diff --git a/.gitattributes b/.gitattributes index 2ab2d3c596..1d88c3ce29 100644 --- a/.gitattributes +++ b/.gitattributes @@ -2,3 +2,4 @@ *.js linguist-language=java *.css linguist-language=java *.html linguist-language=java +*.sh linguist-language=java diff --git a/.gitignore b/.gitignore new file mode 100755 index 0000000000..08ae7cfc9c --- /dev/null +++ b/.gitignore @@ -0,0 +1,6 @@ +node_modules/ +package-lock.json +package.json +yarn.lock +scripts/ +.DS_Store \ No newline at end of file diff --git a/README.md b/README.md index 9bfdbb6c30..4104b77026 100644 --- a/README.md +++ b/README.md @@ -12,18 +12,21 @@

记录并分享每一次成长

- ------ +通过 gitbook 的形式整理了自己的工作和学习经验，[JavaKeeper](http://javakeeper.starfish.ink) 直接访问即可，也推荐大家采用这种形式创建属于自己的“笔记本”，让成长看的见。 + +> 欢迎关注公众号 [JavaKeeper](#公众号) ，有 500+ 本电子书，大佬云集的微信群，等你来撩~ + ## ☕ Java | Project | Version | Article | | :-----: | :-----: | :----------------------------------------------------------- | -| JVM | | [JVM与Java体系结构](/docs/java/JVM/JVM与Java体系结构.md)
[类加载子系统](/docs/java/JVM/类加载子系统.md)
[运行时数据区](/docs/java/JVM/Runtime-Data-Areas.md)
| -| Java8 | | [Java8 通关攻略](/docs/java/java8.md)
| -| JUC | | [不懂Java 内存模型，就先别扯什么高并发](/docs/java/JUC/Java-Memory-Model.md)
[从 Atomic 到 CAS ，竟然衍生出这么多 20k+ 面试题](/docs/java/JUC/volatile.md)
[「阻塞队列」手写生产者消费者、线程池原理面试题真正的答案](https://mp.weixin.qq.com/s/NALM27_K4GIqNmm7kScTAw)
[线程池解毒](/docs/java/JUC/Thread-Pool.md)
| +| JVM | | [JVM与Java体系结构](https://javakeeper.starfish.ink/java/JVM/JVM-Java.html)
[类加载子系统](https://javakeeper.starfish.ink/java/JVM/Class-Loading.html)
[运行时数据区](https://javakeeper.starfish.ink/java/JVM/Runtime-Data-Areas.html)
[看完这篇垃圾回收，和面试官扯皮没问题了](https://javakeeper.starfish.ink/java/JVM/GC.html)
[垃圾回收-实战篇](https://javakeeper.starfish.ink/java/JVM/GC-%E5%AE%9E%E6%88%98.html)
[你有认真了解过自己的Java“对象”吗](https://javakeeper.starfish.ink/java/JVM/Java-Object.html)
[JVM 参数配置](https://javakeeper.starfish.ink/java/JVM/JVM%E5%8F%82%E6%95%B0%E9%85%8D%E7%BD%AE.html)
[谈谈你对 OOM 的认识](https://javakeeper.starfish.ink/java/JVM/OOM.html)
[阿里面试回顾： 说说强引用、软引用、弱引用、虚引用？](https://javakeeper.starfish.ink/java/JVM/Reference.html)
[JVM 性能监控和故障处理工具](https://javakeeper.starfish.ink/java/JVM/JVM%E6%80%A7%E8%83%BD%E7%9B%91%E6%8E%A7%E5%92%8C%E6%95%85%E9%9A%9C%E5%A4%84%E7%90%86%E5%B7%A5%E5%85%B7.html)
| +| Java8 | | [Java8 通关攻略](https://javakeeper.starfish.ink/java/Java-8.html)
| +| JUC | | [不懂Java 内存模型，就先别扯什么高并发](https://javakeeper.starfish.ink/java/JUC/Java-Memory-Model.html)
[面试必问的 volatile，你真的理解了吗](https://javakeeper.starfish.ink/java/JUC/volatile.html)
[从 Atomic 到 CAS ，竟然衍生出这么多 20k+ 面试题](https://javakeeper.starfish.ink/java/JUC/CAS.html)
[「阻塞队列」手写生产者消费者、线程池原理面试题真正的答案](https://javakeeper.starfish.ink/java/JUC/BlockingQueue.html)
[线程池解毒](https://javakeeper.starfish.ink/java/JUC/Thread-Pool.html)
| | NIO | | | @@ -33,8 +36,8 @@ | Project | Version | Article | | :----------------------------------------------------------: | :-----: | :----------------------------------------------------------- | -| ![](https://icongr.am/devicon//mysql-original.svg?size=20) **MySQL** | 5.7.25 | [1、MySQL架构概述](docs/mysql/MySQL-Framework.md)
[2、MySQL存储引擎](docs/mysql/MySQL-Storage-Engines.md)
[3、索引](docs/mysql/MySQL-Index.md)
[4、事务](docs/mysql/MySQL-Transaction.md)
5、表设计
[6、性能优化](docs/mysql/MySQL-Optimization.md)
7、锁机制
8、分区分表分库
9 、主从复制
| -| ![](https://icongr.am/devicon//redis-original.svg?size=20) **Redis** | 5.0.6 | [1、NoSQL概述](docs/data-store/Redis/1.Nosql-Overview.md)
[2、Redis概述](docs/data-store/Redis/2.readRedis.md)
[3、Redis数据类型](docs/data-store/Redis/3.Redis-Datatype.md)
[4、Redis配置](docs/data-store/Redis/4.Redis-Conf.md)
[5、深入理解 Redis 的持久化](docs/data-store/Redis/5.Redis-Persistence.md)
| +| ![](https://icongr.am/devicon//mysql-original.svg?size=20) **MySQL** | 5.7.25 | [1、MySQL架构概述](https://javakeeper.starfish.ink/data-management/MySQL/MySQL-Framework.html)
[2、MySQL存储引擎](https://javakeeper.starfish.ink/data-management/MySQL/MySQL-Storage-Engines.html)
[3、索引](https://javakeeper.starfish.ink/data-management/MySQL/MySQL-Index.html)
[4、事务](https://javakeeper.starfish.ink/data-management/MySQL/MySQL-Transaction.html)
5、表设计
[6、性能优化](docs/data-store/MySQL/MySQL-Optimization.md)
7、锁机制
8、分区分表分库
9 、主从复制
| +| ![](https://icongr.am/devicon//redis-original.svg?size=20) **Redis** | 5.0.6 | [1、NoSQL概述]()
[2、Redis概述](https://javakeeper.starfish.ink/data-management/Redis/ReadRedis.html)
[3、Redis数据类型](https://javakeeper.starfish.ink/data-management/Redis/Redis-Datatype.html)
[4、Redis配置](https://javakeeper.starfish.ink/data-management/Redis/Redis-Conf.html)
[5、深入理解 Redis 的持久化](https://javakeeper.starfish.ink/data-management/Redis/Redis-Persistence.html)
| | **Elasticsearch** | | | | **Amazon S3** | | | | MongoDB | | | @@ -47,7 +50,7 @@ | Project | Version | Article | | :-------: | :-----------------: | :----------------------------------------------------------- | | **Linux** | CentOS release 6.10 | [Linux通关攻略]( ) | -| **Nginx** | 1.16.1 | [Nginx通关攻略](docs/nginx/nginx.md) | +| **Nginx** | 1.16.1 | [Nginx通关攻略](https://mp.weixin.qq.com/s/jA-6tDcrNgd-Wtncj6D6DQ) | @@ -77,7 +80,7 @@ | Project | Version | Article | | :-----: | :-----: | :----------------------------------------------------------- | | MQ | | [Hello MQ](/docs/message-queue/浅谈消息队列及常见的消息中间件.md)
| -| Kafka | 2.12 | [Hello Kafka](/docs/message-queue/Kafka/Hello-Kafka.md)
| +| Kafka | 2.12 | [Hello Kafka](/docs/message-queue/Kafka/Hello-Kafka.md)
[Kafka为什么能那么快的 6 个原因](https://mp.weixin.qq.com/s/dbnpPEF0FBB5A5xH21OoeQ)
| @@ -105,7 +108,7 @@ | Project | Article | | :------------------: | :----------------------------------------------------------- | -| GoF 的 23 种设计模式 | [设计模式前传——要学设计模式你要先知道这些](/docs/design-pattern/Design-Pattern-Overview.md)
[单例模式——我只有一个对象](/docs/design-pattern/Singleton-Pattern.md)
[工厂模式——我有好多对象](/docs/design-pattern/Factory-Pattern.md)
[观察者模式——暗中观察](/docs/design-pattern/Observer-Pattern.md)
[装饰者模式——拒绝继承滥用](/docs/design-pattern/Decorator-Pattern.md)
[责任链模式——更灵活的 if else](/docs/design-pattern/Chain-of-Responsibility-Pattern)
[代理模式——面试官问我Spring AOP中两种代理的区别](https://mp.weixin.qq.com/s/U7eR5Mpu4VBbtPP1livLnA)
| +| GoF 的 23 种设计模式 | [设计模式前传——要学设计模式你要先知道这些](/docs/design-pattern/Design-Pattern-Overview.md)
[单例模式——我只有一个对象](/docs/design-pattern/Singleton-Pattern.md)
[工厂模式——我有好多对象](/docs/design-pattern/Factory-Pattern.md)
[观察者模式——暗中观察](/docs/design-pattern/Observer-Pattern.md)
[装饰者模式——拒绝继承滥用](/docs/design-pattern/Decorator-Pattern.md)
[责任链模式——更灵活的 if else](/docs/design-pattern/Chain-of-Responsibility-Pattern)
[代理模式——面试官问我Spring AOP中两种代理的区别](https://mp.weixin.qq.com/s/U7eR5Mpu4VBbtPP1livLnA)
[原型模式——浅拷贝和深拷贝](http://mp.weixin.qq.com/s?__biz=MzIyNDI3MjY0NQ==&mid=2247485400&idx=1&sn=b83ef5d8d81e54bc46207bf540fc9cf9&chksm=e810cfb2df6746a41e10904fe43611e1385d406a95f680472e72620b91973f8724af9a4d8c37&token=1569764147&lang=zh_CN#rd)
| @@ -168,7 +171,9 @@ -## 扫一扫《毕加索 亚威农少女》，寻找你要的“宝藏” +## 公众号 + +扫一扫《亚威农少女》，寻找你要的“宝藏” ![](https://tva1.sinaimg.cn/large/007S8ZIlly1gf8izv6q5jj30ft0ft4ir.jpg) diff --git a/docs/.DS_Store b/docs/.DS_Store new file mode 100644 index 0000000000..cfba763044 Binary files /dev/null and b/docs/.DS_Store differ diff --git a/docs/.obsidian/app.json b/docs/.obsidian/app.json new file mode 100644 index 0000000000..036417784d --- /dev/null +++ b/docs/.obsidian/app.json @@ -0,0 +1,3 @@ +{ + "communityThemeSortOrder": "download" +} \ No newline at end of file diff --git a/docs/.obsidian/appearance.json b/docs/.obsidian/appearance.json new file mode 100644 index 0000000000..acf703af81 --- /dev/null +++ b/docs/.obsidian/appearance.json @@ -0,0 +1,5 @@ +{ + "theme": "obsidian", + "cssTheme": "Obsidian Nord", + "accentColor": "" +} \ No newline at end of file diff --git a/docs/.obsidian/core-plugins-migration.json b/docs/.obsidian/core-plugins-migration.json new file mode 100644 index 0000000000..318ec8585d --- /dev/null +++ b/docs/.obsidian/core-plugins-migration.json @@ -0,0 +1,31 @@ +{ + "file-explorer": true, + "global-search": true, + "switcher": true, + "graph": true, + "backlink": true, + "outgoing-link": true, + "tag-pane": true, + "page-preview": true, + "daily-notes": true, + "templates": true, + "note-composer": true, + "command-palette": true, + "slash-command": false, + "editor-status": true, + "starred": true, + "markdown-importer": false, + "zk-prefixer": false, + "random-note": false, + "outline": true, + "word-count": true, + "slides": false, + "audio-recorder": false, + "workspaces": false, + "file-recovery": true, + "publish": false, + "sync": false, + "canvas": true, + "bookmarks": true, + "properties": false +} \ No newline at end of file diff --git a/docs/.obsidian/core-plugins.json b/docs/.obsidian/core-plugins.json new file mode 100644 index 0000000000..30410c96ee --- /dev/null +++ b/docs/.obsidian/core-plugins.json @@ -0,0 +1,32 @@ +{ + "file-explorer": true, + "global-search": true, + "switcher": true, + "graph": true, + "backlink": true, + "outgoing-link": true, + "tag-pane": true, + "page-preview": true, + "daily-notes": true, + "templates": true, + "note-composer": true, + "command-palette": true, + "slash-command": false, + "editor-status": true, + "starred": true, + "markdown-importer": false, + "zk-prefixer": false, + "random-note": false, + "outline": true, + "word-count": true, + "slides": false, + "audio-recorder": false, + "workspaces": false, + "file-recovery": true, + "publish": false, + "sync": false, + "canvas": true, + "bookmarks": true, + "properties": false, + "webviewer": false +} \ No newline at end of file diff --git a/docs/.obsidian/graph.json b/docs/.obsidian/graph.json new file mode 100644 index 0000000000..f1c3cdafa8 --- /dev/null +++ b/docs/.obsidian/graph.json @@ -0,0 +1,22 @@ +{ + "collapse-filter": true, + "search": "", + "showTags": false, + "showAttachments": false, + "hideUnresolved": false, + "showOrphans": true, + "collapse-color-groups": true, + "colorGroups": [], + "collapse-display": false, + "showArrow": false, + "textFadeMultiplier": 0, + "nodeSizeMultiplier": 1, + "lineSizeMultiplier": 1, + "collapse-forces": true, + "centerStrength": 0.518713248970312, + "repelStrength": 9.42057291666667, + "linkStrength": 0.345987955729167, + "linkDistance": 250, + "scale": 0.8171482109060346, + "close": true +} \ No newline at end of file diff --git a/docs/.obsidian/hotkeys.json b/docs/.obsidian/hotkeys.json new file mode 100644 index 0000000000..9e26dfeeb6 --- /dev/null +++ b/docs/.obsidian/hotkeys.json @@ -0,0 +1 @@ +{} \ No newline at end of file diff --git a/docs/.obsidian/themes/Minimal.css b/docs/.obsidian/themes/Minimal.css new file mode 100644 index 0000000000..3421fc8c8b --- /dev/null +++ b/docs/.obsidian/themes/Minimal.css @@ -0,0 +1,12636 @@ +/* --------------------------------------------------------------------------- + +Minimal Obsidian 5.3.2 by @kepano + +Important: this is an archived copy of Minimal +only for use with Obsidian 0.15.x and below. + +For Obsidian 0.16+ use Minimal 6.0+ + +--------------------------------------------------------------------------- + +User interface replacement for Obsidian. + +Designed to be used with the Minimal Theme Settings +plugin and the Hider plugin. + +Sponsor my work: +https://www.buymeacoffee.com/kepano + +Readme: +https://github.com/kepano/obsidian-minimal + +----------------------------------------------------------------------------- + +MIT License + +Copyright (c) 2020-2022 Stephan Ango (@kepano) + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in +all copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +*/ +@charset "UTF-8"; +/* Variables */ +body { + --font-text-theme:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Inter,Ubuntu,sans-serif; + --font-editor-theme:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Inter,Ubuntu,sans-serif; + --font-monospace-theme:Menlo,SFMono-Regular,Consolas,"Roboto Mono",monospace; + --font-interface-theme:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Inter,Ubuntu,sans-serif; + --font-editor:var(--font-editor-override), var(--font-text-override), var(--font-editor-theme); + --minimal-version:"You are currently using Minimal 5.3.2\a\aIf you run into any issues, try updating to the latest version of the theme. It is also highly recommended to install Minimal Theme Settings and Contextual Typography plugins.\a\a Full documentation:\a minimal.guide\a\a Support my work:\a buymeacoffee.com/kepano"; } + +:root { + /* Cursor */ + --cursor:default; + /* Font sizes */ + --font-small:13px; + --font-smaller:11px; + --font-smallest:10px; + --font-inputs:13px; + --font-settings:15px; + --font-settings-small:12px; + /* Font weights */ + --normal-weight:400; + --bold-weight:600; + --link-weight:inherit; + /* Headings */ + --title-size:1.1em; + --title-weight:600; + /* Headings */ + --h1:1.125em; + --h2:1.05em; + --h3:1em; + --h4:0.90em; + --h5:0.85em; + --h6:0.85em; + --h1-weight:600; + --h2-weight:600; + --h3-weight:500; + --h4-weight:500; + --h5-weight:500; + --h6-weight:400; + --h1-variant:normal; + --h2-variant:normal; + --h3-variant:normal; + --h4-variant:small-caps; + --h5-variant:small-caps; + --h6-variant:small-caps; + --h1-style:normal; + --h2-style:normal; + --h3-style:normal; + --h4-style:normal; + --h5-style:normal; + --h6-style:normal; + /* Blockquotes */ + --blockquote-style:normal; + /* Line widths */ + --line-width:40rem; + --line-height:1.5; + --max-width:88%; + --max-col-width:18em; + /* Icons */ + --icon-muted:0.5; + --icon-size:18px; + --border-width:1px; + --border-width-alt:1px; + /* Quotes and transclusions */ + --nested-padding:1.1em; + /* Lists */ + --folding-offset:10px; + --list-edit-offset:0.5em; + --list-indent:2em; + --list-spacing:0.075em; + /* Radiuses */ + --radius-s:2px; + --radius-m:5px; + --radius-l:12px; + --radius-xl:16px; + --input-height:32px; + --header-height:40px; + /* Mobile sidebars */ + --mobile-left-sidebar-width:280pt; + --mobile-right-sidebar-width:240pt; + /* Tags */ + --tag-radius:14px; + --tag-border-width:1px; + --top-left-padding-y:0px; + /* Image opacity in dark mode */ + --image-muted:0.7; + /* Spacing */ + --spacing-p: 0.75em; } + +.mod-macos { + --top-left-padding-y:24px; } + +/* Dynamic colors + + Most colors are driven from the following values, meaning that + the backgrounds, borders, and various shades are + automatically generated. + + - Base color is used for the backgrounds, text and borders. + - Accent color is used for links and some interactive elements. + + The colors use HSL (hue, saturation, lightness) + + - Hue (0-360 degrees):0 is red, 120 is green, and 240 is blue + - Saturation (0-100%):0% is desaturated, 100% is full saturation + - Lightness (0-100%):0% is black, 100% is white + +*/ +:root { + --base-h:0; + /* Base hue */ + --base-s:0%; + /* Base saturation */ + --base-d:15%; + /* Base lightness Dark Mode - 0 is black */ + --base-l:96%; + /* Base lightness Light Mode - 100 is white */ + --accent-h:201; + /* Accent hue */ + --accent-s:17%; + /* Accent saturation */ + --accent-d:60%; + /* Accent lightness Dark Mode */ + --accent-l:50%; + /* Accent lightness Light Mode */ + --red:#d04255; + --yellow:#e5b567; + --green:#a8c373; + --orange:#d5763f; + --cyan:#73bbb2; + --blue:#6c99bb; + --purple:#9e86c8; + --pink:#b05279; } + +.theme-light, +.theme-light.minimal-default-light, +body .excalidraw { + --accent-l:50%; + --base-l:96%; + --bg1:white; + --bg2: + hsl( + var(--base-h), + var(--base-s), + var(--base-l) + ); + --bg3: + hsla( + var(--base-h), + var(--base-s), + calc(var(--base-l) - 50%), + 0.12 + ); + --ui1: + hsl( + var(--base-h), + var(--base-s), + calc(var(--base-l) - 6%) + ); + --ui2: + hsl( + var(--base-h), + var(--base-s), + calc(var(--base-l) - 12%) + ); + --ui3: + hsl( + var(--base-h), + var(--base-s), + calc(var(--base-l) - 20%) + ); + --tx1: + hsl( + var(--base-h), + var(--base-s), + calc(var(--base-l) - 90%) + ); + --tx2: + hsl( + var(--base-h), + calc(var(--base-s) - 20%), + calc(var(--base-l) - 45%) + ); + --tx3: + hsl( + var(--base-h), + calc(var(--base-s) - 10%), + calc(var(--base-l) - 25%) + ); + --tx4: + hsl( + var(--base-h), + calc(var(--base-s) - 10%), + calc(var(--base-l) - 60%) + ); + --ax1: + hsl( + var(--accent-h), + var(--accent-s), + var(--accent-l) + ); + --ax2: + hsl( + var(--accent-h), + var(--accent-s), + calc(var(--accent-l) - 10%) + ); + --ax3: + hsl( + var(--accent-h), + var(--accent-s), + calc(var(--accent-l) + 10%) + ); + --hl1: + hsla( + var(--accent-h), + 50%, + calc(var(--base-l) - 20%), + 30% + ); + --hl2:rgba(255, 225, 0, 0.5); } + +.theme-light.minimal-light-contrast .titlebar, +.theme-light.minimal-light-contrast .workspace-fake-target-overlay.is-in-sidebar, +.theme-light.minimal-light-contrast .workspace-ribbon.mod-left:not(.is-collapsed), +.theme-light.minimal-light-contrast .mod-left-split, +.theme-light.minimal-light-contrast.minimal-status-off .status-bar, +.theme-light.minimal-light-contrast.is-mobile .workspace-drawer.mod-left, +.theme-dark, +.theme-dark.minimal-default-dark, +.excalidraw.theme--dark { + --accent-l:60%; + --base-l:15%; + --bg1: + hsl( + var(--base-h), + var(--base-s), + var(--base-l) + ); + --bg2: + hsl( + var(--base-h), + var(--base-s), + calc(var(--base-l) - 2%) + ); + --bg3: + hsla( + var(--base-h), + var(--base-s), + calc(var(--base-l) + 40%), + 0.12 + ); + --ui1: + hsl( + var(--base-h), + var(--base-s), + calc(var(--base-l) + 6%) + ); + --ui2: + hsl( + var(--base-h), + var(--base-s), + calc(var(--base-l) + 12%) + ); + --ui3: + hsl( + var(--base-h), + var(--base-s), + calc(var(--base-l) + 20%) + ); + --tx1: + hsl( + var(--base-h), + calc(var(--base-s) - 10%), + calc(var(--base-l) + 67%) + ); + --tx2: + hsl( + var(--base-h), + calc(var(--base-s) - 20%), + calc(var(--base-l) + 45%) + ); + --tx3: + hsl( + var(--base-h), + calc(var(--base-s) - 10%), + calc(var(--base-l) + 20%) + ); + --tx4: + hsl( + var(--base-h), + calc(var(--base-s) - 10%), + calc(var(--base-l) + 50%) + ); + --ax1: + hsl( + var(--accent-h), + var(--accent-s), + var(--accent-l) + ); + --ax2: + hsl( + var(--accent-h), + var(--accent-s), + calc(var(--accent-l) + 12%) + ); + --ax3: + hsl( + var(--accent-h), + var(--accent-s), + calc(var(--accent-l) - 12%) + ); + --hl1: + hsla( + var(--accent-h), + 70%, + 40%, + 30% + ); + --hl2:rgba(255, 177, 80, 0.3); + --sp1:#fff; } + +.theme-light.minimal-light-white { + --background-primary: white; + --background-secondary: white; + --background-secondary-alt: white; } + +.theme-dark.minimal-dark-black { + --base-d:0%; + --background-primary: black; + --background-secondary: black; + --background-secondary-alt: black; + --background-tertiary: + hsl( + var(--base-h), + var(--base-s), + calc(var(--base-d) + 10%)) ; + --tx1:hsl( + var(--base-h), + var(--base-s), + calc(var(--base-d) + 75%) + ); + --tx2:hsl( + var(--base-h), + var(--base-s), + calc(var(--base-d) + 50%) + ); + --tx3:hsl( + var(--base-h), + var(--base-s), + calc(var(--base-d) + 25%) + ); + --ui1:hsl( + var(--base-h), + var(--base-s), + calc(var(--base-d) + 12%) + ); + --ui2:hsl( + var(--base-h), + var(--base-s), + calc(var(--base-d) + 20%) + ); + --ui3:hsl( + var(--base-h), + var(--base-s), + calc(var(--base-d) + 30%) + ); } + +/* Map colors to semantic Obsidian names */ +.theme-light { + --mono100:black; + --mono0:white; } + +.theme-dark { + --mono100:white; + --mono0:black; } + +.theme-light, +.theme-dark { + --h1-color:var(--text-normal); + --h2-color:var(--text-normal); + --h3-color:var(--text-normal); + --h4-color:var(--text-normal); + --h5-color:var(--text-normal); + --h6-color:var(--text-muted); } + +.theme-light.minimal-light-contrast .workspace-fake-target-overlay.is-in-sidebar, +.theme-light.minimal-light-contrast .titlebar, +.theme-light.minimal-light-contrast .workspace-ribbon.mod-left:not(.is-collapsed), +.theme-light.minimal-light-contrast .mod-left-split, +.theme-light.minimal-light-contrast.minimal-status-off .status-bar, +.theme-light.minimal-light-contrast.is-mobile .workspace-drawer.mod-left, +.theme-dark, +.theme-light, +.excalidraw.theme--dark, +body .excalidraw { + --text-normal: var(--tx1); + --text-bold: var(--tx1); + --text-italic: var(--tx1); + --text-muted: var(--tx2); + --text-faint: var(--tx3); + --title-color: var(--tx1); + --title-color-inactive: var(--tx2); + --text-code: var(--tx4); + --text-error: var(--red); + --text-blockquote: var(--tx2); + --text-accent: var(--ax1); + --text-accent-hover: var(--ax2); + --text-on-accent: white; + --text-selection: var(--hl1); + --text-highlight-bg: var(--hl2); + --background-primary: var(--bg1); + --background-primary-alt: var(--bg2); + --background-secondary: var(--bg2); + --background-secondary-alt: var(--bg1); + --background-tertiary: var(--bg3); + --background-table-rows: var(--bg2); + --background-modifier-form-field: var(--bg1); + --background-modifier-form-field-highlighted: + var(--bg1); + --interactive-hover: var(--ui1); + --interactive-accent: var(--ax3); + --interactive-accent-hover: var(--ax3); + --background-modifier-accent: var(--ax3); + --background-modifier-border: var(--ui1); + --background-modifier-border-hover: var(--ui2); + --background-modifier-border-focus: var(--ui3); + --background-modifier-success: var(--green); + --background-divider: var(--ui1); + --scrollbar-bg: transparent; + --scrollbar-thumb-bg: var(--ui1); + --scrollbar-active-thumb-bg: var(--ui3); + --quote-opening-modifier: var(--ui2); + --modal-border: var(--ui2); + --icon-color: var(--tx2); + --icon-color-hover: var(--tx2); + --icon-color-active: var(--tx1); + --icon-hex: var(--mono0); + --tag-color: var(--tx2); + --tag-bg: transparent; + --tag-bg2: transparent; + --shadow-m: + 0px 2.7px 6.7px rgba(0, 0, 0, 0.04), + 0px 8.9px 22.3px rgba(0, 0, 0, 0.06), + 0px 40px 100px rgba(0, 0, 0, 0.1); + --shadow-l: + 0px 0.8px 4.2px rgba(0, 0, 0, 0.014), + 0px 2px 10px rgba(0, 0, 0, 0.02), + 0px 3.8px 18.8px rgba(0, 0, 0, 0.025), + 0px 6.7px 33.5px rgba(0, 0, 0, 0.03), + 0px 12.5px 62.7px rgba(0, 0, 0, 0.036), + 0px 30px 150px rgba(0, 0, 0, 0.05); } + +.theme-light, +body .excalidraw { + --interactive-normal: var(--bg1); + --interactive-accent-rgb:220, 220, 220; + --active-line-bg:rgba(0,0,0,0.035); + --background-modifier-cover:hsla(var(--base-h),calc(var(--base-s) - 50%),calc(var(--base-l) - 7%),0.7); + --text-highlight-bg-active:rgba(0, 0, 0, 0.1); + /* Errors */ + --background-modifier-error:rgba(255,0,0,0.14); + --background-modifier-error-hover:rgba(255,0,0,0.08); + /* Shadows */ + --shadow-color:rgba(0, 0, 0, 0.1); + --btn-shadow-color:rgba(0, 0, 0, 0.05); } + +.theme-dark, +.excalidraw.theme--dark { + --interactive-normal: var(--bg3); + --interactive-accent-rgb:66, 66, 66; + --active-line-bg:rgba(255,255,255,0.04); + --background-modifier-cover:hsla(var(--base-h),var(--base-s),calc(var(--base-d) - 12%),0.7); + --text-highlight-bg-active:rgba(255, 255, 255, 0.1); + /* Errors */ + --background-modifier-error:rgba(255,20,20,0.12); + --background-modifier-error-hover:rgba(255,20,20,0.18); + /* Shadows */ + --background-modifier-box-shadow:rgba(0, 0, 0, 0.3); + --shadow-color:rgba(0, 0, 0, 0.3); + --btn-shadow-color:rgba(0, 0, 0, 0.2); } + +.theme-light.minimal-light-white { + --background-table-rows: var(--bg2); } + +.theme-light.minimal-light-tonal { + --background-primary: var(--bg2); + --background-primary-alt: var(--bg3); + --background-table-rows: var(--bg3); } + +.theme-dark.minimal-dark-tonal { + --background-secondary: var(--bg1); + --background-table-rows: var(--bg3); } + +.theme-dark.minimal-dark-black { + --background-primary-alt: var(--bg3); + --background-table-rows: var(--bg3); + --modal-border: var(--ui2); + --active-line-bg:rgba(255,255,255,0.085); + --background-modifier-form-field: var(--bg3); + --background-modifier-cover:hsla(var(--base-h),var(--base-s),calc(var(--base-d) + 8%),0.9); + --background-modifier-box-shadow:rgba(0, 0, 0, 1); } + +/* Desktop font sizes */ +body { + --font-adaptive-normal:var(--font-text-size,var(--editor-font-size)); + --font-adaptive-small:var(--font-small); + --font-adaptive-smaller:var(--font-smaller); + --font-adaptive-smallest:var(--font-smallest); + --line-width-adaptive:var(--line-width); + --line-width-wide:calc(var(--line-width) + 12.5%); + --font-code:calc(var(--font-adaptive-normal) * 0.9); + --table-font-size:calc(var(--font-adaptive-normal) * 0.875); } + +/* Phone font sizes */ +@media (max-width: 400pt) { + .is-mobile { + --font-adaptive-small:calc(var(--font-small) + 2px); + --font-adaptive-smaller:calc(var(--font-smaller) + 2px); + --font-adaptive-smallest:calc(var(--font-smallest) + 2px); + --max-width:88%; } } +/* Tablet font sizes */ +@media (min-width: 400pt) { + .is-mobile { + --font-adaptive-small:calc(var(--font-small) + 3px); + --font-adaptive-smaller:calc(var(--font-smaller) + 2px); + --font-adaptive-smallest:calc(var(--font-smallest) + 2px); + --line-width-adaptive:calc(var(--line-width) + 6rem); + --max-width:90%; } } +/* Disabled features */ +/* Disabled features */ +/* Search counts */ +.tree-item-flair:not(.tag-pane-tag-count) { + display: none; } + +/* Folder name */ +.tree-item-inner-subtext { + display: none; } + +/* Obsidian */ +/* Block width snippet */ +.minimal-dev-block-width { + /* Green — Folding offset width */ + /* Red — Max width */ + /* Orange — Wide line width*/ + /* Blue — Normal line width */ } + .minimal-dev-block-width .mod-root .workspace-leaf-content:after { + display: flex; + align-items: flex-end; + content: "\00a0pane\00a0"; + font-size: 12px; + color: gray; + font-family: var(--font-monospace); + width: 100%; + max-width: 100%; + height: 100vh; + top: 0; + z-index: 999; + position: fixed; + pointer-events: none; } + .minimal-dev-block-width.minimal-readable .mod-root .view-header:after { + display: flex; + align-items: flex-end; + color: green; + font-size: 12px; + font-family: var(--font-monospace); + content: " "; + width: var(--folding-offset); + height: 100vh; + border-left: 1px solid green; + border-right: 1px solid green; + background-color: rgba(0, 128, 0, 0.1); + top: 0; + left: max(calc(50% - var(--line-width-adaptive)/2 - 1px), calc(50% - var(--max-width)/2 - 1px)); + z-index: 999; + position: fixed; + pointer-events: none; } + .minimal-dev-block-width.minimal-readable-off .mod-root .view-header:after { + display: flex; + align-items: flex-end; + color: green; + font-size: 12px; + font-family: var(--font-monospace); + content: " "; + width: var(--folding-offset); + height: 100vh; + border-left: 1px solid green; + border-right: 1px solid green; + background-color: rgba(0, 128, 0, 0.1); + top: 0; + left: calc(50% - var(--max-width)/2 - 1px); + z-index: 999; + position: fixed; + pointer-events: none; } + .minimal-dev-block-width .mod-root .view-content:before { + display: flex; + align-items: flex-end; + content: "\00a0max\00a0"; + font-size: 12px; + color: red; + width: var(--max-width); + height: 100vh; + border-left: 1px solid red; + border-right: 1px solid red; + top: 0; + left: 50%; + transform: translate(-50%, 0); + z-index: 999; + position: fixed; + pointer-events: none; } + .minimal-dev-block-width.minimal-readable .mod-root .view-header:before { + display: flex; + align-items: flex-end; + content: "\00a0wide\00a0"; + font-size: 12px; + color: orange; + font-family: var(--font-monospace); + width: var(--line-width-wide); + max-width: var(--max-width); + height: 100vh; + border-left: 1px solid orange; + border-right: 1px solid orange; + background-color: rgba(255, 165, 0, 0.05); + top: 0; + left: 50%; + transform: translate(-50%, 0); + z-index: 999; + position: fixed; + pointer-events: none; } + .minimal-dev-block-width.minimal-readable .mod-root .view-content:after { + display: flex; + align-items: flex-end; + color: blue; + font-size: 12px; + font-family: var(--font-monospace); + content: "\00a0normal"; + width: var(--line-width-adaptive); + max-width: var(--max-width); + height: 100vh; + border-left: 1px solid blue; + border-right: 1px solid blue; + background-color: rgba(0, 0, 255, 0.08); + top: 0; + left: 50%; + transform: translate(-50%, 0); + z-index: 999; + position: fixed; + pointer-events: none; } + +/* Obsidian */ +/* Blockquotes */ +/* Preview */ +.markdown-preview-view blockquote { + border-radius: 0; + border: solid var(--quote-opening-modifier); + border-width: 0px 0px 0px 1px; + background-color: transparent; + padding: 0 0 0 var(--nested-padding); + margin-inline-start: 0; + margin-inline-end: 0; + font-size: var(--blockquote-size); + font-style: var(--blockquote-style); + color: var(--text-blockquote); } + +.cm-s-obsidian span.cm-quote, +.markdown-preview-view blockquote em, +.markdown-preview-view blockquote strong { + color: var(--text-blockquote); } + +/* Editor */ +.markdown-source-view.mod-cm6.is-live-preview .HyperMD-quote, +.markdown-source-view.mod-cm6 .HyperMD-quote { + background-color: transparent; + color: var(--text-blockquote); + font-size: var(--blockquote-size); + font-style: var(--blockquote-style); + border-left: 1px solid var(--quote-opening-modifier); } + +.markdown-source-view.mod-cm6 .cm-blockquote-border { + width: 20px; + display: inline-block; + border-left: none; + border-right: 1px solid var(--quote-opening-modifier); } + +.markdown-source-view.mod-cm6 .cm-hmd-indent-in-quote { + margin-left: 10px; } + +.is-live-preview .cm-hmd-indent-in-quote { + color: var(--text-faint); } + +/* Callouts */ +.is-live-preview.is-readable-line-width > .cm-callout .callout { + max-width: var(--max-width); + margin: 0 auto; } + +/* Checklists, task lists, checkboxes */ +:root { + --checkbox-size:17px; + --checkbox-icon:20px; + --checkbox-radius:50%; + --checkbox-top:2px; + --checkbox-left:0px; + --checkbox-margin:0px 6px 0px -1.35em; } + +.checkbox-square { + --checkbox-size:15px; + --checkbox-icon:17px; + --checkbox-radius:4px; + --checkbox-top:1px; + --checkbox-left:0px; + --checkbox-margin:0px 8px 0px -1.35em; } + +input[type=checkbox] { + -webkit-appearance: none; + appearance: none; + border-radius: var(--checkbox-radius); + border: 1px solid var(--text-faint); + padding: 0; + margin: 0 6px 0 0; + width: var(--checkbox-size); + height: var(--checkbox-size); } + +input[type=checkbox]:hover, +input[type=checkbox]:focus { + outline: 0; + border-color: var(--text-muted); } + +.checklist-plugin-main .group .compact > .toggle .checked, +.is-flashing input[type=checkbox]:checked, +input[type=checkbox]:checked { + background-color: var(--background-modifier-accent); + border: 1px solid var(--background-modifier-accent); + background-position: 44% 55%; + background-size: 70%; + background-repeat: no-repeat; + background-image: url('data:image/svg+xml; utf8, '); } + +.markdown-preview-section > .contains-task-list { + padding-bottom: 0.5em; } + +body .markdown-source-view.mod-cm6 .HyperMD-task-line[data-task]:not([data-task=" "]), +body .markdown-preview-view ul > li.task-list-item.is-checked { + text-decoration: none; + color: var(--text-normal); } + +body.minimal-strike-lists .markdown-source-view.mod-cm6 .HyperMD-task-line[data-task]:is([data-task="x"]), +body.minimal-strike-lists .markdown-preview-view ul li[data-task="x"].task-list-item.is-checked, +body.minimal-strike-lists li[data-task="x"].task-list-item.is-checked { + color: var(--text-faint); + text-decoration: line-through; } + +/* Preview offset */ +ul > li.task-list-item .task-list-item-checkbox { + margin-left: -1.35em; } + +/* Editor */ +.mod-cm6 .HyperMD-task-line[data-task] .task-list-item-checkbox { + margin: -2px 1px 0 -0.6em; } + +.is-mobile .mod-cm6 .HyperMD-task-line[data-task] .task-list-item-checkbox { + margin-left: -0.4em; } + +.is-mobile .markdown-preview-view input[type=checkbox].task-list-item-checkbox { + top: 0.2em; } + +.task-list-item-checkbox, +.markdown-preview-view .task-list-item-checkbox { + filter: none; + width: var(--checkbox-size); + height: var(--checkbox-size); } + +.markdown-preview-view .task-list-item-checkbox { + position: relative; + top: var(--checkbox-top); + left: var(--checkbox-left); + line-height: 0; + margin: var(--checkbox-margin); } + +.markdown-preview-view ul > li.task-list-item { + text-indent: 0; + line-height: var(--line-height); } + +.markdown-preview-view .task-list-item { + padding-inline-start: 0; } + +.side-dock-plugin-panel-inner { + padding-right: 6px; + padding-left: 6px; } + +/* Code blocks */ +/* Live Preview */ +.markdown-source-view.mod-cm6.is-readable-line-width .cm-editor .HyperMD-codeblock.cm-line, +.mod-cm6 .cm-editor .HyperMD-codeblock.cm-line { + padding-left: 10px; + padding-right: 10px; } + +/* Reading */ +.cm-s-obsidian span.cm-inline-code, +.markdown-rendered code, +.markdown-preview-view code { + color: var(--text-code); + font-size: var(--font-code); } + +.markdown-preview-view td code, +.markdown-source-view.mod-cm6 td code { + font-size: calc(var(--font-code) - 2px); } + +.markdown-preview-view pre code { + background-color: transparent; } + +.markdown-preview-view pre, +.markdown-source-view.mod-cm6 .cm-preview-code-block pre.dataview-error, +.mod-cm6 .cm-editor .HyperMD-codeblock.cm-line, +.cm-s-obsidian .HyperMD-codeblock { + color: var(--text-code); + font-size: var(--font-code); } + +button.copy-code-button { + cursor: var(--cursor); + box-shadow: none; + font-size: var(--font-adaptive-smaller); + background-color: transparent; + color: var(--text-faint); + padding: 0.25em 0.75em; } + +button.copy-code-button:hover { + background-color: var(--interactive-normal); + color: var(--text-muted); } + +.theme-light :not(pre) > code[class*="language-"], +.theme-light pre[class*="language-"] { + background-color: var(--background-primary-alt); } + +.theme-light code[class*="language-"], +.theme-light pre[class*="language-"] { + text-shadow: none; } + +.markdown-source-view.mod-cm6 .code-block-flair { + font-size: var(--font-smaller); + padding: 5px 0; + color: var(--text-muted); } + +.cm-s-obsidian .hmd-fold-html-stub, +.cm-s-obsidian .hmd-fold-code-stub, +.cm-s-obsidian.CodeMirror .HyperMD-hover > .HyperMD-hover-content code, +.cm-s-obsidian .cm-formatting-hashtag, +.cm-s-obsidian .cm-inline-code, +.cm-s-obsidian .HyperMD-codeblock, +.cm-s-obsidian .HyperMD-hr, +.cm-s-obsidian .cm-hmd-frontmatter, +.cm-s-obsidian .cm-hmd-orgmode-markup, +.cm-s-obsidian .cm-formatting-code, +.cm-s-obsidian .cm-math, +.cm-s-obsidian span.hmd-fold-math-placeholder, +.cm-s-obsidian .CodeMirror-linewidget kbd, +.cm-s-obsidian .hmd-fold-html kbd +.CodeMirror-code { + font-family: var(--font-monospace); } + +/* Drag ghost */ +body.is-dragging { + cursor: grabbing; + cursor: -webkit-grabbing; } + +.workspace-drop-overlay:before, +.mod-drag { + opacity: 0; + border-radius: 0 !important; } + +.drag-ghost, +.drag-ghost.mod-leaf { + border: none; + background-color: rgba(0, 0, 0, 0.7); + font-size: var(--font-adaptive-small); + padding: 3px 8px 4px; + color: white; + font-weight: 500; + border-radius: 5px; } + +.drag-ghost-icon { + display: none; } + +.drag-ghost-self svg { + margin-right: 4px; + opacity: 0.5; + display: none; } + +.drag-ghost-action { + padding: 0; + font-weight: 400; + color: rgba(255, 255, 255, 0.7); + font-size: var(--font-adaptive-smaller); } + +.mod-drag { + opacity: 0; + border: 2px solid var(--text-accent); + background-color: var(--background-primary); } + +.view-header.is-highlighted:after { + background-color: var(--text-selection); } + +.view-header.is-highlighted .view-actions { + background: transparent; } + +/* +.workspace-fake-target-overlay, +.workspace-fake-target-overlay.is-in-sidebar, +.workspace-drop-overlay, +.view-header.is-highlighted:after { + opacity:0; + background-color:var(--background-primary); +} +*/ +/* Editor mode (CodeMirror 6 Live Preview) */ +/* Fix strange Obsidian ghost textearea bug on right click */ +.CodeMirror-wrap > div > textarea { + opacity: 0; } + +.markdown-source-view.mod-cm6 hr { + border-width: 2px; } + +.mod-cm6 .cm-editor .cm-line { + padding: 0; } + +.cm-editor .cm-content { + padding-top: 0.5em; } + +.markdown-source-view { + color: var(--text-normal); } + +.markdown-source-view.mod-cm6 .cm-scroller { + padding-top: 15px; + padding-left: 0; + padding-right: 0; } + +/* Gutters */ +body:not(.is-mobile) .markdown-source-view.mod-cm6 .cm-gutters { + position: absolute !important; + z-index: 0; } + +.cm-editor .cm-lineNumbers .cm-gutterElement { + min-width: 25px; } + +/* Line numbers */ +@media (max-width: 400pt) { + .cm-editor .cm-lineNumbers .cm-gutterElement { + padding-right: 4px; + padding-left: 8px; } } +.cm-editor .cm-lineNumbers .cm-gutterElement { + font-variant-numeric: tabular-nums; } + +.cm-editor .cm-lineNumbers .cm-gutterElement.cm-active, +.cm-editor .cm-gutterElement.cm-active .cm-heading-marker { + color: var(--text-muted); } + +/* Code execution blocks, e.g. Dataview */ +.markdown-source-view.mod-cm6 .edit-block-button { + cursor: var(--cursor); + color: var(--text-faint); + background-color: var(--background-primary); + top: 0; + right: auto; + left: 0px; + opacity: 0; + transition: opacity 200ms; + padding: 4px 4px 4px 9px; } + .markdown-source-view.mod-cm6 .edit-block-button svg { + margin: 0 !important; } + +.markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-embed-block > .edit-block-button { + width: 30px !important; + padding-left: 7px !important; + transform: none !important; + margin-left: 0 !important; } + +.is-live-preview:not(.is-readable-line-width) .cm-embed-block > .edit-block-button { + padding-left: 0px !important; + margin-left: 0 !important; + transform: none !important; + right: 0; + left: auto; + padding: 4px; } + +.markdown-source-view.mod-cm6 .edit-block-button:hover { + background-color: var(--background-primary); + color: var(--text-muted); } + +.markdown-source-view.mod-cm6 .edit-block-button svg { + opacity: 1; + width: var(--icon-size); + height: var(--icon-size); } + +.markdown-source-view.mod-cm6 .edit-block-button:hover svg { + opacity: 1; } + +.markdown-source-view.mod-cm6 .cm-embed-block { + padding: 0; + border: 0; + border-radius: 0; } + +.markdown-source-view.mod-cm6 .cm-embed-block:hover { + border: 0; } + +/* Live Preview folding */ +.markdown-source-view.mod-cm6 .cm-foldPlaceholder { + color: var(--text-faint); } + +.markdown-source-view.mod-cm6.is-live-preview .HyperMD-quote { + background-color: transparent; + border-left-width: 1px; } + +.cm-editor .cm-foldPlaceholder, +.markdown-source-view.mod-cm6 .cm-fold-indicator .collapse-indicator { + cursor: var(--cursor); } + +.markdown-source-view.mod-cm6 .HyperMD-list-line.HyperMD-list-line-1 .cm-fold-indicator .collapse-indicator { + right: 8px; } + +.markdown-source-view.mod-cm6 .HyperMD-list-line.HyperMD-task-line:not(.HyperMD-list-line-1) .cm-fold-indicator .collapse-indicator { + right: 8px; + width: auto; } + +.markdown-source-view.mod-cm6 .HyperMD-list-line:not(.HyperMD-list-line-1) .cm-fold-indicator .collapse-indicator { + right: -8px; + top: 1px; + width: 26px; } + +ul > li.is-collapsed::marker, +.markdown-source-view.mod-cm6 .is-collapsed ~ .cm-formatting-list .list-bullet:after { + color: var(--text-accent); } + +.cm-gutterElement .collapse-indicator, +.markdown-source-view.mod-cm6 .cm-fold-indicator .collapse-indicator, +.markdown-source-view.mod-cm6 .fold-gutter { + opacity: 0; } + +.cm-gutterElement:hover .collapse-indicator, +.cm-gutterElement .is-collapsed .collapse-indicator, +.markdown-source-view.mod-cm6 .cm-line:hover .cm-fold-indicator .collapse-indicator, +.markdown-source-view.mod-cm6 .cm-fold-indicator.is-collapsed .collapse-indicator, +.markdown-source-view.mod-cm6 .fold-gutter.is-collapsed, +.markdown-source-view.mod-cm6 .fold-gutter:hover, +.markdown-source-view.mod-cm6 .cm-fold-indicator.is-collapsed .collapse-indicator svg { + opacity: 1; } + +/* Live Preview text selection */ +.markdown-source-view.mod-cm6 .cm-line .cm-selection, +.markdown-source-view.mod-cm6 .cm-line .cm-inline-code .cm-selection { + background-color: var(--text-selection); } + +.cm-selectionBackground { + background-color: transparent !important; } + +body .markdown-source-view.mod-cm6.is-readable-line-width:not(.is-rtl) .cm-contentContainer { + max-width: 100%; } + +body:not(.is-mobile).minimal-folding .markdown-source-view.mod-cm6.is-readable-line-width .cm-contentContainer { + max-width: 100%; } + +/* Editor mode (Legacy) */ +.theme-light .token.operator, +.theme-light .token.entity, +.theme-light .token.url, +.theme-light .language-css .token.string, +.theme-light .style .token.string, +.theme-light .cm-operator, +.theme-light .cm-string, +.theme-light .cm-string-2, +.theme-light .cm-link { + background-color: transparent; } + +.markdown-source-view.mod-cm6, +.markdown-source-view.mod-cm5, +.markdown-source-view { + padding: 0; } + +.cm-s-obsidian .CodeMirror-code { + padding-right: 0; } + +.CodeMirror-lines { + padding-bottom: 170px; } + +.CodeMirror pre.CodeMirror-line, +.CodeMirror pre.CodeMirror-line-like { + padding-left: 0; + padding-right: 0; } + +.cm-s-obsidian pre.HyperMD-list-line { + padding-top: 0; } + +.workspace .markdown-preview-view { + padding: 0; } + +.workspace .markdown-preview-view .markdown-embed { + margin: 0; } + +.workspace .markdown-preview-view .markdown-embed-content { + max-height: none; } + +.markdown-embed-title, +.internal-embed .markdown-preview-section { + max-width: 100%; } + +.CodeMirror-linenumber { + font-size: var(--font-adaptive-small) !important; + font-feature-settings: 'tnum'; + color: var(--text-faint); + padding-top: 3px; } + +span.cm-image-marker, +.cm-s-obsidian span.cm-footref.cm-formatting.cm-formatting-link.cm-formatting-link-end, +.cm-s-obsidian .cm-formatting-link + span.cm-link.cm-formatting.cm-formatting-link-end, +.cm-s-obsidian .cm-active span.cm-link.cm-hmd-barelink.cm-formatting-link-start, +.cm-s-obsidian span.cm-link.cm-hmd-barelink.cm-formatting-link-start, +.cm-s-obsidian span.cm-formatting-link { + color: var(--text-faint); } + +/* Editor Mode Footnotes */ +.cm-s-obsidian span.cm-footref { + font-size: var(--font-adaptive-normal); } + +.cm-s-obsidian pre.HyperMD-footnote { + font-size: var(--font-adaptive-small); + padding-left: 20px; } + +/* Editor Mode Quotes */ +.cm-formatting-quote { + color: var(--text-faint) !important; } + +/* Transcluded notes and embeds */ +/* Strict embeds (naked) */ +.embed-strict .internal-embed .markdown-embed { + padding: 0; + border: none; } + +.embed-strict .internal-embed .markdown-embed .markdown-embed-title { + display: none; } + +.embed-strict .internal-embed:not([src*="#^"]) .markdown-embed-link { + width: 30px; } + +.embed-strict.contextual-typography .internal-embed .markdown-preview-view .markdown-preview-sizer > div, +.contextual-typography .embed-strict .internal-embed .markdown-preview-view .markdown-preview-sizer > div { + margin: 0; + width: 100%; } + +.markdown-embed .markdown-preview-view .markdown-preview-sizer { + padding-bottom: 0 !important; } + +.markdown-preview-view.markdown-embed .markdown-preview-sizer, +.markdown-preview-view.is-readable-line-width .markdown-embed .markdown-preview-sizer { + max-width: 100%; + width: 100%; + min-height: 0 !important; + padding-bottom: 0 !important; } + +.markdown-embed .markdown-preview-section div:last-child p, +.markdown-embed .markdown-preview-section div:last-child ul { + margin-block-end: 2px; } + +.markdown-preview-view .markdown-embed { + margin-top: var(--nested-padding); + padding: 0 calc(var(--nested-padding) / 2) 0 var(--nested-padding); } + +.markdown-embed-title { + line-height: 18px; + height: 24px; } + +.internal-embed:not([src*="#^"]) .markdown-embed-link { + right: 0; + width: 100%; } + +.markdown-embed-link, +.file-embed-link { + top: 0px; + right: 0; + text-align: right; } + +.file-embed-link svg, +.markdown-embed-link svg { + width: 16px; + height: 16px; + opacity: 0; } + +.markdown-embed .file-embed-link:hover svg, +.markdown-embed .markdown-embed-link:hover svg { + opacity: 1; } + +.markdown-embed-link:hover, .file-embed-link:hover { + color: var(--text-muted); } + +.markdown-preview-view .markdown-embed-content > .markdown-preview-view { + max-height: none !important; } + +.markdown-embed-content { + max-height: none !important; } + +.markdown-embed .markdown-preview-view { + padding: 0; } + +.internal-embed .markdown-embed { + border: 0; + border-left: 1px solid var(--quote-opening-modifier); + border-radius: 0; } + +/* Headings and fonts */ +h1, h2, h3, h4, h5, strong { + font-weight: var(--bold-weight); } + +h1, h2, h3, h4 { + letter-spacing: -0.02em; } + +body, input, button { + font-family: var(--font-interface); } + +.cm-s-obsidian span.cm-error { + color: var(--red); } + +.markdown-preview-view, +.popover, +.workspace-leaf-content[data-type=markdown] { + font-family: var(--font-text); } + +body, input, button, +.markdown-preview-view, +.markdown-source-view.mod-cm6.is-live-preview .cm-scroller, +.cm-s-obsidian, +.cm-s-obsidian .cm-formatting-hashtag { + font-size: var(--font-adaptive-normal); + font-weight: var(--normal-weight); + line-height: var(--line-height); + -webkit-font-smoothing: subpixel-antialiased; } + +.markdown-source-view.mod-cm6 .cm-scroller, +.markdown-source-view, +.cm-s-obsidian .cm-formatting-hashtag, +.cm-s-obsidian, +.cm-s-obsidian span.cm-formatting-task { + line-height: var(--line-height); + font-family: var(--font-editor); + -webkit-font-smoothing: subpixel-antialiased; } + +/* Use reading font in live preview */ +.lp-reading-font .markdown-source-view.mod-cm6.is-live-preview .cm-scroller { + font-family: var(--font-text); } + +.cm-s-obsidian span.cm-formatting-task { + font-family: var(--font-editor); + line-height: var(--line-height); } + +.cm-s-obsidian .cm-header, +.cm-s-obsidian .cm-strong { + font-weight: var(--bold-weight); } + +strong, +.cm-s-obsidian .cm-strong { + color: var(--text-bold); } + +em, +.cm-s-obsidian .cm-em { + color: var(--text-italic); } + +.cm-formatting-header, +.cm-s-obsidian .cm-formatting-header.cm-header-1, +.cm-s-obsidian .cm-formatting-header.cm-header-2, +.cm-s-obsidian .cm-formatting-header.cm-header-3, +.cm-s-obsidian .cm-formatting-header.cm-header-4, +.cm-s-obsidian .cm-formatting-header.cm-header-5, +.cm-s-obsidian .cm-formatting-header.cm-header-6 { + color: var(--text-faint); } + +.view-header-title, +.file-embed-title, +.markdown-embed-title { + letter-spacing: -0.02em; + text-align: left; + font-size: var(--title-size); + font-weight: var(--title-weight); } + +.view-header-title { + color: var(--title-color-inactive); } + +.file-embed-title, +.markdown-embed-title, +.workspace-leaf.mod-active .view-header-title { + color: var(--title-color); } + +.cm-s-obsidian .HyperMD-header { + line-height: 1.3; } + +.mod-cm6 .cm-editor .HyperMD-header-1, +.mod-cm6 .cm-editor .HyperMD-header-2, +.mod-cm6 .cm-editor .HyperMD-header-3, +.mod-cm6 .cm-editor .HyperMD-header-4, +.mod-cm6 .cm-editor .HyperMD-header-5, +.mod-cm6 .cm-editor .HyperMD-header-6 { + padding-top: 0.5em; } + +h1, +.empty-state-title, +.markdown-rendered h1, +.markdown-preview-view h1, +.cm-s-obsidian .cm-header-1 { + font-variant: var(--h1-variant); + letter-spacing: -0.02em; + line-height: 1.3; + font-family: var(--h1-font); + font-size: var(--h1); + color: var(--h1-color); + font-weight: var(--h1-weight); + font-style: var(--h1-style); } + h1 a, + .empty-state-title a, + .markdown-rendered h1 a, + .markdown-preview-view h1 a, + .cm-s-obsidian .cm-header-1 a { + font-weight: var(--h1-weight); } + +.markdown-rendered h2, +.markdown-preview-view h2, +.cm-s-obsidian .cm-header-2 { + font-variant: var(--h2-variant); + letter-spacing: -0.01em; + line-height: 1.3; + font-family: var(--h2-font); + font-size: var(--h2); + color: var(--h2-color); + font-weight: var(--h2-weight); + font-style: var(--h2-style); } + .markdown-rendered h2 a, + .markdown-preview-view h2 a, + .cm-s-obsidian .cm-header-2 a { + font-weight: var(--h2-weight); } + +.markdown-rendered h3, +.markdown-preview-view h3, +.cm-s-obsidian .cm-header-3 { + font-variant: var(--h3-variant); + letter-spacing: -0.01em; + line-height: 1.4; + font-family: var(--h3-font); + font-size: var(--h3); + color: var(--h3-color); + font-weight: var(--h3-weight); + font-style: var(--h3-style); } + .markdown-rendered h3 a, + .markdown-preview-view h3 a, + .cm-s-obsidian .cm-header-3 a { + font-weight: var(--h3-weight); } + +.markdown-rendered h4, +.markdown-preview-view h4, +.cm-s-obsidian .cm-header-4 { + font-variant: var(--h4-variant); + letter-spacing: 0.02em; + font-family: var(--h4-font); + font-size: var(--h4); + color: var(--h4-color); + font-weight: var(--h4-weight); + font-style: var(--h4-style); } + .markdown-rendered h4 a, + .markdown-preview-view h4 a, + .cm-s-obsidian .cm-header-4 a { + font-weight: var(--h4-weight); } + +.markdown-rendered h5, +.markdown-preview-view h5, +.cm-s-obsidian .cm-header-5 { + font-variant: var(--h5-variant); + letter-spacing: 0.02em; + font-family: var(--h5-font); + font-size: var(--h5); + color: var(--h5-color); + font-weight: var(--h5-weight); + font-style: var(--h5-style); } + .markdown-rendered h5 a, + .markdown-preview-view h5 a, + .cm-s-obsidian .cm-header-5 a { + font-weight: var(--h5-weight); } + +.markdown-rendered h6, +.markdown-preview-view h6, +.cm-s-obsidian .cm-header-6 { + font-variant: var(--h6-variant); + letter-spacing: 0.02em; + font-family: var(--h6-font); + font-size: var(--h6); + color: var(--h6-color); + font-weight: var(--h6-weight); + font-style: var(--h6-style); } + .markdown-rendered h6 a, + .markdown-preview-view h6 a, + .cm-s-obsidian .cm-header-6 a { + font-weight: var(--h6-weight); } + +/* Footnotes */ +/* Preview mode */ +.footnotes-list { + margin-block-start: -10px; + padding-inline-start: 20px; + font-size: var(--font-adaptive-small); } + +.footnotes-list p { + display: inline; + margin-block-end: 0; + margin-block-start: 0; } + +.footnote-ref a { + text-decoration: none; } + +.footnote-backref { + color: var(--text-faint); } + +.footnotes .is-flashing, +.minimal-folding .footnotes .is-flashing { + box-shadow: -1px 0px 0 3px var(--text-highlight-bg); } + +.cm-s-obsidian .HyperMD-footnote, +.footnotes { + font-size: calc(var(--font-adaptive-normal) - 2px); } + +.markdown-preview-view .footnotes hr { + margin: 0.5em 0 1em; + border-width: 1px 0 0 0; } + +/* YAML Frontmatter */ +.theme-dark pre.frontmatter[class*="language-yaml"], +.theme-light pre.frontmatter[class*="language-yaml"] { + padding: 0 0 0px 0; + background: transparent; + font-family: var(--font-text); + line-height: 1.2; + border-radius: 0; + border-bottom: 0px solid var(--background-modifier-border); } + +.markdown-preview-view .table-view-table > thead > tr > th { + border-color: var(--background-modifier-border); } + +.theme-dark .frontmatter .token, +.theme-light .frontmatter .token, +.markdown-preview-section .frontmatter code { + font-family: var(--font-text); + color: var(--text-faint) !important; } + +.markdown-source-view .cm-s-obsidian .cm-hmd-frontmatter { + font-family: var(--font-editor); + color: var(--text-muted); } + +.markdown-preview-section .frontmatter code { + color: var(--text-muted); + font-size: var(--font-adaptive-small); } + +.cm-s-obsidian .cm-hmd-frontmatter, +.cm-s-obsidian .cm-def.cm-hmd-frontmatter { + font-size: var(--font-adaptive-small); + color: var(--text-muted); } + +/* Preview mode */ +.frontmatter code.language-yaml { + padding: 0; } + +.frontmatter-collapse-indicator.collapse-indicator { + display: none; } + +.frontmatter-container .tag { + font-size: var(--font-adaptive-smaller); } + +.frontmatter-container .frontmatter-alias { + color: var(--text-muted); } + +.frontmatter-container { + font-size: var(--font-adaptive-small); + padding: 10px 0; + background: transparent; + border-radius: 0; + margin: 0; + border: 0; + border-bottom: 1px solid var(--background-modifier-border); } + +.frontmatter-container .frontmatter-container-header { + padding: 0; + font-weight: 500; + border-bottom: 0; + font-size: var(--font-adaptive-small); } + +/* File browser */ +.is-mobile .nav-folder.mod-root > .nav-folder-title .nav-folder-title-content { + display: none; } + +.nav-file-tag { + font-weight: 400; } + +.nav-header { + padding: 0; } + +.nav-buttons-container { + padding: 10px 5px 0px 8px; + margin-bottom: 0px !important; + justify-content: flex-start; + border: 0; } + +.nav-files-container { + overflow-x: hidden; + padding-bottom: 50px; } + +body:not(.is-mobile) .nav-folder.mod-root > .nav-folder-title .nav-folder-title-content { + font-weight: 500; + text-transform: uppercase; + letter-spacing: 0.05em; + color: var(--text-muted); + padding-bottom: 7px; + margin-left: -7px; + font-size: var(--font-adaptive-smaller); } + +.nav-folder-title { + margin: 0 0 0 8px; + min-width: auto; + width: calc(100% - 16px); + padding: 0 10px 0 16px; + line-height: 1.5; + cursor: var(--cursor); + border: none; } + +.nav-folder.mod-root > .nav-folder-title.is-being-dragged-over { + background-color: var(--text-selection); } + +.nav-folder-title.is-being-dragged-over { + background-color: var(--text-selection); + border-color: var(--text-selection); + border-radius: var(--radius-m); + border: 0px solid transparent; } + +.nav-folder-title-content { + padding: 1px 4px; } + +.nav-folder-collapse-indicator { + top: 1px; + margin-left: -10px; } + +/* Fix :active state when right-clicking in file explorer */ +.nav-file-title.is-being-dragged, +.nav-folder-title.is-being-dragged, +body:not(.is-grabbing) .nav-file-title.is-being-dragged:hover, +body:not(.is-grabbing) .nav-folder-title.is-being-dragged:hover { + background-color: var(--background-tertiary); + color: var(--text-normal); + box-shadow: 0 0 0 2px var(--background-modifier-border-focus); + z-index: 1; } + +.workspace-leaf.mod-active .nav-folder.has-focus > .nav-folder-title, +.workspace-leaf.mod-active .nav-file.has-focus { + border: none; + background-color: transparent; } + +.nav-file { + margin-left: 12px; + padding-right: 4px; + border: none; } + +.nav-file-title { + width: calc(100% - 30px); + margin: 0 8px 0 -4px; + padding: 0; + border-width: 0; + line-height: 1.6; + border-color: var(--background-secondary); + border-radius: var(--radius-m); + cursor: var(--cursor); } + +.nav-file-title.is-active, +.nav-folder-title.is-active, +.nav-file-title.is-being-dragged, +body:not(.is-grabbing) .nav-folder-title.is-active:hover, +body:not(.is-grabbing) .nav-folder-title:hover, +body:not(.is-grabbing) .nav-file-title.is-active:hover { + background-color: var(--background-tertiary); + color: var(--text-normal); } + +.nav-file-title-content { + width: 100%; + white-space: nowrap; + overflow: hidden; + text-overflow: ellipsis; + padding: 2px 7px; + border: 0; + vertical-align: middle; + cursor: var(--cursor); } + +.drop-indicator { + border-width: 1px; } + +.nav-file-icon { + margin: 1px 0 0 0; + vertical-align: bottom; + padding: 0 0 0 5px; } + +.workspace-leaf-content[data-type=starred] .nav-file-title-content { + width: calc(100% - 15px); } + +.workspace-leaf-content[data-type=starred] .nav-file-icon { + opacity: 0.5; } + +body:not(.is-grabbing) .nav-file-title:hover .nav-folder-collapse-indicator, +body:not(.is-grabbing) .nav-folder-title:hover .nav-folder-collapse-indicator, +body:not(.is-grabbing) .nav-file-title:hover, +body:not(.is-grabbing) .nav-folder-title:hover { + background: transparent; } + +.nav-file-title, +.tree-item-self, +.nav-folder-title, +.is-collapsed .search-result-file-title, +.tag-pane-tag { + font-size: var(--font-adaptive-small); + color: var(--text-muted); } + +.search-result-file-title { + font-size: var(--font-adaptive-small); + color: var(--text-normal); + font-weight: var(--normal-weight); } + +.side-dock-collapsible-section-header { + font-size: var(--font-adaptive-small); + color: var(--text-muted); + cursor: var(--cursor); + margin-right: 0; + margin-left: 0; } + +.side-dock-collapsible-section-header:hover, +.side-dock-collapsible-section-header:not(.is-collapsed) { + color: var(--text-muted); + background: transparent; } + +.tree-view-item-self:hover .tree-view-item-collapse, +.tree-item-self.is-clickable:hover { + color: var(--text-muted); + background: transparent; + cursor: var(--cursor); } + +.tree-item-self.is-clickable { + cursor: var(--cursor); } + +.search-result-collapse-indicator svg, +.search-result-file-title:hover .search-result-collapse-indicator svg, +.side-dock-collapsible-section-header-indicator:hover svg, +.side-dock-collapsible-section-header:hover .side-dock-collapsible-section-header-indicator svg, +.markdown-preview-view .collapse-indicator svg, +.tree-view-item-collapse svg, +.is-collapsed .search-result-collapse-indicator svg, +.nav-folder-collapse-indicator svg, +.side-dock-collapsible-section-header-indicator svg, +.is-collapsed .side-dock-collapsible-section-header-indicator svg { + color: var(--text-faint); + cursor: var(--cursor); } + +.search-result-collapse-indicator, +.search-result-file-title:hover .search-result-collapse-indicator, +.side-dock-collapsible-section-header-indicator:hover, +.side-dock-collapsible-section-header:hover .side-dock-collapsible-section-header-indicator, +.markdown-preview-view .collapse-indicator, +.tree-view-item-collapse, +.is-collapsed .search-result-collapse-indicator, +.nav-folder-collapse-indicator, +.side-dock-collapsible-section-header-indicator, +.is-collapsed .side-dock-collapsible-section-header-indicator { + color: var(--text-faint); + cursor: var(--cursor); } + +.is-collapsed .search-result-file-title:hover, +.search-result-file-title:hover, +.nav-folder-title.is-being-dragged-over .nav-folder-collapse-indicator svg { + color: var(--text-normal); } + +/* --------------- */ +/* Nested items */ +.nav-folder-collapse-indicator, +.tree-item-self .collapse-icon { + color: var(--background-modifier-border-hover); } + +.tree-item-self .collapse-icon { + padding-left: 0; + width: 18px; + margin-left: -18px; + justify-content: center; } + +.tree-item-self:hover .collapse-icon { + color: var(--text-normal); } + +.tree-item-self { + padding-left: 15px; } + +.tree-item { + padding-left: 5px; } + +.tree-item-flair { + font-size: var(--font-adaptive-smaller); + right: 0; + background: transparent; + color: var(--text-faint); } + +.tree-item-flair-outer:after { + content: ''; } + +.tree-item-self.is-clickable { + cursor: var(--cursor); } + +.tree-item-self.is-clickable:hover { + background: transparent; } + +.tree-item-self:hover .tree-item-flair { + background: transparent; + color: var(--text-muted); } + +.tree-item-children { + margin-left: 5px; } + +/* Folding icons in Preview */ +.collapse-indicator svg, +.markdown-preview-view .heading-collapse-indicator.collapse-indicator svg, +.markdown-preview-view ol > li .collapse-indicator svg, +.markdown-preview-view ul > li .collapse-indicator svg { + opacity: 0; } + +h1:hover .heading-collapse-indicator.collapse-indicator svg, +h2:hover .heading-collapse-indicator.collapse-indicator svg, +h3:hover .heading-collapse-indicator.collapse-indicator svg, +h4:hover .heading-collapse-indicator.collapse-indicator svg, +h5:hover .heading-collapse-indicator.collapse-indicator svg, +.HyperMD-header:hover .collapse-indicator svg, +.markdown-preview-view .is-collapsed .collapse-indicator svg, +.markdown-preview-view .collapse-indicator:hover svg, +.collapse-indicator:hover svg { + opacity: 1; } + +.markdown-preview-view div.is-collapsed h1::after, +.markdown-preview-view div.is-collapsed h2::after, +.markdown-preview-view div.is-collapsed h3::after, +.markdown-preview-view div.is-collapsed h4::after, +.markdown-preview-view div.is-collapsed h5::after, +.markdown-preview-view ol .is-collapsed::after, +.markdown-preview-view ul .is-collapsed::after { + content: "..."; + padding: 5px; + color: var(--text-faint); } + +.markdown-preview-view ol > li.task-list-item .collapse-indicator, +.markdown-preview-view ul > li.task-list-item .collapse-indicator { + margin-left: -48px; + position: absolute; } + +.markdown-preview-view ol > li .collapse-indicator { + padding-right: 20px; } + +.markdown-preview-view .heading-collapse-indicator.collapse-indicator { + margin-left: -28px; + padding-right: 7px 8px 7px 0; } + +.markdown-preview-view .collapse-indicator { + position: absolute; + margin-left: -44px; + padding-bottom: 10px; + padding-top: 0px; } + +.markdown-preview-view ul > li:not(.task-list-item) .collapse-indicator { + padding-right: 20px; } + +.list-collapse-indicator .collapse-indicator .collapse-icon { + opacity: 0; } + +.markdown-preview-view ul > li h1, +.markdown-preview-view ul > li h2, +.markdown-preview-view ul > li h3, +.markdown-preview-view ul > li h4 { + display: inline; } + +/* Folding icons in Edit mode */ +.markdown-source-view.mod-cm6.is-folding .cm-contentContainer { + padding-left: 0; } + +.CodeMirror-foldgutter-folded, +.CodeMirror-foldgutter-open { + cursor: var(--cursor); } + +body .frontmatter-collapse-indicator svg.right-triangle { + background-color: currentColor; + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body span[title="Fold line"], +body span[title="Unfold line"] { + position: relative; + font-size: 0; + color: transparent; + display: flex; + height: auto; + align-items: center; } + +body span[title="Fold line"]:hover, +body span[title="Unfold line"]:hover, +body .CodeMirror-foldgutter-open:hover, +body .CodeMirror-foldgutter-folded:hover { + color: var(--text-muted); } + +body span[title="Fold line"]:after, +body span[title="Unfold line"]:after, +body .CodeMirror-foldgutter-open:after, +body .CodeMirror-foldgutter-folded:after { + text-align: center; + color: var(--text-faint); + font-size: 1.25rem; + display: flex; + align-items: center; + justify-content: center; + margin-left: 0px; + width: 1rem; + height: 1rem; } + +body:not(.is-mobile) span[title="Fold line"]:after, +body:not(.is-mobile) span[title="Unfold line"]:after, +body:not(.is-mobile) .CodeMirror-foldgutter-open:after, +body:not(.is-mobile) .CodeMirror-foldgutter-folded:after { + margin-top: 0.35rem; + margin-left: 2px; } + +body .is-mobile .cm-editor .cm-lineNumbers .cm-gutterElement { + padding: 0 3px 0 0px; + min-width: 15px; + text-align: right; + white-space: nowrap; } + +body span[title="Fold line"]:after, +body span[title="Unfold line"]:after { + font-size: 1rem; + line-height: 1; } + +body span[title="Fold line"]:after, +body span[title="Unfold line"]:after { + font-size: 1rem; + line-height: 1; } + +body span[title="Unfold line"]:after, +body .CodeMirror-foldgutter-folded:after { + background-color: var(--text-faint); + height: 12px; + width: 12px; + -webkit-mask-image: url('data:image/svg+xml;utf8,'); + transform: translateY(-2px); + transform: rotate(-90deg); } + +body span[title="Fold line"]:after, +body .CodeMirror-foldgutter-open:after { + background-color: var(--text-faint); + height: 12px; + width: 12px; + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +.is-mobile span[title="Fold line"]:after, +.is-mobile .CodeMirror-foldgutter-open:after { + transform: translateX(-2px) !important; } + +span[title="Fold line"], +.CodeMirror-foldgutter-open:after { + opacity: 0; } + +span[title="Fold line"]:hover, +span[title="Unfold line"], +.CodeMirror-foldgutter-folded:after, +.CodeMirror-code > div:hover .CodeMirror-foldgutter-open:after { + opacity: 1; } + +span[title="Unfold line"]:hover, +.CodeMirror-code > div:hover .CodeMirror-foldgutter-open:hover:after, +.CodeMirror-code > div:hover .CodeMirror-foldgutter-folded:hover:after { + opacity: 1; } + +body.is-mobile span[title="Unfold line"]:after, +body.is-mobile .CodeMirror-foldgutter-folded:after { + content: "›"; + font-family: sans-serif; + transform: translateY(-2px); + transform: rotate(-90deg) translateY(2px) translateX(-0.45em); } + +body.is-mobile span[title="Fold line"]:after, +body.is-mobile .CodeMirror-foldgutter-open:after { + content: "›"; + font-family: sans-serif; + transform: rotate(360deg); } + +/* Icons and icon buttons */ +body svg.right-triangle { + color: var(--text-muted); + background-color: var(--text-muted); + height: 12px; + width: 12px; + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +.nav-action-button svg { + width: 15px; + height: 15px; } + +body .view-header-icon, +body .graph-controls-button, +body .clickable-icon, +body .menu-item-icon, +body .side-dock-ribbon-action, +body .nav-action-button, +body .view-action, +body .workspace-tab-header-inner-icon { + line-height: 0; } + +body .view-header-icon svg path, +body .graph-controls-button svg path, +body .clickable-icon svg path, +body .menu-item-icon svg path, +body .side-dock-ribbon-action svg path, +body .nav-action-button svg path, +body .view-action svg path, +body .workspace-tab-header-inner-icon svg path { + stroke-width: 2px; } + +body .view-action svg.cross path { + stroke-width: 2px; } + +.workspace-ribbon-collapse-btn svg path { + stroke-width: 4px; } + +.nav-action-button svg path { + stroke-width: 2px; } + +.clickable-icon { + cursor: var(--cursor); } + +.graph-controls-button, +.view-action, +.view-header-icon, +.nav-action-button, +.workspace-tab-header, +.side-dock-ribbon-tab, +.side-dock-ribbon-action, +.workspace-tab-header { + background: transparent; + color: var(--icon-color); + opacity: var(--icon-muted); + transition: opacity 0.1s ease-in-out; + cursor: var(--cursor); + line-height: 0; } + +.graph-controls-button, +.view-header-icon, +.workspace-tab-header-inner-icon, +.side-dock-ribbon-action, +.workspace-ribbon-collapse-btn { + margin: 0; + padding: 4px 4px; + height: 26px; + border-radius: var(--radius-m); } + +.view-header-icon { + display: flex; + align-items: center; } + +.workspace-ribbon-collapse-btn { + margin: 0; + padding: 2px 4px; } + +.side-dock-ribbon-action { + border-left: 0; + margin: 0 6px 6px; } + +.nav-action-button, +.workspace-leaf-content[data-type='search'] .nav-action-button, +.workspace-leaf-content[data-type='backlink'] .nav-action-button { + padding: 3px 5px 3px; + margin: 0 0 7px 0px; + height: 26px; + text-align: center; + border-radius: var(--radius-m); } + +.nav-action-button.is-active, +.workspace-leaf-content[data-type='dictionary-view'] .nav-action-button.is-active, +.workspace-leaf-content[data-type='search'] .nav-action-button.is-active, +.workspace-leaf-content[data-type='backlink'] .nav-action-button.is-active, +.workspace-leaf-content[data-type='tag'] .nav-action-button.is-active, +.workspace-tab-header.is-active, +.workspace-leaf-content[data-type='search'] .nav-action-button.is-active { + background: transparent; + color: var(--icon-color); + opacity: 1; + transition: opacity 0.1s ease-in-out; } + +.nav-action-button.is-active, +.workspace-tab-header.is-active:hover { + color: var(--icon-color); } + +.workspace-leaf-content[data-type='search'] .nav-action-button.is-active { + background: transparent; } + +.graph-controls-button:hover, +.view-action:hover, +.view-action.is-active:hover, +.view-header-icon:hover, +.nav-action-button:hover, +.nav-action-button.is-active:hover, +.workspace-tab-header:hover, +.side-dock-ribbon-tab:hover, +.side-dock-ribbon-action:hover { + color: var(--icon-color-hover); + opacity: 1; + transition: opacity 0.1s ease-in-out; } + +.graph-controls-button:hover, +.view-action:hover, +.nav-action-button:hover, +.workspace-leaf-content[data-type='search'] .nav-action-button.is-active:hover, +.workspace-leaf-content[data-type='backlink'] .nav-action-button.is-active:hover, +.workspace-drawer-tab-option-item:hover, +.workspace-drawer-header-icon:hover, +.workspace-tab-header-inner-icon:hover, +.side-dock-ribbon-action:hover { + background-color: var(--background-tertiary); + border-radius: var(--radius-m); } + +/* Search */ +.is-mobile .document-search-container .document-search { + position: relative; } + +.is-mobile .search-input-container:before, +.is-mobile .workspace-leaf-content[data-type='search'] .search-input-container:before, +.is-mobile .document-search-container .document-search:before { + content: " "; + position: absolute; + z-index: 9; + top: 50%; + transform: translateY(-50%); + left: 7px; + display: block; + width: 18px; + height: 18px; + background-color: var(--text-muted); + -webkit-mask-image: url('data:image/svg+xml,%3Csvg xmlns="http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg" width="18" height="18" preserveAspectRatio="xMidYMid meet" viewBox="0 0 24 24"%3E%3Cpath fill="currentColor" fill-rule="evenodd" d="m16.325 14.899l5.38 5.38a1.008 1.008 0 0 1-1.427 1.426l-5.38-5.38a8 8 0 1 1 1.426-1.426ZM10 16a6 6 0 1 0 0-12a6 6 0 0 0 0 12Z"%2F%3E%3C%2Fsvg%3E'); + background-position: 50% 50%; + background-repeat: no-repeat; } + +/* Indentation Guides (Obsidian 0.14.0+) */ +body { + --ig-adjust-reading:-0.65em; + --ig-adjust-edit:-1px; } + +.markdown-rendered.show-indentation-guide li.task-list-item > ul::before, +.markdown-rendered.show-indentation-guide li.task-list-item > ol::before, +.markdown-rendered.show-indentation-guide li > ul::before, +.markdown-rendered.show-indentation-guide li > ol::before { + left: var(--ig-adjust-reading); } + +/* Live Preview */ +.markdown-source-view.mod-cm6 .cm-indent::before { + transform: translateX(var(--ig-adjust-edit)); } + +.is-mobile .markdown-rendered.show-indentation-guide li > ul::before, +.is-mobile .markdown-rendered.show-indentation-guide li > ol::before { + left: calc(0em + var(--ig-adjust-reading)); } +.is-mobile .markdown-source-view.mod-cm6 .cm-indent::before { + transform: translateX(calc(2px + var(--ig-adjust-edit))); } + +/* Links */ +a { + color: var(--text-accent); + font-weight: var(--link-weight); } + +strong a { + color: var(--text-accent); + font-weight: var(--bold-weight); } + +a[href*="obsidian://search"] { + background-image: url("data:image/svg+xml,"); } + +.theme-dark a[href*="obsidian://search"] { + background-image: url("data:image/svg+xml,"); } + +.cm-s-obsidian span.cm-url:hover, +.is-live-preview.cm-s-obsidian span.cm-hmd-internal-link:hover, +.is-live-preview.cm-s-obsidian span.cm-url:hover, +.is-live-preview.cm-s-obsidian span.cm-link:hover { + color: var(--text-accent-hover); } + +a em, +.cm-s-obsidian span.cm-url, +.cm-s-obsidian .cm-url, +.cm-s-obsidian .cm-active .cm-url, +.is-live-preview.cm-s-obsidian .cm-link, +.cm-s-obsidian.mod-cm6 .cm-hmd-internal-link { + color: var(--text-accent); } + +.cm-url, +.cm-link, +.cm-hmd-internal-link { + font-weight: var(--link-weight); } + +.cm-s-obsidian .cm-active span.cm-link.cm-hmd-barelink, +.cm-s-obsidian span.cm-link.cm-hmd-barelink, +.cm-s-obsidian span.cm-link.cm-hmd-barelink:hover { + color: var(--text-normal); } + +.cm-s-obsidian .cm-active .cm-formatting.cm-formatting-link, +.cm-s-obsidian span.cm-image-alt-text.cm-link, +.cm-s-obsidian:not(.is-live-preview) .cm-formatting-link + span.cm-link { + color: var(--text-muted); } + +/* Reader Mode Lists */ +div > ol, +div > ul { + padding-inline-start: 1.4em; } + +ul > li { + min-height: 1.4em; } + +ol > li { + margin-left: 0em; } + +ul { + padding-inline-start: var(--list-indent); } + +ol { + padding-inline-start: var(--list-indent); + margin-left: 0; + list-style: default; } + +.is-mobile { + /* first level */ } + .is-mobile ul > li:not(.task-list-item)::marker { + font-size: 0.8em; } + .is-mobile .markdown-rendered ul, + .is-mobile .markdown-rendered ol { + padding-inline-start: var(--list-indent); } + .is-mobile .markdown-rendered div > ol, + .is-mobile .markdown-rendered div > ul { + padding-inline-start: 2em; } + .is-mobile .el-ol > ol, + .is-mobile .el-ul > ul { + margin-left: 0; } + +/* Live Preview */ +.cm-line:not(.HyperMD-codeblock) { + tab-size: var(--list-indent); } + +.markdown-source-view.mod-cm6 .cm-content .HyperMD-list-line { + margin-left: var(--list-edit-offset) !important; } + +/* Space between list items */ +.markdown-source-view ol > li, +.markdown-source-view ul > li, +.markdown-preview-view ol > li, +.markdown-preview-view ul > li, +.mod-cm6 .HyperMD-list-line.cm-line { + padding-top: var(--list-spacing); + padding-bottom: var(--list-spacing); } + +/* Legacy Editor Mode Lists */ +.cm-formatting-list { + color: var(--text-faint) !important; } + +/* Bullets */ +ul > li::marker, +ol > li::marker { + color: var(--text-faint); } + +ul > li:not(.task-list-item)::marker { + font-family: -apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,sans-serif; + font-size: 0.9em; } + +.mod-cm6 .HyperMD-list-line .list-bullet::after, +.mod-cm6 span.list-bullet::after { + line-height: 0.95em; + font-size: 1.4em; + font-family: -apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,sans-serif; + vertical-align: middle; + color: var(--text-faint); } + +body:not(.is-mobile) .markdown-source-view.mod-cm6 .list-bullet:after { + left: -5px; } + +body:not(.is-mobile) .markdown-source-view.mod-cm6 span.cm-formatting.cm-formatting-list.cm-formatting-list-ol { + margin-left: -5px; } + +/* Modals */ +.progress-bar-message { + color: var(--text-faint); } + +.modal { + box-shadow: var(--shadow-l); + border: none; + background: var(--background-primary); + border-radius: var(--radius-l); + overflow: hidden; + padding: 20px; } + +body:not(.is-mobile) .modal { + border: 1px solid var(--modal-border); } + +.modal.mod-settings .vertical-tab-content-container { + border-left: 1px solid var(--background-divider); + padding-bottom: 0; + padding-right: 0; } + +.modal-title { + text-align: left; + font-size: var(--h2); + line-height: 1.4; } + +.modal-content { + margin-top: 0px; + padding: 2px; + font-size: var(--font-adaptive-small); } + +.modal-content .u-center-text { + text-align: left; + font-size: var(--font-adaptive-small); } + +.modal-button-container { + margin-top: 10px; + gap: 8px; + display: flex; } + .modal-button-container button { + margin-top: 10px; } + +/* Confirm delete */ +.modal-container.mod-confirmation .modal { + width: 480px; + min-width: 0; } +.modal-container.mod-confirmation .modal-content { + margin-top: 10px; } + .modal-container.mod-confirmation .modal-content .setting-item { + margin-top: 10px; } +.modal-container.mod-confirmation .modal-button-container { + display: flex; } + .modal-container.mod-confirmation .modal-button-container > .mod-warning:nth-last-child(3) { + background: transparent; + border: none; + font-weight: 500; + color: var(--text-error); + cursor: pointer; + margin-right: auto; + box-shadow: none; + padding-left: 0; + padding-right: 0; } + .modal-container.mod-confirmation .modal-button-container > .mod-warning:nth-last-child(3):hover { + text-decoration: underline; } + .modal-container.mod-confirmation .modal-button-container > .mod-warning:nth-last-child(2) { + margin-left: auto; } + +/* Close buttons */ +.document-search-close-button, +.modal-close-button { + cursor: var(--cursor); + line-height: 20px; + text-align: center; + height: 24px; + width: 24px; + font-size: 24px; + color: var(--text-faint); + border-radius: var(--radius-m); } + +.modal-close-button { + top: 7px; + right: 7px; + padding: 0; } + +body:not(.is-mobile) .document-search-close-button:hover, +.modal-close-button:hover { + color: var(--text-normal); + background: var(--background-tertiary); } + +.document-search-close-button:before, +.modal-close-button:before { + font-family: Inter,sans-serif; + font-weight: 200; } + +/* Mobile modals */ +.is-mobile { + /* Mobile community themes */ + /* Mobile Community plugins */ + /* Tablet */ + /* Phone */ } + .is-mobile .modal { + width: 100%; + max-width: 100%; + border: none; + padding: 10px; + -webkit-touch-callout: none; + -webkit-user-select: none; + user-select: none; } + .is-mobile .modal, + .is-mobile .modal-bg { + transition: none !important; + transform: none !important; } + .is-mobile .modal.mod-publish, + .is-mobile .modal.mod-community-plugin, + .is-mobile .modal.mod-settings { + width: 100vw; + max-height: 90vh; + padding: 0; } + .is-mobile .mod-confirmation .modal { + border-radius: 15px; } + .is-mobile .mod-confirmation .modal .modal-close-button { + display: none; } + .is-mobile .modal-content { + padding: 0; + border-radius: 15px; } + .is-mobile .modal-button-container { + padding: 0; } + .is-mobile .setting-item:not(.mod-toggle):not(.setting-item-heading) { + flex-grow: 0; } + .is-mobile .vertical-tab-header-group:last-child, + .is-mobile .vertical-tab-content, + .is-mobile .minimal-donation { + padding-bottom: 70px !important; } + .is-mobile .modal.mod-settings .vertical-tab-header:before { + content: "Settings"; + font-weight: 600; + font-size: var(--font-settings); + position: sticky; + display: flex; + height: 54px; + margin-top: 8px; + align-items: center; + justify-content: center; + text-align: center; + border-bottom: 1px solid var(--background-modifier-border); + background: var(--background-primary); + left: 0; + top: 0; + right: 0; + z-index: 1; } + .is-mobile .modal .vertical-tab-header-group-title { + padding: 15px 20px 10px 20px; + text-transform: uppercase; + letter-spacing: 0.05em; } + .is-mobile .modal .vertical-tab-nav-item { + padding: 12px 0px; + margin: 0; + border-radius: 0; + color: var(--text-primary); + border-bottom: 1px solid var(--background-modifier-border); } + .is-mobile .modal .vertical-tab-nav-item:after { + content: " "; + float: right; + width: 20px; + height: 20px; + display: block; + opacity: 0.2; + background: center right no-repeat url("data:image/svg+xml,"); } + .is-mobile.theme-dark .modal .vertical-tab-nav-item:after { + background: center right no-repeat url("data:image/svg+xml,"); } + .is-mobile .vertical-tab-header-group-items { + width: calc(100% - 40px); + margin: 0 auto; } + .is-mobile .modal .vertical-tab-nav-item:first-child { + border-top: 1px solid var(--background-modifier-border); } + .is-mobile .modal.mod-settings .vertical-tab-nav-item { + font-size: var(--font-settings); } + .is-mobile .modal svg.left-arrow-with-tail { + -webkit-mask-image: url("data:image/svg+xml,"); + height: 26px; + width: 26px; } + .is-mobile .modal-close-button { + display: block; + z-index: 2; + top: 10px; + right: 12px; + padding: 4px; + font-size: 34px; + width: 34px; + height: 34px; + background-color: var(--background-primary); } + .is-mobile .modal-close-button:before { + font-weight: 300; + color: var(--text-muted); } + .is-mobile .modal-close-button:hover { + background-color: var(--background-tertiary); } + .is-mobile .mod-community-theme .modal-title { + padding: 10px 20px; } + .is-mobile .modal.mod-community-theme, + .is-mobile .modal.mod-community-theme .modal-content { + height: unset; } + .is-mobile .community-plugin-search { + border: none; } + .is-mobile .community-plugin-item:hover { + background-color: transparent; } + .is-mobile .community-plugin-item { + margin: 0; } + .is-mobile .community-plugin-search .setting-item { + margin-right: 42px; } + .is-mobile .community-plugin-search .setting-item-control { + display: flex; + flex-direction: row; } + .is-mobile .community-plugin-search .setting-item-control button { + width: 40px; + font-size: 0; + margin-left: 10px; + justify-content: center; + color: var(--text-muted); + border: none; + box-shadow: none; + background-color: currentColor; + -webkit-mask: no-repeat center center url('data:image/svg+xml;utf8,'); + -webkit-mask-size: 22px; } + .is-mobile .community-plugin-search .setting-item-control button:hover { + background-color: var(--text-normal); } + .is-mobile .community-plugin-search .search-input-container { + margin: 0; } + .is-mobile .modal.mod-settings .vertical-tabs-container { + display: flex; + overflow: hidden; + border-top-left-radius: 15px; + border-top-right-radius: 15px; } + .is-mobile .community-plugin-details .modal-setting-back-button { + padding: 12px 20px; } + .is-mobile .modal-setting-back-button { + border-bottom: 1px solid var(--background-modifier-border); + display: flex; + margin-top: 8px; + height: 54px; + justify-content: center; + align-items: center; + background-color: var(--color-background); + box-shadow: none; } + .is-mobile .modal-setting-back-button-icon { + position: absolute; + left: 10px; } + .is-mobile .modal-setting-back-button span:nth-child(2) { + flex-grow: 1; + text-align: center; + font-weight: 600; + height: 54px; + display: flex; + align-items: center; + justify-content: center; + color: var(--text-normal); } + .is-mobile .hotkey-list-container .setting-command-hotkeys { + flex: unset; } + .is-mobile .modal.mod-settings .vertical-tab-content-container { + border: 0; } + @media (min-width: 400pt) { + .is-mobile .modal .vertical-tab-header, + .is-mobile .modal .vertical-tabs-container, + .is-mobile .modal .vertical-tab-content-container { + border-radius: 15px !important; } + .is-mobile .modal, + .is-mobile .modal-container .modal.mod-settings { + max-width: 800px; + transform: translateZ(0); + border-radius: 15px; + margin-bottom: 0; + bottom: auto; + overflow: hidden; } + .is-mobile .modal-container .modal.mod-settings .vertical-tabs-container { + transform: translateZ(0); } + .is-mobile .modal-container .modal-bg { + opacity: 0.8 !important; } + .is-mobile .search-input-container input { + width: 100%; } + .is-mobile .modal-setting-back-button, + .is-mobile .modal.mod-settings .vertical-tab-header:before { + margin-top: 0; } } + @media (max-width: 400pt) { + .is-mobile .modal { + border-radius: 0; + border: none; } + .is-mobile .modal.mod-publish, + .is-mobile .modal.mod-community-plugin, + .is-mobile .modal.mod-settings { + max-height: calc(100vh - 32px); + box-shadow: 0 -32px 0 0 var(--background-primary); } + .is-mobile .mod-confirmation .modal { + bottom: 4.5vh; } + .is-mobile .modal .search-input-container { + width: 100%; + margin: 0; } + .is-mobile .modal-close-button { + top: 18px; + right: 0px; + padding: 4px 16px 2px 4px; + width: 46px; } + .is-mobile .modal-close-button:hover { + background: var(--background-primary); } } + +/* Menus */ +.menu { + padding: 7px 5px; + background-color: var(--background-secondary); } + +.menu-item { + font-size: var(--font-adaptive-small); + border-radius: var(--radius-m); + padding: 3px 6px 3px 6px; + margin: 0 2px; + cursor: var(--cursor); + height: auto; + line-height: 20px; + display: flex; + align-items: center; + overflow: hidden; } + .menu-item:hover, .menu-item:hover:not(.is-disabled):not(.is-label), .menu-item.selected:not(.is-disabled):not(.is-label) { + background-color: var(--background-tertiary); } + +.menu-separator { + margin: 8px -5px; } + +.menu-item-icon { + width: 20px; + opacity: 0.6; + line-height: 10px; + position: static; + margin-right: 2px; } + .menu-item-icon svg { + width: 12px; + height: 12px; } + +.menu-item-icon +div.menu-item:hover .menu-item-icon svg, +div.menu-item:hover .menu-item-icon svg path { + color: var(--text-normal); } + +/* Mobile */ +.is-mobile { + /* Tablet */ + /* Phone */ } + .is-mobile:not(.minimal-icons-off) .menu-item-icon svg { + width: 18px; + height: 18px; } + .is-mobile .menu { + border: none; + width: 100%; + max-width: 100%; + left: 0 !important; + -webkit-touch-callout: none; + -webkit-user-select: none; + user-select: none; } + .is-mobile .menu-item { + padding: 5px 10px; + margin: 0; } + .is-mobile .menu-item-icon { + margin-right: 10px; } + .is-mobile .menu-item.is-label { + color: var(--text-normal); + font-weight: var(--bold-weight); } + .is-mobile .menu-item.is-label .menu-item-icon { + display: none; } + @media (min-width: 400pt) { + .is-mobile .menu { + top: 60px !important; + right: 0 !important; + bottom: auto; + left: auto; + margin: 0 auto; + width: 360px; + padding: 10px 10px 10px; + border-radius: 15px; + box-shadow: 0 0 100vh 100vh rgba(0, 0, 0, 0.5); } + .is-mobile .menu .menu-item:hover { + background-color: var(--background-tertiary); } } + @media (max-width: 400pt) { + .is-mobile .menu { + padding-bottom: 30px; } + .is-mobile .menu-item.is-label { + font-size: var(--font-settings-title); } } + +/* Preview mode */ +.markdown-preview-view blockquote, +.markdown-preview-view p, +.markdown-preview-view ol, +.markdown-preview-view ul { + margin-block-start: var(--spacing-p); + margin-block-end: var(--spacing-p); } +.markdown-preview-view ul ol, +.markdown-preview-view ol ol, +.markdown-preview-view ol ul, +.markdown-preview-view ul ul { + margin-block-start: 0em; + margin-block-end: 0em; } +.markdown-preview-view h1, +.markdown-preview-view h2, +.markdown-preview-view h3, +.markdown-preview-view h4, +.markdown-preview-view h5, +.markdown-preview-view h6 { + margin-block-start: 1em; + margin-block-end: var(--spacing-p); } + +.markdown-preview-view hr { + height: 1px; + border-width: 2px 0 0 0; } + +iframe { + border: 0; } + +.markdown-preview-view .mod-highlighted { + transition: background-color 0.3s ease; + background-color: var(--text-selection); + color: inherit; } + +/* Backlinks in Preview */ +.mod-root .workspace-leaf-content[data-type='markdown'] .nav-header { + border-top: 1px solid var(--background-modifier-border); + margin-top: 3em; + position: relative; } + +.mod-root .workspace-leaf-content[data-type='markdown'] .nav-buttons-container, +.mod-root .workspace-leaf-content[data-type='markdown'].backlink-pane, +.mod-root .workspace-leaf-content[data-type='markdown'] .backlink-pane .search-result-container, +.mod-root .workspace-leaf-content[data-type='markdown'] .search-input-container, +.mod-root .workspace-leaf-content[data-type='markdown'] .tree-item, +.mod-root .workspace-leaf-content[data-type='markdown'] .search-empty-state { + padding-left: 0; + margin-left: 0; } + +.is-mobile .workspace-leaf-content:not([data-type='search']) .workspace-leaf-content[data-type='markdown'] .nav-buttons-container { + border-bottom: none; + padding-top: 5px; } + +.mod-root .workspace-leaf-content[data-type='markdown'] .search-input-container { + margin-bottom: 0px; + width: calc(100% - 130px); + margin-top: 10px; } + +.is-mobile .mod-root .workspace-leaf-content[data-type='markdown'] .search-input-container { + width: calc(100% - 160px); } + +.mod-root .workspace-leaf-content[data-type='markdown'] .backlink-pane { + padding-top: 10px; } + +.mod-root .workspace-leaf-content[data-type='markdown'] .nav-buttons-container { + position: absolute; + right: 0; + top: 3px; } + +.mod-root .workspace-leaf-content[data-type='markdown'] .backlink-pane > .tree-item-self:hover, +.mod-root .workspace-leaf-content[data-type='markdown'] .backlink-pane > .tree-item-self { + padding-left: 0px; + text-transform: none; + color: var(--text-normal); + font-size: var(--font-adaptive-normal); + font-weight: 500; + letter-spacing: unset; } + +.mod-root .workspace-leaf-content[data-type='markdown'] .backlink-pane > .tree-item-self.is-collapsed { + color: var(--text-faint); } + +.mod-root .workspace-leaf-content[data-type='markdown'] .backlink-pane > .tree-item-self.is-collapsed:hover { + color: var(--text-muted); } + +.mod-root .workspace-leaf-content[data-type='markdown'] .backlink-pane .search-result-file-title { + font-size: calc(var(--font-adaptive-normal) - 2px); } + +.mod-root .workspace-leaf-content[data-type=markdown] .markdown-source-view .embedded-backlinks .nav-header { + margin-top: 0; } + +/* Embedded searches */ +.internal-query { + border-top: none; + border-bottom: none; } + +.internal-query .internal-query-header { + padding-top: 10px; + justify-content: left; + border-top: 1px solid var(--ui1); } + +.internal-query .internal-query-header-title { + font-weight: 500; + color: var(--text-normal); + font-size: var(--text-adaptive-normal); } + +.internal-query .search-result-container { + border-bottom: 1px solid var(--ui1); } + +/* Default ribbon sidedock icons */ +.workspace-ribbon.mod-left .workspace-ribbon-collapse-btn, +.workspace-ribbon.mod-right .workspace-ribbon-collapse-btn { + opacity: 1; + position: fixed; + width: 26px; + display: flex; + align-items: center; + top: auto; + text-align: center; + bottom: 32px; + z-index: 9; } + +.workspace-ribbon.mod-left .workspace-ribbon-collapse-btn { + left: 8px; } + +.workspace-ribbon.mod-right { + right: 4px; + bottom: 0; + height: 32px; + padding-top: 6px; + position: absolute; + background: transparent; + border: 0; } + +.mod-right .workspace-ribbon-collapse-btn { + background-color: var(--background-primary); } + +.workspace-ribbon-collapse-btn, +.view-action, +.side-dock-ribbon-tab, +.side-dock-ribbon-action { + cursor: var(--cursor); } + +.workspace-ribbon-collapse-btn:hover { + background-color: var(--background-tertiary); } + +.workspace-ribbon { + border-width: var(--border-width-alt); + border-color: var(--background-divider); + background: var(--background-secondary); + flex: 0 0 42px; + padding-top: 7px; } + +.mod-right:not(.is-collapsed) ~ .workspace-split.mod-right-split { + margin-right: 0; } + +.side-dock-settings { + padding-bottom: 20px; } + +body.hider-frameless:not(.hider-ribbon):not(.is-fullscreen) .side-dock-actions { + padding-top: var(--top-left-padding-y); } + +/* Scroll bars */ +body:not(.hider-scrollbars).styled-scrollbars ::-webkit-scrollbar, +body:not(.native-scrollbars) ::-webkit-scrollbar { + width: 11px; + background-color: transparent; } +body:not(.hider-scrollbars).styled-scrollbars ::-webkit-scrollbar:horizontal, +body:not(.native-scrollbars) ::-webkit-scrollbar:horizontal { + height: 11px; } +body:not(.hider-scrollbars).styled-scrollbars ::-webkit-scrollbar-corner, +body:not(.native-scrollbars) ::-webkit-scrollbar-corner { + background-color: transparent; } +body:not(.hider-scrollbars).styled-scrollbars ::-webkit-scrollbar-track, +body:not(.native-scrollbars) ::-webkit-scrollbar-track { + background-color: transparent; } +body:not(.hider-scrollbars).styled-scrollbars ::-webkit-scrollbar-thumb, +body:not(.native-scrollbars) ::-webkit-scrollbar-thumb { + background-clip: padding-box; + border-radius: 20px; + border: 3px solid transparent; + background-color: var(--background-modifier-border); + border-width: 3px 3px 3px 3px; + min-height: 45px; } +body:not(.hider-scrollbars).styled-scrollbars .modal .vertical-tab-header::-webkit-scrollbar-thumb:hover, +body:not(.hider-scrollbars).styled-scrollbars .mod-left-split .workspace-tabs ::-webkit-scrollbar-thumb:hover, +body:not(.hider-scrollbars).styled-scrollbars ::-webkit-scrollbar-thumb:hover, +body:not(.native-scrollbars) .modal .vertical-tab-header::-webkit-scrollbar-thumb:hover, +body:not(.native-scrollbars) .mod-left-split .workspace-tabs ::-webkit-scrollbar-thumb:hover, +body:not(.native-scrollbars) ::-webkit-scrollbar-thumb:hover { + background-color: var(--background-modifier-border-hover); } +body:not(.hider-scrollbars).styled-scrollbars .modal .vertical-tab-header::-webkit-scrollbar-thumb:active, +body:not(.hider-scrollbars).styled-scrollbars .mod-left-split .workspace-tabs ::-webkit-scrollbar-thumb:active, +body:not(.hider-scrollbars).styled-scrollbars ::-webkit-scrollbar-thumb:active, +body:not(.native-scrollbars) .modal .vertical-tab-header::-webkit-scrollbar-thumb:active, +body:not(.native-scrollbars) .mod-left-split .workspace-tabs ::-webkit-scrollbar-thumb:active, +body:not(.native-scrollbars) ::-webkit-scrollbar-thumb:active { + background-color: var(--background-modifier-border-focus); } + +/* Search and replace (in file) */ +.is-flashing { + border-radius: 2px; + box-shadow: 2px 1px 0 4px var(--text-highlight-bg); + transition: all 0s ease-in-out; } + +.minimal-folding .is-flashing { + box-shadow: 5px 1px 0 6px var(--text-highlight-bg); } + +.is-flashing .tag { + border-color: var(--text-highlight-bg-active); } + +.suggestion-container.mod-search-suggestion { + max-width: 240px; } + +.mod-search-suggestion .suggestion-item { + font-size: var(--font-adaptive-small); } + +.mod-search-suggestion .clickable-icon { + margin: 0; } + +.search-suggest-item.mod-group { + font-size: var(--font-adaptive-smaller); } + +.cm-s-obsidian span.obsidian-search-match-highlight { + background: inherit; + background: var(--text-highlight-bg); + padding-left: 0; + padding-right: 0; } + +.markdown-preview-view .search-highlight > div { + box-shadow: 0 0 0px 2px var(--text-normal); + border-radius: 2px; + background: transparent; } + +.markdown-preview-view .search-highlight > div { + opacity: 0.4; } + +.markdown-preview-view .search-highlight > div.is-active { + background: transparent; + border-radius: 2px; + opacity: 1; + mix-blend-mode: normal; + box-shadow: 0 0 0px 3px var(--text-accent); } + +/* Live Preview */ +.cm-s-obsidian span.obsidian-search-match-highlight { + background-color: transparent; + box-shadow: 0 0 0px 3px var(--text-accent); + mix-blend-mode: multiply; + border-radius: 2px; } + +body:not(.is-mobile).borders-title .document-search-container { + padding-top: 0; } + +body input.document-search-input.mod-no-match:hover, +body input.document-replace-input.mod-no-match:hover, +body input.document-search-input.mod-no-match, +body input.document-replace-input.mod-no-match { + background-color: var(--background-primary); } + +body:not(.is-mobile) .document-search-container.mod-replace-mode { + height: 72px; } + +body:not(.is-mobile) .document-replace-buttons, +body:not(.is-mobile) .document-search-buttons { + padding-top: 3px; } + +.document-replace-buttons, +.document-search-buttons { + height: 30px; + padding-top: 0; + gap: 5px; + display: flex; } + +.document-search-button, +.document-search-close-button { + cursor: var(--cursor); + color: var(--text-muted); + font-weight: 500; } + +body:not(.is-mobile) .document-search-button, +body:not(.is-mobile) .document-search-close-button { + background: var(--background-tertiary); + height: 26px; } + +.document-search-button:hover { + box-shadow: none; + background: var(--background-tertiary); } + +body .document-search-close-button { + bottom: 0; + top: 0; + display: inline-flex; + height: 26px; + width: 26px; + line-height: 24px; } + +.document-search-button { + margin: 0; + padding-left: 0.75em; + padding-right: 0.75em; } + +body .document-search-container { + margin-top: 12px; + padding: 0; + height: 38px; + background-color: var(--background-primary); + border-top: none; + width: 100%; } + +.document-search, +.document-replace { + max-width: var(--max-width); + width: var(--line-width); + margin: 0 auto; + padding: 0 5px; } + +.minimal-readable-off .document-search, +.minimal-readable-off .document-replace { + width: 100%; } + +.markdown-source-view.is-searching, +.markdown-source-view.is-replacing, +.markdown-reading-view.is-searching { + flex-direction: column-reverse; } + +body input.document-search-input, +body input.document-replace-input { + margin-top: 2px; + font-size: var(--font-adaptive-small); + border: 1px solid var(--background-modifier-border); + border-radius: var(--radius-m); + height: 28px; + background: var(--background-primary); + transition: border-color 0.1s ease-in-out; } + +input.document-search-input:hover, +input.document-replace-input:hover { + border: 1px solid var(--background-modifier-border-hover); + background: var(--background-primary); + transition: border-color 0.1s ease-in-out; } + +input.document-search-input:focus, +input.document-replace-input:focus { + border: 1px solid var(--background-modifier-border-hover); + background: var(--background-primary); + transition: all 0.1s ease-in-out; } + +.document-search-button { + font-size: var(--font-adaptive-small); } + +/* Mobile */ +.is-mobile .document-search, +.is-mobile .document-replace { + flex-direction: row; } +.is-mobile .document-replace { + padding-top: 6px; } + .is-mobile .document-replace .document-replace-buttons { + flex-shrink: 1; + flex-grow: 0; } +.is-mobile .document-search-container { + padding: 8px 0 8px 0; + background-color: var(--background-primary); + margin: 0 auto 0 auto; + height: auto; + width: 100%; + border-bottom: 1px solid var(--background-modifier-border); + padding-left: var(--folding-offset); } +.is-mobile .document-search, +.is-mobile .document-replace { + margin: 0 auto; + padding-left: 0; + padding-right: 0; + max-width: calc(var(--max-width) + 2%); + width: var(--line-width-adaptive); } +.is-mobile.minimal-readable-off .document-search, +.is-mobile.minimal-readable-off .document-replace { + width: 100%; } +.is-mobile .document-search-container input[type='text'] { + width: auto; + margin: 0 8px 0 0; + height: 36px; + padding: 5px 10px 5px 10px; + border-radius: 6px; + min-width: 90px; + border: 1px solid var(--background-modifier-border); + background-color: var(--background-primary); } +.is-mobile .document-search-container .document-search-input[type='text'] { + padding-left: 30px; } +.is-mobile .document-search .document-search-buttons, +.is-mobile .document-replace button { + flex-grow: 0; } +.is-mobile .document-search-container button.document-search-button { + width: auto; + margin: 0px; + background: transparent; + font-size: 14px; + height: 36px; + padding: 0 2px; + white-space: nowrap; } +.is-mobile .document-search .document-search-close-button, +.is-mobile .document-replace .document-search-close-button { + height: 30px; + line-height: 30px; } + +/* Settings */ +.modal.mod-sync-history, +.modal.mod-sync-log, +.modal.mod-publish, +.modal.mod-community-plugin, +.modal.mod-settings { + width: 90vw; + height: 100vh; + max-height: 90vh; + max-width: 1000px; } + +.modal.mod-settings .vertical-tab-header, +.modal.mod-settings .vertical-tab-content-container { + height: 90vh; } + +.setting-item-name, +.community-plugin-name, +.modal.mod-settings .vertical-tab-content-container { + font-size: var(--font-settings); + line-height: 1.3; } + +.modal .modal-content > h2 { + text-align: left; + font-size: var(--h1); + font-weight: 600; } + +.modal.mod-settings .vertical-tab-content h1, +.modal.mod-settings .vertical-tab-content h2, +.modal.mod-settings .vertical-tab-content h3 { + text-align: left; + font-size: var(--h1); + font-weight: 600; } + +.modal .modal-content > h2:first-child, +.modal.mod-settings .vertical-tab-content > h2:first-child, +.modal.mod-settings .vertical-tab-content > h3:first-child { + margin-top: 0; } + +.community-plugin-search-summary, +.setting-item-description, +.community-plugin-item .community-plugin-author, +.community-plugin-downloads, +.community-plugin-item .community-plugin-desc { + font-size: var(--font-settings-small); + line-height: 1.3; + font-weight: 400; } + +.style-settings-collapse-indicator { + margin-right: 6px; } + +.modal .vertical-tab-nav-item { + font-size: var(--font-small); + line-height: 1.3; } + +.community-plugin-search .setting-item { + margin-right: 10px; } + +.flair.mod-pop { + letter-spacing: 0; + text-transform: none; + vertical-align: unset; + top: -1px; } + +.community-plugin-search { + padding: 20px 0 0 0; + background-color: var(--background-secondary); + border-right: 1px solid var(--background-divider); + flex: 0 0 270px; } + +.community-plugin-search-summary { + border-bottom: 1px solid var(--background-divider); + padding-bottom: 10px; } + +.community-plugin-info p button { + margin-right: 8px; } + +.community-plugin-item { + margin: 0; + cursor: var(--cursor); + padding-top: 15px; + border-bottom: 1px solid var(--background-divider); } + +.community-plugin-item:hover { + background-color: var(--background-tertiary); } + +.community-plugin-item .community-plugin-name { + font-weight: 500; } + +.community-plugin-item .community-plugin-author { + color: var(--text-muted); + padding-bottom: 10px; } + +.community-plugin-item .community-plugin-desc { + color: var(--text-normal); + font-size: var(--font-small); } + +.community-plugin-search .setting-item-info { + flex-grow: 0; } + +.community-plugin-search .search-input-container { + margin-left: -5px; + margin-right: 5px; } + +.modal .community-plugin-search .setting-item-control button { + display: flex; + align-items: center; } + +.setting-item-control button { + padding: 0.5em 0.75em; } + +button.mod-cta, +.modal button, +.modal button.mod-cta a { + font-size: var(--font-settings-small); + height: var(--input-height); + cursor: var(--cursor); + margin-right: 0px; + margin-left: 0px; } + +/* Settings */ +.modal.mod-settings .modal-content { + padding: 0; } +.modal.mod-settings .vertical-tab-content-container { + padding-top: 0; } + .modal.mod-settings .vertical-tab-content-container .vertical-tab-content { + padding-top: 30px; } + +.horizontal-tab-content, +.vertical-tab-content { + background: var(--background-primary); + padding-bottom: 100px; + padding-left: 40px; + padding-right: 40px; } + +.vertical-tab-header, +.vertical-tab-content { + padding-bottom: 100px; } + +.modal.mod-community-plugin .modal-content { + padding: 0; } + +.plugin-list-plugins { + overflow: visible; } + +.clickable-icon { + margin: 0; } + +.installed-plugins-container .clickable-icon { + margin: 0; } + +.installed-plugins-container .clickable-icon[aria-label="Uninstall"] { + margin: 0; } + +.plugin-list-plugins .clickable-icon { + margin: 0; } + +.hotkey-list-container { + padding-right: 0; } + +/* Themes */ +body .modal.mod-community-theme { + max-width: 1000px; } + +.community-theme-container { + padding-top: 10px; } + +.community-theme-container, +.hotkey-settings-container { + height: auto; + overflow: visible; } + +.theme-list { + justify-content: space-evenly; } + +.community-theme-filters-container, +.hotkey-search-container { + padding: 0 0 20px 0; } + +.modal.mod-community-theme { + padding: 0; } + +.modal.mod-community-theme .modal-content { + padding: 30px; } + +.community-theme { + padding: 0; + margin: 0 0 2em 0; + align-items: stretch; + background: transparent; } + +.community-theme-title { + text-align: left; + font-size: var(--font-settings); } + +.community-theme-info + div { + background-color: var(--background-secondary); + display: flex; + align-items: center; + padding: 0; + flex-grow: 1; + border-radius: 20px; } + +.community-theme-info { + line-height: 1; + flex-grow: 0; + padding: 0 0 10px 0; + align-items: flex-end; + justify-content: flex-start; + flex-wrap: wrap; } + +.community-theme-remove-button { + padding: 4px 6px; + display: flex; + color: var(--text-muted); + background-color: transparent; } + +.community-theme .community-theme-screenshot { + max-width: 100%; } + +body:not(.is-mobile) .theme-list { + display: grid; + grid-template-columns: repeat(3, 1fr); + gap: 0 2em; } + body:not(.is-mobile) .theme-list .community-theme { + align-self: stretch; + justify-self: center; + max-width: 100%; + width: 100%; + background-color: var(--background-secondary); + padding: 18px; + border-radius: var(--radius-l); + border: 2px solid transparent; } + body:not(.is-mobile) .theme-list .community-theme:hover { + border: 2px solid var(--text-accent); } + body:not(.is-mobile) .theme-list .community-theme.is-selected { + grid-column: 1/4; + grid-row: 1; + max-width: 100%; + display: grid; + grid-template-columns: 1.5fr 2fr; + padding: 20px 20px; + border-radius: var(--radius-xl); + border-color: transparent; } + body:not(.is-mobile) .theme-list .community-theme.is-selected .community-theme-info { + display: grid; + grid-template-columns: repeat(2, 1fr); + grid-template-rows: 30px 50px 440px; + margin: 0 40px 0 0; } + body:not(.is-mobile) .theme-list .community-theme.is-selected .community-theme-title { + grid-column: 1/3; + grid-row: 1/2; + text-align: left; + font-size: 2em; + font-weight: 500; + margin: 0; } + body:not(.is-mobile) .theme-list .community-theme.is-selected .community-theme-info + div { + display: flex; + align-items: center; + flex-grow: 1; + box-shadow: none; } + body:not(.is-mobile) .theme-list .community-theme.is-selected .community-theme-downloads { + text-align: right; } + body:not(.is-mobile) .theme-list .community-theme.is-selected .community-theme-remove-button { + bottom: 20px; + left: 0px; + right: auto; + top: auto; + color: var(--text-faint); + display: flex; + align-items: center; } + body:not(.is-mobile) .theme-list .community-theme.is-selected .community-theme-remove-button:after { + content: 'Delete theme'; + padding-left: 5px; } + body:not(.is-mobile) .theme-list .community-theme.is-selected .community-theme-remove-button:hover { + color: var(--text-error); } + body:not(.is-mobile) .theme-list .community-theme.is-selected .modal-button-container { + grid-column: 2; + grid-row: 1/2; + margin-top: 0; + margin-left: auto; + margin-right: 0; } + body:not(.is-mobile) .theme-list .community-theme.is-selected .modal-button-container button { + margin: 0; + width: 160px; + height: 36px; + cursor: pointer; + border: none; + box-shadow: inset 0 0 0 1px rgba(0, 0, 0, 0.1); } + body:not(.is-mobile) .theme-list .community-theme.is-selected .modal-button-container button:hover { + background-color: var(--ax2); } + body:not(.is-mobile) .theme-list .community-theme.is-selected .modal-button-container button:not(.mod-cta) { + display: none; } + body:not(.is-mobile) .theme-list .community-theme.is-selected .community-theme-info::after { + grid-column: 1/3; + grid-row: 3/4; + padding-top: 20px; + align-self: flex-start; + justify-self: flex-start; + content: var(--minimal-version); + color: var(--text-normal); + font-size: var(--font-adaptive-normal); + line-height: 1.4; + width: 100%; + position: relative; + white-space: pre-wrap; + text-align: left; + border: none; } + +.community-theme-remove-button { + top: 15px; } + .community-theme-remove-button:hover { + color: var(--text-error); } + +.community-theme.is-selected { + padding-left: 0; + padding-right: 0; + background-color: transparent; + color: var(--text-normal); } + .community-theme.is-selected .community-theme-info + div { + box-shadow: 0px 0.5px 1px 0.5px rgba(0, 0, 0, 0.1), inset 0 0 0 2px var(--text-accent); } + .community-theme.is-selected .community-theme-downloads, + .community-theme.is-selected .community-theme-info { + margin-bottom: 0; + color: var(--text-muted); } + .community-theme.is-selected .community-theme-info .clickable-icon { + width: 100%; + background-color: var(--background-primary); + border: 1px solid var(--background-modifier-border); + color: var(--text-normal); + cursor: pointer; + display: block; + text-align: center; + grid-column: 1/3; + padding: 7px 0; + margin: 20px 0 0; + height: 36px; + border-radius: 5px; + box-shadow: 0 1px 1px 0px var(--btn-shadow-color); } + .community-theme.is-selected .community-theme-info .clickable-icon:hover { + border: 1px solid var(--background-modifier-border-hover); + box-shadow: 0 2px 3px 0px var(--btn-shadow-color); } + .community-theme.is-selected .community-theme-info .clickable-icon::after { + content: "Learn more"; + padding-left: 4px; } + .community-theme.is-selected .modal-button-container .mod-cta { + background-color: var(--interactive-accent); + color: white; } + .community-theme.is-selected .modal-button-container .mod-cta:hover { + background-color: var(--interactive-accent-hover); } + +.modal.mod-settings .vertical-tab-header { + background: var(--background-secondary); + padding-top: 5px; + flex: 0 0 220px; + padding-bottom: 100px; } + +.vertical-tab-header-group-title { + color: var(--text-faint); + text-transform: none; + font-size: 12px; + letter-spacing: 0; + font-weight: 500; } + +.vertical-tab-nav-item { + padding: 5px 8px; + margin: 0 8px 0; + color: var(--text-muted); + font-weight: 400; + border: none; + background: var(--background-secondary); + cursor: var(--cursor); + border-radius: var(--radius-m); } + +.vertical-tab-nav-item:hover { + color: var(--text-normal); } + +.vertical-tab-nav-item.is-active { + color: var(--text-normal); + background-color: var(--background-tertiary); } + +.setting-hotkey { + background-color: var(--background-tertiary); + padding: 3px 4px 3px 8px; + display: flex; + align-items: center; } + +.setting-hotkey-icon.setting-delete-hotkey { + margin-left: 3px; + cursor: var(--cursor); } + +.setting-delete-hotkey:hover { + background-color: transparent; } + +body:not(.minimal-icons) .setting-hotkey-icon.setting-delete-hotkey svg { + width: 16px; + height: 16px; } + +.setting-hotkey.mod-empty { + background: transparent; + color: var(--text-faint); } + +.setting-item { + padding: 0.75rem 0; } + +.setting-item-description { + padding-top: 4px; } + +.setting-item-control { + margin-right: 0; + gap: 8px; } + +/* Status bar */ +.workspace-split.mod-left-split > .workspace-leaf-resize-handle, +.workspace-split.mod-right-split > .workspace-leaf-resize-handle { + height: 100%; } + +.status-bar { + transition: color 200ms linear; + color: var(--text-faint); + font-size: var(--font-adaptive-smaller); + border-top: var(--border-width) solid var(--background-divider); + line-height: 1; + max-height: 24px; } + +.minimal-status-off .status-bar { + background-color: var(--background-secondary); + border-width: var(--border-width); + padding: 2px 6px 4px; } + +body:not(.minimal-status-off) .status-bar { + background-color: var(--background-primary); + z-index: 30; + border-top-left-radius: 5px; + width: auto; + position: absolute; + left: auto; + border: 0; + bottom: 0; + right: 0; + max-height: 26px; + padding: 2px 8px 6px 3px; } + +/* +body.plugin-sliding-panes-rotate-header:not(.minimal-status-off) .status-bar { + border-top:1px solid var(--background-modifier-border); + border-left:1px solid var(--background-modifier-border); +}*/ +.sync-status-icon.mod-working, +.sync-status-icon.mod-success { + color: var(--text-faint); + cursor: var(--cursor); } + +.status-bar:hover .sync-status-icon.mod-working, +.status-bar:hover .sync-status-icon.mod-success, +.status-bar:hover { + color: var(--text-muted); + transition: color 200ms linear; } + +.status-bar .plugin-sync:hover .sync-status-icon.mod-working, +.status-bar .plugin-sync:hover .sync-status-icon.mod-success { + color: var(--text-normal); } + +.status-bar-item-segment { + margin-right: 10px; } + +.status-bar-item, +.sync-status-icon { + display: flex; + align-items: center; } + +.status-bar-item { + padding: 7px 4px; + margin: 0 0 0 0; + cursor: var(--cursor) !important; } + .status-bar-item .status-bar-item-icon { + line-height: 0; } + .status-bar-item.plugin-editor-status:hover, .status-bar-item.plugin-sync:hover, .status-bar-item.cMenu-statusbar-button:hover, .status-bar-item.mod-clickable:hover { + text-align: center; + background-color: var(--background-tertiary) !important; + border-radius: 4px; } + .status-bar-item.plugin-editor-status svg, .status-bar-item.plugin-sync svg { + height: 15px; + width: 15px; } + +/* Syntax highlighting */ +.theme-light code[class*="language-"], +.theme-light pre[class*="language-"], +.theme-dark code[class*="language-"], +.theme-dark pre[class*="language-"] { + color: var(--tx1); } +.theme-light .token.prolog, +.theme-light .token.doctype, +.theme-light .token.cdata, +.theme-light .cm-meta, +.theme-light .cm-qualifier, +.theme-dark .token.prolog, +.theme-dark .token.doctype, +.theme-dark .token.cdata, +.theme-dark .cm-meta, +.theme-dark .cm-qualifier { + color: var(--tx2); } +.theme-light .cm-comment, +.theme-light .token.comment, +.theme-dark .cm-comment, +.theme-dark .token.comment { + color: var(--tx2); } +.theme-light .token.tag, +.theme-light .token.constant, +.theme-light .token.symbol, +.theme-light .token.deleted, +.theme-light .cm-tag, +.theme-dark .token.tag, +.theme-dark .token.constant, +.theme-dark .token.symbol, +.theme-dark .token.deleted, +.theme-dark .cm-tag { + color: var(--red); } +.theme-light .token.punctuation, +.theme-light .cm-punctuation, +.theme-light .cm-bracket, +.theme-light .cm-hr, +.theme-dark .token.punctuation, +.theme-dark .cm-punctuation, +.theme-dark .cm-bracket, +.theme-dark .cm-hr { + color: var(--tx2); } +.theme-light .token.boolean, +.theme-light .token.number, +.theme-light .cm-number, +.theme-dark .token.boolean, +.theme-dark .token.number, +.theme-dark .cm-number { + color: var(--purple); } +.theme-light .token.selector, +.theme-light .token.attr-name, +.theme-light .token.string, +.theme-light .token.char, +.theme-light .token.builtin, +.theme-light .token.inserted, +.theme-light .cm-string, +.theme-light .cm-string-2, +.theme-dark .token.selector, +.theme-dark .token.attr-name, +.theme-dark .token.string, +.theme-dark .token.char, +.theme-dark .token.builtin, +.theme-dark .token.inserted, +.theme-dark .cm-string, +.theme-dark .cm-string-2 { + color: var(--green); } +.theme-light .cm-property, +.theme-light .token.property, +.theme-light .token.operator, +.theme-light .token.entity, +.theme-light .token.url, +.theme-light .language-css .token.string, +.theme-light .style .token.string, +.theme-light .token.variable, +.theme-light .cm-operator, +.theme-light .cm-link, +.theme-light .cm-variable-2, +.theme-light .cm-variable-3, +.theme-dark .cm-property, +.theme-dark .token.property, +.theme-dark .token.operator, +.theme-dark .token.entity, +.theme-dark .token.url, +.theme-dark .language-css .token.string, +.theme-dark .style .token.string, +.theme-dark .token.variable, +.theme-dark .cm-operator, +.theme-dark .cm-link, +.theme-dark .cm-variable-2, +.theme-dark .cm-variable-3 { + color: var(--cyan); } +.theme-light .token.atrule, +.theme-light .token.attr-value, +.theme-light .token.function, +.theme-light .token.class-name, +.theme-light .cm-attribute, +.theme-light .cm-variable, +.theme-light .cm-type, +.theme-light .cm-def, +.theme-dark .token.atrule, +.theme-dark .token.attr-value, +.theme-dark .token.function, +.theme-dark .token.class-name, +.theme-dark .cm-attribute, +.theme-dark .cm-variable, +.theme-dark .cm-type, +.theme-dark .cm-def { + color: var(--yellow); } +.theme-light .token.keyword, +.theme-light .cm-keyword, +.theme-light .cm-builtin, +.theme-dark .token.keyword, +.theme-dark .cm-keyword, +.theme-dark .cm-builtin { + color: var(--pink); } +.theme-light .token.regex, +.theme-light .token.important, +.theme-dark .token.regex, +.theme-dark .token.important { + color: var(--orange); } + +/* Preview mode tables */ +.markdown-source-view.mod-cm6 table { + border-collapse: collapse; } + +.markdown-preview-view table { + margin-block-start: 1em; } + +.markdown-source-view.mod-cm6 td, +.markdown-source-view.mod-cm6 th, +.markdown-preview-view th, +.markdown-preview-view td { + padding: 4px 10px; } + +.markdown-source-view.mod-cm6 td, +.markdown-preview-view td { + font-size: var(--table-font-size); } + +.markdown-source-view.mod-cm6 th, +.markdown-preview-view th { + font-weight: 400; + font-size: var(--table-font-size); + color: var(--text-muted); + border-top: none; + text-align: left; } + .markdown-source-view.mod-cm6 th[align="center"], + .markdown-preview-view th[align="center"] { + text-align: center; } + .markdown-source-view.mod-cm6 th[align="right"], + .markdown-preview-view th[align="right"] { + text-align: right; } + +.markdown-source-view.mod-cm6 th:last-child, +.markdown-source-view.mod-cm6 td:last-child, +.markdown-preview-view th:last-child, +.markdown-preview-view td:last-child { + border-right: none; } + +.markdown-source-view.mod-cm6 th:first-child, +.markdown-source-view.mod-cm6 td:first-child, +.markdown-preview-view th:first-child, +.markdown-preview-view td:first-child { + border-left: none; + padding-left: 0; } + +.markdown-source-view.mod-cm6 tr:last-child td, +.markdown-preview-view tr:last-child td { + border-bottom: none; } + +/* Legacy Editor Tables */ +.CodeMirror pre.HyperMD-table-row { + font-family: var(--font-monospace); + font-size: var(--table-font-size); } + +/* Live Preview Tables */ +.is-live-preview .el-table { + width: 100%; + max-width: 100%; } + +.cm-s-obsidian .HyperMD-table-row { + font-size: var(--table-font-size); } + +.cm-s-obsidian .HyperMD-table-row span.cm-hmd-table-sep, +.cm-hmd-table-sep-dummy { + color: var(--text-faint); + font-weight: 400; } + +/* Tags */ +body.minimal-unstyled-tags .frontmatter-container .tag, +body.minimal-unstyled-tags a.tag, +body.minimal-unstyled-tags .cm-s-obsidian span.cm-hashtag { + color: var(--tag-color); + font-weight: var(--link-weight); + text-decoration: none; } + body.minimal-unstyled-tags .frontmatter-container .tag:hover, + body.minimal-unstyled-tags a.tag:hover, + body.minimal-unstyled-tags .cm-s-obsidian span.cm-hashtag:hover { + color: var(--text-normal); } + +body:not(.minimal-unstyled-tags) .frontmatter-container .tag, +body:not(.minimal-unstyled-tags) a.tag { + background-color: var(--tag-bg); + border: var(--tag-border-width) solid var(--background-modifier-border); + color: var(--tag-color); + font-size: calc(var(--font-adaptive-normal) * 0.8); + font-weight: var(--link-weight); + font-family: var(--font-interface); + padding: 1px 8px; + text-align: center; + text-decoration: none; + vertical-align: middle; + display: inline-block; + margin: 1px 0; + border-radius: var(--tag-radius); } +body:not(.minimal-unstyled-tags) a.tag:hover { + color: var(--text-normal); + border-color: var(--background-modifier-border-hover); + background-color: var(--tag-bg2); } +body:not(.minimal-unstyled-tags) .cm-s-obsidian span.cm-hashtag { + background-color: var(--tag-bg); + border: var(--tag-border-width) solid var(--background-modifier-border); + color: var(--tag-color); + font-size: calc(var(--font-adaptive-normal) * 0.8); + font-family: var(--font-interface); + font-weight: var(--link-weight); + text-align: center; + text-decoration: none; + margin: 0; + vertical-align: text-bottom; + padding-top: 2px; + border-left: none; + border-right: none; + padding-bottom: 3px; + cursor: text; } +body:not(.minimal-unstyled-tags) .cm-s-obsidian span.cm-hashtag:hover { + background-color: var(--tag-bg2); } +body:not(.minimal-unstyled-tags) span.cm-hashtag.cm-hashtag-begin { + border-top-left-radius: var(--tag-radius); + border-bottom-left-radius: var(--tag-radius); + padding-left: 8px; + border-right: none; + border-left: var(--tag-border-width) solid var(--background-modifier-border); } +body:not(.minimal-unstyled-tags) span.cm-hashtag.cm-hashtag-end { + border-top-right-radius: var(--tag-radius); + border-bottom-right-radius: var(--tag-radius); + border-left: none; + padding-right: 8px; + border-right: var(--tag-border-width) solid var(--background-modifier-border); } + +/* Tag pane */ +.tag-container { + padding-left: 15px; } + +.tag-pane-tag-count { + padding: 0; + color: var(--text-faint); } + +.pane-list-item-ending-flair { + background: transparent; } + +.tag-pane-tag { + padding: 2px 5px 2px 5px; + cursor: var(--cursor); } + +.tag-pane-tag:hover { + background: transparent; } + +.nav-file.is-active .nav-file-title:hover { + background: var(--background-tertiary) !important; } + +.nav-file.is-active > .nav-file-title { + background: var(--background-tertiary); } + +/* Tooltips */ +.tooltip { + font-size: var(--font-adaptive-smaller); + line-height: 1.3; + font-weight: 500; + padding: 4px 8px; + border-radius: 4px; + transition: none; + text-align: left; + animation: none; + opacity: 0.8; } + +.tooltip.mod-left, +.tooltip.mod-right { + transform: none; + animation: none; } + +/* Title Bar */ +/* Alignment */ +.title-align-left:not(.plugin-sliding-panes-rotate-header) .view-header-title-container { + margin-left: 5px; } + +.title-align-center:not(.plugin-sliding-panes-rotate-header) .view-header-title { + margin-left: 0; + padding-right: 0; + text-align: center; } + +.title-align-left:not(.plugin-sliding-panes-rotate-header) .view-header-title-container, +.title-align-center:not(.plugin-sliding-panes-rotate-header) .view-header-title-container { + width: auto; + position: static; } + +.mod-macos.hider-frameless:not(.is-fullscreen):not(.plugin-sliding-panes-rotate-header) .workspace-split.mod-left-split.is-collapsed + .mod-root .workspace-leaf:first-of-type .view-header-title-container { + max-width: calc(100% - (var(--traffic-x-space) * 2) - 30px); } + +.mod-macos.is-popout-window.hider-frameless:not(.is-fullscreen):not(.plugin-sliding-panes-rotate-header) .mod-root .workspace-leaf:first-of-type .view-header-title-container { + max-width: calc(100% - (var(--traffic-x-space) * 2) - 30px); } + +.view-header { + height: var(--header-height); + align-items: center; } + +/* Left side title bar icon */ +body:not(.minimal-icons-off) div.view-header-icon svg { + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' enable-background='new 0 0 32 32' viewBox='0 0 32 32' xml:space='preserve'%3E%3Cpath d='M10 6h4v4h-4zm8 0h4v4h-4zm-8 8h4v4h-4zm8 0h4v4h-4zm-8 8h4v4h-4zm8 0h4v4h-4z'/%3E%3Cpath fill='none' d='M0 0h32v32H0z'/%3E%3C/svg%3E"); } + +.view-header-icon { + margin-left: var(--traffic-x-space); + opacity: 0; + top: 0; + left: 4px; + z-index: 20; } + +.show-grabber .view-header-icon { + opacity: var(--icon-muted); } + +.show-grabber .view-header-icon:hover { + opacity: 1; } + +.view-header-icon:hover { + cursor: grab; } + +.view-header-icon:active { + cursor: grabbing; } + +/* Right side title bar icon */ +.view-actions { + margin-right: 1px; + height: calc(var(--header-height) - 1px); + top: 0; + align-items: center; + z-index: 15; + background: var(--background-primary); } + +/* Title area */ +.view-header-title { + padding-right: 80px; } + +/* Fade out title +body:not(.is-mobile) .view-header-title:before { + background:linear-gradient(90deg,transparent 0%,var(--background-primary) 80%); + width:50px; + content:" "; + height:var(--header-height); + display:inline-block; + vertical-align:bottom; + position:absolute; + right:50px; + pointer-events:none; +}*/ +.workspace-leaf-header, +.view-header, +.workspace-leaf.mod-active .view-header, +.workspace-split.mod-root > .workspace-leaf:first-of-type:last-of-type .view-header { + background-color: var(--background-primary) !important; + border-top: none; + border-bottom: none; } + +.view-header-title-container { + padding-left: 0; + padding-right: 0px; + position: absolute; + width: var(--line-width-adaptive); + max-width: var(--max-width); + margin: 0 auto; + left: 0; + right: 0; } + +.view-header-title-container:after { + display: none; } + +.view-actions { + padding: 0px 6px; + margin-right: 0px; + margin-left: auto; + transition: opacity 0.25s ease-in-out; } + +.view-actions .view-action { + margin: 0; + top: 0; + padding: 4px; + border-radius: var(--radius-m); + display: flex; + align-items: center; } + +body:not(.is-mobile) .view-actions .view-action { + height: 26px; } + +.view-action.is-active { + color: var(--icon-color); + opacity: var(--icon-muted); } + +body:not(.is-mobile) .view-actions .view-action:last-child { + margin-left: -1px; } + +body:not(.minimal-focus-mode) .workspace-ribbon:not(.is-collapsed) ~ .mod-root .view-actions, +.minimal-focus-mode .workspace-ribbon:not(.is-collapsed) ~ .mod-root .view-header:hover .view-actions, +.workspace-ribbon.mod-left.is-collapsed ~ .mod-root .view-header:hover .view-actions, +.mod-right.is-collapsed ~ .mod-root .view-header:hover .view-actions, +.view-action.is-active:hover { + opacity: 1; + transition: opacity 0.25s ease-in-out; } + +.view-content { + height: calc(100% - var(--header-height)); } + +/* Window frame */ +body:not(.hider-frameless):not(.is-fullscreen):not(.is-mobile) { + --titlebar-height:28px; + padding-top: var(--titlebar-height) !important; } + +body:not(.hider-frameless):not(.is-fullscreen):not(.is-mobile) .titlebar { + background: var(--background-secondary); + border-bottom: var(--border-width) solid var(--background-divider); + height: var(--titlebar-height) !important; + top: 0 !important; + padding-top: 0 !important; } + +body.hider-frameless .titlebar { + border-bottom: none; } + +.mod-windows .titlebar-button:hover { + background-color: var(--background-primary-alt); } + +.mod-windows .titlebar-button.mod-close:hover { + background-color: var(--background-modifier-error); } + +.mod-windows .mod-close:hover svg { + fill: white !important; + stroke: white !important; } + +.titlebar-button-container { + height: var(--titlebar-height); + top: 0; + display: flex; + align-items: center; } + +.titlebar:hover .titlebar-button-container.mod-left { + opacity: 1; } + +.is-focused .titlebar-text { + color: var(--text-normal); } + +.titlebar-text { + font-weight: 600; + color: var(--text-faint); + letter-spacing: inherit; } + +body:not(.window-title-on) .titlebar-text { + display: none; } + +.titlebar-button:hover { + opacity: 1; + transition: opacity 100ms ease-out; } + +.titlebar-button { + opacity: 0.5; + cursor: var(--cursor); + color: var(--text-muted); + padding: 2px 4px; + border-radius: 3px; + line-height: 1; + display: flex; } + +.titlebar-button:hover { + background-color: var(--background-tertiary); } + +.titlebar-button-container.mod-left .titlebar-button { + margin-right: 5px; } + +.titlebar-button-container.mod-right .titlebar-button { + margin-left: 0; + border-radius: 0; + height: 100%; + align-items: center; + padding: 2px 15px; } + +/* Workspace */ +/* Empty state */ +.empty-state { + background-color: var(--background-primary); + text-align: center; } + +.workspace-leaf-content[data-type="empty"] .view-header, +.empty-state-title { + display: none; } + +.empty-state-action-list { + color: var(--text-normal); + font-size: var(--font-adaptive-normal); } + +/* Empty side pane */ +.pane-empty { + text-align: center; + color: var(--text-faint); + font-size: var(--font-adaptive-small); } + +.workspace-split.mod-root { + background-color: var(--background-primary); } + +.workspace-split.mod-vertical > .workspace-split { + padding: 0; } + +.workspace-split .workspace-tabs { + background: var(--background-primary); } + +.workspace-split:not(.mod-right-split) .workspace-tabs { + background: var(--background-secondary); } + +.workspace-split.mod-root > .workspace-leaf:first-of-type .workspace-leaf-content, +.workspace-split.mod-root > .workspace-leaf:last-of-type .workspace-leaf-content { + border-top-right-radius: 0px; + border-top-left-radius: 0px; } + +/* Resize handles */ +.workspace-split.mod-root.mod-horizontal .workspace-leaf-resize-handle, +.workspace-split.mod-root.mod-vertical .workspace-leaf-resize-handle { + border-width: 1px; } + +.workspace-split.mod-horizontal > * > .workspace-leaf-resize-handle { + height: 3px; + background: transparent; + border-bottom: var(--border-width-alt) solid var(--background-divider); } + +.workspace-split.mod-right-split > .workspace-leaf-resize-handle { + background: transparent; + border-left: var(--border-width-alt) solid var(--background-divider); + width: 3px !important; } + +.workspace-split.mod-vertical > * > .workspace-leaf-resize-handle, +.workspace-split.mod-left-split > .workspace-leaf-resize-handle { + border-right: var(--border-width) solid var(--background-divider); + width: 4px !important; + background: transparent; } + +.workspace-split.mod-right-split > .workspace-leaf-resize-handle:hover, +.workspace-split.mod-horizontal > * > .workspace-leaf-resize-handle:hover, +.workspace-split.mod-vertical > * > .workspace-leaf-resize-handle:hover, +.workspace-split.mod-left-split > .workspace-leaf-resize-handle:hover { + border-color: var(--background-modifier-border-hover); + transition: border-color 0.1s ease-in-out 0.05s, border-width 0.1s ease-in-out 0.05s; + border-width: 2px; } + +.workspace-split.mod-right-split > .workspace-leaf-resize-handle:active, +.workspace-split.mod-horizontal > * > .workspace-leaf-resize-handle:active, +.workspace-split.mod-vertical > * > .workspace-leaf-resize-handle:active, +.workspace-split.mod-left-split > .workspace-leaf-resize-handle:active { + border-color: var(--background-modifier-border-focus); + border-width: 2px; } + +.workspace-tab-container-before, +.workspace-tab-container-after { + width: 0; } + +.workspace-leaf { + border-left: 0px; } + +.workspace-tabs .workspace-leaf, +.workspace-tabs .workspace-leaf.mod-active { + border: none; } + +.mod-horizontal .workspace-leaf { + border-bottom: 0px; + background-color: transparent; + box-shadow: none !important; } + +.workspace-split.mod-right-split .workspace-tabs .workspace-leaf { + border-radius: 0; } + +/* Effects on non-active panels */ +.workspace-tab-container-inner { + background: transparent; + border-radius: 0; + width: 100%; + max-width: 100%; + margin: 0 auto; + padding-left: 5px; } + +.workspace-tabs .workspace-tab-header-container { + border: none; } + +.workspace-sidedock-empty-state + .workspace-tabs .workspace-tab-header-container { + border-bottom: var(--border-width) solid var(--background-divider); } + +.mod-right-split .workspace-tabs .nav-buttons-container { + z-index: 1; } + +.workspace-tab-header.is-before-active .workspace-tab-header-inner, +.workspace-tab-header.is-active, +.workspace-tab-header.is-after-active, +.workspace-tab-header.is-after-active .workspace-tab-header-inner, +.workspace-tab-header.is-before-active, +.workspace-tab-header.is-after-active { + background: transparent; } + +.workspace-tabs { + border: 0; + padding-right: 0; + font-size: 100%; } + +.workspace-tab-container-inner { + padding-left: 6px; } + +.workspace-tab-header-inner { + padding: 0px 0px 0 2px; } + +.workspace-tab-header-container { + height: var(--header-height); + padding: 0; + align-items: center; + background-color: transparent; } + +.workspace-tab-header-container { + border-bottom: var(--border-width) solid var(--background-divider); } + +/* Components */ +/* Audio files */ +.theme-dark audio { + filter: none; } + +.theme-dark audio::-webkit-media-controls-play-button, +.theme-dark audio::-internal-media-controls-overflow-button, +.theme-dark audio::-webkit-media-controls-timeline, +.theme-dark audio::-webkit-media-controls-volume-control-container, +.theme-dark audio::-webkit-media-controls-current-time-display, +.theme-dark audio::-webkit-media-controls-time-remaining-display, +.theme-dark audio::-internal-media-controls-overflow-button { + filter: invert(1); } + +audio { + height: 36px; + border-radius: 4px; } + +audio::-webkit-media-controls-enclosure { + border: 1px solid var(--background-modifier-border); + background-color: var(--background-secondary); } + +audio::-webkit-media-controls-current-time-display { + color: var(--text-normal); + font-family: var(--font-interface); + font-size: var(--font-adaptive-small); + text-shadow: none; } + +audio::-webkit-media-controls-time-remaining-display { + color: var(--text-muted); + font-family: var(--font-interface); + font-size: var(--font-adaptive-small); + text-shadow: none; } + +audio::-webkit-media-controls-panel { + padding: 2px 1.5px; } + +audio::-webkit-media-controls input[pseudo="-internal-media-controls-overflow-button" i]:enabled:hover::-internal-media-controls-button-hover-background { + background-color: transparent; } + +/* Buttons */ +button { + cursor: var(--cursor); } + +button, +.setting-item-control button { + font-family: var(--font-interface); + font-size: var(--font-inputs); + font-weight: 400; + border-radius: var(--radius-m); } + +button:active, +button:focus { + -webkit-appearance: none; + border-color: var(--background-modifier-border-hover); } + +body:not(.is-mobile) button:active, +body:not(.is-mobile) button:focus { + box-shadow: 0 0 0px 2px var(--background-modifier-border-hover); } + +.modal.mod-settings button:not(.mod-cta):not(.mod-warning), +.modal button:not(.mod-warning), +.modal.mod-settings button:not(.mod-warning) { + background-color: var(--interactive-normal); + color: var(--text-normal); + border: 1px solid var(--background-modifier-border); + box-shadow: 0 1px 1px 0px var(--btn-shadow-color); + cursor: var(--cursor); + height: var(--input-height); + line-height: 0; + white-space: nowrap; + transition: background-color 0.2s ease-out, border-color 0.2s ease-out; } + +button.mod-warning { + border: 1px solid var(--background-modifier-error); + color: var(--text-error); + box-shadow: 0 1px 1px 0px var(--btn-shadow-color); + transition: background-color 0.2s ease-out; } + +button.mod-warning:hover { + border: 1px solid var(--background-modifier-error); + color: var(--text-error); + box-shadow: 0 2px 3px 0px var(--btn-shadow-color); + transition: background-color 0.2s ease-out; } + +button:hover, +.modal button:not(.mod-warning):hover, +.modal.mod-settings button:not(.mod-warning):hover { + background-color: var(--interactive-normal); + border-color: var(--background-modifier-border-hover); + box-shadow: 0 2px 3px 0px var(--btn-shadow-color); + transition: background-color 0.2s ease-out, border-color 0.2s ease-out; } + +.is-mobile button.copy-code-button { + width: auto; + margin-right: 4px; } + +/* Dropdowns */ +.dropdown, +body .addChoiceBox #addChoiceTypeSelector { + font-family: var(--font-interface); + font-size: var(--font-inputs); } + +.dropdown, +select { + box-shadow: 0 1px 1px 0px var(--btn-shadow-color); + background-color: var(--interactive-normal); + border-color: var(--background-modifier-border); + transition: border-color 0.1s linear; + height: var(--input-height); + font-family: var(--font-interface); + border-radius: var(--radius-m); } + +.dropdown { + background-image: url("data:image/svg+xml;charset=US-ASCII,<%2Fsvg>"); } + +.theme-dark .dropdown { + background-image: url("data:image/svg+xml;charset=US-ASCII,<%2Fsvg>"); } + +.dropdown:hover, +select:hover { + background-color: var(--interactive-normal); + box-shadow: 0 2px 3px 0px var(--btn-shadow-color); + border-color: var(--background-modifier-border-hover); + transition: all 0.1s linear; } + +.dropdown:focus, +.dropdown:active, +select:focus, +select:active { + -webkit-appearance: none; + border-color: var(--background-modifier-border-hover); } + +body:not(.is-mobile) .dropdown:focus, +body:not(.is-mobile) .dropdown:active, +body:not(.is-mobile) select:focus, +body:not(.is-mobile) select:active { + box-shadow: 0 0 0px 2px var(--background-modifier-border-hover); } + +/* Input fields */ +textarea, +input[type='text'], +input[type='search'], +input[type='email'], +input[type='password'], +input[type='number'] { + font-family: var(--font-interface); + font-size: var(--font-inputs); } + +textarea { + padding: 5px 10px; + transition: box-shadow 0.1s linear; + -webkit-appearance: none; + line-height: 1.3; } + +input[type='text'], +input[type='search'], +input[type='email'], +input[type='password'], +input[type='number'] { + padding: 5px 10px; + -webkit-appearance: none; + transition: box-shadow 0.1s linear; + height: var(--input-height); } + +textarea:hover, +input:hover { + border-color: var(--background-modifier-border-hover); + transition: border-color 0.1s linear, box-shadow 0.1s linear; } + +textarea:active, +textarea:focus, +input[type='text']:active, +input[type='search']:active, +input[type='email']:active, +input[type='password']:active, +input[type='number']:active, +input[type='text']:focus, +input[type='search']:focus, +input[type='email']:focus, +input[type='password']:focus, +input[type='number']:focus { + -webkit-appearance: none; + border-color: var(--background-modifier-border-hover); } + +body:not(.is-mobile) textarea:active, +body:not(.is-mobile) textarea:focus, +body:not(.is-mobile) .dropdown:focus, +body:not(.is-mobile) .dropdown:active, +body:not(.is-mobile) select:focus, +body:not(.is-mobile) select:active, +body:not(.is-mobile) input:focus { + box-shadow: 0 0 0px 2px var(--background-modifier-border-hover); + transition: border-color 0.1s linear, box-shadow 0.1s linear; } + +/* Progress bars */ +.theme-light { + --progress-outline:rgba(0,0,0,0.05); } + +.theme-dark { + --progress-outline:rgba(255,255,255,0.04); } + +.markdown-source-view.is-live-preview progress, +.markdown-preview-view progress { + -webkit-writing-mode: horizontal-tb; + writing-mode: horizontal-tb; + appearance: none; + box-sizing: border-box; + display: inline-block; + height: 5px; + margin-bottom: 4px; + width: 220px; + max-width: 100%; + overflow: hidden; + border-radius: 0px; + border: 0; + vertical-align: -0.2rem; } + .markdown-source-view.is-live-preview progress[value]::-webkit-progress-bar, + .markdown-preview-view progress[value]::-webkit-progress-bar { + background-color: var(--background-tertiary); + box-shadow: inset 0px 0px 0px var(--border-width) var(--progress-outline); + border-radius: 5px; + overflow: hidden; } + .markdown-source-view.is-live-preview progress[value]::-webkit-progress-value, + .markdown-preview-view progress[value]::-webkit-progress-value { + background-color: var(--text-accent); + overflow: hidden; } + .markdown-source-view.is-live-preview progress[value^='1']::-webkit-progress-value, .markdown-source-view.is-live-preview progress[value^='2']::-webkit-progress-value, .markdown-source-view.is-live-preview progress[value^='3']::-webkit-progress-value, + .markdown-preview-view progress[value^='1']::-webkit-progress-value, + .markdown-preview-view progress[value^='2']::-webkit-progress-value, + .markdown-preview-view progress[value^='3']::-webkit-progress-value { + background-color: var(--red); } + .markdown-source-view.is-live-preview progress[value^='4']::-webkit-progress-value, .markdown-source-view.is-live-preview progress[value^='5']::-webkit-progress-value, + .markdown-preview-view progress[value^='4']::-webkit-progress-value, + .markdown-preview-view progress[value^='5']::-webkit-progress-value { + background-color: var(--orange); } + .markdown-source-view.is-live-preview progress[value^='6']::-webkit-progress-value, .markdown-source-view.is-live-preview progress[value^='7']::-webkit-progress-value, + .markdown-preview-view progress[value^='6']::-webkit-progress-value, + .markdown-preview-view progress[value^='7']::-webkit-progress-value { + background-color: var(--yellow); } + .markdown-source-view.is-live-preview progress[value^='8']::-webkit-progress-value, .markdown-source-view.is-live-preview progress[value^='9']::-webkit-progress-value, + .markdown-preview-view progress[value^='8']::-webkit-progress-value, + .markdown-preview-view progress[value^='9']::-webkit-progress-value { + background-color: var(--green); } + .markdown-source-view.is-live-preview progress[value='1']::-webkit-progress-value, .markdown-source-view.is-live-preview progress[value='100']::-webkit-progress-value, + .markdown-preview-view progress[value='1']::-webkit-progress-value, + .markdown-preview-view progress[value='100']::-webkit-progress-value { + background-color: var(--text-accent); } + .markdown-source-view.is-live-preview progress[value='0']::-webkit-progress-value, .markdown-source-view.is-live-preview progress[value='2']::-webkit-progress-value, .markdown-source-view.is-live-preview progress[value='3']::-webkit-progress-value, .markdown-source-view.is-live-preview progress[value='4']::-webkit-progress-value, .markdown-source-view.is-live-preview progress[value='5']::-webkit-progress-value, .markdown-source-view.is-live-preview progress[value='6']::-webkit-progress-value, .markdown-source-view.is-live-preview progress[value='7']::-webkit-progress-value, .markdown-source-view.is-live-preview progress[value='8']::-webkit-progress-value, .markdown-source-view.is-live-preview progress[value='9']::-webkit-progress-value, + .markdown-preview-view progress[value='0']::-webkit-progress-value, + .markdown-preview-view progress[value='2']::-webkit-progress-value, + .markdown-preview-view progress[value='3']::-webkit-progress-value, + .markdown-preview-view progress[value='4']::-webkit-progress-value, + .markdown-preview-view progress[value='5']::-webkit-progress-value, + .markdown-preview-view progress[value='6']::-webkit-progress-value, + .markdown-preview-view progress[value='7']::-webkit-progress-value, + .markdown-preview-view progress[value='8']::-webkit-progress-value, + .markdown-preview-view progress[value='9']::-webkit-progress-value { + background-color: var(--red); } + +/* Range slider input */ +input[type=range] { + background-color: var(--background-modifier-border-hover); + height: 2px; + padding: 0 0px; + -webkit-appearance: none; + cursor: default; + margin: 0; + border-radius: 0px; } + +body:not(.is-mobile) input[type=range]:focus { + box-shadow: none; } + +input[type=range]::-webkit-slider-runnable-track { + background: var(--background-modifier-border-hover); + height: 2px; + margin-top: 0px; } + +input[type=range]::-webkit-slider-thumb { + background: white; + border: 1px solid var(--background-modifier-border-hover); + height: 18px; + width: 18px; + border-radius: 16px; + margin-top: -5px; + transition: all 0.1s linear; + cursor: default; + box-shadow: 0 1px 1px 0px rgba(0, 0, 0, 0.05), 0 2px 4px 0px rgba(0, 0, 0, 0.1); } + +input[type=range]::-webkit-slider-thumb:hover, +input[type=range]::-webkit-slider-thumb:active { + background: white; + border-width: 1; + border: 1px solid var(--background-modifier-border-focus); + box-shadow: 0 1px 2px 0px rgba(0, 0, 0, 0.05), 0 2px 3px 0px rgba(0, 0, 0, 0.2); + transition: all 0.1s linear; } + +body:not(.is-mobile) input[type=range]:focus::-webkit-slider-thumb { + box-shadow: 0 1px 2px 0px rgba(0, 0, 0, 0.05), 0 2px 3px 0px rgba(0, 0, 0, 0.2); } + +/* Toggle switches */ +.checkbox-container { + background-color: var(--background-modifier-border-hover); + box-shadow: inset 0 0px 1px 0px rgba(0, 0, 0, 0.2); + border: none; + width: 40px; + height: 22px; + cursor: var(--cursor); } + .checkbox-container.is-enabled { + border-color: var(--interactive-accent); } + .checkbox-container.is-enabled:after { + transform: translate3d(20px, 0, 0); } + .checkbox-container:after { + background: white; + border: none; + margin: 2px 0 0 0; + height: 18px; + width: 18px; + border-radius: 26px; + transform: translate3d(2px, 0, 0); + box-shadow: 0 1px 2px 0px rgba(0, 0, 0, 0.1); + transition: all 0.1s linear; } + .checkbox-container:hover:after { + box-shadow: 0 2px 3px 0px rgba(0, 0, 0, 0.1); + transition: all 0.1s linear; } + +/* Minimal features */ +/* Active line highlight */ +.active-line-on .cm-line.cm-active, +.active-line-on .markdown-source-view.mod-cm6.is-live-preview .HyperMD-quote.cm-active { + background-color: var(--active-line-bg); + box-shadow: -25vw 0px var(--active-line-bg), 25vw 0 var(--active-line-bg); } + +.borders-low { + --border-width:0px; + --border-width-alt:1px; } + +.borders-none { + --border-width:0px; + --border-width-alt:0px; } + +/* Title borders */ +body.borders-title .workspace-leaf .workspace-leaf-content:not([data-type='empty']):not([data-type='map']):not([data-type='graph']):not([data-type='localgraph']) .view-header, +body.borders-title .workspace-split.mod-root .workspace-leaf:first-of-type:last-of-type .workspace-leaf-content:not([data-type='map']):not([data-type='graph']):not([data-type='empty']):not([data-type='localgraph']) .view-header { + border-bottom: var(--border-width) solid var(--background-divider); } + +body.borders-title .workspace-ribbon.mod-left.is-collapsed { + border-right: var(--border-width) solid var(--background-divider); } + +body:not(.is-fullscreen).mod-macos.hider-frameless.borders-title .mod-left-split > .workspace-tabs:nth-child(3) .workspace-tab-header-container { + border: none; } + +/* MIT License | Copyright (c) Stephan Ango (@kepano) + +Cards snippet for Obsidian + +author: @kepano +version: 1.1.0 + +Support my work: +https://github.com/sponsors/kepano + +*/ +:root { + --cards-min-width:180px; + --cards-max-width:1fr; + --cards-mobile-width:120px; + --cards-image-height:400px; + --cards-padding:1.2em; + --cards-image-fit:contain; + --cards-background:transparent; + --cards-border-width:1px; } + +@media (max-width: 400pt) { + :root { + --cards-min-width:var(--cards-mobile-width); } } +/* Make the grid and basic cards */ +.cards.table-100 table.dataview tbody, +.table-100 .cards table.dataview tbody { + padding: 0.25rem 0.75rem; } + +.cards .el-pre + .el-lang-dataview .table-view-thead { + padding-top: 8px; } + +.cards table.dataview tbody { + clear: both; + padding: 0.5rem 0; + display: grid; + grid-template-columns: repeat(auto-fit, minmax(var(--cards-min-width), var(--cards-max-width))); + grid-column-gap: 0.75rem; + grid-row-gap: 0.75rem; } + +.cards table.dataview > tbody > tr { + background-color: var(--cards-background); + border: var(--cards-border-width) solid var(--background-modifier-border); + display: flex; + flex-direction: column; + margin: 0; + padding: 0 0 calc(var(--cards-padding)/3) 0; + border-radius: 6px; + overflow: hidden; + transition: box-shadow 0.15s linear; } + +.cards table.dataview > tbody > tr:hover { + border: var(--cards-border-width) solid var(--background-modifier-border-hover); + box-shadow: 0 4px 6px 0px rgba(0, 0, 0, 0.05), 0 1px 3px 1px rgba(0, 0, 0, 0.025); + transition: box-shadow 0.15s linear; } + +/* Styling elements inside cards */ +.markdown-source-view.mod-cm6.cards .dataview.table-view-table > tbody > tr > td, +.trim-cols .cards table.dataview tbody > tr > td { + white-space: normal; } + +.markdown-source-view.mod-cm6.cards .dataview.table-view-table > tbody > tr > td, +.cards table.dataview tbody > tr > td { + border-bottom: none; + padding: 0 !important; + line-height: 1.2; + width: calc(100% - var(--cards-padding)); + margin: 0 auto; + overflow: visible !important; + max-width: 100%; + display: flex; } + +.cards table.dataview tbody > tr > td .el-p { + display: block; + width: 100%; } + +.cards table.dataview tbody > tr > td:first-child { + font-weight: var(--bold-weight); } + +.cards table.dataview tbody > tr > td:first-child a { + padding: 0 0 calc(var(--cards-padding)/3); + display: block; } + +.cards table.dataview tbody > tr > td:not(:first-child) { + font-size: 90%; + color: var(--text-muted); } + +@media (max-width: 400pt) { + .cards table.dataview tbody > tr > td:not(:first-child) { + font-size: 80%; } } +/* Helpers */ +.cards-cover.cards table.dataview tbody > tr > td img { + object-fit: cover; } + +.cards-16-9.cards table.dataview tbody > tr > td img { + aspect-ratio: 16/9; } + +.cards-1-1.cards table.dataview tbody > tr > td img { + aspect-ratio: 1/1; } + +.cards-2-1.cards table.dataview tbody > tr > td img { + aspect-ratio: 2/1; } + +.cards-2-3.cards table.dataview tbody > tr > td img { + aspect-ratio: 2/3; } + +.cards-align-bottom.cards table.dataview tbody > tr > td:last-child { + align-items: flex-end; + flex-grow: 1; } + +.cards-cols-1 table.dataview tbody { + grid-template-columns: repeat(1, minmax(0, 1fr)); } + +.cards-cols-2 table.dataview tbody { + grid-template-columns: repeat(2, minmax(0, 1fr)); } + +@media (min-width: 400pt) { + .cards-cols-3 table.dataview tbody { + grid-template-columns: repeat(3, minmax(0, 1fr)); } + + .cards-cols-4 table.dataview tbody { + grid-template-columns: repeat(4, minmax(0, 1fr)); } + + .cards-cols-5 table.dataview tbody { + grid-template-columns: repeat(5, minmax(0, 1fr)); } + + .cards-cols-6 table.dataview tbody { + grid-template-columns: repeat(6, minmax(0, 1fr)); } + + .cards-cols-7 table.dataview tbody { + grid-template-columns: repeat(7, minmax(0, 1fr)); } + + .cards-cols-8 table.dataview tbody { + grid-template-columns: repeat(8, minmax(0, 1fr)); } } +/* Card content */ +/* Paragraphs */ +.cards table.dataview tbody > tr > td > *:not(.el-embed-image) { + padding: calc(var(--cards-padding)/3) 0; } + +.cards table.dataview tbody > tr > td:not(:last-child):not(:first-child) > .el-p:not(.el-embed-image) { + border-bottom: 1px solid var(--background-modifier-border); + width: 100%; } + +/* Links */ +.cards table.dataview tbody > tr > td a { + text-decoration: none; } + +.links-int-on .cards table.dataview tbody > tr > td a { + text-decoration: none; } + +/* Buttons */ +.cards table.dataview tbody > tr > td > button { + width: 100%; + margin: calc(var(--cards-padding)/2) 0; } + +.cards table.dataview tbody > tr > td:last-child > button { + margin-bottom: calc(var(--cards-padding)/6); } + +/* Lists */ +.cards table.dataview tbody > tr > td > ul { + width: 100%; + padding: 0.25em 0 !important; + margin: 0 auto !important; } + +.cards table.dataview tbody > tr > td:not(:last-child) > ul { + border-bottom: 1px solid var(--background-modifier-border); } + +/* Images */ +.cards table.dataview tbody > tr > td .el-embed-image { + background-color: var(--background-secondary); + display: block; + margin: 0 calc(var(--cards-padding)/-2) 0 calc(var(--cards-padding)/-2); + width: calc(100% + var(--cards-padding)); } + +.cards table.dataview tbody > tr > td img { + width: 100%; + object-fit: var(--cards-image-fit); + max-height: var(--cards-image-height); + background-color: var(--background-secondary); + vertical-align: bottom; } + +/* ------------------- */ +/* Block button */ +.markdown-source-view.mod-cm6.cards .edit-block-button { + top: 0px; } + +/* ------------------- */ +/* Sorting */ +.cards.table-100 table.dataview thead > tr, +.table-100 .cards table.dataview thead > tr { + right: 0.75rem; } + +.table-100 .cards table.dataview thead:before, +.cards.table-100 table.dataview thead:before { + margin-right: 0.75rem; } + +.cards table.dataview thead { + user-select: none; + width: 180px; + display: block; + float: right; + position: relative; + text-align: right; + height: 24px; + padding-bottom: 4px; } + +.cards table.dataview thead:before { + content: ''; + position: absolute; + right: 0; + top: 0; + height: var(--icon-size); + background-repeat: no-repeat; + cursor: var(--cursor); + text-align: right; + padding: 4px 10px; + margin-bottom: 2px; + border-radius: 5px; + font-weight: 500; + font-size: var(--font-adaptive-small); } + +.cards table.dataview thead:before { + opacity: 0.25; + background-position: center center; + background-size: var(--icon-size); + background-image: url('data:image/svg+xml;utf8,'); } + +.theme-light .cards table.dataview thead:before { + background-image: url('data:image/svg+xml;utf8,'); } + +.cards table.dataview thead:hover:before { + opacity: 0.5; } + +.cards table.dataview thead > tr { + top: 0; + position: absolute; + display: none; + z-index: 9; + border: 1px solid var(--background-modifier-border); + background-color: var(--background-secondary); + box-shadow: 0 2px 8px var(--background-modifier-box-shadow); + padding: 6px; + border-radius: 6px; + flex-direction: column; + margin: 26px 0 0 0; + width: 100%; } + +.cards table.dataview thead:hover > tr { + display: flex; } + +.cards table.dataview thead > tr > th { + display: block; + padding: 3px 30px 3px 6px !important; + border-radius: 5px; + width: 100%; + font-weight: 400; + color: var(--text-muted); + cursor: var(--cursor); + border: none; + font-size: var(--font-adaptive-small); } + +.cards table.dataview thead > tr > th[sortable-style="sortable-asc"], +.cards table.dataview thead > tr > th[sortable-style="sortable-desc"] { + color: var(--text-normal); } + +.cards table.dataview thead > tr > th:hover { + color: var(--text-normal); + background-color: var(--background-tertiary); } + +/* Checklist icons */ +.cm-formatting.cm-formatting-task.cm-property { + font-family: var(--font-monospace); + font-size: 90%; } + +input[data-task=">"]:checked, +input[data-task="!"]:checked, +input[data-task="-"]:checked, +input[data-task="<"]:checked, +input[data-task="l"]:checked, +input[data-task="*"]:checked, +input[data-task="I"]:checked, +input[data-task="p"]:checked, +input[data-task="f"]:checked, +input[data-task="k"]:checked, +input[data-task="u"]:checked, +input[data-task="w"]:checked, +input[data-task="c"]:checked, +input[data-task="d"]:checked, +input[data-task="b"]:checked, +li[data-task=">"] > input:checked, +li[data-task="!"] > input:checked, +li[data-task="-"] > input:checked, +li[data-task="<"] > input:checked, +li[data-task="l"] > input:checked, +li[data-task="*"] > input:checked, +li[data-task="I"] > input:checked, +li[data-task="p"] > input:checked, +li[data-task="f"] > input:checked, +li[data-task="k"] > input:checked, +li[data-task="u"] > input:checked, +li[data-task="d"] > input:checked, +li[data-task="w"] > input:checked, +li[data-task="c"] > input:checked, +li[data-task="b"] > input:checked, +li[data-task=">"] > p > input:checked, +li[data-task="!"] > p > input:checked, +li[data-task="-"] > p > input:checked, +li[data-task="<"] > p > input:checked, +li[data-task="l"] > p > input:checked, +li[data-task="*"] > p > input:checked, +li[data-task="I"] > p > input:checked, +li[data-task="p"] > p > input:checked, +li[data-task="f"] > p > input:checked, +li[data-task="k"] > p > input:checked, +li[data-task="u"] > p > input:checked, +li[data-task="d"] > p > input:checked, +li[data-task="w"] > p > input:checked, +li[data-task="c"] > p > input:checked, +li[data-task="b"] > p > input:checked { + border: none; + border-radius: 0; + background-image: none; + background-color: currentColor; + -webkit-mask-size: var(--checkbox-icon); + -webkit-mask-position: 50% 50%; } + +/* [>] Forwarded */ +input[data-task=">"]:checked, +li[data-task=">"] > input:checked, +li[data-task=">"] > p > input:checked { + color: var(--text-faint); + transform: rotate(90deg); + -webkit-mask-position: 50% 100%; + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-5 w-5' viewBox='0 0 20 20' fill='currentColor'%3E%3Cpath d='M10.894 2.553a1 1 0 00-1.788 0l-7 14a1 1 0 001.169 1.409l5-1.429A1 1 0 009 15.571V11a1 1 0 112 0v4.571a1 1 0 00.725.962l5 1.428a1 1 0 001.17-1.408l-7-14z' /%3E%3C/svg%3E"); } + +/* [<] Schedule */ +input[data-task="<"]:checked, +li[data-task="<"] > input:checked, +li[data-task="<"] > p > input:checked { + color: var(--text-faint); + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-5 w-5' viewBox='0 0 20 20' fill='currentColor'%3E%3Cpath fill-rule='evenodd' d='M10 18a8 8 0 100-16 8 8 0 000 16zm1-12a1 1 0 10-2 0v4a1 1 0 00.293.707l2.828 2.829a1 1 0 101.415-1.415L11 9.586V6z' clip-rule='evenodd' /%3E%3C/svg%3E"); + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-5 w-5' viewBox='0 0 20 20' fill='currentColor'%3E%3Cpath fill-rule='evenodd' d='M6 2a1 1 0 00-1 1v1H4a2 2 0 00-2 2v10a2 2 0 002 2h12a2 2 0 002-2V6a2 2 0 00-2-2h-1V3a1 1 0 10-2 0v1H7V3a1 1 0 00-1-1zm0 5a1 1 0 000 2h8a1 1 0 100-2H6z' clip-rule='evenodd' /%3E%3C/svg%3E"); } + +/* [?] Question */ +input[data-task="?"]:checked, +li[data-task="?"] > input:checked, +li[data-task="?"] > p > input:checked { + background-color: var(--yellow); + border-color: var(--yellow); + background-position: 50% 50%; + background-size: 200% 90%; + background-image: url('data:image/svg+xml,%3Csvg xmlns="http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg" width="20" height="20" preserveAspectRatio="xMidYMid meet" viewBox="0 0 16 16"%3E%3Cpath fill="white" fill-rule="evenodd" d="M4.475 5.458c-.284 0-.514-.237-.47-.517C4.28 3.24 5.576 2 7.825 2c2.25 0 3.767 1.36 3.767 3.215c0 1.344-.665 2.288-1.79 2.973c-1.1.659-1.414 1.118-1.414 2.01v.03a.5.5 0 0 1-.5.5h-.77a.5.5 0 0 1-.5-.495l-.003-.2c-.043-1.221.477-2.001 1.645-2.712c1.03-.632 1.397-1.135 1.397-2.028c0-.979-.758-1.698-1.926-1.698c-1.009 0-1.71.529-1.938 1.402c-.066.254-.278.461-.54.461h-.777ZM7.496 14c.622 0 1.095-.474 1.095-1.09c0-.618-.473-1.092-1.095-1.092c-.606 0-1.087.474-1.087 1.091S6.89 14 7.496 14Z"%2F%3E%3C%2Fsvg%3E'); } +.theme-dark input[data-task="?"]:checked, +.theme-dark li[data-task="?"] > input:checked, +.theme-dark li[data-task="?"] > p > input:checked { + background-image: url('data:image/svg+xml,%3Csvg xmlns="http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg" width="20" height="20" preserveAspectRatio="xMidYMid meet" viewBox="0 0 16 16"%3E%3Cpath fill="black" fill-opacity="0.8" fill-rule="evenodd" d="M4.475 5.458c-.284 0-.514-.237-.47-.517C4.28 3.24 5.576 2 7.825 2c2.25 0 3.767 1.36 3.767 3.215c0 1.344-.665 2.288-1.79 2.973c-1.1.659-1.414 1.118-1.414 2.01v.03a.5.5 0 0 1-.5.5h-.77a.5.5 0 0 1-.5-.495l-.003-.2c-.043-1.221.477-2.001 1.645-2.712c1.03-.632 1.397-1.135 1.397-2.028c0-.979-.758-1.698-1.926-1.698c-1.009 0-1.71.529-1.938 1.402c-.066.254-.278.461-.54.461h-.777ZM7.496 14c.622 0 1.095-.474 1.095-1.09c0-.618-.473-1.092-1.095-1.092c-.606 0-1.087.474-1.087 1.091S6.89 14 7.496 14Z"%2F%3E%3C%2Fsvg%3E'); } + +/* [/] Incomplete */ +input[data-task="/"]:checked, +li[data-task="/"] > input:checked, +li[data-task="/"] > p > input:checked { + background-image: none; + background-color: transparent; + position: relative; + overflow: hidden; } + input[data-task="/"]:checked:after, + li[data-task="/"] > input:checked:after, + li[data-task="/"] > p > input:checked:after { + content: " "; + display: block; + position: absolute; + background-color: var(--background-modifier-accent); + width: calc(50% - 0.5px); + height: 100%; } + +/* [!] Important */ +input[data-task="!"]:checked, +li[data-task="!"] > input:checked, +li[data-task="!"] > p > input:checked { + color: var(--orange); + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-5 w-5' viewBox='0 0 20 20' fill='currentColor'%3E%3Cpath fill-rule='evenodd' d='M8.257 3.099c.765-1.36 2.722-1.36 3.486 0l5.58 9.92c.75 1.334-.213 2.98-1.742 2.98H4.42c-1.53 0-2.493-1.646-1.743-2.98l5.58-9.92zM11 13a1 1 0 11-2 0 1 1 0 012 0zm-1-8a1 1 0 00-1 1v3a1 1 0 002 0V6a1 1 0 00-1-1z' clip-rule='evenodd' /%3E%3C/svg%3E"); } + +/* ["] Quote */ +input[data-task="“"]:checked, +li[data-task="“"] > input:checked, +li[data-task="“"] > p > input:checked, +input[data-task="\""]:checked, +li[data-task="\""] > input:checked, +li[data-task="\""] > p > input:checked { + background-position: 50% 50%; + background-color: var(--cyan); + border-color: var(--cyan); + background-size: 75%; + background-image: url('data:image/svg+xml,%3Csvg xmlns="http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg" width="20" height="20" preserveAspectRatio="xMidYMid meet" viewBox="0 0 24 24"%3E%3Cpath fill="white" d="M6.5 10c-.223 0-.437.034-.65.065c.069-.232.14-.468.254-.68c.114-.308.292-.575.469-.844c.148-.291.409-.488.601-.737c.201-.242.475-.403.692-.604c.213-.21.492-.315.714-.463c.232-.133.434-.28.65-.35l.539-.222l.474-.197l-.485-1.938l-.597.144c-.191.048-.424.104-.689.171c-.271.05-.56.187-.882.312c-.318.142-.686.238-1.028.466c-.344.218-.741.4-1.091.692c-.339.301-.748.562-1.05.945c-.33.358-.656.734-.909 1.162c-.293.408-.492.856-.702 1.299c-.19.443-.343.896-.468 1.336c-.237.882-.343 1.72-.384 2.437c-.034.718-.014 1.315.028 1.747c.015.204.043.402.063.539l.025.168l.026-.006A4.5 4.5 0 1 0 6.5 10zm11 0c-.223 0-.437.034-.65.065c.069-.232.14-.468.254-.68c.114-.308.292-.575.469-.844c.148-.291.409-.488.601-.737c.201-.242.475-.403.692-.604c.213-.21.492-.315.714-.463c.232-.133.434-.28.65-.35l.539-.222l.474-.197l-.485-1.938l-.597.144c-.191.048-.424.104-.689.171c-.271.05-.56.187-.882.312c-.317.143-.686.238-1.028.467c-.344.218-.741.4-1.091.692c-.339.301-.748.562-1.05.944c-.33.358-.656.734-.909 1.162c-.293.408-.492.856-.702 1.299c-.19.443-.343.896-.468 1.336c-.237.882-.343 1.72-.384 2.437c-.034.718-.014 1.315.028 1.747c.015.204.043.402.063.539l.025.168l.026-.006A4.5 4.5 0 1 0 17.5 10z"%2F%3E%3C%2Fsvg%3E'); } +.theme-dark input[data-task="“"]:checked, +.theme-dark li[data-task="“"] > input:checked, +.theme-dark li[data-task="“"] > p > input:checked, +.theme-dark input[data-task="\""]:checked, +.theme-dark li[data-task="\""] > input:checked, +.theme-dark li[data-task="\""] > p > input:checked { + background-image: url('data:image/svg+xml,%3Csvg xmlns="http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg" width="20" height="20" preserveAspectRatio="xMidYMid meet" viewBox="0 0 24 24"%3E%3Cpath fill="black" fill-opacity="0.7" d="M6.5 10c-.223 0-.437.034-.65.065c.069-.232.14-.468.254-.68c.114-.308.292-.575.469-.844c.148-.291.409-.488.601-.737c.201-.242.475-.403.692-.604c.213-.21.492-.315.714-.463c.232-.133.434-.28.65-.35l.539-.222l.474-.197l-.485-1.938l-.597.144c-.191.048-.424.104-.689.171c-.271.05-.56.187-.882.312c-.318.142-.686.238-1.028.466c-.344.218-.741.4-1.091.692c-.339.301-.748.562-1.05.945c-.33.358-.656.734-.909 1.162c-.293.408-.492.856-.702 1.299c-.19.443-.343.896-.468 1.336c-.237.882-.343 1.72-.384 2.437c-.034.718-.014 1.315.028 1.747c.015.204.043.402.063.539l.025.168l.026-.006A4.5 4.5 0 1 0 6.5 10zm11 0c-.223 0-.437.034-.65.065c.069-.232.14-.468.254-.68c.114-.308.292-.575.469-.844c.148-.291.409-.488.601-.737c.201-.242.475-.403.692-.604c.213-.21.492-.315.714-.463c.232-.133.434-.28.65-.35l.539-.222l.474-.197l-.485-1.938l-.597.144c-.191.048-.424.104-.689.171c-.271.05-.56.187-.882.312c-.317.143-.686.238-1.028.467c-.344.218-.741.4-1.091.692c-.339.301-.748.562-1.05.944c-.33.358-.656.734-.909 1.162c-.293.408-.492.856-.702 1.299c-.19.443-.343.896-.468 1.336c-.237.882-.343 1.72-.384 2.437c-.034.718-.014 1.315.028 1.747c.015.204.043.402.063.539l.025.168l.026-.006A4.5 4.5 0 1 0 17.5 10z"%2F%3E%3C%2Fsvg%3E'); } + +/* [-] Canceled */ +input[data-task="-"]:checked, +li[data-task="-"] > input:checked, +li[data-task="-"] > p > input:checked { + color: var(--text-faint); + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-5 w-5' viewBox='0 0 20 20' fill='currentColor'%3E%3Cpath fill-rule='evenodd' d='M3 10a1 1 0 011-1h12a1 1 0 110 2H4a1 1 0 01-1-1z' clip-rule='evenodd' /%3E%3C/svg%3E"); } + +body:not(.tasks) .markdown-source-view.mod-cm6 .HyperMD-task-line[data-task]:is([data-task="-"]), +body:not(.tasks) .markdown-preview-view ul li[data-task="-"].task-list-item.is-checked, +body:not(.tasks) li[data-task="-"].task-list-item.is-checked { + color: var(--text-faint); + text-decoration: line-through solid var(--text-faint) 1px; } + +/* [*] Star */ +input[data-task="*"]:checked, +li[data-task="*"] > input:checked, +li[data-task="*"] > p > input:checked { + color: var(--yellow); + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-5 w-5' viewBox='0 0 20 20' fill='currentColor'%3E%3Cpath d='M9.049 2.927c.3-.921 1.603-.921 1.902 0l1.07 3.292a1 1 0 00.95.69h3.462c.969 0 1.371 1.24.588 1.81l-2.8 2.034a1 1 0 00-.364 1.118l1.07 3.292c.3.921-.755 1.688-1.54 1.118l-2.8-2.034a1 1 0 00-1.175 0l-2.8 2.034c-.784.57-1.838-.197-1.539-1.118l1.07-3.292a1 1 0 00-.364-1.118L2.98 8.72c-.783-.57-.38-1.81.588-1.81h3.461a1 1 0 00.951-.69l1.07-3.292z' /%3E%3C/svg%3E"); } + +/* [l] Location */ +input[data-task="l"]:checked, +li[data-task="l"] > input:checked, +li[data-task="l"] > p > input:checked { + color: var(--red); + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-5 w-5' viewBox='0 0 20 20' fill='currentColor'%3E%3Cpath fill-rule='evenodd' d='M5.05 4.05a7 7 0 119.9 9.9L10 18.9l-4.95-4.95a7 7 0 010-9.9zM10 11a2 2 0 100-4 2 2 0 000 4z' clip-rule='evenodd' /%3E%3C/svg%3E"); } + +/* [i] Info */ +input[data-task="i"]:checked, +li[data-task="i"] > input:checked, +li[data-task="i"] > p > input:checked { + background-color: var(--blue); + border-color: var(--blue); + background-position: 50%; + background-size: 100%; + background-image: url('data:image/svg+xml,%3Csvg xmlns="http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg" width="20" height="20" preserveAspectRatio="xMidYMid meet" viewBox="0 0 512 512"%3E%3Cpath fill="none" stroke="white" stroke-linecap="round" stroke-linejoin="round" stroke-width="40" d="M196 220h64v172"%2F%3E%3Cpath fill="none" stroke="white" stroke-linecap="round" stroke-miterlimit="10" stroke-width="40" d="M187 396h138"%2F%3E%3Cpath fill="white" d="M256 160a32 32 0 1 1 32-32a32 32 0 0 1-32 32Z"%2F%3E%3C%2Fsvg%3E'); } +.theme-dark input[data-task="i"]:checked, +.theme-dark li[data-task="i"] > input:checked, +.theme-dark li[data-task="i"] > p > input:checked { + background-image: url('data:image/svg+xml,%3Csvg xmlns="http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg" width="20" height="20" preserveAspectRatio="xMidYMid meet" viewBox="0 0 512 512"%3E%3Cpath fill="none" stroke="black" stroke-opacity="0.8" stroke-linecap="round" stroke-linejoin="round" stroke-width="40" d="M196 220h64v172"%2F%3E%3Cpath fill="none" stroke="black" stroke-opacity="0.8" stroke-linecap="round" stroke-miterlimit="10" stroke-width="40" d="M187 396h138"%2F%3E%3Cpath fill="black" fill-opacity="0.8" d="M256 160a32 32 0 1 1 32-32a32 32 0 0 1-32 32Z"%2F%3E%3C%2Fsvg%3E'); } + +/* [S] Amount/savings/money */ +input[data-task="S"]:checked, +li[data-task="S"] > input:checked, +li[data-task="S"] > p > input:checked { + border-color: var(--green); + background-color: var(--green); + background-size: 100%; + background-image: url('data:image/svg+xml,%3Csvg xmlns="http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg" width="20" height="20" preserveAspectRatio="xMidYMid meet" viewBox="0 0 48 48"%3E%3Cpath fill="white" fill-rule="evenodd" d="M26 8a2 2 0 1 0-4 0v2a8 8 0 1 0 0 16v8a4.002 4.002 0 0 1-3.773-2.666a2 2 0 0 0-3.771 1.332A8.003 8.003 0 0 0 22 38v2a2 2 0 1 0 4 0v-2a8 8 0 1 0 0-16v-8a4.002 4.002 0 0 1 3.773 2.666a2 2 0 0 0 3.771-1.332A8.003 8.003 0 0 0 26 10V8Zm-4 6a4 4 0 0 0 0 8v-8Zm4 12v8a4 4 0 0 0 0-8Z" clip-rule="evenodd"%2F%3E%3C%2Fsvg%3E'); } +.theme-dark input[data-task="S"]:checked, +.theme-dark li[data-task="S"] > input:checked, +.theme-dark li[data-task="S"] > p > input:checked { + background-image: url('data:image/svg+xml,%3Csvg xmlns="http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg" width="20" height="20" preserveAspectRatio="xMidYMid meet" viewBox="0 0 48 48"%3E%3Cpath fill-opacity="0.8" fill="black" fill-rule="evenodd" d="M26 8a2 2 0 1 0-4 0v2a8 8 0 1 0 0 16v8a4.002 4.002 0 0 1-3.773-2.666a2 2 0 0 0-3.771 1.332A8.003 8.003 0 0 0 22 38v2a2 2 0 1 0 4 0v-2a8 8 0 1 0 0-16v-8a4.002 4.002 0 0 1 3.773 2.666a2 2 0 0 0 3.771-1.332A8.003 8.003 0 0 0 26 10V8Zm-4 6a4 4 0 0 0 0 8v-8Zm4 12v8a4 4 0 0 0 0-8Z" clip-rule="evenodd"%2F%3E%3C%2Fsvg%3E'); } + +/* [I] Idea/lightbulb */ +input[data-task="I"]:checked, +li[data-task="I"] > input:checked, +li[data-task="I"] > p > input:checked { + color: var(--yellow); + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-5 w-5' viewBox='0 0 20 20' fill='currentColor'%3E%3Cpath d='M11 3a1 1 0 10-2 0v1a1 1 0 102 0V3zM15.657 5.757a1 1 0 00-1.414-1.414l-.707.707a1 1 0 001.414 1.414l.707-.707zM18 10a1 1 0 01-1 1h-1a1 1 0 110-2h1a1 1 0 011 1zM5.05 6.464A1 1 0 106.464 5.05l-.707-.707a1 1 0 00-1.414 1.414l.707.707zM5 10a1 1 0 01-1 1H3a1 1 0 110-2h1a1 1 0 011 1zM8 16v-1h4v1a2 2 0 11-4 0zM12 14c.015-.34.208-.646.477-.859a4 4 0 10-4.954 0c.27.213.462.519.476.859h4.002z' /%3E%3C/svg%3E"); } + +/* [f] Fire */ +input[data-task="f"]:checked, +li[data-task="f"] > input:checked, +li[data-task="f"] > p > input:checked { + color: var(--red); + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-5 w-5' viewBox='0 0 20 20' fill='currentColor'%3E%3Cpath fill-rule='evenodd' d='M12.395 2.553a1 1 0 00-1.45-.385c-.345.23-.614.558-.822.88-.214.33-.403.713-.57 1.116-.334.804-.614 1.768-.84 2.734a31.365 31.365 0 00-.613 3.58 2.64 2.64 0 01-.945-1.067c-.328-.68-.398-1.534-.398-2.654A1 1 0 005.05 6.05 6.981 6.981 0 003 11a7 7 0 1011.95-4.95c-.592-.591-.98-.985-1.348-1.467-.363-.476-.724-1.063-1.207-2.03zM12.12 15.12A3 3 0 017 13s.879.5 2.5.5c0-1 .5-4 1.25-4.5.5 1 .786 1.293 1.371 1.879A2.99 2.99 0 0113 13a2.99 2.99 0 01-.879 2.121z' clip-rule='evenodd' /%3E%3C/svg%3E"); } + +/* [k] Key */ +input[data-task="k"]:checked, +li[data-task="k"] > input:checked, +li[data-task="k"] > p > input:checked { + color: var(--yellow); + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-5 w-5' viewBox='0 0 20 20' fill='currentColor'%3E%3Cpath fill-rule='evenodd' d='M18 8a6 6 0 01-7.743 5.743L10 14l-1 1-1 1H6v2H2v-4l4.257-4.257A6 6 0 1118 8zm-6-4a1 1 0 100 2 2 2 0 012 2 1 1 0 102 0 4 4 0 00-4-4z' clip-rule='evenodd' /%3E%3C/svg%3E"); } + +/* [u] Up */ +input[data-task="u"]:checked, +li[data-task="u"] > input:checked, +li[data-task="u"] > p > input:checked { + color: var(--green); + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-5 w-5' viewBox='0 0 20 20' fill='currentColor'%3E%3Cpath fill-rule='evenodd' d='M12 7a1 1 0 110-2h5a1 1 0 011 1v5a1 1 0 11-2 0V8.414l-4.293 4.293a1 1 0 01-1.414 0L8 10.414l-4.293 4.293a1 1 0 01-1.414-1.414l5-5a1 1 0 011.414 0L11 10.586 14.586 7H12z' clip-rule='evenodd' /%3E%3C/svg%3E"); } + +/* [d] Down */ +input[data-task="d"]:checked, +li[data-task="d"] > input:checked, +li[data-task="d"] > p > input:checked { + color: var(--red); + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-5 w-5' viewBox='0 0 20 20' fill='currentColor'%3E%3Cpath fill-rule='evenodd' d='M12 13a1 1 0 100 2h5a1 1 0 001-1V9a1 1 0 10-2 0v2.586l-4.293-4.293a1 1 0 00-1.414 0L8 9.586 3.707 5.293a1 1 0 00-1.414 1.414l5 5a1 1 0 001.414 0L11 9.414 14.586 13H12z' clip-rule='evenodd' /%3E%3C/svg%3E"); } + +/* [w] Win */ +input[data-task="w"]:checked, +li[data-task="w"] > input:checked, +li[data-task="w"] > p > input:checked { + color: var(--purple); + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-5 w-5' viewBox='0 0 20 20' fill='currentColor'%3E%3Cpath fill-rule='evenodd' d='M6 3a1 1 0 011-1h.01a1 1 0 010 2H7a1 1 0 01-1-1zm2 3a1 1 0 00-2 0v1a2 2 0 00-2 2v1a2 2 0 00-2 2v.683a3.7 3.7 0 011.055.485 1.704 1.704 0 001.89 0 3.704 3.704 0 014.11 0 1.704 1.704 0 001.89 0 3.704 3.704 0 014.11 0 1.704 1.704 0 001.89 0A3.7 3.7 0 0118 12.683V12a2 2 0 00-2-2V9a2 2 0 00-2-2V6a1 1 0 10-2 0v1h-1V6a1 1 0 10-2 0v1H8V6zm10 8.868a3.704 3.704 0 01-4.055-.036 1.704 1.704 0 00-1.89 0 3.704 3.704 0 01-4.11 0 1.704 1.704 0 00-1.89 0A3.704 3.704 0 012 14.868V17a1 1 0 001 1h14a1 1 0 001-1v-2.132zM9 3a1 1 0 011-1h.01a1 1 0 110 2H10a1 1 0 01-1-1zm3 0a1 1 0 011-1h.01a1 1 0 110 2H13a1 1 0 01-1-1z' clip-rule='evenodd' /%3E%3C/svg%3E"); } + +/* [p] Pros */ +input[data-task="p"]:checked, +li[data-task="p"] > input:checked, +li[data-task="p"] > p > input:checked { + color: var(--green); + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-5 w-5' viewBox='0 0 20 20' fill='currentColor'%3E%3Cpath d='M2 10.5a1.5 1.5 0 113 0v6a1.5 1.5 0 01-3 0v-6zM6 10.333v5.43a2 2 0 001.106 1.79l.05.025A4 4 0 008.943 18h5.416a2 2 0 001.962-1.608l1.2-6A2 2 0 0015.56 8H12V4a2 2 0 00-2-2 1 1 0 00-1 1v.667a4 4 0 01-.8 2.4L6.8 7.933a4 4 0 00-.8 2.4z' /%3E%3C/svg%3E"); } + +/* [c] Cons */ +input[data-task="c"]:checked, +li[data-task="c"] > input:checked, +li[data-task="c"] > p > input:checked { + color: var(--orange); + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-5 w-5' viewBox='0 0 20 20' fill='currentColor'%3E%3Cpath d='M18 9.5a1.5 1.5 0 11-3 0v-6a1.5 1.5 0 013 0v6zM14 9.667v-5.43a2 2 0 00-1.105-1.79l-.05-.025A4 4 0 0011.055 2H5.64a2 2 0 00-1.962 1.608l-1.2 6A2 2 0 004.44 12H8v4a2 2 0 002 2 1 1 0 001-1v-.667a4 4 0 01.8-2.4l1.4-1.866a4 4 0 00.8-2.4z' /%3E%3C/svg%3E"); } + +/* [b] Bookmark */ +input[data-task="b"]:checked, +li[data-task="b"] > input:checked, +li[data-task="b"] > p > input:checked { + color: var(--orange); + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-5 w-5' viewBox='0 0 20 20' fill='currentColor'%3E%3Cpath d='M5 4a2 2 0 012-2h6a2 2 0 012 2v14l-5-2.5L5 18V4z' /%3E%3C/svg%3E"); } + +/* Colorful active states */ +.colorful-active { + --sp1:var(--bg1); } + +.colorful-active .nav-file-title.is-active, +.colorful-active #calendar-container .active, +.colorful-active #calendar-container .active.today, +.colorful-active #calendar-container .active:hover, +.colorful-active #calendar-container .day:active, +.colorful-active .vertical-tab-nav-item.is-active, +.colorful-active .nav-file-title.is-being-dragged, +.colorful-active .nav-folder-title.is-being-dragged, +body.colorful-active:not(.is-grabbing) .nav-file-title.is-being-dragged:hover, +body.colorful-active:not(.is-grabbing) .nav-folder-title.is-being-dragged:hover, +body.colorful-active:not(.is-grabbing) .nav-file-title.is-active:hover, +.colorful-active .menu-item.selected:not(.is-disabled):not(.is-label), +.colorful-active .menu-item:hover, +.colorful-active .menu-item:hover:not(.is-disabled):not(.is-label) { + background-color: var(--ax3); + color: var(--sp1); } + +.colorful-active #calendar-container .day:active .dot, +.colorful-active #calendar-container .active .dot, +.colorful-active #calendar-container .today.active .dot { + fill: var(--sp1); } + +.colorful-active .menu-item.selected:not(.is-disabled):not(.is-label) .menu-item-icon, +.colorful-active .menu-item:hover .menu-item-icon { + color: var(--sp1); } + +.colorful-active .nav-file-title-content.is-being-renamed { + color: var(--text-normal); } + +.is-mobile.colorful-active .nav-file-title.is-active { + box-shadow: 0 0 0px 2px var(--ax3); } + +/* +.colorful-active .suggestion-container .suggestion-item:hover, +.colorful-active .modal-container .suggestion-item.is-selected { +}*/ +/* Colorful headings */ +body.colorful-headings { + --h1-color:var(--red); + --h2-color:var(--orange); + --h3-color:var(--yellow); + --h4-color:var(--green); + --h5-color:var(--blue); + --h6-color:var(--purple); } + +/* Icons + +Thank you to Matthew Meyers and Chetachi Ezikeuzor */ +.is-mobile .tree-item-self .collapse-icon { + width: 20px; } + +body:not(.minimal-icons-off) .view-action svg, +body:not(.minimal-icons-off) .workspace-tab-header-inner-icon svg, +body:not(.minimal-icons-off) .nav-action-button svg, +body:not(.minimal-icons-off) .graph-controls-button svg { + width: var(--icon-size); + height: var(--icon-size); } + +body:not(.minimal-icons-off) .menu-item-icon svg { + width: 16px; + height: 16px; } + +body:not(.minimal-icons-off) .workspace-ribbon-collapse-btn svg { + width: var(--icon-size); + height: var(--icon-size); } + +body:not(.minimal-icons-off) svg.any-key, +body:not(.minimal-icons-off) svg.blocks, +body:not(.minimal-icons-off) svg.bar-graph, +body:not(.minimal-icons-off) svg.breadcrumbs-trail-icon, +body:not(.minimal-icons-off) svg.audio-file, +body:not(.minimal-icons-off) svg.bold-glyph, +body:not(.minimal-icons-off) svg.italic-glyph, +body:not(.minimal-icons-off) svg.bracket-glyph, +body:not(.minimal-icons-off) svg.broken-link, +body:not(.minimal-icons-off) svg.bullet-list-glyph, +body:not(.minimal-icons-off) svg.bullet-list, +body:not(.minimal-icons-off) svg.calendar-day, +body:not(.minimal-icons-off) svg.calendar-with-checkmark, +body:not(.minimal-icons-off) svg.check-in-circle, +body:not(.minimal-icons-off) svg.check-small, +body:not(.minimal-icons-off) svg.checkbox-glyph, +body:not(.minimal-icons-off) svg.checkmark, +body:not(.minimal-icons-off) svg.clock, +body:not(.minimal-icons-off) svg.cloud, +body:not(.minimal-icons-off) svg.code-glyph, +body:not(.minimal-icons-off) svg.create-new, +body:not(.minimal-icons-off) svg.cross-in-box, +body:not(.minimal-icons-off) svg.cross, +body:not(.minimal-icons-off) svg.crossed-star, +body:not(.minimal-icons-off) svg.dice, +body:not(.minimal-icons-off) svg.disk, +body:not(.minimal-icons-off) svg.document, +body:not(.minimal-icons-off) svg.documents, +body:not(.minimal-icons-off) svg.dot-network, +body:not(.minimal-icons-off) svg.double-down-arrow-glyph, +body:not(.minimal-icons-off) svg.double-up-arrow-glyph, +body:not(.minimal-icons-off) svg.down-arrow-with-tail, +body:not(.minimal-icons-off) svg.down-chevron-glyph, +body:not(.minimal-icons-off) svg.enter, +body:not(.minimal-icons-off) svg.exit-fullscreen, +body:not(.minimal-icons-off) svg.expand-vertically, +body:not(.minimal-icons-off) svg.excalidraw-icon, +body:not(.minimal-icons-off) svg.filled-pin, +body:not(.minimal-icons-off) svg.folder, +body:not(.minimal-icons-off) svg.fullscreen, +body:not(.minimal-icons-off) svg.gear, +body:not(.minimal-icons-off) svg.globe, +body:not(.minimal-icons-off) svg.hashtag, +body:not(.minimal-icons-off) svg.heading-glyph, +body:not(.minimal-icons-off) svg.go-to-file, +body:not(.minimal-icons-off) svg.help .widget-icon, +body:not(.minimal-icons-off) svg.help, +body:not(.minimal-icons-off) svg.highlight-glyph, +body:not(.minimal-icons-off) svg.horizontal-split, +body:not(.minimal-icons-off) svg.image-file, +body:not(.minimal-icons-off) svg.image-glyph, +body:not(.minimal-icons-off) svg.indent-glyph, +body:not(.minimal-icons-off) svg.info, +body:not(.minimal-icons-off) svg.install, +body:not(.minimal-icons-off) svg.keyboard-glyph, +body:not(.minimal-icons-off) svg.ledger, +body:not(.minimal-icons-off) svg.left-arrow-with-tail, +body:not(.minimal-icons-off) svg.left-arrow, +body:not(.minimal-icons-off) svg.left-chevron-glyph, +body:not(.minimal-icons-off) svg.lines-of-text, +body:not(.minimal-icons-off) svg.link-glyph, +body:not(.minimal-icons-off) svg.link, +body:not(.minimal-icons-off) svg.magnifying-glass, +body:not(.minimal-icons-off) svg.microphone-filled, +body:not(.minimal-icons-off) svg.microphone, +body:not(.minimal-icons-off) svg.minus-with-circle, +body:not(.minimal-icons-off) svg.note-glyph, +body:not(.minimal-icons-off) svg.number-list-glyph, +body:not(.minimal-icons-off) svg.open-vault, +body:not(.minimal-icons-off) svg.pane-layout, +body:not(.minimal-icons-off) svg.paper-plane, +body:not(.minimal-icons-off) svg.paused, +body:not(.minimal-icons-off) svg.pencil, +body:not(.minimal-icons-off) svg.pencil_icon, +body:not(.minimal-icons-off) svg.pin, +body:not(.minimal-icons-off) svg.plus-with-circle, +body:not(.minimal-icons-off) svg.popup-open, +body:not(.minimal-icons-off) svg.presentation, +body:not(.minimal-icons-off) svg.price-tag-glyph, +body:not(.minimal-icons-off) svg.quote-glyph, +body:not(.minimal-icons-off) svg.redo-glyph, +body:not(.minimal-icons-off) svg.reset, +body:not(.minimal-icons-off) svg.right-arrow-with-tail, +body:not(.minimal-icons-off) svg.right-arrow, +body:not(.minimal-icons-off) svg.right-chevron-glyph, +body:not(.minimal-icons-off) svg.right-triangle, +body:not(.minimal-icons-off) svg.run-command, +body:not(.minimal-icons-off) svg.search, +body:not(.minimal-icons-off) svg.ScriptEngine, +body:not(.minimal-icons-off) svg.sheets-in-box, +body:not(.minimal-icons-off) svg.spreadsheet, +body:not(.minimal-icons-off) svg.stacked-levels, +body:not(.minimal-icons-off) svg.star-list, +body:not(.minimal-icons-off) svg.star, +body:not(.minimal-icons-off) svg.strikethrough-glyph, +body:not(.minimal-icons-off) svg.switch, +body:not(.minimal-icons-off) svg.sync-small, +body:not(.minimal-icons-off) svg.sync, +body:not(.minimal-icons-off) svg.tag-glyph, +body:not(.minimal-icons-off) svg.three-horizontal-bars, +body:not(.minimal-icons-off) svg.trash, +body:not(.minimal-icons-off) svg.undo-glyph, +body:not(.minimal-icons-off) svg.unindent-glyph, +body:not(.minimal-icons-off) svg.up-and-down-arrows, +body:not(.minimal-icons-off) svg.up-arrow-with-tail, +body:not(.minimal-icons-off) svg.up-chevron-glyph, +body:not(.minimal-icons-off) svg.vault, +body:not(.minimal-icons-off) svg.vertical-split, +body:not(.minimal-icons-off) svg.vertical-three-dots, +body:not(.minimal-icons-off) svg.wrench-screwdriver-glyph, +body:not(.minimal-icons-off) svg.clock-glyph, +body:not(.minimal-icons-off) svg.command-glyph, +body:not(.minimal-icons-off) svg.add-note-glyph, +body:not(.minimal-icons-off) svg.calendar-glyph, +body:not(.minimal-icons-off) svg.duplicate-glyph, +body:not(.minimal-icons-off) svg.file-explorer-glyph, +body:not(.minimal-icons-off) svg.graph-glyph, +body:not(.minimal-icons-off) svg.import-glyph, +body:not(.minimal-icons-off) svg.languages, +body:not(.minimal-icons-off) svg.links-coming-in, +body:not(.minimal-icons-off) svg.links-going-out, +body:not(.minimal-icons-off) svg.merge-files-glyph, +body:not(.minimal-icons-off) svg.merge-files, +body:not(.minimal-icons-off) svg.open-elsewhere-glyph, +body:not(.minimal-icons-off) svg.obsidian-leaflet-plugin-icon-map, +body:not(.minimal-icons-off) svg.paper-plane-glyph, +body:not(.minimal-icons-off) svg.paste-text, +body:not(.minimal-icons-off) svg.paste, +body:not(.minimal-icons-off) svg.percent-sign-glyph, +body:not(.minimal-icons-off) svg.play-audio-glyph, +body:not(.minimal-icons-off) svg.plus-minus-glyph, +body:not(.minimal-icons-off) svg.presentation-glyph, +body:not(.minimal-icons-off) svg.question-mark-glyph, +body:not(.minimal-icons-off) svg.reading-glasses, +body:not(.minimal-icons-off) svg.restore-file-glyph, +body:not(.minimal-icons-off) svg.scissors-glyph, +body:not(.minimal-icons-off) svg.scissors, +body:not(.minimal-icons-off) svg.search-glyph, +body:not(.minimal-icons-off) svg.select-all-text, +body:not(.minimal-icons-off) svg.split, +body:not(.minimal-icons-off) svg.star-glyph, +body:not(.minimal-icons-off) svg.stop-audio-glyph, +body:not(.minimal-icons-off) svg.sweep, +body:not(.minimal-icons-off) svg.two-blank-pages, +body:not(.minimal-icons-off) svg.tomorrow-glyph, +body:not(.minimal-icons-off) svg.yesterday-glyph, +body:not(.minimal-icons-off) svg.workspace-glyph, +body:not(.minimal-icons-off) svg.box-glyph, +body:not(.minimal-icons-off) svg.wand, +body:not(.minimal-icons-off) svg.longform, +body:not(.minimal-icons-off) svg.changelog { + background-color: currentColor; } + +body:not(.minimal-icons-off) svg.any-key > path, +body:not(.minimal-icons-off) svg.blocks > path, +body:not(.minimal-icons-off) svg.bar-graph > path, +body:not(.minimal-icons-off) svg.breadcrumbs-trail-icon > path, +body:not(.minimal-icons-off) svg.audio-file > path, +body:not(.minimal-icons-off) svg.bold-glyph > path, +body:not(.minimal-icons-off) svg.italic-glyph > path, +body:not(.minimal-icons-off) svg.bracket-glyph > path, +body:not(.minimal-icons-off) svg.broken-link > path, +body:not(.minimal-icons-off) svg.bullet-list-glyph > path, +body:not(.minimal-icons-off) svg.bullet-list > path, +body:not(.minimal-icons-off) svg.calendar-day > path, +body:not(.minimal-icons-off) svg.calendar-with-checkmark > path, +body:not(.minimal-icons-off) svg.check-in-circle > path, +body:not(.minimal-icons-off) svg.check-small > path, +body:not(.minimal-icons-off) svg.checkbox-glyph > path, +body:not(.minimal-icons-off) svg.checkmark > path, +body:not(.minimal-icons-off) svg.clock > path, +body:not(.minimal-icons-off) svg.cloud > path, +body:not(.minimal-icons-off) svg.code-glyph > path, +body:not(.minimal-icons-off) svg.command-glyph > path, +body:not(.minimal-icons-off) svg.create-new > path, +body:not(.minimal-icons-off) svg.cross-in-box > path, +body:not(.minimal-icons-off) svg.cross > path, +body:not(.minimal-icons-off) svg.crossed-star > path, +body:not(.minimal-icons-off) svg.dice > path, +body:not(.minimal-icons-off) svg.disk > path, +body:not(.minimal-icons-off) svg.document > path, +body:not(.minimal-icons-off) svg.documents > path, +body:not(.minimal-icons-off) svg.dot-network > path, +body:not(.minimal-icons-off) svg.double-down-arrow-glyph > path, +body:not(.minimal-icons-off) svg.double-up-arrow-glyph > path, +body:not(.minimal-icons-off) svg.down-arrow-with-tail > path, +body:not(.minimal-icons-off) svg.down-chevron-glyph > path, +body:not(.minimal-icons-off) svg.enter > path, +body:not(.minimal-icons-off) svg.exit-fullscreen > path, +body:not(.minimal-icons-off) svg.expand-vertically > path, +body:not(.minimal-icons-off) svg.excalidraw-icon path, +body:not(.minimal-icons-off) svg.filled-pin > path, +body:not(.minimal-icons-off) svg.folder > path, +body:not(.minimal-icons-off) svg.fullscreen > path, +body:not(.minimal-icons-off) svg.gear > path, +body:not(.minimal-icons-off) svg.hashtag > path, +body:not(.minimal-icons-off) svg.heading-glyph > path, +body:not(.minimal-icons-off) svg.globe > path, +body:not(.minimal-icons-off) svg.go-to-file > path, +body:not(.minimal-icons-off) svg.help .widget-icon > path, +body:not(.minimal-icons-off) svg.help > path, +body:not(.minimal-icons-off) svg.highlight-glyph > path, +body:not(.minimal-icons-off) svg.horizontal-split > path, +body:not(.minimal-icons-off) svg.image-file > path, +body:not(.minimal-icons-off) svg.image-glyph > path, +body:not(.minimal-icons-off) svg.indent-glyph > path, +body:not(.minimal-icons-off) svg.info > path, +body:not(.minimal-icons-off) svg.install > path, +body:not(.minimal-icons-off) svg.keyboard-glyph > path, +body:not(.minimal-icons-off) svg.left-arrow-with-tail > path, +body:not(.minimal-icons-off) svg.left-arrow > path, +body:not(.minimal-icons-off) svg.left-chevron-glyph > path, +body:not(.minimal-icons-off) svg.lines-of-text > path, +body:not(.minimal-icons-off) svg.link-glyph > path, +body:not(.minimal-icons-off) svg.link > path, +body:not(.minimal-icons-off) svg.magnifying-glass > path, +body:not(.minimal-icons-off) svg.microphone-filled > path, +body:not(.minimal-icons-off) svg.microphone > path, +body:not(.minimal-icons-off) svg.minus-with-circle > path, +body:not(.minimal-icons-off) svg.note-glyph > path, +body:not(.minimal-icons-off) svg.number-list-glyph > path, +body:not(.minimal-icons-off) svg.obsidian-leaflet-plugin-icon-map > path, +body:not(.minimal-icons-off) svg.open-vault > path, +body:not(.minimal-icons-off) svg.pane-layout > path, +body:not(.minimal-icons-off) svg.paper-plane > path, +body:not(.minimal-icons-off) svg.paused > path, +body:not(.minimal-icons-off) svg.pencil > path, +body:not(.minimal-icons-off) svg.pencil_icon > path, +body:not(.minimal-icons-off) svg.pin > path, +body:not(.minimal-icons-off) svg.plus-with-circle > path, +body:not(.minimal-icons-off) svg.popup-open > path, +body:not(.minimal-icons-off) svg.presentation > path, +body:not(.minimal-icons-off) svg.price-tag-glyph > path, +body:not(.minimal-icons-off) svg.quote-glyph > path, +body:not(.minimal-icons-off) svg.redo-glyph > path, +body:not(.minimal-icons-off) svg.reset > path, +body:not(.minimal-icons-off) svg.reading-glasses > path, +body:not(.minimal-icons-off) svg.right-arrow-with-tail > path, +body:not(.minimal-icons-off) svg.right-arrow > path, +body:not(.minimal-icons-off) svg.right-chevron-glyph > path, +body:not(.minimal-icons-off) svg.right-triangle > path, +body:not(.minimal-icons-off) svg.run-command > path, +body:not(.minimal-icons-off) svg.ScriptEngine > path, +body:not(.minimal-icons-off) svg.search > path, +body:not(.minimal-icons-off) svg.sheets-in-box > path, +body:not(.minimal-icons-off) svg.spreadsheet > path, +body:not(.minimal-icons-off) svg.stacked-levels > path, +body:not(.minimal-icons-off) svg.star-list > path, +body:not(.minimal-icons-off) svg.star > path, +body:not(.minimal-icons-off) svg.strikethrough-glyph > path, +body:not(.minimal-icons-off) svg.switch > path, +body:not(.minimal-icons-off) svg.sync-small > path, +body:not(.minimal-icons-off) svg.sync > path, +body:not(.minimal-icons-off) svg.tag-glyph > path, +body:not(.minimal-icons-off) svg.three-horizontal-bars > path, +body:not(.minimal-icons-off) svg.trash > path, +body:not(.minimal-icons-off) svg.undo-glyph > path, +body:not(.minimal-icons-off) svg.unindent-glyph > path, +body:not(.minimal-icons-off) svg.up-and-down-arrows > path, +body:not(.minimal-icons-off) svg.up-arrow-with-tail > path, +body:not(.minimal-icons-off) svg.up-chevron-glyph > path, +body:not(.minimal-icons-off) svg.vault > path, +body:not(.minimal-icons-off) svg.vertical-split > path, +body:not(.minimal-icons-off) svg.vertical-three-dots > path, +body:not(.minimal-icons-off) svg.wrench-screwdriver-glyph > path, +body:not(.minimal-icons-off) svg.clock-glyph > path, +body:not(.minimal-icons-off) svg.add-note-glyph > path, +body:not(.minimal-icons-off) svg.calendar-glyph > path, +body:not(.minimal-icons-off) svg.duplicate-glyph > path, +body:not(.minimal-icons-off) svg.file-explorer-glyph > path, +body:not(.minimal-icons-off) svg.graph-glyph > path, +body:not(.minimal-icons-off) svg.import-glyph > path, +body:not(.minimal-icons-off) svg.languages > path, +body:not(.minimal-icons-off) svg.links-coming-in > path, +body:not(.minimal-icons-off) svg.links-going-out > path, +body:not(.minimal-icons-off) svg.merge-files > path, +body:not(.minimal-icons-off) svg.open-elsewhere-glyph > path, +body:not(.minimal-icons-off) svg.paper-plane-glyph > path, +body:not(.minimal-icons-off) svg.paste-text > path, +body:not(.minimal-icons-off) svg.paste > path, +body:not(.minimal-icons-off) svg.percent-sign-glyph > path, +body:not(.minimal-icons-off) svg.play-audio-glyph > path, +body:not(.minimal-icons-off) svg.plus-minus-glyph > path, +body:not(.minimal-icons-off) svg.presentation-glyph > path, +body:not(.minimal-icons-off) svg.question-mark-glyph > path, +body:not(.minimal-icons-off) svg.restore-file-glyph > path, +body:not(.minimal-icons-off) svg.scissors-glyph > path, +body:not(.minimal-icons-off) svg.scissors > path, +body:not(.minimal-icons-off) svg.search-glyph > path, +body:not(.minimal-icons-off) svg.select-all-text > path, +body:not(.minimal-icons-off) svg.split > path, +body:not(.minimal-icons-off) svg.star-glyph > path, +body:not(.minimal-icons-off) svg.stop-audio-glyph > path, +body:not(.minimal-icons-off) svg.sweep > path, +body:not(.minimal-icons-off) svg.two-blank-pages > path, +body:not(.minimal-icons-off) svg.tomorrow-glyph > path, +body:not(.minimal-icons-off) svg.yesterday-glyph > path, +body:not(.minimal-icons-off) svg.workspace-glyph > path, +body:not(.minimal-icons-off) svg.box-glyph > path, +body:not(.minimal-icons-off) svg.wand > path, +body:not(.minimal-icons-off) svg.longform > path, +body:not(.minimal-icons-off) svg.changelog > path { + display: none; } + +body:not(.minimal-icons-off) svg.any-key { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.audio-file { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.bar-graph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.breadcrumbs-trail-icon { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.blocks { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.bold-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.italic-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.bracket-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.broken-link { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.bullet-list-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.bullet-list { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.calendar-with-checkmark { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.check-in-circle { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.check-small { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.checkbox-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.checkmark { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.clock { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.clock-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.cloud { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.code-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.cross-in-box { + -webkit-mask-image: url("data:image/svg+xml,"); } + +body:not(.minimal-icons-off) svg.cross { + -webkit-mask-image: url("data:image/svg+xml,"); + width: var(--icon-size); + height: var(--icon-size); } + +body:not(.minimal-icons-off) svg.crossed-star { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.dice { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.disk { + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-6 w-6' fill='none' viewBox='0 0 24 24' stroke='currentColor'%3E%3Cpath stroke-linecap='round' stroke-linejoin='round' stroke-width='2' d='M8 7H5a2 2 0 00-2 2v9a2 2 0 002 2h14a2 2 0 002-2V9a2 2 0 00-2-2h-3m-1 4l-3 3m0 0l-3-3m3 3V4' /%3E%3C/svg%3E"); } + +body:not(.minimal-icons-off) svg.document { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) .nav-action-button[aria-label="New note"] svg.document, +body:not(.minimal-icons-off) .workspace-leaf-content[data-type="file-explorer"] .nav-action-button:first-child svg.document, +body:not(.minimal-icons-off) svg.create-new { + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-6 w-6' fill='none' viewBox='0 0 24 24' stroke='currentColor'%3E%3Cpath stroke-linecap='round' stroke-linejoin='round' stroke-width='2' d='M11 5H6a2 2 0 00-2 2v11a2 2 0 002 2h11a2 2 0 002-2v-5m-1.414-9.414a2 2 0 112.828 2.828L11.828 15H9v-2.828l8.586-8.586z' /%3E%3C/svg%3E"); } + +body:not(.minimal-icons-off) svg.documents { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.dot-network { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.double-down-arrow-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.double-up-arrow-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.down-arrow-with-tail { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.down-chevron-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.enter { + transform: translate(-2px); + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.excalidraw-icon { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.expand-vertically { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.filled-pin { + transform: rotate(45deg); + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.folder { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) .workspace-tab-header[data-type="file-explorer"] svg.folder, +body:not(.minimal-icons-off) .workspace-tab-header[aria-label="File explorer"] svg.folder { + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-6 w-6' fill='none' viewBox='0 0 24 24' stroke='currentColor'%3E%3Cpath stroke-linecap='round' stroke-linejoin='round' stroke-width='2' d='M3 12l2-2m0 0l7-7 7 7M5 10v10a1 1 0 001 1h3m10-11l2 2m-2-2v10a1 1 0 01-1 1h-3m-6 0a1 1 0 001-1v-4a1 1 0 011-1h2a1 1 0 011 1v4a1 1 0 001 1m-6 0h6' /%3E%3C/svg%3E"); } + +body:not(.minimal-icons-off) .nav-action-button[aria-label="New folder"] svg.folder, +body:not(.minimal-icons-off) .workspace-leaf-content[data-type="file-explorer"] .nav-action-button:nth-child(2) svg.folder { + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-6 w-6' fill='none' viewBox='0 0 24 24' stroke='currentColor'%3E%3Cpath stroke-linecap='round' stroke-linejoin='round' stroke-width='2' d='M9 13h6m-3-3v6m-9 1V7a2 2 0 012-2h6l2 2h6a2 2 0 012 2v8a2 2 0 01-2 2H5a2 2 0 01-2-2z' /%3E%3C/svg%3E"); } + +body:not(.minimal-icons-off) svg.fullscreen { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.ScriptEngine, +body:not(.minimal-icons-off) svg.gear { + -webkit-mask-image: url("data:image/svg+xml,"); } + +body:not(.minimal-icons-off) svg.globe { + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-6 w-6' fill='none' viewBox='0 0 24 24' stroke='currentColor'%3E%3Cpath stroke-linecap='round' stroke-linejoin='round' stroke-width='2' d='M3.055 11H5a2 2 0 012 2v1a2 2 0 002 2 2 2 0 012 2v2.945M8 3.935V5.5A2.5 2.5 0 0010.5 8h.5a2 2 0 012 2 2 2 0 104 0 2 2 0 012-2h1.064M15 20.488V18a2 2 0 012-2h3.064M21 12a9 9 0 11-18 0 9 9 0 0118 0z' /%3E%3C/svg%3E"); } + +body:not(.minimal-icons-off) svg.hashtag { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.heading-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.go-to-file { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.help .widget-icon, +body:not(.minimal-icons-off) svg.help { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.highlight-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.horizontal-split { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.image-file { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.image-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.indent-glyph { + -webkit-mask-image: url('data:image/svg+xml,%3Csvg xmlns="http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg" width="24" height="24" preserveAspectRatio="xMidYMid meet" viewBox="0 0 16 16"%3E%3Cg fill="black"%3E%3Cpath d="M2 3.5a.5.5 0 0 1 .5-.5h11a.5.5 0 0 1 0 1h-11a.5.5 0 0 1-.5-.5zm.646 2.146a.5.5 0 0 1 .708 0l2 2a.5.5 0 0 1 0 .708l-2 2a.5.5 0 0 1-.708-.708L4.293 8L2.646 6.354a.5.5 0 0 1 0-.708zM7 6.5a.5.5 0 0 1 .5-.5h6a.5.5 0 0 1 0 1h-6a.5.5 0 0 1-.5-.5zm0 3a.5.5 0 0 1 .5-.5h6a.5.5 0 0 1 0 1h-6a.5.5 0 0 1-.5-.5zm-5 3a.5.5 0 0 1 .5-.5h11a.5.5 0 0 1 0 1h-11a.5.5 0 0 1-.5-.5z"%2F%3E%3C%2Fg%3E%3C%2Fsvg%3E'); } + +body:not(.minimal-icons-off) svg.info { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.install { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.keyboard-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.left-arrow-with-tail { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.left-arrow { + -webkit-mask-image: url("data:image/svg+xml,"); } + +body:not(.minimal-icons-off) svg.left-chevron-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.reading-glasses, +body:not(.minimal-icons-off) svg.lines-of-text { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.ledger { + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-6 w-6' fill='none' viewBox='0 0 24 24' stroke='currentColor' stroke-width='2'%3E%3Cpath stroke-linecap='round' stroke-linejoin='round' d='M17 9V7a2 2 0 00-2-2H5a2 2 0 00-2 2v6a2 2 0 002 2h2m2 4h10a2 2 0 002-2v-6a2 2 0 00-2-2H9a2 2 0 00-2 2v6a2 2 0 002 2zm7-5a2 2 0 11-4 0 2 2 0 014 0z' /%3E%3C/svg%3E"); } + +body:not(.minimal-icons-off) svg.link-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); + transform: rotate(90deg); } + +body:not(.minimal-icons-off) svg.link { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); + transform: rotate(90deg); } + +body:not(.minimal-icons-off) svg.magnifying-glass { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.microphone-filled { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.microphone { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.minus-with-circle { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.note-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.number-list-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.open-vault { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.obsidian-leaflet-plugin-icon-map { + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-6 w-6' fill='none' viewBox='0 0 24 24' stroke='currentColor'%3E%3Cpath stroke-linecap='round' stroke-linejoin='round' stroke-width='2' d='M9 20l-5.447-2.724A1 1 0 013 16.382V5.618a1 1 0 011.447-.894L9 7m0 13l6-3m-6 3V7m6 10l4.553 2.276A1 1 0 0021 18.382V7.618a1 1 0 00-.553-.894L15 4m0 13V4m0 0L9 7' /%3E%3C/svg%3E"); } + +body:not(.minimal-icons-off) svg.pane-layout { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.paper-plane { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.paused { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +/* Text Generator plugin */ +body:not(.minimal-icons-off) svg.pencil_icon { + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-6 w-6' fill='none' viewBox='0 0 24 24' stroke='currentColor' stroke-width='2'%3E%3Cpath stroke-linecap='round' stroke-linejoin='round' d='M5 3v4M3 5h4M6 17v4m-2-2h4m5-16l2.286 6.857L21 12l-5.714 2.143L13 21l-2.286-6.857L5 12l5.714-2.143L13 3z' /%3E%3C/svg%3E"); } + +body:not(.minimal-icons-off) svg.pencil { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.pin { + transform: rotate(45deg); + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.plus-with-circle { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.popup-open { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.presentation { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.price-tag-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.quote-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) .workspace-tab-header[data-type="dictionary-view"] svg.quote-glyph { + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-6 w-6' fill='none' viewBox='0 0 24 24' stroke='currentColor'%3E%3Cpath stroke-linecap='round' stroke-linejoin='round' stroke-width='2' d='M12 6.253v13m0-13C10.832 5.477 9.246 5 7.5 5S4.168 5.477 3 6.253v13C4.168 18.477 5.754 18 7.5 18s3.332.477 4.5 1.253m0-13C13.168 5.477 14.754 5 16.5 5c1.747 0 3.332.477 4.5 1.253v13C19.832 18.477 18.247 18 16.5 18c-1.746 0-3.332.477-4.5 1.253' /%3E%3C/svg%3E"); } + +body:not(.minimal-icons-off) svg.redo-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.reset { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.right-arrow-with-tail { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.right-arrow { + -webkit-mask-image: url("data:image/svg+xml,"); } + +body:not(.minimal-icons-off) svg.right-chevron-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.right-triangle { + color: var(--text-faint); + background-color: var(--text-faint); + height: 12px; + width: 12px; + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.command-glyph, +body:not(.minimal-icons-off) svg.run-command { + -webkit-mask-image: url("data:image/svg+xml,"); } + +body:not(.minimal-icons-off) svg.search { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.sheets-in-box { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.spreadsheet { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.stacked-levels { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.star-list { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.star { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.strikethrough-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.switch { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.sync-small { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.sync { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.tag-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body.is-mobile:not(.minimal-icons-off) .view-header-icon svg.three-horizontal-bars { + -webkit-mask-image: url("data:image/svg+xml,%3Csvg width='24' height='24' viewBox='0 0 26 21' fill='none' xmlns='http://www.w3.org/2000/svg'%3E%3Cpath d='M8.18555 18.8857H9.87207V1.91309H8.18555V18.8857ZM6.1123 6.2207C6.27702 6.2207 6.42025 6.15983 6.54199 6.03809C6.66374 5.90918 6.72461 5.76953 6.72461 5.61914C6.72461 5.45443 6.66374 5.31478 6.54199 5.2002C6.42025 5.07845 6.27702 5.01758 6.1123 5.01758H3.81348C3.64876 5.01758 3.50553 5.07845 3.38379 5.2002C3.26204 5.31478 3.20117 5.45443 3.20117 5.61914C3.20117 5.76953 3.26204 5.90918 3.38379 6.03809C3.50553 6.15983 3.64876 6.2207 3.81348 6.2207H6.1123ZM6.1123 9.00293C6.27702 9.00293 6.42025 8.94206 6.54199 8.82031C6.66374 8.69857 6.72461 8.55534 6.72461 8.39062C6.72461 8.23307 6.66374 8.09701 6.54199 7.98242C6.42025 7.86068 6.27702 7.7998 6.1123 7.7998H3.81348C3.64876 7.7998 3.50553 7.86068 3.38379 7.98242C3.26204 8.09701 3.20117 8.23307 3.20117 8.39062C3.20117 8.55534 3.26204 8.69857 3.38379 8.82031C3.50553 8.94206 3.64876 9.00293 3.81348 9.00293H6.1123ZM6.1123 11.7744C6.27702 11.7744 6.42025 11.7171 6.54199 11.6025C6.66374 11.4808 6.72461 11.3411 6.72461 11.1836C6.72461 11.0189 6.66374 10.8792 6.54199 10.7646C6.42025 10.6429 6.27702 10.582 6.1123 10.582H3.81348C3.64876 10.582 3.50553 10.6429 3.38379 10.7646C3.26204 10.8792 3.20117 11.0189 3.20117 11.1836C3.20117 11.3411 3.26204 11.4808 3.38379 11.6025C3.50553 11.7171 3.64876 11.7744 3.81348 11.7744H6.1123ZM3.37305 20.2822H21.957C23.0885 20.2822 23.9336 20.0029 24.4922 19.4443C25.0508 18.8929 25.3301 18.0622 25.3301 16.9521V3.83594C25.3301 2.72591 25.0508 1.89518 24.4922 1.34375C23.9336 0.785156 23.0885 0.505859 21.957 0.505859H3.37305C2.2487 0.505859 1.40365 0.785156 0.837891 1.34375C0.279297 1.89518 0 2.72591 0 3.83594V16.9521C0 18.0622 0.279297 18.8929 0.837891 19.4443C1.40365 20.0029 2.2487 20.2822 3.37305 20.2822ZM3.39453 18.5527C2.85742 18.5527 2.44564 18.4131 2.15918 18.1338C1.87272 17.8473 1.72949 17.4248 1.72949 16.8662V3.92188C1.72949 3.36328 1.87272 2.94434 2.15918 2.66504C2.44564 2.37858 2.85742 2.23535 3.39453 2.23535H21.9355C22.4655 2.23535 22.8737 2.37858 23.1602 2.66504C23.4538 2.94434 23.6006 3.36328 23.6006 3.92188V16.8662C23.6006 17.4248 23.4538 17.8473 23.1602 18.1338C22.8737 18.4131 22.4655 18.5527 21.9355 18.5527H3.39453Z' fill='black'/%3E%3C/svg%3E%0A"); } + +body:not(.minimal-icons-off) svg.three-horizontal-bars { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.trash { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.undo-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.unindent-glyph { + -webkit-mask-image: url('data:image/svg+xml,%3Csvg xmlns="http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg" width="24" height="24" preserveAspectRatio="xMidYMid meet" viewBox="0 0 16 16"%3E%3Cg fill="black"%3E%3Cpath d="M2 3.5a.5.5 0 0 1 .5-.5h11a.5.5 0 0 1 0 1h-11a.5.5 0 0 1-.5-.5zm10.646 2.146a.5.5 0 0 1 .708.708L11.707 8l1.647 1.646a.5.5 0 0 1-.708.708l-2-2a.5.5 0 0 1 0-.708l2-2zM2 6.5a.5.5 0 0 1 .5-.5h6a.5.5 0 0 1 0 1h-6a.5.5 0 0 1-.5-.5zm0 3a.5.5 0 0 1 .5-.5h6a.5.5 0 0 1 0 1h-6a.5.5 0 0 1-.5-.5zm0 3a.5.5 0 0 1 .5-.5h11a.5.5 0 0 1 0 1h-11a.5.5 0 0 1-.5-.5z"%2F%3E%3C%2Fg%3E%3C%2Fsvg%3E'); } + +body:not(.minimal-icons-off) svg.up-and-down-arrows { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.up-arrow-with-tail { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.up-chevron-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.vault { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.vertical-split { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.vertical-three-dots { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.wrench-screwdriver-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.add-note-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.calendar-day { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.calendar-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.duplicate-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.file-explorer-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.graph-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.import-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.languages { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.links-coming-in { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.links-going-out { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.merge-files { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.open-elsewhere-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.paper-plane-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.paste-text { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.paste { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.percent-sign-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.play-audio-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.plus-minus-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.presentation-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.question-mark-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.restore-file-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.scissors-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.scissors { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.search-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.select-all-text { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.split { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.star-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.stop-audio-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.sweep { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.two-blank-pages { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.tomorrow-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.yesterday-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.workspace-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.box-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.wand { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.longform { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +body:not(.minimal-icons-off) svg.changelog { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); } + +/* Fancy cursor */ +.fancy-cursor .CodeMirror-cursor { + border: none; + border-left: 2px solid var(--text-accent); } + +.cm-fat-cursor .CodeMirror-cursor { + background-color: var(--text-accent); + opacity: 0.5; + width: 5px; } + +.cm-animate-fat-cursor { + background-color: var(--text-accent); + opacity: 0.5; + width: 5px; } + +/* Reset to default for iOS */ +body .markdown-source-view.mod-cm6 .cm-content { + caret-color: unset; } + +/* Live Preview */ +body.fancy-cursor .markdown-source-view.mod-cm6 .cm-content, +body.fancy-cursor .mod-cm6 .cm-line { + caret-color: var(--text-accent); } + +/* Prompt */ +.fancy-cursor input.prompt-input { + caret-color: var(--text-accent); } + +.nav-folder-children .nav-folder-children { + margin-left: 18px; + padding-left: 0; } + +body:not(.sidebar-lines-off) .nav-folder-children .nav-folder-children { + border-left: 1px solid var(--background-modifier-border); } + +.nav-folder-title { + margin-left: 6px; } + +.nav-file { + margin-left: 10px; } + +.mod-root > .nav-folder-children > .nav-file { + margin-left: 12px; } + +/* Focus mode */ +/* MIT License | Copyright (c) Stephan Ango (@kepano) */ +/* Hide app ribbon */ +.workspace-ribbon.mod-left { + border-left: 0; + transition: none; } + +.minimal-focus-mode .workspace-ribbon.mod-left.is-collapsed { + border-color: transparent; + background-color: var(--background-primary); } + +.minimal-focus-mode .workspace-ribbon.mod-left { + background-color: var(--background-secondary); + transition: background-color 0s linear 0s; } + +.minimal-focus-mode .workspace-ribbon.mod-left.is-collapsed, +.minimal-focus-mode .workspace-ribbon.is-collapsed .workspace-ribbon-collapse-btn { + opacity: 0; + transition: opacity 0.1s ease-in-out 0.1s, background-color 0.1s linear 0.1s; } + +.minimal-focus-mode .workspace-ribbon.mod-left.is-collapsed:hover, +.minimal-focus-mode .workspace-ribbon.is-collapsed:hover .workspace-ribbon-collapse-btn { + opacity: 1; } + +.is-right-sidedock-collapsed .workspace-split.mod-right-split { + margin-right: 0px; } + +body.minimal-focus-mode.borders-title .workspace-ribbon.mod-left.is-collapsed { + border-right: none; } + +/* Collapse header bar */ +body.minimal-focus-mode.borders-title .workspace-leaf .workspace-leaf-content:not([data-type='empty']):not([data-type='map']):not([data-type='graph']):not([data-type='localgraph']) .view-header, +body.minimal-focus-mode.borders-title .workspace-split.mod-root .workspace-leaf:first-of-type:last-of-type .workspace-leaf-content:not([data-type='empty']):not([data-type='map']):not([data-type='graph']):not([data-type='localgraph']) .view-header { + border-bottom: var(--border-width) solid transparent; } + +body.minimal-focus-mode.borders-title .workspace-leaf .workspace-leaf-content:not([data-type=graph]):not([data-type=localgraph]) .view-header:focus-within, +body.minimal-focus-mode.borders-title .workspace-split.mod-root .workspace-leaf:first-of-type:last-of-type .workspace-leaf-content:not([data-type=graph]):not([data-type=empty]):not([data-type=localgraph]) .view-header:focus-within, +body.minimal-focus-mode.borders-title .workspace-leaf .workspace-leaf-content:not([data-type=graph]):not([data-type=localgraph]) .view-header:hover, +body.minimal-focus-mode.borders-title .workspace-split.mod-root .workspace-leaf:first-of-type:last-of-type .workspace-leaf-content:not([data-type=graph]):not([data-type=empty]):not([data-type=localgraph]) .view-header:hover { + border-bottom: var(--border-width) solid var(--background-divider); } + +body:not(.plugin-sliding-panes-rotate-header) .app-container .workspace-split.mod-root > .workspace-leaf .view-header { + transition: height linear 0.1s; } + +body.minimal-focus-mode:not(.plugin-sliding-panes-rotate-header) .app-container .workspace-split.mod-root > .workspace-leaf .view-header { + height: 0em; + transition: all linear 0.1s; } + +body.minimal-focus-mode:not(.plugin-sliding-panes-rotate-header) .view-header::after { + width: 100%; + content: " "; + background-color: transparent; + height: 20px; + position: absolute; + z-index: -9; + top: 0; } + +body.minimal-focus-mode .mod-left:not(.is-pinned) + .mod-root > div:first-of-type .view-header-icon, +body.minimal-focus-mode:not(.plugin-sliding-panes-rotate-header) .view-header-icon, +body.minimal-focus-mode:not(.plugin-sliding-panes-rotate-header) .view-header-title, +body.minimal-focus-mode:not(.plugin-sliding-panes-rotate-header) .view-actions { + opacity: 0; + transition: all linear 0.1s; } + +body.minimal-focus-mode:not(.plugin-sliding-panes-rotate-header) .workspace-split.mod-root .workspace-leaf .view-header:hover, +body.minimal-focus-mode:not(.plugin-sliding-panes-rotate-header) .workspace-split.mod-root .workspace-leaf .view-header:focus-within { + height: calc(var(--header-height) + 2px); + transition: all linear 0.1s; } + +body.minimal-focus-mode .mod-left:not(.is-pinned) + .mod-root > div:first-of-type .view-header:hover .view-header-icon, +body.minimal-focus-mode .mod-left:not(.is-pinned) + .mod-root > div:first-of-type .view-header:focus-within .view-header-icon, +body.minimal-focus-mode.show-grabber .view-header:hover .view-header-icon, +body.minimal-focus-mode.show-grabber .view-header:focus-within .view-header-icon { + opacity: var(--icon-muted); } + +body.minimal-focus-mode .mod-left:not(.is-pinned) + .mod-root > div:first-of-type .view-header:hover .view-header-icon:hover, +body.minimal-focus-mode .mod-left:not(.is-pinned) + .mod-root > div:first-of-type .view-header:focus-within .view-header-icon:hover, +body.minimal-focus-mode .view-header:hover .view-header-icon:hover, +body.minimal-focus-mode .view-header:focus-within .view-header-icon:hover, +body.minimal-focus-mode .view-header:hover .view-actions, +body.minimal-focus-mode .view-header:focus-within .view-actions, +body.minimal-focus-mode .view-header:hover .view-header-title, +body.minimal-focus-mode .view-header:focus-within .view-header-title { + opacity: 1; + transition: all linear 0.1s; } + +.minimal-focus-mode .view-content { + height: 100%; } + +/* Hide status bar */ +.status-bar { + transition: opacity 0.2s ease-in-out; } + +.minimal-focus-mode:not(.minimal-status-off) .status-bar { + opacity: 0; } + +.minimal-focus-mode .status-bar:hover { + opacity: 1; + transition: opacity 0.2s ease-in-out; } + +/* Full width media */ +.full-width-media .markdown-preview-view .image-embed img:not(.emoji):not([width]), +.full-width-media .image-embed img:not(.emoji):not([width]), +.full-width-media .markdown-preview-view audio, +.full-width-media .markdown-preview-view video { + width: 100%; } + +/* Table helper classes for alternate styles */ +/* MIT License | Copyright (c) Stephan Ango (@kepano) */ +.table-small table:not(.calendar) { + --table-font-size:85%; } + +.table-tiny table:not(.calendar) { + --table-font-size:75%; } + +.markdown-source-view.mod-cm6 th, +.markdown-source-view.mod-cm6 td, +.markdown-preview-view .table-view-table > thead > tr > th, +table:not(.calendar) thead > tr > th, +table:not(.calendar) tbody > tr > td, +.table-view-table .tag, +.table-view-table a.tag { + font-size: var(--table-font-size); } + +.row-hover th:first-child, +.row-hover th:first-child, +.row-alt.markdown-source-view.mod-cm6 th:first-child, +.row-alt.markdown-source-view.mod-cm6 td:first-child, +.row-alt table:not(.calendar) th:first-child, +.row-alt table:not(.calendar) tbody > tr > td:first-child, +.table-lines.markdown-source-view.mod-cm6 th:first-child, +.table-lines.markdown-source-view.mod-cm6 td:first-child, +.table-lines table:not(.calendar) thead > tr > th:first-child, +.table-lines table:not(.calendar) tbody > tr > td:first-child { + padding-left: 10px; } + +.row-alt table:not(.calendar) tbody > tr:nth-child(odd), +.col-alt table:not(.calendar) tr > th:nth-child(2n+2), +.col-alt table:not(.calendar) tr > td:nth-child(2n+2) { + padding-left: 10px; + background: var(--background-table-rows); } + +.table-tabular table:not(.calendar) { + font-variant-numeric: tabular-nums; } + +.table-lines table:not(.calendar), +.table-lines .table-view-table { + border: 1px solid var(--background-modifier-border); } + +.table-lines table:not(.calendar) .table-view-table thead > tr > th, +.table-lines table:not(.calendar) .table-view-table > tbody > tr > td { + border-right: 1px solid var(--background-modifier-border); + border-bottom: 1px solid var(--background-modifier-border); + padding: 4px 10px; } + +.table-nowrap thead > tr > th, +.table-nowrap tbody > tr > td { + white-space: nowrap; + text-overflow: ellipsis; + overflow: hidden; } + +.trim-cols .markdown-preview-view.table-wrap .table-view-table > tbody > tr > td, +.trim-cols .markdown-preview-view.table-wrap .table-view-table > thead > tr > th, +.trim-cols .markdown-source-view.mod-cm6.table-wrap .table-view-table > tbody > tr > td, +.trim-cols .markdown-source-view.mod-cm6.table-wrap .table-view-table > thead > tr > th, +.table-nowrap .table-wrap thead > tr > th, +.table-nowrap .table-wrap tbody > tr > td { + white-space: normal; + overflow: auto; } + +.table-numbers table:not(.calendar) { + counter-reset: section; } + +.table-numbers table:not(.calendar) > thead > tr > th:first-child::before { + content: " "; + padding-right: 0.5em; + display: inline-block; + min-width: 2em; } + +.table-numbers table:not(.calendar) > tbody > tr > td:first-child::before { + counter-increment: section; + content: counter(section) " "; + text-align: center; + padding-right: 0.5em; + display: inline-block; + min-width: 2em; + color: var(--text-faint); + font-variant-numeric: tabular-nums; } + +.row-highlight table:not(.calendar) tbody > tr:hover td { + background-color: var(--background-table-rows); } + +.row-lines table:not(.calendar) tbody > tr > td, +.row-lines .table-view-table > tbody > tr > td { + border-bottom: 1px solid var(--background-modifier-border); } + +.row-lines table:not(.calendar) tbody > tr:last-child > td { + border-bottom: none; } + +.col-lines table:not(.calendar) tbody > tr > td:not(:last-child), +.col-lines .table-view-table thead > tr > th:not(:last-child), +.col-lines .table-view-table > tbody > tr > td:not(:last-child) { + border-right: 1px solid var(--background-modifier-border); } + +/* Highlight rows on hover */ +.row-hover { + --row-color-hover: + hsla( + var(--accent-h), + 50%, + 80%, + 20% + ); } + +.theme-dark.row-hover { + --row-color-hover: + hsla( + var(--accent-h), + 30%, + 40%, + 20% + ); } + +.row-hover tr:hover td { + background-color: var(--row-color-hover); } + +/* Dark mode images */ +/* MIT License | Copyright (c) Stephan Ango (@kepano) */ +.theme-dark .markdown-source-view img, +.theme-dark .markdown-preview-view img { + opacity: var(--image-muted); + transition: opacity 0.25s linear; } + +.theme-dark .print-preview img, +.theme-dark .markdown-source-view img:hover, +.theme-dark .markdown-preview-view img:hover { + opacity: 1; + transition: opacity 0.25s linear; } + +/* Invert */ +.theme-dark img[src$="#invert"], +.theme-dark div[src$="#invert"] img, +.theme-dark span[src$="#invert"] img { + filter: invert(1) hue-rotate(180deg); + mix-blend-mode: screen; } + +.theme-dark div[src$="#invert"] { + background-color: var(--background-primary); } + +.theme-light img[src$="#invertW"], +.theme-light div[src$="#invertW"] img, +.theme-light span[src$="invertW"] img { + filter: invert(1) hue-rotate(180deg); } + +/* Circle */ +img[src$="#circle"], +span[src$="#circle"] img, +span[src$="#round"] img { + border-radius: 50%; + aspect-ratio: 1/1; } + +/* Outline */ +img[src$="#outline"], +span[src$="#outline"] img { + border: 1px solid var(--ui1); } + +/* Interface */ +img[src$="#interface"], +span[src$="#interface"] img { + border: 1px solid var(--ui1); + box-shadow: 0px 0.5px 0.9px rgba(0, 0, 0, 0.021), 0px 1.3px 2.5px rgba(0, 0, 0, 0.03), 0px 3px 6px rgba(0, 0, 0, 0.039), 0px 10px 20px rgba(0, 0, 0, 0.06); + margin-top: 10px; + margin-bottom: 15px; + border-radius: var(--radius-m); } + +/* MIT License | Copyright (c) Stephan Ango (@kepano) + +Image Grid snippet for Obsidian + +author: @kepano +version: 3.0.0 + +Support my work: +https://github.com/sponsors/kepano + +*/ +/* Requires Contextual Typography 2.2.1+ */ +div:not(.el-embed-image) + .el-embed-image { + margin-top: 1rem; } + +.el-embed-image { + margin-top: 0.5rem; } + +.contextual-typography .markdown-preview-section > .el-embed-image > p { + margin-block-start: 0; + margin-block-end: 0; } + +.img-grid .markdown-preview-section .el-embed-image img:not(.emoji):not([width]), +.img-grid .markdown-preview-section video { + width: 100%; } + +.img-grid .markdown-preview-section > .el-embed-image > p { + display: grid; + grid-column-gap: 0.5rem; + grid-row-gap: 0; + grid-template-columns: repeat(auto-fit, minmax(0, 1fr)); } + +.img-grid .markdown-preview-section > .el-embed-image > p > br { + display: none; } + +.img-grid .markdown-preview-section > .el-embed-image > p > img { + object-fit: cover; + align-self: stretch; } + +.img-grid .markdown-preview-section > .el-embed-image > p > .internal-embed img { + object-fit: cover; + height: 100%; } + +.img-grid .img-grid-ratio .markdown-preview-section > .el-embed-image > p > .internal-embed img, +.img-grid.img-grid-ratio .markdown-preview-section > .el-embed-image > p > .internal-embed img { + object-fit: contain; + height: 100%; + align-self: center; } + +@media (max-width: 400pt) { + .el-embed-image { + margin-top: 0.25rem; } + + .img-grid .markdown-preview-section > .el-embed-image > p { + grid-column-gap: 0.25rem; } } +/* Image zoom */ +/* MIT License | Copyright (c) Stephan Ango (@kepano) */ +body:not(.zoom-off) .view-content img { + max-width: 100%; + cursor: zoom-in; } + +body:not(.zoom-off) .view-content img:active { + cursor: zoom-out; } + +body:not(.is-mobile):not(.zoom-off) .view-content .markdown-preview-view img[referrerpolicy='no-referrer']:active, +body:not(.is-mobile):not(.zoom-off) .view-content .image-embed:active { + aspect-ratio: unset; + cursor: zoom-out; + display: block; + z-index: 200; + position: fixed; + max-height: calc(100% + 1px); + max-width: 100%; + height: calc(100% + 1px); + width: 100%; + object-fit: contain; + margin: -0.5px auto 0 !important; + text-align: center; + padding: 0; + left: 0; + right: 0; + bottom: 0; } + +body:not(.is-mobile):not(.zoom-off) .view-content .markdown-preview-view img[referrerpolicy='no-referrer']:active { + background-color: var(--background-primary); + padding: 10px; } + +body:not(.is-mobile):not(.zoom-off) .view-content .image-embed:active:after { + background-color: var(--background-primary); + opacity: 0.9; + content: " "; + height: calc(100% + 1px); + width: 100%; + position: fixed; + left: 0; + right: 1px; + z-index: 0; } + +body:not(.is-mobile):not(.zoom-off) .view-content .image-embed:active img { + aspect-ratio: unset; + top: 50%; + z-index: 99; + transform: translateY(-50%); + padding: 0; + margin: 0 auto; + width: calc(100% - 20px); + max-height: 95vh; + object-fit: contain; + left: 0; + right: 0; + bottom: 0; + position: absolute; + opacity: 1; } + +/* MIT License | Copyright (c) Stephan Ango (@kepano) + +Labeled Nav snippet for Obsidian + +author: @kepano +version: 1.2.0 + +Support my work: +https://github.com/sponsors/kepano + +*/ +.labeled-nav .mod-left-split > .workspace-tabs:nth-child(3) .workspace-tab-header-container { + height: auto; } +.labeled-nav .mod-left-split > .workspace-tabs:nth-child(3) .workspace-tab-container-inner { + flex-direction: column; + padding: 8px 8px 4px 8px; + background-color: transparent; } +.labeled-nav .mod-left-split > .workspace-tabs:nth-child(3) .workspace-tab-header { + padding: 0; + margin-bottom: 2px; + border: none; + height: auto; + opacity: 0.75; } + .labeled-nav .mod-left-split > .workspace-tabs:nth-child(3) .workspace-tab-header.is-active, .labeled-nav .mod-left-split > .workspace-tabs:nth-child(3) .workspace-tab-header:hover { + opacity: 1; + background-color: transparent; } + .labeled-nav .mod-left-split > .workspace-tabs:nth-child(3) .workspace-tab-header .workspace-tab-header-inner { + padding: 0; + box-shadow: none; + border: none; + border-radius: 6px; } + .labeled-nav .mod-left-split > .workspace-tabs:nth-child(3) .workspace-tab-header .workspace-tab-header-inner-icon { + border-radius: 6px; + padding: 5px 6px; + margin: 0; + height: 26px; + width: 100%; + opacity: 1; } +.labeled-nav .mod-left-split > .workspace-tabs:nth-child(3) .workspace-tab-header .workspace-tab-header-inner-icon:hover { + background-color: var(--background-tertiary); } +.labeled-nav .mod-left-split > .workspace-tabs:nth-child(3) .workspace-tab-header-inner-icon { + font-size: var(--font-small); + font-weight: 500; + display: flex; + align-items: center; } +.labeled-nav .mod-left-split > .workspace-tabs:nth-child(3) .workspace-tab-header:hover .workspace-tab-header-inner-icon, +.labeled-nav .mod-left-split > .workspace-tabs:nth-child(3) .workspace-tab-header.is-active .workspace-tab-header-inner-icon { + color: var(--icon-color-active); } +.labeled-nav .mod-left-split > .workspace-tabs:nth-child(3) .workspace-tab-header-inner-icon svg { + margin-right: 6px; } +.labeled-nav .mod-left-split > .workspace-tabs:nth-child(3) .workspace-tab-header-container { + border: none; + padding: 0; } +.labeled-nav .mod-left-split > .workspace-tabs:nth-child(3) .workspace-tab-header .workspace-tab-header-inner-icon:after { + content: "Plugin"; } +.labeled-nav .mod-left-split > .workspace-tabs:nth-child(3) .workspace-tab-header[data-type="backlink"] .workspace-tab-header-inner-icon:after { + content: "Backlinks"; } +.labeled-nav .mod-left-split > .workspace-tabs:nth-child(3) .workspace-tab-header[data-type="calendar"] .workspace-tab-header-inner-icon:after { + content: "Calendar"; } +.labeled-nav .mod-left-split > .workspace-tabs:nth-child(3) .workspace-tab-header[data-type="dictionary-view"] .workspace-tab-header-inner-icon:after { + content: "Dictionary"; } +.labeled-nav .mod-left-split > .workspace-tabs:nth-child(3) .workspace-tab-header[data-type="localgraph"] .workspace-tab-header-inner-icon:after, +.labeled-nav .mod-left-split > .workspace-tabs:nth-child(3) .workspace-tab-header[data-type="graph"] .workspace-tab-header-inner-icon:after { + content: "Graph"; } +.labeled-nav .mod-left-split > .workspace-tabs:nth-child(3) .workspace-tab-header[data-type="markdown"] .workspace-tab-header-inner-icon:after { + content: "Note"; } +.labeled-nav .mod-left-split > .workspace-tabs:nth-child(3) .workspace-tab-header[data-type="file-explorer"] .workspace-tab-header-inner-icon:after { + content: "Notes"; } +.labeled-nav .mod-left-split > .workspace-tabs:nth-child(3) .workspace-tab-header[data-type="outgoing-link"] .workspace-tab-header-inner-icon:after { + content: "Outlinks"; } +.labeled-nav .mod-left-split > .workspace-tabs:nth-child(3) .workspace-tab-header[data-type="outline"] .workspace-tab-header-inner-icon:after { + content: "Outline"; } +.labeled-nav .mod-left-split > .workspace-tabs:nth-child(3) .workspace-tab-header[data-type="recent-files"] .workspace-tab-header-inner-icon:after { + content: "Recent"; } +.labeled-nav .mod-left-split > .workspace-tabs:nth-child(3) .workspace-tab-header[data-type="reminder-list"] .workspace-tab-header-inner-icon:after { + content: "Reminders"; } +.labeled-nav .mod-left-split > .workspace-tabs:nth-child(3) .workspace-tab-header[data-type="search"] .workspace-tab-header-inner-icon:after { + content: "Search"; } +.labeled-nav .mod-left-split > .workspace-tabs:nth-child(3) .workspace-tab-header[data-type="starred"] .workspace-tab-header-inner-icon:after { + content: "Starred"; } +.labeled-nav .mod-left-split > .workspace-tabs:nth-child(3) .workspace-tab-header[data-type="style-settings"] .workspace-tab-header-inner-icon:after { + content: "Style"; } +.labeled-nav .mod-left-split > .workspace-tabs:nth-child(3) .workspace-tab-header[data-type="tag"] .workspace-tab-header-inner-icon:after { + content: "Tags"; } + +/* MIT License | Copyright (c) Stephan Ango (@kepano) + +Layout Control snippet for Obsidian + +author: @kepano +version: 2.0.0 + +Support my work: +https://github.com/sponsors/kepano + +*/ +/* Requires Contextual Typography 2.2.1+ */ +/* Switch to flexbox */ +.contextual-typography .markdown-reading-view > .markdown-preview-view { + padding-top: 15px; } + +.contextual-typography .markdown-preview-view.markdown-preview-view.is-readable-line-width .markdown-preview-sizer { + display: flex; + flex-direction: column; + width: 100%; + max-width: 100%; + padding-left: 0; + padding-top: 0; } + +.contextual-typography .markdown-preview-view.is-readable-line-width .markdown-preview-sizer { + align-items: center; + padding-left: 0; } + +.contextual-typography .markdown-preview-view.is-readable-line-width .markdown-preview-sizer > div { + width: var(--max-width); } + +.contextual-typography .markdown-preview-view.is-readable-line-width .markdown-preview-sizer > div { + margin-left: auto; + margin-right: auto; + max-width: var(--max-width); + width: var(--line-width-adaptive); } + +.contextual-typography .markdown-preview-view.is-readable-line-width .markdown-embed .markdown-preview-sizer > div { + max-width: 100%; } + +.contextual-typography .markdown-preview-view.is-readable-line-width .markdown-preview-sizer > .el-table, +.contextual-typography .markdown-preview-view.is-readable-line-width .markdown-preview-sizer > .el-lang-dataview, +.contextual-typography .markdown-preview-view.is-readable-line-width .markdown-preview-sizer > .el-lang-dataviewjs { + width: 100%; + max-width: 100%; + overflow-x: auto; } + +.el-lang-dataviewjs .block-language-dataviewjs .contains-task-list, +.el-lang-dataview .block-language-dataview .contains-task-list { + max-width: 100%; } + +.is-readable-line-width .el-table table, +.is-readable-line-width .el-lang-dataview .dataview.table-view-table, +.is-readable-line-width .el-lang-dataviewjs .dataview.table-view-table { + width: var(--max-width); + max-width: var(--line-width-adaptive); + margin: 0 auto 0.5rem; } + +.markdown-embed .el-table table, +.markdown-embed .el-lang-dataview .dataview.table-view-table { + width: 100%; } + +/* Dataview and tables */ +.table-100 .el-table table, +.table-100 .el-lang-dataviewjs .dataview.table-view-table, +.table-100 .el-lang-dataview .dataview.table-view-table { + max-width: 100% !important; + width: 100% !important; } + +.markdown-preview-view.table-100.is-readable-line-width .el-table table, +.markdown-preview-view.table-100.is-readable-line-width .el-lang-dataview .dataview.table-view-table, +.markdown-preview-view.table-100.is-readable-line-width .el-lang-dataviewjs .dataview.table-view-table { + max-width: 100% !important; + width: 100% !important; } + +.table-max .el-table table, +.table-max .el-lang-dataview .dataview.table-view-table, +.table-max .el-lang-dataviewjs .dataview.table-view-table { + max-width: 100% !important; } + +.markdown-preview-view.table-max .el-table table, +.markdown-preview-view.table-max .el-lang-dataview .dataview.table-view-table +.markdown-preview-view.table-max .el-lang-dataviewjs .dataview.table-view-table { + max-width: 100% !important; } + +.table-wide .markdown-preview-view.is-readable-line-width .el-table table, +.markdown-preview-view.is-readable-line-width.table-wide .el-table table, +.table-wide .markdown-preview-view.is-readable-line-width .el-lang-dataview .dataview.table-view-table, +.markdown-preview-view.is-readable-line-width.table-wide .el-lang-dataview .dataview.table-view-table, +.table-wide .markdown-preview-view.is-readable-line-width .el-lang-dataviewjs .dataview.table-view-table, +.markdown-preview-view.is-readable-line-width.table-wide .el-lang-dataviewjs .dataview.table-view-table { + max-width: var(--line-width-wide) !important; } + +.table-100 table th:first-child, +.table-100 table td:first-child, +.table-100 .dataview.table-view-table th:first-child, +.table-100 .dataview.table-view-table td:first-child, +.table-100 .markdown-source-view.mod-cm6 td:first-child, +.table-100 .markdown-source-view.mod-cm6 th:first-child { + padding-left: 20px; } + +.table-100 table th:last-child, +.table-100 table td:last-child, +.table-100 .dataview.table-view-table th:last-child, +.table-100 .dataview.table-view-table td:last-child { + padding-right: 20px; } + +/* Maps, images and iframes */ +.contextual-typography.chart-max .markdown-preview-view.is-readable-line-width .markdown-preview-sizer > .el-lang-chart, +.contextual-typography .markdown-preview-view.is-readable-line-width.chart-max .markdown-preview-sizer > .el-lang-chart, +.contextual-typography.map-max .markdown-preview-view.is-readable-line-width .markdown-preview-sizer > .el-lang-leaflet, +.contextual-typography .markdown-preview-view.is-readable-line-width.map-max .markdown-preview-sizer > .el-lang-leaflet, +.contextual-typography.iframe-max .markdown-preview-view.is-readable-line-width .markdown-preview-sizer > .el-iframe, +.contextual-typography .markdown-preview-view.is-readable-line-width.iframe-max .markdown-preview-sizer > .el-iframe, +.contextual-typography.img-max .markdown-preview-view.is-readable-line-width .markdown-preview-sizer > .el-embed-image, +.contextual-typography .markdown-preview-view.is-readable-line-width.img-max .markdown-preview-sizer > .el-embed-image { + width: 100%; } + +.contextual-typography.chart-wide .markdown-preview-view.is-readable-line-width .markdown-preview-sizer > .el-lang-chart, +.contextual-typography .markdown-preview-view.is-readable-line-width.chart-wide .markdown-preview-sizer > .el-lang-chart, +.contextual-typography.map-wide .markdown-preview-view.is-readable-line-width .markdown-preview-sizer > .el-lang-leaflet, +.contextual-typography .markdown-preview-view.is-readable-line-width.map-wide .markdown-preview-sizer > .el-lang-leaflet, +.contextual-typography.iframe-wide .markdown-preview-view.is-readable-line-width .markdown-preview-sizer > .el-iframe, +.contextual-typography .markdown-preview-view.is-readable-line-width.iframe-wide .markdown-preview-sizer > .el-iframe, +.contextual-typography.img-wide .markdown-preview-view.is-readable-line-width .markdown-preview-sizer > .el-embed-image, +.contextual-typography .markdown-preview-view.is-readable-line-width.img-wide .markdown-preview-sizer > .el-embed-image { + width: var(--line-width-wide); } + +.contextual-typography.chart-100 .markdown-preview-view.is-readable-line-width .markdown-preview-sizer > .el-lang-chart, +.contextual-typography .markdown-preview-view.is-readable-line-width.chart-100 .markdown-preview-sizer > .el-lang-chart, +.contextual-typography.map-100 .markdown-preview-view.is-readable-line-width .markdown-preview-sizer > .el-lang-leaflet, +.contextual-typography .markdown-preview-view.is-readable-line-width.map-100 .markdown-preview-sizer > .el-lang-leaflet, +.contextual-typography.iframe-100 .markdown-preview-view.is-readable-line-width .markdown-preview-sizer > .el-iframe, +.contextual-typography .markdown-preview-view.iframe-100 .markdown-preview-sizer > .el-iframe, +.contextual-typography.img-100 .markdown-preview-view.is-readable-line-width .markdown-preview-sizer > .el-embed-image, +.contextual-typography .markdown-preview-view.img-100 .markdown-preview-sizer > .el-embed-image { + width: 100%; + max-width: 100%; } + +.is-readable-line-width .el-table table, +.is-readable-line-width .el-lang-dataview .dataview.table-view-table, +.is-readable-line-width .el-lang-dataviewjs .dataview.table-view-table { + max-width: calc(var(--line-width-adaptive) - var(--folding-offset)); } + +.embed-strict .el-embed-page p, +.map-100 .el-lang-leaflet, +.map-max .el-lang-leaflet, +.map-wide .el-lang-leaflet, +.chart-100 .el-lang-chart, +.chart-max .el-lang-chart, +.chart-wide .el-lang-chart, +.table-100 .el-lang-dataview, +.table-max .el-lang-dataview, +.table-wide .el-lang-dataview, +.table-100 .el-lang-dataviewjs, +.table-max .el-lang-dataviewjs, +.table-wide .el-lang-dataviewjs, +.table-100 .el-table, +.table-max .el-table, +.table-wide .el-table, +.iframe-100 .el-iframe, +.iframe-max .el-iframe, +.iframe-wide .el-iframe, +.img-100 .el-embed-image, +.img-max .el-embed-image, +.img-wide .el-embed-image { + --folding-offset:0px; } + +/* Live Preview */ +.chart-max.markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .block-language-chart, +.chart-max .markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .block-language-chart, +.map-max.markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .block-language-leaflet, +.map-max .markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .block-language-leaflet, +.table-max.markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-table-widget > table, +.table-max .markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-table-widget > table, +.table-max.markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-embed-block > .block-language-dataview, +.table-max .markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-embed-block > .block-language-dataview, +.table-max.markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-embed-block > .block-language-dataviewjs, +.table-max .markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-embed-block > .block-language-dataviewjs, +.table-max.markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-embed-block.cm-table-widget > div:not(.edit-block-button), +.table-max .markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-embed-block.cm-table-widget > div:not(.edit-block-button), +.img-max.markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-content > .image-embed, +.img-max .markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-content > .image-embed, +.img-max.markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-content > img, +.img-max .markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-content > img { + width: var(--max-width) !important; + max-width: var(--max-width) !important; + transform: none !important; + padding-left: 0; + margin: 0 auto !important; } + +.chart-wide.markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .block-language-chart, +.chart-wide .markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .block-language-chart, +.map-wide.markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .block-language-leaflet, +.map-wide .markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .block-language-leaflet, +.table-wide.markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-table-widget > table, +.table-wide .markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-table-widget > table, +.table-wide.markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-embed-block > .block-language-dataview, +.table-wide .markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-embed-block > .block-language-dataview, +.table-wide.markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-embed-block > .block-language-dataviewjs, +.table-wide .markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-embed-block > .block-language-dataviewjs, +.table-wide.markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-embed-block.cm-table-widget > div:not(.edit-block-button), +.table-wide .markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-embed-block.cm-table-widget > div:not(.edit-block-button), +.img-wide.markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-content > .image-embed, +.img-wide .markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-content > .image-embed, +.img-wide.markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-content > img, +.img-wide .markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-content > img { + width: var(--line-width-wide) !important; + max-width: var(--max-width); + transform: none !important; + padding-left: 0; + margin: 0 auto !important; } + +.chart-100.markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .block-language-chart, +.chart-100 .markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .block-language-chart, +.map-100.markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .block-language-leaflet, +.map-100 .markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .block-language-leaflet, +.table-100.markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width table, +.table-100 .markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width table, +.table-100.markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-embed-block > .block-language-dataview, +.table-100 .markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-embed-block > .block-language-dataview, +.table-100.markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-embed-block > .block-language-dataviewjs, +.table-100 .markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-embed-block > .block-language-dataviewjs, +.table-100.markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-embed-block.cm-table-widget > div:not(.edit-block-button), +.table-100 .markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-embed-block.cm-table-widget > div:not(.edit-block-button), +.img-100.markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-content > .image-embed, +.img-100 .markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-content > .image-embed, +.img-100.markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-content > img, +.img-100 .markdown-source-view.mod-cm6.is-live-preview.is-readable-line-width .cm-content > img { + width: 100% !important; + max-width: 100% !important; + transform: none !important; + margin: 0 auto !important; + padding-left: 0; } + +/* Mobile */ +@media (max-width: 400pt) { + .markdown-preview-view .el-table th:first-child, + .markdown-preview-view .el-table td:first-child, + .markdown-preview-view .el-lang-dataview th:first-child, + .markdown-preview-view .el-lang-dataview td:first-child + .markdown-preview-view .el-lang-dataviewjs th:first-child, + .markdown-preview-view .el-lang-dataviewjs td:first-child { + padding-left: 6vw; } + + .markdown-preview-view .el-table th:last-child, + .markdown-preview-view .el-table td:last-child, + .markdown-preview-view .el-lang-dataview th:last-child, + .markdown-preview-view .el-lang-dataview td:last-child, + .markdown-preview-view .el-lang-dataviewjs th:last-child, + .markdown-preview-view .el-lang-dataviewjs td:last-child { + padding-right: 6vw; } + + .markdown-preview-view.is-readable-line-width .markdown-preview-sizer > .el-table, + .markdown-preview-view.is-readable-line-width .markdown-preview-sizer > .el-lang-dataview + .markdown-preview-view.is-readable-line-width .markdown-preview-sizer > .el-lang-dataviewjs { + padding-left: 0; + padding-right: 0; } + + .markdown-preview-view.is-readable-line-width .markdown-preview-sizer > .el-table, + .markdown-preview-view .table-view-table table, + .markdown-preview-view.is-readable-line-width .markdown-preview-sizer > .el-lang-dataview + .markdown-preview-view.is-readable-line-width .markdown-preview-sizer > .el-lang-dataviewjs { + width: 100%; } } +/* Custom line width with folding offset */ +@media (max-width: 400pt) { + .is-mobile { + --folding-offset:0px; } } +/* Nudge titlebar */ +body:not(.title-align-center):not(.title-align-left):not(.plugin-sliding-panes-rotate-header) .view-header-title { + padding-left: var(--folding-offset); } + +.markdown-source-view.wide, +.markdown-preview-view.wide { + --line-width-adaptive:var(--line-width-wide); } + +.markdown-source-view.max, +.markdown-preview-view.max { + --line-width-adaptive:300em; + --line-width-wide:300em; } + +/* With readable line width */ +.markdown-preview-view.is-readable-line-width .markdown-preview-sizer { + max-width: var(--max-width); + width: var(--line-width-adaptive); + padding-left: 0; } + +.markdown-source-view.is-readable-line-width .CodeMirror { + padding-left: 0; + padding-right: 0; + margin: 0 auto 0 auto; + width: var(--line-width-adaptive); + max-width: var(--max-width); } + +/* Readable line width off */ +.markdown-reading-view .markdown-preview-view:not(.is-readable-line-width) > .markdown-preview-sizer { + max-width: var(--max-width); + margin: 0 auto; + padding-left: var(--folding-offset); } + +.is-mobile .markdown-source-view.mod-cm6 .cm-gutters { + padding-right: 0; } + +/* Requires Minimal plugin 5.2.1+ */ +.minimal-readable-off .view-header-title-container { + width: var(--max-width); } + +/* Max width for readable-line length off */ +.markdown-source-view.mod-cm6:not(.is-readable-line-width) .cm-contentContainer { + max-width: var(--max-width); + margin: 0 0 0 calc(50% - var(--max-width)/2) !important; + padding-left: var(--folding-offset); } + +.markdown-source-view.mod-cm6 .cm-content > .cm-embed-block[contenteditable=false] { + overflow-x: auto; } + +/* Folding offset */ +.markdown-preview-view.is-readable-line-width .markdown-preview-sizer > div, +.markdown-preview-view.is-readable-line-width .markdown-preview-sizer > div[data-block-language="dataview"], +.markdown-preview-view.is-readable-line-width .markdown-preview-sizer > div[data-block-language="dataviewjs"] { + padding-left: var(--folding-offset); } + +.internal-embed > .markdown-embed, +.popover:not(.hover-editor) { + --folding-offset:0; } + +/* Live Preview */ +.markdown-source-view.mod-cm6.is-line-wrap.is-readable-line-width .cm-content { + max-width: 100%; } + +.markdown-source-view.mod-cm6.is-line-wrap.is-readable-line-width .cm-line:not(.HyperMD-table-row) { + max-width: calc(var(--max-width) - var(--folding-offset)); } + +/* Fill the width of the parent block for nested elements */ +.is-live-preview.is-readable-line-width.embed-strict .internal-embed .markdown-preview-sizer, +.is-readable-line-width .block-language-dataview table.dataview, +.is-readable-line-width .block-language-dataviewjs table.dataview, +.is-live-preview.is-readable-line-width .cm-embed-block table.dataview, +.markdown-source-view.is-live-preview.is-readable-line-width table.NLT__table, +.markdown-preview-view.is-readable-line-width .dataview.result-group .contains-task-list { + width: 100%; + max-width: 100%; + transform: none; + margin-left: auto !important; } + +/* Remove margins when nested */ +.markdown-source-view.mod-cm6.is-readable-line-width .cm-line > .internal-embed, +.markdown-source-view.mod-cm6.is-readable-line-width .cm-line.HyperMD-list-line .internal-embed.image-embed { + margin-left: 0 !important; } + +/* Line width for Live Preview / Editor mode + Gets complicated. + -------------------------------------------*/ +/* Nudge everything slightly to the left to make space for folding and gutters */ +/* This is the big daddy rule for most editor content line types */ +.markdown-source-view.mod-cm6.is-readable-line-width { + /* Don't force width for images that have a width */ } + .markdown-source-view.mod-cm6.is-readable-line-width .internal-embed, + .markdown-source-view.mod-cm6.is-readable-line-width .cm-content > .image-embed, + .markdown-source-view.mod-cm6.is-readable-line-width .cm-line, + .markdown-source-view.mod-cm6.is-readable-line-width .cm-line.HyperMD-quote, + .markdown-source-view.mod-cm6.is-readable-line-width .cm-line.HyperMD-codeblock, + .markdown-source-view.mod-cm6.is-readable-line-width .embedded-backlinks, + .markdown-source-view.mod-cm6.is-readable-line-width .cm-embed-block.cm-callout > .callout, + .markdown-source-view.mod-cm6.is-readable-line-width .cm-html-embed, + .markdown-source-view.mod-cm6.is-readable-line-width .cm-content > img:not([width]), + .markdown-source-view.mod-cm6.is-readable-line-width table { + width: calc(var(--line-width-adaptive) - var(--folding-offset)); + max-width: calc(var(--max-width) - var(--folding-offset)); + margin-right: auto; + margin-left: max(calc(50% + var(--folding-offset) - var(--line-width-adaptive)/2), calc(50% + var(--folding-offset) - var(--max-width)/2)) !important; } + .markdown-source-view.mod-cm6.is-readable-line-width .cm-line > .cm-html-embed { + --folding-offset:0; } + .markdown-source-view.mod-cm6.is-readable-line-width .cm-content > img[width] { + max-width: var(--max-width); + margin-left: max(calc(50% + var(--folding-offset) - var(--line-width-adaptive)/2), calc(50% + var(--folding-offset) - var(--max-width)/2)) !important; } + +.markdown-source-view.mod-cm6.is-readable-line-width .mod-empty, +.markdown-source-view.mod-cm6.is-readable-line-width .cm-embed-block > div, +.markdown-source-view.mod-cm6.is-readable-line-width .cm-embed-block > mjx-container { + width: calc(var(--line-width-adaptive) - var(--folding-offset)); + max-width: calc(var(--max-width) - var(--folding-offset)); + margin-right: auto; + margin-left: max(calc(50% + var(--folding-offset) - var(--line-width-adaptive)/2), calc(50% + var(--folding-offset) - var(--max-width)/2)) !important; } + +/* For lists adding an extra offset value in Edit mode */ +/* Needs .is-line-wrap to override default styling */ +.markdown-source-view.mod-cm6.is-readable-line-width.is-line-wrap .HyperMD-list-line { + width: calc(var(--line-width-adaptive) - var(--folding-offset) - var(--list-edit-offset)); + max-width: calc(var(--max-width) - var(--folding-offset) - var(--list-edit-offset)); + margin-right: auto; + margin-left: max(calc(50% + var(--list-edit-offset) + var(--folding-offset) - var(--line-width-adaptive)/2), calc(50% + var(--list-edit-offset) + var(--folding-offset) - var(--max-width)/2)) !important; } + +/* Dataview lists/checklists + A nightmare mainly because there is no selector that indicates + a list is present inside the dataview block + -------------------------------------------*/ +/* Normal block width */ +/* ------------------ */ +body:not(.table-100):not(.table-max):not(.table-wide) .is-live-preview.is-readable-line-width:not(.table-100):not(.table-max):not(.table-wide) .dataview.list-view-ul, +body:not(.table-100):not(.table-max):not(.table-wide) .is-live-preview.is-readable-line-width:not(.table-100):not(.table-max):not(.table-wide) .dataview > h4, +body:not(.table-100):not(.table-max):not(.table-wide) .is-live-preview.is-readable-line-width:not(.table-100):not(.table-max):not(.table-wide) .dataview.result-group > .contains-task-list, +body:not(.table-100):not(.table-max):not(.table-wide) .is-live-preview.is-readable-line-width:not(.table-100):not(.table-max):not(.table-wide) .dataview.dataview-container > .contains-task-list { + max-width: 100%; + margin-right: auto; + margin-left: auto; + transform: none; } +body:not(.table-100):not(.table-max):not(.table-wide) .markdown-preview-view.is-readable-line-width:not(.table-100):not(.table-max):not(.table-wide) .dataview.list-view-ul, +body:not(.table-100):not(.table-max):not(.table-wide) .markdown-preview-view.is-readable-line-width:not(.table-100):not(.table-max):not(.table-wide) .dataview.dataview-container > .contains-task-list, +body:not(.table-100):not(.table-max):not(.table-wide) .markdown-preview-view.is-readable-line-width:not(.table-100):not(.table-max):not(.table-wide) .block-language-dataviewjs > p, +body:not(.table-100):not(.table-max):not(.table-wide) .markdown-preview-view.is-readable-line-width:not(.table-100):not(.table-max):not(.table-wide) .block-language-dataviewjs > h1, +body:not(.table-100):not(.table-max):not(.table-wide) .markdown-preview-view.is-readable-line-width:not(.table-100):not(.table-max):not(.table-wide) .block-language-dataviewjs > h2, +body:not(.table-100):not(.table-max):not(.table-wide) .markdown-preview-view.is-readable-line-width:not(.table-100):not(.table-max):not(.table-wide) .block-language-dataviewjs > h3, +body:not(.table-100):not(.table-max):not(.table-wide) .markdown-preview-view.is-readable-line-width:not(.table-100):not(.table-max):not(.table-wide) .block-language-dataviewjs > h4, +body:not(.table-100):not(.table-max):not(.table-wide) .markdown-preview-view.is-readable-line-width:not(.table-100):not(.table-max):not(.table-wide) .block-language-dataviewjs h4, +body:not(.table-100):not(.table-max):not(.table-wide) .markdown-preview-view.is-readable-line-width:not(.table-100):not(.table-max):not(.table-wide) .block-language-dataview > h4, +body:not(.table-100):not(.table-max):not(.table-wide) .markdown-preview-view.is-readable-line-width:not(.table-100):not(.table-max):not(.table-wide) .block-language-dataview h4, +body:not(.table-100):not(.table-max):not(.table-wide) .markdown-preview-view.is-readable-line-width:not(.table-100):not(.table-max):not(.table-wide) .dataview.result-group, +body:not(.table-100):not(.table-max):not(.table-wide) .markdown-preview-view.is-readable-line-width:not(.table-100):not(.table-max):not(.table-wide) .dataview.dataview-error { + width: calc(var(--line-width-adaptive) - var(--folding-offset)); + max-width: var(--max-width); + margin-right: auto; + margin-left: auto; } + +/* Wider block widths */ +/* ------------------ */ +.is-live-preview.is-readable-line-width .dataview.list-view-ul, +.is-live-preview.is-readable-line-width .dataview > h4, +.is-live-preview.is-readable-line-width .block-language-dataviewjs h4, +.is-live-preview.is-readable-line-width .dataview .contains-task-list, +.is-live-preview.is-readable-line-width .dataview.dataview-container .contains-task-list { + --folding-offset:10px; + width: calc(var(--line-width-adaptive) - var(--folding-offset)); + max-width: calc(100% - var(--folding-offset)); + transform: translateX(calc(var(--folding-offset)/2)); + margin-right: auto; + margin-left: auto; } + +.table-100 .is-live-preview.is-readable-line-width .dataview.list-view-ul, +.table-100 .is-live-preview.is-readable-line-width .dataview > h4, +.table-100 .is-live-preview.is-readable-line-width .dataview .contains-task-list, +.table-100.is-live-preview.is-readable-line-width .dataview.list-view-ul, +.table-100.is-live-preview.is-readable-line-width .dataview > h4, +.table-100.is-live-preview.is-readable-line-width .dataview .contains-task-list { + max-width: calc(var(--max-width) - var(--folding-offset)); } + +.markdown-preview-view.is-readable-line-width .dataview.list-view-ul, +.markdown-preview-view.is-readable-line-width .dataview .contains-task-list, +.markdown-preview-view.is-readable-line-width .block-language-dataviewjs > p, +.markdown-preview-view.is-readable-line-width .block-language-dataviewjs > h1, +.markdown-preview-view.is-readable-line-width .block-language-dataviewjs > h2, +.markdown-preview-view.is-readable-line-width .block-language-dataviewjs > h3, +.markdown-preview-view.is-readable-line-width .block-language-dataviewjs > h4, +.markdown-preview-view.is-readable-line-width .block-language-dataviewjs h4, +.markdown-preview-view.is-readable-line-width .block-language-dataview > h4, +.markdown-preview-view.is-readable-line-width .block-language-dataview h4, +.markdown-preview-view.is-readable-line-width .dataview.result-group, +.markdown-preview-view.is-readable-line-width .dataview.dataview-error { + --folding-offset:10px; + width: calc(var(--line-width-adaptive) - var(--folding-offset)); + max-width: calc(var(--max-width) - var(--folding-offset)); + margin-left: auto; + margin-right: max(calc(50% - var(--line-width-adaptive)/2), calc(50% - var(--max-width)/2)); } + +/* Links and underline handling*/ +body:not(.links-int-on) a[href*="obsidian://"], +body:not(.links-int-on) .markdown-preview-view .internal-link, +body:not(.links-ext-on) .external-link, +body:not(.links-ext-on) .cm-link .cm-underline, +body:not(.links-ext-on) .cm-s-obsidian span.cm-url, +body:not(.links-int-on) .cm-hmd-internal-link .cm-underline, +body:not(.links-int-on) a.internal-link, +body:not(.links-int-on) .cm-s-obsidian span.cm-hmd-internal-link:hover { + text-decoration: none; } + +.links-int-on .is-live-preview .cm-hmd-internal-link, +.links-int-on .markdown-preview-view .internal-link, +.links-int-on .cm-s-obsidian span.cm-hmd-internal-link, +.markdown-preview-view .internal-link { + text-decoration: underline; } + +.links-ext-on .external-link, +.external-link { + background-position-y: center; + text-decoration: underline; } + +/* Scroll indicator for sidebar containers */ +body:not(.is-translucent):not(.is-mobile) .mod-left-split .item-list, +body:not(.is-translucent):not(.is-mobile) .mod-left-split .nav-files-container, +body:not(.is-translucent):not(.is-mobile) .mod-left-split .workspace-leaf-content[data-type='search'] .search-result-container, +body:not(.is-translucent):not(.is-mobile) .mod-left-split .tag-container, +body:not(.is-translucent):not(.is-mobile) .mod-left-split .outgoing-link-pane, +body:not(.is-translucent):not(.is-mobile) .mod-left-split .backlink-pane { + background: linear-gradient(var(--background-secondary) 10%, rgba(255, 255, 255, 0)) center top, linear-gradient(var(--background-modifier-border) 100%, rgba(0, 0, 0, 0)) center top; + background-repeat: no-repeat; + background-size: 100% 40px, 91% var(--border-width); + background-attachment: local, scroll; } + +body:not(.is-mobile) .mod-right-split .item-list, +body:not(.is-mobile) .mod-right-split .nav-files-container, +body:not(.is-mobile) .mod-right-split .workspace-leaf-content[data-type='search'] .search-result-container, +body:not(.is-mobile) .mod-right-split .tag-container, +body:not(.is-mobile) .mod-right-split .outgoing-link-pane, +body:not(.is-mobile) .mod-right-split .backlink-pane { + background: linear-gradient(var(--background-primary) 10%, rgba(255, 255, 255, 0)) center top, linear-gradient(var(--background-modifier-border) 100%, rgba(0, 0, 0, 0)) center top; + background-repeat: no-repeat; + background-size: 100% 40px, 91% var(--border-width); + background-attachment: local, scroll; } + +/* Sidebar documents */ +.mod-left-split .markdown-preview-sizer > div, +.mod-left-split .cm-contentContainer { + padding-left: 0 !important; + max-width: 100% !important; } + +.workspace > .workspace-split:not(.mod-root) .CodeMirror, +.workspace > .workspace-split:not(.mod-root) .cm-scroller, +.workspace > .workspace-split:not(.mod-root) .markdown-preview-view { + font-size: var(--font-adaptive-small); + line-height: 1.25; } +.workspace > .workspace-split:not(.mod-root) .workspace-leaf-content[data-type=markdown] .markdown-preview-view { + padding: 0 15px; } +.workspace > .workspace-split:not(.mod-root) .workspace-leaf-content[data-type=markdown] .markdown-embed .markdown-preview-view { + padding: 0; } +.workspace > .workspace-split:not(.mod-root) .CodeMirror, +.workspace > .workspace-split:not(.mod-root) .markdown-preview-section, +.workspace > .workspace-split:not(.mod-root) .markdown-preview-sizer { + max-width: 100%; + padding: 0; + width: auto; } +.workspace > .workspace-split:not(.mod-root) .cm-editor { + --folding-offset: 0px; } + +.minimal-folding .workspace > .workspace-split:not(.mod-root) .workspace-leaf-content[data-type=markdown] .allow-fold-headings.markdown-preview-view .markdown-preview-sizer, +.minimal-folding .workspace > .workspace-split:not(.mod-root) .workspace-leaf-content[data-type=markdown] .allow-fold-lists.markdown-preview-view .markdown-preview-sizer { + padding-left: 0; } + +/* Hide embed styling for sidebar documents */ +.workspace > .workspace-split:not(.mod-root) .internal-embed .markdown-embed { + border: none; + padding: 0; } + +.workspace > .workspace-split:not(.mod-root) .CodeMirror-sizer { + padding-left: 10px; } + +/* Hidden tabs +Needs some work + +.mod-right-split { + .workspace-tab-header-container:not(:hover) { + height:0; + opacity:0; + z-index:999; + width:100%; + transition:height 0.1s linear, opacity 0.1s linear; + &::after { + width:100%; + content:" "; + background-color:transparent; + height:20px; + position:absolute; + z-index:99; + top:0; + } + } +} +.workspace-tab-header-container { + transition:height 0.1s linear, opacity 0.1s linear; +} + */ +/* Underline */ +.tab-style-2 .workspace-tab-header-container .workspace-tab-header { + flex-grow: 1; + height: var(--header-height); } + +.tab-style-2 .workspace-tab-container-inner { + padding: 0; } + +.tab-style-2 .workspace-tab-header-container .workspace-tab-header .workspace-tab-header-inner { + justify-content: center; + align-items: center; + border-bottom: 1px solid var(--background-divider); + border-radius: 0; + transition: none; } + +.tab-style-2 .workspace-tab-header-container .workspace-tab-header .workspace-tab-header-inner:hover { + background-color: var(--bg3); } + +.tab-style-2 .workspace-tab-header-container .workspace-tab-header.is-active .workspace-tab-header-inner { + border-bottom: 2px solid var(--ax3); + padding-top: 1px; + color: var(--ax3); } + +.tab-style-2 .workspace-tab-header-inner-icon:hover { + background-color: transparent; } + +/* Wide */ +.tab-style-3 .workspace-sidedock-empty-state + .workspace-tabs .workspace-tab-header-container, +.tab-style-3 .mod-right-split .workspace-sidedock-empty-state + .workspace-tabs .workspace-tab-header-container { + border-bottom: none; } + +.tab-style-3 .workspace-tab-header-container { + padding-left: 7px; + padding-right: 7px; + border: none; } + +.tab-style-3 .workspace-tab-header-container .workspace-tab-header { + flex-grow: 1; } + +.tab-style-3 .workspace-tab-container-inner { + padding: 3px; + background: var(--bg3); + border-radius: 6px; } + +.tab-style-3 .workspace-tab-header-container .workspace-tab-header .workspace-tab-header-inner { + justify-content: center; + align-items: center; + transition: none; + border: 1px solid transparent; } + +.tab-style-3 .workspace-tab-header-container .workspace-tab-header .workspace-tab-header-inner:hover { + background-color: transparent; } + +.tab-style-3:not(.minimal-dark-tonal) .mod-left-split .workspace-tab-header-container .workspace-tab-header.is-active .workspace-tab-header-inner { + background: var(--bg2); } + +.tab-style-3 .workspace-tab-header-container .workspace-tab-header.is-active .workspace-tab-header-inner { + background: var(--bg1); + box-shadow: 0px 1px 1px 0 rgba(0, 0, 0, 0.1); + border-radius: 4px; } + +.tab-style-3.labeled-nav .mod-left-split > .workspace-tabs:nth-child(3) .workspace-tab-header-container .workspace-tab-header.is-active .workspace-tab-header-inner { + background-color: transparent; } + +.tab-style-3 .workspace-tab-header-inner-icon { + height: 18px; + padding: 0; } + +.tab-style-3 .workspace-tab-header-inner-icon:hover { + background-color: transparent; } + +/* Index */ +.tab-style-4 .workspace-sidedock-empty-state + .workspace-tabs .workspace-tab-header-container { + border: none; } + +.tab-style-4 .workspace-tab-header-container .workspace-tab-header { + flex-grow: 1; + height: var(--header-height); } + +.tab-style-4 .workspace-tab-container-inner { + background-color: var(--background-secondary); + padding: 0; } + +.tab-style-4 .workspace-tab-header-container .workspace-tab-header .workspace-tab-header-inner { + justify-content: center; + align-items: center; + border-bottom: none; + border-radius: 0; + transition: none; + border-top: 1px solid transparent; } + +.tab-style-4 .workspace-tab-header-container .workspace-tab-header { + border-bottom: 1px solid var(--background-modifier-border); + opacity: 1; } + +.tab-style-4 .workspace-tab-header-container .workspace-tab-header .workspace-tab-header-inner-icon { + opacity: var(--icon-muted); } + +.tab-style-4 .workspace-tab-header-container .workspace-tab-header.is-active .workspace-tab-header-inner-icon { + opacity: 1; } + +.tab-style-4.hider-frameless:not(.labeled-nav) .mod-left-split > .workspace-tabs:nth-child(3) .workspace-tab-header-container .workspace-tab-header.is-active .workspace-tab-header-inner { + border-top: 1px solid var(--background-modifier-border); } + +.tab-style-4 .workspace-tab-header-container .workspace-tab-header.is-active { + border-bottom: 1px solid transparent; } + +.tab-style-4 .workspace-tab-header-container .workspace-tab-header.is-active { + background-color: var(--background-primary); + border-radius: 0; } + +.tab-style-4 .mod-left-split .workspace-tab-header-container .workspace-tab-header.is-active { + background-color: var(--background-secondary); } + +.tab-style-4 .workspace-tab-header-container .workspace-tab-header.is-active .workspace-tab-header-inner { + box-shadow: 1px 0 var(--background-modifier-border), -1px 0 var(--background-modifier-border); + border-bottom: none; } + +.tab-style-4 .workspace-tab-header-inner-icon:hover { + background-color: transparent; } + +/* Translucent sidebars */ +:root { + --bg-translucency-light:0.7; + --bg-translucency-dark:0.85; } + +.theme-light.frosted-sidebar.is-translucent, +.theme-dark.frosted-sidebar.is-translucent { + --opacity-translucency:1; } + +.is-translucent.frosted-sidebar:not(.hider-ribbon) .workspace-ribbon.mod-left, +.is-translucent.frosted-sidebar .workspace-split:not(.mod-right-split) .workspace-tabs { + background: transparent; } + +.is-translucent.frosted-sidebar:not(.hider-ribbon) .workspace-ribbon.mod-left:after { + background: var(--background-secondary); + opacity: var(--bg-translucency-light); + top: 0px; + left: 0px; + content: ""; + height: 120%; + position: fixed; + width: 42px; + z-index: -10; } + +.is-translucent.frosted-sidebar .mod-left-split .workspace-tabs:after { + background: var(--background-secondary); + opacity: var(--bg-translucency-light); + top: -50px; + content: ""; + height: 120%; + position: fixed; + width: 120%; + z-index: -10; } + +.theme-dark.is-translucent.frosted-sidebar:not(.hider-ribbon) .workspace-ribbon.mod-left:after, +.theme-dark.is-translucent.frosted-sidebar .workspace-split:not(.mod-right-split) .workspace-tabs:after { + opacity: var(--bg-translucency-dark); } + +.theme-light.is-translucent.frosted-sidebar.minimal-light-white .workspace-split:not(.mod-right-split) .workspace-tabs:after { + background: white; } + +.theme-dark.is-translucent.frosted-sidebar.minimal-dark-black .workspace-split:not(.mod-right-split) .workspace-tabs:after { + background: black; } + +.is-translucent .status-bar { + margin: 0; } + +/* Turn off file name trimming */ +.full-file-names .tree-item-inner, +.full-file-names .nav-file-title-content, +.full-file-names .search-result-file-title, +.nav-file-title-content.is-being-renamed { + text-overflow: unset; + white-space: normal; + line-height: 1.35; } + +.full-file-names .nav-file-title { + margin-bottom: 3px; } + +/* Underline headings */ +.theme-light, +.theme-dark { + --h1l:var(--ui1); + --h2l:var(--ui1); + --h3l:var(--ui1); + --h4l:var(--ui1); + --h5l:var(--ui1); + --h6l:var(--ui1); } + +.h1-l .markdown-reading-view h1:not(.embedded-note-title), +.h1-l .mod-cm6 .cm-editor .HyperMD-header-1 { + border-bottom: 1px solid var(--h1l); + padding-bottom: 0.4em; + margin-block-end: 0.6em; } + +.h2-l .markdown-reading-view h2, +.h2-l .mod-cm6 .cm-editor .HyperMD-header-2 { + border-bottom: 1px solid var(--h2l); + padding-bottom: 0.4em; + margin-block-end: 0.6em; } + +.h3-l .markdown-reading-view h3, +.h3-l .mod-cm6 .cm-editor .HyperMD-header-3 { + border-bottom: 1px solid var(--h3l); + padding-bottom: 0.4em; + margin-block-end: 0.6em; } + +.h4-l .markdown-reading-view h4, +.h4-l .mod-cm6 .cm-editor .HyperMD-header-4 { + border-bottom: 1px solid var(--h4l); + padding-bottom: 0.4em; + margin-block-end: 0.6em; } + +.h5-l .markdown-reading-view h5, +.h5-l .mod-cm6 .cm-editor .HyperMD-header-5 { + border-bottom: 1px solid var(--h5l); + padding-bottom: 0.4em; + margin-block-end: 0.6em; } + +.h6-l .markdown-reading-view h6, +.h6-l .mod-cm6 .cm-editor .HyperMD-header-6 { + border-bottom: 1px solid var(--h6l); + padding-bottom: 0.4em; + margin-block-end: 0.6em; } + +/* Mobile */ +/* Mobile styling +/* MIT License | Copyright (c) Stephan Ango (@kepano) */ +/* Needs cleanup +-------------------------------------------------------------------------------- */ +.is-mobile { + --font-settings-title:18px; + --font-settings:16px; + --font-settings-small:13px; + --input-height:38px; + --radius-m:8px; } + +@media (min-width: 400pt) { + .is-mobile { + --input-height:36px; + --radius-m:6px; } } +.hider-tooltips .follow-link-popover { + display: none; } + +.is-mobile .follow-link-popover { + font-family: var(--font-interface); } + +/* Padding reset */ +body.is-mobile { + padding: 0 !important; } + +.is-mobile { + /* Folding on mobile */ } + .is-mobile .titlebar { + height: 0 !important; + padding: 0 !important; + position: relative !important; + border-bottom: none; } + .is-mobile .safe-area-top-cover { + background-color: transparent; } + .is-mobile .horizontal-main-container { + background-color: var(--background-primary); } + .is-mobile .workspace { + border-radius: 0 !important; + transform: none !important; } + .is-mobile .workspace-drawer:not(.is-pinned) { + width: 100vw; + max-width: 360pt; + border: none; + box-shadow: 0 5px 50px 5px rgba(0, 0, 0, 0.05); } + .is-mobile .workspace-drawer.mod-left.is-pinned { + max-width: var(--mobile-left-sidebar-width); + min-width: 150pt; } + .is-mobile .workspace-drawer.mod-right.is-pinned { + max-width: var(--mobile-right-sidebar-width); + min-width: 150pt; } + .is-mobile .workspace-drawer.mod-right.is-pinned { + border-right: none; } + .is-mobile .workspace-leaf-content[data-type=starred] .item-list { + padding-left: 5px; } + .is-mobile .workspace-drawer-tab-container > * { + padding: 0; } + .is-mobile .workspace-drawer-tab-option-item-title, + .is-mobile .workspace-drawer-active-tab-title { + font-size: var(--font-adaptive-small); } + .is-mobile .workspace-drawer-tab-option-item:hover .workspace-drawer-tab-option-item-title, + .is-mobile .workspace-drawer-active-tab-header:hover .workspace-drawer-active-tab-title { + color: var(--text-normal); } + .is-mobile .workspace-drawer-active-tab-header:hover .workspace-drawer-active-tab-back-icon { + color: var(--text-normal); } + .is-mobile .nav-file-title, + .is-mobile .nav-folder-title, + .is-mobile .outline, + .is-mobile .tree-item-self, + .is-mobile .tag-container, + .is-mobile .tag-pane-tag { + font-size: var(--font-adaptive-small); + line-height: 1.5; + margin-bottom: 4px; } + .is-mobile .backlink-pane > .tree-item-self, + .is-mobile .outgoing-link-pane > .tree-item-self { + font-size: var(--font-adaptive-smallest); } + .is-mobile .tree-item-flair { + font-size: var(--font-adaptive-small); } + .is-mobile .nav-files-container { + padding: 5px 5px 5px 5px; } + .is-mobile .search-result-container { + padding-bottom: 20px; } + .is-mobile .search-result-file-match-replace-button { + background-color: var(--background-tertiary); + color: var(--text-normal); } + .is-mobile .search-result-file-matches, + .is-mobile .search-result-file-title { + font-size: var(--font-adaptive-small); } + .is-mobile .cm-editor .cm-foldGutter .cm-gutterElement { + cursor: var(--cursor); } + .is-mobile .cm-editor .cm-foldPlaceholder { + background: transparent; + border-color: transparent; } + .is-mobile .empty-state-action { + border-radius: var(--radius-m); + font-size: var(--font-adaptive-small); } + .is-mobile .workspace-drawer-header { + padding: 20px 10px 0 25px; } + .is-mobile .workspace-drawer-header-name { + font-weight: var(--bold-weight); + color: var(--text-normal); + font-size: 1.125em; } + .is-mobile .workspace-drawer-header-info { + color: var(--text-faint); + font-size: var(--font-adaptive-small); + margin-bottom: 0; } + .is-mobile .mod-left .workspace-drawer-header-info, + .is-mobile .is-mobile.hider-status .workspace-drawer-header-info { + display: none; } + .is-mobile .workspace-drawer-active-tab-header { + margin: 2px 12px 2px; + padding: 8px 0 8px 8px; } + .is-mobile .workspace-leaf-content .item-list, + .is-mobile .tag-container, + .is-mobile .backlink-pane { + padding-top: 10px; } + .is-mobile .outgoing-link-pane, + .is-mobile .backlink-pane { + padding-left: 10px; } + +/* Workspace */ +.workspace-drawer.mod-left .workspace-drawer-inner { + padding-left: 0; } + +.is-mobile .side-dock-ribbon { + background: var(--background-secondary); + border-right: 1px solid var(--background-modifier-border); + z-index: 3; + flex-direction: column; + width: 70px; + padding: 15px 0; + margin-right: 0px; } + +body:not(.is-ios).is-mobile .workspace-drawer-ribbon { + padding: 20px 5px; } + +.is-ios .is-pinned .side-dock-ribbon { + padding: 30px 0 20px 0; } + +body.is-mobile.hider-frameless:not(.hider-ribbon) .side-dock-actions { + padding-top: 5px; } + +.is-mobile .side-dock-actions, .is-mobile .side-dock-settings { + flex-direction: column; + border-radius: 15px; } + +.is-mobile .mod-left .workspace-drawer-header, +.is-mobile .mod-left .workspace-drawer-tab-container { + margin-left: 70px; } + +.is-mobile .side-dock-ribbon .side-dock-ribbon-action { + padding: 9px 5px 2px 5px; + margin: 0 12px 4px; + height: 40px; } + +.is-mobile .side-dock-ribbon .side-dock-ribbon-action svg { + width: 22px; + height: 22px; } + +.is-mobile .workspace-drawer-active-tab-container { + z-index: 2; + background-color: var(--background-primary); } + +.is-mobile .side-dock-actions, +.is-mobile .side-dock-settings { + display: flex; + align-content: center; + justify-content: center; + padding: 0; } + +.is-mobile .workspace-drawer.mod-left:not(.is-pinned) { + border-right: none; } + +.is-mobile .nav-buttons-container { + padding: 0 0 10px 15px; } + +/* Inputs */ +.is-mobile input[type='text'] { + font-size: 14px; + height: var(--input-height); } + +.is-mobile .setting-item-control .search-input-container input { + display: inline-block; + width: 100%; + margin-bottom: 0; } + +.is-mobile .search-input-container input, +.is-mobile .search-input-container input:hover, +.is-mobile .search-input-container input:focus, +.is-mobile .search-input-container input[type='text'], +.is-mobile .workspace-leaf-content[data-type='search'] .search-input-container input { + -webkit-appearance: none; + border-radius: 6px; + height: 36px; + padding: 6px 20px 6px 34px; + font-size: 14px; } + +.is-mobile .search-input-container input::placeholder { + font-size: 14px; } + +.is-mobile .workspace-drawer { + border-width: var(--border-width); } + +.is-mobile .workspace-drawer-inner, +.is-mobile .workspace-drawer-active-tab-container { + background-color: var(--background-secondary); } + +.workspace-drawer-active-tab-icon { + display: none; } + +.is-ios .is-pinned .workspace-drawer-ribbon { + padding: 30px 0 20px 0; } + +.is-ios .workspace-drawer.is-pinned .workspace-drawer-header { + padding-top: 26px; } + +.is-mobile .workspace-split.mod-root { + background-color: var(--background-primary); } + +.is-ios .mod-root .workspace-leaf { + padding-top: 20px; } + +.is-ios .mod-root .workspace-split.mod-horizontal .workspace-leaf:not(:first-of-type) { + padding-top: 0; } + +.is-mobile.minimal-focus-mode .view-actions { + opacity: 1; } + +.is-mobile .workspace-drawer-tab-options { + padding-top: 10px; } + +.is-mobile .workspace-drawer-tab-option-item { + -webkit-touch-callout: none; + -webkit-user-select: none; + user-select: none; + margin: 0 10px; + padding: 8px 10px; + border-radius: var(--radius-m); } + +.is-mobile .workspace-drawer-header-icon { + align-self: start; } + +body.is-mobile:not(.minimal-icons-off) .workspace-drawer-header-icon svg, +body.is-mobile:not(.minimal-icons-off) .nav-action-button svg, +body.is-mobile:not(.minimal-icons-off) .view-action svg { + width: 22px; + height: 22px; } + +.is-mobile.hider-search-suggestions .search-input-suggest-button { + display: none; } + +.is-mobile .search-input-clear-button { + right: 6px; } + +.is-mobile .search-input-clear-button:before { + height: 16px; + width: 16px; } + +.is-mobile .view-header-title { + font-size: var(--title-size); } + +.is-mobile .view-header-title:-webkit-autofill:focus { + font-family: var(--font-interface); + color: red; } + +.is-mobile .view-header-icon { + padding: 16px 6px 16px 7px; + margin-left: 4px; } + +.is-mobile .mod-root .view-header-icon, +.is-mobile .mod-left.is-pinned + .mod-root .view-header-icon { + display: none; } + +.is-mobile .view-action { + padding: 5px 5px 4px; } + +.is-mobile .workspace-leaf-content:not([data-type='search']) .nav-buttons-container { + border-bottom: var(--border-width) solid var(--background-modifier-border); } + +.is-mobile .workspace-leaf-content[data-type='search'] .nav-action-button, +.is-mobile .nav-action-button, +.is-mobile .workspace-drawer-header-icon { + padding: 4px 7px 0 !important; + margin: 5px 2px 2px 0; + text-align: center; + height: 32px; + cursor: var(--cursor); } + +.is-mobile .nav-file-title.is-active { + box-shadow: 0 0 0px 2px var(--background-tertiary); } + +.pull-down-action { + top: 0; + left: 0; + right: 0; + width: 100%; + margin: 0 auto; + padding: 50px 0 20px; + text-align: center; + border-radius: 0; + border: none; + box-shadow: 0 5px 200px var(--background-modifier-box-shadow); } + +.pull-out-action { + top: 0; + height: 100vh; + padding: 30px 10px; + background: transparent; + display: flex; + justify-content: center; + align-content: center; + flex-direction: column; } + +.is-mobile .markdown-preview-view pre { + overflow-x: scroll; } + +.is-mobile .view-header-icon .three-horizontal-bars { + opacity: 0; } + +.is-mobile.plugin-sliding-panes .view-header-title { + mask-image: unset; + -webkit-mask-image: unset; } + +.is-mobile.plugin-sliding-panes-rotate-header .view-header-title { + line-height: 1.2; } + +.is-mobile .workspace-drawer-header-name-text { + white-space: nowrap; + margin-right: 10px; } + +/* --------------- */ +/* Phone */ +@media (max-width: 400pt) { + .is-mobile .view-header-icon { + display: none; } + + /* Disable hover backgrounds on phone */ + .is-mobile .view-action:hover, + .is-mobile .nav-action-button:hover, + .side-dock-ribbon .side-dock-ribbon-action:hover, + .is-mobile .workspace-leaf-content[data-type='search'] .nav-action-button.is-active:hover, + .is-mobile .workspace-leaf-content[data-type='backlink'] .nav-action-button.is-active:hover, + .is-mobile .workspace-drawer-tab-option-item:hover, + .is-mobile .workspace-drawer-header-icon:hover { + background: transparent; } + + .is-mobile .mod-left .workspace-drawer-header-icon { + display: none; } + + .is-ios .workspace-drawer .workspace-drawer-header { + padding-top: 45px; } + + .is-ios .mod-root .workspace-leaf { + padding-top: 40px; } + + .is-mobile .mod-right .workspace-drawer-header div:nth-child(2) { + display: none; } + + .is-mobile .workspace .workspace-drawer-backdrop { + margin-top: -40px; + height: calc(100vh + 50px); + z-index: 9; } + + .is-ios .workspace-drawer-ribbon { + padding: 50px 0 30px 0; } + + .is-mobile .view-header-title-container { + margin-left: 0; } + + .is-mobile .view-header-title { + max-width: calc(100vw - 90px); + padding-right: 20px; + padding-left: calc(50% - var(--max-width)/2 + var(--folding-offset)) !important; + font-size: var(--font-settings-title); + letter-spacing: -0.015em; } + + .is-mobile .workspace-drawer-header-name-text { + font-size: var(--font-settings-title); + letter-spacing: -0.015em; } + + .is-mobile .view-header { + border-bottom: var(--border-width) solid var(--background-modifier-border) !important; } + + .is-mobile .installed-plugins-container { + max-width: 100%; + overflow: hidden; } + + .is-mobile .setting-item-info { + flex: 1 1 auto; } + + .is-mobile .kanban-plugin__board-settings-modal .setting-item-control, + .is-mobile .setting-item-control { + flex: 1 0 auto; + margin-right: 0; + min-width: auto; } + + .is-mobile .checkbox-container { + flex: 1 0 40px; + max-width: 40px; } + + .is-mobile .setting-item-description { + word-break: break-word; + white-space: pre-line; } + + .is-mobile .view-action { + padding: 0 4px 0 4px; + height: 22px; } + + .is-mobile .frontmatter-container .tag, + .is-mobile .cm-s-obsidian span.cm-hashtag, + .is-mobile .tag { + font-size: var(--font-adaptive-smaller); } + + .is-mobile .setting-item-control select, + .is-mobile .setting-item-control input, + .is-mobile .setting-item-control button { + margin-bottom: 5px; } + + .is-mobile .setting-item-control input[type="range"] { + margin-bottom: 10px; } } +/* --------------- */ +/* Tablet */ +@media (min-width: 400pt) { + .mod-left:not(.is-pinned) + .mod-root > div:first-of-type .view-header-icon { + opacity: var(--icon-muted); + display: flex; } + + .mod-left:not(.is-pinned) + .mod-root > div:first-of-type .view-header-icon:hover, + .mod-left:not(.is-pinned) + .mod-root .view-header-icon .three-horizontal-bars { + opacity: 1; } + + .mod-left:not(.is-pinned) + .mod-root .view-header-icon:hover { + background-color: var(--background-tertiary); } + + .is-mobile.is-ios .safe-area-top-cover { + background-color: transparent; } + + .is-mobile .view-action { + padding: 5px 6px 4px; } + + .is-mobile .mod-left:not(.is-pinned) + .mod-root .workspace-leaf:first-of-type .view-header-title-container { + max-width: calc(100% - 102px); } + + /* Animations */ + .is-mobile .menu, + .is-mobile .suggestion-container, + .is-mobile .modal, + .is-mobile .prompt { + transition: unset !important; + transform: unset !important; + animation: unset !important; } + + .is-mobile .community-plugin-search .setting-item { + padding-top: 10px; } + + .is-mobile .setting-item:not(.mod-toggle):not(.setting-item-heading) { + flex-direction: row; + align-items: center; } + + .is-mobile button, + .is-mobile .setting-item-control select, + .is-mobile .setting-item-control input, + .is-mobile .setting-item-control button { + width: auto; } + + .is-mobile .workspace-drawer:not(.is-pinned) { + margin: 30px 16px 0; + height: calc(100vh - 48px); + border-radius: 15px; } + + .is-mobile .setting-item:not(.mod-toggle):not(.setting-item-heading) .setting-item-control { + width: auto; + margin-top: 0; } + + .is-mobile .markdown-preview-view ol > li.task-list-item .collapse-indicator, + .is-mobile .markdown-preview-view ul > li.task-list-item .collapse-indicator { + margin-left: -2.5em; + margin-top: 0.1em; } + + .pull-down-action { + width: 400px; + top: 15px; + padding: 15px; + border-radius: 15px; } } +/* iOS style modals */ +:root { + --ios-radius:10px; + --ios-input-radius:8px; + --ios-shadow:0 5px 100px rgba(0,0,0,0.15); + --ios-muted:#8e8e93; } + +.theme-light { + --ios-blue:#007aff; + --ios-red:#ff3c2f; + --ios-bg-translucent:rgba(255,255,255,0.85); + --ios-bg:white; + --ios-border:rgba(0,0,0,0.1); } + +.theme-dark { + --ios-blue:#0b84ff; + --ios-red:#ff453a; + --ios-bg-translucent:rgba(44,44,46,0.85); + --ios-bg:#2c2c2e; + --ios-border:rgba(255,255,255,0.15); } + +.is-ios { + --text-error:#ff453a; + /* + .mod-confirmation .modal { + width:400px; + max-width:95vw; + overflow:visible; + background-color:rgba(0,0,0,0.07); + padding:0; + border-radius:var(--ios-radius); + box-shadow:var(--ios-shadow); + .modal-title { + text-align:center; + display:none; + } + .modal-content { + border-radius:var(--ios-radius) var(--ios-radius) 0 0; + background-color:var(--ios-bg-translucent); + backdrop-filter:blur(2px); + -webkit-backdrop-filter:blur(2px); + font-size:13px; + margin:0; + text-align:center; + color:var(--ios-muted); + padding:15px; + p { + margin-block-start:0; + margin-block-end:0; + } + } + .setting-item { + margin-top: 15px; + border-top: 0; + flex-direction: column; + .setting-item-info { + padding-bottom:5px; + } + .setting-item-control { + margin:0; + flex-direction: column; + button { + backdrop-filter: none; + -webkit-backdrop-filter: none; + background: transparent; + padding: 20px 0 10px; + border-top: 0; + } + } + } + button { + background-color:var(--ios-bg-translucent); + backdrop-filter:blur(2px); + -webkit-backdrop-filter:blur(2px); + margin:0; + border:none; + height:auto; + padding:28px 0; + line-height:0; + box-shadow:none; + color:var(--ios-blue); + font-weight:400; + border-radius:0; + font-size:18px; + border-top:1px solid var(--ios-border); + } + button:hover { + background-color:transparent; + border:none; + box-shadow:none; + border-top:1px solid var(--ios-border); + } + .modal-button-container { + gap:0; + } + .modal-button-container>.mod-warning:nth-last-child(3), + button.mod-warning { + border-top:1px solid var(--ios-border); + background-color:var(--ios-bg-translucent); + backdrop-filter:blur(2px); + -webkit-backdrop-filter:blur(2px); + color:var(--ios-red); + font-weight:400; + text-decoration:none; + } + .modal-button-container>button:last-child { + border-top:none; + margin-top:10px; + font-weight:600; + border-radius:var(--ios-radius); + background-color:var(--ios-bg); + } + .modal-button-container>button:nth-last-child(2), + .modal-button-container>.mod-warning:nth-last-child(2) { + border-bottom-left-radius:var(--ios-radius); + border-bottom-right-radius:var(--ios-radius); + } + .modal-button-container>button:last-child:hover { + background-color:var(--ios-bg-translucent); + } + } */ } + .is-ios .search-input-container input, + .is-ios .workspace-leaf-content[data-type='search'] .search-input-container input, + .is-ios .document-search-container input[type='text'] { + border-radius: var(--ios-input-radius); + border: 0px; + background-color: var(--background-tertiary); } + .is-ios .search-input-container input:active, .is-ios .search-input-container input:hover, .is-ios .search-input-container input:focus, + .is-ios .workspace-leaf-content[data-type='search'] .search-input-container input:active, + .is-ios .workspace-leaf-content[data-type='search'] .search-input-container input:hover, + .is-ios .workspace-leaf-content[data-type='search'] .search-input-container input:focus, + .is-ios .document-search-container input[type='text']:active, + .is-ios .document-search-container input[type='text']:hover, + .is-ios .document-search-container input[type='text']:focus { + border-radius: var(--ios-input-radius); + border: 0px; + background-color: var(--background-tertiary); } + .is-ios .search-input-container input::placeholder, + .is-ios .workspace-leaf-content[data-type='search'] .search-input-container input::placeholder, + .is-ios .document-search-container input[type='text']::placeholder { + color: var(--text-muted); } + +/* iPad tablet */ +@media (min-width: 400pt) { + .is-ios .mobile-toolbar { + height: 70px; } + .is-ios .mobile-toolbar-options-container { + margin: 0 auto; + display: inline-flex; + width: auto; } } +.mobile-toolbar-off .mobile-toolbar { + display: none; } + +.mobile-toolbar { + width: 100%; + display: flex; + overflow: scroll; + background-color: var(--background-primary); + border-top: 1px solid var(--background-modifier-border); } + +@media (min-width: 400pt) { + .mobile-toolbar-option { + border-radius: 8px; + margin: 6px 0; } + + .mobile-toolbar-option:hover { + background-color: var(--background-tertiary); } } +/* Core plugins */ +/* Backlink pane */ +.outgoing-link-pane, +.backlink-pane { + padding-bottom: 30px; } + +.outgoing-link-pane .search-result-container, +.backlink-pane .search-result-container { + padding: 5px 5px 5px 5px; + margin-left: 0; } + +.outgoing-link-pane .search-result-file-title, +.backlink-pane .search-result-file-title { + padding-left: 15px; } + +.outgoing-link-pane .tree-item-icon, +.outgoing-link-pane > .tree-item-self .collapse-icon, +.backlink-pane > .tree-item-self .collapse-icon { + display: none; } + +.tree-item-self.outgoing-link-item { + padding: 0; + margin-left: 5px; } + +.outgoing-link-pane > .tree-item-self:hover, +.outgoing-link-pane > .tree-item-self, +.backlink-pane > .tree-item-self:hover, +.backlink-pane > .tree-item-self { + padding-left: 15px; + color: var(--text-muted); + text-transform: uppercase; + letter-spacing: 0.05em; + font-size: var(--font-adaptive-smallest); + font-weight: 500; + padding: 5px 7px 5px 10px; + background: transparent; } + +.outgoing-link-pane > .tree-item-self.is-collapsed, +.backlink-pane > .tree-item-self.is-collapsed { + color: var(--text-faint); } + +.outgoing-link-pane .search-result-file-match { + padding: 5px 0; + border: 0; } + +.outgoing-link-pane .search-result-file-match-destination-file { + background: transparent; } + +.search-result-file-match:hover .search-result-file-match-destination-file:hover { + background: transparent; + color: var(--text-normal); } + +/* Graphs */ +.theme-dark, +.theme-light { + --node:var(--text-muted); + --node-focused:var(--text-accent); + --node-tag:var(--red); + --node-attachment:var(--yellow); + --node-unresolved:var(--text-faint); } + +/* Fill color for nodes */ +.graph-view.color-fill { + color: var(--node); } + +/* Fill color for current local node */ +.graph-view.color-fill-focused { + color: var(--node-focused); } + +/* Fill color for nodes on hover */ +.graph-view.color-fill-highlight { + color: var(--node-focused); } + +/* Stroke color for nodes */ +.graph-view.color-circle { + color: var(--node-focused); } + +/* Line color */ +.graph-view.color-line { + color: var(--background-modifier-border); } + +/* Line color on hover */ +.graph-view.color-line-highlight { + color: var(--node-focused); } + +/* Text color */ +.graph-view.color-text { + color: var(--text-normal); } + +/* Tag nodes */ +.theme-dark .graph-view.color-fill-tag, +.theme-light .graph-view.color-fill-tag { + color: var(--node-tag); } + +.theme-dark .graph-view.color-fill-attachment, +.theme-light .graph-view.color-fill-attachment { + color: var(--node-attachment); } + +.theme-dark .graph-view.color-fill-unresolved, +.theme-light .graph-view.color-fill-unresolved { + color: var(--node-unresolved); } + +/* Full bleed (takes up full height) */ +body:not(.is-mobile):not(.plugin-sliding-panes-rotate-header) .workspace-split.mod-root .workspace-leaf-content[data-type='localgraph'] .view-header, +body:not(.is-mobile):not(.plugin-sliding-panes-rotate-header) .workspace-split.mod-root .workspace-leaf-content[data-type='graph'] .view-header { + position: fixed; + background: transparent !important; + width: 100%; } + +body:not(.is-mobile):not(.plugin-sliding-panes-rotate-header) .workspace-leaf-content[data-type='localgraph'] .view-content, +body:not(.is-mobile):not(.plugin-sliding-panes-rotate-header) .workspace-leaf-content[data-type='graph'] .view-content { + height: 100%; } + +body:not(.plugin-sliding-panes-rotate-header) .workspace-leaf-content[data-type='localgraph'] .view-header-title, +body:not(.plugin-sliding-panes-rotate-header) .workspace-leaf-content[data-type='graph'] .view-header-title { + display: none; } + +body:not(.is-mobile):not(.plugin-sliding-panes-rotate-header) .workspace-leaf-content[data-type='localgraph'] .view-actions, +body:not(.is-mobile):not(.plugin-sliding-panes-rotate-header) .workspace-leaf-content[data-type='graph'] .view-actions { + background: transparent; } + +.mod-root .workspace-leaf-content[data-type='localgraph'] .graph-controls, +.mod-root .workspace-leaf-content[data-type='graph'] .graph-controls { + top: 32px; } + +/* Graph controls */ +.graph-controls.is-close { + padding: 6px; + left: 0; + top: 0; } + +.graph-controls-button { + cursor: var(--cursor); } + +.graph-control-section .tree-item-children { + padding-bottom: 15px; } + +.graph-control-section-header { + font-weight: 500; + text-transform: uppercase; + letter-spacing: 0.05em; + font-size: var(--font-adaptive-smallest); + color: var(--text-muted); } + +.graph-control-section-header:hover { + color: var(--text-normal); } + +.graph-controls .search-input-container { + width: 100%; } + +.setting-item.mod-search-setting.has-term-changed .graph-control-search-button, +.graph-controls .graph-control-search-button { + display: none; } + +.graph-controls .setting-item { + padding: 4px 0 0 0; } + +.graph-controls .setting-item-name { + font-size: var(--font-adaptive-small); } + +.graph-controls { + background: var(--background-secondary); + border: 1px solid var(--background-modifier-border); + min-width: 240px; + left: 6px; + margin-top: 6px; + margin-bottom: 0; + padding: 10px 12px 10px 2px; + border-radius: var(--radius-m); } + +.graph-controls input[type='text'], .graph-controls input[type='range'] { + font-size: var(--font-adaptive-small); } + +.graph-controls .mod-cta { + width: 100%; + font-size: var(--font-adaptive-small); + padding: 5px; + margin: 0; } + +.graph-controls-button.mod-animate { + margin-top: 5px; } + +.mod-left-split .graph-controls { + background: var(--background-secondary); } + +.local-graph-jumps-slider-container, +.workspace-split.mod-left-split .local-graph-jumps-slider-container, +.workspace-split.mod-right-split .local-graph-jumps-slider-container, +.workspace-fake-target-overlay .local-graph-jumps-slider-container { + background: transparent; + opacity: 0.6; + padding: 0; + left: 12px; + transition: opacity 0.2s linear; + height: auto; } + +.mod-root .local-graph-jumps-slider-container { + right: 0; + left: 0; + width: var(--line-width-adaptive); + max-width: var(--max-width); + margin: 0 auto; + top: 30px; } + +.workspace-split.mod-left-split .local-graph-jumps-slider-container:hover, +.workspace-split.mod-right-split .local-graph-jumps-slider-container:hover, +.workspace-fake-target-overlay .local-graph-jumps-slider-container:hover, +.local-graph-jumps-slider-container:hover { + opacity: 0.8; + transition: opacity 0.2s linear; } + +/* Outline */ +.outline { + padding: 15px 10px 20px 0; + font-size: var(--font-adaptive-small); } + +.outline .pane-empty { + font-size: var(--font-adaptive-small); + color: var(--text-faint); + padding: 0 0 0 15px; + width: 100%; } + +.outline .tree-item-self { + cursor: var(--cursor); + line-height: 1.4; + margin-bottom: 4px; + font-size: var(--font-adaptive-small); + padding-left: 15px; } + +.tree-item-collapse { + opacity: 1; + left: -5px; + color: var(--text-faint); } + +.outline .tree-item-inner:hover { + color: var(--text-normal); } + +.tree-item-self.is-clickable:hover .tree-item-collapse { + color: var(--text-normal); } + +.outline > .tree-item > .tree-item-self .right-triangle { + opacity: 0; } + +/* Page Preview aka Popovers */ +.theme-dark.minimal-dark-black .popover { + background: var(--bg2); } + +.popover, +.popover.hover-popover { + min-height: 40px; + box-shadow: 0 20px 40px var(--background-modifier-box-shadow); + pointer-events: auto !important; + border: 1px solid var(--background-modifier-border); } + +.popover.hover-popover { + width: 400px; + max-height: 40vh; } + +.popover.hover-popover .markdown-embed { + padding: 0; } + +.popover .markdown-embed-link { + display: none; } + +.popover .markdown-embed .markdown-preview-view { + padding: 10px 20px 30px; } + +.popover.hover-popover .markdown-embed .markdown-embed-content { + max-height: none; } + +.popover.hover-popover.mod-empty { + padding: 20px 20px 20px 20px; + color: var(--text-muted); } + +.popover.hover-popover .markdown-preview-view .table-view-table, +.popover.hover-popover .markdown-embed .markdown-preview-view { + font-size: 1.05em; } + +.popover.hover-popover .markdown-embed h1, +.popover.hover-popover .markdown-embed h2, +.popover.hover-popover .markdown-embed h3, +.popover.hover-popover .markdown-embed h4 { + margin-top: 1rem; } + +/* Prompt */ +/* Used for command palette and quick switcher */ +.prompt { + box-shadow: var(--shadow-m); + padding-bottom: 0; + border: 1px solid var(--modal-border); } + +body:not(.hider-scrollbars) .prompt { + padding-right: 0px; } + +body:not(.hider-scrollbars) .prompt-results { + padding-right: 10px; } + +input.prompt-input { + border: 0; + background: var(--background-primary); + box-shadow: none !important; + padding-left: 10px; + height: 40px; + line-height: 4; + font-size: var(--font-adaptive-normal); } + input.prompt-input:hover { + border: 0; + background: var(--background-primary); + padding-left: 10px; + line-height: 4; } + +.prompt-results { + padding-bottom: 0; } + .prompt-results .suggestion-item:last-child, + .prompt-results .suggestion-empty { + margin-bottom: 10px; } + +.prompt-instructions { + color: var(--text-muted); } + +.prompt-instruction-command { + font-weight: 600; } + +/* +.suggestion-prefix { + font-weight:500; +}*/ +/* In Editor autocomplete */ +.suggestion-container { + box-shadow: 0 5px 40px rgba(0, 0, 0, 0.2); + padding: 0 6px; + border-radius: 8px; + background-color: var(--background-primary); + border: 1px solid var(--background-modifier-border-hover); } + .suggestion-container .suggestion-item { + font-size: calc(var(--font-adaptive-normal) * .9) !important; + cursor: var(--cursor); + padding: 4px 10px 4px 10px; + border-radius: 4px; } + .suggestion-container .suggestion-item:first-child { + margin-top: 6px; } + .suggestion-container .suggestion-item:last-child { + margin-bottom: 6px; } + +.is-mobile .suggestion-container .suggestion-item:first-child { + margin-top: 0; } +.is-mobile .suggestion-container .suggestion-item:last-child { + margin-bottom: 10px; } + +.suggestion-hotkey { + margin-top: 0.25em; } + +.suggestion-flair { + left: auto; + right: 8px; + opacity: 0.25; } + +.prompt-results .suggestion-flair .filled-pin { + display: none; } + +.prompt-results .suggestion-item { + padding: 5px 8px 5px 10px; } + +/* +.prompt .prompt-results { + .suggestion-item { + display:flex; + align-items:center; + .suggestion-prefix { + white-space:pre; + } + .suggestion-content { + white-space:pre; + overflow:hidden; + text-overflow:ellipsis; + flex-grow:1; + padding-right:1em; + } + .suggestion-hotkey { + white-space:pre; + margin-top:0; + } + .suggestion-hotkey:not(:last-child) { + margin:0 5px 0 0; + } + } +} +*/ +.modal-container .suggestion-item.is-selected { + border-radius: var(--radius-m); + background: var(--background-tertiary); } + +.suggestion-item.is-selected { + background: var(--background-tertiary); } + +.suggestion-item, +.suggestion-empty { + font-size: var(--font-adaptive-normal); + cursor: var(--cursor); } + +/* Mobile */ +.is-mobile { + /* Tablet */ + /* Phone */ } + .is-mobile .prompt, + .is-mobile .suggestion-container { + width: 100%; + max-width: 100%; + border: none; + padding: 10px 10px 0 10px; + -webkit-touch-callout: none; + -webkit-user-select: none; + user-select: none; } + .is-mobile .suggestion-container { + left: 0; + right: 0; + margin: 0 auto; + border: none; } + .is-mobile .suggestion-item { + font-size: var(--font-adaptive-normal); + padding-left: 10px; + letter-spacing: 0.001px; } + .is-mobile .prompt-results .suggestion-flair { + display: none; } + .is-mobile input[type='text'].prompt-input, + .is-mobile input[type='text'].prompt-input:hover { + line-height: 2; + padding: 8px; + height: 4.5ex; + font-size: var(--font-adaptive-normal); } + @media (min-width: 400pt) { + .is-mobile .modal-container .prompt { + opacity: 1 !important; } + .is-mobile .prompt { + max-width: 600px; + max-height: 600px; + bottom: auto !important; + border-radius: 15px; + top: 100px !important; } + .is-mobile .suggestion-container { + max-width: 600px; + max-height: 600px; + border-radius: 15px; + bottom: 80px; + border: 1px solid var(--background-modifier-border); } + .is-mobile .modal-container .suggestion-item { + padding: 8px 5px 8px 8px; + border-radius: var(--radius-m); } + .is-mobile .suggestion-flair { + right: 0; + left: auto; + position: absolute; + padding: 10px; } } + @media (max-width: 400pt) { + .is-mobile .suggestion-hotkey { + display: none; } + .is-mobile .suggestion-flair { + right: 0; + left: auto; + position: absolute; + padding: 5px 5px 0 0; } + .is-mobile .suggestion-container { + max-height: 200px; + border-top: 1px solid var(--background-modifier-border); + border-radius: 0; + padding-top: 0; + box-shadow: none; } + .is-mobile .prompt { + border-radius: 0; + border: none; + padding-top: 5px; + padding-bottom: 0; + max-height: calc(100vh - 120px); + top: 120px; } + .is-mobile .suggestion-container .suggestion { + padding-top: 10px; } } + +/* Publish */ +.modal.mod-publish { + max-width: 600px; + padding-left: 0; + padding-right: 0; + padding-bottom: 0; } + +.modal.mod-publish .modal-title { + padding-left: 20px; + padding-bottom: 10px; } + +.mod-publish .modal-content { + padding-left: 20px; + padding-right: 20px; } + +.mod-publish p { + font-size: var(--font-small); } + +.mod-publish .tree-item-flair { + display: unset; } + +.file-tree .mod-new .tree-item-flair, +.file-tree .mod-deleted .tree-item-flair, +.file-tree .mod-to-delete .tree-item-flair, +.file-tree .mod-changed .tree-item-flair { + background: transparent; } + +.file-tree .mod-deleted .tree-item-flair, +.file-tree .mod-to-delete .tree-item-flair { + color: var(--pink); } + +.file-tree .mod-new .tree-item-flair { + color: var(--green); } + +.file-tree .mod-changed .tree-item-flair { + color: var(--yellow); } + +.mod-publish .button-container, +.modal.mod-publish .modal-button-container { + margin-top: 0px; + padding: 10px; + border-top: 1px solid var(--background-modifier-border); + bottom: 0px; + background-color: var(--background-primary); + position: absolute; + width: 100%; + margin-left: -20px; + text-align: center; } + +.publish-changes-info { + padding: 0 0 15px; + margin-bottom: 0; + border-bottom: 1px solid var(--background-modifier-border); } + +.modal.mod-publish .modal-content .publish-sections-container { + max-height: none; + height: auto; + padding: 10px 20px 30px 0; + margin-top: 10px; + margin-right: -20px; + margin-bottom: 80px; } + +.publish-site-settings-container { + max-height: none; + height: auto; + margin-right: -20px; + margin-bottom: 80px; + overflow-x: hidden; } + +.publish-section-header { + padding-bottom: 15px; + border-width: 1px; } + +.password-item { + padding-left: 0; + padding-right: 0; } + +.publish-section-header-text { + font-weight: 600; + color: var(--text-normal); + cursor: var(--cursor); } + +.publish-section-header-text, +.publish-section-header-toggle-collapsed-button, +.publish-section-header-action, +.file-tree-item-header { + cursor: var(--cursor); } + +.publish-section-header-text:hover, +.publish-section-header-toggle-collapsed-button:hover, +.publish-section-header-action:hover { + color: var(--text-normal); + cursor: var(--cursor); } + +.mod-publish .u-pop { + color: var(--text-normal); } + +.publish-section-header-toggle-collapsed-button { + padding: 7px 0 0 3px; + width: 18px; } + +.mod-publish .file-tree-item { + margin-left: 20px; } + +.mod-publish .file-tree-item { + padding: 0; + margin-bottom: 2px; + font-size: var(--font-small); } + +.mod-publish .file-tree-item-checkbox { + filter: hue-rotate(0); } + +.mod-publish .file-tree-item.mod-deleted .flair, +.mod-publish .file-tree-item.mod-to-delete .flair { + background: transparent; + color: #ff3c00; + font-weight: 500; } + +.mod-publish .file-tree-item.mod-new .flair { + background: transparent; + font-weight: 500; + color: #13c152; } + +.mod-publish .site-list-item { + padding-left: 0; + padding-right: 0; } + +.is-mobile { + /* Mobile publish */ + /* Phone */ } + .is-mobile .mod-publish .modal-content { + display: unset; + padding: 10px 10px 10px; + margin-bottom: 120px; + overflow-x: hidden; } + .is-mobile .mod-publish .button-container, + .is-mobile .modal.mod-publish .modal-button-container { + padding: 10px 15px 30px; + margin-left: 0px; + left: 0; } + .is-mobile .modal.mod-publish .modal-title { + padding: 10px 20px; + margin: 0 -10px; + border-bottom: 1px solid var(--background-modifier-border); } + .is-mobile .publish-site-settings-container { + margin-right: 0; + padding: 0; } + .is-mobile .modal.mod-publish .modal-content .publish-sections-container { + margin-right: 0; + padding-right: 0; } + @media (max-width: 400pt) { + .is-mobile .publish-section-header, + .is-mobile .publish-changes-info { + flex-wrap: wrap; + border: none; } + .is-mobile .publish-changes-info .publish-changes-add-linked-btn { + flex-basis: 100%; + margin-top: 10px; } + .is-mobile .publish-section-header-text { + flex-basis: 100%; + margin-bottom: 10px; + margin-left: 20px; + margin-top: -8px; } + .is-mobile .publish-section { + background: var(--background-secondary); + border-radius: 10px; + padding: 12px 12px 1px; } + .is-mobile .publish-changes-switch-site { + flex-grow: 0; + margin-right: 10px; } } + +/* Search */ +.search-result-container.mod-global-search .search-empty-state { + padding-left: 15px; } + +.search-result-file-match { + cursor: var(--cursor) !important; + width: auto; + left: 0; } + +.search-result-file-match:hover { + background: transparent; } + +.search-result-container:before { + height: 1px; } + +.search-result-file-match-replace-button { + background-color: var(--background-primary); + border: 1px solid var(--background-modifier-border); + color: var(--text-muted); + opacity: 1; + top: auto; + right: 18px; + bottom: 1px; + font-weight: 500; + font-size: var(--font-adaptive-smaller); } + +.search-result-hover-button:hover { + background-color: var(--background-tertiary); + color: var(--text-muted); } + +.search-result-file-match-replace-button:hover { + background-color: var(--background-modifier-border); + color: var(--text-normal); } + +.search-result-container.is-loading:before { + background-color: var(--background-modifier-accent); } + +.search-result { + margin-bottom: 0; } + +.search-result-count { + opacity: 1; + color: var(--text-faint); + padding: 0 0 0 5px; } + +.search-result-file-match:before { + top: 0; } + +.search-result-file-match:not(:first-child) { + margin-top: 0px; } + +.search-result-file-match { + margin-top: 0; + margin-bottom: 0; + padding-top: 6px; + padding-bottom: 5px; } + +.search-result-file-matched-text { + background-color: var(--text-selection); } + +.search-input-container input, +.search-input-container input:hover, +.search-input-container input:focus { + font-size: var(--font-adaptive-small); + padding: 5px 28px 5px 10px; + background-color: var(--background-modifier-form-field); } + +.search-input-container { + width: calc(100% - 20px); + margin: 0 0 8px 10px; } + +.workspace-leaf-content .setting-item { + padding: 5px 0; + border: none; } + +.workspace-leaf-content .setting-item-control { + flex-shrink: 0; + flex: 1; } + +.search-input-clear-button { + background: transparent; + border-radius: 50%; + color: var(--text-muted); + cursor: var(--cursor); + top: 0px; + right: 2px; + bottom: 0px; + line-height: 0; + height: calc(var(--input-height) - 2px); + width: 28px; + margin: auto; + padding: 0 0; + text-align: center; + display: flex; + justify-content: center; + align-items: center; + transition: color 0.2s ease-in-out; } + +.search-input-clear-button:hover { + color: var(--text-normal); + transition: color 0.2s ease-in-out; } + +.search-input-clear-button:active { + color: var(--text-normal); + transition: color 0.2s ease-in-out; } + +.search-input-clear-button:before { + content: ''; + height: 13px; + width: 13px; + display: block; + background-color: currentColor; + -webkit-mask-image: url("data:image/svg+xml,"); + -webkit-mask-repeat: no-repeat; } + +.search-input { + max-width: 100%; + margin-left: 0; + width: 500px; } + +input.search-input:focus { + border-color: var(--background-modifier-border); } + +.workspace-leaf-content[data-type='search'] .search-result-file-matches { + padding-left: 0; } + +.search-empty-state { + font-size: var(--font-adaptive-small); + color: var(--text-faint); + padding-left: 5px; + margin: 0; } + +.search-result-container { + padding: 5px 10px 50px 5px; } + +.search-result-file-title { + line-height: 1.3; + padding: 4px 4px 4px 20px; + vertical-align: middle; + cursor: var(--cursor) !important; } + +.tree-item-inner, +.search-result-file-title { + white-space: nowrap; + overflow: hidden; + text-overflow: ellipsis; } + +.search-result-collapse-indicator { + left: 0px; } + +.search-result-file-match { + padding-right: 15px; } + +.search-result-file-match:before { + height: 0.5px; } + +.search-result-file-matches { + font-size: var(--font-adaptive-smaller); + line-height: 1.3; + margin: 3px 0 8px 0px; + padding: 0 0 2px 0; + color: var(--text-muted); + border: 1px solid var(--background-modifier-border); + background: var(--background-primary); + border-radius: var(--radius-m); } + +.search-result:last-child .search-result-file-matches { + border: 1px solid var(--background-modifier-border); } + +.search-result-hover-button.mod-top { + top: 4px; + right: 4px; } + +.search-result-hover-button.mod-bottom { + bottom: 0px; + right: 4px; } + +.search-info-container { + font-size: var(--font-adaptive-smaller); + color: var(--text-faint); + padding-top: 5px; + padding-bottom: 5px; } + +.search-info-more-matches { + font-size: var(--font-adaptive-smaller); + padding-top: 4px; + padding-bottom: 4px; + color: var(--text-normal); } + +.side-dock-collapsible-section-header-indicator { + display: none; } + +.search-result-file-title:hover { + color: var(--text-normal); + background: transparent; } + +.workspace-leaf-content .search-input, +.workspace-leaf-content .search-input:hover, +.workspace-leaf-content .search-input:focus { + font-size: var(--font-adaptive-small); + padding: 7px 10px; + height: 28px; + border-radius: var(--radius-m); + background: var(--background-primary); + border: 1px solid var(--background-modifier-border); + transition: border-color 0.1s ease-in-out; } + +.workspace-leaf-content .search-input:hover { + border-color: var(--background-modifier-border-hover); + transition: border-color 0.1s ease-in-out; } + +.workspace-leaf-content .search-input:focus { + background: var(--background-primary); + border-color: var(--background-modifier-border-focus); + transition: all 0.1s ease-in-out; } + +.search-input-container input::placeholder { + color: var(--text-faint); + font-size: var(--font-adaptive-small); } + +.workspace-split.mod-root .workspace-split.mod-vertical .workspace-leaf-content { + padding-right: 0; } + +.workspace-split.mod-horizontal.mod-right-split { + width: 0; } + +.workspace-split.mod-vertical > .workspace-leaf { + padding-right: 1px; } + +.workspace-leaf-content[data-type=starred] .item-list { + padding-top: 5px; } + +.workspace-leaf-content .view-content { + padding: 0; } + +.workspace-split.mod-right-split .view-content { + padding: 0; + background-color: var(--background-primary); } + +/* Sync */ +/* Sync Log */ +.modal.mod-sync-log { + padding: 20px 0 0 0; } + +.modal.mod-sync-log .modal-title { + padding: 0 20px; } + +.modal.mod-sync-log .modal-content { + padding: 0px; + display: flex; + flex-direction: column; } + +.modal.mod-sync-log .modal-button-container { + border-top: 1px solid var(--background-modifier-border); + padding: 15px; + background-color: var(--background-primary); + margin: 0; } + +.modal.mod-sync-log .sync-log-container { + padding: 16px 20px; + background-color: var(--background-secondary); + flex-grow: 1; + font-size: var(--font-adaptive-small); } + +.sync-log-container .list-item { + padding-left: 0; } + +.modal.mod-sync-log .setting-item.mod-toggle { + padding: 20px; } + +.sync-history-content { + font-size: var(--font-adaptive-small); + border: none; + padding: 20px 40px 20px 20px; + border-radius: 0; } + +body .sync-history-content-container textarea.sync-history-content:active, +body .sync-history-content-container textarea.sync-history-content:focus { + box-shadow: none; } + +/* Sync history */ +.modal.mod-sync-history .modal-content { + padding: 0; } + +.sync-history-content-empty { + padding: 5px 20px; + color: var(--text-muted); + font-size: var(--font-adaptive-small); } + +.sync-history-content-container { + padding: 0; + height: auto; + border-left: 1px solid var(--background-modifier-border); + background-color: var(--background-primary); } + +.sync-history-content-buttons.u-center-text { + text-align: center; + padding: 10px; + margin: 0; + border-top: 1px solid var(--background-modifier-border); } + +.sync-history-content-container .modal-button-container { + margin: 0; + padding: 10px 5px; + border-top: 1px solid var(--background-modifier-border); + background-color: var(--background-primary); + text-align: center; } + +.sync-history-list { + min-width: 220px; } + +.sync-history-list-container { + min-width: 220px; + flex-basis: 230px; + max-height: none; + overflow-y: scroll; + background-color: var(--background-secondary); } + +.sync-history-list { + padding: 10px 10px 0 10px; + overflow: unset; + background-color: var(--background-secondary); } + +.sync-history-list .search-input-container { + width: 100%; + margin: 0; } + +.sync-history-load-more-button { + font-size: var(--font-adaptive-small); + cursor: var(--cursor); + margin: 0px 10px 10px; + border-radius: var(--radius-m); } + +.sync-history-load-more-button:hover { + background-color: var(--background-tertiary); } + +.sync-history-list-item { + border-radius: var(--radius-m); + padding: 4px 8px; + margin-bottom: 4px; + font-size: var(--font-adaptive-small); + cursor: var(--cursor); } + +.sync-history-list-item.is-active, .sync-history-list-item:hover { + background-color: var(--background-tertiary); } + +/* Mobile */ +.is-mobile .sync-status-icon { + margin-top: 2px; } +.is-mobile .sync-history-list { + padding: 10px; + background-color: var(--background-primary); } +.is-mobile .sync-history-list-item { + font-size: var(--font-adaptive-small); + padding: 8px 10px; } +.is-mobile .sync-history-content-container .modal-button-container { + padding: 5px 10px 30px 10px; } +.is-mobile .sync-history-content { + outline: none; + -webkit-appearance: none; + border: 0; + background-color: var(--background-secondary); } +.is-mobile .modal.mod-sync-log .mod-toggle, .is-mobile .modal.mod-sync-log .modal-button-container { + flex: 0; } + +/* --------------- */ +/* Phone */ +@media (max-width: 400pt) { + .is-mobile .modal.mod-sync-log { + width: 100vw; + height: 100vh; + max-height: calc(100vh - 32px); + box-shadow: 0 -32px 0 32px var(--background-primary); + bottom: 0; + padding-bottom: 10px; } } +/* Community plugins */ +/* Banner plugin */ +/* +.markdown-source-view.mod-cm6 .cm-line.has-banner { + width:100% !important; + max-width:100% !important; + transform:none !important; + + .cm-fold-indicator, + .cm-def.cm-hmd-frontmatter { + margin-left:max( + calc(50% + var(--folding-offset) - var(--line-width-adaptive)/2), + calc(50% - var(--max-width)/2) + var(--folding-offset)) !important; + } + .obsidian-banner-icon { + width:calc(var(--line-width-adaptive) - var(--folding-offset)); + max-width:var(--max-width); + margin-left:auto; + margin-right:auto; + transform:translateX(calc(var(--folding-offset)/2)); + } +} */ +.obsidian-banner.solid { + border-bottom: var(--border-width) solid var(--background-divider); } + +.contextual-typography .markdown-preview-view div.has-banner-icon.obsidian-banner-wrapper { + overflow: visible; } + +.theme-dark .markdown-preview-view img.emoji { + opacity: 1; } + +/* Breadcrumbs plugin +body .BC-trail { + border-width: 0 0 1px 0; + border-radius: 0; +} +*/ +/* Buttons plugin */ +body.theme-dark .button-default, +body.theme-light .button-default { + border: none; + box-shadow: none; + height: var(--input-height); + background: var(--background-tertiary); + color: var(--text-normal); + font-size: revert; + font-weight: 500; + transform: none; + transition: all 0.1s linear; + padding: 0 20px; } + +body.theme-dark .button-default:hover, +body.theme-light .button-default:hover { + border: none; + background: var(--background-modifier-border-hover); + box-shadow: none; + transform: none; + transition: all 0.1s linear; } + +body.theme-light .button-default:focus, +body.theme-light .button-default:active, +body.theme-dark .button-default:focus, +body.theme-dark .button-default:active { + box-shadow: none; } + +body .button-default.blue { + background-color: var(--blue) !important; } + +.button-default.red { + background-color: var(--red) !important; } + +.button-default.green { + background-color: var(--green) !important; } + +.button-default.yellow { + background-color: var(--yellow) !important; } + +.button-default.purple { + background-color: var(--purple) !important; } + +/* Calendar plugin */ +.workspace-leaf-content[data-type='calendar'] .view-content { + padding: 5px 0 0 0; } + +#calendar-container { + padding: 0 15px 5px; + --color-background-day-empty:var(--background-secondary-alt); + --color-background-day-active:var(--background-tertiary); + --color-background-day-hover:var(--background-tertiary); + --color-dot:var(--text-faint); + --color-text-title:var(--text-normal); + --color-text-heading:var(--text-muted); + --color-text-day:var(--text-normal); + --color-text-today:var(--text-normal); + --color-arrow:var(--text-faint); + --color-background-day-empty:transparent; } + +#calendar-container .table { + border-collapse: separate; + table-layout: fixed; } + +#calendar-container h2 { + font-weight: 400; + font-size: var(--h2); } + +.mod-root #calendar-container { + width: var(--line-width-adaptive); + max-width: var(--max-width); + margin: 0 auto; + padding: 0; } + +#calendar-container .arrow { + cursor: var(--cursor); + width: 22px; + border-radius: 4px; + padding: 3px 7px; } + +#calendar-container .arrow svg { + width: 12px; + height: 12px; + color: var(--text-faint); + opacity: 0.7; } + +#calendar-container .arrow:hover { + fill: var(--text-muted); + color: var(--text-muted); + background-color: var(--background-tertiary); } + +#calendar-container .arrow:hover svg { + color: var(--text-muted); + opacity: 1; } + +#calendar-container tr th { + padding: 2px 0 4px; + font-weight: 500; + letter-spacing: 0.1em; + font-size: var(--font-adaptive-smallest); } + +#calendar-container tr td { + padding: 2px 0 0 0; + border-radius: var(--radius-m); + cursor: var(--cursor); + border: 1px solid transparent; + transition: none; } + +#calendar-container .nav { + padding: 0; + margin: 10px 5px 10px 5px; } + +#calendar-container .dot { + margin: 0; } + +#calendar-container .year, +#calendar-container .month, +#calendar-container .title { + font-size: var(--font-adaptive-normal); + font-weight: 400; + color: var(--text-normal); } + +#calendar-container .today.active, +#calendar-container .today { + color: var(--text-accent); + font-weight: 600; } + +#calendar-container .today.active .dot, +#calendar-container .today .dot { + fill: var(--text-accent); } + +#calendar-container .active .task { + stroke: var(--text-faint); } + +#calendar-container .active { + color: var(--text-normal); } + +#calendar-container .reset-button { + text-transform: none; + letter-spacing: 0; + font-size: var(--font-adaptive-small); + font-weight: 500; + color: var(--text-muted); + border-radius: 4px; + margin: 0; + padding: 2px 8px; } + +#calendar-container .reset-button:hover { + color: var(--text-normal); + background-color: var(--background-tertiary); } + +#calendar-container .reset-button, +#calendar-container .day { + cursor: var(--cursor); } + +#calendar-container .day.adjacent-month { + color: var(--text-faint); + opacity: 1; } + +#calendar-container .day { + padding: 2px 4px 4px; + font-size: calc(var(--font-adaptive-normal) - 2px); } + +#calendar-container .active, +#calendar-container .active.today, +#calendar-container .week-num:hover, +#calendar-container .day:hover { + background-color: var(--color-background-day-active); } + +#calendar-container .active .dot { + fill: var(--text-faint); } + +#calendar-container .active .task { + stroke: var(--text-faint); } + +/* Charts */ +.block-language-chart canvas, +.block-language-dataviewjs canvas { + margin: 1em 0; } + +.theme-light, +.theme-dark { + --chart-color-1:var(--blue); + --chart-color-2:var(--red); + --chart-color-3:var(--yellow); + --chart-color-4:var(--green); + --chart-color-5:var(--orange); + --chart-color-6:var(--purple); + --chart-color-7:var(--cyan); + --chart-color-8:var(--pink); } + +/* Checklist plugin */ +.checklist-plugin-main .group .classic, +.checklist-plugin-main .group .compact, +.checklist-plugin-main .group svg, +.checklist-plugin-main .group .page { + cursor: var(--cursor); } + +.workspace .view-content .checklist-plugin-main { + padding: 10px 10px 15px 15px; + --todoList-togglePadding--compact:2px; + --todoList-listItemMargin--compact:2px; } + +.checklist-plugin-main .title { + font-weight: 400; + color: var(--text-muted); + font-size: var(--font-adaptive-small); } + +.checklist-plugin-main .group svg { + fill: var(--text-faint); } + +.checklist-plugin-main .group svg:hover { + fill: var(--text-normal); } + +.checklist-plugin-main .group .title:hover { + color: var(--text-normal); } + +.checklist-plugin-main .group:not(:last-child) { + border-bottom: 1px solid var(--background-modifier-border); } + +.checklist-plugin-main .group { + padding: 0 0 2px 0; } + +.checklist-plugin-main .group .classic:last-child, +.checklist-plugin-main .group .compact:last-child { + margin-bottom: 10px; } + +.checklist-plugin-main .group .classic, +.checklist-plugin-main .group .compact { + font-size: var(--font-adaptive-small); } + +.checklist-plugin-main .group .classic, +.checklist-plugin-main .group .compact { + background: transparent; + border-radius: 0; + margin: 1px auto; + padding: 0; } + +.checklist-plugin-main .group .classic .content { + padding: 0; } + +.checklist-plugin-main .group .classic:hover, +.checklist-plugin-main .group .compact:hover { + background: transparent; } + +.markdown-preview-view.checklist-plugin-main ul > li:not(.task-list-item)::before { + display: none; } + +.checklist-plugin-main .group .compact > .toggle .checked { + background: var(--text-accent); + top: -1px; + left: -1px; + height: 18px; + width: 18px; } + +.checklist-plugin-main .compact .toggle:hover { + opacity: 1 !important; } + +.checklist-plugin-main .group .count { + font-size: var(--font-adaptive-smaller); + padding: 0; + background: transparent; + font-weight: 400; + color: var(--text-faint); } + +.checklist-plugin-main .group .group-header:hover .count { + color: var(--text-muted); } + +.checklist-plugin-main .group .checkbox { + border: 1px solid var(--background-modifier-border-hover); + min-height: 18px; + min-width: 18px; + height: 18px; + width: 18px; } + +.checklist-plugin-main .group .checkbox:hover { + border: 1px solid var(--background-modifier-border-focus); } + +.checklist-plugin-main button:active, +.checklist-plugin-main button:focus, +.checklist-plugin-main button:hover { + box-shadow: none !important; } + +.checklist-plugin-main button.collapse { + padding: 0; } + +body:not(.is-mobile) .checklist-plugin-main button.collapse svg { + width: 18px; + height: 18px; } + +/* Checklist plugin mobile */ +.is-mobile .checklist-plugin-main .group-header .title { + flex-grow: 1; + flex-shrink: 0; } + +.is-mobile .checklist-plugin-main button { + width: auto; } + +.is-mobile .checklist-plugin-main.markdown-preview-view ul { + padding-inline-start: 0; } + +.is-mobile .workspace .view-content .checklist-plugin-main { + padding-bottom: 50px; } + +/* cMenu plugin */ +body #cMenuModalBar { + box-shadow: 0px 2px 20px var(--shadow-color); } + +body #cMenuModalBar .cMenuCommandItem { + cursor: var(--cursor); } + +body #cMenuModalBar button.cMenuCommandItem:hover { + background-color: var(--background-tertiary); } + +/* Contextual Typography */ +.el-hr hr { + margin: 1rem 0; } + +.el-p + .el-h1, +.el-p + .el-h2 { + margin-top: 0.75rem; } + +.el-hr + .el-h1, +.el-hr + .el-h2, +.el-h1 + .el-h1, +.el-h1 + .el-h2, +.el-h2 + .el-h2 { + margin-top: 0rem; } + +.el-ol + .el-lang-dataview, +.el-ul + .el-lang-dataview, +.el-p:not(.el-lang-dataview) + .el-lang-dataview, +.el-ol + .el-lang-dataviewjs, +.el-ul + .el-lang-dataviewjs, +.el-p:not(.el-lang-dataviewjs) + .el-lang-dataviewjs, +.el-ol + .el-table, +.el-ul + .el-table, +.el-p + .el-table, +.el-lang-dataviewjs + .el-p, +.el-lang-dataview + .el-p { + margin-top: var(--spacing-p); } + +.el-div + .el-h1, +.el-pre + .el-h1, +.el-lang-leaflet, +.el-lang-leaflet + *, +.el-iframe + .el-p, +.el-p + .el-iframe, +.el-p:not(.el-embed-image) + .el-embed-image, +.el-embed-image + .el-p:not(.el-embed-image) { + margin-top: 1rem; } + +/* Dataview plugin */ +/*body .table-view-table > thead > tr > th, +.markdown-preview-view .table-view-table { + font-size:calc(var(--font-adaptive-normal) - 1px); +}*/ +body .table-view-table > thead > tr > th, +.markdown-preview-view .table-view-table > thead > tr > th { + font-weight: 400; + font-size: var(--table-font-size); + color: var(--text-muted); + border-bottom: 1px solid var(--background-modifier-border); + cursor: var(--cursor); } + +table.dataview ul.dataview-ul { + list-style: none; + padding-inline-start: 0; + margin-block-start: 0em !important; + margin-block-end: 0em !important; } + +.markdown-source-view.mod-cm6 .table-view-table > tbody > tr > td, +.markdown-preview-view .table-view-table > tbody > tr > td { + max-width: var(--max-col-width); } + +body .dataview.small-text { + color: var(--text-faint); } + +/* Remove hover effect */ +body .dataview.task-list-item:hover, +body .dataview.task-list-basic-item:hover, +body .table-view-table > tbody > tr:hover { + background-color: transparent; + box-shadow: none; } + +body .dataview-error { + margin-top: 16px; + background-color: transparent; } + +.markdown-source-view.mod-cm6 .cm-content .dataview.dataview-error, +.dataview.dataview-error { + color: var(--text-muted); } + +/* New error box as of 2022-05 */ +body div.dataview-error-box { + min-height: 0; + border: none; + background-color: transparent; + font-size: var(--table-font-size); + border-radius: var(--radius-m); + padding: 15px 0; } + body div.dataview-error-box p { + margin-block-start: 0; + margin-block-end: 0; + color: var(--text-faint); } + +.markdown-source-view div.dataview-error-box { + margin-top: 15px; } + +/* Trim columns feature */ +.trim-cols .markdown-source-view.mod-cm6 .table-view-table > tbody > tr > td, +.trim-cols .markdown-preview-view .table-view-table > tbody > tr > td, +.trim-cols .markdown-source-view.mod-cm6 .table-view-table > thead > tr > th { + white-space: nowrap; + text-overflow: ellipsis; + overflow: hidden; } + +/* Lists */ +ul .dataview .task-list-item:hover, +ul .dataview .task-list-basic-item:hover { + background-color: transparent; + box-shadow: none; } + +body .dataview.result-group { + padding-left: 0; } + +/* Inline fields */ +body .dataview.inline-field-key, +body .dataview.inline-field-value, +body .dataview .inline-field-standalone-value { + font-family: var(--font-text); + font-size: calc(var(--font-adaptive-normal) - 2px); + background: transparent; + color: var(--text-muted); } + +body .dataview.inline-field-key { + padding: 0; } + +body .dataview .inline-field-standalone-value { + padding: 0; } + +body .dataview.inline-field-key::after { + margin-left: 3px; + content: "|"; + color: var(--background-modifier-border); } + +body .dataview.inline-field-value { + padding: 0 1px 0 3px; } + +/* Calendar */ +.markdown-preview-view .block-language-dataview table.calendar th { + border: none; + cursor: default; + background-image: none; } + +.markdown-preview-view .block-language-dataview table.calendar .day { + font-size: var(--font-adaptive-small); } + +/* Dictionary plugin */ +.workspace-leaf-content .view-content.dictionary-view-content { + padding: 0; } + +div[data-type="dictionary-view"] .contents { + padding-bottom: 2rem; } + +div[data-type="dictionary-view"] .results > .container { + background-color: transparent; + margin-top: 0; + max-width: none; + padding: 0 10px; } + +div[data-type="dictionary-view"] .error, +div[data-type="dictionary-view"] .errorDescription { + text-align: left; + font-size: var(--font-adaptive-small); + padding: 10px 12px 0; + margin: 0; } + +div[data-type="dictionary-view"] .results > .container h3 { + text-transform: uppercase; + letter-spacing: 0.05em; + color: var(--text-muted); + font-size: var(--font-adaptive-smallest); + font-weight: 500; + padding: 5px 7px 0px 2px; + margin-bottom: 6px; } + +div[data-type="dictionary-view"] .container .main { + border-radius: 0; + background-color: transparent; + font-size: var(--font-adaptive-smaller); + line-height: 1.3; + color: var(--text-muted); + padding: 5px 0 0; } + +div[data-type="dictionary-view"] .main .definition { + padding: 10px; + border: 1px solid var(--background-modifier-border); + border-radius: 5px; + margin: 10px 0 5px; + background-color: var(--background-primary); } + +div[data-type="dictionary-view"] .main .definition:last-child { + border: 1px solid var(--background-modifier-border); } + +div[data-type="dictionary-view"] .main .synonyms { + padding: 10px 0 0; } + +div[data-type="dictionary-view"] .main .synonyms p { + margin: 0; } + +div[data-type="dictionary-view"] .main .definition > blockquote { + margin: 0; } + +div[data-type="dictionary-view"] .main .label { + color: var(--text-normal); + margin-bottom: 2px; + font-size: var(--font-adaptive-smaller); + font-weight: 500; } + +div[data-type="dictionary-view"] .main .mark { + color: var(--text-normal); + background-color: var(--text-selection); + box-shadow: none; } + +div[data-type="dictionary-view"] .main > .opener { + font-size: var(--font-adaptive-small); + color: var(--text-normal); + padding-left: 5px; } + +/* Excalidraw Plugin */ +body .excalidraw, +body .excalidraw.theme--dark { + --color-primary-light:var(--text-selection); + --color-primary:var(--interactive-accent); + --color-primary-chubb:var(--interactive-accent-hover); + --color-primary-darker:var(--interactive-accent-hover); + --color-primary-darkest:var(--interactive-accent-hover); + --ui-font:var(--font-interface); + --island-bg-color:var(--background-secondary); + --button-gray-1:var(--background-tertiary); + --button-gray-2:var(--background-tertiary); + --focus-highlight-color:var(--background-modifier-border-focus); + --default-bg-color:var(--background-primary); + --input-border-color:var(--background-modifier-border); + --link-color:var(--text-accent); + --overlay-bg-color:rgba(255, 255, 255, 0.88); + --text-primary-color:var(--text-normal); } + +.workspace-leaf-content[data-type=excalidraw] .view-header .view-header-title-container { + width: auto; } + +body .excalidraw .App-toolbar-container .ToolIcon_type_floating:not(.is-mobile) .ToolIcon__icon { + box-shadow: none; } + +body .excalidraw button, +body .excalidraw .buttonList label { + cursor: var(--cursor); } + +body .excalidraw .Dialog__title { + font-variant: normal; } + +body .excalidraw .reset-zoom-button, +body .excalidraw .HintViewer { + color: var(--text-muted); + font-size: var(--font-small); } + +body .excalidraw .reset-zoom-button { + padding-left: 1em; + padding-right: 1em; } + +body .excalidraw .HintViewer > span { + background-color: transparent; } + +body .excalidraw button:hover { + box-shadow: none; } + +body .excalidraw .Island { + box-shadow: none; + border: 1px solid var(--background-modifier-border); } + +body .excalidraw .ToolIcon { + cursor: var(--cursor); + font-family: var(--font-interface); + background-color: transparent; } + +body .excalidraw label.ToolIcon { + cursor: var(--cursor); + background-color: transparent; } + +/* Electron Window Tweaker */ +:root { + --ewt-traffic-light-y:0px; } + +/* Embedded Note Titles plugin */ +.contextual-typography .markdown-preview-view h1.embedded-note-title { + margin-block-start: 0; + margin-block-end: 0; } + +.embedded-note-titles .markdown-preview-view > h1 { + padding-left: var(--folding-offset) !important; } + +.embedded-note-titles .is-readable-line-width.markdown-preview-view > h1 { + max-width: var(--max-width) !important; + width: var(--line-width-adaptive) !important; } + +.mod-cm6 .cm-editor h1.cm-line.embedded-note-title { + padding-top: var(--embedded-note-title-padding-top); + padding-bottom: var(--embedded-note-title-padding-bottom); } + +/* Attempting focus mode + embedded note titles + +.embedded-note-titles.minimal-focus-mode .markdown-preview-view > h1 { + padding-top:var(--header-height); +} +.embedded-note-titles.minimal-focus-mode .workspace-split.mod-root > .workspace-leaf:first-of-type:last-of-type .CodeMirror-scroller { + margin-top:calc(var(--header-height) - 10px); +}*/ +.embedded-note-titles .CodeMirror-scroll > h1 { + /* ...edit mode styles... */ } + +.embedded-note-titles .is-readable-line-width .CodeMirror-scroll > h1 { + /* ...edit mode styles with readable line width enabled... */ } + +/* Git plugin */ +.git-view-body .opener { + text-transform: uppercase; + letter-spacing: 0.05em; + font-size: var(--font-adaptive-smallest); + font-weight: 500; + padding: 5px 7px 5px 10px; + margin-bottom: 6px; } + +.git-view-body .file-view .opener { + text-transform: none; + letter-spacing: normal; + font-size: var(--font-adaptive-smallest); + font-weight: normal; + padding: initial; + margin-bottom: 0px; } + +.git-view-body .file-view .opener .collapse-icon { + display: flex !important; + margin-left: -7px; } + +.git-view-body { + margin-top: 6px; } + +.git-view-body .file-view { + margin-left: 4px; } + +.git-view-body .file-view main:hover { + color: var(--text-normal); } + +.git-view-body .file-view .tools .type { + display: none !important; } + +.git-view-body .file-view .tools { + opacity: 0; + transition: opacity .1s; } + +.git-view-body .file-view main:hover > .tools { + opacity: 1; } + +.git-view-body .staged { + margin-bottom: 12px; } + +.git-view-body .opener.open { + color: var(--text-normal); } + +div[data-type="git-view"] .search-input-container { + margin-left: 0; + width: 100%; } + +.git-view-body .opener .collapse-icon { + display: none !important; } + +.git-view-body main { + background-color: var(--background-primary) !important; + width: initial !important; } + +.git-view-body .file-view > main:not(.topLevel) { + margin-left: 7px; } + +div[data-type="git-view"] .commit-msg { + min-height: 2.5em !important; + height: 2.5em !important; + padding: 6.5px 8px !important; } + +div[data-type="git-view"] .search-input-clear-button { + bottom: 5.5px; } + +/* Hider plugin */ +/* Frameless mode */ +body.hider-frameless:not(.is-mobile) .workspace-split.mod-left-split > .workspace-tabs { + padding-top: var(--top-left-padding-y); + transition: padding-top 0.2s linear; } + +/* Include support for Electron Window Tweaker */ +body.mod-macos.hider-frameless:not(.is-fullscreen):not(.is-mobile) .workspace-split.mod-left-split > .workspace-tabs:nth-child(3) { + padding-top: calc(var(--top-left-padding-y) + var(--ewt-traffic-light-y)); + transition: padding-top 0.2s linear; } + +body.mod-macos.hider-frameless:not(.hider-ribbon):not(.is-fullscreen):not(.is-mobile) .workspace-ribbon .side-dock-actions { + padding-top: calc(var(--top-left-padding-y) + var(--ewt-traffic-light-y)); } + +.hider-frameless:not(.is-mobile) .workspace-split.mod-right-split > .workspace-tabs, +.hider-frameless:not(.is-mobile) .workspace-split.mod-root .view-header { + padding-top: 0px; } + +.hider-frameless:not(.is-mobile) .workspace-split.mod-right-split > .workspace-tabs ~ .workspace-tabs, +.hider-frameless:not(.is-mobile) .workspace-split.mod-left-split > .workspace-tabs ~ .workspace-tabs { + padding-top: 0px; } + +.hider-frameless.is-fullscreen:not(.is-mobile) .workspace-split.mod-left-split > .workspace-tabs, +.hider-frameless.is-fullscreen:not(.is-mobile) .workspace-split.mod-root .view-header { + padding-top: 0px; } + +/* Adjustments to title bar for traffic light icons */ +:root { + --traffic-x-space:0px; } + +/* Frameless + no ribbon */ +.mod-macos.hider-ribbon.hider-frameless:not(.is-fullscreen):not(.plugin-sliding-panes-rotate-header) .workspace-split.mod-left-split.is-collapsed + .mod-root .workspace-leaf:first-of-type { + --traffic-x-space:64px; } + +/* Frameless + popout */ +.mod-macos.is-popout-window.hider-ribbon.hider-frameless:not(.is-fullscreen):not(.plugin-sliding-panes-rotate-header) .mod-root .workspace-leaf:first-of-type { + --traffic-x-space:64px; } + +/* Frameless */ +.mod-macos.hider-frameless:not(.is-fullscreen):not(.plugin-sliding-panes-rotate-header) .workspace-split.mod-left-split.is-collapsed + .mod-root .workspace-leaf:first-of-type { + --traffic-x-space:22px; } + +/* Remove ribbon border on Mac when frameless */ +.mod-macos.hider-frameless .workspace-ribbon { + border: none; } + +/* --------------- */ +/* App ribbon moved to the bottom edge */ +.hider-ribbon:not(.is-mobile) .workspace-ribbon-collapse-btn { + display: none; } + +.hider-ribbon:not(.is-mobile) .workspace-ribbon.mod-right { + pointer-events: none; } + +.hider-ribbon:not(.is-mobile) .workspace-ribbon.mod-left { + position: absolute; + border-right: 0px; + margin: 0; + height: var(--header-height); + overflow: visible; + flex-basis: 0; + bottom: 0; + top: auto; + display: flex !important; + flex-direction: row; + z-index: 17; + opacity: 0; + transition: opacity 0.25s ease-in-out; + filter: drop-shadow(2px 10px 30px rgba(0, 0, 0, 0.2)); } + +.hider-ribbon:not(.is-mobile) .side-dock-actions, +.hider-ribbon:not(.is-mobile) .side-dock-settings { + display: flex; + border-top: var(--border-width) solid var(--background-modifier-border); + background: var(--background-secondary); + margin: 0; + position: relative; } + +.hider-ribbon:not(.is-mobile) .side-dock-actions { + padding-left: 5px; } + +.hider-ribbon:not(.is-mobile) .side-dock-settings { + border-right: var(--border-width) solid var(--background-modifier-border); + border-top-right-radius: 5px; + padding-right: 10px; } + +.hider-ribbon:not(.is-mobile) .workspace-ribbon.mod-left .side-dock-ribbon-action { + display: flex; + padding: 4px; + margin: 6px 0px 5px 7px; } + +.hider-ribbon:not(.is-mobile) .workspace-ribbon.mod-left:hover { + opacity: 1; + transition: opacity 0.25s ease-in-out; } + +.hider-ribbon:not(.is-mobile) .workspace-ribbon.mod-left .workspace-ribbon-collapse-btn { + opacity: 0; } + +.hider-ribbon:not(.is-mobile) .workspace-split.mod-left-split { + margin: 0; } + +.hider-ribbon:not(.is-mobile) .workspace-leaf-content .item-list { + padding-bottom: 40px; } + +.hider-ribbon .workspace-ribbon { + padding: 0; } + +/* Hover Editor */ +.popover.hover-editor { + --folding-offset:10px; } + +.theme-light, +.theme-dark { + --he-title-bar-inactive-bg:var(--background-secondary); + --he-title-bar-inactive-pinned-bg:var(--background-secondary); + --he-title-bar-active-pinned-bg:var(--background-secondary); + --he-title-bar-active-bg:var(--background-secondary); + --he-title-bar-inactive-fg:var(--text-muted); + --he-title-bar-active-fg:var(--text-normal); + --he-title-bar-font-size:14px; } + +.theme-light { + --popover-shadow: + 0px 2.7px 3.1px rgba(0, 0, 0, 0.032), + 0px 5.9px 8.7px rgba(0, 0, 0, 0.052), + 0px 10.4px 18.1px rgba(0, 0, 0, 0.071), + 0px 20px 40px rgba(0, 0, 0, 0.11) ; } + +.theme-dark { + --popover-shadow: + 0px 2.7px 3.1px rgba(0, 0, 0, 0.081), + 0px 5.9px 8.7px rgba(0, 0, 0, 0.131), + 0px 10.4px 18.1px rgba(0, 0, 0, 0.18), + 0px 20px 40px rgba(0, 0, 0, 0.28) ; } + +.popover.hover-editor:not(.snap-to-viewport) { + --max-width:92%; } + .popover.hover-editor:not(.snap-to-viewport) .markdown-preview-view, + .popover.hover-editor:not(.snap-to-viewport) .markdown-source-view .cm-content { + font-size: 90%; } + +body .popover.hover-editor:not(.is-loaded) { + box-shadow: var(--popover-shadow); } + body .popover.hover-editor:not(.is-loaded) .markdown-preview-view { + padding: 15px 0 0 0; } + body .popover.hover-editor:not(.is-loaded) .view-content { + height: 100%; + background-color: var(--background-primary); } + body .popover.hover-editor:not(.is-loaded) .view-actions { + height: auto; } + body .popover.hover-editor:not(.is-loaded) .popover-content { + border: 1px solid var(--background-modifier-border-hover); } + body .popover.hover-editor:not(.is-loaded) .popover-titlebar { + padding: 0 4px; } + body .popover.hover-editor:not(.is-loaded) .popover-titlebar .popover-title { + padding-left: 4px; + letter-spacing: -.02em; + font-weight: var(--title-weight); } + body .popover.hover-editor:not(.is-loaded) .markdown-embed { + height: auto; + font-size: unset; + line-height: unset; } + body .popover.hover-editor:not(.is-loaded) .markdown-embed .markdown-preview-view { + padding: 0; } + body .popover.hover-editor:not(.is-loaded).show-navbar .popover-titlebar { + border-bottom: var(--border-width) solid var(--background-modifier-border); } + body .popover.hover-editor:not(.is-loaded) .popover-action, + body .popover.hover-editor:not(.is-loaded) .popover-header-icon { + cursor: var(--cursor); + margin: 4px 0; + padding: 4px 3px; + border-radius: var(--radius-m); + color: var(--icon-color); } + body .popover.hover-editor:not(.is-loaded) .popover-action.mod-pin-popover, + body .popover.hover-editor:not(.is-loaded) .popover-header-icon.mod-pin-popover { + padding: 4px 2px; } + body .popover.hover-editor:not(.is-loaded) .popover-action svg, + body .popover.hover-editor:not(.is-loaded) .popover-header-icon svg { + opacity: var(--icon-muted); } + body .popover.hover-editor:not(.is-loaded) .popover-action:hover, + body .popover.hover-editor:not(.is-loaded) .popover-header-icon:hover { + background-color: var(--background-tertiary); + color: var(--icon-color-hover); } + body .popover.hover-editor:not(.is-loaded) .popover-action:hover svg, + body .popover.hover-editor:not(.is-loaded) .popover-header-icon:hover svg { + opacity: 1; + transition: opacity 0.1s ease-in-out; } + body .popover.hover-editor:not(.is-loaded) .popover-action.is-active, + body .popover.hover-editor:not(.is-loaded) .popover-header-icon.is-active { + color: var(--icon-color); } + +/* Kanban plugin */ +body .kanban-plugin__markdown-preview-view { + font-family: var(----text); } + +body .kanban-plugin { + --interactive-accent:var(--text-selection); + --interactive-accent-hover:var(--background-tertiary); + --text-on-accent:var(--text-normal); + background-color: var(--background-primary); } + +body .kanban-plugin__board > div { + margin: 0 auto; } + +body .kanban-plugin__checkbox-label { + font-size: var(--font-adaptive-small); + color: var(--text-muted); } + +body .kanban-plugin__item-markdown ul { + margin: 0; } + +body .kanban-plugin__item-content-wrapper { + box-shadow: none; } + +body .kanban-plugin__grow-wrap > textarea, +body .kanban-plugin__grow-wrap::after { + padding: 0; + border: 0; + border-radius: 0; } + +body:not(.is-mobile) .kanban-plugin__grow-wrap > textarea:focus { + box-shadow: none; } + +body .kanban-plugin__markdown-preview-view, +body .kanban-plugin__grow-wrap > textarea, +body .kanban-plugin__grow-wrap::after, +body .kanban-plugin__item-title p { + font-size: calc(var(--font-adaptive-normal) - 2px); + line-height: 1.3; } + +.kanban-plugin__item-input-actions button, +.kanban-plugin__lane-input-actions button { + font-size: var(--font-adaptive-small); } + +body .kanban-plugin__item { + background-color: var(--background-primary); } + +.kanban-plugin__item-title-wrapper { + align-items: center; } + +body .kanban-plugin__lane-form-wrapper { + border: 1px solid var(--background-modifier-border); } + +body .kanban-plugin__lane-header-wrapper { + border-bottom: 0; } + +body .kanban-plugin__lane-title p, +body .kanban-plugin__lane-header-wrapper .kanban-plugin__grow-wrap > textarea, +body .kanban-plugin__lane-input-wrapper .kanban-plugin__grow-wrap > textarea { + background: transparent; + color: var(--text-normal); + font-size: calc(var(--font-adaptive-normal) - 2px); + font-weight: 500; } + +body .kanban-plugin__item-input-wrapper .kanban-plugin__grow-wrap > textarea { + padding: 0; + border-radius: 0; } + +body .kanban-plugin__item-form .kanban-plugin__grow-wrap { + padding: 6px 8px; + border-radius: 6px; + border: 1px solid var(--background-modifier-border); + background-color: var(--background-primary); } + +body .kanban-plugin__item-input-wrapper .kanban-plugin__grow-wrap > textarea::placeholder { + color: var(--text-faint); } + +body .kanban-plugin__lane button.kanban-plugin__lane-settings-button.is-enabled, +body .kanban-plugin__item .kanban-plugin__item-edit-archive-button, +body .kanban-plugin__item button.kanban-plugin__item-edit-button, +body .kanban-plugin__lane button.kanban-plugin__lane-settings-button, +.kanban-plugin__item-settings-actions > button, +.kanban-plugin__lane-action-wrapper > button { + background: transparent; + transition: color 0.1s ease-in-out; } + +body .kanban-plugin__item .kanban-plugin__item-edit-archive-button:hover, +body .kanban-plugin__item button.kanban-plugin__item-edit-button.is-enabled, +body .kanban-plugin__item button.kanban-plugin__item-edit-button:hover, +body .kanban-plugin__lane button.kanban-plugin__lane-settings-button.is-enabled, +body .kanban-plugin__lane button.kanban-plugin__lane-settings-button:hover { + color: var(--text-normal); + transition: color 0.1s ease-in-out; + background: transparent; } + +body .kanban-plugin__new-lane-button-wrapper { + position: fixed; + bottom: 30px; } + +body .kanban-plugin__lane-items > .kanban-plugin__placeholder:only-child { + border: 1px dashed var(--background-modifier-border); + height: 2em; } + +body .kanban-plugin__item-postfix-button-wrapper { + align-self: flex-start; } + +body .kanban-plugin__item button.kanban-plugin__item-prefix-button.is-enabled, +body .kanban-plugin__item button.kanban-plugin__item-postfix-button.is-enabled, +body .kanban-plugin__lane button.kanban-plugin__lane-settings-button.is-enabled { + color: var(--text-muted); } + +body .kanban-plugin button { + box-shadow: none; + cursor: var(--cursor); } + +body .kanban-plugin__item button.kanban-plugin__item-prefix-button:hover, +body .kanban-plugin__item button.kanban-plugin__item-postfix-button:hover, +body .kanban-plugin__lane button.kanban-plugin__lane-settings-button:hover { + background-color: var(--background-tertiary); } + +body:not(.minimal-icons-off) .kanban-plugin svg.cross { + height: 14px; + width: 14px; } + +body .kanban-plugin__item-button-wrapper > button { + font-size: var(--font-adaptive-small); + color: var(--text-muted); + font-weight: 400; + background: transparent; + height: 32px; } + +body .kanban-plugin__item-button-wrapper > button:hover { + color: var(--text-normal); + background: var(--background-tertiary); } + +body .kanban-plugin__item-button-wrapper > button:focus { + box-shadow: none; } + +body .kanban-plugin__item-button-wrapper { + padding: 1px 6px 5px; + border-top: none; } + +body .kanban-plugin__lane-setting-wrapper > div:last-child { + border: none; + margin: 0; } + +body .kanban-plugin.something-is-dragging { + cursor: grabbing; + cursor: -webkit-grabbing; } + +body .kanban-plugin__item.is-dragging { + box-shadow: 0 5px 30px rgba(0, 0, 0, 0.15), 0 0 0 2px var(--text-selection); } + +body .kanban-plugin__lane.is-dragging { + box-shadow: 0 5px 30px rgba(0, 0, 0, 0.15); + border: 1px solid var(--background-modifier-border); } + +body .kanban-plugin__lane { + background: transparent; + padding: 0; + border: var(--border-width) solid transparent; } + +body { + --kanban-border:var(--border-width); } + +body.theme-dark, +body.minimal-dark-black.theme-dark, +body.minimal-dark-tonal.theme-dark, +body.minimal-light-white.theme-light, +body.minimal-light-tonal.theme-light { + --kanban-border:0px; } + +body .kanban-plugin__lane-items { + border: var(--kanban-border) solid var(--background-modifier-border); + border-bottom: none; + padding: 0 4px; + border-top-left-radius: 8px; + border-top-right-radius: 8px; + margin: 0; + background-color: var(--background-secondary); } + +body .kanban-plugin__item-input-wrapper { + border: 0; + padding-top: 1px; + flex-grow: 1; } + +body .kanban-plugin__item-form, +body .kanban-plugin__item-button-wrapper { + background-color: var(--background-secondary); + border: var(--kanban-border) solid var(--background-modifier-border); + border-top: none; + border-bottom-left-radius: 8px; + border-bottom-right-radius: 8px; } + +body .kanban-plugin__item-form { + padding: 0 4px 5px; } + +body .kanban-plugin__markdown-preview-view ol.contains-task-list .contains-task-list, +body .kanban-plugin__markdown-preview-view ul.contains-task-list .contains-task-list, +body .kanban-plugin__markdown-preview-view ul, .kanban-plugin__markdown-preview-view ol { + padding-inline-start: 1.8em !important; } + +@media (max-width: 400pt) { + .kanban-plugin__board { + flex-direction: column !important; } + + .kanban-plugin__lane { + width: 100% !important; + margin-bottom: 1rem !important; } } +/* Lapel */ +body .cm-heading-marker { + cursor: var(--cursor); + padding-left: 10px; } + +/* Leaflet plugin */ +.theme-light { + --leaflet-buttons:var(--bg1); + --leaflet-borders:rgba(0,0,0,0.1); } + +.theme-dark { + --leaflet-buttons:var(--bg2); + --leaflet-borders:rgba(255,255,255,0.1); } + +.leaflet-top { + transition: top 0.1s linear; } + +.mod-macos.minimal-focus-mode .mod-root .map-100 .markdown-preview-sizer.markdown-preview-section .el-lang-leaflet:nth-child(3) .leaflet-top { + top: calc(18px + var(--ewt-traffic-light-y)); + transition: top 0.1s linear; } + +body .leaflet-container { + background-color: var(--background-secondary); + font-family: var(--font-interface); } + +.map-100 .markdown-preview-sizer.markdown-preview-section .el-lang-leaflet:nth-child(3) { + margin-top: -16px; } + +.leaflet-control-attribution { + display: none; } + +.leaflet-popup-content { + margin: 10px; } + +.block-language-leaflet { + border-radius: var(--radius-m); + overflow: hidden; + border: var(--border-width) solid var(--background-modifier-border); } + +.map-wide .block-language-leaflet { + border-radius: var(--radius-l); } + +.map-max .block-language-leaflet { + border-radius: var(--radius-xl); } + +.workspace-leaf-content[data-type="obsidian-leaflet-map-view"] .block-language-leaflet { + border-radius: 0; + border: none; } + +.map-100 .block-language-leaflet { + border-radius: 0px; + border-left: none; + border-right: none; } + +/* Checkbox */ +.block-language-leaflet .leaflet-control-expandable-list .input-container .input-item > input { + appearance: none; } + +/* Buttons */ +body .block-language-leaflet .leaflet-bar.disabled > a { + background-color: transparent; + opacity: 0.3; } + +body .leaflet-touch .leaflet-bar a:first-child { + border-top-left-radius: 4px; + border-top-right-radius: 4px; } + +body .leaflet-touch .leaflet-bar a:last-child { + border-bottom-left-radius: 4px; + border-bottom-right-radius: 4px; } + +body .leaflet-control-layers-toggle { + border-radius: 4px; } + +body .leaflet-control-layers-toggle, +body .leaflet-control-layers-expanded, +body .block-language-leaflet .leaflet-control-has-actions .control-actions.expanded, +body .block-language-leaflet .leaflet-control-expandable, +body .block-language-leaflet .leaflet-distance-control, +body .leaflet-bar, +body .leaflet-bar a { + background-color: var(--leaflet-buttons); + color: var(--text-muted); + border: none; + user-select: none; } + +body .leaflet-bar a.leaflet-disabled, +body .leaflet-bar a.leaflet-disabled:hover { + background-color: var(--leaflet-buttons); + color: var(--text-faint); + opacity: 0.6; + cursor: not-allowed; } + +body .leaflet-control a { + cursor: var(--cursor); + color: var(--text-normal); } + +body .leaflet-bar a:hover { + background-color: var(--background-tertiary); + color: var(--text-normal); + border: none; } + +body .leaflet-touch .leaflet-control-layers { + background-color: var(--leaflet-buttons); } + +body .leaflet-touch .leaflet-control-layers, +body .leaflet-touch .leaflet-bar { + border-radius: 5px; + box-shadow: 2px 0 8px 0px rgba(0, 0, 0, 0.1); + border: 1px solid var(--ui1); } + +body .block-language-leaflet .leaflet-control-has-actions .control-actions { + box-shadow: 0; + border: 1px solid var(--ui1); } + +body .leaflet-control-expandable-list .leaflet-bar { + box-shadow: none; + border-radius: 0; } + +body .block-language-leaflet .leaflet-distance-control { + padding: 4px 10px; + height: auto; + cursor: var(--cursor) !important; } + +body .block-language-leaflet .leaflet-marker-link-popup > .leaflet-popup-content-wrapper > * { + font-size: var(--font-adaptive-small); + font-family: var(--font-interface); } + +body .block-language-leaflet .leaflet-marker-link-popup > .leaflet-popup-content-wrapper { + padding: 4px 10px !important; } + +.leaflet-marker-icon svg path { + stroke: var(--background-primary); + stroke-width: 18px; } + +/* Map View plugin */ +.map-view-marker-name { + font-weight: 400; } + +.workspace-leaf-content[data-type="map"] .graph-controls { + background-color: var(--background-primary); } + +/* Full bleed */ +body:not(.is-mobile):not(.plugin-sliding-panes-rotate-header) .workspace-split.mod-root .workspace-leaf-content[data-type='map'] .view-header { + position: fixed; + background: transparent !important; + width: 100%; + z-index: 99; } + +body:not(.plugin-sliding-panes-rotate-header) .workspace-leaf-content[data-type='map'] .view-header-title { + display: none; } + +body:not(.is-mobile):not(.plugin-sliding-panes-rotate-header) .workspace-leaf-content[data-type='map'] .view-actions { + background: transparent; } + +body:not(.is-mobile):not(.plugin-sliding-panes-rotate-header) .workspace-leaf-content[data-type='map'] .view-content { + height: 100%; } + +body:not(.is-mobile):not(.plugin-sliding-panes-rotate-header) .workspace-leaf-content[data-type='map'] .leaflet-top.leaflet-right { + top: var(--header-height); } + +/* Metatable */ +.obsidian-metatable { + --metatable-font-size:calc(var(--font-adaptive-normal) - 2px); + --metatable-font-family: var(--font-interface); + --metatable-background:transparent; + --metatable-foreground: var(--text-faint); + --metatable-key-background:transparent; + --metatable-key-border-width:0; + --metatable-key-border-color:transparent; + --metatable-value-background:transparent; + padding-bottom: 0.5rem; } + .obsidian-metatable::part(value), .obsidian-metatable::part(key) { + border-bottom: 0px solid var(--background-modifier-border); + padding: 0.1rem 0; + text-overflow: ellipsis; + overflow: hidden; } + .obsidian-metatable::part(key) { + font-weight: 400; + color: var(--tx3); + font-size: calc(var(--font-adaptive-normal) - 2px); } + .obsidian-metatable::part(value) { + font-size: calc(var(--font-adaptive-normal) - 2px); + color: var(--tx1); } + +/* NL tables */ +body .NLT__header-menu-header-container { + font-size: 85%; } + +body .NLT__button { + background: transparent; + box-shadow: none; + color: var(--text-muted); } + body .NLT__button:hover, body .NLT__button:active, body .NLT__button:focus { + background: transparent; + color: var(--text-normal); + box-shadow: none; } + +.NLT__app .NLT__button { + background: transparent; + border: 1px solid var(--background-modifier-border); + box-shadow: 0 0.5px 1px 0 var(--btn-shadow-color); + color: var(--text-muted); + padding: 2px 8px; } + .NLT__app .NLT__button:hover, .NLT__app .NLT__button:active, .NLT__app .NLT__button:focus { + background: transparent; + border-color: var(--background-modifier-border-hover); + color: var(--text-normal); + box-shadow: 0 0.5px 1px 0 var(--btn-shadow-color); } + +/* +.NLT__header-content { + position:relative; +} +th.NLT__selectable .NLT__header-content:after { + content:" "; + width:16px; + height:16px; + position:absolute; + z-index:999999; + top:50%; + transform:translateY(-50%); + display:inline-block; + -webkit-mask-image:url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-5 w-5' viewBox='0 0 20 20' fill='currentColor' %3E%3Cpath fill-rule='evenodd' d='M11.49 3.17c-.38-1.56-2.6-1.56-2.98 0a1.532 1.532 0 01-2.286.948c-1.372-.836-2.942.734-2.106 2.106.54.886.061 2.042-.947 2.287-1.561.379-1.561 2.6 0 2.978a1.532 1.532 0 01.947 2.287c-.836 1.372.734 2.942 2.106 2.106a1.532 1.532 0 012.287.947c.379 1.561 2.6 1.561 2.978 0a1.533 1.533 0 012.287-.947c1.372.836 2.942-.734 2.106-2.106a1.533 1.533 0 01.947-2.287c1.561-.379 1.561-2.6 0-2.978a1.532 1.532 0 01-.947-2.287c.836-1.372-.734-2.942-2.106-2.106a1.532 1.532 0 01-2.287-.947zM10 13a3 3 0 100-6 3 3 0 000 6z' clip-rule='evenodd' /%3E%3C/svg%3E"); + -webkit-mask-image:url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-5 w-5' viewBox='0 0 20 20' fill='currentColor'%3E%3Cpath d='M6 10a2 2 0 11-4 0 2 2 0 014 0zM12 10a2 2 0 11-4 0 2 2 0 014 0zM16 12a2 2 0 100-4 2 2 0 000 4z' /%3E%3C/svg%3E"); + -webkit-mask-size:16px 16px; + margin:0 0 0 6px; + background-color:var(--text-faint); +}*/ +.NLT__td:nth-last-child(2), +.NLT__th:nth-last-child(2) { + border-right: 0; } + +.NLT__app { + /* Remove Sortable plugin background icons */ } + .NLT__app .NLT__td:last-child, + .NLT__app .NLT__th:last-child { + padding-right: 0; } + .NLT__app .NLT__th { + background-image: none !important; } + .NLT__app th.NLT__selectable:hover { + background-color: transparent; + cursor: var(--cursor); } + +.NLT__menu .NLT__menu-container { + background-color: var(--background-secondary); } +.NLT__menu .NLT__header-menu-item { + font-size: var(--font-adaptive-small); } +.NLT__menu .NLT__header-menu { + padding: 6px 4px; } +.NLT__menu .NLT__drag-menu { + font-size: var(--font-adaptive-small); + padding: 6px 4px; } +.NLT__menu svg { + color: var(--text-faint); + margin-right: 6px; } +.NLT__menu .NLT__selected, +.NLT__menu .NLT__selectable:hover { + background: transparent; } +.NLT__menu .NLT__selected > .NLT__selectable { + background-color: var(--background-tertiary); } +.NLT__menu .NLT__selectable { + cursor: var(--cursor); } +.NLT__menu div.NLT__selectable { + min-width: 110px; + border-radius: var(--radius-m); + padding: 3px 8px 3px 4px; + margin: 1px 2px 1px; + cursor: var(--cursor); + height: auto; + line-height: 20px; } + .NLT__menu div.NLT__selectable:hover { + background-color: var(--background-tertiary); } +.NLT__menu .NLT__textarea { + font-size: var(--table-font-size); } + +.NLT__tfoot tr:hover td { + background-color: transparent; } + +/* Outliner plugin (pre Live Preview) */ +body.outliner-plugin-bls .CodeMirror-line .cm-hmd-list-indent::before { + background-image: linear-gradient(to right, var(--background-modifier-border) 1px, transparent 1px); + background-position-x: 2px; + background-size: var(--font-text-size) 1px; } + +body.outliner-plugin-bls .cm-s-obsidian span.cm-formatting-list { + letter-spacing: unset; } + +body.outliner-plugin-bls .cm-s-obsidian .HyperMD-list-line { + padding-top: 0; } + +body.outliner-plugin-bls .cm-s-obsidian span.cm-formatting-list-ul:before { + color: var(--text-faint); + margin-left: -3px; + margin-top: -5px; } + +body.outliner-plugin-bls.minimal-rel-edit .cm-hmd-list-indent > .cm-tab:after { + content: ""; + border-right: none; } + +body.outliner-plugin-bls .cm-s-obsidian span.cm-formatting-list-ul { + color: transparent !important; } + +body.outliner-plugin-bls .cm-s-obsidian:not(.is-live-preview) .cm-formatting-list-ul:before, +body.outliner-plugin-bls .cm-s-obsidian.is-live-preview .list-bullet:before { + color: var(--text-faint); } + +/* QuickAdd plugin */ +.modal .quickAddPrompt > h1, +.modal .quickAddYesNoPrompt h1 { + margin-top: 0; + text-align: left !important; + font-size: var(--h1); + font-weight: 600; } + +.modal .quickAddYesNoPrompt p { + text-align: left !important; } + +.modal .quickAddYesNoPrompt button { + font-size: var(--font-settings-small); } + +.modal .yesNoPromptButtonContainer { + font-size: var(--font-settings-small); + justify-content: flex-end; } + +.quickAddModal .modal-content { + padding: 20px 2px 5px; } + +div#quick-explorer { + display: flex; } + div#quick-explorer span.explorable { + align-items: center; + color: var(--text-muted); + display: flex; + font-size: var(--font-adaptive-smaller); + line-height: 16px; } + div#quick-explorer span.explorable:last-of-type { + font-size: var(--font-adaptive-smaller); } + div#quick-explorer span.explorable.selected, div#quick-explorer span.explorable:hover { + background-color: unset !important; } + div#quick-explorer span.explorable.selected .explorable-name, div#quick-explorer span.explorable:hover .explorable-name { + color: var(--text-normal); } + div#quick-explorer span.explorable.selected .explorable-separator, div#quick-explorer span.explorable:hover .explorable-separator { + color: var(--text-normal); } + div#quick-explorer .explorable-name { + padding: 0 4px; + border-radius: 4px; } + div#quick-explorer .explorable-separator::before { + content: "\00a0›" !important; + font-size: 1.3em; + font-weight: 400; + margin: 0px; } + +body:not(.colorful-active) .qe-popup-menu .menu-item:not(.is-disabled):not(.is-label):hover, body:not(.colorful-active) .qe-popup-menu .menu-item:not(.is-disabled):not(.is-label).selected { + background-color: var(--background-tertiary); + color: var(--text-normal); } + body:not(.colorful-active) .qe-popup-menu .menu-item:not(.is-disabled):not(.is-label):hover .menu-item-icon, body:not(.colorful-active) .qe-popup-menu .menu-item:not(.is-disabled):not(.is-label).selected .menu-item-icon { + color: var(--text-normal); } + +/* Obsidian Tabs plugin */ +.workspace-leaf-content[data-type="recent-files"] .view-content { + padding-top: 10px; } + +/* Reminder Plugin */ +.mod-root .workspace-leaf-content[data-type="reminder-list"] main { + max-width: var(--max-width); + margin: 0 auto; + padding: 0; } + +/* Popup */ +.modal .reminder-actions .later-select { + font-size: var(--font-settings-small); + vertical-align: bottom; + margin-left: 3px; } +.modal .reminder-actions .icon { + line-height: 1; } + +/* In sidebar */ +:not(.mod-root) .workspace-leaf-content[data-type="reminder-list"] main { + margin: 0 auto; + padding: 15px; } + :not(.mod-root) .workspace-leaf-content[data-type="reminder-list"] main .group-name { + font-weight: 500; + color: var(--text-muted); + font-size: var(--font-adaptive-small); + padding-bottom: 0.5em; + border-bottom: 1px solid var(--background-modifier-border); } + :not(.mod-root) .workspace-leaf-content[data-type="reminder-list"] main .reminder-group .reminder-list-item { + line-height: 1.3; + font-size: var(--font-adaptive-small); } + :not(.mod-root) .workspace-leaf-content[data-type="reminder-list"] main .reminder-group .no-reminders { + color: var(--text-faint); } + :not(.mod-root) .workspace-leaf-content[data-type="reminder-list"] main .reminder-group .reminder-time { + font-family: var(--font-text); + font-size: var(--font-adaptive-small); } + :not(.mod-root) .workspace-leaf-content[data-type="reminder-list"] main .reminder-group .reminder-file { + color: var(--text-faint); } + +/* Calendar picker */ +body .modal .dtchooser { + background-color: transparent; } + body .modal .dtchooser .reminder-calendar .year-month { + font-weight: 400; + font-size: var(--font-adaptive-normal); + padding-bottom: 10px; } + body .modal .dtchooser .reminder-calendar .year-month .month, + body .modal .dtchooser .reminder-calendar .year-month .year { + color: var(--text-normal); } + body .modal .dtchooser .reminder-calendar .year-month .month-nav:first-child { + background-color: currentColor; + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-5 w-5' viewBox='0 0 20 20' fill='currentColor'%3E%3Cpath fill-rule='evenodd' d='M12.707 5.293a1 1 0 010 1.414L9.414 10l3.293 3.293a1 1 0 01-1.414 1.414l-4-4a1 1 0 010-1.414l4-4a1 1 0 011.414 0z' clip-rule='evenodd' /%3E%3C/svg%3E"); } + body .modal .dtchooser .reminder-calendar .year-month .month-nav:last-child { + background-color: currentColor; + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-5 w-5' viewBox='0 0 20 20' fill='currentColor'%3E%3Cpath fill-rule='evenodd' d='M7.293 14.707a1 1 0 010-1.414L10.586 10 7.293 6.707a1 1 0 011.414-1.414l4 4a1 1 0 010 1.414l-4 4a1 1 0 01-1.414 0z' clip-rule='evenodd' /%3E%3C/svg%3E"); } + body .modal .dtchooser .reminder-calendar .year-month .month-nav { + -webkit-mask-size: 20px 20px; + -webkit-mask-repeat: no-repeat; + -webkit-mask-position: 50% 50%; + color: var(--text-faint); + cursor: var(--cursor); + border-radius: var(--radius-m); + padding: 0; + width: 30px; + display: inline-block; } + body .modal .dtchooser .reminder-calendar .year-month .month-nav:hover { + color: var(--text-muted); } + body .modal .dtchooser .reminder-calendar th { + padding: 0.5em 0; + font-size: var(--font-adaptive-smallest); + font-weight: 500; + text-transform: uppercase; + letter-spacing: 0.1em; } + body .modal .dtchooser .reminder-calendar .calendar-date { + transition: background-color 0.1s ease-in; + padding: 0.3em 0; + border-radius: var(--radius-m); } + body .modal .dtchooser .reminder-calendar .calendar-date:hover, body .modal .dtchooser .reminder-calendar .calendar-date.is-selected { + transition: background-color 0.1s ease-in; + background-color: var(--background-tertiary) !important; } + body .modal .dtchooser .reminder-calendar .calendar-date.is-selected { + font-weight: var(--bold-weight); + color: var(--text-accent) !important; } + +/* Sliding Panes aka Andy Mode plugin */ +body.plugin-sliding-panes-rotate-header { + --header-width:40px; } + body.plugin-sliding-panes-rotate-header .view-header-title:before { + display: none; } + +body.plugin-sliding-panes .workspace-split.mod-root { + background-color: var(--background-primary); } +body.plugin-sliding-panes .mod-horizontal .workspace-leaf { + box-shadow: none !important; } +body.plugin-sliding-panes:not(.is-fullscreen) .workspace-split.is-collapsed ~ .workspace-split.mod-root .view-header { + transition: padding 0.1s ease; } +body.plugin-sliding-panes .view-header-title:before { + background: none; } +body.plugin-sliding-panes .view-header { + background: none; } +body.plugin-sliding-panes.plugin-sliding-panes-rotate-header .workspace > .mod-root > .workspace-leaf.mod-active > .workspace-leaf-content > .view-header { + border: none; } +body.plugin-sliding-panes.plugin-sliding-panes-rotate-header .workspace > .mod-root > .workspace-leaf > .workspace-leaf-content > .view-header { + border: none; + text-orientation: sideways; } + body.plugin-sliding-panes.plugin-sliding-panes-rotate-header .workspace > .mod-root > .workspace-leaf > .workspace-leaf-content > .view-header .view-header-icon { + padding: 4px 1px; + margin: 5px 0 0 0; + left: 0; + width: 26px; } + body.plugin-sliding-panes.plugin-sliding-panes-rotate-header .workspace > .mod-root > .workspace-leaf > .workspace-leaf-content > .view-header .view-actions { + padding-bottom: 33px; + margin-left: 0; + height: auto; } + body.plugin-sliding-panes.plugin-sliding-panes-rotate-header .workspace > .mod-root > .workspace-leaf > .workspace-leaf-content > .view-header .view-action { + margin: 3px 0; + padding: 4px 1px; + width: 26px; } +body.plugin-sliding-panes.plugin-sliding-panes-rotate-header .workspace > .mod-root > .workspace-leaf > .workspace-leaf-content > .view-header > .view-header-title-container:before, +body.plugin-sliding-panes.plugin-sliding-panes-rotate-header .app-container .workspace > .mod-root > .workspace-leaf.mod-active > .workspace-leaf-content > .view-header > .view-header-title-container:before { + background: none !important; } +.workspace > .mod-root .view-header-title-container +body.plugin-sliding-panes.plugin-sliding-panes-rotate-header.plugin-sliding-panes-header-alt .workspace > .mod-root .view-header-title { + margin-top: 0; } +body.plugin-sliding-panes.plugin-sliding-panes-rotate-header .workspace > .mod-root .view-header-title-container { + margin-left: 0; + padding-top: 0; } +body.plugin-sliding-panes.plugin-sliding-panes-rotate-header .view-header-title-container { + position: static; } +body.plugin-sliding-panes.plugin-sliding-panes-rotate-header .app-container .workspace > .mod-root > .workspace-leaf > .workspace-leaf-content > .view-header > div { + margin-left: 0; + margin-right: 0; + bottom: 0; } +body.plugin-sliding-panes.plugin-sliding-panes-rotate-header.show-grabber .view-header-icon { + opacity: var(--icon-muted); } +body.plugin-sliding-panes.plugin-sliding-panes-rotate-header .view-header-icon:hover { + opacity: 1; } + +body:not(.plugin-sliding-panes-header-alt).plugin-sliding-panes-rotate-header .workspace > .mod-root > .workspace-leaf > .workspace-leaf-content > .view-header > .view-header-title-container > .view-header-title, +body:not(.plugin-sliding-panes-header-alt).plugin-sliding-panes-rotate-header .workspace > .mod-root > .workspace-split > .workspace-leaf-content > .view-header > .view-header-title-container > .view-header-title { + padding-top: 5px; } + +body.plugin-sliding-panes-stacking .workspace > .mod-root > .workspace-leaf, +body.plugin-sliding-panes .workspace-split.mod-vertical > .workspace-leaf { + box-shadow: 0 0 0 1px var(--background-modifier-border), 1px 0px 15px 0px var(--shadow-color) !important; } + +body.is-mobile.plugin-sliding-panes.plugin-sliding-panes-rotate-header .workspace > .mod-root > .workspace-leaf > .workspace-leaf-content > .view-header .view-header-icon { + height: 30px; } + +/* Space for the hover ribbon in the bottom left */ +body.hider-ribbon.plugin-sliding-panes.plugin-sliding-panes-rotate-header .workspace > .mod-root > .workspace-leaf > .workspace-leaf-content > .view-header .view-actions { + padding-bottom: 50px; } + +body.plugin-sliding-panes.is-fullscreen .view-header-icon { + padding-top: 8px; } + +body.plugin-sliding-panes .mod-root .graph-controls { + top: 20px; + left: 30px; } + +/* Sortable plugin */ +body .markdown-preview-view th, +body .table-view-table > thead > tr > th, +body .markdown-source-view.mod-cm6 .dataview.table-view-table thead.table-view-thead tr th { + cursor: var(--cursor); + background-image: none; } + +/* Live preview */ +.markdown-source-view.mod-cm6 th { + background-repeat: no-repeat; + background-position: right; } + +/* Style Settings preferences */ +.style-settings-container[data-level="2"] { + background: var(--background-secondary); + border: 1px solid var(--ui1); + border-radius: 5px; + padding: 10px 20px; + margin: 2px 0 2px -20px; } + +.workspace-leaf-content[data-type="style-settings"] .view-content { + padding: 0 0 20px var(--folding-offset); } +.workspace-leaf-content[data-type="style-settings"] .view-content > div { + width: var(--line-width-adaptive); + max-width: var(--max-width); + margin: 0 auto; } +.workspace-leaf-content[data-type="style-settings"] .style-settings-heading[data-level="0"] .setting-item-name { + padding-left: 17px; } +.workspace-leaf-content[data-type="style-settings"] .setting-item { + max-width: 100%; + margin: 0 auto; } +.workspace-leaf-content[data-type="style-settings"] .setting-item-name { + position: relative; } +.workspace-leaf-content[data-type="style-settings"] .style-settings-collapse-indicator { + position: absolute; + left: 0; } + +.setting-item-heading.style-settings-heading, +.style-settings-container .style-settings-heading { + cursor: var(--cursor); } + +.modal.mod-settings .setting-item .pickr button.pcr-button { + box-shadow: none; + border-radius: 40px; + height: 24px; + width: 24px; } + +.setting-item .pickr .pcr-button:after, +.setting-item .pickr .pcr-button:before { + border-radius: 40px; + box-shadow: none; + border: none; } + +.setting-item.setting-item-heading.style-settings-heading.is-collapsed { + border-bottom: 1px solid var(--background-modifier-border); } + +.setting-item.setting-item-heading.style-settings-heading { + border: 0; + padding: 10px 0 5px; + margin-bottom: 0; } + +.mod-root .workspace-leaf-content[data-type="style-settings"] .style-settings-container .setting-item:not(.setting-item-heading) { + flex-direction: row; + align-items: center; + padding: 0.5em 0; } + +.workspace-split:not(.mod-root) .workspace-leaf-content[data-type="style-settings"] .setting-item-name { + font-size: var(--font-small); } + +.setting-item .style-settings-import, +.setting-item .style-settings-export { + text-decoration: none; + font-size: var(--font-settings-small); + font-weight: 500; + color: var(--text-muted); + margin: 0; + padding: 2px 8px; + border-radius: 5px; + cursor: var(--cursor); } + +.style-settings-import:hover, +.style-settings-export:hover { + background-color: var(--background-tertiary); + color: var(--text-normal); + cursor: var(--cursor); } + +.themed-color-wrapper > div + div { + margin-top: 0; + margin-left: 6px; } + +.theme-light .themed-color-wrapper > .theme-light { + background-color: transparent; } + +.theme-light .themed-color-wrapper > .theme-dark { + background-color: rgba(0, 0, 0, 0.8); } + +.theme-dark .themed-color-wrapper > .theme-dark { + background-color: transparent; } + +/* Obsidian Tabs plugin */ +body.plugin-tabs .mod-root.workspace-split.mod-vertical > div.workspace-leaf.stayopen .view-header, +body.plugin-tabs .mod-root.workspace-split.mod-vertical > .workspace-split.mod-vertical > div.workspace-leaf .view-header, .plugin-tabs .mod-root.workspace-split.mod-vertical > div.workspace-leaf.mod-active .view-header { + border: none; } + +/* Todoist Plugin Styles */ +body .todoist-query-title { + display: inline; + font-size: var(--h4); + font-variant: var(--h4-variant); + letter-spacing: 0.02em; + color: var(--h4-color); + font-weight: var(--h4-weight); + font-style: var(--h4-style); } + +body .is-live-preview .block-language-todoist { + padding-left: 0; } + +ul.todoist-task-list > li.task-list-item .task-list-item-checkbox { + margin: 0; } + +body .todoist-refresh-button { + display: inline; + float: right; + background: transparent; + padding: 5px 6px 0; + margin-right: 0px; } + +body .is-live-preview .todoist-refresh-button { + margin-right: 30px; } + +body .todoist-refresh-button:hover { + box-shadow: none; + background-color: var(--background-tertiary); } + +.todoist-refresh-button svg { + width: 15px; + height: 15px; + opacity: var(--icon-muted); } + +ul.todoist-task-list { + margin-left: -0.25em; } + +.is-live-preview ul.todoist-task-list { + padding-left: 0; + margin-left: 0.5em; + margin-block-start: 0; + margin-block-end: 0; } + +.contains-task-list.todoist-task-list .task-metadata { + font-size: var(--font-adaptive-small); + display: flex; + color: var(--text-muted); + justify-content: space-between; + margin-left: 0.1em; + margin-bottom: 0.25rem; } + +.is-live-preview .contains-task-list.todoist-task-list .task-metadata { + padding-left: calc(var(--checkbox-size) + 0.6em); } + +.todoist-task-list .task-date.task-overdue { + color: var(--orange); } + +body .todoist-p1 > input[type="checkbox"] { + border: 1px solid var(--red); } + +body .todoist-p1 > input[type="checkbox"]:hover { + opacity: 0.8; } + +body .todoist-p2 > input[type="checkbox"] { + border: 1px solid var(--yellow); } + +body .todoist-p2 > input[type="checkbox"]:hover { + opacity: 0.8; } + +body .todoist-p3 > input[type="checkbox"] { + border: 1px solid var(--blue); } + +body .todoist-p3 > input[type="checkbox"]:hover { + opacity: 0.8; } + +/* Tracker */ +body.theme-light { + --color-axis-label:var(--tx1); + --color-tick-label:var(--tx2); + --color-dot-fill:var(--ax1); + --color-line:var(--ui1); } + +.tracker-axis-label { + font-family: var(--font-interface); } + +.tracker-axis { + color: var(--ui2); } + +/* Color schemes */ +/* Atom */ +.theme-dark.minimal-atom-dark { + --red:#e16d76; + --orange:#d19a66; + --yellow:#cec167; + --green:#98c379; + --cyan:#58b6c2; + --blue:#62afef; + --purple:#c678de; + --pink:#e16d76; } + +.theme-light.minimal-atom-light { + --red:#e45749; + --orange:#b76b02; + --yellow:#c18302; + --green:#50a150; + --cyan:#0d97b3; + --blue:#62afef; + --purple:#a626a4; + --pink:#e45749; } + +.theme-light.minimal-atom-light { + --base-h:106; + --base-s:0%; + --base-l:98%; + --accent-h:209; + --accent-s:100%; + --accent-l:55%; + --bg1:#fafafa; + --bg2:#eaeaeb; + --bg3:#dbdbdc; + --ui1:#dbdbdc; + --ui2:#d8d8d9; + --tx1:#232324; + --tx2:#8e8e90; + --tx3:#a0a1a8; + --ax1:#1a92ff; + --ax3:#566de8; + --hl1:rgba(180,180,183,0.3); + --hl2:rgba(209,154,102,0.35); } + +.theme-light.minimal-atom-light.minimal-light-white { + --bg3:#eaeaeb; } + +.theme-light.minimal-atom-light.minimal-light-contrast .titlebar, +.theme-light.minimal-atom-light.minimal-light-contrast .workspace-fake-target-overlay.is-in-sidebar, +.theme-light.minimal-atom-light.minimal-light-contrast .workspace-ribbon.mod-left:not(.is-collapsed), +.theme-light.minimal-atom-light.minimal-light-contrast .mod-left-split, +.theme-light.minimal-atom-light.minimal-light-contrast.minimal-status-off .status-bar, +.theme-light.minimal-atom-light.minimal-light-contrast.is-mobile .workspace-drawer.mod-left, +.theme-dark.minimal-atom-dark { + --base-h:220; + --base-s:12%; + --base-l:18%; + --accent-h:220; + --accent-s:86%; + --accent-l:65%; + --bg1:#282c34; + --bg2:#21252c; + --bg3:#3a3f4b; + --background-divider:#181a1f; + --tx1:#d8dae1; + --tx2:#898f9d; + --tx3:#5d6370; + --ax1:#578af2; + --ax3:#578af2; + --hl1:rgba(114,123,141,0.3); + --hl2:rgba(209,154,102,0.3); + --sp1:#fff; } + +.theme-dark.minimal-atom-dark.minimal-dark-black { + --base-d:5%; + --bg3:#282c34; + --background-divider:#282c34; } + +/* Dracula */ +.theme-dark.minimal-dracula-dark { + --red:#ff5555; + --yellow:#f1fa8c; + --green:#50fa7b; + --orange:#ffb86c; + --purple:#bd93f9; + --pink:#ff79c6; + --cyan:#8be9fd; + --blue:#6272a4; } + +.theme-light.minimal-dracula-light.minimal-light-contrast .titlebar, +.theme-light.minimal-dracula-light.minimal-light-contrast .workspace-fake-target-overlay.is-in-sidebar, +.theme-light.minimal-dracula-light.minimal-light-contrast .workspace-ribbon.mod-left:not(.is-collapsed), +.theme-light.minimal-dracula-light.minimal-light-contrast .mod-left-split, +.theme-light.minimal-dracula-light.minimal-light-contrast.minimal-status-off .status-bar, +.theme-light.minimal-dracula-light.minimal-light-contrast.is-mobile .workspace-drawer.mod-left, +.theme-dark.minimal-dracula-dark { + --base-h:232; + --base-s:16%; + --base-l:19%; + --accent-h:265; + --accent-s:89%; + --accent-l:78%; + --bg1:#282a37; + --bg2:#21222c; + --ui2:#44475a; + --ui3:#6272a4; + --tx1:#f8f8f2; + --tx2:#949FBE; + --tx3:#6272a4; + --ax3:#ff79c6; + --hl1:rgba(134, 140, 170, 0.3); + --hl2:rgba(189, 147, 249, 0.35); } + +.theme-dark.minimal-dracula-dark.minimal-dark-black { + --ui1:#282a36; } + +/* Everforest */ +.theme-light.minimal-everforest-light { + --red:#f85552; + --orange:#f57d26; + --yellow:#dfa000; + --green:#8da101; + --purple:#df69ba; + --pink:#df69ba; + --cyan:#35a77c; + --blue:#7fbbb3; } + +.theme-dark.minimal-everforest-dark { + --red:#e67e80; + --orange:#e69875; + --yellow:#dbbc7f; + --green:#a7c080; + --purple:#d699b6; + --pink:#d699b6; + --cyan:#83c092; + --blue:#7fbbb3; } + +.theme-light.minimal-everforest-light { + --base-h:46; + --base-s:87%; + --base-l:94%; + --accent-h:81; + --accent-s:37%; + --accent-l:52%; + --bg1:#FDF7E3; + --bg2:#EEEAD5; + --bg3:rgba(206,207,182,.5); + --ui1:#dfdbc8; + --ui2:#bdc3af; + --ui3:#bdc3af; + --tx1:#5C6A72; + --tx2:#829181; + --tx3:#a6b0a0; + --ax1:#93b259; + --ax2:#738555; + --ax3:#93b259; + --hl1:rgba(198,214,152,.4); + --hl2:rgba(222,179,51,.3); } + +.theme-light.minimal-everforest-light.minimal-light-tonal { + --bg2:#EEEAD5; } + +.theme-light.minimal-everforest-light.minimal-light-white { + --bg3:#f3efda; + --ui1:#edead5; } + +.theme-light.minimal-everforest-light.minimal-light-contrast .titlebar, +.theme-light.minimal-everforest-light.minimal-light-contrast .workspace-fake-target-overlay.is-in-sidebar, +.theme-light.minimal-everforest-light.minimal-light-contrast .workspace-ribbon.mod-left:not(.is-collapsed), +.theme-light.minimal-everforest-light.minimal-light-contrast .mod-left-split, +.theme-light.minimal-everforest-light.minimal-light-contrast.minimal-status-off .status-bar, +.theme-light.minimal-everforest-light.minimal-light-contrast.is-mobile .workspace-drawer.mod-left, +.theme-dark.minimal-everforest-dark { + --base-h:203; + --base-s:15%; + --base-l:23%; + --accent-h:81; + --accent-s:34%; + --accent-l:63%; + --bg1:#323D44; + --bg2:#2A343A; + --bg3:#414C54; + --bg3:rgba(78,91,100,0.5); + --ui1:#404c51; + --ui2:#4A555C; + --ui3:#525c62; + --tx1:#d3c6aa; + --tx2:#9da9a0; + --tx3:#7a8478; + --ax1:#A7C080; + --ax2:#c7cca3; + --ax3:#93b259; + --hl1:rgba(134,70,93,.5); + --hl2:rgba(147,185,96,.3); } + +.theme-dark.minimal-everforest-dark.minimal-dark-black { + --hl1:rgba(134,70,93,.4); + --ui1:#2b3339; } + +/* Gruvbox */ +.theme-dark.minimal-gruvbox-dark, +.theme-light.minimal-gruvbox-light { + --red:#cc241d; + --yellow:#d79921; + --green:#98971a; + --orange:#d65d0e; + --purple:#b16286; + --pink:#b16286; + --cyan:#689d6a; + --blue:#458588; } + +.theme-light.minimal-gruvbox-light { + --base-h:49; + --base-s:92%; + --base-l:89%; + --accent-h:24; + --accent-s:88%; + --accent-l:45%; + --bg1:#fcf2c7; + --bg2:#f2e6bd; + --bg3:#ebd9b3; + --ui1:#ebdbb2; + --ui2:#d5c4a1; + --ui3:#bdae93; + --tx1:#282828; + --tx2:#7c7065; + --tx3:#a89a85; + --ax1:#d65d0e; + --ax2:#af3a03; + --ax3:#d65d0d; + --hl1:rgba(192,165,125,.3); + --hl2:rgba(215,153,33,.4); } + +.theme-light.minimal-gruvbox-light.minimal-light-tonal { + --bg2:#fcf2c7; } + +.theme-light.minimal-gruvbox-light.minimal-light-white { + --bg3:#faf5d7; + --ui1:#f2e6bd; } + +.theme-light.minimal-gruvbox-light.minimal-light-contrast .titlebar, +.theme-light.minimal-gruvbox-light.minimal-light-contrast .workspace-fake-target-overlay.is-in-sidebar, +.theme-light.minimal-gruvbox-light.minimal-light-contrast .workspace-ribbon.mod-left:not(.is-collapsed), +.theme-light.minimal-gruvbox-light.minimal-light-contrast .mod-left-split, +.theme-light.minimal-gruvbox-light.minimal-light-contrast.minimal-status-off .status-bar, +.theme-light.minimal-gruvbox-light.minimal-light-contrast.is-mobile .workspace-drawer.mod-left, +.theme-dark.minimal-gruvbox-dark { + --accent-h:24; + --accent-s:88%; + --accent-l:45%; + --bg1:#282828; + --bg2:#1e2021; + --bg3:#3d3836; + --bg3:rgba(62,57,55,0.5); + --ui1:#3c3836; + --ui2:#504945; + --ui3:#665c54; + --tx1:#fbf1c7; + --tx2:#bdae93; + --tx3:#7c6f64; + --ax1:#d65d0e; + --ax2:#fe8019; + --ax3:#d65d0e; + --hl1:rgba(173,149,139,0.3); + --hl2:rgba(215,153,33,.4); } + +.theme-dark.minimal-gruvbox-dark.minimal-dark-black { + --hl1:rgba(173,149,139,0.4); + --ui1:#282828; } + +/* macOS */ +.theme-dark.minimal-macos-dark, +.theme-light.minimal-macos-light { + --red:#ff3b31; + --yellow:#ffcc00; + --green:#2acd41; + --orange:#ff9502; + --purple:#b051de; + --pink:#ff2e55; + --cyan:#02c7be; + --blue:#027aff; } + +.theme-light.minimal-macos-light { + --base-h:106; + --base-s:0%; + --base-l:94%; + --accent-h:212; + --accent-s:100%; + --accent-l:50%; + --bg1:#fff; + --bg2:#f0f0f0; + --bg3:#d7d7d7; + --ui1:#e7e7e7; + --tx1:#454545; + --tx2:#808080; + --tx3:#b0b0b0; + --ax1:#027aff; + --ax2:#0463cc; + --ax3:#007bff; + --hl1:#b3d7ff; } + +.theme-light.minimal-macos-light.minimal-light-tonal { + --bg1:#f0f0f0; + --bg2:#f0f0f0; } + +.theme-light.minimal-macos-light.minimal-light-contrast .titlebar, +.theme-light.minimal-macos-light.minimal-light-contrast .workspace-fake-target-overlay.is-in-sidebar, +.theme-light.minimal-macos-light.minimal-light-contrast .workspace-ribbon.mod-left:not(.is-collapsed), +.theme-light.minimal-macos-light.minimal-light-contrast .mod-left-split, +.theme-light.minimal-macos-light.minimal-light-contrast.minimal-status-off .status-bar, +.theme-light.minimal-macos-light.minimal-light-contrast.is-mobile .workspace-drawer.mod-left, +.theme-dark.minimal-macos-dark { + --base-h:106; + --base-s:0%; + --base-l:12%; + --accent-h:212; + --accent-s:100%; + --accent-l:50%; + --bg1:#1e1e1e; + --bg2:#282828; + --bg3:rgba(255,255,255,0.11); + --background-divider:#000; + --ui1:#373737; + --ui2:#515151; + --ui3:#595959; + --tx1:#dcdcdc; + --tx2:#8c8c8c; + --tx3:#686868; + --ax1:#027aff; + --ax2:#3f9bff; + --ax3:#007bff; + --hl1:rgba(98,169,252,0.5); + --sp1:#fff; } + +.theme-dark.minimal-macos-dark.minimal-dark-black { + --background-divider:#1e1e1e; } + +/* Nord */ +.theme-dark.minimal-nord-dark, +.theme-light.minimal-nord-light { + --red:#BF616A; + --yellow:#EBCB8B; + --green:#A3BE8C; + --orange:#D08770; + --purple:#B48EAD; + --pink:#B48EAD; + --cyan:#88C0D0; + --blue:#81A1C1; } + +.theme-light.minimal-nord-light { + --base-h:221; + --base-s:27%; + --base-l:94%; + --accent-h:213; + --accent-s:32%; + --accent-l:52%; + --bg1:#fff; + --bg2:#eceff4; + --bg3:rgba(157,174,206,0.25); + --ui1:#d8dee9; + --ui2:#BBCADC; + --ui3:#81a1c1; + --tx1:#2e3440; + --tx2:#7D8697; + --tx3:#ADB1B8; + --ax1:#5e81ac; + --ax2:#81a1c1; + --hl2:rgba(208, 135, 112, 0.35); } + +.theme-light.minimal-nord-light.minimal-light-contrast .titlebar, +.theme-light.minimal-nord-light.minimal-light-contrast .workspace-fake-target-overlay.is-in-sidebar, +.theme-light.minimal-nord-light.minimal-light-contrast .workspace-ribbon.mod-left:not(.is-collapsed), +.theme-light.minimal-nord-light.minimal-light-contrast .mod-left-split, +.theme-light.minimal-nord-light.minimal-light-contrast.minimal-status-off .status-bar, +.theme-light.minimal-nord-light.minimal-light-contrast.is-mobile .workspace-drawer.mod-left, +.theme-dark.minimal-nord-dark { + --base-h:220; + --base-s:16%; + --base-l:22%; + --accent-h:213; + --accent-s:32%; + --accent-l:52%; + --bg1:#2e3440; + --bg2:#3b4252; + --bg3:rgba(135,152,190,0.15); + --ui1:#434c5e; + --ui2:#58647b; + --ui3:#5e81ac; + --tx1:#d8dee9; + --tx2:#9eafcc; + --tx3:#4c566a; + --ax3:#5e81ac; + --hl1:rgba(129,142,180,0.3); + --hl2:rgba(208, 135, 112, 0.35); } + +.theme-dark.minimal-nord-dark.minimal-dark-black { + --ui1:#2e3440; } + +/* Notion */ +.theme-light.minimal-notion-light { + --base-h:39; + --base-s:18%; + --base-d:96%; + --accent-h:197; + --accent-s:65%; + --accent-l:71%; + --bg2:#f7f6f4; + --bg3:#e8e7e4; + --ui1:#ededec; + --ui2:#dbdbda; + --ui3:#aaa9a5; + --tx1:#37352f; + --tx2:#72706c; + --tx3:#aaa9a5; + --ax1:#37352f; + --ax2:#000; + --ax3:#2eaadc; + --hl1:rgba(131,201,229,0.3); + --link-weight:500; } + +.theme-light.minimal-notion-light.minimal-light-contrast .titlebar, +.theme-light.minimal-notion-light.minimal-light-contrast .workspace-fake-target-overlay.is-in-sidebar, +.theme-light.minimal-notion-light.minimal-light-contrast .workspace-ribbon.mod-left:not(.is-collapsed), +.theme-light.minimal-notion-light.minimal-light-contrast .mod-left-split, +.theme-light.minimal-notion-light.minimal-light-contrast.minimal-status-off .status-bar, +.theme-light.minimal-notion-light.minimal-light-contrast.is-mobile .workspace-drawer.mod-left, +.theme-dark.minimal-notion-dark { + --base-h:203; + --base-s:8%; + --base-d:20%; + --accent-h:197; + --accent-s:48%; + --accent-l:43%; + --bg1:#2f3437; + --bg2:#373c3f; + --bg3:#4b5053; + --ui1:#3e4245; + --ui2:#585d5f; + --ui3:#585d5f; + --tx1:#ebebeb; + --tx2:#909295; + --tx3:#585d5f; + --ax1:#ebebeb; + --ax2:#fff; + --ax3:#2eaadc; + --hl1:rgba(57,134,164,0.3); + --link-weight:500; } + +.theme-dark.minimal-notion-dark.minimal-dark-black { + --base-d:5%; + --bg3:#232729; + --ui1:#2f3437; } + +/* Solarized */ +.theme-dark.minimal-solarized-dark, +.theme-light.minimal-solarized-light { + --red:#dc322f; + --orange:#cb4b16; + --yellow:#b58900; + --green:#859900; + --cyan:#2aa198; + --blue:#268bd2; + --purple:#6c71c4; + --pink:#d33682; } + +.theme-light.minimal-solarized-light { + --base-h:44; + --base-s:87%; + --base-l:94%; + --accent-h:205; + --accent-s:70%; + --accent-l:48%; + --bg1:#fdf6e3; + --bg2:#eee8d5; + --bg3:rgba(0,0,0,0.062); + --ui1:#e9e1c8; + --ui2:#d0cab8; + --ui3:#d0cab8; + --tx1:#073642; + --tx2:#586e75; + --tx3:#ABB2AC; + --tx4:#586e75; + --ax1:#268bd2; + --hl1:rgba(202,197,182,0.3); + --hl2:rgba(203,75,22,0.3); } + +.theme-light.minimal-solarized-light.minimal-light-tonal { + --bg2:#fdf6e3; } + +.theme-light.minimal-solarized-light.minimal-light-contrast .titlebar, +.theme-light.minimal-solarized-light.minimal-light-contrast .workspace-fake-target-overlay.is-in-sidebar, +.theme-light.minimal-solarized-light.minimal-light-contrast .workspace-ribbon.mod-left:not(.is-collapsed), +.theme-light.minimal-solarized-light.minimal-light-contrast .mod-left-split, +.theme-light.minimal-solarized-light.minimal-light-contrast.minimal-status-off .status-bar, +.theme-light.minimal-solarized-light.minimal-light-contrast.is-mobile .workspace-drawer.mod-left, +.theme-dark.minimal-solarized-dark { + --accent-h:205; + --accent-s:70%; + --accent-l:48%; + --base-h:193; + --base-s:98%; + --base-l:11%; + --bg1:#002b36; + --bg2:#073642; + --bg3:rgba(255,255,255,0.062); + --ui1:#19414B; + --ui2:#274850; + --ui3:#31535B; + --tx1:#93a1a1; + --tx2:#657b83; + --tx3:#31535B; + --tx4:#657b83; + --ax1:#268bd2; + --ax3:#268bd2; + --hl1:rgba(15,81,98,0.3); + --hl2:rgba(203, 75, 22, 0.35); } + +.theme-dark.minimal-solarized-dark.minimal-dark-black { + --hl1:rgba(15,81,98,0.55); + --ui1:#002b36; } + +/* Things */ +.theme-dark.minimal-things-dark, +.theme-light.minimal-things-light { + --red:#FF306C; + --yellow:#FFD500; + --green:#4BBF5E; + --orange:#ff9502; + --purple:#b051de; + --pink:#ff2e55; + --cyan:#49AEA4; } + +.theme-light.minimal-things-light { + --blue:#1b61c2; } + +.theme-dark.minimal-things-dark { + --blue:#4d95f7; } + +.theme-light.minimal-things-light { + --accent-h:215; + --accent-s:76%; + --accent-l:43%; + --bg1:white; + --bg2:#f5f6f8; + --bg3:rgba(162,177,187,0.25); + --ui1:#eef0f4; + --ui2:#D8DADD; + --ui3:#c1c3c6; + --tx1:#26272b; + --tx2:#7D7F84; + --tx3:#a9abb0; + --ax1:#1b61c2; + --ax2:#1C88DD; + --ax3:#1b61c2; + --hl1:#cae2ff; } + +.theme-light.minimal-things-light.minimal-light-tonal { + --ui1:#e6e8ec; } + +.theme-light.minimal-things-light.minimal-light-white { + --bg3:#f5f6f8; } + +.theme-light.minimal-things-light.minimal-light-contrast .titlebar, +.theme-light.minimal-things-light.minimal-light-contrast .workspace-fake-target-overlay.is-in-sidebar, +.theme-light.minimal-things-light.minimal-light-contrast .workspace-ribbon.mod-left:not(.is-collapsed), +.theme-light.minimal-things-light.minimal-light-contrast .mod-left-split, +.theme-light.minimal-things-light.minimal-light-contrast.minimal-status-off .status-bar, +.theme-light.minimal-things-light.minimal-light-contrast.is-mobile .workspace-drawer.mod-left, +.theme-dark.minimal-things-dark { + --base-h:218; + --base-s:9%; + --base-l:15%; + --accent-h:215; + --accent-s:91%; + --accent-l:64%; + --bg1:#24262a; + --bg2:#202225; + --bg3:#3d3f41; + --background-divider:#17191c; + --ui1:#3A3B3F; + --ui2:#45464a; + --ui3:#6c6e70; + --tx1:#fbfbfb; + --tx2:#CBCCCD; + --tx3:#6c6e70; + --ax1:#4d95f7; + --ax2:#79a9ec; + --ax3:#4d95f7; + --hl1:rgba(40,119,236,0.35); + --sp1:#fff; } + +.theme-dark.minimal-things-dark.minimal-dark-black { + --base-d:5%; + --bg3:#24262a; + --background-divider:#24262a; } +/* Plugin compatibility */ + +/* @plugins +core: +- backlink +- command-palette +- daily-notes +- file-explorer +- file-recovery +- global-search +- graph +- outgoing-link +- outline +- page-preview +- publish +- random-note +- starred +- switcher +- sync +- tag-pane +- word-count +community: +- buttons +- dataview +- calendar +- obsidian-charts +- obsidian-checklist-plugin +- obsidian-codemirror-options +- obsidian-dictionary-plugin +- obsidian-embedded-note-titles +- obsidian-excalidraw-plugin +- obsidian-git +- obsidian-hider +- obsidian-hover-editor +- obsidian-kanban +- obsidian-metatable +- obsidian-minimal-settings +- obsidian-outliner +- obsidian-system-dark-mode +- obsidian-style-settings +- quickadd +- sliding-panes-obsidian +- todoist-sync-plugin +*/ +/* @settings + +name: Minimal +id: minimal-style +settings: + - + id: instructions + title: Welcome 👋 + type: heading + level: 2 + collapsed: true + description: Use the Minimal Theme Settings plugin to access hotkeys, adjust features, select fonts, and choose from preset color schemes. Use the settings below for more granular customization. Visit minimal.guide for documentation. + - + id: interface + title: Interface colors + type: heading + level: 2 + collapsed: true + - + id: base + title: Base color + description: Defines all background and border colors unless overridden in more granular settings + type: variable-themed-color + format: hsl-split + default-light: '#' + default-dark: '#' + - + id: accent + title: Accent color + description: Defines link and checkbox colors unless overridden in more granular settings + type: variable-themed-color + format: hsl-split + default-light: '#' + default-dark: '#' + - + id: bg1 + title: Primary background + description: Background color for the main window + type: variable-themed-color + format: hex + default-light: '#' + default-dark: '#' + - + id: bg2 + title: Secondary background + description: Background color for left sidebar and menus + type: variable-themed-color + format: hex + default-light: '#' + default-dark: '#' + - + id: bg3 + title: Active background + description: Background color for hovered buttons and currently selected file + type: variable-themed-color + opacity: true + format: hex + default-light: '#' + default-dark: '#' + - + id: ui1 + title: Border color + type: variable-themed-color + description: For buttons, divider lines, and outlined elements + opacity: true + format: hex + default-light: '#' + default-dark: '#' + - + id: ui2 + title: Highlighted border color + description: Used when hovering over buttons, dividers, and outlined elements + type: variable-themed-color + opacity: true + format: hex + default-light: '#' + default-dark: '#' + - + id: ui3 + title: Active border color + description: Used when clicking buttons and outlined elements + type: variable-themed-color + opacity: true + format: hex + default-light: '#' + default-dark: '#' + - + id: extended-palette + title: Interface extended palette + type: heading + level: 2 + collapsed: true + - + id: red + title: Red + description: Extended palette colors are defaults used for progress bar status, syntax highlighting, colorful headings, and graph nodes + type: variable-themed-color + opacity: true + format: hex + default-light: '#' + default-dark: '#' + - + id: orange + title: Orange + type: variable-themed-color + opacity: true + format: hex + default-light: '#' + default-dark: '#' + - + id: yellow + title: Yellow + type: variable-themed-color + opacity: true + format: hex + default-light: '#' + default-dark: '#' + - + id: green + title: Green + type: variable-themed-color + opacity: true + format: hex + default-light: '#' + default-dark: '#' + - + id: cyan + title: Cyan + type: variable-themed-color + opacity: true + format: hex + default-light: '#' + default-dark: '#' + - + id: blue + title: Blue + type: variable-themed-color + opacity: true + format: hex + default-light: '#' + default-dark: '#' + - + id: purple + title: Purple + type: variable-themed-color + opacity: true + format: hex + default-light: '#' + default-dark: '#' + - + id: pink + title: Pink + type: variable-themed-color + opacity: true + format: hex + default-light: '#' + default-dark: '#' + - + id: active-line + title: Active line + type: heading + level: 2 + collapsed: true + - + id: active-line-on + title: Highlight active line + description: Adds a background to current line in editor + type: class-toggle + default: false + - + id: active-line-bg + title: Active line background + description: Using a low opacity color is recommended to avoid conflicting with highlights + type: variable-themed-color + opacity: true + format: hex + default-light: '#' + default-dark: '#' + - + id: blockquotes + title: Blockquotes + type: heading + level: 2 + collapsed: true + - + id: text-blockquote + title: Blockquotes text color + type: variable-themed-color + format: hex + default-light: '#' + default-dark: '#' + - + id: blockquote-size + title: Blockquotes font size + description: Accepts any CSS font-size value + type: variable-text + default: '' + - + id: blockquote-style + title: Blockquotes font style + type: variable-select + allowEmpty: false + default: normal + options: + - + label: Normal + value: normal + - + label: Italic + value: italic + - + id: code-blocks + title: Code blocks + type: heading + level: 2 + collapsed: true + - + id: text-code + title: Code text color + description: Color of code when syntax highlighting is not present + type: variable-themed-color + format: hex + default-light: '#' + default-dark: '#' + - + id: font-code + title: Code font size + description: Accepts any CSS font-size value + type: variable-text + default: 13px + - + id: embed-blocks + title: Embeds and transclusions + type: heading + level: 2 + collapsed: true + - + id: embed-strict + title: Use strict embed style globally + description: Transclusions appear seamlessly in the flow of text. Can be enabled per file using the embed-strict helper class + type: class-toggle + default: false + - + id: graphs + title: Graphs + type: heading + level: 2 + collapsed: true + - + id: node + title: Node color + description: Changing node colors requires closing and reopening graph panes or restarting Obsidian + type: variable-themed-color + format: hex + default-light: '#' + default-dark: '#' + - + id: node-focused + title: Active node color + type: variable-themed-color + format: hex + default-light: '#' + default-dark: '#' + - + id: node-tag + title: Tag node color + type: variable-themed-color + format: hex + default-light: '#' + default-dark: '#' + - + id: node-attachment + title: Attachment node color + type: variable-themed-color + format: hex + default-light: '#' + default-dark: '#' + - + id: node-unresolved + title: Unresolved node color + type: variable-themed-color + format: hex + default-light: '#' + default-dark: '#' + - + id: headings + title: Headings + type: heading + level: 2 + collapsed: true + - + id: level-1-headings + title: Level 1 Headings + type: heading + level: 3 + collapsed: true + - + id: h1-font + title: H1 font + description: Name of the font as it appears on your system + type: variable-text + default: '' + - + id: h1 + title: H1 font size + description: Accepts any CSS font-size value + type: variable-text + default: 1.125em + - + id: h1-weight + title: H1 font weight + type: variable-number-slider + default: 600 + min: 100 + max: 900 + step: 100 + - + id: h1-color + title: H1 text color + type: variable-themed-color + format: hex + default-light: '#' + default-dark: '#' + - + id: h1-variant + title: H1 font variant + type: variable-select + allowEmpty: false + default: normal + options: + - + label: Normal + value: normal + - + label: Small caps + value: small-caps + - + label: All small caps + value: all-small-caps + - + id: h1-style + title: H1 font style + type: variable-select + allowEmpty: false + default: normal + options: + - + label: Normal + value: normal + - + label: Italic + value: italic + - + id: h1-l + title: H1 divider line + description: Adds a border below the heading + type: class-toggle + default: false + - + id: level-2-headings + title: Level 2 Headings + type: heading + level: 3 + collapsed: true + - + id: h2-font + title: H2 font + description: Name of the font as it appears on your system + type: variable-text + default: '' + - + id: h2 + title: H2 font size + description: Accepts any CSS font-size value + type: variable-text + default: 1em + - + id: h2-weight + title: H2 font weight + type: variable-number-slider + default: 600 + min: 100 + max: 900 + step: 100 + - + id: h2-color + title: H2 text color + type: variable-themed-color + format: hex + default-light: '#' + default-dark: '#' + - + id: h2-variant + title: H2 font variant + type: variable-select + allowEmpty: false + default: normal + options: + - + label: Normal + value: normal + - + label: Small caps + value: small-caps + - + label: All small caps + value: all-small-caps + - + id: h2-style + title: H2 font style + type: variable-select + allowEmpty: false + default: normal + options: + - + label: Normal + value: normal + - + label: Italic + value: italic + - + id: h2-l + title: H2 divider line + description: Adds a border below the heading + type: class-toggle + default: false + - + id: level-3-headings + title: Level 3 Headings + type: heading + level: 3 + collapsed: true + - + id: h3-font + title: H3 font + description: Name of the font as it appears on your system + type: variable-text + default: '' + - + id: h3 + title: H3 font size + description: Accepts any CSS font-size value + type: variable-text + default: 1em + - + id: h3-weight + title: H3 font weight + type: variable-number-slider + default: 600 + min: 100 + max: 900 + step: 100 + - + id: h3-color + title: H3 text color + type: variable-themed-color + format: hex + default-light: '#' + default-dark: '#' + - + id: h3-variant + title: H3 font variant + type: variable-select + allowEmpty: false + default: normal + options: + - + label: Normal + value: normal + - + label: Small caps + value: small-caps + - + label: All small caps + value: all-small-caps + - + id: h3-style + title: H3 font style + type: variable-select + allowEmpty: false + default: normal + options: + - + label: Normal + value: normal + - + label: Italic + value: italic + - + id: h3-l + title: H3 divider line + description: Adds a border below the heading + type: class-toggle + default: false + - + id: level-4-headings + title: Level 4 Headings + type: heading + level: 3 + collapsed: true + - + id: h4-font + title: H4 font + description: Name of the font as it appears on your system + type: variable-text + default: '' + - + id: h4 + title: H4 font size + description: Accepts any CSS font-size value + type: variable-text + default: 0.9em + - + id: h4-weight + title: H4 font weight + type: variable-number-slider + default: 500 + min: 100 + max: 900 + step: 100 + - + id: h4-color + title: H4 text color + type: variable-themed-color + format: hex + default-light: '#' + default-dark: '#' + - + id: h4-variant + title: H4 font variant + type: variable-select + allowEmpty: false + default: small-caps + options: + - + label: Normal + value: normal + - + label: Small caps + value: small-caps + - + label: All small caps + value: all-small-caps + - + id: h4-style + title: H4 font style + type: variable-select + allowEmpty: false + default: normal + options: + - + label: Normal + value: normal + - + label: Italic + value: italic + - + id: h4-l + title: H4 divider line + description: Adds a border below the heading + type: class-toggle + default: false + - + id: level-5-headings + title: Level 5 Headings + type: heading + level: 3 + collapsed: true + - + id: h5-font + title: H5 font + description: Name of the font as it appears on your system + type: variable-text + default: '' + - + id: h5 + title: H5 font size + description: Accepts any CSS font-size value + type: variable-text + default: 0.85em + - + id: h5-weight + title: H5 font weight + type: variable-number-slider + default: 500 + min: 100 + max: 900 + step: 100 + - + id: h5-color + title: H5 text color + type: variable-themed-color + format: hex + default-light: '#' + default-dark: '#' + - + id: h5-variant + title: H5 font variant + type: variable-select + allowEmpty: false + default: small-caps + options: + - + label: Normal + value: normal + - + label: Small caps + value: small-caps + - + label: All small caps + value: all-small-caps + - + id: h5-style + title: H5 font style + type: variable-select + allowEmpty: false + default: normal + options: + - + label: Normal + value: normal + - + label: Italic + value: italic + - + id: h5-l + title: H5 divider line + description: Adds a border below the heading + type: class-toggle + default: false + - + id: level-6-headings + title: Level 6 Headings + type: heading + level: 3 + collapsed: true + - + id: h6-font + title: H6 font + description: Name of the font as it appears on your system + type: variable-text + default: '' + - + id: h6 + title: H6 font size + description: Accepts any CSS font-size value + type: variable-text + default: 0.85em + - + id: h6-weight + title: H6 font weight + type: variable-number-slider + default: 400 + min: 100 + max: 900 + step: 100 + - + id: h6-color + title: H6 text color + type: variable-themed-color + format: hex + default-light: '#' + default-dark: '#' + - + id: h6-variant + title: H6 font variant + type: variable-select + allowEmpty: false + default: small-caps + options: + - + label: Normal + value: normal + - + label: Small caps + value: small-caps + - + label: All small caps + value: all-small-caps + - + id: h6-style + title: H6 font style + type: variable-select + allowEmpty: false + default: normal + options: + - + label: Normal + value: normal + - + label: Italic + value: italic + - + id: h6-l + title: H6 divider line + type: class-toggle + description: Adds a border below the heading + default: false + - + id: icons + title: Icons + type: heading + level: 2 + collapsed: true + - + id: icon-muted + title: Icon opacity (inactive) + type: variable-number-slider + default: 0.5 + min: 0.25 + max: 1 + step: 0.05 + - + id: icon-color + title: Icon color + type: variable-themed-color + opacity: true + format: hex + default-light: '#' + default-dark: '#' + - + id: icon-color-hover + title: Icon color (hover) + type: variable-themed-color + opacity: true + format: hex + default-light: '#' + default-dark: '#' + - + id: icon-color-active + title: Icon color (active) + type: variable-themed-color + opacity: true + format: hex + default-light: '#' + default-dark: '#' + - + id: images + title: Images + type: heading + level: 2 + collapsed: true + - + id: image-muted + title: Image opacity in dark mode + description: Level of fading for images in dark mode. Hover over images to display at full brightness. + type: variable-number-slider + default: 0.7 + min: 0.25 + max: 1 + step: 0.05 + - + id: zoom-off + title: Disable image zoom + description: Turns off click + hold to zoom images + type: class-toggle + - + id: indentation-guides + title: Indentation guides + type: heading + level: 2 + collapsed: true + - + id: ig-adjust-reading + title: Horizontal adjustment in reading mode + type: variable-number-slider + default: -0.65 + min: -1.2 + max: 0 + step: 0.05 + format: em + - + id: ig-adjust-edit + title: Horizontal adjustment in edit mode + type: variable-number-slider + default: -1 + min: -10 + max: 10 + step: 1 + format: px + - + id: links + title: Links + type: heading + level: 2 + collapsed: true + - + id: ax1 + title: Link color + type: variable-themed-color + format: hex + default-light: '#' + default-dark: '#' + - + id: ax2 + title: Link color (hovering) + type: variable-themed-color + format: hex + default-light: '#' + default-dark: '#' + - + id: link-weight + title: Link font weight + type: variable-number-slider + default: 400 + min: 100 + max: 900 + step: 100 + - + id: lists + title: Lists and tasks + type: heading + level: 2 + collapsed: true + - + id: ax3 + title: Checkbox color + description: Background color for completed tasks + type: variable-themed-color + format: hex + default-light: '#' + default-dark: '#' + - + id: checkbox-shape + title: Checkbox shape + type: class-select + allowEmpty: false + default: checkbox-circle + options: + - + label: Circle + value: checkbox-circle + - + label: Square + value: checkbox-square + - + id: minimal-strike-lists + title: Strike completed tasks + description: Adds strikethrough line and greyed text for completed tasks + type: class-toggle + default: false + - + id: list-spacing + title: List item spacing + description: Vertical space between list items in em units + type: variable-number-slider + default: 0.075 + min: 0 + max: 0.3 + step: 0.005 + format: em + - + id: list-indent + title: Nested list indentation + description: Horizontal space from left in em units + type: variable-number-slider + default: 2 + min: 1 + max: 3.5 + step: 0.1 + format: em + - + id: sidebars + title: Sidebars + type: heading + level: 2 + collapsed: true + - + id: tab-style + title: Tab style + description: See documentation for screenshots + type: class-select + allowEmpty: false + default: tab-style-1 + options: + - + label: Compact + value: tab-style-1 + - + label: Pill + value: tab-style-3 + - + label: Underlined + value: tab-style-2 + - + label: Index + value: tab-style-4 + - + id: sidebar-lines-off + title: Disable sidebar relationship lines + description: Turns off lines in file navigation + type: class-toggle + - + id: mobile-left-sidebar-width + title: Mobile left sidebar width + description: Maximum width for pinned left sidebar on mobile + type: variable-number + default: 280 + format: pt + - + id: mobile-right-sidebar-width + title: Mobile right sidebar width + description: Maximum width for pinned right sidebar on mobile + type: variable-number + default: 240 + format: pt + - + id: tables + title: Tables + type: heading + level: 2 + collapsed: true + - + id: table-font-size + title: Table font size + description: All of the following settings apply to all tables globally. To turn on these features on a per-note basis use helper classes. See documentation. + type: variable-text + default: 1em + - + id: row-lines + title: Row lines + description: Display borders between table rows globally + type: class-toggle + default: false + - + id: col-lines + title: Column lines + description: Display borders between table columns globally + type: class-toggle + default: false + - + id: table-lines + title: Cell lines + description: Display borders around all table cells globally + type: class-toggle + default: false + - + id: row-alt + title: Striped rows + description: Display striped background in alternating table rows globally + type: class-toggle + default: false + - + id: col-alt + title: Striped columns + description: Display striped background in alternating table columns globally + type: class-toggle + default: false + - + id: table-tabular + title: Tabular figures + description: Use fixed width numbers in tables globally + type: class-toggle + default: false + - + id: table-numbers + title: Row numbers + description: Display row numbers in tables globally + type: class-toggle + default: false + - + id: table-nowrap + title: Disable line wrap + description: Turn off line wrapping in table cells globally + type: class-toggle + default: false + - + id: row-hover + title: Highlight active row + description: Highlight rows on hover + type: class-toggle + default: false + - + id: row-color-hover + title: Active row background + description: Background color for hovered tables rows + type: variable-themed-color + format: hex + opacity: true + default-light: '#' + default-dark: '#' + - + id: tags + title: Tags + type: heading + level: 2 + collapsed: true + - + id: minimal-unstyled-tags + title: Plain tags + description: Tags will render as normal text, overrides settings below + type: class-toggle + default: false + - + id: tag-radius + title: Tag shape + type: variable-select + default: 14px + options: + - + label: Pill + value: 14px + - + label: Rounded + value: 4px + - + label: Square + value: 0px + - + id: tag-border-width + title: Tag border width + type: variable-select + default: 1px + options: + - + label: None + value: 0 + - + label: Thin + value: 1px + - + label: Thick + value: 2px + - + id: tag-color + title: Tag text color + type: variable-themed-color + opacity: true + format: hex + default-light: '#' + default-dark: '#' + - + id: tag-bg + title: Tag background color + type: variable-themed-color + opacity: true + format: hex + default-light: '#' + default-dark: '#' + - + id: tag-bg2 + title: Tag background color (hover) + type: variable-themed-color + opacity: true + format: hex + default-light: '#' + default-dark: '#' + - + id: text + title: Text + type: heading + level: 2 + collapsed: true + - + id: tx1 + title: Normal text color + type: variable-themed-color + description: Primary text color used by default across all elements + opacity: true + format: hex + default-light: '#' + default-dark: '#' + - + id: hl1 + title: Selected text background + type: variable-themed-color + opacity: true + format: hex + default-light: '#' + default-dark: '#' + - + id: hl2 + title: Highlighted text background + type: variable-themed-color + opacity: true + format: hex + default-light: '#' + default-dark: '#' + - + id: tx2 + title: Muted text color + description: Secondary text such as sidebar note titles and table headings + type: variable-themed-color + opacity: true + format: hex + default-light: '#' + default-dark: '#' + - + id: tx3 + title: Faint text color + description: tertiary text such as input placeholders, empty checkboxes, and disabled statuses + type: variable-themed-color + opacity: true + format: hex + default-light: '#' + default-dark: '#' + - + id: text-italic + title: Italic text color + type: variable-themed-color + format: hex + default-light: '#' + default-dark: '#' + - + id: text-bold + title: Bold text color + type: variable-themed-color + format: hex + default-light: '#' + default-dark: '#' + - + id: bold-weight + title: Bold text weight + type: variable-number-slider + default: 600 + min: 100 + max: 900 + step: 100 + - + id: spacing-p + title: Paragraph spacing + description: Space between paragraphs in reading mode + type: variable-text + default: 0.75em + - + id: titlebar + title: Title bar + type: heading + level: 2 + collapsed: true + - + id: title-alignment + title: Title alignment + description: Position of the text within the title bar + type: class-select + allowEmpty: false + default: title-align-body + options: + - + label: Text body + value: title-align-body + - + label: Left + value: title-align-left + - + label: Center + value: title-align-center + - + id: show-grabber + title: Always show grabber icon + description: Make the dragging handle always visible in the top left corner of a pane + type: class-toggle + default: false + - + id: header-height + title: Title bar height + type: variable-text + default: 42px + - + id: title-size + title: Title font size + description: Accepts any CSS font-size value + type: variable-text + default: 1.1em + - + id: title-weight + title: Title font weight + type: variable-number-slider + default: 600 + min: 100 + max: 900 + step: 100 + - + id: title-color + title: Title text color (active) + type: variable-themed-color + format: hex + default-light: '#' + default-dark: '#' + - + id: title-color-inactive + title: Title text color (inactive) + type: variable-themed-color + format: hex + default-light: '#' + default-dark: '#' + - + id: translucency + title: Translucency + type: heading + level: 2 + collapsed: true + - + id: bg-translucency-light + title: Translucency (light mode) + description: Sidebar translucency in light mode. Requires turning on "Translucent window" in Appearance settings, and "Translucent sidebar" in Minimal settings. + type: variable-number-slider + default: 0.7 + min: 0 + max: 1 + step: 0.05 + - + id: bg-translucency-dark + title: Translucency (dark mode) + description: Sidebar translucency in dark mode + type: variable-number-slider + default: 0.85 + min: 0 + max: 1 + step: 0.05 + +*/ + +/* @settings +name: Minimal Cards +id: minimal-cards-style +settings: + - + id: cards-min-width + title: Card minimum width + type: variable-text + default: 180px + - + id: cards-max-width + title: Card maximum width + description: Default fills the available width, accepts valid CSS units + type: variable-text + default: 1fr + - + id: cards-mobile-width + title: Card minimum width on mobile + type: variable-text + default: 120px + - + id: cards-padding + title: Card padding + type: variable-text + default: 1.2em + - + id: cards-image-height + title: Card maximum image height + type: variable-text + default: 400px + - + id: cards-border-width + title: Card border width + type: variable-text + default: 1px + - + id: cards-background + title: Card background color + type: variable-themed-color + format: hex + default-light: '#' + default-dark: '#' + +*/ + +/* @settings +name: Minimal Mobile +id: minimal-mobile +settings: + - + id: mobile-toolbar-off + title: Disable toolbar + description: Turns off mobile toolbar + type: class-toggle +*/ + +/* @settings +name: Minimal Advanced Settings +id: minimal-advanced +settings: + - + id: window-title-on + title: Display window title + description: Display title in the window frame + type: class-toggle + - + id: styled-scrollbars + title: Styled scrollbars + description: Use styled scrollbars (replaces native scrollbars) + type: class-toggle + - + id: cursor + title: Cursor style + description: The cursor style for UI elements + type: variable-select + default: default + options: + - + label: Default + value: default + - + label: Pointer + value: pointer + - + label: Crosshair + value: crosshair + - + id: font-smaller + title: Smaller font size + description: Font size in px of smaller text + type: variable-number + default: 11 + format: px + - + id: font-smallest + title: Smallest font size + description: Font size in px of smallest text + type: variable-number + default: 10 + format: px + - + id: folding-offset + title: Folding offset + description: Width of the left margin used for folding indicators + type: variable-number-slider + default: 10 + min: 0 + max: 30 + step: 1 + format: px + +*/ diff --git a/docs/.obsidian/themes/Obsidian Nord.css b/docs/.obsidian/themes/Obsidian Nord.css new file mode 100644 index 0000000000..808ba086dd --- /dev/null +++ b/docs/.obsidian/themes/Obsidian Nord.css @@ -0,0 +1,564 @@ + +:root +{ + --dark0: #2e3440; + --dark1: #3b4252; + --dark2: #434c5e; + --dark3: #4c566a; + + --light0: #d8dee9; + --light1: #e5e9f0; + --light2: #eceff4; + --light3: #ffffff; + + --frost0: #8fbcbb; + --frost1: #88c0d0; + --frost2: #81a1c1; + --frost3: #5e81ac; + + --red: #bf616a; + --orange: #d08770; + --yellow: #ebcb8b; + --green: #a3be8c; + --purple: #b48ead; +} + +.theme-dark +{ + --background-primary: var(--dark0); + --background-primary-alt: var(--dark0); + --background-secondary: var(--dark1); + --background-secondary-alt: var(--dark2); + --text-normal: var(--light2); + --text-faint: var(--light0); + --text-muted: var(--light1); + --text-title-h1: var(--red); + --text-title-h2: var(--orange); + --text-title-h3: var(--yellow); + --text-title-h4: var(--green); + --text-title-h5: var(--purple); + --text-title-h6: var(--orange); + --text-link: var(--frost0); + --text-a: var(--frost3); + --text-a-hover: var(--frost2); + --text-mark: rgba(136, 192, 208, 0.3); /* frost1 */ + --pre-code: var(--dark1); + --text-highlight-bg: rgba(163, 190, 140, 0.3); /* green */ + --text-highlight-bg-active: var(--green); + --interactive-accent: var(--frost0); + --interactive-before: var(--dark3); + --background-modifier-border: var(--dark2); + --text-accent: var(--orange); + --interactive-accent-rgb: var(--orange); + --inline-code: var(--frost1); + --code-block: var(--frost1); + --vim-cursor: var(--orange); + --text-selection: var(--dark3); + --text-tag: var(--frost0); + --task-checkbox: var(--frost0); + --table-header: hsl(220, 16%, 16%); + --table-row-even: hsl(220, 16%, 20%); + --table-row-odd: hsl(220, 16%, 24%); + --table-hover: var(--dark3); +} +.theme-light +{ + --background-primary: var(--light3); + --background-primary-alt: var(--light3); + --background-secondary: var(--light2); + --background-secondary-alt: var(--light1); + --text-normal: var(--dark1); + --text-faint: var(--dark3); + --text-muted: var(--dark2); + --text-title-h1: var(--red); + --text-title-h2: var(--orange); + --text-title-h3: var(--yellow); + --text-title-h4: var(--green); + --text-title-h5: var(--purple); + --text-title-h6: var(--orange); + --text-link: var(--frost0); + --text-a: var(--frost3); + --text-a-hover: var(--frost1); + --text-mark: rgba(136, 192, 208, 0.3); /* frost1 */ + --pre-code: var(--light2); + --text-highlight-bg: rgba(235, 203, 139, 0.6); /* yellow */ + --text-highlight-bg-active: var(--yellow); + --interactive-accent: var(--frost0); + --interactive-before: var(--light0); + --background-modifier-border: var(--light1); + --text-accent: var(--orange); + --interactive-accent-rgb: var(--orange); + --inline-code: var(--frost1); + --code-block: var(--frost1); + --vim-cursor: var(--orange); + --text-selection: var(--light0); + --text-tag: var(--frost2); + --task-checkbox: var(--frost0); + --table-header: hsl(218, 27%, 48%); + --table-row-even: hsl(220, 16%, 94%); + --table-row-odd: hsl(220, 16%, 98%); + --table-hover: var(--light1); +} + +body { + --font-text-theme: -apple-system, BlinkMacSystemFont, Segoe UI, Helvetica, Arial, sans-serif, Apple Color Emoji, Segoe UI Emoji; + + --font-monospace-theme: 'Hack Nerd Font', 'Source Code Pro', ui-monospace, SFMono-Regular, SF Mono, Menlo, Consolas, Liberation Mono, monospace; +} + +.theme-dark code[class*="language-"], +.theme-dark pre[class*="language-"], +.theme-light code[class*="language-"], +.theme-light pre[class*="language-"] +{ + text-shadow: none !important; + background-color: var(--pre-code) !important; +} + +.graph-view.color-circle, +.graph-view.color-fill-highlight, +.graph-view.color-line-highlight +{ + color: var(--interactive-accent-rgb) !important; +} +.graph-view.color-text +{ + color: var(--text-a-hover) !important; +} +/* +.graph-view.color-fill +{ + color: var(--background-secondary); +} +.graph-view.color-line +{ + color: var(--background-modifier-border); +} +*/ + +html, +body +{ + /* font-size: 16px !important; */ +} + +strong +{ + font-weight: 600 !important; +} + +a, +.cm-hmd-internal-link +{ + color: var(--text-a) !important; + text-decoration: none !important; +} + +a:hover, +.cm-hmd-internal-link:hover, +.cm-url +{ + color: var(--text-a-hover) !important; + text-decoration: none !important; +} + +a.tag, a.tag:hover { + color: var(--text-tag) !important; + background-color: var(--background-secondary-alt); + padding: 2px 4px; + border-radius: 4px; +} + +a.tag:hover { + text-decoration: underline !important; +} + +mark +{ + background-color: var(--text-mark); +} + +.titlebar { + background-color: var(--background-secondary-alt); +} + +.titlebar-inner { + color: var(--text-normal); +} + +.view-actions a +{ + color: var(--text-normal) !important; +} + +.view-actions a:hover +{ + color: var(--text-a) !important; +} + +.HyperMD-codeblock-bg +{ + background-color: var(--pre-code) !important; +} + +.HyperMD-codeblock +{ + line-height: 1.4em !important; + color: var(--code-block) !important; +} + +.HyperMD-codeblock-begin +{ + border-top-left-radius: 4px !important; + border-top-right-radius: 4px !important; +} + +.HyperMD-codeblock-end +{ + border-bottom-left-radius: 4px !important; + border-bottom-right-radius: 4px !important; +} + + +table { + border: 1px solid var(--background-secondary) !important; + border-collapse: collapse; +} + +th { + font-weight: 600 !important; + border: 0px !important; + text-align: left; + background-color: var(--table-header); + color: var(--frost0); +} + +td { + border-left: 0px !important; + border-right: 0px !important; + border-bottom: 1px solid var(--background-secondary) !important; +} + +tr:nth-child(even){ background-color: var(--table-row-even) } +tr:nth-child(odd){ background-color: var(--table-row-odd) } +tr:hover { background-color: var(--table-hover); } + +thead +{ + border-bottom: 2px solid var(--background-modifier-border) !important; +} + +.HyperMD-table-row +{ + line-height: normal !important; + padding-left: 4px !important; + padding-right: 4px !important; + /* background-color: var(--pre-code) !important; */ +} + +.HyperMD-table-row-0 +{ + /* padding-top: 4px !important; */ +} + +.CodeMirror-foldgutter-folded, +.is-collapsed .nav-folder-collapse-indicator +{ + color: var(--text-a) !important; +} + +.nav-file-tag +{ + color: var(--text-a) !important; +} + +.is-active .nav-file-title +{ + color: var(--text-a) !important; + background-color: var(--background-primary-alt) !important; +} + +.nav-file-title +{ + border-bottom-left-radius: 0 !important; + border-bottom-right-radius: 0 !important; + border-top-left-radius: 0 !important; + border-top-right-radius: 0 !important; +} + +.HyperMD-list-line +{ + padding-top: 0 !important; +} + +.cm-hashtag-begin +{ + color: var(--text-tag) !important; + background-color: var(--background-secondary-alt); + padding: 2px 0 2px 4px; + border-top-left-radius: 4px; + border-bottom-left-radius: 4px; + text-decoration: none !important; +} + +.cm-hashtag-end +{ + color: var(--text-tag) !important; + background-color: var(--background-secondary-alt); + padding: 2px 4px 2px 0; + border-top-right-radius: 4px; + border-bottom-right-radius: 4px; + text-decoration: none !important; +} + +.cm-hashtag-begin:hover, .cm-hashtag-end:hover +{ + text-decoration: underline !important; +} + +.search-result-file-matched-text +{ + color: var(--light3) !important; +} + +.markdown-preview-section pre code, +.markdown-preview-section code +{ + font-size: 0.9em !important; + background-color: var(--pre-code) !important; +} + +.markdown-preview-section pre code +{ + padding: 4px !important; + line-height: 1.4em !important; + display: block !important; + color: var(--code-block) !important; +} + +.markdown-preview-section code +{ + color: var(--inline-code) !important; +} + +.cm-s-obsidian, +.cm-inline-code +{ + -webkit-font-smoothing: auto !important; +} + +.cm-inline-code +{ + color: var(--inline-code) !important; + background-color: var(--pre-code) !important; + padding: 1px !important; +} + +.workspace-leaf-header-title +{ + font-weight: 600 !important; +} + +.side-dock-title +{ + padding-top: 15px !important; + font-size: 20px !important; +} + +.side-dock-ribbon-tab:hover, +.side-dock-ribbon-action:hover, +.side-dock-ribbon-action.is-active:hover, +.nav-action-button:hover, +.side-dock-collapse-btn:hover +{ + color: var(--text-a); +} + +.side-dock +{ + border-right: 0 !important; +} + +/* vertical resize-handle */ +.workspace-split.mod-vertical > * > .workspace-leaf-resize-handle, +.workspace-split.mod-left-split > .workspace-leaf-resize-handle, +.workspace-split.mod-right-split > .workspace-leaf-resize-handle +{ + width: 1px !important; + background-color: var(--background-secondary-alt); +} + +/* horizontal resize-handle */ +.workspace-split.mod-horizontal > * > .workspace-leaf-resize-handle +{ + height: 1px !important; + background-color: var(--background-secondary-alt); +} + +/* Remove vertical split padding */ +.workspace-split.mod-root .workspace-split.mod-vertical .workspace-leaf-content, +.workspace-split.mod-vertical > .workspace-split, +.workspace-split.mod-vertical > .workspace-leaf, +.workspace-tabs +{ + padding-right: 0px; +} + +.markdown-embed-title +{ + font-weight: 600 !important; +} + +.markdown-embed +{ + padding-left: 10px !important; + padding-right: 10px !important; + margin-left: 10px !important; + margin-right: 10px !important; +} + +.cm-header-1.cm-link, +h1 a +{ + color: var(--text-title-h1) !important; +} + +.cm-header-2.cm-link, +h2 a +{ + color: var(--text-title-h2) !important; +} + +.cm-header-3.cm-link, +h3 a +{ + color: var(--text-title-h3) !important; +} +.cm-header-4.cm-link, +h4 a +{ + color: var(--text-title-h4) !important; +} +.cm-header-5.cm-link, +h5 a +{ + color: var(--text-title-h5) !important; +} +.cm-header-6.cm-link, +h6 a +{ + color: var(--text-title-h6) !important; +} + +.cm-header { + font-weight: 500 !important; +} + +.HyperMD-header-1, +.markdown-preview-section h1 +{ + font-weight: 500 !important; + font-size: 2.2em !important; + color: var(--text-title-h1) !important; +} + +.HyperMD-header-2, +.markdown-preview-section h2 +{ + font-weight: 500 !important; + font-size: 2.0em !important; + color: var(--text-title-h2) !important; +} + +.HyperMD-header-3, +.markdown-preview-section h3 +{ + font-weight: 500 !important; + font-size: 1.8em !important; + color: var(--text-title-h3) !important; +} + +.HyperMD-header-4, +.markdown-preview-section h4 +{ + font-weight: 500 !important; + font-size: 1.6em !important; + color: var(--text-title-h4) !important; +} + +.HyperMD-header-5, +.markdown-preview-section h5 +{ + font-weight: 500 !important; + font-size: 1.4em !important; + color: var(--text-title-h5) !important; +} + +.HyperMD-header-6, +.markdown-preview-section h6 +{ + font-weight: 500 !important; + font-size: 1.2em !important; + color: var(--text-title-h6) !important; +} + +.suggestion-item.is-selected +{ + background-color: var(--background-secondary); +} + +.empty-state-action:hover +{ + color: var(--interactive-accent); +} + +.checkbox-container +{ + background-color: var(--interactive-before); +} + +.checkbox-container:after +{ + background-color: var(--background-secondary-alt); +} + +.mod-cta +{ + color: var(--background-secondary-alt) !important; + font-weight: 600 !important; +} + +.mod-cta:hover +{ + background-color: var(--interactive-before) !important; + font-weight: 600 !important; +} + +.CodeMirror-cursor +{ + background-color: var(--vim-cursor) !important; + opacity: 60% !important; +} + +input.task-list-item-checkbox { + border: 1px solid var(--task-checkbox); + appearance: none; + -webkit-appearance: none; +} + +input.task-list-item-checkbox:checked { + background-color: var(--task-checkbox); + box-shadow: inset 0 0 0 2px var(--background-primary); +} + +.mermaid .note +{ + fill: var(--frost3) !important; +} + +.setting-item-control input[type="text"] { + color: var(--text-normal); +} +.setting-item-control input[type="text"]::placeholder { + color: var(--dark3); +} diff --git a/docs/.obsidian/themes/Things.css b/docs/.obsidian/themes/Things.css new file mode 100644 index 0000000000..0ae4ac4edb --- /dev/null +++ b/docs/.obsidian/themes/Things.css @@ -0,0 +1,6967 @@ +/*─────────────────────────────────────────────────────── +THINGS +Version 1.8.2 +Created by @colineckert + +Readme: +https://github.com/colineckert/obsidian-things + +Support my work: +https://www.buymeacoffee.com/colineckert + +Support @kepano +https://www.buymeacoffee.com/kepano + +---------------------------------------------------------------- + +MIT License + +Copyright (c) 2020-2021 Stephan Ango (@kepano) + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. + +---------------------------------------------------------------- + +For help and/or CSS snippets, thanks to: +- @kepano +- @chetachiezikeuzor + +────────────────────────────────────────────────────── */ + +@charset "UTF-8"; +:root { + /*---------------------------------------------------------------- + COLORS + ----------------------------------------------------------------*/ + + --base-h: 212; /* Base hue */ + --base-s: 15%; /* Base saturation */ + --base-d: 13%; /* Base lightness Dark Mode - 0 is black */ + --base-l: 97%; /* Base lightness Light Mode - 100 is white */ + --accent-h: 215; /* Accent hue */ + --accent-s: 75%; /* Accent saturation */ + --accent-d: 70%; /* Accent lightness Dark Mode */ + --accent-l: 60%; /* Accent lightness Light Mode */ + + --blue: #2e80f2; + --pink: #ff82b2; + --green: #3eb4bf; + --yellow: #e5b567; + --orange: #e87d3e; + --red: #e83e3e; + --purple: #9e86c8; + + --light-yellow-highlighter: #fff3a3a6; + --dark-yellow-highlighter: #dbce7e77; + --pink-highlighter: #ffb8eba6; + --red-highlighter: #db3e606e; + --blue-highlighter: #adccffa6; + --dark-blue-highlighter: #adccff5b; + --green-highlighter: #bbfabba6; + --purple-highlighter: #d2b3ffa6; + --orange-highlighter: #ffb86ca6; + --grey-highlighter: #cacfd9a6; + + /* Colors, sizes, weights, padding */ + + --h1-color: var(--text-normal); + --h2-color: var(--blue); + --h3-color: var(--blue); + --h4-color: var(--yellow); + --h5-color: var(--red); + --h6-color: var(--text-muted); + + --strong-color: var(--pink); + --em-color: var(--pink); + + --font-normal: 16px; + --font-small: 13px; + --font-smaller: 11px; + --font-smallest: 10px; + + --font-settings: 15px; + --font-settings-small: 13px; + --font-inputs: 14px; + + --h1: 1.5em; + --h2: 1.3em; + --h3: 1.1em; + --h4: 0.9em; + --h5: 0.85em; + --h6: 0.85em; + + --h1-weight: 700; + --h2-weight: 700; + --h3-weight: 600; + --h4-weight: 500; + --h5-weight: 500; + --h6-weight: 400; + + --normal-weight: 400; /* Switch to 300 if you want thinner default text */ + --bold-weight: 700; /* Switch to 700 if you want thicker bold text */ + --icon-muted: 0.4; + --line-width: 45rem; /* Maximum characters per line */ + --line-height: 1.5; + --border-width: 1px; + --border-width-alt: 1px; + --max-width: 90%; /* Amount of padding around the text, use 90% for narrower padding */ + --nested-padding: 3.5%; /* Amount of padding for quotes and transclusions */ + --input-height: 36px; + --list-indent: 2em; + + --font-todoist-title-size: 1em; + --font-todoist-metadata-size: small; + + --cursor: default; + --h4-transform: uppercase; +} + +/* Desktop fonts */ +body { + /* Font families */ + --font-text-theme: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, + Inter, Ubuntu, sans-serif; + --font-editor-theme: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, + Inter, Ubuntu, sans-serif; + --font-monospace-theme: 'JetBrains', Menlo, SFMono-Regular, Consolas, + 'Roboto Mono', monospace; + --font-interface-theme: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, + Inter, Ubuntu, sans-serif; + --font-editor: var(--font-editor-override), var(--font-text-override), + var(--font-editor-theme); + + /* Font sizes */ + --font-adaptive-normal: var(--font-text-size, var(--editor-font-size)); + --font-adaptive-small: var(--font-small); + --font-adaptive-smaller: var(--font-smaller); + --font-adaptive-smallest: var(--font-smallest); + --line-width-adaptive: var(--line-width); + --line-width-wide: calc(var(--line-width) + 12.5%); + --font-code: calc(var(--font-adaptive-normal) * 0.9); +} + +/* Phone font sizes */ +@media (max-width: 400pt) { + .is-mobile { + --font-adaptive-small: calc(var(--font-small) + 2px); + --font-adaptive-smaller: calc(var(--font-smaller) + 2px); + --font-adaptive-smallest: calc(var(--font-smallest) + 2px); + --max-width: 88%; + } +} +/* Tablet font sizes */ +@media (min-width: 400pt) { + .is-mobile { + --font-adaptive-small: calc(var(--font-small) + 3px); + --font-adaptive-smaller: calc(var(--font-smaller) + 2px); + --font-adaptive-smallest: calc(var(--font-smallest) + 2px); + --line-width-adaptive: calc(var(--line-width) + 6rem); + --max-width: 90%; + } +} + +/*---------------------------------------------------------------- + THEMES +---------------------------------------------------------------- */ + +.theme-light { + --text-normal: hsl(var(--base-h), var(--base-s), calc(var(--base-l) - 80%)); + --text-muted: hsl( + var(--base-h), + calc(var(--base-s) - 5%), + calc(var(--base-l) - 45%) + ); + --text-faint: hsl( + var(--base-h), + calc(var(--base-s) - 5%), + calc(var(--base-l) - 25%) + ); + + --text-accent: hsl(var(--accent-h), var(--accent-s), var(--accent-l)); + --text-accent-hover: hsl( + var(--accent-h), + var(--accent-s), + calc(var(--accent-l) - 10%) + ); + --text-on-accent: white; + --text-selection: hsla(var(--accent-h), 50%, calc(var(--base-l) - 20%), 30%); + --text-highlight-bg: var(--light-yellow-highlighter); + --text-highlight-bg-active: rgba(0, 0, 0, 0.1); + + --background-primary: white; + --background-primary-alt: hsl(var(--base-h), var(--base-s), var(--base-l)); + --background-secondary: hsl(var(--base-h), var(--base-s), var(--base-l)); + --background-secondary-alt: hsl( + var(--base-h), + var(--base-s), + calc(var(--base-l) - 2%) + ); + --background-tertiary: hsl( + var(--base-h), + var(--base-s), + calc(var(--base-l) - 7%) + ); + --background-modifier-border: hsl( + var(--base-h), + var(--base-s), + calc(var(--base-l) - 4%) + ); + --background-modifier-border-hover: hsl( + var(--base-h), + var(--base-s), + calc(var(--base-l) - 12%) + ); + --background-modifier-border-focus: hsl( + var(--base-h), + var(--base-s), + calc(var(--base-l) - 20%) + ); + --background-modifier-form-field: hsl( + var(--base-h), + var(--base-s), + calc(var(--base-l) + 6%) + ); + --background-modifier-form-field-highlighted: hsl( + var(--base-h), + var(--base-s), + calc(var(--base-l) + 8%) + ); + --background-button: white; + + --background-transparent: hsla( + var(--base-h), + var(--base-s), + var(--base-l), + 0 + ); + /* --background-translucent: rgba(255, 255, 255, 0.85); */ + --background-translucent: hsla( + var(--base-h), + var(--base-s), + calc(var(--base-l) + 0%), + 0.8 + ); + --opacity-translucency: 1; + + --icon-color: var(--text-muted); + --icon-hex: 000; + + --background-match-highlight: hsla(var(--accent-h), 40%, 62%, 0.2); + --background-modifier-accent: hsl( + var(--accent-h), + var(--accent-s), + calc(var(--accent-l) + 10%) + ); + + --interactive-accent: hsl( + var(--accent-h), + var(--accent-s), + calc(var(--accent-l) + 10%) + ); + --interactive-accent-hover: hsl( + var(--accent-h), + var(--accent-s), + calc(var(--accent-l) - 0%) + ); + + --interactive-accent-rgb: 220, 220, 220; + + --quote-opening-modifier: hsl( + var(--base-h), + var(--base-s), + calc(var(--base-l) - 10%) + ); + --background-modifier-cover: hsla( + var(--base-h), + var(--base-s), + calc(var(--base-l) - 5%), + 0.7 + ); + --shadow-color: rgba(0, 0, 0, 0.1); + + /* --tag-background-color: rgb(189, 225, 211); */ + --tag-background-color-l: #bde1d3; + /* --tag-font-color: rgb(29, 105, 75); */ + --tag-font-color-l: #1d694b; + + --code-color-l: #5c5c5c; + --code-color: var(--code-color-l); + --atom-gray-1: #383a42; + --atom-gray-2: #383a42; + --atom-red: #e75545; + --atom-green: #4ea24c; + --atom-blue: #3d74f6; + --atom-purple: #a625a4; + --atom-aqua: #0084bc; + --atom-yellow: #e35649; + --atom-orange: #986800; +} + +.theme-dark { + --text-normal: hsl(var(--base-h), var(--base-s), calc(var(--base-d) + 70%)); + --text-muted: hsl(var(--base-h), var(--base-s), calc(var(--base-d) + 45%)); + --text-faint: hsl(var(--base-h), var(--base-s), calc(var(--base-d) + 20%)); + + --text-accent: hsl(var(--accent-h), var(--accent-s), var(--accent-d)); + --text-accent-hover: hsl( + var(--accent-h), + var(--accent-s), + calc(var(--accent-d) + 12%) + ); + --text-on-accent: white; + --text-selection: hsla(var(--accent-h), 70%, 40%, 30%); + --text-highlight-bg: var(--dark-blue-highlighter); + --text-highlight-bg-active: rgba(255, 255, 255, 0.1); + + --background-primary: hsl(var(--base-h), var(--base-s), var(--base-d)); + --background-primary-alt: hsl( + var(--base-h), + var(--base-s), + calc(var(--base-d) - 2%) + ); + --background-secondary: hsl( + var(--base-h), + var(--base-s), + calc(var(--base-d) - 2%) + ); + --background-secondary-alt: hsl(var(--base-h), var(--base-s), var(--base-d)); + --background-tertiary: hsl( + var(--base-h), + var(--base-s), + calc(var(--base-d) + 2%) + ); + --background-modifier-border: hsl( + var(--base-h), + var(--base-s), + calc(var(--base-d) + 4%) + ); + --background-modifier-border-hover: hsl( + var(--base-h), + var(--base-s), + calc(var(--base-d) + 10%) + ); + --background-modifier-border-focus: hsl( + var(--base-h), + var(--base-s), + calc(var(--base-d) + 20%) + ); + --background-modifier-box-shadow: rgba(0, 0, 0, 0.3); + --background-button: hsl( + var(--base-h), + var(--base-s), + calc(var(--base-d) + 2%) + ); + + --background-transparent: hsla( + var(--base-h), + var(--base-s), + var(--base-d), + 0 + ); + --background-translucent: hsla( + var(--base-h), + var(--base-s), + var(--base-d), + 0.8 + ); + --opacity-translucency: 1; + + --background-match-highlight: hsla(var(--accent-h), 40%, 62%, 0.2); + --background-modifier-accent: hsl( + var(--accent-h), + var(--accent-s), + calc(var(--accent-d) - 10%) + ); + + --icon-color: var(--text-muted); + --icon-hex: FFF; + --interactive-accent: hsl( + var(--accent-h), + var(--accent-s), + calc(var(--accent-d) - 20%) + ); + --interactive-accent-hover: hsl( + var(--accent-h), + var(--accent-s), + calc(var(--accent-d) - 15%) + ); + --quote-opening-modifier: hsl( + var(--base-h), + var(--base-s), + calc(var(--base-d) + 10%) + ); + --interactive-accent-rgb: 66, 66, 66; + + --background-modifier-cover: hsla( + var(--base-h), + var(--base-s), + calc(var(--base-d) - 12%), + 0.8 + ); + --shadow-color: rgba(0, 0, 0, 0.3); + + --tag-background-color-d: rgb(29, 105, 75); + --tag-font-color-d: var(--text-normal); + + --code-color-d: #a6a6a6; + --code-color: var(--code-color-d); + --atom-gray-1: #5c6370; + --atom-gray-2: #abb2bf; + --atom-red: #e06c75; + --atom-orange: #d19a66; + --atom-green: #98c379; + --atom-aqua: #56b6c2; + --atom-purple: #c678dd; + --atom-blue: #61afef; + --atom-yellow: #e5c07b; +} + +/* ---------------------------------------------------------------- +Desktop Styling +---------------------------------------------------------------- */ + +/* ---------------------- */ +/* Better Live Preview */ +/* ---------------------- */ + +.is-live-preview { + padding: 0 0.5em !important; +} + +/* Quote blocks */ +.markdown-source-view.mod-cm6.is-live-preview .HyperMD-quote { + border: 0 solid var(--quote-opening-modifier); + border-left-width: 2px; + background-color: var(--background-primary); +} + +/* Live Preview list bullets */ +body:not(.is-mobile) .markdown-source-view.mod-cm6 .list-bullet:after { + left: -3px; +} +.mod-cm6 .HyperMD-list-line .list-bullet::after, +.mod-cm6 span.list-bullet::after { + line-height: 0.95em; + font-size: 1.4em; + font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; + vertical-align: middle; + color: var(--text-faint); +} + +.is-live-preview .HyperMD-header-2 { + border-bottom: 2px solid var(--background-modifier-border); + width: 100%; + padding-bottom: 2px; +} + +/* Temp fix to match Live Preview checkbox color */ +.is-live-preview input[type='checkbox']:checked { + background-color: #00a7c4 !important; +} + +/* Align checkboxes */ +.markdown-source-view.mod-cm6 .task-list-item-checkbox { + vertical-align: sub !important; +} + +/* Align collapse-indicators */ +.is-live-preview .collapse-indicator.collapse-icon { + top: 2px !important; +} + +.cm-strong, +strong { + font-weight: var(--bold-weight) !important; +} + +h1, +h2, +h3, +h4 { + letter-spacing: -0.02em; +} + +h2 { + border-bottom: 2px solid var(--background-modifier-border); + width: 100%; + padding-bottom: 2px; +} + +.popover, +.vertical-tab-content-container, +.workspace-leaf-content[data-type='markdown'] { + font-family: var(--text); +} + +body, +input, +button, +.markdown-preview-view, +.cm-s-obsidian .cm-formatting-hashtag, +.cm-s-obsidian { + font-size: var(--font-adaptive-normal); + font-weight: var(--normal-weight); + line-height: var(--line-height); + -webkit-font-smoothing: subpixel-antialiased; +} + +.markdown-source-view.mod-cm6 .cm-scroller, +.markdown-source-view, +.cm-s-obsidian .cm-formatting-hashtag, +.cm-s-obsidian, +.cm-s-obsidian span.cm-formatting-task { + font-family: var(--font-editor); + -webkit-font-smoothing: subpixel-antialiased; +} + +/* Ensure tags use text font */ +.markdown-source-view.mod-cm6.is-live-preview .cm-hashtag.cm-meta, +.markdown-source-view.mod-cm5 .cm-hashtag.cm-meta { + font-family: var(--font-text-theme); +} + +/* Use reading font in live preview */ +.lp-reading-font .markdown-source-view.mod-cm6.is-live-preview .cm-scroller { + font-family: var(--font-text); +} + +.cm-s-obsidian span.cm-formatting-task { + font-family: var(--font-monospace); /* Editor task is monospace */ + line-height: var(--line-height); +} +.cm-formatting-strong, +.cm-formatting-em, +.cm-formatting.cm-formatting-quote { + color: var(--text-faint) !important; + font-weight: var(--normal-weight); + opacity: 0.8; + letter-spacing: -0.02em; +} +.cm-formatting-header, +.cm-s-obsidian .cm-formatting-header.cm-header-1, +.cm-s-obsidian .cm-formatting-header.cm-header-2, +.cm-s-obsidian .cm-formatting-header.cm-header-3, +.cm-s-obsidian .cm-formatting-header.cm-header-4, +.cm-s-obsidian .cm-formatting-header.cm-header-5, +.cm-s-obsidian .cm-formatting-header.cm-header-6 { + color: var(--text-faint); + font-weight: var(--bold-weight); + opacity: 0.8; + letter-spacing: -0.02em; +} +.view-header-title, +.file-embed-title, +.markdown-embed-title { + letter-spacing: -0.02em; + text-align: left; + font-size: 1.125em; + padding: 10px; +} +.empty-state-title, +.markdown-preview-view h1, +.HyperMD-header-1 .cm-header-1, +.cm-s-obsidian .cm-header-1 { + letter-spacing: -0.02em; + line-height: 1.3; + font-size: var(--h1) !important; + color: var(--h1-color); + font-weight: var(--h1-weight) !important; +} +.markdown-preview-view h2, +.HyperMD-header-2 .cm-header-2, +.cm-s-obsidian .cm-header-2 { + letter-spacing: -0.02em; + line-height: 1.3; + font-size: var(--h2) !important; + color: var(--h2-color); + font-weight: var(--h2-weight) !important; +} +.markdown-preview-view h3, +.HyperMD-header-3 .cm-header-3, +.cm-s-obsidian .cm-header-3 { + letter-spacing: -0em; + line-height: 1.4; + font-size: var(--h3) !important; + color: var(--h3-color); + font-weight: var(--h3-weight) !important; +} +.markdown-preview-view h4, +.HyperMD-header-4 .cm-header-4, +.cm-s-obsidian .cm-header-4 { + letter-spacing: 0.02em; + font-size: var(--h4) !important; + color: var(--h4-color); + font-weight: var(--h4-weight) !important; + text-transform: var(--h4-transform); +} +.markdown-preview-view h5, +.HyperMD-header-5 .cm-header-5, +.cm-s-obsidian .cm-header-5 { + letter-spacing: 0.02em; + font-size: var(--h5) !important; + color: var(--h5-color); + font-weight: var(--h5-weight) !important; +} +.markdown-preview-view h6, +.HyperMD-header-6 .cm-header-6, +.cm-s-obsidian .cm-header-6 { + letter-spacing: 0.02em; + font-size: var(--h6) !important; + color: var(--h6-color); + font-weight: var(--h6-weight) !important; +} + +.markdown-preview-view mark { + margin: 0 -0.05em; + padding: 0.125em 0.15em; + border-radius: 0.2em; + -webkit-box-decoration-break: clone; + box-decoration-break: clone; +} + +/* --------------- */ +/* Highlight styles */ +/* --------------- */ + +span.cm-highlight { + padding: 0.1em 0; + border-radius: 0.2em; + -webkit-box-decoration-break: clone; + box-decoration-break: clone; +} + +span.cm-formatting-highlight { + /*margin: 0 0 0 -0.4em;*/ + padding-left: 0.15em; + padding-right: 0em; + -webkit-box-decoration-break: clone; + box-decoration-break: clone; +} + +.cm-highlight + span.cm-formatting-highlight { + padding-left: 0em; + padding-right: 0.15em; + -webkit-box-decoration-break: clone; + box-decoration-break: clone; +} + +/* --------------- */ +/* Tags */ +/* --------------- */ + +.theme-light .frontmatter-container .tag, +.theme-light a.tag { + background-color: var(--tag-background-color-l); + color: var(--tag-font-color-l); + font-size: var(--font-adaptive-small); + font-weight: 500; + padding: 3px 8px; + text-align: center; + text-decoration: none; + border-radius: 20px; +} +.theme-light a.tag:hover { + color: var(--text-normal); + border-color: var(--background-modifier-border-hover); +} +.theme-dark .frontmatter-container .tag, +.theme-dark a.tag { + background-color: var(--tag-background-color-d); + color: var(--tag-font-color-d); + font-size: var(--font-adaptive-small); + font-weight: 500; + padding: 3px 8px; + text-align: center; + text-decoration: none; + border-radius: 20px; +} +.theme-dark a.tag:hover { + color: var(--text-normal); + border-color: var(--background-modifier-border-hover); +} +.theme-light .cm-s-obsidian span.cm-hashtag { + background-color: var(--tag-background-color-l); + color: var(--tag-font-color-l); + font-size: var(--font-adaptive-small); + font-weight: 500; + text-align: center; + text-decoration: none; + padding-top: 3px; + padding-bottom: 3px; + border-left: none; + border-right: none; + cursor: text; +} +.theme-dark .cm-s-obsidian span.cm-hashtag { + background-color: var(--tag-background-color-d); + color: var(--tag-font-color-d); + font-size: var(--font-adaptive-small); + font-weight: 500; + text-align: center; + text-decoration: none; + padding-top: 3px; + padding-bottom: 3px; + border-left: none; + border-right: none; + cursor: text; +} +span.cm-hashtag.cm-hashtag-begin { + border-top-left-radius: 14px; + border-bottom-left-radius: 14px; + padding-left: 8px; + border-right: none; + border-left: 1px solid var(--background-modifier-border); +} +span.cm-hashtag.cm-hashtag-end { + border-top-right-radius: 14px; + border-bottom-right-radius: 14px; + border-left: none; + padding-right: 8px; + border-right: 1px solid var(--background-modifier-border); +} + +/* --------------- */ +/* Image zoom */ +/* --------------- */ + +/* Image cards */ +img { + border-radius: 4px; + box-shadow: 0 1px 3px rgba(0, 0, 0, 0.12), 0 1px 2px rgba(0, 0, 0, 0.24); + background-color: var(--background-secondary); + /* Background color so PNGs with transparent backgrounds don't look weird */ +} + +.full-width-media .markdown-preview-view .image-embed img:not([width]), +.full-width-media .markdown-preview-view audio, +.full-width-media .markdown-preview-view video { + width: 100%; +} + +.view-content .markdown-preview-view img { + max-width: 100%; + cursor: zoom-in; +} + +body:not(.is-mobile) + .view-content + .markdown-preview-view + img[referrerpolicy='no-referrer']:active, +body:not(.is-mobile) .view-content .image-embed:active { + cursor: zoom-out; + display: block; + z-index: 100; + position: fixed; + max-height: calc(100% + 1px); + max-width: calc(100% - 20px); + height: calc(100% + 1px); + width: 100%; + object-fit: contain; + margin: -0.5px auto 0; + text-align: center; + padding: 0; + left: 0; + right: 0; + bottom: 0; + background: var(--background-translucent); +} +body:not(.is-mobile) + .view-content + .markdown-preview-view + img[referrerpolicy='no-referrer']:active { + padding: 2.5%; +} +body:not(.is-mobile) + .view-content + .markdown-preview-view + .image-embed:active + img { + top: 50%; + transform: translateY(-50%); + padding: 0; + margin: 0 auto; + width: auto; + max-height: 95vh; + left: 0; + right: 0; + bottom: 0; + position: absolute; + opacity: 1; +} +.theme-dark span[src$='#invert'] img { + filter: invert(1) hue-rotate(180deg); + mix-blend-mode: screen; +} + +/* --------------- */ +/* Modals */ +/* --------------- */ + +.modal { + border: none; + background: var(--background-primary); + border-radius: 10px; + overflow: hidden; + padding: 20px 20px 10px; +} +.modal.mod-settings .vertical-tab-content-container { + border-left: 1px solid var(--background-modifier-border); + padding-bottom: 0; + padding-right: 0; +} +.modal.mod-settings, +.modal.mod-settings .vertical-tab-container { + max-width: 1000px; + width: 60vw; + min-height: 20vh; + width: 90vw; + height: 100vh; + max-height: 80vh; + overflow-y: hidden; + border: 1px solid var(--background-modifier-border) !important; +} +.modal.mod-settings .vertical-tab-content-container, +.modal.mod-settings .vertical-tab-header { + height: 80vh; +} +body .modal.mod-community-theme { + max-width: 1000px; + border: 1px solid var(--background-modifier-border); + overflow: hidden; +} +.modal.mod-community-theme { + padding: 0; +} +body:not(.is-mobile) .theme-list .community-theme { + width: 100%; + padding: 18px; + border: 1px solid var(--background-modifier-border); +} +.modal.mod-community-theme .modal-content { + padding: 30px; +} +.modal-title { + text-align: left; + font-size: var(--h2); + line-height: 1.4; + padding-bottom: 0; +} +.modal-content { + margin-top: 0px; + padding: 0; +} +.modal-content .u-center-text { + text-align: left; + font-size: 13px; +} +.community-plugin-name, +.modal.mod-settings .vertical-tab-content-container, +.setting-item-name { + font-size: var(--font-settings); + line-height: 1.4; +} +.community-plugin-downloads, +.community-plugin-item .community-plugin-author, +.community-plugin-item .community-plugin-desc, +.community-plugin-search-summary, +.setting-item-description { + font-size: var(--font-settings-small); + line-height: 1.4; + font-weight: 400; +} +.setting-item-description { + padding-top: 4px; +} +.setting-item-control button, +button { + font-size: var(--font-inputs); + font-weight: 400; +} +.modal button, +.modal button.mod-cta a, +button.mod-cta { + font-size: var(--font-settings-small); + margin-right: 3px; + margin-left: 3px; +} +.dropdown, +body .addChoiceBox #addChoiceTypeSelector { + font-size: var(--font-settings-small); +} +.progress-bar-message { + color: var(--text-faint); +} +input.prompt-input { + border: 0; + background: var(--background-primary); + box-shadow: none !important; + padding-left: 10px; + height: 40px; + line-height: 4; + font-size: var(--font-adaptive-normal); +} +input.prompt-input:hover { + border: 0; + background: var(--background-primary); + padding-left: 10px; + line-height: 4; +} +.suggestion-item { + cursor: var(--cursor); + padding-left: 10px; +} +.suggestion-flair { + left: auto; + right: 16px; + opacity: 0.25; +} +.prompt-results .suggestion-flair .filled-pin { + display: none; +} +.theme-light .modal-container .suggestion-item.is-selected { + border-radius: 6px; + background: var(--background-tertiary); +} +.theme-dark .modal-container .suggestion-item.is-selected { + border-radius: 6px; + background: var(--blue); +} +.menu-item { + margin-bottom: 1px; +} +.suggestion-item.is-selected, +.menu-item:hover:not(.is-disabled):not(.is-label), +.menu-item:hover { + background: var(--background-tertiary); +} +.suggestion-item, +.suggestion-empty { + font-size: var(--font-adaptive-normal); +} +.modal, +.prompt, +.suggestion-container { + box-shadow: 0 5px 30px rgba(0, 0, 0, 0.15); +} +.prompt-instructions { + color: var(--text-muted); + padding: 10px; +} +.prompt-instruction-command { + font-weight: 600; +} +.prompt { + padding-bottom: 0; +} +.prompt-results { + padding-bottom: 10px; +} +.menu { + padding: 6px; +} +.menu-item { + font-size: var(--font-adaptive-small); + border-radius: 5px; + padding: 2px 12px 3px 10px; + height: 26px; + cursor: var(--cursor); + line-height: 20px; +} +.menu-separator { + margin: 6px -5px; +} +.menu-item-icon svg { + width: 12px; + height: 12px; +} +.menu-item-icon { + width: 24px; +} + +/* --------------- */ +/* Sync */ +/* --------------- */ + +.sync-history-content { + font-size: var(--font-adaptive-small); + border: none; + padding: 20px 40px 20px 20px; +} +.sync-history-content-container { + padding: 0; +} +.sync-history-content-container .modal-button-container { + margin: 0; + padding: 10px 5px; + border-top: 1px solid var(--background-modifier-border); + background-color: var(--background-primary); + text-align: center; +} +.sync-history-list-container { + flex-basis: 220px; +} +.sync-history-list { + padding: 10px; + border-right: 1px solid var(--background-modifier-border); + background-color: var(--background-secondary); +} +.sync-history-list-item { + border-radius: 4px; + padding: 4px 8px; + margin-bottom: 4px; + font-size: var(--font-adaptive-small); + cursor: var(--cursor); +} +.sync-history-list-item.is-active, +.sync-history-list-item:hover { + background-color: var(--background-tertiary); +} + +/* --------------- */ +/* YAML Front matter */ +/* --------------- */ + +.theme-dark pre.frontmatter[class*='language-yaml'], +.theme-light pre.frontmatter[class*='language-yaml'] { + padding: 0 0 0px 0; + background: transparent; + font-family: var(--text); + line-height: 1.2; + border-radius: 0; + border-bottom: 0px solid var(--background-modifier-border); +} +.markdown-preview-view .table-view-table > thead > tr > th { + border-color: var(--background-modifier-border); +} +.theme-dark .frontmatter .token, +.theme-light .frontmatter .token, +.markdown-preview-section .frontmatter code { + font-family: var(--text); +} + +.markdown-source-view .cm-s-obsidian .cm-hmd-frontmatter { + font-family: var(--font-monospace); +} + +/* --------------- */ +/* Drag ghost */ +/* --------------- */ + +body.is-dragging { + cursor: grabbing; + cursor: -webkit-grabbing; +} + +.workspace-drop-overlay:before, +.mod-drag, +.drag-ghost { + opacity: 100; + border-radius: 0 !important; +} +.mod-drag { + opacity: 0; + border: 2px solid var(--text-accent); + background-color: var(--background-primary); +} +.view-header.is-highlighted:after { + background-color: var(--text-selection); +} +.view-header.is-highlighted .view-actions { + background: transparent; +} + +/* --------------- */ +/* Workspace */ +/* --------------- */ + +.empty-state { + background-color: var(--background-primary); + text-align: center; +} +.workspace-split.mod-vertical > .workspace-split { + padding: 0; +} +.workspace-split .workspace-tabs { + background: var(--background-primary); +} +.workspace-split:not(.mod-right-split) .workspace-tabs { + background: var(--background-secondary); +} +.workspace-split.mod-root + > .workspace-leaf:first-of-type + .workspace-leaf-content, +.workspace-split.mod-root + > .workspace-leaf:last-of-type + .workspace-leaf-content { + border-top-right-radius: 0px; + border-top-left-radius: 0px; +} +.workspace-split.mod-root.mod-horizontal .workspace-leaf-resize-handle, +.workspace-split.mod-root.mod-vertical .workspace-leaf-resize-handle { + border-width: 1px; +} +.workspace-split.mod-horizontal > * > .workspace-leaf-resize-handle { + height: 2px; + background: transparent; + border-bottom: var(--border-width-alt) solid var(--background-modifier-border); +} +.workspace-split.mod-right-split > .workspace-leaf-resize-handle { + background: transparent; + border-left: var(--border-width-alt) solid var(--background-modifier-border); + width: 3px !important; +} +.workspace-split.mod-vertical > * > .workspace-leaf-resize-handle, +.workspace-split.mod-left-split > .workspace-leaf-resize-handle { + border-right: var(--border-width) solid var(--background-modifier-border); + width: 2px !important; + background: transparent; +} +.workspace-split.mod-right-split > .workspace-leaf-resize-handle:hover, +.workspace-split.mod-horizontal > * > .workspace-leaf-resize-handle:hover, +.workspace-split.mod-vertical > * > .workspace-leaf-resize-handle:hover, +.workspace-split.mod-left-split > .workspace-leaf-resize-handle:hover { + border-color: var(--background-modifier-border-hover); + transition: border-color 0.1s ease-in-out 0.05s, + border-width 0.1s ease-in-out 0.05s; + border-width: 3px; +} +.workspace-split.mod-right-split > .workspace-leaf-resize-handle:active, +.workspace-split.mod-horizontal > * > .workspace-leaf-resize-handle:active, +.workspace-split.mod-vertical > * > .workspace-leaf-resize-handle:active, +.workspace-split.mod-left-split > .workspace-leaf-resize-handle:active { + border-color: var(--background-modifier-border-focus); + border-width: 3px; +} +.workspace-tab-container-before, +.workspace-tab-container-after { + width: 0; +} +.workspace-leaf { + border-left: 0px; +} +.mod-horizontal .workspace-leaf { + border-bottom: 0px; + background-color: transparent; + box-shadow: none !important; +} + +.workspace-tab-header.is-before-active .workspace-tab-header-inner, +.workspace-tab-header.is-active, +.workspace-tab-header.is-after-active, +.workspace-tab-header.is-after-active .workspace-tab-header-inner, +.workspace-tab-header.is-before-active, +.workspace-tab-header.is-after-active { + background: transparent; +} +.workspace-tabs { + border: 0; + padding-right: 0; + font-size: 100%; +} +.workspace-tab-header-container { + border: 0 !important; + height: 40px; + background-color: transparent; +} + +/* --------------- */ +/* Workspace Icons */ +/* --------------- */ + +.nav-action-button svg { + width: 25px; + height: 15px; +} +.workspace-ribbon-collapse-btn svg path { + stroke-width: 3px; +} +.nav-action-button svg path { + stroke-width: 2px; +} +.clickable-icon { + cursor: var(--cursor); +} +.view-header-icon, +.workspace-tab-header, +.nav-action-button, +.side-dock-ribbon-tab, +.view-action { + background: transparent; + color: var(--text-muted); + opacity: var(--icon-muted); + transition: opacity 0.1s ease-in-out; + cursor: var(--cursor); +} +.view-header-icon { + opacity: 0; +} +.is-mobile.show-mobile-hamburger .view-header-icon { + opacity: 1; + transform: scale(1.4); + padding: 0px 15px; + top: 5px; +} +.workspace-leaf-content[data-type='search'] .nav-action-button.is-active, +.workspace-leaf-content[data-type='backlink'] .nav-action-button.is-active, +.workspace-leaf-content[data-type='tag'] .nav-action-button.is-active, +.workspace-tab-header.is-active, +.workspace-leaf-content[data-type='search'] .nav-action-button.is-active { + background: transparent; + color: var(--text-muted); + opacity: 1; + transition: opacity 0.1s ease-in-out; +} +.view-action:hover, +.view-header-icon:hover, +.nav-action-button:hover, +.workspace-tab-header:hover, +.side-dock-ribbon-tab:hover, +.side-dock-ribbon-action:hover { + background: transparent; + color: var(--text-muted); + opacity: 1; + transition: opacity 0 ease-in-out; +} +.workspace-leaf-content[data-type='search'] .nav-action-button.is-active { + background: transparent; +} +.nav-action-button, +.workspace-leaf-content[data-type='search'] .nav-action-button, +.workspace-leaf-content[data-type='backlink'] .nav-action-button { + padding: 0 4px 0 8px; + margin: 0; +} + +/* --------------- */ +/* Workspace Tabs */ +/* --------------- */ + +.workspace-tab-header-container { + height: unset; + padding: 5px 10px 0px 10px; + margin: 5px 0; +} +.theme-light .workspace-tab-header.is-active { + box-shadow: 0px 0px 1px 1px inset var(--background-tertiary); + background-color: var(--background-primary); + border-radius: 6px; +} +.theme-dark .workspace-tab-header.is-active { + box-shadow: 0px 0px 0px 1px inset var(--background-secondary); + background-color: var(--background-tertiary); + border-radius: 6px; +} +.workspace-tab-container-before.is-before-active, +.workspace-tab-container-after.is-after-active, +.workspace-tab-header.is-before-active, +.workspace-tab-header.is-after-active { + background: transparent; +} + +/* --------------- */ +/* Workspace slider */ +/* --------------- */ + +.theme-light .workspace-tab-container-inner { + border-radius: 10px; + background-color: var(--background-secondary-alt) !important; + border: 1px solid var(--background-tertiary); + display: flex; + justify-content: center; + align-items: center; + stroke-width: 0; +} +.theme-dark .workspace-tab-container-inner { + border-radius: 10px; + background-color: var(--background-secondary) !important; + border: 1px solid var(--background-tertiary); + display: flex; + justify-content: center; + align-items: center; + stroke-width: 0; +} +.workspace-tab-header { + background-color: transparent; + border-radius: 10px !important; +} +.workspace-tab-header-inner { + padding: 6px 15px; +} +.workspace-tab-header-inner-icon { + display: flex; + justify-content: center; + align-items: center; +} + +/* --------------- */ +/* Window frame */ +/* --------------- */ + +body:not(.hider-frameless):not(.is-fullscreen):not(.is-mobile) { + --titlebar-height: 28px; + padding-top: var(--titlebar-height) !important; +} +body:not(.hider-frameless):not(.is-fullscreen):not(.is-mobile) .titlebar { + background: var(--background-secondary); + border-bottom: var(--border-width) solid var(--background-modifier-border); + height: var(--titlebar-height) !important; + top: 0 !important; + padding-top: 0 !important; +} +body.hider-frameless .titlebar { + border-bottom: none; +} +.mod-windows .titlebar-button:hover { + background-color: var(--background-primary-alt); +} +.mod-windows .titlebar-button.mod-close:hover { + background-color: var(--background-modifier-error); +} +.mod-windows .mod-close:hover svg { + fill: white !important; + stroke: white !important; +} + +.titlebar-button-container { + height: var(--titlebar-height); + top: 0; + display: flex; + align-items: center; +} +.titlebar:hover .titlebar-button-container.mod-left { + opacity: 1; +} +.titlebar-text { + display: none; + padding-top: 5px; + color: var(--text-faint); + letter-spacing: inherit; +} +.titlebar-button:hover { + opacity: 1; + transition: opacity 100ms ease-out; +} +.titlebar-button { + opacity: 1; + cursor: var(--cursor); + color: var(--text-muted); + padding: 2px 4px; + border-radius: 3px; + line-height: 1; + display: flex; +} +.titlebar-button:hover { + background-color: var(--background-tertiary); +} +.titlebar-button-container.mod-left .titlebar-button { + margin-right: 5px; +} +.titlebar-button-container.mod-right .titlebar-button { + margin-left: 0; + border-radius: 0; + height: 100%; + align-items: center; + padding: 2px 15px; +} + +/* --------------- */ +/* Title Bar */ +/* --------------- */ + +.view-actions { + margin-right: 10px; + z-index: 15; + background: var(--background-primary); +} +.view-header { + height: 40px; +} +.view-header-title { + padding: 0; +} +.workspace-leaf-header, +.view-header { + background-color: var(--background-primary) !important; + border: none !important; +} +.view-header-title-container:after { + display: none; +} + +/* --------------- */ +/* Full borders */ +/* --------------- */ + +body.full-borders .view-header { + border-bottom: 1px solid var(--background-modifier-border) !important; +} +body.full-borders .side-dock-ribbon { + border-right: 1px solid var(--background-modifier-border) !important; +} + +/* --------------- */ +/* Custom line width */ +/* --------------- */ + +.markdown-preview-view.is-readable-line-width .markdown-preview-sizer { + max-width: var(--max-width); + width: var(--line-width-adaptive); +} +.is-mobile .markdown-source-view.mod-cm6.is-readable-line-width .cm-content { + max-width: var(--line-width-adaptive); +} + +.markdown-source-view.is-readable-line-width .CodeMirror { + padding-left: 0; + padding-right: 0; + margin: 0 auto 0 auto; + width: var(--line-width-adaptive); + max-width: var(--max-width); +} +.view-header-title-container { + padding-left: 0; + padding-right: 0; + position: absolute; + max-width: var(--max-width); + width: var(--line-width-adaptive); + margin: 0 auto; + left: 0; + right: 0; +} + +/* --------------- */ +/* EDITOR MODE */ +/* --------------- */ + +/* Fancy cursor */ +/* .CodeMirror-cursor, +.cm-s-obsidian .cm-cursor { + border: none; + border-right: 2px solid var(--text-accent); +} */ +.markdown-source-view.mod-cm6, +.markdown-source-view.mod-cm5, +.markdown-source-view { + padding: 0; +} +.cm-s-obsidian .CodeMirror-code { + padding-right: 0; +} +.CodeMirror-lines { + padding-bottom: 170px; +} +.cm-s-obsidian pre.HyperMD-list-line { + padding-top: 0; +} +.workspace .markdown-preview-view { + padding: 0; +} +.workspace .markdown-preview-view .markdown-embed { + margin: 0; +} +.workspace .markdown-preview-view .markdown-embed-content { + max-height: none; +} +.markdown-embed-title, +.internal-embed .markdown-preview-section { + max-width: 100%; +} +.cm-s-obsidian .HyperMD-header, +.cm-s-obsidian pre.HyperMD-header { + /* Commenting to better align header and content */ + /* padding-left: 0 !important; */ + font-size: 1em !important; +} +.CodeMirror-linenumber { + font-size: var(--font-adaptive-small) !important; + font-feature-settings: 'tnum'; + color: var(--text-faint); + padding-top: 3px; +} +.cm-s-obsidian span.cm-url, +.cm-s-obsidian span.cm-url:hover { + color: var(--text-accent); +} +.cm-s-obsidian span.cm-link { + color: var(--text-muted); +} +.cm-s-obsidian span.cm-hmd-internal-link { + color: var(--text-accent) !important; +} +.cm-s-obsidian span.cm-formatting-link { + color: var(--text-faint) !important; +} + +/* Mermaid */ +.mermaid svg { + width: 100%; +} + +/* Transcluded notes and embeds */ + +.markdown-preview-view.is-readable-line-width + .markdown-embed + .markdown-preview-sizer { + max-width: 100%; + width: 100%; +} + +.markdown-embed h1:first-child { + margin-block-start: 0em; +} + +.markdown-preview-view .markdown-embed { + margin-top: var(--nested-padding); + padding: 0 calc(var(--nested-padding) / 2) 0 var(--nested-padding); +} +.markdown-embed-title { + /* Remove height to fix cutoff bug */ + /* height: 24px; */ + line-height: 18px; +} +.markdown-embed .markdown-preview-sizer:first-child ul { + margin-block-start: 2px; +} +.markdown-embed .markdown-preview-section:last-child p, +.markdown-embed .markdown-preview-section:last-child ul { + margin-block-end: 2px; +} +.internal-embed:not([src*='#^']) .markdown-embed-link { + left: 0; + width: 100%; +} +.markdown-embed-link, +.file-embed-link { + top: 0px; + right: 0; + text-align: right; +} +.file-embed-link svg, +.markdown-embed-link svg { + width: 20px; + opacity: 0; +} +.markdown-embed:hover .file-embed-link svg, +.markdown-embed:hover .markdown-embed-link svg { + opacity: 1; +} +.markdown-preview-view .markdown-embed-content > .markdown-preview-view { + max-height: none !important; +} +.markdown-embed .markdown-preview-view { + padding: 0; +} +.internal-embed .markdown-embed { + border: 0; + border-left: 2px solid var(--quote-opening-modifier); + border-radius: 0; +} + +/* Embedded Searches */ + +.markdown-preview-view .internal-query.is-embed { + border-top: none; + border-bottom: none; +} +.markdown-preview-view .internal-query.is-embed .internal-query-header { + justify-content: start; +} +.markdown-preview-view .internal-query.is-embed .internal-query-header-title { + font-weight: 500; + color: var(--text-normal); + font-size: var(--h2); +} +.internal-query.is-embed .search-result-file-matches { + border-bottom: 0; +} + +/* Editor Mode Footnotes */ + +.cm-s-obsidian span.cm-footref { + font-size: var(--font-adaptive-normal); +} +.cm-s-obsidian pre.HyperMD-footnote { + font-size: var(--font-adaptive-small); + padding-left: 20px; +} + +/* Editor Mode Tables */ + +.CodeMirror pre.HyperMD-table-row { + font-size: calc(var(--font-adaptive-normal) - 1px); + font-family: var(--font-monospace) !important; +} + +/* Editor Mode Lists */ + +.cm-formatting-list { + color: var(--text-faint) !important; +} +/* Editor Mode Quotes */ + +span.cm-formatting.cm-formatting-quote { + color: var(--text-faint) !important; +} + +/* --------------- */ +/* Internal search */ +/* --------------- */ + +.is-flashing { + border-radius: 2px; + box-shadow: 0 2px 0 8px var(--text-highlight-bg); + transition: all 0s ease-in-out; +} +.is-flashing .tag { + border-color: var(--text-highlight-bg-active); +} +.suggestion-container.mod-search-suggestion { + max-width: 280px; +} +.mod-search-suggestion .suggestion-item { + font-size: var(--font-adaptive-small); +} +.mod-search-suggestion .clickable-icon { + margin: 0; +} +.search-suggest-item.mod-group { + font-size: var(--font-adaptive-smaller); +} +.cm-s-obsidian span.obsidian-search-match-highlight { + background: inherit; + background: var(--text-highlight-bg); + padding-left: 0; + padding-right: 0; +} +.markdown-preview-view .search-highlight > div { + box-shadow: 0 0 0px 2px var(--text-normal); + border-radius: 2px; + background: transparent; +} +.markdown-preview-view .search-highlight > div { + opacity: 0.4; +} +.markdown-preview-view .search-highlight > div.is-active { + background: transparent; + border-radius: 2px; + opacity: 1; + mix-blend-mode: normal; + box-shadow: 0 0 0px 3px var(--text-accent); +} +.document-search-container.mod-replace-mode { + height: 90px; +} +.document-search-button, +.document-search-close-button { + cursor: var(--cursor); +} +.document-search-close-button:before { + font-weight: 200; +} +body .document-search-container { + margin-top: 12px; + padding: 0; + height: 38px; + background-color: var(--background-primary); + border-top: none; + width: 100%; +} +.markdown-reading-view.is-searching, +.markdown-source-view.is-replacing, +.markdown-source-view.is-searching { + flex-direction: column-reverse; +} +input.document-search-input, +input.document-replace-input { + margin-top: 2px; + font-size: var(--font-adaptive-small) !important; + border: 1px solid var(--background-modifier-border); + border-radius: 5px; + height: 28px !important; + background: var(--background-primary); + transition: border-color 0.1s ease-in-out; +} +input.document-search-input:hover, +input.document-replace-input:hover { + border: 1px solid var(--background-modifier-border-hover); + background: var(--background-primary); + transition: border-color 0.1s ease-in-out; +} +input.document-search-input:focus, +input.document-replace-input:focus { + border: 1px solid var(--background-modifier-border-focus); + background: var(--background-primary); + transition: all 0.1s ease-in-out; +} +.document-search-button { + font-size: var(--font-adaptive-small); +} + +/* --------------- */ +/* Sidebar documents */ +/* --------------- */ + +.workspace > .workspace-split:not(.mod-root) .CodeMirror, +.workspace > .workspace-split:not(.mod-root) .markdown-preview-view { + font-size: var(--font-adaptive-small); + line-height: 1.2; +} +.workspace + > .workspace-split:not(.mod-root) + .workspace-leaf-content[data-type='markdown'] + .markdown-preview-view { + padding: 0 15px; +} +.workspace + > .workspace-split:not(.mod-root) + .workspace-leaf-content[data-type='markdown'] + .markdown-embed + .markdown-preview-view { + padding: 0; +} +.workspace > .workspace-split:not(.mod-root) .CodeMirror, +.workspace > .workspace-split:not(.mod-root) .markdown-preview-section, +.workspace > .workspace-split:not(.mod-root) .markdown-preview-sizer { + max-width: 100%; + padding: 0; + width: auto; +} + +/* Hide embed styling for sidebar documents */ +.workspace > .workspace-split:not(.mod-root) .internal-embed .markdown-embed { + border: none; + padding: 0; +} + +.workspace > .workspace-split:not(.mod-root) .CodeMirror-sizer { + padding-left: 10px; +} + +/* --------------- */ +/* Turn off file name trimming */ +/* --------------- */ + +.full-file-names .tree-item-inner, +.full-file-names .nav-file-title-content, +.full-file-names .search-result-file-title { + text-overflow: unset; + white-space: normal; + line-height: 1.4; +} + +.full-file-names .nav-file-title { + margin-bottom: 3px; +} + +/* --------------- */ +/* Form inputs */ +/* --------------- */ + +input[type='email'], +input[type='number'], +input[type='password'], +input[type='search'], +/* input[type='text'], */ +textarea { + font-size: var(--font-inputs); +} +textarea { + padding: 5px 10px; + transition: all 0.1s linear; + line-height: 1.3; + -webkit-appearance: none; +} +input[type='text'], +input[type='search'], +input[type='email'], +input[type='password'], +input[type='number'] { + padding: 5px 10px; + transition: all 0.1s linear; + height: var(--input-height); + -webkit-appearance: none; +} +textarea:hover, +select:hover, +input:hover { + border-color: var(--background-modifier-border-hover); + transition: all 0.1s linear; +} +textarea:active, +textarea:focus, +button:active, +button:focus, +.dropdown:focus, +.dropdown:active, +select:focus, +select:active, +input[type='text']:active, +input[type='search']:active, +input[type='email']:active, +input[type='password']:active, +input[type='number']:active, +input[type='text']:focus, +input[type='search']:focus, +input[type='email']:focus, +input[type='password']:focus, +input[type='number']:focus { + -webkit-appearance: none; + border-color: var(--background-modifier-border-hover); +} +body:not(.is-mobile) textarea:active, +body:not(.is-mobile) textarea:focus, +body:not(.is-mobile) button:active, +body:not(.is-mobile) button:focus, +body:not(.is-mobile) .dropdown:focus, +body:not(.is-mobile) .dropdown:active, +body:not(.is-mobile) select:focus, +body:not(.is-mobile) select:active, +body:not(.is-mobile) input:focus { + box-shadow: 0 0 0px 2px var(--background-modifier-border-hover); +} +.modal.mod-settings button:not(.mod-cta):not(.mod-warning), +.modal button:not(.mod-warning), +.modal.mod-settings button:not(.mod-warning) { + background-color: var(--background-button); + color: var(--text-normal); + border: 1px solid var(--background-modifier-border); + box-shadow: 0 1px 1px 0px rgba(0, 0, 0, 0.05); + cursor: var(--cursor); + height: var(--input-height); + line-height: 0; + white-space: nowrap; +} +button:hover, +.modal button:not(.mod-warning):hover, +.modal.mod-settings button:not(.mod-warning):hover { + box-shadow: 0 2px 3px 0px rgba(0, 0, 0, 0.05); + background-color: var(--background-button); + border-color: var(--background-modifier-border-hover); +} +.dropdown, +select { + box-shadow: 0 1px 1px 0px rgba(0, 0, 0, 0.05); + background-color: var(--background-button); + border-color: var(--background-modifier-border); + transition: border-color 0.1s linear; +} +.dropdown:hover, +select:hover { + background-color: var(--background-button); + box-shadow: 0 2px 3px 0px rgba(0, 0, 0, 0.05); +} + +/* --------------- */ +/* Checkboxes */ +/* --------------- */ + +input[type='checkbox'] { + -webkit-appearance: none; + appearance: none; + border-radius: 30%; + border: 2px solid var(--background-modifier-border-hover); + padding: 0; +} +input[type='checkbox']:focus, +input[type='checkbox']:hover { + outline: 0; + border-color: var(--text-faint); +} +.checklist-plugin-main .group .compact > .toggle .checked, +.is-flashing input[type='checkbox']:checked, +input[type='checkbox']:checked { + background-color: var(--blue) !important; + /* border: 2px solid var(--blue); */ + border: none; + background-position: center; + background-size: 70%; + background-repeat: no-repeat; + background-image: url('data:image/svg+xml; utf8, '); +} +.markdown-preview-section > .contains-task-list { + padding-bottom: 0.5em; +} +.markdown-preview-view ul > li.task-list-item.is-checked, +.markdown-source-view.mod-cm6 .HyperMD-task-line[data-task='x'], +.markdown-source-view.mod-cm6 .HyperMD-task-line[data-task='X'] { + text-decoration: none; +} +.markdown-preview-view .task-list-item-checkbox { + width: 16px; + height: 16px; + position: relative; + top: 6px; + line-height: 0; + margin-left: -1.5em; + margin-right: 6px; + filter: none; +} +.markdown-preview-view ul > li.task-list-item { + text-indent: 0; + line-height: 1.4; +} +.markdown-preview-view .task-list-item { + padding-inline-start: 0; +} +.side-dock-plugin-panel-inner { + padding-right: 6px; + padding-left: 6px; +} + +/* --------------- */ +/* Toggle switches */ +/* --------------- */ + +.checkbox-container { + background-color: var(--background-modifier-border-hover); + box-shadow: inset 0 0px 1px 0px rgba(0, 0, 0, 0.2); + border: none; + width: 40px; + height: 24px; + cursor: var(--cursor); +} +.checkbox-container:after { + background: white; + border: none; + margin: 3px 0 0 0; + height: 18px; + width: 18px; + border-radius: 26px; + box-shadow: 0 1px 2px 0px rgba(0, 0, 0, 0.1); + transition: all 0.1s linear; +} +.checkbox-container:hover:after { + box-shadow: 0 2px 3px 0px rgba(0, 0, 0, 0.1); + transition: all 0.1s linear; +} +.checkbox-container.is-enabled { + border-color: var(--interactive-accent); +} + +/* --------------- */ +/* File browser */ +/* --------------- */ + +.nav-header { + padding: 0; +} +.nav-buttons-container { + padding: 10px 5px 0px 5px; + margin-bottom: 0px !important; + justify-content: flex-start; + border: 0; +} +.nav-files-container { + overflow-x: hidden; + padding-bottom: 50px; + padding-left: 5px; +} +.nav-folder-title { + margin: 0 0 0 8px; + min-width: auto; + width: calc(100% - 16px); + padding: 0 10px 0 16px; + line-height: 1.5; + cursor: var(--cursor); +} +.nav-folder-children .nav-folder-children { + margin-left: 20px; + padding-left: 0; + border-left: 1px solid var(--background-modifier-border); +} +.nav-folder.mod-root > .nav-folder-title.is-being-dragged-over { + background-color: var(--text-selection); +} +.nav-folder-title.is-being-dragged-over { + background-color: var(--text-selection); + border-color: var(--text-selection); + border-radius: 6px; + border: 1px solid transparent; +} +.nav-folder-title-content { + padding: 0px 4px 1px 0; + font-weight: 600; +} +.nav-folder-collapse-indicator { + top: 1px; + margin-left: -10px; +} +.nav-file { + margin-left: 12px; + padding-right: 4px; +} +.workspace-leaf.mod-active .nav-folder.has-focus > .nav-folder-title, +.workspace-leaf.mod-active .nav-file.has-focus { + border: 1px solid transparent; +} +.nav-file-title { + width: calc(100% - 30px); + margin: 0 8px 0 -4px; + padding: 2px 2px; + border-width: 0; + line-height: 1.6; + border-color: var(--background-secondary); + border-radius: 6px; + cursor: var(--cursor); +} +.nav-file-title.is-being-dragged, +.nav-file-title.is-active, +body:not(.is-grabbing) .nav-file-title.is-active:hover { + background-color: var(--background-tertiary); + color: var(--text-normal); +} +.nav-file-title-content { + width: 100%; + white-space: nowrap; + overflow: hidden; + text-overflow: ellipsis; + padding: 0 5px; + vertical-align: middle; + cursor: var(--cursor); +} +.drop-indicator { + border-width: 1px; +} +.nav-file-icon { + margin: 1px 0 0 0; + vertical-align: bottom; + padding: 0 0 0 5px; +} +.workspace-leaf-content[data-type='starred'] .nav-file-title-content { + width: calc(100% - 15px); +} +body:not(.is-grabbing) .nav-file-title:hover .nav-folder-collapse-indicator, +body:not(.is-grabbing) .nav-folder-title:hover .nav-folder-collapse-indicator, +body:not(.is-grabbing) .nav-file-title:hover, +body:not(.is-grabbing) .nav-folder-title:hover { + background: transparent; +} + +/* Tooltip */ + +.tooltip { + font-size: var(--font-adaptive-small); + line-height: 1.2; + padding: 4px 8px; + border-radius: 4px; +} + +/* Sidebar font size */ +.nav-file-title, +.tree-item-self, +.nav-folder-title, +.is-collapsed .search-result-file-title, +.tag-pane-tag { + font-size: var(--font-adaptive-small); + color: var(--text-muted); +} +.search-result-file-title { + font-size: var(--font-adaptive-small); + color: var(--text-normal); + font-weight: var(--normal-weight); +} +.side-dock-collapsible-section-header { + font-size: var(--font-adaptive-small); + color: var(--text-muted); + cursor: var(--cursor); + margin-right: 0; + margin-left: 0; +} +.side-dock-collapsible-section-header:hover, +.side-dock-collapsible-section-header:not(.is-collapsed) { + color: var(--text-muted); + background: transparent; +} +.tree-view-item-self:hover .tree-view-item-collapse, +.collapsible-item-self.is-clickable:hover { + color: var(--text-muted); + background: transparent; + cursor: var(--cursor); +} +.collapsible-item-self.is-clickable { + cursor: var(--cursor); +} +.search-result-collapse-indicator svg, +.search-result-file-title:hover .search-result-collapse-indicator svg, +.side-dock-collapsible-section-header-indicator:hover svg, +.side-dock-collapsible-section-header:hover + .side-dock-collapsible-section-header-indicator + svg, +.markdown-preview-view .collapse-indicator svg, +.tree-view-item-collapse svg, +.is-collapsed .search-result-collapse-indicator svg, +.nav-folder-collapse-indicator svg, +.side-dock-collapsible-section-header-indicator svg, +.is-collapsed .side-dock-collapsible-section-header-indicator svg { + color: var(--text-faint); + cursor: var(--cursor); +} +.search-result-collapse-indicator, +.search-result-file-title:hover .search-result-collapse-indicator, +.side-dock-collapsible-section-header-indicator:hover, +.side-dock-collapsible-section-header:hover + .side-dock-collapsible-section-header-indicator, +.markdown-preview-view .collapse-indicator, +.tree-view-item-collapse, +.is-collapsed .search-result-collapse-indicator, +.nav-folder-collapse-indicator, +.side-dock-collapsible-section-header-indicator, +.is-collapsed .side-dock-collapsible-section-header-indicator { + color: var(--text-faint); + cursor: var(--cursor); +} +.nav-folder-title.is-being-dragged-over .nav-folder-collapse-indicator svg { + color: var(--text-normal); +} + +/* --------------- */ +/* Relationship lines */ +/* --------------- */ + +/* Relationship lines in Preview */ + +ul { + position: relative; +} +.markdown-preview-view ul ul::before { + content: ''; + border-right: 1px solid var(--background-modifier-border); + position: absolute; + left: -0.85em !important; + top: 0; + bottom: 0; +} +.markdown-preview-view ul.contains-task-list::before { + top: 5px; +} +.markdown-preview-view .task-list-item-checkbox { + margin-left: -1.3em; +} + +/* Relationship lines in Edit mode */ + +.cm-hmd-list-indent > .cm-tab { + display: inline-block; +} +.cm-hmd-list-indent > .cm-tab:after { + content: ' '; + display: block; + width: 1px; + position: absolute; + top: 1px; + border-right: 1px solid var(--background-modifier-border); + height: 100%; +} + +/* --------------- */ +/* Folding offset */ +/* --------------- */ + +/* Add padding to account for gutter in Edit mode when folding is on */ + +body:not(.plugin-sliding-panes-rotate-header) .view-header-title, +.allow-fold-headings.markdown-preview-view .markdown-preview-sizer, +.allow-fold-lists.markdown-preview-view .markdown-preview-sizer { + padding: 0 8px 0 16px; +} +.allow-fold-lists.markdown-preview-view + .markdown-embed + .markdown-preview-sizer { + padding-left: 0; +} +.is-mobile .markdown-source-view.mod-cm6.is-readable-line-width .cm-gutters, +.is-mobile .markdown-source-view.mod-cm6.is-readable-line-width .cm-content { + transform: translateX(-10px) !important; +} +.CodeMirror-sizer { + padding-right: 12px !important; +} + +/* Folding icons in Preview */ + +.markdown-preview-view .heading-collapse-indicator.collapse-indicator svg, +.markdown-preview-view ol > li .collapse-indicator svg, +.markdown-preview-view ul > li .collapse-indicator svg { + opacity: 0; +} + +h1:hover .heading-collapse-indicator.collapse-indicator svg, +h2:hover .heading-collapse-indicator.collapse-indicator svg, +h3:hover .heading-collapse-indicator.collapse-indicator svg, +h4:hover .heading-collapse-indicator.collapse-indicator svg, +h5:hover .heading-collapse-indicator.collapse-indicator svg, +.markdown-preview-view .is-collapsed .collapse-indicator svg, +.markdown-preview-view .collapse-indicator:hover svg { + opacity: 1; +} +.markdown-preview-view div.is-collapsed h1::after, +.markdown-preview-view div.is-collapsed h2::after, +.markdown-preview-view div.is-collapsed h3::after, +.markdown-preview-view div.is-collapsed h4::after, +.markdown-preview-view div.is-collapsed h5::after, +.markdown-preview-view ol .is-collapsed::after, +.markdown-preview-view ul .is-collapsed::after { + content: '...'; + padding: 5px; + color: var(--text-faint); +} +.markdown-preview-view ol > li.task-list-item .collapse-indicator, +.markdown-preview-view ul > li.task-list-item .collapse-indicator { + position: absolute; + margin-left: -42px; + margin-top: 5px; +} +.markdown-preview-view ol > li .collapse-indicator { + padding-right: 20px; +} +.markdown-preview-view .heading-collapse-indicator.collapse-indicator { + margin-left: -25px; + padding-right: 7px 8px 7px 0; +} +.markdown-preview-view .collapse-indicator { + position: absolute; + margin-left: -42px; + padding-bottom: 10px; + padding-top: 0px; +} +.markdown-preview-view ul > li:not(.task-list-item) .collapse-indicator { + padding-right: 20px; +} +.markdown-preview-view ul > li:not(.task-list-item)::marker { + font-size: 0.9em; +} +.markdown-preview-view ul > li:not(.task-list-item).is-collapsed::before { + background: var(--background-modifier-border); + box-shadow: 3px 0 0px 4px var(--background-modifier-border); +} +.list-collapse-indicator .collapse-indicator .collapse-icon { + opacity: 0; +} +.markdown-preview-view ul > li h1, +.markdown-preview-view ul > li h2, +.markdown-preview-view ul > li h3, +.markdown-preview-view ul > li h4 { + display: inline; +} + +/* Folding icons in Edit mode */ + +span[title='Fold line'], +span[title='Unfold line'] { + margin: 0 0 0 0; + padding: 0 0 1em 0; +} + +.CodeMirror-foldmarker { + color: var(--text-faint); + cursor: default; + margin-left: 5px; +} +.CodeMirror-foldgutter-folded { + cursor: var(--cursor); + margin-top: -3px; + transform: rotate(-90deg); +} +.CodeMirror-foldgutter-open { + cursor: var(--cursor); + margin-top: -1px; + width: 16px; + height: 20px; +} +span[title='Fold line'], +span[title='Unfold line'], +.CodeMirror-foldgutter-folded:after, +.CodeMirror-foldgutter-open:after { + background-repeat: no-repeat; + background-position: 50% 50%; + background-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100' width='8' height='8' class='right-triangle'%3E%3Cpath fill='currentColor' stroke='currentColor' d='M94.9,20.8c-1.4-2.5-4.1-4.1-7.1-4.1H12.2c-3,0-5.7,1.6-7.1,4.1c-1.3,2.4-1.2,5.2,0.2,7.6L43.1,88c1.5,2.3,4,3.7,6.9,3.7 s5.4-1.4,6.9-3.7l37.8-59.6C96.1,26,96.2,23.2,94.9,20.8L94.9,20.8z'%3E%3C/path%3E%3C/svg%3E"); + color: transparent; +} +span[title='Unfold line'] { + background-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100' width='8' height='8' class='right-triangle'%3E%3Cpath fill='currentColor' stroke='currentColor' transform='rotate(-90,50,50)' d='M94.9,20.8c-1.4-2.5-4.1-4.1-7.1-4.1H12.2c-3,0-5.7,1.6-7.1,4.1c-1.3,2.4-1.2,5.2,0.2,7.6L43.1,88c1.5,2.3,4,3.7,6.9,3.7 s5.4-1.4,6.9-3.7l37.8-59.6C96.1,26,96.2,23.2,94.9,20.8L94.9,20.8z'%3E%3C/path%3E%3C/svg%3E"); +} +.theme-dark span[title='Fold line'], +.theme-dark span[title='Unfold line'], +.theme-dark .CodeMirror-foldgutter-folded:after, +.theme-dark .CodeMirror-foldgutter-open:after { + background-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100' width='8' height='8' class='right-triangle'%3E%3Cpath fill='%23FFFFFF' stroke='%23FFFFFF' d='M94.9,20.8c-1.4-2.5-4.1-4.1-7.1-4.1H12.2c-3,0-5.7,1.6-7.1,4.1c-1.3,2.4-1.2,5.2,0.2,7.6L43.1,88c1.5,2.3,4,3.7,6.9,3.7 s5.4-1.4,6.9-3.7l37.8-59.6C96.1,26,96.2,23.2,94.9,20.8L94.9,20.8z'%3E%3C/path%3E%3C/svg%3E"); +} +span[title='Fold line'], +.CodeMirror-foldgutter-open:after { + opacity: 0; +} +span[title='Fold line']:hover, +span[title='Unfold line'], +.CodeMirror-foldgutter-folded:after, +.CodeMirror-code > div:hover .CodeMirror-foldgutter-open:after { + opacity: 0.3; +} +span[title='Unfold line']:hover, +.CodeMirror-code > div:hover .CodeMirror-foldgutter-open:hover:after, +.CodeMirror-code > div:hover .CodeMirror-foldgutter-folded:hover:after { + opacity: 1; +} + +/* --------------- */ +/* Outline */ +/* --------------- */ + +.outline { + padding: 15px 10px 20px 5px; + font-size: var(--font-adaptive-small); +} +.outline .pane-empty { + font-size: var(--font-adaptive-small); + color: var(--text-faint); + padding: 0 0 0 15px; + width: 100%; +} +.outline .collapsible-item-self { + cursor: var(--cursor); + line-height: 1.4; + margin-bottom: 4px; + font-size: var(--font-adaptive-small); + padding-left: 15px; +} +.collapsible-item-collapse { + opacity: 1; + left: -5px; + color: var(--text-faint); +} +.outline .collapsible-item-inner:hover { + color: var(--text-normal); +} +.collapsible-item-self.is-clickable:hover .collapsible-item-collapse { + color: var(--text-normal); +} +.outline > .collapsible-item > .collapsible-item-self .right-triangle { + opacity: 0; +} + +/* --------------- */ +/* Search */ +/* --------------- */ + +.search-result-container.mod-global-search .search-empty-state { + padding-left: 15px; +} +.search-result-file-match { + cursor: var(--cursor) !important; +} +.search-result-file-match:hover { + color: var(--text-normal); + background: transparent; +} +.search-result-container:before { + height: 1px; +} +.search-result-container.is-loading:before { + background-color: var(--background-modifier-accent); +} +.search-result { + margin-bottom: 0; +} +.search-result-count { + opacity: 1; + color: var(--text-faint); + padding: 0 0 0 5px; +} +.search-result-file-match:before { + top: 0; +} +.search-result-file-match:not(:first-child) { + margin-top: 0px; +} +.search-result-file-match { + margin-top: 0; + margin-bottom: 0; + padding-top: 6px; + padding-bottom: 5px; +} +.search-input-container input, +.search-input-container input:hover, +.search-input-container input:focus { + font-size: var(--font-adaptive-small); + padding: 5px 10px; + background-color: var(--background-secondary); +} +.search-input-container { + width: calc(100% - 20px); + margin: 0 0 5px 10px; +} +/* .search-result-file-matched-text { + background-color: var(--text-selection); +} */ +.workspace-leaf-content .setting-item { + padding: 5px 0; + border: none; +} +.workspace-leaf-content .setting-item-control { + flex-shrink: 0; + flex: 1; +} +.search-input-clear-button { + cursor: var(--cursor); + top: 0px; + bottom: 0px; + border-radius: 15px; + line-height: 0px; + height: 15px; + width: 15px; + margin: auto; + padding: 6px 0 0 0; + text-align: center; + vertical-align: middle; + align-items: center; + color: var(--text-faint); +} +.search-input-clear-button:hover { + color: var(--text-normal); +} +.search-input-clear-button:before { + font-size: 22px; + font-weight: 200; +} +.search-input { + max-width: 100%; + margin-left: 0; + width: 500px; +} +input.search-input:focus { + border-color: var(--background-modifier-border); +} +.workspace-leaf-content[data-type='search'] .search-result-file-matches { + border-left: 0; + padding-left: 0; +} +.search-empty-state { + font-size: var(--font-adaptive-small); + color: var(--text-faint); + padding-left: 5px; + margin: 0; +} +.search-result-container { + padding: 5px 10px 50px 0px; +} +.search-result-file-title { + line-height: 1.3; + padding: 4px 4px 4px 24px; + vertical-align: middle; + cursor: var(--cursor) !important; +} +.tree-item-inner, +.search-result-file-title { + white-space: nowrap; + overflow: hidden; + text-overflow: ellipsis; +} +.search-result-collapse-indicator { + left: 0px; +} +.search-result-file-match:before { + height: 0.5px; +} +.search-result-file-matches { + font-size: var(--font-adaptive-small); + line-height: 1.4; + margin-bottom: 8px; + padding: 0 0 6px 0; + color: var(--text-muted); + border-bottom: 1px solid var(--background-modifier-border-focus); +} +.search-info-container { + font-size: var(--font-adaptive-smaller); + color: var(--text-faint); + padding-top: 5px; + padding-bottom: 5px; +} +.search-info-more-matches { + font-size: var(--font-adaptive-smaller); + padding-top: 4px; + padding-bottom: 4px; + color: var(--text-normal); +} +.side-dock-collapsible-section-header-indicator { + display: none; +} +.search-result-file-title:hover { + color: var(--text-normal); + background: transparent; +} +.workspace-leaf-content .search-input, +.workspace-leaf-content .search-input:hover, +.workspace-leaf-content .search-input:focus { + font-size: var(--font-adaptive-small); + padding: 7px 10px; + height: 28px; + border-radius: 5px; + background: var(--background-primary); + border: 1px solid var(--background-modifier-border); + transition: border-color 0.1s ease-in-out; +} +.workspace-leaf-content .search-input:hover { + border-color: var(--background-modifier-border-hover); + transition: border-color 0.1s ease-in-out; +} +.workspace-leaf-content .search-input:focus { + background: var(--background-primary); + border-color: var(--background-modifier-border-focus); + transition: all 0.1s ease-in-out; +} +.search-input-container input::placeholder { + color: var(--text-faint); + font-size: var(--font-adaptive-small); +} +.workspace-split.mod-root + .workspace-split.mod-vertical + .workspace-leaf-content { + padding-right: 0; +} +.workspace-split.mod-horizontal.mod-right-split { + width: 0; +} +.workspace-split.mod-vertical > .workspace-leaf { + padding-right: 1px; +} +.workspace-leaf-content[data-type='starred'] .item-list { + padding-top: 5px; +} +.workspace-leaf-content .view-content, +.workspace-split.mod-right-split .view-content { + padding: 0; +} + +/* --------------- */ +/* Nested items */ +/* --------------- */ + +.nav-folder-collapse-indicator, +.tree-item-self .collapse-icon { + color: var(--background-modifier-border-hover); +} +.tree-item-self .collapse-icon { + padding-left: 0; + width: 15px; + margin-left: -15px; +} +.outline .tree-item-self .collapse-icon { + margin-left: -20px; +} +.tag-container .collapse-icon { + margin-left: -20px; +} +.tree-item-self:hover .collapse-icon { + color: var(--text-normal); +} +.tree-item { + padding-left: 5px; +} +.tree-item-flair { + font-size: var(--font-adaptive-smaller); + right: 0; + background: transparent; + color: var(--text-faint); +} +.tree-item-flair-outer:after { + content: ''; +} +.tree-item-self.is-clickable { + cursor: var(--cursor); +} +.tree-item-self.is-clickable:hover { + background: transparent; +} +.tree-item-self:hover .tree-item-flair { + background: transparent; + color: var(--text-muted); +} +.tree-item-children { + margin-left: 5px; +} + +/* --------------- */ +/* Backlink pane */ +/* --------------- */ + +.outgoing-link-pane, +.backlink-pane { + padding-bottom: 30px; +} +.outgoing-link-pane .search-result-container, +.backlink-pane .search-result-container { + padding: 5px 5px 5px 5px; + margin-left: 0; +} +.outgoing-link-pane .search-result-file-title, +.backlink-pane .search-result-file-title { + padding-left: 15px; +} +.outgoing-link-pane .tree-item-icon, +.outgoing-link-pane > .tree-item-self .collapse-icon, +.backlink-pane > .tree-item-self .collapse-icon { + display: none; +} + +.tree-item-self.outgoing-link-item { + padding: 0; + margin-left: 5px; +} + +.outgoing-link-pane > .tree-item-self:hover, +.outgoing-link-pane > .tree-item-self, +.backlink-pane > .tree-item-self:hover, +.backlink-pane > .tree-item-self { + padding-left: 15px; + color: var(--text-muted); + text-transform: uppercase; + letter-spacing: 0.05em; + font-size: var(--font-adaptive-smaller); + font-weight: 500; + padding: 10px 7px 5px 10px; + background: transparent; +} + +.outgoing-link-pane > .tree-item-self.is-collapsed, +.backlink-pane > .tree-item-self.is-collapsed { + color: var(--text-faint); +} + +.outgoing-link-pane .search-result-file-match { + padding: 5px 0; + border: 0; +} +.outgoing-link-pane .search-result-file-match-destination-file { + background: transparent; +} +.search-result-file-match:hover + .search-result-file-match-destination-file:hover { + background: transparent; + color: var(--text-normal); +} + +/* --------------- */ +/* Tag pane */ +/* --------------- */ + +.tag-container { + padding: 10px 15px; +} +.tag-pane-tag-count { + margin-right: 10px; + color: var(--text-faint); +} +.pane-list-item-ending-flair { + background: transparent; +} +.tag-pane-tag { + padding: 2px 5px 2px 5px; + cursor: var(--cursor); +} +.tag-pane-tag:hover { + background: transparent; +} +.nav-file.is-active .nav-file-title:hover { + background: var(--background-tertiary) !important; +} +.nav-file.is-active > .nav-file-title { + background: var(--background-tertiary); +} + +/* --------------- */ +/* Status bar */ +/* --------------- */ + +.status-bar { + transition: color 0.2s linear; + color: var(--text-faint); + font-size: var(--font-adaptive-smaller); + border-top: var(--border-width) solid var(--background-divider); + line-height: 1; + max-height: 24px; +} +.minimal-status-off .status-bar { + background-color: var(--background-secondary); + border-width: var(--border-width); + padding: 2px 6px 4px; +} +.status-bar { + background-color: var(--background-primary); + z-index: 30; + border-top-left-radius: 5px; + width: auto; + position: absolute; + left: auto; + border: 0; + bottom: 0; + right: 0; + max-height: 26px; + padding: 2px 8px 6px 3px; +} +.sync-status-icon.mod-success, +.sync-status-icon.mod-working { + color: var(--text-faint); + cursor: var(--cursor); +} +.status-bar:hover, +.status-bar:hover .sync-status-icon.mod-success, +.status-bar:hover .sync-status-icon.mod-working { + color: var(--text-muted); + transition: color 0.2s linear; +} +.status-bar .plugin-sync:hover .sync-status-icon.mod-success, +.status-bar .plugin-sync:hover .sync-status-icon.mod-working { + color: var(--text-normal); +} +.status-bar-item-segment { + margin-right: 10px; +} +.status-bar-item { + cursor: var(--cursor) !important; +} +/* .status-bar-item.cMenu-statusbar-button:hover, +.status-bar-item.mod-clickable:hover, +.status-bar-item.plugin-sync:hover { + text-align: center; + background-color: var(--background-tertiary) !important; + border-radius: 4px; +} */ +.status-bar-item { + padding: 7px 4px; + margin: 0; +} +.status-bar-item, +.sync-status-icon { + display: flex; + align-items: center; +} +.status-bar-item.plugin-sync svg { + height: 15px; + width: 15px; +} + +/* --------------- */ +/* Workplace ribbon & sidedock icons */ +/* --------------- */ + +.workspace-ribbon { + flex: 0 0 42px; + padding-top: 7px; +} +.workspace-ribbon.mod-right { + right: 4px; + bottom: 0; + height: 32px; + padding-top: 6px; + position: absolute; + background: 0 0; + border: 0; +} +.workspace-ribbon-collapse-btn { + margin: 0; + padding: 5px 4px; + border-radius: 5px; +} +.mod-right .workspace-ribbon-collapse-btn { + background-color: var(--background-primary); +} +.mod-right .workspace-ribbon-collapse-btn:hover { + background-color: var(--background-tertiary); +} +.workspace-ribbon.mod-left .workspace-ribbon-collapse-btn, +.workspace-ribbon.mod-right .workspace-ribbon-collapse-btn { + opacity: 1; + position: fixed; + width: 26px; + display: flex; + align-items: center; + top: auto; + text-align: center; + bottom: 42px; + right: 15px; + z-index: 9; +} +.workspace-ribbon.mod-left .workspace-ribbon-collapse-btn { + left: 8px; +} +.side-dock-settings { + padding-bottom: 30px; +} +.workspace-ribbon-collapse-btn, +.view-action, +.side-dock-ribbon-tab, +.side-dock-ribbon-action { + cursor: var(--cursor); +} +.workspace-ribbon { + border-width: var(--border-width-alt); + border-color: var(--background-modifier-border); + background: var(--background-secondary); +} +.mod-right:not(.is-collapsed) ~ .workspace-split.mod-right-split { + margin-right: 0; +} +.side-dock-ribbon-action { + padding: 6px 0; +} +body.hider-frameless:not(.hider-ribbon):not(.is-fullscreen) .side-dock-actions { + padding-top: 24px; +} +body.hider-frameless:not(.hider-ribbon):not(.is-fullscreen) + .workspace-ribbon-collapse-btn { + margin: 0; + padding-top: 40px; +} +.workspace-ribbon.mod-right { + right: 7px; /* DO NOT CHANGE */ +} + +/* --------------- */ +/* Preview mode */ +/* --------------- */ + +.markdown-preview-view hr { + height: 1px; + border-width: 2px 0 0 0; +} +a[href*="obsidian://search"] +{ + background-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100' width='17' height='17' class='search'%3E%3Cpath fill='black' stroke='black' stroke-width='2' d='M42,6C23.2,6,8,21.2,8,40s15.2,34,34,34c7.4,0,14.3-2.4,19.9-6.4l26.3,26.3l5.6-5.6l-26-26.1c5.1-6,8.2-13.7,8.2-22.1 C76,21.2,60.8,6,42,6z M42,10c16.6,0,30,13.4,30,30S58.6,70,42,70S12,56.6,12,40S25.4,10,42,10z'%3E%3C/path%3E%3C/svg%3E"); +} +.theme-dark a[href*="obsidian://search"] +{ + background-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 100 100' width='17' height='17' class='search'%3E%3Cpath fill='white' stroke='white' stroke-width='2' d='M42,6C23.2,6,8,21.2,8,40s15.2,34,34,34c7.4,0,14.3-2.4,19.9-6.4l26.3,26.3l5.6-5.6l-26-26.1c5.1-6,8.2-13.7,8.2-22.1 C76,21.2,60.8,6,42,6z M42,10c16.6,0,30,13.4,30,30S58.6,70,42,70S12,56.6,12,40S25.4,10,42,10z'%3E%3C/path%3E%3C/svg%3E"); +} + +/* Style settings to toggle link underlines */ +body:not(.links-int-on) a[href*="obsidian://"], +body:not(.links-int-on) .markdown-preview-view .internal-link, +body:not(.links-int-on) .markdown-source-view.mod-cm6 .cm-hmd-internal-link .cm-underline, +body:not(.links-ext-on) .external-link, +body:not(.links-ext-on) .markdown-source-view.mod-cm6 .cm-link .cm-underline { + text-decoration: none; +} + +.footnotes-list { + margin-block-start: -10px; + padding-inline-start: 20px; + font-size: var(--font-adaptive-small); +} +.footnotes-list p { + display: inline; + margin-block-end: 0; + margin-block-start: 0; +} +.footnote-ref a { + text-decoration: none; +} +.footnote-backref { + color: var(--text-faint); +} +iframe { + border: 0; +} +.markdown-preview-view .mod-highlighted { + transition: background-color 0.3s ease; + background-color: var(--text-selection); + color: inherit; +} + +/* Metadata */ + +.frontmatter-collapse-indicator.collapse-indicator { + display: none; +} +.frontmatter-container .tag { + font-size: var(--font-adaptive-smaller); +} +.frontmatter-container .frontmatter-alias { + color: var(--text-muted); +} +.frontmatter-container { + border: 1px solid var(--background-modifier-border); + font-size: 14px; + color: var(--text-muted); + padding: 6px 14px; + border-radius: 4px; + background-color: var(--background-primary-alt); + position: relative; + margin-top: 16px; +} + +/* Blockquotes */ + +.markdown-preview-view blockquote { + border-radius: 0; + border: solid var(--quote-opening-modifier); + border-width: 0px 0px 0px 2px; + background-color: transparent; + font-style: italic; + padding: 0 0 0 calc(var(--nested-padding) / 2); + margin-inline-start: var(--nested-padding); +} + +.cm-s-obsidian span.cm-quote { + font-style: italic; +} + +body:not(.default-font-color) .cm-s-obsidian span.cm-quote, +body:not(.default-font-color) .markdown-preview-view blockquote { + color: var(--green); +} + +/* --------------- +TEXT MARKINGS +--------------- */ + +/* Hashes */ + +span.cm-formatting { + color: var(--text-faint); +} + +/* Italics */ + +body:not(.default-font-color) em, +body:not(.default-font-color) .cm-s-obsidian .cm-em.cm-header, +body:not(.default-font-color) .cm-s-obsidian .cm-em.cm-header.cm-header-1, +body:not(.default-font-color) .cm-s-obsidian .cm-em.cm-header.cm-header-2, +body:not(.default-font-color) .cm-s-obsidian .cm-em.cm-header.cm-header-3, +body:not(.default-font-color) .cm-s-obsidian .cm-em.cm-header.cm-header-4, +body:not(.default-font-color) .cm-s-obsidian .cm-em.cm-header.cm-header-5, +body:not(.default-font-color) .cm-s-obsidian .cm-em.cm-header.cm-header-6, +body:not(.default-font-color) .cm-s-obsidian .cm-em.cm-header.cm-header-1, +body:not(.default-font-color) .cm-s-obsidian .cm-em.cm-header.cm-header-2, +body:not(.default-font-color) .cm-s-obsidian .cm-em.cm-header.cm-header-3, +body:not(.default-font-color) .cm-s-obsidian .cm-em.cm-header.cm-header-4, +body:not(.default-font-color) .cm-s-obsidian .cm-em.cm-header.cm-header-5, +body:not(.default-font-color) .cm-s-obsidian .cm-em.cm-header.cm-header-6, +body:not(.default-font-color) .markdown-preview-section em, +body:not(.default-font-color) .cm-s-obsidian .cm-em { + font-style: italic; + color: var(--em-color); +} + +/* Bold */ + +body:not(.default-font-color) strong, +body:not(.default-font-color) .cm-s-obsidian .cm-strong.cm-header, +body:not(.default-font-color) .cm-s-obsidian .cm-strong.cm-header.cm-header-1, +body:not(.default-font-color) .cm-s-obsidian .cm-strong.cm-header.cm-header-2, +body:not(.default-font-color) .cm-s-obsidian .cm-strong.cm-header.cm-header-3, +body:not(.default-font-color) .cm-s-obsidian .cm-strong.cm-header.cm-header-4, +body:not(.default-font-color) .cm-s-obsidian .cm-strong.cm-header.cm-header-5, +body:not(.default-font-color) .cm-s-obsidian .cm-strong.cm-header.cm-header-6, +body:not(.default-font-color) .cm-s-obsidian .cm-strong.cm-header.cm-header-1, +body:not(.default-font-color) .cm-s-obsidian .cm-strong.cm-header.cm-header-2, +body:not(.default-font-color) .cm-s-obsidian .cm-strong.cm-header.cm-header-3, +body:not(.default-font-color) .cm-s-obsidian .cm-strong.cm-header.cm-header-4, +body:not(.default-font-color) .cm-s-obsidian .cm-strong.cm-header.cm-header-5, +body:not(.default-font-color) .cm-s-obsidian .cm-strong.cm-header.cm-header-6, +body:not(.default-font-color) .cm-header.cm-header-3.cm-hmd-internal-link, +body:not(.default-font-color) .markdown-preview-section strong, +body:not(.default-font-color) .cm-s-obsidian .cm-strong { + color: var(--strong-color); +} + +/* Strikethrough */ + +del, +.cm-strikethrough { + text-decoration-color: var(--text-muted); + text-decoration-thickness: 2px !important; +} + +/* Tables */ + +.markdown-preview-view th { + font-weight: var(--bold-weight); + text-align: left; + border-top: none; +} +.markdown-preview-view th:last-child, +.markdown-preview-view td:last-child { + border-right: none; +} +.markdown-preview-view th:first-child, +.markdown-preview-view td:first-child { + border-left: none; + padding-left: 0; +} +.markdown-preview-view tr:last-child td { + border-bottom: none; +} + +/* Number Tables */ +.numbertable table { + counter-reset: section; +} +.numbertable table > tbody > tr > td:first-child::before { + counter-increment: section; + content: counter(section) '. '; +} + +/* Color rows */ +.color-rows tr:nth-child(even) { + background: var(--background-primary); +} +.color-rows tr:nth-child(odd) { + background: var(--background-secondary); +} + +/* Lists */ +ul { + padding-inline-start: var(--list-indent); +} +ol { + padding-inline-start: var(--list-indent); + margin-left: 0; + list-style: default; +} +.is-mobile ul > li:not(.task-list-item)::marker { + font-size: 0.8em; +} +.is-mobile .markdown-rendered ol, +.is-mobile .markdown-rendered ul { + padding-inline-start: var(--list-indent); +} +.is-mobile .markdown-rendered div > ol, +.is-mobile .markdown-rendered div > ul { + padding-inline-start: 2em; +} +.is-mobile .el-ol > ol, +.is-mobile .el-ul > ul { + margin-left: 0; +} +.cm-line:not(.HyperMD-codeblock) { + tab-size: var(--list-indent); +} +ul > li { + min-height: 1.4em; +} +ul > li::marker, +ol > li::marker { + color: var(--text-faint); +} +ol > li { + margin-left: 0em; +} + +/* --------------- */ +/* Code */ +/* --------------- */ + +.markdown-preview-view code { + color: var(--code-color); +} +.cm-inline-code { + color: var(--code-color) !important; +} +.theme-light :not(pre) > code[class*='language-'], +.theme-light pre[class*='language-'] { + background-color: var(--background-primary-alt); +} +.theme-light code[class*='language-'], +.theme-light pre[class*='language-'] { + text-shadow: none; +} +/* Horizontal scroll */ +code[class*='language-'], +pre[class*='language-'] { + text-align: left !important; + white-space: pre !important; + word-spacing: normal !important; + word-break: normal !important; + word-wrap: normal !important; + line-height: 1.5 !important; + -moz-tab-size: 4 !important; + -o-tab-size: 4 !important; + tab-size: 4 !important; + -webkit-hyphens: none !important; + -moz-hyphens: none !important; + -ms-hyphens: none !important; + hyphens: none !important; +} +pre[class*='language-'] { + overflow: auto !important; +} +/* ------------------ */ +pre .copy-code-button { + border-radius: 5px; + background-color: var(--background-secondary-alt); +} +pre .copy-code-button:hover { + background-color: var(--background-tertiary); +} +.markdown-preview-section .frontmatter code { + color: var(--text-muted); + font-size: var(--font-adaptive-small); +} +.cm-s-obsidian .hmd-fold-html-stub, +.cm-s-obsidian .hmd-fold-code-stub, +.cm-s-obsidian.CodeMirror .HyperMD-hover > .HyperMD-hover-content code, +.cm-s-obsidian .cm-formatting-hashtag, +.cm-s-obsidian .cm-inline-code, +.cm-s-obsidian .HyperMD-codeblock, +.cm-s-obsidian .HyperMD-hr, +.cm-s-obsidian .cm-hmd-frontmatter, +.cm-s-obsidian .cm-hmd-orgmode-markup, +.cm-s-obsidian .cm-formatting-code, +.cm-s-obsidian .cm-math, +.cm-s-obsidian span.hmd-fold-math-placeholder, +.cm-s-obsidian .CodeMirror-linewidget kbd, +.cm-s-obsidian .hmd-fold-html kbd .CodeMirror-code { + font-family: var(--font-monospace); +} +.cm-s-obsidian .cm-hmd-frontmatter { + font-size: var(--font-adaptive-small); + color: var(--text-muted); +} +.markdown-source-view.mod-cm6 .code-block-flair { + color: var(--text-muted); +} + +/* ------------------- */ +/* Atom coloring */ +/* Source: https://github.com/AGMStudio/prism-theme-one-dark */ +/* ------------------- */ + +.token.comment, +.token.prolog, +.token.doctype, +.token.cdata { + color: var(--atom-gray-1) !important; +} +.token.punctuation, +.cm-hmd-codeblock, +.cm-bracket { + color: var(--atom-gray-2) !important; +} +code[class*='language-'], +.token.selector, +.token.tag, +code .cm-property, +.cm-def { + color: var(--atom-red) !important; +} +.token.property, +.token.boolean, +.token.number, +.token.constant, +.token.symbol, +.token.attr-name, +.token.deleted, +.cm-number { + color: var(--atom-orange) !important; +} +.token.string, +.token.char, +.token.attr-value, +.token.builtin, +.token.inserted, +.cm-hmd-codeblock.cm-string { + color: var(--atom-green) !important; +} +.token.operator, +.cm-operator, +.token.entity, +.token.url, +.language-css .token.string, +.style .token.string { + color: var(--atom-aqua) !important; +} +.token.atrule, +.token.keyword, +.cm-keyword { + color: var(--atom-purple) !important; +} +.token.function, +.token.macro.property, +.cm-variable { + color: var(--atom-blue) !important; +} +.token.class-name, +.cm-atom, +code .cm-tag, +.cm-type, +.theme-dark .cm-variable-2 { + color: var(--atom-yellow) !important; +} +.token.regex, +.token.important, +.token.variable { + color: var(--atom-purple) !important; +} +.token.important, +.token.bold { + font-weight: bold !important; +} +.token.italic { + font-style: italic !important; +} +.token.entity { + cursor: help !important; +} +pre.line-numbers { + position: relative !important; + padding-left: 3.8em !important; + counter-reset: linenumber !important; +} +pre.line-numbers > code { + position: relative !important; +} +.line-numbers .line-numbers-rows { + position: absolute !important; + pointer-events: none !important; + top: 0 !important; + font-size: 100% !important; + left: -3.8em !important; + width: 3em !important; + letter-spacing: -1px !important; + border-right: 0 !important; + -webkit-user-select: none !important; + -moz-user-select: none !important; + -ms-user-select: none !important; + user-select: none !important; +} +.line-numbers-rows > span { + pointer-events: none !important; + display: block !important; + counter-increment: linenumber !important; +} +.line-numbers-rows > span:before { + content: counter(linenumber) !important; + color: var(--syntax-gray-1) !important; + display: block !important; + padding-right: 0.8em !important; + text-align: right !important; +} +.cm-s-obsidian .HyperMD-codeblock { + line-height: 1.5 !important; +} +.markdown-source-view.mod-cm6.is-readable-line-width + .cm-editor + .HyperMD-codeblock.cm-line, +.mod-cm6 .cm-editor .HyperMD-codeblock.cm-line { + padding-left: 10px; + padding-right: 10px; +} +.markdown-source-view.mod-cm6 .code-block-flair { + font-size: var(--font-smaller); + padding: 5px 0; + color: var(--text-muted); +} + +/* --------------- */ +/* Popovers */ +/* --------------- */ + +.popover, +.popover.hover-popover { + min-height: 40px; + box-shadow: 0 20px 40px var(--background-modifier-box-shadow); + pointer-events: auto !important; + border: 1px solid var(--background-modifier-border); +} +.popover.hover-popover { + max-height: 40vh; +} +.popover .markdown-embed-link { + display: none; +} +.popover .markdown-embed .markdown-preview-view { + padding: 10px 20px 30px; +} +.popover.hover-popover .markdown-embed .markdown-embed-content { + max-height: none; +} +.popover.hover-popover.mod-empty { + padding: 20px 20px 20px 20px; + color: var(--text-muted); +} + +.popover.hover-popover .markdown-preview-view .table-view-table, +.popover.hover-popover .markdown-embed .markdown-preview-view { + font-size: 1.05em; +} + +.popover.hover-popover .markdown-embed h1, +.popover.hover-popover .markdown-embed h2, +.popover.hover-popover .markdown-embed h3, +.popover.hover-popover .markdown-embed h4 { + margin-top: 1rem; +} + +/* --------------- */ +/* Graphs */ + +/* Fill color for nodes */ +.graph-view.color-fill { + color: var(--text-muted); +} +/* Fill color for nodes on hover */ +.graph-view.color-fill-highlight { + color: var(--text-accent); +} +/* Stroke color for nodes */ +.graph-view.color-circle { + color: var(--text-accent); +} +/* Line color */ +.graph-view.color-line { + color: var(--background-modifier-border); +} +/* Line color on hover */ +.graph-view.color-line-highlight { + color: var(--text-accent); + border: 0; +} +/* Text color */ +.graph-view.color-text { + color: var(--text-normal); +} +.graph-view.color-fill-unresolved { + color: var(--text-faint); +} + +/* Full bleed (takes up full height) */ + +body:not(.plugin-sliding-panes-rotate-header) + .workspace-leaf-content[data-type='localgraph'] + .view-header, +body:not(.plugin-sliding-panes-rotate-header) + .workspace-leaf-content[data-type='graph'] + .view-header { + position: fixed; + background: transparent !important; + width: 100%; +} +body:not(.plugin-sliding-panes-rotate-header) + .workspace-leaf-content[data-type='localgraph'] + .view-content, +body:not(.plugin-sliding-panes-rotate-header) + .workspace-leaf-content[data-type='graph'] + .view-content { + height: 100%; +} +body:not(.plugin-sliding-panes-rotate-header) + .workspace-leaf-content[data-type='localgraph'] + .view-header-title, +body:not(.plugin-sliding-panes-rotate-header) + .workspace-leaf-content[data-type='graph'] + .view-header-title { + display: none; +} +body:not(.plugin-sliding-panes-rotate-header) + .workspace-leaf-content[data-type='localgraph'] + .view-actions, +body:not(.plugin-sliding-panes-rotate-header) + .workspace-leaf-content[data-type='graph'] + .view-actions { + background: transparent; +} +.mod-root .workspace-leaf-content[data-type='localgraph'] .graph-controls, +.mod-root .workspace-leaf-content[data-type='graph'] .graph-controls { + top: 30px; +} + +.mod-root .workspace-leaf-content[data-type='localgraph'] .graph-controls, +.mod-root .workspace-leaf-content[data-type='graph'] .graph-controls { + top: 30px; +} + +/* Graph controls */ + +.graph-control-section .tree-item-children { + padding-bottom: 15px; +} +.graph-control-section-header { + font-weight: 500; + text-transform: uppercase; + letter-spacing: 0.05em; + font-size: var(--font-adaptive-smallest); + color: var(--text-muted); +} +.graph-controls .search-input-container { + width: 100%; +} +.setting-item.mod-search-setting.has-term-changed .graph-control-search-button, +.graph-controls .graph-control-search-button { + display: none; +} +.graph-controls .setting-item-name { + font-size: var(--font-adaptive-small); +} +.graph-controls { + background: var(--background-primary); + border: none; + min-width: 240px; + left: 0; + top: 10px; + margin-bottom: 0; + padding: 10px 20px 10px 10px; + border-radius: 0; +} +.graph-controls input[type='text'], +.graph-controls input[type='range'] { + font-size: var(--font-adaptive-small); +} +.graph-controls .mod-cta { + width: 100%; + font-size: var(--font-adaptive-small); + padding: 5px; +} + +.mod-left-split .graph-controls { + background: var(--background-secondary); +} +input[type='range'] { + background-color: var(--background-modifier-border-hover); + height: 2px; + padding: 0 0px; + -webkit-appearance: none; + cursor: default; + margin: 0; + border-radius: 0px; +} +input[type='range']::-webkit-slider-runnable-track { + background: var(--background-modifier-border-hover); + height: 2px; + margin-top: 0px; +} +input[type='range']::-webkit-slider-thumb { + background: white; + border: 1px solid var(--background-modifier-border-hover); + height: 18px; + width: 18px; + border-radius: 16px; + margin-top: -5px; + transition: all 0.1s linear; + cursor: default; + box-shadow: 0 1px 1px 0px rgba(0, 0, 0, 0.05), + 0 2px 4px 0px rgba(0, 0, 0, 0.1); +} +input[type='range']::-webkit-slider-thumb:hover, +input[type='range']::-webkit-slider-thumb:active { + background: white; + border-width: 1; + border: 1px solid var(--background-modifier-border-focus); + box-shadow: 0 1px 2px 0px rgba(0, 0, 0, 0.05), + 0 2px 3px 0px rgba(0, 0, 0, 0.2); + transition: all 0.1s linear; +} + +.local-graph-jumps-slider-container, +.workspace-split.mod-left-split .local-graph-jumps-slider-container, +.workspace-split.mod-right-split .local-graph-jumps-slider-container, +.workspace-fake-target-overlay .local-graph-jumps-slider-container { + background: transparent; + opacity: 0.6; + padding: 0; + left: 12px; + transition: opacity 0.2s linear; + height: auto; +} +.mod-root .local-graph-jumps-slider-container { + right: 0; + left: 0; + width: var(--line-width-adaptive); + max-width: var(--max-width); + margin: 0 auto; + top: 30px; +} +.workspace-split.mod-left-split .local-graph-jumps-slider-container:hover, +.workspace-split.mod-right-split .local-graph-jumps-slider-container:hover, +.workspace-fake-target-overlay .local-graph-jumps-slider-container:hover, +.local-graph-jumps-slider-container:hover { + opacity: 0.8; + transition: opacity 0.2s linear; +} + +/* --------------- */ +/* Settings */ +/* --------------- */ + +.horizontal-tab-content, +.vertical-tab-content { + background: var(--background-primary); + padding-bottom: 100px; +} +.vertical-tab-header, +.vertical-tab-content { + padding-bottom: 100px; +} +.plugin-list-plugins { + overflow: visible; +} +.community-theme-container, +.hotkey-settings-container { + height: auto; + overflow: visible; +} +.modal.mod-settings .vertical-tab-header { + background: var(--background-secondary); + padding-top: 5px; + padding-bottom: 25px; +} +.vertical-tab-header-group-title { + color: var(--text-faint); + font-size: 12px; + letter-spacing: 0.05em; + font-weight: var(--bold-weight); +} +.vertical-tab-nav-item { + padding: 4px 10px 4px 17px; + color: var(--text-muted); + border: none; + background: var(--background-secondary); + cursor: var(--cursor); + font-size: var(--font-small); + line-height: 1.4; +} +.vertical-tab-nav-item:hover, +.vertical-tab-nav-item.is-active { + color: var(--text-normal); +} +.setting-hotkey { + background-color: var(--background-modifier-border); + padding: 3px 10px 3px 10px; +} +.setting-hotkey.mod-empty { + background: transparent; +} +.dropdown { + border-color: var(--background-modifier-border); + background-image: url('data:image/svg+xml;charset=US-ASCII,%3Csvg%20xmlns%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg%22%20width%3D%22292.4%22%20height%3D%22292.4%22%3E%3Cpath%20fill%3D%22%23000%22%20d%3D%22M287%2069.4a17.6%2017.6%200%200%200-13-5.4H18.4c-5%200-9.3%201.8-12.9%205.4A17.6%2017.6%200%200%200%200%2082.2c0%205%201.8%209.3%205.4%2012.9l128%20127.9c3.6%203.6%207.8%205.4%2012.8%205.4s9.2-1.8%2012.8-5.4L287%2095c3.5-3.5%205.4-7.8%205.4-12.8%200-5-1.9-9.2-5.5-12.8z%22%2F%3E%3C%2Fsvg%3E'); +} +.theme-dark .dropdown { + background-image: url('data:image/svg+xml;charset=US-ASCII,%3Csvg%20xmlns%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2Fsvg%22%20width%3D%22292.4%22%20height%3D%22292.4%22%3E%3Cpath%20fill%3D%22%23FFF%22%20d%3D%22M287%2069.4a17.6%2017.6%200%200%200-13-5.4H18.4c-5%200-9.3%201.8-12.9%205.4A17.6%2017.6%200%200%200%200%2082.2c0%205%201.8%209.3%205.4%2012.9l128%20127.9c3.6%203.6%207.8%205.4%2012.8%205.4s9.2-1.8%2012.8-5.4L287%2095c3.5-3.5%205.4-7.8%205.4-12.8%200-5-1.9-9.2-5.5-12.8z%22%2F%3E%3C%2Fsvg%3E'); +} + +/* --------------- */ +/* Publish */ +/* --------------- */ + +.modal.mod-publish { + max-width: 600px; + padding-left: 0; + padding-right: 0; + padding-bottom: 0; +} +.modal.mod-publish .modal-title { + padding-left: 20px; + padding-bottom: 10px; +} +.mod-publish .modal-content { + padding-left: 20px; + padding-right: 20px; +} +.mod-publish p { + font-size: var(--font-small); +} +.mod-publish .button-container, +.modal.mod-publish .modal-button-container { + margin-top: 0px; + padding: 10px; + border-top: 1px solid var(--background-modifier-border); + bottom: 0px; + background-color: var(--background-primary); + position: absolute; + width: 100%; + margin-left: -20px; + text-align: center; +} +.publish-changes-info { + padding: 0 0 15px; + margin-bottom: 0; + border-bottom: 1px solid var(--background-modifier-border); +} +.modal.mod-publish .modal-content .publish-sections-container { + max-height: none; + height: auto; + padding: 10px 20px 30px 0; + margin-top: 10px; + margin-right: -20px; + margin-bottom: 80px; +} +.publish-site-settings-container { + max-height: none; + height: auto; + margin-right: -20px; + margin-bottom: 80px; + overflow-x: hidden; +} +.publish-section-header { + padding-bottom: 15px; + border-width: 1px; +} +.password-item { + padding-left: 0; + padding-right: 0; +} +.publish-section-header-text { + font-weight: 600; + color: var(--text-normal); + cursor: var(--cursor); +} +.publish-section-header-text, +.publish-section-header-toggle-collapsed-button, +.publish-section-header-action, +.file-tree-item-header { + cursor: var(--cursor); +} +.publish-section-header-text:hover, +.publish-section-header-toggle-collapsed-button:hover, +.publish-section-header-action:hover { + color: var(--text-normal); + cursor: var(--cursor); +} +.mod-publish .u-pop { + color: var(--text-normal); +} +.publish-section-header-toggle-collapsed-button { + padding: 7px 0 0 3px; + width: 18px; +} +.mod-publish .file-tree-item { + margin-left: 20px; +} +.mod-publish .file-tree-item { + padding: 0; + margin-bottom: 2px; + font-size: var(--font-small); +} +.mod-publish .file-tree-item-checkbox { + filter: hue-rotate(0); +} +.mod-publish .file-tree-item.mod-deleted .flair, +.mod-publish .file-tree-item.mod-to-delete .flair { + background: transparent; + color: #ff3c00; + font-weight: 500; +} +.mod-publish .file-tree-item.mod-new .flair { + background: transparent; + font-weight: 500; + color: #13c152; +} +.mod-publish .site-list-item { + padding-left: 0; + padding-right: 0; +} + +/* --------------- */ +/* Scroll bars */ +/* --------------- */ + +::-webkit-scrollbar { + width: 7px !important; +} +::-webkit-scrollbar-track { + background-color: var(--background-primary); +} +::-webkit-scrollbar-thumb { + border-width: 0px 4px 6px 0px; + border-style: solid; + border-radius: 0 !important; + border-color: var(--background-primary); + background-color: var(--background-modifier-border); + min-height: 40px; +} +.modal .vertical-tab-header::-webkit-scrollbar-track, +.mod-left-split .workspace-tabs ::-webkit-scrollbar-track { + background-color: var(--background-secondary); +} +.modal .vertical-tab-header::-webkit-scrollbar-track-piece, +.mod-left-split .workspace-tabs ::-webkit-scrollbar-track-piece { + background-color: var(--background-secondary); +} +.modal .vertical-tab-header::-webkit-scrollbar-thumb, +.mod-left-split .workspace-tabs ::-webkit-scrollbar-thumb { + border-color: var(--background-secondary); + background-color: var(--background-modifier-border); +} +.modal .vertical-tab-header::-webkit-scrollbar-thumb:hover, +.mod-left-split .workspace-tabs ::-webkit-scrollbar-thumb:hover, +::-webkit-scrollbar-thumb:hover { + background-color: var(--background-modifier-border-hover); +} +.modal .vertical-tab-header::-webkit-scrollbar-thumb:active, +.mod-left-split .workspace-tabs ::-webkit-scrollbar-thumb:active, +::-webkit-scrollbar-thumb:active { + background-color: var(--background-modifier-border-focus); +} + +/* -------------------------------------------------------------------------------- +Mobile styling +-------------------------------------------------------------------------------- */ + +.is-mobile { + --font-settings-title: 18px; + --font-settings: 16px; + --font-settings-small: 13px; + --input-height: 40px; +} +body.is-mobile { + padding: 0 !important; +} +.hider-tooltips .follow-link-popover { + display: none; +} +.is-mobile .workspace-drawer-tab-container > *, +body.is-mobile .view-header-title, +.is-mobile .allow-fold-headings.markdown-preview-view .markdown-preview-sizer, +.is-mobile .allow-fold-lists.markdown-preview-view .markdown-preview-sizer { + padding: 0; +} +.is-mobile .titlebar { + height: 0 !important; + padding: 0 !important; + position: relative !important; + border-bottom: none; +} +.is-mobile .horizontal-main-container { + background-color: var(--background-primary); +} +.is-mobile .safe-area-top-cover { + background-color: var(--background-primary); +} +.is-mobile .workspace { + border-radius: 0 !important; + transform: none !important; +} +.is-mobile .workspace-drawer:not(.is-pinned) { + width: 100vw; + max-width: 360pt; + border: none; + box-shadow: 0 5px 50px 5px rgba(0, 0, 0, 0.05); +} +.is-mobile .workspace-drawer.mod-left.is-pinned { + max-width: 280pt; +} +.is-mobile .workspace-drawer.mod-right.is-pinned { + max-width: 240pt; +} + +.is-mobile .workspace-drawer.mod-right.is-pinned { + border-right: none; +} +.is-mobile .workspace-leaf-content[data-type='starred'] .item-list { + padding-left: 5px; +} +.is-mobile .workspace-drawer-tab-option-item-title, +.is-mobile .workspace-drawer-active-tab-title { + font-size: var(--font-adaptive-small); +} +.is-mobile + .workspace-drawer-tab-option-item:hover + .workspace-drawer-tab-option-item-title, +.is-mobile + .workspace-drawer-active-tab-header:hover + .workspace-drawer-active-tab-title { + color: var(--text-normal); +} +.is-mobile + .workspace-drawer-active-tab-header:hover + .workspace-drawer-active-tab-back-icon { + color: var(--text-normal); +} +.is-mobile .nav-file-title, +.is-mobile .nav-folder-title, +.is-mobile .outline, +.is-mobile .tree-item-self, +.is-mobile .tag-container, +.is-mobile .tag-pane-tag { + font-size: var(--font-adaptive-small); + line-height: 1.5; + margin-bottom: 4px; +} +.is-mobile .backlink-pane > .tree-item-self, +.is-mobile .outgoing-link-pane > .tree-item-self { + font-size: var(--font-adaptive-smallest); +} +.is-mobile .tree-item-flair { + font-size: var(--font-adaptive-small); +} +.is-mobile .nav-files-container { + padding: 5px 5px 5px 5px; +} +.is-mobile .search-result-container { + padding-bottom: 20px; +} +.is-mobile .search-result-file-match-replace-button { + background-color: var(--background-tertiary); + color: var(--text-normal); +} +.is-mobile .search-result-file-matches, +.is-mobile .search-result-file-title { + font-size: var(--font-adaptive-small); +} + +/* Modal close button */ + +.modal-close-button { + top: 2px; + padding: 0; + cursor: var(--cursor); + font-size: 24px; + color: var(--text-faint); +} +.modal-close-button:hover { + color: var(--text-normal); +} +.modal-close-button:before { + font-family: Inter, sans-serif; + font-weight: 200; +} + +.is-mobile .modal-close-button { + display: block; + z-index: 2; + top: 0; + right: 12px; + padding: 4px; + font-size: 34px; + width: 34px; + height: 34px; +} +.is-mobile .modal-close-button:before { + font-weight: 300; + color: var(--text-muted); +} + +/* Folding on mobile */ + +.is-mobile .empty-state-action { + border-radius: 6px; + font-size: var(--font-adaptive-small); +} +.is-mobile .workspace-drawer-header { + padding: 5px 10px 0 20px; +} +body:not(.is-ios).is-mobile .workspace-drawer-ribbon { + padding: 5px; +} +.is-mobile .workspace-drawer-header-name { + font-weight: var(--bold-weight); + color: var(--text-normal); + font-size: 1.125em; + margin-top: 3px; +} +.is-mobile .workspace-drawer-header-info { + color: var(--text-faint); + font-size: var(--font-adaptive-smaller); + margin-bottom: 0; +} +.is-mobile .mod-left .workspace-drawer-header-info, +.is-mobile.hider-status .workspace-drawer-header-info { + display: none; +} +.is-mobile .workspace-drawer-active-tab-header { + margin: 2px 12px 2px; + padding: 8px 0 8px 8px; +} +.is-mobile .workspace-leaf-content .item-list, +.is-mobile .tag-container, +.is-mobile .backlink-pane { + padding-top: 10px; +} +.is-mobile .outgoing-link-pane, +.is-mobile .backlink-pane { + padding-left: 10px; +} +.workspace-drawer.mod-left .workspace-drawer-inner { + padding-left: 0; +} +.is-mobile .workspace-drawer-ribbon { + background: var(--background-secondary); + border-right: 1px solid var(--background-modifier-border); + z-index: 3; + flex-direction: column; + width: 70px; + padding: 15px 0; + margin-right: 0px; +} +.is-ios .is-pinned .workspace-drawer-ribbon { + padding: 30px 0 20px 0; +} +.is-mobile .side-dock-actions, +.is-mobile .side-dock-settings { + flex-direction: column; + border-radius: 15px; +} +.is-mobile .mod-left .workspace-drawer-header, +.is-mobile .mod-left .workspace-drawer-tab-container { + margin-left: 70px; +} +.is-mobile .workspace-drawer-ribbon .side-dock-ribbon-action { + padding: 9px 5px 2px 5px; + margin: 0 12px 4px; + border-radius: 8px; +} +.is-mobile .workspace-drawer-ribbon .side-dock-ribbon-action svg { + width: 22px; + height: 22px; +} +.is-mobile .workspace-drawer-ribbon .side-dock-ribbon-action:hover { + background-color: var(--background-tertiary); + box-shadow: 0 0 0px 1px var(--background-tertiary); +} +.is-mobile .workspace-drawer-active-tab-container { + z-index: 9999; + background-color: var(--background-primary); +} +.is-mobile .side-dock-actions, +.is-mobile .side-dock-settings { + display: flex; + align-content: center; + justify-content: center; + padding: 0; +} +.is-mobile .workspace-drawer.mod-left:not(.is-pinned) { + border-right: none; +} +.is-mobile .modal.mod-publish, +.is-mobile .modal.mod-community-plugin, +.is-mobile .modal.mod-settings { + width: 100vw; + max-height: 90vh; + padding: 0; +} +.is-mobile .vertical-tab-header-group:last-child, +.is-mobile .vertical-tab-content, +.is-mobile .minimal-donation { + padding-bottom: 70px !important; +} +.is-mobile .modal.mod-settings .vertical-tab-header:before { + content: 'Settings'; + font-weight: 600; + font-size: var(--font-settings); + position: sticky; + display: flex; + height: 54px; + margin-top: 8px; + align-items: center; + justify-content: center; + text-align: center; + border-bottom: 1px solid var(--background-modifier-border); + background: var(--background-primary); + left: 0; + top: 0; + right: 0; + z-index: 1; +} +.is-mobile .modal .vertical-tab-header-group-title { + padding: 15px 20px 10px 20px; + text-transform: uppercase; + letter-spacing: 0.05em; +} +.is-mobile .nav-buttons-container { + padding: 0 0 10px 15px; +} +.is-mobile + .workspace-leaf-content:not([data-type='search']) + .nav-buttons-container { + border-bottom: var(--border-width) solid var(--background-modifier-border); +} +.is-mobile input[type='text'] { + font-size: 14px; + height: var(--input-height); +} +.is-mobile .search-input-container input[type='text'] { + border-radius: 50px; + height: 40px; + padding: 10px 20px; + font-size: 14px; + -webkit-appearance: none; +} +.is-mobile .search-input-clear-button { + right: 15px; +} +.is-mobile .modal, +.is-mobile .prompt, +.is-mobile .suggestion-container { + width: 100%; + max-width: 100%; + padding: 10px; + -webkit-touch-callout: none; + -webkit-user-select: none; + user-select: none; +} +.is-mobile .suggestion-container { + margin: 0 auto; + border: none; + left: 0; + right: 0; +} +.is-mobile .suggestion-item { + font-size: var(--font-adaptive-normal); + padding-left: 10px; + letter-spacing: 0.001px; +} +.is-mobile .prompt-results .suggestion-flair { + display: none; +} +.is-mobile input[type='text'].prompt-input, +.is-mobile input[type='text'].prompt-input:hover { + line-height: 2; + padding: 8px; + font-size: var(--font-adaptive-normal); +} +.is-mobile .search-input-container input::placeholder { + font-size: 14px; +} +.is-mobile .modal-setting-back-button { + padding: 20px; + background-color: var(--color-background); + box-shadow: none; +} +.is-mobile .hotkey-list-container .setting-command-hotkeys { + flex: unset; +} +.is-mobile + .markdown-preview-view + input[type='checkbox'].task-list-item-checkbox { + top: 6px; +} +.is-mobile .workspace-drawer { + border-width: var(--border-width); +} +.is-mobile .workspace-drawer-inner, +.is-mobile .workspace-drawer-active-tab-container { + background-color: var(--background-secondary); +} +.is-mobile .menu { + border: none; + width: 100%; + max-width: 100%; + left: 0 !important; + -webkit-touch-callout: none; + -webkit-user-select: none; + user-select: none; +} +.is-ios .is-pinned .workspace-drawer-ribbon { + padding: 30px 0 20px 0; +} +.is-ios .workspace-drawer.is-pinned .workspace-drawer-header { + padding-top: 26px; +} +.is-mobile .workspace-split.mod-root { + background-color: var(--background-primary); +} +.is-ios .mod-root .workspace-leaf { + padding-top: 20px; +} +.is-ios + .mod-root + .workspace-split.mod-horizontal + .workspace-leaf:not(:first-of-type) { + padding-top: 0; +} +.is-mobile.focus-mode .view-actions { + opacity: 1; +} +.is-mobile .workspace-drawer-header-icon { + align-self: start; +} +.is-mobile .workspace-drawer-header-icon svg { + width: 22px; + height: 100%; +} +.is-mobile .workspace-drawer-tab-options { + padding-top: 10px; +} +.is-mobile .workspace-drawer-tab-option-item { + -webkit-touch-callout: none; + -webkit-user-select: none; + user-select: none; + margin: 0 10px; + padding: 8px 10px; + border-radius: 6px; +} +.is-mobile .nav-action-button svg { + width: 22px; + margin: 0; +} +.is-mobile .menu-item { + padding: 5px 10px; +} +.is-mobile .menu-item-icon { + margin-right: 10px; +} +.is-mobile .menu-item-icon svg { + width: 18px; + height: 18px; +} +.is-mobile .view-header-title { + font-size: 125%; +} +.is-mobile .view-action svg { + width: 22px; +} +.is-mobile .view-action { + padding: 5px 5px 4px; + margin: 0; + border-radius: 8px; +} +.is-mobile .workspace-leaf-content[data-type='search'] .nav-action-button, +.is-mobile .nav-action-button, +.is-mobile .workspace-drawer-header-icon { + padding: 5px 7px 0 !important; + margin: 5px 2px 2px 0; + text-align: center; + border-radius: 8px; + cursor: var(--cursor); +} +.is-mobile .nav-file-title.is-active { + box-shadow: 0 0 0px 3px var(--background-tertiary); +} +.pull-down-action { + top: 0; + left: 0; + right: 0; + width: 100%; + margin: 0 auto; + padding: 50px 0 20px; + text-align: center; + border-radius: 0; + border: none; + box-shadow: 0 5px 200px var(--background-modifier-box-shadow); +} +.is-mobile .menu-item.is-label { + color: var(--text-normal); + font-weight: var(--bold-weight); +} +.is-mobile .menu-item.is-label .menu-item-icon { + display: none; +} +.mobile-toolbar { + width: 100%; + text-align: center; + display: flex; + overflow: scroll; + background-color: var(--background-primary); + border-top: 1px solid var(--background-modifier-border); +} +.is-mobile .modal.mod-settings .vertical-tab-content-container { + border: 0; +} +.is-mobile .modal, +.is-mobile .modal-bg { + transition: none !important; + transform: none !important; +} +.is-mobile .document-search-container { + height: 56px; + padding: 10px 15px; +} +.is-mobile .document-search-container input[type='text'] { + width: auto; + margin: 0 5px 0 0; + height: 32px; + padding: 5px 7px; + border-radius: 6px; + border: 1px solid var(--background-modifier-border); + background-color: var(--background-primary); +} +.is-mobile .document-search-container button { + width: auto; + margin: 0px; + background: transparent; + font-size: 14px; + height: 32px; +} +.is-mobile .modal .vertical-tab-header-group:last-child, +.is-mobile .modal .vertical-tab-content { + padding-bottom: 70px !important; +} +.pull-out-action { + top: 0; + height: 100vh; + padding: 30px 10px; + background: transparent; + display: flex; + justify-content: center; + align-content: center; + flex-direction: column; +} +.is-mobile .markdown-preview-view pre { + overflow-x: scroll; +} + +/* Sync */ + +.is-mobile .sync-history-list { + padding: 10px; + background-color: var(--background-primary); +} +.is-mobile .sync-history-list-item { + font-size: var(--font-adaptive-small); + padding: 8px 10px; +} +.is-mobile .sync-history-content-container .modal-button-container { + padding: 5px 10px 30px 10px; +} +.is-mobile .sync-history-content { + outline: none; + -webkit-appearance: none; + border: 0; + background-color: var(--background-secondary); +} +.is-mobile.show-mobile-hamburger .view-header-icon .three-horizontal-bars { + opacity: 1; +} +.is-mobile.show-mobile-hamburger .view-header .view-header-title-container { + left: 50px; +} +.is-mobile.plugin-sliding-panes .view-header-title { + mask-image: unset; + -webkit-mask-image: unset; +} +.is-mobile.plugin-sliding-panes-rotate-header .view-header-title { + line-height: 1.2; +} +.is-mobile .workspace-drawer-header-name-text { + white-space: nowrap; + margin-right: 10px; +} +.is-mobile .mod-community-theme .modal-title { + padding: 10px 20px; +} +.is-mobile .mod-publish .modal-content { + display: unset; + padding: 10px 10px 10px; + margin-bottom: 120px; + overflow-x: hidden; +} +.is-mobile .mod-publish .button-container, +.is-mobile .modal.mod-publish .modal-button-container { + padding: 10px 15px 30px; + margin-left: 0px; + left: 0; +} +.is-mobile .modal.mod-publish .modal-title { + padding: 10px 20px; + margin: 0 -10px; + border-bottom: 1px solid var(--background-modifier-border); +} +.is-mobile .publish-site-settings-container { + margin-right: 0; + padding: 0; +} +.is-mobile .modal.mod-publish .modal-content .publish-sections-container { + margin-right: 0; + padding-right: 0; +} + +/* --------------- */ +/* Phone styling */ +/* --------------- */ + +@media (max-width: 400pt) { + .is-mobile.show-mobile-hamburger .view-header-icon { + display: block; + } + .is-mobile .suggestion-hotkey { + display: none; + } + .is-mobile .modal, + .is-mobile .menu, + .is-mobile .prompt { + border-radius: 0; + } + .is-mobile .suggestion-flair { + right: 0; + left: auto; + position: absolute; + padding: 5px 5px 0 0; + } + .is-mobile .prompt { + border-radius: 0; + padding-top: 5px; + padding-bottom: 0; + max-height: calc(100vh - 120px); + top: 120px; + } + .is-mobile .suggestion-container { + max-height: 200px; + border-top: 1px solid var(--background-modifier-border); + border-radius: 0; + padding-top: 0; + box-shadow: none; + } + .is-mobile .suggestion-container .suggestion { + padding-top: 10px; + } + .workspace-drawer-header-icon .pin { + display: none; + } + /* + .is-mobile .markdown-source-view .cm-scroller > .cm-content { + margin-top:15px; + } */ + .is-ios .workspace-drawer .workspace-drawer-header { + padding-top: 40px; + } + .is-ios .mod-root .workspace-leaf { + padding-top: 40px; + } + .is-mobile .workspace .workspace-drawer-backdrop { + margin-top: -40px; + height: calc(100vh + 50px); + z-index: 9; + } + .is-mobile .modal .vertical-tab-header-group-title { + padding: 20px 20px 10px; + } + .is-mobile .modal .vertical-tab-nav-item { + padding: 3px 20px; + } + .is-ios .workspace-drawer-ribbon { + padding: 40px 0 20px 0; + } + .is-mobile .view-header-title { + max-width: 80vw; + } + .is-mobile .view-header-title { + padding-right: 20px; + font-size: 18px; + } + .is-mobile .workspace-drawer-header-name-text { + font-size: var(--font-settings-title); + letter-spacing: -0.015em; + } + .is-mobile .menu-item.is-label { + font-size: 18px; + } + .is-mobile .view-header { + border-bottom: var(--border-width) solid var(--background-modifier-border) !important; + } + .is-mobile .modal-setting-back-button { + border-bottom: 1px solid var(--background-modifier-border); + } + .is-mobile .installed-plugins-container { + max-width: 100%; + overflow: hidden; + } + .is-mobile .setting-item-info { + flex: 1 1 auto; + } + .is-mobile .kanban-plugin__board-settings-modal .setting-item-control, + .is-mobile .setting-item-control { + flex: 1 0 auto; + margin-right: 0; + min-width: auto; + } + .is-mobile .checkbox-container { + flex: 1 0 40px; + max-width: 40px; + } + .is-mobile .setting-item-description { + word-break: break-word; + white-space: pre-line; + } + .is-mobile .view-action { + padding: 3px 0 0 4px; + margin-top: -4px; + } + .is-mobile .menu { + padding-bottom: 30px; + } + .is-mobile .frontmatter-container .tag, + .is-mobile .cm-s-obsidian span.cm-hashtag, + .is-mobile .tag { + font-size: var(--font-adaptive-smaller); + } + .is-mobile .setting-item-control select, + .is-mobile .setting-item-control input, + .is-mobile .setting-item-control button { + margin-bottom: 5px; + } + .is-mobile .setting-item-control input[type='range'] { + margin-bottom: 10px; + } + .is-mobile .publish-section-header, + .is-mobile .publish-changes-info { + flex-wrap: wrap; + border: none; + } + .is-mobile .publish-changes-info .publish-changes-add-linked-btn { + flex-basis: 100%; + margin-top: 10px; + } + .is-mobile .publish-section-header-text { + flex-basis: 100%; + margin-bottom: 10px; + margin-left: 20px; + margin-top: -8px; + } + .is-mobile .publish-section { + background: var(--background-secondary); + border-radius: 10px; + padding: 12px 12px 1px; + } + .is-mobile .publish-changes-switch-site { + flex-grow: 0; + margin-right: 10px; + } +} + +/* ---------------- */ +/* Mobile toolbar button */ +/* ---------------- */ + +body.is-mobile:not(.floating-button-off):not(.advanced-toolbar) + .view-action:nth-last-of-type(5), +body.is-mobile:not(.floating-button-off):not(.advanced-toolbar) + .view-action:nth-last-of-type(4) { + color: white; + background-color: var(--blue); + opacity: 1; + top: calc(100vh - 90px); + display: flex; + padding: 5px; + position: fixed; + left: 87vw; + transform: translate(-40%, -18%); + justify-content: center; + align-items: center; + width: 53px; + height: 53px; + border-radius: 50% !important; + box-shadow: 0.9px 0.9px 3.6px rgba(0, 0, 0, 0.07), + 2.5px 2.4px 10px rgba(0, 0, 0, 0.1), 6px 5.7px 24.1px rgba(0, 0, 0, 0.13), + 20px 19px 80px rgba(0, 0, 0, 0.2); +} + +body.is-mobile:not(.floating-button-off).advanced-toolbar + .view-action:nth-last-of-type(5), +body.is-mobile:not(.floating-button-off).advanced-toolbar + .view-action:nth-last-of-type(4) { + color: white; + background-color: var(--blue); + opacity: 1; + position: fixed; + top: calc(100vh - 138px); + display: flex; + padding: 5px; + left: 87vw; + transform: translate(-40%, -18%); + justify-content: center; + align-items: center; + width: 53px; + height: 53px; + border-radius: 50% !important; + box-shadow: 0.9px 0.9px 3.6px rgba(0, 0, 0, 0.07), + 2.5px 2.4px 10px rgba(0, 0, 0, 0.1), 6px 5.7px 24.1px rgba(0, 0, 0, 0.13), + 20px 19px 80px rgba(0, 0, 0, 0.2); +} + +/* --------------- */ +/* Tablet styling */ +/* --------------- */ + +@media (min-width: 400pt) { + .mobile-toolbar-option { + border-radius: 8px; + margin: 6px 0; + } + .mobile-toolbar-option:hover { + background-color: var(--background-tertiary); + } + + .is-mobile.is-ios .safe-area-top-cover { + background-color: transparent; + } + .is-mobile .modal, + .is-mobile .modal-container .modal.mod-settings { + max-width: 800px; + transform: translateZ(0); + border-top-left-radius: 20px !important; + border-top-right-radius: 20px !important; + margin-bottom: -15px; + overflow: hidden; + } + .is-mobile .modal-container .modal.mod-settings .vertical-tabs-container { + transform: translateZ(0); + } + .is-mobile .view-action { + padding: 5px 5px 4px; + border-radius: 8px; + } + .is-mobile .view-action:hover, + .is-mobile .nav-action-button:hover, + .is-mobile + .workspace-leaf-content[data-type='search'] + .nav-action-button.is-active:hover, + .is-mobile + .workspace-leaf-content[data-type='backlink'] + .nav-action-button.is-active:hover, + .is-mobile .workspace-drawer-tab-option-item:hover, + .is-mobile .workspace-drawer-header-icon:hover { + background-color: var(--background-tertiary); + box-shadow: 0 0 0 2px var(--background-tertiary); + } + .is-mobile .prompt { + max-width: 600px; + max-height: 600px; + bottom: auto !important; + border-radius: 20px; + top: 100px !important; + } + .is-mobile .suggestion-container { + max-width: 600px; + max-height: 600px; + border-radius: 20px; + bottom: 80px; + border: 1px solid var(--background-modifier-border); + } + .is-mobile .modal-container .suggestion-item { + padding: 10px 5px 10px 10px; + border-radius: 8px; + } + .is-mobile .suggestion-flair { + right: 0; + left: auto; + position: absolute; + padding: 10px; + } + .is-mobile .menu { + top: 60px !important; + right: 0 !important; + bottom: auto; + left: auto; + margin: 0 auto; + width: 360px; + padding: 10px 10px 20px; + border-radius: 15px; + box-shadow: 0 0 100vh 100vh rgba(0, 0, 0, 0.5); + } + /* Animations */ + .is-mobile .menu, + .is-mobile .suggestion-container, + .is-mobile .modal, + .is-mobile .prompt { + transition: unset !important; + transform: unset !important; + animation: unset !important; + } + .is-mobile .modal-container .modal-bg { + opacity: 0.8 !important; + } + .is-mobile .modal-container .prompt { + opacity: 1 !important; + } + .is-mobile .menu .menu-item:hover { + background-color: var(--background-tertiary); + } + .is-mobile .setting-item:not(.mod-toggle):not(.setting-item-heading) { + flex-direction: row; + align-items: center; + } + .is-mobile .setting-item-control select, + .is-mobile .setting-item-control input, + .is-mobile .setting-item-control button { + width: auto; + } + .is-mobile .workspace-drawer:not(.is-pinned) { + margin: 30px 16px 0; + height: calc(100vh - 48px); + border-radius: 15px; + } + .is-mobile + .setting-item:not(.mod-toggle):not(.setting-item-heading) + .setting-item-control { + width: auto; + margin-top: 0; + } + .is-mobile .modal .search-input-container input { + width: 100%; + } + .pull-down-action { + width: 400px; + top: 15px; + padding: 15px; + border-radius: 15px; + } +} + +/*---------------------------------------------------------------- +PLUGINS +----------------------------------------------------------------*/ + +/* --------------- */ +/* Sliding Panes */ +/* --------------- */ + +body.plugin-sliding-panes-rotate-header { + --header-width: 40px; +} +body.plugin-sliding-panes-rotate-header .view-header-title:before { + display: none; +} +body.plugin-sliding-panes .workspace-split.mod-root { + background-color: var(--background-primary); +} +body.plugin-sliding-panes .mod-horizontal .workspace-leaf { + box-shadow: none !important; +} +body.plugin-sliding-panes:not(.is-fullscreen) + .workspace-split.is-collapsed + ~ .workspace-split.mod-root + .view-header { + transition: padding 0.1s ease; +} +body.plugin-sliding-panes .view-header-title:before { + background: 0 0; +} +body.plugin-sliding-panes .view-header { + background: 0 0; +} +body.plugin-sliding-panes.plugin-sliding-panes-rotate-header + .workspace + > .mod-root + > .workspace-leaf.mod-active + > .workspace-leaf-content + > .view-header { + border: none; +} +body.plugin-sliding-panes.plugin-sliding-panes-rotate-header + .workspace + > .mod-root + > .workspace-leaf + > .workspace-leaf-content + > .view-header { + border: none; + text-orientation: sideways; +} +body.plugin-sliding-panes.plugin-sliding-panes-rotate-header + .workspace + > .mod-root + > .workspace-leaf + > .workspace-leaf-content + > .view-header + .view-header-icon { + padding: 4px 1px; + margin: 5px 0 0 0; + left: 0; +} +body.plugin-sliding-panes.plugin-sliding-panes-rotate-header + .workspace + > .mod-root + > .workspace-leaf + > .workspace-leaf-content + > .view-header + .view-actions { + padding-bottom: 33px; + margin-left: 0; + height: auto; +} +body.plugin-sliding-panes.plugin-sliding-panes-rotate-header + .workspace + > .mod-root + > .workspace-leaf + > .workspace-leaf-content + > .view-header + .view-action { + margin: 3px 0; + padding: 4px 1px; +} +body.plugin-sliding-panes.plugin-sliding-panes-rotate-header + .app-container + .workspace + > .mod-root + > .workspace-leaf.mod-active + > .workspace-leaf-content + > .view-header + > .view-header-title-container:before, +body.plugin-sliding-panes.plugin-sliding-panes-rotate-header + .workspace + > .mod-root + > .workspace-leaf + > .workspace-leaf-content + > .view-header + > .view-header-title-container:before { + background: 0 0 !important; +} +.workspace + > .mod-root + .view-header-title-container + body.plugin-sliding-panes.plugin-sliding-panes-rotate-header.plugin-sliding-panes-header-alt + .workspace + > .mod-root + .view-header-title { + margin-top: 0; +} +body.plugin-sliding-panes.plugin-sliding-panes-rotate-header + .workspace + > .mod-root + .view-header-title-container { + margin-left: 0; + padding-top: 0; +} +body.plugin-sliding-panes.plugin-sliding-panes-rotate-header + .view-header-title-container { + position: static; +} +body.plugin-sliding-panes.plugin-sliding-panes-rotate-header + .app-container + .workspace + > .mod-root + > .workspace-leaf + > .workspace-leaf-content + > .view-header + > div { + margin-left: 0; + bottom: 0; +} +body.plugin-sliding-panes.plugin-sliding-panes-rotate-header .view-header-icon { + opacity: var(--icon-muted); +} +body.plugin-sliding-panes.plugin-sliding-panes-rotate-header + .view-header-icon:hover { + opacity: 1; +} +body.plugin-sliding-panes .workspace-split.mod-vertical > .workspace-leaf, +body.plugin-sliding-panes-stacking .workspace > .mod-root > .workspace-leaf { + box-shadow: 0 0 0 1px var(--background-modifier-border), + 1px 0 15px 0 var(--shadow-color) !important; +} +body.is-mobile.plugin-sliding-panes.plugin-sliding-panes-rotate-header + .workspace + > .mod-root + > .workspace-leaf + > .workspace-leaf-content + > .view-header + .view-header-icon { + height: 30px; +} +body.hider-ribbon.plugin-sliding-panes.plugin-sliding-panes-rotate-header + .workspace + > .mod-root + > .workspace-leaf + > .workspace-leaf-content + > .view-header + .view-actions { + padding-bottom: 50px; +} +body.plugin-sliding-panes.is-fullscreen .view-header-icon { + padding-top: 8px; +} +body.plugin-sliding-panes .mod-root .graph-controls { + top: 20px; + left: 30px; +} + +/* --------------- */ +/* Hider */ +/* --------------- */ + +.hider-ribbon:not(.is-mobile) .workspace-ribbon-collapse-btn { + display: none; +} +.hider-ribbon:not(.is-mobile) .workspace-ribbon.mod-right { + pointer-events: none; +} +.hider-ribbon:not(.is-mobile) .workspace-ribbon.mod-left { + position: absolute; + border-right: 0px; + margin: 0; + height: var(--header-height); + overflow: visible; + flex-basis: 0; + bottom: 0; + top: auto; + display: flex !important; + flex-direction: row; + z-index: 17; + opacity: 0; + transition: opacity 0.25s ease-in-out; + filter: drop-shadow(2px 10px 30px rgba(0, 0, 0, 0.2)); +} +.hider-ribbon:not(.is-mobile) .side-dock-actions, +.hider-ribbon:not(.is-mobile) .side-dock-settings { + display: flex; + border-top: var(--border-width) solid var(--background-modifier-border); + background: var(--background-secondary); + margin: 0; + position: relative; +} +.hider-ribbon:not(.is-mobile) .side-dock-actions { + padding-left: 5px; +} +.hider-ribbon:not(.is-mobile) .side-dock-settings { + border-right: var(--border-width) solid var(--background-modifier-border); + border-top-right-radius: 5px; + padding-right: 10px; +} +.hider-ribbon:not(.is-mobile) + .workspace-ribbon.mod-left + .side-dock-ribbon-action { + display: flex; + padding: 4px; + margin: 6px 0px 5px 10px; +} +.hider-ribbon:not(.is-mobile) .workspace-ribbon.mod-left:hover { + opacity: 1; + transition: opacity 0.25s ease-in-out; +} +.hider-ribbon:not(.is-mobile) + .workspace-ribbon.mod-left + .workspace-ribbon-collapse-btn { + border-top: 1px solid var(--background-modifier-border); +} +.hider-ribbon:not(.is-mobile) .workspace-split.mod-left-split { + margin: 0; +} +.hider-ribbon:not(.is-mobile) .workspace-leaf-content .item-list { + padding-bottom: 40px; +} +.hider-ribbon .workspace-ribbon { + padding: 0; +} + +/* --------------- */ +/* View Headers & Actions */ +/* --------------- */ + +.view-header { + align-items: center; +} +.view-actions { + margin-right: 0px; + margin-left: auto; + transition: opacity 0.25s ease-in-out; +} +.view-actions .view-action { + margin-right: 8px; +} +.view-action.is-active { + color: var(--text-faint); + opacity: 1; +} +.view-actions .view-action:last-child { + margin-left: 2px; +} + +/* Frameless mode on macOS only */ + +.hider-frameless:not(.is-mobile) + .workspace-split.mod-right-split + > .workspace-tabs, +.hider-frameless:not(.is-mobile) .workspace-split.mod-root .view-header { + padding-top: 2px; +} +.hider-frameless:not(.is-mobile) + .workspace-split.mod-left-split + > .workspace-tabs { + padding-top: 24px; +} +.hider-frameless:not(.is-mobile) + .workspace-split.mod-right-split + > .workspace-tabs + ~ .workspace-tabs, +.hider-frameless:not(.is-mobile) + .workspace-split.mod-left-split + > .workspace-tabs + ~ .workspace-tabs { + padding-top: 0px; +} +.hider-frameless.is-fullscreen:not(.is-mobile) + .workspace-split.mod-left-split + > .workspace-tabs, +.hider-frameless.is-fullscreen:not(.is-mobile) + .workspace-split.mod-root + .view-header { + padding-top: 0px; +} + +/* Title bar / traffic light icons */ +/* TODO: fix for Live Preview */ +.mod-macos.hider-frameless.hider-ribbon:not(.plugin-sliding-panes-rotate-header) { + --traffic-space: 80px; + --traffic-padding: 60px; +} +.mod-macos.hider-frameless:not(.plugin-sliding-panes-rotate-header) { + --traffic-space: 55px; + --traffic-padding: 20px; +} +.mod-macos.hider-frameless.hider-ribbon:not(.plugin-sliding-panes-rotate-header) { + --traffic-space: 95px; + --traffic-padding: 60px; +} +.mod-macos.hider-frameless:not(.plugin-sliding-panes-rotate-header) { + --traffic-space: 65px; + --traffic-padding: 20px; +} +.mod-macos.hider-frameless:not(.is-fullscreen):not(.plugin-sliding-panes-rotate-header) + .workspace-split.mod-left-split.is-collapsed + + .mod-root + .workspace-leaf:first-of-type + .workspace-leaf-content:not([data-type='graph']) + .view-header-icon { + margin-left: var(--traffic-padding); +} + +body:not(.plugin-sliding-panes-rotate-header) + .app-container + .workspace-split.mod-root + > .workspace-leaf + .view-header { + transition: height linear 0.1s; +} + +:root { + --traffic-x-space: 0px; +} +.mod-macos.hider-frameless:not(.is-fullscreen):not(.plugin-sliding-panes-rotate-header) + .workspace-split.mod-left-split.is-collapsed + + .mod-root + .workspace-leaf:first-of-type + .view-header-title-container { + max-width: calc(100% - (var(--traffic-x-space) * 2) - 30px); +} +.mod-macos.is-popout-window.hider-frameless:not(.is-fullscreen):not(.plugin-sliding-panes-rotate-header) + .mod-root + .workspace-leaf:first-of-type + .view-header-title-container { + max-width: calc(100% - (var(--traffic-x-space) * 2) - 30px); +} +.mod-macos.hider-ribbon.hider-frameless:not(.is-fullscreen):not(.plugin-sliding-panes-rotate-header) + .workspace-split.mod-left-split.is-collapsed + + .mod-root + .workspace-leaf:first-of-type { + --traffic-x-space: 64px; +} +.mod-macos.is-popout-window.hider-ribbon.hider-frameless:not(.is-fullscreen):not(.plugin-sliding-panes-rotate-header) + .mod-root + .workspace-leaf:first-of-type { + --traffic-x-space: 64px; +} +.mod-macos.hider-frameless:not(.is-fullscreen):not(.plugin-sliding-panes-rotate-header) + .workspace-split.mod-left-split.is-collapsed + + .mod-root + .workspace-leaf:first-of-type { + --traffic-x-space: 22px; +} +.mod-macos.hider-frameless .workspace-ribbon { + border: none; +} + +/* --------------- */ +/* Calendar */ +/* --------------- */ + +.workspace-leaf-content[data-type='calendar'] .view-content { + padding: 5px 0 0 0; +} +#calendar-container { + padding: 5px 15px; + --color-background-day-empty: var(--background-secondary-alt); + --color-background-day-active: var(--background-tertiary); + --color-background-day-hover: var(--background-tertiary); + --color-dot: var(--text-faint); + --color-text-title: var(--text-normal); + --color-text-heading: var(--text-muted); + --color-text-day: var(--text-normal); + --color-text-today: var(--text-normal); + --color-arrow: var(--text-faint); + --color-background-day-empty: transparent; +} +#calendar-container .table { + border-collapse: separate; + table-layout: fixed; +} +#calendar-container h2 { + font-size: var(--h2); + font-weight: 400; +} +.mod-root #calendar-container { + width: var(--line-width-adaptive); + max-width: var(--max-width); + margin: 0 auto; + padding: 0; +} +#calendar-container h2 .arrow { + color: var(--text-faint); + cursor: var(--cursor); +} +#calendar-container .arrow:hover { + fill: var(--text-muted); + color: var(--text-muted); +} +#calendar-container tr th { + padding: 2px 0; + font-weight: 500; +} +#calendar-container tr td { + padding: 2px 0 0; + border-radius: 4px; + cursor: var(--cursor); + border: 2px solid transparent; + transition: none; +} +#calendar-container .nav { + padding: 0; + margin: 10px 5px 10px 5px; +} +#calendar-container .dot { + margin: 0; +} +#calendar-container .arrow { + cursor: var(--cursor); +} +#calendar-container .arrow:hover svg { + color: var(--text-muted); +} +#calendar-container .reset-button { + font-size: var(--font-adaptive-smaller); +} +#calendar-container .reset-button:hover { + color: var(--text-normal); +} +#calendar-container .title { + font-size: var(--h1); +} + +#calendar-container .month, +#calendar-container .title { + font-size: var(--font-adaptive-normal); + font-weight: 600; +} +#calendar-container .today { + color: var(--text-accent); + font-weight: 600; +} +#calendar-container .today .dot { + fill: var(--text-accent); +} +#calendar-container .active .task { + stroke: var(--text-faint); +} +#calendar-container .active { + color: var(--text-normal); +} + +#calendar-container .reset-button, +#calendar-container .day { + cursor: var(--cursor); +} +#calendar-container .active, +#calendar-container .active.today, +#calendar-container .week-num:hover, +#calendar-container .day:hover { + background-color: var(--color-background-day-active); +} +#calendar-container .active .dot { + fill: var(--text-faint); +} +#calendar-container .active .task { + stroke: var(--text-faint); +} +#calendar-container .year { + color: var(--text-normal); +} + +/* --------------- */ +/* Kanban */ +/* --------------- */ + +body .kanban-plugin__markdown-preview-view { + font-family: var(--text); +} + +body .workspace-leaf-content[data-type='kanban'] .view-header-title-container { + text-align: center; +} +body .kanban-plugin { + --interactive-accent: var(--text-selection); + --interactive-accent-hover: var(--background-tertiary); + --text-on-accent: var(--text-normal); + background-color: var(--background-primary); +} +body .kanban-plugin__board > div { + margin: 0 auto; +} +body .kanban-plugin__checkbox-label { + font-size: var(--font-adaptive-small); + color: var(--text-muted); +} +body .kanban-plugin__item-markdown ul { + margin: 0; +} +body .kanban-plugin__item-content-wrapper { + box-shadow: none; +} +body .kanban-plugin__grow-wrap > textarea, +body .kanban-plugin__grow-wrap::after { + padding: 0; + border: 0; +} +body .kanban-plugin__grow-wrap > textarea, +body .kanban-plugin__grow-wrap::after, +body .kanban-plugin__item-title p { + font-size: calc(var(--preview-font-size) - 2px); +} +body:not(.is-mobile) .kanban-plugin__grow-wrap > textarea:focus { + box-shadow: none; +} +.kanban-plugin__item-input-actions button, +.kanban-plugin__lane-input-actions button { + font-size: var(--font-adaptive-small); +} +body .kanban-plugin__item { + background-color: var(--background-primary); +} +body .kanban-plugin__lane-header-wrapper { + border-bottom: 0; +} +body .kanban-plugin__lane-header-wrapper .kanban-plugin__grow-wrap > textarea, +body .kanban-plugin__lane-input-wrapper .kanban-plugin__grow-wrap > textarea { + background: transparent; + color: var(--text-normal); + font-size: 0.875rem; + font-weight: 600; +} +body .kanban-plugin__item-input-wrapper { + border: 0; +} +body .kanban-plugin__item-input-wrapper .kanban-plugin__grow-wrap > textarea { + padding: 6px 8px; + border: 1px solid var(--background-modifier-border); +} +body .kanban-plugin__lane button.kanban-plugin__lane-settings-button.is-enabled, +body .kanban-plugin__item .kanban-plugin__item-edit-archive-button, +body .kanban-plugin__item button.kanban-plugin__item-edit-button, +body .kanban-plugin__lane button.kanban-plugin__lane-settings-button, +.kanban-plugin__item-settings-actions > button, +.kanban-plugin__lane-action-wrapper > button { + background: transparent; + transition: color 0.1s ease-in-out; +} +body .kanban-plugin__item .kanban-plugin__item-edit-archive-button:hover, +body .kanban-plugin__item button.kanban-plugin__item-edit-button.is-enabled, +body .kanban-plugin__item button.kanban-plugin__item-edit-button:hover, +body .kanban-plugin__lane button.kanban-plugin__lane-settings-button.is-enabled, +body .kanban-plugin__lane button.kanban-plugin__lane-settings-button:hover { + color: var(--text-normal); + transition: color 0.1s ease-in-out; + background: transparent; +} +body .kanban-plugin__new-lane-button-wrapper { + position: fixed; + bottom: 30px; +} +body .kanban-plugin button { + box-shadow: none; + cursor: var(--cursor); +} +body .kanban-plugin__item-button-wrapper > button { + font-size: var(--font-adaptive-small); + color: var(--text-muted); + background: transparent; +} +body .kanban-plugin__item-button-wrapper > button:hover { + color: var(--text-normal); + background: var(--background-tertiary); +} +body .kanban-plugin__item-button-wrapper { + padding-top: 5px; + border-top: none; +} + +body .kanban-plugin__lane-setting-wrapper > div:last-child { + border: none; + margin: 0; +} + +body .kanban-plugin__item.is-dragging { + box-shadow: 0 5px 30px rgba(0, 0, 0, 0.15), 0 0 0 2px var(--text-selection); +} +body .kanban-plugin__lane.is-dragging { + box-shadow: 0 5px 30px rgba(0, 0, 0, 0.15); + border: 1px solid var(--background-modifier-border); +} + +body .kanban-plugin__lane { + background: var(--background-secondary); + padding: 0; + border-radius: 8px; + border: 1px solid transparent; +} +body .kanban-plugin__lane-items { + padding-bottom: 0; + margin: 0; + background-color: var(--background-secondary); +} + +body + .kanban-plugin__markdown-preview-view + ol.contains-task-list + .contains-task-list, +body + .kanban-plugin__markdown-preview-view + ul.contains-task-list + .contains-task-list, +body .kanban-plugin__markdown-preview-view ul, +.kanban-plugin__markdown-preview-view ol { + padding-inline-start: 24px !important; +} + +@media (max-width: 400pt) { + .kanban-plugin__board { + flex-direction: column !important; + } + + .kanban-plugin__lane { + width: 100% !important; + margin-bottom: 1rem !important; + } +} + +/* --------------- */ +/* Todoist */ +/* --------------- */ + +.todoist-query-title { + display: inline !important; +} +.todoist-refresh-spin { + animation: spin 1s linear infinite; +} +.todoist-refresh-button { + display: inline; + float: right; + margin-left: 8px; + padding: 3px 10px; +} +.todoist-refresh-button:hover { + background-color: var(--background-tertiary); +} +@-webkit-keyframes spin { + 100% { + -webkit-transform: rotate(360deg); + } +} + +/* READER VIEW */ + +.markdown-preview-view + ul + > li.task-list-item + .todoist-p1 + > input[type='checkbox'] { + border: 1px solid #ff757f !important; + background-color: rgba(255, 117, 127, 0.25) !important; +} +.markdown-preview-view + ul + > li.task-list-item + .todoist-p1 + > input[type='checkbox']:hover { + background-color: rgba(255, 117, 127, 0.5) !important; +} +.markdown-preview-view + ul + > li.task-list-item + .todoist-p2 + > input[type='checkbox'] { + border: 1px solid #ffc777 !important; + background-color: rgba(255, 199, 119, 0.25) !important; +} +.markdown-preview-view + ul + > li.task-list-item + .todoist-p2 + > input[type='checkbox']:hover { + background-color: rgba(255, 199, 119, 0.5) !important; +} +.markdown-preview-view + ul + > li.task-list-item + .todoist-p3 + > input[type='checkbox'] { + border: 1px solid #65bcff !important; + background-color: rgba(101, 188, 255, 0.25) !important; +} +.markdown-preview-view + ul + > li.task-list-item + .todoist-p3 + > input[type='checkbox']:hover { + background-color: rgba(101, 188, 255, 0.5) !important; +} +.markdown-preview-view + ul + > li.task-list-item + .todoist-p4 + > input[type='checkbox'] { + border: 1px solid #b4c2f0 !important; + background-color: rgba(180, 194, 240, 0.25) !important; +} +.markdown-preview-view + ul + > li.task-list-item + .todoist-p4 + > input[type='checkbox']:hover { + background-color: rgba(180, 194, 240, 0.5) !important; +} + +/* LIVE PREVIEW */ + +.is-live-preview ul > li.task-list-item .todoist-p1 > input[type='checkbox'] { + border: 1px solid #ff75c6 !important; + background-color: rgba(255, 117, 221, 0.25) !important; +} +.is-live-preview + ul + > li.task-list-item + .todoist-p1 + > input[type='checkbox']:hover { + background-color: rgba(255, 117, 193, 0.5) !important; +} +.is-live-preview ul > li.task-list-item .todoist-p2 > input[type='checkbox'] { + border: 1px solid #ffa3a3 !important; + background-color: rgba(255, 139, 119, 0.25) !important; +} +.is-live-preview + ul + > li.task-list-item + .todoist-p2 + > input[type='checkbox']:hover { + background-color: rgba(255, 154, 154, 0.5) !important; +} +.is-live-preview ul > li.task-list-item .todoist-p3 > input[type='checkbox'] { + border: 1px solid #35bfff !important; + background-color: rgba(67, 233, 255, 0.308) !important; +} +.is-live-preview + ul + > li.task-list-item + .todoist-p3 + > input[type='checkbox']:hover { + background-color: rgba(53, 223, 253, 0.5) !important; +} +.is-live-preview ul > li.task-list-item .todoist-p4 > input[type='checkbox'] { + border: 1px solid #89c6ffd5 !important; + background-color: rgba(150, 170, 179, 0.192) !important; +} +.is-live-preview + ul + > li.task-list-item + .todoist-p4 + > input[type='checkbox']:hover { + background-color: rgba(166, 182, 194, 0.418) !important; +} + +.task-metadata { + font-size: var(--font-todoist-metadata-size); + color: #7a88cf; + margin-left: unset !important; +} +.task-metadata > * { + margin-right: 30px; +} +.task-date.task-overdue { + color: rgba(255, 152, 164, 0.75) !important; +} +.task-calendar-icon, +.task-project-icon, +.task-labels-icon { + vertical-align: middle; + height: 17px; + width: 17px; +} +.todoist-project .todoist-project { + margin-left: 20px; +} +.todoist-section { + margin-left: 20px; +} +.todoist-project .todoist-project-title { + font-weight: 700; + margin-block-end: 0px; +} +.todoist-section .todoist-section-title { + font-size: var(--font-todoist-title-size); + color: #7a88cf; + font-weight: 700; + margin-block-end: 0px; +} +.todoist-error { + border: 1px solid #ff98a4; + background-color: rgba(255, 152, 164, 0.05); + padding: 1em 1em; + margin: 1em 0px; +} +.todoist-error p { + margin: 0 0 1em 0; + font-weight: 600; +} +.todoist-error code { + background-color: unset !important; + padding: unset !important; + margin: unset !important; +} +.todoist-success { + border: 1px solid #c3e88d !important; + background-color: rgba(195, 232, 141, 0.05); + padding: 1em 1em !important; + margin: 1em 0px; +} +.todoist-success p { + margin: 0; + font-weight: 600; +} +.priority-container .priority-1 { + color: #ff98a4; +} +.priority-container .priority-2 { + color: #ffc777; +} +.priority-container .priority-3 { + color: #65bcff; +} +.priority-container .priority-4 { + color: #b4c2f0; +} + +/* --------------- */ +/* Checklist */ +/* --------------- */ + +.checklist-plugin-main .group .classic, +.checklist-plugin-main .group .compact, +.checklist-plugin-main .group svg, +.checklist-plugin-main .group .page { + cursor: var(--cursor); +} +.workspace .view-content .checklist-plugin-main { + padding: 10px 10px 15px 15px; + --todoList-togglePadding--compact: 2px; + --todoList-listItemMargin--compact: 2px; +} +.checklist-plugin-main .title { + font-weight: 400; + color: var(--text-muted); + font-size: var(--font-adaptive-small); +} +.checklist-plugin-main .group svg { + fill: var(--text-faint); +} +.checklist-plugin-main .group svg:hover { + fill: var(--text-normal); +} +.checklist-plugin-main .group .title:hover { + color: var(--text-normal); +} +.checklist-plugin-main .group:not(:last-child) { + border-bottom: 1px solid var(--background-modifier-border); +} +.checklist-plugin-main .group { + padding: 0 0 4px 0; +} +.checklist-plugin-main .group .classic:last-child, +.checklist-plugin-main .group .compact:last-child { + margin-bottom: 10px; +} +.checklist-plugin-main .group .classic, +.checklist-plugin-main .group .compact { + font-size: var(--font-adaptive-small) !important; +} +.checklist-plugin-main .content { + font-size: var(--font-adaptive-small) !important; +} +.checklist-plugin-main .group .classic, +.checklist-plugin-main .group .compact { + background: transparent; + border-radius: 0; + margin: 1px auto; + padding: 0; +} +.checklist-plugin-main .group .classic .content { + padding: 0; +} +.checklist-plugin-main .group .classic:hover, +.checklist-plugin-main .group .compact:hover { + background: transparent; +} +.markdown-preview-view.checklist-plugin-main + ul + > li:not(.task-list-item)::before { + display: none; +} +.checklist-plugin-main .group .compact > .toggle .checked { + background: var(--text-accent); + top: -1px; + left: -1px; + height: 18px; + width: 18px; +} +.checklist-plugin-main .compact .toggle:hover { + opacity: 1 !important; +} +.checklist-plugin-main .group .count { + font-size: var(--font-adaptive-smaller); + background: transparent; + font-weight: 400; + color: var(--text-faint); +} +.checklist-plugin-main .group .group-header:hover .count { + color: var(--text-muted); +} +.checklist-plugin-main .group .checkbox { + border: 2px solid var(--background-modifier-border-focus); + min-height: 18px; + min-width: 18px; + height: 18px; + width: 18px; + border-radius: 30%; +} + +.checklist-plugin-main .group .checkbox:hover { + border: 2px solid var(--background-modifier-border-focus); +} + +.checklist-plugin-main .toggle:hover { + box-shadow: none; +} + +.checklist-plugin-main .container .search { + font-size: var(--font-adaptive-small) !important; + border: 1px solid var(--background-modifier-border) !important; +} + +.checklist-plugin-main .container .settings-container > svg { + width: 100%; +} + +.checklist-plugin-main .checkbox .checked { + border-radius: 30% !important; + background-color: var(--background-modifier-border-focus) !important; + top: calc( + calc(var(--checklist-checkboxSize) - var(--checklist-checkboxCheckedSize)) / + 6 + ); + left: calc( + calc(var(--checklist-checkboxSize) - var(--checklist-checkboxCheckedSize)) / + 6 + ); +} + +/* Checklist mobile styling */ + +.is-mobile .checklist-plugin-main .group-header { + display: flex; + margin-bottom: 12px; +} +.is-mobile .checklist-plugin-main .group-header .title { + font-weight: 500; + color: var(--text-muted); + font-size: var(--font-adaptive-small); +} +.is-mobile .checklist-plugin-main .group-header button { + width: fit-content !important; + margin-left: 5px; +} +.is-mobile .checklist-plugin-main .group .classic { + display: flex; + align-items: center; + padding: 5px 0; +} +.is-mobile .checklist-plugin-main .group .classic .content { + padding: 0; + display: inline-block; +} +.is-mobile .checklist-plugin-main .group .classic .toggle { + padding: 0; + margin-right: 1rem; + width: fit-content !important; + display: inline-block; +} + +/* --------------- */ +/* Dataview */ +/* --------------- */ + +.markdown-preview-view .table-view-table { + font-size: calc(var(--font-adaptive-normal) - 1px); +} +.markdown-preview-view .table-view-table > thead > tr > th { + font-weight: 600; + font-size: calc(var(--font-adaptive-normal) - 1px); + color: var(--text-normal); + border-bottom: 1px solid var(--text-faint); + cursor: var(--cursor); + font-family: var(--font-monospace); +} + +/* --------------- */ +/* Day Planner */ +/* --------------- */ + +.plugin-obsidian-day-planner { + display: flex !important; + align-items: center; +} +.day-planner { + position: relative; + display: flex; + align-items: center; +} + +/* --------------- */ +/* Style Settings */ +/* --------------- */ + +.setting-item-heading.style-settings-heading, +.style-settings-container .style-settings-heading { + cursor: var(--cursor); +} +.modal.mod-settings .setting-item .pickr button.pcr-button { + box-shadow: none; + border-radius: 40px; + height: 24px; + width: 24px; +} +.setting-item .pickr .pcr-button:after, +.setting-item .pickr .pcr-button:before { + border-radius: 40px; + box-shadow: none; + border: none; +} + +/* --------------- */ +/* MacOs-like Translucency */ +/* --------------- */ + +.is-translucent:not(.macOS-translucent).theme-light { + --opacity-translucency: 0.6; +} + +.is-translucent:not(.macOS-translucent).theme-dark { + --opacity-translucency: 0.7; +} + +.is-translucent .workspace-leaf-resize-handle { + opacity: var(--opacity-translucency); + background-color: transparent; +} + +.macOS-translucent.is-translucent.is-translucent ::-webkit-scrollbar { + display: none; +} + +.macOS-translucent.is-translucent .titlebar, +.macOS-translucent.is-translucent .status-bar { + background-color: var(--background-translucent) !important; +} + +.macOS-translucent.is-translucent .titlebar-button:hover { + background-color: var(--background-primary); +} + +.macOS-translucent.is-translucent .workspace { + background-color: var(--background-translucent) !important; +} + +.macOS-translucent.is-translucent .workspace-split .workspace-tabs { + background: var(--background-primary) !important; +} + +.macOS-translucent.is-translucent .workspace-tab-container-inner { + background-color: transparent !important; + border: transparent; +} + +.macOS-translucent.is-translucent .workspace-split .workspace-tabs, +.macOS-translucent.is-translucent .graph-controls, +.macOS-translucent.is-translucent .nav-file-title.is-active { + background-color: transparent !important; + box-shadow: inset -10px 0 4px -10px rgba(0, 0, 0, 0.04); +} + +.focus-mode.macOS-translucent.is-translucent .workspace { + background-color: var(--background-primary) !important; +} + +.macOS-translucent.is-translucent .workspace-ribbon.mod-right, +.macOS-translucent.is-translucent .workspace-ribbon.mod-left { + background: transparent; +} + +.macOS-translucent.is-translucent .mod-horizontal .workspace-leaf { + border-bottom: 0px; + background-color: transparent; + box-shadow: none !important; +} + +.macOS-translucent.is-translucent.theme-light .workspace { + --text-muted: hsl( + var(--base-h), + calc(var(--base-s) - 3%), + calc(var(--base-l) - 50%) + ); + --svg-faint: hsl( + var(--base-h), + calc(var(--base-s) - 3%), + calc(var(--base-l) - 38%) + ); +} + +/* -------------------------------------------------------------------------------- +Icon replacement +Thanks to Kepano, Matthew Meyers, and Chetachi Ezikeuzor +-------------------------------------------------------------------------------- */ + +.tree-item-self .collapse-icon { + width: 20px; +} + +body:not(.minimal-icons-off) .view-action svg, +body:not(.minimal-icons-off) .workspace-tab-header-inner-icon svg, +body:not(.minimal-icons-off) .nav-action-button svg, +body:not(.minimal-icons-off) .graph-controls-button svg { + width: 18px; + height: 18px; +} +body:not(.minimal-icons-off) .menu-item-icon svg { + width: 16px; + height: 16px; +} +body:not(.minimal-icons-off) .workspace-ribbon-collapse-btn svg { + width: 18px; + height: 18px; +} + +body:not(.minimal-icons-off) svg.any-key, +body:not(.minimal-icons-off) svg.blocks, +body:not(.minimal-icons-off) svg.bar-graph, +body:not(.minimal-icons-off) svg.breadcrumbs-trail-icon, +body:not(.minimal-icons-off) svg.audio-file, +body:not(.minimal-icons-off) svg.bold-glyph, +body:not(.minimal-icons-off) svg.italic-glyph, +body:not(.minimal-icons-off) svg.bracket-glyph, +body:not(.minimal-icons-off) svg.broken-link, +body:not(.minimal-icons-off) svg.bullet-list-glyph, +body:not(.minimal-icons-off) svg.bullet-list, +body:not(.minimal-icons-off) svg.calendar-day, +body:not(.minimal-icons-off) svg.calendar-with-checkmark, +body:not(.minimal-icons-off) svg.check-in-circle, +body:not(.minimal-icons-off) svg.check-small, +body:not(.minimal-icons-off) svg.checkbox-glyph, +body:not(.minimal-icons-off) svg.checkmark, +body:not(.minimal-icons-off) svg.clock, +body:not(.minimal-icons-off) svg.cloud, +body:not(.minimal-icons-off) svg.code-glyph, +body:not(.minimal-icons-off) svg.create-new, +body:not(.minimal-icons-off) svg.cross-in-box, +body:not(.minimal-icons-off) svg.cross, +body:not(.minimal-icons-off) svg.crossed-star, +body:not(.minimal-icons-off) svg.dice, +body:not(.minimal-icons-off) svg.disk, +body:not(.minimal-icons-off) svg.document, +body:not(.minimal-icons-off) svg.documents, +body:not(.minimal-icons-off) svg.dot-network, +body:not(.minimal-icons-off) svg.double-down-arrow-glyph, +body:not(.minimal-icons-off) svg.double-up-arrow-glyph, +body:not(.minimal-icons-off) svg.down-arrow-with-tail, +body:not(.minimal-icons-off) svg.down-chevron-glyph, +body:not(.minimal-icons-off) svg.enter, +body:not(.minimal-icons-off) svg.exit-fullscreen, +body:not(.minimal-icons-off) svg.expand-vertically, +body:not(.minimal-icons-off) svg.excalidraw-icon, +body:not(.minimal-icons-off) svg.filled-pin, +body:not(.minimal-icons-off) svg.folder, +body:not(.minimal-icons-off) svg.fullscreen, +body:not(.minimal-icons-off) svg.gear, +body:not(.minimal-icons-off) svg.hashtag, +body:not(.minimal-icons-off) svg.heading-glyph, +body:not(.minimal-icons-off) svg.go-to-file, +body:not(.minimal-icons-off) svg.help .widget-icon, +body:not(.minimal-icons-off) svg.help, +body:not(.minimal-icons-off) svg.highlight-glyph, +body:not(.minimal-icons-off) svg.horizontal-split, +body:not(.minimal-icons-off) svg.image-file, +body:not(.minimal-icons-off) svg.image-glyph, +body:not(.minimal-icons-off) svg.indent-glyph, +body:not(.minimal-icons-off) svg.info, +body:not(.minimal-icons-off) svg.install, +body:not(.minimal-icons-off) svg.keyboard-glyph, +body:not(.minimal-icons-off) svg.left-arrow-with-tail, +body:not(.minimal-icons-off) svg.left-arrow, +body:not(.minimal-icons-off) svg.left-chevron-glyph, +body:not(.minimal-icons-off) svg.lines-of-text, +body:not(.minimal-icons-off) svg.link-glyph, +body:not(.minimal-icons-off) svg.link, +body:not(.minimal-icons-off) svg.magnifying-glass, +body:not(.minimal-icons-off) svg.microphone-filled, +body:not(.minimal-icons-off) svg.microphone, +body:not(.minimal-icons-off) svg.minus-with-circle, +body:not(.minimal-icons-off) svg.note-glyph, +body:not(.minimal-icons-off) svg.number-list-glyph, +body:not(.minimal-icons-off) svg.open-vault, +body:not(.minimal-icons-off) svg.pane-layout, +body:not(.minimal-icons-off) svg.paper-plane, +body:not(.minimal-icons-off) svg.paused, +/*body:not(.minimal-icons-off) svg.pdf-file,*/ +body:not(.minimal-icons-off) svg.pencil, +body:not(.minimal-icons-off) svg.pin, +body:not(.minimal-icons-off) svg.plus-with-circle, +body:not(.minimal-icons-off) svg.popup-open, +body:not(.minimal-icons-off) svg.presentation, +body:not(.minimal-icons-off) svg.price-tag-glyph, +body:not(.minimal-icons-off) svg.quote-glyph, +body:not(.minimal-icons-off) svg.redo-glyph, +body:not(.minimal-icons-off) svg.reset, +body:not(.minimal-icons-off) svg.right-arrow-with-tail, +body:not(.minimal-icons-off) svg.right-arrow, +body:not(.minimal-icons-off) svg.right-chevron-glyph, +body:not(.minimal-icons-off) svg.right-triangle, +body:not(.minimal-icons-off) svg.run-command, +body:not(.minimal-icons-off) svg.search, +body:not(.minimal-icons-off) svg.sheets-in-box, +body:not(.minimal-icons-off) svg.spreadsheet, +body:not(.minimal-icons-off) svg.stacked-levels, +body:not(.minimal-icons-off) svg.star-list, +body:not(.minimal-icons-off) svg.star, +body:not(.minimal-icons-off) svg.strikethrough-glyph, +body:not(.minimal-icons-off) svg.switch, +body:not(.minimal-icons-off) svg.sync-small, +body:not(.minimal-icons-off) svg.sync, +body:not(.minimal-icons-off) svg.tag-glyph, +body:not(.minimal-icons-off) svg.three-horizontal-bars, +body:not(.minimal-icons-off) svg.trash, +body:not(.minimal-icons-off) svg.undo-glyph, +body:not(.minimal-icons-off) svg.unindent-glyph, +body:not(.minimal-icons-off) svg.up-and-down-arrows, +body:not(.minimal-icons-off) svg.up-arrow-with-tail, +body:not(.minimal-icons-off) svg.up-chevron-glyph, +body:not(.minimal-icons-off) svg.vault, +body:not(.minimal-icons-off) svg.vertical-split, +body:not(.minimal-icons-off) svg.vertical-three-dots, +body:not(.minimal-icons-off) svg.wrench-screwdriver-glyph, +body:not(.minimal-icons-off) svg.clock-glyph, +body:not(.minimal-icons-off) svg.command-glyph, +body:not(.minimal-icons-off) svg.add-note-glyph, +body:not(.minimal-icons-off) svg.calendar-glyph, +body:not(.minimal-icons-off) svg.duplicate-glyph, +body:not(.minimal-icons-off) svg.file-explorer-glyph, +body:not(.minimal-icons-off) svg.graph-glyph, +body:not(.minimal-icons-off) svg.import-glyph, +body:not(.minimal-icons-off) svg.languages, +body:not(.minimal-icons-off) svg.links-coming-in, +body:not(.minimal-icons-off) svg.links-going-out, +body:not(.minimal-icons-off) svg.merge-files-glyph, +body:not(.minimal-icons-off) svg.merge-files, +body:not(.minimal-icons-off) svg.open-elsewhere-glyph, +body:not(.minimal-icons-off) svg.paper-plane-glyph, +body:not(.minimal-icons-off) svg.paste-text, +body:not(.minimal-icons-off) svg.paste, +body:not(.minimal-icons-off) svg.percent-sign-glyph, +body:not(.minimal-icons-off) svg.play-audio-glyph, +body:not(.minimal-icons-off) svg.plus-minus-glyph, +body:not(.minimal-icons-off) svg.presentation-glyph, +body:not(.minimal-icons-off) svg.question-mark-glyph, +body:not(.minimal-icons-off) svg.restore-file-glyph, +body:not(.minimal-icons-off) svg.scissors-glyph, +body:not(.minimal-icons-off) svg.scissors, +body:not(.minimal-icons-off) svg.search-glyph, +body:not(.minimal-icons-off) svg.select-all-text, +body:not(.minimal-icons-off) svg.split, +body:not(.minimal-icons-off) svg.star-glyph, +body:not(.minimal-icons-off) svg.stop-audio-glyph, +body:not(.minimal-icons-off) svg.sweep, +body:not(.minimal-icons-off) svg.two-blank-pages, +body:not(.minimal-icons-off) svg.tomorrow-glyph, +body:not(.minimal-icons-off) svg.yesterday-glyph, +body:not(.minimal-icons-off) svg.workspace-glyph, +body:not(.minimal-icons-off) svg.box-glyph, +body:not(.minimal-icons-off) svg.wand, +body:not(.minimal-icons-off) svg.longform, +body:not(.minimal-icons-off) svg.changelog, +body:not(.no-sanctum-icons) svg.reading-glasses { + background-color: currentColor; +} + +body:not(.minimal-icons-off) svg.any-key > path, +body:not(.minimal-icons-off) svg.blocks > path, +body:not(.minimal-icons-off) svg.bar-graph > path, +body:not(.minimal-icons-off) svg.breadcrumbs-trail-icon > path, +body:not(.minimal-icons-off) svg.audio-file > path, +body:not(.minimal-icons-off) svg.bold-glyph > path, +body:not(.minimal-icons-off) svg.italic-glyph > path, +body:not(.minimal-icons-off) svg.bracket-glyph > path, +body:not(.minimal-icons-off) svg.broken-link > path, +body:not(.minimal-icons-off) svg.bullet-list-glyph > path, +body:not(.minimal-icons-off) svg.bullet-list > path, +body:not(.minimal-icons-off) svg.calendar-day > path, +body:not(.minimal-icons-off) svg.calendar-with-checkmark > path, +body:not(.minimal-icons-off) svg.check-in-circle > path, +body:not(.minimal-icons-off) svg.check-small > path, +body:not(.minimal-icons-off) svg.checkbox-glyph > path, +body:not(.minimal-icons-off) svg.checkmark > path, +body:not(.minimal-icons-off) svg.clock > path, +body:not(.minimal-icons-off) svg.cloud > path, +body:not(.minimal-icons-off) svg.code-glyph > path, +body:not(.minimal-icons-off) svg.command-glyph > path, +body:not(.minimal-icons-off) svg.create-new > path, +body:not(.minimal-icons-off) svg.cross-in-box > path, +body:not(.minimal-icons-off) svg.cross > path, +body:not(.minimal-icons-off) svg.crossed-star > path, +body:not(.minimal-icons-off) svg.dice > path, +body:not(.minimal-icons-off) svg.disk > path, +body:not(.minimal-icons-off) svg.document > path, +body:not(.minimal-icons-off) svg.documents > path, +body:not(.minimal-icons-off) svg.dot-network > path, +body:not(.minimal-icons-off) svg.double-down-arrow-glyph > path, +body:not(.minimal-icons-off) svg.double-up-arrow-glyph > path, +body:not(.minimal-icons-off) svg.down-arrow-with-tail > path, +body:not(.minimal-icons-off) svg.down-chevron-glyph > path, +body:not(.minimal-icons-off) svg.enter > path, +body:not(.minimal-icons-off) svg.exit-fullscreen > path, +body:not(.minimal-icons-off) svg.expand-vertically > path, +body:not(.minimal-icons-off) svg.excalidraw-icon > path, +body:not(.minimal-icons-off) svg.filled-pin > path, +body:not(.minimal-icons-off) svg.folder > path, +body:not(.minimal-icons-off) svg.fullscreen > path, +body:not(.minimal-icons-off) svg.gear > path, +body:not(.minimal-icons-off) svg.hashtag > path, +body:not(.minimal-icons-off) svg.heading-glyph > path, +body:not(.minimal-icons-off) svg.go-to-file > path, +body:not(.minimal-icons-off) svg.help .widget-icon > path, +body:not(.minimal-icons-off) svg.help > path, +body:not(.minimal-icons-off) svg.highlight-glyph > path, +body:not(.minimal-icons-off) svg.horizontal-split > path, +body:not(.minimal-icons-off) svg.image-file > path, +body:not(.minimal-icons-off) svg.image-glyph > path, +body:not(.minimal-icons-off) svg.indent-glyph > path, +body:not(.minimal-icons-off) svg.info > path, +body:not(.minimal-icons-off) svg.install > path, +body:not(.minimal-icons-off) svg.keyboard-glyph > path, +body:not(.minimal-icons-off) svg.left-arrow-with-tail > path, +body:not(.minimal-icons-off) svg.left-arrow > path, +body:not(.minimal-icons-off) svg.left-chevron-glyph > path, +body:not(.minimal-icons-off) svg.lines-of-text > path, +body:not(.minimal-icons-off) svg.link-glyph > path, +body:not(.minimal-icons-off) svg.link > path, +body:not(.minimal-icons-off) svg.magnifying-glass > path, +body:not(.minimal-icons-off) svg.microphone-filled > path, +body:not(.minimal-icons-off) svg.microphone > path, +body:not(.minimal-icons-off) svg.minus-with-circle > path, +body:not(.minimal-icons-off) svg.note-glyph > path, +body:not(.minimal-icons-off) svg.number-list-glyph > path, +body:not(.minimal-icons-off) svg.open-vault > path, +body:not(.minimal-icons-off) svg.pane-layout > path, +body:not(.minimal-icons-off) svg.paper-plane > path, +body:not(.minimal-icons-off) svg.paused > path, +/*body:not(.minimal-icons-off) svg.pdf-file > path,*/ +body:not(.minimal-icons-off) svg.pencil > path, +body:not(.minimal-icons-off) svg.pin > path, +body:not(.minimal-icons-off) svg.plus-with-circle > path, +body:not(.minimal-icons-off) svg.popup-open > path, +body:not(.minimal-icons-off) svg.presentation > path, +body:not(.minimal-icons-off) svg.price-tag-glyph > path, +body:not(.minimal-icons-off) svg.quote-glyph > path, +body:not(.minimal-icons-off) svg.redo-glyph > path, +body:not(.minimal-icons-off) svg.reset > path, +body:not(.minimal-icons-off) svg.right-arrow-with-tail > path, +body:not(.minimal-icons-off) svg.right-arrow > path, +body:not(.minimal-icons-off) svg.right-chevron-glyph > path, +body:not(.minimal-icons-off) svg.right-triangle > path, +body:not(.minimal-icons-off) svg.run-command > path, +body:not(.minimal-icons-off) svg.search > path, +body:not(.minimal-icons-off) svg.sheets-in-box > path, +body:not(.minimal-icons-off) svg.spreadsheet > path, +body:not(.minimal-icons-off) svg.stacked-levels > path, +body:not(.minimal-icons-off) svg.star-list > path, +body:not(.minimal-icons-off) svg.star > path, +body:not(.minimal-icons-off) svg.strikethrough-glyph > path, +body:not(.minimal-icons-off) svg.switch > path, +body:not(.minimal-icons-off) svg.sync-small > path, +body:not(.minimal-icons-off) svg.sync > path, +body:not(.minimal-icons-off) svg.tag-glyph > path, +body:not(.minimal-icons-off) svg.three-horizontal-bars > path, +body:not(.minimal-icons-off) svg.trash > path, +body:not(.minimal-icons-off) svg.undo-glyph > path, +body:not(.minimal-icons-off) svg.unindent-glyph > path, +body:not(.minimal-icons-off) svg.up-and-down-arrows > path, +body:not(.minimal-icons-off) svg.up-arrow-with-tail > path, +body:not(.minimal-icons-off) svg.up-chevron-glyph > path, +body:not(.minimal-icons-off) svg.vault > path, +body:not(.minimal-icons-off) svg.vertical-split > path, +body:not(.minimal-icons-off) svg.vertical-three-dots > path, +body:not(.minimal-icons-off) svg.wrench-screwdriver-glyph > path, +body:not(.minimal-icons-off) svg.clock-glyph > path, +body:not(.minimal-icons-off) svg.add-note-glyph > path, +body:not(.minimal-icons-off) svg.calendar-glyph > path, +body:not(.minimal-icons-off) svg.duplicate-glyph > path, +body:not(.minimal-icons-off) svg.file-explorer-glyph > path, +body:not(.minimal-icons-off) svg.graph-glyph > path, +body:not(.minimal-icons-off) svg.import-glyph > path, +body:not(.minimal-icons-off) svg.languages > path, +body:not(.minimal-icons-off) svg.links-coming-in > path, +body:not(.minimal-icons-off) svg.links-going-out > path, +body:not(.minimal-icons-off) svg.merge-files > path, +body:not(.minimal-icons-off) svg.open-elsewhere-glyph > path, +body:not(.minimal-icons-off) svg.paper-plane-glyph > path, +body:not(.minimal-icons-off) svg.paste-text > path, +body:not(.minimal-icons-off) svg.paste > path, +body:not(.minimal-icons-off) svg.percent-sign-glyph > path, +body:not(.minimal-icons-off) svg.play-audio-glyph > path, +body:not(.minimal-icons-off) svg.plus-minus-glyph > path, +body:not(.minimal-icons-off) svg.presentation-glyph > path, +body:not(.minimal-icons-off) svg.question-mark-glyph > path, +body:not(.minimal-icons-off) svg.restore-file-glyph > path, +body:not(.minimal-icons-off) svg.scissors-glyph > path, +body:not(.minimal-icons-off) svg.scissors > path, +body:not(.minimal-icons-off) svg.search-glyph > path, +body:not(.minimal-icons-off) svg.select-all-text > path, +body:not(.minimal-icons-off) svg.split > path, +body:not(.minimal-icons-off) svg.star-glyph > path, +body:not(.minimal-icons-off) svg.stop-audio-glyph > path, +body:not(.minimal-icons-off) svg.sweep > path, +body:not(.minimal-icons-off) svg.two-blank-pages > path, +body:not(.minimal-icons-off) svg.tomorrow-glyph > path, +body:not(.minimal-icons-off) svg.yesterday-glyph > path, +body:not(.minimal-icons-off) svg.workspace-glyph > path, +body:not(.minimal-icons-off) svg.box-glyph > path, +body:not(.minimal-icons-off) svg.wand > path, +body:not(.minimal-icons-off) svg.longform > path, +body:not(.minimal-icons-off) svg.changelog > path, +body:not(.no-sanctum-icons) svg.reading-glasses > path { + display: none; +} + +body:not(.minimal-icons-off) svg.any-key { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.audio-file { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.bar-graph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.breadcrumbs-trail-icon { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.blocks { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.bold-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.italic-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.bracket-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.broken-link { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.bullet-list-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.bullet-list { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.calendar-with-checkmark { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.check-in-circle { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.check-small { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.checkbox-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.checkmark { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.clock { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.clock-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.cloud { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.code-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.cross-in-box { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.cross { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); + -webkit-mask-image: url("data:image/svg+xml,"); + width: 18px; + height: 18px; +} +body:not(.minimal-icons-off) svg.crossed-star { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.dice { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.disk { + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-6 w-6' fill='none' viewBox='0 0 24 24' stroke='currentColor'%3E%3Cpath stroke-linecap='round' stroke-linejoin='round' stroke-width='2' d='M8 7H5a2 2 0 00-2 2v9a2 2 0 002 2h14a2 2 0 002-2V9a2 2 0 00-2-2h-3m-1 4l-3 3m0 0l-3-3m3 3V4' /%3E%3C/svg%3E"); +} +body:not(.no-svg-replace) svg.document { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.no-svg-replace) + .workspace-leaf-content[data-type='starred'] + svg.document { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) + .nav-action-button[aria-label='New note'] + svg.document, +body:not(.minimal-icons-off) svg.create-new { + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-6 w-6' fill='none' viewBox='0 0 24 24' stroke='currentColor'%3E%3Cpath stroke-linecap='round' stroke-linejoin='round' stroke-width='2' d='M11 5H6a2 2 0 00-2 2v11a2 2 0 002 2h11a2 2 0 002-2v-5m-1.414-9.414a2 2 0 112.828 2.828L11.828 15H9v-2.828l8.586-8.586z' /%3E%3C/svg%3E"); +} +body:not(.minimal-icons-off) svg.documents { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) + .workspace-leaf-content[data-type='video'] + .view-header + .view-header-icon + svg.document { + background-color: currentColor; + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' enable-background='new 0 0 32 32' viewBox='0 0 32 32' xml:space='preserve'%3E%3Cpath d='M10 6h4v4h-4zm8 0h4v4h-4zm-8 8h4v4h-4zm8 0h4v4h-4zm-8 8h4v4h-4zm8 0h4v4h-4z'/%3E%3Cpath fill='none' d='M0 0h32v32H0z'/%3E%3C/svg%3E"); +} +body:not(.minimal-icons-off) + .workspace-leaf-content[data-type='markdown'] + .view-header + .view-header-icon + svg.document { + background-color: currentColor; + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' enable-background='new 0 0 32 32' viewBox='0 0 32 32' xml:space='preserve'%3E%3Cpath d='M10 6h4v4h-4zm8 0h4v4h-4zm-8 8h4v4h-4zm8 0h4v4h-4zm-8 8h4v4h-4zm8 0h4v4h-4z'/%3E%3Cpath fill='none' d='M0 0h32v32H0z'/%3E%3C/svg%3E"); +} +body:not(.minimal-icons-off) svg.dot-network { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.double-down-arrow-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.double-up-arrow-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.down-arrow-with-tail { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.down-chevron-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.enter { + transform: translate(-2px); + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.excalidraw-icon { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.expand-vertically { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.filled-pin { + transform: rotate(45deg); + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.folder { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) + .workspace-tab-header[aria-label='File explorer'] + svg.folder { + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-6 w-6' fill='none' viewBox='0 0 24 24' stroke='currentColor'%3E%3Cpath stroke-linecap='round' stroke-linejoin='round' stroke-width='2' d='M3 12l2-2m0 0l7-7 7 7M5 10v10a1 1 0 001 1h3m10-11l2 2m-2-2v10a1 1 0 01-1 1h-3m-6 0a1 1 0 001-1v-4a1 1 0 011-1h2a1 1 0 011 1v4a1 1 0 001 1m-6 0h6' /%3E%3C/svg%3E"); +} +body:not(.minimal-icons-off) + .nav-action-button[aria-label='New folder'] + svg.folder { + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-6 w-6' fill='none' viewBox='0 0 24 24' stroke='currentColor'%3E%3Cpath stroke-linecap='round' stroke-linejoin='round' stroke-width='2' d='M9 13h6m-3-3v6m-9 1V7a2 2 0 012-2h6l2 2h6a2 2 0 012 2v8a2 2 0 01-2 2H5a2 2 0 01-2-2z' /%3E%3C/svg%3E"); +} +body:not(.minimal-icons-off) svg.fullscreen { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.gear { + -webkit-mask-image: url("data:image/svg+xml,"); +} +body:not(.minimal-icons-off) svg.hashtag { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.heading-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.go-to-file { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.help .widget-icon, +body:not(.minimal-icons-off) svg.help { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.highlight-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.horizontal-split { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.image-file { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.image-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.indent-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.info { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.install { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.keyboard-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.left-arrow-with-tail { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.left-arrow { + -webkit-mask-image: url("data:image/svg+xml,"); +} +body:not(.minimal-icons-off) svg.left-chevron-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.lines-of-text { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.link-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); + transform: rotate(90deg); +} +body:not(.minimal-icons-off) svg.link { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); + transform: rotate(90deg); +} +body:not(.minimal-icons-off) svg.magnifying-glass { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.microphone-filled { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.microphone { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.minus-with-circle { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.note-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.number-list-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.open-vault { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.pane-layout { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.paper-plane { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.paused { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.pencil { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.pin { + transform: rotate(45deg); + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.plus-with-circle { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.popup-open { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.presentation { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.price-tag-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.quote-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) + .workspace-tab-header[aria-label='Dictionary'] + svg.quote-glyph { + -webkit-mask-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' class='h-6 w-6' fill='none' viewBox='0 0 24 24' stroke='currentColor'%3E%3Cpath stroke-linecap='round' stroke-linejoin='round' stroke-width='2' d='M12 6.253v13m0-13C10.832 5.477 9.246 5 7.5 5S4.168 5.477 3 6.253v13C4.168 18.477 5.754 18 7.5 18s3.332.477 4.5 1.253m0-13C13.168 5.477 14.754 5 16.5 5c1.747 0 3.332.477 4.5 1.253v13C19.832 18.477 18.247 18 16.5 18c-1.746 0-3.332.477-4.5 1.253' /%3E%3C/svg%3E"); +} +body:not(.minimal-icons-off) svg.redo-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.reset { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.right-arrow-with-tail { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.right-arrow { + -webkit-mask-image: url("data:image/svg+xml,"); +} +body:not(.minimal-icons-off) svg.right-chevron-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.right-triangle { + color: var(--text-faint); + background-color: var(--text-faint); + height: 12px; + width: 12px; + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.command-glyph, +body:not(.minimal-icons-off) svg.run-command { + -webkit-mask-image: url("data:image/svg+xml,"); +} +body:not(.minimal-icons-off) svg.search { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.sheets-in-box { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.spreadsheet { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.stacked-levels { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.star-list { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.star { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.strikethrough-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.switch { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.sync-small { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.sync { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.tag-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.three-horizontal-bars { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.trash { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.undo-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.unindent-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.up-and-down-arrows { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.up-arrow-with-tail { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.up-chevron-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.vault { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.vertical-split { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.vertical-three-dots { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.wrench-screwdriver-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.add-note-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.calendar-day { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.calendar-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.duplicate-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.file-explorer-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.graph-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.import-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.languages { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.links-coming-in { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.links-going-out { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.merge-files { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.open-elsewhere-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.paper-plane-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.paste-text { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.paste { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.percent-sign-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.play-audio-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.plus-minus-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.presentation-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.question-mark-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.restore-file-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.scissors-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.scissors { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.search-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.select-all-text { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.split { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.star-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.stop-audio-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.sweep { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.two-blank-pages { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.tomorrow-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.yesterday-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.workspace-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.box-glyph { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.wand { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.longform { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.changelog { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} +body:not(.minimal-icons-off) svg.reading-glasses { + -webkit-mask-image: url('data:image/svg+xml;utf8,'); +} + +/* ─────────────────────────────────────────────────── */ +/* Plugin Compatibility info for the Obsidian Hub */ +/* ─────────────────────────────────────────────────── */ + +/* @plugins +core: +- backlink +- command-palette +- file-explorer +- global-search +- graph +- outgoing-link +- outline +- page-preview +- starred +- switcher +- tag-pane +- file-recovery +- daily-notes +- random-note +- publish +- sync +- word-count +community: +- sliding-panes-obsidian +- obsidian-codemirror-options +- obsidian-kanban +- dataview +- obsidian-hider +- calendar +- mysnippets-plugin +- cmenu-plugin +- obsidian-outliner +- readwise-official +- tag-wrangler +- todoist-sync-plugin +- templater-obsidian +- obsidian-system-dark-mode +- obsidian-style-settings +*/ + +/* Style Settings */ + +/* @settings +name: Things Theme +id: things-style +settings: + - + id: features + title: Features + type: heading + level: 2 + collapsed: true + - + id: minimal-icons-off + title: Default icons + description: Use default icons instead of minimal set + type: class-toggle + default: false + - + id: full-file-names + title: Show full file names + description: Turn off trimming on files in sidebar + type: class-toggle + - + id: links-int-on + title: Underline internal links + description: Show underlines on internal links + type: class-toggle + default: true + - + id: links-ext-on + title: Underline external links + description: Show underlines on external links + type: class-toggle + default: true + - + id: show-mobile-hamburger + title: Display hamburger menu on mobile + description: Display the top-left hamburger menu on mobile + type: class-toggle + default: false + - + id: fonts + title: Fonts + type: heading + level: 2 + collapsed: true + - + id: text + title: Text font + description: Used in preview mode + type: variable-text + default: -apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif + - + id: text-editor + title: Editor font + description: Used in edit mode + type: variable-text + default: -apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif + - + id: font-monospace + title: Monospace font + description: Used for code blocks and front matter + type: variable-text + default: JetBrains Mono,Menlo,SFMono-Regular,Consolas,"Roboto Mono",monospace + - + id: font-ui + title: UI font + description: Used for buttons, menus and sidebar + type: variable-text + default: -apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Oxygen-Sans,Ubuntu,Cantarell,"Helvetica Neue",sans-serif + - + id: custom-fonts + title: Typography + type: heading + level: 2 + collapsed: true + - + id: default-font-color + title: Default font colors + description: Use the default font color styling for bold, italics, and quotes + type: class-toggle + default: false + - + id: accent-h + title: Link hue color + description: Hue of both internal and external links + type: variable-number-slider + default: 215 + min: 0 + max: 360 + step: 1 + - + id: strong-color + title: Bold font color + type: variable-color + format: hex + default: '#FF82B2' + - + id: em-color + title: Italics font color + type: variable-color + format: hex + default: '#FF82B2' + - + id: green + title: Blockquotes font color + type: variable-color + format: hex + default: '#3EB4BF' + - + id: code-color-l + title: Inline code blocks font color (Light mode) + type: variable-color + format: hex + default: '#5C5C5C' + - + id: code-color-d + title: Inline code blocks font color (Dark mode) + type: variable-color + format: hex + default: '#A6A6A6' + - + id: tag-background-color-l + title: Tag background color (Light mode) + type: variable-color + format: hex + default: '#BDE1D3' + - + id: tag-font-color-l + title: Tag font color (Light mode) + type: variable-color + format: hex + default: '#1D694B' + - + id: tag-background-color-d + title: Tag background color (Dark mode) + type: variable-color + format: hex + default: '#1D694B' + - + id: tag-font-color-d + title: Tag font color (Dark mode) + type: variable-color + format: hex + default: '#' + - + id: editor-font-size + title: Editor font size + description: Font size in em for editor and preview overall font size + type: variable-number + default: 1 + format: em + - + id: font-small + title: Sidebar and tag font size + description: Font size in px of sidebar, tags, and small text + type: variable-number + default: 13 + format: px + - + id: font-smaller + title: Smaller font size + description: Font size in px of smaller text + type: variable-number + default: 11 + format: px + - + id: line-height + title: Body line height + description: Line height of the main text + type: variable-number + default: 1.5 + - + id: line-width + title: Normal line width + description: Number of characters per line + type: variable-number + default: 45 + format: rem + - + id: max-width + title: Maximum line width + description: Percentage of space inside a pane that a line can fill. Recommended values between 80 to 100 + type: variable-number + default: 90 + format: '%' + - + id: headings + title: Headings + type: heading + level: 2 + collapsed: true + - + id: level-1-headings + title: Level 1 Headings + type: heading + level: 3 + collapsed: true + - + id: h1 + title: H1 font size + description: Accepts any CSS font-size value + type: variable-text + default: 1.5em + - + id: h1-weight + title: H1 font weight + description: Accepts numbers representing the CSS font-weight + type: variable-number + default: 700 + - + id: h1-color + title: H1 color + type: variable-color + format: hex + default: '#' + - + id: level-2-headings + title: Level 2 Headings + type: heading + level: 3 + collapsed: true + - + id: h2 + title: H2 font size + description: Accepts any CSS font-size value + type: variable-text + default: 1.3em + - + id: h2-weight + title: H2 font weight + description: Accepts numbers representing the CSS font-weight + type: variable-number + default: 700 + - + id: h2-color + title: H2 color + type: variable-color + format: hex + default: '#2E80F2' + - + id: level-3-headings + title: Level 3 Headings + type: heading + level: 3 + collapsed: true + - + id: h3 + title: H3 font size + description: Accepts any CSS font-size value + type: variable-text + default: 1.1em + - + id: h3-weight + title: H3 font weight + description: Accepts numbers representing the CSS font-weight + type: variable-number + default: 600 + - + id: h3-color + title: H3 color + type: variable-color + format: hex + default: '#2E80F2' + - + id: level-4-headings + title: Level 4 Headings + type: heading + level: 3 + collapsed: true + - + id: h4 + title: H4 font size + description: Accepts any CSS font-size value + type: variable-text + default: 0.9em + - + id: h4-weight + title: H4 font weight + description: Accepts numbers representing the CSS font-weight + type: variable-number + default: 500 + - + id: h4-color + title: H4 color + type: variable-color + format: hex + default: '#E5B567' + - + id: h4-transform + title: H4 transform + description: Transform the H4 heading text + type: variable-select + default: uppercase + options: + - + label: Uppercase + value: uppercase + - + label: None + value: none + - + id: level-5-headings + title: Level 5 Headings + type: heading + level: 3 + collapsed: true + - + id: h5 + title: H5 font size + description: Accepts any CSS font-size value + type: variable-text + default: 0.85em + - + id: h5-weight + title: H5 font weight + description: Accepts numbers representing the CSS font-weight + type: variable-number + default: 500 + - + id: h5-color + title: H5 color + type: variable-color + format: hex + default: '#E83E3E' + - + id: level-6-headings + title: Level 6 Headings + type: heading + level: 3 + collapsed: true + - + id: h6 + title: H6 font size + description: Accepts any CSS font-size value + type: variable-text + default: 0.85em + - + id: h6-weight + title: H6 font weight + description: Accepts numbers representing the CSS font-weight + type: variable-number + default: 400 + - + id: h6-color + title: H6 color + type: variable-color + format: hex + default: '#' + - + id: advanced + title: Advanced + type: heading + level: 2 + collapsed: true + - + title: Disable mobile floating-action button + description: Revert placement of edit/preview button to default in header (mobile) + id: floating-button-off + type: class-toggle + default: false + - + title: MacOS-like translucent window + description: Give workspace a MacOS-like translucency + id: macOS-translucent + type: class-toggle + default: false + - + id: cursor + title: Cursor style + description: The cursor style for UI elements + type: variable-select + default: default + options: + - + label: Default + value: default + - + label: Pointer + value: pointer + - + label: Crosshair + value: crosshair + - + id: credits + title: Credits + type: heading + description: Created with ❤︎ by @colineckert. This theme uses code from Minimal by @kepano. Support @kepano at buymeacoffee.com/kepano and @colineckert at buymeacoffee.com/colineckert + level: 2 + collapsed: true + +*/ diff --git a/docs/.obsidian/workspace b/docs/.obsidian/workspace new file mode 100644 index 0000000000..2005115db2 --- /dev/null +++ b/docs/.obsidian/workspace @@ -0,0 +1,141 @@ +{ + "main": { + "id": "2c9313c97191b3d5", + "type": "split", + "children": [ + { + "id": "17a0267a38d6cd5c", + "type": "leaf", + "state": { + "type": "markdown", + "state": { + "file": "data-management/MySQL/MySQL-Index.md", + "mode": "preview", + "source": false + } + } + } + ], + "direction": "vertical" + }, + "left": { + "id": "c2ba06bd2f318734", + "type": "split", + "children": [ + { + "id": "8aec8ec279c7aff1", + "type": "tabs", + "children": [ + { + "id": "ae760b575766f3f0", + "type": "leaf", + "state": { + "type": "file-explorer", + "state": {} + } + }, + { + "id": "f88a42db0e3bb960", + "type": "leaf", + "state": { + "type": "search", + "state": { + "query": "", + "matchingCase": false, + "explainSearch": false, + "collapseAll": false, + "extraContext": false, + "sortOrder": "alphabetical" + } + } + }, + { + "id": "c7bda26138bbb70d", + "type": "leaf", + "state": { + "type": "starred", + "state": {} + } + } + ] + } + ], + "direction": "horizontal", + "width": 300 + }, + "right": { + "id": "4d54828aaab36d9f", + "type": "split", + "children": [ + { + "id": "4acb1c44f8a68ec8", + "type": "tabs", + "children": [ + { + "id": "f56e43f509009e33", + "type": "leaf", + "state": { + "type": "backlink", + "state": { + "file": "data-management/MySQL/MySQL-Index.md", + "collapseAll": false, + "extraContext": false, + "sortOrder": "alphabetical", + "showSearch": false, + "searchQuery": "", + "backlinkCollapsed": false, + "unlinkedCollapsed": true + } + } + }, + { + "id": "e22cbd6030bde448", + "type": "leaf", + "state": { + "type": "outgoing-link", + "state": { + "file": "data-management/MySQL/MySQL-Index.md", + "linksCollapsed": false, + "unlinkedCollapsed": true + } + } + }, + { + "id": "37585a229386609d", + "type": "leaf", + "state": { + "type": "tag", + "state": { + "sortOrder": "frequency", + "useHierarchy": true + } + } + }, + { + "id": "1b738c49f6ecdf2c", + "type": "leaf", + "state": { + "type": "outline", + "state": { + "file": "data-management/MySQL/MySQL-Index.md" + } + } + } + ], + "currentTab": 3 + } + ], + "direction": "horizontal", + "width": 300 + }, + "active": "17a0267a38d6cd5c", + "lastOpenFiles": [ + "interview/Kafka-FAQ.md", + "work/DPA.md", + "data-structure-algorithms/soultion/Array-Solution.md", + "data-structure-algorithms/Array.md", + "data-structure-algorithms/Recursion.md", + "data-management/Big-Data/Bloom-Filter.md", + "data-structure-algorithms/Linked-List.md" + ] +} \ No newline at end of file diff --git a/docs/.obsidian/workspace.json b/docs/.obsidian/workspace.json new file mode 100644 index 0000000000..d648b6a945 --- /dev/null +++ b/docs/.obsidian/workspace.json @@ -0,0 +1,302 @@ +{ + "main": { + "id": "2c9313c97191b3d5", + "type": "split", + "children": [ + { + "id": "233cec07312ea132", + "type": "tabs", + "children": [ + { + "id": "17a0267a38d6cd5c", + "type": "leaf", + "state": { + "type": "markdown", + "state": { + "file": "data-management/MySQL/MySQL-Index.md", + "mode": "preview", + "source": false + }, + "icon": "lucide-file", + "title": "MySQL-Index" + } + }, + { + "id": "264bec0c6c3126ab", + "type": "leaf", + "state": { + "type": "markdown", + "state": { + "file": "data-structure-algorithms/soultion/LinkedList-Soultion.md", + "mode": "source", + "source": false + }, + "icon": "lucide-file", + "title": "LinkedList-Soultion" + } + }, + { + "id": "e24a131a4a8c13f9", + "type": "leaf", + "state": { + "type": "markdown", + "state": { + "file": "data-structure-algorithms/algorithm/Dynamic-Programming.md", + "mode": "source", + "source": false + }, + "icon": "lucide-file", + "title": "Dynamic-Programming" + } + }, + { + "id": "56243bd84331da43", + "type": "leaf", + "state": { + "type": "release-notes", + "state": { + "currentVersion": "1.8.9" + }, + "icon": "lucide-book-up", + "title": "Release Notes 1.8.9" + } + }, + { + "id": "80d5c719688f185a", + "type": "leaf", + "state": { + "type": "release-notes", + "state": { + "currentVersion": "1.8.10" + }, + "icon": "lucide-book-up", + "title": "Release Notes 1.8.10" + } + }, + { + "id": "c7c1397df03f59cc", + "type": "leaf", + "state": { + "type": "empty", + "state": {}, + "icon": "lucide-file", + "title": "新标签页" + } + }, + { + "id": "f02bfed475530fbe", + "type": "leaf", + "state": { + "type": "markdown", + "state": { + "file": "data-structure-algorithms/soultion/DP-Solution.md", + "mode": "source", + "source": false + }, + "icon": "lucide-file", + "title": "DP-Solution" + } + }, + { + "id": "0d9db99c2c25b1c2", + "type": "leaf", + "state": { + "type": "markdown", + "state": { + "file": "data-structure-algorithms/soultion/DP-Solution.md", + "mode": "source", + "source": false + }, + "icon": "lucide-file", + "title": "DP-Solution" + } + } + ], + "currentTab": 4 + } + ], + "direction": "vertical" + }, + "left": { + "id": "c2ba06bd2f318734", + "type": "split", + "children": [ + { + "id": "8aec8ec279c7aff1", + "type": "tabs", + "children": [ + { + "id": "ae760b575766f3f0", + "type": "leaf", + "state": { + "type": "file-explorer", + "state": { + "sortOrder": "alphabetical", + "autoReveal": false + }, + "icon": "lucide-folder-closed", + "title": "文件列表" + } + }, + { + "id": "f88a42db0e3bb960", + "type": "leaf", + "state": { + "type": "search", + "state": { + "query": "", + "matchingCase": false, + "explainSearch": false, + "collapseAll": false, + "extraContext": false, + "sortOrder": "alphabetical" + }, + "icon": "lucide-search", + "title": "搜索" + } + }, + { + "id": "c7bda26138bbb70d", + "type": "leaf", + "state": { + "type": "starred", + "state": {}, + "icon": "lucide-file", + "title": "插件不再活动" + } + }, + { + "id": "123cfa366479ce4d", + "type": "leaf", + "state": { + "type": "bookmarks", + "state": {}, + "icon": "lucide-bookmark", + "title": "书签" + } + } + ] + } + ], + "direction": "horizontal", + "width": 200 + }, + "right": { + "id": "4d54828aaab36d9f", + "type": "split", + "children": [ + { + "id": "4acb1c44f8a68ec8", + "type": "tabs", + "children": [ + { + "id": "f56e43f509009e33", + "type": "leaf", + "state": { + "type": "backlink", + "state": { + "file": "data-management/Redis/Redis-Datatype.md", + "collapseAll": true, + "extraContext": false, + "sortOrder": "alphabetical", + "showSearch": false, + "searchQuery": "", + "backlinkCollapsed": false, + "unlinkedCollapsed": true + }, + "icon": "links-coming-in", + "title": "Redis-Datatype 的反向链接列表" + } + }, + { + "id": "e22cbd6030bde448", + "type": "leaf", + "state": { + "type": "outgoing-link", + "state": { + "file": "data-management/Redis/Redis-Datatype.md", + "linksCollapsed": false, + "unlinkedCollapsed": true + }, + "icon": "links-going-out", + "title": "Redis-Datatype 的出链列表" + } + }, + { + "id": "37585a229386609d", + "type": "leaf", + "state": { + "type": "tag", + "state": { + "sortOrder": "frequency", + "useHierarchy": true + }, + "icon": "lucide-tags", + "title": "标签" + } + }, + { + "id": "1b738c49f6ecdf2c", + "type": "leaf", + "state": { + "type": "outline", + "state": { + "followCursor": false, + "showSearch": false, + "searchQuery": "" + }, + "icon": "lucide-list", + "title": "大纲" + } + } + ], + "currentTab": 3 + } + ], + "direction": "horizontal", + "width": 300 + }, + "left-ribbon": { + "hiddenItems": { + "switcher:打开快速切换": false, + "graph:查看关系图谱": false, + "canvas:新建白板": false, + "daily-notes:打开/创建今天的日记": false, + "templates:插入模板": false, + "command-palette:打开命令面板": false + } + }, + "active": "80d5c719688f185a", + "lastOpenFiles": [ + "framework/SpringWebFlux/Webflux~.md", + "framework/SpringWebFlux/Webflux.md", + "framework/SpringWebFlux/Untitled.md", + "framework/SpringWebFlux/响应式编程~.md", + "framework/SpringWebFlux/SpringWebFlux~.md", + "framework/SpringWebFlux/WebFlux~.md", + "data-management/Redis/Redis-Datatype.md", + "data-management/Redis/Redis-Database.md", + "data-management/Redis/Nosql-Overview.md", + "data-management/Redis/Nosql-Overview~.md", + "data-management/Redis/reproduce/Cache-Design.md", + "data-management/Redis/reproduce/Redis为什么变慢了-常见延迟问题定位与分析.md", + "data-management/Redis/reproduce/Key 寻址算法.md", + "interview/Redis-FAQ.md", + "data-structure-algorithms/soultion/Binary-Tree-Solution.md", + "未命名文件夹", + "java/IO~.md", + "java/IO.md", + "java/未命名.md", + "interview/Ability-FAQ~.md", + "data-structure-algorithms/algorithm/BFS.md", + "data-structure-algorithms/algorithm/Untitled.md", + "interview/Algorithm.md", + "interview/Alto.md", + "interview/Untitled.md", + "data-structure-algorithms/data-structure/Binary-Tree.md", + "data-structure-algorithms/soultion/DP-Solution.md", + "Untitled.canvas", + "data-structure-algorithms/algorithm", + "data-structure-algorithms/未命名文件夹" + ] +} \ No newline at end of file diff --git a/docs/.vuepress/.DS_Store b/docs/.vuepress/.DS_Store new file mode 100644 index 0000000000..cd94417745 Binary files /dev/null and b/docs/.vuepress/.DS_Store differ diff --git a/docs/.vuepress/config.js b/docs/.vuepress/config.js new file mode 100644 index 0000000000..42bf241df7 --- /dev/null +++ b/docs/.vuepress/config.js @@ -0,0 +1,392 @@ +module.exports = { + theme: require.resolve('./theme/vuepress-theme-reco'), + //theme: 'reco', + base: "/", + title: 'JavaKeeper', + //description: 'Keep On Growing：Java Keeper', + head: [ + ["link", { rel: "icon", href: `/icon.svg` }], + ['meta', { name: 'keywords', content: 'JavaKeeper,Java,Java开发,算法,blog' }], + ['script', {}, ` + var _hmt = _hmt || []; + (function() { + var hm = document.createElement("script"); + hm.src = "https://hm.baidu.com/hm.js?a949a9b30eb86ac0159e735ff8670c03"; + var s = document.getElementsByTagName("script")[0]; + s.parentNode.insertBefore(hm, s); + // 引入谷歌,不需要可删除这段 + var hm1 = document.createElement("script"); + hm1.src = "https://www.googletagmanager.com/gtag/js?id=UA-169923503-1"; + var s1 = document.getElementsByTagName("script")[0]; + s1.parentNode.insertBefore(hm1, s1); + })(); + // 谷歌加载,不需要可删除 + window.dataLayer = window.dataLayer || []; + function gtag(){dataLayer.push(arguments);} + gtag('js', new Date()); + gtag('config', 'UA-169923503-1'); + `], + ], + themeConfig: { + author: '海星', + repo: 'Jstarfish/JavaKeeper', + //logo: './public/img/logo.png', + subSidebar: 'auto',//在所有页面中启用自动生成子侧边栏，原 sidebar 仍然兼容 + nav: [ + { text: 'Java', link: '/java/' , icon: 'icon-java'}, + { text: '数据结构与算法', link: '/data-structure-algorithms/', icon: 'icon-tree' }, + { text: '设计模式', link: '/design-pattern/', icon: 'icon-design' }, + { text: '数据管理', link: '/data-management/', icon: 'icon-ic_datastores'}, + { text: '开发框架', link: '/framework/', icon: 'icon-framework1' }, + { text: '分布式架构', link: '/distribution/', icon: 'icon-distributed' }, + { text: '网络编程', link: '/network/' , icon: 'icon-network'}, + { text: '直击面试', link: '/interview/', icon: 'icon-interview' }, + ], + sidebar: { + "/java/": genJavaSidebar(), + "/data-structure-algorithms/": genDSASidebar(), + "/design-pattern/": genDesignPatternSidebar(), + "/data-management/": genDataManagementSidebar(), + "/framework/": genFrameworkSidebar(), + "/distribution/": genDistributionSidebar(), + "/network/": genNetworkSidebar(), + "/interview/": genInterviewSidebar(), + }, + blogConfig: { + // category: { + // location: 2, // 在导航栏菜单中所占的位置，默认2 + // text: 'Category' // 默认文案 “分类” + // }, + // tag: { + // location: 3, // 在导航栏菜单中所占的位置，默认3 + // text: 'Tag' // 默认文案 “标签” + // } + } + }, + plugins: [ + ['@vuepress-reco/vuepress-plugin-bulletin-popover', { + width: '260px', // 默认 260px + title: '消息提示', + body: [ + { + type: 'title', + content: '
🐳 欢迎关注〖JavaKeeper〗🐳
🎉 500 + Java开发电子书免费获取 🎉

', + style: 'text-aligin: center;width: 100%;' + }, + { + type: 'image', + src: '/qcode.png' + } + ] + //, + // footer: [ + // { + // type: 'button', + // text: '打赏', + // link: '/donate' + // } + // ] + }], + [ + 'vuepress-plugin-mathjax', + { + target: 'svg', + macros: { + '*': '\\times', + }, + }, + ], + ['@vuepress/last-updated'] + ] +} + +function genJavaSidebar() { + return [ + { + title: "Java", + collapsable: true, + children: [ + "Java-8", + "Java-Throwable", + "Online-Error-Check", + ] + }, + { + title: "JVM", + collapsable: true, + sidebarDepth: 2, // 可选的, 默认值是 1 + children: ["JVM/JVM-Java","JVM/Class-Loading","JVM/Runtime-Data-Areas","JVM/GC","JVM/GC-实战", + "JVM/Java-Object", "JVM/JVM参数配置", + "JVM/OOM","JVM/Reference","JVM/JVM性能监控和故障处理工具"] + }, + { + title: "JUC", + collapsable: true, + children: [ + ["JUC/readJUC","开篇——聊聊并发编程"], + "JUC/Java-Memory-Model", + "JUC/volatile","JUC/synchronized","JUC/CAS", + ['JUC/Concurrent-Container','Collection 大局观'], + "JUC/AQS", + "JUC/ThreadLocal", + "JUC/CountDownLatch、CyclicBarrier、Semaphore", + ['JUC/BlockingQueue','阻塞队列'], + "JUC/Thread-Pool", + "JUC/Locks", + "JUC/多个线程顺序执行问题", + ] + }, + { + title: "Other", + collapsable: true, + children: [ + "other/Git-Specification", + ] + } + ]; +} + +function genDSASidebar() { + return [ + { + title: "数据结构", + collapsable: true, + //sidebarDepth: 2, // 可选的, 默认值是 1 + children: [ + ['data-structure/Array','数组'], + ['data-structure/Linked-List','链表'], + ['data-structure/Stack','栈'], + ['data-structure/Queue','队列'], + ['data-structure/Binary-Tree','二叉树'], + ['data-structure/Skip-List','跳表'] + ] + }, + { + title: "算法", + collapsable: true, + children: [ + "complexity", + ['algorithm/Sort', '排序'], + ['algorithm/Binary-Search', '二分查找'], + ['algorithm/Recursion', '递归'], + ['algorithm/Backtracking', '回溯'], + ['algorithm/DFS-BFS', 'DFS vs BFS'], + ['algorithm/Double-Pointer', '双指针'], + ['algorithm/Dynamic-Programming', '动态规划'] + ] + }, + { + title: "刷题", + collapsable: true, + children: [ + ['soultion/Binary-Tree-Solution', '二叉树'], + ['soultion/Array-Solution', '数组'], + ['soultion/String-Solution', '字符串'], + ['soultion/LinkedList-Soultion', '链表'], + ['soultion/DFS-Solution', 'DFS'], + ['soultion/Math-Solution', '数学'], + ['soultion/stock-problems', '股票问题'], + ['soultion/剑指offer', '剑指offer'] + ] + } + ]; +} + +function genDesignPatternSidebar() { + return [ + ['Design-Pattern-Overview', '设计模式前传'], + { + title: "创建型模式", + collapsable: true, + sidebarDepth: 3, // 可选的, 默认值是 1 + children: [ + ['Singleton-Pattern', '单例模式'], + ['Factory-Pattern', '工厂模式'], + ['Prototype-Pattern', '原型模式'], + ['Builder-Pattern', '建造者模式'] + ] + }, + { + title: "结构型模式", + collapsable: true, + sidebarDepth: 3, // 可选的, 默认值是 1 + children: [ + ['Decorator-Pattern', '装饰模式'], + ['Proxy-Pattern', '代理模式'], + ['Adapter-Pattern', '适配器模式'] + ] + }, + { + title: "行为模式", + collapsable: true, + sidebarDepth: 2, // 可选的, 默认值是 1 + children: [ + ['Chain-of-Responsibility-Pattern', '责任链模式'], + ['Observer-Pattern', '观察者模式'], + ['Template-Pattern', '模板方法模式'], + ['Strategy-Pattern', '策略模式'], + ['Facade-Pattern', '外观模式'] + ] + }, + ['Pipeline-Pattern', '管道模式'], + ['Spring-Design.md', 'Spring 中的设计模式'] + ]; +} + +function genDataManagementSidebar(){ + return [ + { + title: "MySQL", + collapsable: true, + //sidebarDepth: 1, // 可选的, 默认值是 1 + children: [ + ['MySQL/MySQL-Framework', 'MySQL 架构介绍'], + ['MySQL/MySQL-Storage-Engines', 'MySQL 存储引擎'], + ['MySQL/MySQL-Index', 'MySQL 索引'], + ['MySQL/MySQL-Transaction', 'MySQL 事务'], + ['MySQL/MySQL-Log', 'MySQL 日志'], + ['MySQL/MySQL-Lock', 'MySQL 锁'], + ['MySQL/MySQL-Select', 'MySQL 查询'], + ['MySQL/MySQL-Optimization', 'MySQL 优化'], + ['MySQL/Three-Normal-Forms', '数据库三范式'] + ] + }, + { + title: "Redis", + collapsable: true, + sidebarDepth: 2, // 可选的, 默认值是 1 + children: [ + ['Redis/ReadRedis', 'Redis 开篇'], + ['Redis/Redis-Datatype', 'Redis 数据类型'], + ['Redis/Redis-Persistence', 'Redis 持久化'], + ['Redis/Redis-Conf', 'Redis 配置'], + ['Redis/Redis-Transaction', 'Redis 事务'], + ['Redis/Redis-Lock', 'Redis 分布式锁'], + ['Redis/Redis-Master-Slave', 'Redis 主从'], + ['Redis/Redis-Sentinel', 'Redis 哨兵'], + ['Redis/Redis-Cluster', 'Redis 集群'], + ['Redis/Redis-MQ', 'Redis 消息队列方案'], + ] + }, + { + title: "Big-Data", + collapsable: true, + children: [ + ['Big-Data/Hello-BigData', '大数据'], + ['Big-Data/Hive', 'Hive'], + ['Big-Data/Bloom-Filter', '布隆过滤器'], + ['Big-Data/Kylin', 'Kylin'], + ['Big-Data/HBase', 'HBase'], + ['Big-Data/Phoenix', 'Phoneix'] + ] + } + + ]; +} + +function genFrameworkSidebar(){ + return [ + { + title: "Spring", + collapsable: true, + sidebarDepth: 2, // 可选的, 默认值是 1 + children: [ + ['Spring/Spring-IOC', 'Spring IOC'], + ['Spring/Spring-IOC-Source', 'Spring IOC 源码解毒'], + ['Spring/Spring-Cycle-Dependency', 'Spring 循环依赖'], + ['Spring/Spring-AOP', 'Spring AOP'], + ['Spring/Spring-MVC', 'Spring MVC'], + ] + }, + { + title: "Spring Boot", + collapsable: true, + sidebarDepth: 2, // 可选的, 默认值是 1 + children: [ + ['SpringBoot/Hello-SpringBoot', 'Hello-SpringBoot'], + // ['SpringBoot/Spring Boot 最流行的 16 条实践解读', 'Spring Boot 最流行的 16 条实践解读'], + // ['SpringBoot/@Scheduled', '@Scheduled'], + ] + }, + { + title: "Quartz", + collapsable: true, + sidebarDepth: 2, // 可选的, 默认值是 1 + children: [ + ['Quartz/Quartz', 'Hello Quartz'], + ['Quartz/Quartz-MySQL', 'jobstore 数据库表结构'], + // ['SpringBoot/@Scheduled', '@Scheduled'], + ] + }, + { + title: "Logging", + collapsable: true, + sidebarDepth: 2, // 可选的, 默认值是 1 + children: [ + ['logging/Java-Logging', 'Hello Logging'] + ] + } + ]; +} + +function genDistributionSidebar(){ + return [ + { + title: "Kafka", + collapsable: true, + sidebarDepth: 2, // 可选的, 默认值是 1 + children: [ + ['message-queue/Kafka/Hello-Kafka', 'Hello-Kafka'], + ['message-queue/Kafka/Kafka-Version', 'Kafka版本问题'], + ['message-queue/Kafka/Kafka-Workflow','Kafka-Workflow'], + ['message-queue/Kafka/Kafka-Producer','Kafka-Producer'], + ['message-queue/Kafka/Kafka-Consumer','Kafka-Consumer'], + ['message-queue/Kafka/Kafka高效读写数据的原因','Kafka高效读写数据的原因'] + ] + }, + { + title: " Zookeeper", + collapsable: true, + children: [ + ['ZooKeeper/Consistency-Protocol','分布式一致性协议'], + ['ZooKeeper/Hello-Zookeeper','Hello Zookeeper'], + ['ZooKeeper/Zookeeper-Use','Zookeeper 实战'], + ] + }, + { + title: "RPC", + collapsable: true, + sidebarDepth: 2, // 可选的, 默认值是 1 + children: [ + ['rpc/Hello-Protocol-Buffers', 'Hello ProtocolBuffers'], + ['rpc/Hello-RPC.md','Hello RPC'], + ['rpc/Hello-gRPC','Hello gRPC'], + ] + }, + ]; +} + +function genNetworkSidebar(){ + return [ + ['RMI', 'RMI远程调用'], + ]; +} + +function genInterviewSidebar(){ + return [ + ['Java-Basics-FAQ', 'Java基础部分'], + ['Java-Collections-FAQ', 'Java集合部分'], + ['JUC-FAQ', 'Java 多线程部分'], + ['JVM-FAQ', 'JVM 部分'], + ['MySQL-FAQ', 'MySQL 部分'], + ['Redis-FAQ', 'Redis 部分'], + ['Network-FAQ', '计算机网络部分'], + ['Kafka-FAQ', 'Kafka 部分'], + ['ZooKeeper-FAQ', 'Zookeeper 部分'], + ['RPC-FAQ', 'RPC 部分'], + ['MyBatis-FAQ', 'MyBatis 部分'], + ['Spring-FAQ', 'Spring 部分'], + ['Design-Pattern-FAQ', '设计模式部分'], + ['Elasticsearch-FAQ', 'Elasticsearch 部分'], + ]; +} \ No newline at end of file diff --git a/docs/.vuepress/dist b/docs/.vuepress/dist new file mode 160000 index 0000000000..105cc8a4b2 --- /dev/null +++ b/docs/.vuepress/dist @@ -0,0 +1 @@ +Subproject commit 105cc8a4b28dbf98f7e847970e21da82cdaba28c diff --git "a/images/end - \345\211\257\346\234\254.png" b/docs/.vuepress/public/end_code.png similarity index 100% rename from "images/end - \345\211\257\346\234\254.png" rename to docs/.vuepress/public/end_code.png diff --git "a/images/\344\272\232\345\250\201\345\206\234\345\260\221\345\245\263.png" b/docs/.vuepress/public/girl_code.png similarity index 100% rename from "images/\344\272\232\345\250\201\345\206\234\345\260\221\345\245\263.png" rename to docs/.vuepress/public/girl_code.png diff --git a/docs/.vuepress/public/icon.svg b/docs/.vuepress/public/icon.svg new file mode 100644 index 0000000000..cede12c59a --- /dev/null +++ b/docs/.vuepress/public/icon.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/images/a.jpg b/docs/.vuepress/public/logo.jpg similarity index 100% rename from images/a.jpg rename to docs/.vuepress/public/logo.jpg diff --git a/docs/.vuepress/public/qcode.png b/docs/.vuepress/public/qcode.png new file mode 100644 index 0000000000..0c93c28828 Binary files /dev/null and b/docs/.vuepress/public/qcode.png differ diff --git a/docs/.vuepress/public/qrcode.jpg b/docs/.vuepress/public/qrcode.jpg new file mode 100644 index 0000000000..3688d2be35 Binary files /dev/null and b/docs/.vuepress/public/qrcode.jpg differ diff --git a/docs/.vuepress/theme/.DS_Store b/docs/.vuepress/theme/.DS_Store new file mode 100644 index 0000000000..b6fd03ec58 Binary files /dev/null and b/docs/.vuepress/theme/.DS_Store differ diff --git a/docs/.vuepress/theme/vuepress-theme-reco/.DS_Store b/docs/.vuepress/theme/vuepress-theme-reco/.DS_Store new file mode 100644 index 0000000000..41406f682d Binary files /dev/null and b/docs/.vuepress/theme/vuepress-theme-reco/.DS_Store differ diff --git a/docs/.vuepress/theme/vuepress-theme-reco/.npmignore b/docs/.vuepress/theme/vuepress-theme-reco/.npmignore new file mode 100644 index 0000000000..d059d27b01 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/.npmignore @@ -0,0 +1,3 @@ +__tests__ +__mocks__ +node_modules diff --git a/docs/.vuepress/theme/vuepress-theme-reco/README.md b/docs/.vuepress/theme/vuepress-theme-reco/README.md new file mode 100644 index 0000000000..2b6d41e485 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/README.md @@ -0,0 +1,88 @@ +

Vue logo

+ +

+ + + + +

+ +## Introduce + +1. 这是一个vuepress主题，旨在添加博客所需的分类、标签墙、分页、评论等功能； +2. 主题追求极简，根据 vuepress 的默认主题修改而成，官方的主题配置仍然适用； +3. 效果：[午后南杂](https://www.recoluan.com) +4. 文档：[vuepress-theme-reco-doc](https://vuepress-theme-reco.recoluan.com) + +## Quick start + +**npx** + +``` +npx @vuepress-reco/theme-cli init my-blog +``` + +**npm** + +```bash +# init +npm install @vuepress-reco/theme-cli -g +theme-cli init my-blog + +# install +cd my-blog +npm install + +# run +npm run dev + +# build +npm run build +``` + +**yarn** + +```bash +# init +yarn global add @vuepress-reco/theme-cli +theme-cli init my-blog + +# install +cd my-blog +yarn install + +# run +yarn dev + +# build +yarn build +``` + +## Preview + +![size.png](https://i.loli.net/2020/01/13/nCbXp13lRG2TNeD.png) + +![style.png](https://i.loli.net/2020/01/13/ke1VirShQRLnEd7.png) + +![dark.png](https://i.loli.net/2020/01/13/Lj6XbwdmDFCYH9k.png) + +![home.png](https://i.loli.net/2020/01/13/nra3kbYSlxojmw4.png) + +## Contributors + +**衷心感谢为此项目贡献过宝贵代码的朋友们** + +|昵称|贡献记录| +|:-:|-| +|[kangxu](https://github.com/kangxukangxu)|[vuepress-theme-reco@0.x](https://github.com/recoluan/vuepress-theme-reco/commit/ec7426a88d50cf8d9f90a7ad9b731a10da7f438b)| +|[Ekko](https://github.com/danranVm)|[vuepress-theme-reco-demo@1.x](https://github.com/recoluan/vuepress-theme-reco-demo/commit/6d2bbe919e7f6564b8c8173558d197e38e024dc5)| + +**衷心感谢美女设计师** + +|昵称|贡献内容| +|:-:|-| +|[Zoey]()|主题图标调整定稿| +|冰冰|主题图标初稿| + +## License +[MIT](https://github.com/recoluan/vuepress-theme-reco/blob/master/LICENSE) diff --git a/docs/.vuepress/theme/vuepress-theme-reco/components/.DS_Store b/docs/.vuepress/theme/vuepress-theme-reco/components/.DS_Store new file mode 100644 index 0000000000..f5d0703465 Binary files /dev/null and b/docs/.vuepress/theme/vuepress-theme-reco/components/.DS_Store differ diff --git a/docs/.vuepress/theme/vuepress-theme-reco/components/AlgoliaSearchBox.vue b/docs/.vuepress/theme/vuepress-theme-reco/components/AlgoliaSearchBox.vue new file mode 100644 index 0000000000..8ed69ae097 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/components/AlgoliaSearchBox.vue @@ -0,0 +1,172 @@ + + + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/components/Common.vue b/docs/.vuepress/theme/vuepress-theme-reco/components/Common.vue new file mode 100644 index 0000000000..2d1bb93b4a --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/components/Common.vue @@ -0,0 +1,209 @@ + + + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/components/DropdownLink.vue b/docs/.vuepress/theme/vuepress-theme-reco/components/DropdownLink.vue new file mode 100644 index 0000000000..8045c0c79e --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/components/DropdownLink.vue @@ -0,0 +1,182 @@ + + + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/components/DropdownTransition.vue b/docs/.vuepress/theme/vuepress-theme-reco/components/DropdownTransition.vue new file mode 100644 index 0000000000..8c711a155b --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/components/DropdownTransition.vue @@ -0,0 +1,33 @@ + + + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/components/Footer.vue b/docs/.vuepress/theme/vuepress-theme-reco/components/Footer.vue new file mode 100644 index 0000000000..a9f0d44592 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/components/Footer.vue @@ -0,0 +1,92 @@ + + + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/components/FriendLink.vue b/docs/.vuepress/theme/vuepress-theme-reco/components/FriendLink.vue new file mode 100644 index 0000000000..073c0431b3 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/components/FriendLink.vue @@ -0,0 +1,225 @@ + + + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/components/Home.vue b/docs/.vuepress/theme/vuepress-theme-reco/components/Home.vue new file mode 100644 index 0000000000..6ffb3aaf16 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/components/Home.vue @@ -0,0 +1,193 @@ + + + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/components/HomeBlog.vue b/docs/.vuepress/theme/vuepress-theme-reco/components/HomeBlog.vue new file mode 100644 index 0000000000..3ac9656708 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/components/HomeBlog.vue @@ -0,0 +1,349 @@ + + + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/components/MobShare.vue b/docs/.vuepress/theme/vuepress-theme-reco/components/MobShare.vue new file mode 100644 index 0000000000..afb81e4a0c --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/components/MobShare.vue @@ -0,0 +1,30 @@ + + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/components/Mode/ModePicker.vue b/docs/.vuepress/theme/vuepress-theme-reco/components/Mode/ModePicker.vue new file mode 100644 index 0000000000..a9ece2fad5 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/components/Mode/ModePicker.vue @@ -0,0 +1,100 @@ + + + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/components/Mode/applyMode.js b/docs/.vuepress/theme/vuepress-theme-reco/components/Mode/applyMode.js new file mode 100644 index 0000000000..44d9a897b0 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/components/Mode/applyMode.js @@ -0,0 +1,39 @@ +import modeOptions from './modeOptions' + +function render (mode) { + const rootElement = document.querySelector(':root') + const options = modeOptions[mode] + const opposite = mode === 'dark' ? 'light' : 'dark' + + for (const k in options) { + rootElement.style.setProperty(k, options[k]) + } + + rootElement.classList.remove(opposite) + rootElement.classList.add(mode) +} + +/** + * Sets a color scheme for the website. + * If browser supports "prefers-color-scheme", 'auto' mode will respect the setting for light or dark mode + * otherwise it will set a dark theme during night time + */ +export default function applyMode (mode) { + if (mode !== 'auto') { + render(mode) + return + } + + const isDarkMode = window.matchMedia('(prefers-color-scheme: dark)').matches + const isLightMode = window.matchMedia('(prefers-color-scheme: light)').matches + + if (isDarkMode) render('dark') + if (isLightMode) render('light') + + if (!isDarkMode && !isLightMode) { + console.log('You specified no preference for a color scheme or your browser does not support it. I schedule dark mode during night time.') + const hour = new Date().getHours() + if (hour < 6 || hour >= 18) render('dark') + else render('light') + } +} diff --git a/docs/.vuepress/theme/vuepress-theme-reco/components/Mode/index.vue b/docs/.vuepress/theme/vuepress-theme-reco/components/Mode/index.vue new file mode 100755 index 0000000000..ec86ab83d2 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/components/Mode/index.vue @@ -0,0 +1,105 @@ + + + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/components/Mode/modeOptions.js b/docs/.vuepress/theme/vuepress-theme-reco/components/Mode/modeOptions.js new file mode 100644 index 0000000000..bb30b0f5b1 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/components/Mode/modeOptions.js @@ -0,0 +1,44 @@ +const modeOptions = { + light: { + '--default-color-10': 'rgba(255, 255, 255, 1)', + '--default-color-9': 'rgba(255, 255, 255, .9)', + '--default-color-8': 'rgba(255, 255, 255, .8)', + '--default-color-7': 'rgba(255, 255, 255, .7)', + '--default-color-6': 'rgba(255, 255, 255, .6)', + '--default-color-5': 'rgba(255, 255, 255, .5)', + '--default-color-4': 'rgba(255, 255, 255, .4)', + '--default-color-3': 'rgba(255, 255, 255, .3)', + '--default-color-2': 'rgba(255, 255, 255, .2)', + '--default-color-1': 'rgba(255, 255, 255, .1)', + '--background-color': '#fff', + '--box-shadow': '0 1px 8px 0 rgba(0, 0, 0, 0.1)', + '--box-shadow-hover': '0 2px 16px 0 rgba(0, 0, 0, 0.2)', + '--text-color': '#242424', + '--text-color-sub': '#7F7F7F', + '--border-color': '#eaecef', + '--code-color': 'rgba(27, 31, 35, 0.05)', + '--mask-color': '#888' + }, + dark: { + '--default-color-10': 'rgba(0, 0, 0, 1)', + '--default-color-9': 'rgba(0, 0, 0, .9)', + '--default-color-8': 'rgba(0, 0, 0, .8)', + '--default-color-7': 'rgba(0, 0, 0, .7)', + '--default-color-6': 'rgba(0, 0, 0, .6)', + '--default-color-5': 'rgba(0, 0, 0, .5)', + '--default-color-4': 'rgba(0, 0, 0, .4)', + '--default-color-3': 'rgba(0, 0, 0, .3)', + '--default-color-2': 'rgba(0, 0, 0, .2)', + '--default-color-1': 'rgba(0, 0, 0, .1)', + '--background-color': '#181818', + '--box-shadow': '0 1px 8px 0 rgba(0, 0, 0, .6)', + '--box-shadow-hover': '0 2px 16px 0 rgba(0, 0, 0, .7)', + '--text-color': 'rgba(255, 255, 255, .8)', + '--text-color-sub': '#8B8B8B', + '--border-color': 'rgba(0, 0, 0, .3)', + '--code-color': 'rgba(0, 0, 0, .3)', + '--mask-color': '#000' + } +} + +export default modeOptions diff --git a/docs/.vuepress/theme/vuepress-theme-reco/components/NavLink.vue b/docs/.vuepress/theme/vuepress-theme-reco/components/NavLink.vue new file mode 100644 index 0000000000..af7fb13c4c --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/components/NavLink.vue @@ -0,0 +1,55 @@ + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/components/NavLinks.vue b/docs/.vuepress/theme/vuepress-theme-reco/components/NavLinks.vue new file mode 100644 index 0000000000..e730f110f3 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/components/NavLinks.vue @@ -0,0 +1,188 @@ + + + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/components/Navbar.vue b/docs/.vuepress/theme/vuepress-theme-reco/components/Navbar.vue new file mode 100644 index 0000000000..8cbb639028 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/components/Navbar.vue @@ -0,0 +1,150 @@ + + + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/components/NoteAbstract.vue b/docs/.vuepress/theme/vuepress-theme-reco/components/NoteAbstract.vue new file mode 100644 index 0000000000..18e1449ff8 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/components/NoteAbstract.vue @@ -0,0 +1,31 @@ + + + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/components/NoteAbstractItem.vue b/docs/.vuepress/theme/vuepress-theme-reco/components/NoteAbstractItem.vue new file mode 100644 index 0000000000..9b5197d1cf --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/components/NoteAbstractItem.vue @@ -0,0 +1,91 @@ + + + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/components/Page.vue b/docs/.vuepress/theme/vuepress-theme-reco/components/Page.vue new file mode 100644 index 0000000000..d551078f11 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/components/Page.vue @@ -0,0 +1,325 @@ + + + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/components/PageInfo.vue b/docs/.vuepress/theme/vuepress-theme-reco/components/PageInfo.vue new file mode 100644 index 0000000000..12c31bd688 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/components/PageInfo.vue @@ -0,0 +1,104 @@ + + + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/components/Password.vue b/docs/.vuepress/theme/vuepress-theme-reco/components/Password.vue new file mode 100644 index 0000000000..d8f99f683c --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/components/Password.vue @@ -0,0 +1,326 @@ + + + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/components/PersonalInfo.vue b/docs/.vuepress/theme/vuepress-theme-reco/components/PersonalInfo.vue new file mode 100644 index 0000000000..aa9ebf57fd --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/components/PersonalInfo.vue @@ -0,0 +1,117 @@ + + + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/components/SearchBox.vue b/docs/.vuepress/theme/vuepress-theme-reco/components/SearchBox.vue new file mode 100644 index 0000000000..3e9fc06e93 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/components/SearchBox.vue @@ -0,0 +1,241 @@ + + + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/components/Sidebar.vue b/docs/.vuepress/theme/vuepress-theme-reco/components/Sidebar.vue new file mode 100644 index 0000000000..4aaae5d86a --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/components/Sidebar.vue @@ -0,0 +1,68 @@ + + + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/components/SidebarButton.vue b/docs/.vuepress/theme/vuepress-theme-reco/components/SidebarButton.vue new file mode 100644 index 0000000000..b8fb4150bd --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/components/SidebarButton.vue @@ -0,0 +1,27 @@ + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/components/SidebarGroup.vue b/docs/.vuepress/theme/vuepress-theme-reco/components/SidebarGroup.vue new file mode 100644 index 0000000000..d82d809b07 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/components/SidebarGroup.vue @@ -0,0 +1,131 @@ + + + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/components/SidebarLink.vue b/docs/.vuepress/theme/vuepress-theme-reco/components/SidebarLink.vue new file mode 100644 index 0000000000..9d3b49cca2 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/components/SidebarLink.vue @@ -0,0 +1,97 @@ + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/components/SidebarLinks.vue b/docs/.vuepress/theme/vuepress-theme-reco/components/SidebarLinks.vue new file mode 100644 index 0000000000..30522ceb5a --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/components/SidebarLinks.vue @@ -0,0 +1,141 @@ + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/components/SubSidebar.vue b/docs/.vuepress/theme/vuepress-theme-reco/components/SubSidebar.vue new file mode 100644 index 0000000000..402161daaa --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/components/SubSidebar.vue @@ -0,0 +1,76 @@ + + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/components/TagList.vue b/docs/.vuepress/theme/vuepress-theme-reco/components/TagList.vue new file mode 100644 index 0000000000..5e593fdb76 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/components/TagList.vue @@ -0,0 +1,57 @@ + + + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/enhanceApp.js b/docs/.vuepress/theme/vuepress-theme-reco/enhanceApp.js new file mode 100644 index 0000000000..eae579f4af --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/enhanceApp.js @@ -0,0 +1,24 @@ +/* eslint-disable no-proto */ +import postMixin from '@theme/mixins/posts' +import localMixin from '@theme/mixins/locales' +import { addLinkToHead, addScriptToHead } from '@theme/helpers/utils' +import { registerCodeThemeCss, interceptRouterError } from '@theme/helpers/other' +import VueCompositionAPI from '@vue/composition-api' + +export default ({ + Vue, + siteData, + isServer, + router +}) => { + Vue.use(VueCompositionAPI) + Vue.mixin(postMixin) + Vue.mixin(localMixin) + if (!isServer) { + addLinkToHead('//at.alicdn.com/t/font_1030519_2ciwdtb4x65.css') + addScriptToHead('//kit.fontawesome.com/51b01de608.js') + registerCodeThemeCss(siteData.themeConfig.codeTheme) + } + + interceptRouterError(router) +} diff --git a/docs/.vuepress/theme/vuepress-theme-reco/global-components/Badge.vue b/docs/.vuepress/theme/vuepress-theme-reco/global-components/Badge.vue new file mode 100644 index 0000000000..dba9e79db3 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/global-components/Badge.vue @@ -0,0 +1,44 @@ + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/helpers/other.js b/docs/.vuepress/theme/vuepress-theme-reco/helpers/other.js new file mode 100644 index 0000000000..ed31239c19 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/helpers/other.js @@ -0,0 +1,36 @@ +/* eslint-disable no-proto */ +import { addLinkToHead } from './utils' +export function getOneColor () { + const tagColorArr = [ + '#e15b64', + '#f47e60', + '#f8b26a', + '#abbd81', + '#849b87', + '#e15b64', + '#f47e60', + '#f8b26a', + '#f26d6d', + '#67cc86', + '#fb9b5f', + '#3498db' + ] + const index = Math.floor(Math.random() * tagColorArr.length) + return tagColorArr[index] +} + +export function registerCodeThemeCss (theme = 'tomorrow') { + const themeArr = ['tomorrow', 'funky', 'okaidia', 'solarizedlight', 'default'] + const href = `//prismjs.com/themes/prism${themeArr.indexOf(theme) > -1 ? `-${theme}` : ''}.css` + + addLinkToHead(href) +} + +export function interceptRouterError (router) { + // 获取原型对象上的 push 函数 + const originalPush = router.__proto__.push + // 修改原型对象中的p ush 方法 + router.__proto__.push = function push (location) { + return originalPush.call(this, location).catch(err => err) + } +} diff --git a/docs/.vuepress/theme/vuepress-theme-reco/helpers/postData.js b/docs/.vuepress/theme/vuepress-theme-reco/helpers/postData.js new file mode 100644 index 0000000000..13c3f356d0 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/helpers/postData.js @@ -0,0 +1,41 @@ +import { compareDate } from '@theme/helpers/utils' + +// 过滤博客数据 +export function filterPosts (posts, isTimeline) { + posts = posts.filter((item, index) => { + const { title, frontmatter: { home, date, publish } } = item + // 过滤多个分类时产生的重复数据 + if (posts.indexOf(item) !== index) { + return false + } else { + const someConditions = home == true || title == undefined || publish === false + const boo = isTimeline === true + ? !(someConditions || date === undefined) + : !someConditions + return boo + } + }) + return posts +} + +// 排序博客数据 +export function sortPostsByStickyAndDate (posts) { + posts.sort((prev, next) => { + const prevSticky = prev.frontmatter.sticky + const nextSticky = next.frontmatter.sticky + if (prevSticky && nextSticky) { + return prevSticky == nextSticky ? compareDate(prev, next) : (prevSticky - nextSticky) + } else if (prevSticky && !nextSticky) { + return -1 + } else if (!prevSticky && nextSticky) { + return 1 + } + return compareDate(prev, next) + }) +} + +export function sortPostsByDate (posts) { + posts.sort((prev, next) => { + return compareDate(prev, next) + }) +} diff --git a/docs/.vuepress/theme/vuepress-theme-reco/helpers/utils.js b/docs/.vuepress/theme/vuepress-theme-reco/helpers/utils.js new file mode 100644 index 0000000000..35db33c996 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/helpers/utils.js @@ -0,0 +1,261 @@ +export const hashRE = /#.*$/ +export const extRE = /\.(md|html)$/ +export const endingSlashRE = /\/$/ +export const outboundRE = /^(https?:|mailto:|tel:)/ + +export function normalize (path) { + return decodeURI(path) + .replace(hashRE, '') + .replace(extRE, '') +} + +export function getHash (path) { + const match = path.match(hashRE) + if (match) { + return match[0] + } +} + +export function isExternal (path) { + return outboundRE.test(path) +} + +export function isMailto (path) { + return /^mailto:/.test(path) +} + +export function isTel (path) { + return /^tel:/.test(path) +} + +export function ensureExt (path) { + if (isExternal(path)) { + return path + } + const hashMatch = path.match(hashRE) + const hash = hashMatch ? hashMatch[0] : '' + const normalized = normalize(path) + + if (endingSlashRE.test(normalized)) { + return path + } + return normalized + '.html' + hash +} + +export function isActive (route, path) { + const routeHash = route.hash + const linkHash = getHash(path) + if (linkHash && routeHash !== linkHash) { + return false + } + const routePath = normalize(route.path) + const pagePath = normalize(path) + return routePath === pagePath +} + +export function resolvePage (pages, rawPath, base) { + if (base) { + rawPath = resolvePath(rawPath, base) + } + const path = normalize(rawPath) + for (let i = 0; i < pages.length; i++) { + if (normalize(pages[i].regularPath) === path) { + return Object.assign({}, pages[i], { + type: 'page', + path: ensureExt(pages[i].path) + }) + } + } + console.error(`[vuepress] No matching page found for sidebar item "${rawPath}"`) + return {} +} + +function resolvePath (relative, base, append) { + const firstChar = relative.charAt(0) + if (firstChar === '/') { + return relative + } + + if (firstChar === '?' || firstChar === '#') { + return base + relative + } + + const stack = base.split('/') + + // remove trailing segment if: + // - not appending + // - appending to trailing slash (last segment is empty) + if (!append || !stack[stack.length - 1]) { + stack.pop() + } + + // resolve relative path + const segments = relative.replace(/^\//, '').split('/') + for (let i = 0; i < segments.length; i++) { + const segment = segments[i] + if (segment === '..') { + stack.pop() + } else if (segment !== '.') { + stack.push(segment) + } + } + + // ensure leading slash + if (stack[0] !== '') { + stack.unshift('') + } + + return stack.join('/') +} + +/** + * @param { Page } page + * @param { string } regularPath + * @param { SiteData } site + * @param { string } localePath + * @returns { SidebarGroup } + */ +export function resolveSidebarItems (page, regularPath, site, localePath) { + const { pages, themeConfig } = site + + const localeConfig = localePath && themeConfig.locales + ? themeConfig.locales[localePath] || themeConfig + : themeConfig + + const sidebarConfig = localeConfig.sidebar || themeConfig.sidebar + + const { base, config } = resolveMatchingConfig(regularPath, sidebarConfig) + return config + ? config.map(item => resolveItem(item, pages, base)) + : [] +} + +export function groupHeaders (headers) { + // group h3s under h2 + headers = headers.map(h => Object.assign({}, h)) + let lastH2 + headers.forEach(h => { + if (h.level === 2) { + lastH2 = h + } else if (lastH2) { + (lastH2.children || (lastH2.children = [])).push(h) + } + }) + return headers.filter(h => h.level === 2) +} + +export function resolveNavLinkItem (linkItem) { + return Object.assign(linkItem, { + type: linkItem.items && linkItem.items.length ? 'links' : 'link' + }) +} + +/** + * @param { Route } route + * @param { Array | Array | [link: string]: SidebarConfig } config + * @returns { base: string, config: SidebarConfig } + */ +export function resolveMatchingConfig (regularPath, config) { + if (Array.isArray(config)) { + return { + base: '/', + config: config + } + } + for (const base in config) { + if (ensureEndingSlash(regularPath).indexOf(encodeURI(base)) === 0) { + return { + base, + config: config[base] + } + } + } + return {} +} + +export function formatDate (time, fmt = 'yyyy-MM-dd hh:mm:ss') { + time = time.replace(/-/g, '/') + const date = new Date(time) + if (/(y+)/.test(fmt)) { + fmt = fmt.replace(RegExp.$1, date.getFullYear() + '').substr(4 - RegExp.$1.length) + } + + const o = { + 'M+': date.getMonth() + 1, + 'd+': date.getDate(), + 'h+': date.getHours(), + 'm+': date.getMinutes(), + 's+': date.getSeconds() + } + + for (const key in o) { + if (RegExp(`(${key})`).test(fmt)) { + const str = o[key] + '' + fmt = fmt.replace(RegExp.$1, str.length === 2 ? str : '0' + str) + } + } + return fmt +} + +// 获取时间的数字类型 +export function getTimeNum (date) { + return new Date(date.frontmatter.date).getTime() +} + +// 比对时间 +export function compareDate (a, b) { + return getTimeNum(b) - getTimeNum(a) +} + +// 向 head 中添加 style +export function addLinkToHead (href) { + const iconLink = document.createElement('link') + iconLink.rel = 'stylesheet' + iconLink.href = href + + document.head.append(iconLink) +} + +// 向 head 中添加 script +export function addScriptToHead (href) { + const iconLink = document.createElement('script') + iconLink.src = href + + document.head.append(iconLink) +} + +function ensureEndingSlash (path) { + return /(\.html|\/)$/.test(path) + ? path + : path + '/' +} + +function resolveItem (item, pages, base, groupDepth = 1) { + if (typeof item === 'string') { + return resolvePage(pages, item, base) + } else if (Array.isArray(item)) { + return Object.assign(resolvePage(pages, item[0], base), { + title: item[1] + }) + } else { + if (groupDepth > 3) { + console.error( + '[vuepress] detected a too deep nested sidebar group.' + ) + } + const children = item.children || [] + if (children.length === 0 && item.path) { + return Object.assign(resolvePage(pages, item.path, base), { + title: item.title + }) + } + return { + type: 'group', + path: item.path, + title: item.title, + sidebarDepth: item.sidebarDepth, + children: children.map(child => resolveItem(child, pages, base, groupDepth + 1)), + collapsable: item.collapsable !== false + } + } +} diff --git a/docs/.vuepress/theme/vuepress-theme-reco/images/bg.svg b/docs/.vuepress/theme/vuepress-theme-reco/images/bg.svg new file mode 100644 index 0000000000..f576b34ae4 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/images/bg.svg @@ -0,0 +1,39 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/images/home-bg.jpg b/docs/.vuepress/theme/vuepress-theme-reco/images/home-bg.jpg new file mode 100644 index 0000000000..9926b804fa Binary files /dev/null and b/docs/.vuepress/theme/vuepress-theme-reco/images/home-bg.jpg differ diff --git a/docs/.vuepress/theme/vuepress-theme-reco/images/home-head.png b/docs/.vuepress/theme/vuepress-theme-reco/images/home-head.png new file mode 100644 index 0000000000..4f1eb9fd6c Binary files /dev/null and b/docs/.vuepress/theme/vuepress-theme-reco/images/home-head.png differ diff --git a/docs/.vuepress/theme/vuepress-theme-reco/images/icon_vuepress_reco.png b/docs/.vuepress/theme/vuepress-theme-reco/images/icon_vuepress_reco.png new file mode 100644 index 0000000000..ac01252905 Binary files /dev/null and b/docs/.vuepress/theme/vuepress-theme-reco/images/icon_vuepress_reco.png differ diff --git a/docs/.vuepress/theme/vuepress-theme-reco/index.js b/docs/.vuepress/theme/vuepress-theme-reco/index.js new file mode 100644 index 0000000000..7f1abf761c --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/index.js @@ -0,0 +1,96 @@ +const path = require('path') + +// Theme API. +module.exports = (options, ctx) => ({ + alias () { + const { themeConfig, siteConfig } = ctx + // resolve algolia + const isAlgoliaSearch = ( + themeConfig.algolia || + Object.keys(siteConfig.locales && themeConfig.locales || {}) + .some(base => themeConfig.locales[base].algolia) + ) + return { + '@AlgoliaSearchBox': isAlgoliaSearch + ? path.resolve(__dirname, 'components/AlgoliaSearchBox.vue') + : path.resolve(__dirname, 'noopModule.js'), + '@SearchBox': path.resolve(__dirname, 'components/SearchBox.vue') + } + }, + + plugins: [ + '@vuepress-reco/back-to-top', + '@vuepress-reco/loading-page', + '@vuepress-reco/pagation', + '@vuepress-reco/comments', + '@vuepress/active-header-links', + ['@vuepress/medium-zoom', { + selector: '.theme-reco-content :not(a) > img' + }], + '@vuepress/plugin-nprogress', + ['@vuepress/plugin-blog', { + permalink: '/:regular', + frontmatters: [ + { + id: 'tags', + keys: ['tags'], + path: '/tag/', + layout: 'Tags', + scopeLayout: 'Tag' + }, + { + id: 'categories', + keys: ['categories'], + path: '/categories/', + layout: 'Categories', + scopeLayout: 'Category' + }, + { + id: 'timeline', + keys: ['timeline'], + path: '/timeline/', + layout: 'TimeLines', + scopeLayout: 'TimeLine' + } + ] + }], + 'vuepress-plugin-smooth-scroll', + ['container', { + type: 'tip', + before: info => `

${info}

`, + after: '
', + defaultTitle: '' + }], + ['container', { + type: 'warning', + before: info => `

${info}

`, + after: '
', + defaultTitle: '' + }], + ['container', { + type: 'danger', + before: info => `

${info}

`, + after: '
', + defaultTitle: '' + }], + ['container', { + type: 'right', + defaultTitle: '' + }], + ['container', { + type: 'theorem', + before: info => `

${info}

`, + after: '
', + defaultTitle: '' + }], + ['container', { + type: 'details', + before: info => `
${info ? `${info}` : ''}\n`, + after: () => '
\n', + defaultTitle: { + '/': 'See More', + '/zh/': '更多' + } + }] + ] +}) diff --git a/docs/.vuepress/theme/vuepress-theme-reco/layouts/404.vue b/docs/.vuepress/theme/vuepress-theme-reco/layouts/404.vue new file mode 100644 index 0000000000..305da8741f --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/layouts/404.vue @@ -0,0 +1,69 @@ + + + + + + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/layouts/Category.vue b/docs/.vuepress/theme/vuepress-theme-reco/layouts/Category.vue new file mode 100644 index 0000000000..8431c9be37 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/layouts/Category.vue @@ -0,0 +1,167 @@ + + + + + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/layouts/Layout.vue b/docs/.vuepress/theme/vuepress-theme-reco/layouts/Layout.vue new file mode 100644 index 0000000000..469e2e8614 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/layouts/Layout.vue @@ -0,0 +1,41 @@ + + + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/layouts/Tag.vue b/docs/.vuepress/theme/vuepress-theme-reco/layouts/Tag.vue new file mode 100644 index 0000000000..8999109505 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/layouts/Tag.vue @@ -0,0 +1,112 @@ + + + + + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/layouts/Tags.vue b/docs/.vuepress/theme/vuepress-theme-reco/layouts/Tags.vue new file mode 100644 index 0000000000..2544b9ae5c --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/layouts/Tags.vue @@ -0,0 +1,103 @@ + + + + + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/layouts/TimeLines.vue b/docs/.vuepress/theme/vuepress-theme-reco/layouts/TimeLines.vue new file mode 100644 index 0000000000..80d6463613 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/layouts/TimeLines.vue @@ -0,0 +1,153 @@ + + + + + + + diff --git a/docs/.vuepress/theme/vuepress-theme-reco/lib/vuepress-theme-reco.js b/docs/.vuepress/theme/vuepress-theme-reco/lib/vuepress-theme-reco.js new file mode 100644 index 0000000000..ee7a9c3ca6 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/lib/vuepress-theme-reco.js @@ -0,0 +1,7 @@ +'use strict' + +module.exports = vuepressThemeReco + +function vuepressThemeReco () { + // TODO +} diff --git a/docs/.vuepress/theme/vuepress-theme-reco/locales/en.js b/docs/.vuepress/theme/vuepress-theme-reco/locales/en.js new file mode 100644 index 0000000000..4bacdfde4f --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/locales/en.js @@ -0,0 +1,11 @@ +export default { + homeBlog: { + article: 'Article', + tag: 'Tag', + category: 'Category', + friendLink: 'Friend Link' + }, + tag: { + all: 'All' + } +} diff --git a/docs/.vuepress/theme/vuepress-theme-reco/locales/index.js b/docs/.vuepress/theme/vuepress-theme-reco/locales/index.js new file mode 100644 index 0000000000..05f2219102 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/locales/index.js @@ -0,0 +1,7 @@ +import zhHans from './zh-hans.js' +import zhHant from './zh-hant.js' +import en from './en.js' +import ja from './ja.js' +import ko from './ko.js' + +export { zhHans, zhHant, en, ja, ko } diff --git a/docs/.vuepress/theme/vuepress-theme-reco/locales/ja.js b/docs/.vuepress/theme/vuepress-theme-reco/locales/ja.js new file mode 100644 index 0000000000..32b6a12528 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/locales/ja.js @@ -0,0 +1,11 @@ +export default { + homeBlog: { + article: '文章', + tag: 'ラベル', + category: '分類', + friendLink: '友情リンク' + }, + tag: { + all: '全部' + } +} diff --git a/docs/.vuepress/theme/vuepress-theme-reco/locales/ko.js b/docs/.vuepress/theme/vuepress-theme-reco/locales/ko.js new file mode 100644 index 0000000000..b65bfbde91 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/locales/ko.js @@ -0,0 +1,11 @@ +export default { + homeBlog: { + article: '글', + tag: '태그', + category: '분류', + friendLink: '링크 참조' + }, + tag: { + all: '전체' + } +} diff --git a/docs/.vuepress/theme/vuepress-theme-reco/locales/zh-hans.js b/docs/.vuepress/theme/vuepress-theme-reco/locales/zh-hans.js new file mode 100644 index 0000000000..6a23c3ab5c --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/locales/zh-hans.js @@ -0,0 +1,11 @@ +export default { + homeBlog: { + article: '文章', + tag: '标签', + category: '分类', + friendLink: '友情链接' + }, + tag: { + all: '全部' + } +} diff --git a/docs/.vuepress/theme/vuepress-theme-reco/locales/zh-hant.js b/docs/.vuepress/theme/vuepress-theme-reco/locales/zh-hant.js new file mode 100644 index 0000000000..d4aa243889 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/locales/zh-hant.js @@ -0,0 +1,11 @@ +export default { + homeBlog: { + article: '文章', + tag: '標簽', + category: '分類', + friendLink: '友情鏈接' + }, + tag: { + all: '全部' + } +} diff --git a/docs/.vuepress/theme/vuepress-theme-reco/mixins/locales.js b/docs/.vuepress/theme/vuepress-theme-reco/mixins/locales.js new file mode 100644 index 0000000000..21509caabd --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/mixins/locales.js @@ -0,0 +1,23 @@ +import { zhHans, zhHant, en, ja, ko } from '../locales/index' + +export default { + computed: { + $recoLocales () { + const recoLocales = this.$themeLocaleConfig.recoLocales || {} + + if (/^zh\-(CN|SG)$/.test(this.$lang)) { + return { ...zhHans, ...recoLocales } + } + if (/^zh\-(HK|MO|TW)$/.test(this.$lang)) { + return { ...zhHant, ...recoLocales } + } + if (/^ja\-JP$/.test(this.$lang)) { + return { ...ja, ...recoLocales } + } + if (/^ko\-KR$/.test(this.$lang)) { + return { ...ko, ...recoLocales } + } + return { ...en, ...recoLocales } + } + } +} diff --git a/docs/.vuepress/theme/vuepress-theme-reco/mixins/moduleTransiton.js b/docs/.vuepress/theme/vuepress-theme-reco/mixins/moduleTransiton.js new file mode 100644 index 0000000000..e77f7cc977 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/mixins/moduleTransiton.js @@ -0,0 +1,21 @@ +export default { + data () { + return { + recoShowModule: false + } + }, + mounted () { + this.recoShowModule = true + }, + watch: { + '$route' (newV, oldV) { + if (newV.path === oldV.path) return + + this.recoShowModule = false + + setTimeout(() => { + this.recoShowModule = true + }, 200) + } + } +} diff --git a/docs/.vuepress/theme/vuepress-theme-reco/mixins/pagination.js b/docs/.vuepress/theme/vuepress-theme-reco/mixins/pagination.js new file mode 100644 index 0000000000..dbefecc70c --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/mixins/pagination.js @@ -0,0 +1,21 @@ +export default { + methods: { + // 获取当前页码 + _getStoragePage () { + const path = window.location.pathname + const currentPage = JSON.parse(sessionStorage.getItem('currentPage')) + + if (currentPage === null || path !== currentPage.path) { + sessionStorage.setItem('currentPage', { page: 1, path: '' }) + return 1 + } + + return parseInt(currentPage.page) + }, + // 设置当前页码 + _setStoragePage (page) { + const path = window.location.pathname + sessionStorage.setItem('currentPage', JSON.stringify({ page, path })) + } + } +} diff --git a/docs/.vuepress/theme/vuepress-theme-reco/mixins/posts.js b/docs/.vuepress/theme/vuepress-theme-reco/mixins/posts.js new file mode 100644 index 0000000000..d248ceed15 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/mixins/posts.js @@ -0,0 +1,78 @@ +import { filterPosts, sortPostsByStickyAndDate, sortPostsByDate } from '../helpers/postData' + +export default { + computed: { + $recoPosts () { + const { + $categories: { list: articles } + } = this + + let posts = articles.reduce((allData, currentData) => { + return [...allData, ...currentData.pages] + }, []) + + posts = filterPosts(posts, false) + sortPostsByStickyAndDate(posts) + + return posts + }, + $recoPostsForTimeline () { + let pages = this.$recoPosts + const formatPages = {} + const formatPagesArr = [] + pages = filterPosts(pages, true) + this.pages = pages.length == 0 ? [] : pages + for (let i = 0, length = pages.length; i < length; i++) { + const page = pages[i] + const pageDateYear = dateFormat(page.frontmatter.date, 'year') + if (formatPages[pageDateYear]) formatPages[pageDateYear].push(page) + else { + formatPages[pageDateYear] = [page] + } + } + + for (const key in formatPages) { + const data = formatPages[key] + sortPostsByDate(data) + formatPagesArr.unshift({ + year: key, + data + }) + } + + return formatPagesArr + }, + $showSubSideBar () { + const { + $themeConfig: { subSidebar: themeSubSidebar, sidebar: themeSidebar }, + $frontmatter: { subSidebar: pageSubSidebar, sidebar: pageSidebar } + } = this + + const headers = this.$page.headers || [] + + if ([pageSubSidebar, pageSidebar].indexOf(false) > -1) { + return false + } else if ([pageSubSidebar, pageSidebar].indexOf('auto') > -1 && headers.length > 0) { + return true + } else if ([themeSubSidebar, themeSidebar].indexOf('auto') > -1 && headers.length > 0) { + return true + } else { + return false + } + } + } +} + +function renderTime (date) { + var dateee = new Date(date).toJSON() + return new Date(+new Date(dateee) + 8 * 3600 * 1000).toISOString().replace(/T/g, ' ').replace(/\.[\d]{3}Z/, '').replace(/-/g, '/') +} +function dateFormat (date, type) { + date = renderTime(date) + const dateObj = new Date(date) + const year = dateObj.getFullYear() + const mon = dateObj.getMonth() + 1 + const day = dateObj.getDate() + if (type == 'year') return year + else return `${mon}-${day}` +} diff --git a/docs/.vuepress/theme/vuepress-theme-reco/noopModule.js b/docs/.vuepress/theme/vuepress-theme-reco/noopModule.js new file mode 100644 index 0000000000..b1c6ea436a --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/noopModule.js @@ -0,0 +1 @@ +export default {} diff --git a/docs/.vuepress/theme/vuepress-theme-reco/package.json b/docs/.vuepress/theme/vuepress-theme-reco/package.json new file mode 100644 index 0000000000..95edd2f155 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/package.json @@ -0,0 +1,38 @@ +{ + "name": "vuepress-theme-reco", + "version": "1.6.1", + "description": "A simple and beautiful vuepress Blog & Doc theme.", + "keywords": [ + "vuepress", + "vue", + "theme" + ], + "homepage": "https://vuepress-theme-reco.recoluan.com/", + "bugs": { + "url": "https://github.com/vuepress-reco/vuepress-theme-reco/issues" + }, + "repository": { + "type": "git", + "url": "git+https://github.com/vuepress-reco/vuepress-theme-reco.git" + }, + "license": "MIT", + "author": "reco_luan", + "main": "index.js", + "dependencies": { + "@vue/composition-api": "^1.0.0-beta.20", + "@vuepress-reco/core": "^1.6.0", + "@vuepress-reco/vuepress-plugin-back-to-top": "^1.6.0", + "@vuepress-reco/vuepress-plugin-comments": "^1.6.0", + "@vuepress-reco/vuepress-plugin-loading-page": "^1.6.0", + "@vuepress-reco/vuepress-plugin-pagation": "^1.6.0", + "@vuepress/plugin-blog": "1.9.2", + "@vuepress/plugin-medium-zoom": "1.5.0", + "docsearch.js": "2.6.3", + "md5": "2.2.1", + "vue-click-outside": "1.1.0", + "vuepress-plugin-smooth-scroll": "^0.0.9" + }, + "gitHead": "c3a1db8487bece9341b6269a0f6ebf910cd5462a", + "_from": "vuepress-theme-reco@0.2.1", + "_resolved": "http://registry.npm.taobao.org/vuepress-theme-reco/download/vuepress-theme-reco-0.2.1.tgz" +} diff --git a/docs/.vuepress/theme/vuepress-theme-reco/styles/arrow.styl b/docs/.vuepress/theme/vuepress-theme-reco/styles/arrow.styl new file mode 100644 index 0000000000..d116e1cc3d --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/styles/arrow.styl @@ -0,0 +1,22 @@ +@require './config' + +.arrow + display inline-block + width 0 + height 0 + &.up + border-left 4px solid transparent + border-right 4px solid transparent + border-bottom 6px solid var(--text-color-sub) + &.down + border-left 4px solid transparent + border-right 4px solid transparent + border-top 6px solid var(--text-color-sub) + &.right + border-top 4px solid transparent + border-bottom 4px solid transparent + border-left 6px solid var(--text-color-sub) + &.left + border-top 4px solid transparent + border-bottom 4px solid transparent + border-right 6px solid var(--text-color-sub) diff --git a/docs/.vuepress/theme/vuepress-theme-reco/styles/code.styl b/docs/.vuepress/theme/vuepress-theme-reco/styles/code.styl new file mode 100644 index 0000000000..b8551c3b6b --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/styles/code.styl @@ -0,0 +1,135 @@ +.content__default + code + color lighten($textColor, 20%) + padding 0.25rem 0.5rem + margin 0 + font-size 0.85em + background-color var(--code-color) + border-radius 3px + .token + &.deleted + color #EC5975 + &.inserted + color $accentColor + +.content__default + pre, pre[class*="language-"] + line-height 1.4 + padding 1.25rem 1.5rem + margin 0.85rem 0 + background-color $codeBgColor + border-radius 6px + overflow auto + code + color #fff + padding 0 + background-color transparent + border-radius 0 + +div[class*="language-"] + position relative + background-color $codeBgColor + border-radius 6px + .highlight-lines + user-select none + padding-top 1.3rem + position absolute + top 0 + left 0 + width 100% + line-height 1.4 + .highlighted + background-color rgba(0, 0, 0, 66%) + pre, pre[class*="language-"] + background transparent + position relative + z-index 1 + &::before + position absolute + z-index 3 + top 0.8em + right 1em + font-size 0.75rem + color rgba(255, 255, 255, 0.4) + &:not(.line-numbers-mode) + .line-numbers-wrapper + display none + &.line-numbers-mode + .highlight-lines .highlighted + position relative + &:before + content ' ' + position absolute + z-index 3 + left 0 + top 0 + display block + width $lineNumbersWrapperWidth + height 100% + background-color rgba(0, 0, 0, 66%) + pre + padding-left $lineNumbersWrapperWidth + 1 rem + vertical-align middle + .line-numbers-wrapper + position absolute + top 0 + width $lineNumbersWrapperWidth + text-align center + color rgba(255, 255, 255, 0.3) + padding 1.25rem 0 + line-height 1.4 + br + user-select none + .line-number + position relative + z-index 4 + user-select none + font-size 0.85em + &::after + content '' + position absolute + z-index 2 + top 0 + left 0 + width $lineNumbersWrapperWidth + height 100% + border-radius 6px 0 0 6px + border-right 1px solid rgba(0, 0, 0, 66%) + background-color $codeBgColor + + +for lang in $codeLang + div{'[class~="language-' + lang + '"]'} + &:before + content ('' + lang) + +div[class~="language-javascript"] + &:before + content "js" + +div[class~="language-typescript"] + &:before + content "ts" + +div[class~="language-markup"] + &:before + content "html" + +div[class~="language-markdown"] + &:before + content "md" + +div[class~="language-json"]:before + content "json" + +div[class~="language-ruby"]:before + content "rb" + +div[class~="language-python"]:before + content "py" + +div[class~="language-bash"]:before + content "sh" + +div[class~="language-php"]:before + content "php" diff --git a/docs/.vuepress/theme/vuepress-theme-reco/styles/custom-blocks.styl b/docs/.vuepress/theme/vuepress-theme-reco/styles/custom-blocks.styl new file mode 100644 index 0000000000..f79e79f5c5 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/styles/custom-blocks.styl @@ -0,0 +1,51 @@ +.custom-block + .custom-block-title + font-weight 600 + margin-bottom -0.4rem + &.tip, &.warning, &.danger + padding .1rem 1.5rem + border-left-width .5rem + border-left-style solid + margin 1rem 0 + &.tip + background-color var(--code-color) + border-color #67cc86 + .title + color #67cc86 + &.warning + background-color var(--code-color) + border-color #fb9b5f + .title + color #fb9b5f + &.danger + background-color var(--code-color) + border-color #f26d6d + .title + color #f26d6d + &.right + color transparentify($textColor, 0.4) + font-size 0.9rem + text-align right + &.theorem + margin 1rem 0 + padding .1rem 1.5rem + border-radius 0.4rem + background-color var(--code-color) + .title + font-weight bold + &.details + display block + position relative + border-radius 2px + margin 1em 0 + padding 1rem + background-color var(--code-color) + h4 + margin-top 0 + figure, p + &:last-child + margin-bottom 0 + padding-bottom 0 + summary + outline none + cursor pointer diff --git a/docs/.vuepress/theme/vuepress-theme-reco/styles/mobile.styl b/docs/.vuepress/theme/vuepress-theme-reco/styles/mobile.styl new file mode 100644 index 0000000000..cb44a7d744 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/styles/mobile.styl @@ -0,0 +1,39 @@ +// @require './config' + +$mobileSidebarWidth = $sidebarWidth * 0.82 + +// narrow desktop / iPad +@media (max-width: $MQNarrow) + .sidebar + font-size 15px + width $mobileSidebarWidth + .page, .password-wrapper-in + margin-left $mobileSidebarWidth + +// wide mobile +@media (max-width: $MQMobile) + .sidebar + top 0 + padding-top $navbarHeight + transform translateX(-100%) + transition transform .2s ease + .page, .password-wrapper-in + margin-left 0 + .theme-container + &.sidebar-open + .sidebar + transform translateX(0) + &.no-navbar + .sidebar + padding-top: 0 + .password-shadow + padding-left 0 + +// narrow mobile +@media (max-width: $MQMobileNarrow) + h1 + font-size 1.9rem + .content__default + div[class*="language-"] + margin 0.85rem -1.5rem + border-radius 0 diff --git a/docs/.vuepress/theme/vuepress-theme-reco/styles/palette.styl b/docs/.vuepress/theme/vuepress-theme-reco/styles/palette.styl new file mode 100644 index 0000000000..ea6620af89 --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/styles/palette.styl @@ -0,0 +1,53 @@ +$darkColor10 = rgba(0, 0, 0, 1) +$darkColor9 = rgba(0, 0, 0, .9) +$darkColor8 = rgba(0, 0, 0, .8) +$darkColor7 = rgba(0, 0, 0, .7) +$darkColor6 = rgba(0, 0, 0, .6) +$darkColor5 = rgba(0, 0, 0, .5) +$darkColor4 = rgba(0, 0, 0, .4) +$darkColor3 = rgba(0, 0, 0, .3) +$darkColor2 = rgba(0, 0, 0, .2) +$darkColor1 = rgba(0, 0, 0, .1) + +$lightColor10 = rgba(255, 255, 255, 1) +$lightColor9 = rgba(255, 255, 255, .9) +$lightColor8 = rgba(255, 255, 255, .8) +$lightColor7 = rgba(255, 255, 255, .7) +$lightColor6 = rgba(255, 255, 255, .6) +$lightColor5 = rgba(255, 255, 255, .5) +$lightColor4 = rgba(255, 255, 255, .4) +$lightColor3 = rgba(255, 255, 255, .3) +$lightColor2 = rgba(255, 255, 255, .2) +$lightColor1 = rgba(255, 255, 255, .1) + +$accentColor = #2A6FDB + +$textShadow = 0 2px 4px $darkColor1; +$borderRadius = .25rem +$lineNumbersWrapperWidth = 2.5rem + +$backgroundColor ?= #fff +$backgroundColorDark ?= #181818 + +$boxShadow = 0 1px 8px 0 $darkColor1 +$boxShadowHover = 0 2px 16px 0 $darkColor2 +$boxShadowDark = 0 1px 8px 0 $darkColor6 +$boxShadowHoverDark = 0 2px 16px 0 $darkColor7 + +$textColor ?= #242424 +$textColorDark ?= $lightColor8 +$textColorSub = #7F7F7F +$textColorSubDark = #8B8B8B + +$borderColor ?= #eaecef +$borderColorDark ?= $darkColor3 + +$codeColor ?= rgba(27, 31, 35, 0.05) +$codeColorDark ?= $darkColor3 + +$maskColor ?= #888 +$maskColorDark ?= #000 + +$homePageWidth = 1126px +$contentWidth = 860px +$sidebarWidth = 18rem diff --git a/docs/.vuepress/theme/vuepress-theme-reco/styles/theme.styl b/docs/.vuepress/theme/vuepress-theme-reco/styles/theme.styl new file mode 100644 index 0000000000..7b30961eec --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/styles/theme.styl @@ -0,0 +1,230 @@ +@require './code' +@require './custom-blocks' +@require './arrow' +@require './wrapper' +@require './toc' + +html, body + padding 0 + margin 0 +body + font-family Ubuntu, -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Oxygen, Cantarell, "Fira Sans", "Droid Sans", "Helvetica Neue", sans-serif + -webkit-font-smoothing antialiased + -moz-osx-font-smoothing grayscale + font-size 15px + color var(--text-color) + background-color var(--background-color) + +.page, .password-wrapper-in + overflow-x: hidden + margin-left $sidebarWidth + +.navbar + position fixed + z-index 20 + top 0 + left 0 + right 0 + height $navbarHeight + box-sizing border-box + +.sidebar-mask + position fixed + z-index 9 + top 0 + left 0 + width 100vw + height 100vh + display none + background-color: rgba(0,0,0,.65); + +.sidebar + font-size 16px + background-color var(--background-color) + width $sidebarWidth + position fixed + z-index 10 + margin 0 + top $navbarHeight + left 0 + bottom 0 + box-sizing border-box + border-right 1px solid var(--border-color) + overflow-y auto + +.content__default:not(.custom) + @extend $wrapper + a:hover + text-decoration underline + p.demo + padding 1rem 1.5rem + border 1px solid #ddd + border-radius 4px + img + max-width 100% + *:first-child + margin-top 0 + +.content__default.custom + padding 0 + margin 0 + img + max-width 100% + +.abstract + img + max-width 100% + +a + font-weight 500 + color $accentColor + text-decoration none + +p a code + font-weight 400 + color $accentColor + +kbd + background #eee + border solid 0.15rem #ddd + border-bottom solid 0.25rem #ddd + border-radius 0.15rem + padding 0 0.15em + +blockquote + font-size .9rem + color #999 + border-left .25rem solid #999 + background-color var(--code-color) + margin 0.5rem 0 + padding .25rem 0 .25rem 1rem + & > p + margin 0 + +ul, ol + padding-left 1.2em + +strong + font-weight 600 + +h1, h2, h3, h4, h5, h6 + font-weight 500 + line-height 1.25 + .content__default:not(.custom) > & + margin-top (2.1rem - $navbarHeight) + padding-top $navbarHeight + margin-bottom 1rem + &:first-child + margin-top -3.5rem; + display: none; + +h1 + font-size 1.6rem + +h2 + font-size 1.4rem + +h3 + font-size 1.2rem + +h4 + font-size 1.1rem + +h5 + font-size 1.0rem + +a.header-anchor + font-size 0.85em + float left + margin-left -0.87em + padding-right 0.23em + margin-top 0.125em + opacity 0 + &:hover + text-decoration none + +code, kbd, .line-number + font-family source-code-pro, Menlo, Monaco, Consolas, "Courier New", monospace + +p, ul, ol + line-height 2.2 + +hr + border 0 + border-top 1px solid var(--border-color) + +table + border-collapse collapse + margin 1rem 0 + display: block + overflow-x: auto + +tr + border-top 1px solid var(--border-color) + &:nth-child(2n) + background-color var(--code-color) + +th, td + border 1px solid var(--border-color) + padding .6em 1em + +.theme-container + &.sidebar-open + .sidebar-mask + display: block + &.no-navbar + .content__default:not(.custom) > h1, h2, h3, h4, h5, h6 + margin-top 1.5rem + padding-top 0 + .sidebar + top 0 + +@media (min-width: ($MQMobile + 1px)) + .theme-container.no-sidebar + .sidebar + display none + .page, .password-wrapper-in + margin-left 0 + +@require 'mobile.styl' + +.iconfont + font-family: "iconfont",Ubuntu,-apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Oxygen,Cantarell,Fira Sans,Droid Sans,Helvetica Neue,sans-serif !important + font-size: 13px; + color: var(--text-color-sub); + + +/************** 滚动条 **************/ +::-webkit-scrollbar + width: 5px; + height: 5px; + +::-webkit-scrollbar-track-piece + // background-color: rgba(0, 0, 0, 0.2); + +::-webkit-scrollbar-thumb:vertical + height: 5px; + background-color: $accentColor; + +::-webkit-scrollbar-thumb:horizontal + width: 5px; + background-color: $accentColor; + +/************** 流程图的滚动条 **************/ +.vuepress-flowchart + overflow: auto + +/************** SW-Update Popup **************/ + +.sw-update-popup + border-radius: $borderRadius!important; + box-shadow: var(--box-shadow)!important; + color: var(--text-color)!important; + background: var(--background-color)!important; + border: none!important; + > button + background: $accentColor; + border-radius: $borderRadius; + color: #fff; + -webkit-tap-highlight-color:rgba(0, 0, 0, 0) + border: none; diff --git a/docs/.vuepress/theme/vuepress-theme-reco/styles/toc.styl b/docs/.vuepress/theme/vuepress-theme-reco/styles/toc.styl new file mode 100644 index 0000000000..d3e71069ba --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/styles/toc.styl @@ -0,0 +1,3 @@ +.table-of-contents + .badge + vertical-align middle diff --git a/docs/.vuepress/theme/vuepress-theme-reco/styles/wrapper.styl b/docs/.vuepress/theme/vuepress-theme-reco/styles/wrapper.styl new file mode 100644 index 0000000000..a99262c71a --- /dev/null +++ b/docs/.vuepress/theme/vuepress-theme-reco/styles/wrapper.styl @@ -0,0 +1,9 @@ +$wrapper + max-width $contentWidth + margin 0 auto + padding 2rem 2.5rem + @media (max-width: $MQNarrow) + padding 2rem + @media (max-width: $MQMobileNarrow) + padding 1.5rem + diff --git a/docs/Go/.DS_Store b/docs/Go/.DS_Store new file mode 100644 index 0000000000..08970f11ff Binary files /dev/null and b/docs/Go/.DS_Store differ diff --git a/docs/Go/Go.md b/docs/Go/Go.md new file mode 100755 index 0000000000..8a533fc908 --- /dev/null +++ b/docs/Go/Go.md @@ -0,0 +1,133 @@ +> Go 语言入门垫脚石 +> +> https://www.runoob.com/go/go-tutorial.html + + + +Go语言是什么? + +Go出自名门Google公司，是一门支持并发 、垃圾回收的编译型高级编程语言。Go兼具静态编译语言的高性能以及动态语言的高开发效率。 该项目的三位领导者均是著名的语言学家: + +Rob Pike: + +Go 语言项目总负责人，贝尔实验室 Unix 团队成员，参与的项目包括 Plan9，Inferno 操作系统和 Limbo 编程语言 + +Ken Thompson: + +贝尔实验室 Unix 团队成员，C 语言、Unix 和 Plan 9 的创始人之一，与 Rob Pike 共同开发了 UTF-8 字符集规范，图灵奖获得者 + +Robert Griesemer: + +参与开发 Java HotSpot 虚拟机、 V8 Javascript engine + + + +**为什么选** **Go****?** + +❑ 语言简单、开发效率高 + ❑ 高效的垃圾回收机制 + ❑ 支持多返回值 + ❑ 更丰富的内置类型:map、slice、channel、interface ❑ 语言层面支持并发编程 + +❑ 编译型语言，编译即测试 ❑ 跨平台编译 + + + +![](/Users/apple/Desktop/screenshot/截屏2022-05-05 下午4.16.44.png) + + + +![](/Users/apple/Desktop/screenshot/截屏2022-05-05 下午4.17.19.png) + +![](/Users/apple/Desktop/screenshot/截屏2022-05-05 下午4.17.43.png) + + + + + +基础概念 + +数据类型和语句 + +Go 程序的测试 + +标准库的用法 + + + +## Hello world + +Go 语言的基础组成有以下几个部分： + +- 包声明 +- 引入包 +- 函数 +- 变量 +- 语句 & 表达式 +- 注释 + +```go +package main + +import "fmt" + +func main() { + /* 这是我的第一个简单的程序 */ + fmt.Println("Hello, World!") +} +``` + +> 1. 第一行代码 *package main* 定义了包名。你必须在源文件中非注释的第一行指明这个文件属于哪个包，如：package main。package main表示一个可独立执行的程序，每个 Go 应用程序都包含一个名为 main 的包。 +> 2. 下一行 *import "fmt"* 告诉 Go 编译器这个程序需要使用 fmt 包（的函数，或其他元素），fmt 包实现了格式化 IO（输入/输出）的函数。 +> 3. 下一行 *func main()* 是程序开始执行的函数。main 函数是每一个可执行程序所必须包含的，一般来说都是在启动后第一个执行的函数（如果有 init() 函数则会先执行该函数）。 +> 4. 下一行 /*...*/ 是注释，在程序执行时将被忽略。单行注释是最常见的注释形式，你可以在任何地方使用以 // 开头的单行注释。多行注释也叫块注释，均已以 /* 开头，并以 */ 结尾，且不可以嵌套使用，多行注释一般用于包的文档描述或注释成块的代码片段。 +> 5. 下一行 *fmt.Println(...)* 可以将字符串输出到控制台，并在最后自动增加换行字符 \n。 +> 使用 fmt.Print("hello, world\n") 可以得到相同的结果。 +> Print 和 Println 这两个函数也支持使用变量，如：fmt.Println(arr)。如果没有特别指定，它们会以默认的打印格式将变量 arr 输出到控制台。 +> 6. 当标识符（包括常量、变量、类型、函数名、结构字段等等）以一个大写字母开头，如：Group1，那么使用这种形式的标识符的对象就可以被外部包的代码所使用（客户端程序需要先导入这个包），这被称为导出（像面向对象语言中的 public）；标识符如果以小写字母开头，则对包外是不可见的，但是他们在整个包的内部是可见并且可用的（像面向对象语言中的 protected ）。 + + + + + +与Java 的区别： + +- 行分隔符：在 Go 程序中，一行代表一个语句结束。每个语句不需要像 C 家族中的其它语言一样以分号 ; 结尾，因为这些工作都将由 Go 编译器自动完成。 + +![](/Users/apple/Desktop/screenshot/截屏2022-05-05 下午4.26.00.png) + +## 工作区和GOPATH + +我们学习 Go 语言时，要做的第一件事，都是根据自己电脑的计算架构（比如，是 32 位的计算机还是 64 位的计算机）以及操作系统（比如，是 Windows 还是 Linux），从[Go 语言官网](https://golang.google.cn)下载对应的二进制包，也就是可以拿来即用的安装包。 + +随后，我们会解压缩安装包、放置到某个目录、配置环境变量，并通过在命令行中输入`go version`来验证是否安装成功。 + +在这个过程中，我们还需要配置 3 个环境变量，也就是 GOROOT、GOPATH 和 GOBIN。这里我可以简单介绍一下。 + +- GOROOT：Go 语言安装根目录的路径，也就是 GO 语言的安装路径。 +- GOPATH：若干工作区目录的路径。是我们自己定义的工作空间。 +- GOBIN：GO 程序生成的可执行文件（executable file）的路径。 + +其中，GOPATH 背后的概念是最多的，也是最重要的。那么，**今天我们的面试问题是：你知道设置 GOPATH 有什么意义吗？** + +关于这个问题，它的**典型回答**是这样的： + +你可以把 GOPATH 简单理解成 Go 语言的工作目录，它的值是一个目录的路径，也可以是多个目录路径，每个目录都代表 Go 语言的一个工作区（workspace）。 + +我们需要利于这些工作区，去放置 Go 语言的源码文件（source file），以及安装（install）后的归档文件（archive file，也就是以“.a”为扩展名的文件）和可执行文件（executable file）。 + +事实上，由于 Go 语言项目在其生命周期内的所有操作（编码、依赖管理、构建、测试、安装等）基本上都是围绕着 GOPATH 和工作区进行的。所以，它的背后至少有 3 个知识点，分别是： + +**1. Go 语言源码的组织方式是怎样的；** + +**2. 你是否了解源码安装后的结果（只有在安装后，Go 语言源码才能被我们或其他代码使用）；** + +**3. 你是否理解构建和安装 Go 程序的过程（这在开发程序以及查找程序问题的时候都很有用，否则你很可能会走弯路）。** + + + +## 命令源码文件 + +**源码文件又分为三种，即：命令源码文件、库源码文件和测试源码文件，它们都有着不同的用途和编写规则。** + +![](https://static001.geekbang.org/resource/image/9d/cb/9d08647d238e21e7184d60c0afe5afcb.png) \ No newline at end of file diff --git a/docs/README.md b/docs/README.md index 3612e34184..6712082265 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,26 +1,25 @@ -### JavaKeeper - -

-
- - - - - -

- -> 希望自己可以持续输入并一直输出，有一个自己的完整知识库，所以命名为 JavaKeeper - - - -### ToDoList - -- [x] OOM -- [ ] Kafka 系列文章 - -> 作为一个深耕在互联网公司的Java码农，在准备换坑的时候，才发现这么些年都没有一个对所学知识的整理，所以就整理下之前的笔记和一些正在学习的内容。文中内容大部分来自自己平时看的学习视频以及一些大牛博客 - -> 每个知识点都会有一篇HELLO系文章，最后会总结一些互联网公司面试问题，用来应对下一份工作面试的提问 - - - +--- +home: true +heroText: JavaKeeper +tagline: 欢迎加入 Java 技术有限委员会 + +heroImage: /logo.jpg +# heroImageStyle: { +# maxWidth: '600px', +# width: '100%', +# display: block, +# margin: '9rem auto 2rem', +# background: '#fff', +# borderRadius: '1rem', +# } +# isShowTitleInHome: false +actionText: 干起来 → +actionLink: /java/ +features: +- title: Java + details: Java 工程师必备架构体系知识总结：涵盖分布式、微服务、RPC等互联网公司常用架构，以及数据结构与算法、数据存储、缓存、搜索等必备技能 +- title: Keep + details: 学技术，最重要的不是怎么学，而是开始学。多少有志青年，想得太多，做的太少，没开始时，在“学什么”中纠结，开始之后，在“怎么学”中迷茫。以梦为马，越骑越傻。诗和远方，越走越慌。不忘初心是对的，但切记要出发 +- title: er + details: 一个嗜好太多能力太小的互联网工具人 +--- diff --git "a/docs/TODO/complexity\347\232\204\345\211\257\346\234\254.md" "b/docs/TODO/complexity\347\232\204\345\211\257\346\234\254.md" deleted file mode 100644 index 32979ae6e1..0000000000 --- "a/docs/TODO/complexity\347\232\204\345\211\257\346\234\254.md" +++ /dev/null @@ -1,40 +0,0 @@ -作为一个伪高级开发，想换坑了，之前也没系统刷过算法题，今天打开LeetCode第一题，看解析突然发现大学时候学的时间复杂度、空间复杂度在我脑中只有$O(n)$、$O(0)$ 这几个符号，具体什么意思，学渣体质的我那会应该就没学会，所以，，，输入并输出一下， - - - -![img](https://aliyun-lc-upload.oss-cn-hangzhou.aliyuncs.com/LCCN-Articles/explore/2018%20%E9%9D%A2%E8%AF%95%E6%B1%87%E6%80%BB/lc_sum.png) - - - - - -算法与数据结构是面试考察的重中之重，也是大家日后学习时需要着重训练的部分。简单的总结一下，大约有这些内容： - - - -## **算法 - Algorithms** - -1. 排序算法：快速排序、归并排序、计数排序 -2. 搜索算法：回溯、递归、剪枝技巧 -3. 图论：最短路、最小生成树、网络流建模 -4. 动态规划：背包问题、最长子序列、计数问题 -5. 基础技巧：分治、倍增、二分、贪心 - -## **数据结构 - Data Structures** - -1. 数组与链表：单 / 双向链表、跳舞链 -2. 栈与队列 -3. 树与图：最近公共祖先、并查集 -4. 哈希表 -5. 堆：大 / 小根堆、可并堆 -6. 字符串：字典树、后缀树 - - - -对于上面总结的这部分内容，力扣已经为大家准备好了相关专题，等待大家来练习啦。 - -算法部分，我们开设了 [初级算法 - 帮助入门](https://leetcode-cn.com/explore/interview/card/top-interview-questions-easy/)、[中级算法 - 巩固训练](https://leetcode-cn.com/explore/interview/card/top-interview-questions-medium/)、 [高级算法 - 提升进阶](https://leetcode-cn.com/explore/interview/card/top-interview-questions-hard/) 三个不同的免费探索主题，包含：数组、字符串、搜索、排序、动态规划、数学、图论等许多内容。大家可以针对自己当前的基础与能力，选择相对应的栏目进行练习。为了能够达到较好的效果，建议小伙伴将所有题目都练习 2～3 遍，吃透每一道题目哦。 - -数据结构部分，我们也开设了一个非常系统性的 [数据结构](https://leetcode-cn.com/explore/learn/) 板块，有练习各类数据结构的探索主题，其中包含：队列与栈、数组与字符串、链表、哈希表、二叉树等丰富的内容。每一个章节都包含文字讲解与生动的图片演示，同时配套相关题目。相信只要认真练习，一定能受益匪浅。 - -力扣将热门面试问题里比较新的题目按照类别进行了整理，以供大家按模块练习。 diff --git a/docs/_images/.DS_Store b/docs/_images/.DS_Store new file mode 100644 index 0000000000..90d90ebc34 Binary files /dev/null and b/docs/_images/.DS_Store differ diff --git a/docs/_images/Spring/.DS_Store b/docs/_images/Spring/.DS_Store new file mode 100644 index 0000000000..d7b14a1102 Binary files /dev/null and b/docs/_images/Spring/.DS_Store differ diff --git a/docs/_images/Spring/BeanFactory.png b/docs/_images/Spring/BeanFactory.png new file mode 100644 index 0000000000..2be2dcc303 Binary files /dev/null and b/docs/_images/Spring/BeanFactory.png differ diff --git a/docs/_images/Spring/aop-demo.svg b/docs/_images/Spring/aop-demo.svg new file mode 100644 index 0000000000..150a14d37c --- /dev/null +++ b/docs/_images/Spring/aop-demo.svg @@ -0,0 +1,3 @@ + + +
验证参数
验证参数
前置日志
前置日志
add()
add()
后置日志
后置日志
验证参数
验证参数
前置日志
前置日志
sub()
sub()
后置日志
后置日志
验证参数
验证参数
前置日志
前置日志
mul()
mul()
后置日志
后置日志
验证参数
验证参数
前置日志
前置日志
div()
div()
后置日志
后置日志
后置日志
后置日志
验证参数
验证参数
前置日志
前置日志
add()
add()
sub()
sub()
mul()
mul()
div()
div()
抽取横切关注点
抽取横切关注点
业务逻辑
业务逻辑
业务逻辑
业务逻辑
切面
切面
AOP
AOP
验证
验证
日志
日志Viewer does not support full SVG 1.1 \ No newline at end of file diff --git a/docs/_images/Spring/beanDefinition.png b/docs/_images/Spring/beanDefinition.png new file mode 100644 index 0000000000..883e566f6b Binary files /dev/null and b/docs/_images/Spring/beanDefinition.png differ diff --git a/docs/_images/Spring/cycle-demo.png b/docs/_images/Spring/cycle-demo.png new file mode 100644 index 0000000000..e27743eff8 Binary files /dev/null and b/docs/_images/Spring/cycle-demo.png differ diff --git a/docs/_images/Spring/cycle-dependency-code.png b/docs/_images/Spring/cycle-dependency-code.png new file mode 100644 index 0000000000..b032b328a0 Binary files /dev/null and b/docs/_images/Spring/cycle-dependency-code.png differ diff --git a/docs/_images/Spring/cycle-dependency-constructor.png b/docs/_images/Spring/cycle-dependency-constructor.png new file mode 100644 index 0000000000..78ee686144 Binary files /dev/null and b/docs/_images/Spring/cycle-dependency-constructor.png differ diff --git a/docs/_images/Spring/cycle-dependency-index.png b/docs/_images/Spring/cycle-dependency-index.png new file mode 100644 index 0000000000..d70f6c4339 Binary files /dev/null and b/docs/_images/Spring/cycle-dependency-index.png differ diff --git a/docs/_images/Spring/getEarlyBeanReference-code.png b/docs/_images/Spring/getEarlyBeanReference-code.png new file mode 100644 index 0000000000..42ce28648e Binary files /dev/null and b/docs/_images/Spring/getEarlyBeanReference-code.png differ diff --git a/docs/_images/Spring/ioc-demo.png b/docs/_images/Spring/ioc-demo.png new file mode 100644 index 0000000000..a6a67efbee Binary files /dev/null and b/docs/_images/Spring/ioc-demo.png differ diff --git a/docs/_images/Spring/spring-aop-demo.svg b/docs/_images/Spring/spring-aop-demo.svg new file mode 100644 index 0000000000..150a14d37c --- /dev/null +++ b/docs/_images/Spring/spring-aop-demo.svg @@ -0,0 +1,3 @@ + + +
验证参数
验证参数
前置日志
前置日志
add()
add()
后置日志
后置日志
验证参数
验证参数
前置日志
前置日志
sub()
sub()
后置日志
后置日志
验证参数
验证参数
前置日志
前置日志
mul()
mul()
后置日志
后置日志
验证参数
验证参数
前置日志
前置日志
div()
div()
后置日志
后置日志
后置日志
后置日志
验证参数
验证参数
前置日志
前置日志
add()
add()
sub()
sub()
mul()
mul()
div()
div()
抽取横切关注点
抽取横切关注点
业务逻辑
业务逻辑
业务逻辑
业务逻辑
切面
切面
AOP
AOP
验证
验证
日志
日志Viewer does not support full SVG 1.1 \ No newline at end of file diff --git a/docs/_images/Spring/spring-aop-log.png b/docs/_images/Spring/spring-aop-log.png new file mode 100644 index 0000000000..718171766a --- /dev/null +++ b/docs/_images/Spring/spring-aop-log.png @@ -0,0 +1,3 @@ + + +
调用者
调用者
Calculator
Calculator
日志代理
日志代理
验证代理
验证代理
记录开始日志
记录开始日志
参数检验
参数检验
记录结束日志
记录结束日志Viewer does not support full SVG 1.1 \ No newline at end of file diff --git a/docs/_images/Spring/spring-aop-log.svg b/docs/_images/Spring/spring-aop-log.svg new file mode 100644 index 0000000000..718171766a --- /dev/null +++ b/docs/_images/Spring/spring-aop-log.svg @@ -0,0 +1,3 @@ + + +
调用者
调用者
Calculator
Calculator
日志代理
日志代理
验证代理
验证代理
记录开始日志
记录开始日志
参数检验
参数检验
记录结束日志
记录结束日志Viewer does not support full SVG 1.1 \ No newline at end of file diff --git a/docs/_images/Spring/spring-createbean.png b/docs/_images/Spring/spring-createbean.png new file mode 100644 index 0000000000..fe377d4d1a Binary files /dev/null and b/docs/_images/Spring/spring-createbean.png differ diff --git a/docs/_images/Spring/spring-getbean.png b/docs/_images/Spring/spring-getbean.png new file mode 100644 index 0000000000..64851f91a7 Binary files /dev/null and b/docs/_images/Spring/spring-getbean.png differ diff --git a/docs/_images/ad/.DS_Store b/docs/_images/ad/.DS_Store new file mode 100644 index 0000000000..21fad860c7 Binary files /dev/null and b/docs/_images/ad/.DS_Store differ diff --git "a/docs/_images/ad/DPA\344\272\247\347\224\237\350\203\214\346\231\257.png" "b/docs/_images/ad/DPA\344\272\247\347\224\237\350\203\214\346\231\257.png" new file mode 100644 index 0000000000..975b83ee7c Binary files /dev/null and "b/docs/_images/ad/DPA\344\272\247\347\224\237\350\203\214\346\231\257.png" differ diff --git a/docs/_images/ad/ad-platforms.png b/docs/_images/ad/ad-platforms.png new file mode 100644 index 0000000000..a240924a57 Binary files /dev/null and b/docs/_images/ad/ad-platforms.png differ diff --git a/docs/_images/ad/dpa-flow.png b/docs/_images/ad/dpa-flow.png new file mode 100644 index 0000000000..2083336574 Binary files /dev/null and b/docs/_images/ad/dpa-flow.png differ diff --git "a/docs/_images/ad/dpa-\345\212\250\346\200\201\345\225\206\345\223\201\345\271\277\345\221\212\344\274\230\345\212\277.png" "b/docs/_images/ad/dpa-\345\212\250\346\200\201\345\225\206\345\223\201\345\271\277\345\221\212\344\274\230\345\212\277.png" new file mode 100644 index 0000000000..989fc09dcb Binary files /dev/null and "b/docs/_images/ad/dpa-\345\212\250\346\200\201\345\225\206\345\223\201\345\271\277\345\221\212\344\274\230\345\212\277.png" differ diff --git "a/docs/_images/ad/dpa\344\270\232\345\212\241.png" "b/docs/_images/ad/dpa\344\270\232\345\212\241.png" new file mode 100644 index 0000000000..20ea48295e Binary files /dev/null and "b/docs/_images/ad/dpa\344\270\232\345\212\241.png" differ diff --git a/docs/_images/ad/mobile_search.png b/docs/_images/ad/mobile_search.png new file mode 100644 index 0000000000..901bfa05d0 Binary files /dev/null and b/docs/_images/ad/mobile_search.png differ diff --git a/docs/_images/ad/real-time-report.png b/docs/_images/ad/real-time-report.png new file mode 100644 index 0000000000..269b8ab4da Binary files /dev/null and b/docs/_images/ad/real-time-report.png differ diff --git a/docs/_images/ad/report-ad.png b/docs/_images/ad/report-ad.png new file mode 100644 index 0000000000..94f63b2e60 Binary files /dev/null and b/docs/_images/ad/report-ad.png differ diff --git a/docs/_images/ad/report-feature.png b/docs/_images/ad/report-feature.png new file mode 100644 index 0000000000..f8030e81cc Binary files /dev/null and b/docs/_images/ad/report-feature.png differ diff --git a/docs/_images/ad/report-framework.png b/docs/_images/ad/report-framework.png new file mode 100644 index 0000000000..586ee859ea --- /dev/null +++ b/docs/_images/ad/report-framework.png @@ -0,0 +1,4 @@ + + + +
网页报告
网页报告
定制报告
定制报告
API  报告
API  报告
移动端报告
移动端报告
数据应用
数据应用
物理存储
物理存储
基础宽表
基础宽表
数据采集
数据采集
 
RSYNC




...
消费数据
消费数据
数据源
数据源
物化视图
物化视图
HBase
HBase
关键词PV
关键词PV
计费Kafka
计费Kafka
Adtech Kafka
Adtech Kafka
点击数据
点击数据
排名数据
排名数据
展现数据
展现数据
逻辑存储
逻辑存储
数据计算
（数据模型）
数据计算（数据模型）...
账号报告
账号报告
计划报告
计划报告
关键词报告
关键词报告
Phoenix
Phoenix
HBase
HBase
HBase
HBase
数据存储
数据存储
分时报告
分时报告
普通创意
普通创意
排名数据
排名数据
   
HDFS / Hive




...
计费日志
计费日志
组报告
组报告
搜索词报告
搜索词报告
分地域报告
分地域报告
分人群报告
分人群报告
创意组件
创意组件
应用组件
应用组件
视频类样式
视频类样式
视频类样式
视频类样式
子链类样式
子链类样式
商品类样式
商品类样式
子链类样式
子链类样式
基础报告
基础报告
维度报告
维度报告
创意报告
创意报告
......
......
样式报告
样式报告
kylin
kylinViewer does not support full SVG 1.1 \ No newline at end of file diff --git a/docs/_images/ad/report-framework.svg b/docs/_images/ad/report-framework.svg new file mode 100644 index 0000000000..586ee859ea --- /dev/null +++ b/docs/_images/ad/report-framework.svg @@ -0,0 +1,4 @@ + + + +
网页报告
网页报告
定制报告
定制报告
API  报告
API  报告
移动端报告
移动端报告
数据应用
数据应用
物理存储
物理存储
基础宽表
基础宽表
数据采集
数据采集
 
RSYNC




...
消费数据
消费数据
数据源
数据源
物化视图
物化视图
HBase
HBase
关键词PV
关键词PV
计费Kafka
计费Kafka
Adtech Kafka
Adtech Kafka
点击数据
点击数据
排名数据
排名数据
展现数据
展现数据
逻辑存储
逻辑存储
数据计算
（数据模型）
数据计算（数据模型）...
账号报告
账号报告
计划报告
计划报告
关键词报告
关键词报告
Phoenix
Phoenix
HBase
HBase
HBase
HBase
数据存储
数据存储
分时报告
分时报告
普通创意
普通创意
排名数据
排名数据
   
HDFS / Hive




...
计费日志
计费日志
组报告
组报告
搜索词报告
搜索词报告
分地域报告
分地域报告
分人群报告
分人群报告
创意组件
创意组件
应用组件
应用组件
视频类样式
视频类样式
视频类样式
视频类样式
子链类样式
子链类样式
商品类样式
商品类样式
子链类样式
子链类样式
基础报告
基础报告
维度报告
维度报告
创意报告
创意报告
......
......
样式报告
样式报告
kylin
kylinViewer does not support full SVG 1.1 \ No newline at end of file diff --git a/docs/_images/ad/report-platform.png b/docs/_images/ad/report-platform.png new file mode 100644 index 0000000000..a674ac297f Binary files /dev/null and b/docs/_images/ad/report-platform.png differ diff --git a/docs/_images/ad/sogou-data-center-ETL.drawio.svg b/docs/_images/ad/sogou-data-center-ETL.drawio.svg new file mode 100644 index 0000000000..a4b1a3ae56 --- /dev/null +++ b/docs/_images/ad/sogou-data-center-ETL.drawio.svg @@ -0,0 +1,4 @@ + + + +
xuripv
xuripv
xuribasedata
xuribasedata
processPvTask
processPvTask
processClickTask
processClickTask
xuricost
xuricost
xuripv_pre
xuripv_pre
xurilogall
xurilogall
xuri_pv_hour
xuri_pv_hour
importPcXuriPvHour2HiveTask
importMobileXuriPvHour2HiveTask
importPcXuriPvHour2Hive...
pv_log
pv_log
1变多，1条日志拆分成多条广告展现
1变多，1条日志拆分成多条广告展现
给每条广告展现设置唯一 pvid, 并拆分素材三元组，计算关键词排名
给每条广告展现设置唯一 pvid, 并拆分素材三元组，计算关键词排名
processPvPreTask
processPvPreTask
一次搜索会打印1条日志，会有多个广告展现，拼接成1条展现日志
一次搜索会打印1条日志，会有多个广告展现，拼接成1条展现日志
xuri_pv_hour
xuri_pv_hour
xuri_cpcreport
xuri_cpcreport
xuri_ideareport
xuri_ideareport
ideareport_view
ideareport_view
cpcreport_view
cpcreport_view
regionreport_view
regionreport_view
hourreport_view
hourreport_view
ipreport_view
ipreport_view
stylereport_styletype
_view

stylereport_styletype...
stylereport_materialtype
_view
stylereport_materialtype...
stylereport_material
_view
stylereport_material...
stylereport_material_
clickposition_view
stylereport_material_...
账号、计划、组、关键词报告
账号、计划、组、关键词报告
地域报告
地域报告
分时报告
分时报告
IP报告
IP报告
创意报告
创意报告
样式维度报告
样式维度报告
素材类型维度报告
素材类型维度报告
素材id维度
素材id维度
素材点击位置维度
素材点击位置维度
cpcreport_view
cpcreport_view
搜索词报告
搜索词报告
大宽表
大宽表
计费组生成，单条计费日志，包含点四元祖和消耗
计费组生成，单条计费日志，包含点四元祖和消耗

Viewer does not support full SVG 1.1 \ No newline at end of file diff --git a/docs/_images/ad/xuri.p4p.sogou.png b/docs/_images/ad/xuri.p4p.sogou.png new file mode 100644 index 0000000000..a674ac297f Binary files /dev/null and b/docs/_images/ad/xuri.p4p.sogou.png differ diff --git "a/docs/_images/ad/\346\220\234\347\264\242DPA.png" "b/docs/_images/ad/\346\220\234\347\264\242DPA.png" new file mode 100644 index 0000000000..a472130715 Binary files /dev/null and "b/docs/_images/ad/\346\220\234\347\264\242DPA.png" differ diff --git a/docs/_images/algorithms/.DS_Store b/docs/_images/algorithms/.DS_Store new file mode 100644 index 0000000000..4f15a9793e Binary files /dev/null and b/docs/_images/algorithms/.DS_Store differ diff --git a/docs/_images/algorithms/stock-problems.png b/docs/_images/algorithms/stock-problems.png new file mode 100644 index 0000000000..e51cca9f95 Binary files /dev/null and b/docs/_images/algorithms/stock-problems.png differ diff --git a/docs/_images/big-data/recommend .png b/docs/_images/big-data/recommend.png similarity index 100% rename from docs/_images/big-data/recommend .png rename to docs/_images/big-data/recommend.png diff --git a/docs/_images/data-structure/.DS_Store b/docs/_images/data-structure/.DS_Store new file mode 100644 index 0000000000..e67b729ad8 Binary files /dev/null and b/docs/_images/data-structure/.DS_Store differ diff --git "a/docs/_images/data-structure/DP_\350\207\252\344\270\213\350\200\214\344\270\212.png" "b/docs/_images/data-structure/DP_\350\207\252\344\270\213\350\200\214\344\270\212.png" new file mode 100644 index 0000000000..8349ff993a Binary files /dev/null and "b/docs/_images/data-structure/DP_\350\207\252\344\270\213\350\200\214\344\270\212.png" differ diff --git a/docs/_images/data-structure/linked-list/double-linkedlist-add.png b/docs/_images/data-structure/linked-list/double-linkedlist-add.png new file mode 100644 index 0000000000..c04cea1ed1 Binary files /dev/null and b/docs/_images/data-structure/linked-list/double-linkedlist-add.png differ diff --git a/docs/_images/data-structure/linked-list/double-linkedlist-del.png b/docs/_images/data-structure/linked-list/double-linkedlist-del.png new file mode 100644 index 0000000000..7b8d0677d2 Binary files /dev/null and b/docs/_images/data-structure/linked-list/double-linkedlist-del.png differ diff --git a/docs/_images/data-structure/linked-list/doule-linkedlist-node.png b/docs/_images/data-structure/linked-list/doule-linkedlist-node.png new file mode 100644 index 0000000000..f0d654b956 Binary files /dev/null and b/docs/_images/data-structure/linked-list/doule-linkedlist-node.png differ diff --git a/docs/_images/data-structure/linked-list/doule-linkedlist.png b/docs/_images/data-structure/linked-list/doule-linkedlist.png new file mode 100644 index 0000000000..ff1865f23c Binary files /dev/null and b/docs/_images/data-structure/linked-list/doule-linkedlist.png differ diff --git a/docs/_images/data-structure/linked-list/single-linkedlist-add.png b/docs/_images/data-structure/linked-list/single-linkedlist-add.png new file mode 100644 index 0000000000..b14464df82 Binary files /dev/null and b/docs/_images/data-structure/linked-list/single-linkedlist-add.png differ diff --git a/docs/_images/data-structure/linked-list/single-linkedlist-del.png b/docs/_images/data-structure/linked-list/single-linkedlist-del.png new file mode 100644 index 0000000000..0ac9ae96b3 Binary files /dev/null and b/docs/_images/data-structure/linked-list/single-linkedlist-del.png differ diff --git a/docs/_images/data-structure/linked-list/single-linkedlist-head.png b/docs/_images/data-structure/linked-list/single-linkedlist-head.png new file mode 100644 index 0000000000..b6b44d72c7 Binary files /dev/null and b/docs/_images/data-structure/linked-list/single-linkedlist-head.png differ diff --git a/docs/_images/data-structure/linked-list/single-linkedlist-node.png b/docs/_images/data-structure/linked-list/single-linkedlist-node.png new file mode 100644 index 0000000000..a3435041f7 Binary files /dev/null and b/docs/_images/data-structure/linked-list/single-linkedlist-node.png differ diff --git a/docs/_images/data-structure/linked-list/single-linkedlist.png b/docs/_images/data-structure/linked-list/single-linkedlist.png new file mode 100644 index 0000000000..fed5021108 Binary files /dev/null and b/docs/_images/data-structure/linked-list/single-linkedlist.png differ diff --git a/docs/_images/data-structure/skip-list/.DS_Store b/docs/_images/data-structure/skip-list/.DS_Store new file mode 100644 index 0000000000..a0f1e2245e Binary files /dev/null and b/docs/_images/data-structure/skip-list/.DS_Store differ diff --git a/docs/_images/data-structure/skip-list/linkedlist.png b/docs/_images/data-structure/skip-list/linkedlist.png new file mode 100644 index 0000000000..81fb6a2374 Binary files /dev/null and b/docs/_images/data-structure/skip-list/linkedlist.png differ diff --git a/docs/_images/data-structure/skip-list/skip-index.png b/docs/_images/data-structure/skip-list/skip-index.png new file mode 100644 index 0000000000..e466cb0a32 Binary files /dev/null and b/docs/_images/data-structure/skip-list/skip-index.png differ diff --git a/docs/_images/data-structure/skip-list/skip-list.png b/docs/_images/data-structure/skip-list/skip-list.png new file mode 100644 index 0000000000..7f9dda32ff Binary files /dev/null and b/docs/_images/data-structure/skip-list/skip-list.png differ diff --git a/docs/_images/data-structure/skip-list/skiplist-banner.png b/docs/_images/data-structure/skip-list/skiplist-banner.png new file mode 100644 index 0000000000..74ced7d76b Binary files /dev/null and b/docs/_images/data-structure/skip-list/skiplist-banner.png differ diff --git a/docs/_images/data-structure/skip-list/skiplist-index-count.png b/docs/_images/data-structure/skip-list/skiplist-index-count.png new file mode 100644 index 0000000000..02e967b496 Binary files /dev/null and b/docs/_images/data-structure/skip-list/skiplist-index-count.png differ diff --git a/docs/_images/data-structure/skip-list/skiplist-insert.png b/docs/_images/data-structure/skip-list/skiplist-insert.png new file mode 100644 index 0000000000..f01cffa5ea Binary files /dev/null and b/docs/_images/data-structure/skip-list/skiplist-insert.png differ diff --git a/docs/_images/data-structure/skip-list/skiplist-resume.png b/docs/_images/data-structure/skip-list/skiplist-resume.png new file mode 100644 index 0000000000..efe9714778 Binary files /dev/null and b/docs/_images/data-structure/skip-list/skiplist-resume.png differ diff --git a/docs/_images/data-structure/skip-list/skiplist.png b/docs/_images/data-structure/skip-list/skiplist.png new file mode 100644 index 0000000000..9c25c31db5 Binary files /dev/null and b/docs/_images/data-structure/skip-list/skiplist.png differ diff --git a/docs/_images/data-structure/tree/68747470733a2f2f747661312e73696e61696d672e636e2f6c617267652f30303753385a496c6c7931676475367871726e6e666a3331393230686f7463762e6a7067.jpeg b/docs/_images/data-structure/tree/68747470733a2f2f747661312e73696e61696d672e636e2f6c617267652f30303753385a496c6c7931676475367871726e6e666a3331393230686f7463762e6a7067.jpeg new file mode 100644 index 0000000000..fdfb661a3d Binary files /dev/null and b/docs/_images/data-structure/tree/68747470733a2f2f747661312e73696e61696d672e636e2f6c617267652f30303753385a496c6c7931676475367871726e6e666a3331393230686f7463762e6a7067.jpeg differ diff --git a/docs/_images/data-structure/tree/array-2-binary-tree.png b/docs/_images/data-structure/tree/array-2-binary-tree.png new file mode 100644 index 0000000000..d3b5e6024f Binary files /dev/null and b/docs/_images/data-structure/tree/array-2-binary-tree.png differ diff --git a/docs/_images/data-structure/tree/avl-tree.png b/docs/_images/data-structure/tree/avl-tree.png new file mode 100644 index 0000000000..d79558af14 Binary files /dev/null and b/docs/_images/data-structure/tree/avl-tree.png differ diff --git a/docs/_images/data-structure/tree/balance-binary-tree.jpeg b/docs/_images/data-structure/tree/balance-binary-tree.jpeg new file mode 100644 index 0000000000..fedb1b3059 Binary files /dev/null and b/docs/_images/data-structure/tree/balance-binary-tree.jpeg differ diff --git a/docs/_images/data-structure/tree/binary-search-tree-insert.gif b/docs/_images/data-structure/tree/binary-search-tree-insert.gif new file mode 100644 index 0000000000..ccf1038bd9 Binary files /dev/null and b/docs/_images/data-structure/tree/binary-search-tree-insert.gif differ diff --git a/docs/_images/data-structure/tree/binary-search-tree-penjee.gif b/docs/_images/data-structure/tree/binary-search-tree-penjee.gif new file mode 100644 index 0000000000..73b8b69b87 Binary files /dev/null and b/docs/_images/data-structure/tree/binary-search-tree-penjee.gif differ diff --git a/docs/_images/data-structure/tree/binary-search-tree.jpeg b/docs/_images/data-structure/tree/binary-search-tree.jpeg new file mode 100644 index 0000000000..599e3ac217 Binary files /dev/null and b/docs/_images/data-structure/tree/binary-search-tree.jpeg differ diff --git a/docs/_images/data-structure/tree/binary-serach-tree-del.png b/docs/_images/data-structure/tree/binary-serach-tree-del.png new file mode 100644 index 0000000000..6c668e6440 Binary files /dev/null and b/docs/_images/data-structure/tree/binary-serach-tree-del.png differ diff --git a/docs/_images/data-structure/tree/binary-tree-2-array.png b/docs/_images/data-structure/tree/binary-tree-2-array.png new file mode 100644 index 0000000000..8df2706690 Binary files /dev/null and b/docs/_images/data-structure/tree/binary-tree-2-array.png differ diff --git a/docs/_images/data-structure/tree/binary-tree-dfs.png b/docs/_images/data-structure/tree/binary-tree-dfs.png new file mode 100644 index 0000000000..a50aba25ba Binary files /dev/null and b/docs/_images/data-structure/tree/binary-tree-dfs.png differ diff --git a/docs/_images/data-structure/tree/binary-tree-inorder.png b/docs/_images/data-structure/tree/binary-tree-inorder.png new file mode 100644 index 0000000000..f5ab39b8e6 Binary files /dev/null and b/docs/_images/data-structure/tree/binary-tree-inorder.png differ diff --git a/docs/_images/data-structure/tree/binary-tree-leveltraverse.png b/docs/_images/data-structure/tree/binary-tree-leveltraverse.png new file mode 100644 index 0000000000..9d80955094 Binary files /dev/null and b/docs/_images/data-structure/tree/binary-tree-leveltraverse.png differ diff --git a/docs/_images/data-structure/tree/binary-tree-node-store.jpeg b/docs/_images/data-structure/tree/binary-tree-node-store.jpeg new file mode 100644 index 0000000000..3ec509d48c Binary files /dev/null and b/docs/_images/data-structure/tree/binary-tree-node-store.jpeg differ diff --git a/docs/_images/data-structure/tree/binary-tree-postorder.jpeg b/docs/_images/data-structure/tree/binary-tree-postorder.jpeg new file mode 100644 index 0000000000..3798cda5e3 Binary files /dev/null and b/docs/_images/data-structure/tree/binary-tree-postorder.jpeg differ diff --git a/docs/_images/data-structure/tree/binary-tree-preorder.png b/docs/_images/data-structure/tree/binary-tree-preorder.png new file mode 100644 index 0000000000..778afd1c5d Binary files /dev/null and b/docs/_images/data-structure/tree/binary-tree-preorder.png differ diff --git a/docs/_images/data-structure/tree/binary-tree-special-case.jpeg b/docs/_images/data-structure/tree/binary-tree-special-case.jpeg new file mode 100644 index 0000000000..3bcb093e7e Binary files /dev/null and b/docs/_images/data-structure/tree/binary-tree-special-case.jpeg differ diff --git a/docs/_images/data-structure/tree/binary-tree-store1.jpeg b/docs/_images/data-structure/tree/binary-tree-store1.jpeg new file mode 100644 index 0000000000..14dfc1db33 Binary files /dev/null and b/docs/_images/data-structure/tree/binary-tree-store1.jpeg differ diff --git a/docs/_images/data-structure/tree/binary-tree-store2.jpeg b/docs/_images/data-structure/tree/binary-tree-store2.jpeg new file mode 100644 index 0000000000..44b74f4583 Binary files /dev/null and b/docs/_images/data-structure/tree/binary-tree-store2.jpeg differ diff --git a/docs/_images/data-structure/tree/binary-tree-structure.jpeg b/docs/_images/data-structure/tree/binary-tree-structure.jpeg new file mode 100644 index 0000000000..38df7334c9 Binary files /dev/null and b/docs/_images/data-structure/tree/binary-tree-structure.jpeg differ diff --git a/docs/_images/data-structure/tree/binary-tree-three-store.jpeg b/docs/_images/data-structure/tree/binary-tree-three-store.jpeg new file mode 100644 index 0000000000..41667730a8 Binary files /dev/null and b/docs/_images/data-structure/tree/binary-tree-three-store.jpeg differ diff --git a/docs/_images/data-structure/tree/skewed-binary-tree.jpeg b/docs/_images/data-structure/tree/skewed-binary-tree.jpeg new file mode 100644 index 0000000000..658ffeb655 Binary files /dev/null and b/docs/_images/data-structure/tree/skewed-binary-tree.jpeg differ diff --git "a/docs/_images/data-structure/\345\215\225\351\223\276\350\241\250.png" "b/docs/_images/data-structure/\345\215\225\351\223\276\350\241\250.png" new file mode 100644 index 0000000000..a3435041f7 Binary files /dev/null and "b/docs/_images/data-structure/\345\215\225\351\223\276\350\241\250.png" differ diff --git "a/docs/_images/data-structure/\345\215\225\351\223\276\350\241\250\345\210\240\351\231\244.png" "b/docs/_images/data-structure/\345\215\225\351\223\276\350\241\250\345\210\240\351\231\244.png" new file mode 100644 index 0000000000..0ac9ae96b3 Binary files /dev/null and "b/docs/_images/data-structure/\345\215\225\351\223\276\350\241\250\345\210\240\351\231\244.png" differ diff --git "a/docs/_images/data-structure/\345\215\225\351\223\276\350\241\250\346\217\222\345\205\245.png" "b/docs/_images/data-structure/\345\215\225\351\223\276\350\241\250\346\217\222\345\205\245.png" new file mode 100644 index 0000000000..b14464df82 Binary files /dev/null and "b/docs/_images/data-structure/\345\215\225\351\223\276\350\241\250\346\217\222\345\205\245.png" differ diff --git "a/docs/_images/data-structure/\345\217\214\351\223\276\350\241\250.png" "b/docs/_images/data-structure/\345\217\214\351\223\276\350\241\250.png" new file mode 100644 index 0000000000..ff1865f23c Binary files /dev/null and "b/docs/_images/data-structure/\345\217\214\351\223\276\350\241\250.png" differ diff --git "a/docs/_images/data-structure/\345\217\214\351\223\276\350\241\250\345\210\240\351\231\244.png" "b/docs/_images/data-structure/\345\217\214\351\223\276\350\241\250\345\210\240\351\231\244.png" new file mode 100644 index 0000000000..7b8d0677d2 Binary files /dev/null and "b/docs/_images/data-structure/\345\217\214\351\223\276\350\241\250\345\210\240\351\231\244.png" differ diff --git "a/docs/_images/data-structure/\345\217\214\351\223\276\350\241\250\346\226\260\345\242\236.png" "b/docs/_images/data-structure/\345\217\214\351\223\276\350\241\250\346\226\260\345\242\236.png" new file mode 100644 index 0000000000..c04cea1ed1 Binary files /dev/null and "b/docs/_images/data-structure/\345\217\214\351\223\276\350\241\250\346\226\260\345\242\236.png" differ diff --git "a/docs/_images/data-structure/\345\217\214\351\223\276\350\241\250\347\273\223\346\236\204.png" "b/docs/_images/data-structure/\345\217\214\351\223\276\350\241\250\347\273\223\346\236\204.png" new file mode 100644 index 0000000000..f0d654b956 Binary files /dev/null and "b/docs/_images/data-structure/\345\217\214\351\223\276\350\241\250\347\273\223\346\236\204.png" differ diff --git "a/docs/_images/data-structure/\345\220\216\345\272\217\351\201\215\345\216\206.png" "b/docs/_images/data-structure/\345\220\216\345\272\217\351\201\215\345\216\206.png" new file mode 100644 index 0000000000..0da2d08089 Binary files /dev/null and "b/docs/_images/data-structure/\345\220\216\345\272\217\351\201\215\345\216\206.png" differ diff --git "a/docs/_images/data-structure/\345\223\221\345\205\203\347\273\223\347\202\271\345\215\225\351\223\276\350\241\250.png" "b/docs/_images/data-structure/\345\223\221\345\205\203\347\273\223\347\202\271\345\215\225\351\223\276\350\241\250.png" new file mode 100644 index 0000000000..b6b44d72c7 Binary files /dev/null and "b/docs/_images/data-structure/\345\223\221\345\205\203\347\273\223\347\202\271\345\215\225\351\223\276\350\241\250.png" differ diff --git "a/docs/_images/data-structure/\345\233\276\345\275\242\347\273\223\346\236\204.png" "b/docs/_images/data-structure/\345\233\276\345\275\242\347\273\223\346\236\204.png" new file mode 100644 index 0000000000..925c6a439d Binary files /dev/null and "b/docs/_images/data-structure/\345\233\276\345\275\242\347\273\223\346\236\204.png" differ diff --git "a/docs/_images/data-structure/\345\261\202\345\272\217\351\201\215\345\216\206.png" "b/docs/_images/data-structure/\345\261\202\345\272\217\351\201\215\345\216\206.png" new file mode 100644 index 0000000000..b46fcfa4ed Binary files /dev/null and "b/docs/_images/data-structure/\345\261\202\345\272\217\351\201\215\345\216\206.png" differ diff --git "a/docs/_images/data-structure/\346\240\221\345\275\242\347\273\223\346\236\204.png" "b/docs/_images/data-structure/\346\240\221\345\275\242\347\273\223\346\236\204.png" new file mode 100644 index 0000000000..a45fcec8bd Binary files /dev/null and "b/docs/_images/data-structure/\346\240\221\345\275\242\347\273\223\346\236\204.png" differ diff --git "a/docs/_images/data-structure/\347\272\277\345\275\242\347\273\223\346\236\204.png" "b/docs/_images/data-structure/\347\272\277\345\275\242\347\273\223\346\236\204.png" new file mode 100644 index 0000000000..1f56f0b876 Binary files /dev/null and "b/docs/_images/data-structure/\347\272\277\345\275\242\347\273\223\346\236\204.png" differ diff --git "a/docs/_images/data-structure/\351\200\222\345\275\222\346\226\220\346\263\242\351\202\243\345\245\221.png" "b/docs/_images/data-structure/\351\200\222\345\275\222\346\226\220\346\263\242\351\202\243\345\245\221.png" new file mode 100644 index 0000000000..45539619f8 Binary files /dev/null and "b/docs/_images/data-structure/\351\200\222\345\275\222\346\226\220\346\263\242\351\202\243\345\245\221.png" differ diff --git "a/docs/_images/data-structure/\351\223\276\345\274\217\347\273\223\346\236\204.png" "b/docs/_images/data-structure/\351\223\276\345\274\217\347\273\223\346\236\204.png" new file mode 100644 index 0000000000..6f6f1d0137 Binary files /dev/null and "b/docs/_images/data-structure/\351\223\276\345\274\217\347\273\223\346\236\204.png" differ diff --git "a/docs/_images/data-structure/\351\223\276\345\274\217\351\230\237\345\210\227.png" "b/docs/_images/data-structure/\351\223\276\345\274\217\351\230\237\345\210\227.png" new file mode 100644 index 0000000000..77ecfce851 Binary files /dev/null and "b/docs/_images/data-structure/\351\223\276\345\274\217\351\230\237\345\210\227.png" differ diff --git "a/docs/_images/data-structure/\351\233\206\345\220\210\347\273\223\346\236\204.png" "b/docs/_images/data-structure/\351\233\206\345\220\210\347\273\223\346\236\204.png" new file mode 100644 index 0000000000..d378642f63 Binary files /dev/null and "b/docs/_images/data-structure/\351\233\206\345\220\210\347\273\223\346\236\204.png" differ diff --git "a/docs/_images/data-structure/\351\241\272\345\272\217\347\273\223\346\236\204.png" "b/docs/_images/data-structure/\351\241\272\345\272\217\347\273\223\346\236\204.png" new file mode 100644 index 0000000000..5ea2a1afdf Binary files /dev/null and "b/docs/_images/data-structure/\351\241\272\345\272\217\347\273\223\346\236\204.png" differ diff --git a/docs/_images/design-pattern/UML-type.png b/docs/_images/design-pattern/UML-type.png deleted file mode 100644 index 35d9bd2829..0000000000 Binary files a/docs/_images/design-pattern/UML-type.png and /dev/null differ diff --git a/docs/_images/design-pattern/abstract-factory-demo.png b/docs/_images/design-pattern/abstract-factory-demo.png deleted file mode 100644 index 7162ebfb46..0000000000 Binary files a/docs/_images/design-pattern/abstract-factory-demo.png and /dev/null differ diff --git a/docs/_images/design-pattern/abstract-factory-uml.png b/docs/_images/design-pattern/abstract-factory-uml.png deleted file mode 100644 index 88ca80d309..0000000000 Binary files a/docs/_images/design-pattern/abstract-factory-uml.png and /dev/null differ diff --git a/docs/_images/design-pattern/decorator-uml.png b/docs/_images/design-pattern/decorator-uml.png deleted file mode 100644 index b534ffad1f..0000000000 Binary files a/docs/_images/design-pattern/decorator-uml.png and /dev/null differ diff --git a/docs/_images/design-pattern/factory-method-uml.png b/docs/_images/design-pattern/factory-method-uml.png deleted file mode 100644 index 7f6a0eb799..0000000000 Binary files a/docs/_images/design-pattern/factory-method-uml.png and /dev/null differ diff --git a/docs/_images/design-pattern/observer-uml.png b/docs/_images/design-pattern/observer-uml.png deleted file mode 100644 index fa14885acb..0000000000 Binary files a/docs/_images/design-pattern/observer-uml.png and /dev/null differ diff --git a/docs/_images/design-pattern/responsibility-pattern-uml.png b/docs/_images/design-pattern/responsibility-pattern-uml.png deleted file mode 100644 index abc7ce238e..0000000000 Binary files a/docs/_images/design-pattern/responsibility-pattern-uml.png and /dev/null differ diff --git a/docs/_images/design-pattern/simple-factory-uml.png b/docs/_images/design-pattern/simple-factory-uml.png deleted file mode 100644 index f82cdab658..0000000000 Binary files a/docs/_images/design-pattern/simple-factory-uml.png and /dev/null differ diff --git a/docs/_images/distribution/cap.jpg b/docs/_images/distribution/cap.jpg new file mode 100644 index 0000000000..81afb6de8c Binary files /dev/null and b/docs/_images/distribution/cap.jpg differ diff --git a/docs/_images/message-queue/mesage-what.png b/docs/_images/distribution/message-queue/mesage-what.png similarity index 100% rename from docs/_images/message-queue/mesage-what.png rename to docs/_images/distribution/message-queue/mesage-what.png diff --git a/docs/_images/message-queue/message-acaticemq.png b/docs/_images/distribution/message-queue/message-acaticemq.png similarity index 100% rename from docs/_images/message-queue/message-acaticemq.png rename to docs/_images/distribution/message-queue/message-acaticemq.png diff --git a/docs/_images/message-queue/message-kafka.png b/docs/_images/distribution/message-queue/message-kafka.png similarity index 100% rename from docs/_images/message-queue/message-kafka.png rename to docs/_images/distribution/message-queue/message-kafka.png diff --git a/docs/_images/message-queue/message-overview.png b/docs/_images/distribution/message-queue/message-overview.png similarity index 100% rename from docs/_images/message-queue/message-overview.png rename to docs/_images/distribution/message-queue/message-overview.png diff --git a/docs/_images/message-queue/message-pubsub-mode.png b/docs/_images/distribution/message-queue/message-pubsub-mode.png similarity index 100% rename from docs/_images/message-queue/message-pubsub-mode.png rename to docs/_images/distribution/message-queue/message-pubsub-mode.png diff --git a/docs/_images/message-queue/message-push-pull.png b/docs/_images/distribution/message-queue/message-push-pull.png similarity index 100% rename from docs/_images/message-queue/message-push-pull.png rename to docs/_images/distribution/message-queue/message-push-pull.png diff --git a/docs/_images/message-queue/message-queue-mode.png b/docs/_images/distribution/message-queue/message-queue-mode.png similarity index 100% rename from docs/_images/message-queue/message-queue-mode.png rename to docs/_images/distribution/message-queue/message-queue-mode.png diff --git a/docs/_images/message-queue/message-queue.png b/docs/_images/distribution/message-queue/message-queue.png similarity index 100% rename from docs/_images/message-queue/message-queue.png rename to docs/_images/distribution/message-queue/message-queue.png diff --git a/docs/_images/message-queue/message-rabbitmq.png b/docs/_images/distribution/message-queue/message-rabbitmq.png similarity index 100% rename from docs/_images/message-queue/message-rabbitmq.png rename to docs/_images/distribution/message-queue/message-rabbitmq.png diff --git a/docs/_images/message-queue/message-rocketmq.png b/docs/_images/distribution/message-queue/message-rocketmq.png similarity index 100% rename from docs/_images/message-queue/message-rocketmq.png rename to docs/_images/distribution/message-queue/message-rocketmq.png diff --git a/docs/_images/message-queue/message-server-mode.png b/docs/_images/distribution/message-queue/message-server-mode.png similarity index 100% rename from docs/_images/message-queue/message-server-mode.png rename to docs/_images/distribution/message-queue/message-server-mode.png diff --git a/docs/_images/message-queue/message-user-1.png b/docs/_images/distribution/message-queue/message-user-1.png similarity index 100% rename from docs/_images/message-queue/message-user-1.png rename to docs/_images/distribution/message-queue/message-user-1.png diff --git a/docs/_images/message-queue/message-user-2.png b/docs/_images/distribution/message-queue/message-user-2.png similarity index 100% rename from docs/_images/message-queue/message-user-2.png rename to docs/_images/distribution/message-queue/message-user-2.png diff --git a/docs/_images/message-queue/message-user-3.png b/docs/_images/distribution/message-queue/message-user-3.png similarity index 100% rename from docs/_images/message-queue/message-user-3.png rename to docs/_images/distribution/message-queue/message-user-3.png diff --git a/docs/_images/message-queue/message-user-4.png b/docs/_images/distribution/message-queue/message-user-4.png similarity index 100% rename from docs/_images/message-queue/message-user-4.png rename to docs/_images/distribution/message-queue/message-user-4.png diff --git a/docs/_images/distribution/message-queue/mq-one2many.jpg b/docs/_images/distribution/message-queue/mq-one2many.jpg new file mode 100644 index 0000000000..eea45d340f Binary files /dev/null and b/docs/_images/distribution/message-queue/mq-one2many.jpg differ diff --git a/docs/_images/distribution/message-queue/mq-point2point.jpg b/docs/_images/distribution/message-queue/mq-point2point.jpg new file mode 100644 index 0000000000..b33db11faa Binary files /dev/null and b/docs/_images/distribution/message-queue/mq-point2point.jpg differ diff --git a/docs/_images/message-queue/mq_index.png b/docs/_images/distribution/message-queue/mq_index.png similarity index 100% rename from docs/_images/message-queue/mq_index.png rename to docs/_images/distribution/message-queue/mq_index.png diff --git a/docs/_images/distribution/message-queue/mq_overview.png b/docs/_images/distribution/message-queue/mq_overview.png new file mode 100644 index 0000000000..99d5762400 Binary files /dev/null and b/docs/_images/distribution/message-queue/mq_overview.png differ diff --git "a/docs/_images/message-queue/\345\260\217\347\213\227\351\222\261\351\222\261.png" "b/docs/_images/distribution/message-queue/\345\260\217\347\213\227\351\222\261\351\222\261.png" similarity index 100% rename from "docs/_images/message-queue/\345\260\217\347\213\227\351\222\261\351\222\261.png" rename to "docs/_images/distribution/message-queue/\345\260\217\347\213\227\351\222\261\351\222\261.png" diff --git a/docs/_images/distribution/zab-commit.png b/docs/_images/distribution/zab-commit.png new file mode 100644 index 0000000000..89650a4e01 Binary files /dev/null and b/docs/_images/distribution/zab-commit.png differ diff --git a/docs/_images/zookeeper/2PC.png b/docs/_images/distribution/zookeeper/2PC.png similarity index 100% rename from docs/_images/zookeeper/2PC.png rename to docs/_images/distribution/zookeeper/2PC.png diff --git a/docs/_images/distribution/zookeeper/3PC.png b/docs/_images/distribution/zookeeper/3PC.png new file mode 100644 index 0000000000..95c7a9c735 Binary files /dev/null and b/docs/_images/distribution/zookeeper/3PC.png differ diff --git a/docs/_images/distribution/zookeeper/paxos.png b/docs/_images/distribution/zookeeper/paxos.png new file mode 100644 index 0000000000..07fe08e5a8 Binary files /dev/null and b/docs/_images/distribution/zookeeper/paxos.png differ diff --git a/docs/_images/distribution/zookeeper/zab-commit.png b/docs/_images/distribution/zookeeper/zab-commit.png new file mode 100644 index 0000000000..b08dc54ec2 Binary files /dev/null and b/docs/_images/distribution/zookeeper/zab-commit.png differ diff --git a/docs/_images/distribution/zookeeper/zab.png b/docs/_images/distribution/zookeeper/zab.png new file mode 100644 index 0000000000..5bf2483a8b Binary files /dev/null and b/docs/_images/distribution/zookeeper/zab.png differ diff --git a/docs/_images/zookeeper/zk-conf.png b/docs/_images/distribution/zookeeper/zk-conf.png similarity index 100% rename from docs/_images/zookeeper/zk-conf.png rename to docs/_images/distribution/zookeeper/zk-conf.png diff --git a/docs/_images/zookeeper/zk-elect.jpg b/docs/_images/distribution/zookeeper/zk-elect.jpg similarity index 100% rename from docs/_images/zookeeper/zk-elect.jpg rename to docs/_images/distribution/zookeeper/zk-elect.jpg diff --git a/docs/_images/zookeeper/zk-listener.png b/docs/_images/distribution/zookeeper/zk-listener.png similarity index 100% rename from docs/_images/zookeeper/zk-listener.png rename to docs/_images/distribution/zookeeper/zk-listener.png diff --git a/docs/_images/zookeeper/zk-loadbalancing.png b/docs/_images/distribution/zookeeper/zk-loadbalancing.png similarity index 100% rename from docs/_images/zookeeper/zk-loadbalancing.png rename to docs/_images/distribution/zookeeper/zk-loadbalancing.png diff --git a/docs/_images/distribution/zookeeper/zk-manage-node.jpg b/docs/_images/distribution/zookeeper/zk-manage-node.jpg new file mode 100644 index 0000000000..b222ef0e21 Binary files /dev/null and b/docs/_images/distribution/zookeeper/zk-manage-node.jpg differ diff --git a/docs/_images/zookeeper/zk-unify-conf.png b/docs/_images/distribution/zookeeper/zk-unify-conf.png similarity index 100% rename from docs/_images/zookeeper/zk-unify-conf.png rename to docs/_images/distribution/zookeeper/zk-unify-conf.png diff --git a/docs/_images/distribution/zookeeper/zk-work.png b/docs/_images/distribution/zookeeper/zk-work.png new file mode 100644 index 0000000000..ab33287294 Binary files /dev/null and b/docs/_images/distribution/zookeeper/zk-work.png differ diff --git a/docs/_images/zookeeper/zk-write-data.png b/docs/_images/distribution/zookeeper/zk-write-data.png similarity index 100% rename from docs/_images/zookeeper/zk-write-data.png rename to docs/_images/distribution/zookeeper/zk-write-data.png diff --git a/docs/_images/zookeeper/zk-znode.png b/docs/_images/distribution/zookeeper/zk-znode.png similarity index 100% rename from docs/_images/zookeeper/zk-znode.png rename to docs/_images/distribution/zookeeper/zk-znode.png diff --git a/docs/_images/end.jpg b/docs/_images/end.jpg new file mode 100644 index 0000000000..3a6ae42eda Binary files /dev/null and b/docs/_images/end.jpg differ diff --git a/docs/_images/java/.DS_Store b/docs/_images/java/.DS_Store new file mode 100644 index 0000000000..c56bcdd79b Binary files /dev/null and b/docs/_images/java/.DS_Store differ diff --git a/docs/_images/java/JVM/java-object.jpg b/docs/_images/java/JVM/java-object.jpg deleted file mode 100644 index bc17bccfaf..0000000000 Binary files a/docs/_images/java/JVM/java-object.jpg and /dev/null differ diff --git a/docs/_images/java/JVM/jvm-bytecode.png b/docs/_images/java/JVM/jvm-bytecode.png deleted file mode 100644 index 19c7898e17..0000000000 Binary files a/docs/_images/java/JVM/jvm-bytecode.png and /dev/null differ diff --git a/docs/_images/java/JVM/jvm-class-load.png b/docs/_images/java/JVM/jvm-class-load.png deleted file mode 100644 index e4f7fd62b6..0000000000 Binary files a/docs/_images/java/JVM/jvm-class-load.png and /dev/null differ diff --git a/docs/_images/java/JVM/jvm-compile.png b/docs/_images/java/JVM/jvm-compile.png deleted file mode 100644 index 3429c6dcc9..0000000000 Binary files a/docs/_images/java/JVM/jvm-compile.png and /dev/null differ diff --git a/docs/_images/java/JVM/jvm-dynamic-linking.png b/docs/_images/java/JVM/jvm-dynamic-linking.png deleted file mode 100644 index d706307643..0000000000 Binary files a/docs/_images/java/JVM/jvm-dynamic-linking.png and /dev/null differ diff --git a/docs/_images/java/JVM/jvm-framework.png b/docs/_images/java/JVM/jvm-framework.png deleted file mode 100644 index e152cf5509..0000000000 Binary files a/docs/_images/java/JVM/jvm-framework.png and /dev/null differ diff --git a/docs/_images/java/JVM/jvm-javap.png b/docs/_images/java/JVM/jvm-javap.png deleted file mode 100644 index 240b310de9..0000000000 Binary files a/docs/_images/java/JVM/jvm-javap.png and /dev/null differ diff --git a/docs/_images/java/JVM/jvm-jdk-jre.png b/docs/_images/java/JVM/jvm-jdk-jre.png deleted file mode 100644 index a7e43e5232..0000000000 Binary files a/docs/_images/java/JVM/jvm-jdk-jre.png and /dev/null differ diff --git a/docs/_images/java/JVM/jvm-pc-counter.png b/docs/_images/java/JVM/jvm-pc-counter.png deleted file mode 100644 index 2a03b82c4d..0000000000 Binary files a/docs/_images/java/JVM/jvm-pc-counter.png and /dev/null differ diff --git a/docs/_images/java/JVM/jvm-stack-frame.png b/docs/_images/java/JVM/jvm-stack-frame.png deleted file mode 100644 index 53b6fd1e63..0000000000 Binary files a/docs/_images/java/JVM/jvm-stack-frame.png and /dev/null differ diff --git "a/docs/_images/java/JVM/\345\240\206\345\206\205\345\255\230\347\251\272\351\227\264.png" "b/docs/_images/java/JVM/\345\240\206\345\206\205\345\255\230\347\251\272\351\227\264.png" deleted file mode 100644 index 0ac7cd1298..0000000000 Binary files "a/docs/_images/java/JVM/\345\240\206\345\206\205\345\255\230\347\251\272\351\227\264.png" and /dev/null differ diff --git "a/docs/_images/java/JVM/\345\244\215\345\210\266\347\256\227\346\263\225.png" "b/docs/_images/java/JVM/\345\244\215\345\210\266\347\256\227\346\263\225.png" deleted file mode 100644 index 85d6e5cffd..0000000000 Binary files "a/docs/_images/java/JVM/\345\244\215\345\210\266\347\256\227\346\263\225.png" and /dev/null differ diff --git a/docs/_images/java/Throwable.jpg b/docs/_images/java/Throwable.jpg new file mode 100644 index 0000000000..9549042b45 Binary files /dev/null and b/docs/_images/java/Throwable.jpg differ diff --git a/docs/_images/java/collection/JDK1.7_ConcurrentHashMap.jpg b/docs/_images/java/collection/JDK1.7_ConcurrentHashMap.jpg new file mode 100644 index 0000000000..9aec3723ac Binary files /dev/null and b/docs/_images/java/collection/JDK1.7_ConcurrentHashMap.jpg differ diff --git a/docs/_images/java/collection/JDK1.8_ConcurrentHashMap.jpg b/docs/_images/java/collection/JDK1.8_ConcurrentHashMap.jpg new file mode 100644 index 0000000000..28f574279b Binary files /dev/null and b/docs/_images/java/collection/JDK1.8_ConcurrentHashMap.jpg differ diff --git a/docs/_images/java/collection/hashMap_threads.png b/docs/_images/java/collection/hashMap_threads.png new file mode 100644 index 0000000000..4f77ecf84e Binary files /dev/null and b/docs/_images/java/collection/hashMap_threads.png differ diff --git a/docs/_images/java/juc/juc.png b/docs/_images/java/juc/juc.png new file mode 100644 index 0000000000..0a6495afc0 Binary files /dev/null and b/docs/_images/java/juc/juc.png differ diff --git a/docs/_images/java/juc/queue.png b/docs/_images/java/juc/queue.png new file mode 100644 index 0000000000..4e90b96bb9 Binary files /dev/null and b/docs/_images/java/juc/queue.png differ diff --git a/docs/_images/java/juc/thread-pool-reject.png b/docs/_images/java/juc/thread-pool-reject.png new file mode 100644 index 0000000000..5e8629ad13 Binary files /dev/null and b/docs/_images/java/juc/thread-pool-reject.png differ diff --git a/docs/_images/java/rmi-dmeo.png b/docs/_images/java/rmi-dmeo.png new file mode 100644 index 0000000000..dee0225764 Binary files /dev/null and b/docs/_images/java/rmi-dmeo.png differ diff --git a/docs/_images/leetcode-hot100-array.png b/docs/_images/leetcode-hot100-array.png new file mode 100644 index 0000000000..31c5607bb7 Binary files /dev/null and b/docs/_images/leetcode-hot100-array.png differ diff --git a/docs/_images/message-queue/Kafka/controller-leader.png b/docs/_images/message-queue/Kafka/controller-leader.png deleted file mode 100644 index cbab054742..0000000000 Binary files a/docs/_images/message-queue/Kafka/controller-leader.png and /dev/null differ diff --git a/docs/_images/message-queue/Kafka/interceptor-demo.png b/docs/_images/message-queue/Kafka/interceptor-demo.png deleted file mode 100644 index 804d1ce455..0000000000 Binary files a/docs/_images/message-queue/Kafka/interceptor-demo.png and /dev/null differ diff --git a/docs/_images/message-queue/Kafka/kafka-ack-slg.png b/docs/_images/message-queue/Kafka/kafka-ack-slg.png deleted file mode 100644 index 494769ff03..0000000000 Binary files a/docs/_images/message-queue/Kafka/kafka-ack-slg.png and /dev/null differ diff --git a/docs/_images/message-queue/Kafka/kafka-ack=-1.png b/docs/_images/message-queue/Kafka/kafka-ack=-1.png deleted file mode 100644 index 0647f7439d..0000000000 Binary files a/docs/_images/message-queue/Kafka/kafka-ack=-1.png and /dev/null differ diff --git a/docs/_images/message-queue/Kafka/kafka-ack=1.png b/docs/_images/message-queue/Kafka/kafka-ack=1.png deleted file mode 100644 index 95be4d96a5..0000000000 Binary files a/docs/_images/message-queue/Kafka/kafka-ack=1.png and /dev/null differ diff --git a/docs/_images/message-queue/Kafka/kafka-apis.png b/docs/_images/message-queue/Kafka/kafka-apis.png deleted file mode 100644 index db6053ccc2..0000000000 Binary files a/docs/_images/message-queue/Kafka/kafka-apis.png and /dev/null differ diff --git a/docs/_images/message-queue/Kafka/kafka-consume-group.png b/docs/_images/message-queue/Kafka/kafka-consume-group.png deleted file mode 100644 index c3931c7a75..0000000000 Binary files a/docs/_images/message-queue/Kafka/kafka-consume-group.png and /dev/null differ diff --git a/docs/_images/message-queue/Kafka/kafka-leo.png b/docs/_images/message-queue/Kafka/kafka-leo.png deleted file mode 100644 index d5f2b6e9c4..0000000000 Binary files a/docs/_images/message-queue/Kafka/kafka-leo.png and /dev/null differ diff --git a/docs/_images/message-queue/Kafka/kafka-partition.jpg b/docs/_images/message-queue/Kafka/kafka-partition.jpg deleted file mode 100644 index e8cd1de32b..0000000000 Binary files a/docs/_images/message-queue/Kafka/kafka-partition.jpg and /dev/null differ diff --git a/docs/_images/message-queue/Kafka/kafka-producer-thread.png b/docs/_images/message-queue/Kafka/kafka-producer-thread.png deleted file mode 100644 index 1ecda6e19c..0000000000 Binary files a/docs/_images/message-queue/Kafka/kafka-producer-thread.png and /dev/null differ diff --git a/docs/_images/message-queue/Kafka/kafka-segement.jpg b/docs/_images/message-queue/Kafka/kafka-segement.jpg deleted file mode 100644 index 8bc98a8dd6..0000000000 Binary files a/docs/_images/message-queue/Kafka/kafka-segement.jpg and /dev/null differ diff --git a/docs/_images/message-queue/Kafka/kafka-start.png b/docs/_images/message-queue/Kafka/kafka-start.png deleted file mode 100644 index 1080ae6a7b..0000000000 Binary files a/docs/_images/message-queue/Kafka/kafka-start.png and /dev/null differ diff --git a/docs/_images/message-queue/Kafka/kafka-streams-data-clean.png b/docs/_images/message-queue/Kafka/kafka-streams-data-clean.png deleted file mode 100644 index 3c6109c9a0..0000000000 Binary files a/docs/_images/message-queue/Kafka/kafka-streams-data-clean.png and /dev/null differ diff --git a/docs/_images/message-queue/Kafka/kafka-workflow.jpg b/docs/_images/message-queue/Kafka/kafka-workflow.jpg deleted file mode 100644 index 49ed343ac4..0000000000 Binary files a/docs/_images/message-queue/Kafka/kafka-workflow.jpg and /dev/null differ diff --git a/docs/_images/message-queue/Kafka/kafka-write-flow.png b/docs/_images/message-queue/Kafka/kafka-write-flow.png deleted file mode 100644 index 487d735acb..0000000000 Binary files a/docs/_images/message-queue/Kafka/kafka-write-flow.png and /dev/null differ diff --git a/docs/_images/message-queue/Kafka/kakfa-java-demo.png b/docs/_images/message-queue/Kafka/kakfa-java-demo.png deleted file mode 100644 index 491cb41ad0..0000000000 Binary files a/docs/_images/message-queue/Kafka/kakfa-java-demo.png and /dev/null differ diff --git a/docs/_images/message-queue/Kafka/kakfa-principle.png b/docs/_images/message-queue/Kafka/kakfa-principle.png deleted file mode 100644 index c1ce1730bc..0000000000 Binary files a/docs/_images/message-queue/Kafka/kakfa-principle.png and /dev/null differ diff --git a/docs/_images/message-queue/Kafka/kakfa-streams-flow.png b/docs/_images/message-queue/Kafka/kakfa-streams-flow.png deleted file mode 100644 index 180c313609..0000000000 Binary files a/docs/_images/message-queue/Kafka/kakfa-streams-flow.png and /dev/null differ diff --git a/docs/_images/message-queue/Kafka/log_anatomy.png b/docs/_images/message-queue/Kafka/log_anatomy.png deleted file mode 100644 index a649499926..0000000000 Binary files a/docs/_images/message-queue/Kafka/log_anatomy.png and /dev/null differ diff --git a/docs/_images/message-queue/Kafka/log_consumer.png b/docs/_images/message-queue/Kafka/log_consumer.png deleted file mode 100644 index fbc45f2060..0000000000 Binary files a/docs/_images/message-queue/Kafka/log_consumer.png and /dev/null differ diff --git a/docs/_images/message-queue/Kafka/mq.png b/docs/_images/message-queue/Kafka/mq.png deleted file mode 100644 index 0eb2e236d7..0000000000 Binary files a/docs/_images/message-queue/Kafka/mq.png and /dev/null differ diff --git a/docs/_images/message-queue/Kafka/sumer-groups.png b/docs/_images/message-queue/Kafka/sumer-groups.png deleted file mode 100644 index 16fe2936cb..0000000000 Binary files a/docs/_images/message-queue/Kafka/sumer-groups.png and /dev/null differ diff --git a/docs/_images/message-queue/Kafka/zero-copy.png b/docs/_images/message-queue/Kafka/zero-copy.png deleted file mode 100644 index f9ac3ba2b3..0000000000 Binary files a/docs/_images/message-queue/Kafka/zero-copy.png and /dev/null differ diff --git a/docs/_images/message-queue/Kafka/zookeeper-store.png b/docs/_images/message-queue/Kafka/zookeeper-store.png deleted file mode 100644 index 143f682a4a..0000000000 Binary files a/docs/_images/message-queue/Kafka/zookeeper-store.png and /dev/null differ diff --git a/docs/_images/mysql/.DS_Store b/docs/_images/mysql/.DS_Store new file mode 100644 index 0000000000..0e7bf57afa Binary files /dev/null and b/docs/_images/mysql/.DS_Store differ diff --git a/docs/_images/mysql/COMPACT-And-REDUNDANT-Row-Format.jpg b/docs/_images/mysql/COMPACT-And-REDUNDANT-Row-Format.jpg deleted file mode 100644 index ac01b41a4c..0000000000 Binary files a/docs/_images/mysql/COMPACT-And-REDUNDANT-Row-Format.jpg and /dev/null differ diff --git a/docs/_images/mysql/Index-advantage.png b/docs/_images/mysql/Index-advantage.png new file mode 100644 index 0000000000..1e6e27fe58 Binary files /dev/null and b/docs/_images/mysql/Index-advantage.png differ diff --git a/docs/_images/mysql/Infimum-Rows-Supremum.jpg b/docs/_images/mysql/Infimum-Rows-Supremum.jpg deleted file mode 100644 index 2894d4b105..0000000000 Binary files a/docs/_images/mysql/Infimum-Rows-Supremum.jpg and /dev/null differ diff --git a/docs/_images/mysql/InnoDB-B-Tree-Node.jpg b/docs/_images/mysql/InnoDB-B-Tree-Node.jpg deleted file mode 100644 index 22409dfe63..0000000000 Binary files a/docs/_images/mysql/InnoDB-B-Tree-Node.jpg and /dev/null differ diff --git a/docs/_images/mysql/MySQL-B+Tree-store.png b/docs/_images/mysql/MySQL-B+Tree-store.png new file mode 100644 index 0000000000..313b63dac7 Binary files /dev/null and b/docs/_images/mysql/MySQL-B+Tree-store.png differ diff --git a/docs/_images/mysql/MySQL-B-Tree.png b/docs/_images/mysql/MySQL-B-Tree.png new file mode 100644 index 0000000000..6020f054ad Binary files /dev/null and b/docs/_images/mysql/MySQL-B-Tree.png differ diff --git a/docs/_images/mysql/MySQL-InnoDB-Index-primary.png b/docs/_images/mysql/MySQL-InnoDB-Index-primary.png new file mode 100644 index 0000000000..d27ab2d139 Binary files /dev/null and b/docs/_images/mysql/MySQL-InnoDB-Index-primary.png differ diff --git a/docs/_images/mysql/MySQL-InnoDB-Index.png b/docs/_images/mysql/MySQL-InnoDB-Index.png new file mode 100644 index 0000000000..fe85c75eac Binary files /dev/null and b/docs/_images/mysql/MySQL-InnoDB-Index.png differ diff --git a/docs/_images/mysql/MySQL-MyISAM-Index.png b/docs/_images/mysql/MySQL-MyISAM-Index.png new file mode 100644 index 0000000000..51a83c831f Binary files /dev/null and b/docs/_images/mysql/MySQL-MyISAM-Index.png differ diff --git a/docs/_images/mysql/MySQL-search-tree.png b/docs/_images/mysql/MySQL-search-tree.png new file mode 100644 index 0000000000..98d2779252 Binary files /dev/null and b/docs/_images/mysql/MySQL-search-tree.png differ diff --git a/docs/_images/mysql/bTree.png b/docs/_images/mysql/bTree.png deleted file mode 100644 index c25ab404d3..0000000000 Binary files a/docs/_images/mysql/bTree.png and /dev/null differ diff --git a/docs/_images/mysql/count+1-solve.png b/docs/_images/mysql/count+1-solve.png new file mode 100644 index 0000000000..ca8f879332 Binary files /dev/null and b/docs/_images/mysql/count+1-solve.png differ diff --git a/docs/_images/mysql/count+1.png b/docs/_images/mysql/count+1.png new file mode 100644 index 0000000000..92f37c68b7 Binary files /dev/null and b/docs/_images/mysql/count+1.png differ diff --git a/docs/_images/mysql/count+1_1.png b/docs/_images/mysql/count+1_1.png new file mode 100644 index 0000000000..22d4654f31 Binary files /dev/null and b/docs/_images/mysql/count+1_1.png differ diff --git a/docs/_images/mysql/count-problem.png b/docs/_images/mysql/count-problem.png new file mode 100644 index 0000000000..6d553c1c62 Binary files /dev/null and b/docs/_images/mysql/count-problem.png differ diff --git a/docs/_images/mysql/expalin.jpg b/docs/_images/mysql/expalin.jpg deleted file mode 100644 index da052f7fe6..0000000000 Binary files a/docs/_images/mysql/expalin.jpg and /dev/null differ diff --git a/docs/_images/mysql/explain-1.png b/docs/_images/mysql/explain-1.png new file mode 100644 index 0000000000..6357a622a0 Binary files /dev/null and b/docs/_images/mysql/explain-1.png differ diff --git a/docs/_images/mysql/explain-2.png b/docs/_images/mysql/explain-2.png new file mode 100644 index 0000000000..de193e71d8 Binary files /dev/null and b/docs/_images/mysql/explain-2.png differ diff --git a/docs/_images/mysql/explain-key.png b/docs/_images/mysql/explain-key.png deleted file mode 100644 index 5d4f617e2f..0000000000 Binary files a/docs/_images/mysql/explain-key.png and /dev/null differ diff --git a/docs/_images/mysql/index-ICP.png b/docs/_images/mysql/index-ICP.png new file mode 100644 index 0000000000..ca84b2cd14 Binary files /dev/null and b/docs/_images/mysql/index-ICP.png differ diff --git a/docs/_images/mysql/mysql-framework1.png b/docs/_images/mysql/mysql-framework1.png deleted file mode 100644 index b5b52403c4..0000000000 Binary files a/docs/_images/mysql/mysql-framework1.png and /dev/null differ diff --git a/docs/_images/mysql/mysql-log-banner.png b/docs/_images/mysql/mysql-log-banner.png new file mode 100644 index 0000000000..0d368dc892 Binary files /dev/null and b/docs/_images/mysql/mysql-log-banner.png differ diff --git a/docs/_images/mysql/mysql.png b/docs/_images/mysql/mysql.png deleted file mode 100644 index 535d627b70..0000000000 Binary files a/docs/_images/mysql/mysql.png and /dev/null differ diff --git a/docs/_images/mysql/optimization-orderby.png b/docs/_images/mysql/optimization-orderby.png deleted file mode 100644 index d13f3656b6..0000000000 Binary files a/docs/_images/mysql/optimization-orderby.png and /dev/null differ diff --git a/docs/_images/mysql/pre-index.png b/docs/_images/mysql/pre-index.png new file mode 100644 index 0000000000..ffc8e22317 Binary files /dev/null and b/docs/_images/mysql/pre-index.png differ diff --git a/docs/_images/mysql/search-index-demo.png b/docs/_images/mysql/search-index-demo.png new file mode 100644 index 0000000000..d99baa5d84 Binary files /dev/null and b/docs/_images/mysql/search-index-demo.png differ diff --git a/docs/_images/mysql/tablespace-segment-extent-page-row.jpg b/docs/_images/mysql/tablespace-segment-extent-page-row.jpg deleted file mode 100644 index 66db246165..0000000000 Binary files a/docs/_images/mysql/tablespace-segment-extent-page-row.jpg and /dev/null differ diff --git a/docs/_images/redis/nosql-index.jpg b/docs/_images/redis/nosql-index.jpg deleted file mode 100644 index 9eb9900db9..0000000000 Binary files a/docs/_images/redis/nosql-index.jpg and /dev/null differ diff --git a/docs/_images/redis/nosqlwhy.png b/docs/_images/redis/nosqlwhy.png deleted file mode 100644 index 0b299c4c19..0000000000 Binary files a/docs/_images/redis/nosqlwhy.png and /dev/null differ diff --git a/docs/_images/redis/redis-aof-conf.jpg b/docs/_images/redis/redis-aof-conf.jpg deleted file mode 100644 index a12c68cdaa..0000000000 Binary files a/docs/_images/redis/redis-aof-conf.jpg and /dev/null differ diff --git a/docs/_images/redis/redis-aof.png b/docs/_images/redis/redis-aof.png deleted file mode 100644 index 8f853bb9e8..0000000000 Binary files a/docs/_images/redis/redis-aof.png and /dev/null differ diff --git a/docs/_images/redis/redis-cap.png b/docs/_images/redis/redis-cap.png deleted file mode 100644 index 9a16045c7f..0000000000 Binary files a/docs/_images/redis/redis-cap.png and /dev/null differ diff --git a/docs/_images/redis/redis-hash.jpg b/docs/_images/redis/redis-hash.jpg deleted file mode 100644 index 2f95be5e61..0000000000 Binary files a/docs/_images/redis/redis-hash.jpg and /dev/null differ diff --git a/docs/_images/redis/redis-rdb.png b/docs/_images/redis/redis-rdb.png deleted file mode 100644 index 1bd57be20f..0000000000 Binary files a/docs/_images/redis/redis-rdb.png and /dev/null differ diff --git a/docs/_images/redis/redis-snapshotting.png b/docs/_images/redis/redis-snapshotting.png deleted file mode 100644 index 2ee487f49e..0000000000 Binary files a/docs/_images/redis/redis-snapshotting.png and /dev/null differ diff --git a/docs/_images/redis/redis-string.jpg b/docs/_images/redis/redis-string.jpg deleted file mode 100644 index 3c4625ae76..0000000000 Binary files a/docs/_images/redis/redis-string.jpg and /dev/null differ diff --git a/docs/_images/redis/redis-transaction-case1.png b/docs/_images/redis/redis-transaction-case1.png deleted file mode 100644 index cc8aac5654..0000000000 Binary files a/docs/_images/redis/redis-transaction-case1.png and /dev/null differ diff --git a/docs/_images/redis/redis-transaction-case2.png b/docs/_images/redis/redis-transaction-case2.png deleted file mode 100644 index cb6a37ba7c..0000000000 Binary files a/docs/_images/redis/redis-transaction-case2.png and /dev/null differ diff --git a/docs/_images/redis/redis-transaction-case3.png b/docs/_images/redis/redis-transaction-case3.png deleted file mode 100644 index 5d7cebea51..0000000000 Binary files a/docs/_images/redis/redis-transaction-case3.png and /dev/null differ diff --git a/docs/_images/redis/redis-transaction-case4.png b/docs/_images/redis/redis-transaction-case4.png deleted file mode 100644 index 4f07372737..0000000000 Binary files a/docs/_images/redis/redis-transaction-case4.png and /dev/null differ diff --git a/docs/_images/redis/redis-transaction-watch1.png b/docs/_images/redis/redis-transaction-watch1.png deleted file mode 100644 index b9b0d7bb2e..0000000000 Binary files a/docs/_images/redis/redis-transaction-watch1.png and /dev/null differ diff --git a/docs/_images/redis/redis-transaction-watch2.png b/docs/_images/redis/redis-transaction-watch2.png deleted file mode 100644 index e82434a228..0000000000 Binary files a/docs/_images/redis/redis-transaction-watch2.png and /dev/null differ diff --git a/docs/_images/redis/redis-transaction-watch3.png b/docs/_images/redis/redis-transaction-watch3.png deleted file mode 100644 index 428da720cf..0000000000 Binary files a/docs/_images/redis/redis-transaction-watch3.png and /dev/null differ diff --git a/docs/_images/redis/redis-watch-demo.jpg b/docs/_images/redis/redis-watch-demo.jpg deleted file mode 100644 index 18668f42f4..0000000000 Binary files a/docs/_images/redis/redis-watch-demo.jpg and /dev/null differ diff --git a/docs/_images/zookeeper/3PC.png b/docs/_images/zookeeper/3PC.png deleted file mode 100644 index 9bb07d2f45..0000000000 Binary files a/docs/_images/zookeeper/3PC.png and /dev/null differ diff --git a/docs/apis/README.md b/docs/apis/README.md deleted file mode 100644 index f0dd970e0d..0000000000 --- a/docs/apis/README.md +++ /dev/null @@ -1,63 +0,0 @@ -# aiWARE APIs - -Veritone's full suite of APIs enables you to easily add cognitive functionality and intelligent features to your custom solution. - -![Integration](../overview/architecture-overview/stack-integration.svg) - -Our API is built around the GraphQL paradigm to provide a more efficient way to deliver data with greater flexibility than a traditional REST approach. -GraphQL is a powerful query language that operates via a single endpoint using conventional HTTP requests and returning JSON responses. -The JSON-based structure not only lets you call multiple nested resources in a single query, it also allows you to define requests so as to specify the exact data that you want sent back. -(No more sifting through a "kitchen sink" REST response. -*You* decide which fields to aggregate in the response.) - -## Base URL {docsify-ignore} - -Veritone API uses a single endpoint for making ad-hoc requests and to integrate API into third-party applications. All requests must be HTTP POST to [https://api.veritone.com/v3/graphql](https://api.veritone.com/v3/graphql) with *application/json* encoded bodies. - - -> ### Base URL for Engine Development -> -> Engines in Veritone follow a different endpoint protocol for accessing the API. -> To ensure successful API execution across different environments, the API base URL is passed in the Task Payload at engine runtime. -> Once your engine receives the Task Payload, use the `veritoneApiBaseUrl` field value to construct the GraphQL endpoint for requests. -> For example: -> -> ```javascript -> const apiUrl = payload.veritoneApiBaseUrl + '/v3/graphql'; -> ``` -> -> It’s important that the standard API endpoint (referenced above) is not hard coded in your engine and that only the base URL provided in the Task Payload is used to make requests. -> For more information, see [Building Engines](/developer/engines/). - - -### Authentication - -All API requests must be authenticated using an API Token. To authenticate your calls, provide the token in the *Authentication* header of the request with a value *Bearer \*. Requests made without this header or with an invalid token will return an error code. For information about generating an API Token, see [Authentication](/apis/authentication). - -### GraphiQL Playground - -To make it easier to explore, write, and test the API, we set up [GraphiQL](https://api.veritone.com/v3/graphiql) — an interactive playground that gives you a code editor with autocomplete, validation, and syntax error highlighting features. Use the GraphiQL interface to construct and execute queries, experiment with different schema modifications, and browse documentation. In addition, GraphiQL bakes authorization right into the schema and automatically passes the *Authentication* header with a valid token when you’re logged into the Veritone system. - -Veritone’s GraphiQL interface is the recommended method for ad-hoc API requests, but calls can be made using any HTTP client. All requests must be HTTP POST to the https://api.veritone.com/v3/graphql endpoint with the *query* parameter and *application/json* encoded bodies. If you’re using a raw HTTP client, the query body contents must be sent in a string with all quotes escaped (see [GraphQL Basics](/apis/tutorials/graphql-basics) for more information). - -### API Documentation - -For full Veritone API documentation, see our [GraphQL docs](https://api.veritone.com/v3/graphqldocs/). - -### We’re here to help! - -We’ve tried to pack our API section with detailed information and a variety of examples to assist you in your development. But if you have questions or need assistance, don’t hesitate to reach out to our Developer Support Team by [email](mailto:devsupport@veritone.com) or on [Slack](https://chat.veritone.com/) for help. - -### In this section - - - -* [The Veritone API Data Model](/apis/data-model) - -* [Using GraphQL](/apis/using-graphql) - -* [Authentication](/apis/authentication) - -* [Error Codes](/apis/error-codes) - -* [Tutorials](/apis/tutorials/) diff --git a/docs/apis/_media/Get-API-Token-1.png b/docs/apis/_media/Get-API-Token-1.png deleted file mode 100644 index 7a75b1e6cd..0000000000 Binary files a/docs/apis/_media/Get-API-Token-1.png and /dev/null differ diff --git a/docs/apis/_media/Get-API-Token-2.png b/docs/apis/_media/Get-API-Token-2.png deleted file mode 100644 index 2482ee15ed..0000000000 Binary files a/docs/apis/_media/Get-API-Token-2.png and /dev/null differ diff --git a/docs/apis/_media/veritone-data-model.jpg b/docs/apis/_media/veritone-data-model.jpg deleted file mode 100644 index f6e2dd2194..0000000000 Binary files a/docs/apis/_media/veritone-data-model.jpg and /dev/null differ diff --git a/docs/apis/_media/veritone-graphql-erd.svg b/docs/apis/_media/veritone-graphql-erd.svg deleted file mode 100644 index 619dbf0093..0000000000 --- a/docs/apis/_media/veritone-graphql-erd.svg +++ /dev/null @@ -1 +0,0 @@ -VTN-21370_COR_VisualAid_HowAPIWorks_Q219_vFinal_outlinedfonts \ No newline at end of file diff --git a/docs/apis/authentication.md b/docs/apis/authentication.md deleted file mode 100644 index 687cccfc45..0000000000 --- a/docs/apis/authentication.md +++ /dev/null @@ -1,40 +0,0 @@ -# Authentication - -Veritone uses token-based authentication for accessing the system and resources. Currently, the following authentication types are provided to meet the security needs of specific actions being performed: - -* **Application:** Third-party applications use OAuth2 protocol for authentication and authorization. OAuth2 provides SSO and generates tokens for users to securely access data without revealing username and password credentials. See [OAuth2 Authentication and Authorization](/developer/applications/oauth) for more information. - -* **Engine:** Veritone uses JSON Web Tokens (JWT) for engine authentication. JWTs provide access to resources specifically related to processing tasks (e.g., a TDO’s media assets) and are passed to engines in the Task Payload at engine runtime. - -* **API Token:** API Tokens provide access to organization-level resources and are generally used to make ad-hoc API requests by passing the token in an *Authorization* header. - -## Creating an API Token {docsify-ignore} - -Veritone’s GraphiQL interface is recommended for exploring, writing and testing the API, but calls can also be made using any HTTP client. -When you’re logged into the Veritone platform, GraphiQL logic automatically passes a valid token in the `Authorization` header of every request. -When making requests using a different client, include a valid API Token in the `Authorization` header with the value `Bearer `. -Requests made without this header or with an invalid token will return an error code. - -An API Token can be generated in the Veritone Admin App by your Organization Administrator. - -To generate an API Token: - -1. Log into the Veritone Platform and select **Admin** from the **App Picker** drop-down. -Veritone Admin opens. - -2. Click the **API Keys** tile. -The API Keys page opens. -![Get API Token](_media/Get-API-Token-1.png) - -3. Click **New API** Key. -The New API Key window opens. -![Get API Token](_media/Get-API-Token-2.png) - -4. Enter a token name and select the permissions needed for the token to perform the required API tasks. -Click **Generate Token** to save. -The Token Generated window opens. - -5. Copy your token and click **Close** when finished. - -> For security, once the Token Generated window closes, the token code will no longer display. -You would have to create a new API key to be able to see a valid API key. diff --git a/docs/apis/data-model.md b/docs/apis/data-model.md deleted file mode 100644 index 5afd01322f..0000000000 --- a/docs/apis/data-model.md +++ /dev/null @@ -1,100 +0,0 @@ -# The Veritone Data Model - -The following diagram presents a high-level overview of the data model -implemented in the Veritone GraphQL API. - -![Entity relationship diagram](_media/veritone-data-model.jpg) - -Typically, each block in the diagram has a corresponding type in the GraphQL -schema. Each type has corresponding queries that retrieve it and mutations -to modify it. - -Taking the `Task` and `Job` blocks from the diagram, we have the following -schema in GraphQL: - -```graphql -type Job { - # Retrieve all tasks contained in this job - tasks: [Task] -} - -type Task { - # Retrieve the job that contains this task - job: Job -} - -type Query { - # Retrieve a single job by ID - job(id: ID!): Job - # Retrieve a list of jobs - jobs: [Job] - # Retrieve a single task by ID - task(id: ID!): Task -} -``` - -Note that it's been simplified, with only basic fields and no filtering or paging. - -The following section describes what each entity means. - -* _Asset_: a piece of data. An asset can be consumed by an engine for analysis, -or it can be produced by an engine to store results. -* _Temporal Data Object_: contains assets and associated time series and other metadata. -* _Engine_: a self-contained, encapsulated program that is integrated into the Veritone -platform and performs either AI analysis or infrastructure functions such as ingestion. Third-party developers in the Veritone ecosystem -can create, deploy, and market their own engines using the -Veritone Developer Application (VDA). -* _Task_: a task is a request to run a single engine. A task is created with a payload containing input to the engine. -* _Job_: a job describes all the _tasks_ necessary to perform a -single high-level operation, such as run a series of engines against -a piece of data or ingest data from a given source. -* _Organization_: a Veritone platform subscriber. An organization -may represent a company, a department within a company, or an -individual developer. -* _User_: a single user account within an organization, representing -a human user (not a service or application). -* _Role_: an abstract, business-focused way of describing a user's permissions -on the Veritone platform. For example, a "Library Editor" can manage libraries -on behalf of the organization. A role collects a set of fine-grained _permissions_. -* _Permission_: a single functional permission on the Veritone platform, -such as "create library" or "view asset". -* _Application_: a custom Veritone client application. Third-party developers in the Veritone ecosystem -can create, deploy, and market their own applications using the -Veritone Developer Application (VDA). -* _Engine Category_: a grouping of engines with similar features. -Expected engine payload and output are common to engines within -an engine category. Engine categories make it easier for users to -find and select the engines they want to use. -* _Engine Whitelist_ / _Engine Blacklist_: users within a -given organization are allowed to search, view, and use a set of -engines determined by the _engine whitelist_ and _engine blacklist_ -for the organization. These lists are managed by Veritone in -accordance with the organization's needs. -* _Build_: engine developers upload an engine _build_ to deploy -or update their engine; a build represents the engine code. -Each engine can have any number of builds representing different -versions of the code, only one of which is active at any given time. -* _Library_: A named collection of entities an organization is interested in identifying in media, e.g. American Politicians. A library's type defines what type of entities it can hold. -* _Library type_: Describes the type of entities the library contains. Used to tailor the UI experience for specific types of entities (for example, People, Ads, etc). -* _Library collaborator_: An external organization that -a library has been shared with. Users in the collaborator -organization can view the library and use it to train -their own models, but cannot modify it. -* _Library engine model_: Data generated during an trainable engine's training step. The model is (in some cases) provided to an engine when the engine is run. A model can optionally contain an asset representing the model data. -* _Entity_: An aggregation of assets (_entity identifiers_) for a defined concept, such as a person, a company or organization, an advertising campaign, a type of object, etc. -* _Entity identifier_: An asset associated with an entity. Examples are headshots for face recognition, voice clips for speaker recognition, ad creative for audio fingerprinting, DLM for transcription, and aliases for transcription. -* _Entity identifier type_: The type of asset associated with an entity. e.g. headshot, logo, voice clip, ad, DLM. The library type defines what identifier types it can support. -* _Mention_: A record that a given entity, tag, or search condition was -matched (or "mentioned") in an engine result. Mentions can be shared and published. -* _Mention comment_: A user comment on an individual mention. Comments are used -to share, publicize, and collaborate on mentions. -* _Mention rating_: A user rating on an individual mention. Ratings are used -to share, publicize, and collaborate on mentions. -* _Watchlist_: An enhanced stored search including a set of search conditions -and filters, effective start and stop date, and other information. Hits against -the search captured in a watchlist generate mentions. -* _Folder_: A folder is a container for organizing and sharing information, -including temporal data objects (media and other data), watchlists, other folders. -* _Root folder_: an organization has an implicit top-level folder for each -folder type (collection, watchlist, etc.). -* _Collection_: a group of related data. A collection can be placed in a folder, or not. Collections can be shared and publicized. diff --git a/docs/apis/error-codes.md b/docs/apis/error-codes.md deleted file mode 100644 index 7b624307f8..0000000000 --- a/docs/apis/error-codes.md +++ /dev/null @@ -1,169 +0,0 @@ -# Error Codes - -Error handling in a GraphQL API is slightly different than error -handling in a typical REST API: - -- there is only one URL endpoint -- a single request can specify any number of queries, mutations, and fields therein - -The GraphQL specification does not explicitly state how errors should be -treated, but there are documented conventions and best practices. The -Veritone API follows these conventions and also leverages a commonly -used library (apollo-error) for error handling consistent with what you -may find in other GraphQL APIs. - -An HTTP status code of 200 indicates that the GraphQL server was able to -parse the incoming query and attempt to resolve all fields. - -Therefore, a non-200 status code indicates that either the query did not -reach the GraphQL server, or that the server was unable to parse it. The -follow table lists several HTTP status code you may encounter and their -meanings. - -| Code | Meaning | -| ---- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| 200 | GraphQL server received the query, parsed it, and attempted to resolve | -| 404 | Not found. In GraphQL, there is only one URL endpoint. Therefore, this error means that the caller's URL path was wrong. | -| 400 | Malformed request. The request should have the 'Content-Type' header set to either 'application/json' or 'multipart/form-data'. The request must contain a 'query' parameter containing JSON, and that JSON should have a 'query' element containing, as a string, a valid GraphQL query. | -| 500 | An internal server error prevented the server from handling the request. This error will not happen under normal circumstances. | -| 502 | HTTP gateway error. This could indicate an internal server error or timeout. Neither will occur under normal circumstances. | - -A HTTP 200 status code will be accompanied by a normal GraphQL response -body in JSON format. Fields that were successfully resolved will have -their data. Fields that *cannot* be successfully resolved will have a -null value and a corresponding error set in the *errors* field. - -Here's an example where we attempt to create three objects and only one -succeeds: - -```graphql -mutation { - create1: createAsset(input: { - containerId: 123 - }) { - id - } - - create2: createAsset(input: { - containerId: "400001249" - type:"media" - contentType: "video/mp4" - }) { - id - } - - create3: createAsset(input: { - containerId: "400001249" - type:"media" - contentType: "video/mp4" - uri: "http://localhost/myvideo.mp4" - }) { - id - } -} -``` - -The response: - -```json -{ - "data": { - "create1": null, - "create2": null, - "create3": { - "id": "e6a8e6b1-955a-4d0c-be3b-d1ff83833a15" - } - }, - "errors": [ - { - "message": "The requested object was not found", - "name": "not_found", - "time_thrown": "2017-12-12T01:15:48.875Z", - "data": { - "objectId": "123", - "objectType": "TemporalDataObject" - } - }, - { - "message": "One of uri or file (upload) must be provided to create an asset.", - "name": "invalid_input", - "time_thrown": "2017-12-12T01:15:48.964Z" - } - ] -} -``` - -Here's another example where we attempt to retrieve three objects, but only one is found: - -```graphql -{ - asset1: asset(id: "2426dbe5-eef3-4167-9da8-fb1eeec61c67") { - id - } - asset2: asset(id: "2426dbe5-eef3-4167-9da8-fb1eeec61c68") { - id - } - asset3: asset(id: "1fa65e5a-8008-48e4-9968-272fbef54cc2") { - id - } -} -``` - -The response: - -```json -{ - "data": { - "asset1": null, - "asset2": null, - "asset3": { - "id": "1fa65e5a-8008-48e4-9968-272fbef54cc2" - } - }, - "errors": [ - { - "message": "The requested object was not found", - "name": "not_found", - "time_thrown": "2017-12-12T01:21:30.243Z", - "data": { - "objectId": "2426dbe5-eef3-4167-9da8-fb1eeec61c67", - "objectType": "Asset" - } - }, - { - "message": "The requested object was not found", - "name": "not_found", - "time_thrown": "2017-12-12T01:21:30.247Z", - "data": { - "objectId": "2426dbe5-eef3-4167-9da8-fb1eeec61c68", - "objectType": "Asset" - } - } - ] -} -``` - -So, to check for errors you should first verify HTTP status 200, and then check for an errors array in the response body. Or, if a field you expected to find a value in has null, look in the `errors` array for an explanation. - -Detailed error information is shown in a consistent format. Each entry in the `errors` array will be an object with the following fields: - -| Field | Description | -| ----- | ----------- | -| message | A human-readable description of the error cause | -| name | A machine-readable error code in string form | -| time_thrown | Timestamp | -| data | Operation-specific supplementation information about the error. For example on a not_found the data field will often have objectId with the ID that could not be found. | - -A standard set of error codes is used across the API. This list may grow over time. - -| Error code | Meaning | Recommended action | -| ---------- | ------- | ------------------ | -| not_found | A requested object was not found. This could mean that the object never existed, existed and has been deleted, or exists but the caller does not have access to it. | Verify that object exists and that the caller has rights to it. | -| not_allowed | The caller is not authorized to perform the requested operation. For example, a user with the Viewer role on the CMS app would get a not_allowed error if attempting to use the createAsset example shown above. | Either provision the caller with the required rights or do not attempt to access the object. | -| invalid_input | Although the query contains syntactically valid GraphQL according to the schema, something was wrong with the parameters on the field. For example, an integer parameter have have been outside the allowed range. | Indicates a client error. Refer to the schema documentation to fix the client code. | -| capacity_exceeded | Server capacity allocated to the client was exceeded while processing the request. | Try again later or, if the query was a complex one that may be expensive to resolve, break it apart or use a smaller page size. | -| authentication_error | The client did not supply a valid authentication token or the token was not of the type required for the requested operation. Not all fields require authentication, but an attempt to access one that does without supplying a valid token will cause this error. | Get and submit a current authentication token. If the token is current, the field may require a specific type of token such as api or user. Consult the schema documentation and correct the client code. | -| not_implemented | The requested query, mutation, or field is not available on this server. The cause may be a configuration issue or operational problem affecting a required subsystem. | Verify that the operation is actually supported by consulting schema docs. If so, try again later or contact Veritone support. | -| service_unavailable | A required service could not be reached. This error can indicate a temporary outage or a misconfiguration. | Try again later or contact Veritone support. | -| service_failure | A required service was accessible, but failed to respond or fulfill a request successfully. This error can indicate a temporary outage or a misconfiguration. | Try again later or contact Veritone support. | -| internal_error | An internal server error prevented the field from being resolved. | Try again later or contact Veritone support. | diff --git a/docs/apis/examples.md b/docs/apis/examples.md deleted file mode 100644 index 9f9c919c62..0000000000 --- a/docs/apis/examples.md +++ /dev/null @@ -1,2104 +0,0 @@ -# API Examples (GraphQL) - -## Authentication - -### Log In and Get Session Token - -```graphql -mutation userLogin { - userLogin(input: {userName: "jdoe@mycompany.com" password: "Password123"}) { - token - } -} -``` - -RESPONSE: - -```json -{ - "data": { - "userLogin": { - "token": "98b80884-a9f5-49ef-a7c6-bab43751218e" - } - } -} -``` - -ERRORS: (User not found) - -```json -{ - "data": { - "userLogin": null - }, - "errors": [ - { - "message": "The requested object or entity could not be found", - "name": "not_found", - "time_thrown": "2019-03-04T17:13:49.691Z", - "data": { - "serviceMessage": "404 - undefined", - "requestId": "0A0B018B:47AA_0A0B664F:0050_5C7D5CCC_98997F2:6132", - "errorId": "d089ba45-e0a5-408f-b5b3-fefa33951f73" - }, - "path": [ - "userLogin" - ], - "locations": [ - { - "line": 2, - "column": 3 - } - ] - } - ] -} -``` - -### Validate Token - -```graphql -mutation validateToken { - validateToken(token: "78cac3d0-684d-486e-8d86-8890dbae72e2") { - token - } -} -``` - -### Log Out User Session - -```graphql -mutation userLogout { - userLogout(token: "3c15e237-94a5-4563-9be7-4882acc7fa74") -} -``` - -## Builds and Deployment - -### Get an Engine's Name, State, Build, and Deployment Model - -```graphql -query { - engine(id:"3929002a-6902-4c59-b81c-7a79fcd8b9c0"){ - id - name - state - deploymentModel - builds(buildStatus:deployed) { - records{ - id - } - } - } -} -``` - -RETURNS: - -```json -{ - "data": { - "engine": { - "id": "3929002a-6902-4c59-b81c-7a79fcd8b9c0", - "name": "Amazon Transcribe - V2F (EU West)", - "state": "active", - "deploymentModel": "NonNetworkIsolated", - "builds": { - "records" : [ ] - } - } - } -} -``` - -### Pause the Build - -Before changing an engine's `deploymentModel`, it should be paused. - -> Use caution not to pause an engine that might currently be in use. - -```graphql -mutation pauseBuild{ - updateEngineBuild(input:{ - id: "6ff2cd01-a648-4ce1-afe4-9392b8821057" - engineId: "0fe8e64c-6c70-4727-9d89-de06f5231a6f" - action: pause - }){ - id - engineId - status - } -} -``` - -Possible values for `action` include the following: - -```graphql -enum BuildUpdateAction { - deploy - pause - unpause - approve - disapprove - invalidate - submit - upload - delete -} -``` - -### Update an Engine's Deployment Model - -> The engine should be paused (as demonstrated above) prior to running this command. - -```graphql -mutation updateModel{ - updateEngine(input:{ - id:"0fe8e64c-6c70-4727-9d89-de06f5231a6f" - deploymentModel: FullyNetworkIsolated - }){ - id - name - deploymentModel - } -} -``` - -### Unpause the Build - -```graphql -mutation unpauseBuild{ - updateEngineBuild(input:{ - id:"6ff2cd01-a648-4ce1-afe4-9392b8821057" - engineId:"0fe8e64c-6c70-4727-9d89-de06f5231a6f" - action:unpause - }){ - id - engineId - status - } -} -``` - -### Deploy a Build - -> If you have changed an engine's `deploymentModel` using the foregoing commands, you should `deploy` it again. - -```graphql -mutation deployBuild{ - updateEngineBuild(input:{ - id:"6ff2cd01-a648-4ce1-afe4-9392b8821057" - engineId:"0fe8e64c-6c70-4727-9d89-de06f5231a6f" - action:deploy - }){ - id - engineId - status - } -} -``` - -## Collections - -### Create a Collection - -```graphql -mutation newCollection { - createCollection(input:{ - name:"Test Collection", - image:"https://edit.co.uk/uploads/2016/12/Image-1-Alternatives-to-stock-photography-Thinkstock.jpg" - folderDescription:"Interesting Folder" - }) { - id - } -} -``` - -RESPONSE: - -```json -{ - "data": { - "createCollection": { - "id": "10570" - } - } -} -``` - -### Get Collections - -```graphql -query getCollections { - collections { - records { - id - } - } -``` - -RESPONSE: - -```json -{ - "data": { - "collections": { - "records": [ - { - "id": "10566" - }, - { - "id": "10570" - }, - { - "id": "10569" - } - ] - } - } -} -``` - -### Update a Collection - -```graphql -mutation { - updateCollection(input: {folderId: "10566", name: "First Collection", folderDescription: "Folder description of first collection"}) { - id - name - } -} -``` - -RESPONSE: - -```json -{ - "data": { - "updateCollection": { - "id": "10566", - "name": "First Collection" - } - } -} -``` - -### Delete a Collection - -```graphql -mutation removeCollection { - deleteCollection(id: "10566") { - id - } -} -``` - -## Ingestion - -### Create TDO and Upload Asset - -This is probably the easiest way to upload a file. -Given a public file URL it will create a container TDO and upload the file as the primary media asset for that TDO. - -> "uri" must be a public URL - -```graphql -mutation createTDOWithAsset { - createTDOWithAsset( - input: { - startDateTime: 1533761172, - stopDateTime: 1533761227, - contentType: "video/mp4", - assetType: "media", - addToIndex: true, - uri: "https://s3.amazonaws.com/hold4fisher/s3Test.mp4" - } - ) - { - id - status - assets { - records { - id - assetType - contentType - signedUri - } - } - } -} -``` - -### Create Empty TDO (No Asset) - -This can be useful for creating a container TDO that you can upload assets into later with the `createAsset` mutation. - -```graphql -mutation createTDO { - createTDO( - input: { - startDateTime: 1548432520, - stopDateTime: 1548436341 - } - ) - { - id - status - } -} -``` - -## Jobs and Tasks - -### Run Engine Job on Existing TDO - -> The last three "engineId" values are needed for the TDO to appear correctly in CMS. - -```graphql -mutation runEngineJob { - createJob( - input: { - targetId: "102014611", - tasks: [ - { - engineId: "8081cc99-c6c2-49b0-ab59-c5901a503508" - }, - { - engineId: "insert-into-index" - }, - { - engineId: "thumbnail-generator" - }, - { - engineId: "mention-generate" - } - ] - } - ) - { - id - } -} -``` - -### Run Engine Job on External File (using Web Stream Adapter) and Add Results to Existing TDO - -> "url" must be public; change only the second "engineId" value. The existing TDO must be empty (no assets). - -```graphql -mutation runRTEngineJob { - createJob(input: { - targetId: "88900861", - tasks: [{ - engineId: "9e611ad7-2d3b-48f6-a51b-0a1ba40feab4", # Webstream Adapter's Engine ID - payload: { - url: "https://s3.amazonaws.com/hold4fisher/s3Test.mp4" - } - }, - { - engineId: "38afb67a-045b-43db-96db-9080073695ab" # Some engine ID you want to use for processing - }] - }) { - id - } -} -``` - -### Run Engine Job with Standby Task - -```graphql -mutation createTranscriptionJobWithStandby { - createJob(input: { - targetId: "53796349", - tasks: [{ - engineId: "transcribe-speechmatics-container-en-us", - standbyTask: { - engineId: "transcribe-speechmatics-container-en-us", - standbyTask: { - engineId: "transcribe-voicebase", - payload: { language: "en-US", priority: "low" } - } - } - }] - }) { - id - tasks { - records { - id - } - } - } -} -``` - -### Run Library-Enabled Engine Job (e.g. Face Recognition) - -> "libraryEngineModelId" can be obtained by running the [Get Library Training Stats](#get-library-training-stats) query in the "Library" section - -```graphql -mutation runLibraryEngineJob { - createJob(input: { - targetId: "119586271" - tasks: [ - { - engineId: "0744df88-6274-490e-b02f-107cae03d991" - payload: { - libraryId: "ef8c7263-6c7c-4b8f-9cb5-93784e3d89f5" - libraryEngineModelId: "5750ca7e-8c4a-4ca4-b9f9-df8617032bd4" - }, - } - ] - }) { - id - targetId - tasks { - records { - id - engineId - order - payload - status - } - } - } -} -``` - -### Create (Kick Off) a Job - -This job will reprocess face detection in Redact. - -```graphql -mutation reprocessFaceDetection { - createJob(input : { - targetId: "331580431" - tasks:[{ - engineId: "b9eca145-3bd6-4e62-83e3-82dbc5858af1", - payload: { - recordingId: "331580431" - confidenceThreshold: 0.7 - videoType: "Bodycam" - } - }] - }){ - id - } -} -``` - -### Get Jobs for a Given TDO - -```graphql -query getJobs { - jobs(targetId: "102014611") { - records { - id - createdDateTime - status - tasks { - records { - id - status - engine { - id - name - category { - name - } - } - } - } - } - } -} -``` - -### Get List of Running Jobs - -In this example, we ask for a `limit` of 50 running jobs. -The `status` can be `pending`, `cancelled`, `queued`, `running`, or `complete`. - -```graphql -query runningJobs { - jobs(status: running, limit: 50) { - count - records { - id - targetId - createdDateTime - tasks { - records { - id - payload - } - } - } - } -} -``` - -### Check Job Status of a Specific Job - -```graphql -query jobStatus { - job(id: "18114402_busvuCo21J") { - status - createdDateTime - targetId - tasks { - records { - status - createdDateTime - modifiedDateTime - id - engine { - id - name - category { - name - } - } - } - } - } -} -``` - -RESPONSE: - -```json -{ - "data": { - "job": { - "status": "complete", - "createdDateTime": "2019-02-26T21:15:59.000Z", - "targetId": "380612136", - "tasks": { - "records": [ - { - "status": "complete", - "createdDateTime": "2019-02-26T21:15:59.000Z", - "modifiedDateTime": "2019-02-26T21:15:59.000Z", - "id": "19020926_wUALxLYqjoDh5N8", - "engine": { - "id": "915bb300-bfa8-4ce6-8498-50d43705a144", - "name": "mention-generate", - "category": { - "name": "Search" - } - } - }, - { - "status": "complete", - "createdDateTime": "2019-02-26T21:15:59.000Z", - "modifiedDateTime": "2019-02-26T21:15:59.000Z", - "id": "19020926_wUALxLYqjolDUtw", - "engine": { - "id": "c2aaa6d7-14fa-f840-f77e-4d2c0b857fa8", - "name": "Add to Index", - "category": { - "name": "Search" - } - } - } - ] - } - } - } -} -``` - -### Check the Status of a Specific Task - -If you know the Task ID, you can do: - -```graphql -query { - task(id:"19020926_wUALxLYqjoDh5N8") { - status - } -} -``` - -RESPONSE: - -```json -{ - "data": { - "task": { - "status": "complete" - } - } -} -``` - -### Get Transcription Jobs - -You can (optionally) use the `dateTimeFilter` field to filter responses by date (as shown below). -This example fetches a max `limit` of 3 transcription jobs, with a `status` of `complete`, created after 9/25/2018. - -```graphql -query getTranscriptionJobs { - jobs(dateTimeFilter: {fromDateTime: "2018-09-25T03:10:40.000Z", field: createdDateTime}, limit: 3, status: complete, engineCategoryIds: ["67cd4dd0-2f75-445d-a6f0-2f297d6cd182"]) { - count - records { - id - targetId - createdDateTime - tasks { - records { - id - payload - } - } - } - } -} -``` - -### Get Information on Most Recent Jobs - -The following query will retrieve information on the most recent 10 jobs. - -```graphql -query { - jobs(orderBy: {field: createdDateTime, direction: desc}, limit: 10) { - records { - id - status - createdDateTime - target { - id - streams { - uri - protocol - } - assets { - records { - id - assetType - uri - details - } - } - } - tasks { - records { - id - status - engine { - id - name - } - queuedDateTime - completedDateTime - payload - } - } - } - } -} -``` - -RESPONSE: - -```json -{ - "data": { - "jobs": { - "records": [ - { - "id": "19020926_0N2iBMnL1D", - "status": "complete", - "createdDateTime": "2019-02-26T21:15:59.000Z", - "target": { - "id": "380612133", - "streams": [], - "assets": { - "records": [ - { - "id": "380612133_33TR5wyYyh", - "assetType": "media", - "uri": "https://inspirent.s3.amazonaws.com/assets/41020140/aca39ac2-181e-4837-a059-378acc6b24bd.mp4", - "details": null - } - ] - } - }, - "tasks": { - "records": [ - { - "id": "19020926_0N2iBMnL1DuhMEl", - "status": "complete", - "engine": { - "id": "915bb300-bfa8-4ce6-8498-50d43705a144", - "name": "mention-generate" - }, - "queuedDateTime": "2019-02-26T21:15:59.000Z", - "completedDateTime": "2019-02-26T21:15:59.000Z", - "payload": { - "organizationId": 17532 - } - }, - { - "id": "19020926_0N2iBMnL1D8bJUX", - "status": "complete", - "engine": { - "id": "c2aaa6d7-14fa-f840-f77e-4d2c0b857fa8", - "name": "Add to Index" - }, - "queuedDateTime": "2019-02-26T21:15:59.000Z", - "completedDateTime": "2019-02-26T21:15:59.000Z", - "payload": { - "organizationId": 17532 - } - } - ] - } - }, - { - "id": "19020926_gYlCLxSgXT", - "status": "complete", - "createdDateTime": "2019-02-26T21:15:59.000Z", - "target": { - "id": "380612134", - "streams": [], - "assets": { - "records": [ - { - "id": "380612134_7dMz22o8KH", - "assetType": "media", - "uri": "https://inspirent.s3.amazonaws.com/assets/41020181/9474d32f-557d-4d72-894e-5cfa6d31d93f.mp4", - "details": null - } - ] - } - }, - "tasks": { - "records": [ - { - "id": "19020926_gYlCLxSgXT7zWh3", - "status": "complete", - "engine": { - "id": "915bb300-bfa8-4ce6-8498-50d43705a144", - "name": "mention-generate" - }, - "queuedDateTime": "2019-02-26T21:15:59.000Z", - "completedDateTime": "2019-02-26T21:15:59.000Z", - "payload": { - "organizationId": 17532 - } - }, - { - "id": "19020926_gYlCLxSgXTiRPjY", - "status": "complete", - "engine": { - "id": "c2aaa6d7-14fa-f840-f77e-4d2c0b857fa8", - "name": "Add to Index" - }, - "queuedDateTime": "2019-02-26T21:15:59.000Z", - "completedDateTime": "2019-02-26T21:15:59.000Z", - "payload": { - "organizationId": 17532 - } - } - ] - } - }, - { - "id": "19020926_sArl2PbD38", - "status": "complete", - "createdDateTime": "2019-02-26T21:15:59.000Z", - "target": { - "id": "380612135", - "streams": [], - "assets": { - "records": [ - { - "id": "380612135_s3N6dLAS5n", - "assetType": "media", - "uri": "https://inspirent.s3.amazonaws.com/assets/41020653/ad14cbf3-9ee1-4a05-84cf-e71326c35537.mp4", - "details": null - } - ] - } - }, - "tasks": { - "records": [ - { - "id": "19020926_sArl2PbD38W6bwr", - "status": "complete", - "engine": { - "id": "915bb300-bfa8-4ce6-8498-50d43705a144", - "name": "mention-generate" - }, - "queuedDateTime": "2019-02-26T21:15:59.000Z", - "completedDateTime": "2019-02-26T21:15:59.000Z", - "payload": { - "organizationId": 17532 - } - }, - { - "id": "19020926_sArl2PbD38fZ6MS", - "status": "complete", - "engine": { - "id": "c2aaa6d7-14fa-f840-f77e-4d2c0b857fa8", - "name": "Add to Index" - }, - "queuedDateTime": "2019-02-26T21:15:59.000Z", - "completedDateTime": "2019-02-26T21:15:59.000Z", - "payload": { - "organizationId": 17532 - } - } - ] - } - }, - { - "id": "19020926_wUALxLYqjo", - "status": "complete", - "createdDateTime": "2019-02-26T21:15:59.000Z", - "target": { - "id": "380612136", - "streams": [], - "assets": { - "records": [ - { - "id": "380612136_qEqiALGKtI", - "assetType": "media", - "uri": "https://inspirent.s3.amazonaws.com/assets/41020144/dbf21c78-909a-4d6f-8cd1-1b55d721ab07.mp3", - "details": null - } - ] - } - }, - "tasks": { - "records": [ - { - "id": "19020926_wUALxLYqjoDh5N8", - "status": "complete", - "engine": { - "id": "915bb300-bfa8-4ce6-8498-50d43705a144", - "name": "mention-generate" - }, - "queuedDateTime": "2019-02-26T21:15:59.000Z", - "completedDateTime": "2019-02-26T21:15:59.000Z", - "payload": { - "organizationId": 17532 - } - }, - { - "id": "19020926_wUALxLYqjolDUtw", - "status": "complete", - "engine": { - "id": "c2aaa6d7-14fa-f840-f77e-4d2c0b857fa8", - "name": "Add to Index" - }, - "queuedDateTime": "2019-02-26T21:15:59.000Z", - "completedDateTime": "2019-02-26T21:15:59.000Z", - "payload": { - "organizationId": 17532 - } - } - ] - } - } - ] - } - } -} -``` - -### Get Information about a Task - -```graphql -query { - task(id:"19020926_wUALxLYqjoDh5N8") { - id - status - name - engine { - id - name - displayName - } - engineId - } -} -``` - -RESPONSE: - -```json -{ - "data": { - "task": { - "id": "19020926_wUALxLYqjoDh5N8", - "status": "complete", - "name": null, - "engine": { - "id": "915bb300-bfa8-4ce6-8498-50d43705a144", - "name": "mention-generate", - "displayName": "mention-generate" - }, - "engineId": "915bb300-bfa8-4ce6-8498-50d43705a144" - } - } -} -``` - -### Get Logs for Tasks - -```graphql -query getLogs { - temporalDataObject(id: "331178425") { - tasks { - records { - engine { - id - name - } - id - status - startedDateTime - completedDateTime - log { - text - uri - } - } - } - } -} -``` - -### Cancel Job in Progress - -```graphql -mutation cancelJob { - cancelJob(id: "18114402_busvuCo21J") { - id - message - } - } - ``` - -### Delete TDO - -```graphql -mutation deleteTDO { - deleteTDO(id: "64953347") { - id - message - } -} -``` - -## Retrieval - -### Get Assets for TDO - -```graphql -query getAssets { - temporalDataObject(id: "280670774") { - assets { - records { - sourceData { - engine { - id - name - } - } - id - createdDateTime - assetType - signedUri - } - } - } -} -``` - -### Get Engine Results in Veritone Standard Format - -```graphql -query getEngineOutput { - engineResults(tdoId: "102014611", engineIds: ["transcribe-speechmatics-container-en-us"]) { - records { - tdoId - engineId - startOffsetMs - stopOffsetMs - jsondata - assetId - userEdited - } - } -} -``` - -### Get Transcription and Speaker Detection Results in Veritone Standard Format - -```graphql -query vtn { - engineResults(tdoId: "107947027", engineIds: ["40356dac-ace5-46c7-aa02-35ef089ca9a4", "transcribe-speechmatics-container-en-us"]) { - records { - jsondata - } - } -} -``` - -### Get Task Output by Job ID - -Task output displays the debug/log information reported by the engine at the completion of the task - -```graphql -query getEngineOutputByJob { - job(id: "18083316_nNXUOSnxJH") { - tasks { - records { - id - output - status - engine { - id - name - category { - name - } - } - } - } - } -} -``` - -### Get Task Output (Log Information) by TDO ID - -Task output displays the debug/log information reported by the engine at the completion of the task - -```graphql -query getEngineOutputByTDO { - temporalDataObject(id: "102014611") { - tasks { - records { - id - engine { - id - name - category { - name - } - } - status - output - } - } - } -} -``` - -### Export Transcription Results - -This example shows how to limit line length to 32 characters for 3 of the 4 transcript formats, while setting it to 50 for `txt`. - -```graphql -mutation createExportRequest { - createExportRequest(input: { - includeMedia: true - tdoData: [{tdoId: "96972470"}, {tdoId: "77041379"}] - outputConfigurations:[ - { - engineId:"transcribe-speechmatics" - categoryId:"67cd4dd0-2f75-445d-a6f0-2f297d6cd182" - formats:[ - { - extension:"srt" - options: { - maxCharacterPerLine:32 - linesPerScreen: 3 - newLineOnPunctuation: true - } - }, - { - extension:"vtt" - options: { - maxCharacterPerLine:32 - linesPerScreen: 2 - newLineOnPunctuation: false - } - }, - { - extension:"ttml" - options: { - maxCharacterPerLine:32 - newLineOnPunctuation: true - } - }, - { - extension:"txt" - options: { - maxCharacterPerLine:50 - } - } - ] - } - ] - }) { - id - status - organizationId - createdDateTime - modifiedDateTime - requestorId - assetUri - } -} -``` - -### Check the status of an export request - -```graphql -query checkExportStatus { - exportRequest(id:"10f6f809-ee36-4c12-97f4-d1d0cf04ea85") { - status - assetUri - requestorId - } -} -``` - -### Get Transcription Output in JSON Format (Legacy Method) - -!> This output format is deprecated in favor of using the [engineResults query](#Get-Engine-Results-in-Veritone-Standard-Format) for `vtn-standard` output -and the [`createExportRequest` mutation](#export-transcription-results) for transforming to other formats. - -```graphql -query getTranscriptJSON { - temporalDataObject(id: "102014611") { - primaryAsset(assetType: "transcript") { - signedUri - transform(transformFunction: Transcript2JSON) - } - details - } -} -``` - -## Search - -See the [search quickstart guide](apis/search-quickstart/) for details on the searchMedia query syntax. - -### Search Structured Data - -This example shows how to search for structured data using the search API. -In this case, we're looking for media where the `QUANTITY` value equals `2` or the `ORG_ID` value equals `"12343"`. - -```graphql -query searchSDO { - searchMedia(search: { - index: ["mine"], - limit: 20, - offset: 0, - query: { - conditions: [ - { operator: "term", field: "QUANTITY", value: 2 } - { operator: "term", field: "ORD_ID", value: "12343" } - ], - operator: "or" - }, - type: "" - }) - { - jsondata - } -} -``` - -## Folders - -### Create a Folder - -Before you can create a folder, you must know the ID of the intended parent folder. -If you want to place the new folder under the `rootFolder`, you can discover the `rootFolder` ID by (for example) running a query using `organizations(id:)`. - -```graphql -mutation { - createFolder(input:{ - name:"scratch" - description:"temporary files", - parentId:"c13df18e-314d-47fb-b4cd-aa171bc1bce6", - rootFolderType:watchlist - }) { - id - name - } -} -``` - -RESPONSE: - -```json -{ - "data": { - "createFolder": { - "id": "4f6e98d5-168a-407d-b22f-d5607bd4a9a6", - "name": "scratch" - } - } -} -``` - -ERRORS: - -Note that all four input fields (`name`, `description`, `parentId`, and `rootFolderType`) are mandatory. -The `rootFolderType` should correspond to the folder type of the `parentId` folder and should be one of `cms`, `collection`, or `watchlist`. - -### List Folders in CMS and Contained TDOs - -```graphql -query listFolders { - rootFolders(type: cms) { - id - name - subfolders { - id - name - childTDOs { - count - records { - id - } - } - } - childTDOs { - count - records { - id - } - } - } -} -``` - -### Get Folder Info by TDO ID - -```graphql -query getFolderInfoByTDO { - temporalDataObject(id: "112971783") { - folders { - id - name - childTDOs { - records { - id - } - } - } - } -} -``` - -### Get Folder Info by Folder ID - -```graphql -query getFolderInfo { - folder(id: "0ef7e0e3-63f5-47b9-8ed1-4c172c5a1e0a") { - id - name - childTDOs { - records { - id - } - } - } -} -``` - -## Library - -### Get Entity Info - -```graphql -query getEntityInfo { - entity(id: "02de18d5-4f70-450c-9716-de949668ec40") { - name - jsondata - } -} -``` - -### Get Unpublished Entities in Library - -```graphql -query unpublishedEntities { - entities(libraryIds: "ffd171b9-d493-41fa-86f1-4e02dd769e73", isPublished: false, limit: 1000) { - count - records { - name - id - } - } -} -``` - -### Publish/Train Library - -```graphql -mutation publishLibrary { - publishLibrary(id: "169f5db0-1464-48b2-b5e1-59fe5c9db7d9") { - name - } -} -``` - -### Get Library Training Stats - -Get trainJobId value for desired engine model (e.g. "Machine Box Facebox Similarity") - -```graphql -query getTrainJobID { - library(id: "1776029a-8447-406a-91bc-2402bf33443b") { - engineModels { - records { - id - createdDateTime - modifiedDateTime - engine { - id - name - } - libraryVersion - trainStatus - trainJobId - } - } - } -} -``` - -### Use trainJobId value to obtain train job details - -```graphql -query getTrainJobDetails { - job(id: "18104324_t5XpqlGOfm") { - tasks { - records { - id - status - startedDateTime - completedDateTime - } - } - } -} -``` - -### Check if Engine is Library-Trainable - -```graphql -query engineLibraryTrainable { - engine(id: "95d62ae8-edc2-4fb9-ad08-fe33646f0ece") { - libraryRequired - } -} -``` - -### Update Entity Thumbnail in library - -First, obtain the URL for the entity's first identifier: - -```graphql -query getEntityImageURL { - entity(id: "3fd506ec-aa31-43b3-b51e-5881a57965a1") { - identifiers(limit: 1) { - records { - url - } - } - } -} -``` - -Then pass the identifier's URL in as the value of `profileImageUrl` - -```graphql -mutation updateEntityThumbnail { - updateEntity(input: { - id: "3fd506ec-aa31-43b3-b51e-5881a57965a1", - profileImageUrl: "https://veritone-aiware-430032233708-us-gov-prod-sled2-recordings.s3-us-gov-west-1.amazonaws.com/3b82ec4a-0679-49cf-999b-7e76b899de56/3fd506ec-aa31-43b3-b51e-5881a57965a1/75a5f021-7c06-495a-9f03-ed5dc209dff3.jpeg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAWIH7LETWO45I3Q7X%2F20190117%2Fus-gov-west-1%2Fs3%2Faws4_request&X-Amz-Date=20190117T232649Z&X-Amz-Expires=86400&X-Amz-Security-Token=FQoDYXdzEDIaDO0%2FlJcJeJRIyJ86myLABEvH3vewwLDedka8Cn%2FcSPfDPA8rGer8JiyVtS7rB4mbYJ8kCkTzmWYdkm9DmBVyNxcMSqxUz9sW7U1yexuERkYcpD5ajBThw2JS8uwQeL4%2FXsO3vlk9ODBjFoT%2BWn%2B7p%2Bti54npv7BPSNHk7I%2BHLMfrSQQB6CmM8RZNAjGiyh%2FWk8Ls2ykyU92lIqAZKdKYbvhNfIGgSalH%2B%2BWnrEzVeBbDKN6ahLcqCbIsqfEIChjGk8XAwvfrTmnkXGjNZWheb6rHnl%2F24chxOyX4utZVuJeJIGZh6%2BOR%2F8TU4pSGiyvTuePmcdZkczLQKLu1g3TB5WbEPR2Ip1t%2F9Xr%2Fc54ph2qBXk4dEMlFHii0DCt4Ov7HXqsQtFS0fpNY3yrg3MxVLNfx5zpsUN0jwSB%2BAWGJ3jIOCe4djCx75dejnxAEse2lxxImPZnIhi1OspqHbb6TxTUfIkNHqLrFQ3Ari%2FtRnjRJID0ocwtziGgY7Husm1lWrJMyyhY9PGv1pHtfCmlXEcNKa0VLz52vsR9sOr6zIhM20GWJ8aSakEbauKBKDUeoA43ZQMZn4p05QAuIHIVOxcJ1VwPdrSLDFgi5MJawZYMgHbjIgSLpn6AD4aaIL7dkX6EiLrKir0OEXAiyU9P33btNVMPgWIWv%2FGGF%2FGakalGhTK8%2FoTlo%2BJIZdxmfk%2FH%2F%2BeAbTtHU0yVDwgh0PIGHUmL1rg2MkOXCEN4fT8dSY99s56Axrsr%2FhYIrimid2CI0AoFzCCyXGmng3VsScUeCoCiXuYPiBQ%3D%3D&X-Amz-Signature=d31cfb938032f55c73251ead4deadfaf0ab46d8b38728f4da998b517dd207859&X-Amz-SignedHeaders=host" - }) { - id - profileImageUrl - } -} -``` - -## Structured Data - -### Get Currently Published Schema ID - -The ID you see in Developer's URL is the `dataRegistry` ID, which contains multiple schemas. -To get the ID of the currently published one (for use in the `createStructuredData` mutation for example), you can use a query like this: - -```graphql -query { - dataRegistry(id: "") { - publishedSchema { - id - } - } -} -``` - -## Users - -### Get an Organization's Users - -Supply one or more `organizationId` values inside an array: - -```graphql -query { - users(organizationIds:[17532]) { - records { - id - name - } - } -} -``` - -### Create a New User - -The only *required* fields are `name` and `organizationId`. - -```graphql -mutation { - createUser(input: { - name: "Arby Arbiti", - requestorId: "960b3fa8-1812-4303-b58d-4f0d227f2afc", - password: "", - organizationId: 17532, - sendNewUserEmail: false, - email: "something@something.com", - firstName: "Arby", - lastName: "Arbiti" - }) - { - id - } -} -``` - -## Miscellaneous - -### Enumerate Cognition Engine Categories - -```graphql -query enumerateCognitionEngineCategories { - engineCategories(type:"Cognition") { - records { - categoryType - totalEngines - id - description - type { - name - } - } - } -} -``` - -RESPONSE: - -```json -{ - "data": { - "engineCategories": { - "records": [ - { - "categoryType": "transcode", - "totalEngines": 28, - "id": "581dbb32-ea5b-4458-bd15-8094942345e3", - "description": null, - "type": { - "name": "Cognition" - } - }, - { - "categoryType": "transcript", - "totalEngines": 787, - "id": "67cd4dd0-2f75-445d-a6f0-2f297d6cd182", - "description": "Convert the spoken word into readable text", - "type": { - "name": "Cognition" - } - }, - { - "categoryType": "sentiment", - "totalEngines": 46, - "id": "f2554098-f14b-4d81-9be1-41d0f992a22f", - "description": "Infer the sentiment or emotion being emitted in media", - "type": { - "name": "Cognition" - } - }, - { - "categoryType": "fingerprint", - "totalEngines": 14, - "id": "17d62b84-8b49-465b-a6be-fe3ea3bc8f05", - "description": "Find the same audio by using audio fingerprints", - "type": { - "name": "Cognition" - } - }, - { - "categoryType": "face", - "totalEngines": 282, - "id": "6faad6b7-0837-45f9-b161-2f6bf31b7a07", - "description": "Detect and Identify multiple faces within rich media content", - "type": { - "name": "Cognition" - } - }, - { - "categoryType": "object", - "totalEngines": 360, - "id": "088a31be-9bd6-4628-a6f0-e4004e362ea0", - "description": "Recognize Object and Logos within the Video Media", - "type": { - "name": "Cognition" - } - }, - { - "categoryType": "translate", - "totalEngines": 157, - "id": "3b2b2ff8-44aa-4db4-9b71-ff96c3bf5923", - "description": "Translate transcribed text from one language to another", - "type": { - "name": "Cognition" - } - }, - { - "categoryType": "audio", - "totalEngines": 10, - "id": "c6e07fe3-f15f-48a7-8914-951b852d54d0", - "description": "Detect characteristics of sound, including alarms, breaking glass, gunshots and more within audio", - "type": { - "name": "Cognition" - } - }, - { - "categoryType": "music", - "totalEngines": 0, - "id": "c96b5d0e-3ce1-4fd7-9c38-d25ddef87a5f", - "description": "Identify the music playing in the background of media", - "type": { - "name": "Cognition" - } - }, - { - "categoryType": "geolocation", - "totalEngines": 5, - "id": "203ad7c2-3dbd-45f9-95a6-855f911563d0", - "description": "Extract Location, acceleration, velocity and altitude from the media", - "type": { - "name": "Cognition" - } - }, - { - "categoryType": "conductor", - "totalEngines": 32, - "id": "892960cb-14c7-4743-a6e2-d6e437d6c5bb", - "description": null, - "type": { - "name": "Cognition" - } - }, - { - "categoryType": "stationPlayout", - "totalEngines": 1, - "id": "935c4838-dcf6-415c-99c4-5ceb0a8944be", - "description": "Find ads and songs for radio station mentions", - "type": { - "name": "Cognition" - } - }, - { - "categoryType": "ocr", - "totalEngines": 108, - "id": "3b4ac603-9bfa-49d3-96b3-25ca3b502325", - "description": "Recognize Text within the Video Media", - "type": { - "name": "Cognition" - } - }, - { - "categoryType": "logo", - "totalEngines": 50, - "id": "5a511c83-2cbd-4f2d-927e-cd03803a8a9c", - "description": "Recognize Logos within the Video Media", - "type": { - "name": "Cognition" - } - }, - { - "categoryType": "utility", - "totalEngines": 13, - "id": "f951fbf9-aa69-47a2-87c8-12dfb51a1f18", - "description": "Miscellaneous utility engines for cognition", - "type": { - "name": "Cognition" - } - }, - { - "categoryType": "speech", - "totalEngines": 4, - "id": "09f48865-c9e5-47b9-be79-8581047477c4", - "description": "Convert text to speech", - "type": { - "name": "Cognition" - } - }, - { - "categoryType": "correlation", - "totalEngines": 18, - "id": "a70df3f6-84a7-4570-b8f4-daa122127e37", - "description": "Correlates structured data to recordings", - "type": { - "name": "Cognition" - } - }, - { - "categoryType": "voice", - "totalEngines": 4, - "id": "19bfa716-309a-41dc-9dac-d07a1e7008cd", - "description": "Detect and Identify multiple voices within rich media content", - "type": { - "name": "Cognition" - } - }, - { - "categoryType": "speaker", - "totalEngines": 14, - "id": "a856c447-1030-4fb0-917f-08179f949c4e", - "description": "Detect the change in speakers within your transcription results", - "type": { - "name": "Cognition" - } - }, - { - "categoryType": "workflow", - "totalEngines": 8, - "id": "c5458876-43d2-41e8-a340-f734702df04a", - "description": "", - "type": { - "name": "Cognition" - } - }, - { - "categoryType": "reducer", - "totalEngines": 3, - "id": "70e54f46-7586-4ff1-876c-5f918357aec6", - "description": "Combine engine outputs from multiple segments", - "type": { - "name": "Cognition" - } - }, - { - "categoryType": "human", - "totalEngines": 11, - "id": "3b3cfe77-da94-4362-9ac2-3ed89747a68b", - "description": "Tasks to be performed by humans", - "type": { - "name": "Cognition" - } - } - ] - } - } -} -``` - -### Search for Engines by Capability - -Find all transcription engines: - -```graphql -query getAllTranscriptionEngines { - engines(categoryId: "67cd4dd0-2f75-445d-a6f0-2f297d6cd182") { - records { - id - name - category { - id - } - } - } -} -``` - -### Get TDO Details (Filename, Tags, etc.) - -```graphql -query getTDODetails { - temporalDataObject(id: "102014611") { - details - } -} -``` - -### Get Duration of Media File (In Seconds) - -```graphql -query getDuration { - temporalDataObject(id: "112971783") { - tasks { - records { - id - mediaLengthSec - } - } - } -} -``` - -### Get Organization Info for Current Session - -```graphql -query getOrgInfo { - me { - organization { - name - id - } - } -} -``` - -### List Available Engines - -?> For demonstration purposes this query uses `limit: 1000`. -In actual practice you will want to use a lower limit and ensure that you are paginating through the list of engines by using the `offset` parameter until you receive an empty list. - -```graphql -query listEngines { - engines(limit: 1000) { - count - records { - id - name - category { - id - name - } - fields { - name - type - max - min - step - info - label - defaultValue - options { - key - value - } - } - } - } -} -``` - -### List Available Engines (Grouped By Category) - -?> For demonstration purposes this query uses `limit: 1000`. -In actual practice you will want to use a lower limit and ensure that you are paginating through the list of engines by using the `offset` parameter until you receive an empty list. - -```graphql -query listEnginesByCategory { - engineCategories { - records { - id - name - engines(limit: 1000) { - records { - id - name - fields { - name - options { - value - } - } - } - } - } - } -} -``` - -### List Available Engine Categories - -```graphql -query listEngineCategories { - engineCategories { - records { - id - name - description - } - } -} -``` - -### List Engines for a Specific Category - -?> For demonstration purposes this query uses `limit: 1000`. -In actual practice you will want to use a lower limit and ensure that you are paginating through the list of engines by using the `offset` parameter until you receive an empty list. - -```graphql -query listEnginesForCategory { - engines(category: "transcription", limit: 1000) { - count - records { - id - name - fields { - name - type - max - min - step - info - label - defaultValue - options { - key - value - } - } - } - } -} -``` - -### List Engine’s Custom Fields - -```graphql -query engineCustomFields { - engine(id: "b396fa74-83ff-4052-88a7-c37808a25673") { - id - fields { - type - name - defaultValue - info - options { - key - value - } - } - } -} -``` - -### Add Engines to White List - -!> This operation requires elevated rights in the system. - -```graphql -mutation addToWhiteList { - addToEngineWhitelist(toAdd:{ - organizationId:13456 - engineIds:["",""] - }){ - organizationId - engines{ - name - id - } - } -} -``` - -### Get All Roles in an Organization - -> This method requires elevated rights in the system. - -```graphql -query getAllRoles { - organization(id: 17532) { - roles { - name - id - permissions { - records { - id - name - description - } - } - } - } -} -``` - -### Fix Video Playback Issue in CMS - -Step 1: List all TDOs and identify which ones have a bad primary asset based on `signedUri` field. - -```graphql -query listPrimaryAsset { - temporalDataObjects(limit: 100) { - records { - id - primaryAsset(assetType:"media"){ - signedUri - } - } - } -} -``` - -Step 2: Identify the right Asset ID to use as the TDO's primary asset. - -```graphql -query primaryAsset { - temporalDataObject(id: "117679345") { - details - primaryAsset(assetType: "media") { - signedUri - type - } - assets { - records { - id - assetType - signedUri - } - } - } -} -``` - -Step 3: Update the primary asset. - -```graphql -mutation setPrimaryAsset { - updateTDO( input: { - id: 117679345, - primaryAsset: - { id: "117679345_t6l92LBhp0", assetType: "media" } - }) - { - id - } -} -``` - -## Real Time - -Step 1: Create a TDO or "Container" for use in Step 2: - -```graphql -mutation createTDOWithAsset { - createTDOWithAsset(input: { - startDateTime: 1548880932, - updateStopDateTimeFromAsset: true, - contentType: "video/mp4", - assetType: "media", addToIndex: true, - uri: "https://s3.amazonaws.com/hold4fisher/Manchester+United+4-3+Real+Madrid+-+UEFA+CL+2002_2003+%5BHD%5D-W7HM1RfNfS4.mp4" - }) - { - id - status - assets { - records { - id - type - contentType - signedUri - } - } - } -} -``` - -RESPONSE: - -```json -{ - "data": { - "createTDOWithAsset": { - "id": "390257661", - "status": "recorded", - "assets": { - "records": [ - { - "id": "390257661_R9Ij9DhD9L", - "type": "media", - "contentType": "video/mp4", - "signedUri": "https://s3.amazonaws.com/hold4fisher/Manchester+United+4-3+Real+Madrid+-+UEFA+CL+2002_2003+%5BHD%5D-W7HM1RfNfS4.mp4" - } - ] - } - } - } -} -``` - -Step 2: Create a Job involving face recognition, using the ID obtained from the response to Step 1: - -```graphql -mutation createJob { - createJob(input: { - targetId: "390257661", - isReprocessJob:true - tasks: [ - {engineId: "5e651457-e102-4d16-a8f2-5c0c34f58851"}]}) { - id - status - } -} -``` - -RESPONSE: - -```json -{ - "data": { - "createJob": { - "id": "19031004_ADKtN72ZWZ", - "status": "pending" - } - } -} -``` - -Step 3: Get the results of the job created in step 2: - -```graphql -query getEngineResults { - engineResults(tdoId: "390257661", engineIds: ["5e651457-e102-4d16-a8f2-5c0c34f58851"]) { - records { - jsondata - } - } -} -``` - -You can also provide the `jobId` instead of the `engineId`: - -```graphql -query getEngineResults { - engineResults(tdoId: "390257661", jobId:"19031004_ADKtN72ZWZ") { - records { - jsondata - } - } -} -``` diff --git a/docs/apis/job-quickstart/README.md b/docs/apis/job-quickstart/README.md deleted file mode 100644 index 7ea6a5872d..0000000000 --- a/docs/apis/job-quickstart/README.md +++ /dev/null @@ -1,541 +0,0 @@ -# Job Quickstart Guide - -## Getting Started - -The aiWARE Job API allows you to easily integrate cognitive functionality -such as object detection, language translation, and voice transcription -with just a few lines of code. -The Job API is a set of calls that takes you through the workflow for -performing cognitive task processing — from file ingestion to engine -processing and retrieving output results. - -aiWARE's API is built around the GraphQL paradigm to provide a more -efficient way to deliver data with greater flexibility than a traditional REST approach. -GraphQL is a [query language](http://graphql.org/learn/queries/) that operates over a single -endpoint using conventional HTTP requests and returning JSON responses. -The structure not only lets you call multiple nested resources in a single query, -it also allows you to define requests so that only requested data is sent back. - -This quickstart guide provides resources, detailed documentation, and example requests and responses to help get your integration up and running to perform the following operations: - -1. [Create a TDO (recording container)](#create-a-tdo) -1. [Create a job](#create-a-job) consisting of one or more tasks -1. [Check the status of a job](#check-the-job-status) -1. [Retrieve job results](#retrieve-job-output) -1. [Delete a TDO (recording container)](#delete-a-tdo-andor-its-content) - -We designed this quickstart to be user friendly and example filled, -but if you have any questions, please don’t hesitate to reach out to our [Developer Support Team](mailto:devsupport@veritone.com) for help. - -### Base URL - -aiWARE uses a single endpoint for accessing the API. -All calls to the API are POST requests and are served over *http* with *application/json* encoded bodies. -The base URL varies based on the geographic region where the services will run. -When configuring your integration, choose the base URL from the list below that supports your geographic location. - -| Region | Base URL | -| ------ | -------- | -| United States (Default) | https://api.veritone.com/v3/graphql | -| United Kingdom | https://api.uk.veritone.com/v3/graphql | - -> The above base URLs are provided for use within SaaS environments. -Applications using GovCloud, on-prem, or other deployments access the API via an endpoint that's custom configured to the private network. - -### Making Sample Requests - -To make it easier to explore, write, and test the API, we set up [GraphiQL](https://api.veritone.com/v3/graphiql) — an interactive playground that gives you a code editor with autocomplete, validation, and syntax error highlighting features. -Use the [GraphiQL interface](https://api.veritone.com/v3/graphiql) to construct and execute queries, experiment with different schema modifications, and browse documentation. -In addition, GraphiQL bakes authorization right into the schema and automatically passes the `Authentication` header with a valid token when you’re logged into the aiWARE system. - -aiWARE’s [GraphiQL interface](https://api.veritone.com/v3/graphiql) is the recommended method for ad hoc API requests, but calls can be made using any HTTP client. -All requests must be HTTP POST to the base URL designated for your geographic region with the `query` parameter and `application/json` encoded bodies. -In addition, requests must be authenticated using an API Token. -Pass the token in your request using the *Authorization* header with a value `Bearer token`. -If you’re using a raw HTTP client, the query body contents must be sent in a string with all quotes escaped. - -The sample requests provided in this documentation are structured for use in our [GraphiQL interface](https://api.veritone.com/v3/graphiql), but we’ve also included the basic cURL structure for your reference below. -Please note that the examples shown throughout this guide do not use client information and are not language specific. -For fields that require account-specific data (such as a containerId), replace the value with your own. -In addition, the sample requests shown are not all-inclusive — they highlight the minimum requirements and relevant information. -Additional attributes for each request can be found in our [GraphQL docs](https://api.veritone.com/v3/graphqldocs/). - -#### Basic cURL Structure for Requests - -```bash -curl -X POST \ - https://api.veritone.com/v3/graphql \ - -H 'authorization: Bearer 31gcf6:2e76022093e64732b4c48f202234394328abcf72d50e4981b8043a19e8d9baac' \ - -H 'content-type: application/json' \ - -d '{"query": "mutation { createTDO( input: { startDateTime: 1507128535, stopDateTime: 1507128542, name: \"My New Video\", description: \"The latest video in the series\" }) { id, status } }" }' -``` - -### Authentication - - - -aiWARE's Job API uses bearer token authentication for requests. To authenticate your calls, provide a valid API Token in the `Authentication` header of the request with the value `Bearer token`. -Requests made without this header or with an invalid token will return an error code. - -An API Token can be generated in the Veritone Admin App by your Organization Administrator. -If your organization does not use the Admin App, please contact your Veritone Account Manager for assistance. - -**To generate an API Token:** - -1. Log into the aiWARE Platform and select **Admin** from the *App Picker* drop-down. The *Admin App* opens. - -1. Click the **API Keys** tile. The *API Key* page opens. - - ![Get API Token](../_media/Get-API-Token-1.png) - -1. Click **New API** Key. The *New API Key* window opens. - - ![Get API Token](../_media/Get-API-Token-2.png) - -1. Enter a token name and select the permissions needed for the token to perform the required API tasks. Click **Generate Token** to save. The *Token Generated *window opens. - -1. Copy your token and click **Close** when finished. - -?> Once the *Token Generated* window closes, the token code no longer displays and it cannot be viewed again. -If you misplace the token, you should delete the token and generate a new one. - -## How to Create a Cognitive Processing Job: High-Level Summary - -For this quickstart, we will run a transcription job on a video file (.mp4) stored at an Amazon S3 URL. -This is a relatively simple example, but it shows the overall pattern you will follow in running cognitive processing jobs. -The approach is to: - - - -1. Create a TDO. -2. Create the job. -3. Poll for status. -4. Obtain the output of the job. -5. Clean up. - -Each of these steps involves its own mutation or query. - -The second step not only specifies the tasks associated with the job, but actually queues and kicks off the job. - -Note that a job can contain one or more *tasks*. -Each task is associated with an engine. -The task will also usually specify an asset to operate against. -(You will see how this works in the [Create a Job](#create-a-job) section below.) -The choice of how many related tasks to wrap in a given job is up to you. -A lot depends on the scenario. -For example, in a job that will create a transcript from an audio or video file, there may -need to be a `Transcoding` task to convert the file to a supported format for processing. - -Some of these concepts will become clearer as you read through the example shown below. - -## Create a TDO - -In aiWARE, jobs and job-related artifacts (such as media files) need to be associated with -a container called a [Temporal Data Object (TDO)](https://api.veritone.com/v3/graphqldocs/temporaldataobject.doc.html). -The first step in the Job workflow is thus to create a TDO. This is easy to do: - -```graphql -mutation createTDO { - createTDO( - input: { - startDateTime: "2019-04-24T21:49:04.412Z", - stopDateTime: "2019-04-24T21:50:04.412Z" - } - ) - { - id - } -} -``` - -?> The `startDateTime` and `stopDateTime` values are required, but can be dummy values. -The only firm requirements are that the values exist, and that the -second value is greater than the first (but not by more than a year). -You can supply an integer here (milliseconds), or any ISO-8601-legal string. -E.g. `"20190424T174428Z"` and `"2019-04-24T21:49:04.412Z"` are both acceptable string values. - -> Set the TDO's `addToIndex` field to `false` if you want to _disable_ automatic indexing of TDO assets for purposes of Search. - -A successful response will look like: - -```json -{ - "data": { - "createTDO": { - "id": "460907869" - } - } -} -``` - -Take note of the `id`. You will need it to create a job. - -## Create a Job - -If you know the URI of the media file you wish to process, you can immediately create the job that will run against that media file. -You just need the URI of the file, the `targetId` of the TDO you created in the previous step, -and the `engineId` values of any engines that will be called upon to ingest and process the file. - -For example, to run a transcription job on an `.mp4` file: - -```graphql -mutation createJob { - createJob(input: { - targetId: "460907869", # the TDO ID (from the previous step) - tasks: [{ - engineId: "9e611ad7-2d3b-48f6-a51b-0a1ba40feab4", # ID of webstream adapter - payload: { - url: "https://s3.amazonaws.com/dev-chunk-cache-tmp/AC.mp4" - } - },{ - engineId: "54525249-da68-4dbf-b6fe-aea9a1aefd4d" # ID of a transcription engine - }] - }) { - id - } -} -``` - -This job has two tasks: one representing ingestion, and the other representing transcription. - -The response will look like: - -```json -{ - "data": { - "createJob": { - "id": "19041508_f2qIbo9qAh" - } - } -} -``` - -Take note of the returned `id` value. -This is the value by which you will refer the job in the next step. - -> If you want to process files larger than 100MB, please see additional information on -[Uploading Large Files](/apis/tutorials/uploading-large-files.md). - -## Check the Job Status - -Jobs run asynchronously. -You can check the job status (and also the individual `status` fields of the tasks) -with a query similar to this: - -```graphql -query queryJobStatus { - job(id:"19041508_f2qIbo9qAh") { - id - status - targetId - tasks{ - records{ - id - status - engine{ - id - name - } - # taskOutput <= For more granular/verbose output, include this field - } - } - } -} -``` - -> The possible job statuses are `cancelled`, `complete`, `pending`, `queued`, or `running`. -Tasks can have those same statuses, as well as `aborted`, `accepted`, -`failed`, `resuming`, `standby_pending`, or `waiting`. - -The response may look similar to: - -```json -{ - "data": { - "job": { - "status": "complete", - "createdDateTime": "2019-04-15T19:45:44.000Z", - "targetId": "460907869", - "tasks": { - "records": [ - { - "status": "complete", - "createdDateTime": "2019-04-15T19:45:44.000Z", - "modifiedDateTime": "2019-04-15T19:49:40.000Z", - "id": "19041615_krfPuFO0jVj713Z", - "engine": { - "id": "54525249-da68-4dbf-b6fe-aea9a1aefd4d", - "name": "Transcription - DR - English (Global)", - "category": { - "name": "Transcription" - } - } - }, - { - "status": "complete", - "createdDateTime": "2019-04-15T19:45:44.000Z", - "modifiedDateTime": "2019-04-15T19:46:42.000Z", - "id": "19041615_krfPuFO0jVOsxjE", - "engine": { - "id": "ea0ada2a-7571-4aa5-9172-b5a7d989b041", - "name": "Stream Ingestor", - "category": { - "name": "Intracategory" - } - } - }, - { - "status": "complete", - "createdDateTime": "2019-04-15T19:45:44.000Z", - "modifiedDateTime": "2019-04-15T19:46:04.000Z", - "id": "19041615_krfPuFO0jVtiZFm", - "engine": { - "id": "9e611ad7-2d3b-48f6-a51b-0a1ba40feab4", - "name": "Webstream Adapter", - "category": { - "name": "Pull" - } - } - } - ] - } - } - } -} -``` - -## Retrieve Job Output - -Once a job's status is `complete`, you can request the output of a task, -such as a transcript, translation, or object detection results. - -Use the TDO `id`, along with the appropriate engine `id`, to query the item of interest. - -```graphql -query getEngineOutput { - engineResults(tdoId: "460907869", - engineIds: ["54525249-da68-4dbf-b6fe-aea9a1aefd4d"]) { # ID of the transcription engine - records { - tdoId - engineId - startOffsetMs - stopOffsetMs - jsondata - assetId - userEdited - } - } -} -``` - -The response will be a (potentially sizable) JSON object. -The following example has a shortened `series` array; normally it is much longer. - -```json -{ - "data": { - "engineResults": { - "records": [ - { - "tdoId": "460907869", - "engineId": "54525249-da68-4dbf-b6fe-aea9a1aefd4d", - "startOffsetMs": 0, - "stopOffsetMs": 172250, - "jsondata": { - "taskId": "19041615_krfPuFO0jVj713Z", - "generatedDateUTC": "0001-01-01T00:00:00Z", - "series": [ - { - "startTimeMs": 0, - "stopTimeMs": 960, - "words": [ - { - "word": "We", - "confidence": 1, - "bestPath": true, - "utteranceLength": 1 - } - ], - "language": "en" - } - ], - "modifiedDateTime": 1555357780000, - "sourceEngineId": "54525249-da68-4dbf-b6fe-aea9a1aefd4d" - }, - "assetId": "450192755_lJFc2CsiOz", - "userEdited": null - } - ] - } - } -} -``` - -As you can see, transcripts contain time-correlated, word-based text fragments with -beginning and ending times. A successful call returns the transcript data, with an `assetId`, and -other details specified in the request. Otherwise, an error is returned. - -## Requesting a Specific Output Format - -Sometimes you may want a particular flavor of output (such as `ttml` or `srt` for captioning). -In that case, you can use the `createExportRequest` mutation: - -```graphql -mutation createExportRequest { - createExportRequest(input: { - includeMedia: false, - tdoData: [{tdoId: "431011721"}], - outputConfigurations: [{ - engineId: "71ab1ba9-e0b8-4215-b4f9-0fc1a1d2b44d", - formats: [{ - extension: "vtt", - options: {newLineOnPunctuation: false} - }] - }] - }) { - id - status - organizationId - createdDateTime - modifiedDateTime - requestorId - assetUri - } -} -``` - -The response: - -```json -{ - "data": { - "createExportRequest": { - "id": "a2efc2bb-e09f-40bf-a2bc-1d25297ac2f7", - "status": "incomplete", - "organizationId": "17532", - "createdDateTime": "2019-04-25T20:45:20.784Z", - "modifiedDateTime": "2019-04-25T20:45:20.784Z", - "requestorId": "960b3fa8-1812-4303-b58d-4f0d227f2afc", - "assetUri": null - } - } -} -``` - -Since an export request may take time to process, you should poll until the status is complete, -using the `id` returned above. - -```graphql -query exportRequest { - exportRequest(id: "a2efc2bb-e09f-40bf-a2bc-1d25297ac2f7") { - status - assetUri - requestorId - } -} -``` - -The response (showing that the export is, in this case, incomplete): - -```json -{ - "data": { - "exportRequest": { - "status": "incomplete", - "assetUri": null, - "requestorId": "960b3fa8-1812-4303-b58d-4f0d227f2afc" - } - } -} -``` - -When the status becomes `complete`, you can retrieve the results at the URL returned in the `assetUri` field. - -## Delete a TDO and/or Its Content - -If a TDO is no longer needed, it can be deleted from an organization’s files -to free up storage space or comply with organizational policies. -The API provides flexible options that allow you to delete a TDO and all of its assets, -or clean up a TDO's content by removing the associated assets so the TDO can be reused -and new assets can be created. - -!> Deleting a TDO data permanently removes contents from aiWARE and they will no longer be accessible via CMS, search, or any other method. -**Deleting a TDO cannot be undone**! - -### Delete a TDO and All Assets - -To delete a TDO and all asset metadata, make a request to the `deleteTDO` mutation -and pass the TDO `id` as an argument. -This operation is processed immediately at the time of the request and -permanently deletes the specified TDO *as well as its assets* from the organization's account. -Any subsequent requests against the TDO or assets will return an error. - -First, we'll look at how to delete the entire TDO. -Then we'll discuss how to remove *just the content items*. - -#### Example: Delete a TDO - -```graphql -mutation{ - deleteTDO(id: "44512341") - { - id - message - } - } -``` - -The response is: - -```json -{ - "data": { - "deleteTDO": { - "id": "44512341", - "message": "TemporalDataObject 44512341 was deleted." - } - } -} -``` - -### Remove TDO Content - -To remove just the asset (content) associated with a TDO, while retaining the TDO/container -and asset metadata, make a request to the `cleanupTDO` mutation with the TDO `id`. -This mutation uses the `options` parameter along with any combination of the values below -to specify the type(s) of data to be deleted. - -* `storage`: Deletes the TDO's assets from storage, including engine results. Asset metadata will remain until the TDO/container is deleted. -* `searchIndex`: Deletes all search index data. The TDO and its assets will no longer be accessible through search. -* `engineResults`: Deletes engine results stored on related task objects. Engine results that are stored as assets will remain unless `storage` is passed as a value in the request. - -!> Requests that do not use the `options` parameter will remove the TDO's content from `storage` and the `search index` by default. - -#### Example: Remove TDO Content (storage, engineResults) - -```graphql -mutation { - cleanupTDO(id: "44512341", options: [storage, engineResults]) { - id - message - } - } -``` - -Response: - - ```json - { - "data": { - "cleanupTDO": { - "id": "44512341", - "message": "Data deleted from 44512341: storage,engineResults" - } - } -} -``` - - diff --git a/docs/apis/jobs-tasks-tdos.md b/docs/apis/jobs-tasks-tdos.md deleted file mode 100644 index 4b1358b3e8..0000000000 --- a/docs/apis/jobs-tasks-tdos.md +++ /dev/null @@ -1,159 +0,0 @@ -# Jobs, Tasks, and TDOs - -Throughout our APIs and this documentation, you will see frequent reference to Jobs, Tasks, and TDOs (Temporal Data Objects). These are the workhorse objects of Veritone's aiWARE platform. - -Let's take a quick look at each of these, in reverse order: - -## TDO (Temporal Data Object) - -The [Temporal Data Object](https://api.veritone.com/v3/graphqldocs/temporaldataobject.doc.html) is an all-purpose container object, aggregating information about jobs, assets, and temporal data (among other things). - -Important facts to know about TDOs are: - -* You will generally need to manage the lifecycle of a TDO yourself. Although some engines may create a TDO on their own, it is far more common that you will submit a TDO — that _you_ created — when kicking off a Job with `createJob()`. - -* When you no longer need a TDO, you can [delete it programmatically](apis/job-quickstart/?id=delete-a-tdo-andor-its-content), or you can [purge its contents](apis/job-quickstart/?id=remove-tdo-content). Otherwise, it lives forever. - -* TDOs you create are generally visible (and thus usable) only by members of your Organization. - -* You will often [create an empty TDO programmatically](apis/job-quickstart/?id=create-a-tdo), then run an ingestion task on it to populate it with a media asset. - -* When processing a media file referenced in your TDO, an engine will produce its own output (e.g., transcription output) in the form of a `vtn-standard` asset, which will be attached to your TDO _by reference_. - -* A TDO can contain multiple assets of multiple types. (See [Asset Types](apis/tutorials/asset-types?id=asset-types) for more information.) - -## Task - -The [Task](https://api.veritone.com/v3/graphqldocs/task.doc.html) is the smallest unit of work in aiWARE. - -Things to know: - -* A Task specifies an [engine](https://api.veritone.com/v3/graphqldocs/engine.doc.html) that will be run against a TDO. - -* Tasks are run as part of a [Job](https://api.veritone.com/v3/graphqldocs/job.doc.html) (see below). - -* A Task can be queried at any time using the GraphQL `task()` method. - -* The possible status values that a Task can have are shown below. - -```graphql -enum TaskStatus { - pending - running - complete - queued - accepted - failed - cancelled - standby_pending - waiting - resuming - aborted - paused -} -``` - -> If a Task finishes with a status of `aborted` or `failed`, it will cause the Job of which it is a part to finish with a status of `failed`. - -## Job - -The [Job](https://api.veritone.com/v3/graphqldocs/job.doc.html) is a higher-level unit of work that wraps one or more Tasks. - -> If you need to aggregate Jobs into an even higher-level unit of work, consider using [Veritone Automate Studio](https://automate.veritone.com/) to create a multi-Job workflow. - -Things to remember: - -* You can create and queue (and thus essentially launch, immediately and asynchronously) a job using the GraphQL `createJob()` method. - -* A Job needs to operate against a TDO. You should specify the TDO's ID in the `targetID` property when you call `createJob()`. - -* The order in which you list Tasks, in your call to `createJob()`, is important. If your Job needs to ingest a media file, the ingestion-engine task should be the first Task in your list of Tasks. - -* You can (and should) check a Job's status using the Job ID returned by `createJob()`. (See [Check the Job Status](apis/job-quickstart/?id=check-the-job-status) for an example of how to do this.) - -* A Job can have any of the status values shown below. - -```graphql -enum JobStatus { - pending - complete - running - cancelled - queued - failed -} -``` - -> Be sure to consult the [Job Quickstart Guide](apis/job-quickstart/) for a more complete discussion of how to create, run, monitor, and obtain data from Jobs. - -## Ingestion - -Ingestion refers to the intake of media files into a CMS, DAM, or MAM system. -When a file is ingested, it is generally copied to a secure location, registered with the host system, and optionally chunked, transcoded, tagged, indexed, thumbnailed, and/or subjected to other "normalizing" operations, such that the system can operate on all ingested files reliably, using the same APIs, with the same expectations, no matter where a file originally came from. - -In Veritone's aiWARE system, a file can undergo cognitive processing if and only if it has been ingested. -The two most common ways to ingest a media file for processing in aiWARE are: - -1\. Create a TDO and pull the media asset into it, in one operation, using `createTDOWithAsset()`. (See [this example](apis/examples?id=create-tdo-and-upload-asset) in our API docs.) - -2\. Create a TDO manually and then run an ingestion job on it using `createJob()` in conjunction with an appropriate ingestion engine (also called an [adapter](developer/adapters/?id=adapter-workflow)). -Veritone aiWARE offers many ready-to-use ingestion engines tailored to various intake scenarios, such as pulling videos (or other files) from YouTube, Google Drive, Dropbox, etc. -To see a list of the available ingestion engines (adapters) in aiWARE, run the following GraphQL query: - - ```graphql - query listIngestionEngines { - engines(filter: { - type: Ingestion - }) { - records { - name - id - } - } - } - ``` - -> You'll commonly use the Webstream Adapter — with ID `"9e611ad7-2d3b-48f6-a51b-0a1ba40feab4"` — to pull files from public URIs. - -### Example Ingestion Job - -The following example assumes that you have already created a TDO with ID `88900861`. - -To pull a media file called `s3Test.mp4` (located at `https://s3.amazonaws.com/holdings/s3Test.mp4`) into aiWARE, you could run the following mutation: - -```graphql -mutation runIngestionJob { - createJob(input: { - targetId: "88900861", - tasks: [{ - engineId: "9e611ad7-2d3b-48f6-a51b-0a1ba40feab4", - payload: { - url: "https://s3.amazonaws.com/holdings/s3Test.mp4" - } - }] - }) { - id - } -} -``` - -Once the ingestion job finishes (assuming it completes normally), you can submit the same TDO as part of a cognitive processing job, _without a need to re-ingest the file._ - -When submitting a TDO that already contains an ingested asset to `createJob()`, it's important that you set the `isReprocessJob` flag to `true`, as shown here: - -```graphql -mutation createJob { - createJob(input: { - targetId: "88900861", - isReprocessJob:true - tasks: [ - { - engineId: "5e651457-e102-4d16-a8f2-5c0c34f58851" - }]}) { - id - status - } -} -``` - -!> Failure to set the `isReprocessJob` flag could result in a hung job. diff --git a/docs/apis/reference/mutation/README.md b/docs/apis/reference/mutation/README.md deleted file mode 100644 index 1f7471e2fd..0000000000 --- a/docs/apis/reference/mutation/README.md +++ /dev/null @@ -1,2552 +0,0 @@ -# Mutation Methods - -The table below gives a quick summary of GraphQL [mutation](https://api.veritone.com/v3/graphqldocs/mutation.doc.html) methods, alphabetized by name. - -Click any name to see the complete method signature, and other info. - -| Method name | Short Description | -| -- | -- | -| [addLibraryDataset](#addlibrarydataset) | Add recordings to a dataset library | -| [addMediaSegment](#addmediasegment) | Add a single media segment to a TemporalDataObject | -| [addTasksToJobs](#addtaskstojobs) | Add tasks to jobs | -| [addToEngineBlacklist](#addtoengineblacklist) | Add to engine blacklist | -| [addToEngineWhitelist](#addtoenginewhitelist) | Add to engine whitelist | -| [applicationWorkflow](#applicationworkflow) | Apply an application workflow step, such as "submit" or "approve" | -| [bulkDeleteContextMenuExtensions](#bulkdeletecontextmenuextensions) | Bulk delete context meu extensions | -| [bulkUpdateWatchlist](#bulkupdatewatchlist) | Apply bulk updates to watchlists | -| [cancelJob](#canceljob) | Cancel a job | -| [changePassword](#changepassword) | Change the current authenticated user's password | -| [cleanupTDO](#cleanuptdo) | Delete partial information from a temporal data object | -| [createApplication](#createapplication) | Create a new application | -| [createAsset](#createasset) | Create a media asset | -| [createCognitiveSearch](#createcognitivesearch) | Create cognitive search | -| [createCollection](#createcollection) | Create (ingest) a structured data object | -| [createCollectionMention](#createcollectionmention) | Add a mention to a collection | -| [createCollectionMentions](#createcollectionmentions) | Create collection mentions | -| [createCreative](#createcreative) | Create a creative | -| [createDataRegistry](#createdataregistry) | Create a structured data registry schema metadata | -| [createEngine](#createengine) | Create a new engine | -| [createEngineBuild](#createenginebuild) | Create an engine build | -| [createEntity](#createentity) | Create a new entity | -| [createEntityIdentifier](#createentityidentifier) | Create an entity identifier | -| [createEntityIdentifierType](#createentityidentifiertype) | Create an entity identifier type, such as "face" or "image" | -| [createEvent](#createevent) | Create a new event | -| [createExportRequest](#createexportrequest) | Create an export request | -| [createFolder](#createfolder) | Create a new folder | -| [createFolderContentTempate](#createfoldercontenttempate) | Create new content template into a folder | -| [createIngestionConfiguration](#createingestionconfiguration) | Create an ingestion configuration | -| [createJob](#createjob) | Create a job | -| [createLibrary](#createlibrary) | Create a new library | -| [createLibraryConfiguration](#createlibraryconfiguration) | Create Dataset Library Configuration | -| [createLibraryEngineModel](#createlibraryenginemodel) | Create a library engine model | -| [createLibraryType](#createlibrarytype) | Create a library type, such as "ad" or "people" | -| [createMediaShare](#createmediashare) | Create Media Share | -| [createMention](#createmention) | Create a mention object | -| [createMentionComment](#creatementioncomment) | Create a mention comment | -| [createMentionExportRequest](#creatementionexportrequest) | Create a mention export request | -| [createMentionRating](#creatementionrating) | Create a mention rating | -| [createMentions](#creatementions) | Create Mention in bulk | -| [createOrganization](#createorganization) | Create a new organization | -| [createPasswordResetRequest](#createpasswordresetrequest) | Create a password reset request | -| [createPasswordUpdateRequest](#createpasswordupdaterequest) | Force a user to update password on next login | -| [createProcessTemplate](#createprocesstemplate) | Create a processTemplate in CMS | -| [createRootFolders](#createrootfolders) | Create root folder for an organization | -| [createSavedSearch](#createsavedsearch) | Create a new Saved Search | -| [createStructuredData](#createstructureddata) | Create (ingest) a structured data object | -| [createSubscription](#createsubscription) | Create subscription | -| [createTDO](#createtdo) | Create a new temporal data object | -| [createTDOWithAsset](#createtdowithasset) | Create a TDO and an asset with a single call | -| [createTaskLog](#createtasklog) | Create a task log by using multipart form POST | -| [createTriggers](#createtriggers) | Create trigger for events or types | -| [createUser](#createuser) | Create a new user within an organization | -| [createWatchlist](#createwatchlist) | Create watchlist | -| [createWidget](#createwidget) | Creates a widget associated with a collection | -| [deleteApplication](#deleteapplication) | Delete an application | -| [deleteAsset](#deleteasset) | Delete an asset | -| [deleteCognitiveSearch](#deletecognitivesearch) | Delete cognitive search | -| [deleteCollection](#deletecollection) | Delete Collection | -| [deleteCollectionMention](#deletecollectionmention) | Remove a mention from a collection | -| [deleteCreative](#deletecreative) | Delete a creative | -| [deleteEngine](#deleteengine) | Delete an engine | -| [deleteEngineBuild](#deleteenginebuild) | Delete an engine build | -| [deleteEntity](#deleteentity) | Delete an entity | -| [deleteEntityIdentifier](#deleteentityidentifier) | Delete an entity identifier | -| [deleteFolder](#deletefolder) | Delete a folder | -| [deleteFolderContentTempate](#deletefoldercontenttempate) | Delete existing folder content template by folderContentTemplateId | -| [deleteFromEngineBlacklist](#deletefromengineblacklist) | Delete from engine blacklist | -| [deleteFromEngineWhitelist](#deletefromenginewhitelist) | Delete from engine whitelist | -| [deleteIngestionConfiguration](#deleteingestionconfiguration) | Delete an ingestion configuration | -| [deleteLibrary](#deletelibrary) | Delete a library | -| [deleteLibraryConfiguration](#deletelibraryconfiguration) | Delete Dataset Library Configuration | -| [deleteLibraryDataset](#deletelibrarydataset) | Remove recordings from a dataset library | -| [deleteLibraryEngineModel](#deletelibraryenginemodel) | Delete a library engine model | -| [deleteMentionComment](#deletementioncomment) | Delete a mention comment | -| [deleteMentionRating](#deletementionrating) | Delete a mention rating | -| [deleteProcessTemplate](#deleteprocesstemplate) | Delete a processTemplate by ID in CMS | -| [deleteSavedSearch](#deletesavedsearch) | Delete a saved search | -| [deleteStructuredData](#deletestructureddata) | Delete a structured data object | -| [deleteSubscription](#deletesubscription) | Delete subscription | -| [deleteTDO](#deletetdo) | Delete a temporal data object | -| [deleteTrigger](#deletetrigger) | Delete a registed trigger by ID | -| [deleteUser](#deleteuser) | Delete a user | -| [deleteWatchlist](#deletewatchlist) | Delete watchlist | -| [emitEvent](#emitevent) | Emit an event | -| [emitSystemEvent](#emitsystemevent) | Emit a system-level emit | -| [engineWorkflow](#engineworkflow) | Apply an application workflow step, such as "submit" or "approve" | -| [fileTemporalDataObject](#filetemporaldataobject) | File a TemporalDataObject in a folder | -| [fileWatchlist](#filewatchlist) | File watchlist | -| [getCurrentUserPasswordToken](#getcurrentuserpasswordtoken) | Get password token info for current user | -| [getEngineJWT](#getenginejwt) | JWT tokens with a more limited scoped token to specific resources to the recording, task, and job and also has no organization association | -| [moveFolder](#movefolder) | Move a folder from one parent folder to another | -| [moveTemporalDataObject](#movetemporaldataobject) | Moves a TemporalDataObject from one parent folder to another | -| [pollTask](#polltask) | Poll a task | -| [publishLibrary](#publishlibrary) | Publish a new version of a library | -| [refreshToken](#refreshtoken) | Refresh a user token, returning a fresh token so that the client can continue to authenticate to the API | -| [replaceSavedSearch](#replacesavedsearch) | Mark existing saved search profile as deleted Create new saved search profile | -| [requestClone](#requestclone) | Start a clone job | -| [retryJob](#retryjob) | Retry a job | -| [sendEmail](#sendemail) | Send a basic email | -| [setWorkflowRuntimeStorageData](#setworkflowruntimestoragedata) | Create or Update Workflow data | -| [shareCollection](#sharecollection) | Share a collection, allowing other organizations to view the data it contains | -| [shareFolder](#sharefolder) | Share a folder with other organizations | -| [shareMention](#sharemention) | Share mention | -| [shareMentionFromCollection](#sharementionfromcollection) | Share a mention from a collection | -| [shareMentionInBulk](#sharementioninbulk) | Share mentions in bulk | -| [startWorkflowRuntime](#startworkflowruntime) | Start a Veritone Workflow instance | -| [stopWorkflowRuntime](#stopworkflowruntime) | Shutdown Veritone Workflow instance | -| [subscribeEvent](#subscribeevent) | Subscribe to an event | -| [unfileTemporalDataObject](#unfiletemporaldataobject) | Unfile a TemporalDataObject from a folder | -| [unfileWatchlist](#unfilewatchlist) | Unfile watchlist | -| [unsubscribeEvent](#unsubscribeevent) | Unsubscribe to an event | -| [updateApplication](#updateapplication) | Update a custom application | -| [updateAsset](#updateasset) | Update an asset | -| [updateCognitiveSearch](#updatecognitivesearch) | Update cognitive search | -| [updateCollection](#updatecollection) | Update a collection | -| [updateCreative](#updatecreative) | Update a creative | -| [updateCurrentUser](#updatecurrentuser) | Update the current authenticated user | -| [updateDataRegistry](#updatedataregistry) | Update a structured data registry schema metadata | -| [updateEngine](#updateengine) | Update an engine | -| [updateEngineBuild](#updateenginebuild) | Update an engine build | -| [updateEntity](#updateentity) | Update an entity | -| [updateEntityIdentifier](#updateentityidentifier) | Update entity identifier | -| [updateEntityIdentifierType](#updateentityidentifiertype) | Update an entity identifier type | -| [updateEvent](#updateevent) | Update an event | -| [updateExportRequest](#updateexportrequest) | Update an export request | -| [updateFolder](#updatefolder) | Update an existing folder | -| [updateFolderContentTempate](#updatefoldercontenttempate) | Update existing content template by folderContentTemplateId | -| [updateIngestionConfiguration](#updateingestionconfiguration) | Update an ingestion configuration | -| [updateJobs](#updatejobs) | Update jobs | -| [updateLibrary](#updatelibrary) | Update an existing library | -| [updateLibraryConfiguration](#updatelibraryconfiguration) | Update Dataset Library Configuration | -| [updateLibraryEngineModel](#updatelibraryenginemodel) | Update a library engine model | -| [updateLibraryType](#updatelibrarytype) | Update a library type | -| [updateMention](#updatemention) | Update a mention object | -| [updateMentionComment](#updatementioncomment) | Update a mention comment | -| [updateMentionExportRequest](#updatementionexportrequest) | Update status or assetURI of a mentionExportRequest Often use when the file export was completed or downloaded | -| [updateMentionRating](#updatementionrating) | Update a mention rating | -| [updateMentions](#updatementions) | Update a set of mentions | -| [updateOrganization](#updateorganization) | Update an organization | -| [updateProcessTemplate](#updateprocesstemplate) | Update a processTemplate by ID in CMS | -| [updateSchemaState](#updateschemastate) | Update schema state | -| [updateSharedCollectionHistory](#updatesharedcollectionhistory) | Update shared collection history | -| [updateSharedCollectionMentions](#updatesharedcollectionmentions) | Update shared collection mentions | -| [updateSubscription](#updatesubscription) | Update subscription | -| [updateTDO](#updatetdo) | Update a temporal data object | -| [updateTask](#updatetask) | Update a task | -| [updateUser](#updateuser) | Update an existing user | -| [updateWatchlist](#updatewatchlist) | Update watchlist | -| [updateWidget](#updatewidget) | Updates a widget | -| [uploadEngineResult](#uploadengineresult) | Upload and store an engine result | -| [upsertSchemaDraft](#upsertschemadraft) | Update a structured data registry schema | -| [userLogin](#userlogin) | Login as a user | -| [userLogout](#userlogout) | Logout user and invalidate user token | -| [validateEngineOutput](#validateengineoutput) | Validates if an engine output conforms to the engine output guidelines | -| [validateToken](#validatetoken) | Validate a user token | -| [verifyJWT](#verifyjwt) | Verify JWT token | - -#### addLibraryDataset - -Add recordings to a dataset library - -_**Arguments**_
- -`input:` - -```graphql -addLibraryDataset(input: AddLibraryDataset!): LibraryDataset -``` - -*See also:*
[AddLibraryDataset](https://api.veritone.com/v3/graphqldocs/addlibrarydataset.doc.html), [LibraryDataset](https://api.veritone.com/v3/graphqldocs/librarydataset.doc.html) - ---- -#### addTasksToJobs - -Arguments -input: - -_**Arguments**_
- -```graphql -addTasksToJobs(input: AddTasksToJobs): AddTasksToJobsResponse -``` - -*See also:*
[AddTasksToJobs](https://api.veritone.com/v3/graphqldocs/addtaskstojobs.doc.html), [AddTasksToJobsResponse](https://api.veritone.com/v3/graphqldocs/addtaskstojobsresponse.doc.html) - ---- -#### addToEngineBlacklist - -Arguments -toAdd: - -_**Arguments**_
- -```graphql -addToEngineBlacklist(toAdd: SetEngineBlacklist!): EngineBlacklist -``` - -*See also:*
[SetEngineBlacklist](https://api.veritone.com/v3/graphqldocs/setengineblacklist.doc.html), [EngineBlacklist](https://api.veritone.com/v3/graphqldocs/engineblacklist.doc.html) - ---- -#### addToEngineWhitelist - -Arguments -toAdd: - -_**Arguments**_
- -```graphql -addToEngineWhitelist(toAdd: SetEngineWhitelist!): EngineWhitelist -``` - -*See also:*
[SetEngineWhitelist](https://api.veritone.com/v3/graphqldocs/setenginewhitelist.doc.html), [EngineWhitelist](https://api.veritone.com/v3/graphqldocs/enginewhitelist.doc.html) - ---- -#### applicationWorkflow - -Apply an application workflow step, such as "submit" or "approve" - -_**Arguments**_
- -`input:` Fields required to apply a application workflow step - -```graphql -applicationWorkflow(input: ApplicationWorkflow): Application -``` - -*See also:*
[ApplicationWorkflow](https://api.veritone.com/v3/graphqldocs/applicationworkflow.doc.html), [Application](https://api.veritone.com/v3/graphqldocs/application.doc.html) - ---- -#### bulkDeleteContextMenuExtensions - -Bulk delete context meu extensions. - -_**Arguments**_
- -`input:` - -```graphql -bulkDeleteContextMenuExtensions( - input: BulkDeleteContextMenuExtensions -): ContextMenuExtensionList -``` - -*See also:*
[BulkDeleteContextMenuExtensions](https://api.veritone.com/v3/graphqldocs/bulkdeletecontextmenuextensions.doc.html), [ContextMenuExtensionList](https://api.veritone.com/v3/graphqldocs/contextmenuextensionlist.doc.html) - ---- -#### bulkUpdateWatchlist - -Apply bulk updates to watchlists. -This mutation is currently available only to Veritone operations. - -_**Arguments**_
- -`filter:` A filter indicating which watchlists should be updated. - -At least one filter condition must be provided. - -Only watchlists for the user's organization will be updated. - -`input:` Fields used to update a watchlist. - -```graphql -bulkUpdateWatchlist( - filter: BulkUpdateWatchlistFilter!, - input: BulkUpdateWatchlist -): WatchlistList -``` - -*See also:*
[BulkUpdateWatchlistFilter](https://api.veritone.com/v3/graphqldocs/bulkupdatewatchlistfilter.doc.html), [BulkUpdateWatchlist](https://api.veritone.com/v3/graphqldocs/bulkupdatewatchlist.doc.html), [WatchlistList](https://api.veritone.com/v3/graphqldocs/watchlistlist.doc.html) - ---- -#### cancelJob - -Cancel a job. This action effectively deletes the job, -although a records of job and task execution remains in -Veritone's database. - -_**Arguments**_
- -`id:` Supply the ID of the job to delete. - -```graphql -cancelJob(id: ID!): DeletePayload -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [DeletePayload](https://api.veritone.com/v3/graphqldocs/deletepayload.doc.html) - ---- -#### changePassword - -Change the current authenticated user's password - -_**Arguments**_
- -`input:` Fields needed to change password - -```graphql -changePassword(input: ChangePassword!): User -``` - -*See also:*
[ChangePassword](https://api.veritone.com/v3/graphqldocs/changepassword.doc.html), [User](https://api.veritone.com/v3/graphqldocs/user.doc.html) - ---- -#### cleanupTDO - -Delete partial information from a temporal data object. -Use the delete options to control exactly which data is deleted. -The default is to delete objects from storage and the search index, -while leaving TDO-level metadata and task engine results intact. -To permanently delete the TDO, use delete TDO. - -_**Arguments**_
- -`id:` Supply the ID of the TDO to clean up. - -`options:` Supply a list of cleanup options. See [TDOCleanupOption](https://api.veritone.com/v3/graphqldocs/tdocleanupoption.doc.html) -for details. If not provided, the server will use default settings. - -```graphql -cleanupTDO(id: ID!, options: [TDOCleanupOption!]): DeletePayload -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [TDOCleanupOption](https://api.veritone.com/v3/graphqldocs/tdocleanupoption.doc.html), [DeletePayload](https://api.veritone.com/v3/graphqldocs/deletepayload.doc.html) - ---- -#### createApplication - -Create a new application. An application must -go through a sequence of workflow steps before -it is available in production. See the VDA documentation -for details. - -_**Arguments**_
- -`input:` Fields needed to create a new custom application. - -```graphql -createApplication(input: CreateApplication): Application -``` - -*See also:*
[CreateApplication](https://api.veritone.com/v3/graphqldocs/createapplication.doc.html), [Application](https://api.veritone.com/v3/graphqldocs/application.doc.html) - ---- -#### createAsset - -Create a media asset. Optionally, upload content using -multipart form POST. - -_**Arguments**_
- -`input:` Fields needed to create an asset. - -```graphql -createAsset(input: CreateAsset!): Asset -``` - -*See also:*
[CreateAsset](https://api.veritone.com/v3/graphqldocs/createasset.doc.html), [Asset](https://api.veritone.com/v3/graphqldocs/asset.doc.html) - ---- -#### createCognitiveSearch - -_**Arguments**_
- -`input:` - -```graphql -createCognitiveSearch(input: CreateCognitiveSearch): CognitiveSearch -``` - -*See also:*
[CreateCognitiveSearch](https://api.veritone.com/v3/graphqldocs/createcognitivesearch.doc.html), [CognitiveSearch](https://api.veritone.com/v3/graphqldocs/cognitivesearch.doc.html) - ---- -#### createCollection - -Create (ingest) a structured data object - -_**Arguments**_
- -`input:` Fields required to create new collection - -```graphql -createCollection(input: CreateCollection): Collection -``` - -*See also:*
[CreateCollection](https://api.veritone.com/v3/graphqldocs/createcollection.doc.html), [Collection](https://api.veritone.com/v3/graphqldocs/collection.doc.html) - ---- -#### createCollectionMention - -Add a mention to a collection - -_**Arguments**_
- -`input:` Fields needed to add a mention to a collection - -```graphql -createCollectionMention(input: CollectionMentionInput): CollectionMention -``` - -*See also:*
[CollectionMentionInput](https://api.veritone.com/v3/graphqldocs/collectionmentioninput.doc.html), [CollectionMention](https://api.veritone.com/v3/graphqldocs/collectionmention.doc.html) - ---- -#### createCreative - -Create a creative - -_**Arguments**_
- -`input:` - -```graphql -createCreative(input: CreateCreative!): Creative! -``` - -*See also:*
[CreateCreative](https://api.veritone.com/v3/graphqldocs/createcreative.doc.html), [Creative](https://api.veritone.com/v3/graphqldocs/creative.doc.html) - ---- -#### createDataRegistry - -Create a structured data registry schema metadata. - -_**Arguments**_
- -`input:` - -```graphql -createDataRegistry(input: CreateDataRegistry!): DataRegistry -``` - -*See also:*
[CreateDataRegistry](https://api.veritone.com/v3/graphqldocs/createdataregistry.doc.html), [DataRegistry](https://api.veritone.com/v3/graphqldocs/dataregistry.doc.html) - ---- -#### createEngine - -Create a new engine. The engine will need to go -through a sequence of workflow steps before -use in production. See VDA documentation for details. - -_**Arguments**_
- -`input:` Fields needed to create a new engine - -```graphql -createEngine(input: CreateEngine): Engine -``` - -*See also:*
[CreateEngine](https://api.veritone.com/v3/graphqldocs/createengine.doc.html), [Engine](https://api.veritone.com/v3/graphqldocs/engine.doc.html) - ---- -#### createEngineBuild - -Create an engine build. - -_**Arguments**_
- -`input:` Fields needed to create an engine build. - -```graphql -createEngineBuild(input: CreateBuild!): Build -``` - -*See also:*
[CreateBuild](https://api.veritone.com/v3/graphqldocs/createbuild.doc.html), [Build](https://api.veritone.com/v3/graphqldocs/build.doc.html) - ---- -#### createEntity - -Create a new entity. - -_**Arguments**_
- -`input:` Fields required to create a new entity. - -```graphql -createEntity(input: CreateEntity!): Entity -``` - -*See also:*
[CreateEntity](https://api.veritone.com/v3/graphqldocs/createentity.doc.html), [Entity](https://api.veritone.com/v3/graphqldocs/entity.doc.html) - ---- -#### createEntityIdentifier - -Create an entity identifier. -This mutation accepts file uploads. To use this mutation and upload a file, -send a multipart form POST containing two parameters: `query`, with the -GraphQL query, and `file` containing the file itself. -For more information see the documentation at -https://veritone-developer.atlassian.net/wiki/spaces/DOC/pages/13893791/GraphQL. - -_**Arguments**_
- -`input:` Fields needed to create an entity identifier. - -```graphql -createEntityIdentifier(input: CreateEntityIdentifier!): EntityIdentifier -``` - -*See also:*
[CreateEntityIdentifier](https://api.veritone.com/v3/graphqldocs/createentityidentifier.doc.html), [EntityIdentifier](https://api.veritone.com/v3/graphqldocs/entityidentifier.doc.html) - ---- -#### createEntityIdentifierType - -Create an entity identifier type, such as "face" or "image". -Entity identifier types are typically created or modified -only by Veritone engineering. Most libraries and -entities will use existing entity identifier types. - -_**Arguments**_
- -`input:` Fields required to create an entity identifier type. - -```graphql -createEntityIdentifierType( - input: CreateEntityIdentifierType! -): EntityIdentifierType -``` - -*See also:*
[CreateEntityIdentifierType](https://api.veritone.com/v3/graphqldocs/createentityidentifiertype.doc.html), [EntityIdentifierType](https://api.veritone.com/v3/graphqldocs/entityidentifiertype.doc.html) - ---- -#### createEvent - -Create a new event - -_**Arguments**_
- -`input:` - -```graphql -createEvent(input: CreateEvent!): Event! -``` - -*See also:*
[CreateEvent](https://api.veritone.com/v3/graphqldocs/createevent.doc.html), [Event](https://api.veritone.com/v3/graphqldocs/event.doc.html) - ---- -#### createExportRequest - -Create an export request. The requested TDO data, possibly including -TDO media and engine results, will be exported offline. - -_**Arguments**_
- -`input:` Input data required to create the export request - -```graphql -createExportRequest(input: CreateExportRequest!): ExportRequest! -``` -*See also:*
[CreateExportRequest](https://api.veritone.com/v3/graphqldocs/createexportrequest.doc.html), [ExportRequest](https://api.veritone.com/v3/graphqldocs/exportrequest.doc.html) - ---- -#### createFolder - -Create a new folder - -_**Arguments**_
- -`input:` Fields needed to create a new folder. - -```graphql -createFolder(input: CreateFolder): Folder -``` - -*See also:*
[CreateFolder](https://api.veritone.com/v3/graphqldocs/createfolder.doc.html), [Folder](https://api.veritone.com/v3/graphqldocs/folder.doc.html) - ---- -#### createFolderContentTempate - -Create new content template into a folder - -_**Arguments**_
- -`input:` - -```graphql -createFolderContentTempate( - input: CreateFolderContentTempate! -): FolderContentTemplate! -``` - -*See also:*
[CreateFolderContentTempate](https://api.veritone.com/v3/graphqldocs/createfoldercontenttempate.doc.html), [FolderContentTemplate](https://api.veritone.com/v3/graphqldocs/foldercontenttemplate.doc.html) - ---- -#### createIngestionConfiguration - -Create an ingestion configuration - -_**Arguments**_
- -`input:` - -```graphql -createIngestionConfiguration( - input: CreateIngestionConfiguration -): IngestionConfiguration -``` - -*See also:*
[CreateIngestionConfiguration](https://api.veritone.com/v3/graphqldocs/createingestionconfiguration.doc.html), [IngestionConfiguration](https://api.veritone.com/v3/graphqldocs/ingestionconfiguration.doc.html) - ---- -#### createJob - -Create a job - -_**Arguments**_
- -`input:` Fields required to create a job. - -```graphql -createJob(input: CreateJob): Job -``` - -*See also:*
[CreateJob](https://api.veritone.com/v3/graphqldocs/createjob.doc.html), [Job](https://api.veritone.com/v3/graphqldocs/job.doc.html) - ---- -#### createLibrary - -Create a new library. -Once the library is created, the client can add -entities and entity identifiers. Note that the -library type determines what types of entity identifiers -can be used within the library. - -_**Arguments**_
- -`input:` Fields needed to create a new library. - -```graphql -createLibrary(input: CreateLibrary!): Library -``` - -*See also:*
[CreateLibrary](https://api.veritone.com/v3/graphqldocs/createlibrary.doc.html), [Library](https://api.veritone.com/v3/graphqldocs/library.doc.html) - ---- -#### createLibraryConfiguration - -Create Dataset Library Configuration - -_**Arguments**_
- -`input:` Fields required to create library configuration - -```graphql -createLibraryConfiguration( - input: CreateLibraryConfiguration! -): LibraryConfiguration -``` - -*See also:*
[CreateLibraryConfiguration](https://api.veritone.com/v3/graphqldocs/createlibraryconfiguration.doc.html), [LibraryConfiguration](https://api.veritone.com/v3/graphqldocs/libraryconfiguration.doc.html) - ---- -#### createLibraryEngineModel - -Create a library engine model. - -_**Arguments**_
- -`input:` Fields required to create a library engine model. - -```graphql -createLibraryEngineModel( - input: CreateLibraryEngineModel! -): LibraryEngineModel -``` - -*See also:*
[CreateLibraryEngineModel](https://api.veritone.com/v3/graphqldocs/createlibraryenginemodel.doc.html), [LibraryEngineModel](https://api.veritone.com/v3/graphqldocs/libraryenginemodel.doc.html) - ---- -#### createLibraryType - -Create a library type, such as "ad" or "people". -Entity identifier types are typically created or modified -only by Veritone engineering. Most libraries -will use existing entity identifier types. - -_**Arguments**_
- -`input:` Fields needed to create a new library type. - -```graphql -createLibraryType(input: CreateLibraryType!): LibraryType -``` - -*See also:*
[CreateLibraryType](https://api.veritone.com/v3/graphqldocs/createlibrarytype.doc.html), [LibraryType](https://api.veritone.com/v3/graphqldocs/librarytype.doc.html) - ---- -#### createMediaShare - -Create Media Share. Returning the url of the share - -_**Arguments**_
- -`input:` - -```graphql -createMediaShare(input: CreateMediaShare!): CreatedMediaShare! -``` - -*See also:*
[CreateMediaShare](https://api.veritone.com/v3/graphqldocs/createmediashare.doc.html), [CreatedMediaShare](https://api.veritone.com/v3/graphqldocs/createdmediashare.doc.html) - ---- -#### createMention - -Create a mention object - -_**Arguments**_
- -`input:` - -```graphql -createMention(input: CreateMention!): Mention -``` - -*See also:*
[CreateMention](https://api.veritone.com/v3/graphqldocs/createmention.doc.html), [Mention](https://api.veritone.com/v3/graphqldocs/mention.doc.html) - ---- -#### createMentionComment - -Create a mention comment - -_**Arguments**_
- -`input:` Fields needed to create a mention comment - -```graphql -createMentionComment(input: CreateMentionComment): MentionComment -``` - -*See also:*
[CreateMentionComment](https://api.veritone.com/v3/graphqldocs/creatementioncomment.doc.html), [MentionComment](https://api.veritone.com/v3/graphqldocs/mentioncomment.doc.html) - ---- -#### createMentionExportRequest - -Create a mention export request. The requested mentionFilters including -The mention export file csv will be exported offline. - -_**Arguments**_
- -`input:` Input data required to create the export request - -```graphql -createMentionExportRequest( - input: CreateMentionExportRequest! -): ExportRequest! -``` - -*See also:*
[CreateMentionExportRequest](https://api.veritone.com/v3/graphqldocs/creatementionexportrequest.doc.html), [ExportRequest](https://api.veritone.com/v3/graphqldocs/exportrequest.doc.html) - ---- -#### createMentionRating - -Create a mention rating - -_**Arguments**_
- -`input:` Fields needed to create a mention rating - -```graphql -createMentionRating(input: CreateMentionRating): MentionRating -``` - -*See also:*
[CreateMentionRating](https://api.veritone.com/v3/graphqldocs/creatementionrating.doc.html), [MentionRating](https://api.veritone.com/v3/graphqldocs/mentionrating.doc.html) - ---- -#### createMentions - -Create Mention in bulk. The input should be an array of createMentions - -_**Arguments**_
- -`input:` - -```graphql -createMentions(input: CreateMentions!): MentionList -``` - -*See also:*
[CreateMentions](https://api.veritone.com/v3/graphqldocs/creatementions.doc.html), [MentionList](https://api.veritone.com/v3/graphqldocs/mentionlist.doc.html) - ---- -#### createOrganization - -Create a new organization. - -_**Arguments**_
- -`input:` Fields needed to create an organization. - -```graphql -createOrganization(input: CreateOrganization!): Organization -``` - -*See also:*
[CreateOrganization](https://api.veritone.com/v3/graphqldocs/createorganization.doc.html), [Organization](https://api.veritone.com/v3/graphqldocs/organization.doc.html) - ---- -#### createPasswordResetRequest - -Create a password reset request. This mutation is used on behalf -of a user who needs to reset their password. It operates only on -the currently authenicated user (based on the authentication token provided). - -_**Arguments**_
- -`input:` - -```graphql -createPasswordResetRequest( - input: CreatePasswordResetRequest -): CreatePasswordResetRequestPayload -``` - -*See also:*
[CreatePasswordResetRequest](https://api.veritone.com/v3/graphqldocs/createpasswordresetrequest.doc.html), [CreatePasswordResetRequestPayload](https://api.veritone.com/v3/graphqldocs/createpasswordresetrequestpayload.doc.html) - ---- -#### createPasswordUpdateRequest - -Force a user to update password on next login. -This mutation is used by administrators. - -_**Arguments**_
- -`input:` Fields needed to create a password update request - -```graphql -createPasswordUpdateRequest( - input: CreatePasswordUpdateRequest -): User -``` - -*See also:*
[CreatePasswordUpdateRequest](https://api.veritone.com/v3/graphqldocs/createpasswordupdaterequest.doc.html), [User](https://api.veritone.com/v3/graphqldocs/user.doc.html) - ---- -#### createProcessTemplate - -Create a processTemplate in CMS - -_**Arguments**_
- -`input:` - -```graphql -createProcessTemplate(input: CreateProcessTemplate!): ProcessTemplate! -``` - -*See also:*
[CreateProcessTemplate](https://api.veritone.com/v3/graphqldocs/createprocesstemplate.doc.html), [ProcessTemplate](https://api.veritone.com/v3/graphqldocs/processtemplate.doc.html) - ---- -#### createRootFolders - -Create root folder for an organization - -_**Arguments**_
- -`rootFolderType:` The type of root folder to create - -```graphql -createRootFolders(rootFolderType: RootFolderType): [Folder] -``` - -*See also:*
[RootFolderType](https://api.veritone.com/v3/graphqldocs/rootfoldertype.doc.html), [Folder](https://api.veritone.com/v3/graphqldocs/folder.doc.html) - ---- -#### createSavedSearch - -Create a new Saved Search - -_**Arguments**_
- -`input:` - -```graphql -createSavedSearch(input: CreateSavedSearch!): SavedSearch! -``` - -*See also:*
[CreateSavedSearch](https://api.veritone.com/v3/graphqldocs/createsavedsearch.doc.html), [SavedSearch](https://api.veritone.com/v3/graphqldocs/savedsearch.doc.html) - ---- -#### createStructuredData - -Create (ingest) a structured data object - -_**Arguments**_
- -`input:` - -```graphql -createStructuredData(input: CreateStructuredData!): StructuredData -``` - -*See also:*
[CreateStructuredData](https://api.veritone.com/v3/graphqldocs/createstructureddata.doc.html), [StructuredData](https://api.veritone.com/v3/graphqldocs/structureddata.doc.html) - ---- -#### createSubscription - -_**Arguments**_
- -`input:` - -```graphql -createSubscription(input: CreateSubscription!): Subscription -``` - -*See also:*
[CreateSubscription](https://api.veritone.com/v3/graphqldocs/createsubscription.doc.html), [Subscription](https://api.veritone.com/v3/graphqldocs/subscription.doc.html) - ---- -#### createTDO - -Create a new temporal data object - -_**Arguments**_
- -`input:` Fields required to create a TDO - -```graphql -createTDO(input: CreateTDO): TemporalDataObject -``` - -*See also:*
[CreateTDO](https://api.veritone.com/v3/graphqldocs/createtdo.doc.html), [TemporalDataObject](https://api.veritone.com/v3/graphqldocs/temporaldataobject.doc.html) - ---- -#### createTDOWithAsset - -Create a TDO and an asset with a single call - -_**Arguments**_
- -`input:` Input fields necessary to create the TDO and asset - -```graphql -createTDOWithAsset(input: CreateTDOWithAsset): TemporalDataObject -``` - -*See also:*
[CreateTDOWithAsset](https://api.veritone.com/v3/graphqldocs/createtdowithasset.doc.html), [TemporalDataObject](https://api.veritone.com/v3/graphqldocs/temporaldataobject.doc.html) - ---- -#### createTaskLog - -Create a task log by using -multipart form POST. - -_**Arguments**_
- -`input:` Fields needed to create a task log. - -```graphql -createTaskLog(input: CreateTaskLog!): TaskLog -``` - -*See also:*
[CreateTaskLog](https://api.veritone.com/v3/graphqldocs/createtasklog.doc.html), [TaskLog](https://api.veritone.com/v3/graphqldocs/tasklog.doc.html) - ---- -#### createTriggers - -Create trigger for events or types. - -_**Arguments**_
- -`input:` - -```graphql -createTriggers(input: CreateTriggers!): [Trigger] -``` - -*See also:*
[CreateTriggers](https://api.veritone.com/v3/graphqldocs/createtriggers.doc.html), [Trigger](https://api.veritone.com/v3/graphqldocs/trigger.doc.html) - ---- -#### createUser - -Create a new user within an organization. - -_**Arguments**_
- -`input:` Fields needed to create a user. - -```graphql -createUser(input: CreateUser): User -``` - -*See also:*
[CreateUser](https://api.veritone.com/v3/graphqldocs/createuser.doc.html), [User](https://api.veritone.com/v3/graphqldocs/user.doc.html) - ---- -#### createWatchlist - -_**Arguments**_
- -`input:` - -```graphql -createWatchlist(input: CreateWatchlist!): Watchlist -``` - -*See also:*
[CreateWatchlist](https://api.veritone.com/v3/graphqldocs/createwatchlist.doc.html), [Watchlist](https://api.veritone.com/v3/graphqldocs/watchlist.doc.html) - ---- -#### createWidget - -Creates a widget associated with a collection - -_**Arguments**_
- -`input:` Fields needed to create a new widget - -```graphql -createWidget(input: CreateWidget): Widget -``` - -*See also:*
[CreateWidget](https://api.veritone.com/v3/graphqldocs/createwidget.doc.html), [Widget](https://api.veritone.com/v3/graphqldocs/widget.doc.html) - ---- -#### deleteApplication - -Delete an application - -_**Arguments**_
- -`id:` Supply the ID of the application to delete. - -```graphql -deleteApplication(id: ID!): DeletePayload -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [DeletePayload](https://api.veritone.com/v3/graphqldocs/deletepayload.doc.html) - ---- -#### deleteAsset - -Delete an asset - -_**Arguments**_
- -`id:` Provide the ID of the asset to delete. - -```graphql -deleteAsset(id: ID!): DeletePayload -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [DeletePayload](https://api.veritone.com/v3/graphqldocs/deletepayload.doc.html) - ---- -#### deleteCognitiveSearch - -_**Arguments**_
- -`id:` - -```graphql -deleteCognitiveSearch(id: ID!): DeletePayload -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [DeletePayload](https://api.veritone.com/v3/graphqldocs/deletepayload.doc.html) - ---- -#### deleteCollection - -Delete Collection - -_**Arguments**_
- -`folderId:` @deprecated(`reason:` "folderId has been renamed to id. -Use id.") - -`id:` Supply the ID of the folder or collection to delete - -```graphql -deleteCollection(folderId: ID, id: ID): DeletePayload -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [DeletePayload](https://api.veritone.com/v3/graphqldocs/deletepayload.doc.html) - ---- -#### deleteCollectionMention - -Remove a mention from a collection - -_**Arguments**_
- -`input:` Fields needed to delete a mention from a collection - -```graphql -deleteCollectionMention(input: CollectionMentionInput): CollectionMention -``` - -*See also:*
[CollectionMentionInput](https://api.veritone.com/v3/graphqldocs/collectionmentioninput.doc.html), [CollectionMention](https://api.veritone.com/v3/graphqldocs/collectionmention.doc.html) - ---- -#### deleteCreative - -Delete a creative - -_**Arguments**_
- -`id:` - -```graphql -deleteCreative(id: ID!): DeletePayload! -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [DeletePayload](https://api.veritone.com/v3/graphqldocs/deletepayload.doc.html) - ---- -#### deleteEngine - -Delete an engine - -_**Arguments**_
- -`id:` Provide the ID of the engine to delete - -```graphql -deleteEngine(id: ID!): DeletePayload -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [DeletePayload](https://api.veritone.com/v3/graphqldocs/deletepayload.doc.html) - ---- -#### deleteEngineBuild - -Delete an engine build - -_**Arguments**_
- -`input:` Fields needed to delete an engine build. - -```graphql -deleteEngineBuild(input: DeleteBuild!): DeletePayload -``` - -*See also:*
[DeleteBuild](https://api.veritone.com/v3/graphqldocs/deletebuild.doc.html), [DeletePayload](https://api.veritone.com/v3/graphqldocs/deletepayload.doc.html) - ---- -#### deleteEntity - -Delete an entity. This mutation will also delete all associated -entity identifiers and associated objects. - -_**Arguments**_
- -`id:` Supply the ID of the entity to delete. - -```graphql -deleteEntity(id: ID!): DeletePayload -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [DeletePayload](https://api.veritone.com/v3/graphqldocs/deletepayload.doc.html) - ---- -#### deleteEntityIdentifier - -Delete an entity identifier - -_**Arguments**_
- -`id:` Supply the ID of the entity identifier to delete. - -```graphql -deleteEntityIdentifier(id: ID!): DeletePayload -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [DeletePayload](https://api.veritone.com/v3/graphqldocs/deletepayload.doc.html) - ---- -#### deleteFolder - -Delete a folder - -_**Arguments**_
- -`input:` Fields needed to delete a folder - -```graphql -deleteFolder(input: DeleteFolder): DeletePayload -``` - -*See also:*
[DeleteFolder](https://api.veritone.com/v3/graphqldocs/deletefolder.doc.html), [DeletePayload](https://api.veritone.com/v3/graphqldocs/deletepayload.doc.html) - ---- -#### deleteFolderContentTempate - -Delete existing folder content template by folderContentTemplateId - -_**Arguments**_
- -`id:` Folder Content Template Id - -```graphql -deleteFolderContentTempate(id: ID!): DeletePayload! -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [DeletePayload](https://api.veritone.com/v3/graphqldocs/deletepayload.doc.html) - ---- -#### deleteFromEngineBlacklist - -_**Arguments**_
- -`toDelete:` - -```graphql -deleteFromEngineBlacklist( - toDelete: SetEngineBlacklist! -): EngineBlacklist -``` - -*See also:*
[SetEngineBlacklist](https://api.veritone.com/v3/graphqldocs/setengineblacklist.doc.html), [EngineBlacklist](https://api.veritone.com/v3/graphqldocs/engineblacklist.doc.html) - ---- -#### deleteFromEngineWhitelist - -_**Arguments**_
- -`toDelete:` - -```graphql -deleteFromEngineWhitelist( - toDelete: SetEngineBlacklist! -): EngineWhitelist -``` - -*See also:*
[SetEngineBlacklist](https://api.veritone.com/v3/graphqldocs/setengineblacklist.doc.html), [EngineWhitelist](https://api.veritone.com/v3/graphqldocs/enginewhitelist.doc.html) - ---- -#### deleteIngestionConfiguration - -Delete an ingestion configuration - -_**Arguments**_
- -`id:` ID of the ingestion configuration to delete - -```graphql -deleteIngestionConfiguration(id: ID!): DeletePayload -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [DeletePayload](https://api.veritone.com/v3/graphqldocs/deletepayload.doc.html) - ---- -#### deleteLibrary - -Delete a library. This mutation will also delete all entities, -entity identifiers, library engine models, and associated objects. - -_**Arguments**_
- -`id:` Provide the ID of the library to delete. - -```graphql -deleteLibrary(id: ID!): DeletePayload -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [DeletePayload](https://api.veritone.com/v3/graphqldocs/deletepayload.doc.html) - ---- -#### deleteLibraryConfiguration - -Delete Dataset Library Configuration - -_**Arguments**_
- -`id:` Supply configuration ID to delete. - -```graphql -deleteLibraryConfiguration(id: ID!): DeletePayload -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [DeletePayload](https://api.veritone.com/v3/graphqldocs/deletepayload.doc.html) - ---- -#### deleteLibraryDataset - -Remove recordings from a dataset library - -_**Arguments**_
- -`input:` - -```graphql -deleteLibraryDataset(input: DeleteLibraryDataset!): DeleteLibraryDatasetPayload -``` - -*See also:*
[DeleteLibraryDataset](https://api.veritone.com/v3/graphqldocs/deletelibrarydataset.doc.html), [DeleteLibraryDatasetPayload](https://api.veritone.com/v3/graphqldocs/deletelibrarydatasetpayload.doc.html) - ---- -#### deleteLibraryEngineModel - -Delete a library engine model - -_**Arguments**_
- -`id:` Supply the ID of the library engine model to delete. - -```graphql -deleteLibraryEngineModel(id: ID!): DeletePayload -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [DeletePayload](https://api.veritone.com/v3/graphqldocs/deletepayload.doc.html) - ---- -#### deleteMentionComment - -Delete a mention comment - -_**Arguments**_
- -`input:` Fields needed to delete a mention comment - -```graphql -deleteMentionComment(input: DeleteMentionComment): DeletePayload -``` - -*See also:*
[DeleteMentionComment](https://api.veritone.com/v3/graphqldocs/deletementioncomment.doc.html), [DeletePayload](https://api.veritone.com/v3/graphqldocs/deletepayload.doc.html) - ---- -#### deleteMentionRating - -Delete a mention rating - -_**Arguments**_
- -`input:` Fields needed to delete a mention rating. - -```graphql -deleteMentionRating(input: DeleteMentionRating): DeletePayload -``` - -*See also:*
[DeleteMentionRating](https://api.veritone.com/v3/graphqldocs/deletementionrating.doc.html), [DeletePayload](https://api.veritone.com/v3/graphqldocs/deletepayload.doc.html) - ---- -#### deleteSavedSearch - -Delete a saved search - -_**Arguments**_
- -`id:` - -```graphql -deleteSavedSearch(id: ID!): DeletePayload! -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [DeletePayload](https://api.veritone.com/v3/graphqldocs/deletepayload.doc.html) - ---- -#### deleteStructuredData - -Delete a structured data object - -_**Arguments**_
- -`input:` - -```graphql -deleteStructuredData(input: DeleteStructuredData!): DeletePayload -``` - -*See also:*
[DeleteStructuredData](https://api.veritone.com/v3/graphqldocs/deletestructureddata.doc.html), [DeletePayload](https://api.veritone.com/v3/graphqldocs/deletepayload.doc.html) - ---- -#### deleteSubscription - -_**Arguments**_
- -`id:` - -```graphql -deleteSubscription(id: ID!): DeletePayload -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [DeletePayload](https://api.veritone.com/v3/graphqldocs/deletepayload.doc.html) - ---- -#### deleteTDO - -Delete a temporal data object. The TDO metadata, its assets and -all storage objects, and search index data are deleted. -Engine results stored in related task objects are not. -cleanupTDO can be used to selectively delete certain data on the TDO. - -_**Arguments**_
- -`id:` Supply the ID of the TDO to delete - -```graphql -deleteTDO(id: ID!): DeletePayload -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [DeletePayload](https://api.veritone.com/v3/graphqldocs/deletepayload.doc.html) - ---- -#### deleteTrigger - -Delete a registed trigger by ID. - -_**Arguments**_
- -`id:` - -```graphql -deleteTrigger(id: ID!): DeletePayload -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [DeletePayload](https://api.veritone.com/v3/graphqldocs/deletepayload.doc.html) - ---- -#### deleteUser - -Delete a user - -_**Arguments**_
- -`id:` Supply the ID of the user to delete. - -```graphql -deleteUser(id: ID!): DeletePayload -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [DeletePayload](https://api.veritone.com/v3/graphqldocs/deletepayload.doc.html) - ---- -#### deleteWatchlist - -_**Arguments**_
- -`id:` - -```graphql -deleteWatchlist(id: ID!): DeletePayload -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [DeletePayload](https://api.veritone.com/v3/graphqldocs/deletepayload.doc.html) - ---- -#### emitEvent - -Emit an event - -_**Arguments**_
- -`input:` - -```graphql -emitEvent(input: EmitEvent!): EmitEventResponse! -``` - -*See also:*
[EmitEvent](https://api.veritone.com/v3/graphqldocs/emitevent.doc.html), [EmitEventResponse](https://api.veritone.com/v3/graphqldocs/emiteventresponse.doc.html) - ---- -#### emitSystemEvent - -Emit a system-level emit. This mutation is used only by -Veritone platform components. - -_**Arguments**_
- -`input:` Data required to create the event - -```graphql -emitSystemEvent(input: EmitSystemEvent!): SystemEventInfo! -} -``` - -*See also:*
[EmitSystemEvent](https://api.veritone.com/v3/graphqldocs/emitsystemevent.doc.html), [SystemEventInfo](https://api.veritone.com/v3/graphqldocs/systemeventinfo.doc.html) - ---- -#### engineWorkflow - -Apply an application workflow step, such as "submit" or "approve" - -_**Arguments**_
- -`input:` Fields required to apply a engine workflow step - -```graphql -engineWorkflow(input: EngineWorkflow): Engine -``` - -*See also:*
[EngineWorkflow](https://api.veritone.com/v3/graphqldocs/engineworkflow.doc.html), [Engine](https://api.veritone.com/v3/graphqldocs/engine.doc.html) - ---- -#### fileTemporalDataObject - -File a TemporalDataObject in a folder. A given TemporalDataObject can -be filed in any number of folders, or none. Filing causes the TemporalDataObject -and its assets to be visible within the folder. - -_**Arguments**_
- -`input:` The fields needed to file a TemporalDataObject in a -folder - -```graphql -fileTemporalDataObject(input: FileTemporalDataObject!): TemporalDataObject -``` - -*See also:*
[FileTemporalDataObject](https://api.veritone.com/v3/graphqldocs/filetemporaldataobject.doc.html), [TemporalDataObject](https://api.veritone.com/v3/graphqldocs/temporaldataobject.doc.html) - ---- -#### fileWatchlist - -_**Arguments**_
- -`input:` - -```graphql -fileWatchlist(input: FileWatchlist!): Watchlist -``` - -*See also:*
[FileWatchlist](https://api.veritone.com/v3/graphqldocs/filewatchlist.doc.html), [Watchlist](https://api.veritone.com/v3/graphqldocs/watchlist.doc.html) - ---- -#### getCurrentUserPasswordToken - -Get password token info for current user - -_**Arguments**_
- -`input:` - -```graphql -getCurrentUserPasswordToken( - input: GetCurrentUserPasswordToken! -): PasswordTokenInfo! -``` - -*See also:*
[GetCurrentUserPasswordToken](https://api.veritone.com/v3/graphqldocs/getcurrentuserpasswordtoken.doc.html), [PasswordTokenInfo](https://api.veritone.com/v3/graphqldocs/passwordtokeninfo.doc.html) - ---- -#### getEngineJWT - -JWT tokens with a more limited scoped token to specific -resources to the recording, task, and job -and also has no organization association. - -_**Arguments**_
- -`input:` - -```graphql -getEngineJWT(input: getEngineJWT!): JWTTokenInfo! -``` - -*See also:*
[getEngineJWT](https://api.veritone.com/v3/graphqldocs/getenginejwt.doc.html), [JWTTokenInfo](https://api.veritone.com/v3/graphqldocs/jwttokeninfo.doc.html) - ---- -#### moveFolder - -Move a folder from one parent folder to another. - -_**Arguments**_
- -`input:` Fields needed to move a folder - -```graphql -moveFolder(input: MoveFolder): Folder -``` - -*See also:*
[MoveFolder](https://api.veritone.com/v3/graphqldocs/movefolder.doc.html), [Folder](https://api.veritone.com/v3/graphqldocs/folder.doc.html) - ---- -#### moveTemporalDataObject - -Moves a TemporalDataObject from one parent folder to another. -Any other folders the TemporalDataObject is filed in are unaffected. - -_**Arguments**_
- -`input:` Fields need to move a TemporalDataObject - -```graphql -moveTemporalDataObject(input: MoveTemporalDataObject!): TemporalDataObject -``` - -*See also:*
[MoveTemporalDataObject](https://api.veritone.com/v3/graphqldocs/movetemporaldataobject.doc.html), [TemporalDataObject](https://api.veritone.com/v3/graphqldocs/temporaldataobject.doc.html) - ---- -#### pollTask - -Poll a task - -_**Arguments**_
- -`input:` Fields required to poll a task. - -```graphql -pollTask(input: PollTask): Task -``` - -*See also:*
[PollTask](https://api.veritone.com/v3/graphqldocs/polltask.doc.html), [Task](https://api.veritone.com/v3/graphqldocs/task.doc.html) - ---- -#### publishLibrary - -Publish a new version of a library. -Increments library version by one and trains compatible engines. - -_**Arguments**_
- -`id:` ID of the library to publish - -```graphql -publishLibrary(id: ID!): Library -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [Library](https://api.veritone.com/v3/graphqldocs/library.doc.html) - ---- -#### refreshToken - -Refresh a user token, returning a fresh token so that the client -can continue to authenticate to the API. - -_**Arguments**_
- -`token:` - -```graphql -refreshToken(token: String!): LoginInfo -``` - -*See also:*
[String](https://api.veritone.com/v3/graphqldocs/string.doc.html), [LoginInfo](https://api.veritone.com/v3/graphqldocs/logininfo.doc.html) - ---- -#### replaceSavedSearch - -Mark existing saved search profile as deleted -Create new saved search profile - -_**Arguments**_
- -`input:` - -```graphql -replaceSavedSearch(input: ReplaceSavedSearch!): SavedSearch! -``` - -*See also:*
[ReplaceSavedSearch](https://api.veritone.com/v3/graphqldocs/replacesavedsearch.doc.html), [SavedSearch](https://api.veritone.com/v3/graphqldocs/savedsearch.doc.html) - ---- -#### requestClone - -Start a clone job. A clone creates a new TDO -that links back to an existing TDO's assets -instead of creating new ones and is used -primarily to handle sample media. - -_**Arguments**_
- -`input:` Fields needed to request a new clone job. - -```graphql -requestClone(input: RequestClone): CloneRequest -``` - -*See also:*
[RequestClone](https://api.veritone.com/v3/graphqldocs/requestclone.doc.html), [CloneRequest](https://api.veritone.com/v3/graphqldocs/clonerequest.doc.html) - ---- -#### retryJob - -Retry a job. This action applies only to jobs -that are in a failure state. The task sequence -for the job will be restarted in its original -configuration. - -_**Arguments**_
- -`id:` Supply the ID of the job to retry. - -```graphql -retryJob(id: ID!): Job -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [Job](https://api.veritone.com/v3/graphqldocs/job.doc.html) - ---- -#### sendEmail - -Send a basic email. Mutation returns true for a success message. - -_**Arguments**_
- -`input:` - -```graphql -sendEmail(input: SendEmail!): Boolean! -``` - -*See also:*
[SendEmail](https://api.veritone.com/v3/graphqldocs/sendemail.doc.html), [Boolean](https://api.veritone.com/v3/graphqldocs/boolean.doc.html) - ---- -#### setWorkflowRuntimeStorageData - -Create or Update Workflow data. - -_**Arguments**_
- -`workflowRuntimeId:` - -`input:` - -```graphql -setWorkflowRuntimeStorageData( - workflowRuntimeId: ID!, - input: CreateWorkflowRuntimeStorageData! -): WorkflowRuntimeStorageData! -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [CreateWorkflowRuntimeStorageData](https://api.veritone.com/v3/graphqldocs/createworkflowruntimestoragedata.doc.html), [WorkflowRuntimeStorageData](https://api.veritone.com/v3/graphqldocs/workflowruntimestoragedata.doc.html) - ---- -#### shareCollection - -Share a collection, allowing other organizations to view the data -it contains. - -_**Arguments**_
- -`input:` Fields needed to share a collection - -```graphql -shareCollection(input: ShareCollection): Share -``` - -*See also:*
[ShareCollection](https://api.veritone.com/v3/graphqldocs/sharecollection.doc.html), [Share](https://api.veritone.com/v3/graphqldocs/share.doc.html) - ---- -#### shareFolder - -Share a folder with other organizations - -_**Arguments**_
- -`input:` - -```graphql -shareFolder(input: ShareFolderInput): Folder -``` - -*See also:*
[ShareFolderInput](https://api.veritone.com/v3/graphqldocs/sharefolderinput.doc.html), [Folder](https://api.veritone.com/v3/graphqldocs/folder.doc.html) - ---- -#### shareMention - -Share mention - -_**Arguments**_
- -`input:` - -```graphql -shareMention(input: ShareMention): Share -``` - -*See also:*
[ShareMention](https://api.veritone.com/v3/graphqldocs/sharemention.doc.html), [Share](https://api.veritone.com/v3/graphqldocs/share.doc.html) - ---- -#### shareMentionFromCollection - -Share a mention from a collection - -_**Arguments**_
- -`input:` Fields needed to share a mention - -```graphql -shareMentionFromCollection( - input: ShareMentionFromCollection -): Share -``` - -*See also:*
[ShareMentionFromCollection](https://api.veritone.com/v3/graphqldocs/sharementionfromcollection.doc.html), [Share](https://api.veritone.com/v3/graphqldocs/share.doc.html) - ---- -#### shareMentionInBulk - -Share mentions in bulk - -_**Arguments**_
- -`input:` - -```graphql -shareMentionInBulk(input: ShareMentionInBulk): [Share] -``` - -*See also:*
[ShareMentionInBulk](https://api.veritone.com/v3/graphqldocs/sharementioninbulk.doc.html), [Share](https://api.veritone.com/v3/graphqldocs/share.doc.html) - ---- -#### startWorkflowRuntime - -Start a Veritone Workflow instance - -_**Arguments**_
- -`workflowRuntimeId:` - -`orgId:` - -`generateAuthToken:` - -```graphql -startWorkflowRuntime( - workflowRuntimeId: ID!, - orgId: ID!, - generateAuthToken: Boolean -): WorkflowRuntimeResponse! -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [Boolean](https://api.veritone.com/v3/graphqldocs/boolean.doc.html), [WorkflowRuntimeResponse](https://api.veritone.com/v3/graphqldocs/workflowruntimeresponse.doc.html) - ---- -#### stopWorkflowRuntime - -Shut down Veritone Workflow instance - -_**Arguments**_
- -`workflowRuntimeId:` - -```graphql -stopWorkflowRuntime(workflowRuntimeId: ID!): WorkflowRuntimeResponse! -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [WorkflowRuntimeResponse](https://api.veritone.com/v3/graphqldocs/workflowruntimeresponse.doc.html) - ---- -#### subscribeEvent - -Subscribe to an event - -_**Arguments**_
- -`input:` - -```graphql -subscribeEvent(input: SubscribeEvent!): ID! -``` - -*See also:*
[SubscribeEvent](https://api.veritone.com/v3/graphqldocs/subscribeevent.doc.html), [ID](https://api.veritone.com/v3/graphqldocs/id.doc.html) - ---- -#### unfileTemporalDataObject - -Unfile a TemporalDataObject from a folder. This causes the TemporalDataObject -and its assets to disappear from the folder, but does not otherwise affect -either the TDO or the folder and does not change access controls. - -_**Arguments**_
- -`input:` The fields needed to file a TemporalDataObject in a -folder - -```graphql -unfileTemporalDataObject( - input: UnfileTemporalDataObject! -): TemporalDataObject -``` - -*See also:*
[UnfileTemporalDataObject](https://api.veritone.com/v3/graphqldocs/unfiletemporaldataobject.doc.html), [TemporalDataObject](https://api.veritone.com/v3/graphqldocs/temporaldataobject.doc.html) - ---- -#### unfileWatchlist - -_**Arguments**_
- -`input:` - -```graphql -unfileWatchlist(input: UnfileWatchlist!): Watchlist -``` - -*See also:*
[UnfileWatchlist](https://api.veritone.com/v3/graphqldocs/unfilewatchlist.doc.html), [Watchlist](https://api.veritone.com/v3/graphqldocs/watchlist.doc.html) - ---- -#### unsubscribeEvent - -Unsubscribe to an event - -_**Arguments**_
- -`id:` - -```graphql -unsubscribeEvent(id: ID!): UnsubscribeEvent! -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [UnsubscribeEvent](https://api.veritone.com/v3/graphqldocs/unsubscribeevent.doc.html) - ---- -#### updateApplication - -Update a custom application. Applications are subject to -specific workflows. The current application state determines -what updates can be made to it. See VDA documentation for details. - -_**Arguments**_
- -`input:` Fields required to update a custom application. - -```graphql -updateApplication(input: UpdateApplication): Application -``` - -*See also:*
[UpdateApplication](https://api.veritone.com/v3/graphqldocs/updateapplication.doc.html), [Application](https://api.veritone.com/v3/graphqldocs/application.doc.html) - ---- -#### updateAsset - -Update an asset - -_**Arguments**_
- -`input:` Fields needed to update an asset. - -```graphql -updateAsset(input: UpdateAsset!): Asset -``` - -*See also:*
[UpdateAsset](https://api.veritone.com/v3/graphqldocs/updateasset.doc.html), [Asset](https://api.veritone.com/v3/graphqldocs/asset.doc.html) - ---- -#### updateCognitiveSearch - -_**Arguments**_
- -`input:'` - -```graphql -updateCognitiveSearch(input: UpdateCognitiveSearch): CognitiveSearch -``` - -*See also:*
[UpdateCognitiveSearch](https://api.veritone.com/v3/graphqldocs/updatecognitivesearch.doc.html), [CognitiveSearch](https://api.veritone.com/v3/graphqldocs/cognitivesearch.doc.html) - ---- -#### updateCollection - -Update a collection - -_**Arguments**_
- -`input:` Fields needed to update a collection - -```graphql -updateCollection(input: UpdateCollection): Collection -``` - -*See also:*
[UpdateCollection](https://api.veritone.com/v3/graphqldocs/updatecollection.doc.html), [Collection](https://api.veritone.com/v3/graphqldocs/collection.doc.html) - ---- -#### updateCreative - -Update a creative - -_**Arguments**_
- -`input:` - -```graphql -updateCreative(input: UpdateCreative!): Creative! -``` - -*See also:*
[UpdateCreative](https://api.veritone.com/v3/graphqldocs/updatecreative.doc.html), [Creative](https://api.veritone.com/v3/graphqldocs/creative.doc.html) - ---- -#### updateCurrentUser - -Update the current authenticated user - -_**Arguments**_
- -`input:` - -```graphql -updateCurrentUser(input: UpdateCurrentUser!): User! -``` - -*See also:*
[UpdateCurrentUser](https://api.veritone.com/v3/graphqldocs/updatecurrentuser.doc.html), [User](https://api.veritone.com/v3/graphqldocs/user.doc.html) - ---- -#### updateDataRegistry - -Update a structured data registry schema metadata. - -_**Arguments**_
- -`input:` - -```graphql -updateDataRegistry(input: UpdateDataRegistry!): DataRegistry -``` - -*See also:*
[UpdateDataRegistry](https://api.veritone.com/v3/graphqldocs/updatedataregistry.doc.html), [DataRegistry](https://api.veritone.com/v3/graphqldocs/dataregistry.doc.html) - ---- -#### updateEngine - -Update an engine. Engines are subject to specific -workflow steps. An engine's state determines what -updates can be made to it. See VDA documentation for -details. - -_**Arguments**_
- -`input:` Fields needed to update an engine - -```graphql -updateEngine(input: UpdateEngine): Engine -``` - -*See also:*
[UpdateEngine](https://api.veritone.com/v3/graphqldocs/updateengine.doc.html), [Engine](https://api.veritone.com/v3/graphqldocs/engine.doc.html) - ---- -#### updateEngineBuild - -Update an engine build. Engine builds are subject to -specific workflow steps. A build's state determines what -updates can be made to it. See VDA documentation for details. - -_**Arguments**_
- -`input:` Fields needed to update an engine build. - -```graphql -updateEngineBuild(input: UpdateBuild!): Build -``` - -*See also:*
[UpdateBuild](https://api.veritone.com/v3/graphqldocs/updatebuild.doc.html), [Build](https://api.veritone.com/v3/graphqldocs/build.doc.html) - ---- -#### updateEntity - -Update an entity. - -_**Arguments**_
- -`input:` Fields required to update an entity. - -```graphql -updateEntity(input: UpdateEntity!): Entity -``` - -*See also:*
[UpdateEntity](https://api.veritone.com/v3/graphqldocs/updateentity.doc.html), [Entity](https://api.veritone.com/v3/graphqldocs/entity.doc.html) - ---- -#### updateEntityIdentifier - -_**Arguments**_
- -`input:` Fields required to update an entity identifier.` - -```graphql -updateEntityIdentifier(input: UpdateEntityIdentifier!): EntityIdentifier -``` - -*See also:*
[UpdateEntityIdentifier](https://api.veritone.com/v3/graphqldocs/updateentityidentifier.doc.html), [EntityIdentifier](https://api.veritone.com/v3/graphqldocs/entityidentifier.doc.html) - ---- -#### updateEntityIdentifierType - -Update an entity identifier type. - -_**Arguments**_
- -`input:` Fields required to update an entity identifier type. - -```graphql -updateEntityIdentifierType( - input: UpdateEntityIdentifierType! -): EntityIdentifierType -``` - -*See also:*
[UpdateEntityIdentifierType](https://api.veritone.com/v3/graphqldocs/updateentityidentifiertype.doc.html), [EntityIdentifierType](https://api.veritone.com/v3/graphqldocs/entityidentifiertype.doc.html) - ---- -#### updateEvent - -Update an event - -_**Arguments**_
- -`input:` - -```graphql -updateEvent(input: UpdateEvent!): Event! -``` - -*See also:*
[UpdateEvent](https://api.veritone.com/v3/graphqldocs/updateevent.doc.html), [Event](https://api.veritone.com/v3/graphqldocs/event.doc.html) - ---- -#### updateExportRequest - -Update an export request - -_**Arguments**_
- -`input:` Input data required to update an export request - -```graphql -updateExportRequest(input: UpdateExportRequest!): ExportRequest! -``` - -*See also:*
[UpdateExportRequest](https://api.veritone.com/v3/graphqldocs/updateexportrequest.doc.html), [ExportRequest](https://api.veritone.com/v3/graphqldocs/exportrequest.doc.html) - ---- -#### updateFolder - -Update an existing folder - -_**Arguments**_
- -`input:` Fields needed to update a folder. - -```graphql -updateFolder(input: UpdateFolder): Folder -``` - -*See also:*
[UpdateFolder](https://api.veritone.com/v3/graphqldocs/updatefolder.doc.html), [Folder](https://api.veritone.com/v3/graphqldocs/folder.doc.html) - ---- -#### updateFolderContentTempate - -Update existing content template by folderContentTemplateId - -_**Arguments**_
- -`input:` - -```graphql -updateFolderContentTempate( - input: UpdateFolderContentTempate! -): FolderContentTemplate! -``` - -*See also:*
[UpdateFolderContentTempate](https://api.veritone.com/v3/graphqldocs/updatefoldercontenttempate.doc.html), [FolderContentTemplate](https://api.veritone.com/v3/graphqldocs/foldercontenttemplate.doc.html) - ---- -#### updateIngestionConfiguration - -Update an ingestion configuration - -_**Arguments**_
- -`input:` - -```graphql -updateIngestionConfiguration( - input: UpdateIngestionConfiguration -): IngestionConfiguration -``` - -*See also:*
[UpdateIngestionConfiguration](https://api.veritone.com/v3/graphqldocs/updateingestionconfiguration.doc.html), [IngestionConfiguration](https://api.veritone.com/v3/graphqldocs/ingestionconfiguration.doc.html) - ---- -#### updateJobs - -_**Arguments**_
- -`input:` - -```graphql -updateJobs(input: UpdateJobs!): JobList -``` - -*See also:*
[UpdateJobs](https://api.veritone.com/v3/graphqldocs/updatejobs.doc.html), [JobList](https://api.veritone.com/v3/graphqldocs/joblist.doc.html) - ---- -#### updateLibrary - -Update an existing library. - -_**Arguments**_
- -`input:` Fields needed to update a library - -```graphql -updateLibrary(input: UpdateLibrary!): Library -``` - -*See also:*
[UpdateLibrary](https://api.veritone.com/v3/graphqldocs/updatelibrary.doc.html), [Library](https://api.veritone.com/v3/graphqldocs/library.doc.html) - ---- -#### updateLibraryConfiguration - -Update Dataset Library Configuration - -_**Arguments**_
- -`input:` Fields required to create library configuration - -```graphql -updateLibraryConfiguration( - input: UpdateLibraryConfiguration! -): LibraryConfiguration -``` - -*See also:*
[UpdateLibraryConfiguration](https://api.veritone.com/v3/graphqldocs/updatelibraryconfiguration.doc.html), [LibraryConfiguration](https://api.veritone.com/v3/graphqldocs/libraryconfiguration.doc.html) - ---- -#### updateLibraryEngineModel - -Update a library engine model - -_**Arguments**_
- -`input:` Fields required to update a library engine model - -```graphql -updateLibraryEngineModel( - input: UpdateLibraryEngineModel! -): LibraryEngineModel -``` - -*See also:*
[UpdateLibraryEngineModel](https://api.veritone.com/v3/graphqldocs/updatelibraryenginemodel.doc.html), [LibraryEngineModel](https://api.veritone.com/v3/graphqldocs/libraryenginemodel.doc.html) - ---- -#### updateLibraryType - -Update a library type. - -_**Arguments**_
- -`input:` Fields needed to update a library type. - -```graphql -updateLibraryType(input: UpdateLibraryType!): LibraryType -``` - -*See also:*
[UpdateLibraryType](https://api.veritone.com/v3/graphqldocs/updatelibrarytype.doc.html), [LibraryType](https://api.veritone.com/v3/graphqldocs/librarytype.doc.html) - ---- -#### updateMention - -Update a mention object - -_**Arguments**_
- -`input:` - -```graphql -updateMention(input: UpdateMention!): Mention -``` - -*See also:*
[UpdateMention](https://api.veritone.com/v3/graphqldocs/updatemention.doc.html), [Mention](https://api.veritone.com/v3/graphqldocs/mention.doc.html) - ---- -#### updateMentionComment - -Update a mention comment - -_**Arguments**_
- -`input:` Fields needed to update a mention comment - -```graphql -updateMentionComment(input: UpdateMentionComment): MentionComment -``` - -*See also:*
[UpdateMentionComment](https://api.veritone.com/v3/graphqldocs/updatementioncomment.doc.html), [MentionComment](https://api.veritone.com/v3/graphqldocs/mentioncomment.doc.html) - ---- -#### updateMentionExportRequest - -Update status or assetURI of a mentionExportRequest -Often use when the file export was completed or downloaded - -_**Arguments**_
- -`input:` - -```graphql -updateMentionExportRequest( - input: UpdateMentionExportRequest! -): ExportRequest! -``` - -*See also:*
[UpdateMentionExportRequest](https://api.veritone.com/v3/graphqldocs/updatementionexportrequest.doc.html), [ExportRequest](https://api.veritone.com/v3/graphqldocs/exportrequest.doc.html) - ---- -#### updateMentionRating - -Update a mention rating - -_**Arguments**_
- -`input:` Fields needed to update a mention rating - -```graphql -updateMentionRating(input: UpdateMentionRating): MentionRating -``` - -*See also:*
[UpdateMentionRating](https://api.veritone.com/v3/graphqldocs/updatementionrating.doc.html), [MentionRating](https://api.veritone.com/v3/graphqldocs/mentionrating.doc.html) - ---- -#### updateMentions - -Update a set of mentions - -_**Arguments**_
- -`input:` - -```graphql -updateMentions(input: UpdateMentions!): [Mention] -``` - -*See also:*
[UpdateMentions](https://api.veritone.com/v3/graphqldocs/updatementions.doc.html), [Mention](https://api.veritone.com/v3/graphqldocs/mention.doc.html) - ---- -#### updateOrganization - -Update an organization - -_**Arguments**_
- -`input:` Fields required to update an organization. - -```graphql -updateOrganization(input: UpdateOrganization!): Organization -``` - -*See also:*
[UpdateOrganization](https://api.veritone.com/v3/graphqldocs/updateorganization.doc.html), [Organization](https://api.veritone.com/v3/graphqldocs/organization.doc.html) - ---- -#### updateProcessTemplate - -Update a processTemplate by ID in CMS - -_**Arguments**_
- -`input:` - -```graphql -updateProcessTemplate(input: UpdateProcessTemplate!): ProcessTemplate! -``` - -*See also:*
[UpdateProcessTemplate](https://api.veritone.com/v3/graphqldocs/updateprocesstemplate.doc.html), [ProcessTemplate](https://api.veritone.com/v3/graphqldocs/processtemplate.doc.html) - ---- -#### updateSchemaState - -_**Arguments**_
- -`input:` - -```graphql -updateSchemaState(input: UpdateSchemaState!): Schema -``` - -*See also:*
[UpdateSchemaState](https://api.veritone.com/v3/graphqldocs/updateschemastate.doc.html), [Schema](https://api.veritone.com/v3/graphqldocs/schema.doc.html) - ---- -#### updateSubscription - -_**Arguments**_
- -`input:` - -```graphql -updateSubscription(input: UpdateSubscription!): Subscription -``` - -*See also:*
[UpdateSubscription](https://api.veritone.com/v3/graphqldocs/updatesubscription.doc.html), [Subscription](https://api.veritone.com/v3/graphqldocs/subscription.doc.html) - ---- -#### updateTDO - -Update a temporal data object - -_**Arguments**_
- -`input:` Fields required to update a TDO - -```graphql -updateTDO(input: UpdateTDO): TemporalDataObject -``` - -*See also:*
[UpdateTDO](https://api.veritone.com/v3/graphqldocs/updatetdo.doc.html), [TemporalDataObject](https://api.veritone.com/v3/graphqldocs/temporaldataobject.doc.html) - ---- -#### updateTask - -Update a task - -_**Arguments**_
- -`input:` Fields required to update a task. - -```graphql -updateTask(input: UpdateTask): Task -``` - -*See also:*
[UpdateTask](https://api.veritone.com/v3/graphqldocs/updatetask.doc.html), [Task](https://api.veritone.com/v3/graphqldocs/task.doc.html) - ---- -#### updateUser - -Update an existing user - -_**Arguments**_
- -`input:` Fields needed to update a user - -```graphql -updateUser(input: UpdateUser): User -``` - -*See also:*
[UpdateUser](https://api.veritone.com/v3/graphqldocs/updateuser.doc.html), [User](https://api.veritone.com/v3/graphqldocs/user.doc.html) - ---- -#### updateWatchlist - -_**Arguments**_
- -`input:` - -```graphql -updateWatchlist(input: UpdateWatchlist!): Watchlist -``` - -*See also:*
[UpdateWatchlist](https://api.veritone.com/v3/graphqldocs/updatewatchlist.doc.html), [Watchlist](https://api.veritone.com/v3/graphqldocs/watchlist.doc.html) - ---- -#### updateWidget - -Updates a widget - -_**Arguments**_
- -`input:` Fields needed to update a widget - -```graphql -updateWidget(input: UpdateWidget): Widget -``` - -*See also:*
[UpdateWidget](https://api.veritone.com/v3/graphqldocs/updatewidget.doc.html), [Widget](https://api.veritone.com/v3/graphqldocs/widget.doc.html) - ---- -#### uploadEngineResult - -Upload and store an engine result. The result will be stored as an -asset associated with the target TemporalDataObject and the -task will be updated accordingly. -Use a multipart form POST to all this mutation. - -_**Arguments**_
- -`input:` Fields needed to upload and store an engine result - -```graphql -uploadEngineResult(input: UploadEngineResult!): Asset -``` - -*See also:*
[UploadEngineResult](https://api.veritone.com/v3/graphqldocs/uploadengineresult.doc.html), [Asset](https://api.veritone.com/v3/graphqldocs/asset.doc.html) - ---- -#### upsertSchemaDraft - -Update a structured data registry schema. - -_**Arguments**_
- -`input:` - -```graphql -upsertSchemaDraft(input: UpsertSchemaDraft!): Schema -``` - -*See also:*
[UpsertSchemaDraft](https://api.veritone.com/v3/graphqldocs/upsertschemadraft.doc.html), [Schema](https://api.veritone.com/v3/graphqldocs/schema.doc.html) - ---- -#### userLogin - -Login as a user. This mutation does not require an existing authentication -context (via `Authorization` header with bearer token, cookie, etc.). -Instead, the client supplies credentials to this mutation, which then -authenticates the user and sets up the authentication context. -The returned tokens can be used to authenticate future requests. - -_**Arguments**_
- -`input:` Fields needed to log in - -```graphql -userLogin(input: UserLogin): LoginInfo -``` - -*See also:*
[UserLogin](https://api.veritone.com/v3/graphqldocs/userlogin.doc.html), [LoginInfo](https://api.veritone.com/v3/graphqldocs/logininfo.doc.html) - ---- -#### userLogout - -Logout user and invalidate user token - -_**Arguments**_
- -`token:` User token that should be invalidated - -```graphql -userLogout(token: String!): Boolean -``` - -*See also:*
[String](https://api.veritone.com/v3/graphqldocs/string.doc.html), [Boolean](https://api.veritone.com/v3/graphqldocs/boolean.doc.html) - ---- -#### validateEngineOutput - -Validates if an engine output conforms to the engine output guidelines - -_**Arguments**_
- -`input:` - -```graphql -validateEngineOutput(input: JSONData!): Boolean! -``` - -*See also:*
[JSONData](https://api.veritone.com/v3/graphqldocs/jsondata.doc.html), [Boolean](https://api.veritone.com/v3/graphqldocs/boolean.doc.html) - ---- -#### validateToken - -Validate a user token. This mutation is used by services to determine -if the token provided by a given client is valid. - -_**Arguments**_
- -`token:` - -```graphql -validateToken(token: String!): LoginInfo -``` - -*See also:*
[String](https://api.veritone.com/v3/graphqldocs/string.doc.html), [LoginInfo](https://api.veritone.com/v3/graphqldocs/logininfo.doc.html) - ---- -#### verifyJWT - -Verify JWT token - -_**Arguments**_
- -`jwtToken:` - -```graphql -verifyJWT(jwtToken: String!): VerifyJWTPayload -``` - -*See also:*
[String](https://api.veritone.com/v3/graphqldocs/string.doc.html), [VerifyJWTPayload](https://api.veritone.com/v3/graphqldocs/verifyjwtpayload.doc.html) diff --git a/docs/apis/reference/query/README.md b/docs/apis/reference/query/README.md deleted file mode 100644 index a8d8b3b945..0000000000 --- a/docs/apis/reference/query/README.md +++ /dev/null @@ -1,1927 +0,0 @@ -# Query Methods - -The table below gives a quick summary of GraphQL [query](https://api.veritone.com/v3/graphqldocs/query.doc.html) methods, alphabetized by name. - -Click any name to see the complete method signature, and other info. - -| Method name | Short Description | -| -- | -- | -| [application](#application) | Retrieve a single application | -| [applications](#applications) | Retrieve applications | -| [asset](#asset) | Retrieve a single Asset | -| [auditLog](#auditlog) | Examine entries from the audit log | -| [cloneRequests](#clonerequests) | Retrieve clone job entries | -| [cognitiveSearch](#cognitivesearch) | Cognitive search | -| [collection](#collection) | Collection | -| [collections](#collections) | Collections | -| [creative](#creative) | Get creative by id with current organizationId | -| [dataRegistries](#dataregistries) | Data registries | -| [dataRegistry](#dataregistry) | Data registry | -| [engine](#engine) | Retrieve a single engine by ID | -| [engineBuild](#enginebuild) | Engine build | -| [engineCategories](#enginecategories) | Retrieve engine categories | -| [engineCategory](#enginecategory) | Retrieve a specific engine category | -| [engineResults](#engineresults) | Retrieves engine results by TDO and engine ID or by job ID | -| [engines](#engines) | Retrieve engines | -| [entities](#entities) | Retrieve a list of entities across libraries | -| [entity](#entity) | Retrieve a specific entity | -| [entityIdentifierType](#entityidentifiertype) | Entity identifier type | -| [entityIdentifierTypes](#entityidentifiertypes) | Retrieve entity identifier types | -| [event](#event) | Retrieve a event by id | -| [events](#events) | Retrieve a list of events by application | -| [exportRequest](#exportrequest) | Export request | -| [exportRequests](#exportrequests) | Retrieve a list of export requests | -| [folder](#folder) | Retrieve a single folder | -| [getSignedWritableUrl](#getsignedwritableurl) | Returns a signed writable S3 URL | -| [getSignedWritableUrls](#getsignedwritableurls) | Return writable storage URLs in bulk | -| [graphqlServiceInfo](#graphqlserviceinfo) | Returns information about the GraphQL server, useful for diagnostics | -| [groups](#groups) | Retrieve groups | -| [ingestionConfiguration](#ingestionconfiguration) | Retrieve a single ingestion configuration | -| [ingestionConfigurations](#ingestionconfigurations) | Retrieve ingestion configurations | -| [job](#job) | Retrieve a single job | -| [jobs](#jobs) | Retrieve jobs | -| [libraries](#libraries) | Retrieve libraries and entities | -| [library](#library) | Retrieve a specific library | -| [libraryConfiguration](#libraryconfiguration) | Retrieve library configuration | -| [libraryEngineModel](#libraryenginemodel) | Retrieve a specific library engine model | -| [libraryType](#librarytype) | Retrieve a single library type | -| [libraryTypes](#librarytypes) | Retrieve all library types | -| [me](#me) | Retrieve information for the current logged-in user | -| [mediaShare](#mediashare) | Get the media share by media shareId | -| [mention](#mention) | Retrieve a single mention | -| [mentionStatusOptions](#mentionstatusoptions) | | -| [mentions](#mentions) | Mentions | -| [myRights](#myrights) | | -| [organization](#organization) | Retrieve a single organization | -| [organizations](#organizations) | Retrieve organizations | -| [permissions](#permissions) | Retrieve permissions | -| [processTemplate](#processtemplate) | Get process templates by id | -| [processTemplates](#processtemplates) | Get list process templates by id or current organizationId | -| [rootFolders](#rootfolders) | Retrieve the root folders for an organization | -| [savedSearches](#savedsearches) | Fetch all saved searches that the current user has made -| [schema](#schema) | Schema | -| [schemaProperties](#schemaproperties) | Schema properties | -| [schemas](#schemas) | Retrieve a list of schemas for structured data ingestions | -| [searchMedia](#searchmedia) | Search for media across an index | -| [searchMentions](#searchmentions) | Search for mentions across an index | -| [sharedCollection](#sharedcollection) | Retrieve a shared collection | -| [sharedCollectionHistory](#sharedcollectionhistory) | Retrieve shared collection history records | -| [sharedFolders](#sharedfolders) | Retrieve the shared folders for an organization | -| [sharedMention](#sharedmention) | Retrieve a shared mention | -| [structuredData](#structureddata) | Retrieve a structured data object | -| [structuredDataObject](#structureddataobject) | Retrieve a structured data object | -| [structuredDataObjects](#structureddataobjects) | Retrieve a paginated list of structured data object | -| [subscription](#subscription) | Subscription | -| [task](#task) | Retrieve a single task by ID | -| [temporalDataObject](#temporaldataobject) | Retrieve a single temporal data object | -| [temporalDataObjects](#temporaldataobjects) | Retrieve a list of temporal data objects | -| [timeZones](#timezones) | This query returns information about time zones recognized by this server | -| [tokens](#tokens) | Retrieve user's organization API tokens | -| [trigger](#trigger) | Trigger | -| [triggers](#triggers) | | -| [user](#user) | Retrieve an individual user | -| [users](#users) | Retrieve users | -| [watchlist](#watchlist) | Watchlist | -| [watchlists](#watchlists) | Watchlists | -| [widget](#widget) | Retrieve a single Widget | -| [workflowRuntime](#workflowruntime) | Retrieve Veritone Workflow instance status by ID | -| [workflowRuntimeStorageData](#workflowruntimestoragedata) | Get a specific workflowRuntimeData based on dataKey | - -#### application - -Retrieve a single application - -_**Arguments**_
- -`id:` The application ID - -```graphql -application(id: ID!): Application -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [Application](https://api.veritone.com/v3/graphqldocs/application.doc.html) - ---- -#### applications - -Retrieve applications. These are custom applications integrated into -the Veritone platform using the VDA framework. - -_**Arguments**_
- -`id:` Provide an ID to retrieve a single specific application. - -`status:` Provide a status, such as "draft" or "active" - -`owned:` If true, return only applications owned by the user's organization. - -`offset:` Provide an offset to skip to a certain element in the result, for paging. - -`limit:` Specify maximum number of results to retrieve in this result. Page size. - - -```graphql -applications( - id: ID, - status: ApplicationStatus, - owned: Boolean, - offset: Int, - limit: Int -): ApplicationList -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [ApplicationStatus](https://api.veritone.com/v3/graphqldocs/applicationstatus.doc.html), [Boolean](https://api.veritone.com/v3/graphqldocs/boolean.doc.html), [Int](https://api.veritone.com/v3/graphqldocs/int.doc.html), [ApplicationList](https://api.veritone.com/v3/graphqldocs/applicationlist.doc.html) - ---- -#### asset - -Retrieve a single Asset - -_**Arguments**_
- -`id:` The asset ID - -```graphql -asset(id: ID!): Asset -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [Asset](https://api.veritone.com/v3/graphqldocs/asset.doc.html) - ---- -#### auditLog - -Examine entries from the audit log. All operations that modify data are -written to the audit log. Only entries for the user's own organization -can be queried. -All queries are bracketed by a time window. A default time window is applied -if the `toDateTime` and/or `fromDateTime` parameters are not provided. -The maximum time window length is 30 days. -Only Veritone and organization administrators can use this query. - -_**Arguments**_
- -`toDateTime:` Date/time up to which entries will be returned. In other words, the end of the query time window. Defaults to the current time. - -`fromDateTime:` Date/time from which entries will be returned. In other words, the start of the query time window. Defaults to the `toDateTime` minus 7 days. - -`organizationId:` Organization ID to query records for. This value can only be used by Veritone administrators. Any value provided by user administrators will be ignored. - -`userName:` User name on audit entry. Must be exact match. - -`clientIpAddress:` IP address of the client that generated the audit action. Must be exact match. - -`clientUserAgent:` HTTP user agent of the client that generated the audit action. Must be exact match. - -`eventType:` The event type, such as `Create`, -`Update`, or Delete`. Must be exact match. - -`objectId:` The ID of the object involved in the audit action. The format of this ID varies by object type. Must be exact match. - -`objectType:` The type of the object involved in the audit action, such as -`Watchlist` or -`TemporalDataObject`. Must be exact match. - -`success:` Whether or not the action was successful. - -`id:` The unique ID of an audit log entry. Multiple values can be provided. - -`offset:` Offset into result set, for paging. - -`limit:` Limit on result size, for paging (page size). Audit queries are lightweight so the default of 100 is higher than the default offset used elsewhere in the API. - -`orderBy:` Order information. Default is order by createdDateTime` descending. - - -```graphql -auditLog( - toDateTime: DateTime, - fromDateTime: DateTime, - organizationId: ID, - userName: String, - clientIpAddress: String, - clientUserAgent: String, - eventType: String, - objectId: ID, - objectType: String, - success: Boolean, - id: [ID!], - offset: Int, - limit: Int, - orderBy: [AuditLogOrderBy!] -): AuditLogEntryList! -``` - -*See also:*
[DateTime](https://api.veritone.com/v3/graphqldocs/datetime.doc.html), [ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [String](https://api.veritone.com/v3/graphqldocs/string.doc.html), [Boolean](https://api.veritone.com/v3/graphqldocs/boolean.doc.html), [Int](https://api.veritone.com/v3/graphqldocs/int.doc.html), [AuditLogOrderBy](https://api.veritone.com/v3/graphqldocs/auditlogorderby.doc.html), [AuditLogEntryList](https://api.veritone.com/v3/graphqldocs/auditlogentrylist.doc.html) - ---- -#### cloneRequests - -Retrieve clone job entries - -_**Arguments**_
- -`id:` Provide an ID to retrieve a single specific clone request. - -`applicationId:` Application ID to get clone requests for. Defaults to the user's own application. - -`offset:` - -`limit:` - -```graphql -cloneRequests(id: ID, applicationId: ID, offset: Int, limit: Int): CloneRequestList -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [Int](https://api.veritone.com/v3/graphqldocs/int.doc.html), [CloneRequestList](https://api.veritone.com/v3/graphqldocs/clonerequestlist.doc.html) - ---- -#### cognitiveSearch - -_**Arguments**_
- -```graphql -cognitiveSearch(id: ID!): CognitiveSearch! -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [CognitiveSearch](https://api.veritone.com/v3/graphqldocs/cognitivesearch.doc.html) - ---- -#### collection - -_**Arguments**_
- -`id:` - -```graphql -collection(id: ID!): Collection! -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [Collection](https://api.veritone.com/v3/graphqldocs/collection.doc.html) - ---- -#### collections - -_**Arguments**_
- - id: - name: - mentionId: - offset: - limit: - -```graphql -collections(id: ID, name: String, mentionId: ID, offset: Int, limit: Int): CollectionList! -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [String](https://api.veritone.com/v3/graphqldocs/string.doc.html), [Int](https://api.veritone.com/v3/graphqldocs/int.doc.html), [CollectionList](https://api.veritone.com/v3/graphqldocs/collectionlist.doc.html) - ---- -#### creative - -Get creative by id with current organizationId - -_**Arguments**_
- -`id:` - - -```graphql -creative(id: ID!): Creative! -} -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [Creative](https://api.veritone.com/v3/graphqldocs/creative.doc.html) - ---- -#### dataRegistries - -_**Arguments**_
- - id: - name: - nameMatch: - offset: - limit: - orderBy: - orderDirection: - filterByOwnership: - -```graphql -dataRegistries( - id: ID, - name: String, - nameMatch: StringMatch, - offset: Int, - limit: Int, - orderBy: DataRegistryOrderBy, - orderDirection: OrderDirection, - filterByOwnership: SchemaOwnership -): DataRegistryList -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [String](https://api.veritone.com/v3/graphqldocs/string.doc.html), [StringMatch](https://api.veritone.com/v3/graphqldocs/stringmatch.doc.html), [Int](https://api.veritone.com/v3/graphqldocs/int.doc.html), [DataRegistryOrderBy](https://api.veritone.com/v3/graphqldocs/dataregistryorderby.doc.html), [OrderDirection](https://api.veritone.com/v3/graphqldocs/orderdirection.doc.html), [SchemaOwnership](https://api.veritone.com/v3/graphqldocs/schemaownership.doc.html), [DataRegistryList](https://api.veritone.com/v3/graphqldocs/dataregistrylist.doc.html) - ---- -#### dataRegistry - -_**Arguments**_
- -`id:` - -```graphql -dataRegistry(id: ID!): DataRegistry -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [DataRegistry](https://api.veritone.com/v3/graphqldocs/dataregistry.doc.html) - ---- -#### engine - -Retrieve a single engine by ID - -_**Arguments**_
- -`id:` Provide the engine ID - - -```graphql -engine(id: ID!): Engine -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [Engine](https://api.veritone.com/v3/graphqldocs/engine.doc.html) - ---- -#### engineBuild - -_**Arguments**_
- - id: Provide the build ID - -```graphql -engineBuild(id: ID!): Build -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [Build](https://api.veritone.com/v3/graphqldocs/build.doc.html) - ---- -#### engineCategories - -Retrieve engine categories - -_**Arguments**_
- -`id:` Provide an ID to retrieve a single specific engine category. - -`name:` Provide a name, or part of one, to search by category name - -`type:` Return all categories of an engine type - -`offset:` Specify maximum number of results to retrieve in this result. Page size. - -`limit:` Specify maximum number of results to retrieve in this result. - - -```graphql -engineCategories( - id: ID, - name: String, - type: String, - offset: Int, - limit: Int -): EngineCategoryList -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [String](https://api.veritone.com/v3/graphqldocs/string.doc.html), [Int](https://api.veritone.com/v3/graphqldocs/int.doc.html), [EngineCategoryList](https://api.veritone.com/v3/graphqldocs/enginecategorylist.doc.html) - ---- -#### engineCategory - -Retrieve a specific engine category - -_**Arguments**_
- -`id:` Supply the ID of the engine category to retrieve - - -```graphql -engineCategory(id: ID!): EngineCategory -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [EngineCategory](https://api.veritone.com/v3/graphqldocs/enginecategory.doc.html) - ---- -#### engineResults - -Retrieves engine results by TDO and engine ID or by job ID. - -_**Arguments**_
- -`tdoId:` Provide the ID of the TDO containing engine results to retrieve. If this parameter is used, engineIds or engineCategoryIds must also be set. Results for _only_ the specified TDO will be returned. - -`sourceId:` Provide the ID of the Source containing engine results to retrieve. If this parameter is used, engineIds or engineCategoryIds must also be set. This takes priority over tdoId. - -`engineIds:` Provide one or more engine IDs to retrieve engine results by ID. This parameter is mandatory if tdoId is used, but optional if jobId or engineCategory is used. - -`engineCategoryIds:` Provide one or more category IDs to get all results from that categroy. - -`jobId:` Provide a job ID to retrieve engine results for the job. - -`mentionId:` Provide a mention ID to retrieve engine results for the mention. - -`startOffsetMs:` Start offset ms for the results. - -`stopOffsetMs:` End offset ms for the results. - -`startDate:` Start date for the results. Takes priority over startOffsetMs. - -`stopDate:` End date for the results. Takes priority over stopOffsetMs. - -`ignoreUserEdited:` Whether or not to exclude user edited engine results. Defaults to false. - -`fallbackTdoId:` A TDO ID can be provided for use if the provided sourceId` and/or mentionId` parameters do not resolve to a logical set of TDOs. Depending on parameter settings and available data, results from other TDOs can be included in the response. - - -```graphql -engineResults( - tdoId: ID, - sourceId: ID, - engineIds: [ID!], - engineCategoryIds: [ID!], - jobId: ID, - mentionId: ID, - startOffsetMs: Int, - stopOffsetMs: Int, - startDate: DateTime, - stopDate: DateTime, - ignoreUserEdited: Boolean, - fallbackTdoId: ID -): EngineResultList -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [Int](https://api.veritone.com/v3/graphqldocs/int.doc.html), [DateTime](https://api.veritone.com/v3/graphqldocs/datetime.doc.html), [Boolean](https://api.veritone.com/v3/graphqldocs/boolean.doc.html), [EngineResultList](https://api.veritone.com/v3/graphqldocs/engineresultlist.doc.html) - ---- -#### engines - -Retrieve engines - -_**Arguments**_
- -`id:` Provide an ID to retrieve a single specific engine. - -`ids:` - -`categoryId:` Provide a category ID to filter by engine category. - -`category:` provide a category name or ID to filter by engine category - -`state:` Provide a list of states to filter by engine state. - -`owned:` If true, return only engines owned by the user's organization. - -`libraryRequired:` If true, return only engines that require a library. - -`createsTDO:` If true, return only engines that create their own TDO. If false, return only engines that do not create a TDO. If not set, return either. - -`name:` Provide a name, or part of a name, to search by engine name - -`offset:` Specify maximum number of results to retrieve in this result. Page size. - -`limit:` Specify maximum number of results to retrieve in this result. - -`filter:` Filters for engine attributes - -`orderBy:` Provide a list of EngineSortField to sort by. - - -```graphql -engines( - id: ID, - ids: [ID!], - categoryId: String, - category: String, - state: [EngineState], - owned: Boolean, - libraryRequired: Boolean, - createsTDO: Boolean, - name: String, - offset: Int, - limit: Int, - filter: EngineFilter, - orderBy: [EngineSortField] -): EngineList -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [String](https://api.veritone.com/v3/graphqldocs/string.doc.html), [EngineState](https://api.veritone.com/v3/graphqldocs/enginestate.doc.html), [Boolean](https://api.veritone.com/v3/graphqldocs/boolean.doc.html), [Int](https://api.veritone.com/v3/graphqldocs/int.doc.html), [EngineFilter](https://api.veritone.com/v3/graphqldocs/enginefilter.doc.html), [EngineSortField](https://api.veritone.com/v3/graphqldocs/enginesortfield.doc.html), [EngineList](https://api.veritone.com/v3/graphqldocs/enginelist.doc.html) - ---- -#### entities - -Retrieve a list of entities across libraries - -_**Arguments**_
- -`ids:` Provide a list of entity IDs to retrieve those entities - -`libraryIds:` Provide a list of library IDs to retrieve entities across multiple libraries. - -`isPublished:` - -`identifierTypeId:` - -`name:` - -`offset:` - -`limit:` - -`orderBy:` - -`orderDirection:` - - -```graphql -entities( - ids: [ID!], - libraryIds: [ID!], - isPublished: Boolean, - identifierTypeId: ID, - name: String, - offset: Int, - limit: Int, - orderBy: LibraryEntityOrderBy, - orderDirection: OrderDirection -): EntityList -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [Boolean](https://api.veritone.com/v3/graphqldocs/boolean.doc.html), [String](https://api.veritone.com/v3/graphqldocs/string.doc.html), [Int](https://api.veritone.com/v3/graphqldocs/int.doc.html), [LibraryEntityOrderBy](https://api.veritone.com/v3/graphqldocs/libraryentityorderby.doc.html), [OrderDirection](https://api.veritone.com/v3/graphqldocs/orderdirection.doc.html), [EntityList](https://api.veritone.com/v3/graphqldocs/entitylist.doc.html) - ---- -#### entity - -Retrieve a specific entity - -_**Arguments**_
- -`id:` Provide an entity ID. - - -```graphql -entity(id: ID!): Entity -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [Entity](https://api.veritone.com/v3/graphqldocs/entity.doc.html) - ---- -#### entityIdentifierType - -_**Arguments**_
- -` id:` Provide the entity identifier type ID - -```graphql -entityIdentifierType(id: ID!): EntityIdentifierType -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [EntityIdentifierType](https://api.veritone.com/v3/graphqldocs/entityidentifiertype.doc.html) - ---- -#### entityIdentifierTypes - -Retrieve entity identifier types - -_**Arguments**_
- -`id:` Provide an ID to retrieve a single specific entity identifier type. - -`offset:` Provide an offset to skip to a certain element in the result, for paging. - -`limit:` Specify maximum number of results to retrieve in this result. Page size. - - -```graphql -entityIdentifierTypes(id: ID, offset: Int, limit: Int): EntityIdentifierTypeList -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [Int](https://api.veritone.com/v3/graphqldocs/int.doc.html), [EntityIdentifierTypeList](https://api.veritone.com/v3/graphqldocs/entityidentifiertypelist.doc.html) - ---- -#### event - -Retrieve a event by id - -_**Arguments**_
- -`id:` - - -```graphql -event(id: ID!): Event! -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [Event](https://api.veritone.com/v3/graphqldocs/event.doc.html) - ---- -#### events - -Retrieve a list of events by application - -_**Arguments**_
- -`application:` Provide an application to retrieve all its events. Use 'system' to list all public system events. - -`offset:` Provide an offset to skip to a certain element in the result, for paging. - -`limit:` Specify maximum number of results to retrieve in this result. Page size. - - -```graphql -events(application: String!, offset: Int, limit: Int): EventList! -``` - -*See also:*
[String](https://api.veritone.com/v3/graphqldocs/string.doc.html), [Int](https://api.veritone.com/v3/graphqldocs/int.doc.html), [EventList](https://api.veritone.com/v3/graphqldocs/eventlist.doc.html) - ---- -#### exportRequest - -_**Arguments**_
- -`id:` - -`event:` Provide an event to retrieve export request. Should be -`exportRequest` or `mentionExportRequest`. -Default value is 'exportRequest' - -```graphql -exportRequest(id: ID!, event: ExportRequestEvent): ExportRequest! -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [ExportRequestEvent](https://api.veritone.com/v3/graphqldocs/exportrequestevent.doc.html), [ExportRequest](https://api.veritone.com/v3/graphqldocs/exportrequest.doc.html) - ---- -#### exportRequests - -Retrieve a list of export requests - -_**Arguments**_
- -`id:` Provide an ID to retrieve a single export request - -`offset:` Provide an offset to skip to a certain element in the result, for paging. - -`limit:` Specify maximum number of results to retrieve in this result. Page size. - -`status:` Provide a list of status options to filter by status - -`event:` Provide an event to retrieve export request. Should be exportRequest' or 'mentionExportRequest' Default value is 'exportRequest' - -```graphql -exportRequests( - id: ID, - offset: Int, - limit: Int, - status: [ExportRequestStatus!], - event: ExportRequestEvent -): ExportRequestList! -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [Int](https://api.veritone.com/v3/graphqldocs/int.doc.html), [ExportRequestStatus](https://api.veritone.com/v3/graphqldocs/exportrequeststatus.doc.html), [ExportRequestEvent](https://api.veritone.com/v3/graphqldocs/exportrequestevent.doc.html), [ExportRequestList](https://api.veritone.com/v3/graphqldocs/exportrequestlist.doc.html) - ---- -#### folder - -Retrieve a single folder. Used to navigate the folder tree structure. - -_**Arguments**_
- -`id:` Provide an ID to retrieve a single specific user. - -```graphql -folder(id: ID!): Folder -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [Folder](https://api.veritone.com/v3/graphqldocs/folder.doc.html) - ---- -#### getSignedWritableUrl - -Returns a signed writable S3 URL. A client can then -upload to this URL with an HTTP PUT without providing -any additional authorization (_Note_: The push must be a PUT. -A POST will fail.) - -_**Arguments**_
- -`key:` Optional key of the object to generate a writable URL for. If not provided, a new, unique key will be generated. If a key is provided and resembles a file name with extension delimited by .), a UUID will be inserted into the file name, leaving the extension intact. If a key is provided and does not resemble a file name, a UUID will be appended. - -`type:` Optional type of resource, such as - -`asset`, - -`thumbnail`, or - -`preview` - -`path:` Optional extended path information. If the uploaded content will be contained within a container such as a -`TemporalDataObject` (for -`asset`) or -`Library` for -`entityIdentifier`), the ID of the object should be provided here. - -`organizationId:` Optional organization ID. Normally this value is computed by the server based on the authorization token used for the request. Is is used only by Veritone platform components. - -```graphql -getSignedWritableUrl( - key: String, - type: String, - path: String, - organizationId: ID -): WritableUrlInfo -``` - -*See also:*
[String](https://api.veritone.com/v3/graphqldocs/string.doc.html), [ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [WritableUrlInfo](https://api.veritone.com/v3/graphqldocs/writableurlinfo.doc.html) - ---- -#### getSignedWritableUrls - -Return writable storage URLs in bulk. -A maximum of 1000 can be created in one call. -See `getSignedWritableUrl` for details on usage of the -response contents. - -_**Arguments**_
- -`number:` Number of signed URLs to return - -`type:` Optional type of resource, such as -`asset`, -`thumbnail`, or -`preview` - -`path:` Optional extended path information. If the uploaded content will be contained within a container such as a - -`TemporalDataObject` (for -`asset`) or -`Library` for -`entityIdentifier`), the ID of the object should be provided here. - -`organizationId:` Optional organization ID. Normally this value is computed by the server based on the authorization token used for the request. Is is used only by Veritone platform components. - - -```graphql -getSignedWritableUrls( - number: Int!, - type: String, - path: String, - organizationId: ID -): [WritableUrlInfo!]! -``` - -*See also:*
[Int](https://api.veritone.com/v3/graphqldocs/int.doc.html), [String](https://api.veritone.com/v3/graphqldocs/string.doc.html), [ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [WritableUrlInfo](https://api.veritone.com/v3/graphqldocs/writableurlinfo.doc.html) - ---- -#### graphqlServiceInfo - -Returns information about the GraphQL server, useful -for diagnostics. This data is primarily used by Veritone -development, and some fields may be restricted to Veritone administrators. - -_**Arguments**_
- -```graphql -graphqlServiceInfo: GraphQLServiceInfo -``` - -*See also:*
[GraphQLServiceInfo](https://api.veritone.com/v3/graphqldocs/graphqlserviceinfo.doc.html) - ---- -#### groups - -Retrieve groups - -_**Arguments**_
- -`id:` Provide an ID to retrieve a specific group by ID - -`ids:` Provide IDs to retrieve multiple groups by ID - -`name:` Provide a name, or part of one, to search for groups by name - -`organizationIds:` Provide a list of organization IDs to retrieve groups defined within certain organizations. - -`offset:` Provide an offset to skip to a certain element in the result, for paging. - -`limit:` Specify maximum number of results to retrieve in this result. Page size. - - -```graphql -groups(id: ID, ids: [ID], name: String, organizationIds: [ID], offset: Int, limit: Int): GroupList -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [String](https://api.veritone.com/v3/graphqldocs/string.doc.html), [Int](https://api.veritone.com/v3/graphqldocs/int.doc.html), [GroupList](https://api.veritone.com/v3/graphqldocs/grouplist.doc.html) - ---- -#### ingestionConfiguration - -Retrieve a single ingestion configuration - -_**Arguments**_
- -`id:` The configuration ID - - -```graphql -ingestionConfiguration(id: ID!): IngestionConfiguration -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [IngestionConfiguration](https://api.veritone.com/v3/graphqldocs/ingestionconfiguration.doc.html) - ---- -#### ingestionConfigurations - -Retrieve ingestion configurations - -_**Arguments**_
- -`id:` Supply an ingestion configuration ID to retrieve a single Ingestion - -`offset:` Offset - -`limit:` Limit - -`name:` - -`startDate:` - -`endDate:` - -`sources:` Specify one or more sources to filter by source type - -`applicationId:` Supply an application ID to retrieve configurations only for that application. - -`emailAddress:` Email address configured for ingestion - -```graphql -ingestionConfigurations( - id: ID, - offset: Int, - limit: Int, - name: String, - startDate: DateTime, - endDate: DateTime, - sources: [String!], - applicationId: ID, - emailAddress: String -): IngestionConfigurationList -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [Int](https://api.veritone.com/v3/graphqldocs/int.doc.html), [String](https://api.veritone.com/v3/graphqldocs/string.doc.html), [DateTime](https://api.veritone.com/v3/graphqldocs/datetime.doc.html), [IngestionConfigurationList](https://api.veritone.com/v3/graphqldocs/ingestionconfigurationlist.doc.html) - ---- -#### job - -Retrieve a single job - -_**Arguments**_
- -`id:` the job ID - -```graphql -job(id: ID!): Job -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [Job](https://api.veritone.com/v3/graphqldocs/job.doc.html) - ---- -#### jobs - -Retrieve jobs - -_**Arguments**_
- -`hasTargetTDO:` - -`id:` Provide an ID to retrieve a single specific job. - -`status:` Provide a list of status strings to filter by status - -`applicationStatus:` - -`offset:` Provide an offset to skip to a certain element in the result, for paging. - -`limit:` Specify the maximum number of results to included in this response, or page size. - -`applicationId:` Provide an application ID to filter jobs for a given application. Defaults to the user's own application. - -`targetId:` Provide a target ID to get the set of jobs running against a particular TDO. - -`clusterId:` Provide a cluster ID to get the jobs running on a specific cluster - -`scheduledJobIds:` Provide a list of scheduled job IDs to get jobs associated with the scheduled jobs - -`hasScheduledJobId:` Return only jobs that are (true) or are not false) associated with a scheduled job - -`orderBy:` Provide sort information. The default is to sort by createdDateTime descending. - -`dateTimeFilter:` Filter by date/time field. - -`applicationIds:` Provide list of application IDs to filter jobs. Defaults to the user's own application. - -`engineIds:` Provide a list of engine IDs to filter for jobs that contain tasks for the specified engines. - -`engineCategoryIds:` Provide a list of engine category IDs to filter for jobs that contain tasks for engines in the specific categories. - - -```graphql -jobs( - hasTargetTDO: Boolean, - id: ID, - status: [JobStatusFilter!], - applicationStatus: String, - offset: Int, - limit: Int, - applicationId: ID, - targetId: ID, - clusterId: ID, - scheduledJobIds: [ID!], - hasScheduledJobId: Boolean, - orderBy: [JobSortField!], - dateTimeFilter: [JobDateTimeFilter!], - applicationIds: [ID], - engineIds: [ID!], - engineCategoryIds: [ID!] -): JobList -``` - -*See also:*
[Boolean](https://api.veritone.com/v3/graphqldocs/boolean.doc.html), [ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [JobStatusFilter](https://api.veritone.com/v3/graphqldocs/jobstatusfilter.doc.html), [String](https://api.veritone.com/v3/graphqldocs/string.doc.html), [Int](https://api.veritone.com/v3/graphqldocs/int.doc.html), [JobSortField](https://api.veritone.com/v3/graphqldocs/jobsortfield.doc.html), [JobDateTimeFilter](https://api.veritone.com/v3/graphqldocs/jobdatetimefilter.doc.html), [JobList](https://api.veritone.com/v3/graphqldocs/joblist.doc.html) - ---- -#### libraries - -Retrieve libraries and entities - -_**Arguments**_
- -`id:` Provide an ID to retrieve a single specific library. - -`name:` Provide a name string to search by name. - -`type:` Provide the name or ID of a library to search for libraries that contain that type. - -`entityIdentifierTypeIds:` Provide the id of an entity identifier type to search for libraries that correlate to that type. - -`includeOwnedOnly:` Specify true if only libraries owned by the user's organization should be returned. Otherwise, shared libraries will be included. - -`offset:` Provide an offset to skip to a certain element in the result, for paging. - -`limit:` Specify maximum number of results to retrieve in this result. Page size. - -`orderBy:` Specify a field to order by - -`orderDirection:` Specify the direction to order by - - -```graphql -libraries( - id: ID, - name: String, - type: String, - entityIdentifierTypeIds: [String!], - includeOwnedOnly: Boolean, - offset: Int, - limit: Int, - orderBy: LibraryOrderBy, - orderDirection: OrderDirection -): LibraryList -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [String](https://api.veritone.com/v3/graphqldocs/string.doc.html), [Boolean](https://api.veritone.com/v3/graphqldocs/boolean.doc.html), [Int](https://api.veritone.com/v3/graphqldocs/int.doc.html), [LibraryOrderBy](https://api.veritone.com/v3/graphqldocs/libraryorderby.doc.html), [OrderDirection](https://api.veritone.com/v3/graphqldocs/orderdirection.doc.html), [LibraryList](https://api.veritone.com/v3/graphqldocs/librarylist.doc.html) - ---- -#### library - -Retrieve a specific library - -_**Arguments**_
- -`id:` Provide a library ID. - -```graphql -library(id: ID!): Library -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [Library](https://api.veritone.com/v3/graphqldocs/library.doc.html) - ---- -#### libraryConfiguration - -Retrieve library configuration - -_**Arguments**_
- -`id:` Provide configuration id - - -```graphql -libraryConfiguration(id: ID!): LibraryConfiguration -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [LibraryConfiguration](https://api.veritone.com/v3/graphqldocs/libraryconfiguration.doc.html) - ---- -#### libraryEngineModel - -Retrieve a specific library engine model - -_**Arguments**_
- -`id:` Provide the library engine model ID - -```graphql -libraryEngineModel(id: ID!): LibraryEngineModel -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [LibraryEngineModel](https://api.veritone.com/v3/graphqldocs/libraryenginemodel.doc.html) - ---- -#### libraryType - -Retrieve a single library type - -_**Arguments**_
- -`id:` Provide an ID to retrieve a single specific library type. - -```graphql -libraryType(id: ID): LibraryType -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [LibraryType](https://api.veritone.com/v3/graphqldocs/librarytype.doc.html) - ---- -#### libraryTypes - -Retrieve all library types - -_**Arguments**_
- -`id:` Provide an ID to retrieve a single specific library type. - -`offset:` Provide an offset to skip to a certain element in the result, for paging. - -`limit:` Specify maximum number of results to retrieve in this result. Page size. - -```graphql -libraryTypes(id: ID, offset: Int, limit: Int): LibraryTypeList -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [Int](https://api.veritone.com/v3/graphqldocs/int.doc.html), [LibraryTypeList](https://api.veritone.com/v3/graphqldocs/librarytypelist.doc.html) - ---- -#### me - -Retrieve information for the current logged-in user - -_**Arguments**_
- -```graphql -me: User -``` - -*See also:*
[User](https://api.veritone.com/v3/graphqldocs/user.doc.html) - ---- -#### mediaShare - -Get the media share by media shareId - -_**Arguments**_
- -`id:` - -```graphql -mediaShare(id: ID!): MediaShare! -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [MediaShare](https://api.veritone.com/v3/graphqldocs/mediashare.doc.html) - ---- -#### mention - -Retrieve a single mention - -_**Arguments**_
- -`mentionId:` The mention ID - -`limit:` Comments pagination - limit - -`offset:` Comments pagination - limit - -`userId:` The user who owns the mention. - -```graphql -mention(mentionId: ID!, limit: Int, offset: Int, userId: String): Mention -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [Int](https://api.veritone.com/v3/graphqldocs/int.doc.html), [String](https://api.veritone.com/v3/graphqldocs/string.doc.html), [Mention](https://api.veritone.com/v3/graphqldocs/mention.doc.html) - ---- -#### mentionStatusOptions - -_**Arguments**_
- -```graphql -mentionStatusOptions: [MentionStatus!]! -``` - -*See also:*
[MentionStatus](https://api.veritone.com/v3/graphqldocs/mentionstatus.doc.html) - ---- -#### mentions - - _**Arguments**_
- -`id:` - -`watchlistId:` Get mentions created from the specified watchlist - -`sourceId:` Get mentions associated with the specified source - -`sourceTypeId:` Get mentions associated with sources of the -specified source type - -`tdoId:` Get mentions associated directly with the specific TDO -dateTimeFilter: Specify date/time filters against mention -fields. -Querying for mentions can be expensive. If the query does not -include a filter by `id`, `tdoId`, `sourceId`, `watchlistId`, or -a user-provided `dateTimeFilter`, a default filter of the -past 7 days is applied. - -`orderBy:` Set order information on the query. Multiple fields -are supported. - -`offset:` - -`limit:` - -```graphql -mentions( - id: ID, - watchlistId: ID, - sourceId: ID, - sourceTypeId: ID, - tdoId: ID, - dateTimeFilter: [MentionDateTimeFilter!], - orderBy: [MentionOrderBy!], - offset: Int, - limit: Int -): MentionList -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [MentionDateTimeFilter](https://api.veritone.com/v3/graphqldocs/mentiondatetimefilter.doc.html), [MentionOrderBy](https://api.veritone.com/v3/graphqldocs/mentionorderby.doc.html), [Int](https://api.veritone.com/v3/graphqldocs/int.doc.html), [MentionList](https://api.veritone.com/v3/graphqldocs/mentionlist.doc.html) - ---- -#### myRights - -_**Arguments**_
- -```graphql -myRights: RightsListing -``` - -*See also:*
[RightsListing](https://api.veritone.com/v3/graphqldocs/rightslisting.doc.html) - ---- -#### organization - -Retrieve a single organization - -_**Arguments**_
- -`id:` The organization ID TODO take application ID as well as org ID - -```graphql -organization(id: ID!): Organization -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [Organization](https://api.veritone.com/v3/graphqldocs/organization.doc.html) - ---- -#### organizations - -Retrieve organizations - -_**Arguments**_
- -`id:` Provide an ID to retrieve a single specific organization. - -`offset:` Provide an offset to skip to a certain element in the result, for paging. - -`limit:` Specify maximum number of results to retrieve in this result. Page size. - -`kvpProperty:` Provide a property from the organization kvp to filter the organizaion list. - -`kvpValue:` Provide value to for the kvpFeature filter. If not present the filter becomes `kvpProperty` existence filter. - -```graphql -organizations( - id: ID, - offset: Int, - limit: Int, - kvpProperty: String, - kvpValue: String -): OrganizationList -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [Int](https://api.veritone.com/v3/graphqldocs/int.doc.html), [String](https://api.veritone.com/v3/graphqldocs/string.doc.html), [OrganizationList](https://api.veritone.com/v3/graphqldocs/organizationlist.doc.html) - ---- -#### permissions - -Retrieve permissions - -_**Arguments**_
- -`id:` Provide an ID to retrieve a single specific permission. - -`name:` - -`offset:` Provide an offset to skip to a certain element in the result, for paging. - -`limit:` Specify maximum number of results to retrieve in this result. Page size. - -```graphql -permissions(id: ID, name: String, offset: Int, limit: Int): PermissionList -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [String](https://api.veritone.com/v3/graphqldocs/string.doc.html), [Int](https://api.veritone.com/v3/graphqldocs/int.doc.html), [PermissionList](https://api.veritone.com/v3/graphqldocs/permissionlist.doc.html) - ---- -#### processTemplate - -Get process templates by id - -_**Arguments**_
- -`id:` - -```graphql -processTemplate(id: ID!): ProcessTemplate! -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [ProcessTemplate](https://api.veritone.com/v3/graphqldocs/processtemplate.doc.html) - ---- -#### processTemplates - -Get list process templates by id or current organizationId - -_**Arguments**_
- -`id:` - -`offset:` - -`limit:` - -```graphql -processTemplates(id: ID, offset: Int, limit: Int): ProcessTemplateList! -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [Int](https://api.veritone.com/v3/graphqldocs/int.doc.html), [ProcessTemplateList](https://api.veritone.com/v3/graphqldocs/processtemplatelist.doc.html) - ---- -#### rootFolders - -Retrieve the root folders for an organization - -_**Arguments**_
- -`type:` The type of root folder to retrieve - -```graphql -rootFolders(type: RootFolderType): [Folder] -``` - -*See also:*
[RootFolderType](https://api.veritone.com/v3/graphqldocs/rootfoldertype.doc.html), [Folder](https://api.veritone.com/v3/graphqldocs/folder.doc.html) - ---- -#### savedSearches - -Fetch all saved searches that the current user has made -Fetch all saved searches that have been shared with -the current users organization -Include any saved searches that the user has created - -_**Arguments**_
- -`offset:` - -`limit:` - -`includeShared:` - -`filterByName:` - -`orderBy:` - -`orderDirection:` - - -```graphql -savedSearches( - offset: Int, - limit: Int, - includeShared: Boolean, - filterByName: String, - orderBy: SavedSearchOrderBy, - orderDirection: OrderDirection -): SavedSearchList! -``` - -*See also:*
[Int](https://api.veritone.com/v3/graphqldocs/int.doc.html), [Boolean](https://api.veritone.com/v3/graphqldocs/boolean.doc.html), [String](https://api.veritone.com/v3/graphqldocs/string.doc.html), [SavedSearchOrderBy](https://api.veritone.com/v3/graphqldocs/savedsearchorderby.doc.html), [OrderDirection](https://api.veritone.com/v3/graphqldocs/orderdirection.doc.html), [SavedSearchList](https://api.veritone.com/v3/graphqldocs/savedsearchlist.doc.html) - ---- -#### schema - -_**Arguments**_
- -`id:` - -```graphql -schema(id: ID!): Schema -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [Schema](https://api.veritone.com/v3/graphqldocs/schema.doc.html) - ---- -#### schemaProperties - - _**Arguments**_
- -`dataRegistryVersion:` - -`search:` - -`limit:` Limit - -`offset:` Offset - - -```graphql -schemaProperties( - dataRegistryVersion: [DataRegistryVersion!], - search: String, - limit: Int, - offset: Int -): SchemaPropertyList -``` - -*See also:*
[DataRegistryVersion](https://api.veritone.com/v3/graphqldocs/dataregistryversion.doc.html), [String](https://api.veritone.com/v3/graphqldocs/string.doc.html), [Int](https://api.veritone.com/v3/graphqldocs/int.doc.html), [SchemaPropertyList](https://api.veritone.com/v3/graphqldocs/schemapropertylist.doc.html) - ---- -#### schemas - -Retrieve a list of schemas for structured data ingestions - -_**Arguments**_
- -`id:` Id of a schema to retrieve - -`ids:` Ids of schemas to retrieve - -`dataRegistryId:` Specify the id of the DataRegistry to get schemas - -`status:` Specify one or more statuses to filter by schema status - -`majorVersion:` Specify a major version to filter schemas - -`name:` Specify a data registry name to filter schemas - -`nameMatch:` The strategy used to find data registry name - -`limit:` Limit - -`offset:` Offset - -`orderBy:` Specify one or more fields and direction to order results - - -```graphql -schemas( - id: ID, - ids: [ID!], - dataRegistryId: ID, - status: [SchemaStatus!], - majorVersion: Int, - name: String, - nameMatch: StringMatch, - limit: Int, - offset: Int, - orderBy: [SchemaOrder] -): SchemaList -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [SchemaStatus](https://api.veritone.com/v3/graphqldocs/schemastatus.doc.html), [Int](https://api.veritone.com/v3/graphqldocs/int.doc.html), [String](https://api.veritone.com/v3/graphqldocs/string.doc.html), [StringMatch](https://api.veritone.com/v3/graphqldocs/stringmatch.doc.html), [SchemaOrder](https://api.veritone.com/v3/graphqldocs/schemaorder.doc.html), [SchemaList](https://api.veritone.com/v3/graphqldocs/schemalist.doc.html) - ---- -#### searchMedia - -Search for media across an index. -This query requires a user token. - -_**Arguments**_
- -`search:` JSON structure containing the search query. TODO link to syntax documentation - -```graphql -searchMedia(search: JSONData!): SearchResult -``` - -*See also:*
[JSONData](https://api.veritone.com/v3/graphqldocs/jsondata.doc.html), [SearchResult](https://api.veritone.com/v3/graphqldocs/searchresult.doc.html) - ---- -#### searchMentions - -Search for mentions across an index. -This query requires a user token. - -_**Arguments**_
- -`search:` JSON structure containing the search query. TODO link to syntax documentation - -```graphql -searchMentions(search: JSONData!): SearchResult -``` - -*See also:*
[JSONData](https://api.veritone.com/v3/graphqldocs/jsondata.doc.html), [SearchResult](https://api.veritone.com/v3/graphqldocs/searchresult.doc.html) - ---- -#### sharedCollection - -Retrieve a shared collection - -_**Arguments**_
- -`shareId:` share token - -```graphql -sharedCollection(shareId: ID!): SharedCollection -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [SharedCollection](https://api.veritone.com/v3/graphqldocs/sharedcollection.doc.html) - ---- -#### sharedCollectionHistory - -Retrieve shared collection history records - -_**Arguments**_
- -`ids:` Provide an ID to retrieve a single history record. - -`folderId:` Provide a folder ID to filter by collection. - -`shareId:` Provide a share ID to filter by share ID. - -`offset:` Specify maximum number of results to retrieve in this result. Page size. - -`limit:` Specify maximum number of results to retrieve in this result. - - -```graphql -sharedCollectionHistory( - ids: [ID!], - folderId: ID, - shareId: String, - offset: Int, - limit: Int -): SharedCollectionHistoryList! -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [String](https://api.veritone.com/v3/graphqldocs/string.doc.html), [Int](https://api.veritone.com/v3/graphqldocs/int.doc.html), [SharedCollectionHistoryList](https://api.veritone.com/v3/graphqldocs/sharedcollectionhistorylist.doc.html) - ---- -#### sharedFolders - -Retrieve the shared folders for an organization - -_**Arguments**_
- -```graphql -sharedFolders: [Folder] -``` - -*See also:*
[Folder](https://api.veritone.com/v3/graphqldocs/folder.doc.html) - ---- -#### sharedMention - -Retrieve a shared mention - -_**Arguments**_
- -`shareId:` share token - -```graphql -sharedMention(shareId: ID!): SharedMention -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [SharedMention](https://api.veritone.com/v3/graphqldocs/sharedmention.doc.html) - ---- -#### structuredData - -Retrieve a structured data object - -_**Arguments**_
- -`id:` Supply the ID of the structured data object to retrieve. This will override filters. - -`schemaId:` Schema Id for the structured data object to retrieve - -```graphql -structuredData(id: ID!, schemaId: ID!): StructuredData -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [StructuredData](https://api.veritone.com/v3/graphqldocs/structureddata.doc.html) - ---- -#### structuredDataObject - -Retrieve a structured data object - -_**Arguments**_
- -`id:` Supply the ID of the structured data object to retrieve. This will override filters. - -`schemaId:` Schema Id for the structured data object to retrieve - -```graphql -structuredDataObject(id: ID!, schemaId: ID!): StructuredData -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [StructuredData](https://api.veritone.com/v3/graphqldocs/structureddata.doc.html) - ---- -#### structuredDataObjects - -Retrieve a paginated list of structured data object - -_**Arguments**_
- -`id:` Supply the ID of the structured data object to retrieve. This will override filters. - -`ids:` List of Ids of the structured data objects to retrieve. This will override filters. - -`schemaId:` Schema Id for the structured data object to retrieve - -`orderBy:` - -`limit:` - -`offset:` - -`owned:` - -`filter:` Query to filter SDO. Supports operations such as and, or, eq, gt, lt, etc. TODO link to syntax documentation - -```graphql -structuredDataObjects( - id: ID, - ids: [ID!], - schemaId: ID!, - orderBy: [StructuredDataOrderBy!], - limit: Int, - offset: Int, - owned: Boolean, - filter: JSONData -): StructuredDataList -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [StructuredDataOrderBy](https://api.veritone.com/v3/graphqldocs/structureddataorderby.doc.html), [Int](https://api.veritone.com/v3/graphqldocs/int.doc.html), [Boolean](https://api.veritone.com/v3/graphqldocs/boolean.doc.html), [JSONData](https://api.veritone.com/v3/graphqldocs/jsondata.doc.html), [StructuredDataList](https://api.veritone.com/v3/graphqldocs/structureddatalist.doc.html) - ---- -#### subscription - -_**Arguments**_
- -`id:` - -```graphql -subscription(id: ID!): Subscription! -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [Subscription](https://api.veritone.com/v3/graphqldocs/subscription.doc.html) - ---- -#### task - -Retrieve a single task by ID - -_**Arguments**_
- -`id:` Provide the task ID. - -```graphql -task(id: ID!): Task -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [Task](https://api.veritone.com/v3/graphqldocs/task.doc.html) - ---- -#### temporalDataObject - -Retrieve a single temporal data object - -_**Arguments**_
- -`id:` the TDO ID - -```graphql -temporalDataObject(id: ID!): TemporalDataObject -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [TemporalDataObject](https://api.veritone.com/v3/graphqldocs/temporaldataobject.doc.html) - ---- -#### temporalDataObjects - -Retrieve a list of temporal data objects. - -_**Arguments**_
- -`applicationId:` Application ID to get TDOs for. Defaults to the user's own application. - -`id:` Provide an ID to retrieve a single specific TDO. - -`offset:` Provide an offset to skip to a certain element in the result, for paging. - -`limit:` Specify maximum number of results to retrieve in this result. Page size. - -`sourceId:` Optionally, specify a source ID. TDOs ingested from this source will be returned. - -`scheduledJobId:` Optionally, specify a scheduled job ID. TDOs ingested under this scheduled job will be returned. - -`sampleMedia:` Whether to retrieve only tdos with the specified sampleMedia value - -`includePublic:` Whether to retrieve public data that is not part of the user's organization. The default is false. Pass true to include public data in the result set. - -`orderBy:` - -`orderDirection:` - -`dateTimeFilter:` Provide optional filters against any date/time field to filter objects within a given time window. Matching objects must meet all of the given conditions. - -`mentionId:` Retrieve TDOs associated with the given mention - -```graphql -temporalDataObjects( - applicationId: ID, - id: ID, - offset: Int, - limit: Int, - sourceId: ID, - scheduledJobId: ID, - sampleMedia: Boolean, - includePublic: Boolean, - orderBy: TemporalDataObjectOrderBy, - orderDirection: OrderDirection, - dateTimeFilter: [TemporalDataObjectDateTimeFilter!], - mentionId: ID -): TDOList -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [Int](https://api.veritone.com/v3/graphqldocs/int.doc.html), [Boolean](https://api.veritone.com/v3/graphqldocs/boolean.doc.html), [TemporalDataObjectOrderBy](https://api.veritone.com/v3/graphqldocs/temporaldataobjectorderby.doc.html), [OrderDirection](https://api.veritone.com/v3/graphqldocs/orderdirection.doc.html), [TemporalDataObjectDateTimeFilter](https://api.veritone.com/v3/graphqldocs/temporaldataobjectdatetimefilter.doc.html), [TDOList](https://api.veritone.com/v3/graphqldocs/tdolist.doc.html) - ---- -#### timeZones - -This query returns information about time zones recognized by this -server. The information is static and does not change. - -_**Arguments**_
- -```graphql -timeZones: [TimeZone!]! -``` - -*See also:*
[TimeZone](https://api.veritone.com/v3/graphqldocs/timezone.doc.html) - ---- -#### tokens - -Retrieve user's organization API tokens - -_**Arguments**_
- -```graphql -tokens: [Token] -``` - -*See also:*
[Token](https://api.veritone.com/v3/graphqldocs/token.doc.html) - ---- -#### trigger - -Arguments -id: - -_**Arguments**_
- -```graphql -trigger(id: ID!): Trigger -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [Trigger](https://api.veritone.com/v3/graphqldocs/trigger.doc.html) - ---- -#### triggers - -_**Arguments**_
- -None. - -```graphql -triggers: [Trigger] -``` - -*See also:*
[Trigger](https://api.veritone.com/v3/graphqldocs/trigger.doc.html) - ---- -#### user - -Retrieve an individual user - -_**Arguments**_
- -`id:` The user ID. A user ID is a string in UUID format. - -`organizationIds:` - - -```graphql -user(id: ID!, organizationIds: [ID]): User -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [User](https://api.veritone.com/v3/graphqldocs/user.doc.html) - ---- -#### users - -Retrieve users - -_**Arguments**_
- -`id:` Provide an ID to retrieve a single specific user. A user ID is a string in UUID format. - -`ids:` Provide IDs to retrieve multiple users by ID. - -`name:` Provide a name, or part of one, to search by name. - -`organizationIds:` Provide a list of organization IDs to filter your search by organization. - -`offset:` Provide an offset to skip to a certain element in the result, for paging. - -`limit:` Specify maximum number of results to retrieve in this result. Page size. - -`includeAllOrgUsers:` Include all organization users. - - -```graphql -users( - id: ID, - ids: [ID], - name: String, - organizationIds: [ID], - offset: Int, - limit: Int, - includeAllOrgUsers: Boolean -): UserList -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [String](https://api.veritone.com/v3/graphqldocs/string.doc.html), [Int](https://api.veritone.com/v3/graphqldocs/int.doc.html), [Boolean](https://api.veritone.com/v3/graphqldocs/boolean.doc.html), [UserList](https://api.veritone.com/v3/graphqldocs/userlist.doc.html) - ---- -#### watchlist - -_**Arguments**_
- -`id:` - -```graphql -watchlist(id: ID!): Watchlist -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [Watchlist](https://api.veritone.com/v3/graphqldocs/watchlist.doc.html) - ---- -#### watchlists - -_**Arguments**_
- - id: - - maxStopDateTime: - - minStopDateTime: - - minStartDateTime: - - maxStartDateTime: - - name: - - offset: - - limit: - - orderBy: - - orderDirection: - - isDisabled: Set `true` to include only disabled watchlist or `false` to include only enabled watchlists. By default, - both are included. - -```graphql -watchlists( - id: ID, - maxStopDateTime: DateTime, - minStopDateTime: DateTime, - minStartDateTime: DateTime, - maxStartDateTime: DateTime, - name: String, - offset: Int, - limit: Int, - orderBy: WatchlistOrderBy, - orderDirection: OrderDirection, - isDisabled: Boolean -): WatchlistList -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [DateTime](https://api.veritone.com/v3/graphqldocs/datetime.doc.html), [String](https://api.veritone.com/v3/graphqldocs/string.doc.html), [Int](https://api.veritone.com/v3/graphqldocs/int.doc.html), [WatchlistOrderBy](https://api.veritone.com/v3/graphqldocs/watchlistorderby.doc.html), [OrderDirection](https://api.veritone.com/v3/graphqldocs/orderdirection.doc.html), [Boolean](https://api.veritone.com/v3/graphqldocs/boolean.doc.html), [WatchlistList](https://api.veritone.com/v3/graphqldocs/watchlistlist.doc.html) - ---- -#### widget - -Retrieve a single Widget - -_**Arguments**_
- -`id:` The widget ID - -`collectionId:` - -`organizationId:` - -```graphql -widget(id: ID!, collectionId: ID!, organizationId: ID!): Widget -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [Widget](https://api.veritone.com/v3/graphqldocs/widget.doc.html) - ---- -#### workflowRuntime - -Retrieve Veritone Workflow instance status by ID - -_**Arguments**_
- -`workflowRuntimeId:` an ID - -```graphql -workflowRuntime(workflowRuntimeId: ID!): WorkflowRuntimeResponse! -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [WorkflowRuntimeResponse](https://api.veritone.com/v3/graphqldocs/workflowruntimeresponse.doc.html) - ---- -#### workflowRuntimeStorageData - -Get a specific `WorkflowRuntimeStorageDataList based on workflow runtime ID - -_**Arguments**_
- -`workflowRuntimeId:` Unique ID of the workflow instance - -`storageKey:` The unique ID to retrieve a single workflow data object - -`storageKeyPrefix:` A prefix filter used to return a set of workflow data items whose dataKey starts with dataKeyPrefix - -`offset:` Offset for paging - -`limit:` Limit on result size, for paging (page size). Note that workflow runtime data can be arbitrarily large, therefore smaller paging should be preferred. - -```graphql -workflowRuntimeStorageData( - workflowRuntimeId: ID!, - storageKey: String, - storageKeyPrefix: String, - offset: Int, - limit: Int -): WorkflowRuntimeStorageDataList! -``` - -*See also:*
[ID](https://api.veritone.com/v3/graphqldocs/id.doc.html), [String](https://api.veritone.com/v3/graphqldocs/string.doc.html), [Int](https://api.veritone.com/v3/graphqldocs/int.doc.html), [WorkflowRuntimeStorageDataList](https://api.veritone.com/v3/graphqldocs/workflowruntimestoragedatalist.doc.html) diff --git a/docs/apis/schema/listing.md b/docs/apis/schema/listing.md deleted file mode 100644 index 3c2658d93e..0000000000 --- a/docs/apis/schema/listing.md +++ /dev/null @@ -1,108 +0,0 @@ -# The Veritone GraphQL Schema Files - -Some commonly available tooling for GraphQL, including code generators, -requires a schema reference. - -Current V3 schema files are available at: - -* [schema.graphql](/schemas/api/v3/schema.graphql ':ignore') -* [schema.json](/schemas/api/v3/schema.json ':ignore') - -The GraphQL server provides introspection queries that can be used to -dynamically inspect the schema. These queries have complete access to -all schema metadata in the same way that the server itself does. - -Access these queries with the special `__schema` and `__type` queries. - -For example: - -```graphql -query { - __schema { - types { - name - kind - description - fields { - name - type { - name - description - } - description - args { - name - description - type { - name - description - } - defaultValue - } - } - } - } -} -``` - -This query generates: - -```json -{ - "data": { - "__schema": { - "types": [ - { - "name": "Query", - "kind": "OBJECT", - "description": "Queries are used to retrieve data. If you're new to our API,\ntry the me query to explore the information you have access to.\nHit ctrl-space at any time to activate field completion hints, and\nmouse over a field or parameter to see its documentation.", - "fields": [ - { - "name": "temporalDataObjects", - "type": { - "name": "TDOList", - "description": "" - }, - "description": "Retrieve a list of temporal data objects.", - "args": [ - { - "name": "applicationId", - "description": "Application ID to get TDOs for. Defaults to the user's own application.", - "type": { - "name": "ID", - "description": "The ID scalar type represents a unique identifier, often used to refetch an object or as key for a cache. The ID type appears in a JSON response as a String; however, it is not intended to be human-readable. When expected as an input type, any string (such as \"4\") or integer (such as 4) input value will be accepted as an ID." - }, - "defaultValue": null - }, - { - "name": "id", - "description": "Provide an ID to retrieve a single specific TDO.", - "type": { - "name": "ID", - "description": "The ID scalar type represents a unique identifier, often used to refetch an object or as key for a cache. The ID type appears in a JSON response as a String; however, it is not intended to be human-readable. When expected as an input type, any string (such as \"4\") or integer (such as 4) input value will be accepted as an ID." - }, - "defaultValue": null - }, - { - "name": "offset", - "description": "Provide an offset to skip to a certain element in the result, for paging.", - "type": { - "name": "Int", - "description": "The Int scalar type represents non-fractional signed whole numeric values. Int can represent values between -(2^31) and 2^31 - 1. " - }, - "defaultValue": "0" - }, - { - "name": "limit", - "description": "Specify maximum number of results to retrieve in this result. Page size.", - "type": { - "name": "Int", - "description": "The Int scalar type represents non-fractional signed whole numeric values. Int can represent values between -(2^31) and 2^31 - 1. " - }, - "defaultValue": "30" - }, -``` - -GraphQL schema files are auto-generated in `.graphql` and `.json` format. - -To generate your own copies from Veritone's, or any other, GraphQL endpoint, try GraphCool's get-graphql-schema tool: https://github.com/graphcool/get-graphql-schema. diff --git a/docs/apis/search-quickstart/Get-API-Token-1.png b/docs/apis/search-quickstart/Get-API-Token-1.png deleted file mode 100644 index 7a75b1e6cd..0000000000 Binary files a/docs/apis/search-quickstart/Get-API-Token-1.png and /dev/null differ diff --git a/docs/apis/search-quickstart/Get-API-Token-2.png b/docs/apis/search-quickstart/Get-API-Token-2.png deleted file mode 100644 index 2482ee15ed..0000000000 Binary files a/docs/apis/search-quickstart/Get-API-Token-2.png and /dev/null differ diff --git a/docs/apis/search-quickstart/README.md b/docs/apis/search-quickstart/README.md deleted file mode 100644 index 72f5e78790..0000000000 --- a/docs/apis/search-quickstart/README.md +++ /dev/null @@ -1,2859 +0,0 @@ -# Search Quickstart Guide - -## Getting Started - -Veritone Search API includes rich, full-text search features to query against an organization’s public and private media indexes. The search function is highly customizable includes core functions that support parsing, aggregation, and auto-completion. Searches are performed using a variety of supported query types, expressed in JSON. Most search operations make use of optional query configurations that allow you to control the search behavior and narrow results by defining aspects of the content and values. - -Veritone API is built around the GraphQL paradigm to provide a more efficient way to deliver data with greater flexibility than a traditional REST approach. GraphQL is a query language that operates over a single endpoint using conventional HTTP requests and returning JSON responses. The structure not only lets you call multiple nested resources in a single query, it also allows you to define requests so that the query you send matches the data you receive. - -To make effective use of the Search API, you’ll need to know a few things about how data is stored in Veritone, the various options for structuring queries, and requirements for performing successful requests. This quickstart guide provides everything you need to help get your integration up and running as quickly as possible. We designed this quickstart to be user-friendly and example-filled, but if you have any questions, please don’t hesitate to reach out to our [Developer Support Team](mailto:devsupport@veritone.com) for help. - -### **Base URL** - -Veritone uses a single endpoint for accessing the API. All calls to the API are *POST* requests and are served over *http* with *application/json* encoded bodies. The base URL varies based on the geographic region where the services will run. When configuring your integration, choose the base URL that supports your geographic location from the list below. - - - - - - - - - - - - - - -
RegionBase URL
United States[https://api.veritone.com/v3/graphql](https://api.veritone.com/v3/graphql)
Europe[https://api.uk.veritone.com/v3/graphql](https://api.uk.veritone.com/v3/graphql)
- -*Note:* The above base URLs are provided for use within SaaS environments. On-prem deployments access the API using an endpoint that's custom configured to the environment. - -### **Making Sample Requests** - -To make it easier to explore, write, and test the API, we set up [GraphiQL](https://api.veritone.com/v3/graphiql) — an interactive playground that gives you a code editor with autocomplete, validation, and syntax error highlighting features. Use the GraphiQL interface to construct and execute queries, experiment with different schema modifications, and browse documentation. In addition, GraphiQL bakes authorization right into the schema and automatically passes the *Authentication* header with a valid token when you’re logged into the Veritone system. - -Veritone’s GraphiQL interface is the recommended method for ad hoc API requests, but calls can be made using any HTTP client. All requests must be HTTP POST to the base URL for your geographic region with the *query* parameter and *application/json* encoded bodies. In addition, requests must be authenticated using an API Token. Pass the token in your request using the *Authorization* header with a value *Bearer token*. If you’re using a raw HTTP client, the query body contents must be sent in a string with all quotes escaped. - -### **Authentication** - -Veritone Job API uses bearer token authentication for requests. To authenticate your calls, provide a valid API Token in the *Authentication* header of the request with the value *Bearer token*. Requests made without this header or with an invalid token will return an error code. - -An API Token can be generated in the Veritone Admin App by your organization administrator. If your organization does not use the Admin App, please contact your Veritone Account Manager for assistance. - -**To generate an API Token:** -1. Log into the Veritone Platform and select **Admin** from the *App Picker* drop-down. The *Admin App* opens. -2. Click the **API Keys** tile. The *API Key* page opens. - -![Get API Token](Get-API-Token-1.png) - -3. Click **New API** Key. The *New API Key* window opens. - -![Get API Token](Get-API-Token-2.png) - -4. Enter a token name and select the permissions needed for the token to perform the required API tasks. Click **Generate Token** to save. The *Token Generated* window opens. -5. Copy your token and click **Close** when finished. - -*Note:* Once the *Token Generated* window closes, the token code no longer displays and it cannot be viewed again. - -### Relevance - -Search operations return results in order of relevance — the result that’s most relevant to the search query is the first item in the result set, and the least relevant is last. Relevance scoring is based on three primary factors: - -* **Term Frequency:** How often does the term appear in the field? The more often, the more relevant. A field containing five instances of the same term is more likely to be relevant than a field containing just one mention. -* **Inverse Index Frequency:** How often does each term appear in the index? The more often, the less relevant. Terms that appear in many records have a lower weight than more-uncommon terms. -* **Field Length:** How long is the field? The longer it is, the less likely it is that words in the field will be relevant. A term appearing in a field with a short title carries more weight than the same term appearing in a long content field. - -## Query Basics & Syntax - -Veritone Search API gives you the flexibility to build a variety of query types to search and retrieve indexed media and mentions content. The Search API allows you to combine a series of simple elements together to construct queries as simple or as complex as you’d like in JSON format. Although queries are customizable, there is a common structure and set of core parameters that each must use. In addition, there are a number of optional filters, components, and syntax options that can be specified to modify a query. - -### Content Type - -Searches in Veritone are performed against two types of content: media and mentions. Each request must specify one of the following search content types as the root operation of the request. -* **Search Media:** The *Search Media* operation searches media files and assets for matching records. -* **Search Mentions:** The *Search Mentions* operation searches for matching records in mentions and watchlists. - -#### **Required Query Parameters** - -Regardless of the level of complexity, each search query in Veritone operates on four core elements: an *index*, a *field*, an *operator*, and an operator value or variable(s). - -##### **Index** - -All search functionality runs against Veritone’s public and private index databases. The *index* field defines whether to search the organization’s public or private index (or both) for matching documents. There are two possible *index* values: "global," which refers to the public media index, and "mine", which refers to private media uploaded to an account. Each request must specify at least one *index* value enclosed in brackets. - -##### **Field** - -Content in Veritone is made searchable by mapping to specific document and data types. The *field* parameter defines the type of data/document to be searched, and each query must specify a value for the *field* property to be successful. If a *field* value is not provided, an empty result set will be returned. The *field* parameter uses a definitive set of values. See *Appendix 1: Fields* for more information. - -##### **Operator** - -Operators are the heart of the query — they describe the type of search that will be performed. Each *operator* uses one or more additional properties that add greater definition to the query criteria. All queries must specify at least one *operator*. - -The table below provides an overview of the different operators that can be used. Additional information about each of the operators is provided in the section on *Query Schema and Types*. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Operator Name
Description
rangeFinds matching items within a specified date/time range.
termFinds an exact match for a word or phrase.
termsFinds exact matches for multiple words or phrases.
query_stringSearches plain text strings composed of terms, phrases, and Boolean logic.
word_proximityFinds exact matches for words that are not located next to one another.
existsChecks for the presence of a specified engine output data type.
andFinds records for matching values when all of the conditions are true.
orFinds matching records when at least one of the conditions is true.
query_objectSearches for an array of objects with a common field.
- -#### **Grouping** - -To apply two or more Booleans to a single field, group multiple terms or clauses together with parenthesis. - -Below is an example of a search for Kobe or Bryant and basketball or Lakers. - -``` -value: "(kobe OR bryant) AND (basketball OR lakers)" -``` - -#### **Escaping Special Characters** - -Veritone Search API supports escaping special characters in a query string to ensure they aren’t interpreted by the query parser. If any of the following special characters have been enabled, use a double backslash before the character to search for it. - -Special characters: + - ! ( ) { } [ ] ^ " ~ * ? : \ - -In the example below, the ! is escaped since it’s part of the company name "ideaDes!gns". -``` -value: "ideaDes\\!gns" -``` - -Use a single backslash to escape quotes around a phrase in a query string. The example below escapes the double quotes around "Kobe Bryant" to ensure they’re interpreted literally and not as string closers. -``` -value: "\"kobe bryant\" AND (basketball OR lakers)" -``` - -## Query Schema and Types - -Veritone Search API accepts a variety of query types, features, and options that make it easy to build queries as simple or complex as you’d like. - -Queries can be constructed in multiple ways to search terms, phrases, ranges of time, and more. In addition, the Search API supports query modifiers (such as Wildcards and Select Statements) to provide an even broader range of searching options. - -### **Schema** - -A schema is created by supplying a *Search Media* or *Search Mentions* content type as the root operation and then providing a query object with an *operator*, *index*, *field*, and a list of other properties that describe the data you want to retrieve. All search requests specify JSON data as the response field. - -#### **Sample Search Query Schema** - -```graphql - query{ => The base query parameter used to set up the GraphQL query object. (required) - -------request fields----------- - searchMedia(search:{ => The content type to be searched and search variable. (required) - index: array => Specifies whether to search the public ("global") or private (“mine) index. (required) - query: object => The query object that defines the pipeline for the data to be returned. (required) - field: string => The data or document type to be searched. (required) - operator: string => The type of search to be performed. Common types include term, range, and query_string. (required) - value: string => The exact data to be searched for in a field. (required) - offset: integer => The zero-based index of the first record to show in the response. (optional) - limit: integer => The maximum number of results to return. (optional) - }){ - -------return fields------------ - jsondata: => A JSON object with response data. Each search request must specify jsondata as the return field. (required) - } -} -``` - -### **Query Types** - -Veritone supports several types of queries, which are identified by the *operator* field. Queries are organized in a hierarchical structure and use nested fields in curvy brackets. The schema uses the *query* parameter as a wrapper around a subquery that contains an *operator*, *field*, and a *value* or *variables*. The various query types allow you to customize the search — available options are described below. - -#### **Term/Terms** - -The *term/terms* operators are used to find exact matches of keywords or phrases. A *term/terms* query can match text, numbers, and dates. Searches produce a list of records that contain the keywords or phrases, no matter where they appear in the text. - -*Important Notes:* -- Enclose single keywords and phrases in quotation marks (e.g., "basketball", “Mike Jones”). -- A phrase includes two or more words together, separated by a space (e.g., "free throw line"). The order of the terms in a phrase is respected. Only instances where the words appear in the same order as the input value are returned. - -**Search by Term**
-The *term* operator is used to find an exact match for a single keyword or phrase. - -The following example shows a *term* search against the global index to find media files where the word "Mike" appears in the program name. - -``` -query: { - operator: "term" - field: "programName" - value: "mike" - } -} -``` - -**Search by Terms**
-The *terms* operator allows you to search for multiple keywords or phrases. Search terms can include any string of words or phrases separated by commas. When submitting multiple search terms, search uses a default *OR* behavior and will return results that contain an exact match of any of the words/phrases. - -*Important Note:* When building a *terms* query, note that the *terms* operator and the *values* property are written in plural form. - -The example below is a *terms* search query for the words "football" and “touchdown” found in transcript text in the user’s private media index. -``` -query: { - operator: "terms" - field: "transcript.transcript" - values: ["football", "touchdown"] - } -``` - -#### **Date/Time Range Search** - -The *range* operator can be used to find content that falls into a specified time frame. This query supports the following types of ranges: -- **Inclusive Range:** Uses a combination of comparison property filters to find matching content between a specific starting and ending date. -- **Open-Ended Range:** Uses a single comparison property filter to search before or after a specific point in time. - -A *range* query must include at least one comparison property and specify the date as the value (using[ Unix/Epoch Timestamp](https://www.epochconverter.com/) format in milliseconds and UTC time zone). - -**Comparison Properties** - - - - - - - - - - - - - - - - - - - - - - -
Name
Description
gtgreater than: Searches for documents created after the specified date/time.
ltless than: Searches for documents created before the specified date/time.
gtegreater than or equal to: Searches for documents created on or after the specified date/time.
lteless than or equal to: Searches for documents created on or before the specified date/time.
- -**Inclusive Date Range** - -To search for records created between specified dates, use two of the comparison property options to define the *to* and *from* dates. - -The following example is a search against the public index for the 10 most recent media files timestamped on or after 1/26/2017, 5:54:00 AM, and before 6/23/2017, 6:30:00 AM. -``` -query: { - operator: "range" - field: "absoluteStartTimeMs" - gte: 1485417954000 - lt: 1498199400000 - } -} -``` - -**Open-Ended Date Range** - -To search with an open-ended range (e.g., find files before a specified date), use just one of the comparison property options. - -The example below shows a search for the 30 most recent media files in the public index timestamped after 3/15/2017 at 6:30:00 AM. -``` -query: { - operator: "range" - field: "absoluteStartTimeMs" - lt: 1485417954000 - } -``` -#### **Query String** - -The *query_string* operator performs full-text searches and constructs queries as text strings using terms, phrases, and Boolean logic. The biggest advantage of the query string is its syntax that parses multiple structured queries into simpler ones. - -*Important Notes:* -- The query string supports more complex query constructs, such as wildcard searches. -- Multiple query statements that are not joined by the *and* operator will return results that match any of the conditions. -- Date and time values are not supported in a query string. - -The below example will return results transcripts where "Kobe Bryant" or “Lakers” was found. -``` - query: { - operator: "query_string" - field: "transcript.transcript" - value: "\"kobe bryant\" lakers" - } -``` - -**Using AND/OR in a Query String** - -You can specify a broader or more narrow search by using the *AND* and *OR* operators between values in a query string. When using *AND/OR* within a quoted string, they are treated as part of the field value (not the main query operator) and must be written in all uppercase. In addition, multiple terms in a query string can be grouped together using parenthesis to make the logic clear. Note that terms in parentheses are processed first. - -The following example is a search for transcripts where "Kobe Bryant" was found with either the word “basketball” or the word “Lakers.” -``` -query: { - operator: "query_string" - field: "transcript.transcript" - value: "\"kobe bryant\" AND (basketball OR lakers)" - } -``` - -#### **Word Proximity** - -While the *terms* query searches for specified terms in the exact order as the input value, a *word proximity* search allows the words to be separated or to appear in a different order. The *word proximity* operator uses the *inOrder* property as a Boolean to find terms where they do not appear together, and the *distance* property to specify the maximum number of words that can separate them. - -**Word Proximity Properties** - - - - - - - - - - - - - - - - - - - - - - - -
Name
Required
Type
Description
Example
inOrderyesBooleanA Boolean that when set to false searches for all of the words in an order different than the input value. Note that if the distance property has a value of 0, inOrder will be set to true.inOrder: false
distanceyesintegerThe number of non-search words that can separate the specified terms to be found. To match words in the same order as the input value, use a distance value of 0. (Note that a 0 value sets inOrder to true.) To transpose two words, enter a value of 1.distance: 10
- -The following query finds the terms "NFL," “football,” and “game” in transcript text when they appear within 10 words of one another in any order. -``` - query: { - operator: "word_proximity" - field: "transcript.transcript" - values: ["nfl football game"] - inOrder: false - distance: 10 - } -``` - -#### **Exists** - -The *exists* operator is used to check for the presence of a specific field. This query is useful for retrieving matching media files of a specific data type. If the specified *field* is not found, the operator will consider it non-existent and it will return no results. - -The example below is a search for media files with the field "veritone-file.mimetype". -``` -query: { - operator: "exists" - name: "veritone-file.mimetype" - } -``` - -#### **And / Or (as Query Operators)** - -The *and* and *or* operators allow you to combine two or more query clauses with Boolean logic to create broader or more narrow search results. These operators chain conditional statements together and always evaluate to true or false. As main query operators, *and/or* are case insensitive and can be written as uppercase or lowercase. It’s important to note that when using these operators in compound queries, *and* takes precedence over *or*. - -**and Operator** - -The *and* operator matches documents where both terms exist anywhere in the text of a single record. - -The search below returns results for "Morning Show AM" mentions found on November 11, 2017. -``` - query: { - operator: "and", - conditions: [ - { - operator: "term" - field: "trackingUnitName" - value: "Morning Show AM" - }, - { - operator: "term" - field: "mentionDate" - value: "2017-11-09" - } - ] - } -``` - -**or Operator** - -The *or* operator connects two conditions and returns results if either condition is true. - -The example below shows a search for the name "Bob" or “Joe” or “Sue” in a transcript or records that created on or after November 17, 2017 at 9:00 AM. -``` - query: { - operator: "or", - conditions: [ - { - operator: "terms", - field: "transcript.transcript" - values: ["bob", "joe", "sue"] - }, - { - operator: "range", - field: "absoluteStartTimeMs" - gte: 1510909200000 - } - ] - } -``` - -**Combining and & or** - -The *and* and *or* (and *not*) operators can be combined as multi-level subqueries to create a compound condition. When these operators are used together, it’s important to note that *not* is evaluated first, then *and*, and finally *or*. - -The example below is a search for records found between November 17, 2017 at 9:00 AM and December 1, 2017 at noon where the word "basketball" or “Lakers” was found in a transcript, or the NBA logo was detected. -``` - operator: "and", - conditions: [ - { - operator: "range" - field: "absoluteStartTimeMs, - gte: 1510909200000 - lte: 1512129600000 - }, - { - operator: "or" - conditions: [ - { - operator: "query_string" - field: "transcript.transcript" - value: ["basketball", "lakers"] - } - operator: "term" - field: "logo-recognition.series.found" - value: "NBA" - } - ] - } - ] -} -``` - -#### **Query Object** - -The *query_object* operator allows an array of objects to be queried independently of one another. Each *query_object* uses a Boolean operator (e.g., *and*, *or*, *not*) to combine a list of nested subqueries. Nested subqueries can use any operator type. - -The below example is a search for an ESPN logo or the words "touchdown" and “ruled” or “college” in a transcript on or before December 11, 2017 at 9:11 PM. -``` - operator: "query_object" - query:{ - operator: "or" - conditions: [{ - operator: "range" - field: "absoluteStartTimeMs" - lte: 1513026660000 - }, - { - operator: "term" - field: "logo-recognition.series.found" - value: "ESPN" - }, - { - operator: "query_string" - field: "transcript.transcript" - value: "touchdown AND (ruled OR college)" - } - ] - } -``` - -### **Query Modifiers** - -#### **Not** - -A *not* modifier returns search results that do not include the specified values. Any of the query operators can be negated by adding the property *not* with a value of *true*. Note that negation is currently unsupported at the compound operator level. - -The below example is a search that excludes the terms "NFL," “football,” and “game.” -``` - query: { - operator: "terms" - field: "transcript.transcript" - values: ["nfl", "football", "game"] - not: true - distance: 10 -} -``` - -#### **Wildcard Searches** - -The *wildcard* modifier is useful when you want to search various forms of a word. Wildcard searches can be run on individual terms, using ? to replace a single character, and * to replace zero or more characters. For example, he* returns results that would include her, help, hello, helicopter and any other words that begin with "he". Searching he? will only match three-letter words that start with “he”, such as hem, hen, and her. - -*Important Notes:* -- There can be more than one wildcard in a single search term or phrase, and the two wildcard characters can be used in combination. (e.g., j*?? will match words with three or more characters that start with the letter j.) -- Wildcard matching is only supported within single terms and not within phrase queries. (e.g., m*th* will match "method" but not “meet there”.) -- A wildcard symbol cannot be used as the first character of a search. - -#### **Select Statements** - -A *select* query modifier lets you retrieve only the data that you want. It also allows you to combine data from multiple *field* property sources. Submitting a request with the *select* parameter returns full record matches for all of the specified values. To use the *select* filter, enter a comma-separated list of *field* names to return. - -The example below is a search for records where "Kobe Bryant" along with either the word “basketball” or the word “Lakers” is found in either a transcript or a file uploaded to Veritone CMS. -``` - query: { - operator: "query_string" - field: "transcript.transcript" - value: "\"kobe bryant\" AND (basketball OR lakers)" - }, - select: ["transcript.transcript", "veritone-file"] -} -``` - -## Sample Requests and Responses - -Veritone’s GraphiQL interface is a service you can use to easily interact with the Search API. We recommend using GraphiQL ([https://api.veritone.com/v3/graphiql](https://api.veritone.com/v3/graphiql)) for making test API requests, but calls can also be made using a different HTTP client. All requests must be HTTP POST to the Veritone GraphQL endpoint with the *query* parameter and *application/json* encoded bodies. In addition, requests must be authenticated with an API Token. Pass the token in your request using the *Authorization* header with a value *Bearer *. If you’re using a raw HTTP client, the query body contents must be sent in a string (not an object) with all quotes escaped. - -Following are a variety of example queries that demonstrate how to put everything together. The sample requests provided are structured for use in our GraphiQL interface, but we’ve also included the basic cURL structure for your reference below. Please note that the examples shown do not use client information and are not language specific. For fields that require account-specific data (such as a Recording ID), replace the value with your own. - -### **Basic cURL Structure** - -```bash -curl -X POST \ - https://api.veritone.com/v3/graphql \ - -H 'authorization: Bearer 2079b07c-1a6f-4c2e-b534-a1aaa7f7fe42' \ - -H 'content-type: application/json' \ - -d '{ - "query": " query { searchMedia(search: { offset: 0, limit: 2, index: [\"global\"], query: { operator: \"term\", field: \"programName\", value: \"basketball\" }}) { jsondata } }" -}' -``` - -### **Sample Request 1: Query Object Search** - -The following example is a media search of both indexes where NBA was recognized as text and the words "Kobe Bryant" and “basketball” or “Lakers” was found in a transcript. This request also includes a select statement that limits the search to transcripts and Veritone files. -```graphql -query{ - searchMedia(search:{ - offset: 0 - limit: 1 - index: ["mine"] - operator: "query_object" - query:{ - operator: "or" - conditions: [{ - operator: "range" - field: "absoluteStartTimeMs" - lt: 1485417954000 - }, - { - operator: "term" - field: "text-recognition.series.ocrtext" - value: "nba" - }, - { - operator: "query_string" - field: "transcript.transcript" - value: "\"kobe bryant\" AND (basketball OR lakers)" - } - ] - } -}) { - jsondata - } -} -``` - -#### **Sample Response 1: Query Object Search** - -```json -{ - "data": { - "searchMedia": { - "jsondata": { - "results": [ - { - "recording": { - "recordingId": "16322767", - "fileLocation": "https://inspirent.s3.amazonaws.com/assets/16322767/b8936c78-2186-72e5-93af-5c5aa13dd982.mp4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAI7L6G7PCOOOLA7MQ%2F20171215%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20171215T155327Z&X-Amz-Expires=604800&X-Amz-Signature=49a0bec006693905a66cfb0972167170b14947e05c66dbcf98ed705f31726a03&X-Amz-SignedHeaders=host", - "fileType": "video/mp4", - "programId": "-1", - "programName": "Private Media", - "mediaSourceId": "-1", - "mediaSourceTypeId": "5", - "tags": [ - { - "value": "TEDx", - "displayName": "TEDx" - }, - { - "value": "Tolo West", - "displayName": "Tolo West" - } - ], - "sliceTime": 1484780795, - "mediaStartTime": 1484780045, - "aibDuration": 810, - "isOwn": true, - "hitStartTime": 1484780795, - "hitEndTime": 1484780855 - }, - "startDateTime": 1484780795, - "stopDateTime": 1484780855, - "hits": [] - } - ], - "totalResults": 728274, - "limit": 1, - "from": 0, - "to": 0, - "searchToken": "171a1b10-e9c0-11e7-bae7-61a10650068b", - "timestamp": 1513353207 - } - } - } -} -``` - -### **Sample Request 2: Exists Query** - -The example below is a media search against the private index for records with a filename. -```graphql -query{ - searchMedia(search:{ - offset: 0 - limit: 1 - index: ["mine"] - query:{ - operator: "exists" - name: "veritone-file.filename" - } - }) { - jsondata - } -} -``` - -#### **Sample Response 2: Exists Query** - -```json -{ - "data": { - "searchMedia": { - "jsondata": { - "results": [ - { - "recording": { - "recordingId": "43033727", - "fileLocation": "https://inspirent.s3.amazonaws.com/assets/43033727/ea9d9845-775b-48cd-aada-16fa56894ba0.mp4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAI7L6G7PCOOOLA7MQ%2F20171215%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20171215T181255Z&X-Amz-Expires=604800&X-Amz-Signature=2fd918d5ac20979bd27d365bfa455904cf1726307fddbd362a28a8bd9a0a81a8&X-Amz-SignedHeaders=host", - "fileType": "video/mp4", - "programId": "-1", - "programName": "Weekly Talkshow", - "programLiveImage": "https://inspirent.s3.amazonaws.com/assets/43033727/fe693b30-18ae-47c7-984f-530eab61d7.jpeg", - "mediaSourceId": "-1", - "mediaSourceTypeId": "5", - "sliceTime": 1512682022, - "mediaStartTime": 1512681992, - "aibDuration": 90, - "isOwn": true, - "hitStartTime": 1512682022, - "hitEndTime": 1512682082 - }, - "startDateTime": 1512682022, - "stopDateTime": 1512682082, - "hits": [ - { - "veritone-file": { - "filename": "Veritone_v06.mp4", - "mimetype": "video/mp4", - "size": 162533502 - } - } - ] - } - ], - "totalResults": 733275, - "limit": 1, - "from": 0, - "to": 0, - "searchToken": "930f0960-e1c3-11e7-9e94-eba5f6b5faf7", - "timestamp": 1513361576 - } - } - } -} -``` - -### **Sample Request 3: "or" Query with Negation** - -The example below is a media search of the public index for Coke or Pepsi logos but not Redbull. -```graphql -query{ - searchMedia(search:{ - offset: 0 - limit: 1 - index: ["global"] - query:{ - operator: "or" - conditions: [{ - operator: "term" - field: "logo-recognition.series.found" - value: "Coke" - }, - { - operator: "term" - field: "logo-recognition.series.found" - value: "Pepsi" - }, - { - operator: "term" - field: "logo-recognition.series.found" - value: "Redbull" - not: true - }] - } - }) { - jsondata - } -} -``` - -#### **Sample Response 3: "or" Query with Negation** - -```json -{ - "data": { - "searchMedia": { - "jsondata": { - "results": [ - { - "recording": { - "recordingId": "43842334", - "fileLocation": "https://inspirent.s3.amazonaws.com/tv-recordings/vv-stream/39898/2017/12/14/processed/39898_20171214_0720_263a786f-3d59-4572-9752-984ded917573.mp4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAI7L6G7PCOOOLA7MQ%2F20171214%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20171214T211125Z&X-Amz-Expires=604800&X-Amz-Signature=af6b9acc5b113949fb3d01ec3464c14e10449bd0bd51d250bdd906504588caa2&X-Amz-SignedHeaders=host", - "fileType": "video/mp4", - "programId": "27405", - "programName": "Prime Time", - "programImage": "https://s3.amazonaws.com/prod-veritone-ugc/programs/27405/3tWiUgZNS42vWB55dED3_GRftjRV6_400x400.jpg", - "mediaSourceId": "37802", - "mediaSourceTypeId": "2", - "isPublic": true, - "sliceTime": 1513236060, - "mediaStartTime": 1513236000, - "aibDuration": 120, - "isOwn": false, - "hitStartTime": 1513236064, - "hitEndTime": 1513236067 - }, - "startDateTime": 1513236060, - "stopDateTime": 1513236120, - "hits": [ - { - "logo-recognition": { - "source": "bf50297a-55f1-5a27-17a1-d213ae0c7f55", - "series": [ - { - "end": 67000, - "found": "Pepsi", - "salience": 0.300785630941391, - "start": 64000 - } - ] - } - } - ] - } - ], - "totalResults": 808, - "limit": 1, - "from": 0, - "to": 0, - "searchToken": "57d785e0-9213-11e7-9a71-df8e48a9af47", - "timestamp": 1513285885 - } - } - } -} -``` - -### **Sample Request 4: Word Proximity Search** - -The example below is a media search of the public index for instances where the words emergency, broadcast, and fire are found in any order within 10 words of one another. -```graphql -query{ - searchMedia(search:{ - offset: 0 - limit: 1 - index: ["global"] - query:{ - operator: "word_proximity" - field: "transcript.transcript" - inOrder: false - distance: 10 - values: ["emergency", "broadcast", "fire"] - } - }) { - jsondata - } -} -``` - -#### **Sample Response 4: Word Proximity Search** - -```json -{ - "data": { - "searchMedia": { - "jsondata": { - "results": [ - { - "recording": { - "recordingId": "36744245", - "fileLocation": "https://inspirent.s3.amazonaws.com/recordings/1832ec2e-7342-41e5-4396-b0d681009e21_original.mp3?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAI7L6G7PCOOOLA7MQ%2F20171214%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20171214T211646Z&X-Amz-Expires=604800&X-Amz-Signature=6865eb270e11e9cae746230ed2247a2f336edbe3e7cde199af487319facf1c43&X-Amz-SignedHeaders=host", - "fileType": "audio/mpeg", - "programId": "26372", - "programName": "Morning Drive", - "programImage": "https://s3.amazonaws.com/prod-veritone-ugc/programs/26972/y3wVqN69SreXR6FREBqQ_WINS-AM.jpg", - "programLiveImage": "https://s3.amazonaws.com/prod-veritone-ugc/programs/25072/WpH0ipIYSzKA2Brvl0DX_BczwZNdTzmfmRbA3nTJA_MwvD0M4bRr2ELAcK8uQD_wins%2525201010.JPG", - "mediaSourceId": "32063", - "mediaSourceTypeId": "1", - "isPublic": true, - "sliceTime": 1503398731, - "mediaStartTime": 1503398701, - "aibDuration": 90, - "isOwn": false, - "hitStartTime": 1503398752, - "hitEndTime": 1503398754 - }, - "startDateTime": 1503398731, - "stopDateTime": 1503398791, - "hits": [ - { - "transcript": { - "source": "temporal", - "transcript": [ - { - "hits": [ - { - "queryTerm": "broadcast", - "startTime": 51.829, - "endTime": 52.51 - }, - { - "queryTerm": "fire", - "startTime": 52.579, - "endTime": 52.949 - }, - { - "queryTerm": "emergency", - "startTime": 53.44, - "endTime": 53.989 - } - ], - "startTime": 43.01, - "endTime": 72.479, - "text": "seriously in a train crash in suburban Philadelphia early today a SEPTA commuter train struck an unoccupied parked train at a terminal in Upper Darby broadcast a fire recorded the emergency call Act of one market Acura one market town on her you know . Her and want to honor him by the N.T.S.B. is trying to determine the cause we have mostly clear skies but some low clouds a little bit a haze around seventy five degrees heading for ninety" - } - ] - } - } - ] - } - ], - "totalResults": 95, - "limit": 1, - "from": 0, - "to": 0, - "searchToken": "17334140-e374-11e7-b0e6-a586c4474dec", - "timestamp": 1513286206 - } - } - } -} -``` - -### **Sample Request 5: Query String with Negation** - -The example below is a media search of the private index for the word Paris when it does not appear with Las Vegas. -```graphql -query{ - searchMedia(search:{ - offset: 0 - limit: 1 - index: ["mine"] - query:{ - operator: "query_string" - field: "transcript.transcript" - value: "paris NOT \"las vegas\"" - } - }) { - jsondata - } -} -``` - -#### **Sample Response 5: Query String with Negation** - -```json -{ - "data": { - "searchMedia": { - "jsondata": { - "results": [ - { - "recording": { - "recordingId": "36397022", - "fileLocation": "https://inspirent.s3.amazonaws.com/assets/36397022/17996ce6-1d2e-43c7-9398-d6f359f56d96.mp3?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAI7L6G7PCOOOLA7MQ%2F20171214%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20171214T212013Z&X-Amz-Expires=604800&X-Amz-Signature=b7459f5c243c4c6a28a803b083c27265a605fd2d3426ba47f945a423fef6839c&X-Amz-SignedHeaders=host", - "fileType": "audio/mp3", - "programId": "-1", - "programName": "Morning Chat", - "programLiveImage": "https://s3.amazonaws.com/veritone-ugc/temp%2FprogramLiveImageURL%2F74DgqpH2RMyu2i5MtSb5_Test(1).png", - "mediaSourceId": "-1", - "mediaSourceTypeId": "5", - "sliceTime": 1501005830, - "mediaStartTime": 1501005380, - "aibDuration": 510, - "isOwn": true, - "hitStartTime": 1501005841, - "hitEndTime": 1501005842 - }, - "startDateTime": 1501005830, - "stopDateTime": 1501005890, - "hits": [ - { - "transcript": { - "source": "temporal", - "transcript": [ - { - "hits": [ - { - "queryTerm": "Paris", - "startTime": 461.869, - "endTime": 462.159 - } - ], - "startTime": 461.869, - "endTime": 491.61, - "text": "Paris Jackson is in the upside Wow Wow That's great Naomi star airs Tuesdays on Fox and you can follow her on Twitter at Naomi Campbell thank you so much for calling in Naomi thank you SO MUCH THANK YOU HAVE A GREAT DAY AND . The first say fifty percent . At Macy's It feels good to give back together with the help generosity of our customers and employees we gave back fifty two million dollars to charities nationwide last year" - } - ] - } - } - ] - } - ], - "totalResults": 18071, - "limit": 1, - "from": 0, - "to": 0, - "searchToken": "92ab8800-7a14-11e7-8c04-bd019caca41d", - "timestamp": 1513286413 - } - } - } -} -``` - -### **Sample Request 6: Compound Query** - -The following example is a media search of both indexes for either of two faces and a 20th Century Fox logo or an occurrence of Fox Sports found in text. -```graphql -query{ - searchMedia(search:{ - offset: 0 - limit: 1 - index: ["global", "mine"] - query:{ - operator: "or" - conditions: [{ - operator: "terms" - field: "face-recognition.series.entityId" - values: ["3513c7da-3cde-4444-888e-ddd73e6d0cd9", "f34245b0-096b-4c3d-9646-6271d7d260d1"] - } - { - operator: "or" - conditions: [{ - operator: "term" - field: "logo-recognition.series.found" - value: "20th Century Fox" - }, - { - operator: "term" - field: "text-recognition.series.ocrtext" - value: "fox sports" - } - ] - } - ] - } -}) { - jsondata - } -} -``` - -#### **Sample Response 6: Compound Query** - -```json -{ - "data": { - "searchMedia": { - "jsondata": { - "results": [ - { - "recording": { - "recordingId": "43592937", - "fileLocation": "https://inspirent.s3.amazonaws.com/tv-recordings/17/12/14/6f0026ec-70a6-4ded-4a01-4c4d0ad4df75.mp4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAI7L6G7PCOOOLA7MQ%2F20171214%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20171214T212939Z&X-Amz-Expires=604800&X-Amz-Signature=059b209149eaf2e4111d56d2b33d0c804f40ca3789b26736db787ca9981d8f45&X-Amz-SignedHeaders=host", - "fileType": "video/mp4", - "programId": "29684", - "programName": "Weekday Afternoons", - "programImage": "https://s3.amazonaws.com/veritone-ugc/programs/25084/XbZ1Aj4JSHmrBNIuWUaW_200px-BBC_World_News_red.svg.png", - "programLiveImage": "https://s3.amazonaws.com/prod-veritone-ugc/programs/25084/qOyniPW3SrmTjVw7B1ow_4de7730beea8e20ba24de2be5ac03cb6.jpg", - "mediaSourceId": "39546", - "mediaSourceTypeId": "2", - "isPublic": true, - "sliceTime": 1513286310, - "mediaStartTime": 1513286100, - "aibDuration": 270, - "isOwn": false, - "hitStartTime": 1513286316, - "hitEndTime": 1513286317 - }, - "startDateTime": 1513286310, - "stopDateTime": 1513286370, - "hits": [ - { - "logo-recognition": { - "source": "bf5a703a-55f1-5a27-17a1-d213ae0c7f55", - "series": [ - { - "end": 217000, - "found": "20th Century Fox", - "salience": 0.5717288255691528, - "start": 216000 - } - ] - } - } - ] - } - ], - "totalResults": 4886, - "limit": 1, - "from": 0, - "to": 0, - "searchToken": "e3cc7680-e115-36e7-9e94-eba5f6b5faf7", - "timestamp": 1513286979 - } - } - } -} -``` - -### **Sample Request 7: Mentions Search Using the "and" Operator** - -The example below is a search for mentions with the name "Dallas Cowboys Super Bowl" and have a verification status of “Pending”. -```graphql -query{ - searchMentions(search:{ - offset: 0 - limit: 1 - index: ["mine"] - query:{ - operator: "and" - conditions: [{ - operator: "term" - field: "trackingUnitName" - value: "Dallas Cowboys Super Bowl" - }, - { - operator: "term" - field: "mentionStatusId" - value: "1" - }] - } - }) { - jsondata - } -} -``` - -#### **Sample Response 7: Mentions Search Using "and" Operator** - -```json -{ - "data": { - "searchMentions": { - "jsondata": { - "results": [ - { - "id": 47569938, - "programFormatName": "Information and News", - "mentionDate": "2017-01-31T07:59:18.000Z", - "mediaStartTime": "2017-01-31T07:45:01.000Z", - "mediaId": 20017455, - "metadata": { - "filename": "AM-RADIO", - "veritone-file": { - "size": 0, - "filename": "AM-RADIO", - "mimetype": "audio/mpeg" - }, - "veritone-media-source": { - "mediaSourceId": "14326", - "mediaSourceTypeId": "1" - }, - "veritone-program": { - "programId": "3828", - "programName": "AM-RADIO Morning Talk", - "programImage": "https://s3.amazonaws.com/veritone-ugc/cb5e59d4-a986-4e2b-b525-482319df3350%2FbrdProgram%2F2uGsLVKsQeiKN3UuHufC_941478_10151455644772706_951533539_n.jpg", - "programLiveImage": "https://s3.amazonaws.com/prod-veritone-ugc/cb5e59d4-a986-4e2b-b525-482319df3350%2FbrdProgram%2FwwEn3Ya9RgmMQwUEGoD1_LUkKlgZQS36ncUbY8Iz7_2520to%2520live2.JPG" - } - }, - "fileLocation": "https://inspirent.s3.amazonaws.com/recordings/9605ea97-87df-428e-6740-720df8b8691c_original.mp3?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAI7L6G7PCOOOLA7MQ%2F20171215%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20171215T205111Z&X-Amz-Expires=604800&X-Amz-Signature=00f62a6e2367c109320c98b9aea190cd28d82ac347eeeca030f42810b7ab75e3&X-Amz-SignedHeaders=host", - "fileType": "audio/mpeg", - "snippets": [ - { - "startTime": 857.62, - "endTime": 887.33, - "text": "eighty first women's Australian Open final Monica Seles beat Steffi Graf four six six three six two in one thousand nine hundred twenty eight the Dallas Cowboys beat the Buffalo Bills thirty to thirteen in Atlanta running back Emmitt Smith wins the M.V.P. and fourth consecutive Super Bowl game loss to the Cowboys twenty sixteen at the one hundred fourth women's Australian Open final six three six six four that time capsule your", - "hits": [ - { - "startTime": 865.7, - "endTime": 865.929, - "queryTerm": "Dallas" - }, - { - "startTime": 865.93, - "endTime": 866.07, - "queryTerm": "Cowboys" - }, - { - "startTime": 872.74, - "endTime": 873.31, - "queryTerm": "Super" - }, - { - "startTime": 873.31, - "endTime": 873.43, - "queryTerm": "Bowl" - } - ] - } - ], - "userSnippets": null, - "advertiserId": 0, - "advertiserName": "", - "brandId": 0, - "brandImage": "", - "brandName": "", - "campaignId": 0, - "campaignName": "", - "organizationId": 7295, - "organizationName": "Demo Organization", - "trackingUnitId": 10032, - "trackingUnitName": "Dallas Cowboys Super Bowl", - "mentionStatusId": 1, - "mediaSourceTypeId": 1, - "mediaSourceTypeName": "Audio", - "mediaSourceId": 14326, - "mediaSourceName": "AM-RADIO Morning Talk", - "isNational": true, - "spotTypeId": null, - "spotTypeName": null, - "programId": 3828, - "programName": "AM-RADIO", - "programImage": "https://s3.amazonaws.com/prod-veritone-ugc/cb5e59d4-a986-4e2b-b525-482319df3350%2FbrdProgram%2F2uGsLVKsQeiKN3UuHufC_941478_10151455644772706_951533539_n.jpg", - "programLiveImage": "https://s3.amazonaws.com/veritone-ugc/cb5e52b4-a986-4e2b-b525-482319df3350%2FbrdProgram%2FwwEn3Ya9RgmMQwUEGoD1_LUkKlgZQS36ncUbY8Iz7_2520to%2520live2.JPG", - "impressions": 1894, - "audience": [ - { - "gender": "men", - "age_group": "35-44", - "audience": 11, - "isTargetMatch": true - }, - { - "gender": "men", - "age_group": "45-49", - "audience": 121, - "isTargetMatch": true - }, - { - "gender": "men", - "age_group": "50-54", - "audience": 474, - "isTargetMatch": true - }, - { - "gender": "men", - "age_group": "65+", - "audience": 95, - "isTargetMatch": true - }, - { - "gender": "women", - "age_group": "50-54", - "audience": 19, - "isTargetMatch": false - }, - { - "gender": "women", - "age_group": "65+", - "audience": 693, - "isTargetMatch": false - }, - { - "gender": "men", - "age_group": "55-64", - "audience": 481, - "isTargetMatch": true - } - ], - "targetAudience": { - "gender": 1, - "genderName": "M", - "ageGroup": [ - 0, - 5 - ], - "ageGroupMin": 18, - "ageGroupMax": 0, - "impressions": 1182 - }, - "audienceMarketCount": 3, - "audienceAffiliateCount": 1, - "rating": null, - "ratings": null, - "comments": null, - "markets": [ - { - "marketId": 54, - "marketName": "Des Moines-Ames, IA" - } - ], - "marketId": null, - "marketName": null, - "hourOfDay": 7, - "dayOfWeek": 2, - "dayOfMonth": 31, - "month": 1, - "year": 2017, - "isMatch": true, - "mentionStatusName": "Pending Verification", - "complianceStatusId": null, - "cognitiveEngineResults": null, - "hits": 4 - } - ], - "totalResults": 579, - "limit": 1, - "from": 0, - "to": 0, - "searchToken": "ae882400-e1d9-11e7-947b-339cddca931e", - "timestamp": 1513371071 - } - } - } -} -``` -## Appendix 1: Fields - -The table below defines the possible values currently available for the *field* property. Fields are grouped by category and include a name, a type, and one or more possible values. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Field Name

Type

Description

Example

Recording
-Searches for various aspects of recordings in public and private indexes.
absoluteStartTimeMsintegerAbsolute start date and time in Unix/Epoch timestamp format.

-* Must use range operator.		operator: "range"
-field: "absoluteStartTimeMs"
-gte: "1497808410"
absoluteStopTimeMsintegerAbsolute stop date and time in Unix/Epoch timestamp format.

-· Must use range operator.		operator: "range"
-field: "absoluteStopTimeMs"
-gte: "1497808410"
recordingIdstringThe unique ID associated with the recording container.operator: "term"
-field: "recordingId"
-value: "39590300"
fileLocationstringThe URL of the file.operator: "term"
-field: "fileLocation"
-value: "https://inspirent.s3.amazonaws.com/assets/39070300/789eb7b2-6c4d-4ec4-8a79-f4600b7e31c1.mp4"
fileTypestringThe recording file MIME type.

-· Value must be entered as lowercase.		operator: "term"
-field: "fileType"
-value: "video/mp4"
relativeStartTimeMsintegerThe starting time of the recording in milliseconds.

-· Must use range operator.		operator: "range"
-field: "relativeStartTimeMs"
-lte: "1497808410"
relativeStopTimeMsintegerThe ending time of the recording in milliseconds.

-· Must use range operator.		operator: "range"
-field: "relativeStopTimeMs"
-lte: "1497808410"
isPublicBooleanA Boolean that searches the public index when set to true.operator: "term"
-field: "isPublic"
-value: true
programNamestringThe user-entered name of the program.

-· Must use query_string operator and the value must be entered as lowercase.		operator: "query_string"
-field: "programName"
-value: "this week in tech"
mediaSourceIdstringA UUID assigned by the system that identifies the media ingestion source.operator: "term"
-field: "mediaSourceId"
-value: "39262"
mediaSourceTypeIdstringThe ID associated with the media source type.operator: "term"
-field: "mediaSourceTypeId"
-value: "4"
Mention
-Searches for various aspects of mentions and watchlists in an organization’s private media index.
idstringThe unique ID associated with the mention.operator: "term"
-field: "id"
-value: "40392711"
programFormatNamestringThe radio format name as defined by Wikipedia/Radio\_Format.operator: "term"
-field: "programFormatName"
-value: "Sport"
mentionDatetype: date -format: date_ optional\_timeThe date and time the mention was created.

-· The date is mandatory and the time is optional.
-· UTC time zone.
-· Date format: "yyyy-mm-dd"
-· The time element "T" must follow the date.
-· Time format: hh/mm/ss/ms		operator: "range"
-field: "mentionDate"
-gte: "2017-04-13T20:09:09.000"
mediaStartTimetype: date -format: date_ optional_timeThe radio/TV broadcast time of the media.

-· The date is mandatory and the time is optional.
-· UTC time zone.
-· Date format: "yyyy-mm-dd"
-· The time element "T" must follow the date when adding a time to the value.
-· Time format: hh/mm/ss/ms
-· Uses the term or range operator.		operator: "range"
-field: "mediaStartTime"
-gte: "2017-04-13T20:09:09.000"

-operator: "term"
-field: "mediaStartTime"
-value: "2017-04-13T20:09:09.000"
mediaSourceIdstringThe ID of the media source.operator: "term"
-field: "mediaSourceId"
-value: "12734"
mediaSourceNamestringThe name of the media source.operator: "term"
-field: "mediaSourceName"
-value: "WPDH-FM"
mediaSourceTypeIdintegerThe ID of the media source type.

-Possible values:
-"1" (Broadcast Radio)
-"2" (Broadcast TV)
-"3" (YouTube)
-"4" (Podcast)
-"5" (Private Media)		operator: "term"
-field: "mediaSourceTypeId"
-value: "2"
metadata.veritone-program.primaryMediaSourceIdstringThe primary media source ID metadata. Uses the term or range operator.operator: "term"
-field: "metadata.veritone-program.primaryMediaSourceId"
-value: "21296"
metadata.veritone-program.programIdstringThe program ID metadata.operator: "term"
-field: "metadata.veritone-program.programId"
-value: "28224"
metadata.veritone-program.programNamestringThe program name metadata.operator: "term"
-field: "metadata.veritone-program.programName"
-value: "AM Afternoon News 4p6p"
veritone-cognitive-profile-paramsobjectThe search query parameters used to generate the mention.operator: "term"
-field: "veritone-cognitive-profile-params"
-value: "hello"
isNationalBooleanA Boolean that when set to true searches for programs that are national.operator: "term"
-field: "isNational"
-value: false
audience.genderstringThe gender of the segment.

-Possible values:
-"men"
-"women"		operator: "term"
-field: "audience.gender"
-value: "men"
audience.age_groupstringThe age group string for the segment.

-Possible values:
-"6-11"
-"12-17"
-"18-24"
-"25-34"
-"35-44"
-"45-49"
-"50-54"
-"55-64"
-"65+"		operator: "term"
-field: "audience.age_group"
-value: "45-49"
audience.audienceintegerThe number of impressions for the segment.operator: "range"
-field: "audience.audience"
-gt: "1100"
audience.isTargetMatchBooleanSpecifies whether the audience segment matches (true) or doesn’t match (false) the target audience.operator: "term"
-field: "audience.isTargetMatch"
-value: true
targetAudience.genderintegerThe target audience gender.

-Possible values:
-"1" (men)
-"2" (women)
-"3" (people)		operator: "term"
-field: "targetAudience.gender"
-value: "3"
targetAudience.genderNamestringThe gender code of the target audience.

-Possible values are as follows and must be capitalized:
-"M" (male)
-"F" (female)
-"P" (people)		operator: "term"
-field: "targetAudience.genderName"
-value: "F"
targetAudience.ageGroupintegerThe target audience age group range. Enter one or more of the possible values to search a target audience age group.

-Possible values:
-"0" (18-24)
-"1" (25-34)
-"2" (35-44)
-"3" (45-54)
-"4" (55-64)
-"5" (65+)		operator: "terms"
-field: "targetAudience.ageGroup"
-values: ["2", "4"]
targetAudience.ageGroupMinintegerThe lower bound of the target age group.operator: "term"
-field: "targetAudience.ageGroupMin"
-value: "25"
targetAudience.ageGroupMaxintegerThe upper bound of the target age group.operator: "term"
-field: "targetAudience.ageGroupMax"
-value: "54"
targetAudience.impressionsintegerThe total number of impressions in the target audience.operator: "range"
-field: "targetAudience.impressions"
-gt: "15000"
metadata.veritone-file.filenamestringThe name associated with the media file.operator: "term"
-field: "metadata.veritone-file.filename"
-value: "AM Afternoon News18"
metadata.veritone-file.mimetypestringThe file type metadata.operator: "term"
-field: "metadata.veritone-file.mimetype"
-value: "audio/mpeg"
metadata.veritone-file.size integerThe file size metadata. Uses term or range operator.operator: "range"
-field: "metadata.veritone-file.size"
-gte: "2000"
metadata.veritone-media-source.mediaSourceIdstringThe media source ID metadata.operator: "term"
-field: "metadata.veritone-media-source.mediaSourceId"
-value: "16390"
metadata.veritone-media-source.mediaSourceNamestringThe media source name.operator: "term"
-field: "metadata.veritone-media-source.mediaSourceName"
-value: "Morning Drive"
metadata.veritone-media-source.mediaSourceTypeIdintegerThe identifier of the media source type.

-Possible values:
-"1" (Broadcast Radio)
-"2" (Broadcast TV)
-"3" (YouTube)
-"4" (Podcast)
-"5" (Private Media)		operator: "term"
-field: "metadata.veritone-media-source.mediaSourceTypeId"
-value: "1"
fileTypestringThe file type of the mention.operator: "term"
-field: "fileType"
-value: "audio/mpeg"
snippets.hits.startTimeintegerThe start time of the snippet hit.

-· Uses term or range operator.		operator: "term"
-field: "snippets.hits.startTime"
-value: "824.73"
snippets.hits.endTimeintegerThe ending time of the snippet hit.

-· Uses term or range operator.		operator: "term"
-field: "snippets.hits.endTime"
-value: "869.95"
snippets.hits.queryTermstringA word to search for in the text of the mention hit.

-· Value must be all lowercase.		operator: "term"
-field: "snippets.hits.queryTerm"
-value: "nfl"
snippets.startTimeintegerThe starting time of the snippet.

-· Uses term or range operator.		operator: "term"
-field: "snippets.startTime"
-value: "819.839"
snippets endTimeintegerThe ending time of the snippet.

-· Uses term or range operator.		operator: "term"
-field: "snippets.endTime"
-value: "840.16"
snippets.textstringOne or more words from the snippet text.

-· Value must be all lowercase.
-· Uses query_string or term operator.		operator: "query_string"
-field: "snippets.text"
-value: "team will go on to victory"

-operator: "term"
-field: "snippets.text"
-value: "team"
userSnippets.startTimeintegerThe start time of the user-edited mention text.

-· Uses term or range operator.		operator: "term"
-field: " userSnippets startTime"
-value: "78"

-operator: "range"
-field: " userSnippets startTime"
-gte: "78"
userSnippets.endTimeintegerThe end time of the user-edited mention text.

-· Uses term or range operator.		operator: "term"
-field: "userSnippets endTime"
-value: "78"

-operator: "range"
-field: "userSnippets endTime"
-lte: "179"
userSnippets.snippets.textstringThe user-edited mention text.

-· Value must be all lowercase.		operator: "query_string"
-field: "userSnippets.snippets.text"
-value: "team will"

-operator: "term"
-field: " userSnippets.snippets.text"
-value: "team"
organizationIdstringThe unique ID associated with the organization.operator: "term"
-field: "organizationId"
-value: "7685"
organizationNamestringThe name of the organization.operator: "term"
-field: "organizationName"
-value: "ABC Organization"
trackingUnitIdstringThe unique ID of the mention.

-· Uses term or range operator.		operator: "term"
-field: "trackingUnitId"
-value: "20068"
trackingUnitNamestringThe user-entered name of the mention. Values for this field are case sensitive and must be entered as they are saved in the index. -operator: "term"
-field: "trackingUnitName"
-value: "Dallas Cowboys Super Bowl"
mentionStatusIdintegerThe status of the mention’s verification.

-Possible values:
-"1" (Pending Verification)
-"2" (Verified)
-"3" (Needs Review)
-"4" (Request Bonus)
-"5" (Invalid)
-"6" (Processing Verification)
-"7" (Auto Verified)		operator: "term"
-field: "mentionStatusId"
-value: "2"
mediaSourceTypeIdintegerThe media source type identifier.

-Possible values:
-"1" (Broadcast Radio)
-"2" (Broadcast TV)
-"3" (YouTube)
-"4" (Podcast)
-"5" (Private Media)		operator: "term"
-field: "mediaSourceTypeId"
-value: "2"
mediaSourceTypeNamestringThe media source name.

-Possible values:
-"Audio"
-"Broadcast TV"
-"Podcast"
-"YouTube"
-"Private Media"		operator: "term"
-field: "mediaSourceTypeName"
-value: "Audio"
mediaSourceIdstringThe ID of the media source.

-· Uses term or range operator.		operator: "term"
-field: "mediaSourceId"
-value: "38674"
mediaSourceNamestringThe user-defined program name.operator: "term"
-field: "mediaSourceName"
-value: "The Weekly Podcast"
spotTypeIdintegerThe ID of the spot type:

-Possible values:
-"17" (Bonus)
-"15" (Live :05)
-"16" (Live :10)
-"1" (Live :15)
-"2" (Live :30)
-"3" (Live :60)
-"14" (MicroMention)
-"9" (Non Voiced :05)
-"10" (Non Voiced :10)
-"11" (Non Voiced :15)
-"12" (Non Voiced :30)
-"13" (Non Voiced :60)
-"4" (Voiced :05)
-"5" (Voiced :10)
-"6" (Voiced :15)
-"7" (Voiced :30)
-"8" (Voiced :60)		operator: "term"
-field: "spotTypeId"
-value: "17"
spotTypeNamestringThe name of the spot type.

-Possible values:
-"Bonus"
-"Live :05"
-"Live :10"
-"Live :15"
-"Live :30"
-"Live :60"
-"MicroMention"
-"Non Voiced :05"
-"Non Voiced :10"
-"Non Voiced :15"
-"Non Voiced :30"
-"Non Voiced :60"
-"Voiced :05"
-"Voiced :10"
-"Voiced :15"
-"Voiced :30"
-"Voiced :60"		operator: "term"
-field: "spotTypeName"
-value: "Bonus"
programIdstringThe unique ID associated with the program.operator: "term"
-field: "programId"
-value: "22979"
programName.rawstringThe user-defined name of the program.operator: "term"
-field: "programName.raw"
-value: "Morning Drive - MF 10a1p"
impressionsintegerThe audience count.operator: "term"
-field: "impressions"
-value: "8"
audienceMarketCountintegerThe number of markets where the media was broadcast.operator: "term"
-field: "audienceMarketCount"
-value: "2"
audienceAffiliateCountintegerThe number of affiliates for the primary media source.operator: "term"
-field: "audienceAffiliateCount"
-value: "1"
ratingfloatThe number of user-given stars to a watchlist mention.

-Possible values:
-"1"
-"2"
-"3"
-"4"
-"5"		operator: "term"
-field: "rating"
-value: "5"
ratings.rating_idstringThe unique ID of the mention rating.operator: "term"
-field: "ratings.rating_id"
-value: "38387"
ratings.rating_valueintegerThe number user-given stars to a watchlist mention.

-Possible values:
-"1"
-"2"
-"3"
-"4"
-"5"		operator: "term"
-field: "ratings.rating_value"
-value: "5"
ratings.date\_createdtype: date -format: date_ optional_timeThe date and time that the mention rating was created.

-· The date is mandatory and the time is optional.
-· UTC time zone
-· Date format: "yyyy-mm-dd"
-· The time element "T" must follow the date when adding a time to the value.
-· Time format: hh/mm/ss/ms
-· Uses the term, query_string or range operators.		operator: "range"
-field: "ratings.date_created"
-lte: "2017-10-13T00:12:34.44948"
comments.idstringThe unique ID of the comment.operator: "term"
-field: "comments.id"
-value: "43770"
comments.textstringText entered as the mention comment.

-· Value must be entered as all lowercase.
-· Uses both the term and query_string operators.		operator: "query_string"
-field: "comments.text"
-value: "provides information"
comments.date\_modifiedtype: date -format: date_ optional_timeThe date and time that the comment was created.

-· The date is mandatory and the time is optional.
-· UTC time zone.
-· Date format: "yyyy-mm-dd"
-· The time element "T" must follow the date when adding a time to the value.
-· Time format: hh/mm/ss/ms
-· Uses the term, query_string or range operators.		operator: "range"
-field: "comments.date_modified"
-lte: "2017-10-20T01:43:35.0909"
markets.marketIdintegerThe market ID to use as a filter when searching for mentions. (See "Appendix 2: Markets" for a list of all market name values.)

-· Uses term and range operators.		operator: "term"
-field: "markets.marketId"
-value: "112"
markets.marketNamestringThe market name to use as a filter when searching for mentions. (See "Appendix 2: Markets" for a list of all market name values.)operator: "term"
-field: "markets.marketName"
-value: "Los Angeles, CA"
marketIdintegerThe market ID to use as a filter when searching for mentions. (See "Appendix 2: Markets" for a list of all market name values.)

-· Uses term and range operators.		operator: "term"
-field: "marketId"
-value: "112"
marketNamestringThe market name to use as a filter when searching for mentions. (See "Appendix 2: Markets" for a list of all market name values.)operator: "term"
-field: "marketName"
-value: "Los Angeles, CA"
hourOfDayintegerThe hour of day (24-hour format) to use as a filter when searching for mentions.operator: "term"
-field: "hourOfDay"
-value: "8"
dayOfWeekintegerThe numeric value assigned to a day of the week to use as a filter when searching for mentions.

-· Uses the range and term operators.

-Possible values:
-"1" (Monday)
-"2" (Tuesday)
-"3" (Wednesday)
-"4" (Thursday)
-"5" (Friday)
-"6" (Saturday)
-"7" (Sunday)		operator: "term"
-field: "dayOfWeek"
-value: "4"
dayOfMonthintegerThe numeric value assigned to a day of the month to use as a filter when searching for mentions.

-· Uses the range and term operators.		operator: "term"
-field: "dayOfMonth"
-value: "13"
monthintegerThe numeric value assigned to a calendar month to use as a filter when searching for mentions.

-· Uses the range and term operators.		operator: "term"
-field: "month"
-value: "4"
yearintegerA four-digit year to use as a filter when searching for mentions.

-· Uses the range and term operators.		operator: "range"
-field: "month"
-gte: "2015"
mentionStatusNamestringA mention status to use as a filter when searching for mentions.

-· Values are case sensitive and must use initial caps.

-Possible values:
-"Pending Verification"
-"Auto Verified"
-"Invalid"
-"Needs Review"
-"Processing Verification"
-"Request Bonus"
-"Verified"		operator: "term"
-field: "mentionStatusName"
-value: "Verified" -
Face Recognition
-Searches for records generated by a facial engine and other detected face attributes.
face-recognition.sourcestringThe ID of the face recognition engine to use as a filter when searching. -For a list of possible values, make a request to the Engines query. Pass "category" as an argument with "facial" as the value, and specify "id" and "name" as return fields.operator: "term"
-field: "face-recognition.source"
-value: "84b513bd-d64d-3a35-9d42-579c8611fdbf"
face-recognition.series.entityIdstringThe unique ID associated with the library entity (such as a person or an organization). -operator: "term"
-field: "face-recognition.series.entityId"
-value: "423aa55e-6135-4f10-9585-f65a1f3d909e"
face-recognition.series.startstringThe time in milliseconds from the beginning of the file where the fingerprint began.

-· Uses range and term operators.		operator: "value"
-field: "face-recognition.series.start"
-lte: "54000"
face-recognition.series.endstringThe time in milliseconds from the beginning of the file where the fingerprint ended.

-· Uses range and term operators.		operator: "value"
-field: "face-recognition.series.end"
-gte: "55000"
face-recognition.series.libraryIdstringThe unique ID associated with the library.operator: "term"
-field: "face-recognition.series.libraryId"
-value: "0fb02432-dcb1-40b0-bb6a-3f7e481aae3e"
face-recognition.series.originalImagestringThe URL of the library cover image.operator: "term"
-field: "face-recognition.series.originalImage"
-value: "https://s3-us-west-1.amazonaws.com/prod-veritone-face/aed3ce42-97d0-46f1-89f9-48df4ba0ace2/35914952/318af5mf.983887/setId-0_frame-792.png"
Fingerprint
-Searches for records generated by a fingerprint engine and other detected fingerprint attributes.
fingerprint.sourcestringThe ID of the fingerprint engine to use as a filter when searching.

-For a list of possible values, make a request to the Engines query. Pass "category" as an argument with "fingerprint" as the value, and specify "id" and "name" as return fields.		operator: "term"
-field: "fingerprint.source"
-value: "c5eff1ad-d53e-8bb9-fb1f-1866da84d0b3"
fingerprint.series.startintegerThe time in milliseconds from the beginning of the file where the fingerprint began.

-· Uses range and term operators.		operator: "range"
-field: "fingerprint.series.start"
-lte: "22000"
fingerprint.series.endintegerThe time in milliseconds from the beginning of the file where the fingerprint ended.

-· Uses range and term operators.		operator: "range"
-field: "fingerprint.series.end"
-gte: "21000"
fingerprint.series.entityIdstringThe unique Entity ID associated with the fingerprint.operator: "term"
-field: "fingerprint.series.entityId"
-value: "e49c5344-622b-42bb-8e82-514d96d2df88"
fingerprint.series.entityIdentifierIdstringThe unique Entity Identifier ID associated with the fingerprint.operator: "term"
-field: "fingerprint.series.entityIdentifierId"
-value: "e4f95339-622b-42bb-8e82-514d96d2df88"
fingerprint.series.libraryIdstringThe Library ID associated with the fingerprint.operator: "term"
-field: "fingerprint.series.libraryId"
-value: "c034a463-2a82-4be5-b263-61f3b61e5282"
fingerprint.series.scorefloatA probability score that indicates the similarity between features in a fingerprint sample and a stored entity. This score can be used as a filter to prioritize more salient fingerprints when searching. Scores closer to 0.0 are less similar, while scores closer to 1.0 are high in similarity.

-· Uses range and term operators.		operator: "range"
-field: "fingerprint.series.score"
-gte: ".055"
Logo Recognition
-Searches for records generated by a logo engine and other detected logo attributes.
logo-recognition.sourcestringThe ID of the logo recognition engine to use as a filter when searching.

-For a list of possible values, make a request to the Engines query. Pass "category" as an argument with "logo recognition" as the value, and specify "id" and "name" as return fields.		operator: "term"
-field: "logo-recognition.source"
-value: "bf50303a-55f1-5a27-17a1-d213ae0c7f55"
logo-recognition.series.foundstringThe name of the detected logo. -· Case-sensitive – value must match indexed logo name.operator: "query_string"
-field: "logo-recognition.series.found"
-value: "ESPN"
logo-recognition.series.startintegerThe time that the logo was first identified in milliseconds from the beginning of the recording.

-· Uses range operator only.		operator: "range"
-field: "logo-recognition.series.start"
-gte: "20700"
logo-recognition.series.endintegerThe time that the logo was last identified in milliseconds from the beginning of the recording.

-· Uses range operator only.		operator: "range"
-field: "logo-recognition.series.end"
-lte: "20800"
Object Recognition
-Searches for records generated by an object recognition engine and various attributes of detected objects.
object-recognition.sourcestringThe ID of the object recognition engine to use as a filter when searching.

-For a list of possible values, make a request to the Engines query. Pass "category" as an argument with "object detection" as the value, and specify "id" and "name" as return fields.		operator: "term"
-field: "object-recognition.source"
-value: "070148f0-3fd0-1bdc-b412-1e79b8aa36a2"
object-recognition.series.boundingPoly.xstringThe "x" coordinate of the object found on the screen.

-· Uses range operator.		operator: "range"
-field: "object-recognition.series.boundingPoly.x"
-lte: "418"
object-recognition.series.boundingPoly.ystringThe "y" coordinate of the object found on the screen.

-· Uses range operator.		operator: "range"
-field: "object-recognition.series.boundingPoly.y"
-gte: "269"
object-recognition.series.foundstringA label for the detected object. (e.g., person, laptop)operator: "term"
-field: "object-recognition.series.found"
-value: "girl"
object-recognition.series.startstringThe time in milliseconds from the beginning of the file where the object was first identified.

-· Uses range and term operators.		operator: "range"
-field: "object-recognition.series.start"
-gte: "59000"
object-recognition.series.endstringThe time in milliseconds from the beginning of the file where the object was last identified.

-· Uses range and term operators.		operator: "range"
-field: "object-recognition.series.end"
-lte: "60000"
object-recognition.series.saliencyfloatA probability score that measures the similarity of a recognized object to a reference model. This score can be used as a filter to prioritize more salient objects when searching. Scores closer to 0.0 are less similar, while scores closer to 1.0 are high in similarity.

-· Uses range and term operators.		operator: "range"
-field: "object-recognition.series.saliency"
-gte: "80"
OCR (Text Recognition)
-Searches for records generated by an OCR engine and performs full-text search for various attributes of recognized text. All queries should use the query_string operator.
text-recognition.sourcestringThe ID of the text recognition engine to use as a filter when searching.

-For a list of possible values, make a request to the Engines query. Pass "category" as an argument with "text recognition" as the value, and specify "id" and "name" as return fields.		operator: "term"
-field: "text-recognition.source"
-value: "9a6ac62d-a881-8884-6ee0-f15ab84fcbe2"
text-recognition.series.startintegerThe time in milliseconds from the beginning of the file where the text was first recognized.

-· Uses range and term operators.		operator: "range"
-field: "text-recognition.series.start"
-gte: "544000"
text-recognition.series.endintegerThe time in milliseconds from the beginning of the file where the text was last recognized.

-· Uses range and term operators.		operator: "range"
-field: "text-recognition.series.end"
-lte: "545000"
text-recognition.series.ocrtextstringOne or more identified words displayed as text.

-· Value must be all lowercase.
-· Uses term and query_string operators.		operator: "term"
-field: "text-recognition.series.ocrtext"
-value: "diego"
Sentiment
-Searches for records generated by a sentiment engine and other attributes for detected sentiment.
sentiment-veritone.sourcestringThe ID of the sentiment engine to use as a filter when searching.

-For a list of possible values, make a request to the Engines query. Pass "category" as an argument with "sentiment" as the value, and specify "id" and "name" as return fields.		operator: "term"
-field: "sentiment-veritone.source"
-value: "104a6b61-5d7d-8a1c-fa32-d37b4c931e7c"
sentiment-veritone.series.startstringThe time in milliseconds from the beginning of the file where the sentiment was first detected.

-· Uses range and term operators.		operator: "range"
-field: "sentiment-veritone.series.start"
-lte: "84000"
sentiment-veritone.series.endstringThe time in milliseconds from the beginning of the file where the sentiment was last detected.

-· Uses range and term operators.		operator: "range"
-field: "sentiment-veritone.series.start"
-gte: "844690"
sentiment-veritone.series.scorefloatA probability score that measures the similarity of a person’s sentiment to a reference model. This score can be used as a filter to prioritize more salient instances when searching. Scores closer to 0.0 are less similar, while scores closer to 1.0 are high in similarity. The value must include 0 before the decimal.

-· Uses range and term operators.		operator: "range"
-field: "sentiment-veritone.series.start"
-gte: 0.81
Transcript
-Searches for records generated by a transcription engine and words/phrases in a transcript.
transcript.sourcestringThe ID of the transcription engine to use as a filter when searching.

-For a list of possible values, make a request to the Engines query. Pass "category" as an argument with "transcription" as the value, and specify "id" and "name" as return fields.		operator: "term"
-field: "transcript.source"
-value: "818aa3da-0123-33a4-8c5b-2bc3e569c52a"
transcript.transcriptstringThis field is used to find keywords or phrases in transcription text.

-· Value must be entered as lowercase. -		operator: "query_string"
-field: "transcript.transcript"
-value: "so you know atom"
Tags
-Searches for user-created tags. This field’s values are case-sensitive and must be entered exactly as they appear in Veritone to return a match.
tags.key -stringThis field is part of a key value pair (using the query object type and value field below) in a complex query.operator: "term"
-field: "tags.key"
-value: "host"
tags.valuestringThis field can be used in a complex query as part of a key value pair (using the query object type and key field above) or as a stand-alone in a simple query.operator: "term"
-field: "tags.value"
-value: "Ellen"
tags.displayNamestringThis field is used in simple queries to find a tag by name.operator: "term"
-field: "tags.displayName"
-value: "Adam Levine"
Veritone Job
-Search options for various aspects of tasks run on a recording.
veritone-job.insert-into-indexbooleanSearches for tasks that were indexed when set to true.operator: "term"
-field: "veritone-job.insert-into-index"
-value: true
veritone-job.mention-generatebooleanSearches for tasks that generated a mention when set to true.operator: "term"
-field: "veritone-job.mention-generate"
-value: true
veritone-job.statusstringThe status of the job as a whole.

-Possible values are:
-· accepted: The job is valid and has been accepted for processing.
-· running: The job is running.
-· failed: At least one task failed and execution was terminated.
-· complete: All tasks in the job finished normally.		operator: "term"
-field: "veritone-job.status"
-value: "complete"
veritone-job.{engineId}booleanReturns jobs for the stated engine ID when set to true.

-For a list of possible engine ID values, call the Engines query. Pass "records" as a request parameter and "id" and "name" as return fields.		operator: "term"
-field: "veritone-job.imagedetection-facerecognition-veritone"
-value: true
Veritone File
-Full text search for various attributes of files imported to CMS.
veritone-file.filenamestringThe name of the file. Value must be input as all lowercase.operator: "term"
-field: "veritone-file.filename"
-value: "morning drive 15"
veritone-file.mimetypestringThe file type. Value must be input as all lowercase.operator: "term"
-field: "veritone-file.mimetype"
-value: "video/mp4"
veritone-file.sizelongThe size of the file.operator: "term"
-field: "veritone-file.size"
-value: 0
Veritone Public
-Searches various aspects of nationally broadcasted programs.
veritone-public.isNationalbooleanSpecifies whether a program is national when set to true.operator: "term"
-field: "veritone-public.isNational"
-value: true
veritone-public.marketCountintegerSearches for a specific number of markets.operator: "term"
-field: "veritone-public.marketCount"
-value: "3"
veritone-public.marketsstringSearches for recordings in specified markets. Refer to "Appendix 2: Markets" for a list of possible values.operator: "terms"
-field: "veritone-public.markets"
-values: ["88", "50", "167"]
veritone-public.primaryMarketstringSearches for recordings in the specified primary market. Refer to "Appendix 2: Markets" for a list of possible valuesoperator: "term"
-field: "veritone-public.primaryMarket"
-value: "167"
veritone-public.primaryStatestringSearches for recordings in the specified primary US state. Enter values using the two-letter state abbreviation.operator: "term"
-field: "veritone-public.primaryState"
-value: "CA"
veritone-public.stateCountintegerSearches for a specific number of states.operator: "term"
-field: "veritone-public.stateCount"
-value: "3"
veritone-public.statesstringSearches for recordings in the specified US states. Enter values using the two-letter state abbreviation.operator: "terms"
-field: "veritone-public.states"
-values: ["CA", "NY", "WA"]
- -## Appendix 2: Markets - -The following list provides the possible market ID values for the *veritone-public.markets* and *veritone-public.primaryMarket* fields in Appendix 1. - - - - - - - - - - -
Market ID/Market NameMarket ID/Market Name
"1" "Abilene-Sweetwater, TX"
-"2" "Albany-Schenectady-Troy, NY"
-"3" "Albany, GA"
-"4" "Albuquerque-Santa Fe, NM"
-"5" "Alexandria, LA"
-"6" "Alpena, MI"
-"7" "Amarillo, TX"
-"8" "Anchorage, AK"
-"9" "Atlanta, GA"
-"10" "Augusta, GA"
-"11" "Austin, TX"
-"12" "Bakersfield, CA"
-"13" "Baltimore, MD"
-"14" "Bangor, ME"
-"15" "Baton Rouge, LA"
-"16" "Beaumont-Port Arthur, TX"
-"17" "Bend, OR"
-"18" "Billings, MT"
-"19" "Biloxi-Gulfport, MS"
-"20" "Binghamton, NY"
-"21" "Birmingham, AL"
-"22" "Bluefield-Beckley-Oak Hill, WV"
-"23" "Boise, ID"
-"24" "Boston, MA"
-"25" "Bowling Green, KY"
-"26" "Buffalo, NY"
-"27" "Burlington, VT-Plattsburgh, NY"
-"28" "Butte-Bozeman, MT"
-"29" "Casper-Riverton, WY"
-"30" "Cedar Rapids-Waterloo-Iowa City-Dubuque, IA"
-"31" "Champagne-Springfield-Decatur, IL"
-"32" "Charleston-Huntington, WV"
-"33" "Charleston, SC"
-"34" "Charlotte, NC"
-"35" "Charlottesville, VA"
-"36" "Chattanooga, TN"
-"37" "Cheyenne, WY-Scottsbluff, NE"
-"38" "Chicago, IL"
-"39" "Chico-Redding, CA"
-"40" "Cincinnati, OH"
-"41" "Clarksburg-Weston, WV"
-"42" "Cleveland-Akron, OH"
-"43" "Colorado Springs-Pueblo, CO"
-"44" "Columbia-Jefferson City, MO"
-"45" "Columbia, SC"
-"46" "Columbus-Tupelo-West Point, MS"
-"47" "Columbus, GA"
-"48" "Columbus, OH"
-"49" "Corpus Christi, TX"
-"50" "Dallas-Ft. Worth, TX"
-"51" "Davenport, IA-Rock Island-Moline, IL"
-"52" "Dayton, OH"
-"53" "Denver, CO"
-"54" "Des Moines-Ames, IA"
-"55" "Detroit, MI"
-"56" "Dothan, AL"
-"57" "Duluth, MN-Superior, WI"
-"58" "El Paso, TX"
-"59" "Elmira, NY"
-"60" "Erie, PA"
-"61" "Eugene, OR"
-"62" "Eureka, CA"
-"63" "Evansville, IN"
-"64" "Fairbanks, AK"
-"65" "Fargo-Valley City, ND"
-"66" "Flint-Saginaw-Bay City, MI"
-"67" "Fresno-Visalia, CA"
-"68" "Ft. Myers-Naples, FL"
-"69" "Ft. Smith-Fayetteville-Springdale-Rogers, AR"
-"70" "Ft. Wayne, IN"
-"71" "Gainesville, FL"
-"72" "Glendive, MT"
-"73" "Grand Junction-Montrose, CO"
-"74" "Grand Rapids-Kalamazoo-Battle Creek, MI"
-"75" "Great Falls, MT"
-"76" "Green Bay-Appleton, WI"
-"77" 8Greensboro-High Point-Winston Salem, NC"
-"78" "Greenville-New Bern-Washington, NC"
-"79" "Greenville-Spartanburg, SC-Asheville, NC"
-"80" "Greenwood-Greenville, MS"
-"81" "Harlingen-Weslaco-Brownsville-McAllen, TX"
-"82" "Harrisburg-Lancaster-Lebanon-York, PA"
-"83" "Harrisonburg, VA"
-"84" "Hartford-New Haven, CT"
-"85" "Hattiesburg-Laurel, MS"
-"86" "Helena, MT"
-"87" "Honolulu, HI"
-"88" "Houston, TX"
-"89" "Huntsville-Decatur-Florence, AL"
-"90" "Idaho Falls-Pocatello, ID"
-"91" "Indianapolis, IN"
-"92" "Jackson, MS"
-"93" "Jackson, TN"
-"94" "Jacksonville, FL"
-"95" "Johnstown-Altoona, PA"
-"96" "Jonesboro, AR"
-"97" "Joplin, MO-Pittsburg, KS"
-"98" "Juneau, AK"
-"99" "Kansas City, KS-MO"
-"100" "Knoxville, TN"
-"101" "La Crosse-Eau Claire, WI"
-"102" "Lafayette, IN"
-"103" "Lafayette, LA"
-"104" "Lake Charles, LA"
-"105" "Lansing, MI"
-"106" "Laredo, TX"
-"107" "Las Vegas, NV"		"108" "Lexington, KY"
-"109" "Lima, OH"
-"110" "Lincoln-Hastings-Kearney, NE"
-"111" "Little Rock-Pine Bluff, AR"
-"112" "Los Angeles, CA"
-"113" "Louisville, KY"
-"114" "Lubbock, TX"
-"115" "Macon, GA"
-"116" "Madison, WI"
-"117" "Mankato, MN"
-"118" "Marquette, MI"
-"119" "Medford-Klamath Falls, OR"
-"120" "Memphis, TN"
-"121" "Meridian, MS"
-"122" "Miami – Ft. Lauderdale, FL"
-"123" "Milwaukee, WI"
-"124" "Minneapolis – St. Paul, MN"
-"125" "Minot-Bismarck-Dickinson, ND"
-"126" "Missoula, MT"
-"127" "Mobile, AL-Pensacola, FL"
-"128" "Monroe, LA-El Dorado, AR"
-"129" "Monterey-Salinas, CA"
-"130" "Montgomery, AL"
-"131" "Myrtle Beach-Florence, SC"
-"132" "Nashville, TN"
-"133" "New Orleans, LA"
-"134" "New York, NY"
-"135" "Norfolk-Portsmouth-Newport News, VA"
-"136" "North Platte, NE"
-"137" "Odessa-Midland, TX"
-"138" "Oklahoma City, OK"
-"139" "Omaha, NE"
-"140" "Orlando-Daytona Beach-Melbourne, FL"
-"141" "Ottumwa, IA-Kirksville, MO"
-"142" "Paducah-Cape Girardeau-Harrisburg-Mt Vernon"
-"143" "Palm Springs, CA"
-"144" "Panama City, FL"
-"145" "Parkersburg, WV"
-"146" "Peoria-Bloomington, IL"
-"147" "Philadelphia, PA"
-"148" "Phoenix, AZ"
-"149" "Pittsburgh, PA"
-"150" "Portland-Auburn, ME"
-"151" "Portland, OR"
-"152" "Presque Isle, ME"
-"153" "Providence, RI-New Bedford, MA"
-"154" "Quincy, IL-Hannibal, MO-Keokuk, IA"
-"155" "Raleigh-Durham, NC"
-"156" "Rapid City, SD"
-"157" "Reno, NV"
-"158" "Richmond-Petersburg, VA"
-"159" "Roanoke-Lynchburg, VA"
-"160" "Rochester, MN-Mason City, IA-Austin, MN"
-"161" "Rochester, NY"
-"162" "Rockford, IL"
-"163" "Sacramento-Stockton-Modesto, CA"
-"164" "Salisbury, MD"
-"165" "Salt Lake City, UT"
-"166" "San Angelo, TX"
-"167" "San Antonio, TX"
-"168" "San Diego, CA"
-"169" "San Francisco-Oakland-San Jose, CA"
-"170" "Santa Barbara-Santa Maria-San Luis Obispo,"
-"171" "Savannah, GA"
-"172" "Seattle-Tacoma, WA"
-"173" "Sherman, TX – Ada, OK"
-"174" "Shreveport, LA"
-"175" "Sioux City, IA"
-"176" "Sioux Falls-Mitchell, SD"
-"177" "South Bend-Elkhart, IN"
-"178" "Spokane, WA"
-"179" "Springfield-Holyoke, MA"
-"180" "Springfield, MO"
-"181" "St. Joseph, MO"
-"182" "St. Louis, MO"
-"183" "Syracuse, NY"
-"184" "Tallahassee, FL-Thomasville, GA"
-"185" "Tampa-St Petersburg-Sarasota, FL"
-"186" "Terre Haute, IN"
-"187" "Toledo, OH"
-"188" "Topeka, KS"
-"189" "Traverse City-Cadillac, MI"
-"190" "Tri-Cities, TN-VA"
-"191" "Tucson, AZ"
-"192" "Tulsa, OK"
-"193" "Twin Falls, ID"
-"194" "Tyler-Longview, TX"
-"195" "Utica, NY"
-"196" "Victoria, TX"
-"197" "Waco-Temple-Bryan, TX"
-"198" "Washington, DC"
-"199" "Watertown, NY"
-"200" "Wausau-Rhinelander, WI"
-"201" "West Palm Beach-Ft. Pierce, FL"
-"202" "Wheeling, WV- Steubenville, OH"
-"203" "Wichita – Hutchinson, KS"
-"204" "Wichita Falls, TX –Lawton, OK"
-"205" "Wilkes Barre-Scranton, PA"
-"206" "Wilmington, NC"
-"207" "Yakima-Pasco-Richland-Kennewick, WA"
-"208" "Youngstown, OH"
-"209" "Yuma, AZ-El Centro, CA"
-"210" "Zanesville, OH"
-"211" "Adelaide, SA"
-"212" "Melbourne, VIC"
-"213" "Brisbane, QLD"
diff --git a/docs/apis/tutorials/README.md b/docs/apis/tutorials/README.md deleted file mode 100644 index fd9d22e125..0000000000 --- a/docs/apis/tutorials/README.md +++ /dev/null @@ -1,58 +0,0 @@ - - -# Tutorials - -## Featured - -
-cognition -

-How to Train a Cognitive Engine
In aiWARE, training is practically an automatic process. But first, you need to know how to set up libraries of learnable entities. That part's easy — and fun, too! Follow along as we lead you through the GraphQL queries for doing facial recognition. -
-
-


- -
-cognition -

-How to Build Your Own Cognitive Engine
Would you believe it takes only 25 lines of JavaScript to create a Hello World cognitive engine using NodeJS? Learn how to create engine builds with Docker, test them locally, and onboard your creation to Veritone's aiWARE platform. It's easy! -
-
-
- -## The Full List - -* [GraphQL API Basics](apis/tutorials/graphql-basics.md) -* [Build Your Own AI App](developer/applications/app-tutorial/) -* [Build Your Own Cognitive Engine](developer/engines/tutorial/) -* [Customizing Engine Output](developer/engines/tutorial/customizing-engine-output) -* [Customizing Engine Input](developer/engines/tutorial/engine-custom-fields) -* [How to Train a Cognitive Engine](developer/engines/tutorial/engine-training-tutorial) -* [Clean up TDO data](apis/tutorials/cleanup-tdo.md) -* [Creating Export Requests](apis/tutorials/create-export-request/) -* [Posting Engine Results](apis/tutorials/engine-results.md) -* [Lookup Available Engines](apis/tutorials/get-engines.md) -* [Error Handling in the GraphQL API](apis/tutorials/graphql-error-handling.md) -* [Uploading and Processing Files](apis/tutorials/upload-and-process.md) -* [Uploading Large Files](apis/tutorials/uploading-large-files.md) -* [Handling File Upload Errors](apis/tutorials/file-upload-error-handling.md) -* [Authentication and Authorization Tokens](apis/tutorials/tokens.md) -* [Paging](apis/tutorials/paging.md) -* [Asset Types](apis/tutorials/asset-types.md) -* [API Examples](apis/examples.md) — GraphQL-based API tips and tricks -* [Job Quickstart Guide](apis/job-quickstart/) -* [Search Quickstart Guide](apis/search-quickstart/) - -_This page changes frequently, so be sure to bookmark it and come back often!_ \ No newline at end of file diff --git a/docs/apis/tutorials/asset-types.md b/docs/apis/tutorials/asset-types.md deleted file mode 100644 index 053ec5fd7e..0000000000 --- a/docs/apis/tutorials/asset-types.md +++ /dev/null @@ -1,29 +0,0 @@ -# Asset Types - -When you create assets on a temporal data object (TDO), you are required to specify a `type`. -aiWARE supports the following values for the `type` field: - -| Type | Meaning | -| ---- | ------- | -| `media` | Used for anything that’s a file that could be processed. A TDO can have more than one `media` asset to represent various copies of the file or ancillary versions of the original file. When a TDO is processed, the asset that will be used for processing is determined by which of the `media` assets is identified as the `primaryMedia(type: "asset")`. | -| `vtn-standard` | Used for engine output documents. See the [engine output standard](/developer/engines/standards/engine-output/) section for information. | -| `thumbnail` | Used for storing lower-resolution image thumbnail previews of media files. The default thumbnail that is displayed in CMS and other apps is based on the value of the TDO's `thumbnailUrl` property. | -| `content-template` | TDO content templates are extra metadata appended to a TDO. The contents of each `content-template` asset will conform to a particular schema designated by `asset.sourceData.schema`. | -| `transcript` | A legacy asset type used for storing TTML transcripts. Do not use this type; use `vtn-standard` instead. | - -## Custom Asset Types - -You can define your own custom asset types by simply writing an asset with a type value beginning with `x-` (e.g. `x-my-asset-type`). -This will allow you to save asset types specific to your custom workflows and applications. -No special handling will be applied to assets whose type begins with `x-`. - -## Reading Assets - -While it is sometimes necessary to read asset metadata through the [assets] query and access the asset contents directly by downloading the file present at the asset's `signedUri`, -more often you will want to use higher-level APIs to access the information stored in assets. - -- For accessing **audio and video media**, the signedUri returned for the primary media asset is often a reference to our `media-streamer`, -our DASH/HLS-compatible media streaming service that can be used to clip and stitch files and streams. -- For accessing **engine output**, use the [engineResults](/apis/reference/query/?id=engineresults) query. -It normalizes multiple versions of engine output to our most recent standard and can retrieve time-based sections of content. -- For accessing **thumbnails**, use the `temporalDataObject.thumbnail` property, which will return a signed URI. diff --git a/docs/apis/tutorials/cleanup-tdo.md b/docs/apis/tutorials/cleanup-tdo.md deleted file mode 100644 index 01fae6f959..0000000000 --- a/docs/apis/tutorials/cleanup-tdo.md +++ /dev/null @@ -1,49 +0,0 @@ -# Cleaning up TDO data - -As part of the cognition workflow, organizations may upload large amounts of content (media files, etc.) to object storage. Furthermore, as engines run, they may store their results in additional assets and tables. Sometimes it may become necessary to clean up, or delete, some of this content either to save space or comply with certain policies. But the organization may not wish to entirely delete all data. - -The Veritone GraphQL API allows users fine-grained control over what data is delete is what is left on Veritone's servers. - -In the GraphQL schema a temporal data object, or TDO, is the top-level container for media assets and engine results (TDO is called "recording" in the CMS application). To clean up data in a TDO, using the following procedure. - -First, log into the Veritone platform if you haven't already. - -You'll need to know the ID of the TDO, or recording, to operate on. You can get this from CMS – when you click on the media detail in CMS, the ID is the final part of the URL that appears in the browser's address bar. - -Then you can open the interactive GraphQL user interface, called GraphiQL, at https://api.veritone.com/v3/graphiql. - -On the left panel you can type your GraphQL query. A GraphQL query is like a small computer program that uses a simplified language to interact with Veritone's services. The UI has built-in help and auto-completion (CTRL-space) to assist. - -A query that modified information is called a mutation. The specific mutation used to clean up TDO content is cleanupTDO. - -To use the default settings, type this and hit the arrow button or CTRL-ENTER to execute the query: - -```graphql -mutation { - cleanupTDO(id: "") { - id - message - } -} -``` - -On the right panel, you will see a result indicating that the operation ran successfully and the requested data was deleted. The TDO itself is still in the database, along with the records of all engine tasks run against it. - -For more precise control over the data that is deleted, you can pass an additional options parameter. Possible values are: - -* `storage`: Indicates that all assets should be deleted from storage, including those used to store engine results. Metadata about the assets will remain until the container `TemporalDataObject` is permanently deleted. -* `searchIndex`: Indicates that all search index data should be deleted. The `TemporalDataObject` and its assets will no longer be accessible through search.* -* `engineResults`: Indicates that engine results stored on related task objects should be deleted. Engine results stored as assets will remain until assets are removed using the storage option. - -The default behavior is to use the `storage` and `searchIndex` settings. To change this, pass any combination of valid option values as shown below: - -```graphql -mutation { - cleanupTDO(id: "", options: [engineResults, storage]) { - id - message - } -} -``` - -To delete a TDO entirely, use the `deleteTDO` mutation. This will delete the TDO and all metadata about its assets, but will not delete any task records or engine results. diff --git a/docs/apis/tutorials/create-export-request/README.md b/docs/apis/tutorials/create-export-request/README.md deleted file mode 100644 index c47ee20285..0000000000 --- a/docs/apis/tutorials/create-export-request/README.md +++ /dev/null @@ -1,54 +0,0 @@ -# Creating Export Requests - -Export requests can be made to export data out of aiWARE in specific formats. -They are requested via the [`createExportRequest` mutation](/apis/reference/mutation/?id=createexportrequest). - -An example mutation might look like this: - -```graphql -mutation createExportRequest { - createExportRequest(input: { - includeMedia: true - tdoData: [{tdoId: "96972470"}, {tdoId: "77041379"}] - outputConfigurations:[ - { - engineId:"" - categoryId:"67cd4dd0-2f75-445d-a6f0-2f297d6cd182" - formats:[ - { - extension:"ttml" - options: { - maxCharacterPerLine:32 - newLineOnPunctuation: true - } - } - ] - } - ] - }) { - id - status - organizationId - createdDateTime - modifiedDateTime - requestorId - assetUri - } -} -``` - -See the [API reference](https://api.veritone.com/v3/graphqldocs/createexportrequest.doc.html) for full documentation on all the available parameters. - -## Format-Specific Options - -The `options` block under `formats` is (by definition) optional. -When it is used, please note that different options apply to different formats. - - -| Format | Available Options | -| ---- | ---- | -| `txt` | `maxCharacterPerLine`
`withSpeakerData`
`timeGapToSeparateParagraphMs` | -| `ttml` | `maxCharacterPerLine`
`newLineOnPunctuation`
`withSpeakerData` | -| `vtt` | `maxCharacterPerLine`
`newLineOnPunctuation`
`withSpeakerData`
`linesPerScreen` | -| `srt` | `maxCharacterPerLine`
`newLineOnPunctuation`
`withSpeakerData`
`linesPerScreen` | - diff --git a/docs/apis/tutorials/create-tdo-with-asset.md b/docs/apis/tutorials/create-tdo-with-asset.md deleted file mode 100644 index edbd356a34..0000000000 --- a/docs/apis/tutorials/create-tdo-with-asset.md +++ /dev/null @@ -1,60 +0,0 @@ -# Uploading assets - -Recall that in the Veritone data model, a piece of data such as a -media recording, face image, or PDF is an _asset_. Every asset is created -in a container that stores additional metadata and ties associated assets -together for engine processing and searching; this container is called -a _temporal data object_. - -[Upload and process](/apis/tutorials/upload-and-process) describes a typical control -flow in which a client uploads some data, runs an engine to process the -data, and downloads results. The basic mutations involved are `createTDO` -and `createAsset`. - -As a shortcut, the `createTDOWithAsset` can be used to create a TDO and -upload an asset in a single step. This mutation is useful for simple cases -where the client just needs to upload a single piece of data for processing. - -This mutation works just like `createAsset` in that the caller can pass a file -using multipart form post or a plain URI to store a reference. See [GraphQL Basics](/apis/tutorials/graphql-basics) -for details on making such requests. - -To use `createTDOWithAsset`, provide basic metadata about the asset. In this -case we'll store a reference: - -```gql -mutation { - createTDOWithAsset(input: { - assetType:"media" - uri: "https://static.veritone.com/samples/media.mp4" - startDateTime: "2017-12-28T22:30:57.000Z" - stopDateTime: "2017-12-28T22:30:58.000Z" - }) { - id - primaryAsset(assetType: "media") { - id - uri - } - } -} -``` - -Note that we've included put the primary asset in the response fields section -so that we get back the new asset ID along with the new TDO ID: - -```json -{ - "data": { - "createTDOWithAsset": { - "id": "400004331", - "primaryAsset": { - "id": "747f3d1b-eec1-4b38-8495-be38fb70d88b", - "uri": "https://static.veritone.com/samples/media.mp4" - } - } - } -} -``` - -This mutation provides quickest, simplest way to get data into the platform and -start processing. diff --git a/docs/apis/tutorials/engine-fields-in-cms.png b/docs/apis/tutorials/engine-fields-in-cms.png deleted file mode 100644 index 89a22f01ff..0000000000 Binary files a/docs/apis/tutorials/engine-fields-in-cms.png and /dev/null differ diff --git a/docs/apis/tutorials/engine-results.md b/docs/apis/tutorials/engine-results.md deleted file mode 100644 index 7b70b54e4f..0000000000 --- a/docs/apis/tutorials/engine-results.md +++ /dev/null @@ -1,133 +0,0 @@ -# Posting Engine Results - -!> This article applies to `batch` engines only. -Batch engines post results once and for all at the end of their processing. -For information on how `chunk` engines should post results, please see the specific instructions for your engine's -[cognitive capability](/developer/engines/cognitive/?id=capabilities) or the general instructions for use of the -[Engine Toolkit](../../developer/engines/toolkit/). - -After an engine finishes processing and produces a result, several things -have to happen in order for the engine results to be available to -client applications and other engines: - -- the task status must be updated to `complete` -- the engine result must be set on the task's `output` property -- an *asset* must be created containing the engine's result -- task and engine source information must be set in the asset's metadata - -This list assumes that the engine completed its work without error. -When the engine is unable to finish normally, the task status should be set to `failed`, and appropriate details about the task should be specified in the `updateTask` mutation. - (See the [UpdateTask](https://api.veritone.com/v3/graphqldocs/updatetask.doc.html) schema definition for more information.) - -> When reporting errors, we recommend that you use one of the predefined error codes in [TaskFailureReason](https://api.veritone.com/v3/graphqldocs/taskfailurereason.doc.html), if possible. -The use of these codes is optional but can greatly facilitate troubleshooting. - -If the engine finishes its work without error, you can use `updateTask` (with a status of `complete`) to post the engine's results. -For convenience, however, the Veritone API provides a more streamlined -mutation, `uploadEngineResult`, that will report all necessary update info given a task ID and -result payload. -(See the [UploadEngineResult](https://api.veritone.com/v3/graphqldocs/uploadengineresult.doc.html) schema specification.) -Use of this mutation insulates engine developers from -changes in the various requirements listed above, since Veritone engineers will -update the mutation implementation as needed. - -The `uploadEngineResult` mutation will accept engine results via the `file` parameter in a multipart form POST request, or using the `outputString` query parameter. -The former use case is more common, as most engines create files containing their results. -The following examples assume use of multipart form upload. -(See [Request Basics](/apis/tutorials/graphql-basics) for details and sample code for multipart form post.) - -Here's a sample query that creates a transcript engine result. -Note that the `assetType` and `contentType` values will depend on the type of -engine. - -```graphql -mutation { - uploadEngineResult(input: { - taskId: "" - assetType: "v-transcript" - contentType: "application/ttml" - }) { - id - type - uri - contentType - sourceData { - taskId - name - engineId - } - } - } -``` - -The _asset_ can contain any format: JSON, XML, binary, etc. -However, the _task output_ metadata property must contain valid JSON. -`uploadEngineResult` makes this easier for developers to manage by -automatically transforming the engine result into JSON before setting it -on the task metadata (the asset contains unmodified engine results, exactly -as uploaded by the client). The entire result string is placed into a single -JSON value: - -```json - { "data": "..."} -``` - -To override the default key, `data`, use the `outputJsonKey` parameter. -In this example, we'll change it to `transcriptXml`. - -```graphql -mutation { - uploadEngineResult(input: { - taskId: "" - assetType: "v-transcript" - contentType: "application/ttml" - outputJsonKey: "transcriptXml" - }) { - id - } - } -``` - -The resulting value will then be: - -```json -{ "transcriptXml": "..."} -``` - -And, again, the asset will contain the raw XML. - -By default, the task will be marked `complete`. To override this behavior, -use the `completeTask` parameter: - -```graphql -mutation { - uploadEngineResult(input: { - taskId: "" - assetType: "v-transcript" - contentType: "application/ttml" - outputJsonKey: "transcriptXml" - completeTask: false - }) { - id - } - } -``` - -Now say we do not want to produce a file, but rather, feed our engine results -directly from memory into the mutation. - -```graphql -mutation { - uploadEngineResult(input: { - taskId: "" - assetType: "v-transcript" - contentType: "application/ttml" - outputString: "" - }) { - id - } - } -``` - -The output is treated exactly the same way as a file upload with regard to -asset creation and task output JSON mapping. diff --git a/docs/apis/tutorials/file-upload-error-handling.md b/docs/apis/tutorials/file-upload-error-handling.md deleted file mode 100644 index 85e03a0a3f..0000000000 --- a/docs/apis/tutorials/file-upload-error-handling.md +++ /dev/null @@ -1,72 +0,0 @@ -# Handling File Upload Errors - -Veritone supports a number of methods for easily and securely uploading files, however there are certain conditions that can cause an upload to fail. If you encounter an error when uploading a file, review the information provided below to help determine the cause and find suggested courses of action you can take to resolve it. - -To learn more about uploading content to Veritone, including supported upload methods, source types, and size limits, see our [Uploading Large Files](apis/tutorials/uploading-large-files.md) tutorial. - -## Local System File Upload Error - -Attempting to upload or attach a file that exceeds 100MB will return an HTTP 413 error response that looks similar to the following: - -```json -{ - "errors":[ - { - "code":"LIMIT_FILE_SIZE", - "field":"file", - "storageErrors":[], - "data":{ - "limitInBytes":104857600, - "requestSize":"2295200106", - "errorId":"ab3efd8f-c0de-4c84-b299-1d7698b4a9b8" - }, - "name":"invalid_input", - "message":"The file upload was larger than the limit allowed by this server. The maximum file upload size is 104857600 bytes (100 mb)." - } - ] -} -``` - -If your file exceeds the allowable limit, there are two options to work around the 100MB restriction. - -* Split the file into smaller chunks. Although 100MB is a reasonable size for most artifacts (such as long multimedia recordings), cognitive engine processing and analysis performs more efficiently and reliably with smaller object sizes. -* If you’re unable to divide a file larger than 100MB, use the [raw](#direct-httphttps-url-upload) or [pre-signed](#pre-signed-url-upload) URL methods to upload the file without splitting it. - -## Pre-Signed URL Upload Error - -A pre-signed URL upload failure will result in a 403 error. If you receive a 403 error, be sure your request conforms to the following guidelines: - -* Verify that you are doing an `HTTP PUT` (not `GET` or `POST`) and that the URL has not expired. The `expiresInSeconds` field indicates the amount of time remaining (in seconds) before the URL expires. -* If you’re using a client-side HTTP library, check to be sure that no headers have been added to the request and that the URL has not been modified in any way. - -## Query Size Error - -Requests that exceed the maximum allowed query size will return a HTTP 413 response with the following message in the response body: - -```json -{ - "errors":[ - { - "data":{ - "limitInBytes":1000000, - "requestSize":"1200011", - "errorId":"752b9354-8fbd-4071-9a2b-522add55b944" - }, - "name":"invalid_input", - "message":"The request payload was larger than the limit allowed by this server. The maximum JSON request size is 10000000 bytes (10 mb)." - } - ] -} -``` - -Typically, this error is encountered when the request payload exceeds the maximum 10MB limit. Below we describe two common causes of this error and some suggested courses of action for correction. - -### Automated Querying - -A manually constructed query is unlikely to exceed the size capacity. However, queries that are machine-generated through a loop or other input type may attempt to retrieve or edit too many objects in batch and, as a result, they will exceed the allowable limit. You can work around this issue by modifying your code and splitting the query into batches of bounded size. Then, submit the smaller queries using multiple sequential requests. - -### Arbitrary JSON Input - -Although some mutation and data type fields take arbitrary JSON as input, these fields are not designed to consume large objects. An input field with a large JSON representation could alone exceed the allowable 10MB limit and cause the request to fail. For example, the output field of the updateTask mutation could contain a large JSON input value. In this case, even though the base GraphQL query may be small, the size of the output field value could exceed the maximum query size limit. - -To work around this issue, simply reduce the size of the payload by either splitting the input into multiple objects or by uploading it as a file. diff --git a/docs/apis/tutorials/get-engines.md b/docs/apis/tutorials/get-engines.md deleted file mode 100644 index 787b840c96..0000000000 --- a/docs/apis/tutorials/get-engines.md +++ /dev/null @@ -1,111 +0,0 @@ -# Looking Up Available Engines - -The list of available engines in the Veritone platform is constantly growing and -dependent on the user's permissions. -To get information on what engines are available to run and what options they have, -you can use the GraphQL API. - -Here's a sample query that lists all the engine categories and their associated -available engines. - -```graphql -query { - engineCategories { - records { - id - name - engines(limit: 200) { - records { - id - name - fields { - name - options { - value - } - } - } - } - } - } -} -``` - -```json -{ - "data": { - "engineCategories": { - "records": [ - { - "id": "3b2b2ff8-44aa-4db4-9b71-ff96c3bf5923", - "name": "Translate", - "engines": { - "records": [ - { - "id": "6f47f6d4-57e0-4c2c-9f4f-a3e95e7a725e", - "name": "Translate - D", - "fields": [] - }, - { - "id": "388d951e-c90c-f001-9d6b-8bb70b9e6267", - "name": "Jupiter", - "fields": [ - { - "name": "target", - "type": "Picklist", - "options": [ - { - "value": "en" - }, - { - "value": "af" - }, - { - "value": "ar" - } -... -``` - -The `fields` property on the engines defines the set of input parameters the engine accepts. -You can see how the various options behave by running engines against media in CMS using -the "Advanced Cognitive Settings" mode and viewing the dropdown options. - -![setting engine field values in CMS](engine-fields-in-cms.png) - -In practice, you would probably want to limit your returned dataset by specifying -a particular engine category or searching by engine name - -```graphql -query { - engines(name: "Supernova", categoryId: "67cd4dd0-2f75-445d-a6f0-2f297d6cd182") { - records { - id - name - } - } -} -``` - -```json -{ - "data": { - "engines": { - "records": [ - { - "id": "906b4561-5c53-1d3b-6038-aff9c153545c", - "name": "Supernova-Russian" - }, - { - "id": "3cbc180c-0b60-25f5-a850-c06d71f89e30", - "name": "Supernova-Japanese" - } - ] - } - } -} -``` - -The engine IDs returned are what you would submit as the `engineId` in task definitions. -The fields returned specify the keys and values you would submit as the `payload` in -task definitions. See [Uploading and Processing Files](/apis/tutorials/upload-and-process) for -details on how to use those values. diff --git a/docs/apis/tutorials/graphql-basics.md b/docs/apis/tutorials/graphql-basics.md deleted file mode 100644 index e6008cf9a9..0000000000 --- a/docs/apis/tutorials/graphql-basics.md +++ /dev/null @@ -1,254 +0,0 @@ -# GraphQL API Basics - -This tutorial outlines the basic, raw format of a GraphQL API request. - -It assumes that you know what GraphQL is -and understand terms like _schema_, _type_, _field_, _mutation_, and _query_, -and are ready to start using the Veritone APIs. - -This page includes examples using `curl`, Postman, Javascript, and Python. - -## Basic request format - -Every request is an HTTP POST. - -Two types of requests are accepted: plain JSON and multipart form POST. For most requests you'll use plain JSON. Multipart form POST is useful for mutations that accept a file upload. - -To make a plain JSON request: - -- use HTTP POST -- set the `Content-Type` header to `application/json` -- most operations require authentication. If authenticating, set the `Authorization` header to `Bearer: `. -- the request body must contain valid JSON. The JSON must contain a `query` element, with a string value containing the GraphQL query. -- optionally, you can include a `variables` element with a map of variable values referenced in the query - -Tools like `curl` or `wget` are often used to demonstrate how to use REST APIs. With typical REST APIs, this is essential because every operation has a different URL format and request payload. -In a GraphQL API, every request is the same. Thus, we'll -use `curl` to demonstrate the basic structure of a request. - -However, once this basic structure is understand, `curl` is -a poor tool for exploring GraphQL APIs. Use GraphiQL instead. - -```bash -console$ curl https://api.veritone.com/v3/graphql \ -> -H 'Authorization: Bearer ' \ -> -H 'Content-Type: application/json' \ -> -X POST \ -> -d '{ "query":" query { asset(id: \"\") { id assetType } }" }' -``` - -Note that: - -- The `Authorization` header is used for authentication -- The `Content-Type` header is set to `application/json` -- HTTP POST is used -- The request body contains JSON. -- That JSON has a single element, `query` -- The value of `query` is a string containing the GraphQL query. -- Since the request body must parse as valid JSON, it must all -be on one line and be properly escaped (note the escaped double quotes). - -The API returns plain JSON: - -```json -{"data":{"asset":{"id":"","assetType":"transcript"}}} -``` - -You can also use the multipart form POST request format. The response and -the way authorization is sent are exactly the same. For manual testing -a form request can be a little easier to format since the GraphQL query -is sent directly in the `query` form field instead of embedded in JSON. -Note that double quotes inside the GraphQL query in this format do -not have to be escaped. - -```bash -curl -v \ -> -H 'Authorization: Bearer ' \ -> https://api.veritone.com/v3/graphql \ -> -F query='query { asset(id: "") { id assetType } }' -``` - -Here's an example of a basic request in Postman. -First we set the `Authorization` header using a valid token: -![Set auth token in Postman](postman-auth.png) - -Then we send the GraphQL query using multipart form post: -![Send GraphQL query](postman-query.png) - -## Using GraphQL Variables - -It is often convenient to write GraphQL queries using -variables to inject parameterized values, such as the asset ID -in the example above. To do so you must declare the variable -in the query signature, and after that you can reference it -within the query body. The GraphQL server handles any -parsing or formatting of the variable value, as long as a -value of the right type is provided. Note that the type (`ID!`) -in the variable declaration much match the type where it is -used (the `id` parameter on `asset`, which is `ID!`). - -Here's the rewritten query: - -```graphql -query assetById($assetId: ID!) { - asset(id: $assetId) { - id - assetType - } -} -``` - -The client then provides variable values in a separate JSON object. -Here is the `curl` example: - -```bash -console$ curl https://api.veritone.com/v3/graphql \ -> -H 'Authorization: Bearer ' \ -> -H 'Content-Type: application/json' \ -> -X POST \ -> -d '{ "query":" query byId($assetId: ID!) { asset(id: $assetId) { id assetType } }", "variables": " {\"assetId\":\"\"}" }' -``` - -If the request is not formed correctly, you'll receive a 400 error back from the GraphQL server. -The error message will indicate the problem. Typical causes include: - -- invalid or unspecified content type -- the request body does not contain valid JSON with a `query` element -- the string value of the `query` element is not escaped properly -- the string value of the `query` element does not contain a valid -GraphQL query -- variables are referenced in the query but not properly declared or sent in the `variables` parameter - -Here's a Postman sample query using variables: -![GraphQL query with variables](/apis/tutorials/postman-varquery.png) - -And here we bind the variable values: -![GraphQL query with variables](/apis/tutorials/postman-varvars.png) - -## Uploading files with multipart form post - -The GraphQL server also accepts multipart form post requests as described above. -You can submit plain queries -this way if convenient, but the primary purpose of the method is to support file upload for mutations -that accept them. Here is a `curl` example: - -```bash -curl -v \ -> -H 'Authorization: Bearer ' \ -> https://api.veritone.com/v3/graphql \ -> -F query='mutation create($containerId: ID!, $assetType: String!) { createAsset(input: { containerId: $containerId, assetType: $assetType}) { id }}' \ -> -F variables='{ "containerId": "", "assetType": "" }' \ -> -F file=@ -``` - -Note the following differences when using multipart form post: - -- the GraphQL query is provided as a plain string parameter (it is not embedded in JSON) -- variables are defined in a JSON object (exclude this parameter if your query does not use variables) -- a single `file` element contains the file to upload. This convention is used across all Veritone GraphQL APIs that accept a file upload. -- all other aspects of the request -- URL, HTTP method, authentication, etc. -- are the same as a plain JSON request. - -Here's another example using Postman. -Here's the mutation: -![Mutation with file upload](postman-file1.png) -Now we bind variables: -![Variables with file upload](postman-file2.png) -And last, we attach the file: -![Attaching the file](postman-file3.png) - -Use of a client library that handles multipart form post is strongly recommended. - -Below is a bare-bones example using Javascript under NodeJS. - -```javascript -global.fetch = require('node-fetch');; -const request = require('superagent'); -let size = data.length; -let tdoId = ""; -let query = `mutation { - createAsset(input: { - containerId: "${tdoId}" - contentType: "video/mp4" - assetType: "media" - }) { - id - uri - } -} - `; - -let headers = { - Authorization: 'Bearer ' + -}; -request - .post('https://api.veritone.com/v3/graphql') - .set(headers) - .field('query', query) - .field('filename', fileName) - .attach('file', Buffer.from(data, 'utf8'), fileName) - .end(function gotResponse(err, response) { - if (!err) { - let responseData = JSON.parse(response.text); - console.log("new asset created with id "+ responseData.data.createAsset.id); - } - }); -``` - -Here's another example using Python's `requests` library. It uses `createAsset` -as an example. However, the request format is exactly the same for all mutations -that accept file upload (`updateLibraryEngineModel`, `uploadEngineResult`, etc.). -The only difference is in the GraphQL query string. - -```python -import requests -import json -import os - -# basic function that creates an asset using the Veritone GraphQL API using -# multipart form POST -# tdoId: ID of the TDO that will contain the new asset -# filePath: local path to the file to upload -# assetType: asset type, such as "media" or "transcript" -# token: a valid Veritone authentication token -def createAsset(tdoId, filePath, assetType, token): - # This is the GraphQL query string, using GraphQL variables. - query = ''' - mutation createAsset($assetType: String!, $containerId: ID!){ - createAsset(input: { - assetType: $assetType - containerId: $containerId - }) { - id - assetType - contentType - uri - } - } - ''' - # Our variable map will contain the parameter values. - variables = { - 'assetType': assetType, - 'containerId': tdoId - } - # Set up the files for requests library multipart form upload. - # First determine the base file name from the path - fileName = os.path.basename(filePath) - # Now set up the dict containing the file itself - files = { - 'file': (fileName, open(filePath)) - } - # Set up authorization header - headers = { - 'Authorization': 'Bearer %s' % token - } - # Finally set up the other form fields. - data = { - 'query': query, - # Note that variables should have a string containing JSON. - 'variables': json.dumps(variables) - } - - # Make the request and print the result - r = requests.post('https://api.veritone.com/v3/graphql', files=files, data=data, headers=headers) - print(r.text) -``` diff --git a/docs/apis/tutorials/graphql-error-handling.md b/docs/apis/tutorials/graphql-error-handling.md deleted file mode 100644 index c9e549af6a..0000000000 --- a/docs/apis/tutorials/graphql-error-handling.md +++ /dev/null @@ -1,158 +0,0 @@ -# Error Handling in the GraphQL API - -Error handling in a GraphQL API is slightly different than error handling in a typical REST API: - -* There is only one URL endpoint -* A single request can specify any number of queries, mutations, and fields therein - -The GraphQL specification does not explicitly state how errors should be treated, but there are documented conventions and best practices. The Veritone API follows these conventions and also leverages a commonly used library (apollo-error) for error handling consistent with what you may find in other GraphQL APIs. - -An HTTP status code of 200 indicates that the GraphQL server was able to parse the incoming query and attempt to resolve all fields. - -Therefore, a non-200 status code indicates that either the query did not reach the GraphQL server, or that the server was unable to parse it. The follow table lists several HTTP status code you may encounter and their meanings. - -|Code|Meaning| -|----|-------| -|200| GraphQL server received the query, parsed it, and attempted to resolve -| 404| Not found. In GraphQL, there is only one URL endpoint. Therefore, this error means that the caller's URL path was wrong.| -| 400 | Malformed request. The request should have the Content-Type header set to either `application/json` or `multipart/form-data`. The request must contain a query parameter containing JSON, and that JSON should have a query element containing, as a string, a valid GraphQL query.| -| 429 | The request has been rate-limited and should be retried. Rate limits are currently applied across a 10-second window to authentication token (for user sessions and API keys) and organization. The server response complies with the specification for HTTP 429 (https://tools.ietf.org/html/rfc6585#section-4). The response header `Retry-After` contains the time period in seconds after which the request can be retried. Retrying before that time will generate another 429. On rare occasions, the server may emit 429s to prevent errors under exceptional load levels. The response body contains additional details about why the request was subject to rate limiting. -| 500 | An internal server error prevented the server from handling the request. This error will not happen under normal circumstances.| -| 502 | HTTP gateway error. This could indicate an internal server error or timeout. Neither will occur under normal circumstances.| - -A HTTP 200 status code will be accompanied by a normal GraphQL response body in JSON format. Fields that were successfully resolved will have their data. Fields that cannot be successfully resolved will have a null value and a corresponding error set in the errors field. - -Here's an example where we attempt to create three objects and only one succeeds: - -```graphql -mutation { - create1: createAsset(input: { - containerId: 123 - }) { - id - } - - create2: createAsset(input: { - containerId: "400001249" - type:"media" - contentType: "video/mp4" - }) { - id - } - - create3: createAsset(input: { - containerId: "400001249" - type:"media" - contentType: "video/mp4" - uri: "http://localhost/myvideo.mp4" - }) { - id - } -} -``` - -The response: - -```json -{ - "data": { - "create1": null, - "create2": null, - "create3": { - "id": "e6a8e6b1-955a-4d0c-be3b-d1ff83833a15" - } - }, - "errors": [ - { - "message": "The requested object was not found", - "name": "not_found", - "time_thrown": "2017-12-12T01:15:48.875Z", - "data": { - "objectId": "123", - "objectType": "TemporalDataObject" - } - }, - { - "message": "One of uri or file (upload) must be provided to create an asset.", - "name": "invalid_input", - "time_thrown": "2017-12-12T01:15:48.964Z", - } - ] -} -``` - -Here's another example where we attempt to retrieve three objects, but only one is found: - -```graphql -query { - asset1: asset(id: "2426dbe5-eef3-4167-9da8-fb1eeec61c67") { - id - } - asset2: asset(id: "2426dbe5-eef3-4167-9da8-fb1eeec61c68") { - id - } - asset3: asset(id: "1fa65e5a-8008-48e4-9968-272fbef54cc2") { - id - } -} -``` - -Result: - -```json -{ - "data": { - "asset1": null, - "asset2": null, - "asset3": { - "id": "1fa65e5a-8008-48e4-9968-272fbef54cc2" - } - }, - "errors": [ - { - "message": "The requested object was not found", - "name": "not_found", - "time_thrown": "2017-12-12T01:21:30.243Z", - "data": { - "objectId": "2426dbe5-eef3-4167-9da8-fb1eeec61c67", - "objectType": "Asset" - } - }, - { - "message": "The requested object was not found", - "name": "not_found", - "time_thrown": "2017-12-12T01:21:30.247Z", - "data": { - "objectId": "2426dbe5-eef3-4167-9da8-fb1eeec61c68", - "objectType": "Asset" - } - } - ] -} -``` - -So, to check for errors you should first verify HTTP status 200, and then check for an errors array in the response body. Or, if a field you expected to find a value in has null, look in the errors object for an explanation. - -Error information is shown in a consistent format: - -|Field|Description| -|-----|-----------| -|`message`| A human-readable description of the error cause| -|`name` |A machine-readable error code in string form| -|`time_thrown`| Timestamp| -|`data` |Operation-specific supplementation information about the error. For example on a `not_found` the data field will often have `objectId` with the ID that could not be found.| - -A standard set of error codes is used across the API. This list may grow over time. - -|Error code|Meaning|Recommended action| -|----------|-------|------------------| -|`not_found` |A requested object was not found. This could mean that the object never existed, existed and has been deleted, or exists but the caller does not have access to it. Verify that object exists and that the caller has rights to it.| -|`not_allowed` |The caller is not authorized to perform the requested operation. For example, a user with the Viewer role on the CMS app would get a not_allowed error if attempting to use the createAsset example shown above. Either provision the caller with the required rights or do not attempt to access the object.| -|`invalid_input`| Although the query contains syntactically valid GraphQL according to the schema, something was wrong with the parameters on the field. For example, an integer parameter have have been outside the allowed range. Indicates a client error. Refer to the schema documentation to fix the client code.| -|`capacity_exceeded` |Server capacity allocated to the client was exceeded while processing the request. Try again later or, if the query was a complex one that may be expensive to resolve, break it apart or use a smaller page size.| -|`authentication_error` |The client did not supply a valid authentication token or the token was not of the type required for the requested operation. Not all fields require authentication, but an attempt to access one that does without supplying a valid token will cause this error. Get and submit a current authentication token. If the token is current, the field may require a specific type of token such as `api` or `user`. Consult the schema documentation and correct the client code.| -|`rate_limited` |The request was subject to rate limiting. See HTTP 429, above.|Retry the request after the time interval in seconds specified in the `Retry-After` response header.| -|`not_implemented` |The requested query, mutation, or field is not available on this server. The cause may be a configuration issue or operational problem affecting a required subsystem. Verify that the operation is actually supported by consulting schema docs. If so, try again later or contact Veritone support. -|`service_unavailable`|A required service could not be reached. This error can indicate a temporary outage or a misconfiguration. Try again later or contact Veritone support.| -|`service_failure`| A required service was accessible, but failed to respond or fulfill a request successfully. This error can indicate a temporary outage or a misconfiguration. Try again later or contact Veritone support.| -|`internal_error` |An internal server error prevented the field from being resolved. Try again later or contact Veritone support.| diff --git a/docs/apis/tutorials/paging.md b/docs/apis/tutorials/paging.md deleted file mode 100644 index b6124facf6..0000000000 --- a/docs/apis/tutorials/paging.md +++ /dev/null @@ -1,323 +0,0 @@ -# Paging - -Many queries can return a large number of results. Consider -a simple query over jobs -- a typical organization will have -thousands, if not millions, of jobs in the system. -Similarly, an organization may have many thousands of temporal data objects. - -It isn't realistic for any client, or for the server, to attempt to -process or display the entire result set at once. Therefore, the API -support paging in a fairly typical way. - -Some client developers may have implemented paging in a REST API. -Paging in our GraphQL is very similar, but GraphQL poses a potential -complexity. We'll take the simple case first. - -## Array Types in GraphQL - -In the GraphQL schema, each field (recall that a query is simply a -field on the special type `Query`) declares its return type. -For example, `name: String` indicates that the field is a string. -Brackets indicate that the field is an _array_ of the type. -For example, `names: [String]` would be an array of strings. - -Some fields on the Veritone schema use simple arrays. -For example, the `User` type has a list of user settings. -Here's the schema definition: - -```graphql -# Settings for the user -userSettings: [UserSetting!] -``` - -And here's a sample query: - -```graphql -query { - me { - userSettings { - key - value - } - } -} -``` - -The values within the field are returned as a JSON array: - -```json -{ - "data": { - "me": { - "userSettings": [ - { - "key":"favoriteAnimal", - "value":"hedgehog" - }, { - "key":"favoriteFood", - "value":"deep-fried turtle" - } - ] - } - } -} -``` - -This method works well for fields where the list size is known to be small. -In this case, we know that the number of user preferences will not be large; -even the pickiest user would have less than 100. Any server or client -implementation can easily hold and process such a list in memory, and -a client UI could support it with at most simple scrolling. - -However, this does not work for cases such as those described above, -where the number of results can be in the hundreds, thousands, or millions. -Such fields support paging. - -## Paged Fields - -A common paradigm for paged fields is implemented across the API. - -Every paged field returns a _list_ type. -For example, the `jobs` query returns `JobList`. -Each object type has its own list type, but they all comply with the -same schema. The only difference is in the field type of the `records` -array, which contains the actual objects. - -Further, each paged field takes a standard pair of parameters that control -paging. -For example, here is the definition for `jobs`: - -```graphql -type Query { - jobs( - # Provide an offset to skip to a certain element in the result, for paging. - offset: Int = 0 - # Specify the maximum number of results to included in this response, or page size. - limit: Int = 30 - ): JobList! -} - -type JobList implements Page { - # Jobs retrieved - records: [Job!] - # The starting index for records that were returned in this query. - offset: Int! - # Maximum number of results that were retrieved in this query; page size - limit: Int! - # Number of records returned in this response - count: Int -} -``` - -Here we'll ask for the first page of three: - -```graphql -query { - jobs (offset: 0 limit: 3){ - count - offset - limit - records { - id - } - } -} -``` - -`JobList` represents a single page of results. The actual `Job` objects -are contained in the `records` field. - -```json -{ - "data": { - "jobs": { - "offset": 0, - "limit": 3, - "count": 3, - "records": [ - { - "id": "bf133402-4945-4b0c-950f-f46c9b935139" - }, - { - "id": "fa254e47-0b0d-41d5-9671-2006581a3606" - }, - { - "id": "cc7f95bb-e5a0-4a1e-9a11-04e6b58dc1be" - } - ] - } - } -} -``` - -As you can see, `records` is an array of objects, each of which contains -the fields we requested under `records` in our query. `offset` and `limit` -reflect the values passed by the client. `count` is the actual number of -results returned. - -A _total_ count of all possible results that can be returned by the query -across all pages cannot be reliably computed by the server in a performant -way for all queries. Thus, this value is not included in the schema. -The client cannot know ahead of time how many results there are; it must -iterate over the pages until it reaches the end. - -The API follows the following contract across all paged fields: - -* the default offset is 0 (first page) -* there is a default page size, almost always 30 (documented per field) -* the number of objects returned in `records` will be less than or equal -to the value set for `limit` -* `count` will equal the size of the `records` array -* if `count` is less than `limit`, there are no more results available; the client -has reached the last page -* a request for a nonexistent page (`offset` greater than total possible results) -returns an empty page, not an error - -Therefore, the client can iterate over pages until it reaches a page with -less than the requested number of results. That is the last page. If the -total number of results divides evenly by the page size, the last page will -have size zero. - -## Paging and Nested Fields - -Let's take a more complex query that retrieves nested fields: - -```graphql -query { - jobs (offset: 0 limit: 3){ - count - offset - limit - records { - id - tasks (offset: 0 limit: 3){ - count - offset - limit - records { - id - } - } - } - } -} -``` - -Note that: - -* the nested `tasks` field _also_ is paged -* the page parameters for `jobs` and its nested `tasks` are independent - -It returns: - -```json -{ - "data": { - "jobs": { - "count": 3, - "offset": 0, - "limit": 3, - "records": [ - { - "id": "fb32d786-b5c8-4982-b7b1-cb25e4e5c03f", - "tasks": { - "count": 3, - "offset": 0, - "limit": 3, - "records": [ - { - "id": "fb32d786-b5c8-4982-b7b1-cb25e4e5c03f-c7fb88d8-8cbb-43b1-ae20-3d8c737308c5" - }, - { - "id": "fb32d786-b5c8-4982-b7b1-cb25e4e5c03f-7971dc08-04c3-4b94-a7d2-6082fe9e1950" - }, - { - "id": "fb32d786-b5c8-4982-b7b1-cb25e4e5c03f-b17c3776-24bc-4cb4-9513-b1724db85e34" - } - ] - } - }, - { - "id": "56e352e8-6c63-4d9f-9995-03c96f6e6ec5", - "tasks": { - "count": 1, - "offset": 0, - "limit": 3, - "records": [ - { - "id": "56e352e8-6c63-4d9f-9995-03c96f6e6ec5-4e133ed8-08b0-493d-8c6a-44a1daf98e3b" - } - ] - } - }, - { - "id": "cc7f95bb-e5a0-4a1e-9a11-04e6b58dc1be", - "tasks": { - "count": 1, - "offset": 0, - "limit": 3, - "records": [ - { - "id": "cc7f95bb-e5a0-4a1e-9a11-04e6b58dc1be-fa8e969d-0e8b-45b7-ba79-857d1efdb656" - } - ] - } - } - ] - } - } -} -``` - -As we examine the response, some important facts become apparent. - -* the `tasks` field involves retrieving and populating a separate set of -objects _per job_ in top-level job page. Thus, this query can be considerably -more expensive than the original `jobs` query even though the page size is -the same. -* both `jobs` and the `tasks` field within a given `Job` are paged. -Therefore, to get the full list of all tasks for all jobs a client would need -to implemented nested or recursive paging: for each page of `Job` results, -iterate over the jobs, and for each one iterate over its `tasks` results. - -The implications of nested paging for the client depend entirely on what -that client needs to do. Should it iterate recursively over all results? -This might be appropriate for, for instance, a script that exports a -library full of entities and entity identifiers (three nested levels of paging). -Or should it get only the top level (jobs, in our example) and drill into -pages sub-fields on demand? For most interactive user interfaces, that is the -most effective model. - -In all cases it is important to choose an appropriate page size that -will allow each HTTP request to complete in an acceptable amount of time. - -## Choosing an Appropriate Page Size - -The default page size is suitable for most almost all cases. -A smaller page size can always be used without ill effect. - -### Large Page Sizes - -In some cases you may want to make query that returns minimal information -about each object but an entire result set. For example, you might want -to get the entire list of engine names and IDs within a given category. -Simply set a very large page size: - -```graphql -query { - engines(categoryId: "6faad6b7-0837-45f9-b161-2f6bf31b7a07", limit: 500) { - records { - id - name - } - } -} -``` - -Note that this is a good practice _only_ for queries that include a small -selection of scalar fields. - -Some queries may enforce a maximum page size. This varies per query. -If you make a query with a `limit` value that exceeds the maximum allowed -value you will receive a `invalid_input` error with message and payload -describing the problem. diff --git a/docs/apis/tutorials/postman-auth.png b/docs/apis/tutorials/postman-auth.png deleted file mode 100644 index a558cf101c..0000000000 Binary files a/docs/apis/tutorials/postman-auth.png and /dev/null differ diff --git a/docs/apis/tutorials/postman-file1.png b/docs/apis/tutorials/postman-file1.png deleted file mode 100644 index a8c198c1f6..0000000000 Binary files a/docs/apis/tutorials/postman-file1.png and /dev/null differ diff --git a/docs/apis/tutorials/postman-file2.png b/docs/apis/tutorials/postman-file2.png deleted file mode 100644 index 24ed6556ca..0000000000 Binary files a/docs/apis/tutorials/postman-file2.png and /dev/null differ diff --git a/docs/apis/tutorials/postman-file3.png b/docs/apis/tutorials/postman-file3.png deleted file mode 100644 index e5a9d6ad77..0000000000 Binary files a/docs/apis/tutorials/postman-file3.png and /dev/null differ diff --git a/docs/apis/tutorials/postman-query.png b/docs/apis/tutorials/postman-query.png deleted file mode 100644 index 410a6ea0b5..0000000000 Binary files a/docs/apis/tutorials/postman-query.png and /dev/null differ diff --git a/docs/apis/tutorials/postman-varquery.png b/docs/apis/tutorials/postman-varquery.png deleted file mode 100644 index 62a123d7a4..0000000000 Binary files a/docs/apis/tutorials/postman-varquery.png and /dev/null differ diff --git a/docs/apis/tutorials/postman-varvars.png b/docs/apis/tutorials/postman-varvars.png deleted file mode 100644 index 759a281dfe..0000000000 Binary files a/docs/apis/tutorials/postman-varvars.png and /dev/null differ diff --git a/docs/apis/tutorials/tokens.md b/docs/apis/tutorials/tokens.md deleted file mode 100644 index 150c6f39dc..0000000000 --- a/docs/apis/tutorials/tokens.md +++ /dev/null @@ -1,538 +0,0 @@ -# Authentication and Authorization Tokens - -This tutorial covers some common questions Veritone API users might have, such as - -* what is a token; -* where do I get one; -* what am I allowed to do with it? - -First, as noted on the main [authentication page](/apis/authentication), the Veritone API -accepts authentication in the standard form of a bearer token in -the HTTP `Authorization` header: - -```bash -curl https://api.veritone.com/v3/graphql -H 'Authorization: Bearer ' -H 'Content-Type: application/json' -d '{"query" : "query { me { id }}"}' -``` - -This is the _only_ authentication method accepted by the API. -Interactive user interfaces acquire and submit this token on behalf of the user. - -A token is always an opaque string of letters, numbers, and the characters `-` and `:`. - -The token serves to 1) identify the client to the Veritone platform and -2) describe the rights and privileges the client has over resources. - -There are several types of tokens that a client developer may come across, -depending on whether they are developing an engine, developing an application, -or doing ad-hoc testing. - -## User Tokens - -A user, or session-based token, is scoped to an individual user within -an organization. User tokens are not persistent; they expire. - -User tokens should be used within any interactive, end-user application, -whether it be web-based, mobile, or otherwise. The user must have an -account on the Veritone platform within the target environment (for example, -[http://https://www.veritone.com/login/#/]). A user account is created within -an organization by organization administrators or Veritone staff. - -Each user is granted certain privileges, or roles, within the organization. -Each role carries with it certain functional rights enforced within the API. -For example, a user with the "Developer Editor" role within the "Developer" -platform application has rights required to list, create, and update -the organization's engines with `engines(...)`, `createEngine(...)`, -`updateEngine(...)`, and other related queries and mutation. - -To provision a user account or modify roles, contact your organization -administrator or Veritone support. The organization administrator can -manage users with the main Veritone administration interface. - -The roles that are available to assign to users depend on the applications -and features that are provisioned for the organization. For example, -and organization must be entitled to use the Veritone Developer Application (VDA) -in order for any of its users to have the "Developer" roles. - -If you are _using_ an interactive user application, the application will -authenticate you, acquire a token, and send the token on your behalf. -You might need to have certain roles on your account in order for the -application's features to work, but you do not need to worry about -tokens or authentication. - -If you are _developing_ an interactive user application, then the -application code will need to authenticate the user to acquire the session -token and then provide that token with every API request. - -The preferred method to authenticate in this case is _OAuth flow_, as described -[here](/apis/authentication). OAuth is an industry standard and is -secure and flexible with a rich set of tooling and infrastructure to -support it. - -In certain situations a client application might not be able to use OAuth. -Such clients can authenticate directly to the Veritone API using the -`userLogin` mutation. - -```graphql -mutation { - userLogin(input: { - userName: "sampleuser@veritone.com" - password: "sample password" - }) { - token - } -} -``` - -If successful, a token is returned: - -```json -{ - "data": { - "userLogin": { - "token": "c45360ca-5110-3cdb-1252-ae9dda1e29ce" - } - } -} -``` - -This user token can then be used to authenticate to the API. - -_Important_: An application should log in _once_ and use the resulting -token for the duration of the session. Do _not_ log in before every request! - -## API Keys - -Some ecosystem developers only build system integrations or engines -and may never have a cause to deal with user accounts or tokens. - -Such developers will generally rely on specially provisioned, persistent tokens called -API keys. - -API keys can be used for testing engine code and for -intended for long-running system components. For example, if you have a -service such as a webhook handler that needs to use the Veritone API, -you can acquire an API key with appropriate rights and configure your -service to send it when authenticating to the API. - -To provision an API key, your organization administrator can use -the administration interface. - -* Go to the Veritone Admin app -* Navigate to the organization overview page for your organization -* Click the "API Keys" tile -* Click "Add token" -* Give the token a distinct name that indicates its purpose (for eample, "webhook-handler") -* Select appropriate rights. Use caution and do not add more rights than are strictly necessary. -They can be updated later if necessary. -* When you click "Save", the entire token will be displayed. -Copy it to the clipboard and store it somewhere safe. -For security reasons, the UI does not display actual tokens in the listing. -* If necessary, come back to this tile later to edit the token and change its rights. -You can also revoke the token from this page. - -_Warning_: API keys are sensitive information. They are persistent, -lasting until revoked, and enable broad access to your organization's data. -Store them only in secure locations and give them out only to trusted -users with a true need for the access. - -In some cases Veritone support may serve as organization administrator and -provide API key. - -Like user tokens, API key are given functional rights. Within a given -operation, and API keys has implicit access to all of your organization's -data. For example, an API key with the 'read TDO' right can read -_all_ of your organization's TDOs. - -## Engine Tokens - -Engine processing poses a complex authorization problem. - -As a cognitive processing consumer, you create or ingest content -and then run a variety of engines against that content. Whether you -do so using the Veritone CMS application or some other method, -you select content and select engines. You implicitly authorize -the engines to access _that specific content_. - -The engines -can include core Veritone engines as well as third-party engines -produced by the Veritone developer ecosystem. You might, as a consumer, -not even know exactly which engines are being run. - -For the engine to work and produce results, the engine code must -have access to the content you ran it on (typically a TDO and associated -assets). The engine must also be able to store its output and associate -it with your content (typically as a new asset on the target TDO). - -However, the engine should _not_ be able to access any _other_ content -within your organization. Nor should it be able to modify your content -in unexpected ways. For example, say you have uploaded two media -files to CMS: - -* `meetingRecording.mp4` -* `securityCameraStream-2018-05-08.mp4` - -Each upload results in a new TDO with a single media asset. -You then use the CMS interface to run the default transcription engine -against `meetingRecording.mp4`. This creates a new job with several tasks: -one for transcoding, one for transcription, perhaps others. - -The transcription engine will use the Veritone API to retrieve and process your content and store its results. -The engine should be able to: - -* retrieve metadata about `meetingRecording.mp4` -* download `meetingRecording.mp4` -* update the status of its own task -* create and upload an asset containing the transcript and attach it to -the TDO for `meetingRecording.mp4` - -The engine should _not_ be able to: - -* retrieve metadata about `securityCameraStream-2018-05-08.mp4` -* delete the media asset for `meetingRecording.mp4` -* update the status of the transcoding engine's task _or_ any other task -* retrieve or modify other types of data in your organization, such as -user information, libraries, etc. - -In other words, when processing a task, an engine should have permission -_only_ to access the data targeted in the task and write its own results -to the appropriate location. - -To enforce these constraints, the platform generates a special, -single-use token for each task. The token comes with a built-in set -of rights that allow the engine to perform its intended function -on the target data _and nothing else_. Since the token is intended -for one-time use in a single engine task, it expires after an -appropriate time window. - -When examining the rights associated with an engine token through the -API (see below for details), you will see something like this: - -```json -{ - "myRights": { - "operations": [ - "asset:uri", - "asset:all", - "recording:read", - "recording:update", - "task:update" - ], - "resources": { - "Job": [ - "1923036a-abac-482a-9e68-d10d43f42849" - ], - "Task": [ - "1923036a-abac-482a-9e68-d10d43f42849-eaf8bc0c-a197-4691-9c24-f8d34b791acb" - ], - "TemporalDataObject": [ - "400000148" - ] - } - } -} -``` - -Note that the rights include a limited set of functional permissions appropriate -for the engine (in this case, read a TDO and update its task) _and_ -rights to a specific set of objects: the job and task object under which -the engine was invoked, and the TDO that it will process. - -Certain rights are also derived from the those rights declared in the token. -For example, an engine can access an asset it just created, as long as it -sets source data on that asset with the correct task ID. - -Any attempt using this engine token (whether by the engine at runtime or -in manual testing) to perform any other type of operation or access -any other data will result in a `not_allowed` error. For example, -with the above sample token rights, an engine cannot invoke -`updateTDO` on _any_ TDO (including its target) or call `updateTask` on -any task other than the ID specified in `resources.Task`. - -Engine tokens are generated by the platform system components during -engine execution and are intended for one-time use for a specific job. -Thus, they are not suitable for testing engine code during development. -For that purpose, use an API key as described above. - -## Examining and Troubleshooting Tokens - -The Veritone GraphQL API authorizes access by functional rights -at the level of individual fields within the top-level GraphQL query. -Error responses follow the general pattern described in -[Error Codes](/apis/error-codes.md) -- the HTTP request will return -with HTTP 200, the `errors` section of the payload will be non-empty -and contain the message and other information about the error, -and the affected field in the `data` element -will be `null`. The error type is listed in the `name` field. -Most authorization errors surface as `not_allowed`. You may occasionally -see `not_found`. - -The API enforces authorization on _resources_ as well as operations. -For example, if you create a temporal data object (TDO) and do not make it -public, users from other organizations do not have any access to it. -If you do make the TDO public, then users from other organizations -can read the TDO and its data, but not modify it in any way. - -The error messages and payloads that the API returns include -as much detail as is possible to share without potentially -compromising the confidentiality and integrity of your data. - -### "not_allowed" Errors - -A `not_allowed` error can occur under normal conditions when a client -attempts to access some field, query, or mutation that it is not -authorized to use. It does not indicate an error in the server or API. - -Resolution steps may include: - -* Switch tokens. For example, if you are using a user token to test -engine code, switch to an engine token or API token for a more realistic -test -* Remove the affected field, query, or mutation from your request. If -the field is not strictly necessary for your use case, simply do not use it. -* Get help from your organization admin to give your user or API token -additional rights. For example, if you are developing an engine, you should -probably have the "developer editor" role. If you are writing a system -integration that creates jobs using an API token, that token may need -"full job permission" rights. - -To handle a `not_allowed` error, first examine the entire response payload -and the token itself. - -Here's a sample `createEngine` attempt: - -```graphql -mutation { - createEngine(input: { - name: "foo" - categoryId: "bar" - deploymentModel: HumanReview} - ) { - id - } -} -``` - -And here's the response for a token that does not have sufficient rights -to create an engine: - -```json -{ - "errors": [ - { - "message": "The authenticated user or token is not authorized to perform the requested action.", - "data": { - "field": "createEngine", - "type": "Mutation", - "rights": [ - "asset:uri", - "job:create", - "job:read", - "job:update", - "job:delete", - "task:update", - "recording:create", - "recording:read", - "recording:update", - "recording:delete", - "report:create", - "ingestion:read", - "ingestion:create", - "ingestion:update", - "ingestion:delete" - ] - }, - "name": "not_allowed", - "time_thrown": "2018-05-09T21:48:00.680Z" - } - ], - "data": { - "createEngine": null - } -} -``` - -The Veritone API provides some queries that can be useful for understanding -what client information is associated with a given token and what rights are -associated with it. - -For example, the following query will return some information about the -authenticated user, their organization, and the functional rights available. -This is just an example - additional fields can be added to the user and -organization fields as needed. - -```graphql -query { - me { - id - name - organization { - id - name - } - } - myRights { - operations - resources - } -} -``` - -Since GraphiQL is an interactive, browser-based application that requires -the user be logged into the platform, it implicitly will only use the user-scoped -session token. It isn't ideal for testing with or troubleshooting the other -token types. For this purpose (only) we use a raw HTTP client such as `curl`. - -```bash -curl https://api.veritone.com/v3/graphql \ - -H 'Content-Type: application/json' \ - -H 'Authorization: Bearer ' \ - -d '{"query": "query { me { id name organization { id name }} myRights { operations resources }}"}' -``` - -The details of the response will depend on the token in question. -Just for example, for a user who has the developer editor role -we might see something like the following. - -```json -{ - "data": { - "me": { - "id": "e92d0333-4ac2-69a0-4d95-dbc2eb5240c3", - "name": "sampleuser@veritone.com", - "organization": { - "id": "1111111111", - "name": "Sample Organization" - } - }, - "myRights": { - "operations": [ - "job.create", - "job.read", - "job.update", - "job.delete", - "task.create", - "task.read", - "task.update", - "task.delete", - "recording.create", - "recording.read", - "recording.update", - "recording.delete", - "developer.access", - "developer.docker.org.push", - "developer.docker.org.pull", - "developer.engine.read", - "developer.engine.update", - "developer.engine.create", - "developer.engine.delete", - "developer.engine.disable", - "developer.engine.enable", - "developer.build.read", - "developer.build.update", - "developer.build.delete", - "developer.build.deploy", - "developer.build.submit", - "developer.build.pause", - "developer.build.unpause", - "developer.build.approve", - "developer.build.disapprove", - "developer.task.read" - ], - "resources": {} - } - } -} -``` - -The details of the `myRights` payload are used and managed internally -and subject to change. However, they may still suffice to give a general -idea of what rights a token has and why certain GraphQL queries or -mutations return a `not_allowed` error. For example, `job.create` is required -for `createJob`. - -You may also encounter a `not_allowed` if you attempt an unauthorized -operation on a resource that you can read, but not edit. -For example, if you attempt to run `updateTDO` on a TDO that is owned -by another organization but is public (readable to you), you will -receive `not_allowed`. Similarly, if you run `updateTDO` on a TDO that -is owned by your organization _but_ your user or API key does not have -permission to update TDOs, you will receive `not_allowed`. - -### "not_found" Errors - -If you do not have read access to a given resource, for example if it is -owned by another organization and is private, the resource is effectively -invisible to you through the API. It will not be returned, even in partial -form, in any listing. For example, the following query returns only TDOs -to which the client has at least read access: - -```graphql -query { - temporalDataObjects(includePublic: true) { - records { - id - } - } -} -``` - -Inaccessible resources are simply not returned. There is no error. - -However, this query attempts to access a specific resource by ID: - -```graphql -query { - temporalDataObject(id: 4000051912345) { - id - } -} -``` - -If the resource does not exist _or_ does exist but you do not have access -to it, you'll receive a `not_found`: - -```json -{ - "data": { - "temporalDataObject": null - }, - "errors": [ - { - "message": "The requested object was not found", - "name": "not_found", - "time_thrown": "2018-05-09T23:05:14.864Z", - "data": { - "objectId": "4000051912345", - "objectType": "TemporalDataObject", - "errorId": "062884db-f227-4bfe-ad44-370c30b855bf" - } - } - ] -} -``` - -In the latter case, you must of course have got the ID from somewhere in -order to bootstrap the query. The object may have been made private, -or un-shared, at some point. Or it may have been passed through some side channel -(email, instant message, etc.) by a user who does have access to it. -In any case, unauthorized objects are effectively invisible; the caller is -not entitled to know if they exist or not. - -An unexpected `not_found` can indicate that: - -* the resource has been deleted -* the resource is owned by another organization and your access has been revoked -* you are inadvertently using a token for a different org (such as a personal - free developer account, versus an organization for you company) -* you are using an engine token that was created for a different job (this is a common mis-step) - -Resolutions may include: - -* Just don't attempt to access the resource. Ignore the error and continue, -or, if you're testing, use a resource that you do have access to. -* Locate the owner of the resource (whoever gave you its ID) and ask them to -share it with your organization or make it public (they might, of course, decline to do so). -* If you're using an API token, ensure that it is for the correct organization. -* If you're using an engine token, generate a fresh engine payload and test task. -This payload will contain a new token with appropriate rights for the -resources referenced in the task. diff --git a/docs/apis/tutorials/upload-and-process.md b/docs/apis/tutorials/upload-and-process.md deleted file mode 100644 index 459fa35b8e..0000000000 --- a/docs/apis/tutorials/upload-and-process.md +++ /dev/null @@ -1,232 +0,0 @@ -# Uploading and Processing Files - -A very common workflow for Veritone integrations is to upload a file, process it, and -download the results. In Veritone's API, that flow is modeled by the following requests: - -1. Create a Temporal Data Object (TDO) -2. Create an Asset under the TDO -3. Create a job to run specific engines against the TDO -4. Check for job completion -5. When job is complete, download the results - -_Veritone's Sample React App implements this request flow in its -[example workflow](https://github.com/veritone/veritone-sample-app-react/blob/master/src/modules/mediaExample/index.js#L126) -and is a good example to look at or copy and paste from._ - -## 1. Create a Temporal Data Object (TDO) - -Temporal Data Objects (TDOs) are the main container for all data in Veritone's system. -They contain merely a start and stop time (in UTC) so results can be correlated to -Veritone's master timeline in its Temporal Elastic Database™. - -Here's a sample query that creates a TDO and returns its ID and date range. The ID will -be used in subsequent calls. - -```graphql -mutation { - createTDO(input: { - startDateTime: "2018-01-01T10:00:00" - stopDateTime: "2018-01-01T11:00:00" - }) { - id - startDateTime - stopDateTime - } -} -``` - -```json -{ - "data": { - "createTDO": { - "id": "52359027", - "startDateTime": "2018-01-01T10:00:00.000Z", - "stopDateTime": "2018-01-01T11:00:00.000Z" - } - } -} -``` - -## 2. Upload an Asset - -Assets are always contained in TDOs. Here's an example request to add an asset -through GraphQL. - -```graphql -mutation { - createAsset(input: { - containerId: "52359027", - assetType: "media", - contentType: "video/mp4", - uri: "" - }) { - id - } -} -``` - -```json -{ - "data": { - "createAsset": { - "id": "be8611e8-d833-4ce9-b731-ca8f7b153e38" - } - } -} -``` - -Alternatively, if the file is not available on a publicly accessible URL, you can issue -a `multipart/form-data` request with the file contents in the file form field. - -## 3. Create a Job - -To process the file you've uploaded, you construct a job that has one or more tasks. -Each task contains an engineId and optionally a JSON payload defining field selections -and/or library information (for trainable engines). - -To explore which engines and fields are available, see the tutorial on -[looking up available engines](/apis/tutorials/get-engines). - -```graphql -mutation { - createJob(input: { - targetId: "52359027", - tasks: [{ - engineId: "" - }, { - engineId: "", - payload: { - picklistField: "value", - multiPicklistField: ["value", "another value"] - } - }] - }) { - id - } -} -``` - -```json -{ - "data": { - "createJob": { - "id": "cc60a74e-be58-4366-8f8e-4a2590852b96" - } - } -} -``` - -## 4. Check for Job Completion - -Jobs run asynchronously, run their tasks in parallel, and can be queried for their status. -Using the job ID you received when creating the job, you will want to periodically check -on the status of the job and its associated tasks. - -```graphql -query { - job(id: "cc60a74e-be58-4366-8f8e-4a2590852b96") { - status - tasks { - records { - id - engineId - status - } - } - } -} -``` - -```json -{ - "data": { - "job": { - "status": "running", - "tasks": { - "records": [ - { - "id": "cc60a74e-be58-4366-8f8e-4a2590852b96-a60847d3-4f40-4929-abbc-1406ee2d1c46", - "engineId": "fc004413-89f0-132a-60b2-b94522fb7e66", - "status": "running" - }, - { - "id": "cc60a74e-be58-4366-8f8e-4a2590852b96-55ad3c97-0d55-4e32-b397-2c7b737a4b84", - "engineId": "f44aa80e-4650-c55c-58e7-49c965019790", - "status": "pending" - } - ] - } - } - } -} -``` - -## 5. Download Results - -When the job completes, there are two places to look for information: - -- The `taskStatus` on the task: where engines can report a single piece of information -- The `assets` in the TDO: where engines can write multiple formats of output - -```graphql -query { - job(id: "cc60a74e-be58-4366-8f8e-4a2590852b96") { - targetId - tasks { - records { - engineId - taskOutput - } - } - } - - temporalDataObject(id: "52359027") { - assets { - records { - assetType - signedUri - sourceData { - taskId - engineId - } - } - } - } -} -``` - -```json -{ - "data": { - "job": { - "targetId": "52359027", - "tasks": { - "records": [ - { - "engineId": "fc004413-89f0-132a-60b2-b94522fb7e66", - "taskOutput": {} - }, - { - "engineId": "f44aa80e-4650-c55c-58e7-49c965019790", - "taskOutput": null - } - ] - } - }, - "temporalDataObject": { - "assets": { - "records": [ - { - "assetType": "media", - "signedUri": "", - "sourceData": { - "taskId": null, - "engineId": "48b690ee-8a36-4ede-b1e0-d27b035ac8bd" - } - } - ] - } - } - } -} -``` diff --git a/docs/apis/tutorials/uploading-large-files.md b/docs/apis/tutorials/uploading-large-files.md deleted file mode 100644 index fbc7b446d6..0000000000 --- a/docs/apis/tutorials/uploading-large-files.md +++ /dev/null @@ -1,592 +0,0 @@ -# Uploading Large Files - -Veritone provides API for securely and reliably uploading files to your account. Several mutations in the Veritone API accept file upload, but if your data is large in size or exceeds the maximum upload limit, you may experience slower upload speed, timeouts, or failed requests. - -For best performance and to make file uploading as easy and flexible as possible, Veritone supports multiple methods that allow you to upload files of virtually any size. Depending on the size of the data and the requirements of your environment, you can choose from the following options and send upload requests in the way that best meets your needs: - -* **Local System Upload:** Attach and upload a file 100MB in size or less using a local file path. This method includes an option to chunk large files into a sequence of parts and upload them in parallel. -* **HTTP/HTTPS URL Upload:** Upload a file larger than 100MB using a publicly accessible HTTP or HTTPS URL to the file. -* **Pre-Signed URL Upload:** Generate a temporary URL that provides a path to upload content from your server to Veritone for a limited amount of time. - -This tutorial includes everything you need to know about performing successful file uploads. It describes basic content storage concepts, outlines the general file upload process, defines size requirements, details the different upload methods, covers some common pitfalls, and provides corrective actions you can take to resolve errors. You can read this tutorial end-to-end to get all the details or use the links below to jump to a specific section if you already know what you’re looking for. - -* [Content Storage and Upload Concepts](#content-storage-and-upload-concepts) -* [Size Limitations](#size-limitations) -* [Local System File Upload](#local-file-system-upload) -* [HTTP/HTTPS URL File Upload](#direct-httphttps-url-upload) -* [Pre-signed URL Upload](#pre-signed-url-upload) -* [Handling Errors](#handling-errors) - -## Content Storage and Upload Concepts - -### Content Storage Overview - -Content that’s ingested and stored in Veritone is organized as *containers* and *assets*. - -* **Container:** A container is similar to a conventional system directory or folder. It holds an uploaded file and any number of additional assets, which are renditions of the original uploaded file (e.g., a thumbnail generated from the original file). The Veritone GraphQL API refers to a container as a "Temporal Data Object" (or TDO). Each container has a unique TDO ID that’s used to identify it and connect it to its associated assets. - -* **Asset:** An asset is much like a file. It represents an object in a container and consists of binary or text content and metadata. Videos, images, and transcripts that are stored in Veritone are examples of assets. Each asset is identified by a unique ID and contains the TDO ID it’s associated with as well as the URL path to the location where the file is stored in Veritone. When a file is uploaded to a container, it becomes the container’s original asset. While an asset can only be associated with one container, a container may hold multiple rendition assets of the original uploaded file. Some rendition assets are created by default based the original asset’s file type. Examples of renditions include generating a thumbnail image or encoding the original file into a format compatible for engine processing. Additional rendition assets are also produced when a task is executed on an existing asset, such as creating a transcription from an audio file. - -### File Upload Process Overview - -The general file upload process consists of two distinct functions that occur in a single request. A call to the `createTDOWithAsset` mutation first creates an empty container (or TDO) and then uploads the file to it. Once a file is successfully uploaded, it’s stored as an asset and can be used as the input for performing other actions, such as cognitive processing. - -## Size Limitations - -To safeguard the performance and integrity of the system, the Veritone API has built-in size limiting mechanisms on inbound requests and file uploads. These limits apply to both the `application/json` and `multipart/form-data` request protocols. The following sections provide details on the the limits accepted by Veritone to help you in setting up and managing your requests. - -### Maximum Query Size - -Veritone supports a maximum request size of 10MB. This limit exceeds the size of most typical requests and is well-suited for most purposes. There’s no limit to the number of elements a request contains as long as the overall query size doesn’t exceed 10MB. It’s important to note that this limitation applies only to the size of the GraphQL request body — external resources contained in a query (e.g., a file attachment) do not count toward the 10MB limit. Calls that exceed the maximum will return an error. If you encounter a query size error, see the [Handling Errors](#handling-errors) section for possible error codes that can be returned along with suggested actions you can take to resolve them. - -### Maximum File Upload Size - -The Veritone API enforces certain file size limitations to ensure the reliability of data uploads. These limits are based on what can be reliably uploaded with a single request in a reasonable amount of time. The following table shows the size limits per request for various file types and upload methods. When uploading a single file, the defined limit is the maximum allowed size for that file. When a file is uploaded in parts, it’s the maximum allowed size for each of the file chunks. Files that are larger than the limits cannot be transferred in a single request. Veritone provides options to accommodate larger file uploads, which are outlined in the *Recommended Best Practice* column of the table and described in detail later in this guide. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

File Size Limits

-
File Type
Upload Method
Max Size Limit Per Request
Recommended Best Practice
video or audioLocal Path100MBFor files that exceed 100MB: -Split the file into smaller chunks and upload as a multipart/form request. --OR- -Upload the file via URL
URLNo limit*Split larger files into chunks no more than 15 minutes in length.
image, text, or applicationLocal Path100MBFor files that exceed 100MB: -Split the file into smaller chunks and upload as a multipart/form request. --OR- -Upload the file via URL
URLNo limit*Split files that exceed 250MB into smaller chunks.
*Note:* Although there are no restrictions imposed by Veritone for URL uploads, it’s important to note that cognition engines may set their own limits for data processing and analysis. Therefore, it’s recommended to split large files into smaller chunks to optimize performance and reduce the risk of error.
- - -## Local File System Upload - -You can upload files from your local system up to 100MB in a single operation by doing a `multipart/form-data` HTTP request. This method structures requests in multiple parts that represent different characteristics of the file. In each request, form field data specifying information about the file and a query containing details about the asset are sent along with `Authentication` and `Content-Type` headers. When a file is successfully uploaded, a `TDO/container ID` and `URL` to the file’s storage location are returned in the response. These values can then be used with other Veritione API to perform additional actions, such as cognitive processing. - -Files that are larger than the 100MB limit can be split into smaller, more manageable pieces and uploaded independently, in any order, and in parallel. All file chunks should be the same size (up to 100MB each), except for the last chunk, which can be any size under 100MB. If you want to send a large file without dividing it, you can use the raw or pre-signed URL upload method. - -> If you encounter an error during a local file upload, see the [Handling Errors](#handling-errors) section for possible error codes that can be returned along with suggested actions you can take to resolve them. - -To upload a local file, make a request to the `createTdoWithAsset` mutation. When structuring your request, be sure to set the `Content-Type` header to `multipart/form-data` and use the `form-data` keys `query`, `file`, and `filename` in the body. Currently, GraphiQL does not support multipart/form requests, so a different HTTP client must be used for making sample calls. - -If you’re uploading a large file that’s segmented into smaller parts, upload the first chunk of the file in the `createTdoWithAsset` request and then use the `TDO ID` returned in the response to upload the remaining chunks to the same container in the `createAsset` mutation. - -### Step 1. Create Container and Upload File - -Create the TDO/container and upload a local file up to 100 MB or the primary file chunk of a larger, split file. - -#### Sample Request Payload — Create TDO with Asset (local file upload) - -```graphql --H content-type: => A header that specifies the content type. Enter "multipart/form-data" as the value.(required) --F filename => The name of the first file chunk to upload. The value must match the name of the saved file. (required) --F file => The path of the file to upload. (required) --F query=mutation { ------------request fields----------- - createTDOWithAsset(input:{ => The mutation type and input variable. (required) - startDateTime: "string" => The starting date and time of the file to be uploaded in [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) format. (required) - stopDateTime: "string" => The ending date and time in [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) format. The value is calculated by adding the length of the file to the startDateTime value. If a value is not provided, a 15-minute default value will be applied. (optional) - contentType: "string" => The MIME type of the file being uploaded (e.g., "audio/mp3"). If a value is not provided, the default value "video/mp4" will be applied. (optional) - assetType: "string" => A label that classifies the file to be uploaded, such as "transcript," "media," or "text." If a value is not specified, the default value "media" will be applied. (optional) - addToIndex: => A Boolean that adds the uploaded file to the search index when set to true. (optional and recommended) - }){ ------------return fields----------- - id => The unique ID associated with the TDO/container, provided by the server. (required) - status => The status of the request’s progress. (required) - assets{ => The Assets object parameter that contains the TDO's assets. (required) - records { => The Records object parameter that contains data specific to individual assets. (required) - id => The unique ID of the new asset, provided by the server. (required) - type => A label that classifies the asset. This reflects the value specified in the request or the default value "media" if no request value was set. (required) - contentType => The MIME type of the asset. This reflects the value specified in the request or the default value "video/mp4" if no request value was set. (required) - signedUri => The secure URI of the asset. (required) -``` - -#### Sample Request: Create TDO with Asset (local file upload) - -```bash -curl -X POST \ - https://api.veritone.com/v3/graphql \ - -H 'authorization: Bearer 31gcf6:2e76022093e64732b4c48f202234394328abcf72d50e4981b8043a19e8d9baac' \ - -H 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW' \ - -F 'filename=rescued puppies road trip.mp4' \ - -F 'file=@/Users/bobjones/Downloads/rescued puppies road trip.mp4' \ - -F 'query=mutation { - createTDOWithAsset( - input: { - startDateTime: 1507128535 - stopDateTime: 1507128542 - contentType: "video/mp4" - assetType: "media" - - addToIndex: true - } ) { - id - status - assets{ - records { - id - type - contentType - signedUri - } - } - } -}' -``` - -#### Sample Response: Create TDO with Asset (local file upload) - -```json -{ - "data": { - "createTDOWithAsset": { - "id": "44512341", - "status": "recorded", - "assets": { - "records": [ - { - "id": "7acfab47-f648-4cfc-9042-74b4cafb1605", - "type": "media", - "contentType": "video/mp4", - "signedUri": "https://inspirent.s3.amazonaws.com/assets/44512341/fd3bd480-2c23-4107-a283-3a2b54d6e512.mp4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAJ6G7HKQBDKQ35MFQ%2F20180713%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180713T155226Z&X-Amz-Expires=3600&X-Amz-Security-Token=FQoDYXdzEF0aDAJG7F%2FR22OpH5fwNCK3A10D6Sx3CYJ6G7kk0ykfjR59f4cLoDNE87BbY2BE0L6ivCD1GN2ZQP4XV8Im6NxQogv5aNqsFnolF5kY9fT%2BSRQwOPNkQA%2FGqPiFJMHkjwvz5zl8SvkW%2FvfAlhIEFEcVKR36jQxNZcrHi10TwMGj3w88bs1lSGz2NOIKZwKOrFSHioxkznZhHLMCzYImVQNmoX2g8qA%2BZVUlsuBh4QvlL3pg7ww%2BXpPaQ1U6vNmpIZEuZIw2TBG%2BmZh9NwNWqFuRtvfxX%2FzoKq71wmI%2F%2F4CGdQ9t%2BUbbWLvT0iVdv7fvoZPbRr6iGvfBsQ2SAD7tXM2iqz2XO%2B8cqKRbeBZ%2BLEzN3K2v2LIGOaHp1%2BDpyYSjo0WcHAVx2KWkq%2BrdvfLH%2Bl%2FANyRMOQdEZrlbkS0ZMrtLIQU4b74ylkslkwhbpP6K%2FxvdDSi7TPQH4xRNdz2OCiELHWKVs%2FHc5Gx1BUrg%2F4C%2BeXTIWFHY2qrqgOhnfW3wZq4p4J%2BZCNbORIOnePb21ZIQtUVQWOEuvmwKKpsz4eDDV%2FvZduq%2Bhkwtwhcr8AimWDbcJyoL9jRwrzqwI9ri12RP7O%2FwYT410jIoxJyi2gU%3D&X-Amz-Signature=e75c68ab486a356da011ba413c88ebb7db3eeacbfa1ebd5b68d469211e736e09&X-Amz-SignedHeaders=host" - } - ] - } - } - } -} -``` - -### Step 2. Upload Additional File Chunks (Split Files Only) - -Once the TDO is created and primary file chunk is uploaded, use the `TDO/container ID` returned in the response from the previous step and make individual calls to the `createAsset` mutation to upload each of the remaining file chunks. - -#### Sample Request Payload — Create Asset (local file upload) - -```graphql --H content-type: => A header that specifies the content type. Enter "multipart/form-data" as the value.(required) --F filename => The name of the file chunk to upload. The value must match the name of the saved file. (required) --F file => The path of the file to upload. (required) --F query=mutation { ------------request fields----------- - createAsset(input:{ => The mutation type and input variable. (required) - containerId: "string" => The TDO/Container ID returned in the createTDOWithAsset response. (required) - contentType: "string" => The MIME type of the file being uploaded (e.g., "audio/mp3"). If a value is not provided, the default value "video/mp4" will be applied. (optional) - assetType: "string" => A label that classifies the file to be uploaded, such as "transcript," “media,” or “text.” If a value is not specified, the default value "media" will be applied. (optional) - }){ ------------return fields----------- - id => The unique ID of the new asset, provided by the server. (required) - assetType => A label that classifies the asset. This reflects the value specified in the request or the default value "media" if no request value was set. - contentType => The MIME type of the asset (e.g., "audio/mp3"). This reflects the value specified in the request or the default value "video/mp4" if no request value was set. (required) - containerId => The unique ID associated with the TDO/container, provided by the server. (required) - signedUri => The secure URI of the asset. (required) -``` - -#### Sample Request — Create Asset (local file upload) - -```bash -curl -X POST \ - https://api.veritone.com/v3/graphql \ - -H 'authorization: Bearer 31rzg6:2e76022093e64732b4c48f202234394328abcf72d50e4981b8043a19e8d9baac' \ - -H 'cache-control: no-cache' \ - -H 'content-type: multipart/form-data; boundary=----WebKitFormBoundary7MA4YWxkTrZu0gW' \ - -H 'postman-token: e73ec9f6-050a-0c1f-2a50-8ca09ad8ac4d' \ - -F 'query=mutation { - createAsset( - input: { - containerId: "44512341" - contentType: "video/mp4" - type: "media" - }) { - id - assetType - contentType - containerId - signedUri -} -} -' \ - -F 'filename=rescued puppies road trip.mp4' \ - -F 'file=@/Users/lisafontaine/Downloads/rescued puppies road trip.mp4' -``` - -#### Sample Response — Create Asset (local file upload) - -```json -{ - "data": { - "createAsset": { - "id": "eaf6795e-9e9a-435b-9878-cde23c261d38", - "assetType": "media", - "contentType": "video/mp4", - "containerId": "44512341", - "signedUri": "https://inspirent.s3.amazonaws.com/assets/44512341/eaf6795e-9e9a-435b-9878-cde23c261d38.mp4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAJPATTVSBG43PWIIA%2F20180723%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180723T180739Z&X-Amz-Expires=3600&X-Amz-Security-Token=FQoDYXdzEGIaDJ3ILmUthQ1diK3A4dlKa1djhm%2Bq3qtrSsxd7m4HDp8HSNt6rOoHISeQrRZpL9%2FSAqRNkNsb6nW4%2FV7aXtdbhWPw4naw5TuVAQga7qHx3aIQ6nyYZQ5%2BqgJh0ZRHp6XrGXKTbngb5PrqyBoE%2FxTyvXF6a5l3odiJeHSrZ5vjVgINKRqlRTUH%2FI0KIBwn7WBXaBoLsjfV1ilwJiGnFD2Mlv03kx90zgf8fiog9rI3LXd6TIBDUtWDX5VrIKp3u19ddjWes%2FSsV6W9K1BG0gs97kwCdn%2BGD%2BQF5znIzO79P8Mle%2BnFeW5opZQWuLXb3zi8k7qaj13WU411VhxGq46mNSg5iT0V%2FddoKYehXprfbJajC%2BVkP5LRHDuLPuWz0FfeHfK%2BpKMiZ35ZMhNmu2dE9cu%2BUB9DoUuRYGYkdZNp7wvFhHG2x%2FsacvE0n1m0VtbBHr%2B5qrxaaKC01XSvbB28T%2BxQkmoaTx4BsDZNRsmsDuEwLRu9Aqz3vRJovHUqeqQpS02SSoHSyncrNq%2BxBswJyS%2BuwNAx6LJRu4%2FL5Qx3wk%2Bv6RqGVqoOAxTuy9p9hOrw%2B%2F6BsYyZJrSja7tFJsTaocGYwgo0ZjY2gU%3D&X-Amz-Signature=ff191f86b188891c2eb9ff654459afce2fdbf7ce9b45810f159cc3b30e8141&X-Amz-SignedHeaders=host" - } - } -} -``` - -## Direct HTTP/HTTPS URL Upload - -If the file you want to upload is publicly available online, you can pass the HTTP or HTTPS URL in the request instead of uploading the actual file data. This method allows you to upload files of almost any size, making it useful for adding large files that exceed 100MB in size. Although Veritone does not impose any limits on the size of the file being uploaded using a URL, it’s important to note that cognition engines can set their own limits for data processing and analysis. To prevent performance degradation and to ensure data processing reliability, it’s recommended to follow the [file size limit](#size-limitations) best practices. - -When uploading a file from a URL, include the `uri` parameter in the request with the publicly accessible URL to the file set as the value. Files uploaded from a URL are not saved in the database — only the URL path is stored, which is used to redirect users directly to the file location. This provides you with additional privacy and management of your data by allowing you to control how long the file is available. If a saved URL becomes invalid, the file is no longer accessible. - -### Upload a File via URL - -To upload a file using a URL, make a call to the `createTDOWithAsset` mutation and pass the URL in the input body of the request. - -The following example creates a TDO/container and uploads a file from the specified URL location. - -#### Sample Request Payload — Create TDO with Asset (URL upload) - -```graphql -mutation { ------------request fields----------- - createTDOWithAsset(input:{ => The mutation type and input variable. - startDateTime: "string" => The starting date and time of the file to be uploaded in [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) format. (required) - stopDateTime: "string" => The ending date and time in [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) format. The value is calculated by adding the length of the file to the startDateTime value. If a value is not provided, a 15-minute default value will be applied. (optional) - addToIndex: => A Boolean that adds the uploaded file to the search index when set to true. (optional and recommended) - contentType: "string" => The MIME type of the file being uploaded (e.g., "audio/mp3"). If a value is not provided, the default value "video/mp4" will be applied. (optional) - assetType: "string" => A label that classifies the file to be uploaded, such as "transcript," "media," or "text." If a value is not specified, the default value "media" will be applied. (optional) - addToIndex: => A Boolean that adds the uploaded file to the search index when set to true. (optional and recommended) - uri: "string" => The publicly available URL to the file. - }){ ------------return fields----------- - id => The unique ID associated with the TDO/container, provided by the server. - status => The status of the request’s progress. - assets{ => The Assets object parameter that contains the TDO's assets. - records { => The Records object parameter that contains data specific to individual assets. - id => The unique ID of the new asset, provided by the server. - type => A label that classifies the asset. This is the value specified in the request or the default value "media" if no request value was set. - contentType => The MIME type of the asset. This is the value specified in the request or the default value "video/mp4" if no request value was set. - signedUri => The secure URI of the asset. -``` - -#### Sample Request — Create TDO with Asset (URL upload) - -```graphql -mutation { - createTDOWithAsset( - input:{ - startDateTime: 1507128535 - stopDateTime: 1507128542 - contentType: "video/mp4" - assetType: "media" - uri: "https://www.youtube.com/watch?v=LUzGYV-_qkQ" - } - ) { - id - status - assets{ - records { - id - type - contentType - signedUri - } - } - } -} -``` - -#### Sample Response — Create TDO with Asset (URL upload) - -```json -{ - "data": { - "createTDOWithAsset": { - "id": "44512341", - "status": "recorded", - "assets": { - "records": [ - { - "id": "7acfab47-f648-4cfc-9042-74b4cafb1605", - "type": "media", - "contentType": "video/mp4", - "signedUri": "https://www.youtube.com/watch?v=LUzGYV-_qkQ" - } - ] - } - } - } -} -``` - -## Pre-Signed URL Upload - -For added security, you can use a pre-signed URL to upload a file of nearly any size from your own server. A pre-signed URL is a temporary link that can be used access to a file in your storage facility and upload it directly to Veritone’s S3 without making it publicly available. - -Pre-signed URLs are scoped to allow access to a specific operation (PUT), bucket, and key for a limited amount of time. They also account for security permissions required to access the file once it’s been uploaded to Veritone. Uploading a file with this method involves the following steps: - -1. Generate a pre-signed URL. -2. Make an HTTP PUT request to the signed URL to upload the file. -3. Create an asset with the uploaded file. - -In addition to generating the pre-signed `url`, requests will also return an `unsignedUrl` and a `getUrl`. These additional URLs are used for different purposes and are only effective once the file has been uploaded to Veritone. The `unsignedUrl` indicates the upload location of the file and is used to create an asset with the uploaded file. The `getUrl` allows the file object to be retrieved after the pre-signed URL expires. This is useful if you want to make the file data available without giving free-range access to your storage. - -### Additional Notes - -* By default, a pre-signed URL is valid for 10,800 seconds (or three hours). -* Veritone does not impose any limits on the size of the file being uploaded with a pre-signed URL, but it’s important to note that cognition engines can set their own limits for data processing. To prevent performance degradation and to ensure the reliability of data processing, it’s recommended to follow the best practices related to file size limits. -* A separate pre-signed URL must be created for each chunk in a large file upload. -* To upload a file using your own pre-signed URL, follow the [Direct HTTP/HTTPS Upload](#direct-httphttps-url-upload) instructions and provide the path in the `uri` field of the request. - -> If you encounter an error using the pre-signed URL upload method, see the [Handling Errors](#handling-errors) section for a list of possible error codes that can be returned along with suggested actions you can take to resolve them. - -### Step 1 — Generate a Pre-Signed URL - -To generate a pre-signed URL, make a request to the `getSignedWritableUrl` query as demonstrated in the example below. - -#### Sample Request Payload — Get Signed Writable URL - -```graphql -query { ------------request fields----------- - getSignedWritableUrl => The mutation type. - (key: "filename.txt") => The unique identifier for an object within a bucket. Every object in a bucket has exactly one key. This is an optional parameter to specify a custom prefix for the object key. A key can be anything, but it’s most commonly a file name. - { ------------return fields----------- - key => The unique UUID for the file object to be uploaded. If a key prefix was included in the request, the UUID will be appended to the specified input value. - bucket => The name of the s3 bucket where the object will reside. - url => The signed, writable URL used to directly upload the file. The URL accepts HTTP PUT (only). - getUrl => A signed URL that can be used with HTTP GET to retrieve the contents of the file after it has been uploaded. - unsignedUrl => The raw, unsigned URL to the object. Once the file has been uploaded, this URL value is used to create an asset. - expiresInSeconds => The amount of time (in seconds) that the user has to start uploading the file. This is set to 10800 seconds by default, meaning the user will have 3 hours to start sending the file. If file upload takes longer than 3 hours to complete, the connection will not be closed. - } -} -``` - -#### Sample Request — Get Signed Writable URL - -```graphql -query { - getSignedWritableUrl(key: "your-filename.mp4") { - key - bucket - url - getUrl - unsignedUrl - expiresInSeconds - } -} -``` - -#### Sample Response — Get Signed Writable URL - -```json -{ - "data": { - "getSignedWritableUrl": { - "key": "rescued-puppies-roadtrip-b15e37e7-2fe6-48e9-9d57-865983bebd55.mp4", - "bucket": "prod-api.veritone.com", - "url": "https://s3.amazonaws.com/prod-api.veritone.com/rescued-puppies-roadtrip-b15e37e7-2fe6-48e9-9d57-865983bebd55.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJSGMPJHUC4ZLIYMQ%2F20180507%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180507T223634Z&X-Amz-Expires=10800&X-Amz-Signature=1eed5e973843e397510def0325437571d350b3ad90320a4f8dc9f4d9b503f798&X-Amz-SignedHeaders=host", - "getUrl": "https://s3.amazonaws.com/prod-api.veritone.com/your-filename-b15e37e7-2fe6-48e9-9d57-865983bebd55.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJSGMPJHUC4ZLIYMQ%2F20180507%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180507T223634Z&X-Amz-Expires=10800&X-Amz-Signature=25b1753f8c46a204c2a5f1e9c4bb23fdb98cf1d2c0a4d02936ac2978417b8ab1&X-Amz-SignedHeaders=host", - "unsignedUrl": "https://s3.amazonaws.com/prod-api.veritone.com/your-filename-b15e37e7-2fe6-48e9-9d57-865983bebd55.mp4", - "expiresInSeconds": 10800 - } - } -} -``` - -### Step 2. Use the Pre-Signed URL to Upload a File - -To upload the file using the pre-signed URL, make a PUT request to the `url` value returned in the previous step. Successful requests will return a 200 response with no additional data. - -#### Sample Request Structure — Upload File Using Pre-Signed URL - -```bash -curl - -X PUT ' \ - -d "@/path/to/filename" -``` - -#### Sample Request — Upload File Using Pre-Signed URL - -```bash -curl -v --X PUT -"https://s3.amazonaws.com/prod-api.veritone.com/rescued-puppies-roadtrip-b15e37e7-2fe6-48e9-9d57-865983bebd55.mp4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAIO2KSNIJXJIKRDJQ%2F20180724%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180724T144431Z&X-Amz-Expires=10800&X-Amz-Security-Token=FQoDYXdzEHcaDHs3x4EGqZ0Dw%2B3q1yK3A4T%2F0FdzMxilgPNhEHH55WltvT6LDTDMtPHFur02%2FabCgnvjdQvPXhw1DNsSZewayHYoYdREYeZcYj5K3hMuCBV7RRZl%2F5n%2BfNhDo4GkSzYMeXAPdoJMYBEaaivBNV386AZyUy%2BQ79B4Ij5jVNAiP62KqbsvSzIHonlpX6nflAzJMxohs%2F38nCSpwfRJ97QyaFvTIy9ekiv%2BGHuWXci3nRFIEWwaHyoiLq1sNzhQ4D2LDkb2dZ2AnBFrPusvB%2B%2FULbxIy41XBr0JycfM0Mlc55yPnaZr%2FzIB7KKNI2vESGFu7EPkoZdwwX58wvaCkXPaFV7l3a3jeMeASKN00Fg5JfpSYZltiHK9%2FSqHPZso1uRBBshJiYrla4okNiR0r4zfq%2BkO9pYJDQZ1UNy81Dw6yjcSLNB438M0F3oIYQj2J1lHGe9%2Bok9koE4umB9sC3Wvgz46O25Q3MN9H6zk4cw1Nf0UzeEoKduO1B3gZ7HD7IJT2dcN5L3y5Qih1QrJMS2GeKo7Z75j5a1EeZu7%2F0fqaX1nTafEfM3pLQu0qlRhiwjmMiBgFdpDsLN3Iz4f%2BG3N5tJ%2BPK2lHIwo2OPc2gU%3D&X-Amz-Signature=ac1ad0a9194aee797047ebfea7abee7f1cd40d9e00f97942db29042ac4032d47&X-Amz-SignedHeaders=host" - -d "@rescued-puppies-roadtrip.mp4" -``` - -### Step 3. Create an Asset - -Once the file is uploaded, pass the `unsignedUrl` value returned in the response of Step 1 to the `createTDOWithAsset` mutation to create a TDO/container and file asset. - -#### Sample Request Payload — Create TDO with Asset (pre-signed URL upload) - -```graphql -mutation { ------------request fields----------- - createTDOWithAsset(input:{ => The mutation type and input variable. - startDateTime: integer => The starting date and time of the file to be uploaded in [Unix/Epoch (https://www.epochconverter.com/) timestamp format. - stopDateTime: integer => The ending date and time in [Unix/Epoch](https://www.epochconverter.com/) timestamp format, calculated by adding the length of the file to the startDateTime. If a value is not specified, a 15-minute default value will be applied. (optional) - addToIndex: => A Boolean that adds the uploaded file to the search index when set to true. (optional and recommended) - contentType: "string" => The MIME type of the file being uploaded (e.g., audio/mp3). If a value is not specified, the default value "video/mp4" will be applied. (optional) - assetType: "string" => A label that classifies the file to be uploaded, such as "transcript," “media,” or “text.” If a value is not specified, the default value "media" will be applied. (optional) - uri: "string" => The URL location of the file. Use the unsignedUrl value returned in the getSignedWritableUrl response (Step 1). - }){ ------------return fields----------- - id => The unique ID associated with the TDO/container, provided by the server. - status => The status of the request’s progress. - assets{ => The Assets object parameter that contains the TDO's assets. - records { => The Records object parameter that contains data specific to individual assets. - id => The unique ID of the new asset, provided by the server. - type => A label that classifies the asset. This is the value specified in the request or the default value "media" if no request value was set. - contentType => The MIME type of the asset (e.g., audio/mp3). - signedUri => The secure URI of the asset. - } - } - } -} -``` - -#### Sample Request — Create TDO with Asset (pre-signed URL upload) - -```graphql -mutation { - createTDOWithAsset( - input:{ - startDateTime: 1507128535 - stopDateTime: 1507128542 - contentType: "video/mp4" - assetType: "media" - uri: "https://s3.amazonaws.com/prod-api.veritone.com/your-filename-b15e37e7-2fe6-48e9-9d57-865983bebd55.mp4" - } - ) { - id - status - assets{ - records { - id - type - contentType - signedUri - } - } - } -} -``` - -#### Sample Response — Create TDO with Asset (pre-signed URL upload) - -```json -{ - "data": { - "createTDOWithAsset": { - "id": "44512341", - "status": "recorded", - "assets": { - "records": [ - { - "id": "b15e37e7-2fe6-48e9-9d57-865983bebd55", - "type": "media", - "contentType": "video/mp4", - "signedUri": "https://s3.amazonaws.com/prod-api.veritone.com/your-filename-b15e37e7-2fe6-48e9-9d57-865983bebd55.mp4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAJNBZJF3A6AQ7XYZQ%2F20180724%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20180724T150615Z&X-Amz-Expires=3600&X-Amz-Security-Token=FQoDYXdzEHcaDNfCYgj1hIAxUC1NJCK3A5FG9Z2goJG12uAqPBMaHpMqDdZpFynoeeW0HB85xBKPxn3sTutO84skuR7O2y3XyajQdtsVtFFlZQo85gE26LKwI8aHJE1iQ%2Bjg90bmom1iEmqLwv7ZSIzlnVp8rMQbCob1JyVuBly%2Bd7KVRGwzkvThnVI1SJAcnnV3oNdPP2I%2FQcZLRuIgRrnJTOVQb3PU3RecWfjBc1zAc9VR7rDyqNxg48Nt4uxqdZfwdZZALG9%2F5AZrGTx%2B%2Fe3KsOfSrridcfe1J3XPyybLPBGJr5x%2F4BhKL3yY%2FuCNuCosm2h4pbU0sw1KisGg4wstn0DSLQuG8EAd8hmbCFRlJoelfOVFJShq4WWZp9X1tdUTZ1hMs9%2FGe2PL3X0WAuIylH1SFZCPMrU3tHLMbMWI6i6cZ7wI9IzQO%2BKXgL07G1IDcvFKpOZVE5QrSsmDh73R7L%2B3NTitMUlTDH6WVV4WvishMWAqULRh0oIbMtIMPUhBfBWtyW%2F%2Fq2w%2B3sAIOZoEYgdTXS8PGfmnt0GK58EroO5MC2oEhrIVz4wqhmbj%2B7elQa7NtvdXlgDae9u9HJLd1PXBJ5zYAQNTR%2Fi3qfko1%2Bfc2gU%3D&X-Amz-Signature=c1fe009a408aea900ba909e3402e6f25915b8f0ab2fbe842cec3468e816fcdf9&X-Amz-SignedHeaders=host" - } - ] - } - } - } -} -``` - -## Handling Errors - -### Query Size Error - -Requests that exceed the maximum allowed query size will return a HTTP 413 response with the following message in the response body: - -```json -{ - "errors":[ - { - "data":{ - "limitInBytes":1000000, - "requestSize":"1200011", - "errorId":"752b9354-8fbd-4071-9a2b-522add55b944" - }, - "name":"invalid_input", - "message":"The request payload was larger than the limit allowed by this server. The maximum JSON request size is 10000000 bytes (10 mb)." - } - ] -} -``` - -Typically, this error is encountered when the request payload exceeds the maximum 10MB limit. Below we describe two common causes of this error and some suggested courses of action for correction. - -#### Automated Querying - -A manually constructed query is unlikely to exceed the size capacity. However, queries that are machine-generated through a loop or other input type may attempt to retrieve or edit too many objects in batch and, as a result, they will exceed the allowable limit. You can work around this issue by modifying your code and splitting the query into batches of bounded size. Then, submit the smaller queries using multiple sequential requests. - -#### Arbitrary JSON Input - -Although some mutation and data type fields take arbitrary JSON as input, these fields are not designed to consume large objects. An input field with a large JSON representation could alone exceed the allowable 10MB limit and cause the request to fail. For example, the output field of the updateTask mutation could contain a large JSON input value. In this case, even though the base GraphQL query may be small, the size of the output field value could exceed the maximum query size limit. - -To work around this issue, simply reduce the size of the payload by either splitting the input into multiple objects or by uploading it as a file. - -### Local System File Upload Error - -Attempting to upload or attach a file that exceeds 100MB will return an HTTP 413 error response that looks similar to the following: - -```json -{ - "errors":[ - { - "code":"LIMIT_FILE_SIZE", - "field":"file", - "storageErrors":[], - "data":{ - "limitInBytes":104857600, - "requestSize":"2295200106", - "errorId":"ab3efd8f-c0de-4c84-b299-1d7698b4a9b8" - }, - "name":"invalid_input", - "message":"The file upload was larger than the limit allowed by this server. The maximum file upload size is 104857600 bytes (100 mb)." - } - ] -} -``` - -If your file exceeds the allowable limit, there are two options to work around the 100MB restriction. - -* Split the file into smaller chunks. Although 100MB is a reasonable size for most artifacts (such as long multimedia recordings), cognitive engine processing and analysis performs more efficiently and reliably with smaller object sizes. -* If you’re unable to divide a file larger than 100MB, use the [raw](#direct-httphttps-url-upload) or [pre-signed](#pre-signed-url-upload) URL methods to upload the file without splitting it. - -### Pre-Signed URL Upload Errors - -If you receive a 403 error, there are a few things you can check. - -* Verify that you are doing an `HTTP PUT` (not `GET` or `POST`) and that the URL has not expired. The `expiresInSeconds` field indicates the amount of time remaining (in seconds) before the URL expires. -* If you’re using a client-side HTTP library, check to be sure that no headers have been added to the request and that the URL has not been modified in any way. diff --git a/docs/apis/using-graphql.md b/docs/apis/using-graphql.md deleted file mode 100644 index bb460cfd6d..0000000000 --- a/docs/apis/using-graphql.md +++ /dev/null @@ -1,231 +0,0 @@ -# Using GraphQL - -## What is GraphQL? - -GraphQL (http://graphql.org/) is a query language. It allows users to format queries that specify exactly what information they want to send or receive, using a formal schema defined by the GraphQL provider. - -GraphQL initiated with Facebook and is now used by a number of major API providers, including Github. A rich and rapidly expanding ecosystem of GraphQL tooling, much of it open source, helps both GraphQL providers and clients to quickly build data-centric applications and services. - -## Why is Veritone using GraphQL for its APIs? - -Veritone uses GraphQL to provide a clean, unified interface to its data and operations. GraphQL has many benefits over a traditional REST API. - -* Users can control exactly what information they get back with each operation -* The schema is self-documenting and reduces errors and time wasted from bad input -* The GraphiQL user interface allows a user to easily explore the schema and all the information available to them - -## GraphQL API Quick Start - -To get started right away: - -* Sign in to the Veritone platform at https://developer.veritone.com -* Go to the GraphiQL user interface at https://api.veritone.com/v3/graphiql. You will automatically be authenticated through to the API and have access to live data for your organization. - -* Try a simple query such as me and use `ctrl`+`space` (or `command`+`space` for Mac) hints to see the fields available. -* See static schema documentation at https://api.veritone.com/v3/graphqldocs/ -* Hit the API directly at https://api.veritone.com/v3/graphql. You'll need to provide a valid authentication token as described at [here](/apis/authentication). -* See [Basics](/apis/tutorials/graphql-basics) and [Examples](/apis/tutorials/) - -## How do I use GraphQL? - -For detailed information about GraphQL, see the main GraphQL site (http://graphql.org). -This document contains a high-level overview and some information specific to Veritone's implementation. - -A GraphQL _schema_ defines _types_ and _fields_ on those types. That's it. A schema defines a set of queries as entry points to the data model. In a typical schema you will see something like this: - -```graphql -schema { - query: Query -} -type Query { - objects(objectName: String): [MyObjectType] -} -type MyObjectType { - name: String -} -``` - -However, that's just convention -- a query is, in the schema, just a field on a type. In this example, `query` is the root of our schema. -To query, you send a string in GraphQL that specifies the fields you want, and any parameters on them. So for this simple schema we might send: - -```graphql -query { - objects(objectName: "my sample object?") { - name - } -} -``` - -Here we are specifying the top-level field on the schema (`query`),and asking for one field (`objects`). `objects` is a field that takes a single optional parameter, `objectName`, and returns a list of `MyObjectType`. We've passed in the parameter `objectName: "my sample object"`. -It's up to the server how to resolve each requested field and interpret the parameters. All data might reside in a single back-end database schema. On the other hand, the server might populate some fields by reaching out to other sources such as REST services or even external services. -Take this example: - -```graphql -type MyObjectType { - name: String data: ObjectData -} -type ObjectData { - ... -} -``` - -`name` and the list of `MyObjectType` might live in one backend database, while the server populates data for each object by calling out to an external REST service. -To modify data, you use a _mutation_. A mutation is formatted like a query and can return information like a query, but modifies data in some way (create, update, or delete). -Mutations take, as parameters, special types called inputs. An input contains those fields necessary to create or modify an instance of the related type. -Continuing our example, we might have - -```graphql -type MyObjectType { - # A name for the object, provided by the user. Required. - name: String! - # A unique and invariant server-defined ID for the object - id: ID! -} -input CreateMyObject { - # Provide a name for your object name: String! -} -mutation { - createMyObject(input: CreateMyObject!): MyObjectType -} -``` - -We've defined an object type that has some fields provided by the user, and some fields controlled by the server. Our input type contains only those fields that can (and must) be provided by the user. Our mutation take the input type and returns a new object. - -```graphql -mutation { - createMyObject(input: { - name: "my new object" - }) { - id - name - } -} -``` - -It would return something like: - -```json -{ - "data": { - "createMyObject": { - "id": "37dbf368-7c76-45c8-8b96-c1e90b0c5ec2", - "name": "my new object" - } - } -} -``` - -## Veritone's schema - -Now let's take a look at the Veritone schema. You can refer to the following links. You can access the GraphiQL user interface at https://api.veritone.com/v3/graphiql. - -If you're new to our schema, try using the me query to explore the data you have access to. - -```graphql -query { - me { - id - name - # use to get a list of fields available at each level - } -} -``` - -Queries that support paging all use a consistent set of parameters and return fields based on the interface `Page`. You specify an optional offset and limit as parameters, and receive back an object with fields `offset`, `limit`, `count` (number of objects actually returned), and `records` containing the list of results. - -Veritone's platform is full-featured and growing constantly, so the schema is necessarily complex. For most users, the best way to explore it is interactively using GraphiQL. You can also consult the static documentation at https://api.veritone.com/v3/graphqldocs/ or review the high-level entity relationship diagram below. - -You can find raw GraphQL schema files [here](/apis/schema/listing) . - -## GraphQL hints and tips - -Use GraphiQL to explore the schema, make ad hoc queries to find data, and test and debug queries and mutations for your applications. - -You can request as many fields as you want on a given type -- including the top-level query. Thus, you can, effectively, make multiple queries in a single request. - -GraphQL lets you structure complex queries that retrieve, aggregate, and marshal large amounts of data. Use with caution. Complex queries over large data sets can affect performance. - -You can retrieve the same field twice, say to apply different parameters, using aliases. For example: - -```graphql -query { - firstUser: user(name:"smith") { - ... - } - secondUser: user(name:"jones") { - ... - } - thirdUser: user(id:"...") { - ... - } -} -``` - -GraphQL supports interfaces, which define common fields for a set of types, and unions, which specify a grouping of types with no common fields. A field that uses an interface or union can include any of several types. All types that implement a given interface have common fields that can be requested in the usual way.However, to request a type-specific field you must use a fragment. Here is an example using `TemporalDataObject`, which accepts an interface, `Metadata` in its `metadata` field. We'll ask for a common field, `name`, along with some fields that are specific to `Program`. - -```graphql -{ - temporalDataObjects(limit: 30) { - records { - id - metadata { - name - ...on Program { - image - liveImage - } - } - } - } -} -``` - -The result will look something like this: - -```json -{ - "data": { - "temporalDataObjects": { - "records": [ - { - "id": "21098452", - "metadata": [], - "status": "uploaded", - "startDateTime": 0, - "stopDateTime": 0 - }, - { - "id": "21098441", - "metadata": [], - "status": null, - "startDateTime": 0, - "stopDateTime": 0 - }, - { - "id": "21098183", - "metadata": [ - { - "name": "CloneData", - "__typename": "CloneData" - }, - { - "__typename": "FileData", - "name": "FileData" - }, - { - "__typename": "JSONObject", - "name": "veritone-permissions" - }, - { - "name": "Program", - "__typename": "Program", - "image": "", - "liveImage": "https://s3.amazonaws.com/dev-veritone-ugc/demo_temp%2FprogramImageURL%2Fo8poaodeRpmUJBNK3tKT_cw.png" - } - ] - } - ] - } - } -} -``` diff --git a/docs/architecture/Refactoring.md b/docs/architecture/Refactoring.md new file mode 100644 index 0000000000..e5c9c48acb --- /dev/null +++ b/docs/architecture/Refactoring.md @@ -0,0 +1,35 @@ +# 重构 + +> 重构是在不改变软件可观察行为的前提下改善其内部结构 + + + +重构（名词）：对软件内部结构的一种调整，目的是在不改变「软件之可察行为」前提下，提高其可理解性，降低其修改成本。 + + + +重构（动词）：使用一系列重构准则（手法〕，在不改变「软件之可察行为」前提 下，调整其结构。 + + + + + + + + + +## 测试 + +测试驱动开发（Test-Driven Development,TDD） + + + + + +做法 + +创造一个新函数，根据这个函数的意图来对它命名（以它”做什么“来命名，而不是以它”怎么做“命名） + + + +《重构 改善既有代码的设计》 \ No newline at end of file diff --git "a/docs/architecture/\347\247\222\346\235\200\347\263\273\347\273\237\350\256\276\350\256\241.md" "b/docs/architecture/\347\247\222\346\235\200\347\263\273\347\273\237\350\256\276\350\256\241.md" new file mode 100755 index 0000000000..bd47b14253 --- /dev/null +++ "b/docs/architecture/\347\247\222\346\235\200\347\263\273\347\273\237\350\256\276\350\256\241.md" @@ -0,0 +1,26 @@ +**秒杀系统本质上就是一个满足大并发、高性能和高可用的分布式系统**。 + +## 架构原则：“4 要 1 不要” + +1. 数据要尽量少 + + 所谓“数据要尽量少”，首先是指用户请求的数据能少就少。请求的数据包括上传给系统的数据和系统返回给用户 的数据（通常就是网页）。 + +2. 请求数要尽量少 +3. 路径要尽量短 +4. 依赖要尽量少 +5. 不要有单点 + + + +# 动静分离 + +所谓“动静分离”，其实就是把用户请求的数据（如 HTML 页面）划分为“动态数据”和“静态数据”。 + +**第一，你应该把静态数据缓存到离用户最近的地方**。静态数据就是那些相对不会变化的数据，因此我们可以把它们缓存起来。缓存到哪里呢？常见的就三种，用户浏览器里、CDN 上或者在服务端的 Cache 中。你应该根据情况，把它们尽量缓存到离用户最近的地方。 + +**第二，静态化改造就是要直接缓存 HTTP 连接**。相较于普通的数据缓存而言，你肯定还听过系统的静态化改造。静态化改造是直接缓存 HTTP 连接而不是仅仅缓存数据，如下图所示，Web 代理服务器根据请求 URL，直接取出对应的 HTTP 响应头和响应体然后直接返回，这个响应过程简单得连 HTTP 协议都不用重新组装，甚至连 HTTP 请求头也不需要解析。 + +![](https://static001.geekbang.org/resource/image/2c/46/2c608715621afc9c95570dce00a87546.jpg) + +第三，让谁来缓存静态数据也很重要。不同语言写的 Cache 软件处理缓存数据的效率也各不相同。以 Java 为例，因为 Java 系统本身也有其弱点（比如不擅长处理大量连接请求，每个连接消耗的内存较多，Servlet 容器解析 HTTP 协议较慢），所以你可以不在 Java 层做缓存，而是直接在 Web 服务器层上做，这样你就可以屏蔽 Java 语言层面的一些弱点；而相比起来，Web 服务器（如 Nginx、Apache、Varnish）也更擅长处理大并发的静态文件请求。 \ No newline at end of file diff --git "a/docs/assets/css/theme-simple - \345\211\257\346\234\254.css" "b/docs/assets/css/theme-simple - \345\211\257\346\234\254.css" deleted file mode 100644 index c01c631844..0000000000 --- "a/docs/assets/css/theme-simple - \345\211\257\346\234\254.css" +++ /dev/null @@ -1,2 +0,0 @@ -.github-corner{position:absolute;z-index:40;top:0;right:0;border-bottom:0;text-decoration:none}.github-corner svg{height:70px;width:70px;fill:var(--theme-color);color:var(--base-background-color)}.github-corner:hover .octo-arm{-webkit-animation:octocat-wave 560ms ease-in-out;animation:octocat-wave 560ms ease-in-out}@-webkit-keyframes octocat-wave{0%,100%{-webkit-transform:rotate(0);transform:rotate(0)}20%,60%{-webkit-transform:rotate(-25deg);transform:rotate(-25deg)}40%,80%{-webkit-transform:rotate(10deg);transform:rotate(10deg)}}@keyframes octocat-wave{0%,100%{-webkit-transform:rotate(0);transform:rotate(0)}20%,60%{-webkit-transform:rotate(-25deg);transform:rotate(-25deg)}40%,80%{-webkit-transform:rotate(10deg);transform:rotate(10deg)}}.progress{position:fixed;z-index:60;top:0;left:0;right:0;height:3px;width:0;background-color:var(--theme-color);transition:width var(--duration-fast),opacity calc(var(--duration-fast) * 2)}body.ready-transition:after,body.ready-transition>*:not(.progress){opacity:0;transition:opacity var(--spinner-transition-duration)}body.ready-transition:after{content:'';position:absolute;z-index:1000;top:calc(50% - (var(--spinner-size) / 2));left:calc(50% - (var(--spinner-size) / 2));height:var(--spinner-size);width:var(--spinner-size);border:var(--spinner-track-width, 0) solid var(--spinner-track-color);border-left-color:var(--theme-color);border-left-color:var(--theme-color);border-radius:50%;-webkit-animation:spinner var(--duration-slow) infinite linear;animation:spinner var(--duration-slow) infinite linear}body.ready-transition.ready-spinner:after{opacity:1}body.ready-transition.ready-fix:after{opacity:0}body.ready-transition.ready-fix>*:not(.progress){opacity:1;transition-delay:var(--spinner-transition-duration)}@-webkit-keyframes spinner{0%{-webkit-transform:rotate(0deg);transform:rotate(0deg)}100%{-webkit-transform:rotate(360deg);transform:rotate(360deg)}}@keyframes spinner{0%{-webkit-transform:rotate(0deg);transform:rotate(0deg)}100%{-webkit-transform:rotate(360deg);transform:rotate(360deg)}}*,*:before,*:after{box-sizing:inherit;font-size:inherit;-webkit-overflow-scrolling:touch;-webkit-tap-highlight-color:rgba(0,0,0,0);-webkit-text-size-adjust:none;-webkit-touch-callout:none}:root{box-sizing:border-box;background-color:var(--base-background-color);font-size:var(--base-font-size);font-weight:var(--base-font-weight);line-height:var(--base-line-height);letter-spacing:var(--base-letter-spacing);color:var(--base-color);-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;font-smoothing:antialiased}html,button,input,optgroup,select,textarea{font-family:var(--base-font-family)}button,input,optgroup,select,textarea{font-size:100%;margin:0}a{text-decoration:none;-webkit-text-decoration-skip:ink;text-decoration-skip-ink:auto}body{margin:0}hr{height:0;margin:2em 0;border:none;border-bottom:var(--hr-border, 0)}img{border:0}main{display:block}main.hidden{display:none}mark{background:var(--mark-background);color:var(--mark-color)}pre{font-family:var(--pre-font-family);font-size:var(--pre-font-size);font-weight:var(--pre-font-weight);line-height:var(--pre-line-height)}small{display:inline-block;font-size:var(--small-font-size)}strong{font-weight:var(--strong-font-weight);color:var(--strong-color, currentColor)}sub,sup{font-size:var(--subsup-font-size);line-height:0;position:relative;vertical-align:baseline}sub{bottom:-0.25em}sup{top:-0.5em}body:not([data-platform^="Mac"]) *{scrollbar-color:hsla(var(--mono-hue), var(--mono-saturation), 50%, 0.3) hsla(var(--mono-hue), var(--mono-saturation), 50%, 0.1);scrollbar-width:thin}body:not([data-platform^="Mac"]) * ::-webkit-scrollbar{width:5px;height:5px}body:not([data-platform^="Mac"]) * ::-webkit-scrollbar-thumb{background:hsla(var(--mono-hue), var(--mono-saturation), 50%, 0.3)}body:not([data-platform^="Mac"]) * ::-webkit-scrollbar-track{background:hsla(var(--mono-hue), var(--mono-saturation), 50%, 0.1)}::selection{background:var(--selection-color)}.emoji{height:var(--emoji-size);vertical-align:middle}.task-list-item{list-style:none}.task-list-item input{margin-right:0.5em;margin-left:0;vertical-align:0.075em}.markdown-section code[class*="lang-"],.markdown-section pre[data-lang]{font-family:var(--code-font-family);font-size:var(--code-font-size);font-weight:var(--code-font-weight);letter-spacing:normal;line-height:var(--code-block-line-height);-moz-tab-size:var(--code-tab-size);-o-tab-size:var(--code-tab-size);tab-size:var(--code-tab-size);text-align:left;white-space:pre;word-spacing:normal;word-wrap:normal;word-break:normal;-webkit-hyphens:none;-ms-hyphens:none;hyphens:none}.markdown-section pre[data-lang]{position:relative;overflow:hidden;margin:var(--code-block-margin);padding:0;border-radius:var(--code-block-border-radius)}.markdown-section pre[data-lang]::after{content:attr(data-lang);position:absolute;top:0.75em;right:0.75em;opacity:0.6;color:inherit;font-size:var(--font-size-s);line-height:1}.markdown-section pre[data-lang] code{display:block;overflow:auto;padding:var(--code-block-padding)}code[class*="lang-"],pre[data-lang]{color:var(--code-theme-text)}pre[data-lang]::selection,pre[data-lang] ::selection,code[class*="lang-"]::selection,code[class*="lang-"] ::selection{background:var(--code-theme-selection, var(--selection-color))}:not(pre)>code[class*="lang-"],pre[data-lang]{background:var(--code-theme-background)}.namespace{opacity:0.7}.token.comment,.token.prolog,.token.doctype,.token.cdata{color:var(--code-theme-comment)}.token.punctuation{color:var(--code-theme-punctuation)}.token.property,.token.tag,.token.boolean,.token.number,.token.constant,.token.symbol,.token.deleted{color:var(--code-theme-tag)}.token.selector,.token.attr-name,.token.string,.token.char,.token.builtin,.token.inserted{color:var(--code-theme-selector)}.token.operator,.token.entity,.token.url,.language-css .token.string,.style .token.string{color:var(--code-theme-operator)}.token.atrule,.token.attr-value,.token.keyword{color:var(--code-theme-keyword)}.token.function{color:var(--code-theme-function)}.token.regex,.token.important,.token.variable{color:var(--code-theme-variable)}.token.important,.token.bold{font-weight:bold}.token.italic{font-style:italic}.token.entity{cursor:help}.markdown-section{position:relative;max-width:var(--content-max-width);margin:0 auto;padding:2rem 45px}.app-nav:not(:empty) ~ main .markdown-section{padding-top:3.5rem}.markdown-section figure,.markdown-section p,.markdown-section ol,.markdown-section ul{margin:1em 0}.markdown-section ol,.markdown-section ul{padding-left:1.5rem}.markdown-section ol ol,.markdown-section ol ul,.markdown-section ul ol,.markdown-section ul ul{margin-top:0.15rem;margin-bottom:0.15rem}.markdown-section a{border-bottom:var(--link-border-bottom);color:var(--link-color);-webkit-text-decoration:var(--link-text-decoration);text-decoration:var(--link-text-decoration);-webkit-text-decoration-color:var(--link-text-decoration-color);text-decoration-color:var(--link-text-decoration-color)}.markdown-section a:hover{border-bottom:var(--link-border-bottom--hover, var(--link-border-bottom, 0));color:var(--link-color--hover, var(--link-color));-webkit-text-decoration:var(--link-text-decoration--hover, var(--link-text-decoration));text-decoration:var(--link-text-decoration--hover, var(--link-text-decoration));-webkit-text-decoration-color:var(--link-text-decoration-color--hover, var(--link-text-decoration-color));text-decoration-color:var(--link-text-decoration-color--hover, var(--link-text-decoration-color))}.markdown-section a.anchor{border-bottom:0;color:inherit;text-decoration:none}.markdown-section a.anchor:hover{text-decoration:underline}.markdown-section blockquote{overflow:visible;margin:2em 0;padding:1.5em;border-width:var(--blockquote-border-width, 0);border-style:var(--blockquote-border-style);border-color:var(--blockquote-border-color);border-radius:var(--blockquote-border-radius);background:var(--blockquote-background);color:var(--blockquote-color);font-family:var(--blockquote-font-family);font-size:var(--blockquote-font-size);font-style:var(--blockquote-font-style);font-weight:var(--blockquote-font-weight);quotes:"“" "”" "‘" "’"}.markdown-section blockquote em{font-family:var(--blockquote-em-font-family);font-size:var(--blockquote-em-font-size);font-style:var(--blockquote-em-font-style);font-weight:var(--blockquote-em-font-weight)}.markdown-section blockquote p:first-child{margin-top:0}.markdown-section blockquote p:first-child:before,.markdown-section blockquote p:first-child:after{color:var(--blockquote-quotes-color);font-family:var(--blockquote-quotes-font-family);font-size:var(--blockquote-quotes-font-size);line-height:0}.markdown-section blockquote p:first-child:before{content:var(--blockquote-quotes-open);margin-right:0.15em;vertical-align:-0.45em}.markdown-section blockquote p:first-child:after{content:var(--blockquote-quotes-close);margin-left:0.15em;vertical-align:-0.55em}.markdown-section blockquote p:last-child{margin-bottom:0}.markdown-section code{font-family:var(--code-font-family);font-size:var(--code-font-size);font-weight:var(--code-font-weight);line-height:inherit}.markdown-section code:not([class*="lang-"]):not([class*="language-"]){margin:var(--code-inline-margin);padding:var(--code-inline-padding);border-radius:var(--code-inline-border-radius);background:var(--code-inline-background);color:var(--code-inline-color, currentColor);white-space:nowrap}.markdown-section h1:first-child,.markdown-section h2:first-child,.markdown-section h3:first-child,.markdown-section h4:first-child,.markdown-section h5:first-child,.markdown-section h6:first-child{margin-top:0}.markdown-section h1+h2,.markdown-section h1+h3,.markdown-section h1+h4,.markdown-section h1+h5,.markdown-section h1+h6,.markdown-section h2+h3,.markdown-section h2+h4,.markdown-section h2+h5,.markdown-section h2+h6,.markdown-section h3+h4,.markdown-section h3+h5,.markdown-section h3+h6,.markdown-section h4+h5,.markdown-section h4+h6,.markdown-section h5+h6{margin-top:1rem;font-size: 1rem;}.markdown-section h1{margin:var(--heading-h1-margin, var(--heading-margin));padding:var(--heading-h1-padding, var(--heading-padding));border-width:var(--heading-h1-border-width, 0);border-style:var(--heading-h1-border-style);border-color:var(--heading-h1-border-color);font-family:var(--heading-h1-font-family, var(--heading-font-family));font-size:var(--heading-h1-font-size);font-weight:var(--heading-h1-font-weight, var(--heading-font-weight));line-height:var(--base-line-height);color:var(--heading-h1-color, var(--heading-color))}.markdown-section h2{margin:var(--heading-h2-margin, var(--heading-margin));padding:var(--heading-h2-padding, var(--heading-padding));border-width:var(--heading-h2-border-width, 0);border-style:var(--heading-h2-border-style);border-color:var(--heading-h2-border-color);font-family:var(--heading-h2-font-family, var(--heading-font-family));font-size:var(--heading-h2-font-size);font-weight:var(--heading-h2-font-weight, var(--heading-font-weight));line-height:var(--base-line-height);color:var(--heading-h2-color, var(--heading-color))}.markdown-section h3{margin:var(--heading-h3-margin, var(--heading-margin));padding:var(--heading-h3-padding, var(--heading-padding));border-width:var(--heading-h3-border-width, 0);border-style:var(--heading-h3-border-style);border-color:var(--heading-h3-border-color);font-family:var(--heading-h3-font-family, var(--heading-font-family));font-size:var(--heading-h3-font-size);font-weight:var(--heading-h3-font-weight, var(--heading-font-weight));color:var(--heading-h3-color, var(--heading-color))}.markdown-section h4{margin:var(--heading-h4-margin, var(--heading-margin));padding:var(--heading-h4-padding, var(--heading-padding));border-width:var(--heading-h4-border-width, 0);border-style:var(--heading-h4-border-style);border-color:var(--heading-h4-border-color);font-family:var(--heading-h4-font-family, var(--heading-font-family));font-size:var(--heading-h4-font-size);font-weight:var(--heading-h4-font-weight, var(--heading-font-weight));color:var(--heading-h4-color, var(--heading-color))}.markdown-section h5{margin:var(--heading-h5-margin, var(--heading-margin));padding:var(--heading-h5-padding, var(--heading-padding));border-width:var(--heading-h5-border-width, 0);border-style:var(--heading-h5-border-style);border-color:var(--heading-h5-border-color);font-family:var(--heading-h5-font-family, var(--heading-font-family));font-size:var(--heading-h5-font-size);font-weight:var(--heading-h5-font-weight, var(--heading-font-weight));color:var(--heading-h5-color, var(--heading-color))}.markdown-section h6{margin:var(--heading-h6-margin, var(--heading-margin));padding:var(--heading-h6-padding, var(--heading-padding));border-width:var(--heading-h6-border-width, 0);border-style:var(--heading-h6-border-style);border-color:var(--heading-h6-border-color);font-family:var(--heading-h6-font-family, var(--heading-font-family));font-size:var(--heading-h6-font-size);font-weight:var(--heading-h6-font-weight, var(--heading-font-weight));color:var(--heading-h6-color, var(--heading-color))}.markdown-section iframe{margin:1em 0}.markdown-section img{max-width:100%}.markdown-section kbd{display:inline-block;min-width:var(--kbd-min-width);margin:var(--kbd-margin);padding:var(--kbd-padding);border:var(--kbd-border);border-radius:var(--kbd-border-radius);background:var(--kbd-background);font-family:inherit;font-size:var(--kbd-font-size);text-align:center;letter-spacing:0;line-height:1;color:var(--kbd-color)}.markdown-section kbd+kbd{margin-left:-0.15em}.markdown-section table{display:block;overflow:auto;margin:1rem 0;border-spacing:0;border-collapse:collapse}.markdown-section th,.markdown-section td{padding:var(--table-cell-padding)}.markdown-section th:not([align]){text-align:left}.markdown-section thead{border-color:var(--table-head-border-color);border-style:solid;border-width:var(--table-head-border-width, 0);background:var(--table-head-background)}.markdown-section th{font-weight:var(--table-head-font-weight);color:var(--strong-color)}.markdown-section td{border-color:var(--table-cell-border-color);border-style:solid;border-width:var(--table-cell-border-width, 0)}.markdown-section tbody{border-color:var(--table-body-border-color);border-style:solid;border-width:var(--table-body-border-width, 0)}.markdown-section tbody tr:nth-child(odd){background:var(--table-row-odd-background)}.markdown-section tbody tr:nth-child(even){background:var(--table-row-even-background)}.markdown-section>ul .task-list-item{margin-left:-1.25em}.markdown-section>ul .task-list-item .task-list-item{margin-left:0}.markdown-section .table-wrapper table{display:table;width:100%}.markdown-section .table-wrapper td::before{display:none}@media (max-width: 30em){.markdown-section .table-wrapper tbody,.markdown-section .table-wrapper tr,.markdown-section .table-wrapper td{display:block}.markdown-section .table-wrapper th,.markdown-section .table-wrapper td{border:none}.markdown-section .table-wrapper thead{display:none}.markdown-section .table-wrapper tr{border-color:var(--table-cell-border-color);border-style:solid;border-width:var(--table-cell-border-width, 0);padding:var(--table-cell-padding)}.markdown-section .table-wrapper tr:not(:last-child){border-bottom:0}.markdown-section .table-wrapper td{display:flex;padding:0.15em 0}.markdown-section .table-wrapper td::before{display:block;min-width:8em;max-width:8em;font-weight:bold;text-align:left}}.markdown-section .tip,.markdown-section .warn{position:relative;margin:2em 0;padding:var(--notice-padding);border-width:var(--notice-border-width, 0);border-style:var(--notice-border-style);border-color:var(--notice-border-color);border-radius:var(--notice-border-radius);background:var(--notice-background);font-family:var(--notice-font-family);font-weight:var(--notice-font-weight);color:var(--notice-color)}.markdown-section .tip:before,.markdown-section .warn:before{display:inline-block;position:var(--notice-before-position, relative);top:var(--notice-before-top);left:var(--notice-before-left);height:var(--notice-before-height);width:var(--notice-before-width);margin:var(--notice-before-margin);padding:var(--notice-before-padding);border-radius:var(--notice-before-border-radius);line-height:var(--notice-before-line-height);font-family:var(--notice-before-font-family);font-size:var(--notice-before-font-size);font-weight:var(--notice-before-font-weight);text-align:center}.markdown-section .tip{border-width:var(--notice-important-border-width, var(--notice-border-width, 0));border-style:var(--notice-important-border-style, var(--notice-border-style));border-color:var(--notice-important-border-color, var(--notice-border-color));background:var(--notice-important-background, var(--notice-background));color:var(--notice-important-color, var(--notice-color))}.markdown-section .tip:before{content:var(--notice-important-before-content, var(--notice-before-content));background:var(--notice-important-before-background, var(--notice-before-background));color:var(--notice-important-before-color, var(--notice-before-color))}.markdown-section .warn{border-width:var(--notice-tip-border-width, var(--notice-border-width, 0));border-style:var(--notice-tip-border-style, var(--notice-border-style));border-color:var(--notice-tip-border-color, var(--notice-border-color));background:var(--notice-tip-background, var(--notice-background));color:var(--notice-tip-color, var(--notice-color))}.markdown-section .warn:before{content:var(--notice-tip-before-content, var(--notice-before-content));background:var(--notice-tip-before-background, var(--notice-before-background));color:var(--notice-tip-before-color, var(--notice-before-color))}.cover{display:none;position:relative;z-index:20;min-height:100vh;flex-direction:column;align-items:center;justify-content:center;padding:calc(var(--cover-border-inset, 0px) + var(--cover-border-width, 0px));color:var(--cover-color);text-align:var(--cover-text-align)}@media screen and (-ms-high-contrast: active), screen and (-ms-high-contrast: none){.cover{height:100vh}}.cover:before,.cover:after{content:'';position:absolute}.cover:before{top:0;bottom:0;left:0;right:0;background-blend-mode:var(--cover-background-blend-mode);background-color:var(--cover-background-color);background-image:var(--cover-background-image);background-position:var(--cover-background-position);background-repeat:var(--cover-background-repeat);background-size:var(--cover-background-size)}.cover:after{top:var(--cover-border-inset, 0);bottom:var(--cover-border-inset, 0);left:var(--cover-border-inset, 0);right:var(--cover-border-inset, 0);border-width:var(--cover-border-width, 0);border-style:solid;border-color:var(--cover-border-color)}.cover a{border-bottom:var(--cover-link-border-bottom);color:var(--cover-link-color);-webkit-text-decoration:var(--cover-link-text-decoration);text-decoration:var(--cover-link-text-decoration);-webkit-text-decoration-color:var(--cover-link-text-decoration-color);text-decoration-color:var(--cover-link-text-decoration-color)}.cover a:hover{border-bottom:var(--cover-link-border-bottom--hover, var(--cover-link-border-bottom));color:var(--cover-link-color--hover, var(--cover-link-color));-webkit-text-decoration:var(--cover-link-text-decoration--hover, var(--cover-link-text-decoration));text-decoration:var(--cover-link-text-decoration--hover, var(--cover-link-text-decoration));-webkit-text-decoration-color:var(--cover-link-text-decoration-color--hover, var(--cover-link-text-decoration-color));text-decoration-color:var(--cover-link-text-decoration-color--hover, var(--cover-link-text-decoration-color))}.cover h1{color:var(--cover-heading-color);position:relative;margin:0;font-size:var(--cover-heading-font-size);font-weight:var(--cover-heading-font-weight);line-height:1.2}.cover h1 a,.cover h1 a:hover{display:block;border-bottom:none;color:inherit;text-decoration:none}.cover h1 small{position:absolute;bottom:0;margin-left:0.5em}.cover h1 span{font-size:calc(var(--cover-heading-font-size-min) * 1px)}@media (min-width: 26em){.cover h1 span{font-size:calc((var(--cover-heading-font-size-min) * 1px) + (var(--cover-heading-font-size-max) - var(--cover-heading-font-size-min)) * ((100vw - 420px) / (1024 - 420)))}}@media (min-width: 64em){.cover h1 span{font-size:calc(var(--cover-heading-font-size-max) * 1px)}}.cover blockquote{margin:0;color:var(--cover-blockquote-color);font-size:var(--cover-blockquote-font-size)}.cover blockquote a{color:inherit}.cover ul{padding:0;list-style-type:none}.cover .cover-main{position:relative;z-index:1;max-width:var(--cover-max-width);margin:var(--cover-margin);padding:0 45px}.cover .cover-main>p:last-child{margin:1.25em -.25em}.cover .cover-main>p:last-child a{display:block;margin:.375em .25em;padding:var(--cover-button-padding);border:var(--cover-button-border);border-radius:var(--cover-button-border-radius);box-shadow:var(--cover-button-box-shadow);background:var(--cover-button-background);text-align:center;-webkit-text-decoration:var(--cover-button-text-decoration);text-decoration:var(--cover-button-text-decoration);-webkit-text-decoration-color:var(--cover-button-text-decoration-color);text-decoration-color:var(--cover-button-text-decoration-color);color:var(--cover-button-color);white-space:nowrap;transition:var(--cover-button-transition)}.cover .cover-main>p:last-child a:hover{border:var(--cover-button-border--hover, var(--cover-button-border));box-shadow:var(--cover-button-box-shadow--hover, var(--cover-button-box-shadow));background:var(--cover-button-background--hover, var(--cover-button-background));-webkit-text-decoration:var(--cover-button-text-decoration--hover, var(--cover-button-text-decoration));text-decoration:var(--cover-button-text-decoration--hover, var(--cover-button-text-decoration));-webkit-text-decoration-color:var(--cover-button-text-decoration-color--hover, var(--cover-button-text-decoration-color));text-decoration-color:var(--cover-button-text-decoration-color--hover, var(--cover-button-text-decoration-color));color:var(--cover-button-color--hover, var(--cover-button-color))}.cover .cover-main>p:last-child a:first-child{border:var(--cover-button-primary-border, var(--cover-button-border));box-shadow:var(--cover-button-primary-box-shadow, var(--cover-button-box-shadow));background:var(--cover-button-primary-background, var(--cover-button-background));-webkit-text-decoration:var(--cover-button-primary-text-decoration, var(--cover-button-text-decoration));text-decoration:var(--cover-button-primary-text-decoration, var(--cover-button-text-decoration));-webkit-text-decoration-color:var(--cover-button-primary-text-decoration-color, var(--cover-button-text-decoration-color));text-decoration-color:var(--cover-button-primary-text-decoration-color, var(--cover-button-text-decoration-color));color:var(--cover-button-primary-color, var(--cover-button-color))}.cover .cover-main>p:last-child a:first-child:hover{border:var(--cover-button-primary-border--hover, var(--cover-button-border--hover, var(--cover-button-primary-border, var(--cover-button-border))));box-shadow:var(--cover-button-primary-box-shadow--hover, var(--cover-button-box-shadow--hover, var(--cover-button-primary-box-shadow, var(--cover-button-box-shadow))));background:var(--cover-button-primary-background--hover, var(--cover-button-background--hover, var(--cover-button-primary-background, var(--cover-button-background))));-webkit-text-decoration:var(--cover-button-primary-text-decoration--hover, var(--cover-button-text-decoration--hover, var(--cover-button-primary-text-decoration, var(--cover-button-text-decoration))));text-decoration:var(--cover-button-primary-text-decoration--hover, var(--cover-button-text-decoration--hover, var(--cover-button-primary-text-decoration, var(--cover-button-text-decoration))));-webkit-text-decoration-color:var(--cover-button-primary-text-decoration-color--hover, var(--cover-button-text-decoration-color--hover, var(--cover-button-primary-text-decoration-color, var(--cover-button-text-decoration-color))));text-decoration-color:var(--cover-button-primary-text-decoration-color--hover, var(--cover-button-text-decoration-color--hover, var(--cover-button-primary-text-decoration-color, var(--cover-button-text-decoration-color))));color:var(--cover-button-primary-color--hover, var(--cover-button-color--hover, var(--cover-button-primary-color, var(--cover-button-color))))}@media (min-width: 30.01em){.cover .cover-main>p:last-child a{display:inline-block}}.cover .mask{visibility:var(--cover-background-mask-visibility, hidden);position:absolute;top:0;bottom:0;left:0;right:0;background-color:var(--cover-background-mask-color);opacity:var(--cover-background-mask-opacity)}.cover.has-mask .mask{visibility:visible}.cover.show{display:flex}.app-nav{position:absolute;z-index:30;top:calc(35px - (0.5em * var(--base-line-height)));left:45px;right:80px;text-align:right}.app-nav.no-badge{right:45px}.app-nav li>img,.app-nav li>a>img{margin-top:-0.25em;vertical-align:middle}.app-nav li>img:first-child,.app-nav li>a>img:first-child{margin-right:0.5em}.app-nav ul,.app-nav li{margin:0;padding:0;list-style:none}.app-nav li{position:relative}.app-nav li a{display:block;line-height:1;transition:var(--navbar-root-transition)}.app-nav>ul>li{display:inline-block;margin:var(--navbar-root-margin)}.app-nav>ul>li:first-child{margin-left:0}.app-nav>ul>li:last-child{margin-right:0}.app-nav>ul>li>a,.app-nav>ul>li>span{padding:var(--navbar-root-padding);border-width:var(--navbar-root-border-width, 0);border-style:var(--navbar-root-border-style);border-color:var(--navbar-root-border-color);border-radius:var(--navbar-root-border-radius);background:var(--navbar-root-background);color:var(--navbar-root-color);-webkit-text-decoration:var(--navbar-root-text-decoration);text-decoration:var(--navbar-root-text-decoration);-webkit-text-decoration-color:var(--navbar-root-text-decoration-color);text-decoration-color:var(--navbar-root-text-decoration-color)}.app-nav>ul>li>a:hover,.app-nav>ul>li>span:hover{background:var(--navbar-root-background--hover, var(--navbar-root-background));border-style:var(--navbar-root-border-style--hover, var(--navbar-root-border-style));border-color:var(--navbar-root-border-color--hover, var(--navbar-root-border-color));color:var(--navbar-root-color--hover, var(--navbar-root-color));-webkit-text-decoration:var(--navbar-root-text-decoration--hover, var(--navbar-root-text-decoration));text-decoration:var(--navbar-root-text-decoration--hover, var(--navbar-root-text-decoration));-webkit-text-decoration-color:var(--navbar-root-text-decoration-color--hover, var(--navbar-root-text-decoration-color));text-decoration-color:var(--navbar-root-text-decoration-color--hover, var(--navbar-root-text-decoration-color))}.app-nav>ul>li>a:not(:last-child),.app-nav>ul>li>span:not(:last-child){padding:var(--navbar-menu-root-padding, var(--navbar-root-padding));background:var(--navbar-menu-root-background, var(--navbar-root-background))}.app-nav>ul>li>a:not(:last-child):hover,.app-nav>ul>li>span:not(:last-child):hover{background:var(--navbar-menu-root-background--hover, var(--navbar-menu-root-background, var(--navbar-root-background--hover, var(--navbar-root-background))))}.app-nav>ul>li>a.active{background:var(--navbar-root-background--active, var(--navbar-root-background));border-style:var(--navbar-root-border-style--active, var(--navbar-root-border-style));border-color:var(--navbar-root-border-color--active, var(--navbar-root-border-color));color:var(--navbar-root-color--active, var(--navbar-root-color));-webkit-text-decoration:var(--navbar-root-text-decoration--active, var(--navbar-root-text-decoration));text-decoration:var(--navbar-root-text-decoration--active, var(--navbar-root-text-decoration));-webkit-text-decoration-color:var(--navbar-root-text-decoration-color--active, var(--navbar-root-text-decoration-color));text-decoration-color:var(--navbar-root-text-decoration-color--active, var(--navbar-root-text-decoration-color))}.app-nav>ul>li>a.active:not(:last-child):hover{background:var(--navbar-menu-root-background--active, var(--navbar-menu-root-background, var(--navbar-root-background--active, var(--navbar-root-background))))}.app-nav>ul>li ul{visibility:hidden;position:absolute;top:100%;right:50%;overflow-y:auto;box-sizing:border-box;max-height:calc(50vh);padding:var(--navbar-menu-padding);border-width:var(--navbar-menu-border-width, 0);border-style:solid;border-color:var(--navbar-menu-border-color);border-radius:var(--navbar-menu-border-radius);background:var(--navbar-menu-background);box-shadow:var(--navbar-menu-box-shadow);text-align:left;white-space:nowrap;opacity:0;-webkit-transform:translate(50%, -0.35em);transform:translate(50%, -0.35em);transition:var(--navbar-menu-transition)}.app-nav>ul>li ul li{white-space:nowrap}.app-nav>ul>li ul a{margin:var(--navbar-menu-link-margin);padding:var(--navbar-menu-link-padding);border-width:var(--navbar-menu-link-border-width, 0);border-style:var(--navbar-menu-link-border-style);border-color:var(--navbar-menu-link-border-color);border-radius:var(--navbar-menu-link-border-radius);background:var(--navbar-menu-link-background);color:var(--navbar-menu-link-color);-webkit-text-decoration:var(--navbar-menu-link-text-decoration);text-decoration:var(--navbar-menu-link-text-decoration);-webkit-text-decoration-color:var(--navbar-menu-link-text-decoration-color);text-decoration-color:var(--navbar-menu-link-text-decoration-color)}.app-nav>ul>li ul a:hover{background:var(--navbar-menu-link-background--hover, var(--navbar-menu-link-background));border-style:var(--navbar-menu-link-border-style--hover, var(--navbar-menu-link-border-style));border-color:var(--navbar-menu-link-border-color--hover, var(--navbar-menu-link-border-color));color:var(--navbar-menu-link-color--hover, var(--navbar-menu-link-color));-webkit-text-decoration:var(--navbar-menu-link-text-decoration--hover, var(--navbar-menu-link-text-decoration));text-decoration:var(--navbar-menu-link-text-decoration--hover, var(--navbar-menu-link-text-decoration));-webkit-text-decoration-color:var(--navbar-menu-link-text-decoration-color--hover, var(--navbar-menu-link-text-decoration-color));text-decoration-color:var(--navbar-menu-link-text-decoration-color--hover, var(--navbar-menu-link-text-decoration-color))}.app-nav>ul>li ul a.active{background:var(--navbar-menu-link-background--active, var(--navbar-menu-link-background));border-style:var(--navbar-menu-link-border-style--active, var(--navbar-menu-link-border-style));border-color:var(--navbar-menu-link-border-color--active, var(--navbar-menu-link-border-color));color:var(--navbar-menu-link-color--active, var(--navbar-menu-link-color));-webkit-text-decoration:var(--navbar-menu-link-text-decoration--active, var(--navbar-menu-link-text-decoration));text-decoration:var(--navbar-menu-link-text-decoration--active, var(--navbar-menu-link-text-decoration));-webkit-text-decoration-color:var(--navbar-menu-link-text-decoration-color--active, var(--navbar-menu-link-text-decoration-color));text-decoration-color:var(--navbar-menu-link-text-decoration-color--active, var(--navbar-menu-link-text-decoration-color))}.app-nav>ul>li:hover ul,.app-nav>ul>li:focus ul,.app-nav>ul>li.focus-within ul{visibility:visible;opacity:1;-webkit-transform:translate(50%, 0);transform:translate(50%, 0)}.sidebar,.sidebar-toggle,main>.content{transition:all var(--sidebar-transition-duration) ease-out}@media (min-width: 48em){nav.app-nav{margin-left:var(--sidebar-width)}}main{position:relative;overflow-x:hidden;min-height:100vh}@media (min-width: 48em){main>.content{margin-left:var(--sidebar-width)}}.sidebar{display:flex;flex-direction:column;position:fixed;z-index:10;top:0;right:100%;overflow-x:hidden;overflow-y:auto;height:100vh;width:var(--sidebar-width);padding:var(--sidebar-padding);border-width:var(--sidebar-border-width);border-style:solid;border-color:var(--sidebar-border-color);background:var(--sidebar-background)}.sidebar>h1{margin:0;margin:var(--sidebar-name-margin);padding:var(--sidebar-name-padding);background:var(--sidebar-name-background);color:var(--sidebar-name-color);font-family:var(--sidebar-name-font-family);font-size:var(--sidebar-name-font-size);font-weight:var(--sidebar-name-font-weight);text-align:var(--sidebar-name-text-align)}.sidebar>h1 img{max-width:100%}.sidebar>h1 .app-name-link{color:var(--sidebar-name-color)}body:not([data-platform^="Mac"]) .sidebar::-webkit-scrollbar{width:5px}body:not([data-platform^="Mac"]) .sidebar::-webkit-scrollbar-thumb{border-radius:50vw}@media (min-width: 48em){.sidebar{position:absolute;-webkit-transform:translateX(var(--sidebar-width));transform:translateX(var(--sidebar-width))}}@media print{.sidebar{display:none}}.sidebar-nav,.sidebar nav{order:1;margin:var(--sidebar-nav-margin);padding:var(--sidebar-nav-padding);background:var(--sidebar-nav-background)}.sidebar-nav ul,.sidebar nav ul{margin:0;padding:0;list-style:none}.sidebar-nav ul ul,.sidebar nav ul ul{margin-left:var(--sidebar-nav-indent)}.sidebar-nav a,.sidebar nav a{display:block;overflow:hidden;margin:var(--sidebar-nav-link-margin);padding:var(--sidebar-nav-link-padding);border-width:var(--sidebar-nav-link-border-width, 0);border-style:var(--sidebar-nav-link-border-style);border-color:var(--sidebar-nav-link-border-color);border-radius:var(--sidebar-nav-link-border-radius);background-color:var(--sidebar-nav-link-background-color);background-image:var(--sidebar-nav-link-background-image);background-position:var(--sidebar-nav-link-background-position);background-repeat:var(--sidebar-nav-link-background-repeat);background-size:var(--sidebar-nav-link-background-size);color:var(--sidebar-nav-link-color);font-weight:var(--sidebar-nav-link-font-weight);white-space:nowrap;-webkit-text-decoration:var(--sidebar-nav-link-text-decoration);text-decoration:var(--sidebar-nav-link-text-decoration);-webkit-text-decoration-color:var(--sidebar-nav-link-text-decoration-color);text-decoration-color:var(--sidebar-nav-link-text-decoration-color);text-overflow:ellipsis;transition:var(--sidebar-nav-link-transition)}.sidebar-nav a img,.sidebar nav a img{margin-top:-0.25em;vertical-align:middle}.sidebar-nav a img:first-child,.sidebar nav a img:first-child{margin-right:0.5em}.sidebar-nav a:hover,.sidebar nav a:hover{border-width:var(--sidebar-nav-link-border-width--hover, var(--sidebar-nav-link-border-width, 0));border-style:var(--sidebar-nav-link-border-style--hover, var(--sidebar-nav-link-border-style));border-color:var(--sidebar-nav-link-border-color--hover, var(--sidebar-nav-link-border-color));background-color:var(--sidebar-nav-link-background-color--hover, var(--sidebar-nav-link-background-color));background-image:var(--sidebar-nav-link-background-image--hover, var(--sidebar-nav-link-background-image));background-position:var(--sidebar-nav-link-background-position--hover, var(--sidebar-nav-link-background-position));background-size:var(--sidebar-nav-link-background-size--hover, var(--sidebar-nav-link-background-size));color:var(--sidebar-nav-link-color--hover, var(--sidebar-nav-link-color));font-weight:var(--sidebar-nav-link-font-weight--hover, var(--sidebar-nav-link-font-weight));-webkit-text-decoration:var(--sidebar-nav-link-text-decoration--hover, var(--sidebar-nav-link-text-decoration));text-decoration:var(--sidebar-nav-link-text-decoration--hover, var(--sidebar-nav-link-text-decoration));-webkit-text-decoration-color:var(--sidebar-nav-link-text-decoration-color);text-decoration-color:var(--sidebar-nav-link-text-decoration-color)}.sidebar-nav ul>li>span,.sidebar-nav ul>li>strong,.sidebar nav ul>li>span,.sidebar nav ul>li>strong{display:block;margin:var(--sidebar-nav-strong-margin);padding:var(--sidebar-nav-strong-padding);border-width:var(--sidebar-nav-strong-border-width, 0);border-style:solid;border-color:var(--sidebar-nav-strong-border-color);color:var(--sidebar-nav-strong-color);font-size:var(--sidebar-nav-strong-font-size);font-weight:var(--sidebar-nav-strong-font-weight);text-transform:var(--sidebar-nav-strong-text-transform)}.sidebar-nav ul>li>span+ul,.sidebar-nav ul>li>strong+ul,.sidebar nav ul>li>span+ul,.sidebar nav ul>li>strong+ul{margin-left:0}.sidebar-nav ul>li:first-child>span,.sidebar-nav ul>li:first-child>strong,.sidebar nav ul>li:first-child>span,.sidebar nav ul>li:first-child>strong{margin-top:0}.sidebar-nav::-webkit-scrollbar,.sidebar nav::-webkit-scrollbar{width:0}@supports (width: env(safe-area-inset)){@media only screen and (orientation: landscape){.sidebar-nav,.sidebar nav{margin-left:calc(env(safe-area-inset-left) / 2)}}}.sidebar-nav li>a:before,.sidebar-nav li>strong:before{display:inline-block}.sidebar-nav li>a{background-repeat:var(--sidebar-nav-pagelink-background-repeat);background-size:var(--sidebar-nav-pagelink-background-size)}.sidebar-nav li>a[href^="#/"]:not([href*="?id="]){transition:var(--sidebar-nav-pagelink-transition)}.sidebar-nav li>a[href^="#/"]:not([href*="?id="]),.sidebar-nav li>a[href^="#/"]:not([href*="?id="]) ~ ul a{padding:var(--sidebar-nav-pagelink-padding, var(--sidebar-nav-link-padding))}.sidebar-nav li>a[href^="#/"]:not([href*="?id="]):only-child{background-image:var(--sidebar-nav-pagelink-background-image);background-position:var(--sidebar-nav-pagelink-background-position)}.sidebar-nav li>a[href^="#/"]:not([href*="?id="]):not(:only-child){background-image:var(--sidebar-nav-pagelink-background-image--loaded, var(--sidebar-nav-pagelink-background-image));background-position:var(--sidebar-nav-pagelink-background-position--loaded, var(--sidebar-nav-pagelink-background-image))}.sidebar-nav li.active>a,.sidebar-nav li.collapse>a{border-width:var(--sidebar-nav-link-border-width--active, var(--sidebar-nav-link-border-width));border-style:var(--sidebar-nav-link-border-style--active, var(--sidebar-nav-link-border-style));border-color:var(--sidebar-nav-link-border-color--active, var(--sidebar-nav-link-border-color));background-color:var(--sidebar-nav-link-background-color--active, var(--sidebar-nav-link-background-color));background-image:var(--sidebar-nav-link-background-image--active, var(--sidebar-nav-link-background-image));background-position:var(--sidebar-nav-link-background-position--active, var(--sidebar-nav-link-background-position));background-size:var(--sidebar-nav-link-background-size--active, var(--sidebar-nav-link-background-size));color:var(--sidebar-nav-link-color--active, var(--sidebar-nav-link-color));font-weight:var(--sidebar-nav-link-font-weight--active, var(--sidebar-nav-link-font-weight));-webkit-text-decoration:var(--sidebar-nav-link-text-decoration--active, var(--sidebar-nav-link-text-decoration));text-decoration:var(--sidebar-nav-link-text-decoration--active, var(--sidebar-nav-link-text-decoration));-webkit-text-decoration-color:var(--sidebar-nav-link-text-decoration-color);text-decoration-color:var(--sidebar-nav-link-text-decoration-color)}.sidebar-nav li.active>a[href^="#/"]:not([href*="?id="]):not(:only-child){background-image:var(--sidebar-nav-pagelink-background-image--active, var(--sidebar-nav-pagelink-background-image--loaded, var(--sidebar-nav-pagelink-background-image)));background-position:var(--sidebar-nav-pagelink-background-position--active, var(--sidebar-nav-pagelink-background-position--loaded, var(--sidebar-nav-pagelink-background-image)))}.sidebar-nav li.collapse>a[href^="#/"]:not([href*="?id="]):not(:only-child){background-image:var(--sidebar-nav-pagelink-background-image--collapse, var(--sidebar-nav-pagelink-background-image--loaded, var(--sidebar-nav-pagelink-background-image)));background-position:var(--sidebar-nav-pagelink-background-position--collapse, var(--sidebar-nav-pagelink-background-position--loaded, var(--sidebar-nav-pagelink-background-image)))}.sidebar-nav li.collapse .app-sub-sidebar{display:none}.sidebar-nav>ul>li>a:before{content:var(--sidebar-nav-link-before-content-l1, var(--sidebar-nav-link-before-content));margin:var(--sidebar-nav-link-before-margin-l1, var(--sidebar-nav-link-before-margin));color:var(--sidebar-nav-link-before-color-l1, var(--sidebar-nav-link-before-color))}.sidebar-nav>ul>li.active>a:before{content:var(--sidebar-nav-link-before-content-l1--active, var(--sidebar-nav-link-before-content--active, var(--sidebar-nav-link-before-content-l1, var(--sidebar-nav-link-before-content))));color:var(--sidebar-nav-link-before-color-l1--active, var(--sidebar-nav-link-before-color--active, var(--sidebar-nav-link-before-color-l1, var(--sidebar-nav-link-before-color))))}.sidebar-nav>ul>li>ul>li>a:before{content:var(--sidebar-nav-link-before-content-l2, var(--sidebar-nav-link-before-content));margin:var(--sidebar-nav-link-before-margin-l2, var(--sidebar-nav-link-before-margin));color:var(--sidebar-nav-link-before-color-l2, var(--sidebar-nav-link-before-color))}.sidebar-nav>ul>li>ul>li.active>a:before{content:var(--sidebar-nav-link-before-content-l2--active, var(--sidebar-nav-link-before-content--active, var(--sidebar-nav-link-before-content-l2, var(--sidebar-nav-link-before-content))));color:var(--sidebar-nav-link-before-color-l2--active, var(--sidebar-nav-link-before-color--active, var(--sidebar-nav-link-before-color-l2, var(--sidebar-nav-link-before-color))))}.sidebar-nav>ul>li>ul>li>ul>li>a:before{content:var(--sidebar-nav-link-before-content-l3, var(--sidebar-nav-link-before-content));margin:var(--sidebar-nav-link-before-margin-l3, var(--sidebar-nav-link-before-margin));color:var(--sidebar-nav-link-before-color-l3, var(--sidebar-nav-link-before-color))}.sidebar-nav>ul>li>ul>li>ul>li.active>a:before{content:var(--sidebar-nav-link-before-content-l3--active, var(--sidebar-nav-link-before-content--active, var(--sidebar-nav-link-before-content-l3, var(--sidebar-nav-link-before-content))));color:var(--sidebar-nav-link-before-color-l3--active, var(--sidebar-nav-link-before-color--active, var(--sidebar-nav-link-before-color-l3, var(--sidebar-nav-link-before-color))))}.sidebar-nav>ul>li>ul>li>ul>li>ul>li>a:before{content:var(--sidebar-nav-link-before-content-l4, var(--sidebar-nav-link-before-content));margin:var(--sidebar-nav-link-before-margin-l4, var(--sidebar-nav-link-before-margin));color:var(--sidebar-nav-link-before-color-l4, var(--sidebar-nav-link-before-color))}.sidebar-nav>ul>li>ul>li>ul>li>ul>li.active>a:before{content:var(--sidebar-nav-link-before-content-l4--active, var(--sidebar-nav-link-before-content--active, var(--sidebar-nav-link-before-content-l4, var(--sidebar-nav-link-before-content))));color:var(--sidebar-nav-link-before-color-l4--active, var(--sidebar-nav-link-before-color--active, var(--sidebar-nav-link-before-color-l4, var(--sidebar-nav-link-before-color))))}.sidebar-nav>:last-child{margin-bottom:2rem}.sidebar-toggle,.sidebar-toggle-button{width:var(--sidebar-toggle-width);outline:none}.sidebar-toggle{position:fixed;z-index:11;top:0;bottom:0;left:0;max-width:40px;margin:0;padding:0;border:0;background:transparent;-webkit-appearance:none;-moz-appearance:none;appearance:none;cursor:pointer}.sidebar-toggle .sidebar-toggle-button{position:absolute;top:var(--sidebar-toggle-offset-top);left:var(--sidebar-toggle-offset-left);height:var(--sidebar-toggle-height);border-radius:var(--sidebar-toggle-border-radius);border-width:var(--sidebar-toggle-border-width);border-style:var(--sidebar-toggle-border-style);border-color:var(--sidebar-toggle-border-color);background:var(--sidebar-toggle-background, transparent);color:var(--sidebar-toggle-icon-color)}.sidebar-toggle span{position:absolute;top:calc(50% - (var(--sidebar-toggle-icon-stroke-width) / 2));left:calc(50% - (var(--sidebar-toggle-icon-width) / 2));height:var(--sidebar-toggle-icon-stroke-width);width:var(--sidebar-toggle-icon-width);background-color:currentColor}.sidebar-toggle span:nth-child(1){margin-top:calc(0px - (var(--sidebar-toggle-icon-height) / 2))}.sidebar-toggle span:nth-child(3){margin-top:calc((var(--sidebar-toggle-icon-height) / 2))}@media (min-width: 48em){.sidebar-toggle{position:absolute;overflow:visible;top:var(--sidebar-toggle-offset-top);bottom:auto;left:0;height:var(--sidebar-toggle-height);-webkit-transform:translateX(var(--sidebar-width));transform:translateX(var(--sidebar-width))}.sidebar-toggle .sidebar-toggle-button{top:0}}@media print{.sidebar-toggle{display:none}}@media (max-width: 47.99em){body.close .sidebar,body.close .sidebar-toggle,body.close main>.content{-webkit-transform:translateX(var(--sidebar-width));transform:translateX(var(--sidebar-width))}}@media (min-width: 48em){body.close main>.content{-webkit-transform:translateX(0);transform:translateX(0)}}@media (max-width: 47.99em){body.close nav.app-nav,body.close .github-corner{display:none}}@media (min-width: 48em){body.close .sidebar,body.close .sidebar-toggle{-webkit-transform:translateX(0);transform:translateX(0)}}@media (min-width: 48em){body.close nav.app-nav{margin-left:0}}@media (max-width: 47.99em){body.close .sidebar-toggle{width:100%;max-width:none}body.close .sidebar-toggle span{margin-top:0}body.close .sidebar-toggle span:nth-child(1){-webkit-transform:rotate(45deg);transform:rotate(45deg)}body.close .sidebar-toggle span:nth-child(2){display:none}body.close .sidebar-toggle span:nth-child(3){-webkit-transform:rotate(-45deg);transform:rotate(-45deg)}}@media (min-width: 48em){body.close main>.content{margin-left:0}}@media (min-width: 48em){body.sticky .sidebar,body.sticky .sidebar-toggle{position:fixed}}body .docsify-copy-code-button,body .docsify-copy-code-button:after{border-radius:var(--border-radius-m, 0);border-top-left-radius:0;border-bottom-right-radius:0;background:var(--copycode-background);color:var(--copycode-color)}body .docsify-copy-code-button span{border-radius:var(--border-radius-s, 0)}body .docsify-pagination-container{border-top:var(--pagination-border-top);color:var(--pagination-color)}body .pagination-item-label{font-size:var(--pagination-label-font-size)}body .pagination-item-label svg{color:var(--pagination-label-color);height:var(--pagination-chevron-height);stroke:var(--pagination-chevron-stroke);stroke-linecap:var(--pagination-chevron-stroke-linecap);stroke-linejoin:var(--pagination-chevron-stroke-linecap);stroke-width:var(--pagination-chevron-stroke-width)}body .pagination-item-title{color:var(--pagination-title-color);font-size:var(--pagination-title-font-size)}body .app-name.hide{display:block}body .sidebar{padding:var(--sidebar-padding)}.sidebar .search{margin:0;padding:0;border:0}.sidebar .search input{padding:0;line-height:1;font-size:inherit}.sidebar .search .clear-button{width:auto}.sidebar .search .clear-button svg{-webkit-transform:scale(1);transform:scale(1)}.sidebar .search .matching-post{border:none}.sidebar .search p{font-size:inherit}.sidebar .search{order:var(--search-flex-order);margin:var(--search-margin);padding:var(--search-padding);background:var(--search-background)}.sidebar .search a{color:inherit}.sidebar .search h2{margin:var(--search-result-heading-margin);font-size:var(--search-result-heading-font-size);font-weight:var(--search-result-heading-font-weight);color:var(--search-result-heading-color)}.sidebar .search .input-wrap{margin:var(--search-input-margin);background-color:var(--search-input-background-color);border-width:var(--search-input-border-width, 0);border-style:solid;border-color:var(--search-input-border-color);border-radius:var(--search-input-border-radius)}.sidebar .search input[type="search"]{min-width:0;padding:var(--search-input-padding);border:none;background-color:transparent;background-image:var(--search-input-background-image);background-position:var(--search-input-background-position);background-repeat:var(--search-input-background-repeat);background-size:var(--search-input-background-size);font-size:var(--search-input-font-size);color:var(--search-input-color);transition:var(--search-input-transition)}.sidebar .search input[type="search"]::-ms-clear{display:none}.sidebar .search input[type="search"]::-webkit-input-placeholder{color:var(--search-input-placeholder-color, gray)}.sidebar .search input[type="search"]:-ms-input-placeholder{color:var(--search-input-placeholder-color, gray)}.sidebar .search input[type="search"]::-ms-input-placeholder{color:var(--search-input-placeholder-color, gray)}.sidebar .search input[type="search"]::placeholder{color:var(--search-input-placeholder-color, gray)}.sidebar .search input[type="search"]::-webkit-input-placeholder{line-height:normal}.sidebar .search input[type="search"]:focus{background-color:var(--search-input-background-color--focus, var(--search-input-background-color));background-image:var(--search-input-background-image--focus, var(--search-input-background-image));background-position:var(--search-input-background-position--focus, var(--search-input-background-position));background-size:var(--search-input-background-size--focus, var(--search-input-background-size))}@supports (width: env(safe-area-inset)){@media only screen and (orientation: landscape){.sidebar .search input[type="search"]{margin-left:calc(env(safe-area-inset-left) / 2)}}}.sidebar .search p{overflow:hidden;text-overflow:ellipsis;-webkit-line-clamp:2}.sidebar .search p:empty{text-align:center}.sidebar .search .clear-button{margin:0 15px 0 0;padding:0;border:none;line-height:1;background:transparent;cursor:pointer}.sidebar .search .clear-button svg circle{fill:var(--search-clear-icon-color1, gray)}.sidebar .search .clear-button svg path{stroke:var(--search-clear-icon-color2, #fff)}.sidebar .search.show ~ *:not(h1){display:none}.sidebar .search .results-panel{display:none;color:var(--search-result-item-color);font-size:var(--search-result-item-font-size);font-weight:var(--search-result-item-font-weight)}.sidebar .search .results-panel.show{display:block}.sidebar .search .matching-post{margin:var(--search-result-item-margin);padding:var(--search-result-item-padding)}.sidebar .search .matching-post,.sidebar .search .matching-post:last-child{border-width:var(--search-result-item-border-width, 0) !important;border-style:var(--search-result-item-border-style);border-color:var(--search-result-item-border-color)}.sidebar .search .matching-post p{margin:0}.sidebar .search .search-keyword{margin:var(--search-result-keyword-margin);padding:var(--search-result-keyword-padding);border-radius:var(--search-result-keyword-border-radius);background-color:var(--search-result-keyword-background);color:var(--search-result-keyword-color, currentColor);font-style:normal;font-weight:var(--search-result-keyword-font-weight)}.medium-zoom-overlay,.medium-zoom-image--open{z-index:50 !important}.medium-zoom-overlay{background:var(--zoomimage-overlay-background) !important}:root{--mono-hue: 113;--mono-saturation: 0%;--mono-shade3: hsl(var(--mono-hue), var(--mono-saturation), 20%);--mono-shade2: hsl(var(--mono-hue), var(--mono-saturation), 30%);--mono-shade1: hsl(var(--mono-hue), var(--mono-saturation), 40%);--mono-base: hsl(var(--mono-hue), var(--mono-saturation), 50%);--mono-tint1: hsl(var(--mono-hue), var(--mono-saturation), 70%);--mono-tint2: hsl(var(--mono-hue), var(--mono-saturation), 89%);--mono-tint3: hsl(var(--mono-hue), var(--mono-saturation), 97%);--theme-hue: 204;--theme-saturation: 90%;--theme-lightness: 45%;--theme-color: hsl(var(--theme-hue), var(--theme-saturation), var(--theme-lightness));--modular-scale: 1.333;--modular-scale--2: calc(var(--modular-scale--1) / var(--modular-scale));--modular-scale--1: calc(var(--modular-scale-1) / var(--modular-scale));--modular-scale-1: 1rem;--modular-scale-2: calc(var(--modular-scale-1) * var(--modular-scale));--modular-scale-3: calc(var(--modular-scale-2) * var(--modular-scale));--modular-scale-4: calc(var(--modular-scale-3) * var(--modular-scale));--modular-scale-5: calc(var(--modular-scale-4) * var(--modular-scale));--font-size-xxxl: var(--modular-scale-5);--font-size-xxl: var(--modular-scale-4);--font-size-xl: var(--modular-scale-3);--font-size-l: var(--modular-scale-2);--font-size-m: var(--modular-scale-1);--font-size-s: var(--modular-scale--1);--font-size-xs: var(--modular-scale--2);--duration-slow: 1s;--duration-medium: 0.5s;--duration-fast: 0.25s;--spinner-size: 60px;--spinner-track-width: 4px;--spinner-track-color: rgba(0, 0, 0, 0.15);--spinner-transition-duration: var(--duration-medium)}:root{--base-background-color: #fff;--base-color: var(--mono-shade2);--base-font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol";--base-font-size: 16px;--base-font-weight: normal;--base-line-height: 1.7;--emoji-size: calc(var(--base-line-height) * 1em);--hr-border: 1px solid var(--mono-tint2);--mark-background: #ffecb3;--pre-font-family: var(--code-font-family);--pre-font-size: var(--code-font-size);--pre-font-weight: normal;--selection-color: #b4d5fe;--small-font-size: var(--font-size-s);--strong-color: var(--heading-color);--strong-font-weight: 600;--subsup-font-size: var(--font-size-s)}:root{--content-max-width: 55em;--blockquote-background: var(--mono-tint3);--blockquote-border-style: solid;--blockquote-border-radius: var(--border-radius-m);--blockquote-em-font-weight: normal;--blockquote-font-weight: normal;--code-font-family: Inconsolata, Consolas, Menlo, Monaco, "Andale Mono WT", "Andale Mono", "Lucida Console", "DejaVu Sans Mono", "Bitstream Vera Sans Mono", "Courier New", Courier, monospace;--code-font-size: calc(var(--font-size-m) * 0.95);--code-font-weight: normal;--code-tab-size: 4;--code-block-border-radius: var(--border-radius-m);--code-block-line-height: var(--base-line-height);--code-block-margin: 1em 0;--code-block-padding: 1.75em 1.5em 1.5em 1.5em;--code-inline-background: var(--code-theme-background);--code-inline-border-radius: var(--border-radius-s);--code-inline-color: var(--code-theme-text);--code-inline-margin: 0 0.15em;--code-inline-padding: 0.125em 0.4em;--code-theme-background: var(--mono-tint3);--heading-color: var(--mono-shade3);--heading-margin: 2.5rem 0 0;--heading-h1-border-style: solid;--heading-h1-font-size: var(--font-size-xxl);--heading-h2-border-style: solid;--heading-h2-font-size: var(--font-size-xl);--heading-h3-border-style: solid;--heading-h3-font-size: var(--font-size-l);--heading-h4-border-style: solid;--heading-h4-font-size: var(--font-size-m);--heading-h5-border-style: solid;--heading-h5-font-size: var(--font-size-s);--heading-h6-border-style: solid;--heading-h6-font-size: var(--font-size-xs);--kbd-background: var(--mono-tint3);--kbd-border-radius: var(--border-radius-m);--kbd-margin: 0 0.3em;--kbd-min-width: 2.5em;--kbd-padding: 0.65em 0.5em;--link-text-decoration: underline;--notice-background: var(--mono-tint3);--notice-border-radius: var(--border-radius-m);--notice-border-style: solid;--notice-font-weight: normal;--notice-padding: 1em 1.5em;--notice-before-font-weight: normal;--table-cell-padding: 0.75em 0.5em;--table-head-border-color: var(--table-cell-border-color);--table-head-font-weight: var(--strong-font-weight);--table-row-odd-background: var(--mono-tint3)}:root{--cover-margin: 0 auto;--cover-max-width: 40em;--cover-text-align: center;--cover-background-color: var(--base-background-color);--cover-background-mask-color: var(--base-background-color);--cover-background-mask-opacity: 0.8;--cover-background-position: center center;--cover-background-repeat: no-repeat;--cover-background-size: cover;--cover-blockquote-font-size: var(--font-size-l);--cover-border-color: var(--theme-color);--cover-button-border: 1px solid var(--theme-color);--cover-button-border-radius: var(--border-radius-m);--cover-button-color: var(--theme-color);--cover-button-padding: 0.5em 2rem;--cover-button-text-decoration: none;--cover-button-transition: all var(--duration-fast) ease-in-out;--cover-button-primary-background: var(--theme-color);--cover-button-primary-border: 1px solid var(--theme-color);--cover-button-primary-color: #fff;--cover-heading-color: var(--theme-color);--cover-heading-font-size: var(--font-size-xxl);--cover-heading-font-weight: normal;--cover-link-text-decoration: underline }:root{--navbar-root-border-style: solid;--navbar-root-margin: 0 0 0 1.5em;--navbar-root-transition: all var(--duration-fast);--navbar-menu-background: var(--base-background-color);--navbar-menu-border-radius: var(--border-radius-m);--navbar-menu-box-shadow: rgba(45,45,45,0.05) 0px 0px 1px, rgba(49,49,49,0.05) 0px 1px 2px, rgba(42,42,42,0.05) 0px 2px 4px, rgba(32,32,32,0.05) 0px 4px 8px, rgba(49,49,49,0.05) 0px 8px 16px, rgba(35,35,35,0.05) 0px 16px 32px;--navbar-menu-padding: 0.5em;--navbar-menu-transition: all var(--duration-fast);--navbar-menu-link-border-style: solid;--navbar-menu-link-margin: 0.75em 0.5em;--navbar-menu-link-padding: 0.2em 0 }:root{--copycode-background: #808080;--copycode-color: #fff}:root{--docsifytabs-border-color: var(--mono-tint2);--docsifytabs-border-radius-px: var(--border-radius-s);--docsifytabs-tab-background: var(--mono-tint3);--docsifytabs-tab-color: var(--mono-tint1)}:root{--pagination-border-top: 1px solid var(--mono-tint2);--pagination-chevron-height: 0.8em;--pagination-chevron-stroke: currentColor;--pagination-chevron-stroke-linecap: round;--pagination-chevron-stroke-width: 1px;--pagination-label-font-size: var(--font-size-s);--pagination-title-font-size: var(--font-size-l)}:root{--search-margin: 1.5rem 0 0;--search-input-background-repeat: no-repeat;--search-input-border-color: var(--mono-tint1);--search-input-border-width: 1px;--search-input-padding: 0.5em;--search-flex-order: 1;--search-result-heading-color: var(--heading-color);--search-result-heading-font-size: var(--base-font-size);--search-result-heading-font-weight: normal;--search-result-heading-margin: 0 0 0.25em;--search-result-item-border-color: var(--mono-tint2);--search-result-item-border-style: solid;--search-result-item-border-width: 0 0 1px 0;--search-result-item-font-weight: normal;--search-result-item-padding: 1em 0;--search-result-keyword-background: var(--mark-background);--search-result-keyword-border-radius: var(--border-radius-s);--search-result-keyword-color: var(--mark-color);--search-result-keyword-font-weight: normal;--search-result-keyword-margin: 0 0.1em;--search-result-keyword-padding: 0.2em 0}:root{--zoomimage-overlay-background: rgba(0, 0, 0, 0.875)}:root{--sidebar-background: var(--base-background-color);--sidebar-border-width: 0;--sidebar-padding: 0 25px;--sidebar-transition-duration: var(--duration-fast);--sidebar-width: 17rem;--sidebar-name-font-size: var(--font-size-l);--sidebar-name-font-weight: normal;--sidebar-name-margin: 1.5rem 0 0;--sidebar-name-text-align: center;--sidebar-nav-strong-border-color: var(--sidebar-border-color);--sidebar-nav-strong-color: var(--heading-color);--sidebar-nav-strong-font-weight: var(--strong-font-weight);--sidebar-nav-strong-margin: 1.5em 0 0.5em;--sidebar-nav-strong-padding: 0.25em 0;--sidebar-nav-indent: 1em;--sidebar-nav-margin: 1.5rem 0 0;--sidebar-nav-link-border-style: solid;--sidebar-nav-link-border-width: 0;--sidebar-nav-link-color: var(--base-color);--sidebar-nav-link-font-weight: normal;--sidebar-nav-link-padding: 0.25em 0;--sidebar-nav-link-text-decoration--active: underline;--sidebar-nav-link-text-decoration--hover: underline;--sidebar-nav-link-before-margin: 0 0.35em 0 0;--sidebar-nav-pagelink-background-repeat: no-repeat;--sidebar-nav-pagelink-transition: var(--sidebar-nav-link-transition);--sidebar-toggle-border-radius: var(--border-radius-s);--sidebar-toggle-border-style: solid;--sidebar-toggle-border-width: 0;--sidebar-toggle-height: 36px;--sidebar-toggle-icon-color: var(--base-color);--sidebar-toggle-icon-height: 10px;--sidebar-toggle-icon-stroke-width: 1px;--sidebar-toggle-icon-width: 16px;--sidebar-toggle-offset-left: 0;--sidebar-toggle-offset-top: calc(35px - (var(--sidebar-toggle-height) / 2));--sidebar-toggle-width: 44px}:root{--code-theme-background: #f3f3f3;--code-theme-comment: #6e8090;--code-theme-function: #dd4a68;--code-theme-keyword: #07a;--code-theme-operator: #a67f59;--code-theme-punctuation: #999;--code-theme-selection: #b3d4fc;--code-theme-selector: #690;--code-theme-tag: #905;--code-theme-text: #333;--code-theme-variable: #e90}:root{--border-radius-s: 2px;--border-radius-m: 4px;--border-radius-l: 8px;--strong-font-weight: 600;--blockquote-border-color: var(--theme-color);--blockquote-border-radius: 0 var(--border-radius-m) var(--border-radius-m) 0;--blockquote-border-width: 0 0 0 4px;--code-inline-background: var(--mono-tint2);--code-theme-background: var(--mono-tint3);--heading-font-weight: var(--strong-font-weight);--heading-h1-font-weight: 400;--heading-h2-font-weight: 400;--heading-h2-border-color: var(--mono-tint2);--heading-h2-border-width: 0 0 1px 0;--heading-h2-margin: 2.5rem 0 1.5rem;--heading-h2-padding: 0 0 1rem 0;--kbd-border: 1px solid var(--mono-tint2);--notice-border-radius: 0 var(--border-radius-m) var(--border-radius-m) 0;--notice-border-width: 0 0 0 4px;--notice-padding: 1em 1.5em 1em 3em;--notice-before-border-radius: 100%;--notice-before-font-weight: bold;--notice-before-height: 1.5em;--notice-before-left: 0.75em;--notice-before-line-height: 1.5;--notice-before-margin: 0 0.25em 0 0;--notice-before-position: absolute;--notice-before-width: var(--notice-before-height);--notice-important-background: hsl(340, 60%, 96%);--notice-important-border-color: hsl(340, 90%, 45%);--notice-important-before-background: var(--notice-important-border-color) url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 16 16' fill='%23fff'%3E%3Cpath d='M10 14C10 15.1 9.1 16 8 16 6.9 16 6 15.1 6 14 6 12.9 6.9 12 8 12 9.1 12 10 12.9 10 14Z'/%3E%3Cpath d='M10 1.6C10 1.2 9.8 0.9 9.6 0.7 9.2 0.3 8.6 0 8 0 7.4 0 6.8 0.2 6.5 0.6 6.2 0.9 6 1.2 6 1.6 6 1.7 6 1.8 6 1.9L6.8 9.6C6.9 9.9 7 10.1 7.2 10.2 7.4 10.4 7.7 10.5 8 10.5 8.3 10.5 8.6 10.4 8.8 10.3 9 10.1 9.1 9.9 9.2 9.6L10 1.9C10 1.8 10 1.7 10 1.6Z'/%3E%3C/svg%3E") center / 0.875em no-repeat;--notice-important-before-color: #fff;--notice-important-before-content: "";--notice-tip-background: hsl(204, 60%, 96%);--notice-tip-border-color: hsl(204, 90%, 45%);--notice-tip-before-background: var(--notice-tip-border-color) url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 16 16' fill='%23fff'%3E%3Cpath d='M9.1 0C10.2 0 10.7 0.7 10.7 1.6 10.7 2.6 9.8 3.6 8.6 3.6 7.6 3.6 7 3 7 2 7 1.1 7.7 0 9.1 0Z'/%3E%3Cpath d='M5.8 16C5 16 4.4 15.5 5 13.2L5.9 9.1C6.1 8.5 6.1 8.2 5.9 8.2 5.7 8.2 4.6 8.6 3.9 9.1L3.5 8.4C5.6 6.6 7.9 5.6 8.9 5.6 9.8 5.6 9.9 6.6 9.5 8.2L8.4 12.5C8.2 13.2 8.3 13.5 8.5 13.5 8.7 13.5 9.6 13.2 10.4 12.5L10.9 13.2C8.9 15.2 6.7 16 5.8 16Z'/%3E%3C/svg%3E") center / 0.875em no-repeat;--notice-tip-before-color: #fff;--notice-tip-before-content: "";--table-cell-border-color: var(--mono-tint2);--table-cell-border-width: 1px 0;--cover-background-color: hsl(var(--theme-hue), 25%, 60%);--cover-background-image: radial-gradient(ellipse at center 115%, rgba(255, 255, 255, 0.9), transparent);--cover-blockquote-color: var(--strong-color);--cover-heading-color: #fff;--cover-heading-font-size-max: 56;--cover-heading-font-size-min: 34;--cover-heading-font-weight: 200;--navbar-root-color--active: var(--theme-color);--navbar-menu-border-radius: var(--border-radius-m);--navbar-menu-root-background: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='9.6' height='6' viewBox='0 0 9.6 6'%3E%3Cpath d='M1.5 1.5l3.3 3 3.3-3' stroke-width='1.5' stroke='rgb%28179, 179, 179%29' fill='none' stroke-linecap='square' stroke-linejoin='miter' vector-effect='non-scaling-stroke'/%3E%3C/svg%3E") right no-repeat;--navbar-menu-root-padding: 0 18px 0 0;--search-input-background-color: #fff;--search-input-background-image: url("data:image/svg+xml,%3Csvg height='20px' width='20px' viewBox='0 0 24 24' fill='none' stroke='rgba(0, 0, 0, 0.3)' stroke-width='1.5' stroke-linecap='round' stroke-linejoin='round' preserveAspectRatio='xMidYMid meet' xmlns='http://www.w3.org/2000/svg'%3E%3Ccircle cx='10.5' cy='10.5' r='7.5' vector-effect='non-scaling-stroke'%3E%3C/circle%3E%3Cline x1='21' y1='21' x2='15.8' y2='15.8' vector-effect='non-scaling-stroke'%3E%3C/line%3E%3C/svg%3E");--search-input-background-position: 21px center;--search-input-border-color: var(--sidebar-border-color);--search-input-border-width: 1px 0;--search-input-margin: 0 -25px;--search-input-padding: 0.65em 1em 0.65em 50px;--search-input-placeholder-color: rgba(0, 0, 0, 0.4);--search-clear-icon-color1: rgba(0, 0, 0, 0.3);--search-result-heading-font-weight: var(--strong-font-weight);--search-result-item-border-color: var(--sidebar-border-color);--search-result-keyword-border-radius: var(--border-radius-s);--sidebar-background: var(--mono-tint3);--sidebar-border-color: var(--mono-tint2);--sidebar-border-width: 0 1px 0 0;--sidebar-name-color: var(--theme-color);--sidebar-name-font-weight: 300;--sidebar-nav-strong-border-width: 0 0 1px 0;--sidebar-nav-strong-font-size: smaller;--sidebar-nav-strong-margin: 2em -25px 0.75em 0;--sidebar-nav-strong-padding: 0.25em 0 0.75em 0;--sidebar-nav-strong-text-transform: uppercase;--sidebar-nav-link-border-color: transparent;--sidebar-nav-link-border-color--active: var(--theme-color);--sidebar-nav-link-border-width: 0 4px 0 0;--sidebar-nav-link-color--active: var(--theme-color);--sidebar-nav-link-margin: 0 -25px 0 0;--sidebar-nav-link-text-decoration: none;--sidebar-nav-link-text-decoration--active: none;--sidebar-nav-link-text-decoration--hover: underline;--sidebar-nav-link-before-content-l3: '-';--sidebar-nav-pagelink-background-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='7' height='11.2' viewBox='0 0 7 11.2'%3E%3Cpath d='M1.5 1.5l4 4.1 -4 4.1' stroke-width='1.5' stroke='rgb%28179, 179, 179%29' fill='none' stroke-linecap='square' stroke-linejoin='miter' vector-effect='non-scaling-stroke'/%3E%3C/svg%3E");--sidebar-nav-pagelink-background-image--active: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='11.2' height='7' viewBox='0 0 11.2 7'%3E%3Cpath d='M1.5 1.5l4.1 4 4.1-4' stroke-width='1.5' stroke='rgb%2811, 135, 218%29' fill='none' stroke-linecap='square' stroke-linejoin='miter' vector-effect='non-scaling-stroke'/%3E%3C/svg%3E");--sidebar-nav-pagelink-background-image--collapse: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='7' height='11.2' viewBox='0 0 7 11.2'%3E%3Cpath d='M1.5 1.5l4 4.1 -4 4.1' stroke-width='1.5' stroke='rgb%2811, 135, 218%29' fill='none' stroke-linecap='square' stroke-linejoin='miter' vector-effect='non-scaling-stroke'/%3E%3C/svg%3E");--sidebar-nav-pagelink-background-image--loaded: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='11.2' height='7' viewBox='0 0 11.2 7'%3E%3Cpath d='M1.5 1.5l4.1 4 4.1-4' stroke-width='1.5' stroke='rgb%2811, 135, 218%29' fill='none' stroke-linecap='square' stroke-linejoin='miter' vector-effect='non-scaling-stroke'/%3E%3C/svg%3E");--sidebar-nav-pagelink-background-position: 3px center;--sidebar-nav-pagelink-background-position--active: left center;--sidebar-nav-pagelink-background-position--collapse: var(--sidebar-nav-pagelink-background-position);--sidebar-nav-pagelink-background-position--loaded: var(--sidebar-nav-pagelink-background-position--active);--sidebar-nav-pagelink-padding: 0.25em 0 0.25em 20px;--sidebar-nav-pagelink-transition: none;--sidebar-toggle-background: var(--sidebar-border-color);--sidebar-toggle-border-radius: 0 var(--border-radius-s) var(--border-radius-s) 0;--sidebar-toggle-width: 32px} -/*# sourceMappingURL=theme-simple.css.map */ \ No newline at end of file diff --git a/docs/assets/css/theme-simple.css b/docs/assets/css/theme-simple.css index 2c4269923a..6bc4adc996 100644 --- a/docs/assets/css/theme-simple.css +++ b/docs/assets/css/theme-simple.css @@ -1,2 +1,2136 @@ -.github-corner{position:absolute;z-index:40;top:0;right:0;border-bottom:0;text-decoration:none}.github-corner svg{height:70px;width:70px;fill:var(--theme-color);color:var(--base-background-color)}.github-corner:hover .octo-arm{-webkit-animation:octocat-wave 560ms ease-in-out;animation:octocat-wave 560ms ease-in-out}@-webkit-keyframes octocat-wave{0%,100%{-webkit-transform:rotate(0);transform:rotate(0)}20%,60%{-webkit-transform:rotate(-25deg);transform:rotate(-25deg)}40%,80%{-webkit-transform:rotate(10deg);transform:rotate(10deg)}}@keyframes octocat-wave{0%,100%{-webkit-transform:rotate(0);transform:rotate(0)}20%,60%{-webkit-transform:rotate(-25deg);transform:rotate(-25deg)}40%,80%{-webkit-transform:rotate(10deg);transform:rotate(10deg)}}.progress{position:fixed;z-index:60;top:0;left:0;right:0;height:3px;width:0;background-color:var(--theme-color);transition:width var(--duration-fast),opacity calc(var(--duration-fast) * 2)}body.ready-transition:after,body.ready-transition>*:not(.progress){opacity:0;transition:opacity var(--spinner-transition-duration)}body.ready-transition:after{content:'';position:absolute;z-index:1000;top:calc(50% - (var(--spinner-size) / 2));left:calc(50% - (var(--spinner-size) / 2));height:var(--spinner-size);width:var(--spinner-size);border:var(--spinner-track-width, 0) solid var(--spinner-track-color);border-left-color:var(--theme-color);border-left-color:var(--theme-color);border-radius:50%;-webkit-animation:spinner var(--duration-slow) infinite linear;animation:spinner var(--duration-slow) infinite linear}body.ready-transition.ready-spinner:after{opacity:1}body.ready-transition.ready-fix:after{opacity:0}body.ready-transition.ready-fix>*:not(.progress){opacity:1;transition-delay:var(--spinner-transition-duration)}@-webkit-keyframes spinner{0%{-webkit-transform:rotate(0deg);transform:rotate(0deg)}100%{-webkit-transform:rotate(360deg);transform:rotate(360deg)}}@keyframes spinner{0%{-webkit-transform:rotate(0deg);transform:rotate(0deg)}100%{-webkit-transform:rotate(360deg);transform:rotate(360deg)}}*,*:before,*:after{box-sizing:inherit;font-size:inherit;-webkit-overflow-scrolling:touch;-webkit-tap-highlight-color:rgba(0,0,0,0);-webkit-text-size-adjust:none;-webkit-touch-callout:none}:root{box-sizing:border-box;background-color:var(--base-background-color);font-size:var(--base-font-size);font-weight:var(--base-font-weight);line-height:var(--base-line-height);letter-spacing:var(--base-letter-spacing);color:var(--base-color);-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale;font-smoothing:antialiased}html,button,input,optgroup,select,textarea{font-family:var(--base-font-family)}button,input,optgroup,select,textarea{font-size:100%;margin:0}a{text-decoration:none;-webkit-text-decoration-skip:ink;text-decoration-skip-ink:auto}body{margin:0}hr{height:0;margin:2em 0;border:none;border-bottom:var(--hr-border, 0)}img{border:0}main{display:block}main.hidden{display:none}mark{background:var(--mark-background);color:var(--mark-color)}pre{font-family:var(--pre-font-family);font-size:var(--pre-font-size);font-weight:var(--pre-font-weight);line-height:var(--pre-line-height)}small{display:inline-block;font-size:var(--small-font-size)}strong{font-weight:var(--strong-font-weight);color:var(--strong-color, currentColor)}sub,sup{font-size:var(--subsup-font-size);line-height:0;position:relative;vertical-align:baseline}sub{bottom:-0.25em}sup{top:-0.5em}body:not([data-platform^="Mac"]) *{scrollbar-color:hsla(var(--mono-hue), var(--mono-saturation), 50%, 0.3) hsla(var(--mono-hue), var(--mono-saturation), 50%, 0.1);scrollbar-width:thin}body:not([data-platform^="Mac"]) * ::-webkit-scrollbar{width:5px;height:5px}body:not([data-platform^="Mac"]) * ::-webkit-scrollbar-thumb{background:hsla(var(--mono-hue), var(--mono-saturation), 50%, 0.3)}body:not([data-platform^="Mac"]) * ::-webkit-scrollbar-track{background:hsla(var(--mono-hue), var(--mono-saturation), 50%, 0.1)}::selection{background:var(--selection-color)}.emoji{height:var(--emoji-size);vertical-align:middle}.task-list-item{list-style:none}.task-list-item input{margin-right:0.5em;margin-left:0;vertical-align:0.075em}.markdown-section code[class*="lang-"],.markdown-section pre[data-lang]{font-family:var(--code-font-family);font-size:var(--code-font-size);font-weight:var(--code-font-weight);letter-spacing:normal;line-height:var(--code-block-line-height);-moz-tab-size:var(--code-tab-size);-o-tab-size:var(--code-tab-size);tab-size:var(--code-tab-size);text-align:left;white-space:pre;word-spacing:normal;word-wrap:normal;word-break:normal;-webkit-hyphens:none;-ms-hyphens:none;hyphens:none}.markdown-section pre[data-lang]{position:relative;overflow:hidden;margin:var(--code-block-margin);padding:0;border-radius:var(--code-block-border-radius)}.markdown-section pre[data-lang]::after{content:attr(data-lang);position:absolute;top:0.75em;right:0.75em;opacity:0.6;color:inherit;font-size:var(--font-size-s);line-height:1}.markdown-section pre[data-lang] code{display:block;overflow:auto;padding:var(--code-block-padding)}code[class*="lang-"],pre[data-lang]{color:var(--code-theme-text)}pre[data-lang]::selection,pre[data-lang] ::selection,code[class*="lang-"]::selection,code[class*="lang-"] ::selection{background:var(--code-theme-selection, var(--selection-color))}:not(pre)>code[class*="lang-"],pre[data-lang]{background:var(--code-theme-background)}.namespace{opacity:0.7}.token.comment,.token.prolog,.token.doctype,.token.cdata{color:var(--code-theme-comment)}.token.punctuation{color:var(--code-theme-punctuation)}.token.property,.token.tag,.token.boolean,.token.number,.token.constant,.token.symbol,.token.deleted{color:var(--code-theme-tag)}.token.selector,.token.attr-name,.token.string,.token.char,.token.builtin,.token.inserted{color:var(--code-theme-selector)}.token.operator,.token.entity,.token.url,.language-css .token.string,.style .token.string{color:var(--code-theme-operator)}.token.atrule,.token.attr-value,.token.keyword{color:var(--code-theme-keyword)}.token.function{color:var(--code-theme-function)}.token.regex,.token.important,.token.variable{color:var(--code-theme-variable)}.token.important,.token.bold{font-weight:bold}.token.italic{font-style:italic}.token.entity{cursor:help}.markdown-section{position:relative;max-width:var(--content-max-width);margin:0 auto;padding:2rem 45px}.app-nav:not(:empty) ~ main .markdown-section{padding-top:3.5rem}.markdown-section figure,.markdown-section p,.markdown-section ol,.markdown-section ul{margin:1em 0}.markdown-section ol,.markdown-section ul{padding-left:1.5rem}.markdown-section ol ol,.markdown-section ol ul,.markdown-section ul ol,.markdown-section ul ul{margin-top:0.15rem;margin-bottom:0.15rem}.markdown-section a{border-bottom:var(--link-border-bottom);color:var(--link-color);-webkit-text-decoration:var(--link-text-decoration);text-decoration:var(--link-text-decoration);-webkit-text-decoration-color:var(--link-text-decoration-color);text-decoration-color:var(--link-text-decoration-color)}.markdown-section a:hover{border-bottom:var(--link-border-bottom--hover, var(--link-border-bottom, 0));color:var(--link-color--hover, var(--link-color));-webkit-text-decoration:var(--link-text-decoration--hover, var(--link-text-decoration));text-decoration:var(--link-text-decoration--hover, var(--link-text-decoration));-webkit-text-decoration-color:var(--link-text-decoration-color--hover, var(--link-text-decoration-color));text-decoration-color:var(--link-text-decoration-color--hover, var(--link-text-decoration-color))}.markdown-section a.anchor{border-bottom:0;color:inherit;text-decoration:none}.markdown-section a.anchor:hover{text-decoration:underline}.markdown-section blockquote{overflow:visible;margin:2em 0;padding:1.5em;border-width:var(--blockquote-border-width, 0);border-style:var(--blockquote-border-style);border-color:var(--blockquote-border-color);border-radius:var(--blockquote-border-radius);background:var(--blockquote-background);color:var(--blockquote-color);font-family:var(--blockquote-font-family);font-size:var(--blockquote-font-size);font-style:var(--blockquote-font-style);font-weight:var(--blockquote-font-weight);quotes:"“" "”" "‘" "’"}.markdown-section blockquote em{font-family:var(--blockquote-em-font-family);font-size:var(--blockquote-em-font-size);font-style:var(--blockquote-em-font-style);font-weight:var(--blockquote-em-font-weight)}.markdown-section blockquote p:first-child{margin-top:0}.markdown-section blockquote p:first-child:before,.markdown-section blockquote p:first-child:after{color:var(--blockquote-quotes-color);font-family:var(--blockquote-quotes-font-family);font-size:var(--blockquote-quotes-font-size);line-height:0}.markdown-section blockquote p:first-child:before{content:var(--blockquote-quotes-open);margin-right:0.15em;vertical-align:-0.45em}.markdown-section blockquote p:first-child:after{content:var(--blockquote-quotes-close);margin-left:0.15em;vertical-align:-0.55em}.markdown-section blockquote p:last-child{margin-bottom:0}.markdown-section code{font-family:var(--code-font-family);font-size:var(--code-font-size);font-weight:var(--code-font-weight);line-height:inherit}.markdown-section code:not([class*="lang-"]):not([class*="language-"]){margin:var(--code-inline-margin);padding:var(--code-inline-padding);border-radius:var(--code-inline-border-radius);background:var(--code-inline-background);color:var(--code-inline-color, currentColor);white-space:nowrap}.markdown-section h1:first-child,.markdown-section h2:first-child,.markdown-section h3:first-child,.markdown-section h4:first-child,.markdown-section h5:first-child,.markdown-section h6:first-child{margin-top:0}.markdown-section h1+h2,.markdown-section h1+h3,.markdown-section h1+h4,.markdown-section h1+h5,.markdown-section h1+h6,.markdown-section h2+h3,.markdown-section h2+h4,.markdown-section h2+h5,.markdown-section h2+h6,.markdown-section h3+h4,.markdown-section h3+h5,.markdown-section h3+h6,.markdown-section h4+h5,.markdown-section h4+h6,.markdown-section h5+h6{margin-top:1rem;font-size: 1rem;}.markdown-section h1{margin:var(--heading-h1-margin, var(--heading-margin));padding:var(--heading-h1-padding, var(--heading-padding));border-width:var(--heading-h1-border-width, 0);border-style:var(--heading-h1-border-style);border-color:var(--heading-h1-border-color);font-family:var(--heading-h1-font-family, var(--heading-font-family));font-size:var(--heading-h1-font-size);font-weight:var(--heading-h1-font-weight, var(--heading-font-weight));line-height:var(--base-line-height);color:var(--heading-h1-color, var(--heading-color))}.markdown-section h2{margin:var(--heading-h2-margin, var(--heading-margin));padding:var(--heading-h2-padding, var(--heading-padding));border-width:var(--heading-h2-border-width, 0);border-style:var(--heading-h2-border-style);border-color:var(--heading-h2-border-color);font-family:var(--heading-h2-font-family, var(--heading-font-family));font-size:var(--heading-h2-font-size);font-weight:var(--heading-h2-font-weight, var(--heading-font-weight));line-height:var(--base-line-height);color:var(--heading-h2-color, var(--heading-color))}.markdown-section h3{margin:var(--heading-h3-margin, var(--heading-margin));padding:var(--heading-h3-padding, var(--heading-padding));border-width:var(--heading-h3-border-width, 0);border-style:var(--heading-h3-border-style);border-color:var(--heading-h3-border-color);font-family:var(--heading-h3-font-family, var(--heading-font-family));font-size:var(--heading-h3-font-size);font-weight:var(--heading-h3-font-weight, var(--heading-font-weight));color:var(--heading-h3-color, var(--heading-color))}.markdown-section h4{margin:var(--heading-h4-margin, var(--heading-margin));padding:var(--heading-h4-padding, var(--heading-padding));border-width:var(--heading-h4-border-width, 0);border-style:var(--heading-h4-border-style);border-color:var(--heading-h4-border-color);font-family:var(--heading-h4-font-family, var(--heading-font-family));font-size:var(--heading-h4-font-size);font-weight:var(--heading-h4-font-weight, var(--heading-font-weight));color:var(--heading-h4-color, var(--heading-color))}.markdown-section h6{margin:var(--heading-h5-margin, var(--heading-margin));padding:var(--heading-h5-padding, var(--heading-padding));border-width:var(--heading-h5-border-width, 0);border-style:var(--heading-h5-border-style);border-color:var(--heading-h5-border-color);font-family:var(--heading-h5-font-family, var(--heading-font-family));font-size:var(--heading-h5-font-size);font-weight:var(--heading-h5-font-weight, var(--heading-font-weight));color:var(--heading-h5-color, var(--heading-color))}.markdown-section h6{margin:var(--heading-h6-margin, var(--heading-margin));padding:var(--heading-h6-padding, var(--heading-padding));border-width:var(--heading-h6-border-width, 0);border-style:var(--heading-h6-border-style);border-color:var(--heading-h6-border-color);font-family:var(--heading-h6-font-family, var(--heading-font-family));font-size:var(--heading-h6-font-size);font-weight:var(--heading-h6-font-weight, var(--heading-font-weight));color:var(--heading-h6-color, var(--heading-color))}.markdown-section iframe{margin:1em 0}.markdown-section img{max-width:100%}.markdown-section kbd{display:inline-block;min-width:var(--kbd-min-width);margin:var(--kbd-margin);padding:var(--kbd-padding);border:var(--kbd-border);border-radius:var(--kbd-border-radius);background:var(--kbd-background);font-family:inherit;font-size:var(--kbd-font-size);text-align:center;letter-spacing:0;line-height:1;color:var(--kbd-color)}.markdown-section kbd+kbd{margin-left:-0.15em}.markdown-section table{display:block;overflow:auto;margin:1rem 0;border-spacing:0;border-collapse:collapse}.markdown-section th,.markdown-section td{padding:var(--table-cell-padding)}.markdown-section th:not([align]){text-align:left}.markdown-section thead{border-color:var(--table-head-border-color);border-style:solid;border-width:var(--table-head-border-width, 0);background:var(--table-head-background)}.markdown-section th{font-weight:var(--table-head-font-weight);color:var(--strong-color)}.markdown-section td{border-color:var(--table-cell-border-color);border-style:solid;border-width:var(--table-cell-border-width, 0)}.markdown-section tbody{border-color:var(--table-body-border-color);border-style:solid;border-width:var(--table-body-border-width, 0)}.markdown-section tbody tr:nth-child(odd){background:var(--table-row-odd-background)}.markdown-section tbody tr:nth-child(even){background:var(--table-row-even-background)}.markdown-section>ul .task-list-item{margin-left:-1.25em}.markdown-section>ul .task-list-item .task-list-item{margin-left:0}.markdown-section .table-wrapper table{display:table;width:100%}.markdown-section .table-wrapper td::before{display:none}@media (max-width: 30em){.markdown-section .table-wrapper tbody,.markdown-section .table-wrapper tr,.markdown-section .table-wrapper td{display:block}.markdown-section .table-wrapper th,.markdown-section .table-wrapper td{border:none}.markdown-section .table-wrapper thead{display:none}.markdown-section .table-wrapper tr{border-color:var(--table-cell-border-color);border-style:solid;border-width:var(--table-cell-border-width, 0);padding:var(--table-cell-padding)}.markdown-section .table-wrapper tr:not(:last-child){border-bottom:0}.markdown-section .table-wrapper td{display:flex;padding:0.15em 0}.markdown-section .table-wrapper td::before{display:block;min-width:8em;max-width:8em;font-weight:bold;text-align:left}}.markdown-section .tip,.markdown-section .warn{position:relative;margin:2em 0;padding:var(--notice-padding);border-width:var(--notice-border-width, 0);border-style:var(--notice-border-style);border-color:var(--notice-border-color);border-radius:var(--notice-border-radius);background:var(--notice-background);font-family:var(--notice-font-family);font-weight:var(--notice-font-weight);color:var(--notice-color)}.markdown-section .tip:before,.markdown-section .warn:before{display:inline-block;position:var(--notice-before-position, relative);top:var(--notice-before-top);left:var(--notice-before-left);height:var(--notice-before-height);width:var(--notice-before-width);margin:var(--notice-before-margin);padding:var(--notice-before-padding);border-radius:var(--notice-before-border-radius);line-height:var(--notice-before-line-height);font-family:var(--notice-before-font-family);font-size:var(--notice-before-font-size);font-weight:var(--notice-before-font-weight);text-align:center}.markdown-section .tip{border-width:var(--notice-important-border-width, var(--notice-border-width, 0));border-style:var(--notice-important-border-style, var(--notice-border-style));border-color:var(--notice-important-border-color, var(--notice-border-color));background:var(--notice-important-background, var(--notice-background));color:var(--notice-important-color, var(--notice-color))}.markdown-section .tip:before{content:var(--notice-important-before-content, var(--notice-before-content));background:var(--notice-important-before-background, var(--notice-before-background));color:var(--notice-important-before-color, var(--notice-before-color))}.markdown-section .warn{border-width:var(--notice-tip-border-width, var(--notice-border-width, 0));border-style:var(--notice-tip-border-style, var(--notice-border-style));border-color:var(--notice-tip-border-color, var(--notice-border-color));background:var(--notice-tip-background, var(--notice-background));color:var(--notice-tip-color, var(--notice-color))}.markdown-section .warn:before{content:var(--notice-tip-before-content, var(--notice-before-content));background:var(--notice-tip-before-background, var(--notice-before-background));color:var(--notice-tip-before-color, var(--notice-before-color))}.cover{display:none;position:relative;z-index:20;min-height:100vh;flex-direction:column;align-items:center;justify-content:center;padding:calc(var(--cover-border-inset, 0px) + var(--cover-border-width, 0px));color:var(--cover-color);text-align:var(--cover-text-align)}@media screen and (-ms-high-contrast: active), screen and (-ms-high-contrast: none){.cover{height:100vh}}.cover:before,.cover:after{content:'';position:absolute}.cover:before{bottom:0;left:0;right:0;background-blend-mode:var(--cover-background-blend-mode);background-color:var(--cover-background-color);background-image:var(--cover-background-image);background-position:var(--cover-background-position);background-repeat:var(--cover-background-repeat);background-size:var(--cover-background-size)}.cover:after{top:var(--cover-border-inset, 0);bottom:var(--cover-border-inset, 0);left:var(--cover-border-inset, 0);right:var(--cover-border-inset, 0);border-width:var(--cover-border-width, 0);border-style:solid;border-color:var(--cover-border-color)}.cover a{border-bottom:var(--cover-link-border-bottom);color:var(--cover-link-color);-webkit-text-decoration:var(--cover-link-text-decoration);text-decoration:var(--cover-link-text-decoration);-webkit-text-decoration-color:var(--cover-link-text-decoration-color);text-decoration-color:var(--cover-link-text-decoration-color)}.cover a:hover{border-bottom:var(--cover-link-border-bottom--hover, var(--cover-link-border-bottom));color:var(--cover-link-color--hover, var(--cover-link-color));-webkit-text-decoration:var(--cover-link-text-decoration--hover, var(--cover-link-text-decoration));text-decoration:var(--cover-link-text-decoration--hover, var(--cover-link-text-decoration));-webkit-text-decoration-color:var(--cover-link-text-decoration-color--hover, var(--cover-link-text-decoration-color));text-decoration-color:var(--cover-link-text-decoration-color--hover, var(--cover-link-text-decoration-color))}.cover h1{color:var(--cover-heading-color);position:relative;margin:0;font-size:var(--cover-heading-font-size);font-weight:var(--cover-heading-font-weight);line-height:1.2}.cover h1 a,.cover h1 a:hover{display:block;border-bottom:none;color:inherit;text-decoration:none}.cover h1 small{position:absolute;bottom:0;margin-left:0.5em}.cover h1 span{font-size:calc(var(--cover-heading-font-size-min) * 1px)}@media (min-width: 26em){.cover h1 span{font-size:calc((var(--cover-heading-font-size-min) * 1px) + (var(--cover-heading-font-size-max) - var(--cover-heading-font-size-min)) * ((100vw - 420px) / (1024 - 420)))}}@media (min-width: 64em){.cover h1 span{font-size:calc(var(--cover-heading-font-size-max) * 1px)}}.cover blockquote{margin:0;color:var(--cover-blockquote-color);font-size:var(--cover-blockquote-font-size)}.cover blockquote a{color:inherit}.cover ul{padding:0;list-style-type:none}.cover .cover-main{position:relative;z-index:1;max-width:var(--cover-max-width);margin:var(--cover-margin);padding:0 45px}.cover .cover-main>p:last-child{margin:1.25em -.25em}.cover .cover-main>p:last-child a{display:block;margin:.375em .25em;padding:var(--cover-button-padding);border:var(--cover-button-border);border-radius:var(--cover-button-border-radius);box-shadow:var(--cover-button-box-shadow);background:var(--cover-button-background);text-align:center;-webkit-text-decoration:var(--cover-button-text-decoration);text-decoration:var(--cover-button-text-decoration);-webkit-text-decoration-color:var(--cover-button-text-decoration-color);text-decoration-color:var(--cover-button-text-decoration-color);color:var(--cover-button-color);white-space:nowrap;transition:var(--cover-button-transition)}.cover .cover-main>p:last-child a:hover{border:var(--cover-button-border--hover, var(--cover-button-border));box-shadow:var(--cover-button-box-shadow--hover, var(--cover-button-box-shadow));background:var(--cover-button-background--hover, var(--cover-button-background));-webkit-text-decoration:var(--cover-button-text-decoration--hover, var(--cover-button-text-decoration));text-decoration:var(--cover-button-text-decoration--hover, var(--cover-button-text-decoration));-webkit-text-decoration-color:var(--cover-button-text-decoration-color--hover, var(--cover-button-text-decoration-color));text-decoration-color:var(--cover-button-text-decoration-color--hover, var(--cover-button-text-decoration-color));color:var(--cover-button-color--hover, var(--cover-button-color))}.cover .cover-main>p:last-child a:first-child{border:var(--cover-button-primary-border, var(--cover-button-border));box-shadow:var(--cover-button-primary-box-shadow, var(--cover-button-box-shadow));background:var(--cover-button-primary-background, var(--cover-button-background));-webkit-text-decoration:var(--cover-button-primary-text-decoration, var(--cover-button-text-decoration));text-decoration:var(--cover-button-primary-text-decoration, var(--cover-button-text-decoration));-webkit-text-decoration-color:var(--cover-button-primary-text-decoration-color, var(--cover-button-text-decoration-color));text-decoration-color:var(--cover-button-primary-text-decoration-color, var(--cover-button-text-decoration-color));color:var(--cover-button-primary-color, var(--cover-button-color))}.cover .cover-main>p:last-child a:first-child:hover{border:var(--cover-button-primary-border--hover, var(--cover-button-border--hover, var(--cover-button-primary-border, var(--cover-button-border))));box-shadow:var(--cover-button-primary-box-shadow--hover, var(--cover-button-box-shadow--hover, var(--cover-button-primary-box-shadow, var(--cover-button-box-shadow))));background:var(--cover-button-primary-background--hover, var(--cover-button-background--hover, var(--cover-button-primary-background, var(--cover-button-background))));-webkit-text-decoration:var(--cover-button-primary-text-decoration--hover, var(--cover-button-text-decoration--hover, var(--cover-button-primary-text-decoration, var(--cover-button-text-decoration))));text-decoration:var(--cover-button-primary-text-decoration--hover, var(--cover-button-text-decoration--hover, var(--cover-button-primary-text-decoration, var(--cover-button-text-decoration))));-webkit-text-decoration-color:var(--cover-button-primary-text-decoration-color--hover, var(--cover-button-text-decoration-color--hover, var(--cover-button-primary-text-decoration-color, var(--cover-button-text-decoration-color))));text-decoration-color:var(--cover-button-primary-text-decoration-color--hover, var(--cover-button-text-decoration-color--hover, var(--cover-button-primary-text-decoration-color, var(--cover-button-text-decoration-color))));color:var(--cover-button-primary-color--hover, var(--cover-button-color--hover, var(--cover-button-primary-color, var(--cover-button-color))))}@media (min-width: 30.01em){.cover .cover-main>p:last-child a{display:inline-block}}.cover .mask{visibility:var(--cover-background-mask-visibility, hidden);position:absolute;top:0;bottom:0;left:0;right:0;background-color:var(--cover-background-mask-color);opacity:var(--cover-background-mask-opacity)}.cover.has-mask .mask{visibility:visible}.cover.show{display:flex}.app-nav{position:absolute;z-index:30;top:calc(35px - (0.5em * var(--base-line-height)));left:45px;right:80px;text-align:right}.app-nav.no-badge{right:45px}.app-nav li>img,.app-nav li>a>img{margin-top:-0.25em;vertical-align:middle}.app-nav li>img:first-child,.app-nav li>a>img:first-child{margin-right:0.5em}.app-nav ul,.app-nav li{margin:0;padding:0;list-style:none}.app-nav li{position:relative}.app-nav li a{display:block;line-height:1;transition:var(--navbar-root-transition)}.app-nav>ul>li{display:inline-block;margin:var(--navbar-root-margin)}.app-nav>ul>li:first-child{margin-left:0}.app-nav>ul>li:last-child{margin-right:0}.app-nav>ul>li>a,.app-nav>ul>li>span{padding:var(--navbar-root-padding);border-width:var(--navbar-root-border-width, 0);border-style:var(--navbar-root-border-style);border-color:var(--navbar-root-border-color);border-radius:var(--navbar-root-border-radius);background:var(--navbar-root-background);color:var(--navbar-root-color);-webkit-text-decoration:var(--navbar-root-text-decoration);text-decoration:var(--navbar-root-text-decoration);-webkit-text-decoration-color:var(--navbar-root-text-decoration-color);text-decoration-color:var(--navbar-root-text-decoration-color)}.app-nav>ul>li>a:hover,.app-nav>ul>li>span:hover{background:var(--navbar-root-background--hover, var(--navbar-root-background));border-style:var(--navbar-root-border-style--hover, var(--navbar-root-border-style));border-color:var(--navbar-root-border-color--hover, var(--navbar-root-border-color));color:var(--navbar-root-color--hover, var(--navbar-root-color));-webkit-text-decoration:var(--navbar-root-text-decoration--hover, var(--navbar-root-text-decoration));text-decoration:var(--navbar-root-text-decoration--hover, var(--navbar-root-text-decoration));-webkit-text-decoration-color:var(--navbar-root-text-decoration-color--hover, var(--navbar-root-text-decoration-color));text-decoration-color:var(--navbar-root-text-decoration-color--hover, var(--navbar-root-text-decoration-color))}.app-nav>ul>li>a:not(:last-child),.app-nav>ul>li>span:not(:last-child){padding:var(--navbar-menu-root-padding, var(--navbar-root-padding));background:var(--navbar-menu-root-background, var(--navbar-root-background))}.app-nav>ul>li>a:not(:last-child):hover,.app-nav>ul>li>span:not(:last-child):hover{background:var(--navbar-menu-root-background--hover, var(--navbar-menu-root-background, var(--navbar-root-background--hover, var(--navbar-root-background))))}.app-nav>ul>li>a.active{background:var(--navbar-root-background--active, var(--navbar-root-background));border-style:var(--navbar-root-border-style--active, var(--navbar-root-border-style));border-color:var(--navbar-root-border-color--active, var(--navbar-root-border-color));color:var(--navbar-root-color--active, var(--navbar-root-color));-webkit-text-decoration:var(--navbar-root-text-decoration--active, var(--navbar-root-text-decoration));text-decoration:var(--navbar-root-text-decoration--active, var(--navbar-root-text-decoration));-webkit-text-decoration-color:var(--navbar-root-text-decoration-color--active, var(--navbar-root-text-decoration-color));text-decoration-color:var(--navbar-root-text-decoration-color--active, var(--navbar-root-text-decoration-color))}.app-nav>ul>li>a.active:not(:last-child):hover{background:var(--navbar-menu-root-background--active, var(--navbar-menu-root-background, var(--navbar-root-background--active, var(--navbar-root-background))))}.app-nav>ul>li ul{visibility:hidden;position:absolute;top:100%;right:50%;overflow-y:auto;box-sizing:border-box;max-height:calc(50vh);padding:var(--navbar-menu-padding);border-width:var(--navbar-menu-border-width, 0);border-style:solid;border-color:var(--navbar-menu-border-color);border-radius:var(--navbar-menu-border-radius);background:var(--navbar-menu-background);box-shadow:var(--navbar-menu-box-shadow);text-align:left;white-space:nowrap;opacity:0;-webkit-transform:translate(50%, -0.35em);transform:translate(50%, -0.35em);transition:var(--navbar-menu-transition)}.app-nav>ul>li ul li{white-space:nowrap}.app-nav>ul>li ul a{margin:var(--navbar-menu-link-margin);padding:var(--navbar-menu-link-padding);border-width:var(--navbar-menu-link-border-width, 0);border-style:var(--navbar-menu-link-border-style);border-color:var(--navbar-menu-link-border-color);border-radius:var(--navbar-menu-link-border-radius);background:var(--navbar-menu-link-background);color:var(--navbar-menu-link-color);-webkit-text-decoration:var(--navbar-menu-link-text-decoration);text-decoration:var(--navbar-menu-link-text-decoration);-webkit-text-decoration-color:var(--navbar-menu-link-text-decoration-color);text-decoration-color:var(--navbar-menu-link-text-decoration-color)}.app-nav>ul>li ul a:hover{background:var(--navbar-menu-link-background--hover, var(--navbar-menu-link-background));border-style:var(--navbar-menu-link-border-style--hover, var(--navbar-menu-link-border-style));border-color:var(--navbar-menu-link-border-color--hover, var(--navbar-menu-link-border-color));color:var(--navbar-menu-link-color--hover, var(--navbar-menu-link-color));-webkit-text-decoration:var(--navbar-menu-link-text-decoration--hover, var(--navbar-menu-link-text-decoration));text-decoration:var(--navbar-menu-link-text-decoration--hover, var(--navbar-menu-link-text-decoration));-webkit-text-decoration-color:var(--navbar-menu-link-text-decoration-color--hover, var(--navbar-menu-link-text-decoration-color));text-decoration-color:var(--navbar-menu-link-text-decoration-color--hover, var(--navbar-menu-link-text-decoration-color))}.app-nav>ul>li ul a.active{background:var(--navbar-menu-link-background--active, var(--navbar-menu-link-background));border-style:var(--navbar-menu-link-border-style--active, var(--navbar-menu-link-border-style));border-color:var(--navbar-menu-link-border-color--active, var(--navbar-menu-link-border-color));color:var(--navbar-menu-link-color--active, var(--navbar-menu-link-color));-webkit-text-decoration:var(--navbar-menu-link-text-decoration--active, var(--navbar-menu-link-text-decoration));text-decoration:var(--navbar-menu-link-text-decoration--active, var(--navbar-menu-link-text-decoration));-webkit-text-decoration-color:var(--navbar-menu-link-text-decoration-color--active, var(--navbar-menu-link-text-decoration-color));text-decoration-color:var(--navbar-menu-link-text-decoration-color--active, var(--navbar-menu-link-text-decoration-color))}.app-nav>ul>li:hover ul,.app-nav>ul>li:focus ul,.app-nav>ul>li.focus-within ul{visibility:visible;opacity:1;-webkit-transform:translate(50%, 0);transform:translate(50%, 0)}.sidebar,.sidebar-toggle,main>.content{transition:all var(--sidebar-transition-duration) ease-out}@media (min-width: 48em){nav.app-nav{margin-left:var(--sidebar-width)}}main{position:relative;overflow-x:hidden;min-height:100vh}@media (min-width: 48em){main>.content{margin-left:var(--sidebar-width)}}.sidebar{display:flex;flex-direction:column;position:fixed;z-index:10;top:0;right:100%;overflow-x:hidden;overflow-y:auto;height:100vh;width:var(--sidebar-width);padding:var(--sidebar-padding);border-width:var(--sidebar-border-width);border-style:solid;border-color:var(--sidebar-border-color);background:var(--sidebar-background)}.sidebar>h1{margin:0;margin:var(--sidebar-name-margin);padding:var(--sidebar-name-padding);background:var(--sidebar-name-background);color:var(--sidebar-name-color);font-family:var(--sidebar-name-font-family);font-size:var(--sidebar-name-font-size);font-weight:var(--sidebar-name-font-weight);text-align:var(--sidebar-name-text-align)}.sidebar>h1 img{max-width:100%}.sidebar>h1 .app-name-link{color:var(--sidebar-name-color)}body:not([data-platform^="Mac"]) .sidebar::-webkit-scrollbar{width:5px}body:not([data-platform^="Mac"]) .sidebar::-webkit-scrollbar-thumb{border-radius:50vw}@media (min-width: 48em){.sidebar{position:absolute;-webkit-transform:translateX(var(--sidebar-width));transform:translateX(var(--sidebar-width))}}@media print{.sidebar{display:none}}.sidebar-nav,.sidebar nav{order:1;margin:var(--sidebar-nav-margin);padding:var(--sidebar-nav-padding);background:var(--sidebar-nav-background)}.sidebar-nav ul,.sidebar nav ul{margin:0;padding:0;list-style:none}.sidebar-nav ul ul,.sidebar nav ul ul{margin-left:var(--sidebar-nav-indent)}.sidebar-nav a,.sidebar nav a{display:block;overflow:hidden;margin:var(--sidebar-nav-link-margin);padding:var(--sidebar-nav-link-padding);border-width:var(--sidebar-nav-link-border-width, 0);border-style:var(--sidebar-nav-link-border-style);border-color:var(--sidebar-nav-link-border-color);border-radius:var(--sidebar-nav-link-border-radius);background-color:var(--sidebar-nav-link-background-color);background-image:var(--sidebar-nav-link-background-image);background-position:var(--sidebar-nav-link-background-position);background-repeat:var(--sidebar-nav-link-background-repeat);background-size:var(--sidebar-nav-link-background-size);color:var(--sidebar-nav-link-color);font-weight:var(--sidebar-nav-link-font-weight);white-space:nowrap;-webkit-text-decoration:var(--sidebar-nav-link-text-decoration);text-decoration:var(--sidebar-nav-link-text-decoration);-webkit-text-decoration-color:var(--sidebar-nav-link-text-decoration-color);text-decoration-color:var(--sidebar-nav-link-text-decoration-color);text-overflow:ellipsis;transition:var(--sidebar-nav-link-transition)}.sidebar-nav a img,.sidebar nav a img{margin-top:-0.25em;vertical-align:middle}.sidebar-nav a img:first-child,.sidebar nav a img:first-child{margin-right:0.5em}.sidebar-nav a:hover,.sidebar nav a:hover{border-width:var(--sidebar-nav-link-border-width--hover, var(--sidebar-nav-link-border-width, 0));border-style:var(--sidebar-nav-link-border-style--hover, var(--sidebar-nav-link-border-style));border-color:var(--sidebar-nav-link-border-color--hover, var(--sidebar-nav-link-border-color));background-color:var(--sidebar-nav-link-background-color--hover, var(--sidebar-nav-link-background-color));background-image:var(--sidebar-nav-link-background-image--hover, var(--sidebar-nav-link-background-image));background-position:var(--sidebar-nav-link-background-position--hover, var(--sidebar-nav-link-background-position));background-size:var(--sidebar-nav-link-background-size--hover, var(--sidebar-nav-link-background-size));color:var(--sidebar-nav-link-color--hover, var(--sidebar-nav-link-color));font-weight:var(--sidebar-nav-link-font-weight--hover, var(--sidebar-nav-link-font-weight));-webkit-text-decoration:var(--sidebar-nav-link-text-decoration--hover, var(--sidebar-nav-link-text-decoration));text-decoration:var(--sidebar-nav-link-text-decoration--hover, var(--sidebar-nav-link-text-decoration));-webkit-text-decoration-color:var(--sidebar-nav-link-text-decoration-color);text-decoration-color:var(--sidebar-nav-link-text-decoration-color)}.sidebar-nav ul>li>span,.sidebar-nav ul>li>strong,.sidebar nav ul>li>span,.sidebar nav ul>li>strong{display:block;margin:var(--sidebar-nav-strong-margin);padding:var(--sidebar-nav-strong-padding);border-width:var(--sidebar-nav-strong-border-width, 0);border-style:solid;border-color:var(--sidebar-nav-strong-border-color);color:var(--sidebar-nav-strong-color);font-size:1rem;font-weight:var(--sidebar-nav-strong-font-weight);text-transform:var(--sidebar-nav-strong-text-transform)}.sidebar-nav ul>li>span+ul,.sidebar-nav ul>li>strong+ul,.sidebar nav ul>li>span+ul,.sidebar nav ul>li>strong+ul{margin-left:0}.sidebar-nav ul>li:first-child>span,.sidebar-nav ul>li:first-child>strong,.sidebar nav ul>li:first-child>span,.sidebar nav ul>li:first-child>strong{margin-top:0}.sidebar-nav::-webkit-scrollbar,.sidebar nav::-webkit-scrollbar{width:0}@supports (width: env(safe-area-inset)){@media only screen and (orientation: landscape){.sidebar-nav,.sidebar nav{margin-left:calc(env(safe-area-inset-left) / 2)}}}.sidebar-nav li>a:before,.sidebar-nav li>strong:before{display:inline-block}.sidebar-nav li>a{background-repeat:var(--sidebar-nav-pagelink-background-repeat);background-size:var(--sidebar-nav-pagelink-background-size)}.sidebar-nav li>a[href^="#/"]:not([href*="?id="]){transition:var(--sidebar-nav-pagelink-transition)}.sidebar-nav li>a[href^="#/"]:not([href*="?id="]),.sidebar-nav li>a[href^="#/"]:not([href*="?id="]) ~ ul a{padding:var(--sidebar-nav-pagelink-padding, var(--sidebar-nav-link-padding))}.sidebar-nav li>a[href^="#/"]:not([href*="?id="]):only-child{background-image:var(--sidebar-nav-pagelink-background-image);background-position:var(--sidebar-nav-pagelink-background-position)}.sidebar-nav li>a[href^="#/"]:not([href*="?id="]):not(:only-child){background-image:var(--sidebar-nav-pagelink-background-image--loaded, var(--sidebar-nav-pagelink-background-image));background-position:var(--sidebar-nav-pagelink-background-position--loaded, var(--sidebar-nav-pagelink-background-image))}.sidebar-nav li.active>a,.sidebar-nav li.collapse>a{border-width:var(--sidebar-nav-link-border-width--active, var(--sidebar-nav-link-border-width));border-style:var(--sidebar-nav-link-border-style--active, var(--sidebar-nav-link-border-style));border-color:var(--sidebar-nav-link-border-color--active, var(--sidebar-nav-link-border-color));background-color:var(--sidebar-nav-link-background-color--active, var(--sidebar-nav-link-background-color));background-image:var(--sidebar-nav-link-background-image--active, var(--sidebar-nav-link-background-image));background-position:var(--sidebar-nav-link-background-position--active, var(--sidebar-nav-link-background-position));background-size:var(--sidebar-nav-link-background-size--active, var(--sidebar-nav-link-background-size));color:var(--sidebar-nav-link-color--active, var(--sidebar-nav-link-color));font-weight:var(--sidebar-nav-link-font-weight--active, var(--sidebar-nav-link-font-weight));-webkit-text-decoration:var(--sidebar-nav-link-text-decoration--active, var(--sidebar-nav-link-text-decoration));text-decoration:var(--sidebar-nav-link-text-decoration--active, var(--sidebar-nav-link-text-decoration));-webkit-text-decoration-color:var(--sidebar-nav-link-text-decoration-color);text-decoration-color:var(--sidebar-nav-link-text-decoration-color)}.sidebar-nav li.active>a[href^="#/"]:not([href*="?id="]):not(:only-child){background-image:var(--sidebar-nav-pagelink-background-image--active, var(--sidebar-nav-pagelink-background-image--loaded, var(--sidebar-nav-pagelink-background-image)));background-position:var(--sidebar-nav-pagelink-background-position--active, var(--sidebar-nav-pagelink-background-position--loaded, var(--sidebar-nav-pagelink-background-image)))}.sidebar-nav li.collapse>a[href^="#/"]:not([href*="?id="]):not(:only-child){background-image:var(--sidebar-nav-pagelink-background-image--collapse, var(--sidebar-nav-pagelink-background-image--loaded, var(--sidebar-nav-pagelink-background-image)));background-position:var(--sidebar-nav-pagelink-background-position--collapse, var(--sidebar-nav-pagelink-background-position--loaded, var(--sidebar-nav-pagelink-background-image)))}.sidebar-nav li.collapse .app-sub-sidebar{display:none}.sidebar-nav>ul>li>a:before{content:var(--sidebar-nav-link-before-content-l1, var(--sidebar-nav-link-before-content));margin:var(--sidebar-nav-link-before-margin-l1, var(--sidebar-nav-link-before-margin));color:var(--sidebar-nav-link-before-color-l1, var(--sidebar-nav-link-before-color))}.sidebar-nav>ul>li.active>a:before{content:var(--sidebar-nav-link-before-content-l1--active, var(--sidebar-nav-link-before-content--active, var(--sidebar-nav-link-before-content-l1, var(--sidebar-nav-link-before-content))));color:var(--sidebar-nav-link-before-color-l1--active, var(--sidebar-nav-link-before-color--active, var(--sidebar-nav-link-before-color-l1, var(--sidebar-nav-link-before-color))))}.sidebar-nav>ul>li>ul>li>a:before{content:var(--sidebar-nav-link-before-content-l2, var(--sidebar-nav-link-before-content));margin:var(--sidebar-nav-link-before-margin-l2, var(--sidebar-nav-link-before-margin));color:var(--sidebar-nav-link-before-color-l2, var(--sidebar-nav-link-before-color))}.sidebar-nav>ul>li>ul>li.active>a:before{content:var(--sidebar-nav-link-before-content-l2--active, var(--sidebar-nav-link-before-content--active, var(--sidebar-nav-link-before-content-l2, var(--sidebar-nav-link-before-content))));color:var(--sidebar-nav-link-before-color-l2--active, var(--sidebar-nav-link-before-color--active, var(--sidebar-nav-link-before-color-l2, var(--sidebar-nav-link-before-color))))}.sidebar-nav>ul>li>ul>li>ul>li>a:before{content:var(--sidebar-nav-link-before-content-l3, var(--sidebar-nav-link-before-content));margin:var(--sidebar-nav-link-before-margin-l3, var(--sidebar-nav-link-before-margin));color:var(--sidebar-nav-link-before-color-l3, var(--sidebar-nav-link-before-color))}.sidebar-nav>ul>li>ul>li>ul>li.active>a:before{content:var(--sidebar-nav-link-before-content-l3--active, var(--sidebar-nav-link-before-content--active, var(--sidebar-nav-link-before-content-l3, var(--sidebar-nav-link-before-content))));color:var(--sidebar-nav-link-before-color-l3--active, var(--sidebar-nav-link-before-color--active, var(--sidebar-nav-link-before-color-l3, var(--sidebar-nav-link-before-color))))}.sidebar-nav>ul>li>ul>li>ul>li>ul>li>a:before{content:var(--sidebar-nav-link-before-content-l4, var(--sidebar-nav-link-before-content));margin:var(--sidebar-nav-link-before-margin-l4, var(--sidebar-nav-link-before-margin));color:var(--sidebar-nav-link-before-color-l4, var(--sidebar-nav-link-before-color))}.sidebar-nav>ul>li>ul>li>ul>li>ul>li.active>a:before{content:var(--sidebar-nav-link-before-content-l4--active, var(--sidebar-nav-link-before-content--active, var(--sidebar-nav-link-before-content-l4, var(--sidebar-nav-link-before-content))));color:var(--sidebar-nav-link-before-color-l4--active, var(--sidebar-nav-link-before-color--active, var(--sidebar-nav-link-before-color-l4, var(--sidebar-nav-link-before-color))))}.sidebar-nav>:last-child{margin-bottom:2rem}.sidebar-toggle,.sidebar-toggle-button{width:var(--sidebar-toggle-width);outline:none}.sidebar-toggle{position:fixed;z-index:11;top:0;bottom:0;left:0;max-width:40px;margin:0;padding:0;border:0;background:transparent;-webkit-appearance:none;-moz-appearance:none;appearance:none;cursor:pointer}.sidebar-toggle .sidebar-toggle-button{position:absolute;top:var(--sidebar-toggle-offset-top);left:var(--sidebar-toggle-offset-left);height:var(--sidebar-toggle-height);border-radius:var(--sidebar-toggle-border-radius);border-width:var(--sidebar-toggle-border-width);border-style:var(--sidebar-toggle-border-style);border-color:var(--sidebar-toggle-border-color);background:var(--sidebar-toggle-background, transparent);color:var(--sidebar-toggle-icon-color)}.sidebar-toggle span{position:absolute;top:calc(50% - (var(--sidebar-toggle-icon-stroke-width) / 2));left:calc(50% - (var(--sidebar-toggle-icon-width) / 2));height:var(--sidebar-toggle-icon-stroke-width);width:var(--sidebar-toggle-icon-width);background-color:currentColor}.sidebar-toggle span:nth-child(1){margin-top:calc(0px - (var(--sidebar-toggle-icon-height) / 2))}.sidebar-toggle span:nth-child(3){margin-top:calc((var(--sidebar-toggle-icon-height) / 2))}@media (min-width: 48em){.sidebar-toggle{position:absolute;overflow:visible;top:var(--sidebar-toggle-offset-top);bottom:auto;left:0;height:var(--sidebar-toggle-height);-webkit-transform:translateX(var(--sidebar-width));transform:translateX(var(--sidebar-width))}.sidebar-toggle .sidebar-toggle-button{top:0}}@media print{.sidebar-toggle{display:none}}@media (max-width: 47.99em){body.close .sidebar,body.close .sidebar-toggle,body.close main>.content{-webkit-transform:translateX(var(--sidebar-width));transform:translateX(var(--sidebar-width))}}@media (min-width: 48em){body.close main>.content{-webkit-transform:translateX(0);transform:translateX(0)}}@media (max-width: 47.99em){body.close nav.app-nav,body.close .github-corner{display:none}}@media (min-width: 48em){body.close .sidebar,body.close .sidebar-toggle{-webkit-transform:translateX(0);transform:translateX(0)}}@media (min-width: 48em){body.close nav.app-nav{margin-left:0}}@media (max-width: 47.99em){body.close .sidebar-toggle{width:100%;max-width:none}body.close .sidebar-toggle span{margin-top:0}body.close .sidebar-toggle span:nth-child(1){-webkit-transform:rotate(45deg);transform:rotate(45deg)}body.close .sidebar-toggle span:nth-child(2){display:none}body.close .sidebar-toggle span:nth-child(3){-webkit-transform:rotate(-45deg);transform:rotate(-45deg)}}@media (min-width: 48em){body.close main>.content{margin-left:0}}@media (min-width: 48em){body.sticky .sidebar,body.sticky .sidebar-toggle{position:fixed}}body .docsify-copy-code-button,body .docsify-copy-code-button:after{border-radius:var(--border-radius-m, 0);border-top-left-radius:0;border-bottom-right-radius:0;background:var(--copycode-background);color:var(--copycode-color)}body .docsify-copy-code-button span{border-radius:var(--border-radius-s, 0)}body .docsify-pagination-container{border-top:var(--pagination-border-top);color:var(--pagination-color)}body .pagination-item-label{font-size:var(--pagination-label-font-size)}body .pagination-item-label svg{color:var(--pagination-label-color);height:var(--pagination-chevron-height);stroke:var(--pagination-chevron-stroke);stroke-linecap:var(--pagination-chevron-stroke-linecap);stroke-linejoin:var(--pagination-chevron-stroke-linecap);stroke-width:var(--pagination-chevron-stroke-width)}body .pagination-item-title{color:var(--pagination-title-color);font-size:var(--pagination-title-font-size)}body .app-name.hide{display:block}body .sidebar{padding:var(--sidebar-padding)}.sidebar .search{margin:0;padding:0;border:0}.sidebar .search input{padding:0;line-height:1;font-size:inherit}.sidebar .search .clear-button{width:auto}.sidebar .search .clear-button svg{-webkit-transform:scale(1);transform:scale(1)}.sidebar .search .matching-post{border:none}.sidebar .search p{font-size:inherit}.sidebar .search{order:var(--search-flex-order);margin:var(--search-margin);padding:var(--search-padding);background:var(--search-background)}.sidebar .search a{color:inherit}.sidebar .search h2{margin:var(--search-result-heading-margin);font-size:var(--search-result-heading-font-size);font-weight:var(--search-result-heading-font-weight);color:var(--search-result-heading-color)}.sidebar .search .input-wrap{margin:var(--search-input-margin);background-color:var(--search-input-background-color);border-width:var(--search-input-border-width, 0);border-style:solid;border-color:var(--search-input-border-color);border-radius:var(--search-input-border-radius)}.sidebar .search input[type="search"]{min-width:0;padding:var(--search-input-padding);border:none;background-color:transparent;background-image:var(--search-input-background-image);background-position:var(--search-input-background-position);background-repeat:var(--search-input-background-repeat);background-size:var(--search-input-background-size);font-size:var(--search-input-font-size);color:var(--search-input-color);transition:var(--search-input-transition)}.sidebar .search input[type="search"]::-ms-clear{display:none}.sidebar .search input[type="search"]::-webkit-input-placeholder{color:var(--search-input-placeholder-color, gray)}.sidebar .search input[type="search"]:-ms-input-placeholder{color:var(--search-input-placeholder-color, gray)}.sidebar .search input[type="search"]::-ms-input-placeholder{color:var(--search-input-placeholder-color, gray)}.sidebar .search input[type="search"]::placeholder{color:var(--search-input-placeholder-color, gray)}.sidebar .search input[type="search"]::-webkit-input-placeholder{line-height:normal}.sidebar .search input[type="search"]:focus{background-color:var(--search-input-background-color--focus, var(--search-input-background-color));background-image:var(--search-input-background-image--focus, var(--search-input-background-image));background-position:var(--search-input-background-position--focus, var(--search-input-background-position));background-size:var(--search-input-background-size--focus, var(--search-input-background-size))}@supports (width: env(safe-area-inset)){@media only screen and (orientation: landscape){.sidebar .search input[type="search"]{margin-left:calc(env(safe-area-inset-left) / 2)}}}.sidebar .search p{overflow:hidden;text-overflow:ellipsis;-webkit-line-clamp:2}.sidebar .search p:empty{text-align:center}.sidebar .search .clear-button{margin:0 15px 0 0;padding:0;border:none;line-height:1;background:transparent;cursor:pointer}.sidebar .search .clear-button svg circle{fill:var(--search-clear-icon-color1, gray)}.sidebar .search .clear-button svg path{stroke:var(--search-clear-icon-color2, #fff)}.sidebar .search.show ~ *:not(h1){display:none}.sidebar .search .results-panel{display:none;color:var(--search-result-item-color);font-size:var(--search-result-item-font-size);font-weight:var(--search-result-item-font-weight)}.sidebar .search .results-panel.show{display:block}.sidebar .search .matching-post{margin:var(--search-result-item-margin);padding:var(--search-result-item-padding)}.sidebar .search .matching-post,.sidebar .search .matching-post:last-child{border-width:var(--search-result-item-border-width, 0) !important;border-style:var(--search-result-item-border-style);border-color:var(--search-result-item-border-color)}.sidebar .search .matching-post p{margin:0}.sidebar .search .search-keyword{margin:var(--search-result-keyword-margin);padding:var(--search-result-keyword-padding);border-radius:var(--search-result-keyword-border-radius);background-color:var(--search-result-keyword-background);color:var(--search-result-keyword-color, currentColor);font-style:normal;font-weight:var(--search-result-keyword-font-weight)}.medium-zoom-overlay,.medium-zoom-image--open{z-index:50 !important}.medium-zoom-overlay{background:var(--zoomimage-overlay-background) !important}:root{--mono-hue: 113;--mono-saturation: 0%;--mono-shade3: hsl(var(--mono-hue), var(--mono-saturation), 20%);--mono-shade2: hsl(var(--mono-hue), var(--mono-saturation), 30%);--mono-shade1: hsl(var(--mono-hue), var(--mono-saturation), 40%);--mono-base: hsl(var(--mono-hue), var(--mono-saturation), 50%);--mono-tint1: hsl(var(--mono-hue), var(--mono-saturation), 70%);--mono-tint2: hsl(var(--mono-hue), var(--mono-saturation), 89%);--mono-tint3: hsl(var(--mono-hue), var(--mono-saturation), 97%);--theme-hue: 204;--theme-saturation: 90%;--theme-lightness: 45%;--theme-color: hsl(var(--theme-hue), var(--theme-saturation), var(--theme-lightness));--modular-scale: 1.333;--modular-scale--2: calc(var(--modular-scale--1) / var(--modular-scale));--modular-scale--1: calc(var(--modular-scale-1) / var(--modular-scale));--modular-scale-1: 1rem;--modular-scale-2: calc(var(--modular-scale-1) * var(--modular-scale));--modular-scale-3: calc(var(--modular-scale-2) * var(--modular-scale));--modular-scale-4: calc(var(--modular-scale-3) * var(--modular-scale));--modular-scale-5: calc(var(--modular-scale-4) * var(--modular-scale));--font-size-xxxl: var(--modular-scale-5);--font-size-xxl: var(--modular-scale-4);--font-size-xl: var(--modular-scale-3);--font-size-l: var(--modular-scale-2);--font-size-m: var(--modular-scale-1);--font-size-s: var(--modular-scale--1);--font-size-xs: var(--modular-scale--2);--duration-slow: 1s;--duration-medium: 0.5s;--duration-fast: 0.25s;--spinner-size: 60px;--spinner-track-width: 4px;--spinner-track-color: rgba(0, 0, 0, 0.15);--spinner-transition-duration: var(--duration-medium)}:root{--base-background-color: #fff;--base-color: var(--mono-shade2);--base-font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol";--base-font-size: 16px;--base-font-weight: normal;--base-line-height: 1.7;--emoji-size: calc(var(--base-line-height) * 1em);--hr-border: 1px solid var(--mono-tint2);--mark-background: #ffecb3;--pre-font-family: var(--code-font-family);--pre-font-size: var(--code-font-size);--pre-font-weight: normal;--selection-color: #b4d5fe;--small-font-size: var(--font-size-s);--strong-color: var(--heading-color);--strong-font-weight: 600;--subsup-font-size: var(--font-size-s)}:root{--content-max-width: 55em;--blockquote-background: var(--mono-tint3);--blockquote-border-style: solid;--blockquote-border-radius: var(--border-radius-m);--blockquote-em-font-weight: normal;--blockquote-font-weight: normal;--code-font-family: Inconsolata, Consolas, Menlo, Monaco, "Andale Mono WT", "Andale Mono", "Lucida Console", "DejaVu Sans Mono", "Bitstream Vera Sans Mono", "Courier New", Courier, monospace;--code-font-size: calc(var(--font-size-m) * 0.95);--code-font-weight: normal;--code-tab-size: 4;--code-block-border-radius: var(--border-radius-m);--code-block-line-height: var(--base-line-height);--code-block-margin: 1em 0;--code-block-padding: 1.75em 1.5em 1.5em 1.5em;--code-inline-background: var(--code-theme-background);--code-inline-border-radius: var(--border-radius-s);--code-inline-color: var(--code-theme-text);--code-inline-margin: 0 0.15em;--code-inline-padding: 0.125em 0.4em;--code-theme-background: var(--mono-tint3);--heading-color: var(--mono-shade3);--heading-margin: 2.5rem 0 0;--heading-h1-border-style: solid;--heading-h1-font-size: var(--font-size-xxl);--heading-h2-border-style: solid;--heading-h2-font-size: var(--font-size-xl);--heading-h3-border-style: solid;--heading-h3-font-size: var(--font-size-l);--heading-h4-border-style: solid;--heading-h4-font-size: var(--font-size-m);--heading-h5-border-style: solid;--heading-h5-font-size: var(--font-size-s);--heading-h6-border-style: solid;--heading-h6-font-size: var(--font-size-xs);--kbd-background: var(--mono-tint3);--kbd-border-radius: var(--border-radius-m);--kbd-margin: 0 0.3em;--kbd-min-width: 2.5em;--kbd-padding: 0.65em 0.5em;--link-text-decoration: underline;--notice-background: var(--mono-tint3);--notice-border-radius: var(--border-radius-m);--notice-border-style: solid;--notice-font-weight: normal;--notice-padding: 1em 1.5em;--notice-before-font-weight: normal;--table-cell-padding: 0.75em 0.5em;--table-head-border-color: var(--table-cell-border-color);--table-head-font-weight: var(--strong-font-weight);--table-row-odd-background: var(--mono-tint3)}:root{--cover-margin: 0 auto;--cover-max-width: 40em;--cover-text-align: center;--cover-background-color: var(--base-background-color);--cover-background-mask-color: var(--base-background-color);--cover-background-mask-opacity: 0.8;--cover-background-position: center center;--cover-background-repeat: no-repeat;--cover-background-size: cover;--cover-blockquote-font-size: var(--font-size-l);--cover-border-color: var(--theme-color);--cover-button-border: 1px solid var(--theme-color);--cover-button-border-radius: var(--border-radius-m);--cover-button-color: var(--theme-color);--cover-button-padding: 0.5em 2rem;--cover-button-text-decoration: none;--cover-button-transition: all var(--duration-fast) ease-in-out;--cover-button-primary-background: var(--theme-color);--cover-button-primary-border: 1px solid var(--theme-color);--cover-button-primary-color: #fff;--cover-heading-color: var(--theme-color);--cover-heading-font-size: var(--font-size-xxl);--cover-heading-font-weight: normal;--cover-link-text-decoration: underline }:root{--navbar-root-border-style: solid;--navbar-root-margin: 0 0 0 1.5em;--navbar-root-transition: all var(--duration-fast);--navbar-menu-background: var(--base-background-color);--navbar-menu-border-radius: var(--border-radius-m);--navbar-menu-box-shadow: rgba(45,45,45,0.05) 0px 0px 1px, rgba(49,49,49,0.05) 0px 1px 2px, rgba(42,42,42,0.05) 0px 2px 4px, rgba(32,32,32,0.05) 0px 4px 8px, rgba(49,49,49,0.05) 0px 8px 16px, rgba(35,35,35,0.05) 0px 16px 32px;--navbar-menu-padding: 0.5em;--navbar-menu-transition: all var(--duration-fast);--navbar-menu-link-border-style: solid;--navbar-menu-link-margin: 0.75em 0.5em;--navbar-menu-link-padding: 0.2em 0 }:root{--copycode-background: #808080;--copycode-color: #fff}:root{--docsifytabs-border-color: var(--mono-tint2);--docsifytabs-border-radius-px: var(--border-radius-s);--docsifytabs-tab-background: var(--mono-tint3);--docsifytabs-tab-color: var(--mono-tint1)}:root{--pagination-border-top: 1px solid var(--mono-tint2);--pagination-chevron-height: 0.8em;--pagination-chevron-stroke: currentColor;--pagination-chevron-stroke-linecap: round;--pagination-chevron-stroke-width: 1px;--pagination-label-font-size: var(--font-size-s);--pagination-title-font-size: var(--font-size-l)}:root{--search-margin: 1.5rem 0 0;--search-input-background-repeat: no-repeat;--search-input-border-color: var(--mono-tint1);--search-input-border-width: 1px;--search-input-padding: 0.5em;--search-flex-order: 1;--search-result-heading-color: var(--heading-color);--search-result-heading-font-size: var(--base-font-size);--search-result-heading-font-weight: normal;--search-result-heading-margin: 0 0 0.25em;--search-result-item-border-color: var(--mono-tint2);--search-result-item-border-style: solid;--search-result-item-border-width: 0 0 1px 0;--search-result-item-font-weight: normal;--search-result-item-padding: 1em 0;--search-result-keyword-background: var(--mark-background);--search-result-keyword-border-radius: var(--border-radius-s);--search-result-keyword-color: var(--mark-color);--search-result-keyword-font-weight: normal;--search-result-keyword-margin: 0 0.1em;--search-result-keyword-padding: 0.2em 0}:root{--zoomimage-overlay-background: rgba(0, 0, 0, 0.875)}:root{--sidebar-background: var(--base-background-color);--sidebar-border-width: 0;--sidebar-padding: 0 25px;--sidebar-transition-duration: var(--duration-fast);--sidebar-width: 17rem;--sidebar-name-font-size: var(--font-size-l);--sidebar-name-font-weight: normal;--sidebar-name-margin: 1.5rem 0 0;--sidebar-name-text-align: center;--sidebar-nav-strong-border-color: var(--sidebar-border-color);--sidebar-nav-strong-color: var(--heading-color);--sidebar-nav-strong-font-weight: var(--strong-font-weight);--sidebar-nav-strong-margin: 1.5em 0 0.5em;--sidebar-nav-strong-padding: 0.25em 0;--sidebar-nav-indent: 1em;--sidebar-nav-margin: 1.5rem 0 0;--sidebar-nav-link-border-style: solid;--sidebar-nav-link-border-width: 0;--sidebar-nav-link-color: var(--base-color);--sidebar-nav-link-font-weight: normal;--sidebar-nav-link-padding: 0.25em 0;--sidebar-nav-link-text-decoration--active: underline;--sidebar-nav-link-text-decoration--hover: underline;--sidebar-nav-link-before-margin: 0 0.35em 0 0;--sidebar-nav-pagelink-background-repeat: no-repeat;--sidebar-nav-pagelink-transition: var(--sidebar-nav-link-transition);--sidebar-toggle-border-radius: var(--border-radius-s);--sidebar-toggle-border-style: solid;--sidebar-toggle-border-width: 0;--sidebar-toggle-height: 36px;--sidebar-toggle-icon-color: var(--base-color);--sidebar-toggle-icon-height: 10px;--sidebar-toggle-icon-stroke-width: 1px;--sidebar-toggle-icon-width: 16px;--sidebar-toggle-offset-left: 0;--sidebar-toggle-offset-top: calc(35px - (var(--sidebar-toggle-height) / 2));--sidebar-toggle-width: 44px}:root{--code-theme-background: #f3f3f3;--code-theme-comment: #6e8090;--code-theme-function: #dd4a68;--code-theme-keyword: #07a;--code-theme-operator: #a67f59;--code-theme-punctuation: #999;--code-theme-selection: #b3d4fc;--code-theme-selector: #690;--code-theme-tag: #905;--code-theme-text: #333;--code-theme-variable: #e90}:root{--border-radius-s: 2px;--border-radius-m: 4px;--border-radius-l: 8px;--strong-font-weight: 600;--blockquote-border-color: var(--theme-color);--blockquote-border-radius: 0 var(--border-radius-m) var(--border-radius-m) 0;--blockquote-border-width: 0 0 0 4px;--code-inline-background: var(--mono-tint2);--code-theme-background: var(--mono-tint3);--heading-font-weight: var(--strong-font-weight);--heading-h1-font-weight: 400;--heading-h2-font-weight: 400;--heading-h2-border-color: var(--mono-tint2);--heading-h2-border-width: 0 0 1px 0;--heading-h2-margin: 2.5rem 0 1.5rem;--heading-h2-padding: 0 0 1rem 0;--kbd-border: 1px solid var(--mono-tint2);--notice-border-radius: 0 var(--border-radius-m) var(--border-radius-m) 0;--notice-border-width: 0 0 0 4px;--notice-padding: 1em 1.5em 1em 3em;--notice-before-border-radius: 100%;--notice-before-font-weight: bold;--notice-before-height: 1.5em;--notice-before-left: 0.75em;--notice-before-line-height: 1.5;--notice-before-margin: 0 0.25em 0 0;--notice-before-position: absolute;--notice-before-width: var(--notice-before-height);--notice-important-background: hsl(340, 60%, 96%);--notice-important-border-color: hsl(340, 90%, 45%);--notice-important-before-background: var(--notice-important-border-color) url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 16 16' fill='%23fff'%3E%3Cpath d='M10 14C10 15.1 9.1 16 8 16 6.9 16 6 15.1 6 14 6 12.9 6.9 12 8 12 9.1 12 10 12.9 10 14Z'/%3E%3Cpath d='M10 1.6C10 1.2 9.8 0.9 9.6 0.7 9.2 0.3 8.6 0 8 0 7.4 0 6.8 0.2 6.5 0.6 6.2 0.9 6 1.2 6 1.6 6 1.7 6 1.8 6 1.9L6.8 9.6C6.9 9.9 7 10.1 7.2 10.2 7.4 10.4 7.7 10.5 8 10.5 8.3 10.5 8.6 10.4 8.8 10.3 9 10.1 9.1 9.9 9.2 9.6L10 1.9C10 1.8 10 1.7 10 1.6Z'/%3E%3C/svg%3E") center / 0.875em no-repeat;--notice-important-before-color: #fff;--notice-important-before-content: "";--notice-tip-background: hsl(204, 60%, 96%);--notice-tip-border-color: hsl(204, 90%, 45%);--notice-tip-before-background: var(--notice-tip-border-color) url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 16 16' fill='%23fff'%3E%3Cpath d='M9.1 0C10.2 0 10.7 0.7 10.7 1.6 10.7 2.6 9.8 3.6 8.6 3.6 7.6 3.6 7 3 7 2 7 1.1 7.7 0 9.1 0Z'/%3E%3Cpath d='M5.8 16C5 16 4.4 15.5 5 13.2L5.9 9.1C6.1 8.5 6.1 8.2 5.9 8.2 5.7 8.2 4.6 8.6 3.9 9.1L3.5 8.4C5.6 6.6 7.9 5.6 8.9 5.6 9.8 5.6 9.9 6.6 9.5 8.2L8.4 12.5C8.2 13.2 8.3 13.5 8.5 13.5 8.7 13.5 9.6 13.2 10.4 12.5L10.9 13.2C8.9 15.2 6.7 16 5.8 16Z'/%3E%3C/svg%3E") center / 0.875em no-repeat;--notice-tip-before-color: #fff;--notice-tip-before-content: "";--table-cell-border-color: var(--mono-tint2);--table-cell-border-width: 1px 0;--cover-background-color: hsl(var(--theme-hue), 25%, 60%);--cover-background-image: radial-gradient(ellipse at center 115%, rgba(255, 255, 255, 0.9), transparent);--cover-blockquote-color: var(--strong-color);--cover-heading-color: #fff;--cover-heading-font-size-max: 56;--cover-heading-font-size-min: 34;--cover-heading-font-weight: 200;--navbar-root-color--active: var(--theme-color);--navbar-menu-border-radius: var(--border-radius-m);--navbar-menu-root-background: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='9.6' height='6' viewBox='0 0 9.6 6'%3E%3Cpath d='M1.5 1.5l3.3 3 3.3-3' stroke-width='1.5' stroke='rgb%28179, 179, 179%29' fill='none' stroke-linecap='square' stroke-linejoin='miter' vector-effect='non-scaling-stroke'/%3E%3C/svg%3E") right no-repeat;--navbar-menu-root-padding: 0 18px 0 0;--search-input-background-color: #fff;--search-input-background-image: url("data:image/svg+xml,%3Csvg height='20px' width='20px' viewBox='0 0 24 24' fill='none' stroke='rgba(0, 0, 0, 0.3)' stroke-width='1.5' stroke-linecap='round' stroke-linejoin='round' preserveAspectRatio='xMidYMid meet' xmlns='http://www.w3.org/2000/svg'%3E%3Ccircle cx='10.5' cy='10.5' r='7.5' vector-effect='non-scaling-stroke'%3E%3C/circle%3E%3Cline x1='21' y1='21' x2='15.8' y2='15.8' vector-effect='non-scaling-stroke'%3E%3C/line%3E%3C/svg%3E");--search-input-background-position: 21px center;--search-input-border-color: var(--sidebar-border-color);--search-input-border-width: 1px 0;--search-input-margin: 0 -25px;--search-input-padding: 0.65em 1em 0.65em 50px;--search-input-placeholder-color: rgba(0, 0, 0, 0.4);--search-clear-icon-color1: rgba(0, 0, 0, 0.3);--search-result-heading-font-weight: var(--strong-font-weight);--search-result-item-border-color: var(--sidebar-border-color);--search-result-keyword-border-radius: var(--border-radius-s);--sidebar-background: var(--mono-tint3);--sidebar-border-color: var(--mono-tint2);--sidebar-border-width: 0 1px 0 0;--sidebar-name-color: var(--theme-color);--sidebar-name-font-weight: 300;--sidebar-nav-strong-border-width: 0 0 1px 0;--sidebar-nav-strong-font-size: smaller;--sidebar-nav-strong-margin: 2em -25px 0.75em 0;--sidebar-nav-strong-padding: 0.25em 0 0.75em 0;--sidebar-nav-strong-text-transform: uppercase;--sidebar-nav-link-border-color: transparent;--sidebar-nav-link-border-color--active: var(--theme-color);--sidebar-nav-link-border-width: 0 4px 0 0;--sidebar-nav-link-color--active: var(--theme-color);--sidebar-nav-link-margin: 0 -25px 0 0;--sidebar-nav-link-text-decoration: none;--sidebar-nav-link-text-decoration--active: none;--sidebar-nav-link-text-decoration--hover: underline;--sidebar-nav-link-before-content-l3: '-';--sidebar-nav-pagelink-background-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='7' height='11.2' viewBox='0 0 7 11.2'%3E%3Cpath d='M1.5 1.5l4 4.1 -4 4.1' stroke-width='1.5' stroke='rgb%28179, 179, 179%29' fill='none' stroke-linecap='square' stroke-linejoin='miter' vector-effect='non-scaling-stroke'/%3E%3C/svg%3E");--sidebar-nav-pagelink-background-image--active: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='11.2' height='7' viewBox='0 0 11.2 7'%3E%3Cpath d='M1.5 1.5l4.1 4 4.1-4' stroke-width='1.5' stroke='rgb%2811, 135, 218%29' fill='none' stroke-linecap='square' stroke-linejoin='miter' vector-effect='non-scaling-stroke'/%3E%3C/svg%3E");--sidebar-nav-pagelink-background-image--collapse: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='7' height='11.2' viewBox='0 0 7 11.2'%3E%3Cpath d='M1.5 1.5l4 4.1 -4 4.1' stroke-width='1.5' stroke='rgb%2811, 135, 218%29' fill='none' stroke-linecap='square' stroke-linejoin='miter' vector-effect='non-scaling-stroke'/%3E%3C/svg%3E");--sidebar-nav-pagelink-background-image--loaded: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='11.2' height='7' viewBox='0 0 11.2 7'%3E%3Cpath d='M1.5 1.5l4.1 4 4.1-4' stroke-width='1.5' stroke='rgb%2811, 135, 218%29' fill='none' stroke-linecap='square' stroke-linejoin='miter' vector-effect='non-scaling-stroke'/%3E%3C/svg%3E");--sidebar-nav-pagelink-background-position: 3px center;--sidebar-nav-pagelink-background-position--active: left center;--sidebar-nav-pagelink-background-position--collapse: var(--sidebar-nav-pagelink-background-position);--sidebar-nav-pagelink-background-position--loaded: var(--sidebar-nav-pagelink-background-position--active);--sidebar-nav-pagelink-padding: 0.25em 0 0.25em 20px;--sidebar-nav-pagelink-transition: none;--sidebar-toggle-background: var(--sidebar-border-color);--sidebar-toggle-border-radius: 0 var(--border-radius-s) var(--border-radius-s) 0;--sidebar-toggle-width: 32px} +.github-corner { + position: absolute; + z-index: 40; + top: 0; + right: 0; + border-bottom: 0; + text-decoration: none +} + +.github-corner svg { + height: 70px; + width: 70px; + fill: var(--theme-color); + color: var(--base-background-color) +} + +.github-corner:hover .octo-arm { + -webkit-animation: octocat-wave 560ms ease-in-out; + animation: octocat-wave 560ms ease-in-out +} + +@-webkit-keyframes octocat-wave { + 0%, 100% { + transform: rotate(0) + } + 20%, 60% { + transform: rotate(-25deg) + } + 40%, 80% { + transform: rotate(10deg) + } +} + +@keyframes octocat-wave { + 0%, 100% { + transform: rotate(0) + } + 20%, 60% { + transform: rotate(-25deg) + } + 40%, 80% { + transform: rotate(10deg) + } +} + +.progress { + position: fixed; + z-index: 2147483647; + top: 0; + left: 0; + right: 0; + height: 3px; + width: 0; + background-color: var(--theme-color); + transition: width var(--duration-fast), opacity calc(var(--duration-fast) * 2) +} + +body.ready-transition:after, body.ready-transition > *:not(.progress) { + opacity: 0; + transition: opacity var(--spinner-transition-duration) +} + +body.ready-transition:after { + content: ''; + position: absolute; + z-index: 1000; + top: calc(50% - (var(--spinner-size) / 2)); + left: calc(50% - (var(--spinner-size) / 2)); + height: var(--spinner-size); + width: var(--spinner-size); + border: var(--spinner-track-width, 0) solid var(--spinner-track-color); + border-left-color: var(--theme-color); + border-left-color: var(--theme-color); + border-radius: 50%; + -webkit-animation: spinner var(--duration-slow) infinite linear; + animation: spinner var(--duration-slow) infinite linear +} + +body.ready-transition.ready-spinner:after { + opacity: 1 +} + +body.ready-transition.ready-fix:after { + opacity: 0 +} + +body.ready-transition.ready-fix > *:not(.progress) { + opacity: 1; + transition-delay: var(--spinner-transition-duration) +} + +@-webkit-keyframes spinner { + 0% { + transform: rotate(0deg) + } + 100% { + transform: rotate(360deg) + } +} + +@keyframes spinner { + 0% { + transform: rotate(0deg) + } + 100% { + transform: rotate(360deg) + } +} + +*, *:before, *:after { + box-sizing: inherit; + font-size: inherit; + -webkit-overflow-scrolling: touch; + -webkit-tap-highlight-color: rgba(0, 0, 0, 0); + -webkit-text-size-adjust: none; + -webkit-touch-callout: none +} + +:root { + box-sizing: border-box; + background-color: var(--base-background-color); + font-size: var(--base-font-size); + font-weight: var(--base-font-weight); + line-height: var(--base-line-height); + letter-spacing: var(--base-letter-spacing); + color: var(--base-color); + -webkit-font-smoothing: antialiased; + -moz-osx-font-smoothing: grayscale; + font-smoothing: antialiased +} + +html, button, input, optgroup, select, textarea { + font-family: var(--base-font-family) +} + +button, input, optgroup, select, textarea { + font-size: 100%; + margin: 0 +} + +a { + text-decoration: none; + -webkit-text-decoration-skip: ink; + text-decoration-skip-ink: auto +} + +body { + margin: 0 +} + +hr { + height: 0; + margin: 2em 0; + border: none; + border-bottom: var(--hr-border, 0) +} + +img { + max-width: 100%; + border: 0 +} + +main { + display: block +} + +main.hidden { + display: none +} + +mark { + background: var(--mark-background); + color: var(--mark-color) +} + +pre { + font-family: var(--pre-font-family); + font-size: var(--pre-font-size); + font-weight: var(--pre-font-weight); + line-height: var(--pre-line-height) +} + +small { + display: inline-block; + font-size: var(--small-font-size) +} + +strong { + font-weight: var(--strong-font-weight); + color: var(--strong-color, currentColor) +} + +sub, sup { + font-size: var(--subsup-font-size); + line-height: 0; + position: relative; + vertical-align: baseline +} + +sub { + bottom: -0.25em +} + +sup { + top: -0.5em +} + +body:not([data-platform^="Mac"]) * { + scrollbar-color: hsla(var(--mono-hue), var(--mono-saturation), 50%, 0.3) hsla(var(--mono-hue), var(--mono-saturation), 50%, 0.1); + scrollbar-width: thin +} + +body:not([data-platform^="Mac"]) * ::-webkit-scrollbar { + width: 5px; + height: 5px +} + +body:not([data-platform^="Mac"]) * ::-webkit-scrollbar-thumb { + background: hsla(var(--mono-hue), var(--mono-saturation), 50%, 0.3) +} + +body:not([data-platform^="Mac"]) * ::-webkit-scrollbar-track { + background: hsla(var(--mono-hue), var(--mono-saturation), 50%, 0.1) +} + +::-moz-selection { + background: var(--selection-color) +} + +::selection { + background: var(--selection-color) +} + +.emoji { + height: var(--emoji-size); + vertical-align: middle +} + +.task-list-item { + list-style: none +} + +.task-list-item input { + margin-right: 0.5em; + margin-left: 0; + vertical-align: 0.075em +} + +.markdown-section code[class*="lang-"], .markdown-section pre[data-lang] { + font-family: var(--code-font-family); + font-size: var(--code-font-size); + font-weight: var(--code-font-weight); + letter-spacing: normal; + line-height: var(--code-block-line-height); + -moz-tab-size: var(--code-tab-size); + -o-tab-size: var(--code-tab-size); + tab-size: var(--code-tab-size); + text-align: left; + white-space: pre; + word-spacing: normal; + word-wrap: normal; + word-break: normal; + -webkit-hyphens: none; + -ms-hyphens: none; + hyphens: none +} + +.markdown-section pre[data-lang] { + position: relative; + overflow: hidden; + margin: var(--code-block-margin); + padding: 0; + border-radius: var(--code-block-border-radius) +} + +.markdown-section pre[data-lang]::after { + content: attr(data-lang); + position: absolute; + top: 0.75em; + right: 0.75em; + opacity: 0.6; + color: inherit; + font-size: var(--font-size-s); + line-height: 1 +} + +.markdown-section pre[data-lang] code { + display: block; + overflow: auto; + padding: var(--code-block-padding) +} + +code[class*="lang-"], pre[data-lang] { + color: var(--code-theme-text) +} + +pre[data-lang]::-moz-selection, pre[data-lang] ::-moz-selection, code[class*="lang-"]::-moz-selection, code[class*="lang-"] ::-moz-selection { + background: var(--code-theme-selection, var(--selection-color)) +} + +pre[data-lang]::-moz-selection, pre[data-lang] ::-moz-selection, code[class*="lang-"]::-moz-selection, code[class*="lang-"] ::-moz-selection { + background: var(--code-theme-selection, var(--selection-color)) +} + +pre[data-lang]::selection, pre[data-lang] ::selection, code[class*="lang-"]::selection, code[class*="lang-"] ::selection { + background: var(--code-theme-selection, var(--selection-color)) +} + +:not(pre) > code[class*="lang-"], pre[data-lang] { + background: var(--code-theme-background) +} + +.namespace { + opacity: 0.7 +} + +.token.comment, .token.prolog, .token.doctype, .token.cdata { + color: var(--code-theme-comment) +} + +.token.punctuation { + color: var(--code-theme-punctuation) +} + +.token.property, .token.tag, .token.boolean, .token.number, .token.constant, .token.symbol, .token.deleted { + color: var(--code-theme-tag) +} + +.token.selector, .token.attr-name, .token.string, .token.char, .token.builtin, .token.inserted { + color: var(--code-theme-selector) +} + +.token.operator, .token.entity, .token.url, .language-css .token.string, .style .token.string { + color: var(--code-theme-operator) +} + +.token.atrule, .token.attr-value, .token.keyword { + color: var(--code-theme-keyword) +} + +.token.function { + color: var(--code-theme-function) +} + +.token.regex, .token.important, .token.variable { + color: var(--code-theme-variable) +} + +.token.important, .token.bold { + font-weight: bold +} + +.token.italic { + font-style: italic +} + +.token.entity { + cursor: help +} + +.markdown-section { + position: relative; + max-width: var(--content-max-width); + margin: 0 auto; + padding: 2rem 45px +} + +.app-nav:not(:empty) ~ main .markdown-section { + padding-top: 3.5rem +} + +.markdown-section figure, .markdown-section p, .markdown-section ol, .markdown-section ul { + margin: 1em 0 +} + +.markdown-section ol, .markdown-section ul { + padding-left: 1.5rem +} + +.markdown-section ol ol, .markdown-section ol ul, .markdown-section ul ol, .markdown-section ul ul { + margin-top: 0.15rem; + margin-bottom: 0.15rem +} + +.markdown-section a { + border-bottom: var(--link-border-bottom); + color: var(--link-color); + -webkit-text-decoration: var(--link-text-decoration); + text-decoration: var(--link-text-decoration); + -webkit-text-decoration-color: var(--link-text-decoration-color); + text-decoration-color: var(--link-text-decoration-color) +} + +.markdown-section a:hover { + border-bottom: var(--link-border-bottom--hover, var(--link-border-bottom, 0)); + color: var(--link-color--hover, var(--link-color)); + -webkit-text-decoration: var(--link-text-decoration--hover, var(--link-text-decoration)); + text-decoration: var(--link-text-decoration--hover, var(--link-text-decoration)); + -webkit-text-decoration-color: var(--link-text-decoration-color--hover, var(--link-text-decoration-color)); + text-decoration-color: var(--link-text-decoration-color--hover, var(--link-text-decoration-color)) +} + +.markdown-section a.anchor { + border-bottom: 0; + color: inherit; + text-decoration: none +} + +.markdown-section a.anchor:hover { + text-decoration: underline +} + +.markdown-section blockquote { + overflow: visible; + margin: 2em 0; + padding: 1.5em; + border-width: var(--blockquote-border-width, 0); + border-style: var(--blockquote-border-style); + border-color: var(--blockquote-border-color); + border-radius: var(--blockquote-border-radius); + background: var(--blockquote-background); + color: var(--blockquote-color); + font-family: var(--blockquote-font-family); + font-size: var(--blockquote-font-size); + font-style: var(--blockquote-font-style); + font-weight: var(--blockquote-font-weight); + quotes: "“" "”" "‘" "’" +} + +.markdown-section blockquote em { + font-family: var(--blockquote-em-font-family); + font-size: var(--blockquote-em-font-size); + font-style: var(--blockquote-em-font-style); + font-weight: var(--blockquote-em-font-weight) +} + +.markdown-section blockquote p:first-child { + margin-top: 0 +} + +.markdown-section blockquote p:first-child:before, .markdown-section blockquote p:first-child:after { + color: var(--blockquote-quotes-color); + font-family: var(--blockquote-quotes-font-family); + font-size: var(--blockquote-quotes-font-size); + line-height: 0 +} + +.markdown-section blockquote p:first-child:before { + content: var(--blockquote-quotes-open); + margin-right: 0.15em; + vertical-align: -0.45em +} + +.markdown-section blockquote p:first-child:after { + content: var(--blockquote-quotes-close); + margin-left: 0.15em; + vertical-align: -0.55em +} + +.markdown-section blockquote p:last-child { + margin-bottom: 0 +} + +.markdown-section code { + font-family: var(--code-font-family); + font-size: var(--code-font-size); + font-weight: var(--code-font-weight); + line-height: inherit +} + +.markdown-section code:not([class*="lang-"]):not([class*="language-"]) { + margin: var(--code-inline-margin); + padding: var(--code-inline-padding); + border-radius: var(--code-inline-border-radius); + background: var(--code-inline-background); + color: var(--code-inline-color, currentColor); + white-space: nowrap +} + +.markdown-section h1:first-child, .markdown-section h2:first-child, .markdown-section h3:first-child, .markdown-section h4:first-child, .markdown-section h5:first-child, .markdown-section h6:first-child { + margin-top: 0 +} + +.markdown-section h1 + h2, .markdown-section h1 + h3, .markdown-section h1 + h4, .markdown-section h1 + h5, .markdown-section h1 + h6, .markdown-section h2 + h3, .markdown-section h2 + h4, .markdown-section h2 + h5, .markdown-section h2 + h6, .markdown-section h3 + h4, .markdown-section h3 + h5, .markdown-section h3 + h6, .markdown-section h4 + h5, .markdown-section h4 + h6, .markdown-section h5 + h6 { + margin-top: 1rem +} + +.markdown-section h1 { + margin: var(--heading-h1-margin, var(--heading-margin)); + padding: var(--heading-h1-padding, var(--heading-padding)); + border-width: var(--heading-h1-border-width, 0); + border-style: var(--heading-h1-border-style); + border-color: var(--heading-h1-border-color); + font-family: var(--heading-h1-font-family, var(--heading-font-family)); + font-size: var(--heading-h1-font-size); + font-weight: var(--heading-h1-font-weight, var(--heading-font-weight)); + line-height: var(--base-line-height); + color: var(--heading-h1-color, var(--heading-color)) +} + +.markdown-section h2 { + margin: var(--heading-h2-margin, var(--heading-margin)); + padding: var(--heading-h2-padding, var(--heading-padding)); + border-width: var(--heading-h2-border-width, 0); + border-style: var(--heading-h2-border-style); + border-color: var(--heading-h2-border-color); + font-family: var(--heading-h2-font-family, var(--heading-font-family)); + font-size: var(--heading-h2-font-size); + font-weight: var(--heading-h2-font-weight, var(--heading-font-weight)); + line-height: var(--base-line-height); + color: var(--heading-h2-color, var(--heading-color)) +} + +.markdown-section h3 { + margin: var(--heading-h3-margin, var(--heading-margin)); + padding: var(--heading-h3-padding, var(--heading-padding)); + border-width: var(--heading-h3-border-width, 0); + border-style: var(--heading-h3-border-style); + border-color: var(--heading-h3-border-color); + font-family: var(--heading-h3-font-family, var(--heading-font-family)); + font-size: var(--heading-h3-font-size); + font-weight: var(--heading-h3-font-weight, var(--heading-font-weight)); + color: var(--heading-h3-color, var(--heading-color)) +} + +.markdown-section h4 { + margin: var(--heading-h4-margin, var(--heading-margin)); + padding: var(--heading-h4-padding, var(--heading-padding)); + border-width: var(--heading-h4-border-width, 0); + border-style: var(--heading-h4-border-style); + border-color: var(--heading-h4-border-color); + font-family: var(--heading-h4-font-family, var(--heading-font-family)); + font-size: var(--heading-h4-font-size); + font-weight: var(--heading-h4-font-weight, var(--heading-font-weight)); + color: var(--heading-h4-color, var(--heading-color)) +} + +.markdown-section h5 { + margin: var(--heading-h5-margin, var(--heading-margin)); + padding: var(--heading-h5-padding, var(--heading-padding)); + border-width: var(--heading-h5-border-width, 0); + border-style: var(--heading-h5-border-style); + border-color: var(--heading-h5-border-color); + font-family: var(--heading-h5-font-family, var(--heading-font-family)); + font-weight: var(--heading-h5-font-weight, var(--heading-font-weight)); + color: var(--heading-h5-color, var(--heading-color)) +} + +.markdown-section h6 { + margin: var(--heading-h6-margin, var(--heading-margin)); + padding: var(--heading-h6-padding, var(--heading-padding)); + border-width: var(--heading-h6-border-width, 0); + border-style: var(--heading-h6-border-style); + border-color: var(--heading-h6-border-color); + font-family: var(--heading-h6-font-family, var(--heading-font-family)); + font-size: var(--heading-h6-font-size); + font-weight: var(--heading-h6-font-weight, var(--heading-font-weight)); + color: var(--heading-h6-color, var(--heading-color)) +} + +.markdown-section iframe { + margin: 1em 0 +} + +.markdown-section img { + max-width: 100% +} + +.markdown-section kbd { + display: inline-block; + min-width: var(--kbd-min-width); + margin: var(--kbd-margin); + padding: var(--kbd-padding); + border: var(--kbd-border); + border-radius: var(--kbd-border-radius); + background: var(--kbd-background); + font-family: inherit; + font-size: var(--kbd-font-size); + text-align: center; + letter-spacing: 0; + line-height: 1; + color: var(--kbd-color) +} + +.markdown-section kbd + kbd { + margin-left: -0.15em +} + +.markdown-section table { + display: block; + overflow: auto; + margin: 1rem 0; + border-spacing: 0; + border-collapse: collapse +} + +.markdown-section th, .markdown-section td { + padding: var(--table-cell-padding) +} + +.markdown-section th:not([align]) { + text-align: left +} + +.markdown-section thead { + border-color: var(--table-head-border-color); + border-style: solid; + border-width: var(--table-head-border-width, 0); + background: var(--table-head-background) +} + +.markdown-section th { + font-weight: var(--table-head-font-weight); + color: var(--strong-color) +} + +.markdown-section td { + border-color: var(--table-cell-border-color); + border-style: solid; + border-width: var(--table-cell-border-width, 0) +} + +.markdown-section tbody { + border-color: var(--table-body-border-color); + border-style: solid; + border-width: var(--table-body-border-width, 0) +} + +.markdown-section tbody tr:nth-child(odd) { + background: var(--table-row-odd-background) +} + +.markdown-section tbody tr:nth-child(even) { + background: var(--table-row-even-background) +} + +.markdown-section > ul .task-list-item { + margin-left: -1.25em +} + +.markdown-section > ul .task-list-item .task-list-item { + margin-left: 0 +} + +.markdown-section .table-wrapper { + overflow-x: auto +} + +.markdown-section .table-wrapper table { + display: table; + width: 100% +} + +.markdown-section .table-wrapper td::before { + display: none +} + +@media (max-width: 30em) { + .markdown-section .table-wrapper tbody, .markdown-section .table-wrapper tr, .markdown-section .table-wrapper td { + display: block + } + + .markdown-section .table-wrapper th, .markdown-section .table-wrapper td { + border: none + } + + .markdown-section .table-wrapper thead { + display: none + } + + .markdown-section .table-wrapper tr { + border-color: var(--table-cell-border-color); + border-style: solid; + border-width: var(--table-cell-border-width, 0); + padding: var(--table-cell-padding) + } + + .markdown-section .table-wrapper tr:not(:last-child) { + border-bottom: 0 + } + + .markdown-section .table-wrapper td { + padding: 0.15em 0 0.15em 8em + } + + .markdown-section .table-wrapper td::before { + display: inline-block; + float: left; + width: 8em; + margin-left: -8em; + font-weight: bold; + text-align: left + } +} + +.markdown-section .tip, .markdown-section .warn { + position: relative; + margin: 2em 0; + padding: var(--notice-padding); + border-width: var(--notice-border-width, 0); + border-style: var(--notice-border-style); + border-color: var(--notice-border-color); + border-radius: var(--notice-border-radius); + background: var(--notice-background); + font-family: var(--notice-font-family); + font-weight: var(--notice-font-weight); + color: var(--notice-color) +} + +.markdown-section .tip:before, .markdown-section .warn:before { + display: inline-block; + position: var(--notice-before-position, relative); + top: var(--notice-before-top); + left: var(--notice-before-left); + height: var(--notice-before-height); + width: var(--notice-before-width); + margin: var(--notice-before-margin); + padding: var(--notice-before-padding); + border-radius: var(--notice-before-border-radius); + line-height: var(--notice-before-line-height); + font-family: var(--notice-before-font-family); + font-size: var(--notice-before-font-size); + font-weight: var(--notice-before-font-weight); + text-align: center +} + +.markdown-section .tip { + border-width: var(--notice-important-border-width, var(--notice-border-width, 0)); + border-style: var(--notice-important-border-style, var(--notice-border-style)); + border-color: var(--notice-important-border-color, var(--notice-border-color)); + background: var(--notice-important-background, var(--notice-background)); + color: var(--notice-important-color, var(--notice-color)) +} + +.markdown-section .tip:before { + content: var(--notice-important-before-content, var(--notice-before-content)); + background: var(--notice-important-before-background, var(--notice-before-background)); + color: var(--notice-important-before-color, var(--notice-before-color)) +} + +.markdown-section .warn { + border-width: var(--notice-tip-border-width, var(--notice-border-width, 0)); + border-style: var(--notice-tip-border-style, var(--notice-border-style)); + border-color: var(--notice-tip-border-color, var(--notice-border-color)); + background: var(--notice-tip-background, var(--notice-background)); + color: var(--notice-tip-color, var(--notice-color)) +} + +.markdown-section .warn:before { + content: var(--notice-tip-before-content, var(--notice-before-content)); + background: var(--notice-tip-before-background, var(--notice-before-background)); + color: var(--notice-tip-before-color, var(--notice-before-color)) +} + +.cover { + display: none; + position: relative; + z-index: 20; + min-height: 100vh; + flex-direction: column; + align-items: center; + justify-content: center; + padding: calc(var(--cover-border-inset, 0px) + var(--cover-border-width, 0px)); + color: var(--cover-color); + text-align: var(--cover-text-align) +} + +@media screen and (-ms-high-contrast: active), screen and (-ms-high-contrast: none) { + .cover { + height: 100vh + } +} + +.cover:before, .cover:after { + /*content: '';*/ + position: absolute +} + +.cover:before { + top: 0; + bottom: 0; + left: 0; + right: 0; + background-blend-mode: var(--cover-background-blend-mode); + background-color: var(--cover-background-color); + background-image: var(--cover-background-image); + background-position: var(--cover-background-position); + background-repeat: var(--cover-background-repeat); + background-size: var(--cover-background-size) +} + +.cover:after { + top: var(--cover-border-inset, 0); + bottom: var(--cover-border-inset, 0); + left: var(--cover-border-inset, 0); + right: var(--cover-border-inset, 0); + border-width: var(--cover-border-width, 0); + border-style: solid; + border-color: var(--cover-border-color) +} + +.cover a { + border-bottom: var(--cover-link-border-bottom); + color: var(--cover-link-color); + -webkit-text-decoration: var(--cover-link-text-decoration); + text-decoration: var(--cover-link-text-decoration); + -webkit-text-decoration-color: var(--cover-link-text-decoration-color); + text-decoration-color: var(--cover-link-text-decoration-color) +} + +.cover a:hover { + border-bottom: var(--cover-link-border-bottom--hover, var(--cover-link-border-bottom)); + color: var(--cover-link-color--hover, var(--cover-link-color)); + -webkit-text-decoration: var(--cover-link-text-decoration--hover, var(--cover-link-text-decoration)); + text-decoration: var(--cover-link-text-decoration--hover, var(--cover-link-text-decoration)); + -webkit-text-decoration-color: var(--cover-link-text-decoration-color--hover, var(--cover-link-text-decoration-color)); + text-decoration-color: var(--cover-link-text-decoration-color--hover, var(--cover-link-text-decoration-color)) +} + +.cover h1 { + color: var(--cover-heading-color); + position: relative; + margin: 0; + font-size: var(--cover-heading-font-size); + font-weight: var(--cover-heading-font-weight); + line-height: 1.2 +} + +.cover h1 a, .cover h1 a:hover { + display: block; + border-bottom: none; + color: inherit; + text-decoration: none +} + +.cover h1 small { + position: absolute; + bottom: 0; + margin-left: 0.5em +} + +.cover h1 span { + font-size: calc(var(--cover-heading-font-size-min) * 1px) +} + +@media (min-width: 26em) { + .cover h1 span { + font-size: calc((var(--cover-heading-font-size-min) * 1px) + (var(--cover-heading-font-size-max) - var(--cover-heading-font-size-min)) * ((100vw - 420px) / (1024 - 420))) + } +} + +@media (min-width: 64em) { + .cover h1 span { + font-size: calc(var(--cover-heading-font-size-max) * 1px) + } +} + +.cover blockquote { + margin: 0; + color: var(--cover-blockquote-color); + font-size: var(--cover-blockquote-font-size) +} + +.cover blockquote a { + color: inherit +} + +.cover ul { + padding: 0; + list-style-type: none +} + +.cover .cover-main { + position: relative; + z-index: 1; + max-width: var(--cover-max-width); + margin: var(--cover-margin); + padding: 0 45px +} + +.cover .cover-main > p:last-child { + margin: 1.25em -.25em +} + +.cover .cover-main > p:last-child a { + display: block; + margin: .375em .25em; + padding: var(--cover-button-padding); + border: var(--cover-button-border); + border-radius: var(--cover-button-border-radius); + box-shadow: var(--cover-button-box-shadow); + background: var(--cover-button-background); + text-align: center; + -webkit-text-decoration: var(--cover-button-text-decoration); + text-decoration: var(--cover-button-text-decoration); + -webkit-text-decoration-color: var(--cover-button-text-decoration-color); + text-decoration-color: var(--cover-button-text-decoration-color); + color: var(--cover-button-color); + white-space: nowrap; + transition: var(--cover-button-transition) +} + +.cover .cover-main > p:last-child a:hover { + border: var(--cover-button-border--hover, var(--cover-button-border)); + box-shadow: var(--cover-button-box-shadow--hover, var(--cover-button-box-shadow)); + background: var(--cover-button-background--hover, var(--cover-button-background)); + -webkit-text-decoration: var(--cover-button-text-decoration--hover, var(--cover-button-text-decoration)); + text-decoration: var(--cover-button-text-decoration--hover, var(--cover-button-text-decoration)); + -webkit-text-decoration-color: var(--cover-button-text-decoration-color--hover, var(--cover-button-text-decoration-color)); + text-decoration-color: var(--cover-button-text-decoration-color--hover, var(--cover-button-text-decoration-color)); + color: var(--cover-button-color--hover, var(--cover-button-color)) +} + +.cover .cover-main > p:last-child a:first-child { + border: var(--cover-button-primary-border, var(--cover-button-border)); + box-shadow: var(--cover-button-primary-box-shadow, var(--cover-button-box-shadow)); + background: var(--cover-button-primary-background, var(--cover-button-background)); + -webkit-text-decoration: var(--cover-button-primary-text-decoration, var(--cover-button-text-decoration)); + text-decoration: var(--cover-button-primary-text-decoration, var(--cover-button-text-decoration)); + -webkit-text-decoration-color: var(--cover-button-primary-text-decoration-color, var(--cover-button-text-decoration-color)); + text-decoration-color: var(--cover-button-primary-text-decoration-color, var(--cover-button-text-decoration-color)); + color: var(--cover-button-primary-color, var(--cover-button-color)) +} + +.cover .cover-main > p:last-child a:first-child:hover { + border: var(--cover-button-primary-border--hover, var(--cover-button-border--hover, var(--cover-button-primary-border, var(--cover-button-border)))); + box-shadow: var(--cover-button-primary-box-shadow--hover, var(--cover-button-box-shadow--hover, var(--cover-button-primary-box-shadow, var(--cover-button-box-shadow)))); + background: var(--cover-button-primary-background--hover, var(--cover-button-background--hover, var(--cover-button-primary-background, var(--cover-button-background)))); + -webkit-text-decoration: var(--cover-button-primary-text-decoration--hover, var(--cover-button-text-decoration--hover, var(--cover-button-primary-text-decoration, var(--cover-button-text-decoration)))); + text-decoration: var(--cover-button-primary-text-decoration--hover, var(--cover-button-text-decoration--hover, var(--cover-button-primary-text-decoration, var(--cover-button-text-decoration)))); + -webkit-text-decoration-color: var(--cover-button-primary-text-decoration-color--hover, var(--cover-button-text-decoration-color--hover, var(--cover-button-primary-text-decoration-color, var(--cover-button-text-decoration-color)))); + text-decoration-color: var(--cover-button-primary-text-decoration-color--hover, var(--cover-button-text-decoration-color--hover, var(--cover-button-primary-text-decoration-color, var(--cover-button-text-decoration-color)))); + color: var(--cover-button-primary-color--hover, var(--cover-button-color--hover, var(--cover-button-primary-color, var(--cover-button-color)))) +} + +@media (min-width: 30.01em) { + .cover .cover-main > p:last-child a { + display: inline-block + } +} + +.cover .mask { + visibility: var(--cover-background-mask-visibility, hidden); + position: absolute; + top: 0; + bottom: 0; + left: 0; + right: 0; + background-color: var(--cover-background-mask-color); + opacity: var(--cover-background-mask-opacity) +} + +.cover.has-mask .mask { + visibility: visible +} + +.cover.show { + display: flex +} + +.app-nav { + position: absolute; + z-index: 30; + top: calc(35px - (0.5em * var(--base-line-height))); + left: 45px; + right: 80px; + text-align: right +} + +.app-nav.no-badge { + right: 45px +} + +.app-nav li > img, .app-nav li > a > img { + margin-top: -0.25em; + vertical-align: middle +} + +.app-nav li > img:first-child, .app-nav li > a > img:first-child { + margin-right: 0.5em +} + +.app-nav ul, .app-nav li { + margin: 0; + padding: 0; + list-style: none +} + +.app-nav li { + position: relative +} + +.app-nav li a { + display: block; + line-height: 1; + transition: var(--navbar-root-transition) +} + +.app-nav > ul > li { + display: inline-block; + margin: var(--navbar-root-margin) +} + +.app-nav > ul > li:first-child { + margin-left: 0 +} + +.app-nav > ul > li:last-child { + margin-right: 0 +} + +.app-nav > ul > li > a, .app-nav > ul > li > span { + padding: var(--navbar-root-padding); + border-width: var(--navbar-root-border-width, 0); + border-style: var(--navbar-root-border-style); + border-color: var(--navbar-root-border-color); + border-radius: var(--navbar-root-border-radius); + background: var(--navbar-root-background); + color: var(--navbar-root-color); + -webkit-text-decoration: var(--navbar-root-text-decoration); + text-decoration: var(--navbar-root-text-decoration); + -webkit-text-decoration-color: var(--navbar-root-text-decoration-color); + text-decoration-color: var(--navbar-root-text-decoration-color) +} + +.app-nav > ul > li > a:hover, .app-nav > ul > li > span:hover { + background: var(--navbar-root-background--hover, var(--navbar-root-background)); + border-style: var(--navbar-root-border-style--hover, var(--navbar-root-border-style)); + border-color: var(--navbar-root-border-color--hover, var(--navbar-root-border-color)); + color: var(--navbar-root-color--hover, var(--navbar-root-color)); + -webkit-text-decoration: var(--navbar-root-text-decoration--hover, var(--navbar-root-text-decoration)); + text-decoration: var(--navbar-root-text-decoration--hover, var(--navbar-root-text-decoration)); + -webkit-text-decoration-color: var(--navbar-root-text-decoration-color--hover, var(--navbar-root-text-decoration-color)); + text-decoration-color: var(--navbar-root-text-decoration-color--hover, var(--navbar-root-text-decoration-color)) +} + +.app-nav > ul > li > a:not(:last-child), .app-nav > ul > li > span:not(:last-child) { + padding: var(--navbar-menu-root-padding, var(--navbar-root-padding)); + background: var(--navbar-menu-root-background, var(--navbar-root-background)) +} + +.app-nav > ul > li > a:not(:last-child):hover, .app-nav > ul > li > span:not(:last-child):hover { + background: var(--navbar-menu-root-background--hover, var(--navbar-menu-root-background, var(--navbar-root-background--hover, var(--navbar-root-background)))) +} + +.app-nav > ul > li > a.active { + background: var(--navbar-root-background--active, var(--navbar-root-background)); + border-style: var(--navbar-root-border-style--active, var(--navbar-root-border-style)); + border-color: var(--navbar-root-border-color--active, var(--navbar-root-border-color)); + color: var(--navbar-root-color--active, var(--navbar-root-color)); + -webkit-text-decoration: var(--navbar-root-text-decoration--active, var(--navbar-root-text-decoration)); + text-decoration: var(--navbar-root-text-decoration--active, var(--navbar-root-text-decoration)); + -webkit-text-decoration-color: var(--navbar-root-text-decoration-color--active, var(--navbar-root-text-decoration-color)); + text-decoration-color: var(--navbar-root-text-decoration-color--active, var(--navbar-root-text-decoration-color)) +} + +.app-nav > ul > li > a.active:not(:last-child):hover { + background: var(--navbar-menu-root-background--active, var(--navbar-menu-root-background, var(--navbar-root-background--active, var(--navbar-root-background)))) +} + +.app-nav > ul > li ul { + visibility: hidden; + position: absolute; + top: 100%; + right: 50%; + overflow-y: auto; + box-sizing: border-box; + max-height: calc(50vh); + padding: var(--navbar-menu-padding); + border-width: var(--navbar-menu-border-width, 0); + border-style: solid; + border-color: var(--navbar-menu-border-color); + border-radius: var(--navbar-menu-border-radius); + background: var(--navbar-menu-background); + box-shadow: var(--navbar-menu-box-shadow); + text-align: left; + white-space: nowrap; + opacity: 0; + transform: translate(50%, -0.35em); + transition: var(--navbar-menu-transition) +} + +.app-nav > ul > li ul li { + white-space: nowrap +} + +.app-nav > ul > li ul a { + margin: var(--navbar-menu-link-margin); + padding: var(--navbar-menu-link-padding); + border-width: var(--navbar-menu-link-border-width, 0); + border-style: var(--navbar-menu-link-border-style); + border-color: var(--navbar-menu-link-border-color); + border-radius: var(--navbar-menu-link-border-radius); + background: var(--navbar-menu-link-background); + color: var(--navbar-menu-link-color); + -webkit-text-decoration: var(--navbar-menu-link-text-decoration); + text-decoration: var(--navbar-menu-link-text-decoration); + -webkit-text-decoration-color: var(--navbar-menu-link-text-decoration-color); + text-decoration-color: var(--navbar-menu-link-text-decoration-color) +} + +.app-nav > ul > li ul a:hover { + background: var(--navbar-menu-link-background--hover, var(--navbar-menu-link-background)); + border-style: var(--navbar-menu-link-border-style--hover, var(--navbar-menu-link-border-style)); + border-color: var(--navbar-menu-link-border-color--hover, var(--navbar-menu-link-border-color)); + color: var(--navbar-menu-link-color--hover, var(--navbar-menu-link-color)); + -webkit-text-decoration: var(--navbar-menu-link-text-decoration--hover, var(--navbar-menu-link-text-decoration)); + text-decoration: var(--navbar-menu-link-text-decoration--hover, var(--navbar-menu-link-text-decoration)); + -webkit-text-decoration-color: var(--navbar-menu-link-text-decoration-color--hover, var(--navbar-menu-link-text-decoration-color)); + text-decoration-color: var(--navbar-menu-link-text-decoration-color--hover, var(--navbar-menu-link-text-decoration-color)) +} + +.app-nav > ul > li ul a.active { + background: var(--navbar-menu-link-background--active, var(--navbar-menu-link-background)); + border-style: var(--navbar-menu-link-border-style--active, var(--navbar-menu-link-border-style)); + border-color: var(--navbar-menu-link-border-color--active, var(--navbar-menu-link-border-color)); + color: var(--navbar-menu-link-color--active, var(--navbar-menu-link-color)); + -webkit-text-decoration: var(--navbar-menu-link-text-decoration--active, var(--navbar-menu-link-text-decoration)); + text-decoration: var(--navbar-menu-link-text-decoration--active, var(--navbar-menu-link-text-decoration)); + -webkit-text-decoration-color: var(--navbar-menu-link-text-decoration-color--active, var(--navbar-menu-link-text-decoration-color)); + text-decoration-color: var(--navbar-menu-link-text-decoration-color--active, var(--navbar-menu-link-text-decoration-color)) +} + +.app-nav > ul > li:hover ul, .app-nav > ul > li:focus ul, .app-nav > ul > li.focus-within ul { + visibility: visible; + opacity: 1; + transform: translate(50%, 0) +} + +.sidebar, .sidebar-toggle, main > .content { + transition: all var(--sidebar-transition-duration) ease-out +} + +@media (min-width: 48em) { + nav.app-nav { + margin-left: var(--sidebar-width) + } +} + +main { + position: relative; + overflow-x: hidden; + min-height: 100vh +} + +@media (min-width: 48em) { + main > .content { + margin-left: var(--sidebar-width) + } +} + +.sidebar { + display: flex; + flex-direction: column; + position: fixed; + z-index: 10; + top: 0; + right: 100%; + overflow-x: hidden; + overflow-y: auto; + height: 100vh; + width: var(--sidebar-width); + padding: var(--sidebar-padding); + border-width: var(--sidebar-border-width); + border-style: solid; + border-color: var(--sidebar-border-color); + background: var(--sidebar-background) +} + +.sidebar > h1 { + margin: 0; + margin: var(--sidebar-name-margin); + padding: var(--sidebar-name-padding); + background: var(--sidebar-name-background); + color: var(--sidebar-name-color); + font-family: var(--sidebar-name-font-family); + font-size: var(--sidebar-name-font-size); + font-weight: var(--sidebar-name-font-weight); + text-align: var(--sidebar-name-text-align) +} + +.sidebar > h1 img { + max-width: 100% +} + +.sidebar > h1 .app-name-link { + color: var(--sidebar-name-color) +} + +body:not([data-platform^="Mac"]) .sidebar::-webkit-scrollbar { + width: 5px +} + +body:not([data-platform^="Mac"]) .sidebar::-webkit-scrollbar-thumb { + border-radius: 50vw +} + +@media (min-width: 48em) { + .sidebar { + position: absolute; + transform: translateX(var(--sidebar-width)) + } +} + +@media print { + .sidebar { + display: none + } +} + +.sidebar-nav, .sidebar nav { + order: 1; + margin: var(--sidebar-nav-margin); + padding: var(--sidebar-nav-padding); + background: var(--sidebar-nav-background) +} + +.sidebar-nav ul, .sidebar nav ul { + margin: 0; + padding: 0; + list-style: none +} + +.sidebar-nav ul ul, .sidebar nav ul ul { + margin-left: var(--sidebar-nav-indent) +} + +.sidebar-nav a, .sidebar nav a { + display: block; + overflow: hidden; + margin: var(--sidebar-nav-link-margin); + padding: var(--sidebar-nav-link-padding); + border-width: var(--sidebar-nav-link-border-width, 0); + border-style: var(--sidebar-nav-link-border-style); + border-color: var(--sidebar-nav-link-border-color); + border-radius: var(--sidebar-nav-link-border-radius); + background-color: var(--sidebar-nav-link-background-color); + background-image: var(--sidebar-nav-link-background-image); + background-position: var(--sidebar-nav-link-background-position); + background-repeat: var(--sidebar-nav-link-background-repeat); + background-size: var(--sidebar-nav-link-background-size); + color: var(--sidebar-nav-link-color); + font-weight: var(--sidebar-nav-link-font-weight); + white-space: nowrap; + -webkit-text-decoration: var(--sidebar-nav-link-text-decoration); + text-decoration: var(--sidebar-nav-link-text-decoration); + -webkit-text-decoration-color: var(--sidebar-nav-link-text-decoration-color); + text-decoration-color: var(--sidebar-nav-link-text-decoration-color); + text-overflow: ellipsis; + transition: var(--sidebar-nav-link-transition) +} + +.sidebar-nav a img, .sidebar nav a img { + margin-top: -0.25em; + vertical-align: middle +} + +.sidebar-nav a img:first-child, .sidebar nav a img:first-child { + margin-right: 0.5em +} + +.sidebar-nav a:hover, .sidebar nav a:hover { + border-width: var(--sidebar-nav-link-border-width--hover, var(--sidebar-nav-link-border-width, 0)); + border-style: var(--sidebar-nav-link-border-style--hover, var(--sidebar-nav-link-border-style)); + border-color: var(--sidebar-nav-link-border-color--hover, var(--sidebar-nav-link-border-color)); + background-color: var(--sidebar-nav-link-background-color--hover, var(--sidebar-nav-link-background-color)); + background-image: var(--sidebar-nav-link-background-image--hover, var(--sidebar-nav-link-background-image)); + background-position: var(--sidebar-nav-link-background-position--hover, var(--sidebar-nav-link-background-position)); + background-size: var(--sidebar-nav-link-background-size--hover, var(--sidebar-nav-link-background-size)); + color: var(--sidebar-nav-link-color--hover, var(--sidebar-nav-link-color)); + font-weight: var(--sidebar-nav-link-font-weight--hover, var(--sidebar-nav-link-font-weight)); + -webkit-text-decoration: var(--sidebar-nav-link-text-decoration--hover, var(--sidebar-nav-link-text-decoration)); + text-decoration: var(--sidebar-nav-link-text-decoration--hover, var(--sidebar-nav-link-text-decoration)); + -webkit-text-decoration-color: var(--sidebar-nav-link-text-decoration-color); + text-decoration-color: var(--sidebar-nav-link-text-decoration-color) +} + +.sidebar-nav ul > li > span, .sidebar-nav ul > li > strong, .sidebar nav ul > li > span, .sidebar nav ul > li > strong { + display: block; + margin: var(--sidebar-nav-strong-margin); + padding: var(--sidebar-nav-strong-padding); + border-width: var(--sidebar-nav-strong-border-width, 0); + border-style: solid; + border-color: var(--sidebar-nav-strong-border-color); + color: var(--sidebar-nav-strong-color); + font-size: 1rem; + font-weight: var(--sidebar-nav-strong-font-weight); + text-transform: var(--sidebar-nav-strong-text-transform) +} + +.sidebar-nav ul > li > span + ul, .sidebar-nav ul > li > strong + ul, .sidebar nav ul > li > span + ul, .sidebar nav ul > li > strong + ul { + margin-left: 0 +} + +.sidebar-nav ul > li:first-child > span, .sidebar-nav ul > li:first-child > strong, .sidebar nav ul > li:first-child > span, .sidebar nav ul > li:first-child > strong { + margin-top: 0 +} + +.sidebar-nav::-webkit-scrollbar, .sidebar nav::-webkit-scrollbar { + width: 0 +} + +@supports (width: env(safe-area-inset)) { + @media only screen and (orientation: landscape) { + .sidebar-nav, .sidebar nav { + margin-left: calc(env(safe-area-inset-left) / 2) + } + } +} + +.sidebar-nav li > a:before, .sidebar-nav li > strong:before { + display: inline-block +} + +.sidebar-nav li > a { + background-repeat: var(--sidebar-nav-pagelink-background-repeat); + background-size: var(--sidebar-nav-pagelink-background-size) +} + +.sidebar-nav li > a[href^="#/"]:not([href*="?id="]) { + transition: var(--sidebar-nav-pagelink-transition) +} + +.sidebar-nav li > a[href^="#/"]:not([href*="?id="]), .sidebar-nav li > a[href^="#/"]:not([href*="?id="]) ~ ul a { + padding: var(--sidebar-nav-pagelink-padding, var(--sidebar-nav-link-padding)) +} + +.sidebar-nav li > a[href^="#/"]:not([href*="?id="]):only-child { + background-image: var(--sidebar-nav-pagelink-background-image); + background-position: var(--sidebar-nav-pagelink-background-position) +} + +.sidebar-nav li > a[href^="#/"]:not([href*="?id="]):not(:only-child) { + background-image: var(--sidebar-nav-pagelink-background-image--loaded, var(--sidebar-nav-pagelink-background-image)); + background-position: var(--sidebar-nav-pagelink-background-position--loaded, var(--sidebar-nav-pagelink-background-image)) +} + +.sidebar-nav li.active > a, .sidebar-nav li.collapse > a { + border-width: var(--sidebar-nav-link-border-width--active, var(--sidebar-nav-link-border-width)); + border-style: var(--sidebar-nav-link-border-style--active, var(--sidebar-nav-link-border-style)); + border-color: var(--sidebar-nav-link-border-color--active, var(--sidebar-nav-link-border-color)); + background-color: var(--sidebar-nav-link-background-color--active, var(--sidebar-nav-link-background-color)); + background-image: var(--sidebar-nav-link-background-image--active, var(--sidebar-nav-link-background-image)); + background-position: var(--sidebar-nav-link-background-position--active, var(--sidebar-nav-link-background-position)); + background-size: var(--sidebar-nav-link-background-size--active, var(--sidebar-nav-link-background-size)); + color: var(--sidebar-nav-link-color--active, var(--sidebar-nav-link-color)); + font-weight: var(--sidebar-nav-link-font-weight--active, var(--sidebar-nav-link-font-weight)); + -webkit-text-decoration: var(--sidebar-nav-link-text-decoration--active, var(--sidebar-nav-link-text-decoration)); + text-decoration: var(--sidebar-nav-link-text-decoration--active, var(--sidebar-nav-link-text-decoration)); + -webkit-text-decoration-color: var(--sidebar-nav-link-text-decoration-color); + text-decoration-color: var(--sidebar-nav-link-text-decoration-color) +} + +.sidebar-nav li.active > a[href^="#/"]:not([href*="?id="]):not(:only-child) { + background-image: var(--sidebar-nav-pagelink-background-image--active, var(--sidebar-nav-pagelink-background-image--loaded, var(--sidebar-nav-pagelink-background-image))); + background-position: var(--sidebar-nav-pagelink-background-position--active, var(--sidebar-nav-pagelink-background-position--loaded, var(--sidebar-nav-pagelink-background-image))) +} + +.sidebar-nav li.collapse > a[href^="#/"]:not([href*="?id="]):not(:only-child) { + background-image: var(--sidebar-nav-pagelink-background-image--collapse, var(--sidebar-nav-pagelink-background-image--loaded, var(--sidebar-nav-pagelink-background-image))); + background-position: var(--sidebar-nav-pagelink-background-position--collapse, var(--sidebar-nav-pagelink-background-position--loaded, var(--sidebar-nav-pagelink-background-image))) +} + +.sidebar-nav li.collapse .app-sub-sidebar { + display: none +} + +.sidebar-nav > ul > li > a:before { + content: var(--sidebar-nav-link-before-content-l1, var(--sidebar-nav-link-before-content)); + margin: var(--sidebar-nav-link-before-margin-l1, var(--sidebar-nav-link-before-margin)); + color: var(--sidebar-nav-link-before-color-l1, var(--sidebar-nav-link-before-color)) +} + +.sidebar-nav > ul > li.active > a:before { + content: var(--sidebar-nav-link-before-content-l1--active, var(--sidebar-nav-link-before-content--active, var(--sidebar-nav-link-before-content-l1, var(--sidebar-nav-link-before-content)))); + color: var(--sidebar-nav-link-before-color-l1--active, var(--sidebar-nav-link-before-color--active, var(--sidebar-nav-link-before-color-l1, var(--sidebar-nav-link-before-color)))) +} + +.sidebar-nav > ul > li > ul > li > a:before { + content: var(--sidebar-nav-link-before-content-l2, var(--sidebar-nav-link-before-content)); + margin: var(--sidebar-nav-link-before-margin-l2, var(--sidebar-nav-link-before-margin)); + color: var(--sidebar-nav-link-before-color-l2, var(--sidebar-nav-link-before-color)) +} + +.sidebar-nav > ul > li > ul > li.active > a:before { + content: var(--sidebar-nav-link-before-content-l2--active, var(--sidebar-nav-link-before-content--active, var(--sidebar-nav-link-before-content-l2, var(--sidebar-nav-link-before-content)))); + color: var(--sidebar-nav-link-before-color-l2--active, var(--sidebar-nav-link-before-color--active, var(--sidebar-nav-link-before-color-l2, var(--sidebar-nav-link-before-color)))) +} + +.sidebar-nav > ul > li > ul > li > ul > li > a:before { + content: var(--sidebar-nav-link-before-content-l3, var(--sidebar-nav-link-before-content)); + margin: var(--sidebar-nav-link-before-margin-l3, var(--sidebar-nav-link-before-margin)); + color: var(--sidebar-nav-link-before-color-l3, var(--sidebar-nav-link-before-color)) +} + +.sidebar-nav > ul > li > ul > li > ul > li.active > a:before { + content: var(--sidebar-nav-link-before-content-l3--active, var(--sidebar-nav-link-before-content--active, var(--sidebar-nav-link-before-content-l3, var(--sidebar-nav-link-before-content)))); + color: var(--sidebar-nav-link-before-color-l3--active, var(--sidebar-nav-link-before-color--active, var(--sidebar-nav-link-before-color-l3, var(--sidebar-nav-link-before-color)))) +} + +.sidebar-nav > ul > li > ul > li > ul > li > ul > li > a:before { + content: var(--sidebar-nav-link-before-content-l4, var(--sidebar-nav-link-before-content)); + margin: var(--sidebar-nav-link-before-margin-l4, var(--sidebar-nav-link-before-margin)); + color: var(--sidebar-nav-link-before-color-l4, var(--sidebar-nav-link-before-color)) +} + +.sidebar-nav > ul > li > ul > li > ul > li > ul > li.active > a:before { + content: var(--sidebar-nav-link-before-content-l4--active, var(--sidebar-nav-link-before-content--active, var(--sidebar-nav-link-before-content-l4, var(--sidebar-nav-link-before-content)))); + color: var(--sidebar-nav-link-before-color-l4--active, var(--sidebar-nav-link-before-color--active, var(--sidebar-nav-link-before-color-l4, var(--sidebar-nav-link-before-color)))) +} + +.sidebar-nav > :last-child { + margin-bottom: 2rem +} + +.sidebar-toggle, .sidebar-toggle-button { + width: var(--sidebar-toggle-width); + outline: none +} + +.sidebar-toggle { + position: fixed; + z-index: 11; + top: 0; + bottom: 0; + left: 0; + max-width: 40px; + margin: 0; + padding: 0; + border: 0; + background: transparent; + -webkit-appearance: none; + -moz-appearance: none; + appearance: none; + cursor: pointer +} + +.sidebar-toggle .sidebar-toggle-button { + position: absolute; + top: var(--sidebar-toggle-offset-top); + left: var(--sidebar-toggle-offset-left); + height: var(--sidebar-toggle-height); + border-radius: var(--sidebar-toggle-border-radius); + border-width: var(--sidebar-toggle-border-width); + border-style: var(--sidebar-toggle-border-style); + border-color: var(--sidebar-toggle-border-color); + background: var(--sidebar-toggle-background, transparent); + color: var(--sidebar-toggle-icon-color) +} + +.sidebar-toggle span { + position: absolute; + top: calc(50% - (var(--sidebar-toggle-icon-stroke-width) / 2)); + left: calc(50% - (var(--sidebar-toggle-icon-width) / 2)); + height: var(--sidebar-toggle-icon-stroke-width); + width: var(--sidebar-toggle-icon-width); + background-color: currentColor +} + +.sidebar-toggle span:nth-child(1) { + margin-top: calc(0px - (var(--sidebar-toggle-icon-height) / 2)) +} + +.sidebar-toggle span:nth-child(3) { + margin-top: calc((var(--sidebar-toggle-icon-height) / 2)) +} + +@media (min-width: 48em) { + .sidebar-toggle { + position: absolute; + overflow: visible; + top: var(--sidebar-toggle-offset-top); + bottom: auto; + left: 0; + height: var(--sidebar-toggle-height); + transform: translateX(var(--sidebar-width)) + } + + .sidebar-toggle .sidebar-toggle-button { + top: 0 + } +} + +@media print { + .sidebar-toggle { + display: none + } +} + +@media (max-width: 47.99em) { + body.close .sidebar, body.close .sidebar-toggle, body.close main > .content { + transform: translateX(var(--sidebar-width)) + } +} + +@media (min-width: 48em) { + body.close main > .content { + transform: translateX(0) + } +} + +@media (max-width: 47.99em) { + body.close nav.app-nav, body.close .github-corner { + display: none + } +} + +@media (min-width: 48em) { + body.close .sidebar, body.close .sidebar-toggle { + transform: translateX(0) + } +} + +@media (min-width: 48em) { + body.close nav.app-nav { + margin-left: 0 + } +} + +@media (max-width: 47.99em) { + body.close .sidebar-toggle { + width: 100%; + max-width: none + } + + body.close .sidebar-toggle span { + margin-top: 0 + } + + body.close .sidebar-toggle span:nth-child(1) { + transform: rotate(45deg) + } + + body.close .sidebar-toggle span:nth-child(2) { + display: none + } + + body.close .sidebar-toggle span:nth-child(3) { + transform: rotate(-45deg) + } +} + +@media (min-width: 48em) { + body.close main > .content { + margin-left: 0 + } +} + +@media (min-width: 48em) { + body.sticky .sidebar, body.sticky .sidebar-toggle { + position: fixed + } +} + +body .docsify-copy-code-button, body .docsify-copy-code-button:after { + border-radius: var(--border-radius-m, 0); + border-top-left-radius: 0; + border-bottom-right-radius: 0; + background: var(--copycode-background); + color: var(--copycode-color) +} + +body .docsify-copy-code-button span { + border-radius: var(--border-radius-s, 0) +} + +body .docsify-pagination-container { + border-top: var(--pagination-border-top); + color: var(--pagination-color) +} + +body .pagination-item-label { + font-size: var(--pagination-label-font-size) +} + +body .pagination-item-label svg { + color: var(--pagination-label-color); + height: var(--pagination-chevron-height); + stroke: var(--pagination-chevron-stroke); + stroke-linecap: var(--pagination-chevron-stroke-linecap); + stroke-linejoin: var(--pagination-chevron-stroke-linecap); + stroke-width: var(--pagination-chevron-stroke-width) +} + +body .pagination-item-title { + color: var(--pagination-title-color); + font-size: var(--pagination-title-font-size) +} + +body .app-name.hide { + display: block +} + +body .sidebar { + padding: var(--sidebar-padding) +} + +.sidebar .search { + margin: 0; + padding: 0; + border: 0 +} + +.sidebar .search input { + padding: 0; + line-height: 1; + font-size: inherit +} + +.sidebar .search .clear-button { + width: auto +} + +.sidebar .search .clear-button svg { + transform: scale(1) +} + +.sidebar .search .matching-post { + border: none +} + +.sidebar .search p { + font-size: inherit +} + +.sidebar .search { + order: var(--search-flex-order); + margin: var(--search-margin); + padding: var(--search-padding); + background: var(--search-background) +} + +.sidebar .search a { + color: inherit +} + +.sidebar .search h2 { + margin: var(--search-result-heading-margin); + font-size: var(--search-result-heading-font-size); + font-weight: var(--search-result-heading-font-weight); + color: var(--search-result-heading-color) +} + +.sidebar .search .input-wrap { + margin: var(--search-input-margin); + background-color: var(--search-input-background-color); + border-width: var(--search-input-border-width, 0); + border-style: solid; + border-color: var(--search-input-border-color); + border-radius: var(--search-input-border-radius) +} + +.sidebar .search input[type="search"] { + min-width: 0; + padding: var(--search-input-padding); + border: none; + background-color: transparent; + background-image: var(--search-input-background-image); + background-position: var(--search-input-background-position); + background-repeat: var(--search-input-background-repeat); + background-size: var(--search-input-background-size); + font-size: var(--search-input-font-size); + color: var(--search-input-color); + transition: var(--search-input-transition) +} + +.sidebar .search input[type="search"]::-ms-clear { + display: none +} + +.sidebar .search input[type="search"]::-webkit-input-placeholder { + color: var(--search-input-placeholder-color, gray) +} + +.sidebar .search input[type="search"]::-moz-placeholder { + color: var(--search-input-placeholder-color, gray) +} + +.sidebar .search input[type="search"]:-ms-input-placeholder { + color: var(--search-input-placeholder-color, gray) +} + +.sidebar .search input[type="search"]::placeholder { + color: var(--search-input-placeholder-color, gray) +} + +.sidebar .search input[type="search"]::-webkit-input-placeholder { + line-height: normal +} + +.sidebar .search input[type="search"]:focus { + background-color: var(--search-input-background-color--focus, var(--search-input-background-color)); + background-image: var(--search-input-background-image--focus, var(--search-input-background-image)); + background-position: var(--search-input-background-position--focus, var(--search-input-background-position)); + background-size: var(--search-input-background-size--focus, var(--search-input-background-size)) +} + +@supports (width: env(safe-area-inset)) { + @media only screen and (orientation: landscape) { + .sidebar .search input[type="search"] { + margin-left: calc(env(safe-area-inset-left) / 2) + } + } +} + +.sidebar .search p { + overflow: hidden; + text-overflow: ellipsis; + -webkit-box-orient: vertical; + -webkit-line-clamp: 2 +} + +.sidebar .search p:empty { + text-align: center +} + +.sidebar .search .clear-button { + margin: 0 15px 0 0; + padding: 0; + border: none; + line-height: 1; + background: transparent; + cursor: pointer +} + +.sidebar .search .clear-button svg circle { + fill: var(--search-clear-icon-color1, gray) +} + +.sidebar .search .clear-button svg path { + stroke: var(--search-clear-icon-color2, #fff) +} + +.sidebar .search.show ~ *:not(h1) { + display: none +} + +.sidebar .search .results-panel { + display: none; + color: var(--search-result-item-color); + font-size: var(--search-result-item-font-size); + font-weight: var(--search-result-item-font-weight) +} + +.sidebar .search .results-panel.show { + display: block +} + +.sidebar .search .matching-post { + margin: var(--search-result-item-margin); + padding: var(--search-result-item-padding) +} + +.sidebar .search .matching-post, .sidebar .search .matching-post:last-child { + border-width: var(--search-result-item-border-width, 0) !important; + border-style: var(--search-result-item-border-style); + border-color: var(--search-result-item-border-color) +} + +.sidebar .search .matching-post p { + margin: 0 +} + +.sidebar .search .search-keyword { + margin: var(--search-result-keyword-margin); + padding: var(--search-result-keyword-padding); + border-radius: var(--search-result-keyword-border-radius); + background-color: var(--search-result-keyword-background); + color: var(--search-result-keyword-color, currentColor); + font-style: normal; + font-weight: var(--search-result-keyword-font-weight) +} + +.medium-zoom-overlay, .medium-zoom-image--open, .medium-zoom-image--opened { + z-index: 2147483646 !important +} + +.medium-zoom-overlay { + background: var(--zoomimage-overlay-background) !important +} + +:root { + --mono-hue: 113; + --mono-saturation: 0%; + --mono-shade3: hsl(var(--mono-hue), var(--mono-saturation), 20%); + --mono-shade2: hsl(var(--mono-hue), var(--mono-saturation), 30%); + --mono-shade1: hsl(var(--mono-hue), var(--mono-saturation), 40%); + --mono-base: hsl(var(--mono-hue), var(--mono-saturation), 50%); + --mono-tint1: hsl(var(--mono-hue), var(--mono-saturation), 70%); + --mono-tint2: hsl(var(--mono-hue), var(--mono-saturation), 89%); + --mono-tint3: hsl(var(--mono-hue), var(--mono-saturation), 97%); + --theme-hue: 204; + --theme-saturation: 90%; + --theme-lightness: 45%; + --theme-color: hsl(var(--theme-hue), var(--theme-saturation), var(--theme-lightness)); + --modular-scale: 1.333; + --modular-scale--2: calc(var(--modular-scale--1) / var(--modular-scale)); + --modular-scale--1: calc(var(--modular-scale-1) / var(--modular-scale)); + --modular-scale-1: 1rem; + --modular-scale-2: calc(var(--modular-scale-1) * var(--modular-scale)); + --modular-scale-3: calc(var(--modular-scale-2) * var(--modular-scale)); + --modular-scale-4: calc(var(--modular-scale-3) * var(--modular-scale)); + --modular-scale-5: calc(var(--modular-scale-4) * var(--modular-scale)); + --font-size-xxxl: var(--modular-scale-5); + --font-size-xxl: var(--modular-scale-4); + --font-size-xl: var(--modular-scale-3); + --font-size-l: var(--modular-scale-2); + --font-size-m: var(--modular-scale-1); + --font-size-s: var(--modular-scale--1); + --font-size-xs: var(--modular-scale--2); + --duration-slow: 1s; + --duration-medium: 0.5s; + --duration-fast: 0.25s; + --spinner-size: 60px; + --spinner-track-width: 4px; + --spinner-track-color: rgba(0, 0, 0, 0.15); + --spinner-transition-duration: var(--duration-medium) +} + +:root { + --base-background-color: #fff; + --base-color: var(--mono-shade2); + --base-font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", "Segoe UI Symbol"; + --base-font-size: 16px; + --base-font-weight: normal; + --base-line-height: 1.7; + --emoji-size: calc(var(--base-line-height) * 1em); + --hr-border: 1px solid var(--mono-tint2); + --mark-background: #ffecb3; + --pre-font-family: var(--code-font-family); + --pre-font-size: var(--code-font-size); + --pre-font-weight: normal; + --selection-color: #b4d5fe; + --small-font-size: var(--font-size-s); + --strong-color: var(--heading-color); + --strong-font-weight: 600; + --subsup-font-size: var(--font-size-s) +} + +:root { + --content-max-width: 55em; + --blockquote-background: var(--mono-tint3); + --blockquote-border-style: solid; + --blockquote-border-radius: var(--border-radius-m); + --blockquote-em-font-weight: normal; + --blockquote-font-weight: normal; + --code-font-family: Inconsolata, Consolas, Menlo, Monaco, "Andale Mono WT", "Andale Mono", "Lucida Console", "DejaVu Sans Mono", "Bitstream Vera Sans Mono", "Courier New", Courier, monospace; + --code-font-size: calc(var(--font-size-m) * 0.95); + --code-font-weight: normal; + --code-tab-size: 4; + --code-block-border-radius: var(--border-radius-m); + --code-block-line-height: var(--base-line-height); + --code-block-margin: 1em 0; + --code-block-padding: 1.75em 1.5em 1.5em 1.5em; + --code-inline-background: var(--code-theme-background); + --code-inline-border-radius: var(--border-radius-s); + --code-inline-color: var(--code-theme-text); + --code-inline-margin: 0 0.15em; + --code-inline-padding: 0.125em 0.4em; + --code-theme-background: var(--mono-tint3); + --heading-color: var(--mono-shade3); + --heading-margin: 2.5rem 0 0; + --heading-h1-border-style: solid; + --heading-h1-font-size: var(--font-size-xxl); + --heading-h2-border-style: solid; + --heading-h2-font-size: var(--font-size-xl); + --heading-h3-border-style: solid; + --heading-h3-font-size: var(--font-size-l); + --heading-h4-border-style: solid; + --heading-h4-font-size: var(--font-size-m); + --heading-h5-border-style: solid; + --heading-h5-font-size: var(--font-size-s); + --heading-h6-border-style: solid; + --heading-h6-font-size: var(--font-size-xs); + --kbd-background: var(--mono-tint3); + --kbd-border-radius: var(--border-radius-m); + --kbd-margin: 0 0.3em; + --kbd-min-width: 2.5em; + --kbd-padding: 0.65em 0.5em; + --link-text-decoration: underline; + --notice-background: var(--mono-tint3); + --notice-border-radius: var(--border-radius-m); + --notice-border-style: solid; + --notice-font-weight: normal; + --notice-padding: 1em 1.5em; + --notice-before-font-weight: normal; + --table-cell-padding: 0.75em 0.5em; + --table-head-border-color: var(--table-cell-border-color); + --table-head-font-weight: var(--strong-font-weight); + --table-row-odd-background: var(--mono-tint3) +} + +:root { + --cover-margin: 0 auto; + --cover-max-width: 40em; + --cover-text-align: center; + --cover-background-color: var(--base-background-color); + --cover-background-mask-color: var(--base-background-color); + --cover-background-mask-opacity: 0.8; + --cover-background-position: center center; + --cover-background-repeat: no-repeat; + --cover-background-size: cover; + --cover-blockquote-font-size: var(--font-size-l); + --cover-border-color: var(--theme-color); + --cover-button-border: 1px solid var(--theme-color); + --cover-button-border-radius: var(--border-radius-m); + --cover-button-color: var(--theme-color); + --cover-button-padding: 0.5em 2rem; + --cover-button-text-decoration: none; + --cover-button-transition: all var(--duration-fast) ease-in-out; + --cover-button-primary-background: var(--theme-color); + --cover-button-primary-border: 1px solid var(--theme-color); + --cover-button-primary-color: #fff; + --cover-heading-color: var(--theme-color); + --cover-heading-font-size: var(--font-size-xxl); + --cover-heading-font-weight: normal; + --cover-link-text-decoration: underline +} + +:root { + --navbar-root-border-style: solid; + --navbar-root-margin: 0 0 0 1.5em; + --navbar-root-transition: all var(--duration-fast); + --navbar-menu-background: var(--base-background-color); + --navbar-menu-border-radius: var(--border-radius-m); + --navbar-menu-box-shadow: rgba(45, 45, 45, 0.05) 0px 0px 1px, rgba(49, 49, 49, 0.05) 0px 1px 2px, rgba(42, 42, 42, 0.05) 0px 2px 4px, rgba(32, 32, 32, 0.05) 0px 4px 8px, rgba(49, 49, 49, 0.05) 0px 8px 16px, rgba(35, 35, 35, 0.05) 0px 16px 32px; + --navbar-menu-padding: 0.5em; + --navbar-menu-transition: all var(--duration-fast); + --navbar-menu-link-border-style: solid; + --navbar-menu-link-margin: 0.75em 0.5em; + --navbar-menu-link-padding: 0.2em 0 +} + +:root { + --copycode-background: #808080; + --copycode-color: #fff +} + +:root { + --docsifytabs-border-color: var(--mono-tint2); + --docsifytabs-border-radius-px: var(--border-radius-s); + --docsifytabs-tab-background: var(--mono-tint3); + --docsifytabs-tab-color: var(--mono-tint1) +} + +:root { + --pagination-border-top: 1px solid var(--mono-tint2); + --pagination-chevron-height: 0.8em; + --pagination-chevron-stroke: currentColor; + --pagination-chevron-stroke-linecap: round; + --pagination-chevron-stroke-width: 1px; + --pagination-label-font-size: var(--font-size-s); + --pagination-title-font-size: var(--font-size-l) +} + +:root { + --search-margin: 1.5rem 0 0; + --search-input-background-repeat: no-repeat; + --search-input-border-color: var(--mono-tint1); + --search-input-border-width: 1px; + --search-input-padding: 0.5em; + --search-flex-order: 1; + --search-result-heading-color: var(--heading-color); + --search-result-heading-font-size: var(--base-font-size); + --search-result-heading-font-weight: normal; + --search-result-heading-margin: 0 0 0.25em; + --search-result-item-border-color: var(--mono-tint2); + --search-result-item-border-style: solid; + --search-result-item-border-width: 0 0 1px 0; + --search-result-item-font-weight: normal; + --search-result-item-padding: 1em 0; + --search-result-keyword-background: var(--mark-background); + --search-result-keyword-border-radius: var(--border-radius-s); + --search-result-keyword-color: var(--mark-color); + --search-result-keyword-font-weight: normal; + --search-result-keyword-margin: 0 0.1em; + --search-result-keyword-padding: 0.2em 0 +} + +:root { + --zoomimage-overlay-background: rgba(0, 0, 0, 0.875) +} + +:root { + --sidebar-background: var(--base-background-color); + --sidebar-border-width: 0; + --sidebar-padding: 0 25px; + --sidebar-transition-duration: var(--duration-fast); + --sidebar-width: 17rem; + --sidebar-name-font-size: var(--font-size-l); + --sidebar-name-font-weight: normal; + --sidebar-name-margin: 1.5rem 0 0; + --sidebar-name-text-align: center; + --sidebar-nav-strong-border-color: var(--sidebar-border-color); + --sidebar-nav-strong-color: var(--heading-color); + --sidebar-nav-strong-font-weight: var(--strong-font-weight); + --sidebar-nav-strong-margin: 1.5em 0 0.5em; + --sidebar-nav-strong-padding: 0.25em 0; + --sidebar-nav-indent: 1em; + --sidebar-nav-margin: 1.5rem 0 0; + --sidebar-nav-link-border-style: solid; + --sidebar-nav-link-border-width: 0; + --sidebar-nav-link-color: var(--base-color); + --sidebar-nav-link-font-weight: normal; + --sidebar-nav-link-padding: 0.25em 0; + --sidebar-nav-link-text-decoration--active: underline; + --sidebar-nav-link-text-decoration--hover: underline; + --sidebar-nav-link-before-margin: 0 0.35em 0 0; + --sidebar-nav-pagelink-background-repeat: no-repeat; + --sidebar-nav-pagelink-transition: var(--sidebar-nav-link-transition); + --sidebar-toggle-border-radius: var(--border-radius-s); + --sidebar-toggle-border-style: solid; + --sidebar-toggle-border-width: 0; + --sidebar-toggle-height: 36px; + --sidebar-toggle-icon-color: var(--base-color); + --sidebar-toggle-icon-height: 10px; + --sidebar-toggle-icon-stroke-width: 1px; + --sidebar-toggle-icon-width: 16px; + --sidebar-toggle-offset-left: 0; + --sidebar-toggle-offset-top: calc(35px - (var(--sidebar-toggle-height) / 2)); + --sidebar-toggle-width: 44px +} + +:root { + --code-theme-background: #f3f3f3; + --code-theme-comment: #6e8090; + --code-theme-function: #dd4a68; + --code-theme-keyword: #07a; + --code-theme-operator: #a67f59; + --code-theme-punctuation: #999; + --code-theme-selection: #b3d4fc; + --code-theme-selector: #690; + --code-theme-tag: #905; + --code-theme-text: #333; + --code-theme-variable: #e90 +} + +:root { + --border-radius-s: 2px; + --border-radius-m: 4px; + --border-radius-l: 8px; + --strong-font-weight: 600; + --blockquote-border-color: var(--theme-color); + --blockquote-border-radius: 0 var(--border-radius-m) var(--border-radius-m) 0; + --blockquote-border-width: 0 0 0 4px; + --code-inline-background: var(--mono-tint2); + --code-theme-background: var(--mono-tint3); + --heading-font-weight: var(--strong-font-weight); + --heading-h1-font-weight: 400; + --heading-h2-font-weight: 400; + --heading-h2-border-color: var(--mono-tint2); + --heading-h2-border-width: 0 0 1px 0; + --heading-h2-margin: 2.5rem 0 1.5rem; + --heading-h2-padding: 0 0 1rem 0; + --kbd-border: 1px solid var(--mono-tint2); + --notice-border-radius: 0 var(--border-radius-m) var(--border-radius-m) 0; + --notice-border-width: 0 0 0 4px; + --notice-padding: 1em 1.5em 1em 3em; + --notice-before-border-radius: 100%; + --notice-before-font-weight: bold; + --notice-before-height: 1.5em; + --notice-before-left: 0.75em; + --notice-before-line-height: 1.5; + --notice-before-margin: 0 0.25em 0 0; + --notice-before-position: absolute; + --notice-before-width: var(--notice-before-height); + --notice-important-background: hsl(340, 60%, 96%); + --notice-important-border-color: hsl(340, 90%, 45%); + --notice-important-before-background: var(--notice-important-border-color) url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 16 16' fill='%23fff'%3E%3Cpath d='M10 14C10 15.1 9.1 16 8 16 6.9 16 6 15.1 6 14 6 12.9 6.9 12 8 12 9.1 12 10 12.9 10 14Z'/%3E%3Cpath d='M10 1.6C10 1.2 9.8 0.9 9.6 0.7 9.2 0.3 8.6 0 8 0 7.4 0 6.8 0.2 6.5 0.6 6.2 0.9 6 1.2 6 1.6 6 1.7 6 1.8 6 1.9L6.8 9.6C6.9 9.9 7 10.1 7.2 10.2 7.4 10.4 7.7 10.5 8 10.5 8.3 10.5 8.6 10.4 8.8 10.3 9 10.1 9.1 9.9 9.2 9.6L10 1.9C10 1.8 10 1.7 10 1.6Z'/%3E%3C/svg%3E") center / 0.875em no-repeat; + --notice-important-before-color: #fff; + --notice-important-before-content: ""; + --notice-tip-background: hsl(204, 60%, 96%); + --notice-tip-border-color: hsl(204, 90%, 45%); + --notice-tip-before-background: var(--notice-tip-border-color) url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 16 16' fill='%23fff'%3E%3Cpath d='M9.1 0C10.2 0 10.7 0.7 10.7 1.6 10.7 2.6 9.8 3.6 8.6 3.6 7.6 3.6 7 3 7 2 7 1.1 7.7 0 9.1 0Z'/%3E%3Cpath d='M5.8 16C5 16 4.4 15.5 5 13.2L5.9 9.1C6.1 8.5 6.1 8.2 5.9 8.2 5.7 8.2 4.6 8.6 3.9 9.1L3.5 8.4C5.6 6.6 7.9 5.6 8.9 5.6 9.8 5.6 9.9 6.6 9.5 8.2L8.4 12.5C8.2 13.2 8.3 13.5 8.5 13.5 8.7 13.5 9.6 13.2 10.4 12.5L10.9 13.2C8.9 15.2 6.7 16 5.8 16Z'/%3E%3C/svg%3E") center / 0.875em no-repeat; + --notice-tip-before-color: #fff; + --notice-tip-before-content: ""; + --table-cell-border-color: var(--mono-tint2); + --table-cell-border-width: 1px 0; + --cover-background-color: hsl(var(--theme-hue), 25%, 60%); + --cover-background-image: radial-gradient(ellipse at center 115%, rgba(255, 255, 255, 0.9), transparent); + --cover-blockquote-color: var(--strong-color); + --cover-heading-color: #fff; + --cover-heading-font-size-max: 56; + --cover-heading-font-size-min: 34; + --cover-heading-font-weight: 200; + --navbar-root-color--active: var(--theme-color); + --navbar-menu-border-radius: var(--border-radius-m); + --navbar-menu-root-background: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='9.6' height='6' viewBox='0 0 9.6 6'%3E%3Cpath d='M1.5 1.5l3.3 3 3.3-3' stroke-width='1.5' stroke='rgb%28179, 179, 179%29' fill='none' stroke-linecap='square' stroke-linejoin='miter' vector-effect='non-scaling-stroke'/%3E%3C/svg%3E") right no-repeat; + --navbar-menu-root-padding: 0 18px 0 0; + --search-input-background-color: #fff; + --search-input-background-image: url("data:image/svg+xml,%3Csvg height='20px' width='20px' viewBox='0 0 24 24' fill='none' stroke='rgba(0, 0, 0, 0.3)' stroke-width='1.5' stroke-linecap='round' stroke-linejoin='round' preserveAspectRatio='xMidYMid meet' xmlns='http://www.w3.org/2000/svg'%3E%3Ccircle cx='10.5' cy='10.5' r='7.5' vector-effect='non-scaling-stroke'%3E%3C/circle%3E%3Cline x1='21' y1='21' x2='15.8' y2='15.8' vector-effect='non-scaling-stroke'%3E%3C/line%3E%3C/svg%3E"); + --search-input-background-position: 21px center; + --search-input-border-color: var(--sidebar-border-color); + --search-input-border-width: 1px 0; + --search-input-margin: 0 -25px; + --search-input-padding: 0.65em 1em 0.65em 50px; + --search-input-placeholder-color: rgba(0, 0, 0, 0.4); + --search-clear-icon-color1: rgba(0, 0, 0, 0.3); + --search-result-heading-font-weight: var(--strong-font-weight); + --search-result-item-border-color: var(--sidebar-border-color); + --search-result-keyword-border-radius: var(--border-radius-s); + --sidebar-background: var(--mono-tint3); + --sidebar-border-color: var(--mono-tint2); + --sidebar-border-width: 0 1px 0 0; + --sidebar-name-color: var(--theme-color); + --sidebar-name-font-weight: 300; + --sidebar-nav-strong-border-width: 0 0 1px 0; + --sidebar-nav-strong-font-size: smaller; + --sidebar-nav-strong-margin: 2em -25px 0.75em 0; + --sidebar-nav-strong-padding: 0.25em 0 0.75em 0; + --sidebar-nav-strong-text-transform: uppercase; + --sidebar-nav-link-border-color: transparent; + --sidebar-nav-link-border-color--active: var(--theme-color); + --sidebar-nav-link-border-width: 0 4px 0 0; + --sidebar-nav-link-color--active: var(--theme-color); + --sidebar-nav-link-margin: 0 -25px 0 0; + --sidebar-nav-link-text-decoration: none; + --sidebar-nav-link-text-decoration--active: none; + --sidebar-nav-link-text-decoration--hover: underline; + --sidebar-nav-link-before-content-l3: '-'; + --sidebar-nav-pagelink-background-image: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='7' height='11.2' viewBox='0 0 7 11.2'%3E%3Cpath d='M1.5 1.5l4 4.1 -4 4.1' stroke-width='1.5' stroke='rgb%28179, 179, 179%29' fill='none' stroke-linecap='square' stroke-linejoin='miter' vector-effect='non-scaling-stroke'/%3E%3C/svg%3E"); + --sidebar-nav-pagelink-background-image--active: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='11.2' height='7' viewBox='0 0 11.2 7'%3E%3Cpath d='M1.5 1.5l4.1 4 4.1-4' stroke-width='1.5' stroke='rgb%2811, 135, 218%29' fill='none' stroke-linecap='square' stroke-linejoin='miter' vector-effect='non-scaling-stroke'/%3E%3C/svg%3E"); + --sidebar-nav-pagelink-background-image--collapse: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='7' height='11.2' viewBox='0 0 7 11.2'%3E%3Cpath d='M1.5 1.5l4 4.1 -4 4.1' stroke-width='1.5' stroke='rgb%2811, 135, 218%29' fill='none' stroke-linecap='square' stroke-linejoin='miter' vector-effect='non-scaling-stroke'/%3E%3C/svg%3E"); + --sidebar-nav-pagelink-background-image--loaded: url("data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='11.2' height='7' viewBox='0 0 11.2 7'%3E%3Cpath d='M1.5 1.5l4.1 4 4.1-4' stroke-width='1.5' stroke='rgb%2811, 135, 218%29' fill='none' stroke-linecap='square' stroke-linejoin='miter' vector-effect='non-scaling-stroke'/%3E%3C/svg%3E"); + --sidebar-nav-pagelink-background-position: 3px center; + --sidebar-nav-pagelink-background-position--active: left center; + --sidebar-nav-pagelink-background-position--collapse: var(--sidebar-nav-pagelink-background-position); + --sidebar-nav-pagelink-background-position--loaded: var(--sidebar-nav-pagelink-background-position--active); + --sidebar-nav-pagelink-padding: 0.25em 0 0.25em 20px; + --sidebar-nav-pagelink-transition: none; + --sidebar-toggle-background: var(--sidebar-border-color); + --sidebar-toggle-border-radius: 0 var(--border-radius-s) var(--border-radius-s) 0; + --sidebar-toggle-width: 32px +} + /*# sourceMappingURL=theme-simple.css.map */ \ No newline at end of file diff --git a/docs/big-data/Hadoop-MapReduce.md b/docs/big-data/Hadoop-MapReduce.md deleted file mode 100644 index 34c8104756..0000000000 --- a/docs/big-data/Hadoop-MapReduce.md +++ /dev/null @@ -1,21 +0,0 @@ - https://hadoop.apache.org/docs/r1.0.4/cn/index.html - - - -## MapReduce 概述 - -### 1.1 MapReduce 定义 - -MapReduce是一个**分布式运算程序的编程框架**，是用户开发“基于Hadoop的数据分析应用”的核心框架。 - -MapReduce核心功能是将**用户编写的业务逻辑代码**和**自带默认组件**整合成一个完整的**分布式运算程序**，并发运行在一个Hadoop集群上。 - - - -### 1.2 MapReduce 优缺点 - -#### 1.2.1 优点 - -1．MapReduce 易于编程：它简单的实现一些接口，就可以完成一个分布式程序，这个分布式程序可 以分布到大量廉价的PC机器上运行。也就是说你写一个分布式程序，跟写 一个简单的串行程序是一模一样的。就是因为这个特点使得MapReduce编 程变得非常流行。 - -2．良好的扩展性 当你的计算资源不能得到满足的时候，你可以通过简单的增加机器来扩展 它的计算能力。 MapReduce优缺点 1.2.1 优点 3．高容错性 MapReduce设计的初衷就是使程序能够部署在廉价的PC机器上，这就要求 它具有很高的容错性。比如其中一台机器挂了，它可以把上面的计算任务 转移到另外一个节点上运行，不至于这个任务运行失败，而且这个过程不 需要人工参与，而完全是由Hadoop内部完成的。 4．适合PB级以上海量数据的离线处理 可以实现上千台服务器集群并发工作，提供数据处理能力。 \ No newline at end of file diff --git a/docs/collection/.DS_Store b/docs/collection/.DS_Store new file mode 100644 index 0000000000..5fe2d83286 Binary files /dev/null and b/docs/collection/.DS_Store differ diff --git "a/docs/collection/23\344\270\255\350\256\276\350\256\241\346\250\241\345\274\217\351\200\232\344\277\227\350\247\243\351\207\212.md" "b/docs/collection/23\344\270\255\350\256\276\350\256\241\346\250\241\345\274\217\351\200\232\344\277\227\350\247\243\351\207\212.md" new file mode 100644 index 0000000000..e5f7149410 --- /dev/null +++ "b/docs/collection/23\344\270\255\350\256\276\350\256\241\346\250\241\345\274\217\351\200\232\344\277\227\350\247\243\351\207\212.md" @@ -0,0 +1,149 @@ +### 01 工厂方法 + +追 MM 少不了请吃饭了，麦当劳的鸡翅和肯德基的鸡翅都是 MM 爱吃的东西，虽然口味有所不同，但不管你带 MM 去麦当劳或肯德基，只管向服务员说「来四个鸡翅」就行了。麦当劳和肯德基就是生产鸡翅的 Factory 工厂模式：客户类和工厂类分开。 + +消费者任何时候需要某种产品，只需向工厂请求即可。消费者无须修改就可以接纳新产品。缺点是当产品修改时，工厂类也要做相应的修改。如：如何创建及如何向客户端提供。 + +### 02 建造者模式 + +MM 最爱听的就是「我爱你」这句话了，见到不同地方的 MM，要能够用她们的方言跟她说这句话哦，我有一个多种语言翻译机，上面每种语言都有一个按键，见到 MM 我只要按对应的键，它就能够用相应的语言说出「我爱你」这句话了，国外的 MM 也可以轻松搞掂，这就是我的「我爱你」builder。 + +建造模式：将产品的内部表象和产品的生成过程分割开来，从而使一个建造过程生成具有不同的内部表象的产品对象。建造模式使得产品内部表象可以独立的变化，客户不必知道产品内部组成的细节。建造模式可以强制实行一种分步骤进行的建造过程。 + +### 03 抽象工厂 + +请 MM 去麦当劳吃汉堡，不同的 MM 有不同的口味，要每个都记住是一件烦人的事情，我一般采用 Factory Method 模式，带着 MM 到服务员那儿，说「要一个汉堡」，具体要什么样的汉堡呢，让 MM 直接跟服务员说就行了。 + +工厂方法模式：核心工厂类不再负责所有产品的创建，而是将具体创建的工作交给子类去做，成为一个抽象工厂角色，仅负责给出具体工厂类必须实现的接口，而不接触哪一个产品类应当被实例化这种细节。 + +### 04 原型模式 + +跟 MM 用 QQ 聊天，一定要说些深情的话语了，我搜集了好多肉麻的情话，需要时只要 copy 出来放到 QQ 里面就行了，这就是我的情话 prototype 了。（100 块钱一份，你要不要） + +原始模型模式：通过给出一个原型对象来指明所要创建的对象的类型，然后用复制这个原型对象的方法创建出更多同类型的对象。原始模型模式允许动态的增加或减少产品类，产品类不需要非得有任何事先确定的等级结构，原始模型模式适用于任何的等级结构。缺点是每一个类都必须配备一个克隆方法。 + +### 05 单态模式 + +俺有 6 个漂亮的老婆，她们的老公都是我，我就是我们家里的老公 Sigleton，她们只要说道「老公」，都是指的同一个人，那就是我 (刚才做了个梦啦，哪有这么好的事) + +单例模式：单例模式确保某一个类只有一个实例，而且自行实例化并向整个系统提供这个实例单例模式。单例模式只应在有真正的 “单一实例” 的需求时才可使用。 + +### 06 适配器模式 + +在朋友聚会上碰到了一个美女 Sarah，从香港来的，可我不会说粤语，她不会说普通话，只好求助于我的朋友 kent 了，他作为我和 Sarah 之间的 Adapter，让我和 Sarah 可以相互交谈了 (也不知道他会不会耍我) + +适配器（变压器）模式：把一个类的接口变换成客户端所期待的另一种接口，从而使原本因接口原因不匹配而无法一起工作的两个类能够一起工作。适配类可以根据参数返还一个合适的实例给客户端。 + +### 07 桥梁模式 + +早上碰到 MM，要说早上好，晚上碰到 MM，要说晚上好；碰到 MM 穿了件新衣服，要说你的衣服好漂亮哦，碰到 MM 新做的发型，要说你的头发好漂亮哦。不要问我 “早上碰到 MM 新做了个发型怎么说” 这种问题，自己用 BRIDGE 组合一下不就行了 + +桥梁模式：将抽象化与实现化脱耦，使得二者可以独立的变化，也就是说将他们之间的强关联变成弱关联，也就是指在一个软件系统的抽象化和实现化之间使用组合 / 聚合关系而不是继承关系，从而使两者可以独立的变化。 + +### 08 合成模式 + +Mary 今天过生日。“我过生日，你要送我一件礼物。”“嗯，好吧，去商店，你自己挑。”“这件 T 恤挺漂亮，买，这条裙子好看，买，这个包也不错，买。”“喂，买了三件了呀，我只答应送一件礼物的哦。”“什么呀，T 恤加裙子加包包，正好配成一套呀，小姐，麻烦你包起来。”“……”，MM 都会用 Composite 模式了，你会了没有？ + +合成模式：合成模式将对象组织到树结构中，可以用来描述整体与部分的关系。合成模式就是一个处理对象的树结构的模式。合成模式把部分与整体的关系用树结构表示出来。合成模式使得客户端把一个个单独的成分对象和由他们复合而成的合成对象同等看待。 + +### 09 装饰模式 + +Mary 过完轮到 Sarly 过生日，还是不要叫她自己挑了，不然这个月伙食费肯定玩完，拿出我去年在华山顶上照的照片，在背面写上 “最好的的礼物，就是爱你的 Fita”，再到街上礼品店买了个像框（卖礼品的 MM 也很漂亮哦），再找隔壁搞美术设计的 Mike 设计了一个漂亮的盒子装起来……，我们都是 Decorator，最终都在修饰我这个人呀，怎么样，看懂了吗？ + +装饰模式：装饰模式以对客户端透明的方式扩展对象的功能，是继承关系的一个替代方案，提供比继承更多的灵活性。动态给一个对象增加功能，这些功能可以再动态的撤消。增加由一些基本功能的排列组合而产生的非常大量的功能。 + +### 10 门面模式 + +我有一个专业的 Nikon 相机，我就喜欢自己手动调光圈、快门，这样照出来的照片才专业，但 MM 可不懂这些，教了半天也不会。幸好相机有 Facade 设计模式，把相机调整到自动档，只要对准目标按快门就行了，一切由相机自动调整，这样 MM 也可以用这个相机给我拍张照片了。门面模式：外部与一个子系统的通信必须通过一个统一的门面对象进行。 + +门面模式提供一个高层次的接口，使得子系统更易于使用。每一个子系统只有一个门面类，而且此门面类只有一个实例，也就是说它是一个单例模式。但整个系统可以有多个门面类。 + +### 11 享元模式 + +每天跟 MM 发短信，手指都累死了，最近买了个新手机，可以把一些常用的句子存在手机里，要用的时候，直接拿出来，在前面加上 MM 的名字就可以发送了，再不用一个字一个字敲了。共享的句子就是 Flyweight，MM 的名字就是提取出来的外部特征，根据上下文情况使用。享元模式：FLYWEIGHT 在拳击比赛中指最轻量级。 + +享元模式以共享的方式高效的支持大量的细粒度对象。享元模式能做到共享的关键是区分内蕴状态和外蕴状态。内蕴状态存储在享元内部，不会随环境的改变而有所不同。外蕴状态是随环境的改变而改变的。外蕴状态不能影响内蕴状态，它们是相互独立的。 + +将可以共享的状态和不可以共享的状态从常规类中区分开来，将不可以共享的状态从类里剔除出去。客户端不可以直接创建被共享的对象，而应当使用一个工厂对象负责创建被共享的对象。享元模式大幅度的降低内存中对象的数量。 + +### 12 代理模式 + +跟 MM 在网上聊天，一开头总是 “hi, 你好”,“你从哪儿来呀？”“你多大了？”“身高多少呀？” 这些话，真烦人，写个程序做为我的 Proxy 吧，凡是接收到这些话都设置好了自己的回答，接收到其他的话时再通知我回答，怎么样，酷吧。 + +代理模式：代理模式给某一个对象提供一个代理对象，并由代理对象控制对源对象的引用。代理就是一个人或一个机构代表另一个人或者一个机构采取行动。某些情况下，客户不想或者不能够直接引用一个对象，代理对象可以在客户和目标对象直接起到中介的作用。 + +客户端分辨不出代理主题对象与真实主题对象。代理模式可以并不知道真正的被代理对象，而仅仅持有一个被代理对象的接口，这时候代理对象不能够创建被代理对象，被代理对象必须有系统的其他角色代为创建并传入。 + +### 13 责任链模式 + +晚上去上英语课，为了好开溜坐到了最后一排，哇，前面坐了好几个漂亮的 MM 哎，找张纸条，写上 “Hi, 可以做我的女朋友吗？如果不愿意请向前传”，纸条就一个接一个的传上去了，糟糕，传到第一排的 MM 把纸条传给老师了，听说是个老处女呀，快跑！ + +责任链模式：在责任链模式中，很多对象由每一个对象对其下家的引用而接起来形成一条链。请求在这个链上传递，直到链上的某一个对象决定处理此请求。客户并不知道链上的哪一个对象最终处理这个请求，系统可以在不影响客户端的情况下动态的重新组织链和分配责任。处理者有两个选择：承担责任或者把责任推给下家。一个请求可以最终不被任何接收端对象所接受。 + +### 14 命令模式 + +俺有一个 MM 家里管得特别严，没法见面，只好借助于她弟弟在我们俩之间传送信息，她对我有什么指示，就写一张纸条让她弟弟带给我。这不，她弟弟又传送过来一个 COMMAND，为了感谢他，我请他吃了碗杂酱面，哪知道他说：“我同时给我姐姐三个男朋友送 COMMAND，就数你最小气，才请我吃面。” + +命令模式：命令模式把一个请求或者操作封装到一个对象中。命令模式把发出命令的责任和执行命令的责任分割开，委派给不同的对象。命令模式允许请求的一方和发送的一方独立开来，使得请求的一方不必知道接收请求的一方的接口，更不必知道请求是怎么被接收，以及操作是否执行，何时被执行以及是怎么被执行的。系统支持命令的撤消。 + +### 15 解释器模式 + +俺有一个《泡 MM 真经》，上面有各种泡 MM 的攻略，比如说去吃西餐的步骤、去看电影的方法等等，跟 MM 约会时，只要做一个 Interpreter，照着上面的脚本执行就可以了。 + +解释器模式：给定一个语言后，解释器模式可以定义出其文法的一种表示，并同时提供一个解释器。客户端可以使用这个解释器来解释这个语言中的句子。解释器模式将描述怎样在有了一个简单的文法后，使用模式设计解释这些语句。 + +在解释器模式里面提到的语言是指任何解释器对象能够解释的任何组合。在解释器模式中需要定义一个代表文法的命令类的等级结构，也就是一系列的组合规则。每一个命令对象都有一个解释方法，代表对命令对象的解释。命令对象的等级结构中的对象的任何排列组合都是一个语言。 + +### 16 迭代模式 + +我爱上了 Mary，不顾一切的向她求婚。Mary：“想要我跟你结婚，得答应我的条件” 我：“什么条件我都答应，你说吧” Mary：“我看上了那个一克拉的钻石” 我：“我买，我买，还有吗？” Mary：“我看上了湖边的那栋别墅” 我：“我买，我买，还有吗？” Mary：“我看上那辆法拉利跑车” 我脑袋嗡的一声，坐在椅子上，一咬牙：“我买，我买，还有吗？” + +迭代模式：迭代模式可以顺序访问一个聚集中的元素而不必暴露聚集的内部表象。多个对象聚在一起形成的总体称之为聚集，聚集对象是能够包容一组对象的容器对象。迭代子模式将迭代逻辑封装到一个独立的子对象中，从而与聚集本身隔开。 + +迭代模式简化了聚集的界面。每一个聚集对象都可以有一个或一个以上的迭代子对象，每一个迭代子的迭代状态可以是彼此独立的。迭代算法可以独立于聚集角色变化。 + +### 17 调停者模式 + +四个 MM 打麻将，相互之间谁应该给谁多少钱算不清楚了，幸亏当时我在旁边，按照各自的筹码数算钱，赚了钱的从我这里拿，赔了钱的也付给我，一切就 OK 啦，俺得到了四个 MM 的电话。调停者模式：调停者模式包装了一系列对象相互作用的方式，使得这些对象不必相互明显作用。从而使他们可以松散偶合。 + +当某些对象之间的作用发生改变时，不会立即影响其他的一些对象之间的作用。保证这些作用可以彼此独立的变化。调停者模式将多对多的相互作用转化为一对多的相互作用。调停者模式将对象的行为和协作抽象化，把对象在小尺度的行为上与其他对象的相互作用分开处理。 + +### 18 备忘录模式 + +同时跟几个 MM 聊天时，一定要记清楚刚才跟 MM 说了些什么话，不然 MM 发现了会不高兴的哦，幸亏我有个备忘录，刚才与哪个 MM 说了什么话我都拷贝一份放到备忘录里面保存，这样可以随时察看以前的记录啦。 + +备忘录模式：备忘录对象是一个用来存储另外一个对象内部状态的快照的对象。备忘录模式的用意是在不破坏封装的条件下，将一个对象的状态捉住，并外部化，存储起来，从而可以在将来合适的时候把这个对象还原到存储起来的状态。 + +### 19 观察者模式 + +想知道咱们公司最新 MM 情报吗？加入公司的 MM 情报邮件组就行了，tom 负责搜集情报，他发现的新情报不用一个一个通知我们，直接发布给邮件组，我们作为订阅者（观察者）就可以及时收到情报啦。 + +观察者模式：观察者模式定义了一种一队多的依赖关系，让多个观察者对象同时监听某一个主题对象。这个主题对象在状态上发生变化时，会通知所有观察者对象，使他们能够自动更新自己。 + +### 20 状态模式 + +跟 MM 交往时，一定要注意她的状态哦，在不同的状态时她的行为会有不同，比如你约她今天晚上去看电影，对你没兴趣的 MM 就会说 “有事情啦”，对你不讨厌但还没喜欢上的 MM 就会说 “好啊，不过可以带上我同事么？”，已经喜欢上你的 MM 就会说 “几点钟？看完电影再去泡吧怎么样？”，当然你看电影过程中表现良好的话，也可以把 MM 的状态从不讨厌不喜欢变成喜欢哦。 + +状态模式：状态模式允许一个对象在其内部状态改变的时候改变行为。这个对象看上去象是改变了它的类一样。状态模式把所研究的对象的行为包装在不同的状态对象里，每一个状态对象都属于一个抽象状态类的一个子类。 + +状态模式的意图是让一个对象在其内部状态改变的时候，其行为也随之改变。状态模式需要对每一个系统可能取得的状态创立一个状态类的子类。当系统的状态变化时，系统便改变所选的子类。 + +### 21 策略模式 + +跟不同类型的 MM 约会，要用不同的策略，有的请电影比较好，有的则去吃小吃效果不错，有的去海边浪漫最合适，单目的都是为了得到 MM 的芳心，我的追 MM 锦囊中有好多 Strategy 哦。策略模式：策略模式针对一组算法，将每一个算法封装到具有共同接口的独立的类中，从而使得它们可以相互替换。 + +策略模式使得算法可以在不影响到客户端的情况下发生变化。策略模把行为和环境分开。环境类负责维持和查询行为类，各种算法在具体的策略类中提供。由于算法和环境独立开来，算法的增减，修改都不会影响到环境和客户端。 + +### 22 模板方法模式 + +看过《如何说服女生上床》这部经典文章吗？女生从认识到上床的不变的步骤分为巧遇、打破僵局、展开追求、接吻、前戏、动手、爱抚、进去八大步骤 (Template method)，但每个步骤针对不同的情况，都有不一样的做法，这就要看你随机应变啦 (具体实现)； + +模板方法模式：模板方法模式准备一个抽象类，将部分逻辑以具体方法以及具体构造子的形式实现，然后声明一些抽象方法来迫使子类实现剩余的逻辑。不同的子类可以以不同的方式实现这些抽象方法，从而对剩余的逻辑有不同的实现。先制定一个顶级逻辑框架，而将逻辑的细节留给具体的子类去实现。 + +### 23 访问者模式 + +情人节到了，要给每个 MM 送一束鲜花和一张卡片，可是每个 MM 送的花都要针对她个人的特点，每张卡片也要根据个人的特点来挑，我一个人哪搞得清楚，还是找花店老板和礼品店老板做一下 Visitor，让花店老板根据 MM 的特点选一束花，让礼品店老板也根据每个人特点选一张卡，这样就轻松多了； + +访问者模式：访问者模式的目的是封装一些施加于某种数据结构元素之上的操作。一旦这些操作需要修改的话，接受这个操作的数据结构可以保持不变。访问者模式适用于数据结构相对未定的系统，它把数据结构和作用于结构上的操作之间的耦合解脱开，使得操作集合可以相对自由的演化。访问者模式使得增加新的操作变的很容易，就是增加一个新的访问者类。 + +访问者模式将有关的行为集中到一个访问者对象中，而不是分散到一个个的节点类中。当使用访问者模式时，要将尽可能多的对象浏览逻辑放在访问者类中，而不是放到它的子类中。访问者模式可以跨过几个类的等级结构访问属于不同的等级结构的成员类。 \ No newline at end of file diff --git "a/docs/collection/API \351\235\242\350\257\225\345\233\233\350\277\236\346\235\200\357\274\232\346\216\245\345\217\243\345\246\202\344\275\225\350\256\276\350\256\241\357\274\237\345\256\211\345\205\250\345\246\202\344\275\225\344\277\235\350\257\201\357\274\237\347\255\276\345\220\215\345\246\202\344\275\225\345\256\236\347\216\260\357\274\237\351\230\262\351\207\215\345\246\202\344\275\225\345\256\236\347\216\260\357\274\237.md" "b/docs/collection/API \351\235\242\350\257\225\345\233\233\350\277\236\346\235\200\357\274\232\346\216\245\345\217\243\345\246\202\344\275\225\350\256\276\350\256\241\357\274\237\345\256\211\345\205\250\345\246\202\344\275\225\344\277\235\350\257\201\357\274\237\347\255\276\345\220\215\345\246\202\344\275\225\345\256\236\347\216\260\357\274\237\351\230\262\351\207\215\345\246\202\344\275\225\345\256\236\347\216\260\357\274\237.md" new file mode 100644 index 0000000000..687bd0eec2 --- /dev/null +++ "b/docs/collection/API \351\235\242\350\257\225\345\233\233\350\277\236\346\235\200\357\274\232\346\216\245\345\217\243\345\246\202\344\275\225\350\256\276\350\256\241\357\274\237\345\256\211\345\205\250\345\246\202\344\275\225\344\277\235\350\257\201\357\274\237\347\255\276\345\220\215\345\246\202\344\275\225\345\256\236\347\216\260\357\274\237\351\230\262\351\207\215\345\246\202\344\275\225\345\256\236\347\216\260\357\274\237.md" @@ -0,0 +1,742 @@ +> 在实际的业务中，难免会跟第三方系统进行数据的交互与传递，那么如何保证数据在传输过程中的安全呢（防窃取）？ +> +> 除了https的协议之外，能不能加上通用的一套算法以及规范来保证传输的安全性呢？ + +下面我们就来讨论下常用的一些API设计的安全方法，可能不一定是最好的，有更牛逼的实现方式，但是这篇是我自己的经验分享. + +### 一、token 简介 + +Token：访问令牌access token, 用于接口中, 用于标识接口调用者的身份、凭证，减少用户名和密码的传输次数。一般情况下客户端(接口调用方)需要先向服务器端申请一个接口调用的账号，服务器会给出一个appId和一个key, key用于参数签名使用，注意key保存到客户端，需要做一些安全处理，防止泄露。 + +Token的值一般是UUID，服务端生成Token后需要将token做为key，将一些和token关联的信息作为value保存到缓存服务器中(redis)，当一个请求过来后，服务器就去缓存服务器中查询这个Token是否存在，存在则调用接口，不存在返回接口错误，一般通过拦截器或者过滤器来实现，Token分为两种： + +- API Token(接口令牌): 用于访问不需要用户登录的接口，如登录、注册、一些基本数据的获取等。 获取接口令牌需要拿appId、timestamp和sign来换，sign=加密(timestamp+key) +- USER Token(用户令牌): 用于访问需要用户登录之后的接口，如：获取我的基本信息、保存、修改、删除等操作。获取用户令牌需要拿用户名和密码来换 + +关于Token的时效性：token可以是一次性的、也可以在一段时间范围内是有效的，具体使用哪种看业务需要。 + +一般情况下接口最好使用https协议，如果使用http协议，Token机制只是一种减少被黑的可能性，其实只能防君子不能防小人。 + +一般token、timestamp和sign 三个参数会在接口中会同时作为参数传递，每个参数都有各自的用途。 + + + +### 二、timestamp 简介 + +timestamp: 时间戳，是客户端调用接口时对应的当前时间戳，时间戳用于防止DoS攻击。当黑客劫持了请求的url去DoS攻击，每次调用接口时接口都会判断服务器当前系统时间和接口中传的的timestamp的差值，如果这个差值超过某个设置的时间(假如5分钟)，那么这个请求将被拦截掉，如果在设置的超时时间范围内，是不能阻止DoS攻击的。 timestamp机制只能减轻DoS攻击的时间，缩短攻击时间。如果黑客修改了时间戳的值可通过sign签名机制来处理。 + +#### DoS + +DoS是Denial of Service的简称，即拒绝服务，造成DoS的攻击行为被称为DoS攻击，其目的是使计算机或网络无法提供正常的服务。最常见的DoS攻击有计算机网络带宽攻击和连通性攻击。 + +DoS攻击是指故意的攻击网络协议实现的缺陷或直接通过野蛮手段残忍地耗尽被攻击对象的资源，目的是让目标计算机或网络无法提供正常的服务或资源访问，使目标系统服务系统停止响应甚至崩溃，而在此攻击中并不包括侵入目标服务器或目标网络设备。这些服务资源包括网络带宽，文件系统空间容量，开放的进程或者允许的连接。这种攻击会导致资源的匮乏，无论计算机的处理速度多快、内存容量多大、网络带宽的速度多快都无法避免这种攻击带来的后果。 + +- Pingflood: 该攻击在短时间内向目的主机发送大量ping包，造成网络堵塞或主机资源耗尽。 +- Synflood: 该攻击以多个随机的源主机地址向目的主机发送SYN包，而在收到目的主机的SYN ACK后并不回应，这样，目的主机就为这些源主机建立了大量的连接队列，而且由于没有收到ACK一直维护着这些队列，造成了资源的大量消耗而不能向正常请求提供服务。 + +- Smurf：该攻击向一个子网的广播地址发一个带有特定请求（如ICMP回应请求）的包，并且将源地址伪装成想要攻击的主机地址。子网上所有主机都回应广播包请求而向被攻击主机发包，使该主机受到攻击。 +- Land-based：攻击者将一个包的源地址和目的地址都设置为目标主机的地址，然后将该包通过IP欺骗的方式发送给被攻击主机，这种包可以造成被攻击主机因试图与自己建立连接而陷入死循环，从而很大程度地降低了系统性能。 +- Ping of Death：根据TCP/IP的规范，一个包的长度最大为65536字节。尽管一个包的长度不能超过65536字节，但是一个包分成的多个片段的叠加却能做到。当一个主机收到了长度大于65536字节的包时，就是受到了Ping of Death攻击，该攻击会造成主机的宕机。 +- Teardrop：IP数据包在网络传递时，数据包可以分成更小的片段。攻击者可以通过发送两段（或者更多）数据包来实现TearDrop攻击。第一个包的偏移量为0，长度为N，第二个包的偏移量小于N。为了合并这些数据段，TCP/IP堆栈会分配超乎寻常的巨大资源，从而造成系统资源的缺乏甚至机器的重新启动。 +- PingSweep：使用ICMP Echo轮询多个主机。 + +### 三、sign 简介 + +nonce：随机值，是客户端随机生成的值，作为参数传递过来，随机值的目的是增加sign签名的多变性。随机值一般是数字和字母的组合，6位长度，随机值的组成和长度没有固定规则。 + +sign: 一般用于参数签名，防止参数被非法篡改，最常见的是修改金额等重要敏感参数， sign的值一般是将所有非空参数按照升续排序然后+token+key+timestamp+nonce(随机数)拼接在一起，然后使用某种加密算法进行加密，作为接口中的一个参数sign来传递，也可以将sign放到请求头中。接口在网络传输过程中如果被黑客挟持，并修改其中的参数值，然后再继续调用接口，虽然参数的值被修改了，但是因为黑客不知道sign是如何计算出来的，不知道sign都有哪些值构成，不知道以怎样的顺序拼接在一起的，最重要的是不知道签名字符串中的key是什么，所以黑客可以篡改参数的值，但没法修改sign的值，当服务器调用接口前会按照sign的规则重新计算出sign的值然后和接口传递的sign参数的值做比较，如果相等表示参数值没有被篡改，如果不等，表示参数被非法篡改了，就不执行接口了。 + +### 四、防止重复提交 + +对于一些重要的操作需要防止客户端重复提交的(如非幂等性重要操作)，具体办法是当请求第一次提交时将sign作为key保存到redis，并设置超时时间，超时时间和Timestamp中设置的差值相同。当同一个请求第二次访问时会先检测redis是否存在该sign，如果存在则证明重复提交了，接口就不再继续调用了。如果sign在缓存服务器中因过期时间到了，而被删除了，此时当这个url再次请求服务器时，因token的过期时间和sign的过期时间一直，sign过期也意味着token过期，那样同样的url再访问服务器会因token错误会被拦截掉，这就是为什么sign和token的过期时间要保持一致的原因。拒绝重复调用机制确保URL被别人截获了也无法使用（如抓取数据）。 + +对于哪些接口需要防止重复提交可以自定义个注解来标记。 + +> 注意：所有的安全措施都用上的话有时候难免太过复杂，在实际项目中需要根据自身情况作出裁剪，比如可以只使用签名机制就可以保证信息不会被篡改，或者定向提供服务的时候只用Token机制就可以了。如何裁剪，全看项目实际情况和对接口安全性的要求。 + +### 五、使用流程 + +1. 接口调用方(客户端)向接口提供方(服务器)申请接口调用账号，申请成功后，接口提供方会给接口调用方一个appId和一个key参数 +2. 客户端携带参数appId、timestamp、sign去调用服务器端的API token，其中sign=加密(appId + timestamp + key) +3. 客户端拿着api_token 去访问不需要登录就能访问的接口 +4. 当访问用户需要登录的接口时，客户端跳转到登录页面，通过用户名和密码调用登录接口，登录接口会返回一个usertoken, 客户端拿着usertoken 去访问需要登录才能访问的接口 + +sign的作用是防止参数被篡改，客户端调用服务端时需要传递sign参数，服务器响应客户端时也可以返回一个sign用于客户度校验返回的值是否被非法篡改了。客户端传的sign和服务器端响应的sign算法可能会不同。 + + + +### 六、示例代码 + +#### 1. dependency + +```xml + + org.springframework.boot + spring-boot-starter-data-redis + + + redis.clients + jedis + 2.9.0 + + + + org.springframework.boot + spring-boot-starter-web + +``` + +#### 2. RedisConfiguration + +```java +@Configuration +public class RedisConfiguration { + @Bean + public JedisConnectionFactory jedisConnectionFactory(){ + return new JedisConnectionFactory(); + } + + /** + * 支持存储对象 + * @return + */ + @Bean + public RedisTemplate redisTemplate(){ + RedisTemplate redisTemplate = new StringRedisTemplate(); + redisTemplate.setConnectionFactory(jedisConnectionFactory()); + Jackson2JsonRedisSerializer jackson2JsonRedisSerializer = new Jackson2JsonRedisSerializer(Object.class); + ObjectMapper objectMapper = new ObjectMapper(); + objectMapper.setVisibility(PropertyAccessor.ALL, JsonAutoDetect.Visibility.ANY); + objectMapper.enableDefaultTyping(ObjectMapper.DefaultTyping.NON_FINAL); + + jackson2JsonRedisSerializer.setObjectMapper(objectMapper); + redisTemplate.setValueSerializer(jackson2JsonRedisSerializer); + redisTemplate.afterPropertiesSet(); + + return redisTemplate; + } +} +``` + +#### 3. TokenController + +```java +@Slf4j +@RestController +@RequestMapping("/api/token") +public class TokenController { + + @Autowired + private RedisTemplate redisTemplate; + + /** + * API Token + * + * @param sign + * @return + */ + @PostMapping("/api_token") + public ApiResponse apiToken(String appId, @RequestHeader("timestamp") String timestamp, @RequestHeader("sign") String sign) { + Assert.isTrue(!StringUtils.isEmpty(appId) && !StringUtils.isEmpty(timestamp) && !StringUtils.isEmpty(sign), "参数错误"); + + long reqeustInterval = System.currentTimeMillis() - Long.valueOf(timestamp); + Assert.isTrue(reqeustInterval < 5 * 60 * 1000, "请求过期，请重新请求"); + + // 1. 根据appId查询数据库获取appSecret + AppInfo appInfo = new AppInfo("1", "12345678954556"); + + // 2. 校验签名 + String signString = timestamp + appId + appInfo.getKey(); + String signature = MD5Util.encode(signString); + log.info(signature); + Assert.isTrue(signature.equals(sign), "签名错误"); + + // 3. 如果正确生成一个token保存到redis中，如果错误返回错误信息 + AccessToken accessToken = this.saveToken(0, appInfo, null); + + return ApiResponse.success(accessToken); + } + + + @NotRepeatSubmit(5000) + @PostMapping("user_token") + public ApiResponse userToken(String username, String password) { + // 根据用户名查询密码, 并比较密码(密码可以RSA加密一下) + UserInfo userInfo = new UserInfo(username, "81255cb0dca1a5f304328a70ac85dcbd", "111111"); + String pwd = password + userInfo.getSalt(); + String passwordMD5 = MD5Util.encode(pwd); + Assert.isTrue(passwordMD5.equals(userInfo.getPassword()), "密码错误"); + + // 2. 保存Token + AppInfo appInfo = new AppInfo("1", "12345678954556"); + AccessToken accessToken = this.saveToken(1, appInfo, userInfo); + userInfo.setAccessToken(accessToken); + return ApiResponse.success(userInfo); + } + + private AccessToken saveToken(int tokenType, AppInfo appInfo, UserInfo userInfo) { + String token = UUID.randomUUID().toString(); + + // token有效期为2小时 + Calendar calendar = Calendar.getInstance(); + calendar.setTime(new Date()); + calendar.add(Calendar.SECOND, 7200); + Date expireTime = calendar.getTime(); + + // 4. 保存token + ValueOperations operations = redisTemplate.opsForValue(); + TokenInfo tokenInfo = new TokenInfo(); + tokenInfo.setTokenType(tokenType); + tokenInfo.setAppInfo(appInfo); + + if (tokenType == 1) { + tokenInfo.setUserInfo(userInfo); + } + + operations.set(token, tokenInfo, 7200, TimeUnit.SECONDS); + + AccessToken accessToken = new AccessToken(token, expireTime); + + return accessToken; + } + + public static void main(String[] args) { + long timestamp = System.currentTimeMillis(); + System.out.println(timestamp); + String signString = timestamp + "1" + "12345678954556"; + String sign = MD5Util.encode(signString); + System.out.println(sign); + + System.out.println("-------------------"); + signString = "password=123456&username=1&12345678954556" + "ff03e64b-427b-45a7-b78b-47d9e8597d3b1529815393153sdfsdfsfs" + timestamp + "A1scr6"; + sign = MD5Util.encode(signString); + System.out.println(sign); + } +} +``` + +#### 4. WebMvcConfiguration + +```java +@Configuration +public class WebMvcConfiguration extends WebMvcConfigurationSupport { + + private static final String[] excludePathPatterns = {"/api/token/api_token"}; + + @Autowired + private TokenInterceptor tokenInterceptor; + + @Override + public void addInterceptors(InterceptorRegistry registry) { + super.addInterceptors(registry); + registry.addInterceptor(tokenInterceptor) + .addPathPatterns("/api/**") + .excludePathPatterns(excludePathPatterns); + } +} +``` + +#### 5. TokenInterceptor + +```java +@Component +public class TokenInterceptor extends HandlerInterceptorAdapter { + + @Autowired + private RedisTemplate redisTemplate; + + /** + * + * @param request + * @param response + * @param handler 访问的目标方法 + * @return + * @throws Exception + */ + @Override + public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception { + String token = request.getHeader("token"); + String timestamp = request.getHeader("timestamp"); + // 随机字符串 + String nonce = request.getHeader("nonce"); + String sign = request.getHeader("sign"); + Assert.isTrue(!StringUtils.isEmpty(token) && !StringUtils.isEmpty(timestamp) && !StringUtils.isEmpty(sign), "参数错误"); + + // 获取超时时间 + NotRepeatSubmit notRepeatSubmit = ApiUtil.getNotRepeatSubmit(handler); + long expireTime = notRepeatSubmit == null ? 5 * 60 * 1000 : notRepeatSubmit.value(); + + // 2. 请求时间间隔 + long reqeustInterval = System.currentTimeMillis() - Long.valueOf(timestamp); + Assert.isTrue(reqeustInterval < expireTime, "请求超时，请重新请求"); + + // 3. 校验Token是否存在 + ValueOperations tokenRedis = redisTemplate.opsForValue(); + TokenInfo tokenInfo = tokenRedis.get(token); + Assert.notNull(tokenInfo, "token错误"); + + // 4. 校验签名(将所有的参数加进来，防止别人篡改参数) 所有参数看参数名升续排序拼接成url + // 请求参数 + token + timestamp + nonce + String signString = ApiUtil.concatSignString(request) + tokenInfo.getAppInfo().getKey() + token + timestamp + nonce; + String signature = MD5Util.encode(signString); + boolean flag = signature.equals(sign); + Assert.isTrue(flag, "签名错误"); + + // 5. 拒绝重复调用(第一次访问时存储，过期时间和请求超时时间保持一致), 只有标注不允许重复提交注解的才会校验 + if (notRepeatSubmit != null) { + ValueOperations signRedis = redisTemplate.opsForValue(); + boolean exists = redisTemplate.hasKey(sign); + Assert.isTrue(!exists, "请勿重复提交"); + signRedis.set(sign, 0, expireTime, TimeUnit.MILLISECONDS); + } + + return super.preHandle(request, response, handler); + } +} +``` + +#### 6. MD5Util ----MD5工具类，加密生成数字签名 + +```java +public class MD5Util { + + private static final String hexDigits[] = { "0", "1", "2", "3", "4", "5", + "6", "7", "8", "9", "a", "b", "c", "d", "e", "f" }; + + private static String byteArrayToHexString(byte b[]) { + StringBuffer resultSb = new StringBuffer(); + for (int i = 0; i < b.length; i++) + resultSb.append(byteToHexString(b[i])); + + return resultSb.toString(); + } + + private static String byteToHexString(byte b) { + int n = b; + if (n < 0) + n += 256; + int d1 = n / 16; + int d2 = n % 16; + return hexDigits[d1] + hexDigits[d2]; + } + + public static String encode(String origin) { + return encode(origin, "UTF-8"); + } + public static String encode(String origin, String charsetname) { + String resultString = null; + try { + resultString = new String(origin); + MessageDigest md = MessageDigest.getInstance("MD5"); + if (charsetname == null || "".equals(charsetname)) + resultString = byteArrayToHexString(md.digest(resultString + .getBytes())); + else + resultString = byteArrayToHexString(md.digest(resultString + .getBytes(charsetname))); + } catch (Exception exception) { + } + return resultString; + } +} +``` + +#### 7. @NotRepeatSubmit -----自定义注解，防止重复提交。 + +```java +/** + * 禁止重复提交 + */ +@Target(ElementType.METHOD) +@Retention(RetentionPolicy.RUNTIME) +public @interface NotRepeatSubmit { + /** 过期时间，单位毫秒 **/ + long value() default 5000; +} +``` + +#### 8. AccessToken + +```java +@Data +@AllArgsConstructor +public class AccessToken { + /** token */ + private String token; + + /** 失效时间 */ + private Date expireTime; +} +``` + +#### 9. AppInfo + +```java +@Data +@NoArgsConstructor +@AllArgsConstructor +public class AppInfo { + /** App id */ + private String appId; + /** API 秘钥 */ + private String key; +} +``` + +#### 10. TokenInfo + +```java +@Data +public class TokenInfo { + /** token类型: api:0 、user:1 */ + private Integer tokenType; + + /** App 信息 */ + private AppInfo appInfo; + + /** 用户其他数据 */ + private UserInfo userInfo; +} +``` + +#### 11. UserInfo + +```java +@Data +public class UserInfo { + /** 用户名 */ + private String username; + /** 手机号 */ + private String mobile; + /** 邮箱 */ + private String email; + /** 密码 */ + private String password; + /** 盐 */ + private String salt; + + private AccessToken accessToken; + + public UserInfo(String username, String password, String salt) { + this.username = username; + this.password = password; + this.salt = salt; + } +} +``` + +#### 12. ApiCodeEnum + +```java +/** + * 错误码code可以使用纯数字,使用不同区间标识一类错误，也可以使用纯字符，也可以使用前缀+编号 + * + * 错误码：ERR + 编号 + * + * 可以使用日志级别的前缀作为错误类型区分 Info(I) Error(E) Warning(W) + * + * 或者以业务模块 + 错误号 + * + * TODO 错误码设计 + * + * Alipay 用了两个code，两个msg(https://docs.open.alipay.com/api_1/alipay.trade.pay) + */ +public enum ApiCodeEnum { + SUCCESS("10000", "success"), + UNKNOW_ERROR("ERR0001","未知错误"), + PARAMETER_ERROR("ERR0002","参数错误"), + TOKEN_EXPIRE("ERR0003","认证过期"), + REQUEST_TIMEOUT("ERR0004","请求超时"), + SIGN_ERROR("ERR0005","签名错误"), + REPEAT_SUBMIT("ERR0006","请不要频繁操作"), + ; + + /** 代码 */ + private String code; + + /** 结果 */ + private String msg; + + ApiCodeEnum(String code, String msg) { + this.code = code; + this.msg = msg; + } + + public String getCode() { + return code; + } + + public String getMsg() { + return msg; + } +} +``` + +#### 13. ApiResult + +```java +@Data +@NoArgsConstructor +@AllArgsConstructor +public class ApiResult { + + /** 代码 */ + private String code; + + /** 结果 */ + private String msg; +} +``` + +#### 14. ApiUtil -------这个参考支付宝加密的算法写的.我直接Copy过来了。 + +```java +public class ApiUtil { + + /** + * 按参数名升续拼接参数 + * @param request + * @return + */ + public static String concatSignString(HttpServletRequest request) { + Map paramterMap = new HashMap<>(); + request.getParameterMap().forEach((key, value) -> paramterMap.put(key, value[0])); + // 按照key升续排序，然后拼接参数 + Set keySet = paramterMap.keySet(); + String[] keyArray = keySet.toArray(new String[keySet.size()]); + Arrays.sort(keyArray); + StringBuilder sb = new StringBuilder(); + for (String k : keyArray) { + // 或略掉的字段 + if (k.equals("sign")) { + continue; + } + if (paramterMap.get(k).trim().length() > 0) { + // 参数值为空，则不参与签名 + sb.append(k).append("=").append(paramterMap.get(k).trim()).append("&"); + } + } + + return sb.toString(); + } + + public static String concatSignString(Map map) { + Map paramterMap = new HashMap<>(); + map.forEach((key, value) -> paramterMap.put(key, value)); + // 按照key升续排序，然后拼接参数 + Set keySet = paramterMap.keySet(); + String[] keyArray = keySet.toArray(new String[keySet.size()]); + Arrays.sort(keyArray); + StringBuilder sb = new StringBuilder(); + for (String k : keyArray) { + if (paramterMap.get(k).trim().length() > 0) { + // 参数值为空，则不参与签名 + sb.append(k).append("=").append(paramterMap.get(k).trim()).append("&"); + } + } + return sb.toString(); + } + + /** + * 获取方法上的@NotRepeatSubmit注解 + * @param handler + * @return + */ + public static NotRepeatSubmit getNotRepeatSubmit(Object handler) { + if (handler instanceof HandlerMethod) { + HandlerMethod handlerMethod = (HandlerMethod) handler; + Method method = handlerMethod.getMethod(); + NotRepeatSubmit annotation = method.getAnnotation(NotRepeatSubmit.class); + + return annotation; + } + + return null; + } +} +``` + +#### 15. ApiResponse + +```java +@Data +@Slf4j +public class ApiResponse { + /** 结果 */ + private ApiResult result; + + /** 数据 */ + private T data; + + /** 签名 */ + private String sign; + + + public static ApiResponse success(T data) { + return response(ApiCodeEnum.SUCCESS.getCode(), ApiCodeEnum.SUCCESS.getMsg(), data); + } + + public static ApiResponse error(String code, String msg) { + return response(code, msg, null); + } + + public static ApiResponse response(String code, String msg, T data) { + ApiResult result = new ApiResult(code, msg); + ApiResponse response = new ApiResponse(); + response.setResult(result); + response.setData(data); + + String sign = signData(data); + response.setSign(sign); + + return response; + } + + private static String signData(T data) { + // TODO 查询key + String key = "12345678954556"; + Map responseMap = null; + try { + responseMap = getFields(data); + } catch (IllegalAccessException e) { + return null; + } + String urlComponent = ApiUtil.concatSignString(responseMap); + String signature = urlComponent + "key=" + key; + String sign = MD5Util.encode(signature); + + return sign; + } + + /** + * @param data 反射的对象,获取对象的字段名和值 + * @throws IllegalArgumentException + * @throws IllegalAccessException + */ + public static Map getFields(Object data) throws IllegalAccessException, IllegalArgumentException { + if (data == null) return null; + Map map = new HashMap<>(); + Field[] fields = data.getClass().getDeclaredFields(); + for (int i = 0; i < fields.length; i++) { + Field field = fields[i]; + field.setAccessible(true); + + String name = field.getName(); + Object value = field.get(data); + if (field.get(data) != null) { + map.put(name, value.toString()); + } + } + + return map; + } +} +``` + + + +### 七、ThreadLocal + +ThreadLocal是线程内的全局上下文。就是在单个线程中，方法之间共享的内存，每个方法都可以从该上下文中获取值和修改值。 + +##### 实际案例： + +在调用api时都会传一个token参数，通常会写一个拦截器来校验token是否合法，我们可以通过token找到对应的用户信息(User)，如果token合法，然后将用户信息存储到ThreadLocal中，这样无论是在controller、service、dao的哪一层都能访问到该用户的信息。作用类似于Web中的request作用域。 + +传统方式我们要在方法中访问某个变量，可以通过传参的形式往方法中传参，如果多个方法都要使用那么每个方法都要传参；如果使用ThreadLocal所有方法就不需要传该参数了，每个方法都可以通过ThreadLocal来访问该值。 + +- ThreadLocalUtil.set("key", value); 保存值 +- T value = ThreadLocalUtil.get("key"); 获取值 + +```java +public class ThreadLocalUtil { + private static final ThreadLocal> threadLocal = new ThreadLocal() { + @Override + protected Map initialValue() { + return new HashMap<>(4); + } + }; + + + public static Map getThreadLocal(){ + return threadLocal.get(); + } + + public static T get(String key) { + Map map = (Map)threadLocal.get(); + return (T)map.get(key); + } + + public static T get(String key,T defaultValue) { + Map map = (Map)threadLocal.get(); + return (T)map.get(key) == null ? defaultValue : (T)map.get(key); + } + + public static void set(String key, Object value) { + Map map = (Map)threadLocal.get(); + map.put(key, value); + } + + public static void set(Map keyValueMap) { + Map map = (Map)threadLocal.get(); + map.putAll(keyValueMap); + } + + public static void remove() { + threadLocal.remove(); + } + + public static Map fetchVarsByPrefix(String prefix) { + Map vars = new HashMap<>(); + if( prefix == null ){ + return vars; + } + Map map = (Map)threadLocal.get(); + Set set = map.entrySet(); + + for( Map.Entry entry : set){ + Object key = entry.getKey(); + if( key instanceof String ){ + if( ((String) key).startsWith(prefix) ){ + vars.put((String)key,(T)entry.getValue()); + } + } + } + return vars; + } + + public static T remove(String key) { + Map map = (Map)threadLocal.get(); + return (T)map.remove(key); + } + + public static void clear(String prefix) { + if( prefix == null ){ + return; + } + Map map = (Map)threadLocal.get(); + Set set = map.entrySet(); + List removeKeys = new ArrayList<>(); + + for( Map.Entry entry : set ){ + Object key = entry.getKey(); + if( key instanceof String ){ + if( ((String) key).startsWith(prefix) ){ + removeKeys.add((String)key); + } + } + } + for( String key : removeKeys ){ + map.remove(key); + } + } +} +``` + + + +### 总结: + +这个是目前第三方数据接口交互过程中常用的一些参数与使用示例，希望对大家有点帮助。 + +当然如果为了保证更加的安全，可以加上RSA,RSA2，AES等等加密方式，保证了数据的更加的安全，但是唯一的缺点是加密与解密比较耗费CPU的资源. \ No newline at end of file diff --git a/docs/collection/DataGrip.md b/docs/collection/DataGrip.md new file mode 100644 index 0000000000..a962239e22 --- /dev/null +++ b/docs/collection/DataGrip.md @@ -0,0 +1,305 @@ +> 最近在参与一个离职同事的数据交接工作，发现他用的 MySQL 工具竟然不用写完整的 sql 就可以查询，随便写了几个字段，竟然就有提示，感觉很高级，然后就也下载体验了下，真香 + +最近看到一款数据库客户端工具，DataGrip，是大名鼎鼎的JetBrains公司出品的，就是那个出品Intellij IDEA的公司。DataGrip是一款数据库管理客户端工具，方便连接到数据库服务器，执行sql、创建表、创建索引以及导出数据等。之前试用的客户端工具是dbvisualizer，但是在试用了DataGrip以后，我就决定抛弃dbvisualizer。 + +### DataGrip 初识 + +我相信，当你第一眼看到DataGrip以后，会有一种惊艳的感觉，就好比你第一眼看到一个姑娘，就是那么一瞥，你对自己说，就是她了！废话不多说，来看看DataGrip的常用功能。安装过程也很简单，双击安装，下一步，中间会让你选择主题，本人选择的是经典的Darcula，安装完成后，启动，界面如下 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoictRBzKxic8j9dRI7nsyl1hoZEK43ZSy2OtzuUXmxWdibdy0XQJlL1fm2Q/640?wx_fmt=png) + +相信使用过I DEA 的同学会感到很亲切。接下来管理数据库驱动。DataGrip 支持主流的数据库，File->DataSource + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoic5KdicqdQ85q9PPLD901icwia5XmwvzJq4zfOYuTbMB6ce4vvckpAZ8tdA/640?wx_fmt=png) + + + +也可以在 Database 视图中展开绿色的 + 号，添加数据库连接 + +![](https://mmbiz.qpic.cn/mmbiz_jpg/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoicXZOwodlwqInkibBg5F1XbIauicwd2Fw7G3H1S2YvK5t5Zu10l2wKO9Lw/640?wx_fmt=jpeg) + + +选择需要连接的数据库类型 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoicNYK1FQRHwetysopFWR5r71ibHtkKR8YdjuhSwhEBRP7XUQicR1dHqkjw/640?wx_fmt=png) + +在面板中，左上部分列出了已经建立的数据库连接，点击各项，右侧会展示当前连接的配置信息，General 面板中，可以配置数据库连接的信息，如主机、用户名、密码等，不同数据库配置信息不完全相同，填入数据库 URL，注意，URL 后有个选项，可以选择直接填入url，那么就不需要单独填主机名、端口等信息了。 + + + +Driver 部分显示数据库驱动信息，如果还没有下载过驱动，底部会有个警告，提示缺少驱动 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoicWGVjDAOibJgJSRneAFAsDgzvPsPbiaH2P7hXkZhVoATlicCxcN41lhCgg/640?wx_fmt=png) + +点击 Driver 后的数据库类型，会跳转到驱动下载页面，点击 download，下载完会显示驱动包 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoicXejbL3cMGTmPexovjAQwsa4fUicjugjCVzZl6Mxh7ZYpG9E6Zt65cOA/640?wx_fmt=png) + + + +如果下载的驱动有问题，可以手动添加本地驱动包，在试用过程中，创建Oracle连接时，下载的驱动包就有问题，提示缺少class，点击右侧绿色的+号，选择本地下载好的jar包，通过右侧上下箭头，将导入的jar包移到最上位置就OK了 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoicwy3x45tMxdnCfoicI1PsmtIZCCfXfj9pWxm5QQibxg7jhzslYuzibHMww/640?wx_fmt=png) + + + +点击 Test Connection，查看配置是否正确，接下来就可以使用了。 + +打开DataGrip，选择File->Settings，当前面板显示了常用设置项 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoicVE3b6dMIKrEDRx81Zd0kKl9PXPpqTiaLcYf2W1esaFc2qK1V7dE9aVA/640?wx_fmt=png) + + + +基本上默认设置就足够了，要更改设置也很简单，左侧菜单已经分类好了，第一项是数据库相关的配置，第二项是配置外观的，在这里可以修改主题，key map修改快捷键，editor配置编辑器相关设置，在这里可以修改编辑器字体，展开edit项，Editor->Color & Fonts->Font + +![img](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoicqdKaSNmibxvB9DWmaIgFkUnU3h7vcWyVRENzLmJjy4J7RErVOicyKfmg/640?wx_fmt=png) + +需要将当前主题保存一下，点击save as，起个名，选择重命名后的主题就能修改了，这里我选择习惯的Conurier New字体，大小为14号，点击右下角的apply，点击OK + +![img](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoicDiahykWhUkG2EX0z3kyNR0qvxltib4pSxmahsJkrmPOvERR0C4pSqPJA/640?wx_fmt=png) + +其他的没啥好设置的了。 + +### 常用操作 + +接下来，我们来使用DataGrip完成数据库的常用操作，包括查询数据、修改数据，创建数据库、表等。 + +![img](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoicI709MNMK8UicvNSAsgykzmmw77DAEoVIr1j124mNY7G9SDTrzVOIb9Q/640?wx_fmt=png) + +#### 操作数据 + +左上区域显示了当前数据库连接，展开后会显示数据库表等信息，如果展开后没有任何信息，需要选中数据库连接，点击上面的旋转图标同步一下，下方有个More Schema选项，点击可以切换不同的schema。 + +右键选中的数据库连接，选择open console，就可以在右侧的控制台中书写sql语句了。 + +![img](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoiczuFkDoeeDJjdaHQZyeh5P7zc6nGB9LrkiaxzEXhuCs20GeQCbZEJQ1A/640?wx_fmt=png) + +DataGrip的智能提示非常爽，无论是标准的sql关键字，还是表名、字段名，甚至数据库特定的字段，都能提示，不得不感叹这智能提示太强大了，Intellij IDEA的智能提示也是秒杀eclipse。 + +写完sql语句后，可以选中，电子左上侧绿色箭头执行 + +![img](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoicaJa4y30Y7pSxRibD4jSwYHsB8r8UHsVGWuxRf4BOonWJLByaVcf1P7w/640?wx_fmt=png) + +也可以使用快捷键Ctrl+Enter，选中情况下，会直接执行该sql，未选中情况下，如果控制台中有多条sql，会提示你要执行哪条sql。之前习惯了dbvisualizer中的操作，dbvisualizer中光标停留在当前sql上(sql以分号结尾)，按下Ctrl+.快捷键会自动执行当前sql，其实DataGrip也能设置，在setting->Database-General中 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoic1aQl1PlM5HXKWPBS0l1MTV8BMFwtjpS4zXkxx3LIw6o6ZqONwtw8hQ/640?wx_fmt=png) + +语句执行时默认是提示，改成smallest statement后，光标停留在当前语句时，按下Ctrl+Enter就会直接执行当前语句。 + +语句的执行结果在底部显示 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoicYARKV8IyTwHQVQ6K6h3icHWc6PGlXyaH1tD4Su2kDgxlMGUMCEIJrgA/640?wx_fmt=png) + +如果某列的宽度太窄，可以鼠标点击该列的任意一个，使用快捷键Ctrl+Shift+左右箭头可以调整宽度，如果要调整所有列的宽度，可以点击左上角红框部分，选择所有行，使用快捷键Ctrl+Shift+左右箭头调整 + +添加行、删除行也很方便，上部的+、-按钮能直接添加行或删除选中的行，编辑列同样也很方便，双击要修改的列，输入修改后的值，鼠标在其他部分点击就完成修改了 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoicGibgrJAvjUFDzS3JSuibWI931SibRX8jLX7NVYr72ohsicic2jibVpCHHd6g/640?wx_fmt=png) + +有的时候我们要把某个字段置为null，不是空字符串""，DataGrip也提供了渐变的操作，直接在列上右键，选择set null + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoicStZ9aVmDfXQ83bFwcSiaC5nFknqbNWGH86DSGBGWU7HOjcjI6X85feg/640?wx_fmt=png) + +对于需要多窗口查看结果的，即希望查询结果在新的tab中展示，可以点击pin tab按钮，那新查询将不会再当前tab中展示，而是新打开一个tab + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoicuBeKeTte8JDCuaLxueibf0o3yyHhNrVBewsZuPD2s2tibGxaCsKZH6fQ/640?wx_fmt=png) + +旁边的output控制台显示了执行sql的日志信息，能看到sql执行的时间等信息 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoicibOTVJVZyBCCMPdRwqzia86icKCibbFCuiad1tRiacgdEkYDvcyZibNiaO3HeQ/640?wx_fmt=png) + +我就问这么吊的工具，还有谁！！！ + +#### 操作表 + +要新建表也是相当简单、智能，选中数据库连接，点击绿色+号下选择table + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoic25XP1iaaLaHOb338zbxuwiawuSoibalPDuqsgYvGULstzxsNUaZx4qGug/640?wx_fmt=png) + +在新打开的窗口中，可以填写表信息 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoicGRxzbno3Q1Yux2iaNsPP7FWxC6Hnzv7o2ziaiaenMWCGBjja4eBlcOfOA/640?wx_fmt=png) + +我就问你看到这个窗口兴奋不兴奋！！！ + +顶部可以填写表名、表注释，中间可以点击右侧绿色+号添加列，列类型type也是能自动补全，default右侧的消息框图标点击后能对列添加注释，旁边的几个tab可以设置索引及外键 + +所有这些操作的DDL都会直接在底部显示 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoicPPUfBt50xckCJ4mw0OHqBVac0nj5hSyGK0k8aVpoq057EumjicF5Vvw/640?wx_fmt=png) + +我就问你怕不怕 + +表建完后，可以点击下图中的table图标，打开表查看视图 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoicqV1AB1K1GlSgYH8NqO68NYMjEHoH0nlnOleNib84UQhjhPGwgQsKxCA/640?wx_fmt=png) + +可以查看表的数据，也能查看DDL语句 + +这些基本功能的设计、体验，已经惊艳到我了，接下来就是数据的导出。 + +#### 导出数据 + +DataGrip的导出功能也是相当强大 + +选择需要导出数据的表，右键，Dump Data To File + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoichdDHwo8ylicv4zTKxA63NviaLBzJR7guNJ5Z4eXhEAKGK4ko4AN03V5Q/640?wx_fmt=png) + +即可以导出insert、update形式的sql语句，也能导出为html、csv、json格式的数据 + +也可以在查询结果视图中导出 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoic92yWKRoaYkuWHzHhmspqCA73bjp9ibw8neaOjI0UMGs1no8JJnKQm9A/640?wx_fmt=png) + +点击右上角下载图标，在弹出窗口中可以选择不同的导出方式，如sql insert、sql update、csv格式等 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoicabK0hpwvCKpoMjc9dmr9HgjfFJ2GfsmNS0g7hxjMdWgOLyyc3VaTfg/640?wx_fmt=png) + +如果是导出到csv格式，还能控制导出的格式 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoiclP64Q7s8zvoNkmn66cXs2NtEianu3Hh9R8Z1YSiaS694HUPfPrYlnNNw/640?wx_fmt=png) + +导出后用excel打开是这种结果 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoicsHJJXfIpcqvyTvFKmNgc5L8GNc2oz9VCE8uibD5putzR20Anbl4oYnQ/640?wx_fmt=png) + + + +#### 导入数据 + +除了能导出数据外，还能导入数据 + +选择表，右键->Import from File，选择要导入的文件 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoic16veMlLHspBzhzDzasYOr9KXPgpDicq6U7mHl91r59d64LZUmkjT0WQ/640?wx_fmt=png) + +注意，导出的时候如果勾选了左侧的两个header选项，导入的时候如果有header，也要勾选，不然会提示列个数不匹配 + +### 骚操作 + +#### 1、关键字导航： + +当在datagrip的文本编辑区域编写sql时，按住键盘Ctrl键不放，同时鼠标移动到sql关键字上，比如表名、字段名称、或者是函数名上，鼠标会变成手型，关键字会变蓝，并加了下划线，点击，会自动定位到左侧对象树，并选中点击的对象 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoicaSapDQicnKTecKRHqicDfHfGRAyJWlflXtIiawQ8GThSy1zdDnvNzDAiaA/640?wx_fmt=png) + +#### 2、快速导航到指定的表、视图、函数等： + +在datagrip中，使用Ctrl+N快捷键，弹出一个搜索框，输入需要导航的名称，回车即可 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoicEdYgsxOdibKicJElWEMibzD7rQhIpznTnJunJkzD5gByTcEIFWuJ8Rx3g/640?wx_fmt=png) + +#### 3、全局搜索 + +连续两次按下shift键，或者鼠标点击右上角的搜索图标，弹出搜索框，搜索任何你想搜索的东西 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoicPpjXq6ZweWdLricQFIzib1ghmkcFESibPASbjFCkl9ZIoavk7zmpuQZaA/640?wx_fmt=png) + +#### 4、结果集搜索 + +在查询结果集视图区域点击鼠标，按下Ctrl+F快捷键，弹出搜索框，输入搜索内容，支持正则表达式、过滤结果 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoicyPbgMCXo86E3iciaaaVn3dsv7MDEmfLaOg39de7eQPfV1mu2iaUc7An7w/640?wx_fmt=png) + +#### 5、导航到关联数据 + +表之间会有外检关联，查询的时候，能直接定位到关联数据，或者被关联数据，例如user1表有个外检字段classroom指向classroom表的主键id，在查询classroom表数据的时候，可以在id字段上右键，go to，referencing data + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoicgibU4bibmkb5wdic0L9Lliayk9ia72CtomOqkaiaWoRTlZHFPxYN2hrgqCXQ/640?wx_fmt=png) + +选择要显示第一条数据还是显示所有数据 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoicgLViaE7BQZG71SUnibvR4TrQ2qSczdNEr0da0zddOqqEKdoZKSz05h5w/640?wx_fmt=png) + +会自动打开关联表的数据 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoicLuatgQITfX9LoiaNUPibG18fpibMOPEeib8Lp3mKI3z1XfqqlCl6YPQK2Q/640?wx_fmt=png) + +相反，查询字表的数据时，也能自动定位到父表 + +#### 6、结果集数据过滤 + +对于使用table edit（对象树中选中表，右键->table editor）打开的结果集，可以使用条件继续过滤结果集，如下图所示，可以在结果集左上角输入款中输入where条件过滤 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoicrJzyYIy9dIRxyRSVicGvyJUribMVLrYOH2jowQ0jIzGgMEWeeAoQSfFg/640?wx_fmt=png) + +也可以对着需要过滤数据的列右键，filter by过滤 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoic5elJHMv7XSYKziabwj2vOJfbRPR7RQYicda0pu240qxfngZ7x1sE2Ssg/640?wx_fmt=png) + +#### 7、行转列 + +对于字段比较多的表，查看数据要左右推动，可以切换成列显示，在结果集视图区域使用Ctrl+Q快捷键 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoiccVAPLhZ389NPKk9Vlad4X6iaOgstPKPm9XicHUiauibUns8icuRHO5d8Fmw/640?wx_fmt=png) + +### 智能化 + +#### 1、变量重命名 + +鼠标点击需要重命名的变量，按下Shift+F6快捷键，弹出重命名对话框，输入新的名称 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoic3picnCsiaDNxork2jaMUXYp3I7IrlVCEzkuCyAJq6lGqIeGIz8K5yr6A/640?wx_fmt=png) + +#### 2、自动检测无法解析的对象 + +如果表名、字段名不存在，datagrip会自动提示，此时对着有问题的表名或字段名，按下Alt+Enter，会自动提示是否创建表或添加字段 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoicKzTNibV2cdWfncwZuQzjwRXCu01ibzFCa7jStJpvFumVa0FkoCphHrMg/640?wx_fmt=png) + +#### 3、权限定字段名 + +对于查询使用表别名的，而字段中没有使用别名前缀的，datagrip能自动添加前缀，鼠标停留在需要添加别名前缀的字段上，使用Alt+Enter快捷键 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoicXf7IW0iaVPZFwQG2kjyqwVsK6yicxK76ZiaTPRoCicH4biaUmF3P9hu5NMA/640?wx_fmt=png) + +#### 4、通配符自动展开 + +查询的时候我们会使用select _查询所有列，这是不好的习惯，datagrip能快速展开列，光标定位到_后面，按下Alt+Enter快捷键 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoiclG3MjwhSX94QwLIUXvOh343t7fb3aeBAdR6uULYjX82dYtCic3d6kVg/640?wx_fmt=png) + +#### 5、大写自动转换 + +sql使用大写形式是个好的习惯，如果使用了小写，可以将光标停留在需要转换的字段或表名上，使用Ctrl+shift+U快捷键自动转换 + +#### 6、sql格式化 + +选中需要格式化的sql代码，使用Ctrl+Alt+L快捷键 + + + +### 强大的编辑器 + +> datagrip提供了一个功能强大的编辑器，实现了notpad++的列编辑模式 + +#### 1、多光标模式 + +在编辑sql的时候，可能需要同时输入或同时删除一些字符，按下alt+shift，同时鼠标在不同的位置点击，会出现多个光标 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoicwEDPSIDFpL1EH2KWJ4AkL0wWWRjO7tRobgOGDEj82oeeUJEWDlJ1wQ/640?wx_fmt=png) + +#### 2、代码注释 + +选中要注释的代码，按下Ctrl+/或Ctrl+shift+/快捷键，能注释代码，或取消注释 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoicOuwWOuIL7fKU89tKW0WlnJRuTH6eURsBGJ6gc8zgo3muPLEe5kic4zA/640?wx_fmt=png) + +#### 3、列编辑 + +按住键盘Alt键，同时按下鼠标左键拖动，能选择多列，拷贝黏贴等操作 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoicBV1T8OnkzSsxqpW5Wj2gKXE9zcyfbGuEXUnvGeXg8ZDxmAGnfic4hAg/640?wx_fmt=png) + +#### 4、代码历史 + +在文本编辑器中，邮件，local history，show history，可以查看使用过的sql历史 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoic9G5vPgynLADjSeslibBPE92RQ1XL2H69QNQbSRJmGaaWPeeAPibQMficg/640?wx_fmt=png) + +#### 5、命令历史 + +![](https://mmbiz.qpic.cn/mmbiz_png/eukZ9J6BEiacgYOmicqibauZ8YgyP7bQWoiclN7FOVXVjlfcfXY8BBpF3qWpSXV53BungvsvDssyP8iasJPicp4JB8sQ/640?wx_fmt=png) + diff --git "a/docs/collection/GitHub \351\252\232\346\223\215\344\275\234\357\274\214\344\270\252\344\272\272\351\241\265\350\277\230\350\203\275\350\277\231\344\271\210\347\216\251\357\274\237.md" "b/docs/collection/GitHub \351\252\232\346\223\215\344\275\234\357\274\214\344\270\252\344\272\272\351\241\265\350\277\230\350\203\275\350\277\231\344\271\210\347\216\251\357\274\237.md" new file mode 100644 index 0000000000..45a4228bd6 --- /dev/null +++ "b/docs/collection/GitHub \351\252\232\346\223\215\344\275\234\357\274\214\344\270\252\344\272\272\351\241\265\350\277\230\350\203\275\350\277\231\344\271\210\347\216\251\357\274\237.md" @@ -0,0 +1,41 @@ +# GitHub 骚操作，个人页还能这么玩？ + +![](https://tva1.sinaimg.cn/large/007S8ZIlly1gh3ducoo1sj31kw0sgtez.jpg) + +之前写过一篇 GitHub 骚操作的文章，今天要介绍的算是骚样式 + +> 虽然代码写的少，但是逼格不能丢呀 + +继『全球最大同性交友平台』GitHub 被微软收购之后，经常会有一些新花样，ui变化，界面变化，最近一次的改版相信大家已经看到了，布局上增加了右边栏，显示了更多的信息，同时整个 Repo 上侧所占的宽度达到了屏幕的 100%。 + +![](https://tva1.sinaimg.cn/large/007S8ZIlly1gh37kl6y5mj326u0ruqbp.jpg) + + + +之前如果我们想在首页有一个自我介绍，一般都是用 Gists，现在有了Github Profile ReadMe 可以帮我们实现自定义个人主页介绍，看看我新搞的主页：https://github.com/Jstarfish。 + +![](https://tva1.sinaimg.cn/large/007S8ZIlly1gh3dni4jwtj31rr0u04qp.jpg) + + + +一、首先在 GitHub 上建立一个与自己 GitHub 账户同名的仓库，记得勾选 README。 + +![](https://tva1.sinaimg.cn/large/007S8ZIlly1gh37ju6ovij310b0u0wkn.jpg) + + + +二、该仓库的 README.md 就会显示在首页了，直接编辑就可以，接着就靠自己随意发挥了。 + +![](https://tva1.sinaimg.cn/large/007S8ZIlly1gh386tecrnj31ee05w0td.jpg) + + + +三、“抄袭”大牛们的创意 + +https://github.com/anuraghazra/github-readme-stats 可以在你的 README 中 获取动态生成的 GitHub 统计信息 + +https://github.com/kautukkundan/Awesome-Profile-README-templates 这个仓库收集了很多比较有创意的README 模板，我们可以参考着自己修改。 + + + +最后，让我们为同性的友谊干杯🍻，秀出自己的主页。 \ No newline at end of file diff --git "a/docs/collection/IDEA \351\230\262\346\255\242\345\206\231\344\273\243\347\240\201\346\262\211\350\277\267\346\217\222\344\273\266.md" "b/docs/collection/IDEA \351\230\262\346\255\242\345\206\231\344\273\243\347\240\201\346\262\211\350\277\267\346\217\222\344\273\266.md" new file mode 100755 index 0000000000..122aa428d7 --- /dev/null +++ "b/docs/collection/IDEA \351\230\262\346\255\242\345\206\231\344\273\243\347\240\201\346\262\211\350\277\267\346\217\222\344\273\266.md" @@ -0,0 +1,144 @@ +## 前言 + +当初年少懵懂，那年夏天填志愿选专业，父母听其他长辈说选择计算机专业好。从那以后，我的身上就有了计院深深的烙印。从寝室到机房，从机房到图书馆，C、C++、Java、只要是想写点自己感兴趣的东西，一坐就是几个小时，但那时年轻，起身，收拾，一路小跑会女神，轻轻松松。现在工作了，毫无意外的做着开发的工作，长时间久坐。写代码一忙起来就忘了起来活动一下，也不怎么喝水。经常等到忙完了就感觉腰和腿不舒服。直到今年的体检报告一下来，才幡然醒：:没有一个好身体,就不能好好打工，让老板过上他自己想要的生活了. + +试过用手机提醒自己，但是没用。小米手环的久坐提醒功能也开着，有时候写代码正入神的，时间到了也就点一下就关了，还是没什么作用。所以我想究竟是我太赖了，还是用Idea写代码容易沉迷，总之不可能是改需求有意思。所以打算为自己开发一款小小的Idea防沉迷插件，我叫她【StopCoding】。她应该可以设置每隔多少分钟，就弹出一个提醒对话框，一旦对话框弹出来，idea 的代码编辑框就自动失去了焦点，什么都不能操作，到这还不算完，关键是这个对话框得关不了，并且还显示着休息倒计时，还有即使我修改了系统时间，这个倒计时也依然有效，除非我打开任务管理器，关闭 Idea 的进程，然后再重新启动 Idea。但是想一下想，idea 都都关了，还是休息一下吧。 + +下面就介绍一下她简单的使用教程和开发教程 + +## 安装使用教程 + +### 安装 + +1. 在idea中直接搜索安装StopCoding插件(官方已经审核通过) + +![](https://p6-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/b7c4529d9ad74e52b25c9e3ff8bde987~tplv-k3u1fbpfcp-watermark.image) + +2. 内网开发的小伙伴 可以下载之后进行本地安装 ：https://github.com/jogeen/StopCoding/releases/tag/20210104-V1.0 + +- 本地安装： + +![img](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/b3a879d9342042f1889177c5417119d1~tplv-k3u1fbpfcp-watermark.image) + +### 使用 + +- Step1. 然后在菜单栏中tools->StopCoding + +![img](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/d444991c76a64791af61f506b1c4ae16~tplv-k3u1fbpfcp-watermark.image) + +- Step2. 设置适合你的参数然后保存。 + +![img](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/1e55c783876e40728480b34aa3f326cf~tplv-k3u1fbpfcp-watermark.image) + +- Step3. 然后快乐的Coding吧，再不用担心自己会沉迷了。工作时间结束,她会弹出下框进行提醒,当然,这个框是关不掉的.只有你休息了足够的时间它才会自动关闭. + +![img](https://p9-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/38cef698f7084ec69fb9d7a862119aec~tplv-k3u1fbpfcp-watermark.image) + +## 开发教程 + +这个插件非常的简约,界面操作也很简单。所使用的技术基本上都是java的基础编程知识。所以小伙伴感兴趣的话，一起看看吧。 + +### 技术范围 + +- 插件工程的基本结构 +- Swing 主要负责两个对话框的交互 +- Timer 作为最基本的定时器选择 + +### 插件工程结构 + +![img](https://p9-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/c6fe38bd434f44659072012a56d238fd~tplv-k3u1fbpfcp-watermark.image) + +- plugin.xml + +这是插件工程的核心配置文件，里面每一项的解释，可以参考第一篇的介绍[核心配置文件说明](https://juejin.cn/post/6844904127990857742)。 + +- data包 + - SettingData,配置信息对应model + - DataCenter,作为运行时的数据中心，都是些静态的全局变量 +- service + - TimerService 这个定时计算的核心代码 +- task + - RestTask 休息时的定时任务 + - WorkTask 工作时的定时任务 +- ui + - SettingDialog 设置信息的对话框 + - TipsDialog 休息时提醒的对话框 +- StopCodingSettingAction 启动入口的action + +### Swing + +其实在idea中开发Swing项目的界面非常简单。因为idea提供了一系列可视化的操作，以及控件布局的拖拽。接下来就简单的介绍一下对话框的创建过程和添加事件。 + +#### 创建对话框 + +- Step1 + +![img](https://p9-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/63ee7f02dd04402c81c7a228f96ca49f~tplv-k3u1fbpfcp-watermark.image) + +- Step2 + +![img](https://p6-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/18ce507706784d348edbb459f8e8f46e~tplv-k3u1fbpfcp-watermark.image) + +- Step3 + +![img](https://p9-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/43d91050e4f145b48319e88f075af49d~tplv-k3u1fbpfcp-watermark.image) + +- 注：这里并没有详细的展开Swing的讲解，因为界面的这个东西，需要大家多去自己实践。这里就不做手册式的赘述了。 + +#### 添加事件 + +其实，刚才创建的这个对话框里的两个按钮都是默认已经创建好了点击事件的。 + +```java +public class TestDialog extends JDialog { + private JPanel contentPane; + private JButton buttonOK; + private JButton buttonCancel; + + public TestDialog() { + setContentPane(contentPane); + setModal(true); + getRootPane().setDefaultButton(buttonOK); + + buttonOK.addActionListener(new ActionListener() { + public void actionPerformed(ActionEvent e) { + onOK(); + } + }); //这是给OK按钮绑定点击事件的监听器 + + buttonCancel.addActionListener(new ActionListener() { + public void actionPerformed(ActionEvent e) { + onCancel(); + } + });//这是给取消按钮绑定点击事件的监听器 + //其他代码 + } +``` + +当然我们也可以其它任何控件去创建不同的事件监听器。这里可以通过界面操作创建很多种监听器，只要你需要，就可以使用。 + +- step1 + +![img](https://p1-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/2799b4f1cb724270a8ee561d5dc8040b~tplv-k3u1fbpfcp-watermark.image) + +- step2 + +![img](https://p1-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/9ef64de972ef41dfaa95d325fe9ceccb~tplv-k3u1fbpfcp-watermark.image) + +### Timer定时器 + +在这个插件里面，需要用到定时的功能，同时去计算公国和休息的时间。所以使用JDK自带的Timer，非常的方便。下面我Timer的常用的api放在这里，就清楚它的使用了。 + +- 构造方法 + +![img](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/1a074e309ff846fb94b09fc2ae94efca~tplv-k3u1fbpfcp-watermark.image) + +- 成员防范 + +![img](https://p1-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/a576b3deae2b423990c37fb3be1982f6~tplv-k3u1fbpfcp-watermark.image) + +- 主要是schedule去添加一个定时任务，和使用cancel去取消任务停止定时器。 + +### 最后 + +相信有了这些基本介绍，感谢兴趣的小伙伴想去看看源码和尝试自己写一个小插件就没什么大问题了。不说了，我得休息了。希望这个插件能帮到作为程序员得你，和这篇文章对你有一点点启发。当然麻烦小伙伴点个赞，鼓励一下打工人。 \ No newline at end of file diff --git a/docs/others/Idea2020.1.md b/docs/collection/Idea2020.1.md similarity index 100% rename from docs/others/Idea2020.1.md rename to docs/collection/Idea2020.1.md diff --git a/docs/others/JVM-Issue.md b/docs/collection/JVM-Issue.md similarity index 100% rename from docs/others/JVM-Issue.md rename to docs/collection/JVM-Issue.md diff --git "a/docs/others/Java \345\255\246\344\271\240\350\267\257\347\272\277.md" "b/docs/collection/Java \345\255\246\344\271\240\350\267\257\347\272\277.md" similarity index 100% rename from "docs/others/Java \345\255\246\344\271\240\350\267\257\347\272\277.md" rename to "docs/collection/Java \345\255\246\344\271\240\350\267\257\347\272\277.md" diff --git "a/docs/others/Java\344\270\2559\344\270\252\345\244\204\347\220\206Exception\347\232\204\346\234\200\344\275\263\345\256\236\350\267\265.md" "b/docs/collection/Java\344\270\2559\344\270\252\345\244\204\347\220\206Exception\347\232\204\346\234\200\344\275\263\345\256\236\350\267\265.md" similarity index 100% rename from "docs/others/Java\344\270\2559\344\270\252\345\244\204\347\220\206Exception\347\232\204\346\234\200\344\275\263\345\256\236\350\267\265.md" rename to "docs/collection/Java\344\270\2559\344\270\252\345\244\204\347\220\206Exception\347\232\204\346\234\200\344\275\263\345\256\236\350\267\265.md" diff --git "a/docs/collection/MySQL \344\274\230\345\214\226\345\267\245\345\205\267.md" "b/docs/collection/MySQL \344\274\230\345\214\226\345\267\245\345\205\267.md" new file mode 100755 index 0000000000..1ae7dd0a2b --- /dev/null +++ "b/docs/collection/MySQL \344\274\230\345\214\226\345\267\245\345\205\267.md" @@ -0,0 +1,175 @@ +> 对于正在运行的 MySQL，性能如何，参数设置的是否合理，账号设置的是否存在安全隐患，你是否了然于胸呢？ +> +> 俗话说工欲善其事，必先利其器，定期对你的 MySQL 数据库进行一个体检，是保证数据库安全运行的重要手段，因为，好的工具是使你的工作效率倍增！ +> +> 今天和大家分享几个 MySQL 优化的工具，你可以使用它们对你的 MySQL 进行一个体检，生成 awr 报告，让你从整体上把握你的数据库的性能情况。 + +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/00000.png) + + + +## 一、mysqltuner.pl + +是 MySQL 一个常用的数据库性能诊断工具，主要检查参数设置的合理性包括日志文件、存储引擎、安全建议及性能分析。针对潜在的问题，给出改进的建议。是 MySQL 优化的好帮手。 + +在上一版本中，MySQLTuner支持 MySQL / MariaDB / Percona Server 的约 300 个指标。 + +项目地址：[https://github.com/major/MySQLTuner-perl](https://links.jianshu.com/go?to=https%3A%2F%2Fgithub.com%2Fmajor%2FMySQLTuner-perl) + +### 1.1 下载 + +```cpp +[root@localhost ~]#wget https://raw.githubusercontent.com/major/MySQLTuner-perl/master/mysqltuner.pl +``` + +### 1.2 使用 + +```csharp +[root@localhost ~]# ./mysqltuner.pl --socket /var/lib/mysql/mysql.sock + >> MySQLTuner 1.7.4 - Major Hayden + >> Bug reports, feature requests, and downloads at http://mysqltuner.com/ + >> Run with '--help' for additional options and output filtering +[--] Skipped version check for MySQLTuner script +Please enter your MySQL administrative login: root +Please enter your MySQL administrative password: [OK] Currently running supported MySQL version 5.7.23 +[OK] Operating on 64-bit architecture +``` + +### 1.3 报告分析 + +1）重要关注[!!]（中括号有叹号的项）例如[!!] Maximum possible memory usage: 4.8G (244.13% of installed RAM)，表示内存已经严重用超了。 + +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/11111.png) + +2）关注最后给的建议“Recommendations ”。 + +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/22222.png) + +## 二、tuning-primer.sh + +MySQL 的另一个优化工具，针于MySQL的整体进行一个体检，对潜在的问题，给出优化的建议。 + +项目地址：[https://github.com/BMDan/tuning-primer.sh](https://links.jianshu.com/go?to=https%3A%2F%2Fgithub.com%2FBMDan%2Ftuning-primer.sh) + +目前，支持检测和优化建议的内容如下： + +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/33333.png) + +2.1 下载 + +```cpp +[root@localhost ~]#wget https://launchpad.net/mysql-tuning-primer/trunk/1.6-r1/+download/tuning-primer.sh +``` + +2.2 使用 + +```ruby +[root@localhost ~]# [root@localhost dba]# ./tuning-primer.sh + + -- MYSQL PERFORMANCE TUNING PRIMER -- + - By: Matthew Montgomery - +``` + +2.3 报告分析 + +重点查看有红色告警的选项，根据建议结合自己系统的实际情况进行修改，例如： + +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/44444.png) + + + +## 三、pt-variable-advisor + +pt-variable-advisor 可以分析MySQL变量并就可能出现的问题提出建议。 + +3.1 安装 + +[https://www.percona.com/downloads/percona-toolkit/LATEST/](https://links.jianshu.com/go?to=https%3A%2F%2Fwww.percona.com%2Fdownloads%2Fpercona-toolkit%2FLATEST%2F) + +```csharp +[root@localhost ~]#wget https://www.percona.com/downloads/percona-toolkit/3.0.13/binary/redhat/7/x86_64/percona-toolkit-3.0.13-re85ce15-el7-x86_64-bundle.tar +[root@localhost ~]#yum install percona-toolkit-3.0.13-1.el7.x86_64.rpm +``` + +3.2 使用 + +pt-variable-advisor是pt工具集的一个子工具，主要用来诊断你的参数设置是否合理。 + +```csharp +[root@localhost ~]# pt-variable-advisor localhost --socket /var/lib/mysql/mysql.sock +``` + +3.3 报告分析 + +重点关注有WARN的信息的条目，例如： + +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/55555.png) + + + +## 四、pt-qurey-digest + +pt-query-digest 主要功能是从日志、进程列表和tcpdump分析MySQL查询。 + +4.1安装 + +具体参考3.1节 + +4.2使用 + +pt-query-digest主要用来分析mysql的慢日志，与mysqldumpshow工具相比，py-query_digest 工具的分析结果更具体，更完善。 + +```csharp +[root@localhost ~]# pt-query-digest /var/lib/mysql/slowtest-slow.log +``` + +4.3 常见用法分析 + +1）直接分析慢查询文件: + +```cpp +pt-query-digest /var/lib/mysql/slowtest-slow.log > slow_report.log +``` + +2）分析最近12小时内的查询： + +```cpp +pt-query-digest --since=12h /var/lib/mysql/slowtest-slow.log > slow_report2.log +``` + +3）分析指定时间范围内的查询： + +```csharp +pt-query-digest /var/lib/mysql/slowtest-slow.log --since '2017-01-07 09:30:00' --until '2017-01-07 10:00:00'> > slow_report3.log +``` + +4）分析指含有select语句的慢查询 + +```dart +pt-query-digest --filter '$event->{fingerprint} =~ m/^select/i' /var/lib/mysql/slowtest-slow.log> slow_report4.log +``` + +5）针对某个用户的慢查询 + +```dart +pt-query-digest --filter '($event->{user} || "") =~ m/^root/i' /var/lib/mysql/slowtest-slow.log> slow_report5.log +``` + +6）查询所有所有的全表扫描或full join的慢查询 + +```rust +pt-query-digest --filter '(($event->{Full_scan} || "") eq "yes") ||(($event->{Full_join} || "") eq "yes")' /var/lib/mysql/slowtest-slow.log> slow_report6.log +``` + +4.4 报告分析 + +- 第一部分：总体统计结果 Overall：总共有多少条查询 Time range：查询执行的时间范围 unique：唯一查询数量，即对查询条件进行参数化以后，总共有多少个不同的查询 total：总计 min：最小 max：最大 avg：平均 95%：把所有值从小到大排列，位置位于95%的那个数，这个数一般最具有参考价值 median：中位数，把所有值从小到大排列，位置位于中间那个数 +- 第二部分：查询分组统计结果 Rank：所有语句的排名，默认按查询时间降序排列，通过--order-by指定 Query ID：语句的ID，（去掉多余空格和文本字符，计算hash值） Response：总的响应时间 time：该查询在本次分析中总的时间占比 calls：执行次数，即本次分析总共有多少条这种类型的查询语句 R/Call：平均每次执行的响应时间 V/M：响应时间Variance-to-mean的比率 Item：查询对象 +- 第三部分：每一种查询的详细统计结果 ID：查询的ID号，和上图的Query ID对应 Databases：数据库名 Users：各个用户执行的次数（占比） Query_time distribution ：查询时间分布, 长短体现区间占比。Tables：查询中涉及到的表 Explain：SQL语句 + + + +> 来源：https://www.jianshu.com/p/cb2be017d5a9 + +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/end%20(13).jpg) + diff --git "a/docs/collection/Redis \344\270\272\344\273\200\344\271\210\345\217\230\346\205\242\344\272\206.md" "b/docs/collection/Redis \344\270\272\344\273\200\344\271\210\345\217\230\346\205\242\344\272\206.md" new file mode 100644 index 0000000000..7f9e0b6941 --- /dev/null +++ "b/docs/collection/Redis \344\270\272\344\273\200\344\271\210\345\217\230\346\205\242\344\272\206.md" @@ -0,0 +1,234 @@ +Redis 作为内存数据库，拥有非常高的性能，单个实例的QPS能够达到10W左右。但我们在使用Redis时，经常时不时会出现访问延迟很大的情况，如果你不知道Redis的内部实现原理，在排查问题时就会一头雾水。 + +很多时候，Redis出现访问延迟变大，都与我们的使用不当或运维不合理导致的。 + +这篇文章我们就来分析一下Redis在使用过程中，经常会遇到的延迟问题以及如何定位和分析。 + + + +## 使用复杂度高的命令 + +如果在使用Redis时，发现访问延迟突然增大，如何进行排查？ + +首先，第一步，建议你去查看一下Redis的慢日志。Redis提供了慢日志命令的统计功能，我们通过以下设置，就可以查看有哪些命令在执行时延迟比较大。 + +首先设置Redis的慢日志阈值，只有超过阈值的命令才会被记录，这里的单位是微妙，例如设置慢日志的阈值为5毫秒，同时设置只保留最近1000条慢日志记录： + +``` +# 命令执行超过5毫秒记录慢日志 +CONFIG SET slowlog-log-slower-than 5000 +# 只保留最近1000条慢日志 +CONFIG SET slowlog-max-len 1000 +``` + +设置完成之后，所有执行的命令如果延迟大于5毫秒，都会被Redis记录下来，我们执行`SLOWLOG get 5`查询最近5条慢日志： + +``` +127.0.0.1:6379> SLOWLOG get 5 +1) 1) (integer) 32693 # 慢日志ID + 2) (integer) 1593763337 # 执行时间 + 3) (integer) 5299 # 执行耗时(微妙) + 4) 1) "LRANGE" # 具体执行的命令和参数 + 2) "user_list_2000" + 3) "0" + 4) "-1" +2) 1) (integer) 32692 + 2) (integer) 1593763337 + 3) (integer) 5044 + 4) 1) "GET" + 2) "book_price_1000" +... +``` + +通过查看慢日志记录，我们就可以知道在什么时间执行哪些命令比较耗时，如果你的业务**经常使用`O(n)`以上复杂度的命令**，例如`sort`、`sunion`、`zunionstore`，或者在执行`O(n)`命令时操作的数据量比较大，这些情况下Redis处理数据时就会很耗时。 + +如果你的服务请求量并不大，但Redis实例的CPU使用率很高，很有可能是使用了复杂度高的命令导致的。 + +解决方案就是，不使用这些复杂度较高的命令，并且一次不要获取太多的数据，每次尽量操作少量的数据，让Redis可以及时处理返回。 + +## 存储大key + +如果查询慢日志发现，并不是复杂度较高的命令导致的，例如都是`SET`、`DELETE`操作出现在慢日志记录中，那么你就要怀疑是否存在Redis写入了大key的情况。 + +Redis在写入数据时，需要为新的数据分配内存，当从Redis中删除数据时，它会释放对应的内存空间。 + +如果一个key写入的数据非常大，Redis在**分配内存时也会比较耗时**。同样的，当删除这个key的数据时，**释放内存也会耗时比较久**。 + +你需要检查你的业务代码，是否存在写入大key的情况，需要评估写入数据量的大小，业务层应该避免一个key存入过大的数据量。 + +那么有没有什么办法可以扫描现在Redis中是否存在大key的数据吗？ + +Redis也提供了扫描大key的方法： + +``` +redis-cli -h $host -p $port --bigkeys -i 0.01 +``` + +使用上面的命令就可以扫描出整个实例key大小的分布情况，它是以类型维度来展示的。 + +需要注意的是当我们在线上实例进行大key扫描时，Redis的QPS会突增，为了降低扫描过程中对Redis的影响，我们需要控制扫描的频率，使用`-i`参数控制即可，它表示扫描过程中每次扫描的时间间隔，单位是秒。 + +使用这个命令的原理，其实就是Redis在内部执行`scan`命令，遍历所有key，然后针对不同类型的key执行`strlen`、`llen`、`hlen`、`scard`、`zcard`来获取字符串的长度以及容器类型(list/dict/set/zset)的元素个数。 + +而对于容器类型的key，只能扫描出元素最多的key，但元素最多的key不一定占用内存最多，这一点需要我们注意下。不过使用这个命令一般我们是可以对整个实例中key的分布情况有比较清晰的了解。 + +针对大key的问题，Redis官方在4.0版本推出了`lazy-free`的机制，用于异步释放大key的内存，降低对Redis性能的影响。即使这样，我们也不建议使用大key，大key在集群的迁移过程中，也会影响到迁移的性能，这个后面在介绍集群相关的文章时，会再详细介绍到。 + +## 集中过期 + +有时你会发现，平时在使用Redis时没有延时比较大的情况，但在某个时间点突然出现一波延时，而且**报慢的时间点很有规律，例如某个整点，或者间隔多久就会发生一次**。 + +如果出现这种情况，就需要考虑是否存在大量key集中过期的情况。 + +如果有大量的key在某个固定时间点集中过期，在这个时间点访问Redis时，就有可能导致延迟增加。 + +Redis的过期策略采用主动过期+懒惰过期两种策略： + +- 主动过期：Redis内部维护一个定时任务，默认每隔100毫秒会从过期字典中随机取出20个key，删除过期的key，如果过期key的比例超过了25%，则继续获取20个key，删除过期的key，循环往复，直到过期key的比例下降到25%或者这次任务的执行耗时超过了25毫秒，才会退出循环 +- 懒惰过期：只有当访问某个key时，才判断这个key是否已过期，如果已经过期，则从实例中删除 + +注意，**Redis的主动过期的定时任务，也是在Redis主线程中执行的**，也就是说如果在执行主动过期的过程中，出现了需要大量删除过期key的情况，那么在业务访问时，必须等这个过期任务执行结束，才可以处理业务请求。此时就会出现，业务访问延时增大的问题，最大延迟为25毫秒。 + +而且这个访问延迟的情况，**不会记录在慢日志里**。慢日志中**只记录真正执行某个命令的耗时**，Redis主动过期策略执行在操作命令之前，如果操作命令耗时达不到慢日志阈值，它是不会计算在慢日志统计中的，但我们的业务却感到了延迟增大。 + +此时你需要检查你的业务，是否真的存在集中过期的代码，一般集中过期使用的命令是`expireat`或`pexpireat`命令，在代码中搜索这个关键字就可以了。 + +如果你的业务确实需要集中过期掉某些key，又不想导致Redis发生抖动，有什么优化方案？ + +解决方案是，**在集中过期时增加一个随机时间，把这些需要过期的key的时间打散即可。** + +伪代码可以这么写： + +``` +# 在过期时间点之后的5分钟内随机过期掉 +redis.expireat(key, expire_time + random(300)) +``` + +这样Redis在处理过期时，不会因为集中删除key导致压力过大，阻塞主线程。 + +另外，除了业务使用需要注意此问题之外，还可以通过运维手段来及时发现这种情况。 + +做法是我们需要把Redis的各项运行数据监控起来，执行`info`可以拿到所有的运行数据，在这里我们需要重点关注`expired_keys`这一项，它代表整个实例到目前为止，累计删除过期key的数量。 + +我们需要对这个指标监控，当在**很短时间内这个指标出现突增**时，需要及时报警出来，然后与业务报慢的时间点对比分析，确认时间是否一致，如果一致，则可以认为确实是因为这个原因导致的延迟增大。 + +## 实例内存达到上限 + +有时我们把Redis当做纯缓存使用，就会给实例设置一个内存上限`maxmemory`，然后开启LRU淘汰策略。 + +当实例的内存达到了`maxmemory`后，你会发现之后的每次写入新的数据，有可能变慢了。 + +导致变慢的原因是，当Redis内存达到`maxmemory`后，每次写入新的数据之前，必须先踢出一部分数据，让内存维持在`maxmemory`之下。 + +这个踢出旧数据的逻辑也是需要消耗时间的，而具体耗时的长短，要取决于配置的淘汰策略： + +- allkeys-lru：不管key是否设置了过期，淘汰最近最少访问的key +- volatile-lru：只淘汰最近最少访问并设置过期的key +- allkeys-random：不管key是否设置了过期，随机淘汰 +- volatile-random：只随机淘汰有设置过期的key +- allkeys-ttl：不管key是否设置了过期，淘汰即将过期的key +- noeviction：不淘汰任何key，满容后再写入直接报错 +- allkeys-lfu：不管key是否设置了过期，淘汰访问频率最低的key（4.0+支持） +- volatile-lfu：只淘汰访问频率最低的过期key（4.0+支持） + +具体使用哪种策略，需要根据业务场景来决定。 + +我们最常使用的一般是`allkeys-lru`或`volatile-lru`策略，它们的处理逻辑是，每次从实例中随机取出一批key（可配置），然后淘汰一个最少访问的key，之后把剩下的key暂存到一个池子中，继续随机取出一批key，并与之前池子中的key比较，再淘汰一个最少访问的key。以此循环，直到内存降到`maxmemory`之下。 + +如果使用的是`allkeys-random`或`volatile-random`策略，那么就会快很多，因为是随机淘汰，那么就少了比较key访问频率时间的消耗了，随机拿出一批key后直接淘汰即可，因此这个策略要比上面的LRU策略执行快一些。 + +但以上这些逻辑都是在访问Redis时，**真正命令执行之前执行的**，也就是它会影响我们访问Redis时执行的命令。 + +另外，如果此时Redis实例中有存储大key，那么**在淘汰大key释放内存时，这个耗时会更加久，延迟更大**，这需要我们格外注意。 + +如果你的业务访问量非常大，并且必须设置`maxmemory`限制实例的内存上限，同时面临淘汰key导致延迟增大的的情况，要想缓解这种情况，除了上面说的避免存储大key、使用随机淘汰策略之外，也可以考虑**拆分实例**的方法来缓解，拆分实例可以把一个实例淘汰key的压力**分摊到多个实例**上，可以在一定程度降低延迟。 + +## fork耗时严重 + +如果你的Redis开启了自动生成RDB和AOF重写功能，那么有可能在后台生成RDB和AOF重写时导致Redis的访问延迟增大，而等这些任务执行完毕后，延迟情况消失。 + +遇到这种情况，一般就是执行生成RDB和AOF重写任务导致的。 + +生成RDB和AOF都需要父进程`fork`出一个子进程进行数据的持久化，**在`fork`执行过程中，父进程需要拷贝内存页表给子进程，如果整个实例内存占用很大，那么需要拷贝的内存页表会比较耗时，此过程会消耗大量的CPU资源，在完成`fork`之前，整个实例会被阻塞住，无法处理任何请求，如果此时CPU资源紧张，那么`fork`的时间会更长，甚至达到秒级。这会严重影响Redis的性能**。 + +具体原理也可以参考我之前写的文章：[Redis持久化是如何做的？RDB和AOF对比分析](http://kaito-kidd.com/2020/06/29/redis-persistence-rdb-aof/)。 + +我们可以执行`info`命令，查看最后一次`fork`执行的耗时`latest_fork_usec`，单位微妙。这个时间就是整个实例阻塞无法处理请求的时间。 + +除了因为备份的原因生成RDB之外，**在主从节点第一次建立数据同步时**，主节点也会生成RDB文件给从节点进行一次全量同步，这时也会对Redis产生性能影响。 + +要想避免这种情况，我们需要规划好数据备份的周期，建议在**从节点上执行备份，而且最好放在低峰期执行**。如果对于丢失数据不敏感的业务，那么不建议开启AOF和AOF重写功能。 + +另外，`fork`的耗时也与系统有关，如果把Redis部署在虚拟机上，那么这个时间也会增大。所以使用Redis时建议部署在物理机上，降低`fork`的影响。 + +## 绑定CPU + +很多时候，我们在部署服务时，为了提高性能，降低程序在使用多个CPU时上下文切换的性能损耗，一般会采用进程绑定CPU的操作。 + +但在使用Redis时，我们不建议这么干，原因如下。 + +**绑定CPU的Redis，在进行数据持久化时，`fork`出的子进程，子进程会继承父进程的CPU使用偏好，而此时子进程会消耗大量的CPU资源进行数据持久化，子进程会与主进程发生CPU争抢，这也会导致主进程的CPU资源不足访问延迟增大。** + +所以在部署Redis进程时，如果需要开启RDB和AOF重写机制，一定不能进行CPU绑定操作！ + +## 开启AOF + +上面提到了，当执行AOF文件重写时会因为`fork`执行耗时导致Redis延迟增大，除了这个之外，如果开启AOF机制，设置的策略不合理，也会导致性能问题。 + +开启AOF后，Redis会把写入的命令实时写入到文件中，但写入文件的过程是先写入内存，等内存中的数据超过一定阈值或达到一定时间后，内存中的内容才会被真正写入到磁盘中。 + +AOF为了保证文件写入磁盘的安全性，提供了3种刷盘机制： + +- `appendfsync always`：每次写入都刷盘，对性能影响最大，占用磁盘IO比较高，数据安全性最高 +- `appendfsync everysec`：1秒刷一次盘，对性能影响相对较小，节点宕机时最多丢失1秒的数据 +- `appendfsync no`：按照操作系统的机制刷盘，对性能影响最小，数据安全性低，节点宕机丢失数据取决于操作系统刷盘机制 + +当使用第一种机制`appendfsync always`时，Redis每处理一次写命令，都会把这个命令写入磁盘，而且**这个操作是在主线程中执行的**。 + +内存中的的数据写入磁盘，这个会加重磁盘的IO负担，操作磁盘成本要比操作内存的代价大得多。如果写入量很大，那么每次更新都会写入磁盘，此时机器的磁盘IO就会非常高，拖慢Redis的性能，因此我们不建议使用这种机制。 + +与第一种机制对比，`appendfsync everysec`会每隔1秒刷盘，而`appendfsync no`取决于操作系统的刷盘时间，安全性不高。因此我们推荐使用`appendfsync everysec`这种方式，在最坏的情况下，只会丢失1秒的数据，但它能保持较好的访问性能。 + +当然，对于有些业务场景，对丢失数据并不敏感，也可以不开启AOF。 + +## 使用Swap + +如果你发现Redis突然变得非常慢，**每次访问的耗时都达到了几百毫秒甚至秒级**，那此时就检查Redis是否使用到了Swap，这种情况下Redis基本上已经无法提供高性能的服务。 + +我们知道，操作系统提供了Swap机制，目的是为了当内存不足时，可以把一部分内存中的数据换到磁盘上，以达到对内存使用的缓冲。 + +但当内存中的数据被换到磁盘上后，访问这些数据就需要从磁盘中读取，这个速度要比内存慢太多！ + +**尤其是针对Redis这种高性能的内存数据库来说，如果Redis中的内存被换到磁盘上，对于Redis这种性能极其敏感的数据库，这个操作时间是无法接受的。** + +我们需要检查机器的内存使用情况，确认是否确实是因为内存不足导致使用到了Swap。 + +如果确实使用到了Swap，要及时整理内存空间，释放出足够的内存供Redis使用，然后释放Redis的Swap，让Redis重新使用内存。 + +释放Redis的Swap过程通常要重启实例，为了避免重启实例对业务的影响，一般先进行主从切换，然后释放旧主节点的Swap，重新启动服务，待数据同步完成后，再切换回主节点即可。 + +可见，当Redis使用到Swap后，此时的Redis的高性能基本被废掉，所以我们需要提前预防这种情况。 + +**我们需要对Redis机器的内存和Swap使用情况进行监控，在内存不足和使用到Swap时及时报警出来，及时进行相应的处理。** + +## 网卡负载过高 + +如果以上产生性能问题的场景，你都规避掉了，而且Redis也稳定运行了很长时间，但在某个时间点之后开始，访问Redis开始变慢了，而且一直持续到现在，这种情况是什么原因导致的？ + +之前我们就遇到这种问题，**特点就是从某个时间点之后就开始变慢，并且一直持续**。这时你需要检查一下机器的网卡流量，是否存在网卡流量被跑满的情况。 + +**网卡负载过高，在网络层和TCP层就会出现数据发送延迟、数据丢包等情况。Redis的高性能除了内存之外，就在于网络IO，请求量突增会导致网卡负载变高。** + +如果出现这种情况，你需要排查这个机器上的哪个Redis实例的流量过大占满了网络带宽，然后确认流量突增是否属于业务正常情况，如果属于那就需要及时扩容或迁移实例，避免这个机器的其他实例受到影响。 + +运维层面，我们需要对机器的各项指标增加监控，包括网络流量，在达到阈值时提前报警，及时与业务确认并扩容。 + +## 总结 + +以上我们总结了Redis中常见的可能导致延迟增大甚至阻塞的场景，这其中既涉及到了业务的使用问题，也涉及到Redis的运维问题。 + +可见，要想保证Redis高性能的运行，其中涉及到CPU、内存、网络，甚至磁盘的方方面面，其中还包括操作系统的相关特性的使用。 + +作为开发人员，我们需要了解Redis的运行机制，例如各个命令的执行时间复杂度、数据过期策略、数据淘汰策略等，使用合理的命令，并结合业务场景进行优化。 + +作为DBA运维人员，需要了解数据持久化、操作系统`fork`原理、Swap机制等，并对Redis的容量进行合理规划，预留足够的机器资源，对机器做好完善的监控，才能保证Redis的稳定运行。 \ No newline at end of file diff --git "a/docs/collection/Spring Boot \351\233\206\346\210\220 JUnit5.md" "b/docs/collection/Spring Boot \351\233\206\346\210\220 JUnit5.md" new file mode 100755 index 0000000000..a8aef82090 --- /dev/null +++ "b/docs/collection/Spring Boot \351\233\206\346\210\220 JUnit5.md" @@ -0,0 +1,171 @@ +### 为什么使用JUnit5 + +- JUnit4 被广泛使用，但是许多场景下使用起来语法较为繁琐，JUnit5 中支持 lambda 表达式，语法简单且代码不冗余。 +- JUnit5 易扩展，包容性强，可以接入其他的测试引擎。 +- 功能更强大提供了新的断言机制、参数化测试、重复性测试等新功能。 +- ps：开发人员为什么还要测试，单测写这么规范有必要吗？其实单测是开发人员必备技能，只不过很多开发人员开发任务太重导致调试完就不管了，没有系统化得单元测试，单元测试在系统重构时能发挥巨大的作用，可以在重构后快速测试新的接口是否与重构前有出入。 + +### 简介 + +![](https://img2020.cnblogs.com/blog/1543774/202010/1543774-20201013233417780-386312048.png) + +如图，JUnit5 结构如下： + +- **JUnit Platform**： 这是Junit提供的平台功能模块，通过它，其它的测试引擎都可以接入Junit实现接口和执行。 +- **JUnit JUpiter**：这是JUnit5的核心，是一个基于JUnit Platform的引擎实现，它包含许多丰富的新特性来使得自动化测试更加方便和强大。 +- **JUnit Vintage**：这个模块是兼容JUnit3、JUnit4版本的测试引擎，使得旧版本的自动化测试也可以在JUnit5下正常运行。 + +### 依赖引入 + +我们以 `SpringBoot2.3.1 `为例，引入如下依赖，防止使用旧的 junit4 相关接口我们将其依赖排除。 + +```xml + + org.springframework.boot + spring-boot-starter-test + test + + + org.junit.vintage + junit-vintage-engine + + + +``` + +### 常用注解 + +- @BeforeEach：在每个单元测试方法执行前都执行一遍 +- @BeforeAll：在每个单元测试方法执行前执行一遍（只执行一次） +- @DisplayName("商品入库测试")：用于指定单元测试的名称 +- @Disabled：当前单元测试置为无效，即单元测试时跳过该测试 +- @RepeatedTest(n)：重复性测试，即执行n次 +- @ParameterizedTest：参数化测试， +- @ValueSource(ints = {1, 2, 3})：参数化测试提供数据 + +### 断言 + +JUnit Jupiter 提供了强大的断言方法用以验证结果，在使用时需要借助 java8 的新特性 lambda 表达式，均是来自`org.junit.jupiter.api.Assertions `包的 `static` 方法。 + +`assertTrue` 与 `assertFalse` 用来判断条件是否为 `true` 或 `false` + +```java +@Test +@DisplayName("测试断言equals") +void testEquals() { + assertTrue(3 < 4); +} +``` + +`assertNull` 与 `assertNotNull` 用来判断条件是否为 `null` + +```java +@Test +@DisplayName("测试断言NotNull") +void testNotNull() { + assertNotNull(new Object()); +} +``` + +`assertThrows` 用来判断执行抛出的异常是否符合预期，并可以使用异常类型接收返回值进行其他操作 + +```java +@Test +@DisplayName("测试断言抛异常") +void testThrows() { + ArithmeticException arithExcep = assertThrows(ArithmeticException.class, () -> { + int m = 5/0; + }); + assertEquals("/ by zero", arithExcep.getMessage()); +} +``` + +`assertTimeout`用来判断执行过程是否超时 + +```java +@Test +@DisplayName("测试断言超时") +void testTimeOut() { + String actualResult = assertTimeout(ofSeconds(2), () -> { + Thread.sleep(1000); + return "a result"; + }); + System.out.println(actualResult); +} +``` + +`assertAll` 是组合断言，当它内部所有断言正确执行完才算通过 + +```java +@Test +@DisplayName("测试组合断言") +void testAll() { + assertAll("测试item商品下单", + () -> { + //模拟用户余额扣减 + assertTrue(1 < 2, "余额不足"); + }, + () -> { + //模拟item数据库扣减库存 + assertTrue(3 < 4); + }, + () -> { + //模拟交易流水落库 + assertNotNull(new Object()); + } + ); +} +``` + +### 重复性测试 + +在许多场景中我们需要对同一个接口方法进行重复测试，例如对幂等性接口的测试。 + +JUnit Jupiter 通过使用 `@RepeatedTest(n)` 指定需要重复的次数 + +```java +@RepeatedTest(3) +@DisplayName("重复测试") +void repeatedTest() { + System.out.println("调用"); +} +``` + +![](https://img2020.cnblogs.com/blog/1543774/202010/1543774-20201013233431933-1368900431.png) + +### 参数化测试 + +参数化测试可以按照多个参数分别运行多次单元测试这里有点类似于重复性测试，只不过每次运行传入的参数不用。需要使用到 `@ParameterizedTest`，同时也需要 `@ValueSource` 提供一组数据，它支持八种基本类型以及 `String` 和自定义对象类型，使用极其方便。 + +```java +@ParameterizedTest +@ValueSource(ints = {1, 2, 3}) +@DisplayName("参数化测试") +void paramTest(int a) { + assertTrue(a > 0 && a < 4); +} +``` + +### 内嵌测试 + +JUnit5 提供了嵌套单元测试的功能，可以更好展示测试类之间的业务逻辑关系，我们通常是一个业务对应一个测试类，有业务关系的类其实可以写在一起。这样有利于进行测试。而且内联的写法可以大大减少不必要的类，精简项目，防止类爆炸等一系列问题。 + +```java +@SpringBootTest +@AutoConfigureMockMvc +@DisplayName("Junit5单元测试") +public class MockTest { + //.... + @Nested + @DisplayName("内嵌订单测试") + class OrderTestClas { + @Test + @DisplayName("取消订单") + void cancelOrder() { + int status = -1; + System.out.println("取消订单成功,订单状态为:"+status); + } + } +} +``` + diff --git "a/docs/others/fastjson\346\274\217\346\264\236.md" "b/docs/collection/fastjson\346\274\217\346\264\236.md" similarity index 100% rename from "docs/others/fastjson\346\274\217\346\264\236.md" rename to "docs/collection/fastjson\346\274\217\346\264\236.md" diff --git a/docs/.nojekyll "b/docs/collection/feed/\346\234\252\345\221\275\345\220\215.md" old mode 100644 new mode 100755 similarity index 100% rename from docs/.nojekyll rename to "docs/collection/feed/\346\234\252\345\221\275\345\220\215.md" diff --git "a/docs/collection/google-api\350\256\276\350\256\241\346\214\207\345\215\227.md" "b/docs/collection/google-api\350\256\276\350\256\241\346\214\207\345\215\227.md" new file mode 100644 index 0000000000..2739260264 --- /dev/null +++ "b/docs/collection/google-api\350\256\276\350\256\241\346\214\207\345\215\227.md" @@ -0,0 +1,1767 @@ +# API 设计指南 + +## 简介 + +这是联网 API 的通用设计指南。它自 2014 年起在 Google 内部使用，是 Google 在设计 [Cloud API](https://cloud.google.com/apis/docs/overview?hl=zh-cn) 和其他 [Google API](https://github.com/googleapis/googleapis) 时遵循的指南。我们在此公开此设计指南，目的是为外部开发者提供信息，使我们所有人都能更轻松地协同工作。 + +[Cloud Endpoints](https://cloud.google.com/endpoints/docs/grpc?hl=zh-cn) 开发者可能会发现本指南在设计 gRPC API 时特别有用，我们强烈建议此类开发者遵循这些设计原则。不过，我们并不强制要求使用本指南。您可以使用 Cloud Endpoints 和 gRPC，而无需遵循本指南。 + +本指南同时适用于 REST API 和 RPC API，尤其适用于 gRPC API。gRPC API 使用 [Protocol Buffers](https://cloud.google.com/apis/design/proto3?hl=zh-cn) 定义其 API 表面 (surface) 和 [API 服务配置](https://github.com/googleapis/googleapis)，以配置其 API 服务，包括 HTTP 映射、日志记录和监控。 Google API 和 Cloud Endpoints gRPC API 使用 HTTP 映射功能进行 JSON/HTTP 到 Protocol Buffers/RPC 的[转码](https://cloud.google.com/endpoints/docs/transcoding?hl=zh-cn)。 + +本指南是一份活文档，随着时间的推移，我们会采纳和批准新的风格和设计模式，为本指南增加相关内容。本着这种精神，我们会不断完善本指南，并为 API 设计的艺术和技巧提供充足的空间。 + +## 本文档中使用的惯例 + +本文档中使用的要求级别关键字（“必须”、“不得”、“必需”，“应”、“不应”、“应该”、“不应该”、“建议”、“可以”和“可选”）将按 [RFC 2119 ](https://www.ietf.org/rfc/rfc2119.txt)中的描述进行解释。 + +在本文档中，这些关键字使用**粗体**突出显示。 + + + +# 基于资源的设计 + +本设计指南的目标是帮助开发者设计**简单、一致且易用**的网络 API。同时，它还有助于将 RPC API（基于套接字）与 REST API（基于 HTTP）的设计融合起来。 + +RPC API 通常根据接口和方法设计。随着时间的推移，接口和方法越来越多，最终结果可能是形成一个庞大而混乱的 API 接口，因为开发者必须单独学习每种方法。显然，这既耗时又容易出错。 + +引入 [REST](http://en.wikipedia.org/wiki/Representational_state_transfer) 架构风格主要是为了与 HTTP/1.1 配合使用，但也有助于解决这个问题。其核心原则是定义可以用少量方法控制的命名资源。这些资源和方法被称为 API 的“名词”和“动词”。使用 HTTP 协议时，资源名称自然映射到网址，方法自然映射到 HTTP 的 `POST`、`GET`、`PUT`、`PATCH` 和 `DELETE`。这使得要学习的内容减少了很多，因为开发人员可以专注于资源及其关系，并假定它们拥有的标准方法同样很少。 + +近来，HTTP REST API 在互联网上取得了巨大成功。2010 年，大约 74％ 的公共网络 API 是 HTTP REST API。 + +虽然 HTTP REST API 在互联网上非常流行，但它们承载的流量比传统的 RPC API 要小。例如，美国高峰时段大约一半的互联网流量是视频内容，显然出于性能考虑，很少有人会使用 REST API 来传送此类内容。在数据中心内，许多公司使用基于套接字的 RPC API 来承载大多数网络流量，这可能涉及比公共 REST API 高几个数量级的数据（以字节为单位）。 + +在实际使用中，人们会出于不同目的选择 RPC API 和 HTTP REST API，理想情况下，API 平台应该为所有类型的 API 提供最佳支持。本设计指南可帮助您设计和构建符合此原则的 API。它将面向资源的设计原则应用于通用 API 设计并定义了许多常见的设计模式，从而提高可用性并降低复杂性。 + +**注意**：本设计指南介绍了如何将 REST 原则应用于 API 设计，与编程语言、操作系统或网络协议无关。这不仅仅是一个创建 REST API 的指南。 + +## 什么是 REST API？ + +REST API 是可单独寻址的“资源”（API 中的“名词”）的“集合”。资源通过[资源名称](https://cloud.google.com/apis/design/resource_names?hl=zh-cn)被引用，并通过一组“方法”（也称为“动词”或“操作”）进行控制。 + +REST Google API 的标准方法（也称为“REST 方法”）包括 `List`、`Get`、`Create`、`Update` 和 `Delete`。API 设计者还可以使用“自定义方法”（也称为“自定义动词”或“自定义操作”）来实现无法轻易映射到标准方法的功能（例如数据库事务）。 + +**注意**：自定义动词并不意味着创建自定义 HTTP 动词来支持自定义方法。对基于 HTTP 的 API 而言，它们只是映射到最合适的 HTTP 动词。 + +## 设计流程 + +设计指南建议在设计面向资源的 API 时采取以下步骤（更多细节将在下面的特定部分中介绍）： + +- 确定 API 提供的资源类型。 +- 确定资源之间的关系。 +- 根据类型和关系确定资源名称方案。 +- 确定资源架构。 +- 将最小的方法集附加到资源。 + +## 资源 + +面向资源的 API 通常被构建为资源层次结构，其中每个节点是一个“简单资源”或“集合资源”。 为方便起见，它们通常被分别称为资源和集合。 + +- 一个集合包含**相同类型**的资源列表。 例如，一个用户拥有一组联系人。 +- 资源具有一些状态和零个或多个子资源。 每个子资源可以是一个简单资源或一个集合资源。 + +例如，Gmail API 有一组用户，每个用户都有一组消息、一组线程、一组标签、一个个人资料资源和若干设置资源。 + +虽然存储系统和 REST API 之间存在一些概念上的对应，但具有面向资源 API 的服务不一定是数据库，并且在解释资源和方法方面具有极大的灵活性。例如，创建日历事件（资源）可以为参与者创建附加事件、向参与者发送电子邮件邀请、预约会议室以及更新视频会议时间安排。 + +## 方法 + +面向资源的 API 的关键特性是，强调资源（数据模型）甚于资源上执行的方法（功能）。典型的面向资源的 API 使用少量方法公开大量资源。方法可以是标准方法或自定义方法。对于本指南，标准方法有：`List`、`Get`、`Create`、`Update` 和 `Delete`。 + +如果 API 功能能够自然映射到标准方法，则**应该**在 API 设计中使用该方法。对于不会自然映射到某一标准方法的功能，**可以**使用自定义方法。[自定义方法](https://cloud.google.com/apis/design/custom_methods?hl=zh-cn)提供与传统 RPC API 相同的设计自由度，可用于实现常见的编程模式，例如数据库事务或数据分析。 + +## 示例 + +以下部分介绍了如何将面向资源的 API 设计应用于大规模服务的一些实际示例。您可以在 [Google API](https://github.com/googleapis/googleapis) 代码库中找到更多示例。 + +在这些示例中，星号表示列表中的一个特定资源。 + +### Gmail API + +Gmail API 服务实现了 Gmail API 并公开了大多数 Gmail 功能。它具有以下资源模型： + +- API 服务： + + ``` + gmail.googleapis.com + ``` + + - 用户集合： + + ``` + users/* + ``` + + 。每个用户都拥有以下资源。 + + - 消息集合：`users/*/messages/*`。 + - 线程集合：`users/*/threads/*`。 + - 标签集合：`users/*/labels/*`。 + - 变更历史记录集合：`users/*/history/*`。 + - 表示用户个人资料的资源：`users/*/profile`。 + - 表示用户设置的资源：`users/*/settings`。 + +### Cloud Pub/Sub API + +`pubsub.googleapis.com` 服务实现了 [Cloud Pub/Sub AP](https://cloud.google.com/pubsub?hl=zh-cn)，后者定义以下资源模型： + +- API 服务： + + ``` + pubsub.googleapis.com + ``` + + - 主题集合：`projects/*/topics/*`。 + - 订阅集合：`projects/*/subscriptions/*`。 + +**注意**：Pub/Sub API 的其他实现可以选择不同的资源命名方案。 + +### Cloud Spanner API + +`spanner.googleapis.com` 服务实现了 [Cloud Spanner API](https://cloud.google.com/spanner?hl=zh-cn)，后者定义了以下资源模型： + +- API 服务： + + ``` + spanner.googleapis.com + ``` + + - 实例集合： + + ``` + projects/*/instances/* + ``` + + 。 + + - 实例操作的集合：`projects/*/instances/*/operations/*`。 + - 数据库的集合：`projects/*/instances/*/databases/*`。 + - 数据库操作的集合：`projects/*/instances/*/databases/*/operations/*`。 + - 数据库会话的集合：`projects/*/instances/*/databases/*/sessions/*`。 + + + + + +# 资源名称 + +在面向资源的 API 中，“资源”是被命名的实体，“资源名称”是它们的标识符。每个资源都**必须**具有自己唯一的资源名称。 资源名称由资源自身的 ID、任何父资源的 ID 及其 API 服务名称组成。在下文中，我们将查看资源 ID 以及如何构建资源名称。 + +gRPC API 应使用无传输协议的 [URI](http://tools.ietf.org/html/rfc3986) 作为资源名称。它们通常遵循 REST 网址规则，其行为与网络文件路径非常相似。它们可以轻松映射到 REST 网址：如需了解详情，请参阅[标准方法](https://cloud.google.com/apis/design/standard_methods?hl=zh-cn)部分。 + +“集合”是一种特殊的资源，包含相同类型的子资源列表。例如，目录是文件资源的集合。集合的资源 ID 称为集合 ID。 + +资源名称由集合 ID 和资源 ID 构成，按分层方式组织并以正斜杠分隔。如果资源包含子资源，则子资源的名称由父资源名称后跟子资源的 ID 组成，也以正斜杠分隔。 + +示例 1：存储服务具有一组 `buckets`，其中每个存储分区都有一组 `objects`： + +| API 服务名称 | 集合 ID | 资源 ID | 集合 ID | 资源 ID | +| :----------------------- | :------- | :--------- | :------- | :--------- | +| //storage.googleapis.com | /buckets | /bucket-id | /objects | /object-id | + +示例 2：电子邮件服务具有一组 `users`。每个用户都有一个 `settings` 子资源，而 `settings` 子资源拥有包括 `customFrom` 在内的许多其他子资源： + +| API 服务名称 | 集合 ID | 资源 ID | 资源 ID | 资源 ID | +| :-------------------- | :------ | :---------------- | :-------- | :---------- | +| //mail.googleapis.com | /users | /name@example.com | /settings | /customFrom | + +API 生产者可以为资源和集合 ID 选择任何可接受的值，只要它们在资源层次结构中是唯一的。您可以在下文中找到有关选择适当的资源和集合 ID 的更多准则。 + +通过拆分资源名称（例如 `name.split("/")[n]`），可以获得单个集合 ID 和资源 ID（假设任何段都不包含正斜杠）。 + +## 完整资源名称 + +无传输协议的 [URI](http://tools.ietf.org/html/rfc3986) 由 [DNS 兼容的](http://tools.ietf.org/html/rfc1035) API 服务名称和资源路径组成。资源路径也称为“相对资源名称”。 例如： + +``` +"//library.googleapis.com/shelves/shelf1/books/book2" +``` + +API 服务名称供客户端定位 API 服务端点；它**可以**是仅限内部服务的虚构 DNS 名称。如果 API 服务名称在上下文中很明显，则通常使用相对资源名称。 + +## 相对资源名称 + +开头没有“/”的 URI 路径 ([path-noscheme](http://tools.ietf.org/html/rfc3986#appendix-A)）。它标识 API 服务中的资源。例如： + +``` +"shelves/shelf1/books/book2" +``` + +## 资源 ID + +标识其父资源中资源的非空 URI 段 ([segment-nz-nc](http://tools.ietf.org/html/rfc3986#appendix-A))，请参见上文的示例。 + +资源名称末尾的资源 ID **可以**具有多个 URI 段。例如： + +| 集合 ID | 资源 ID | +| :------ | :------------------- | +| files | /source/py/parser.py | + +API 服务**应该**尽可能使用网址友好的资源 ID。 资源 ID **必须**被清楚地记录，无论它们是由客户端、服务器还是其中一个分配的。例如，文件名通常由客户端分配，而电子邮件消息 ID 通常由服务器分配。 + +## 集合 ID + +标识其父资源中集合资源的非空 URI 段 ([segment-nz-nc](http://tools.ietf.org/html/rfc3986#appendix-A))，请参见上文的示例。 + +由于集合 ID 通常出现在生成的客户端库中，因此它们**必须**符合以下要求： + +- **必须**是有效的 C/C++ 标识符。 + +- **必须**是复数形式的首字母小写驼峰体。如果该词语没有合适的复数形式，例如“evidence（证据）”和“weather（天气）”，则**应该**使用单数形式。 + +- **必须**使用简明扼要的英文词语。 + +- 应该 + + 避免过于笼统的词语，或对其进行限定后再使用。例如， + + ``` + rowValues + ``` + + + + 优先于 + + + + ``` + values + ``` + + 。 + + 应该 + + 避免在不加以限定的情况下使用以下词语： + + - elements + - entries + - instances + - items + - objects + - resources + - types + - values + +## 资源名称和网址 + +虽然完整的资源名称类似于普通网址，但两者并不相同。单个资源可以由不同的 API 版本、API 协议或 API 网络端点公开。完整资源名称未指明此类信息，因此在实际使用中必须将其映射到特定的 API 版本和 API 协议。 + +要通过 REST API 使用完整资源名称，**必须**将其转换为 REST 网址，实现方法为在服务名称之前添加 HTTPS 传输协议、在资源路径之前添加 API 主要版本以及对资源路径进行网址转义。例如： + +``` +// This is a calendar event resource name. +"//calendar.googleapis.com/users/john smith/events/123" + +// This is the corresponding HTTP URL. +"https://calendar.googleapis.com/v3/users/john%20smith/events/123" +``` + +## 资源名称为字符串 + +除非存在向后兼容问题，否则 Google API **必须**使用纯字符串来表示资源名称。资源名称**应该**像普通文件路径一样处理，并且它们不支持 % 编码。 + +对于资源定义，第一个字段**应该**是资源名称的字符串字段，并且**应该**称为 `name`。 + +**注意**：以下代码示例使用 [gRPC 转码](https://github.com/googleapis/googleapis/blob/master/google/api/http.proto)语法。请点击链接以查看详细信息。 + +例如： + +```proto +service LibraryService { + rpc GetBook(GetBookRequest) returns (Book) { + option (google.api.http) = { + get: "/v1/{name=shelves/*/books/*}" + }; + }; + rpc CreateBook(CreateBookRequest) returns (Book) { + option (google.api.http) = { + post: "/v1/{parent=shelves/*}/books" + body: "book" + }; + }; +} + +message Book { + // Resource name of the book. It must have the format of "shelves/*/books/*". + // For example: "shelves/shelf1/books/book2". + string name = 1; + + // ... other properties +} + +message GetBookRequest { + // Resource name of a book. For example: "shelves/shelf1/books/book2". + string name = 1; +} + +message CreateBookRequest { + // Resource name of the parent resource where to create the book. + // For example: "shelves/shelf1". + string parent = 1; + // The Book resource to be created. Client must not set the `Book.name` field. + Book book = 2; +} +``` + +**注意**：为了保证资源名称的一致性，前导正斜杠**不得**被任何网址模板变量捕获。例如，**必须**使用网址模板 `"/v1/{name=shelves/*/books/*}"`，而非 `"/v1{name=/shelves/*/books/*}"`。 + +## 问题 + +### 问：为什么不使用资源 ID 来标识资源？ + +答：任何大型系统都有很多种资源。在使用资源 ID 来标识资源的时候，我们实际上是使用特定于资源的元组来标识资源，例如 `(bucket, object)` 或 `(user, album, photo)`。这会带来几个主要问题： + +- 开发者必须了解并记住这些匿名元组。 +- 传递元组通常比传递字符串更难。 +- 集中式基础架构（例如日志记录和访问控制系统）不理解专用元组。 +- 专用元组限制了 API 设计的灵活性，例如提供可重复使用的 API 接口。例如，[长时间运行的操作](https://github.com/googleapis/googleapis/tree/master/google/longrunning)可以与许多其他 API 接口一起使用，因为它们使用灵活的资源名称。 + +### 问：为什么特殊字段名为 name 而不是 id？ + +答：特殊字段以资源“名称”的概念命名。一般来说，我们发现 `name` 的概念让开发者感到困惑。例如，文件名实际上只是名称还是完整路径？通过预留标准字段 `name`，开发者不得不选择更合适的词语，例如 `display_name` 或 `title` 或 `full_name`。 + + + +# 标准方法 + +本章定义了标准方法（即 `List`、`Get`、`Create`、`Update` 和 `Delete` 的概念。标准方法可降低复杂性并提高一致性。[Google API](https://github.com/googleapis/googleapis) 代码库中超过 70％ 的 API 方法都是标准方法，这使得它们更易于学习和使用。 + +下表描述了如何将标准方法映射到 HTTP 方法： + +| 标准方法 | HTTP 映射 | HTTP 请求正文 | HTTP 响应正文 | +| :----------------------------------------------------------- | :---------------------------- | :------------ | :------------------------ | +| [`List`](https://cloud.google.com/apis/design/standard_methods?hl=zh-cn#list) | `GET ` | 无 | 资源*列表 | +| [`Get`](https://cloud.google.com/apis/design/standard_methods?hl=zh-cn#get) | `GET ` | 无 | 资源* | +| [`Create`](https://cloud.google.com/apis/design/standard_methods?hl=zh-cn#create) | `POST ` | 资源 | 资源* | +| [`Update`](https://cloud.google.com/apis/design/standard_methods?hl=zh-cn#update) | `PUT or PATCH ` | 资源 | 资源* | +| [`Delete`](https://cloud.google.com/apis/design/standard_methods?hl=zh-cn#delete) | `DELETE ` | 不适用 | `google.protobuf.Empty`** | + +*如果方法支持响应字段掩码以指定要返回的字段子集，则 `List`、`Get`、`Create` 和 `Update` 方法返回的资源**可能**包含部分数据。在某些情况下，API 平台对所有方法的字段掩码提供原生支持。 + +**从不立即移除资源的 `Delete` 方法（例如更新标志或创建长时间运行的删除操作）返回的响应**应该**包含长时间运行的操作或修改后的资源。 + +对于在单个 API 调用的时间跨度内未完成的请求，标准方法还**可以**返回[长时间运行的操作](https://github.com/googleapis/googleapis/blob/master/google/longrunning/operations.proto)。 + +以下部分详细描述了每种标准方法。 这些示例显示了.proto 文件中定义的方法和 HTTP 映射的特殊注释。您可以在 [Google API](https://github.com/googleapis/googleapis) 代码库中找到许多使用标准方法的示例。 + +## 列出 + +`List` 方法将一个集合名称和零个或多个参数作为输入，并返回与输入匹配的资源列表。 + +`List` 通常用于搜索资源。`List` 适用于来自单个集合的数据，该集合的大小有限且不进行缓存。对于更广泛的情况，**应该**使用[自定义方法](https://cloud.google.com/apis/design/custom_methods?hl=zh-cn) `Search`。 + +批量 get（例如，获取多个资源 ID 并为每个 ID 返回对象的方法）**应该**被实现为自定义 `BatchGet` 方法，而不是 `List` 方法。但是，如果您有一个已经存在的可提供相同功能的 `List` 方法，**可以**出于此目的重复使用 `List` 方法。如果您使用的是自定义 `BatchGet` 方法，则**应该**将其映射到 `HTTP GET`。 + +适用的常见模式：[分页](https://cloud.google.com/apis/design/design_patterns?hl=zh-cn#list_pagination)、[结果排序](https://cloud.google.com/apis/design/design_patterns?hl=zh-cn#sorting_order)。 + +适用的命名规则：[过滤字段](https://cloud.google.com/apis/design/naming_convention?hl=zh-cn#list_filter_field)、[结果字段](https://cloud.google.com/apis/design/naming_convention?hl=zh-cn#list_response)。 + +HTTP 映射： + +- `List` 方法 **必须**使用 HTTP `GET` 动词。 +- 接收其资源正在列出的集合名称的请求消息字段**应该**映射到网址路径。如果集合名称映射到网址路径，则网址模板的最后一段（[集合 ID](https://cloud.google.com/apis/design/resource_names?hl=zh-cn#CollectionId)）**必须**是字面量。 +- 所有剩余的请求消息字段**应该**映射到网址查询参数。 +- 没有请求正文，API 配置**不得**声明 `body` 子句。 +- 响应正文**应该**包含资源列表以及可选元数据。 + +示例： + +```proto +// Lists books in a shelf. +rpc ListBooks(ListBooksRequest) returns (ListBooksResponse) { + // List method maps to HTTP GET. + option (google.api.http) = { + // The `parent` captures the parent resource name, such as "shelves/shelf1". + get: "/v1/{parent=shelves/*}/books" + }; +} + +message ListBooksRequest { + // The parent resource name, for example, "shelves/shelf1". + string parent = 1; + + // The maximum number of items to return. + int32 page_size = 2; + + // The next_page_token value returned from a previous List request, if any. + string page_token = 3; +} + +message ListBooksResponse { + // The field name should match the noun "books" in the method name. There + // will be a maximum number of items returned based on the page_size field + // in the request. + repeated Book books = 1; + + // Token to retrieve the next page of results, or empty if there are no + // more results in the list. + string next_page_token = 2; +} +``` + +## 获取 + +`Get` 方法需要一个资源名称和零个或多个参数作为输入，并返回指定的资源。 + +HTTP 映射： + +- `Get` 方法 **必须**使用 HTTP `GET` 动词。 +- 接收资源名称的请求消息字段**应该**映射到网址路径。 +- 所有剩余的请求消息字段**应该**映射到网址查询参数。 +- 没有请求正文，API 配置**不得**声明 `body` 子句。 +- 返回的资源**应该**映射到整个响应正文。 + +示例： + +```proto +// Gets a book. +rpc GetBook(GetBookRequest) returns (Book) { + // Get maps to HTTP GET. Resource name is mapped to the URL. No body. + option (google.api.http) = { + // Note the URL template variable which captures the multi-segment resource + // name of the requested book, such as "shelves/shelf1/books/book2" + get: "/v1/{name=shelves/*/books/*}" + }; +} + +message GetBookRequest { + // The field will contain name of the resource requested, for example: + // "shelves/shelf1/books/book2" + string name = 1; +} +``` + +## 创建 + +`Create` 方法需要一个父资源名称、一个资源以及零个或多个参数作为输入。它在指定的父资源下创建新资源，并返回新建的资源。 + +如果 API 支持创建资源，则**应该**为每一个可以创建的资源类型设置 `Create` 方法。 + +HTTP 映射： + +- `Create` 方法 **必须**使用 HTTP `POST` 动词。 +- 请求消息**应该**具有字段 `parent`，以指定要在其中创建资源的父资源名称。 +- 包含资源的请求消息字段**必须**映射到请求正文。如果将 `google.api.http` 注释用于 `Create` 方法，则**必须**使用 `body: ""` 表单。 +- 该请求**可以**包含名为 `_id` 的字段，以允许调用者选择客户端分配的 ID。该字段**可以**在资源内。 +- 所有剩余的请求消息字段**应该**映射到网址查询参数。 +- 返回的资源**应该**映射到整个 HTTP 响应正文。 + +如果 `Create` 方法支持客户端分配的资源名称并且资源已存在，则请求**应该**失败并显示错误代码 `ALREADY_EXISTS` 或使用服务器分配的不同的资源名称，并且文档应该清楚地记录创建的资源名称可能与传入的不同。 + +`Create` 方法**必须**使用输入资源，以便在资源架构更改时，无需同时更新请求架构和资源架构。对于客户端无法设置的资源字段，**必须**将它们记录为“仅限输出”字段。 + +示例： + +```proto +// Creates a book in a shelf. +rpc CreateBook(CreateBookRequest) returns (Book) { + // Create maps to HTTP POST. URL path as the collection name. + // HTTP request body contains the resource. + option (google.api.http) = { + // The `parent` captures the parent resource name, such as "shelves/1". + post: "/v1/{parent=shelves/*}/books" + body: "book" + }; +} + +message CreateBookRequest { + // The parent resource name where the book is to be created. + string parent = 1; + + // The book id to use for this book. + string book_id = 3; + + // The book resource to create. + // The field name should match the Noun in the method name. + Book book = 2; +} + +rpc CreateShelf(CreateShelfRequest) returns (Shelf) { + option (google.api.http) = { + post: "/v1/shelves" + body: "shelf" + }; +} + +message CreateShelfRequest { + Shelf shelf = 1; +} +``` + +## 更新 + +`Update` 方法需要一条包含一个资源的请求消息和零个或多个参数作为输入。它更新指定的资源及其属性，并返回更新后的资源。 + +除了包含资源[名称或父资源](https://cloud.google.com/apis/design/resource_names?hl=zh-cn#Definitions)的属性之外，`Update` 方法**应该**可以改变可变资源的属性。`Update` 方法 **不得**包含任何“重命名”或“移动”资源的功能，这些功能**应该**由自定义方法来处理。 + +HTTP 映射： + +- 标准 `Update` 方法**应该**支持部分资源更新，并将 HTTP 动词 `PATCH` 与名为 `update_mask` 的 `FieldMask`字段一起使用。应忽略客户端提供的作为输入的[输出字段](https://cloud.google.com/apis/design/design_patterns?hl=zh-cn#output_fields)。 +- 需要更高级修补语义的 `Update` 方法（例如附加到重复字段）**应该**由[自定义方法](https://cloud.google.com/apis/design/custom_methods?hl=zh-cn)提供。 +- 如果 `Update` 方法仅支持完整资源更新，则**必须**使用 HTTP 动词 `PUT`。但是，强烈建议不要进行完整更新，因为在添加新资源字段时会出现向后兼容性问题。 +- 接收资源名称的消息字段**必须**映射到网址路径。该字段**可以**位于资源消息本身中。 +- 包含资源的请求消息字段**必须**映射到请求正文。 +- 所有剩余的请求消息字段**必须**映射到网址查询参数。 +- 响应消息**必须**是更新的资源本身。 + +如果 API 接受客户端分配的资源名称，则服务器**可以**允许客户端指定不存在的资源名称并创建新资源。 否则，使用不存在的资源名称的 `Update` 方法**应该**失败。 如果这是唯一的错误条件，则**应该**使用错误代码 `NOT_FOUND`。 + +具有支持资源创建的 `Update` 方法的 API 还**应该**提供 `Create` 方法。原因是，如果 `Update` 方法是唯一的方法，则它将不知道如何创建资源。 + +例如： + +```proto +// Updates a book. +rpc UpdateBook(UpdateBookRequest) returns (Book) { + // Update maps to HTTP PATCH. Resource name is mapped to a URL path. + // Resource is contained in the HTTP request body. + option (google.api.http) = { + // Note the URL template variable which captures the resource name of the + // book to update. + patch: "/v1/{book.name=shelves/*/books/*}" + body: "book" + }; +} + +message UpdateBookRequest { + // The book resource which replaces the resource on the server. + Book book = 1; + + // The update mask applies to the resource. For the `FieldMask` definition, + // see https://developers.google.com/protocol-buffers/docs/reference/google.protobuf#fieldmask + FieldMask update_mask = 2; +} +``` + +## 删除 + +`Delete` 方法需要一个资源名称和零个或多个参数作为输入，并删除或计划删除指定的资源。`Delete` 方法 **应该**返回 `google.protobuf.Empty`。 + +API **不应该**依赖于 `Delete` 方法返回的任何信息，因为它**不能**重复调用。 + +HTTP 映射： + +- `Delete` 方法 **必须**使用 HTTP `DELETE` 动词。 +- 接收资源名称的请求消息字段**应该**映射到网址路径。 +- 所有剩余的请求消息字段**应该**映射到网址查询参数。 +- 没有请求正文，API 配置**不得**声明 `body` 子句。 +- 如果 `Delete` 方法立即移除资源，则**应该**返回空响应。 +- 如果 `Delete` 方法启动长时间运行的操作，则**应该**返回长时间运行的操作。 +- 如果 `Delete` 方法仅将资源标记为已删除，则**应该**返回更新后的资源。 + +对 `Delete` 方法的调用在效果上应该是[幂等](http://tools.ietf.org/html/rfc2616#section-9.1.2)的，但不需要产生相同的响应。任意数量的 `Delete` 请求都**应该**导致资源（最终）被删除，但只有第一个请求会产生成功代码。后续请求应生成 `google.rpc.Code.NOT_FOUND`。 + +例如： + +```proto +// Deletes a book. +rpc DeleteBook(DeleteBookRequest) returns (google.protobuf.Empty) { + // Delete maps to HTTP DELETE. Resource name maps to the URL path. + // There is no request body. + option (google.api.http) = { + // Note the URL template variable capturing the multi-segment name of the + // book resource to be deleted, such as "shelves/shelf1/books/book2" + delete: "/v1/{name=shelves/*/books/*}" + }; +} + +message DeleteBookRequest { + // The resource name of the book to be deleted, for example: + // "shelves/shelf1/books/book2" + string name = 1; +} +``` + + + +# 自定义方法 + +本章将讨论如何在 API 设计中使用自定义方法。 + +自定义方法是指 5 个标准方法之外的 API 方法。这些方法**应该**仅用于标准方法不易表达的功能。通常情况下，API 设计者**应该**尽可能优先考虑使用标准方法，而不是自定义方法。标准方法具有大多数开发者熟悉的更简单且定义明确的语义，因此更易于使用且不易出错。另一项优势是 API 平台更加了解和支持标准方法，例如计费、错误处理、日志记录、监控。 + +自定义方法可以与资源、集合或服务关联。 它**可以**接受任意请求和返回任意响应，并且还支持流式请求和响应。 + +自定义方法名称**必须**遵循[方法命名惯例](https://cloud.google.com/apis/design/naming_convention?hl=zh-cn#method_names)。 + +## HTTP 映射 + +对于自定义方法，它们**应该**使用以下通用 HTTP 映射： + +``` +https://service.name/v1/some/resource/name:customVerb +``` + +使用 `:` 而不是 `/` 将自定义动词与资源名称分开以便支持任意路径。例如，恢复删除文件可以映射到 `POST /files/a/long/file/name:undelete` + +选择 HTTP 映射时，**应**遵循以下准则： + +- 自定义方法**应该**使用 HTTP `POST` 动词，因为该动词具有最灵活的语义，但作为替代 get 或 list 的方法（如有可能，**可以**使用 `GET`）除外。（详情请参阅第三条。） +- 自定义方法**不应该**使用 HTTP `PATCH`，但**可以**使用其他 HTTP 动词。在这种情况下，方法**必须**遵循该动词的标准 [HTTP 语义](https://tools.ietf.org/html/rfc2616#section-9)。 +- 请注意，使用 HTTP `GET` 的自定义方法**必须**具有幂等性并且无负面影响。例如，在资源上实现特殊视图的自定义方法**应该**使用 HTTP `GET`。 +- 接收与自定义方法关联的资源或集合的资源名称的请求消息字段**应该**映射到网址路径。 +- 网址路径**必须**以包含冒号（后跟自定义动词）的后缀结尾。 +- 如果用于自定义方法的 HTTP 动词允许 HTTP 请求正文（其适用于 `POST`、`PUT`、`PATCH` 或自定义 HTTP 动词），则此自定义方法的 HTTP 配置**必须**使用 `body: "*"` 子句，所有其他请求消息字段都**应**映射到 HTTP 请求正文。 +- 如果用于自定义方法的 HTTP 动词不接受 HTTP 请求正文（`GET`、`DELETE`），则此方法的 HTTP 配置**不得**使用 `body` 子句，并且所有其他请求消息字段都**应**映射到网址查询参数。 + +**警告**：如果一个服务会实现多个 API，API 生产者**必须**仔细创建服务配置，以避免 API 之间的自定义动词发生冲突。 + +```proto +// This is a service level custom method. +rpc Watch(WatchRequest) returns (WatchResponse) { + // Custom method maps to HTTP POST. All request parameters go into body. + option (google.api.http) = { + post: "/v1:watch" + body: "*" + }; +} + +// This is a collection level custom method. +rpc ClearEvents(ClearEventsRequest) returns (ClearEventsResponse) { + option (google.api.http) = { + post: "/v3/events:clear" + body: "*" + }; +} + +// This is a resource level custom method. +rpc CancelEvent(CancelEventRequest) returns (CancelEventResponse) { + option (google.api.http) = { + post: "/v3/{name=events/*}:cancel" + body: "*" + }; +} + +// This is a batch get custom method. +rpc BatchGetEvents(BatchGetEventsRequest) returns (BatchGetEventsResponse) { + // The batch get method maps to HTTP GET verb. + option (google.api.http) = { + get: "/v3/events:batchGet" + }; +} +``` + +## 用例 + +自定义方法适用于以下场景： + +- **重启虚拟机。** 设计备选方案可能是“在重启集合中创建一个重启资源”，这会让人感觉过于复杂，或者“虚拟机具有可变状态，客户端可以将状态从 RUNNING 更新到 RESTARTING”，这会产生可能存在哪些其他状态转换的问题。 此外，重启是一个常见概念，可以合理转化为一个自定义方法，从直观上来说符合开发者的预期。 +- **发送邮件。** 创建一个电子邮件消息不一定意味着要发送它（草稿）。与设计备选方案（将消息移动到“发件箱”集合）相比，自定义方法更容易被 API 用户发现，并且可以更直接地对概念进行建模。 +- **提拔员工。** 如果作为标准 `update` 方法实现，客户端需要复制企业提拔流程管理政策，以确保提拔发生在正确的级别，并属于同一职业阶梯等等。 +- **批处理方法。** 对于对性能要求苛刻的方法，提供自定义批处理方法**可以**有助于减少每个请求的开销。例如，[accounts.locations.batchGet](https://developers.google.com/my-business/reference/rest/v4/accounts.locations/batchGet?hl=zh-cn)。 + +以下是标准方法比自定义方法更适用的示例： + +- 使用不同查询参数的查询资源（使用带有标准列表过滤的标准 `list` 方法）。 +- 简单的资源属性更改（使用带有字段掩码的标准 `update` 方法）。 +- 关闭一个通知（使用标准 `delete` 方法）。 + +## 常用自定义方法 + +以下是常用或有用的自定义方法名称的精选列表。API 设计者在引入自己的名称之前**应该**考虑使用这些名称，以提高 API 之间的一致性。 + +| 方法名称 | 自定义动词 | HTTP 动词 | 备注 | +| :------- | :---------- | :-------- | :----------------------------------------------------------- | +| 取消 | `:cancel` | `POST` | 取消一个未完成的操作，例如 [`operations.cancel`](https://github.com/googleapis/googleapis/blob/master/google/longrunning/operations.proto#L100)。 | +| batchGet | `:batchGet` | `GET` | 批量获取多个资源。如需了解详情，请参阅[列表描述](https://cloud.google.com/apis/design/standard_methods?hl=zh-cn#list)。 | +| 移动 | `:move` | `POST` | 将资源从一个父级移动到另一个父级，例如 [`folders.move`](https://cloud.google.com/resource-manager/reference/rest/v2/folders/move?hl=zh-cn)。 | +| 搜索 | `:search` | `GET` | List 的替代方法，用于获取不符合 List 语义的数据，例如 [`services.search`](https://cloud.google.com/service-infrastructure/docs/service-consumer-management/reference/rest/v1/services/search?hl=zh-cn)。 | +| 恢复删除 | `:undelete` | `POST` | 恢复之前删除的资源，例如 [`services.undelete`](https://cloud.google.com/service-infrastructure/docs/service-management/reference/rest/v1/services/undelete?hl=zh-cn)。建议的保留期限为 30 天。 | + + + +# 标准字段 + +本节介绍了需要类似概念时应使用的一组标准消息字段定义。这将确保相同的概念在不同 API 中具有相同的名称和语义。 + +| 名称 | 类型 | 说明 | +| :---------------- | :----------------------------------------------------------- | :----------------------------------------------------------- | +| `name` | `string` | `name` 字段应包含[相对资源名称](https://cloud.google.com/apis/design/resource_names?hl=zh-cn#relative_resource_name)。 | +| `parent` | `string` | 对于资源定义和 List/Create 请求，`parent` 字段应包含父级[相对资源名称](https://cloud.google.com/apis/design/resource_names?hl=zh-cn#relative_resource_name)。 | +| `create_time` | [`Timestamp`](https://github.com/google/protobuf/blob/master/src/google/protobuf/timestamp.proto) | 创建实体的时间戳。 | +| `update_time` | [`Timestamp`](https://github.com/google/protobuf/blob/master/src/google/protobuf/timestamp.proto) | 最后更新实体的时间戳。注意：执行 create/patch/delete 操作时会更新 update_time。 | +| `delete_time` | [`Timestamp`](https://github.com/google/protobuf/blob/master/src/google/protobuf/timestamp.proto) | 删除实体的时间戳，仅当它支持保留时才适用。 | +| `expire_time` | [`Timestamp`](https://github.com/google/protobuf/blob/master/src/google/protobuf/timestamp.proto) | 实体到期时的到期时间戳。 | +| `start_time` | [`Timestamp`](https://github.com/google/protobuf/blob/master/src/google/protobuf/timestamp.proto) | 标记某个时间段开始的时间戳。 | +| `end_time` | [`Timestamp`](https://github.com/google/protobuf/blob/master/src/google/protobuf/timestamp.proto) | 标记某个时间段或操作结束的时间戳（无论其成功与否）。 | +| `read_time` | [`Timestamp`](https://github.com/google/protobuf/blob/master/src/google/protobuf/timestamp.proto) | 应读取（如果在请求中使用）或已读取（如果在响应中使用）特定实体的时间戳。 | +| `time_zone` | `string` | 时区名称。它应该是 [IANA TZ](http://www.iana.org/time-zones) 名称，例如“America/Los_Angeles”。如需了解详情，请参阅 https://en.wikipedia.org/wiki/List_of_tz_database_time_zones。 | +| `region_code` | `string` | 位置的 Unicode 国家/地区代码 (CLDR)，例如“US”和“419”。如需了解详情，请访问 http://www.unicode.org/reports/tr35/#unicode_region_subtag。 | +| `language_code` | `string` | BCP-47 语言代码，例如“en-US”或“sr-Latn”。如需了解详情，请参阅 http://www.unicode.org/reports/tr35/#Unicode_locale_identifier。 | +| `mime_type` | `string` | IANA 发布的 MIME 类型（也称为媒体类型）。如需了解详情，请参阅 https://www.iana.org/assignments/media-types/media-types.xhtml。 | +| `display_name` | `string` | 实体的显示名称。 | +| `title` | `string` | 实体的官方名称，例如公司名称。它应被视为 `display_name` 的正式版本。 | +| `description` | `string` | 实体的一个或多个文本描述段落。 | +| `filter` | `string` | List 方法的标准过滤器参数。 | +| `query` | `string` | 如果应用于搜索方法（即 [`:search`](https://cloud.google.com/apis/design/custom_methods?hl=zh-cn#common_custom_methods)），则与 `filter` 相同。 | +| `page_token` | `string` | List 请求中的分页令牌。 | +| `page_size` | `int32` | List 请求中的分页大小。 | +| `total_size` | `int32` | 列表中与分页无关的项目总数。 | +| `next_page_token` | `string` | List 响应中的下一个分页令牌。它应该用作后续请求的 `page_token`。空值表示不再有结果。 | +| `order_by` | `string` | 指定 List 请求的结果排序。 | +| `request_id` | `string` | 用于检测重复请求的唯一字符串 ID。 | +| `resume_token` | `string` | 用于恢复流式传输请求的不透明令牌。 | +| `labels` | `map` | 表示 Cloud 资源标签。 | +| `show_deleted` | `bool` | 如果资源允许恢复删除行为，相应的 List 方法必须具有 `show_deleted` 字段，以便客户端可以发现已删除的资源。 | +| `update_mask` | [`FieldMask`](https://github.com/google/protobuf/blob/master/src/google/protobuf/field_mask.proto) | 它用于 `Update` 请求消息，该消息用于对资源执行部分更新。此掩码与资源相关，而不是与请求消息相关。 | +| `validate_only` | `bool` | 如果为 true，则表示仅应验证给定请求，而不执行该请求。 | + +### 系统参数 + +除标准字段外，Google API 还支持可以在所有 API 方法中使用的一组共同请求参数，这些参数称为“系统参数”。如需了解详情，请参阅[系统参数](https://cloud.google.com/apis/docs/system-parameters?hl=zh-cn)。 + + + +# 错误 + +本章简要介绍了 Google API 的错误模型。它还为开发者提供了正确生成和处理错误的通用指南。 + +Google API 使用简单的协议无关错误模型，以便我们在不同的 API、不同的 API 协议（如 gRPC 或 HTTP）和不同的错误上下文（例如异步、批处理或工作流错误）中能够有一致的体验。 + +## 错误模型 + +Google API 的错误模型由 [`google.rpc.Status`](https://github.com/googleapis/googleapis/blob/master/google/rpc/status.proto) 逻辑定义，该实例在发生 API 错误时返回给客户端。以下代码段显示了错误模型的总体设计： + +```proto +package google.rpc; + +// The `Status` type defines a logical error model that is suitable for +// different programming environments, including REST APIs and RPC APIs. +message Status { + // A simple error code that can be easily handled by the client. The + // actual error code is defined by `google.rpc.Code`. + int32 code = 1; + + // A developer-facing human-readable error message in English. It should + // both explain the error and offer an actionable resolution to it. + string message = 2; + + // Additional error information that the client code can use to handle + // the error, such as retry info or a help link. + repeated google.protobuf.Any details = 3; +} +``` + +由于大多数 Google API 采用面向资源的 API 设计，因此错误处理遵循相同的设计原则，使用一小组标准错误配合大量资源。例如，服务器没有定义不同类型的“找不到”错误，而是使用一个标准 `google.rpc.Code.NOT_FOUND` 错误代码并告诉客户端找不到哪个特定资源。错误空间变小降低了文档的复杂性，在客户端库中提供了更好的惯用映射，并降低了客户端的逻辑复杂性，同时不限制是否包含可操作信息。 + +### 错误代码 + +Google API **必须**使用 [`google.rpc.Code`](https://github.com/googleapis/googleapis/blob/master/google/rpc/code.proto) 定义的规范错误代码。单个 API **应**避免定义其他错误代码，因为开发人员不太可能编写用于处理大量错误代码的逻辑。作为参考，每个 API 调用平均处理 3 个错误代码意味着大多数应用的逻辑只是用于错误处理，这对开发人员而言并非好体验。 + +### 错误消息 + +错误消息应该可以帮助用户轻松快捷地**理解和解决** API 错误。通常，在编写错误消息时请考虑以下准则： + +- 不要假设用户是您 API 的专家用户。用户可能是客户端开发人员、操作人员、IT 人员或应用的最终用户。 +- 不要假设用户了解有关服务实现的任何信息，或者熟悉错误的上下文（例如日志分析）。 +- 如果可能，应构建错误消息，以便技术用户（但不一定是 API 开发人员）可以响应错误并改正。 +- 确保错误消息内容简洁。如果需要，请提供一个链接，便于有疑问的读者提问、提供反馈或详细了解错误消息中不方便说明的信息。此外，可使用详细信息字段来提供更多信息。 + +**警告**：错误消息不属于 API 表面。它们随时都会更改，恕不另行通知。应用代码**不得**严重依赖于错误消息。 + +### 错误详情 + +Google API 为错误详细信息定义了一组标准错误负载，您可在 [google/rpc/error_details.proto](https://github.com/googleapis/googleapis/blob/master/google/rpc/error_details.proto) 中找到这些错误负载。 它们涵盖了对于 API 错误的最常见需求，例如配额失败和无效参数。与错误代码一样，开发者应尽可能使用这些标准载荷。 + +只有在可以帮助应用代码处理错误的情况下，才应引入其他错误详细信息类型。如果错误信息只能由人工处理，则应根据错误消息内容，让开发人员手动处理，而不是引入其他错误详细信息类型。 + +下面是一些示例 `error_details` 载荷： + +- `ErrorInfo` 提供既**稳定**又**可扩展**的结构化错误信息。 +- `RetryInfo`：描述客户端何时可以重试失败的请求，这些内容可能在以下方法中返回：`Code.UNAVAILABLE` 或 `Code.ABORTED` +- `QuotaFailure`：描述配额检查失败的方式，这些内容可能在以下方法中返回：`Code.RESOURCE_EXHAUSTED` +- `BadRequest`：描述客户端请求中的违规行为，这些内容可能在以下方法中返回：`Code.INVALID_ARGUMENT` + +### 错误信息 + +[`ErrorInfo`](https://github.com/googleapis/googleapis/blob/master/google/rpc/error_details.proto#L110) 是一种特殊种类的错误负载。它提供人类和应用可使用的**稳定且可扩展**错误信息。每个 `ErrorInfo` 包含 3 条信息：错误网域、错误原因和一组错误元数据，如[示例](https://translate.googleapis.com/language/translate/v2?key=invalid&q=hello&source=en&target=es&format=text&$.xgafv=2)。如需了解详情，请参阅 [`ErrorInfo`](https://github.com/googleapis/googleapis/blob/master/google/rpc/error_details.proto#L110) 定义。 + +对于 Google API，主要错误网域是 `googleapis.com`，相应的错误原因由 `google.api.ErrorReason` 枚举定义。如需了解详情，请参阅 [`google.api.ErrorReason`](https://github.com/googleapis/googleapis/blob/master/google/api/error_reason.proto) 定义。 + +### 错误本地化 + +[`google.rpc.Status`](https://github.com/googleapis/googleapis/blob/master/google/rpc/status.proto) 中的 `message` 字段面向开发人员，**必须**使用英语。 + +如果需要面向用户的错误消息，请使用 [`google.rpc.LocalizedMessage`](https://github.com/googleapis/googleapis/blob/master/google/rpc/error_details.proto) 作为您的详细信息字段。虽然 [`google.rpc.LocalizedMessage`](https://github.com/googleapis/googleapis/blob/master/google/rpc/error_details.proto) 中的消息字段可以进行本地化，请确保 [`google.rpc.Status`](https://github.com/googleapis/googleapis/blob/master/google/rpc/status.proto) 中的消息字段使用英语。 + +默认情况下，API 服务应使用经过身份验证的用户的语言区域设置或 HTTP `Accept-Language` 标头或请求中的 `language_code` 参数来确定本地化的语言。 + +## 错误映射 + +可以在不同的编程环境中访问 Google API。每种环境通常都有自己的错误处理方法。以下部分介绍了错误模型在常用环境中的映射方式。 + +### HTTP 映射 + +虽然 proto3 消息具有原生 JSON 编码，但 Google 的 API 平台对 Google 的 JSON HTTP API 使用了不同的错误架构，以实现向后兼容性。 + +架构： + +```proto +// The error format v2 for Google JSON REST APIs. +// +// NOTE: This schema is not used for other wire protocols. +message Error { + // This message has the same semantics as `google.rpc.Status`. It uses HTTP + // status code instead of gRPC status code. It has an extra field `status` + // for backward compatibility with Google API Client Libraries. + message Status { + // The HTTP status code that corresponds to `google.rpc.Status.code`. + int32 code = 1; + // This corresponds to `google.rpc.Status.message`. + string message = 2; + // This is the enum version for `google.rpc.Status.code`. + google.rpc.Code status = 4; + // This corresponds to `google.rpc.Status.details`. + repeated google.protobuf.Any details = 5; + } + // The actual error payload. The nested message structure is for backward + // compatibility with Google API client libraries. It also makes the error + // more readable to developers. + Status error = 1; +} +``` + +例如： + +```shell +$ curl 'https://translate.googleapis.com/language/translate/v2?key=invalid&q=hello&source=en&target=es&format=text&$.xgafv=2' +{ + "error": { + "code": 400, + "message": "API key not valid. Please pass a valid API key.", + "status": "INVALID_ARGUMENT", + "details": [ + { + "@type": "type.googleapis.com/google.rpc.ErrorInfo", + "reason": "API_KEY_INVALID", + "domain": "googleapis.com", + "metadata": { + "service": "translate.googleapis.com" + } + } + ] + } +} +``` + +### gRPC 映射 + +不同的 RPC 协议采用不同方式映射错误模型。对于 [gRPC](http://grpc.io/)，生成的代码和每种支持语言的运行时库为错误模型提供原生支持。您可在 gRPC 的 API 文档中了解更多信息。例如，请参阅 gRPC Java 的 [`io.grpc.Status`](https://grpc.github.io/grpc-java/javadoc/io/grpc/Status.html)。 + +### 客户端库映射 + +Google 客户端库可能会根据语言选择采用不同方式表达错误，以与既定习语保持一致。例如，[google-cloud-go](https://github.com/GoogleCloudPlatform/google-cloud-go) 库将返回一个错误，该错误实现与 [`google.rpc.Status`](https://github.com/googleapis/googleapis/blob/master/google/rpc/status.proto) 相同的接口，而 [google-cloud-java](https://github.com/googleapis/google-cloud-java) 将引发异常。 + +## 处理错误 + +下面的表格包含在 [`google.rpc.Code`](https://github.com/googleapis/googleapis/blob/master/google/rpc/code.proto) 中定义的所有 gRPC 错误代码及其原因的简短描述。要处理错误，您可以检查返回状态代码的说明并相应地修改您的调用。 + +| HTTP | gRPC | 说明 | +| :--- | :-------------------- | :----------------------------------------------------------- | +| 200 | `OK` | 无错误。 | +| 400 | `INVALID_ARGUMENT` | 客户端指定了无效参数。如需了解详情，请查看错误消息和错误详细信息。 | +| 400 | `FAILED_PRECONDITION` | 请求无法在当前系统状态下执行，例如删除非空目录。 | +| 400 | `OUT_OF_RANGE` | 客户端指定了无效范围。 | +| 401 | `UNAUTHENTICATED` | 由于 OAuth 令牌丢失、无效或过期，请求未通过身份验证。 | +| 403 | `PERMISSION_DENIED` | 客户端权限不足。这可能是因为 OAuth 令牌没有正确的范围、客户端没有权限或者 API 尚未启用。 | +| 404 | `NOT_FOUND` | 未找到指定的资源。 | +| 409 | `ABORTED` | 并发冲突，例如读取/修改/写入冲突。 | +| 409 | `ALREADY_EXISTS` | 客户端尝试创建的资源已存在。 | +| 429 | `RESOURCE_EXHAUSTED` | 资源配额不足或达到速率限制。如需了解详情，客户端应该查找 google.rpc.QuotaFailure 错误详细信息。 | +| 499 | `CANCELLED` | 请求被客户端取消。 | +| 500 | `DATA_LOSS` | 出现不可恢复的数据丢失或数据损坏。客户端应该向用户报告错误。 | +| 500 | `UNKNOWN` | 出现未知的服务器错误。通常是服务器错误。 | +| 500 | `INTERNAL` | 出现内部服务器错误。通常是服务器错误。 | +| 501 | `NOT_IMPLEMENTED` | API 方法未通过服务器实现。 | +| 502 | 不适用 | 到达服务器前发生网络错误。通常是网络中断或配置错误。 | +| 503 | `UNAVAILABLE` | 服务不可用。通常是服务器已关闭。 | +| 504 | `DEADLINE_EXCEEDED` | 超出请求时限。仅当调用者设置的时限比方法的默认时限短（即请求的时限不足以让服务器处理请求）并且请求未在时限范围内完成时，才会发生这种情况。 | + +**警告**：Google API 可以并发检查 API 请求的多个前提条件。返回某个错误代码并不满足其他前提条件。应用代码**不得**依赖前提条件检查的排序。 + +### 重试错误 + +客户端**可能**使用指数退避算法重试 503 UNAVAILABLE 错误。 除非另有说明，否则最小延迟应为 1 秒。 除非另有说明，否则默认重试重复应当仅一次。 + +对于 429 RESOURCE_EXHAUSTED 错误，客户端可能会在更高层级以最少 30 秒的延迟重试。此类重试仅对长时间运行的后台作业有用。 + +对于所有其他错误，重试请求可能并不适用。首先确保您的请求具有幂等性，并查看 [`google.rpc.RetryInfo`](https://github.com/googleapis/googleapis/blob/master/google/rpc/error_details.proto#L40) 以获取指导。 + +### 传播错误 + +如果您的 API 服务依赖于其他服务，则不应盲目地将这些服务的错误传播到您的客户端。在翻译错误时，我们建议执行以下操作： + +- 隐藏实现详细信息和机密信息。 +- 调整负责该错误的一方。例如，从另一个服务接收 `INVALID_ARGUMENT` 错误的服务器应该将 `INTERNAL` 传播给它自己的调用者。 + +### 重现错误 + +如果您通过分析日志和监控功能无法解决错误，则应该尝试通过简单且可重复的测试来重现错误。您可以使用该测试来收集问题排查的相关信息，您可以在联系技术支持团队时提供这些信息。 + +我们建议您使用 [`oauth2l`](https://github.com/google/oauth2l)、[`curl -v`](https://curl.se/docs/manpage.html) 和[系统参数](https://cloud.google.com/apis/docs/system-parameters?hl=zh-cn)重现 Google API 中的错误。它们可以共同重现几乎所有 Google API 请求，并为您提供详细的调试信息。如需了解详情，请参阅您要调用的 API 的相应文档页面。 + +**注意**：如需进行问题排查，您应该在请求网址中添加 `&$.xgafv=2` 以选择错误格式 v2。出于兼容性方面的考虑，某些 Google API 默认使用错误格式 v1。 + +## 生成错误 + +如果您是服务器开发者，则应该生成包含足够信息的错误，以帮助客户端开发者理解并解决问题。同时，您必须重视用户数据的安全性和隐私性，避免在错误消息和错误详细信息中披露敏感信息，因为错误通常会被记录下来并且可能被其他人访问。例如，诸如“客户端 IP 地址不在 allowlist 12.0.0.0/8 上”之类的错误消息会披露服务器端政策的相关信息，用户可能无法访问日志。 + +要生成正确的错误，首先需要熟悉 [`google.rpc.Code`](https://github.com/googleapis/googleapis/blob/master/google/rpc/code.proto)，然后才能为每个错误条件选择最合适的错误代码。服务器应用可以并行检查多个错误条件，并返回第一个错误条件。 + +下表列出了每个错误代码和恰当的错误消息示例。 + +| HTTP | gRPC | 错误消息示例 | +| :--- | :-------------------- | :-------------------------------------------------- | +| 400 | `INVALID_ARGUMENT` | 请求字段 x.y.z 是 xxx，预期为 [yyy, zzz] 内的一个。 | +| 400 | `FAILED_PRECONDITION` | 资源 xxx 是非空目录，因此无法删除。 | +| 400 | `OUT_OF_RANGE` | 参数“age”超出范围 [0,125]。 | +| 401 | `UNAUTHENTICATED` | 身份验证凭据无效。 | +| 403 | `PERMISSION_DENIED` | 使用权限“xxx”处理资源“yyy”被拒绝。 | +| 404 | `NOT_FOUND` | 找不到资源“xxx”。 | +| 409 | `ABORTED` | 无法锁定资源“xxx”。 | +| 409 | `ALREADY_EXISTS` | 资源“xxx”已经存在。 | +| 429 | `RESOURCE_EXHAUSTED` | 超出配额限制“xxx”。 | +| 499 | `CANCELLED` | 请求被客户端取消。 | +| 500 | `DATA_LOSS` | 请参阅注释。 | +| 500 | `UNKNOWN` | 请参阅注释。 | +| 500 | `INTERNAL` | 请参阅注释。 | +| 501 | `NOT_IMPLEMENTED` | 方法“xxx”未实现。 | +| 503 | `UNAVAILABLE` | 请参阅注释。 | +| 504 | `DEADLINE_EXCEEDED` | 请参阅备注。 | + +**注意**：由于客户端无法修复服务器错误，因此生成其他错误详细信息没有任何用处。为避免在错误条件下泄露敏感信息，建议不要生成任何错误消息，而仅生成 `google.rpc.DebugInfo` 错误详细信息。`DebugInfo` 专为服务器端的日志记录而设计，**不得**发送到客户端。 + +### 错误负载 + +`google.rpc` 软件包定义了一组标准错误载荷，它们优先于自定义错误载荷。下表列出了每个错误代码及其匹配的标准错误负载（如果适用）。 我们建议高级应用在处理错误时在 `google.rpc.Status` 中查找这些错误负载。 + +| HTTP | gRPC | 建议的错误详细信息 | +| :--- | :-------------------- | :------------------------------- | +| 400 | `INVALID_ARGUMENT` | `google.rpc.BadRequest` | +| 400 | `FAILED_PRECONDITION` | `google.rpc.PreconditionFailure` | +| 400 | `OUT_OF_RANGE` | `google.rpc.BadRequest` | +| 401 | `UNAUTHENTICATED` | `google.rpc.ErrorInfo` | +| 403 | `PERMISSION_DENIED` | `google.rpc.ErrorInfo` | +| 404 | `NOT_FOUND` | `google.rpc.ResourceInfo` | +| 409 | `ABORTED` | | +| 409 | `ALREADY_EXISTS` | `google.rpc.ResourceInfo` | +| 429 | `RESOURCE_EXHAUSTED` | `google.rpc.QuotaFailure` | +| 499 | `CANCELLED` | | +| 500 | `DATA_LOSS` | | +| 500 | `UNKNOWN` | | +| 500 | `INTERNAL` | | +| 501 | `NOT_IMPLEMENTED` | | +| 503 | `UNAVAILABLE` | | +| 504 | `DEADLINE_EXCEEDED` | | + + + +# 命名规则 + +为了跨众多 API 长期为开发者提供一致的体验，API 使用的名称都**应**具有以下特点： + +- 简单 +- 直观 +- 一致 + +这包括接口、资源、集合、方法和消息的名称。 + +由于很多开发者不是以英语为母语，所以这些命名惯例的目标之一是确保大多数开发者可以轻松理解 API。对于方法和资源，我们鼓励使用简单、一致和少量的词汇来命名。 + +- API 中使用的名称**应**采用正确的美式英语。例如，使用美式英语的 license、color，而非英式英语的 licence、colour。 +- 为了简化命名，**可以**使用已被广泛接受的简写形式或缩写。例如，API 优于 Application Programming Interface。 +- 尽量使用直观、熟悉的术语。例如，如果描述移除（和销毁）一个资源，则删除优于擦除。 +- 使用相同的名称或术语命名同样的概念，包括跨 API 共享的概念。 +- 避免名称过载。使用不同的名称命名不同的概念。 +- 避免在 API 的上下文以及范围更大的 Google API 生态系统中使用含糊不清、过于笼统的名称。这些名称可能导致对 API 概念的误解。相反，应选择能准确描述 API 概念的名称。这对定义一阶 API 元素（例如资源）的名称尤其重要。没有需避免名称的明确列表，因为每个名称都必须放在其他名称的上下文中进行评估。实例、信息和服务的名称都曾出现过这类问题。所选择的名称应清楚地描述 API 概念（例如：什么的实例？），并将其与其他相关概念区分开（例如：“alert”是指规则、信号还是通知？）。 +- 仔细考虑使用的名称是否可能与常用编程语言中的关键字存在冲突。您**可以**使用这些名称，但在 API 审核期间可能会触发额外的审查。因此应明智而审慎地使用。 + +## 产品名称 + +产品名称是指 API 的产品营销名称，例如 Google Calendar API。API、界面、文档、服务条款、对帐单和商业合同等信息中使用的产品名称**必须**一致。Google API **必须**使用产品团队和营销团队批准的产品名称。 + +下表显示了所有相关 API 名称及其一致性的示例。如需详细了解各自名称及其命名惯例，请参阅本页面下方的详细信息。 + +| API 名称 | 示例 | +| :------------- | :----------------------------------- | +| **产品名称** | `Google Calendar API` | +| **服务名称** | `calendar.googleapis.com` | +| **软件包名称** | `google.calendar.v3` | +| **接口名称** | `google.calendar.v3.CalendarService` | +| **来源目录** | `//google/calendar/v3` | +| **API 名称** | `calendar` | + +## 服务名称 + +服务名称**应该**是语法上有效的 DNS 名称（遵循 [RFC 1035](http://www.ietf.org/rfc/rfc1035.txt)），可以解析为一个或多个网络地址。公开的 Google API 的服务名称采用 `xxx.googleapis.com` 格式。例如，Google 日历的服务名称是 `calendar.googleapis.com`。 + +如果一个 API 是由多项服务组成，则**应**采用更容易发现的命名方式。要做到这点，一种方法是使服务名称共享一个通用前缀。例如，`build.googleapis.com` 和 `buildresults.googleapis.com` 服务都属于 Google Build API。 + +## 软件包名称 + +API .proto 文件中声明的软件包名称**应该**与产品名称和服务名称保持一致。软件包名称**应该**使用单数组件名称，以避免混合使用单数和复数组件名称。软件包名称**不能**使用下划线。进行版本控制的 API 的软件包名称**必须**以此版本结尾。例如： + +```proto +// Google Calendar API +package google.calendar.v3; +``` + +与服务无直接关联的抽象 API（例如 Google Watcher API）**应该**使用与产品名称一致的 proto 软件包名称： + +```proto +// Google Watcher API +package google.watcher.v1; +``` + +API .proto 文件中指定的 Java 软件包名称**必须**与带有标准 Java 软件包名称前缀（`com.`、`edu.`、`net.` 等）的 proto 软件包名称相匹配。例如： + +```proto +package google.calendar.v3; + +// Specifies Java package name, using the standard prefix "com." +option java_package = "com.google.calendar.v3"; +``` + +## 集合 ID + +[集合 ID](https://cloud.google.com/apis/design/resource_names?hl=zh-cn#collection_id) **应**采用复数和 `lowerCamelCase`（小驼峰式命名法）格式，并遵循美式英语拼写和语义。例如：`events`、`children` 或 `deletedEvents`。 + +## 接口名称 + +为了避免与[服务名称](https://cloud.google.com/apis/design/naming_convention?hl=zh-cn#service_names)（例如 `pubsub.googleapis.com`）混淆，术语 *“接口名称”*是指在 .proto 文件中定义 `service` 时使用的名称： + +```proto +// Library is the interface name. +service Library { + rpc ListBooks(...) returns (...); + rpc ... +} +``` + +您可以将服务名称视为对一组 API 实际实现的引用，而接口名称则是 API 的抽象定义。 + +接口名称**应该**使用直观的名词，例如 Calendar 或 Blob。该名称**不得**与编程语言及其运行时库（如 File）中的成熟概念相冲突。 + +在极少数情况下，接口名称会与 API 中的其他名称相冲突，此时**应该**使用后缀（例如 `Api` 或 `Service`）来消除歧义。 + +## 方法名称 + +服务**可以**在其 IDL 规范中定义一个或多个远程过程调用 (RPC) 方法，这些方法需与集合和资源上的方法对应。方法名称**应**采用大驼峰式命名格式并遵循 `VerbNoun` 的命名惯例，其中 Noun（名词）通常是资源类型。 + +| 动词 | 名词 | 方法名称 | 请求消息 | 响应消息 | +| :------- | :----- | :----------- | :------------------ | :---------------------- | +| `List` | `Book` | `ListBooks` | `ListBooksRequest` | `ListBooksResponse` | +| `Get` | `Book` | `GetBook` | `GetBookRequest` | `Book` | +| `Create` | `Book` | `CreateBook` | `CreateBookRequest` | `Book` | +| `Update` | `Book` | `UpdateBook` | `UpdateBookRequest` | `Book` | +| `Rename` | `Book` | `RenameBook` | `RenameBookRequest` | `RenameBookResponse` | +| `Delete` | `Book` | `DeleteBook` | `DeleteBookRequest` | `google.protobuf.Empty` | + +方法名称的动词部分**应该**使用用于要求或命令的[祈使语气](https://en.wikipedia.org/wiki/Imperative_mood#English)，而不是用于提问的陈述语气。 + +如果关于 API 子资源的方法名称使用提问动词（经常使用陈述语气表示），则容易让人混淆。例如，要求 API 创建一本书，这显然是 `CreateBook`（祈使语气），但是询问 API 关于图书发行商的状态可能会使用陈述语气，例如 `IsBookPublisherApproved` 或 `NeedsPublisherApproval`。若要在此类情况下继续使用祈使语气，请使用“check”(`CheckBookPublisherApproved`) 和“validate”(`ValidateBookPublisher`) 等命令。 + +方法名称**不应**包含介词（例如“For”、“With”、“At”、“To”）。通常，带有介词的方法名称表示正在使用新方法，应将一个字段添加到现有方法中，或者该方法应使用不同的动词。 + +例如，如果 `CreateBook` 消息已存在且您正在考虑添加 `CreateBookFromDictation`，请考虑使用 `TranscribeBook` 方法。 + +## 消息名称 + +消息名称**应该**简洁明了。避免不必要或多余的字词。如果不存在无形容词的相应消息，则通常可以省略形容词。例如，如果没有非共享代理设置，则 `SharedProxySettings` 中的 `Shared` 是多余的。 + +消息名称**不应**包含介词（例如“With”、“For”）。通常，带有介词的消息名称可以通过消息上的可选字段来更好地表示。 + +### 请求和响应消息 + +RPC 方法的请求和响应消息**应该**分别以带有后缀 `Request` 和 `Response` 的方法名称命名，除非方法请求或响应类型为以下类型： + +- 一条空消息（使用 `google.protobuf.Empty`）、 +- 一个资源类型，或 +- 一个表示操作的资源 + +这通常适用于在标准方法 `Get`、`Create`、`Update` 或 `Delete` 中使用的请求或响应。 + +## 枚举名称 + +枚举类型**必须**使用 UpperCamelCase 格式的名称。 + +枚举值**必须**使用 CAPITALIZED_NAMES_WITH_UNDERSCORES 格式。每个枚举值**必须**以分号（而不是逗号）结尾。第一个值**应该**命名为 ENUM_TYPE_UNSPECIFIED，因为在枚举值未明确指定时系统会返回此值。 + +```proto +enum FooBar { + // The first value represents the default and must be == 0. + FOO_BAR_UNSPECIFIED = 0; + FIRST_VALUE = 1; + SECOND_VALUE = 2; +} +``` + +### 封装容器 + +封装 proto2 枚举类型（其中 `0` 值具有非 `UNSPECIFIED` 的含义）的消息**应该**以后缀 `Value` 来命名，并具有名为 `value` 的单个字段。 + +```proto +enum OldEnum { + VALID = 0; + OTHER_VALID = 1; +} +message OldEnumValue { + OldEnum value = 1; +} +``` + +## 字段名称 + +.proto 文件中的字段定义**必须**使用 lower_case_underscore_separated_names 格式。这些名称将映射到每种编程语言的生成代码中的原生命名惯例。 + +字段名称**不应**包含介词（例如“for”、“during”、“at”），例如： + +- `reason_for_error` 应该改成 `error_reason` +- `cpu_usage_at_time_of_failure` 应该改成 `failure_time_cpu_usage` + +字段名称**不应**使用后置形容词（名词后面的修饰符），例如： + +- `items_collected` 应该改成 `collected_items` +- `objects_imported` 应该改成 `imported_objects` + +### 重复字段名称 + +API 中的重复字段**必须**使用正确的复数形式。这符合现有 Google API 的命名惯例和外部开发者的共同预期。 + +### 时间和时间段 + +要表示一个与任何时区或日历无关的时间点，**应该**使用 `google.protobuf.Timestamp`，并且字段名称**应该**以 `time`（例如 `start_time` 和 `end_time`）结尾。 + +如果时间指向一个活动，则字段名称**应该**采用 `verb_time` 的形式，例如 `create_time` 和 `update_time`。请勿使用动词的过去时态，例如 `created_time` 或 `last_updated_time`。 + +要表示与任何日历和概念（如“天”或“月”）无关的两个时间点之间的时间跨度，**应该**使用 `google.protobuf.Duration`。 + +```proto +message FlightRecord { + google.protobuf.Timestamp takeoff_time = 1; + google.protobuf.Duration flight_duration = 2; +} +``` + +如果由于历史性或兼容性原因（包括系统时间、时长、推迟和延迟），您必须使用整数类型表示与时间相关的字段，那么字段名称**必须**采用以下格式： + +``` +xxx_{time|duration|delay|latency}_{seconds|millis|micros|nanos} +message Email { + int64 send_time_millis = 1; + int64 receive_time_millis = 2; +} +``` + +如果由于历史性或兼容性原因，您必须使用字符串类型表示时间戳，则字段名称**不应该**包含任何单位后缀。字符串表示形式**应该**使用 RFC 3339 格式，例如“2014-07-30T10:43:17Z”。 + +### 日期和时间 + +对于与时区和时段无关的日期，**应该**使用 `google.type.Date`，并且该名称应具有后缀 `_date`。如果日期必须表示为字符串，则应采用 ISO 8601 日期格式 YYYY-MM-DD，例如 2014-07-30。 + +对于与时区和日期无关的时间，**应该**使用 `google.type.TimeOfDay`，并且该名称应具有后缀 `_time`。如果时间必须表示为字符串，则应采用 ISO 8601 24 小时制格式 HH:MM:SS[.FFF]，例如 14:55:01.672。 + +```proto +message StoreOpening { + google.type.Date opening_date = 1; + google.type.TimeOfDay opening_time = 2; +} +``` + +### 数量 + +由整数类型表示的数量**必须**包含度量单位。 + +``` +xxx_{bytes|width_pixels|meters} +``` + +如果数量是条目计数，则该字段**应该**具有后缀 `_count`，例如 `node_count`。 + +### 列表过滤器字段 + +如果 API 支持对 `List` 方法返回的资源进行过滤，则包含过滤器表达式的字段**应该**命名为 `filter`。例如： + +```proto +message ListBooksRequest { + // The parent resource name. + string parent = 1; + + // The filter expression. + string filter = 2; +} +``` + +### 列表响应 + +`List` 方法的响应消息（包含资源列表）中的字段名称**必须**是资源名称本身的复数形式。例如，`CalendarApi.ListEvents()` 方法**必须**为返回的资源列表定义一个响应消息`ListEventsResponse`，其中包含一个名为 `events` 的重复字段。 + +```proto +service CalendarApi { + rpc ListEvents(ListEventsRequest) returns (ListEventsResponse) { + option (google.api.http) = { + get: "/v3/{parent=calendars/*}/events"; + }; + } +} + +message ListEventsRequest { + string parent = 1; + int32 page_size = 2; + string page_token = 3; +} + +message ListEventsResponse { + repeated Event events = 1; + string next_page_token = 2; +} +``` + +## 驼峰式命名法 + +除字段名称和枚举值外，`.proto` 文件中的所有定义都**必须**使用由 [Google Java 样式](https://google.github.io/styleguide/javaguide.html#s5.3-camel-case)定义的 UpperCamelCase 格式的名称。 + +## 名称缩写 + +对于软件开发者熟知的名称缩写，例如 `config` 和 `spec`，**应该**在 API 定义中使用这些缩写，而非完整名称。这将使源代码易于读写。而在正式文档中，**应该**使用完整名称。示例： + +- config (configuration) +- id (identifier) +- spec (specification) +- stats (statistics) + + + +# 常见设计模式 + +## 空响应 + +标准的 `Delete` 方法**应该**返回 `google.protobuf.Empty`，除非它正在执行“软”删除，在这种情况下，该方法**应该**返回状态已更新的资源，以指示正在进行删除。 + +自定义方法**应该**有自己的 `XxxResponse` 消息（即使为空），因为它们的功能很可能会随着时间的推移而增长并需要返回其他数据。 + +## 表示范围 + +表示范围的字段**应该**使用半开区间和命名惯例 `[start_xxx, end_xxx)`，例如 `[start_key, end_key)` 或 `[start_time, end_time)`。通常 C ++ STL 库和 Java 标准库会使用半开区间语义。API **应该**避免使用其他表示范围的方式，例如 `(index, count)` 或 `[first, last]`。 + +## 资源标签 + +在面向资源的 API 中，资源架构由 API 定义。要让客户端将少量简单元数据附加到资源（例如，将虚拟机资源标记为数据库服务器），API **应该** 使用 `google.api.LabelDescriptor` 中描述的资源标签设计模式。 + +为此，API 设计**应该**将 `map labels` 字段添加到资源定义中。 + +```proto +message Book { + string name = 1; + map labels = 2; +} +``` + +## 长时间运行的操作 + +如果某个 API 方法通常需要很长时间才能完成，您可以通过适当设计，让其向客户端返回“长时间运行的操作”资源，客户端可以使用该资源来跟踪进度和接收结果。 [Operation](https://github.com/googleapis/googleapis/blob/master/google/longrunning/operations.proto) 定义了一个标准接口来使用长时间运行的操作。 各个 API **不得**为长时间运行的操作定义自己的接口，以避免不一致性。 + +操作资源**必须**作为响应消息直接返回，操作的任何直接后果都**应该**反映在 API 中。例如，在创建资源时，即便资源表明其尚未准备就绪，该资源也**应该**出现在 LIST 和 GET 方法中。操作完成后，如果方法并未长时间运行，则 `Operation.response` 字段应包含本应直接返回的消息。 + +操作可以使用 `Operation.metadata` 字段提供有关其进度的信息。即使初始实现没有填充 `metadata` 字段，API 也**应该**为此元数据定义消息。 + +## 列表分页 + +可列表集合**应该**支持分页，即使结果通常很小。 + +**说明**：如果某个 API 从一开始就不支持分页，稍后再支持它就比较麻烦，因为添加分页会破坏 API 的行为。 不知道 API 正在使用分页的客户端可能会错误地认为他们收到了完整的结果，而实际上只收到了第一页。 + +为了在 `List` 方法中支持分页（在多个页面中返回列表结果），API **应该**： + +- 在 `List` 方法的请求消息中定义 `string` 字段 `page_token`。客户端使用此字段请求列表结果的特定页面。 +- 在 `List` 方法的请求消息中定义 `int32` 字段 `page_size`。客户端使用此字段指定服务器返回的最大结果数。服务器**可以**进一步限制单个页面中返回的最大结果数。如果 `page_size` 为 `0`，则服务器将决定要返回的结果数。 +- 在 `List` 方法的响应消息中定义 `string` 字段 `next_page_token`。此字段表示用于检索下一页结果的分页令牌。如果值为 `""`，则表示请求没有进一步的结果。 + +要检索下一页结果，客户端**应该**在后续的 `List` 方法调用中（在请求消息的 `page_token` 字段中）传递响应的 `next_page_token` 值： + +```proto +rpc ListBooks(ListBooksRequest) returns (ListBooksResponse); + +message ListBooksRequest { + string parent = 1; + int32 page_size = 2; + string page_token = 3; +} + +message ListBooksResponse { + repeated Book books = 1; + string next_page_token = 2; +} +``` + +当客户端传入除页面令牌之外的查询参数时，如果查询参数与页面令牌不一致，则服务**必须**使请求失败。 + +页面令牌内容**应该**是可在网址中安全使用的 base64 编码的协议缓冲区。 这使得内容可以在避免兼容性问题的情况下演变。如果页面令牌包含潜在的敏感信息，则**应该**对该信息进行加密。服务**必须**通过以下方法之一防止篡改页面令牌导致数据意外暴露： + +- 要求在后续请求中重新指定查询参数。 +- 仅在页面令牌中引用服务器端会话状态。 +- 加密并签署页面令牌中的查询参数，并在每次调用时重新验证并重新授权这些参数。 + +分页的实现也**可以**提供名为 `total_size` 的 `int32` 字段中的项目总数。 + +## 列出子集合 + +有时，API 需要让客户跨子集执行 `List/Search` 操作。例如，“API 图书馆”有一组书架，每个书架都有一系列书籍，而客户希望在所有书架上搜索某一本书。在这种情况下，建议在子集合上使用标准 `List`，并为父集合指定通配符集合 ID `"-"`。对于“API 图书馆”示例，我们可以使用以下 REST API 请求： + +``` +GET https://library.googleapis.com/v1/shelves/-/books?filter=xxx +``` + +**注意**：选择 `"-"` 而不是 `"*"` 的原因是为了避免需要进行 URL 转义。 + +## 从子集合中获取唯一资源 + +有时子集合中的资源具有在其父集合中唯一的标识符。此时，在不知道哪个父集合包含它的情况下使用 `Get` 检索该资源可能很有用。在这种情况下，建议对资源使用标准 `Get`，并为资源在其中是唯一的所有父集合指定通配符集合 ID `"-"`。例如，在 API 图书馆中，如果书籍在所有书架上的所有书籍中都是唯一的，我们可以使用以下 REST API 请求： + +``` +GET https://library.googleapis.com/v1/shelves/-/books/{id} +``` + +响应此调用的资源名称**必须**使用资源的规范名称，并使用实际的父集合标识符而不是每个父集合都使用 `"-"`。例如，上面的请求应返回名称为 `shelves/shelf713/books/book8141`（而不是 `shelves/-/books/book8141`）的资源。 + +## 排序顺序 + +如果 API 方法允许客户端指定列表结果的排序顺序，则请求消息**应该**包含一个字段： + +```proto +string order_by = ...; +``` + +字符串值**应该**遵循 SQL 语法：逗号分隔的字段列表。例如：`"foo,bar"`。默认排序顺序为升序。要将字段指定为降序，**应该**将后缀 `" desc"` 附加到字段名称中。例如：`"foo desc,bar"`。 + +语法中的冗余空格字符是无关紧要的。 `"foo,bar desc"` 和 `" foo , bar desc "` 是等效的。 + +## 提交验证请求 + +如果 API 方法有副作用，并且需要验证请求而不导致此类副作用，则请求消息**应该**包含一个字段： + +```proto +bool validate_only = ...; +``` + +如果此字段设置为 `true`，则服务器**不得**执行任何副作用，仅执行与完整请求一致的特定于实现的验证。 + +如果验证成功，则**必须**返回 `google.rpc.Code.OK`，并且任何使用相同请求消息的完整请求**不得**返回 `google.rpc.Code.INVALID_ARGUMENT`。请注意，由于其他错误（例如 `google.rpc.Code.ALREADY_EXISTS` 或争用情况），请求可能仍然会失败。 + +## 请求重复 + +对于网络 API，幂等 API 方法是首选，因为它们可以在网络故障后安全地重试。但是，某些 API 方法不能轻易为幂等（例如创建资源），并且需要避免不必要的重复。对于此类用例，请求消息**应**包含唯一 ID（如 UUID），服务器将使用该 ID 检测重复并确保请求仅被处理一次。 + +```proto +// A unique request ID for server to detect duplicated requests. +// This field **should** be named as `request_id`. +string request_id = ...; +``` + +如果检测到重复请求，则服务器**应该**返回先前成功请求的响应，因为客户端很可能未收到先前的响应。 + +## 枚举默认值 + +每个枚举定义**必须**以 `0` 值条目开头，当未明确指定枚举值时，**应**使用该条目。API **必须**记录如何处理 `0` 值。 + +如果存在共同的默认行为，则**应该**使用枚举值 `0`，并且 API 应记录预期的行为。 + +如果没有共同的默认行为，则枚举值 `0` **应该**被命名为 `ENUM_TYPE_UNSPECIFIED` 并且在使用时**应该**使用错误 `INVALID_ARGUMENT` 拒绝。 + +```proto +enum Isolation { + // Not specified. + ISOLATION_UNSPECIFIED = 0; + // Reads from a snapshot. Collisions occur if all reads and writes cannot be + // logically serialized with concurrent transactions. + SERIALIZABLE = 1; + // Reads from a snapshot. Collisions occur if concurrent transactions write + // to the same rows. + SNAPSHOT = 2; + ... +} + +// When unspecified, the server will use an isolation level of SNAPSHOT or +// better. +Isolation level = 1; +``` + +惯用名称**可以**用于 `0` 值。例如，`google.rpc.Code.OK` 是指定缺少错误代码的惯用方法。在这种情况下，在枚举类型的上下文中，`OK` 在语义上等同于 `UNSPECIFIED`。 + +如果存在本质上合理且安全的默认值，则该值**可以**用于“0”值。例如，`BASIC` 是[资源视图](https://cloud.google.com/apis/design/design_patterns?hl=zh-cn#resource_view)枚举中的“0”值。 + +## 语法规则 + +在 API 设计中，通常需要为某些数据格式定义简单的语法，例如可接受的文本输入。为了在所有 API 中提供一致的开发者体验并减少学习曲线，API 设计人员**必须**使用以下扩展巴科斯范式（Extended Backus-Naur Form，简写为“EBNF”）语法的变体来定义这样的语法： + +``` +Production = name "=" [ Expression ] ";" ; +Expression = Alternative { "|" Alternative } ; +Alternative = Term { Term } ; +Term = name | TOKEN | Group | Option | Repetition ; +Group = "(" Expression ")" ; +Option = "[" Expression "]" ; +Repetition = "{" Expression "}" ; +``` + +**注意**：`TOKEN` 表示在语法之外定义的终端符号。 + +## 整数类型 + +在 API 设计中，**不应该**使用 `uint32` 和 `fixed32` 等无符号整数类型，因为某些重要的编程语言和系统（如 Java，JavaScript 和 OpenAPI）不太支持它们。并且它们更有可能导致溢出错误。另一个问题是，不同的 API 很可能会对同一事件使用不匹配的有符号和无符号类型。 + +当有符号整数类型用于负值无意义的事物（例如大小或超时）时，值 `-1` （且**仅有** `-1`）**可以**用于表示特殊含义，例如文件结尾 (EOF)、无限超时、无限制配额限制或未知年龄。**必须**明确记录此类用法以避免混淆。如果隐式默认值 `0` 的行为不是非常明显，API 提供方也应对其进行记录。 + +## 部分响应 + +有时，API 客户端只需要响应消息中的特定数据子集。为了支持此类用例，某些 API 平台为部分响应提供原生支持。Google API Platform 通过响应字段掩码来支持它。任何 REST API 调用都有一个隐式系统查询参数 `$fields`，它是 `google.protobuf.FieldMask` 值的 JSON 表示形式。在发送回客户端之前，响应消息将由 `$fields` 过滤。API 平台会自动为所有 API 方法处理此逻辑。 + +``` +GET https://library.googleapis.com/v1/shelves?$fields=name +``` + +## 资源视图 + +为了减少网络流量，有时可允许客户端限制服务器应在其响应中返回的资源部分，即返回资源视图而不是完整的资源表示形式。API 中的资源视图支持是通过向方法请求添加一个参数来实现的，该参数允许客户端指定希望在响应中接收的资源视图。 + +该参数具有以下特点： + +- **应该**是 `enum` 类型 +- **必须**命名为 `view`。 + +枚举的每个值定义将在服务器的响应中返回资源的哪些部分（哪些字段）。为每个 `view` 值返回的具体内容是由实现定义的，**应该**在 API 文档中指定。 + +```proto +package google.example.library.v1; + +service Library { + rpc ListBooks(ListBooksRequest) returns (ListBooksResponse) { + option (google.api.http) = { + get: "/v1/{name=shelves/*}/books" + } + }; +} + +enum BookView { + // Not specified, equivalent to BASIC. + BOOK_VIEW_UNSPECIFIED = 0; + + // Server responses only include author, title, ISBN and unique book ID. + // The default value. + BASIC = 1; + + // Full representation of the book is returned in server responses, + // including contents of the book. + FULL = 2; +} + +message ListBooksRequest { + string name = 1; + + // Specifies which parts of the book resource should be returned + // in the response. + BookView view = 2; +} +``` + +此构造将映射到网址中，例如： + +``` +GET https://library.googleapis.com/v1/shelves/shelf1/books?view=BASIC +``` + +您可以在本设计指南的[标准方法](https://cloud.google.com/apis/design/standard_methods?hl=zh-cn)一章中找到有关定义方法、请求和响应的更多信息。 + +## ETag + +ETag 是一个不透明标识符，允许客户端发出条件请求。 为了支持 ETag，API **应该**在资源定义中包含字符串字段 `etag`，并且其语义**必须**符合 ETag 的常见用法。通常，`etag` 包含服务器计算的资源的指纹。如需了解更多详情，请参阅 [Wikipedia](https://en.wikipedia.org/wiki/HTTP_ETag) 和 [RFC 7232](https://tools.ietf.org/html/rfc7232#section-2.3)。 + +ETag 可以被强验证或弱验证，其中弱验证的 ETag 以 `W/` 为前缀。在本上下文中，强验证意味着具有相同 ETag 的两个资源具有逐字节相同的内容和相同的额外字段（即，内容-类型）。这意味着强验证的 ETag 允许缓存部分响应并在稍后组合。 + +相反，具有相同的弱验证 ETag 值的资源意味着表示法在语义上是等效的，但不一定逐字节相同，因此不适合字节范围请求的响应缓存。 + +例如： + +``` +// This is a strong ETag, including the quotes. +"1a2f3e4d5b6c7c" +// This is a weak ETag, including the prefix and quotes. +W/"1a2b3c4d5ef" +``` + +请牢记，引号是 ETag 值的一部分并且必须存在，以符合 [RFC 7232](https://tools.ietf.org/html/rfc7232#section-2.3)。这意味着 ETag 的 JSON 表示法最终会对引号进行转义。例如，ETag 在 JSON 资源正文中表示为： + +``` +// Strong +{ "etag": "\"1a2f3e4d5b6c7c\"", "name": "...", ... } +// Weak +{ "etag": "W/\"1a2b3c4d5ef\"", "name": "...", ... } +``` + +ETag 中允许的字符摘要： + +- 仅限可打印的 ASCII + - RFC 2732 允许的非 ASCII 字符，但对开发者不太友好 +- 不能有空格 +- 除上述位置外，不能有双引号 +- 遵从 RFC 7232 的推荐，避免使用反斜杠，以防止在转义时出现混淆 + +## 输出字段 + +API 可能希望区分客户端提供的作为输入的字段，以及由服务器返回的仅在特定资源上输出的字段。对于仅限输出的字段，**应该**记录字段属性。 + +请注意，如果在请求中设置了或在 `google.protobuf.FieldMask` 中包括了仅输出字段，则服务器**必须**接受请求并且不出现错误。服务器**必须**忽略仅限输出字段的存在及任何提示。此建议的原因是因为客户端经常将服务器返回的资源作为另一个请求的输入重新使用，例如，检索到的 `Book` 稍后将在 UPDATE 方法中被重新使用。如果对仅限输出字段进行验证，则会导致客户端需要额外清除仅限输出字段。 + +```proto +message Book { + string name = 1; + // Output only. + Timestamp create_time = 2; +} +``` + +## 单例资源 + +当只有一个资源实例存在于其父资源中（如果没有父资源，则在 API 中）时，可以使用单例资源。 + +单例资源**必须**省略标准的 `Create` 和 `Delete` 方法；在创建或删除父资源时即隐式创建或删除了单例资源（如果没有父资源，则单例资源隐式存在）。**必须**使用标准的 `Get` 和 `Update` 方法，以及任意适合您的用例的自定义方法访问该资源。 + +例如，具有 `User` 资源的 API 可以将每个用户的设置公开为 `Settings` 单例。 + +```proto +rpc GetSettings(GetSettingsRequest) returns (Settings) { + option (google.api.http) = { + get: "/v1/{name=users/*/settings}" + }; +} + +rpc UpdateSettings(UpdateSettingsRequest) returns (Settings) { + option (google.api.http) = { + patch: "/v1/{settings.name=users/*/settings}" + body: "settings" + }; +} + +[...] + +message Settings { + string name = 1; + // Settings fields omitted. +} + +message GetSettingsRequest { + string name = 1; +} + +message UpdateSettingsRequest { + Settings settings = 1; + // Field mask to support partial updates. + FieldMask update_mask = 2; +} +``` + +## 流式半关闭 + +对于任何双向或客户端流传输 API，服务器**应该**依赖 RPC 系统提供的、客户端发起的半关闭来完成客户端流。无需定义显式完成消息。 + +客户端需要在半关闭之前发送的任何信息都**必须**定义为请求消息的一部分。 + +## 网域范围名称 + +网域范围名称是以 DNS 域名为前缀的实体名称，旨在防止名称发生冲突。当不同的组织以分散的方式定义其实体名称时，这种设计模式很有用。其语法类似于没有架构的 URI。 + +网域范围名称广泛用于 Google API 和 Kubernetes API，例如： + +- Protobuf `Any` 类型的表示形式：`type.googleapis.com/google.protobuf.Any` +- Stackdriver 指标类型：`compute.googleapis.com/instance/cpu/utilization` +- 标签键：`cloud.googleapis.com/location` +- Kubernetes API 版本：`networking.k8s.io/v1` +- `x-kubernetes-group-version-kind` OpenAPI 扩展程序中的 `kind` 字段。 + +## 布尔值与枚举与字符串 + +在设计 API 方法时，您通常会为特定功能（例如启用跟踪或停用缓存）提供一组选择。实现此目的的常用方法是引入 `bool`、`enum` 或 `string` 类型的请求字段。对于给定用例，要使用哪种正确的类型通常不是很明显。推荐的选项如下： + +- 如果我们希望获得固定的设计并且有意不想扩展该功能，请使用 `bool` 类型。例如 `bool enable_tracing` 或 `bool enable_pretty_print`。 +- 如果我们希望获得灵活的设计，但不希望该设计频繁更改，请使用 `enum` 类型。一般的经验法则是枚举定义每年仅更改一次或更少。例如 `enum TlsVersion` 或 `enum HttpVersion`。 +- 如果我们采用开放式设计或者可以根据外部标准频繁更改设计，请使用 `string` 类型。必须明确记录支持的值。例如： + - 由 [Unicode 区域](http://www.unicode.org/reports/tr35/#unicode_region_subtag)定义的 `string region_code`。 + - 由 [Unicode 语言区域](http://www.unicode.org/reports/tr35/#Unicode_locale_identifier)定义的 `string language_code`。 + +## 数据保留 + +在设计 API 服务时，数据保留是服务可靠性相当重要的部分。通常，用户数据会被软件错误或人为错误误删。没有数据保留和相应的取消删除功能，一个简单的错误就可能对业务造成灾难性的影响。 + +通常，我们建议 API 服务采用以下数据保留政策： + +- 对于用户元数据、用户设置和其他重要信息，应保留 30 天的数据。例如，监控指标、项目元数据和服务定义。 +- 对于大量用户内容，应保留 7 天的数据。例如，二进制 blob 和数据库表。 +- 对于暂时性状态或费用昂贵的存储服务，如果可行，应保留 1 天的数据。例如，Memcache 实例和 Redis 服务器。 + +在数据保留期限期间，可以删除数据而不会丢失数据。如果免费提供数据保留的成本很高，则服务可以提供付费的数据保留。 + +## 大型载荷 + +联网 API 通常依赖于多个网络层作为其数据路径。大多数网络层对请求和响应大小有硬性限制。32 MB 是很多系统中常用的限制。 + +在设计处理大于 10 MB 的载荷的 API 方法时，我们应该谨慎选择合适的策略，以确保易用性和满足未来增长的需求。对于 Google API，我们建议使用流式传输或媒体上传/下载来处理大型载荷。如使用流式传输，服务器会逐步地同步处理大量数据，例如 Cloud Spanner API。如使用媒体，大量数据会流经大型存储系统（如 Google Cloud Storage），服务器可以异步处理数据，例如 Google Drive API。 + + + +# 文档 + +本部分介绍了如何向 API 添加内嵌文档。大多数 API 还拥有概览、教程和简要参考文档，这些内容本设计指南并不涉及。如需了解 API、资源和方法命名，请参阅[命名惯例](https://cloud.google.com/apis/design/naming_convention?hl=zh-cn)。 + + + +## proto 文件中的注释格式 + +使用常用的 Protocol Buffers `//` 注释格式向 `.proto` 文件添加注释。 + +```proto +// Creates a shelf in the library, and returns the new Shelf. +rpc CreateShelf(CreateShelfRequest) returns (Shelf) { + option (google.api.http) = { post: "/v1/shelves" body: "shelf" }; +} +``` + +## 服务配置中的注释 + +另一种向 `.proto` 文件添加文档注释的方法是，您可以在其 YAML 服务配置文件中向 API 添加内嵌文档。如果两个文件中都记录了相同的元素，则 YAML 文件中的文档将优先于 `.proto` 中的文档。 + +```yaml +documentation: + summary: Gets and lists social activities + overview: A simple example service that lets you get and list possible social activities + rules: + - selector: google.social.Social.GetActivity + description: Gets a social activity. If the activity does not exist, returns Code.NOT_FOUND. +... +``` + +如果您有多个服务使用相同的 `.proto` 文件，并且您希望提供服务专用文档，则可能需要使用此方法。YAML 文档规则还允许您向 API 说明添加更详细的 `overview`。但一般首选向 `.proto` 文件添加文档注释。 + +与向 `.proto` 添加注释一样，您可以使用 Markdown 在 YAML 文件注释中提供其他格式设置。 + +## API 说明 + +API 说明是说明 API 功能的短语（以行为动词开头）。在您的 `.proto` 文件中，API 说明作为注释添加到相应的 `service` 中，如以下示例所示： + +```proto +// Manages books and shelves in a simple digital library. +service LibraryService { +... +} +``` + +以下是一些 API 说明示例： + +- 与世界各地的朋友分享最新动态、照片、视频等。 +- 访问云托管的机器学习服务，轻松构建响应数据流的智能应用。 + +## 资源说明 + +资源说明是描述资源表示的内容的句子。如果您需要添加更多细节，请使用更多句子。在您的 `.proto` 文件中，API 说明作为注释添加到相应的消息类型中，如以下示例所示： + +```proto +// A book resource in the Library API. +message Book { + ... +} +``` + +以下是一些资源说明示例： + +- 用户待办事项列表中的一项任务。每项任务具有唯一的优先级。 +- 用户日历上的一个事件。 + +## 字段和参数说明 + +描述字段或参数定义的名词短语，如以下示例所示： + +- 本系列的主题数量。 +- 经纬度坐标的精度，以米为单位。 必须是非负数。 +- 标记是否为本系列的提交资源返回附件网址值。`series.insert` 的默认值为 `true`。 +- 投票信息的容器。仅在记录投票信息时出现。 +- 目前未使用或已弃用。 + +字段和参数说明**应该**描述哪些值有效和无效。请记住，工程师们会通过一切可能的途径导致服务失败，并且他们无法读取底层代码来澄清任何不清楚的信息。 + +对于字符串，说明**应该**描述语法和允许的字符以及任何所需的编码。例如： + +- 集合 [A-a0-9] 中的 1-255 个字符 +- 遵循 RFC 2332 惯例且以 / 开头的有效网址路径字符串。长度上限为 500 个字符。 + +说明**应该**指定任何默认值或行为，但**可以**省略描述实际为 null 的默认值。 + +如果字段值是**必需**、**仅限输入**、**仅限输出**，则**應該**在字段说明开头记录这些值。默认情况下，所有字段和参数都是可选的。例如： + +```proto +message Table { + // Required. The resource name of the table. + string name = 1; + + // Input only. Whether to validate table creation without actually doing it. + bool validate_only = 2; + + // Output only. The timestamp when the table was created. Assigned by + // the server. + google.protobuf.Timestamp create_time = 3; + + // The display name of the table. + string display_name = 4; +} +``` + +**注意**：只要可行且有用，字段说明就**应该**提供示例值。 + +## 方法说明 + +方法说明是一个指明方法效果及其操作资源的句子。它通常以第三人称的现在时态动词（即以“s”结尾的动词）开头。如果需要添加详细信息，请使用更多句子。以下是一些示例： + +- 列出已通过身份验证的用户的日历事件。 +- 使用请求中包含的数据来更新日历事件。 +- 从已通过身份验证的用户的位置历史记录中删除一个位置记录。 +- 在已通过身份验证的用户的位置历史记录中，使用请求中包含的数据创建或更新一个位置记录。如果已存在具有相同时间戳值的位置资源，则所提供的数据会覆盖现有数据。 + +## 所有说明的核对清单 + +确保每个说明都是简短而完整的，能够让用户在没有其他关于该 API 的信息的情况下所理解。在大多数情况下，都有更多需要说明的内容，而不仅仅是重复显而易见的信息。例如，`series.insert` 方法的说明不应该只是说“插入一个序列”。虽然您的命名应该包含信息，但大多数读者还是会阅读说明，因为他们需要名称之外的更多信息。如果您不确定需要在说明中包括哪些其他内容，请尝试回答以下所有相关问题： + +- 它是什么？ +- 如果成功了它会执行什么操作？如果失败了它会执行什么操作？什么可能导致它失败及如何导致它失败？ +- 它具有幂等性吗？ +- 它的单位是什么？（例如：米、度、像素。） +- 它接受什么范围的值？此范围是否包含边界值？ +- 它有什么副作用？ +- 应该如何使用它？ +- 可能会导致它失败的常见错误有哪些？ +- 它总是存在吗？（例如：“用于投票信息的容器。仅在记录投票信息时存在。”） +- 它有默认设置吗？ + +## 惯例 + +本部分列出了文本说明和文档的一些使用惯例。例如，在说明标识符时，请使用“ID”（全部大写），而不是“Id”或“id”。在引用该数据格式时，请使用“JSON”而不是“Json”或“json”。以 `code font` 显示所有字段/参数名称。将文字字符串值以 `code font` 表示，并将其放入引号中。 + +- ID +- JSON +- RPC +- REST +- `property_name` 或 `"string_literal"` +- `true`/`false` + +### 要求级别 + +要设置预期或陈述要求级别，请使用以下术语：必须、不得、必需、应、不应、应该、不应该、建议、可以和可选。 + +如需了解这些术语的含义，请参阅 [RFC 2119](https://www.ietf.org/rfc/rfc2119.txt) 中的定义。 建议您在 API 文档中包含 RFC 摘要中的语句。 + +确定哪些术语既符合您的要求，又能为开发者提供灵活性。如果从技术上来说，您的 API 还支持其他选项，则请勿使用绝对术语（如“必须”）。 + +## 语言风格 + +如[命名惯例](https://cloud.google.com/apis/design/naming_convention?hl=zh-cn)中所述，我们建议在编写文档注释时使用简单、一致的词汇和风格。注释应该易于母语非英语的读者理解，所以应避免行话、俚语、复杂的隐喻、引用流行文化，或包含任何其他不容易翻译的内容。使用友好、专业的风格直接与阅读注释的开发者交流，并尽可能简明扼要。请记住，大多数读者的目的是了解如何使用 API，而不是阅读您的文档！ + + + diff --git "a/docs/collection/\344\272\254\344\270\234\350\220\245\351\224\200\346\212\225\346\224\276\345\271\263\345\217\260\344\275\216\344\273\243\347\240\201\357\274\210Low-Code\357\274\211\345\256\236\350\267\265.md" "b/docs/collection/\344\272\254\344\270\234\350\220\245\351\224\200\346\212\225\346\224\276\345\271\263\345\217\260\344\275\216\344\273\243\347\240\201\357\274\210Low-Code\357\274\211\345\256\236\350\267\265.md" new file mode 100755 index 0000000000..c9f391a2d3 --- /dev/null +++ "b/docs/collection/\344\272\254\344\270\234\350\220\245\351\224\200\346\212\225\346\224\276\345\271\263\345\217\260\344\275\216\344\273\243\347\240\201\357\274\210Low-Code\357\274\211\345\256\236\350\267\265.md" @@ -0,0 +1,103 @@ +## 前言 + +过去的2020年，一场忽如其来的“新冠”病毒，迅速的重塑了当今整个世界的方方面面，作为现代社会中很重要的领域之一——软件科技行业，当然也深受其影响。 + +各个软件科技公司都开始着重考虑节省人力成本和提高人力效率。在不可逆的社会浪潮之中，大家都开始想方设法地翻找各自的“工具箱”，看看有没有什么工具可以帮助各自在巨浪中生存下去。在大家的努力下，一条2014年由Forrester提出的“低代码（Low-Code）”技术构想被大家所重新拾起，成为了互联网软件开发界新进“红人”。 + +本文将介绍低代码的相关概念，以及结合京东营销投放平台自身情况，把从0到1落地低代码的实践进行介绍，从这个项目的角度进行经验分享。 + +## 什么是低代码 + +“Low-Code” 是什么？ + +如果你第一次听说，会不会猜测这是指代码写的很差劲的贬义词？或者说是不是指很底层编译语言？ + +但是其都不是，那到底是什么意思？ + +作为一名搜商比情商还高的程序员，本能的第一时间进行了网上搜索，首先找到的是全球网络上最大且最受大众欢迎的参考工具书wikipedia上的一个词条：Low-code development platform。 + +wikipedia词条上给的定义是： + +> A low-code development platform (LCDP) is software that provides an development environment programmers use to create application software through graphical user interfaces and configuration instead of traditional hand-coded computer programming. + +从中我们可以提炼两个关键信息，1. 是一个应用软件的开发环境。2. 提供的是易用友好的可视化方式进行编程。 + +## 解决了什么 + +从低代码的定义上，我们可以看出低代码平台可以解决三大软件开发方面的问题：减少重复工作、聚合平台能力、形成一体化生态。 + +**减少重复工作**，设计合理的低代码平台会尽可能保证服务能力的原子性，可以消灭绝大部分繁琐和重复的代码，最大化软件复用。再往深一些看，低代码不只是少些代码而已，代码写的少，bug也就越少，要测的代码也少了，后续的应用构建、部署、管理等多个环节都减少了，对于软件开发的整个生命周期的人力都节省了。 + +**聚合平台能力**，低代码平台强调服务能力的复用。我们一般认为使用过的成熟服务相对新服务是更可信的，更多的服务复用，意味着可以提高服务的性能、成本、稳定性、安全性、可持续发展能力等。这同时帮助平台在能力服务方面越来越丰富、健壮，更好的服务可以引入更多业务使用，继而继续扩展服务，形成良性循环。 + +**形成一体化生态**，当聚合了越来越多的平台能力后，平台可以提供多层次多粒度复用手段，比如页面组件库、逻辑函数库、应用模板库等，引入更多的用户，甚至可以引入第三方研发一起实现平台共建，从而让平台服务的聚合逐步升级成为设计、研发、使用三位一体的平台生态。 + +### 可能存在的问题 + +软件开发界有一句名言： + +> 软件开发没有“银弹”。 + +意思是说对于软件开发来说，没有一种方法是可以适用所有场景的。当然，低代码也有一些自身的局限。 + +首先，低代码或许可以降级开发门槛，但**复杂度并不会降低**。可视化开发的自由度越高，组件粒度就越细，配置的复杂度就越高。同时，平台上的各种可视化组件、逻辑动作和部署环境都是黑盒，如果内部出问题无法排查和解决。 + +其次，由于低代码更多考虑的是服务复用性、通用性，导致它更加适用于一些通用业务，对于一些定制化要求很高的需求并不友好。所以，对于低代码平台的设计，我们在实践方面也一直在寻求“易用”和“复杂度”之间的平衡。 + +### 为什么我们要做低代码改造 + +在解释这个问题之前，先简单介绍一下京东营销投放平台的素材管理功能是什么。 + +一句话概括，就是用户可以通过系统，维护自己的素材池（素材可以是商品、广告、优惠券等），针对素材进行增删改查，素材池以一个整体在前台进行投放，投放时可以使用各种策略：个性化千人千面、人工干预、热度等。 + +对于素材管理的多种服务，我们可以分成两大类：**通用标准能力和自定义业务能力**。 + +通用能力包括素材组、场次、素材方面的增删改查管理，个性化千人千面、人工干预、热度等的投放策略。 + +定制化的能力一般都是和具体素材类型相关的，比如数据校验、数据封装、数据渲染等。 + +![](https://tva1.sinaimg.cn/large/008i3skNly1gx67h248amj30u007t756.jpg) + +通过这样的分析，我们可以发现对于通用的标准能力是非常适合使用低代码设计的，标准化沉淀下来的通用能力，可以在不同的素材上进行复用。新素材的开发只需要关注业务向的定制能力，这样大大减少了新素材接入的开发成本。 + +### 架构设计 + +对于低代码的架构设计，核心点应该为开发者尽可能屏蔽底层技术细节、减少不必要的技术复杂度，并支撑其更好地应对业务复杂度（满足灵活通用的业务场景需求）。 + +设计的难点关键处在于如何解耦业务和技术复杂度，做到让基础设施下沉，形成能力的标准化可复用。 + +![图片](https://tva1.sinaimg.cn/large/008i3skNly1gx67heixlwj30u00t4763.jpg) + +在我们分析的基础上，对于京东营销投放平台的素材管理能力进行了低代码平台设计。下面是我们的架构图： + +![图片](https://tva1.sinaimg.cn/large/008i3skNly1gx67hliim4j30u00n9djc.jpg) + +架构上，在后端服务方面，我们标准化沉淀下来素材管理的通用能力，形成标准化流程；在定制能力上，我们支持在素材类型垂直的进行自定义能力开发接入。 + +中间层我们设计一套适配层，针对服务后端和前端的标准化对接。 + +在前端设计上，我们主要引入可视化表单和列表的配置能力，把前端的素材内容形成各种各样可复用的组件，通过灵活搭建的方式进行前端页面渲染配置（注：图中drip是我们部门自研的一套工具集，包含脚手架，组件等前端常用功能）。 + +### 京东营销投放平台低代码实践成果 + +低代码平台上线以后，对于产品、研发、测试的工作方式都进行了转变，从原来传统的关注全流程业务定制开发，转变为了主要平台研发关注基础能力开发和轻量业务研发关注业务逻辑开发。 + +最终我们上线的低代码平台，核心可视化配置分成两部分，**表单配置和列表页配置**。 + +![图片](https://tva1.sinaimg.cn/large/008i3skNly1gx67hz1qv8j30u009tq3r.jpg) + +表单配置核心包括基础组件选择、表单页配置和组件数据配置。基础组件列出了平台所支持的所有可视化组件；表单页配置支持动态配置表单项；组件配置针对表单页上配置的表单项，进行进一步的数据绑定配置，支持和后端协议的打通。 + +![图片](https://tva1.sinaimg.cn/large/008i3skNly1gx67hz1qv8j30u009tq3r.jpg) + +列表页配置核心包括基础组件选择和表格布局配置。基础组件列出列表渲染所支持的各种可视化组件，表格布局支持动态配置列表展示项。 + +![图片](https://tva1.sinaimg.cn/large/008i3skNly1gx67i50qgoj30u00dn0ti.jpg) + +投放低代码平台上线以来，**大大提高到了研发人效、增强了系统稳定性**。以今年2021年上半年最忙的五月份来说（支持年中618大促），一个月时间投放平台支持接入了14种新素材，整体单素材接入需要的平均研发人力从原来的8人日降低到2人日。 + +## 结语 + +数据显示，中国企业的数字化转型市场需求大概需要5亿个新的应用或者APP，这个庞大的需求，如果按照传统的软件开发模式，不仅成本高昂，产品的输出和供给也受到限制。低代码平台的出现，在某一些场景下，**发挥低代码配置灵活和复用性高的特点**，可以更快更好的满足市场需求。 + +本文也是趁着低代码潮流，在2021年初起对京东营销投放平台的素材管理能力进行从0到1 低代码改造，感谢整个团队的付出和部门内部的支持，在日常工作较饱和的情况下，愿意支持内部系统创新。在一路的摸索过程中，也遇到了很多问题，收获了很多的宝贵经验，最终使得低代码平台1.0版本很快的在4月份上线，2.0版本在7月份上线，现在的低代码平台已经完全覆盖素材接入管理流程，相应的人效红利也在逐步的展现，让团队有更多的时间精力去做其他的探索。 \ No newline at end of file diff --git "a/docs/collection/\345\233\276\350\247\243git \345\270\270\347\224\250\345\221\275\344\273\244.md" "b/docs/collection/\345\233\276\350\247\243git \345\270\270\347\224\250\345\221\275\344\273\244.md" new file mode 100644 index 0000000000..d9900cc79a --- /dev/null +++ "b/docs/collection/\345\233\276\350\247\243git \345\270\270\347\224\250\345\221\275\344\273\244.md" @@ -0,0 +1,138 @@ +本文图解 Git 中的最常用命令，如果你稍微理解 Git 的工作原理，这篇文章能够让你理解的更透彻。 + +## 基本用法 + +![](https://mmbiz.qpic.cn/mmbiz_png/v1JN0W4OpXjuEjicbH8PQibIRkiaNHr9ibqusjLdmEbIVyxtkjfdtVF9qLMkEkVD69ZwzCuOHiaEFczrkAtib8ic7JznA/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +上面的四条命令在工作目录、暂存目录(也叫做索引)和仓库之间复制文件。 + +- `git add files` 把当前文件放入暂存区域。 +- `git commit` 给暂存区域生成快照并提交。 +- `git reset -- files` 用来撤销最后一次 `git add files`，你也可以用 `git reset` 撤销所有暂存区域文件。 +- `git checkout -- files` 把文件从暂存区域复制到工作目录，用来丢弃本地修改。 + +你可以用 git reset -p, git checkout -p, or git add -p进入交互模式。 + +也可以跳过暂存区域直接从仓库取出文件或者直接提交代码。 + +![图片](https://mmbiz.qpic.cn/mmbiz_png/v1JN0W4OpXjuEjicbH8PQibIRkiaNHr9ibqud6LC2aicjNKDWT21Hia4rNsykKdUKvNLicCGEb17M3RSz3ica06iaGsJ26Q/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +- `git commit -a` 相当于运行 git add 把所有当前目录下的文件加入暂存区域再运行。 +- `git commit files` 进行一次包含最后一次提交加上工作目录中文件快照的提交。并且文件被添加到暂存区域。 +- `git checkout HEAD -- files` 回滚到复制最后一次提交。 + +## 约定 + +后文中以下面的形式使用图片。 + +![](https://mmbiz.qpic.cn/mmbiz_png/v1JN0W4OpXjuEjicbH8PQibIRkiaNHr9ibqu7NiaX2TCRf2LmScVxMq6Vuuu27PCD5zQARVFmicaicOTqmf58KDHjsWFA/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +绿色的5位字符表示提交的ID，分别指向父节点。分支用橘色显示，分别指向特定的提交。当前分支由附在其上的HEAD标识。这张图片里显示最后5次提交，ed489是最新提交。main分支指向此次提交，另一个stable分支指向祖父提交节点。 + +## 命令详解 + +### Diff + +有许多种方法查看两次提交之间的变动。下面是一些示例。 + +![](https://mmbiz.qpic.cn/mmbiz_png/v1JN0W4OpXjuEjicbH8PQibIRkiaNHr9ibqulXh7n4XFXW9Su6bIuaYrO1QVSG0DUx3XQV39r47DtTibzpKMWybdCaA/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +### Commit + +提交时，git用暂存区域的文件创建一个新的提交，并把此时的节点设为父节点。然后把当前分支指向新的提交节点。下图中，当前分支是main。在运行命令之前，main指向ed489，提交后，main指向新的节点f0cec并以ed489作为父节点。 + +![图片](https://mmbiz.qpic.cn/mmbiz_png/v1JN0W4OpXjuEjicbH8PQibIRkiaNHr9ibquicW9XtH5x8uZYXgxTibAzI6NMHyYQicTQpx088lCjODPjyntB6fxcqEwA/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +即便当前分支是某次提交的祖父节点，git会同样操作。下图中，在main分支的祖父节点stable分支进行一次提交，生成了1800b。这样，stable分支就不再是main分支的祖父节点。此时，**合并** (或者 **衍合**) 是必须的。 + +![图片](https://mmbiz.qpic.cn/mmbiz_png/v1JN0W4OpXjuEjicbH8PQibIRkiaNHr9ibquaQAps6sd0gF62lnMliafhlMU1JnZ0hwmroPmw3zA4mWBYSpS6JEPbAQ/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +如果想更改一次提交，使用 `git commit --amend`。git会使用与当前提交相同的父节点进行一次新提交，旧的提交会被取消。 + +![图片](https://mmbiz.qpic.cn/mmbiz_png/v1JN0W4OpXjuEjicbH8PQibIRkiaNHr9ibquVBHmp05AuxQYOLNbyISvKFSpibicGXcN80BFQDGMfWeuFniaekxQw98Wg/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +另一个例子是分离HEAD提交，后文讲。 + +### Checkout + +checkout命令用于从历史提交（或者暂存区域）中拷贝文件到工作目录，也可用于切换分支。 + +当给定某个文件名（或者打开-p选项，或者文件名和-p选项同时打开）时，git会从指定的提交中拷贝文件到暂存区域和工作目录。比如，git checkout HEAD~ foo.c 会将提交节点HEAD~(即当前提交节点的父节点)中的foo.c复制到工作目录并且加到暂存区域中。（如果命令中没有指定提交节点，则会从暂存区域中拷贝内容。）注意当前分支不会发生变化。 + +![图片](https://mmbiz.qpic.cn/mmbiz_png/v1JN0W4OpXjuEjicbH8PQibIRkiaNHr9ibquYRJlVeV37BX7p2RjOnhiaicKrXmVA5cET0wsFEtibEmiaicia99uvrBhicDog/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +当不指定文件名，而是给出一个（本地）分支时，那么HEAD标识会移动到那个分支（也就是说，我们“切换”到那个分支了），然后暂存区域和工作目录中的内容会和HEAD对应的提交节点一致。新提交节点（下图中的a47c3）中的所有文件都会被复制（到暂存区域和工作目录中）；只存在于老的提交节点（ed489）中的文件会被删除；不属于上述两者的文件会被忽略，不受影响。 + +![图片](https://mmbiz.qpic.cn/mmbiz_png/v1JN0W4OpXjuEjicbH8PQibIRkiaNHr9ibquwAYoaS7XxctZHcMJRd9X2Uj0XuJJvEu97JCZE9zaicxrlv0icDxaIhAw/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +如果既没有指定文件名，也没有指定分支名，而是一个标签、远程分支、SHA-1值或者是像main~3类似的东西，就得到一个匿名分支，称作detached HEAD（被分离的HEAD标识）。这样可以很方便地在历史版本之间互相切换。比如说你想要编译1.6.6.1版本的git，你可以运行git checkout v1.6.6.1（这是一个标签，而非分支名），编译，安装，然后切换回另一个分支，比如说git checkout main。然而，当提交操作涉及到“分离的HEAD”时，其行为会略有不同，详情见在下面。 + +![图片](https://mmbiz.qpic.cn/mmbiz_png/v1JN0W4OpXjuEjicbH8PQibIRkiaNHr9ibqu2YZpLEkZIPkhQt9QhMSDgZEqqKRmUfJ88OEAJ4wHR2VEm7N4icOFN7A/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +### HEAD标识处于分离状态时的提交操作 + +当HEAD处于分离状态（不依附于任一分支）时，提交操作可以正常进行，但是不会更新任何已命名的分支。(你可以认为这是在更新一个匿名分支。) + +![图片](https://mmbiz.qpic.cn/mmbiz_png/v1JN0W4OpXjuEjicbH8PQibIRkiaNHr9ibqua9280QsWR9xas8oZ5VVqTicr8mKI9Xw4uicksmSMABnDxwXn2TJicQExQ/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +一旦此后你切换到别的分支，比如说main，那么这个提交节点（可能）再也不会被引用到，然后就会被丢弃掉了。注意这个命令之后就不会有东西引用2eecb。 + +![图片](https://mmbiz.qpic.cn/mmbiz_png/v1JN0W4OpXjuEjicbH8PQibIRkiaNHr9ibqupnjPcO08ibU5I35DVVPsx6y7KpbIklBRonjWOFDibZAKQuIMOpqVDm6w/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +但是，如果你想保存这个状态，可以用命令git checkout -b name来创建一个新的分支。 + +![图片](https://mmbiz.qpic.cn/mmbiz_png/v1JN0W4OpXjuEjicbH8PQibIRkiaNHr9ibquCxouWHoFCoQH8trhDRvSYXia7qjpAskdzFgpIwmGzMretUuwwsUAvBA/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +### Reset + +reset命令把当前分支指向另一个位置，并且有选择的变动工作目录和索引。也用来在从历史仓库中复制文件到索引，而不动工作目录。 + +如果不给选项，那么当前分支指向到那个提交。如果用--hard选项，那么工作目录也更新，如果用--soft选项，那么都不变。 + +![图片](https://mmbiz.qpic.cn/mmbiz_png/v1JN0W4OpXjuEjicbH8PQibIRkiaNHr9ibquR2ssrwZCkzWpAicgu6ka8coVOwia46EHpmnmibVjicrbxF6bX4VXfPiaLEw/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +如果没有给出提交点的版本号，那么默认用HEAD。这样，分支指向不变，但是索引会回滚到最后一次提交，如果用--hard选项，工作目录也同样。 + +![图片](https://mmbiz.qpic.cn/mmbiz_png/v1JN0W4OpXjuEjicbH8PQibIRkiaNHr9ibqu4aYibTEhy50rlJC7foIuM354xEGwPDa5wpVRkoUkZ2l8STF4xob4hjQ/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +如果给了文件名(或者 -p选项), 那么工作效果和带文件名的checkout差不多，除了索引被更新。 + +![图片](https://mmbiz.qpic.cn/mmbiz_png/v1JN0W4OpXjuEjicbH8PQibIRkiaNHr9ibqul6v6bPIWC2R2Hgq5RdurdLs2AkW0aZvZJz3EqMbo3hkb2G7T78VZibg/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +### Merge + +merge 命令把不同分支合并起来。合并前，索引必须和当前提交相同。如果另一个分支是当前提交的祖父节点，那么合并命令将什么也不做。另一种情况是如果当前提交是另一个分支的祖父节点，就导致fast-forward合并。指向只是简单的移动，并生成一个新的提交。 + +![图片](https://mmbiz.qpic.cn/mmbiz_png/v1JN0W4OpXjuEjicbH8PQibIRkiaNHr9ibqufNibdMOHpQeibRTtmOmNAkW2jGcMl1UsDWeuw2N4sKjrXDckg2FR2BFA/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +否则就是一次真正的合并。默认把当前提交(ed489 如下所示)和另一个提交(33104)以及他们的共同祖父节点(b325c)进行一次三方合并。结果是先保存当前目录和索引，然后和父节点33104一起做一次新提交。 + +![图片](https://mmbiz.qpic.cn/mmbiz_png/v1JN0W4OpXjuEjicbH8PQibIRkiaNHr9ibqu7GCEon8ZIjiaUnX3OOwibRJJnrPmdTwibGAHDibh9sthaYCjvYAFGAr27g/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +### Cherry Pick + +cherry-pick命令"复制"一个提交节点并在当前分支做一次完全一样的新提交。 + +![图片](https://mmbiz.qpic.cn/mmbiz_png/v1JN0W4OpXjuEjicbH8PQibIRkiaNHr9ibquoRiat32geHjtWMakEDd8Ldr65lLxb9zFN5aSR8ub2ogDeFPwibr6wFBQ/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +### Rebase + +衍合是合并命令的另一种选择。合并把两个父分支合并进行一次提交，提交历史不是线性的。衍合在当前分支上重演另一个分支的历史，提交历史是线性的。本质上，这是线性化的自动的 cherry-pick + +![图片](https://mmbiz.qpic.cn/mmbiz_png/v1JN0W4OpXjuEjicbH8PQibIRkiaNHr9ibqunYS5PT91Cgu74kKP9HK1biaUCxs7rvWT49u5OZIA0MAbLoTEIxicEKpA/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +上面的命令都在topic分支中进行，而不是main分支，在main分支上重演，并且把分支指向新的节点。注意旧提交没有被引用，将被回收。 + +要限制回滚范围，使用--onto选项。下面的命令在main分支上重演当前分支从169a6以来的最近几个提交，即2c33a。 + +![图片](https://mmbiz.qpic.cn/mmbiz_png/v1JN0W4OpXjuEjicbH8PQibIRkiaNHr9ibqu8oVsj0EWGa0IEXc1USnHJfl7GKJDbHkiaZj5k3XdwTSwZhfibcq4gqWg/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +同样有 `git rebase --interactive` 让你更方便的完成一些复杂操作，比如丢弃、重排、修改、合并提交。 + + + +## 技术说明 + +文件内容并没有真正存储在索引(.git/index)或者提交对象中，而是以blob的形式分别存储在数据库中(.git/objects)，并用SHA-1值来校验。索引文件用识别码列出相关的blob文件以及别的数据。对于提交来说，以树(tree)的形式存储，同样用对于的哈希值识别。树对应着工作目录中的文件夹，树中包含的树或者blob对象对应着相应的子目录和文件。每次提交都存储下它的上一级树的识别码。 + +如果用detached HEAD提交，那么最后一次提交会被 the reflog for HEAD 引用。但是过一段时间就失效，最终被回收，与git commit --amend或者git rebase很像。 \ No newline at end of file diff --git "a/docs/collection/\345\256\271\347\201\276 vs \345\244\207\344\273\275.md" "b/docs/collection/\345\256\271\347\201\276 vs \345\244\207\344\273\275.md" new file mode 100755 index 0000000000..90bd03c304 --- /dev/null +++ "b/docs/collection/\345\256\271\347\201\276 vs \345\244\207\344\273\275.md" @@ -0,0 +1,102 @@ +> 数据中心运行突发故障(如：天灾不可避免的灾难)是无法预测的，计算机里的数据就像扫雷游戏一样，十面埋伏充满雷区，随时都有可能Game Over，容灾备份就是数据安全的最后防线，但是你可以避免由数据中心发生故障而丢失数据引发的数据丢失的局面。 +> +> 今天为大家介绍“容灾和备份的区别”以及一些尽可能减少发生运行故障失败机会，并加强企业的数据备份环境的简单要点。 + + + +![](https://tva1.sinaimg.cn/large/008i3skNly1grd6t7zqqpj31900u0e84.jpg) + +### 什么是容灾 ？ + +容灾系统是指在相隔较远的异地，建立两套或多套功能相同的IT系统，互相之间可以进行健康状态监视和功能切换，当一处系统因意外（如火灾、地震等）停止工作时，整个应用系统可以切换到另一处，使得该系统功能可以继续正常工作。 + +容灾技术是系统的高可用性技术的一个组成部分，容灾系统更加强调处理外界环境对系统的影响，特别是灾难性事件对整个IT节点的影响，提供节点级别的系统恢复功能。 + + + +### 容灾的分类 + +从其对系统的保护程度来分，可以将容灾系统分为：数据容灾和应用容灾，数据容灾就是指建立一个异地的数据系统，该系统是本地关键应用数据的一个实时复制。 + +应用容灾是在数据容灾的基础上，在异地建立一套完整的与本地生产系统相当的备份应用系统（可以是互为备份），在灾难情况下，远程系统迅速接管业务运行，数据容灾是抗御灾难的保障，而应用容灾则是容灾系统建设的目标。 + + + +### 容灾和备份有什么联系 ？ + +容灾备份实际上是两个概念，容灾是为了在遭遇灾害时能保证信息系统能正常运行，帮助企业实现业务连续性的目标，备份是为了应对灾难来临时造成的数据丢失问题。在容灾备份一体化产品出现之前，容灾系统与备份系统是独立的。容灾备份产品的最终目标是帮助企业应对人为误操作、软件错误、病毒入侵等'软'性灾害以及硬件故障、自然灾害等“硬”性灾害。 + + + +### 容灾和备份的区别 + +一般意义上，备份指的是数据备份或系统备份，容灾指的是不在同一机房的数据备份或应用系统备份。备份采用备份软件技术实现，而容灾通过复制或镜像软件实现，两者的根本区别在于： + +1. 容灾主要针对火灾、地震等重大自然灾害，因此备份中心与主中心间必须保证一定的安全距离；数据备份在同一数据中心进行。 +2. 容灾系统不仅保护数据，更重要的目的在于保证业务的连续性；而数据备份系统只保护数据的安全性。 +3. 容灾保证数据的完整性；备份则只能恢复出备份时间点以前的数据。 +4. 容灾是在线过程；备份是离线过程。 +5. 容灾系统中，两地的数据是实时一致的；备份的数据则具有一定的时效性。 +6. 故障情况下，容灾系统的切换时间是几秒钟至几分钟；而备份系统的恢复时间可能几小时到几十小时。 + + + +### 容灾的分类 + +#### 1、数据级 + +数据级容灾是最基础的手段，指通过建立异地容灾中心，做数据的远程备份，在灾难发生之后要确保原有的数据不会丢失或者遭到破坏，但在数据级容灾这个级别，发生灾难时应用是会中断的。可以简单的把这种容灾方式理解成一个远程的数据备份中心，就是建立一个数据的备份系统或者一个容灾系统，比如数据库、文件等等。 + +- 优点：费用比较低，构建实施相对简单 +- 缺点：数据级容灾的恢复时间比较长 + + + +#### 2、应用级 + +应用级容灾是在数据级容灾的基础之上，在备份站点同样构建一套相同的应用系统，通过同步或异步复制技术，这样可以保证关键应用在允许的时间范围内恢复运行，尽可能减少灾难带来的损失，让用户基本感受不到灾难的发生。应用级容灾就是建立一个应用的备份系统，比如一套OA系统正在运行，在另一个地方建立一套同样的OA系统。 + +- 优点：提供的服务是完整、可靠、安全的，确保业务的连续性 +- 缺点：费用较高，需要更多软件的实现 + + + +#### 3、业务级 + +业务级容灾是全业务的灾备，除了必要的IT相关技术，还要求具备全部的基础设施。 + +- 优点：保障业务的连续性 +- 缺点：费用很高，还需要场所费用的投入，实施难度大。 + + + +### 备份等级 + +容灾备份是通过在异地建立和维护一个备份存储系统，利用地理上的分离来保证系统和数据对灾难性事件的抵御能力。根据容灾系统对灾难的抵抗程度，可分为数据容灾和应用容灾。数据容灾是指建立一个异地的数据系统，该系统是对本地系统关键应用数据实时复制。 + +当出现灾难时，可由异地系统迅速接替本地系统而保证业务的连续性。应用容灾比数据容灾层次更高，即在异地建立一套完整的、与本地数据系统相当的备份应用系统（可以同本地应用系统互为备份，也可与本地应用系统共同工作）。 + +在灾难出现后，远程应用系统迅速接管或承担本地应用系统的业务运行，设计一个容灾备份系统，需要考虑多方面的因素，如备份/恢复数据量大小、应用数据中心和备援数据中心之间的距离和数据传输方式、灾难发生时所要求的恢复速度、备援中心的管理及投入资金等，根据这些因素和不同的应用场合，通常可将容灾备份分为四个等级。 + +#### 第0级：没有备援中心 + +这一级容灾备份，实际上没有灾难恢复能力，它只在本地进行数据备份，并且被备份的数据只在本地保存，没有送往异地。 + +#### 第1级：本地磁带备份，异地保存 + +在本地将关键数据备份，然后送到异地保存。灾难发生后，按预定数据恢复程序恢复系统和数据。这种方案成本低、易于配置。但当数据量增大时，存在存储介质难管理的问题，并且当灾难发生时存在大量数据难以及时恢复的问题。为了解决此问题，灾难发生时，先恢复关键数据，后恢复非关键数据。欢迎关注微信公众号：朱小厮的博客。 + +#### 第2级：热备份站点备份 + +在异地建立一个热备份点，通过网络进行数据备份。也就是通过网络以同步或异步方式，把主站点的数据备份到备份站点，备份站点一般只备份数据，不承担业务。当出现灾难时，备份站点接替主站点的业务，从而维护业务运行的连续性。 + +#### 第3级：活动备援中心 + +在相隔较远的地方分别建立两个数据中心，它们都处于工作状态，并进行相互数据备份。当某个数据中心发生灾难时，另一个数据中心接替其工作任务。这种级别的备份根据实际要求和投入资金的多少，又可分为两种： + +1. 两个数据中心之间只限于关键数据的相互备份； + +2. 两个数据中心之间互为镜像，即零数据丢失等。 + +零数据丢失是目前要求最高的一种容灾备份方式，它要求不管什么灾难发生，系统都能保证数据的安全。所以，它需要配置复杂的管理软件和专用的硬件设备，需要投资相对而言是最大的，但恢复速度也是。 + diff --git "a/docs/others/\345\271\266\344\270\215\346\230\257\346\211\200\346\234\211\351\241\271\347\233\256\351\203\275\351\200\202\345\220\210\345\276\256\346\234\215\345\212\241.md" "b/docs/collection/\345\271\266\344\270\215\346\230\257\346\211\200\346\234\211\351\241\271\347\233\256\351\203\275\351\200\202\345\220\210\345\276\256\346\234\215\345\212\241.md" similarity index 100% rename from "docs/others/\345\271\266\344\270\215\346\230\257\346\211\200\346\234\211\351\241\271\347\233\256\351\203\275\351\200\202\345\220\210\345\276\256\346\234\215\345\212\241.md" rename to "docs/collection/\345\271\266\344\270\215\346\230\257\346\211\200\346\234\211\351\241\271\347\233\256\351\203\275\351\200\202\345\220\210\345\276\256\346\234\215\345\212\241.md" diff --git "a/docs/others/\346\234\200\350\257\246\347\273\206\347\232\204 IDEA \344\270\255\344\275\277\347\224\250 Debug \346\225\231\347\250\213.md" "b/docs/collection/\346\234\200\350\257\246\347\273\206\347\232\204 IDEA \344\270\255\344\275\277\347\224\250 Debug \346\225\231\347\250\213.md" similarity index 100% rename from "docs/others/\346\234\200\350\257\246\347\273\206\347\232\204 IDEA \344\270\255\344\275\277\347\224\250 Debug \346\225\231\347\250\213.md" rename to "docs/collection/\346\234\200\350\257\246\347\273\206\347\232\204 IDEA \344\270\255\344\275\277\347\224\250 Debug \346\225\231\347\250\213.md" diff --git "a/docs/collection/\347\247\222\346\235\200\347\263\273\347\273\237\350\256\276\350\256\241.md" "b/docs/collection/\347\247\222\346\235\200\347\263\273\347\273\237\350\256\276\350\256\241.md" new file mode 100644 index 0000000000..7b79d923d7 --- /dev/null +++ "b/docs/collection/\347\247\222\346\235\200\347\263\273\347\273\237\350\256\276\350\256\241.md" @@ -0,0 +1,165 @@ +# 怎么能设计出一个骚气的秒杀系统 + +前言：秒杀系统相信很多人见过，比如京东或者淘宝的秒杀，小米手机的秒杀，那么秒杀系统的后台是如何实现的呢？我们如何设计一个秒杀系统呢？对于秒杀系统应该考虑哪些问题？如何设计出健壮的秒杀系统？本期我们就来探讨一下这个问题： + +![img](https://tva1.sinaimg.cn/large/007S8ZIlly1gj8wmvkxr7j30xa0eq79r.jpg) + +**一、秒杀系统应该考虑的问题** + +**二、秒杀系统的设计和技术方案** + +**三、系统架构图和总结** + + + +## 一、秒杀应该考虑哪些问题 + +### 1.1 超卖问题 + + 分析秒杀的业务场景,最重要的有一点就是超卖问题，假如备货只有100个，但是最终超卖了200，一般来讲秒杀系统的价格都比较低，如果超卖将严重影响公司的财产利益，因此首当其冲的就是解决商品的超卖问题。 + +### 1.2 高并发 + +秒杀具有时间短、并发量大的特点，秒杀持续时间只有几分钟，而一般公司都为了制造轰动效应，会以极低的价格来吸引用户，因此参与抢购的用户会非常的多。短时间内会有大量请求涌进来，后端如何防止并发过高造成缓存击穿或者失效，击垮数据库都是需要考虑的问题。 + +### 1.3 接口防刷 + +现在的秒杀大多都会出来针对秒杀对应的软件，这类软件会模拟不断向后台服务器发起请求，一秒几百次都是很常见的，如何防止这类软件的重复无效请求，防止不断发起的请求也是需要我们针对性考虑的 + +### 1.4 秒杀url + +对于普通用户来讲，看到的只是一个比较简单的秒杀页面，在未达到规定时间，秒杀按钮是灰色的，一旦到达规定时间，灰色按钮变成可点击状态。这部分是针对小白用户的，如果是稍微有点电脑功底的用户，会通过F12看浏览器的network看到秒杀的url，通过特定软件去请求也可以实现秒杀。或者提前知道秒杀url的人，一请求就直接实现秒杀了。这个问题我们需要考虑解决 + +### 1.5 数据库设计 + +秒杀有把我们服务器击垮的风险，如果让它与我们的其他业务使用在同一个数据库中，耦合在一起，就很有可能牵连和影响其他的业务。如何防止这类问题发生，就算秒杀发生了宕机、服务器卡死问题，也应该让他尽量不影响线上正常进行的业务 + +### 1.6 大量请求问题 + +按照1.2的考虑，就算使用缓存还是不足以应对短时间的高并发的流量的冲击。如何承载这样巨大的访问量，同时提供稳定低时延的服务保证，是需要面对的一大挑战。我们来算一笔账，假如使用的是redis缓存，单台redis服务器可承受的QPS大概是4W左右，如果一个秒杀吸引的用户量足够多的话，单QPS可能达到几十万，单体redis还是不足以支撑如此巨大的请求量。缓存会被击穿，直接渗透到DB,从而击垮mysql.后台会将会大量报错 + +## 二、秒杀系统的设计和技术方案 + +### 2.1 秒杀系统数据库设计 + +针对1.5提出的秒杀数据库的问题，因此应该单独设计一个秒杀数据库，防止因为秒杀活动的高并发访问拖垮整个网站。这里只需要两张表，一张是秒杀订单表，一张是秒杀货品表 + +![img](https://img2018.cnblogs.com/blog/1066538/201907/1066538-20190729235900632-459948622.png) + + ![img](https://img2018.cnblogs.com/blog/1066538/201908/1066538-20190804004816292-1018723015.png) + +其实应该还有几张表，商品表：可以关联goods_id查到具体的商品信息，商品图像、名称、平时价格、秒杀价格等，还有用户表：根据用户user_id可以查询到用户昵称、用户手机号，收货地址等其他额外信息，这个具体就不给出实例了。 + +### 2.2 秒杀url的设计 + +为了避免有程序访问经验的人通过下单页面url直接访问后台接口来秒杀货品，我们需要将秒杀的url实现动态化，即使是开发整个系统的人都无法在秒杀开始前知道秒杀的url。具体的做法就是通过md5加密一串随机字符作为秒杀的url，然后前端访问后台获取具体的url，后台校验通过之后才可以继续秒杀。 + +### 2.3 秒杀页面静态化 + +将商品的描述、参数、成交记录、图像、评价等全部写入到一个静态页面，用户请求不需要通过访问后端服务器，不需要经过数据库，直接在前台客户端生成，这样可以最大可能的减少服务器的压力。具体的方法可以使用freemarker模板技术，建立网页模板，填充数据，然后渲染网页 + +### 2.4 单体redis升级为集群redis + +秒杀是一个读多写少的场景，使用redis做缓存再合适不过。不过考虑到缓存击穿问题，我们应该构建redis集群，采用哨兵模式，可以提升redis的性能和可用性。 + +### 2.5 使用nginx + +nginx是一个高性能web服务器，它的并发能力可以达到几万，而tomcat只有几百。通过nginx映射客户端请求，再分发到后台tomcat服务器集群中可以大大提升并发能力。 + +### 2.6 精简sql + +典型的一个场景是在进行扣减库存的时候，传统的做法是先查询库存，再去update。这样的话需要两个sql，而实际上一个sql我们就可以完成的。可以用这样的做法：update miaosha_goods set stock =stock-1 where goos_id ={#goods_id} and version = #{version} and sock>0;这样的话，就可以保证库存不会超卖并且一次更新库存,还有注意一点这里使用了版本号的乐观锁，相比较悲观锁，它的性能较好。 + +### 2.7 redis预减库存 + +很多请求进来，都需要后台查询库存,这是一个频繁读的场景。可以使用redis来预减库存，在秒杀开始前可以在redis设值，比如redis.set(goodsId,100),这里预放的库存为100可以设值为常量),每次下单成功之后,Integer stock = (Integer)redis.get(goosId); 然后判断sock的值，如果小于常量值就减去1;不过注意当取消的时候,需要增加库存，增加库存的时候也得注意不能大于之间设定的总库存数(查询库存和扣减库存需要原子操作，此时可以借助lua脚本)下次下单再获取库存的时候,直接从redis里面查就可以了。 + +### 2.8 接口限流 + +秒杀最终的本质是数据库的更新，但是有很多大量无效的请求，我们最终要做的就是如何把这些无效的请求过滤掉，防止渗透到数据库。限流的话，需要入手的方面很多： + +#### 2.8.1 前端限流 + +首先第一步就是通过前端限流，用户在秒杀按钮点击以后发起请求，那么在接下来的5秒是无法点击(通过设置按钮为disable)。这一小举措开发起来成本很小，但是很有效。 + +#### 2.8.2 同一个用户xx秒内重复请求直接拒绝 + +具体多少秒需要根据实际业务和秒杀的人数而定，一般限定为10秒。具体的做法就是通过redis的键过期策略，首先对每个请求都从String value = redis.get(userId);如果获取到这个 + +value为空或者为null，表示它是有效的请求，然后放行这个请求。如果不为空表示它是重复性请求，直接丢掉这个请求。如果有效,采用redis.setexpire(userId,value,10).value可以是任意值，一般放业务属性比较好,这个是设置以userId为key，10秒的过期时间(10秒后,key对应的值自动为null) + +#### 2.8.3 令牌桶算法限流 + +接口限流的策略有很多，我们这里采用令牌桶算法。令牌桶算法的基本思路是每个请求尝试获取一个令牌，后端只处理持有令牌的请求，生产令牌的速度和效率我们都可以自己限定，guava提供了RateLimter的api供我们使用。以下做一个简单的例子,注意需要引入guava + +```java +public class TestRateLimiter { + + public static void main(String[] args) { + //1秒产生1个令牌 + final RateLimiter rateLimiter = RateLimiter.create(1); + for (int i = 0; i < 10; i++) { + //该方法会阻塞线程，直到令牌桶中能取到令牌为止才继续向下执行。 + double waitTime= rateLimiter.acquire(); + System.out.println("任务执行" + i + "等待时间" + waitTime); + } + System.out.println("执行结束"); + } +} +``` + + 上面代码的思路就是通过RateLimiter来限定我们的令牌桶每秒产生1个令牌(生产的效率比较低)，循环10次去执行任务。acquire会阻塞当前线程直到获取到令牌，也就是如果任务没有获取到令牌，会一直等待。那么请求就会卡在我们限定的时间内才可以继续往下走，这个方法返回的是线程具体等待的时间。执行如下; + +![img](https://img2018.cnblogs.com/blog/1066538/201908/1066538-20190802152410567-941309775.png) + +可以看到任务执行的过程中，第1个是无需等待的，因为已经在开始的第1秒生产出了令牌。接下来的任务请求就必须等到令牌桶产生了令牌才可以继续往下执行。如果没有获取到就会阻塞(有一个停顿的过程)。不过这个方式不太好，因为用户如果在客户端请求，如果较多的话,直接后台在生产token就会卡顿(用户体验较差)，它是不会抛弃任务的，我们需要一个更优秀的策略:**如果超过某个时间没有获取到，直接拒绝该任务**。接下来再来个案例： + +```java +public class TestRateLimiter2 { + + public static void main(String[] args) { + final RateLimiter rateLimiter = RateLimiter.create(1); + + for (int i = 0; i < 10; i++) { + long timeOut = (long) 0.5; + boolean isValid = rateLimiter.tryAcquire(timeOut, TimeUnit.SECONDS); + System.out.println("任务" + i + "执行是否有效:" + isValid); + if (!isValid) { + continue; + } + System.out.println("任务" + i + "在执行"); + } + System.out.println("结束"); + } +} +``` + +其中用到了tryAcquire方法，这个方法的主要作用是设定一个超时的时间，如果在指定的时间内**预估(注意是预估并不会真实的等待)，**如果能拿到令牌就返回true，如果拿不到就返回false.然后我们让无效的直接跳过，这里设定每秒生产1个令牌，让每个任务尝试在 + +0.5秒获取令牌，如果获取不到,就直接跳过这个任务(放在秒杀环境里就是直接抛弃这个请求)；程序实际运行如下： + +![img](https://img2018.cnblogs.com/blog/1066538/201908/1066538-20190802154618921-1185631982.png) + +只有第1个获取到了令牌，顺利执行了，下面的基本都直接抛弃了，因为0.5秒内，令牌桶(1秒1个)来不及生产就肯定获取不到返回false了。 + +**这个限流策略的效率有多高呢？假如我们的并发请求是400万瞬间的请求,将令牌产生的效率设为每秒20个，每次尝试获取令牌的时间是0.05秒，那么最终测试下来的结果是，每次只会放行4个左右的请求,大量的请求会被拒绝,这就是令牌桶算法的优秀之处。** + +### 2.9 异步下单 + +为了提升下单的效率，并且防止下单服务的失败。需要将下单这一操作进行异步处理。最常采用的办法是使用队列，队列最显著的三个优点：**异步、削峰、解耦**。这里可以采用rabbitmq，在后台经过了限流、库存校验之后，流入到这一步骤的就是有效请求。然后发送到队列里，队列接受消息，异步下单。下完单，入库没有问题可以用短信通知用户秒杀成功。假如失败的话,可以采用补偿机制，重试。 + +### 3.0 服务降级 + +假如在秒杀过程中出现了某个服务器宕机，或者服务不可用，应该做好后备工作。之前的博客里有介绍通过Hystrix进行服务熔断和降级，可以开发一个备用服务，假如服务器真的宕机了，直接给用户一个友好的提示返回，而不是直接卡死，服务器错误等生硬的反馈。 + +## 三、总结 + +秒杀流程图： + +![img](https://tva1.sinaimg.cn/large/007S8ZIlly1gj8woov6i4j31lg0u0qf8.jpg) + + + + + +这就是我设计出来的秒杀流程图,当然不同的秒杀体量针对的技术选型都不一样，这个流程可以支撑起几十万的流量，如果是成千万破亿那就得重新设计了。比如数据库的分库分表、队列改成用kafka、redis增加集群数量等手段。通过本次设计主要是要表明的是我们如何应对高并发的处理，并开始尝试解决它，在工作中多思考、多动手能提升我们的能力水平，加油！如果本篇有任何错误，请麻烦指出来，不胜感激。 \ No newline at end of file diff --git "a/docs/others/\347\273\237\344\270\200\345\274\202\345\270\270\345\244\204\347\220\206.md" "b/docs/collection/\347\273\237\344\270\200\345\274\202\345\270\270\345\244\204\347\220\206.md" similarity index 100% rename from "docs/others/\347\273\237\344\270\200\345\274\202\345\270\270\345\244\204\347\220\206.md" rename to "docs/collection/\347\273\237\344\270\200\345\274\202\345\270\270\345\244\204\347\220\206.md" diff --git "a/docs/others/\350\201\212\350\201\212\347\256\200\345\216\206.md" "b/docs/collection/\350\201\212\350\201\212\347\256\200\345\216\206.md" similarity index 94% rename from "docs/others/\350\201\212\350\201\212\347\256\200\345\216\206.md" rename to "docs/collection/\350\201\212\350\201\212\347\256\200\345\216\206.md" index 946b8a2245..02a169fe2f 100644 --- "a/docs/others/\350\201\212\350\201\212\347\256\200\345\216\206.md" +++ "b/docs/collection/\350\201\212\350\201\212\347\256\200\345\216\206.md" @@ -343,4 +343,34 @@ codeKK 上一些内推职位是我发的，所以有时会收到一个仅含标 -所有举例不针对个人，只是把自己的感受经验分享出来，希望大家都有个靓丽的简历，为一份好的工作开个头 \ No newline at end of file +所有举例不针对个人，只是把自己的感受经验分享出来，希望大家都有个靓丽的简历，为一份好的工作开个头 + + + + + +不要把简历作为附件发出去 + + 关注简历投递时间 + +对照用人单位的要求写简历 + +用私人邮箱发主题鲜明的应聘邮件 + +在招聘网站填写资料时姓名一栏加上简短的特长自述 + +经常刷新简历(排在前面，更容易被人 事经理找到) + +忌向一单位申请多职(用人单位会认为你非常盲目，没有目标 + +，缺乏主见) + +不要只应聘最近三天的职位（一些职位已经是半个月甚至两个 + +月的，应聘的人少，成功率反而高 + +最好的简历标题，首推应聘xx岗位的人——X年岗位经验诚心求职XXX部门XXX岗位 姓名XX 139XXX + +次一点的简历标题——应聘XXX岗位 + +最次的简历标题——求职 \ No newline at end of file diff --git "a/docs/collection/\351\235\242\350\257\225\345\256\230\351\227\256\357\274\232\344\270\272\344\273\200\344\271\210SpringBoot\347\232\204 jar \345\217\257\344\273\245\347\233\264\346\216\245\350\277\220\350\241\214\357\274\237.md" "b/docs/collection/\351\235\242\350\257\225\345\256\230\351\227\256\357\274\232\344\270\272\344\273\200\344\271\210SpringBoot\347\232\204 jar \345\217\257\344\273\245\347\233\264\346\216\245\350\277\220\350\241\214\357\274\237.md" new file mode 100644 index 0000000000..21bbffd193 --- /dev/null +++ "b/docs/collection/\351\235\242\350\257\225\345\256\230\351\227\256\357\274\232\344\270\272\344\273\200\344\271\210SpringBoot\347\232\204 jar \345\217\257\344\273\245\347\233\264\346\216\245\350\277\220\350\241\214\357\274\237.md" @@ -0,0 +1,319 @@ +SpringBoot提供了一个插件spring-boot-maven-plugin用于把程序打包成一个可执行的jar包。在pom文件里加入这个插件即可： + +```xml + + + + org.springframework.boot + spring-boot-maven-plugin + + + +``` + +打包完生成的 `executable-jar-1.0-SNAPSHOT.jar` 内部的结构如下： + +``` +├── META-INF +│ ├── MANIFEST.MF +│ └── maven +│ └── spring.study +│ └── executable-jar +│ ├── pom.properties +│ └── pom.xml +├── lib +│ ├── aopalliance-1.0.jar +│ ├── classmate-1.1.0.jar +│ ├── spring-boot-1.3.5.RELEASE.jar +│ ├── spring-boot-autoconfigure-1.3.5.RELEASE.jar +│ ├── ... +├── org +│ └── springframework +│ └── boot +│ └── loader +│ ├── ExecutableArchiveLauncher$1.class +│ ├── ... +└── spring + └── study + └── executablejar + └── ExecutableJarApplication.class +``` + +然后可以直接执行jar包就能启动程序了： + +```sh +java -jar executable-jar-1.0-SNAPSHOT.jar +``` + + + +打包出来fat jar内部有4种文件类型： + +1. META-INF文件夹：程序入口，其中MANIFEST.MF用于描述jar包的信息 +2. lib目录：放置第三方依赖的jar包，比如springboot的一些jar包 +3. spring boot loader相关的代码 +4. 模块自身的代码 + +MANIFEST.MF 文件的内容： + +```properties +Manifest-Version: 1.0 +Implementation-Title: executable-jar +Implementation-Version: 1.0-SNAPSHOT +Archiver-Version: Plexus Archiver +Built-By: Format +Start-Class: spring.study.executablejar.ExecutableJarApplication +Implementation-Vendor-Id: spring.study +Spring-Boot-Version: 1.3.5.RELEASE +Created-By: Apache Maven 3.2.3 +Build-Jdk: 1.8.0_20 +Implementation-Vendor: Pivotal Software, Inc. +Main-Class: org.springframework.boot.loader.JarLauncher +``` + +我们看到，它的Main-Class是 `org.springframework.boot.loader.JarLauncher`，当我们使用 java -jar 执行 jar 包的时候会调用JarLauncher 的 main 方法，而不是我们编写的 SpringApplication。 + +那么 JarLauncher 这个类是的作用是什么的？ + +它是 SpringBoot 内部提供的工具 Spring Boot Loader 提供的一个用于执行 Application 类的工具类(fat jar内部有spring loader相关的代码就是因为这里用到了)。相当于Spring Boot Loader提供了一套标准用于执行SpringBoot打包出来的jar + +### Spring Boot Loader抽象的一些类 + +- 抽象类Launcher：各种Launcher的基础抽象类，用于启动应用程序；跟Archive配合使用；目前有3种实现，分别是JarLauncher、WarLauncher以及PropertiesLauncher + +- Archive：归档文件的基础抽象类。JarFileArchive就是jar包文件的抽象。它提供了一些方法比如getUrl会返回这个Archive对应的URL；getManifest方法会获得Manifest数据等。ExplodedArchive是文件目录的抽象 + +- JarFile：对jar包的封装，每个JarFileArchive都会对应一个JarFile。JarFile被构造的时候会解析内部结构，去获取jar包里的各个文件或文件夹，这些文件或文件夹会被封装到Entry中，也存储在JarFileArchive中。如果Entry是个jar，会解析成JarFileArchive。 + +比如一个JarFileArchive对应的URL为： + +``` +jar:file:/Users/format/Develop/gitrepository/springboot-analysis/springboot-executable-jar/target/executable-jar-1.0-SNAPSHOT.jar!/ +``` + +它对应的JarFile为： + +``` +/Users/format/Develop/gitrepository/springboot-analysis/springboot-executable-jar/target/executable-jar-1.0-SNAPSHOT.jar +``` + +这个JarFile有很多Entry，比如： + +``` +META-INF/ +META-INF/MANIFEST.MF +spring/ +spring/study/ +.... +spring/study/executablejar/ExecutableJarApplication.class +lib/spring-boot-starter-1.3.5.RELEASE.jar +lib/spring-boot-1.3.5.RELEASE.jar +... +``` + +JarFileArchive内部的一些依赖jar对应的URL(SpringBoot使用org.springframework.boot.loader.jar.Handler处理器来处理这些URL)： + +``` +jar:file:/Users/Format/Develop/gitrepository/springboot-analysis/springboot-executable-jar/target/executable-jar-1.0-SNAPSHOT.jar!/lib/spring-boot-starter-web-1.3.5.RELEASE.jar!/ + +jar:file:/Users/Format/Develop/gitrepository/springboot-analysis/springboot-executable-jar/target/executable-jar-1.0-SNAPSHOT.jar!/lib/spring-boot-loader-1.3.5.RELEASE.jar!/org/springframework/boot/loader/JarLauncher.class +``` + +我们看到如果有jar包中包含jar，或者jar包中包含jar包里面的class文件，那么会使用 **!/** 分隔开，这种方式只有`org.springframework.boot.loader.jar.Handler` 能处理，它是 SpringBoot 内部扩展出来的一种 URL 协议。 + +### JarLauncher 的执行过程 + +JarLauncher的main方法： + +```java +public static void main(String[] args) { + // 构造JarLauncher，然后调用它的launch方法。参数是控制台传递的 + new JarLauncher().launch(args); +} +``` + +JarLauncher被构造的时候会调用父类ExecutableArchiveLauncher的构造方法。 + +ExecutableArchiveLauncher的构造方法内部会去构造Archive，这里构造了JarFileArchive。构造JarFileArchive的过程中还会构造很多东西，比如JarFile，Entry … + +JarLauncher的launch方法： + +```java +protected void launch(String[] args) { + try { + // 在系统属性中设置注册了自定义的URL处理器：org.springframework.boot.loader.jar.Handler。如果URL中没有指定处理器，会去系统属性中查询 + JarFile.registerUrlProtocolHandler(); + // getClassPathArchives方法在会去找lib目录下对应的第三方依赖JarFileArchive，同时也会项目自身的JarFileArchive + // 根据getClassPathArchives得到的JarFileArchive集合去创建类加载器ClassLoader。这里会构造一个LaunchedURLClassLoader类加载器，这个类加载器继承URLClassLoader，并使用这些JarFileArchive集合的URL构造成URLClassPath + // LaunchedURLClassLoader类加载器的父类加载器是当前执行类JarLauncher的类加载器 + ClassLoader classLoader = createClassLoader(getClassPathArchives()); + // getMainClass方法会去项目自身的Archive中的Manifest中找出key为Start-Class的类 + // 调用重载方法launch + launch(args, getMainClass(), classLoader); + } + catch (Exception ex) { + ex.printStackTrace(); + System.exit(1); + } +} + +// Archive的getMainClass方法 +// 这里会找出spring.study.executablejar.ExecutableJarApplication这个类 +public String getMainClass() throws Exception { + Manifest manifest = getManifest(); + String mainClass = null; + if (manifest != null) { + mainClass = manifest.getMainAttributes().getValue("Start-Class"); + } + if (mainClass == null) { + throw new IllegalStateException( + "No 'Start-Class' manifest entry specified in " + this); + } + return mainClass; +} + +// launch重载方法 +protected void launch(String[] args, String mainClass, ClassLoader classLoader) + throws Exception { + // 创建一个MainMethodRunner，并把args和Start-Class传递给它 + Runnable runner = createMainMethodRunner(mainClass, args, classLoader); + // 构造新线程 + Thread runnerThread = new Thread(runner); + // 线程设置类加载器以及名字，然后启动 + runnerThread.setContextClassLoader(classLoader); + runnerThread.setName(Thread.currentThread().getName()); + runnerThread.start(); +} +``` + + + +MainMethodRunner的run方法： + +```java +@Override +public void run() { + try { + // 根据Start-Class进行实例化 + Class mainClass = Thread.currentThread().getContextClassLoader() + .loadClass(this.mainClassName); + // 找出main方法 + Method mainMethod = mainClass.getDeclaredMethod("main", String[].class); + // 如果main方法不存在，抛出异常 + if (mainMethod == null) { + throw new IllegalStateException( + this.mainClassName + " does not have a main method"); + } + // 调用 + mainMethod.invoke(null, new Object[] { this.args }); + } + catch (Exception ex) { + UncaughtExceptionHandler handler = Thread.currentThread() + .getUncaughtExceptionHandler(); + if (handler != null) { + handler.uncaughtException(Thread.currentThread(), ex); + } + throw new RuntimeException(ex); + } +} +``` + +Start-Class的main方法调用之后，内部会构造Spring容器，启动内置Servlet容器等过程。 这些过程我们都已经分析过了。 + +### 关于自定义的类加载器LaunchedURLClassLoader + +LaunchedURLClassLoader重写了loadClass方法，也就是说它修改了默认的类加载方式(先看该类是否已加载这部分不变，后面真正去加载类的规则改变了，不再是直接从父类加载器中去加载)。LaunchedURLClassLoader定义了自己的类加载规则： + +```java +private Class doLoadClass(String name) throws ClassNotFoundException { + + // 1) Try the root class loader + try { + if (this.rootClassLoader != null) { + return this.rootClassLoader.loadClass(name); + } + } + catch (Exception ex) { + // Ignore and continue + } + + // 2) Try to find locally + try { + findPackage(name); + Class cls = findClass(name); + return cls; + } + catch (Exception ex) { + // Ignore and continue + } + + // 3) Use standard loading + return super.loadClass(name, false); +} +``` + +加载规则： + +1. 如果根类加载器存在，调用它的加载方法。这里是根类加载是ExtClassLoader +2. 调用LaunchedURLClassLoader自身的findClass方法，也就是URLClassLoader的findClass方法 +3. 调用父类的loadClass方法，也就是执行默认的类加载顺序(从BootstrapClassLoader开始从下往下寻找) + +LaunchedURLClassLoader自身的findClass方法： + +```java +protected Class findClass(final String name) + throws ClassNotFoundException +{ + try { + return AccessController.doPrivileged( + new PrivilegedExceptionAction>() { + public Class run() throws ClassNotFoundException { + // 把类名解析成路径并加上.class后缀 + String path = name.replace('.', '/').concat(".class"); + // 基于之前得到的第三方jar包依赖以及自己的jar包得到URL数组，进行遍历找出对应类名的资源 + // 比如path是org/springframework/boot/loader/JarLauncher.class，它在jar:file:/Users/Format/Develop/gitrepository/springboot-analysis/springboot-executable-jar/target/executable-jar-1.0-SNAPSHOT.jar!/lib/spring-boot-loader-1.3.5.RELEASE.jar!/中被找出 + // 那么找出的资源对应的URL为jar:file:/Users/Format/Develop/gitrepository/springboot-analysis/springboot-executable-jar/target/executable-jar-1.0-SNAPSHOT.jar!/lib/spring-boot-loader-1.3.5.RELEASE.jar!/org/springframework/boot/loader/JarLauncher.class + Resource res = ucp.getResource(path, false); + if (res != null) { // 找到了资源 + try { + return defineClass(name, res); + } catch (IOException e) { + throw new ClassNotFoundException(name, e); + } + } else { // 找不到资源的话直接抛出ClassNotFoundException异常 + throw new ClassNotFoundException(name); + } + } + }, acc); + } catch (java.security.PrivilegedActionException pae) { + throw (ClassNotFoundException) pae.getException(); + } +} +``` + +下面是LaunchedURLClassLoader的一个测试： + +```java +// 注册org.springframework.boot.loader.jar.Handler URL协议处理器 +JarFile.registerUrlProtocolHandler(); +// 构造LaunchedURLClassLoader类加载器，这里使用了2个URL，分别对应jar包中依赖包spring-boot-loader和spring-boot，使用 "!/" 分开，需要org.springframework.boot.loader.jar.Handler处理器处理 +LaunchedURLClassLoader classLoader = new LaunchedURLClassLoader( + new URL[] { + new URL("jar:file:/Users/Format/Develop/gitrepository/springboot-analysis/springboot-executable-jar/target/executable-jar-1.0-SNAPSHOT.jar!/lib/spring-boot-loader-1.3.5.RELEASE.jar!/") + , new URL("jar:file:/Users/Format/Develop/gitrepository/springboot-analysis/springboot-executable-jar/target/executable-jar-1.0-SNAPSHOT.jar!/lib/spring-boot-1.3.5.RELEASE.jar!/") + }, + LaunchedURLClassLoaderTest.class.getClassLoader()); + +// 加载类 +// 这2个类都会在第二步本地查找中被找出(URLClassLoader的findClass方法) +classLoader.loadClass("org.springframework.boot.loader.JarLauncher"); +classLoader.loadClass("org.springframework.boot.SpringApplication"); +// 在第三步使用默认的加载顺序在ApplicationClassLoader中被找出 +classLoader.loadClass("org.springframework.boot.autoconfigure.web.DispatcherServletAutoConfiguration"); +``` + +### Spring Boot Loader的作用 + +SpringBoot在可执行jar包中定义了自己的一套规则，比如第三方依赖jar包在/lib目录下，jar包的URL路径使用自定义的规则并且这个规则需要使用 `org.springframework.boot.loader.jar.Handler` 处理器处理。它的Main-Class使用JarLauncher，如果是war包，使用WarLauncher执行。这些Launcher内部都会另起一个线程启动自定义的SpringApplication类。 + +这些特性通过spring-boot-maven-plugin插件打包完成。 \ No newline at end of file diff --git "a/docs/collection/\351\235\242\350\257\225\345\256\230\351\227\256\357\274\232\351\253\230\345\271\266\345\217\221\344\270\213\357\274\214\344\275\240\351\203\275\346\200\216\344\271\210\351\200\211\346\213\251\346\234\200\344\274\230\347\232\204\347\272\277\347\250\213\346\225\260\357\274\237.md" "b/docs/collection/\351\235\242\350\257\225\345\256\230\351\227\256\357\274\232\351\253\230\345\271\266\345\217\221\344\270\213\357\274\214\344\275\240\351\203\275\346\200\216\344\271\210\351\200\211\346\213\251\346\234\200\344\274\230\347\232\204\347\272\277\347\250\213\346\225\260\357\274\237.md" new file mode 100644 index 0000000000..14ed6d3a53 --- /dev/null +++ "b/docs/collection/\351\235\242\350\257\225\345\256\230\351\227\256\357\274\232\351\253\230\345\271\266\345\217\221\344\270\213\357\274\214\344\275\240\351\203\275\346\200\216\344\271\210\351\200\211\346\213\251\346\234\200\344\274\230\347\232\204\347\272\277\347\250\213\346\225\260\357\274\237.md" @@ -0,0 +1,134 @@ +> 来源 | https://urlify.cn/YFbMNf + +为了加快程序处理速度，我们会将问题分解成若干个并发执行的任务。并且创建线程池，将任务委派给线程池中的线程，以便使它们可以并发地执行。在高并发的情况下采用线程池，可以有效降低线程创建释放的时间花销及资源开销，如不使用线程池，有可能造成系统创建大量线程而导致消耗完系统内存以及 “过度切换”（在 JVM 中采用的处理机制为时间片轮转，减少了线程间的相互切换） 。 + +但是有一个很大的问题摆在我们面前，即我们希望尽可能多地创建任务，但由于资源所限我们又不能创建过多的线程。那么在高并发的情况下，我们怎么选择最优的线程数量呢？选择原则又是什么呢？ + +### 一、理论分析 + +关于如何计算并发线程数，有两种说法。 + +**第一种，《Java Concurrency in Practice》即《java 并发编程实践》8.2 节 170 页** + +对于计算密集型的任务，一个有 Ncpu 个处理器的系统通常通过使用一个 Ncpu + 1 个线程的线程池来获得最优的利用率（计算密集型的线程恰好在某时因为发生一个页错误或者因其他原因而暂停，刚好有一个 “额外” 的线程，可以确保在这种情况下 CPU 周期不会中断工作）。 + +对于包含了 I/O 和其他阻塞操作的任务，不是所有的线程都会在所有的时间被调度，因此你需要一个更大的池。为了正确地设置线程池的长度，你必须估算出任务花在等待的时间与用来计算的时间的比率；这个估算值不必十分精确，而且可以通过一些监控工具获得。你还可以选择另一种方法来调节线程池的大小，在一个基准负载下，使用 几种不同大小的线程池运行你的应用程序，并观察 CPU 利用率的水平。 + +给定下列定义： + +``` +Ncpu = CPU的数量 +Ucpu = 目标CPU的使用率， 0 <= Ucpu <= 1 +W/C = 等待时间与计算时间的比率 +为保持处理器达到期望的使用率，最优的池的大小等于： +Nthreads = Ncpu x Ucpu x (1 + W/C) +``` + +你可以使用 Runtime 来获得 CPU 的数目： + +``` +int N_CPUS = Runtime.getRuntime().availableProcessors(); +``` + +当然，CPU 周期并不是唯一你可以使用线程池管理的资源。其他可以约束资源池大小的资源包括：内存、文件句柄、套接字句柄和数据库连接等。计算这些类型资源池的大小约束非常简单：首先累加出每一个任务需要的这些资源的总童，然后除以可用的总量。所得的结果是池大小的上限。 + +当任务需要使用池化的资源时，比如数据库连接，那么线程池的长度和资源池的长度会相互影响。如果每一个任务都需要一个数据库连接，那么连接池的大小就限制了线程池的有效大小；类似地，当线程池中的任务是连接池的唯一消费者时，那么线程池的大小反而又会限制了连接池的有效大小。 + +如上，在《Java Concurrency in Practice》一书中，给出了估算线程池大小的公式： + +``` +Nthreads = Ncpu x Ucpu x (1 + W/C)，其中 +Ncpu = CPU核心数 +Ucpu = CPU使用率，0~1 +W/C = 等待时间与计算时间的比率 +``` + +**第二种，《Programming Concurrency on the JVM Mastering》即《Java 虚拟机并发编程》2.1 节 12 页** + +为了解决上述难题，我们希望至少可以创建处理器核心数那么多个线程。这就保证了有尽可能多地处理器核心可以投入到解决问题的工作中去。通过下面的代码，我们可以很容易地获取到系统可用的处理器核心数： + +``` +Runtime.getRuntime().availableProcessors(); +``` + +所以，应用程序的最小线程数应该等于可用的处理器核数。如果所有的任务都是计算密集型的，则创建处理器可用核心数那么多个线程就可以了。在这种情况下，创建更多的线程对程序性能而言反而是不利的。因为当有多个仟务处于就绪状态时，处理器核心需要在线程间频繁进行上下文切换，而这种切换对程序性能损耗较大。但如果任务都是 IO 密集型的，那么我们就需要开更多的线程来提高性能。 + +当一个任务执行 IO 操作时，其线程将被阻塞，于是处理器可以立即进行上下文切换以便处理其他就绪线程。如果我们只有处理器可用核心数那么多个线程的话，则即使有待执行的任务也无法处理，因为我们已经拿不出更多的线程供处理器调度了。 + +如果任务有 50% 的时间处于阻塞状态，则程序所需线程数为处理器可用核心数的两倍。如果任务被阻塞的时间少于 50%，即这些任务是计算密集型的，则程序所需线程数将随之减少，但最少也不应低于处理器的核心数。如果任务被阻塞的时间大于执行时间，即该任务是 IO 密集型的，我们就需要创建比处理器核心数大几倍数量的线程。我们可以计算出程序所需线程的总数，总结如下： + +- 线程数 = CPU 可用核心数 /(1 - 阻塞系数），其中阻塞系数的取值在 0 和 1 之间。 +- 计算密集型任务的阻塞系数为 0，而 IO 密集型任务的阻塞系数则接近 1。一个完全阻塞的任务是注定要挂掉的，所以我们无须担心阻塞系数会达到 1。 + +为了更好地确定程序所需线程数，我们需要知道下面两个关键参数： + +- 处理器可用核心数； +- 任务的阻塞系数； + +第一个参数很容易确定，我们甚至可以用之前的方法在运行时查到这个值。但确定阻塞系数就稍微困难一些。我们可以先试着猜测，抑或采用一些性能分析工具或 java.lang.management API 来确定线程花在系统 IO 操作上的时间与 CPU 密集任务所耗时间的比值。如上，在《Programming Concurrency on the JVM Mastering》一书中，给出了估算线程池大小的公式： + +**线程数 = Ncpu /（1 - 阻塞系数）** + +对于说法一，假设 CPU 100% 运转，即撇开 CPU 使用率这个因素，线程数 = Ncpu x (1 + W/C)。 + +现在假设将方法二的公式等于方法一公式，即 Ncpu /（1 - 阻塞系数）= Ncpu x (1 + W/C)，推导出：阻塞系数 = W / (W + C)，即阻塞系数 = 阻塞时间 /（阻塞时间 + 计算时间），这个结论在方法二后续中得到印证，如下： + +> 由于对 Web 服务的请求大部分时间都花在等待服务器响应上了，所以阻塞系数会相当高，因此程序需要开的线程数可能是处理器核心数的若干倍。假设阻塞系数是 0.9，即每个任务 90% 的时间处于阻塞状态而只有 10% 的时间在干活，则在双核处理器上我们就需要开 20 个线程（使用第 2.1 节的公式计算）。如果有很多只股票要处理的话，我们可以在 8 核处理器上开到 80 个线程来处理该任务。 + +由此可见，说法一和说法二其实是一个公式。 + +### 二、实际应用 + +那么实际使用中并发线程数如何设置呢？我们先看一道题目： + +> 假设要求一个系统的 TPS（Transaction Per Second 或者 Task Per Second）至少为 20，然后假设每个 Transaction 由一个线程完成，继续假设平均每个线程处理一个 Transaction 的时间为 4s。那么问题转化为： + +如何设计线程池大小，使得可以在 1s 内处理完 20 个 Transaction？ + +计算过程很简单，每个线程的处理能力为 0.25TPS，那么要达到 20TPS，显然需要 20/0.25=80 个线程。 + +这个理论上成立的，但是实际情况中，一个系统最快的部分是 CPU，所以决定一个系统吞吐量上限的是 CPU。增强 CPU 处理能力，可以提高系统吞吐量上限。在考虑时需要把 CPU 吞吐量加进去。 + +分析如下（我们以说法一公式为例）： + +**Nthreads = Ncpu x (1 + W/C)** + +即线程等待时间所占比例越高，需要越多线程。线程 CPU 时间所占比例越高，需要越少线程。这就可以划分成两种任务类型： + +**IO 密集型** 一般情况下，如果存在 IO，那么肯定 W/C > 1（阻塞耗时一般都是计算耗时的很多倍），但是需要考虑系统内存有限（每开启一个线程都需要内存空间），这里需要在服务器上测试具体多少个线程数适合（CPU 占比、线程数、总耗时、内存消耗）。如果不想去测试，保守点取 1 即可，Nthreads = Ncpu x (1 + 1) = 2Ncpu。这样设置一般都 OK。 + +**计算密集型** 假设没有等待 W = 0，则 W/C = 0。Nthreads = Ncpu。 + +根据短板效应，真实的系统吞吐量并不能单纯根据 CPU 来计算。那要提高系统吞吐量，就需要从 “系统短板”（比如网络延迟、IO）着手： + +- 尽量提高短板操作的并行化比率，比如多线程下载技术； +- 增强短板能力，比如用 NIO 替代 IO； + +第一条可以联系到 Amdahl 定律，这条定律定义了串行系统并行化后的加速比计算公式：加速比 = 优化前系统耗时 / 优化后系统耗时 加速比越大，表明系统并行化的优化效果越好。Addahl 定律还给出了系统并行度、CPU 数目和加速比的关系，加速比为 Speedup，系统串行化比率（指串行执行代码所占比率）为 F，CPU 数目为 N：Speedup <= 1 / (F + (1-F)/N) + +当 N 足够大时，串行化比率 F 越小，加速比 Speedup 越大。 + +这时候又抛出是否线程池一定比但线程高效的问题？ + +答案是否定的，比如 Redis 就是单线程的，但它却非常高效，基本操作都能达到十万量级 / s。从线程这个角度来看，部分原因在于： + +- 多线程带来线程上下文切换开销，单线程就没有这种开销； +- 锁； + +当然 “Redis 很快” 更本质的原因在于： + +Redis 基本都是内存操作，这种情况下单线程可以很高效地利用 CPU。而多线程适用场景一般是：存在相当比例的 IO 和网络操作。 + +总的来说，应用情况不同，采取多线程 / 单线程策略不同；线程池情况下，不同的估算，目的和出发点是一致的。 + +至此结论为： + +IO 密集型 = 2Ncpu（可以测试后自己控制大小，2Ncpu 一般没问题）（常出现于线程中：数据库数据交互、文件上传下载、网络数据传输等等） + +计算密集型 = Ncpu（常出现于线程中：复杂算法） + +当然说法一中还有一种说法： + +> 对于计算密集型的任务，一个有 Ncpu 个处理器的系统通常通过使用一个 Ncpu + 1 个线程的线程池来获得最优的利用率（计算密集型的线程恰好在某时因为发生一个页错误或者因其他原因而暂停，刚好有一个 “额外” 的线程，可以确保在这种情况下 CPU 周期不会中断工作）。 + +即，计算密集型 = Ncpu + 1，但是这种做法导致的多一个 CPU 上下文切换是否值得，这里不考虑。读者可自己考量。 \ No newline at end of file diff --git a/docs/coverpage.md b/docs/coverpage.md deleted file mode 100644 index 685726e2d2..0000000000 --- a/docs/coverpage.md +++ /dev/null @@ -1,10 +0,0 @@ -![](/_media/icon.svg) - - - - -> `Keep On Growing`：**Java Keeper** - -[GitHub](https://github.com/Jstarfish/Technical-Learning) -[Get Started](#JavaKeeper) - diff --git a/docs/css.txt b/docs/css.txt deleted file mode 100644 index 1604ae4050..0000000000 --- a/docs/css.txt +++ /dev/null @@ -1,348 +0,0 @@ -/*自定义样式，实时生效*/ - -/*自定义样式，实时生效*/ - -/* 全局属性 - * 页边距 padding:30px; - * 全文字体 font-family:ptima-Regular; - * 英文换行 word-break:break-all; - */ -#nice { - padding: 0px; -} - -.nice-wx-box-pc { - width: 100%; - padding: 10px 10px 10px 10px; - box-shadow: none; -} - -.nice-wx-box { - width: 100%; - padding: 10px 10px 10px 10px; - box-shadow: none; -} - -#nice h1, h2, h3, h4, h5, h6 { - margin-top: 25px; - margin-bottom: 1px; - font-weight: bold; - color: black; -} - -/* 段落，下方未标注标签参数均同此处 - * 上边距 margin-top:5px; - * 下边距 margin-bottom:5px; - * 行高 line-height:26px; - * 词间距 word-spacing:3px; - * 字间距 letter-spacing:3px; - * 对齐 text-align:left; - * 颜色 color:#3e3e3e; - * 字体大小 font-size:16px; - * 首行缩进 text-indent:2em; - */ -#nice p { - margin:10px 10px; - line-height:1.75; - letter-spacing:1.6px; - font-size: 15px; - word-spacing:0.1em; -} - -/* 一级标题 */ -#nice h1 { - border-bottom: 2px solid #FF6827; - font-size: 1.4em; - text-align: center; -} - -/* 一级标题内容 */ -#nice h1 span { - font-size: 1.4em; - display:inline-block; - font-weight: bold; - //background: #0e88eb; - color:#ffffff; - color: #FF6827; - padding:3px 10px 1px; - border-top-right-radius:3px; - border-top-left-radius:3px; - margin-right:3px; -} - -/* 一级标题修饰 请参考有实例的主题 */ -#nice h1:after { -} - -/* 二级标题 */ -#nice h2 { - text-align:left; - margin:20px 10px 0px 0px; -} - -/* 二级标题内容 */ -#nice h2 span { - font-family:STHeitiSC-Light; - font-size: 20px; - color:#FF6827; - font-weight:bolder; - display:inline-block; - padding-left:10px; - border-left:5px solid #FF6827; -} - -/* 二级标题修饰 请参考有实例的主题 */ -#nice h2:after { -} - -/* 三级标题 */ -#nice h3 { - font-size: 18px; - color: #0e88eb; - -} - -/* 三级标题内容 */ -#nice h3 span { - font-size: 18px; - color: #000000; -} - -/* 三级标题修饰 请参考有实例的主题 */ -#nice h3:after { -} - -/* 无序列表整体样式 - * list-style-type: square|circle|disc; - */ -#nice ul { -} - -/* 无序列表整体样式 - * list-style-type: upper-roman|lower-greek|lower-alpha; - */ -#nice ol { -} - -/* 列表内容，不要设置li - */ -#nice li section { - font-size: 15px; -} - -/* 引用 - * 左边缘颜色 border-left-color:black; - * 背景色 background:gray; - */ -#nice blockquote { - font-style:normal; - border-left:none; - padding:10px; - position:relative; - line-height:1.8; - border-radius:7px 7px 7px 7px; - color: #0e88eb; - background:#fff; - box-shadow:#84A1A8 0px 10px 15px; - margin-bottom: 40px; - margin-top: -14px; -} -#nice blockquote:before { - //content:"★ "; - content:"👨‍💻 导读：“"; - display:inline; - color: #000000; - font-size:1em; - //font-family:Arial,serif; - line-height:1em; - font-weight:500; -} - -/* 引用文字 */ -#nice blockquote p { - color: #000000; - font-size:14px; - display:inline; -} -#nice blockquote:after { - content:"”"; - float:right; - display:inline; - color:#000000; - font-size:1em; - line-height:1em; - font-weight:500; -} - -/* 链接 - * border-bottom: 1px solid #009688; - */ -#nice a { - color: #000000; - border-bottom: 0px solid #ff3502; - font-family:STHeitiSC-Light; -} - -/* 加粗 */ -#nice strong { - font-weight: border; - //color: #FF6827; -} - -/* 斜体 */ -#nice em { - color: #FF6827; - letter-spacing:0.3em; -} - -/* 加粗斜体 */ -#nice strong em { - color: #0e88eb; - letter-spacing:0.3em; -} - -/* 删除线 */ -#nice del { -} - -/* 分隔线 - * 粗细、样式和颜色 - * border-top:1px solid #3e3e3e; - */ -#nice hr { - height:1px; - padding:0; - margin-top: 30px; - margin-bottom: 30px; - border:none; - border-top:medium solidid #333; - text-align:center; - background-image:linear-gradient(to right,rgba(248,57,41,0),#FF6827,rgba(248,57,41,0)); -} - -/* 图片 - * 宽度 width:80%; - * 居中 margin:0 auto; - * 居左 margin:0 0; - */ -#nice img { - border-radius:0px 0px 5px 5px; - display:block; - margin:20px auto; - width:100%; - height:100%; - object-fit:contain; - box-shadow:#84A1A8 0px 10px 15px; -} - -/* 图片描述文字 */ -#nice figcaption { - display:block; - font-size:12px; - font-family:PingFangSC-Light; -} - -/* 行内代码 */ -#nice p code,li code { - font-size: 14px; - word-wrap: break-word; - padding: 2px 4px; - border-radius: 4px; - margin: 0 2px; - color: #FF6827; - background-color: rgba(27,31,35,.05); - font-family: Operator Mono, Consolas, Monaco, Menlo, monospace; - word-break: break-all; -} - -#nice li code { - font-size: 14px; - word-wrap: break-word; - padding: 2px 4px; - border-radius: 4px; - margin: 0 2px; - color: #FF6827; - background-color: rgba(27,31,35,.05); - font-family: Operator Mono, Consolas, Monaco, Menlo, monospace; - word-break: break-all; -} - -/* 非微信代码块 - * 代码块不换行 display:-webkit-box !important; - * 代码块换行 display:block; - */ -#nice pre code { -} - -/* - * 表格内的单元格 - * 字体大小 font-size: 16px; - * 边框 border: 1px solid #ccc; - * 内边距 padding: 5px 10px; - */ -#nice table tr th, -#nice table tr td { - font-size: 15px; -} - -/* 脚注文字 */ -#nice .footnote-word { - color: #FF6827; -} - -/* 脚注上标 */ -#nice .footnote-ref { - color: #FF6827; -} - -/* 非微信代码块 - * 代码块不换行 display:-webkit-box !important; - * 代码块换行 display:block; - */ -#nice pre code { -} - -/* 脚注文字 */ -#nice .footnote-word { - color: #FF6827; -} - -/* 脚注上标 */ -#nice .footnote-ref { - color: #FF6827; -} - -/*脚注链接样式*/ -#nice .footnote-item em { - color: #FF6827; - font-size:12px; -} - -/* "参考资料"四个字 - * 内容 content: "参考资料"; - */ -#nice .footnotes-sep:before { -} - -/* 参考资料编号 */ -#nice .footnote-num { -} - -/* 参考资料文字 */ -#nice .footnote-item p { -} - -/* 参考资料解释 */ -#nice .footnote-item p em { -} - -/* 行间公式 - * 最大宽度 max-width: 300% !important; - */ -#nice .block-equation svg { -} - -/* 行内公式 - */ -#nice .inline-equation svg { -} \ No newline at end of file diff --git a/docs/data-management/.DS_Store b/docs/data-management/.DS_Store new file mode 100644 index 0000000000..c40a6e6781 Binary files /dev/null and b/docs/data-management/.DS_Store differ diff --git a/docs/data-management/Big-Data/.DS_Store b/docs/data-management/Big-Data/.DS_Store new file mode 100644 index 0000000000..b24b0adada Binary files /dev/null and b/docs/data-management/Big-Data/.DS_Store differ diff --git a/docs/big-data/Bloom-Filter.md b/docs/data-management/Big-Data/Bloom-Filter.md similarity index 96% rename from docs/big-data/Bloom-Filter.md rename to docs/data-management/Big-Data/Bloom-Filter.md index 0acb95379a..9bffa3d59d 100644 --- a/docs/big-data/Bloom-Filter.md +++ b/docs/data-management/Big-Data/Bloom-Filter.md @@ -1,5 +1,3 @@ ->文章收录在 GitHub [JavaKeeper](https://github.com/Jstarfish/JavaKeeper) ，N线互联网开发必备技能兵器谱 - ## 什么是 BloomFilter **布隆过滤器**（英语：Bloom Filter）是 1970 年由布隆提出的。它实际上是一个很长的二进制向量和一系列随机映射函数。主要用于判断一个元素是否在一个集合中。 @@ -381,15 +379,19 @@ public class RedissonBloomFilterDemo { 由于使用较少，暂不深入。 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gdu79z432kj30ku0aumy2.jpg) + + +> 文章持续更新，可以微信搜「 **JavaKeeper** 」第一时间阅读，无套路领取 500+ 本电子书和 30+ 视频教学和源码，本文 **GitHub** [github.com/JavaKeeper](https://github.com/Jstarfish/JavaKeeper) 已经收录，Javaer 开发、面试必备技能兵器谱，有你想要的。 + + ## 参考与感谢 -https://www.cs.cmu.edu/~dga/papers/cuckoo-conext2014.pdf +- https://www.cs.cmu.edu/~dga/papers/cuckoo-conext2014.pdf -http://www.justdojava.com/2019/10/22/bloomfilter/ +- http://www.justdojava.com/2019/10/22/bloomfilter/ -https://www.cnblogs.com/cpselvis/p/6265825.html +- https://www.cnblogs.com/cpselvis/p/6265825.html -https://juejin.im/post/5cc5aa7ce51d456e431adac5 +- https://juejin.im/post/5cc5aa7ce51d456e431adac5 diff --git a/docs/data-management/Big-Data/Doris.md b/docs/data-management/Big-Data/Doris.md new file mode 100644 index 0000000000..2a649d38a9 --- /dev/null +++ b/docs/data-management/Big-Data/Doris.md @@ -0,0 +1,29 @@ +## 背景 + +Doris 由百度大数据部研发 ( 之前叫百度 Palo，2018年贡献到 Apache 社区后，更名为 Apache Doris )， Doris 从最初的只为解决百度凤巢报表的专用系统，到现在在百度内部，已有有超过200个产品线在使用，部署机器超过1000台，单一业务最大可达到上百 TB。 + +Apache Doris 作为一款开源的 MPP 分析型数据库产品，主要用于解决近实时的报表和多维分析。不仅能够在亚秒级响应时间即可获得查询结果，有效的支持实时数据分析。相较于其他业界比较火的 OLAP 数据库系统，Doris 的分布式架构非常简洁，支持弹性伸缩，易于运维，节省大量人力和时间成本。目前国内社区比较活跃，也有腾讯、京东、美团、小米等大厂在使用。 + + + +## 功能特性 + + + + + +更多应用场景可以参考以下链接： + +- [知乎用户画像与实时数据的架构与实践](https://mp.weixin.qq.com/s/i5qbiKN6ruOk2Snpyy6DBw) +- [Apache Doris 在京东广告平台的应用](https://mp.weixin.qq.com/s?__biz=Mzg5MDEyODc1OA==&mid=2247483954&idx=1&sn=2bf970d108d8cbda3b4984b48dd5d88e&chksm=cfe0122bf8979b3dbe0e5540e87173bd0cc0fd816c3f5d5366c8dd4c2e1cb61def8475b177c7&scene=178&cur_album_id=1536842014548393984#rd) +- [Apache Doris 在京东搜索实时 OLAP 中的应用实践](https://xie.infoq.cn/article/7d0671528ba454e67476bd76f) +- [基于Apache Doris 的小米增长分析平台实践](https://blog.csdn.net/DorisDB/article/details/108402104) +- [基于 Doris 的有道精品课数据中台建设实践](https://mp.weixin.qq.com/s?__biz=Mzg5MDEyODc1OA==&mid=2247484929&idx=1&sn=3ce7fc004042f07c17cdd50a37ae2f1b&chksm=cfe01618f8979f0ed8c10bc319cfae5c8e58cd49abb416e4f9cdd9a90546905cbe0e0501a8b6&scene=178&cur_album_id=1536842014548393984#rd) +- [Doris on ES 在快手商业化的最佳实践](https://mp.weixin.qq.com/s?__biz=Mzg5MDEyODc1OA==&mid=2247484996&idx=1&sn=6b1332612548927beb6f05ae739c11cc&chksm=cfe0165df8979f4bfd0cebfd4ec9bf4f76ece4320ad78415ad7b169cf31609ba7793cf733cef&scene=178&cur_album_id=1536842014548393984#rd) +- [Apache Doris 在韵达物流领域的应用实践](https://mp.weixin.qq.com/s?__biz=Mzg5MDEyODc1OA==&mid=2247492503&idx=1&sn=af7a930ccab3db3b3e3902ee1340f669&chksm=cfe3f38ef8947a98b0670993ed496ac9fba708eaefa264afcd7a8b093ce7d4dda332bb9aa3ca&scene=178&cur_album_id=1536842014548393984#rd) +- [百度基于 Iceberg 拓展 Doris 数据湖能力的实践](https://mp.weixin.qq.com/s?__biz=Mzg5MDEyODc1OA==&mid=2247494320&idx=1&sn=bb4b9ba450d750d30bc4f3fd51634e7e&chksm=cfe3faa9f89473bf66b2c530ae57a5b1d2d2111fad4eda9254f03e4471c69de43b8d2951fdb7&scene=178&cur_album_id=1536842014548393984#rd) +- [新东方在线教育实时数仓的落地实践](https://mp.weixin.qq.com/s?__biz=Mzg5MDEyODc1OA==&mid=2247500698&idx=1&sn=c8df28edc926ed266f8b25f330f501b5&chksm=cfe3d383f8945a95d1fc7ae4b497df9a3fc8dd5a90cc65f957f21e9061e8065ba52959cf274b&scene=178&cur_album_id=1536842014548393984#rd) +- [Doris在作业帮实时数仓中的应用&实践](https://mp.weixin.qq.com/s?__biz=Mzg5MDEyODc1OA==&mid=2247484565&idx=1&sn=bc4c7e6f1408b659cbdc235d609aa26a&chksm=cfe0148cf8979d9a6ec7af22073e1ef2aba9420d6f5af7ddc39f23b3aab226e07c02cd46883d&scene=178&cur_album_id=1536842014548393984#rd) +- [Doris 在百度用户画像人群业务的应用](https://mp.weixin.qq.com/s?__biz=Mzg5MDEyODc1OA==&mid=2247484637&idx=1&sn=962f71e4dd089af01437de74517bb403&chksm=cfe014c4f8979dd25823ee5b40373b09247a296910c60d282de7a2331c9a452045a615b144e0&scene=178&cur_album_id=1536842014548393984#rd) +- [Apache Doris在美团外卖数仓中的应用实践](https://tech.meituan.com/2020/04/09/doris-in-meituan-waimai.html) +- …… \ No newline at end of file diff --git a/docs/data-management/Big-Data/HBase.md b/docs/data-management/Big-Data/HBase.md new file mode 100755 index 0000000000..d34e43f028 --- /dev/null +++ b/docs/data-management/Big-Data/HBase.md @@ -0,0 +1,427 @@ +--- +title: HBase +date: 2023-03-09 +tags: + - HBase +categories: Big Data +--- + +![](https://hbase.apache.org/images/hbase_logo_with_orca_large.png) + +# 一、HBase 简介 + +### 1.1 HBase 定义 + +HBase 是一种分布式、可扩展、支持海量数据存储的 NoSQL 数据库。 + +### 1.2 HBase 的起源 + +HBase 是一个基于 HDFS 的分布式、面向列的开源数据库，是一个结构化数据的分布式存储系统，利用 HBase 技术可在廉价 PC Server上搭建起大规模结构化存储集群。 + +HBase 的原型是 Google 的 BigTable 论文，受到了该论文思想的启发，目前作为 Hadoop 的子项目来开发维护，用于支持结构化的数据存储。 + +[Apache](http://www.apache.org/) HBase™是 [Hadoop](http://hadoop.apache.org/) 数据库，这是一个分布式，可扩展的大数据存储。 + +当您需要随机，实时读取/写入您的大数据时使用Apache HBase™。该项目的目标是托管非常大的表 - 数十亿行×数百万列 - 在商品硬件集群上。Apache HBase是一个开源的，分布式的，版本化的非关系数据库，其模型是由 Chang 等人在 Google 的 [Bigtable：一种用于结构化数据](http://research.google.com/archive/bigtable.html)的[分布式存储系统](http://research.google.com/archive/bigtable.html)之后建模的。就像Bigtable利用Google文件系统提供的分布式数据存储一样，Apache HBase 在 Hadoop 和HDFS 之上提供了类似 Bigtable 的功能。 + + + +### 1.3 **HBase**在商业项目中的能力 + +每天： + +1. 消息量：发送和接收的消息数超过60亿 +2. 将近1000亿条数据的读写 +3. 高峰期每秒150万左右操作 +4. 整体读取数据占有约55%，写入占有45% +5. 超过2PB的数据，涉及冗余共6PB数据 +6. 数据每月大概增长300千兆字节。 + + + +### 1.4 特性 + +Hbase是一种NoSQL数据库，这意味着它不像传统的RDBMS数据库那样支持SQL作为查询语言。Hbase是一种分布式存储的数据库，技术上来讲，它更像是分布式存储而不是分布式数据库，它缺少很多RDBMS系统的特性，比如列类型，辅助索引，触发器，和高级查询语言等。那Hbase有什么特性呢？如下： + +- 强读写一致，但是不是“最终一致性”的数据存储，这使得它非常适合高速的计算聚合 +- 自动分片，通过Region分散在集群中，当行数增长的时候，Region也会自动的切分和再分配 +- 自动的故障转移 +- Hadoop/HDFS集成，和HDFS开箱即用，不用太麻烦的衔接 +- 丰富的“简洁，高效”API，Thrift/REST API，Java API +- 块缓存，布隆过滤器，可以高效的列查询优化 +- 操作管理，Hbase提供了内置的web界面来操作，还可以监控JMX指标 + + + +### 1.5 什么时候用Hbase？ + +Hbase不适合解决所有的问题： + +- 首先数据库量要足够多，如果有十亿及百亿行数据，那么Hbase是一个很好的选项，如果只有几百万行甚至不到的数据量，RDBMS是一个很好的选择。因为数据量小的话，真正能工作的机器量少，剩余的机器都处于空闲的状态 +- 其次，如果你不需要辅助索引，静态类型的列，事务等特性，一个已经用RDBMS的系统想要切换到Hbase，则需要重新设计系统。 +- 最后，保证硬件资源足够，每个HDFS集群在少于5个节点的时候，都不能表现的很好。因为HDFS默认的复制数量是3，再加上一个NameNode。 + +Hbase在单机环境也能运行，但是请在开发环境的时候使用 + + + +### 1.6 HBase 数据模型 + +逻辑上，HBase 的数据模型同关系型数据库很类似，数据存储在一张表中，有行有列。但从 HBase 的底层物理存储结构（K-V）来看，HBase 更像是一个 **multi-dimensional map**（多维度map）。 + +#### 1.6.1 HBase逻辑结构 + +![](https://img.starfish.ink/big-data/hbase-logic.png) + +> 通过横向切分 Region 和纵向切分 列族 来存储大数据 + +#### 1.6.2 HBase 物理存储结构 + +![](https://img.starfish.ink/big-data/hbase-physical-structure.png) + + + +### 1.7 数据模型(相关术语) + +#### Name Space + +命名空间，类似于关系型数据库的 database 概念，每个命名空间下有多个表。HBase 两个自带的命名空间，分别是 hbase 和 default，hbase 中存放的是 HBase 内置的表，default 表是用户默认使用的命名空间。 +一个表可以自由选择是否有命名空间，如果创建表的时候加上了命名空间后，这个表名字以 `:`作为区分！ + +#### Table + +类似于关系型数据库的表概念。不同的是，HBase 定义表时只需要声明列族即可，数据属性，比如超时时间（TTL），压缩算法（COMPRESSION）等，都在列族的定义中定义，不需要声明具体的列。 +这意味着，往 HBase 写入数据时，字段可以动态、按需指定。因此，和关系型数据库相比，HBase 能够轻松应对字段变更的场景。 + +#### Row + +HBase 表中的每行数据都由一个 RowKey 和多个 Column（列）组成。一个行包含了多个列，这些列通过列族来分类，行中的数据所属列族只能从该表所定义的列族中选取，不能定义这个表中不存在的列族，否则报错 `NoSuchColumnFamilyException`。 + +#### RowKey + +Rowkey 由用户指定的一串不重复的字符串定义，是一行的唯一标识！数据是按照 RowKey 的字典顺序存储的，并且查询数据时只能根据 RowKey 进行检索，所以 RowKey 的设计十分重要。 +如果使用了之前已经定义的 RowKey，那么会将之前的数据更新掉！ + +#### Column Family + +列族是多个列的集合。一个列族可以动态地灵活定义多个列。表的相关属性大部分都定义在列族上，同一个表里的不同列族可以有完全不同的属性配置，但是同一个列族内的所有列都会有相同的属性。 +列族存在的意义是 HBase 会把相同列族的列尽量放在同一台机器上，所以说，如果想让某几个列被放到一起，你就给他们定义相同的列族。 +官方建议一张表的列族定义的越少越好，列族太多会极大程度地降低数据库性能，且目前版本 Hbase 的架构，容易出 BUG。 + +#### Column Qualifier + +Hbase 中的列是可以随意定义的，一个行中的列不限名字、不限数量，只限定列族。因此列必须依赖于列族存在！列的名称前必须带着其所属的列族！例如 info：name，info：age。 +因为 HBase 中的列全部都是灵活的，可以随便定义的，因此创建表的时候并不需要指定列！列只有在你插入第一条数据的时候才会生成。其他行有没有当前行相同的列是不确定，只有在扫描数据的时候才能得知！ + +#### TimeStamp + +用于标识数据的不同版本（version）。时间戳默认由系统指定，也可以由用户显式指定。 +在读取单元格的数据时，版本号可以省略，如果不指定，Hbase默认会获取最后一个版本的数据返回！ + +#### Cell + +由 `{rowkey, column Family：column Qualifier, time Stamp}` 唯一确定的单元。 +Cell 中的数据是没有类型的，全部是字节码形式存储。 + +#### Region + +Region 由一个表的若干行组成！在 Region 中行的排序按照行键（rowkey）字典排序。 +Region 不能跨 RegionSever，且当数据量大的时候，HBase 会拆分 Region。 +Region 由 RegionServer 进程管理。HBase 在进行负载均衡的时候，一个 Region 有可能会从当前 RegionServer移动到其他 RegionServer 上。 +Region 是基于 HDFS 的，它的所有数据存取操作都是调用了 HDFS 的客户端接口来实现的。 + + + +### 1.8 HBase基本架构 + +![](https://img.starfish.ink/big-data/hbase-framework.png) + +- Zookeeper，作为分布式的协调。RegionServer也会把自己的信息写到ZooKeeper中。 +- HDFS是Hbase运行的底层文件系统 +- RegionServer，理解为数据节点，存储数据的。 +- Master RegionServer要实时的向Master报告信息。Master知道全局的RegionServer运行情况，可以控制RegionServer的故障转移和Region的切分。 + + + +详细点的： + +![](https://img.starfish.ink/big-data/hbase-framework1.png) + +架构角色： + +#### Region Server + +RegionServer是一个服务，负责多个Region的管理。其实现类为HRegionServer，主要作用如下: + +- 对于数据的操作：get, put, delete； +- 对于Region的操作：splitRegion、compactRegion。 +- 客户端从ZooKeeper获取RegionServer的地址，从而调用相应的服务，获取数据。 + +#### Master + +Master是所有Region Server的管理者，其实现类为HMaster，主要作用如下： + +- 对于表的操作：create, delete, alter，这些操作可能需要跨多个ReginServer，因此需要Master来进行协调！ +- 对于RegionServer的操作：分配regions到每个RegionServer，监控每个RegionServer的状态，负载均衡和故障转移。 +- 即使Master进程宕机，集群依然可以执行数据的读写，只是不能进行表的创建和修改等操作！当然Master也不能宕机太久，有很多必要的操作，比如创建表、修改列族配置，以及更重要的分割和合并都需要它的操作。 + +#### Zookeeper + +RegionServer非常依赖ZooKeeper服务，ZooKeeper管理了HBase所有RegionServer的信息，包括具体的数据段存放在哪个RegionServer上。 +客户端每次与HBase连接，其实都是先与ZooKeeper通信，查询出哪个RegionServer需要连接，然后再连接RegionServer。 +Zookeeper中记录了读取数据所需要的元数据表hbase:meata,因此关闭Zookeeper后，客户端是无法实现读操作的！ +HBase 通过 Zookeeper 来做 Master 的高可用、RegionServer 的监控、元数据的入口以及集群配置的维护等工作 + +#### HDFS + +HDFS为Hbase提供最终的底层数据存储服务，同时为HBase提供高可用的支持。 + + +#### 架构细化 + +![](https://img.starfish.ink/big-data/hbase-framework3.png) + +- HMaster是Master Server的实现，负责监控集群中的RegionServer实例，同时是所有metadata改变的接口，在集群中，通常运行在NameNode上面，[这里有一篇更细的HMaster介绍](https://links.jianshu.com/go?to=http%3A%2F%2Fblog.zahoor.in%2F2012%2F08%2Fhbase-hmaster-architecture%2F) + - HMasterInterface暴露的接口，Table(createTable, modifyTable, removeTable, enable, disable),ColumnFamily (addColumn, modifyColumn, removeColumn),Region (move, assign, unassign) + - Master运行的后台线程：LoadBalancer线程，控制region来平衡集群的负载。CatalogJanitor线程，周期性的检查hbase:meta表。 +- HRegionServer是RegionServer的实现，服务和管理Regions，集群中RegionServer运行在DataNode + - HRegionRegionInterface暴露接口：Data (get, put, delete, next, etc.)，Region (splitRegion, compactRegion, etc.) + - RegionServer后台线程：CompactSplitThread，MajorCompactionChecker，MemStoreFlusher，LogRoller +- Regions，代表table，Region有多个Store(列簇)，Store有一个Memstore和多个StoreFiles(HFiles)，StoreFiles的底层是Block。 + + + + + +## 二、Hello HBase + +https://hbase.apache.org/book.html#quickstart + + + +## 三、HBase 进阶 + +#### 3.1 Hbase中RegionServer架构 + +![](https://img.starfish.ink/big-data/hbase-regionserver.png) + +##### 1）StoreFile + +保存实际数据的物理文件，StoreFile 以 Hfile的形式存储在HDFS上。每个Store会有一个或多个StoreFile（HFile），数据在每个StoreFile中都是有序的。 + +##### 2）MemStore + +写缓存，由于HFile中的数据要求是有序的，所以数据是先存储在MemStore中，排好序后，等到达刷写时机才会刷写到HFile，每次刷写都会形成一个新的HFile。 + +##### 3）WAL + +由于数据要经MemStore排序后才能刷写到HFile，但把数据保存在内存中会有很高的概率导致数据丢失，为了解决这个问题，数据会先写在一个叫做Write-Ahead logfile的文件中，然后再写入MemStore中。所以在系统出现故障的时候，数据可以通过这个日志文件重建。(l类似于NameNode中fsimage和edits_log的作用) +每间隔hbase.regionserver.optionallogflushinterval(默认1s)， HBase会把操作从内存写入WAL。 +一个RegionServer上的所有Region共享一个WAL实例。 +WAL的检查间隔由hbase.regionserver.logroll.period定义，默认值为1小时。检查的内容是把当前WAL中的操作跟实际持久化到HDFS上的操作比较，看哪些操作已经被持久化了，被持久化的操作就会被移动到.oldlogs文件夹内（这个文件夹也是在HDFS上的）。一个WAL实例包含有多个WAL文件。WAL文件的最大数量通过hbase.regionserver.maxlogs（默认是32）参数来定义。 + +##### 4）BlockCache + +读缓存，每次查询出的数据会缓存在BlockCache中，方便下次查询。 + + + +### 3.2 Hbase读写流程 + +#### 一、写数据流程 + +![](https://img.starfish.ink/big-data/hbase-write-read.png) + +写流程： + +1. Client先访问zookeeper，获取hbase:meta表位于哪个Region Server。 +2. 访问对应的Region Server，获取hbase:meta表，根据读请求的namespace:table/rowkey，查询出目标数据位于哪个Region Server中的哪个Region中。并将该table的region信息以及meta表的位置信息缓存在客户端的meta cache，方便下次访问。 +3. 与目标Region Server进行通讯； +4. 将数据顺序写入（追加）到WAL； +5. 将数据写入对应的MemStore，数据会在MemStore进行排序； +6. 向客户端发送ack； +7. 等达到MemStore的刷写时机后，将数据刷写到HFile。 + +读写都是交给regionserver负责处理！ + + + +写数据分两步： + +- step1: 找所写数据region所在的regionserver +- step2： regionserver写数据 + + + +① 如何知道应该将请求发送给哪个regionserver? + +> 如何知道region和regionserver的对应关系？ hbase:meta +> 如何知道 hbase:meta 所在的regionserver? 查询zk的/hbase/meta-region-server获取 + +当获取meta表后，如何得知当前的数据应该放入哪个region? 继而得知如何放入哪个regionserver? + +> 根据当前数据插入的表获取所有的region,再根据插入数据的rowkey和region的startkey和endkey进行比较，判断当前数据应该放入哪个region! +> 再读取region所在行的info:server列，获取对应的regionserver即可！ + +②向regionserver发送写请求 + +a) 由regionserver找到对应region的WAL对象，进行预写日志记录 +b) 数据根据列族，写入到对应的store对象的memstore中 +c) 通知客户端写成功 + +③后续 + +memstore满的时候，或触发了其他的flush条件，memstore中的数据会被刷写到storefile中！ + 书写后，过期的WAL文件会移动到oldWALS目录中。 + + + + + +#### 二、读取数据流程 + +![](https://img.starfish.ink/big-data/hbase-read.png) + +读流程 + +1. Client先访问zookeeper，获取hbase:meta表位于哪个Region Server。 +2. 访问对应的Region Server，获取hbase:meta表，根据读请求的namespace:table/rowkey，查询出目标数据位于哪个Region Server中的哪个Region中。并将该table的region信息以及meta表的位置信息缓存在客户端的meta cache，方便下次访问。 +3. 与目标Region Server进行通讯； +4. 分别在Block Cache（读缓存），MemStore和Store File（HFile）中查询目标数据，并将查到的所有数据进行合并。此处所有数据是指同一条数据的不同版本（time stamp）或者不同的类型（Put/Delete）。 +5. 将查询到的数据块（Block，HFile数据存储单元，默认大小为64KB）缓存到Block Cache。 +6. 将合并后的最终结果返回给客户端。 + + + +总体分两步： + +①找查询的region对应的regionserver +②regionserver处理读请求，返回数据 + +step1: + +a) 查询zk的/hbase/meta-region-server获取 hbase:meta表的regionserver +b) 向 hbase:meta表的regionserver 发请求，下载meta表，缓存到客户端本地，方便以后使用 +c) 从hbase:meta表中，根据查询的rowkey和表名，确定要查询的region，继而确定region所在的regionserver + +step2: + +d) 向这些region的regionserver发请求，查询指定列族的数据 +e) 先查memstore，再查blockcache， 再查storefile +f) 如果扫到了storefile，那么数据所在的block(16k)会被缓存如blockcache + + + +#### 三、存储设计 + +在Hbase中，表被分割成多个更小的块然后分散的存储在不同的服务器上，这些小块叫做Regions，存放Regions的地方叫做RegionServer。Master进程负责处理不同的RegionServer之间的Region的分发。在Hbase实现中HRegionServer和HRegion类代表RegionServer和Region。HRegionServer除了包含一些HRegions之外，还处理两种类型的文件用于数据存储 + +- HLog， 预写日志文件，也叫做WAL(write-ahead log) +- HFile 真实的数据存储文件 + +##### HLog + +- MasterProcWAL：HMaster记录管理操作，比如解决冲突的服务器，表创建和其它DDLs等操作到它的WAL文件中，这个WALs存储在MasterProcWALs目录下，它不像RegionServer的WALs，HMaster的WAL也支持弹性操作，就是如果Master服务器挂了，其它的Master接管的时候继续操作这个文件。 + +- WAL记录所有的Hbase数据改变，如果一个RegionServer在MemStore进行FLush的时候挂掉了，WAL可以保证数据的改变被应用到。如果写WAL失败了，那么修改数据的完整操作就是失败的。 + + - 通常情况，每个RegionServer只有一个WAL实例。在2.0之前，WAL的实现叫做HLog + - WAL位于*/hbase/WALs/*目录下 + - MultiWAL: 如果每个RegionServer只有一个WAL，由于HDFS必须是连续的，导致必须写WAL连续的，然后出现性能问题。MultiWAL可以让RegionServer同时写多个WAL并行的，通过HDFS底层的多管道，最终提升总的吞吐量，但是不会提升单个Region的吞吐量。 + +- WAL的配置： + + ```jsx + // 启用multiwal + + hbase.wal.provider + multiwal + + ``` + +[Wiki百科关于WAL](https://links.jianshu.com/go?to=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FWrite-ahead_logging) + +##### HFile + +HFile是Hbase在HDFS中存储数据的格式，它包含多层的索引，这样在Hbase检索数据的时候就不用完全的加载整个文件。索引的大小(keys的大小，数据量的大小)影响block的大小，在大数据集的情况下，block的大小设置为每个RegionServer 1GB也是常见的。 + +> 探讨数据库的数据存储方式，其实就是探讨数据如何在磁盘上进行有效的组织。因为我们通常以如何高效读取和消费数据为目的，而不是数据存储本身。 + +###### Hfile生成方式 + +起初，HFile中并没有任何Block，数据还存在于MemStore中。 + +Flush发生时，创建HFile Writer，第一个空的Data Block出现，初始化后的Data Block中为Header部分预留了空间，Header部分用来存放一个Data Block的元数据信息。 + +而后，位于MemStore中的KeyValues被一个个append到位于内存中的第一个Data Block中： + +**注**：如果配置了Data Block Encoding，则会在Append KeyValue的时候进行同步编码，编码后的数据不再是单纯的KeyValue模式。Data Block Encoding是HBase为了降低KeyValue结构性膨胀而提供的内部编码机制。 + +![img](https:////upload-images.jianshu.io/upload_images/426671-fc9efc43916684b1.png?imageMogr2/auto-orient/strip|imageView2/2/w/1118/format/webp) + + + +###### 读写简流程 + +![img](https:////upload-images.jianshu.io/upload_images/426671-726c0d6e0f57814a.png?imageMogr2/auto-orient/strip|imageView2/2/w/1200/format/webp) + + + + + +## 四、HBase API + +> 不过在公司使用的时候，一般不使用原生的Hbase API，使用原生的API会导致访问不可监控，影响系统稳定性，以致于版本升级的不可控。 + + + +## 五、与 Hive的集成 + +https://cwiki.apache.org/confluence/display/Hive/HBaseIntegration + +### HBase 与 Hive 的对比 + +1．Hive + +(1) 数据仓库 + +Hive 的本质其实就相当于将 HDFS 中已经存储的文件在 Mysql 中做了一个双射关系，以方便使用 HQL 去管理查询。 + +(2) 用于数据分析、清洗 + +Hive 适用于离线的数据分析和清洗，延迟较高。 + +(3) 基于 HDFS、MapReduce + +Hive 存储的数据依旧在 DataNode 上，编写的 HQL 语句终将是转换为 MapReduce 代码执行。 + +2．HBase + +(1) 数据库 + +是一种面向列族存储的非关系型数据库。 + +(2) 用于存储结构化和非结构化的数据 + +适用于单表非关系型数据的存储，不适合做关联查询，类似 JOIN 等操作。 + +(3) 基于 HDFS + +数据持久化存储的体现形式是 HFile，存放于 DataNode 中，被 ResionServer 以 region 的形式进行管理。 + +(4) 延迟较低，接入在线业务使用 + +面对大量的企业数据，HBase 可以直线单表大量数据的存储，同时提供了高效的数据访问速度。 + + + +## 来源与感谢： + +- https://hbase.apache.org/ + +- https://hbase.apache.org/book.html#_preface + +- [《入门HBase，看这一篇就够了》](https://www.jianshu.com/p/b23800d9b227) + +- [《hbase 系列文章》](https://blog.csdn.net/weixin_42796403/category_10748539.html) + diff --git a/docs/data-management/Big-Data/HDFS.md b/docs/data-management/Big-Data/HDFS.md new file mode 100644 index 0000000000..b9a7d262ed --- /dev/null +++ b/docs/data-management/Big-Data/HDFS.md @@ -0,0 +1,68 @@ +# HDFS + +## HDFS 概述 + +### HDFS产生背景 + +随着数据量越来越大，在一个操作系统存不下所有的数据，那么就分配到更多的操作系统管理的磁盘中，但是不方便管理和维护，迫切需要一种系统来管理多台机器上的文件，这就是 **分布式文件管理系统**。HDFS 只是分布式文件管理系统中的一种。 + +### HDFS定义 + +HDFS（Hadoop Distributed File System），它是一个**文件系统**，用于存储文件，通过目录树来定位文件；其次，它是**分布式的**，由很多服务器联合起来实现其功能，集群中的服务器有各自的角色。 + +HDFS的使用场景：适合一次写入，多次读出的场景，且不支持文件的修改。适合用来做**数据分析**，并不适合用来做网盘应用。 + +### HDFS 优缺点 + +**优点：** + +- 高容错性 + - 数据自动保存多个副本。它通过增加副本的形式，提高容错性 + - 某一个副本丢失以后，它可以自动恢复 +- 适合处理大数据 + - 数据规模：能够处理数据规模达到GB、TB、甚至PB级别的数据 + - 文件规模：能够处理百万规模以上的文件数量，数量相当之大 +- 可构建在廉价机器上，通过多副本机制，提高可靠性。 + + + +**缺点：** + +- 不适合低延时数据访问，比如毫秒级的存储数据，是做不到的 +- 无法高效的对大量小文件进行存储 + - 存储大量小文件的话，它会占用 NameNode 大量的内存来存储文件目录和块信息。这样是不可取的，因为 NameNode 的内存总是有限的 + - 小文件存储的寻址时间会超过读取时间，它违反了 HDFS 的设计目标 +- 不支持并发写入、文件随机修改 + - 一个文件只能有一个写，不允许多个线程同时写 + - 仅支持数据 append（追加），不支持文件的随机修改 + + + +### HDFS 组成架构 + +![Hadoop分布式文件系统：架构和设计](https://hadoop.apache.org/docs/r1.0.4/cn/images/hdfsarchitecture.gif) + +**NameNode**（nn）：就是Master，它是一个主管、管理者。 + +1. 管理HDFS的名称空间； +2. 配置副本策略； +3. 管理数据块（Block）映射信息； +4. 处理客户端读写请求。 + +**DataNode**：就是Slave。NameNode 下达命令，DataNode执行实际的操作。 + +1. 存储实际的数据块； +2. 执行数据块的读/写操作。 + +**Client**：就是客户端。 + +1. 文件切分。文件上传HDFS的时候，Client将文件切分成一个一个的Block，然后进行上传； +2. 与NameNode交互，获取文件的位置信息； +3. 与DataNode交互，读取或者写入数据； +4. Client提供一些命令来管理HDFS，比如NameNode格式化； +5. Client可以通过一些命令来访问HDFS，比如对HDFS增删查改操作； + +**Secondary NameNode**：并非NameNode的热备。当NameNode挂掉的时候，它并不 能马上替换NameNode并提供服务。 + +1. 辅助NameNode，分担其工作量，比如定期合并Fsimage和Edits，并推送给NameNode ； +2. 在紧急情况下，可辅助恢复NameNode。 \ No newline at end of file diff --git a/docs/data-management/Big-Data/Hadoop-MapReduce.md b/docs/data-management/Big-Data/Hadoop-MapReduce.md new file mode 100644 index 0000000000..f045da3944 --- /dev/null +++ b/docs/data-management/Big-Data/Hadoop-MapReduce.md @@ -0,0 +1,1617 @@ +> 官方文档才是最好的入门学习文档：https://hadoop.apache.org/docs/r1.0.4/cn/mapred_tutorial.html + +## 一、MapReduce 概述 + +### 1.1 MapReduce 定义 + +MapReduce 是一个**分布式运算程序的编程框架**，是用户开发“基于Hadoop的数据分析应用”的核心框架。基于它写出来的应用程序能够运行在由上千个商用机器组成的大型集群上，并以一种可靠容错的方式并行处理上T级别的数据集。 + +MapReduce 核心功能是将**用户编写的业务逻辑代码**和**自带默认组件**整合成一个完整的**分布式运算程序**，并发运行在一个 Hadoop 集群上。 + +一个Map/Reduce *作业（job）* 通常会把输入的数据集切分为若干独立的数据块，由 *map任务（task）*以完全并行的方式处理它们。框架会对map的输出先进行排序， 然后把结果输入给*reduce任务*。通常作业的输入和输出都会被存储在文件系统中。 整个框架负责任务的调度和监控，以及重新执行已经失败的任务。 + +通常，Map/Reduce框架和[分布式文件系统](https://hadoop.apache.org/docs/r1.0.4/cn/hdfs_design.html)是运行在一组相同的节点上的，也就是说，计算节点和存储节点通常在一起。这种配置允许框架在那些已经存好数据的节点上高效地调度任务，这可以使整个集群的网络带宽被非常高效地利用。 + +Map/Reduce框架由一个单独的 master JobTracker 和每个集群节点一个 slave TaskTracker 共同组成。master 负责调度构成一个作业的所有任务，这些任务分布在不同的 slave 上，master 监控它们的执行，重新执行已经失败的任务。而 slave 仅负责执行由 master 指派的任务。 + +应用程序至少应该指明输入/输出的位置（路径），并通过实现合适的接口或抽象类提供map和reduce函数。再加上其他作业的参数，就构成了*作业配置（job configuration）*。然后，Hadoop 的 *job client*提交作业（jar包/可执行程序等）和配置信息给 JobTracker，后者负责分发这些软件和配置信息给 slave、调度任务并监控它们的执行，同时提供状态和诊断信息给 job-client。 + +虽然Hadoop框架是用JavaTM实现的，但Map/Reduce应用程序则不一定要用 Java来写 。 + +- [Hadoop Streaming](https://hadoop.apache.org/core/docs/r0.18.2/api/org/apache/hadoop/streaming/package-summary.html)是一种运行作业的实用工具，它允许用户创建和运行任何可执行程序 （例如：Shell工具）来做为mapper和reducer。 +- [Hadoop Pipes](https://hadoop.apache.org/core/docs/r0.18.2/api/org/apache/hadoop/mapred/pipes/package-summary.html)是一个与[SWIG](http://www.swig.org/)兼容的C++ API （没有基于JNITM技术），它也可用于实现Map/Reduce应用程序。 + +### 1.2 MapReduce 优缺点 + +#### 优点 + +1. **易于编程** + + 它简单的实现一些接口，就可以完成一个分布式程序，这个分布式程序可以分布到大量廉价的 PC机器上运行。也就是说你写一个分布式程序，跟写一个简单的串行程序是一模一样的。就是因为这个特点使得MapReduce 编程变得非常流行。 + +2. **良好的扩展性** + + 当你的计算资源不能得到满足的时候，你可以通过简单的增加机器来扩展它的计算能力。 MapReduce + +3. **高容错性** + + MapReduce 设计的初衷就是使程序能够部署在廉价的 PC 机器上，这就要求它具有很高的容错性。比如其中一台机器挂了，它可以把上面的计算任务 转移到另外一个节点上运行，不至于这个任务运行失败，而且这个过程不需要人工参与，而完全是由 Hadoop 内部完成的。 + +4. **适合 PB 级以上海量数据的离线处理** + + 可以实现上千台服务器集群并发工作，提供数据处理能力。 + +#### 缺点 + +1. **不擅长实时计算** + + MapReduce 无法像 MySQL 一样，在毫秒或者秒级内返回结果。 + +2. **不擅长流式计算** + + 流式计算的输入数据是动态的，而 MapReduce 的输入数据集是静态的，不能动态变化。这是因为MapReduce 自身的设计特点决定了数据源必须是静态的。 + +3. **不擅长DAG（有向图）计算** + + 多个应用程序存在依赖关系，后一个应用程序的输入为前一个的输出。在这种情况下，MapReduce 并不是不能做，而是使用后，每个 MapReduce作业的输出结果都会写入到磁盘，会造成大量的磁盘 IO，导致性能非常的低下。 + + + +### 1.3 MapReduce 核心思想 + +![image-20200723104319403](https://imgkr.cn-bj.ufileos.com/3d068497-63b8-4e14-8100-8e63f9d835a9.png) + + + +1. 分布式的运算程序往往需要分成至少 2 个阶段。 +2. 第一个阶段的 MapTask 并发实例，完全并行运行，互不相干。 +3. 第二个阶段的 ReduceTask 并发实例互不相干，但是他们的数据依赖于上一个阶段的所有 MapTask 并发实例的输出。 +4. MapReduce 编程模型只能包含一个 Map 阶段和一个 Reduce 阶段，如果用户的业务逻辑非常复杂，那就只能多个 MapReduce 程序，串行运行。 + + + +### 1.4 MapReduce 进程 + +一个完整的 MapReduce 程序在分布式运行时有三类实例进程： + +1. MrAppMaster： 负责整个程序的过程调度及状态协调 +2. MapTask： 负责 Map 阶段的整个数据处理流程 +3. ReduceTask：负责 Reduce 阶段的整个数据处理流程 + + + +### 1.5 官方 WordCount 源码 + +采用反编译工具反编译源码，发现 WordCount 案例有 Map 类、Reduce 类和驱动类。且数据的类型是 Hadoop 自身封装的序列化类型。 + + + +### 1.6 常用数据序列化类型 + +![image-20200723104612516](https://imgkr.cn-bj.ufileos.com/9b598cd2-756c-48b7-8526-4d075dad916b.png) + + + +### 1.7 MapReduce 编程规范 + +用户编写的程序分成三个部分：Mapper、Reducer 和 Driver。 + +1、Mapper 阶段 + +​ ① 用户自定义的 Mapper 要继承自己的父类 + +​ ② Mapper 的输入数据是 KV 对的形式（KV的类型可自定义） + +​ ③ Mapper 中的业务逻辑写在 map() 方法中 + +​ ④ Mapper 的输出数据是 KV 对的形式（KV 的类型可自定义） + +​ ⑤ map() 方法（MapTask 进程）对每一个调用一次 + +2、Reducer 阶段 + +​ ① 用户自定义的 Reducer 要继承自己的父类 + +​ ② Reducer 的输入数据类型对应 Mapper 的输出数据类型，也是KV + +​ ③ Reducer 的业务逻辑写在 reduce() 方法中 + +​ ④ ReduceTask 进程对每一组相同k的组调用一次 reduce() 方法 + +3、Driver 阶段 + +​ 相当于 YARN 集群的客户端，用于提交我们整个程序到 YARN 集群，提交的是封装了 MapReduce 程序相关运 行参数的 job 对象 + + + +### 1.8 输入与输出 + +Map/Reduce框架运转在 键值对上，也就是说， 框架把作业的输入看为是一组 键值对，同样也产出一组 键值对做为作业的输出，这两组键值对的类型可能不同。 + +框架需要对key和value的类(classes)进行序列化操作， 因此，这些类需要实现 [Writable](https://hadoop.apache.org/core/docs/r0.18.2/api/org/apache/hadoop/io/Writable.html)接口。 另外，为了方便框架执行排序操作，key类必须实现 [WritableComparable](https://hadoop.apache.org/core/docs/r0.18.2/api/org/apache/hadoop/io/WritableComparable.html)接口。 + +一个Map/Reduce 作业的输入和输出类型如下所示： + +(input) -> **map** -> -> **combine** -> -> **reduce** -> (output) + + + +### 1.9 hello world + +> 我们用官方提供的 WordCount 例子 + + + + + +## 二、Hadoop 序列化 + +### 2.1 序列化概述 + +#### 2.1.1 什么是序列化 + +序列化就是把内存中的对象，转换成字节序列（或其他数据传输协议）以便于存储到磁盘（持久化）和网络传输。 反序列化就是将收到字节序列（或其他数据传输协议）或者是磁盘的持久化数据，转换成内存中的对象。 + +#### 2.1.2 为什么要序列化 + +一般来说，“活的”对象只生存在内存里，关机断电就没有了。而且“活的” 对象只能由本地的进程使用，不能被发送到网络上的另外一台计算机。 然而序列化可以存储“活的”对象，可以将“活的”对象发送到远程计算机。 + +#### 2.1.3 为什么不用Java的序列化 + +Java 的序列化是一个重量级序列化框架（Serializable），一个对象被序列化后，会附带很多额外的信息（各种校验信息，Header，继承体系等），不便于在网络中高效传输。所以，Hadoop 自己开发了一套序列化机制（Writable）。 + +**Hadoop序列化特点：** + +1. 紧凑 ：高效使用存储空间 +2. 快速：读写数据的额外开销小 +3. 可扩展：随着通信协议的升级而可升级 +4. 互操作：支持多语言的交互 + +#### 2.2 自定义 bean 对象实现序列化接口（Writable） + +在企业开发中往往常用的基本序列化类型不能满足所有需求，比如在 Hadoop 框架内部传递一个 bean 对象，那么该对象就需要实现序列化接口。 具体实现 bean 对象序列化步骤如下 7 步。 + +1. 必须实现 Writable 接口 + +2. 反序列化时，需要反射调用空参构造函数，所以必须有空参构造 + + ```java + public FlowBean() { + super(); + } + ``` + +3. 重写序列化方法 + + ```java + @Override + public void write(DataOutput out) throws IOException { + out.writeLong(upFlow); + out.writeLong(downFlow); + out.writeLong(sumFlow); + } + ``` + +4. 重写反序列化方法 + + ```java + @Override + public void readFields(DataInput in) throws IOException { + upFlow = in.readLong(); + downFlow = in.readLong(); + sumFlow = in.readLong(); + } + ``` + +5. 注意反序列化的顺序和序列化的顺序完全一致 + +6. 要想把结果显示在文件中，需要重写 toString()，可用”\t”分开，方便后续用 + +7. 如果需要将自定义的 bean 放在 key 中传输，则还需要实现 Comparable 接口，因为 MapReduce 框中的 Shuffle 过程要求对 key 必须能排序 + + ```java + @Override + public int compareTo(FlowBean o) { + // 倒序排列，从大到小 + return this.sumFlow > o.getSumFlow() ? -1 : 1; + } + ``` + + + +#### 2.3 序列化案例实操 + +1、需求：统计每一个手机号耗费的总上行流量、下行流量、总流量 + +​ ① 输入数据 phone_data .txt + +​ ② 输入数据格式： + +![image-20200723183657595](https://imgkr.cn-bj.ufileos.com/883a403a-23e7-46ae-85f0-2ee6e6706c34.png) + +​ ③ 期望输出数据格式 + + ![](https://imgkr.cn-bj.ufileos.com/4c27e7ef-9adb-41ab-8d70-a5b6db428309.png) + + + +2、需求分析 + +![](https://imgkr.cn-bj.ufileos.com/d75006cd-24a9-48a6-ac7f-a738979c0584.png) + +3、编写 MapReduce 程序 + +① 编写流量统计的 Bean 对象 + +```java +public class FlowBean implements Writable { + + private long upFlow;// 上行流量 + private long downFlow;// 下行流量 + private long sumFlow;// 总流量 + + // 空参构造， 为了后续反射用 + public FlowBean() { + super(); + } + + public FlowBean(long upFlow, long downFlow) { + super(); + this.upFlow = upFlow; + this.downFlow = downFlow; + sumFlow = upFlow + downFlow; + } + + // 序列化方法 + @Override + public void write(DataOutput out) throws IOException { + + out.writeLong(upFlow); + out.writeLong(downFlow); + out.writeLong(sumFlow); + } + + // 反序列化方法 + @Override + public void readFields(DataInput in) throws IOException { + // 必须要求和序列化方法顺序一致 + upFlow = in.readLong(); + downFlow = in.readLong(); + sumFlow = in.readLong(); + } + + @Override + public String toString() { + return upFlow + "\t" + downFlow + "\t" + sumFlow; + } + + public long getUpFlow() { + return upFlow; + } + + public void setUpFlow(long upFlow) { + this.upFlow = upFlow; + } + + public long getDownFlow() { + return downFlow; + } + + public void setDownFlow(long downFlow) { + this.downFlow = downFlow; + } + + public long getSumFlow() { + return sumFlow; + } + + public void setSumFlow(long sumFlow) { + this.sumFlow = sumFlow; + } + + public void set(long upFlow2, long downFlow2) { + + upFlow = upFlow2; + downFlow = downFlow2; + sumFlow = upFlow2 + downFlow2; + } +} +``` + +② 编写 mapper + +```java +public class FlowCountMapper extends Mapper { + + Text k = new Text(); + FlowBean v = new FlowBean(); + + @Override + protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { + // 7 13560436666 120.196.100.99 1116 954 200 + + // 1 获取一行 + String line = value.toString(); + + // 2 切割 \t + String[] fields = line.split("\t"); + + // 3 封装对象 + k.set(fields[1]);// 封装手机号 + + long upFlow = Long.parseLong(fields[fields.length - 3]); + long downFlow = Long.parseLong(fields[fields.length - 2]); + + v.setUpFlow(upFlow); + v.setDownFlow(downFlow); +// v.set(upFlow,downFlow); + + // 4 写出 + context.write(k, v); + } +} +``` + +③ 编写 Reducer 类 + +```java +public class FlowCountReducer extends Reducer { + + FlowBean v = new FlowBean(); + + @Override + protected void reduce(Text key, Iterable values, Context context) + throws IOException, InterruptedException { +// 13568436656 2481 24681 30000 +// 13568436656 1116 954 20000 + + long sum_upFlow = 0; + long sum_downFlow = 0; + + // 1 累加求和 + for (FlowBean flowBean : values) { + + sum_upFlow += flowBean.getUpFlow(); + sum_downFlow += flowBean.getDownFlow(); + } + + v.set(sum_upFlow, sum_downFlow); + + // 2 写出 + context.write(key, v); + } +} +``` + +④ 编写 Driver 驱动类 + +```java +public class FlowsumDriver { + + public static void main(String[] args) throws IOException, ClassNotFoundException, InterruptedException { + + args = new String[]{"e:/input/inputflow","e:/output1"}; + + Configuration conf = new Configuration(); + // 1 获取job对象 + Job job = Job.getInstance(conf ); + + // 2 设置jar的路径 + job.setJarByClass(FlowsumDriver.class); + + // 3 关联mapper和reducer + job.setMapperClass(FlowCountMapper.class); + job.setReducerClass(FlowCountReducer.class); + + // 4 设置mapper输出的key和value类型 + job.setMapOutputKeyClass(Text.class); + job.setMapOutputValueClass(FlowBean.class); + + // 5 设置最终输出的key和value类型 + job.setOutputKeyClass(Text.class); + job.setOutputValueClass(FlowBean.class); + +// job.setPartitionerClass(ProvincePartitioner.class); +// +// job.setNumReduceTasks(6); +// + + // 6 设置输入输出路径 + FileInputFormat.setInputPaths(job, new Path(args[0])); + FileOutputFormat.setOutputPath(job, new Path(args[1])); + + // 7 提交job + boolean result = job.waitForCompletion(true); + + System.exit(result?0 :1); + } +} +``` + + + +## 三、MapReduce 框架原理 + +### 3.1 InputFormat 数据输入 + +#### 3.1.1 切片与 MapTask 并行度决定机制 + +1. 问题引出 + + MapTask 的并行度决定 Map 阶段的任务处理并发度，进而影响到整个 Job 的处理速度。 + + 思考：1G 的数据，启动 8 个 MapTask，可以提高集群的并发处理能力。那么 1K 的数据，也启动 8 个 MapTask，会提高集群性能吗？MapTask 并行任务是否越多越好呢？哪些因素影响了 MapTask 并行度？ + +2. MapTask 并行度决定机制 + + **数据块**：Block 是 HDFS 物理上把数据分成一块一块。 + + **数据切片**：数据切片只是在逻辑上对输入进行分片，并不会在磁盘上将其切分成片进行存储。 + +![](https://imgkr.cn-bj.ufileos.com/a2240c23-3cf4-42dc-b8cb-dfd4969cb795.png) + + + +#### 3.1.2 Job 提交流程源码和切片源码详解 + + 1．Job 提交流程源码详解，如图 4-8 所示 + +```java +waitForCompletion() +submit(); +// 1 建立连接 +connect(); +// 1）创建提交 Job 的代理 +new Cluster(getConfiguration()); +// （1）判断是本地 yarn 还是远程 +initialize(jobTrackAddr, conf); +// 2 提交 job +submitter.submitJobInternal(Job.this, cluster) +// 1）创建给集群提交数据的 Stag 路径 +Path jobStagingArea = +JobSubmissionFiles.getStagingDir(cluster, conf); +// 2）获取 jobid ，并创建 Job 路径 +JobID jobId = submitClient.getNewJobID(); +// 3）拷贝 jar 包到集群 +copyAndConfigureFiles(job, submitJobDir); +rUploader.uploadFiles(job, jobSubmitDir); +// 4）计算切片，生成切片规划文件 +writeSplits(job, submitJobDir); +maps = writeNewSplits(job, jobSubmitDir); +input.getSplits(job); +// 5）向 Stag 路径写 XML 配置文件 +writeConf(conf, submitJobFile); +conf.writeXml(out); +// 6）提交 Job,返回提交状态 +status = submitClient.submitJob(jobId, +submitJobDir.toString(), job.getCredentials()); +``` + +![](https://imgkr.cn-bj.ufileos.com/adfae885-c0a0-47e1-94d9-19976e3aeb85.png) + + + +2．FileInputFormat 切片源码解析(input.getSplits(job)) + +（1）程序先找到你数据存储的目录。 + +（2）开始遍历处理（规划切片）目录下的每一个文件 + +（3）遍历第一个文件ss.txt + +​ a）获取文件大小fs.sizeOf(ss.txt) + +​ b）计算切片大小 computeSplitSize(Math.max(minSize,Math.min(maxSize,blocksize)))=blocksize=128M c）默认情况下，切片大小=blocksize + +​ d）开始切，形成第1个切片：ss.txt—0:128M 第2个切片ss.txt—128:256M 第3个切片ss.txt—256M:300M （每次切片时，都要判断切完剩下的部分是否大于块的1.1倍，不大于1.1倍就划分一块切片） + +​ e）将切片信息写到一个切片规划文件中 + +​ f）整个切片的核心过程在getSplit()方法中完成 + +​ g）InputSplit只记录了切片的元数据信息，比如起始位置、长度以及所在的节点列表等。 + +（4）提交切片规划文件到YARN上，YARN上的MrAppMaster就可以根据切片规划文件计算开启MapTask个数。 + + + +#### 3.1.3 FileInputFormat 切片机制 + +1、切片机制 + +（1）简单地按照文件的内容长度进行切片 + +（2）切片大小，默认等于Block大小 + +（3）切片时不考虑数据集整体，而是逐个针对每一个文件单独切片 + +2、案例分析 + +![](https://imgkr.cn-bj.ufileos.com/f60b287b-5328-42a2-8009-ec1fea3fe7af.png) + + + +（1）源码中计算切片大小的公式 + +``` + Math.max(minSize, Math.min(maxSize, blockSize)); + +mapreduce.input.fileinputformat.split.minsize=1 默认值为1 + +mapreduce.input.fileinputformat.split.maxsize= Long.MAXValue 默认值Long.MAXValue +``` + + 因此，默认情况下，切片大小=blocksize。 + +（2）切片大小设置 + +maxsize（切片最大值）：参数如果调得比blockSize小，则会让切片变小，而且就等于配置的这个参数的值。 minsize（切片最小值）：参数调的比blockSize大，则可以让切片变得比blockSize还大。 + +（3）获取切片信息API + +``` +// 获取切片的文件名称 + +String name = inputSplit.getPath().getName(); + + // 根据文件类型获取切片信息 + + FileSplit inputSplit = (FileSplit) context.getInputSplit(); +``` + +#### 3.1.4 CombineTextInputFormat 切片机制 + +框架默认的 TextInputFormat 切片机制是对任务按文件规划切片，不管文件多小，都会是一个单独的切片，都会交给一个 MapTask，这样如果有大量小文件，就会产生大量的 MapTask，处理效率极其低下。 + +1、应用场景： + +CombineTextInputFormat 用于小文件过多的场景，它可以将多个小文件从逻辑上规划到 一个切片中，这样，多个小文件就可以交给一个 MapTask 处理。 + +2、虚拟存储切片最大值设置 + +``` +CombineTextInputFormat.setMaxInputSplitSize(job, 4194304);// 4m +``` + +注意：虚拟存储切片最大值设置最好根据实际的小文件大小情况来设置具体的值。 + +3、切片机制 + +生成切片过程包括：虚拟存储过程和切片过程二部分。 + +![](https://imgkr.cn-bj.ufileos.com/7111778f-3e86-46ec-8404-2e0ffe7dfe31.png) + +（1）虚拟存储过程： + +将输入目录下所有文件大小，依次和设置的 setMaxInputSplitSize 值比较，如果不大于设置的最大值，逻辑上划分一个块。如果输入文件大于设置的最大值且大于两倍， 那么以最大值切割一块；当剩余数据大小超过设置的最大值且不大于最大值 2 倍，此时 将文件均分成 2 个虚拟存储块（防止出现太小切片）。 + +例如 setMaxInputSplitSize 值为 4M，输入文件大小为 8.02M，则先逻辑上分成一个 4M。剩余的大小为 4.02M，如果按照 4M 逻辑划分，就会出现 0.02M 的小的虚拟存储 文件，所以将剩余的 4.02M 文件切分成（2.01M 和 2.01M）两个文件。 + +（2）切片过程： + +​ （a）判断虚拟存储的文件大小是否大于 setMaxInputSplitSize 值，大于等于则单独 形成一个切片。 + +​ （b）如果不大于则跟下一个虚拟存储文件进行合并，共同形成一个切片。 + +​ （c）测试举例：有 4 个小文件大小分别为 1.7M、5.1M、3.4M 以及 6.8M 这四个小 文件，则虚拟存储之后形成 6 个文件块，大小分别为： 1.7M，（2.55M、2.55M），3.4M 以及（3.4M、3.4M） + +最终会形成 3 个切片，大小分别为： （1.7+2.55）M，（2.55+3.4）M，（3.4+3.4）M 3.1.5 + +#### 3.1.5 CombineTextInputFormat 案例实操 + + 1．需求 + +将输入的大量小文件合并成一个切片统一处理。 + +（1）输入数据 准备 4 个小文件 + +（2）期望： 期望一个切片处理 4 个文件 + +2．实现过程 + + （1）不做任何处理，运行 1.6 节的 WordCount 案例程序，观察切片个数为 4。 + +（2）在 WordcountDriver 中增加如下代码，运行程序，并观察运行的切片个数为 3。 + +​ （a）驱动类中添加代码如下： + +``` +// 如果不设置 InputFormat，它默认用的是 TextInputFormat.class +job.setInputFormatClass(CombineTextInputFormat.class); +//虚拟存储切片最大值设置 4m +CombineTextInputFormat.setMaxInputSplitSize(job, 4194304); +``` + +​ （b）运行结果为 3 个切片。 + +（3）在 WordcountDriver 中增加如下代码，运行程序，并观察运行的切片个数为 1。 + +​ （a）驱动中添加代码如下： + +``` +// 如果不设置 InputFormat，它默认用的是 TextInputFormat.class +job.setInputFormatClass(CombineTextInputFormat.class); +//虚拟存储切片最大值设置 20m +CombineTextInputFormat.setMaxInputSplitSize(job, 20971520); +``` + +​ （b）运行结果为 1 个切片。 + +#### 3.1.6 FileInputFormat 实现类 + +思考：在运行MapReduce程序时，输入的文件格式包括：基于行的日志文件、 二进制格式文件、数据库表等。那么，针对不同的数据类型，MapReduce 是如何读取这些数据的呢？ + + FileInputFormat 常见的接口实现类包括： TextInputFormat 、 KeyValueTextInputFormat、NLineInputFormat、CombineTextInputFormat和自定义 InputFormat等。 + +##### 1．TextInputFormat + + TextInputFormat是默认的FileInputFormat实现类。按行读取每条记录。键是存储该行在整个文件中的 起始字节偏移量， LongWritable类 型。值是这行的内容，不包括任何行终止符（换行符和回车符）， Text类型。 + +以下是一个示例，比如，一个分片包含了如下4条文本记录。 + +``` +Rich learning form +Intelligent learning engine +Learning more convenient +From the real demand for more close to the enterprise +``` + +每条记录表示为以下键/值对： + +``` +(0,Rich learning form) +(19,Intelligent learning engine) +(47,Learning more convenient) +(72,From the real demand for more close to the enterprise) +``` + +##### 2．KeyValueTextInputFormat + +每 一 行 均 为 一 条 记 录 ， 被 分 隔 符 分 割 为 key ， value 。 可 以 通 过 在 驱 动 类 中 设 置 conf.set(KeyValueLineRecordReader.KEY_VALUE_SEPERATOR, "\t"); 来设定分隔符。默认分隔符是tab（\t）。 + +以下是一个示例，输入是一个包含4条记录的分片。其中——>表示一个（水平方向的）制表符。 + +``` +line1 ——>Rich learning form +line2 ——>Intelligent learning engine +line3 ——>Learning more convenient +line4 ——>From the real demand for more close to the enterprise +``` + +每条记录表示为以下键/值对： + +``` +(line1,Rich learning form) +(line2,Intelligent learning engine) +(line3,Learning more convenient) +(line4,From the real demand for more close to the enterprise) +``` + +此时的键是每行排在制表符之前的Text序列。 + +##### 3．NLineInputFormat + +如 果 使 用 NlineInputFormat ， 代表每个 map 进 程 处 理 的 InputSplit 不再按 Block 块 去 划 分 ， 而 是 按 NlineInputFormat指定的行数N来划分。即输入文件的总行数/N=切片数，如果不整除，切片数=商+1。 以下是一个示例，仍然以上面的4行输入为例。 + +``` +Rich learning form +Intelligent learning engine +Learning more convenient +From the real demand for more close to the enterprise +``` + +例如，如果N是2，则每个输入分片包含两行。开启2个MapTask。 + +``` +(0,Rich learning form) +(19,Intelligent learning engine) +``` + +另一个 mapper 则收到后两行： + +``` +(47,Learning more convenient) +(72,From the real demand for more close to the enterprise) +``` + +这里的键和值与TextInputFormat生成的一样。 + + + +#### 3.1.7 KeyValueTextInputFormat 使用案例 + +1．需求 + +统计输入文件中每一行的第一个单词相同的行数。 + +（1）输入数据 + +``` +banzhang ni hao +xihuan hadoop banzhang +banzhang ni hao +xihuan hadoop banzhang +``` + +（2）期望结果数据 + +``` +banzhang 2 +xihuan 2 +``` + +2．需求分析 + +![](https://imgkr.cn-bj.ufileos.com/5c536c82-af5c-410c-9245-8fdacce6ab9b.png) + +code: priv.starfish.hadoop.mr.kv + + + +#### 3.1.8 NLineInputFormat 使用案例 + + 1．需求 + + 对每个单词进行个数统计，要求根据每个输入文件的行数来规定输出多少个切片。此案 例要求每三行放入一个切片中。 + +（1）输入数据 + +``` +banzhang ni hao +xihuan hadoop banzhang +banzhang ni hao +xihuan hadoop banzhang +banzhang ni hao +xihuan hadoop banzhang +banzhang ni hao +xihuan hadoop banzhang +banzhang ni hao +xihuan hadoop banzhang banzhang ni hao +xihuan hadoop banzhang +``` + +（2）期望输出数据 + +``` +Number of splits:4 +``` + +2．需求分析 + +![](https://imgkr.cn-bj.ufileos.com/06bf6edf-4763-465f-bcd3-d228d0524998.png) + + + +3．代码实现 + +``` +priv.starfish.hadoop.mr.nline包下 +``` + +4．测试 （1）输入数据 + +``` +banzhang ni hao +xihuan hadoop banzhang +banzhang ni hao +xihuan hadoop banzhang +banzhang ni hao +xihuan hadoop banzhang +banzhang ni hao +xihuan hadoop banzhang +banzhang ni hao +xihuan hadoop banzhang banzhang ni hao +``` + +（2）输出结果的切片数，如图 4-10 所示： + +![](https://imgkr.cn-bj.ufileos.com/245975cc-6eb9-4e48-ad70-aa99c30e371e.png) + + + +#### 3.1.9 自定义 InputFormat + +在企业开发中，Hadoop框架自带的InputFormat类型不能满足所有应用场 景，需要自定义InputFormat来解决实际问题。 自定义InputFormat步骤如下： + +（1）自定义一个类继承FileInputFormat。 + +（2）改写RecordReader，实现一次读取一个完整文件封装为KV。 + + （3）在输出时使用SequenceFileOutPutFormat输出合并文件。 + + + +#### 3.1.10 自定义 InputFormat 案例实操 + +无论 HDFS 还是 MapReduce，在处理小文件时效率都非常低，但又难免面临处理大量 小文件的场景，此时，就需要有相应解决方案。可以自定义 InputFormat 实现小文件的合并。 + +1．需求 + + 将多个小文件合并成一个 SequenceFile 文件（SequenceFile 文件是 Hadoop 用来存储二 进制形式的 key-value 对的文件格式），SequenceFile 里面存储着多个文件，存储的形式为文 件路径+名称为 key，文件内容为 value。 + +（1）输入数据 + +![image-20200724153412425](https://imgkr.cn-bj.ufileos.com/311531ae-4e34-4510-a824-ef931a4bae7b.png) + +（2）期望输出文件格式 + +![image-20200724153427396](https://imgkr.cn-bj.ufileos.com/8f18a0f7-7fbd-42ad-bc3b-cc5a09e5692b.png) + + + +2．需求分析 + +1、自定义一个类继承FileInputFormat + +（1）重写isSplitable()方法，返回false不可切割 + +（2）重写createRecordReader()，创建自定义的RecordReader对象，并初始化 + +2、改写RecordReader，实现一次读取一个完整文件封装为KV + +（1）采用IO流一次读取一个文件输出到value中，因为设置了不可切片，最终把所有文件都封装到了value中 （2）获取文件路径信息+名称，并设置key + +3、设置Driver + +``` +// （1）设置输入的inputFormat + + job.setInputFormatClass(WholeFileInputformat.class); + +// （2）设置输出的outputFormat + +job.setOutputFormatClass(SequenceFileOutputFormat.class); +``` + +3．程序实现 + +``` +priv.starfish.hadoop.mr.inputformat包 +``` + + + +### 3.2 MapReduce 工作流程 + +1．流程示意图，如图 4-6，4-7 所示 + +![](https://imgkr.cn-bj.ufileos.com/26929967-2e21-4ed2-8fc2-0da3ab759204.png) + +![](https://imgkr.cn-bj.ufileos.com/7a10044f-c359-4cb7-80eb-09782bd39af8.png) + +2．流程详解 + +上面的流程是整个 MapReduce 最全工作流程，但是 Shuffle 过程只是从第 7 步开始到第 16 步结束，具体 Shuffle 过程详解，如下： + +1. MapTask 收集我们的 map()方法输出的 kv 对，放到内存缓冲区中 +2. 从内存缓冲区不断溢出本地磁盘文件，可能会溢出多个文件 +3. 多个溢出文件会被合并成大的溢出文件 +4. 在溢出过程及合并的过程中，都要调用 Partitioner 进行分区和针对 key 进行排序 +5. ReduceTask 根据自己的分区号，去各个 MapTask 机器上取相应的结果分区数据 +6. ReduceTask 会取到同一个分区的来自不同 MapTask 的结果文件，ReduceTask 会将这 些文件再进行合并（归并排序） +7. 合并成大文件后，Shuffle 的过程也就结束了，后面进入 ReduceTask 的逻辑运算过程 （从文件中取出一个一个的键值对 Group，调用用户自定义的 reduce()方法） + +3．注意 + +Shuffle 中的缓冲区大小会影响到 MapReduce 程序的执行效率，原则上说，缓冲区越大， 磁盘 io 的次数越少，执行速度就越快。 + +缓冲区的大小可以通过参数调整，参数：io.sort.mb 默认 100M。 + +4．源码解析流程 + +``` +context.write(k, NullWritable.get()); +output.write(key, value); +collector.collect(key, value,partitioner.getPartition(key, value, partitions)); +HashPartitioner(); +collect() +close() +collect.flush() +sortAndSpill() +sort() QuickSort +mergeParts(); +collector.close(); + +``` + +### 3.3 Shuffle 机制 + +#### 3.3.1 Shuffle 机制 + +Map 方法之后，Reduce 方法之前的数据处理过程称之为 Shuffle。如图 4-14 所示。 + +![](https://imgkr.cn-bj.ufileos.com/19fe35b4-d486-488e-afbd-cfeea4ae3093.png) + +#### 3.3.2 Partition 分区 + +1、问题引出 + +要求将统计结果按照条件输出到不同文件中（分区）。比如：将统计结果 按照手机归属地不同省份输出到不同文件中（分区） + +2、默认Partitioner分区 + +``` +public class HashPartitioner extends Partitioner { + public int getPartition(K key, V value, int numReduceTasks) { + return (key.hashCode() & Integer.MAX_VALUE) % numReduceTasks; + } +} +``` + +默认分区是根据key的hashCode对ReduceTasks个数取模得到的。用户没法 控制哪个key存储到哪个分区。 + +3、自定义Partitioner步骤 + +（1）自定义类继承Partitioner，重写getPartition()方法 + +``` +public class CustomPartitioner extends Partitioner { + @Override + public int getPartition(Text key, FlowBean value, int numPartitions) { + // 控制分区代码逻辑 + … … + return partition; + } +} +``` + +（2）在Job驱动中，设置自定义Partitioner + +``` + job.setPartitionerClass(CustomPartitioner.class); +``` + +（3）自定义Partition后，要根据自定义Partitioner的逻辑设置相应数量的ReduceTask + +``` +job.setNumReduceTasks(5); +``` + +4、分区总结 + +（1）如果ReduceTask的数量> getPartition的结果数，则会多产生几个空的输出文件part-r-000xx； + +（2）如果1 bean.getSumFlow()) { +result = -1; +}else if (sumFlow < bean.getSumFlow()) { +result = 1; +}else { +result = 0; +} +return result; +} + +``` + +#### 3.3.5 WritableComparable 排序案例实操（全排序） + +1．需求 + +根据案例 2.3 产生的结果再次对总流量进行排序。 + +（1）输入数据 + + 原始数据 phone_data .txt + + 第一次处理后的数据 part-r-00000 + +（2）期望输出数据 + +``` +13509468723 7335 110349 117684 +13736230513 2481 24681 27162 +13956435636 132 1512 1644 +13846544121 264 0 264 +。。。 。。。 + +``` + +2．需求分析 + +![](https://imgkr.cn-bj.ufileos.com/d5cb65cd-3a55-492b-8279-499bed05cc3e.png) + + + +3．代码实现 + +``` +priv.starfish.hadoop.mr.sort +``` + + + +#### 3.3.6 WritableComparable 排序案例实操（区内排序） + +1．需求 + +要求每个省份手机号输出的文件中按照总流量内部排序。 + +2．需求分析 + +基于前一个需求，增加自定义分区类，分区按照省份手机号设置。 + +![](https://imgkr.cn-bj.ufileos.com/2aef3942-35a7-43eb-9157-8437f1d1444e.png) + +3．案例实操 + +（1）增加自定义分区类 + +``` +priv.starfish.hadoop.mr.sort ProvincePartitioner +``` + +（2）在驱动类中添加分区类 + +``` +// 加载自定义分区类 +job.setPartitionerClass(ProvincePartitioner.class); +// 设置 Reducetask 个数 +job.setNumReduceTasks(5); + +``` + +#### 3.3.7 Combiner 合并 + +（1）Combiner是MR程序中Mapper和Reducer之外的一种组件。 + +（2）Combiner组件的父类就是Reducer。 + +（3）Combiner和Reducer的区别在于运行的位置 + +- Combiner是在每一个MapTask所在的节点运行; + +- Reducer是接收全局所有Mapper的输出结果； + +（4）Combiner的意义就是对每一个MapTask的输出进行局部汇总，以减小网络传输量。 + +（5）Combiner能够应用的前提是不能影响最终的业务逻辑，而且，Combiner的输出kv 应该跟Reducer的输入kv类型要对应起来。 + +（6）自定义 Combiner 实现步骤 + +​ （a）自定义一个 Combiner 继承 Reducer，重写 Reduce 方法 + +``` +public class WordcountCombiner extends Reducer{ +@Override +protected void reduce(Text key, Iterable +values,Context context) throws IOException, +InterruptedException { + // 1 汇总操作 +int count = 0; +for(IntWritable v :values){ +count += v.get(); +} + // 2 写出 +context.write(key, new IntWritable(count)); +} +} +``` + +（b）在 Job 驱动类中设置： + +``` +job.setCombinerClass(WordcountCombiner.class); +``` + +#### 3.3.8 Combiner 合并案例实操 + +1．需求 + +统计过程中对每一个 MapTask 的输出进行局部汇总，以减小网络传输量即采用 Combiner 功能。 + +（1）数据输入 hello.txt + +（2）期望输出数据 + +期望：Combine 输入数据多，输出时经过合并，输出数据降低。 + +2．需求分析 + +![](https://imgkr.cn-bj.ufileos.com/1f301c7a-675e-4f1e-b647-5dd55adf232a.png) + + + +3．案例实操-方案一 + +1）增加一个 WordcountCombiner 类继承 Reducer + +``` +import java.io.IOException; +import org.apache.hadoop.io.IntWritable; +import org.apache.hadoop.io.Text; +import org.apache.hadoop.mapreduce.Reducer; +public class WordcountCombiner extends Reducer{ +IntWritable v = new IntWritable(); +@Override +protected void reduce(Text key, Iterable +values, Context context) throws IOException, +InterruptedException { + // 1 汇总 +int sum = 0; +for(IntWritable value :values){ +sum += value.get(); +} +v.set(sum); +// 2 写出 +context.write(key, v); +} +} + +``` + +2）在 WordcountDriver 驱动类中指定 Combiner + +``` +// 指定需要使用 combiner，以及用哪个类作为 combiner 的逻辑 +job.setCombinerClass(WordcountCombiner.class); + +``` + +4．案例实操-方案二 + +1）将 WordcountReducer 作为 Combiner 在 WordcountDriver 驱动类中指定 + +``` +// 指定需要使用 Combiner，以及用哪个类作为 Combiner 的逻辑 +job.setCombinerClass(WordcountReducer.class); + +``` + +运行程序，如图 4-16，4-17 所示 + +![image-20200724174441853](https://imgkr.cn-bj.ufileos.com/c0935098-c23f-4be9-8f3b-812bc11833aa.png) + +#### 3.3.9 GroupingComparator 分组（辅助排序） + +对 Reduce 阶段的数据根据某一个或几个字段进行分组。 + +分组排序步骤： + +（1）自定义类继承 WritableComparator + +（2）重写 compare()方法 + +``` +@Override public int compare(WritableComparable a, WritableComparable b) { // 比较的业务逻辑 return result; } +``` + +（3）创建一个构造将比较对象的类传给父类 + +``` +protected OrderGroupingComparator() { super(OrderBean.class, true); } +``` + +#### 3.3.10 GroupingComparator 分组案例实操 + +1．需求 有如下订单数据 + +![](https://imgkr.cn-bj.ufileos.com/f83db6e2-380f-4434-bc5a-e154b720d068.png) + +现在需要求出每一个订单中最贵的商品。 + +（1）输入数据 GroupingComparator.txt + +（2）期望输出数据 + +``` +1 222.8 +2 722.4 +3 232.8 +``` + +2．需求分析 + +（1）利用“订单 id 和成交金额”作为 key，可以将 Map 阶段读取到的所有订单数据按 照 id 升序排序，如果 id 相同再按照金额降序排序，发送到 Reduce。 + +（2）在 Reduce 端利用 groupingComparator 将订单 id 相同的 kv 聚合成组，然后取第一 个即是该订单中最贵商品，如图 4-18 所示。 + +![](https://imgkr.cn-bj.ufileos.com/0d635313-8935-4423-a9f1-d0d8d5940b25.png) + +3．代码实现 + +``` +priv.starfish.hadoop.mr.order +``` + + + +### 3.4 MapTask 工作机制 + +![](https://imgkr.cn-bj.ufileos.com/caae2e63-73a1-49a5-b1f8-e9d090d92bad.png) + +（1）Read 阶段：MapTask 通过用户编写的 RecordReader，从输入 InputSplit 中解析出 一个个 key/value。 + +（2）Map 阶段：该节点主要是将解析出的 key/value 交给用户编写 map()函数处理，并 产生一系列新的 key/value。 + +（3）Collect 收集阶段：在用户编写 map()函数中，当数据处理完成后，一般会调用 OutputCollector.collect()输出结果。在该函数内部，它会将生成的 key/value 分区（调用 Partitioner），并写入一个环形内存缓冲区中。 + +（4）Spill 阶段：即“溢写”，当环形缓冲区满后，MapReduce 会将数据写到本地磁盘上， 生成一个临时文件。需要注意的是，将数据写入本地磁盘之前，先要对数据进行一次本地排 序，并在必要时对数据进行合并、压缩等操作。 + +溢写阶段详情： + + 步骤 1：利用快速排序算法对缓存区内的数据进行排序，排序方式是，先按照分区编号 Partition 进行排序，然后按照 key 进行排序。这样，经过排序后，数据以分区为单位聚集在 一起，且同一分区内所有数据按照 key 有序。 + +步骤 2：按照分区编号由小到大依次将每个分区中的数据写入任务工作目录下的临时文 件 output/spillN.out（N 表示当前溢写次数）中。如果用户设置了 Combiner，则写入文件之 前，对每个分区中的数据进行一次聚集操作。 + +步骤 3：将分区数据的元信息写到内存索引数据结构 SpillRecord 中，其中每个分区的元 信息包括在临时文件中的偏移量、压缩前数据大小和压缩后数据大小。如果当前内存索引大 小超过 1MB，则将内存索引写到文件 output/spillN.out.index 中。 + +（5）Combine 阶段：当所有数据处理完成后，MapTask 对所有临时文件进行一次合并， 以确保最终只会生成一个数据文件。 + + 当所有数据处理完后，MapTask 会将所有临时文件合并成一个大文件，并保存到文件 output/file.out 中，同时生成相应的索引文件 output/file.out.index。 在进行文件合并过程中，MapTask 以分区为单位进行合并。对于某个分区，它将采用多 轮递归合并的方式。每轮合并 io.sort.factor（默认 10）个文件，并将产生的文件重新加入待 合并列表中，对文件排序后，重复以上过程，直到最终得到一个大文件。 + +让每个 MapTask 最终只生成一个数据文件，可避免同时打开大量文件和同时读取大量 小文件产生的随机读取带来的开销。 + + + +### 3.5 ReduceTask 工作机制 + +![](https://imgkr.cn-bj.ufileos.com/e8fd10d6-55aa-4de0-9f4c-80c1d3d4b419.png) + +（1）Copy 阶段：ReduceTask 从各个 MapTask 上远程拷贝一片数据，并针对某一片数 据，如果其大小超过一定阈值，则写到磁盘上，否则直接放到内存中。 + +（2）Merge 阶段：在远程拷贝数据的同时，ReduceTask 启动了两个后台线程对内存和 磁盘上的文件进行合并，以防止内存使用过多或磁盘上文件过多。 + +（3）Sort 阶段：按照 MapReduce 语义，用户编写 reduce()函数输入数据是按 key 进行 聚集的一组数据。为了将 key 相同的数据聚在一起，Hadoop 采用了基于排序的策略。由于 各个 MapTask 已经实现对自己的处理结果进行了局部排序，因此，ReduceTask 只需对所有 数据进行一次归并排序即可。 + +（4）Reduce 阶段：reduce()函数将计算结果写到 HDFS 上。 + + + + 2．设置 ReduceTask 并行度（个数） + + ReduceTask 的并行度同样影响整个 Job 的执行并发度和执行效率，但与 MapTask 的并 发数由切片数决定不同，ReduceTask 数量的决定是可以直接手动设置： + +``` +// 默认值是 1，手动设置为 4 + +job.setNumReduceTasks(4); +``` + +3．实验：测试 ReduceTask 多少合适 + +（1）实验环境：1 个 Master 节点，16 个 Slave 节点：CPU:8GHZ，内存: 2G + +（2）实验结论： + +![](https://imgkr.cn-bj.ufileos.com/e28d97eb-befa-4910-b8a3-b5f23a46d9a8.png) + +4．注意事项 + +（1）ReduceTask=0，表示没有Reduce阶段，输出文件个数和Map个数一致。 + +（2）ReduceTask默认值就是1，所以输出文件个数为一个。 + + （3）如果数据分布不均匀，就有可能在Reduce阶段产生数据倾斜 + +（4）ReduceTask数量并不是任意设置，还要考虑业务逻辑需求，有些情况下，需要计 算全局汇总结果，就只能有1个ReduceTask。 + +（5）具体多少个ReduceTask，需要根据集群性能而定。 + +（6）如果分区数不是1，但是ReduceTask为1，是否执行分区过程。答案是：不执行分 区过程。因为在MapTask的源码中，执行分区的前提是先判断ReduceNum个数是否大于1。 不大于1肯定不执行。 + + + +### 3.6 OutputFormat 数据输出 + +#### 3.6.1 OutputFormat 接口实现类 + +OutputFormat是MapReduce输出的基类，所有实现MapReduce输出都实现了 OutputFormat 接口。下面我们介绍几种常见的OutputFormat实现类。 + +1．文本输出TextOutputFormat + +默 认的输出格式是TextOutputFormat，它把每条记录写为文本行。它的键和值可以是任 意类型，因为TextOutputFormat调用toString()方法把它们转换为字符串。 + + 2．SequenceFileOutputFormat + +将SequenceFileOutputFormat输出作为后续 MapReduce任务的输入，这便是一种好的输出 格式，因为它的格式紧凑，很容易被压缩。 + +3．自定义OutputFormat + +根据用户需求，自定义实现输出。 + + + +#### 3.6.2 自定义 OutputFormat + +1．使用场景 + +为了实现控制最终文件的输出路径和输出格式，可以自定义OutputFormat。 例如：要在一个MapReduce程序中根据数据的不同输出两类结果到不同目 录，这类灵活的输出需求可以通过自定义OutputFormat来实现。 + + 2．自定义OutputFormat步骤 + +（1）自定义一个类继承FileOutputFormat。 + +（2）改写RecordWriter，具体改写输出数据的方法write()。 + +#### 3.6.3 自定义 OutputFormat 案例实操 + +1．需求 + +过滤输入的 log 日志，包含 atguigu 的网站输出到 e:/atguigu.log，不包含 atguigu 的网站 输出到 e:/other.log。 （1）输入数据 log.txt + +（2）期望输出数据 atguigu.log other.log + +2．需求分析 + +![](https://imgkr.cn-bj.ufileos.com/ffca8111-9117-4252-b27f-724424ad3be8.png) + +3．案例实操 + +``` +priv.starfish.hadoop.mr.outputformat +``` + + + +### 3.7 Join 多种应用 + +#### 3.7.1 Reduce Join + +Reduce Join工作原理 + +Map端的主要工作：为来自不同表或文件的key/value对，打标签以区别不同 来源的记录。然后用连接字段作为key，其余部分和新加的标志作为value，最后 进行输出。 + +Reduce端的主要工作：在Reduce端以连接字段作为key的分组已经完成，我 们只需要在每一个分组当中将那些来源于不同文件的记录(在Map阶段已经打标 志)分开，最后进行合并就ok了。 + + + +#### 3.7.2 Reduce Join 案例实操 + +![](https://imgkr.cn-bj.ufileos.com/a38f7498-985f-404a-80ee-e23e10350e37.png) + +2．需求分析 通过将关联条件作为 Map 输出的 key，将两表满足 Join 条件的数据并携带数据所来源 的文件信息，发往同一个 ReduceTask，在 Reduce 中进行数据的串联，如图 4-20 所示。 + +![](https://imgkr.cn-bj.ufileos.com/a95d8957-45c1-4683-9103-ec8cade040cd.png) + +3．代码实现 1）创建商品和订合并后的 Bean 类 + +``` +priv.starfish.hadoop.mr.table +``` + +4．测试 + +5．总结 + +缺点：这种方式中，合并的操作是在Reduce阶段完成，Reduce端的处理压力 太大，Map节点的运算负载则很低，资源利用率不高，且在Reduce阶段极易产生 数据倾斜。 解决方案：Map端实现数据合并 + + + +#### 3.7.3 Map Join + +1．使用场景 Map Join 适用于一张表十分小、一张表很大的场景。 + +2．优点 + +思考：在 Reduce 端处理过多的表，非常容易产生数据倾斜。怎么办？ + +在 Map 端缓存多张表，提前处理业务逻辑，这样增加 Map 端业务，减少 Reduce 端数 据的压力，尽可能的减少数据倾斜。 + + 3．具体办法：采用 DistributedCache + +（1）在 Mapper 的 setup 阶段，将文件读取到缓存集合中。 + +（2）在驱动函数中加载缓存。 // 缓存普通文件到 Task 运行节点。 + + job.addCacheFile(new URI("file://e:/cache/pd.txt")); + +#### 3.7.4 Map Join 案例实操 + +![](https://imgkr.cn-bj.ufileos.com/6cde022c-a905-4ce0-8db8-f1ecbaef29de.png) + +2．需求分析 MapJoin 适用于关联表中有小表的情形。 + +![](https://imgkr.cn-bj.ufileos.com/e918c4d2-e8e9-444e-8100-a2ec952bed26.png) + +3．实现代码 + + （1）先在驱动模块中添加缓存文件 + +``` +priv.starfish.hadoop.mr.cache +``` + + + +### 3.8 计数器应用 + +Hadoop为每个作业维护若干内置计数器，以描述多项指标。例如，某些计数器记录 已处理的字节数和记录数，使用户可监控已处理的输入数据量和已产生的输出数据量。 + +1．计数器API + +（1）采用枚举的方式统计计数 + +``` +enum MyCounter{MALFORORMED,NORMAL} +//对枚举定义的自定义计数器加1 +context.getCounter(MyCounter.MALFORORMED).increment(1); +``` + +（2）采用计数器组、计数器名称的方式统计 + +``` + context.getCounter("counterGroup", "counter").increment(1); +``` + +组名和计数器名称随便起，但最好有意义。 + +（3）计数结果在程序运行后的控制台上查看。 + + 2. 计数器案例实操 详见数据清洗案例 + + + +### 3.9 数据清洗（ETL） + +在运行核心业务 MapReduce 程序之前，往往要先对数据进行清洗，清理掉不符合用户 要求的数据。清理的过程往往只需要运行 Mapper 程序，不需要运行 Reduce 程序。 + +#### 3.9.1 数据清洗案例实操-简单解析版 + +1．需求 + +去除日志中字段长度小于等于 11 的日志。 + +（1）输入数据 web.log + +（2）期望输出数据 每行字段长度都大于 11。 + +2．需求分析 + +需要在 Map 阶段对输入的数据根据规则进行过滤清洗。 + +3．实现代码 + +（1）编写 LogMapper 类 + +``` +priv.starfish.hadoop.mr.log +``` + +#### 3.9.2 数据清洗案例实操-复杂解析版 + +1．需求 + +对 Web 访问日志中的各字段识别切分，去除日志中不合法的记录。根据清洗规则， 输出过滤后的数据。 + +（1）输入数据 web.log + +（2）期望输出数据 都是合法的数据 + +2．实现代码 + +（1）定义一个 bean，用来记录日志数据中的各数据字段 + + + +### 3.10 MapReduce 开发总结 + +在编写 MapReduce 程序时，需要考虑如下几个方面： + +##### 1> 输入数据接口：InputFormat + +1. 默认使用的实现类是：TextInputFormat +2. TextInputFormat 的功能逻辑是：一次读一行文本，然后将该行的起始偏移量作为 key，行内容作为 value 返回 +3. KeyValueTextInputFormat 每一行均为一条记录，被分隔符分割为 key，value。默认分隔符是 tab（\t） +4. NlineInputFormat 按照指定的行数 N 来划分切片 +5. CombineTextInputFormat 可以把多个小文件合并成一个切片处理，提高处理效率 +6. 用户还可以自定义 InputFormat + +##### 2> 逻辑处理接口：Mapper + +用户根据业务需求实现其中三个方法：`map()` `setup()` `cleanup () ` + +##### 3> Partitioner 分区 + +1. 有默认实现 HashPartitioner ， 逻辑是根据 key 的哈希值 和 numReduces 来返回一个分区号；`key.hashCode()&Integer.MAXVALUE % numReduces ` +2. 如果业务上有特别的需求，可以自定义分区。 + +##### 4> Comparable 排序 + +1. 当我们用自定义的对象作为 key 来输出时，就必须要实现 WritableComparable 接口，重写其中的compareTo()方法 +2. 部分排序：对最终输出的每一个文件进行内部排序 +3. 全排序：对所有数据进行排序，通常只有一个Reduce +4. 二次排序：排序的条件有两个 + +##### 5> Combiner合并 + +Combiner 合并可以提高程序执行效率，减少IO传输。但是使用时必须不能影 响原有的业务处理结果。 + +##### 6> Reduce端分组：GroupingComparator + +在 Reduce 端对 key 进行分组。应用于：在接收的 key 为 bean 对象时，想让一个或几个字段相同（全部字段比较不相同）的 key 进入到同一个 reduce 方法时，可以采用分组排序。 + +##### 7> 逻辑处理接口：Reducer + + 用户根据业务需求实现其中三个方法：`reduce()` `setup()` `cleanup ()` + +##### 8> 输出数据接口：OutputFormat + +1. 默认实现类是 TextOutputFormat，功能逻辑是：将每一个KV对，向目标文本文件输出一行 +2. 将 SequenceFileOutputFormat 输出作为后续 MapReduce 任务的输入，这便是一种好的输出格式，因为它的格式紧凑，很容易被压缩 +3. 用户还可以自定义 OutputFormat \ No newline at end of file diff --git a/docs/big-data/Hello-BigData.md b/docs/data-management/Big-Data/Hello-BigData.md similarity index 98% rename from docs/big-data/Hello-BigData.md rename to docs/data-management/Big-Data/Hello-BigData.md index 853219193c..d11e8c8933 100644 --- a/docs/big-data/Hello-BigData.md +++ b/docs/data-management/Big-Data/Hello-BigData.md @@ -44,7 +44,7 @@ ### 1.4 企业数据部的一般组织结构 -![](../_images/big-data/company.png) +![](../../_images/big-data/company.png) @@ -124,7 +124,7 @@ Hortonworks 文档较好。 -![](../_images/big-data/hadoop.png) +![](../../_images/big-data/hadoop.png) #### 2.5.1 HDFS 架构概述 @@ -156,7 +156,7 @@ MapReduce 将计算过程分为两个阶段：Map 和 Reduce 2）Reduce 阶段对 Map 结果进行汇总 -![](../_images/big-data/MapReduce.png) +![](../../_images/big-data/MapReduce.png) @@ -166,7 +166,7 @@ MapReduce 将计算过程分为两个阶段：Map 和 Reduce ### 2.6 大数据技术生态体系 -![](../_images/big-data/system.png) +![](../../_images/big-data/system.png) 图中涉及的技术名词解释如下： @@ -204,7 +204,7 @@ MapReduce 将计算过程分为两个阶段：Map 和 Reduce ### 2.7 推荐系统框架图 -![](../_images/big-data/recommend .png) +![](../../_images/big-data/recommend.png) diff --git a/docs/data-management/Big-Data/Hive-Explain.md b/docs/data-management/Big-Data/Hive-Explain.md new file mode 100644 index 0000000000..984ca38cf7 --- /dev/null +++ b/docs/data-management/Big-Data/Hive-Explain.md @@ -0,0 +1,56 @@ +> [官方文档](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Explain) + + + +我们知道 MySQL 有执行计划， + +Hive 的底层就是 MapReduce 的编程实现，我们可以通过执行计划详细的了解执行过程。 + +### 语法 + +Hive 和 MySQL 类似，也提供了一个 `EXPLAIN` 命令来显示查询语句的执行计划，如下 + +```sql +EXPLAIN [EXTENDED|CBO|AST|DEPENDENCY|AUTHORIZATION|LOCKS|VECTORIZATION|ANALYZE] query +``` + +`EXPLAIN + 可选参数 + 查询语句` + +可选参数： + +- EXTENDED： +- CBO： +- AST： +- DEPENDENCY： +- AUTHORIZATION： +- LOCKS： +- VECTORIZATION： +- ANALYZE： + +### 示例 + +直接 `EXPLAN`，不加任何参数 + +```bash +hive> explain select * from xurilogall where dt='2020-03-01'; +OK +STAGE DEPENDENCIES: + Stage-0 is a root stage // + +STAGE PLANS: + Stage: Stage-0 + Fetch Operator + limit: -1 + Processor Tree: + TableScan + alias: xurilogall + Statistics: Num rows: 3391560 Data size: 5860617060 Basic stats: COMPLETE Column stats: NONE + Select Operator + expressions: log_time (type: string), cpctranid (type: bigint), msg_type (type: int), accountid (type: bigint), cpcplanid (type: bigint), clickdate (type: string), cost (type: decimal(10,0)), costa (type: decimal(10,0)), costb (type: decimal(10,0)), costc (type: decimal(10,0)), servicetype (type: int), cpcid (type: bigint), keyword (type: string), position (type: bigint), cpcgrpid (type: bigint), ip (type: string), pid (type: string), istest (type: int), agentid (type: int), regioncode (type: int), type (type: int), msgid (type: string), account_budgetday (type: int), cpcplan_budgetday (type: int), validity (type: int), ideaid (type: bigint), querykey (type: string), codeid (type: int), domainid (type: int), topdomainid (type: int), subdomainid (type: int), clickdevice (type: int), clickdevicestat (type: int), cpcplan_devicetype (type: int), extraideaid (type: int), linkpicid (type: int), cpcstyle_type (type: int), complete_pid (type: string), place_holder (type: int), pattern_match (type: string), showregion (type: string), eesf (type: string), extend_reserved (type: bigint), cumulate_status (type: smallint), cumulate_amount (type: bigint), real_account_budget (type: bigint), plantype (type: int), matchtype (type: smallint), max_price (type: bigint), '2020-03-01' (type: string) + outputColumnNames: _col0, _col1, _col2, _col3, _col4, _col5, _col6, _col7, _col8, _col9, _col10, _col11, _col12, _col13, _col14, _col15, _col16, _col17, _col18, _col19, _col20, _col21, _col22, _col23, _col24, _col25, _col26, _col27, _col28, _col29, _col30, _col31, _col32, _col33, _col34, _col35, _col36, _col37, _col38, _col39, _col40, _col41, _col42, _col43, _col44, _col45, _col46, _col47, _col48, _col49 + Statistics: Num rows: 3391560 Data size: 5860617060 Basic stats: COMPLETE Column stats: NONE + ListSink + +Time taken: 0.26 seconds, Fetched: 17 row(s) +``` + diff --git a/docs/data-management/Big-Data/Hive.md b/docs/data-management/Big-Data/Hive.md new file mode 100644 index 0000000000..548fb247db --- /dev/null +++ b/docs/data-management/Big-Data/Hive.md @@ -0,0 +1,199 @@ +# Hive + +> https://www.cnblogs.com/qingyunzong/p/8707885.html +> +> 《[Hive SQL之数据类型和存储格式](https://www.cnblogs.com/qingyunzong/p/8733924.html)》 +> +> [Hive的DDL操作](https://www.cnblogs.com/qingyunzong/p/8723271.html) + + + +## 什么是 Hive + +1. Hive 由 Facebook 实现并开源 + +2. 是基于 Hadoop 的一个数据仓库工具 + +3. 可以将结构化的数据映射为一张数据库表 + +4. 并提供 HQL(Hive SQL)查询功能 + +5. 底层数据是存储在 HDFS 上 + +6. Hive的本质是将 SQL 语句转换为 MapReduce 任务运行 + +7. 使不熟悉 MapReduce 的用户很方便地利用 HQL 处理和计算 HDFS 上的结构化的数据，适用于离线的批量数据计算。 + + + +数据仓库之父比尔·恩门（Bill Inmon）在 1991 年出版的“Building the Data Warehouse”（《建 立数据仓库》）一书中所提出的定义被广泛接受——数据仓库（Data Warehouse）是一个面 向主题的（Subject Oriented）、集成的（Integrated）、相对稳定的（Non-Volatile）、反映历史 变化（Time Variant）的数据集合，用于支持管理决策(Decision Making Support)。 + +　　Hive 依赖于 HDFS 存储数据，Hive 将 HQL 转换成 MapReduce 执行，所以说 Hive 是基于 Hadoop 的一个数据仓库工具，实质就是一款基于 HDFS 的 MapReduce 计算框架，对存储在 HDFS 中的数据进行分析和管理 + +![img](https://images2018.cnblogs.com/blog/1228818/201804/1228818-20180403192903767-826182114.png) + + + +## 为什么使用 Hive + +直接使用 MapReduce 所面临的问题： + +　　1、人员学习成本太高 + +　　2、项目周期要求太短 + +　　3、MapReduce实现复杂查询逻辑开发难度太大 + +为什么要使用 Hive： + +　　1、更友好的接口：操作接口采用类 SQL 的语法，提供快速开发的能力 + +　　2、更低的学习成本：避免了写 MapReduce，减少开发人员的学习成本 + +　　3、更好的扩展性：可自由扩展集群规模而无需重启服务，还支持用户自定义函数 + + + +### Hive 特点 + +**优点**： + +　　1、**可扩展性,横向扩展**，Hive 可以自由的扩展集群的规模，一般情况下不需要重启服务 横向扩展：通过分担压力的方式扩展集群的规模 纵向扩展：一台服务器cpu i7-6700k 4核心8线程，8核心16线程，内存64G => 128G + +　　2、**延展性**，Hive 支持自定义函数，用户可以根据自己的需求来实现自己的函数 + +　　3、**良好的容错性**，可以保障即使有节点出现问题，SQL 语句仍可完成执行 + +**缺点**： + +　　1、**Hive 不支持记录级别的增删改操作**，但是用户可以通过查询生成新表或者将查询结 果导入到文件中（当前选择的 hive-2.3.2 的版本支持记录级别的插入操作） + +　　2、**Hive 的查询延时很严重**，因为 MapReduce Job 的启动过程消耗很长时间，所以不能 用在交互查询系统中。 + +　　3、**Hive 不支持事务**（因为不没有增删改，所以主要用来做 OLAP（联机分析处理），而 不是 OLTP（联机事务处理），这就是数据处理的两大级别）。 + + + +### Hive 和 RDBMS 的对比 + +![img](https://images2018.cnblogs.com/blog/1228818/201804/1228818-20180403193352838-1398998715.png) + +总结： + +　　Hive 具有 SQL 数据库的外表，但应用场景完全不同，**Hive 只适合用来做海量离线数 据统计分析，也就是数据仓库**。 + + + + + +## Hive的架构 + +![img](https://images2018.cnblogs.com/blog/1228818/201804/1228818-20180403193501903-1989526977.png) + +从上图看出hive的内部架构由四部分组成： + + + +### 1、用户接口: shell/CLI, jdbc/odbc, webui Command Line Interface + +　　CLI，Shell 终端命令行（Command Line Interface），采用交互形式使用 Hive 命令行与 Hive 进行交互，最常用（学习，调试，生产） + +　　JDBC/ODBC，是 Hive 的基于 JDBC 操作提供的客户端，用户（开发员，运维人员）通过 这连接至 Hive server 服务 + +　　Web UI，通过浏览器访问 Hive + + + +### 2、跨语言服务 ： thrift server 提供了一种能力，让用户可以使用多种不同的语言来操纵hive + +　　Thrift 是 Facebook 开发的一个软件框架，可以用来进行可扩展且跨语言的服务的开发， Hive 集成了该服务，能让不同的编程语言调用 Hive 的接口 + + + +### 3、底层的Driver： 驱动器Driver，编译器Compiler，优化器Optimizer，执行器Executor + +　　Driver 组件完成 HQL 查询语句从词法分析，语法分析，编译，优化，以及生成逻辑执行 计划的生成。生成的逻辑执行计划存储在 HDFS 中，并随后由 MapReduce 调用执行 + +　　Hive 的核心是驱动引擎， 驱动引擎由四部分组成： + +　　　　(1) 解释器：解释器的作用是将 HiveSQL 语句转换为抽象语法树（AST） + +　　　　(2) 编译器：编译器是将语法树编译为逻辑执行计划 + +　　　　(3) 优化器：优化器是对逻辑执行计划进行优化 + +　　　　(4) 执行器：执行器是调用底层的运行框架执行逻辑执行计划 + + + +### 4、元数据存储系统 ： RDBMS MySQL + +　　**元数据**，通俗的讲，就是存储在 Hive 中的数据的描述信息。 + +　　Hive 中的元数据通常包括：表的名字，表的列和分区及其属性，表的属性（内部表和 外部表），表的数据所在目录 + +　　Metastore 默认存在自带的 Derby 数据库中。缺点就是不适合多用户操作，并且数据存 储目录不固定。数据库跟着 Hive 走，极度不方便管理 + +　　解决方案：通常存我们自己创建的 MySQL 库（本地 或 远程） + +　　Hive 和 MySQL 之间通过 MetaStore 服务交互 + + + +### 执行流程 + +　　HiveQL 通过命令行或者客户端提交，经过 Compiler 编译器，运用 MetaStore 中的元数 据进行类型检测和语法分析，生成一个逻辑方案(Logical Plan)，然后通过的优化处理，产生 一个 MapReduce 任务。 + +[回到顶部](https://www.cnblogs.com/qingyunzong/p/8707885.html#_labelTop) + +## Hive的数据组织 + +1、Hive 的存储结构包括**数据库、表、视图、分区和表数据**等。数据库，表，分区等等都对 应 HDFS 上的一个目录。表数据对应 HDFS 对应目录下的文件。 + +2、Hive 中所有的数据都存储在 HDFS 中，没有专门的数据存储格式，因为 **Hive 是读模式** （Schema On Read），可支持 TextFile，SequenceFile，RCFile 或者自定义格式等 + +3、 只需要在创建表的时候告诉 Hive 数据中的**列分隔符和行分隔符**，Hive 就可以解析数据 + +　　Hive 的默认列分隔符：控制符 **Ctrl + A，\x01 Hive** 的 + +　　Hive 的默认行分隔符：换行符 **\n** + +4、Hive 中包含以下数据模型： + +　　**database**：在 HDFS 中表现为${hive.metastore.warehouse.dir}目录下一个文件夹 + +　　**table**：在 HDFS 中表现所属 database 目录下一个文件夹 + +　　**external table**：与 table 类似，不过其数据存放位置可以指定任意 HDFS 目录路径 + +　　**partition**：在 HDFS 中表现为 table 目录下的子目录 + +　　**bucket**：在 HDFS 中表现为同一个表目录或者分区目录下根据某个字段的值进行 hash 散 列之后的多个文件 + +　　**view**：与传统数据库类似，只读，基于基本表创建 + +5、Hive 的元数据存储在 RDBMS 中，除元数据外的其它所有数据都基于 HDFS 存储。默认情 况下，Hive 元数据保存在内嵌的 Derby 数据库中，只能允许一个会话连接，只适合简单的 测试。实际生产环境中不适用，为了支持多用户会话，则需要一个独立的元数据库，使用 MySQL 作为元数据库，Hive 内部对 MySQL 提供了很好的支持。 + +6、Hive 中的表分为内部表、外部表、分区表和 Bucket 表 + +**内部表和外部表的区别：** + +　　**删除内部表，删除表元数据和数据** + +　　**删除外部表，删除元数据，不删除数据** + +**内部表和外部表的使用选择：** + +　　大多数情况，他们的区别不明显，如果数据的所有处理都在 Hive 中进行，那么倾向于 选择内部表，但是如果 Hive 和其他工具要针对相同的数据集进行处理，外部表更合适。 + +　　使用外部表访问存储在 HDFS 上的初始数据，然后通过 Hive 转换数据并存到内部表中 + +　　使用外部表的场景是针对一个数据集有多个不同的 Schema + +　　通过外部表和内部表的区别和使用选择的对比可以看出来，hive 其实仅仅只是对存储在 HDFS 上的数据提供了一种新的抽象。而不是管理存储在 HDFS 上的数据。所以不管创建内部 表还是外部表，都可以对 hive 表的数据存储目录中的数据进行增删操作。 + +**分区表和分桶表的区别：** + +　　Hive 数据表可以根据某些字段进行分区操作，细化数据管理，可以让部分查询更快。同 时表和分区也可以进一步被划分为 Buckets，分桶表的原理和 MapReduce 编程中的 HashPartitioner 的原理类似。 + +　　分区和分桶都是细化数据管理，但是分区表是手动添加区分，由于 Hive 是读模式，所 以对添加进分区的数据不做模式校验，分桶表中的数据是按照某些分桶字段进行 hash 散列 形成的多个文件，所以数据的准确性也高很多 \ No newline at end of file diff --git a/docs/data-management/Big-Data/Kylin.md b/docs/data-management/Big-Data/Kylin.md new file mode 100644 index 0000000000..e6dc3c4dad --- /dev/null +++ b/docs/data-management/Big-Data/Kylin.md @@ -0,0 +1,153 @@ +--- +title: Kylin +date: 2023-03-09 +tags: + - OLAP +categories: OLAP +--- + +![](https://siteprod-cdn.kyligence.io/wp-content/uploads/2019/02/apache-kylin896512.png) + +> 之前总觉得 Javaer 对数据层面的掌握，在 MySQL、缓存、MQ 这些就行了，可是现如今这个行业是真“卷”啊，这不，大数据相关知识也得了解了解。这玩意一般是用来做大数据的固化查询的，也早就应用在了各个互联网公司，后续还有篇实际使用。 +> +> Apache Kylin™是一个开源的、分布式的分析型数据仓库，提供 Hadoop/Spark 之上的 SQL 查询接口及多维分析（OLAP）能力以支持超大规模数据，最初由 eBay 开发并贡献至开源社区。它能在亚秒内查询巨大的表。 +> +> 有完善的中文文档：http://kylin.apache.org/cn/docs/ + + + +## 前言 + +随着移动互联网、物联网等技术的发展，近些年人类所积累的数据正在呈爆炸式的增长，大数据时代已经来临。但是海量数据的收集只是大数据技术的第一步，如何让数据产生价值才是大数据领域的终极目标。Hadoop 的出现解决了数据存储问题，但如何对海量数据进行OLAP 查询，却一直令人十分头疼。 + +企业中的查询大致可分为**即席查询**和**定制查询**两种。之前出现的很多 OLAP 引擎，包括 Hive、Presto、SparkSQL 等，虽然在很大程度上降低了数据分析的难度，但它们都只适用于即席查询的场景。它们的优点是查询灵活，但是随着数据量和计算复杂度的增长，响应时间不能得到保证。而定制查询多数情况下是对用户的操作做出实时反应，Hive 等查询引擎动辄数分钟甚至数十分钟的响应时间显然是不能满足需求的。在很长一段时间里，企业只能对数据仓库中的数据进行提前计算，再将算好后的结果存储在 MySQL 等关系型数据库中，再提供给用户进行查询。但是当业务复杂度和数据量逐渐升高后，使用这套方案的开发成本和维护成本都显著上升。因此，如何对已经固化下来的查询进行亚秒级返回一直是企业应用中的一个痛点。 + +在这种情况下，Apache Kylin 应运而生。不同于“大规模并行处理”（Massive Parallel Processing，MPP）架构的 Hive、Presto 等，Apache Kylin 采用“**预计算**”的模式，用户只需要提前定义好查询维度，Kylin 将帮助我们进行计算，并将结果存储到 **HBase** 中，为海量数据的查询和分析提供亚秒级返回，是一种典型的“**空间换时间**”的解决方案。Apache Kylin 的出现不仅很好地解决了海量数据快速查询的问题，也避免了手动开发和维护提前计算程序带来的一系列麻烦。 + +Apache Kylin 最初由 eBay 公司开发，并贡献给 Apache 基金会，但是目前 Apache Kylin 的核心开发团队已经自立门户，创建了 Kyligence 公司。值得一提的是，Apache Kylin 是第一个由中国人主导的 Apache 顶级项目（2017 年 4 月 19 日，华为的 CarbonData 成为 Apache 顶级项目，因此 Apache Kylin 不再是唯一由国人贡献的 Apache 顶级项目）。由于互联网技术和开源思想进入我国的时间较晚，开源软件的世界一直是由西方国家主导，在数据领域也不例外。从 Hadoop 到 Spark，再到最近大热的机器学习平台 TenserFlow 等，均是如此。但近些年来，我们很欣喜地看到以 Apache Kylin 为首的各种以国人主导的开源项目不断地涌现出来，这些技术不断缩小着我国与西方开源技术强国之间的差距，提升我国技术人员在国际开源社区的影响力。 + +## 一、核心概念 + +在了解 Apache Kylin 的使用以前，我们有必要先来了解一下在 Apache Kylin 中会出现的核心概念。 + +### 数据仓库 + +Data Warehouse，简称 DW，中文名数据仓库，是商业智能（BI）中的核心部分。主要是将不同数据源的数据整合到一起，通过多维分析等方式为企业提供决策支持和报表生成。那么它与我们熟悉的传统关系型数据库有什么不同呢？ + +简而言之，用途不同。数据库面向事务，而数据仓库面向分析。数据库一般存储在线的业务数据，需要对上层业务的改变做出实时反应，涉及到增删查改等操作，所以需要遵循三大范式，需要 ACID。而数据仓库中存储的则主要是历史数据，主要目的是为企业决策提供支持，所以可能存在大量数据冗余，但利于多个维度查询，为决策者提供更多观察视角。 + +在传统 BI 领域中，数据仓库的数据同样存储在 Oracle、MySQL 等数据库中，而在大数据领域中最常用的数据仓库就是 Apache Hive，Hive 也是 Apache Kylin 默认的数据源。 + +### OLAP + +OLAP（Online Analytical Process），联机分析处理，以多维度的方式分析数据，一般带有主观的查询需求，多应用在数据仓库。 + +与之对应的是 OLTP（Online Transaction Process），联机事务处理，侧重于数据库的增删查改等常用业务操作。了解了上面数据库与数据仓库的区别后，OLAP 与 OLTP 的区别就不难理解了。 + +### 维度和度量 + +维度和度量是数据分析领域中两个常用的概念。 + +简单地说，维度就是观察数据的角度。比如传感器的采集数据，可以从时间的维度来观察： + +![](https://img.starfish.ink/big-data/kylin-dimension.png) + +也可以进一步细化，从时间和设备两个角度观察： + +![](https://img.starfish.ink/big-data/kylin-dimension2.png) + +**维度**一般是离散的值，比如时间维度上的每一个独立的日期，或者设备维度上的每一个独立的设备。因此统计时可以把维度相同的记录聚合在一起，然后应用聚合函数做累加、均值、最大值、最小值等聚合计算。 + +**度量**就是被聚合的统计值，也就是聚合运算的结果，它一般是连续的值，如以上两个图中的温度值，或是其他测量点，比如湿度等等。通过对度量的比较和分析，我们就可以对数据做出评估，比如这个月设备运行是否稳定，某个设备的平均温度是否明显高于其他同类设备等等。 + +### Cube 和 Cuboid + +了解了维度和度量之后，我们可以将数据模型上的所有字段进行分类：它们要么是维度，要么是度量。根据定义好的维度和度量，我们就可以构建 Cube 了。 + +对于一个给定的数据模型，我们可以对其上的所有维度进行组合。对于 N 个维度来说，组合所有可能性共有 2 的 N 次方种。对于每一种维度的组合，将度量做聚合计算，然后将运算的结果保存为一个物化视图，称为 Cuboid。所有维度组合的 Cuboid 作为一个整体，被称为 Cube。 + +举个例子。假设有一个电商的销售数据集，其中维度包括时间（Time）、商品（Item）、地点（Location）和供应商（Supplier），度量为销售额（GMV）。那么所有维度的组合就有 2 的 4 次方，即 16 种，比如一维度（1D）的组合有[Time]、[Item]、[Location]、[Supplier] 4 种；二维度（2D）的组合有[Time Item]、[Time Location]、[Time Supplier]、[Item Location]、[Item Supplier]、[Location Supplier] 6种；三维度（3D）的组合也有 4 种；最后零维度（0D）和四维度（4D）的组合各有 1 种，总共 16 种。 + +计算 Cubiod，即按维度来聚合销售额。如果用 SQL 语句来表达计算 Cuboid [Time, Location]，那么 SQL 语句如下： + +```sql +select Time, Location, Sum(GMV) as GMV from Sales group by Time, Location +``` + +将计算的结果保存为物化视图，所有 Cuboid 物化视图的总称就是 Cube。 + +> Cube 中只包含聚合数据，所以用户的所有查询都应该是聚合查询 (包含 “group by”)，不能出现 select * 这种 + +### 事实表和维度表 + +事实表（Fact Table）是指存储有事实记录的表，如系统日志、销售记录、传感器数值等；事实表的记录是动态增长的，所以它的体积通常远大于维度表。 + +维度表（Dimension Table）或维表，也称为查找表（Lookup Table），是与事实表相对应的一种表；它保存了维度的属性值，可以跟事实表做关联；相当于将事实表上经常重复的属性抽取、规范出来用一张表进行管理。常见的维度表有：日期表（存储与日期对应的周、月、季度等属性）、地区表（包含国家、省/州、城市等属性）等。维度表的变化通常不会太大。使用维度表有许多好处： + +- 缩小了事实表的大小。 +- 便于维度的管理和维护，增加、删除和修改维度的属性，不必对事实表的大量记录进行改动。 +- 维度表可以为多个事实表重用。 + +### 星形模型 + +星形模型（Star Schema）是数据挖掘中常用的几种多维数据模型之一。它的特点是只有一张事实表，以及零到多个维度表，事实表与维度表通过主外键相关联，维度表之间没有关联，就像许多小星星围绕在一颗恒星周围，所以名为星形模型。 + +另一种常用的模型是**雪花模型**（SnowFlake Schema），就是将星形模型中的某些维表抽取成更细粒度的维表，然后让维表之间也进行关联，这种形状酷似雪花的的模型称为雪花模型。 + +还有一种更为复杂的模型，具有多个事实表，维表可以在不同事实表之间公用，这种模型被称为**星座模型**。 + +不过，Kylin 目前只支持星形模型和雪花模型。 + + + +## 二、技术架构 + +Apache Kylin 系统主要可以分为**离线构建**和**在线查询**两部分。 + +![](http://kylin.apache.org/assets/images/kylin_diagram.png) + +上图左侧为数据源，目前 Kylin 默认的数据源是 Apache Hive，保存着待分析的用户数据。 + +根据元数据的定义，构建引擎从数据源抽取数据，并构建 Cube。数据以关系表的形式输入，并且必须符合星形模型。构建技术主要为 MapReduce（Spark目前在beta版本）。构建后的 Cube 保存在右侧存储引擎中，目前 Kylin 默认的存储为 Apache HBase。 + +完成离线构建后，用户可以从上方的查询系统发送 SQL 进行查询分析。Kylin 提供了 RESTful API、JDBC/ODBC 接口供用户调用。无论从哪个接口进入，SQL 最终都会来到 REST 服务层，再转交给查询引擎进行处理。查询引擎解析 SQL，生成基于关系表的逻辑执行计划，然后将其转译为基于 Cube 的物理执行计划，最后查询预计算生成的 Cube 并产生结果。整个过程不会访问原始数据源。如果用户提交的查询语句未在 Kylin 中预先定义，Kylin 会返回一个错误。 + +值得一提的是，Kylin 对数据源、执行引擎和 Cube 存储三个核心模块提取出了抽象层，这意味着这三个模块可以被任意地扩展和替换。比如可以使用 Spark 替代 MapReduce 作为 Cube 的构建引擎，使用 Cassandra 替代 HBase 作为 Cube 计算后数据的存储等。良好的扩展性使得 Kylin 可以在这个技术发展日新月异的时代方便地使用更先进的技术替代现有技术，做到与时俱进，也使用户可以针对自己的业务特点对 Kylin 进行深度定制。 + + + +Apache Kylin 的这种架构使得它拥有许多非常棒的特性： + +- SQL 接口： + + Kylin 主要的对外接口就是以 SQL 的形式提供的。SQL 简单易用的特性极大地降低了 Kylin 的学习成本，不论是数据分析师还是 Web开发程序员都能从中收益。 + +- 支持海量数据集 + + 不论是 Hive、SparkSQL，还是 Impala、Presto，都改变不了这样一个事实：查询时间随着数据量的增长而线性增长。 + + 而 Apache Kylin 使用预计算技术打破了这一点。Kylin 在数据集规模上的局限性主要取决于维度的个数和基数，而不是数据集的大小，所以 Kylin 能更好地支持海量数据集的查询。 + +- 亚秒级响应 + + 同样受益于预计算技术，Kylin 的查询速度非常快，因为复杂的连接、聚合等操作都在 Cube 的构建过程中已经完成了。 + +- 水平扩展 + + Apache Kylin 同样可以使用集群部署方式进行水平扩展。但部署多个节点只能提高 Kylin 处理查询的能力，而不能提升它的预计算能力。 + +- 可视化集成 + + Apache Kylin 提供了 ODBC/JDBC 接口和 RESTful API，可以很方便地与 Tableau 等数据可视化工具集成。数据团队也可以在开放的API 上进行二次开发。 + + + + + +### 参考与感谢： + +- 原文：[《一文读懂Apache Kylin》](https://www.jianshu.com/p/abd5e90ab051) +- [《Apache Kylin 在百度地图的实践》](https://www.infoq.cn/article/practis-of-apache-kylin-in-baidu-map/) +- 美团技术团队：[Apache Kylin的实践与优化](https://tech.meituan.com/2020/11/19/apache-kylin-practice-in-meituan.html) +- [【硬刚Kylin】Kylin入门/原理/调优/OLAP解决方案和行业典型应用](https://www.modb.pro/db/79232) + diff --git a/docs/data-management/Big-Data/OLAP.md b/docs/data-management/Big-Data/OLAP.md new file mode 100755 index 0000000000..bca7f45014 --- /dev/null +++ b/docs/data-management/Big-Data/OLAP.md @@ -0,0 +1 @@ +![](https://cdn.educba.com/academy/wp-content/uploads/2019/04/What-is-OLAP.jpg) \ No newline at end of file diff --git a/docs/data-management/Big-Data/Phoenix.md b/docs/data-management/Big-Data/Phoenix.md new file mode 100755 index 0000000000..74efa69593 --- /dev/null +++ b/docs/data-management/Big-Data/Phoenix.md @@ -0,0 +1,235 @@ +## 一、Phoenix 简介 + +Phoenix 最早是 saleforce 的一个开源项目，后来成为 Apache 的顶级项目。 + +Phoenix 构建在 HBase 之上的开源 SQL 层。能够让我们使用标准的 JDBC API 去建表, 插入数据和查询 HBase 中的数据, 从而可以避免使用 HBase 的客户端 API。在我们的应用和 HBase 之间添加了 Phoenix, 并不会降低性能, 而且我们也少写了很多代码。 + +phoneix的本质就是定义了大量的协处理器，使用协处理器帮助我们完成HBase的操作！ + + + +## 二、Phoenix 特点 + +将 SQL 查询编译为 HBase 扫描 +确定扫描 Rowkey 的最佳开始和结束位置 +扫描并行执行 +将 where 子句推送到服务器端的过滤器 +通过协处理器进行聚合操作 +完美支持 HBase 二级索引创建 +DML命令以及通过DDL命令创建和操作表和版本化增量更改。 +容易集成：如Spark，Hive，Pig，Flume和Map Reduce。 + + + +## 三、Phoenix 架构 + +![](https://img-blog.csdnimg.cn/20210131165822416.png) + + + +## 四、和Hbase中数据的关系映射 + +| 模型 | HBase | Phoneix(SQL) | +| ---- | ------------------------------------ | ------------------------ | +| 库 | namespace | database | +| 表 | table | table | +| 列族 | column Family cf:cq | 列 | +| 列名 | column Quailfier | | +| 值 | value | | +| 行键 | rowkey（唯一） dt\|area\|city\|adsid | 主键(dt,area,city,adsid) | + +​ Phoenix 将 HBase 的数据模型映射到关系型模型中！ + +![](https://img-blog.csdnimg.cn/20210131165851572.png) + +## 五、Phoenix使用场景 + +### 5.1 场景一：新建表 + +通过phoneix在hbase中创建表，通过phoneix向hbase的表执行增删改查！ + +```sql +--使用默认的0号列族 +CREATE TABLE IF NOT EXISTS "us_population" ( + state CHAR(2) NOT NULL, + city VARCHAR NOT NULL, + population BIGINT + CONSTRAINT my_pk PRIMARY KEY (state, city)); + +--如果希望列族有意义 +CREATE TABLE IF NOT EXISTS us_population ( + state CHAR(2) NOT NULL, + city VARCHAR NOT NULL, + info.population BIGINT + CONSTRAINT my_pk PRIMARY KEY (state, city)); + +``` + +默认所有的小写，都会转大写！在查询时的小写也会转大写！ + +如果必须用小写，需要加"", 在以后操作时，都需要加"",尽量不要使用小写！ + +### 5.2 场景二：映射Hbase中已有表 + +hbase中已经存在了一个表，在phoneix中建表，映射上，进行操作！ + +在phoneix中，只读操作！ 创建一个View! + +```sql +CREATE VIEW IF NOT EXISTS "t2" ( + id VARCHAR PRIMARY KEY , + "cf1"."name" VARCHAR , + "cf2"."age" VARCHAR , + "cf2"."gender" VARCHAR + ); + +``` + +在phoneix中，可读可写操作！ 创建一个Table! + +```sql +CREATE TABLE IF NOT EXISTS "t4" ( + id VARCHAR PRIMARY KEY , + "cf1"."name" VARCHAR , + "cf2"."age" VARCHAR , + "cf2"."gender" VARCHAR + ) column_encoded_bytes=0; +``` + + + +## 六、Phoenix使用语法 + +进入Phoenix客户端界面 + +``` +[hadoop@hadoop101 phoenix]$ /opt/module/phoenix/bin/sqlline.py hadoop102,hadoop103,hadoop104:2181 +``` + + + +### 1）显示所有表 + +``` +!table 或 !tables +``` + + + +### 2）创建表 + +``` +CREATE TABLE IF NOT EXISTS us_population ( + state CHAR(2) NOT NULL, + city VARCHAR NOT NULL, + population BIGINT + CONSTRAINT my_pk PRIMARY KEY (state, city)); +``` + +说明: + +- char类型必须添加长度限制 +- varchar 可以不用长度限制 +- 主键映射到 HBase 中会成为 Rowkey. 如果有多个主键(联合主键), 会把多个主键的值拼成 rowkey +- 在 Phoenix 中, 默认会把表名,字段名等自动转换成大写. 如果要使用消息, 需要把他们用双引号括起来. + + + +### 3）插入数据 + +``` +upsert into us_population values('NY','NewYork',8143197); +upsert into us_population values('CA','Los Angeles',3844829); +upsert into us_population values('IL','Chicago',2842518); +说明: upset可以看成是update和insert的结合体. +``` + +### 4）查询记录 + +``` +select * from US_POPULATION; +select * from us_population where state='NY'; +``` + + +5）删除记录 + +``` +delete from us_population where state='NY'; +``` + + +6）删除表 + +``` +drop table us_population; +``` + + +7）退出命令行 + +``` +! quit +``` + + + +## 七、使用JDBC连接 + +添加如下依赖： + + +```xml + + + org.apache.phoenix + phoenix-core + 5.0.0-HBase-2.0 + + + + + org.apache.hadoop + hadoop-common + 3.1.1 + +``` + +测试连接： + +```java +import java.sql.*; + +public class MyPhoenix { +public static void main(String[] args) throws SQLException, ClassNotFoundException { + + //Class.forName("org.apache.phoenix.jdbc.PhoenixDriver"); + + Connection connection = DriverManager.getConnection("jdbc:phoenix:hadoop102:2181"); + + String sql = "select * from US_POPULATION"; + + PreparedStatement ps = connection.prepareStatement(sql); + + ResultSet resultSet = ps.executeQuery(); + + while (resultSet.next()){ + System.out.println(resultSet.getString("STATE")+" " + + resultSet.getString("CITY") + " " + + resultSet.getLong("POPULATION")); + + } + + resultSet.close(); + + ps.close(); + + connection.close(); +} +} +``` + + + + + \ No newline at end of file diff --git a/docs/data-structure/binary-tree.md b/docs/data-management/Big-Data/README.md similarity index 100% rename from docs/data-structure/binary-tree.md rename to docs/data-management/Big-Data/README.md diff --git "a/docs/data-management/Big-Data/\346\216\250\350\215\220\347\263\273\347\273\237\344\271\213\347\224\250\346\210\267\347\224\273\345\203\217\345\237\272\347\241\200\342\200\224\342\200\224\345\244\247\346\225\260\346\215\256\346\227\266\344\273\243.md" "b/docs/data-management/Big-Data/\346\216\250\350\215\220\347\263\273\347\273\237\344\271\213\347\224\250\346\210\267\347\224\273\345\203\217\345\237\272\347\241\200\342\200\224\342\200\224\345\244\247\346\225\260\346\215\256\346\227\266\344\273\243.md" new file mode 100644 index 0000000000..d3e991dfe1 --- /dev/null +++ "b/docs/data-management/Big-Data/\346\216\250\350\215\220\347\263\273\347\273\237\344\271\213\347\224\250\346\210\267\347\224\273\345\203\217\345\237\272\347\241\200\342\200\224\342\200\224\345\244\247\346\225\260\346\215\256\346\227\266\344\273\243.md" @@ -0,0 +1,341 @@ +## 推荐系统之用户画像基础——大数据时代 + +> 在互联网步入大数据时代后，用户行为给企业的产品和服务带来了一系列的改变和重塑，其中最大的变化在于，用户的一切行为在企业面前是可“追溯”“分析”的。企业内保存了大量的原始数据和各种业务数据，这是企业经营活动的真实记录，如何更加有效地利用这些数据进行分析和评估，成为企业基于更大数据量背景的问题所在。随着大数据技术的深入研究与应用，企业的关注点日益聚焦在如何利用大数据来为精细化运营和精准营销服务，而要做精细化运营，首先要建立本企业的用户画像。 + +## 画像简介 + +用户画像，即用户信息标签化，通过收集用户的社会属性、消费习惯、偏好特征等各个维度的数据，进而对用户或者产品特征属性进行刻画，并对这些特征进行分析、统计，挖掘潜在价值信息，从而抽象出用户的信息全貌，如图1-1所示。用户画像可看作企业应用大数据的根基，是定向广告投放与个性化推荐的前置条件，为数据驱动运营奠定了基础。由此看来，如何从海量数据中挖掘出有价值的信息越发重要。 + +![图1-1　某用户标签化](https://mmbiz.qpic.cn/mmbiz_png/zHbzQPKIBPh4WJCtSFtGYXlpCqDmKJVpTpUfVXDSfibTbvS7tCuiappvNj35twj1ZibMCx0W6dfcFibFVyCSRLVicQg/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + + + +大数据已经兴起多年，其对于互联网公司的应用来说已经如水、电、空气对于人们的生活一样，成为不可或缺的重要组成部分。从基础设施建设到应用层面，主要有数据平台搭建及运维管理、数据仓库开发、上层应用的统计分析、报表生成及可视化、用户画像建模、个性化推荐与精准营销等应用方向。 + +很多公司在大数据基础建设上投入很多，也做了不少报表，但业务部门觉得大数据和传统报表没什么区别，也没能体会大数据对业务有什么帮助和价值，究其原因，其实是“数据静止在数据仓库，是死的”。 + +而用户画像可以帮助大数据“走出”数据仓库，针对用户进行个性化推荐、精准营销、个性化服务等多样化服务，是大数据落地应用的一个重要方向。数据应用体系的层级划分如图1-2所示。 + +![图1-2　数据应用体系的层级划分](https://mmbiz.qpic.cn/mmbiz_png/zHbzQPKIBPh4WJCtSFtGYXlpCqDmKJVpaPuUNSibZqNyWibpYpNG1Xp40zKZsZEd9mzEHezPfticyG1nR9YgKXJzA/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + + + +### 标签类型： + +用户画像建模其实就是对用户“打标签”，从对用户打标签的方式来看，一般分为3种类型（如图1-3所示）：①统计类标签；②规则类标签；③机器学习挖掘类标签。 + +![图1-3　标签类型](https://mmbiz.qpic.cn/mmbiz_png/zHbzQPKIBPh4WJCtSFtGYXlpCqDmKJVp4TQDdvTUze7YPI0L5gh1CB6jlUfyurzePUGpFqfzq0QSN1ubkibsTUg/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + + + +下面我们介绍这3种类型的标签的区别： + +**① 统计类标签** + +这类标签是最为基础也最为常见的标签类型，例如，对于某个用户来说，其性别、年龄、城市、星座、近7日活跃时长、近7日活跃天数、近7日活跃次数等字段可以从用户注册数据、用户访问、消费数据中统计得出。该类标签构成了用户画像的基础。 + +**② 规则类标签** + +该类标签基于用户行为及确定的规则产生。例如，对平台上“消费活跃”用户这一口径的定义为“近30天交易次数≥2”。在实际开发画像的过程中，由于运营人员对业务更为熟悉，而数据人员对数据的结构、分布、特征更为熟悉，因此规则类标签的规则由运营人员和数据人员共同协商确定； + +**③ 机器学习挖掘类标签** + +该类标签通过机器学习挖掘产生，用于对用户的某些属性或某些行为进行预测判断。例如，根据一个用户的行为习惯判断该用户是男性还是女性、根据一个用户的消费习惯判断其对某商品的偏好程度。该类标签需要通过算法挖掘产生。 + +在项目工程实践中，一般统计类和规则类的标签即可以满足应用需求，在开发中占有较大比例。机器学习挖掘类标签多用于预测场景，如判断用户性别、用户购买商品偏好、用户流失意向等。一般地，机器学习标签开发周期较长，开发成本较高，因此其开发所占比例较小。 + +## **数据架构** + +在整个工程化方案中，系统依赖的基础设施包括Spark、Hive、HBase、Airflow、MySQL、Redis、Elasticsearch。除去基础设施外，系统主体还包括Spark Streaming、ETL、产品端3个重要组成部分。图1-4所示是用户画像数仓架构图，下面对其进行详细介绍。 + +![图1-4　用户画像数仓架构](https://mmbiz.qpic.cn/mmbiz_png/zHbzQPKIBPh4WJCtSFtGYXlpCqDmKJVp2TKMOXBhDvHAsk8MSrxduyprWjDvTEZee8rN8Xj4CxAdFq814k0kAg/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + + + +图1-4下方虚线框中为常见的数据仓库ETL加工流程，也就是将每日的业务数据、日志数据、埋点数据等经过ETL过程，加工到数据仓库对应的ODS层、DW层、DM层中。 + +中间的虚线框即为用户画像建模的主要环节，用户画像不是产生数据的源头，而是对基于数据仓库ODS层、DW层、DM层中与用户相关数据的二次建模加工。在ETL过程中将用户标签计算结果写入Hive，由于不同数据库有不同的应用场景，后续需要进一步将数据同步到MySQL、HBase、Elasticsearch等数据库中。 + +- Hive：存储用户标签计算结果、用户人群计算结果、用户特征库计算结果。 +- MySQL：存储标签元数据，监控相关数据，导出到业务系统的数据。 +- HBase：存储线上接口实时调用类数据。 +- Elasticsearch：支持海量数据的实时查询分析，用于存储用户人群计算、用户群透视分析所需的用户标签数据（由于用户人群计算、用户群透视分析的条件转化成的SQL语句多条件嵌套较为复杂，使用Impala执行也需花费大量时间）。 + +用户标签数据在Hive中加工完成后，部分标签通过Sqoop同步到MySQL数据库，提供用于BI报表展示的数据、多维透视分析数据、圈人服务数据；另一部分标签同步到HBase数据库用于产品的线上个性化推荐。 + + + +## 主要覆盖模块 + +搭建一套用户画像方案整体来说需要考虑8个模块的建设，如图1-5所示。 + +- 用户画像基础：需要了解、明确用户画像是什么，包含哪些模块，数据仓库架构是什么样子，开发流程，表结构设计，ETL设计等。这些都是框架，大方向的规划，只有明确了方向后续才能做好项目的排期和人员投入预算。这对于评估每个开发阶段重要指标和关键产出非常重要，重点可看1.4节。 +- 数据指标体系：根据业务线梳理，包括用户属性、用户行为、用户消费、风险控制等维度的指标体系。 +- 标签数据存储：标签相关数据可存储在Hive、MySQL、HBase、Elasticsearch等数据库中，不同存储方式适用于不同的应用场景。 +- 标签数据开发：用户画像工程化的重点模块，包含统计类、规则类、挖掘类、流式计算类标签的开发，以及人群计算功能的开发，打通画像数据和各业务系统之间的通路，提供接口服务等开发内容。 + +![图1-5　用户画像主要覆盖模块](https://mmbiz.qpic.cn/mmbiz_png/zHbzQPKIBPh4WJCtSFtGYXlpCqDmKJVp4NkIeMicco9SeWrz14feew70WcEVREcicckVTrJos40icCIy9KshQK87Q/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + + + +- 开发性能调优：标签加工、人群计算等脚本上线调度后，为了缩短调度时间、保障数据的稳定性等，需要对开发的脚本进行迭代重构、调优。 +- 作业流程调度：标签加工、人群计算、同步数据到业务系统、数据监控预警等脚本开发完成后，需要调度工具把整套流程调度起来。本书讲解了Airflow这款开源ETL工具在调度画像相关任务脚本上的应用。 +- 用户画像产品化：为了能让用户数据更好地服务于业务方，需要以产品化的形态应用在业务上。产品化的模块主要包括标签视图、用户标签查询、用户分群、透视分析等。 +- 用户画像应用：画像的应用场景包括用户特征分析、短信、邮件、站内信、Push消息的精准推送、客服针对用户的不同话术、针对高价值用户的极速退货退款等VIP服务应用。 + + + +## 开发阶段流程 + +本节主要介绍画像系统开发上线的流程以及各阶段的关键产出。 + +### 1. 开发上线流程 + +用户画像建设项目流程，如图1-6所示。 + +![img](https://mmbiz.qpic.cn/mmbiz_png/zHbzQPKIBPh4WJCtSFtGYXlpCqDmKJVpcfpqNsE9sT17YPdxlia8ibFpQian868ianT5aianlqUZrPChknIiaLJ7OvNg/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +图1-6　用户画像建设项目流程 + +#### **第一阶段：目标解读** + +在建立用户画像前，首先需要明确用户画像服务于企业的对象，再根据业务方需求，明确未来产品建设目标和用户画像分析之后的预期效果。 + +一般而言，用户画像的服务对象包括运营人员和数据分析人员。不同业务方对用户画像的需求有不同的侧重点，就运营人员来说，他们需要分析用户的特征、定位用户行为偏好，做商品或内容的个性化推送以提高点击转化率，所以画像的侧重点就落在了用户个人行为偏好上；就数据分析人员来说，他们需要分析用户行为特征，做好用户的流失预警工作，还可根据用户的消费偏好做更有针对性的精准营销。 + +#### **第二阶段：任务分解与需求调研** + +经过第一阶段的需求调研和目标解读，我们已经明确了用户画像的服务对象与应用场景，接下来需要针对服务对象的需求侧重点，结合产品现有业务体系和“数据字典”规约实体和标签之间的关联关系，明确分析维度。就后文将要介绍的案例而言，需要从用户属性画像、用户行为画像、用户偏好画像、用户群体偏好画像等角度去进行业务建模。 + +#### **第三阶段：需求场景讨论与明确** + +在本阶段，数据运营人员需要根据与需求方的沟通结果，输出产品用户画像需求文档，在该文档中明确画像应用场景、最终开发出的标签内容与应用方式，并就该文档与需求方反复沟通并确认无误。 + +#### **第四阶段：应用场景与数据口径确认** + +经过第三个阶段明确了需求场景与最终实现的标签维度、标签类型后，数据运营人员需要结合业务与数据仓库中已有的相关表，明确与各业务场景相关的数据口径。在该阶段中，数据运营方需要输出产品用户画像开发文档，该文档需要明确应用场景、标签开发的模型、涉及的数据库与表以及应用实施流程。该文档不需要再与运营方讨论，只需面向数据运营团队内部就开发实施流程达成一致意见即可。 + +#### 第五阶段：特征选取与模型数据落表 + +本阶段中数据分析挖掘人员需要根据前面明确的需求场景进行业务建模，写好HQL逻辑，将相应的模型逻辑写入临时表中，并抽取数据校验是否符合业务场景需求。 + +#### 第六阶段：线下模型数据验收与测试 + +数据仓库团队的人员将相关数据落表后，设置定时调度任务，定期增量更新数据。数据运营人员需要验收数仓加工的HQL逻辑是否符合需求，根据业务需求抽取表中数据查看其是否在合理范围内，如果发现问题要及时反馈给数据仓库人员调整代码逻辑和行为权重的数值。 + +#### 第七阶段：线上模型发布与效果追踪 + +经过第六阶段，数据通过验收之后，会通过Git进行版本管理，部署上线。使用Git进行版本管理，上线后通过持续追踪标签应用效果及业务方反馈，调整优化模型及相关权重配置。 + +### 2. 各阶段关键产出 + +为保证程序上线的准时性和稳定性，需要规划好各阶段的任务排期和关键产出。画像体系的开发分为几个主要阶段，包括前期指标体系梳理、用户标签开发、ETL调度开发、打通数据服务层、画像产品端开发、面向业务方推广应用、为业务方提供营销策略的解决方案等，如表1-1所示。 + +![](https://mmbiz.qpic.cn/mmbiz_png/zHbzQPKIBPh4WJCtSFtGYXlpCqDmKJVp0jJTYd2gzWVpeUGTZDUgd7JYRJaC9RROKXwmBophBXStoJlEfBXI1g/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +![表1-1　用户画像项目各阶段关键产出](https://mmbiz.qpic.cn/mmbiz_png/zHbzQPKIBPh4WJCtSFtGYXlpCqDmKJVpAHB1mTebJEYNyyOFm14GLVYTND1JUGgKib84naFia1xAfM6xAHrRFY2Q/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +![img](https://mmbiz.qpic.cn/mmbiz_png/zHbzQPKIBPh4WJCtSFtGYXlpCqDmKJVpVUuPIfqyzAubkeHWgcicNyIpfkmVSkibUyQBlibSGKBcBIaCGgnPYs3fw/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + + + +- 标签开发：根据业务需求和应用场景梳理标签指标体系，调研业务上定义的数据口径，确认数据来源，开发相应的标签。标签开发在整个画像项目周期中占有较大比重。 +- ETL调度开发：梳理需要调度的各任务之间的依赖关系，开发调度脚本及调度监控告警脚本，上线调度系统。 +- 打通服务层接口：为了让画像数据走出数据仓库，应用到用户身上，需要打通数据仓库和各业务系统的接口。 +- 画像产品化：需要产品经理与业务人员、技术开发人员一起对接业务需求点和产品功能实现形式，画产品原型，确定工作排期。Java Web端开发完成后，需要数据开发人员向对应的库表中灌入数据。 +- 开发调优：在画像的数据和产品端搭建好架构、能提供稳定服务的基础上，为了让调度任务执行起来更加高效、提供服务更加稳健，需要对标签计算脚本、调度脚本、数据同步脚本等相关计算任务进行重构优化。 +- 面向业务方推广应用：用户画像最终的价值产出点是业务方应用画像数据进行用户分析，多渠道触达运营用户，分析ROI，提升用户活跃度或营收。因此，面向业务人员推广画像系统的使用方式、提供针对具体业务场景的解决方案显得尤为重要。在该阶段，相关人员需要撰写画像的使用文档，提供业务支持。 + + + +## **画像应用的落地** + +用户画像最终的价值还是要落地运行，为业务带来实际价值。这里需要开发标签的数据工程师和需求方相互协作，将标签应用到业务中。否则开发完标签后，数据还是只停留在数据仓库中，没有为业务决策带来积极作用。 + +画像开发过程中，还需要开发人员组织数据分析、运营、客服等团队的人员进行画像应用上的推广。对于数据分析人员来说，可能会关注用户画像开发了哪些表、哪些字段以及字段的口径定义；对运营、客服等业务人员来说，可能更关注用户标签定义的口径，如何在Web端使用画像产品进行分析、圈定用户进行定向营销，以及应用在业务上数据的准确性和及时性。 + +只有业务人员在日常工作中真正应用画像数据、画像产品，才能更好地推动画像标签的迭代优化，带来流量提升和营收增长，产出业绩价值。 + + + +## 某用户画像案例 + +这里通过一个实践案例来将大家更好地带入实际开发画像、应用画像标签的场景中。本节主要介绍案例背景及相关的元数据，以及开发标签中可以设计的表结构样式。 + +在本案例的开发工作中，基于Spark计算引擎，主要涉及的语言包括HiveQL、Python、Scala、Shell等。 + +### 1. 案例背景介绍 + +某图书电商网站拥有超过千万的网购用户群体，所售各品类图书100余万种。用户在平台上可进行浏览、搜索、收藏、下单、购买等行为。商城的运营需要解决两个问题：一方面在企业产品线逐渐扩张、信息资源过载的背景下，如何在兼顾自身商业目标的同时更好地满足消费者的需求，为用户带来更个性化的购物体验，通过内容的精准推荐，更好地提高用户的点击转化率；另一方面在用户规模不断增长的背景下，运营方考虑建立用户流失预警机制，及时识别将要流失的用户群体，采取运营措施挽回用户。 + +商城自建立以来，数据仓库中积累着大量的业务数据、日志数据及埋点数据。如何充分挖掘沉淀在数据仓库中的数据的价值，有效支持用户画像的建设，成为当前的重要工作。 + +### 2. 相关元数据 + +在本案例中，可以获取的数据按其类型分为：业务类数据和用户行为数据。其中业务类数据是指用户在平台上下单、购买、收藏物品、货物配送等与业务相关的数据；用户行为数据是指用户搜索某条信息、访问某个页面、点击某个按钮、提交某个表单等通过操作行为产生（在解析日志的埋点表中）的数据。 + +涉及数据仓库中的表主要包括用户信息表、商品订单表、图书信息表、图书类目表、App端日志表、Web端日志表、商品评论表等。下面就用户画像建模过程中会用到的一些数据表做详细介绍。 + +**① 用户信息表** + +用户信息表（见表1-2）存放有关用户的各种信息，如用户姓名、年龄、性别、电话号码、归属地等信息。 + +![表1-2　用户信息表（dim.user_basic_info）](https://mmbiz.qpic.cn/mmbiz_png/zHbzQPKIBPh4WJCtSFtGYXlpCqDmKJVppfm8eJYIV7hgzbSjl6Q77g2PHIBPA5TksO2icYXH5taogN9iaaZ3908A/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + + + +**② 商品订单表** + +商品订单表（见表1-3）存放商品订单的各类信息，包括订单编号、用户id、用户姓名、订单生成时间、订单状态等信息。 + +![表1-3　商品订单表（dw.order_info_fact）](https://mmbiz.qpic.cn/mmbiz_png/zHbzQPKIBPh4WJCtSFtGYXlpCqDmKJVpRcDWcJic2G9BnuP9e2WwrbeiciblSrvzZQpjArEvUZo8daVs6qorSIAhA/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + + + +**③ 埋点日志表** + +埋点日志表（见表1-4）存放用户访问App时点击相关控件的打点记录。通过在客户端做埋点，从日志数据中解析出来。 + +![表1-4　埋点日志表（ods.page_event_log）](https://mmbiz.qpic.cn/mmbiz_png/zHbzQPKIBPh4WJCtSFtGYXlpCqDmKJVp2aMbDUGibicZJoPQLhs0lm9zBrzicxDSnT28DrxQk0Xia25aAhKjBSicq0w/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +![img](https://mmbiz.qpic.cn/mmbiz_png/zHbzQPKIBPh4WJCtSFtGYXlpCqDmKJVp7XEFMHAaqB08qH21WVID1p4gQfuicKG0GyFpx2z3HHLLXwla0C6PboA/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + + + +**④ 访问日志表** + +访问日志表（见表1-5）存放用户访问App的相关信息及用户的LBS相关信息，通过在客户端埋点，从日志数据中解析出来。 + +![表1-5　访问日志表（ods.page_view_log）](https://mmbiz.qpic.cn/mmbiz_png/zHbzQPKIBPh4WJCtSFtGYXlpCqDmKJVpfqsB5PI3DVZXSib7MPuqsuQDxXOykrMLhc5S6s9PMfzL10vBLgM2jhw/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + + + +**⑤ 商品评论表** + +商品评论表（见表1-6）存放用户对商品的评论信息。 + +![表1-6　商品评论表（dw.book_comment）](https://mmbiz.qpic.cn/mmbiz_png/zHbzQPKIBPh4WJCtSFtGYXlpCqDmKJVpiawexfHaricV4A8fib6rhLSReYmp2ibqpaFK6O2CZLiaiagiaxC1gQm4LaSzg/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + + + +**⑥ 搜索日志表** + +搜索日志表（见表1-7）存放用户在App端搜索相关的日志数据。 + +![表1-7　搜索日志表（dw.app_search_log）](https://mmbiz.qpic.cn/mmbiz_png/zHbzQPKIBPh4WJCtSFtGYXlpCqDmKJVpIdInicOXMTwLuEY8zuzuRiclvCUiaibg7iaFLQFZ73Lxv80FuEtibEK71eFg/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + + + +**⑦ 用户收藏表** + +用户收藏表（见表1-8）记录用户收藏图书的数据。 + +![表1-8　用户收藏表（dw.book_collection_df）](https://mmbiz.qpic.cn/mmbiz_png/zHbzQPKIBPh4WJCtSFtGYXlpCqDmKJVp9mNAPicfIDmabvYSdgh0nw8QUpib3PnemxibIPeHdLnSPyQdHjibPekP9A/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + + + +**⑧ 购物车信息表** + +购物车信息表（见表1-9）记录用户将图书加入购物车的数据。 + +![表1-9　购物车信息表（dw.shopping_cart_df）](https://mmbiz.qpic.cn/mmbiz_png/zHbzQPKIBPh4WJCtSFtGYXlpCqDmKJVpscGQ28EKumicicbo81WaavCCskjkZyEdAurwx8ck3TvahH3cLIXGuCLw/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + + + +### 3. 画像表结构设计 + +表结构设计也是画像开发过程中需要解决的一个重要问题。 + +表结构设计的重点是要考虑存储哪些信息、如何存储（数据分区）、如何应用（如何抽取标签）这3个方面的问题。 + +不同业务背景有不同的设计方式，这里提供两种设计思路：一是每日全量数据的表结构；二是每日增量数据的表结构。 + +Hive需要对输入进行全盘扫描来满足查询条件，通过使用分区可以优化查询。对于用户标签这种日加工数据，随着时间的推移，分区数量的变动也是均匀的。 + +每日全量数据，即该表的日期分区中记录着截止到当天的全量用户数据。例如，“select count(*) from userprofile where data='20180701'”这条语句查询的是userprofile表截止到2018年7月1日的全量用户数据。日全量数据的优势是方便查询，缺点是不便于探查更细粒度的用户行为。 + +每日增量数据，即该表的日期分区中记录着当日的用户行为数据。例如，同样是“select count(*) from userprofile where data='20180701'”，这条语句查询的是userprofile表在2018年7月1日记录的当日用户行为数据。日增量数据可视为ODS层的用户行为画像，在应用时还需要基于该增量数据做进一步的建模加工。 + +下面详细介绍这两种表结构的设计方法。 + +**① 日全量数据** + +日全量数据表中，在每天对应的日期分区中插入截止到当天为止的全量数据，用户进行查询时，只需查询最近一天的数据即可获得最新全量数据。下面以一个具体的日全量表结构的例子来进行说明。 + +![](https://mmbiz.qpic.cn/mmbiz_png/zHbzQPKIBPh4WJCtSFtGYXlpCqDmKJVpiaxzH9qcfqQYUk3WEM1VBTUWEIk1QejuZx9hY4fU44mDQJAJyxnYMYg/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +这里userid表示用户id，labelweight表示标签权重，theme表示标签归属的二级主题，labelid表示一个标签id。通过“日期 +标签归属的二级主题+标签id”的方式进行分区，设置三个分区字段更便于开发和查询数据。该表结构下的标签权重仅考虑统计类型标签的权重，如：历史购买金额标签对应的权重为金额数量，用户近30日访问天数为对应的天数，该权重值的计算未考虑较为复杂的用户行为次数、行为类型、行为距今时间等复杂情况。 + +通过表名末尾追加“_all”的规范化命名形式，可直观看出这是一张日全量表。 + +例如，对于主题类型为“会员”的标签，插入“20190101”日的全量数据，可通过语句： + +insert overwrite table dw. userprofile_userlabel_all partition(data_date= '20190101', theme= 'member', labelid='ATTRITUBE_U_05_001')来实现。 + +查询截止到“20190101”日的被打上会员标签的用户量，可通过语句： + +select count(distinct userid) from dw.userprofile_userlabel_all where data_date='20190101'来实现。 + +**② 日增量数据** + +日增量数据表，即在每天的日期分区中插入当天业务运行产生的数据，用户进行查询时通过限制查询的日期范围，就可以找出在特定时间范围内被打上特定标签的用户。下面以一个具体的日增量表结构的例子来说明。 + +![img](https://mmbiz.qpic.cn/mmbiz_png/zHbzQPKIBPh4WJCtSFtGYXlpCqDmKJVpXZ9SVNMN9zkrsJ2kxAibPjgD1x0uohVp2KM4fIcI2LZxVcq5gSSBZ0Q/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +这里，labelid表示标签名称；cookieid表示用户id；act_cnt表示用户当日行为次数，如用户当日浏览某三级品类商品3次，则打上次数为3；tag_type_id为标签类型,如母婴、3C、数码等不同类型；act_type_id表示行为类型，如浏览、搜索、收藏、下单等行为。分区方式为按日期分区，插入当日数据。 + +通过表名末尾追加“_append”的规范化命名形式，可直观看出这是一张日增量表。 + +例如，某用户在“20180701”日浏览某3C电子商品4次（act_cnt），即给该用户（userid）打上商品对应的三级品类标签（tagid），标签类型（tag_type_id）为3C电子商品，行为类型（act_type_id）为浏览。这里可以通过对标签类型和行为类型两个字段配置维度表的方式，对数据进行管理。例如对于行为类型（act_type_id）字段，可以设定1为购买行为、2为浏览行为、3为收藏行为等，在行为标签表中以数值定义用户行为类型，在维度表中维护每个数值对应的具体含义。 + +该日增量数据表可视为ODS层用户行为标签明细。在查询过程中，例如对于某用户id为001的用户，查询其在“20180701”日到“20180707”日被打上的标签，可通过命令： + +select * from dw.userprofile_act_feature_append where userid = '001' and data_date>='20180701' and data_date<= '20180707'查询。 + +该日增量的表结构记录了用户每天的行为带来的标签，但未计算打在用户身上标签的权重，计算权重时还需做进一步建模加工。标签权重算法详见4.6节的内容。 + +**③ 关于宽表设计** + +用户画像表结构如何设计，没有一定要遵循的固定的格式，符合业务需要、能满足应用即可。下面通过两个宽表设计的案例，提供另一种解决方案的思路。 + +用户属性宽表设计（见表1-10），主要记录用户基本属性信息。 + +![表1-10　用户属性宽表设计](https://mmbiz.qpic.cn/mmbiz_png/zHbzQPKIBPh4WJCtSFtGYXlpCqDmKJVpkNDibfGT0pSyqjNbfVwM2PSeKAY4n26sFypntGAo5l85yaGUzazwOBQ/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +![img](https://mmbiz.qpic.cn/mmbiz_png/zHbzQPKIBPh4WJCtSFtGYXlpCqDmKJVpwh1DVqY6Fs99cS0vrBqLlYfmSD8cPuforHIhM9ghBFTibLgAAibghWJw/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + + + +用户日活跃宽表设计（见表1-11），主要记录用户每天访问的信息。 + +![表1-11　用户日活跃宽表设计](https://mmbiz.qpic.cn/mmbiz_png/zHbzQPKIBPh4WJCtSFtGYXlpCqDmKJVpqI1iaop228U8kFh3jH8135hetKYbCY43TUxW0xyMatXoESMIKNLx3zQ/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + + + +## **定性类画像** + +本书重点讲解如何运用大数据定量刻画用户画像，然而对于用户的刻画除了定量维度外，定性刻画也是常见手段。定性类画像多见于用户研究等运营类岗位，通过电话调研、网络调研问卷、当面深入访谈、网上第三方权威数据等方式收集用户信息，帮助其理解用户。这种定性类调研相比大数据定量刻画用户来说，可以更精确地了解用户需求和行为特征，但这个样本量是有限的，得出的结论也不一定能代表大部分用户的观点。 + +通过制定调研问卷表，我们可以收集用户基本信息以及设置一个或多个场景，专访用户或网络回收调研问卷，在分析问卷数据后获取用户的画像特征。目前市场上“问卷星”等第三方问卷调查平台可提供用户问卷设计、链接发放、采集数据和信息、调研结果分析等一系列功能，如图1-7所示。 + +![图1-7　某调研问卷示例（截图自“问卷星”）](https://mmbiz.qpic.cn/mmbiz_png/zHbzQPKIBPh4WJCtSFtGYXlpCqDmKJVpGLgsz12LW3fsKuAavTmUTNNZNzm4AEJ2jb6ehTN6dnfVLIT29MeIjA/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + + + +根据回收的调研问卷，可结合统计数据进一步分析用户画像特征（如图1-8所示）。 + +![图1-8　回收的调研问卷（截图自“问卷星”）](https://mmbiz.qpic.cn/mmbiz_png/zHbzQPKIBPh4WJCtSFtGYXlpCqDmKJVpQtmS6NNeV81wP7uuzjuRcJzznrCPPq5Inic0UvBwbL6aCLyT295YlJA/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + + + +## **小结** + +本文主要介绍了用户画像的一些基础知识，包括画像的简介、标签类型、整个画像系统的数据架构，开发画像系统主要覆盖的8个模块，以及开发过程中的各阶段关键产出。初步介绍了画像系统的轮廓概貌，帮助读者对于如何设计画像系统、开发周期、画像的应用方式等有宏观的初步的了解。 + +——本文摘自机械工业出版社华章图书 + +《用户画像方法论与工程化解决方案》 + +**作者介绍：** + +赵宏田，资深大数据技术专家。擅长Hadoop、Spark等大数据技术，以及业务数据分析、数据仓库开发、爬虫、用户画像系统搭建等。著有畅销书《数据化运营：系统方法与实践案例》 《用户画像：方法论与工程化解决方案》。 \ No newline at end of file diff --git a/docs/data-store/Elasticsearch/ES-FAQ.md b/docs/data-management/Elasticsearch/ES-FAQ.md similarity index 100% rename from docs/data-store/Elasticsearch/ES-FAQ.md rename to docs/data-management/Elasticsearch/ES-FAQ.md diff --git a/docs/data-management/MySQL/.DS_Store b/docs/data-management/MySQL/.DS_Store new file mode 100644 index 0000000000..62c3b6dcb5 Binary files /dev/null and b/docs/data-management/MySQL/.DS_Store differ diff --git a/docs/data-management/MySQL/MySQL-Framework.md b/docs/data-management/MySQL/MySQL-Framework.md new file mode 100644 index 0000000000..f616a373f6 --- /dev/null +++ b/docs/data-management/MySQL/MySQL-Framework.md @@ -0,0 +1,179 @@ +--- +title: MySQL架构介绍 +date: 2022-08-25 +tags: + - MySQL +categories: MySQL +--- + +![](https://img.starfish.ink/mysql/banner-mysql-architecture.png) + +> Hello，我是海星。 +> +> 学习 MySQL 第一步，不是去学 select 、update，而是先要对他的整体架构设计有个大概的了解，先高屋建瓴，然后逐一攻破。 + +和其它数据库相比，MySQL 有点与众不同，它的架构可以在多种不同场景中应用并发挥良好作用。主要体现在存储引擎的架构上，**插件式的存储引擎架构将查询处理和其它的系统任务以及数据的存储提取相分离**。这种架构可以根据业务的需求和实际需要选择合适的存储引擎。 + +下边是 MySQL 官网中 8.0 版本的一个图，我们展开看一下，对 MySQL 整体架构和可插拔的存储引擎先有个总体回顾。 + +![](https://img.starfish.ink/mysql/architecture.png) + +## 1. 连接层 + +要使用 MySQL，第一步肯定要与他进行连接。 + +最上层就是一些客户端和连接服务，包含本地 socket 通信和大多数基于客户端/服务端工具实现的类似于 tcp/ip 的通信。**主要完成一些类似于建立连接、授权认证、及相关的安全方案**。 + +```mysql +# -h 指定 MySQL 服务得 IP 地址，如果是连接本地的 MySQL服务，可以不用这个参数； +# -u 指定用户名，管理员角色名为 root； +# -p 指定密码，如果命令行中不填写密码（为了密码安全，建议不要在命令行写密码），就需要在交互对话里面输入密码 +mysql -h$ip -u$user -p +``` + +输入密码后，就成功建立了连接，我们可以用 `show processlist` 查看当前所有数据库连接的 `session` 状态 + +> 连接状态，一般是`休眠`（sleep），`查询`（query），`连接`（connect），如果一条 SQL 语句是`query`状态，而且`time`时间很长，说明存在`问题` + +![](https://img.starfish.ink/mysql/show_processlist.png) + +其中的 Command 列显示为“Sleep”的这一行，就表示现在系统里面有一个空闲连接 + +客户端如果太长时间没动静，连接器就会自动将它断开。这个时间是由参数 `wait_timeout` 控制的，默认值是 8 小时。 + +```mysql +mysql> show variables like 'wait_timeout'; ++---------------+-------+ +| Variable_name | Value | ++---------------+-------+ +| wait_timeout | 28800 | ++---------------+-------+ +1 row in set (0.00 sec) +``` + +当然，MySQL 对连接数量也是有限制的，最大连接数由 `max_connections` 参数控制 + +```mysql +mysql> show variables like 'max_connections'; ++-----------------+-------+ +| Variable_name | Value | ++-----------------+-------+ +| max_connections | 151 | ++-----------------+-------+ +1 row in set (0.00 sec) +``` + + + +## 2. 服务层 + +第二层架构完成了大部分的核心功能， 包括查询解析、优化、缓存、以及所有的内置函数，所有跨存储引擎的功能也都在这一层实现，包括触发器、存储过程、视图等 + +### 查询缓存 + +第一步的连接建立后，我们就可以执行 select 语句了。执行逻辑就会来到第二步：查询缓存。 + +MySQL 拿到一个查询请求后，会先到查询缓存看看，之前是不是执行过这条语句。之前执行过的语句及其结果可能会以 key-value 对的形式，被直接缓存在内存中。key 是查询的语句，value 是查询的结果。如果你的查询能够直接在这个缓存中找到 key，那么这个 value 就会被直接返回给客户端。 + +> 对一个表的更新，就会把该表上的所有查询缓存清空，所以更新比较频繁的表，查询缓存的命中率就极低，所以不建议使用，官方已经在 8.0 版本移除该功能了。 +> +> 之前版本的 MySQL 也提供“按需使用”的方式。我们可以将参数 query_cache_type 设置成 DEMAND，这样对默认的 SQL 语句就都不使用查询缓存。 +> +> **Note** +> +> The query cache is deprecated as of MySQL 5.7.20, and is removed in MySQL 8.0. + +### 分析器 + +如果没有命中查询缓存，就要开始真正执行语句了。首先，MySQL 需要知道你要做什么，因此需要对 SQL 语句做解析。 + +解析器会做如下两件事情。 + +- 第一件事情，**词法分析**。MySQL 会根据你输入的字符串识别出关键字出来，构建出 SQL 语法树，这样方便后面模块获取 SQL 类型、表名、字段名、 where 条件等等。 + +- 第二件事情，**语法分析**。根据词法分析的结果，语法解析器会根据语法规则，判断你输入的这个 SQL 语句是否满足 MySQL 语法。 + +如果我们输入的 SQL 语句语法不对，就会在解析器这个阶段报错。比如，我下面这条查询语句，把 from 写成了 form，这时 MySQL 解析器就会给报错。 + +```mysql +mysql> select * form user; +ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'form user' at line 1 +``` + +### 优化器 + +经过了分析器，MySQL 就知道你要做什么了。在开始执行之前，还要先经过优化器的处理。比如重写查询、决定表的读取顺序，选择合适的索引等 + +> 比如你执行下面这样的语句，这个语句是执行两个表的join： +> +> ```mysql +> select * from t1 join t2 using(ID) where t1.c=10 and t2.d=20;` +> ``` +> +> - 既可以先从表t1里面取出c=10的记录的ID值，再根据ID值关联到表t2，再判断t2里面d的值是否等于20。 +> - 也可以先从表t2里面取出d=20的记录的ID值，再根据ID值关联到t1，再判断t1里面c的值是否等于10。 +> +> 这两种执行方法的逻辑结果是一样的，但是执行的效率会有不同，而优化器的作用就是决定选择使用哪一个方案。 + +### 执行器 + +* MySQL 通过分析器知道了你要做什么，通过优化器知道了该怎么做，于是就进入了执行器阶段，开始执行语句。 +* 开始执行的时候，要先判断一下你对这个表有没有执行查询的权限，如果没有，就会返回没有权限的错误 +* 如果有权限，就打开表继续执行。打开表的时候，执行器就会根据表的引擎定义，去使用这个引擎提供的接口。 + +>```mysql +> select * from T where ID=10; +>``` +> +>比如我们这个例子中的表T中，ID字段没有索引，那么执行器的执行流程是这样的： +> +>1. 调用 InnoDB 引擎接口取这个表的第一行，判断 ID 值是不是 10，如果不是则跳过，如果是则将这行存在结果集中； +>2. 调用引擎接口取“下一行”，重复相同的判断逻辑，直到取到这个表的最后一行。 +>3. 执行器将上述遍历过程中所有满足条件的行组成的记录集作为结果集返回给客户端。 + +* 至此，这个整个语句就执行完成了。一条查询语句的执行过程一般是经过连接器、分析器、优化器、执行器等功能模块，最后到达存储引擎。 + +> 对于有索引的表，执行的逻辑也差不多。第一次调用的是“取满足条件的第一行”这个接口，之后循环取“满足条件的下一行”这个接口，这些接口都是引擎中已经定义好的。 +> +> 你会在数据库的慢查询日志中看到一个 rows_examined 的字段，表示这个语句执行过程中扫描了多少行。这个值就是在执行器每次调用引擎获取数据行的时候累加的。 +> +> 在有些场景下，执行器调用一次，在引擎内部则扫描了多行，因此**引擎扫描行数跟 rows_examined 并不是完全相同的。** + +### SQL 接口 + +用于接收客户端发送的各种 SQL 命令，返回用户需要查询的结果，比如 DML、DDL、存储过程、视图、触发器这些 + + + +## 3. 引擎层 + +存储引擎层，存储引擎真正的负责了 MySQL 中数据的存储和提取，服务器通过 API 与存储引擎进行通信。 + +不同的存储引擎具有的功能不同，这样我们可以根据自己的实际需要进行选取。 + + + +## 4. 存储层 + +数据存储层，主要是将数据存储在运行于该设备的文件系统之上，并完成与存储引擎的交互。 + + + +### MySQL 的查询流程大致是？ + +> 一条 SQL 查询语句是如何执行的？ +1. **客户端请求**：MySQL 客户端通过协议与 MySQL 服务器建连接，发送查询语句，先检查查询缓存，如果命中，直接返回结果，否则进行语句解析（MySQL 8.0 已取消了缓存） +2. **查询接收**：连接器接收请求，管理连接 +3. **解析器**：对 SQL 进行词法分析和语法分析，转换为解析树 +4. **优化器**：优化器生成执行计划，选择最优索引和连接顺序 +5. **查询执行器**：执行器执行查询，通过存储引擎接口获取数据 +6. **存储引擎**：存储引擎检索数据，返回给执行器 +7. **返回结果**：结果通过连接器返回给客户端 + +![](https://img.starfish.ink/mysql/MySQL-select-flow.png) + + +## Reference + +- 《高性能 MySQL》 +- 《MySQL 实战 45 讲》 diff --git a/docs/data-management/MySQL/MySQL-Index.md b/docs/data-management/MySQL/MySQL-Index.md new file mode 100644 index 0000000000..fe6c8f80c3 --- /dev/null +++ b/docs/data-management/MySQL/MySQL-Index.md @@ -0,0 +1,634 @@ +--- +title: MySQL索引——妈妈再也不用担心我不会索引了 +date: 2022-02-12 +tags: + - MySQL +categories: MySQL +--- + +![](https://img.starfish.ink/mysql/banner-mysql-index.png) + +>Hello，我是海星。 +> +>先来一道经典的服务端面试题，看下你会吗 +> +>“如果有这样一个查询 `select * from table where a=1 group by b order by c;` 如果每个字段都有一个单列索引，索引会生效吗？如果是复合索引，能说下几种情况吗？“ +> +>这篇文章算是一个 MySQL 索引的知识梳理，包括索引的一些概念、B 树的结构、和索引的原理以及一些索引策略的知识，祝好 + + + +## 一、索引基础回顾 + +### 索引是什么 + +- MYSQL 官方对索引的定义为：索引（Index）是帮助 MySQL 高效获取数据的数据结构，所以说**索引的本质是：数据结构** + +- 索引的目的在于提高查询效率，可以类比字典、 火车站的车次表、图书的目录等 。 + +- 可以简单的理解为“排好序的快速查找数据结构”，数据本身之外，**数据库还维护着一个满足特定查找算法的数据结构**，这些数据结构以某种方式引用（指向）数据，这样就可以在这些数据结构上实现高级查找算法。这种数据结构，就是索引。 + + > 常见的索引模型其实有很多，哈希表、有序数组，各种搜索树都可以实现索引结构 + + 下图是一种可能的索引方式示例（二叉搜索树） + + ![](https://img.starfish.ink/mysql/search-index-demo.png) + + 上图左边是一张简单的`学生成绩表`，只有学号 id 和成绩 score 两列（最左边的是数据的物理地址） + + 比如我们想要快速查指定成绩的学生，通过构建一个右边的二叉搜索树当索引，索引节点就是成绩数据，节点指向对应数据记录物理地址的指针，这样就可以运用二叉查找在一定的复杂度内获取到对应的数据，从而快速检索出符合条件的学生信息。 + +- 索引本身也很大，不可能全部存储在内存中，**一般以索引文件的形式存储在磁盘上** + + + +### 优势 + +- 索引大大减少了服务器需要扫描的数据量（提高数据检索效率） +- 索引可以帮助服务器避免排序和临时表（降低数据排序的成本，降低 CPU 的消耗） +- 索引可以将随机 I/O 变为顺序 I/O（降低数据库 IO 成本） + + + +### 劣势 + +- 索引也是一张表，保存了主键和索引字段，并指向实体表的记录，所以也需要占用内存 +- 虽然索引大大提高了查询速度，同时却会降低更新表的速度，如对表进行 INSERT、UPDATE 和 DELETE。 + 因为更新表时，MySQL 不仅要保存数据，还要保存一下索引文件每次更新添加了索引列的字段，都会调整因为更新所带来的键值变化后的索引信息 + + + +### 索引分类 + +我们从 3 个角度看下索引的分类 + +**从逻辑角度** + +- 主键索引：主键索引是一种特殊的唯一索引，不允许有空值 +- 普通索引或者单列索引：每个索引只包含单个列，一个表可以有多个单列索引 +- 多列索引（复合索引、联合索引）：复合索引指多个字段上创建的索引，只有在查询条件中使用了创建索引时的第一个字段，索引才会被使用。 +- 唯一索引或者非唯一索引 +- Full-Text 全文索引：它查找的是文本中的关键词，而不是直接比较索引中的值 +- 空间索引：空间索引是对空间数据类型的字段建立的索引 + +**数据结构角度** + +- Hash 索引：主要就是通过 Hash 算法，将数据库字段数据转换成定长的 Hash 值，与这条数据的行指针一并存入 Hash 表的对应位置；如果发生 Hash 碰撞，则在对应 Hash 键下以链表形式存储。查询时，就再次对待查关键字再次执行相同的 Hash 算法，得到 Hash 值，到对应 Hash 表对应位置取出数据即可，Memory 引擎是支持非唯一哈希索引的，如果发生 Hash 碰撞，会以链表的方式存放多个记录在同一哈希条目中。使用 Hash 索引的数据库并不多， 目前有 Memory 引擎和 NDB 引擎支持 Hash 索引。 + + 缺点是，只支持等值比较查询，像 = 、 in() 这种，不支持范围查找，比如 where id > 10 这种，也不能排序。 + +- B+ 树索引（下文会详细讲） + +**从物理存储角度** + +- 聚集索引（clustered index） + +- 非聚集索引（non-clustered index），也叫辅助索引（secondary index） + + 聚集索引和非聚集索引都是 B+ 树结构 + + + +## 二、MySQL 索引结构 + +索引可以有很多种结构类型，这样可以为不同的场景提供更好的性能。 + +> **首先要明白索引（index）是在存储引擎（storage engine）层面实现的，而不是 server 层面**。不是所有的存储引擎都支持所有的索引类型。即使多个存储引擎支持某一索引类型，它们的实现和行为也可能有所差别。 +> +> 像有的 二* 面试官上来就会问：`MySQL 为什么不用 Hash 结构做索引? ` +> +> 我会直接来一句，不好意思，MySQL 也会用 Hash 做索引，Memory 存储引擎就支持 Hash 索引。只是场景用的少，Hash 结构更适用于只有等值查询的场景 +> +> 为什么不用二叉搜索树呢？ 这就很简单了，二叉树的叉叉上只有两个数，数据量太多的话，那得多少层呀。 + + + +### 磁盘 IO + +介绍索引结构之前，我们先了解下[磁盘IO与预读](https://tech.meituan.com/2014/06/30/mysql-index.html "MySQL索引原理及慢查询优化") + +> 磁盘读取数据靠的是机械运动，每次读取数据花费的时间可以分为寻道时间、旋转延迟、传输时间三个部分 +> +> - 寻道时间指的是磁臂移动到指定磁道所需要的时间，主流磁盘一般在 5ms 以下； +> - 旋转延迟就是我们经常听说的磁盘转速，比如一个磁盘 7200 转，表示每分钟能转 7200 次，也就是说 1 秒钟能转 120 次，旋转延迟就是 `1/120/2 = 4.17ms`； +> - 传输时间指的是从磁盘读出或将数据写入磁盘的时间，一般在零点几毫秒，相对于前两个时间可以忽略不计。 +> +> ![](https://img.starfish.ink/mysql/disk-io.png) +> +> 那么访问一次磁盘的时间，即一次磁盘 IO 的时间约等于 `5+4.17 = 9ms` 左右，听起来还挺不错的，但要知道一台 500 -MIPS 的机器每秒可以执行 5 亿条指令，因为指令依靠的是电的性质，换句话说执行一次 IO 的时间可以执行 40 万条指令，数据库动辄十万百万乃至千万级数据，每次 9 毫秒的时间，显然是个灾难。下图是计算机硬件延迟的对比图，供大家参考： +> +> ![](https://img.starfish.ink/mysql/7f46a0a4.png) +> +> 考虑到磁盘 IO 是非常高昂的操作，计算机操作系统做了一些优化，当一次 IO 时，不光把当前磁盘地址的数据，而是把相邻的数据也都读取到内存缓冲区内，因为局部预读性原理告诉我们，当计算机访问一个地址的数据的时候，与其相邻的数据也会很快被访问到。每一次 IO 读取的数据我们称之为**一页(page)**。具体一页有多大数据跟操作系统有关，一般为 4k 或 8k，也就是我们读取一页内的数据时候，实际上才发生了一次 IO，这个理论对于索引的数据结构设计非常有帮助。 +> + + + +那是不应该有一种数据结构，可以在每次查找数据时把磁盘 IO 次数控制在一个很小的数量级， B+ 树就这样应用而生。 + +### 心里有点 B 树 + +有一点面试经验的同学，可能都碰到过这么一道面试题：MySQL InnoDB 索引为什么用 B+ 树，不用 B 树 + +> B-Tree == B Tree，他两是一个东西，没有 B 减树 这玩意 + +先大概（仔细）看下维基百科的概述： + +> 在 B 树中，内部（非叶子）节点可以拥有可变数量的子节点（数量范围预先定义好）。当数据被插入或从一个节点中移除，它的子节点数量发生变化。为了维持在预先设定的数量范围内，内部节点可能会被合并或者分离。因为子节点数量有一定的允许范围，所以B 树不需要像其他自平衡查找树那样频繁地重新保持平衡，但是由于节点没有被完全填充，可能浪费了一些空间。子节点数量的上界和下界依特定的实现而设置。例如，在一个 2-3 B树（通常简称[2-3树](https://zh.wikipedia.org/wiki/2-3树)），每一个内部节点只能有 2 或 3 个子节点。 +> +> **B 树中每一个内部节点会包含一定数量的键，键将节点的子树分开**。例如，如果一个内部节点有 3 个子节点（子树），那么它就必须有两个键： *a*1 和 *a*2 。左边子树的所有值都必须小于 *a*1 ，中间子树的所有值都必须在 *a*1 和 *a*2 之间，右边子树的所有值都必须大于 *a*2 。 +> +> 在存取节点数据所耗时间远超过处理节点数据所耗时间的情况下，Ｂ树在可选的实现中拥有很多优势，因为存取节点的开销被分摊到里层节点的多次操作上。这通常出现在当节点存储在二级存储器如硬盘存储器上。通过最大化内部里层节点的子节点的数量，树的高度减小，存取节点的开销被缩减。另外，重新平衡树的动作也更少出现。子节点的最大数量取决于，每个子节点必需存储的信息量，和完整磁盘块的大小或者二次存储器中类似的容量。虽然 2-3 树更易于解释，实际运用中，Ｂ树使用[二级存储器](https://zh.wikipedia.org/w/index.php?title=二级存储器&action=edit&redlink=1)，需要大量数目的子节点来提升效率。 +> +> 而 B+ 树 又是 B 树的变种，B+ 树结构，所有的数据都存放在叶子节点上，且把叶子节点通过指针连接到一起，形成了一条数据链表，以加快相邻数据的检索效率。 + +推荐一个数据结构可视化网站：https://www.cs.usfca.edu/~galles/visualization/Algorithms.html，可以用来生成各种数据结构 + +将 `[11,13,15,16,20,23,25,30,23,27]` 用 B 树 和 B+ 树存储，看下结构 + +![](https://img.starfish.ink/mysql/BTree-vs-B%2BTree.png) + +#### **B 树和 B+ 树区别** + +B-Tree 和 B+Tree 都是为磁盘等外存储设备设计的一种平衡查找树。 + +| 关键词 | B-树 | B+树 | 备注 | +| -------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | ----------------------------------- | +| 最大分支，最小分支 | 每个结点最多有m个分支（子树），最少⌈m/2⌉（中间结点）个分支或者2个分支（是根节点非叶子结点）。 | 同左 | m阶对应的就是就是最大分支 | +| n个关键字与分支的关系 | 分支等于n+1 | 分支等于n | 无 | +| 关键字个数（B+树关键字个数要多） | 大于等于⌈m/2⌉-1小于等于m-1 | 大于等于⌈m/2⌉小于等于m | B+树关键字个数要多，+体现在的地方。 | +| 叶子结点相同点 | 每个节点中的元素互不相等且按照从小到大排列；所有的叶子结点都位于同一层。 | 同左 | 无 | +| 叶子结点不相同 | 不包含信息 | 叶子结点包含信息，指针指向记录。 | 无 | +| 叶子结点之间的关系 | 无 | B+树上有一个指针指向关键字最小的叶子结点，所有叶子节点之间链接成一个线性链表 | 无 | +| 非叶子结点 | 一个关键字对应一个记录的存储地址 | 只起到索引的作用 | 无 | +| 存储结构 | 相同 | 同左 | 无 | + + + +### 为什么要用 B+ 树 + +心里有了磁盘 IO 和 B 树的概念，接下来就顺理成章了。磁盘 IO 次数越少，那查询效率肯定就越高。而 IO 次数又取决于 B+ 树的高度 + +我们以 InnoDB 存储引擎来说明。 + +系统从磁盘读取数据到内存时是以磁盘块（block）为基本单位的，位于同一个磁盘块中的数据会被一次性读取出来，而不是需要什么取什么。 + +InnoDB 存储引擎中有页（Page）的概念，页是其磁盘管理的最小单位。InnoDB 存储引擎中默认每个页的大小为16KB，可通过参数 `innodb_page_size` 将页的大小设置为 4K、8K、16K，在 MySQL 中可通过如下命令查看页的大小：`show variables like 'innodb_page_size';` + +而系统一个磁盘块的存储空间往往没有这么大，因此 InnoDB 每次申请磁盘空间时都会是若干地址连续磁盘块来达到页的大小 16KB。InnoDB 在把磁盘数据读入到磁盘时会以页为基本单位，在查询数据时如果一个页中的每条数据都能有助于定位数据记录的位置，这将会减少磁盘 I/O 次数，提高查询效率。 + +#### 举个例子 + +索引是为了更快的查询到数据，MySQL 数据行可能会很多内容 + +以范围查找为例简单看下，B Tree 结构查询 [10-25] 的数据（从根节点开始，随机查找一样的道理，只是我画的图只有 2 层，说服力强的不是那么明显罢了） + +1. 加载根节点，第一个节点元素15，大于10【磁盘 I/O 操作第 1 次】 +2. 通过根节点的左子节点地址加载，找到 11，13【磁盘 I/O 操作第 2 次】 +3. 重新加载根节点，找到中间节点数据 16，20【磁盘 I/O 操作第 3 次】 +4. 再次加载根节点，23 小于 25，再加载右子节点，找到 25，结束【磁盘 I/O 操作第 4 次】 + +![](https://img.starfish.ink/mysql/MySQL-B%2BTree-store.png) + +而 B+ 树对范围查找就简单了，数据都在最下边的叶子节点下，而且链起来了，我只需找到第一个然后遍历就行（暂且不考虑页分裂等其他问题）。 + + + +#### 解答 + +> 为什么 MySQL 索引要用 B+ 树不是 B 树？ + +B+Tree 是在 B-Tree 基础上的一种优化，使其更适合实现外存储索引结构。 + +用 B+ 树不用 B 树考虑的是 IO 对性能的影响，B 树的每个节点都存储数据，而 B+ 树只有叶子节点才存储数据，所以查找相同数据量的情况下，B 树的高度更高，IO 更频繁。数据库索引是存储在磁盘上的，当数据量大时，就不能把整个索引全部加载到内存了，只能逐一加载每一个磁盘页（对应索引树的节点）。其中在 MySQL 底层对 B+ 树进行进一步优化：**在叶子节点中是双向链表，且在链表的头结点和尾节点也是循环指向的**。 + +B-Tree 结构图每个节点中不仅要包含数据的 key 值，还有 data 值。而每一个页的存储空间是有限的，如果 data 数据较大时将会导致每个节点（即一个页）能存储的 key 的数量很小，当存储的数据量很大时同样会导致 B-Tree 的深度较大，增大查询时的磁盘 I/O 次数，进而影响查询效率。在 B+Tree 中，**所有数据记录节点都是按照键值大小顺序存放在同一层的叶子节点上**，而非叶子节点上只存储 key 值信息，这样可以大大加大每个节点存储的 key 值数量，降低 B+Tree 的高度。 + +> IO 次数取决于 B+ 数的高度 h，假设当前数据表的数据为 N，每个磁盘块的数据项的数量是 m，则有 `h=㏒(m+1)N`，当数据量 N 一定的情况下，m 越大，h 越小；而 `m = 磁盘块的大小 / 数据项的大小`，磁盘块的大小也就是一个数据页的大小，是固定的，如果数据项占的空间越小，数据项的数量越多，树的高度越低。这就是为什么每个数据项，即索引字段要尽量的小，比如 int 占 4 字节，要比 bigint 8 字节少一半。这也是为什么 B+ 树要求把真实的数据放到叶子节点而不是内层节点，一旦放到内层节点，磁盘块的数据项会大幅度下降，导致树增高。当数据项等于 1 时将会退化成线性表。 + + + +## 三、MyISAM 和 InnoDB 索引原理 + +### MyISAM 主键索引与辅助索引的结构 + +MyISAM 引擎的索引文件和数据文件是分离的。**MyISAM 引擎索引结构的叶子节点的数据域，存放的并不是实际的数据记录，而是数据记录的地址**。索引文件与数据文件分离，这样的索引称为"**非聚簇索引**"。MyISAM 的主索引与辅助索引区别并不大，主键索引就是一个名为 PRIMARY 的唯一非空索引。 + +> 术语 “聚簇” 表示数据行和相邻的键值紧凑的存储在一起 + +![](https://img.starfish.ink/mysql/MySQL-MyISAM-Index.png) + +在 MyISAM 中，索引（含叶子节点）存放在单独的 `.myi` 文件中，叶子节点存放的是数据的物理地址偏移量（通过偏移量访问就是随机访问，速度很快）。 + +主索引是指主键索引，键值不可能重复；辅助索引则是普通索引，键值可能重复。 + +通过索引查找数据的流程：先从索引文件中查找到索引节点，从中拿到数据的文件指针，再到数据文件中通过文件指针定位了具体的数据。 + +辅助索引类似。 + + + +### InnoDB 主键索引与辅助索引的结构 + +**InnoDB 引擎索引结构的叶子节点的数据域，存放的就是实际的数据记录**（对于主索引，此处会存放表中所有的数据记录；对于辅助索引此处会引用主键，检索的时候通过主键到主键索引中找到对应数据行），或者说，**InnoDB 的数据文件本身就是主键索引文件**，这样的索引被称为"“**聚簇索引**”，一个表只能有一个聚簇索引。 + +#### 主键索引： + +我们知道 InnoDB 索引是聚集索引，它的索引和数据是存入同一个 `.idb` 文件中的，因此它的索引结构是在同一个树节点中同时存放索引和数据，如下图中最底层的叶子节点有三行数据，对应于数据表中的 id、name、score 数据项。 + +![](https://img.starfish.ink/mysql/MySQL-InnoDB-Index-primary.png) + +在 Innodb 中，索引分叶子节点和非叶子节点，非叶子节点就像新华字典的目录，单独存放在索引段中，叶子节点则是顺序排列的，在数据段中。 + +InnoDB 的数据文件可以按照表来切分（只需要开启`innodb_file_per_table)`，切分后存放在`xxx.ibd`中，不切分存放在 `xxx.ibdata`中。 + +从 MySQL 5.6.6 版本开始，它的默认值就是 ON 了。 + +> 扩展点：建议将这个值设置为 ON。因为，一个表单独存储为一个文件更容易管理，而且在你不需要这个表的时候，通过 drop table 命令，系统就会直接删除这个文件。而如果是放在共享表空间中，即使表删掉了，空间也是不会回收的。 +> +> 所以会碰到这种情况，数据库占用空间太大后，把一个最大的表删掉了一半的数据，表文件的大小还是没变~ +> +> 在 MySQL 8.0 版本以前，表结构是存在以.frm 为后缀的文件里。而 MySQL 8.0 版本，则已经允许把表结构定义放在系统数据表中了。 + +#### 辅助（非主键）索引： + +这次我们以示例中学生表中的 name 列建立辅助索引，它的索引结构跟主键索引的结构有很大差别，在最底层的叶子结点有两行数据，第一行的字符串是辅助索引，按照 ASCII 码进行排序，第二行的整数是主键的值。 + +这就意味着，对 name 列进行条件搜索，需要两个步骤： + +1. 在辅助索引上检索 name，到达其叶子节点获取对应的主键； +2. 使用主键在主索引上再进行对应的检索操作 + +这也就是所谓的“**回表查询**” + +![](https://img.starfish.ink/mysql/MySQL-InnoDB-Index.png) + +**InnoDB 索引结构需要注意的点** + +1. 数据文件本身就是索引文件 +2. 表数据文件本身就是按 B+Tree 组织的一个索引结构文件 +3. 聚集索引中叶节点包含了完整的数据记录 +4. InnoDB 表必须要有主键，并且推荐使用整型自增主键 + +正如我们上面介绍 InnoDB 存储结构，索引与数据是共同存储的，不管是主键索引还是辅助索引，在查找时都是通过先查找到索引节点才能拿到相对应的数据，如果我们在设计表结构时没有显式指定索引列的话，MySQL 会从表中选择数据不重复的列建立索引，如果没有符合的列，则 MySQL 自动为 InnoDB 表生成一个隐含字段作为主键，并且这个字段长度为 6 个字节，类型为整型。 + +> 你可能在一些建表规范里面见到过类似的描述，要求建表语句里一定要有自增主键。当然事无绝对，我们来分析一下哪些场景下应该使用自增主键，而哪些场景下不应该。 +> +> 自增主键的插入数据模式，正符合了递增插入的场景。每次插入一条新记录，都是追加操作，都不涉及到挪动其他记录，也不会触发叶子节点的分裂。 +> +> 而有业务逻辑的字段做主键，则往往不容易保证有序插入，这样写数据成本相对较高。 +> +> 除了考虑性能外，我们还可以从存储空间的角度来看。假设你的表中确实有一个唯一字段，比如字符串类型的身份证号，那应该用身份证号做主键，还是用自增字段做主键呢？ +> +> 由于每个非主键索引的叶子节点上都是主键的值。如果用身份证号做主键，那么每个二级索引的叶子节点占用约 20 个字节，而如果用整型做主键，则只要 4 个字节，如果是长整型（bigint）则是 8 个字节。 +> +> **显然，主键长度越小，普通索引的叶子节点就越小，普通索引占用的空间也就越小。** +> +> 所以，从性能和存储空间方面考量，自增主键往往是更合理的选择。 +> +> 有没有什么场景适合用业务字段直接做主键的呢？还是有的。比如，有些业务的场景需求是这样的： +> +> 1. 只有一个索引； +> 2. 该索引必须是唯一索引。 +> +> 你一定看出来了，这就是典型的 KV 场景。 +> +> 由于没有其他索引，所以也就不用考虑其他索引的叶子节点大小的问题。 +> +> 这时候我们就要优先考虑上一段提到的“尽量使用主键查询”原则，直接将这个索引设置为主键，可以避免每次查询需要搜索两棵树。 + +![](https://img.starfish.ink/mysql/MySQL-secondary-index.png) + +## 四、索引策略 + +### 哪些情况需要创建索引 + +1. 主键自动建立唯一索引 + +2. 频繁作为查询条件的字段 + +3. 查询中与其他表关联的字段，外键关系建立索引 + +4. 单键/组合索引的选择问题，who? 高并发下倾向创建组合索引 + +5. 查询中排序的字段，排序字段通过索引访问大幅提高排序速度 + +6. 查询中统计或分组字段 + + + +### 哪些情况不要创建索引 + +1. 表记录太少 +2. 经常增删改的表 +3. 数据重复且分布均匀的表字段，只应该为最经常查询和最经常排序的数据列建立索引（如果某个数据类包含太多的重复数据，建立索引没有太大意义） +4. 频繁更新的字段不适合创建索引（会加重IO负担） +5. where 条件里用不到的字段不创建索引 + + + +### [高效索引](高性能的索引策略 "《高性能MySQL》") + +> 整理自《高性能 MySQL》 + +#### 独立的列 + +**如果查询中的列不是独立的，MySQL 就不会使用索引**。“独立的列”是指索引不能是表达式的一部分，也不能是函数的参数。 + +比如： + +```mysql +EXPLAIN SELECT * FROM mydb.sys_user where user_id = 2; +``` + +在 sys_user 表中，user_id 是主键，有主键索引，索引 explain 出来结果就是： + +![](https://img.starfish.ink/mysql/explain-1.png) + +可见这次查询使用了PRIMARY KEY来优化查询，如果变成这样： + +```mysql +EXPLAIN SELECT * FROM mydb.sys_user where user_id + 1 = 2; +``` + +结果就是： + +![](https://img.starfish.ink/mysql/explain-2.png) + + + +#### 前缀索引 + +前缀索引其实就是对文本的前几个字符（具体是几个字符在建立索引时指定）建立索引，这样建立起来的索引占用空间更小，所以查询更快。 + +```mysql +ALTER TABLE table_name ADD KEY(column_name(prefix_length)); +ALTER TABLE table_name ADD index index_name(column_name(prefix_length)); +``` + +对于内容很长的列，比如 blob, text 或者很长的 varchar 列，必须使用前缀索引，MySQL 不允许索引这些列的完整长度。 + +所以问题就在于要选择合适长度的前缀，即 prefix_length。前缀太短，选择性太低，前缀太长，索引占用空间太大。 + +![](https://img.starfish.ink/mysql/pre-index.png) + +比如上图中，两个不同的索引同样执行下面的语句 + +```mysql +select id,name,email from user where emai='zhangsan@qq.com' +``` + +执行效果会有很大的差别，普通索引 `idx_email ` 找到满足条件的记录后，再返回主键索引取出数据即可，而前缀索引会多次查到 `zhangs`，然后返回主键索引取出数据进行对比，会扫描多次数据行。 + +如果前缀索引取前 7 个字节构建的话 `idx_pre_email(7)`，就只需要扫描一行。 + +所以使用前缀索引，定义好长度，就可以做到既节省空间，又不用额外增加太多的查询成本。 + +为了决定前缀的合适长度，需要找到最常见的值的列表，然后和最常见的前缀列进行比较。 + +前缀索引是一种能使索引更小、更快的有效办法，但另一方面也有缺点：**MySQL 无法使用前缀索引做 ORDER BY 和 GROUP BY，也无法使用前缀索引做『覆盖索引』**。 + +> 一个常见的场景是针对很长的十六进制唯一 ID 使用前缀索引。 +> +> 身份证号这样的数据如何索引？ +> +> - **使用倒序存储**：如果你存储身份证号的时候把它倒过来存，每次查询的时候，你可以这么写 +> +> ```mysql +> select field_list from t where id_card = reverse('input_id_card_string'); +> ``` +> +> - **使用 hash 字段。**你可以在表上再创建一个整数字段，来保存身份证的校验码，同时在这个字段上创建索引。 +> +> ```mysql +> alter table t add id_card_crc int unsigned, add index(id_card_crc); +> --查询 +> select field_list from t where id_card_crc=crc32('input_id_card_string') and id_card='input_id_card_string' +> ``` + + + +#### 覆盖索引 + +**覆盖索引**（Covering Index），或者叫索引覆盖， 也就是平时所说的**不需要回表操作** + +- 就是 select 的数据列只用从索引中就能够取得，不必读取数据行，MySQL 可以利用索引返回 select 列表中的字段，而不必根据索引再次读取数据文件，换句话说**查询列要被所建的索引覆盖**。 + +- 索引是高效找到行的一个方法，但是一般数据库也能使用索引找到一个列的数据，因此它不必读取整个行。毕竟索引叶子节点存储了它们索引的数据，当能通过读取索引就可以得到想要的数据，那就不需要读取行了。一个索引包含（覆盖）满足查询结果的数据就叫做覆盖索引。 + +- **判断标准** + + 使用 explain，可以通过输出的 extra 列来判断，对于一个索引覆盖查询，显示为 **using index**，MySQL 查询优化器在执行查询前会决定是否有索引覆盖查询 + + + +#### 多列索引（复合索引、联合索引） + +组合索引(concatenated index)：由多个列构成的索引，如 `create index idx_emp on emp(col1, col2, col3, ……)`，则我们称 idx_emp 索引为组合索引。 + + + +**在多个列上建立独立的单列索引大部分情况下并不能提高 MySQL 的查询性能**。对于下面的查询 where 条件，这两个单列索引都是不好的选择： + +```mysql +SELECT user_id,user_name FROM mydb.sys_user where user_id = 1 or user_name = 'zhang3'; +``` + +MySQL 5.0 版本之前，MySQL 会对这个查询使用全表扫描，除非改写成两个查询 UNION 的方式。 + +MySQL 5.0 和后续版本引入了一种叫做“**索引合并**”的策略，查询能够同时使用这两个单列索引进行扫描，并将结果合并。这种算法有三个变种：OR 条件的联合（union），AND 条件的相交（intersection），组合前两种情况的联合及相交。索引合并策略有时候是一种优化的结果，但实际上更多时候说明了表上的索引建得很糟糕： + +1. 当出现服务器对多个索引做相交操作时（多个AND条件），通常意味着需要一个包含所有相关列的多列索引，而不是多个独立的单列索引。 + +2. 当出现服务器对多个索引做联合操作时（多个OR条件），通常需要耗费大量的 CPU 和内存资源在算法的缓存、排序和合并操作上。特别是当其中有些索引的选择性不高，需要合并扫描返回的大量数据的时候。 + +3. 如果在 explain 中看到有索引合并，应该好好检查一下查询和表的结构，看是不是已经是最优的。 + + + +##### 最左前缀原则 + +在组合索引中有一个重要的概念：引导列(leading column)，在上面的例子中，col1 列为引导列。当我们进行查询时可以使用 ”where col1 = ? ”，也可以使用 ”where col1 = ? and col2 = ?”，这样的限制条件都会使用索引，但是”where col2 = ? ”查询就不会使用该索引。**所以限制条件中包含先导列时，该限制条件才会使用该组合索引。** + +> 举个栗子： +> +> 当 B+ 树的数据项是复合的数据结构，比如(name,age,sex)的时候，B+ 树是按照从左到右的顺序来建立搜索树的，比如当(张三,20,F)这样的数据来检索的时候，B+ 树会优先比较 name 来确定下一步的所搜方向，如果 name 相同再依次比较 age 和 sex，最后得到检索的数据；但当 (20,F) 这样的没有 name 的数据来的时候，B+ 树就不知道下一步该查哪个节点，因为建立搜索树的时候 name 就是第一个比较因子，必须要先根据 name 来搜索才能知道下一步去哪里查询。比如当 (张三,F) 这样的数据来检索时，B+ 树可以用 name 来指定搜索方向，但下一个字段 age 的缺失，所以只能把名字等于张三的数据都找到，然后再匹配性别是 F 的数据了， 这个是非常重要的性质，即**索引的最左匹配特性**。 + +![](https://img.starfish.ink/mysql/composite-index.png) + +可以看到，索引项是按照索引定义里面出现的字段顺序排序的。 + +当你的逻辑需求是查到所有名字是“Bob”的人时，可以快速定位到 ID = 2，然后向后遍历得到所有需要的结果。 + +如果你要查的是所有名字第一个字母是“B”的人，你的 SQL 语句的条件是"where name like ‘B %’"。这时，你也能够用上这个索引，查找到第一个符合条件的记录是 ID=2，然后向后遍历，直到不满足条件为止。 + +可以看到，不只是索引的全部定义，只要满足最左前缀，就可以利用索引来加速检索。这个最左前缀可以是联合索引的最左 N 个字段，也可以是字符串索引的最左 M 个字符。 + +那么就会出现一个问题：**在建立联合索引的时候，如何安排索引内的字段顺序。** + +这里我们的评估标准是，索引的复用能力。因为可以支持最左前缀，所以当已经有了 (a,b) 这个联合索引后，一般就不需要单独在 a 上建立索引了。因此，**第一原则是，如果通过调整顺序，可以少维护一个索引，那么这个顺序往往就是需要优先考虑采用的。** + + + +##### 索引下推 + +上一段我们说到满足最左前缀原则的时候，最左前缀可以用于在索引中定位记录。这时，你可能要问，那些不符合最左前缀的部分，会怎么样呢？ + +我们还是以联合索引（name,age,sex）为例。如果现在有一个需求：检索出表中“名字第一个字是 B，而且年龄是 19 岁的所有男孩”。那么，SQL 语句是这么写的： + +```mysql +mysql> select * from tuser where name like 'B %' and age=19 and sex=F; +``` + +你已经知道了前缀索引规则，所以这个语句在搜索索引树的时候，只能用 “B”，找到第一个满足条件的记录 ID = 2。当然，这还不错，总比全表扫描要好。(组合索引满足最左匹配，但是遇到非等值判断时匹配停止) + +然后呢？ + +当然是判断其他条件是否满足。 + +在 MySQL 5.6 之前，只能从 ID = 2 开始一个个回表。到主键索引上找出数据行，再对比字段值。 + +而 MySQL 5.6 引入的**索引下推优化**（index condition pushdown)， 可以在索引遍历过程中，对索引中包含的字段先做判断，直接过滤掉不满足条件的记录，减少回表次数。 + +![](https://img.starfish.ink/mysql/index-ICP.png) + +> 索引下推在**非主键索引**上的优化，可以有效减少回表的次数，大大提升了查询的效率 + + + +#### 使用索引扫描来做排序 + +MySQL 有两种方式可以生成有序的结果，通过排序操作或者按照索引顺序扫描，如果 explain 的 type 列的值为 index，则说明 MySQL 使用了索引扫描来做排序（不要和 extra 列的 Using index 搞混了，那个是使用了覆盖索引查询）。 + +扫描索引本身是很快的，因为只需要从一条索引记录移动到紧接着的下一条记录，但如果索引不能覆盖查询所需的全部列，那就不得不每扫描一条索引记录就回表查询一次对应的整行，这基本上都是随机 I/O，因此按索引顺序读取数据的速度通常要比顺序地全表扫描慢，尤其是在 I/O 密集型的工作负载时。 + +**MySQL 可以使用同一个索引既满足排序，又用于查找行，因此，如果可能，设计索引时应该尽可能地同时满足这两种任务，这样是最好的**。 + +**只有当索引的列顺序和 order by 子句的顺序完全一致，并且所有列的排序方向（倒序或升序，创建索引时可以指定 ASC 或 DESC）都一样时，MySQL 才能使用索引来对结果做排序**，如果查询需要关联多张表，则只有当 order by 子句引用的字段全部为第一个表时，才能使用索引做排序，order by 子句和查找型查询的限制是一样的，需要满足索引的最左前缀的要求，否则 MySQL 都需要执行排序操作，而无法使用索引排序。 + + + +#### 压缩（前缀压缩）索引 + +MyISAM 使用前缀压缩来减少索引的大小，从而让更多的索引可以放入内存中，这在某些情况下能极大地提高性能。 + +默认只压缩字符串，但通过参数设置也可以对整数做压缩。 + +MyISAM 压缩每个索引块的方法是，先完全保存索引块中的第一个值，然后将其他值和第一个值进行比较得到相同前缀的字节数和剩余的不同后缀部分，把这部分存储起来即可。 + +例如，索引块中的第一个值是“perform“，第二个值是”performance“，那么第二个值的前缀压缩后存储的是类似”7,ance“这样的形式。MyISAM 对**行指针**也采用类似的前缀压缩方式。 + +压缩块使用更少的空间，代价是某些操作可能更慢。因为每个值的压缩前缀都依赖前面的值，所以 MyISAM 查找时无法在索引块使用二分查找而只能从头开始扫描。正序的扫描速度还不错，但是如果是倒序扫描——例如 ORDER BY DESC——就不是很好了。所有在块中查找某一行的操作平均都需要扫描半个索引块。 + +测试表明，对于 CPU 密集型应用，因为扫描需要随机查找，压缩索引使得 MyISAM 在索引查找上要慢好几倍。压缩索引的倒序扫描就更慢了。压缩索引需要在 CPU 内存资源与磁盘之间做权衡。压缩索引可能只需要十分之一大小的磁盘空间，如果是 I/O 密集型应用，对某些查询带来的好处会比成本多很多。 + +可以在 CREATE TABLE 语句中指定 PACK_KEYS 参数来控制索引压缩的方式。 + + + +#### 重复索引和冗余索引 + +MySQL 允许在相同列上创建多个索引，无论是有意的还是无意的。有意的用途没想明白~ + +**重复索引是指在相同的列上按照相同的顺序创建的相同类型的索引**。应该避免这样创建重复索引，发现以后也应该立即移除。 + +冗余索引和重复索引有一些不同。如果创建了索引(A,B)，再创建索引(A)就是冗余索引，因为这只是前一个索引的前缀索引。因此索引(A,B)也可以当做索引(A)来使用（这种冗余只是对 B-Tree 索引来说的）。但是如果再创建索引(B,A)，则不是冗余索引，索引(B)也不是，因为B不是索引(A,B)的最左前缀。另外，其他不同类型的索引（例如哈希索引或者全文索引）也不会是 B-Tree 索引的冗余索引，而无论覆盖的索引列是什么。 + + + +#### 未使用的索引 + +除了冗余索引和重复索引，可能还会有一些服务器永远不使用的索引，这样的索引完全是累赘，建议考虑删除，有两个工具可以帮助定位未使用的索引： + +1. 在 percona server 或者 mariadb 中先打开 userstat=ON 服务器变量，默认是关闭的，然后让服务器运行一段时间，再通过查询` information_schema.index_statistics` 就能查到每个索引的使用频率。 + +2. 使用 percona toolkit 中的 pt-index-usage 工具，该工具可以读取查询日志，并对日志中的每个查询进行explain 操作，然后打印出关于索引和查询的报告，这个工具不仅可以找出哪些索引是未使用的，还可以了解查询的执行计划。 + + + +## 五、索引优化 + +### 导致 SQL 执行慢的原因 + +1. 硬件问题。如网络速度慢，内存不足，I/O 吞吐量小，磁盘空间满了等 + +2. 没有索引或者索引失效 + +3. 数据过多（分库分表） + +4. 服务器调优及各个参数设置（调整my.cnf） + + + +### 索引优化 + +```mysql +CREATE TABLE hero( + id INT NOT NULL auto_increment, + name VARCHAR(100) NOT NULL, + phone CHAR(11) NOT NULL, + country varchar(100) NOT NULL, + PRIMARY KEY (id), + KEY idx_name_phone (name, phone) +); +``` + +1. 全值匹配我最爱（就是搜索条件中的列和索引列一致） + + > `select name, phone from hero where name = 'star' and phone = '13266666666'` + > + > 因为有「查询优化器」的存在，所有搜索条件调换顺序，改成 `phone = '13266666666' and name = 'star'` 无影响 +2. 最佳左前缀法则，比如建立了一个联合索引(a,b,c)，那么其实我们可利用的索引就有(a) (a,b)（a,c）(a,b,c) +3. 不在索引列上做任何操作（计算、函数、(自动or手动)类型转换），会导致索引失效而转向全表扫描 +4. 存储引擎不能使用索引中范围条件右边的列 + + ```mysql + -- 只能用到 name 列 + SELECT * FROM hero WHERE name > 'Join' AND name < 'Lily' AND phone > '13222223333'; + + -- 但是如果左边的列是精确查找，则右边的列可以进行范围查找, 也可以用到 phone 列 + SELECT * FROM hero WHERE name = 'Join' AND phone > '13222223333'; + ``` +5. 尽量使用覆盖索引(只访问索引的查询(索引列和查询列一致))，减少 select * +6. `is null` , `is not null` 也无法使用索引 +7. `like "xxxx%"` 是可以用到索引的，`like "%xxxx"` 则不行(like "%xxx%" 同理)。like 以通配符开头('%abc...')索引失效会变成全表扫描的操作， +8. 字符串不加单引号索引失效（隐式类型转换） +9. 少用 or，用它来连接时会索引失效（这个其实不是绝对的，or 走索引与否，还和优化器的**预估**有关，5.0 之后出现的 index merge 技术就是优化这个的） +10. <，<=，=，>，>=，BETWEEN，IN 可用到索引，<>，not in ，!= 则不行，会导致全表扫描 +11. 使用联合索引时，ASC、DESC 混用会导致索引失效 + + + +### [建索引的几大原则](https://tech.meituan.com/2014/06/30/mysql-index.html "MySQL索引原理及慢查询优化") + +1. 最左前缀匹配原则，非常重要的原则，MySQL 会一直向右匹配直到遇到范围查询(>、<、between、like)就停止匹配，比如 `a = 1 and b = 2 and c > 3 and d = 4` 如果建立(a,b,c,d)顺序的索引，d 是用不到索引的，如果建立(a,b,d,c)的索引则都可以用到，a,b,d 的顺序可以任意调整。 + +2. = 和 in 可以乱序，比如 `a = 1 and b = 2 and c = 3` 建立(a,b,c)索引可以任意顺序，MySQL 的查询优化器会帮你优化成索引可以识别的形式。 + +3. 尽量选择区分度高的列作为索引，区分度的公式是 `count(distinct col)/count(*)`，表示字段不重复的比例，比例越大我们扫描的记录数越少，唯一键的区分度是 1，而一些状态、性别字段可能在大数据面前区分度就是 0，那可能有人会问，这个比例有什么经验值吗？使用场景不同，这个值也很难确定，一般需要 join 的字段我们都要求是 0.1 以上，即平均 1 条扫描 10 条记录。 + +4. 索引列不能参与计算，保持列“干净”，比如 `from_unixtime(create_time) = ’2014-05-29’` 就不能使用到索引，原因很简单，b+ 树中存的都是数据表中的字段值，但进行检索时，需要把所有元素都应用函数才能比较，显然成本太大。所以语句应该写成 `create_time = unix_timestamp(’2014-05-29’)`。 + +5. 尽量的扩展索引，不要新建索引。比如表中已经有 a 的索引，现在要加(a,b)的索引，那么只需要修改原来的索引即可。 + +6. 索引列的类型尽量小 + - 数据类型越小，索引占用的存储空间就越少，在一个数据页内就可以放下更多的记录，从而减少磁盘`I/O`带来的性能损耗 + + + + +> 我有一个公众号「 **JavaKeeper** 」 +> +> 我还有一个 **GitBook** [github.com/JavaKeeper](https://github.com/Jstarfish/JavaKeeper) + + + +## Reference + +- https://dev.mysql.com/doc/refman/8.0/en/create-index.html +- https://zh.wikipedia.org/wiki/B%E6%A0%91 +- https://medium.com/@mena.meseha/what-is-the-difference-between-mysql-innodb-b-tree-index-and-hash-index-ed8f2ce66d69 +- https://www.javatpoint.com/b-tree +- https://blog.csdn.net/Abysscarry/article/details/80792876 +- 《MySQL 实战 45 讲》 +- 《高性能 MySQL》 diff --git a/docs/data-management/MySQL/MySQL-Lock.md b/docs/data-management/MySQL/MySQL-Lock.md new file mode 100644 index 0000000000..9f7fa29e92 --- /dev/null +++ b/docs/data-management/MySQL/MySQL-Lock.md @@ -0,0 +1,486 @@ +--- +title: MySQL 锁 +date: 2022-02-15 +tags: + - MySQL 锁 +categories: MySQL +--- + +![](https://img.starfish.ink/mysql/banner-mysql-lock.png) + +> Hello, 我是海星。 +> +> 锁是计算机协调多个进程或线程并发访问某一资源的机制。 +> +> 数据库锁定机制简单来说，就是数据库为了保证数据的一致性，而使各种共享资源在被并发访问变得有序所设计的一种规则。主要用来处理并发问题。 + + 为什么需要锁，只有并发操作时候才有锁的必要，并发事务访问相同记录的情况大致可以划分为 3 种： + +- `读-读`情况：即并发事务相继读取相同的记录 +- `写-写`情况：即并发事务相继对相同的记录做出改动 +- `读-写`或`写-读`情况：也就是一个事务进行读取操作，另一个进行改动操作。 + + + +## 一、锁的分类有哪些 + + +#### 按操作粒度分类： + +> 为了尽可能提高数据库的并发度，每次锁定的数据范围越小越好，理论上每次只锁定当前操作的数据的方案会得到最大的并发度，但是管理锁是很耗资源的事情（涉及获取，检查，释放锁等动作），因此数据库系统需要在高并发响应和系统性能两方面进行平衡，这样就产生了“锁粒度（Lock granularity）”的概念。 + +- **全局锁**：对整个数据库实例加锁，可以用 `Flush tables with read lock (FTWRL)`设置为只读，就相当于加全局锁了。**全局锁的典型使用场景是，做全库逻辑备份。**也就是把整库每个表都 select 出来存成文本。 +- **页级锁**：对数据页（通常是连续的几个行）加锁，控制并发事务对该页的访问。（ BDB 存储引擎使用页级锁） +- **表级锁**：开销小，加锁快；不会出现死锁；锁定粒度大，发生锁冲突的概率较高，并发度最低； + - 表锁的语法是`lock tables … read/write` + - 另一类表级的锁是 MDL（metadata lock)。MDL 不需要显式使用，在访问一个表的时候会被自动加上。MDL 的作用是，保证读写的正确性。 + +- **行级锁**：开销大，加锁慢；会出现死锁；锁定粒度最小，发生锁冲突的概率最低，并发度也最高； + +适用：从锁的角度来说，表级锁更适合于以查询为主，只有少量按索引条件更新数据的应用，如 Web 应用；而行级锁则更适合于有大量按索引条件并发更新少量不同数据，同时又有并发查询的应用，如一些在线事务处理（OLTP）系统。 + + MySQL 不同的存储引擎支持不同的锁机制，所有的存储引擎都以自己的方式实现了锁机制 + +| | 行锁 | 表锁 | 页锁 | +| ------ | ---- | ---- | ---- | +| MyISAM | | √ | | +| BDB | | √ | √ | +| InnoDB | √ | √ | | +| Memory | | √ | | + +#### 按加锁机制分类 + +**乐观锁与悲观锁是两种并发控制的思想，可用于解决丢失更新问题** + +- 乐观锁会“乐观地”假定大概率不会发生并发更新冲突，访问、处理数据过程中不加锁，只在更新数据时再根据版本号或时间戳判断是否有冲突，有则处理，无则提交事务； + +- 悲观锁会“悲观地”假定大概率会发生并发更新冲突，访问、处理数据前就加排他锁，在整个数据处理过程中锁定数据，事务提交或回滚后才释放锁； + +#### 按锁模式（算法）分类 + +- 记录锁（Record Lock）：行级锁的特定类型，锁定单个行，确保其他事务无法同时修改或读取该行 + +- 间隙锁（Gap Lock）：对索引项之间的“间隙”加锁，锁定记录的范围（对第一条记录前的间隙或最后一条将记录后的间隙加锁），不包含索引项本身。其他事务不能在锁范围内插入数据，这样就防止了别的事务新增幻影行 + +- MDL（Metadata Lock）：锁定数据库对象的元数据，如表结构，用于保证数据定义的一致性 +- 临建锁(next-key Lock)： 锁定索引项本身和索引范围。即 Record Lock 和 Gap Lock 的结合。可解决幻读问题。 + +#### 按属性分类： + +- **读锁**（共享锁）： + + - `共享锁`，英文名：`Shared Locks`，简称`S锁`。在事务要读取一条记录时，需要先获取该记录的`S锁`。 + + - 针对同一份数据，多个读操作可以同时进行，不会互相影响 + + - ```mysql + SELECT ... LOCK IN SHARE MODE; //对读取的记录加S锁 + ``` + +- **写锁**（独占锁、排他锁）： + + - 当前写操作没有完成前，它会阻断其他写锁和读锁 + + - `独占锁`，也常称`排他锁`，英文名：`Exclusive Locks`，简称`X锁`。在事务要改动一条记录时，需要先获取该记录的`X锁` + + - ```mysql + SELECT ... FOR UPDATE; //对读取的记录加X锁 + ``` + +#### 按状态分类 + +- 意向共享锁（Intention Shared Lock）：表级锁的辅助锁，表示事务要在某个表或页级锁上获取共享锁。 +- 意向排它锁（Intention Exclusive Lock）：表级锁的辅助锁，表示事务要在某个表或页级锁上获取排它锁。 + + + +## 二、全局锁 + +要使用全局锁，则要执行这条命令： + +```sql +flush tables with read lock +``` + +执行后，**整个数据库就处于只读状态了**，这时其他线程执行以下操作，都会被阻塞 + +如果要释放全局锁，则要执行这条命令： + +```sql +unlock tables +``` + +全局锁主要应用于做**全库逻辑备份**，这样在备份数据库期间，不会因为数据或表结构的更新，而出现备份文件的数据与预期的不一样。 + + + +## 三、表锁 + +MySQL 里面表级别的锁有这几种： + +- 表锁 +- 元数据锁（MDL） +- 意向锁 +- AUTO-INC 锁 + +#### 表锁 + +MySQL 支持多种存储引擎，不同存储引擎对锁的支持也是不一样的。 + +对于`MyISAM`、`MEMORY`、`MERGE`这些存储引擎来说，它们只支持表级锁，而且这些引擎并不支持事务，所以使用这些存储引擎的锁一般都是针对当前会话来说的。 + +#### 元数据锁 + +**另一类表级的锁是 MDL（metadata lock)。**MDL 不需要显式使用，在访问一个表的时候会被自动加上。MDL 的作用是，保证读写的正确性。你可以想象一下，如果一个查询正在遍历一个表中的数据，而执行期间另一个线程对这个表结构做变更，删了一列，那么查询线程拿到的结果跟表结构对不上，肯定是不行的。 + +因此，在 MySQL 5.5 版本中引入了 MDL，当对一个表做增删改查操作的时候，加 MDL 读锁；当要对表做结构变更操作的时候，加 MDL 写锁。 + +- 读锁之间不互斥，因此你可以有多个线程同时对一张表增删改查。 + +- 读写锁之间、写锁之间是互斥的，用来保证变更表结构操作的安全性。因此，如果有两个线程要同时给一个表加字段，其中一个要等另一个执行完才能开始执行。 + +虽然 MDL 锁是系统默认会加的，但却是你不能忽略的一个机制。比如下面这个例子，我经常看到有人掉到这个坑里：给一个小表加个字段，导致整个库挂了。 + +1. 首先，线程 A 先启用了事务（但是一直不提交），然后执行一条 select 语句，此时就先对该表加上 MDL 读锁； +2. 然后，线程 B 也执行了同样的 select 语句，此时并不会阻塞，因为「读读」并不冲突； +3. 接着，线程 C 修改了表字段，此时由于线程 A 的事务并没有提交，也就是 MDL 读锁还在占用着，这时线程 C 就无法申请到 MDL 写锁，就会被阻塞， + +那么在线程 C 阻塞后，后续有对该表的 select 语句，就都会被阻塞，如果此时有大量该表的 select 语句的请求到来，就会有大量的线程被阻塞住，这时数据库的线程很快就会爆满了。 + +> 为什么线程 C 因为申请不到 MDL 写锁，而导致后续的申请读锁的查询操作也会被阻塞？ + +这是因为申请 MDL 锁的操作会形成一个队列，队列中**写锁获取优先级高于读锁**，一旦出现 MDL 写锁等待，会阻塞后续该表的所有 CRUD 操作。 + +所以为了能安全的对表结构进行变更，在对表结构变更前，先要看看数据库中的长事务，是否有事务已经对表加上了 MDL 读锁，如果可以考虑 kill 掉这个长事务，然后再做表结构的变更。 + +#### 表级别的意向锁 + +- 意向共享锁，英文名：`Intention Shared Lock`，简称`IS锁`。在使用 InnoDB 引擎的表里对某些记录加上「共享锁」之前，需要先在表级别加上一个「意向共享锁」； +- 意向独占锁，英文名：`Intention Exclusive Lock`，简称`IX锁`。在使用 InnoDB 引擎的表里对某些纪录加上「独占锁」之前，需要先在表级别加上一个「意向独占锁」； + +`IS锁`和`IX锁`的使命只是为了后续在加表级别的`S锁`和`X锁`时判断表中是否有已经被加锁的记录，以避免用遍历的方式来查看表中有没有上锁的记录。 + +#### AUTO-INC 锁 + +表里的主键通常都会设置成自增的，这是通过对主键字段声明 `AUTO_INCREMENT` 属性实现的。 + +之后在插入数据时，可以不指定主键的值，数据库会自动给主键赋值递增的值，这主要是通过 **AUTO-INC 锁**实现的。 + +AUTO-INC 锁是特殊的表锁机制，锁**不是在一个事务提交后才释放，而是在执行完插入语句后就会立即释放**。 + +**在插入数据时，会加一个表级别的 AUTO-INC 锁**，然后为被 `AUTO_INCREMENT` 修饰的字段赋值递增的值，等插入语句执行完成后，才会把 AUTO-INC 锁释放掉。 + +那么，一个事务在持有 AUTO-INC 锁的过程中，其他事务如果要向该表插入语句都会被阻塞，从而保证插入数据时，被 `AUTO_INCREMENT` 修饰的字段的值是连续递增的。 + +但是， AUTO-INC 锁再对大量数据进行插入的时候，会影响插入性能，因为另一个事务中的插入会被阻塞。 + +因此， 在 MySQL 5.1.22 版本开始，InnoDB 存储引擎提供了一种**轻量级的锁**来实现自增。 + +一样也是在插入数据的时候，会为被 `AUTO_INCREMENT` 修饰的字段加上轻量级锁，**然后给该字段赋值一个自增的值，就把这个轻量级锁释放了，而不需要等待整个插入语句执行完后才释放锁**。 + +InnoDB 存储引擎提供了个 `innodb_autoinc_lock_mode` 的系统变量，是用来控制选择用 AUTO-INC 锁，还是轻量级的锁。 + +- 当 innodb_autoinc_lock_mode = 0，就采用 AUTO-INC 锁，语句执行结束后才释放锁； +- 当 innodb_autoinc_lock_mode = 2，就采用轻量级锁，申请自增主键后就释放锁，并不需要等语句执行后才释放。 +- 当 innodb_autoinc_lock_mode = 1，相当于两种方式混着来 + - 普通 insert 语句，自增锁在申请之后就马上释放； + - 类似 insert … select 这样的批量插入数据的语句，自增锁还是要等语句结束后才被释放； + +当 innodb_autoinc_lock_mode = 2 是性能最高的方式，但是当搭配 binlog 的日志格式是 statement 一起使用的时候，可能会造成不同事务中的插入语句为 AUTO_INCREMENT 修饰的列生成的值是交叉的，在「主从复制的场景」中会发生**数据不一致的问题**。 + + + +#### 如何加表锁 + +MyISAM 在执行查询语句（SELECT）前，会自动给涉及的所有表加读锁，在执行更新操作（UPDATE、DELETE、INSERT等）前，会自动给涉及的表加写锁，这个过程并不需要用户干预，因此，用户一般不需要直接用 LOCK TABLE 命令给 MyISAM 表显式加锁。 + +#### MyISAM 表锁优化建议 + +对于 MyISAM 存储引擎，虽然使用表级锁定在锁定实现的过程中比实现行级锁定或者页级锁所带来的附加成本都要小，锁定本身所消耗的资源也是最少。但是由于锁定的颗粒度比较大，所以造成锁定资源的争用情况也会比其他的锁定级别都要多，从而在较大程度上会降低并发处理能力。所以，在优化MyISAM存储引擎锁定问题的时候，最关键的就是如何让其提高并发度。由于锁定级别是不可能改变的了，所以我们首先需要**尽可能让锁定的时间变短**，然后就是让可能并发进行的操作尽可能的并发。 + +看看哪些表被加锁了: + +```mysql +mysql>show open tables; +``` + +1. ##### 查询表级锁争用情况 + +MySQL 内部有两组专门的状态变量记录系统内部锁资源争用情况： + +```mysql +mysql> show status like 'table%'; +``` + +这里有两个状态变量记录 MySQL 内部表级锁定的情况，两个变量说明如下： + +- Table_locks_immediate：产生表级锁定的次数，表示可以立即获取锁的查询次数，每立即获取锁值加1 + +- Table_locks_waited：出现表级锁定争用而发生等待的次数(不能立即获取锁的次数，每等待一次锁值加1)，此值高则说明存在着较严重的表级锁争用情况 + +两个状态值都是从系统启动后开始记录，出现一次对应的事件则数量加1。如果这里的Table_locks_waited状态值比较高，那么说明系统中表级锁定争用现象比较严重，就需要进一步分析为什么会有较多的锁定资源争用了。 + +> 此外，Myisam的读写锁调度是写优先，这也是myisam不适合做写为主表的引擎。因为写锁后，其他线程不能做任何操作，大量的更新会使查询很难得到锁，从而造成永远阻塞 + +2. **缩短锁定时间** + + 如何让锁定时间尽可能的短呢？唯一的办法就是让我们的Query执行时间尽可能的短。 + +- **尽量减少大的复杂Query，将复杂Query分拆成几个小的Query分布进行；** +- **尽可能的建立足够高效的索引，让数据检索更迅速；** +- **尽量让MyISAM存储引擎的表只存放必要的信息，控制字段类型；** +- **利用合适的机会优化MyISAM表数据文件。** + +3. 分离能并行的操作 + + 说到MyISAM的表锁，而且是读写互相阻塞的表锁，可能有些人会认为在MyISAM存储引擎的表上就只能是完全的串行化，没办法再并行了。大家不要忘记了，MyISAM的存储引擎还有一个非常有用的特性，那就是ConcurrentInsert（并发插入）的特性。 + + MyISAM存储引擎有一个控制是否打开Concurrent Insert功能的参数选项：`concurrent_insert`，可以设置为0，1或者2。三个值的具体说明如下： + +- concurrent_insert=2，无论MyISAM表中有没有空洞，都允许在表尾并发插入记录； + +- concurrent_insert=1，如果MyISAM表中没有空洞（即表的中间没有被删除的行），MyISAM允许在一个进程读表的同时，另一个进程从表尾插入记录。这也是MySQL的默认设置； + +- concurrent_insert=0，不允许并发插入。 + + 可以利用MyISAM存储引擎的并发插入特性，来解决应用中对同一表查询和插入的锁争用。例如，将concurrent_insert系统变量设为2，总是允许并发插入；同时，通过定期在系统空闲时段执行OPTIMIZE TABLE语句来整理空间碎片，收回因删除记录而产生的中间空洞。 + +4. 合理利用读写优先级 + + MyISAM存储引擎的是读写互相阻塞的，那么，一个进程请求某个MyISAM表的读锁，同时另一个进程也请求同一表的写锁，MySQL如何处理呢？ + + 答案是写进程先获得锁。不仅如此，即使读请求先到锁等待队列，写请求后到，写锁也会插到读锁请求之前。 + + 这是因为MySQL的表级锁定对于读和写是有不同优先级设定的，默认情况下是写优先级要大于读优先级。 + + 所以，如果我们可以根据各自系统环境的差异决定读与写的优先级： + + 通过执行命令SET LOW_PRIORITY_UPDATES=1，使该连接读比写的优先级高。如果我们的系统是一个以读为主，可以设置此参数，如果以写为主，则不用设置； + + 通过指定INSERT、UPDATE、DELETE语句的LOW_PRIORITY属性，降低该语句的优先级。 + + 虽然上面方法都是要么更新优先，要么查询优先的方法，但还是可以用其来解决查询相对重要的应用（如用户登录系统）中，读锁等待严重的问题。 + + 另外，MySQL也提供了一种折中的办法来调节读写冲突，即给系统参数max_write_lock_count设置一个合适的值，当一个表的读锁达到这个值后，MySQL就暂时将写请求的优先级降低，给读进程一定获得锁的机会。 + + 这里还要强调一点：一些需要长时间运行的查询操作，也会使写进程“饿死”，因此，应用中应尽量避免出现长时间运行的查询操作，不要总想用一条SELECT语句来解决问题，因为这种看似巧妙的SQL语句，往往比较复杂，执行时间较长，在可能的情况下可以通过使用中间表等措施对SQL语句做一定的“分解”，使每一步查询都能在较短时间完成，从而减少锁冲突。如果复杂查询不可避免，应尽量安排在数据库空闲时段执行，比如一些定期统计可以安排在夜间执行。 + + + +## 四、行锁 + +InnoDB 引擎是支持行级锁的，而 MyISAM 引擎并不支持行级锁。 + +> InnoDB 与 MyISAM 的最大不同有两点：一是支持事务（TRANSACTION）；二是采用了行级锁 + +`行锁`，也称为`记录锁`，顾名思义就是在记录上加的锁。 + + + +行级锁的类型主要有三类： + +- Record Lock，记录锁，也就是仅仅把一条记录锁上； +- Gap Lock，间隙锁，锁定一个范围，但是不包含记录本身； +- Next-Key Lock：Record Lock + Gap Lock 的组合，锁定一个范围，并且锁定记录本身。 + +#### Record Lock + +Innodb 存储引擎由于实现了行级锁定，虽然在锁定机制的实现方面所带来的性能损耗可能比表级锁定会要更高一些，但是在整体并发处理能力方面要远远优于 MyISAM 的表级锁定的。当系统并发量较高的时候，Innodb 的整体性能和 MyISAM 相比就会有比较明显的优势了。 + +Record Lock 称为记录锁，锁住的是一条记录。而且记录锁是有 S 锁和 X 锁之分的： + +- 当一个事务对一条记录加了 S 型记录锁后，其他事务也可以继续对该记录加 S 型记录锁（S 型与 S 锁兼容），但是不可以对该记录加 X 型记录锁（S 型与 X 锁不兼容）; +- 当一个事务对一条记录加了 X 型记录锁后，其他事务既不可以对该记录加 S 型记录锁（S 型与 X 锁不兼容），也不可以对该记录加 X 型记录锁（X 型与 X 锁不兼容）。 + +举个例子，当一个事务执行了下面这条语句： + +```sql +mysql > begin; +mysql > select * from t where id = 4 for update; +``` + +就是对 t 表中主键 id 为 4 的这条记录加上 X 型的记录锁，这样其他事务就无法对这条记录进行修改了。 + +![](https://img.starfish.ink/mysql/MySQL-record-lock.png) + +当事务执行 commit 后，事务过程中生成的锁都会被释放。 + + + +#### Gap Lock + +Gap Lock 称为间隙锁，只存在于可重复读隔离级别，目的是为了解决可重复读隔离级别下幻读的现象。 + +假设，表中有一个范围 id 为（4，8）间隙锁，那么其他事务就无法插入 id = 5、6、7 的记录了，这样就有效的防止幻读现象的发生。 + +![](https://img.starfish.ink/mysql/MySQL-gap-lock.png) + +间隙锁虽然存在 X 型间隙锁和 S 型间隙锁，但是并没有什么区别，**间隙锁之间是兼容的，即两个事务可以同时持有包含共同间隙范围的间隙锁，并不存在互斥关系，因为间隙锁的目的是防止插入幻影记录而提出的**。 + + + +#### Next-Key Lock + +Next-Key Lock 称为临键锁，是 Record Lock + Gap Lock 的组合，锁定一个范围，并且锁定记录本身。 + +假设，表中有一个范围 id 为（4，8] 的 next-key lock，那么其他事务即不能插入 id = 5,6,7 记录，也不能修改 id = 8 这条记录。 + +![](https://img.starfish.ink/mysql/MySQL-next-key-lock.png) + +所以，next-key lock 即能保护该记录，又能阻止其他事务将新记录插入到被保护记录前面的间隙中。 + +**next-key lock 是包含间隙锁+记录锁的，如果一个事务获取了 X 型的 next-key lock，那么另外一个事务在获取相同范围的 X 型的 next-key lock 时，是会被阻塞的**。 + +比如，一个事务持有了范围为 (4, 8] 的 X 型的 next-key lock，那么另外一个事务在获取相同范围的 X 型的 next-key lock 时，就会被阻塞。 + +虽然相同范围的间隙锁是多个事务相互兼容的，但对于记录锁，我们是要考虑 X 型与 S 型关系，X 型的记录锁与 X 型的记录锁是冲突的。 + + + +#### 插入意向锁 + +一个事务在插入一条记录的时候，需要判断插入位置是否已被其他事务加了间隙锁（next-key lock 也包含间隙锁）。 + +如果有的话，插入操作就会发生**阻塞**，直到拥有间隙锁的那个事务提交为止（释放间隙锁的时刻），在此期间会生成一个**插入意向锁**，表明有事务想在某个区间插入新记录，但是现在处于等待状态。 + +举个例子，假设事务 A 已经对表加了一个范围 id 为（4，8）间隙锁。 + +当事务 A 还没提交的时候，事务 B 向该表插入一条 id = 5 的新记录，这时会判断到插入的位置已经被事务 A 加了间隙锁，于是事物 B 会生成一个插入意向锁，然后将锁的状态设置为等待状态（*PS：MySQL 加锁时，是先生成锁结构，然后设置锁的状态，如果锁状态是等待状态，并不是意味着事务成功获取到了锁，只有当锁状态为正常状态时，才代表事务成功获取到了锁*），此时事务 B 就会发生阻塞，直到事务 A 提交了事务。 + +插入意向锁名字虽然有意向锁，但是它并**不是意向锁，它是一种特殊的间隙锁，属于行级别锁**。 + +如果说间隙锁锁住的是一个区间，那么「插入意向锁」锁住的就是一个点。因而从这个角度来说，插入意向锁确实是一种特殊的间隙锁。 + +插入意向锁与间隙锁的另一个非常重要的差别是：尽管「插入意向锁」也属于间隙锁，但两个事务却不能在同一时间内，一个拥有间隙锁，另一个拥有该间隙区间内的插入意向锁（当然，插入意向锁如果不在间隙锁区间内则是可以的）。 + + + +#### InnoDB 行锁实现方式 + +**InnoDB 行锁是通过给索引上的索引项加锁来实现的，只有通过索引条件检索数据，InnoDB 才使用行级锁，否则，InnoDB 将使用表锁** + +在实际应用中，要特别注意 InnoDB 行锁的这一特性，不然的话，可能导致大量的锁冲突，从而影响并发性能。下面通过一些实际例子来加以说明。 + +1. 在不通过索引条件查询的时候，InnoDB 确实使用的是表锁，而不是行锁。 +2. 由于 MySQL 的行锁是针对索引加的锁，不是针对记录加的锁，所以虽然是访问不同行的记录，但是如果是使用相同的索引键，是会出现锁冲突的。 +3. 当表有多个索引的时候，不同的事务可以使用不同的索引锁定不同的行，另外，不论是使用主键索引、唯一索引或普通索引，InnoDB 都会使用行锁来对数据加锁。 +4. 即便在条件中使用了索引字段，但是否使用索引来检索数据是由 MySQL 通过判断不同执行计划的代价来决定的，如果 MySQL 认为全表扫描效率更高，比如对一些很小的表，它就不会使用索引，这种情况下 InnoDB 将使用表锁，而不是行锁。因此，**在分析锁冲突时，别忘了检查 SQL 的执行计划，以确认是否真正使用了索引**。 + + + +#### 如何分析行锁定 + +通过检查 `InnoDB_row_lock` 状态变量来分析系统上的行锁的争夺情况 + +```mysql +mysql>show status like 'innodb_row_lock%'; +``` + +![](https://img.starfish.ink/mysql/innodb_row_lock.png) + +- `Innodb_row_lock_current_waits`：当前正在等待锁定的数量； +- `Innodb_row_lock_time`：从系统启动到现在锁定总时间长度； +- `Innodb_row_lock_time_avg`：每次等待所花平均时间； +- `Innodb_row_lock_time_max`：从系统启动到现在等待最常的一次所花的时间； +- `Innodb_row_lock_waits`：系统启动后到现在总共等待的次数； + +#### 行锁优化 + +- 尽可能让所有数据检索都通过索引来完成，避免无索引行锁升级为表锁 + +- 合理设计索引，尽量缩小锁的范围 + +- 尽可能较少检索条件，避免间隙锁 + +- 尽量控制事务大小，减少锁定资源量和时间长度 + +- 尽可能低级别事务隔离 + + + +### 页锁 + +开销和加锁时间界于表锁和行锁之间；会出现死锁；锁定粒度界于表锁和行锁之间，并发度一般。 + + + +## 五、InnoDB 锁内存结构 + +```mysql +*# 事务T1* +SELECT * FROM hero LOCK IN SHARE MODE*;* +``` + +很显然这条语句需要为`hero表`中的所有记录进行加锁，那是不是需要为每条记录都生成一个`锁结构`呢？ + +在对不同记录加锁时，如果符合下边这些条件： + +- 在同一个事务中进行加锁操作 +- 被加锁的记录在同一个页面中 +- 加锁的类型是一样的 +- 等待状态是一样的 + +那么这些记录的锁就可以被放到一个`锁结构`中 + +`锁`是一个内存结构，InnoDB中用 `lock_t` 这个结构来定义(8.0)： + +```c +struct lock_t { + /** transaction owning the lock */ + trx_t *trx; + + /** list of the locks of the transaction */ + UT_LIST_NODE_T(lock_t) trx_locks; + + /** Index for a record lock */ + dict_index_t *index; + + /** Hash chain node for a record lock. The link node in a singly + linked list, used by the hash table. */ + lock_t *hash; + + union { + /** Table lock */ + lock_table_t tab_lock; + + /** Record lock */ + lock_rec_t rec_lock; + }; +``` + + + +## 六、死锁 + +![](https://img.starfish.ink/mysql/deadlock.jpeg) + +死锁是指两个或者多个事务在同一资源上互相占用，并请求锁定对方占用的资源，从而导致恶性循环的现象。当多个事务试图以不同的顺序锁定资源时，就可能会产生死锁。多个事务同时锁定同一个资源时，也会产生死锁。 + +![](https://img.starfish.ink/mysql/MySQL-dead-lock-demo.png) + +为了解决这种问题，数据库系统实现了各种死锁检测和死锁超时机制。 + +当出现死锁以后，有两种策略： + +- 一种策略是，直接进入等待，直到超时。这个超时时间可以通过参数 innodb_lock_wait_timeout 来设置。 +- 另一种策略是，发起死锁检测，发现死锁后，主动回滚死锁链条中的某一个事务，让其他事务得以继续执行。将参数 innodb_deadlock_detect 设置为 on，表示开启这个逻辑。 + +在 InnoDB 中，`innodb_lock_wait_timeout` 的默认值是 50s，意味着如果采用第一个策略，当出现死锁以后，第一个被锁住的线程要过 50s 才会超时退出，然后其他线程才有可能继续执行。对于在线服务来说，这个等待时间往往是无法接受的。 + +但是，我们又不可能直接把这个时间设置成一个很小的值，比如 1s。这样当出现死锁的时候，确实很快就可以解开，但如果不是死锁，而是简单的锁等待呢？所以，超时时间设置太短的话，会出现很多误伤。 + +所以，正常情况下我们还是要采用第二种策略，即：主动死锁检测，而且 `innodb_deadlock_detect` 的默认值本身就是 on。主动死锁检测在发生死锁的时候，是能够快速发现并进行处理的，但是它也是有额外负担的。 + + + +## 参考 + +- 《高性能 MySQL》 +- 《MySQL技术内幕：innodb》 +- 《MySQL实战45讲》 +- 《从根儿上理解MySQL》 + + + + + diff --git a/docs/data-management/MySQL/MySQL-Log.md b/docs/data-management/MySQL/MySQL-Log.md new file mode 100755 index 0000000000..76ad66835e --- /dev/null +++ b/docs/data-management/MySQL/MySQL-Log.md @@ -0,0 +1,940 @@ +--- +title: MySQL 是怎么想的，搞这么多种日志 +date: 2022-08-01 +tags: + - MySQL + - MySQL 日志 +categories: MySQL +--- + +![](https://img.starfish.ink/mysql/banner-mysql-log.png) + +> Hello，我是海星。 +> +> 不管是 DB 还是其他组件，很多看似奇怪的问题，答案往往就藏在日志里。 +> +> 这一篇聊聊 MySQL 的一些日志文件 + +MySQL日志文件：用来记录 MySQL 实例对某种条件做出响应时写入的文件，大概可分为：通用查询日志、慢查询日志、错误日志、二进制日志、中继日志、重做日志和回滚日志。 + +他们都有什么作用和关联，我们一起捋一捋（基于 InnoDB） + +![](https://img.starfish.ink/mysql/logs.png) + +我们从一条 SQL 语句说起吧 + +```mysql +UPDATE fish SET type = 2 WHERE name = 'starfish'; +``` + +先说个不怎么被提到的，通用查询日志 + +## 一、通用查询日志(general log) + +通用查询日志记录了所有用户的连接开始时间和截止时间，以及发给 MySQL 数据库服务器的所有 SQL 指令。当我们的数据发生异常时，开启通用查询日志，还原操作时的具体场景，可以帮助我们准确定位问题。 + +```mysql +mysql> SHOW VARIABLES LIKE '%general%'; ++------------------+-------------------------------------------------+ +| Variable_name | Value | ++------------------+-------------------------------------------------+ +| general_log | OFF | +| general_log_file | /usr/local/mysql/data/starfishdeMacBook-Pro.log | ++------------------+-------------------------------------------------+ +2 rows in set (0.00 sec) +``` + +通用日志默认是关闭的，我们开启后看下效果 + +```mysql +SET GLOBAL general_log = 'ON' +## SET @@gobal.generl_log_file = /usr/local/mysql/data/starfishdeMacBook-Pro.log +``` + +客户端执行 update 指令，查看日志目录如下，这玩意会记录从数据库连接到终止之间的所有记录，而且都是文本数据，太占资源了，所以一般没有开启的。 + +![](https://img.starfish.ink/mysql/generl_log.png) + + + +## 二、重做日志(redo log) + +了解这块知识，我们先要知道这么几个前置知识点，先看下官网的 InnoDB 架构图，有个大概印象 + +![](https://img.starfish.ink/mysql/innodb-architecture.png) + +> #### 什么是 随机 IO 和 顺序 IO +> +> 磁盘读写数据的两种方式。随机 IO 需要先找到地址，再读写数据，每次拿到的地址都是随机的。比如 MySQL 执行增删改操作时。就像送外卖，每一单送的地址都不一样，到处跑，效率极低。而顺序 IO，由于地址是连贯的，找到地址后，一次可以读写许多数据，效率比较高，比如在末尾追加日志这种。就像送外卖，所有的单子地址都在一栋楼，一下可以送很多，效率很高。 + +> #### 什么是数据页？ +> +> MySQL 的 InnoDB 存储引擎以 Data Page（数据页）作为磁盘和内存之间交互的基本单位，他的大小一般为默认值 16K。 +> +> 从数据页的作用来分，可以分为 Free Page（空闲页）、Clean Page（干净页）、Dirty Page（脏页）； +> +> - **当内存数据页 和 磁盘数据页内容不一致的时候，这个内存页 就为 “脏页”**。将内存中的数据同步到磁盘中的这个过程就被称为**“刷脏”** +> - **内存数据页写入磁盘后，内存数据页 和 磁盘数据页内容一致，称之为 “干净页”** +> +> 从类型来分的话，还可以分成存放 UNDO 日志的页、存放 INODE 信息的页、存放表空间头部信息的页等。 + +> #### 什么是缓冲池 | buffer pool？ +> +> 关系型数据库的特点就是需要对磁盘中大量的数据进行存取，所以有时候也被叫做基于磁盘的数据库。正是因为数据库需要频繁对磁盘进行 IO 操作，为了改善因为直接读写磁盘导致的 IO 性能问题，所以引入了缓冲池。 +> +> 不过不论是什么类型的页面，每当我们从页面中读取或写入数据时，都必须先将其从硬盘上加载到内存中的`buffer pool`中（也就是说内存中的页面其实就是硬盘中页面的一个副本），然后才能对内存中页面进行读取或写入。如果要修改内存中的页面，为了减少磁盘 I/O，修改后的页面并不立即同步到磁盘，而是作为`脏页`继续呆在内存中，等待后续合适时机将其刷新到硬盘（一般是有后台线程异步刷新），将该页刷到磁盘的操作称为 刷脏页 （本句是重点，后面要吃）。 +> +> #### 内存缓冲区 +>> +> InnoDB 存储引擎是基于磁盘存储的，并将其中的记录按照页的方式进行管理。由于 CPU 速度与磁盘速度之间的鸿沟，基于磁盘的数据库系统通常使用内存缓冲区技术来提高数据库的整体性能。 +> +> ##### Page Cache +> +> InnoDB 会将读取过的页缓存在内存中，并采取最近最少使用（Least Recently Used，LRU） 算法将缓冲池作为列表进行管理，以增加缓存命中率。 +> +> InnoDB 对 LRU 算法进行了一定的改进，执行**中点插入策略**，默认前 5/8 为`New Sublist`，存储经常被使用的热点页，后 3/8 为`Old Sublist`。新读入的 page 默认被加在`old Sublist`的头部，如果在运行过程中 `old Sublist` 的数据被访问到了，那么这个页就会被移动到 `New Sublist` 的头部。 +> +> ![](https://img.starfish.ink/mysql/lru-buffer-pool.png) +> +> 每当 InnoDB 缓存新的数据页时，会优先从 `Free List` 中查找空余的缓存区域，如果不存在，那么就需要从 `LRU List` 淘汰一定的尾部节点。不管数据页位于 `New Sublist` 还是 `Old Sublist`，如果没有被访问到，那么最终都会被移动到 `LRU List` 的尾部作为牺牲者。 +> +> ##### Change Buffer +> +> Change Buffer 用于记录数据的修改，因为 InnoDB 的辅助索引不同于聚集索引的顺序插入，如果每次修改二级索引都直接写入磁盘，则会有大量频繁的随机 IO。 +> +> InnoDB 从 1.0.x 版本开始引入了 Change Buffer，主要目的是将对**非唯一辅助索引**页的操作缓存下来，如果辅助索引页已经在缓冲区了，则直接修改；如果不在，则先将修改保存到 Change Buffer。当对应辅助索引页读取到缓冲区时，才将 Change Buffer 的数据合并到真正的辅助索引页中，以此减少辅助索引的随机 IO，并达到操作合并的效果。 +> +> ![](https://img.starfish.ink/mysql/innodb-change-buffer.png) +> +> 在 MySQL 5.5 之前 Change Buffer 其实叫 Insert Buffer，最初只支持 INSERT 操作的缓存，随着支持操作类型的增加，改名为 Change Buffer，现在 InnoDB 存储引擎可以对 INSERT、DELETE、UPDATE 都进行缓冲，对应着：Insert Buffer、Delete Buffer、Purge buffer。 +> +> ##### Double Write Buffer +> +> 当发生数据库宕机时，可能存储引擎正在写入某个页到表中，而这个页只写了一部分，比如 16KB 的页，只写了前 4KB，之后就发生了宕机。虽然可以通过日志进行数据恢复，但是如果这个页本身已经发生了损坏，再对其进行重新写入是没有意义的。因此 InnoDB 引入 Double Write Buffer 解决数据页的半写问题。 +> +> Double Write Buffer 大小默认为 2M，即 128 个数据页。其中分为两部分，一部分留给`batch write`，提供批量刷新脏页的操作，另一部分是`single page write`，留给用户线程发起的单页刷脏操作。 +> +> 在对缓冲池的脏页进行刷新时，脏页并不是直接写到磁盘，而是会通过`memcpy()`函数将脏页先复制到内存中的 Double Write Buffer 中，如果 Double Write Buffer 写满了，那么就会调用`fsync()`系统调用，一次性将 Double Write Buffer 所有的数据写入到磁盘中，因为这个过程是顺序写入，开销几乎可以忽略。在确保写入成功后，再使用异步 IO 把各个数据页写回自己的表空间中。 + +### 2.1 为什么需要 redo log + +好了，我们步入主题 redo log，这里比较喜欢丁奇老师的比喻 + +> 酒店掌柜有一个粉板，专门用来记录客人的赊账记录。如果赊账的人不多，可以把顾客名和账目写在板上。但如果赊账的人多了，粉板总会有记不下的时候，这个时候掌柜一定还有一个专门记录赊账的账本。 +> +> 如果有人要赊账或者还账的话，掌柜一般有两种做法： +> +> - 一种做法是直接把账本翻出来，把这次赊的账加上去或者扣除掉； +> - 另一种做法是先在粉板上记下这次的账，等打烊以后再把账本翻出来核算。 +> +> 在生意红火柜台很忙时，掌柜一定会选择后者，因为前者操作实在是太麻烦了。首先，你得找到这个人的赊账总额那条记录。你想想，密密麻麻几十页，掌柜要找到那个名字，可能还得带上老花镜慢慢找，找到之后再拿出算盘计算，最后再将结果写回到账本上。 +> +> 这整个过程想想都麻烦。相比之下，还是先在粉板上记一下方便。你想想，如果掌柜没有粉板的帮助，每次记账都得翻账本，效率是不是低得让人难以忍受？ +> + +- 同样，在 MySQL 里也有这个问题，Innodb 是以`页`为单位进行磁盘交互的，而一个事务很可能只修改一个数据页里面的几个字节，这个时候将完整的数据页刷到磁盘的话，整个过程 IO 成本、查找成本都很高。 + +- 一个事务可能涉及修改多个数据页，并且这些数据页在物理上并不连续，使用随机 IO 写入性能太差！ + +这样的性能问题，我们是不能忍的，前面我们讲到数据页在缓冲池中被修改会变成脏页。如果这时宕机，脏页就会失效，这就导致我们修改的数据丢失了，也就无法保证事务的**持久性**。 + +为了解决这些问题，MySQL 的设计者就用了类似酒店掌柜粉板的思路，设计了`redo log`，**具体来说就是只记录事务对数据页做了哪些修改**，这样就能完美地解决性能问题了(相对而言文件更小并且是顺序 IO)。 + +而粉板和账本配合的整个过程，其实就类似 MySQL 里经常说到的 **WAL 技术**（Write-Ahead Loging），它的关键点就是**先写日志，再写磁盘**，也就是先写粉板，等不忙的时候再写账本。这又解决了我们的持久性问题。 + +具体来说，当有一条记录需要更新的时候，InnoDB 引擎就会先把记录写到 `redo log`（粉板）里面，并更新内存，这个时候更新就算完成了。同时，InnoDB 引擎会在适当的时候，将这个操作记录更新到磁盘里面，而这个更新往往是在系统比较空闲的时候做，这就像打烊以后掌柜做的事。 + +> DBA 口中的**日志先行**说的就是这个 WAL 技术。 +> +> 记录下对磁盘中某某页某某位置数据的修改结果的 redo log，这种日志被称为**物理日志**，可以节省很多磁盘空间。 +> +> 最开始看到的通用查询日志，记录了所有数据库的操作，我们叫**逻辑日志**，还有下边会说的 binlog、undo log 也都属于逻辑日志。 + + + +### 2.2 组织 redo log + +#### 「redo log 结构」 + +MySQL redo日志是一组日志文件，在 MySQL 8.0.30 版本中，MySQL 会生成 32 个 redo log 文件（老版本默认 2 个） + +![](https://img.starfish.ink/mysql/ib_redo-file.png) + +> [`innodb_log_file_size`](https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_log_file_size) and [`innodb_log_files_in_group`](https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_log_files_in_group) are deprecated in MySQL 8.0.30. These variables are superseded by [`innodb_redo_log_capacity`](https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_redo_log_capacity). For more information, see [Section 15.6.5, “Redo Log”](https://dev.mysql.com/doc/refman/8.0/en/innodb-redo-log.html). + +为了应对 InnoDB 各种各样不同的需求，到 MySQL 8.0 为止，已经有多达 65 种的 REDO 记录，每种都不太一样，我们看下比较通用的结构了解了解就 OK（主要有作用于 Page，作用于 Space 以及提供额外信息的 Logic 类型的三大类） + +![](https://img.starfish.ink/mysql/redolog-content.png) + +比如 `MLOG_WRITE_STRING` 类型的 REDO 表示写入一串数据，但是因为不能确定写入的数据占多少字节，所以需要在日志结构中添加一个长度字段来表示写入了多长的数据。 + +除此之外，还有一些复杂的 REDO 类型来记录一些复杂的操作。例如插入一条数据，并不仅仅只是在数据页中插入一条数据，还可能会导致数据页和索引页的分裂，可能要修改数据页中的头信息（Page Header）、目录槽信息（Page Directory）等等。 + +#### 「Mini-Transaction」 + +一个事务中可能有多个增删改的 SQL语句，而一个 SQL 语句在执行过程中可能修改若干个页面，会有多个操作。 + +> 例如一个 INSERT 语句： +> +> - 如果表没有主键，会去更新内存中的 `Max Row ID` 属性，并在其值为 `256` 的倍数时，将其刷新到`系统表空间`的页号为 `7` 的`Max Row ID `属性处。 +> - 接着向聚簇索引插入数据，这个过程要根据索引找到要插入的缓存页位置，向数据页插入记录。这个过程还可能会涉及数据页和索引页的分裂，那就会增加或修改一些缓存页，移动页中的记录。 +> - 如果有二级索引，还会向二级索引中插入记录。 +> +> 最后还可能要改动一些系统页面，比如要修改各种段、区的统计信息，各种链表的统计信息等等。 + +所以 InnoDB 将执行语句的过程中产生的 `redo log` 划分成了若干个不可分割的组，一组 `redo log` 就是对底层页面的一次原子访问，这个原子访问也称为 `Mini-Transaction`，简称 **mtr**。一个 `mtr` 就包含一组 `redo log`，在崩溃恢复时这一组 `redo log` 就是一个不可分割的整体。 + +#### 「redo log block」 + +`redo log` 并不是一条一条写入磁盘的日志文件中的，而且一个原子操作的 `mtr` 包含一组 `redo log`，一条一条的写就无法保证写磁盘的原子性了。 + +磁盘是块设备，InnoDB 中也用 Block 的概念来读写数据，设计了一个 `redo log block` 的数据结构，称为重做日志块（`block`），重做日志块跟缓存页有点类似，只不过日志块记录的是一条条 redo log。 + +`S_FILE_LOG_BLOCK_SIZE` 等于磁盘扇区的大小 512B，每次 IO 读写的最小单位都是一个 Block。 + +一个 `redo log block` 固定 `512字节` 大小，由三个部分组成： + +- 12 字节的 **Block Header**，主要记录一些额外的信息，包括文件信息、log 版本、lsn 等 +- Block 中剩余的中间 498 个字节就是 REDO 真正内容的存放位置 + +- Block 末尾是 4 字节的 **Block Tailer**，记录当前 Block 的 Checksum，通过这个值，读取 Log 时可以明确 Block 数据有没有被完整写盘。 + +#### 「redo log 组成」 + +前置知识点说了，缓冲池有提效的功效，所以 redo log 也不是直接干到日志文件（磁盘中），而是有个类似缓冲池的 `redo log buffer`（内存中），在写 redo log 时会先写 `redo log buffer` + +> 用户态下的缓冲区数据是无法直接写入磁盘的。因为中间必须经过操作系统的内核空间缓冲区（OS Buffer）。 + +写入 `redo log buffer` 后，再写入 `OS Buffer`，然后操作系统调用 `fsync()` 函数将日志刷到磁盘。 + +![](https://img.starfish.ink/mysql/redolog-buf.png) + +> 扩展点： +> +> - redo log buffer 里面的内容，既然是在操作系统调用 fsync() 函数持久化到磁盘的，那如果事务执行期间 MySQL 发生异常重启，那这部分日志就丢了。由于事务并没有提交，所以这时日志丢了也不会有损失。 +> - fsync() 的时机不是我们控制的，那就有可能在事务还没提交的时候，redo log buffer 中的部分日志被持久化到磁盘中 +> +> 所以，redo log 是存在不同状态的 +> +> 这三种状态分别是： +> +> 1. 存在 redo log buffer 中，物理上是在 MySQL 进程内存中； +> 2. 写到磁盘 (write)，但是没有持久化（fsync)，物理上是在文件系统的 page cache 里面； +> 3. 持久化到磁盘，对应的是 hard disk。 +> +> ![](https://img.starfish.ink/mysql/redo-status.png) +> +> 日志写到 redo log buffer 是很快的，wirte 到 page cache 也差不多，但是持久化到磁盘的速度就慢多了。 +> +> 为了控制 redo log 的写入策略，InnoDB 提供了 `innodb_flush_log_at_trx_commit` 参数，它有三种可能取值（刷盘策略还会说到）： +> +> 1. 设置为 0 的时候，表示每次事务提交时都只是把 redo log 留在 redo log buffer 中 ; +> 2. 设置为 1 的时候，表示每次事务提交时都将 redo log 直接持久化到磁盘； +> 3. 设置为 2 的时候，表示每次事务提交时都只是把 redo log 写到 page cache。 +> +> InnoDB 有一个后台线程，每隔 1 秒，就会把 redo log buffer 中的日志，调用 write 写到文件系统的 page cache，然后调用 fsync 持久化到磁盘。 +> +> 注意，事务执行中间过程的 redo log 也是直接写在 redo log buffer 中的，这些 redo log 也会被后台线程一起持久化到磁盘。也就是说，一个没有提交的事务的 redo log，也是可能已经持久化到磁盘的。 + +写完 redo log buffer 后，我们就要顺序追加日志了，可是每次往哪里写，肯定需要个标识的，类似 offset，小结一下接着聊。 + + + +#### 「redo 结构小结」 + +我们把这几个 redo 内容串起来，其实就是 redo log 有 32 个文件，每个文件以 Block 为单位划分，多个文件首尾相连顺序写入 REDO 内容，Redo 又按不同类型有不同内容。 + +一个 `mtr` 中的 redo log 实际上是先写到 redo log buffer，然后再”找机会“ 将一个个 mtr 的日志记录复制到`block`中，最后在一些时机将`block`刷新到磁盘日志文件中。 + +redo 文件结构大致是下图这样： + +![](https://img.starfish.ink/mysql/redolog-flow.png) + + + +### 2.3 写入 redo log + +刚才说了，会找机会将 block 刷盘，那到底是什么时候呢？ + +#### 刷盘时机 + +写入到日志文件（刷新到磁盘）的时机有这么 3 种： + +- MySQL 正常关闭时 +- 每秒刷新一次 +- redo log buffer 剩余空间小于 1/2 时（内存不够用了，要先将脏页写到磁盘） + +每次事务提交时都将缓存在 redo log buffer 里的 redo log 直接持久化到磁盘，这个策略可由 `innodb_flush_log_at_trx_commit` 参数控制 + +#### 刷盘策略 + +`innodb_flush_log_at_trx_commit` 的值可以是 1、2、0 + +- 当设置该值为 1 时，每次事务提交都要做一次 fsync，这是最安全的配置，即使宕机也不会丢失事务，这是默认值； + +- 当设置为 2 时，则在事务提交时只做 write 操作，只保证写到系统的 page cache，因此实例 crash 不会丢失事务，但宕机则可能丢失事务； + +- 当设置为 0 时，事务提交不会触发 redo 写操作，而是留给后台线程每秒一次的刷盘操作，因此实例 crash 最多丢失 1 秒钟内的事务 + + ![](https://img.starfish.ink/mysql/redolog-flush.png) + + +#### 日志逻辑序列号（log sequence number，LSN） + +刷盘时机、刷盘策略看着好像挺合适，如果刷盘还没结束，服务器 GG（宕机）了呢？ 知道你不慌，redo 可以用来保证持久性嘛~ + +重启服务后，我们肯定需要通过 redo 重放来恢复数据，但是从哪开始恢复呢？ + +为解决这个问题 InnoDB 为 redo log 记录了序列号，这被称为 LSN（Log Sequence Number），可以理解为偏移量。 + +在 MySQL Innodb 引擎中 LSN 是一个非常重要的概念，表示从日志记录创建开始到特定的日志记录已经写入的字节数，LSN 的计算是包含每个 BLOCK 的头和尾字段的。 + +在 InnoDB 的日志系统中，LSN 无处不在，它既用于表示修改脏页时的日志序号，也用于记录 checkpoint，通过 LSN，可以具体的定位到其在 redo log 文件中的位置。 + +> 那如何由一个给定 LSN 的日志，在日志文件中找到它存储的位置的偏移量并能正确的读出来呢。所有的日志文件要属于日志组，而在 log_group_t 里的 lsn 和 lsn_offset 字段已经记录了某个日志 lsn 和其存放在文件内的偏移量之间的对应关系。我们可以利用存储在 group 内的 lsn 和给定 lsn 之间的相对位置，来计算出给定 lsn 在文件中的存储位置。(具体怎么算我们先不讨论) + +LSN 是单调递增的，用来对应 redo log 的一个个写入点。每次写入长度为 length 的 redo log， LSN 的值就会加上 length。越新的日志 LSN 越大。 + +InnoDB 用检查点（ checkpoint_lsn ）指示未被刷盘的数据从这里开始，用 lsn 指示下一个应该被写入日志的位置。不过由于有 redo log buffer 的缘故，实际被写入磁盘的位置往往比 lsn 要小。 + +redo log 采用逻辑环形结构来复用空间（循环写入），这种环形结构一般需要几个指针去配合使用 + +![](https://img.starfish.ink/mysql/redolog-lsn.png) + + + +如果 lsn 追上了 checkpoint，就意味着 **redo log 文件满了，这时 MySQL 不能再执行新的更新操作，也就是说 MySQL 会被阻塞**（*所以针对并发量大的系统，适当设置 redo log 的文件大小非常重要*），此时**会停下来将 Buffer Pool 中的脏页刷新到磁盘中，然后标记 redo log 哪些记录可以被擦除，接着对旧的 redo log 记录进行擦除，等擦除完旧记录腾出了空间，checkpoint 就会往后移动（图中顺时针）**，然后 MySQL 恢复正常运行，继续执行新的更新操作。 + +> #### 组提交（group commit） +> +> redo log 的刷盘操作将会是最终影响 MySQL TPS 的瓶颈所在。为了缓解这一问题，MySQL 使用了组提交，将多个刷盘操作合并成一个，如果说 10 个事务依次排队刷盘的时间成本是 10，那么将这 10 个事务一次性一起刷盘的时间成本则近似于 1。 +> +> 当开启 binlog 时（下边还会介绍） +> +> 为了保证 redo log 和 binlog 的数据一致性，MySQL 使用了二阶段提交，由 binlog 作为事务的协调者。而引入二阶段提交使得binlog 又成为了性能瓶颈，先前的 Redo log 组提交也成了摆设。为了再次缓解这一问题，MySQL 增加了 binlog 的组提交，目的同样是将 binlog 的多个刷盘操作合并成一个，结合 redo log 本身已经实现的组提交，分为三个阶段(Flush 阶段、Sync 阶段、Commit 阶段)完成 binlog 组提交，最大化每次刷盘的收益，弱化磁盘瓶颈，提高性能。 +> +> 通常我们说 MySQL 的“双 1”配置，指的就是 sync_binlog 和 innodb_flush_log_at_trx_commit 都设置成 1。也就是说，一个事务完整提交前，需要等待两次刷盘，一次是 redo log（prepare 阶段），一次是 binlog。 +> +> 这时候，你可能有一个疑问，这意味着我从 MySQL 看到的 TPS 是每秒两万的话，每秒就会写四万次磁盘。但是，我用工具测试出来，磁盘能力也就两万左右，怎么能实现两万的 TPS？ +> +> 解释这个问题，就要用到组提交（group commit）机制了。假设有三个并发事务 (trx1, trx2, trx3) 在 prepare 阶段，都写完 redo log buffer，持久化到磁盘的过程，对应的 LSN 分别是 50、120 和 160。 +> +> 1. trx1 是第一个到达的，会被选为这组的 leader； +> 2. 等 trx1 要开始写盘的时候，这个组里面已经有了三个事务，这时候 LSN 也变成了 160； +> 3. trx1 去写盘的时候，带的就是 LSN=160，因此等 trx1 返回时，所有 LSN 小于等于 160 的 redo log，都已经被持久化到磁盘； +> 4. 这时候 trx2 和 trx3 就可以直接返回了。 +> +> 所以，一次组提交里面，组员越多，节约磁盘 IOPS 的效果越好。 + +#### Checkpoint + +CheckPoint 的意思是检查点，用于推进 Redo Log 的失效。当触发 Checkpoint 后，会去看 Flush List 中最早的那个节点 old_lsn 是多少，也就是说当前 Flush List 还剩的最早被修改的数据页的 redo log lsn 是多少，并且将这个 lsn 记录到 Checkpoint 中，因为在这之前被修改的数据页都已经刷新到磁盘了，对应的 redo log 也就无效了，所以说之后在这个 old_lsn 之后的 redo log 才是有用的。这就解释了之前说的 redo log 文件组如何覆盖无效日志。 + + + +#### 一个重做全过程的示例 + +我们小结下 redo log 的过程 + +![](https://img.starfish.ink/mysql/redolog-write.png) + +以更新事务为例 + +1. 将原始数据读入内存，修改数据的内存副本。 + +2. 先将内存中 Buffer pool 的脏页写入到 Redo log buffer 当中**记录数据的变化**。然后再将 redo log buffer 当中记录数据变化的日志通过 **顺序IO** 刷新到磁盘的 redo log file 当中 + + > 在缓冲池中有一条 Flush 链表用来维护被修改的数据页面，也就是脏页所组成的链表。 + +3. 写入操作系统 **Page Cache** + +4. 刷盘，将重做日志缓冲区中的内容刷新到重做日志文件（如果此时脏页没刷完，会通过redo log 重放来恢复数据） + +5. 唤醒用户线程完成 commit + +6. 随后正常将内存中的脏页刷回磁盘。 + +### 2.4 redo 小结 + +有了 redo log，InnoDB 就可以保证即使数据库发生异常重启，之前提交的记录都不会丢失，这个能力称为 **crash-safe**。 + +所以有了 redo log，再通过 WAL 技术，InnoDB 就可以保证即使数据库发生异常重启，之前已提交的记录都不会丢失，这个能力称为 **crash-safe**（崩溃恢复）。可以看出来， **redo log 保证了事务四大特性中的持久性**。 + +redo log 作用： + +- **实现事务的持久性，让 MySQL 有 crash-safe 的能力**，能够保证 MySQL 在任何时间段突然崩溃，重启后之前已提交的记录都不会丢失； +- **将写操作从「随机写」变成了「顺序写」**，提升 MySQL 写入磁盘的性能。 + + + +## 三、回滚日志(undo log) + +回滚日志的作用是进行事务回滚，是 Innodb 存储引擎层生成的日志，实现了事务中的**原子性**，主要**用于事务回滚和 MVCC**。 + +当事务执行的时候，回滚日志中记录了事务中每次数据更新前的状态。当事务需要回滚的时候，可以通过读取回滚日志，恢复到指定的位置。另一方面，回滚日志也可以让其他的事务读取到这个事务对数据更改之前的值，从而确保了其他事务可以不受这个事务修改数据的影响。 + +> undo Log 是 InnoDB 十分重要的组成部分，它的作用横贯 InnoDB 中两个最主要的部分，并发控制（Concurrency Control）和故障恢复（Crash Recovery）。 +> +> - Undo Log 用来记录每次修改之前的历史值，配合 Redo Log 用于故障恢复 + +#### 3.1 为什么需要 undo log + +##### 事务回滚 + +由于如硬件故障，软件 Bug，运维操作等原因的存在，数据库在任何时刻都有突然崩溃的可能。 + +这个时候没有完成提交的事务可能已经有部分数据写入了磁盘，如果不加处理，会违反数据库对**原子性**的保证。 + +针对这个问题，直观的想法是等到事务真正提交时，才能允许这个事务的任何修改落盘，也就是 No-Steal 策略。显而易见，这种做法一方面造成很大的内存空间压力，另一方面提交时的大量随机 IO 会极大的影响性能。因此，数据库实现中通常会在正常事务进行中，就不断的连续写入 Undo Log，来记录本次修改之前的历史值。当 Crash 真正发生时，可以在 Recovery 过程中通过回放 Undo Log 将未提交事务的修改抹掉。InnoDB 采用的就是这种方式。 + +##### MVCC（Multi-Versioin Concurrency Control） + +用于 MVCC（实现非锁定读），读取一行记录时，若已被其他事务占据，则通过 undo 读取之前的版本。 + +为了避免只读事务与写事务之间的冲突，避免写操作等待读操作，几乎所有的主流数据库都采用了多版本并发控制（MVCC）的方式，也就是为每条记录保存多份历史数据供读事务访问，新的写入只需要添加新的版本即可，无需等待。 + +InnoDB 在这里复用了 Undo Log 中已经记录的历史版本数据来满足 MVCC 的需求。 + +InnoDB 中其实是把 Undo 当做一种数据来维护和使用的，也就是说，Undo Log 日志本身也像其他的数据库数据一样，会写自己对应的Redo Log，通过 Redo Log 来保证自己的原子性。因此，更合适的称呼应该是 **Undo Data**。 + + + +> 我们在执行执行一条“增删改”语句的时候，虽然没有输入 begin 开启事务和 commit 提交事务，但是 MySQL 会**隐式开启事务**来执行“增删改”语句的，执行完就自动提交事务的，这样就保证了执行完“增删改”语句后，我们可以及时在数据库表看到“增删改”的结果了。 +> +> 执行一条语句是否自动提交事务，是由 `autocommit` 参数决定的，默认是开启。所以，执行一条 update 语句也是会使用事务的。 +> + + + +#### 3.2 组织 undo log + +> 前边我们简单提过下，Undo Log 属于逻辑日志，为什么不用物理日志呢？ +> +> Undo Log 需要的是事务之间的并发，以及方便的多版本数据维护，其重放逻辑不希望因 DB 的物理存储变化而变化。因此，InnoDB 中的 Undo Log 采用了基于事务的 **Logical Logging** 的方式。 +> + +![](https://img.starfish.ink/mysql/innodb-architecture-undo.png) + +> 各个版本的 MySQL，undo tablespaces 存储有一些差距，我们以 8.0 版本说明 + +#### 「undo log 的两种类型」 + +根据行为的不同，undo log 分为两种：insert undo log 和 update undo log + +- **insert undo log，是在 insert 操作中产生的。** + + insert 操作的记录只对事务本身可见，对于其它事务此记录是不可见的，所以 insert undo log 可以在事务提交后直接删除而不需要进行purge操作。 + + Insert Undo Record 仅仅是为了可能的事务回滚准备的，并不在 MVCC 功能中承担作用。 + + ![](https://img.starfish.ink/mysql/insert-undo-log.png) + + 存在一组长度不定的 Key Fields，因为对应表的主键可能由多个 field 组成，这里需要记录 Record 完整的主键信息，回滚的时候可以通过这个信息在索引中定位到对应的 Record。 + +- **update undo log 是 update 或 delete 操作中产生。** + + 由于 MVCC 需要保留 Record 的多个历史版本，当某个 Record 的历史版本还在被使用时，这个 Record 是不能被真正的删除的。 + + 因此，当需要删除时，其实只是修改对应 Record 的Delete Mark标记。对应的，如果这时这个Record又重新插入，其实也只是修改一下Delete Mark标记，也就是将这两种情况的delete和insert转变成了update操作。再加上常规的Record修改，因此这里的Update Undo Record会对应三种Type： + + - TRX_UNDO_UPD_EXIST_REC + - TRX_UNDO_DEL_MARK_REC + - TRX_UNDO_UPD_DEL_REC。 + + 他们的存储内容也类似，我们看下 TRX_UNDO_UPD_EXIST_REC + + ![](https://img.starfish.ink/mysql/update-undo-log.png) + + 除了跟 Insert Undo Record 相同的头尾信息，以及主键 Key Fileds 之外，Update Undo Record 增加了： + + - Transaction Id记录了产生这个历史版本事务Id，用作后续MVCC中的版本可见性判断 + - Rollptr指向的是该记录的上一个版本的位置，包括space number，page number和page内的offset。沿着Rollptr可以找到一个Record的所有历史版本。 + - Update Fields中记录的就是当前这个Record版本相对于其之后的一次修改的Delta信息，包括所有被修改的Field的编号，长度和历史值。 + + > 因为会对已经存在的记录产生影响，为了提供 MVCC机制，因此 update undo log 不能在事务提交时就进行删除，而是将事务提交时放到入 history list 上，等待 purge 线程进行最后的删除操作 + > + > 为了更好的支持并发，InnoDB的多版本一致性读是采用了基于回滚段的的方式。另外，对于更新和删除操作，InnoDB并不是真正的删除原来的记录，而是设置记录的delete mark为1。因此为了解决数据Page和Undo Log膨胀的问题，需要引入purge机制进行回收 + > + > 为了保证事务并发操作时，在写各自的undo log时不产生冲突，InnoDB采用回滚段的方式来维护undo log的并发写入和持久化。回滚段实际上是一种 Undo 文件组织方式 + + + +每当 InnoDB 中需要修改某个 Record 时，都会将其历史版本写入一个 Undo Log 中，对应的 Undo Record 是 Update 类型。 + +当插入新的 Record 时，还没有一个历史版本，但为了方便事务回滚时做逆向（Delete）操作，这里还是会写入一个 Insert 类型的 Undo Record。 + +#### 「组织方式」 + +每一次的修改都会产生至少一个 Undo Record，那么大量 Undo Record 如何组织起来，来支持高效的访问和管理呢？ + +每个事务其实会修改一组的 Record，对应的也就会产生一组 Undo Record，这些 Undo Record 首尾相连就组成了这个事务的**Undo Log**。除了一个个的 Undo Record 之外，还在开头增加了一个Undo Log Header 来记录一些必要的控制信息，因此，一个 Undo Log 的结构如下所示： + +![](https://img.starfish.ink/mysql/undo-log-header.png) + +- Trx Id：事务Id +- Trx No：事务的提交顺序，也会用这个来判断是否能Purge +- Delete Mark：标明该Undo Log中有没有TRX_UNDO_DEL_MARK_REC类型的Undo Record，避免Purge时不必要的扫描 +- Log Start Offset：记录Undo Log Header的结束位置，方便之后Header中增加内容时的兼容 +- Xid and DDL Flags：Flag信息 +- Table ID if DDL：表ID +- Next Undo Log：标记后边的Undo Log +- Prev Undo Log：标记前边的Undo Log +- History List Node： + +索引中的同一个 Record 被不同事务修改，会产生不同的历史版本，这些历史版本又通过 **Rollptr** 串成一个链表，供 MVCC 使用。如下图所示： + +![](https://img.starfish.ink/mysql/undo-log-logicial.png) + +> 示例中有三个事务操作了表 t 上，主键 id 是 1 的记录，首先事务 X 插入了一条记录，事务 Y、Z 去修改了这条记录。X，Y，Z 三个事务分别有自己的逻辑上连续的三条 Undo Log，每条 Undo Log 有自己的 Undo Log Header。从索引中的这条 Record 沿着 Rollptr 可以依次找到这三个事务 Undo Log 中关于这条记录的历史版本。同时可以看出，Insert 类型 Undo Record 中只记录了对应的主键值：id=1，而 Update 类型的 Undo Record 中还记录了对应的历史版本的生成事务 Trx_id，以及被修改的 name 的历史值。 + +undo 是逻辑日志，只是将数据库**逻辑的**恢复到执行语句或事务之前。 + +我们知道 InnoDB 中默认以 块 为单位存储，一个块默认是 16KB。那么如何用固定的块大小承载不定长的 Undo Log，以实现高效的空间分配、复用，避免空间浪费。InnoDB 的**基本思路**是让多个较小的 Undo Log 紧凑存在一个 Undo Page 中，而对较大的 Undo Log 则随着不断的写入，按需分配足够多的 Undo Page 分散承载 + +![](https://img.starfish.ink/mysql/undo-log-physical.png) + +Undo 的物理组织格式是—— Undo Segment，它会持有至少一个 Undo Page。 + +InnoDB 中的 Undo 文件中准备了大量的 Undo Segment 的槽位，按照1024一组划分为**Rollback Segment**。 + +Undo 的文件组织格式是——Undo Tablespace，每个 Undo Tablespace 最多会包含 128 个 Rollback Segment。MySQL 8.0 最多支持 127 个独立的 Undo Tablespace。 + +在内存中也会维护对应的数据结构来管理 Undo Log，我们就不深入了。 + + + +#### 3.3 MVCC 是如何实现的 + +多版本的目的是为了避免写事务和读事务的互相等待，那么每个读事务都需要在不对 Record 加 Lock 的情况下， 找到对应的应该看到的历史版本。所谓历史版本就是假设在该只读事务开始的时候对整个 DB 打一个快照，之后该事务的所有读请求都从这个快照上获取。当然实现上不能真正去为每个事务打一个快照，这个时间空间成本都太高了。 + +> MVCC 的实现还有一个概念：快照读，快照信息就记录在 undo 中 +> +> 所谓快照读，就是读取的是快照数据，即快照生成的那一刻的数据，像我们常用的**普通的SELECT语句在不加锁情况下就是快照读**。如： +> +> ```mysql +> SELECT * FROM t WHERE ... +> ``` +> +> 和快照读相对应的另外一个概念叫做当前读，当前读就是读取最新数据，所以，**加锁的 SELECT，或者对数据进行增删改都会进行当前读**，比如： +> +> ```mysql +> SELECT * FROM t LOCK IN SHARE MODE; +> +> SELECT * FROM t FOR UPDATE; +> +> INSERT INTO t ... +> +> DELETE FROM t ... +> +> UPDATE t ... +> ``` + +InnoDB **通过 ReadView + undo log 实现 MVCC** + +对于「读提交」和「可重复读」隔离级别的事务来说，它们的快照读（普通 select 语句）是通过 Read View + undo log 来实现的，它们的区别在于创建 Read View 的时机不同： + +- 「读提交」隔离级别是在每个 select 都会生成一个新的 Read View，也意味着，事务期间的多次读取同一条数据，前后两次读的数据可能会出现不一致，因为可能这期间另外一个事务修改了该记录，并提交了事务。 +- 「可重复读」隔离级别是启动事务时生成一个 Read View，然后整个事务期间都在用这个 Read View，这样就保证了在事务期间读到的数据都是事务启动前的记录。 + +这两个隔离级别实现是通过「事务的 Read View 里的字段」和「记录中的两个隐藏列（trx_id 和 roll_pointer）」的比对，如果不满足可见性，就会顺着 undo log 版本链里找到满足其可见性的记录，从而控制并发事务访问同一个记录时的行为，这就叫 MVCC（多版本并发控制）。 + +InnoDB 的做法，是在读事务第一次读取的时候获取一份 ReadView，并一直持有，其中记录所有当前活跃的写事务 ID，由于写事务的 ID 是自增分配的，通过这个 ReadView 我们可以知道在这一瞬间，哪些事务已经提交哪些还在运行，根据 Read Committed 的要求，未提交的事务的修改就是不应该被看见的，对应地，已经提交的事务的修改应该被看到。 + +> **Read View 主要来帮我们解决可见性的问题的**, 即他会来告诉我们本次事务应该看到哪个快照，不应该看到哪个快照。 +> +> 在 Read View 中有几个重要的属性： +> +> - trx_ids，系统当前未提交的事务 ID 的列表。 +> - low_limit_id，未提交的事务中最大的事务 ID。 +> - up_limit_id，未提交的事务中最小的事务 ID。 +> - creator_trx_id，创建这个 Read View 的事务 ID。 +> +> 每开启一个事务，我们都会从数据库中获得一个事务 ID，这个事务 ID 是自增长的，通过 ID 大小，我们就可以判断事务的时间顺序。 + +作为存储历史版本的 Undo Record，其中记录的 trx_id 就是做这个可见性判断的，对应的主索引的 Record 上也有这个值。当一个读事务拿着自己的 ReadView 访问某个表索引上的记录时，会通过比较 Record 上的 trx_id 确定是否是可见的版本，如果不可见就沿着 Record 或 Undo Record 中记录的 rollptr 一路找更老的历史版本。 + +具体的事务 id，指向 undo log 的指针 rollptr，这些信息是放在哪里呢，这就是我们常说的 InnoDB 隐藏字段了 + +> #### InnoDB存储引擎的行结构 +> +> InnoDB 表数据的组织方式为主键聚簇索引，二级索引中采用的是(索引键值, 主键键值)的组合来唯一确定一条记录。 +> InnoDB表数据为主键聚簇索引,mysql默认为每个索引行添加了4个隐藏的字段,分别是： +> +> - DB_ROW_ID：InnoDB引擎中一个表只能有一个主键,用于聚簇索引,如果表没有定义主键会选择第一个非Null 的唯一索引作为主键,如果还没有,生成一个隐藏的DB_ROW_ID作为主键构造聚簇索引。 +> - DB_TRX_ID：最近更改该行数据的事务ID。 +> - DB_ROLL_PTR：回滚指针，指向这条记录的上一个版本，其实他指向的就是 Undo Log 中的上一个版本的快照的地址 +> - DELETE BIT：索引删除标志，如果DB删除了一条数据,是优先通知索引将该标志位设置为1,然后通过(purge)清除线程去异步删除真实的数据。 +> + +如下图所示，事务 R 需要查询表 t 上的 id 为 1 的记录，R 开始时事务 X 已经提交，事务 Y 还在运行，事务 Z 还没开始，这些信息都被记录在了事务 R 的 ReadView 中。事务 R 从索引中找到对应的这条 Record[1, stafish]，对应的 trx_id 是 Z，不可见。沿着 Rollptr 找到Undo 中的前一版本[1, fish]，对应的 trx_id 是 Y，不可见。继续沿着 Rollptr 找到[1, star]，trx_id是 X 可见，返回结果。 + +![](https://img.starfish.ink/mysql/undo-log-mvcc.png) + +前面提到过，作为 Logical Log，Undo 中记录的其实是前后两个版本的 diff 信息，而读操作最终是要获得完整的 Record 内容的，也就是说这个沿着 rollptr 指针一路查找的过程中需要用 Undo Record 中的 diff 内容依次构造出对应的历史版本，这个过程在函数 **row_search_mvcc **中，其中 **trx_undo_prev_version_build** 会根据当前的 rollptr 找到对应的 Undo Record 位置，这里如果是 rollptr指向的是 insert 类型，或者找到了已经 Purge 了的位置，说明到头了，会直接返回失败。否则，就会解析对应的 Undo Record，恢复出trx_id、指向下一条 Undo Record 的 rollptr、主键信息，diff 信息 update vector 等信息。之后通过 **row_upd_rec_in_place**，用update vector 修改当前持有的 Record 拷贝中的信息，获得 Record 的这个历史版本。之后调用自己 ReadView 的 **changes_visible** 判断可见性，如果可见则返回用户。完成这个历史版本的读取。 + + + +#### 3.4 Undo Log 清理 + +我们已经知道，InnoDB 在 Undo Log 中保存了多份历史版本来实现 MVCC，当某个历史版本已经确认不会被任何现有的和未来的事务看到的时候，就应该被清理掉。 + +InnoDB中每个写事务结束时都会拿一个递增的编号**trx_no**作为事务的提交序号，而每个读事务会在自己的ReadView中记录自己开始的时候看到的最大的trx_no为**m_low_limit_no**。那么，如果一个事务的trx_no小于当前所有活跃的读事务Readview中的这个**m_low_limit_no**，说明这个事务在所有的读开始之前已经提交了，其修改的新版本是可见的， 因此不再需要通过undo构建之前的版本，这个事务的Undo Log也就可以被清理了。 + + + +> redo log 和 undo log 区别在哪？ +> +> - redo log 记录了此次事务「**完成后**」的数据状态，记录的是更新**之后**的值； +> - undo log 记录了此次事务「**开始前**」的数据状态，记录的是更新**之前**的值； + + + +## 四、二进制日志(binlog) + +前面我们讲过，MySQL 整体来看，其实就有两块：一块是 Server 层，它主要做的是 MySQL 功能层面的事情；还有一块是引擎层，负责存储相关的具体事宜。上面我们聊到的 redo log 和 undo log 是 InnoDB 引擎特有的日志，而 Server 层也有自己的日志，称为 binlog（二进制日志）。 + +二进制日志，也被叫做「归档日志」，主要**用于数据备份和主从复制** + +- **主从复制**：在 `Master` 端开启 `binlog`，然后将 `binlog` 发送到各个 `Slave` 端，`Slave` 端重放 `binlog` 从而达到主从数据一致 +- **数据恢复**：可以用 `mysqldump` 做数据备份，binlog 格式是二进制日志，可以使用 `mysqlbinlog` 工具解析，实现数据恢复 + +二进制日志主要记录数据库的更新事件，比如创建数据表、更新表中的数据、数据更新所花费的时长等信息。通过这些信息，我们可以再现数据更新操作的全过程。而且，由于日志的延续性和时效性，我们还可以利用日志，完成无损失的数据恢复和主从服务器之间的数据同步。 + + + +### 4.1 binlog VS redolog + +是不会有点疑惑，binlog 和 redo log 是不是有点重复？这个问题跟 MySQL 的时间线有关系。 + +因为最开始 MySQL 里并没有 InnoDB 引擎。MySQL 自带的引擎是 MyISAM，但是 MyISAM 没有 crash-safe 的能力，binlog 日志只能用于归档。而 InnoDB 是另一个公司以插件形式引入 MySQL 的，既然只依靠 binlog 是没有 crash-safe 能力的，所以 InnoDB 使用另外一套日志系统——也就是 redo log 来实现 crash-safe 能力。 + +这两种日志有以下四点区别。 + +1. redo log 是 InnoDB 引擎特有的；binlog 是 MySQL 的 Server 层实现的，所有引擎都可以使用。 +2. redo log 是物理日志，记录的是“在某个数据页上做了什么修改”；binlog 是逻辑日志，记录的是这个语句的原始逻辑，比如“给 ID=2 这一行的 c 字段加 1 ”。 +3. redo log 是循环写的，空间固定会用完；binlog 是可以追加写入的。“追加写”是指 binlog 文件写到一定大小后会切换到下一个，并不会覆盖以前的日志。 +4. redo log 用于掉电等故障恢复。binlog 用于备份恢复、主从复制 + + + +### 4.2 查看 binlog + +查看二进制日志主要有 3 种情况，分别是查看当前正在写入的二进制日志、查看所有的二进制日志和查看二进制日志中的所有数据更新事件。 + +```mysql +mysql> show master status; ++---------------+----------+--------------+------------------+-------------------+ +| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set | ++---------------+----------+--------------+------------------+-------------------+ +| binlog.000002 | 27736 | | | | ++---------------+----------+--------------+------------------+-------------------+ +1 row in set (0.00 sec) +``` + +```mysql +mysql> show binary logs; ++---------------+-----------+-----------+ +| Log_name | File_size | Encrypted | ++---------------+-----------+-----------+ +| binlog.000001 | 638 | No | +| binlog.000002 | 27736 | No | ++---------------+-----------+-----------+ +2 rows in set (0.01 sec) +``` + +```mysql +mysql> show variables like '%binlog_format%'; ++---------------+-------+ +| Variable_name | Value | ++---------------+-------+ +| binlog_format | ROW | ++---------------+-------+ +1 row in set (0.00 sec) +``` + + + +### 4.3 Binary logging formats + +`binlog`日志有三种格式，分别为`STATMENT`、`ROW`和`MIXED`。 + +> 在 `MySQL 5.7.7`之前，默认的格式是`STATEMENT`，`MySQL 5.7.7`之后，默认值是 `ROW`。日志格式通过 `binlog-format` 指定。 + +- `STATMENT` ：基于 SQL 语句的复制(`statement-based replication, SBR`)，每一条会修改数据的 sql 语句会记录到 binlog 中**。** + - 优点：不需要记录每一行的变化，减少了 binlog 日志量，节约了 IO, 从而提高了性能； + - 缺点：在某些情况下会导致主从数据不一致，比如执行`sysdate()`、`slepp()`等。 + +- `ROW` ：基于行的复制(`row-based replication, RBR`)，不记录每条sql语句的上下文信息，仅需记录哪条数据被修改了**。 ** + - 优点：不会出现某些特定情况下的存储过程、或function、或trigger的调用和触发无法被正确复制的问题**；** + - 缺点：会产生大量的日志，尤其是 `alter table` 的时候会让日志暴涨 + +- `MIXED` ：基于 STATMENT 和 ROW 两种模式的混合复制(`mixed-based replication, MBR`)，mixed 格式的意思是，MySQL 自己会判断这条 SQL 语句是否可能引起主备不一致，如果有可能，就用 row 格式，否则就用 statement 格式 + +> ``` +>SET GLOBAL binlog_format = 'STATEMENT'; +> SET GLOBAL binlog_format = 'ROW'; +> SET GLOBAL binlog_format = 'MIXED'; +> ``` +> + + + +### 4.4 binlog 的写入机制 + +binlog 的写入逻辑比较简单：事务执行过程中，先把日志写到 binlog cache，事务提交的时候，再把 binlog cache 写到 binlog 文件中。 + +一个事务的 binlog 是不能被拆开的，因此不论这个事务多大，也要确保一次性写入。这就涉及到了 binlog cache 的保存问题。 + +系统给 binlog cache 分配了一片内存，每个线程一个，参数 `binlog_cache_size` 用于控制单个线程内 binlog cache 所占内存的大小。如果超过了这个参数规定的大小，就要暂存到磁盘。 + +事务提交的时候，执行器把 binlog cache 里的完整事务写入到 binlog 中，并清空 binlog cache。 + +![](https://img.starfish.ink/mysql/binlog-cache.png) + + + +可以看到，每个线程有自己 binlog cache，但是共用同一份 binlog 文件。 + +- 图中的 write，指的就是指把日志写入到文件系统的 page cache，并没有把数据持久化到磁盘，所以速度比较快。 +- 图中的 fsync，才是将数据持久化到磁盘的操作。一般情况下，我们认为 fsync 才占磁盘的 IOPS。 + +write 和 fsync 的时机，是由参数 sync_binlog 控制的： + +1. sync_binlog=0 的时候，表示每次提交事务都只 write，不 fsync； +2. sync_binlog=1 的时候，表示每次提交事务都会执行 fsync； +3. sync_binlog=N(N>1) 的时候，表示每次提交事务都 write，但累积 N 个事务后才 fsync。 + +因此，在出现 IO 瓶颈的场景里，将 sync_binlog 设置成一个比较大的值，可以提升性能。在实际的业务场景中，考虑到丢失日志量的可控性，一般不建议将这个参数设成 0，比较常见的是将其设置为 100~1000 中的某个数值。 + +但是，将 sync_binlog 设置为 N，对应的风险是：如果主机发生异常重启，会丢失最近 N 个事务的 binlog 日志。 + + + +### 4.5 update 语句执行过程 + +比较重要的 undo、redo、binlog 都介绍完了，我们来看执行器和 InnoDB 引擎在执行一个简单的 update 语句时的内部流程。`update t set name='starfish' where id = 1;` + +1. 执行器先找引擎取 id=1 这一行。id 是主键，引擎直接用树搜索找到这一行。如果 id=1 这一行所在的数据页本来就在内存（buffer pool）中，就直接返回给执行器；否则，需要先从磁盘读入内存，然后再返回。 +2. 执行器拿到引擎给的行数据，更新行数据，再调用引擎接口写入这行新数据。 +3. 引擎将这行新数据更新到内存中，同时将这个更新操作记录到 redo log 里面，此时 redo log 处于 prepare 状态。然后告知执行器执行完成了，随时可以提交事务。 +4. 执行器生成这个操作的 binlog，并把 binlog 写入磁盘。 +5. 执行器调用引擎的提交事务接口，引擎把刚刚写入的 redo log 改成提交（commit）状态，更新完成。 + +这里我给出这个 update 语句的执行流程图，图中浅色框表示是在 InnoDB 内部执行的，深色框表示是在执行器中执行的。 + +![](https://img.starfish.ink/mysql/update-flow.png) + +你可能注意到了，最后三步看上去有点“绕”，将 redo log 的写入拆成了两个步骤：prepare 和 commit，这就是"两阶段提交"。 + +### 4.6 两阶段提交 + +为什么必须有“两阶段提交”呢？这是为了让两份日志之间的逻辑一致。要说明这个问题，我们得从文章开头的那个问题说起：**怎样让数据库恢复到半个月内任意一秒的状态？** + +前面我们说过了，binlog 会记录所有的逻辑操作，并且是采用“追加写”的形式。如果你的 DBA 承诺说半个月内可以恢复，那么备份系统中一定会保存最近半个月的所有 binlog，同时系统会定期做整库备份。这里的“定期”取决于系统的重要性，可以是一天一备，也可以是一周一备。 + +当需要恢复到指定的某一秒时，比如某天下午两点发现中午十二点有一次误删表，需要找回数据，那你可以这么做： + +- 首先，找到最近的一次全量备份，如果你运气好，可能就是昨天晚上的一个备份，从这个备份恢复到临时库； +- 然后，从备份的时间点开始，将备份的 binlog 依次取出来，重放到中午误删表之前的那个时刻。 + +这样你的临时库就跟误删之前的线上库一样了，然后你可以把表数据从临时库取出来，按需要恢复到线上库去。 + +好了，说完了数据恢复过程，我们回来说说，为什么日志需要“两阶段提交”。这里不妨用反证法来进行解释。 + +由于 redo log 和 binlog 是两个独立的逻辑，如果不用两阶段提交，要么就是先写完 redo log 再写 binlog，或者采用反过来的顺序。我们看看这两种方式会有什么问题。 + +仍然用前面的 update 语句来做例子。假设当前 ID=2 的行，字段 c 的值是 0，再假设执行 update 语句过程中在写完第一个日志后，第二个日志还没有写完期间发生了 crash，会出现什么情况呢？ + +1. **先写 redo log 后写 binlog**。假设在 redo log 写完，binlog 还没有写完的时候，MySQL 进程异常重启。由于我们前面说过的，redo log 写完之后，系统即使崩溃，仍然能够把数据恢复回来，所以恢复后这一行 c 的值是 1。 + 但是由于 binlog 没写完就 crash 了，这时候 binlog 里面就没有记录这个语句。因此，之后备份日志的时候，存起来的 binlog 里面就没有这条语句。 + 然后你会发现，如果需要用这个 binlog 来恢复临时库的话，由于这个语句的 binlog 丢失，这个临时库就会少了这一次更新，恢复出来的这一行 c 的值就是 0，与原库的值不同。 +2. **先写 binlog 后写 redo log**。如果在 binlog 写完之后 crash，由于 redo log 还没写，崩溃恢复以后这个事务无效，所以这一行 c 的值是 0。但是 binlog 里面已经记录了“把 c 从 0 改成 1”这个日志。所以，在之后用 binlog 来恢复的时候就多了一个事务出来，恢复出来的这一行 c 的值就是 1，与原库的值不同。 + +可以看到，如果不使用“两阶段提交”，那么数据库的状态就有可能和用它的日志恢复出来的库的状态不一致。 + +你可能会说，这个概率是不是很低，平时也没有什么动不动就需要恢复临时库的场景呀？ + +其实不是的，不只是误操作后需要用这个过程来恢复数据。当你需要扩容的时候，也就是需要再多搭建一些备库来增加系统的读能力的时候，现在常见的做法也是用全量备份加上应用 binlog 来实现的，这个“不一致”就会导致你的线上出现主从数据库不一致的情况。 + +简单说，redo log 和 binlog 都可以用于表示事务的提交状态，而两阶段提交就是让这两个状态保持逻辑上的一致。 + + + +### 4.7 主从同步 + +MySQL主从同步的作用主要有以下几点： + +- 故障切换。 +- 提供一定程度上的备份服务。 +- 实现MySQL数据库的读写分离。 + +MySQL 的主从复制依赖于 binlog ，也就是记录 MySQL 上的所有变化并以二进制形式保存在磁盘上。复制的过程就是将 binlog 中的数据从主库传输到从库上。 + +这个过程一般是**异步**的，也就是主库上执行事务操作的线程不会等待复制 binlog 的线程同步完成。 + +![](https://img.starfish.ink/mysql/master-slave.png) + +具体详细过程如下： + +- MySQL 主库在收到客户端提交事务的请求之后，会先写入 binlog，再提交事务，更新存储引擎中的数据，事务提交完成后，返回给客户端“操作成功”的响应。 +- 从库会创建一个专门的 I/O 线程，连接主库的 log dump 线程，来接收主库的 binlog 日志，再把 binlog 信息写入 relay log 的中继日志里，再返回给主库“复制成功”的响应。 +- 从库会创建一个用于回放 binlog 的线程，去读 relay log 中继日志，然后回放 binlog 更新存储引擎中的数据，最终实现主从的数据一致性。 + +在完成主从复制之后，你就可以在写数据时只写主库，在读数据时只读从库，这样即使写请求会锁表或者锁记录，也不会影响读请求的执行。 + +> #### 中继日志(relay log) +> +> 中继日志只在主从服务器架构的从服务器上存在。从服务器为了与主服务器保持一致，要从主服务器读取二进制日志的内容，并且把读取到的信息写入本地的日志文件中，这个从服务器本地的日志文件就叫中继日志。然后，从服务器读取中继日志，并根据中继日志的内容对从服务器的数据进行更新，完成主从服务器的数据同步。 +> +> Relay log(中继日志)是在MySQL主从复制时产生的日志，在MySQL的主从复制主要涉及到三个线程: +> +> 1) Log dump线程：向从库的IO线程传输主库的Binlog日志 +> +> 2) IO线程：向主库请求Binlog日志，并将Binlog日志写入到本地的relay log中。 +> +> 3) SQL线程：读取Relay log日志，将其解析为SQL语句并逐一执行。 +> +> + +> MySQL 主从复制还有哪些模型？ + +主要有三种： + +- **同步复制**：MySQL 主库提交事务的线程要等待所有从库的复制成功响应，才返回客户端结果。这种方式在实际项目中，基本上没法用，原因有两个：一是性能很差，因为要复制到所有节点才返回响应；二是可用性也很差，主库和所有从库任何一个数据库出问题，都会影响业务。 +- **异步复制**（默认模型）：MySQL 主库提交事务的线程并不会等待 binlog 同步到各从库，就返回客户端结果。这种模式一旦主库宕机，数据就会发生丢失。 +- **半同步复制**：MySQL 5.7 版本之后增加的一种复制方式，介于两者之间，事务线程不用等待所有的从库复制成功响应，只要一部分复制成功响应回来就行，比如一主二从的集群，只要数据成功复制到任意一个从库上，主库的事务线程就可以返回给客户端。这种**半同步复制的方式，兼顾了异步复制和同步复制的优点，即使出现主库宕机，至少还有一个从库有最新的数据，不存在数据丢失的风险**。 + + + +## 五、错误日志(error log) + +错误日志记录了 MySQL 服务器启动、停止运行的时间，以及系统启动、运行和停止过程中的诊断信息，包括错误、警告和提示等。当我们的数据库服务器发生系统故障时，错误日志是发现问题、解决故障的首选。 + +错误日志默认是开启的 + +```mysql +mysql> show variables like '%log_error%'; ++----------------------------+----------------------------------------+ +| Variable_name | Value | ++----------------------------+----------------------------------------+ +| binlog_error_action | ABORT_SERVER | +| log_error | /usr/local/mysql/data/mysqld.local.err | +| log_error_services | log_filter_internal; log_sink_internal | +| log_error_suppression_list | | +| log_error_verbosity | 2 | ++----------------------------+----------------------------------------+ +5 rows in set (0.01 sec) +``` + +我们可以看到错误日志的地址，当出现数据库不能正常启动、使用的时候，第一个查的就是错误日志，有时候错误日志中也会有些优化信息，比如告诉我们需要增大 InnoDB 引擎的 redo log 这种。 + + + +## 六、慢查询日志(slow query log) + +慢查询日志用来记录执行时间超过指定时长的查询。它的主要作用是，帮助我们发现那些执行时间特别长的 SQL 查询，并且有针对性地进行优化，从而提高系统的整体效率。当我们的数据库服务器发生阻塞、运行变慢的时候，检查一下慢查询日志，找到那些慢查询，对解决问题很有帮助。 + +```mysql +mysql> show variables like '%slow_query%'; ++---------------------+------------------------------------------------------+ +| Variable_name | Value | ++---------------------+------------------------------------------------------+ +| slow_query_log | OFF | +| slow_query_log_file | /usr/local/mysql/data/starfishdeMacBook-Pro-slow.log | ++---------------------+------------------------------------------------------+ +2 rows in set (0.02 sec) +``` + +默认也是关闭的，其实我们说的慢查询日志，有两个值 + +> Mac 没有 my.ini/my.cnf 文件，需要自己搞，我们只对本次生效吧。`set global slow_query_log=1` + +```mysql +mysql> show variables like '%long_query_time%'; ++-----------------+-----------+ +| Variable_name | Value | ++-----------------+-----------+ +| long_query_time | 10.000000 | ++-----------------+-----------+ +1 row in set (0.01 sec) + +mysql> show variables like '%row_limit%'; ++------------------------+-------+ +| Variable_name | Value | ++------------------------+-------+ +| min_examined_row_limit | 0 | ++------------------------+-------+ +1 row in set (0.01 sec) +``` + +`long_query_time`：慢查询的时间阈值 + +`min_examined_row_limit`：查询扫描过的最少记录数，因为这个值默认是 0，所以常被我们忽略 + +查询的执行时间和最少扫描记录，共同组成了判别一个查询是否是慢查询的条件。如果查询扫描过的记录数大于等于这个变量的值，并且查询执行时间超过 long_query_time 的值，那么，这个查询就被记录到慢查询日志中；反之，则不被记录到慢查询日志中。 + + + +## 小结 + +感谢你读到这里，送你两道面试题吧 + +### 说下 一条 MySQL 更新语句的执行流程是怎样的吧？ + +```mysql +mysql> update t set name='starfish' where salary > 999999; +``` + +server 层和 InnoDB 层之间是如何沟通： + +1. salary 有二级索引，行器先找引擎取扫描区间的第一行。根据这条二级索引记录中的主键值执行回表操作（即通过聚簇索引的B+树根节点一层一层向下找，直到在叶子节点中找到相应记录），将获取到的聚簇索引记录返回给 server 层。 +2. server 层得到聚簇索引记录后，会看一下更新前的记录和更新后的记录是否一样，如果一样的话就不更新了，如果不一样的话就把更新前的记录和更新后的记录都当作参数传给 InnoDB 层，让 InnoDB 真正的执行更新记录的操作 +3. InnoDB 收到更新请求后，先更新记录的聚簇索引记录，再更新记录的二级索引记录。最后将更新结果返回给 server 层 +4. server 层继续向 InnoDB 索要下一条记录，由于已经通过 B+ 树定位到二级索引扫描区间 `[999999, +∞)` 的第一条二级索引记录，而记录又是被串联成单向链表，所以 InnoDB 直接通过记录头信息的 `next_record` 的属性即可获取到下一条二级索引记录。然后通过该二级索引的主键值进行回表操作，获取到完整的聚簇索引记录再返回给 server 层。 +5. 就这样一层一层的处理 + +具体执行流程： + +1. 先在 B+ 树中定位到该记录（这个过程也被称作**加锁读**），如果该记录所在的页面不在 buffer pool 里，先将其加载到 buffer pool 里再读取。 + +2. 首先更新聚簇索引记录。 更新聚簇索引记录时： + + ① 先向 Undo 页面写 undo 日志。不过由于这是在更改页面，所以修改 Undo 页面前需要先记录一下相应的 redo 日志。 + + ② 将这个更新操作记录到 redo log 里面，此时 redo log 处于 prepare 状态。然后告知执行器执行完成了，随时可以提交事务。 + + > 这里可以会有点疑惑。我们可以直接理解成先写 undo 再写 redo，这里修改后的页面并没有加入 buffer pool 的 flush 链表，记录的 redo 日志也没有加入到 redo log buffer。当这个函数执行完后，才会：先将这个过程产生的 redo 日志写入到 redo log buffer，再将这个过程修改的页面加入到 buffer pool 的 flush 链表中。 + +3. 更新其他的二级索引记录。 + + > 更新二级索引记录时不会再记录 undo 日志，但由于是在修改页面内容，会先记录相应的 redo 日志。 + +4. 记录该语句对应的 binlog 日志，此时记录的 binlog 并没有刷新到硬盘上的 binlog 日志文件，在事务提交时才会统一将该事务运行过程中的所有 binlog 日志刷新到硬盘。 + +5. 引擎把刚刚写入的 redo log 改成提交（commit）状态，更新完成。 + +### 日志执行顺序？ + +时序上先 undo log，redo log 先 prepare， 再写 binlog，最后再把 redo log commit + +![](https://img.starfish.ink/mysql/log-seq.png) + +### 为什么需要记录 REDO + +redo log 是 Innodb 存储引擎层生成的日志，实现了事务中的**持久性**，主要**用于掉电等故障恢复**： + +1. 在系统遇到故障的恢复过程中，可以修复未完成的事务修改的数据。 +2. InnoDB 为了提高数据存取的效率，减少磁盘操作的频率，对数据的更新操作不会立即写到磁盘上，而是把数据更新先保存在内存中(**InnoDB Buffer Pool**)，积累到一定程度，再集中进行磁盘读写操作。这样就存在一个问题：一旦出现宕机或者停电等异常情况，内存中保存的数据更新操作可能会丢失。为了保证数据库本身的一致性和**持久性**，InnoDB 维护了 REDO LOG。修改 Page 之前需要先将修改的内容记录到 REDO 中，并保证 REDO LOG 早于对应的 Page 落盘，也就是常说的 WAL。当故障发生导致内存数据丢失后，InnoDB 会在重启时，通过重放 REDO，将 Page 恢复到崩溃前的状态。 + +回答面试官问题时候，如果能指明不同版本的差异，会加分的 + + + +## References + +- https://dev.mysql.com/doc/refman/8.0/en/server-logs.html +- https://dev.mysql.com/doc/dev/mysql-server/latest/PAGE_INNODB_REDO_LOG_THREADS.html +- https://dev.mysql.com/doc/refman/8.0/en/innodb-redo-log.html +- [《庖丁解InnoDB之UNDO LOG》](http://mysql.taobao.org/monthly/2021/10/01/) +- [庖丁解InnoDB之Undo LOG](http://catkang.github.io/2021/10/30/mysql-undo.html) +- https://www.51cto.com/article/639652.html \ No newline at end of file diff --git a/docs/data-management/MySQL/MySQL-Master-Slave.md b/docs/data-management/MySQL/MySQL-Master-Slave.md new file mode 100644 index 0000000000..fb7ba2e62f --- /dev/null +++ b/docs/data-management/MySQL/MySQL-Master-Slave.md @@ -0,0 +1,13 @@ +TODO + + + +![](https://img.starfish.ink/mysql/log-seq.png) + +备库 B 跟主库 A 之间维持了一个长连接。主库 A 内部有一个线程，专门用于服务备库 B 的这个长连接。一个事务日志同步的完整过程是这样的： + +1. 在备库 B 上通过 change master 命令，设置主库 A 的 IP、端口、用户名、密码，以及要从哪个位置开始请求 binlog，这个位置包含文件名和日志偏移量。 +2. 在备库 B 上执行 start slave 命令，这时候备库会启动两个线程，就是图中的 io_thread 和 sql_thread。其中 io_thread 负责与主库建立连接。 +3. 主库 A 校验完用户名、密码后，开始按照备库 B 传过来的位置，从本地读取 binlog，发给 B。 +4. 备库 B 拿到 binlog 后，写到本地文件，称为中转日志（relay log）。 +5. sql_thread 读取中转日志，解析出日志里的命令，并执行。 \ No newline at end of file diff --git a/docs/data-management/MySQL/MySQL-Optimization.md b/docs/data-management/MySQL/MySQL-Optimization.md new file mode 100644 index 0000000000..89292e3b0e --- /dev/null +++ b/docs/data-management/MySQL/MySQL-Optimization.md @@ -0,0 +1,689 @@ +--- +title: MySQL 优化 +date: 2024-05-09 +tags: + - MySQL +categories: MySQL +--- + +![](https://img.starfish.ink/mysql/banner-mysql-optimization.png) + +> 《高性能MySQL》给出的性能定义：完成某件任务所需要的的时间度量，性能既响应时间。 +> +> 我们主要探讨 Select 的优化，包括 MySQL Server 做了哪些工作以及我们作为开发，如何定位问题，以及如何优化，怎么写出高性能 SQL + + + +## 一、MySQL Server 优化了什么 + +### MySQL Query Optimizer + +MySQL 中有专门负责优化 SELECT 语句的优化器模块，主要功能：通过计算分析系统中收集到的统计信息，为客户端请求的 Query 提供他认为最优的执行计划（他认为最优的数据检索方式，但不见得是DBA认为是最优的，这部分最耗费时间） + +当客户端向 MySQL 请求一条 Query，命令解析器模块完成请求分类，区别出是 SELECT 并转发给 MySQL Query Optimizer 时，MySQL Query Optimizer 首先会对整条 Query 进行优化，处理掉一些常量表达式的预算，直接换算成常量值。并对 Query 中的查询条件进行简化和转换，如去掉一些无用或显而易见的条件、结构调整等。然后分析 Query 中的 Hint 信息（如果有），看显示 Hint 信息是否可以完全确定该 Query 的执行计划。如果没有 Hint 或Hint 信息还不足以完全确定执行计划，则会读取所涉及对象的统计信息，根据 Query 进行写相应的计算分析，然后再得出最后的执行计划。 + +MySQL 查询优化器是一个复杂的组件，它的主要任务是确定执行给定查询的最优方式。以下是 MySQL 查询优化器在处理查询时所进行的一些关键活动： + +#### 1.1 解析查询： + +优化器首先解析查询语句，理解其语法和语义。 + +#### 1.2 词法和语法分析： + +检查查询语句是否符合SQL语法规则。 + +#### 1.3 语义分析： + +确保查询引用的所有数据库对象（如表、列、别名等）都是存在的，并且用户具有相应的访问权限。 + +#### 1.4 查询重写： + +可能对查询进行一些变换，以提高其效率。例如，使用等价变换简化查询或应用数据库的视图定义。 + +#### 1.5 确定执行计划： + +优化器会生成一个或多个可能的执行计划，并估算每个计划的成本（如I/O操作、CPU使用等）。 + +#### 1.6 选择最佳执行计划： + +执行成本包括 I/O 成本和 CPU 成本。MySQL 有一套自己的计算公式，在一条单表查询语句真正执行之前，MySQL 的查询优化器会找出执行该语句所有可能使用的方案，对比之后找出成本最低的方案，这个成本最低的方案就是所谓的`执行计划`，之后才会调用存储引擎提供的接口真正的执行查询。 + +#### 1.7 索引选择： + +确定是否使用索引以及使用哪个索引。考虑因素包括索引的选择性、查询条件、索引的前缀等。 + +#### 1.8 表访问顺序： + +对于涉及多个表的查询，优化器决定最佳的表访问顺序，以减少数据的访问量。 + +#### 1.9 连接算法选择： + +join 我们每天都在用，左、右连接、内连接就不详细介绍了 + +![img](https://res.cloudinary.com/practicaldev/image/fetch/s--c4weS2Kt--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rscdema0dwgx4n91eg0p.png) + +对于连接操作，优化器会选择最合适的算法，如嵌套循环、块嵌套循环、哈希连接等。 + +- **嵌套循环连接（Nested-Loop Join）**：驱动表只访问一次，但被驱动表却可能被多次访问，访问次数取决于对驱动表执行单表查询后的结果集中的记录条数的连接执行方式称之为`嵌套循环连接`。 + + 左（外）连接的驱动表就是左边的那个表，右（外）连接的驱动表就是右边的那个表。内连接驱动表就无所谓了。 + +- **基于块的嵌套循环连接（Block Nested-Loop Join）**: 块嵌套循环连接是嵌套循环连接的优化版本。每次访问被驱动表，被驱动表的记录会被加载到内存中，与驱动表匹配，然后清理内存，然后再取下一条，这样 I/O 成本是超级高的。 + + 所以为了减少了对被驱动表的访问次数，引入了 `join buffer` 的概念，执行连接查询前申请的一块固定大小的内存，先把若干条驱动表结果集中的记录装在这个`join buffer`中，然后开始扫描被驱动表，每一条被驱动表的记录一次性和`join buffer`中的多条驱动表记录做匹配，因为匹配的过程都是在内存中完成的，所以这样可以显著减少被驱动表的`I/O`代价。 + +#### 1.10 子查询优化： + +对于子查询，优化器决定是将其物化、转换为半连接、还是其他形式。 + +#### 1.11 谓词下推： + +将查询条件（谓词）尽可能地下推到存储引擎层面，以便尽早过滤数据。 + +#### 1.12 分区修剪： + +如果表被分区，优化器会识别出只需要扫描的分区。 + +#### 1.13 排序和分组优化： + +优化器会考虑使用索引来执行排序和分组操作。 + +#### 1.14 临时表和物化： + +优化器可能会决定使用临时表来存储中间结果，以简化查询。 + +#### 1.15 并行查询执行： + +对于某些查询，优化器可以决定使用并行执行来提高性能。 + +#### 1.16 执行计划缓存： + +如果可能，优化器会重用之前缓存的执行计划，以减少解析和优化的开销。 + +#### 1.17 生成执行语句： + +最终，优化器生成用于执行查询的底层指令。 + +#### 1.18 监控和调整： + +优化器的行为可以通过各种参数进行调整，以适应特定的工作负载和系统配置。 + +#### 1.19 统计信息更新： + +优化器依赖于表和索引的统计信息来做出决策，因此需要确保统计信息是最新的。 + +InnoDB存储引擎的统计数据收集是数据库性能优化的重要组成部分，因为这些统计数据会被MySQL查询优化器用来生成查询执行计划。以下是InnoDB统计数据收集的一些关键点： + +1. **统计数据存储方式**： + - InnoDB提供了两种存储统计数据的方式：永久性统计数据和非永久性统计数据。 + - 永久性统计数据存储在磁盘上，服务器重启后依然存在。 + - 非永久性统计数据存储在内存中，服务器关闭时会被清除。 +2. **系统变量控制**： + - `innodb_stats_persistent`：控制是否使用永久性统计数据，默认在MySQL 5.6.6之后的版本中为ON。 + - `innodb_stats_persistent_sample_pages`：控制永久性统计数据采样的页数，默认值为2012。 + - `innodb_stats_transient_sample_pages`：控制非永久性统计数据采样的页数，默认值为82。 +3. **统计信息的更新**： + - `ANALYZE TABLE`：可以用于手动更新统计信息，它将重新计算表的统计数据3。 + - `innodb_stats_auto_recalc`：控制是否自动重新计算统计数据，默认为ON24。 +4. **统计数据的收集**： + - InnoDB通过采样页面来估计表中的行数和其他统计信息24。 + - `innodb_table_stats`和`innodb_index_stats`：这两个内部表存储了关于表和索引的统计数据24。 +5. **特定表的统计数据属性**： + - 在创建或修改表时，可以通过`STATS_PERSISTENT`、`STATS_AUTO_RECALC`和`STATS_SAMPLE_PAGES`属性来控制表的统计数据行为234。 +6. **NULL值的处理**： + - `innodb_stats_method`变量决定了在统计索引列不重复值的数量时如何对待NULL值310。 +7. **手动更新统计数据**： + - 可以手动更新`innodb_table_stats`和`innodb_index_stats`表中的统计数据，之后需要使用`FLUSH TABLE`命令让优化器重新加载统计信息4。 +8. **非永久性统计数据**： + - 当`innodb_stats_persistent`设置为OFF时，新创建的表将使用非永久性统计数据，这些数据存储在内存中45。 +9. **统计数据的自动更新**： + - 如果表中数据变动超过一定比例（默认10%），并且`innodb_stats_auto_recalc`为ON，InnoDB将自动更新统计数据34。 + +通过以上信息，我们了解到 InnoDB 的统计数据收集是一个动态的过程，旨在帮助优化器做出更好的查询执行计划决策。数据库管理员可以根据系统的具体需求和性能指标来调整相关的系统变量，以优化统计数据的收集和使用。 + +> 问个问题：为什么 InnoDB `rows`这个统计项的值是估计值呢？ +> +> `InnoDB`统计一个表中有多少行记录的套路大概是这样的：按照一定算法（并不是纯粹随机的）选取几个叶子节点页面，计算每个页面中主键值记录数量，然后计算平均一个页面中主键值的记录数量乘以全部叶子节点的数量就算是该表的`n_rows`值。 + +通过`EXPLAIN`或`EXPLAIN ANALYZE`命令可以查看查询优化器的执行计划，这有助于理解查询的执行方式，并据此进行优化。优化器的目标是找到最快、最高效的执行计划，但有时它也可能做出不理想的决策，特别是在数据量变化或统计信息不准确时。在这种情况下，可以通过调整索引、修改查询或使用SQL提示词来引导优化器做出更好的选择。 + + + +## 二、业务开发者可以优化什么 + +![](https://miro.medium.com/v2/resize:fit:1002/1*cegOtzJsnsPmLxZWHU1gWg.png) + + + + + +假设性能优化就是在一定负载下尽可能的降低响应时间。那我们作为一名业务开发，想优化 MySQL，一般就是优化 CRUD 性能，那优化的前提肯定是比较烂，才优化，要么是服务器或者网络烂，要么是 DB 设计的烂，当然更多的一般是 SQL 写的烂。 + +![](https://image.dbbqb.com/202405181617/c383939ea6ba00933fd84e2989230659/pXOzq) + +### 2.1 影响 MySQL 的性能因素 | 常见瓶颈 + +- ##### 硬件资源 + + - CPU：CPU 的性能直接影响到 MySQL 的计算速度。多核 CPU 可以提高并发处理能力 + + - 内存：足够的内存可以有效减少磁盘 I/O，提高缓存效率 + + - 磁盘：磁盘的 I/O 性能对数据库读写速度有显著影响，特别是对于读密集型操作。比如装入数据远大于内存容量的时候，磁盘 I/O 就会达到瓶颈 + +- ##### 网络 + +​ 网络带宽和延迟也会影响分布式数据库或应用服务器与数据库服务器之间的通信效率 + +- ##### DB 设计 + + - 存储引擎的选择 + - 参数配置，如 `innodb_buffer_pool_size`，对数据库性能有决定性影响 + +- ##### 连接数和线程管理： + + - 高并发时，连接数和线程的高效管理对性能至关重要 + +- ##### 数据库设计 + + - 合理的表结构设计、索引优化、数据类型选择等都会影响性能 + - 比如哪种超大文本、二进制媒体数据啥的就别往 MySQL 放了 + +- ##### 开发的技术水平 + + - 开发的水平对性能的影响，查询语句写的烂，比如不管啥上来就 `SELECT *`，一个劲的 `left join` + - 索引失效（单值 复合） + +当然锁、长事务这类使用中的坑，会对性能造成影响，我也举不全。 + + + +### 2.2 性能分析 + +#### 先查外忧 + +外忧，就是我们业务开发，一般情况下不用解决，或者一般这锅背不到我们头上的问题，比如硬件、网络这种 + +> 查看 Linux 系统性能的常用命令 +> +> MySQL 数据库是常见的两个瓶颈是 CPU 和 I/O 的瓶颈。CPU 在饱和的时候一般发生在数据装入内存或从磁盘上读取数据时候，磁盘 I/O 瓶颈发生在装入数据远大于内存容量的时候，如果应用分布在网络上，那么查询量相当大的时候那么瓶颈就会出现在网络上。Linux 中我们常用 mpstat、vmstat、iostat、sar 和 top 来查看系统的性能状态。 +> +> `mpstat`： mpstat 是 Multiprocessor Statistics 的缩写，是实时系统监控工具。其报告为 CPU 的一些统计信息，这些信息存放在 /proc/stat 文件中。在多CPUs系统里，其不但能查看所有CPU的平均状况信息，而且能够查看特定CPU的信息。mpstat最大的特点是可以查看多核心cpu中每个计算核心的统计数据，而类似工具vmstat只能查看系统整体cpu情况。 +> +> `vmstat`：vmstat 命令是最常见的 Linux/Unix 监控工具，可以展现给定时间间隔的服务器的状态值，包括服务器的 CPU 使用率，内存使用，虚拟内存交换情况，IO 读写情况。这个命令是我查看 Linux/Unix 最喜爱的命令，一个是 Linux/Unix 都支持，二是相比top，我可以看到整个机器的CPU、内存、IO的使用情况，而不是单单看到各个进程的CPU使用率和内存使用率(使用场景不一样)。 +> +> `iostat`: 主要用于监控系统设备的 IO 负载情况，iostat 首次运行时显示自系统启动开始的各项统计信息，之后运行 iostat 将显示自上次运行该命令以后的统计信息。用户可以通过指定统计的次数和时间来获得所需的统计信息。 +> +> `sar`： sar（System Activity Reporter系统活动情况报告）是目前 Linux 上最为全面的系统性能分析工具之一，可以从多方面对系统的活动进行报告，包括：文件的读写情况、系统调用的使用情况、磁盘 I/O、CPU效率、内存使用状况、进程活动及 IPC 有关的活动等。 +> +> `top`：top 命令是 Linux 下常用的性能分析工具，能够实时显示系统中各个进程的资源占用状况，类似于 Windows 的任务管理器。top 显示系统当前的进程和其他状况，是一个动态显示过程,即可以通过用户按键来不断刷新当前状态.如果在前台执行该命令，它将独占前台，直到用户终止该程序为止。比较准确的说，top 命令提供了实时的对系统处理器的状态监视。它将显示系统中 CPU 最“敏感”的任务列表。该命令可以按 CPU 使用。内存使用和执行时间对任务进行排序；而且该命令的很多特性都可以通过交互式命令或者在个人定制文件中进行设定。 + +除了服务器硬件的性能瓶颈，对于 MySQL 系统本身，我们可以使用工具来优化数据库的性能，通常有三种：使用索引，使用 EXPLAIN 分析查询以及调整 MySQL 的内部配置。 + + + +#### 再定内患 | MySQL 常见性能分析手段 + +在优化 MySQL 时，通常需要对数据库进行分析，常见的分析手段有**慢查询日志**，**EXPLAIN 分析查询**，**profiling 分析**以及**show命令查询系统状态及系统变量**，通过定位分析性能的瓶颈，才能更好的优化数据库系统的性能。 + +##### 2.2.1 性能瓶颈定位 + +我们可以通过 show 命令查看 MySQL 状态及变量，找到系统的瓶颈： + +```mysql +Mysql> show status ——显示状态信息（扩展show status like ‘XXX’） + +Mysql> show variables ——显示系统变量（扩展show variables like ‘XXX’） + +Mysql> show innodb status ——显示InnoDB存储引擎的状态 + +Mysql> show processlist ——查看当前SQL执行，包括执行状态、是否锁表等 + +Shell> mysqladmin variables -u username -p password——显示系统变量 + +Shell> mysqladmin extended-status -u username -p password——显示状态信息 +``` + + + +##### 2.2.2 Explain(执行计划) + +- 是什么：使用 Explain 关键字可以模拟优化器执行 SQL 查询语句，从而知道 MySQL 是如何处理你的 SQL 语句的。分析你的查询语句或是表结构的性能瓶颈 +- 能干吗 + - 表的读取顺序 + - 数据读取操作的操作类型 + - 哪些索引可以使用 + - 哪些索引被实际使用 + - 表之间的引用 + - 每张表有多少行被优化器查询 + +- 怎么玩 + + - Explain + SQL语句 + - 执行计划包含的信息 + + 这里我建两个一样的表 t1、t2 用于示例说明 + + ```mysql + CREATE TABLE t1 ( + id INT NOT NULL AUTO_INCREMENT, + col1 VARCHAR(100), + col2 INT, + col3 VARCHAR(100), + part1 VARCHAR(100), + part2 VARCHAR(100), + part3 VARCHAR(100), + common_field VARCHAR(100), + PRIMARY KEY (id), + KEY idx_key1 (col1), + UNIQUE KEY idx_key2 (col2), + KEY idx_key3 (col3), + KEY idx_key_part(part1, part2, part3) + ) Engine=InnoDB CHARSET=utf8; + ``` + +- 各字段解释 + + - **id**（select 查询的序列号，包含一组数字，表示查询中执行 select 子句或操作表的顺序） + + - id 相同，执行顺序从上往下 + - id 不同，如果是子查询，id 的序号会递增，id 值越大优先级越高，越先被执行 + - id 相同不同，同时存在，相同的属于一组，从上往下执行 + + - **select_type**（查询的类型，用于区别普通查询、联合查询、子查询等复杂查询） + + - **SIMPLE** ：简单的 select 查询，查询中不包含子查询或 UNION + - **PRIMARY**：查询中若包含任何复杂的子部分，最外层查询被标记为 PRIMARY + - **SUBQUERY**：在 select 或 where 列表中包含了子查询 + - **DERIVED**：在 from 列表中包含的子查询被标记为 DERIVED，mysql 会递归执行这些子查询，把结果放在临时表里 + - **UNION**：若第二个 select 出现在 UNION 之后，则被标记为 UNION，若 UNION 包含在 from 子句的子查询中，外层 select 将被标记为 DERIVED + - **UNION RESULT**：从 UNION 表获取结果的 select + + - **table**（显示这一行的数据是关于哪张表的） + + - **partitions**（匹配的分区信息，高版本才有的） + + - **type**（显示查询使用了那种类型，从最好到最差依次排列 **system > const > eq_ref > ref > fulltext > ref_or_null > index_merge > unique_subquery > index_subquery > range > index > ALL** ） + + - system：表只有一行记录（等于系统表），是 const 类型的特例，平时不会出现 + - const：表示通过索引一次就找到了，const 用于比较 primary key 或 unique 索引，因为只要匹配一行数据，所以很快，如将主键置于 where 列表中，mysql 就能将该查询转换为一个常量 + - eq_ref：唯一性索引扫描，对于每个索引键，表中只有一条记录与之匹配，常见于主键或唯一索引扫描 + - ref：非唯一性索引扫描，范围匹配某个单独值得所有行。本质上也是一种索引访问，他返回所有匹配某个单独值的行，然而，它可能也会找到多个符合条件的行，多以他应该属于查找和扫描的混合体 + - ref_or_null：当对普通二级索引进行等值匹配查询，该索引列的值也可以是`NULL`值时，那么对该表的访问方法就可能是ref_or_null + - index_merge: 在某些场景下可以使用`Intersection`、`Union`、`Sort-Union`这三种索引合并的方式来执行查询 + - range：只检索给定范围的行，使用一个索引来选择行。key 列显示使用了哪个索引，一般就是在你的 where 语句中出现了between、<、>、in等的查询，这种范围扫描索引比全表扫描要好，因为它只需开始于索引的某一点，而结束于另一点，不用扫描全部索引 + - index：Full Index Scan，index 于 ALL 区别为 index 类型只遍历索引树。通常比 ALL 快，因为索引文件通常比数据文件小。（**也就是说虽然 all 和 index 都是读全表，但 index 是从索引中读取的，而 all 是从硬盘中读的**） + - all：Full Table Scan，将遍历全表找到匹配的行 + + > 一般来说，得保证查询至少达到 range 级别，最好到达 ref + + - **possible_keys**（显示可能应用在这张表中的索引，一个或多个，查询涉及到的字段若存在索引，则该索引将被列出，但不一定被查询实际使用） + + - **key** + + - 实际使用的索引，如果为NULL，则没有使用索引 + + - **查询中若指定了使用了覆盖索引，则该索引和查询的 select 字段重叠，仅出现在 key 列表中**![](https://img.starfish.ink/mysql/explain-key.png) + + - **key_len** + + - 表示索引中使用的字节数，可通过该列计算查询中使用的索引的长度。在不损失精确性的情况下，长度越短越好 + - key_len 显示的值为索引字段的最大可能长度，并非实际使用长度，即 key_len 是根据表定义计算而得，不是通过表内检索出的 + + - **ref** （显示索引的哪一列被使用了，如果可能的话，是一个常数。哪些列或常量被用于查找索引列上的值）![](https://img.starfish.ink/mysql/explain-ref.png) + + - **rows** （根据表统计信息及索引选用情况，大致估算找到所需的记录所需要读取的行数） + + - **filtered**（某个表经过搜索条件过滤后剩余记录条数的百分比） + + - **Extra**（包含不适合在其他列中显示但十分重要的额外信息） + + 额外信息有好几十个，我们看几个常见的 + + 1. `using filesort`：说明 MySQL 会对数据使用一个外部的索引排序，不是按照表内的索引顺序进行读取。MySQL 中无法利用索引完成的排序操作称为“文件排序”![](https://img.starfish.ink/mysql/explain-extra-using-filesort.png) + + 2. `Using temporary`：使用了临时表保存中间结果，比如去重、排序之类的，比如我们在执行许多包含`DISTINCT`、`GROUP BY`、`UNION`等子句的查询过程中，如果不能有效利用索引来完成查询，`MySQL`很有可能寻求通过建立内部的临时表来执行查询。![](https://img.starfish.ink/mysql/explain-extra-using-tmp.png) + + 3. `using index`：表示相应的 select 操作中使用了覆盖索引，避免访问了表的数据行，效率不错，如果同时出现 `using where`，表明索引被用来执行索引键值的查找；否则索引被用来读取数据而非执行查找操作![](https://img.starfish.ink/mysql/explain-extra-using-index.png) + + 4. `using where`：当某个搜索条件需要在`server层`进行判断时![](https://img.starfish.ink/mysql/explain-extra-using-where.png) + + 5. `using join buffer`：使用了连接缓存![](https://img.starfish.ink/mysql/explain-extra-using-join-buffer.png) + + 6. `impossible where`：where 子句的值总是 false，不能用来获取任何元祖![](https://img.starfish.ink/mysql/explain-extra-impossible-where.png) + + 7. `Using index condition` : 查询使用了索引，但是查询条件不能完全由索引本身来满足![](https://img.starfish.ink/mysql/explain-extra-using-index-condition.png) + + `Using index condition `通常出现在以下几种情况： + + - **索引条件下推（Index Condition Pushdown, ICP）**：这是 MySQL 的一个优化策略，它将查询条件的过滤逻辑“下推”到存储引擎层，而不是在服务器层处理。这样可以减少从存储引擎检索的数据量，从而提高查询效率。 + - **部分索引**：当查询条件只涉及索引的一部分列时，MySQL 可以使用索引来快速定位到满足条件的行，但是可能需要回表（即访问表的实际数据行）来检查剩余的条件。 + - **复合索引**：在使用复合索引（即索引包含多个列）的情况下，如果查询条件只匹配索引的前几列，那么剩余的列可能需要通过 `Using index condition` 来进一步过滤。 + + 8. `select tables optimized away`：在没有group by子句的情况下，基于索引优化操作或对于MyISAM存储引擎优化COUNT(*)操作，不必等到执行阶段再进行计算，查询执行计划生成的阶段即完成优化 + + 9. `distinct`：优化distinct操作，在找到第一匹配的元祖后即停止找同样值的动作 + + > **Json格式的执行计划** + > + > MySQL 5.7及以上版本支持使用`EXPLAIN FORMAT=JSON`命令，该命令返回查询的执行计划，以 JSON 格式展示。 + > + > `EXPLAIN` 的 JSON 输出包含了多个层次和字段，以下是一些主要的字段： + > + > - **`query_block`**: 表示查询块，对应于`EXPLAIN`表格中的一行。对于简单查询，只有一个查询块；对于包含子查询或UNION的复杂查询，会有多个查询块。 + > - **`select_id`**: 查询块的唯一标识符。 + > - **`select_type`**: 查询类型（如`SIMPLE`、`PRIMARY`、`UNION`等）。 + > - **`table`**: 正在访问的表名。 + > - **`partitions`**: 表分区信息。 + > - **`join`**: 如果是连接查询，这里会提供连接的详细信息。 + > - **`condition`**: 应用的条件。 + > - **`used_columns`**: 实际使用到的列。 + > - **`attached_condition`**: 附加条件，如`WHERE`子句或`ON`子句的条件。 + > - **`output`**: 表示查询块的输出，通常是一个子查询或派生表。 + > - **`cost_model`**: 包含成本模型相关的信息，如： + > - **`rows_estimated`**: 估计需要读取的行数。 + > - **`rows_examined`**: 实际检查的行数。 + > - **`cost`**: 查询的成本。 + > - **`execution_info`**: 包含执行信息，如： + > - **`execution_mode`**: 执行模式（如`PACKET_BASED`或`ROW_BASED`）。 + > + > ```mysql + > mysql> EXPLAIN FORMAT=JSON SELECT * FROM t1 INNER JOIN t2 ON t1.col1 = t2.col2 WHERE t1.common_field = 'a'\G + > *************************** 1. row *************************** + > EXPLAIN: { + > "query_block": { + > "select_id": 1, + > "cost_info": { + > "query_cost": "0.70" + > }, + > "nested_loop": [ + > { + > "table": { + > "table_name": "t1", # t1表是驱动表 + > "access_type": "ALL", # 访问方法为ALL，意味着使用全表扫描访问 + > "possible_keys": [ # 可能使用的索引 + > "idx_key1" + > ], + > "rows_examined_per_scan": 1, + > "rows_produced_per_join": 1, + > "filtered": "100.00", + > "cost_info": { + > "read_cost": "0.25", + > "eval_cost": "0.10", + > "prefix_cost": "0.35", + > "data_read_per_join": "1K" + > }, + > "used_columns": [ + > "id", + > "col1", + > "col2", + > "col3", + > "part1", + > "part2", + > "part3", + > "common_field" + > ], + > "attached_condition": "((`fish`.`t1`.`common_field` = 'a') and (`fish`.`t1`.`col1` is not null))" + > } + > }, + > { + > "table": { + > "table_name": "t2", + > "access_type": "eq_ref", + > "possible_keys": [ + > "idx_key2" + > ], + > "key": "idx_key2", + > "used_key_parts": [ + > "col2" + > ], + > "key_length": "5", + > "ref": [ + > "fish.t1.col1" + > ], + > "rows_examined_per_scan": 1, + > "rows_produced_per_join": 1, + > "filtered": "100.00", + > "index_condition": "(cast(`fish`.`t1`.`col1` as double) = cast(`fish`.`t2`.`col2` as double))", + > "cost_info": { + > "read_cost": "0.25", + > "eval_cost": "0.10", + > "prefix_cost": "0.70", + > "data_read_per_join": "1K" + > }, + > "used_columns": [ + > "id", + > "col1", + > "col2", + > "col3", + > "part1", + > "part2", + > "part3", + > "common_field" + > ] + > } + > } + > ] + > } + > } + > ``` + +##### 2.2.3 OPTIMIZER TRACE + +MySQL 的 `OPTIMIZER TRACE` 特性提供了一种深入理解查询优化器如何决定执行计划的方法。通过这个特性，你可以获取关于查询优化过程的详细信息，包括优化器所做的选择、考虑的替代方案、成本估算等 + +MySQL 5.6.3 版本开始，`OPTIMIZER TRACE` 作为一个系统变量引入。要使用它，你需要设置 `optimizer_trace` 变量为 `ON`，并确保你有 `TRACE` 权限。 + +启用 `OPTIMIZER TRACE`： + +```mysql +SET optimizer_trace="enabled=on"; +``` + +执行你的查询后，获取优化器跟踪结果： + +```mysql +SELECT * FROM information_schema.OPTIMIZER_TRACE; +``` + +`OPTIMIZER_TRACE `表包含了多个列，提供了优化器决策过程的详细信息。以下是一些关键列： + +- **`query`**: 执行的SQL查询。 +- **`trace`**: 优化过程的详细跟踪信息，通常以JSON格式展示。 +- **`missed_uses`**: 优化器未能使用的潜在优化。 +- **`step`**: 优化过程中的步骤编号。 +- **`level`**: 跟踪信息的层次级别。 +- **`OK`**: 指示步骤是否成功完成。 +- **`reason`**: 如果步骤未成功，原因说明。 + + + +##### 2.2.4 慢查询日志 + +MySQL 的慢查询日志是 MySQL 提供的一种日志记录，它用来记录在 MySQL 中响应时间超过阈值的语句，具体指运行时间超过`long_query_time` 值的SQL，则会被记录到慢查询日志中。 + +- `long_query_time` 的默认值为10，意思是运行10 秒以上的语句。 +- 默认情况下，MySQL 数据库没有开启慢查询日志，需要手动设置参数开启。 +- 如果不是调优需要的话，一般不建议启动该参数。 + +**开启慢查询日志** + +临时配置： + +```mysql +mysql> set global slow_query_log='ON'; +mysql> set global slow_query_log_file='/var/lib/mysql/hostname-slow.log'; +mysql> set global long_query_time=2; +``` + +可以用 `select sleep(4)` 验证是否成功开启。 + +在生产环境中，如果手工分析日志，查找、分析SQL，还是比较费劲的，所以 MySQL 提供了日志分析工具 `mysqldumpslow`。 + +> 通过 mysqldumpslow --help 查看操作帮助信息 +> +> - 得到返回记录集最多的10个SQL +> +> `mysqldumpslow -s r -t 10 /var/lib/mysql/hostname-slow.log` +> +> - 得到访问次数最多的10个SQL +> +> `mysqldumpslow -s c -t 10 /var/lib/mysql/hostname-slow.log` +> +> - 得到按照时间排序的前10条里面含有左连接的查询语句 +> +> `mysqldumpslow -s t -t 10 -g "left join" /var/lib/mysql/hostname-slow.log` +> +> - 也可以和管道配合使用 +> +> `mysqldumpslow -s r -t 10 /var/lib/mysql/hostname-slow.log | more` + +**也可使用 pt-query-digest 分析 RDS MySQL 慢查询日志** + + + +##### 2.2.5 Show Profile 分析查询 + +通过慢日志查询可以知道哪些 SQL 语句执行效率低下，通过 explain 我们可以得知 SQL 语句的具体执行情况，索引使用等，还可以结合`Show Profile` 命令查看执行状态。 + +- `Show Profile`是 MySQL 提供可以用来分析当前会话中语句执行的资源消耗情况。可以用于 SQL 的调优的测量 +- 默认情况下，参数处于关闭状态，并保存最近 15 次的运行结果 +- 分析步骤 + +1. 启用profiling并执行查询： + + ```mysql + SET profiling = 1; + SELECT * FROM t1 WHERE col1 = 'a'; + ``` + +2. 查看所有PROFILES： + + ```mysql + SHOW PROFILES; + ``` + +3. 查看特定查询的 PROFILE, 进行诊断： + + ```mysql + SHOW PROFILE CPU, BLOCK IO FOR QUERY query_id; + ``` + +![](https://img.starfish.ink/mysql/show-profile.png) + + + +#### 2.3 性能优化 + +##### 2.3.1 索引优化 + +1. **选择合适的索引类型**：根据查询需求，选择适合的索引类型，如普通索引、唯一索引、全文索引等。 +2. **合理设计索引**：创建索引时，应考虑列的选择性，即不同值的比例，选择性高的列更应优先索引。 +3. **使用覆盖索引**：创建覆盖索引，使得查询可以直接从索引中获取所需的数据，而无需回表查询。 +4. **遵循最左前缀匹配原则**：在使用复合索引时，查询条件应该包含索引最左边的列，以确保索引的有效使用。 +5. **避免冗余和重复索引**：定期检查并删除重复或冗余的索引，以减少维护成本和提高性能。 +6. **使用索引条件下推（ICP）**：在MySQL 5.6及以上版本，利用索引条件下推减少服务器层的数据处理。 +7. **索引维护**：定期对索引进行维护，如重建索引，以保持索引的性能和效率。 +8. **使用EXPLAIN分析查询**：通过EXPLAIN命令分析查询的执行计划，确保索引被正确使用。 +9. **避免在索引列上进行运算或使用函数**：这会导致索引失效，从而进行全表扫描。 +10. **负向条件索引**：负向条件查询（如不等于、not in等）不会使用索引，建议用in查询优化。 +11. **对文本建立前缀索引**：对于长文本字段，考虑建立前缀索引以减少索引大小并提高效率。 +12. **建立索引的列不为NULL**：NULL值会影响索引的有效性，尽量确保索引列不包含NULL值。 +13. **明确知道只会返回一条记录时，使用limit 1**：这可以提高查询效率。 +14. **范围查询字段放最后**：在复合索引中，将范围查询的字段放在最后，以提高查询效率。 +15. **尽量全值匹配**：尽量使用索引列的全值匹配，以提高索引的使用效率。 +16. **Like查询时，左侧不要加%**：这会导致索引失效，应尽量避免。 +17. **注意null/not null对索引的影响**：null值可能会影响索引的使用，需要特别注意。 +18. **字符类型务必加上引号**：确保字符类型的值在使用时加上引号，以利用索引。 +19. **OR关键字左右尽量都为索引列**：确保OR条件的两侧字段都是索引列，以提高查询效率。 + +这些策略可以帮助提高 MySQL 数据库的性能，但应根据具体的数据分布和查询模式来设计索引。索引不是越多越好，不恰当的索引可能会降低数据库的插入、更新和删除性能 + + + +##### 2.3.2 查询优化 + +1. **减少返回的列**：避免使用`SELECT *`，只选择需要的列。 +2. **减少返回的行**：使用WHERE子句精确过滤数据、使用LIMIT子句限制返回结果的数量。 +3. **优化JOIN操作**： + - 确保JOIN操作中的表已经正确索引。 + - 使用小表驱动大表的原则。 + - 考虑表的连接顺序，减少中间结果集的大小。 +4. **优化子查询**：将子查询转换为连接（JOIN）操作，以减少数据库的嵌套查询开销。 +5. **使用临时表或派生表**：对于复杂的子查询，可以考虑使用临时表或派生表来简化查询。 +6. **使用聚合函数和GROUP BY优化**： + - 在适当的情况下使用聚合函数减少数据量。 + - 确保GROUP BY子句中的列上有索引。 +7. **避免在WHERE子句中使用函数**：这可能导致索引失效，从而进行全表扫描。 +8. **优化ORDER BY子句**：如果可能，使用索引排序来优化排序操作。 +9. **使用UNION ALL代替UNION**：如果不需要去重，使用UNION ALL可以提高性能。 +10. **优化数据表结构**：避免冗余字段，使用合适的数据类型。 +11. **使用分区表**：对于非常大的表，使用分区可以提高查询效率。 +12. **使用缓存**：对于重复查询的数据，使用缓存来减少数据库的访问。 +13. **避免使用SELECT DISTINCT**：DISTINCT操作会降低查询性能，仅在必要时使用。 +14. **优化LIKE语句**：避免在LIKE语句中使用前导通配符（如'%keyword'）。 +15. **使用合适的事务隔离级别**：降低事务隔离级别可以减少锁的竞争，但要注意数据一致性。 +16. **监控和优化慢查询**：开启慢查询日志，定期分析慢查询，找出性能瓶颈。 +17. **批量操作**：对于插入或更新操作，尽量使用批量操作来减少事务开销。 +18. **避免在索引列上使用OR条件**：OR条件可能导致查询优化器放弃使用索引。 +19. **使用物化视图**：对于复杂的查询，可以考虑使用物化视图来预先计算并存储结果。 + + + +##### 2.3.3 数据类型优化 + +MySQL 支持的数据类型非常多，选择正确的数据类型对于获取高性能至关重要。不管存储哪种类型的数据，下面几个简单的原则都有助于做出更好的选择。 + +- 更小的通常更好：一般情况下，应该尽量使用可以正确存储数据的最小数据类型。 + + 简单就好：简单的数据类型通常需要更少的CPU周期。例如，整数比字符操作代价更低，因为字符集和校对规则（排序规则）使字符比较比整型比较复杂。 + +- 尽量避免NULL：通常情况下最好指定列为NOT NULL + + + +> 规范： +> +> - 必须把字段定义为NOT NULL并且提供默认值 +> - null的列使索引/索引统计/值比较都更加复杂，对MySQL来说更难优化 +> - null 这种类型MySQL内部需要进行特殊处理，增加数据库处理记录的复杂性；同等条件下，表中有较多空字段的时候，数据库的处理性能会降低很多 +> - null值需要更多的存储空，无论是表还是索引中每行中的null的列都需要额外的空间来标识 +> - 对null 的处理时候，只能采用is null或is not null，而不能采用=、in、<、<>、!=、not in这些操作符号。如：where name!=’shenjian’，如果存在name为null值的记录，查询结果就不会包含name为null值的记录 +> +> - **禁止使用TEXT、BLOB类型**：会浪费更多的磁盘和内存空间，非必要的大量的大字段查询会淘汰掉热数据，导致内存命中率急剧降低，影响数据库性能 +> +> - **禁止使用小数存储货币** +> - 必须使用varchar(20)存储手机号 +> +> - **禁止使用ENUM，可使用TINYINT代替** +> +> - 禁止在更新十分频繁、区分度不高的属性上建立索引 +> +> - 禁止使用INSERT INTO t_xxx VALUES(xxx)，必须显示指定插入的列属性 +> +> - 禁止使用属性隐式转换：SELECT uid FROM t_user WHERE phone=13812345678 会导致全表扫描 +> +> - 禁止在WHERE条件的属性上使用函数或者表达式 +> +> - `SELECT uid FROM t_user WHERE from_unixtime(day)>='2017-02-15' `会导致全表扫描 +> +> - 正确的写法是：`SELECT uid FROM t_user WHERE day>= unix_timestamp('2017-02-15 00:00:00')` +> +> - 建立组合索引，必须把区分度高的字段放在前面 +> +> - 禁止负向查询，以及%开头的模糊查询 +> +> - 负向查询条件：NOT、!=、<>、!<、!>、NOT IN、NOT LIKE等，会导致全表扫描 +> - %开头的模糊查询，会导致全表扫描 +> +> - 禁止大表使用JOIN查询，禁止大表使用子查询：会产生临时表，消耗较多内存与CPU，极大影响数据库性能 +> + +### References + +- [58到家数据库30条军规解读](https://mp.weixin.qq.com/s?__biz=MjM5ODYxMDA5OQ==&mid=2651959906&idx=1&sn=2cbdc66cfb5b53cf4327a1e0d18d9b4a&chksm=bd2d07be8a5a8ea86dc3c04eced3f411ee5ec207f73d317245e1fefea1628feb037ad71531bc&scene=21#wechat_redirect) + + + + + diff --git a/docs/data-store/MySQL/MySQL-Segmentation.md b/docs/data-management/MySQL/MySQL-Segmentation.md similarity index 75% rename from docs/data-store/MySQL/MySQL-Segmentation.md rename to docs/data-management/MySQL/MySQL-Segmentation.md index 6ba5aeda9d..c406866d38 100644 --- a/docs/data-store/MySQL/MySQL-Segmentation.md +++ b/docs/data-management/MySQL/MySQL-Segmentation.md @@ -36,48 +36,77 @@ subtopic -### 分区类型及操作 - -#### RANGE分区 - -mysql将会根据指定的拆分策略，,把数据放在不同的表文件上。相当于在文件上,被拆成了小块.但是,对外给客户的感觉还是一张表，透明的。 - - 按照 range 来分，就是每个库一段连续的数据，这个一般是按比如**时间范围**来的，但是这种一般较少用，因为很容易产生热点问题，大量的流量都打在最新的数据上了。 - - range 来分，好处在于说，扩容的时候很简单 - -#### List分区 - -MySQL中的LIST分区在很多方面类似于RANGE分区。和按照RANGE分区一样，每个分区必须明确定义。它们的主要区别在于，LIST分区中每个分区的定义和选择是基于某列的值从属于一个值列表集中的一个值，而RANGE分区是从属于一个连续区间值的集合。 - -#### 其它 - -- Hash分区: hash 分发，好处在于说，可以平均分配每个库的数据量和请求压力；坏处在于说扩容起来比较麻烦，会有一个数据迁移的过程，之前的数据需要重新计算 hash 值重新分配到不同的库或表 - -- Key分区 - -- 子分区 - -#### 对NULL值的处理 - - -MySQL中的分区在禁止空值NULL上没有进行处理，无论它是一个列值还是一个用户定义表达式的值，一般而言，在这种情况下MySQL把NULL当做零。如果你不希望出现类似情况，建议在设计表时声明该列“NOT NULL” - +- **RANGE分区**：基于属于一个给定连续区间的列值，把多行分配给分区。mysql将会根据指定的拆分策略，把数据放在不同的表文件上。相当于在文件上,被拆成了小块.但是,对外给客户的感觉还是一张表，透明的。 + + 按照 range 来分，就是每个库一段连续的数据，这个一般是按比如**时间范围**来的，比如交易表啊，销售表啊等，可以根据年月来存放数据。可能会产生热点问题，大量的流量都打在最新的数据上了。 + + range 来分，好处在于说，扩容的时候很简单。 + + ```mysql + CREATE TABLE sales ( + id INT, + amount DECIMAL(10,2), + sale_date DATE + ) + PARTITION BY RANGE (YEAR(sale_date)) ( + PARTITION p0 VALUES LESS THAN (2000), + PARTITION p1 VALUES LESS THAN (2005), + PARTITION p2 VALUES LESS THAN (2010), + PARTITION p3 VALUES LESS THAN MAXVALUE + ); + ``` + +- **LIST分区**：按列表划分，类似于RANGE分区，但使用的是明确的值列表。 + + 它们的主要区别在于，LIST分区中每个分区的定义和选择是基于某列的值从属于一个值列表集中的一个值，而RANGE分区是从属于一个连续区间值的集合。 + + ```mysql + CREATE TABLE customers ( + id INT, + name VARCHAR(50), + country VARCHAR(50) + ) + PARTITION BY LIST (country) ( + PARTITION p0 VALUES IN ('USA', 'Canada'), + PARTITION p1 VALUES IN ('UK', 'France'), + PARTITION p2 VALUES IN ('Germany', 'Italy') + ); + ``` + +- **HASH分区**：按哈希算法划分，将数据根据某个列的哈希值均匀分布到不同的分区中。 + + hash 分发，好处在于说，可以平均分配每个库的数据量和请求压力；坏处在于说扩容起来比较麻烦，会有一个数据迁移的过程，之前的数据需要重新计算 hash 值重新分配到不同的库或表 + + ```mysql + CREATE TABLE orders ( + id INT, + order_date DATE, + customer_id INT + ) + PARTITION BY HASH(id) PARTITIONS 4; + ``` + +- **KEY分区**：类似于HASH分区，但使用MySQL内部的哈希函数。 + + ```mysql + CREATE TABLE logs ( + id INT, + log_date DATE + ) + PARTITION BY KEY(id) PARTITIONS 4; + ``` + **看上去分区表很帅气，为什么大部分互联网还是更多的选择自己分库分表来水平扩展咧？** -回答： +- 分区表，分区键设计不太灵活，如果不走分区键，很容易出现全表锁 -1）分区表，分区键设计不太灵活，如果不走分区键，很容易出现全表锁 - -2）一旦数据量并发量上来，如果在分区表实施关联，就是一个灾难 - -3）自己分库分表，自己掌控业务场景与访问模式，可控。分区表，研发写了一个sql，都不确定mysql是怎么玩的，不太可控 - -4）运维的坑，嘿嘿 +- 一旦数据并发量上来，如果在分区表实施关联，就是一个灾难 +- 自己分库分表，自己掌控业务场景与访问模式，可控。分区表，研发写了一个sql，都不确定mysql是怎么玩的，不太可控 + ## Mysql分库 @@ -158,11 +187,11 @@ Atlas是由 Qihoo 360, Web平台部基础架构团队开发维护的一个基于 TDDL所处的位置（tddl通用数据访问层，部署在客户端的jar包，用于将用户的SQL路由到指定的数据库中）： -![image-20191205103623544](../_images/mysql/tddl.png) +![image-20191205103623544](../../_images/mysql/tddl.png) 淘宝很早就对数据进行过分库的处理， 上层系统连接多个数据库，中间有一个叫做DBRoute的路由来对数据进行统一访问。DBRoute对数据进行多库的操作、数据的整合，让上层系统像操作 一个数据库一样操作多个库。但是随着数据量的增长，对于库表的分法有了更高的要求，例如，你的商品数据到了百亿级别的时候，任何一个库都无法存放了，于是 分成2个、4个、8个、16个、32个……直到1024个、2048个。好，分成这么多，数据能够存放了，那怎么查询它？这时候，数据查询的中间件就要能 够承担这个重任了，它对上层来说，必须像查询一个数据库一样来查询数据，还要像查询一个数据库一样快（每条查询在几毫秒内完成），TDDL就承担了这样一 个工作。在外面有些系统也用DAL（数据访问层） 这个概念来命名这个中间件。 -![image-20191205103631139](../_images/mysql/TDDL2.png) +![image-20191205103631139](../../_images/mysql/TDDL2.png) 系出名门，淘宝诞生。功能强大，阿里开源（部分） 主要优点： diff --git a/docs/data-management/MySQL/MySQL-Storage-Engines.md b/docs/data-management/MySQL/MySQL-Storage-Engines.md new file mode 100644 index 0000000000..19f5543935 --- /dev/null +++ b/docs/data-management/MySQL/MySQL-Storage-Engines.md @@ -0,0 +1,257 @@ +--- +title: MySQL Storage Engine +date: 2023-05-31 +tags: + - MySQL +categories: MySQL +--- + +![img](https://dbastack.com/wp-content/uploads/2024/09/MySQL-Storage-Engine-3.png.webp) + +> 存储引擎是 MySQL 的组件，用于处理不同表类型的 SQL 操作。不同的存储引擎提供不同的存储机制、索引技巧、锁定水平等功能，使用不同的存储引擎，还可以获得特定的功能。 +> +> 使用哪一种引擎可以灵活选择，**一个数据库中多个表可以使用不同引擎以满足各种性能和实际需求**，使用合适的存储引擎，将会提高整个数据库的性能 。 +> +> MySQL 服务器使用可插拔的存储引擎体系结构，可以从运行中的MySQL服务器加载或卸载存储引擎 。 + +> [MySQL 5.7 可供选择的存储引擎](https://dev.mysql.com/doc/refman/5.7/en/storage-engines.html) + +## 一、存储引擎的作用与架构 + +MySQL 存储引擎是数据库的底层核心组件，负责数据的**存储、检索、事务控制**以及**并发管理**。其架构采用**插件式设计**，允许用户根据业务需求灵活选择引擎类型，例如 InnoDB、MyISAM、Memory 等。这种设计将**查询处理**与**数据存储**解耦，提升了系统的可扩展性和灵活性 。 + +MySQL 的体系架构分为四层： + +- **连接层**：管理客户端连接、认证与线程分配，支持 SSL 安全协议。 +- **核心服务层**：处理 SQL 解析、优化、缓存及内置函数执行。 +- **存储引擎层**：实际负责数据的存储和提取，支持多引擎扩展。 +- **数据存储层**：通过文件系统与存储引擎交互，管理物理文件。 + + + +## 二、核心存储引擎详解 + +### 2.1 常用存储引擎 + +**查看存储引擎** + +```mysql +-- 查看支持的存储引擎 +SHOW ENGINES + +-- 查看默认存储引擎 +SHOW VARIABLES LIKE 'storage_engine' + +--查看具体某一个表所使用的存储引擎，这个默认存储引擎被修改了！ +show create table tablename + +--准确查看某个数据库中的某一表所使用的存储引擎 +show table status like 'tablename' +show table status from database where name="tablename" +``` + +以下是 MySQL 主要存储引擎的对比表格，整合了各引擎的核心特性及适用场景，结合最新版本（MySQL 8.0+）特性更新： + +| **存储引擎** | **核心特性** | **事务支持** | **锁级别** | **索引类型** | **文件结构** | **适用场景** | +| ---------------------- | ------------------------------------------------------------ | ------------ | -------------- | ----------------------- | -------------------------------------- | -------------------------------- | +| **InnoDB** | 支持ACID事务、行级锁、MVCC、外键约束，具备崩溃恢复能力，默认使用聚簇索引 | ✅ | 行锁/表锁 | B+Tree/全文索引（5.6+） | `.ibd`（数据+索引）、`.frm`（表结构） | 高并发OLTP（电商交易、金融系统） | +| **MyISAM** | 非事务型，表级锁，支持全文索引和压缩表，查询速度快 | ❌ | 表锁 | B+Tree/全文索引 | `.MYD`（数据）、`.MYI`（索引）、`.frm` | 静态报表、日志分析、只读业务 | +| **Memory** | 数据全内存存储，哈希索引加速查询，重启后数据丢失 | ❌ | 表锁 | Hash/B-Tree | `.frm`（仅表结构） | 临时表、会话缓存、高速缓存层 | +| **Archive** | 仅支持INSERT/SELECT，Zlib压缩存储（压缩率10:1），无索引 | ❌ | 行锁（仅插入） | ❌ | `.ARZ`（数据）、`.ARM`（元数据） | 历史数据归档、审计日志 | +| **CSV** | 数据以CSV格式存储，可直接文本编辑，不支持索引 | ❌ | 表锁 | ❌ | `.CSV`（数据）、`.CSM`（元数据） | 数据导入/导出中间表 | +| **Blackhole** | 写入数据即丢弃，仅保留二进制日志，用于复制链路中继 | ❌ | ❌ | ❌ | `.frm`（仅表结构） | 主从复制中继、性能测试 | +| **Federated** | 代理访问远程表，本地无实际数据存储 | ❌ | 依赖远程表引擎 | 依赖远程表引擎 | `.frm`（仅表结构） | 分布式数据聚合 | +| **NDB** | 集群式存储引擎，支持数据自动分片和高可用性 | ✅ | 行锁 | Hash/B-Tree | 数据存储在集群节点 | MySQL Cluster分布式系统 | +| **Merge** | 聚合多个MyISAM表，逻辑上作为单个表操作 | ❌ | 表锁 | B-Tree | `.MRG`（聚合定义）、底层使用MyISAM文件 | 分库分表聚合查询 | +| **Performance Schema** | 内置性能监控引擎，采集服务器运行时指标 | ❌ | ❌ | ❌ | 内存存储，无物理文件 | 性能监控与诊断 | + + + +### 2.2 存储引擎架构演进 + +**1. MySQL 8.0 关键改进** + +- 原子 DDL：DDL操作（如CREATE TABLE）具备事务性，失败时自动回滚元数据变更 +- 数据字典升级：系统表全部转为InnoDB引擎，替代原有的.frm文件，实现事务化元数据管理 +- Redo日志优化：MySQL 8.0.30+ 引入 `innodb_redo_log_capacity` 参数替代旧版日志配置，支持动态调整redo日志大小 + + **2. 物理文件结构变化** + +| 文件类型 | 5.7及之前版本 | 8.0+版本 | 作用 | +| -------------- | ------------- | ------------------ | ------------------ | +| 表结构定义文件 | .frm | .sdi (JSON格式) | 存储表结构元数据 6 | +| 事务日志 | ibdata1 | undo_001, undo_002 | 独立UNDO表空间 | +| 数据文件 | .ibd | .ibd | 表数据与索引存储 | +| 临时文件 | ibtmp1 | ibtmp1 | 临时表空间 | + +> 示例：通过 `SHOW CREATE TABLE` 可查看SDI元数据，支持 JSON 格式导出 + + + +### 2.3 Innodb 引擎的 4 大特性 + +#### **1. 插入缓冲（Insert Buffer / Change Buffer）** + +- **作用**：优化非唯一二级索引的插入、删除、更新（即 DML 操作）性能，减少磁盘随机 I/O 开销。 +- 原理： + - 当非唯一索引页不在内存中时，操作会被暂存到 Change Buffer（内存区域）中，而非直接写入磁盘。 + - 后续通过合并（Merge）操作，将多个离散的修改批量写入磁盘，减少 I/O 次数。 +- 适用条件： + - 仅针对非唯一二级索引。 + - 可通过参数 `innodb_change_buffer_max_size` 调整缓冲区大小（默认 25% 缓冲池）。 + +#### 2. 二次写（Double Write） + +- 作用：防止因部分页写入（Partial Page Write）导致的数据页损坏，确保崩溃恢复的可靠性。 +- 流程： + - 脏页刷盘时，先写入内存的 Doublewrite Buffer，再分两次（每次 1MB）顺序写入共享表空间的连续磁盘区域。 + - 若数据页写入过程中崩溃，恢复时从共享表空间副本还原损坏页，再通过 Redo Log 恢复。 +- 意义：牺牲少量顺序 I/O 换取数据完整性，避免因随机 I/O 中断导致数据丢失。 + +#### 3. 自适应哈希索引（Adaptive Hash Index, AHI） + +- 作用：自动为高频访问的索引页创建哈希索引，加速查询速度（尤其等值查询）。 + +- 触发条件： + + - 同一索引被连续访问 17 次以上。 + - 某页被访问超过 100 次，且访问模式一致（如固定 WHERE 条件）。 + +- 限制 + + ：仅对热点数据生效，无法手动指定，可通过参数 `innodb_adaptive_hash_index` 启用或关闭。 + +#### 4. 预读（Read Ahead） + +- 作用：基于空间局部性原理，异步预加载相邻数据页到缓冲池，减少未来查询的磁盘 I/O。 +- 模式： + - 线性预读：按顺序访问的页超过阈值时，预加载下一批连续页（默认 64 页为一个块）。 + - 随机预读（已废弃）：当某块中部分页在缓冲池时，预加载剩余页，但因性能问题被弃用。 + +#### 其他重要特性补充 + +尽管上述四点是核心性能优化特性，但 InnoDB 的其他关键能力也值得注意： + +- 事务支持：通过 ACID 特性（原子性、一致性、隔离性、持久性）保障数据一致性。 +- 行级锁与外键约束：支持高并发与数据完整性。 +- **崩溃恢复**：结合 Redo Log 和 Double Write 实现快速恢复 + + + +### 2.4 数据的存储 + +在整个数据库体系结构中，我们可以使用不同的存储引擎来存储数据，而绝大多数存储引擎都以二进制的形式存储数据；我们来看下InnoDB 中对数据是如何存储的。 + +在 InnoDB 存储引擎中，所有的数据都被逻辑地存放在表空间中，表空间（tablespace）是存储引擎中最高的存储逻辑单位，在表空间的下面又包括段（segment）、区（extent）、页（page） + +![](https://img.starfish.ink/mysql/table-space.jpg) + + + + 同一个数据库实例的所有表空间都有相同的页大小；默认情况下，表空间中的页大小都为 16KB，当然也可以通过改变 `innodb_page_size` 选项对默认大小进行修改，需要注意的是不同的页大小最终也会导致区大小的不同 + +对于 16KB 的页来说，连续的 64 个页就是一个区，也就是 1 个区默认占用 1 MB 空间的大小。 + +#### 数据页结构 + +页是 InnoDB 存储引擎管理数据的最小磁盘单位，一个页的大小一般是 `16KB`。 + +`InnoDB` 为了不同的目的而设计了许多种不同类型的`页`，比如存放表空间头部信息的页，存放 `Insert Buffer` 信息的页，存放 `INODE`信息的页，存放 `undo` 日志信息的页等等等等。 + + B-Tree 节点就是实际存放表中数据的页面，我们在这里将要介绍页是如何组织和存储记录的；首先，一个 InnoDB 页有以下七个部分： + +![](https://img.starfish.ink/mysql/innodb-b-tree-node.jpg) + +有的部分占用的字节数是确定的，有的部分占用的字节数是不确定的。 + +| 名称 | 中文名 | 占用空间大小 | 简单描述 | +| -------------------- | ------------------ | ------------ | ------------------------ | +| `File Header` | 文件头部 | `38`字节 | 页的一些通用信息 | +| `Page Header` | 页面头部 | `56`字节 | 数据页专有的一些信息 | +| `Infimum + Supremum` | 最小记录和最大记录 | `26`字节 | 两个虚拟的行记录 | +| `User Records` | 用户记录 | 不确定 | 实际存储的行记录内容 | +| `Free Space` | 空闲空间 | 不确定 | 页中尚未使用的空间 | +| `Page Directory` | 页面目录 | 不确定 | 页中的某些记录的相对位置 | +| `File Trailer` | 文件尾部 | `8`字节 | 校验页是否完整 | + +在页的 7 个组成部分中，我们自己存储的记录会按照我们指定的`行格式`存储到 `User Records` 部分。但是在一开始生成页的时候，其实并没有 `User Records` 这个部分，每当我们插入一条记录，都会从 `Free Space` 部分，也就是尚未使用的存储空间中申请一个记录大小的空间划分到 `User Records` 部分，当 `Free Space` 部分的空间全部被 `User Records` 部分替代掉之后，也就意味着这个页使用完了，如果还有新的记录插入的话，就需要去申请新的页了，这个过程的图示如下： + +![](https://img.starfish.ink/mysql/page-application-record.png) + + + +#### 如何存储表 + +MySQL 使用 InnoDB 存储表时，会将表的定义和数据索引等信息分开存储，其中前者存储在 `.frm` 文件中，后者存储在 `.ibd` 文件中。 + +#### .frm 文件 + +无论在 MySQL 中选择了哪个存储引擎，所有的 MySQL 表都会在硬盘上创建一个 `.frm` 文件用来描述表的格式或者说定义；`.frm` 文件的格式在不同的平台上都是相同的。 + +> MySQL 官方文档中的 [11.1 MySQL .frm File Format](https://dev.mysql.com/doc/internals/en/frm-file-format.html) 一文对于 `.frm` 文件格式中的二进制的内容有着非常详细的表述。 + +#### .ibd 文件 + +InnoDB 中用于存储数据的文件总共有两个部分，一是系统表空间文件，包括 `ibdata1`、`ibdata2` 等文件，其中存储了 InnoDB 系统信息和用户数据库表数据和索引，是所有表公用的。 + +当打开 `innodb_file_per_table` 选项时，`.ibd` 文件就是每一个表独有的表空间，文件存储了当前表的数据和相关的索引数据。 + +#### 如何存储记录 | InnoDB 行格式 + +InnoDB 存储引擎和大多数数据库一样，记录是以行的形式存储的，每个 16KB 大小的页中可以存放多条行记录。 + +它可以使用不同的行格式进行存储。 + +InnoDB 早期的文件格式为 `Antelope`，可以定义两种行记录格式，分别是 `Compact` 和 `Redundant`，InnoDB 1.0.x 版本开始引入了新的文件格式 `Barracuda`。`Barracuda `文件格式下拥有两种新的行记录格式：`Compressed` 和 `Dynamic`。 + +> [InnoDB Row Formats](https://dev.mysql.com/doc/refman/5.7/en/innodb-row-format.html#innodb-row-format-redundant) + +![](https://img.starfish.ink/mysql/innodb-row-format.png) + +MySQL 5.7 版本支持以上格式的行存储方式。 + +我们可以在创建或修改表的语句中指定行格式： + +```mysql +CREATE TABLE 表名 (列的信息) ROW_FORMAT=行格式名称 + +ALTER TABLE 表名 ROW_FORMAT=行格式名称 +``` + +`Compact `行记录格式是在 MySQL 5.0 中引入的，其首部是一个非 NULL 变长列长度列表，并且是逆序放置的，其长度为： + +- 若列的长度小于等于 255 字节，用 1 个字节表示； +- 若列的长度大于 255 字节，用 2 个字节表示。 + +![Compact row format](https://miro.medium.com/v2/resize:fit:1400/1*wNIUPIn4jo9kKbLvsmSUDQ.png) + +变长字段的长度最大不可以超过 2 字节，这是因为 MySQL 数据库中 VARCHAR 类型的最大长度限制为 65535。变长字段之后的第二个部分是 NULL 标志位，该标志位指示了该行数据中某列是否为 NULL 值，有则用 1 表示，NULL 标志位也是不定长的。接下来是记录头部信息，固定占用 5 字节。 + +`Redundant` 是 MySQL 5.0 版本之前 InnoDB 的行记录格式，`Redundant` 行记录格式的首部是每一列长度偏移列表，同样是逆序存放的。从整体上看，`Compact `格式的存储空间减少了约 20%，但代价是某些操作会增加 CPU 的使用。 + +`Dynamic` 和 `Compressed `是 `Compact `行记录格式的变种，`Compressed `会对存储在其中的行数据会以 `zlib` 的算法进行压缩，因此对于 BLOB、TEXT、VARCHAR 这类大长度类型的数据能够进行非常有效的存储。 + +> 高版本，比如 8.3 默认使用的是 Dynamic +> +> ```sql +> SELECT @@innodb_default_row_format; +> ``` + + + +#### 行溢出数据 + +当 InnoDB 存储极长的 TEXT 或者 BLOB 这类大对象时，MySQL 并不会直接将所有的内容都存放在数据页中。因为 InnoDB 存储引擎使用 B+Tree 组织索引，每个页中至少应该有两条行记录，因此，如果页中只能存放下一条记录，那么 InnoDB 存储引擎会自动将行数据存放到溢出页中。 + +如果我们使用 `Compact` 或 `Redundant` 格式，那么会将行数据中的前 768 个字节存储在数据页中，后面的数据会通过指针指向 Uncompressed BLOB Page。 + +但是如果我们使用新的行记录格式 `Compressed` 或者 `Dynamic` 时只会在行记录中保存 20 个字节的指针，实际的数据都会存放在溢出页面中。 + + + +### 参考与引用： + +- https://www.linkedin.com/pulse/leverage-innodb-architecture-optimize-django-model-design-bouslama +- [踏雪无痕-InnoDB存储引擎](https://www.cnblogs.com/chenpingzhao/p/9177324.html) +- [MySQL 与 InnoDB 存储引擎总结](https://wingsxdu.com/posts/database/mysql/innodb/) + diff --git a/docs/data-management/MySQL/MySQL-Transaction.md b/docs/data-management/MySQL/MySQL-Transaction.md new file mode 100644 index 0000000000..45e21bd4f4 --- /dev/null +++ b/docs/data-management/MySQL/MySQL-Transaction.md @@ -0,0 +1,513 @@ +--- +title: MySQL 事务 +date: 2022-02-01 +tags: + - MySQL +categories: MySQL +--- + +![](https://img.starfish.ink/mysql/banner-mysql-transaction.png) + +> Hello，我是海星。 +> +> MySQL 事务，最熟悉经典的例子，就是你给我转账的例子了，要经过查询余额，减你的钱，加我的钱，这一系列操作必须保证是一体的，这些数据库操作的集合就构成了一个事务。 +> +> MySQL 事务也是在存储引擎层面实现的，大家用 InnoDB 取代 MyISAM 引擎很重要的一个原因就是 InnoDB 支持事务。 + + + +## 一、事务基本要素 — ACID + +事务是由一组 SQL 语句组成的逻辑处理单元，具有 4 个属性，通常简称为事务的 ACID 属性。 + +![](https://img.starfish.ink/mysql/ACID.png) + +- **A (Atomicity) 原子性**：整个事务中的所有操作，要么全部完成，要么全部不完成，不可能停滞在中间某个环节。事务在执行过程中发生错误，会被回滚（Rollback）到事务开始前的状态，就像这个事务从来没有执行过一样。 +- **C (Consistency) 一致性**：在事务开始之前和事务结束以后，数据库的完整性约束没有被破坏。 +- **I (Isolation)隔离性**：一个事务所做的修改在最终提交以前，对其他事务是不可见的。这种属性有时称为『串行化』，为了防止事务操作间的混淆，必须串行化或序列化请求，使得在同一时间仅有一个请求用于同一数据。 +- **D (Durability) 持久性**：在事务完成以后，该事务对数据库所作的更改便持久的保存在数据库中，并不会被回滚。 + + + +## 二、MySQL 中事务的使用 + + MySQL 的服务层不管理事务，而是由下层的存储引擎实现。MySQL 提供了两种事务型的存储引擎：InnoDB 和 NDB。 + +**MySQL支持本地事务的语句：** + +```mysql +START TRANSACTION | BEGIN [WORK] +COMMIT [WORK] [AND [NO] CHAIN] [[NO] RELEASE] +ROLLBACK [WORK] [AND [NO] CHAIN] [[NO] RELEASE] +SET AUTOCOMMIT = {0 | 1} +``` + +- START TRANSACTION 或 BEGIN 语句：开始一项新的事务。 +- COMMIT 和 ROLLBACK：用来提交或者回滚事务。 +- CHAIN 和 RELEASE 子句：分别用来定义在事务提交或者回滚之后的操作，CHAIN 会立即启动一个新事物，并且和刚才的事务具有相同的隔离级别，RELEASE 则会断开和客户端的连接。 +- SET AUTOCOMMIT 可以修改当前连接的提交方式， 如果设置了 SET AUTOCOMMIT=0，则设置之后的所有事务都需要通过明确的命令进行提交或者回滚 + +**事务使用注意点：** + +- 如果在锁表期间，用 start transaction 命令开始一个新事务，会造成一个隐含的 unlock tables 被执行。 +- 在同一个事务中，最好不使用不同存储引擎的表，否则 ROLLBACK 时需要对非事务类型的表进行特别的处理，因为 COMMIT、ROLLBACK 只能对事务类型的表进行提交和回滚。 +- 和 Oracle 的事务管理相同，所有的 DDL 语句是不能回滚的，并且部分的 DDL 语句会造成隐式的提交。 +- 在事务中可以通过定义 SAVEPOINT（例如：mysql> savepoint test; 定义 savepoint，名称为 test），指定回滚事务的一个部分，但是不能指定提交事务的一个部分。对于复杂的应用，可以定义多个不同的 SAVEPOINT，满足不同的条件时，回滚 + 不同的 SAVEPOINT。需要注意的是，如果定义了相同名字的 SAVEPOINT，则后面定义的SAVEPOINT 会覆盖之前的定义。对于不再需要使用的 SAVEPOINT，可以通过 RELEASE SAVEPOINT 命令删除 SAVEPOINT， 删除后的 SAVEPOINT， 不能再执行 ROLLBACK TO SAVEPOINT命令。 + +**自动提交（autocommit）：** +Mysql 默认采用自动提交模式，可以通过设置 `autocommit` 变量来启用或禁用自动提交模式 + +- **隐式锁定** + + InnoDB 在事务执行过程中，使用两阶段锁协议： + + 随时都可以执行锁定，InnoDB 会根据隔离级别在需要的时候自动加锁； + + 锁只有在执行 commit 或者 rollback 的时候才会释放，并且所有的锁都是在**同一时刻**被释放。 + +- **显式锁定** + + InnoDB 也支持通过特定的语句进行显示锁定（存储引擎层）： + +```mysql +select ... lock in share mode //共享锁 +select ... for update //排他锁 +``` + +​ MySQL Server 层的显示锁定： + +```mysql +lock table 和 unlock table +``` + + + +## 三、事务隔离级别 + +当数据库上有多个事务同时执行的时候，就可能出现脏读（dirty read）、不可重复读（non-repeatable read）、幻读（phantom read）的问题，为了解决这些问题，就有了“隔离级别”的概念。 + +> **并发事务处理带来的问题** +> +> - 更新丢失（Lost Update)： 事务 A 和事务 B 选择同一行，然后基于最初选定的值更新该行时，由于两个事务都不知道彼此的存在，就会发生丢失更新问题 +> - 脏读(Dirty Reads)：事务 A 读取了事务 B 更新的数据，然后 B 回滚操作，那么 A 读取到的数据是脏数据 +> - 不可重复读（Non-Repeatable Reads)：事务 A 多次读取同一数据，事务 B 在事务 A 多次读取的过程中，对数据作了更新并提交，导致事务 A 多次读取同一数据时，结果不一致。 +> - 幻读（Phantom Reads)：系统管理员 A 将数据库中所有学生的成绩从具体分数改为 ABCDE 等级，但是系统管理员 B 就在这个时候插入了一条具体分数的记录，当系统管理员 A 改结束后发现还有一条记录没有改过来，就好像发生了幻觉一样，这就叫幻读。 + +> **幻读和不可重复读的区别：** +> +> - 不可重复读的重点是修改：在同一事务中，同样的条件，第一次读的数据和第二次读的数据不一样。（因为中间有其他事务提交了修改） +> - 幻读的重点在于新增或者删除：在同一事务中，同样的条件,，第一次和第二次读出来的记录数不一样。（因为中间有其他事务提交了插入/删除） +> + +> **并发事务处理带来的问题的解决办法：** +> +> - “更新丢失”通常是应该完全避免的。但防止更新丢失，并不能单靠数据库事务控制器来解决，需要应用程序对要更新的数据加必要的锁来解决，因此，防止更新丢失应该是应用的责任。 +> +> - “脏读” 、 “不可重复读”和“幻读” ，其实都是数据库读一致性问题，必须由数据库提供一定的事务隔离机制来解决： +> +> - 一种是加锁：在读取数据前，对其加锁，阻止其他事务对数据进行修改。 +> - 另一种是数据多版本并发控制（MultiVersion Concurrency Control，简称 **MVCC** 或 MCC），也称为多版本数据库：不用加任何锁， 通过一定机制生成一个数据请求时间点的一致性数据快照 （Snapshot)， 并用这个快照来提供一定级别 （语句级或事务级） 的一致性读取。从用户的角度来看，好象是数据库可以提供同一数据的多个版本。 +> + +在谈隔离级别之前，你首先要知道，你隔离得越严实，效率就会越低。因此很多时候，我们都要在二者之间寻找一个平衡点。 + +数据库事务的隔离级别有 4 种，由低到高分别为 + +- 读未提交（read uncommitted） +- 读提交（read committed） +- 可重复读（repeatable read） +- 串行化（serializable ） + +查看当前数据库的事务隔离级别： + +```mysql +mysql> show variables like 'transaction_isolation'; ++-----------------------+-----------------+ +| Variable_name | Value | ++-----------------------+-----------------+ +| transaction_isolation | REPEATABLE-READ | ++-----------------------+-----------------+ +``` + +> 通俗理解就是 +> +> - 读未提交：别人改数据的事务尚未提交，我在我的事务中也能读到。 +> - 读已提交：别人改数据的事务已经提交，我在我的事务中才能读到。 +>- 可重复读：别人改数据的事务已经提交，我在我的事务中也不去读。 +> - 串行：我的事务尚未提交，别人就别想改数据。 + +#### Read uncommitted + +**读未提交，就是一个事务可以读取另一个未提交事务的数据**。 + +事例：老板要给程序员发工资，程序员的工资是3.6万/月。但是发工资时老板不小心按错了数字，按成3.9万/月，该钱已经打到程序员的户口，但是事务还没有提交，就在这时，程序员去查看自己这个月的工资，发现比往常多了3千元，以为涨工资了非常高兴。但是老板及时发现了不对，马上回滚差点就提交了的事务，将数字改成3.6万再提交。 + +分析：实际程序员这个月的工资还是3.6万，但是程序员看到的是3.9万。他看到的是老板还没提交事务时的数据。这就是脏读。 + +那怎么解决脏读呢？Read committed！读提交，能解决脏读问题。 + +#### Read committed + +**读提交，顾名思义，就是一个事务要等另一个事务提交后才能读取数据**。 + +事例：程序员拿着信用卡去享受生活（卡里当然是只有3.6万），当他埋单时（程序员事务开启），收费系统事先检测到他的卡里有3.6万，就在这个时候！！程序员的妻子要把钱全部转出充当家用，并提交。当收费系统准备扣款时，再检测卡里的金额，发现已经没钱了（第二次检测金额当然要等待妻子转出金额事务提交完）。程序员就会很郁闷，明明卡里是有钱的… + +分析：这就是读提交，若有事务对数据进行更新（UPDATE）操作时，读操作事务要等待这个更新操作事务提交后才能读取数据，可以解决脏读问题。但在这个事例中，出现了一个事务范围内两个相同的查询却返回了不同数据，这就是不可重复读。 + +那怎么解决可能的不可重复读问题？Repeatable read ！ + +#### Repeatable read + +**可重复读，就是在开始读取数据（事务开启）时，不再允许修改操作。 MySQL 的默认事务隔离级别** + +事例：程序员拿着信用卡去享受生活（卡里当然是只有3.6万），当他埋单时（事务开启，不允许其他事务的UPDATE修改操作），收费系统事先检测到他的卡里有3.6万。这个时候他的妻子不能转出金额了。接下来收费系统就可以扣款了。 + +分析：重复读可以解决不可重复读问题。写到这里，应该明白的一点就是，不可重复读对应的是修改，即 UPDATE 操作。但是可能还会有幻读问题。因为幻读问题对应的是插入 INSERT 操作，而不是 UPDATE 操作。 + +**什么时候会出现幻读？** + +事例：程序员某一天去消费，花了2千元，然后他的妻子去查看他今天的消费记录（全表扫描FTS，妻子事务开启），看到确实是花了2千元，就在这个时候，程序员花了1万买了一部电脑，即新增 INSERT 了一条消费记录，并提交。当妻子打印程序员的消费记录清单时（妻子事务提交），发现花了1.2万元，似乎出现了幻觉，这就是幻读。 + +那怎么解决幻读问题？Serializable！ + +#### Serializable 序列化 + +串行化是最高的事务隔离级别，在该级别下，事务串行化顺序执行，可以避免脏读、不可重复读与幻读。 + +简单来说，Serializable会在读取的每一行数据上都加锁，所以可能导致大量的超时和锁争用问题。这种事务隔离级别效率低下，比较耗数据库性能，一般不使用。 + + + +#### demo + +```mysql +create table T(t int) engine=InnoDB; +insert into T(t) values(1); +``` + +![](https://img.starfish.ink/mysql/transaction-demo.png) + +- “读未提交”：则 V1 的值就是 2。这时候事务 B 虽然还没有提交，但是结果已经被 A 看到了。因此，V2、V3 也都是 2。 +- “读提交”：则 V1 是 1，V2 的值是 2。事务 B 的更新在提交后才能被 A 看到。所以， V3 的值也是 2。 +- “可重复读”：则 V1、V2 是 1，V3 是 2。之所以 V2 还是 1，遵循的就是这个要求：事务在执行期间看到的数据前后必须是一致的。 +- “串行化”：则在事务 B 执行“将 1 改成 2”的时候，会被锁住。直到事务 A 提交后，事务 B 才可以继续执行。所以从 A 的角度看， V1、V2 值是 1，V3 的值是 2。 + +> - 读未提交：别人改数据的事务尚未提交，我在我的事务中也能读到。 +> - 读已提交：别人改数据的事务已经提交，我在我的事务中才能读到。 +> - 可重复读：别人改数据的事务已经提交，我在我的事务中也不去读。 +> - 串行：我的事务尚未提交，别人就别想改数据。 + +| 事务隔离级别 | 读数据一致性 | 脏读 | 不可重复读 | 幻读 | +| ---------------------------- | ---------------------------------------- | ---- | ---------- | ---- | +| 读未提交（read-uncommitted） | 最低级被，只能保证不读取物理上损坏的数据 | 是 | 是 | 是 | +| 读已提交（read-committed） | 语句级 | 否 | 是 | 是 | +| 可重复读（repeatable-read） | 事务级 | 否 | 否 | 是 | +| 串行化（serializable） | 最高级别，事务级 | 否 | 否 | 否 | + +需要说明的是，事务隔离级别和数据访问的并发性是对立的，事务隔离级别越高并发性就越差。所以要根据具体的应用来确定合适的事务隔离级别，这个地方没有万能的原则。 + + + +## 三、MVCC 多版本并发控制 + +#### 核心概念 + +1. **快照读（Snapshot Read）**： + - 每个事务在开始时，会获取一个数据快照，事务在读取数据时，总是读取该快照中的数据。 + - 这意味着即使在事务进行期间，其他事务对数据的更新也不会影响当前事务的读取。 +2. **版本链（Version Chain）**： + - 每个数据行都有多个版本，每个版本包含数据和元数据（如创建时间、删除时间等）。 + - 新版本的数据行会被链接到旧版本的数据行，形成一个版本链。 +3. **隐式锁（Implicit Locking）**： + - MVCC 通过版本管理避免了显式锁定，减少了锁争用问题。 + - 对于读取操作，事务读取其开始时的快照数据，不会被写操作阻塞。 + +#### MVCC 的底层实现 + +1. **数据行的多版本存储**： + - 每个数据行在物理存储上会有多个版本，每个版本包含该行在特定时间点的值。 + - 数据行版本包含元数据，如事务ID（Transaction ID）、创建时间戳和删除时间戳。 +2. **快照读取**： + - 每个事务在开始时，会记录当前系统的事务ID作为快照ID。 + - 读取数据时，只读取那些创建时间戳早于快照ID，并且删除时间戳为空或晚于快照ID的数据版本。 +3. **事务提交和版本更新**： + - 当一个事务对数据行进行更新时，会创建一个新的数据版本，并将其链接到现有版本链上。 + - 旧版本仍然存在，直到没有任何活动事务需要访问它们。 + +#### MVCC 在MySQL中的实现 + +MySQL InnoDB 存储引擎使用 MVCC 来实现可重复读（REPEATABLE READ）隔离级别，避免脏读、不可重复读和幻读问题。具体机制如下： + +1. **隐藏列**： + - InnoDB 在每行记录中存储两个隐藏列：`trx_id`（事务ID）和`roll_pointer`（回滚指针）。 + - `trx_id` 记录最后一次修改该行的事务ID，`roll_pointer` 指向该行的上一版本。 + +> 其实，InnoDB下的 Compact 行结构，有三个隐藏的列 +> +> | 列名 | 是否必须 | 描述 | +> | -------------- | -------- | ------------------------------------------------------------ | +> | row_id | 否 | 行ID，唯一标识一条记录（如果定义主键，它就没有啦） | +> | transaction_id | 是 | 事务ID | +> | roll_pointer | 是 | DB_ROLL_PTR是一个回滚指针，用于配合undo日志，指向上一个旧版本 | + +2. **Undo日志**： + + - 每次数据更新时，InnoDB 会在 Undo 日志中记录旧版本数据。 + + - 如果需要读取旧版本数据，InnoDB 会通过 `roll_pointer` 找到 Undo 日志中的旧版本。 + +3. **一致性视图（Consistent Read View）**： + + - InnoDB 为每个事务创建一致性视图，记录当前活动的所有事务ID。 + + - 读取数据时，会根据一致性视图决定哪些版本的数据对当前事务可见。 + + + +在 MySQL 中，实际上每条记录在更新的时候都会同时记录一条回滚操作。记录上的最新值，通过回滚操作，都可以得到前一个状态的值。 + +假设一个值从 1 被按顺序改成了 2、3、4，在回滚日志里面就会有类似下面的记录。 + +![](https://img.starfish.ink/mysql/mvcc-read-view.png) + +当前值是 4，但是在查询这条记录的时候，不同时刻启动的事务会有不同的 read-view。如图中看到的，在视图 A、B、C 里面，这一个记录的值分别是 1、2、4，同一条记录在系统中可以存在多个版本，就是数据库的多版本并发控制（MVCC）。对于 read-view A，要得到 1，就必须将当前值依次执行图中所有的回滚操作得到。 + +同时你会发现，即使现在有另外一个事务正在将 4 改成 5，这个事务跟 read-view A、B、C 对应的事务是不会冲突的。 + + + +MySQL 的大多数事务型存储引擎实现都不是简单的行级锁。基于提升并发性考虑，一般都同时实现了多版本并发控制（MVCC），包括Oracle、PostgreSQL。只是实现机制各不相同。 + +可以认为 MVCC 是行级锁的一个变种，但它在很多情况下避免了加锁操作，因此开销更低。虽然实现机制有所不同，但大都实现了非阻塞的读操作，写操作也只是锁定必要的行。 + +MVCC 的实现是通过保存数据在某个时间点的快照来实现的。也就是说不管需要执行多长时间，每个事物看到的数据都是一致的。 + +典型的 MVCC 实现方式，分为**乐观（optimistic）并发控制和悲观（pressimistic）并发控制**。 + +下边通过 InnoDB 的简化版行为来说明 MVCC 是如何工作的。 + +InnoDB 的 MVCC，是通过在每行记录后面保存两个隐藏的列来实现。这两个列，一个保存了行的创建时间，一个保存行的过期时间（删除时间）。当然存储的并不是真实的时间，而是系统版本号（system version number）。每开始一个新的事务，系统版本号都会自动递增。事务开始时刻的系统版本号会作为事务的版本号，用来和查询到的每行记录的版本号进行比较。 + +**REPEATABLE READ（可重读）隔离级别下 MVCC 如何工作：** + +- **SELECT** + + InnoDB 会根据以下两个条件检查每行记录： + + - InnoDB 只查找版本早于当前事务版本的数据行，这样可以确保事务读取的行，要么是在开始事务之前已经存在要么是事务自身插入或者修改过的 + - 行的删除版本号要么未定义，要么大于当前事务版本号，这样可以确保事务读取到的行在事务开始之前未被删除 + +​ 只有符合上述两个条件的才会被查询出来 + +- **INSERT** + + InnoDB 为新插入的每一行保存当前系统版本号作为行版本号 + +- **DELETE** + + InnoDB 为删除的每一行保存当前系统版本号作为行删除标识 + +- **UPDATE** + + InnoDB 为插入的一行新记录，保存当前系统版本号作为行版本号，同时保存当前系统版本号到原来的行作为删除标识 + +保存这两个额外系统版本号，使大多数操作都不用加锁。使数据操作简单，性能很好，并且也能保证只会读取到符合要求的行。不足之处是每行记录都需要额外的存储空间，需要做更多的行检查工作和一些额外的维护工作。 + +MVCC 只在 COMMITTED READ（读提交）和 REPEATABLE READ（可重复读）两种隔离级别下工作。 + +> 所谓的`MVCC`，就是通过生成一个`ReadView`，然后通过`ReadView`找到符合条件的记录版本（历史版本是由`undo日志`构建的），其实就像是在生成`ReadView`的那个时刻做了一次时间静止（就像用相机拍了一个快照），查询语句只能读到在生成`ReadView`之前已提交事务所做的更改，在生成`ReadView`之前未提交的事务或者之后才开启的事务所做的更改是看不到的。而写操作肯定针对的是最新版本的记录，读记录的历史版本和改动记录的最新版本本身并不冲突，也就是采用`MVCC`时，`读-写`操作并不冲突。 + + + +## 四、事务的实现 + +> 事务的隔离性是通过锁实现，而事务的原子性、一致性和持久性则是通过事务日志实现 。 + +### 一致性读（Consistent Reads） + +事务利用`MVCC`进行的读取操作称为`一致性读`，或者`一致性无锁读`，有的地方也称之为`快照读`。所有普通的`SELECT`语句（`plain SELECT`）在`READ COMMITTED`、`REPEATABLE READ`隔离级别下都算是`一致性读`，比方说： + +```sql +SELECT * FROM t; +SELECT * FROM t1 INNER JOIN t2 ON t1.col1 = t2.col2 +``` + +`一致性读`并不会对表中的任何记录做`加锁`操作，其他事务可以自由的对表中的记录做改动。 + +### 事务日志 + +事务日志可以帮助提高事务效率： + +- 使用事务日志，存储引擎在修改表的数据时只需要修改其内存拷贝，再把该修改行为记录到持久在硬盘上的事务日志中，而不用每次都将修改的数据本身持久到磁盘。 +- 事务日志采用的是**追加**的方式，因此写日志的操作是磁盘上一小块区域内的顺序 I/O，而不像随机 I/O 需要在磁盘的多个地方移动磁头，所以采用事务日志的方式相对来说要快得多。 +- 事务日志持久以后，内存中被修改的数据在后台可以慢慢刷回到磁盘。 +- 如果数据的修改已经记录到事务日志并持久化，但数据本身没有写回到磁盘，此时系统崩溃，存储引擎在重启时能够自动恢复这一部分修改的数据。 + +目前来说，大多数存储引擎都是这样实现的，我们通常称之为**预写式日志**（Write-Ahead Logging），修改数据需要写两次磁盘。 + + + +事务的实现是基于数据库的存储引擎。不同的存储引擎对事务的支持程度不一样。MySQL 中支持事务的存储引擎有 InnoDB 和 NDB。 + +事务的实现就是如何实现 ACID 特性。 + +- **RR隔离级别下间隙锁才有效，RC隔离级别下没有间隙锁；** +- **RR隔离级别下为了解决“幻读”问题：“快照读”依靠MVCC控制，“当前读”通过间隙锁解决；** +- **间隙锁和行锁合称next-key lock，每个next-key lock是前开后闭区间；** +- **间隙锁的引入，可能会导致同样语句锁住更大的范围，影响并发度。** + + + +### 重做日志 + +**redo log（重做日志**） 实现持久化 + +在 InnoDB 的存储引擎中，事务日志通过重做(redo)日志和 InnoDB 存储引擎的日志缓冲(InnoDB Log Buffer)实现。事务开启时，事务中的操作，都会先写入存储引擎的日志缓冲中，在事务提交之前，这些缓冲的日志都需要提前刷新到磁盘上持久化，这就是 DBA 们口中常说的“日志先行”(Write-Ahead Logging)。 + +当事务提交之后，在 Buffer Pool 中映射的数据文件才会慢慢刷新到磁盘。此时如果数据库崩溃或者宕机，那么当系统重启进行恢复时，就可以根据 redo log 中记录的日志，把数据库恢复到崩溃前的一个状态。未完成的事务，可以继续提交，也可以选择回滚，这基于恢复的策略而定。 + +在系统启动的时候，就已经为 redo log 分配了一块连续的存储空间，以顺序追加的方式记录 Redo Log，通过顺序 I/O 来改善性能。所有的事务共享 redo log 的存储空间，它们的 redo log 按语句的执行顺序，依次交替的记录在一起。 + +> InnoDB 的 redo log 是固定大小的，比如可以配置为一组 4 个文件，每个文件的大小是 1GB，那么这块“粉板”总共就可以记录 4GB 的操作。从头开始写，写到末尾就又回到开头循环写，如下面这个图所示。 +> +> ![](https://static001.geekbang.org/resource/image/16/a7/16a7950217b3f0f4ed02db5db59562a7.png) +> +> write pos 是当前记录的位置，一边写一边后移，写到第 3 号文件末尾后就回到 0 号文件开头。checkpoint 是当前要擦除的位置，也是往后推移并且循环的，擦除记录前要把记录更新到数据文件。 +> +> write pos 和 checkpoint 之间的是“粉板”上还空着的部分，可以用来记录新的操作。如果 write pos 追上 checkpoint，表示“粉板”满了，这时候不能再执行新的更新，得停下来先擦掉一些记录，把 checkpoint 推进一下。 +> +> 有了 redo log，InnoDB 就可以保证即使数据库发生异常重启，之前提交的记录都不会丢失，这个能力称为**crash-safe**。 + + + +### 回滚日志 + +**undo log（回滚日志）** 实现原子性 + +undo log 主要为事务的回滚服务。在事务执行的过程中，除了记录 redo log，还会记录一定量的 undo log。undo log 记录了数据在每个操作前的状态，如果事务执行过程中需要回滚，就可以根据 undo log 进行回滚操作。单个事务的回滚，只会回滚当前事务做的操作，并不会影响到其他的事务做的操作。 + + + +二种日志均可以视为一种恢复操作，redo_log 是恢复提交事务修改的页操作，而 undo_log 是回滚行记录到特定版本。二者记录的内容也不同，redo_log 是**物理日志**，记录页的物理修改操作，而 undo_log 是**逻辑日志**，根据每行记录进行记录。 + +> 回滚日志可以**理解**为，我们在事务中使用的每一条 `INSERT` 都对应了一条 `DELETE`，每一条 `UPDATE` 也都对应一条相反的 `UPDATE` 语句。 +> +> 假设一个值从 1 被按顺序改成了 2、3、4，在回滚日志里面就会有类似下面的记录。 +> +> ![](https://img.starfish.ink/mysql/mvcc-read-view.png) +> +> 当前值是 4，但是在查询这条记录的时候，不同时刻启动的事务会有不同的 read-view。如图中看到的，在视图 A、B、C 里面，这一个记录的值分别是 1、2、4，同一条记录在系统中可以存在多个版本，就是数据库的多版本并发控制（MVCC）。对于 read-view A，要得到 1，就必须将当前值依次执行图中所有的回滚操作得到。 +> +> 同时你会发现，即使现在有另外一个事务正在将 4 改成 5，这个事务跟 read-view A、B、C 对应的事务是不会冲突的。 +> +> +> +> 在可重复读隔离级别下，事务在启动的时候就“拍了个快照”。注意，这个快照是基于整库的。 +> +> 这时，你会说这看上去不太现实啊。如果一个库有 100G，那么我启动一个事务，MySQL 就要拷贝 100G 的数据出来，这个过程得多慢啊。可是，我平时的事务执行起来很快啊。 +> +> 实际上，我们并不需要拷贝出这 100G 的数据。我们先来看看这个快照是怎么实现的。 +> +> InnoDB 里面每个事务有一个唯一的事务 ID，叫作 transaction id。它是在事务开始的时候向 InnoDB 的事务系统申请的，是按申请顺序严格递增的。 +> +> 而每行数据也都是有多个版本的。每次事务更新数据的时候，都会生成一个新的数据版本，并且把 transaction id 赋值给这个数据版本的事务 ID，记为 row trx_id。同时，旧的数据版本要保留，并且在新的数据版本中，能够有信息可以直接拿到它。 +> +> 也就是说，数据表中的一行记录，其实可能有多个版本 (row)，每个版本有自己的 row trx_id。 +> +> 如图 2 所示，就是一个记录被多个事务连续更新后的状态。 +> +> ![](https://static001.geekbang.org/resource/image/68/ed/68d08d277a6f7926a41cc5541d3dfced.png) +> +> 图中虚线框里是同一行数据的 4 个版本，当前最新版本是 V4，k 的值是 22，它是被 transaction id 为 25 的事务更新的，因此它的 row trx_id 也是 25 +> +> 你可能会问，前面的文章不是说，语句更新会生成 undo log（回滚日志）吗？那么，**undo log 在哪呢？** +> +> 实际上，图 2 中的三个虚线箭头，就是 undo log；而 V1、V2、V3 并不是物理上真实存在的，而是每次需要的时候根据当前版本和 undo log 计算出来的。比如，需要 V2 的时候，就是通过 V4 依次执行 U3、U2 算出来。 +> +> 明白了多版本和 row trx_id 的概念后，我们再来想一下，InnoDB 是怎么定义那个“100G”的快照的。 +> +> 按照可重复读的定义，一个事务启动的时候，能够看到所有已经提交的事务结果。但是之后，这个事务执行期间，其他事务的更新对它不可见。 +> +> 因此，一个事务只需要在启动的时候声明说，“**以我启动的时刻为准，如果一个数据版本是在我启动之前生成的，就认；如果是我启动以后才生成的，我就不认，我必须要找到它的上一个版本**”。 +> +> 在实现上， InnoDB 为每个事务构造了一个数组，用来保存这个事务启动瞬间，当前正在“活跃”的所有事务 ID。“活跃”指的就是，启动了但还没提交。 +> +> 数组里面事务 ID 的最小值记为低水位，当前系统里面已经创建过的事务 ID 的最大值加 1 记为高水位。 +> +> 这个视图数组和高水位，就组成了当前事务的一致性视图（read-view）。 +> +> 而数据版本的可见性规则，就是基于数据的 row trx_id 和这个一致性视图的对比结果得到的。 +> +> 这个视图数组把所有的 row trx_id 分成了几种不同的情况。 +> +> ![](https://static001.geekbang.org/resource/image/88/5e/882114aaf55861832b4270d44507695e.png) +> +> + + + + + + + +在数据库系统中，事务的原子性和持久性是由事务日志（transaction log）保证的，在实现时也就是上面提到的两种日志，前者用于对事务的影响进行撤销，后者在错误处理时对已经提交的事务进行重做，它们能保证两点： + +1. 发生错误或者需要回滚的事务能够成功回滚（原子性）； +2. 在事务提交后，数据没来得及写会磁盘就宕机时，在下次重新启动后能够成功恢复数据（持久性）； + + + +### MySQL对分布式事务的支持 + +[官方分布式事务文档](https://dev.mysql.com/doc/refman/5.7/en/xa.html ) + +分布式事务的实现方式有很多，既可以采用innoDB提供的原生的事务支持，也可以采用消息队列来实现分布式事务的最终一致性。这里我们主要聊一下innoDB对分布式事务的支持。 + +MySQL 从 5.0.3 开始支持分布式事务，**当前分布式事务只支持 InnoDB 存储引擎**。一个分布式事务会涉及多个行动，这些行动本身是事务性的。所有行动都必须一起成功完成，或者一起被回滚。 + +![img](../../_images/mysql/mysql-xa-transactions.png) + +如图，mysql的分布式事务模型。模型中分三块：应用程序（AP）、资源管理器（RM）、事务管理器（TM）: + +- 应用程序：定义了事务的边界，指定需要做哪些事务； +- 资源管理器：提供了访问事务的方法，通常一个数据库就是一个资源管理器； +- 事务管理器：协调参与了全局事务中的各个事务。 + +分布式事务采用两段式提交（two-phase commit）的方式： + +- 第一阶段所有的事务节点开始准备，告诉事务管理器ready。 +- 第二阶段事务管理器告诉每个节点是commit还是rollback。如果有一个节点失败，就需要全局的节点全部rollback，以此保障事务的原子性。 + +分布式事务（XA 事务）的 SQL 语法主要包括： + +```mysql +XA {START|BEGIN} xid [JOIN|RESUME] +``` + +虽然 MySQL 支持分布式事务，但是在测试过程中，还是发现存在一些问题： +如果分支事务在达到 prepare 状态时，数据库异常重新启动，服务器重新启动以后，可以继续对分支事务进行提交或者回滚得操作，但是提交的事务没有写 binlog，存在一定的隐患，可能导致使用 binlog 恢复丢失部分数据。如果存在复制的数据库，则有可能导致主从数据库的数据不一致。 + +如果分支事务在执行到 prepare 状态时，数据库异常，且不能再正常启动，需要使用备份和 binlog 来恢复数据，那么那些在 prepare 状态的分支事务因为并没有记录到 binlog，所以不能通过 binlog 进行恢复，在数据库恢复后，将丢失这部分的数据。 + +如果分支事务的客户端连接异常中止，那么数据库会自动回滚未完成的分支事务，如果此时分支事务已经执行到 prepare 状态， 那么这个分布式事务的其他分支可能已经成功提交，如果这个分支回滚，可能导致分布式事务的不完整，丢失部分分支事务的内容。 +总之， MySQL 的分布式事务还存在比较严重的缺陷， 在数据库或者应用异常的情况下，可能会导致分布式事务的不完整。如果应用对于数据的完整性要求不是很高，则可以考虑使用。如果应用对事务的完整性有比较高的要求，那么对于当前的版本，则不推荐使用分布式事务。 + + + +## 总结 + + + + + +## References + +- [『浅入深出』MySQL 中事务的实现](https://draveness.me/mysql-transaction/) +- [数据库事务与MySQL事务总结](https://zhuanlan.zhihu.com/p/29166694) + + + + + + \ No newline at end of file diff --git a/docs/data-management/MySQL/MySQL-select.md b/docs/data-management/MySQL/MySQL-select.md new file mode 100644 index 0000000000..ac0c35efdd --- /dev/null +++ b/docs/data-management/MySQL/MySQL-select.md @@ -0,0 +1,287 @@ +--- +title: MySQL 查询 +date: 2023-03-31 +tags: + - MySQL +categories: MySQL +--- + +![](https://img.starfish.ink/mysql/banner-mysql-select.png) + +![The Essential Guide to SQL’s Execution Order](https://www.kdnuggets.com/wp-content/uploads/ferrer_essential_guide_sql_execution_order_1.png) + +> `SQL`的全称是`Structured Query Language`，翻译后就是`结构化查询语言`。 + +## Order By + +在开发应用的时候，一定会经常碰到需要根据指定的字段排序来显示结果的需求。 + +```mysql +CREATE TABLE `t` ( + `id` int(11) NOT NULL, + `city` varchar(16) NOT NULL, + `name` varchar(16) NOT NULL, + `age` int(11) NOT NULL, + PRIMARY KEY (`id`), + KEY `city` (`city`) +) ENGINE=InnoDB; +``` + +业务需求来了， + +![](/Users/starfish/Documents/截图/截屏2023-04-19 09.33.52.png) + +Extra 这个字段中的“Using filesort”表示的就是需要排序，MySQL 会给每个线程分配一块内存用于排序，称为 sort_buffer。 + +```mysql +mysql> show variables like 'sort_buffer_size'; ++------------------+--------+ +| Variable_name | Value | ++------------------+--------+ +| sort_buffer_size | 262144 | ++------------------+--------+ +1 row in set (0.00 sec) +``` + +### 全字段排序 + +这个语句的大概执行流程是这样的： + +1. 初始化 sort_buffer，确定放入 name、city、age 这三个字段； +2. 从索引 city 找到第一个满足 city='北京’ 条件的主键 id； +3. 到主键 id 索引取出整行，取 name、city、age 三个字段的值，存入 sort_buffer 中； +4. 从索引 city 取下一个记录的主键 id； +5. 重复步骤 3、4 直到 city 的值不满足查询条件为止； +6. 对 sort_buffer 中的数据按照字段 name 做快速排序； +7. 按照排序结果取前 3 行返回给客户端。 + +我们暂且把这个排序过程，称为『全字段排序』 + +“按 name 排序”这个动作，可能在内存中完成，也可能需要使用外部排序，这取决于排序所需的内存和参数 sort_buffer_size。 + +sort_buffer_size，就是 MySQL 为排序开辟的内存（sort_buffer）的大小。如果要排序的数据量小于 sort_buffer_size，排序就在内存中完成。但如果排序数据量太大，内存放不下，则不得不利用磁盘临时文件辅助排序。 + +### rowid 排序 + +在上面这个算法过程里面，只对原表的数据读了一遍，剩下的操作都是在 sort_buffer 和临时文件中执行的。但这个算法有一个问题，就是如果查询要返回的字段很多的话，那么 sort_buffer 里面要放的字段数太多，这样内存里能够同时放下的行数很少，要分成很多个临时文件，排序的性能会很差。 + +所以如果单行很大，这个方法效率不够好。 + +那么，**如果 MySQL 认为排序的单行长度太大会怎么做呢？** + +接下来，我来修改一个参数，让 MySQL 采用另外一种算法。 + +```mysql +SET max_length_for_sort_data = 16; +``` + +max_length_for_sort_data，是 MySQL 中专门控制用于排序的行数据的长度的一个参数。它的意思是，如果单行的长度超过这个值，MySQL 就认为单行太大，要换一个算法。 + +city、name、age 这三个字段的定义总长度是 36，我把 max_length_for_sort_data 设置为 16，我们再来看看计算过程有什么改变。 + +新的算法放入 sort_buffer 的字段，只有要排序的列（即 name 字段）和主键 id。 + +但这时，排序的结果就因为少了 city 和 age 字段的值，不能直接返回了，整个执行流程就变成如下所示的样子： + +1. 初始化 sort_buffer，确定放入两个字段，即 name 和 id； +2. 从索引 city 找到第一个满足 city='杭州’条件的主键 id； +3. 到主键 id 索引取出整行，取 name、id 这两个字段，存入 sort_buffer 中； +4. 从索引 city 取下一个记录的主键 id； +5. 重复步骤 3、4 直到不满足 city='杭州’条件为止； +6. 对 sort_buffer 中的数据按照字段 name 进行排序； +7. 遍历排序结果，取前 3 行，并按照 id 的值回到原表中取出 city、name 和 age 三个字段返回给客户端。 + +对全字段排序流程你会发现，rowid 排序多访问了一次表 t 的主键索引，就是步骤 7。 + + + +### 全字段排序 VS rowid 排序 + +如果 MySQL 实在是担心排序内存太小，会影响排序效率，才会采用 rowid 排序算法，这样排序过程中一次可以排序更多行，但是需要再回到原表去取数据。 + +如果 MySQL 认为内存足够大，会优先选择全字段排序，把需要的字段都放到 sort_buffer 中，这样排序后就会直接从内存里面返回查询结果了，不用再回到原表去取数据。 + +这也就体现了 MySQL 的一个设计思想：**如果内存够，就要多利用内存，尽量减少磁盘访问。** + + + +当然，如果我们创建了覆盖索引，就不需要再排序了 + +```mysql +alter table t add index city_user_age(city, name, age); +``` + +![](/Users/starfish/Documents/截图/截屏2023-04-19 10.21.30.png) + + + +## 常见通用的 Join 查询 + +### SQL执行顺序 + +- 手写 + + ```mysql + SELECT DISTINCT + FROM + JOIN ON + WHERE + GROUP BY + HAVING + ORDER BY + LIMIT + ``` + +- 机读 + + ```mysql + FROM + ON + JOIN + WHERE + GROUP BY + HAVING + SELECT + DISTINCT + ORDER BY + LIMIT + ``` + +- 总结 + + ![The Essential Guide to SQL’s Execution Order](https://img.starfish.ink/mysql/ferrer_essential_guide_sql_execution_order_6.png) + +### Join图 + +![sql-joins](https://img.starfish.ink/mysql/sql-joins.jpg) + +### demo + +#### 建表SQL + +```plsql +CREATE TABLE `tbl_dept` ( + `id` INT(11) NOT NULL AUTO_INCREMENT, + `deptName` VARCHAR(30) DEFAULT NULL, + `locAdd` VARCHAR(40) DEFAULT NULL, + PRIMARY KEY (`id`) +) ENGINE=INNODB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8; + +CREATE TABLE `tbl_emp` ( + `id` INT(11) NOT NULL AUTO_INCREMENT, + `name` VARCHAR(20) DEFAULT NULL, + `deptId` INT(11) DEFAULT NULL, + PRIMARY KEY (`id`), + KEY `fk_dept_id` (`deptId`) + #CONSTRAINT `fk_dept_id` FOREIGN KEY (`deptId`) REFERENCES `tbl_dept` (`id`) +) ENGINE=INNODB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8; + + +INSERT INTO tbl_dept(deptName,locAdd) VALUES('RD',11); +INSERT INTO tbl_dept(deptName,locAdd) VALUES('HR',12); +INSERT INTO tbl_dept(deptName,locAdd) VALUES('MK',13); +INSERT INTO tbl_dept(deptName,locAdd) VALUES('MIS',14); +INSERT INTO tbl_dept(deptName,locAdd) VALUES('FD',15); +INSERT INTO tbl_emp(NAME,deptId) VALUES('z3',1); +INSERT INTO tbl_emp(NAME,deptId) VALUES('z4',1); +INSERT INTO tbl_emp(NAME,deptId) VALUES('z5',1); +INSERT INTO tbl_emp(NAME,deptId) VALUES('w5',2); +INSERT INTO tbl_emp(NAME,deptId) VALUES('w6',2); +INSERT INTO tbl_emp(NAME,deptId) VALUES('s7',3); +INSERT INTO tbl_emp(NAME,deptId) VALUES('s8',4); +INSERT INTO tbl_emp(NAME,deptId) VALUES('s9',51); + +``` + +#### 7种JOIN + +1. A、B两表共有 + + ```mysql + select * from tbl_emp a **inner join** tbl_dept b on a.deptId = b.id; + ``` + +2. A、B两表共有+A的独有 + + ```mysql + select * from tbl_emp a **left join** tbl_dept b on a.deptId = b.id; + ``` + +3. A、B两表共有+B的独有 + + ```mysql + select * from tbl_emp a **right join** tbl_dept b on a.deptId = b.id; + ``` + +4. A的独有 + + ```mysql + select * from tbl_emp a left join tbl_dept b on a.deptId = b.id where b.id is null; + ``` + +5. B的独有 + + ```mysql + select * from tbl_emp a right join tbl_dept b on a.deptId = b.id where a.deptId is null; + ``` + +6. AB全有 + + +**MySQL Full Join的实现 因为MySQL不支持FULL JOIN,替代方法:left join + union(可去除重复数据)+ right join** + + ```mysql + SELECT * FROM tbl_emp A LEFT JOIN tbl_dept B ON A.deptId = B.id + UNION + SELECT * FROM tbl_emp A RIGHT JOIN tbl_dept B ON A.deptId = B.id + ``` + +7. A的独有+B的独有 + + ```mysql + SELECT * FROM tbl_emp A LEFT JOIN tbl_dept B ON A.deptId = B.id WHERE B.`id` IS NULL + UNION + SELECT * FROM tbl_emp A RIGHT JOIN tbl_dept B ON A.deptId = B.id WHERE A.`deptId` IS NULL; + ``` + + + +## count() + +count() 是一个聚合函数，对于返回的结果集，一行行地判断，如果 count 函数的参数不是 NULL，累计值就加 1，否则不加。最后返回累计值。 + +### count(*) 的实现方式 + +你首先要明确的是，在不同的 MySQL 引擎中，count(*) 有不同的实现方式。 + +- MyISAM 引擎把一个表的总行数存在了磁盘上，因此执行 count(*) 的时候会直接返回这个数，效率很高； +- 而 InnoDB 引擎就麻烦了，它执行 count(*) 的时候，需要把数据一行一行地从引擎里面读出来，然后累积计数。 + +> 那**为什么 InnoDB 不跟 MyISAM 一样，也把数字存起来呢？** +> +> 这是因为即使是在同一个时刻的多个查询，由于多版本并发控制（MVCC）的原因，InnoDB 表“应该返回多少行”也是不确定的。 + + + + + +**对于 count(主键 id) 来说**，InnoDB 引擎会遍历整张表，把每一行的 id 值都取出来，返回给 server 层。server 层拿到 id 后，判断是不可能为空的，就按行累加。 + +**对于 count(1) 来说**，InnoDB 引擎遍历整张表，但不取值。server 层对于返回的每一行，放一个数字“1”进去，判断是不可能为空的，按行累加。 + +单看这两个用法的差别的话，你能对比出来，count(1) 执行得要比 count(主键 id) 快。因为从引擎返回 id 会涉及到解析数据行，以及拷贝字段值的操作。 + +**对于 count(字段) 来说**： + +1. 如果这个“字段”是定义为 not null 的话，一行行地从记录里面读出这个字段，判断不能为 null，按行累加； +2. 如果这个“字段”定义允许为 null，那么执行的时候，判断到有可能是 null，还要把值取出来再判断一下，不是 null 才累加。 + + + +按照效率排序的话，count(字段) 数据库的设计三范式: +> +> - 1NF：字段不可分。第一范式要求数据原子化 +> - 2NF：唯一性 一个表只说明一个事物。有主键，非主键字段依赖主键。第二范式消除部分依赖 +> - 3NF：非主键字段不能相互依赖。第三范式消除传递依赖，从而提高数据的完整性和一致性 + +在当今信息化时代，数据库已经成为各行各业中不可或缺的一部分。从小型应用到企业级系统，数据库都在背后默默支撑着数据的存储、管理与分析。而在数据库设计中，**三范式**（3NF）是关系型数据库模型设计中最为基础和重要的规范化原则之一。三范式的核心目标是通过规范化数据库的结构，减少冗余数据，增强数据一致性和完整性，避免更新异常，确保数据的高效存储与访问。 + +### 什么是数据库规范化？ + +数据库规范化（Normalization）是指对数据库表的设计进行优化的过程，通过一系列规则来消除冗余数据，并提高数据的组织方式。规范化不仅帮助减少数据的重复性，还确保数据之间的逻辑关系清晰，减少因数据修改引发的不一致问题。 + +规范化的核心思想是通过分解数据库表，降低数据冗余性和依赖性。在实际操作中，数据库规范化通常分为多个阶段，称为**范式**。每一范式都是在前一范式的基础上进一步精化和完善的。三范式是数据库设计中最为基础且常用的范式，它通过消除部分依赖和传递依赖等问题，保证数据库的高效性与一致性。 + +### 第一范式（1NF） + +**第一范式**是数据库规范化的最基础范式，它的要求是：**表中的每个列必须包含原子值，即每个字段只能存储不可再分的单一数据单元**。简而言之，1NF 要求每个列的数据必须是“原子的”，也就是没有重复数据和多值属性。 + +#### 1NF的具体要求： + +1. 每列中的数据必须是不可分割的单一数据项。 +2. 每一行必须唯一，不允许有重复的记录。 +3. 表中的每个字段都必须具有唯一的标识符（即主键）。 + +**示例**： 假设我们有一个学生信息表，如下所示： + +| 学生ID | 姓名 | 电话号码 | +| ------ | ---- | ---------------------- | +| 1 | 张三 | 1234567890, 0987654321 | +| 2 | 李四 | 1357924680, 2468013579 | + +从上表可以看出，"电话号码"字段存储了多个值，违反了1NF。要满足1NF要求，电话号码字段应拆分为多行或多列，像这样： + +| 学生ID | 姓名 | 电话号码 | +| ------ | ---- | ---------- | +| 1 | 张三 | 1234567890 | +| 1 | 张三 | 0987654321 | +| 2 | 李四 | 1357924680 | +| 2 | 李四 | 2468013579 | + +这样，每个字段就变成了原子值，符合了1NF的要求。 + +### 第二范式（2NF） + +第二范式建立在第一范式的基础之上，要求**消除部分依赖**。具体而言，2NF要求表中的所有非主属性（即不参与主键的字段）必须完全依赖于主键，而不能仅依赖于主键的一部分。换句话说，表中的每一个非主属性必须依赖于完整的主键，而不是主键的某一部分。 + +#### 2NF的要求： + +1. 表必须满足1NF。 +2. 表中的每个非主键列必须完全依赖于整个主键。 +3. 如果表有复合主键（由多个列组成的主键），则不能有部分依赖。 + +**示例**： 考虑以下的订单表： + +| 订单ID | 产品ID | 产品名称 | 数量 | 单价 | +| ------ | ------ | -------- | ---- | ---- | +| 1 | 101 | 手机 | 2 | 2000 | +| 1 | 102 | 耳机 | 1 | 500 | +| 2 | 101 | 手机 | 1 | 2000 | + +在这个例子中，复合主键由**订单ID**和**产品ID**组成。虽然**产品名称**只依赖于**产品ID**，但它却被存储在了当前的表中，这种依赖关系违反了2NF。我们可以通过拆分表格来消除部分依赖： + +**订单表：** + +| 订单ID | 产品ID | 数量 | 单价 | +| ------ | ------ | ---- | ---- | +| 1 | 101 | 2 | 2000 | +| 1 | 102 | 1 | 500 | +| 2 | 101 | 1 | 2000 | + +**产品表：** + +| 产品ID | 产品名称 | +| ------ | -------- | +| 101 | 手机 | +| 102 | 耳机 | + +通过这样的拆分，我们确保了所有非主键字段完全依赖于复合主键，符合了2NF。 + +### 第三范式（3NF） + +第三范式是在第二范式的基础上进一步规范化，它要求消除**传递依赖**。传递依赖是指某个非主键列依赖于另一个非主键列，而这个非主键列又依赖于主键。换句话说，第三范式要求表中的每个非主属性都直接依赖于主键，而不是间接依赖。 + +#### 3NF的要求： + +1. 表必须满足2NF。 +2. 表中的每个非主键列必须直接依赖于主键，不能依赖于其他非主键列。 + +**示例**： 考虑以下员工表： + +| 员工ID | 部门ID | 部门名称 | 部门经理 | +| ------ | ------ | -------- | -------- | +| 1 | 101 | 销售部 | 王经理 | +| 2 | 102 | 技术部 | 张经理 | + +在这个例子中，**部门名称**和**部门经理**依赖于**部门ID**，而**部门ID**又依赖于**员工ID**，这就构成了传递依赖。为了满足3NF，我们应该将部门相关信息提取到单独的表中： + +**员工表：** + +| 员工ID | 部门ID | +| ------ | ------ | +| 1 | 101 | +| 2 | 102 | + +**部门表：** + +| 部门ID | 部门名称 | 部门经理 | +| ------ | -------- | -------- | +| 101 | 销售部 | 王经理 | +| 102 | 技术部 | 张经理 | + +通过这样的拆分，我们消除了传递依赖，确保了表符合3NF的要求。 + +### 规范化的优点与实际应用 + +数据库规范化的主要优点是减少冗余数据，提高数据的一致性和完整性。通过规范化，数据的修改操作变得更加简便，不容易出现更新异常，例如**插入异常**、**删除异常**和**更新异常**。 + +然而，在实际应用中，数据库设计并非总是严格遵循最高范式。过度规范化可能会导致表的拆分过多，从而影响查询性能。在这种情况下，数据库设计师可能会选择在一定程度上**反规范化**，即故意增加一些冗余数据，以提高查询效率。 + +### 总结 + +数据库的三范式是关系型数据库设计的基础，它通过消除数据冗余和不必要的依赖关系，帮助提高数据的一致性和完整性。第一范式要求数据原子化，第二范式消除部分依赖，而第三范式消除传递依赖。在实际的数据库设计中，三范式为我们提供了科学的规范化思路，但在一些特定场景下，可能需要适当进行反规范化来优化查询性能。 + +了解并掌握三范式的概念，能帮助开发人员在设计数据库时做出更加高效且一致的决策，提升数据库系统的整体性能和可维护性。 + diff --git a/docs/data-store/MySQL/readMySQL.md b/docs/data-management/MySQL/readMySQL.md similarity index 86% rename from docs/data-store/MySQL/readMySQL.md rename to docs/data-management/MySQL/readMySQL.md index 20e4d4e390..c8f198ff9f 100644 --- a/docs/data-store/MySQL/readMySQL.md +++ b/docs/data-management/MySQL/readMySQL.md @@ -3,9 +3,6 @@ ---
- -![img](../../_images/mysql/mysql.png) -

@@ -26,7 +23,7 @@ MySQL是个啥，就说一句话——**MySQL是一个关系型数据库管理 - +![](/Users/starfish/oceanus/picBed/mysql/question-list.png) diff --git "a/docs/data-management/MySQL/reproduce/Int(4)\345\222\214Int(11) \351\200\211\345\223\252\344\270\252\357\274\237.md" "b/docs/data-management/MySQL/reproduce/Int(4)\345\222\214Int(11) \351\200\211\345\223\252\344\270\252\357\274\237.md" new file mode 100644 index 0000000000..50cac305fe --- /dev/null +++ "b/docs/data-management/MySQL/reproduce/Int(4)\345\222\214Int(11) \351\200\211\345\223\252\344\270\252\357\274\237.md" @@ -0,0 +1,143 @@ +# Int(4)和Int(11) 选哪个？ + +### 缘起 + +大家平时在进行数据库设计的时候，如果遇到需要存储整数类型的数据的时候，通常会优先使用Int这个整数类型，在处理20亿级别的正负数值存储上，Int类型是完全能够满足日常需求的了。 + +但是在进行数据库建表语句书写的时候，大家经常会见到Int类型的后面会带上1个括号，里面跟上1个数值，通常要么是4，要么是11。如下： + +![](https://tva1.sinaimg.cn/large/008i3skNly1gs5wsboui1j30kw05wq3z.jpg) + +![](https://tva1.sinaimg.cn/large/008i3skNly1gs5wt6igczj30n30a0jtv.jpg) + +这 int 括号里面的数值，究竟是什么意思呢？有的开发者认为，这个数值是用来限制Int类型能够存储的数字的长度的( 类似char、varchar括号数值 )；有的开发者则认为，在存储相同数字的情况下，Int(4)会比Int(11)在存储上节省更多的存储空间。 + +那么实际情况究竟是怎样的呢？在实际的数据库设计中，究竟应该使用Int(4)还是Int(11)呢？又或者是应该什么都不写呢，只用默认的Int呢？ + +让我们开启今天的MySQL数据库之Int类型之旅。^_^ + +### 存储比较 + +为了方便测试Int(4)、Int(11)、以及默认的Int，在存储上是否存在差别，我们分别创建t_int_four、t_int_eleven、t_int_default表，如下： + +![](https://tva1.sinaimg.cn/large/008i3skNly1gs5wu71qc3j309509egm8.jpg) + +接下来，我们向t_int_four表的my_int_four字段，插入Int( 有符号 )类型的最大值2147483647，如下： + +![](https://tva1.sinaimg.cn/large/008i3skNly1gs5wutnlz1j30nt0b3t9z.jpg) + +如上图，我们看到m_int_four字段的Int(4)类型，并没有影响到Int类型最大值2147483647的插入。可见这个Int括号中的数值4，并不是对数字长度的存储进行限制的，也就是说，只要不超过Int( 有符号 )类型的最小值和最大值的范围，都是可以正确存储的。 + +接下来我们向t_int_eleven表的my_int_eleven字段，插入Int类型的最大值2147483647。如下： + +![](https://tva1.sinaimg.cn/large/008i3skNly1gs5wvf5stlj30mh0eawgh.jpg) + +如上图，我们看到，my_int_eleven字段的Int(11)，也成功存储了Int类型的最大值2147483647。 + +接下来我们向t_int_default表的my_int_default字段，插入Int类型的最大值。如下： + +![](https://tva1.sinaimg.cn/large/008i3skNly1gs5ww32k2ej30my0evmza.jpg) + +如上图，my_int_default字段的Int类型，也成功存储了Int类型的最大值。这也就进一步证明了，Int类型括号中的数值，对该列字段的数值的存储长度是没有任何影响的，只要不超出Int类型的数值范围，都是可以被正确存储的。 + +这里我们发现Int类型的最大值2147483647，是一个10位长度的数字，那么my_int_eleven字段的Int(11)，能否突破Int类型的最大值，存储11位长度的数值呢？ + +我们存入一个11位的数字，如下： + +![](https://tva1.sinaimg.cn/large/008i3skNly1gs5wwqwaq7j30mo0caabr.jpg) + +如上图，这里我们发现，即便设置了Int(11)的列字段，依然无法突破Int类型的数值范围存储限制，最终还是只允许存储Int( 有符号 )类型的有效数值范围。 + +那么Int(4)、Int(11）、以及默认的Int，在存储空间的占用上是否存在差别呢？是否相同位数长度的数值，Int(4)就比Int(11)节省更多的物理存储空间呢？ + +接下来，我们打开磁盘上的表t_int_four、t_int_eleven、t_int_default这3个表的表空间文件( 后缀名是.ibd )，打开进行对比，如下： + +![](https://tva1.sinaimg.cn/large/008i3skNly1gs5wx9o043j30gu0eyjwt.jpg) + +![](https://tva1.sinaimg.cn/large/008i3skNly1gs5x5pjz7nj30gr0ey79c.jpg) + +![](https://tva1.sinaimg.cn/large/008i3skNly1gs5wxsw8r9j30gr0fvdla.jpg) + +从上图的元数据中，我们看到不管是t_int_four的Int(4)，还是t_int_eleven的Int(11)，甚至是t_int_default的Int，在存储空间占用上，都是用了4个字节的空间大小，这里的FF FF FF FF，就是我们所存储的Int类型的最大值2147483647，如下： + +![](https://tva1.sinaimg.cn/large/008i3skNly1gs5wyahmyjj308w06mglr.jpg) + +由于我们的Int类型是有符号的，也就是能存储负数。所以这里的4294967295需要除以2，得到有符号的正数2147483647，就是Int( 有符号 )类型的最大值了。 + +![](https://tva1.sinaimg.cn/large/008i3skNly1gs5wytp4lcj308w06jjrk.jpg) + +关于有符号整数的存储及计算方法，大家可以在网上自行查询脑补，这里就不再陈述了。 + +### ZEROFILL + +经过上面的示例，我们已经知道Int类型括号中的数值，并不是控制录入数据的数值位数长度的，那么它究竟是用来干什么的呢？ + +在《MySQL中文参考手册》中，数值类型的列字段，都有一个叫做ZEROFILL的可选属性，如下： + +![](https://tva1.sinaimg.cn/large/008i3skNly1gs5x002vtvj30nz0gyn03.jpg) + +按照文档中的介绍，如果一个数值类型的列字段，加上了ZEROFILL的属性后，该列类型会自动变为Unsigned( 无符号 )类型，并具备自动补0的功能。 + +那么究竟是什么样的效果呢？下面我们看一个示例。 + +![](https://tva1.sinaimg.cn/large/008i3skNly1gs5x0j3qlnj30ob0e5di8.jpg) + +如上图，在t_int_zerofill表中，我们分别创建了表示Int(4)、Int(11)、Int的3个字段：my_int_four、my_int_eleven、my_int_default。 + +我们在插入了1条每个字段值为数字1的数据后，发现查询出来的结果表中，自动在每个Int类型的字段列上，补了数字0。使得每一个数字列中的数值长度，正好等于该列字段Int括号中数值的长度。如下： + +![](https://tva1.sinaimg.cn/large/008i3skNly1gs5x11gmn3j30gx03vt97.jpg) + +上面表格中，我们发现，如果Int类型的列字段中，存储的数值的位长度，小于Int括号中的数值( 后面统一叫做ZEROFILL长度 )，MySQL在查询显示的时候，就会自动在该列的存储数值的左边，进行补0。使得整个显示值的总长度，等于Int列类型的ZEROFILL长度。如果使用的是Int默认类型，则按照Int( 无符号 )类型存储的最大数值的位长度进行补0，这里Int( 无符号 )类型的最大值为4294967295，也就是10位。所以my_int_default在显示数字1时，补9位0+1位数字(1)，正好是10位。 + +接下来，我们插入1条各字段数值为1234的数据，如下： + +![](https://tva1.sinaimg.cn/large/008i3skNly1gs5x1jofo1j30o80dlgne.jpg) + +如上图，这里我们看到，在插入1234之后，my_int_four的Int(4)，就没有再进行补0了，因为数字1234的位长度，正好等于my_int_four列字段的ZEROFILL长度，也就是Int(4)。而其它列my_int_eleven和my_int_default，依然按照各自的ZEROFILL长度进行补0显示。 + +### 混合示例 + +经过上面的示例，我们知道Int类型的ZEROFILL长度参数的用法，那么其他的数值类型，是否也同样使用ZEROFILL长度参数的用法规则呢？ + +我们创建t_int_complex_zerofill表，并设置不同的数值类型的列字段，如下： + +![](https://tva1.sinaimg.cn/large/008i3skNly1gs5x20eygfj30az050q3g.jpg) + +接下来，插入1条测试数据，分别为每一个列字段，设置该数值列类型的无符号最大数值，如下： + +![](https://tva1.sinaimg.cn/large/008i3skNly1gs5x2gu6mmj60x80extc902.jpg) + +如上图，各种数值类型的ZEROFILL长度参数，依然对数值列类型本身的存储大小，是没有影响的。 + +接下来我们插入1条，各列字段数值为1的测试数据，如下： + +![](https://tva1.sinaimg.cn/large/008i3skNly1gs5x2znti6j30us0dodi4.jpg) + +如上图，各种数字类型的ZEROFILL长度参数都起到了预期的补0作用。 + +### 总结 + +本篇主要针对MySQL数据库设计中，Int类型的列字段中，括号中不同补0长度的数值设置，以及其他具备相似特性的数值列类型，进行对了对比和分析。 + +经过不同的示例分析，我们知道了数值列类型的补0长度的数值设置，也就是ZEROFILL的长度参数设置，对数值列类型本身的存储是没有任何影响的。 + +在数值列类型的字段上，如果没有显示声明“ZEROFILL”标识的话，只要存储的数值不超过该数字列类型( 有符号 )的数值范围，就都可以正确存储。也就是说Int(4)和Int(11)在存储大小上，是没有任何差别和限制的。 + +数值列类型的补0长度的数值设置，只有在该数值列类型的字段上，显示声明“ZEROFILL”标识之后，才会按照数值列类型的补0长度( ZEROFILL长度参数 )，进行补0显示。 + +那么为什么在很多MySQL的建表语句中，会经常见到没有ZEROFILL标识的Int(4)和Int(11)的写法呢？ + +阿K认为，这里可能主要有2种方向的考虑。 + +- Int(11)：因为Int( 有符号 )类型的最大数值为2147483647，位长度是10。那么在存储的时候，就能从括号中看出来，在进行数据插入的时候，就不要输入11位长度的数字，比如：12345678901，就是超出范围的非法数据。用作数字插入的预警作用。 + +- Int(4)：因为Int列类型的存储空间大小为4字节，在设计和存储的时候，就能够从括号中看出来Int列类型的占用空间大小，从而结合char、varchar等其它列的类型，方便的计算出来每条数据在插入的时候，所占用存储空间的大小，从而帮助开发者进行存储优化和数据库表优化。比如上面的t_int_complex_zerofill示例表， + +![](https://tva1.sinaimg.cn/large/008i3skNly1gs5x42w3xrj30az050q3g.jpg) + +我们很容易就能从各个数值列字段的ZEROFILL长度中，累加计算出，每向表中增加1条数据，就会用掉18个字节的存储空间，公式如下： + + TINYINT(1) + SMALLINT(2) + MEDIUMINT(3) + INT(4) + BIGINT(8) = 18字节 + +在实际的数据库设计开发中，每位设计者的观点和想法都不尽相同，都有自己的设计考量。关于Int数值类型的字段设计，究竟Int(4)和Int(11)谁更好更美呢？作为开发人员的您，又是怎么看待它们的呢？ diff --git a/docs/data-management/MySQL/reproduce/MySQL-count.md b/docs/data-management/MySQL/reproduce/MySQL-count.md new file mode 100644 index 0000000000..aa359617cd --- /dev/null +++ b/docs/data-management/MySQL/reproduce/MySQL-count.md @@ -0,0 +1,181 @@ +> 《MySQL 实战45 讲》 + +在开发系统的时候，你可能经常需要计算一个表的行数，比如一个交易系统的所有变更记录总数。这时候你可能会想，一条 select count(*) from t 语句不就解决了吗？ + +但是，你会发现随着系统中记录数越来越多，这条语句执行得也会越来越慢。然后你可能就想了，MySQL 怎么这么笨啊，记个总数，每次要查的时候直接读出来，不就好了吗。 + +那么今天，我们就来聊聊 count(*) 语句到底是怎样实现的，以及 MySQL 为什么会这么实现。然后，我会再和你说说，如果应用中有这种频繁变更并需要统计表行数的需求，业务设计上可以怎么做。 + +# count(*) 的实现方式 + +你首先要明确的是，在不同的 MySQL 引擎中，count(*) 有不同的实现方式。 + +- MyISAM 引擎把一个表的总行数存在了磁盘上，因此执行 count(*) 的时候会直接返回这个数，效率很高； +- 而 InnoDB 引擎就麻烦了，它执行 count(*) 的时候，需要把数据一行一行地从引擎里面读出来，然后累积计数。 + +**这里需要注意的是，我们在这篇文章里讨论的是没有过滤条件的 count(*)，如果加了 where 条件的话，MyISAM 表也是不能返回得这么快的。** + +在前面的文章中，我们一起分析了为什么要使用 InnoDB，因为不论是在事务支持、并发能力还是在数据安全方面，InnoDB 都优于 MyISAM。我猜你的表也一定是用了 InnoDB 引擎。这就是当你的记录数越来越多的时候，计算一个表的总行数会越来越慢的原因。 + +那**为什么 InnoDB 不跟 MyISAM 一样，也把数字存起来呢？** + +这是因为即使是在同一个时刻的多个查询，由于多版本并发控制（MVCC）的原因，InnoDB 表“应该返回多少行”也是不确定的。这里，我用一个算 count(*) 的例子来为你解释一下。 + +假设表 t 中现在有 10000 条记录，我们设计了三个用户并行的会话。 + +- 会话 A 先启动事务并查询一次表的总行数； +- 会话 B 启动事务，插入一行后记录后，查询表的总行数； +- 会话 C 先启动一个单独的语句，插入一行记录后，查询表的总行数。 + +我们假设从上到下是按照时间顺序执行的，同一行语句是在同一时刻执行的。 + +![会话 A、B、C 的执行流程](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/mysql/count-problem.png) + +你会看到，在最后一个时刻，三个会话 A、B、C 会同时查询表 t 的总行数，但拿到的结果却不同。 + +这和 InnoDB 的事务设计有关系，可重复读是它默认的隔离级别，在代码上就是通过多版本并发控制，也就是 MVCC 来实现的。每一行记录都要判断自己是否对这个会话可见，因此对于 count(*) 请求来说，InnoDB 只好把数据一行一行地读出依次判断，可见的行才能够用于计算“基于这个查询”的表的总行数。 + +当然，现在这个看上去笨笨的 MySQL，在执行 count(*) 操作的时候还是做了优化的。 + +你知道的，InnoDB 是索引组织表，主键索引树的叶子节点是数据，而普通索引树的叶子节点是主键值。所以，普通索引树比主键索引树小很多。对于 count(*) 这样的操作，遍历哪个索引树得到的结果逻辑上都是一样的。因此，MySQL 优化器会找到最小的那棵树来遍历。**在保证逻辑正确的前提下，尽量减少扫描的数据量，是数据库系统设计的通用法则之一。** + +如果你用过 show table status 命令的话，就会发现这个命令的输出结果里面也有一个 TABLE_ROWS 用于显示这个表当前有多少行，这个命令执行挺快的，那这个 TABLE_ROWS 能代替 count(*) 吗？ + +实际上，TABLE_ROWS 就是从这个采样估算得来的，因此它也很不准。有多不准呢，官方文档说误差可能达到 40% 到 50%。**所以，show table status 命令显示的行数也不能直接使用。** + +到这里我们小结一下： + +- MyISAM 表虽然 count(*) 很快，但是不支持事务； +- show table status 命令虽然返回很快，但是不准确； +- InnoDB 表直接 count(*) 会遍历全表，虽然结果准确，但会导致性能问题。 + +那么，回到文章开头的问题，如果你现在有一个页面经常要显示交易系统的操作记录总数，到底应该怎么办呢？答案是，我们只能自己计数。 + +接下来，我们讨论一下，看看自己计数有哪些方法，以及每种方法的优缺点有哪些。 + +这里，我先和你说一下这些方法的基本思路：你需要自己找一个地方，把操作记录表的行数存起来。 + +# 用缓存系统保存计数 + +对于更新很频繁的库来说，你可能会第一时间想到，用缓存系统来支持。 + +你可以用一个 Redis 服务来保存这个表的总行数。这个表每被插入一行 Redis 计数就加 1，每被删除一行 Redis 计数就减 1。这种方式下，读和更新操作都很快，但你再想一下这种方式存在什么问题吗？ + +没错，缓存系统可能会丢失更新。 + +Redis 的数据不能永久地留在内存里，所以你会找一个地方把这个值定期地持久化存储起来。但即使这样，仍然可能丢失更新。试想如果刚刚在数据表中插入了一行，Redis 中保存的值也加了 1，然后 Redis 异常重启了，重启后你要从存储 redis 数据的地方把这个值读回来，而刚刚加 1 的这个计数操作却丢失了。 + +当然了，这还是有解的。比如，Redis 异常重启以后，到数据库里面单独执行一次 count(*) 获取真实的行数，再把这个值写回到 Redis 里就可以了。异常重启毕竟不是经常出现的情况，这一次全表扫描的成本，还是可以接受的。 + +但实际上，**将计数保存在缓存系统中的方式，还不只是丢失更新的问题。即使 Redis 正常工作，这个值还是逻辑上不精确的。** + +你可以设想一下有这么一个页面，要显示操作记录的总数，同时还要显示最近操作的 100 条记录。那么，这个页面的逻辑就需要先到 Redis 里面取出计数，再到数据表里面取数据记录。 + +我们是这么定义不精确的： + +1. 一种是，查到的 100 行结果里面有最新插入记录，而 Redis 的计数里还没加 1； +2. 另一种是，查到的 100 行结果里没有最新插入的记录，而 Redis 的计数里已经加了 1。 + +这两种情况，都是逻辑不一致的。 + +我们一起来看看这个时序图。 + +![img](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/mysql/count+1.png) + +上图中，会话 A 是一个插入交易记录的逻辑，往数据表里插入一行 R，然后 Redis 计数加 1；会话 B 就是查询页面显示时需要的数据。 + +在上图 的这个时序里，在 T3 时刻会话 B 来查询的时候，会显示出新插入的 R 这个记录，但是 Redis 的计数还没加 1。这时候，就会出现我们说的数据不一致。 + +你一定会说，这是因为我们执行新增记录逻辑时候，是先写数据表，再改 Redis 计数。而读的时候是先读 Redis，再读数据表，这个顺序是相反的。那么，如果保持顺序一样的话，是不是就没问题了？我们现在把会话 A 的更新顺序换一下，再看看执行结果。 + +![img](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/mysql/count+1_1.png) + + + +你会发现，这时候反过来了，会话 B 在 T3 时刻查询的时候，Redis 计数加了 1 了，但还查不到新插入的 R 这一行，也是数据不一致的情况。 + +在并发系统里面，我们是无法精确控制不同线程的执行时刻的，因为存在图中的这种操作序列，所以，我们说即使 Redis 正常工作，这个计数值还是逻辑上不精确的。 + +# 在数据库保存计数 + +根据上面的分析，用缓存系统保存计数有丢失数据和计数不精确的问题。那么，**如果我们把这个计数直接放到数据库里单独的一张计数表 C 中，又会怎么样呢？** + +首先，这解决了崩溃丢失的问题，InnoDB 是支持崩溃恢复不丢数据的。 + +> 备注：关于 InnoDB 的崩溃恢复，你可以再回顾一下第 2 篇文章[《日志系统：一条 SQL 更新语句是如何执行的？》](https://time.geekbang.org/column/article/68633)中的相关内容。 + +然后，我们再看看能不能解决计数不精确的问题。 + +你会说，这不一样吗？无非就是把图 3 中对 Redis 的操作，改成了对计数表 C 的操作。只要出现图 3 的这种执行序列，这个问题还是无解的吧？ + +这个问题还真不是无解的。 + +我们这篇文章要解决的问题，都是由于 InnoDB 要支持事务，从而导致 InnoDB 表不能把 count(*) 直接存起来，然后查询的时候直接返回形成的。 + +所谓以子之矛攻子之盾，现在我们就利用“事务”这个特性，把问题解决掉。 + +![图 4 会话 A、B 的执行时序图](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/mysql/count+1-solve.png) + +我们来看下现在的执行结果。虽然会话 B 的读操作仍然是在 T3 执行的，但是因为这时候更新事务还没有提交，所以计数值加 1 这个操作对会话 B 还不可见。 + +因此，会话 B 看到的结果里， 查计数值和“最近 100 条记录”看到的结果，逻辑上就是一致的。 + +# 不同的 count 用法 + +在 select count(?) from t 这样的查询语句里面，count(*)、count(主键 id)、count(字段) 和 count(1) 等不同用法的性能，有哪些差别。 + +需要注意的是，下面的讨论还是基于 InnoDB 引擎的。 + +这里，首先你要弄清楚 count() 的语义。count() 是一个聚合函数，对于返回的结果集，一行行地判断，如果 count 函数的参数不是 NULL，累计值就加 1，否则不加。最后返回累计值。 + +所以，count(*)、count(主键 id) 和 count(1) 都表示返回满足条件的结果集的总行数；而 count(字段），则表示返回满足条件的数据行里面，参数“字段”不为 NULL 的总个数。 + +至于分析性能差别的时候，你可以记住这么几个原则： + +1. server 层要什么就给什么； +2. InnoDB 只给必要的值； +3. 现在的优化器只优化了 count(*) 的语义为“取行数”，其他“显而易见”的优化并没有做。 + +这是什么意思呢？接下来，我们就一个个地来看看。 + +**对于 count(主键 id) 来说**，InnoDB 引擎会遍历整张表，把每一行的 id 值都取出来，返回给 server 层。server 层拿到 id 后，判断是不可能为空的，就按行累加。 + +**对于 count(1) 来说**，InnoDB 引擎遍历整张表，但不取值。server 层对于返回的每一行，放一个数字“1”进去，判断是不可能为空的，按行累加。 + +单看这两个用法的差别的话，你能对比出来，count(1) 执行得要比 count(主键 id) 快。因为从引擎返回 id 会涉及到解析数据行，以及拷贝字段值的操作。 + +**对于 count(字段) 来说**： + +1. 如果这个“字段”是定义为 not null 的话，一行行地从记录里面读出这个字段，判断不能为 null，按行累加； +2. 如果这个“字段”定义允许为 null，那么执行的时候，判断到有可能是 null，还要把值取出来再判断一下，不是 null 才累加。 + +也就是前面的第一条原则，server 层要什么字段，InnoDB 就返回什么字段。 + +**但是 count(\*) 是例外**，并不会把全部字段取出来，而是专门做了优化，不取值。count(*) 肯定不是 null，按行累加。 + +看到这里，你一定会说，优化器就不能自己判断一下吗，主键 id 肯定非空啊，为什么不能按照 count(*) 来处理，多么简单的优化啊。 + +当然，MySQL 专门针对这个语句进行优化，也不是不可以。但是这种需要专门优化的情况太多了，而且 MySQL 已经优化过 count(*) 了，你直接使用这种用法就可以了。 + +所以结论是：按照效率排序的话，count(字段) 知识这个东西，看来真的要温故而知新，一直不用，都要忘记了。 +> +> 业务很简单：需要批量插入一些数据，数据来源可能是其他数据库的表，也可能是一个外部excel的导入。 +> +> 那么问题来了，是不是每次插入之前都要查一遍，看看重不重复，在代码里筛选一下数据，重复的就过滤掉呢？ +> +> 向大数据数据库中插入值时，还要判断插入是否重复，然后插入。如何提高效率？ + +解决的办法有很多种，不同的场景解决方案也不一样，数据量很小的情况下，怎么搞都行，但是数据量很大的时候，这就不是一个简单的问题了。 + +几百万的数据，不可能查出来去重处理！ + +看一下常用到的解决方案。 + +### 1、insert ignore into + +> 当插入数据时，如出现错误时，如重复数据，将不返回错误，只以警告形式返回。所以使用ignore请确保语句本身没有问题，否则也会被忽略掉。例如： + +```mysql +INSERT IGNORE INTO user (name) VALUES ('telami') +``` + +> 这种方法很简便，但是有一种可能，就是插入不是因为重复数据报错，而是因为其他原因报错的，也同样被忽略了～ + +### 2、on duplicate key update + +当primary或者unique重复时，则执行update语句，如update后为无用语句，如id=id，则同1功能相同，但错误不会被忽略掉。 + +在公众号后端架构师后台回复“架构整洁”，获取一份惊喜礼包。 + +例如，为了实现name重复的数据插入不报错，可使用一下语句： + +```mysql +INSERT INTO user (name) VALUES ('telami') ON duplicate KEY UPDATE id = id +``` + +这种方法有个前提条件，就是，需要插入的约束，需要是主键或者唯一约束（在你的业务中那个要作为唯一的判断就将那个字段设置为唯一约束也就是unique key）。 + +### 3、insert … select … where not exist + +根据select的条件判断是否插入，可以不光通过primary 和unique来判断，也可通过其它条件。例如： + +```mysql +INSERT INTO user (name) SELECT 'telami' FROM dual WHERE NOT EXISTS (SELECT id FROM user WHERE id = 1) +``` + +这种方法其实就是使用了mysql的一个临时表的方式，但是里面使用到了子查询，效率也会有一点点影响，如果能使用上面的就不使用这个。 + +### 4、replace into + +如果存在primary or unique相同的记录，则先删除掉。再插入新记录。 + +```mysql +REPLACE INTO user SELECT 1, 'telami' FROM books +``` + +这种方法就是不管原来有没有相同的记录，都会先删除掉然后再插入。 + +### 实践 + +选择的是第二种方式 + +```xml + + insert into user (id,username,mobile_number) + values + + ( + #{item.id}, + #{item.username}, + #{item.mobileNumber} + ) + + ON duplicate KEY UPDATE id = id + +``` + +这里用的是Mybatis，批量插入的一个操作，**mobile_number**已经加了唯一约束。这样在批量插入时，如果存在手机号相同的话，是不会再插入了的。 \ No newline at end of file diff --git "a/docs/data-store/MySQL/\344\272\222\350\201\224\347\275\221\345\270\270\347\224\250\345\210\206\345\272\223\345\210\206\350\241\250\346\226\271\346\241\210.md" "b/docs/data-management/MySQL/reproduce/\344\272\222\350\201\224\347\275\221\345\270\270\347\224\250\345\210\206\345\272\223\345\210\206\350\241\250\346\226\271\346\241\210.md" similarity index 100% rename from "docs/data-store/MySQL/\344\272\222\350\201\224\347\275\221\345\270\270\347\224\250\345\210\206\345\272\223\345\210\206\350\241\250\346\226\271\346\241\210.md" rename to "docs/data-management/MySQL/reproduce/\344\272\222\350\201\224\347\275\221\345\270\270\347\224\250\345\210\206\345\272\223\345\210\206\350\241\250\346\226\271\346\241\210.md" diff --git "a/docs/data-store/MySQL/\346\200\247\350\203\275\344\274\230\345\214\226\344\271\213\345\210\206\351\241\265\346\237\245\350\257\242.md" "b/docs/data-management/MySQL/reproduce/\346\200\247\350\203\275\344\274\230\345\214\226\344\271\213\345\210\206\351\241\265\346\237\245\350\257\242.md" similarity index 100% rename from "docs/data-store/MySQL/\346\200\247\350\203\275\344\274\230\345\214\226\344\271\213\345\210\206\351\241\265\346\237\245\350\257\242.md" rename to "docs/data-management/MySQL/reproduce/\346\200\247\350\203\275\344\274\230\345\214\226\344\271\213\345\210\206\351\241\265\346\237\245\350\257\242.md" diff --git a/docs/data-management/README.md b/docs/data-management/README.md new file mode 100644 index 0000000000..d5d3ae0c88 --- /dev/null +++ b/docs/data-management/README.md @@ -0,0 +1 @@ +![](https://tva1.sinaimg.cn/large/008eGmZEly1gmibi0ficej318y0u07wi.jpg) \ No newline at end of file diff --git a/docs/data-management/Redis/.DS_Store b/docs/data-management/Redis/.DS_Store new file mode 100644 index 0000000000..45706ae4e1 Binary files /dev/null and b/docs/data-management/Redis/.DS_Store differ diff --git a/docs/data-store/Redis/1.Nosql-Overview.md b/docs/data-management/Redis/Nosql-Overview.md similarity index 72% rename from docs/data-store/Redis/1.Nosql-Overview.md rename to docs/data-management/Redis/Nosql-Overview.md index d432e219c0..4e5c48ff95 100644 --- a/docs/data-store/Redis/1.Nosql-Overview.md +++ b/docs/data-management/Redis/Nosql-Overview.md @@ -1,9 +1,5 @@ -![nosql-index.jpg](http://ww1.sinaimg.cn/large/9b9f09a9ly1g9yp975bslj211c0fq75c.jpg) - # NoSQL的前世今生 -> Java大猿帅成长手册，**GitHub** [JavaEgg](https://github.com/Jstarfish/JavaEgg) ，N线互联网开发必备技能兵器谱 - ### 啥玩意： NoSQL(NoSQL = Not Only SQL )，“不仅仅是SQL”，泛指**非关系型的数据库**。随着互联网web2.0网站的兴起，传统的关系数据库在处理web2.0网站，特别是超大规模和高并发的SNS类型的web2.0纯动态网站已经显得力不从心，暴露了很多难以克服的问题，而非关系型的数据库则由于其本身的特点得到了非常迅速的发展。NoSQL数据库的产生就是为了解决大规模数据集合多重数据种类带来的挑战，尤其是大数据应用难题，包括超大规模数据的存储。（例如谷歌或Facebook每天为他们的用户收集万亿比特的数据）。这些类型的数据存储不需要固定的模式，无需多余操作就可以横向扩展。 @@ -14,7 +10,7 @@ NoSQL(NoSQL = Not Only SQL )，“不仅仅是SQL”，泛指**非关系型的 #### 1. 单机MySQL的美好年代 -​ 在以前，一个网站的访问量一般都不大，用单个数据库完全可以轻松应付。在那个时候，更多的都是静态网页，动态交互类型的网站不多。上述架构下，我们来看看数据存储的瓶颈是什么？ +在以前，一个网站的访问量一般都不大，用单个数据库完全可以轻松应付。在那个时候，更多的都是静态网页，动态交互类型的网站不多。上述架构下，我们来看看数据存储的瓶颈是什么？ - 数据量的总大小 一个机器放不下时 - 数据的索引（B+ Tree）一个机器的内存放不下时 @@ -22,29 +18,29 @@ NoSQL(NoSQL = Not Only SQL )，“不仅仅是SQL”，泛指**非关系型的 #### 2. Memcached(缓存)+MySQL+垂直拆分 -​ 后来，随着访问量的上升，几乎大部分使用MySQL架构的网站在数据库上都开始出现了性能问题，web程序不再仅仅专注在功能上，同时也在追求性能。程序员们开始大量的使用**缓存技术**来缓解数据库的压力，优化数据库的结构和索引。开始比较流行的是通过**文件缓存**来缓解数据库压力，但是当访问量继续增大的时候，多台web机器通过文件缓存不能共享，大量的小文件缓存也带了了比较高的IO压力。在这个时候，Memcached就自然的成为一个非常时尚的技术产品。 +后来，随着访问量的上升，几乎大部分使用MySQL架构的网站在数据库上都开始出现了性能问题，web程序不再仅仅专注在功能上，同时也在追求性能。程序员们开始大量的使用**缓存技术**来缓解数据库的压力，优化数据库的结构和索引。开始比较流行的是通过**文件缓存**来缓解数据库压力，但是当访问量继续增大的时候，多台web机器通过文件缓存不能共享，大量的小文件缓存也带了了比较高的IO压力。在这个时候，Memcached就自然的成为一个非常时尚的技术产品。 -​ Memcached作为一个**独立的分布式的缓存服务器**，为多个web服务器提供了一个共享的高性能缓存服务，在Memcached服务器上，又发展了根据hash算法来进行多台Memcached缓存服务的扩展，然后又出现了一致性hash来解决增加或减少缓存服务器导致重新hash带来的大量缓存失效的弊端 +Memcached作为一个**独立的分布式的缓存服务器**，为多个web服务器提供了一个共享的高性能缓存服务，在Memcached服务器上，又发展了根据hash算法来进行多台Memcached缓存服务的扩展，然后又出现了一致性hash来解决增加或减少缓存服务器导致重新hash带来的大量缓存失效的弊端 #### 3. Mysql主从读写分离 -​ 由于数据库的写入压力增加，Memcached只能缓解数据库的读取压力。读写集中在一个数据库上让数据库不堪重负，大部分网站开始**使用主从复制技术来达到读写分离，以提高读写性能和读库的可扩展性**。**Mysql的master-slave模式**成为这个时候的网站标配了。 +由于数据库的写入压力增加，Memcached只能缓解数据库的读取压力。读写集中在一个数据库上让数据库不堪重负，大部分网站开始**使用主从复制技术来达到读写分离，以提高读写性能和读库的可扩展性**。**Mysql的master-slave模式**成为这个时候的网站标配了。 #### 4. 分表分库+水平拆分+mysql集群 -​ 在Memcached的高速缓存，MySQL的主从复制，读写分离的基础之上，这时MySQL主库的写压力开始出现瓶颈，而数据量的持续猛增，由于**MyISAM**使用**表锁**，在高并发下会出现严重的锁问题，大量的高并发MySQL应用开始使用**InnoDB**引擎代替MyISAM。 +在Memcached的高速缓存，MySQL的主从复制，读写分离的基础之上，这时MySQL主库的写压力开始出现瓶颈，而数据量的持续猛增，由于**MyISAM**使用**表锁**，在高并发下会出现严重的锁问题，大量的高并发MySQL应用开始使用**InnoDB**引擎代替MyISAM。 -​ 同时，开始流行**使用分表分库来缓解写压力和数据增长的扩展问题**。这个时候，分表分库成了一个热门技术，是面试的热门问题也是业界讨论的热门技术问题。也就在这个时候，MySQL推出了还不太稳定的表分区，这也给技术实力一般的公司带来了希望。虽然MySQL推出了MySQL Cluster集群，但性能也不能很好满足互联网的要求，只是在高可靠性上提供了非常大的保证。 +同时，开始流行**使用分表分库来缓解写压力和数据增长的扩展问题**。这个时候，分表分库成了一个热门技术，是面试的热门问题也是业界讨论的热门技术问题。也就在这个时候，MySQL推出了还不太稳定的表分区，这也给技术实力一般的公司带来了希望。虽然MySQL推出了MySQL Cluster集群，但性能也不能很好满足互联网的要求，只是在高可靠性上提供了非常大的保证。 #### 5. MySQL的扩展性瓶颈 -​ MySQL数据库也经常存储一些大文本字段，导致数据库表非常的大，在做数据库恢复的时候就导致非常的慢，不容易快速恢复数据库。比如1000万4KB大小的文本就接近40GB的大小，如果能把这些数据从MySQL省去，MySQL将变得非常的小。关系数据库很强大，但是它并不能很好的应付所有的应用场景。MySQL的扩展性差（需要复杂的技术来实现），大数据下IO压力大，表结构更改困难，正是当前使用MySQL的开发人员面临的问题。 +MySQL数据库也经常存储一些大文本字段，导致数据库表非常的大，在做数据库恢复的时候就导致非常的慢，不容易快速恢复数据库。比如1000万4KB大小的文本就接近40GB的大小，如果能把这些数据从MySQL省去，MySQL将变得非常的小。关系数据库很强大，但是它并不能很好的应付所有的应用场景。MySQL的扩展性差（需要复杂的技术来实现），大数据下IO压力大，表结构更改困难，正是当前使用MySQL的开发人员面临的问题。 #### 6. 为什么用NoSQL -​ 今天我们可以通过第三方平台（如：Google,Facebook等）可以很容易的**访问和抓取数据**（爬虫私密信息有风险哈）。用户的个人信息，社交网络，地理位置，用户生成的数据和用户操作日志已经成倍的增加。我们如果要对这些用户数据进行挖掘，那SQL数据库已经不适合这些应用了, NoSQL数据库的发展也不能很好的处理这些大的数据。 +今天我们可以通过第三方平台（如：Google,Facebook等）可以很容易的**访问和抓取数据**（爬虫私密信息有风险哈）。用户的个人信息，社交网络，地理位置，用户生成的数据和用户操作日志已经成倍的增加。我们如果要对这些用户数据进行挖掘，那SQL数据库已经不适合这些应用了, NoSQL数据库的发展也不能很好的处理这些大的数据。 -![1.png](https://i.loli.net/2019/11/17/WvJC95bHnUjGuF2.png) +![](https://i.loli.net/2019/11/17/WvJC95bHnUjGuF2.png) diff --git a/docs/data-management/Redis/ReadRedis.md b/docs/data-management/Redis/ReadRedis.md new file mode 100644 index 0000000000..2ecbb031b8 --- /dev/null +++ b/docs/data-management/Redis/ReadRedis.md @@ -0,0 +1,110 @@ +![img](https://redis.io/wp-content/uploads/2014/05/redis_289_art.png) + + + +> 作为一名后端开发工程师，工作中肯定会用到 Redis，面试中八成也会被问到 Redis 相关的问题。所以系统学习，深入学习还是很有必要的。 +> +> 建立属于自己的完整的 Reids 知识框架。 +> +> 带着问题去系统学习，有一个自己的问题画像，最后梳理成自己的“武功秘籍” +> +> 看下极客时间中的一个 Redis 问题画像图： +> +> ![img](https://static001.geekbang.org/resource/image/70/b4/70a5bc1ddc9e3579a2fcb8a5d44118b4.jpeg) + + + +## Redis 简介 + +Redis: **REmote DIctionary Server**(远程字典服务器)。 + +Redis 是一个全开源免费（BSD许可）的，使用 C 语言编写，内存中的数据结构存储系统，它可以用作**数据库、缓存和消息中间件**。一般作为一个高性能的(key/value)分布式内存数据库，基于**内存**运行并支持持久化的 NoSQL 数据库，是当前最热门的 NoSql 数据库之一，也被人们称为**数据结构服务器** + +它支持多种数据结构，如字符串（Strings）、哈希（Hashes）、列表（Lists）、集合（Sets）、有序集合（Sorted Sets）、位图（bitmaps）、HyperLogLogs 和地理空间索引（geospatial indexes），并带有半持久化存储的选项。 + +### 主要特点 + +1. **高性能**：Redis 的读写速度非常快，支持每秒百万级的请求处理能力。其所有数据都存储在内存中，保证了极高的读写性能。 +2. **多种数据结构**：Redis 不仅支持简单的键值对，还支持多种高级数据结构，如列表、集合、有序集合、哈希等，可以满足复杂的数据存储需求。 +3. **持久化**：Redis 支持多种持久化机制，如 RDB（快照）和 AOF（追加文件），可以将内存中的数据持久化到磁盘，保证数据的持久性。 +4. **高可用性**：通过 Redis 的复制（Replication）、Sentinel 和 Cluster 特性，可以实现高可用性和自动故障转移。复制机制允许数据从主节点复制到多个从节点，从而提高数据的冗余性和读取的可扩展性。 +5. **Lua脚本**：Redis 内置 Lua 脚本引擎，支持在服务器端运行复杂的脚本，减少了网络往返次数，提高了操作的原子性。 +6. **事务支持**：Redis 支持事务，通过 MULTI、EXEC、DISCARD 和 WATCH 等命令实现事务操作。 +7. **发布/订阅**：Redis 提供了发布/订阅功能，可以实现消息通知和实时消息传递。 +8. **丰富的生态系统**：Redis 拥有丰富的客户端库，支持多种编程语言，包括 C、C++、Java、Python、Go、Node.js等。 + +### 应用场景 + +1. **缓存**：Redis 作为缓存系统，可以极大地提高数据读取速度，减轻数据库的压力。 +2. **会话存储**：利用 Redis 的高性能和持久化特性，可以用于存储用户会话信息。 +3. **实时分析**：利用 Redis 的集合和有序集合，可以进行实时数据分析和排名。 +4. **消息队列**：利用 Redis 的列表和发布/订阅特性，可以实现简单的消息队列系统。 +5. **计数器和限流**：利用 Redis 的原子递增操作，可以实现高效的计数器和限流机制。 + +> **具体以某一论坛为例：** +> +> - 记录帖子的点赞数、评论数和点击数 (hash)。 +> - 记录用户的帖子 ID 列表 (排序)，便于快速显示用户的帖子列表 (zset)。 +> - 记录帖子的标题、摘要、作者和封面信息，用于列表页展示 (hash)。 +> - 记录帖子的点赞用户 ID 列表，评论 ID 列表，用于显示和去重计数 (zset)。 +> - 缓存近期热帖内容 (帖子内容空间占用比较大)，减少数据库压力 (hash)。 +> - 记录帖子的相关文章 ID，根据内容推荐相关帖子 (list)。 +> - 如果帖子 ID 是整数自增的，可以使用 Redis 来分配帖子 ID(计数器)。 +> - 收藏集和帖子之间的关系 (zset)。 +> - 记录热榜帖子 ID 列表，总热榜和分类热榜 (zset)。 +> - 缓存用户行为历史，进行恶意行为过滤 (zset,hash)。 + +Redis 的官网地址，非常好记，是 redis.io。（域名后缀io属于国家域名，是 british Indian Ocean territory，即英属印度洋领地）目前，Vmware 在资助着 Redis 项目的开发和维护。 + + + +**安装** + +```shell +$ wget http://download.redis.io/releases/redis-5.0.6.tar.gz +$ tar xzf redis-5.0.6.tar.gz +$ cd redis-5.0.6 +$ make +``` + +新版本的编译文件在 src 中（之前在bin目录），启动 server + +```sh +$ src/redis-server +``` + +启动客户端 + +```shell +$ src/redis-cli +redis> set foo bar +OK +redis> get foo +"bar" +``` + + + +## Redis 知识全景 + +![](/Users/starfish/Downloads/79da7093ed998a99d9abe91e610b74e7.jpg) + +“两大维度”就是指系统维度和应用维度，“三大主线”也就是指高性能、高可靠和高可扩展（可以简称为“三高”）。 + +Redis 作为庞大的键值数据库，可以说遍地都是知识，一抓一大把，我们怎么能快速地知道该学哪些呢？别急，接下来就要看“三大主线”的魔力了。 + +别看技术点是零碎的，其实你完全可以按照这三大主线，给它们分下类，就像图片中展示的那样，具体如下： + +**高性能主线**，包括线程模型、数据结构、持久化、网络框架； + +**高可靠主线**，包括主从复制、哨兵机制； + +**高可扩展主线**，包括数据分片、负载均衡。 + + + + + +## 推荐阅读 + +[《我是如何学习Redis的？高效学习Redis的路径和方法分享》](http://kaito-kidd.com/2020/09/09/how-i-learned-redis/) \ No newline at end of file diff --git "a/docs/data-management/Redis/Redis \346\234\211\347\224\250\346\214\207\344\273\244.md" "b/docs/data-management/Redis/Redis \346\234\211\347\224\250\346\214\207\344\273\244.md" new file mode 100644 index 0000000000..8a096f1a0e --- /dev/null +++ "b/docs/data-management/Redis/Redis \346\234\211\347\224\250\346\214\207\344\273\244.md" @@ -0,0 +1,146 @@ +## Info 指令 + +在使用 Redis 时，时常会遇到很多问题需要诊断，在诊断之前需要了解 Redis 的运行状态，通过强大的 Info 指令，你可以清晰地知道 Redis 内部一系列运行参数。 + +Info 指令显示的信息非常繁多，分为 9 大块，每个块都有非常多的参数，这 9 个块分别是: + +1. Server 服务器运行的环境参数 +2. Clients 客户端相关信息 +3. Memory 服务器运行内存统计数据 +4. Persistence 持久化信息 +5. Stats 通用统计数据 +6. Replication 主从复制相关信息 +7. CPU CPU 使用情况 +8. Cluster 集群信息 +9. KeySpace 键值对统计数量信息 + +Info 可以一次性获取所有的信息，也可以按块取信息。 + +``` +# 获取所有信息 +> info +# 获取内存相关信息 +> info memory +# 获取复制相关信息 +> info replication + +#Redis 连接了多少客户端？ +> info clients + +#Redis 内存占用多大 ? +> redis-cli info memory | grep used | grep human +used_memory_human:827.46K # 内存分配器 (jemalloc) 从操作系统分配的内存总量 +used_memory_rss_human:3.61M # 操作系统看到的内存占用 ,top 命令看到的内存 +used_memory_peak_human:829.41K # Redis 内存消耗的峰值 +used_memory_lua_human:37.00K # lua 脚本引擎占用的内存大小 + +#复制积压缓冲区多大？ +> redis-cli info replication |grep backlog +repl_backlog_active:0 +repl_backlog_size:1048576 # 这个就是积压缓冲区大小 +repl_backlog_first_byte_offset:0 +repl_backlog_histlen:0 +``` + +考虑到参数非常繁多，一一说明工作量巨大，下面我只挑一些关键性的、非常实用和最常用的参数进行详细讲解。如果读者想要了解所有的参数细节，请参考阅读 [Redis 官网文档](https://link.juejin.cn/?target=https%3A%2F%2Fredis.io%2Fcommands%2Finfo)。 + + + + + +## Redis 每秒执行多少次指令？ + + + +![img](https://p1-jj.byteimg.com/tos-cn-i-t2oaga2asx/gold-user-assets/2018/7/16/164a14ce6633c24a~tplv-t2oaga2asx-watermark.awebp) + + + +这个信息在 Stats 块里，可以通过 `info stats` 看到。 + +``` +# ops_per_sec: operations per second，也就是每秒操作数 +> redis-cli info stats |grep ops +instantaneous_ops_per_sec:789 +``` + + + +![img](https://p1-jj.byteimg.com/tos-cn-i-t2oaga2asx/gold-user-assets/2018/7/13/1649181bd00bed33~tplv-t2oaga2asx-watermark.awebp) + +以上，表示 ops 是 789，也就是所有客户端每秒会发送 789 条指令到服务器执行。极限情况下，Redis 可以每秒执行 10w 次指令，CPU 几乎完全榨干。如果 qps 过高，可以考虑通过 `monitor` 指令快速观察一下究竟是哪些 key 访问比较频繁，从而在相应的业务上进行优化，以减少 IO 次数。`monitor` 指令会瞬间吐出来巨量的指令文本，所以一般在执行 `monitor` 后立即 `ctrl+c`中断输出。 + + + +``` +> redis-cli monitor +``` + +## Redis 连接了多少客户端？ + +这个信息在 Clients 块里，可以通过 `info clients` 看到。 + +``` +> redis-cli info clients +# Clients +connected_clients:124 # 这个就是正在连接的客户端数量 +client_longest_output_list:0 +client_biggest_input_buf:0 +blocked_clients:0 +``` + +这个信息也是比较有用的，通过观察这个数量可以确定是否存在意料之外的连接。如果发现这个数量不对劲，接着就可以使用`client list`指令列出所有的客户端链接地址来确定源头。 + +关于客户端的数量还有个重要的参数需要观察，那就是`rejected_connections`，它表示因为超出最大连接数限制而被拒绝的客户端连接次数，如果这个数字很大，意味着服务器的最大连接数设置的过低需要调整 `maxclients` 参数。 + +``` +> redis-cli info stats |grep reject +rejected_connections:0 +``` + +## Redis 内存占用多大 ? + + + +![img](https://p1-jj.byteimg.com/tos-cn-i-t2oaga2asx/gold-user-assets/2018/7/16/164a14efc8e3e44b~tplv-t2oaga2asx-watermark.awebp) + + + +这个信息在 Memory 块里，可以通过 `info memory` 看到。 + +``` +> redis-cli info memory | grep used | grep human +used_memory_human:827.46K # 内存分配器 (jemalloc) 从操作系统分配的内存总量 +used_memory_rss_human:3.61M # 操作系统看到的内存占用 ,top 命令看到的内存 +used_memory_peak_human:829.41K # Redis 内存消耗的峰值 +used_memory_lua_human:37.00K # lua 脚本引擎占用的内存大小 +``` + +如果单个 Redis 内存占用过大，并且在业务上没有太多压缩的空间的话，可以考虑集群化了。 + +## 复制积压缓冲区多大？ + +这个信息在 Replication 块里，可以通过 `info replication` 看到。 + +``` +> redis-cli info replication |grep backlog +repl_backlog_active:0 +repl_backlog_size:1048576 # 这个就是积压缓冲区大小 +repl_backlog_first_byte_offset:0 +repl_backlog_histlen:0 +``` + +复制积压缓冲区大小非常重要，它严重影响到主从复制的效率。当从库因为网络原因临时断开了主库的复制，然后网络恢复了，又重新连上的时候，这段断开的时间内发生在 master 上的修改操作指令都会放在积压缓冲区中，这样从库可以通过积压缓冲区恢复中断的主从同步过程。 + +积压缓冲区是环形的，后来的指令会覆盖掉前面的内容。如果从库断开的时间过长，或者缓冲区的大小设置的太小，都会导致从库无法快速恢复中断的主从同步过程，因为中间的修改指令被覆盖掉了。这时候从库就会进行全量同步模式，非常耗费 CPU 和网络资源。 + +如果有多个从库复制，积压缓冲区是共享的，它不会因为从库过多而线性增长。如果实例的修改指令请求很频繁，那就把积压缓冲区调大一些，几十个 M 大小差不多了，如果很闲，那就设置为几个 M。 + +``` +> redis-cli info stats | grep sync +sync_full:0 +sync_partial_ok:0 +sync_partial_err:0 # 半同步失败次数 +``` + +通过查看`sync_partial_err`变量的次数来决定是否需要扩大积压缓冲区，它表示主从半同步复制失败的次数。 \ No newline at end of file diff --git a/docs/data-management/Redis/Redis-Cache-Model.md b/docs/data-management/Redis/Redis-Cache-Model.md new file mode 100644 index 0000000000..9cb60433a4 --- /dev/null +++ b/docs/data-management/Redis/Redis-Cache-Model.md @@ -0,0 +1,191 @@ +### **什么是缓存模式？** + +缓存模式是指在系统中如何设计和实现缓存机制，用以快速存取数据并减轻后端数据存储的负载。Redis 提供了灵活多样的缓存策略，常见模式包括： + +1. **直写模式（Write Through）** +2. **回写模式（Write Back）** +3. **旁路缓存模式（Cache Aside）** +4. **只缓存模式（Read Through）** + +通过合理选择和优化这些模式，可以满足不同业务场景对性能、数据一致性和可用性的需求。 + +------ + +### **Redis 缓存模式详解** + +#### **1. Cache Aside（旁路缓存）** + +Cache Aside 模式又称为 **Lazy Loading**，是使用最广泛的缓存模式之一，通常由业务代码显式管理缓存和数据库，核心思想是： + +- 数据从数据库加载到缓存中，缓存作为数据库的一个“旁路”。 +- 应用程序负责读取缓存，缓存未命中时再从数据库读取并更新缓存。 + +**读请求**： + +- 先从缓存中读取数据； +- 如果缓存中没有数据（缓存未命中），从数据库中获取数据，将数据写入缓存，返回给客户端。 + +**写请求**： + +- 数据写入数据库后，将缓存中的数据清除或更新。 + +**适用场景：** + +- 数据更新较少但读取频率较高的场景，例如商品详情、热搜榜单。 +- 对数据一致性要求不严格的系统。 + +**优缺点：** + +- **优点**：简单易用，缓存与数据库解耦。 +- **缺点**：缓存预热需要时间，容易出现缓存击穿问题。 + + + +#### 2. Write Through（直写模式） + +在直写模式中，应用程序的所有写操作都会同时更新缓存和数据库： + +- 数据写入数据库的同时同步写入缓存。 + +**工作流程：** + +1. 应用将数据同时写入 Redis 和数据库。 +2. 读取时直接从 Redis 获取数据。 + +**适用场景：** + +- 需要缓存和数据库一致性非常高的场景，例如账户余额、订单状态等敏感数据。 + +**优缺点：** + +- **优点**：一致性高，数据实时同步。 +- **缺点**：写入速度较慢，因为每次写操作都需要更新两处存储。 + + + +#### 3. Write Back（回写模式） + +在回写模式下，数据首先写入缓存，之后异步写入数据库： + +- 写入数据库的操作由后台线程或任务队列完成。 + +**工作流程：** + +1. 数据先写入 Redis。 +2. Redis 异步将数据批量写入数据库。 + +**适用场景：** + +- 写频率高、读频率较低且对一致性要求不严格的场景，例如日志系统。 + +**优缺点：** + +- **优点**：写入性能高，因为写数据库是异步的。 +- **缺点**：可能导致数据丢失，如果缓存写入后还未同步到数据库时发生故障。 + + + +#### 4. Read Through（只缓存模式） + +在这种模式中，所有的读写操作都必须通过缓存完成： + +- 缓存未命中时，应用从数据库中加载数据并自动更新缓存。 + +**工作流程：** + +1. 应用读取 Redis，如果未命中，自动从数据库加载并更新。 +2. 写入时同步更新缓存和数据库。 + +**适用场景：** + +- 读多写少且对实时性要求较高的场景。 + +**优缺点：** + +- **优点**：应用层逻辑简单。 +- **缺点**：依赖于缓存层，缓存崩溃可能导致大量数据库请求。 + + + +#### 5、Refresh Ahead（提前刷新缓存） + +Refresh Ahead 模式通过提前异步加载数据，防止缓存失效时查询数据库的性能抖动。 + +- 基于预设的过期时间，在缓存即将失效前，后台异步加载数据并更新缓存。 + + + +#### 6、Singleflight 模式 + +Singleflight 是一种**抑制重复请求**的模式，用于解决缓存未命中时的高并发问题。 + +- 当多个请求同时查询相同的缓存未命中数据时，只有一个请求会执行数据库查询，其余请求等待结果返回。 + +**核心思路** + +- 当多个并发请求同时访问**同一个缓存键**，导致缓存未命中时，**Singleflight** 机制保证只有**一个请求**去执行实际的数据库查询或计算操作，其余请求会等待第一个请求的结果完成后直接返回相同的结果。 +- 这样可以有效避免重复查询或计算，减少对数据库、API 等资源的压力。 + +**流程** + +1. 多个请求到来时，检测是否已有请求正在执行。 + - 如果没有请求在执行，则当前请求负责查询数据。 + - 如果已有请求在执行，其他请求进入等待队列。 +2. **第一个请求执行数据库查询或计算操作，并保存结果。** +3. **等待中的请求获取到第一个请求的结果，直接返回相同数据。** + + + +### Redis 缓存模式的优化策略 + +#### **1. 缓存雪崩** + +缓存雪崩指缓存集中失效时，大量请求直接击穿数据库，导致数据库压力激增甚至崩溃。 + +**解决方案：** + +- **设置随机过期时间**：避免大量缓存同时失效。 +- **多级缓存**：在 Redis 之上增加本地缓存（如 Guava）。 +- **请求限流**：通过限流机制控制瞬时流量。 + +#### **2. 缓存击穿** + +缓存击穿指某个热点数据缓存失效后，短时间内大量请求直接打到数据库。 + +**解决方案：** + +- **热点数据预加载**：提前将热点数据缓存。 +- **加互斥锁**：在缓存未命中时，通过锁机制防止数据库过载。 + +#### 3. 缓存穿透 + +缓存穿透指用户请求的数据既不在缓存中，也不存在于数据库中，导致所有请求打到数据库。 + +**解决方案：** + +- **布隆过滤器**：拦截无效请求，减少数据库查询。 +- **缓存空结果**：将不存在的数据写入缓存，避免重复查询。 + + + +### Redis 缓存模式的应用场景 + +#### **1. 电商秒杀** + +在高并发的秒杀场景中，Redis 通常用于缓存商品库存数据。典型模式为： + +- 使用 `Cache Aside` 模式缓存库存数据，避免频繁访问数据库。 +- 配合分布式锁，防止超卖问题。 + +#### **2. 社交网络** + +在社交平台上，Redis 用于存储用户会话、好友关系等数据： + +- 使用 `Write Through` 模式保证数据一致性。 +- 通过 Redis 的 Set 结构实现快速去重和交集运算。 + +#### **3. 实时排行榜** + +Redis 的 Sorted Set 结构非常适合实现排行榜功能： + +- 使用 `Cache Aside` 模式，定期将缓存中的排行榜数据同步到数据库。 \ No newline at end of file diff --git a/docs/data-management/Redis/Redis-Cluster.md b/docs/data-management/Redis/Redis-Cluster.md new file mode 100644 index 0000000000..856af90b08 --- /dev/null +++ b/docs/data-management/Redis/Redis-Cluster.md @@ -0,0 +1,581 @@ +--- +title: Redis 集群 +date: 2021-10-11 +tags: + - Redis +categories: Redis +--- + +## 一、Redis 集群是啥 + +我们先回顾下前边介绍的几种 Redis 高可用方案：持久化、主从同步和哨兵机制。但这些方案仍有痛点，其中最主要的问题就是存储能力受单机限制，以及没办法实现写操作的负载均衡。 + +Redis 集群刚好解决了上述问题，实现了较为完善的高可用方案。 + + + +### 1.1 Redis 集群化 + +集群，即 Redis Cluster，是 Redis 3.0 开始引入的分布式存储方案。 + +集群由多个节点(Node)组成，Redis 的数据分布在这些节点中。集群中的节点分为主节点和从节点：只有主节点负责读写请求和集群信息的维护；从节点只进行主节点数据和状态信息的复制。 + + + +### 1.2 集群的主要作用 + +1. **数据分区**： 数据分区 *(或称数据分片)* 是集群最核心的功能。集群将数据分散到多个节点，**一方面** 突破了 Redis 单机内存大小的限制，**存储容量大大增加**；**另一方面** 每个主节点都可以对外提供读服务和写服务，**大大提高了集群的响应能力**。 + + Redis 单机内存大小受限问题，例如，如果单机内存太大，`bgsave` 和 `bgrewriteaof` 的 `fork` 操作可能导致主进程阻塞，主从环境下主机切换时可能导致从节点长时间无法提供服务，全量复制阶段主节点的复制缓冲区可能溢出…… + +2. **高可用**： 集群支持主从复制和主节点的 **自动故障转移** *（与哨兵类似）*，当任一节点发生故障时，集群仍然可以对外提供服务。 + +![redis-cluster-framework](https://img.starfish.ink/redis/redis-cluster-framework.png) + +上图展示了 **Redis Cluster** 典型的架构图，集群中的每一个 Redis 节点都 **互相两两相连**，客户端任意 **直连** 到集群中的 **任意一台**，就可以对其他 Redis 节点进行 **读写** 的操作。 + + + +### 1.3 Redis 集群的基本原理 + +![](https://img.starfish.ink/redis/redis-cluster-slot.png) + +Redis 集群中内置了 `16384` 个哈希槽。当客户端连接到 Redis 集群之后，会同时得到一份关于这个 **集群的配置信息**，当客户端具体对某一个 `key` 值进行操作时，会计算出它的一个 Hash 值，然后把结果对 `16384` **求余数**，这样每个 `key` 都会对应一个编号在 `0-16383` 之间的哈希槽，Redis 会根据节点数量 **大致均等** 的将哈希槽映射到不同的节点。 + +再结合集群的配置信息就能够知道这个 `key` 值应该存储在哪一个具体的 Redis 节点中，如果不属于自己管，那么就会使用一个特殊的 `MOVED` 命令来进行一个跳转，告诉客户端去连接这个节点以获取数据： + +```bash +GET x +-MOVED 3999 127.0.0.1:6381 +``` + +`MOVED` 指令第一个参数 `3999` 是 `key` 对应的槽位编号，后面是目标节点地址，`MOVED` 命令前面有一个减号，表示这是一个错误的消息。客户端在收到 `MOVED` 指令后，就立即纠正本地的 **槽位映射表**，那么下一次再访问 `key` 时就能够到正确的地方去获取了。 + + + +## 二、Hello World + +#### 2.1 创建集群节点配置文件 + +创建六个配置文件，分别命名为：`redis_7000.conf`/`redis_7001.conf`…..`redis_7005.conf`，然后根据不同的端口号修改对应的端口值就好了（方便管理可以将这些配置文件放在同一个目录下，我这里放在了 `cluster_config` 目录下）： + +```bash +# 后台执行 +daemonize yes +# 端口号 +port 7000 +# 启动集群模式 +cluster-enabled yes +# 每一个集群节点都有一个配置文件，这个文件是不能手动编辑的。确保每一个集群节点的配置文件不通 +cluster-config-file nodes-7000.conf +# 集群节点的超时时间，单位：ms，超时后集群会认为该节点失败 +cluster-node-timeout 5000 +# 最后将 appendonly 改成 yes(AOF 持久化) +appendonly yes +``` + +#### 2.2 启动 Redis 实例 + +启动刚才配置的 6 个 Redis 实例 + +```bash +redis-server cluster_config/redis_7000.conf +redis-server cluster_config/redis_7001.conf +redis-server cluster_config/redis_7002.conf +redis-server cluster_config/redis_7003.conf +redis-server cluster_config/redis_7004.conf +redis-server cluster_config/redis_7005.conf +``` + +然后执行 `ps -ef | grep redis` 查看是否启动成功： + +![](https://img.starfish.ink/redis/redis-sentinel-ps.png) + +可以看到 `6` 个 Redis 节点都以集群的方式成功启动了，**但是现在每个节点还处于独立的状态**，也就是说它们每一个都各自成了一个集群，还没有互相联系起来，我们需要手动地把他们之间建立起联系。 + +#### 2.3 建立集群 + +创建集群，其实就是节点执行下列命令（Redis 5 之后的方式，之前的版本可以使用 redis-trib.rb 创建）： + +```bash +redis-cli --cluster create 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 --cluster-replicas 1 +``` + +这里稍微解释一下这个 `--replicas 1` 的意思是：我们希望为集群中的每个主节点创建一个从节点。 + +观察控制台输出： + +![](https://img.starfish.ink/redis/redis-cluster-new.jpg) + +看到 `[OK]` 的信息之后，就表示集群已经搭建成功了，可以看到，这里我们正确地创建了三主三从的集群。 + +（这里可能会遇到一些坑，槽没有被完全覆盖，或者 node 不为空这种错误） + +#### 2.4 验证集群 + +我们先使用 `redic-cli` 任意连接一个节点： + +```bash +redis-cli -c -h 127.0.0.1 -p 7000 +127.0.0.1:7000> +``` + +`-c` 表示集群模式；`-h` 指定 ip 地址；`-p` 指定端口。 + +然后随便 `set` 一些值观察控制台输入： + +```bash +127.0.0.1:7000> set name javakeeper +-> Redirected to slot [5798] located at 127.0.0.1:7001 +OK +127.0.0.1:7001> +``` + +可以看到这里 Redis 自动帮我们进行了 `Redirected` 操作跳转到了 `7001` 这个实例上。 + +我们再使用 `cluster info` *(查看集群信息)* 和 `cluster nodes` *(查看节点列表)* 来分别看看：*(任意节点输入均可)* + +![](https://img.starfish.ink/redis/cluster-info.png) + + + +## 三、深入集群原理 + +Redis 集群最核心的功能就是数据分区，数据分区之后又伴随着通信机制和数据结构的建设，所以我们从这 3 个方面来一一深入 + +### 3.1 数据分区方案 + +数据分区有**顺序分区**、**哈希分区**等，其中哈希分区由于其天然的随机性，使用广泛；集群的分区方案便是哈希分区的一种。 + +哈希分区的基本思路是：对数据的特征值（如key）进行哈希，然后根据哈希值决定数据落在哪个节点。常见的哈希分区包括：哈希取余分区、一致性哈希分区、带虚拟节点的一致性哈希分区等。 + +#### 方案一：哈希取余分区 + +哈希取余分区思路非常简单：计算 `key` 的 hash 值，然后对节点数量进行取余，从而决定数据映射到哪个节点上。 + +不过该方案最大的问题是，**当新增或删减节点时**，节点数量发生变化，系统中所有的数据都需要 **重新计算映射关系**，引发大规模数据迁移。 + +这种方式的突出优点是简单性，常用于数据库的分库分表规则，一般采用预分区的方式，提前根据数据量规划好分区数，比如划分为 512 或 1024 张表，保证可支撑未来一段时间的数据量，再根据负载情况将表迁移到其他数据库中。扩容时通常采用翻倍扩容，避免数据映射全部被打乱导致全量迁移的情况 + +#### 方案二：一致性哈希分区 + +一致性哈希算法将 **整个哈希值空间** 组织成一个虚拟的圆环，范围一般是 0 - $2^{32}$，对于每一个数据，根据 `key` 计算 hash 值，确定数据在环上的位置，然后从此位置沿顺时针行走，找到的第一台服务器就是其应该映射到的服务器： + +![](https://img.starfish.ink/redis/redis-consistency.png) + +与哈希取余分区相比，一致性哈希分区将 **增减节点的影响限制在相邻节点**。以上图为例，如果在 `node1` 和 `node2` 之间增加 `node5`，则只有 `node2` 中的一部分数据会迁移到 `node5`；如果去掉 `node2`，则原 `node2` 中的数据只会迁移到 `node3` 中，只有 `node3` 会受影响。 + +一致性哈希分区的主要问题在于，当 **节点数量较少** 时，增加或删减节点，**对单个节点的影响可能很大**，造成数据的严重不平衡。还是以上图为例，如果去掉 `node2`，`node3` 中的数据由总数据的 `1/4` 左右变为 `1/2` 左右，与其他节点相比负载过高。 + +#### 方案三：带有虚拟节点的一致性哈希分区 + +该方案在 **一致性哈希分区的基础上**，引入了 **虚拟节点** 的概念。Redis 集群使用的便是该方案，其中的虚拟节点称为 **槽（slot）**。槽是介于数据和实际节点之间的虚拟概念，每个实际节点包含一定数量的槽，每个槽包含哈希值在一定范围内的数据。槽的范围一般远大于节点数。 + +在使用了槽的一致性哈希分区中，**槽是数据管理和迁移的基本单位**。槽 **解耦** 了 **数据和实际节点** 之间的关系，增加或删除节点对系统的影响很小。仍以上图为例，系统中有 `4` 个实际节点，假设为其分配 `16` 个槽(0-15)； + +- 槽 0-3 位于 node1；4-7 位于 node2；以此类推…. + +如果此时删除 `node2`，只需要将槽 4-7 重新分配即可，例如槽 4-5 分配给 `node1`，槽 6 分配给 `node3`，槽 7 分配给 `node4`；可以看出删除 `node2` 后，数据在其他节点的分布仍然较为均衡。 + + + +Redis 虚拟槽分区的特点: + +- 解耦数据和节点之间的关系，简化了节点扩容和收缩难度。 + +- 节点自身维护槽的映射关系，不需要客户端或者代理服务维护槽分区元数据。 + +- 支持节点、槽、键之间的映射查询，用于数据路由、在线伸缩等场景。 + + + +### 3.2 集群功能限制 + +Redis 集群相对单机在功能上存在一些限制，需要开发人员提前了解，在使用时做好规避。限制如下: + +- key 批量操作支持有限。如 mset、mget，目前只支持具有相同 slot 值的 key 执行批量操作。对于映射为不同 slot 值的 key 由于执行 mget、mget 等操作可能存在于多个节点上因此不被支持。 + + (为此，Redis 引入 HashTag 的概念，使得数据分布算法可以根据 key 的某一部分进行计算，让相关的两条记录落到同一个数据分片，**当一个key包含 {} 的时候，就不对整个 key 做 hash，而仅对 {} 包括的字符串做 hash**。Pipeline 同样可以受益于 hash_tag) + + ```bash + 127.0.0.1:7000> mset javaframework Spring cframework Libevent + (error) CROSSSLOT Keys in request don't hash to the same slot + 127.0.0.1:7000> mset java{framework} Spring c{framework} Libevent + -> Redirected to slot [10840] located at 127.0.0.1:7001 + OK + 127.0.0.1:7001> mget java{framework} c{framework} + 1) "Spring" + 2) "Libevent" + 127.0.0.1:7001> + ``` + +- key 事务操作支持有限。同理只支持多 key 在同一节点上的事务操作，当多个 key 分布在不同的节点上时无法使用事务功能。 + +- key 作为数据分区的最小粒度，因此不能将一个大的键值对象如 hash、list 等映射到不同的节点。 + +- 不支持多数据库空间。单机下的 Redis 可以支持 16 个数据库，集群模式下只能使用一个数据库空间，即 db0。 + +- 复制结构只支持一层，从节点只能复制主节点，不支持嵌套树状复制结构。 + + + +### 3.3 节点通信 + +集群的建立离不开节点之间的通信，例如我们上面启动六个集群节点之后通过 `redis-cli` 命令帮助我们搭建起来了集群，实际上背后每个集群之间的两两连接是通过了 `CLUSTER MEET` 命令发送 `MEET` 消息完成的。 + +通信过程说明： + +1. 集群中的每个节点都会单独开辟一个 TCP 通道，用于节点之间彼此通信，通信端口号在基础端口上加 10000 +2. 每个节点在固定周期内通过特定规则选择几个节点发送 ping 消息 +3. 接收到 ping 消息的节点用 pong 消息作为响应 + +集群中每个节点通过一定规则挑选要通信的节点，每个节点可能知道全部节点，也可能仅知道部分节点，只要这些节点彼此可以正常通信，最终它们会达到一致的状态。当节点出故障、新节点加入、主从角色变化、槽信息 变更等事件发生时，通过不断的 `ping/pong` 消息通信，经过一段时间后所有的节点都会知道整个集群全部节点的最新状态，从而达到集群状态同步的目的。 + +#### 两个端口 + +在 **哨兵系统** 中，节点分为 **数据节点** 和 **哨兵节点**：前者存储数据，后者实现额外的控制功能。在 **集群** 中，没有数据节点与非数据节点之分：**所有的节点都存储数据，也都参与集群状态的维护**。为此，集群中的每个节点，都提供了两个 TCP 端口： + +- **普通端口：** 即我们在前面指定的端口 *(7000等)*。普通端口主要用于为客户端提供服务 *（与单机节点类似）*；但在节点间数据迁移时也会使用。 +- **集群端口：** 端口号是普通端口 + 10000 *（10000是固定值，无法改变）*，如 `7000` 节点的集群端口为 `17000`。**集群端口只用于节点之间的通信**，如搭建集群、增减节点、故障转移等操作时节点间的通信；不要使用客户端连接集群接口。为了保证集群可以正常工作，在配置防火墙时，要同时开启普通端口和集群端口。 + + + +#### Gossip 协议 + +> 对于一个分布式集群来说，它的良好运行离不开集群节点信息和节点状态的正常维护。为了实现这一目标，通常我们可以选择**中心化**的方法，使用一个第三方系统，比如 Zookeeper 或 etcd，来维护集群节点的信息、状态等。同时，我们也可以选择**去中心化**的方法，让每个节点都维护彼此的信息、状态，并且使用集群通信协议 Gossip 在节点间传播更新的信息，从而实现每个节点都能拥有一致的信息。下图就展示了这两种集群节点信息维护的方法，你可以看下。 +> +> ![](https://img.starfish.ink/redis/redis-gossip.png) + +节点间通信，按照通信协议可以分为几种类型：单对单、广播、Gossip 协议等。重点是广播和 Gossip 的对比。 + +- 广播是指向集群内所有节点发送消息。**优点** 是集群的收敛速度快(集群收敛是指集群内所有节点获得的集群信息是一致的)，**缺点** 是每条消息都要发送给所有节点，CPU、带宽等消耗较大。 + +- Gossip 协议的特点是：在节点数量有限的网络中，**每个节点都 “随机” 的与部分节点通信** *（并不是真正的随机，而是根据特定的规则选择通信的节点），经过一番杂乱无章的通信，每个节点的状态很快会达到一致。Gossip 协议的 **优点** 有负载 (比广播)* 低、去中心化、容错性高 *(因为通信有冗余)* 等；**缺点** 主要是集群的收敛速度慢。 + + （为什么需要随机呢？ ） + + Gossip 协议工作原理就是节点彼此不断通信交换信息，一段时间后所有的节点都会知道集群完整的信息，这种方式类似**流言传播**。Gossip 协议的主要职责就是信息交换。信息交换的载体就是节点彼此发送的 Gossip 消息，了解这些消息有助于我们理解集群如何完成信息交换。 + +#### 消息类型 + +集群中的节点采用 **固定频率（每秒10次）** 的 **定时任务** 进行通信相关的工作：判断是否需要发送消息及消息类型、确定接收节点、发送消息等。如果集群状态发生了变化，如增减节点、槽状态变更，通过节点间的通信，所有节点会很快得知整个集群的状态，使集群收敛。 + +节点间发送的消息主要分为 `5` 种：`meet 消息`、`ping 消息`、`pong 消息`、`fail 消息`、`publish 消息`。不同的消息类型，通信协议、发送的频率和时机、接收节点的选择等是不同的： + +![](https://img.starfish.ink/redis/redis-message-type.png) + +- **MEET 消息：** 用于通知新节点加入。消息发送者通知接收者加入到当前集群，meet 消息通信正常完成后，接收节点会加入到集群中并进行周期性的 ping、pong 消息交换。 +- **PING 消息：** 集群里每个节点每秒钟会选择部分节点发送 `PING` 消息，接收者收到消息后会回复一个 `PONG` 消息。**PING 消息的内容是自身节点和部分其他节点的状态信息**，作用是彼此交换信息，以及检测节点是否在线。`PING` 消息使用 Gossip 协议发送，接收节点的选择兼顾了收敛速度和带宽成本（内部频繁进行信息交换，而且 ping/pong 消息会携带当前节点和部分其他节点的状态数据，势必会加重带宽和计算的负担，所以选择需要通信的节点列表就很重要了），**具体规则如下**： + 1. 随机找 5 个节点，在其中选择最久没有通信的 1 个节点； + 2. 扫描节点列表，选择最近一次收到 `PONG` 消息时间大于 `cluster_node_timeout / 2` 的所有节点，防止这些节点长时间未更新。 + +![](https://img.starfish.ink/redis/redis-cluster-ping.png) + +- **PONG消息：** `PONG` 消息封装了自身状态数据。可以分为两种： + 1. **第一种** 是在接到 `MEET/PING` 消息后回复的 `PONG` 消息； + 2. **第二种** 是指节点向集群广播 `PONG` 消息，这样其他节点可以获知该节点的最新信息，例如故障恢复后新的主节点会广播 `PONG` 消息。 +- **FAIL 消息：** 当节点判定集群内另一个节点下线时，会向集群内广播一个 fail 消息，其他节点接收到 fail 消息之后把对应节点更新为下线状态。 +- **PUBLISH 消息：** 节点收到 `PUBLISH` 命令后，会先执行该命令，然后向集群广播这一消息，接收节点也会执行该 `PUBLISH` 命令。 + + + +#### 消息结构 + +所有的消息格式划分为：**消息头**和**消息体**。消息头包含发送节点自身状态数据，接收节点根据消息头就可以获取到发送节点的相关数据，结构如下（ `src/cluster.h` 目录下可以大概看下源码）: + +```c +typedef struct { + char sig[4]; /* 信号标示 */ + uint32_t totlen; /* 消息总长度 */ + uint16_t ver; /* 协议版本 */ + uint16_t port; /* TCP base port number. */ + uint16_t type; /* 消息类型 */ + uint16_t count; /* Only used for some kind of messages. */ + uint64_t currentEpoch; /* 当前发送节点的配置纪元 */ + uint64_t configEpoch; /* 主节点/从节点的主节点配置纪元 */ + uint64_t offset; /* 复制偏移量 */ + char sender[CLUSTER_NAMELEN]; /* Name of the sender node */ + unsigned char myslots[CLUSTER_SLOTS/8]; + char slaveof[CLUSTER_NAMELEN]; + char myip[NET_IP_STR_LEN]; /* Sender IP, if not all zeroed. */ + char notused1[34]; /* 34 bytes reserved for future usage. */ + uint16_t cport; /* Sender TCP cluster bus port */ + uint16_t flags; /* 发送节点标识,区分主从角色，是否下线等 */ + unsigned char state; /* 发送节点所处的集群状态 */ + unsigned char mflags[3]; /* 消息标识: CLUSTERMSG_FLAG[012]_... */ + union clusterMsgData data; /* 消息正文 */ +} clusterMsg; +``` + +集群内所有的消息都采用相同的**消息头**结构 `clusterMsg`，它包含了发送节点关键信息，如节点 id、槽映射、节点标识(主从角色，是否下线)等。 + +**消息体** 在 Redis 内部采用 `clusterMsgData` 结构声明，结构如下: + +```c +union clusterMsgData { + //Ping、Pong和Meet消息类型对应的数据结构 + struct { + /* Array of N clusterMsgDataGossip structures */ + clusterMsgDataGossip gossip[1]; + } ping; + + //Fail消息类型对应的数据结构 + struct { + clusterMsgDataFail about; + } fail; + + //Publish消息类型对应的数据结构 + struct { + clusterMsgDataPublish msg; + } publish; + + //Update消息类型对应的数据结构 + struct { + clusterMsgDataUpdate nodecfg; + } update; + + //Module消息类型对应的数据结构 + struct { + clusterMsgModule msg; + } module; +}; +``` + +消息体 `clusterMsgData` 定义发送消息的数据，其中 ping、meet、pong 都采用 `clusterMsgDataGossip` 数组作为消息体数据，实际消息类型使用消息头的 type 属性区分。每个消息体包含该节点的多个 `clusterMsgDataGossip` 结构数据，用于信息交换，结构如下: + +```c +typedef struct { + char nodename[CLUSTER_NAMELEN]; //节点名称 + uint32_t ping_sent; //节点发送Ping的时间 + uint32_t pong_received; //节点收到Pong的时间 + char ip[NET_IP_STR_LEN]; //节点IP + uint16_t port; //节点和客户端的通信端口 + uint16_t cport; //节点用于集群通信的端口 + uint16_t flags; //节点的标记 + uint32_t notused1; //未用字段 +} clusterMsgDataGossip; +``` + +从 clusterMsgDataGossip 数据结构中，我们可以看到，它里面包含了节点的基本信息，比如节点名称、IP 和通信端口，以及使用 Ping、Pong 消息发送和接收时间来表示的节点运行状态。 + +消息交互的过程就是解析消息头和消息体的过程 + +- 解析消息头过程：消息头包含了发送节点的信息，如果发送节点是新节点且消息是 meet 类型，则加入到本地节点列表；如果是已知节点，则尝试更新发送节点的状态，如槽映射关系、主从角色等状态。 +- 解析消息体过程：如果消息体的 `clusterMsgDataGossip` 数组包含的节点是新节点，则尝试发起与新节点的 meet 握手流程；如果是已知节点，则根据 `clusterMsgDataGossip` 中的 flags 字段判断该节点是否下线，用于故障转移 + +消息处理完后回复 pong 消息，内容同样包含消息头和消息体，发送节点接收到回复的 pong 消息后，采用类似的流程解析处理消息并更新与接收节点最后通信时间，完成一次消息通信。 + + + +#### 数据结构 + +节点需要专门的数据结构来存储集群的状态。所谓集群的状态，是一个比较大的概念，包括：集群是否处于上线状态、集群中有哪些节点、节点是否可达、节点的主从状态、槽的分布…… + +节点为了存储集群状态而提供的数据结构中，最关键的是 `clusterNode` 和 `clusterState` 结构：前者记录了一个节点的状态，后者记录了集群作为一个整体的状态。 + +`clusterNode` 结构保存了 **一个节点的当前状态**，包括创建时间、节点 id、ip 和端口号等。每个节点都会用一个 `clusterNode` 结构记录自己的状态，并为集群内所有其他节点都创建一个 `clusterNode` 结构来记录节点状态。 + +下面列举了 `clusterNode` 的部分字段，并说明了字段的含义和作用（`src/cluster.h`）： + +```c +typedef struct clusterNode { + //节点创建时间 + mstime_t ctime; + //节点id + char name[REDIS_CLUSTER_NAMELEN]; + //节点的ip和端口号 + char ip[REDIS_IP_STR_LEN]; + int port; + //节点标识：整型，每个bit都代表了不同状态，如节点的主从状态、是否在线、是否在握手等 + int flags; + //配置纪元：故障转移时起作用，类似于哨兵的配置纪元 + uint64_t configEpoch; + //槽在该节点中的分布：占用16384/8个字节，16384个比特；每个比特对应一个槽：比特值为1，则该比特对应的槽在节点中；比特值为0，则该比特对应的槽不在节点中 + unsigned char slots[16384/8]; + //节点中槽的数量 + int numslots; + ………… +} clusterNode; +``` + +除了上述字段，`clusterNode` 还包含节点连接、主从复制、故障发现和转移需要的信息等。 + +`clusterState` 结构保存了在当前节点视角下，集群所处的状态。主要字段包括（`src/cluster.h`）： + +```c +typedef struct clusterState { + clusterNode *myself; //自身节点 + uint64_t currentEpoch; //配置纪元 + //集群状态：在线还是下线 + int state; + //集群中至少包含一个槽的节点数量 + int size; + //哈希表，节点名称->clusterNode节点指针 + dict *nodes; + //槽分布信息：数组的每个元素都是一个指向clusterNode结构的指针；如果槽还没有分配给任何节点，则为NULL + clusterNode *slots[16384]; + ………… +} clusterState; +``` + +除此之外，`clusterState` 还包括故障转移、槽迁移等需要的信息。 + + + +### 3.4 集群自动故障转移 + +Redis 集群自身实现了高可用。高可用首先需要解决集群部分失败的场景：当集群内少量节点出现故障时通过自动故障转移保证集群可以正常对外提供服务。 + +之前我们了解了哨兵机制的故障发现和故障转移。集群的实现有些思路类似。 + +#### 3.4.1 故障发现 + +通过定时任务发送 PING 消息检测其他节点状态；节点下线分为**主观下线**和**客观下线**； + +- 主观下线：集群中每个节点都会定期向其他节点发送 ping 消息，接收节点回复 pong 消息作为响应。如果在 `cluster-node-timeout` 时间内通信一直失败，则发送节点会认为接收节点存在故障，把接收节点标记为主观下线(pfail)状态 + +- 当某个节点判断另一个节点主观下线后，相应的节点状态会跟随消息在集群内传播。ping/pong 消息的消息体会携带集群 1/10 的其他节点状态数据， 当接受节点发现消息体中含有主观下线的节点状态时，会在本地找到故障节点的 ClusterNode 结构，保存到下线报告链表中。结构如下: + + ```c + struct clusterNode { /* 认为是主观下线的clusterNode结构 */ + list *fail_reports; /* 记录了所有其他节点对该节点的下线报告 */ ... + }; + ``` + + 通过 Gossip 消息传播，集群内节点不断收集到故障节点的下线报告。当半数以上持有槽的主节点都标记某个节点是主观下线时。触发客观下线流程。 + + 这里有两个问题: + + 1. 为什么必须是负责槽的主节点参与故障发现决策? + + 因为集群模式下 只有处理槽的主节点才负责读写请求和集群槽等关键信息维护，而从节点只进行主节点数据和状态信息的复制。 + + 2. 为什么半数以上处理槽的主节点? + + 必须半数以上是为了应对网络分区等原因造成的集群分割情况，被分割的小集群因为无法完成从主观下线到 客观下线这一关键过程，从而防止小集群完成故障转移之后继续对外提供服务。 + +#### 3.4.2 故障恢复 + +故障节点变为客观下线后，如果下线节点是持有槽的主节点则需要在它的从节点中选出一个替换它，从而保证集群的高可用。 + +下线主节点的所有从节点承担故障恢复的义务，当从节点通过内部定时任务发现自身复制的主节点进入客观下线时，将会触发故障恢复流程： + +1. 资格检查 + + 每个从节点都要检查最后与主节点断线时间，判断是否有资格替换故障的主节点。如果从节点与主节点断线时间超过 `cluster-node-time*cluster-slave-validity-factor`，则当前从节点不具备故障转移资格。参数 `cluster-slave- validity-factor` 用于从节点的有效因子，默认为 10。 + +2. 准备选举时间 + + 当从节点符合故障转移资格后，更新触发故障选举的时间，只有到达该时间后才能执行后续流程 + + ```c + struct clusterState { + ... + mstime_t failover_auth_time; /* 记录之前或者下次将要执行故障选举时间 */ + int failover_auth_rank; /* 记录当前从节点排名 */ } + ``` + + 这里之所以采用延迟触发机制，主要是通过对多个从节点使用不同的延迟选举时间来支持优先级问题。复制偏移量越大说明从节点延迟越低，那么它应该具有更高的优先级来替换故障主节点。 + +3. 发起选举 + + 当从节点定时任务检测到达故障选举时间(`failover_auth_time`)到达后，发起选举流程如下: + + - 更新配置纪元 + + 配置纪元是一个只增不减的整数，每个主节点自身维护一个配置纪元 (`clusterNode.configEpoch`)标示当前主节点的版本，所有主节点的配置纪元 都不相等，从节点会复制主节点的配置纪元。整个集群又维护一个全局的配置纪元(`clusterState.current Epoch`)，用于记录集群内所有主节点配置纪元的最大版本。 + + - 广播选举消息 + +4. 选举投票 + + 只有持有槽的主节点才会处理故障选举消息 (`FAILOVER_AUTH_REQUEST`)，因为每个持有槽的节点在一个配置纪元内都有唯一的一张选票，当接到第一个请求投票的从节点消息时回复 `FAILOVER_AUTH_ACK` 消息作为投票，之后相同配置纪元内其他从节点的选举消息将忽略。 + + 投票过程其实是一个领导者选举的过程，如集群内有 N 个持有槽的主节点代表有 N 张选票。由于在每个配置纪元内持有槽的主节点只能投票给一个 从节点，因此只能有一个从节点获得N/2+1的选票，保证能够找出唯一的从节点。 + + Redis 集群没有直接使用从节点进行领导者选举，主要因为从节点数必须大于等于 3 个才能保证凑够 N/2+1 个节点，将导致从节点资源浪费。使用集群内所有持有槽的主节点进行领导者选举，即使只有一个从节点也可以完成选举过程。 + + 当从节点收集到 N/2+1 个持有槽的主节点投票时，从节点可以执行替换主节点操作，例如集群内有 5 个持有槽的主节点，主节点 b 故障后还有 4 个， 当其中一个从节点收集到 3 张投票时代表获得了足够的选票可以进行替换主节点操作。 + +![](https://img.starfish.ink/redis/redis-cluster-vote.png) + +5. 替换主节点 + + 当从节点收集到足够的选票之后，触发替换主节点操作: + + - 当前从节点取消复制变为主节点。 + + - 执行 clusterDelSlot 操作撤销故障主节点负责的槽，并执行 clusterAddSlot 把这些槽委派给自己。 + + - 向集群广播自己的 pong 消息，通知集群内所有的节点当前从节点变为主节点并接管了故障主节点的槽信息。 + + + +与哨兵一样，集群只实现了主节点的故障转移；从节点故障时只会被下线，不会进行故障转移。因此，使用集群时，应谨慎使用读写分离技术，因为从节点故障会导致读服务不可用，可用性变差。 + + + +### 3.5 客户端访问集群 + +#### 3.5.1 redis-cli + +当节点收到 redis-cli 发来的命令(如 set/get )时，过程如下： + +1. 计算 key 属于哪个槽：`CRC16(key) & 16383` + + 集群提供的 cluster keyslot 命令也是使用上述公式实现 + + ```bash + 127.0.0.1:7001> cluster keyslot k1 + (integer) 12706 + ``` + +2. 判断 key 所在的槽是否在当前节点 + + 假设 key 位于第 i 个槽，`clusterState.slots[i]` 则指向了槽所在的节点，如果 `clusterState.slots[i]==clusterState.myself`，说明槽在当前节点，可以直接在当前节点执行命令；否则，说明槽不在当前节点，则查询槽所在节点的地址(`clusterState.slots[i].ip/port`)，并将其包装到 MOVED 错误中返回给 redis-cli。 + +3. redis-cli 收到 MOVED 错误后，根据返回的 ip 和 port 重新发送请求 + +像 redis-cli 这种客户端又叫 Dummy(傀儡)客户端，它优点是代码实现简单，对客户端协议影响较小，只需要根据重定向信息再次发送请求即可。但是它的弊端很明显，每次执行键命令前都要到 Redis 上进行重定向才能找到要执行命令的节点，额外增加了 IO 开销， 这不是Redis 集群高效的使用方式。正因为如此通常集群客户端都采用另一 种实现：Smart(智能)客户端。 + +#### 3.5.2 Smart 客户端 + +大多数开发语言的 Redis 客户端都采用 Smart 客户端支持集群协议。Smart 客户端通过在内部维护 slot→node 的映射关系，本地就可实现键到节点的查找，从而保证 IO 效率的最大化，而 MOVED 重定向负责协助 Smart 客户端更新 slot→node 映射。 + +以 Jedis 为例，说明 Smart 客户端操作集 群的流程： + +1. 首先在 JedisCluster 初始化时会选择一个运行节点，初始化槽和节点映射关系，使用 `cluster slots` 命令完成 + + ```bash + 127.0.0.1:7001> cluster slots + 1) 1) (integer) 10923 // 开始槽范围 + 2) (integer) 16383 // 结束槽范围 + 3) 1) "127.0.0.1" //主节点ip + 2) (integer) 7002 //从节点端口 + 3) "911f517c6cc3501d42b9bba4aa58b5633375abb5" + 4) 1) "127.0.0.1" + 2) (integer) 7004 + 3) "897b5abb7f79117bb645d79671ced5bebcb855ad" + 2) 1) (integer) 5461 + 2) (integer) 10922 + 3) 1) "127.0.0.1" + 2) (integer) 7001 + 3) "f130468fc299509d709b0867b111342576f00b23" + 4) 1) "127.0.0.1" + 2) (integer) 7003 + 3) "cc122bc7aa0d36da44a5d8c7fbe061940aa81a1d" + 127.0.0.1:7001> + ``` + +2. JedisCluster 解析 cluster slots 结果缓存在本地，并为每个节点创建唯一的 JedisPool 连接池。映射关系在 `JedisClusterInfoCache `类中 +3. 当执行命令时，JedisCluster 根据 key->slot->node 选择需要连接的节点，发送命令。如果成功，则命令执行完毕。如果执行失败，则会随机选择其他节点进行重试，并在出现 MOVED 错误时，使用 cluster slots 重新同步 slot->node 的映射关系 + + + +### 参考与来源 + +1. https://www.mybluelinux.com/redis-explained/ +2. https://redis.io/topics/cluster-tutorial +3. 《Redis 设计与实现》 +4. 《Redis 开发与运维》 +5. https://www.cnblogs.com/kismetv/p/9853040.html \ No newline at end of file diff --git a/docs/data-store/Redis/4.Redis-Conf.md b/docs/data-management/Redis/Redis-Conf.md similarity index 97% rename from docs/data-store/Redis/4.Redis-Conf.md rename to docs/data-management/Redis/Redis-Conf.md index 6531a7a172..e9d84b8574 100644 --- a/docs/data-store/Redis/4.Redis-Conf.md +++ b/docs/data-management/Redis/Redis-Conf.md @@ -1,15 +1,33 @@ ## 简单介绍 -**units单位** +**units 单位** -- 配置大小单位,开头定义了一些基本的度量单位，只支持bytes，不支持bit +- 配置大小单位，开头定义了一些基本的度量单位，只支持 bytes，不支持 bit - 对大小写不敏感 +> Note on units: when memory size is needed, it is possible to specify +> +> it in the usual form of 1k 5GB 4M and so forth: +> +> 1k => 1000 bytes +> +> 1kb => 1024 bytes +> +> 1m => 1000000 bytes +> +> 1mb => 1024*1024 bytes +> +> 1g => 1000000000 bytes +> +> 1gb => 1024*1024*1024 bytes +> +> units are case insensitive so 1GB 1Gb 1gB are all the same. + ### INCLUDES包含 -- 和我们的Struts2配置文件类似，可以通过includes包含，redis.conf可以作为总闸，包含其他 +- 可以通过 includes 包含，redis.conf 可以作为总闸，包含其他 @@ -150,11 +168,11 @@ redis.conf 配置项说明如下： **logfile stdout** -8. 设置数据库的数量，默认数据库为0，可以使用SELECT 命令在连接上指定数据库id +8. 设置数据库的数量，默认数据库为0，可以使用SELECT \命令在连接上指定数据库id **databases 16** -9. 指定在多长时间内，有多少次更新操作，就将数据同步到数据文件，可以多个条件配合 save +9. 指定在多长时间内，有多少次更新操作，就将数据同步到数据文件，可以多个条件配合 save \ \ Redis默认配置文件中提供了三个条件： @@ -182,13 +200,13 @@ redis.conf 配置项说明如下： 13. 设置当本机为slav服务时，设置master服务的IP地址及端口，在Redis启动时，它会自动从master进行数据同步 - **slaveof ** + **slaveof \ \** 14. 当master服务设置了密码保护时，slav服务连接master的密码 - **masterauth ** + **masterauth \** -15. 设置Redis连接密码，如果配置了连接密码，客户端在连接Redis时需要通过AUTH 命令提供密码，默认关闭 +15. 设置Redis连接密码，如果配置了连接密码，客户端在连接Redis时需要通过AUTH \命令提供密码，默认关闭 **requirepass foobared** @@ -198,7 +216,7 @@ redis.conf 配置项说明如下： 17. 指定Redis最大内存限制，Redis在启动时会把数据加载到内存中，达到最大内存后，Redis会先尝试清除已到期或即将到期的Key，当此方法处理 后，仍然到达最大内存设置，将无法再进行写入操作，但仍然可以进行读取操作。Redis新的vm机制，会把Key存放内存，Value会存放在swap区 - **maxmemory ** + **maxmemory \** 18. 指定是否在每次更新操作后进行日志记录，Redis在默认情况下是异步的把数据写入磁盘，如果不开启，可能会在断电时导致一段时间内的数据丢失。因为 redis本身同步数据文件是按上面save条件来同步的，所以有的数据会在一段时间内只存在于内存中。默认为no diff --git a/docs/data-store/Redis/Redis-Database.md b/docs/data-management/Redis/Redis-Database.md similarity index 95% rename from docs/data-store/Redis/Redis-Database.md rename to docs/data-management/Redis/Redis-Database.md index dcb71d3493..9c71d14892 100644 --- a/docs/data-store/Redis/Redis-Database.md +++ b/docs/data-management/Redis/Redis-Database.md @@ -1,10 +1,8 @@ # Redis-Database - - Redis 如何表示一个数据库？数据库操作是如何实现的？ -> 这边文章是基于源码来让我们理解 Redis 的，不管是我们自己下载 redis 还是直接在 Github 上看源码，我们先要了解下 redis 更目录下的重要目录 +> 这篇文章是基于源码来让我们理解 Redis 的，不管是我们自己下载 redis 还是直接在 Github 上看源码，我们先要了解下 redis 根目录下的重要目录 > > - `src`：用C编写的Redis实现 > - `tests`：包含在Tcl中实现的单元测试 @@ -18,8 +16,6 @@ Redis 如何表示一个数据库？数据库操作是如何实现的？ 理解程序如何工作的最简单方法是理解它使用的数据结构。 从 `redis/src` 目录下可以看到 server 的源码文件（基于 `redis-6.0.5`，redis3.0 叫 `redis.c` 和 `redis.h`）。 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gfrqvz1u4uj31ci04yq3l.jpg) - Redis的主头文件 `server.h` 中定义了各种结构体，比如Redis 对象`redisObject` 、存储结构`redisDb `、客户端`client` 等等。 ```c @@ -118,5 +114,4 @@ Redis 解决哈希碰撞的方式 和 Java 中的 HashMap 类似，采取链表 - -https://redisbook.readthedocs.io/en/latest/index.html \ No newline at end of file +> https://redisbook.readthedocs.io/en/latest/index.html diff --git a/docs/data-management/Redis/Redis-Datatype.md b/docs/data-management/Redis/Redis-Datatype.md new file mode 100644 index 0000000000..fc8ff7ad04 --- /dev/null +++ b/docs/data-management/Redis/Redis-Datatype.md @@ -0,0 +1,725 @@ +--- +title: Redis 数据类型篇 +date: 2022-08-25 +tags: + - Redis +categories: Redis +--- + +> 一提到 Redis，我们的脑子里马上就会出现一个词：“快。” +> +> 数据库这么多，为啥 Redis 能有这么突出的表现呢？一方面，这是因为它是内存数据库，所有操作都在内存上完成，内存的访问速度本身就很快。另一方面，这要归功于它的数据结构。 +> +> 这是因为，键值对是按一定的数据结构来组织的，操作键值对最终就是对数据结构进行增删改查操作，所以高效的数据结构是 Redis 快速处理数据的基础。 + + + +我们都知道 Redis 是个 KV 数据库，那 KV 结构的数据在 Redis 中是如何存储的呢？ + +## 一、KV 如何存储？ + +为了实现从键到值的快速访问，Redis 使用了一个哈希表来保存所有键值对。一个哈希表，其实就是一个数组，数组的每个元素称为一个哈希桶。类似我们的 HashMap + +看到这里，你可能会问了：“如果值是集合类型的话，作为数组元素的哈希桶怎么来保存呢？” + +其实，**哈希桶中的元素保存的并不是值本身，而是指向具体值的指针**。这也就是说，不管值是 String，还是集合类型，哈希桶中的元素都是指向它们的指针。 + +在下图中，可以看到，哈希桶中的 entry 元素中保存了 `*key` 和 `*value` 指针，分别指向了实际的键和值，这样一来，即使值是一个集合，也可以通过 `*value` 指针被查找到。 + +![](https://img.starfish.ink/redis/redis-kv.png) + +因为这个哈希表保存了所有的键值对，所以，也把它称为**全局哈希表**。哈希表的最大好处很明显，就是让我们可以用 $O(1)$ 的时间复杂度来快速查找到键值对——我们只需要计算键的哈希值，就可以知道它所对应的哈希桶位置，然后就可以访问相应的 entry 元素。 + +```c +struct redisObject { + unsigned type:4; // 类型 + unsigned encoding:4; // 编码 + unsigned lru:LRU_BITS; // 对象最后一次被访问的时间 + int refcount; //引用计数 + void *ptr; //指向实际值的指针 +}; +``` + +你看，这个查找过程主要依赖于哈希计算，和数据量的多少并没有直接关系。也就是说，不管哈希表里有 10 万个键还是 100 万个键，我们只需要一次计算就能找到相应的键。但是，如果你只是了解了哈希表的 $O(1)$ 复杂度和快速查找特性，那么，当你往 Redis 中写入大量数据后，就可能发现操作有时候会突然变慢了。这其实是因为你忽略了一个潜在的风险点，那就是哈希表的冲突问题和 rehash 可能带来的操作阻塞。 + +### 为什么哈希表操作变慢了？ + +当你往哈希表中写入更多数据时，哈希冲突是不可避免的问题。这里的哈希冲突，也就是指，两个 key 的哈希值和哈希桶计算对应关系时，正好落在了同一个哈希桶中。毕竟，哈希桶的个数通常要少于 key 的数量，这也就是说，难免会有一些 key 的哈希值对应到了同一个哈希桶中。 + +Redis 解决哈希冲突的方式，就是链式哈希。和 JDK7 中的 HahsMap 类似，链式哈希也很容易理解，**就是指同一个哈希桶中的多个元素用一个链表来保存，它们之间依次用指针连接**。 + +![](https://img.starfish.ink/redis/1*gsGJWchCH4V3BukF9xkHpA.jpeg) + +但是，这里依然存在一个问题，哈希冲突链上的元素只能通过指针逐一查找再操作。如果哈希表里写入的数据越来越多，哈希冲突可能也会越来越多，这就会导致某些哈希冲突链过长，进而导致这个链上的元素查找耗时长，效率降低。对于追求“快”的 Redis 来说，这是不太能接受的。 + +所以，Redis 会对哈希表做 rehash 操作。rehash 也就是增加现有的哈希桶数量，让逐渐增多的 entry 元素能在更多的桶之间分散保存，减少单个桶中的元素数量，从而减少单个桶中的冲突。那具体怎么做呢？ + +![img](https://img.starfish.ink/redis/1*2alok5x1yuMJ3Z0V2RK--w.jpeg) + +其实，为了使 rehash 操作更高效，Redis 默认使用了两个全局哈希表：哈希表 1 和哈希表 2。 + +```c +struct dict { + dictType *type; + + dictEntry **ht_table[2]; + unsigned long ht_used[2]; + + long rehashidx; /* rehashing not in progress if rehashidx == -1 */ + + /* Keep small vars at end for optimal (minimal) struct padding */ + int16_t pauserehash; /* If >0 rehashing is paused (<0 indicates coding error) */ + signed char ht_size_exp[2]; /* exponent of size. (size = 1<渐进式 rehash。 + +简单来说就是在第二步拷贝数据时，Redis 仍然正常处理客户端请求，每处理一个请求时，从哈希表 1 中的第一个索引位置开始，顺带着将这个索引位置上的所有 entries 拷贝到哈希表 2 中；等处理下一个请求时，再顺带拷贝哈希表 1 中的下一个索引位置的 entries。 + +渐进式 rehash 这样就巧妙地把一次性大量拷贝的开销，分摊到了多次处理请求的过程中，避免了耗时操作，保证了数据的快速访问。 + +好了，到这里，你应该就能理解，Redis 的键和值是怎么通过哈希表组织的了。对于 String 类型来说，找到哈希桶就能直接增删改查了，所以，哈希表的 $O(1)$ 操作复杂度也就是它的复杂度了。但是，对于集合类型来说，即使找到哈希桶了，还要在集合中再进一步操作。 + +所以集合的操作效率又与集合的底层数据结构有关，接下来我们再说 Redis 的底层数据结构~ + + + +## 一、Redis 的五种基本数据类型和其数据结构 + +![](https://img.starfish.ink/redis/redis-data-type.drawio.png) + +由于 Redis 是基于标准 C 写的，只有最基础的数据类型，因此 Redis 为了满足对外使用的 5 种基本数据类型，开发了属于自己**独有的一套基础数据结构**。 + +**Redis** 有 5 种基础数据类型，它们分别是：**string(字符串)**、**list(列表)**、**hash(字典)**、**set(集合)** 和 **zset(有序集合)**。 + +Redis 底层的数据结构包括：**简单动态数组SDS、链表、字典、跳跃链表、整数集合、快速列表、压缩列表、对象。** + +Redis 为了平衡空间和时间效率，针对 value 的具体类型在底层会采用不同的数据结构来实现，其中哈希表和压缩列表是复用比较多的数据结构，如下图展示了对外数据类型和底层数据结构之间的映射关系： + +![](https://img.starfish.ink/redis/redis-data-types.png) + +下面我们具体看下各种数据类型的底层实现和操作。 + +> 安装好 Redis，我们可以使用 `redis-cli` 来对 Redis 进行命令行的操作，当然 Redis 官方也提供了在线的调试器，你也可以在里面敲入命令进行操作：http://try.redis.io/#run + + + +### 1、String（字符串） + +String 类型是二进制安全的。意思是 Redis 的 String 可以包含任何数据。比如 jpg 图片或者序列化的对象 。 + +Redis 的字符串是动态字符串，是可以修改的字符串，**内部结构实现上类似于 Java 的 ArrayList**，采用预分配冗余空间的方式来减少内存的频繁分配，内部为当前字符串实际分配的空间 capacity 一般要高于实际字符串长度 len。当字符串长度小于 1M 时，扩容都是加倍现有的空间，如果超过 1M，扩容时一次只会多扩 1M 的空间。需要注意的是字符串最大长度为 512M。 + +Redis 没有直接使用 C 语言传统的字符串表示（以空字符结尾的字符数组，以下简称 C 字符串）， 而是自己构建了一种名为简单动态字符串（simple dynamic string，SDS）的抽象类型， 并将 SDS 用作 Redis 的默认字符串表示。 + +根据传统， C 语言使用长度为 `N+1` 的字符数组来表示长度为 `N` 的字符串， 并且字符数组的最后一个元素总是空字符 `'\0'` 。 + +比如说， 下图就展示了一个值为 `"Redis"` 的 C 字符串： + +![](https://img.starfish.ink/redis/c-string.png) + +C 语言使用的这种简单的字符串表示方式， 并不能满足 Redis 对字符串在安全性、效率、以及功能方面的要求 + +[下面说明 SDS 比 C 字符串更适用于 Redis 的原因](http://redisbook.com/preview/sds/different_between_sds_and_c_string.html "SDS 与 C 字符串的区别")： + +- **常数复杂度获取字符串长度** + + 因为 C 字符串并不记录自身的长度信息， 所以为了获取一个 C 字符串的长度， 程序必须遍历整个字符串，对遇到的每个字符进行计数， 直到遇到代表字符串结尾的空字符为止， 这个操作的复杂度为 $O(N)$ + + 和 C 字符串不同， 因为 SDS 在 `len` 属性中记录了 SDS 本身的长度， 所以获取一个 SDS 长度的复杂度仅为 $O(1)$ + + 举个例子， 对于下图所示的 SDS 来说， 程序只要访问 SDS 的 `len` 属性， 就可以立即知道 SDS 的长度为 `5` 字节： + + ![](https://img.starfish.ink/redis/redis-sds.png) + + > **len**：当前字符串的长度（不包括末尾的 null 字符）。这使得 Redis 不需要在每次操作时都遍历整个字符串来获取其长度。 + > + > **alloc**：当前为字符串分配的总空间（包括字符串数据和额外的内存空间）。由于 Redis 使用的是动态分配内存，因此可以避免频繁的内存分配和释放。 + > + > **buf**：实际的字符串数据部分，存储字符串的字符数组。Redis 通过这个区域存储字符串的内容。 + > + > ```c + > struct sdshdr { + > int len; // 当前字符串的长度 + > int alloc; // 为字符串分配的空间, 2.X 版本用一个 free 表示当前字符串缓冲区中未使用的内存量 + > unsigned char buf[]; // 字符串数据 + > }; + > ``` + + 通过使用 SDS 而不是 C 字符串， Redis 将获取字符串长度所需的复杂度从 $O(N)$ 降低到了 $O(1)$ ， 这确保了获取字符串长度的工作不会成为 Redis 的性能瓶颈 + +- **缓冲区溢出/内存泄漏** + + 跟上述问题原因一样，如果执行拼接 or 缩短字符串的操作，操作不当就很容易造成上述问题； + +- **减少修改字符串带来的内存分配次数** + + 因为 C 字符串并不记录自身的长度， 所以对于一个包含了 `N` 个字符的 C 字符串来说， 这个 C 字符串的底层实现总是一个 `N+1` 个字符长的数组（额外的一个字符空间用于保存空字符）。 + + 因为 C 字符串的长度和底层数组的长度之间存在着这种关联性， 所以每次增长或者缩短一个 C 字符串， 程序都总要对保存这个 C 字符串的数组进行一次内存重分配操作： + + - 如果程序执行的是增长字符串的操作， 比如拼接操作（append）， 那么在执行这个操作之前， 程序需要先通过内存重分配来扩展底层数组的空间大小 —— 如果忘了这一步就会产生缓冲区溢出。 + - 如果程序执行的是缩短字符串的操作， 比如截断操作（trim）， 那么在执行这个操作之后， 程序需要通过内存重分配来释放字符串不再使用的那部分空间 —— 如果忘了这一步就会产生内存泄漏。 + + 为了避免 C 字符串的这种缺陷， SDS 通过未使用空间解除了字符串长度和底层数组长度之间的关联： 在 SDS 中， `buf` 数组的长度不一定就是字符数量加一， 数组里面可以包含未使用的字节， 而这些字节的数量就由 SDS 的 `free` 属性记录。 + + 通过未使用空间， SDS 实现了空间预分配和惰性空间释放两种优化策略。 + +- **SDS 如何保证二进制安全**： + + - **不依赖于 null 字符**：**二进制安全**意味着 SDS 字符串可以包含任何数据，包括 null 字节。传统的 C 字符串依赖于 null 字符（'\0'）来表示字符串的结束，但 Redis 的 SDS 不依赖于 null 字符来确定字符串的结束位置。 + - **动态扩展**：SDS 会根据需要动态地扩展其内部缓冲区。Redis 会使用 `alloc` 字段来记录已分配的内存大小。当你向 SDS 中追加数据时，Redis 会确保分配足够的内存，而不需要担心数据的终止符。 + - **不需要二次编码**：二进制数据直接存储在 SDS 的 `buf` 区域内，不需要进行任何编码或转换。因此，Redis 可以原样存储任意二进制数据。 + + > C 语言中的字符串必须符合某种编码（比如 ASCII），并且除了字符串的末尾之外， 字符串里面不能包含空字符， 否则最先被程序读入的空字符将被误认为是字符串结尾 —— 这些限制使得 C 字符串只能保存文本数据， 而不能保存像图片、音频、视频、压缩文件这样的二进制数据 + +- **编码方式**： + - **int**：当值可以用整数表示时，Redis 会使用整数编码。这样可以节省内存，并且在执行数值操作时更高效。 + - **embstr**：这是一种特殊的编码方式，用于存储短字符串。当字符串的长度小于或等于 44 字节时，Redis 会使用 embstr 编码。只读，修改后自动转为 raw。这种编码方式将字符串直接存储在 Redis 对象的内部，这样可以减少内存分配和内存拷贝的次数，提高性能。 + - **raw**：当字符串值不是整数时，Redis 会使用 raw 编码。raw 编码就是简单地将字符串存储为字节序列。Redis 会根据客户端发送的字节序列来存储字符串，因此可以存储任何类型的数据，包括二进制数据。 + +> 可以通过 `TYPE KEY_NAME` 查看 key 所存储的值的类型验证下。 + +### 2、List（列表） + +**Redis 的列表相当于 Java 语言里面的 LinkedList，注意它是链表而不是数组。这意味着 list 的插入和删除操作非常快，时间复杂度为 O(1)，但是索引定位很慢，时间复杂度为 O(n)** + +当列表弹出了最后一个元素之后，该数据结构自动被删除，内存被回收。 + +Redis 的列表结构常用来做异步队列使用。将需要延后处理的任务结构体序列化成字符串塞进 Redis 的列表，另一个线程从这个列表中轮询数据进行处理 + +**右边进左边出：队列** + +```shell +> rpush books python java golang +(integer) 3 +\> llen books + (integer) 3 +\> lpop books + "python" +\> lpop books + "java" +\> lpop books +"golang" +\> lpop books +(nil) +``` + +**右边进右边出：栈** + +```shell +> rpush books python java golang + (integer) 3 + \> rpop books +"golang" +\> rpop books +"java" +\> rpop books +"python" + \> rpop books + (nil) +``` + + + +#### [列表的实现](http://redisbook.com/preview/adlist/implementation.html) + +列表这种数据类型支持存储一组数据。这种数据类型对应两种实现方法，一种是压缩列表（ziplist），另一种是双向循环链表。 + +当列表中存储的数据量比较小的时候，列表就可以采用压缩列表的方式实现。具体需要同时满足下面两个条件： + +- 列表中保存的单个数据（有可能是字符串类型的）小于 64 字节； +- 列表中数据个数少于 512 个。 + +从 Redis 3.2 版本开始，列表的底层实现由压缩列表组成的快速列表（quicklist）所取代。 + +- 快速列表由多个 ziplist 节点组成，每个节点使用指针连接，形成一个链表。 +- 这种方式结合了压缩列表的内存效率和小元素的快速访问，以及双向链表的灵活性。 + +>听到“压缩”两个字，直观的反应就是节省内存。之所以说这种存储结构节省内存，是相较于数组的存储思路而言的。 +> +>它并不是基础数据结构，而是 Redis 自己设计的一种数据存储结构。它有点儿类似数组，通过一片连续的内存空间，来存储数据。不过，它跟数组不同的一点是，它允许存储的数据大小不同。听到“压缩”两个字，直观的反应就是节省内存。之所以说这种存储结构节省内存，是相较于数组的存储思路而言的。我们知道，数组要求每个元素的大小相同，如果我们要存储不同长度的字符串，那我们就需要用最大长度的字符串大小作为元素的大小（假设是 20 个字节）。那当我们存储小于 20 个字节长度的字符串的时候，便会浪费部分存储空间。压缩列表这种存储结构，一方面比较节省内存，另一方面可以支持不同类型数据的存储。而且，因为数据存储在一片连续的内存空间，通过键来获取值为列表类型的数据，读取的效率也非常高。当列表中存储的数据量比较大的时候，也就是不能同时满足刚刚讲的两个条件的时候，列表就要通过双向循环链表来实现了。 + +我们可以从 [源码](https://github.com/redis/redis/blob/unstable/src/adlist.h "redis源码") 的 `adlist.h/listNode` 来看到对其的定义： + +```c +/* Node, List, and Iterator are the only data structures used currently. */ +typedef struct listNode { + struct listNode *prev; // 前置节点 + struct listNode *next; // 后置节点 + void *value; // 节点的值 +} listNode; + +typedef struct listIter { + listNode *next; + int direction; +} listIter; + +typedef struct list { + listNode *head; // 表头节点 + listNode *tail; // 表尾节点 + void *(*dup)(void *ptr); // 节点值复制函数 + void (*free)(void *ptr); // 节点值释放函数 + int (*match)(void *ptr, void *key); // 节点值对比函数 + unsigned long len; // 链表所包含的节点数量 +} list; +``` + + + +### 3、Hash（字典） + +Redis hash 是一个键值对集合。KV 模式不变，但 V 又是一个键值对。 + +#### 1. **Redis Hash 的实现** + +Redis 使用的是 **哈希表（Hash Table）** 来存储 `Hash` 类型的数据。每个 `Hash` 对象都由一个或多个字段（field）和相应的值（value）组成。 + +在 Redis 中，哈希表的数据结构是通过 **`dict`（字典）** 来实现的，`dict` 的结构包括键和值，其中每个键都是哈希表中的字段（field），而值是字段的值。 + +哈希表的基本实现如下： + +- **哈希表**：采用 **开地址法** 来处理哈希冲突。每个 `Hash` 存储为一个哈希表，哈希表根据一定的哈希算法将键映射到一个位置。 +- **动态扩展**：当哈希表存储的元素数量达到一定的阈值时，Redis 会自动进行 **rehash（重哈希）** 操作，重新分配内存并调整哈希表的大小。 + +Redis 字典由 **嵌套的三层结构** 构成，采用链地址法处理哈希冲突： + +```c +// 哈希表结构 +typedef struct dictht { + dictEntry **table; // 二维数组（哈希桶） + unsigned long size; // 总槽位数（2^n 对齐） + unsigned long sizemask; // 槽位掩码（size-1） + unsigned long used; // 已用槽位数量 +} dictht; + +// 字典结构 +typedef struct dict { + dictType *type; // 类型特定函数（实现多态） + void *privdata; // 私有数据 + dictht ht[2]; // 双哈希表（用于渐进式rehash） + long rehashidx; // rehash进度标记（-1表示未进行） +} dict; + +// 哈希节点结构 +typedef struct dictEntry { + void *key; // 键（SDS字符串） + union { + void *val; // 值（Redis对象） + uint64_t u64; + int64_t s64; + } v; + struct dictEntry *next; // 链地址法指针 +} dictEntry; +``` + +#### 2. **Redis Hash 的内部结构** + +Redis 中的 `Hash` 是由 **哈希表** 和 **ziplist** 两种不同的数据结构实现的，具体使用哪一种结构取决于哈希表中键值对的数量和大小。 + +2.1 **哈希表（Hash Table）** + +- 哈希表是 Redis 中实现字典（`dict`）的基础数据结构，使用 **哈希冲突解决方法**（例如链地址法或开地址法）来存储键值对。 +- 每个哈希表由两个数组组成：**键数组（key array）** 和 **值数组（value array）**，它们通过哈希算法映射到表中的相应位置。 + +2.2 **Ziplist（压缩列表）** + +- **ziplist** 是一种内存高效的列表数据结构，在 Redis 中用于存储小型的 `Hash`，当哈希表中的元素个数较少时，Redis 会使用 `ziplist` 来节省内存。 +- Ziplist 是连续内存块，采用压缩存储。它适用于存储小量数据，避免哈希表内存开销。 + +2.3 **切换机制** + +> - 字典中保存的键和值的大小都要小于 64 字节； +> - 字典中键值对的个数要小于 512 个。 + +当 Redis 中 `Hash` 的元素数量较小时，会使用 `ziplist`；当元素数量增多时，会切换到使用哈希表的方式。 + +- **小型 Hash**：当哈希表的字段数量很少时，Redis 会使用 `ziplist` 来存储 `Hash`，因为它可以节省内存。 +- **大型 Hash**：当字段数量较多时，Redis 会将 `ziplist` 转换为 `哈希表` 来优化性能和内存管理。 + +#### 3. Rehash（重哈希） + +**Rehash** 是 Redis 用来扩展哈希表的一种机制。随着 `Hash` 中元素数量的增加，哈希表的负载因子（load factor）会逐渐增大，最终可能会导致哈希冲突增多，降低查询效率。为了解决这个问题，Redis 会在哈希表负载因子达到一定阈值时，执行 **rehash** 操作，即扩展哈希表。 + +3.1 **Rehash 的过程** + +- **扩展哈希表**：Redis 会将哈希表的大小翻倍，并将现有的数据重新映射到新的哈希表中。扩展哈希表的目的是减少哈希冲突，提高查找效率。 + +- **渐进式 rehash（Incremental Rehash）**：Redis 在执行重哈希时，并不会一次性将所有数据都重新映射到新的哈希表中，这样可以避免大量的阻塞操作。Redis 会分阶段地逐步迁移哈希表中的元素。这一过程通过增量的方式进行，逐步从旧哈希表中取出元素，放入新哈希表。 + + 这种增量迁移的方式保证了 **rehash** 操作不会一次性占用过多的 CPU 时间，避免了阻塞。 + +3.2 **Rehash 的触发条件** + +Redis 会在以下情况下触发 rehash 操作： + +- 当哈希表的元素数量超过哈希表容量的负载因子阈值时（例如，默认阈值为 1），Redis 会开始进行 rehash 操作。 +- 当哈希表的空间变得非常紧张，Redis 会执行扩展操作。 + +扩容就会涉及到键值对的迁移。具体来说，迁移操作会在以下两种情况下进行： + +1. **Lazy Rehashing（懒惰重哈希）：** Redis 采用了懒惰重哈希的策略，即在进行哈希表扩容时，并不会立即将所有键值对都重新散列到新的存储桶中。而是在有需要的时候，例如进行读取操作时，才会将相应的键值对从旧存储桶迁移到新存储桶中。这种方式避免了一次性大规模的迁移操作，减少了扩容期间的阻塞时间。 +2. **Redis 事件循环（Event Loop）：** Redis 会在事件循环中定期执行一些任务，包括一些与哈希表相关的操作。在事件循环中，Redis会检查是否有需要进行迁移的键值对，并将它们从旧存储桶迁移到新存储桶中。这样可以保证在系统负载较轻的时候进行迁移，减少对服务性能的影响。 + + + +#### [字典的实现](http://redisbook.com/preview/dict/datastruct.html "字典的实现") + +Redis 字典源码由 `dict.h/dictht` 结构定义： + +```c +typedef struct dictEntry { + void *key; + union { + void *val; + uint64_t u64; + int64_t s64; + double d; + } v; + struct dictEntry *next; +} dictEntry; + +/* This is our hash table structure. Every dictionary has two of this as we + * implement incremental rehashing, for the old to the new table. */ +typedef struct dictht { + dictEntry **table; // 哈希表数组 + unsigned long size; // 哈希表大小 + unsigned long sizemask; // 哈希表大小掩码，用于计算索引值,总是等于 size - 1 + unsigned long used; // 该哈希表已有节点的数量 +} dictht; + +typedef struct dict { + dictType *type; // 类型特定函数 + void *privdata; // 私有数据 + dictht ht[2]; // 哈希表 + long rehashidx; /* rehashing not in progress if rehashidx == -1 */ + unsigned long iterators; /* number of iterators currently running */ +} dict; +``` + + + +### 4、Set（集合） + +集合这种数据类型用来存储一组不重复的数据。这种数据类型也有两种实现方法，一种是基于整数集合，另一种是基于散列表。当要存储的数据，同时满足下面这样两个条件的时候，Redis 就采用整数集合（intset），来实现集合这种数据类型。 + +- 存储的数据都是整数； +- 存储的数据元素个数不超过 512 个。 + +当不能同时满足这两个条件的时候，Redis 就使用散列表来存储集合中的数据。 + +Redis 的 Set 是 String 类型的无序集合。它是通过 HashTable 实现的， 相当于 Java 语言里面的 HashSet，它内部的键值对是无序的唯一的。它的内部实现相当于一个特殊的字典，字典中所有的 value 都是一个值 `NULL`。 + +当集合中最后一个元素移除之后，数据结构自动删除，内存被回收。 + +```c +// 定义整数集合结构 +typedef struct intset { + uint32_t encoding; // 编码方式 + uint32_t length; // 集合长度 + int8_t contents[]; // 元素数组 +} intset; + +// 定义哈希表节点结构 +typedef struct dictEntry { + void *key; // 键 + union { + void *val; // 值 + uint64_t u64; + int64_t s64; + double d; + } v; + struct dictEntry *next; // 指向下一个节点的指针 +} dictEntry; + +// 定义哈希表结构 +typedef struct dictht { + dictEntry **table; // 存储桶数组 + unsigned long size; // 存储桶数量 + unsigned long sizemask; // 存储桶数量掩码 + unsigned long used; // 已使用存储桶数量 +} dictht; + +// 定义集合结构 +typedef struct { + uint32_t encoding; // 编码方式 + dictht *ht; // 哈希表 + intset *is; // 整数集合 +} set; +``` + + + +### 5、Zset(sorted set：有序集合) + +zset 和 set 一样也是 String 类型元素的集合，且不允许重复的成员。不同的是每个元素都会关联一个 double 类型的分数。 + +Redis 正是通过分数来为集合中的成员进行从小到大的排序。zset 的成员是唯一的，但分数(score)却可以重复。 + +它类似于 Java 的 SortedSet 和 HashMap 的结合体，一方面它是一个 set，保证了内部 value 的唯一性，另一方面它可以给每个 value 赋予一个 score，代表这个 value 的排序权重。它的内部实现用的是一种叫做「**跳跃列表**」的数据结构。 + +zset 中最后一个 value 被移除后，数据结构自动删除，内存被回收。 + +实际上，跟 Redis 的其他数据类型一样，有序集合也并不仅仅只有跳表这一种实现方式。当数据量比较小的时候，Redis 会用压缩列表来实现有序集合。具体点说就是，使用压缩列表来实现有序集合的前提，有这样两个： + +- 所有数据的大小都要小于 64 字节； +- 元素个数要小于 128 个。 + + + +> **为什么 Redis 要用跳表来实现有序集合，而不是红黑树？** +> +> Redis 中的有序集合是通过跳表来实现的，严格点讲，其实还用到了散列表。不过散列表我们后面才会讲到，所以我们现在暂且忽略这部分。如果你去查看 Redis 的开发手册，就会发现，Redis 中的有序集合支持的核心操作主要有下面这几个： +> +> - 插入一个数据； +> - 删除一个数据； +> - 查找一个数据； +> - 按照区间查找数据（比如查找值在[100, 356]之间的数据）； +> - 迭代输出有序序列。 +> +> 其中，插入、删除、查找以及迭代输出有序序列这几个操作，红黑树也可以完成，时间复杂度跟跳表是一样的。但是，按照区间来查找数据这个操作，红黑树的效率没有跳表高。对于按照区间查找数据这个操作，跳表可以做到 $O(logn)$ 的时间复杂度定位区间的起点，然后在原始链表中顺序往后遍历就可以了。这样做非常高效。当然，Redis 之所以用跳表来实现有序集合，还有其他原因，比如，跳表更容易代码实现。虽然跳表的实现也不简单，但比起红黑树来说还是好懂、好写多了，而简单就意味着可读性好，不容易出错。还有，跳表更加灵活，它可以通过改变索引构建策略，有效平衡执行效率和内存消耗。不过，跳表也不能完全替代红黑树。因为红黑树比跳表的出现要早一些，很多编程语言中的 Map 类型都是通过红黑树来实现的。我们做业务开发的时候，直接拿来用就可以了，不用费劲自己去实现一个红黑树，但是跳表并没有一个现成的实现，所以在开发中，如果你想使用跳表，必须要自己实现。 + + + +## 二、其他数据类型 + +### Bitmap + +Redis 的 Bitmap 数据结构是一种基于 String 类型的位数组，它允许用户将字符串当作位向量来使用，并对这些位执行位操作。Bitmap 并不是 Redis 中的一个独立数据类型，而是通过在 String 类型上定义的一组位操作命令来实现的。由于 Redis 的 String 类型是二进制安全的，最大长度可以达到 512 MB，因此可以表示最多 $2^{32}$​ 个不同的位。 + +Bitmap 在 Redis 中的使用场景包括但不限于： + +1. **集合表示**：当集合的成员对应于整数 0 到 N 时，Bitmap 可以高效地表示这种集合。 +2. **对象权限**：每个位代表一个特定的权限，类似于文件系统存储权限的方式。 +3. **签到系统**：记录用户在特定时间段内的签到状态。 +4. **用户在线状态**：跟踪大量用户的在线或离线状态。 + +Bitmap 操作的基本命令包括： + +- `SETBIT key offset value`：设置或清除 key 中 offset 位置的位值（只能是 0 或 1）。 +- `GETBIT key offset`：获取 key 中 offset 位置的位值，如果 key 不存在，则返回 0。 + +Bitmap 还支持更复杂的位操作，如： + +- `BITOP operation destkey key [key ...]`：对一个或多个 key 的 Bitmap 进行位操作（AND、OR、NOT、XOR）并将结果保存到 destkey。 +- `BITCOUNT key [start] [end]`：计算 key 中位数为 1 的数量，可选地在指定的 start 和 end 范围内进行计数。 + +Bitmap 在存储空间方面非常高效，例如，表示一亿个用户的登录状态，每个用户用一个位来表示，总共只需要 12 MB 的内存空间。 + +在实际应用中，Bitmap 可以用于实现诸如亿级数据统计、用户行为跟踪等大规模数据集的高效管理。 + +总的来说，Redis 的 Bitmap 是一种非常节省空间且功能强大的数据结构，适用于需要对大量二进制数据进行操作的场景。 + +```sh +127.0.0.1:6379> setbit s 1 1 +(integer) 0 +127.0.0.1:6379> setbit s 2 1 +(integer) 0 +127.0.0.1:6379> setbit s 4 1 +(integer) 0 +127.0.0.1:6379> setbit s 9 1 +(integer) 0 +127.0.0.1:6379> setbit s 10 1 +(integer) 0 +127.0.0.1:6379> setbit s 13 1 +(integer) 0 +127.0.0.1:6379> setbit s 15 1 +(integer) 0 +127.0.0.1:6379> get s +"he" +``` + +上面这个例子可以理解为「零存整取」，同样我们还也可以「零存零取」，「整存零取」。「零存」就是使用 setbit 对位值进行逐个设置，「整存」就是使用字符串一次性填充所有位数组，覆盖掉旧值。 + + + +### HyperLogLog + +Redis 在 2.8.9 版本添加了 HyperLogLog 结构。 + +Redis HyperLogLog 是一种用于基数统计的数据结构，它提供了一个近似的、不精确的解决方案来估算集合中唯一元素的数量，即集合的基数。HyperLogLog 特别适用于需要处理大量数据并且对精度要求不是特别高的场景，因为它使用非常少的内存（通常每个 HyperLogLog 实例只需要 12.4KB 左右，无论集合中有多少元素）。 + +HyperLogLog 的主要特点包括： + +1. **近似统计**：HyperLogLog 不保证精确计算基数，但它提供了一个非常接近真实值的近似值。 +2. **内存效率**：HyperLogLog 能够使用固定大小的内存来估算基数，这使得它在处理大规模数据集时非常有用。 +3. **可合并性**：多个 HyperLogLog 实例可以合并，以估算多个集合的并集基数。 + +HyperLogLog 的主要命令包括： + +- `PFADD key element [element ...]`：向 HyperLogLog 数据结构添加元素。如果 key 不存在，它将被创建。 +- `PFCOUNT key`：返回给定 HyperLogLog 的近似基数。 +- `PFMERGE destkey sourcekey [sourcekey ...]`：将多个 HyperLogLog 结构合并到一个单独的 HyperLogLog 结构中。 + +HyperLogLog 的工作原理基于一个数学算法，它使用一个固定大小的位数组和一些哈希函数。当添加一个新元素时，它被哈希函数映射到一个位数组的索引，并根据哈希值的前几位来设置位数组中的位。基数估算是通过分析位数组中 0 的位置来完成的。 + +由于 HyperLogLog 提供的是近似值，它有一个标准误差率，通常在 0.81% 左右。这意味着如果实际基数是 1000，HyperLogLog 估算的基数可能在 992 到 1008 之间。 + +HyperLogLog 是处理大数据集基数统计的理想选择，尤其是当数据集太大而无法在内存中完全加载时。它在数据挖掘、日志分析、用户行为分析等领域有着广泛的应用。 + +场景：可以用来统计站点的UV... + +> [HyperLogLog图解](http://content.research.neustar.biz/blog/hll.html ) + + + + + +## 三、Redis常见数据类型和命令查阅： + +[Redis命令中心](http://www.redis.cn/commands.html) + +[Redis 命令参考](http://redisdoc.com/ ) + +#### Key（键）常用命令 + +| 命令 | 用法 | 描述 | 示例 | +| ------------ | ---------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | +| **DEL** | DEL key [key ...] | 删除给定的一个或多个 key。 不存在的 key 会被忽略 | | +| DUMP | DUMP key | 序列化给定 key ，并返回被序列化的值，使用 RESTORE 命令可以将这个值反序列化为 Redis 键 | | +| EXISTS | EXISTS key | 检查给定 key 是否存在 | | +| **EXPIRE** | EXPIRE key seconds | 为给定 key 设置生存时间，当 key 过期时(生存时间为 0 )，它会被自动删除 | | +| **PERSIST** | PERSIST key | 移除 key 的过期时间，key 将持久保持。 | | +| **EXPIREAT** | EXPIREAT key timestamp | EXPIREAT 的作用和 EXPIRE 类似，都用于为 key 设置生存时间。 不同在于 EXPIREAT 命令接受的时间参数是 UNIX 时间戳(unix timestamp) | EXPIREAT cache 1355292000 # 这个 key 将在 2012.12.12 过期 | +| **KEYS** | KEYS pattern | 查找所有符合给定模式 pattern 的 key | KEYS * # 匹配数据库内所有 key | +| MOVE | MOVE key db | 将当前数据库的 key 移动到给定的数据库 db 当中如果当前数据库(源数据库)和给定数据库(目标数据库)有相同名字的给定 key ，或者 key 不存在于当前数据库，那么 MOVE 没有任何效果。 因此，也可以利用这一特性，将 MOVE 当作锁(locking)原语(primitive) | MOVE song 1 # 将 song 移动到数据库 1 | +| **TTL** | TTL key | 以秒为单位，返回给定 key 的剩余生存时间(TTL, time to live)当 key 不存在时，返回 -2 。当 key 存在但没有设置剩余生存时间时，返回 -1 。否则，以秒为单位，返回 key 的剩余生存时间 | | +| PTTL | PTTL key | 以毫秒为单位返回 key 的剩余的过期时间。 | | +| **TYPE** | TYPE key | 返回 key 所储存的值的类型 | | +| RENAME | RENAME key newkey | 将 key 改名为 newkey 。当 key 和 newkey 相同，或者 key 不存在时，返回一个错误。当 newkey 已经存在时， RENAME 命令将覆盖旧值 | | + + + +#### String （字符串）常用命令 + +| 命令 | 用法 | 描述 | 示例 | +| ----------- | ----------------------------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | +| **SET** | SET key value [EX seconds] [PX milliseconds] [NX\|XX] | 将字符串值 value 关联到 key 。如果 key 已经持有其他值， SET 就覆写旧值，无视类型 | SET key "value" | +| **MSET** | MSET key value [key value ...] | 同时设置一个或多个 key-value 对。如果某个给定 key 已经存在，那么 MSET 会用新值覆盖原来的旧值，如果这不是你所希望的效果，请考虑使用 MSETNX 命令：它只会在所有给定 key 都不存在的情况下进行设置操作 | MSET date "2012.3.30" time "11:00 a.m." weather "sunny" | +| **SETNX** | SETNX key value | 将 key 的值设为 value ，当且仅当 key 不存在。若给定的 key 已经存在，则 SETNX 不做任何动作 SETNX 是『SET if Not eXists』(如果不存在，则 SET)的简写 | | +| MSETNX | MSETNX key value [key value ...] | 同时设置一个或多个 key-value 对，当且仅当所有给定 key 都不存在。即使只有一个给定 key 已存在， MSETNX 也会拒绝执行所有给定 key 的设置操作 | | +| SETRANGE | SETRANGE key offset value | 用 value 参数覆写(overwrite)给定 key 所储存的字符串值，从偏移量 offset 开始。不存在的 key 当作空白字符串处理 | | +| SETBIT | SETBIT key offset value | 对 key 所储存的字符串值，设置或清除指定偏移量上的位(bit) | GETBIT bit 100 # bit 默认被初始化为 0 | +| SETEX | SETEX key seconds value | 将值 value 关联到 key ，并将 key 的生存时间设为 seconds (以秒为单位)。如果 key 已经存在， SETEX 命令将覆写旧值。 | | +| PSETEX | PSETEX key milliseconds value | 这个命令和 SETEX 命令相似，但它以毫秒为单位设置 key 的生存时间，而不是像 SETEX 命令那样，以秒为单位 | | +| STRLEN | STRLEN key | 返回 key 所储存的字符串值的长度。当 key 储存的不是字符串值时，返回一个错误 | | +| **GET** | GET key | 返回 key 所关联的字符串值。如果 key 不存在那么返回特殊值 nil | | +| **MGET** | MGET key [key ...] | 返回所有(一个或多个)给定 key 的值。如果给定的 key 里面，有某个 key 不存在，那么这个 key 返回特殊值 nil 。因此，该命令永不失败 | | +| GETRANGE | GETRANGE key start end | 返回 key 中字符串值的子字符串，字符串的截取范围由 start 和 end 两个偏移量决定(包括 start 和 end 在内)。负数偏移量表示从字符串最后开始计数， -1 表示最后一个字符， -2 表示倒数第二个，以此类推。 | GETRANGE greeting 0 4 | +| GETSET | GETSET key value | 将给定 key 的值设为 value ，并返回 key 的旧值(old value)。当 key 存在但不是字符串类型时，返回一个错误。 | | +| GETBIT | GETBIT key offset | 对 key 所储存的字符串值，获取指定偏移量上的位(bit)。当 offset 比字符串值的长度大，或者 key 不存在时，返回 0 | | +| **APPEND** | APPEND key value | 如果 key 已经存在并且是一个字符串， APPEND 命令将 value 追加到 key 原来的值的末尾。如果 key 不存在， APPEND 就简单地将给定 key 设为 value ，就像执行 SET key value 一样 | | +| **DECR** | DECR key | 将 key 中储存的数字值减一。如果 key 不存在，那么 key 的值会先被初始化为 0 ，然后再执行 DECR 操作 | redis> SET failure_times 10OK redis> DECR failure_times(integer) 9 | +| **DECRBY** | DECRBY key decrement | 将 key 所储存的值减去减量 decrement | | +| **INCR** | INCR key | 将 key 中储存的数字值增一。如果 key 不存在，那么 key 的值会先被初始化为 0 ，然后再执行 INCR 操作 | | +| **INCRBY** | INCRBY key increment | 将 key 所储存的值加上增量 increment | | +| INCRBYFLOAT | INCRBYFLOAT key increment | 为 key 中所储存的值加上浮点数增量 increment | INCRBYFLOAT mykey 0.1 | +| BITCOUNT | BITCOUNT key [start] [end] | 计算给定字符串中，被设置为 1 的比特位的数量 | | +| BITOP | BITOP operation destkey key [key ...] | 对一个或多个保存二进制位的字符串 key 进行位元操作，并将结果保存到 destkey 上。 | | + + + +#### List（列表）常用命令 + +| 命令 | 用法 | 描述 | 示例 | +| ---------- | ------------------------------------- | ------------------------------------------------------------ | --------------------------------------------- | +| **LPUSH** | LPUSH key value [value ...] | 将一个或多个值 value 插入到列表 key 的表头如果有多个 value 值，那么各个 value 值按从左到右的顺序依次插入到表头 | 正着进反着出 | +| LPUSHX | LPUSHX key value | 将值 value 插入到列表 key 的表头，当且仅当 key 存在并且是一个列表。和 LPUSH 命令相反，当 key 不存在时， LPUSHX 命令什么也不做 | | +| **RPUSH** | RPUSH key value [value ...] | 将一个或多个值 value 插入到列表 key 的表尾(最右边) | 怎么进怎么出 | +| RPUSHX | RPUSHX key value | 将值 value 插入到列表 key 的表尾，当且仅当 key 存在并且是一个列表。和 RPUSH 命令相反，当 key 不存在时， RPUSHX 命令什么也不做。 | | +| **LPOP** | LPOP key | 移除并返回列表 key 的头元素。 | | +| **BLPOP** | BLPOP key [key ...] timeout | 移出并获取列表的第一个元素， 如果列表没有元素会阻塞列表直到等待超时或发现可弹出元素为止 | | +| **RPOP** | RPOP key | 移除并返回列表 key 的尾元素。 | | +| **BRPOP** | BRPOP key [key ...] timeout | 移出并获取列表的最后一个元素， 如果列表没有元素会阻塞列表直到等待超时或发现可弹出元素为止。 | | +| BRPOPLPUSH | BRPOPLPUSH source destination timeout | 从列表中弹出一个值，将弹出的元素插入到另外一个列表中并返回它； 如果列表没有元素会阻塞列表直到等待超时或发现可弹出元素为止。 | | +| RPOPLPUSH | RPOPLPUSH source destinationb | 命令 RPOPLPUSH 在一个原子时间内，执行以下两个动作：将列表 source 中的最后一个元素(尾元素)弹出，并返回给客户端。将 source 弹出的元素插入到列表 destination ，作为 destination 列表的的头元素 | RPOPLPUSH list01 list02 | +| **LSET** | LSET key index value | 将列表 key 下标为 index 的元素的值设置为 value | | +| **LLEN** | LLEN key | 返回列表 key 的长度。如果 key 不存在，则 key 被解释为一个空列表，返回 0 .如果 key 不是列表类型，返回一个错误 | | +| **LINDEX** | LINDEX key index | 返回列表 key 中，下标为 index 的元素。下标(index)参数 start 和 stop 都以 0 为底，也就是说，以 0 表示列表的第一个元素，以 1 表示列表的第二个元素，以此类推。相当于 Java 链表的`get(int index)`方法，它需要对链表进行遍历，性能随着参数`index`增大而变差。 | | +| **LRANGE** | LRANGE key start stop | 返回列表 key 中指定区间内的元素，区间以偏移量 start 和 stop 指定 | | +| LREM | LREM key count value | 根据参数 count 的值，移除列表中与参数 value 相等的元素 | | +| LTRIM | LTRIM key start stop | 对一个列表进行修剪(trim)，就是说，让列表只保留指定区间内的元素，不在指定区间之内的元素都将被删除 | | +| LINSERT | LINSERT key BEFORE\|AFTER pivot value | 将值 value 插入到列表 key 当中，位于值 pivot 之前或之后。当 pivot 不存在于列表 key 时，不执行任何操作。当 key 不存在时， key 被视为空列表，不执行任何操作。如果 key 不是列表类型，返回一个错误。 | LINSERT list01 before c++ c#(在c++之前加上C#) | + +#### **Hash**（哈希表）常用命令 + +| 命令 | 用法 | 描述 | 示例 | +| ------------ | ---------------------------------------------- | ------------------------------------------------------------ | ---- | +| **HSET** | HSET key field value | 将哈希表 key 中的域 field 的值设为 value 。如果 key 不存在，一个新的哈希表被创建并进行 HSET 操作。如果域 field 已经存在于哈希表中，旧值将被覆盖。 | | +| **HMSET** | HMSET key field value [field value ...] | 同时将多个 field-value (域-值)对设置到哈希表 key 中。此命令会覆盖哈希表中已存在的域。 | | +| **HSETNX** | HSETNX key field value | 将哈希表 key 中的域 field 的值设置为 value ，当且仅当域 field 不存在。若域 field 已经存在，该操作无效 | | +| **HGET** | HGET key field | 返回哈希表 key 中给定域 field 的值 | | +| **HMGET** | HMGET key field [field ...] | 返回哈希表 key 中，一个或多个给定域的值。 | | +| **HGETALL** | HGETALL key | 返回哈希表 key 中，所有的域和值。在返回值里，紧跟每个域名(field name)之后是域的值(value)，所以返回值的长度是哈希表大小的两倍 | | +| HDEL | HDEL key field [field ...] | 删除哈希表 key 中的一个或多个指定域，不存在的域将被忽略 | | +| HEXISTS | HEXISTS key field | 查看哈希表 key 中，给定域 field 是否存在 | | +| HLEN | HLEN key | 返回哈希表 key 中域的数量 | | +| **HKEYS** | HKEYS key | 返回哈希表 key 中的所有域 | | +| **HVALS** | HVALS key | 返回哈希表 key 中所有域的值 | | +| HSTRLEN | HSTRLEN key field | 返回哈希表 key 中，与给定域 field 相关联的值的字符串长度（string length）。如果给定的键或者域不存在，那么命令返回 0 | | +| HINCRBY | HINCRBY key field increment | 为哈希表 key 中的域 field 的值加上增量 increment | | +| HINCRBYFLOAT | HINCRBYFLOAT key field increment | 为哈希表 key 中的域 field 加上浮点数增量 increment | | +| HSCAN | HSCAN key cursor [MATCH pattern] [COUNT count] | 迭代哈希表中的键值对。 | | + + + +#### Set（集合）常用命令 + +| 命令 | 用法 | 描述 | 示例 | +| ------------- | ---------------------------------------------- | ------------------------------------------------------------ | ---- | +| **SADD** | SADD key member [member ...] | 将一个或多个 member 元素加入到集合 key 当中，已经存在于集合的 member 元素将被忽略。假如 key 不存在，则创建一个只包含 member 元素作成员的集合。当 key 不是集合类型时，返回一个错误 | | +| **SCARD** | SCARD key | 返回集合 key 的基数(集合中元素的数量)。 | | +| **SDIFF** | SDIFF key [key ...] | 返回一个集合的全部成员，该集合是所有给定集合之间的差集。不存在的 key 被视为空集。 | 差集 | +| SDIFFSTORE | SDIFFSTORE destination key [key ...] | 这个命令的作用和 SDIFF 类似，但它将结果保存到 destination 集合，而不是简单地返回结果集。如果 destination 集合已经存在，则将其覆盖。destination 可以是 key 本身。 | | +| **SINTER** | SINTER key [key ...] | 返回一个集合的全部成员，该集合是所有给定集合的交集。不存在的 key 被视为空集。当给定集合当中有一个空集时，结果也为空集(根据集合运算定律) | 交集 | +| SINTERSTORE | SINTERSTORE destination key [key ...] | 这个命令类似于 SINTER 命令，但它将结果保存到 destination 集合，而不是简单地返回结果集。如果 destination 集合已经存在，则将其覆盖。destination 可以是 key 本身 | | +| **SUNION** | SUNION key [key ...] | 返回一个集合的全部成员，该集合是所有给定集合的并集。不存在的 key 被视为空集 | 并集 | +| SUNIONSTORE | SUNIONSTORE destination key [key ...] | 这个命令类似于 SUNION 命令，但它将结果保存到 destination 集合，而不是简单地返回结果集。如果 destination 已经存在，则将其覆盖。destination 可以是 key 本身 | | +| **SMEMBERS** | SMEMBERS key | 返回集合 key 中的所有成员。不存在的 key 被视为空集合 | | +| SRANDMEMBER | SRANDMEMBER key [count] | 如果命令执行时，只提供了 key 参数，那么返回集合中的一个随机元素 | | +| **SISMEMBER** | SISMEMBER key member | 判断 member 元素是否集合 key 的成员 | | +| SMOVE | SMOVE source destination member | 将 member 元素从 source 集合移动到 destination 集合。 | | +| SPOP | SPOP key | 移除并返回集合中的一个随机元素。如果只想获取一个随机元素，但不想该元素从集合中被移除的话，可以使用 SRANDMEMBER 命令。 | | +| **SREM** | SREM key member [member ...] | 移除集合 key 中的一个或多个 member 元素，不存在的 member 元素会被忽略。当 key 不是集合类型，返回一个错误 | | +| SSCAN | SSCAN key cursor [MATCH pattern] [COUNT count] | 迭代集合中的元素 | | + + + +#### SortedSet（有序集合）常用命令 + +| 命令 | 用法 | 描述 | 示例 | +| ---------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | ---- | +| **ZADD** | ZADD key score1 member1 [score2 member2] | 向有序集合添加一个或多个成员，或者更新已存在成员的分数 | | +| **ZCARD** | ZCARD key | 返回有序集 key 的基数。 | | +| **ZCOUNT** | ZCOUNT key min max | 返回有序集 key 中， score 值在 min 和 max 之间(默认包括 score 值等于 min 或 max )的成员的数量。关于参数 min 和 max 的详细使用方法，请参考 ZRANGEBYSCORE 命令。 | | +| **ZRANGE** | ZRANGE key start stop [WITHSCORES] | 返回有序集 key 中，指定区间内的成员。其中成员的位置按 score 值递增(从小到大)来排序 | | +| **ZREVRANGE** | ZREVRANGE key start stop [WITHSCORES] | 返回有序集 key 中，指定区间内的成员。其中成员的位置按 score 值递减(从大到小)来排列。具有相同 score 值的成员按字典序的逆序(reverse lexicographical order)排列。除了成员按 score 值递减的次序排列这一点外， ZREVRANGE 命令的其他方面和 ZRANGE 命令一样。 | | +| ZREVRANGEBYSCORE | ZREVRANGEBYSCORE key max min [WITHSCORES] [LIMIT offset count] | 返回有序集 key 中， score 值介于 max 和 min 之间(默认包括等于 max 或 min )的所有的成员。有序集成员按 score 值递减(从大到小)的次序排列。 | | +| ZREVRANK | ZREVRANK key member | 返回有序集 key 中成员 member 的排名。其中有序集成员按 score 值递减(从大到小)排序。排名以 0 为底，也就是说， score 值最大的成员排名为 0 。使用 ZRANK 命令可以获得成员按 score 值递增(从小到大)排列的排名。 | | +| ZSCORE | ZSCORE key member | 返回有序集 key 中，成员 member 的 score 值。如果 member 元素不是有序集 key 的成员，或 key 不存在，返回 nil 。 | | +| ZRANGEBYSCORE | ZRANGEBYSCORE key min max [WITHSCORES] [LIMIT offset count] | 返回有序集 key 中，所有 score 值介于 min 和 max 之间(包括等于 min 或 max )的成员。有序集成员按 score 值递增(从小到大)次序排列。 | | +| ZRANK | ZRANK key member | 返回有序集 key 中成员 member 的排名。其中有序集成员按 score 值递增(从小到大)顺序排列。 | | +| **ZINCRBY** | ZINCRBY key increment member | 为有序集 key 的成员 member 的 score 值加上增量 increment | | +| ZREM | ZREM key member [member ...] | 移除有序集 key 中的一个或多个成员，不存在的成员将被忽略。当 key 存在但不是有序集类型时，返回一个错误。 | | +| ZREMRANGEBYRANK | ZREMRANGEBYRANK key start stop | 移除有序集 key 中，指定排名(rank)区间内的所有成员 | | +| ZREMRANGEBYSCORE | ZREMRANGEBYSCORE key min max | 移除有序集 key 中，所有 score 值介于 min 和 max 之间(包括等于 min 或 max )的成员。 | | +| ZUNIONSTORE | ZUNIONSTORE destination numkeys key [key ...] [WEIGHTS weight [weight ...]] [AGGREGATE SUM\|MIN\|MAX] | 计算给定的一个或多个有序集的并集，其中给定 key 的数量必须以 numkeys 参数指定，并将该并集(结果集)储存到 destination 。默认情况下，结果集中某个成员的 score 值是所有给定集下该成员 score 值之 和 。 | | +| ZINTERSTORE | ZINTERSTORE destination numkeys key [key ...] [WEIGHTS weight [weight ...]] [AGGREGATE SUM\|MIN\|MAX] | 计算给定的一个或多个有序集的交集，其中给定 key 的数量必须以 numkeys 参数指定，并将该交集(结果集)储存到 destination 。默认情况下，结果集中某个成员的 score 值是所有给定集下该成员 score 值之和. | | +| ZSCAN | ZSCAN key cursor [MATCH pattern] [COUNT count] | 迭代有序集合中的元素（包括元素成员和元素分值） | | +| ZRANGEBYLEX | ZRANGEBYLEX key min max [LIMIT offset count] | 当有序集合的所有成员都具有相同的分值时，有序集合的元素会根据成员的字典序（lexicographical ordering）来进行排序，而这个命令则可以返回给定的有序集合键 key 中，值介于 min 和 max 之间的成员。 | | +| ZLEXCOUNT | ZLEXCOUNT key min max | 对于一个所有成员的分值都相同的有序集合键 key 来说，这个命令会返回该集合中，成员介于 min 和 max 范围内的元素数量。这个命令的 min 参数和 max 参数的意义和 ZRANGEBYLEX 命令的 min 参数和 max 参数的意义一样 | | +| ZREMRANGEBYLEX | ZREMRANGEBYLEX key min max | 对于一个所有成员的分值都相同的有序集合键 key 来说，这个命令会移除该集合中，成员介于 min 和 max 范围内的所有元素。这个命令的 min 参数和 max 参数的意义和 ZRANGEBYLEX 命令的 min 参数和 max 参数的意义一样 | | + + + + + + + diff --git a/docs/data-management/Redis/Redis-IO-Model.md b/docs/data-management/Redis/Redis-IO-Model.md new file mode 100644 index 0000000000..497291bb3a --- /dev/null +++ b/docs/data-management/Redis/Redis-IO-Model.md @@ -0,0 +1,22 @@ +## 为什么单线程的 Redis 能那么快？ + +首先，我们要知道一个事实，我们通常说，Redis 是单线程，主要是指 **Redis 的网络 IO 和键值对读写是由一个线程来完成的，这也是 Redis 对外提供键值存储服务的主要流程**。但 Redis 的其他功能，比如持久化、异步删除、集群数据同步等，其实是由额外的线程执行的。 + + + +### Redis 为什么用单线程？ + +- Redis是基于内存的操作，CPU不是Redis的瓶颈，Redis的瓶颈最有可能是机器内存的大小或者网络带宽。既然单线程容易实现，而且CPU不会成为瓶颈，那就顺理成章地采用单线程的方案了。 +- 不需要各种锁的性能消耗：多线程编程模式面临的共享资源的并发访问控制问题。并发访问控制一直是多线程开发中的一个难点问题，如果没有精细的设计，比如说，只是简单地采用一个粗粒度互斥锁，就会出现不理想的结果：即使增加了线程，大部分线程也在等待获取访问共享资源的互斥锁，并行变串行，系统吞吐率并没有随着线程的增加而增加。 + + + +### 单线程 Redis 为什么那么快？ + +通常来说，单线程的处理能力要比多线程差很多，但是 Redis 却能使用单线程模型达到每秒数十万级别的处理能力，这是为什么呢？其实，这是 Redis 多方面设计选择的一个综合结果。 + +一方面，Redis 的大部分操作在内存上完成，再加上它采用了高效的数据结构，例如哈希表和跳表，这是它实现高性能的一个重要原因。 + +另一方面，采用单线程，避免了不必要的上下文切换和竞争条件 + +最后一个，就是 Redis 采用了多路复用机制，使其在网络 IO 操作中能并发处理大量的客户端请求，实现高吞吐率。 \ No newline at end of file diff --git a/docs/data-management/Redis/Redis-Lock.md b/docs/data-management/Redis/Redis-Lock.md new file mode 100644 index 0000000000..5719d23d85 --- /dev/null +++ b/docs/data-management/Redis/Redis-Lock.md @@ -0,0 +1,870 @@ +--- +title: 分布式锁 +date: 2021-10-09 +tags: + - Redis +categories: Redis +--- + +![](https://img.starfish.ink/redis/redis-lock-banner.jpg) + +> 分布式锁的文章其实早就烂大街了，但有些“菜鸟”写的太浅，或者自己估计都没搞明白，没用过，看完后我更懵逼了，有些“大牛”写的吧，又太高级，只能看懂前半部分，后边就开始讲论文了，也比较懵逼，所以还得我这个中不溜的来总结下 +> +> 文章拢共分为几个部分： +> +> - 什么是分布式锁 +> - 分布式锁的实现要求 +> - 基于 Redisson 实现的 Redis 分布式锁 +> - 再简单说下 RedLock + +## 一、什么是分布式锁 + +**分布式~~锁**，要这么念，首先得是『分布式』，然后才是『锁』 + +- 分布式：这里的分布式指的是分布式系统，涉及到好多技术和理论，包括 CAP 理论、分布式存储、分布式事务、分布式锁... + + > 分布式系统是由一组通过网络进行通信、为了完成共同的任务而协调工作的计算机节点组成的系统。 + > + > 分布式系统的出现是为了用廉价的、普通的机器完成单个计算机无法完成的计算、存储任务。其目的是**利用更多的机器，处理更多的数据**。 + +- 锁：对对，就是你想的那个，Javer 学的第一个锁应该就是 `synchronized` + + > Java 初级面试问题，来拼写下 **赛克瑞纳挨日的** + + 从锁的使用场景有来看下边这 3 种锁： + + **线程锁**：`synchronized` 是用在方法或代码块中的，我们把它叫『线程锁』，线程锁的实现其实是靠线程之间共享内存实现的，说白了就是内存中的一个整型数，有空闲、上锁这类状态，比如 synchronized 是在对象头中的 Mark Word 有个锁状态标志，Lock 的实现类大部分都有个叫 `volatile int state` 的共享变量来做状态标志。 + + **进程锁**：为了控制同一操作系统中多个进程访问某个共享资源，因为进程具有独立性，各个进程无法访问其他进程的资源，因此无法通过 synchronized 等线程锁实现进程锁。比如说，我们的同一个 linux 服务器，部署了好几个 Java 项目，有可能同时访问或操作服务器上的相同数据，这就需要进程锁，一般可以用『文件锁』来达到进程互斥。 + + **分布式锁**：随着用户越来越多，我们上了好多服务器，原本有个定时给客户发邮件的任务，如果不加以控制的话，到点后每台机器跑一次任务，客户就会收到 N 条邮件，这就需要通过分布式锁来互斥了。 + + > 书面解释：分布式锁是控制分布式系统或不同系统之间共同访问共享资源的一种锁实现，如果不同的系统或同一个系统的不同主机之间共享了某个资源时，往往需要互斥来防止彼此干扰来保证一致性。 + + + +知道了什么是分布式锁，接下来就到了技术选型环节 + +![](http://img.doutula.com/production/uploads/image/2018/01/03/20180103987632_tEBevG.jpg) + +## 二、分布式锁要怎么搞 + +要实现一个分布式锁，我们一般选择集群机器都可以操作的外部系统，然后各个机器都去这个外部系统申请锁。 + +这个外部系统一般需要满足如下要求才能胜任： + +1. **互斥**：在任意时刻，只能有一个客户端能持有锁。 +2. **防止死锁**：即使有一个客户端在持有锁的期间崩溃而没有主动解锁，也能保证后续其他客户端能加锁。所以锁一般要有一个过期时间。 +4. **独占性**：解铃还须系铃人，加锁和解锁必须是同一个客户端，一把锁只能有一把钥匙，客户端自己的锁不能被别人给解开，当然也不能去开别人的锁。 +4. **容错**：外部系统不能太“脆弱”，要保证外部系统的正常运行，客户端才可以加锁和解锁。 + + + +我觉得可以这么类比： + +好多商贩要租用某个仓库，同一时刻，只能给一个商贩租用，且只能有一把钥匙，还得有固定的“租期”，到期后要回收的，当然最重要的是仓库门不能坏了，要不锁都锁不住。这不就是分布式锁吗？ + +> 感慨自己真是个爱技术爱生活的程序猿~~ +> +> 其实锁，本质上就是用来进行防重操作的（数据一致性），像查询这种幂等操作，就不需要费这劲 + + + +直接上结论： + +分布式锁一般有三种实现方式：**1. 数据库乐观锁；2. 基于 Redis 的分布式锁；3. 基于 ZooKeeper 的分布式锁。** + +但为了追求更好的性能，我们通常会选择使用 Redis 或 Zookeeper 来做。 + +> 想必也有喜欢问为什么的同学，那数据库客观锁怎么就性能不好了? +> +> 使用数据库乐观锁，包括主键防重，版本号控制。但是这两种方法各有利弊。 +> +> - 使用主键冲突的策略进行防重，在并发量非常高的情况下对数据库性能会有影响，尤其是应用数据表和主键冲突表在一个库的时候，表现更加明显。还有就是在 MySQL 数据库中采用主键冲突防重，在大并发情况下有可能会造成锁表现象，比较好的办法是在程序中生产主键进行防重。 +> +> - 使用版本号策略 +> +> 这个策略源于 MySQL 的 MVCC 机制，使用这个策略其实本身没有什么问题，唯一的问题就是对数据表侵入较大，我们要为每个表设计一个版本号字段，然后写一条判断 SQL 每次进行判断。 + + + +第三趴，编码 + +## 三、基于 Redis 的分布式锁 + +> 其实 Redis 官网已经给出了实现：https://redis.io/topics/distlock，说各种书籍和博客用了各种手段去用 Redis 实现分布式锁，建议用 Redlock 实现，这样更规范、更安全。我们循序渐进来看 + +我们默认指定大家用的是 Redis 2.6.12 及更高的版本，就不再去讲 `setnx`、`expire` 这种了，直接 `set` 命令加锁 + +```sh +set key value[expiration EX seconds|PX milliseconds] [NX|XX] +``` + +eg: + +```bash +SET resource_name my_random_value NX PX 30000 +``` + +> *SET* 命令的行为可以通过一系列参数来修改 +> +> - `EX second` ：设置键的过期时间为 `second` 秒。 `SET key value EX second` 效果等同于 `SETEX key second value` 。 +> - `PX millisecond` ：设置键的过期时间为 `millisecond` 毫秒。 `SET key value PX millisecond` 效果等同于 `PSETEX key millisecond value` 。 +> - `NX` ：只在键不存在时，才对键进行设置操作。 `SET key value NX` 效果等同于 `SETNX key value` 。 +> - `XX` ：只在键已经存在时，才对键进行设置操作。 + +这条指令的意思：当 key——resource_name 不存在时创建这样的 key，设值为 my_random_value，并设置过期时间 30000 毫秒。 + +别看这干了两件事，因为 Redis 是单线程的，这一条指令不会被打断，所以是原子性的操作。 + + + +Redis 实现分布式锁的主要步骤： + +1. 指定一个 key 作为锁标记，存入 Redis 中，指定一个 **唯一的标识** 作为 value。 +2. 当 key 不存在时才能设置值，确保同一时间只有一个客户端进程获得锁，满足 **互斥性** 特性。 +3. 设置一个过期时间，防止因系统异常导致没能删除这个 key，满足 **防死锁** 特性。 +4. 当处理完业务之后需要清除这个 key 来释放锁，清除 key 时需要校验 value 值，需要满足 **解铃还须系铃人** 。 + +设置一个随机值的意思是在解锁时候判断 key 的值和我们存储的随机数是不是一样，一样的话，才是自己的锁，直接 `del` 解锁就行。 + +当然这个两个操作要保证原子性，所以 Redis 给出了一段 lua 脚本（Redis 服务器会单线程原子性执行 lua 脚本，保证 lua 脚本在处理的过程中不会被任意其它请求打断。）： + +```lua +if redis.call("get",KEYS[1]) == ARGV[1] then + return redis.call("del",KEYS[1]) +else + return 0 +end +``` + + + +### 问题： + +我们先抛出两个问题思考： + +1. 获取锁时，过期时间要设置多少合适呢？ + + 预估一个合适的时间，其实没那么容易，比如操作资源的时间最慢可能要 10s，而我们只设置了 5s 就过期，那就存在锁提前过期的风险。这个问题先记下，我们一会看下 Javaer 要怎么在代码中用 Redis 锁。 + +2. 容错性如何保证呢？ + + Redis 挂了怎么办，你可能会说上主从、上集群，但也会出现这样的极端情况，当我们上锁后，主节点就挂了，这个时候还没来的急同步到从节点，主从切换后锁还是丢了 + +带着这两个问题，我们接着看 + + + +### Redisson 实现代码 + +Redisson 是 Redis 官方的分布式锁组件。GitHub 地址：[https://github.com/redisson/redisson](https://zhuanlan.zhihu.com/write) + +> Redisson 是一个在 Redis 的基础上实现的 Java 驻内存数据网格（In-Memory Data Grid）。它不仅提供了一系列的分布式的 Java 常用对象，还实现了可重入锁（Reentrant Lock）、公平锁（Fair Lock、联锁（MultiLock）、 红锁（RedLock）、 读写锁（ReadWriteLock）等，还提供了许多分布式服务。Redisson 提供了使用 Redis 的最简单和最便捷的方法。Redisson 的宗旨是促进使用者对 Redis 的关注分离（Separation of Concern），从而让使用者能够将精力更集中地放在处理业务逻辑上。 + +Redisson 现在已经很强大了，github 的 wiki 也很详细，分布式锁的介绍直接戳 [Distributed locks and synchronizers](https://github.com/redisson/redisson/wiki/8.-Distributed-locks-and-synchronizers) + +Redisson 支持单点模式、主从模式、哨兵模式、集群模式，只是配置的不同，我们以单点模式来看下怎么使用，代码很简单，都已经为我们封装好了，直接拿来用就好，详细的 demo，我放在了 github: starfish-learn-redisson 上，这里就不一步步来了 + +```java +RLock lock = redisson.getLock("myLock"); +``` + +RLock 提供了各种锁方法，我们来解读下这个接口方法， + +> 注：代码为 3.16.2 版本，可以看到继承自 JDK 的 Lock 接口，和 Reddsion 的异步锁接口 RLockAsync(这个我们先不研究) + +#### RLock + +![](https://img.starfish.ink/redis/RLock.png) + +```java +public interface RLock extends Lock, RLockAsync { + + /** + * 获取锁的名字 + */ + String getName(); + + /** + * 这个叫终端锁操作，表示该锁可以被中断 假如A和B同时调这个方法，A获取锁，B为获取锁，那么B线程可以通过 + * Thread.currentThread().interrupt(); 方法真正中断该线程 + */ + void lockInterruptibly(long leaseTime, TimeUnit unit) throws InterruptedException; + + /** + * 这个应该是最常用的，尝试获取锁 + * waitTimeout 尝试获取锁的最大等待时间，超过这个值，则认为获取锁失败 + * leaseTime 锁的持有时间,超过这个时间锁会自动失效（值应设置为大于业务处理的时间，确保在锁有效期内业务能处理完） + */ + boolean tryLock(long waitTime, long leaseTime, TimeUnit unit) throws InterruptedException; + + /** + * 锁的有效期设置为 leaseTime，过期后自动失效 + * 如果 leaseTime 设置为 -1, 表示不主动过期 + */ + void lock(long leaseTime, TimeUnit unit); + + /** + * Unlocks the lock independently of its state + */ + boolean forceUnlock(); + + /** + * 检查是否被另一个线程锁住 + */ + boolean isLocked(); + + /** + * 检查当前线线程是否持有该锁 + */ + boolean isHeldByCurrentThread(); + + /** + * 这个就明了了，检查指定线程是否持有锁 + */ + boolean isHeldByThread(long threadId); + + /** + * 返回当前线程持有锁的次数 + */ + int getHoldCount(); + + /** + * 返回锁的剩余时间 + * @return time in milliseconds + * -2 if the lock does not exist. + * -1 if the lock exists but has no associated expire. + */ + long remainTimeToLive(); + +} +``` + +#### Demo + +```java +Config config = new Config(); +config.useSingleServer().setAddress("redis://127.0.0.1:6379").setPassword("").setDatabase(1); +RedissonClient redissonClient = Redisson.create(config); +RLock disLock = redissonClient.getLock("mylock"); +boolean isLock; +try { + /** + * 尝试获取锁的最大等待时间是 100 秒，超过这个值还没获取到，就认为获取失败 + * 锁的持有时间是 10 秒 + */ + isLock = disLock.tryLock(100, 10, TimeUnit.MILLISECONDS); + if (isLock) { + //做自己的业务 + Thread.sleep(10000); + } +} catch (Exception e) { + e.printStackTrace(); +} finally { + disLock.unlock(); +} +``` + +就是这么简单，Redisson 已经做好了封装，使用起来 so easy，如果使用主从、哨兵、集群这种也只是配置不同。 + +#### 原理 + +> 看源码小 tips，最好是 fork 到自己的仓库，然后拉到本地，边看边注释，然后提交到自己的仓库，也方便之后再看，不想这么麻烦的，也可以直接看我的 Jstarfish/redisson + +先看下 RLock 的类关系 + +![](https://img.starfish.ink/redis/RLock-UML.png) + +跟着源码，可以发现 RedissonLock 是 RLock 的直接实现，也是我们加锁、解锁操作的核心类 + +##### 加锁 + +主要的加锁方法就下边这两个，区别也很简单，一个有等待时间，一个没有，所以我们挑个复杂的看（源码包含了另一个的绝大部分） + +```java +boolean tryLock(long waitTime, long leaseTime, TimeUnit unit) throws InterruptedException; +void lock(long leaseTime, TimeUnit unit); +``` + +**RedissonLock.tryLock** + +```java +@Override +public boolean tryLock(long waitTime, long leaseTime, TimeUnit unit) throws InterruptedException { + // 获取等锁的最长时间 + long time = unit.toMillis(waitTime); + long current = System.currentTimeMillis(); + //取得当前线程id（判断是否可重入锁的关键） + long threadId = Thread.currentThread().getId(); + // 【核心点1】尝试获取锁，若返回值为null，则表示已获取到锁，返回的ttl就是key的剩余存活时间 + Long ttl = tryAcquire(waitTime, leaseTime, unit, threadId); + if (ttl == null) { + return true; + } + // 还可以容忍的等待时长 = 获取锁能容忍的最大等待时长 - 执行完上述操作流程的时间 + time -= System.currentTimeMillis() - current; + if (time <= 0) { + //等不到了，直接返回失败 + acquireFailed(waitTime, unit, threadId); + return false; + } + + current = System.currentTimeMillis(); + /** + * 【核心点2】 + * 订阅解锁消息 redisson_lock__channel:{$KEY}，并通过await方法阻塞等待锁释放，解决了无效的锁申请浪费资源的问题： + * 基于信息量，当锁被其它资源占用时，当前线程通过 Redis 的 channel 订阅锁的释放事件，一旦锁释放会发消息通知待等待的线程进行竞争 + * 当 this.await返回false，说明等待时间已经超出获取锁最大等待时间，取消订阅并返回获取锁失败 + * 当 this.await返回true，进入循环尝试获取锁 + */ + RFuture subscribeFuture = subscribe(threadId); + //await 方法内部是用CountDownLatch来实现阻塞，获取subscribe异步执行的结果（应用了Netty 的 Future） + if (!subscribeFuture.await(time, TimeUnit.MILLISECONDS)) { + if (!subscribeFuture.cancel(false)) { + subscribeFuture.onComplete((res, e) -> { + if (e == null) { + unsubscribe(subscribeFuture, threadId); + } + }); + } + acquireFailed(waitTime, unit, threadId); + return false; + } + + // ttl 不为空，表示已经有这样的key了，只能阻塞等待 + try { + time -= System.currentTimeMillis() - current; + if (time <= 0) { + acquireFailed(waitTime, unit, threadId); + return false; + } + + // 来个死循环，继续尝试着获取锁 + while (true) { + long currentTime = System.currentTimeMillis(); + ttl = tryAcquire(waitTime, leaseTime, unit, threadId); + if (ttl == null) { + return true; + } + + time -= System.currentTimeMillis() - currentTime; + if (time <= 0) { + acquireFailed(waitTime, unit, threadId); + return false; + } + + currentTime = System.currentTimeMillis(); + + /** + * 【核心点3】根据锁TTL，调整阻塞等待时长； + * 1、latch其实是个信号量Semaphore，调用其tryAcquire方法会让当前线程阻塞一段时间，避免在while循环中频繁请求获锁； + * 当其他线程释放了占用的锁，会广播解锁消息，监听器接收解锁消息，并释放信号量，最终会唤醒阻塞在这里的线程 + * 2、该Semaphore的release方法，会在订阅解锁消息的监听器消息处理方法org.redisson.pubsub.LockPubSub#onMessage调用； + */ + //调用信号量的方法来阻塞线程，时长为锁等待时间和租期时间中较小的那个 + if (ttl >= 0 && ttl < time) { + subscribeFuture.getNow().getLatch().tryAcquire(ttl, TimeUnit.MILLISECONDS); + } else { + subscribeFuture.getNow().getLatch().tryAcquire(time, TimeUnit.MILLISECONDS); + } + + time -= System.currentTimeMillis() - currentTime; + if (time <= 0) { + acquireFailed(waitTime, unit, threadId); + return false; + } + } + } finally { + // 获取到锁或者抛出中断异常，退订redisson_lock__channel:{$KEY}，不再关注解锁事件 + unsubscribe(subscribeFuture, threadId); + } +} +``` + +接着看注释中提到的 3 个核心点 + +**核心点1-尝试加锁：RedissonLock.tryAcquireAsync** + +```java +private RFuture tryAcquireAsync(long waitTime, long leaseTime, TimeUnit unit, long threadId) { + RFuture ttlRemainingFuture; + // leaseTime != -1 说明没过期 + if (leaseTime != -1) { + // 实质是异步执行加锁Lua脚本 + ttlRemainingFuture = tryLockInnerAsync(waitTime, leaseTime, unit, threadId, RedisCommands.EVAL_LONG); + } else { + // 否则，已经过期了,传参变为新的时间（续期后） + ttlRemainingFuture = tryLockInnerAsync(waitTime, internalLockLeaseTime, + TimeUnit.MILLISECONDS, threadId, RedisCommands.EVAL_LONG); + } + ttlRemainingFuture.onComplete((ttlRemaining, e) -> { + if (e != null) { + return; + } + + // lock acquired + if (ttlRemaining == null) { + if (leaseTime != -1) { + internalLockLeaseTime = unit.toMillis(leaseTime); + } else { + // 续期 + scheduleExpirationRenewal(threadId); + } + } + }); + return ttlRemainingFuture; +} +``` + +**异步执行加锁 Lua 脚本：RedissonLock.tryLockInnerAsync** + +```java + RFuture tryLockInnerAsync(long waitTime, long leaseTime, TimeUnit unit, long threadId, RedisStrictCommand command) { + return evalWriteAsync(getRawName(), LongCodec.INSTANCE, command, + // 1.如果缓存中的key不存在，则执行 hincrby 命令(hincrby key UUID+threadId 1), 设值重入次数1 + // 然后通过 pexpire 命令设置锁的过期时间(即锁的租约时间) + // 返回空值 nil ，表示获取锁成功 + "if (redis.call('exists', KEYS[1]) == 0) then " + + "redis.call('hincrby', KEYS[1], ARGV[2], 1); " + + "redis.call('pexpire', KEYS[1], ARGV[1]); " + + "return nil; " + + "end; " + + // 如果key已经存在，并且value也匹配，表示是当前线程持有的锁，则执行 hincrby 命令，重入次数加1，并且设置失效时间 + "if (redis.call('hexists', KEYS[1], ARGV[2]) == 1) then " + + "redis.call('hincrby', KEYS[1], ARGV[2], 1); " + + "redis.call('pexpire', KEYS[1], ARGV[1]); " + + "return nil; " + + "end; " + + //如果key已经存在，但是value不匹配，说明锁已经被其他线程持有，通过 pttl 命令获取锁的剩余存活时间并返回，至此获取锁失败 + "return redis.call('pttl', KEYS[1]);", + Collections.singletonList(getRawName()), unit.toMillis(leaseTime), getLockName(threadId)); +} +``` + +> - KEYS[1] 就是 Collections.singletonList(getName())，表示分布式锁的key； +> - ARGV[1] 就是internalLockLeaseTime，即锁的租约时间（持有锁的有效时间），默认30s； +> - ARGV[2] 就是getLockName(threadId)，是获取锁时set的唯一值 value，即UUID+threadId + +**看门狗续期：RedissonBaseLock.scheduleExpirationRenewal** + +```java +// 基于线程ID定时调度和续期 +protected void scheduleExpirationRenewal(long threadId) { + // 新建一个ExpirationEntry记录线程重入计数 + ExpirationEntry entry = new ExpirationEntry(); + ExpirationEntry oldEntry = EXPIRATION_RENEWAL_MAP.putIfAbsent(getEntryName(), entry); + if (oldEntry != null) { + // 当前进行的当前线程重入加锁 + oldEntry.addThreadId(threadId); + } else { + // 当前进行的当前线程首次加锁 + entry.addThreadId(threadId); + // 首次新建ExpirationEntry需要触发续期方法，记录续期的任务句柄 + renewExpiration(); + } +} + +// 处理续期 +private void renewExpiration() { + // 根据entryName获取ExpirationEntry实例，如果为空，说明在cancelExpirationRenewal()方法已经被移除，一般是解锁的时候触发 + ExpirationEntry ee = EXPIRATION_RENEWAL_MAP.get(getEntryName()); + if (ee == null) { + return; + } + + // 新建一个定时任务，这个就是看门狗的实现，io.netty.util.Timeout是Netty结合时间轮使用的定时任务实例 + Timeout task = commandExecutor.getConnectionManager().newTimeout(new TimerTask() { + @Override + public void run(Timeout timeout) throws Exception { + // 这里是重复外面的那个逻辑， + ExpirationEntry ent = EXPIRATION_RENEWAL_MAP.get(getEntryName()); + if (ent == null) { + return; + } + // 获取ExpirationEntry中首个线程ID，如果为空说明调用过cancelExpirationRenewal()方法清空持有的线程重入计数，一般是锁已经释放的场景 + Long threadId = ent.getFirstThreadId(); + if (threadId == null) { + return; + } + // 向Redis异步发送续期的命令 + RFuture future = renewExpirationAsync(threadId); + future.onComplete((res, e) -> { + // 抛出异常，续期失败，只打印日志和直接终止任务 + if (e != null) { + log.error("Can't update lock " + getRawName() + " expiration", e); + EXPIRATION_RENEWAL_MAP.remove(getEntryName()); + return; + } + // 返回true证明续期成功，则递归调用续期方法（重新调度自己），续期失败说明对应的锁已经不存在，直接返回，不再递归 + if (res) { + // reschedule itself + renewExpiration(); + } else { + cancelExpirationRenewal(null); + } + }); + }// 这里的执行频率为leaseTime转换为ms单位下的三分之一，由于leaseTime初始值为-1的情况下才会进入续期逻辑，那么这里的执行频率为lockWatchdogTimeout的三分之一 + }, internalLockLeaseTime / 3, TimeUnit.MILLISECONDS); + // ExpirationEntry实例持有调度任务实例 + ee.setTimeout(task); +} +``` + + + +**核心点2-订阅解锁消息：RedissonLock.subscribe** + +```java +protected final LockPubSub pubSub; + +public RedissonLock(CommandAsyncExecutor commandExecutor, String name) { + super(commandExecutor, name); + this.commandExecutor = commandExecutor; + this.internalLockLeaseTime = commandExecutor.getConnectionManager().getCfg().getLockWatchdogTimeout(); + //在构造器中初始化pubSub,跟着这几个get方法会发现他们都是在构造器中初始化的，在PublishSubscribeService中会有 + // private final AsyncSemaphore[] locks = new AsyncSemaphore[50]; 这样一段代码，初始化了一组信号量 + this.pubSub = commandExecutor.getConnectionManager().getSubscribeService().getLockPubSub(); +} + +protected RFuture subscribe(long threadId) { + return pubSub.subscribe(getEntryName(), getChannelName()); +} + +// 在LockPubSub中注册一个entryName -> RedissonLockEntry的哈希映射，RedissonLockEntry实例中存放着RPromise结果，一个信号量形式的锁和订阅方法重入计数器 +public RFuture subscribe(String entryName, String channelName) { + AsyncSemaphore semaphore = service.getSemaphore(new ChannelName(channelName)); + RPromise newPromise = new RedissonPromise<>(); + semaphore.acquire(() -> { + if (!newPromise.setUncancellable()) { + semaphore.release(); + return; + } + + E entry = entries.get(entryName); + if (entry != null) { + entry.acquire(); + semaphore.release(); + entry.getPromise().onComplete(new TransferListener(newPromise)); + return; + } + + E value = createEntry(newPromise); + value.acquire(); + + E oldValue = entries.putIfAbsent(entryName, value); + if (oldValue != null) { + oldValue.acquire(); + semaphore.release(); + oldValue.getPromise().onComplete(new TransferListener(newPromise)); + return; + } + + RedisPubSubListener listener = createListener(channelName, value); + service.subscribe(LongCodec.INSTANCE, channelName, semaphore, listener); + }); + + return newPromise; +} +``` + +核心点 3 比较简单，就不说了 + + + +##### 解锁 + +**RedissonLock.unlock()** + +```java +@Override +public void unlock() { + try { + // 获取当前调用解锁操作的线程ID + get(unlockAsync(Thread.currentThread().getId())); + } catch (RedisException e) { + // IllegalMonitorStateException一般是A线程加锁，B线程解锁，内部判断线程状态不一致抛出的 + if (e.getCause() instanceof IllegalMonitorStateException) { + throw (IllegalMonitorStateException) e.getCause(); + } else { + throw e; + } + } +} +``` + +**RedissonBaseLock.unlockAsync** + +```java +@Override +public RFuture unlockAsync(long threadId) { + // 构建一个结果RedissonPromise + RPromise result = new RedissonPromise<>(); + // 返回的RFuture如果持有的结果为true，说明解锁成功，返回NULL说明线程ID异常，加锁和解锁的客户端线程不是同一个线程 + RFuture future = unlockInnerAsync(threadId); + + future.onComplete((opStatus, e) -> { + // 取消看门狗的续期任务 + cancelExpirationRenewal(threadId); + + if (e != null) { + result.tryFailure(e); + return; + } + + if (opStatus == null) { + IllegalMonitorStateException cause = new IllegalMonitorStateException("attempt to unlock lock, not locked by current thread by node id: " + + id + " thread-id: " + threadId); + result.tryFailure(cause); + return; + } + + result.trySuccess(null); + }); + + return result; +} +``` + +**RedissonLock.unlockInnerAsync** + +```java +// 真正的内部解锁的方法，执行解锁的Lua脚本 +protected RFuture unlockInnerAsync(long threadId) { + return evalWriteAsync(getRawName(), LongCodec.INSTANCE, RedisCommands.EVAL_BOOLEAN, + //如果分布式锁存在，但是value不匹配，表示锁已经被其他线程占用，无权释放锁，那么直接返回空值（解铃还须系铃人） + "if (redis.call('hexists', KEYS[1], ARGV[3]) == 0) then " + + "return nil;" + + "end; " + + //如果value匹配，则就是当前线程占有分布式锁，那么将重入次数减1 + "local counter = redis.call('hincrby', KEYS[1], ARGV[3], -1); " + + //重入次数减1后的值如果大于0，表示分布式锁有重入过，那么只能更新失效时间，还不能删除 + "if (counter > 0) then " + + "redis.call('pexpire', KEYS[1], ARGV[2]); " + + "return 0; " + + "else " + + //重入次数减1后的值如果为0，这时就可以删除这个KEY，并发布解锁消息，返回1 + "redis.call('del', KEYS[1]); " + + "redis.call('publish', KEYS[2], ARGV[1]); " + + "return 1; " + + "end; " + + "return nil;", + //这5个参数分别对应KEYS[1]，KEYS[2]，ARGV[1]，ARGV[2]和ARGV[3] + Arrays.asList(getRawName(), getChannelName()), LockPubSub.UNLOCK_MESSAGE, internalLockLeaseTime, getLockName(threadId)); +} +``` + + + +我只列出了一小部分代码，更多的内容还是得自己动手 + +从源码中，我们可以看到 Redisson 帮我们解决了抛出的第一个问题：失效时间设置多长时间为好？ + +Redisson 提供了看门狗，每获得一个锁时，只设置一个很短的超时时间，同时起一个线程在每次快要到超时时间时去刷新锁的超时时间。在释放锁的同时结束这个线程。 + +但是没有解决节点挂掉，丢失锁的问题，接着来~ + + + +![](https://i01piccdn.sogoucdn.com/5c535b46a06ec4d8) + +## 四、RedLock + +我们上边介绍的分布式锁，在某些极端情况下仍然是有缺陷的 + +1. 客户端长时间内阻塞导致锁失效 + + 客户端 1 得到了锁，因为网络问题或者 GC 等原因导致长时间阻塞，然后业务程序还没执行完锁就过期了，这时候客户端 2 也能正常拿到锁，可能会导致线程安全的问题。 + +2. Redis 服务器时钟漂移 + + 如果 Redis 服务器的机器时间发生了向前跳跃，就会导致这个 key 过早超时失效，比如说客户端 1 拿到锁后，key 还没有到过期时间，但是 Redis 服务器的时间比客户端快了 2 分钟，导致 key 提前就失效了，这时候，如果客户端 1 还没有释放锁的话，就可能导致多个客户端同时持有同一把锁的问题。 + +3. 单点实例安全问题 + + 如果 Redis 是单机模式的，如果挂了的话，那所有的客户端都获取不到锁了，假设你是主从模式，但 Redis 的主从同步是异步进行的，如果 Redis 主宕机了，这个时候从机并没有同步到这一把锁，那么机器 B 再次申请的时候就会再次申请到这把锁，这也是问题 + +为了解决这些个问题 Redis 作者提出了 RedLock 红锁的算法，在 Redission 中也对 RedLock 进行了实现。 + +> Redis 官网对 redLock 算法的介绍大致如下：[The Redlock algorithm](https://redis.io/topics/distlock) +> +> 在分布式版本的算法里我们假设我们有 **N 个 Redis master** 节点，这些节点都是完全独立的，我们不用任何复制或者其他隐含的分布式协调机制。之前我们已经描述了在 Redis 单实例下怎么安全地获取和释放锁。我们确保将在每（N) 个实例上使用此方法获取和释放锁。在我们的例子里面我们设置 N=5，这是一个比较合理的设置，所以我们需要在 5 台机器或者虚拟机上面运行这些实例，这样保证他们不会同时都宕掉。为了取到锁，客户端应该执行以下操作: +> +> 1. 获取当前 Unix 时间，以毫秒为单位。 +> 2. 依次尝试从 5 个实例，使用相同的 key 和具有唯一性的 value（例如UUID）获取锁。当向 Redis 请求获取锁时，客户端应该设置一个尝试从某个 Reids 实例获取锁的最大等待时间（超过这个时间，则立马询问下一个实例），这个超时时间应该小于锁的失效时间。例如你的锁自动失效时间为 10 秒，则超时时间应该在 5-50 毫秒之间。这样可以避免服务器端 Redis 已经挂掉的情况下，客户端还在死死地等待响应结果。如果服务器端没有在规定时间内响应，客户端应该尽快尝试去另外一个 Redis 实例请求获取锁。 +> 3. 客户端使用当前时间减去开始获取锁时间（步骤1记录的时间）就得到获取锁消耗的时间。当且仅当从大多数（N/2+1，这里是3个节点）的 Redis 节点都取到锁，并且使用的总耗时小于锁失效时间时，锁才算获取成功。 +> 4. 如果取到了锁，key 的真正有效时间 = 有效时间（获取锁时设置的 key 的自动超时时间） - 获取锁的总耗时（询问各个 Redis 实例的总耗时之和）（步骤 3 计算的结果）。 +> 5. 如果因为某些原因，最终获取锁失败（即没有在至少 “N/2+1 ”个 Redis 实例取到锁或者“获取锁的总耗时”超过了“有效时间”），客户端应该在所有的 Redis 实例上进行解锁（即便某些 Redis 实例根本就没有加锁成功，这样可以防止某些节点获取到锁但是客户端没有得到响应而导致接下来的一段时间不能被重新获取锁）。 + +总结下就是： + +1. 客户端在多个 Redis 实例上申请加锁，必须保证大多数节点加锁成功 + + 解决容错性问题，部分实例异常，剩下的还能加锁成功 + +2. 大多数节点加锁的总耗时，要小于锁设置的过期时间 + + 多实例操作，可能存在网络延迟、丢包、超时等问题，所以就算是大多数节点加锁成功，如果加锁的累积耗时超过了锁的过期时间，那有些节点上的锁可能也已经失效了，还是没有意义的 + +3. 释放锁，要向全部节点发起释放锁请求 + + 如果部分节点加锁成功，但最后由于异常导致大部分节点没加锁成功，就要释放掉所有的，各节点要保持一致 + +> 关于 RedLock，两位分布式大佬，Antirez 和 Martin 还进行过一场争论，感兴趣的也可以看看 + +```java +Config config1 = new Config(); +config1.useSingleServer().setAddress("127.0.0.1:6379"); +RedissonClient redissonClient1 = Redisson.create(config1); + +Config config2 = new Config(); +config2.useSingleServer().setAddress("127.0.0.1:5378"); +RedissonClient redissonClient2 = Redisson.create(config2); + +Config config3 = new Config(); +config3.useSingleServer().setAddress("127.0.0.1:5379"); +RedissonClient redissonClient3 = Redisson.create(config3); + +/** + * 获取多个 RLock 对象 + */ +RLock lock1 = redissonClient1.getLock(lockKey); +RLock lock2 = redissonClient2.getLock(lockKey); +RLock lock3 = redissonClient3.getLock(lockKey); + +/** + * 根据多个 RLock 对象构建 RedissonRedLock （最核心的差别就在这里） + */ +RedissonRedLock redLock = new RedissonRedLock(lock1, lock2, lock3); + +try { + /** + * 4.尝试获取锁 + * waitTimeout 尝试获取锁的最大等待时间，超过这个值，则认为获取锁失败 + * leaseTime 锁的持有时间,超过这个时间锁会自动失效（值应设置为大于业务处理的时间，确保在锁有效期内业务能处理完） + */ + boolean res = redLock.tryLock(100, 10, TimeUnit.SECONDS); + if (res) { + //成功获得锁，在这里处理业务 + } +} catch (Exception e) { + throw new RuntimeException("aquire lock fail"); +}finally{ + //无论如何, 最后都要解锁 + redLock.unlock(); +} +``` + +最核心的变化就是需要构建多个 RLock ，然后根据多个 RLock 构建成一个 RedissonRedLock，因为 redLock 算法是建立在多个互相独立的 Redis 环境之上的（为了区分可以叫为 Redission node），Redission node 节点既可以是单机模式(single)，也可以是主从模式(master/salve)，哨兵模式(sentinal)，或者集群模式(cluster)。这就意味着，不能跟以往这样只搭建 1个 cluster、或 1个 sentinel 集群，或是1套主从架构就了事了，需要为 RedissonRedLock 额外搭建多几套独立的 Redission 节点。 + + **RedissonMultiLock.tryLock** + +```java +@Override +public boolean tryLock(long waitTime, long leaseTime, TimeUnit unit) throws InterruptedException { + // try { + // return tryLockAsync(waitTime, leaseTime, unit).get(); + // } catch (ExecutionException e) { + // throw new IllegalStateException(e); + // } + long newLeaseTime = -1; + if (leaseTime != -1) { + if (waitTime == -1) { + newLeaseTime = unit.toMillis(leaseTime); + } else { + newLeaseTime = unit.toMillis(waitTime)*2; + } + } + + long time = System.currentTimeMillis(); + long remainTime = -1; + if (waitTime != -1) { + remainTime = unit.toMillis(waitTime); + } + long lockWaitTime = calcLockWaitTime(remainTime); + + //允许加锁失败节点个数限制（N-(N/2+1)） + int failedLocksLimit = failedLocksLimit(); + List acquiredLocks = new ArrayList<>(locks.size()); + // 遍历所有节点通过EVAL命令执行lua加锁 + for (ListIterator iterator = locks.listIterator(); iterator.hasNext();) { + RLock lock = iterator.next(); + boolean lockAcquired; + try { + // 对节点尝试加锁 + if (waitTime == -1 && leaseTime == -1) { + lockAcquired = lock.tryLock(); + } else { + long awaitTime = Math.min(lockWaitTime, remainTime); + lockAcquired = lock.tryLock(awaitTime, newLeaseTime, TimeUnit.MILLISECONDS); + } + } catch (RedisResponseTimeoutException e) { + // 如果抛出这类异常，为了防止加锁成功，但是响应失败，需要解锁所有节点 + unlockInner(Arrays.asList(lock)); + lockAcquired = false; + } catch (Exception e) { + lockAcquired = false; + } + + if (lockAcquired) { + acquiredLocks.add(lock); + } else { + /* + * 计算已经申请锁失败的节点是否已经到达 允许加锁失败节点个数限制 （N-(N/2+1)） + * 如果已经到达， 就认定最终申请锁失败，则没有必要继续从后面的节点申请了 + * 因为 Redlock 算法要求至少N/2+1 个节点都加锁成功，才算最终的锁申请成功 + */ + if (locks.size() - acquiredLocks.size() == failedLocksLimit()) { + break; + } + + if (failedLocksLimit == 0) { + unlockInner(acquiredLocks); + if (waitTime == -1) { + return false; + } + failedLocksLimit = failedLocksLimit(); + acquiredLocks.clear(); + // reset iterator + while (iterator.hasPrevious()) { + iterator.previous(); + } + } else { + failedLocksLimit--; + } + } + //计算 目前从各个节点获取锁已经消耗的总时间，如果已经等于最大等待时间，则认定最终申请锁失败，返回false + if (remainTime != -1) { + remainTime -= System.currentTimeMillis() - time; + time = System.currentTimeMillis(); + if (remainTime <= 0) { + unlockInner(acquiredLocks); + return false; + } + } + } + + if (leaseTime != -1) { + acquiredLocks.stream() + .map(l -> (RedissonLock) l) + .map(l -> l.expireAsync(unit.toMillis(leaseTime), TimeUnit.MILLISECONDS)) + .forEach(f -> f.syncUninterruptibly()); + } + + return true; +} +``` + + + +##### 参考与感谢 + +- [《Redis —— Distributed locks with Redis》](https://redis.io/topics/distlock) +- [《Redisson —— Distributed locks and synchronizers》](https://github.com/redisson/redisson/wiki/8.-Distributed-locks-and-synchronizers) +- [慢谈 Redis 实现分布式锁 以及 Redisson 源码解析](https://crazyfzw.github.io/2019/08/24/distributed-locks-with-redis/) +- [理解Redisson中分布式锁的实现](https://www.cnblogs.com/throwable/p/14264804.html) + + + + + diff --git a/docs/data-management/Redis/Redis-MQ.md b/docs/data-management/Redis/Redis-MQ.md new file mode 100644 index 0000000000..64527748f9 --- /dev/null +++ b/docs/data-management/Redis/Redis-MQ.md @@ -0,0 +1,498 @@ +--- +title: Redis 消息队列的三种方案（List、Streams、Pub/Sub） +date: 2022-2-9 +tags: + - Redis +categories: Redis +--- + +![](https://img.starfish.ink/redis/redis-mq-banner.jpg) + +现如今的互联网应用大都是采用 **分布式系统架构** 设计的，所以 **消息队列** 已经逐渐成为企业应用系统 **内部通信** 的核心手段， + +它具有 **低耦合**、**可靠投递**、**广播**、**流量控制**、**最终一致性** 等一系列功能。 + +当前使用较多的 **消息队列** 有 `RabbitMQ`、`RocketMQ`、`ActiveMQ`、`Kafka`、`ZeroMQ`、`MetaMQ` 等，而部分**数据库** 如 `Redis`、`MySQL` 以及 `phxsql` ，如果硬搞的话，其实也可实现消息队列的功能。 + + + +可能有人觉得，各种开源的 MQ 已经足够使用了，为什么需要用 Redis 实现 MQ 呢？ + +- 有些简单的业务场景，可能不需要重量级的 MQ 组件（相比 Redis 来说，Kafka 和 RabbitMQ 都算是重量级的消息队列） + +那你有考虑过用 Redis 做消息队列吗？ + +这一章，我会结合消息队列的特点和 Redis 做消息队列的使用方式，以及实际项目中的使用，来和大家探讨下 Redis 消息队列的方案。 + + + +## 一、回顾消息队列 + +> **消息队列** 是指利用 **高效可靠** 的 **消息传递机制** 进行与平台无关的 **数据交流**，并基于**数据通信**来进行分布式系统的集成。 +> +> 通过提供 **消息传递** 和 **消息排队** 模型，它可以在 **分布式环境** 下提供 **应用解耦**、**弹性伸缩**、**冗余存储**、**流量削峰**、**异步通信**、**数据同步** 等等功能，其作为 **分布式系统架构** 中的一个重要组件，有着举足轻重的地位。 + +![](https://img.starfish.ink/redis/mq.jpg) + + + + + + + +现在回顾下，我们使用的消息队列，一般都有什么样的特点： + +- 三个角色：生产者、消费者、消息处理中心 +- 异步处理模式：**生产者** 将消息发送到一条 **虚拟的通道**（消息队列）上，而无须等待响应。**消费者** 则 **订阅** 或是 **监听** 该通道，取出消息。两者互不干扰，甚至都不需要同时在线，也就是我们说的 **松耦合** +- 可靠性：消息要可以保证不丢失、不重复消费、有时可能还需要顺序性的保证 + + + +撇开我们常用的消息中间件不说，你觉得 Redis 的哪些数据类型可以满足 MQ 的常规需求~~ + +## 二、Redis 实现消息队列 + +思来想去，只有 List 和 Streams 两种数据类型，可以实现消息队列的这些需求，当然，Redis 还提供了发布、订阅(pub/sub) 模式。 + +我们逐一看下这 3 种方式的使用和场景。 + +### 2.1 List 实现消息队列 + +Redis 列表是简单的字符串列表，按照插入顺序排序。你可以添加一个元素到列表的头部（左边）或者尾部（右边）。 + +所以常用来做**异步队列**使用。将需要延后处理的任务结构体序列化成字符串塞进 Redis 的列表，另一个线程从这个列表中轮询数据进行处理。 + +Redis 提供了好几对 List 指令，先大概看下这些命令，混个眼熟 + +#### List 常用命令 + +| 命令 | 用法 | 描述 | 示例 | +| ---------- | ------------------------------------- | ------------------------------------------------------------ | --------------------------------------------- | +| **LPUSH** | LPUSH key value [value ...] | 将一个或多个值 value 插入到列表 key 的表头如果有多个 value 值，那么各个 value 值按从左到右的顺序依次插入到表头 | 正着进反着出 | +| LPUSHX | LPUSHX key value | 将值 value 插入到列表 key 的表头，当且仅当 key 存在并且是一个列表。和 LPUSH 命令相反，当 key 不存在时， LPUSHX 命令什么也不做 | | +| **RPUSH** | RPUSH key value [value ...] | 将一个或多个值 value 插入到列表 key 的表尾(最右边) | 怎么进怎么出 | +| RPUSHX | RPUSHX key value | 将值 value 插入到列表 key 的表尾，当且仅当 key 存在并且是一个列表。和 RPUSH 命令相反，当 key 不存在时， RPUSHX 命令什么也不做。 | | +| **LPOP** | LPOP key | 移除并返回列表 key 的头元素。 | | +| **BLPOP** | BLPOP key [key ...] timeout | 移出并获取列表的第一个元素， 如果列表没有元素会阻塞列表直到等待超时或发现可弹出元素为止 | | +| **RPOP** | RPOP key | 移除并返回列表 key 的尾元素。 | | +| **BRPOP** | BRPOP key [key ...] timeout | 移出并获取列表的最后一个元素， 如果列表没有元素会阻塞列表直到等待超时或发现可弹出元素为止。 | | +| BRPOPLPUSH | BRPOPLPUSH source destination timeout | 从列表中弹出一个值，将弹出的元素插入到另外一个列表中并返回它； 如果列表没有元素会阻塞列表直到等待超时或发现可弹出元素为止。 | | +| RPOPLPUSH | RPOPLPUSH source destinationb | 命令 RPOPLPUSH 在一个原子时间内，执行以下两个动作：将列表 source 中的最后一个元素(尾元素)弹出，并返回给客户端。将 source 弹出的元素插入到列表 destination ，作为 destination 列表的的头元素 | RPOPLPUSH list01 list02 | +| **LSET** | LSET key index value | 将列表 key 下标为 index 的元素的值设置为 value | | +| **LLEN** | LLEN key | 返回列表 key 的长度。如果 key 不存在，则 key 被解释为一个空列表，返回 0 .如果 key 不是列表类型，返回一个错误 | | +| **LINDEX** | LINDEX key index | 返回列表 key 中，下标为 index 的元素。下标(index)参数 start 和 stop 都以 0 为底，也就是说，以 0 表示列表的第一个元素，以 1 表示列表的第二个元素，以此类推。 | | +| **LRANGE** | LRANGE key start stop | 返回列表 key 中指定区间内的元素，区间以偏移量 start 和 stop 指定 | | +| LREM | LREM key count value | 根据参数 count 的值，移除列表中与参数 value 相等的元素 | | +| LTRIM | LTRIM key start stop | 对一个列表进行修剪(trim)，就是说，让列表只保留指定区间内的元素，不在指定区间之内的元素都将被删除 | | +| LINSERT | LINSERT key BEFORE\|AFTER pivot value | 将值 value 插入到列表 key 当中，位于值 pivot 之前或之后。当 pivot 不存在于列表 key 时，不执行任何操作。当 key 不存在时， key 被视为空列表，不执行任何操作。如果 key 不是列表类型，返回一个错误。 | LINSERT list01 before c++ c#(在c++之前加上C#) | + +挑几个弹入、弹出的命令就可以组合出很多**姿势** + +- LPUSH、RPOP 左进右出 + +- RPUSH、LPOP 右进左出 + +```shell +127.0.0.1:6379> lpush mylist a a b c d e +(integer) 6 +127.0.0.1:6379> rpop mylist +"a" +127.0.0.1:6379> rpop mylist +"a" +127.0.0.1:6379> rpop mylist +"b" +127.0.0.1:6379> +``` + +![redis-RPOP](https://img.starfish.ink/redis/redis-rpop.jpg) + + + +#### 即时消费问题 + +通过 `LPUSH`，`RPOP` 这样的方式，会存在一个性能风险点，就是消费者如果想要及时的处理数据，就要在程序中写个类似 `while(true)` 这样的逻辑，不停的去调用 RPOP 或 LPOP 命令，这就会给消费者程序带来些不必要的性能损失。 + +所以，Redis 还提供了 `BLPOP`、`BRPOP` 这种阻塞式读取的命令（带 B-Bloking的都是阻塞式），客户端在没有读到队列数据时，自动阻塞，直到有新的数据写入队列，再开始读取新数据。这种方式就节省了不必要的 CPU 开销。 + +- LPUSH、BRPOP 左进右阻塞出 + +- RPUSH、BLPOP 右进左阻塞出 + +```shell +127.0.0.1:6379> lpush yourlist a b c d +(integer) 4 +127.0.0.1:6379> blpop yourlist 10 +1) "yourlist" +2) "d" +127.0.0.1:6379> blpop yourlist 10 +1) "yourlist" +2) "c" +127.0.0.1:6379> blpop yourlist 10 +1) "yourlist" +2) "b" +127.0.0.1:6379> blpop yourlist 10 +1) "yourlist" +2) "a" +127.0.0.1:6379> blpop yourlist 10 +(nil) +(10.02s) +``` + +**如果将超时时间设置为 0 时，即可无限等待，直到弹出消息** + + + +因为 Redis 单线程的特点，所以在消费数据时，同一个消息不会同时被多个 `consumer` 消费掉，但是需要我们考虑消费不成功的情况。 + +#### 可靠队列模式 | ack 机制 + +以上方式中， List 队列中的消息一经发送出去，便从队列里删除。如果由于网络原因消费者没有收到消息，或者消费者在处理这条消息的过程中崩溃了，就再也无法还原出这条消息。究其原因，就是缺少消息确认机制。 + +为了保证消息的可靠性，消息队列都会有完善的消息确认机制（Acknowledge），即消费者向队列报告消息已收到或已处理的机制。 + +**Redis List 怎么搞一搞呢？** + +再看上边的表格中，有两个命令， `RPOPLPUSH`、`BRPOPLPUSH` （阻塞）从一个 list 中获取消息的同时把这条消息复制到另一个 list 里（可以当做备份），而且这个过程是原子的。 + +这样我们就可以在业务流程安全结束后，再删除队列元素，实现消息确认机制。 + +```c +127.0.0.1:6379> rpush myqueue one +(integer) 1 +127.0.0.1:6379> rpush myqueue two +(integer) 2 +127.0.0.1:6379> rpush myqueue three +(integer) 3 +127.0.0.1:6379> rpoplpush myqueue queuebak +"three" +127.0.0.1:6379> lrange myqueue 0 -1 +1) "one" +2) "two" +127.0.0.1:6379> lrange queuebak 0 -1 +1) "three" +``` + +![redis-rpoplpush](https://img.starfish.ink/redis/redis-rpoplpush.jpg) + + + +之前做过的项目中就有用到这样的方式去处理数据，数据标识从一个 List 取出后放入另一个 List，业务操作安全执行完成后，再去删除 List 中的数据，如果有问题的话，很好回滚。 + + + +当然，还有更特殊的场景，可以通过 **zset** 来实现**延时消息队列**，原理就是将消息加到 zset 结构后，将要被消费的时间戳设置为对应的 score 即可，只要业务数据不会是重复数据就 OK。 + + + +### 2.2 订阅与发布实现消息队列 + +我们都知道消息模型有两种 + +- 点对点： Point-to-Point(P2P) +- 发布订阅：Publish/Subscribe(Pub/Sub) + +List 实现方式其实就是点对点的模式，下边我们再看下 Redis 的发布订阅模式（消息多播），这才是“根正苗红”的 Redis MQ + +![redis-pub_sub](https://img.starfish.ink/redis/redis-pub_sub.jpg) + +"发布/订阅"模式同样可以实现进程间的消息传递，其原理如下: + +"发布/订阅"模式包含两种角色，分别是发布者和订阅者。订阅者可以订阅一个或者多个频道(channel)，而发布者可以向指定的频道(channel)发送消息，所有订阅此频道的订阅者都会收到此消息。 + +Redis 通过 `PUBLISH` 、 `SUBSCRIBE` 等命令实现了订阅与发布模式， 这个功能提供两种信息机制， 分别是订阅/发布到频道和订阅/发布到模式。 + +这个 **频道** 和 **模式** 有什么区别呢？ + +频道我们可以先理解为是个 Redis 的 key 值，而模式，可以理解为是一个类似正则匹配的 Key，只是个可以匹配给定模式的频道。这样就不需要显式的去订阅多个名称了，可以通过模式订阅这种方式，一次性关注多个频道。 + +我们启动三个 Redis 客户端看下效果： + +![redis-subscribe](https://img.starfish.ink/redis/redis-subscribe.jpg) + +先启动两个客户端订阅（subscribe） 名字叫 framework 的频道，然后第三个客户端往 framework 发消息，可以看到前两个客户端都会接收到对应的消息： + +![redis-publish](https://img.starfish.ink/redis/redis-publish.jpg) + +我们可以看到订阅的客户端每次可以收到一个 3 个参数的消息，分别为： + +- 消息的种类 +- 始发频道的名称 +- 实际的消息 + +再来看下订阅符合给定**模式**的频道，这回订阅的命令是 `PSUBSCRIBE` + +![redis-psubscribe](https://img.starfish.ink/redis/redis-psubscribe.jpg) + +我们往 `java.framework` 这个频道发送了一条消息，不止订阅了该频道的 Consumer1 和 Consumer2 可以接收到消息，订阅了模式 `java.*` 的 Consumer3 和 Consumer4 也可以接收到消息。 + +![redis-psubscribe1](https://img.starfish.ink/redis/redis-psubscribe-demo.jpg) + +#### Pub/Sub 常用命令： + +| 命令 | 用法 | 描述 | +| ------------ | ------------------------------------------- | -------------------------------- | +| PSUBSCRIBE | PSUBSCRIBE pattern [pattern ...] | 订阅一个或多个符合给定模式的频道 | +| PUBSUB | PUBSUB subcommand [argument [argument ...]] | 查看订阅与发布系统状态 | +| PUBLISH | PUBLISH channel message | 将信息发送到指定的频道 | +| PUNSUBSCRIBE | PUNSUBSCRIBE [pattern [pattern ...]] | 退订所有给定模式的频道 | +| SUBSCRIBE | SUBSCRIBE channel [channel ...] | 订阅给定的一个或多个频道的信息 | +| UNSUBSCRIBE | UNSUBSCRIBE [channel [channel ...]] | 指退订给定的频道 | + + + +### 2.3 Streams 实现消息队列 + +Redis 发布订阅 (pub/sub) 有个缺点就是消息无法持久化，如果出现网络断开、Redis 宕机等，消息就会被丢弃。而且也没有 Ack 机制来保证数据的可靠性，假设一个消费者都没有，那消息就直接被丢弃了。 + +> 后来 Redis 的父亲 Antirez，又单独开启了一个叫 Disque 的项目来完善这些问题，但是没有做起来，github 的更新也定格在了 5 年前，所以我们就不讨论了。 + +Redis 5.0 版本新增了一个更强大的数据结构——**Stream**。它提供了消息的持久化和主备复制功能，可以让任何客户端访问任何时刻的数据，并且能记住每一个客户端的访问位置，还能保证消息不丢失。 + +它就像是个仅追加内容的**消息链表**，把所有加入的消息都串起来，每个消息都有一个唯一的 ID 和对应的内容。而且消息是持久化的。 + +![redis-stream](https://img.starfish.ink/redis/redis-stream.png) + + + + + +每个 Stream 都有唯一的名称，它就是 Redis 的 key，在我们首次使用 xadd 指令追加消息时自动创建。 + + + +Stream 是 Redis 专门为消息队列设计的数据类型，所以提供了丰富的消息队列操作命令。 + +#### Stream 常用命令 + +| 命令 | 描述 | 用法 | +| -------------------- | -------------------------------------------- | ------------------------------------------------------------ | +| **XADD** | 添加消息到末尾，保证有序，可以自动生成唯一ID | XADD key ID field value [field value ...] | +| XTRIM | 对流进行修剪，限制长度 | XTRIM key MAXLEN [~] count | +| XDEL | 删除消息 | XDEL key ID [ID ...] | +| XLEN | 获取流包含的元素数量，即消息长度 | XLEN key | +| XRANGE | 获取消息列表，会自动过滤已经删除的消息 | XRANGE key start end [COUNT count] | +| XREVRANGE | 反向获取消息列表，ID 从大到小 | XREVRANGE key end start [COUNT count] | +| **XREAD** | 以阻塞或非阻塞方式获取消息列表 | XREAD [COUNT count] [BLOCK milliseconds] STREAMS key [key ...] id [id ...] | +| **XGROUP CREATE** | 创建消费者组 | XGROUP [CREATE key groupname id-or-$] [SETID key groupname id-or-$] [DESTROY key groupname] [DELCONSUMER key groupname consumername] | +| **XREADGROUP GROUP** | 读取消费者组中的消息 | XREADGROUP GROUP group consumer [COUNT count] [BLOCK milliseconds] [NOACK] STREAMS key [key ...] ID [ID ...] | +| XACK | 将消息标记为"已处理" | XACK key group ID [ID ...] | +| XGROUP SETID | 为消费者组设置新的最后递送消息ID | XGROUP SETID [CREATE key groupname id-or-$] [SETID key groupname id-or-$] [DESTROY key groupname] | +| XGROUP DELCONSUMER | 删除消费者 | XGROUP DELCONSUMER [CREATE key groupname id-or-$] [SETID key groupname id-or-$] [DESTROY key groupname] | +| XGROUP DESTROY | 删除消费者组 | XGROUP DESTROY [CREATE key groupname id-or-$] [SETID key groupname id-or-$] [DESTROY key groupname] [DEL | +| XPENDING | 显示待处理消息的相关信息 | XPENDING key group [start end count] [consumer] | +| XCLAIM | 转移消息的归属权 | XCLAIM key group consumer min-idle-time ID [ID ...] [IDLE ms] [TIME ms-unix-time] [RETRYCOUNT count] | +| XINFO | 查看流和消费者组的相关信息 | XINFO [CONSUMERS key groupname] [GROUPS key] [STREAM key] [HELP] | +| XINFO GROUPS | 打印消费者组的信息 | XINFO GROUPS [CONSUMERS key groupname] [GROUPS key] [STREAM key] [HELP] | +| XINFO STREAM | 打印流信息 | XINFO STREAM [CONSUMERS key groupname] [GROUPS key] [STREAM key] [HELP] | + + + +#### CRUD 工程师上线 + +##### 增删改查来一波 + +```shell +# * 号表示服务器自动生成 ID，后面顺序跟着一堆 key/value +127.0.0.1:6379> xadd mystream * f1 v1 f2 v2 f3 v3 +"1609404470049-0" ## 生成的消息 ID，有两部分组成，毫秒时间戳-该毫秒内产生的第1条消息 + +# 消息ID 必须要比上个 ID 大 +127.0.0.1:6379> xadd mystream 123 f4 v4 +(error) ERR The ID specified in XADD is equal or smaller than the target stream top item + +# 自定义ID +127.0.0.1:6379> xadd mystream 1609404470049-1 f4 v4 +"1609404470049-1" + +# -表示最小值 , + 表示最大值,也可以指定最大消息ID，或最小消息ID，配合 -、+ 使用 +127.0.0.1:6379> xrange mystream - + +1) 1) "1609404470049-0" + 2) 1) "f1" + 2) "v1" + 3) "f2" + 4) "v2" + 5) "f3" + 6) "v3" +2) 1) "1609404470049-1" + 2) 1) "f4" + 2) "v4" + +127.0.0.1:6379> xdel mystream 1609404470049-1 +(integer) 1 +127.0.0.1:6379> xlen mystream +(integer) 1 +# 删除整个 stream +127.0.0.1:6379> del mystream +(integer) 1 +``` + +##### 独立消费 + +`xread` 以阻塞或非阻塞方式获取消息列表，指定 `BLOCK` 选项即表示阻塞，超时时间 0 毫秒（意味着永不超时） + +```shell +# 从ID是0-0的开始读前2条 +127.0.0.1:6379> xread count 2 streams mystream 0 +1) 1) "mystream" + 2) 1) 1) "1609405178536-0" + 2) 1) "f5" + 2) "v5" + 2) 1) "1609405198676-0" + 2) 1) "f1" + 2) "v1" + 3) "f2" + 4) "v2" + +# 阻塞的从尾部读取流，开启新的客户端xadd后发现这里就读到了,block 0 表示永久阻塞 +127.0.0.1:6379> xread block 0 streams mystream $ +1) 1) "mystream" + 2) 1) 1) "1609408791503-0" + 2) 1) "f6" + 2) "v6" +(42.37s) +``` + +可以看到，我并没有给流 `mystream` 传入一个常规的 ID，而是传入了一个特殊的 ID `$`这个特殊的 ID 意思是 **XREAD** 应该使用流 `mystream` 已经存储的最大 ID 作为最后一个 ID。以便我们仅接收从我们开始监听时间以后的新消息。这在某种程度上相似于 Unix 命令`tail -f`。 + +当然，也可以指定任意有效的 ID。 + +而且， `XREAD` 的阻塞形式还可以同时监听多个 Strema，只需要指定多个键名即可。 + +```c +127.0.0.1:6379> xread block 0 streams mystream yourstream $ $ +``` + + + +##### 创建消费者组 + +`xread` 虽然可以扇形分发到 N 个客户端，然而，在某些问题中，我们想要做的不是向许多客户端提供相同的消息流，而是从同一流向许多客户端提供不同的消息子集。比如下图这样，三个消费者按轮训的方式去消费一个 Stream。 + +![redis-stream-cg](https://img.starfish.ink/redis/redis-stream-cg.jpg) + +Redis Stream 借鉴了很多 Kafka 的设计。 + +- **Consumer Group**：有了消费组的概念，每个消费组状态独立，互不影响，一个消费组可以有多个消费者 + +- **last_delivered_id** ：每个消费组会有个游标 last_delivered_id 在数组之上往前移动，表示当前消费组已经消费到哪条消息了 +- **pending_ids** ：消费者的状态变量，作用是维护消费者的未确认的 id。 pending_ids 记录了当前已经被客户端读取的消息，但是还没有 ack。如果客户端没有 ack，这个变量里面的消息 ID 会越来越多，一旦某个消息被 ack，它就开始减少。这个 pending_ids 变量在 Redis 官方被称之为 `PEL`，也就是 `Pending Entries List`，这是一个很核心的数据结构，它用来确保客户端至少消费了消息一次，而不会在网络传输的中途丢失了没处理。 + +![redis-group-strucure](https://img.starfish.ink/redis/redis-group-strucure.png) + +Stream 不像 Kafka 那样有分区的概念，如果想实现类似分区的功能，就要在客户端使用一定的策略将消息写到不同的 Stream。 + +- `xgroup create`：创建消费者组 +- `xgreadgroup`：读取消费组中的消息 +- `xack`：ack 掉指定消息 + +![](https://img.starfish.ink/redis/redis-xgroup.jpg) + +```shell +# 创建消费者组的时候必须指定 ID, ID 为 0 表示从头开始消费，为 $ 表示只消费新的消息，也可以自己指定 +127.0.0.1:6379> xgroup create mystream mygroup $ +OK + +# 查看流和消费者组的相关信息，可以查看流、也可以单独查看流下的某个组的信息 +127.0.0.1:6379> xinfo stream mystream + 1) "length" + 2) (integer) 4 # 共 4 个消息 + 3) "radix-tree-keys" + 4) (integer) 1 + 5) "radix-tree-nodes" + 6) (integer) 2 + 7) "last-generated-id" + 8) "1609408943089-0" + 9) "groups" +10) (integer) 1 # 一个消费组 +11) "first-entry" # 第一个消息 +12) 1) "1609405178536-0" + 2) 1) "f5" + 2) "v5" +13) "last-entry" # 最后一个消息 +14) 1) "1609408943089-0" + 2) 1) "f6" + 2) "v6" +127.0.0.1:6379> +``` + +##### 按消费组消费 + +Stream 提供了 `xreadgroup` 指令可以进行消费组的组内消费，需要提供消费组名称、消费者名称和起始消息 ID。它同 `xread` 一样，也可以阻塞等待新消息。读到新消息后，对应的消息 ID 就会进入消费者的 PEL(正在处理的消息) 结构里，客户端处理完毕后使用 `xack` 指令通知服务器，本条消息已经处理完毕，该消息 ID 就会从 PEL 中移除。 + +```shell +# 消费组 mygroup1 中的 消费者 c1 从 mystream 中 消费组数据 +# > 号表示从当前消费组的 last_delivered_id 后面开始读 +# 每当消费者读取一条消息，last_delivered_id 变量就会前进 +127.0.0.1:6379> xreadgroup group mygroup1 c1 count 1 streams mystream > +1) 1) "mystream" + 2) 1) 1) "1609727806627-0" + 2) 1) "f1" + 2) "v1" + 3) "f2" + 4) "v2" + 5) "f3" + 6) "v3" +127.0.0.1:6379> xreadgroup group mygroup1 c1 count 1 streams mystream > +1) 1) "mystream" + 2) 1) 1) "1609727818650-0" + 2) 1) "f4" + 2) "v4" +# 已经没有消息可读了 +127.0.0.1:6379> xreadgroup group mygroup1 c1 count 2 streams mystream > +(nil) + +# 还可以阻塞式的消费 +127.0.0.1:6379> xreadgroup group mygroup1 c2 block 0 streams mystream > +µ1) 1) "mystream" + 2) 1) 1) "1609728270632-0" + 2) 1) "f5" + 2) "v5" +(89.36s) + +# 观察消费组信息 +127.0.0.1:6379> xinfo groups mystream +1) 1) "name" + 2) "mygroup1" + 3) "consumers" + 4) (integer) 2 # 2个消费者 + 5) "pending" + 6) (integer) 3 # 共 3 条正在处理的信息还没有 ack + 7) "last-delivered-id" + 8) "1609728270632-0" + +127.0.0.1:6379> xack mystream mygroup1 1609727806627-0 # ack掉指定消息 +(integer) 1 +``` + + + +尝鲜到此结束，就不继续深入了。 + +个人感觉，就目前来说，Stream 还是不能当做主流的 MQ 来使用的，而且使用案例也比较少，慎用。 + + + +## 写在最后 + +- 当然，还有需要注意的就是，业务上避免过度复用一个 Redis。既用它做缓存、做计算，还拿它做任务队列，这样的话 Redis 会很累的。 + +- 没有绝对好的技术、只有对业务最友好的技术，共勉 + + + +> 以梦为马，越骑越傻。诗和远方，越走越慌。不忘初心是对的，但切记要出发，加油吧，程序员。 + +![](https://img.starfish.ink/oceanus/end.jpg) + + + +## 参考 + +- [ 《Redis 设计与实现》](https://redisbook.readthedocs.io/en/latest/feature/pubsub.html) + +- [Redis 官网](https://redis.io/topics/pubsub) + +- https://segmentfault.com/a/1190000012244418 + +- https://www.cnblogs.com/williamjie/p/11201654.html + diff --git a/docs/data-management/Redis/Redis-Master-Slave.md b/docs/data-management/Redis/Redis-Master-Slave.md new file mode 100644 index 0000000000..bce755ff33 --- /dev/null +++ b/docs/data-management/Redis/Redis-Master-Slave.md @@ -0,0 +1,289 @@ +--- +title: Redis 主从复制 +date: 2021-10-08 +tags: + - Redis +categories: Redis +--- + +![](https://img.starfish.ink/redis/redis-master-slave-banner.png) + +> 我们总说的 Redis 具有高可靠性，其实，这里有两层含义：一是数据尽量少丢失，二是服务尽量少中断。 +> +> AOF 和 RDB 保证了前者，而对于后者，Redis 的做法就是增加副本冗余量，将一份数据同时保存在多个实例上，来避免单点故障。即使有一个实例出现了故障，需要过一段时间才能恢复，其他实例也可以对外提供服务，不会影响业务使用。 +> +> 这就是 Redis 的主从模式，主从库之间采用的是读写分离的方式。 + +![](https://img.starfish.ink/redis/redis-master-slave-mode.png) + +### 一、主从复制是啥 + +**主从复制**，或者叫 主从同步，是指将一台 Redis 服务器的数据，复制到其他的 Redis 服务器。前者称为 **主节点(master)**，后者称为 **从节点(slave)**。且数据的复制是 **单向** 的，只能由主节点到从节点。 + +Redis 主从复制支持 **主从同步** 和 **从从同步** 两种，后者是 Redis 后续版本新增的功能，以减轻主节点的同步负担。 + +### 二、主从复制的目的 + +- **数据冗余：** 主从复制实现了数据的热备份，是持久化之外的一种数据冗余方式。 +- **故障恢复：** 当主节点出现问题时，可以由从节点提供服务，实现快速的故障恢复 *(实际上是一种服务的冗余)*。 +- **负载均衡：** 在主从复制的基础上，配合读写分离，可以由主节点提供写服务，由从节点提供读服务 *（即写 Redis 数据时应用连接主节点，读 Redis 数据时应用连接从节点）*，分担服务器负载。尤其是在写少读多的场景下，通过多个从节点分担读负载，可以大大提高 Redis 服务器的并发量。 +- **高可用基石：** 除了上述作用以外，主从复制还是哨兵和集群能够实施的 **基础**，因此说主从复制是 Redis 高可用的基础。 + + + +### 三、Hello world + +当我们启动多个 Redis 实例的时候，它们相互之间就可以通过 `replicaof`（Redis 5.0 之前使用 `slaveof`，当然目前该命名也是向后兼容的，高版本也可以使用）命令形成主库和从库的关系 + +以下三种方式是 **完全等效** 的： + +- **配置文件**：在从服务器的配置文件中加入：`replicaof ` +- **启动命令**：redis-server 启动命令后加入 `--replicaof ` +- **客户端命令**：Redis 服务器启动后，直接通过客户端执行命令：`replicaof `，让该 Redis 实例成为从节点。 + +需要注意的是：**主从复制的开启，完全是在从节点发起的，不需要我们在主节点做任何事情**。即： **配从(库)不配主(库)** + +#### 3.1 本地启动两个节点 + +在正确安装好 Redis 之后，我们可以使用 `redis-server --port ` 的方式指定创建两个不同端口的 Redis 实例，例如，下方我分别创建了一个 `6379` 和 `6380` 的两个 Redis 实例： + +```bash +# 创建一个端口为 6379 的 Redis 实例 +redis-server --port 6379 +# 创建一个端口为 6380 的 Redis 实例 +redis-server --port 6380 +``` + +此时两个 Redis 节点启动后，都默认为 **主节点**。 + +#### 3.2 建立复制 + +我们在 `6380` 端口的节点中执行 `replicaof` 命令，使之变为从节点： + +```bash +# 在 6380 端口的 Redis 实例中使用控制台 +redis-cli -p 6380 +# 成为本地 6379 端口实例的从节点 +127.0.0.1:6380> REPLICAOF 127.0.0.1 6379 +OK +``` + +#### 3.3 观察效果 + +下面我们来验证一下，主节点的数据是否会复制到从节点之中： + +先在 **从节点** 中查询一个 **不存在** 的 key： + +```bash +127.0.0.1:6380> GET k1 +(nil) +``` + +再在 **主节点** 中添加这个 key + +```bash +127.0.0.1:6379> SET k1 v1 +OK +``` + +此时再从 **从节点** 中查询，会发现已经从 **主节点** 同步到 **从节点**： + +``` +127.0.0.1:6380> GET k1 +“v1” +``` + +#### 3.4 查看信息 + +可以通过 **info replication** 查看当前节点的复制信息 + +```bash +127.0.0.1:6380> REPLICAOF 127.0.0.1 6379 +OK +127.0.0.1:6380> info replication +# Replication +role:slave +master_host:127.0.0.1 +master_port:6379 +master_link_status:up +master_last_io_seconds_ago:7 +master_sync_in_progress:0 +slave_repl_offset:654 +slave_priority:100 +slave_read_only:1 +connected_slaves:0 +master_replid:be40ad10c509c150291dc571035dcb2eef835a38 +master_replid2:0000000000000000000000000000000000000000 +master_repl_offset:654 +second_repl_offset:-1 +repl_backlog_active:1 +repl_backlog_size:1048576 +repl_backlog_first_byte_offset:655 +repl_backlog_histlen:0 +``` + +#### 3.5 断开复制 + +通过 `REPLICAOF host port` 命令建立主从复制关系以后，可以通过 `replicaof no one` 断开。需要注意的是，从节点断开复制后，**不会删除已有的数据**，只是不再接受主节点新的数据变化。 + +```bash +127.0.0.1:6380> replicaof no one +OK +``` + + + +### 四、主从复制的工作过程 + +> 带着这几个疑问继续 +> +> - 主从库同步是如何完成的呢？ +> - 主库数据是一次性传给从库，还是分批同步？ +> - 如果主从库间的网络断连了，数据还能保持一致吗？ + +Redis 主从库之间的同步，在不同阶段有不同的处理方式，我们先来看下主从库通过 `replicaof` 建立连接之后，第一次同步是怎么进行的 + +#### 4.1 全量复制 | 快照同步 + +![redis-replicaof](https://img.starfish.ink/redis/redis-replicaof.png) + +为了节省篇幅，我把主要的步骤都 **浓缩** 在了上图中，其实也可以 **简化成三个阶段：建立连接阶段-数据同步阶段-命令传播阶段**。 + +1. 第一阶段是主从库间建立连接、协商同步的过程，主要是为全量复制做准备。在这一步，从库和主库建立起连接，并告诉主库即将进行同步，主库确认回复后，主从库间就可以开始同步了。 + + 具体来说，从库给主库发送 psync 命令，表示要进行数据同步，主库根据这个命令的参数来启动复制。psync 命令包含了主库的 runID 和复制进度 offset 两个参数。（2.8 版本之前的 `SYNC` 不做介绍） + + - runID，是每个 Redis 实例启动时都会自动生成的一个随机 ID，用来唯一标记这个实例。当从库和主库第一次复制时，因为不知道主库的 runID，所以将 runID 设为“ ？”。 + - offset，此时设为 -1，表示第一次复制。 + + 主库收到 psync 命令后，会用 **FULLRESYNC** 响应命令带上两个参数：主库 runID 和主库目前的复制进度 offset，返回给从库。从库收到响应后，会记录下这两个参数。 + + 这里有个地方需要注意，**FULLRESYNC 响应表示第一次复制采用的全量复制，也就是说，主库会把当前所有的数据都复制给从库**。 + + > 这一步其实还有很多其他的流程，比如从节点会发送 ping 检查 socket 连接是否可用。如果 master 设置了 requirepass ，那 slave 节点就必须设置 masterauth 选项来进行身份验证... + +2. 第二阶段，主库将所有数据同步给从库。从库收到数据后，在本地完成数据加载。这个过程依赖于内存快照生成的 RDB 文件。 + + 具体来说，主库执行 bgsave 命令，生成 RDB 文件，接着将文件发给从库。从库接收到 RDB 文件后，会先清空当前数据库，然后加载 RDB 文件。这是因为从库在通过 replicaof 命令开始和主库同步前，可能保存了其他数据。为了避免之前数据的影响，从库需要先把当前数据库清空。 + + 在主库将数据同步给从库的过程中，主库不会被阻塞，仍然可以正常接收请求。否则，Redis 的服务就被中断了。但是，这些请求中的写操作并没有记录到刚刚生成的 RDB 文件中。为了保证主从库的数据一致性，主库会在内存中用专门的 `replication buffer`，记录 RDB 文件生成后收到的所有写操作。 + +3. 最后，也就是第三个阶段，主库会把第二阶段执行过程中新收到的写命令，再发送给从库。 + + 具体的操作是，当主库完成 RDB 文件发送后，就会把此时 `replication buffer` 中的修改操作发给从库，从库再重新执行这些操作。这样一来，主从库就实现同步了。 + + > 主节点在生成 RDB 文件时，会将新的写命令（例如 `SET`、`DEL` 等）追加到**复制积压缓冲区**中，同时这些命令也会通过网络直接发送到所有已连接的从节点。 + > + > 这些写操作是以 **Redis 协议格式（RESP）** 逐条发送的。 + > + > - 双管齐下（RDB + 命令传播） + > + > ##### **数据传输协议：RESP**(Redis Serialization Protocol) + > + > - Redis 的所有数据通信，包括 RDB 文件和增量数据的发送，均基于其内部协议 **RESP（Redis Serialization Protocol）**。 + > - 增量数据是以 Redis 命令流的形式，序列化为 RESP 格式后通过 TCP 连接发送的 + + + +##### 主库压力问题 | 主从级联模式 + +从主从库之间的第一次数据同步过程，可以看到，一次全量复制中，对于主库来说，需要完成两个耗时的操作：生成 RDB 文件和传输 RDB 文件。 + +如果从库数量很多，而且都要和主库进行全量复制的话，就会导致主库忙于 fork 子进程生成 RDB 文件，进行数据全量同步。fork 这个操作会阻塞主线程处理正常请求，从而导致主库响应应用程序的请求速度变慢。此外，传输 RDB 文件也会占用主库的网络带宽，同样会给主库的资源使用带来压力。 + +所以 Redis 也支持 “ 主 - 从 - 从” 这样的模式。 + +其实就是通过级联的方式，将主库的压力分担给部分从库。 + +我们在部署主从集群的时候，可以手动选择一个从库（比如选择内存资源配置较高的从库），用于级联其他的从库。然后，我们可以再选择一些从库（例如三分之一的从库），在这些从库上执行如下命令，让它们和刚才所选的从库，建立起主从关系。 + +```bash +replicaof 所选从库的IP 6379 +``` + +再看下文章开头的图。 + +![](https://img.starfish.ink/redis/redis-master-slave-mode.png) + +##### 无盘复制 + +主节点在进行快照同步时，会进行很重的文件 IO 操作，特别是对于非 SSD 磁盘存储时，快照会对系统的负载产生较大影响。特别是当系统正在进行 AOF 的 fsync 操作时如果发生快照，fsync 将会被推迟执行，这就会严重影响主节点的服务效率。 + +所以从 Redis 2.8.18 版开始支持无盘复制。所谓无盘复制是指主服务器直接通过套接字 socket 将快照内容发送到从节点，生成快照是一个遍历的过程，主节点会一边遍历内存，一边将序列化的内容发送到从节点，从节点还是跟之前一样，先将接收到的内容存储到磁盘文件中，再进行一次性加载。 + + + +#### 4.2 命令传播 + +一旦主从库完成了全量复制，它们之间就会一直维护一个网络连接，主库会通过这个连接将后续陆续收到的命令操作再同步给从库，这个过程也称为 **基于长连接的命令传播**，可以避免频繁建立连接的开销。 + +##### 心跳机制 + +在命令传播阶段，除了发送写命令，主从节点还维持着心跳机制：PING 和 REPLCONF ACK。心跳机制对于主从复制的超时判断、数据安全等有作用。 + +- 每隔指定的时间，**从节点会向主节点发送 PING 命令**， 并报告复制流的处理情况。 + + PING 发送的频率由 `repl-ping-slave-period` 参数控制，单位是秒，默认值是 10s。 + +- 在命令传播阶段，**从节点会向主节点发送 REPLCONF ACK** 命令，频率是每秒 1 次；命令格式为：`REPLCONF ACK {offset}`，其中 offset 指从节点保存的复制偏移量。REPLCONF ACK 命令的作用包括： + - 实时监测主从节点网络状态：该命令会被主节点用于复制超时的判断。此外，在主节点中使用 info Replication，可以看到其从节点的状态中的 lag 值，代表的是主节点上次收到该 REPLCONF ACK 命令的时间间隔，在正常情况下，该值应该是 0 或 1 + - 检测命令丢失：从节点发送了自身的 offset，主节点会与自己的 offset 对比，如果从节点数据缺失（如网络丢包），主节点会推送缺失的数据（这里也会利用复制积压缓冲区）。**注意**，offset 和复制积压缓冲区，不仅可以用于部分复制，也可以用于处理命令丢失等情形；区别在于前者是在断线重连后进行的，而后者是在主从节点没有断线的情况下进行的。 + - 辅助保证从节点的数量和延迟：Redis 主节点中使用 `min-slaves-to-write` 和 `min-slaves-max-lag` 参数，来保证主节点在不安全的情况下不会执行写命令；所谓不安全，是指从节点数量太少，或延迟过高。例如 `min-slaves-to-write` 和 `min-slaves-max-lag` 分别是 3 和 10，含义是如果从节点数量小于 3 个，或所有从节点的延迟值都大于 10s，则主节点拒绝执行写命令。而这里从节点延迟值的获取，就是通过主节点接收到 REPLCONF ACK 命令的时间来判断的，即前面所说的 info Replication 中的 lag 值。 + +不过， 因为 Redis 主从使用异步复制， 这就意味着当主节点挂掉时，从节点可能没有收到全部的同步消息，这部分未同步的消息就丢失了。如果主从延迟特别大，那么丢失的数据就可能会特别多。 + + + +#### 4.3 增量复制 | 部分复制 + +你以为这样主从同步就结束了？ + +万一网络断连或者网络阻塞，主从库之间的长连接就断了，接下来就面临一个继续同步的问题。 + +在 Redis 2.8 之前，如果主从库在命令传播时出现了网络闪断，那么，从库就会和主库重新进行一次全量复制，开销非常大。 + +从 Redis 2.8 开始，网络断了之后，主从库会采用 **增量复制** 的方式继续同步。 + +增量复制的原理主要是靠主从节点分别维护一个 **复制偏移量**，有了这个偏移量，断线重连之后一比较，之后就可以仅仅把从服务器断线之后缺失的这部分数据给补回来了。 + +全量复制中有 `replication buffer` 这样的缓存区来保存 RDB 文件生成后收到的所有写操作，增量复制中也有一个缓存区，叫 `repl_backlog_buffer` ，默认是 1M。 + +> 当主从库断连后，主库会把断连期间收到的写操作命令，写入 `replication buffer`，同时也会把这些操作命令也写入 `repl_backlog_buffer` 这个缓冲区。 + +`repl_backlog_buffer` 是一个环形缓冲区，主库会记录自己写到的位置，从库则会记录自己已经读到的位置。 + +主库对应的偏移量是 `master_repl_offset`，从库的偏移量 `slave_repl_offset` 。正常情况下，这两个偏移量基本相等。 + +![](https://img.starfish.ink/redis/redis-backlog_buffer.png) + +在网络断连阶段，主库可能会收到新的写操作命令，这时，`master_repl_offset` 会大于 `slave_repl_offset`。此时，主库只用把 `master_repl_offset` 和 `slave_repl_offset` 之间的命令操作同步给从库就可以了。 + +![](https://img.starfish.ink/redis/redis-increment-copy.png) + +> PS：因为 repl_backlog_buffer 是一个环形缓冲区（可以理解为是一个定长的环形数组），所以在缓冲区写满后，主库会继续写入，此时，就会覆盖掉之前写入的操作。**如果从库的读取速度比较慢，就有可能导致从库还未读取的操作被主库新写的操作覆盖了，这会导致主从库间的数据不一致**。如果从库和主库**断连时间过长**，造成它在主库 repl_backlog_buffer 的 slave_repl_offset 位置上的数据已经被覆盖掉了，此时从库和主库间将进行全量复制。 +> +> 因此，我们要想办法避免这一情况，一般而言，我们可以调整 repl_backlog_size 这个参数。这个参数和所需的缓冲空间大小有关。缓冲空间的计算公式是：缓冲空间大小 = 主库写入命令速度 * 操作大小 - 主从库间网络传输命令速度 * 操作大小。在实际应用中，考虑到可能存在一些突发的请求压力，我们通常需要把这个缓冲空间扩大一倍，即 repl_backlog_size = 缓冲空间大小 * 2，这也就是 repl_backlog_size 的最终值。 +> +> 举个例子，如果主库每秒写入 2000 个操作，每个操作的大小为 2KB，网络每秒能传输 1000 个操作，那么，有 1000 个操作需要缓冲起来，这就至少需要 2MB 的缓冲空间。否则，新写的命令就会覆盖掉旧操作了。为了应对可能的突发压力，我们最终把 repl_backlog_size 设为 4MB。 +> +> 这样一来，增量复制时主从库的数据不一致风险就降低了。不过，如果并发请求量非常大，连两倍的缓冲空间都存不下新操作请求的话，此时，主从库数据仍然可能不一致。 +> + + + +### 五、小结 + +Redis 的主从库同步的基本原理，总结来说，有三种模式：全量复制、基于长连接的命令传播，以及增量复制。 + +全量复制虽然耗时，但是对于从库来说，如果是第一次同步，全量复制是无法避免的，所以，**一个 Redis 实例的数据库不要太大**，一个实例大小在几 GB 级别比较合适，这样可以减少 RDB 文件生成、传输和重新加载的开销。另外，为了避免多个从库同时和主库进行全量复制，给主库过大的同步压力，我们也可以采用“主 - 从 - 从”这一级联模式，来缓解主库的压力。 + +我们常用一主二仆的配置，即一个主节点对应两个从节点。 + +主从复制是 Redis 分布式的基础，Redis 的高可用离开了主从复制将无从进行。 + + + +## 参考 + +- https://www.cnblogs.com/kismetv/p/9236731.html +- 《Redis 核心技术与实战》 \ No newline at end of file diff --git a/docs/data-management/Redis/Redis-Persistence.md b/docs/data-management/Redis/Redis-Persistence.md new file mode 100644 index 0000000000..9bb9414067 --- /dev/null +++ b/docs/data-management/Redis/Redis-Persistence.md @@ -0,0 +1,412 @@ +--- +title: Redis的持久化机制 +date: 2020-12-20 +tags: + - Redis +categories: Redis +--- + +![](https://images.pexels.com/photos/33278/disc-reader-reading-arm-hard-drive.jpg?cs=srgb&dl=pexels-pixabay-33278.jpg) + +> 带着疑问，或者是面试问题去看 Redis 的持久化，或许会有不一样的视角，这几个问题你废了吗？ +> +> - Redis 有哪几种持久化方式？有什么区别？ +> +> - 如何选择合适的持久化方式？项目中用的哪种，为什么？ +> +> - aof 如果文件越来越大，怎么办？ +> +> - Redis 采用 aof 持久化时，数据是先写入内存，还是先写入日志，为什么？ + +Redis 的数据全部在内存里，如果突然宕机，数据就会全部丢失，因此必须有一种机制来保证 Redis 的数据不会因为故障而丢失，这种机制就是 Redis 的持久化机制，它会将内存中的数据库状态 **保存到磁盘** 中。 + +**Redis 有两种持久化的方式：快照（`RDB`文件）和追加式文件（`AOF`文件)** + + + +## RDB（Redis DataBase） + +#### 是什么 + +**在指定的时间间隔内将内存中的所有数据集快照写入磁盘**，也就是行话讲的 Snapshot 快照，它执行的是**全量快照**，它恢复时是将快照文件直接读到内存里。 + +> 这就类似于照片，当你拍照时，一张照片就能把拍照那一瞬间的形象完全记下来。 +> +> ![](https://i04piccdn.sogoucdn.com/2c4975a0d2f847c6) + +Redis 会单独创建（fork）一个子进程来进行持久化，会先将数据写入到一个临时文件中，待持久化过程都结束了，再用这个临时文件替换上次持久化好的文件。整个过程中，主进程是不进行任何 IO 操作的，这就确保了极高的性能，如果需要进行大规模数据的恢复，且对于数据恢复的完整性不是非常敏感，那 RDB 方式是比较高效的。 + +RDB 的缺点是最后一次持久化后的数据可能丢失。 + +> **?** What ? Redis 不是单进程的吗? + +> Redis 使用操作系统的多进程 COW(Copy On Write) 机制来实现快照持久化（在执行快照的同时，正常处理写操作）， fork 是类 Unix 操作系统上**创建进程**的主要方法。COW(Copy On Write)是计算机编程中使用的一种优化策略。 +> +> fork 的作用是复制一个与当前进程一样的进程。新进程的所有数据（变量、环境变量、程序计数器等）数值都和原进程一致，但是是一个全新的进程，并作为原进程的子进程。 子进程读取数据，然后序列化写到磁盘中。 + +#### 配置 + +**配置位置**： SNAPSHOTTING + +![](https://img.starfish.ink/redis/redis-snapshotting.jpg) + +rdb 默认保存的是 **dump.rdb** 文件，如下（不可读） + +![](https://img.starfish.ink/redis/redis-rdb-file.jpg) + +你可以对 Redis 进行设置， 让它在“ N 秒内数据集至少有 M 个改动”这一条件被满足时， 自动保存一次数据集。 + +比如说， 以下设置会让 Redis 在满足“ 60 秒内有至少有 1000 个键被改动”这一条件时， 自动保存一次数据集： + +`save 60 1000 ` + +#### 触发 RDB 快照 + +除了通过配置文件的方式，自动触发生成快照，也可以使用命令手动触发 + +- **save**：save 时只管保存，在主线程中执行，会导致阻塞，所以请慎用； +- **bgsave**：可以理解为 `background save` ，当执行 bgsave 命令时，redis 会 fork 出一个子进程，专门用于写入 RDB 文件，避免了主线程的阻塞，这也是 Redis RDB 文件生成的默认配置。可以通过 `lastsave` 命令获取最后一次成功执行快照的时间 +- 执行 **flushall** 命令，也会产生 dump.rdb 文件，但里面是空的，无意义 +- 客户端执行 shutdown 关闭 redis 时，也会触发快照 + +> 简单来说，bgsave 子进程是由主线程 fork 生成的，可以共享主线程的所有内存数据。bgsave 子进程运行后，开始读取主线程的内存数据，并把它们写入 RDB 文件。此时，如果主线程对这些数据也都是读操作（例如图中的键值对K1），那么，主线程和 bgsave 子进程相互不影响。但是，如果主线程要修改一块数据（例如图中的键值对 K3），那么，这块数据就会被复制一份，生成该数据的副本。然后，bgsave 子进程会把这个副本数据写入 RDB 文件，而在这个过程中，主线程仍然可以直接修改原来的数据。 +> +> ![](https://img.starfish.ink/redis/redis-bgsave.jpg) +> +> 当 Redis 触发 RDB 快照时，它会通过 `fork` 系统调用来创建一个子进程，子进程负责将内存中的数据写入到磁盘，而父进程则继续响应客户端请求。**写时复制（COW）** 机制使得在 fork 后，父子进程共享同一份内存页，只有当父进程或子进程对内存进行修改时，操作系统才会复制出新的内存页，从而减少了内存复制的开销。 +> +> 尽管如此，`fork` 子进程的操作本身仍然是有一定开销的，尤其在数据量非常大的时候，尽管采用了 COW，但以下因素可能会导致 **性能下降**： +> +> - **大量数据**：当 Redis 中的数据量非常大时（几千万或几亿条键值对），fork 子进程仍然需要花费一定的时间来生成 RDB 文件，即使是写时复制也会带来一定的延迟。 +> - **系统资源压力**：如果机器的内存不足，频繁的 fork 和写时复制操作可能导致系统的 **内存不足** 和 **CPU 高负载**，尤其是在数据量较大的时候。 +> - **磁盘 I/O**：即便使用写时复制，子进程最终需要将数据写入磁盘，磁盘 I/O 的性能也会影响快照的时间。如果磁盘写入速度较慢，生成 RDB 文件的过程可能会很慢。 +> +> 如果 Redis 在执行一次 RDB 快照时，快照尚未完成，新的写操作又触发了一个新的 RDB 快照，那么会发生 **重叠触发** 的问题。 +> +> - **新的 RDB 快照会等待前一个快照完成**。 +> - 如果 **`save` 配置的触发条件**（如 `save 900 1`）满足且当前快照未完成，Redis 会 **阻塞新的快照请求**，直到当前的快照操作完成。这是为了避免对系统资源的过度消耗，防止多次快照操作同时进行。 + + + +#### 快照的运作方式 + +当 Redis 需要保存 dump.rdb 文件时， 服务器执行以下操作： + +1. Redis 调用 `fork()`，产生一个子进程，此时同时拥有父进程和子进程。 +2. 父进程继续处理 client 请求，子进程负责将内存内容写入到临时文件。由于 os 的写时复制机制，父子进程会共享相同的物理页面，当父进程处理写请求时， os 会为父进程要修改的页面创建副本，而不是写共享的页面。所以子进程的地址空间内的数据是 fork 时刻整个数据库的一个快照。 +3. 当子进程完成对新 RDB 文件的写入时，Redis 用新 RDB 文件替换原来的 RDB 文件，并删除旧的 RDB 文件。 + +这种工作方式使得 Redis 可以从写时复制（copy-on-write）机制中获益。 + +#### 如何恢复 + +将备份文件 (dump.rdb) 移动到 Redis 安装目录并启动服务即可（`CONFIG GET dir` 获取目录） + +![](https://img.starfish.ink/redis/redis-rdb-bak.jpg) + + + +#### 优势 + +- 一旦采用该方式，那么你的整个 Redis 数据库将只包含一个文件，这对于**文件备份**而言是非常完美的。比如，你可能打算每个小时归档一次最近 24 小时的数据，同时还要每天归档一次最近 30 天的数据。通过这样的备份策略，一旦系统出现灾难性故障，我们可以非常容易的进行恢复。**适合大规模的数据恢复** +- 对于灾难恢复而言，RDB 是非常不错的选择。因为我们可以非常轻松的将一个单独的文件压缩后再转移到其它存储介质上。 +- 性能最大化。对于 Redis 的服务进程而言，在开始持久化时，它唯一需要做的只是 fork 出子进程，之后再由子进程完成这些持久化的工作，这样就可以极大的避免服务进程执行 IO 操作了。 + +#### 劣势 + +- 如果你想保证数据的高可用性，即最大限度的避免数据丢失，那么 RDB 不是一个很好的选择。因为系统一旦在定时持久化之前出现宕机现象，此前没有来得及写入磁盘的数据都将丢失（丢失最后一次快照后的所有修改）。 +- 由于 RDB 是通过 fork 子进程来协助完成数据持久化工作的，内存中的数据被克隆了一份，大致 2 倍的膨胀性需要考虑，因此，如果当数据集较大时，可能会导致整个服务器停止服务几百毫秒，甚至是 1 秒钟。 + +> 可能你也会和我有同样的疑问，反正是不阻塞的，我每秒做一次快照，不就可以最大限度的避免数据丢失了吗？ +> +> 一方面，频繁将全量数据写入磁盘，会给磁盘带来很大压力，多个快照竞争有限的磁盘带宽，前一个快照还没有做完，后一个又开始做了，容易造成恶性循环。 +> +> 另一方面，bgsave 子进程需要通过 fork 操作从主线程创建出来。虽然，子进程在创建后不会再阻塞主线程，但是，**fork 这个创建过程本身会阻塞主线程**，而且主线程的内存越大，阻塞时间越长。如果频繁 fork 出 bgsave 子进程，这就会频繁阻塞主线程了。 + +#### 如何停止 + +动态停止 RDB 保存规则的方法：`redis-cli config set save ""`，或者修改配置文件，重启即可。 + +#### 小总结 + +![](https://img.starfish.ink/redis/redis-rdb-summary.jpg) + +- RDB 是一个非常紧凑的文件 + +- RDB 在保存 RDB 文件时父进程唯一需要做的就是 fork 出一个子进程，接下来的工作全部由子进程来做，父进程不需要再做其他 IO 操作，所以 RDB 持久化方式可以最大化 Redis 的性能 + +- 数据丢失风险大 + +- RDB 需要经常 fork 子进程来保存数据集到硬盘上，当数据集比较大的时候，fork 的过程是非常耗时的，可能会导致 Redis 在一些毫秒级不能响应客户端的请求 + + + +## AOF（Append Only File） + +#### 是什么 + +以日志的形式来记录每个写操作，将 Redis 执行过的所有**写指令**记录下来(读操作不记录)，只许追加文件但不可以改写文件，Redis 启动之初会读取该文件重新构建数据，也就是「**重放**」。换言之，Redis 重启的话就根据日志文件的内容将写指令从前到后执行一次以完成数据的恢复工作。 + +#### 配置 + +AOF 默认保存的是 **appendonly.aof ** 文件 + +**配置位置**： APPEND ONLY MODE + +![](https://img.starfish.ink/redis/redis-aof-conf.png) + + + +#### AOF 启动/修复/恢复 + +- 正常恢复 + + - 启动：修改默认的 appendonly no，改为 yes + - 将有数据的 aof 文件复制一份保存到对应目录(`config get dir`) + - 恢复：重启 redis 然后重新加载 + +- 异常恢复 + + - 启动：修改默认的 appendonly no，改为 yes + - 备份被写坏的 AOF 文件 + - 修复：**redis-check-aof --fix** 进行修复 + AOF 文件 + - 恢复：重启 redis 然后重新加载 + + + +#### AOF 日志是如何实现的？ + +说到日志，我们比较熟悉的是数据库的写前日志（Write Ahead Log, WAL），也就是说，在实际写数据前，先把修改的数据记到日志文件中，以便故障时进行恢复（DBA 们常说的“日志先行”）。 + +不过，AOF 日志正好相反，它是写后日志，“写后”的意思是 Redis 是先执行命令，把数据写入内存，然后才记录日志，如下图所示： + +![](https://img.starfish.ink/redis/redis-aof-write-log.png) + +> Tip：日志先行的方式，如果宕机后，还可以通过之前保存的日志恢复到之前的数据状态。可是 AOF 后写日志的方式，如果宕机后，不就会把写入到内存的数据丢失吗？ +> +> 那 AOF 为什么要先执行命令再记日志呢？要回答这个问题，我们要先知道 AOF 里记录了什么内容。 + +传统数据库的日志，例如 redo log（重做日志），记录的是修改后的数据，而 AOF 里记录的是 Redis 收到的每一条命令，这些命令是以文本形式保存的。 + +我们以 Redis 收到 “set k1 v1” 命令后记录的日志为例，看看 AOF 日志的内容。其中，“*2” 表示当前命令有两个部分，每部分都是由 `“$+数字” `开头，后面紧跟着具体的命令、键或值。这里，“数字”表示这部分中的命令、键或值一共有多少字节。 + +例如，`*2` 表示有两个部分，`$6` 表示 6 个字节，也就是下边的 “SELECT” 命令，`$1` 表示 1 个字节，也就是下边的 “0” 命令，合起来就是 `SELECT 0`，选择 0 库。下边的指令同理，就很好理解了 `SET K1 V1`。 + +![](https://img.starfish.ink/redis/redis-aof-file.png) + +但是，为了避免额外的检查开销，**Redis 在向 AOF 里面记录日志的时候，并不会先去对这些命令进行语法检查。所以，如果先记日志再执行命令的话，日志中就有可能记录了错误的命令，Redis 在使用日志恢复数据时，就可能会出错。而写后日志这种方式，就是先让系统执行命令，只有命令能执行成功，才会被记录到日志中，否则，系统就会直接向客户端报错**。所以，Redis 使用写后日志这一方式的一大好处是，可以避免出现记录错误命令的情况。除此之外，AOF 还有一个好处：它是在命令执行后才记录日志，所以**不会阻塞当前的写操作**。 + + + +**不过，AOF 也有两个潜在的风险。** + +- 首先，如果刚执行完一个命令，还没有来得及记日志就宕机了，那么这个命令和相应的数据就有丢失的风险。如果此时 Redis 是用作缓存，还可以从后端数据库重新读入数据进行恢复，但是，如果 Redis 是直接用作数据库的话，此时，因为命令没有记入日志，所以就无法用日志进行恢复了。 + +- 其次，AOF 虽然避免了对当前命令的阻塞，但可能会给下一个操作带来阻塞风险。这是因为，AOF 日志也是在主线程中执行的，如果在把日志文件写入磁盘时，磁盘写压力大，就会导致写盘很慢，进而导致后续的操作也无法执行了。 + +仔细分析的话，你就会发现，这两个风险都是和 AOF 写回磁盘的时机相关的。这也就意味着，如果我们能够控制一个写命令执行完后 AOF 日志写回磁盘的时机，这两个风险就解除了。 + +接着，我们就看下 Redis 提供的写回策略，或者叫 AOF 耐久性。 + + + +#### 三种写回策略 | AOF 耐久性 + +你可以配置 Redis 多久才将数据 fsync 到磁盘一次。AOF 机制给我们提供了三个选择： + +- **appendfsync always**，同步写回：每个写命令执行完，立马同步地将日志写回磁盘； + +- **appendfsync everysec**，每秒写回：每个写命令执行完，只是先把日志写到 AOF 文件的内存缓冲区，每隔一秒把缓冲区中的内容写入磁盘； + + 但是当这一次的 fsync 调用时长超过 1 秒时。Redis 会采取延迟 fsync 的策略，再等一秒钟。也就是在两秒后再进行 fsync，这一次的 fsync 就不管会执行多长时间都会进行。这时候由于在 fsync 时文件描述符会被阻塞，所以当前的写操作就会阻塞。 + +- **appendfsync no**，操作系统控制的写回：每个写命令执行完，只是先把日志写到 AOF 文件的内存缓冲区，由操作系统决定何时将缓冲区内容写回磁盘。**对大多数 Linux 操作系统，是每 30 秒进行一次 fsync，将缓冲区中的数据写到磁盘上**。 + +针对避免主线程阻塞和减少数据丢失问题，这三种写回策略都无法做到两全其美。我们来分析下其中的原因。 + +- “同步写回”可以做到基本不丢数据，但是它在每一个写命令后都有一个慢速的落盘操作，不可避免地会影响主线程性能； +- 虽然“操作系统控制的写回”在写完缓冲区后，就可以继续执行后续的命令，但是落盘的时机已经不在 Redis 手中了，只要 AOF 记录没有写回磁盘，一旦宕机对应的数据就丢失了； +- “每秒写回”采用一秒写回一次的频率，避免了“同步写回”的性能开销，虽然减少了对系统性能的影响，但是如果发生宕机，上一秒内未落盘的命令操作仍然会丢失。所以，这只能算是，在避免影响主线程性能和避免数据丢失两者间取了个折中。 + +| 配置项 | 写回时机 | 优点 | 缺点 | +| -------- | ------------------ | ------------------------ | -------------------------------------------- | +| Always | 同步写回 | 可靠性高，数据基本不丢失 | 每个写命令都要落盘，性能影响较大，慢但是安全 | +| Everysec | 每秒写回 | 性能适中 | 宕机时丢失1秒内的数据 | +| No | 操作系统控制的写回 | 性能好 | 宕机时丢失数据较多 | + +到这里，我们就可以根据系统对高性能和高可靠性的要求，来选择使用哪种写回策略了。 + +***总结一下就是：想要获得高性能，就选择 No 策略；如果想要得到高可靠性保证，就选择 Always 策略；如果允许数据有一点丢失，又希望性能别受太大影响的话，那么就选择 Everysec 策略。*** + + + +> **Tip**：试想一下，如果我们开启 AOF 运行个一年半载的，AOF 文件是不是会越来越大，先不说占用资源的问题，如果宕机重启，以 AOF 文件重做数据，肯定是个特别漫长的过程，所以 Redis 提供了对 AOF 文件的“瘦身”机制。 + +#### rewrite（AOF 重写） + +- 是什么：AOF 采用文件追加方式，文件会越来越大，为了避免出现这种情况，新增了重写机制，当 AOF 文件的大小超过所设定的阈值时，**Redis 就会启动 AOF 文件的内容压缩，只保留可以恢复数据的最小指令集**，可以使用命令 `bgrewriteaof`，这个操作相当于对 AOF 文件“瘦身”。在重写的时候，是根据这个键值对当前的最新状态，为它生成对应的写入命令。这样一来，一个键值对在重写日志中只用一条命令就行了，而且，在日志恢复时，只用执行这条命令，就可以直接完成这个键值对的写入了。 + + ![](https://i01piccdn.sogoucdn.com/8700d2e646eddccf) + +- **重写原理**：AOF 文件持续增长而过大时，会 fork 出一条新进程来将文件重写(也是先写临时文件最后再rename)，遍历新进程的内存中数据，转换成一条条的操作指令，再序列化到一个新的 AOF 文件中。 + + > PS: 重写 AOF 文件的操作，并没有读取旧的 AOF 文件，而是将整个内存中的数据库内容用命令的方式重写了一个新的 AOF 文件，这点和快照有点类似。 + +- 触发机制： + + - **AOF 文件增长比例超过一定阈值**：Redis 会记录上次重写时的 AOF 大小，默认配置是当 AOF 文件大小是上次 rewrite 后大小的**一倍**且文件大于 64M 时触发 + + ``` + auto-aof-rewrite-min-size 64mb //指定触发重写的 AOF 文件最小大小（默认为 64MB） + auto-aof-rewrite-percentage 100 //指定 AOF 文件增长的百分比（默认为 100%） + ``` + + - **手动触发 AOF 重写**： 通过执行 `BGREWRITEAOF` 命令，用户可以手动触发 AOF 重写 + + 我们在客户端输入两次 `set k1 v1` ，然后比较 `bgrewriteaof` 前后两次的 appendonly.aof 文件（先要关闭混合持久化）![bgrewriteaof](https://img.starfish.ink/redis/bgrewriteaof.png) + + + +#### 如果 AOF 文件出错了，怎么办？ + +服务器可能在程序正在对 AOF 文件进行写入时停机， 如果停机造成了 AOF 文件出错（corrupt）， 那么 Redis 在重启时会拒绝载入这个 AOF 文件， 从而确保数据的一致性不会被破坏。 + +当发生这种情况时， 可以用以下方法来修复出错的 AOF 文件： + +1. 为现有的 AOF 文件创建一个备份。 +2. 使用 Redis 附带的 redis-check-aof 程序，对原来的 AOF 文件进行修复。 + +**$ redis-check-aof --fix** + +1. （可选）使用 diff -u 对比修复后的 AOF 文件和原始 AOF 文件的备份，查看两个文件之间的不同之处。 +2. 重启 Redis 服务器，等待服务器载入修复后的 AOF 文件，并进行数据恢复。 + +#### AOF 运作方式 | 后台重写 + +AOF 重写和 RDB 创建快照一样，都巧妙地利用了写时复制机制。 + +不过， 使用子进程也有一个问题需要解决： 因为子进程在进行 AOF 重写期间， 主进程还需要继续处理命令， 而新的命令可能对现有的数据进行修改， 这会让当前数据库的数据和重写后的 AOF 文件中的数据不一致。 + +为了解决这个问题， Redis 增加了一个 AOF 重写缓存， 这个缓存在 fork 出子进程之后开始启用， Redis 主进程在接到新的写命令之后， 除了会将这个写命令的协议内容追加到现有的 AOF 文件之外， 还会追加到这个缓存中： + +以下是 AOF 重写的执行步骤： + +1. **创建一个新的 AOF 文件**：Redis 执行 `fork()` ，它会**异步地**在后台创建一个新的 AOF 文件（通常命名为 `appendonly.aof.new`）。现在同时拥有父进程和子进程。 + +2. **重写的内容——只保留当前数据库状态**：子进程开始将新 AOF 文件的内容写入到临时文件。 + + 在执行 AOF 重写时，Redis 不会记录每个命令的详细内容，而是通过以下方式来确保重写后的 AOF 文件有效： + + - **仅记录当前数据集的重建命令**：Redis 会根据数据库当前的状态生成一系列 `SET`、`HSET`、`LPUSH` 等命令，这些命令能够重建当前数据库的状态。这些命令是可以直接执行的、能够恢复数据的命令。 + - **压缩冗余命令**：在重写过程中，Redis 会去除一些无意义或重复的命令。例如，如果有大量重复的 `SET key value` 操作，Redis 只会保留最新的 `SET` 命令，而忽略之前的操作。 + +3. **复制 AOF文件**：对于所有新执行的写入命令，父进程一边将它们累积到一个内存缓存中，一边将这些改动追加到现有 AOF 文件（即 `appendonly.aof`）的末尾： 这样即使在重写的中途发生停机，现有的 AOF 文件也还是安全的，这个过程称为 **追加操作**。 + +4. **重写完成后的替换**：当子进程完成重写工作时，它给父进程发送一个信号，父进程在接收到信号之后，将内存缓存中的所有数据追加到新 AOF 文件的末尾。 + +5. **删除过时的 AOF 文件**：搞定！现在 Redis 原子地用新文件替换旧文件，之后所有命令都会直接追加到新 AOF 文件的末尾。 + +![](https://img.starfish.ink/redis/redis-aof-rewrite-work.png) + +#### 优势 + +- 该机制可以带来更高的数据安全性，即数据持久性。Redis 中提供了 3 种同步策略，即**每秒同步、每修改同步和不同步**。事实上，每秒同步也是异步完成的，其效率也是非常高的，所差的是一旦系统出现宕机现象，那么这一秒钟之内修改的数据将会丢失。而每修改同步，我们可以将其视为同步持久化，即每次发生的数据变化都会被立即记录到磁盘中。可以预见，这种方式在效率上是最低的。至于无同步，无需多言，我想大家都能正确的理解它。 +- 由于该机制对日志文件的写入操作采用的是 append 模式，因此在写入过程中即使出现宕机现象，也不会破坏日志文件中已经存在的内容。然而如果我们本次操作只是写入了一半数据就出现了系统崩溃问题，不用担心，在 Redis 下一次启动之前，我们可以通过 **redis-check-aof** 工具来帮助我们解决数据一致性的问题。 +- 如果日志过大，Redis 可以自动启用 rewrite 机制。即 Redis 以 append 模式不断的将修改数据写入到老的磁盘文件中，同时 Redis 还会创建一个新的文件用于记录此期间有哪些修改命令被执行。因此在进行 rewrite 切换时可以更好的保证数据安全性。 +- AOF 包含一个格式清晰、易于理解的日志文件用于记录所有的修改操作。事实上，我们也可以通过该文件完成数据的重建。因此 AOF 文件的内容非常容易被人读懂， 对文件进行分析（parse）也很轻松。 导出（export） AOF 文件也非常简单： 举个例子， 如果你不小心执行了 [FLUSHALL](http://redisdoc.com/server/flushall.html#flushall) 命令， 但只要 AOF 文件未被重写， 那么只要停止服务器， 移除 AOF 文件末尾的 FLUSHALL 命令， 并重启 Redis ， 就可以将数据集恢复到 FLUSHALL 执行之前的状态。 + +#### 劣势 + +- 对于相同数量的数据集而言，AOF 文件通常要大于 RDB 文件。恢复速度慢于 RDB。 +- 根据同步策略的不同，AOF 在运行效率上往往会慢于 RDB。总之，每秒同步策略的效率是比较高的，同步禁用策略的效率和 RDB 一样高效。 + +#### 总结 + +![](https://img.starfish.ink/redis/redis-aof-summary.png) + +- AOF 文件是一个只进行追加的日志文件 +- Redis 可以在 AOF 文件体积变得过大时，自动在后台对 AOF 进行重写 +- AOF 文件有序的保存了对数据库执行的所有写入操作，这些写入操作以 Redis 协议的格式保存，因此 AOF 文件的内容非常容易被人读懂，对文件进行分析也很轻松 +- 对于相同的数据集来说，AOF 文件的体积通常需要大于 RDB 文件的体积 +- 根据所使用的 fsync 策略，AOF 的速度可能会慢于 RDB + +**怎么从 RDB 持久化切换到 AOF 持久化** + +在 Redis 2.2 或以上版本，可以在不重启的情况下，从 RDB 切换到 AOF ： + +1. 为最新的 dump.rdb 文件创建一个备份。 + +2. 将备份放到一个安全的地方。 + +3. 执行以下两条命令： + + ```redis + redis-cli> CONFIG SET appendonly yes + + redis-cli> CONFIG SET save "" + ``` + +4. 确保命令执行之后，数据库的键的数量没有改变。 + +5. 确保写命令会被正确地追加到 AOF 文件的末尾。 + +步骤 3 执行的第一条命令开启了 AOF 功能： Redis 会阻塞直到初始 AOF 文件创建完成为止， 之后 Redis 会继续处理命令请求， 并开始将写入命令追加到 AOF 文件末尾。 + +步骤 3 执行的第二条命令用于关闭 RDB 功能。 这一步是可选的， 如果你愿意的话， 也可以同时使用 RDB 和 AOF 这两种持久化功能。 + +别忘了在 redis.conf 中打开 AOF 功能！ 否则的话， 服务器重启之后， 之前通过 CONFIG SET 设置的配置就会被遗忘， 程序会按原来的配置来启动服务器。 + + + +## Which one + +- RDB 持久化方式能够在指定的时间间隔能对你的数据进行快照存储 + +- AOF 持久化方式记录每次对服务器写的操作，当服务器重启的时候会重新执行这些命令来恢复原始的数据，AOF 命令以 Redis 协议追加保存每次写的操作到文件末尾。Redis 还能对 AOF 文件进行后台重写（**bgrewriteaof**），使得 AOF 文件的体积不至于过大 + +- 只做缓存：如果你只希望你的数据在服务器运行的时候存在，你也可以不使用任何持久化方式。 + +- 同时开启两种持久化方式 + + - 在这种情况下,当 Redis 重启的时候会优先载入 AOF 文件来恢复原始的数据，因为在通常情况下 AOF 文件保存的数据集要比 RDB 文件保存的数据集要完整。 + - RDB 的数据不实时，同时使用两者时服务器重启也只会找 AOF 文件。那要不要只使用AOF 呢？建议不要，因为 RDB 更适合用于备份数据库(AOF 在不断变化不好备份)，快速重启，而且不会有 AOF 可能潜在的 bug，留着作为一个万一的手段。 + +#### 性能建议 + +- 因为 RDB 文件只用作后备用途，建议只在 Slave上持久化 RDB 文件，而且只要 15 分钟备份一次就够了，只保留 `save 900 1` 这条规则。 +- 如果 Enalbe AOF，好处是在最恶劣情况下也只会丢失不超过两秒数据，启动脚本较简单只 load 自己的 AOF 文件就可以了。代价一是带来了持续的 IO，二是 AOF rewrite 的最后将 rewrite 过程中产生的新数据写到新文件造成的阻塞几乎是不可避免的。只要硬盘许可，应该尽量减少 AOF rewrite 的频率，AOF 重写的基础大小默认值 64M 太小了，可以设到 5G 以上。默认超过原大小 100% 大小时重写可以改到适当的数值。 +- 如果不 Enable AOF ，仅靠 Master-Slave Replication 实现高可用性也可以。能省掉一大笔 IO ，也减少了rewrite 时带来的系统波动。代价是如果 Master/Slav e同时宕掉，会丢失十几分钟的数据，启动脚本也要比较两个 Master/Slave 中的 RDB 文件，载入较新的那个。 + + + +#### Redis 4.0 混合持久化 + +Redis 4.0 中提出了一个**混合使用 AOF 日志和内存快照的方法**。简单来说，内存快照以一定的频率执行，在两次快照之间，使用 AOF 日志记录这期间的所有命令操作。也就是将 RDB 文件的内容和增量的 AOF 日志文件存在一起。这里的 AOF 日志不再是全量的日志，而是自持久化开始到持久化结束的这段时间发生的增量 AOF 日志。 + +同样我们执行 3 次 `set k1 v1`，然后手动瘦身 `bgrewriteaof` 后，查看 appendonly.aof 文件： + +![](https://img.starfish.ink/redis/redis-mix-persistence-file.png) + +这样做的好处是可以结合 rdb 和 aof 的优点，快速加载同时避免丢失过多的数据，缺点是 aof 里面的 rdb 部分就是压缩格式不再是 aof 格式，可读性差。 + +这样一来，快照不用很频繁地执行，这就避免了频繁 fork 对主线程的影响。而且，AOF 日志也只用记录两次快照间的操作，也就是说，不需要记录所有操作了，因此，就不会出现文件过大的情况了，也可以避免重写开销。 + +4.0 版本的混合持久化功能 **默认关闭**，我们可以通过 `aof-use-rdb-preamble` 配置参数控制该功能的启用。5.0 版本之后 **默认开启**。 + +如下图所示，两次快照中间时刻的修改，用 AOF 日志记录，等到第二次做全量快照时，就可以清空 AOF 日志，因为此时的修改都已经记录到快照中了，恢复时就不再用日志了。 + +![](https://img.starfish.ink/redis/redis-mix-persistence.png) + +这个方法既能享受到 RDB 文件快速恢复的好处，又能享受到 AOF 只记录操作命令的简单优势，有点“鱼和熊掌可以兼得”的意思。 + + + +## 参考 + +[ 1 ] :《Redis 核心技术与实战》 + +[ 2 ] :《Redis设计与实现》 + +[ 3 ] : https://www.wmyskxz.com/2020/03/13/redis-7-chi-jiu-hua-yi-wen-liao-jie/ + +[ 4 ] : https://redis.io/topics/persistence \ No newline at end of file diff --git a/docs/data-management/Redis/Redis-Sentinel.md b/docs/data-management/Redis/Redis-Sentinel.md new file mode 100644 index 0000000000..fcab77ecf2 --- /dev/null +++ b/docs/data-management/Redis/Redis-Sentinel.md @@ -0,0 +1,428 @@ +--- +title: Redis 哨兵模式 +date: 2021-10-08 +tags: + - Redis +categories: Redis +--- + +![](https://img.starfish.ink/redis/redis-sentinel-banner.png) + +> 我们知道 Reids 提供了主从模式的机制，来保证可用性，可是如果主库发生故障了，那就直接会影响到从库的同步，怎么办呢？ +> +> 所以，如果主库挂了，我们就需要运行一个新主库，比如说把一个从库切换为主库，把它当成主库。这就涉及到三个问题： +> +> 1. 主库真的挂了吗？ +> 2. 该选择哪个从库作为主库？ +> 3. 怎么把新主库的相关信息通知给从库和客户端呢？ +> +> 围绕这 3 个问题，我们来看下不需要人工干预就可以解决这三个问题的 Redis 哨兵。 +> +> ![](https://i02piccdn.sogoucdn.com/e9220e7b03e0103b) + +### 一、Redis Sentinel 哨兵 + +![](https://img.starfish.ink/redis/redis-sentinel.png) + +上图 展示了一个典型的哨兵架构图，它由两部分组成，哨兵节点和数据节点： + +- **哨兵节点：** 哨兵系统由一个或多个哨兵节点组成，哨兵节点是特殊的 Redis 节点，不存储数据； +- **数据节点：** 主节点和从节点都是数据节点； + +在复制的基础上，哨兵实现了 **自动化的故障恢复** 功能，下面是官方对于哨兵功能的描述： + +- **监控（Monitoring）：** 哨兵会不断地检查主节点和从节点是否运作正常。 + + 监控是指哨兵进程在运行时，周期性地给所有的主从库发送 PING 命令，检测它们是否仍然在线运行。如果从库没有在规定时间内响应哨兵的 PING 命令，哨兵就会把它标记为“下线状态”；同样，如果主库也没有在规定时间内响应哨兵的 PING 命令，哨兵就会判定主库下线，然后开始自动切换主库的流程。 + +- **通知（Notification）：** 当被监控的某个 Redis 服务器出现问题时， 哨兵可以通过 API 向管理员或者其他应用程序发送通知。 + + 在执行通知任务时，哨兵会把新主库的连接信息发给其他从库，让它们执行 `replicaof` 命令，和新主库建立连接，并进行数据复制。同时，哨兵会把新主库的连接信息通知给客户端，让它们把请求操作发到新主库上。 + +- **自动故障转移（Automatic failover）/ 选主：** 当 **主节点** 不能正常工作时，哨兵会开始 **自动故障转移操作**，它会将失效主节点的其中一个 **从节点升级为新的主节点**，并让其他从节点改为复制新的主节点。 + +- **配置提供者（Configuration provider）：** 客户端在初始化时，通过连接哨兵来获得当前 Redis 服务的主节点地址。 + + 当客户端试图连接失效的主服务器时， 集群也会向客户端返回新主服务器的地址，使得集群可以使用新主服务器代替失效服务器。 + +其中，监控和自动故障转移功能，使得哨兵可以及时发现主节点故障并完成转移。而配置提供者和通知功能，则需要在与客户端的交互中才能体现。 + + + +### 二、 Hello Wolrd + +#### 2.1 部署主从节点 + +哨兵系统中的主从节点，与普通的主从节点配置是一样的，并不需要做任何额外配置。 + +下面分别是主节点（port=6379）和 2 个从节点（port=6380、6381）的配置文件： + +```bash +#redis.conf master +port 6379 +daemonize yes +logfile "6379.log" +dbfilename "dump-6379.rdb" + +#redis_6380.conf +port 6380 +daemonize yes +logfile "6380.log" +dbfilename "dump-6380.rdb" +replicaof 127.0.0.1 6379 + +#redis_6381.conf +port 6381 +daemonize yes +logfile "6381.log" +dbfilename "dump-6381.rdb" +replicaof 127.0.0.1 6379 +``` + +然后我们可以执行 `redis-server ` 来根据配置文件启动不同的 Redis 实例，依次启动主从节点： + +```bash +redis-server redis.conf +redis-server redis_6380.conf +redis-server redis_6381.conf +``` + +节点启动后，我们执行 `redis-cli` 默认连接到我们端口为 `6379` 的主节点执行 `info Replication` 检查一下主从状态是否正常：(可以看到下方正确地显示了两个从节点) + +```bash +127.0.0.1:6379> info replication +# Replication +role:master +connected_slaves:2 +slave0:ip=127.0.0.1,port=6380,state=online,offset=154,lag=1 +slave1:ip=127.0.0.1,port=6381,state=online,offset=140,lag=1 +master_replid:52a58d69125881d3af366d0559439377a70ae879 +master_replid2:0000000000000000000000000000000000000000 +master_repl_offset:154 +second_repl_offset:-1 +repl_backlog_active:1 +repl_backlog_size:1048576 +repl_backlog_first_byte_offset:1 +repl_backlog_histlen:154 +``` + + + +#### 2.2 部署哨兵节点 + +按照上面同样的方法，我们给哨兵节点也创建三个配置文件。*(哨兵节点本质上是特殊的 Redis 节点，所以配置几乎没什么差别，只是在端口上做区分就好，每个哨兵只需要配置监控主节点，就可以自动发现其他的哨兵节点和从节点)* + +```bash +# redis-sentinel-26379.conf +port 26379 +daemonize yes +logfile "26379.log" +sentinel monitor mymaster 127.0.0.1 6379 2 + +# redis-sentinel-26380.conf +port 26380 +daemonize yes +logfile "26380.log" +sentinel monitor mymaster 127.0.0.1 6379 2 + +# redis-sentinel-26381.conf +port 26381 +daemonize yes +logfile "26381.log" +sentinel monitor mymaster 127.0.0.1 6379 2 +``` + +其中，`sentinel monitor mymaster 127.0.0.1 6379 2` 配置的含义是：该哨兵节点监控 `127.0.0.1:6379` 这个主节点，该主节点的名称是 `mymaster`，最后的 `2` 的含义与主节点的故障判定有关：至少需要 `2` 个哨兵节点同意，才能判定主节点故障并进行故障转移。 + +启动 3 个哨兵节点： + +```bash +redis-sentinel redis-sentinel-26379.conf +redis-sentinel redis-sentinel-26380.conf +redis-server redis-sentinel-26381.conf --sentinel #等同于 redis-sentinel redis-sentinel-26381.conf +``` + +使用 `redis-cil` 工具连接哨兵节点，并执行 `info Sentinel` 命令来查看是否已经在监视主节点了： + +```bash +redis-cli -p 26380 +127.0.0.1:26380> info sentinel +# Sentinel +sentinel_masters:1 +sentinel_tilt:0 +sentinel_running_scripts:0 +sentinel_scripts_queue_length:0 +sentinel_simulate_failure_flags:0 +master0:name=mymaster,status=ok,address=127.0.0.1:6379,slaves=2,sentinels=3 +``` + +此时你打开刚才写好的哨兵配置文件，你还会发现出现了一些变化。 + +#### 2.3 演示故障转移 + +我们先看下我们启动的 redis 进程，3 个数据节点，3 个哨兵节点 + +![](https://img.starfish.ink/redis/redis-sentinel-ps.png) + +使用 `kill` 命令来杀掉主节点，**同时** 在哨兵节点中执行 `info Sentinel` 命令来观察故障节点的过程： + +如果 **刚杀掉瞬间** 在哨兵节点中执行 `info` 命令来查看，会发现主节点还没有切换过来，因为哨兵发现主节点故障并转移需要一段时间： + +```bash +# 第一时间查看哨兵节点发现并未转移，还在 6379 端口 +127.0.0.1:26379> info Sentinel +# Sentinel +sentinel_masters:1 +sentinel_tilt:0 +sentinel_running_scripts:0 +sentinel_scripts_queue_length:0 +sentinel_simulate_failure_flags:0 +master0:name=mymaster,status=ok,address=127.0.0.1:6379,slaves=2,sentinels=3 +``` + +一段时间之后你再执行 `info` 命令，查看，你就会发现主节点已经切换成了 `6381` 端口的从节点： + +```bash +# 过一段时间之后在执行，发现已经切换了 6381 端口 +127.0.0.1:26379> info Sentinel +# Sentinel +sentinel_masters:1 +sentinel_tilt:0 +sentinel_running_scripts:0 +sentinel_scripts_queue_length:0 +sentinel_simulate_failure_flags:0 +master0:name=mymaster,status=ok,address=127.0.0.1:6381,slaves=2,sentinels=3 +``` + +但同时还可以发现，**哨兵节点认为新的主节点仍然有两个从节点** *(上方 slaves=2)*，这是因为哨兵在将 `6381` 切换成主节点的同时，将 `6379` 节点置为其从节点。虽然 `6379` 从节点已经挂掉，但是由于 **哨兵并不会对从节点进行客观下线**，因此认为该从节点一直存在。当 `6379` 节点重新启动后，会自动变成 `6381` 节点的从节点。 + +另外，在故障转移的阶段，哨兵和主从节点的配置文件都会被改写： + +- **对于主从节点：** 主要是 `slaveof` 配置的变化，新的主节点没有了 `slaveof` 配置，其从节点则 `slaveof` 新的主节点。 +- **对于哨兵节点：** 除了主从节点信息的变化，纪元(epoch) *(记录当前集群状态的参数)* 也会变化，纪元相关的参数都 +1 了。 + + + +### 三、哨兵机制的工作流程 + +其实哨兵主要负责的就是三个任务：**监控**、**选主**和**通知**。 + +在监控和选主过程中，哨兵都需要做一些决策，比如 + +- 在监控任务中，哨兵需要判断主库、从库是否处于下线状态 +- 在选主任务中，哨兵也要决定选择哪个从库实例作为主库 + +这就引出了两个概念，“主观下线”和“客观下线” + +#### 3.1 主观下线和客观下线 + +我先解释下什么是“主观下线”。 + +**哨兵进程会使用 PING 命令检测它自己和主、从库的网络连接情况，用来判断实例的状态**。如果哨兵发现主库或从库对 PING 命令的响应超时了，那么，哨兵就会先把它标记为“主观下线”。 + +如果检测的是从库，那么，哨兵简单地把它标记为“主观下线”就行了，因为从库的下线影响一般不太大，集群的对外服务不会间断。 + +但是，如果检测的是主库，那么，哨兵还不能简单地把它标记为“主观下线”，开启主从切换。因为很有可能存在这么一个情况：那就是哨兵误判了，其实主库并没有故障。可是，一旦启动了主从切换，后续的选主和通知操作都会带来额外的计算和通信开销。 + +为了避免这些不必要的开销，我们要特别注意误判的情况。 + +误判一般会发生在**集群网络压力较大、网络拥塞，或者是主库本身压力较大**的情况下。一旦哨兵判断主库下线了，就会开始选择新主库，并让从库和新主库进行数据同步，这个过程本身就会有开销，例如，哨兵要花时间选出新主库，从库也需要花时间和新主库同步。 + +那怎么减少误判呢？ + +在日常生活中，当我们要对一些重要的事情做判断的时候，经常会和家人或朋友一起商量一下，然后再做决定。 + +哨兵机制也是类似的，它通常会采用多实例组成的集群模式进行部署，这也被称为**哨兵集群**。引入多个哨兵实例一起来判断，就可以避免单个哨兵因为自身网络状况不好，而误判主库下线的情况。同时，多个哨兵的网络同时不稳定的概率较小，由它们一起做决策，误判率也能降低。 + +在判断主库是否下线时，不能由一个哨兵说了算，只有大多数的哨兵实例，都判断主库已经“主观下线”了，主库才会被标记为“**客观下线**”，这个叫法也是表明主库下线成为一个客观事实了。这个判断原则就是：少数服从多数。同时，这会进一步触发哨兵开始主从切换流程。 + +> **需要特别注意的是，客观下线是主节点才有的概念；如果从节点和哨兵节点发生故障，被哨兵主观下线后，不会再有后续的客观下线和故障转移操作。** + +#### 3.2 选举领导者哨兵节点 + +当主节点被判断客观下线以后，各个哨兵节点会进行协商，选举出一个领导者哨兵节点，并由该领导者节点对其进行故障转移操作。 + +监视该主节点的所有哨兵都有可能被选为领导者，选举使用的算法是 Raft 算法；Raft 算法的基本思路是先到先得：即在一轮选举中，哨兵 A 向 B 发送成为领导者的申请，如果 B 没有同意过其他哨兵，则会同意 A 成为领导者。(Raft 算法：https://raft.github.io/) + +#### 3.3 故障转移 + +接着，选举出的领导者哨兵，开始故障转移操作，大概分 3 步： + +1. 第一步要做的就是在已下线主服务器属下的所有从服务器中，挑选出一个状态良好、数据完整的从服务器 +2. 第二步，更新主从状态，向选出的从服务器发送 `slaveof no one` 命令，将这个从服务器转换为主服务器，并通过 `slaveof` 命令让其他节点成为其从节点 +3. 第三步将已下线的主节点设置为从节点 + +细说下第一步的选主过程 + +一般来说，我把哨兵选择新主库的过程称为“**筛选 + 打分**”。 + +筛选就是先过滤掉不健康的从节点，那些被标记为主观下线、已断线、或者最后一次回复 PING 命令的时间大于五秒钟的从服务器都会被 **淘汰**。 + +打分就是按 Redis 给定的三个规则，给剩下的从库逐个打分，将得分最高的从库选为新主库，这个规则分别是： + +- 优先级最高的从库得分最高 + + 用户可以通过 `slave-priority` 配置项，给不同的从库设置不同优先级。比如，你有两个从库，它们的内存大小不一样，你可以手动给内存大的实例设置一个高优先级。在选主时，哨兵会给优先级高的从库打高分，如果有一个从库优先级最高，那么它就是新主库了。如果从库的优先级都一样，那么哨兵开始第二轮打分。 + +- 和旧主库同步程度最接近的从库得分高 + + 从库的 `slave_repl_offset` 需要最接近 `master_repl_offset`，即得分最高。 + +- ID 号小的从库得分高 + + 每个实例都会有一个 runid，这个 ID 就类似于这里的从库的编号。目前，Redis 在选主库时，有一个默认的规定：在优先级和复制进度都相同的情况下，ID 号最小的从库得分最高，会被选为新主库。 + +![](https://img.starfish.ink/redis/redis-sentinel-select-master.jpg) + +> ##### **哨兵模式是否会出现脑裂问题？** +> +> - **哨兵模式下存在脑裂风险。** +> - 当网络分区或通信异常时，可能导致旧主节点未完全下线，新的主节点被选出，导致两个主节点同时存在，形成**脑裂**问题。 +> +> #### **解决方法：** +> +> 1. **主节点心跳检测**：通过哨兵的客观下线判断，多数哨兵节点确认主节点下线，减少误判。 +> 2. **客户端重连机制**：客户端连接断开后，需要重新通过哨兵获取正确的主节点地址。 +> 3. **配置防止脑裂**：`quorum` 参数设置哨兵节点的投票数量，避免少数节点误判主节点下线。 + +### 四、哨兵集群的原理 + +实际上，一旦多个实例组成了**哨兵集群**，即使有哨兵实例出现故障挂掉了，其他哨兵还能继续协作完成主从库切换的工作，包括判定主库是不是处于下线状态，选择新主库，以及通知从库和客户端。 + +认真看到这里的话，应该会有个疑问，我们在 hello world 环节中只是在哨兵节点加了一条配置 + +```bash +sentinel monitor mymaster 127.0.0.1 6379 2 +``` + +怎么就能组成一个哨兵集群呢？ + +一套合理的监控机制是哨兵节点判定节点不可达的重要保证，Redis 哨兵通过**三个定时监控任务**完成对各个节点发现和监控： + +1. **每隔 10 秒，每个哨兵节点会向主节点和从节点发送 info 命令获取最新的拓扑结构** + + 这个定时任务的作用具体可以表现在三个方面： + + - 通过向主节点执行 *info* 命令，获取从节点的信息，这也是为什么哨兵节点不需要显式配置监控从节点。 + - 当有新的从节点加入时都可以立刻感知出来。 + - 节点不可达或者故障转移后，可以通过 info 命令实时更新节点拓扑信息 + +2. **每隔 2 秒，每个哨兵节点会向 Redis 数据节点的 \__sentinel\_\_:hello 频道上发送该哨兵节点对于主节点的判断以及当前哨兵节点的信息，同时每个哨兵节点也会订阅该频道，来了解其他哨兵节点以及它们对主节点的判断。** + + 这个定时任务可以完成以下两个工作： + + - 发现新的哨兵节点：通过订阅主节点的 `__sentinel__：hello` 了解其他的哨兵节点信息，如果是新加入的哨兵节点，将该哨兵节点信息保存起来，并与该哨兵节点创建连接。 + - 哨兵节点之间交换主节点的状态，作为后面客观下线以及领导者选举的依据。 + +3. **每隔 1 秒，每个哨兵节点会向主节点、从节点、其余哨兵节点发送一条 ping 命令做一次心跳检测，来确认这些节点当前是否可达。** + + 通过这个定时任务，哨兵节点对主节点、从节点、其余哨兵节点都建立起连接，实现了对每个节点的监控，这个定时任务是节点失败判定的重要依据 + + + +#### 4.1 基于 pub/sub 机制的哨兵集群组成 + +哨兵实例之间可以相互发现，要归功于 Redis 提供的 pub/sub 机制，也就是 发布/订阅 机制。 + +哨兵只要和主库建立起了连接，就可以在主库上发布消息了，比如说发布它自己的连接信息（IP 和端口）。同时，它也可以从主库上订阅消息，获得其他哨兵发布的连接信息。当多个哨兵实例都在主库上做了发布和订阅操作后，它们之间就能知道彼此的 IP 地址和端口。 + +在主从集群中，主库上有一个名为 "**\_ \_sentinel\_ \_:hello**" 的频道，不同哨兵就是通过它来相互发现，实现互相通信的。 + +举个例子，具体说明一下。 + +在下图中，哨兵 sentinel_26379 把自己的 IP（127.0.0.1）和端口（26379）发布到频道上，哨兵 26380 和 26381 订阅了该频道。那么此时，其他哨兵就可以从这个频道直接获取哨兵 sentinel_26379 的 IP 地址和端口号。通过这个方式，各个哨兵之间就可以建立网络连接，哨兵集群就形成了。它们相互间可以通过网络连接进行通信，比如说对主库有没有下线这件事儿进行判断和协商。 + +![](https://img.starfish.ink/redis/redis-sentinel-cluster.png) + +#### 4.2 哨兵和从库的连接 + +哨兵除了彼此之间建立起连接形成集群外，还需要和从库建立连接。这是因为，在哨兵的监控任务中，它需要对主从库都进行心跳判断，而且在主从库切换完成后，它还需要通知从库，让它们和新主库进行同步。 + +**哨兵是如何知道从库的 IP 地址和端口的呢？** + +**这是由哨兵向主库发送 INFO 命令来完成的。** + +就像下图所示，哨兵 sentinel_26380 给主库发送 INFO 命令，主库接受到这个命令后，就会把从库列表返回给哨兵。接着，哨兵就可以根据从库列表中的连接信息，和每个从库建立连接，并在这个连接上持续地对从库进行监控。senetinel_26379 和 senetinel_26381 可以通过相同的方法和从库建立连接。 + +![](https://img.starfish.ink/redis/redis-sentinel-slave.png) + +#### 4.3 哨兵和客户端的连接 + +但是，哨兵不能只和主、从库连接。因为，主从库切换后，客户端也需要知道新主库的连接信息，才能向新主库发送请求操作。所以，哨兵还需要完成把新主库的信息告诉客户端这个任务。 + +在实际使用哨兵时，我们有时会遇到这样的问题：如何在客户端通过监控了解哨兵进行主从切换的过程呢？比如说，主从切换进行到哪一步了？这其实就是要求，客户端能够获取到哨兵集群在监控、选主、切换这个过程中发生的各种事件。此时，我们仍然可以**依赖 pub/sub 机制**，来帮助我们完成哨兵和客户端间的信息同步。 + +从本质上说，哨兵就是一个运行在特定模式下的 Redis 实例，只不过它并不服务请求操作，只是完成监控、选主和通知的任务。所以，每个哨兵实例也提供 pub/sub 机制，客户端可以从哨兵订阅消息。 + +哨兵提供的消息订阅频道有很多，不同频道包含了主从库切换过程中的不同关键事件。（这里就不一一列出了） + +知道了这些频道之后，你就可以让客户端从哨兵这里订阅消息了。具体的操作步骤是，客户端读取哨兵的配置文件后，可以获得哨兵的地址和端口，和哨兵建立网络连接。然后，我们可以在客户端执行订阅命令，来获取不同的事件消息。 + +举个例子，你可以执行如下命令，来订阅“所有实例进入客观下线状态的事件”： + +``` +SUBSCRIBE +odown +``` + +当然，你也可以执行如下命令，订阅所有的事件： + +``` +PSUBSCRIBE * +``` + + + +### 五、小结 + +Redis 哨兵是 Redis 的高可用实现方案：故障发现、故障自动转移、配置中心、客户端通知。 + +#### 5.1 哨兵机制其实就有三大功能： + +- 监控：监控主库运行状态，并判断主库是否客观下线； + +- 选主：在主库客观下线后，选取新主库； +- 通知：选出新主库后，通知从库和客户端。 + +#### 5.2 一个哨兵，实际上可以监控多个主节点，通过配置多条 sentinel monitor 即可实现。 + +#### 5.3 哨兵集群的关键机制： + +- 哨兵集群是基于 pub/sub 机制组成的 +- 基于 INFO 命令的从库列表，这可以帮助哨兵和从库建立连接 +- 基于哨兵自身的 pub/sub 功能，这实现了客户端和哨兵之间的事件通知 + + + +`Sentinel` 与 `Redis` **主节点** 和 **从节点** 交互的命令，主要包括： + +| 命令 | 作 用 | +| --------- | ------------------------------------------------------------ | +| PING | `Sentinel` 向 `Redis` 节点发送 `PING` 命令，检查节点的状态 | +| INFO | `Sentinel` 向 `Redis` 节点发送 `INFO` 命令，获取它的 **从节点信息** | +| PUBLISH | `Sentinel` 向其监控的 `Redis` 节点 `__sentinel__:hello` 这个 `channel` 发布 **自己的信息** 及 **主节点** 相关的配置 | +| SUBSCRIBE | `Sentinel` 通过订阅 `Redis` **主节点** 和 **从节点** 的 `__sentinel__:hello` 这个 `channnel`，获取正在监控相同服务的其他 `Sentinel` 节点 | + +`Sentinel` 与 `Sentinel` 交互的命令，主要包括： + +| 命令 | 作 用 | +| ------------------------------- | ------------------------------------------------------------ | +| PING | `Sentinel` 向其他 `Sentinel` 节点发送 `PING` 命令，检查节点的状态 | +| SENTINEL:is-master-down-by-addr | 和其他 `Sentinel` 协商 **主节点** 的状态，如果 **主节点** 处于 `SDOWN` 状态，则投票自动选出新的 **主节点** | + +#### 5.4 建议 + +- 尽可能在不同物理机上部署 Redis 哨兵所有节点 + +- Redis 哨兵中的哨兵节点个数应该为大于等于 3 且最好为奇数 + + 推荐奇数个节点，主要是从成本上考虑，因为，集群中，半数以上节点认为主节点故障了，才会选举新的节点。这样的话奇数个节点和偶数个节点允许宕机的节点数就是一样的，比如 3 个节点和 4 个节点都只允许宕机一台，那么为什么要搞 4 个节点去浪费服务资源呢？但是 4 个节点的性能和容量肯定是更高的哈。 + + + + +### References + +- 《Redis 开发与运维》 +- 《Redis 核心技术与实战》 +- https://redis.io/topics/sentinel +- https://www.cnblogs.com/kismetv/p/9609938.html \ No newline at end of file diff --git a/docs/data-management/Redis/Redis-Transaction.md b/docs/data-management/Redis/Redis-Transaction.md new file mode 100644 index 0000000000..97c95bcb8c --- /dev/null +++ b/docs/data-management/Redis/Redis-Transaction.md @@ -0,0 +1,302 @@ +--- +title: Redis 事务 +date: 2021-10-09 +tags: + - Redis +categories: Redis +--- + +> 文章收录在 GitHub [JavaKeeper](https://github.com/Jstarfish/JavaKeeper) ，N线互联网开发必备技能兵器谱 + +![](https://img.starfish.ink/redis/redis-reansaction-banner.jpg) + +> 假设现在有这样一个业务，用户获取的某些数据来自第三方接口信息，为避免频繁请求第三方接口，我们往往会加一层缓存，缓存肯定要有时效性，假设我们要存储的结构是 hash（没有String的'**SET anotherkey "will expire in a minute" EX 60**'这种原子操作），我们既要批量去放入缓存，又要保证每个 key 都加上过期时间（以防 key 永不过期），这时候事务操作是个比较好的选择 + +为了确保连续多个操作的原子性，我们常用的数据库都会有事务的支持，Redis 也不例外。但它又和关系型数据库不太一样。 + +每个事务的操作都有 begin、commit 和 rollback，begin 指示事务的开始，commit 指示事务的提交，rollback 指示事务的回滚。它大致的形式如下 + +```java +begin(); +try { + command1(); + command2(); + .... + commit(); +} catch(Exception e) { + rollback(); +} +``` + +Redis 在形式上看起来也差不多，分为三个阶段 + +1. 开启事务（multi） +2. 命令入队（业务操作） +3. 执行事务（exec）或取消事务（discard） + +```sh +> multi +OK +> incr star +QUEUED +> incr star +QUEUED +> exec +(integer) 1 +(integer) 2 +``` + +上面的指令演示了一个完整的事务过程，所有的指令在 exec 之前不执行，而是缓存在服务器的一个事务队列中，服务器一旦收到 exec 指令，才开始执行整个事务队列，执行完毕后一次性返回所有指令的运行结果。 + + + +Redis 事务可以一次执行多个命令，本质是一组命令的集合。一个事务中的所有命令都会序列化，按顺序地串行化执行而不会被其它命令插入，不许加塞。 + +可以保证一个队列中，一次性、顺序性、排他性的执行一系列命令（Redis 事务的主要作用其实就是串联多个命令防止别的命令插队） + +官方文档是这么说的 + +> 事务可以一次执行多个命令， 并且带有以下两个重要的保证： +> +> - 事务是一个单独的隔离操作：事务中的所有命令都会序列化、按顺序地执行。事务在执行的过程中，不会被其他客户端发送来的命令请求所打断。 +> - 事务是一个原子操作：事务中的命令要么全部被执行，要么全部都不执行 + +这个原子操作，和关系型 DB 的原子性不太一样，它不能完全保证原子性，后边会介绍。 + + + +### Redis 事务的几个命令 + +| 命令 | 描述 | +| ------- | ------------------------------------------------------------ | +| MULTI | 标记一个事务块的开始 | +| EXEC | 执行所有事务块内的命令 | +| DISCARD | 取消事务，放弃执行事务块内的所有命令 | +| WATCH | 监视一个（或多个）key，如果在事务执行之前这个（或多个）key被其他命令所改动，那么事务将被打断 | +| UNWATCH | 取消 WATCH 命令对所有 keys 的监视 | + +[MULTI](http://redisdoc.com/transaction/multi.html#multi) 命令用于开启一个事务，它总是返回 OK 。 + +MULTI 执行之后， 客户端可以继续向服务器发送任意多条命令， 这些命令不会立即被执行， 而是被放到一个队列中， 当 [EXEC](http://redisdoc.com/transaction/exec.html#exec) 命令被调用时， 所有队列中的命令才会被执行。 + +另一方面， 通过调用 [DISCARD](http://redisdoc.com/transaction/discard.html#discard) ， 客户端可以清空事务队列， 并放弃执行事务。 + +废话不多说，直接操作起来看结果更好理解~ + +### 一帆风顺 + +**正常执行**（可以批处理，挺爽，每条操作成功的话都会各取所需，互不影响） + +![](https://img.starfish.ink/redis/redis-transaction-case1.png) + +**放弃事务**（discard 操作表示放弃事务，之前的操作都不算数） + +![](https://img.starfish.ink/redis/redis-transaction-case2.png) + + + +思考个问题：假设我们有个有过期时间的 key，在事务操作中 key 失效了，那执行 exec 的时候会成功吗？ + + + +### 事务中的错误 + +上边规规矩矩的操作，看着还挺好，可是**事务是为解决数据安全操作提出的**，我们用 Redis 事务的时候，可能会遇上以下两种错误： + +- 事务在执行 `EXEC` 之前，入队的命令可能会出错。比如说，命令可能会产生语法错误（参数数量错误，参数名错误等等），或者其他更严重的错误，比如内存不足（如果服务器使用 `maxmemory` 设置了最大内存限制的话）。 +- 命令可能在 `EXEC` 调用之后失败。举个例子，事务中的命令可能处理了错误类型的键，比如将列表命令用在了字符串键上面，诸如此类。 + +Redis 针对如上两种错误采用了不同的处理策略，对于发生在 `EXEC` 执行之前的错误，服务器会对命令入队失败的情况进行记录，并在客户端调用 `EXEC` 命令时，拒绝执行并自动放弃这个事务（Redis 2.6.5 之前的做法是检查命令入队所得的返回值：如果命令入队时返回 QUEUED ，那么入队成功；否则，就是入队失败） + +对于那些在 `EXEC` 命令执行之后所产生的错误， 并没有对它们进行特别处理： 即使事务中有某个/某些命令在执行时产生了错误， 事务中的其他命令仍然会继续执行。 + +**全体连坐**（某一条操作记录报错的话，exec 后所有操作都不会成功） + +![](https://img.starfish.ink/redis/redis-transaction-case3.png) + +**冤头债主**（示例中 k1 被设置为 String 类型，decr k1 可以放入操作队列中，因为只有在执行的时候才可以判断出语句错误，其他正确的会被正常执行） + +![](https://img.starfish.ink/redis/redis-transaction-case4.png) + + + +### 为什么 Redis 不支持回滚 + +如果你有使用关系式数据库的经验，那么 “Redis 在事务失败时不进行回滚，而是继续执行余下的命令”这种做法可能会让你觉得有点奇怪。 + +以下是官方的*自夸*： + +> - Redis 命令只会因为错误的语法而失败（并且这些问题不能在入队时发现），或是命令用在了错误类型的键上面：这也就是说，从实用性的角度来说，失败的命令是由编程错误造成的，而这些错误应该在开发的过程中被发现，而不应该出现在生产环境中。 +> - 因为不需要对回滚进行支持，所以 Redis 的内部可以保持简单且快速。 +> +> 有种观点认为 Redis 处理事务的做法会产生 bug ， 然而需要注意的是， 在通常情况下， 回滚并不能解决编程错误带来的问题。 举个例子， 如果你本来想通过 `INCR` 命令将键的值加上 1 ， 却不小心加上了 2 ， 又或者对错误类型的键执行了 `INCR` ， 回滚是没有办法处理这些情况的。 +> +> 鉴于没有任何机制能避免程序员自己造成的错误， 并且这类错误通常不会在生产环境中出现， 所以 Redis 选择了更简单、更快速的无回滚方式来处理事务。 + +![](https://i04piccdn.sogoucdn.com/d2ab1c04cc178f61) + + + +### 带 Watch 的事务 + +`WATCH` 命令用于在事务开始之前监视任意数量的键： 当调用 EXEC 命令执行事务时， 如果任意一个被监视的键已经被其他客户端修改了， 那么整个事务将被打断，不再执行， 直接返回失败。 + +WATCH 命令可以被调用多次。 对键的监视从 WATCH 执行之后开始生效， 直到调用 EXEC 为止。 + +用户还可以在单个 WATCH 命令中监视任意多个键， 就像这样： + +``` +redis> WATCH key1 key2 key3 +OK +``` + +**当 `EXEC` 被调用时， 不管事务是否成功执行， 对所有键的监视都会被取消**。另外， 当客户端断开连接时， 该客户端对键的监视也会被取消。 + +我们看个简单的例子，用 watch 监控我的账号余额（一周100零花钱的我），正常消费 + +![](https://img.starfish.ink/redis/redis-transaction-watch1.png) + +但这个卡，还绑定了我媳妇的支付宝，如果在我消费的时候，她也消费了，会怎么样呢？ + +犯困的我去楼下 711 买了包烟，买了瓶水，这时候我媳妇在超市直接刷了 100，此时余额不足的我还在挑口香糖来着，，， + +![](https://img.starfish.ink/redis/redis-transaction-watch2.png) + +这时候我去结账，发现刷卡失败（事务中断），尴尬的一批 + +![](https://img.starfish.ink/redis/redis-transaction-watch3.png) + + + +你可能没看明白 watch 有啥用，我们再来看下，如果还是同样的场景，我们没有 `watch balance` ，事务不会失败，储蓄卡成负数，是不不太符合业务呢 + +> 当然，这里也会出现只要你媳妇刷了你的卡，就没办法刷成功的问题，这时候可以先查下余额，重新开启事务继续刷 + +![](https://img.starfish.ink/redis/redis-transaction-watch4.png) + + + +使用无参数的 `UNWATCH` 命令可以手动取消对所有键的监视。 对于一些需要改动多个键的事务，有时候程序需要同时对多个键进行加锁， 然后检查这些键的当前值是否符合程序的要求。 当值达不到要求时， 就可以使用 `UNWATCH` 命令来取消目前对键的监视， 中途放弃这个事务， 并等待事务的下次尝试。 + + + +**watch指令，类似乐观锁**，事务提交时，如果 key 的值已被别的客户端改变，比如某个 list 已被别的客户端push/pop 过了，整个事务队列都不会被执行。（当然也可以用 Redis 实现分布式锁来保证安全性，属于悲观锁） + +通过 watch 命令在事务执行之前监控了多个 keys，倘若在 watch 之后有任何 key 的值发生变化，exec 命令执行的事务都将被放弃，同时返回 Null 应答以通知调用者事务执行失败。 + +> **悲观锁** +> +> 悲观锁(Pessimistic Lock)，顾名思义，就是很悲观，每次去拿数据的时候都认为别人会修改，所以每次在拿数据的时候都会上锁，这样别人想拿这个数据就会 block 直到它拿到锁。传统的关系型数据库里边就用到了很多这种锁机制，比如行锁，表锁等，读锁，写锁等，都是在做操作之前先上锁 +> +> **乐观锁** +> +> 乐观锁(Optimistic Lock)，顾名思义，就是很乐观，每次去拿数据的时候都认为别人不会修改，所以不会上锁，但是在更新的时候会判断一下在此期间别人有没有去更新这个数据，可以使用版本号等机制。乐观锁适用于多读的应用类型，这样可以提高吞吐量。 +> +> 乐观锁策略：提交版本必须大于记录当前版本才能执行更新 + + + +### [WATCH 命令的实现原理](https://redisbook.readthedocs.io/en/latest/feature/transaction.html#id3 "Redis设计与实现") + +在代表数据库的 `server.h/redisDb` 结构类型中， 都保存了一个 `watched_keys` 字典， 字典的键是这个数据库被监视的键， 而字典的值是一个链表， 链表中保存了所有监视这个键的客户端，如下图。 + +![Redis设计与实现](https://img.starfish.ink/redis/redis-watch-key.png) + +```c +typedef struct redisDb { + dict *dict; /* The keyspace for this DB */ + dict *expires; /* Timeout of keys with a timeout set */ + dict *blocking_keys; /* Keys with clients waiting for data (BLPOP)*/ + dict *ready_keys; /* Blocked keys that received a PUSH */ + dict *watched_keys; /* WATCHED keys for MULTI/EXEC CAS */ + int id; /* Database ID */ + long long avg_ttl; /* Average TTL, just for stats */ + unsigned long expires_cursor; /* Cursor of the active expire cycle. */ + list *defrag_later; /* List of key names to attempt to defrag one by one, gradually. */ +} redisDb; + +list *watched_keys; /* Keys WATCHED for MULTI/EXEC CAS */ +``` + +`WATCH` 命令的作用， 就是将当前客户端和要监视的键在 `watched_keys` 中进行关联。 + +举个例子， 如果当前客户端为 `client99` ， 那么当客户端执行 `WATCH key2 key3` 时， 前面展示的 `watched_keys` 将被修改成这个样子： + +![图：Redis设计与实现](https://img.starfish.ink/redis/redis-watch-client99.png) + +通过 `watched_keys` 字典， 如果程序想检查某个键是否被监视， 那么它只要检查字典中是否存在这个键即可； 如果程序要获取监视某个键的所有客户端， 那么只要取出键的值（一个链表）， 然后对链表进行遍历即可。 + + + +在任何对数据库键空间（key space）进行修改的命令成功执行之后 （比如 FLUSHDB、SET 、DEL、LPUSH、 SADD，诸如此类）， `multi.c/touchWatchedKey` 函数都会被调用 —— 它会去 `watched_keys` 字典， 看是否有客户端在监视已经被命令修改的键， 如果有的话， 程序将所有监视这个/这些被修改键的客户端的 `REDIS_DIRTY_CAS` 选项打开： + +![图：Redis设计与实现](https://img.starfish.ink/redis/redis-transaction-client-cut.png) + +```c +void multiCommand(client *c) { + // 不能在事务中嵌套事务 + if (c->flags & CLIENT_MULTI) { + addReplyError(c,"MULTI calls can not be nested"); + return; + } + // 打开事务 FLAG + c->flags |= CLIENT_MULTI; + addReply(c,shared.ok); +} + +/* "Touch" a key, so that if this key is being WATCHed by some client the + * next EXEC will fail. */ +void touchWatchedKey(redisDb *db, robj *key) { + list *clients; + listIter li; + listNode *ln; + // 字典为空，没有任何键被监视 + if (dictSize(db->watched_keys) == 0) return; + // 获取所有监视这个键的客户端 + clients = dictFetchValue(db->watched_keys, key); + if (!clients) return; + + // 遍历所有客户端，打开他们的 CLIENT_DIRTY_CAS 标识 + listRewind(clients,&li); + while((ln = listNext(&li))) { + client *c = listNodeValue(ln); + + c->flags |= CLIENT_DIRTY_CAS; + } +} +``` + +当客户端发送 EXEC 命令、触发事务执行时， 服务器会对客户端的状态进行检查： + +- 如果客户端的 `CLIENT_DIRTY_CAS` 选项已经被打开，那么说明被客户端监视的键至少有一个已经被修改了，事务的安全性已经被破坏。服务器会放弃执行这个事务，直接向客户端返回空回复，表示事务执行失败。 +- 如果 `CLIENT_DIRTY_CAS` 选项没有被打开，那么说明所有监视键都安全，服务器正式执行事务。 + + + +### 小总结： + +#### 3 个阶段 + +- 开启：以 MULTI 开始一个事务 +- 入队：将多个命令入队到事务中，接到这些命令并不会立即执行，而是放到等待执行的事务队列里面 +- 执行：由 EXEC 命令触发事务 + +#### 3 个特性 + +- 单独的隔离操作：事务中的所有命令都会序列化、按顺序地执行。事务在执行的过程中，不会被其他客户端发送来的命令请求所打断。 +- **没有隔离级别的概念**：队列中的命令没有提交之前都不会实际的被执行，因为事务提交前任何指令都不会被实际执行，也就不存在”事务内的查询要看到事务里的更新，在事务外查询不能看到”这个让人万分头痛的问题 +- 不保证原子性：Redis 同一个事务中如果有一条命令执行失败，其后的命令仍然会被执行，没有回滚 + +在传统的关系式数据库中，常常用 ACID 性质来检验事务功能的安全性。Redis 事务保证了其中的一致性（C）和隔离性（I），但并不保证原子性（A）和持久性（D）。 + + + +**最后** + +Redis 事务在发送每个指令到事务缓存队列时都要经过一次网络读写，当一个事务内部的指令较多时，需要的网络 IO 时间也会线性增长。所以通常 Redis 的客户端在执行事务时都会结合 pipeline 一起使用，这样可以将多次 IO 操作压缩为单次 IO 操作。 + + + +### 参考资料 + +[1] Redis设计与实现: *https://redisbook.readthedocs.io/en/latest/feature/transaction.html#id3* \ No newline at end of file diff --git a/docs/data-management/Redis/reproduce/.DS_Store b/docs/data-management/Redis/reproduce/.DS_Store new file mode 100644 index 0000000000..d342460f02 Binary files /dev/null and b/docs/data-management/Redis/reproduce/.DS_Store differ diff --git a/docs/data-management/Redis/reproduce/Cache-Design.md b/docs/data-management/Redis/reproduce/Cache-Design.md new file mode 100644 index 0000000000..73926e7c53 --- /dev/null +++ b/docs/data-management/Redis/reproduce/Cache-Design.md @@ -0,0 +1,157 @@ +# 缓存设计 + +### 你的系统需要加个缓存设计吗 | 缓存的收益和成本 + +其实我们目前做的系统或多或少都会有些缓存层的设计，比如为了降低存储层（MySQL）的读写压力，我们一般会考虑加上像 Memcache 或者 Redis 这类全内存缓存层。加速读写的同时还能有效降低后端负载。 + +但是加缓存，也是有些成本需要考虑的，比如： + +- 数据不一致性问题 +- 代码维护成本 +- 运维成本 + +### 缓存更新策略 + +缓存中的数据通常都是有生命周期的，需要在指定时间后被删除或更新，这样可以保证缓存空间在一个可控的范围。 + +但是缓存中的数据会和数据 源中的真实数据有一段时间窗口的不一致，需要利用某些策略进行更新。下 面将分别从使用场景、一致性、开发人员开发/维护成本三个方面介绍三种 缓存的更新策略。 + +1.LRU/LFU/FIFO算法剔除 + +使用场景。剔除算法通常用于缓存使用量超过了预设的最大值时候，如 何对现有的数据进行剔除。例如Redis使用maxmemory-policy这个配置作为内 存最大值后对于数据的剔除策略。 + +一致性。要清理哪些数据是由具体算法决定，开发人员只能决定使用哪 +种算法，所以数据的一致性是最差的。 + +维护成本。算法不需要开发人员自己来实现，通常只需要配置最大 maxmemory和对应的策略即可。开发人员只需要知道每种算法的含义，选择 适合自己的算法即可。 + +2.超时剔除 + +使用场景。超时剔除通过给缓存数据设置过期时间，让其在过期时间后 自动删除，例如Redis提供的expire命令。如果业务可以容忍一段时间内，缓 存层数据和存储层数据不一致，那么可以为其设置过期时间。在数据过期 后，再从真实数据源获取数据，重新放到缓存并设置过期时间。例如一个视频的描述信息，可以容忍几分钟内数据不一致，但是涉及交易方面的业务， +后果可想而知。 + 一致性。一段时间窗口内(取决于过期时间长短)存在一致性问题，即 +缓存数据和真实数据源的数据不一致。 + +维护成本。维护成本不是很高，只需设置expire过期时间即可，当然前 提是应用方允许这段时间可能发生的数据不一致。 + +3.主动更新 使用场景。应用方对于数据的一致性要求高，需要在真实数据更新后， + +立即更新缓存数据。例如可以利用消息系统或者其他方式通知缓存更新。 + 一致性。一致性最高，但如果主动更新发生了问题，那么这条数据很可 +能很长时间不会更新，所以建议结合超时剔除一起使用效果会更好。 + 维护成本。维护成本会比较高，开发者需要自己来完成更新，并保证更 +新操作的正确性。 + +![](/Users/apple/Desktop/screenshot/截屏2021-03-15 下午7.06.48.png) + +·低一致性业务建议配置最大内存和淘汰策略的方式使用。 ·高一致性业务可以结合使用超时剔除和主动更新，这样即使主动更新出了问题，也能保证数据过期时间后删除脏数据。 + + + +### 缓存粒度控制 + +我们项目中一般在DB上层加个缓存，减轻DB压力，但是是需要缓存多少数据呢，是10条、100条、还是所有数据呢？这就是缓存粒度问题。 + +通用性。缓存全部数据比部分数据更加通用，但从实际经验看，很长时 +间内应用只需要几个重要的属性。 + 空间占用。缓存全部数据要比部分数据占用更多的空间，可能存在以下 +问题: + +·全部数据会造成内存的浪费。 ·全部数据可能每次传输产生的网络流量会比较大，耗时相对较大，在 + +极端情况下会阻塞网络。 + ·全部数据的序列化和反序列化的CPU开销更大。 代码维护。全部数据的优势更加明显，而部分数据一旦要加新字段需要修改业务代码，而且修改后通常还需要刷新缓存数据。 + +### 缓存穿透 + +缓存穿透是指查询一个根本不存在的数据，缓存层和存储层都不会命 中，通常出于容错的考虑，如果从存储层查不到数据则不写入缓存层，如图 11-3所示整个过程分为如下3步: + +1)缓存层不命中。 + +2)存储层不命中，不将空结果写回缓存。 + +3)返回空结果。 + +缓存穿透问题可能会使后端存储负载加大，由于很多后端存储不具备高 +并发性，甚至可能造成后端存储宕掉。通常可以在程序中分别统计总调用 +数、缓存层命中数、存储层命中数，如果发现大量存储层空命中，可能就是 +出现了缓存穿透问题。 + + 造成缓存穿透的基本原因有两个。第一，自身业务代码或者数据出现问 +题，第二，一些恶意攻击、爬虫等造成大量空命中。下面我们来看一下如何 +解决缓存穿透问题。 + +1.缓存空对象 + +当第2步存储层不命中后，仍然将空对象保留到缓存层中，之后再访问这个数据将会从缓存中获取，这样就保护了后端数据源。 + +2.布隆过滤器拦截 + +在访问缓存层和存储层之前，将存在的key用布隆过滤 器提前保存起来，做第一层拦截。例如:一个推荐系统有4亿个用户id，每 个小时算法工程师会根据每个用户之前历史行为计算出推荐数据放到存储层 中，但是最新的用户由于没有历史行为，就会发生缓存穿透的行为，为此可 以将所有推荐数据的用户做成布隆过滤器。如果布隆过滤器认为该用户id不 存在，那么就不会访问存储层，在一定程度保护了存储层。 + +### 缓存雪崩 + +什么是缓存雪崩:由于缓存层承载着大量请求，有效地 保护了存储层，但是如果缓存层由于某些原因不能提供服务，于是所有的请 求都会达到存储层，存储层的调用量会暴增，造成存储层也会级联宕机的情 况。缓存雪崩的英文原意是stampeding herd(奔逃的野牛)，指的是缓存层 宕掉后，流量会像奔逃的野牛一样，打向后端存储。 + +![](/Users/apple/Desktop/screenshot/截屏2021-03-15 下午7.26.03.png) + +预防和解决缓存雪崩问题，可以从以下三个方面进行着手。 + +1)保证缓存层服务高可用性。和飞机都有多个引擎一样，如果缓存层 设计成高可用的，即使个别节点、个别机器、甚至是机房宕掉，依然可以提 供服务，例如前面介绍过的Redis Sentinel和Redis Cluster都实现了高可用。 + +2)依赖隔离组件为后端限流并降级。无论是缓存层还是存储层都会有 出错的概率，可以将它们视同为资源。作为并发量较大的系统，假如有一个 资源不可用，可能会造成线程全部阻塞(hang)在这个资源上，造成整个系 统不可用。降级机制在高并发系统中是非常普遍的:比如推荐服务中，如果 个性化推荐服务不可用，可以降级补充热点数据，不至于造成前端页面是开 天窗。在实际项目中，我们需要对重要的资源(例如Redis、MySQL、 HBase、外部接口)都进行隔离，让每种资源都单独运行在自己的线程池 中，即使个别资源出现了问题，对其他服务没有影响。但是线程池如何管 理，比如如何关闭资源池、开启资源池、资源池阀值管理，这些做起来还是 相当复杂的。这里推荐一个Java依赖隔离工具 Hystrix(https://github.com/netflix/hystrix)，如图11-15所示。Hystrix是解决依 赖隔离的利器，但是该内容已经超出本书的范围，同时只适用于Java应用， 所以这里不会详细介绍 + +3)提前演练。在项目上线前，演练缓存层宕掉后，应用以及后端的负 载情况以及可能出现的问题，在此基础上做一些预案设定。 + + + +### 热点key重建优化 + +开发人员使用“缓存+过期时间”的策略既可以加速数据读写，又保证数 据的定期更新，这种模式基本能够满足绝大部分需求。但是有两个问题如果 同时出现，可能就会对应用造成致命的危害: + +·当前key是一个热点key(例如一个热门的娱乐新闻)，并发量非常 大。 + +·重建缓存不能在短时间完成，可能是一个复杂计算，例如复杂的 SQL、多次IO、多个依赖等。 + +在缓存失效的瞬间，有大量线程来重建缓存(如图11-16所示)，造成 后端负载加大，甚至可能会让应用崩溃。 + +要解决这个问题也不是很复杂，但是不能为了解决这个问题给系统带来 +更多的麻烦，所以需要制定如下目标: + +·减少重建缓存的次数。 + +·数据尽可能一致。 + +·较少的潜在危险。 + +![](/Users/apple/Desktop/screenshot/截屏2021-03-15 下午7.29.40.png) + + + +1.互斥锁(mutex key) + +此方法只允许一个线程重建缓存，其他线程等待重建缓存的线程执行 完，重新从缓存获取数据即可，整个过程如图11-17所示。 + +![](/Users/apple/Desktop/screenshot/截屏2021-03-15 下午8.15.13.png) + + + +2.永远不过期 + +“永远不过期”包含两层意思: + +·从缓存层面来看，确实没有设置过期时间，所以不会出现热点key过期 后产生的问题，也就是“物理”不过期。 + +·从功能层面来看，为每个value设置一个逻辑过期时间，当发现超过逻 辑过期时间后，会使用单独的线程去构建缓存。 + +整个过程如图11-18所示。 + +从实战看，此方法有效杜绝了热点key产生的问题，但唯一不足的就是 重构缓存期间，会出现数据不一致的情况，这取决于应用方是否容忍这种不 一致。 + + + + + + + +### 无底洞优化 \ No newline at end of file diff --git "a/docs/data-store/Redis/Key \345\257\273\345\235\200\347\256\227\346\263\225.md" "b/docs/data-management/Redis/reproduce/Key \345\257\273\345\235\200\347\256\227\346\263\225.md" similarity index 100% rename from "docs/data-store/Redis/Key \345\257\273\345\235\200\347\256\227\346\263\225.md" rename to "docs/data-management/Redis/reproduce/Key \345\257\273\345\235\200\347\256\227\346\263\225.md" diff --git "a/docs/data-management/Redis/reproduce/Redis\344\270\272\344\273\200\344\271\210\345\217\230\346\205\242\344\272\206-\345\270\270\350\247\201\345\273\266\350\277\237\351\227\256\351\242\230\345\256\232\344\275\215\344\270\216\345\210\206\346\236\220.md" "b/docs/data-management/Redis/reproduce/Redis\344\270\272\344\273\200\344\271\210\345\217\230\346\205\242\344\272\206-\345\270\270\350\247\201\345\273\266\350\277\237\351\227\256\351\242\230\345\256\232\344\275\215\344\270\216\345\210\206\346\236\220.md" new file mode 100755 index 0000000000..73778eece2 --- /dev/null +++ "b/docs/data-management/Redis/reproduce/Redis\344\270\272\344\273\200\344\271\210\345\217\230\346\205\242\344\272\206-\345\270\270\350\247\201\345\273\266\350\277\237\351\227\256\351\242\230\345\256\232\344\275\215\344\270\216\345\210\206\346\236\220.md" @@ -0,0 +1,248 @@ +> Redis作为内存数据库，拥有非常高的性能，单个实例的QPS能够达到10W左右。但我们在使用Redis时，经常时不时会出现访问延迟很大的情况，如果你不知道Redis的内部实现原理，在排查问题时就会一头雾水。 +> +> 很多时候，Redis出现访问延迟变大，都与我们的使用不当或运维不合理导致的。 +> +> 这篇文章我们就来分析一下Redis在使用过程中，经常会遇到的延迟问题以及如何定位和分析。 + + + +## 使用复杂度高的命令 + +如果在使用Redis时，发现访问延迟突然增大，如何进行排查？ + +首先，第一步，建议你去查看一下Redis的慢日志。Redis提供了慢日志命令的统计功能，我们通过以下设置，就可以查看有哪些命令在执行时延迟比较大。 + +首先设置Redis的慢日志阈值，只有超过阈值的命令才会被记录，这里的单位是微秒，例如设置慢日志的阈值为5毫秒，同时设置只保留最近1000条慢日志记录： + +``` +# 命令执行超过5毫秒记录慢日志 +CONFIG SET slowlog-log-slower-than 5000 +# 只保留最近1000条慢日志 +CONFIG SET slowlog-max-len 1000 +``` + +设置完成之后，所有执行的命令如果延迟大于5毫秒，都会被Redis记录下来，我们执行`SLOWLOG get 5`查询最近5条慢日志： + +``` +127.0.0.1:6379> SLOWLOG get 5 +1) 1) (integer) 32693 # 慢日志ID + 2) (integer) 1593763337 # 执行时间 + 3) (integer) 5299 # 执行耗时(微妙) + 4) 1) "LRANGE" # 具体执行的命令和参数 + 2) "user_list_2000" + 3) "0" + 4) "-1" +2) 1) (integer) 32692 + 2) (integer) 1593763337 + 3) (integer) 5044 + 4) 1) "GET" + 2) "book_price_1000" +... +``` + +通过查看慢日志记录，我们就可以知道在什么时间执行哪些命令比较耗时，如果你的业务**经常使用`O(N)`以上复杂度的命令**，例如`sort`、`sunion`、`zunionstore`，或者在执行`O(N)`命令时操作的数据量比较大，这些情况下Redis处理数据时就会很耗时。 + +如果你的服务请求量并不大，但Redis实例的CPU使用率很高，很有可能是使用了复杂度高的命令导致的。 + +解决方案就是，不使用这些复杂度较高的命令，并且一次不要获取太多的数据，每次尽量操作少量的数据，让Redis可以及时处理返回。 + +## 存储bigkey + +> 在Redis中，一个字符串最大512MB，一个二级数据结构（例如hash、list、set、zset）可以存储大约40亿个(2^32-1)个元素，但实际上中如果下面两种情况，我就会认为它是bigkey。 +> +> - **字符串类型**：它的big体现在单个value值很大，一般认为超过10KB就是bigkey。 +> - **非字符串类型**：哈希、列表、集合、有序集合，它们的big体现在元素个数太多。 + +如果查询慢日志发现，并不是复杂度较高的命令导致的，例如都是`SET`、`DELETE`操作出现在慢日志记录中，那么你就要怀疑是否存在Redis写入了bigkey的情况。 + +Redis在写入数据时，需要为新的数据分配内存，当从Redis中删除数据时，它会释放对应的内存空间。 + +如果一个key写入的数据非常大，Redis在**分配内存时也会比较耗时**。同样的，当删除这个key的数据时，**释放内存也会耗时比较久**。 + +你需要检查你的业务代码，是否存在写入bigkey的情况，需要评估写入数据量的大小，业务层应该避免一个key存入过大的数据量。 + +那么有没有什么办法可以扫描现在Redis中是否存在bigkey的数据吗？ + +Redis也提供了扫描bigkey的方法： + +``` +redis-cli -h $host -p $port --bigkeys -i 0.01 +``` + +使用上面的命令就可以扫描出整个实例key大小的分布情况，它是以类型维度来展示的。 + +需要注意的是当我们在线上实例进行bigkey扫描时，Redis的QPS会突增，为了降低扫描过程中对Redis的影响，我们需要控制扫描的频率，使用`-i`参数控制即可，它表示扫描过程中每次扫描的时间间隔，单位是秒。 + +使用这个命令的原理，其实就是Redis在内部执行`scan`命令，遍历所有key，然后针对不同类型的key执行`strlen`、`llen`、`hlen`、`scard`、`zcard`来获取字符串的长度以及容器类型(list/dict/set/zset)的元素个数。 + +而对于容器类型的key，只能扫描出元素最多的key，但元素最多的key不一定占用内存最多，这一点需要我们注意下。不过使用这个命令一般我们是可以对整个实例中key的分布情况有比较清晰的了解。 + +针对bigkey的问题，Redis官方在4.0版本推出了`lazy-free`的机制，用于异步释放bigkey的内存，降低对Redis性能的影响。即使这样，我们也不建议使用bigkey，bigkey在集群的迁移过程中，也会影响到迁移的性能，这个后面在介绍集群相关的文章时，会再详细介绍到。 + +## 集中过期 + +有时你会发现，平时在使用Redis时没有延时比较大的情况，但在某个时间点突然出现一波延时，而且**报慢的时间点很有规律，例如某个整点，或者间隔多久就会发生一次**。 + +如果出现这种情况，就需要考虑是否存在大量key集中过期的情况。 + +如果有大量的key在某个固定时间点集中过期，在这个时间点访问Redis时，就有可能导致延迟增加。 + +Redis的过期策略采用主动过期+懒惰过期两种策略： + +- 主动过期：Redis内部维护一个定时任务，默认每隔100毫秒会从过期字典中随机取出20个key，删除过期的key，如果过期key的比例超过了25%，则继续获取20个key，删除过期的key，循环往复，直到过期key的比例下降到25%或者这次任务的执行耗时超过了25毫秒，才会退出循环 +- 懒惰过期：只有当访问某个key时，才判断这个key是否已过期，如果已经过期，则从实例中删除 + +注意，**Redis的主动过期的定时任务，也是在Redis主线程中执行的**，也就是说如果在执行主动过期的过程中，出现了需要大量删除过期key的情况，那么在业务访问时，必须等这个过期任务执行结束，才可以处理业务请求。此时就会出现，业务访问延时增大的问题，最大延迟为25毫秒。 + +而且这个访问延迟的情况，**不会记录在慢日志里**。慢日志中**只记录真正执行某个命令的耗时**，Redis主动过期策略执行在操作命令之前，如果操作命令耗时达不到慢日志阈值，它是不会计算在慢日志统计中的，但我们的业务却感到了延迟增大。 + +此时你需要检查你的业务，是否真的存在集中过期的代码，一般集中过期使用的命令是`expireat`或`pexpireat`命令，在代码中搜索这个关键字就可以了。 + +如果你的业务确实需要集中过期掉某些key，又不想导致Redis发生抖动，有什么优化方案？ + +解决方案是，**在集中过期时增加一个随机时间，把这些需要过期的key的时间打散即可。** + +伪代码可以这么写： + +``` +# 在过期时间点之后的5分钟内随机过期掉 +redis.expireat(key, expire_time + random(300)) +``` + +这样Redis在处理过期时，不会因为集中删除key导致压力过大，阻塞主线程。 + +另外，除了业务使用需要注意此问题之外，还可以通过运维手段来及时发现这种情况。 + +做法是我们需要把Redis的各项运行数据监控起来，执行`info`可以拿到所有的运行数据，在这里我们需要重点关注`expired_keys`这一项，它代表整个实例到目前为止，累计删除过期key的数量。 + +我们需要对这个指标监控，当在**很短时间内这个指标出现突增**时，需要及时报警出来，然后与业务报慢的时间点对比分析，确认时间是否一致，如果一致，则可以认为确实是因为这个原因导致的延迟增大。 + +## 实例内存达到上限 + +有时我们把Redis当做纯缓存使用，就会给实例设置一个内存上限`maxmemory`，然后开启LRU淘汰策略。 + +当实例的内存达到了`maxmemory`后，你会发现之后的每次写入新的数据，有可能变慢了。 + +导致变慢的原因是，当Redis内存达到`maxmemory`后，每次写入新的数据之前，必须先踢出一部分数据，让内存维持在`maxmemory`之下。 + +这个踢出旧数据的逻辑也是需要消耗时间的，而具体耗时的长短，要取决于配置的淘汰策略： + +- allkeys-lru：不管key是否设置了过期，淘汰最近最少访问的key +- volatile-lru：只淘汰最近最少访问并设置过期的key +- allkeys-random：不管key是否设置了过期，随机淘汰 +- volatile-random：只随机淘汰有设置过期的key +- allkeys-ttl：不管key是否设置了过期，淘汰即将过期的key +- noeviction：不淘汰任何key，满容后再写入直接报错 +- allkeys-lfu：不管key是否设置了过期，淘汰访问频率最低的key（4.0+支持） +- volatile-lfu：只淘汰访问频率最低的过期key（4.0+支持） + +具体使用哪种策略，需要根据业务场景来决定。 + +我们最常使用的一般是`allkeys-lru`或`volatile-lru`策略，它们的处理逻辑是，每次从实例中随机取出一批key（可配置），然后淘汰一个最少访问的key，之后把剩下的key暂存到一个池子中，继续随机取出一批key，并与之前池子中的key比较，再淘汰一个最少访问的key。以此循环，直到内存降到`maxmemory`之下。 + +如果使用的是`allkeys-random`或`volatile-random`策略，那么就会快很多，因为是随机淘汰，那么就少了比较key访问频率时间的消耗了，随机拿出一批key后直接淘汰即可，因此这个策略要比上面的LRU策略执行快一些。 + +但以上这些逻辑都是在访问Redis时，**真正命令执行之前执行的**，也就是它会影响我们访问Redis时执行的命令。 + +另外，如果此时Redis实例中有存储bigkey，那么**在淘汰bigkey释放内存时，这个耗时会更加久，延迟更大**，这需要我们格外注意。 + +如果你的业务访问量非常大，并且必须设置`maxmemory`限制实例的内存上限，同时面临淘汰key导致延迟增大的的情况，要想缓解这种情况，除了上面说的避免存储bigkey、使用随机淘汰策略之外，也可以考虑**拆分实例**的方法来缓解，拆分实例可以把一个实例淘汰key的压力**分摊到多个实例**上，可以在一定程度降低延迟。 + +## fork耗时严重 + +如果你的Redis开启了自动生成RDB和AOF重写功能，那么有可能在后台生成RDB和AOF重写时导致Redis的访问延迟增大，而等这些任务执行完毕后，延迟情况消失。 + +遇到这种情况，一般就是执行生成RDB和AOF重写任务导致的。 + +生成RDB和AOF都需要父进程`fork`出一个子进程进行数据的持久化，**在`fork`执行过程中，父进程需要拷贝内存页表给子进程，如果整个实例内存占用很大，那么需要拷贝的内存页表会比较耗时，此过程会消耗大量的CPU资源，在完成`fork`之前，整个实例会被阻塞住，无法处理任何请求，如果此时CPU资源紧张，那么`fork`的时间会更长，甚至达到秒级。这会严重影响Redis的性能**。 + +具体原理也可以参考我之前写的文章：[Redis持久化是如何做的？RDB和AOF对比分析](http://kaito-kidd.com/2020/06/29/redis-persistence-rdb-aof/)。 + +我们可以执行`info`命令，查看最后一次`fork`执行的耗时`latest_fork_usec`，单位微妙。这个时间就是整个实例阻塞无法处理请求的时间。 + +除了因为备份的原因生成RDB之外，**在主从节点第一次建立数据同步时**，主节点也会生成RDB文件给从节点进行一次全量同步，这时也会对Redis产生性能影响。 + +要想避免这种情况，我们需要规划好数据备份的周期，建议在**从节点上执行备份，而且最好放在低峰期执行**。如果对于丢失数据不敏感的业务，那么不建议开启AOF和AOF重写功能。 + +另外，`fork`的耗时也与系统有关，如果把Redis部署在虚拟机上，那么这个时间也会增大。所以使用Redis时建议部署在物理机上，降低`fork`的影响。 + +## 绑定CPU + +很多时候，我们在部署服务时，为了提高性能，降低程序在使用多个CPU时上下文切换的性能损耗，一般会采用进程绑定CPU的操作。 + +但在使用Redis时，我们不建议这么干，原因如下。 + +**绑定CPU的Redis，在进行数据持久化时，`fork`出的子进程，子进程会继承父进程的CPU使用偏好，而此时子进程会消耗大量的CPU资源进行数据持久化，子进程会与主进程发生CPU争抢，这也会导致主进程的CPU资源不足访问延迟增大。** + +所以在部署Redis进程时，如果需要开启RDB和AOF重写机制，一定不能进行CPU绑定操作！ + +## AOF配置不合理 + +上面提到了，当执行AOF文件重写时会因为`fork`执行耗时导致Redis延迟增大，除了这个之外，如果开启AOF机制，设置的策略不合理，也会导致性能问题。 + +开启AOF后，Redis会把写入的命令实时写入到文件中，但写入文件的过程是先写入内存，等内存中的数据超过一定阈值或达到一定时间后，内存中的内容才会被真正写入到磁盘中。 + +AOF为了保证文件写入磁盘的安全性，提供了3种刷盘机制： + +- `appendfsync always`：每次写入都刷盘，对性能影响最大，占用磁盘IO比较高，数据安全性最高 +- `appendfsync everysec`：1秒刷一次盘，对性能影响相对较小，节点宕机时最多丢失1秒的数据 +- `appendfsync no`：按照操作系统的机制刷盘，对性能影响最小，数据安全性低，节点宕机丢失数据取决于操作系统刷盘机制 + +当使用第一种机制`appendfsync always`时，Redis每处理一次写命令，都会把这个命令写入磁盘，而且**这个操作是在主线程中执行的**。 + +内存中的的数据写入磁盘，这个会加重磁盘的IO负担，操作磁盘成本要比操作内存的代价大得多。如果写入量很大，那么每次更新都会写入磁盘，此时机器的磁盘IO就会非常高，拖慢Redis的性能，因此我们不建议使用这种机制。 + +与第一种机制对比，`appendfsync everysec`会每隔1秒刷盘，而`appendfsync no`取决于操作系统的刷盘时间，安全性不高。因此我们推荐使用`appendfsync everysec`这种方式，在最坏的情况下，只会丢失1秒的数据，但它能保持较好的访问性能。 + +当然，对于有些业务场景，对丢失数据并不敏感，也可以不开启AOF。 + +## 使用Swap + +如果你发现Redis突然变得非常慢，**每次访问的耗时都达到了几百毫秒甚至秒级**，那此时就检查Redis是否使用到了Swap，这种情况下Redis基本上已经无法提供高性能的服务。 + +我们知道，操作系统提供了Swap机制，目的是为了当内存不足时，可以把一部分内存中的数据换到磁盘上，以达到对内存使用的缓冲。 + +但当内存中的数据被换到磁盘上后，访问这些数据就需要从磁盘中读取，这个速度要比内存慢太多！ + +**尤其是针对Redis这种高性能的内存数据库来说，如果Redis中的内存被换到磁盘上，对于Redis这种性能极其敏感的数据库，这个操作时间是无法接受的。** + +我们需要检查机器的内存使用情况，确认是否确实是因为内存不足导致使用到了Swap。 + +如果确实使用到了Swap，要及时整理内存空间，释放出足够的内存供Redis使用，然后释放Redis的Swap，让Redis重新使用内存。 + +释放Redis的Swap过程通常要重启实例，为了避免重启实例对业务的影响，一般先进行主从切换，然后释放旧主节点的Swap，重新启动服务，待数据同步完成后，再切换回主节点即可。 + +可见，当Redis使用到Swap后，此时的Redis的高性能基本被废掉，所以我们需要提前预防这种情况。 + +**我们需要对Redis机器的内存和Swap使用情况进行监控，在内存不足和使用到Swap时及时报警出来，及时进行相应的处理。** + +## 网卡负载过高 + +如果以上产生性能问题的场景，你都规避掉了，而且Redis也稳定运行了很长时间，但在某个时间点之后开始，访问Redis开始变慢了，而且一直持续到现在，这种情况是什么原因导致的？ + +之前我们就遇到这种问题，**特点就是从某个时间点之后就开始变慢，并且一直持续**。这时你需要检查一下机器的网卡流量，是否存在网卡流量被跑满的情况。 + +**网卡负载过高，在网络层和TCP层就会出现数据发送延迟、数据丢包等情况。Redis的高性能除了内存之外，就在于网络IO，请求量突增会导致网卡负载变高。** + +如果出现这种情况，你需要排查这个机器上的哪个Redis实例的流量过大占满了网络带宽，然后确认流量突增是否属于业务正常情况，如果属于那就需要及时扩容或迁移实例，避免这个机器的其他实例受到影响。 + +运维层面，我们需要对机器的各项指标增加监控，包括网络流量，在达到阈值时提前报警，及时与业务确认并扩容。 + +## 总结 + +以上我们总结了Redis中常见的可能导致延迟增大甚至阻塞的场景，这其中既涉及到了业务的使用问题，也涉及到Redis的运维问题。 + +可见，要想保证Redis高性能的运行，其中涉及到CPU、内存、网络，甚至磁盘的方方面面，其中还包括操作系统的相关特性的使用。 + +作为开发人员，我们需要了解Redis的运行机制，例如各个命令的执行时间复杂度、数据过期策略、数据淘汰策略等，使用合理的命令，并结合业务场景进行优化。 + +作为DBA运维人员，需要了解数据持久化、操作系统`fork`原理、Swap机制等，并对Redis的容量进行合理规划，预留足够的机器资源，对机器做好完善的监控，才能保证Redis的稳定运行。 + + + +**来源** + +> [《Redis为什么变慢了？常见延迟问题定位与分析》](http://kaito-kidd.com/2020/07/03/redis-latency-analysis/) +> +> 作者：Kaito + diff --git a/docs/data-store/MySQL/1.MySQL.md b/docs/data-store/MySQL/1.MySQL.md deleted file mode 100644 index 7c46523333..0000000000 --- a/docs/data-store/MySQL/1.MySQL.md +++ /dev/null @@ -1,33 +0,0 @@ -## 数据类型 - -### 整型 - -| 整型 | 存储范围 | 字节 | -| ------------ | ------------------------------------------------------------ | ---- | -| tinyint | 有符号值：-128到127（-2^7到2^7-1） 无符号值：0到255（0到2^8-1） | 1 | -| smallint | 有符号值：-32768到32767（-2^15到2^15-1） 无符号值：0到65535（0到2^16-1） | 2 | -| mediumint | 有符号值：-2^23到2^23-1 无符号值：0到2^24-1 | 3 | -| int(integer) | 有符号值：-2^31到2^31-1 无符号值：0到2^32-1 | 4 | -| bigint | 有符号值：-2^63到2^63-1 无符号值：0到2^64-1 | 8 | - -``` -可使用unsigned控制是否有正负 -可以使用zerofill来进行前导零填充 -``` - - - -https://blog.csdn.net/csdn_c_/article/details/78305742 - - - -### 浮点型 - -| 整型 | 存储范围 | 字节 | -| ------------ | ------------------------------------------------------------ | ---- | -| float | 有符号值：-128到127（-2^7到2^7-1） 无符号值：0到255（0到2^8-1） | 1 | -| smallint | 有符号值：-32768到32767（-2^15到2^15-1） 无符号值：0到65535（0到2^16-1） | 2 | -| mediumint | 有符号值：-2^23到2^23-1 无符号值：0到2^24-1 | 3 | -| int(integer) | 有符号值：-2^31到2^31-1 无符号值：0到2^32-1 | 4 | -| bigint | 有符号值：-2^63到2^63-1 无符号值：0到2^64-1 | 8 | - diff --git a/docs/data-store/MySQL/MySQL-FAQ.md b/docs/data-store/MySQL/MySQL-FAQ.md deleted file mode 100644 index cc56e8ffe4..0000000000 --- a/docs/data-store/MySQL/MySQL-FAQ.md +++ /dev/null @@ -1,1705 +0,0 @@ -![](https://imgkr.cn-bj.ufileos.com/9e22c4b4-0db5-4f4f-9636-974875d4018f.jpg) - -> 写在之前：不建议那种上来就是各种面试题罗列，然后背书式的去记忆，对技术的提升帮助很小，对正经面试也没什么帮助，有点东西的面试官深挖下就懵逼了。 -> -> 个人建议把面试题看作是费曼学习法中的回顾、简化的环节，准备面试的时候，跟着题目先自己讲给自己听，看看自己会满意吗，不满意就继续学习这个点，如此反复，好的offer离你不远的，奥利给 - -## 一、MySQL架构 - -和其它数据库相比，MySQL有点与众不同，它的架构可以在多种不同场景中应用并发挥良好作用。主要体现在存储引擎的架构上，**插件式的存储引擎架构将查询处理和其它的系统任务以及数据的存储提取相分离**。这种架构可以根据业务的需求和实际需要选择合适的存储引擎。 - -![](https://img2018.cnblogs.com/blog/1383365/201902/1383365-20190201092513900-638761565.png) - - - -- **连接层**：最上层是一些客户端和连接服务。**主要完成一些类似于连接处理、授权认证、及相关的安全方案**。在该层上引入了线程池的概念，为通过认证安全接入的客户端提供线程。同样在该层上可以实现基于SSL的安全链接。服务器也会为安全接入的每个客户端验证它所具有的操作权限。 - -- **服务层**：第二层服务层，主要完成大部分的核心服务功能， 包括查询解析、分析、优化、缓存、以及所有的内置函数，所有跨存储引擎的功能也都在这一层实现，包括触发器、存储过程、视图等 - -- **引擎层**：第三层存储引擎层，存储引擎真正的负责了MySQL中数据的存储和提取，服务器通过API与存储引擎进行通信。不同的存储引擎具有的功能不同，这样我们可以根据自己的实际需要进行选取 - -- **存储层**：第四层为数据存储层，主要是将数据存储在运行于该设备的文件系统之上，并完成与存储引擎的交互 - - - -> 画出 MySQL 架构图，这种变态问题都能问的出来 -> -> MySQL 的查询流程具体是？or 一条SQL语句在MySQL中如何执行的？ -> - -客户端请求 ---> 连接器（验证用户身份，给予权限） ---> 查询缓存（存在缓存则直接返回，不存在则执行后续操作） ---> 分析器（对SQL进行词法分析和语法分析操作） ---> 优化器（主要对执行的sql优化选择最优的执行方案方法） ---> 执行器（执行时会先看用户是否有执行权限，有才去使用这个引擎提供的接口） ---> 去引擎层获取数据返回（如果开启查询缓存则会缓存查询结果） - -![img](https://pic2.zhimg.com/80/v2-0d2070e8f84c4801adbfa03bda1f98d9_720w.jpg) - ------- - - - -> 说说MySQL有哪些存储引擎？都有哪些区别？ - -## 二、存储引擎 - -存储引擎是MySQL的组件，用于处理不同表类型的SQL操作。不同的存储引擎提供不同的存储机制、索引技巧、锁定水平等功能，使用不同的存储引擎，还可以获得特定的功能。 - -使用哪一种引擎可以灵活选择，**一个数据库中多个表可以使用不同引擎以满足各种性能和实际需求**，使用合适的存储引擎，将会提高整个数据库的性能 。 - - MySQL服务器使用**可插拔**的存储引擎体系结构，可以从运行中的 MySQL 服务器加载或卸载存储引擎 。 - -### 查看存储引擎 - -```mysql --- 查看支持的存储引擎 -SHOW ENGINES - --- 查看默认存储引擎 -SHOW VARIABLES LIKE 'storage_engine' - ---查看具体某一个表所使用的存储引擎，这个默认存储引擎被修改了！ -show create table tablename - ---准确查看某个数据库中的某一表所使用的存储引擎 -show table status like 'tablename' -show table status from database where name="tablename" -``` - -### 设置存储引擎 - -```mysql --- 建表时指定存储引擎。默认的就是INNODB，不需要设置 -CREATE TABLE t1 (i INT) ENGINE = INNODB; -CREATE TABLE t2 (i INT) ENGINE = CSV; -CREATE TABLE t3 (i INT) ENGINE = MEMORY; - --- 修改存储引擎 -ALTER TABLE t ENGINE = InnoDB; - --- 修改默认存储引擎，也可以在配置文件my.cnf中修改默认引擎 -SET default_storage_engine=NDBCLUSTER; -``` - -默认情况下，每当 `CREATE TABLE` 或 `ALTER TABLE` 不能使用默认存储引擎时，都会生成一个警告。为了防止在所需的引擎不可用时出现令人困惑的意外行为，可以启用 `NO_ENGINE_SUBSTITUTION SQL` 模式。如果所需的引擎不可用，则此设置将产生错误而不是警告，并且不会创建或更改表 - - - -### 存储引擎对比 - -常见的存储引擎就 InnoDB、MyISAM、Memory、NDB。 - -InnoDB 现在是 MySQL 默认的存储引擎，支持**事务、行级锁定和外键** - -#### 文件存储结构对比 - -在 MySQL中建立任何一张数据表，在其数据目录对应的数据库目录下都有对应表的 `.frm` 文件，`.frm` 文件是用来保存每个数据表的元数据(meta)信息，包括表结构的定义等，与数据库存储引擎无关，也就是任何存储引擎的数据表都必须有`.frm`文件，命名方式为 数据表名.frm，如user.frm。 - -查看MySQL 数据保存在哪里：`show variables like 'data%'` - -MyISAM 物理文件结构为： - -- `.frm`文件：与表相关的元数据信息都存放在frm文件，包括表结构的定义信息等 -- `.MYD` (`MYData`) 文件：MyISAM 存储引擎专用，用于存储MyISAM 表的数据 -- `.MYI` (`MYIndex`)文件：MyISAM 存储引擎专用，用于存储MyISAM 表的索引相关信息 - -InnoDB 物理文件结构为： - -- `.frm` 文件：与表相关的元数据信息都存放在frm文件，包括表结构的定义信息等 - -- `.ibd` 文件或 `.ibdata` 文件： 这两种文件都是存放 InnoDB 数据的文件，之所以有两种文件形式存放 InnoDB 的数据，是因为 InnoDB 的数据存储方式能够通过配置来决定是使用**共享表空间**存放存储数据，还是用**独享表空间**存放存储数据。 - - 独享表空间存储方式使用`.ibd`文件，并且每个表一个`.ibd`文件 - 共享表空间存储方式使用`.ibdata`文件，所有表共同使用一个`.ibdata`文件（或多个，可自己配置） - -> ps：正经公司，这些都有专业运维去做，数据备份、恢复啥的，让我一个 Javaer 搞这的话，加钱不？ - -#### 面试这么回答 - -1. InnoDB 支持事务，MyISAM 不支持事务。这是 MySQL 将默认存储引擎从 MyISAM 变成 InnoDB 的重要原因之一； -2. InnoDB 支持外键，而 MyISAM 不支持。对一个包含外键的 InnoDB 表转为 MYISAM 会失败； -3. InnoDB 是聚簇索引，MyISAM 是非聚簇索引。聚簇索引的文件存放在主键索引的叶子节点上，因此 InnoDB 必须要有主键，通过主键索引效率很高。但是辅助索引需要两次查询，先查询到主键，然后再通过主键查询到数据。因此，主键不应该过大，因为主键太大，其他索引也都会很大。而 MyISAM 是非聚集索引，数据文件是分离的，索引保存的是数据文件的指针。主键索引和辅助索引是独立的。 -4. InnoDB 不保存表的具体行数，执行` select count(*) from table` 时需要全表扫描。而 MyISAM 用一个变量保存了整个表的行数，执行上述语句时只需要读出该变量即可，速度很快； -5. InnoDB 最小的锁粒度是行锁，MyISAM 最小的锁粒度是表锁。一个更新语句会锁住整张表，导致其他查询和更新都会被阻塞，因此并发访问受限。这也是 MySQL 将默认存储引擎从 MyISAM 变成 InnoDB 的重要原因之一； - -| 对比项 | MyISAM | InnoDB | -| -------- | -------------------------------------------------------- | ------------------------------------------------------------ | -| 主外键 | 不支持 | 支持 | -| 事务 | 不支持 | 支持 | -| 行表锁 | 表锁，即使操作一条记录也会锁住整个表，不适合高并发的操作 | 行锁,操作时只锁某一行，不对其它行有影响，适合高并发的操作 | -| 缓存 | 只缓存索引，不缓存真实数据 | 不仅缓存索引还要缓存真实数据，对内存要求较高，而且内存大小对性能有决定性的影响 | -| 表空间 | 小 | 大 | -| 关注点 | 性能 | 事务 | -| 默认安装 | 是 | 是 | - - - -> 一张表，里面有ID自增主键，当insert了17条记录之后，删除了第15,16,17条记录，再把Mysql重启，再insert一条记录，这条记录的ID是18还是15 ？ - -如果表的类型是MyISAM，那么是18。因为MyISAM表会把自增主键的最大ID 记录到数据文件中，重启MySQL自增主键的最大ID也不会丢失； - -如果表的类型是InnoDB，那么是15。因为InnoDB 表只是把自增主键的最大ID记录到内存中，所以重启数据库或对表进行OPTION操作，都会导致最大ID丢失。 - -> 哪个存储引擎执行 select count(*) 更快，为什么? - -MyISAM更快，因为MyISAM内部维护了一个计数器，可以直接调取。 - -- 在 MyISAM 存储引擎中，把表的总行数存储在磁盘上，当执行 select count(\*) from t 时，直接返回总数据。 - -- 在 InnoDB 存储引擎中，跟 MyISAM 不一样，没有将总行数存储在磁盘上，当执行 select count(\*) from t 时，会先把数据读出来，一行一行的累加，最后返回总数量。 - -InnoDB 中 count(\*) 语句是在执行的时候，全表扫描统计总数量，所以当数据越来越大时，语句就越来越耗时了，为什么 InnoDB 引擎不像 MyISAM 引擎一样，将总行数存储到磁盘上？这跟 InnoDB 的事务特性有关，由于多版本并发控制（MVCC）的原因，InnoDB 表“应该返回多少行”也是不确定的。 - - - -## 三、数据类型 - -主要包括以下五大类： - -- 整数类型：BIT、BOOL、TINY INT、SMALL INT、MEDIUM INT、 INT、 BIG INT -- 浮点数类型：FLOAT、DOUBLE、DECIMAL -- 字符串类型：CHAR、VARCHAR、TINY TEXT、TEXT、MEDIUM TEXT、LONGTEXT、TINY BLOB、BLOB、MEDIUM BLOB、LONG BLOB -- 日期类型：Date、DateTime、TimeStamp、Time、Year -- 其他数据类型：BINARY、VARBINARY、ENUM、SET、Geometry、Point、MultiPoint、LineString、MultiLineString、Polygon、GeometryCollection等 - -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gf29eij8zsj316x0u0gsn.jpg) - -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gf29fk2f4rj31ac0gi0w3.jpg) - -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gf29g2azwtj31a80nywit.jpg) - - - -> CHAR 和 VARCHAR 的区别？ - -char是固定长度，varchar长度可变： - -char(n) 和 varchar(n) 中括号中 n 代表字符的个数，并不代表字节个数，比如 CHAR(30) 就可以存储 30 个字符。 - -存储时，前者不管实际存储数据的长度，直接按 char 规定的长度分配存储空间；而后者会根据实际存储的数据分配最终的存储空间 - -相同点： - -1. char(n)，varchar(n)中的n都代表字符的个数 -2. 超过char，varchar最大长度n的限制后，字符串会被截断。 - -不同点： - -1. char不论实际存储的字符数都会占用n个字符的空间，而varchar只会占用实际字符应该占用的字节空间加1（实际长度length，0<=length<255）或加2（length>255）。因为varchar保存数据时除了要保存字符串之外还会加一个字节来记录长度（如果列声明长度大于255则使用两个字节来保存长度）。 -2. 能存储的最大空间限制不一样：char的存储上限为255字节。 -3. char在存储时会截断尾部的空格，而varchar不会。 - -char是适合存储很短的、一般固定长度的字符串。例如，char非常适合存储密码的MD5值，因为这是一个定长的值。对于非常短的列，char比varchar在存储空间上也更有效率。 - - - -> 列的字符串类型可以是什么？ - -字符串类型是：SET、BLOB、ENUM、CHAR、CHAR、TEXT、VARCHAR - - - -> BLOB和TEXT有什么区别？ - -BLOB是一个二进制对象，可以容纳可变数量的数据。有四种类型的BLOB：TINYBLOB、BLOB、MEDIUMBLO和 LONGBLOB - -TEXT是一个不区分大小写的BLOB。四种TEXT类型：TINYTEXT、TEXT、MEDIUMTEXT 和 LONGTEXT。 - -BLOB 保存二进制数据，TEXT 保存字符数据。 - ------- - - - -## 四、索引 - ->说说你对 MySQL 索引的理解？ -> ->数据库索引的原理，为什么要用 B+树，为什么不用二叉树？ -> ->聚集索引与非聚集索引的区别？ -> ->InnoDB引擎中的索引策略，了解过吗？ -> ->创建索引的方式有哪些？ -> ->聚簇索引/非聚簇索引，mysql索引底层实现，为什么不用B-tree，为什么不用hash，叶子结点存放的是数据还是指向数据的内存地址，使用索引需要注意的几个地方？ - -- MYSQL官方对索引的定义为：索引（Index）是帮助MySQL高效获取数据的数据结构，所以说**索引的本质是：数据结构** - -- 索引的目的在于提高查询效率，可以类比字典、 火车站的车次表、图书的目录等 。 - -- 可以简单的理解为“排好序的快速查找数据结构”，数据本身之外，**数据库还维护者一个满足特定查找算法的数据结构**，这些数据结构以某种方式引用（指向）数据，这样就可以在这些数据结构上实现高级查找算法。这种数据结构，就是索引。下图是一种可能的索引方式示例。 - - ![](https://tva1.sinaimg.cn/large/007S8ZIlly1gf3u9tli6gj30gt08xdg2.jpg) - - 左边的数据表，一共有两列七条记录，最左边的是数据记录的物理地址 - - 为了加快Col2的查找，可以维护一个右边所示的二叉查找树，每个节点分别包含索引键值，和一个指向对应数据记录物理地址的指针，这样就可以运用二叉查找在一定的复杂度内获取到对应的数据，从而快速检索出符合条件的记录。 - -- 索引本身也很大，不可能全部存储在内存中，**一般以索引文件的形式存储在磁盘上** - -- 平常说的索引，没有特别指明的话，就是B+树（多路搜索树，不一定是二叉树）结构组织的索引。其中聚集索引，次要索引，覆盖索引，复合索引，前缀索引，唯一索引默认都是使用B+树索引，统称索引。此外还有哈希索引等。 - - - - -### 基本语法： - -- 创建： - - - 创建索引：`CREATE [UNIQUE] INDEX indexName ON mytable(username(length));` - - 如果是CHAR，VARCHAR类型，length可以小于字段实际长度；如果是BLOB和TEXT类型，必须指定 length。 - - - 修改表结构(添加索引)：`ALTER table tableName ADD [UNIQUE] INDEX indexName(columnName)` - -- 删除：`DROP INDEX [indexName] ON mytable;` - -- 查看：`SHOW INDEX FROM table_name\G` --可以通过添加 \G 来格式化输出信息。 - -- 使用ALERT命令 - - - `ALTER TABLE tbl_name ADD PRIMARY KEY (column_list):` 该语句添加一个主键，这意味着索引值必须是唯一的，且不能为NULL。 - - `ALTER TABLE tbl_name ADD UNIQUE index_name (column_list` 这条语句创建索引的值必须是唯一的（除了NULL外，NULL可能会出现多次）。 - - `ALTER TABLE tbl_name ADD INDEX index_name (column_list)` 添加普通索引，索引值可出现多次。 - - `ALTER TABLE tbl_name ADD FULLTEXT index_name (column_list)`该语句指定了索引为 FULLTEXT ，用于全文索引。 - - - -### 优势 - -- **提高数据检索效率，降低数据库IO成本** - -- **降低数据排序的成本，降低CPU的消耗** - - - -### 劣势 - -- 索引也是一张表，保存了主键和索引字段，并指向实体表的记录，所以也需要占用内存 -- 虽然索引大大提高了查询速度，同时却会降低更新表的速度，如对表进行INSERT、UPDATE和DELETE。 - 因为更新表时，MySQL不仅要保存数据，还要保存一下索引文件每次更新添加了索引列的字段， - 都会调整因为更新所带来的键值变化后的索引信息 - - - -### MySQL索引分类 - -#### 数据结构角度 - -- B+树索引 -- Hash索引 -- Full-Text全文索引 -- R-Tree索引 - -#### 从物理存储角度 - -- 聚集索引（clustered index） - -- 非聚集索引（non-clustered index），也叫辅助索引（secondary index） - - 聚集索引和非聚集索引都是B+树结构 - -#### 从逻辑角度 - -- 主键索引：主键索引是一种特殊的唯一索引，不允许有空值 -- 普通索引或者单列索引：每个索引只包含单个列，一个表可以有多个单列索引 -- 多列索引（复合索引、联合索引）：复合索引指多个字段上创建的索引，只有在查询条件中使用了创建索引时的第一个字段，索引才会被使用。使用复合索引时遵循最左前缀集合 -- 唯一索引或者非唯一索引 -- 空间索引：空间索引是对空间数据类型的字段建立的索引，MYSQL中的空间数据类型有4种，分别是GEOMETRY、POINT、LINESTRING、POLYGON。 - MYSQL使用SPATIAL关键字进行扩展，使得能够用于创建正规索引类型的语法创建空间索引。创建空间索引的列，必须将其声明为NOT NULL，空间索引只能在存储引擎为MYISAM的表中创建 - - - -> 为什么MySQL 索引中用B+tree，不用B-tree 或者其他树，为什么不用 Hash 索引 -> -> 聚簇索引/非聚簇索引，MySQL 索引底层实现，叶子结点存放的是数据还是指向数据的内存地址，使用索引需要注意的几个地方？ -> -> 使用索引查询一定能提高查询的性能吗？为什么? - -### MySQL索引结构 - -**首先要明白索引（index）是在存储引擎（storage engine）层面实现的，而不是server层面**。不是所有的存储引擎都支持所有的索引类型。即使多个存储引擎支持某一索引类型，它们的实现和行为也可能有所差别。 - -#### B+Tree索引 - -MyISAM 和 InnoDB 存储引擎，都使用 B+Tree的数据结构，它相对与 B-Tree结构，所有的数据都存放在叶子节点上，且把叶子节点通过指针连接到一起，形成了一条数据链表，以加快相邻数据的检索效率。 - -**先了解下 B-Tree 和 B+Tree 的区别** - -##### B-Tree - -B-Tree是为磁盘等外存储设备设计的一种平衡查找树。 - -系统从磁盘读取数据到内存时是以磁盘块（block）为基本单位的，位于同一个磁盘块中的数据会被一次性读取出来，而不是需要什么取什么。 - -InnoDB 存储引擎中有页（Page）的概念，页是其磁盘管理的最小单位。InnoDB 存储引擎中默认每个页的大小为16KB，可通过参数 `innodb_page_size` 将页的大小设置为 4K、8K、16K，在 MySQL 中可通过如下命令查看页的大小：`show variables like 'innodb_page_size';` - -而系统一个磁盘块的存储空间往往没有这么大，因此 InnoDB 每次申请磁盘空间时都会是若干地址连续磁盘块来达到页的大小 16KB。InnoDB 在把磁盘数据读入到磁盘时会以页为基本单位，在查询数据时如果一个页中的每条数据都能有助于定位数据记录的位置，这将会减少磁盘I/O次数，提高查询效率。 - -B-Tree 结构的数据可以让系统高效的找到数据所在的磁盘块。为了描述 B-Tree，首先定义一条记录为一个二元组[key, data] ，key为记录的键值，对应表中的主键值，data 为一行记录中除主键外的数据。对于不同的记录，key值互不相同。 - -一棵m阶的B-Tree有如下特性：  -1. 每个节点最多有m个孩子  -2. 除了根节点和叶子节点外，其它每个节点至少有Ceil(m/2)个孩子。 -3. 若根节点不是叶子节点，则至少有2个孩子  -4. 所有叶子节点都在同一层，且不包含其它关键字信息  -5. 每个非终端节点包含n个关键字信息（P0,P1,…Pn, k1,…kn）  -6. 关键字的个数n满足：ceil(m/2)-1 <= n <= m-1  -7. ki(i=1,…n)为关键字，且关键字升序排序 -8. Pi(i=1,…n)为指向子树根节点的指针。P(i-1)指向的子树的所有节点关键字均小于ki，但都大于k(i-1) - -B-Tree 中的每个节点根据实际情况可以包含大量的关键字信息和分支，如下图所示为一个 3 阶的 B-Tree： - -![索引](https://img-blog.csdn.net/20160202204827368) - -每个节点占用一个盘块的磁盘空间，一个节点上有两个升序排序的关键字和三个指向子树根节点的指针，指针存储的是子节点所在磁盘块的地址。两个关键词划分成的三个范围域对应三个指针指向的子树的数据的范围域。以根节点为例，关键字为17和35，P1指针指向的子树的数据范围为小于17，P2指针指向的子树的数据范围为17~35，P3指针指向的子树的数据范围为大于35。 - -模拟查找关键字29的过程： - -1. 根据根节点找到磁盘块1，读入内存。【磁盘I/O操作第1次】 -2. 比较关键字29在区间（17,35），找到磁盘块1的指针P2。 -3. 根据P2指针找到磁盘块3，读入内存。【磁盘I/O操作第2次】 -4. 比较关键字29在区间（26,30），找到磁盘块3的指针P2。 -5. 根据P2指针找到磁盘块8，读入内存。【磁盘I/O操作第3次】 -6. 在磁盘块8中的关键字列表中找到关键字29。 - -分析上面过程，发现需要3次磁盘I/O操作，和3次内存查找操作。由于内存中的关键字是一个有序表结构，可以利用二分法查找提高效率。而3次磁盘I/O操作是影响整个B-Tree查找效率的决定因素。B-Tree相对于AVLTree缩减了节点个数，使每次磁盘I/O取到内存的数据都发挥了作用，从而提高了查询效率。 - -##### B+Tree - -B+Tree 是在 B-Tree 基础上的一种优化，使其更适合实现外存储索引结构，InnoDB 存储引擎就是用 B+Tree 实现其索引结构。 - -从上一节中的B-Tree结构图中可以看到每个节点中不仅包含数据的key值，还有data值。而每一个页的存储空间是有限的，如果data数据较大时将会导致每个节点（即一个页）能存储的key的数量很小，当存储的数据量很大时同样会导致B-Tree的深度较大，增大查询时的磁盘I/O次数，进而影响查询效率。在B+Tree中，**所有数据记录节点都是按照键值大小顺序存放在同一层的叶子节点上**，而非叶子节点上只存储key值信息，这样可以大大加大每个节点存储的key值数量，降低B+Tree的高度。 - -B+Tree相对于B-Tree有几点不同： - -1. 非叶子节点只存储键值信息； -2. 所有叶子节点之间都有一个链指针； -3. 数据记录都存放在叶子节点中 - -将上一节中的B-Tree优化，由于B+Tree的非叶子节点只存储键值信息，假设每个磁盘块能存储4个键值及指针信息，则变成B+Tree后其结构如下图所示： -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gf3t57jvq1j30sc0aj0tj.jpg) - -通常在B+Tree上有两个头指针，一个指向根节点，另一个指向关键字最小的叶子节点，而且所有叶子节点（即数据节点）之间是一种链式环结构。因此可以对B+Tree进行两种查找运算：一种是对于主键的范围查找和分页查找，另一种是从根节点开始，进行随机查找。 - -可能上面例子中只有22条数据记录，看不出B+Tree的优点，下面做一个推算： - -InnoDB存储引擎中页的大小为16KB，一般表的主键类型为INT（占用4个字节）或BIGINT（占用8个字节），指针类型也一般为4或8个字节，也就是说一个页（B+Tree中的一个节点）中大概存储16KB/(8B+8B)=1K个键值（因为是估值，为方便计算，这里的K取值为10^3）。也就是说一个深度为3的B+Tree索引可以维护10^3 * 10^3 * 10^3 = 10亿 条记录。 - -实际情况中每个节点可能不能填充满，因此在数据库中，B+Tree的高度一般都在2-4层。MySQL的InnoDB存储引擎在设计时是将根节点常驻内存的，也就是说查找某一键值的行记录时最多只需要1~3次磁盘I/O操作。 - -B+Tree性质 - -1. 通过上面的分析，我们知道IO次数取决于b+数的高度h，假设当前数据表的数据为N，每个磁盘块的数据项的数量是m，则有h=㏒(m+1)N，当数据量N一定的情况下，m越大，h越小；而m = 磁盘块的大小 / 数据项的大小，磁盘块的大小也就是一个数据页的大小，是固定的，如果数据项占的空间越小，数据项的数量越多，树的高度越低。这就是为什么每个数据项，即索引字段要尽量的小，比如int占4字节，要比bigint8字节少一半。这也是为什么b+树要求把真实的数据放到叶子节点而不是内层节点，一旦放到内层节点，磁盘块的数据项会大幅度下降，导致树增高。当数据项等于1时将会退化成线性表。 -2. 当b+树的数据项是复合的数据结构，比如(name,age,sex)的时候，b+数是按照从左到右的顺序来建立搜索树的，比如当(张三,20,F)这样的数据来检索的时候，b+树会优先比较name来确定下一步的所搜方向，如果name相同再依次比较age和sex，最后得到检索的数据；但当(20,F)这样的没有name的数据来的时候，b+树就不知道下一步该查哪个节点，因为建立搜索树的时候name就是第一个比较因子，必须要先根据name来搜索才能知道下一步去哪里查询。比如当(张三,F)这样的数据来检索时，b+树可以用name来指定搜索方向，但下一个字段age的缺失，所以只能把名字等于张三的数据都找到，然后再匹配性别是F的数据了， 这个是非常重要的性质，即**索引的最左匹配特性**。 - - - -##### MyISAM主键索引与辅助索引的结构 - -MyISAM引擎的索引文件和数据文件是分离的。**MyISAM引擎索引结构的叶子节点的数据域，存放的并不是实际的数据记录，而是数据记录的地址**。索引文件与数据文件分离，这样的索引称为"**非聚簇索引**"。MyISAM的主索引与辅助索引区别并不大，只是主键索引不能有重复的关键字。 - -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gewoy5bddkj31bp0u04lv.jpg) - -在MyISAM中，索引（含叶子节点）存放在单独的.myi文件中，叶子节点存放的是数据的物理地址偏移量（通过偏移量访问就是随机访问，速度很快）。 - -主索引是指主键索引，键值不可能重复；辅助索引则是普通索引，键值可能重复。 - -通过索引查找数据的流程：先从索引文件中查找到索引节点，从中拿到数据的文件指针，再到数据文件中通过文件指针定位了具体的数据。辅助索引类似。 - - - -##### InnoDB主键索引与辅助索引的结构 - -**InnoDB引擎索引结构的叶子节点的数据域，存放的就是实际的数据记录**（对于主索引，此处会存放表中所有的数据记录；对于辅助索引此处会引用主键，检索的时候通过主键到主键索引中找到对应数据行），或者说，**InnoDB的数据文件本身就是主键索引文件**，这样的索引被称为"“聚簇索引”，一个表只能有一个聚簇索引。 - -###### 主键索引： - -我们知道InnoDB索引是聚集索引，它的索引和数据是存入同一个.idb文件中的，因此它的索引结构是在同一个树节点中同时存放索引和数据，如下图中最底层的叶子节点有三行数据，对应于数据表中的id、stu_id、name数据项。 - -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gewoy2lhr5j320d0u016k.jpg) - -在Innodb中，索引分叶子节点和非叶子节点，非叶子节点就像新华字典的目录，单独存放在索引段中，叶子节点则是顺序排列的，在数据段中。Innodb的数据文件可以按照表来切分（只需要开启`innodb_file_per_table)`，切分后存放在`xxx.ibd`中，默认不切分，存放在`xxx.ibdata`中。 - -###### 辅助（非主键）索引： - -这次我们以示例中学生表中的name列建立辅助索引，它的索引结构跟主键索引的结构有很大差别，在最底层的叶子结点有两行数据，第一行的字符串是辅助索引，按照ASCII码进行排序，第二行的整数是主键的值。 - -这就意味着，对name列进行条件搜索，需要两个步骤： - -① 在辅助索引上检索name，到达其叶子节点获取对应的主键； - -② 使用主键在主索引上再进行对应的检索操作 - -这也就是所谓的“**回表查询**” - -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gewsc7l623j320r0u0gwt.jpg) - - - -**InnoDB 索引结构需要注意的点** - -1. 数据文件本身就是索引文件 - -2. 表数据文件本身就是按 B+Tree 组织的一个索引结构文件 -3. 聚集索引中叶节点包含了完整的数据记录 -4. InnoDB 表必须要有主键，并且推荐使用整型自增主键 - -正如我们上面介绍 InnoDB 存储结构，索引与数据是共同存储的，不管是主键索引还是辅助索引，在查找时都是通过先查找到索引节点才能拿到相对应的数据，如果我们在设计表结构时没有显式指定索引列的话，MySQL 会从表中选择数据不重复的列建立索引，如果没有符合的列，则 MySQL 自动为 InnoDB 表生成一个隐含字段作为主键，并且这个字段长度为6个字节，类型为整型。 - -> 那为什么推荐使用整型自增主键而不是选择UUID？ - -- UUID是字符串，比整型消耗更多的存储空间； - -- 在B+树中进行查找时需要跟经过的节点值比较大小，整型数据的比较运算比字符串更快速； - -- 自增的整型索引在磁盘中会连续存储，在读取一页数据时也是连续；UUID是随机产生的，读取的上下两行数据存储是分散的，不适合执行where id > 5 && id < 20的条件查询语句。 - -- 在插入或删除数据时，整型自增主键会在叶子结点的末尾建立新的叶子节点，不会破坏左侧子树的结构；UUID主键很容易出现这样的情况，B+树为了维持自身的特性，有可能会进行结构的重构，消耗更多的时间。 - - -> 为什么非主键索引结构叶子节点存储的是主键值？ - -保证数据一致性和节省存储空间，可以这么理解：商城系统订单表会存储一个用户ID作为关联外键，而不推荐存储完整的用户信息，因为当我们用户表中的信息（真实名称、手机号、收货地址···）修改后，不需要再次维护订单表的用户数据，同时也节省了存储空间。 - - - -#### Hash索引 - -- 主要就是通过Hash算法（常见的Hash算法有直接定址法、平方取中法、折叠法、除数取余法、随机数法），将数据库字段数据转换成定长的Hash值，与这条数据的行指针一并存入Hash表的对应位置；如果发生Hash碰撞（两个不同关键字的Hash值相同），则在对应Hash键下以链表形式存储。 - - 检索算法：在检索查询时，就再次对待查关键字再次执行相同的Hash算法，得到Hash值，到对应Hash表对应位置取出数据即可，如果发生Hash碰撞，则需要在取值时进行筛选。目前使用Hash索引的数据库并不多，主要有Memory等。 - - MySQL目前有Memory引擎和NDB引擎支持Hash索引。 - - -#### full-text全文索引 - -- 全文索引也是MyISAM的一种特殊索引类型，主要用于全文索引，InnoDB从MYSQL5.6版本提供对全文索引的支持。 - -- 它用于替代效率较低的LIKE模糊匹配操作，而且可以通过多字段组合的全文索引一次性全模糊匹配多个字段。 -- 同样使用B-Tree存放索引数据，但使用的是特定的算法，将字段数据分割后再进行索引（一般每4个字节一次分割），索引文件存储的是分割前的索引字符串集合，与分割后的索引信息，对应Btree结构的节点存储的是分割后的词信息以及它在分割前的索引字符串集合中的位置。 - -#### R-Tree空间索引 - -空间索引是MyISAM的一种特殊索引类型，主要用于地理空间数据类型 - - - -> 为什么Mysql索引要用B+树不是B树？ - -用B+树不用B树考虑的是IO对性能的影响，B树的每个节点都存储数据，而B+树只有叶子节点才存储数据，所以查找相同数据量的情况下，B树的高度更高，IO更频繁。数据库索引是存储在磁盘上的，当数据量大时，就不能把整个索引全部加载到内存了，只能逐一加载每一个磁盘页（对应索引树的节点）。其中在MySQL底层对B+树进行进一步优化：在叶子节点中是双向链表，且在链表的头结点和尾节点也是循环指向的。 - - - -> 面试官：为何不采用Hash方式？ - -因为Hash索引底层是哈希表，哈希表是一种以key-value存储数据的结构，所以多个数据在存储关系上是完全没有任何顺序关系的，所以，对于区间查询是无法直接通过索引查询的，就需要全表扫描。所以，哈希索引只适用于等值查询的场景。而B+ Tree是一种多路平衡查询树，所以他的节点是天然有序的（左子节点小于父节点、父节点小于右子节点），所以对于范围查询的时候不需要做全表扫描。 - -哈希索引不支持多列联合索引的最左匹配规则，如果有大量重复键值得情况下，哈希索引的效率会很低，因为存在哈希碰撞问题。 - - - -### 哪些情况需要创建索引 - -1. 主键自动建立唯一索引 - -2. 频繁作为查询条件的字段 - -3. 查询中与其他表关联的字段，外键关系建立索引 - -4. 单键/组合索引的选择问题，高并发下倾向创建组合索引 - -5. 查询中排序的字段，排序字段通过索引访问大幅提高排序速度 - -6. 查询中统计或分组字段 - - - -### 哪些情况不要创建索引 - -1. 表记录太少 -2. 经常增删改的表 -3. 数据重复且分布均匀的表字段，只应该为最经常查询和最经常排序的数据列建立索引（如果某个数据类包含太多的重复数据，建立索引没有太大意义） -4. 频繁更新的字段不适合创建索引（会加重IO负担） -5. where条件里用不到的字段不创建索引 - - - -### MySQL高效索引 - -**覆盖索引**（Covering Index）,或者叫索引覆盖， 也就是平时所说的不需要回表操作 - -- 就是select的数据列只用从索引中就能够取得，不必读取数据行，MySQL可以利用索引返回select列表中的字段，而不必根据索引再次读取数据文件，换句话说**查询列要被所建的索引覆盖**。 - -- 索引是高效找到行的一个方法，但是一般数据库也能使用索引找到一个列的数据，因此它不必读取整个行。毕竟索引叶子节点存储了它们索引的数据，当能通过读取索引就可以得到想要的数据，那就不需要读取行了。一个索引包含（覆盖）满足查询结果的数据就叫做覆盖索引。 - -- **判断标准** - - 使用explain，可以通过输出的extra列来判断，对于一个索引覆盖查询，显示为**using index**，MySQL查询优化器在执行查询前会决定是否有索引覆盖查询 - - - -## 五、MySQL查询 - -> count(*) 和 count(1)和count(列名)区别 ps：这道题说法有点多 - -执行效果上： - -- count(*)包括了所有的列，相当于行数，在统计结果的时候，不会忽略列值为NULL -- count(1)包括了所有列，用1代表代码行，在统计结果的时候，不会忽略列值为NULL -- count(列名)只包括列名那一列，在统计结果的时候，会忽略列值为空（这里的空不是只空字符串或者0，而是表示null）的计数，即某个字段值为NULL时，不统计。 - -执行效率上： - -- 列名为主键，count(列名)会比count(1)快 -- 列名不为主键，count(1)会比count(列名)快 -- 如果表多个列并且没有主键，则 count(1) 的执行效率优于 count(*) -- 如果有主键，则 select count（主键）的执行效率是最优的 -- 如果表只有一个字段，则 select count(*) 最优。 - - - -> MySQL中 in和 exists 的区别？ - -- exists：exists对外表用loop逐条查询，每次查询都会查看exists的条件语句，当exists里的条件语句能够返回记录行时（无论记录行是的多少，只要能返回），条件就为真，返回当前loop到的这条记录；反之，如果exists里的条件语句不能返回记录行，则当前loop到的这条记录被丢弃，exists的条件就像一个bool条件，当能返回结果集则为true，不能返回结果集则为false -- in：in查询相当于多个or条件的叠加 - -```mysql -SELECT * FROM A WHERE A.id IN (SELECT id FROM B); -SELECT * FROM A WHERE EXISTS (SELECT * from B WHERE B.id = A.id); -``` - -**如果查询的两个表大小相当，那么用in和exists差别不大**。 - -如果两个表中一个较小，一个是大表，则子查询表大的用exists，子查询表小的用in： - - - -> UNION和UNION ALL的区别? - -UNION和UNION ALL都是将两个结果集合并为一个，**两个要联合的SQL语句 字段个数必须一样，而且字段类型要“相容”（一致）；** - -- UNION在进行表连接后会筛选掉重复的数据记录（效率较低），而UNION ALL则不会去掉重复的数据记录； - -- UNION会按照字段的顺序进行排序，而UNION ALL只是简单的将两个结果合并就返回； - - - -### SQL执行顺序 - -- 手写 - - ```mysql - SELECT DISTINCT - FROM - JOIN ON - WHERE - GROUP BY - HAVING - ORDER BY - LIMIT - ``` - -- 机读 - - ```mysql - FROM - ON - JOIN - WHERE - GROUP BY - HAVING - SELECT - DISTINCT - ORDER BY - LIMIT - ``` - -- 总结 - - ![sql-parse](https://tva1.sinaimg.cn/large/007S8ZIlly1gf3t8jyy81j30s2083wg2.jpg) - - - -> mysql 的内连接、左连接、右连接有什么区别？ -> -> 什么是内连接、外连接、交叉连接、笛卡尔积呢？ - -### Join图 - -![sql-joins](https://tva1.sinaimg.cn/large/007S8ZIlly1gf3t8novxpj30qu0l4wi7.jpg) - ------- - - - -## 六、MySQL 事务 - -> 事务的隔离级别有哪些？MySQL的默认隔离级别是什么？ -> -> 什么是幻读，脏读，不可重复读呢？ -> -> MySQL事务的四大特性以及实现原理 -> -> MVCC熟悉吗，它的底层原理？ - -MySQL 事务主要用于处理操作量大，复杂度高的数据。比如说，在人员管理系统中，你删除一个人员，你即需要删除人员的基本资料，也要删除和该人员相关的信息，如信箱，文章等等，这样，这些数据库操作语句就构成一个事务！ - - - -### ACID — 事务基本要素 - -![](https://tva1.sinaimg.cn/large/007S8ZIlly1geu10kkswnj305q05mweo.jpg) - -事务是由一组SQL语句组成的逻辑处理单元，具有4个属性，通常简称为事务的ACID属性。 - -- **A (Atomicity) 原子性**：整个事务中的所有操作，要么全部完成，要么全部不完成，不可能停滞在中间某个环节。事务在执行过程中发生错误，会被回滚（Rollback）到事务开始前的状态，就像这个事务从来没有执行过一样 -- **C (Consistency) 一致性**：在事务开始之前和事务结束以后，数据库的完整性约束没有被破坏 -- **I (Isolation)隔离性**：一个事务的执行不能其它事务干扰。即一个事务内部的操作及使用的数据对其它并发事务是隔离的，并发执行的各个事务之间不能互相干扰 -- **D (Durability) 持久性**：在事务完成以后，该事务所对数据库所作的更改便持久的保存在数据库之中，并不会被回滚 - - - -**并发事务处理带来的问题** - -- 更新丢失（Lost Update)： 事务A和事务B选择同一行，然后基于最初选定的值更新该行时，由于两个事务都不知道彼此的存在，就会发生丢失更新问题 -- 脏读(Dirty Reads)：事务A读取了事务B更新的数据，然后B回滚操作，那么A读取到的数据是脏数据 -- 不可重复读（Non-Repeatable Reads)：事务 A 多次读取同一数据，事务B在事务A多次读取的过程中，对数据作了更新并提交，导致事务A多次读取同一数据时，结果不一致。 -- 幻读（Phantom Reads)：幻读与不可重复读类似。它发生在一个事务A读取了几行数据，接着另一个并发事务B插入了一些数据时。在随后的查询中，事务A就会发现多了一些原本不存在的记录，就好像发生了幻觉一样，所以称为幻读。 - - - -**幻读和不可重复读的区别：** - -- **不可重复读的重点是修改**：在同一事务中，同样的条件，第一次读的数据和第二次读的数据不一样。（因为中间有其他事务提交了修改） -- **幻读的重点在于新增或者删除**：在同一事务中，同样的条件,，第一次和第二次读出来的记录数不一样。（因为中间有其他事务提交了插入/删除） - - - -**并发事务处理带来的问题的解决办法：** - -- “更新丢失”通常是应该完全避免的。但防止更新丢失，并不能单靠数据库事务控制器来解决，需要应用程序对要更新的数据加必要的锁来解决，因此，防止更新丢失应该是应用的责任。 - -- “脏读” 、 “不可重复读”和“幻读” ，其实都是数据库读一致性问题，必须由数据库提供一定的事务隔离机制来解决： - - - 一种是加锁：在读取数据前，对其加锁，阻止其他事务对数据进行修改。 - - 另一种是数据多版本并发控制（MultiVersion Concurrency Control，简称 **MVCC** 或 MCC），也称为多版本数据库：不用加任何锁， 通过一定机制生成一个数据请求时间点的一致性数据快照 （Snapshot)， 并用这个快照来提供一定级别 （语句级或事务级） 的一致性读取。从用户的角度来看，好象是数据库可以提供同一数据的多个版本。 - - - -### 事务隔离级别 - -数据库事务的隔离级别有4种，由低到高分别为 - -- **READ-UNCOMMITTED(读未提交)：** 最低的隔离级别，允许读取尚未提交的数据变更，**可能会导致脏读、幻读或不可重复读**。 -- **READ-COMMITTED(读已提交)：** 允许读取并发事务已经提交的数据，**可以阻止脏读，但是幻读或不可重复读仍有可能发生**。 -- **REPEATABLE-READ(可重复读)：** 对同一字段的多次读取结果都是一致的，除非数据是被本身事务自己所修改，**可以阻止脏读和不可重复读，但幻读仍有可能发生**。 -- **SERIALIZABLE(可串行化)：** 最高的隔离级别，完全服从ACID的隔离级别。所有的事务依次逐个执行，这样事务之间就完全不可能产生干扰，也就是说，**该级别可以防止脏读、不可重复读以及幻读**。 - -查看当前数据库的事务隔离级别： - -```mysql -show variables like 'tx_isolation' -``` - -下面通过事例一一阐述在事务的并发操作中可能会出现脏读，不可重复读，幻读和事务隔离级别的联系。 - -数据库的事务隔离越严格，并发副作用越小，但付出的代价就越大，因为事务隔离实质上就是使事务在一定程度上“串行化”进行，这显然与“并发”是矛盾的。同时，不同的应用对读一致性和事务隔离程度的要求也是不同的，比如许多应用对“不可重复读”和“幻读”并不敏感，可能更关心数据并发访问的能力。 - -#### Read uncommitted - -读未提交，就是一个事务可以读取另一个未提交事务的数据。 - -事例：老板要给程序员发工资，程序员的工资是3.6万/月。但是发工资时老板不小心按错了数字，按成3.9万/月，该钱已经打到程序员的户口，但是事务还没有提交，就在这时，程序员去查看自己这个月的工资，发现比往常多了3千元，以为涨工资了非常高兴。但是老板及时发现了不对，马上回滚差点就提交了的事务，将数字改成3.6万再提交。 - -分析：实际程序员这个月的工资还是3.6万，但是程序员看到的是3.9万。他看到的是老板还没提交事务时的数据。这就是脏读。 - -那怎么解决脏读呢？Read committed！读提交，能解决脏读问题。 - -#### Read committed - -读提交，顾名思义，就是一个事务要等另一个事务提交后才能读取数据。 - -事例：程序员拿着信用卡去享受生活（卡里当然是只有3.6万），当他埋单时（程序员事务开启），收费系统事先检测到他的卡里有3.6万，就在这个时候！！程序员的妻子要把钱全部转出充当家用，并提交。当收费系统准备扣款时，再检测卡里的金额，发现已经没钱了（第二次检测金额当然要等待妻子转出金额事务提交完）。程序员就会很郁闷，明明卡里是有钱的… - -分析：这就是读提交，若有事务对数据进行更新（UPDATE）操作时，读操作事务要等待这个更新操作事务提交后才能读取数据，可以解决脏读问题。但在这个事例中，出现了一个事务范围内两个相同的查询却返回了不同数据，这就是**不可重复读**。 - -那怎么解决可能的不可重复读问题？Repeatable read ！ - -#### Repeatable read - -重复读，就是在开始读取数据（事务开启）时，不再允许修改操作。 **MySQL的默认事务隔离级别** - -事例：程序员拿着信用卡去享受生活（卡里当然是只有3.6万），当他埋单时（事务开启，不允许其他事务的UPDATE修改操作），收费系统事先检测到他的卡里有3.6万。这个时候他的妻子不能转出金额了。接下来收费系统就可以扣款了。 - -分析：重复读可以解决不可重复读问题。写到这里，应该明白的一点就是，**不可重复读对应的是修改，即UPDATE操作。但是可能还会有幻读问题。因为幻读问题对应的是插入INSERT操作，而不是UPDATE操作**。 - -**什么时候会出现幻读？** - -事例：程序员某一天去消费，花了2千元，然后他的妻子去查看他今天的消费记录（全表扫描FTS，妻子事务开启），看到确实是花了2千元，就在这个时候，程序员花了1万买了一部电脑，即新增INSERT了一条消费记录，并提交。当妻子打印程序员的消费记录清单时（妻子事务提交），发现花了1.2万元，似乎出现了幻觉，这就是幻读。 - -那怎么解决幻读问题？Serializable！ - -#### Serializable 序列化 - -Serializable 是最高的事务隔离级别，在该级别下，事务串行化顺序执行，可以避免脏读、不可重复读与幻读。简单来说，Serializable会在读取的每一行数据上都加锁，所以可能导致大量的超时和锁争用问题。这种事务隔离级别效率低下，比较耗数据库性能，一般不使用。 - - - -#### 比较 - -| 事务隔离级别 | 读数据一致性 | 脏读 | 不可重复读 | 幻读 | -| ---------------------------- | ---------------------------------------- | ---- | ---------- | ---- | -| 读未提交（read-uncommitted） | 最低级被，只能保证不读取物理上损坏的数据 | 是 | 是 | 是 | -| 读已提交（read-committed） | 语句级 | 否 | 是 | 是 | -| 可重复读（repeatable-read） | 事务级 | 否 | 否 | 是 | -| 串行化（serializable） | 最高级别，事务级 | 否 | 否 | 否 | - -需要说明的是，事务隔离级别和数据访问的并发性是对立的，事务隔离级别越高并发性就越差。所以要根据具体的应用来确定合适的事务隔离级别，这个地方没有万能的原则。 - - - -MySQL InnoDB 存储引擎的默认支持的隔离级别是 **REPEATABLE-READ（可重读）**。我们可以通过`SELECT @@tx_isolation;`命令来查看，MySQL 8.0 该命令改为`SELECT @@transaction_isolation;` - -这里需要注意的是：与 SQL 标准不同的地方在于InnoDB 存储引擎在 **REPEATABLE-READ（可重读）**事务隔离级别下使用的是Next-Key Lock 算法，因此可以避免幻读的产生，这与其他数据库系统(如 SQL Server)是不同的。所以说InnoDB 存储引擎的默认支持的隔离级别是 REPEATABLE-READ（可重读）已经可以完全保证事务的隔离性要求，即达到了 SQL标准的 **SERIALIZABLE(可串行化)**隔离级别，而且保留了比较好的并发性能。 - -因为隔离级别越低，事务请求的锁越少，所以大部分数据库系统的隔离级别都是**READ-COMMITTED(读已提交):**，但是你要知道的是InnoDB 存储引擎默认使用 **REPEATABLE-READ（可重读）**并不会有任何性能损失。 - - - -### MVCC 多版本并发控制 - -MySQL的大多数事务型存储引擎实现都不是简单的行级锁。基于提升并发性考虑，一般都同时实现了多版本并发控制（MVCC），包括Oracle、PostgreSQL。只是实现机制各不相同。 - -可以认为 MVCC 是行级锁的一个变种，但它在很多情况下避免了加锁操作，因此开销更低。虽然实现机制有所不同，但大都实现了非阻塞的读操作，写操作也只是锁定必要的行。 - -MVCC 的实现是通过保存数据在某个时间点的快照来实现的。也就是说不管需要执行多长时间，每个事物看到的数据都是一致的。 - -典型的MVCC实现方式，分为**乐观（optimistic）并发控制和悲观（pressimistic）并发控制**。下边通过 InnoDB的简化版行为来说明 MVCC 是如何工作的。 - -InnoDB 的 MVCC，是通过在每行记录后面保存两个隐藏的列来实现。这两个列，一个保存了行的创建时间，一个保存行的过期时间（删除时间）。当然存储的并不是真实的时间，而是系统版本号（system version number）。每开始一个新的事务，系统版本号都会自动递增。事务开始时刻的系统版本号会作为事务的版本号，用来和查询到的每行记录的版本号进行比较。 - -**REPEATABLE READ（可重读）隔离级别下MVCC如何工作：** - -- SELECT - - InnoDB会根据以下两个条件检查每行记录： - - - InnoDB只查找版本早于当前事务版本的数据行，这样可以确保事务读取的行，要么是在开始事务之前已经存在要么是事务自身插入或者修改过的 - - - 行的删除版本号要么未定义，要么大于当前事务版本号，这样可以确保事务读取到的行在事务开始之前未被删除 - - 只有符合上述两个条件的才会被查询出来 - -- INSERT：InnoDB为新插入的每一行保存当前系统版本号作为行版本号 - -- DELETE：InnoDB为删除的每一行保存当前系统版本号作为行删除标识 - -- UPDATE：InnoDB为插入的一行新纪录保存当前系统版本号作为行版本号，同时保存当前系统版本号到原来的行作为删除标识 - - -保存这两个额外系统版本号，使大多数操作都不用加锁。使数据操作简单，性能很好，并且也能保证只会读取到符合要求的行。不足之处是每行记录都需要额外的存储空间，需要做更多的行检查工作和一些额外的维护工作。 - -MVCC 只在 COMMITTED READ（读提交）和REPEATABLE READ（可重复读）两种隔离级别下工作。 - - - -### 事务日志 - -InnoDB 使用日志来减少提交事务时的开销。因为日志中已经记录了事务，就无须在每个事务提交时把缓冲池的脏块刷新(flush)到磁盘中。 - -事务修改的数据和索引通常会映射到表空间的随机位置，所以刷新这些变更到磁盘需要很多随机 IO。 - -InnoDB 假设使用常规磁盘，随机IO比顺序IO昂贵得多，因为一个IO请求需要时间把磁头移到正确的位置，然后等待磁盘上读出需要的部分，再转到开始位置。 - -InnoDB 用日志把随机IO变成顺序IO。一旦日志安全写到磁盘，事务就持久化了，即使断电了，InnoDB可以重放日志并且恢复已经提交的事务。 - -InnoDB 使用一个后台线程智能地刷新这些变更到数据文件。这个线程可以批量组合写入，使得数据写入更顺序，以提高效率。 - -事务日志可以帮助提高事务效率： - -- 使用事务日志，存储引擎在修改表的数据时只需要修改其内存拷贝，再把该修改行为记录到持久在硬盘上的事务日志中，而不用每次都将修改的数据本身持久到磁盘。 -- 事务日志采用的是追加的方式，因此写日志的操作是磁盘上一小块区域内的顺序I/O，而不像随机I/O需要在磁盘的多个地方移动磁头，所以采用事务日志的方式相对来说要快得多。 -- 事务日志持久以后，内存中被修改的数据在后台可以慢慢刷回到磁盘。 -- 如果数据的修改已经记录到事务日志并持久化，但数据本身没有写回到磁盘，此时系统崩溃，存储引擎在重启时能够自动恢复这一部分修改的数据。 - -目前来说，大多数存储引擎都是这样实现的，我们通常称之为**预写式日志**（Write-Ahead Logging），修改数据需要写两次磁盘。 - - - -### 事务的实现 - -事务的实现是基于数据库的存储引擎。不同的存储引擎对事务的支持程度不一样。MySQL 中支持事务的存储引擎有 InnoDB 和 NDB。 - -事务的实现就是如何实现ACID特性。 - -事务的隔离性是通过锁实现，而事务的原子性、一致性和持久性则是通过事务日志实现 。 - - - -> 事务是如何通过日志来实现的，说得越深入越好。 - -事务日志包括：**重做日志redo**和**回滚日志undo** - -- **redo log（重做日志**） 实现持久化和原子性 - - 在innoDB的存储引擎中，事务日志通过重做(redo)日志和innoDB存储引擎的日志缓冲(InnoDB Log Buffer)实现。事务开启时，事务中的操作，都会先写入存储引擎的日志缓冲中，在事务提交之前，这些缓冲的日志都需要提前刷新到磁盘上持久化，这就是DBA们口中常说的“日志先行”(Write-Ahead Logging)。当事务提交之后，在Buffer Pool中映射的数据文件才会慢慢刷新到磁盘。此时如果数据库崩溃或者宕机，那么当系统重启进行恢复时，就可以根据redo log中记录的日志，把数据库恢复到崩溃前的一个状态。未完成的事务，可以继续提交，也可以选择回滚，这基于恢复的策略而定。 - - 在系统启动的时候，就已经为redo log分配了一块连续的存储空间，以顺序追加的方式记录Redo Log，通过顺序IO来改善性能。所有的事务共享redo log的存储空间，它们的Redo Log按语句的执行顺序，依次交替的记录在一起。 - -- **undo log（回滚日志）** 实现一致性 - - undo log 主要为事务的回滚服务。在事务执行的过程中，除了记录redo log，还会记录一定量的undo log。undo log记录了数据在每个操作前的状态，如果事务执行过程中需要回滚，就可以根据undo log进行回滚操作。单个事务的回滚，只会回滚当前事务做的操作，并不会影响到其他的事务做的操作。 - - Undo记录的是已部分完成并且写入硬盘的未完成的事务，默认情况下回滚日志是记录下表空间中的（共享表空间或者独享表空间） - -二种日志均可以视为一种恢复操作，redo_log是恢复提交事务修改的页操作，而undo_log是回滚行记录到特定版本。二者记录的内容也不同，redo_log是物理日志，记录页的物理修改操作，而undo_log是逻辑日志，根据每行记录进行记录。 - - - -> 又引出个问题：你知道MySQL 有多少种日志吗？ - -- **错误日志**：记录出错信息，也记录一些警告信息或者正确的信息。 - -- **查询日志**：记录所有对数据库请求的信息，不论这些请求是否得到了正确的执行。 - -- **慢查询日志**：设置一个阈值，将运行时间超过该值的所有SQL语句都记录到慢查询的日志文件中。 - -- **二进制日志**：记录对数据库执行更改的所有操作。 - -- **中继日志**：中继日志也是二进制日志，用来给slave 库恢复 - -- **事务日志**：重做日志redo和回滚日志undo - - - -> 分布式事务相关问题，可能还会问到 2PC、3PC，，， - -### MySQL对分布式事务的支持 - -分布式事务的实现方式有很多，既可以采用 InnoDB 提供的原生的事务支持，也可以采用消息队列来实现分布式事务的最终一致性。这里我们主要聊一下 InnoDB 对分布式事务的支持。 - -MySQL 从 5.0.3 InnoDB 存储引擎开始支持XA协议的分布式事务。一个分布式事务会涉及多个行动，这些行动本身是事务性的。所有行动都必须一起成功完成，或者一起被回滚。 - -在MySQL中，使用分布式事务涉及一个或多个资源管理器和一个事务管理器。 - -![](https://imgkr.cn-bj.ufileos.com/8d48c5e1-c849-413e-8e5a-e96529235f58.png) - -如图，MySQL 的分布式事务模型。模型中分三块：应用程序（AP）、资源管理器（RM）、事务管理器（TM）: - -- 应用程序：定义了事务的边界，指定需要做哪些事务； -- 资源管理器：提供了访问事务的方法，通常一个数据库就是一个资源管理器； -- 事务管理器：协调参与了全局事务中的各个事务。 - -分布式事务采用两段式提交（two-phase commit）的方式： - -- 第一阶段所有的事务节点开始准备，告诉事务管理器ready。 -- 第二阶段事务管理器告诉每个节点是commit还是rollback。如果有一个节点失败，就需要全局的节点全部rollback，以此保障事务的原子性。 - ------- - - - -## 七、MySQL锁机制 - -> 数据库的乐观锁和悲观锁？ -> -> MySQL 中有哪几种锁，列举一下？ -> -> MySQL中InnoDB引擎的行锁是怎么实现的？ -> -> MySQL 间隙锁有没有了解，死锁有没有了解，写一段会造成死锁的 sql 语句，死锁发生了如何解决，MySQL 有没有提供什么机制去解决死锁 - -锁是计算机协调多个进程或线程并发访问某一资源的机制。 - -在数据库中，除传统的计算资源（如CPU、RAM、I/O等）的争用以外，数据也是一种供许多用户共享的资源。数据库锁定机制简单来说，就是数据库为了保证数据的一致性，而使各种共享资源在被并发访问变得有序所设计的一种规则。 - -打个比方，我们到淘宝上买一件商品，商品只有一件库存，这个时候如果还有另一个人买，那么如何解决是你买到还是另一个人买到的问题？这里肯定要用到事物，我们先从库存表中取出物品数量，然后插入订单，付款后插入付款表信息，然后更新商品数量。在这个过程中，使用锁可以对有限的资源进行保护，解决隔离和并发的矛盾。 - - - -### 锁的分类 - -**从对数据操作的类型分类**： - -- **读锁**（共享锁）：针对同一份数据，多个读操作可以同时进行，不会互相影响 - -- **写锁**（排他锁）：当前写操作没有完成前，它会阻断其他写锁和读锁 - -**从对数据操作的粒度分类**： - - -为了尽可能提高数据库的并发度，每次锁定的数据范围越小越好，理论上每次只锁定当前操作的数据的方案会得到最大的并发度，但是管理锁是很耗资源的事情（涉及获取，检查，释放锁等动作），因此数据库系统需要在高并发响应和系统性能两方面进行平衡，这样就产生了“锁粒度（Lock granularity）”的概念。 - -- **表级锁**：开销小，加锁快；不会出现死锁；锁定粒度大，发生锁冲突的概率最高，并发度最低（MyISAM 和 MEMORY 存储引擎采用的是表级锁）； - -- **行级锁**：开销大，加锁慢；会出现死锁；锁定粒度最小，发生锁冲突的概率最低，并发度也最高（InnoDB 存储引擎既支持行级锁也支持表级锁，但默认情况下是采用行级锁）； - -- **页面锁**：开销和加锁时间界于表锁和行锁之间；会出现死锁；锁定粒度界于表锁和行锁之间，并发度一般。 - -适用：从锁的角度来说，表级锁更适合于以查询为主，只有少量按索引条件更新数据的应用，如Web应用；而行级锁则更适合于有大量按索引条件并发更新少量不同数据，同时又有并发查询的应用，如一些在线事务处理（OLTP）系统。 - -| | 行锁 | 表锁 | 页锁 | -| ------ | ---- | ---- | ---- | -| MyISAM | | √ | | -| BDB | | √ | √ | -| InnoDB | √ | √ | | -| Memory | | √ | | - - - -### MyISAM 表锁 - -MyISAM 的表锁有两种模式： - -- 表共享读锁 （Table Read Lock）：不会阻塞其他用户对同一表的读请求，但会阻塞对同一表的写请求； -- 表独占写锁 （Table Write Lock）：会阻塞其他用户对同一表的读和写操作； - -MyISAM 表的读操作与写操作之间，以及写操作之间是串行的。当一个线程获得对一个表的写锁后， 只有持有锁的线程可以对表进行更新操作。 其他线程的读、 写操作都会等待，直到锁被释放为止。 - -默认情况下，写锁比读锁具有更高的优先级：当一个锁释放时，这个锁会优先给写锁队列中等候的获取锁请求，然后再给读锁队列中等候的获取锁请求。 - - - -### InnoDB 行锁 - -InnoDB 实现了以下两种类型的**行锁**： - -- 共享锁（S）：允许一个事务去读一行，阻止其他事务获得相同数据集的排他锁。 -- 排他锁（X）：允许获得排他锁的事务更新数据，阻止其他事务取得相同数据集的共享读锁和排他写锁。 - -为了允许行锁和表锁共存，实现多粒度锁机制，InnoDB 还有两种内部使用的意向锁（Intention Locks），这两种意向锁都是**表锁**： - -- 意向共享锁（IS）：事务打算给数据行加行共享锁，事务在给一个数据行加共享锁前必须先取得该表的 IS 锁。 -- 意向排他锁（IX）：事务打算给数据行加行排他锁，事务在给一个数据行加排他锁前必须先取得该表的 IX 锁。 - -**索引失效会导致行锁变表锁**。比如 vchar 查询不写单引号的情况。 - -#### 加锁机制 - -**乐观锁与悲观锁是两种并发控制的思想，可用于解决丢失更新问题** - -乐观锁会“乐观地”假定大概率不会发生并发更新冲突，访问、处理数据过程中不加锁，只在更新数据时再根据版本号或时间戳判断是否有冲突，有则处理，无则提交事务。用数据版本（Version）记录机制实现，这是乐观锁最常用的一种实现方式 - -悲观锁会“悲观地”假定大概率会发生并发更新冲突，访问、处理数据前就加排他锁，在整个数据处理过程中锁定数据，事务提交或回滚后才释放锁。另外与乐观锁相对应的，**悲观锁是由数据库自己实现了的，要用的时候，我们直接调用数据库的相关语句就可以了。** - - - -#### 锁模式(InnoDB有三种行锁的算法) - -- **记录锁(Record Locks)**： 单个行记录上的锁。对索引项加锁，锁定符合条件的行。其他事务不能修改和删除加锁项； - - ```mysql - SELECT * FROM table WHERE id = 1 FOR UPDATE; - ``` - - 它会在 id=1 的记录上加上记录锁，以阻止其他事务插入，更新，删除 id=1 这一行 - - 在通过 主键索引 与 唯一索引 对数据行进行 UPDATE 操作时，也会对该行数据加记录锁： - - ```mysql - -- id 列为主键列或唯一索引列 - UPDATE SET age = 50 WHERE id = 1; - ``` - -- **间隙锁（Gap Locks）**： 当我们使用范围条件而不是相等条件检索数据，并请求共享或排他锁时，InnoDB会给符合条件的已有数据记录的索引项加锁。对于键值在条件范围内但并不存在的记录，叫做“间隙”。 - - InnoDB 也会对这个“间隙”加锁，这种锁机制就是所谓的间隙锁。 - - 对索引项之间的“间隙”加锁，锁定记录的范围（对第一条记录前的间隙或最后一条将记录后的间隙加锁），不包含索引项本身。其他事务不能在锁范围内插入数据，这样就防止了别的事务新增幻影行。 - - 间隙锁基于非唯一索引，它锁定一段范围内的索引记录。间隙锁基于下面将会提到的`Next-Key Locking` 算法，请务必牢记：**使用间隙锁锁住的是一个区间，而不仅仅是这个区间中的每一条数据**。 - - ```mysql - SELECT * FROM table WHERE id BETWEN 1 AND 10 FOR UPDATE; - ``` - - 即所有在`（1，10）`区间内的记录行都会被锁住，所有id 为 2、3、4、5、6、7、8、9 的数据行的插入会被阻塞，但是 1 和 10 两条记录行并不会被锁住。 - - GAP锁的目的，是为了防止同一事务的两次当前读，出现幻读的情况 - -- **临键锁(Next-key Locks)**： **临键锁**，是**记录锁与间隙锁的组合**，它的封锁范围，既包含索引记录，又包含索引区间。(临键锁的主要目的，也是为了避免**幻读**(Phantom Read)。如果把事务的隔离级别降级为RC，临键锁则也会失效。) - - Next-Key 可以理解为一种特殊的**间隙锁**，也可以理解为一种特殊的**算法**。通过**临建锁**可以解决幻读的问题。 每个数据行上的非唯一索引列上都会存在一把临键锁，当某个事务持有该数据行的临键锁时，会锁住一段左开右闭区间的数据。需要强调的一点是，`InnoDB` 中行级锁是基于索引实现的，临键锁只与非唯一索引列有关，在唯一索引列（包括主键列）上不存在临键锁。 - - 对于行的查询，都是采用该方法，主要目的是解决幻读的问题。 - -> select for update有什么含义，会锁表还是锁行还是其他 - -for update 仅适用于InnoDB，且必须在事务块(BEGIN/COMMIT)中才能生效。在进行事务操作时，通过“for update”语句，MySQL会对查询结果集中每行数据都添加排他锁，其他线程对该记录的更新与删除操作都会阻塞。排他锁包含行锁、表锁。 - -InnoDB这种行锁实现特点意味着：只有通过索引条件检索数据，InnoDB才使用行级锁，否则，InnoDB将使用表锁！ -假设有个表单 products ，里面有id跟name二个栏位，id是主键。 - -- 明确指定主键，并且有此笔资料，row lock - -```mysql -SELECT * FROM products WHERE id='3' FOR UPDATE; -SELECT * FROM products WHERE id='3' and type=1 FOR UPDATE; -``` - -- 明确指定主键，若查无此笔资料，无lock - -```mysql -SELECT * FROM products WHERE id='-1' FOR UPDATE; -``` - -- 无主键，table lock - -```mysql -SELECT * FROM products WHERE name='Mouse' FOR UPDATE; -``` - -- 主键不明确，table lock - -```mysql -SELECT * FROM products WHERE id<>'3' FOR UPDATE; -``` - -- 主键不明确，table lock - -```mysql -SELECT * FROM products WHERE id LIKE '3' FOR UPDATE; -``` - -**注1**: FOR UPDATE仅适用于InnoDB，且必须在交易区块(BEGIN/COMMIT)中才能生效。 -**注2**: 要测试锁定的状况，可以利用MySQL的Command Mode ，开二个视窗来做测试。 - - - -> MySQL 遇到过死锁问题吗，你是如何解决的？ - -### 死锁 - -**死锁产生**： - -- 死锁是指两个或多个事务在同一资源上相互占用，并请求锁定对方占用的资源，从而导致恶性循环 -- 当事务试图以不同的顺序锁定资源时，就可能产生死锁。多个事务同时锁定同一个资源时也可能会产生死锁 -- 锁的行为和顺序和存储引擎相关。以同样的顺序执行语句，有些存储引擎会产生死锁有些不会——死锁有双重原因：真正的数据冲突；存储引擎的实现方式。 - -**检测死锁**：数据库系统实现了各种死锁检测和死锁超时的机制。InnoDB存储引擎能检测到死锁的循环依赖并立即返回一个错误。 - -**死锁恢复**：死锁发生以后，只有部分或完全回滚其中一个事务，才能打破死锁，InnoDB目前处理死锁的方法是，将持有最少行级排他锁的事务进行回滚。所以事务型应用程序在设计时必须考虑如何处理死锁，多数情况下只需要重新执行因死锁回滚的事务即可。 - -**外部锁的死锁检测**：发生死锁后，InnoDB 一般都能自动检测到，并使一个事务释放锁并回退，另一个事务获得锁，继续完成事务。但在涉及外部锁，或涉及表锁的情况下，InnoDB 并不能完全自动检测到死锁， 这需要通过设置锁等待超时参数 innodb_lock_wait_timeout 来解决 - -**死锁影响性能**：死锁会影响性能而不是会产生严重错误，因为InnoDB会自动检测死锁状况并回滚其中一个受影响的事务。在高并发系统上，当许多线程等待同一个锁时，死锁检测可能导致速度变慢。 有时当发生死锁时，禁用死锁检测（使用innodb_deadlock_detect配置选项）可能会更有效，这时可以依赖`innodb_lock_wait_timeout`设置进行事务回滚。 - - - -**MyISAM避免死锁**： - -- 在自动加锁的情况下，MyISAM 总是一次获得 SQL 语句所需要的全部锁，所以 MyISAM 表不会出现死锁。 - -**InnoDB避免死锁**： - -- 为了在单个InnoDB表上执行多个并发写入操作时避免死锁，可以在事务开始时通过为预期要修改的每个元祖（行）使用`SELECT ... FOR UPDATE`语句来获取必要的锁，即使这些行的更改语句是在之后才执行的。 -- 在事务中，如果要更新记录，应该直接申请足够级别的锁，即排他锁，而不应先申请共享锁、更新时再申请排他锁，因为这时候当用户再申请排他锁时，其他事务可能又已经获得了相同记录的共享锁，从而造成锁冲突，甚至死锁 -- 如果事务需要修改或锁定多个表，则应在每个事务中以相同的顺序使用加锁语句。 在应用中，如果不同的程序会并发存取多个表，应尽量约定以相同的顺序来访问表，这样可以大大降低产生死锁的机会 -- 通过`SELECT ... LOCK IN SHARE MODE`获取行的读锁后，如果当前事务再需要对该记录进行更新操作，则很有可能造成死锁。 -- 改变事务隔离级别 - -如果出现死锁，可以用 `show engine innodb status; `命令来确定最后一个死锁产生的原因。返回结果中包括死锁相关事务的详细信息，如引发死锁的 SQL 语句，事务已经获得的锁，正在等待什么锁，以及被回滚的事务等。据此可以分析死锁产生的原因和改进措施。 - ------- - - - -## 八、MySQL调优 - -> 日常工作中你是怎么优化SQL的？ -> -> SQL优化的一般步骤是什么，怎么看执行计划（explain），如何理解其中各个字段的含义？ -> -> 如何写sql能够有效的使用到复合索引？ -> -> 一条sql执行过长的时间，你如何优化，从哪些方面入手？ -> -> 什么是最左前缀原则？什么是最左匹配原则？ - -### 影响mysql的性能因素 - -- 业务需求对MySQL的影响(合适合度) - -- 存储定位对MySQL的影响 - - 不适合放进MySQL的数据 - - 二进制多媒体数据 - - 流水队列数据 - - 超大文本数据 - - 需要放进缓存的数据 - - 系统各种配置及规则数据 - - 活跃用户的基本信息数据 - - 活跃用户的个性化定制信息数据 - - 准实时的统计信息数据 - - 其他一些访问频繁但变更较少的数据 - -- Schema设计对系统的性能影响 - - 尽量减少对数据库访问的请求 - - 尽量减少无用数据的查询请求 - -- 硬件环境对系统性能的影响 - - - -### 性能分析 - -#### MySQL Query Optimizer - -1. MySQL 中有专门负责优化 SELECT 语句的优化器模块，主要功能：通过计算分析系统中收集到的统计信息，为客户端请求的 Query 提供他认为最优的执行计划（他认为最优的数据检索方式，但不见得是 DBA 认为是最优的，这部分最耗费时间） - -2. 当客户端向 MySQL 请求一条 Query，命令解析器模块完成请求分类，区别出是 SELECT 并转发给 MySQL Query Optimize r时，MySQL Query Optimizer 首先会对整条 Query 进行优化，处理掉一些常量表达式的预算，直接换算成常量值。并对 Query 中的查询条件进行简化和转换，如去掉一些无用或显而易见的条件、结构调整等。然后分析 Query 中的 Hint 信息（如果有），看显示 Hint 信息是否可以完全确定该 Query 的执行计划。如果没有 Hint 或 Hint 信息还不足以完全确定执行计划，则会读取所涉及对象的统计信息，根据 Query 进行写相应的计算分析，然后再得出最后的执行计划。 - -#### MySQL常见瓶颈 - -- CPU：CPU在饱和的时候一般发生在数据装入内存或从磁盘上读取数据时候 - -- IO：磁盘I/O瓶颈发生在装入数据远大于内存容量的时候 - -- 服务器硬件的性能瓶颈：top，free，iostat 和 vmstat来查看系统的性能状态 - -#### 性能下降SQL慢 执行时间长 等待时间长 原因分析 - -- 查询语句写的烂 -- 索引失效（单值、复合） -- 关联查询太多join（设计缺陷或不得已的需求） -- 服务器调优及各个参数设置（缓冲、线程数等） - - - -#### MySQL常见性能分析手段 - -在优化MySQL时，通常需要对数据库进行分析，常见的分析手段有**慢查询日志**，**EXPLAIN 分析查询**，**profiling分析**以及**show命令查询系统状态及系统变量**，通过定位分析性能的瓶颈，才能更好的优化数据库系统的性能。 - -##### 性能瓶颈定位 - -我们可以通过 show 命令查看 MySQL 状态及变量，找到系统的瓶颈： - -```mysql -Mysql> show status ——显示状态信息（扩展show status like ‘XXX’） - -Mysql> show variables ——显示系统变量（扩展show variables like ‘XXX’） - -Mysql> show innodb status ——显示InnoDB存储引擎的状态 - -Mysql> show processlist ——查看当前SQL执行，包括执行状态、是否锁表等 - -Shell> mysqladmin variables -u username -p password——显示系统变量 - -Shell> mysqladmin extended-status -u username -p password——显示状态信息 -``` - - - -##### Explain(执行计划) - -是什么：使用 **Explain** 关键字可以模拟优化器执行SQL查询语句，从而知道 MySQL 是如何处理你的 SQL 语句的。分析你的查询语句或是表结构的性能瓶颈 - -能干吗： -- 表的读取顺序 -- 数据读取操作的操作类型 -- 哪些索引可以使用 -- 哪些索引被实际使用 -- 表之间的引用 -- 每张表有多少行被优化器查询 - -怎么玩： - -- Explain + SQL语句 -- 执行计划包含的信息（如果有分区表的话还会有**partitions**） - -![expalin](https://tva1.sinaimg.cn/large/007S8ZIlly1gf2hsjk9zcj30kq01adfn.jpg) - -各字段解释 - -- **id**（select 查询的序列号，包含一组数字，表示查询中执行select子句或操作表的顺序） - - - id相同，执行顺序从上往下 - - id全不同，如果是子查询，id的序号会递增，id值越大优先级越高，越先被执行 - - id部分相同，执行顺序是先按照数字大的先执行，然后数字相同的按照从上往下的顺序执行 - -- **select_type**（查询类型，用于区别普通查询、联合查询、子查询等复杂查询） - - - **SIMPLE** ：简单的select查询，查询中不包含子查询或UNION - - **PRIMARY**：查询中若包含任何复杂的子部分，最外层查询被标记为PRIMARY - - **SUBQUERY**：在select或where列表中包含了子查询 - - **DERIVED**：在from列表中包含的子查询被标记为DERIVED，MySQL会递归执行这些子查询，把结果放在临时表里 - - **UNION**：若第二个select出现在UNION之后，则被标记为UNION，若UNION包含在from子句的子查询中，外层select将被标记为DERIVED - - **UNION RESULT**：从UNION表获取结果的select - -- **table**（显示这一行的数据是关于哪张表的） - -- **type**（显示查询使用了那种类型，从最好到最差依次排列 **system > const > eq_ref > ref > fulltext > ref_or_null > index_merge > unique_subquery > index_subquery > range > index > ALL** ） - - - system：表只有一行记录（等于系统表），是 const 类型的特例，平时不会出现 - - const：表示通过索引一次就找到了，const 用于比较 primary key 或 unique 索引，因为只要匹配一行数据，所以很快，如将主键置于 where 列表中，mysql 就能将该查询转换为一个常量 - - eq_ref：唯一性索引扫描，对于每个索引键，表中只有一条记录与之匹配，常见于主键或唯一索引扫描 - - ref：非唯一性索引扫描，范围匹配某个单独值得所有行。本质上也是一种索引访问，他返回所有匹配某个单独值的行，然而，它可能也会找到多个符合条件的行，多以他应该属于查找和扫描的混合体 - - range：只检索给定范围的行，使用一个索引来选择行。key列显示使用了哪个索引，一般就是在你的where语句中出现了between、<、>、in等的查询，这种范围扫描索引比全表扫描要好，因为它只需开始于索引的某一点，而结束于另一点，不用扫描全部索引 - - index：Full Index Scan，index于ALL区别为index类型只遍历索引树。通常比ALL快，因为索引文件通常比数据文件小。（**也就是说虽然all和index都是读全表，但index是从索引中读取的，而all是从硬盘中读的**） - - ALL：Full Table Scan，将遍历全表找到匹配的行 - - tip: 一般来说，得保证查询至少达到range级别，最好到达ref - -- **possible_keys**（显示可能应用在这张表中的索引，一个或多个，查询涉及到的字段若存在索引，则该索引将被列出，但不一定被查询实际使用） - -- **key** - - - 实际使用的索引，如果为NULL，则没有使用索引 - - - **查询中若使用了覆盖索引，则该索引和查询的 select 字段重叠，仅出现在key列表中** - -![explain-key](https://tva1.sinaimg.cn/large/007S8ZIlly1gf2hsty7iaj30nt0373yb.jpg) - -- **key_len** - - - 表示索引中使用的字节数，可通过该列计算查询中使用的索引的长度。在不损失精确性的情况下，长度越短越好 - - key_len显示的值为索引字段的最大可能长度，并非实际使用长度，即key_len是根据表定义计算而得，不是通过表内检索出的 - -- **ref** （显示索引的哪一列被使用了，如果可能的话，是一个常数。哪些列或常量被用于查找索引列上的值） - -- **rows** （根据表统计信息及索引选用情况，大致估算找到所需的记录所需要读取的行数） - -- **Extra**（包含不适合在其他列中显示但十分重要的额外信息） - - 1. using filesort: 说明mysql会对数据使用一个外部的索引排序，不是按照表内的索引顺序进行读取。mysql中无法利用索引完成的排序操作称为“文件排序”。常见于order by和group by语句中 - - 2. Using temporary：使用了临时表保存中间结果，mysql在对查询结果排序时使用临时表。常见于排序order by和分组查询group by。 - - 3. using index：表示相应的select操作中使用了覆盖索引，避免访问了表的数据行，效率不错，如果同时出现using where，表明索引被用来执行索引键值的查找；否则索引被用来读取数据而非执行查找操作 - - 4. using where：使用了where过滤 - - 5. using join buffer：使用了连接缓存 - - 6. impossible where：where子句的值总是false，不能用来获取任何元祖 - - 7. select tables optimized away：在没有group by子句的情况下，基于索引优化操作或对于MyISAM存储引擎优化COUNT(*)操作，不必等到执行阶段再进行计算，查询执行计划生成的阶段即完成优化 - - 8. distinct：优化distinct操作，在找到第一匹配的元祖后即停止找同样值的动作 - - - -**case**: - -![explain-demo](https://tva1.sinaimg.cn/large/007S8ZIlly1gf2hszmc0lj30lc05w75c.jpg) - -1. 第一行（执行顺序4）：id列为1，表示是union里的第一个select，select_type列的primary表示该查询为外层查询，table列被标记为，表示查询结果来自一个衍生表，其中derived3中3代表该查询衍生自第三个select查询，即id为3的select。【select d1.name......】 - -2. 第二行（执行顺序2）：id为3，是整个查询中第三个select的一部分。因查询包含在from中，所以为derived。【select id,name from t1 where other_column=''】 -3. 第三行（执行顺序3）：select列表中的子查询select_type为subquery，为整个查询中的第二个select。【select id from t3】 -4. 第四行（执行顺序1）：select_type为union，说明第四个select是union里的第二个select，最先执行【select name,id from t2】 -5. 第五行（执行顺序5）：代表从union的临时表中读取行的阶段，table列的表示用第一个和第四个select的结果进行union操作。【两个结果union操作】 - - - -##### 慢查询日志 - -MySQL 的慢查询日志是 MySQL 提供的一种日志记录，它用来记录在 MySQL 中响应时间超过阈值的语句，具体指运行时间超过 `long_query_time` 值的 SQL，则会被记录到慢查询日志中。 - -- `long_query_time` 的默认值为10，意思是运行10秒以上的语句 -- 默认情况下，MySQL数据库没有开启慢查询日志，需要手动设置参数开启 - -**查看开启状态** - -```mysql -SHOW VARIABLES LIKE '%slow_query_log%' -``` - -**开启慢查询日志** - -- 临时配置： - -```mysql -mysql> set global slow_query_log='ON'; -mysql> set global slow_query_log_file='/var/lib/mysql/hostname-slow.log'; -mysql> set global long_query_time=2; -``` - -​ 也可set文件位置，系统会默认给一个缺省文件host_name-slow.log - -​ 使用set操作开启慢查询日志只对当前数据库生效，如果MySQL重启则会失效。 - -- 永久配置 - - 修改配置文件my.cnf或my.ini，在[mysqld]一行下面加入两个配置参数 - -```mysql -[mysqld] -slow_query_log = ON -slow_query_log_file = /var/lib/mysql/hostname-slow.log -long_query_time = 3 -``` - -注：log-slow-queries 参数为慢查询日志存放的位置，一般这个目录要有 MySQL 的运行帐号的可写权限，一般都将这个目录设置为 MySQL 的数据存放目录；long_query_time=2 中的 2 表示查询超过两秒才记录；在my.cnf或者 my.ini 中添加 log-queries-not-using-indexes 参数，表示记录下没有使用索引的查询。 - -可以用 `select sleep(4)` 验证是否成功开启。 - -在生产环境中，如果手工分析日志，查找、分析SQL，还是比较费劲的，所以MySQL提供了日志分析工具**mysqldumpslow**。 - -通过 mysqldumpslow --help 查看操作帮助信息 - -- 得到返回记录集最多的10个SQL - - `mysqldumpslow -s r -t 10 /var/lib/mysql/hostname-slow.log` - -- 得到访问次数最多的10个SQL - - `mysqldumpslow -s c -t 10 /var/lib/mysql/hostname-slow.log` - -- 得到按照时间排序的前10条里面含有左连接的查询语句 - - `mysqldumpslow -s t -t 10 -g "left join" /var/lib/mysql/hostname-slow.log` - -- 也可以和管道配合使用 - - `mysqldumpslow -s r -t 10 /var/lib/mysql/hostname-slow.log | more` - -**也可使用 pt-query-digest 分析 RDS MySQL 慢查询日志** - - - -##### Show Profile 分析查询 - -通过慢日志查询可以知道哪些 SQL 语句执行效率低下，通过 explain 我们可以得知 SQL 语句的具体执行情况，索引使用等，还可以结合`Show Profile`命令查看执行状态。 - -- Show Profile 是 MySQL 提供可以用来分析当前会话中语句执行的资源消耗情况。可以用于SQL的调优的测量 - -- 默认情况下，参数处于关闭状态，并保存最近15次的运行结果 - -- 分析步骤 - - 1. 是否支持，看看当前的mysql版本是否支持 - - ```mysql - mysql>Show variables like 'profiling'; --默认是关闭，使用前需要开启 - ``` - - 2. 开启功能，默认是关闭，使用前需要开启 - - ```mysql - mysql>set profiling=1; - ``` - - 3. 运行SQL - - 4. 查看结果 - - ```mysql - mysql> show profiles; - +----------+------------+---------------------------------+ - | Query_ID | Duration | Query | - +----------+------------+---------------------------------+ - | 1 | 0.00385450 | show variables like "profiling" | - | 2 | 0.00170050 | show variables like "profiling" | - | 3 | 0.00038025 | select * from t_base_user | - +----------+------------+---------------------------------+ - ``` - ``` - - 5. 诊断SQL，show profile cpu,block io for query id(上一步前面的问题SQL数字号码) - - 6. 日常开发需要注意的结论 - - - converting HEAP to MyISAM 查询结果太大，内存都不够用了往磁盘上搬了。 - - - create tmp table 创建临时表，这个要注意 - - - Copying to tmp table on disk 把内存临时表复制到磁盘 - - - locked - ``` - - - -> 查询中哪些情况不会使用索引？ - -### 性能优化 - -#### 索引优化 - -1. 全值匹配我最爱 -2. 最佳左前缀法则，比如建立了一个联合索引(a,b,c)，那么其实我们可利用的索引就有(a), (a,b), (a,b,c) -3. 不在索引列上做任何操作（计算、函数、(自动or手动)类型转换），会导致索引失效而转向全表扫描 -4. 存储引擎不能使用索引中范围条件右边的列 -5. 尽量使用覆盖索引(只访问索引的查询(索引列和查询列一致))，减少select -7. is null ,is not null 也无法使用索引 -8. like "xxxx%" 是可以用到索引的，like "%xxxx" 则不行(like "%xxx%" 同理)。like以通配符开头('%abc...')索引失效会变成全表扫描的操作， -9. 字符串不加单引号索引失效 -10. 少用or，用它来连接时会索引失效 -10. <，<=，=，>，>=，BETWEEN，IN 可用到索引，<>，not in ，!= 则不行，会导致全表扫描 - - - -**一般性建议** - -- 对于单键索引，尽量选择针对当前query过滤性更好的索引 - -- 在选择组合索引的时候，当前Query中过滤性最好的字段在索引字段顺序中，位置越靠前越好。 - -- 在选择组合索引的时候，尽量选择可以能够包含当前query中的where字句中更多字段的索引 - -- 尽可能通过分析统计信息和调整query的写法来达到选择合适索引的目的 - -- 少用Hint强制索引 - - - -#### 查询优化 - -**永远小标驱动大表（小的数据集驱动大的数据集）** - -```mysql -slect * from A where id in (select id from B)`等价于 -#等价于 -select id from B -select * from A where A.id=B.id -``` - -当 B 表的数据集必须小于 A 表的数据集时，用 in 优于 exists - -```mysql -select * from A where exists (select 1 from B where B.id=A.id) -#等价于 -select * from A -select * from B where B.id = A.id` -``` - -当 A 表的数据集小于B表的数据集时，用 exists优于用 in - -注意：A表与B表的ID字段应建立索引。 - - - -**order by关键字优化** - -- order by子句，尽量使用 Index 方式排序，避免使用 FileSort 方式排序 - -- MySQL 支持两种方式的排序，FileSort 和 Index，Index效率高，它指 MySQL 扫描索引本身完成排序，FileSort 效率较低； -- ORDER BY 满足两种情况，会使用Index方式排序；①ORDER BY语句使用索引最左前列 ②使用where子句与ORDER BY子句条件列组合满足索引最左前列 - -- 尽可能在索引列上完成排序操作，遵照索引建的最佳最前缀 -- 如果不在索引列上，filesort 有两种算法，mysql就要启动双路排序和单路排序 - - 双路排序：MySQL 4.1之前是使用双路排序,字面意思就是两次扫描磁盘，最终得到数据 - - 单路排序：从磁盘读取查询需要的所有列，按照order by 列在 buffer对它们进行排序，然后扫描排序后的列表进行输出，效率高于双路排序 - -- 优化策略 - - - 增大sort_buffer_size参数的设置 - - 增大max_lencth_for_sort_data参数的设置 - - - - -**GROUP BY关键字优化** - -- group by实质是先排序后进行分组，遵照索引建的最佳左前缀 -- 当无法使用索引列，增大 `max_length_for_sort_data` 参数的设置，增大`sort_buffer_size`参数的设置 -- where高于having，能写在where限定的条件就不要去having限定了 - - - -#### 数据类型优化 - -MySQL 支持的数据类型非常多，选择正确的数据类型对于获取高性能至关重要。不管存储哪种类型的数据，下面几个简单的原则都有助于做出更好的选择。 - -- 更小的通常更好：一般情况下，应该尽量使用可以正确存储数据的最小数据类型。 - - 简单就好：简单的数据类型通常需要更少的CPU周期。例如，整数比字符操作代价更低，因为字符集和校对规则（排序规则）使字符比较比整型比较复杂。 - -- 尽量避免NULL：通常情况下最好指定列为NOT NULL - ------- - - - -## 九、分区、分表、分库 - -### MySQL分区 - -一般情况下我们创建的表对应一组存储文件，使用`MyISAM`存储引擎时是一个`.MYI`和`.MYD`文件，使用`Innodb`存储引擎时是一个`.ibd`和`.frm`（表结构）文件。 - -当数据量较大时（一般千万条记录级别以上），MySQL的性能就会开始下降，这时我们就需要将数据分散到多组存储文件，保证其单个文件的执行效率 - -**能干嘛** - -- 逻辑数据分割 -- 提高单一的写和读应用速度 -- 提高分区范围读查询的速度 -- 分割数据能够有多个不同的物理文件路径 -- 高效的保存历史数据 - -**怎么玩** - -首先查看当前数据库是否支持分区 - -- MySQL5.6以及之前版本： - - ```mysql - SHOW VARIABLES LIKE '%partition%'; - ``` - -- MySQL5.6： - - ```mysql - show plugins; - ``` - -**分区类型及操作** - -- **RANGE分区**：基于属于一个给定连续区间的列值，把多行分配给分区。mysql将会根据指定的拆分策略，,把数据放在不同的表文件上。相当于在文件上,被拆成了小块.但是,对外给客户的感觉还是一张表，透明的。 - - 按照 range 来分，就是每个库一段连续的数据，这个一般是按比如**时间范围**来的，比如交易表啊，销售表啊等，可以根据年月来存放数据。可能会产生热点问题，大量的流量都打在最新的数据上了。 - - range 来分，好处在于说，扩容的时候很简单。 - -- **LIST分区**：类似于按RANGE分区，每个分区必须明确定义。它们的主要区别在于，LIST分区中每个分区的定义和选择是基于某列的值从属于一个值列表集中的一个值，而RANGE分区是从属于一个连续区间值的集合。 - -- **HASH分区**：基于用户定义的表达式的返回值来进行选择的分区，该表达式使用将要插入到表中的这些行的列值进行计算。这个函数可以包含MySQL 中有效的、产生非负整数值的任何表达式。 - - hash 分发，好处在于说，可以平均分配每个库的数据量和请求压力；坏处在于说扩容起来比较麻烦，会有一个数据迁移的过程，之前的数据需要重新计算 hash 值重新分配到不同的库或表 - -- **KEY分区**：类似于按HASH分区，区别在于KEY分区只支持计算一列或多列，且MySQL服务器提供其自身的哈希函数。必须有一列或多列包含整数值。 - -**看上去分区表很帅气，为什么大部分互联网还是更多的选择自己分库分表来水平扩展咧？** - -- 分区表，分区键设计不太灵活，如果不走分区键，很容易出现全表锁 -- 一旦数据并发量上来，如果在分区表实施关联，就是一个灾难 -- 自己分库分表，自己掌控业务场景与访问模式，可控。分区表，研发写了一个sql，都不确定mysql是怎么玩的，不太可控 - - - -> 随着业务的发展，业务越来越复杂，应用的模块越来越多，总的数据量很大，高并发读写操作均超过单个数据库服务器的处理能力怎么办？ - -这个时候就出现了**数据分片**，数据分片指按照某个维度将存放在单一数据库中的数据分散地存放至多个数据库或表中。数据分片的有效手段就是对关系型数据库进行分库和分表。 - -区别于分区的是，分区一般都是放在单机里的，用的比较多的是时间范围分区，方便归档。只不过分库分表需要代码实现，分区则是mysql内部实现。分库分表和分区并不冲突，可以结合使用。 - - - ->说说分库与分表的设计 - -### MySQL分表 - -分表有两种分割方式，一种垂直拆分，另一种水平拆分。 - -- **垂直拆分** - - 垂直分表，通常是按照业务功能的使用频次，把主要的、热门的字段放在一起做为主要表。然后把不常用的，按照各自的业务属性进行聚集，拆分到不同的次要表中；主要表和次要表的关系一般都是一对一的。 - -- **水平拆分(数据分片)** - - 单表的容量不超过500W，否则建议水平拆分。是把一个表复制成同样表结构的不同表，然后把数据按照一定的规则划分，分别存储到这些表中，从而保证单表的容量不会太大，提升性能；当然这些结构一样的表，可以放在一个或多个数据库中。 - - 水平分割的几种方法： - - - 使用MD5哈希，做法是对UID进行md5加密，然后取前几位（我们这里取前两位），然后就可以将不同的UID哈希到不同的用户表（user_xx）中了。 - - 还可根据时间放入不同的表，比如：article_201601，article_201602。 - - 按热度拆分，高点击率的词条生成各自的一张表，低热度的词条都放在一张大表里，待低热度的词条达到一定的贴数后，再把低热度的表单独拆分成一张表。 - - 根据ID的值放入对应的表，第一个表user_0000，第二个100万的用户数据放在第二 个表user_0001中，随用户增加，直接添加用户表就行了。 - -![](https://tva1.sinaimg.cn/large/007S8ZIlly1geuibkd9mjj31ns0u0aj1.jpg) - - - -### MySQL分库 - -> 为什么要分库? - -数据库集群环境后都是多台 slave，基本满足了读取操作; 但是写入或者说大数据、频繁的写入操作对master性能影响就比较大，这个时候，单库并不能解决大规模并发写入的问题，所以就会考虑分库。 - -> 分库是什么？ - -一个库里表太多了，导致了海量数据，系统性能下降，把原本存储于一个库的表拆分存储到多个库上， 通常是将表按照功能模块、关系密切程度划分出来，部署到不同库上。 - -优点： - -- 减少增量数据写入时的锁对查询的影响 - -- 由于单表数量下降，常见的查询操作由于减少了需要扫描的记录，使得单表单次查询所需的检索行数变少，减少了磁盘IO，时延变短 - -但是它无法解决单表数据量太大的问题 - - - -**分库分表后的难题** - -分布式事务的问题，数据的完整性和一致性问题。 - -数据操作维度问题：用户、交易、订单各个不同的维度，用户查询维度、产品数据分析维度的不同对比分析角度。 跨库联合查询的问题，可能需要两次查询 跨节点的count、order by、group by以及聚合函数问题，可能需要分别在各个节点上得到结果后在应用程序端进行合并 额外的数据管理负担，如：访问数据表的导航定位 额外的数据运算压力，如：需要在多个节点执行，然后再合并计算程序编码开发难度提升，没有太好的框架解决，更多依赖业务看如何分，如何合，是个难题。 - - - -> 配主从，正经公司的话，也不会让 Javaer 去搞的，但还是要知道 - -## 十、主从复制 - -### 复制的基本原理 - -- slave 会从 master 读取 binlog 来进行数据同步 - -- 三个步骤 - - 1. master将改变记录到二进制日志（binary log）。这些记录过程叫做二进制日志事件，binary log events； - 2. salve 将 master 的 binary log events 拷贝到它的中继日志（relay log）; - 3. slave 重做中继日志中的事件，将改变应用到自己的数据库中。MySQL 复制是异步且是串行化的。 - - ![img](http://img.wandouip.com/crawler/article/201942/94aec4abf353527cbbe2bef5a484471d) - -### 复制的基本原则 - -- 每个 slave只有一个 master -- 每个 salve只能有一个唯一的服务器 ID -- 每个master可以有多个salve - -### 复制的最大问题 - -- 延时 - ------- - - - -## 十一、其他问题 - -### 说一说三个范式 - -- 第一范式（1NF）：数据库表中的字段都是单一属性的，不可再分。这个单一属性由基本类型构成，包括整型、实数、字符型、逻辑型、日期型等。 -- 第二范式（2NF）：数据库表中不存在非关键字段对任一候选关键字段的部分函数依赖（部分函数依赖指的是存在组合关键字中的某些字段决定非关键字段的情况），也即所有非关键字段都完全依赖于任意一组候选关键字。 -- 第三范式（3NF）：在第二范式的基础上，数据表中如果不存在非关键字段对任一候选关键字段的传递函数依赖则符合第三范式。所谓传递函数依赖，指的是如 果存在"A → B → C"的决定关系，则C传递函数依赖于A。因此，满足第三范式的数据库表应该不存在如下依赖关系： 关键字段 → 非关键字段 x → 非关键字段y - - - -### 百万级别或以上的数据如何删除 - -关于索引：由于索引需要额外的维护成本，因为索引文件是单独存在的文件,所以当我们对数据的增加,修改,删除,都会产生额外的对索引文件的操作,这些操作需要消耗额外的IO,会降低增/改/删的执行效率。所以，在我们删除数据库百万级别数据的时候，查询MySQL官方手册得知删除数据的速度和创建的索引数量是成正比的。 - -1. 所以我们想要删除百万数据的时候可以先删除索引（此时大概耗时三分多钟） -2. 然后删除其中无用数据（此过程需要不到两分钟） -3. 删除完成后重新创建索引(此时数据较少了)创建索引也非常快，约十分钟左右。 -4. 与之前的直接删除绝对是要快速很多，更别说万一删除中断,一切删除会回滚。那更是坑了。 - - - - - - -## 参考与感谢： - -https://zhuanlan.zhihu.com/p/29150809 - -https://juejin.im/post/5e3eb616f265da570d734dcb#heading-105 - -https://blog.csdn.net/yin767833376/article/details/81511377 - - - - - - - - - - - - - - \ No newline at end of file diff --git a/docs/data-store/MySQL/MySQL-Framework.md b/docs/data-store/MySQL/MySQL-Framework.md deleted file mode 100644 index 9f1b8d5c49..0000000000 --- a/docs/data-store/MySQL/MySQL-Framework.md +++ /dev/null @@ -1,38 +0,0 @@ -# MySQL架构介绍 - -和其它数据库相比，MySQL有点与众不同，它的架构可以在多种不同场景中应用并发挥良好作用。主要体现在存储引擎的架构上，**插件式的存储引擎架构将查询处理和其它的系统任务以及数据的存储提取相分离**。这种架构可以根据业务的需求和实际需要选择合适的存储引擎。 - -![mysql-framework](../../_images/mysql/mysql-framework.png) - - - -## 1. 连接层 - -最上层是一些客户端和连接服务，包含本地socket通信和大多数基于客户端/服务端工具实现的类似于tcp/ip的通信。**主要完成一些类似于连接处理、授权认证、及相关的安全方案**。在该层上引入了线程池的概念，为通过认证安全接入的客户端提供线程。同样在该层上可以实现基于SSL的安全链接。服务器也会为安全接入的每个客户端验证它所具有的操作权限。 - -## 2. 服务层 - -第二层架构主要完成大部分的核心服务功能， 包括查询解析、分析、优化、缓存、以及所有的内置函数，所有跨存储引擎的功能也都在这一层实现，包括触发器、存储过程、视图等 - -## 3.引擎层 - -存储引擎层，存储引擎真正的负责了MySQL中数据的存储和提取，服务器通过API与存储引擎进行通信。不同的存储引擎具有的功能不同，这样我们可以根据自己的实际需要进行选取。 - -## 4.存储层 - -数据存储层，主要是将数据存储在运行于该设备的文件系统之上，并完成与存储引擎的交互。 - - - -更符合程序员审美的MySQL服务器逻辑架构图 - -![](../../_images/mysql/mysql-framework1.png) - -## 查询说明 - -mysql的查询流程大致是： - -1. mysql客户端通过协议与mysql服务器建连接，发送查询语句，先检查查询缓存，如果命中，直接返回结果，否则进行语句解析 -2. 有一系列预处理，比如检查语句是否写正确了，然后是查询优化（比如是否使用索引扫描，如果是一个不可能的条件，则提前终止），生成查询计划，然后查询引擎启动，开始执行查询，从底层存储引擎调用API获取数据，最后返回给客户端。怎么存数据、怎么取数据，都与存储引擎有关。 -3. 然后，mysql默认使用的BTREE索引，并且一个大方向是，无论怎么折腾sql，至少在目前来说，mysql最多只用到表中的一个索引。 - diff --git a/docs/data-store/MySQL/MySQL-Index.md b/docs/data-store/MySQL/MySQL-Index.md deleted file mode 100644 index 6fb1afa32b..0000000000 --- a/docs/data-store/MySQL/MySQL-Index.md +++ /dev/null @@ -1,216 +0,0 @@ -# MySQL索引 - -### 是什么 - -- MYSQL官方对索引的定义为：索引（Index）是帮助MySQL高效获取数据的数据结构，所以说**索引的本质是：数据结构** - -- 索引的目的在于提高查询效率，可以类比字典、 火车站的车次表、图书的目录等 。 - -- 可以简单的理解为“排好序的快速查找数据结构”，数据本身之外，**数据库还维护者一个满足特定查找算法的数据结构**，这些数据结构以某种方式引用（指向）数据，这样就可以在这些数据结构上实现高级查找算法。这种数据结构，就是索引。下图是一种可能的索引方式示例。 - - - - ![1.png](https://i.loli.net/2019/11/13/czD8xJ2AYrFpaRP.png) - - 左边的数据表，一共有两列七条记录，最左边的是数据记录的物理地址 - - 为了加快Col2的查找，可以维护一个右边所示的二叉查找树，每个节点分别包含索引键值，和一个指向对应数据记录物理地址的指针，这样就可以运用二叉查找在一定的复杂度内获取到对应的数据，从而快速检索出符合条件的记录。 - -- 索引本身也很大，不可能全部存储在内存中，**一般以索引文件的形式存储在磁盘上** - -- 平常说的索引，没有特别指明的话，就是B+树（多路搜索树，不一定是二叉树）结构组织的索引。其中聚集索引，次要索引，覆盖索引，符合索引，前缀索引，唯一索引默认都是使用B+树索引，统称索引。此外还有哈希索引等。 - - - -### 优势 - -- **提高数据检索效率，降低数据库IO成本** - -- **降低数据排序的成本，降低CPU的消耗** - - - -### 劣势 - -- 索引也是一张表，保存了主键和索引字段，并指向实体表的记录，所以也需要占用内存 -- 虽然索引大大提高了查询速度，同时却会降低更新表的速度，如对表进行INSERT、UPDATE和DELETE。 -因为更新表时，MySQL不仅要保存数据，还要保存一下索引文件每次更新添加了索引列的字段， - 都会调整因为更新所带来的键值变化后的索引信息 - -### mysql索引分类 - -- **单值索引**：一个索引只包含单个列，一个表可以有多个单列索引 -- **唯一索引**：索引列的值必须唯一，但允许有空值 -- **复合索引**：一个索引包含多个列 - - - -#### 基本语法： - -- 创建： - - - 创建索引：**CREATE [UNIQUE] INDEX indexName ON mytable(username(length))**; - - ?> **Tip** 如果是CHAR，VARCHAR类型，length可以小于字段实际长度；如果是BLOB和TEXT类型，必须指定 length。 - - - 修改表结构(添加索引)：**ALTER table tableName ADD [UNIQUE] INDEX indexName(columnName)** - -- 删除： **DROP INDEX [indexName] ON mytable;** - -- 查看： **SHOW INDEX FROM table_name\G** --可以通过添加 \G 来格式化输出信息。 - -- 使用ALERT命令 - - - **ALTER TABLE tbl_name ADD PRIMARY KEY (column_list):** 该语句添加一个主键，这意味着索引值必须是唯一的，且不能为NULL。 - - - **ALTER TABLE tbl_name ADD UNIQUE index_name (column_list):** 这条语句创建索引的值必须是唯一的（除了NULL外，NULL可能会出现多次）。 - - **ALTER TABLE tbl_name ADD INDEX index_name (column_list):** 添加普通索引，索引值可出现多次。 - - **ALTER TABLE tbl_name ADD FULLTEXT index_name (column_list):**该语句指定了索引为 FULLTEXT ，用于全文索引。 - - - -### 哪些情况需要创建索引 - -1. 主键自动建立唯一索引 - -2. 频繁作为查询条件的字段 - -3. 查询中与其他表关联的字段，外键关系建立索引 - -4. 单键/组合索引的选择问题，who?高并发下倾向创建组合索引 - -5. 查询中排序的字段，排序字段通过索引访问大幅提高排序速度 - -6. 查询中统计或分组字段 - - - -### 哪些情况不要创建索引 - -1. 表记录太少 -2. 经常增删改的表 -3. 数据重复且分布均匀的表字段，只应该为最经常查询和最经常排序的数据列建立索引（如果某个数据类包含太多的重复数据，建立索引没有太大意义） -4. 频繁更新的字段不适合创建索引（会加重IO负担） -5. where条件里用不到的字段不创建索引 - - - -### MySQL索引结构 - -介绍索引之前，先了解下**磁盘IO与预读** - -磁盘读取数据靠的是机械运动，每次读取数据花费的时间可以分为寻道时间、旋转延迟、传输时间三个部分，寻道时间指的是磁臂移动到指定磁道所需要的时间，主流磁盘一般在5ms以下；旋转延迟就是我们经常听说的磁盘转速，比如一个磁盘7200转，表示每分钟能转7200次，也就是说1秒钟能转120次，旋转延迟就是1/120/2 = 4.17ms；传输时间指的是从磁盘读出或将数据写入磁盘的时间，一般在零点几毫秒，相对于前两个时间可以忽略不计。那么访问一次磁盘的时间，即一次磁盘IO的时间约等于5+4.17 = 9ms左右，听起来还挺不错的，但要知道一台500 -MIPS的机器每秒可以执行5亿条指令，因为指令依靠的是电的性质，换句话说执行一次IO的时间可以执行40万条指令，数据库动辄十万百万乃至千万级数据，每次9毫秒的时间，显然是个灾难。下图是计算机硬件延迟的对比图，供大家参考： - -![various-system-software-hardware-latencies](https://awps-assets.meituan.net/mit-x/blog-images-bundle-2014/7f46a0a4.png) - - 考虑到磁盘IO是非常高昂的操作，计算机操作系统做了一些优化，当一次IO时，不光把当前磁盘地址的数据，而是把相邻的数据也都读取到内存缓冲区内，因为局部预读性原理告诉我们，当计算机访问一个地址的数据的时候，与其相邻的数据也会很快被访问到。每一次IO读取的数据我们称之为**一页(page)**。具体一页有多大数据跟操作系统有关，一般为4k或8k，也就是我们读取一页内的数据时候，实际上才发生了一次IO，这个理论对于索引的数据结构设计非常有帮助。 - - - -?> **首先要明白索引（index）是在存储引擎（storage engine）层面实现的，而不是server层面**。不是所有的存储引擎都支持所有的索引类型。即使多个存储引擎支持某一索引类型，它们的实现和行为也可能有所差别。 - - - -#### **BTree索引** - -- MyISAM存储引擎，使用B+Tree的数据结构，它相对与BTree结构，所有的数据都存放在叶子节点上，且把叶子节点通过指针连接到一起，形成了一条数据链表，以加快相邻数据的检索效率 - -- **MyISAM引擎索引结构的叶子节点的数据域，存放的并不是实际的数据记录，而是数据记录的地址**。索引文件与数据文件分离，这样的索引称为***"非聚簇索引"***。MyISAM的主索引与辅助索引区别并不大，只是主键索引不能有重复的关键字。 - -- **InnoDB引擎索引结构的叶子节点的数据域，存放的就是实际的数据记录**（对于主索引，此处会存放表中所有的数据记录；对于辅助索引此处会引用主键，检索的时候通过主键到主键索引中找到对应数据行），或者说，InnoDB的数据文件本身就是主键索引文件，这样的索引被称为***“聚簇索引”***，**一个表只能有一个聚簇索引** - -- 检索原理 - - ![bTree](../../_images/mysql/bTree.png) - - 【初始化介绍】 - 上图是一颗b+树，浅蓝色的块我们称之为一个磁盘块，可以看到每个磁盘块包含几个数据项（深蓝色所示）和指针（黄色所示），如磁盘块1包含数据项17和35，包含指针P1、P2、P3， - P1表示小于17的磁盘块，P2表示在17和35之间的磁盘块，P3表示大于35的磁盘块。 - 真实的数据存在于叶子节点即3、5、9、10、13、15、28、29、36、60、75、79、90、99。 - 非叶子节点只不存储真实的数据，只存储指引搜索方向的数据项，如17、35并不真实存在于数据表中。 - - -【查找过程】 - 如果要查找数据项29，那么首先会把磁盘块1由磁盘加载到内存，此时发生一次IO，在内存中用二分查找确定29在17和35之间，锁定磁盘块1的P2指针，内存时间因为非常短（相比磁盘的IO）可以忽略不计，通过磁盘块1的P2指针的磁盘地址把磁盘块3由磁盘加载到内存，发生第二次IO，29在26和30之间，锁定磁盘块3的P2指针，通过指针加载磁盘块8到内存，发生第三次IO，同时内存中做二分查找找到29，结束查询，总计三次IO。 - -真实的情况是，3层的b+树可以表示上百万的数据，如果上百万的数据查找只需要三次IO，性能提高将是巨大的，如果没有索引，每个数据项都要发生一次IO，那么总共需要百万次的IO，显然成本非常非常高。 - - #### b+树性质 - - 1. 通过上面的分析，我们知道IO次数取决于b+数的高度h，假设当前数据表的数据为N，每个磁盘块的数据项的数量是m，则有h=㏒(m+1)N，当数据量N一定的情况下，m越大，h越小；而m = 磁盘块的大小 / 数据项的大小，磁盘块的大小也就是一个数据页的大小，是固定的，如果数据项占的空间越小，数据项的数量越多，树的高度越低。这就是为什么每个数据项，即索引字段要尽量的小，比如int占4字节，要比bigint8字节少一半。这也是为什么b+树要求把真实的数据放到叶子节点而不是内层节点，一旦放到内层节点，磁盘块的数据项会大幅度下降，导致树增高。当数据项等于1时将会退化成线性表。 - - 2. 当b+树的数据项是复合的数据结构，比如(name,age,sex)的时候，b+数是按照从左到右的顺序来建立搜索树的，比如当(张三,20,F)这样的数据来检索的时候，b+树会优先比较name来确定下一步的所搜方向，如果name相同再依次比较age和sex，最后得到检索的数据；但当(20,F)这样的没有name的数据来的时候，b+树就不知道下一步该查哪个节点，因为建立搜索树的时候name就是第一个比较因子，必须要先根据name来搜索才能知道下一步去哪里查询。比如当(张三,F)这样的数据来检索时，b+树可以用name来指定搜索方向，但下一个字段age的缺失，所以只能把名字等于张三的数据都找到，然后再匹配性别是F的数据了， 这个是非常重要的性质，即索引的最左匹配特性。 - -#### **Hash索引** - -- 主要就是通过Hash算法（常见的Hash算法有直接定址法、平方取中法、折叠法、除数取余法、随机数法），将数据库字段数据转换成定长的Hash值，与这条数据的行指针一并存入Hash表的对应位置；如果发生Hash碰撞（两个不同关键字的Hash值相同），则在对应Hash键下以链表形式存储。 - - 检索算法：在检索查询时，就再次对待查关键字再次执行相同的Hash算法，得到Hash值，到对应Hash表对应位置取出数据即可，如果发生Hash碰撞，则需要在取值时进行筛选。目前使用Hash索引的数据库并不多，主要有Memory等。 - - MySQL目前有Memory引擎和NDB引擎支持Hash索引。 - -- Hash索引的弊端 - - 一般来说，索引的检索效率非常高，可以一次定位，不像B-Tree索引需要进行从根节点到叶节点的多次IO操作。有利必有弊，Hash算法在索引的应用也有很多弊端。 - - 1. Hash索引仅仅能满足等值的查询，范围查询不保证结果正确。因为数据在经过Hash算法后，其大小关系就可能发生变化。 - 2. Hash索引不能被排序。同样是因为数据经过Hash算法后，大小关系就可能发生变化，排序是没有意义的。 - 3. Hash索引不能避免表数据的扫描。因为发生Hash碰撞时，仅仅比较Hash值是不够的，需要比较实际的值以判定是否符合要求。 - 4. Hash索引在发生大量Hash值相同的情况时性能不一定比B-Tree索引高。因为碰撞情况会导致多次的表数据的扫描，造成整体性能的低下，可以通过采用合适的Hash算法一定程度解决这个问题。 - 5. Hash索引不能使用部分索引键查询。因为当使用组合索引情况时，是把多个数据库列数据合并后再计算Hash值，所以对单独列数据计算Hash值是没有意义的。 - -#### **full-text全文索引** - -- 全文索引也是MyISAM的一种特殊索引类型，主要用于全文索引，InnoDB从MYSQL5.6版本提供对全文索引的支持。 - -- 它用于替代效率较低的LIKE模糊匹配操作，而且可以通过多字段组合的全文索引一次性全模糊匹配多个字段。 -- 同样使用B-Tree存放索引数据，但使用的是特定的算法，将字段数据分割后再进行索引（一般每4个字节一次分割），索引文件存储的是分割前的索引字符串集合，与分割后的索引信息，对应Btree结构的节点存储的是分割后的词信息以及它在分割前的索引字符串集合中的位置。 - -#### **R-Tree空间索引** - - 空间索引是MyISAM的一种特殊索引类型，主要用于地理空间数据类型 - - - -### MySQL索引分类 - -#### 数据结构角度 - -- B+树索引 -- Hash索引 -- Full-Text全文索引 -- R-Tree索引 - -#### 从物理存储角度 - -- 聚集索引（clustered index） - -- 非聚集索引（non-clustered index），也叫辅助索引（secondary index） - - 聚集索引和非聚集索引都是B+树结构 - -#### 从逻辑角度 - -- 主键索引：主键索引是一种特殊的唯一索引，不允许有空值 -- 普通索引或者单列索引 -- 多列索引（复合索引）：复合索引指多个字段上创建的索引，只有在查询条件中使用了创建索引时的第一个字段，索引才会被使用。使用复合索引时遵循最左前缀集合 -- 唯一索引或者非唯一索引 -- 空间索引：空间索引是对空间数据类型的字段建立的索引，MYSQL中的空间数据类型有4种，分别是GEOMETRY、POINT、LINESTRING、POLYGON。 - MYSQL使用SPATIAL关键字进行扩展，使得能够用于创建正规索引类型的语法创建空间索引。创建空间索引的列，必须将其声明为NOT NULL，空间索引只能在存储引擎为MYISAM的表中创建 - - - -### MySQL高效索引 - -**覆盖索引**（Covering Index）,或者叫索引覆盖， 也就是平时所说的不需要回表操作 - -- 就是select的数据列只用从索引中就能够取得，不必读取数据行，MySQL可以利用索引返回select列表中的字段，而不必根据索引再次读取数据文件，换句话说**查询列要被所建的索引覆盖**。 -- 索引是高效找到行的一个方法，但是一般数据库也能使用索引找到一个列的数据，因此它不必读取整个行。毕竟索引叶子节点存储了它们索引的数据，当能通过读取索引就可以得到想要的数据，那就不需要读取行了。一个索引包含（覆盖）满足查询结果的数据就叫做覆盖索引。 - -- **判断标准** - - 使用explain，可以通过输出的extra列来判断，对于一个索引覆盖查询，显示为**using index**,MySQL查询优化器在执行查询前会决定是否有索引覆盖查询 - - - -> [美团技术-MySQL索引原理及慢查询优化](https://tech.meituan.com/2014/06/30/mysql-index.html) \ No newline at end of file diff --git a/docs/data-store/MySQL/MySQL-Lock.md b/docs/data-store/MySQL/MySQL-Lock.md deleted file mode 100644 index 130caed2bb..0000000000 --- a/docs/data-store/MySQL/MySQL-Lock.md +++ /dev/null @@ -1,321 +0,0 @@ -# MySQL锁 - -锁是计算机协调多个进程或线程并发访问某一资源的机制。 - -在数据库中，除传统的计算资源（如CPU、RAM、I/O等）的争用以外，数据也是一种供许多用户共享的资源。数据库锁定机制简单来说，就是数据库为了保证数据的一致性，而使各种共享资源在被并发访问变得有序所设计的一种规则 - -打个比方，我们到淘宝上买一件商品，商品只有一件库存，这个时候如果还有另一个人买，那么如何解决是你买到还是另一个人买到的问题？ - - -这里肯定要用到事物，我们先从库存表中取出物品数量，然后插入订单，付款后插入付款表信息，然后更新商品数量。在这个过程中，使用锁可以对有限的资源进行保护，解决隔离和并发的矛盾。 - - - -## 锁的分类 - -#### 从对数据操作的类型分类： - -- **读锁**（共享锁）：针对同一份数据，多个读操作可以同时进行，不会互相影响 - -- **写锁**（排他锁）：当前写操作没有完成前，它会阻断其他写锁和读锁。 - -#### 从对数据操作的粒度分类： - - -为了尽可能提高数据库的并发度，每次锁定的数据范围越小越好，理论上每次只锁定当前操作的数据的方案会得到最大的并发度，但是管理锁是很耗资源的事情（涉及获取，检查，释放锁等动作），因此数据库系统需要在高并发响应和系统性能两方面进行平衡，这样就产生了“锁粒度（Lock granularity）”的概念。 - -一种提高共享资源并发性的方式是让锁定对象更有选择性。尽量只锁定需要修改的部分数据，而不是所有的资源。更理想的方式是，只对会修改的数据片进行精确的锁定。任何时候，在给定的资源上，锁定的数据量越少，则系统的并发程度越高，只要相互之间不发生冲突即可。 - -- **表级锁**：开销小，加锁快；不会出现死锁；锁定粒度大，发生锁冲突的概率最高，并发度最低； - -- **行级锁**：开销大，加锁慢；会出现死锁；锁定粒度最小，发生锁冲突的概率最低，并发度也最高； - -- **页面锁**：开销和加锁时间界于表锁和行锁之间；会出现死锁；锁定粒度界于表锁和行锁之间，并发度一般。 - -适用：从锁的角度来说，表级锁更适合于以查询为主，只有少量按索引条件更新数据的应用，如Web应用；而行级锁则更适合于有大量按索引条件并发更新少量不同数据，同时又有并发查询的应用，如一些在线事务处理（OLTP）系统。 - - - -#### 加锁机制 - -**乐观锁与悲观锁是两种并发控制的思想，可用于解决丢失更新问题** - -乐观锁会“乐观地”假定大概率不会发生并发更新冲突，访问、处理数据过程中不加锁，只在更新数据时再根据版本号或时间戳判断是否有冲突，有则处理，无则提交事务； - -悲观锁会“悲观地”假定大概率会发生并发更新冲突，访问、处理数据前就加排他锁，在整个数据处理过程中锁定数据，事务提交或回滚后才释放锁； - - - -#### 锁模式 - -- 记录锁： 对索引项加锁，锁定符合条件的行。其他事务不能修改 和删除加锁项； -- gap锁： 对索引项之间的“间隙”加锁，锁定记录的范围（对第一条记录前的间隙或最后一条将记录后的间隙加锁），不包含索引项本身。其他事务不能在锁范围内插入数据，这样就防止了别的事务新增幻影行。 -- next-key锁： 锁定索引项本身和索引范围。即Record Lock和Gap Lock的结合。可解决幻读问题。 -- 意向锁 -- 插入意向锁 - - - - MySQL 不同的存储引擎支持不同的锁机制，所有的存储引擎都以自己的方式实现了锁机制 - -| | 行锁 | 表锁 | 页锁 | -| ------ | ---- | ---- | ---- | -| MyISAM | | √ | | -| BDB | | √ | √ | -| InnoDB | √ | √ | | -| Memory | | √ | | - - - -### 表锁（偏读） - -#### 特点： - -**偏向MyISAM存储引擎，开销小，加锁快，无死锁；锁定粒度大，发生锁冲突的概率最高，并发度最低** - -MyISAM在执行查询语句（SELECT）前，会自动给涉及的所有表加读锁，在执行增删改操作前，会自动给涉及的表加写锁。 -MySQL的表级锁有两种模式： - -- 表共享读锁（Table Read Lock） -- 表独占写锁（Table Write Lock） - -| 锁类型 | 可否兼容 | 读锁 | 写锁 | -| ------ | -------- | ---- | ---- | -| 读锁 | 是 | 是 | 否 | -| 写锁 | 是 | 否 | 否 | - - - 结合上表，所以对MyISAM表进行操作，会有以下情况： - -1. 对MyISAM表的读操作（加读锁），不会阻塞其他进程对同一表的读请求，但会阻塞对同一表的写请求。只有当读锁释放后，才会执行其它进程的写操作。 -2. 对MyISAM表的写操作（加写锁），会阻塞其他进程对同一表的读和写操作，只有当写锁释放后，才会执行其它进程的读写操作。 - -简而言之，就是读锁会阻塞写，但是不会堵塞读。而写锁则会把读和写都堵塞。 - -MyISAM表的读操作与写操作之间，以及写操作之间是串行的。当一个线程获得对一个表的写锁后，只有持有锁的线程可以对表进行更新操作。其他线程的读、写操作都会等待，直到锁被释放为止。 - -#### 如何加表锁 - -MyISAM在执行查询语句（SELECT）前，会自动给涉及的所有表加读锁，在执行更新操作（UPDATE、DELETE、INSERT等）前，会自动给涉及的表加写锁，这个过程并不需要用户干预，因此，用户一般不需要直接用LOCK TABLE命令给MyISAM表显式加锁。 - -#### MyISAM表锁优化建议 - -对于MyISAM存储引擎，虽然使用表级锁定在锁定实现的过程中比实现行级锁定或者页级锁所带来的附加成本都要小，锁定本身所消耗的资源也是最少。但是由于锁定的颗粒度比较大，所以造成锁定资源的争用情况也会比其他的锁定级别都要多，从而在较大程度上会降低并发处理能力。所以，在优化MyISAM存储引擎锁定问题的时候，最关键的就是如何让其提高并发度。由于锁定级别是不可能改变的了，所以我们首先需要**尽可能让锁定的时间变短**，然后就是让可能并发进行的操作尽可能的并发。 - -看看哪些表被加锁了: - -```mysql -mysql>show open tables; -``` - -1. ##### 查询表级锁争用情况 - -MySQL内部有两组专门的状态变量记录系统内部锁资源争用情况： - -```mysql -mysql> show status like 'table%'; -``` - -![image-20191203171450621](C:\Users\jiahaixin\AppData\Roaming\Typora\typora-user-images\image-20191203171450621.png) - -这里有两个状态变量记录MySQL内部表级锁定的情况，两个变量说明如下： - -- Table_locks_immediate：产生表级锁定的次数，表示可以立即获取锁的查询次数，每立即获取锁值加1 - -- Table_locks_waited：出现表级锁定争用而发生等待的次数(不能立即获取锁的次数，每等待一次锁值加1)，此值高则说明存在着较严重的表级锁争用情况 - -两个状态值都是从系统启动后开始记录，出现一次对应的事件则数量加1。如果这里的Table_locks_waited状态值比较高，那么说明系统中表级锁定争用现象比较严重，就需要进一步分析为什么会有较多的锁定资源争用了。 - -?> 此外，Myisam的读写锁调度是写优先，这也是myisam不适合做写为主表的引擎。因为写锁后，其他线程不能做任何操作，大量的更新会使查询很难得到锁，从而造成永远阻塞 - -2. **缩短锁定时间** - - 如何让锁定时间尽可能的短呢？唯一的办法就是让我们的Query执行时间尽可能的短。 - -- **尽两减少大的复杂Query，将复杂Query分拆成几个小的Query分布进行；** -- **尽可能的建立足够高效的索引，让数据检索更迅速；** -- **尽量让MyISAM存储引擎的表只存放必要的信息，控制字段类型；** -- **利用合适的机会优化MyISAM表数据文件。** - -3. 分离能并行的操作 - - 说到MyISAM的表锁，而且是读写互相阻塞的表锁，可能有些人会认为在MyISAM存储引擎的表上就只能是完全的串行化，没办法再并行了。大家不要忘记了，MyISAM的存储引擎还有一个非常有用的特性，那就是ConcurrentInsert（并发插入）的特性。 - - MyISAM存储引擎有一个控制是否打开Concurrent Insert功能的参数选项：`concurrent_insert`，可以设置为0，1或者2。三个值的具体说明如下： - -- concurrent_insert=2，无论MyISAM表中有没有空洞，都允许在表尾并发插入记录； - -- concurrent_insert=1，如果MyISAM表中没有空洞（即表的中间没有被删除的行），MyISAM允许在一个进程读表的同时，另一个进程从表尾插入记录。这也是MySQL的默认设置； - -- concurrent_insert=0，不允许并发插入。 - - 可以利用MyISAM存储引擎的并发插入特性，来解决应用中对同一表查询和插入的锁争用。例如，将concurrent_insert系统变量设为2，总是允许并发插入；同时，通过定期在系统空闲时段执行OPTIMIZE TABLE语句来整理空间碎片，收回因删除记录而产生的中间空洞。 - -4. 合理利用读写优先级 - - MyISAM存储引擎的是读写互相阻塞的，那么，一个进程请求某个MyISAM表的读锁，同时另一个进程也请求同一表的写锁，MySQL如何处理呢？ - - 答案是写进程先获得锁。不仅如此，即使读请求先到锁等待队列，写请求后到，写锁也会插到读锁请求之前。 - - 这是因为MySQL的表级锁定对于读和写是有不同优先级设定的，默认情况下是写优先级要大于读优先级。 - - 所以，如果我们可以根据各自系统环境的差异决定读与写的优先级： - - 通过执行命令SET LOW_PRIORITY_UPDATES=1，使该连接读比写的优先级高。如果我们的系统是一个以读为主，可以设置此参数，如果以写为主，则不用设置； - - 通过指定INSERT、UPDATE、DELETE语句的LOW_PRIORITY属性，降低该语句的优先级。 - - 虽然上面方法都是要么更新优先，要么查询优先的方法，但还是可以用其来解决查询相对重要的应用（如用户登录系统）中，读锁等待严重的问题。 - - 另外，MySQL也提供了一种折中的办法来调节读写冲突，即给系统参数max_write_lock_count设置一个合适的值，当一个表的读锁达到这个值后，MySQL就暂时将写请求的优先级降低，给读进程一定获得锁的机会。 - - 这里还要强调一点：一些需要长时间运行的查询操作，也会使写进程“饿死”，因此，应用中应尽量避免出现长时间运行的查询操作，不要总想用一条SELECT语句来解决问题，因为这种看似巧妙的SQL语句，往往比较复杂，执行时间较长，在可能的情况下可以通过使用中间表等措施对SQL语句做一定的“分解”，使每一步查询都能在较短时间完成，从而减少锁冲突。如果复杂查询不可避免，应尽量安排在数据库空闲时段执行，比如一些定期统计可以安排在夜间执行。 - - - -### 行锁（偏写） - -- 偏向InnoDB存储引擎，开销大，加锁慢；会出现死锁；锁定粒度最小，发生锁冲突的概率最低,并发度也最高。 - -- InnoDB与MyISAM的最大不同有两点：一是支持事务（TRANSACTION）；二是采用了行级锁 - - - -Innodb存储引擎由于实现了行级锁定，虽然在锁定机制的实现方面所带来的性能损耗可能比表级锁定会要更高一些，但是在整体并发处理能力方面要远远优于MyISAM的表级锁定的。当系统并发量较高的时候，Innodb的整体性能和MyISAM相比就会有比较明显的优势了。 - - - -1. InnoDB锁定模式及实现机制 - - InnoDB的行级锁定同样分为两种类型，**共享锁和排他锁**，而在锁定机制的实现过程中为了让行级锁定和表级锁定共存，InnoDB也同样使用了**意向锁**（表级锁定）的概念，也就有了**意向共享锁**和**意向排他锁**这两种。 - - 当一个事务需要给自己需要的某个资源加锁的时候，如果遇到一个共享锁正锁定着自己需要的资源的时候，自己可以再加一个共享锁，不过不能加排他锁。但是，如果遇到自己需要锁定的资源已经被一个排他锁占有之后，则只能等待该锁定释放资源之后自己才能获取锁定资源并添加自己的锁定。而意向锁的作用就是当一个事务在需要获取资源锁定的时候，如果遇到自己需要的资源已经被排他锁占用的时候，该事务可以需要锁定行的表上面添加一个合适的意向锁。如果自己需要一个共享锁，那么就在表上面添加一个意向共享锁。而如果自己需要的是某行（或者某些行）上面添加一个排他锁的话，则先在表上面添加一个意向排他锁。意向共享锁可以同时并存多个，但是意向排他锁同时只能有一个存在。所以，可以说**InnoDB的锁定模式实际上可以分为四种：共享锁（S），排他锁（X），意向共享锁（IS）和意向排他锁（IX）**，我们可以通过以下表格来总结上面这四种所的共存逻辑关系： - -![img](G:/youdaoLocalData/jstarfish@126.com/e78be9952e844cdd95279667202076c5/4-1417117507.png) - -如果一个事务请求的锁模式与当前的锁兼容，InnoDB就将请求的锁授予该事务；反之，如果两者不兼容，该事务就要等待锁释放。 - -意向锁是InnoDB自动加的，不需用户干预。**对于UPDATE、DELETE和INSERT语句，InnoDB会自动给涉及数据集加排他锁**（X)；对于普通SELECT语句，InnoDB不会加任何锁；事务可以通过以下语句显示给记录集加共享锁或排他锁。 - -共享锁（S）：SELECT * FROM table_name WHERE ... LOCK IN SHARE MODE - -排他锁（X)：SELECT * FROM table_name WHERE ... FOR UPDATE - -用SELECT ... IN SHARE MODE获得共享锁，主要用在需要数据依存关系时来确认某行记录是否存在，并确保没有人对这个记录进行UPDATE或者DELETE操作。 - -但是如果当前事务也需要对该记录进行更新操作，则很有可能造成死锁，对于锁定行记录后需要进行更新操作的应用，应该使用SELECT... FOR UPDATE方式获得排他锁。 - -2. InnoDB行锁实现方式 - - **InnoDB行锁是通过给索引上的索引项加锁来实现的，只有通过索引条件检索数据，InnoDB才使用行级锁，否则，InnoDB将使用表锁** - - 在实际应用中，要特别注意InnoDB行锁的这一特性，不然的话，可能导致大量的锁冲突，从而影响并发性能。下面通过一些实际例子来加以说明。 - - （1）在不通过索引条件查询的时候，InnoDB确实使用的是表锁，而不是行锁。 - - （2）由于MySQL的行锁是针对索引加的锁，不是针对记录加的锁，所以虽然是访问不同行的记录，但是如果是使用相同的索引键，是会出现锁冲突的。 - - （3）当表有多个索引的时候，不同的事务可以使用不同的索引锁定不同的行，另外，不论是使用主键索引、唯一索引或普通索引，InnoDB都会使用行锁来对数据加锁。 - - （4）即便在条件中使用了索引字段，但是否使用索引来检索数据是由MySQL通过判断不同执行计划的代价来决定的，如果MySQL认为全表扫描效率更高，比如对一些很小的表，它就不会使用索引，这种情况下InnoDB将使用表锁，而不是行锁。因此，**在分析锁冲突时，别忘了检查SQL的执行计划，以确认是否真正使用了索引**。 - - - -#### 如何分析行锁定 - -通过检查InnoDB_row_lock状态变量来分析系统上的行锁的争夺情况 - -```mysql -mysql>show status like 'innodb_row_lock%'; -``` - -![image-20191204150506938](../_images/mysql/raw-lock.png) - - -对各个状态量的说明如下： - -Innodb_row_lock_current_waits：当前正在等待锁定的数量； -Innodb_row_lock_time：从系统启动到现在锁定总时间长度； -Innodb_row_lock_time_avg：每次等待所花平均时间； -Innodb_row_lock_time_max：从系统启动到现在等待最常的一次所花的时间； -Innodb_row_lock_waits：系统启动后到现在总共等待的次数； -对于这5个状态变量，比较重要的主要是 - Innodb_row_lock_time_avg（等待平均时长）， - Innodb_row_lock_waits（等待总次数） - Innodb_row_lock_time（等待总时长）这三项。 -尤其是当等待次数很高，而且每次等待时长也不小的时候，我们就需要分析系统中为什么会有如此多的等待，然后根据分析结果着手指定优化计划。 - - - -#### 行锁优化 - -- 尽可能让所有数据检索都通过索引来完成，避免无索引行锁升级为表锁。 - -- 合理设计索引，尽量缩小锁的范围 - -- 尽可能较少检索条件，避免间隙锁 - -- 尽量控制事务大小，减少锁定资源量和时间长度 - -- 尽可能低级别事务隔离 - - - -### 页锁 - -开销和加锁时间界于表锁和行锁之间；会出现死锁；锁定粒度界于表锁和行锁之间，并发度一般。 - - - - - -## 死锁 - -死锁是指两个或者多个事务在同一资源上互相占用，并请求锁定对方占用的资源，从而导致恶性循环的现象。当多个事务试图以不同的顺序锁定资源时，就可能会产生死锁。多个事务同时锁定同一个资源时，也会产生死锁。 - - - -乐观锁 - -悲观锁 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/docs/data-store/MySQL/MySQL-Master-Slave.md b/docs/data-store/MySQL/MySQL-Master-Slave.md deleted file mode 100644 index 30404ce4c5..0000000000 --- a/docs/data-store/MySQL/MySQL-Master-Slave.md +++ /dev/null @@ -1 +0,0 @@ -TODO \ No newline at end of file diff --git a/docs/data-store/MySQL/MySQL-Optimization.md b/docs/data-store/MySQL/MySQL-Optimization.md deleted file mode 100644 index 02f27b2866..0000000000 --- a/docs/data-store/MySQL/MySQL-Optimization.md +++ /dev/null @@ -1,430 +0,0 @@ -《高性能MySQL》给出的性能定义：完成某件任务所需要的的时间度量，性能既响应时间。 - -假设性能优化就是在一定负载下尽可能的降低响应时间。 - -性能监测工具： **New Relic** **OneAPM** - -## 1. 影响mysql的性能因素 - -##### 1.1 业务需求对mysql的影响(合适合度) - -##### 1.2 存储定位对mysql的影响 - -- 不适合放进mysql的数据 - - 二进制多媒体数据 - - 流水队列数据 - - 超大文本数据 -- 需要放进缓存的数据 - - 系统各种配置及规则数据 - - 活跃用户的基本信息数据 - - 活跃用户的个性化定制信息数据 - - 准实时的统计信息数据 - - 其他一些访问频繁但变更较少的数据 - -##### 1.3 Schema设计对系统的性能影响 - -- 尽量减少对数据库访问的请求 -- 尽量减少无用数据的查询请求 - -##### 1.4 硬件环境对系统性能的影响 - -**典型OLTP应用系统** - - 什么是OLTP：OLTP即联机事务处理，就是我们经常说的关系数据库，意即记录即时的增、删、改、查，就是我们经常应用的东西，这是数据库的基础 - -对于各种数据库系统环境中大家最常见的OLTP系统，其特点是并发量大，整体数据量比较多，但每次访问的数据比较少，且访问的数据比较离散，活跃数据占总体数据的比例不是太大。对于这类系统的数据库实际上是最难维护，最难以优化的，对主机整体性能要求也是最高的。因为不仅访问量很高，数据量也不小。 - -针对上面的这些特点和分析，我们可以对OLTP的得出一个大致的方向。 虽然系统总体数据量较大，但是系统活跃数据在数据总量中所占的比例不大，那么我们可以通过扩大内存容量来尽可能多的将活跃数据cache到内存中； 虽然IO访问非常频繁，但是每次访问的数据量较少且很离散，那么我们对磁盘存储的要求是IOPS表现要很好，吞吐量是次要因素； 并发量很高，CPU每秒所要处理的请求自然也就很多，所以CPU处理能力需要比较强劲； 虽然与客户端的每次交互的数据量并不是特别大，但是网络交互非常频繁，所以主机与客户端交互的网络设备对流量能力也要求不能太弱。 - -**典型OLAP应用系统** - -用于数据分析的OLAP系统的主要特点就是数据量非常大，并发访问不多，但每次访问所需要检索的数据量都比较多，而且数据访问相对较为集中，没有太明显的活跃数据概念。 - -什么是OLAP：OLAP即联机分析处理，是数据仓库的核心部心，所谓数据仓库是对于大量已经由OLTP形成的数据的一种分析型的数据库，用于处理商业智能、决策支持等重要的决策信息；数据仓库是在数据库应用到一定程序之后而对历史数据的加工与分析 基于OLAP系统的各种特点和相应的分析，针对OLAP系统硬件优化的大致策略如下： 数据量非常大，所以磁盘存储系统的单位容量需要尽量大一些； 单次访问数据量较大，而且访问数据比较集中，那么对IO系统的性能要求是需要有尽可能大的每秒IO吞吐量，所以应该选用每秒吞吐量尽可能大的磁盘； 虽然IO性能要求也比较高，但是并发请求较少，所以CPU处理能力较难成为性能瓶颈，所以CPU处理能力没有太苛刻的要求； - -虽然每次请求的访问量很大，但是执行过程中的数据大都不会返回给客户端，最终返回给客户端的数据量都较小，所以和客户端交互的网络设备要求并不是太高； - -此外，由于OLAP系统由于其每次运算过程较长，可以很好的并行化，所以一般的OLAP系统都是由多台主机构成的一个集群，而集群中主机与主机之间的数据交互量一般来说都是非常大的，所以在集群中主机之间的网络设备要求很高。 - - - -## 2. 性能分析 - -### 2.1 MySQL常见瓶颈 - -- CPU：CPU在饱和的时候一般发生在数据装入内存或从磁盘上读取数据时候 - -- IO：磁盘I/O瓶颈发生在装入数据远大于内存容量的时候 - -- 服务器硬件的性能瓶颈：top,free, iostat和vmstat来查看系统的性能状态 - - - -**查看Linux系统性能的常用命令** - -MySQL数据库是常见的两个瓶颈是CPU和I/O的瓶颈。CPU在饱和的时候一般发生在数据装入内存或从磁盘上读取数据时候，磁盘I/O瓶颈发生在装入数据远大于内存容量的时候，如果应用分布在网络上，那么查询量相当大的时候那么瓶颈就会出现在网络上。Linux中我们常用mpstat、vmstat、iostat、sar和top来查看系统的性能状态。 - -`mpstat`： mpstat是Multiprocessor Statistics的缩写，是实时系统监控工具。其报告为CPU的一些统计信息，这些信息存放在/proc/stat文件中。在多CPUs系统里，其不但能查看所有CPU的平均状况信息，而且能够查看特定CPU的信息。mpstat最大的特点是可以查看多核心cpu中每个计算核心的统计数据，而类似工具vmstat只能查看系统整体cpu情况。 - -`vmstat`：vmstat命令是最常见的Linux/Unix监控工具，可以展现给定时间间隔的服务器的状态值，包括服务器的CPU使用率，内存使用，虚拟内存交换情况，IO读写情况。这个命令是我查看Linux/Unix最喜爱的命令，一个是Linux/Unix都支持，二是相比top，我可以看到整个机器的CPU、内存、IO的使用情况，而不是单单看到各个进程的CPU使用率和内存使用率(使用场景不一样)。 - -`iostat`: 主要用于监控系统设备的IO负载情况，iostat首次运行时显示自系统启动开始的各项统计信息，之后运行iostat将显示自上次运行该命令以后的统计信息。用户可以通过指定统计的次数和时间来获得所需的统计信息。 - -`sar`： sar（System Activity Reporter系统活动情况报告）是目前 Linux 上最为全面的系统性能分析工具之一，可以从多方面对系统的活动进行报告，包括：文件的读写情况、系统调用的使用情况、磁盘I/O、CPU效率、内存使用状况、进程活动及IPC有关的活动等。 - -`top`：top命令是Linux下常用的性能分析工具，能够实时显示系统中各个进程的资源占用状况，类似于Windows的任务管理器。top显示系统当前的进程和其他状况，是一个动态显示过程,即可以通过用户按键来不断刷新当前状态.如果在前台执行该命令，它将独占前台，直到用户终止该程序为止。比较准确的说，top命令提供了实时的对系统处理器的状态监视。它将显示系统中CPU最“敏感”的任务列表。该命令可以按CPU使用。内存使用和执行时间对任务进行排序；而且该命令的很多特性都可以通过交互式命令或者在个人定制文件中进行设定。 - -除了服务器硬件的性能瓶颈，对于MySQL系统本身，我们可以使用工具来优化数据库的性能，通常有三种：使用索引，使用EXPLAIN分析查询以及调整MySQL的内部配置。 - - - -### 2.2 性能下降SQL慢 执行时间长 等待时间长 原因分析 - -- 查询语句写的烂 -- 索引失效（单值 复合） -- 关联查询太多join（设计缺陷或不得已的需求） -- 服务器调优及各个参数设置（缓冲、线程数等） - - - -### 2.3 MySql Query Optimizer - -1. Mysql中有专门负责优化SELECT语句的优化器模块，主要功能：通过计算分析系统中收集到的统计信息，为客户端请求的Query提供他认为最优的执行计划（他认为最优的数据检索方式，但不见得是DBA认为是最优的，这部分最耗费时间） - -2. 当客户端向MySQL 请求一条Query，命令解析器模块完成请求分类，区别出是 SELECT 并转发给MySQL Query Optimizer时，MySQL Query Optimizer 首先会对整条Query进行优化，处理掉一些常量表达式的预算，直接换算成常量值。并对 Query 中的查询条件进行简化和转换，如去掉一些无用或显而易见的条件、结构调整等。然后分析 Query 中的 Hint 信息（如果有），看显示Hint信息是否可以完全确定该Query 的执行计划。如果没有 Hint 或Hint 信息还不足以完全确定执行计划，则会读取所涉及对象的统计信息，根据 Query 进行写相应的计算分析，然后再得出最后的执行计划。 - - - -### 2.4 MySQL常见性能分析手段 - -在优化MySQL时，通常需要对数据库进行分析，常见的分析手段有**慢查询日志**，**EXPLAIN 分析查询**，**profiling分析**以及**show命令查询系统状态及系统变量**，通过定位分析性能的瓶颈，才能更好的优化数据库系统的性能。 - -#### 2.4.1 性能瓶颈定位 - -我们可以通过show命令查看MySQL状态及变量，找到系统的瓶颈： - -```shell -Mysql> show status ——显示状态信息（扩展show status like ‘XXX’） - -Mysql> show variables ——显示系统变量（扩展show variables like ‘XXX’） - -Mysql> show innodb status ——显示InnoDB存储引擎的状态 - -Mysql> show processlist ——查看当前SQL执行，包括执行状态、是否锁表等 - -Shell> mysqladmin variables -u username -p password——显示系统变量 - -Shell> mysqladmin extended-status -u username -p password——显示状态信息 -``` - - - -#### 2.4.2 Explain(执行计划) - -- 是什么：使用Explain关键字可以模拟优化器执行SQL查询语句，从而知道MySQL是如何处理你的SQL语句的。分析你的查询语句或是表结构的性能瓶颈 -- 能干吗 - - 表的读取顺序 - - 数据读取操作的操作类型 - - 哪些索引可以使用 - - 哪些索引被实际使用 - - 表之间的引用 - - 每张表有多少行被优化器查询 - -- 怎么玩 - - - Explain + SQL语句 - - 执行计划包含的信息 - -![expalin](../_images/mysql/expalin.jpg) - -- 各字段解释 - - - **id**（select查询的序列号，包含一组数字，表示查询中执行select子句或操作表的顺序） - - - id相同，执行顺序从上往下 - - id不同，如果是子查询，id的序号会递增，id值越大优先级越高，越先被执行 - - id相同不同，同时存在 - - - **select_type**（查询的类型，用于区别普通查询、联合查询、子查询等复杂查询） - - - **SIMPLE** ：简单的select查询，查询中不包含子查询或UNION - - **PRIMARY**：查询中若包含任何复杂的子部分，最外层查询被标记为PRIMARY - - **SUBQUERY**：在select或where列表中包含了子查询 - - **DERIVED**：在from列表中包含的子查询被标记为DERIVED，mysql会递归执行这些子查询，把结果放在临时表里 - - **UNION**：若第二个select出现在UNION之后，则被标记为UNION，若UNION包含在from子句的子查询中，外层select将被标记为DERIVED - - **UNION RESULT**：从UNION表获取结果的select - - - **table**（显示这一行的数据是关于哪张表的） - - - **type**（显示查询使用了那种类型，从最好到最差依次排列 **system > const > eq_ref > ref > fulltext > ref_or_null > index_merge > unique_subquery > index_subquery > range > index > ALL** ） - - - system：表只有 一行记录（等于系统表），是const类型的特例，平时不会出现 - - const：表示通过索引一次就找到了，const用于比较primary key或unique索引，因为只要匹配一行数据，所以很快，如将主键置于where列表中，mysql就能将该查询转换为一个常量 - - eq_ref：唯一性索引扫描，对于每个索引键，表中只有一条记录与之匹配，常见于主键或唯一索引扫描 - - ref：非唯一性索引扫描，范围匹配某个单独值得所有行。本质上也是一种索引访问，他返回所有匹配某个单独值的行，然而，它可能也会找到多个符合条件的行，多以他应该属于查找和扫描的混合体 - - range：只检索给定范围的行，使用一个索引来选择行。key列显示使用了哪个索引，一般就是在你的where语句中出现了between、<、>、in等的查询，这种范围扫描索引比全表扫描要好，因为它只需开始于索引的某一点，而结束于另一点，不用扫描全部索引 - - index：Full Index Scan，index于ALL区别为index类型只遍历索引树。通常比ALL快，因为索引文件通常比数据文件小。（**也就是说虽然all和index都是读全表，但index是从索引中读取的，而all是从硬盘中读的**） - - all：Full Table Scan，将遍历全表找到匹配的行 - - ?> 一般来说，得保证查询至少达到range级别，最好到达ref - - - **possible_keys**（显示可能应用在这张表中的索引，一个或多个，查询涉及到的字段若存在索引，则该索引将被列出，但不一定被查询实际使用） - - - **key** - - - （实际使用的索引，如果为NULL，则没有使用索引） - - - **查询中若使用了覆盖索引，则该索引和查询的select字段重叠，仅出现在key列表中** - - ![explain-key](../_images/mysql/explain-key.png) - - - **key_len** - - 表示索引中使用的字节数，可通过该列计算查询中使用的索引的长度。在不损失精确性的情况下，长度越短越好 - - key_len显示的值为索引字段的最大可能长度，并非实际使用长度，即key_len是根据表定义计算而得，不是通过表内检索出的 - - - **ref** （显示索引的哪一列被使用了，如果可能的话，是一个常数。哪些列或常量被用于查找索引列上的值） - - **rows** （根据表统计信息及索引选用情况，大致估算找到所需的记录所需要读取的行数） - - **Extra**（包含不适合在其他列中显示但十分重要的额外信息） - - 1. using filesort: 说明mysql会对数据使用一个外部的索引排序，不是按照表内的索引顺序进行读取。mysql中无法利用索引完成的排序操作称为“文件排序”![img](../_images/mysql/explain-extra-1.png) - - 2. Using temporary：使用了临时表保存中间结果，mysql在对查询结果排序时使用临时表。常见于排序order by和分组查询group by。![explain-extra-2](../_images/mysql/explain-extra-2.png) - - 3. using index：表示相应的select操作中使用了覆盖索引，避免访问了表的数据行，效率不错，如果同时出现using where，表明索引被用来执行索引键值的查找；否则索引被用来读取数据而非执行查找操作![explain-extra-3](../_images/mysql/explain-extra-3.png) - - 4. using where：使用了where过滤 - - 5. using join buffer：使用了连接缓存 - - 6. impossible where：where子句的值总是false，不能用来获取任何元祖 - - 7. select tables optimized away：在没有group by子句的情况下，基于索引优化操作或对于MyISAM存储引擎优化COUNT(*)操作，不必等到执行阶段再进行计算，查询执行计划生成的阶段即完成优化 - - 8. distinct：优化distinct操作，在找到第一匹配的元祖后即停止找同样值的动作 - - - -- case: - -![explain-demo](../_images/mysql/explain-demo.png) - - - -1. 第一行（执行顺序4）：id列为1，表示是union里的第一个select，select_type列的primary表 示该查询为外层查询，table列被标记为，表示查询结果来自一个衍生表，其中derived3中3代表该查询衍生自第三个select查询，即id为3的select。【select d1.name......】 - -2. 第二行（执行顺序2）：id为3，是整个查询中第三个select的一部分。因查询包含在from中，所以为derived。【select id,name from t1 where other_column=''】 -3. 第三行（执行顺序3）：select列表中的子查询select_type为subquery，为整个查询中的第二个select。【select id from t3】 -4. 第四行（执行顺序1）：select_type为union，说明第四个select是union里的第二个select，最先执行【select name,id from t2】 -5. 第五行（执行顺序5）：代表从union的临时表中读取行的阶段，table列的表示用第一个和第四个select的结果进行union操作。【两个结果union操作】 - - - -#### 2.4.3 慢查询日志 - -MySQL的慢查询日志是MySQL提供的一种日志记录，它用来记录在MySQL中响应时间超过阈值的语句，具体指运行时间超过long_query_time值的SQL，则会被记录到慢查询日志中。 - -- long_query_time的默认值为10，意思是运行10秒以上的语句。 -- 默认情况下，MySQL数据库没有开启慢查询日志，需要手动设置参数开启。 -- 如果不是调优需要的话，一般不建议启动该参数。 - -**查看开启状态** - -`SHOW VARIABLES LIKE '%slow_query_log%'` - -**开启慢查询日志** - -- 临时配置： - -```mysql -mysql> set global slow_query_log='ON'; -mysql> set global slow_query_log_file='/var/lib/mysql/hostname-slow.log'; -mysql> set global long_query_time=2; -``` - -​ 也可set文件位置，系统会默认给一个缺省文件host_name-slow.log - -​ 使用set操作开启慢查询日志只对当前数据库生效，如果MySQL重启则会失效。 - -- 永久配置 - - 修改配置文件my.cnf或my.ini，在[mysqld]一行下面加入两个配置参数 - -```cnf -[mysqld] -slow_query_log = ON -slow_query_log_file = /var/lib/mysql/hostname-slow.log -long_query_time = 3 -``` - -​ 注：log-slow-queries参数为慢查询日志存放的位置，一般这个目录要有mysql的运行帐号的可写权限，一般都 将这个目录设置为mysql的数据存放目录；long_query_time=2中的2表示查询超过两秒才记录；在my.cnf或者 my.ini中添加log-queries-not-using-indexes参数，表示记录下没有使用索引的查询。 - -可以用 `select sleep(4)` 验证是否成功开启。 - -在生产环境中，如果手工分析日志，查找、分析SQL，还是比较费劲的，所以MySQL提供了日志分析工具mysqldumpslow。 - -通过 mysqldumpslow --help查看操作帮助信息 - -- 得到返回记录集最多的10个SQL - - `mysqldumpslow -s r -t 10 /var/lib/mysql/hostname-slow.log` - -- 得到访问次数最多的10个SQL - - `mysqldumpslow -s c -t 10 /var/lib/mysql/hostname-slow.log` - -- 得到按照时间排序的前10条里面含有左连接的查询语句 - - `mysqldumpslow -s t -t 10 -g "left join" /var/lib/mysql/hostname-slow.log` - -- 也可以和管道配合使用 - - `mysqldumpslow -s r -t 10 /var/lib/mysql/hostname-slow.log | more` - -**也可使用 pt-query-digest 分析 RDS MySQL 慢查询日志** - - - -#### 2.4.4 Show Profile分析查询 - -通过慢日志查询可以知道哪些SQL语句执行效率低下，通过explain我们可以得知SQL语句的具体执行情况，索引使用等，还可以结合Show Profile命令查看执行状态。 - -- Show Profile是mysql提供可以用来分析当前会话中语句执行的资源消耗情况。可以用于SQL的调优的测量 - -- 默认情况下，参数处于关闭状态，并保存最近15次的运行结果 - -- 分析步骤 - - 1. 是否支持，看看当前的mysql版本是否支持 - - ```mysql - mysql>Show variables like 'profiling'; --默认是关闭，使用前需要开启 - ``` - - 2. 开启功能，默认是关闭，使用前需要开启 - - ```mysql - mysql>set profiling=1; - ``` - - 3. 运行SQL - - 4. 查看结果，show profiles; - - 5. 诊断SQL，show profile cpu,block io for query 上一步前面的问题SQL数字号码; - - 6. 日常开发需要注意的结论 - - - converting HEAP to MyISAM 查询结果太大，内存都不够用了往磁盘上搬了。 - - - create tmp table 创建临时表，这个要注意 - - - Copying to tmp table on disk 把内存临时表复制到磁盘 - - - locked - - - -## 3. 性能优化 - -### 3.1 索引优化 - -1. 全值匹配我最爱 -2. 最佳左前缀法则 -3. 不在索引列上做任何操作（计算、函数、(自动or手动)类型转换），会导致索引失效而转向全表扫描 -4. 存储引擎不能使用索引中范围条件右边的列 -5. 尽量使用覆盖索引(只访问索引的查询(索引列和查询列一致))，减少select -6. mysql 在使用不等于(!= 或者<>)的时候无法使用索引会导致全表扫描 -7. is null ,is not null 也无法使用索引 -8. like以通配符开头('%abc...')mysql索引失效会变成全表扫描的操作 -9. 字符串不加单引号索引失效 -10. 少用or,用它来连接时会索引失效 - - - -**一般性建议** - -- 对于单键索引，尽量选择针对当前query过滤性更好的索引 - -- 在选择组合索引的时候，当前Query中过滤性最好的字段在索引字段顺序中，位置越靠前越好。 - -- 在选择组合索引的时候，尽量选择可以能够包含当前query中的where字句中更多字段的索引 - -- 尽可能通过分析统计信息和调整query的写法来达到选择合适索引的目的 - -- 少用Hint强制索引 - - - -### 3.2 查询优化 - -- **永远小标驱动大表（小的数据集驱动大的数据集）** - - `slect * from A where id in (select id from B)` - - 等价于 - - `select id from B` - - `select * from A where A.id=B.id` - - 当B表的数据集必须小于A表的数据集时，用in优于exists - - `select * from A where exists (select 1 from B where B.id=A.id)` - - 等价于 - - `select *from A` - - `select * from B where B.id = A.id` - - 当A表的数据集小于B表的数据集时，用exists优于用in - - 注意：A表与B表的ID字段应建立索引。 - - - -- order by关键字优化 - -- - order by子句，尽量使用Index方式排序，避免使用FileSort方式排序 - -- - - mysql支持两种方式的排序，FileSort和Index,Index效率高，它指MySQL扫描索引本身完成排序，FileSort效率较低； - - ORDER BY 满足两种情况，会使用Index方式排序；①ORDER BY语句使用索引最左前列 ②使用where子句与ORDER BY子句条件列组合满足索引最左前列 - -![optimization-orderby](../_images/mysql/optimization-orderby.png) - -- - 尽可能在索引列上完成排序操作，遵照索引建的最佳最前缀 - - 如果不在索引列上，filesort有两种算法，mysql就要启动双路排序和单路排序 - -- - - 双路排序 - - 单路排序 - - 由于单路是后出的，总体而言好过双路 - -- - 优化策略 - -- - - 增大sort_buffer_size参数的设置 - - - 增大max_lencth_for_sort_data参数的设置 - - ![optimization-orderby2](../_images/mysql/optimization-orderby2.png) - -- GROUP BY关键字优化 - - group by实质是先排序后进行分组，遵照索引建的最佳左前缀 - - 当无法使用索引列，增大max_length_for_sort_data参数的设置+增大sort_buffer_size参数的设置 - - where高于having，能写在where限定的条件就不要去having限定了。 - - - -### 3.3 数据类型优化 - -MySQL支持的数据类型非常多，选择正确的数据类型对于获取高性能至关重要。不管存储哪种类型的数据，下面几个简单的原则都有助于做出更好的选择。 - -- 更小的通常更好：一般情况下，应该尽量使用可以正确存储数据的最小数据类型。 - - 简单就好：简单的数据类型通常需要更少的CPU周期。例如，整数比字符操作代价更低，因为字符集和校对规则（排序规则）使字符比较比整型比较复杂。 - -- 尽量避免NULL：通常情况下最好指定列为NOT NULL - - - -> https://www.jianshu.com/p/3c79039e82aa - diff --git a/docs/data-store/MySQL/MySQL-Schema.md b/docs/data-store/MySQL/MySQL-Schema.md deleted file mode 100644 index 411e779fcb..0000000000 --- a/docs/data-store/MySQL/MySQL-Schema.md +++ /dev/null @@ -1,237 +0,0 @@ -- 尽量避免过度设计，例如会导致极其复杂查询的schema设计，或者有很多列的表设计 -- 使用小而简单的合适数据类型，除非真实数据模型中有确切的需要，否则应该尽可能的避免使用NULL值。 -- 尽量使用相同的数据类型存储相似或相关的值，尤其是要在关联条件中使用的列。 -- 注意可变长字符串，其在临时表和排序时可能导致悲观的按最大长度分配内存。 -- 尽量使用整型定义标识列。 -- 避免使用MySQL已经遗弃的特性，例如指定浮点数的精度，或者整数的显示宽度。 -- 小心使用ENUM和SET。 -- 最好避免使用BIT。 - - - - - -# 58到家数据库30条军规解读 - -**军规适用场景**：并发量大、数据量大的互联网业务 - -**军规**：介绍内容 - -**解读**：讲解原因，解读比军规更重要 - - - -## 一、基础规范 - -**（1）必须使用InnoDB存储引擎** - -解读：支持事务、行级锁、并发性能更好、CPU及内存缓存页优化使得资源利用率更高 - - - -**（2）必须使用UTF8字符集** - -解读：万国码，无需转码，无乱码风险，节省空间 - - - -**（3）数据表、数据字段必须加入中文注释** - -解读：N年后谁tm知道这个r1,r2,r3字段是干嘛的 - - - -**（4）禁止使用存储过程、视图、触发器、Event** - -解读：高并发大数据的互联网业务，架构设计思路是“解放数据库CPU，将计算转移到服务层”，并发量大的情况下，这些功能很可能将数据库拖死，业务逻辑放到服务层具备更好的扩展性，能够轻易实现“增机器就加性能”。数据库擅长存储与索引，CPU计算还是上移吧 - - - -**（5）禁止存储大文件或者大照片** - -解读：为何要让数据库做它不擅长的事情？大文件和照片存储在文件系统，数据库里存URI多好 - - - -## 二、命名规范 - -**（6）只允许使用内网域名，而不是ip连接数据库** - - - -**（7）线上环境、开发环境、测试环境数据库内网域名遵循命名规范** - -业务名称：xxx - -线上环境：dj.xxx.db - -开发环境：dj.xxx.rdb - -测试环境：dj.xxx.tdb - -**从库**在名称后加-s标识，**备库**在名称后加-ss标识 - -线上从库：dj.xxx-s.db - -线上备库：dj.xxx-sss.db - - -**（8）库名、表名、字段名：小写，下划线风格，不超过32个字符，必须见名知意，禁止拼音英文混用 - -（9）表名t_xxx，非唯一索引名idx_xxx，唯一索引名uniq_xxx** - -**（10）单实例表数目必须小于500** - - -**（11）单表列数目必须小于30** - -解读： - -a）主键递增，数据行写入可以提高插入性能，可以避免page分裂，减少表碎片提升空间和内存的使用 - -b）主键要选择较短的数据类型， Innodb引擎普通索引都会保存主键的值，较短的数据类型可以有效的减少索引的磁盘空间，提高索引的缓存效率 - -c） 无主键的表删除，在row模式的主从架构，会导致备库夯住 - - - -**（13）禁止使用外键，如果有外键完整性约束，需要应用程序控制** - -解读：外键会导致表与表之间耦合，update与delete操作都会涉及相关联的表，十分影响sql 的性能，甚至会造成死锁。高并发情况下容易造成数据库性能，大数据高并发业务场景数据库使用以性能优先 - - - -## 四、字段设计规范 - -**（14）必须把字段定义为NOT NULL并且提供默认值** - -解读： - -a）null的列使索引/索引统计/值比较都更加复杂，对MySQL来说更难优化 - -b）null 这种类型MySQL内部需要进行特殊处理，增加数据库处理记录的复杂性；同等条件下，表中有较多空字段的时候，数据库的处理性能会降低很多 - -c）null值需要更多的存储空，无论是表还是索引中每行中的null的列都需要额外的空间来标识 - -d）对null 的处理时候，只能采用is null或is not null，而不能采用=、in、<、<>、!=、not in这些操作符号。如：where name!=’shenjian’，如果存在name为null值的记录，查询结果就不会包含name为null值的记录 - - - -**（15）禁止使用TEXT、BLOB类型** - -解读：会浪费更多的磁盘和内存空间，非必要的大量的大字段查询会淘汰掉热数据，导致内存命中率急剧降低，影响数据库性能 - - - -**（16）禁止使用小数存储货币** - -解读：使用整数吧，小数容易导致钱对不上 - - - -**（17）必须使用varchar(20)存储手机号** - -解读： - -a）涉及到区号或者国家代号，可能出现+-() - -b）手机号会去做数学运算么？ - -c）varchar可以支持模糊查询，例如：like“138%” - - - -**（18）禁止使用ENUM，可使用TINYINT代替** - -解读： - -a）增加新的ENUM值要做DDL操作 - -b）ENUM的内部实际存储就是整数，你以为自己定义的是字符串？ - - - -## 五、索引设计规范 - -**（19）单表索引建议控制在5个以内** - - - -**（20）单索引字段数不允许超过5个** - -解读：字段超过5个时，实际已经起不到有效过滤数据的作用了 - - - -**（21）禁止在更新十分频繁、区分度不高的属性上建立索引** - -解读： - -a）更新会变更B+树，更新频繁的字段建立索引会大大降低数据库性能 - -b）“性别”这种区分度不大的属性，建立索引是没有什么意义的，不能有效过滤数据，性能与全表扫描类似 - - - -**（22）建立组合索引，必须把区分度高的字段放在前面** - -解读：能够更加有效的过滤数据 - - - -## 六、SQL使用规范 - -**（23）禁止使用SELECT \*，只获取必要的字段，需要显示说明列属性** - -解读： - -a）读取不需要的列会增加CPU、IO、NET消耗 - -b）不能有效的利用覆盖索引 - -c）使用SELECT *容易在增加或者删除字段后出现程序BUG - - - -**（24）禁止使用INSERT INTO t_xxx VALUES(xxx)，必须显示指定插入的列属性** - -解读：容易在增加或者删除字段后出现程序BUG - - - -**（25）禁止使用属性隐式转换** - -解读：SELECT uid FROM t_user WHERE phone=13812345678 会导致全表扫描，而不能命中phone索引，猜猜为什么？（这个线上问题不止出现过一次） - - - -**（26）禁止在WHERE条件的属性上使用函数或者表达式** - -解读：SELECT uid FROM t_user WHERE from_unixtime(day)>='2017-02-15' 会导致全表扫描 - -正确的写法是：SELECT uid FROM t_user WHERE day>= unix_timestamp('2017-02-15 00:00:00') - - - -**（27）禁止负向查询，以及%开头的模糊查询** - -解读： - -a）负向查询条件：NOT、!=、<>、!<、!>、NOT IN、NOT LIKE等，会导致全表扫描 - -b）%开头的模糊查询，会导致全表扫描 - - - -**（28）禁止大表使用JOIN查询，禁止大表使用子查询** - -解读：会产生临时表，消耗较多内存与CPU，极大影响数据库性能 - - - -**（29）禁止使用OR条件，必须改为IN查询** - -解读：旧版本Mysql的OR查询是不能命中索引的，即使能命中索引，为何要让数据库耗费更多的CPU帮助实施查询优化呢？ - - -**（30）应用程序必须捕获SQL异常，并有相应处理** \ No newline at end of file diff --git a/docs/data-store/MySQL/MySQL-Storage-Engines.md b/docs/data-store/MySQL/MySQL-Storage-Engines.md deleted file mode 100644 index 4f5c5094e4..0000000000 --- a/docs/data-store/MySQL/MySQL-Storage-Engines.md +++ /dev/null @@ -1,246 +0,0 @@ -# Mysql Storage Engines - -存储引擎是MySQL的组件，用于处理不同表类型的SQL操作。不同的存储引擎提供不同的存储机制、索引技巧、锁定水平等功能，使用不同的存储引擎，还可以获得特定的功能。 - -使用哪一种引擎可以灵活选择，**一个数据库中多个表可以使用不同引擎以满足各种性能和实际需求**，使用合适的存储引擎，将会提高整个数据库的性能 。 - - MySQL服务器使用可插拔的存储引擎体系结构，可以从运行中的MySQL服务器加载或卸载存储引擎 。 - -> [MySQL 5.7 可供选择的存储引擎](https://dev.mysql.com/doc/refman/5.7/en/storage-engines.html) - -### 查看存储引擎 - -```mysql --- 查看支持的存储引擎 -SHOW ENGINES - --- 查看默认存储引擎 -SHOW VARIABLES LIKE 'storage_engine' - ---查看具体某一个表所使用的存储引擎，这个默认存储引擎被修改了！ -show create table tablename - ---准确查看某个数据库中的某一表所使用的存储引擎 -show table status like 'tablename' -show table status from database where name="tablename" -``` - - - -![mysql-engines](../../_images/mysql/mysql-engines.png) - - - -### 设置存储引擎 - -```mysql --- 建表时指定存储引擎。默认的就是INNODB，不需要设置 -CREATE TABLE t1 (i INT) ENGINE = INNODB; -CREATE TABLE t2 (i INT) ENGINE = CSV; -CREATE TABLE t3 (i INT) ENGINE = MEMORY; - --- 修改存储引擎 -ALTER TABLE t ENGINE = InnoDB; - --- 修改默认存储引擎，也可以在配置文件my.cnf中修改默认引擎 -SET default_storage_engine=NDBCLUSTER; -``` - - 默认情况下，每当CREATE TABLE或ALTER TABLE不能使用默认存储引擎时，都会生成一个警告。为了防止在所需的引擎不可用时出现令人困惑的意外行为，可以启用`NO_ENGINE_SUBSTITUTION SQL`模式。如果所需的引擎不可用，则此设置将产生错误而不是警告，并且不会创建或更改表 - - - -### 常用存储引擎 - -#### InnoDB - -**InnoDB是MySQL5.7 默认的存储引擎，主要特性有** - -- InnoDB存储引擎维护自己的缓冲池，在访问数据时将表和索引数据缓存在主内存中 - -- 支持事务 - -- 支持外键 - -- B-Tree索引 - -- 不支持集群 - -- 聚簇索引 - -- 行锁 - -- 支持地理位置的数据类型和索引 - - - -#### MyISAM - -在 5.1 版本之前，MyISAM 是 MySQL 的默认存储引擎，MyISAM 并发性比较差，使用的场景比较少，主要特点是 - -每个MyISAM表存储在磁盘上的三个文件中 。这些文件的名称以表名开头，并有一个扩展名来指示文件类型 。 - -`.frm`文件存储表的格式。 `.MYD` (`MYData`) 文件存储表的数据。 `.MYI` (`MYIndex`) 文件存储索引。 - - **MyISAM表具有以下特征** - -- 每个MyISAM表最大索引数是64，这可以通过重新编译来改变。每个索引最大的列数是16 -- 每个MyISAM表都支持一个`AUTO_INCREMENT`的内部列。当执行`INSERT`或者`UPDATE`操作的时候，MyISAM自动更新这个列，这使得`AUTO_INCREMENT`列更快。 -- 当把删除和更新及插入操作混合使用的时候，动态尺寸的行产生更少碎片。这要通过合并相邻被删除的块，若下一个块被删除，就扩展到下一块自动完成 - -- MyISAM支持**并发插入** -- **可以将数据文件和索引文件放在不同物理设备上的不同目录中**，以更快地使用数据目录和索引目录表选项来创建表 -- BLOB和TEXT列可以被索引 -- **NULL被允许在索引的列中**，这个值占每个键的0~1个字节 -- 每个字符列可以有不同的字符集 -- **`MyISAM` 表使用 B-tree 索引** -- MyISAM表的行最大限制为 (2^32)^2 (1.844E+19) -- 大文件（达到63位文件长度）在支持大文件的文件系统和操作系统上被支持 -- 键的最大长度为1000字节，这也可以通过重新编译来改变，对于键长度超过250字节的情况，一个超过1024字节的键将被用上 - -- VARCHAR支持固定或动态记录长度 -- 表中VARCHAR和CHAR列的长度总和有可能达到64KB -- 任意长度的唯一约束 - -- All data values are stored with the low byte first. This makes the data machine and operating system independent. - -- All numeric key values are stored with the high byte first to permit better index compression - - todo：最后两条没搞懂啥意思 - - - -### 存储引擎对比 - -| 对比项 | MyISAM | InnoDB | -| -------- | -------------------------------------------------------- | ------------------------------------------------------------ | -| 主外键 | 不支持 | 支持 | -| 事务 | 不支持 | 支持 | -| 行表锁 | 表锁，即使操作一条记录也会锁住整个表，不适合高并发的操作 | 行锁,操作时只锁某一行，不对其它行有影响，
适合高并发的操作 | -| 缓存 | 只缓存索引，不缓存真实数据 | 不仅缓存索引还要缓存真实数据，对内存要求较高，而且内存大小对性能有决定性的影响 | -| 表空间 | 小 | 大 | -| 关注点 | 性能 | 事务 | -| 默认安装 | 是 | 是 | - - - -官方提供的多种引擎对比 - -| Feature | MyISAM | Memory | InnoDB | Archive | NDB | -| ------------------------------------------ | ------------ | ---------------- | ------------ | ------------ | ------------ | -| **B-tree indexes** | Yes | Yes | Yes | No | No | -| **Backup/point-in-time recovery** (note 1) | Yes | Yes | Yes | Yes | Yes | -| **Cluster database support** | No | No | No | No | Yes | -| **Clustered indexes** | No | No | Yes | No | No | -| **Compressed data** | Yes (note 2) | No | Yes | Yes | No | -| **Data caches** | No | N/A | Yes | No | Yes | -| **Encrypted data** | Yes (note 3) | Yes (note 3) | Yes (note 4) | Yes (note 3) | Yes (note 3) | -| **Foreign key support** | No | No | Yes | No | Yes (note 5) | -| **Full-text search indexes** | Yes | No | Yes (note 6) | No | No | -| **Geospatial data type support** | Yes | No | Yes | Yes | Yes | -| **Geospatial indexing support** | Yes | No | Yes (note 7) | No | No | -| **Hash indexes** | No | Yes | No (note 8) | No | Yes | -| **Index caches** | Yes | N/A | Yes | No | Yes | -| **Locking granularity** | Table | Table | Row | Row | Row | -| **MVCC** | No | No | Yes | No | No | -| **Replication support** (note 1) | Yes | Limited (note 9) | Yes | Yes | Yes | -| **Storage limits** | 256TB | RAM | 64TB | None | 384EB | -| **T-tree indexes** | No | No | No | No | Yes | -| **Transactions** | No | No | Yes | No | Yes | -| **Update statistics for data dictionary** | Yes | Yes | Yes | Yes | Yes | - - - -### 数据的存储 - -在整个数据库体系结构中，我们可以使用不同的存储引擎来存储数据，而绝大多数存储引擎都以二进制的形式存储数据；这一节会介绍 InnoDB 中对数据是如何存储的。 - -在 InnoDB 存储引擎中，所有的数据都被逻辑地存放在表空间中，表空间（tablespace）是存储引擎中最高的存储逻辑单位，在表空间的下面又包括段（segment）、区（extent）、页（page） - - ![tablespace-segment-extent-page-row](../../_images/mysql/tablespace-segment-extent-page-row.jpg) - - 同一个数据库实例的所有表空间都有相同的页大小；默认情况下，表空间中的页大小都为 16KB，当然也可以通过改变 `innodb_page_size` 选项对默认大小进行修改，需要注意的是不同的页大小最终也会导致区大小的不同 - -![Relation Between Page Size - Extent Size](https://raw.githubusercontent.com/Draveness/Analyze/master/contents/Database/images/mysql/Relation%20Between%20Page%20Size%20-%20Extent%20Size.png) - -从图中可以看出，在 InnoDB 存储引擎中，一个区的大小最小为 1MB，页的数量最少为 64 个。 - - - -#### 如何存储表 - -MySQL 使用 InnoDB 存储表时，会将表的定义和数据索引等信息分开存储，其中前者存储在 `.frm` 文件中，后者存储在 `.ibd` 文件中，这一节就会对这两种不同的文件分别进行介绍。 - -![frm-and-ibd-file](https://raw.githubusercontent.com/Draveness/Analyze/master/contents/Database/images/mysql/frm-and-ibd-file.jpg) - -#### .frm 文件 - -无论在 MySQL 中选择了哪个存储引擎，所有的 MySQL 表都会在硬盘上创建一个 `.frm` 文件用来描述表的格式或者说定义；`.frm` 文件的格式在不同的平台上都是相同的。 - -```mysql -`CREATE TABLE test_frm(`` ``column1 CHAR(5),`` ``column2 INTEGER``);` -``` - -当我们使用上面的代码创建表时，会在磁盘上的 `datadir` 文件夹中生成一个 `test_frm.frm` 的文件，这个文件中就包含了表结构相关的信息： - -![frm-file-hex](../../_images/mysql/frm-file-hex.png) - -> MySQL 官方文档中的 [11.1 MySQL .frm File Format](https://dev.mysql.com/doc/internals/en/frm-file-format.html) 一文对于 `.frm` 文件格式中的二进制的内容有着非常详细的表述。 - -#### .ibd 文件 - -InnoDB 中用于存储数据的文件总共有两个部分，一是系统表空间文件，包括 `ibdata1`、`ibdata2` 等文件，其中存储了 InnoDB 系统信息和用户数据库表数据和索引，是所有表公用的。 - -当打开 `innodb_file_per_table` 选项时，`.ibd` 文件就是每一个表独有的表空间，文件存储了当前表的数据和相关的索引数据。 - -#### 如何存储记录 - -与现有的大多数存储引擎一样，InnoDB 使用页作为磁盘管理的最小单位；数据在 InnoDB 存储引擎中都是按行存储的，每个 16KB 大小的页中可以存放 2-7992 行的记录。（至少是2条记录，最多是7992条记录） - -当 InnoDB 存储数据时，它可以使用不同的行格式进行存储；MySQL 5.7 版本支持以下格式的行存储方式： - -![Antelope-Barracuda-Row-Format](../../_images/mysql/Antelope-Barracuda-Row-Format.jpg) - -Antelope 是 InnoDB 最开始支持的文件格式，它包含两种行格式 Compact 和 Redundant，它最开始并没有名字；Antelope 的名字是在新的文件格式 Barracuda 出现后才起的，Barracuda 的出现引入了两种新的行格式 Compressed 和 Dynamic；InnoDB 对于文件格式都会向前兼容，而官方文档中也对之后会出现的新文件格式预先定义好了名字：Cheetah、Dragon、Elk 等等。 - -两种行记录格式 Compact 和 Redundant 在磁盘上按照以下方式存储： - -![COMPACT-And-REDUNDANT-Row-Format](../../_images/mysql/COMPACT-And-REDUNDANT-Row-Format.jpg) - -Compact 和 Redundant 格式最大的不同就是记录格式的第一个部分；在 Compact 中，行记录的第一部分倒序存放了一行数据中列的长度（Length），而 Redundant 中存的是每一列的偏移量（Offset），从总体上上看，Compact 行记录格式相比 Redundant 格式能够减少 20% 的存储空间。 - -#### 行溢出数据 - -当 InnoDB 使用 Compact 或者 Redundant 格式存储极长的 VARCHAR 或者 BLOB 这类大对象时，我们并不会直接将所有的内容都存放在数据页节点中，而是将行数据中的前 768 个字节存储在数据页中，后面会通过偏移量指向溢出页。 - -![Row-Overflo](../../_images/mysql/Row-Overflow.jpg) - -但是当我们使用新的行记录格式 Compressed 或者 Dynamic 时都只会在行记录中保存 20 个字节的指针，实际的数据都会存放在溢出页面中。 - -![Row-Overflow-in-Barracuda](../../_images/mysql/Row-Overflow-in-Barracuda.jpg) - -当然在实际存储中，可能会对不同长度的 TEXT 和 BLOB 列进行优化，不过这就不是本文关注的重点了。 - -> 想要了解更多与 InnoDB 存储引擎中记录的数据格式的相关信息，可以阅读 [InnoDB Record Structure](https://dev.mysql.com/doc/internals/en/innodb-record-structure.html) - -#### 数据页结构 - -页是 InnoDB 存储引擎管理数据的最小磁盘单位，而 B-Tree 节点就是实际存放表中数据的页面，我们在这里将要介绍页是如何组织和存储记录的；首先，一个 InnoDB 页有以下七个部分： - -![InnoDB-B-Tree-Node](../../_images/mysql/InnoDB-B-Tree-Node.jpg) - -每一个页中包含了两对 header/trailer：内部的 Page Header/Page Directory 关心的是页的状态信息，而 Fil Header/Fil Trailer 关心的是记录页的头信息。 - -在页的头部和尾部之间就是用户记录和空闲空间了，每一个数据页中都包含 Infimum 和 Supremum 这两个虚拟的记录（可以理解为占位符），Infimum 记录是比该页中任何主键值都要小的值，Supremum 是该页中的最大值： - -![Infimum-Rows-Supremum](../../_images/mysql/Infimum-Rows-Supremum.jpg) - -User Records 就是整个页面中真正用于存放行记录的部分，而 Free Space 就是空余空间了，它是一个链表的数据结构，为了保证插入和删除的效率，整个页面并不会按照主键顺序对所有记录进行排序，它会自动从左侧向右寻找空白节点进行插入，行记录在物理存储上并不是按照顺序的，它们之间的顺序是由 `next_record` 这一指针控制的。 - -B+ 树在查找对应的记录时，并不会直接从树中找出对应的行记录，它只能获取记录所在的页，将整个页加载到内存中，再通过 Page Directory 中存储的稀疏索引和 `n_owned`、`next_record` 属性取出对应的记录，不过因为这一操作是在内存中进行的，所以通常会忽略这部分查找的耗时。 - -InnoDB 存储引擎中对数据的存储是一个非常复杂的话题，这一节中也只是对表、行记录以及页面的存储进行一定的分析和介绍，虽然作者相信这部分知识对于大部分开发者已经足够了，但是想要真正消化这部分内容还需要很多的努力和实践。 - - - -> [踏雪无痕-InnoDB存储引擎](https://www.cnblogs.com/chenpingzhao/p/9177324.html) \ No newline at end of file diff --git a/docs/data-store/MySQL/MySQL-Transaction.md b/docs/data-store/MySQL/MySQL-Transaction.md deleted file mode 100644 index e9f7a56e40..0000000000 --- a/docs/data-store/MySQL/MySQL-Transaction.md +++ /dev/null @@ -1,298 +0,0 @@ -# MySQL 事务 - -MySQL 事务主要用于处理操作量大，复杂度高的数据。比如说，在人员管理系统中，你删除一个人员，你即需要删除人员的基本资料，也要删除和该人员相关的信息，如信箱，文章等等，这样，这些数据库操作语句就构成一个事务！ - - - -### ACID — 事务基本要素 - -事务是由一组SQL语句组成的逻辑处理单元，具有4个属性，通常简称为事务的ACID属性。 - -- **A (Atomicity) 原子性**：整个事务中的所有操作，要么全部完成，要么全部不完成，不可能停滞在中间某个环节。事务在执行过程中发生错误，会被回滚（Rollback）到事务开始前的状态，就像这个事务从来没有执行过一样。 -- **C (Consistency) 一致性**：在事务开始之前和事务结束以后，数据库的完整性约束没有被破坏。 -- **I (Isolation)隔离性**：隔离状态执行事务，使它们好像是系统在给定时间内执行的唯一操作。如果有两个事务，运行在相同的时间内，执行 相同的功能，事务的隔离性将确保每一事务在系统中认为只有该事务在使用系统。这种属性有时称为串行化，为了防止事务操作间的混淆，必须串行化或序列化请 求，使得在同一时间仅有一个请求用于同一数据。 -- **D (Durability) 持久性**：在事务完成以后，该事务所对数据库所作的更改便持久的保存在数据库之中，并不会被回滚。 - - - -### 事务隔离级别 - -**并发事务处理带来的问题** - -- 更新丢失（Lost Update)： 事务A和事务B选择同一行，然后基于最初选定的值更新该行时，由于两个事务都不知道彼此的存在，就会发生丢失更新问题 -- 脏读(Dirty Reads)：事务A读取了事务B更新的数据，然后B回滚操作，那么A读取到的数据是脏数据 -- 不可重复读（Non-Repeatable Reads)：事务 A 多次读取同一数据，事务 B 在事务A多次读取的过程中，对数据作了更新并提交，导致事务A多次读取同一数据时，结果 不一致。 -- 幻读（Phantom Reads)：系统管理员A将数据库中所有学生的成绩从具体分数改为ABCDE等级，但是系统管理员B就在这个时候插入了一条具体分数的记录，当系统管理员A改结束后发现还有一条记录没有改过来，就好像发生了幻觉一样，这就叫幻读。 - - - -**幻读和不可重复读的区别：** - -- 不可重复读的重点是修改：在同一事务中，同样的条件，第一次读的数据和第二次读的数据不一样。（因为中间有其他事务提交了修改） -- 幻读的重点在于新增或者删除：在同一事务中，同样的条件,，第一次和第二次读出来的记录数不一样。（因为中间有其他事务提交了插入/删除） - - - -**并发事务处理带来的问题的解决办法：** - -- “更新丢失”通常是应该完全避免的。但防止更新丢失，并不能单靠数据库事务控制器来解决，需要应用程序对要更新的数据加必要的锁来解决，因此，防止更新丢失应该是应用的责任。 - -- “脏读” 、 “不可重复读”和“幻读” ，其实都是数据库读一致性问题，必须由数据库提供一定的事务隔离机制来解决： - - - 一种是加锁：在读取数据前，对其加锁，阻止其他事务对数据进行修改。 - - 另一种是数据多版本并发控制（MultiVersion Concurrency Control，简称 **MVCC** 或 MCC），也称为多版本数据库：不用加任何锁， 通过一定机制生成一个数据请求时间点的一致性数据快照 （Snapshot)， 并用这个快照来提供一定级别 （语句级或事务级） 的一致性读取。从用户的角度来看，好象是数据库可以提供同一数据的多个版本。 - - - -查看当前数据库的事务隔离级别： - -```mysql -show variables like 'tx_isolation' -``` - - - -数据库事务的隔离级别有4种，由低到高分别为Read uncommitted 、Read committed 、Repeatable read 、Serializable 。下面通过事例一一阐述在事务的并发操作中可能会出现脏读，不可重复读，幻读和事务隔离级别的联系。 - -数据库的事务隔离越严格，并发副作用越小，但付出的代价就越大，因为事务隔离实质上就是使事务在一定程度上“串行化”进行，这显然与“并发”是矛盾的。同时，不同的应用对读一致性和事务隔离程度的要求也是不同的，比如许多应用对“不可重复读”和“幻读”并不敏感，可能更关系数据并发访问的能力。 - -#### Read uncommitted - -读未提交，就是一个事务可以读取另一个未提交事务的数据。 - -事例：老板要给程序员发工资，程序员的工资是3.6万/月。但是发工资时老板不小心按错了数字，按成3.9万/月，该钱已经打到程序员的户口，但是事务还没有提交，就在这时，程序员去查看自己这个月的工资，发现比往常多了3千元，以为涨工资了非常高兴。但是老板及时发现了不对，马上回滚差点就提交了的事务，将数字改成3.6万再提交。 - -分析：实际程序员这个月的工资还是3.6万，但是程序员看到的是3.9万。他看到的是老板还没提交事务时的数据。这就是脏读。 - -那怎么解决脏读呢？Read committed！读提交，能解决脏读问题。 - -#### Read committed - -读提交，顾名思义，就是一个事务要等另一个事务提交后才能读取数据。 - -事例：程序员拿着信用卡去享受生活（卡里当然是只有3.6万），当他埋单时（程序员事务开启），收费系统事先检测到他的卡里有3.6万，就在这个时候！！程序员的妻子要把钱全部转出充当家用，并提交。当收费系统准备扣款时，再检测卡里的金额，发现已经没钱了（第二次检测金额当然要等待妻子转出金额事务提交完）。程序员就会很郁闷，明明卡里是有钱的… - -分析：这就是读提交，若有事务对数据进行更新（UPDATE）操作时，读操作事务要等待这个更新操作事务提交后才能读取数据，可以解决脏读问题。但在这个事例中，出现了一个事务范围内两个相同的查询却返回了不同数据，这就是不可重复读。 - -那怎么解决可能的不可重复读问题？Repeatable read ！ - -#### Repeatable read - -重复读，就是在开始读取数据（事务开启）时，不再允许修改操作。 MySQL的默认事务隔离级别 - -事例：程序员拿着信用卡去享受生活（卡里当然是只有3.6万），当他埋单时（事务开启，不允许其他事务的UPDATE修改操作），收费系统事先检测到他的卡里有3.6万。这个时候他的妻子不能转出金额了。接下来收费系统就可以扣款了。 - -分析：重复读可以解决不可重复读问题。写到这里，应该明白的一点就是，不可重复读对应的是修改，即UPDATE操作。但是可能还会有幻读问题。因为幻读问题对应的是插入INSERT操作，而不是UPDATE操作。 - -**什么时候会出现幻读？** - -事例：程序员某一天去消费，花了2千元，然后他的妻子去查看他今天的消费记录（全表扫描FTS，妻子事务开启），看到确实是花了2千元，就在这个时候，程序员花了1万买了一部电脑，即新增INSERT了一条消费记录，并提交。当妻子打印程序员的消费记录清单时（妻子事务提交），发现花了1.2万元，似乎出现了幻觉，这就是幻读。 - -那怎么解决幻读问题？Serializable！ - -#### Serializable 序列化 - -Serializable 是最高的事务隔离级别，在该级别下，事务串行化顺序执行，可以避免脏读、不可重复读与幻读。简单来说，Serializable会在读取的每一行数据上都加锁，所以可能导致大量的超时和锁争用问题。这种事务隔离级别效率低下，比较耗数据库性能，一般不使用。 - - - -| 事务隔离级别 | 读数据一致性 | 脏读 | 不可重复读 | 幻读 | -| ---------------------------- | ---------------------------------------- | ---- | ---------- | ---- | -| 读未提交（read-uncommitted） | 最低级被，只能保证不读取物理上损坏的数据 | 是 | 是 | 是 | -| 读已提交（read-committed） | 语句级 | 否 | 是 | 是 | -| 可重复读（repeatable-read） | 事务级 | 否 | 否 | 是 | -| 串行化（serializable） | 最高级别，事务级 | 否 | 否 | 否 | - - - -需要说明的是，事务隔离级别和数据访问的并发性是对立的，事务隔离级别越高并发性就越差。所以要根据具体的应用来确定合适的事务隔离级别，这个地方没有万能的原则。 - - - -### MVCC 多版本并发控制 - -MySQL的大多数事务型存储引擎实现都不是简单的行级锁。基于提升并发性考虑，一般都同时实现了多版本并发控制（MVCC），包括Oracle、PostgreSQL。只是实现机制各不相同。 - -可以认为MVCC是行级锁的一个变种，但它在很多情况下避免了加锁操作，因此开销更低。虽然实现机制有所不同，但大都实现了非阻塞的读操作，写操作也只是锁定必要的行。 - -MVCC的实现是通过保存数据在某个时间点的快照来实现的。也就是说不管需要执行多长时间，每个事物看到的数据都是一致的。 - -典型的MVCC实现方式，分为**乐观（optimistic）并发控制和悲观（pressimistic）并发控制**。下边通过InnoDB的简化版行为来说明MVCC是如何工作的。 - -InnoDB的MVCC，是通过在每行记录后面保存两个隐藏的列来实现。这两个列，一个保存了行的创建时间，一个保存行的过期时间（删除时间）。当然存储的并不是真实的时间，而是系统版本号（system version number）。每开始一个新的事务，系统版本号都会自动递增。事务开始时刻的系统版本号会作为事务的版本号，用来和查询到的每行记录的版本号进行比较。 - -**REPEATABLE READ（可重读）隔离级别下MVCC如何工作：** - -- SELECT - -InnoDB会根据以下两个条件检查每行记录： - -1. InnoDB只查找版本早于当前事务版本的数据行，这样可以确保事务读取的行,要么是在开始事务之前已经存在要么是事务自身插入或者修改过的 - -2. 行的删除版本号要么未定义，要么大于当前事务版本号，这样可以确保事务读取到的行在事务开始之前未被删除 - - 只有符合上述两个条件的才会被查询出来 - -- INSERT - - InnoDB为新插入的每一行保存当前系统版本号作为行版本号 - -- DELETE - - InnoDB为删除的每一行保存当前系统版本号作为行删除标识 - -- UPDATE - - InnoDB为插入的一行新纪录保存当前系统版本号作为行版本号，同时保存当前系统版本号到原来的行作为删除标识 - -保存这两个额外系统版本号，使大多数操作都不用加锁。使数据操作简单，性能很好，并且也能保证只会读取到符合要求的行。不足之处是每行记录都需要额外的存储空间，需要做更多的行检查工作和一些额外的维护工作。 - -MVCC只在COMMITTED READ（读提交）和REPEATABLE READ（可重复读）两种隔离级别下工作。 - - - -### 事务日志 - -事务日志可以帮助提高事务效率： - -- 使用事务日志，存储引擎在修改表的数据时只需要修改其内存拷贝，再把该修改行为记录到持久在硬盘上的事务日志中，而不用每次都将修改的数据本身持久到磁盘。 -- 事务日志采用的是追加的方式，因此写日志的操作是磁盘上一小块区域内的顺序I/O，而不像随机I/O需要在磁盘的多个地方移动磁头，所以采用事务日志的方式相对来说要快得多。 -- 事务日志持久以后，内存中被修改的数据在后台可以慢慢刷回到磁盘。 -- 如果数据的修改已经记录到事务日志并持久化，但数据本身没有写回到磁盘，此时系统崩溃，存储引擎在重启时能够自动恢复这一部分修改的数据。 - -目前来说，大多数存储引擎都是这样实现的，我们通常称之为**预写式日志**（Write-Ahead Logging），修改数据需要写两次磁盘。 - - - -### 事务的实现 - - 事务的实现是基于数据库的存储引擎。不同的存储引擎对事务的支持程度不一样。mysql中支持事务的存储引擎有innoDB和NDB。 - -事务的实现就是如何实现ACID特性。 - -innoDB是mysql默认的存储引擎，默认的隔离级别是RR（Repeatable Read），并且在RR的隔离级别下更进一步，通过多版本**并发控制**（MVCC，Multiversion Concurrency Control ）解决不可重复读问题，加上间隙锁（也就是并发控制）解决幻读问题。因此innoDB的RR隔离级别其实实现了串行化级别的效果，而且保留了比较好的并发性能。 - - - -?> 事务的隔离性是通过锁实现，而事务的原子性、一致性和持久性则是通过事务日志实现 。 - - - -**redo log（重做日志**） 实现持久化和原子性。 - -在innoDB的存储引擎中，事务日志通过重做(redo)日志和innoDB存储引擎的日志缓冲(InnoDB Log Buffer)实现。事务开启时，事务中的操作，都会先写入存储引擎的日志缓冲中，在事务提交之前，这些缓冲的日志都需要提前刷新到磁盘上持久化，这就是DBA们口中常说的“日志先行”(Write-Ahead Logging)。当事务提交之后，在Buffer Pool中映射的数据文件才会慢慢刷新到磁盘。此时如果数据库崩溃或者宕机，那么当系统重启进行恢复时，就可以根据redo log中记录的日志，把数据库恢复到崩溃前的一个状态。未完成的事务，可以继续提交，也可以选择回滚，这基于恢复的策略而定。 - -在系统启动的时候，就已经为redo log分配了一块连续的存储空间,以顺序追加的方式记录Redo Log,通过顺序IO来改善性能。所有的事务共享redo log的存储空间，它们的Redo Log按语句的执行顺序，依次交替的记录在一起。 - - - - **undo log** 实现一致性 - - undo log主要为事务的回滚服务。在事务执行的过程中，除了记录redo log，还会记录一定量的undo log。undo log记录了数据在每个操作前的状态，如果事务执行过程中需要回滚，就可以根据undo log进行回滚操作。单个事务的回滚，只会回滚当前事务做的操作，并不会影响到其他的事务做的操作。 - - - -二种日志均可以视为一种恢复操作，redo_log是恢复提交事务修改的页操作，而undo_log是回滚行记录到特定版本。二者记录的内容也不同，redo_log是物理日志，记录页的物理修改操作，而undo_log是逻辑日志，根据每行记录进行记录。 - - - -### Mysql中的事务使用 - - MySQL的服务层不管理事务，而是由下层的存储引擎实现。MySQL提供了两种事务型的存储引擎：InnoDB和NDB。 - -**MySQL支持本地事务的语句：** - -```mysql -START TRANSACTION | BEGIN [WORK] -COMMIT [WORK] [AND [NO] CHAIN] [[NO] RELEASE] -ROLLBACK [WORK] [AND [NO] CHAIN] [[NO] RELEASE] -SET AUTOCOMMIT = {0 | 1} -``` - -- START TRANSACTION 或 BEGIN 语句：开始一项新的事务。 -- COMMIT 和 ROLLBACK：用来提交或者回滚事务。 -- CHAIN 和 RELEASE 子句：分别用来定义在事务提交或者回滚之后的操作，CHAIN 会立即启动一个新事物，并且和刚才的事务具有相同的隔离级别，RELEASE 则会断开和客户端的连接。 -- SET AUTOCOMMIT 可以修改当前连接的提交方式， 如果设置了 SET AUTOCOMMIT=0，则设置之后的所有事务都需要通过明确的命令进行提交或者回滚 - -**事务使用注意点：** - -- 如果在锁表期间，用 start transaction 命令开始一个新事务，会造成一个隐含的 unlock - tables 被执行。 -- 在同一个事务中，最好不使用不同存储引擎的表，否则 ROLLBACK 时需要对非事 - 务类型的表进行特别的处理，因为 COMMIT、ROLLBACK 只能对事务类型的表进行提交和回滚。 -- 和 Oracle 的事务管理相同，所有的 DDL 语句是不能回滚的，并且部分的 DDL 语句会造成隐式的提交。 -- 在事务中可以通过定义 SAVEPOINT（例如：mysql> savepoint test; 定义 savepoint，名称为 test），指定回滚事务的一个部分，但是不能指定提交事务的一个部分。对于复杂的应用，可以定义多个不同的 SAVEPOINT，满足不同的条件时，回滚 - 不同的 SAVEPOINT。需要注意的是，如果定义了相同名字的 SAVEPOINT，则后面定义的SAVEPOINT 会覆盖之前的定义。对于不再需要使用的 SAVEPOINT，可以通过 RELEASE SAVEPOINT 命令删除 SAVEPOINT， 删除后的 SAVEPOINT， 不能再执行 ROLLBACK TO SAVEPOINT命令。 - -**自动提交（autocommit）：** -Mysql默认采用自动提交模式，可以通过设置autocommit变量来启用或禁用自动提交模式 - -- **隐式锁定** - - InnoDB在事务执行过程中，使用两阶段锁协议： - - 随时都可以执行锁定，InnoDB会根据隔离级别在需要的时候自动加锁； - - 锁只有在执行commit或者rollback的时候才会释放，并且所有的锁都是在**同一时刻**被释放。 - -- **显式锁定** - - InnoDB也支持通过特定的语句进行显示锁定（存储引擎层）： - -```mysql -select ... lock in share mode //共享锁 -select ... for update //排他锁 -``` - -​ MySQL Server层的显示锁定： - -```mysql -lock table和unlock table -``` - - - -### MySQL对分布式事务的支持 - -[官方分布式事务文档](https://dev.mysql.com/doc/refman/5.7/en/xa.html ) - -分布式事务的实现方式有很多，既可以采用innoDB提供的原生的事务支持，也可以采用消息队列来实现分布式事务的最终一致性。这里我们主要聊一下innoDB对分布式事务的支持。 - -MySQL 从 5.0.3 开始支持分布式事务，**当前分布式事务只支持 InnoDB 存储引擎**。一个分布式事务会涉及多个行动，这些行动本身是事务性的。所有行动都必须一起成功完成，或者一起被回滚。 - -![img](../../_images/mysql/mysql-xa-transactions.png) - -如图，mysql的分布式事务模型。模型中分三块：应用程序（AP）、资源管理器（RM）、事务管理器（TM）: - -- 应用程序：定义了事务的边界，指定需要做哪些事务； -- 资源管理器：提供了访问事务的方法，通常一个数据库就是一个资源管理器； -- 事务管理器：协调参与了全局事务中的各个事务。 - -分布式事务采用两段式提交（two-phase commit）的方式： - -- 第一阶段所有的事务节点开始准备，告诉事务管理器ready。 -- 第二阶段事务管理器告诉每个节点是commit还是rollback。如果有一个节点失败，就需要全局的节点全部rollback，以此保障事务的原子性。 - -分布式事务（XA 事务）的 SQL 语法主要包括： - -```mysql -XA {START|BEGIN} xid [JOIN|RESUME] -``` - -虽然 MySQL 支持分布式事务，但是在测试过程中，还是发现存在一些问题： -如果分支事务在达到 prepare 状态时，数据库异常重新启动，服务器重新启动以后，可以继续对分支事务进行提交或者回滚得操作，但是提交的事务没有写 binlog，存在一定的隐患，可能导致使用 binlog 恢复丢失部分数据。如果存在复制的数据库，则有可能导致主从数据库的数据不一致。 - -如果分支事务在执行到 prepare 状态时，数据库异常，且不能再正常启动，需要使用备份和 binlog 来恢复数据，那么那些在 prepare 状态的分支事务因为并没有记录到 binlog，所以不能通过 binlog 进行恢复，在数据库恢复后，将丢失这部分的数据。 - -如果分支事务的客户端连接异常中止，那么数据库会自动回滚未完成的分支事务，如果此时分支事务已经执行到 prepare 状态， 那么这个分布式事务的其他分支可能已经成功提交，如果这个分支回滚，可能导致分布式事务的不完整，丢失部分分支事务的内容。 -总之， MySQL 的分布式事务还存在比较严重的缺陷， 在数据库或者应用异常的情况下，可能会导致分布式事务的不完整。如果应用对于数据的完整性要求不是很高，则可以考虑使用。如果应用对事务的完整性有比较高的要求，那么对于当前的版本，则不推荐使用分布式事务。 - - - -> [数据库事务与MySQL事务总结](https://zhuanlan.zhihu.com/p/29166694) - - - - - - \ No newline at end of file diff --git a/docs/data-store/MySQL/MySQL-select.md b/docs/data-store/MySQL/MySQL-select.md deleted file mode 100644 index 7831d41f21..0000000000 --- a/docs/data-store/MySQL/MySQL-select.md +++ /dev/null @@ -1,133 +0,0 @@ -## 常见通用的Join查询 - -### SQL执行顺序 - -- 手写 - - ```mysql - SELECT DISTINCT - FROM - JOIN ON - WHERE - GROUP BY - HAVING - ORDER BY - LIMIT - ``` - -- 机读 - - ```mysql - FROM - ON - JOIN - WHERE - GROUP BY - HAVING - SELECT - DISTINCT - ORDER BY - LIMIT - ``` - -- 总结 - - ![sql-parse](../../_images/mysql/sql-parse.png) - - - -### Join图 - -![sql-joins](../../_images/mysql/sql-joins.jpg) - -### demo - -#### 建表SQL - -```plsql -CREATE TABLE `tbl_dept` ( - `id` INT(11) NOT NULL AUTO_INCREMENT, - `deptName` VARCHAR(30) DEFAULT NULL, - `locAdd` VARCHAR(40) DEFAULT NULL, - PRIMARY KEY (`id`) -) ENGINE=INNODB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8; - -CREATE TABLE `tbl_emp` ( - `id` INT(11) NOT NULL AUTO_INCREMENT, - `name` VARCHAR(20) DEFAULT NULL, - `deptId` INT(11) DEFAULT NULL, - PRIMARY KEY (`id`), - KEY `fk_dept_id` (`deptId`) - #CONSTRAINT `fk_dept_id` FOREIGN KEY (`deptId`) REFERENCES `tbl_dept` (`id`) -) ENGINE=INNODB AUTO_INCREMENT=1 DEFAULT CHARSET=utf8; - - -INSERT INTO tbl_dept(deptName,locAdd) VALUES('RD',11); -INSERT INTO tbl_dept(deptName,locAdd) VALUES('HR',12); -INSERT INTO tbl_dept(deptName,locAdd) VALUES('MK',13); -INSERT INTO tbl_dept(deptName,locAdd) VALUES('MIS',14); -INSERT INTO tbl_dept(deptName,locAdd) VALUES('FD',15); -INSERT INTO tbl_emp(NAME,deptId) VALUES('z3',1); -INSERT INTO tbl_emp(NAME,deptId) VALUES('z4',1); -INSERT INTO tbl_emp(NAME,deptId) VALUES('z5',1); -INSERT INTO tbl_emp(NAME,deptId) VALUES('w5',2); -INSERT INTO tbl_emp(NAME,deptId) VALUES('w6',2); -INSERT INTO tbl_emp(NAME,deptId) VALUES('s7',3); -INSERT INTO tbl_emp(NAME,deptId) VALUES('s8',4); -INSERT INTO tbl_emp(NAME,deptId) VALUES('s9',51); - -``` - -#### 7种JOIN - -1. A、B两表共有 - - ```mysql - select * from tbl_emp a **inner join** tbl_dept b on a.deptId = b.id; - ``` - -2. A、B两表共有+A的独有 - - ```mysql - select * from tbl_emp a **left join** tbl_dept b on a.deptId = b.id; - ``` - -3. A、B两表共有+B的独有 - - ```mysql - select * from tbl_emp a **right join** tbl_dept b on a.deptId = b.id; - ``` - -4. A的独有 - - ```mysql - select * from tbl_emp a left join tbl_dept b on a.deptId = b.id where b.id is null; - ``` - -5. B的独有 - - ```mysql - select * from tbl_emp a right join tbl_dept b on a.deptId = b.id where a.deptId is null; - ``` - -6. AB全有 - - -**MySQL Full Join的实现 因为MySQL不支持FULL JOIN,替代方法:left join + union(可去除重复数据)+ right join** - - ```mysql - SELECT * FROM tbl_emp A LEFT JOIN tbl_dept B ON A.deptId = B.id - UNION - SELECT * FROM tbl_emp A RIGHT JOIN tbl_dept B ON A.deptId = B.id - ``` - -7. A的独有+B的独有 - - ```mysql - SELECT * FROM tbl_emp A LEFT JOIN tbl_dept B ON A.deptId = B.id WHERE B.`id` IS NULL - UNION - SELECT * FROM tbl_emp A RIGHT JOIN tbl_dept B ON A.deptId = B.id WHERE A.`deptId` IS NULL; - ``` - - - diff --git a/docs/data-store/MySQL/sidebar.md b/docs/data-store/MySQL/sidebar.md deleted file mode 100644 index 9b7daedc22..0000000000 --- a/docs/data-store/MySQL/sidebar.md +++ /dev/null @@ -1,41 +0,0 @@ -- **Java基础** -- **数据存储和缓存** -- [![MySQL](https://icongr.am/devicon/mysql-original.svg?&size=16)MySQL](data-store/MySQL/readMySQL.md) - - [MySQL 架构介绍](data-store/MySQL/MySQL-Framework.md) - - [MySQL 存储引擎](data-store/MySQL/MySQL-Storage-Engines.md) - - [MySQL 索引](data-store/MySQL/MySQL-index.md) - - [MySQL 事务](data-store/MySQL/MySQL-Transaction.md) - - [MySQL 优化](data-store/MySQL/MySQL-Optimization.md) - - [MySQL 锁机制](data-store/MySQL/MySQL-Lock.md) - - [MySQL 分表分库](data-store/MySQL/MySQL-Segmentation.md) - - [MySQL 主从复制](data-store/MySQL/MySQL-Master-Slave.md) -- [![img](../../_media/redis-original.svg?&size=16)Redis](data-store/Redis/2.readRedis.md) -- [![mongoDB](../../_media/mongodb-original.svg)mongoDB]( https://redis.io/ ) -- [![Elasticsearch](../../_media/elasticsearch.svg) Elasticsearch]( https://redis.io/ ) -- [![S3](../../_media/amazonwebservices-original.svg)S3]( https://aws.amazon.com/cn/s3/ ) -- FastDFS(OSS) -- **单体架构** -- [缕清各种Java Logging](logging/Java-Logging.md) -- [hello logback](logging/logback简单使用.md) -- **微服务架构** -- Spring Boot -- Spring Cloud -- **面向服务架构** -- [![message](../../_media/message.svg) 消息中间件](message-queue/readMQ.md) -- [![Nginx](../../_media/nginx-original.svg)Nginx](nginx/nginx.md) -- **工程化与工具** -- [![Maven](https://icongr.am/fontawesome/maxcdn.svg?&size=16)Maven](logging/logback简单使用.md) -- [![Git](../../_media/git-original.svg?&size=16) Git](logging/logback简单使用.md) -- Sonar -- **大数据** -- HDFS -- **性能优化** -- JVM优化 -- web调优 -- DB调优 -- **其他** -- [![Linux](../../_media/linux-original.svg)Linux](linux/linux.md) -- [![]( https://icongr.am/entypo/key.svg?&size=16)设计模式](design-pattern/readme.md) -- **Links** -- [![Github](../../_media/github-original.svg)Github](https://github.com/jhildenbiddle/docsify-tabs) -- [![Blog](https://icongr.am/simple/aboutme.svg?colored&size=16)My Blog](https://www.lazyegg.net) diff --git a/docs/data-store/Redis/2.readRedis.md b/docs/data-store/Redis/2.readRedis.md deleted file mode 100644 index 4830a78540..0000000000 --- a/docs/data-store/Redis/2.readRedis.md +++ /dev/null @@ -1,105 +0,0 @@ -![5b557a0f2856b.jpg](http://ww1.sinaimg.cn/large/9b9f09a9ly1g9ypjztws7j20pl0cdmyu.jpg) - -## Redis简介 - -Redis:**REmote DIctionary Server**(远程字典服务器)。 - -Redis 是一个全开源免费（BSD许可）的，内存中的数据结构存储系统，它可以用作**数据库、缓存和消息中间件**。一般作为一个高性能的(key/value)分布式内存数据库，基于**内存**运行并支持持久化的NoSQL数据库，是当前最热门的NoSql数据库之一,也被人们称为**数据结构服务器** - -
- -### Redis 介绍 - -​ redis是一个开源的、使用C语言编写的、支持网络交互的、可基于内存也可持久化的Key-Value数据库。 - -​ redis是一个key-value存储系统。和Memcached类似，它支持存储的value类型相对更多，包括**string(字符串)、list(链表)、set(集合)、zset(sorted set --有序集合)和hash（哈希类型）****。**这些数据类型都支持push/pop、add/remove及取交集并集和差集及更丰富的操作，而且这些操作都是原子性的。在此基础上，redis支持各种不同方式的排序。与memcached一样，为了保证效率，数据都是缓存在内存中。区别的是redis会周期性的把更新的数据写入磁盘或者把修改操作写入追加的记录文件，并且在此基础上实现了master-slave(主从)同步，所以Redis也可以被看成是一个数据结构服务器。 - -​ **Redis 是一个高性能的key-value数据库**。 redis的出现，很大程度补偿了memcached这类key/value存储的不足，在部 分场合可以对关系数据库起到很好的补充作用。它提供了Java，C/C++，C#，PHP，JavaScript，Perl，Object-C，Python，Ruby，Erlang等客户端，使用很方便。 - -​ Redis支持主从同步。数据可以从主服务器向任意数量的从服务器上同步，从服务器可以是关联其他从服务器的主服务器。这使得Redis可执行单层树复制。存盘可以有意无意的对数据进行写操作。由于完全实现了发布/订阅机制，使得从数据库在任何地方同步树时，可订阅一个频道并接收主服务器完整的消息发布记录。同步对读取操作的可扩展性和数据冗余很有帮助。 - -​ redis的官网地址，非常好记，是redis.io。（域名后缀io属于国家域名，是british Indian Ocean territory，即英属印度洋领地）目前，Vmware在资助着redis项目的开发和维护 - -​ Redis的所有数据都是保存在内存中，然后不定期的通过异步方式保存到磁盘上(这称为“半持久化模式”)；也可以把每一次数据变化都写入到一个append only file(aof)里面(这称为“全持久化模式”)。这就是redis提供的两种持久化的方式，RDB（Redis DataBase）和AOF（Append Only File）。 - -
- -### Redis 特点 - -Redis是一个开源，先进的key-value存储，并用于构建高性能，可扩展的Web应用程序的完美解决方案。 - -Redis从它的许多竞争继承来的三个主要特点： - -- Redis数据库完全在内存中，使用磁盘仅用于持久性。 - -- 相比许多键值数据存储，Redis拥有一套较为丰富的数据类型。 - -- Redis可以将数据复制到任意数量的从服务器。 - -
- -### Redis 优势 - -- 异常快速：Redis的速度非常快，每秒能执行约11万集合，每秒约81000+条记录。SET操作每秒钟 110000 次，GET操作每秒钟 81000 次，网站一般使用Redis作为**缓存服务器**。 -- 支持**丰富的数据类型**：Redis支持大多数开发人员已经知道像列表，集合，有序集合，散列数据类型。这使得它非常容易解决各种各样的问题，因为我们知道哪些问题是可以处理通过它的数据类型更好。 -- 操作都是**原子性**：所有Redis操作是原子的，这保证了如果两个客户端同时访问的Redis服务器将获得更新后的值。 -- MultiUtility工具：Redis是一个多功能实用工具，可以在很多如：缓存，消息传递队列中使用（Redis原生支持发布/订阅），在应用程序中，如：Web应用程序会话，网站页面点击数等任何短暂的数据； - -
- -### redis使用场景 - -- 取最新N个数据的操作 -- 排行榜应用,取TOP N 操作 -- 需要精确设定过期时间的应用 -- 定时器、计数器应用 -- Uniq操作,获取某段时间所有数据排重值 -- 实时系统,反垃圾系统 -- Pub/Sub构建实时消息系统 -- 构建队列系统 -- 缓存 - - - -**具体以某一论坛为例：** - -- 记录帖子的点赞数、评论数和点击数 (hash)。 -- 记录用户的帖子 ID 列表 (排序)，便于快速显示用户的帖子列表 (zset)。 -- 记录帖子的标题、摘要、作者和封面信息，用于列表页展示 (hash)。 -- 记录帖子的点赞用户 ID 列表，评论 ID 列表，用于显示和去重计数 (zset)。 -- 缓存近期热帖内容 (帖子内容空间占用比较大)，减少数据库压力 (hash)。 -- 记录帖子的相关文章 ID，根据内容推荐相关帖子 (list)。 -- 如果帖子 ID 是整数自增的，可以使用 Redis 来分配帖子 ID(计数器)。 -- 收藏集和帖子之间的关系 (zset)。 -- 记录热榜帖子 ID 列表，总热榜和分类热榜 (zset)。 -- 缓存用户行为历史，进行恶意行为过滤 (zset,hash)。 - -
- -**安装** - -``` -$ wget http://download.redis.io/releases/redis-5.0.6.tar.gz -$ tar xzf redis-5.0.6.tar.gz -$ cd redis-5.0.6 -$ make -``` - -新版本的编译文件在src中（之前在bin目录），启动server - -``` -$ src/redis-server -``` - -启动客户端 - -``` -$ src/redis-cli -redis> set foo bar -OK -redis> get foo -"bar" -``` - - - diff --git a/docs/data-store/Redis/3.Redis-Datatype.md b/docs/data-store/Redis/3.Redis-Datatype.md deleted file mode 100644 index d4d153d417..0000000000 --- a/docs/data-store/Redis/3.Redis-Datatype.md +++ /dev/null @@ -1,313 +0,0 @@ -> Redis 有哪几种数据类型，各自底层是怎么实现的？ -> -> 你们项目中哪个地方用到了什么类型，怎么使用的？ -> -> 和 Java 中相似的数据结构的对比，说一下？ - -### Redis的五种基本数据类型 - -#### String（字符串） - -String 是redis最基本的类型，你可以理解成与Memcached一模一样的类型，一个key对应一个value。 - -string类型是二进制安全的。意思是redis的string可以包含任何数据。比如jpg图片或者序列化的对象 。 - -Redis 的字符串是动态字符串，是可以修改的字符串，内部结构实现上类似于 Java 的 ArrayList，采用预分配冗余空间的方式来减少内存的频繁分配，如图中所示，内部为当前字符串实际分配的空间 capacity 一般要高于实际字符串长度 len。当字符串长度小于 1M 时，扩容都是加倍现有的空间，如果超过 1M，扩容时一次只会多扩 1M 的空间。需要注意的是字符串最大长度为 512M。 - -![redis-string.jpg](http://ww1.sinaimg.cn/large/9b9f09a9ly1g9ypoobef5j20fw04pq2p.jpg) - -#### Hash（字典） - -Redis hash 是一个键值对集合。KV模式不变，但V是一个键值对。 - -Redis hash是一个string类型的field和value的映射表，hash特别适合用于存储对象。 - -Redis 的字典相当于 Java 语言里面的 HashMap，它是无序字典， 内部实现结构上同 Java 的 HashMap 也是一致的，同样的数组 + 链表二维结构。第一维 hash 的数组位置碰撞时，就会将碰撞的元素使用链表串接起来。 - -![redis-hash.jpg](http://ww1.sinaimg.cn/large/9b9f09a9ly1g9ypp2rs05j20g307pa9u.jpg) - -#### List（列表） - -Redis 列表是简单的字符串列表，按照插入顺序排序。你可以添加一个元素导列表的头部（左边）或者尾部（右边）。 - -它的底层实际是个链表， - -**Redis 的列表相当于 Java 语言里面的 LinkedList，注意它是链表而不是数组。这意味着 list 的插入和删除操作非常快，时间复杂度为 O(1)，但是索引定位很慢，时间复杂度为 O(n)** - -Redis 的列表结构常用来做异步队列使用。将需要延后处理的任务结构体序列化成字符串塞进 Redis 的列表，另一个线程从这个列表中轮询数据进行处理 - -##### 右边进左边出：队列 - -```shell -> rpush books python java golang -(integer) 3 -\> llen books - (integer) 3 -\> lpop books - "python" -\> lpop books - "java" -\> lpop books -"golang" -\> lpop books -(nil) -``` - -##### 右边进右边出：栈 - -```shell -> rpush books python java golang - (integer) 3 - \> rpop books -"golang" -\> rpop books -"java" -\> rpop books -"python" - \> rpop books - (nil) -``` - - - -#### Set（集合） - -Redis的Set是string类型的无序集合。它是通过HashTable实现的， 相当于 Java 语言里面的 HashSet，它内部的键值对是无序的唯一的。它的内部实现相当于一个特殊的字典，字典中所有的 value 都是一个值`NULL`。 - -#### zset(sorted set：有序集合) - -Redis zset 和 set 一样也是string类型元素的集合,且不允许重复的成员。不同的是每个元素都会关联一个double类型的分数。 - -redis正是通过分数来为集合中的成员进行从小到大的排序。zset的成员是唯一的,但分数(score)却可以重复。 - - 它类似于 Java 的 SortedSet 和 HashMap 的结合体，一方面它是一个 set，保证了内部 value 的唯一性，另一方面它可以给每个 value 赋予一个 score，代表这个 value 的排序权重。它的内部实现用的是一种叫做「跳跃列表」的数据结构。 - - - -**redis常见数据类型查阅：** - -- [Redis命令参考](http://redisdoc.com/) -- [Redis命令中心](http://www.redis.cn/commands.html) - - - -#### **Key**（键）常用命令 - -| 命令 | 用法 | 描述 | 示例 | -| ------------ | ---------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | -| **DEL** | DEL key [key ...] | 删除给定的一个或多个 key。 不存在的 key 会被忽略 | | -| DUMP | DUMP key | 序列化给定 key ，并返回被序列化的值，使用 RESTORE 命令可以将这个值反序列化为 Redis 键 | | -| EXISTS | EXISTS key | 检查给定 key 是否存在 | | -| **EXPIRE** | EXPIRE key seconds | 为给定 key 设置生存时间，当 key 过期时(生存时间为 0 )，它会被自动删除 | | -| **PERSIST** | PERSIST key | 移除 key 的过期时间，key 将持久保持。 | | -| **EXPIREAT** | EXPIREAT key timestamp | EXPIREAT 的作用和 EXPIRE 类似，都用于为 key 设置生存时间。 不同在于 EXPIREAT 命令接受的时间参数是 UNIX 时间戳(unix timestamp) | EXPIREAT cache 1355292000 # 这个 key 将在 2012.12.12 过期 | -| **KEYS** | KEYS pattern | 查找所有符合给定模式 pattern 的 key | KEYS * # 匹配数据库内所有 key | -| MOVE | MOVE key db | 将当前数据库的 key 移动到给定的数据库 db 当中如果当前数据库(源数据库)和给定数据库(目标数据库)有相同名字的给定 key ，或者 key 不存在于当前数据库，那么 MOVE 没有任何效果。 因此，也可以利用这一特性，将 MOVE 当作锁(locking)原语(primitive) | MOVE song 1 # 将 song 移动到数据库 1 | -| **TTL** | TTL key | 以秒为单位，返回给定 key 的剩余生存时间(TTL, time to live)当 key 不存在时，返回 -2 。当 key 存在但没有设置剩余生存时间时，返回 -1 。否则，以秒为单位，返回 key 的剩余生存时间 | | -| PTTL | PTTL key | 以毫秒为单位返回 key 的剩余的过期时间。 | | -| **TYPE** | TYPE key | 返回 key 所储存的值的类型 | | -| RENAME | RENAME key newkey | 将 key 改名为 newkey 。当 key 和 newkey 相同，或者 key 不存在时，返回一个错误。当 newkey 已经存在时， RENAME 命令将覆盖旧值 | | - -#### **String** （字符串）常用命令 - -| 命令 | 用法 | 描述 | 示例 | -| ----------- | ----------------------------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | -| **SET** | SET key value [EX seconds] [PX milliseconds] [NX\|XX] | 将字符串值 value 关联到 key 。如果 key 已经持有其他值， SET 就覆写旧值，无视类型 | SET key "value" | -| **MSET** | MSET key value [key value ...] | 同时设置一个或多个 key-value 对。如果某个给定 key 已经存在，那么 MSET 会用新值覆盖原来的旧值，如果这不是你所希望的效果，请考虑使用 MSETNX 命令：它只会在所有给定 key 都不存在的情况下进行设置操作 | MSET date "2012.3.30" time "11:00 a.m." weather "sunny" | -| **SETNX** | SETNX key value | 将 key 的值设为 value ，当且仅当 key 不存在。若给定的 key 已经存在，则 SETNX 不做任何动作 SETNX 是『SET if Not eXists』(如果不存在，则 SET)的简写 | | -| MSETNX | MSETNX key value [key value ...] | 同时设置一个或多个 key-value 对，当且仅当所有给定 key 都不存在。即使只有一个给定 key 已存在， MSETNX 也会拒绝执行所有给定 key 的设置操作 | | -| SETRANGE | SETRANGE key offset value | 用 value 参数覆写(overwrite)给定 key 所储存的字符串值，从偏移量 offset 开始。不存在的 key 当作空白字符串处理 | | -| SETBIT | SETBIT key offset value | 对 key 所储存的字符串值，设置或清除指定偏移量上的位(bit) | GETBIT bit 100 # bit 默认被初始化为 0 | -| SETEX | SETEX key seconds value | 将值 value 关联到 key ，并将 key 的生存时间设为 seconds (以秒为单位)。如果 key 已经存在， SETEX 命令将覆写旧值。 | | -| PSETEX | PSETEX key milliseconds value | 这个命令和 SETEX 命令相似，但它以毫秒为单位设置 key 的生存时间，而不是像 SETEX 命令那样，以秒为单位 | | -| STRLEN | STRLEN key | 返回 key 所储存的字符串值的长度。当 key 储存的不是字符串值时，返回一个错误 | | -| **GET** | GET key | 返回 key 所关联的字符串值。如果 key 不存在那么返回特殊值 nil | | -| **MGET** | MGET key [key ...] | 返回所有(一个或多个)给定 key 的值。如果给定的 key 里面，有某个 key 不存在，那么这个 key 返回特殊值 nil 。因此，该命令永不失败 | | -| GETRANGE | GETRANGE key start end | 返回 key 中字符串值的子字符串，字符串的截取范围由 start 和 end 两个偏移量决定(包括 start 和 end 在内)。负数偏移量表示从字符串最后开始计数， -1 表示最后一个字符， -2 表示倒数第二个，以此类推。 | GETRANGE greeting 0 4 | -| GETSET | GETSET key value | 将给定 key 的值设为 value ，并返回 key 的旧值(old value)。当 key 存在但不是字符串类型时，返回一个错误。 | | -| GETBIT | GETBIT key offset | 对 key 所储存的字符串值，获取指定偏移量上的位(bit)。当 offset 比字符串值的长度大，或者 key 不存在时，返回 0 | | -| **APPEND** | APPEND key value | 如果 key 已经存在并且是一个字符串， APPEND 命令将 value 追加到 key 原来的值的末尾。如果 key 不存在， APPEND 就简单地将给定 key 设为 value ，就像执行 SET key value 一样 | | -| **DECR** | DECR key | 将 key 中储存的数字值减一。如果 key 不存在，那么 key 的值会先被初始化为 0 ，然后再执行 DECR 操作 | redis> SET failure_times 10OK redis> DECR failure_times(integer) 9 | -| **DECRBY** | DECRBY key decrement | 将 key 所储存的值减去减量 decrement | | -| **INCR** | INCR key | 将 key 中储存的数字值增一。如果 key 不存在，那么 key 的值会先被初始化为 0 ，然后再执行 INCR 操作 | | -| **INCRBY** | INCRBY key increment | 将 key 所储存的值加上增量 increment | | -| INCRBYFLOAT | INCRBYFLOAT key increment | 为 key 中所储存的值加上浮点数增量 increment | INCRBYFLOAT mykey 0.1 | -| BITCOUNT | BITCOUNT key [start] [end] | 计算给定字符串中，被设置为 1 的比特位的数量 | | -| BITOP | BITOP operation destkey key [key ...] | 对一个或多个保存二进制位的字符串 key 进行位元操作，并将结果保存到 destkey 上。 | | - -##### ☆☆位图： - -​ String命令中包含了一种特殊的操作，直接操作bit，某些特殊场景下，会节省存储空间。可以在存取bool型数据的场景使用，比如存取用户男女比例，用户某一段日期签到记录， - -​ 在我们平时开发过程中，会有一些 bool 型数据需要存取，比如用户一年的签到记录，签了是 1，没签是 0，要记录 365 天。如果使用普通的 key/value，每个用户要记录 365 个，当用户上亿的时候，需要的存储空间是惊人的。 - -为了解决这个问题，Redis 提供了位图数据结构，这样每天的签到记录只占据一个位，365 天就是 365 个位，46 个字节 (一个稍长一点的字符串) 就可以完全容纳下，这就大大节约了存储空间。 - - - -![img](https://user-gold-cdn.xitu.io/2018/7/2/1645926f4520d0ce?imageslim) - - - -位图不是特殊的数据结构，它的内容其实就是普通的字符串，也就是 byte 数组。我们可以使用普通的 get/set 直接获取和设置整个位图的内容，也可以使用位图操作 getbit/setbit 等将 byte 数组看成「位数组」来处理 - - Redis 的位数组是自动扩展，如果设置了某个偏移位置超出了现有的内容范围，就会自动将位数组进行零扩充。 - -![img](https://user-gold-cdn.xitu.io/2018/7/2/16459860644097de?imageslim) - -接下来我们使用 redis-cli 设置第一个字符，也就是位数组的前 8 位，我们只需要设置值为 1 的位，如上图所示，h 字符只有 1/2/4 位需要设置，e 字符只有 9/10/13/15 位需要设置。值得注意的是位数组的顺序和字符的位顺序是相反的。 - -``` -127.0.0.1:6379> setbit s 1 1 -(integer) 0 -127.0.0.1:6379> setbit s 2 1 -(integer) 0 -127.0.0.1:6379> setbit s 4 1 -(integer) 0 -127.0.0.1:6379> setbit s 9 1 -(integer) 0 -127.0.0.1:6379> setbit s 10 1 -(integer) 0 -127.0.0.1:6379> setbit s 13 1 -(integer) 0 -127.0.0.1:6379> setbit s 15 1 -(integer) 0 -127.0.0.1:6379> get s -"he" -``` - -上面这个例子可以理解为「零存整取」，同样我们还也可以「零存零取」，「整存零取」。「零存」就是使用 setbit 对位值进行逐个设置，「整存」就是使用字符串一次性填充所有位数组，覆盖掉旧值。 - -bitcount和bitop, bitpos,bitfield 都是操作位图的指令。 - - - -#### List（列表）常用命令 - -| 命令 | 用法 | 描述 | 示例 | -| ---------- | ------------------------------------- | ------------------------------------------------------------ | --------------------------------------------- | -| **LPUSH** | LPUSH key value [value ...] | 将一个或多个值 value 插入到列表 key 的表头如果有多个 value 值，那么各个 value 值按从左到右的顺序依次插入到表头 | 正着进反着出 | -| LPUSHX | LPUSHX key value | 将值 value 插入到列表 key 的表头，当且仅当 key 存在并且是一个列表。和 LPUSH 命令相反，当 key 不存在时， LPUSHX 命令什么也不做 | | -| **RPUSH** | RPUSH key value [value ...] | 将一个或多个值 value 插入到列表 key 的表尾(最右边) | 怎么进怎么出 | -| RPUSHX | RPUSHX key value | 将值 value 插入到列表 key 的表尾，当且仅当 key 存在并且是一个列表。和 RPUSH 命令相反，当 key 不存在时， RPUSHX 命令什么也不做。 | | -| **LPOP** | LPOP key | 移除并返回列表 key 的头元素。 | | -| **BLPOP** | BLPOP key [key ...] timeout | 移出并获取列表的第一个元素， 如果列表没有元素会阻塞列表直到等待超时或发现可弹出元素为止 | | -| **RPOP** | RPOP key | 移除并返回列表 key 的尾元素。 | | -| **BRPOP** | BRPOP key [key ...] timeout | 移出并获取列表的最后一个元素， 如果列表没有元素会阻塞列表直到等待超时或发现可弹出元素为止。 | | -| BRPOPLPUSH | BRPOPLPUSH source destination timeout | 从列表中弹出一个值，将弹出的元素插入到另外一个列表中并返回它； 如果列表没有元素会阻塞列表直到等待超时或发现可弹出元素为止。 | | -| RPOPLPUSH | RPOPLPUSH source destinationb | 命令 RPOPLPUSH 在一个原子时间内，执行以下两个动作：将列表 source 中的最后一个元素(尾元素)弹出，并返回给客户端。将 source 弹出的元素插入到列表 destination ，作为 destination 列表的的头元素 | RPOPLPUSH list01 list02 | -| **LSET** | LSET key index value | 将列表 key 下标为 index 的元素的值设置为 value | | -| **LLEN** | LLEN key | 返回列表 key 的长度。如果 key 不存在，则 key 被解释为一个空列表，返回 0 .如果 key 不是列表类型，返回一个错误 | | -| **LINDEX** | LINDEX key index | 返回列表 key 中，下标为 index 的元素。下标(index)参数 start 和 stop 都以 0 为底，也就是说，以 0 表示列表的第一个元素，以 1 表示列表的第二个元素，以此类推。相当于 Java 链表的`get(int index)`方法，它需要对链表进行遍历，性能随着参数`index`增大而变差。 | | -| **LRANGE** | LRANGE key start stop | 返回列表 key 中指定区间内的元素，区间以偏移量 start 和 stop 指定 | | -| LREM | LREM key count value | 根据参数 count 的值，移除列表中与参数 value 相等的元素 | | -| LTRIM | LTRIM key start stop | 对一个列表进行修剪(trim)，就是说，让列表只保留指定区间内的元素，不在指定区间之内的元素都将被删除 | | -| LINSERT | LINSERT key BEFORE\|AFTER pivot value | 将值 value 插入到列表 key 当中，位于值 pivot 之前或之后。当 pivot 不存在于列表 key 时，不执行任何操作。当 key 不存在时， key 被视为空列表，不执行任何操作。如果 key 不是列表类型，返回一个错误。 | LINSERT list01 before c++ c#(在c++之前加上C#) | - -#### **Hash**（哈希表）常用命令 - -| 命令 | 用法 | 描述 | 示例 | -| ------------ | ---------------------------------------------- | ------------------------------------------------------------ | ---- | -| **HSET** | HSET key field value | 将哈希表 key 中的域 field 的值设为 value 。如果 key 不存在，一个新的哈希表被创建并进行 HSET 操作。如果域 field 已经存在于哈希表中，旧值将被覆盖。 | | -| **HMSET** | HMSET key field value [field value ...] | 同时将多个 field-value (域-值)对设置到哈希表 key 中。此命令会覆盖哈希表中已存在的域。 | | -| **HSETNX** | HSETNX key field value | 将哈希表 key 中的域 field 的值设置为 value ，当且仅当域 field 不存在。若域 field 已经存在，该操作无效 | | -| **HGET** | HGET key field | 返回哈希表 key 中给定域 field 的值 | | -| **HMGET** | HMGET key field [field ...] | 返回哈希表 key 中，一个或多个给定域的值。 | | -| **HGETALL** | HGETALL key | 返回哈希表 key 中，所有的域和值。在返回值里，紧跟每个域名(field name)之后是域的值(value)，所以返回值的长度是哈希表大小的两倍 | | -| HDEL | HDEL key field [field ...] | 删除哈希表 key 中的一个或多个指定域，不存在的域将被忽略 | | -| HEXISTS | HEXISTS key field | 查看哈希表 key 中，给定域 field 是否存在 | | -| HLEN | HLEN key | 返回哈希表 key 中域的数量 | | -| **HKEYS** | HKEYS key | 返回哈希表 key 中的所有域 | | -| **HVALS** | HVALS key | 返回哈希表 key 中所有域的值 | | -| HSTRLEN | HSTRLEN key field | 返回哈希表 key 中，与给定域 field 相关联的值的字符串长度（string length）。如果给定的键或者域不存在，那么命令返回 0 | | -| HINCRBY | HINCRBY key field increment | 为哈希表 key 中的域 field 的值加上增量 increment | | -| HINCRBYFLOAT | HINCRBYFLOAT key field increment | 为哈希表 key 中的域 field 加上浮点数增量 increment | | -| HSCAN | HSCAN key cursor [MATCH pattern] [COUNT count] | 迭代哈希表中的键值对。 | | - -#### **Set**（集合）常用命令 - -| 命令 | 用法 | 描述 | 示例 | -| ------------- | ---------------------------------------------- | ------------------------------------------------------------ | ---- | -| **SADD** | SADD key member [member ...] | 将一个或多个 member 元素加入到集合 key 当中，已经存在于集合的 member 元素将被忽略。假如 key 不存在，则创建一个只包含 member 元素作成员的集合。当 key 不是集合类型时，返回一个错误 | | -| **SCARD** | SCARD key | 返回集合 key 的基数(集合中元素的数量)。 | | -| **SDIFF** | SDIFF key [key ...] | 返回一个集合的全部成员，该集合是所有给定集合之间的差集。不存在的 key 被视为空集。 | 差集 | -| SDIFFSTORE | SDIFFSTORE destination key [key ...] | 这个命令的作用和 SDIFF 类似，但它将结果保存到 destination 集合，而不是简单地返回结果集。如果 destination 集合已经存在，则将其覆盖。destination 可以是 key 本身。 | | -| **SINTER** | SINTER key [key ...] | 返回一个集合的全部成员，该集合是所有给定集合的交集。不存在的 key 被视为空集。当给定集合当中有一个空集时，结果也为空集(根据集合运算定律) | 交集 | -| SINTERSTORE | SINTERSTORE destination key [key ...] | 这个命令类似于 SINTER 命令，但它将结果保存到 destination 集合，而不是简单地返回结果集。如果 destination 集合已经存在，则将其覆盖。destination 可以是 key 本身 | | -| **SUNION** | SUNION key [key ...] | 返回一个集合的全部成员，该集合是所有给定集合的并集。不存在的 key 被视为空集 | 并集 | -| SUNIONSTORE | SUNIONSTORE destination key [key ...] | 这个命令类似于 SUNION 命令，但它将结果保存到 destination 集合，而不是简单地返回结果集。如果 destination 已经存在，则将其覆盖。destination 可以是 key 本身 | | -| **SMEMBERS** | SMEMBERS key | 返回集合 key 中的所有成员。不存在的 key 被视为空集合 | | -| SRANDMEMBER | SRANDMEMBER key [count] | 如果命令执行时，只提供了 key 参数，那么返回集合中的一个随机元素 | | -| **SISMEMBER** | SISMEMBER key member | 判断 member 元素是否集合 key 的成员 | | -| SMOVE | SMOVE source destination member | 将 member 元素从 source 集合移动到 destination 集合。 | | -| SPOP | SPOP key | 移除并返回集合中的一个随机元素。如果只想获取一个随机元素，但不想该元素从集合中被移除的话，可以使用 SRANDMEMBER 命令。 | | -| **SREM** | SREM key member [member ...] | 移除集合 key 中的一个或多个 member 元素，不存在的 member 元素会被忽略。当 key 不是集合类型，返回一个错误 | | -| SSCAN | SSCAN key cursor [MATCH pattern] [COUNT count] | 迭代集合中的元素 | | - -#### **SortedSet**（有序集合）常用命令 - -| 命令 | 用法 | 描述 | 示例 | -| ---------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | ---- | -| **ZADD** | ZADD key score1 member1 [score2 member2] | 向有序集合添加一个或多个成员，或者更新已存在成员的分数 | | -| **ZCARD** | ZCARD key | 返回有序集 key 的基数。 | | -| **ZCOUNT** | ZCOUNT key min max | 返回有序集 key 中， score 值在 min 和 max 之间(默认包括 score 值等于 min 或 max )的成员的数量。关于参数 min 和 max 的详细使用方法，请参考 ZRANGEBYSCORE 命令。 | | -| **ZRANGE** | ZRANGE key start stop [WITHSCORES] | 返回有序集 key 中，指定区间内的成员。其中成员的位置按 score 值递增(从小到大)来排序 | | -| **ZREVRANGE** | ZREVRANGE key start stop [WITHSCORES] | 返回有序集 key 中，指定区间内的成员。其中成员的位置按 score 值递减(从大到小)来排列。具有相同 score 值的成员按字典序的逆序(reverse lexicographical order)排列。除了成员按 score 值递减的次序排列这一点外， ZREVRANGE 命令的其他方面和 ZRANGE 命令一样。 | | -| ZREVRANGEBYSCORE | ZREVRANGEBYSCORE key max min [WITHSCORES] [LIMIT offset count] | 返回有序集 key 中， score 值介于 max 和 min 之间(默认包括等于 max 或 min )的所有的成员。有序集成员按 score 值递减(从大到小)的次序排列。 | | -| ZREVRANK | ZREVRANK key member | 返回有序集 key 中成员 member 的排名。其中有序集成员按 score 值递减(从大到小)排序。排名以 0 为底，也就是说， score 值最大的成员排名为 0 。使用 ZRANK 命令可以获得成员按 score 值递增(从小到大)排列的排名。 | | -| ZSCORE | ZSCORE key member | 返回有序集 key 中，成员 member 的 score 值。如果 member 元素不是有序集 key 的成员，或 key 不存在，返回 nil 。 | | -| ZRANGEBYSCORE | ZRANGEBYSCORE key min max [WITHSCORES] [LIMIT offset count] | 返回有序集 key 中，所有 score 值介于 min 和 max 之间(包括等于 min 或 max )的成员。有序集成员按 score 值递增(从小到大)次序排列。 | | -| ZRANK | ZRANK key member | 返回有序集 key 中成员 member 的排名。其中有序集成员按 score 值递增(从小到大)顺序排列。 | | -| **ZINCRBY** | ZINCRBY key increment member | 为有序集 key 的成员 member 的 score 值加上增量 increment | | -| ZREM | ZREM key member [member ...] | 移除有序集 key 中的一个或多个成员，不存在的成员将被忽略。当 key 存在但不是有序集类型时，返回一个错误。 | | -| ZREMRANGEBYRANK | ZREMRANGEBYRANK key start stop | 移除有序集 key 中，指定排名(rank)区间内的所有成员 | | -| ZREMRANGEBYSCORE | ZREMRANGEBYSCORE key min max | 移除有序集 key 中，所有 score 值介于 min 和 max 之间(包括等于 min 或 max )的成员。 | | -| ZUNIONSTORE | ZUNIONSTORE destination numkeys key [key ...] [WEIGHTS weight [weight ...]] [AGGREGATE SUM\|MIN\|MAX] | 计算给定的一个或多个有序集的并集，其中给定 key 的数量必须以 numkeys 参数指定，并将该并集(结果集)储存到 destination 。默认情况下，结果集中某个成员的 score 值是所有给定集下该成员 score 值之 和 。 | | -| ZINTERSTORE | ZINTERSTORE destination numkeys key [key ...] [WEIGHTS weight [weight ...]] [AGGREGATE SUM\|MIN\|MAX] | 计算给定的一个或多个有序集的交集，其中给定 key 的数量必须以 numkeys 参数指定，并将该交集(结果集)储存到 destination 。默认情况下，结果集中某个成员的 score 值是所有给定集下该成员 score 值之和. | | -| ZSCAN | ZSCAN key cursor [MATCH pattern] [COUNT count] | 迭代有序集合中的元素（包括元素成员和元素分值） | | -| ZRANGEBYLEX | ZRANGEBYLEX key min max [LIMIT offset count] | 当有序集合的所有成员都具有相同的分值时，有序集合的元素会根据成员的字典序（lexicographical ordering）来进行排序，而这个命令则可以返回给定的有序集合键 key 中，值介于 min 和 max 之间的成员。 | | -| ZLEXCOUNT | ZLEXCOUNT key min max | 对于一个所有成员的分值都相同的有序集合键 key 来说，这个命令会返回该集合中，成员介于 min 和 max 范围内的元素数量。这个命令的 min 参数和 max 参数的意义和 ZRANGEBYLEX 命令的 min 参数和 max 参数的意义一样 | | -| ZREMRANGEBYLEX | ZREMRANGEBYLEX key min max | 对于一个所有成员的分值都相同的有序集合键 key 来说，这个命令会移除该集合中，成员介于 min 和 max 范围内的所有元素。这个命令的 min 参数和 max 参数的意义和 ZRANGEBYLEX 命令的 min 参数和 max 参数的意义一样 | | - - - -### HyperLogLog - -Redis 在 2.8.9 版本添加了 HyperLogLog 结构。 - -场景：可以用来统计站点的UV... - -Redis HyperLogLog 是用来做基数统计的算法，HyperLogLog 的优点是，在输入元素的数量或者体积非常非常大时，计算基数所需的空间总是固定 的、并且是很小的。但是会有误差。 - -| 命令 | 用法 | 描述 | -| ------- | ------------------------------------------ | ----------------------------------------- | -| pfadd | [PFADD key element [element ...] | 添加指定元素到 HyperLogLog 中 | -| pfcount | [PFCOUNT key [key ...] | 返回给定 HyperLogLog 的基数估算值。 | -| pfmerge | [PFMERGE destkey sourcekey [sourcekey ...] | 将多个 HyperLogLog 合并为一个 HyperLogLog | - -```java -public class JedisTest { - public static void main(String[] args) { - Jedis jedis = new Jedis(); - for (int i = 0; i < 100000; i++) { - jedis.pfadd("codehole", "user" + i); - } - long total = jedis.pfcount("codehole"); - System.out.printf("%d %d\n", 100000, total); - jedis.close(); - } -} -``` - -[HyperLogLog图解](http://content.research.neustar.biz/blog/hll.html ) - - - -### redis的所有指令 - -[各指令介绍](https://github.com/antirez/redis-doc/tree/617a8109020fa299efb543277c1fea3915652509/commands) - - - - - diff --git a/docs/data-store/Redis/5.Redis-Persistence.md b/docs/data-store/Redis/5.Redis-Persistence.md deleted file mode 100644 index b1289f18c7..0000000000 --- a/docs/data-store/Redis/5.Redis-Persistence.md +++ /dev/null @@ -1,247 +0,0 @@ -> Redis 两种备份方式的区别，项目中用的哪种，为什么？ - -# Redis的持久化机制 - -Redis 的数据全部在内存里，如果突然宕机，数据就会全部丢失，因此必须有一种机制来保证 Redis 的数据不会因为故障而丢失，这种机制就是 Redis 的持久化机制。 - -**Redis有两种持久化的方式：快照（`RDB`文件）和追加式文件（`AOF`文件)** - - - -## RDB（Redis DataBase） - -![img](https://imgkr.cn-bj.ufileos.com/c43c7260-ddf2-4b6f-9098-bb7890487575.gif) - -#### 是什么 - -**在指定的时间间隔内将内存中的数据集快照写入磁盘**，也就是行话讲的Snapshot快照，它恢复时是将快照文件直接读到内存里。 - -Redis会单独创建（fork）一个子进程来进行持久化，会先将数据写入到一个临时文件中，待持久化过程都结束了，再用这个临时文件替换上次持久化好的文件。整个过程中，主进程是不进行任何IO操作的，这就确保了极高的性能，如果需要进行大规模数据的恢复，且对于数据恢复的完整性不是非常敏感，那 RDB 方式要比 AOF 方式更加的高效。RDB的缺点是最后一次持久化后的数据可能丢失。 - -**?** What ? Redis 不是单进程的吗? - -Redis 使用操作系统的多进程 COW(Copy On Write) 机制来实现快照持久化， fork是类Unix操作系统上**创建进程**的主要方法。COW(Copy On Write)是计算机编程中使用的一种优化策略。 - -#### Fork - -fork 的作用是复制一个与当前进程一样的进程。新进程的所有数据（变量、环境变量、程序计数器等）数值都和原进程一致，但是是一个全新的进程，并作为原进程的子进程。 子进程读取数据，然后序列化写到磁盘中 - -rdb 默认保存的是**dump.rdb**文件 - -你可以对 Redis 进行设置， 让它在“ N 秒内数据集至少有 M 个改动”这一条件被满足时， 自动保存一次数据集。 - -你也可以通过调用 [SAVE](http://redisdoc.com/server/save.html#save) 或者 [BGSAVE](http://redisdoc.com/server/bgsave.html#bgsave) ， 手动让 Redis 进行数据集保存操作。 - -比如说， 以下设置会让 Redis 在满足“ 60 秒内有至少有 1000 个键被改动”这一条件时， 自动保存一次数据集： - -`save 60 1000 ` - -这种持久化方式被称为快照（snapshot）。 - -**配置位置**： SNAPSHOTTING - -![redis-snapshotting.png](https://i.loli.net/2019/12/24/JjaO9RohLpvFw4k.png) - -#### 如何触发RDB快照 - -- 配置文件中默认的快照配置 - - 冷拷贝后重新使用 可以`cp dump.rdb dump_new.rdb` - -- 命令save或者是bgsave - - - **Save**：save时只管保存，其它不管，全部阻塞 - - **BGSAVE**：Redis会在后台异步进行快照操作，快照同时还可以响应客户端请求。可以通过lastsave命令获取最后一次成功执行快照的时间 - - 执行**flushall**命令，也会产生dump.rdb文件，但里面是空的，无意义 - -#### 快照的运作方式 - -当 Redis 需要保存 dump.rdb 文件时， 服务器执行以下操作： - -1. Redis 调用 fork() ，产生一个子进程，此时同时拥有父进程和子进程。 -2. 子进程将数据集写入到一个临时 RDB 文件中。 -3. 当子进程完成对新 RDB 文件的写入时，Redis 用新 RDB 文件替换原来的 RDB 文件，并删除旧的 RDB 文件。 - -这种工作方式使得 Redis 可以从写时复制（copy-on-write）机制中获益。 - -#### 如何恢复 - -将备份文件 (dump.rdb) 移动到 redis 安装目录并启动服务即可（CONFIG GET dir获取目录） - -#### 优势 - -- 一旦采用该方式，那么你的整个Redis数据库将只包含一个文件，这对于**文件备份**而言是非常完美的。比如，你可能打算每个小时归档一次最近24小时的数据，同时还要每天归档一次最近30天的数据。通过这样的备份策略，一旦系统出现灾难性故障，我们可以非常容易的进行恢复。**适合大规模的数据恢复** -- 对于灾难恢复而言，RDB是非常不错的选择。因为我们可以非常轻松的将一个单独的文件压缩后再转移到其它存储介质上。 -- 性能最大化。对于Redis的服务进程而言，在开始持久化时，它唯一需要做的只是fork出子进程，之后再由子进程完成这些持久化的工作，这样就可以极大的避免服务进程执行IO操作了。 -- 相比于AOF机制，如果数据集很大，RDB的启动效率会更高。 - -#### 劣势 - -- 如果你想保证数据的高可用性，即最大限度的避免数据丢失，那么RDB将不是一个很好的选择。因为系统一旦在定时持久化之前出现宕机现象，此前没有来得及写入磁盘的数据都将丢失（丢失最后一次快照后的所有修改）。 -- 由于RDB是通过fork子进程来协助完成数据持久化工作的，内存中的数据被克隆了一份，大致2倍的膨胀性需要考虑，因此，如果当数据集较大时，可能会导致整个服务器停止服务几百毫秒，甚至是1秒钟。 - -#### 如何停止 - -动态停止RDB保存规则的方法：`redis-cli config set save ""` - -#### 总结 - -![redis-rdb.png](https://i.loli.net/2019/11/18/75zeUafOsNTkIlw.png) - -- RDB是一个非常紧凑的文件 - -- RDB在保存RDB文件时父进程唯一需要做的就是fork出一个子进程，接下来的工作全部由子进程来做，父进程不需要再做其他IO操作，所以RDB持久化方式可以最大化redis的性能 - -- 与AOF相比，在恢复大的数据集的时候，RDB方式会更快一些 - -- 数据丢失风险大 - -- RDB需要经常fork子进程来保存数据集到硬盘上，当数据集比较大的时候，fork的过程是非常耗时的，可能会导致redis在一些毫秒级不能响应客户端的请求 - - - -## AOF（Append Only File） - -#### 是什么 - -以日志的形式来记录每个写操作，将 Redis 执行过的所有写指令记录下来(读操作不记录)，只许追加文件但不可以改写文件，redis 启动之初会读取该文件重新构建数据，也就是「重放」。换言之，redis 重启的话就根据日志文件的内容将写指令从前到后执行一次以完成数据的恢复工作。 - -AOF 默认保存的是 **appendonly.aof ** 文件 - -**配置位置**： APPEND ONLY MODE - -![redis-aof-conf.jpg](https://i.loli.net/2019/12/24/p2BfU6eyV8miv3N.jpg) - -#### AOF启动/修复/恢复 - -- 正常恢复 - - - 启动：设置Yes 修改默认的appendonly no，改为yes - - 将有数据的 aof 文件复制一份保存到对应目录(config get dir) - - 恢复：重启redis然后重新加载 - -- 异常恢复 - - - 启动：设置Yes 修改默认的appendonly no，改为yes - - 备份被写坏的AOF文件 - - 修复：**redis-check-aof --fix**进行修复 + AOF文件 - - 恢复：重启redis然后重新加载 - -#### rewrite（AOF 重写） - -- 是什么：AOF采用文件追加方式，文件会越来越大为避免出现此种情况，新增了重写机制，当 AOF 文件的大小超过所设定的阈值时，Redis就会启动 AOF 文件的内容压缩，只保留可以恢复数据的最小指令集，可以使用命令`bgrewriteaof`，这个操作相当于对AOF文件“瘦身”。 -- 重写原理：AOF 文件持续增长而过大时，会 fork 出一条新进程来将文件重写(也是先写临时文件最后再rename)，遍历新进程的内存中数据，每条记录有一条的 Set 语句。重写 aof 文件的操作，并没有读取旧的aof文件，而是将整个内存中的数据库内容用命令的方式重写了一个新的 aof 文件，这点和快照有点类似 -- 触发机制：Redis 会记录上次重写时的 AOF 大小，默认配置是当 AOF 文件大小是上次 rewrite 后大小的一倍且文件大于64M 时触发 - -#### AOF耐久性 - -你可以配置 Redis 多久才将数据 fsync 到磁盘一次。 - -有三个选项： - -- 每次有新命令追加到 AOF 文件时就执行一次 fsync ：非常慢，也非常安全。 -- 每秒 fsync 一次：足够快（和使用 RDB 持久化差不多），并且在故障时只会丢失 1 秒钟的数据。 -- 从不 fsync ：将数据交给操作系统来处理。更快，也更不安全的选择。 - -推荐（并且也是默认）的措施为每秒 fsync 一次， 这种 fsync 策略可以兼顾速度和安全性。 - -总是 fsync 的策略在实际使用中非常慢，频繁调用 fsync 注定了这种策略不可能快得起来。 - -#### 如果 AOF 文件出错了，怎么办？ - -服务器可能在程序正在对 AOF 文件进行写入时停机， 如果停机造成了 AOF 文件出错（corrupt）， 那么 Redis 在重启时会拒绝载入这个 AOF 文件， 从而确保数据的一致性不会被破坏。 - -当发生这种情况时， 可以用以下方法来修复出错的 AOF 文件： - -1. 为现有的 AOF 文件创建一个备份。 -2. 使用 Redis 附带的 redis-check-aof 程序，对原来的 AOF 文件进行修复。 - -**$ redis-check-aof --fix** - -1. （可选）使用 diff -u 对比修复后的 AOF 文件和原始 AOF 文件的备份，查看两个文件之间的不同之处。 -2. 重启 Redis 服务器，等待服务器载入修复后的 AOF 文件，并进行数据恢复。 - -#### AOF运作方式 - -AOF 重写和 RDB 创建快照一样，都巧妙地利用了写时复制机制。 - -以下是 AOF 重写的执行步骤： - -1. Redis 执行 fork() ，现在同时拥有父进程和子进程。 -2. 子进程开始将新 AOF 文件的内容写入到临时文件。 -3. 对于所有新执行的写入命令，父进程一边将它们累积到一个内存缓存中，一边将这些改动追加到现有 AOF 文件的末尾： 这样即使在重写的中途发生停机，现有的 AOF 文件也还是安全的。 -4. 当子进程完成重写工作时，它给父进程发送一个信号，父进程在接收到信号之后，将内存缓存中的所有数据追加到新 AOF 文件的末尾。 -5. 搞定！现在 Redis 原子地用新文件替换旧文件，之后所有命令都会直接追加到新 AOF 文件的末尾。 - -#### 优势 - -- 该机制可以带来更高的数据安全性，即数据持久性。Redis中提供了3种同步策略，即**每秒同步、每修改同步和不同步**。事实上，每秒同步也是异步完成的，其效率也是非常高的，所差的是一旦系统出现宕机现象，那么这一秒钟之内修改的数据将会丢失。而每修改同步，我们可以将其视为同步持久化，即每次发生的数据变化都会被立即记录到磁盘中。可以预见，这种方式在效率上是最低的。至于无同步，无需多言，我想大家都能正确的理解它。 -- 由于该机制对日志文件的写入操作采用的是append模式，因此在写入过程中即使出现宕机现象，也不会破坏日志文件中已经存在的内容。然而如果我们本次操作只是写入了一半数据就出现了系统崩溃问题，不用担心，在Redis下一次启动之前，我们可以通过 **redis-check-aof** 工具来帮助我们解决数据一致性的问题。 -- 如果日志过大，Redis 可以自动启用 rewrite 机制。即Redis以append模式不断的将修改数据写入到老的磁盘文件中，同时Redis还会创建一个新的文件用于记录此期间有哪些修改命令被执行。因此在进行rewrite切换时可以更好的保证数据安全性。 -- AOF 包含一个格式清晰、易于理解的日志文件用于记录所有的修改操作。事实上，我们也可以通过该文件完成数据的重建。因此 AOF 文件的内容非常容易被人读懂， 对文件进行分析（parse）也很轻松。 导出（export） AOF 文件也非常简单： 举个例子， 如果你不小心执行了 [FLUSHALL](http://redisdoc.com/server/flushall.html#flushall) 命令， 但只要 AOF 文件未被重写， 那么只要停止服务器， 移除 AOF 文件末尾的 FLUSHALL 命令， 并重启 Redis ， 就可以将数据集恢复到 FLUSHALL 执行之前的状态。 - -#### 劣势 - -- 对于相同数量的数据集而言，AOF文件通常要大于RDB文件。恢复速度慢于rdb。 -- 根据同步策略的不同，AOF在运行效率上往往会慢于RDB。总之，每秒同步策略的效率是比较高的，同步禁用策略的效率和RDB一样高效。 - -#### 总结 - -![redis-aof.png](https://i.loli.net/2019/12/25/2YAgKxsSTRHlqao.png) - -- AOF 文件是一个只进行追加的日志文件 -- Redis 可以在 AOF 文件体积变得过大时，自动在后台对 AOF 进行重写 -- AOF文件有序的保存了对数据库执行的所有写入操作，这些写入操作以Redis协议的格式保存，因此AOF文件的内容非常容易被人读懂，对文件进行分析也很轻松 -- 对于相同的数据集来说，AOF 文件的体积通常需要大于 RDB 文件的体积 -- 根据所使用的 fsync 策略，AOF 的速度可能会慢于 RDB - -**怎么从 RDB 持久化切换到 AOF 持久化** - -在 Redis 2.2 或以上版本，可以在不重启的情况下，从 RDB 切换到 AOF ： - -1. 为最新的 dump.rdb 文件创建一个备份。 -2. 将备份放到一个安全的地方。 -3. 执行以下两条命令： - -```redis - redis-cli> CONFIG SET appendonly yes - - redis-cli> CONFIG SET save "" -``` - -1. 确保命令执行之后，数据库的键的数量没有改变。 -2. 确保写命令会被正确地追加到 AOF 文件的末尾。 - -步骤 3 执行的第一条命令开启了 AOF 功能： Redis 会阻塞直到初始 AOF 文件创建完成为止， 之后 Redis 会继续处理命令请求， 并开始将写入命令追加到 AOF 文件末尾。 - -步骤 3 执行的第二条命令用于关闭 RDB 功能。 这一步是可选的， 如果你愿意的话， 也可以同时使用 RDB 和 AOF 这两种持久化功能。 - -别忘了在 redis.conf 中打开 AOF 功能！ 否则的话， 服务器重启之后， 之前通过 CONFIG SET 设置的配置就会被遗忘， 程序会按原来的配置来启动服务器。 - - - -## Which one - -- RDB 持久化方式能够在指定的时间间隔能对你的数据进行快照存储 - -- AOF 持久化方式记录每次对服务器写的操作，当服务器重启的时候会重新执行这些命令来恢复原始的数据，AOF命令以 redis 协议追加保存每次写的操作到文件末尾。Redis还能对AOF文件进行后台重写（**bgrewriteaof**）,使得 AOF 文件的体积不至于过大 - -- 只做缓存：如果你只希望你的数据在服务器运行的时候存在，你也可以不使用任何持久化方式。 - -- 同时开启两种持久化方式 - - - 在这种情况下,当 redis 重启的时候会优先载入 AOF 文件来恢复原始的数据,因为在通常情况下 AOF 文件保存的数据集要比 RDB 文件保存的数据集要完整。 - - RDB 的数据不实时，同时使用两者时服务器重启也只会找 AOF 文件。那要不要只使用AOF 呢？建议不要，因为 RDB 更适合用于备份数据库(AOF 在不断变化不好备份)，快速重启，而且不会有 AOF 可能潜在的bug，留着作为一个万一的手段。 - -#### 性能建议 - -- 因为 RDB 文件只用作后备用途，建议只在 Slave上持久化 RDB 文件，而且只要15分钟备份一次就够了，只保留save 900 1这条规则。 -- 如果Enalbe AOF，好处是在最恶劣情况下也只会丢失不超过两秒数据，启动脚本较简单只load自己的 AOF 文件就可以了。代价一是带来了持续的 IO，二是 AOF rewrite 的最后将 rewrite 过程中产生的新数据写到新文件造成的阻塞几乎是不可避免的。只要硬盘许可，应该尽量减少 AOF rewrite 的频率，AOF 重写的基础大小默认值64M太小了，可以设到5G以上。默认超过原大小100%大小时重写可以改到适当的数值。 -- 如果不 Enable AOF ，仅靠 Master-Slave Replication 实现高可用性也可以。能省掉一大笔 IO ，也减少了rewrite 时带来的系统波动。代价是如果 Master/Slav e同时宕掉，会丢失十几分钟的数据，启动脚本也要比较两个Master/Slave中的RDB文件，载入较新的那个。 - - - -> [Redis Persistence](https://redis.io/topics/persistence) -> 某免费教学视频 -> -> https://www.wmyskxz.com/2020/03/13/redis-7-chi-jiu-hua-yi-wen-liao-jie/ \ No newline at end of file diff --git a/docs/data-store/Redis/6.Redis-Transaction.md b/docs/data-store/Redis/6.Redis-Transaction.md deleted file mode 100644 index 711b41610d..0000000000 --- a/docs/data-store/Redis/6.Redis-Transaction.md +++ /dev/null @@ -1,183 +0,0 @@ -### Redis事务的几个命令 - -| 命令 | 描述 | -| ------- | ------------------------------------------------------------ | -| MULTI | 标记一个事务块的开始 | -| EXEC | 执行所有事务块内的命令 | -| DISCARD | 取消事务，放弃执行事务块内的所有命令 | -| WATCH | 监视一个（或多个）key，如果在事务执行之前这个（或多个）key被其他命令所改动，那么事务将被打断 | -| UNWATCH | 取消WATCH命令对所有keys的监视 | - -可以一次执行多个命令，本质是一组命令的集合。一个事务中的所有命令都会序列化，按顺序地串行化执行而不会被其它命令插入，不许加塞 - -能干嘛：一个队列中，一次性、顺序性、排他性的执行一系列命令 - -事务可以一次执行多个命令， 并且带有以下两个重要的保证： - -- 事务是一个单独的隔离操作：事务中的所有命令都会序列化、按顺序地执行。事务在执行的过程中，不会被其他客户端发送来的命令请求所打断。 -- 事务是一个原子操作：事务中的命令要么全部被执行，要么全部都不执行。 - -[MULTI](http://redisdoc.com/transaction/multi.html#multi) 命令用于开启一个事务，它总是返回 OK 。 - -[MULTI](http://redisdoc.com/transaction/multi.html#multi) 执行之后， 客户端可以继续向服务器发送任意多条命令， 这些命令不会立即被执行， 而是被放到一个队列中， 当 [EXEC](http://redisdoc.com/transaction/exec.html#exec) 命令被调用时， 所有队列中的命令才会被执行。 - -另一方面， 通过调用 [DISCARD](http://redisdoc.com/transaction/discard.html#discard) ， 客户端可以清空事务队列， 并放弃执行事务。 - -### 事务中的错误 - -使用事务时可能会遇上以下两种错误： - -- 事务在执行 [EXEC](http://redisdoc.com/transaction/exec.html#exec) 之前，入队的命令可能会出错。比如说，命令可能会产生语法错误（参数数量错误，参数名错误，等等），或者其他更严重的错误，比如内存不足（如果服务器使用 maxmemory 设置了最大内存限制的话）。 -- 命令可能在 [EXEC](http://redisdoc.com/transaction/exec.html#exec) 调用之后失败。举个例子，事务中的命令可能处理了错误类型的键，比如将列表命令用在了字符串键上面，诸如此类。 - -对于发生在 [EXEC](http://redisdoc.com/transaction/exec.html#exec) 执行之前的错误，客户端以前的做法是检查命令入队所得的返回值：如果命令入队时返回 QUEUED ，那么入队成功；否则，就是入队失败。如果有命令在入队时失败，那么大部分客户端都会停止并取消这个事务。 - -不过，从 Redis 2.6.5 开始，服务器会对命令入队失败的情况进行记录，并在客户端调用 [EXEC](http://redisdoc.com/transaction/exec.html#exec) 命令时，拒绝执行并自动放弃这个事务。 - -在 Redis 2.6.5 以前， Redis 只执行事务中那些入队成功的命令，而忽略那些入队失败的命令。 而新的处理方式则使得在流水线（pipeline）中包含事务变得简单，因为发送事务和读取事务的回复都只需要和服务器进行一次通讯。 - -至于那些在 [EXEC](http://redisdoc.com/transaction/exec.html#exec) 命令执行之后所产生的错误， 并没有对它们进行特别处理： 即使事务中有某个/某些命令在执行时产生了错误， 事务中的其他命令仍然会继续执行。 - -### 为什么 Redis 不支持回滚 - -如果你有使用关系式数据库的经验， 那么 “Redis 在事务失败时不进行回滚，而是继续执行余下的命令”这种做法可能会让你觉得有点奇怪。 - -以下是这种做法的优点： - -- Redis 命令只会因为错误的语法而失败（并且这些问题不能在入队时发现），或是命令用在了错误类型的键上面：这也就是说，从实用性的角度来说，失败的命令是由编程错误造成的，而这些错误应该在开发的过程中被发现，而不应该出现在生产环境中。 -- 因为不需要对回滚进行支持，所以 Redis 的内部可以保持简单且快速。 - -有种观点认为 Redis 处理事务的做法会产生 bug ， 然而需要注意的是， 在通常情况下， 回滚并不能解决编程错误带来的问题。 举个例子， 如果你本来想通过 [INCR](http://redisdoc.com/string/incr.html#incr) 命令将键的值加上 1 ， 却不小心加上了 2 ， 又或者对错误类型的键执行了 [INCR](http://redisdoc.com/string/incr.html#incr) ， 回滚是没有办法处理这些情况的。 - -鉴于没有任何机制能避免程序员自己造成的错误， 并且这类错误通常不会在生产环境中出现， 所以 Redis 选择了更简单、更快速的无回滚方式来处理事务。 - -简单示例： - -- case1：正常执行（可以批处理 挺爽 每条操作成功的话都会各取所需 互不影响） - - ![redis-transaction-case1.png](http://ww1.sinaimg.cn/large/9b9f09a9ly1g9yqu69x1ej209q05hmx3.jpg) - -- Case2：放弃事务（discard操作表示放弃事务） - - ![redis-transaction-case2.png](http://ww1.sinaimg.cn/large/9b9f09a9ly1g9yqw17zn7j208n03uq2t.jpg) - -- Case3：全体连坐（某一条操作记录报错的话，exec后所有操作都不会成功） - -![redis-transaction-case3.png](http://ww1.sinaimg.cn/large/9b9f09a9ly1g9yqy1ebddj20eh04z3yi.jpg) - -- Case4：冤头债主（示例中 k1被设置为String类型，decr k1可以放入操作队列中，因为只有在执行的时候才可以判断出语句错误，其他正确的会被正常执行） - - ![redis-transaction-case4.png](http://ww1.sinaimg.cn/large/9b9f09a9ly1g9yr1chai1j20df07k0ss.jpg) - -- Case5：watch监控 - - - -### 使用 check-and-set 操作实现乐观锁 - -[WATCH](http://redisdoc.com/transaction/watch.html#watch) 命令可以为 Redis 事务提供 check-and-set （CAS）行为。 - -被 [WATCH](http://redisdoc.com/transaction/watch.html#watch) 的键会被监视，并会发觉这些键是否被改动过了。 如果有至少一个被监视的键在 [EXEC](http://redisdoc.com/transaction/exec.html#exec) 执行之前被修改了， 那么整个事务都会被取消， [EXEC](http://redisdoc.com/transaction/exec.html#exec) 返回空多条批量回复（null multi-bulk reply）来表示事务已经失败。 - -举个例子， 假设我们需要原子性地为某个值进行增 1 操作（假设 [INCR](http://redisdoc.com/string/incr.html#incr) 不存在）。 - -首先我们可能会这样做： - -``` -val = GET mykey -val = val + 1 -SET mykey $val -``` - -上面的这个实现在只有一个客户端的时候可以执行得很好。 但是， 当多个客户端同时对同一个键进行这样的操作时， 就会产生竞争条件。 - -举个例子， 如果客户端 A 和 B 都读取了键原来的值， 比如 10 ， 那么两个客户端都会将键的值设为 11 ， 但正确的结果应该是 12 才对。 - -有了 [WATCH](http://redisdoc.com/transaction/watch.html#watch) ， 我们就可以轻松地解决这类问题了： - -``` -WATCH mykey -val = GET mykey -val = val + 1 -MULTI -SET mykey $val -EXEC -``` - -使用上面的代码， 如果在 WATCH 执行之后， EXEC 执行之前， 有其他客户端修改了 mykey 的值， 那么当前客户端的事务就会失败。 程序需要做的， 就是不断重试这个操作， 直到没有发生碰撞为止。 - -这种形式的锁被称作乐观锁， 它是一种非常强大的锁机制。 并且因为大多数情况下， 不同的客户端会访问不同的键， 碰撞的情况一般都很少， 所以通常并不需要进行重试。 - -**了解 WATCH** - -WATCH 使得 EXEC 命令需要有条件地执行： 事务只能在所有被监视键都没有被修改的前提下执行， 如果这个前提不能满足的话，事务就不会被执行。 - -如果你使用 WATCH 监视了一个带过期时间的键， 那么即使这个键过期了， 事务仍然可以正常执行， 关于这方面的详细情况，请看这个帖子： http://code.google.com/p/redis/issues/detail?id=270 - -WATCH命令可以被调用多次。 对键的监视从 WATCH 执行之后开始生效， 直到调用 EXEC 为止。 - -用户还可以在单个 WATCH 命令中监视任意多个键， 就像这样： - -``` -redis> WATCH key1 key2 key3 OK -``` - -**当** [**EXEC**](http://redisdoc.com/transaction/exec.html#exec) **被调用时， 不管事务是否成功执行， 对所有键的监视都会被取消。**另外， 当客户端断开连接时， 该客户端对键的监视也会被取消。 - -使用无参数的 UNWATCH 命令可以手动取消对所有键的监视。 对于一些需要改动多个键的事务， 有时候程序需要同时对多个键进行加锁， 然后检查这些键的当前值是否符合程序的要求。 当值达不到要求时， 就可以使用 UNWATCH 命令来取消目前对键的监视， 中途放弃这个事务， 并等待事务的下次尝试。 - - - -#### 悲观锁/乐观锁/CAS(Check And Set) - -##### 悲观锁 - - 悲观锁(Pessimistic Lock), 顾名思义，就是很悲观，每次去拿数据的时候都认为别人会修改，所以每次在拿数据的时候都会上锁，这样别人想拿这个数据就会block直到它拿到锁。传统的关系型数据库里边就用到了很多这种锁机制，比如行锁，表锁等，读锁，写锁等，都是在做操作之前先上锁 - -##### 乐观锁 - -乐观锁(Optimistic Lock), 顾名思义，就是很乐观，每次去拿数据的时候都认为别人不会修改，所以不会上锁，但是在更新的时候会判断一下在此期间别人有没有去更新这个数据，可以使用版本号等机制。乐观锁适用于多读的应用类型，这样可以提高吞吐量，乐观锁策略:提交版本必须大于记录当前版本才能执行更新 - -##### CAS - -初始化信用卡可用余额和欠额, 加塞篡改，先监控再开启multi，保证两笔金额变动在同一个事务内 - -有加塞篡改：监控了key，如果key被修改了，后面一个事务的执行失效 - -![](H:\Technical-Learning\docs\_images\redis\redis-watch-demo.jpg) - - unwatch - -一旦执行了exec之前加的监控锁都会被取消掉了 - - - -小结： - -**watch指令，类似乐观锁**，事务提交时，如果key的值已被别的客户端改变，比如某个list已被别的客户端push/pop过了，整个事务队列都不会被执行。 - -通过watch命令在事务执行之前监控了多个keys，倘若在watch之后有任何key的值发生变化，exec命令执行的事务都将被放弃，同时返回Nullmulit-bulk应答以通知调用者事务执行失败 - -![redis-transaction-watch1.png](https://i.loli.net/2019/11/18/oWGvR1H78gCh4dN.png) - -![redis-transaction-watch2.png](https://i.loli.net/2019/11/18/JR1YpD2HSIQ7OEd.png) - -![redis-transaction-watch3.png](https://i.loli.net/2019/11/18/Hp8ilxgOGcbnTJ2.png) - -### 事务3阶段3特性 - -#### 3阶段 - -- 开启：以MULTI开始一个事务 - -- 入队：将多个命令入队到事务中，接到这些命令并不会立即执行，而是放到等待执行的事务队列里面 - -- 执行：由EXEC命令触发事务 - -#### 3特性 - -- 单独的隔离操作：事务中的所有命令都会序列化、按顺序地执行。事务在执行的过程中，不会被其他客户端发送来的命令请求所打断。 - -- **没有隔离级别的概念**：队列中的命令没有提交之前都不会实际的被执行，因为事务提交前任何指令都不会被实际执行，也就不存在”事务内的查询要看到事务里的更新，在事务外查询不能看到”这个让人万分头痛的问题 - -- 不保证原子性：redis同一个事务中如果有一条命令执行失败，其后的命令仍然会被执行，没有回滚 \ No newline at end of file diff --git a/docs/data-store/Redis/Redis-BloomFilter.md b/docs/data-store/Redis/Redis-BloomFilter.md deleted file mode 100644 index c4d5222149..0000000000 --- a/docs/data-store/Redis/Redis-BloomFilter.md +++ /dev/null @@ -1,338 +0,0 @@ -## 隆过滤器是什么？ - -布隆过滤器可以理解为一个不怎么精确的 set 结构，当你使用它的 contains 方法判断某个对象是否存在时，它可能会误判。但是布隆过滤器也不是特别不精确，只要参数设置的合理，它的精确度可以控制的相对足够精确，只会有小小的误判概率。 - -当布隆过滤器说某个值存在时，这个值可能不存在；当它说不存在时，那就肯定不存在。打个比方，当它说不认识你时，肯定就不认识；当它说见过你时，可能根本就没见过面，不过因为你的脸跟它认识的人中某脸比较相似 (某些熟脸的系数组合)，所以误判以前见过你。 - -套在上面的使用场景中，布隆过滤器能准确过滤掉那些已经看过的内容，那些没有看过的新内容，它也会过滤掉极小一部分 (误判)，但是绝大多数新内容它都能准确识别。这样就可以完全保证推荐给用户的内容都是无重复的。 - -## Redis 中的布隆过滤器 - -Redis 官方提供的布隆过滤器到了 Redis 4.0 提供了插件功能之后才正式登场。布隆过滤器作为一个插件加载到 Redis Server 中，给 Redis 提供了强大的布隆去重功能。 - -下面我们来体验一下 Redis 4.0 的布隆过滤器，为了省去繁琐安装过程，我们直接用 Docker 吧。 - -``` -> docker pull redislabs/rebloom # 拉取镜像 -> docker run -p6379:6379 redislabs/rebloom # 运行容器 -> redis-cli # 连接容器中的 redis 服务 -``` - -如果上面三条指令执行没有问题，下面就可以体验布隆过滤器了。 - -## 布隆过滤器基本使用 - -布隆过滤器有二个基本指令，`bf.add` 添加元素，`bf.exists` 查询元素是否存在，它的用法和 set 集合的 sadd 和 sismember 差不多。注意 `bf.add` 只能一次添加一个元素，如果想要一次添加多个，就需要用到 `bf.madd` 指令。同样如果需要一次查询多个元素是否存在，就需要用到 `bf.mexists` 指令。 - -``` -127.0.0.1:6379> bf.add codehole user1 -(integer) 1 -127.0.0.1:6379> bf.add codehole user2 -(integer) 1 -127.0.0.1:6379> bf.add codehole user3 -(integer) 1 -127.0.0.1:6379> bf.exists codehole user1 -(integer) 1 -127.0.0.1:6379> bf.exists codehole user2 -(integer) 1 -127.0.0.1:6379> bf.exists codehole user3 -(integer) 1 -127.0.0.1:6379> bf.exists codehole user4 -(integer) 0 -127.0.0.1:6379> bf.madd codehole user4 user5 user6 -1) (integer) 1 -2) (integer) 1 -3) (integer) 1 -127.0.0.1:6379> bf.mexists codehole user4 user5 user6 user7 -1) (integer) 1 -2) (integer) 1 -3) (integer) 1 -4) (integer) 0 -``` - - - -Java 客户端 Jedis-2.x 没有提供指令扩展机制，所以你无法直接使用 Jedis 来访问 Redis Module 提供的 [bf.xxx](http://bf.xxx/) 指令。RedisLabs 提供了一个单独的包 [JReBloom](https://github.com/RedisLabs/JReBloom)，但是它是基于 Jedis-3.0，Jedis-3.0 这个包目前还没有进入 release，没有进入 maven 的中央仓库，需要在 Github 上下载。在使用上很不方便，如果怕麻烦，还可以使用 [lettuce](https://github.com/lettuce-io/lettuce-core)，它是另一个 Redis 的客户端，相比 Jedis 而言，它很早就支持了指令扩展。 - -``` -public class BloomTest { - - public static void main(String[] args) { - Client client = new Client(); - - client.delete("codehole"); - for (int i = 0; i < 100000; i++) { - client.add("codehole", "user" + i); - boolean ret = client.exists("codehole", "user" + i); - if (!ret) { - System.out.println(i); - break; - } - } - - client.close(); - } - -} -``` - -执行上面的代码后，你会张大了嘴巴发现居然没有输出，塞进去了 100000 个元素，还是没有误判，这是怎么回事？如果你不死心的话，可以将数字再加一个 0 试试，你会发现依然没有误判。 - - 原因就在于布隆过滤器对于已经见过的元素肯定不会误判，它只会误判那些没见过的元素。所以我们要稍微改一下上面的脚本，使用 bf.exists 去查找没见过的元素，看看它是不是以为自己见过了。 - -```java -public class BloomTest { - - public static void main(String[] args) { - Client client = new Client(); - - client.delete("codehole"); - for (int i = 0; i < 100000; i++) { - client.add("codehole", "user" + i); - boolean ret = client.exists("codehole", "user" + (i + 1)); - if (ret) { - System.out.println(i); - break; - } - } - - client.close(); - } - -} -``` - - - -运行后，我们看到了输出是 214，也就是到第 214 的时候，它出现了误判。 - -那如何来测量误判率呢？我们先随机出一堆字符串，然后切分为 2 组，将其中一组塞入布隆过滤器，然后再判断另外一组的字符串存在与否，取误判的个数和字符串总量一半的百分比作为误判率。 - -```java -public class BloomTest { - - private String chars; - { - StringBuilder builder = new StringBuilder(); - for (int i = 0; i < 26; i++) { - builder.append((char) ('a' + i)); - } - chars = builder.toString(); - } - - private String randomString(int n) { - StringBuilder builder = new StringBuilder(); - for (int i = 0; i < n; i++) { - int idx = ThreadLocalRandom.current().nextInt(chars.length()); - builder.append(chars.charAt(idx)); - } - return builder.toString(); - } - - private List randomUsers(int n) { - List users = new ArrayList<>(); - for (int i = 0; i < 100000; i++) { - users.add(randomString(64)); - } - return users; - } - - public static void main(String[] args) { - BloomTest bloomer = new BloomTest(); - List users = bloomer.randomUsers(100000); - List usersTrain = users.subList(0, users.size() / 2); - List usersTest = users.subList(users.size() / 2, users.size()); - - Client client = new Client(); - client.delete("codehole"); - for (String user : usersTrain) { - client.add("codehole", user); - } - int falses = 0; - for (String user : usersTest) { - boolean ret = client.exists("codehole", user); - if (ret) { - falses++; - } - } - System.out.printf("%d %d\n", falses, usersTest.size()); - client.close(); - } - -} -``` - -运行一下，等待大约一分钟，输出: - -``` -total users 100000 -all trained -628 50000 -``` - -可以看到误判率大约 1% 多点。你也许会问这个误判率还是有点高啊，有没有办法降低一点？答案是有的。 - -我们上面使用的布隆过滤器只是默认参数的布隆过滤器，它在我们第一次 add 的时候自动创建。Redis 其实还提供了自定义参数的布隆过滤器，需要我们在 add 之前使用`bf.reserve`指令显式创建。如果对应的 key 已经存在，`bf.reserve`会报错。`bf.reserve`有三个参数，分别是 key, `error_rate`和`initial_size`。错误率越低，需要的空间越大。`initial_size`参数表示预计放入的元素数量，当实际数量超出这个数值时，误判率会上升。 - -所以需要提前设置一个较大的数值避免超出导致误判率升高。如果不使用 bf.reserve，默认的`error_rate`是 0.01，默认的`initial_size`是 100。 - - 接下来我们使用 bf.reserve 改造一下上面的脚本： - -Java 版本： - -``` -public class BloomTest { - - private String chars; - { - StringBuilder builder = new StringBuilder(); - for (int i = 0; i < 26; i++) { - builder.append((char) ('a' + i)); - } - chars = builder.toString(); - } - - private String randomString(int n) { - StringBuilder builder = new StringBuilder(); - for (int i = 0; i < n; i++) { - int idx = ThreadLocalRandom.current().nextInt(chars.length()); - builder.append(chars.charAt(idx)); - } - return builder.toString(); - } - - private List randomUsers(int n) { - List users = new ArrayList<>(); - for (int i = 0; i < 100000; i++) { - users.add(randomString(64)); - } - return users; - } - - public static void main(String[] args) { - BloomTest bloomer = new BloomTest(); - List users = bloomer.randomUsers(100000); - List usersTrain = users.subList(0, users.size() / 2); - List usersTest = users.subList(users.size() / 2, users.size()); - - Client client = new Client(); - client.delete("codehole"); - // 对应 bf.reserve 指令 - client.createFilter("codehole", 50000, 0.001); - for (String user : usersTrain) { - client.add("codehole", user); - } - int falses = 0; - for (String user : usersTest) { - boolean ret = client.exists("codehole", user); - if (ret) { - falses++; - } - } - System.out.printf("%d %d\n", falses, usersTest.size()); - client.close(); - } - -} -``` - -运行一下，等待约 1 分钟，输出如下： - -``` -total users 100000 -all trained -6 50000 -``` - -我们看到了误判率大约 0.012%，比预计的 0.1% 低很多，不过布隆的概率是有误差的，只要不比预计误判率高太多，都是正常现象。 - -## 注意事项 - -布隆过滤器的`initial_size`估计的过大，会浪费存储空间，估计的过小，就会影响准确率，用户在使用之前一定要尽可能地精确估计好元素数量，还需要加上一定的冗余空间以避免实际元素可能会意外高出估计值很多。 - -布隆过滤器的`error_rate`越小，需要的存储空间就越大，对于不需要过于精确的场合，`error_rate`设置稍大一点也无伤大雅。比如在新闻去重上而言，误判率高一点只会让小部分文章不能让合适的人看到，文章的整体阅读量不会因为这点误判率就带来巨大的改变。 - -## 布隆过滤器的原理 - -学会了布隆过滤器的使用，下面有必要把原理解释一下，不然读者还会继续蒙在鼓里 - - - -![img](https://user-gold-cdn.xitu.io/2018/7/4/16464301a0e26416?imageView2/0/w/1280/h/960/format/webp/ignore-error/1) - - - -每个布隆过滤器对应到 Redis 的数据结构里面就是一个大型的位数组和几个不一样的无偏 hash 函数。所谓无偏就是能够把元素的 hash 值算得比较均匀。 - -向布隆过滤器中添加 key 时，会使用多个 hash 函数对 key 进行 hash 算得一个整数索引值然后对位数组长度进行取模运算得到一个位置，每个 hash 函数都会算得一个不同的位置。再把位数组的这几个位置都置为 1 就完成了 add 操作。 - -向布隆过滤器询问 key 是否存在时，跟 add 一样，也会把 hash 的几个位置都算出来，看看位数组中这几个位置是否都为 1，只要有一个位为 0，那么说明布隆过滤器中这个 key 不存在。如果都是 1，这并不能说明这个 key 就一定存在，只是极有可能存在，因为这些位被置为 1 可能是因为其它的 key 存在所致。如果这个位数组比较稀疏，判断正确的概率就会很大，如果这个位数组比较拥挤，判断正确的概率就会降低。具体的概率计算公式比较复杂，感兴趣可以阅读扩展阅读，非常烧脑，不建议读者细看。 - -使用时不要让实际元素远大于初始化大小，当实际元素开始超出初始化大小时，应该对布隆过滤器进行重建，重新分配一个 size 更大的过滤器，再将所有的历史元素批量 add 进去 (这就要求我们在其它的存储器中记录所有的历史元素)。因为 error_rate 不会因为数量超出就急剧增加，这就给我们重建过滤器提供了较为宽松的时间。 - -## 空间占用估计 - -布隆过滤器的空间占用有一个简单的计算公式，但是推导比较繁琐，这里就省去推导过程了，直接引出计算公式，感兴趣的读者可以点击「扩展阅读」深入理解公式的推导过程。 - -布隆过滤器有两个参数，第一个是预计元素的数量 n，第二个是错误率 f。公式根据这两个输入得到两个输出，第一个输出是位数组的长度 l，也就是需要的存储空间大小 (bit)，第二个输出是 hash 函数的最佳数量 k。hash 函数的数量也会直接影响到错误率，最佳的数量会有最低的错误率。 - -``` -k=0.7*(l/n) # 约等于 -f=0.6185^(l/n) # ^ 表示次方计算，也就是 math.pow -``` - -从公式中可以看出 - -1. 位数组相对越长 (l/n)，错误率 f 越低，这个和直观上理解是一致的 -2. 位数组相对越长 (l/n)，hash 函数需要的最佳数量也越多，影响计算效率 -3. 当一个元素平均需要 1 个字节 (8bit) 的指纹空间时 (l/n=8)，错误率大约为 2% -4. 错误率为 10%，一个元素需要的平均指纹空间为 4.792 个 bit，大约为 5bit -5. 错误率为 1%，一个元素需要的平均指纹空间为 9.585 个 bit，大约为 10bit -6. 错误率为 0.1%，一个元素需要的平均指纹空间为 14.377 个 bit，大约为 15bit - -你也许会想，如果一个元素需要占据 15 个 bit，那相对 set 集合的空间优势是不是就没有那么明显了？这里需要明确的是，set 中会存储每个元素的内容，而布隆过滤器仅仅存储元素的指纹。元素的内容大小就是字符串的长度，它一般会有多个字节，甚至是几十个上百个字节，每个元素本身还需要一个指针被 set 集合来引用，这个指针又会占去 4 个字节或 8 个字节，取决于系统是 32bit 还是 64bit。而指纹空间只有接近 2 个字节，所以布隆过滤器的空间优势还是非常明显的。 - -如果读者觉得公式计算起来太麻烦，也没有关系，有很多现成的网站已经支持计算空间占用的功能了，我们只要把参数输进去，就可以直接看到结果，比如 [布隆计算器](https://krisives.github.io/bloom-calculator/)。 - - - -![img](https://user-gold-cdn.xitu.io/2018/7/4/16464885cf1f74c2?imageView2/0/w/1280/h/960/format/webp/ignore-error/1) - - - -## 实际元素超出时，误判率会怎样变化 - -当实际元素超出预计元素时，错误率会有多大变化，它会急剧上升么，还是平缓地上升，这就需要另外一个公式，引入参数 t 表示实际元素和预计元素的倍数 t - -``` -f=(1-0.5^t)^k # 极限近似，k 是 hash 函数的最佳数量 -``` - -当 t 增大时，错误率，f 也会跟着增大，分别选择错误率为 10%,1%,0.1% 的 k 值，画出它的曲线进行直观观察 - - - -![img](https://user-gold-cdn.xitu.io/2018/7/5/164685454156e8e2?imageView2/0/w/1280/h/960/format/webp/ignore-error/1) - - - -从这个图中可以看出曲线还是比较陡峭的 - -1. 错误率为 10% 时，倍数比为 2 时，错误率就会升至接近 40%，这个就比较危险了 -2. 错误率为 1% 时，倍数比为 2 时，错误率升至 15%，也挺可怕的 -3. 错误率为 0.1%，倍数比为 2 时，错误率升至 5%，也比较悬了 - -## 用不上 Redis4.0 怎么办？ - -Redis 4.0 之前也有第三方的布隆过滤器 lib 使用，只不过在实现上使用 redis 的位图来实现的，性能上也要差不少。比如一次 exists 查询会涉及到多次 getbit 操作，网络开销相比而言会高出不少。另外在实现上这些第三方 lib 也不尽完美，比如 pyrebloom 库就不支持重连和重试，在使用时需要对它做一层封装后才能在生产环境中使用。 - -1. [Python Redis Bloom Filter](https://github.com/robinhoodmarkets/pyreBloom) -2. [Java Redis Bloom Filter](https://github.com/Baqend/Orestes-Bloomfilter) - -## 布隆过滤器的其它应用 - -在爬虫系统中，我们需要对 URL 进行去重，已经爬过的网页就可以不用爬了。但是 URL 太多了，几千万几个亿，如果用一个集合装下这些 URL 地址那是非常浪费空间的。这时候就可以考虑使用布隆过滤器。它可以大幅降低去重存储消耗，只不过也会使得爬虫系统错过少量的页面。 - -布隆过滤器在 NoSQL 数据库领域使用非常广泛，我们平时用到的 HBase、Cassandra 还有 LevelDB、RocksDB 内部都有布隆过滤器结构，布隆过滤器可以显著降低数据库的 IO 请求数量。当用户来查询某个 row 时，可以先通过内存中的布隆过滤器过滤掉大量不存在的 row 请求，然后再去磁盘进行查询。 - -邮箱系统的垃圾邮件过滤功能也普遍用到了布隆过滤器，因为用了这个过滤器，所以平时也会遇到某些正常的邮件被放进了垃圾邮件目录中，这个就是误判所致，概率很低。 \ No newline at end of file diff --git a/docs/data-store/Redis/Redis-clients.md b/docs/data-store/Redis/Redis-clients.md deleted file mode 100644 index e4d8e4051a..0000000000 --- a/docs/data-store/Redis/Redis-clients.md +++ /dev/null @@ -1,3 +0,0 @@ - https://redis.io/clients#java - -![image-20191225101834710](C:\Users\jiahaixin\AppData\Roaming\Typora\typora-user-images\image-20191225101834710.png) \ No newline at end of file diff --git a/docs/data-store/Redis/Reids-Lock.md b/docs/data-store/Redis/Reids-Lock.md deleted file mode 100644 index 64d14e182c..0000000000 --- a/docs/data-store/Redis/Reids-Lock.md +++ /dev/null @@ -1,79 +0,0 @@ -# Redis分布式锁 - -## 一、什么是分布式锁？ - -要介绍分布式锁，首先要提到与分布式锁相对应的是线程锁、进程锁。 - -**线程锁**：主要用来给方法、代码块加锁。当某个方法或代码使用锁，在同一时刻仅有一个线程执行该方法或该代码段。线程锁只在同一JVM中有效果，因为线程锁的实现在根本上是依靠线程之间共享内存实现的，比如synchronized是共享对象头，显示锁Lock是共享某个变量（state）。 - -**进程锁**：为了控制同一操作系统中多个进程访问某个共享资源，因为进程具有独立性，各个进程无法访问其他进程的资源， 可以使用本地系统的信号量控制 。 - -**分布式锁**：分布式锁是控制分布式系统或不同系统之间共同访问共享资源的一种锁实现，如果不同的系统或同一个系统的不同主机之间共享了某个资源时，往往需要互斥来防止彼此干扰来保证一致性。 - - - -分布式锁一般有三种实现方式：**1. 数据库乐观锁；2. 基于Redis的分布式锁；3. 基于ZooKeeper的分布式锁。** - - - -TODO:乐观锁、悲观锁 - - - -### 可靠性 - -首先，为了确保分布式锁可用，我们至少要确保锁的实现同时满足以下四个条件： - -1. 互斥性。在任意时刻，只有一个客户端能持有锁。 -2. 不会发生死锁。即使有一个客户端在持有锁的期间崩溃而没有主动解锁，也能保证后续其他客户端能加锁。 -3. 具有容错性。只要大部分的Redis节点正常运行，客户端就可以加锁和解锁。 -4. 解铃还须系铃人。加锁和解锁必须是同一个客户端，客户端自己不能把别人加的锁给解了。 - - - -## 基于 Redis 做分布式锁 - -setnx(key, value)：“set if not exits”，若该key-value不存在，则成功加入缓存并且返回1，存在返回0。 - -get(key)：获得key对应的value值，若不存在则返回nil。 - -getset(key, value)：先获取key对应的value值，若不存在则返回nil，然后将旧的value更新为新的value。 - -expire(key, seconds)：设置key-value的有效期为seconds秒。 - - - -Redis 2.8 版本中作者加入了 set 指令的扩展参数，使得 setnx 和 expire 指令可以一起执行，彻底解决了分布式锁的乱象。 - -\> set lock:codehole true ex 5 nx OK ... **do** something critical ... > del lock:codehole - -上面这个指令就是 setnx 和 expire 组合在一起的原子指令，它就是分布式锁的奥义所在。 - -``` -set key value[expiration EX seconds|PX milliseconds] [NX|XX] -``` - - - -### 基于 redisson 做分布式锁 - -redisson 是 redis 官方的分布式锁组件。GitHub 地址：[https://github.com/redisson/redisson](https://zhuanlan.zhihu.com/write) - -上面的这个问题 ——> 失效时间设置多长时间为好？这个问题在 redisson 的做法是：每获得一个锁时，只设置一个很短的超时时间，同时起一个线程在每次快要到超时时间时去刷新锁的超时时间。在释放锁的同时结束这个线程。 - -```java -RedissonClient redissonClient = Redisson.create(); -RLock rLock = redissonClient.getLock("resourceName"); -//直接加锁 -//rLock.lock(); -//尝试加锁5秒，锁过期时间10秒 -rLock.tryLock(5,10,TimeUnit.SECONDS); -//非阻塞异步加锁 -RFuture rFuture = rLock.tryLockAsync(5,10,TimeUnit.SECONDS);rLock.unlock(); -``` - - - -## RedLock - -我们想象一个这样的场景当机器A申请到一把锁之后，如果Redis主宕机了，这个时候从机并没有同步到这一把锁，那么机器B再次申请的时候就会再次申请到这把锁，为了解决这个问题Redis作者提出了RedLock红锁的算法,在Redission中也对RedLock进行了实现。 \ No newline at end of file diff --git a/docs/data-store/Redis/sidebar.md b/docs/data-store/Redis/sidebar.md deleted file mode 100644 index f5ca71a385..0000000000 --- a/docs/data-store/Redis/sidebar.md +++ /dev/null @@ -1,40 +0,0 @@ -- **Java基础** -- [JUC](#) -- [NIO](#) -- **数据存储和缓存** -- [![MySQL](../_media/mysql-original.svg)MySQL](mysql/readMySQL.md) -- [![Redis](../_media/redis-original.svg) Redis](redis/2.readRedis.md) - - [NoSQL 简介](redis/1.Nosql-Overview.md) - - [Redis 数据类型](redis/3.Redis-Datatype.md) - - [redis.conf](redis/4.Redis-Conf.md) - - [Redis 持久化](redis/5.Redis-Persistence.md) - - [Redis 事务](redis/6.Redis-Transaction.md) - - [Redis FAQ](redis/Redis-FAQ.md) -- [![mongoDB](../_media/mongodb-original.svg)mongoDB]( https://redis.io/ ) -- [![ **Elasticsearch** ](../_media/elasticsearch.svg) Elasticsearch]( https://redis.io/ ) -- [![S3](../_media/amazonwebservices-original.svg)S3]( https://aws.amazon.com/cn/s3/ ) -- FastDFS(OSS) -- **单体架构** -- [缕清各种Java Logging](logging/Java-Logging.md) -- [hello logback](logging/logback简单使用.md) -- **微服务架构** -- Spring Boot -- Spring Cloud -- **面向服务架构** -- [![message](../_media/message.svg) 消息中间件](message-queue/readMQ.md) -- [![Nginx](../_media/nginx-original.svg)Nginx](nginx/nginx.md) -- **工程化与工具** -- [![Maven](https://icongram.jgog.in/fontawesome/maxcdn.svg?&size=16)Maven](logging/logback简单使用.md) -- [![Git](../_media/git-original.svg?&size=16) Git](logging/logback简单使用.md) -- Sonar -- **大数据** -- HDFS -- **性能优化** -- JVM优化 -- web调优 -- DB调优 -- **其他** -- [![Linux](https://icongram.jgog.in/devicon/linux-original.svg?&size=16)Linux](linux/linux.md) -- **Links** -- [![Github](https://icongram.jgog.in/simple/github.svg?color=808080&size=16)Github](https://github.com/jhildenbiddle/docsify-tabs) -- [![Blog](https://icongram.jgog.in/simple/aboutme.svg?colored&size=16)My Blog](https://www.lazyegg.net) diff --git a/docs/data-structure-algorithms/.DS_Store b/docs/data-structure-algorithms/.DS_Store new file mode 100644 index 0000000000..661b6ce50c Binary files /dev/null and b/docs/data-structure-algorithms/.DS_Store differ diff --git a/docs/data-structure-algorithms/BTree.md b/docs/data-structure-algorithms/BTree.md new file mode 100644 index 0000000000..0e2ab78d7c --- /dev/null +++ b/docs/data-structure-algorithms/BTree.md @@ -0,0 +1,175 @@ +## B树 + +B树也是一种用于查找的平衡树，但是它不是二叉树。 + +B-tree 树即 B 树**，B即Balanced，平衡的意思。因为B树的原英文名称为B-tree，而国内很多人喜欢把B-tree译作B-树，其实，这是个非常不好的直译，很容易让人产生误解。如可能会以为B-树是一种树，而B树又是另一种树。而事实上是，B-tree就是指的B树**。特此说明。 + + + +**B树的定义：**B树（B-tree）是一种树状数据结构，能够用来存储排序后的数据。这种数据结构能够让查找数据、循序存取、插入数据及删除的动作，都在对数时间内完成。 + +B树，概括来说是一个一般化的二叉查找树，可以拥有多于2个子节点。与自平衡二叉查找树不同，B-树为系统最优化大块数据的读和写操作。B-tree算法减少定位记录时所经历的中间过程，从而加快存取速度。这种数据结构常被应用在数据库和文件系统的实作上。 + +在B树中查找给定关键字的方法是，首先把根结点取来，在根结点所包含的关键字K1,…,Kn查找给定的关键字（可用顺序查找或二分查找法），若找到等于给定值的关键字，则查找成功；否则，一定可以确定要查找的关键字在Ki与Ki+1之间，Pi为指向子树根节点的指针，此时取指针Pi所指的结点继续查找，直至找到，或指针Pi为空时查找失败。 + +B树作为一种多路搜索树（并不是二叉的）： + +B树的性质 + +M为树的阶数，B-树或为空树，否则满足下列条件： + +1. 定义任意非叶子结点最多只有M个儿子；且M>2； +2. 根结点的儿子数为[2, M]； +3. 除根结点以外的非叶子结点的儿子数为[M/2, M]； +4. 每个结点存放至少M/2-1（取上整）和至多M-1个关键字；（至少2个关键字） +5. 非叶子结点的关键字个数=指向儿子的指针个数-1； +6. 非叶子结点的关键字：K[1], K[2], …, K[M-1]；且K[i] < K[i+1]； +7. 非叶子结点的指针：P[1], P[2], …, P[M]；其中P[1]指向关键字小于K[1]的子树，P[M]指向关键字大于K[M-1]的子树，其它P[i]指向关键字属于(K[i-1], K[i])的子树； +8. 所有叶子结点位于同一层； + +​ 如下图为一个M=3的B树示例： + +![img](https://camo.githubusercontent.com/dfde3dddb226018bc185288ac84dc380f77be859/687474703a2f2f696d672e626c6f672e6373646e2e6e65742f3230313630363132313135353530313737) + +　　B树创建的示意图： + +![img](https://files.cnblogs.com/yangecnu/btreebuild.gif) + + + +B-树的搜索，从根结点开始，对结点内的关键字（有序）序列进行二分查找，如果命中则结束，否则进入查询关键字所属范围的儿子结点；重复，直到所对应的儿子指针为空，或已经是叶子结点。 + + + +B-Tree、B+Tree、红黑树、B*Tree数据结构 https://blog.csdn.net/zhangliangzi/article/details/51367639 + +## 平衡多路查找树（B-Tree） + +B-Tree是为磁盘等外存储设备设计的一种平衡查找树。因此在讲B-Tree之前先了解下磁盘的相关知识。 + +系统从磁盘读取数据到内存时是以磁盘块（block）为基本单位的，位于同一个磁盘块中的数据会被一次性读取出来，而不是需要什么取什么。 + +InnoDB存储引擎中有页（Page）的概念，页是其磁盘管理的最小单位。InnoDB存储引擎中默认每个页的大小为16KB，可通过参数innodb_page_size将页的大小设置为4K、8K、16K，在[MySQL](http://lib.csdn.net/base/mysql)中可通过如下命令查看页的大小： + +``` +mysql> show variables like 'innodb_page_size'; +``` + +- 1 + +- 1 + +而系统一个磁盘块的存储空间往往没有这么大，因此InnoDB每次申请磁盘空间时都会是若干地址连续磁盘块来达到页的大小16KB。InnoDB在把磁盘数据读入到磁盘时会以页为基本单位，在查询数据时如果一个页中的每条数据都能有助于定位数据记录的位置，这将会减少磁盘I/O次数，提高查询效率。 + +B-Tree结构的数据可以让系统高效的找到数据所在的磁盘块。为了描述B-Tree，首先定义一条记录为一个二元组[key, data] ，key为记录的键值，对应表中的主键值，data为一行记录中除主键外的数据。对于不同的记录，key值互不相同。 + +一棵m阶的B-Tree有如下特性：  + +1. 每个节点最多有m个孩子。  +2. 除了根节点和叶子节点外，其它每个节点至少有Ceil(m/2)个孩子（其中ceil(x)是一个取上限的函数）。  +3. 若根节点不是叶子节点，则至少有2个孩子  +4. 所有叶子节点都在同一层，且不包含其它关键字信息  +5. 每个非终端节点包含n个关键字信息（P0,P1,…Pn, k1,…kn）  +6. 关键字的个数n满足：ceil(m/2)-1 <= n <= m-1  +7. ki(i=1,…n)为关键字，且关键字升序排序。  +8. Pi(i=1,…n)为指向子树根节点的指针。P(i-1)指向的子树的所有节点关键字均小于ki，但都大于k(i-1) + +B-Tree中的每个节点根据实际情况可以包含大量的关键字信息和分支，如下图所示为一个3阶的B-Tree： +![索引](https://img-blog.csdn.net/20160202204827368) + +每个节点占用一个盘块的磁盘空间，一个节点上有两个升序排序的关键字和三个指向子树根节点的指针，指针存储的是子节点所在磁盘块的地址。两个关键词划分成的三个范围域对应三个指针指向的子树的数据的范围域。以根节点为例，关键字为17和35，P1指针指向的子树的数据范围为小于17，P2指针指向的子树的数据范围为17~35，P3指针指向的子树的数据范围为大于35。 + +模拟查找关键字29的过程： + +1. 根据根节点找到磁盘块1，读入内存。【磁盘I/O操作第1次】 +2. 比较关键字29在区间（17,35），找到磁盘块1的指针P2。 +3. 根据P2指针找到磁盘块3，读入内存。【磁盘I/O操作第2次】 +4. 比较关键字29在区间（26,30），找到磁盘块3的指针P2。 +5. 根据P2指针找到磁盘块8，读入内存。【磁盘I/O操作第3次】 +6. 在磁盘块8中的关键字列表中找到关键字29。 + +分析上面过程，发现需要3次磁盘I/O操作，和3次内存查找操作。由于内存中的关键字是一个有序表结构，可以利用二分法查找提高效率。而3次磁盘I/O操作是影响整个B-Tree查找效率的决定因素。B-Tree相对于AVLTree缩减了节点个数，使每次磁盘I/O取到内存的数据都发挥了作用，从而提高了查询效率。 + + + +## B+树 + +B+树是B树的变体，也是一种多路搜索树： + +　　1) 其定义基本与B-树相同，除了： + +　　2) 非叶子结点的子树指针与关键字个数相同； + +　　3) 非叶子结点的子树指针P[i]，指向关键字值属于[K[i], K[i+1])的子树（B-树是开区间）； + +　　4) 为所有叶子结点增加一个链指针； + +　　5) 所有关键字都在叶子结点出现； + +　　下图为M=3的B+树的示意图： + +![img](http://p.blog.csdn.net/images/p_blog_csdn_net/manesking/5.JPG) + +　　B+树的搜索与B树也基本相同，区别是B+树只有达到叶子结点才命中（B树可以在非叶子结点命中），其性能也等价于在关键字全集做一次二分查找； + +　　**B+的性质：** + +　　1.所有关键字都出现在叶子结点的链表中（稠密索引），且链表中的关键字恰好是有序的； + +　　2.不可能在非叶子结点命中； + +　　3.非叶子结点相当于是叶子结点的索引（稀疏索引），叶子结点相当于是存储（关键字）数据的数据层； + +　　4.更适合文件索引系统。 + +　　下面为一个B+树创建的示意图： + +![img](https://files.cnblogs.com/yangecnu/Bplustreebuild.gif) + +B+Tree是在B-Tree基础上的一种优化，使其更适合实现外存储索引结构，InnoDB存储引擎就是用B+Tree实现其索引结构。 + +从上一节中的B-Tree结构图中可以看到每个节点中不仅包含数据的key值，还有data值。而每一个页的存储空间是有限的，如果data数据较大时将会导致每个节点（即一个页）能存储的key的数量很小，当存储的数据量很大时同样会导致B-Tree的深度较大，增大查询时的磁盘I/O次数，进而影响查询效率。在B+Tree中，所有数据记录节点都是按照键值大小顺序存放在同一层的叶子节点上，而非叶子节点上只存储key值信息，这样可以大大加大每个节点存储的key值数量，降低B+Tree的高度。 + +B+Tree相对于B-Tree有几点不同： + +1. 非叶子节点只存储键值信息。 +2. 所有叶子节点之间都有一个链指针。 +3. 数据记录都存放在叶子节点中。 + +将上一节中的B-Tree优化，由于B+Tree的非叶子节点只存储键值信息，假设每个磁盘块能存储4个键值及指针信息，则变成B+Tree后其结构如下图所示： +![索引](https://img-blog.csdn.net/20160202205105560) + +通常在B+Tree上有两个头指针，一个指向根节点，另一个指向关键字最小的叶子节点，而且所有叶子节点（即数据节点）之间是一种链式环结构。因此可以对B+Tree进行两种查找运算：一种是对于主键的范围查找和分页查找，另一种是从根节点开始，进行随机查找。 + +可能上面例子中只有22条数据记录，看不出B+Tree的优点，下面做一个推算： + +InnoDB存储引擎中页的大小为16KB，一般表的主键类型为INT（占用4个字节）或BIGINT（占用8个字节），指针类型也一般为4或8个字节，也就是说一个页（B+Tree中的一个节点）中大概存储16KB/(8B+8B)=1K个键值（因为是估值，为方便计算，这里的K取值为〖10〗^3）。也就是说一个深度为3的B+Tree索引可以维护10^3 * 10^3 * 10^3 = 10亿 条记录。 + +实际情况中每个节点可能不能填充满，因此在数据库中，B+Tree的高度一般都在2~4层。[mysql](http://lib.csdn.net/base/mysql)的InnoDB存储引擎在设计时是将根节点常驻内存的，也就是说查找某一键值的行记录时最多只需要1~3次磁盘I/O操作。 + +数据库中的B+Tree索引可以分为聚集索引（clustered index）和辅助索引（secondary index）。上面的B+Tree示例图在数据库中的实现即为聚集索引，聚集索引的B+Tree中的叶子节点存放的是整张表的行记录数据。辅助索引与聚集索引的区别在于辅助索引的叶子节点并不包含行记录的全部数据，而是存储相应行数据的聚集索引键，即主键。当通过辅助索引来查询数据时，InnoDB存储引擎会遍历辅助索引找到主键，然后再通过主键在聚集索引中找到完整的行记录数据。 + + + +## B*树 + +　　B*树是B+树的变体，在B+树的非根和非叶子结点再增加指向兄弟的指针，将结点的最低利用率从1/2提高到2/3。 + +　　B*树如下图所示： + +![img](http://p.blog.csdn.net/images/p_blog_csdn_net/manesking/6.JPG) + +　　B*树定义了非叶子结点关键字个数至少为(2/3)*M，即块的最低使用率为2/3（代替B+树的1/2）； + +　　B+树的分裂：当一个结点满时，分配一个新的结点，并将原结点中1/2的数据复制到新结点，最后在父结点中增加新结点的指针；B+树的分裂只影响原结点和父结点，而不会影响兄弟结点，所以它不需要指向兄弟的指针； + +　　B*树的分裂：当一个结点满时，如果它的下一个兄弟结点未满，那么将一部分数据移到兄弟结点中，再在原结点插入关键字，最后修改父结点中兄弟结点的关键字（因为兄弟结点的关键字范围改变了）；如果兄弟也满了，则在原结点与兄弟结点之间增加新结点，并各复制1/3的数据到新结点，最后在父结点增加新结点的指针； + +　　所以，B*树分配新结点的概率比B+树要低，空间使用率更高。 + + + + + + + diff --git a/docs/data-structure-algorithms/README.md b/docs/data-structure-algorithms/README.md new file mode 100644 index 0000000000..c306a84679 --- /dev/null +++ b/docs/data-structure-algorithms/README.md @@ -0,0 +1,546 @@ +--- +title: 数据结构与算法：Java开发者的必备修炼 +date: 2025-05-09 +categories: Algorithm +--- + +# 🚀 数据结构与算法开篇 + +![数据结构与算法](https://images.unsplash.com/photo-1518186233392-c232efbf2373?w=800&h=400&fit=crop&crop=center) + +> 💡 **关于怎么刷题的帖子**： +> +> - 📖 《论如何4个月高效刷满 500 题并形成长期记忆》 https://leetcode-cn.com/circle/discuss/jq9Zke/ + +--- + +## 🎯 专栏介绍 + +本专栏致力于为Java开发者提供全面的数据结构与算法学习资源，涵盖从基础概念到高级应用的完整知识体系。通过系统性的学习和实践，帮助开发者提升编程能力，掌握解决复杂问题的核心技能。 + +### 📚 专栏特色 + +- 🎯 **系统性学习**：从基础到进阶，循序渐进 +- 💻 **Java实现**：所有代码示例均使用Java语言 +- 🔥 **实战导向**：结合LeetCode经典题目 +- 📊 **可视化理解**：丰富的图表和动画演示 +- 🎨 **美观排版**：精心设计的文档格式 + +--- + +## 📊 第一部分：数据结构分类 + +数据结构是计算机存储、组织数据的方式，是算法的基础。掌握各种数据结构的特点和应用场景，是成为优秀程序员的必经之路。 + +### 📏 线性数据结构 + +线性数据结构是指数据元素之间存在一对一关系的数据结构，元素按线性顺序排列。 + +#### 📋 1. 数组 (Array) +``` +数组结构示意图： +Index: [0] [1] [2] [3] [4] +Value: [12][45][78][23][56] + ↑ ↑ ↑ ↑ ↑ + 连续的内存地址空间 +``` +- **特点**：连续存储，支持随机访问 +- **时间复杂度**：访问O(1)，插入/删除O(n) +- **Java实现**：`int[]`、`ArrayList` + +#### 🔗 2. 链表 (Linked List) +``` +单链表结构示意图： +[Data|Next] -> [Data|Next] -> [Data|Next] -> null + | | | + 节点1 节点2 节点3 + +双链表结构示意图： +null <- [Prev|Data|Next] <-> [Prev|Data|Next] <-> [Prev|Data|Next] -> null +``` +- **特点**：动态存储，插入删除高效 +- **时间复杂度**：访问O(n)，插入/删除O(1) +- **Java实现**：`LinkedList` + +#### 📚 3. 栈 (Stack) +``` +栈结构示意图： + | | <- top (栈顶) + | C | + | B | + | A | + |_____| <- bottom (栈底) + +操作：LIFO (后进先出) +``` +- **特点**：后进先出(LIFO) +- **时间复杂度**：push/pop/peek都是O(1) +- **Java实现**：`Stack`、`Deque` + +#### 🚶 4. 队列 (Queue) +``` +队列结构示意图： +出队 <- [A][B][C][D] <- 入队 + front rear + +操作：FIFO (先进先出) +``` +- **特点**：先进先出(FIFO) +- **时间复杂度**：enqueue/dequeue都是O(1) +- **Java实现**：`Queue`、`LinkedList` + +### 🌳 非线性数据结构 + +非线性数据结构是指数据元素之间存在一对多或多对多关系的数据结构，形成复杂的层次或网状结构。 + +#### 🌲 5. 二叉树 (Binary Tree) +``` +二叉树结构示意图： + A (根节点) + / \ + B C + / \ / \ + D E F G (叶子节点) + +层次： +第0层： A +第1层： B C +第2层： D E F G +``` +- **特点**：每个节点最多有两个子节点 +- **时间复杂度**：搜索/插入/删除O(logn)~O(n) +- **Java实现**：`TreeMap`、`TreeSet` + +#### 🏔️ 6. 堆 (Heap) +``` +最大堆示意图： + 90 + / \ + 80 70 + / \ / \ + 60 50 40 30 + +特点：父节点 >= 子节点 (最大堆) +数组表示：[90,80,70,60,50,40,30] +``` +- **特点**：完全二叉树，堆序性质 +- **时间复杂度**：插入/删除O(logn)，查找最值O(1) +- **Java实现**：`PriorityQueue` + +#### 🕸️ 7. 图 (Graph) +``` +无向图示意图： + A --- B + /| |\ + / | | \ + D | | C + | | / + E --- F + +邻接矩阵表示： + A B C D E F +A [ 0 1 0 1 1 0 ] +B [ 1 0 1 0 0 1 ] +C [ 0 1 0 0 0 1 ] +D [ 1 0 0 0 1 0 ] +E [ 1 0 0 1 0 1 ] +F [ 0 1 1 0 1 0 ] +``` +- **特点**：顶点和边的集合，表示复杂关系 +- **时间复杂度**：遍历O(V+E)，最短路径O(V²) +- **Java实现**：`Map>` + +#### 🗂️ 8. 哈希表 (Hash Table) +``` +哈希表结构示意图： +Hash函数：key → hash(key) % size → index + +Key: "apple" → hash("apple") = 5 → index: 5 % 8 = 5 +Key: "banana" → hash("banana")= 3 → index: 3 % 8 = 3 + +Table: +[0] → null +[1] → null +[2] → null +[3] → "banana" → value +[4] → null +[5] → "apple" → value +[6] → null +[7] → null + +冲突处理（链地址法）： +[3] → ["banana",value1] → ["grape",value2] → null +``` +- **特点**：基于键值对，快速查找 +- **时间复杂度**：平均O(1)，最坏O(n) +- **Java实现**：`HashMap`、`HashSet` + +--- + +## ⚡ 第二部分：常见算法 + +算法是解决问题的步骤和方法，是程序设计的核心。掌握各种算法的思想和实现，能够帮助我们高效地解决各种复杂问题。 + +### 🔢 1. 排序算法 + +排序算法是计算机科学中最基础也是最重要的算法之一，掌握各种排序算法的特点和应用场景至关重要。 + +| 算法 | 时间复杂度（平均） | 时间复杂度（最坏） | 空间复杂度 | 稳定性 | +|------|-------------------|-------------------|------------|--------| +| 冒泡排序 | O(n²) | O(n²) | O(1) | 稳定 | +| 选择排序 | O(n²) | O(n²) | O(1) | 不稳定 | +| 插入排序 | O(n²) | O(n²) | O(1) | 稳定 | +| 快速排序 | O(nlogn) | O(n²) | O(logn) | 不稳定 | +| 归并排序 | O(nlogn) | O(nlogn) | O(n) | 稳定 | +| 堆排序 | O(nlogn) | O(nlogn) | O(1) | 不稳定 | +| 计数排序 | O(n+k) | O(n+k) | O(k) | 稳定 | +| 基数排序 | O(nk) | O(nk) | O(n+k) | 稳定 | + +### 🔍 2. 搜索算法 + +搜索算法用于在数据集合中查找特定元素，不同的搜索算法适用于不同的场景。 + +- **线性搜索**：O(n) - 适用于无序数组 +- **二分搜索**：O(logn) - 适用于有序数组 +- **哈希查找**：O(1) - 适用于HashMap/HashSet +- **树搜索**：O(logn) - 适用于二叉搜索树 +- **图搜索**：DFS/BFS - O(V+E) + +### 🕸️ 3. 图算法 + +图算法是处理图结构数据的重要工具，广泛应用于网络分析、路径规划等领域。 + +#### 🔍 3.1 图遍历 +- **深度优先搜索(DFS)**：O(V+E) +- **广度优先搜索(BFS)**：O(V+E) + +#### 🛣️ 3.2 最短路径 +- **Dijkstra算法**：O((V+E)logV) - 单源最短路径 +- **Floyd-Warshall算法**：O(V³) - 全源最短路径 +- **Bellman-Ford算法**：O(VE) - 含负权边 + +#### 🌲 3.3 最小生成树 +- **Prim算法**：O(ElogV) +- **Kruskal算法**：O(ElogE) + +#### 📋 3.4 拓扑排序 +- **Kahn算法**：O(V+E) +- **DFS算法**：O(V+E) + +### 🎯 4. 动态规划 + +动态规划是一种通过把原问题分解为相对简单的子问题的方式求解复杂问题的方法。 + +#### 🏗️ 4.1 基础DP +- **斐波那契数列**：O(n) +- **爬楼梯**：O(n) +- **最大子序和**：O(n) + +#### 📝 4.2 序列DP +- **最长递增子序列(LIS)**：O(nlogn) +- **最长公共子序列(LCS)**：O(mn) +- **编辑距离**：O(mn) + +#### 🎒 4.3 背包问题 +- **0-1背包**：O(nW) +- **完全背包**：O(nW) +- **多重背包**：O(nWlogM) + +#### 📏 4.4 区间DP +- **最长回文子串**：O(n²) +- **矩阵链乘法**：O(n³) + +### 🎯 5. 贪心算法 + +贪心算法是一种在每一步选择中都采取在当前状态下最好或最优的选择，从而希望导致结果是最好或最优的算法。 + +- **活动选择问题**：O(nlogn) +- **分数背包**：O(nlogn) +- **最小生成树(Prim/Kruskal)**：O(ElogV) +- **霍夫曼编码**：O(nlogn) +- **区间调度**：O(nlogn) + +### 🔄 6. 分治算法 + +分治算法是一种很重要的算法，字面上的解释是"分而治之"，就是把一个复杂的问题分成两个或更多的相同或相似的子问题。 + +- **归并排序**：O(nlogn) +- **快速排序**：O(nlogn) +- **二分搜索**：O(logn) +- **最大子数组和**：O(nlogn) +- **最近点对**：O(nlogn) + +### 🔙 7. 回溯算法 + +回溯算法是一种通过穷举所有可能情况来找到所有解的算法。当探索到某一步时，发现原先选择并不优或达不到目标，就退回一步重新选择。 + +- **N皇后问题**：O(N!) +- **数独求解**：O(9^(n*n)) +- **全排列**：O(n!) +- **子集生成**：O(2^n) +- **组合问题**：O(C(n,k)) + +### 📝 8. 字符串算法 + +字符串算法是处理文本数据的重要工具，广泛应用于文本搜索、模式匹配等领域。 + +- **KMP算法**：O(n+m) - 字符串匹配 +- **Rabin-Karp算法**：O(n+m) - 字符串匹配 +- **最长公共前缀**：O(S) - S为所有字符串长度和 +- **字典树(Trie)**：插入/查找 O(m) +- **后缀数组**：O(nlogn) + +### 🧮 9. 数学算法 + +数学算法是解决数学问题的计算方法，在编程中经常需要用到各种数学算法。 + +- **最大公约数(GCD)**：O(logn) +- **快速幂**：O(logn) +- **素数筛选**：O(nloglogn) +- **模运算**：O(1) +- **组合数学**：O(nlogn) + +### 🔢 10. 位运算算法 + +位运算是计算机中最底层的运算，掌握位运算技巧可以写出更高效的代码。 + +- **位运算基础**：O(1) +- **状态压缩DP**：O(n*2^m) +- **子集枚举**：O(2^n) +- **位操作技巧**：O(1) + +### 🏗️ 11. 高级数据结构算法 + +高级数据结构算法是建立在基础数据结构之上的复杂算法，能够解决更复杂的问题。 + +- **并查集**：O(α(n)) - 接近常数时间 +- **线段树**：O(logn) - 区间查询/更新 +- **树状数组**：O(logn) - 前缀和查询 +- **平衡树(AVL/红黑树)**：O(logn) +- **跳表**：O(logn) + +--- + +## 🎯 第三部分：LeetCode经典题目 + +LeetCode是程序员刷题的重要平台，通过系统性的刷题练习，可以快速提升算法能力。以下是按类型分类的经典题目。 + +### 📋 1. 数组类题目 + +数组是最基础的数据结构，掌握数组的各种操作技巧是算法学习的基础。 + +#### 🔧 基础操作 +- **1. 两数之和** - 哈希表优化 +- **26. 删除排序数组中的重复项** - 双指针 +- **27. 移除元素** - 双指针 +- **88. 合并两个有序数组** - 双指针 + +#### 🔍 搜索与查找 +- **33. 搜索旋转排序数组** - 二分搜索 +- **34. 在排序数组中查找元素的第一个和最后一个位置** - 二分搜索 +- **35. 搜索插入位置** - 二分搜索 +- **153. 寻找旋转排序数组中的最小值** - 二分搜索 + +#### 👆 双指针技巧 +- **15. 三数之和** - 排序+双指针 +- **16. 最接近的三数之和** - 排序+双指针 +- **18. 四数之和** - 排序+双指针 +- **42. 接雨水** - 双指针 +- **11. 盛最多水的容器** - 双指针 + +#### 🪟 滑动窗口 +- **3. 无重复字符的最长子串** - 滑动窗口 +- **76. 最小覆盖子串** - 滑动窗口 +- **209. 长度最小的子数组** - 滑动窗口 +- **438. 找到字符串中所有字母异位词** - 滑动窗口 + +### 🔗 2. 链表类题目 + +链表是动态数据结构，掌握链表的操作技巧对于理解指针和递归非常重要。 + +#### 🔧 基础操作 +- **206. 反转链表** - 迭代/递归 +- **21. 合并两个有序链表** - 双指针 +- **83. 删除排序链表中的重复元素** - 单指针 +- **82. 删除排序链表中的重复元素 II** - 双指针 + +#### 👆 双指针技巧 +- **141. 环形链表** - 快慢指针 +- **142. 环形链表 II** - 快慢指针 +- **160. 相交链表** - 双指针 +- **19. 删除链表的倒数第 N 个结点** - 快慢指针 + +#### 🔄 复杂操作 +- **234. 回文链表** - 快慢指针+反转 +- **143. 重排链表** - 找中点+反转+合并 +- **148. 排序链表** - 归并排序 +- **23. 合并K个排序链表** - 分治/优先队列 + +### 📚 3. 栈与队列类题目 + +栈和队列是重要的线性数据结构，在算法中有着广泛的应用。 + +#### 📚 栈的应用 +- **20. 有效的括号** - 栈匹配 +- **155. 最小栈** - 辅助栈 +- **225. 用队列实现栈** - 设计问题 +- **232. 用栈实现队列** - 设计问题 + +#### 📈 单调栈 +- **496. 下一个更大元素 I** - 单调栈 +- **503. 下一个更大元素 II** - 单调栈 +- **739. 每日温度** - 单调栈 +- **84. 柱状图中最大的矩形** - 单调栈 + +### 4. 树类题目 + +#### 二叉树遍历 +- **94. 二叉树的中序遍历** - 递归/迭代 +- **144. 二叉树的前序遍历** - 递归/迭代 +- **145. 二叉树的后序遍历** - 递归/迭代 +- **102. 二叉树的层序遍历** - BFS + +#### 二叉树性质 +- **104. 二叉树的最大深度** - DFS +- **111. 二叉树的最小深度** - BFS/DFS +- **226. 翻转二叉树** - DFS +- **101. 对称二叉树** - DFS + +#### 二叉搜索树 +- **98. 验证二叉搜索树** - 中序遍历 +- **700. 二叉搜索树中的搜索** - 二分搜索 +- **701. 二叉搜索树中的插入操作** - 递归 +- **450. 删除二叉搜索树中的节点** - 递归 + +#### 树的路径问题 +- **112. 路径总和** - DFS +- **113. 路径总和 II** - DFS+回溯 +- **124. 二叉树中的最大路径和** - DFS +- **257. 二叉树的所有路径** - DFS+回溯 + +### 5. 图类题目 + +#### 图的遍历 +- **200. 岛屿数量** - DFS/BFS +- **695. 岛屿的最大面积** - DFS/BFS +- **994. 腐烂的橘子** - BFS +- **130. 被围绕的区域** - DFS/BFS + +#### 拓扑排序 +- **207. 课程表** - 拓扑排序 +- **210. 课程表 II** - 拓扑排序 + +#### 联通分量 +- **547. 省份数量** - 并查集/DFS +- **684. 冗余连接** - 并查集 +- **721. 账户合并** - 并查集 + +### 6. 动态规划类题目 + +#### 基础DP +- **70. 爬楼梯** - 基础DP +- **198. 打家劫舍** - 线性DP +- **213. 打家劫舍 II** - 环形DP +- **53. 最大子序和** - Kadane算法 + +#### 序列DP +- **300. 最长递增子序列** - LIS +- **1143. 最长公共子序列** - LCS +- **72. 编辑距离** - 字符串DP +- **5. 最长回文子串** - 区间DP + +#### 背包问题 +- **416. 分割等和子集** - 0-1背包 +- **494. 目标和** - 0-1背包 +- **322. 零钱兑换** - 完全背包 +- **518. 零钱兑换 II** - 完全背包 + +#### 状态机型DP +- **121. 买卖股票的最佳时机** - 状态机DP +- **122. 买卖股票的最佳时机 II** - 状态机DP +- **123. 买卖股票的最佳时机 III** - 状态机DP +- **188. 买卖股票的最佳时机 IV** - 状态机DP + +### 7. 回溯算法类题目 + +#### 排列组合 +- **46. 全排列** - 回溯 +- **47. 全排列 II** - 回溯+去重 +- **77. 组合** - 回溯 +- **78. 子集** - 回溯 + +#### 分割问题 +- **131. 分割回文串** - 回溯+动态规划 +- **93. 复原IP地址** - 回溯 + +#### 棋盘问题 +- **51. N 皇后** - 回溯 +- **37. 解数独** - 回溯 + +### 8. 贪心算法类题目 + +#### 区间问题 +- **435. 无重叠区间** - 贪心 +- **452. 用最少数量的箭引爆气球** - 贪心 +- **55. 跳跃游戏** - 贪心 +- **45. 跳跃游戏 II** - 贪心 + +### 9. 字符串类题目 + +#### 字符串匹配 +- **28. 实现 strStr()** - KMP算法 +- **459. 重复的子字符串** - KMP算法 + +#### 回文串 +- **125. 验证回文串** - 双指针 +- **5. 最长回文子串** - 中心扩展 +- **647. 回文子串** - 中心扩展 + +### 10. 位运算类题目 + +- **136. 只出现一次的数字** - 异或运算 +- **191. 位1的个数** - 位运算 +- **338. 比特位计数** - 位运算+DP +- **461. 汉明距离** - 位运算 + + + +### 🎯 刷题建议 +1. **由易到难**：从Easy开始，逐步提升到Hard +2. **分类练习**：按数据结构和算法类型分类刷题 +3. **反复练习**：重要题目要反复练习，形成肌肉记忆 +4. **总结模板**：每种题型都要总结解题模板 +5. **时间管理**：每天固定时间刷题，保持连续性 +6. **记录总结**：建立错题本，定期回顾总结 + +--- + +## 🛠️ 学习工具推荐 + +### 📚 在线平台 +- [LeetCode中国](https://leetcode-cn.com/) - 主要刷题平台 +- [牛客网](https://www.nowcoder.com/) - 面试真题 +- [AcWing](https://www.acwing.com/) - 算法竞赛 +- [洛谷](https://www.luogu.com.cn/) - 算法学习 + +### 🎥 学习资源 +- [算法可视化](https://visualgo.net/) - 算法动画演示 +- [VisuAlgo](https://visualgo.net/) - 数据结构可视化 +- [算法导论](https://mitpress.mit.edu/books/introduction-algorithms) - 经典教材 +- [算法4](https://algs4.cs.princeton.edu/) - Java实现 + +### 💻 开发工具 +- **IDE**：IntelliJ IDEA、Eclipse +- **调试**：LeetCode插件、本地调试 +- **版本控制**：Git管理代码 +- **笔记**：Markdown记录学习心得 + +--- + +> 💡 **记住**：数据结构和算法是程序员的内功，需要持续练习和积累 +> +> 🚀 **建议**：每天至少刷一道LeetCode，坚持100天必有收获 +> +> 📚 **资源**：[LeetCode中国](https://leetcode-cn.com/) | [算法可视化](https://visualgo.net/) +> +> 🎉 **加油**：相信通过系统性的学习，你一定能够掌握数据结构和算法的精髓！ \ No newline at end of file diff --git a/docs/data-structure-algorithms/algorithm/Backtracking.md b/docs/data-structure-algorithms/algorithm/Backtracking.md new file mode 100755 index 0000000000..bdce9eaa4f --- /dev/null +++ b/docs/data-structure-algorithms/algorithm/Backtracking.md @@ -0,0 +1,834 @@ +--- +title: 回溯算法 +date: 2023-05-09 +tags: + - back tracking +categories: Algorithm +--- + +![](https://img.starfish.ink/leetcode/backtracking-banner.png) + +> 🔍 **回溯算法**是解决很多算法问题的常见思想，它也是传统的人工智能的方法，其本质是 **在树形问题中寻找解** 。 +> +> 回溯算法实际上是一个类似枚举的搜索尝试过程，主要是在**搜索尝试**过程中寻找问题的解，当发现已不满足求解条件时，就"**回溯**"返回，尝试别的路径。所以也可以叫做**回溯搜索法**。 +> +> 💡 回溯是递归的副产品，只要有递归就会有回溯。 + +# 一、回溯算法 + +回溯算法是一种**深度优先搜索**（DFS）的算法，它通过递归的方式，逐步建立解空间树，从根节点开始，逐层深入，直到找到一个解或路径不可行时回退到上一个状态（即回溯）。每一步的尝试可能会有多个选择，回溯算法通过剪枝来减少不必要的计算。 + +> 🌲 **深度优先搜索** 算法（英语：Depth-First-Search，DFS）是一种用于遍历或搜索树或图的算法。这个算法会 **尽可能深** 的搜索树的分支。当结点 `v` 的所在边都己被探寻过，搜索将 **回溯** 到发现结点 `v` 的那条边的起始结点。这一过程一直进行到已发现从源结点可达的所有结点为止。如果还存在未被发现的结点，则选择其中一个作为源结点并重复以上过程，整个进程反复进行直到所有结点都被访问为止。 + +回溯的基本步骤通常包含以下几个要素： + +- 🎯 **选择**：在当前状态下，做出一个选择，进入下一个状态。 +- ⚖️ **约束**：每一步的选择必须满足问题的约束条件。 +- 🎪 **目标**：找到一个解或判断是否无法继续。 +- ↩️ **回溯**：如果当前的选择不符合目标，撤销当前的选择，回到上一状态继续尝试其他可能的选择。 + +### 🧠 核心思想 + +**回溯法** 采用试错的思想，它尝试分步的去解决一个问题。 + +1. 🔍 **穷举所有可能的解**：回溯法通过在每个状态下尝试不同的选择，来遍历解空间树。每个分支代表着做出的一个选择，每次递归都尝试不同的路径，直到找到一个解或回到根节点。 +2. ✂️ **剪枝**：在回溯过程中，我们可能会遇到一些不符合约束条件的选择，这时候可以及时退出当前分支，避免无谓的计算，这被称为"剪枝"。剪枝是提高回溯算法效率的关键，能减少不必要的计算。 +3. 🌲 **深度优先搜索（DFS）**：回溯算法在解空间树中深度优先遍历，尝试选择每个分支。直到走到树的叶子节点或回溯到一个不满足条件的节点。 + + + +### 🏗️ 基本框架 + +回溯算法的基本框架可以用递归来实现，通常包含以下几个步骤： + +1. 🎯 **选择和扩展**：选择一个可行的扩展步骤，扩展当前的解。 +2. ✅ **约束检查**：检查当前扩展后的解是否满足问题的约束条件。 +3. 🔄 **递归调用**：如果当前解满足约束条件，则递归地尝试扩展该解。 +4. ↩️ **回溯**：如果当前解不满足约束条件，或所有扩展步骤都已经尝试，则回溯到上一步，尝试其他可能的扩展步骤。 + +以下是回溯算法的一般伪代码： + +```java +result = [] +function backtrack(solution, candidates): //入参可以理解为 路径, 选择列表 + if solution 是一个完整解: //满足结束条件 + result.add(solution) // 处理当前完整解 + return + for candidate in candidates: + if candidate 满足约束条件: + solution.add(candidate) // 扩展解 + backtrack(solution, new_candidates) // 递归调用 + solution.remove(candidate) // 回溯,撤销选择 + +``` + +对应到 java 的一般框架如下： + +```java +public void backtrack(List tempList, int start, int[] nums) { + // 1. 终止条件 + if (tempList.size() == nums.length) { + // 找到一个解 + result.add(new ArrayList<>(tempList)); + return; + } + + for (int i = start; i < nums.length; i++) { + // 2. 剪枝：跳过相同的数字，避免重复 + if (i > start && nums[i] == nums[i - 1]) { + continue; + } + + // 3. 做出选择 + tempList.add(nums[i]); + + // 4. 递归 + backtrack(tempList, i + 1, nums); + + // 5. 撤销选择 + tempList.remove(tempList.size() - 1); + } +} +``` + +**其实就是 for 循环里面的递归，在递归调用之前「做选择」，在递归调用之后「撤销选择」** + +> 💡 **关键理解**：回溯算法 = 递归 + 选择 + 撤销选择 + + + +### 🎯 常见题型 + +回溯法，一般可以解决如下几种问题： + +- 🎲 **组合问题**：N个数里面按一定规则找出k个数的集合 + + - **题目示例**：`LeetCode 39. 组合总和`，`LeetCode 40. 组合总和 II`，`LeetCode 77. 组合` + - **解题思路**： 组合问题要求我们在给定的数组中选取若干个数字，组合成目标值或某种形式的子集。回溯算法的基本思路是从一个起点开始，选择当前数字或者跳过当前数字，直到找到一个合法的组合。组合问题通常有**去重**的要求，避免重复的组合。 + +- 🔄 **排列问题**：N个数按一定规则全排列，有几种排列方式 + + - **题目示例**：`LeetCode 46. 全排列`，`LeetCode 47. 全排列 II`，`LeetCode 31. 下一个排列` + - **解题思路**： 排列问题要求我们通过给定的数字生成所有可能的排列。回溯算法通过递归生成所有排列，通过交换位置来改变元素的顺序。对于全排列 II 这类题目，必须处理重复元素的问题，确保生成的排列不重复。 + +- 📦 **子集问题**：一个N个数的集合里有多少符合条件的子集 + + - **题目示例**：`LeetCode 78. 子集`，`LeetCode 90. 子集 II` + - **解题思路**： 子集问题要求我们生成数组的所有子集，回溯算法通过递归生成每个可能的子集。在生成子集时，每个元素有两种选择——要么包含它，要么不包含它。因此，回溯法通过逐步选择来生成所有的子集。 + +- ✂️ **切割问题**：一个字符串按一定规则有几种切割方式 + + - 题目实例：` LeetCode 416. Partition Equal Subset Sum`，`LeetCode 698. Partition to K Equal Sum Subsets` + - 解题思路：回溯法适用于切割问题中的"探索所有可能的分割方式"的场景。特别是在无法直接通过动态规划推导出最优解时，回溯法通过递归尝试所有可能的分割方式，并通过剪枝减少不必要的计算 + +- ♟️ **棋盘问题**： + + - **题目示例**：`LeetCode 37. 解数独`，`LeetCode 51. N 皇后`，`LeetCode 52. N 皇后 II` + + **解题思路**： 棋盘问题常常涉及到在二维数组中进行回溯搜索，比如在数独中填入数字，或者在 N 皇后问题中放置皇后。回溯法在这里用于逐步尝试每个位置，满足棋盘的约束条件，直到找到一个解或者回溯到一个合法的状态。 + +- 🗺️ **图的遍历问题**： + + - **题目示例**：`LeetCode 79. 单词搜索`，`LeetCode 130. 被围绕的区域` + - **解题思路**： 回溯算法在图遍历中的应用主要是通过递归搜索路径。常见的问题是从某个起点出发，寻找是否存在某个目标路径。通过回溯算法，可以逐步尝试每一个可能的路径，直到找到符合条件的解。 + + + + +### ⚡ 回溯算法的优化技巧 + +1. ✂️ **剪枝**：在递归过程中，如果当前路径不符合问题约束，就提前返回，避免继续深入。例如在排列问题中，遇到重复的数字时可以跳过该分支。 +2. 📊 **排序**：对输入数据进行排序，有助于我们在递归时判断是否可以剪枝，尤其是在去重的场景下。 +3. 🗜️ **状态压缩**：在一些问题中，使用位运算或其他方式对状态进行压缩，可以显著减少存储空间和计算时间。例如在解决旅行商问题时，常常使用状态压缩来存储已经访问的节点。 +4. 🛑 **提前终止**：如果在递归过程中发现某条路径不可能达到目标（例如目标已经超过了剩余可用值），可以直接结束该分支，节省时间。 + + + +# 二、热门面试题 + +## 🎲 排列、组合类 + +> 无论是排列、组合还是子集问题，简单说无非就是让你从序列 `nums` 中以给定规则取若干元素，主要有以下几种变体： +> +> **元素无重不可复选，即 `nums` 中的元素都是唯一的，每个元素最多只能被使用一次，这也是最基本的形式**。 +> +> - 以组合为例，如果输入 `nums = [2,3,6,7]`，和为 7 的组合应该只有 `[7]`。 +> +> **元素可重不可复选，即 `nums` 中的元素可以存在重复，每个元素最多只能被使用一次**。 +> +> - 以组合为例，如果输入 `nums = [2,5,2,1,2]`，和为 7 的组合应该有两种 `[2,2,2,1]` 和 `[5,2]`。 +> +> **元素无重可复选，即 `nums` 中的元素都是唯一的，每个元素可以被使用若干次**。 +> +> - 以组合为例，如果输入 `nums = [2,3,6,7]`，和为 7 的组合应该有两种 `[2,2,3]` 和 `[7]`。 +> +> 当然，也可以说有第四种形式，即元素可重可复选。但既然元素可复选，那又何必存在重复元素呢？元素去重之后就等同于形式三，所以这种情况不用考虑。 +> +> 上面用组合问题举的例子，但排列、组合、子集问题都可以有这三种基本形式，所以共有 9 种变化。 +> +> 除此之外，题目也可以再添加各种限制条件，比如让你求和为 `target` 且元素个数为 `k` 的组合，那这么一来又可以衍生出一堆变体，怪不得面试笔试中经常考到排列组合这种基本题型。 +> +> **但无论形式怎么变化，其本质就是穷举所有解，而这些解呈现树形结构，所以合理使用回溯算法框架，稍改代码框架即可把这些问题一网打尽**。 + + + +### 🎯 一、元素无重不可复选 + +#### 📦 [子集_78](https://leetcode.cn/problems/subsets/) + +> 给你一个整数数组 `nums` ，数组中的元素 **互不相同** 。返回该数组所有可能的子集（幂集）。 +> +> 解集 **不能** 包含重复的子集。你可以按 **任意顺序** 返回解集。 +> +> ``` +> 输入：nums = [1,2,3] +> 输出：[[],[1],[2],[1,2],[3],[1,3],[2,3],[1,2,3]] +> ``` + +**💡 思路**： + +**子集的特性**： + +- 对于给定的数组 `[1, 2, 3]`，它的所有子集应该包括空集、单个元素的子集、两个元素的组合和完整数组。 +- 每个元素都有两种选择：要么加入子集，要么不加入子集。 + +**回溯算法**： + +- 使用回溯的方式可以从空集开始，逐步添加元素来生成所有子集。 +- 从当前的元素出发，尝试包含它或者不包含它，然后递归地处理下一个元素。 + +参数定义： + +- `res`：一个列表，存储最终的所有子集，类型是 `List>`。 + +- `track`：一个临时列表，记录当前路径（即当前递归中形成的子集）。 +- `start`：当前递归要开始的位置（即考虑从哪个位置开始生成子集）。这个 `start` 是非常重要的，它确保了我们在递归时不会重复生成相同的子集。 + +完成回溯树的遍历就收集了所有子集。 + +![](https://img.starfish.ink/leetcode/leetcode-78.png) + +```java +public List> subsets(int[] nums) { + List> res = new ArrayList<>(); + // 记录回溯算法的递归路径 + List track = new ArrayList<>(); + + if (nums.length == 0) { + return res; + } + + backtrack(nums, 0, res, track); + return res; +} + +private void backtrack(int[] nums, int start, List> res, List track) { + + res.add(new ArrayList<>(track)); + + // 回溯算法标准框架 + for (int i = start; i < nums.length; i++) { + // 做选择 + track.add(nums[i]); + // 通过 start 参数控制树枝的遍历，避免产生重复的子集 + backtrack(nums, i + 1, res, track); + // 撤销选择 + track.remove(track.size() - 1); + } +} +``` + + + +#### 🎲 [组合_77](https://leetcode.cn/problems/combinations/) + +> 给定两个整数 `n` 和 `k`，返回范围 `[1, n]` 中所有可能的 `k` 个数的组合。你可以按 **任何顺序** 返回答案。 +> +> ``` +> 输入：n = 4, k = 2 +> 输出： +> [ +> [2,4], +> [3,4], +> [2,3], +> [1,2], +> [1,3], +> [1,4], +> ] +> ``` + +**💡 思路**：翻译一下就变成子集问题了：**给你输入一个数组 `nums = [1,2..,n]` 和一个正整数 `k`，请你生成所有大小为 `k` 的子集**。 + +![](https://img.starfish.ink/leetcode/leetcode-77.png) + +反映到代码上，只需要稍改 base case，控制算法仅仅收集第 `k` 层节点的值即可： + +```java +public List> combine(int n, int k) { + List> res = new ArrayList<>(); + // 记录回溯算法的递归路径 + List track = new ArrayList<>(); + // start 从 1 开始即可 + backtrack(n, k, 1, track, res); + return res; +} + +private void backtrack(int n, int k, int start, List track, List> res) { + // 遍历到了第 k 层，收集当前节点的值 + if (track.size() == k) { + res.add(new ArrayList<>(track)); // 深拷贝 + return; + } + + // 从当前数字开始尝试 + for (int i = start; i <= n; i++) { + track.add(i); + // 通过 start 参数控制树枝的遍历，避免产生重复的子集 + backtrack(n, k, i + 1, track, res); // 递归选下一个数字 + track.remove(track.size() - 1); // 撤销选择 + } +} +``` + + + +#### 🔄 [全排列_46](https://leetcode.cn/problems/permutations/description/) + +> 给定一个不含重复数字的数组 `nums` ，返回其 *所有可能的全排列* 。你可以 **按任意顺序** 返回答案。 +> +> ``` +> 输入：nums = [1,2,3] +> 输出：[[1,2,3],[1,3,2],[2,1,3],[2,3,1],[3,1,2],[3,2,1]] +> ``` + +**💡 思路**：组合/子集问题使用 `start` 变量保证元素 `nums[start]` 之后只会出现 `nums[start+1..]`中的元素，通过固定元素的相对位置保证不出现重复的子集。 + +**但排列问题本身就是让你穷举元素的位置，`nums[i]` 之后也可以出现 `nums[i]` 左边的元素，所以之前的那一套玩不转了，需要额外使用 `used` 数组来标记哪些元素还可以被选择**。 + +全排列共有 `n!` 个，我们可以按阶乘举例的思想，画出「回溯树」 + +![](https://img.starfish.ink/leetcode/leetcode-46.png) + +> 回溯树是一种树状结构，树的每个节点表示一个状态（即当前的选择或部分解），树的每条边表示一次决策的选择。在回溯过程中，我们从根节点开始，递归地选择下一个数字，每次递归都相当于进入树的下一层。 + +> **labuladong 称为 决策树，你在每个节点上其实都在做决策**。因为比如你选了 2 之后，只能再选 1 或者 3，全排列是不允许重复使用数字的。**`[2]` 就是「路径」，记录你已经做过的选择；`[1,3]` 就是「选择列表」，表示你当前可以做出的选择；「结束条件」就是遍历到树的底层叶子节点，这里也就是选择列表为空的时候**。 + +```java +public class Solution { + public List> permute(int[] nums) { + List> res = new ArrayList<>(); + // 记录「路径」 + List track = new ArrayList<>(); + boolean[] used = new boolean[nums.length]; // 标记数字是否被使用过 + backtrack(nums, used, track, res); + return res; + } + + private void backtrack(int[] nums, boolean[] used, List track, List> res) { + // 当排列的大小达到nums.length时，说明当前排列完成 + if (track.size() == nums.length) { + res.add(new ArrayList<>(track)); // 将当前排列加入结果 + return; + } + + // 尝试每一个数字 + for (int i = 0; i < nums.length; i++) { + if (used[i]) continue; // 如果当前数字已经被使用过，则跳过，剪枝操作 + + // 做选择 + track.add(nums[i]); + used[i] = true; // 标记当前数字为已使用 + + // 递归进入下一层 + backtrack(nums, used, track, res); + + // 撤销选择 + track.remove(track.size() - 1); + used[i] = false; // 回溯时将当前数字标记为未使用 + } + } +} +``` + + + +### 🔄 二、元素可重不可复选 + +#### 📦 [子集 II_90](https://leetcode.cn/problems/subsets-ii/) + +> 给你一个整数数组 `nums` ，其中可能包含重复元素，请你返回该数组所有可能的 子集（幂集）。 +> +> 解集 **不能** 包含重复的子集。返回的解集中，子集可以按 **任意顺序** 排列。 +> +> ``` +> 输入：nums = [1,2,2] +> 输出：[[],[1],[1,2],[1,2,2],[2],[2,2]] +> ``` + +**💡 思路**：该问题的关键是**去重**（剪枝），**体现在代码上，需要先进行排序，让相同的元素靠在一起，如果发现 `nums[i] == nums[i-1]`，则跳过** + +> LeetCode 78 **Subsets** 问题并没有重复的子集。我们生成的是所有可能的子集，并且不需要考虑去除重复的子集，因为给定的数组 `nums` 不含重复元素。而在 **Subsets II** 中，由于输入数组可能包含重复元素，所以我们需要特殊处理来避免生成重复的子集。 + +```java +public class Solution { + public List> subsetsWithDup(int[] nums) { + List> res = new ArrayList<>(); + Arrays.sort(nums); // 排序，确保相同的元素相邻 + backtrack(nums, 0, new ArrayList<>(), res); + return res; + } + + private void backtrack(int[] nums, int start, List track, List> res) { + // 每次递归时，将当前的track添加到结果中 + res.add(new ArrayList<>(track)); + + // 从start位置开始遍历 + for (int i = start; i < nums.length; i++) { + // 如果当前元素与前一个元素相同，并且前一个元素没有被选择，跳过当前元素 + if (i > start && nums[i] == nums[i - 1]) { + continue; // 剪枝 + } + + // 做选择 + track.add(nums[i]); + // 递归进入下一层 + backtrack(nums, i + 1, track, res); + // 撤销选择 + track.remove(track.size() - 1); + } + } +} +``` + + + +#### 🎯 [组合总和 II_40](https://leetcode.cn/problems/combination-sum-ii/) + +> 给定一个候选人编号的集合 `candidates` 和一个目标数 `target` ，找出 `candidates` 中所有可以使数字和为 `target` 的组合。 +> +> `candidates` 中的每个数字在每个组合中只能使用 **一次** 。 +> +> **注意：**解集不能包含重复的组合。 +> +> ``` +> 输入: candidates = [10,1,2,7,6,1,5], target = 8, +> 输出: +> [ +> [1,1,6], +> [1,2,5], +> [1,7], +> [2,6] +> ] +> ``` + +**💡 思路**：说这是一个组合问题，其实换个问法就变成子集问题了：请你计算 `candidates` 中所有和为 `target` 的子集。 + +1. **排序**：首先对 `candidates` 数组进行排序，排序后的数组方便处理重复数字。 +2. **递归选择**：在递归过程中，确保如果当前数字和上一个数字相同，且上一个数字没有被选择过，则跳过当前数字，从而避免重复组合。 +3. **递归终止条件**：如果 `target` 变为 0，表示找到了一个符合条件的组合；如果 `target` 小于 0，表示当前路径不合法，应该回溯。 + +```java +public class Solution { + public List> combinationSum2(int[] candidates, int target) { + List> res = new ArrayList<>(); + Arrays.sort(candidates); // 排序，便于后续去重 + backtrack(candidates, target, 0, new ArrayList<>(), res); + return res; + } + + private void backtrack(int[] candidates, int target, int start, List track, List> res) { + // 当目标值为0时，找到一个符合条件的组合 + if (target == 0) { + res.add(new ArrayList<>(track)); // 复制当前组合并加入结果 + return; + } + + // 遍历候选数组 + for (int i = start; i < candidates.length; i++) { + // 剪枝：当前数字大于目标值，后续不可能有合法的组合 + if (candidates[i] > target) { + break; + } + // 剪枝：跳过重复的数字 + if (i > start && candidates[i] == candidates[i - 1]) { + continue; + } + + // 做选择：选择当前数字 + track.add(candidates[i]); + // 递归，注意i + 1表示下一个位置，确保每个数字只使用一次 + backtrack(candidates, target - candidates[i], i + 1, track, res); + // 撤销选择 + track.remove(track.size() - 1); + } + } +} + +``` + + + +#### 🔄 [全排列 II_47](https://leetcode.cn/problems/permutations-ii/) + +> 给定一个可包含重复数字的序列 `nums` ，***按任意顺序*** 返回所有不重复的全排列。 +> +> ``` +> 输入：nums = [1,1,2] +> 输出： +> [[1,1,2], +> [1,2,1], +> [2,1,1]] +> ``` + +**💡 思路**：典型的回溯 + +1. **排序**：排序的目的是为了能够在回溯时做出剪枝选择。如果两个数字相同，并且前一个数字未被使用过，那么就可以跳过当前数字，避免产生重复的排列 +2. **回溯过程**：`backtrack` 是核心的递归函数。它每次递归时尝试将 `nums` 中的元素添加到 `track` 列表中。当 `track` 的大小等于 `nums` 的长度时，说明我们找到了一个排列，加入到结果 `res` 中。 + - **标记数字是否被使用**：用一个布尔数组 `used[]` 来标记当前数字是否已经在某一层递归中被使用过，避免重复排列。 + - **剪枝条件**：如果当前数字和前一个数字相同，并且前一个数字还没有被使用过，那么跳过当前数字，因为此时使用相同的数字会导致重复的排列。 +3. **递归回溯的选择与撤销**： + - **做选择**：选择当前数字 `nums[i]`，将其标记为已用，并添加到当前排列中。 + - **递归**：继续递归地选择下一个数字，直到 `track` 的大小等于 `nums` 的大小。 + - **撤销选择**：递归回到上一层时，撤销刚刚的选择，将当前数字从 `track` 中移除，并将其标记为未使用。 + +> 剪枝的逻辑： +> +> - **第一轮回溯**：选择第一个 `1`，然后继续递归选择，生成一个排列。 +> +> - **第二轮回溯**：当我们尝试选择第二个 `1` 时，假如第一个 `1` 没有被使用，第二个 `1` 会被选择，这时就会生成一个和第一轮相同的排列。为避免这种情况，我们需要 **剪枝**。 + +```java +public class Solution { + public List> permuteUnique(int[] nums) { + List> res = new ArrayList<>(); + Arrays.sort(nums); // 排序，确保相同元素相邻 + backtrack(nums, new boolean[nums.length], new ArrayList<>(), res); + return res; + } + + private void backtrack(int[] nums, boolean[] used, List track, List> res) { + // 当排列的大小达到nums.length时，找到一个合法排列 + if (track.size() == nums.length) { + res.add(new ArrayList<>(track)); // 加入当前排列 + return; + } + + // 遍历数组，递归生成排列 + for (int i = 0; i < nums.length; i++) { + // 剪枝：当前元素已经被使用过，跳过 + if (used[i]) continue; + // 剪枝：跳过相同的元素，避免生成重复排列 + if (i > 0 && nums[i] == nums[i - 1] && !used[i - 1]) continue; + + // 做选择 + track.add(nums[i]); + used[i] = true; // 标记当前元素已使用 + + // 递归 + backtrack(nums, used, track, res); + + // 撤销选择 + track.remove(track.size() - 1); + used[i] = false; // 标记当前元素未使用 + } + } +} + +``` + + + +### ♻️ 三、元素无重复可复选 + +输入数组无重复元素，但每个元素可以被无限次使用 + +#### 🎯 [组合总和_39](https://leetcode.cn/problems/combination-sum/) + +> 给你一个 **无重复元素** 的整数数组 `candidates` 和一个目标整数 `target` ，找出 `candidates` 中可以使数字和为目标数 `target` 的 所有 **不同组合** ，并以列表形式返回。你可以按 **任意顺序** 返回这些组合。 +> +> `candidates` 中的 **同一个** 数字可以 **无限制重复被选取** 。如果至少一个数字的被选数量不同，则两种组合是不同的。 +> +> 对于给定的输入，保证和为 `target` 的不同组合数少于 `150` 个。 +> +> ``` +> 输入：candidates = [2,3,6,7], target = 7 +> 输出：[[2,2,3],[7]] +> 解释： +> 2 和 3 可以形成一组候选，2 + 2 + 3 = 7 。注意 2 可以使用多次。 +> 7 也是一个候选， 7 = 7 。 +> 仅有这两种组合。 +> ``` + +**💡 思路**：**元素无重可复选，即 `nums` 中的元素都是唯一的，每个元素可以被使用若干次**，只要删掉去重逻辑即可 + +```java +public class Solution { + public List> combinationSum(int[] candidates, int target) { + List> res = new ArrayList<>(); + List track = new ArrayList<>(); + backtrack(candidates, target, 0, track, res); + return res; + } + + private void backtrack(int[] candidates, int target, int start, List track, List> res) { + // 如果目标值为0，表示当前组合符合条件 + if (target == 0) { + res.add(new ArrayList<>(track)); // 将当前组合加入结果 + return; + } + + // 遍历候选数组 + for (int i = start; i < candidates.length; i++) { + // 如果当前数字大于目标值，跳过 + if (candidates[i] > target) continue; + + // 做选择：选择当前数字 + track.add(candidates[i]); + + // 递归，注意这里传入 i，因为可以重复选择当前数字 + backtrack(candidates, target - candidates[i], i, track, res); + + // 撤销选择 + track.remove(track.size() - 1); + } + } +} +``` + + + +## 其他问题 + +### 📞 [电话号码的字母组合_17](https://leetcode.cn/problems/letter-combinations-of-a-phone-number/) + +> 给定一个仅包含数字 2-9 的字符串，返回所有它能表示的字母组合。答案可以按 任意顺序 返回。 +> +> 给出数字到字母的映射如下（与电话按键相同）。注意 1 不对应任何字母。 +> +> ![img](https://assets.leetcode-cn.com/aliyun-lc-upload/uploads/2021/11/09/200px-telephone-keypad2svg.png) +> +> ``` +> 输入：digits = "23" +> 输出：["ad","ae","af","bd","be","bf","cd","ce","cf"] +> ``` + +**💡 思路**：回溯，递归地尝试每一位数字对应的所有字母，直到找出所有有效的组合 + +首先，我们需要将每个数字 2 到 9 映射到其对应的字母，可以用 Map , 也可以用数组。然后就是递归处理。 + +**递归终止条件**：当当前组合的长度与输入的数字字符串长度相同，就说明我们已经得到了一个有效的组合，可以将其加入结果集。 + +![](http://img.starfish.ink/leetcode/leetcode-letterCombinations.png) + +```java +public class Solution { + public List letterCombinations(String digits) { + List res = new ArrayList<>(); + if (digits == null || digits.length() == 0) { + return res; // 如果输入为空，返回空结果 + } + + // 数字到字母的映射 + String[] mapping = { + "", "", "abc", "def", "ghi", "jkl", "mno", "pqrs", "tuv", "wxyz" + }; + + // 使用回溯法生成字母组合 + backtrack(digits, 0, mapping, new StringBuilder(), res); + return res; + } + + private void backtrack(String digits, int index, String[] mapping, StringBuilder current, List res) { + // 如果当前组合的长度等于输入的长度，说明已经生成了一个有效的字母组合 + if (index == digits.length()) { + res.add(current.toString()); + return; + } + + // 获取当前数字对应的字母 + String letters = mapping[digits.charAt(index) - '0']; + + // 递归选择字母 + for (char letter : letters.toCharArray()) { + current.append(letter); // 选择一个字母 + backtrack(digits, index + 1, mapping, current, res); // 递归处理下一个数字 + current.deleteCharAt(current.length() - 1); // 撤销选择 + } + } +} +``` + + + +### 🔗 [括号生成_22](https://leetcode.cn/problems/generate-parentheses/) + +> 数字 `n` 代表生成括号的对数，请你设计一个函数，用于能够生成所有可能的并且 **有效的** 括号组合。 +> +> ``` +> 输入：n = 3 +> 输出：["((()))","(()())","(())()","()(())","()()()"] +> ``` + +**💡 思路**： + +![](http://img.starfish.ink/leetcode/leetcode-generate-parentheses.png) + +```java + +public List generateParenthesis(int n) { + List res = new ArrayList<>(); + // 回溯过程中的路径 + StringBuilder track = new StringBuilder(); + if (n == 0) { + return res; + } + trackback(n, n, res, track); + return res; + } + + // 可用的左括号数量为 left 个，可用的右括号数量为 right 个 + private void trackback(int left, int right, List res, StringBuilder track) { + //如果剩余的左括号数量大于右括号数量 + if (left < 0 || right < 0 || left > right) { + return; + } + + // 当所有括号都恰好用完时，得到一个合法的括号组合 + if (left == 0 && right == 0) { + res.add(track.toString()); + return; + } + + // 做选择，尝试放一个左括号 + track.append('('); + trackback(left - 1, right, res, track); + // 撤销选择 + track.deleteCharAt(track.length() - 1); + + track.append(')'); + trackback(left, right - 1, res, track); + track.deleteCharAt(track.length() - 1); + + } +``` + + + +### ♛ [N 皇后_51](https://leetcode.cn/problems/n-queens/) + +> 按照国际象棋的规则，皇后可以攻击与之处在同一行或同一列或同一斜线上的棋子。 +> +> **n 皇后问题** 研究的是如何将 `n` 个皇后放置在 `n×n` 的棋盘上，并且使皇后彼此之间不能相互攻击。 +> +> 给你一个整数 `n` ，返回所有不同的 **n 皇后问题** 的解决方案。 +> +> 每一种解法包含一个不同的 **n 皇后问题** 的棋子放置方案，该方案中 `'Q'` 和 `'.'` 分别代表了皇后和空位。 +> +> ![](https://assets.leetcode.com/uploads/2020/11/13/queens.jpg) +> +> ``` +>输入：n = 4 +> 输出：[[".Q..","...Q","Q...","..Q."],["..Q.","Q...","...Q",".Q.."]] +> 解释：如上图所示，4 皇后问题存在两个不同的解法。 +> ``` + +**💡 思路**：通过回溯算法逐行放置皇后，每次递归时确保当前行、列和对角线不被其他皇后攻击。通过标记已占用的列和对角线，避免重复搜索，最终生成所有合法的解。 + +- 如果在某一列或对角线处已有皇后，就不能在该位置放置皇后。我们可以使用三个辅助数组来追踪列和对角线的使用情况： + - `cols[i]`：表示第 `i` 列是否已经放置了皇后。 + - `diag1[i]`：表示从左上到右下的对角线（`row - col`）是否已经有皇后。 + - `diag2[i]`：表示从右上到左下的对角线（`row + col`）是否已经有皇后。 + +- 主对角线是从左上角到右下角的对角线、副对角线是从右上角到左下角的对角线 + +```java +public class NQueens { + + public List> solveNQueens(int N) { + List> result = new ArrayList<>(); + char[][] board = new char[N][N]; + + // 初始化棋盘，每个位置设为'.' + for (int i = 0; i < N; i++) { + for (int j = 0; j < N; j++) { + board[i][j] = '.'; + } + } + + // 用来记录列、主对角线、副对角线是否已被占用 + boolean[] cols = new boolean[N]; // 列占用标记 + boolean[] diag1 = new boolean[2 * N - 1]; // 主对角线占用标记 + boolean[] diag2 = new boolean[2 * N - 1]; // 副对角线占用标记 + + backtrack(N, 0, board, cols, diag1, diag2, result); + return result; + } + + // 回溯函数 + private void backtrack(int N, int row, char[][] board, boolean[] cols, boolean[] diag1, boolean[] diag2, List> result) { + if (row == N) { // 如果已经放置了 N 个皇后 + List solution = new ArrayList<>(); + for (int i = 0; i < N; i++) { + solution.add(new String(board[i])); // 将每一行转化为字符串并添加到结果中 + } + result.add(solution); + return; + } + + // 遍历每一列，尝试放置皇后 + for (int col = 0; col < N; col++) { + // 判断当前位置是否可以放置皇后 + if (cols[col] || diag1[row - col + (N - 1)] || diag2[row + col]) { + continue; // 如果列、主对角线或副对角线已被占用，跳过当前列 + } + + // 放置皇后 + board[row][col] = 'Q'; + cols[col] = true; // 标记该列已被占用 + diag1[row - col + (N - 1)] = true; // 标记主对角线已被占用 + diag2[row + col] = true; // 标记副对角线已被占用 + + // 递归放置下一行的皇后 + backtrack(N, row + 1, board, cols, diag1, diag2, result); + + // 回溯，撤销当前位置的选择 + board[row][col] = '.'; + cols[col] = false; + diag1[row - col + (N - 1)] = false; + diag2[row + col] = false; + } + } + + // 打印结果 + public void printSolutions(List> solutions) { + for (List solution : solutions) { + for (String row : solution) { + System.out.println(row); + } + System.out.println(); + } + } + + public static void main(String[] args) { + NQueens nq = new NQueens(); + List> solutions = nq.solveNQueens(4); + nq.printSolutions(solutions); + } +} + +``` + + + + + +## 📚 参考与感谢： + +- https://yuminlee2.medium.com/combinations-and-combination-sum-3ed2accc8d12 +- https://medium.com/@sunshine990316/leetcode-python-backtracking-summary-medium-1-e8ae88839e85 +- https://blog.devgenius.io/10-daily-practice-problems-day-18-f7293b55224d +- [hello 算法- 回溯算法](https://www.hello-algo.com/chapter_backtracking/backtracking_algorithm/#1312) + +--- + +> 🎉 **恭喜你完成了回溯算法的学习！** 回溯算法是解决很多复杂问题的强大工具，掌握了它，你就拥有了解决排列、组合、子集等问题的钥匙。记住：**选择 → 递归 → 撤销选择** 是回溯的核心思想！ diff --git a/docs/data-structure-algorithms/algorithm/Binary-Search.md b/docs/data-structure-algorithms/algorithm/Binary-Search.md new file mode 100755 index 0000000000..9fb5f32679 --- /dev/null +++ b/docs/data-structure-algorithms/algorithm/Binary-Search.md @@ -0,0 +1,1030 @@ +--- +title: 二分查找 +date: 2023-02-09 +tags: + - binary-search + - algorithms +categories: algorithms +--- + +![](https://img.starfish.ink/algorithm/binary-search-banner.png) + +> 二分查找【折半查找】，一种简单高效的搜索算法，一般是利用有序数组的特性，通过逐步比较中间元素来快速定位目标值。 +> +> 二分查找并不简单，Knuth 大佬（发明 KMP 算法的那位）都说二分查找：**思路很简单，细节是魔鬼**。比如二分查找让人头疼的细节问题，到底要给 `mid` 加一还是减一，while 里到底用 `<=` 还是 `<`。 + +## 一、二分查找基础框架 + +```java +int binarySearch(int[] nums, int target) { + int left = 0, right = ...; + + while(...) { + int mid = left + (right - left) / 2; + if (nums[mid] == target) { + ... + } else if (nums[mid] < target) { + left = ... + } else if (nums[mid] > target) { + right = ... + } + } + return ...; +} +``` + +**分析二分查找的一个技巧是：不要出现 else，而是把所有情况用 else if 写清楚，这样可以清楚地展现所有细节**。本文都会使用 else if，旨在讲清楚，读者理解后可自行简化。 + +其中 `...` 标记的部分，就是可能出现细节问题的地方，当你见到一个二分查找的代码时，首先注意这几个地方。 + +**另外提前说明一下，计算 `mid` 时需要防止溢出**，代码中 `left + (right - left) / 2` 就和 `(left + right) / 2` 的结果相同，但是有效防止了 `left` 和 `right` 太大，直接相加导致溢出的情况。 + + + +## 二、二分查找性能分析 + +**时间复杂度**：二分查找的时间复杂度为 $O(log n)$​，其中 n 是数组的长度。这是因为每次比较后，搜索范围都会减半，非常高效。 + +> logn 是一个非常“恐怖”的数量级，即便 n 非常非常大，对应的 logn 也很小。比如 n 等于 2 的 32 次方，这个数很大了吧？大约是 42 亿。也就是说，如果我们在 42 亿个数据中用二分查找一个数据，最多需要比较 32 次。 + +**空间复杂度**： + +- 迭代法：$O(1)$，因为只需要常数级别的额外空间。 +- 递归法：$O(log n)$，因为递归调用会占用栈空间。 + +**最坏情况**：最坏情况下，目标值位于数组两端或不存在，需要$log n$次比较才能确定。 + +**二分查找与其他搜索算法的比较**： + +- 线性搜索：线性搜索简单，时间复杂度为$O(n)$，但在大规模数据集上效率较低。 + +- 哈希：哈希在查找上能提供平均$O(1)$的时间复杂度，但需要额外空间存储哈希表，且对数据有序性无要求。 + + + +## 三、刷刷热题 + +- 二分查找，可以用循环（迭代）实现，也可以用递归实现 +- 二分查找依赖的是顺序表结构（也就是数组） +- 二分查找针对的是有序数组 +- 数据量太小太大都不是很适用二分（太小直接顺序遍历就够了，太大的话对连续内存空间要求更高） + +### [二分查找『704』](https://leetcode.cn/problems/binary-search/)（基本的二分搜索） + +> 给定一个 n 个元素有序的（升序）整型数组 nums 和一个目标值 target ，写一个函数搜索 nums 中的 target，如果目标值存在返回下标，否则返回 -1。 +> + +```java +int binarySearch(int[] nums, int target) { + int left = 0; + int right = nums.length - 1; // 注意 + + while(left <= right) { + int mid = left + (right - left) / 2; + if(nums[mid] == target) + return mid; + else if (nums[mid] < target) + left = mid + 1; // 注意 + else if (nums[mid] > target) + right = mid - 1; // 注意 + } + return -1; +} +``` + +**时间复杂度**：O(log n)，每次都将搜索范围缩小一半 +**空间复杂度**：O(1)，只使用常量级别的额外空间 + +**1、为什么 while 循环的条件中是 <=，而不是 <**？ + +答：因为初始化 `right` 的赋值是 `nums.length - 1`，即最后一个元素的索引，而不是 `nums.length`。 + +这二者可能出现在不同功能的二分查找中，区别是：前者相当于两端都闭区间 `[left, right]`，后者相当于左闭右开区间 `[left, right)`。因为索引大小为 `nums.length` 是越界的，所以我们把 `right` 这一边视为开区间。 + +我们这个算法中使用的是前者 `[left, right]` 两端都闭的区间。**这个区间其实就是每次进行搜索的区间**。 + +**2、为什么 `left = mid + 1`，`right = mid - 1`？我看有的代码是 `right = mid` 或者 `left = mid`，没有这些加加减减，到底怎么回事，怎么判断**？ + +答：这也是二分查找的一个难点，不过只要你能理解前面的内容，就能够很容易判断。 + +刚才明确了「搜索区间」这个概念，而且本算法的搜索区间是两端都闭的，即 `[left, right]`。那么当我们发现索引 `mid` 不是要找的 `target` 时，下一步应该去搜索哪里呢？ + +当然是去搜索区间 `[left, mid-1]` 或者区间 `[mid+1, right]` 对不对？**因为 `mid` 已经搜索过，应该从搜索区间中去除**。 + +> ##### 1. **左闭右闭区间 `[left, right]`** +> +> - **循环条件**：`while (left <= right)`，因为 `left == right` 时区间仍有意义。 +> - 边界调整： +> - `nums[mid] < target` → `left = mid + 1`（排除 `mid` 左侧） +> - `nums[mid] > target` → `right = mid - 1`（排除 `mid` 右侧） +> - 适用场景：明确目标值存在于数组时，直接返回下标。 +> +> ##### 2. **左闭右开区间 `[left, right)`** +> +> - **初始化**：`right = nums.length`。 +> - **循环条件**：`while (left < right)`，因为 `left == right` 时区间为空。 +> - 边界调整： +> - `nums[mid] < target` → `left = mid + 1` +> - `nums[mid] > target` → `right = mid`（右开，不包含 `mid`） +> - **适用场景**：需要处理目标值可能不在数组中的情况，例如插入位置问题 + +> 比如说给你有序数组 `nums = [1,2,2,2,3]`，`target` 为 2，此算法返回的索引是 2，没错。但是如果我想得到 `target` 的左侧边界，即索引 1，或者我想得到 `target` 的右侧边界，即索引 3，这样的话此算法是无法处理的。 +> +> 所以又有了一些含有重复元素，带有边界问题的二分。 + +### 寻找左侧边界的二分搜索 + +```java +public int leftBound(int[] nums, int target) { + int left = 0; + int right = nums.length - 1; + while (left <= right) { + int mid = (right - left) / 2 + left; + if (nums[mid] > target) { + right = mid - 1; + } else if (nums[mid] < target) { + left = mid + 1; + } else { + //mid 是第一个元素，或者前一个元素不等于查找值，锁定，且返回的是mid + if (mid == 0 || nums[mid - 1] != target) return mid; + else right = mid - 1; + } + } + return -1; +} +``` + +### 寻找右侧边界的二分查找 + +```java +public int rightBound(int[] nums, int target){ + int left = 0; + int right = nums.length - 1; + while(left <= right){ + int mid = left + (right - left)/2; + if(nums[mid] > target){ + right = mid - 1; + }else if(nums[mid] < target){ + left = mid +1; + }else{ + if(mid == nums.length - 1 || nums[mid +1] != target) return mid; + else left = mid + 1; + } + } + return -1; +} +``` + +### 查找第一个大于等于给定值的元素 + +```JAVA +//查找第一个大于等于给定值的元素 1,3,5,7,9 找出第一个大于等于5的元素 +public int firstNum(int[] nums, int target) { + int left = 0; + int right = nums.length - 1; + while (left <= right) { + int mid = left + (right - left) / 2; + if (nums[mid] >= target) { + if (mid == 0 || nums[mid - 1] < target) return mid; + else right = mid - 1; + } else { + left = mid + 1; + } + } + return -1; +} +``` + + + +### [搜索旋转排序数组『33』](https://leetcode-cn.com/problems/search-in-rotated-sorted-array/) + +> 整数数组 nums 按升序排列，数组中的值 互不相同 。 +> +> 在传递给函数之前，nums 在预先未知的某个下标 k（0 <= k < nums.length）上进行了 旋转，使数组变为 [nums[k], nums[k+1], ..., nums[n-1], nums[0], nums[1], ..., nums[k-1]]（下标 从 0 开始 计数）。例如， [0,1,2,4,5,6,7] 在下标 3 处经旋转后可能变为 [4,5,6,7,0,1,2] 。 +> +> 给你 旋转后 的数组 nums 和一个整数 target ，如果 nums 中存在这个目标值 target ，则返回它的下标，否则返回 -1 。 +> +> ``` +> 输入：nums = [4,5,6,7,0,1,2], target = 0 +> 输出：4 +> ``` +> +> ``` +> 输入：nums = [4,5,6,7,0,1,2], target = 3 +> 输出：-1 +> ``` + +**思路**： + +对于有序数组（部分有序也可以），可以使用二分查找的方法查找元素。 + +旋转数组后，依然是局部有序，从数组中间分成左右两部分后，一定有一部分是有序的 + +- 如果 [L, mid - 1] 是有序数组，且 target 的大小满足 [nums[L],nums[mid]，则我们应该将搜索范围缩小至 [L, mid - 1]，否则在 [mid + 1, R] 中寻找。 +- 如果 [mid, R] 是有序数组，且 target 的大小满足 ({nums}[mid+1],{nums}[R]]，则我们应该将搜索范围缩小至 [mid + 1, R]，否则在 [l, mid - 1] 中寻找。 + +```java +public int search(int[] nums, int target) { + if (nums == null || nums.length == 0) return -1; + + int left = 0, right = nums.length - 1; + + while (left <= right) { + int mid = left + (right - left) / 2; + + if (nums[mid] == target) { + return mid; // 找到目标 + } + + // 判断左半部分是否有序 + if (nums[left] <= nums[mid]) { + // 左半部分有序 + if (nums[left] <= target && target < nums[mid]) { + right = mid - 1; // 目标在左半部分 + } else { + left = mid + 1; // 目标在右半部分 + } + } else { + // 右半部分有序 + if (nums[mid] < target && target <= nums[right]) { + left = mid + 1; // 目标在右半部分 + } else { + right = mid - 1; // 目标在左半部分 + } + } + } + + return -1; // 未找到目标 +} +``` + +**时间复杂度**：$O(log n) $，二分查找的时间复杂度 +**空间复杂度**：$O(1)$，只使用常量级别的额外空间 + + + +### [在排序数组中查找元素的第一个和最后一个位置『34』](https://leetcode-cn.com/problems/find-first-and-last-position-of-element-in-sorted-array/) + +> 给定一个按照升序排列的整数数组 nums，和一个目标值 target。找出给定目标值在数组中的开始位置和结束位置。 +> +> 如果数组中不存在目标值 target，返回 [-1, -1]。 +> +> 你可以设计并实现时间复杂度为 $O(log n) $ 的算法解决此问题吗？ +> +> ``` +> 输入：nums = [5,7,7,8,8,10], target = 8 +> 输出：[3,4] +> ``` +> +> ``` +> 输入：nums = [5,7,7,8,8,10], target = 6 +> 输出：[-1,-1] +> ``` + +**思路**：二分法寻找左右边界值 + +```java +public int[] searchRange(int[] nums, int target) { + int first = binarySearch(nums, target, true); + int last = binarySearch(nums, target, false); + return new int[]{first, last}; +} + +public int binarySearch(int[] nums, int target, boolean findLast) { + int length = nums.length; + int left = 0, right = length - 1; + //结果，因为可能有多个值，所以需要先保存起来 + int index = -1; + while (left <= right) { + //取中间值 + int middle = left + (right - left) / 2; + + //找到相同的值（只有这个地方和普通二分查找有不同） + if (nums[middle] == target) { + //先赋值一下，肯定是找到了，只是不知道这个值是不是在区域的边界内 + index = middle; + //如果是查找最后的 + if (findLast) { + //那我们将浮标移动到下一个值试探一下后面的值还是否有target + left = middle + 1; + } else { + //否则，就是查找第一个值，也是同理，移动指针到上一个值去试探一下上一个值是不是等于target + right = middle - 1; + } + + //下面2个就是普通的二分查找流程，大于小于都移动指针 + } else if (nums[middle] < target) { + left = middle + 1; + } else { + right = middle - 1; + } + + } + return index; +} +``` + +**时间复杂度**：$O(log n) $，需要进行两次二分查找 +**空间复杂度**：$O(1)$，只使用常量级别的额外空间 + + + +### [搜索插入位置『35』](https://leetcode.cn/problems/search-insert-position/) + +> 给定一个排序数组和一个目标值，在数组中找到目标值，并返回其索引。如果目标值不存在于数组中，返回它将会被按顺序插入的位置。请必须使用时间复杂度为 `O(log n)` 的算法。 +> +> ``` +> 输入: nums = [1,3,5,6], target = 2 +> 输出: 1 +> ``` + +```java +public int searchInsert(int[] nums, int target) { + int left = 0; + int right = nums.length - 1; + //注意：特例处理 + if (nums[left] > target) return 0; + if (nums[right] < target) return right + 1; + while (left <= right) { + int mid = left + (right - left) / 2; + if (nums[mid] > target) { + right = mid - 1; + } else if (nums[mid] < target) { + left = mid + 1; + } else { + return mid; + } + } + //注意：这里如果没有查到，返回left,也就是需要插入的位置 + return left; + } +``` + +**时间复杂度**：$O(log n) $，标准的二分查找时间复杂度 +**空间复杂度**：$O(1)$，只使用常量级别的额外空间 + + + +### [寻找旋转排序数组中的最小值『153』](https://leetcode-cn.com/problems/find-minimum-in-rotated-sorted-array/) + +> 已知一个长度为 n 的数组，预先按照升序排列，经由 1 到 n 次 旋转 后，得到输入数组。例如，原数组 nums = [0,1,2,4,5,6,7] 在变化后可能得到： +> 若旋转 4 次，则可以得到 [4,5,6,7,0,1,2] 若旋转 7 次，则可以得到 [0,1,2,4,5,6,7] +> 注意，数组 [a[0], a[1], a[2], ..., a[n-1]] 旋转一次 的结果为数组 [a[n-1], a[0], a[1], a[2], ..., a[n-2]] 。 +> +>给你一个元素值 互不相同 的数组 nums ，它原来是一个升序排列的数组，并按上述情形进行了多次旋转。请你找出并返回数组中的 最小元素 。 +> +>你必须设计一个时间复杂度为 $O(log n)$ 的算法解决此问题。 +> +>``` +> 输入：nums = [3,4,5,1,2] +> 输出：1 +> 解释：原数组为 [1,2,3,4,5] ，旋转 3 次得到输入数组。 +> ``` +> + +**思路**： + +升序数组+旋转，仍然是部分有序，考虑用二分查找。 + +![](https://assets.leetcode-cn.com/solution-static/153/1.png) + +> 我们先搞清楚题目中的数组是通过怎样的变化得来的，基本上就是等于将整个数组向右平移 + +> 这种二分查找难就难在，arr[mid] 跟谁比。 +> +> 我们的目的是：当进行一次比较时，一定能够确定答案在 mid 的某一侧。一次比较为 arr[mid] 跟谁比的问题。 +> 一般的比较原则有： +> +> - 如果有目标值 target，那么直接让 arr[mid] 和 target 比较即可。 +> - 如果没有目标值，一般可以考虑 **端点** +> +> 如果中值 < 右值，则最小值在左半边，可以收缩右边界。 +> 如果中值 > 右值，则最小值在右半边，可以收缩左边界。 + +旋转数组，最小值右侧的元素肯定都小于或等于数组中的最后一个元素 `nums[n-1]`，左侧元素都大于 `num[n-1]` + +```java +public static int findMin(int[] nums) { + int left = 0; + int right = nums.length - 1; + //左闭右开 + while (left < right) { + int mid = left + (right - left) / 2; + //疑问：为什么right = mid;而不是 right = mid-1; + //解答：{4,5,1,2,3}，如果right = mid-1，则丢失了最小值1 + if (nums[mid] < nums[right]) { + right = mid; + } else { + left = mid + 1; + } + } + //循环结束条件，left = right，最小值输出nums[left]或nums[right]均可 + return nums[left]; +} +``` + +**时间复杂度**：$O(log n) $，每次都将搜索范围缩小一半 +**空间复杂度**：$O(1)$，只使用常量级别的额外空间 + +**如果是求旋转数组中的最大值呢** + +```java +public static int findMax(int[] nums) { + int left = 0; + int right = nums.length - 1; + + while (left < right) { + int mid = left + (right - left) >> 1; + + //因为向下取整，left可能会等于mid，所以要考虑 + if (nums[left] < nums[right]) { + return nums[right]; + } + + //[left,mid] 是递增的，最大值只会在[mid,right]中 + if (nums[left] < nums[mid]) { + left = mid; + } else { + //[mid,right]递增，最大值只会在[left, mid-1]中 + right = mid - 1; + } + } + return nums[left]; +} +``` + +### [寻找重复数『287』](https://leetcode-cn.com/problems/find-the-duplicate-number/) + +> 长度为 n+1 的数组，元素在 1~n 之间，有且仅有一个重复数（可能重复多次）。要求不修改数组且只用 O (1) 空间。 +> +> ``` +>输入：nums = [1,3,4,2,2] +> 输出：2 +>``` +> +> ``` +> 输入：nums = [3,1,3,4,2] +> 输出：3 +>``` + +**思路**： + +- 统计小于等于 mid 的元素个数 +- 若 count > mid，说明重复数在 [1, mid] 区间 +- 否则在 [mid+1, n] 区间 + +> 抽屉原理：把 `10` 个苹果放进 `9` 个抽屉，至少有一个抽屉里至少放 `2` 个苹果。 + +```java +public int findDuplicate(int[] nums) { + int left = 0; + int right = nums.length - 1; + while (left <= right) { + int mid = left + (right - left) / 2; + + // nums 中小于等于 mid 的元素的个数 + int count = 0; + for (int num : nums) { + //看这里，是 <= mid，而不是 nums[mid] + if (num <= mid) { + count += 1; + } + } + + // 根据抽屉原理，小于等于 4 的个数如果严格大于 4 个，此时重复元素一定出现在 [1..4] 区间里 + if (count > mid) { + // 重复元素位于区间 [left..mid] + right = mid - 1; + } else { + // if 分析正确了以后，else 搜索的区间就是 if 的反面区间 [mid + 1..right] + left = mid + 1; + } + } + return left; +} +``` + +**时间复杂度**：$O(n log n)$，每次需要遍历数组O(n) +**空间复杂度**：$O(1)$，只使用常量级别的额外空间 + + + +### [寻找峰值『162』](https://leetcode-cn.com/problems/find-peak-element/) + +> 峰值元素是指其值严格大于左右相邻值的元素。 +> +> 给你一个整数数组 nums，找到峰值元素并返回其索引。数组可能包含多个峰值，在这种情况下，返回 任何一个峰值 所在位置即可。 +> +> 你可以假设 nums[-1] = nums[n] = -∞ 。 +> +> 你必须实现时间复杂度为 $O(log n) $的算法来解决此问题。 +> +> ``` +> 输入：nums = [1,2,3,1] +> 输出：2 +> 解释：3 是峰值元素，你的函数应该返回其索引 2。 +> ``` +> +> ``` +> 输入：nums = [1,2,1,3,5,6,4] +> 输出：1 或 5 +> 解释：你的函数可以返回索引 1，其峰值元素为 2； +>   或者返回索引 5， 其峰值元素为 6。 +> ``` + +**思路**： + +- 比较 `nums[mid]` 和 `nums[mid+1]` +- 如果 `nums[mid] < nums[mid+1]`，说明右侧一定有峰值 +- 否则左侧一定有峰值 + +**为什么有效**： + +- 峰值条件 `nums[i] > nums[i+1]` 保证单调性 +- 二分查找每次可以缩小一半搜索范围 + +```java +public int findPeakElement(int[] nums) { + int left = 0, right = nums.length - 1; + + while (left < right) { + int mid = left + (right - left) / 2; + + if (nums[mid] < nums[mid + 1]) { + // 右侧有峰值 + left = mid + 1; + } else { + // 左侧有峰值 + right = mid; + } + } + + return left; // left == right，即为峰值位置 +} +``` + +**时间复杂度**：$O(log n) $，二分查找的时间复杂度 +**空间复杂度**：$O(1)$，只使用常量级别的额外空间 + + + +### [搜索二维矩阵『74』](https://leetcode.cn/problems/search-a-2d-matrix/) + +> 给你一个满足以下两点的 `m x n` 矩阵： +> +> - 每行从左到右递增 +> - 每行第一个数大于上一行最后一个数 +> +> 判断目标值 `target` 是否在矩阵中。 +> +> 给你一个整数 `target` ，如果 `target` 在矩阵中，返回 `true` ；否则，返回 `false` 。 +> +> ![img](https://assets.leetcode.com/uploads/2020/10/05/mat.jpg) +> +> ``` +> 输入：matrix = [[1,3,5,7],[10,11,16,20],[23,30,34,60]], target = 3 +> 输出：true +> ``` + +**思路**：由于每行的第一个元素都大于前一行的最后一个元素，整个矩阵可以被看作一个**完全有序的一维数组**。 + +例如，上面的示例矩阵可以被 “拉直” 为：`[1, 3, 5, 7, 10, 11, 16, 20, 23, 30, 34, 60]` + +要实现 “一次二分”，必须能将一维数组的索引 `mid`（二分查找中的中间位置）转换回二维矩阵的 `(行号, 列号)`，公式如下： + +- **行号 = mid /n**（整除，因为每一行有 `n` 个元素，商表示当前元素在第几行） +- **列号 = mid % n**（取余，余数表示当前元素在该行的第几列） + +举例： + +- 一维索引 `mid = 5`，`n = 4`（列数）： + - 行号 = 5 / 4 = 1（第 2 行，索引从 0 开始） + - 列号 = 5 % 4 = 1（第 2 列） + - 对应矩阵值：`matrix[1][1] = 11`，与一维数组索引 5 的值一致。 + +```java +public boolean searchMatrix(int[][] matrix, int target) { + // 1. 处理边界情况：矩阵为空（行数为 0） + int m = matrix.length; + if (m == 0) { + return false; + } + + // 2. 处理边界情况：矩阵列数为空（每行没有元素） + int n = matrix[0].length; + if (n == 0) { + return false; + } + + // 3. 初始化二分查找的左右指针（对应一维数组的起始和末尾索引） + int left = 0; + int right = m * n - 1; // 总元素数 = 行数 × 列数，末尾索引 = 总元素数 - 1 + + // 4. 二分查找循环：left <= right 确保不遗漏元素 + while (left <= right) { + // 计算中间索引：避免 left + right 直接相加导致整数溢出 + int mid = left + (right - left) / 2; + + // 关键：将一维索引 mid 转换为二维矩阵的 (row, col) 坐标 + int row = mid / n; // 行号 = 中间索引 ÷ 列数（整除） + int col = mid % n; // 列号 = 中间索引 % 列数（取余） + + // 获取当前中间位置的矩阵值 + int midValue = matrix[row][col]; + + // 5. 比较 midValue 与 target，调整二分范围 + if (midValue == target) { + // 找到目标值，直接返回 true + return true; + } else if (midValue < target) { + // 中间值比目标小：目标在右半部分，left 移到 mid + 1 + left = mid + 1; + } else { + // 中间值比目标大：目标在左半部分，right 移到 mid - 1 + right = mid - 1; + } + } + + // 6. 循环结束仍未找到，说明目标不在矩阵中 + return false; +} +``` + +**时间复杂度**：$O(log(mn)) $：二分查找的时间复杂度是 `O(log(总元素数))`，总元素数为 `m×n`，因此复杂度为 `O(log(m×n))` +**空间复杂度**：$O(1)$：仅使用了 `m、n、left、right、mid、row、col、midValue` 等常数个变量，没有额外占用空间 + + + +### [搜索二维矩阵 II『240』](https://leetcode-cn.com/problems/search-a-2d-matrix-ii/) + +> [剑指 Offer 04. 二维数组中的查找](https://leetcode-cn.com/problems/er-wei-shu-zu-zhong-de-cha-zhao-lcof/) 一样的题目 +> +> 在一个 n * m 的二维数组中，每一行都按照从左到右递增的顺序排序，每一列都按照从上到下递增的顺序排序。请完成一个高效的函数，输入这样的一个二维数组和一个整数，判断数组中是否含有该整数。 +> +> 现有矩阵 matrix 如下： +> +> ![](https://assets.leetcode-cn.com/aliyun-lc-upload/uploads/2020/11/25/searchgrid2.jpg) +> +> ``` +> 输入：matrix = [[1,4,7,11,15],[2,5,8,12,19],[3,6,9,16,22],[10,13,14,17,24],[18,21,23,26,30]], target = 5 +> 输出：true +> ``` + +**思路**： + +站在左下角或者右上角看。这个矩阵其实就像是一个Binary Search Tree。然后，聪明的大家应该知道怎么做了。 + +![](https://img.starfish.ink/data-structure/searchgrid2-solution.jpg) + +有序的数组，我们首先应该想到二分 + +```java +public boolean searchMatrix(int[][] matrix, int target) { + // 1. 处理边界情况：矩阵为空、行数为0、或列数为0 + if (matrix == null || matrix.length == 0 || matrix[0].length == 0) { + return false; + } + + // 2. 获取矩阵的行数 m 和列数 n + int m = matrix.length; + int n = matrix[0].length; + + // 3. 初始化指针，从左下角开始 + int row = m - 1; // 行指针指向最后一行 + int col = 0; // 列指针指向第一列 + + // 4. 循环条件：行指针不能越界（>= 0），列指针不能越界（< n） + while (row >= 0 && col < n) { + // 获取当前指针指向的元素值 + int current = matrix[row][col]; + + // 5. 比较 current 与 target + if (current == target) { + // 找到了目标值，直接返回 true + return true; + } else if (current < target) { + // 当前值小于目标值：目标值不可能在当前行（因为行是递增的） + // 将列指针向右移动，去更大的区域查找 + col++; + } else { // current > target + // 当前值大于目标值：目标值不可能在当前列（因为列是递增的） + // 将行指针向上移动，去更小的区域查找 + row--; + } + } + + // 6. 如果循环结束仍未找到，说明目标值不存在于矩阵中 + return false; + } +} +``` + +**时间复杂度**：$O(m + n)$，其中m是行数，n是列数，最坏情况下需要遍历m+n个元素 +**空间复杂度**：$O(1)$，只使用常量级别的额外空间 + + + +### [最长递增子序列『300』](https://leetcode.cn/problems/longest-increasing-subsequence/) + +> 给你一个整数数组 `nums` ，找到其中最长严格递增子序列的长度。 +> +> **子序列** 是由数组派生而来的序列，删除（或不删除）数组中的元素而不改变其余元素的顺序。例如，`[3,6,2,7]`是数组 `[0,3,1,6,2,2,7]` 的子序列。 +> +> ``` +> 输入：nums = [10,9,2,5,3,7,101,18] +> 输出：4 +> 解释：最长递增子序列是 [2,3,7,101]，因此长度为 4。 +> ``` + +**思路**： + +**动态规划解法**： + +```java +public int lengthOfLIS(int[] nums) { + if (nums == null || nums.length == 0) return 0; + int[] dp = new int[nums.length]; + //初始时，每个元素自身构成一个长度为1的子序列 + Arrays.fill(dp, 1); + // 记录全局最长递增子序列的长度，初始为1 + int maxLen = 1; + + for (int i = 1; i < nums.length; i++) { + // 对于每个i，检查所有j < i的元素 + for (int j = 0; j < i; j++) { + // 如果nums[j] < nums[i]，说明nums[i]可以接在nums[j]后面 + if (nums[j] < nums[i]) { + // 更新dp[i]为dp[j] + 1和当前dp[i]的较大值 + dp[i] = Math.max(dp[i], dp[j] + 1); + } + } + // 更新全局最大值 + maxLen = Math.max(maxLen, dp[i]); + } + return maxLen; +} +``` + +**二分查找优化解法**： +```java +public int lengthOfLIS(int[] nums) { + if (nums == null || nums.length == 0) return 0; + + List tails = new ArrayList<>(); + + for (int num : nums) { + // 二分查找第一个大于等于num的位置 + int left = 0, right = tails.size(); + while (left < right) { + int mid = left + (right - left) / 2; + if (tails.get(mid) < num) { + left = mid + 1; + } else { + right = mid; + } + } + + // 如果找到末尾，说明num比所有元素都大，直接添加 + if (left == tails.size()) { + tails.add(num); + } else { + // 否则替换找到的位置 + tails.set(left, num); + } + } + + return tails.size(); +} +``` + +**时间复杂度**： +- 动态规划：$O(n²)$ +- 二分查找优化：$O(n log n)$ + +**空间复杂度**：$O(n)$，需要额外的数组空间 + + + +### [寻找两个正序数组的中位数『4』](https://leetcode.cn/problems/median-of-two-sorted-arrays/) + +> 给定两个大小分别为 `m` 和 `n` 的正序（从小到大）数组 `nums1` 和 `nums2`。请你找出并返回这两个正序数组的 **中位数** 。 +> +> 算法的时间复杂度应该为 `O(log (m+n))` 。 +> +> ``` +> 输入：nums1 = [1,3], nums2 = [2] +> 输出：2.00000 +> 解释：合并数组 = [1,2,3] ，中位数 2 +> ``` + +**思路**： + +中位数是指将一组数据从小到大排序后，位于中间位置的数值。如果数据集中的元素数量是奇数，中位数就是中间的那个元素；如果是偶数，则中位数是中间两个元素的平均值。 + +这道题要求时间复杂度为 O(log(m+n))，提示我们使用二分查找。关键思路是： + +1. **问题转化**：寻找第 k 小的元素，其中 k = (m+n+1)/2（奇数情况）或需要找第k和第k+1小的元素（偶数情况） +2. **二分搜索**：在较短的数组上进行二分，确保左半部分元素个数等于右半部分 +3. **分割线性质**：左半部分的最大值 ≤ 右半部分的最小值 + +```java +public double findMedianSortedArrays(int[] nums1, int[] nums2) { + // 确保 nums1 是较短的数组 + if (nums1.length > nums2.length) { + return findMedianSortedArrays(nums2, nums1); + } + + int m = nums1.length; + int n = nums2.length; + int left = 0, right = m; + + while (left <= right) { + // nums1的分割点 + int cut1 = (left + right) / 2; + // nums2的分割点 + int cut2 = (m + n + 1) / 2 - cut1; + + // 处理边界情况 + int left1 = (cut1 == 0) ? Integer.MIN_VALUE : nums1[cut1 - 1]; + int left2 = (cut2 == 0) ? Integer.MIN_VALUE : nums2[cut2 - 1]; + int right1 = (cut1 == m) ? Integer.MAX_VALUE : nums1[cut1]; + int right2 = (cut2 == n) ? Integer.MAX_VALUE : nums2[cut2]; + + // 找到正确的分割 + if (left1 <= right2 && left2 <= right1) { + // 总长度为偶数 + if ((m + n) % 2 == 0) { + return (Math.max(left1, left2) + Math.min(right1, right2)) / 2.0; + } else { + // 总长度为奇数 + return Math.max(left1, left2); + } + } + // nums1分割点太靠右 + else if (left1 > right2) { + right = cut1 - 1; + } + // nums1分割点太靠左 + else { + left = cut1 + 1; + } + } + + return 0.0; +} +``` + +**算法步骤详解**： +1. 确保在较短数组上进行二分搜索，减少搜索空间 +2. 计算两个数组的分割点，使得左半部分元素个数 = 右半部分元素个数（或多1个） +3. 检查分割是否正确：左半部分最大值 ≤ 右半部分最小值 +4. 根据总长度奇偶性计算中位数 + +**时间复杂度**：$O(log(min(m,n)))$，在较短数组上进行二分搜索 +**空间复杂度**：$O(1)$，只使用常量级别的额外空间 + + + +## 四、常见题目补充 + + +### [x 的平方根『69』](https://leetcode.cn/problems/sqrtx/) + +> 给你一个非负整数 `x`，计算并返回 `x` 的平方根的整数部分。 + +**思路**：在区间 `[1, x/2]` 上二分，寻找最大的 `mid` 使得 `mid*mid <= x`。注意用 `long` 防止乘法溢出。 + +```java +public int mySqrt(int x) { + if (x < 2) return x; + int left = 1, right = x / 2, ans = 0; + while (left <= right) { + int mid = left + (right - left) / 2; + if ((long) mid * mid <= x) { + ans = mid; + left = mid + 1; + } else { + right = mid - 1; + } + } + return ans; +} +``` + +**时间复杂度**：$O(log x)$ +**空间复杂度**：$O(1)$ + + +### [第一个错误的版本『278』](https://leetcode.cn/problems/first-bad-version/) + +> 有 `n` 个版本，从 1 到 n，给你 `isBadVersion(version)` 接口。找出第一个错误版本。 + +**思路**：典型“找左边界”。`isBad(mid)` 为真时收缩到左侧，并记录答案。 + +```java +// isBadVersion(version) 由系统提供 +public int firstBadVersion(int n) { + int left = 1, right = n, ans = n; + while (left <= right) { + int mid = left + (right - left) / 2; + if (isBadVersion(mid)) { + ans = mid; + right = mid - 1; + } else { + left = mid + 1; + } + } + return ans; +} +``` + +**时间复杂度**：$O(log n) $ +**空间复杂度**：$O(1)$ + + +### [有序数组中的单一元素『540』](https://leetcode.cn/problems/single-element-in-a-sorted-array/) + +> 一个按升序排列的数组，除了某个元素只出现一次外，其余每个元素均出现两次。找出这个元素。 + +**思路**：利用“成对对齐”的性质。让 `mid` 偶数化（`mid -= mid % 2`），若 `nums[mid] == nums[mid+1]`，唯一元素在右侧；否则在左侧（含 `mid`）。 + +```java +public int singleNonDuplicate(int[] nums) { + int left = 0, right = nums.length - 1; + while (left < right) { + int mid = left + (right - left) / 2; + if (mid % 2 == 1) mid--; // 偶数化 + if (nums[mid] == nums[mid + 1]) { + left = mid + 2; + } else { + right = mid; + } + } + return nums[left]; +} +``` + +**时间复杂度**：$O(log n) $ +**空间复杂度**：$O(1)$ + + +### [爱吃香蕉的珂珂『875』](https://leetcode.cn/problems/koko-eating-bananas/) + +> 给定每堆香蕉数量 `piles` 和总小时数 `h`，最小化吃速 `k` 使得能在 `h` 小时内吃完。 + +**思路**：对答案 `k` 二分。检验函数为以速率 `k` 需要的总小时数是否 `<= h`。 + +```java +public int minEatingSpeed(int[] piles, int h) { + int left = 1, right = 0; + for (int p : piles) right = Math.max(right, p); + while (left < right) { + int mid = left + (right - left) / 2; + long hours = 0; + for (int p : piles) { + hours += (p + mid - 1) / mid; // 向上取整 + } + if (hours <= h) right = mid; + else left = mid + 1; + } + return left; +} +``` + +**时间复杂度**：$O(n log max(piles))$ +**空间复杂度**：$O(1)$ + + +### [在 D 天内送达包裹的能力『1011』](https://leetcode.cn/problems/capacity-to-ship-packages-within-d-days/) + +> 给定包裹重量数组 `weights` 与天数 `days`，求最小运力使得能按顺序在 `days` 天内送达。 + +**思路**：对答案（运力）二分。下界是最大单件重量，上界是总重量。检验函数为用给定运力需要的天数是否 `<= days`。 + +```java +public int shipWithinDays(int[] weights, int days) { + int left = 0, right = 0; + for (int w : weights) { + left = Math.max(left, w); + right += w; + } + while (left < right) { + int mid = left + (right - left) / 2; + int need = 1, cur = 0; + for (int w : weights) { + if (cur + w > mid) { + need++; + cur = 0; + } + cur += w; + } + if (need <= days) right = mid; + else left = mid + 1; + } + return left; +} +``` + +**时间复杂度**：$O(n log(sum(weights)))$ +**空间复杂度**：$O(1)$ diff --git a/docs/data-structure-algorithms/algorithm/DFS-BFS.md b/docs/data-structure-algorithms/algorithm/DFS-BFS.md new file mode 100644 index 0000000000..658a24b592 --- /dev/null +++ b/docs/data-structure-algorithms/algorithm/DFS-BFS.md @@ -0,0 +1,627 @@ +--- +title: DFS 与 BFS +date: 2025-03-09 +tags: + - Algorithm +categories: BFS DFS +--- + +![](https://images.pexels.com/photos/3769697/pexels-photo-3769697.jpeg?auto=compress&cs=tinysrgb&w=1200) + +> 在线性结构中，按照顺序一个一个地看到所有的元素，称为线性遍历。在非线性结构中，由于元素之间的组织方式变得复杂，就有了不同的遍历行为。其中最常见的遍历有：**深度优先遍历**（Depth-First-Search）和**广度优先遍历**（Breadth-First-Search）。它们的思想非常简单，但是在算法的世界里发挥着巨大的作用，也是面试高频考点。 +> + +「遍历」和「搜索」可以看作是两个等价概念，通过遍历 **所有** 的可能的情况达到搜索的目的。遍历是手段，搜索是目的。因此「优先遍历」也叫「优先搜索」。 + + + +## 一、DFS 与 BFS的核心原理 + +![](https://img.starfish.ink/leetcode/DFS%20AND%20BFS.png) + +1. **DFS（深度优先搜索）** + + - 核心思想：优先沿一条路径深入探索，直到无法继续再回溯到上一个节点继续搜索，类似“不撞南墙不回头”。 + - **适用场景**：寻找所有可能路径、拓扑排序、连通性问题等。 + - **实现方式**：递归（隐式栈）或显式栈。 + + ```Java + void dfs(TreeNode node) { + if (node == null) return; + // 处理当前节点 + dfs(node.left); // 深入左子树 + dfs(node.right); // 深入右子树 + } + ``` + +2. **BFS（广度优先搜索）** + + - 核心思想：按层次逐层遍历，优先访问同一层的所有节点，常用于最短路径问题。 + - **适用场景**：层序遍历、最短路径（无权图）、扩散类问题。 + - **实现方式**：队列（FIFO）。 + + ```Java + void bfs(TreeNode root) { + Queue queue = new LinkedList<>(); + queue.offer(root); + while (!queue.isEmpty()) { + int size = queue.size(); + for (int i = 0; i < size; i++) { + TreeNode node = queue.poll(); + // 处理当前节点 + if (node.left != null) queue.offer(node.left); + if (node.right != null) queue.offer(node.right); + } + } + } + ``` + +只是比较两段代码的话，最直观的感受就是：DFS 遍历的代码比 BFS 简洁太多了！这是因为递归的方式隐含地使用了系统的 栈，我们不需要自己维护一个数据结构。如果只是简单地将二叉树遍历一遍，那么 DFS 显然是更方便的选择。 + + + +## 二、勇往直前的深度优先搜索 + +### 2.1 深度优先遍历的形象描述 + +「一条路走到底，不撞南墙不回头」是对「深度优先遍历」的最直观描述。 + +说明： + +- 深度优先遍历只要前面有可以走的路，就会一直向前走，直到无路可走才会回头； +- 「无路可走」有两种情况：① 遇到了墙；② 遇到了已经走过的路； +- 在「无路可走」的时候，沿着原路返回，直到回到了还有未走过的路的路口，尝试继续走没有走过的路径； +- 有一些路径没有走到，这是因为找到了出口，程序就停止了； +- 「深度优先遍历」也叫「深度优先搜索」，遍历是行为的描述，搜索是目的（用途）； +- 遍历不是很深奥的事情，把 **所有** 可能的情况都看一遍，才能说「找到了目标元素」或者「没找到目标元素」。遍历也称为 **穷举**，穷举的思想在人类看来虽然很不起眼，但借助 **计算机强大的计算能力**，穷举可以帮助我们解决很多专业领域知识不能解决的问题。 + + + +### 2.2 树的深度优先遍历 + +我们以「二叉树」的深度优先遍历为例，介绍树的深度优先遍历。 + +二叉树的深度优先遍历从「根结点」开始，依次 「递归地」 遍历「左子树」的所有结点和「右子树」的所有结点。 + +![](https://pic.leetcode-cn.com/1614684442-vUBRvf-image.png) + +> 事实上，「根结点 → 右子树 → 左子树」也是一种深度优先遍历的方式，为了符合人们「先左再右」的习惯。如果没有特别说明，树的深度优先遍历默认都按照 「根结点 → 左子树 → 右子树」 的方式进行。 + +**二叉树深度优先遍历的递归终止条件**：遍历完一棵树的 **所有** 叶子结点，等价于遍历到 **空结点**。 + +二叉树的深度优先遍历可以分为：前序遍历、中序遍历和后序遍历。 + +![img](https://writings.sh/assets/images/posts/binary-tree-traversal/binary-tree-dfs-order.jpg) + +- 前序遍历：根节点 → 左子树 → 右子树 +- 中序遍历： 左子树 → 根节点 → 右子树 +- 后序遍历：左子树 → 右子树 → 根节点 + +> 友情提示：后序遍历是非常重要的遍历方式，解决很多树的问题都采用了后序遍历的思想，请大家务必重点理解「后序遍历」一层一层向上传递信息的遍历方式。并在做题的过程中仔细体会「后序遍历」思想的应用。 + +**为什么前、中、后序遍历都是深度优先遍历** + +可以把树的深度优先遍历想象成一只蚂蚁，从根结点绕着树的外延走一圈。每一个结点的外延按照下图分成三个部分：前序遍历是第一部分，中序遍历是第二部分，后序遍历是第三部分。 + + + +**重要性质** + +根据定义不难得到以下性质。 + + - 性质 1：二叉树的 前序遍历 序列，根结点一定是 最先 访问到的结点； + - 性质 2：二叉树的 后序遍历 序列，根结点一定是 最后 访问到的结点； + - 性质 3：根结点把二叉树的 中序遍历 序列划分成两个部分，第一部分的所有结点构成了根结点的左子树，第二部分的所有结点构成了根结点的右子树。 + +> 根据这些性质，可以完成「力扣」第 105 题、第 106 题 + + + +### 2.3 图的深度优先遍历 + +深度优先遍历有「回头」的过程，在树中由于不存在「环」（回路），对于每一个结点来说，每一个结点只会被递归处理一次。而「图」中由于存在「环」（回路），就需要 记录已经被递归处理的结点（通常使用布尔数组或者哈希表），以免结点被重复遍历到。 + +**说明**：深度优先遍历的结果通常与图的顶点如何存储有关，所以图的深度优先遍历的结果并不唯一。 + +#### [课程表](https://leetcode.cn/problems/course-schedule/) + +> 你这个学期必须选修 `numCourses` 门课程，记为 `0` 到 `numCourses - 1` 。 +> +> 在选修某些课程之前需要一些先修课程。 先修课程按数组 `prerequisites` 给出，其中 `prerequisites[i] = [ai, bi]` ，表示如果要学习课程 `ai` 则 **必须** 先学习课程 `bi` 。 +> +> - 例如，先修课程对 `[0, 1]` 表示：想要学习课程 `0` ，你需要先完成课程 `1` 。 +> +> 请你判断是否可能完成所有课程的学习？如果可以，返回 `true` ；否则，返回 `false` 。 +> +> ``` +> 输入：numCourses = 2, prerequisites = [[1,0],[0,1]] +> 输出：false +> 解释：总共有 2 门课程。学习课程 1 之前，你需要先完成​课程 0 ；并且学习课程 0 之前，你还应先完成课程 1 。这是不可能的。 +> ``` + +思路：对于课程表问题，实际上是在寻找一个有向无环图（DAG）的环，以确定是否存在一个有效的课程学习顺序。 + +```java +public class Solution { + private boolean hasCycle = false; + + public boolean canFinish(int numCourses, int[][] prerequisites) { + // 构建图的邻接表表示 + List> graph = new ArrayList<>(); + for (int i = 0; i < numCourses; i++) { + graph.add(new ArrayList<>()); + } + for (int[] prerequisite : prerequisites) { + int course = prerequisite[0]; + int prerequisiteCourse = prerequisite[1]; + graph.get(prerequisiteCourse).add(course); + } + + // 初始化访问状态数组 + boolean[] visited = new boolean[numCourses]; + boolean[] recursionStack = new boolean[numCourses]; + + // 对每个节点进行DFS遍历 + for (int i = 0; i < numCourses; i++) { + if (!visited[i]) { + dfs(graph, visited, recursionStack, i); + } + // 如果在DFS过程中检测到了环，则提前返回false + if (hasCycle) { + return false; + } + } + + // 如果没有检测到环，则返回true + return true; + } + + private void dfs(List> graph, boolean[] visited, boolean[] recursionStack, int node) { + // 将当前节点标记为已访问 + visited[node] = true; + // 将当前节点加入递归栈 + recursionStack[node] = true; + + // 遍历当前节点的所有邻接节点 + for (int neighbor : graph.get(node)) { + // 如果邻接节点未被访问，则递归访问 + if (!visited[neighbor]) { + dfs(graph, visited, recursionStack, neighbor); + } else if (recursionStack[neighbor]) { + // 如果邻接节点已经在递归栈中，说明存在环 + hasCycle = true; + } + } + + // 将当前节点从递归栈中移除 + recursionStack[node] = false; + } +} +``` + + + +### 2.4 深度优先遍历的两种实现方式 + +在深度优先遍历的过程中，需要将 当前遍历到的结点 的相邻结点 暂时保存 起来，以便在回退的时候可以继续访问它们。遍历到的结点的顺序呈现「后进先出」的特点，因此 深度优先遍历可以通过「栈」实现。 + +再者，深度优先遍历有明显的递归结构。我们知道支持递归实现的数据结构也是栈。因此实现深度优先遍历有以下两种方式： + +- 编写递归方法； +- 编写栈，通过迭代的方式实现。 + +![image.png](https://pic.leetcode-cn.com/1608890373-CFNdrG-image.png) + +### 2.5 DFS 算法框架 + +#### **1. 基础模板（以全排列为例）** + +```java +// 全局变量：记录结果和路径 +List> result = new ArrayList<>(); +List path = new ArrayList<>(); +boolean[] visited; // 访问标记数组 + +void dfs(int[] nums, int depth) { + // 终止条件：路径长度达到要求 + if (depth == nums.length) { + result.add(new ArrayList<>(path)); + return; + } + + // 遍历选择列表 + for (int i = 0; i < nums.length; i++) { + if (!visited[i]) { + // 做选择：标记已访问，加入路径 + visited[i] = true; + path.add(nums[i]); + // 递归进入下一层 + dfs(nums, depth + 1); + // 撤销选择：回溯 + visited[i] = false; + path.remove(path.size() - 1); + } + } +} +``` + +**关键点**： + +- **路径记录**：通过`path`列表保存当前路径。 +- **访问标记**：使用`visited`数组避免重复访问。 +- **递归与回溯**：递归调用后必须撤销选择以恢复状态 + +#### **2. 二维矩阵遍历框架（如岛屿问题）** + +```java +void dfs(int[][] grid, int i, int j) { + // 边界检查 + if (i < 0 || j < 0 || i >= grid.length || j >= grid[0].length) return; + // 终止条件：遇到非陆地或已访问 + if (grid[i][j] != '1') return; + + // 标记为已访问（直接修改原矩阵） + grid[i][j] = '0'; + // 四个方向递归 + dfs(grid, i + 1, j); // 下 + dfs(grid, i - 1, j); // 上 + dfs(grid, i, j + 1); // 右 + dfs(grid, i, j - 1); // 左 +} +``` + +**适用场景**：矩阵中的连通性问题（如岛屿数量、迷宫路径） + + + +### 2.6 练习 + +> https://leetcode.cn/problem-list/depth-first-search/ + +请大家通过这些问题体会 「**如何设计递归函数的返回值**」 帮助我们解决问题。并理解这些简单的问题其实都是「深度优先遍历」的思想中「后序遍历」思想的体现，真正程序在执行的时候，是通过「一层一层向上汇报」的方式，最终在根结点汇总整棵树遍历的结果。 + +1. 完成「力扣」第 104 题：二叉树的最大深度（简单）：设计递归函数的返回值； +2. 完成「力扣」第 111 题：二叉树的最小深度（简单）：设计递归函数的返回值； +3. 完成「力扣」第 112 题：路径总和（简单）：设计递归函数的返回值； +4. 完成「力扣」第 226 题：翻转二叉树（简单）：前中后序遍历、广度优先遍历均可，中序遍历有一个小小的坑； +5. 完成「力扣」第 100 题：相同的树（简单）：设计递归函数的返回值； +6. 完成「力扣」第 101 题：对称二叉树（简单）：设计递归函数的返回值； +7. 完成「力扣」第 129 题：求根到叶子节点数字之和（中等）：设计递归函数的返回值。 +8. 完成「力扣」第 236 题：二叉树的最近公共祖先（中等）：使用后序遍历的典型问题。 + +请大家完成下面这些树中的问题，加深对前序遍历序列、中序遍历序列、后序遍历序列的理解。 + +9. 完成「力扣」第 105 题：从前序与中序遍历序列构造二叉树（中等）； +10. 完成「力扣」第 106 题：从中序与后序遍历序列构造二叉树（中等）； +11. 完成「力扣」第 1008 题：前序遍历构造二叉搜索树（中等）； + +12. 完成「力扣」第 1028 题：从先序遍历还原二叉树（困难）。 + +> 友情提示：需要用到后序遍历思想的一些经典问题，这些问题可能有一些难度，可以不用急于完成。先做后面的问题，见多了类似的问题以后，慢慢理解「后序遍历」一层一层向上汇报，在根结点汇总的遍历思想。 + + + +### 2.7 总结 + +- 遍历可以用于搜索，思想是穷举，遍历是实现搜索的手段； +- 树的「前、中、后」序遍历都是深度优先遍历； +- 树的后序遍历很重要； +- 由于图中存在环（回路），图的深度优先遍历需要记录已经访问过的结点，以避免重复访问； +- 遍历是一种简单、朴素但是很重要的算法思想，很多树和图的问题就是在树和图上执行一次遍历，在遍历的过程中记录有用的信息，得到需要结果，区别在于为了解决不同的问题，在遍历的时候传递了不同的 与问题相关 的数据。 + + + + + +## 三、齐头并进的广度优先搜索 + +> DFS（深度优先搜索）和 BFS（广度优先搜索）就像孪生兄弟，提到一个总是想起另一个。然而在实际使用中，我们用 DFS 的时候远远多于 BFS。那么，是不是 BFS 就没有什么用呢？ +> +> 如果我们使用 DFS/BFS 只是为了遍历一棵树、一张图上的所有结点的话，那么 DFS 和 BFS 的能力没什么差别，我们当然更倾向于更方便写、空间复杂度更低的 DFS 遍历。不过，某些使用场景是 DFS 做不到的，只能使用 BFS 遍历。 +> + +![image.png](https://pic.leetcode-cn.com/1611483686-FJqdzm-image.png) + +「广度优先遍历」的思想在生活中随处可见： + +如果我们要找一个医生或者律师，我们会先在自己的一度人脉中遍历（查找），如果没有找到，继续在自己的二度人脉中遍历（查找），直到找到为止。 + +### 3.1 广度优先遍历借助「队列」实现 + +广度优先遍历呈现出「一层一层向外扩张」的特点，**先看到的结点先遍历，后看到的结点后遍历**，因此「广度优先遍历」可以借助「队列」实现。 + +![11-01-05.gif](https://pic.leetcode-cn.com/1609663109-jjxZav-11-01-05.gif) + +**说明**：遍历到一个结点时，如果这个结点有左（右）孩子结点，依次将它们加入队列。 + +> 友情提示：广度优先遍历的写法相对固定，我们不建议大家背代码、记模板。在深刻理解广度优先遍历的应用场景（找无权图的最短路径），借助「队列」实现的基础上，多做练习，写对代码就是自然而然的事情了 + +我们先介绍「树」的广度优先遍历，再介绍「图」的广度优先遍历。事实上，它们是非常像的。 + + + +### 3.2 树的广度优先遍历 + +二叉树的层序遍历 + +> 给你一个二叉树，请你返回其按 层序遍历 得到的节点值。 （即逐层地，从左到右访问所有节点）。 +> + +思路分析： + +- 题目要求我们一层一层输出树的结点的值，很明显需要使用「广度优先遍历」实现； +- 广度优先遍历借助「队列」实现； + +- 注意： + - 这样写 `for (int i = 0; i < queue.size(); i++) { `代码是不能通过测评的，这是因为 `queue.size()` 在循环中是变量。正确的做法是：每一次在队列中取出元素的个数须要先暂存起来； + - 子结点入队的时候，非空的判断很重要：在队列的队首元素出队的时候，一定要在左（右）子结点非空的时候才将左（右）子结点入队。 +- 树的广度优先遍历的写法模式相对固定： + - 使用队列； + - 在队列非空的时候，动态取出队首元素； + - 取出队首元素的时候，把队首元素相邻的结点（非空）加入队列。 + +大家在做题的过程中需要多加练习，融汇贯通，不须要死记硬背。 + +![img](https://pic.leetcode-cn.com/94cd1fa999df0276f1dae77a9cca83f4cabda9e2e0b8571cd9550a8ee3545f56.gif) + +```java +public List> levelOrder(TreeNode root) { + List> result = new ArrayList<>(); // 存储遍历结果的列表 + if (root == null) { + return result; // 如果树为空，直接返回空列表 + } + + Queue queue = new LinkedList<>(); // 创建一个队列用于层序遍历 + queue.offer(root); // 将根节点加入队列 + + while (!queue.isEmpty()) { // 当队列不为空时继续遍历 + int levelSize = queue.size(); // 当前层的节点数 + List currentLevel = new ArrayList<>(); // 存储当前层节点值的列表 + + for (int i = 0; i < levelSize; i++) { + TreeNode currentNode = queue.poll(); // 从队列中取出一个节点 + currentLevel.add(currentNode.val); // 将节点值加入当前层列表 + + // 如果左子节点不为空，将其加入队列 + if (currentNode.left != null) { + queue.offer(currentNode.left); + } + // 如果右子节点不为空，将其加入队列 + if (currentNode.right != null) { + queue.offer(currentNode.right); + } + } + + result.add(currentLevel); // 将当前层列表加入结果列表 + } + + return result; // 返回遍历结果 +} +``` + + + +### 3.3 BFS 算法框架 + +**基础模板（队列+访问标记）** + +```java +public int bfs(Node start, Node target) { + Queue queue = new LinkedList<>(); // 核心队列结构 + Set visited = new HashSet<>(); // 防止重复访问 + int step = 0; // 记录扩散步数 + + queue.offer(start); + visited.add(start); + + while (!queue.isEmpty()) { + int levelSize = queue.size(); // 当前层节点数 + for (int i = 0; i < levelSize; i++) { // 遍历当前层所有节点 + Node cur = queue.poll(); + // 终止条件（根据问题场景调整） + if (cur.equals(target)) return step; + + // 扩散相邻节点（根据数据结构调整） + for (Node neighbor : getNeighbors(cur)) { + if (!visited.contains(neighbor)) { + queue.offer(neighbor); + visited.add(neighbor); + } + } + } + step++; // 步数递增 + } + return -1; // 未找到目标 +} +``` + +**关键点** ： + +- **队列控制层次**：通过 `levelSize` 逐层遍历，保证找到最短路径。 +- **访问标记**：避免重复访问（如矩阵问题可改为修改原数据）。 +- **扩散逻辑**：`getNeighbors()` 需根据具体数据结构实现（如二叉树、图、网格等）。 + + + +### 3.4 使用广度优先遍历得到无权图的最短路径 + +在 无权图 中，由于广度优先遍历本身的特点，假设源点为 source，只有在遍历到 所有 距离源点 source 的距离为 d 的所有结点以后，才能遍历到所有 距离源点 source 的距离为 d + 1 的所有结点。也可以使用「两点之间、线段最短」这条经验来辅助理解如下结论：从源点 source 到目标结点 target 走直线走过的路径一定是最短的。 + +> 在一棵树中，一个结点到另一个结点的路径是唯一的，但在图中，结点之间可能有多条路径，其中哪条路最近呢？这一类问题称为最短路径问题。最短路径问题也是 BFS 的典型应用，而且其方法与层序遍历关系密切。 +> +> 在二叉树中，BFS 可以实现一层一层的遍历。在图中同样如此。从源点出发，BFS 首先遍历到第一层结点，到源点的距离为 1，然后遍历到第二层结点，到源点的距离为 2…… 可以看到，用 BFS 的话，距离源点更近的点会先被遍历到，这样就能找到到某个点的最短路径了。 +> +> ![层序遍历与最短路径](https://pic.leetcode-cn.com/01a3617511b1070216582ae59136888072116ccba360ab7c2aa60fc273351b85.jpg) +> +> 小贴士： +> +> 很多同学一看到「最短路径」，就条件反射地想到「Dijkstra 算法」。为什么 BFS 遍历也能找到最短路径呢？ +> +> 这是因为，Dijkstra 算法解决的是带权最短路径问题，而我们这里关注的是无权最短路径问题。也可以看成每条边的权重都是 1。这样的最短路径问题，用 BFS 求解就行了。 +> +> 在面试中，你可能更希望写 BFS 而不是 Dijkstra。毕竟，敢保证自己能写对 Dijkstra 算法的人不多。 +> +> 最短路径问题属于图算法。由于图的表示和描述比较复杂，本文用比较简单的网格结构代替。网格结构是一种特殊的图，它的表示和遍历都比较简单，适合作为练习题。在 LeetCode 中，最短路径问题也以网格结构为主。 + + + +### 3.5 图论中的最短路径问题概述 + +在图中，由于 图中存在环，和深度优先遍历一样，广度优先遍历也需要在遍历的时候记录已经遍历过的结点。特别注意：将结点添加到队列以后，一定要马上标记为「已经访问」，否则相同结点会重复入队，这一点在初学的时候很容易忽略。如果很难理解这样做的必要性，建议大家在代码中打印出队列中的元素进行调试：在图中，如果入队的时候不马上标记为「已访问」，相同的结点会重复入队，这是不对的。 + +另外一点还需要强调，广度优先遍历用于求解「无权图」的最短路径，因此一定要认清「无权图」这个前提条件。如果是带权图，就需要使用相应的专门的算法去解决它们。事实上，这些「专门」的算法的思想也都基于「广度优先遍历」的思想，我们为大家例举如下： + +- 带权有向图、且所有权重都非负的单源最短路径问题：使用 Dijkstra 算法； +- 带权有向图的单源最短路径问题：Bellman-Ford 算法； + +- 一个图的所有结点对的最短路径问题：Floy-Warshall 算法。 + +这里列出的以三位计算机科学家的名字命名的算法，大家可以在《算法导论》这本经典著作的第 24 章、第 25 章找到相关知识的介绍。值得说明的是：应用任何一种算法，都需要认清使用算法的前提，不满足前提直接套用算法是不可取的。深刻理解应用算法的前提，也是学习算法的重要方法。例如我们在学习「二分查找」算法、「滑动窗口」算法的时候，就可以问自己，这个问题为什么可以使用「二分查找」，为什么可以使用「滑动窗口」。我们知道一个问题可以使用「优先队列」解决，是什么样的需求促使我们想到使用「优先队列」，而不是「红黑树（平衡二叉搜索树）」，想清楚使用算法（数据结构）的前提更重要。 + +> [无向图中连通分量的数目『323』](https://leetcode.cn/problems/number-of-connected-components-in-an-undirected-graph/) +> +> 你有一个包含 `n` 个节点的图。给定一个整数 `n` 和一个数组 `edges` ，其中 `edges[i] = [ai, bi]` 表示图中 `ai` 和 `bi` 之间有一条边。 +> +> 返回 *图中已连接分量的数目* 。 +> +> ![img](https://assets.leetcode.com/uploads/2021/03/14/conn1-graph.jpg) +> +> ``` +> 输入: n = 5, edges = [[0, 1], [1, 2], [3, 4]] +> 输出: 2 +> ``` +> +> 思路分析： +> +> 首先需要对输入数组进行处理，由于 n 个结点的编号从 0 到 n - 1 ，因此可以使用「嵌套数组」表示邻接表，具体实现请见参考代码； +> 然后遍历每一个顶点，对每一个顶点执行一次广度优先遍历，注意：在遍历的过程中使用 visited 布尔数组记录已经遍历过的结点。 +> +> ```java +> public int countComponents(int n, int[][] edges) { +> // 第 1 步：构建图 +> List[] adj = new ArrayList[n]; +> for (int i = 0; i < n; i++) { +> adj[i] = new ArrayList<>(); +> } +> // 无向图，所以需要添加双向引用 +> for (int[] edge : edges) { +> adj[edge[0]].add(edge[1]); +> adj[edge[1]].add(edge[0]); +> } +> +> // 第 2 步：开始广度优先遍历 +> int res = 0; +> boolean[] visited = new boolean[n]; +> for (int i = 0; i < n; i++) { +> if (!visited[i]) { +> bfs(adj, i, visited); +> res++; +> } +> } +> return res; +> } +> +> /** +> * @param adj 邻接表 +> * @param u 从 u 这个顶点开始广度优先遍历 +> * @param visited 全局使用的 visited 布尔数组 +> */ +> private void bfs(List[] adj, int u, boolean[] visited) { +> Queue queue = new LinkedList<>(); +> queue.offer(u); +> visited[u] = true; +> +> while (!queue.isEmpty()) { +> Integer front = queue.poll(); +> // 获得队首结点的所有后继结点 +> List successors = adj[front]; +> for (int successor : successors) { +> if (!visited[successor]) { +> queue.offer(successor); +> // 特别注意：在加入队列以后一定要将该结点标记为访问，否则会出现结果重复入队的情况 +> visited[successor] = true; +> } +> } +> } +> } +> ``` +> +> 复杂度分析： +> +> - 时间复杂度：O(V + E)O(V+E)，这里 EE 是边的条数，即数组 edges 的长度，初始化的时候遍历数组得到邻接表。这里 VV 为输入整数 n，遍历的过程是每一个结点执行一次深度优先遍历，时间复杂度为 O(V)O(V)； +> - 空间复杂度：O(V + E)O(V+E)，综合考虑邻接表 O(V + E)O(V+E)、visited 数组 O(V)O(V)、队列的长度 O(V)O(V) 三者得到。 +> 说明：和深度优先遍历一样，图的广度优先遍历的结果并不唯一，与每个结点的相邻结点的访问顺序有关。 + +### 3.6 练习 + +> 友情提示：第 1 - 4 题是广度优先遍历的变形问题，写对这些问题有助于掌握广度优先遍历的代码编写逻辑和细节。 + +1. 完成「力扣」第 107 题：二叉树的层次遍历 II（简单）； +2. 完成《剑指 Offer》第 32 - I 题：从上到下打印二叉树（中等）； +3. 完成《剑指 Offer》第 32 - III 题：从上到下打印二叉树 III（中等）； +4. 完成「力扣」第 103 题：二叉树的锯齿形层次遍历（中等）； +5. 完成「力扣」第 429 题：N 叉树的层序遍历（中等）； +6. 完成「力扣」第 993 题：二叉树的堂兄弟节点（中等）； + + + +#### 二维矩阵遍历（岛屿问题） + +```java +void bfs(char[][] grid, int x, int y) { + int[][] dirs = {{1,0}, {-1,0}, {0,1}, {0,-1}}; + Queue queue = new LinkedList<>(); + queue.offer(new int[]{x, y}); + grid[x][y] = '0'; // 直接修改矩阵代替visited + + while (!queue.isEmpty()) { + int[] pos = queue.poll(); + for (int[] d : dirs) { + int nx = pos[0] + d[0], ny = pos[1] + d[1]; + if (nx >=0 && ny >=0 && nx < grid.length && ny < grid[0].length + && grid[nx][ny] == '1') { + queue.offer(new int[]{nx, ny}); + grid[nx][ny] = '0'; // 标记为已访问 + } + } + } +} +``` + +**优化点** ： + +- 直接修改原矩阵代替`visited`集合，节省空间。 +- 四方向扩散的坐标计算。 + +#### 图的邻接表遍历 + +```java +public void bfsGraph(Map> graph, int start) { + Queue queue = new LinkedList<>(); + boolean[] visited = new boolean[graph.size()]; + queue.offer(start); + visited[start] = true; + + while (!queue.isEmpty()) { + int node = queue.poll(); + System.out.print(node + " "); + for (int neighbor : graph.get(node)) { + if (!visited[neighbor]) { + queue.offer(neighbor); + visited[neighbor] = true; + } + } + } +} +``` + +**数据结构** ： + +- 使用`Map`或邻接表存储图结构。 +- 通过布尔数组标记访问状态。 + + + +## Reference + +- https://leetcode-cn.com/problems/binary-tree-level-order-traversal/solution/bfs-de-shi-yong-chang-jing-zong-jie-ceng-xu-bian-l/ diff --git a/docs/data-structure-algorithms/algorithm/Double-Pointer.md b/docs/data-structure-algorithms/algorithm/Double-Pointer.md new file mode 100755 index 0000000000..69bcd7b253 --- /dev/null +++ b/docs/data-structure-algorithms/algorithm/Double-Pointer.md @@ -0,0 +1,1466 @@ +--- +title: 双指针 +date: 2023-05-17 +tags: + - pointers + - algorithms +categories: algorithms +--- + +![](https://img.starfish.ink/leetcode/two-pointer-banner.png) + +> 在数组中并没有真正意义上的指针，但我们可以把索引当做数组中的指针 +> +> 归纳下双指针算法，其实总共就三类 +> +> - 左右指针，数组和字符串问题 +> - 快慢指针，主要是成环问题 +> - 滑动窗口，针对子串问题 + + + +## 一、左右指针 + +![](https://img.starfish.ink/leetcode/two-point.png) + +左右指针在数组中其实就是两个索引值，两个指针相向而行或者相背而行 + +Javaer 一般这么表示： + +```java +int left = 0; +int right = arr.length - 1; +while(left < right) + *** +``` + +这两个指针 **相向交替移动**， 看着像二分查找是吧，二分也属于左右指针。 + + + +### [反转字符串](https://leetcode.cn/problems/reverse-string/) + +> 编写一个函数，其作用是将输入的字符串反转过来。输入字符串以字符数组 char[] 的形式给出。 +> +> 不要给另外的数组分配额外的空间，你必须原地修改输入数组、使用 O(1) 的额外空间解决这一问题。 +> +> 你可以假设数组中的所有字符都是 ASCII 码表中的可打印字符。 +> +> ``` +> 输入：["h","e","l","l","o"] +> 输出：["o","l","l","e","h"] +> ``` +> + +思路： + +- 因为要反转，所以就不需要相向移动了，如果用双指针思路的话，其实就是遍历中交换左右指针的字符 + +```java +public void reverseString(char[] s) { + int left = 0; + int right = s.length - 1; + while (left < right){ + char tmp = s[left]; + s[left] = s[right]; + s[right] = tmp; + left++; + right--; + } +} +``` + + + +### [两数之和 II - 输入有序数组](https://leetcode.cn/problems/two-sum-ii-input-array-is-sorted/) + +> 给你一个下标从 1 开始的整数数组 numbers ，该数组已按 非递减顺序排列 ，请你从数组中找出满足相加之和等于目标数 target 的两个数。如果设这两个数分别是 numbers[index1] 和 numbers[index2] ，则 1 <= index1 < index2 <= numbers.length 。 +> +> 以长度为 2 的整数数组 [index1, index2] 的形式返回这两个整数的下标 index1 和 index2。 +> +> 你可以假设每个输入 只对应唯一的答案 ，而且你 不可以 重复使用相同的元素。 +> +> 你所设计的解决方案必须只使用常量级的额外空间。 +> +> ``` +> 输入：numbers = [2,7,11,15], target = 9 +> 输出：[1,2] +> 解释：2 与 7 之和等于目标数 9 。因此 index1 = 1, index2 = 2 。返回 [1, 2] 。 +> ``` + +直接用左右指针套就可以 + +```java +public int[] twoSum(int[] nums, int target) { + int left = 0; + int rigth = nums.length - 1; + while (left < rigth) { + int tmp = nums[left] + nums[rigth]; + if (target == tmp) { + //数组下标是从1开始的 + return new int[]{left + 1, rigth + 1}; + } else if (tmp > target) { + rigth--; //右移 + } else { + left++; //左移 + } + } + return new int[]{-1, -1}; +} +``` + + + +### [三数之和](https://leetcode.cn/problems/3sum/) + +> 给你一个整数数组 nums ，判断是否存在三元组 [nums[i], nums[j], nums[k]] 满足 i != j、i != k 且 j != k ，同时还满足 nums[i] + nums[j] + nums[k] == 0 。请你返回所有和为 0 且不重复的三元组。 +> +> 注意：答案中不可以包含重复的三元组。 +> + +思路：**排序、双指针、去重** + +第一个想法是，这三个数，两个指针？ + +- 对数组排序，固定一个数 $nums[i]$ ，然后遍历数组，并移动左右指针求和，判断是否有等于 0 的情况 + +- 特例： + - 排序后第一个数就大于 0，不干了 + + - 有三个需要去重的地方 + - `nums[i] == nums[i - 1]` 直接跳过本次遍历 + + > **避免重复三元组：** + > + > - 我们从第一个元素开始遍历数组，逐步往后移动。如果当前的 `nums[i]` 和前一个 `nums[i - 1]` 相同，说明我们已经处理过以 `nums[i - 1]` 为起点的组合（即已经找过包含 `nums[i - 1]` 的三元组），此时再处理 `nums[i]` 会导致生成重复的三元组，因此可以跳过。 + > - 如果我们检查 `nums[i] == nums[i + 1]`，由于 `nums[i + 1]` 还没有被处理，这种方式无法避免重复，并且会产生错误的逻辑。 + + - `nums[left] == nums[left + 1]` 移动指针，即去重 + + - `nums[right] == nums[right - 1]` 移动指针 + + > **避免重复的配对：** + > + > 在每次固定一个 `nums[i]` 后，剩下的两数之和问题通常使用双指针法来解决。双指针的左右指针 `left` 和 `right` 分别从数组的两端向中间逼近，寻找合适的配对。 + > + > 为了**避免相同的数字被重复使用**，导致重复的三元组，双指针法中也需要跳过相同的元素。 + > + > - 左指针跳过重复元素： + > - 如果 `nums[left] == nums[left + 1]`，说明接下来的数字与之前处理过的数字相同。为了避免生成相同的三元组，我们将 `left` 向右移动跳过这个重复的数字。 + > - 右指针跳过重复元素： + > - 同样地，`nums[right] == nums[right - 1]` 也会导致重复的配对，因此右指针也要向左移动，跳过这个重复数字。 + +```java +public List> threeSum(int[] nums) { + //存放结果list + List> result = new ArrayList<>(); + int length = nums.length; + //特例判断 + if (length < 3) { + return result; + } + Arrays.sort(nums); + for (int i = 0; i < length; i++) { + //排序后的第一个数字就大于0，就说明没有符合要求的结果 + if (nums[i] > 0) break; + + //去重, 不能是 nums[i] == nums[i +1 ]，因为顺序遍历的逻辑使得前一个元素已经被处理过，而后续的元素还没有处理 + if (i > 0 && nums[i] == nums[i - 1]) continue; + //左右指针 + int l = i + 1; + int r = length - 1; + while (l < r) { + int sum = nums[i] + nums[l] + nums[r]; + if (sum == 0) { + result.add(Arrays.asList(nums[i], nums[l], nums[r])); + //去重（相同数字的话就移动指针） + //在将左指针和右指针移动的时候，先对左右指针的值，进行判断,以防[0,0,0]这样的造成数组越界 + //不要用成 if 判断，只跳过 1 条，还会有重复的，且需要再加上 l 0) r--; + } + } + return result; +} +``` + + + +### 盛最多水的容器 + +> 给你 n 个非负整数 a1，a2，...，an，每个数代表坐标中的一个点 (i, ai) 。在坐标内画 n 条垂直线，垂直线 i 的两个端点分别为 (i, ai) 和 (i, 0) 。找出其中的两条线，使得它们与 x 轴共同构成的容器可以容纳最多的水。 +> +> ``` +> 输入：[1,8,6,2,5,4,8,3,7] +> 输出：49 +> 解释：图中垂直线代表输入数组 [1,8,6,2,5,4,8,3,7]。在此情况下，容器能够容纳水（表示为蓝色部分）的最大值为 49。 +> ``` +> +> ![](https://aliyun-lc-upload.oss-cn-hangzhou.aliyuncs.com/aliyun-lc-upload/uploads/2018/07/25/question_11.jpg) + +**思路**： + +- 求得是水量，水量 = 两个指针指向的数字中较小值 * 指针之间的距离（水桶原理，最短的板才不会漏水） +- 为了求最大水量，我们需要存储所有条件的水量，进行比较才行 +- **双指针相向移动**，循环收窄，直到两个指针相遇 +- 往哪个方向移动，需要考虑清楚，如果我们移动数字较大的那个指针，那么前者「两个指针指向的数字中较小值」不会增加，后者「指针之间的距离」会减小，那么这个乘积会更小，所以我们移动**数字较小的那个指针** + +```java +public int maxArea(int[] height){ + int left = 0; + int right = height.length - 1; + //需要保存各个阶段的值 + int result = 0; + while(left < right){ + //水量 = 两个指针指向的数字中较小值∗指针之间的距离 + int area = Math.min(height[left],height[right]) * (right - left); + result = Math.max(result,area); + //移动数字较小的指针 + if(height[left] <= height[right]){ + left ++; + }else{ + right--; + } + } + return result; +} +``` + + + +### [验证回文串](https://leetcode.cn/problems/valid-palindrome/) + +> 如果在将所有大写字符转换为小写字符、并移除所有非字母数字字符之后，短语正着读和反着读都一样。则可以认为该短语是一个 回文串 。 +> +> 字母和数字都属于字母数字字符。 +> +> 给你一个字符串 s，如果它是 回文串 ，返回 true ；否则，返回 false 。 +> +> ``` +> 输入: "A man, a plan, a canal: Panama" +> 输出: true +> 解释："amanaplanacanalpanama" 是回文串 +> ``` + +思路： + +- 没看题解前，因为这个例子中有各种逗号、空格啥的，我第一想到的其实就是先遍历放在一个数组里，然后再去判断，看题解可以在原字符串完成，降低了空间复杂度 +- 首先需要知道三个 API + - `Character.isLetterOrDigit` 确定指定的字符是否为字母或数字 + - `Character.toLowerCase` 将大写字符转换为小写 + - `public char charAt(int index)` String 中的方法，用于返回指定索引处的字符 +- 双指针，每移动一步，判断这两个值是不是相同 +- 两个指针相遇，则是回文串 + +```java +public boolean isPalindrome(String s) { + // 转换为小写并去掉非字母和数字的字符 + int left = 0, right = s.length() - 1; + + while (left < right) { + // 忽略左边非字母和数字字符 + while (left < right && !Character.isLetterOrDigit(s.charAt(left))) { + left++; + } + // 忽略右边非字母和数字字符 + while (left < right && !Character.isLetterOrDigit(s.charAt(right))) { + right--; + } + // 比较两边字符 + if (Character.toLowerCase(s.charAt(left)) != Character.toLowerCase(s.charAt(right))) { + return false; + } + left++; + right--; + } + return true; +} +``` + + + +### 二分查找 + +有重复数字的话，返回的其实就是最右匹配 + +```java +public static int search(int[] nums, int target) { + int left = 0; + int right = nums.length - 1; + while (left <= right) { + //不直接使用（right+left）/2 是考虑数据大的时候溢出 + int mid = (right - left) / 2 + left; + int tmp = nums[mid]; + if (tmp == target) { + return mid; + } else if (tmp > target) { + //右指针移到中间位置 - 1,也避免不存在的target造成死循环 + right = mid - 1; + } else { + // + left = mid + 1; + } + } + return -1; +} +``` + + + +## 二、快慢指针 + +「快慢指针」，也称为「同步指针」，所谓快慢指针，就是两个指针同向而行，一快一慢。快慢指针处理的大都是链表问题。 + +![](https://img.starfish.ink/leetcode/fast-slow-point.png) + +### [环形链表](https://leetcode-cn.com/problems/linked-list-cycle/) + +> 给你一个链表的头节点 head ，判断链表中是否有环。 +> +> 如果链表中有某个节点，可以通过连续跟踪 next 指针再次到达，则链表中存在环。 为了表示给定链表中的环，评测系统内部使用整数 pos 来表示链表尾连接到链表中的位置（索引从 0 开始）。注意：pos 不作为参数进行传递 。仅仅是为了标识链表的实际情况。 +> +> 如果链表中存在环 ，则返回 true 。 否则，返回 false 。 +> +> ![](https://assets.leetcode-cn.com/aliyun-lc-upload/uploads/2018/12/07/circularlinkedlist.png) + +思路： + +- 快慢指针，两个指针，一快一慢的话，慢指针每次只移动一步，而快指针每次移动两步。初始时，慢指针在位置 head，而快指针在位置 head.next。这样一来，如果在移动的过程中，快指针反过来追上慢指针，就说明该链表为环形链表。否则快指针将到达链表尾部，该链表不为环形链表。 + +```java +public boolean hasCycle(ListNode head) { + if (head == null || head.next == null) { + return false; + } + // 龟兔起跑 + ListNode fast = head; + ListNode slow = head; + + while (fast != null && fast.next != null) { + // 龟走一步 + slow = slow.next; + // 兔走两步 + fast = fast.next.next; + if (slow == fast) { + return true; + } + } + return false; +} +``` + + + +### [环形链表II](https://leetcode-cn.com/problems/linked-list-cycle-ii) + +> 给定一个链表，返回链表开始入环的第一个节点。 如果链表无环，则返回 `null`。 +> +> ``` +> 输入：head = [3,2,0,-4], pos = 1 +> 输出：返回索引为 1 的链表节点 +> 解释：链表中有一个环，其尾部连接到第二个节点。 +> ``` + +思路： + +- 最初，我就把有环理解错了，看题解觉得快慢指针相交的地方就是入环的节点 + +- 假设环是这样的，slow 指针进入环后，又走了 b 的距离与 fast 相遇 + + ![fig1](https://assets.leetcode-cn.com/solution-static/142/142_fig1.png) + +1. **检测是否有环**：通过快慢指针来判断链表中是否存在环。慢指针一次走一步，快指针一次走两步。如果链表中有环，两个指针最终会相遇；如果没有环，快指针会到达链表末尾。 + +2. **找到环的起点**： +- 当快慢指针相遇时，我们已经确认链表中存在环。 + +- 从相遇点开始，慢指针保持不动，快指针回到链表头部，此时两个指针每次都走一步。两个指针会在环的起点再次相遇。 + +```java +public ListNode detectCycle(ListNode head) { + if (head == null || head.next == null) { + return null; + } + + ListNode slow = head; + ListNode fast = head; + + // 判断是否有环 + while (fast != null && fast.next != null) { + slow = slow.next; + fast = fast.next.next; + // 快慢指针相遇，说明有环 + if (slow == fast) { + break; + } + } + + // 如果没有环 + if (fast == null || fast.next == null) { + return null; + } + + // 快指针回到起点，慢指针保持在相遇点 + fast = head; + while (fast != slow) { + fast = fast.next; + slow = slow.next; + } + + // 此时快慢指针相遇的地方就是环的起点 + return slow; +} +``` + + + +### [链表的中间结点](https://leetcode.cn/problems/middle-of-the-linked-list/) + +> 给定一个头结点为 `head` 的非空单链表，返回链表的中间结点。 +> +> 如果有两个中间结点，则返回第二个中间结点。(给定链表的结点数介于 `1` 和 `100` 之间。) +> +> ``` +> 输入：[1,2,3,4,5] +> 输出：此列表中的结点 3 (序列化形式：[3,4,5]) +> 返回的结点值为 3 。 (测评系统对该结点序列化表述是 [3,4,5])。 +> 注意，我们返回了一个 ListNode 类型的对象 ans，这样： +> ans.val = 3, ans.next.val = 4, ans.next.next.val = 5, 以及 ans.next.next.next = NULL. +> ``` + +思路： + +- 快慢指针遍历，当 `fast` 到达链表的末尾时，`slow` 必然位于中间 + +```java +public ListNode middleNode(ListNode head) { + ListNode fast = head; + ListNode slow = head; + while (fast != null && fast.next != null) { + slow = slow.next; + fast = fast.next.next; + } + return slow; +} +``` + + + +### [ 回文链表](https://leetcode.cn/problems/palindrome-linked-list/) + +> 给你一个单链表的头节点 head ，请你判断该链表是否为回文链表。如果是，返回 true ；否则，返回 false +> +> ``` +> 输入：head = [1,2,2,1] +> 输出：true +> ``` + +思路： + +- 双指针：将值复制到数组中后用双指针法 +- 或者使用快慢指针来确定中间结点，然后反转后半段链表，将前半部分链表和后半部分进行比较 + +```java +public boolean isPalindrome(ListNode head) { + List vals = new ArrayList(); + + // 将链表的值复制到数组中 + ListNode currentNode = head; + while (currentNode != null) { + vals.add(currentNode.val); + currentNode = currentNode.next; + } + + // 使用双指针判断是否回文 + int front = 0; + int back = vals.size() - 1; + while (front < back) { + if (!vals.get(front).equals(vals.get(back))) { + return false; + } + front++; + back--; + } + return true; +} +``` + + + +### 删除链表的倒数第 N 个结点 + +> 给你一个链表，删除链表的倒数第 n 个结点，并且返回链表的头结点。 +> +> ``` +> 输入：head = [1,2,3,4,5], n = 2 +> 输出：[1,2,3,5] +> ``` + +思路： + +1. 计算链表长度：从头节点开始对链表进行一次遍历，得到链表的长度，随后我们再从头节点开始对链表进行一次遍历，当遍历到第 L−n+1 个节点时，它就是我们需要删除的节点（为了与题目中的 n 保持一致，节点的编号从 1 开始） +2. 栈：根据栈「先进后出」的原则，我们弹出栈的第 n 个节点就是需要删除的节点，并且目前栈顶的节点就是待删除节点的前驱节点 +3. 双指针：由于我们需要找到倒数第 n 个节点，因此我们可以使用两个指针 first 和 second 同时对链表进行遍历，并且 first 比 second 超前 n 个节点。当 first 遍历到链表的末尾时，second 就恰好处于倒数第 n 个节点。 + +```java +public ListNode removeNthFromEnd(ListNode head, int n) { + ListNode dummy = new ListNode(0, head); + ListNode first = head; + ListNode second = dummy; + + //让 first 指针先移动 n 步 + for (int i = 0; i < n; ++i) { + first = first.next; + } + while (first != null) { + first = first.next; + second = second.next; + } + second.next = second.next.next; + return dummy.next; +} +``` + + + +### [删除有序数组中的重复项](https://leetcode.cn/problems/remove-duplicates-from-sorted-array/) + +> 给你一个 升序排列 的数组 nums ，请你 原地 删除重复出现的元素，使每个元素 只出现一次 ，返回删除后数组的新长度。元素的 相对顺序 应该保持 一致 。 +> +> 由于在某些语言中不能改变数组的长度，所以必须将结果放在数组nums的第一部分。更规范地说，如果在删除重复项之后有 k 个元素，那么 nums 的前 k 个元素应该保存最终结果。 +> +> 将最终结果插入 nums 的前 k 个位置后返回 k 。 +> +> 不要使用额外的空间，你必须在 原地 修改输入数组 并在使用 O(1) 额外空间的条件下完成。 +> +> ``` +> 输入：nums = [1,1,2] +> 输出：2, nums = [1,2,_] +> 解释：函数应该返回新的长度 2 ，并且原数组 nums 的前两个元素被修改为 1, 2 。不需要考虑数组中超出新长度后面的元素。 +> ``` + +**思路**： + +- 数组有序，那相等的元素在数组中的下标一定是连续的 +- 使用快慢指针，快指针表示遍历数组到达的下标位置，慢指针表示下一个不同元素要填入的下标位置 +- 第一个元素不需要删除，所有快慢指针都从下标 1 开始 + +```java +public static int removeDuplicates(int[] nums) { + if (nums == null) { + return 0; + } + int fast = 1; + int slow = 1; + while (fast < nums.length) { + //和前一个值比较 + if (nums[fast] != nums[fast - 1]) { + //不一样的话，把快指针的值放在慢指针上，实现了去重，并往前移动慢指针 + nums[slow] = nums[fast]; + ++slow; + } + //相等的话，移动快指针就行 + ++fast; + } + //慢指针的位置就是不重复的数量 + return slow; +} +``` + + + +### 最长连续递增序列 + +> 给定一个未经排序的整数数组，找到最长且 连续递增的子序列，并返回该序列的长度。 +> +> 连续递增的子序列 可以由两个下标 l 和 r（l < r）确定，如果对于每个 l <= i < r，都有 nums[i] < nums[i + 1] ，那么子序列 [nums[l], nums[l + 1], ..., nums[r - 1], nums[r]] 就是连续递增子序列。 +> +> ``` +> 输入：nums = [1,3,5,4,7] +> 输出：3 +> 解释：最长连续递增序列是 [1,3,5], 长度为3。尽管 [1,3,5,7] 也是升序的子序列, 但它不是连续的，因为 5 和 7 在原数组里被 4 隔开。 +> ``` + +思路分析： + +- 这个题的思路和删除有序数组中的重复项，很像 + +```java +public int findLengthOfLCIS(int[] nums) { + int result = 0; + int fast = 0; + int slow = 0; + while (fast < nums.length) { + //前一个数大于后一个数的时候 + if (fast > 0 || nums[fast - 1] > nums[fast]) { + slow = fast; + } + fast++; + result = Math.max(result, fast - slow); + } + return result; +} +``` + + + +## 三、滑动窗口 + +有一类数组上的问题，需要使用两个指针变量（我们称为左指针和右指针），同向、交替向右移动完成任务。这样的过程像极了一个窗口在平面上滑动的过程，因此我们将解决这一类问题的算法称为「滑动窗口」问题 + +![](https://img.starfish.ink/mysql/1*m1WP0k9cHRkcTixpfayOdA.gif) + +滑动窗口，就是两个指针齐头并进，好像一个窗口一样，不断往前滑。 + +滑动窗口算法通过维护一个动态调整的窗口范围，高效解决子串、子数组、限流等场景问题。其核心逻辑可概括为以下步骤： + +1. **初始化窗口** 使用双指针 `left` 和 `right` 定义窗口边界，初始状态均指向起点。 +2. **扩展右边界** 移动 `right` 指针，将新元素加入窗口，并根据问题需求更新状态（如统计字符频率或请求计数）。 +3. **收缩左边界** 当窗口满足特定条件（如达到限流阈值或包含冗余元素），逐步移动 `left` 指针缩小窗口，直至不满足条件为止。 +4. **记录结果** 在窗口状态变化的每个阶段，捕获符合要求的解（如最长子串长度或限流通过状态）。 + + + +子串问题，几乎都是滑动窗口。滑动窗口算法技巧的思路，就是维护一个窗口，不断滑动，然后更新答案，该算法的大致逻辑如下： + +```java +int left = 0, right = 0; + +while (right < s.size()) { + // 增大窗口 + window.add(s[right]); + right++; + + while (window needs shrink) { + // 缩小窗口 + window.remove(s[left]); + left++; + } +} +``` + +> 以下是适用于字符串处理、限流等场景的通用框架： +> +> ```java +> public class SlidingWindow { +> +> // 核心双指针定义 +> int left = 0, right = 0; +> // 窗口状态容器（如哈希表、数组） +> Map window = new HashMap<>(); +> // 结果记录 +> List result = new ArrayList<>(); +> +> public void slidingWindow(String s, String target) { +> // 初始化目标状态（如字符频率） +> Map need = new HashMap<>(); +> for (char c : target.toCharArray()) { +> need.put(c, need.getOrDefault(c, 0) + 1); +> } +> +> while (right < s.length()) { +> char c = s.charAt(right); +> right++; +> // 更新窗口状态 +> window.put(c, window.getOrDefault(c, 0) + 1); +> +> // 窗口收缩条件 +> while (window.get(c) > need.getOrDefault(c, 0)) { +> char d = s.charAt(left); +> left++; +> // 更新窗口状态 +> window.put(d, window.get(d) - 1); +> } +> +> // 记录结果（如最小覆盖子串长度） +> if (right - left == target.length()) { +> result.add(left); +> } +> } +> } +> } +> ``` +> +> + +### 3.1 同向交替移动的两个变量 + +有一类数组上的问题，问我们固定长度的滑动窗口的性质，这类问题还算相对简单。 + +#### [子数组最大平均数 I（643）](https://leetcode-cn.com/problems/maximum-average-subarray-i/) + +> 给定 `n` 个整数，找出平均数最大且长度为 `k` 的连续子数组，并输出该最大平均数。 +> +> ``` +> 输入：[1,12,-5,-6,50,3], k = 4 +> 输出：12.75 +> 解释：最大平均数 (12-5-6+50)/4 = 51/4 = 12.75 +> ``` + +**思路**： + +- 长度为固定的 K，想到用滑动窗口 +- 保存每个窗口的值，取这 k 个数的最大和就可以得出最大平均数 +- 怎么保存每个窗口的值，这一步 + +```java +public static double getMaxAverage(int[] nums, int k) { + int sum = 0; + //先求出前k个数的和 + for (int i = 0; i < nums.length; i++) { + sum += nums[i]; + } + //目前最大的数是前k个数 + int result = sum; + //然后从第 K 个数开始移动，保存移动中的和值，返回最大的 + for (int i = k; i < nums.length; i++) { + sum = sum - nums[i - k] + nums[i]; + result = Math.max(result, sum); + } + //返回的是double + return 1.0 * result / k; +} +``` + + + +### 3.2 不定长度的滑动窗口 + +#### [无重复字符的最长子串_3](https://leetcode-cn.com/problems/longest-substring-without-repeating-characters/) + +> 给定一个字符串 `s` ，请你找出其中不含有重复字符的 **最长子串** 的长度。 +> +> ``` +> 输入: s = "abcabcbb" +> 输出: 3 +> 解释: 因为无重复字符的最长子串是 "abc"，所以其长度为 3。 +> ``` + +思路： + +- 滑动窗口，其实就是一个队列，比如例题中的 abcabcbb，进入这个队列（窗口）为 abc 满足题目要求，当再进入 a，队列变成了 abca，这时候不满足要求。所以，我们要移动这个队列 +- 如何移动？我们只要把队列的左边的元素移出就行了，直到满足题目要求！ +- 一直维持这样的队列，找出队列出现最长的长度时候，求出解！ + +```java +int lengthOfLongestSubstring(String s) { + Map window = new HashMap<>(); + + int left = 0, right = 0; + int res = 0; // 记录结果 + while (right < s.length()) { + char c = s.charAt(right); + right++; + // 进行窗口内数据的一系列更新 + window.put(c, window.getOrDefault(c, 0) + 1); + // 判断左侧窗口是否要收缩,字符串重复时收缩，注意这里是 while 不是 if + while (window.get(c) > 1) { + char d = s.charAt(left); + left++; + // 进行窗口内数据的一系列更新 + window.put(d, window.get(d) - 1); + } + // 在这里更新答案，我们窗口是左闭右开的，所以窗口实际包含的字符数是 right - left，无需 +1 + res = Math.max(res, right - left); + } + return res; +} +``` + + + +#### [最小覆盖子串（76）](https://leetcode-cn.com/problems/minimum-window-substring/) + +> 给你一个字符串 `s` 、一个字符串 `t` 。返回 `s` 中涵盖 `t` 所有字符的最小子串。如果 `s` 中不存在涵盖 `t` 所有字符的子串，则返回空字符串 `""` 。 +> +> ``` +> 输入：s = "ADOBECODEBANC", t = "ABC" +> 输出："BANC" +> ``` + +思路： + +1. 我们在字符串 `S` 中使用双指针中的左右指针技巧，初始化 `left = right = 0`，把索引**左闭右开**区间 `[left, right)` 称为一个「窗口」。 + + > [!IMPORTANT] + > + > 为什么要「左闭右开」区间 + > + > **理论上你可以设计两端都开或者两端都闭的区间，但设计为左闭右开区间是最方便处理的**。 + > + > 因为这样初始化 `left = right = 0` 时区间 `[0, 0)` 中没有元素，但只要让 `right` 向右移动（扩大）一位，区间 `[0, 1)` 就包含一个元素 `0` 了。 + > + > 如果你设置为两端都开的区间，那么让 `right` 向右移动一位后开区间 `(0, 1)` 仍然没有元素；如果你设置为两端都闭的区间，那么初始区间 `[0, 0]` 就包含了一个元素。这两种情况都会给边界处理带来不必要的麻烦。 + +2. 我们先不断地增加 `right` 指针扩大窗口 `[left, right)`，直到窗口中的字符串符合要求（包含了 `T` 中的所有字符）。 + +3. 此时，我们停止增加 `right`，转而不断增加 `left` 指针缩小窗口 `[left, right)`，直到窗口中的字符串不再符合要求（不包含 `T` 中的所有字符了）。同时，每次增加 `left`，我们都要更新一轮结果。 + +4. 重复第 2 和第 3 步，直到 `right` 到达字符串 `S` 的尽头。 + +```java +public String minWindow(String s, String t) { + // 两个map，window 记录窗口中的字符频率，need 记录t中字符的频率 + HashMap window = new HashMap<>(); + HashMap need = new HashMap<>(); + + for (int i = 0; i < t.length(); i++) { + char c = t.charAt(i); + //算出每个字符的数量，有可能有重复的 + need.put(c, need.getOrDefault(c, 0) + 1); + } + + //左开右闭的区间，然后创建移动窗口 + int left = 0, right = 0; + // 窗口中满足need条件的字符个数， valid == need.size 说明窗口满足条件 + int valid = 0; + //记录最小覆盖子串的开始索引和长度 + int start = 0, len = Integer.MAX_VALUE; + while (right < s.length()) { + // c 代表将移入窗口的字符 + char c = s.charAt(right); + //扩大窗口 + right++; + + //先判断当前滑动窗口右端（right 指针处）的字符 c 是否是目标字符串 t 中的一个字符 + if (need.containsKey(c)) { + window.put(c, window.getOrDefault(c, 0) + 1); + //检查当前字符 c 的频率在滑动窗口中是否达到了目标字符串 t 中所要求的频率 + if (window.get(c).equals(need.get(c))) { + valid++; + } + } + //判断左窗口是否需要收缩 + while (valid == need.size()) { + //如果当前滑动窗口的长度比已记录的最小长度 len 更短，则说明找到了一个更小的符合条件的覆盖子串 + if (right - left < len) { + start = left; + len = right - left; + } + //d 是将移除窗口的字符 + char d = s.charAt(left); + left++; //缩小窗口 + + //更新窗口，收缩，更新窗口中的字符频率并检查是否还满足覆盖条件 + if (need.containsKey(d)) { + if (window.get(d).equals(need.get(d))) { + valid--; + window.put(d, window.get(d) - 1); + } + } + } + } + //返回最小覆盖子串 + return len == Integer.MAX_VALUE ? "" : s.substring(start, start + len); +} +``` + + + +#### [字符串的排列（567）](https://leetcode.cn/problems/permutation-in-string/description/) + +> 给你两个字符串 `s1` 和 `s2` ，写一个函数来判断 `s2` 是否包含 `s1` 的排列。如果是，返回 `true` ；否则，返回 `false` 。 +> +> 换句话说，`s1` 的排列之一是 `s2` 的 **子串** 。 +> +> ``` +> 输入：s1 = "ab" s2 = "eidbaooo" +> 输出：true +> 解释：s2 包含 s1 的排列之一 ("ba"). +> ``` + +思路： + +通过滑动窗口（Sliding Window）和字符频率统计来解决 + +和上一题基本一致，只是 移动 `left` 缩小窗口的时机是窗口大小大于 `t.length()` 时，当发现 `valid == need.size()` 时，就说明窗口中就是一个合法的排列 + +```java +// 判断 s 中是否存在 t 的排列 +public boolean checkInclusion(String t, String s) { + Map need = new HashMap<>(); + Map window = new HashMap<>(); + for (char c : t.toCharArray()) { + need.put(c, need.getOrDefault(c, 0) + 1); + } + + int left = 0, right = 0; + int valid = 0; + while (right < s.length()) { + char c = s.charAt(right); + right++; + // 进行窗口内数据的一系列更新 + if (need.containsKey(c)) { + window.put(c, window.getOrDefault(c, 0) + 1); + if (window.get(c).intValue() == need.get(c).intValue()) + valid++; + } + + // 判断左侧窗口是否要收缩 + while (right - left >= t.length()) { + // 在这里判断是否找到了合法的子串 + if (valid == need.size()) + return true; + char d = s.charAt(left); + left++; + // 进行窗口内数据的一系列更新 + if (need.containsKey(d)) { + if (window.get(d).intValue() == need.get(d).intValue()) + valid--; + window.put(d, window.get(d) - 1); + } + } + } + // 未找到符合条件的子串 + return false; +} +``` + + + +#### [替换后的最长重复字符（424）](https://leetcode-cn.com/problems/longest-repeating-character-replacement/) + +> 给你一个仅由大写英文字母组成的字符串，你可以将任意位置上的字符替换成另外的字符，总共可最多替换 k 次。在执行上述操作后，找到包含重复字母的最长子串的长度。 +> +> 注意：字符串长度 和 k 不会超过 10^4 +> +> ``` +> 输入：s = "ABAB", k = 2 +> 输出：4 +> 解释：用两个'A'替换为两个'B',反之亦然。 +> ``` +> +> ``` +> 输入：s = "AABABBA", k = 1 +> 输出：4 +> 解释：将中间的一个'A'替换为'B',字符串变为 "AABBBBA"。子串 "BBBB" 有最长重复字母, 答案为 4。 +> ``` + +思路： + +```java +public int characterReplacement(String s, int k) { + int len = s.length(); + if (len < 2) { + return len; + } + + char[] charArray = s.toCharArray(); + int left = 0; + int right = 0; + + int res = 0; + int maxCount = 0; + int[] freq = new int[26]; + // [left, right) 内最多替换 k 个字符可以得到只有一种字符的子串 + while (right < len){ + freq[charArray[right] - 'A']++; + // 在这里维护 maxCount，因为每一次右边界读入一个字符，字符频数增加，才会使得 maxCount 增加 + maxCount = Math.max(maxCount, freq[charArray[right] - 'A']); + right++; + + if (right - left > maxCount + k){ + // 说明此时 k 不够用 + // 把其它不是最多出现的字符替换以后，都不能填满这个滑动的窗口，这个时候须要考虑左边界向右移动 + // 移出滑动窗口的时候，频数数组须要相应地做减法 + freq[charArray[left] - 'A']--; + left++; + } + res = Math.max(res, right - left); + } + return res; +} +``` + + + +### 3.3 计数问题 + +#### 至多包含两个不同字符的最长子串 + +> 给定一个字符串 `s`，找出 **至多** 包含两个不同字符的最长子串 `t` ，并返回该子串的长度。 +> +> ``` +> 输入: "eceba" +> 输出: 3 +> 解释: t 是 "ece"，长度为3。 +> ``` + + + +```java +public int lengthOfLongestSubstringTwoDistinct(String s) { + if (s == null || s.length() == 0) { + return 0; + } + + // 滑动窗口的左指针 + int left = 0; + // 记录滑动窗口内的字符及其出现的频率 + Map map = new HashMap<>(); + // 记录最长子串的长度 + int maxLen = 0; + + // 遍历整个字符串 + for (int right = 0; right < s.length(); right++) { + // 右指针的字符 + char c = s.charAt(right); + // 将字符 c 加入到窗口中，并更新其出现的次数 + map.put(c, map.getOrDefault(c, 0) + 1); + + // 当窗口内的不同字符数超过 2 时，开始缩小窗口 + while (map.size() > 2) { + // 左指针的字符 + char leftChar = s.charAt(left); + // 减少左指针字符的频率 + map.put(leftChar, map.get(leftChar) - 1); + // 如果左指针字符的频率为 0，则从窗口中移除该字符 + if (map.get(leftChar) == 0) { + map.remove(leftChar); + } + // 移动左指针，缩小窗口 + left++; + } + + // 更新最大长度 + maxLen = Math.max(maxLen, right - left + 1); + } + + return maxLen; +} + + +``` + + + +#### 至多包含 K 个不同字符的最长子串_340 + +> 给定一个字符串 `s`，找出 **至多** 包含 `k` 个不同字符的最长子串 `T`。 +> +> ``` +> 输入: s = "eceba", k = 2 +> 输出: 3 +> 解释: 则 T 为 "ece"，所以长度为 3。 +> ``` + +```java +public int lengthOfLongestSubstringKDistinct(String s, int k) { + if (s == null || s.length() == 0 || k == 0) { + return 0; + } + + // 滑动窗口的左指针 + int left = 0; + // 记录滑动窗口内的字符及其出现的频率 + Map map = new HashMap<>(); + // 记录最长子串的长度 + int maxLen = 0; + + // 遍历整个字符串 + for (int right = 0; right < s.length(); right++) { + // 右指针的字符 + char c = s.charAt(right); + // 将字符 c 加入到窗口中，并更新其出现的次数 + map.put(c, map.getOrDefault(c, 0) + 1); + + // 当窗口内的不同字符数超过 K 时，开始缩小窗口 + while (map.size() > k) { + // 左指针的字符 + char leftChar = s.charAt(left); + // 减少左指针字符的频率 + map.put(leftChar, map.get(leftChar) - 1); + // 如果左指针字符的频率为 0，则从窗口中移除该字符 + if (map.get(leftChar) == 0) { + map.remove(leftChar); + } + // 移动左指针，缩小窗口 + left++; + } + + // 更新最大长度 + maxLen = Math.max(maxLen, right - left + 1); + } + + return maxLen; +} +``` + + + +#### [ 区间子数组个数_795](https://leetcode.cn/problems/number-of-subarrays-with-bounded-maximum/) + +> 给定一个元素都是正整数的数组`A` ，正整数 `L` 以及 `R` (`L <= R`)。 +> +> 求连续、非空且其中最大元素满足大于等于`L` 小于等于`R`的子数组个数。 +> +> ``` +> 例如 : +> 输入: +> A = [2, 1, 4, 3] +> L = 2 +> R = 3 +> 输出: 3 +> 解释: 满足条件的子数组: [2], [2, 1], [3]. +> ``` + +```java +public int numSubarrayBoundedMax(int[] nums, int left, int right) { + int count = 0; + int start = -1; // 记录大于right的元素下标 + int last_bounded = -1; // 记录符合[Left, Right]区间的元素下标 + + for (int i = 0; i < nums.length; i++) { + if (nums[i] > right) { + // 遇到大于right的元素，重置窗口起始点 + start = i; + } + if (nums[i] >= left && nums[i] <= right) { + // 记录符合区间条件的元素下标 + last_bounded = i; + } + // 计算当前有效子数组的数量 + count += last_bounded - start; + } + + return count; +} +``` + + + + + +### 3.4 使用数据结构维护窗口性质 + +有一类问题只是名字上叫「滑动窗口」，但解决这一类问题需要用到常见的数据结构。这一节给出的问题可以当做例题进行学习，一些比较复杂的问题是基于这些问题衍生出来的。 + +#### 滑动窗口最大值 + +#### 滑动窗口中位数 + + + + + +## 四、其他双指针问题 + +#### [最长回文子串_5](https://leetcode.cn/problems/longest-palindromic-substring/) + +> 给你一个字符串 `s`，找到 `s` 中最长的 回文子串。 + +```java +public static String longestPalindrome(String s){ + //处理边界 + if(s == null || s.length() < 2){ + return s; + } + + //初始化start和maxLength变量，用来记录最长回文子串的起始位置和长度 + int start = 0, maxLength = 0; + + //遍历每个字符 + for (int i = 0; i < s.length(); i++) { + //以当前字符为中心的奇数长度回文串 + int len1 = centerExpand(s, i, i); + //以当前字符和下一个字符之间的中心的偶数长度回文串 + int len2 = centerExpand(s, i, i+1); + + int len = Math.max(len1, len2); + + //当前找到的回文串大于之前的记录，更新start和maxLength + if(len > maxLength){ + // i 是当前扩展的中心位置， len 是找到的回文串的总长度，我们要用这两个值计算出起始位置 start + // （len - 1)/2 为什么呢，计算中心到回文串起始位置的距离， 为什么不用 len/2， 这里考虑的是奇数偶数的通用性，比如'abcba' 和 'abba' 或者 'cabbad'，巧妙的同时处理两种，不需要分别考虑 + start = i - (len - 1)/2; + maxLength = len; + } + + } + + return s.substring(start, start + maxLength); +} + +private static int centerExpand(String s, int left, int right){ + while(left >= 0 && right < s.length() && s.charAt(left) == s.charAt(right)){ + left --; + right ++; + } + //这个的含义： 假设扩展过程中，left 和 right 已经超出了回文返回， 此时回文范围是 (left+1,right-1), 那么回文长度= (right-1)-(left+1)+1=right-left-1 + return right - left - 1; +} +``` + + + +#### [合并两个有序数组_88](https://leetcode-cn.com/problems/merge-sorted-array/) + + + +#### [下一个排列_31](https://leetcode.cn/problems/next-permutation/) + +> 整数数组的一个 **排列** 就是将其所有成员以序列或线性顺序排列。 +> +> - 例如，`arr = [1,2,3]` ，以下这些都可以视作 `arr` 的排列：`[1,2,3]`、`[1,3,2]`、`[3,1,2]`、`[2,3,1]` 。 +> +> 整数数组的 **下一个排列** 是指其整数的下一个字典序更大的排列。更正式地，如果数组的所有排列根据其字典顺序从小到大排列在一个容器中，那么数组的 **下一个排列** 就是在这个有序容器中排在它后面的那个排列。如果不存在下一个更大的排列，那么这个数组必须重排为字典序最小的排列（即，其元素按升序排列）。 +> +> - 例如，`arr = [1,2,3]` 的下一个排列是 `[1,3,2]` 。 +> - 类似地，`arr = [2,3,1]` 的下一个排列是 `[3,1,2]` 。 +> - 而 `arr = [3,2,1]` 的下一个排列是 `[1,2,3]` ，因为 `[3,2,1]` 不存在一个字典序更大的排列。 +> +> 给你一个整数数组 `nums` ，找出 `nums` 的下一个排列。 +> +> 必须**[ 原地 ](https://baike.baidu.com/item/原地算法)**修改，只允许使用额外常数空间。 + +**Approach**: + +1. 我们希望下一个数 比当前数大，这样才满足 “下一个排列” 的定义。因此只需要 将后面的「大数」与前面的「小数」交换，就能得到一个更大的数。比如 123456，将 5 和 6 交换就能得到一个更大的数 123465。 +2. 我们还希望下一个数 增加的幅度尽可能的小，这样才满足“下一个排列与当前排列紧邻“的要求。为了满足这个要求，我们需要： + - 在 尽可能靠右的低位 进行交换，需要 从后向前 查找 + - 将一个 尽可能小的「大数」 与前面的「小数」交换。比如 123465，下一个排列应该把 5 和 4 交换而不是把 6 和 4 交换 + - 将「大数」换到前面后，需要将「大数」后面的所有数 重置为升序 + +该算法可以分为三个步骤： + +1. **从右向左找到第一个升序对**（即`nums[i] < nums[i + 1]`），记为`i`。如果找不到，则说明数组已经是最大的排列，直接将数组反转为最小的排列。 +2. **从右向左找到第一个比`nums[i]`大的数**，记为`j`，然后交换`nums[i]`和`nums[j]`。 +3. **反转`i + 1`之后的数组**，使其变成最小的排列。 + +```java +public void nextPermutation(int[] nums){ + + //为什么从倒数第二个元素开始，因为我们第一步要从右往左找到第一个“升序对”， + int i = nums.length - 2; + //step 1： 找到第一个下降的元素 + while (i >= 0 && nums[i] >= nums[i + 1]) { + i--; + } + + //step2 : 如果找到了 i, 找到第一个比 nums[i] 大的元素 j + if (i > 0) { + int j = nums.length - 1; + while (j >= 0 && nums[j] <= nums[i]) { + j--; + } + //交换i 和 j 的位置 + swap(nums, i, j); + + } + + // step3: 反转从 start 开始到末尾的部分（不需要重新排序，是因为这半部分再交换前就是降序的，我们第一步找的升序对） + reverse(nums, i+1); +} + +private void swap(int[] nums, int i, int j){ + int tmp = nums[i]; + nums[i] = nums[j]; + nums[j] = tmp; +} + +private void reverse(int[] nums, int start){ + int end = nums.length - 1; + while(start < end){ + swap(nums, start, end); + start ++; + end --; + } +} +``` + + + +#### [颜色分类_75](https://leetcode.cn/problems/sort-colors/) + +> 给定一个包含红色、白色和蓝色、共 `n` 个元素的数组 `nums` ，**[原地](https://baike.baidu.com/item/原地算法)** 对它们进行排序，使得相同颜色的元素相邻，并按照红色、白色、蓝色顺序排列。 +> +> 我们使用整数 `0`、 `1` 和 `2` 分别表示红色、白色和蓝色。 +> +> 必须在不使用库内置的 sort 函数的情况下解决这个问题。 +> +> ``` +> 输入：nums = [2,0,2,1,1,0] +> 输出：[0,0,1,1,2,2] +> ``` + +**Approach**: + +荷兰国旗问题 + +我们可以使用三个指针：`low`、`mid` 和 `high`，分别用来处理 0、1 和 2 的排序问题。 + +- `low` 表示红色 (0) 的边界，指向的元素是 1 的位置，即把所有 0 放在 `low` 的左边。 +- `mid` 表示当前处理的元素索引。 +- `high` 表示蓝色 (2) 的边界，指向的元素是 2 的位置，把所有 2 放在 `high` 的右边。 + +**算法步骤：** + +1. 初始化：`low = 0`，`mid = 0`，`high = nums.length - 1`。 +2. 当 `mid <= high` 时，进行以下判断： + - 如果 `nums[mid] == 0`，将其与 `nums[low]` 交换，并将 `low` 和 `mid` 都加 1。 + - 如果 `nums[mid] == 1`，只需将 `mid` 加 1，因为 1 已经在正确的位置。 + - 如果 `nums[mid] == 2`，将其与 `nums[high]` 交换，并将 `high` 减 1，但 `mid` 不动，因为交换过来的数还未处理。 + +```java +public void sortColors(int[] nums) { + int low = 0, mid = 0, high = nums.length - 1; + + while (mid <= high) { + if (nums[mid] == 0) { + // 交换 nums[mid] 和 nums[low] + swap(nums, low, mid); + low++; + mid++; + } else if (nums[mid] == 1) { + mid++; + } else if (nums[mid] == 2) { + // 交换 nums[mid] 和 nums[high] + swap(nums, mid, high); + high--; + } + } +} + +private void swap(int[] nums, int i, int j) { + int temp = nums[i]; + nums[i] = nums[j]; + nums[j] = temp; +} + +``` + +- 时间复杂度：O(n)，每个元素只遍历一次。 + +- 空间复杂度：O(1)，不需要额外的空间，只在原数组中进行操作。 + +双指针方法的话，就是两次遍历。 + +```java +public void sortColors(int[] nums) { + int left = 0; + int right = nums.length - 1; + + // 第一次遍历，把 0 移动到数组的左边 + for (int i = 0; i <= right; i++) { + if (nums[i] == 0) { + swap(nums, i, left); + left++; + } + } + + // 第二次遍历，把 2 移动到数组的右边 + for (int i = nums.length - 1; i >= left; i--) { + if (nums[i] == 2) { + swap(nums, i, right); + right--; + } + } +} + +private void swap(int[] nums, int i, int j) { + int temp = nums[i]; + nums[i] = nums[j]; + nums[j] = temp; +} + +``` + + + +#### [排序链表_148](https://leetcode.cn/problems/sort-list/description/) + +> 给你链表的头结点 `head` ，请将其按 **升序** 排列并返回 **排序后的链表** 。 +> +> ``` +> 输入：head = [4,2,1,3] +> 输出：[1,2,3,4] +> ``` + +**Approach**: 要将链表排序，并且时间复杂度要求为 O(nlog⁡n)O(n \log n)O(nlogn)，这提示我们需要使用 **归并排序**。归并排序的特点就是时间复杂度是 O(nlog⁡n)O(n \log n)O(nlogn)，并且它在链表上的表现很好，因为链表的分割和合并操作相对容易。 + +具体实现步骤： + +1. **分割链表**：我们可以使用 **快慢指针** 来找到链表的中点，从而将链表一分为二。 +2. **递归排序**：分别对左右两部分链表进行排序。 +3. **合并有序链表**：最后将两个已经排序好的链表合并成一个有序链表。 + +```java + +public ListNode sortList(ListNode head) { + // base case: if the list is empty or contains a single element, it's already sorted + if (head == null || head.next == null) { + return head; + } + + // Step 1: split the linked list into two halves + ListNode mid = getMiddle(head); + // right 为链表右半部分的头结点 + ListNode right = mid.next; + mid.next = null; //断开 + + // Step 2: recursively sort both halves + ListNode leftSorted = sortList(head); + ListNode rightSorted = sortList(right); + + // Step 3: merge the sorted halves + return mergeTwoLists(leftSorted, rightSorted); +} + +// Helper method to find the middle node of the linked list +private ListNode getMiddle(ListNode head) { + ListNode slow = head; + ListNode fast = head; + + while (fast != null && fast.next != null) { + slow = slow.next; + fast = fast.next.next; + } + + return slow; +} + +// Helper method to merge two sorted linked lists +private ListNode mergeTwoLists(ListNode l1, ListNode l2) { + ListNode dummy = new ListNode(-1); + ListNode current = dummy; + + while (l1 != null && l2 != null) { + if (l1.val < l2.val) { + current.next = l1; + l1 = l1.next; + } else { + current.next = l2; + l2 = l2.next; + } + current = current.next; + } + + // Append the remaining elements of either list + if (l1 != null) { + current.next = l1; + } else { + current.next = l2; + } + + return dummy.next; +} +``` + + + +### 总结 + +区间不同的定义决定了不同的初始化逻辑、遍历过程中的逻辑。 + +- 移除元素 +- 删除排序数组中的重复项 II +- 移动零 + + + diff --git a/docs/data-structure-algorithms/algorithm/Dynamic-Programming.md b/docs/data-structure-algorithms/algorithm/Dynamic-Programming.md new file mode 100644 index 0000000000..9ea856aedc --- /dev/null +++ b/docs/data-structure-algorithms/algorithm/Dynamic-Programming.md @@ -0,0 +1,929 @@ +--- +title: 动态规划——刷题有套路 +date: 2024-03-09 +tags: + - Algorithm +categories: Algorithm +--- + +> 动态规划，简直就是刷题模板、套路届的典范 + +## 一、前言 + +为了面试，不，不，为了提高技术能力，我重拾算法有一段时间了，但是每次都把动态规划放在了后边，因为这个大名鼎鼎的名字，听着就感觉很牛逼，很难学的样子。 + +> **动态规划**（英语：Dynamic programming，简称DP）是运筹学的一个分支，是求解决策过程(decision process)最优化的数学方法。20 世纪 50 年代初美国数学家 R.E.Bellman 等人在研究多阶段决策过程(multistep decision process) 的优化问题时，提出了著名的最优化原理 (principle of optimality)，把多阶段过程转化为一系列单阶段问题，逐个求解，创立了解决这类过程优化问题的新方法——动态规划。1957 年出版了他的名著 Dynamic Programming，这是该领域的第一本著作。 +> +> 动态规划问世以来，在经济管理、生产调度、工程技术和最优控制等方面得到了广泛的应用。例如最短路线、库存管理、资源分配、设备更新、排序、装载等问题，用动态规划方法比用其它方法求解更为方便。 +> +> 虽然动态规划主要用于求解以时间划分阶段的动态过程的优化问题，但是一些与时间无关的静态规划(如线性规划、非线性规划)，只要人为地引进时间因素，把它视为多阶段决策过程，也可以用动态规划方法方便地求解。 +> + +看完之后，我说了一句脏话，然后就开始找相关文章了。 + + + +## 二、写在前面 + +计算机归根结底只会做一件事：穷举。 + +所有的算法都是在让计算机【如何聪明地穷举】而已，动态规划也是如此。 + +> A ： "1+1+1+1+1+1+1+1 =？等式的值是多少" +> +> B ： 计算 "8" +> +> A : 在上面等式的左边写上 "1+" 呢？ "此时等式的值为多少" +> +> B : 很快得出答案 "9" +> +> A : "你怎么这么快就知道答案了" +> +> B : "只要在8的基础上加1就行了" +> +> A : "所以你不用重新计算，因为你记住了第一个等式的值为8!动态规划算法也可以说是 '记住求过的解来节省时间'" + +本文将会从以下角度来讲解动态规划： + +- 什么是动态规划 +- 动态规划从入门到进阶 +- 再谈动态规划 + + + +## 三、动态规划是什么 + +动态规划(dynamic programming)是运筹学的一个分支，是解决**「多阶段决策」**过程最优化的一种数学方法。 + +**一般用来求最值问题**，多数情况下它可以采用**「自下而上」**的递推方式来得出每个子问题的最优解（即**「最优子结构」**），进而自然而然地得出依赖子问题的原问题的最优解。 + +有几个比较眼生的概念，我们看下： + +- **多阶段决策**：比如说我们有一个复杂的问题要处理，我们可以按问题的时间或从空间关系分解成几个互相联系的阶段，使每个阶段的决策问题都是一个比较容易求解的“**子问题**”，这样依次做完每个阶段的最优决策后，他们就构成了整个问题的最优决策。简单地说，就是每做一次决策就可以得到解的一部分，当所有决策做完之后，完整的解就“浮出水面”了。有一种**大事化小，小事化了**的感觉。 + +- **最优子结构**：在我们拆成一个个子问题的时候，每个子问题一定都有一个最优解，既然它分解的子问题是全局最优解，那么依赖于它们解的原问题自然也是全局最优解。比如说，你的原问题是考出最高的总成绩，那么你的子问题就是要把语文考到最高，数学考到最高…… 为了每门课考到最高，你要把每门课相应的选择题分数拿到最高，填空题分数拿到最高…… 当然，最终就是你每门课都是满分，这就是最高的总成绩。![img](https://assets.leetcode-cn.com/aliyun-lc-upload/uploads/2020/08/06/1-1.png) + +- **自下而上**：或者叫自底向上，对应的肯定有**自上而下**（自顶向下） + + - 啥叫**自顶向下**，比如我们求解递归问题，画递归树的时候，是从上向下延伸，都是从一个规模较大的原问题比如说 f(20)，向下逐渐分解规模，直到 f(1) 和 f(2) 触底，然后逐层返回答案，这就叫「自顶向下」，比如我们用递归法计算斐波那契数列的时候 + + ![](https://img.starfish.ink/leetcode/up2down.png) + + + + - 反过来，自底向上，肯定就是从最底下，最简单，问题规模最小的 f(1) 和 f(2) 开始往上推，直到推到我们想要的答案 f(20)，这就是动态规划的思路，这也是为什么动态规划一般都脱离了递归，而是由循环迭代完成计算。 + + ![](https://img.starfish.ink/leetcode/down2up.png) + + + +从递归树中我们可以看到，自顶向下的递归算法，我们会求两次 f(18)，三次 f(17)，，，这就存在了大量的**「重复子问题」**，这样暴力穷举的话效率会极其低下，为了解决重复子问题，我们可以通过「**备忘录**」或者「**DP table**」来优化穷举过程（记忆化递归法），避免不必要的计算。 + +怎样才能自下而上的求出每个子问题的最优解呢，可以肯定子问题之间是有一定联系的，即**迭代递推公式**，也叫「**状态转移方程**」，实际上就是描述问题结构的数学形式。（把 `f(n)` 想做一个状态 `n`，这个状态 `n` 是由状态 `n - 1` 和状态 `n - 2` 相加转移而来，这就叫状态转移，仅此而已） + +> 动态规划中当前的状态往往依赖于前一阶段的状态和前一阶段的决策结果。例如我们知道了第 i 个阶段的状态 Si 以及决策 Ui，那么第 i+1 阶段的状态 Si+1 也就确定了。所以解决动态规划问题的关键就是确定状态转移方程，一旦状态转移方程确定了，那么我们就可以根据方程式进行编码。 + +> 通常许多子问题非常相似，为此动态规划法试图仅仅解决每个子问题一次，具有天然剪枝的功能，从而减少计算量：一旦某个给定子问题的解已经算出，则将其记忆化存储，以便下次需要同一个子问题解之时直接查表。这种做法在重复子问题的数目关于输入的规模呈指数增长时特别有用。 +> +> 动态规划在查找有很多**重叠子问题**的情况的最优解时有效。它将问题重新组合成子问题。为了避免多次解决这些子问题，它们的结果都逐渐被计算并被保存，从简单的问题直到整个问题都被解决。因此，动态规划保存递归时的结果，而不会在解决同样问题时再花费时间。 +> +> 动态规划只能应用于有**最优子结构**的问题。最优子结构的意思是局部最优解能决定全局最优解（对有些问题这个要求并不能完全满足，故有时需要引入一定的近似）。简单地说，问题能够分解成子问题来解决。 + +以上提到的**重叠(复)子问题、最优子结构、状态转移方程就是动态规划三要素**。 + +> 解决动态规划问题的核心：找出子问题及其子问题与原问题的关系 + + + +### 斐波那契数列 + +PS：我们先从一个简单的斐波那契数列来进一步理解下重叠子问题与状态转移方程（斐波那契数列并不是严格意义上的动态规划，因为它没有求最值，所以也没涉及到最优子结构的问题） + +**1、暴力递归** + +斐波那契数列的数学形式就是递归的，写成代码就是这样： + +```java +int fib(int N) { + if (N == 1 || N == 2) return 1; + return fib(N - 1) + fib(N - 2); +} +``` + +这个不用多说了，我们在 **自顶向下** 那部分画出的就是它的递归树，他有大量的重复计算问题，比如 `f(18)` 被计算了两次，而且你可以看到，以 `f(18)` 为根的这个递归树体量巨大，多算一遍，会耗费巨大的时间。更何况，还不止 `f(18)` 这一个节点被重复计算，所以这个算法及其低效。 + +这就是动态规划问题的第一个性质：**重叠子问题**。下面，我们想办法解决这个问题。 + +**2、带备忘录的递归解法** + +明确了问题，其实就已经把问题解决了一半。即然耗时的原因是重复计算，那么我们可以造一个「备忘录」，每次算出某个子问题的答案后别急着返回，先记到「备忘录」里再返回；每次遇到一个子问题就先去「备忘录」里查一查，如果发现之前已经解决过这个问题了，直接把答案拿出来用，不要再耗时去计算了。相当于我们业务开发中的缓存。 + +一般使用一个数组充当这个「备忘录」，当然也可以使用哈希表（字典），思想都是一样的，也有人叫 **记忆化递归法**。 + +```java +public static int fib(int n) { + if (n == 1 || n == 2) { + return 1; + } + //用一个数组充当备忘录，保存记录 + int[] dp = new int[n + 1]; + //初始值 + dp[0] = 0; + dp[1] = 1; + for(int i = 2; i <= n; i++) { + dp[i] = dp[i - 1] + dp[i - 2]; + } + return dp[n]; +} +// 用 hash 表当备忘录 +public int fib(int n) { + if (n == 1 || n == 2) { + return 1; + } + if (hashMap.containsKey(n)) { + return hashMap.get(n); + } else { + int result = fib(n - 1) + fib(n - 2); + hashMap.put(n,result); + return result; + } +} +``` + +带「备忘录」的递归算法，把一棵存在巨量冗余的递归树通过「**剪枝」**，改造成了一幅不存在冗余的递归图，极大减少了子问题（即递归图中节点）的个数。 + +**3、动态规划解法** + +有了上一步「备忘录」的启发，**自顶向下**的递推，每次“缓存”之前的结果，那**自底向上**的推算不也可以吗？而且推算的时候，我们只需要存储之前的两个状态就行，还省了很多空间，我靠，真是个天才，这就是，**动态规划**的做法。 + +![](https://img.starfish.ink/leetcode/down2up.png) + +画个图就很好理解了，我们一层一层的往上计算，得到最后的结果。 + +斐波那契数列的定义其实就是个**状态转移方程**：$f(n) = f(n-1) + f(n-2)$，$f(n)$ 就是子问题的状态，这个状态是由 $f(n-1)$ 和 $f(n-2)$ 这两个状态相加转移过来的，这就是状态转移。是不又有点理解了？ + +最难的状态转移方程有了，看下代码，定义三个变量循环迭代完成计算，搞定 + +```java +public int fib(int n) { + if (n == 1 || n == 2) { + return 1; + } + int result = 0, pre = 1, next = 2; + for (int i = 3; i < n + 1; i++) { + result = pre + next; + pre = next; + next = result; + } + return result; +} +``` + +有没有发现，这个状态转移方程的写法，和暴力破解有着千丝万缕的联系。其实状态转移方程直接代表着暴力解法。 + + + +## 四、什么样的题目适合用动态规划 + +可以使用动态规划的问题一般都有一些特点可以遵循。如题目的问法一般是三种方式： + +1. 求最大值/最小值（除了类似找出数组中最大值这种） + + 乘积最大子数组、最长回文子串、最长上升子序列等等 + +2. 求可行性（True 或 False） + + 凑领钱、字符串交错组成问题 + +3. 求方案总数 + + 硬币组合问题、路径规划问题 + +如果你碰到一个问题，是问你这三个问题之一的，那么有 90% 的概率是可以使用动态规划来求解。 + + + +一个问题是否能够用动态规划算法来解决，需要看这个问题是否能被分解为更小的问题（子问题）。而子问题往下细分为更小的子问题的时候往往会遇到重复的子问题，我们只处理同一个子问题一次，将它的结果保存起来，这就是动态规划最大的特点。 + +接下来就要去理解动态规划的思路了，通常情况下，DP 题可从下面 4 个要素去逐步剖析： + +**1. 状态是什么** + +**2. 状态转移方程是什么** + +**3. 状态的初始值是什么** + +**4. 问题要求的最后答案是什么** + + + +## 五、套路解题 + +动态规划是用大白话说就是一个算法范例（或者理解为一个方法论，模板），**通过将其分解为子问题来解决给定的复杂问题，并存储子问题的结果，以避免再次计算相同的结果**。 + +我们知道了动态规划三要素：重叠子问题、最优子结构、状态转移方程。 + +那要解决一个动态规划问题的大概步骤，就围绕这三要素展开： + +1. **划分阶段：**分析题目可以用动态规划解决，那就先看这个问题如何划分成各个子问题 + +2. **状态定义**：也有叫选择状态的，其实就是定义子问题，我理解其实就是看求解的结果，我们一般用数组来存储子问题结果，所以状态我们一般定义为 $dp[i]$，表示规模为 i 的问题的解，$dp[i-1]$ 就是规模为 i-1 的子问题的解 + +3. **确定决策并写出状态转移方程：**听名字就觉得牛逼的一步，肯定也是最难的一步，其实就是我们从 f(1)、f(2)、f(3) ... f(n-1) 一步步递推出 f(n) 的表达式，也就是说，dp[n] 一定会和 dp[n-1], dp[n-2]....存在某种关系的，这一步就是找出数组元素的关系式，比如斐波那契数列的关系式 $dp[n] = dp[n-1] + dp[n-2]$ + + > 一般来说函数的参数就是状态转移中会变化的量，也就是上面说到的「状态」；函数的返回值就是题目要求我们计算的 + +4. **找出初始值（包括边界条件）：**既然状态转移方程式写好了，但是还需要一个**支点**来撬动它进行不断的计算下去，比如斐波那契数列中的 f(1)=1，f(2)=1，就是初始值 + +5. **优化**：思考有没有可以优化的点 + + + +**写出状态转移方程是最困难的**，这也就是为什么很多朋友觉得动态规划问题困难的原因，我来提供我研究出来的一个思维框架，辅助你思考状态转移方程： + +**明确 base case -> 明确「状态」-> 明确「选择」 -> 定义 dp 数组/函数的含义**。 + +按上面的套路走，最后的结果就可以套这个框架： + +```java +# 初始化 base case +dp[0][0][...] = base +# 进行状态转移 +for 状态1 in 状态1的所有取值： + for 状态2 in 状态2的所有取值： + for ... + dp[状态1][状态2][...] = 求最值(选择1，选择2...) +``` + +下面通过几道经典、且极其常见的面试题来看下动态规划解题套路 + + + +## 六、找感觉（刷题） + +斐波那契数列上手后，我们用解题套路看下 leetcode_70，据说是道正宗的动态规划问题。 + +### 1、[爬楼梯](https://leetcode-cn.com/problems/climbing-stairs/)（leetcode_70） + +> 假设你正在爬楼梯。需要 n 阶你才能到达楼顶。每次你可以爬 1 或 2 个台阶。你有多少种不同的方法可以爬到楼顶呢？注意：给定 n 是一个正整数。 +> +> ``` +>输入： 2 +> 输出： 2 +> 解释： 有两种方法可以爬到楼顶。 +> 1. 1 阶 + 1 阶 +> 2. 2 阶 +> ``` + +这是一道 easy 题，又想说脏话了，当时我拿到这么一道题后，一点想法都没有。 + +#### 分析题目 + +不管了，“穷举思想” + +假设 n = 5，有 5 级楼梯要爬。每次都有 2 种选择：爬 1 级或爬 2 级。 + +如果爬 1 级，则剩下 4 级要爬。 + +如果爬 2 级，则剩下 3 级要爬。 + +这分出了 2 个子问题：爬 4 级楼梯有几种方式？爬 3 级楼梯有几种方式？ + +爬 5 级楼梯的方式数 = 爬 4 级楼梯的方式数 + 爬 3 级楼梯的方式数，这样往下递归分析 + +爬 4 级楼梯的方式数 = 爬 3 级楼梯的方式数 + 爬 2 级楼梯的方式数 + +> 第二次做的时候，我没有用 『自底向上』，而是用『自上向下』的举例，陷入了一种错误 +> +> 我想的是 f(n) = f(n-1) + 1，从上往下的算，留出一级，肯定只能是爬 1 级这一种，所以 +> +> f(5) = f(4) + 1 ....... + +这不是上一节的斐波那契数列吗？？？？？ + +用 $f(x)$ 表示爬到第 x 级台阶的方案数，考虑最后一步可能跨了一级台阶，也可能跨了两级台阶，所以我们可以列出如下式子： + +$f(x) = f(x - 1) + f(x - 2)$ + +它意味着爬到第 $x$ 级台阶的方案数是爬到第 $x - 1$ 级台阶的方案数和爬到第 $x - 2$ 级台阶的方案数的和。 + +我们看下这道题的思路： + +“穷举”之后，发现可以拆分成各个子问题（或者一看问题是多少种方案），推断可以用动态规划 + +1. **定义状态**：用 $dp[n]$ 表示最后的的结果 +2. **初始状态**：只有 1 级台阶的话 dp[0] =0，dp[1] = 1 +3. **状态转移方程**：dp[i] = dp[i - 1] + dp[i - 2] +4. **返回结果**： dp[n] 即我们要的结果 + +```java +public int climbStairs(int n) { + // 创建一个数组来保存历史数据 + int[] dp = new int[n + 1]; + // 给出初始值, 爬楼梯的初始值 + dp[0] = 0; + dp[1] = 1; + for(int i = 2; i <= n; i++) { + //写出状态转移方程 + dp[i] = dp[i - 1] + dp[i - 2]; + } + return dp[n]; +} +``` + +优化方案就是按斐波那契，只保存前两个状态值，优化空间的方案，这里就不多说了。 + + + +### 2、[最大子数组和](https://leetcode-cn.com/problems/maximum-subarray/)（leetcode_53） + +> 给定一个整数数组 nums ，找到一个具有最大和的连续子数组（子数组最少包含一个元素），返回其最大和。 +> +> ``` +>输入: [-2,1,-3,4,-1,2,1,-5,4] +> 输出: 6 +> 解释: 连续子数组 [4,-1,2,1] 的和最大，为 6。 +> ``` +> + +#### 小技巧——涨知识 + +拿到这类题目，避免不了的是要遍历，假设我们给定数组 [a,b,c,d,e] ，通常我们遍历子串或者子序列有如下三种遍历方式： + +- 以某个元素开头的所有子序列，比如以 a 开头的子序列 [a]，[a, b]，[ a, b, c] ... 接着是以 b 开头的子序列 [b]，[b, c]，[b,c,d] ... 接着是 c 开头、d 开头... +- 以子序列的长度为基准，比如先遍历出子序列长度为 1 的子序列，再遍历出长度为 2 的 ... +- 以某个元素结尾的所有子序列，比如以 a 结束的子序列只有 [a]，以 b 结束的子序列 [a,b]，[b]，以 c 结束的子序列 [a,b,c]，[b,c]，[c]，以 d 结束的 ... + +想想这道题，用哪种遍历方式合适一些呢？ + +用哪种遍历方式，可以逐个分析嘛。第一种遍历方式通常用于暴力解法，第二种后边我们也会用到（最长回文子串），第三种由于可以产生递推关系，动态规划问题用的挺多的。 + +#### 分析题目 + +拿到这个题目先理解了意思，我们要求的是连续子数组的和最大那一个，所以我们肯定要遍历出所有子数组，并把每个子数组的和保存起来，然后去比较找到最大的那个就是我们要的结果了。 + +用暴力法从头遍历所有子序列，用两个变量，一个记录最大和，一个记录当前和，双层循环也可以解决。 + +最值问题，我们分析用动态规划解题： + +1. **定义状态**：$dp[i]$ 表示索引从 0 到 i 的元素组成的数组中最大子序和； + +2. **初始状态**：如果数组只有 1 个元素，那 dp 数组的第一个元素也就是数组的第一个元素本身，**`dp[0] = nums[0]`**; + +3. **状态转移方程**：因为我们的数组中可能有负数的情况，从头遍历的话，如有下一个元素是负数的话，和反而更小了，所以我们遍历以元素结尾的子序列，若前一个元素大于 0（$nums[i-1] > 0$），则将它加到当前元素上 $nums[i] = nums[i-1]+nums[i]$。最终 $nums[i]$ 保存的是以原先数组中 $nums[i]$ 结尾的最大子序列和。最后整体遍历一遍 $nums[i]$ 就能找到整个数组最大的子序列和啦。所以状态转移方程： + + $dp[i]=\max \{nums[i],dp[i−1]+nums[i]\}$ + +4. **输出结果**：转移方程只是保存了当前元素的最大和，我们要求的是最终的那个最大值，所以需要从 dp[i] 中找到最大值返回 + +```java +public int maxSubArray(int[] nums) { + //特判 + if (nums == null || nums.length == 0) { + return 0; + } + //初始化 + int length = nums.length; + int[] dp = new int[length]; + // 初始值,只有一个元素的时候最大和即它本身 + dp[0] = nums[0]; + int ans = nums[0]; + // 状态转移 + for (int i = 1; i < length; i++) { + // 取当前元素的值 和 当前元素的值加上一次结果的值 中最大数 + dp[i] = Math.max(nums[i], dp[i - 1] + nums[i]); + // 输出结果：和最大数对比 取大 + ans = Math.max(ans, dp[i]); + } + return ans; +} +``` + +#### 优化 + +同样的套路，考虑到 $f(i)$ 只和 $f(i - 1)$ 相关，于是我们可以只用一个变量 pre 来维护对于当前 $f(i)$ 的 $f(i - 1)$ 的值是多少，从而让空间复杂度降低到 $O(1)$，这有点类似「滚动数组」的思想 + +```java +public int maxSubArray(int[] nums) { + int pre = 0, maxAns = nums[0]; + for (int x : nums) { + pre = Math.max(pre + x, x); + maxAns = Math.max(maxAns, pre); + } + return maxAns; +} +``` + + + +### 3、[ 打家劫舍](https://leetcode-cn.com/problems/house-robber/)（leetcode_198） + +> 你是一个专业的小偷，计划偷窃沿街的房屋。每间房内都藏有一定的现金，影响你偷窃的唯一制约因素就是相邻的房屋装有相互连通的防盗系统，如果两间相邻的房屋在同一晚上被小偷闯入，系统会自动报警。 +> +> 给定一个代表每个房屋存放金额的非负整数数组，计算你不触动警报装置的情况下 ，一夜之内能够偷窃到的最高金额。 +> +> ``` +>输入：[1,2,3,1] +> 输出：4 +> 解释：偷窃 1 号房屋 (金额 = 1) ，然后偷窃 3 号房屋 (金额 = 3)。 +>   偷窃到的最高金额 = 1 + 3 = 4 。 +> ``` +> +> +>提示： +> +>0 <= nums.length <= 100 +> 0 <= nums[i] <= 400 + +#### 分析题目 + +拿到这个题目，看到这道题是 easy，我飘了，心想，这不就两种情况吧，奇数列求和，偶数列求和，比较后得出结果，太简单了吧吧吧吧~（个人感觉官网给的这两个例子误导了聪明的我） + +但是回头一想，这一道动态规划标签下的题，不能被我这么容易的解决，反过来想，小偷也不可能非得从第一家或者第二家隔一间偷一偷，可能是从中间随便一间开始的，比如 [4,3,2,5,1] ，我就会去偷第一家和第四家。 + +1. **定义状态**：用 $dp[i]$ 存储前 i 间房屋能偷窃到的最高总金额 + +2. **初始状态**：如果只有一间房子，只能偷一间了，最大金额 dp[0]，如果有两间房子，偷钱最多的那一件，最大金额 $\max \{dp[0],dp[1]\}$，即 + + $$ f(n)= \begin{cases} dp[0]=nums[0], & \text{只有一间房屋，则偷窃该房屋}\\ dp[1]=\max \{dp[0],dp[1]\},& \text{只有两间房屋，选择其中金额较高的房屋进行偷窃} \end{cases} $$ + +3. **状态转移方程**：由于不可以在相邻的房屋闯入，所以在当前位置 n 房屋可盗窃的最大值，要么就是 n-1 房屋可盗窃的最大值，要么就是 n-2 房屋可盗窃的最大值加上当前房屋的值，二者之间取最大值，即 $dp[i]=\max \{dp[i-1],dp[i−2]+nums[i]\}$ + +4. **输出结果**：$dp[n−1]$ + +```java +public int rob(int[] nums) { + //特判 + if (nums == null || nums.length == 0) return 0; + //创建动态数组 + int length = nums.length; + int[] dp = new int[length]; + //初始状态 + dp[0] = nums[0]; + dp[1] = Math.max(nums[0], nums[1]); + //转移方程 + for (int i = 2; i < length; i++) { + dp[i] = Math.max(dp[i - 1], dp[i - 2] + nums[i]); + } + //取值：当前下标，即length-1 + return dp[length - 1]; +} +``` + +#### 优化 + +同样的优化套路，上述方法使用了数组存储结果。但是每间房屋的最高总金额只和该房屋的前两间房屋的最高总金额相关，因此可以使用滚动数组，在每个时刻只需要存储前两间房屋的最高总金额。和斐波那契数列优化同理。 + +```java +public int rob(int[] nums) { + if (nums == null || nums.length == 0) { + return 0; + } + int length = nums.length; + if (length == 1) { + return nums[0]; + } + int first = nums[0], second = Math.max(nums[0], nums[1]); + for (int i = 2; i < length; i++) { + int temp = second; + second = Math.max(first + nums[i], second); + first = temp; + } + return second; +} +``` + + + +### 4、[不同路径](https://leetcode-cn.com/problems/unique-paths/)（leetcode_62） + +> 一个机器人位于一个 m x n 网格的左上角 （起始点在下图中标记为“Start” ）。 +> +> 机器人每次只能向下或者向右移动一步。机器人试图达到网格的右下角（在下图中标记为“Finish”）。 +> +> 问总共有多少条不同的路径？ +> +> ![](https://assets.leetcode-cn.com/aliyun-lc-upload/uploads/2018/10/22/robot_maze.png) +> +> 例如，上图是一个7 x 3 的网格。有多少可能的路径？ +> +> ``` +>输入: m = 3, n = 2 +> 输出: 3 +> 解释: +> 从左上角开始，总共有 3 条路径可以到达右下角。 +> +> 1. 向右 -> 向右 -> 向下 +> 2. 向右 -> 向下 -> 向右 +> 3. 向下 -> 向右 -> 向右 +> ``` +> +> ``` +>输入: m = 7, n = 3 +>输出: 28 +> ``` +> +> **提示：** +> +> - `1 <= m, n <= 100` +> - 题目数据保证答案小于等于 `2 * 10 ^ 9` + +#### 分析题目 + +1. **定义状态**：这是个求方案总数的问题，大概率可以用动态规划，由于是个表格，像地图一样，有坐标，我们用二维数组来存储结果 $dp[m][n]$，代表到达位置 (m, n) 的所有路径的总数 + +2. **初始状态**：$dp[m][n]$ 表示到坐标 (m,n) 的路径条数。由于机器人从 m=0,n=0 出发，每次只能向下或者向右移动，因此，在所有坐标为（0，m) 的位置机器人要到达的话只有一条路径（一直向下）；在所有坐标为（n,0) 的位置，机器人要到达也只有一条路径（一直向右）在机器人走第 0 行，第 0 列的时候，无论怎么走，都只有 1 种走法。因此初始值是： + + ```java + dp[0] [0….n-1] = 1; // 机器人一直向右走，第 0 列统统为 1 + dp[0…m-1] [0] = 1; // 机器人一直向下走，第 0 列统统为 1 + ``` + +3. **状态转移方程**：要到达任一位置 (m,n) 的总路径条数，总是等于位置 (m-1,n) 的路径条数加上位置（m,n-1) 的路径条数。即 $dp[m][n] = dp[m-1][n] + dp[m][n-1]$ + +4. **输出结果**：由于数组是从下标 0 开始算起的，所以 $dp[m - 1][n - 1]$ 才是我们要的结果 + +![](https://img.starfish.ink/algorithm/uniquePaths.png) + +```java +public int uniquePaths(int m, int n) { + + //定义二维数组保存路径 + int dp[][] = new int[m][n]; + + //定义初始值 + for (int i = 0; i < m; i++) { + dp[i][0] = 1; + } + + for (int j = 0; j < n; j++) { + dp[0][j] = 1; + } + + // 排除初始值的情况，都从 1 开始循环 + for (int i = 1; i < m; i++) { + for (int j = 1; j < n; j++) { + dp[i][j] = dp[i - 1][j] + dp[i][j - 1]; + } + } + // 由于数组是从下标 0 开始算起的，所以dp[m - 1][n - 1] 是我们要的结果 + return dp[m - 1][n - 1]; +} +``` + +#### 优化 + +我们的状态转移方程： $dp[m][n] = dp[m - 1][n] + dp[m][n - 1]$，可以看到我们其实只需要 $dp[m - 1][n]$ 和 $dp[m][n - 1]$，只需要记录这两个数就可以了，这其实相当于转换为一维数组，dp[i] = dp[1] + dp[0] + +```java +public int uniquePaths(int m, int n) { + int[] dp = new int[n]; + Arrays.fill(dp, 1); + for (int i = 1; i < m; i++) { + for (int j = 1; j < n; j++) { + dp[j] = dp[j - 1] + dp[j]; + } + } + return dp[n - 1]; +} +``` + + + +### 5、[零钱兑换](https://leetcode-cn.com/problems/coin-change/)（leetcode_322） + +> 给定不同面额的硬币 coins 和一个总金额 amount。编写一个函数来计算可以凑成总金额所需的最少的硬币个数。如果没有任何一种硬币组合能组成总金额，返回 -1。 (你可以认为每种硬币的数量是无限的。) +> +> ``` +>输入: coins = [1, 2, 5], amount = 11 +> 输出: 3 +> 解释: 11 = 5 + 5 + 1 +> ``` +> +> ``` +>输入: coins = [2], amount = 3 +>输出: -1 +> ``` +> + +#### 分析题目 + +题目问最少的硬币个数，一般可用 DP 来解，看看可以拆分子问题不 + +以 coins = [1, 2, 5]，amount = 11 为例。我们要求组成 11 的最少硬币数，可以考虑组合中的最后一个硬币分别是1，2，5 的情况，比如： + +- 最后一个硬币是 1 的话，最少硬币数应该为【组成 10 的最少硬币数】+ 1枚（1块硬币） +- 最后一个硬币是 2 的话，最少硬币数应该为【组成 9 的最少硬币数】+ 1枚（2块硬币） +- 最后一个硬币是 5 的话，最少硬币数应该为【组成 6 的最少硬币数】+ 1枚（5块硬币） + +在这 3 种情况中硬币数最少的那个就是结果 + +按同样的道理，我们也可以分别再求出组成 10 的最少硬币数，组成 9 的最少硬币数，组成 6 的最少硬币数。。。DP 的套路无疑了。 + +1. **定义状态**：凑齐总金额 i 需要最少硬币个数 $dp[i]$ + +2. **初始状态**：假设金额是 0 ，那就不需要硬币了，即 $dp[0] = 0$ + +3. **状态转移方程**：从例子中我们可以看出结果可以这么表示 $dp[11]=\min \{dp[10]+1,dp[9]+2,dp[6]+5\}$，但我们不知道最后一枚硬币的面值是多少，所以我们需要枚举每个硬币面额值，从中选出最小值 + + ```java + for(int coin : coins){ + result = Math.min(result,1+dp[amout-coin]) + } + ``` + +4. **输出结果**： $dp[amout]$ + +```java +public int coinChange(int[] coins, int amount) { + //定义数组 + int[] dp = new int[amount + 1]; + + int max = amount + 1; + // 初始化每个值为 amount+1，这样当最终求得的 dp[amount] 为 amount+1 时，说明问题无解, 或者初始化一个特殊值 + Arrays.fill(dp, max); + + //初始值 + dp[0] = 0; + // 外层 for 循环在遍历所有可能得金额，（从1到amount） + //dp[i]上的值不断选择已含有硬币值当前位置的数组值 + 1，min保证每一次保存的是最小值 + for (int i = 1; i < amount + 1; i++) { + //内层循环所有硬币面额 + for (int coin : coins) { + //如果i= coin) { + //分两种情况，使用硬币coin和不使用，取最小值 + dp[i] = Math.min(dp[i - coin] + 1, dp[i]); + } + } + } + return dp[amount] == amount + 1 ? -1 : dp[amount]; +} +``` + +> 为啥 `dp` 数组初始化为 `amount + 1` 呢，因为凑成 `amount` 金额的硬币数最多只可能等于 `amount`（全用 1 元面值的硬币），所以初始化为 `amount + 1` 就相当于初始化为正无穷，便于后续取最小值。为啥不直接初始化为 int 型的最大值 `Integer.MAX_VALUE` 呢？因为后面有 `dp[i - coin] + 1`，这就会导致整型溢出。 +> +> 最终，dp[amout] 就是凑成总金额所需的最少硬币数，如果dp[amount] 仍是初始化的较大值，说明无法凑出，返回 -1。 + + + +### 6、[买卖股票的最佳时机](https://leetcode-cn.com/problems/best-time-to-buy-and-sell-stock/)（leetocode_121） + +> 给定一个数组，它的第 i 个元素是一支给定股票第 i 天的价格。 +> +>如果你最多只允许完成一笔交易（即买入和卖出一支股票一次），设计一个算法来计算你所能获取的最大利润。 +> +>注意：你不能在买入股票前卖出股票。 +> +>``` +> 输入: [7,1,5,3,6,4] +>输出: 5 +> 解释: 在第 2 天（股票价格 = 1）的时候买入，在第 5 天（股票价格 = 6）的时候卖出，最大利润 = 6-1 = 5 。 +> 注意利润不能是 7-1 = 6, 因为卖出价格需要大于买入价格；同时，你不能在买入前卖出股票。 +> ``` +> +> ``` +> 输入: [7,6,4,3,1] +>输出: 0 +> 解释: 在这种情况下, 没有交易完成, 所以最大利润为 0。 +>``` + +#### 分析题目 + +我们需要找出给定数组中两个数字之间的最大差值（即，最大利润）。此外，第二个数字（卖出价格）必须大于第一个数字（买入价格） + +```java +public int dp(int[] prices) { + int length = prices.length; + if (length == 0) { + return 0; + } + int dp[] = new int[length]; + //保存一个最小值 + int minPrice = prices[0]; + for (int i = 1; i < length; i++) { + minPrice = Math.min(minPrice, prices[i]); + dp[i] = Math.max(dp[i - 1], prices[i] - minPrice); + } + return dp[length - 1]; +} +``` + + + +### 7、最长回文子串（leetcode_5） + +> 给定一个字符串 s，找到 s 中最长的回文子串。你可以假设 s 的最大长度为 1000。 +> +> ``` +>输入: "babad" +> 输出: "bab" +> 注意: "aba" 也是一个有效答案。 +> ``` + +#### 分析题目 + +回文的意思是正着念和倒着念一样，如：大波美人鱼人美波大 + +建立二维数组 `dp` ，找出所有的回文子串。 + +回文串两边加上两个相同字符，会形成一个新的回文串 。 + +![](https://writings.sh/assets/images/posts/algorithm-longest-palindromic-substring/longest-palindromic-substring-dp-2-1.jpeg) + + + +我们用`dp[i][j]` 记录子串 `i..j` 是否为回文串 。 + +![img](https://writings.sh/assets/images/posts/algorithm-longest-palindromic-substring/longest-palindromic-substring-dp-2-2.jpeg) + +首先，单个字符就形成一个回文串，所以，所有 `dp[i][i] = true` 。 + +![img](https://writings.sh/assets/images/posts/algorithm-longest-palindromic-substring/longest-palindromic-substring-dp-2-3.jpeg) + +然后，容易得到递推关系： + +如果字符 `s[i]` 和 `s[j]` 相等，并且子串 `i+1..j-1` 是回文串的话，子串 `i..j` 也是回文串。 + +也就是，如果 `s[i] == s[j]` 且 `dp[i+1][j-1] = true` 时，`dp[i][j] = true` 。 + +![img](https://writings.sh/assets/images/posts/algorithm-longest-palindromic-substring/longest-palindromic-substring-dp-2-4.jpeg) + +这是本方法中主要的递推关系。 + +不过仍要注意边界情况，即 子串 `i+1..j-1` 的有效性 ，当 `i+1 <= j-1` 时，它才有效。 + +反之，如果不满足，此时 `j <= i+1` ，也就是子串 `i..j` 最多有两个字符， 如果两个字符 `s[i]` 和 `s[j]` 相等，那么是回文串。 + +```java +public String longestPalindrome_1(String s) { + int length = s.length(); + if (length < 2) { + return s; + } + boolean dp[][] = new boolean[length][length]; + for (int i = 0; i < length; i++) { + dp[i][i] = true; + } + + char[] chars = s.toCharArray(); + + //通过最大长度定位回文串位置，或者也可以用个数组记录int[] res = new int[2]; + int maxLen = 1; + int begin = 0; + for (int r = 1; r < length; r++) { + for (int l = 0; l < r; l++) { + if (chars[l] != chars[r]) { + dp[l][r] = false; + } else { + // 特例，如果 是 abaa 这种，需要最后一个和第一个也相等，但是他们距离大于等于了3，所以还需要往里判断 + if (r - l < 3) { + dp[l][r] = true; + } else { + dp[l][r] = dp[l + 1][r - 1]; + } + } + + //只要 dp[l][r] == true 成立，就表示子串 s[i..j] 是回文，此时记录回文长度和起始位置 + if (dp[l][r] && r - l + 1 > maxLen) { + maxLen = r - l + 1; + begin = l; + } + } + } + return s.substring(begin, begin + maxLen); +} +``` + + + +### 8、数字三角形问题 + +``` +7 +3 8 +8 1 0 +2 7 4 4 +4 5 2 6 5 +``` + +从上到下选择一条路，使得经过的数字之和最大。 + +路径上的每一步只能往左下或者右下走。 + +#### 分析题目 + +递归解法 + +可以看出每走第n行第m列时有两种后续:向下或者向右下。由于最后一行可以确定,当做边界条件,所以我们自然而然想到递归求解 + +```java +class Solution{ + + public int getMax(){ + int MAX = 101; + int[][] D = new int[MAX][MAX]; //存储数字三角形 + int n; //n表示层数 + int i = 0; int j = 0; + int maxSum = getMaxSum(D,n,i,j); + return maxSum; + } + + public int getMaxSum(int[][] D,int n,int i,int j){ + if(i == n){ + return D[i][j]; + } + int x = getMaxSum(D,n,i+1,j); + int y = getMaxSum(D,n,i+1,j+1); + return Math.max(x,y)+D[i][j]; + } +} +``` + + + + + +## 总结 + +![](https://pic2.zhimg.com/80/v2-4e3a7d5ae4bb76ce96bc3393013f13f8_720w.jpg?source=1940ef5c) + + + +## 番外篇 + +### 动态规划与其它算法的关系 + +#### **1. 贪心算法（Greedy Algorithm）** + +**核心思想**：每一步都做出当前状态下的**局部最优选择**（即 “贪心选择”），不考虑该选择对后续步骤的影响，最终通过局部最优的累积试图得到全局最优解。本质是 “短视的”：只看眼前，不回头。 + +**特点**： + +- 必须满足**贪心选择性质**：局部最优选择能导致全局最优解（否则贪心会失效）。 +- 子问题无依赖：每一步的选择不影响后续子问题的求解（子问题独立）。 + +**适用场景**： + +- 最优子结构明确（如霍夫曼编码） +- 贪心选择性质成立（如活动选择问题） +- 无需全局最优验证（如零钱兑换特殊场景） + +#### 2. 分治法（Divide and Conquer） + +**核心思想**： 将原问题**分解为若干个规模更小、结构相同的子问题**，递归求解子问题后，**合并子问题的解**得到原问题的解。 +本质是 “分而治之”：拆分独立子问题，逐个击破再整合。 + +**特点**： + +- **递归结构**：子问题相互独立（无重叠） +- **合并成本高**：结果合并是关键步骤 +- **并行潜力**：子问题可并发求解 + +**适用场景**： + +- 问题可自然拆分（如排序、树操作） +- 子问题规模相似（如二分搜索） +- 合并操作复杂度可控（如归并排序） + +#### **3. 动态规划（Dynamic Programming）** + +**核心思想**： 对于具有**重叠子问题**和**最优子结构**的问题，将其分解为子问题后，**存储子问题的解（记忆化）** 以避免重复计算，通过子问题的解推导出原问题的解。本质是 “精打细算的”：记录历史，避免重复劳动。 + +**特点**： + +- **最优子结构**：全局最优包含局部最优 +- **重叠子问题**：子问题反复出现（用表存储） +- **状态转移方程**：定义问题间递推关系 + +**适用场景**： + +- 子问题重叠（如斐波那契数列） +- 多阶段决策最优解（如背包问题） +- 需要回溯最优路径（如最长公共子序列） + +| **特性** | 贪心算法 | 分治法 | 动态规划 | +| -------------- | ---------------------- | -------------------- | ------------------ | +| **决策依据** | 当前局部最优 | 子问题独立解 | 历史子问题最优解 | +| **子问题关系** | 无重复计算 | 完全独立 | 高度重叠 | +| **解空间处理** | 永不回溯 | 显式分割 | 存储+重用 | +| **时间复杂度** | 通常最低 | 中等（依赖合并成本） | 通常较高 | +| **结果保证** | **不保证全局最优** | 保证正确解 | **保证全局最优** | +| **经典案例** | Dijkstra算法、活动选择 | 归并排序、快速排序 | 背包问题、最短路径 | + + + +## Reference + +- http://netedu.xauat.edu.cn/jpkc/netedu/jpkc/ycx/kcjy/kejian/pdf/05.pdf + +- https://leetcode-cn.com/circle/article/lxC3ZB/ + +- https://labuladong.gitbook.io/algo/dong-tai-gui-hua-xi-lie/dong-tai-gui-hua-xiang-jie-jin-jie + +- https://www.zhihu.com/question/39948290 + +- https://zhuanlan.zhihu.com/p/26743197 + +- https://writings.sh/post/algorithm-longest-palindromic-substrings + diff --git a/docs/data-structure-algorithms/algorithm/Greedy.md b/docs/data-structure-algorithms/algorithm/Greedy.md new file mode 100644 index 0000000000..c08936ef44 --- /dev/null +++ b/docs/data-structure-algorithms/algorithm/Greedy.md @@ -0,0 +1,118 @@ +--- +title: 贪心算法——刷题有套路 +date: 2024-09-09 +tags: + - Algorithm + - Greedy +categories: Algorithm +--- + +![](https://cdn.pixabay.com/photo/2025/04/09/14/55/candy-9524410_1280.jpg) + +> 贪心算法（greedy algorithm）是一种常见的解决优化问题的算法，其基本思想是在问题的每个决策阶段，都选择当前看起来最优的选择，即贪心地做出局部最优的决策，以期获得全局最优解。贪心算法简洁且高效，在许多实际问题中有着广泛的应用。 +> +> 贪心算法和动态规划都常用于解决优化问题。它们之间存在一些相似之处，比如都依赖最优子结构性质，但工作原理不同。 +> +> - 动态规划会根据之前阶段的所有决策来考虑当前决策，并使用过去子问题的解来构建当前子问题的解。 +> - 贪心算法不会考虑过去的决策，而是一路向前地进行贪心选择，不断缩小问题范围，直至问题被解决。 + + + +### 贪心算法的应用场景 + +解决一个问题需要多个步骤，每一个步骤有多种选择。可以使用贪心算法解决的问题，每一步只需要解决一个子问题，只做出一种选择，就可以完成任务。 + +### 贪心算法特性 + +较于动态规划，贪心算法的使用条件更加苛刻，其主要关注问题的两个性质。 + +- **贪心选择性质**：只有当局部最优选择始终可以导致全局最优解时，贪心算法才能保证得到最优解。 +- **最优子结构**：原问题的最优解包含子问题的最优解。 + +#### 贪心算法与回溯算法、动态规划的区别 + +「解决一个问题需要多个步骤，每一个步骤有多种选择」这样的描述我们在「回溯算法」「动态规划」算法中都会看到。它们的区别如下： + +- 「回溯算法」需要记录每一个步骤、每一个选择，用于回答所有具体解的问题； +- 「动态规划」需要记录的是每一个步骤、所有选择的汇总值（最大、最小或者计数）； +- 「贪心算法」由于适用的问题，每一个步骤只有一种选择，一般而言只需要记录与当前步骤相关的变量的值。 + +对于不同的求解目标和不同的问题场景，需要使用不同的算法。 + +#### 可以使用「贪心算法」的问题需要满足的条件 + +- 最优子结构：规模较大的问题的解由规模较小的子问题的解组成，区别于「动态规划」，可以使用「贪心算法」的问题「规模较大的问题的解」只由其中一个「规模较小的子问题的解」决定； +- 无后效性：后面阶段的求解不会修改前面阶段已经计算好的结果； + +- 贪心选择性质：从局部最优解可以得到全局最优解。 + +对「最优子结构」和「无后效性」的理解同「动态规划」，「贪心选择性质」是「贪心算法」最需要关注的内容。 + + + +### 贪心算法解题步骤 + +贪心问题的解决流程大体可分为以下三步。 + +1. **问题分析**：梳理与理解问题特性，包括状态定义、优化目标和约束条件等。这一步在回溯和动态规划中都有涉及。 +2. **确定贪心策略**：确定如何在每一步中做出贪心选择。这个策略能够在每一步减小问题的规模，并最终解决整个问题。 +3. **正确性证明**：通常需要证明问题具有贪心选择性质和最优子结构。这个步骤可能需要用到数学证明，例如归纳法或反证法等。 + +确定贪心策略是求解问题的核心步骤，但实施起来可能并不容易，主要有以下原因。 + +- **不同问题的贪心策略的差异较大**。对于许多问题来说，贪心策略比较浅显，我们通过一些大概的思考与尝试就能得出。而对于一些复杂问题，贪心策略可能非常隐蔽，这种情况就非常考验个人的解题经验与算法能力了。 +- **某些贪心策略具有较强的迷惑性**。当我们满怀信心设计好贪心策略，写出解题代码并提交运行，很可能发现部分测试样例无法通过。这是因为设计的贪心策略只是“部分正确”的，上文介绍的零钱兑换就是一个典型案例。 + + + +#### 「力扣」第 455 题：分发饼干（简单） + +> 假设你是一位很棒的家长，想要给你的孩子们一些小饼干。但是，每个孩子最多只能给一块饼干。 +> +> 对每个孩子 i，都有一个胃口值 g[i]，这是能让孩子们满足胃口的饼干的最小尺寸；并且每块饼干 j，都有一个尺寸 s[j] 。如果 s[j] >= g[i]，我们可以将这个饼干 j 分配给孩子 i ，这个孩子会得到满足。你的目标是尽可能满足越多数量的孩子，并输出这个最大数值。 +> +> 示例 1: +> +> 输入: g = [1,2,3], s = [1,1] +> 输出: 1 +> 解释: +> 你有三个孩子和两块小饼干，3个孩子的胃口值分别是：1,2,3。 +> 虽然你有两块小饼干，由于他们的尺寸都是1，你只能让胃口值是1的孩子满足。 +> 所以你应该输出1。 + +```java +class Solution { + public int findContentChildren(int[] g, int[] s) { + Arrays.sort(g); + Arrays.sort(s); + int numOfChildren = g.length, numOfCookies = s.length; + int count = 0; + for (int i = 0, j = 0; i < numOfChildren && j < numOfCookies; i++, j++) { + while (j < numOfCookies && g[i] > s[j]) { + j++; + } + if (j < numOfCookies) { + count++; + } + } + return count; + } +} +``` + +「贪心算法」总是做出在当前看来最好的选择就可以完成任务； +解决「贪心算法」几乎没有套路，到底如何贪心，贪什么与我们要解决的问题密切相关。因此刚开始学习「贪心算法」的时候需要学习和模仿，然后才有直觉，猜测一个问题可能需要使用「贪心算法」，进而尝试证明，学会证明。 + + + +## 贪心算法典型例题 + +贪心算法常常应用在满足贪心选择性质和最优子结构的优化问题中，以下列举了一些典型的贪心算法问题。 + +- **硬币找零问题**：在某些硬币组合下，贪心算法总是可以得到最优解。 +- **区间调度问题**：假设你有一些任务，每个任务在一段时间内进行，你的目标是完成尽可能多的任务。如果每次都选择结束时间最早的任务，那么贪心算法就可以得到最优解。 +- **分数背包问题**：给定一组物品和一个载重量，你的目标是选择一组物品，使得总重量不超过载重量，且总价值最大。如果每次都选择性价比最高（价值 / 重量）的物品，那么贪心算法在一些情况下可以得到最优解。 +- **股票买卖问题**：给定一组股票的历史价格，你可以进行多次买卖，但如果你已经持有股票，那么在卖出之前不能再买，目标是获取最大利润。 +- **霍夫曼编码**：霍夫曼编码是一种用于无损数据压缩的贪心算法。通过构建霍夫曼树，每次选择出现频率最低的两个节点合并，最后得到的霍夫曼树的带权路径长度（编码长度）最小。 +- **Dijkstra 算法**：它是一种解决给定源顶点到其余各顶点的最短路径问题的贪心算法。 + diff --git a/docs/data-structure-algorithms/algorithm/Recursion.md b/docs/data-structure-algorithms/algorithm/Recursion.md new file mode 100644 index 0000000000..f8a1992021 --- /dev/null +++ b/docs/data-structure-algorithms/algorithm/Recursion.md @@ -0,0 +1,389 @@ +--- +title: 递归算法 +date: 2024-05-09 +tags: + - Recursion +categories: Algorithm +--- + +![](https://img.starfish.ink/algorithm/recursion-banner.png) + + + +### 什么是递归 + +递归的基本思想是某个函数直接或者间接地调用自身，这样就把原问题的求解转换为许多性质相同但是规模更小的子问题。我们只需要关注如何把原问题划分成符合条件的子问题，而不需要去研究这个子问题是如何被解决的。 + +**简单地说，就是如果在函数中存在着调用函数本身的情况，这种现象就叫递归。** + +你以前肯定写过递归，只是有可能某些不知道这就是递归罢了。 + +以阶乘函数为例，在 factorial 函数中存在着 `factorial(n - 1)` 的调用，所以此函数是递归函数 + +```java +public long factorial(int n) { + if (n < =1) { + return 1; + } + return n * factorial(n - 1) +} +``` + +进一步剖析「递归」，先有「递」再有「归」，「递」的意思是将问题拆解成子问题来解决， 子问题再拆解成子子问题，...，直到被拆解的子问题无需再拆分成更细的子问题（即可以求解），「归」是说最小的子问题解决了，那么它的上一层子问题也就解决了，上一层的子问题解决了，上上层子问题自然也就解决了,....,直到最开始的问题解决，文字说可能有点抽象，那我们就以阶层 f(6) 为例来看下它的「递」和「归」。 + +![img](https://img.starfish.ink/algorithm/recursion.png) + +求解问题 `f(6)`，由于 `f(6) = n * f(5)`, 所以 `f(6)` 需要拆解成 `f(5)` 子问题进行求解，同理 `f(5) = n * f(4) `,也需要进一步拆分,... ,直到 `f(1)`, 这是「递」，`f(1) `解决了，由于 `f(2) = 2 f(1) = 2` 也解决了,.... `f(n)` 到最后也解决了，这是「归」，所以递归的本质是能把问题拆分成具有**相同解决思路**的子问题，。。。直到最后被拆解的子问题再也不能拆分，解决了最小粒度可求解的子问题后，在「归」的过程中自然顺其自然地解决了最开始的问题。 + + + +### 递归原理 + +递归的基本思想是某个函数直接或者间接地调用自身，这样就把原问题的求解转换为许多性质相同但是规模更小的子问题。 + +我们只需要关注如何把原问题划分成符合条件的子问题，而不需要去研究这个子问题是如何被解决的。 + +递归和枚举的区别在于：枚举是横向地把问题划分，然后依次求解子问题，而递归是把问题逐级分解，是纵向的拆分。 + +```java +int func(你今年几岁) { + // 最简子问题，递归终止条件 + if (你1999年几岁) return 我0岁; + // 递归调用，缩小规模 + return func(你去年几岁) + 1; +} +``` + +任何一个有意义的递归算法总是两部分组成：**递归调用**和**递归终止条件**。 + +为了确保递归函数不会导致无限循环，它应具有以下属性： + +1. 一个简单的`基本案例（basic case）`（或一些案例） —— 能够不使用递归来产生答案的终止方案。 +2. 一组规则，也称作`递推关系（recurrence relation）`，可将所有其他情况拆分到基本案例。 + +注意，函数可能会有多个位置进行自我调用。 + +```java +public returnType recursiveFunction(parameters) { + // 1. 递归终止条件（Base Case） + if (isBaseCase(parameters)) { + return baseCaseResult; // 返回问题的基本解，避免继续递归 + } + + // 2. 递归调用（Recursive Call） + // 将当前问题分解为更小的子问题 + return recursiveFunction(modifiedParameters); + + // 或者对于分治问题，递归调用多个子问题： + // return combineResults( + // recursiveFunction(subProblem1), + // recursiveFunction(subProblem2) + // ); +} + +``` + + + +### 递归需要满足的三个条件 + +只要同时满足以下三个条件，就可以用递归来解决。 + +1. **一个问题的解可以分解为几个子问题的解** + + 何为子问题？子问题就是数据规模更小的问题。比如前面的案例，要知道“自己在哪一排”，可以分解为“前一排的人在哪一排”这样的一个子问题。 + +2. **这个问题与分解之后的子问题，除了数据规模不同，求解思路完全一样** + + 如案例所示，求解“自己在哪一排”的思路，和前面一排人求解“自己在哪一排”的思路是一模一样的。 + +3. **存在递归终止条件** + + 把问题分解为子问题，把子问题再分解为子子问题，一层一层分解下去，不能存在无限循环，这就需要有终止条件。 + + + +### 怎样编写递归代码 + +写递归代码，可以按三步走： + +**第一要素：明确你这个函数想要干什么** + +首先，你需要明确你要解决的问题，以及这个问题是否适合用递归来解决。 + +也就是说，我们先不管函数里面的代码什么，而是要先明白，你这个函数是要用来干什么。 + +例如，我定义了一个函数，算 n 的阶乘 + +```java +// 算 n 的阶乘(假设n不为0) +int factorial(int n){ + +} +``` + +**第二要素：寻找递归结束条件** + +递归基（Base Case）是递归函数结束的条件。如果没有递归基，递归函数会无限循环下去。递归基应该是问题的最小单位，在这种情况下，函数可以直接返回结果而无需进一步递归。 + +也就是说，我们需要找出**当参数为啥时，递归结束，之后直接把结果返回**，请注意，这个时候我们必须能根据这个参数的值，能够**直接**知道函数的结果是什么。 + +例如，上面那个例子，当 n = 1 时，那你应该能够直接知道 f(n) 是啥吧？此时，f(1) = 1。完善我们函数内部的代码，把第二要素加进代码里面 + +```java +// 算 n 的阶乘(假设n不为0) +int f(int n){ + if(n == 1){ + return 1; + } +} +``` + +有人可能会说，当 n = 2 时，那我们可以直接知道 f(n) 等于多少啊，那我可以把 n = 2 作为递归的结束条件吗？ + +当然可以，只要你觉得参数是什么时，你能够直接知道函数的结果，那么你就可以把这个参数作为结束的条件，所以下面这段代码也是可以的。 + +```java +// 算 n 的阶乘(假设n>=2) +int f(int n){ + if(n == 2){ + return 2; + } +} +``` + +注意我代码里面写的注释，假设 n >= 2，因为如果 n = 1时，会被漏掉，当 n <= 2时，f(n) = n，所以为了更加严谨，我们可以写成这样： + +```java +// 算 n 的阶乘(假设n不为0) +int f(int n){ + if(n <= 2){ + return n; + } +} +``` + +**第三要素：定义递归关系** + +第三要素就是，我们要**不断缩小参数的范围**，缩小之后，我们可以通过一些辅助的变量或者操作，使原函数的结果不变。 + +例如，f(n) 这个范围比较大，我们可以让 `f(n) = n * f(n-1)`。这样，范围就由 n 变成了 n-1 了，范围变小了，并且为了原函数 f(n) 不变，我们需要让 f(n-1) 乘以 n。 + +说白了，就是要找到原函数的一个等价关系式，`f(n)` 的等价关系式为 `n * f(n-1)`，即 `f(n) = n * f(n-1)`。 + +> **写递归代码的关键就是找到如何将大问题分解为小问题的规律，请求基于此写出递推公式，然后再推敲终止条件，最后将递推公式和终止条件翻译成代码**。 +> +> 当我们面对一个问题需要分解为多个子问题的时候，递归代码往往没那么好理解，比如第二个案例，人脑几乎没办法把整个“递”和“归”的过程一步一步都想清楚。 +> +> 计算机擅长做重复的事情，所以递归正符合它的胃口。而我们人脑更喜欢平铺直叙的思维方式。当我们看到递归时，我们总想把递归平铺展开，脑子里就会循环，一层一层往下调，然后再一层一层返回，试图想搞清楚计算机每一步都是怎么执行的，这样就很容易被绕进去。 +> +> 对于递归代码，这种试图想清楚整个递和归过程的做法，实际上是进入了一个思维误区。很多时候，我们理解起来比较吃力，主要原因就是自己给自己制造了这种理解障碍。那正确的思维方式应该是怎样的呢？ +> +> 如果一个问题 A 可以分解为若干子问题 B、C、D，可以假设子问题 B、C、D 已经解决，在此基础上思考如何解决问题 A。而且，只需要思考问题 A 与子问题 B、C、D 两层之间的关系即可，不需要一层一层往下思考子问题与子子问题，子子问题与子子子问题之间的关系。屏蔽掉递归细节，这样子理解起来就简单多了。 +> +> 换句话说就是：千万不要跳进这个函数里面企图探究更多细节，否则就会陷入无穷的细节无法自拔，人脑能压几个栈啊。 +> +> 所以，编写递归代码的关键是：**只要遇到递归，我们就把它抽象成一个递推公式，不用想一层层的调用关系，不要试图用人脑去分解递归的每个步骤**。 + + + +#### 递归代码要警惕堆栈溢出 + +递归调用的深度受限于程序的栈空间。如果递归深度太深，可能会导致栈溢出。对于深度较大的递归问题，可以考虑使用迭代方法或尾递归优化（Tail Recursion Optimization）。 + + + +#### 递归代码要警惕重复计算 + +在某些问题中（如斐波那契数列），直接递归可能导致大量重复计算，影响效率。可以使用记忆化（Memoization）或动态规划（Dynamic Programming）来优化递归，避免重复计算。 + + + +### 案例 + +#### 斐波那契数列 + +> 斐波那契数列的是这样一个数列：1、1、2、3、5、8、13、21、34....，即第一项 f(1) = 1,第二项 f(2) = 1.....,第 n 项目为 f(n) = f(n-1) + f(n-2)。求第 n 项的值是多少。 + +```java +int f(int n){ + // 1.先写递归结束条件 + if(n <= 2){ + return 1; + } + // 2.接着写等价关系式 + return f(n-1) + f(n - 2); +} +``` + + + +#### 小青蛙跳台阶 + +> 一只青蛙一次可以跳上1级台阶，也可以跳上2级。求该青蛙跳上一个n级的台阶总共有多少种跳法。 + +每次跳的时候，小青蛙可以跳一个台阶，也可以跳两个台阶，也就是说，每次跳的时候，小青蛙有两种跳法。 + +第一种跳法：第一次我跳了一个台阶，那么还剩下n-1个台阶还没跳，剩下的n-1个台阶的跳法有f(n-1)种。 + +第二种跳法：第一次跳了两个台阶，那么还剩下n-2个台阶还没，剩下的n-2个台阶的跳法有f(n-2)种。 + +所以，小青蛙的全部跳法就是这两种跳法之和了，即 f(n) = f(n-1) + f(n-2)。至此，等价关系式就求出来了 + +```java +int f(int n){ + // 1.先写递归结束条件 + if(n == 1){ + return 1; + } + // 2.接着写等价关系式 + ruturn f(n-1) + f(n-2); +} +``` + +大家觉得上面的代码对不对？ + +答是不大对，当 n = 2 时，显然会有 f(2) = f(1) + f(0)。我们知道，f(0) = 0，按道理是递归结束，不用继续往下调用的，但我们上面的代码逻辑中，会继续调用 f(0) = f(-1) + f(-2)。这会导致无限调用，进入**死循环**。 + +这也是我要和你们说的，关于**递归结束条件是否够严谨问题**，有很多人在使用递归的时候，由于结束条件不够严谨，导致出现死循环。也就是说，当我们在第二步找出了一个递归结束条件的时候，可以把结束条件写进代码，然后进行第三步，但是**请注意**，当我们第三步找出等价函数之后，还得再返回去第二步，根据第三步函数的调用关系，会不会出现一些漏掉的结束条件。就像上面，f(n-2)这个函数的调用，有可能出现 f(0) 的情况，导致死循环，所以我们把它补上。代码如下： + +```java +int f(int n){ + //f(0) = 0,f(1) = 1，等价于 n<=1时，f(n) = n。 + if(n <= 1){ + return n; + } + ruturn f(n-1) + f(n-2); +} +``` + +#### 反转单链表 + +> 反转单链表。例如链表为：1->2->3->4。反转后为 4->3->2->1 + +链表的节点定义如下： + +```java +class Node{ + int date; + Node next; +} +``` + +这个的等价关系不像 n 是个数值那样，比较容易寻找。但是我告诉你，它的等价条件中，一定是范围不断在缩小，对于链表来说，就是链表的节点个数不断在变小 + +```java +//用递归的方法反转链表 +public Node reverseList(Node head){ + // 1.递归结束条件 + if (head == null || head.next == null) { + return head; + } + // 递归反转 子链表 + Node newList = reverseList(head.next); + // 改变 1，2节点的指向。 + // 通过 head.next获取节点2 + Node t1 = head.next; + // 让 2 的 next 指向 2 + t1.next = head; + // 1 的 next 指向 null. + head.next = null; + // 把调整之后的链表返回。 + return newList; + } +``` + + + +#### [两两交换链表中的节点](https://leetcode.cn/problems/swap-nodes-in-pairs/) + +> 给你一个链表，两两交换其中相邻的节点，并返回交换后链表的头节点。你必须在不修改节点内部的值的情况下完成本题（即，只能进行节点交换）。 +> +> ![img](https://assets.leetcode.com/uploads/2020/10/03/swap_ex1.jpg) +> +> ``` +> 输入：head = [1,2,3,4] +> 输出：[2,1,4,3] +> ``` + +当前节点 next，指向当前节点，指针互换 + +```java + public ListNode swapPairs(ListNode head) { + // 基本情况：链表为空或只有一个节点 + if (head == null || head.next == null) { + return head; + } + + // 交换前两个节点 + ListNode first = head; + ListNode second = head.next; + + // 递归交换后续的节点 + first.next = swapPairs(second.next); + + // 交换后的第二个节点成为新的头节点 + second.next = first; + + // 返回新的头节点 + return second; + } +``` + +下边这么写也可以，少了一个局部变量，交换操作和递归调用在一行内完成。 + +```java +public ListNode swapPairs(ListNode head) { + //递归的终止条件 + if(head==null || head.next==null) { + return head; + } + //假设链表是 1->2->3->4 + //这句就先保存节点2 + ListNode tmp = head.next; + //继续递归，处理节点3->4 + //当递归结束返回后，就变成了4->3 + //于是head节点就指向了4，变成1->4->3 + head.next = swapPairs(tmp.next); + //将2节点指向1 + tmp.next = head; + return tmp; +} +``` + +当然，也可以迭代实现~ + +![img](https://miro.medium.com/v2/resize:fit:1400/1*imD5_rA0Hkov-kWXSUdr2Q.png) + +```java +public class Solution { + public ListNode swapPairs(ListNode head) { + // 创建虚拟头节点，指向链表的头节点 + ListNode dummy = new ListNode(0); + dummy.next = head; + + // 当前节点指针，初始化为虚拟头节点 + ListNode current = dummy; + + // 遍历链表 + while (current.next != null && current.next.next != null) { + // 初始化两个要交换的节点 + ListNode first = current.next; + ListNode second = current.next.next; + + // 交换这两个节点 + first.next = second.next; + second.next = first; + current.next = second; + + // 移动 current 指针到下一个需要交换的位置 + current = first; + } + + // 返回交换后的链表头 + return dummy.next; + } +} + +``` + diff --git a/docs/data-structure-algorithms/algorithm/Sort.md b/docs/data-structure-algorithms/algorithm/Sort.md new file mode 100644 index 0000000000..dbbc9842d3 --- /dev/null +++ b/docs/data-structure-algorithms/algorithm/Sort.md @@ -0,0 +1,826 @@ +--- +title: 排序 +date: 2023-10-09 +tags: + - Algorithm +categories: Algorithm +--- + +![img](https://miro.medium.com/v2/resize:fit:1400/1*ZY2e2BNXTYBf9aBBHALCAw.png) + +> 🔢 **排序算法**，从接触计算机学科就会遇到的一个问题。 + +排序算法可以分为**内部排序**和**外部排序**： +- 🧠 **内部排序**：数据记录在内存中进行排序 +- 💾 **外部排序**：因排序的数据很大，一次不能容纳全部的排序记录，在排序过程中需要访问外存 + +常见的内部排序算法有：**插入排序、希尔排序、选择排序、冒泡排序、归并排序、快速排序、堆排序、基数排序**等。 + +| 排序算法 | 平均时间复杂度 | 最好情况 | 最坏情况 | 空间复杂度 | 稳定性 | +| :----------------------: | -------------- | ----------------------------------------------------- | --------------------------------------------------- | ---------------------------------------------------------- | ---------------------------------------------- | +| 冒泡排序-Bubble Sort | $O(n²)$ | $O(n)$（当数组已经有序时，只需遍历一次） | $O(n²)$（当数组是逆序时，需要多次交换） | $O(1)$（原地排序） | 稳定 | +| 选择排序-Selection Sort | $O(n²)$ | $O(n²)$ | $O(n²)$ | $O(1)$（原地排序） | 不稳定（交换元素时可能破坏相对顺序） | +| 插入排序-Insertion Sort | $O(n²)$ | $O(n)$（当数组已经有序时） | $O(n²)$（当数组是逆序时） | $O(1)$（原地排序） | 稳定 | +| 快速排序-Quick Sort | $O(n \log n)$ | $O(n \log n)$（每次划分的子数组大小相等） | $O(n²)$（每次选取的基准值使得数组划分非常不平衡） | $O(\log n)$（对于递归栈） $O(n)$（最坏情况递归栈深度为 n） | 不稳定（交换可能改变相同元素的相对顺序） | +| 希尔排序-Shell Sort | $O(n \log² n)$ | $O(n \log n)$ | $O(n²)$（不同的增量序列有不同的最坏情况） | $O(1)$（原地排序） | 不稳定 | +| 归并排序-Merge Sort | $O(n \log n)$ | $O(n \log n)$ | $O(n \log n)$ | $O(n)$（需要额外的空间用于辅助数组） | 稳定 | +| 堆排序-Heap Sort | $O(n \log n)$ | $O(n \log n)$ | $O(n \log n)$ | $O(1)$（原地排序） | 不稳定（在调整堆时可能改变相同元素的相对顺序） | +| 计数排序-Counting Sort） | $O(n + k)$ | $O(n + k)$（k 是数组中元素的取值范围） | $O(n + k)$ | $O(n + k)$（需要额外的数组来存储计数结果） | 稳定 | +| 桶排序-Bucket Sort） | $O(n + k)$ | $O(n + k)$（k 是桶的数量，n 是元素数量） | $O(n²)$（所有元素都集中到一个桶里，退化成冒泡排序） | $O(n + k)$ | 稳定 | +| 基数排序-Radix Sort | $O(d(n + k))$ | $O(d(n + k))$（d 是位数，k 是取值范围，n 是元素数量） | $O(d(n + k))$ | $O(n + k)$ | 稳定 | + +## 📊 排序算法分类 + +十种常见排序算法可以分为两大类： + +### 🔄 非线性时间比较类排序 +通过比较来决定元素间的相对次序，由于其时间复杂度不能突破$O(nlogn)$，因此称为非线性时间比较类排序。 + +> 💡 **特点**：需要比较元素大小，时间复杂度下界为 $O(n\log n)$ + +### ⚡ 线性时间非比较类排序 +不通过比较来决定元素间的相对次序，它可以突破基于比较排序的时间下界，以线性时间运行，因此称为线性时间非比较类排序。 + +> 💡 **特点**：利用数据特性，可以达到线性时间复杂度 + + + +## 🫧 冒泡排序 + +冒泡排序（Bubble Sort）也是一种简单直观的排序算法。它重复地走访过要排序的数列，一次比较两个元素，如果他们的顺序错误就把他们交换过来。走访数列的工作是重复地进行直到没有再需要交换，也就是说该数列已经排序完成。这个算法的名字由来是因为越小的元素会经由交换慢慢"浮"到数列的顶端。 + +> 🎯 **算法特点**： +> - 简单易懂，适合学习排序思想 +> - 时间复杂度：$O(n²)$ +> - 空间复杂度：$O(1)$ +> - 稳定排序 + +作为最简单的排序算法之一，冒泡排序给我的感觉就像 Abandon 在单词书里出现的感觉一样，每次都在第一页第一位，所以最熟悉。冒泡排序还有一种优化算法，就是立一个 flag，当在一趟序列遍历中元素没有发生交换，则证明该序列已经有序。但这种改进对于提升性能来说并没有什么太大作用。 + +### 1. 算法步骤 + +1. 🔍 **比较相邻元素**：如果第一个比第二个大，就交换他们两个 +2. 🔄 **重复比较**：对每一对相邻元素作同样的工作，从开始第一对到结尾的最后一对。这步做完后，最后的元素会是最大的数 +3. 🎯 **缩小范围**：针对所有的元素重复以上的步骤，除了最后一个 +4. 🔁 **重复过程**：重复步骤1~3，直到排序完成 + +### 2. 动图演示 + +![img](https://www.runoob.com/wp-content/uploads/2019/03/bubbleSort.gif) + + + +### 3. 性能分析 + +- 🚀 **什么时候最快**：当输入的数据已经是正序时（都已经是正序了，我还要你冒泡排序有何用啊） +- 🐌 **什么时候最慢**：当输入的数据是反序时（写一个 for 循环反序输出数据不就行了，干嘛要用你冒泡排序呢，我是闲的吗） + +```java +//冒泡排序，a 表示数组， n 表示数组大小 +public void bubbleSort(int[] a) { + int n = a.length; + if (n <= 1) return; + // 外层循环遍历每一个元素 + for (int i = 0; i < n; i++) { + //提前退出冒泡循环的标志位 + boolean flag = false; + // 内层循环进行元素的比较与交换 + for (int j = 0; j < n - i - 1; j++) { + if (a[j] > a[j+1]) { + int temp = a[j]; + a[j] = a[j+1]; + a[j+1] = temp; + flag = true; //表示有数据交换 + } + } + if (!flag) break; //没有数据交换，提前退出。 + } +} +``` + +嵌套循环，应该立马就可以得出这个算法的时间复杂度为 $O(n²)$。 + +冒泡的过程只涉及相邻数据的交换操作，只需要常量级的临时空间，所以它的空间复杂度是 $O(1)$，是一个原地排序算法。 + + + +## 🎯 选择排序 + +选择排序的思路是这样的：首先，找到数组中最小的元素，拎出来，将它和数组的第一个元素交换位置，第二步，在剩下的元素中继续寻找最小的元素，拎出来，和数组的第二个元素交换位置，如此循环，直到整个数组排序完成。 + +> 🎯 **算法特点**： +> - 简单直观，容易理解 +> - 时间复杂度：$O(n²)$（无论什么情况） +> - 空间复杂度：$O(1)$ +> - 不稳定排序 + +选择排序是一种简单直观的排序算法，无论什么数据进去都是 $O(n²)$ 的时间复杂度。所以用到它的时候，数据规模越小越好。唯一的好处可能就是不占用额外的内存空间了吧。 + +### 1. 算法步骤 + +1. 🔍 **找最小元素**：首先在未排序序列中找到最小（大）元素，存放到排序序列的起始位置 +2. 🔄 **继续寻找**：再从剩余未排序元素中继续寻找最小（大）元素，然后放到已排序序列的末尾 +3. 🔁 **重复过程**：重复第二步，直到所有元素均排序完毕 + +### 2. 动图演示 + +![img](https://www.runoob.com/wp-content/uploads/2019/03/selectionSort.gif) + +```java +public void selectionSort(int [] arrs) { + for (int i = 0; i < arrs.length; i++) { + //最小元素下标 + int min = i; + for (int j = i +1; j < arrs.length; j++) { + if (arrs[j] < arrs[min]) { + min = j; + } + } + //如果当前位置i的元素已经是未排序部分的最小元素，就不需要交换了 + if(min != i){ + //交换位置 + int temp = arrs[i]; + arrs[i] = arrs[min]; + arrs[min] = temp; + } + } + } +} +``` + + + +## 📝 插入排序 + +插入排序的代码实现虽然没有冒泡排序和选择排序那么简单粗暴，但它的原理应该是最容易理解的了，因为只要打过扑克牌的人都应该能够秒懂。 + +> 🎯 **算法特点**： +> - 像整理扑克牌一样直观 +> - 时间复杂度：$O(n²)$（最坏），$O(n)$（最好） +> - 空间复杂度：$O(1)$ +> - 稳定排序 + +它的工作原理为将待排列元素划分为「已排序」和「未排序」两部分，每次从「未排序的」元素中选择一个插入到「已排序的」元素中的正确位置。![insertion sort animate example](https://oi-wiki.org/basic/images/insertion-sort-animate.svg) + +插入排序和冒泡排序一样，也有一种优化算法，叫做拆半插入。 + +### 1. 算法步骤 + +1. 🎯 **起始点**：从第一个元素开始（下标为 0 的元素），该元素可以认为已经被排序 +2. 🔍 **取下一个元素**：取出下一个元素，在已经排序的元素序列中**从后向前**扫描 +3. 🔄 **比较移动**：如果该元素（已排序）大于新元素，将该元素移到下一位置 +4. 🔁 **重复比较**：重复步骤3，直到找到已排序的元素小于或者等于新元素的位置 +5. 📍 **插入位置**：将新元素插入到该位置后 +6. 🔁 **重复过程**：重复步骤2~5 + +### 2. 动图演示 + +![img](https://www.runoob.com/wp-content/uploads/2019/03/insertionSort.gif) + +```java +public void insertionSort(int[] arr) { + // 从下标为1的元素开始选择合适的位置插入，因为下标为0的只有一个元素，默认是有序的 + for (int i = 1; i < arr.length; i++) { + int current = arr[i]; // 当前待插入元素 + int j = i - 1; // 已排序部分的末尾索引 + + // 从后向前扫描，找到插入位置 + while (j >= 0 && arr[j] > current) { + arr[j + 1] = arr[j]; // 元素后移 + j--; // 索引左移 + } + arr[j + 1] = current; // 插入到正确位置 + } +} +``` + + + +## ⚡ 快速排序 + +快速排序的核心思想是分治法，分而治之。它的实现方式是每次从序列中选出一个基准值，其他数依次和基准值做比较，比基准值大的放右边，比基准值小的放左边，然后再对左边和右边的两组数分别选出一个基准值，进行同样的比较移动，重复步骤，直到最后都变成单个元素，整个数组就成了有序的序列。 + +> 🎯 **算法特点**： +> - 分治法思想，效率高 +> - 时间复杂度：$O(n\log n)$（平均），$O(n²)$（最坏） +> - 空间复杂度：$O(\log n)$ +> - 不稳定排序 + +> 快速排序的最坏运行情况是 $O(n²)$，比如说顺序数列的快排。但它的平摊期望时间是 $O(nlogn)$，且 $O(nlogn)$ 记号中隐含的常数因子很小，比复杂度稳定等于 $O(nlogn) $的归并排序要小很多。所以，对绝大多数顺序性较弱的随机数列而言，快速排序总是优于归并排序。 +> +> ![img](https://miro.medium.com/v2/resize:fit:1400/1*DXQsNsa-DeGjsMtp7V6z9A.png) + +### 1. 算法步骤 + +1. 🎯 **选择基准值**：从数组中选择一个元素作为基准值（pivot）。常见的选择方法有选取第一个元素、最后一个元素、中间元素或随机选取一个元素 +2. 🔄 **分区（Partition）**：遍历数组，将所有小于基准值的元素放在基准值的左侧，大于基准值的元素放在右侧。基准值放置在它的正确位置上 +3. 🔁 **递归排序**：对基准值左右两边的子数组分别进行递归排序，直到每个子数组的元素个数为 0 或 1，此时数组已经有序 + +递归的最底部情形，是数列的大小是零或一，也就是数组都已经被排序好了。虽然一直递归下去，但是这个算法总会退出，因为在每次的迭代（iteration）中，它至少会把一个元素摆到它最后的位置去。 + +### 2. 动图演示 + +![img](https://www.runoob.com/wp-content/uploads/2019/03/quickSort.gif) + +```java +public class QuickSort { + // 对外暴露的排序方法：传入待排序数组 + public static void sort(int[] arr) { + if (arr == null || arr.length <= 1) { + return; // 数组为空或只有1个元素，无需排序 + } + // 调用递归方法：初始排序范围是整个数组（从0到最后一个元素） + quickSort(arr, 0, arr.length - 1); + } + + // 递归排序方法：排序 arr 的 [left, right] 区间 + private static void quickSort(int[] arr, int left, int right) { + // 递归终止条件：当 left >= right 时，子数组只有1个元素或为空，无需排序 + if (left >= right) { + return; + } + + // 1. 分区操作：返回基准值最终的索引位置 + // （把比基准小的放左边，比基准大的放右边，基准在中间） + int pivotIndex = partition(arr, left, right); + + quickSort(arr, left, pivotIndex - 1); + quickSort(arr, pivotIndex + 1, right); + } + + // 分区核心方法：选 arr[right] 为基准，完成分区并返回基准索引 + private static int partition(int[] arr, int left, int right) { + // 基准值：这里选当前区间的最后一个元素 + int pivot = arr[right]; + // i 指针：指向“小于基准的区域”的最后一个元素（初始时区域为空，i = left-1） + int i = left - 1; + + // j 指针：遍历当前区间的所有元素（从 left 到 right-1，跳过基准） + for (int j = left; j < right; j++) { + // 如果当前元素 arr[j] 小于等于基准，就加入“小于基准的区域” + if (arr[j] <= pivot) { + i++; // 先扩大“小于基准的区域” + swap(arr, i, j); // 交换 arr[i] 和 arr[j]，把 arr[j] 放进区域 + } + } + + // 最后：把基准值放到“小于基准区域”的后面（i+1 位置） + // 此时 i+1 就是基准的最终位置（左边都<=基准，右边都>基准） + swap(arr, i + 1, right); + return i + 1; + } + + // 辅助方法：交换数组中两个位置的元素 + private static void swap(int[] arr, int a, int b) { + int temp = arr[a]; + arr[a] = arr[b]; + arr[b] = temp; + } + +} +``` + +**📊 时间复杂度分析** + +**最优情况：$O(n \log n)$** + +- **适用场景**：每次分区操作都能将数组**均匀划分**（基准值接近中位数） +- **推导过程**： + - 每次分区将数组分为两个近似相等的子数组 + - 递归深度为 $\log_2 n$（分治次数），每层需遍历 $n$ 个元素 + - 总比较次数满足递推公式：$T(n) = 2T(n/2) + O(n)$ + - 通过主定理或递归树展开可得时间复杂度为 $O(n \log n)$ + +**最坏情况：$O(n²)$** + +- **适用场景**：每次分区划分极不均衡（如基准值始终为最大/最小元素） +- **常见案例**：输入数组已完全有序或逆序，且基准固定选择首/尾元素 +- **推导过程**： + - 每次分区仅减少一个元素（类似冒泡排序） + - 总比较次数为等差数列求和：$(n-1) + (n-2) + \cdots + 1 = \frac{n(n-1)}{2} = O(n^2)$ + - 递归深度为 $n-1$ 层，导致时间复杂度退化为 $O(n²)$ + +**平均情况：$O(n \log n)$** + +- **适用场景**：输入数据**随机分布**，基准值随机选取 + + + +## 🔀 归并排序 + +> ![img](https://miro.medium.com/v2/resize:fit:1400/1*1gyAaMcfcGIuZqrLJNDmgA.png) + +归并排序（Merge sort）是建立在归并操作上的一种有效的排序算法。该算法是采用分治法（Divide and Conquer）的一个非常典型的应用。 + +> 🎯 **算法特点**： +> - 分治法典型应用 +> - 时间复杂度：$O(n\log n)$（稳定） +> - 空间复杂度：$O(n)$ +> - 稳定排序 + +分治，就是分而治之，将一个大问题分解成小的子问题来解决。小的问题解决了，大问题也就解决了。 + +分治思想和递归思想很像。分治算法一般都是用递归来实现的。**分治是一种解决问题的处理思想，递归是一种编程技巧**，这两者并不冲突。 + +作为一种典型的分而治之思想的算法应用，归并排序的实现有两种方法： + +- 🔄 **自上而下的递归** +- 🔁 **自下而上的迭代**（所有递归的方法都可以用迭代重写，所以就有了第 2 种方法） + +和选择排序一样，归并排序的性能不受输入数据的影响，但表现比选择排序好的多，因为始终都是 $O(n\log n)$ 的时间复杂度。代价是需要额外的内存空间。 + +### 1. 算法步骤 + +1. 🔄 **分解**：将数组分成两半，递归地对每一半进行归并排序，直到每个子数组的大小为1（单个元素是有序的） +2. 🔀 **合并**：将两个有序子数组合并成一个有序数组 + +### 2. 动图演示 + +![img](https://www.runoob.com/wp-content/uploads/2019/03/mergeSort.gif) + + + +```java +public class MergeSort { + // 主排序函数 + public static void mergeSort(int[] arr) { + if (arr.length < 2) return; // 基本情况 + int mid = arr.length / 2; + + // 分解 + int[] left = new int[mid]; + int[] right = new int[arr.length - mid]; + + // 填充左子数组 + for (int i = 0; i < mid; i++) { + left[i] = arr[i]; + } + + // 填充右子数组 + for (int i = mid; i < arr.length; i++) { + right[i - mid] = arr[i]; + } + + // 递归排序 + mergeSort(left); + mergeSort(right); + + // 合并已排序的子数组 + merge(arr, left, right); + } + + // 合并两个有序数组 + private static void merge(int[] arr, int[] left, int[] right) { + // i、j、k 分别代表左子数组、右子数组、合并数组的指针 + int i = 0, j = 0, k = 0; + + // 合并两个有序数组，直到子数组都插入到合并数组 + while (i < left.length && j < right.length) { + //比较 left[i] 和 right[j], 左右哪边小的，就放入合并数组，指针要 +1 + if (left[i] <= right[j]) { + arr[k++] = left[i++]; + } else { + arr[k++] = right[j++]; + } + } + + // 复制剩余元素 + while (i < left.length) { + arr[k++] = left[i++]; + } + while (j < right.length) { + arr[k++] = right[j++]; + } + } + + public static void main(String[] args) { + int[] arr = {38, 27, 43, 3, 9, 82, 10}; + mergeSort(arr); + System.out.println("排序后的数组:"); + for (int num : arr) { + System.out.print(num + " "); + } + } +} + +``` + +![img](https://miro.medium.com/v2/resize:fit:1400/1*rx1sSHQEwVI0H9_6DP0S-Q.png) + + + +## 🌳 堆排序 + +> ![img](https://miro.medium.com/v2/resize:fit:1400/1*FNfDwQYa3wN-zcma1c1bfQ.png) + +堆排序（Heapsort）是指利用堆这种数据结构所设计的一种排序算法。堆积是一个近似完全二叉树的结构，并同时满足堆积的性质：即子结点的键值或索引总是小于（或者大于）它的父节点。堆排序可以说是一种利用堆的概念来排序的选择排序。 + +> 🎯 **算法特点**： +> - 基于堆数据结构 +> - 时间复杂度：$O(n\log n)$ +> - 空间复杂度：$O(1)$ +> - 不稳定排序 + +分为两种方法： + +1. 📈 **大顶堆**：每个节点的值都大于或等于其子节点的值，在堆排序算法中用于升序排列 +2. 📉 **小顶堆**：每个节点的值都小于或等于其子节点的值，在堆排序算法中用于降序排列 + +堆排序的平均时间复杂度为 $Ο(n\log n)$。 + +### 1. 算法步骤 + +1. 🏗️ **构建最大堆**： + - 首先将无序数组转换为一个**最大堆**。最大堆是一个完全二叉树，其中每个节点的值都大于或等于其子节点的值 + - 最大堆的根节点（堆顶）是整个堆中的最大元素 + +2. 🔄 **反复取出堆顶元素**： + - 将堆顶元素（最大值）与堆的最后一个元素交换，然后减少堆的大小，堆顶元素移到数组末尾 + - 调整剩余的元素使其重新成为一个最大堆 + - 重复这个过程，直到所有元素有序 + +### 2. 动图演示 + +![](https://img2018.cnblogs.com/blog/1258817/201904/1258817-20190420150936225-1441021270.gif) + +```java +public class HeapSort { + // 主排序函数 + public static void heapSort(int[] arr) { + int n = arr.length; + + // 1. 构建最大堆 + for (int i = n / 2 - 1; i >= 0; i--) { + heapify(arr, n, i); + } + + // 2. 逐步将堆顶元素与末尾元素交换，并缩小堆的范围 + for (int i = n - 1; i > 0; i--) { + // 将当前堆顶（最大值）移到末尾 + swap(arr, 0, i); + + // 重新调整堆，使剩余元素保持最大堆性质 + heapify(arr, i, 0); + } + } + + // 调整堆的函数 + private static void heapify(int[] arr, int n, int i) { + int largest = i; // 设当前节点为最大值 + int left = 2 * i + 1; // 左子节点 + int right = 2 * i + 2; // 右子节点 + + // 如果左子节点比当前最大值大 + if (left < n && arr[left] > arr[largest]) { + largest = left; + } + + // 如果右子节点比当前最大值大 + if (right < n && arr[right] > arr[largest]) { + largest = right; + } + + // 如果最大值不是根节点，则交换，并递归调整 + if (largest != i) { + swap(arr, i, largest); + heapify(arr, n, largest); + } + } + + // 交换两个元素的值 + private static void swap(int[] arr, int i, int j) { + int temp = arr[i]; + arr[i] = arr[j]; + arr[j] = temp; + } +} + +``` + + + +## 🔢 计数排序 + +**计数排序（Counting Sort）** 是一种基于计数的非比较排序算法，适用于对**非负整数**进行排序。计数排序通过统计每个元素出现的次数，然后利用这些信息将元素直接放到它们在有序数组中的位置，从而实现排序。 + +> 🎯 **算法特点**： +> - 非比较排序算法 +> - 时间复杂度：$O(n + k)$ +> - 空间复杂度：$O(k)$ +> - 稳定排序 +> - 适用于数据范围不大的场景 + +计数排序的时间复杂度为 `O(n + k)`，其中 `n` 是输入数据的大小，`k` 是数据范围的大小。它特别适用于数据范围不大但数据量较多的场景。 + +计数排序的核心在于将输入的数据值转化为键存储在额外开辟的数组空间中。作为一种线性时间复杂度的排序，计数排序要求输入的数据必须是有确定范围的整数。 + +### 1. 算法步骤 + +1. 🔍 **找到最大值和最小值**：找到数组中最大值和最小值，以确定计数数组的范围 +2. 🏗️ **创建计数数组**：创建一个计数数组，数组长度为 `max - min + 1`，用来记录每个元素出现的次数 +3. 📊 **统计每个元素的出现次数**：遍历原数组，将每个元素出现的次数记录在计数数组中 +4. 🔄 **累积计数**：将计数数组变为累积计数数组，使其表示元素在有序数组中的位置 +5. 📝 **回填到结果数组**：根据累积计数数组，回填到结果数组，得到最终的排序结果 + +### 2. 动图演示 + +[![动图演示](https://github.com/hustcc/JS-Sorting-Algorithm/raw/master/res/countingSort.gif)](https://github.com/hustcc/JS-Sorting-Algorithm/blob/master/res/countingSort.gif) + +```java +import java.util.Arrays; + +public class CountingSort { + // 计数排序函数 + public static void countingSort(int[] arr) { + if (arr.length == 0) return; + + // 1. 找到数组中的最大值和最小值 + int max = arr[0]; + int min = arr[0]; + for (int i = 1; i < arr.length; i++) { + if (arr[i] > max) { + max = arr[i]; + } else if (arr[i] < min) { + min = arr[i]; + } + } + + // 2. 创建计数数组 + int range = max - min + 1; + int[] count = new int[range]; + + // 3. 统计每个元素的出现次数 + for (int i = 0; i < arr.length; i++) { + count[arr[i] - min]++; + } + + // 4. 计算累积计数，确定元素的最终位置 + for (int i = 1; i < count.length; i++) { + count[i] += count[i - 1]; + } + + // 5. 创建结果数组，并根据累积计数将元素放到正确位置 + int[] output = new int[arr.length]; + for (int i = arr.length - 1; i >= 0; i--) { + output[count[arr[i] - min] - 1] = arr[i]; + count[arr[i] - min]--; + } + + // 6. 将排序好的结果复制回原数组 + for (int i = 0; i < arr.length; i++) { + arr[i] = output[i]; + } + } +} + +``` + + + +## 🪣 桶排序 + +桶排序是计数排序的升级版。它利用了函数的映射关系，高效与否的关键就在于这个映射函数的确定。 + +桶排序（Bucket Sort）是一种基于**分配**的排序算法，尤其适用于**均匀分布**的数据。它通过将元素分布到多个桶中，分别对每个桶进行排序，最后将各个桶中的元素合并起来，得到一个有序数组。 + +> 🎯 **算法特点**： +> - 基于分配的排序算法 +> - 时间复杂度：$O(n + k)$（平均） +> - 空间复杂度：$O(n + k)$ +> - 稳定排序 +> - 适用于均匀分布的数据 + +桶排序的平均时间复杂度为 `O(n + k)`，其中 `n` 是数据的数量，`k` 是桶的数量。桶排序通常适用于小范围的、均匀分布的浮点数或整数。 + +为了使桶排序更加高效，我们需要做到这两点： + +1. 📈 **增加桶的数量**：在额外空间充足的情况下，尽量增大桶的数量 +2. 🎯 **均匀分配**：使用的映射函数能够将输入的 N 个数据均匀的分配到 K 个桶中 + +同时，对于桶中元素的排序，选择何种比较排序算法对于性能的影响至关重要。 + +### 1. 算法步骤 + +1. 🏗️ **创建桶**：创建若干个桶，每个桶表示一个数值范围 +2. 📊 **将元素分配到桶中**：根据元素的大小，将它们分配到对应的桶中 +3. 🔄 **对每个桶内部排序**：对每个桶中的元素分别进行排序（可以使用插入排序、快速排序等） +4. 🔀 **合并所有桶中的元素**：依次将每个桶中的元素合并起来，得到最终的有序数组 + +![img](https://miro.medium.com/v2/resize:fit:1400/1*LwRT4hPsAKJ5iPrjGTwwMg.png) + +```java +import java.util.ArrayList; +import java.util.Collections; + +public class BucketSort { + // 主函数：进行桶排序 + public static void bucketSort(float[] arr, int n) { + if (n <= 0) return; + + // 1. 创建 n 个桶，每个桶是一个空的 ArrayList + ArrayList[] buckets = new ArrayList[n]; + + for (int i = 0; i < n; i++) { + buckets[i] = new ArrayList(); + } + + // 2. 将数组中的元素分配到各个桶中 + for (int i = 0; i < n; i++) { + int bucketIndex = (int) arr[i] * n; // 根据值分配桶 + buckets[bucketIndex].add(arr[i]); + } + + // 3. 对每个桶中的元素进行排序 + for (int i = 0; i < n; i++) { + Collections.sort(buckets[i]); // 可以使用任意内置排序算法 + } + + // 4. 合并所有桶中的元素，形成最终的排序数组 + int index = 0; + for (int i = 0; i < n; i++) { + for (Float value : buckets[i]) { + arr[index++] = value; + } + } + } + + // 测试主函数 + public static void main(String[] args) { + float[] arr = { (float)0.42, (float)0.32, (float)0.33, (float)0.52, (float)0.37, (float)0.47, (float)0.51 }; + int n = arr.length; + bucketSort(arr, n); + + System.out.println("排序后的数组："); + for (float value : arr) { + System.out.print(value + " "); + } + } +} + +``` + +**性能分析** + +- 🚀 **什么时候最快**：当输入的数据可以均匀的分配到每一个桶中 +- 🐌 **什么时候最慢**：当输入的数据被分配到了同一个桶中 + +> 💡 **学习提示**：思路一定要理解了，不背题哈，比如有些直接问你 +> +> 📝 **例题**：已知一组记录的排序码为（46，79，56，38，40，80, 95，24），写出对其进行快速排序的第一趟的划分结果 + + + +## 🔢 基数排序 + +**基数排序（Radix Sort）** 是一种**非比较排序算法**，用于对整数或字符串等进行排序。它的核心思想是将数据按位（如个位、十位、百位等）进行排序，从低位到高位依次进行。基数排序的时间复杂度为 `O(n * k)`，其中 `n` 是待排序元素的个数，`k` 是数字的最大位数或字符串的长度。由于它不涉及比较操作，基数排序适用于一些特定的场景，如排序长度相同的字符串或范围固定的整数。 + +> 🎯 **算法特点**： +> - 非比较排序算法 +> - 时间复杂度：$O(n \times k)$ +> - 空间复杂度：$O(n + k)$ +> - 稳定排序 +> - 适用于位数固定的数据 + +基数排序有两种实现方式： + +- 🔢 **LSD（Least Significant Digit）**：从最低位（个位）开始排序，常用的实现方式 +- 🔢 **MSD（Most Significant Digit）**：从最高位开始排序 + +基数排序是一种非比较型整数排序算法，其原理是将整数按位数切割成不同的数字，然后按每个位数分别比较。由于整数也可以表达字符串（比如名字或日期）和特定格式的浮点数，所以基数排序也不是只能使用于整数。 + +### 1. 算法步骤（LSD 实现） + +1. 🔍 **确定最大位数**：找出待排序数据中最大数的位数 +2. 🔄 **按位排序**：从最低位到最高位，对每一位进行排序。每次排序使用**稳定的排序算法**（如计数排序或桶排序），确保相同位数的元素相对位置不变 + +### 2. 动图演示 + +[![动图演示](https://github.com/hustcc/JS-Sorting-Algorithm/raw/master/res/radixSort.gif)](https://github.com/hustcc/JS-Sorting-Algorithm/blob/master/res/radixSort.gif) + +```java +import java.util.Arrays; + +public class RadixSort { + // 主函数：进行基数排序 + public static void radixSort(int[] arr) { + // 找到数组中的最大数，确定最大位数 + int max = getMax(arr); + + // 从个位开始，对每一位进行排序 + for (int exp = 1; max / exp > 0; exp *= 10) { + countingSortByDigit(arr, exp); + } + } + + // 找到数组中的最大值 + private static int getMax(int[] arr) { + int max = arr[0]; + for (int i = 1; i < arr.length; i++) { + if (arr[i] > max) { + max = arr[i]; + } + } + return max; + } + + // 根据当前位数进行计数排序 + private static void countingSortByDigit(int[] arr, int exp) { + int n = arr.length; + int[] output = new int[n]; // 输出数组 + int[] count = new int[10]; // 计数数组（0-9，用于处理每一位上的数字） + + // 1. 统计每个数字在当前位的出现次数 + for (int i = 0; i < n; i++) { + int digit = (arr[i] / exp) % 10; + count[digit]++; + } + + // 2. 计算累积计数 + for (int i = 1; i < 10; i++) { + count[i] += count[i - 1]; + } + + // 3. 从右到左遍历数组，按当前位将元素放入正确位置 + for (int i = n - 1; i >= 0; i--) { + int digit = (arr[i] / exp) % 10; + output[count[digit] - 1] = arr[i]; + count[digit]--; + } + + // 4. 将排序好的结果复制回原数组 + for (int i = 0; i < n; i++) { + arr[i] = output[i]; + } + } + +} + +``` + + + +## 🐚 希尔排序 + +希尔排序这个名字，来源于它的发明者希尔，也称作"缩小增量排序"，是插入排序的一种更高效的改进版本。但希尔排序是非稳定排序算法。 + +**希尔排序（Shell Sort）** 是一种**基于插入排序**的排序算法，又称为**缩小增量排序**，它通过将数组按一定间隔分成若干个子数组，分别进行插入排序，逐步缩小间隔，最终进行一次标准的插入排序。通过这种方式，希尔排序能够减少数据移动次数，使得整体排序过程更为高效。 + +> 🎯 **算法特点**： +> - 插入排序的改进版本 +> - 时间复杂度：$O(n^{1.3})$ 到 $O(n^2)$（取决于增量序列） +> - 空间复杂度：$O(1)$ +> - 不稳定排序 + +希尔排序的时间复杂度依赖于**增量序列**的选择，通常在 `O(n^1.3)` 到 `O(n^2)` 之间。 + +### 1. 算法步骤 + +1. 🎯 **确定增量序列**：首先确定一个增量序列 `gap`，通常初始的 `gap` 为数组长度的一半，然后逐步缩小 +2. 🔄 **分组排序**：将数组按 `gap` 分组，对每个分组进行插入排序。`gap` 表示当前元素与其分组中的前一个元素的间隔 +3. 📉 **缩小增量并继续排序**：每次将 `gap` 缩小一半，重复分组排序过程，直到 `gap = 1`，即对整个数组进行一次标准的插入排序 +4. ✅ **最终排序完成**：当 `gap` 变为 1 时，希尔排序相当于执行了一次插入排序，此时数组已经接近有序，因此插入排序的效率非常高 + +### 2. 动图演示 + +![img](https://mmbiz.qpic.cn/mmbiz_gif/951TjTgiabkzow2ORRzgpfHIGAKIAWlXm6GpRDRhiczgOdibbGBtpibtIhX4YRzibicUyEOSVh3JZBHtiaZPN30X1WOhA/640?wx_fmt=gif&tp=webp&wxfrom=5&wx_lazy=1) + +```java +import java.util.Arrays; + +public class ShellSort { + // 主函数：希尔排序 + public static void shellSort(int[] arr) { + int n = arr.length; + + // 1. 初始 gap 为数组长度的一半 + for (int gap = n / 2; gap > 0; gap /= 2) { + // 2. 对每个子数组进行插入排序 + for (int i = gap; i < n; i++) { + int temp = arr[i]; + int j = i; + + // 3. 对当前分组进行插入排序 + while (j >= gap && arr[j - gap] > temp) { + arr[j] = arr[j - gap]; + j -= gap; + } + + // 将 temp 插入到正确的位置 + arr[j] = temp; + } + } + } +} +``` + + + + + +## 📚 参考与感谢 + +- https://yuminlee2.medium.com/sorting-algorithms-summary-f17ea88a9174 + +--- + +> 🎉 **恭喜你完成了排序算法的学习！** 排序是计算机科学的基础，掌握了这些算法，你就拥有了解决各种排序问题的强大工具。记住：**选择合适的排序算法比掌握所有算法更重要**！ diff --git a/docs/data-structure-algorithms/complexity.md b/docs/data-structure-algorithms/complexity.md new file mode 100644 index 0000000000..c347f82d35 --- /dev/null +++ b/docs/data-structure-algorithms/complexity.md @@ -0,0 +1,1362 @@ +--- +title: 算法复杂度分析：开发者的必备技能 +date: 2025-05-09 +categories: Algorithm +--- + +> 高级工程师title的我，最近琢磨着好好刷刷算法题更高级一些，然鹅，当我准备回忆大学和面试时候学的数据结构之时，我发现自己对这个算法复杂度的记忆只有OOOOOooo +> +> 文章收录在 GitHub [JavaKeeper](https://github.com/Jstarfish/JavaKeeper) ，N线互联网开发必备技能兵器谱 + +## 前言 + +作为Java后端开发者，我们每天都在编写代码解决业务问题。但你是否思考过：**为什么有些系统在用户量上升时响应越来越慢？为什么某些查询在数据量大时会卡死？为什么同样功能的代码，有些占用内存巨大？** + +答案就在算法复杂度中。 + +算法（Algorithm）是指用来操作数据、解决程序问题的一组方法。对于同一个问题，使用不同的算法，也许最终得到的结果是一样的，但在过程中消耗的资源和时间却会有很大的区别。 + +在实际的Java开发中，复杂度分析帮助我们： +- **预测系统性能瓶颈**：在代码上线前就能预判性能问题 +- **优化关键路径**：识别出最需要优化的代码段 +- **合理选择数据结构**：HashMap vs TreeMap，ArrayList vs LinkedList +- **面试加分项**：大厂面试必考，体现算法功底 + +那么我们应该如何去衡量不同算法之间的优劣呢？ + +## 复杂度分析的两个维度 + +主要还是从算法所占用的「时间」和「空间」两个维度去考量： + +- **时间维度**：是指执行当前算法所消耗的时间，我们通常用「时间复杂度」来描述。 +- **空间维度**：是指执行当前算法需要占用多少内存空间，我们通常用「空间复杂度」来描述。 + +因此，评价一个算法的效率主要是看它的时间复杂度和空间复杂度情况。然而，有的时候时间和空间却又是「鱼和熊掌」，不可兼得的，那么我们就需要从中去取一个平衡点。 + +> 数据结构和算法本身解决的是"快"和"省"的问题，即如何让代码运行得更快，如何让代码更省存储空间。所以，执行效率是算法一个非常重要的考量指标。 + +## **时间复杂度** + +一个算法执行所耗费的时间，从理论上是不能算出来的，必须上机运行测试才能知道。而且测试环境中的硬件性能和测试数据规模都会对其有很大的影响。我们也不可能也没有必要对每个算法都上机测试，只需知道哪个算法花费的时间多，哪个算法花费的时间少就可以了。并且一个算法花费的时间与算法中语句的执行次数成正比例，哪个算法中语句执行次数多，它花费时间就多。一个算法中的语句执行次数称为语句频度或「**时间频度**」。记为T(n)。 + +时间频度T(n)中，n 称为问题的规模，当 n 不断变化时，时间频度 T(n) 也会不断变化。但有时我们想知道它变化时呈现什么规律，为此我们引入时间复杂度的概念。算法的时间复杂度也就是算法的时间度量，记作：$T(n) = O(f(n))$。它表示随问题规模 n 的增大，算法执行时间的增长率和 f(n) 的增长率相同，称作算法的**渐进时间复杂度**，简称「**时间复杂度**」。 + +这种表示方法我们称为「 **大O符号表示法** 」，又称为**渐进符号**，是用于描述函数渐进行为的数学符号 + +## 如何分析算法的时间复杂度 + +### 复杂度分析的核心思路 + +**核心原则**：关注当输入规模n趋向无穷大时，算法执行时间的增长趋势，而非具体的执行时间。 + +**分析步骤**： +1. **识别基本操作**：找出算法中最频繁执行的操作 +2. **计算执行次数**：分析这个操作随输入规模n的执行次数 +3. **忽略低阶项**：只保留增长最快的项 +4. **忽略系数**：去掉常数因子 + +### 复杂度分析的实战方法 + +#### 方法1：数循环层数 +这是最直观的分析方法： + +**单层循环 → O(n)** +```java +for (int i = 0; i < n; i++) { + // 基本操作 +} +// 分析过程：循环n次，每次执行基本操作1次，总共n次 → O(n) +``` + +**双层嵌套循环 → O(n²)** +```java +for (int i = 0; i < n; i++) { // 外层：n次 + for (int j = 0; j < n; j++) { // 内层：每轮n次 + // 基本操作 + } +} +// 分析过程：外层n次 × 内层n次 = n² 次 → O(n²) +``` + +**特殊情况：内层循环次数递减** + +```java +for (int i = 0; i < n; i++) { // 外层：n次 + for (int j = i; j < n; j++) { // 内层：第i轮执行(n-i)次 + // 基本操作 + } +} +// 分析过程：总次数 = n + (n-1) + (n-2) + ... + 1 = n(n+1)/2 ≈ n²/2 +// 忽略系数：O(n²) +``` + +#### 方法2：分析递归调用 +递归算法的复杂度 = 递归调用总次数 × 每次调用的复杂度 + +**递归分析实例：计算斐波那契数列** +```java +int fibonacci(int n) { + if (n <= 1) return n; // 基本情况：O(1) + return fibonacci(n-1) + fibonacci(n-2); // 递归调用：2次 +} +``` + +**分析过程：** +1. **画出递归树**： + ``` + fib(n) + ├── fib(n-1) + │ ├── fib(n-2) + │ └── fib(n-3) + └── fib(n-2) + ├── fib(n-3) + └── fib(n-4) + ``` + +2. **分析递归深度**：最深到fib(0)，深度约为n + +3. **分析每层节点数**：每层节点数翻倍，第k层有2^k个节点 + +4. **计算总节点数**：2^0 + 2^1 + ... + 2^n ≈ 2^n + +5. **得出复杂度**：O(2^n) + +**为什么这么慢？** 因为有大量重复计算！fib(n-2)既在fib(n-1)中计算，又在fib(n)中计算。 + +#### 方法3：分析分治算法 +分治算法的复杂度可以用递推关系式分析。 + +**二分查找分析** +```java +int binarySearch(int[] arr, int target, int left, int right) { + if (left > right) return -1; // 基本情况 + + int mid = (left + right) / 2; // O(1)操作 + if (arr[mid] == target) return mid; // O(1)操作 + else if (arr[mid] < target) + return binarySearch(arr, target, mid + 1, right); // 递归一半 + else + return binarySearch(arr, target, left, mid - 1); // 递归一半 +} +``` + +**分析过程：** +1. **建立递推关系**：T(n) = T(n/2) + O(1) + - 每次递归处理一半的数据 + - 除了递归外，其他操作都是O(1) + +2. **展开递推式**： + - T(n) = T(n/2) + 1 + - T(n/2) = T(n/4) + 1 + - T(n/4) = T(n/8) + 1 + - ... + - T(1) = 1 + +3. **求解**：总共需要log₂n层递归，每层O(1)，所以T(n) = O(logn) + +### 常见复杂度等级及其识别 + +| 复杂度 | 识别特征 | 典型例子 | 数据规模感受 | +|--------|----------|----------|-------------| +| **O(1)** | 不随n变化 | 数组索引访问、HashMap查找 | 任何规模都很快 | +| **O(logn)** | 每次减半 | 二分查找、平衡树操作 | 100万数据只需20步 | +| **O(n)** | 单层循环 | 数组遍历、链表查找 | n=1000万时还可接受 | +| **O(nlogn)** | 分治+合并 | 归并排序、快速排序 | n=100万时性能良好 | +| **O(n²)** | 双层循环 | 冒泡排序、暴力匹配 | n>1000就开始慢了 | +| **O(2^n)** | 无优化递归 | 递归斐波那契 | n>30就要等很久 | + +## 复杂度分析实战练习 + +学会了分析方法，让我们通过几个经典例子来实际练习，重点关注**分析过程**而不是记忆结果。 + +### 练习1：分析冒泡排序的复杂度 + +```java +public void bubbleSort(int[] arr) { + int n = arr.length; + for (int i = 0; i < n - 1; i++) { + for (int j = 0; j < n - i - 1; j++) { + if (arr[j] > arr[j + 1]) { + swap(arr, j, j + 1); + } + } + } +} +``` + +**分析过程：** +1. **识别基本操作**：比较和交换操作 `arr[j] > arr[j + 1]` +2. **分析循环次数**： + - 外层循环：i从0到n-2，共(n-1)次 + - 内层循环：第i轮时，j从0到(n-i-2)，共(n-i-1)次 +3. **计算总次数**： + - 第0轮：n-1次比较 + - 第1轮：n-2次比较 + - 第2轮：n-3次比较 + - ... + - 第(n-2)轮：1次比较 + - 总计：(n-1) + (n-2) + ... + 1 = n(n-1)/2 次 +4. **简化结果**：n(n-1)/2 ≈ n²/2，忽略系数得到 **O(n²)** + +### 练习2：分析HashMap查找的复杂度 + +```java +// HashMap的get操作 +public V get(Object key) { + int hash = hash(key); // 计算hash值，O(1) + int index = hash % table.length; // 计算索引，O(1) + Node node = table[index]; // 直接访问数组，O(1) + + // 在链表/红黑树中查找 + while (node != null) { + if (node.key.equals(key)) { + return node.value; + } + node = node.next; // 遍历链表 + } + return null; +} +``` + +**分析过程：** +1. **理想情况**：没有hash冲突，直接命中 → **O(1)** +2. **最坏情况**：所有元素都hash到同一个位置，形成长度为n的链表 → **O(n)** +3. **平均情况**：hash函数分布均匀，每个链表长度约为1 → **O(1)** + +**为什么说HashMap是O(1)？** 指的是平均情况下的复杂度。 + +### 练习3：分析递归求阶乘的复杂度 + +```java +public int factorial(int n) { + if (n <= 1) return 1; // 基本情况 + return n * factorial(n - 1); // 递归调用 +} +``` + +**分析过程：** +1. **递归深度**：从n递减到1，深度为n +2. **每层工作量**：除了递归调用，只有一次乘法运算，O(1) +3. **总复杂度**：n层 × 每层O(1) = **O(n)** + +**递归调用栈：** +``` +factorial(5) +├── 5 * factorial(4) + ├── 4 * factorial(3) + ├── 3 * factorial(2) + ├── 2 * factorial(1) + └── 1 +``` + +### 练习4：分析快速排序的复杂度 + +```java +public void quickSort(int[] arr, int low, int high) { + if (low < high) { + int pivot = partition(arr, low, high); // 分区，O(n) + quickSort(arr, low, pivot - 1); // 递归左半部分 + quickSort(arr, pivot + 1, high); // 递归右半部分 + } +} +``` + +**分析过程：** +1. **最好情况**：每次都能均匀分割 + - 递归深度：log₂n + - 每层工作量：O(n) + - 总复杂度：O(nlogn) + +2. **最坏情况**：每次都是最不均匀的分割（已排序数组） + - 递归深度：n + - 每层工作量：O(n) + - 总复杂度：O(n²) + +3. **平均情况**：O(nlogn) + +**关键洞察**：快速排序的性能很大程度上取决于pivot的选择。 + +### 复杂度分析的误区和技巧 + +#### 常见误区 + +1. **误区1：认为递归一定比循环慢** + ```java + // 递归版本：O(logn) + int binarySearchRecursive(int[] arr, int target, int left, int right) { + if (left > right) return -1; + int mid = (left + right) / 2; + if (arr[mid] == target) return mid; + else if (arr[mid] < target) + return binarySearchRecursive(arr, target, mid + 1, right); + else + return binarySearchRecursive(arr, target, left, mid - 1); + } + + // 循环版本：同样O(logn) + int binarySearchIterative(int[] arr, int target) { + int left = 0, right = arr.length - 1; + while (left <= right) { + int mid = (left + right) / 2; + if (arr[mid] == target) return mid; + else if (arr[mid] < target) left = mid + 1; + else right = mid - 1; + } + return -1; + } + ``` + +2. **误区2：认为代码行数多就复杂度高** + ```java + // 虽然代码很长，但复杂度是O(1) + public boolean isValidSudoku(char[][] board) { + for (int i = 0; i < 9; i++) { // 固定9次 + for (int j = 0; j < 9; j++) { // 固定9次 + // 检查逻辑... + } + } + return true; // 9×9=81次操作，是常数，所以O(1) + } + ``` + +#### 实用技巧 + +1. **看循环变量的变化规律** + ```java + // 线性增长 → O(n) + for (int i = 0; i < n; i++) + + // 指数增长 → O(logn) + for (int i = 1; i < n; i *= 2) + + // 嵌套循环 → O(n²) + for (int i = 0; i < n; i++) + for (int j = 0; j < n; j++) + ``` + +2. **分析递归的关键问题** + - 递归了多少次？ + - 每次递归处理多大的子问题？ + - 除了递归还做了什么？ + +3. **利用已知的复杂度** + - 排序通常是O(nlogn) + - 哈希表查找通常是O(1) + - 数组遍历是O(n) + - 二分查找是O(logn) + +除此之外，其实还有平均情况复杂度、最好时间复杂度、最坏时间复杂度。。。一般没有特殊说明的情况下，都是指最坏时间复杂度。 + +------ + +## **空间复杂度分析方法** + +空间复杂度分析相对简单，主要关注算法运行过程中需要的**额外存储空间**。 + +### 空间复杂度的组成 +1. **算法本身**：代码指令占用的空间（通常忽略） +2. **输入数据**：输入参数占用的空间（通常忽略） +3. **额外空间**：算法运行时临时申请的空间（主要分析对象） + +### 空间复杂度分析步骤 + +#### 第一步：识别额外空间的来源 +- **局部变量**：函数内声明的变量 +- **数据结构**：数组、链表、栈、队列等 +- **递归调用栈**：递归函数的调用栈 + +#### 第二步：分析空间随输入规模的变化 + +**O(1) 常量空间** +```java +public void swap(int[] arr, int i, int j) { + int temp = arr[i]; // 只使用了一个额外变量 + arr[i] = arr[j]; + arr[j] = temp; +} +// 无论数组多大，只用了temp一个额外空间 → O(1) +``` + +**O(n) 线性空间** +```java +public int[] copyArray(int[] arr) { + int[] newArr = new int[arr.length]; // 创建了n大小的新数组 + for (int i = 0; i < arr.length; i++) { + newArr[i] = arr[i]; + } + return newArr; +} +// 创建了大小为n的数组 → O(n) +``` + +**递归空间复杂度分析** +```java +public int factorial(int n) { + if (n <= 1) return 1; + return n * factorial(n - 1); +} +``` + +**分析过程：** +1. **递归深度**：从n到1，总共n层 +2. **每层空间**：每层只有局部变量n，占用O(1)空间 +3. **总空间**：n层 × O(1) = O(n) + +**递归调用栈示意：** +``` +factorial(5) ← 需要保存参数5 +├── factorial(4) ← 需要保存参数4 + ├── factorial(3) ← 需要保存参数3 + ├── factorial(2) ← 需要保存参数2 + └── factorial(1) ← 需要保存参数1 +``` + +### 空间复杂度优化技巧 + +#### 技巧1：滚动数组 +```java +// ❌ 原始动态规划：O(n)空间 +public int climbStairs(int n) { + int[] dp = new int[n + 1]; // 需要n+1大小的数组 + dp[0] = 1; + dp[1] = 1; + for (int i = 2; i <= n; i++) { + dp[i] = dp[i-1] + dp[i-2]; + } + return dp[n]; +} + +// ✅ 空间优化：O(1)空间 +public int climbStairs(int n) { + if (n <= 1) return 1; + int prev1 = 1, prev2 = 1; // 只需要两个变量 + for (int i = 2; i <= n; i++) { + int current = prev1 + prev2; + prev1 = prev2; + prev2 = current; + } + return prev2; +} +``` + +#### 技巧2：就地修改 +```java +// ❌ 需要额外数组：O(n)空间 +public int[] reverseArray(int[] arr) { + int[] newArr = new int[arr.length]; + for (int i = 0; i < arr.length; i++) { + newArr[i] = arr[arr.length - 1 - i]; + } + return newArr; +} + +// ✅ 就地修改：O(1)空间 +public void reverseArrayInPlace(int[] arr) { + int left = 0, right = arr.length - 1; + while (left < right) { + int temp = arr[left]; + arr[left] = arr[right]; + arr[right] = temp; + left++; + right--; + } +} +``` + +#### 技巧3：递归转迭代 +```java +// ❌ 递归版本：O(n)空间（调用栈） +public boolean hasPath(TreeNode root, int sum) { + if (root == null) return sum == 0; + return hasPath(root.left, sum - root.val) || + hasPath(root.right, sum - root.val); +} + +// ✅ 迭代版本：O(h)空间（h为树高） +public boolean hasPathIterative(TreeNode root, int sum) { + if (root == null) return false; + Stack nodeStack = new Stack<>(); + Stack sumStack = new Stack<>(); + + nodeStack.push(root); + sumStack.push(sum - root.val); + + while (!nodeStack.isEmpty()) { + TreeNode node = nodeStack.pop(); + int currSum = sumStack.pop(); + + if (node.left == null && node.right == null && currSum == 0) { + return true; + } + + if (node.left != null) { + nodeStack.push(node.left); + sumStack.push(currSum - node.left.val); + } + if (node.right != null) { + nodeStack.push(node.right); + sumStack.push(currSum - node.right.val); + } + } + return false; +} +``` + +------ + +## 复杂度速查表 + +来源：https://liam.page/2016/06/20/big-O-cheat-sheet/ 源地址：https://www.bigocheatsheet.com/ + +------ + +#### 1. 线性递归 - O(n)空间 + +**LeetCode应用：链表递归** + +```java +// 反转链表（递归版本） +public ListNode reverseList(ListNode head) { + if (head == null || head.next == null) { + return head; + } + + ListNode newHead = reverseList(head.next); // 递归调用 + head.next.next = head; + head.next = null; + + return newHead; +} +// 递归深度等于链表长度n，空间复杂度O(n) + +// 计算链表长度（递归） +public int getLength(ListNode head) { + if (head == null) return 0; + return 1 + getLength(head.next); +} +// 空间复杂度：O(n)，因为递归栈深度为n +``` + +#### 2. 二分递归 - O(logn)空间 + +**示例：二分查找（递归版本）** + +```java +public int binarySearch(int[] nums, int target, int left, int right) { + if (left > right) return -1; + + int mid = left + (right - left) / 2; + if (nums[mid] == target) { + return mid; + } else if (nums[mid] < target) { + return binarySearch(nums, target, mid + 1, right); + } else { + return binarySearch(nums, target, left, mid - 1); + } +} +// 每次递归都将问题规模减半，递归深度为log₂n，空间复杂度O(logn) +``` + +**LeetCode应用：树的遍历** +```java +// 二叉树的最大深度 +public int maxDepth(TreeNode root) { + if (root == null) return 0; + + int leftDepth = maxDepth(root.left); // 递归左子树 + int rightDepth = maxDepth(root.right); // 递归右子树 + + return Math.max(leftDepth, rightDepth) + 1; +} +// 平衡树：递归深度为O(logn)，空间复杂度O(logn) +// 最坏情况（链状树）：递归深度为O(n)，空间复杂度O(n) +``` + +#### 3. 多分支递归 - 注意陷阱！ + +**错误理解：认为空间复杂度是O(2^n)** +```java +// 斐波那契数列（递归版本） +public int fibonacci(int n) { + if (n <= 1) return n; + return fibonacci(n - 1) + fibonacci(n - 2); +} +``` + +**正确分析：** +- **时间复杂度**：O(2^n) - 因为有2^n个函数调用 +- **空间复杂度**：O(n) - 因为递归栈的最大深度是n + +> 关键理解：虽然有很多递归调用，但任何时刻调用栈的深度最多是n层 + +#### 4. 记忆化递归的空间复杂度 + +```java +// 带记忆化的斐波那契 +private Map memo = new HashMap<>(); + +public int fibonacciMemo(int n) { + if (n <= 1) return n; + + if (memo.containsKey(n)) { + return memo.get(n); + } + + int result = fibonacciMemo(n - 1) + fibonacciMemo(n - 2); + memo.put(n, result); + return result; +} +// 时间复杂度：O(n) +// 空间复杂度：O(n) = 递归栈O(n) + 缓存数组O(n) +``` + +### LeetCode中的空间复杂度优化技巧 + +#### 1. 滚动数组优化 +```java +// 原始DP：O(n)空间 +int[] dp = new int[n]; + +// 优化后：O(1)空间 +int prev1 = dp[0], prev2 = dp[1]; +``` + +#### 2. 就地修改避免额外空间 +```java +// 数组去重（就地修改） +public int removeDuplicates(int[] nums) { + if (nums.length == 0) return 0; + + int i = 0; + for (int j = 1; j < nums.length; j++) { + if (nums[j] != nums[i]) { + i++; + nums[i] = nums[j]; // 就地修改，不需要额外空间 + } + } + return i + 1; +} // 空间复杂度：O(1) +``` + +#### 3. 递归转迭代 +```java +// 递归版本：O(n)空间 +public void inorderRecursive(TreeNode root) { + if (root == null) return; + inorderRecursive(root.left); + System.out.println(root.val); + inorderRecursive(root.right); +} + +// 迭代版本：O(h)空间，h为树的高度 +public void inorderIterative(TreeNode root) { + Stack stack = new Stack<>(); + TreeNode current = root; + + while (current != null || !stack.isEmpty()) { + while (current != null) { + stack.push(current); + current = current.left; + } + current = stack.pop(); + System.out.println(current.val); + current = current.right; + } +} +``` + +### 常见面试问题：时间空间复杂度权衡 + +| 算法 | 时间复杂度 | 空间复杂度 | 权衡点 | +|------|------------|------------|--------| +| 快速排序 | O(nlogn) | O(logn) | 递归栈空间 | +| 归并排序 | O(nlogn) | O(n) | 需要额外数组 | +| 堆排序 | O(nlogn) | O(1) | 就地排序 | +| 计数排序 | O(n+k) | O(k) | 需要额外计数数组 | +| 哈希表查找 | O(1) | O(n) | 用空间换时间 | +| 二分查找 | O(logn) | O(1) | 要求数组有序 | + +------ + +## 复杂度分析在实际开发中的应用 + +### 系统设计中的复杂度考量 + +#### 1. 数据库查询优化 +```java +// ❌ 没有索引的查询 - O(n) +SELECT * FROM users WHERE email = 'user@example.com'; + +// ✅ 有索引的查询 - O(logn) +CREATE INDEX idx_email ON users(email); +SELECT * FROM users WHERE email = 'user@example.com'; +``` + +**在Java中的体现：** +```java +// JPA查询优化 +@Entity +public class User { + @Column(name = "email") + @Index(name = "idx_email") // 添加索引 + private String email; +} + +// 避免N+1查询问题 +@Query("SELECT u FROM User u JOIN FETCH u.orders WHERE u.id = :userId") +User findUserWithOrders(@Param("userId") Long userId); +``` + +#### 2. 缓存策略选择 +```java +// Redis缓存 vs 数据库查询的复杂度对比 +public class UserService { + + // ❌ 每次都查数据库 - O(logn)到O(n) + public User getUserById(Long id) { + return userRepository.findById(id); + } + + // ✅ 使用缓存 - O(1) + @Cacheable(value = "users", key = "#id") + public User getUserByIdCached(Long id) { + return userRepository.findById(id); + } +} +``` + +#### 3. 分页查询的复杂度陷阱 +```java +// ❌ 深度分页性能问题 +// LIMIT 1000000, 20 在MySQL中是O(n)，需要跳过100万条记录 +public Page getUsers(int page, int size) { + return userRepository.findAll(PageRequest.of(page, size)); +} + +// ✅ 游标分页优化为O(logn) +public List getUsersAfterCursor(Long lastUserId, int limit) { + return userRepository.findByIdGreaterThanOrderById(lastUserId, + PageRequest.of(0, limit)); +} +``` + +### 微服务架构中的复杂度问题 + +#### 1. 接口聚合的复杂度 +```java +// ❌ 串行调用多个服务 - O(n) +@RestController +public class OrderController { + + public OrderDetailVO getOrderDetail(Long orderId) { + Order order = orderService.getOrder(orderId); // 100ms + User user = userService.getUser(order.getUserId()); // 100ms + Product product = productService.getProduct(order.getProductId()); // 100ms + // 总耗时：300ms + + return buildOrderDetail(order, user, product); + } +} + +// ✅ 并行调用优化 - O(1) +@Async +public CompletableFuture getOrderDetailAsync(Long orderId) { + CompletableFuture orderFuture = + CompletableFuture.supplyAsync(() -> orderService.getOrder(orderId)); + + CompletableFuture userFuture = orderFuture.thenCompose(order -> + CompletableFuture.supplyAsync(() -> userService.getUser(order.getUserId()))); + + CompletableFuture productFuture = orderFuture.thenCompose(order -> + CompletableFuture.supplyAsync(() -> productService.getProduct(order.getProductId()))); + + return CompletableFuture.allOf(orderFuture, userFuture, productFuture) + .thenApply(v -> buildOrderDetail(orderFuture.join(), userFuture.join(), productFuture.join())); + // 总耗时：约100ms +} +``` + +#### 2. 批量处理优化 +```java +// ❌ 循环调用接口 - O(n) +public void processUsers(List userIds) { + for (Long userId : userIds) { + User user = userService.getUser(userId); // 每次一个网络调用 + processUser(user); + } + // 1000个用户 = 1000次网络调用 +} + +// ✅ 批量调用 - O(1) +public void processUsersBatch(List userIds) { + List users = userService.getUsers(userIds); // 一次网络调用 + users.forEach(this::processUser); + // 1000个用户 = 1次网络调用 +} +``` + +### 大数据处理场景 + +#### 1. 日志分析系统 +```java +// 实时日志处理的复杂度考量 +@Component +public class LogProcessor { + + // ❌ 实时逐条处理 - 无法处理高并发 + public void processLogRealtimeSync(LogEvent event) { + // 同步处理每条日志 + analyzeLog(event); // 假设需要10ms + saveToDatabase(event); // 假设需要5ms + // 每秒最多处理：1000/(10+5) = 66条 + } + + // ✅ 批量异步处理 - 提升吞吐量 + @EventListener + @Async + public void processLogBatch(List events) { + // 批量处理，均摊网络开销 + List results = batchAnalyze(events); // 批量分析 + batchSaveToDatabase(results); // 批量保存 + // 每秒可处理：数万条 + } +} +``` + +#### 2. 报表查询优化 +```java +// 大数据量报表查询的复杂度优化 +@Service +public class ReportService { + + // ❌ 实时聚合计算 - O(n)，数据量大时很慢 + public SalesReport getDailySalesReport(LocalDate date) { + List orders = orderRepository.findByCreateDate(date); + // 需要遍历所有订单进行聚合计算 + return calculateSalesReport(orders); + } + + // ✅ 预计算报表 - O(1)查询 + @Scheduled(cron = "0 0 1 * * ?") // 每天凌晨1点执行 + public void generateDailyReports() { + LocalDate yesterday = LocalDate.now().minusDays(1); + SalesReport report = calculateSalesReport( + orderRepository.findByCreateDate(yesterday)); + reportRepository.save(report); // 预计算结果 + } + + public SalesReport getDailySalesReportCached(LocalDate date) { + return reportRepository.findByDate(date); // O(1)查询 + } +} +``` + +### Spring Boot性能优化实战 + +#### 1. 启动时间优化 +```java +// 启动时间复杂度优化 +@SpringBootApplication +public class Application { + + // ❌ 启动时加载大量数据 + @PostConstruct + public void initData() { + // 启动时从数据库加载10万条配置 - 增加启动时间 + configService.loadAllConfigs(); // O(n) + } + + // ✅ 懒加载优化 + @Bean + @Lazy + public ConfigCache configCache() { + return new ConfigCache(); // 需要时才初始化 + } +} +``` + +#### 2. 内存使用优化 +```java +// 内存复杂度优化 +@RestController +public class DataController { + + // ❌ 一次性加载所有数据到内存 - O(n)空间 + public List getAllData() { + List allData = dataRepository.findAll(); // 可能有百万条数据 + return allData.stream() + .map(this::convertToVO) + .collect(Collectors.toList()); // 内存可能溢出 + } + + // ✅ 流式处理 - O(1)空间 + public void exportAllData(HttpServletResponse response) { + try (Stream dataStream = dataRepository.findAllStream()) { + dataStream.map(this::convertToVO) + .forEach(vo -> writeToResponse(response, vo)); // 逐条处理 + } + } +} +``` + +### 消息队列的复杂度考量 + +```java +// 消息处理的复杂度优化 +@Component +public class MessageProcessor { + + // ❌ 顺序处理消息 - O(n)时间 + @RabbitListener(queues = "order.queue") + public void processOrderSync(OrderMessage message) { + // 顺序处理，一个消息处理完才处理下一个 + validateOrder(message); // 50ms + saveOrder(message); // 100ms + sendNotification(message); // 30ms + // 总计180ms/消息，吞吐量低 + } + + // ✅ 并行处理消息 - 提升吞吐量 + @RabbitListener(queues = "order.queue", concurrency = "10") + public void processOrderAsync(OrderMessage message) { + CompletableFuture.allOf( + CompletableFuture.runAsync(() -> validateOrder(message)), + CompletableFuture.runAsync(() -> saveOrder(message)), + CompletableFuture.runAsync(() -> sendNotification(message)) + ).join(); + // 并行处理，吞吐量大幅提升 + } +} +``` + +### 实际项目中的复杂度思考框架 + +#### 1. 性能需求评估 +```java +// 根据业务规模选择合适的算法复杂度 +public class BusinessScaleAnalysis { + + // 小规模系统（<1万用户） + public void smallScale() { + // O(n²)算法可以接受 + // 简单直接的实现优先 + } + + // 中等规模系统（1万-100万用户） + public void mediumScale() { + // 需要O(nlogn)或更优算法 + // 开始考虑缓存、索引优化 + } + + // 大规模系统（>100万用户） + public void largeScale() { + // 必须O(logn)或O(1)算法 + // 分布式缓存、数据库分片等 + } +} +``` + +#### 2. 技术选型的复杂度考量 +```java +// 不同技术栈的复杂度特性 +public class TechStackComplexity { + + // HashMap vs TreeMap选择 + Map userCache; + + // 如果只需要快速查找：HashMap O(1) + userCache = new HashMap<>(); + + // 如果需要有序遍历：TreeMap O(logn) + userCache = new TreeMap<>(); + + // ArrayList vs LinkedList选择 + List users; + + // 频繁随机访问：ArrayList O(1) + users = new ArrayList<>(); + + // 频繁插入删除：LinkedList O(1) + users = new LinkedList<>(); +} +``` + +## 复杂度优化的最佳实践 + +### LeetCode刷题的复杂度优化套路 + +#### 1. 暴力解法 → 哈希表优化（降时间复杂度） +```java +// 模式：O(n²) → O(n) +// 适用场景：查找、匹配类问题 + +// 示例：两数之和类问题 +// 暴力：双层循环查找 O(n²) +// 优化：哈希表存储 O(n) +Map map = new HashMap<>(); +``` + +#### 2. 递归 → 动态规划（消除重复计算） +```java +// 模式：O(2^n) → O(n) +// 适用场景：有重叠子问题的递归 + +// 递归优化三步走： +// 1. 记忆化递归（自顶向下） +Map memo = new HashMap<>(); + +// 2. 动态规划（自底向上） +int[] dp = new int[n+1]; + +// 3. 空间优化（滚动变量） +int prev1 = 0, prev2 = 1; +``` + +#### 3. 排序 + 双指针（降低循环层数） +```java +// 模式：O(n²) → O(nlogn) +// 适用场景：需要查找配对的问题 + +Arrays.sort(nums); // O(nlogn) +int left = 0, right = nums.length - 1; // O(n) +// 总体：O(nlogn)，比O(n²)好 +``` + +#### 4. 二分查找（利用有序性） +```java +// 模式：O(n) → O(logn) +// 适用场景：在有序数组中查找 + +// 关键：寻找单调性 +while (left <= right) { + int mid = left + (right - left) / 2; + // 根据mid位置的性质决定搜索方向 +} +``` + +### Java集合类的复杂度选择指南 + +#### 1. List选择策略 +```java +// 根据操作频率选择 +public class ListChoiceGuide { + + // 频繁随机访问 + 较少插入删除 → ArrayList + List data = new ArrayList<>(); // get: O(1), add: O(1) + + // 频繁插入删除 + 较少随机访问 → LinkedList + List data = new LinkedList<>(); // add/remove: O(1), get: O(n) + + // 需要线程安全 → Vector 或 Collections.synchronizedList + List data = new Vector<>(); + + // 不可变列表 → Arrays.asList 或 List.of + List data = List.of("a", "b", "c"); +} +``` + +#### 2. Map选择策略 +```java +public class MapChoiceGuide { + + // 一般情况，追求最快查找 → HashMap + Map cache = new HashMap<>(); // O(1) + + // 需要有序遍历 → TreeMap + Map sortedCache = new TreeMap<>(); // O(logn) + + // 需要插入顺序 → LinkedHashMap + Map orderedCache = new LinkedHashMap<>(); // O(1) + 顺序 + + // 高并发环境 → ConcurrentHashMap + Map concurrentCache = new ConcurrentHashMap<>(); +} +``` + +#### 3. Set选择策略 +```java +public class SetChoiceGuide { + + // 一般去重需求 → HashSet + Set uniqueItems = new HashSet<>(); // O(1) + + // 需要有序 → TreeSet + Set sortedItems = new TreeSet<>(); // O(logn) + + // 需要保持插入顺序 → LinkedHashSet + Set orderedItems = new LinkedHashSet<>(); // O(1) + 顺序 +} +``` + +### 常见性能陷阱及避免方法 + +#### 1. 字符串操作陷阱 +```java +// ❌ String拼接陷阱 - O(n²) +String result = ""; +for (String str : list) { + result += str; // 每次都创建新字符串 +} + +// ✅ 使用StringBuilder - O(n) +StringBuilder sb = new StringBuilder(); +for (String str : list) { + sb.append(str); +} +String result = sb.toString(); + +// ✅ Java 8 Stream方式 - O(n) +String result = list.stream().collect(Collectors.joining()); +``` + +#### 2. 集合遍历陷阱 +```java +// ❌ 在遍历中修改集合 - 可能O(n²) +for (int i = 0; i < list.size(); i++) { + if (shouldRemove(list.get(i))) { + list.remove(i); // ArrayList.remove()是O(n)操作 + i--; // 还要调整索引 + } +} + +// ✅ 使用Iterator - O(n) +Iterator iterator = list.iterator(); +while (iterator.hasNext()) { + if (shouldRemove(iterator.next())) { + iterator.remove(); // O(1)操作 + } +} + +// ✅ 使用removeIf - O(n) +list.removeIf(this::shouldRemove); +``` + +#### 3. 数据库查询陷阱 +```java +// ❌ N+1查询问题 +public List getOrdersWithUser(List orderIds) { + List orders = orderRepository.findByIds(orderIds); // 1次查询 + return orders.stream().map(order -> { + User user = userRepository.findById(order.getUserId()); // N次查询 + return new OrderVO(order, user); + }).collect(Collectors.toList()); +} + +// ✅ 批量查询优化 +public List getOrdersWithUserOptimized(List orderIds) { + List orders = orderRepository.findByIds(orderIds); + Set userIds = orders.stream().map(Order::getUserId).collect(Collectors.toSet()); + Map userMap = userRepository.findByIds(userIds) + .stream().collect(Collectors.toMap(User::getId, user -> user)); + + return orders.stream().map(order -> { + User user = userMap.get(order.getUserId()); // O(1)查找 + return new OrderVO(order, user); + }).collect(Collectors.toList()); +} +``` + +### 复杂度分析的实用技巧 + +#### 1. 快速估算方法 +```java +// 根据数据规模快速判断可接受的复杂度 +public class ComplexityEstimation { + + public void analyzeDataScale(int n) { + if (n <= 10) { + // 任何算法都可以，包括O(n!) + System.out.println("可以使用任何算法"); + } else if (n <= 20) { + // O(2^n)可以接受，如回溯算法 + System.out.println("指数级算法可接受"); + } else if (n <= 100) { + // O(n³)勉强可以 + System.out.println("三次方算法勉强可以"); + } else if (n <= 1000) { + // O(n²)是上限 + System.out.println("平方级算法是上限"); + } else if (n <= 100000) { + // 必须O(nlogn)或更优 + System.out.println("必须线性对数级或更优"); + } else { + // 必须O(n)或O(logn) + System.out.println("必须线性级或对数级"); + } + } +} +``` + +#### 2. 复杂度分析检查清单 +```java +// 代码复杂度自检清单 +public class ComplexityCheckList { + + public void checkComplexity() { + // 1. 有几层嵌套循环？ + // - 1层 → O(n) + // - 2层 → O(n²) + // - k层 → O(n^k) + + // 2. 有递归调用吗？ + // - 递归深度 × 每层复杂度 = 总复杂度 + + // 3. 有二分查找吗？ + // - 每次减半 → O(logn) + + // 4. 有排序操作吗？ + // - 通用排序 → O(nlogn) + // - 特殊排序 → 可能O(n) + + // 5. 使用了什么数据结构？ + // - HashMap → O(1) + // - TreeMap → O(logn) + // - LinkedList查找 → O(n) + } +} +``` + +#### 3. 空间复杂度优化技巧 +```java +// 常见空间优化模式 +public class SpaceOptimization { + + // 1. 滚动数组优化DP + public int optimizedDP(int n) { + // 原始：int[] dp = new int[n]; // O(n)空间 + // 优化：只保存必要的状态 + int prev = 0, curr = 1; // O(1)空间 + return curr; + } + + // 2. 就地修改避免额外空间 + public void reverseArrayInPlace(int[] arr) { + int left = 0, right = arr.length - 1; + while (left < right) { + // 就地交换，不需要额外数组 + int temp = arr[left]; + arr[left] = arr[right]; + arr[right] = temp; + left++; + right--; + } + } + + // 3. 流式处理大数据 + public void processLargeData() { + // 避免一次性加载所有数据到内存 + try (Stream lines = Files.lines(Paths.get("large-file.txt"))) { + lines.filter(line -> line.contains("target")) + .forEach(this::processLine); // 逐行处理 + } + } +} +``` + +## 复杂度速查表 + +![](https://tva1.sinaimg.cn/large/00831rSTly1gcbecysa6qj319w01w0sy.jpg) + +#### 大-O 复杂度曲线 + +![](https://tva1.sinaimg.cn/large/00831rSTly1gcbed227xkj317s0qy77v.jpg) + +#### 抽象数据结构的操作复杂度 + +![](https://tva1.sinaimg.cn/large/00831rSTly1gcbed4ehcnj30xz0u0n4m.jpg) + + + +#### 数组排序 + +![](https://tva1.sinaimg.cn/large/00831rSTly1gcbeda0vscj316f0u0jx3.jpg) + +#### 图操作 + +![](https://tva1.sinaimg.cn/large/00831rSTly1gcbede8tjnj31ac08gwg1.jpg) + +#### 堆操作 + +![](https://tva1.sinaimg.cn/large/00831rSTly1gcbedhhqq0j31a40j8n0w.jpg) + + + + + +## 总结 + +### 核心要点回顾 + +通过本文的学习，我们掌握了算法复杂度分析的核心知识： + +#### 1. 理论基础 +- **时间复杂度**：衡量算法执行时间随输入规模增长的趋势 +- **空间复杂度**：衡量算法占用内存空间随输入规模增长的趋势 +- **大O符号**：用于描述算法复杂度的数学工具 + +#### 2. 常见复杂度等级（从优到劣） +1. **O(1)** - 常数阶：HashMap查找、数组随机访问 +2. **O(logn)** - 对数阶：二分查找、TreeMap操作 +3. **O(n)** - 线性阶：数组遍历、链表查找 +4. **O(nlogn)** - 线性对数阶：快速排序、归并排序 +5. **O(n²)** - 平方阶：冒泡排序、嵌套循环 +6. **O(2^n)** - 指数阶：递归斐波那契（需要避免） + +#### 3. LeetCode刷题复杂度优化套路 +- **暴力 → 哈希表**：O(n²) → O(n) +- **递归 → 动态规划**：O(2^n) → O(n) +- **暴力查找 → 二分查找**：O(n) → O(logn) +- **嵌套循环 → 双指针**：O(n²) → O(n) + +#### 4. Java开发实战经验 +- **集合选择**：根据操作频率选择ArrayList/LinkedList、HashMap/TreeMap +- **避免陷阱**：字符串拼接、集合遍历修改、N+1查询 +- **空间优化**：滚动数组、就地修改、流式处理 + +### 给Java后端开发者的建议 + +#### 学习路径 +1. **理论基础**：掌握本文的复杂度分析方法 +2. **刷题实战**：在LeetCode上练习复杂度分析 +3. **项目应用**：在实际项目中运用复杂度思维 +4. **持续优化**：定期review代码的性能瓶颈 + +#### 面试准备 +- **必备技能**：能快速分析代码的时间空间复杂度 +- **常考题型**：两数之和、排序算法、动态规划、树遍历 +- **优化思路**：从暴力解法开始，逐步优化到最优解 +- **实际应用**：结合项目经验谈复杂度优化案例 + +#### 职场应用 +- **代码review**：关注性能复杂度，不只是功能正确性 +- **系统设计**：提前评估算法复杂度，避免性能瓶颈 +- **技术选型**：基于复杂度分析选择合适的数据结构和算法 +- **性能调优**：使用复杂度分析定位和解决性能问题 + +### 进阶学习方向 + +如果你想在算法和数据结构方面更进一步，建议关注： + +1. **高级数据结构**：并查集、线段树、字典树 +2. **算法设计模式**：分治、贪心、回溯、动态规划 +3. **系统设计**：分布式系统中的复杂度考量 +4. **性能优化**：JVM调优、数据库优化、缓存策略 + +### 写在最后 + +复杂度分析不是纸上谈兵的理论知识，而是每个Java后端开发者都应该掌握的实用技能。它帮助我们： + +- **写出更高效的代码** +- **在面试中展现技术功底** +- **在系统设计时做出正确决策** +- **在性能调优时快速定位问题** + +希望这篇文章能帮助你建立起完整的复杂度分析知识体系。记住，**理论学习 + 刷题实战 + 项目应用** 才是掌握算法复杂度的最佳路径。 + +--- + +**下期预告**：我们将深入探讨**数组与链表**的底层实现和应用技巧，敬请期待！ + +> 如果觉得文章对你有帮助，欢迎点赞分享，你的支持是我创作的动力！ +> +> 更多Java技术文章请关注 GitHub [JavaKeeper](https://github.com/Jstarfish/JavaKeeper) \ No newline at end of file diff --git a/docs/data-structure-algorithms/data-structure/Array.md b/docs/data-structure-algorithms/data-structure/Array.md new file mode 100644 index 0000000000..81505f83ea --- /dev/null +++ b/docs/data-structure-algorithms/data-structure/Array.md @@ -0,0 +1,412 @@ +### 数组 + +数组可以说是最基本最常见的数据结构。数组一般用来存储相同类型的数据，可通过数组名和下标进行数据的访问和更新。数组中元素的存储是按照先后顺序进行的，同时在内存中也是按照这个顺序进行连续存放。数组相邻元素之间的内存地址的间隔一般就是数组数据类型的大小。 + +因为 `字符串` 是由字符数组形成的，所以二者是相似的。大多数面试问题都属于这个范畴。 + + + +## 前言 + +具体介绍数组之前，我们先来了解一下集合、列表和数组的概念之间的差别。 + +### 集合 + +集合一般被定义为：由一个或多个确定的元素所构成的整体。 + +通俗来讲，集合就是将一组事物组合在一起。你可以将力扣的题库看作一个集合： + +![1.png](https://tva1.sinaimg.cn/large/008i3skNly1gqozoy8b55j30xq047tbg.jpg) + + + +也可以将力扣商店里的礼品看作一个集合： + +![2.png](https://tva1.sinaimg.cn/large/008i3skNly1gqozpjcnmvj31dv0lngxp.jpg) + +甚至可以将桌面上的物品当作一个集合。 + +集合有什么特性呢？ + +首先，**集合里的元素类型不一定相同。** 你可以将商品看作一个集合，也可以将整个商店看作一个集合，这个商店中有人或者其他物品也没有关系。 + +其次，**集合里的元素没有顺序。** 我们不会这样讲：我想要集合中的第三个元素，因为集合是没有顺序的。 + +事实上，这样的集合并不直接存在于编程语言中。然而，实际编程语言中的很多数据结构，就是在集合的基础上添加了一些规则形成的。 + + + +### 列表 + +列表（又称线性列表）的定义为：是一种数据项构成的有限序列，即按照一定的线性顺序，排列而成的数据项的集合。 + +列表的概念是在集合的特征上形成的，它具有顺序，且长度是可变的。你可以把它看作一张购物清单： + +![3.png](https://tva1.sinaimg.cn/large/008i3skNly1gqozqdf70vj30dg0cwmxx.jpg) + +在这张清单中： + +- 购物清单中的条目代表的类型可能不同，但是按照一定顺序进行了排列； +- 购物清单的长度是可变的，你可以向购物清单中增加、删除条目。 + +在编程语言中，列表最常见的表现形式有数组和链表，而我们熟悉的栈和队列则是两种特殊类型的列表。除此之外，向列表中添加、删除元素的具体实现方式会根据编程语言的不同而有所区分。 + + + +### 数组 + +数组是列表的实现方式之一，也是面试中经常涉及到的数据结构。 + +正如前面提到的，数组是列表的实现方式，它具有列表的特征，同时也具有自己的一些特征。然而，在具体的编程语言中，数组这个数据结构的实现方式具有一定差别。比如 C++ 和 Java 中，数组中的元素类型必须保持一致，而 Python 中则可以不同。Python 中的数组叫做 list，具有更多的高级功能。 + +那么如何从宏观上区分列表和数组呢？这里有一个重要的概念：**索引**。 + +首先，数组会用一些名为 `索引` 的数字来标识每项数据在数组中的位置，且在大多数编程语言中，索引是从 `0` 算起的。我们可以根据数组中的索引，快速访问数组中的元素。 + +![4.png](https://tva1.sinaimg.cn/large/008i3skNly1gqozrdvktdj30iy06qmx4.jpg) + +**而列表中没有索引，这是数组与列表最大的不同点**。 + +其次，数组中的元素在内存中是连续存储的，且每个元素占用相同大小的内存。 + +![5.png](https://tva1.sinaimg.cn/large/008i3skNly1gqozrke8wuj30ux0gr409.jpg) + +相反，列表中的元素在内存中可能彼此相邻，也可能不相邻。比如列表的另一种实现方式——链表，它的元素在内存中则不一定是连续的。 + + + +## 数组的操作 + +### 读取元素 + +读取数组中的元素，即通过数组的索引访问数组中的元素。 + +这里的索引其实就是内存地址，值得一提的是，计算机可以跳跃到任意的内存地址上，这就意味着只要计算出数组中元素的内存地址，则可以一步访问到数组中的元素。 + +可以形象地将计算机中的内存看作一系列排列好的格子，这些格子中，每一个格子对应一个内存地址，数据会存储在不同的格子中。 + +![1.png](https://tva1.sinaimg.cn/large/008i3skNly1gqozs98zimj30zk0k0ab5.jpg) + +而对于数组，计算机会在内存中申请一段 **连续** 的空间，并且会记下索引为 `0` 处的内存地址。例如对于一个数组 `['oranges', 'apples', 'bananas', 'pears', 'tomatoes']`，为了方便起见，我们假设每个元素只占用一个字节，它的索引与内存地址的关系如下图所示。 + +![2.png](https://tva1.sinaimg.cn/large/008i3skNly1gqozslroucj30zk0akdgn.jpg) + +当我们访问数组中索引为 `3` 处的元素时，计算机会进行如下计算： + +- 找到该数组的索引 `0` 的内存地址： `2008`； +- `pears` 的索引为 `3`，计算该元素的内存地址为 `2008 + 3 = 2011`； + +接下来，计算机就可以在直接通过该地址访问到数组中索引为 `3` 的元素了，计算过程很快，因此可以将整个访问过程只看作一个动作，因此时间复杂度为 $O(1)$。 + + + +### 查找元素 + +前面我们谈到计算机只会保存数组中索引为 `0` 处元素的内存地址，因此当计算机想要知道数组中是否包含某个元素时，只能从索引 `0` 处开始，逐步向后查询。 + +还是上面的例子，如果我们要查找数组中是否包含元素 `pears`，计算机会从索引 `0` 开始，逐个比较对应的元素，直到找到该元素后停止搜索，或到达数组的末尾后停止。 + +![3.gif](https://tva1.sinaimg.cn/large/008i3skNly1gqozxdyr51g316o0dce89.gif) + +我们发现，该数组的长度为 `5`，最坏情况下（比如我们查找元素 `tomatoes` 或查找数组中不包含的元素），我们需要查询数组中的每个元素，因此时间复杂度为$ O(N)$，*N* 为数组的长度。 + + + +### 插入元素 + +假如我们想在原有的数组中再插入一个元素 `flowers` 呢？ + +如果要将该元素插入到数组的末尾，只需要一步。即计算机通过数组的长度和位置计算出即将插入元素的内存地址，然后将该元素插入到指定位置即可。 + +![4.gif](https://tva1.sinaimg.cn/large/008i3skNly1gqozvafsuog30uh0dcu0x.gif) + +然而，如果要将该元素插入到数组中的其他位置，则会有所区别，这时我们首先需要为该元素所要插入的位置`腾出` 空间，然后进行插入操作。比如，我们想要在索引 `2` 处插入 `flowers`。 + +![5.gif](https://tva1.sinaimg.cn/large/008i3skNly1gqozvjtbgtg30uh0dcu0z.gif) + +我们发现，如果需要频繁地对数组元素进行插入操作，会造成时间的浪费。事实上，另一种数据结构，即链表可以有效解决这个问题。 + + + +### 删除元素 + +删除元素与插入元素的操作类似，当我们删除掉数组中的某个元素后，数组中会留下 `空缺` 的位置，而数组中的元素在内存中是连续的，这就使得后面的元素需对该位置进行 `填补` 操作。 + +以删除索引 `1` 中的元素 `apples` 为例，具体过程如图所示。 + +![6.gif](https://tva1.sinaimg.cn/large/008i3skNly1gqozvs74g7g30uh0dcu0x.gif) + +同样地，数组的长度为 `5`，最坏情况下，我们删除第一个元素，后面的 `4` 个元素需要向前移动，加上删除操作，共需执行 `5` 步，因此时间复杂度为 $O(N)$，*N* 为数组的长度。 + + + + + +## 二维数组 + +二维数组是一种结构较为特殊的数组，只是将数组中的每个元素变成了一维数组。 + +![1.png](https://tva1.sinaimg.cn/large/008i3skNly1gqozw53udoj30zk0ftdgm.jpg) + +所以二维数组的本质上仍然是一个一维数组，内部的一维数组仍然从索引 `0` 开始，我们可以将它看作一个矩阵，并处理矩阵的相关问题。 + + + +### 示例 + +类似一维数组，对于一个二维数组 `A = [[1, 2, 3, 4],[2, 4, 5, 6],[1, 4, 6, 8]]`，计算机同样会在内存中申请一段 **连续** 的空间，并记录第一行数组的索引位置，即 `A[0][0]` 的内存地址，它的索引与内存地址的关系如下图所示。 + +![2.png](https://tva1.sinaimg.cn/large/008i3skNly1gqozwtoa8xj30zk0ftq6g.jpg) + +注意，实际数组中的元素由于类型的不同会占用不同的字节数，因此每个方格地址之间的差值可能不为 `1`。 + +实际题目中，往往使用二维数据处理矩阵类相关问题，包括矩阵旋转、对角线遍历，以及对子矩阵的操作等。 + + + + + +## 刷题 + +![leetcode-hot100-array](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/algorithms/leetcode-hot100-array.png) + +### [1. 两数之和](https://leetcode-cn.com/problems/two-sum/) + +> 给定一个整数数组 nums 和一个目标值 target，请你在该数组中找出和为目标值的那 两个 整数，并返回他们的数组下标。 +> +> 你可以假设每种输入只会对应一个答案。但是，数组中同一个元素不能使用两遍。 +> +> ``` +>输入：nums = [2,7,11,15], target = 9 +> 输出：[0,1] +>解释：因为 nums[0] + nums[1] == 9 ，返回 [0, 1] 。 +> ``` + + + + + +### 寻找两个正序数组的中位数(4) + +> 给定两个大小为 m 和 n 的正序（从小到大）数组 nums1 和 nums2。 +> +> 请你找出这两个正序数组的中位数，并且要求算法的时间复杂度为 O(log(m + n))。 +> +> 你可以假设 nums1 和 nums2 不会同时为空。 +> +> +> +> 示例 1: +> +> nums1 = [1, 3] +> nums2 = [2] +> +> 则中位数是 2.0 +> 示例 2: +> +> nums1 = [1, 2] +> nums2 = [3, 4] +> +> 则中位数是 (2 + 3)/2 = 2.5 +> + +二分查找 + +给定两个有序数组，要求找到两个有序数组的中位数，最直观的思路有以下两种： + +- 使用归并的方式，合并两个有序数组，得到一个大的有序数组。大的有序数组的中间位置的元素，即为中位数。 + +- 不需要合并两个有序数组，只要找到中位数的位置即可。由于两个数组的长度已知，因此中位数对应的两个数组的下标之和也是已知的。维护两个指针，初始时分别指向两个数组的下标 00 的位置，每次将指向较小值的指针后移一位（如果一个指针已经到达数组末尾，则只需要移动另一个数组的指针），直到到达中位数的位置。 + + + + + +### [215. 数组中的第K个最大元素](https://leetcode-cn.com/problems/kth-largest-element-in-an-array/) + +> 给定整数数组 nums 和整数 k，请返回数组中第 k 个最大的元素。 +> +> 请注意，你需要找的是数组排序后的第 k 个最大的元素，而不是第 k 个不同的元素。 +> +> +> +> 示例 1: +> +> 输入: [3,2,1,5,6,4] 和 k = 2 +> 输出: 5 +> +> 输入: [3,2,3,1,2,4,5,5,6] 和 k = 4 +> 输出: 4 + + + +### [11. 盛最多水的容器](https://leetcode-cn.com/problems/container-with-most-water/) + +> 给你 n 个非负整数 a1，a2，...，an，每个数代表坐标中的一个点 (i, ai) 。在坐标内画 n 条垂直线，垂直线 i 的两个端点分别为 (i, ai) 和 (i, 0) 。找出其中的两条线，使得它们与 x 轴共同构成的容器可以容纳最多的水。 +> +> 说明：你不能倾斜容器。 +> +> +> +> 示例 1： +> +> ![img](https://aliyun-lc-upload.oss-cn-hangzhou.aliyuncs.com/aliyun-lc-upload/uploads/2018/07/25/question_11.jpg) +> +> 输入：[1,8,6,2,5,4,8,3,7] +> 输出：49 +> 解释：图中垂直线代表输入数组 [1,8,6,2,5,4,8,3,7]。在此情况下，容器能够容纳水（表示为蓝色部分）的最大值为 49。 +> +> 输入：height = [1,1] +> 输出：1 +> +> 输入：height = [4,3,2,1,4] +> 输出：16 +> +> 输入：height = [1,2,1] +> 输出：2 + +双指针，，从两头开始内卷，先卷了挫的那头 + + + +### [200. 岛屿数量](https://leetcode-cn.com/problems/number-of-islands/) + +> 给你一个由 '1'（陆地）和 '0'（水）组成的的二维网格，请你计算网格中岛屿的数量。 +> +> 岛屿总是被水包围，并且每座岛屿只能由水平方向和/或竖直方向上相邻的陆地连接形成。 +> +> 此外，你可以假设该网格的四条边均被水包围。 +> +> +> +> 示例 ： +> +> ``` +> 输入：grid = [ +> ["1","1","1","1","0"], +> ["1","1","0","1","0"], +> ["1","1","0","0","0"], +> ["0","0","0","0","0"] +> ] +> 输出：1 +> ``` +> +> ``` +> 输入：grid = [ +> ["1","1","0","0","0"], +> ["1","1","0","0","0"], +> ["0","0","1","0","0"], +> ["0","0","0","1","1"] +> ] +> 输出：3 +> ``` + + + + + +### 移动零(283) + +> 给定一个数组 nums，编写一个函数将所有 0 移动到数组的末尾，同时保持非零元素的相对顺序。 +> +> 示例: +> +> 输入: [0,1,0,3,12] +> 输出: [1,3,12,0,0] +> 说明: +> +> 必须在原数组上操作，不能拷贝额外的数组。 +> 尽量减少操作次数。 + + + +### 排序算法() + + + +### 买卖股票的最佳时机(121) + +> 给定一个数组，它的第 i 个元素是一支给定股票第 i 天的价格。 +> +> 如果你最多只允许完成一笔交易（即买入和卖出一支股票一次），设计一个算法来计算你所能获取的最大利润。 +> +> 注意：你不能在买入股票前卖出股票。 +> +> 示例 1: +> +> 输入: [7,1,5,3,6,4] +> 输出: 5 +> 解释: 在第 2 天（股票价格 = 1）的时候买入，在第 5 天（股票价格 = 6）的时候卖出，最大利润 = 6-1 = 5 。 +> 注意利润不能是 7-1 = 6, 因为卖出价格需要大于买入价格；同时，你不能在买入前卖出股票。 +> 示例 2: +> +> 输入: [7,6,4,3,1] +> 输出: 0 +> 解释: 在这种情况下, 没有交易完成, 所以最大利润为 0。 + + + + + +### [15. 三数之和](https://leetcode-cn.com/problems/3sum/) + +> 给你一个包含 n 个整数的数组 nums，判断 nums 中是否存在三个元素 a，b，c ，使得 a + b + c = 0 ？请你找出所有满足条件且不重复的三元组。 +> +> 注意：答案中不可以包含重复的三元组。 +> +> +> +> 示例： +> +> 给定数组 nums = [-1, 0, 1, 2, -1, -4]， +> +> 满足要求的三元组集合为： +> [ +> [-1, 0, 1], +> [-1, -1, 2] +> ] + + + + + +### 最小路径和(64) + +> 给定一个包含非负整数的 m x n 网格，请找出一条从左上角到右下角的路径，使得路径上的数字总和为最小。 +> +> 说明：每次只能向下或者向右移动一步。 +> +> 示例: +> +> 输入: +> [ +> [1,3,1], +> [1,5,1], +> [4,2,1] +> ] +> 输出: 7 +> 解释: 因为路径 1→3→1→1→1 的总和最小。 + + + + + + + +### 双索引技巧-对撞指针 + +### 双索引技巧-滑动窗口 + + + + + + + + + + + + + +### 稀疏数组 \ No newline at end of file diff --git a/docs/data-structure/tree.md b/docs/data-structure-algorithms/data-structure/Binary-Tree.md old mode 100644 new mode 100755 similarity index 53% rename from docs/data-structure/tree.md rename to docs/data-structure-algorithms/data-structure/Binary-Tree.md index 8a89cd44be..4f22ac687b --- a/docs/data-structure/tree.md +++ b/docs/data-structure-algorithms/data-structure/Binary-Tree.md @@ -1,76 +1,101 @@ -# 数据结构中各种树 +--- +title: 程序员心里得有点树——重学数据结构之二叉树 +date: 2022-06-09 +tags: + - data-structure + - binary-tree +categories: data-structure +--- ## 前言 -面试必问的数据结构内容也比较多，分块整理一下 ，先学习树 +> 重学二叉树 ## 树 树是一种数据结构，它是由n（n>=0）个有限节点组成一个具有层次关系的集合，存储的是具有“一对多”关系的数据元素的集合。把它叫做“树”是因为它看起来像一棵倒挂的树，也就是说它是根朝上，而叶朝下的 -- 除根节点之外的节点被划分为非空集，其中每个节点将被称为子树 -- 树的节点要么保持它们之间的父子关系，要么它们是姐妹节点 +- 除根节点之外的节点被划分为非空集，其中每个节点被称为子树 +- 树的节点父子关系，就是姐妹（兄弟）关系 - 在通用树中，一个节点可以具有任意数量的子节点，但它只能有一个父节点 -- 下图显示了一棵树，其中节点`A`是树的根节点，而其他节点可以看作是`A`的子节点 +- 下图就是一棵树，节点 `A` 为根节点，而其他节点可以看作是 `A` 的子节点 -![tree-demo](https://tva1.sinaimg.cn/large/007S8ZIlly1gdu6xqrnnfj31920hotcv.jpg) +![tree-demo](https://img.starfish.ink/data-structure/tree-demo.png) -​ -### 基本术语 -- **根节点** ： 树中最顶端的节点，根没有父节点 -- **子树**： 如果根节点不为空，则树`T1`，`T2`和`T3`称为根节点的子树。 -- **父节点（Parent）**：如果节点拥有子节点，则该节点为子节点的父节点。 -- **叶节点**：没有子节点的节点，是树的末端节点。 -- **路径**： 连续边的序列称为路径。 在上图所示的树中，节点`E`的路径为`A→B→E`。 -- **祖先节点**： 节点的祖先是从根到该节点的路径上的任何前节点。根节点没有祖先节点。 在上图所示的树中，节点`F`的祖先是`B`和`A`。 -- **度**： 节点的度数等于子节点数。 在上图所示的树中，节点`B`的度数为`2`。叶子节点的度数总是`0`，而在完整的二叉树中，每个节点的度数等于`2`。 -- **边（Edge）**：两个节点中间的链接。 -- **高度（Height）/深度（Depth）**：树中层的数量。比如只有 Level 0,Level 1,Level 2 则高度为 3。 -- **级别编号**： 为树的每个节点分配一个级别编号，使得每个节点都存在于高于其父级的一个级别。树的根节点始终是级别`0`。 -- **层级（Level）**：根为 Level 0 层，根的子节点为 Level 1 层，以此类推。 -- 有序树、无序树：如果将树中的各个子树看成是从左到右是有次序的，则称该树是有序树；若不考虑子树的顺序称为无序树。 -- 森林：m（m>=0）棵互不交互的树的集合。对树中每个结点而言，其子树的集合即为森林。 +### [基本术语](https://www.bootwiki.com/datastructure/data-structure-tree.html) +- **根节点** ：树中最顶端的节点，根没有父节点（树中至少有一个节点——根） +- **子树**： 如果根节点不为空，则树 `B`，`C` 和 `D` 称为根节点的子树（树中各子树是互不相交的集合） +- **父节点（Parent）**：如果节点拥有子节点，则该节点为子节点的父节点 +- **叶节点**：没有子节点的节点，是树的末端节点 +- **边（Edge）**：两个节点中间的链接 +- **路径**： 连续边的序列称为路径。 在上图所示的树中，节点`E`的路径为`A→B→E` +- **祖先节点**： 节点的祖先是从根到该节点的路径上的任何前节点。根节点没有祖先节点。 在上图所示的树中，节点`F`的祖先是`B`和`A` +- **度**： 节点的度数等于子节点数。 在上图所示的树中，节点`B`的度数为`2`。叶子节点的度数总是`0`，而在完整的二叉树中，每个节点的度数等于`2` +- **高度（Height）**：[根]节点到叶子节点的最常路径（边数） +- 深度（Depth）：根节点到这个节点所经历的边的个数 +- **级别编号**： 为树的每个节点分配一个级别编号，使得每个节点都存在于高于其父级的一个级别。树的根节点始终是级别`0`。 +- **层级（Level）**：根为 Level 0 层，根的子节点为 Level 1 层，以此类推 +- 有序树、无序树：如果将树中的各个子树看成是从左到右是有次序的，则称该树是有序树；若不考虑子树的顺序称为无序树 +- 森林：m（m>=0）棵互不交互的树的集合。对树中每个结点而言，其子树的集合即为森林 +![](https://static001.geekbang.org/resource/image/50/b4/50f89510ad1f7570791dd12f4e9adeb4.jpg) ### 基本操作 +1. 构造空树（初始化） + +2. 销毁树（将树置为空树） + +3. 求双亲函数 + +4. 求孩子节点函数 + +5. 插入子树 + +6. 遍历操作 + + ...... + + +> 其他都好理解，主要回顾下几种遍历操作，也是面试常客 + #### 遍历 -遍历的含义就是把树的所有节点（Node）按照**某种顺序**访问一遍。包括**前序**，**中序**，**后续**，**广度优先**（队列），**深度优先**（栈）5种遍历方法 +遍历的含义就是把树的所有节点（Node）按照**某种顺序**访问一遍。包括**前序**，**中序**，**后续**，**广度优先**（队列），**深度优先**（栈）5 种遍历方法 -| 遍历方法 | 顺序 | 示意图 | 顺序 | 应用 | -| :------: | :----------------------: | :----------------------------------------------------------: | -------- | :----------------------------------------------------------: | -| 前序 | **根 ➜ 左 ➜ 右** | ![img](https://tva1.sinaimg.cn/large/007S8ZIlly1gdufbw5witj30c90crwf9.jpg) | 12457836 | 想在节点上直接执行操作（或输出结果）使用先序 | -| 中序 | **左 ➜ 根 ➜ 右** | ![img](https://charlesliuyx.github.io/2018/10/22/%E3%80%90%E7%9B%B4%E8%A7%82%E7%AE%97%E6%B3%95%E3%80%91%E6%A0%91%E7%9A%84%E5%9F%BA%E6%9C%AC%E6%93%8D%E4%BD%9C/In-Order.png) | 42758136 | 在**二分搜索树**中，中序遍历的顺序符合从小到大（或从大到小）顺序的 要输出排序好的结果使用中序 | -| 后序 | **左 ➜ 右 ➜ 根** | ![img](https://charlesliuyx.github.io/2018/10/22/%E3%80%90%E7%9B%B4%E8%A7%82%E7%AE%97%E6%B3%95%E3%80%91%E6%A0%91%E7%9A%84%E5%9F%BA%E6%9C%AC%E6%93%8D%E4%BD%9C/Post-Order.png) | 47852631 | 后续遍历的特点是在执行操作时，肯定**已经遍历过该节点的左右子节点** 适用于进行破坏性操作 比如删除所有节点，比如判断树中是否存在相同子树 | -| 广度优先 | **层序，横向访问** | ![img](https://charlesliuyx.github.io/2018/10/22/%E3%80%90%E7%9B%B4%E8%A7%82%E7%AE%97%E6%B3%95%E3%80%91%E6%A0%91%E7%9A%84%E5%9F%BA%E6%9C%AC%E6%93%8D%E4%BD%9C/Breadth-First.png) | | 当**树的高度非常高**（非常瘦） 使用广度优先剑节省空间 | -| 深度优先 | **纵向，探底到叶子节点** | ![img](https://charlesliuyx.github.io/2018/10/22/%E3%80%90%E7%9B%B4%E8%A7%82%E7%AE%97%E6%B3%95%E3%80%91%E6%A0%91%E7%9A%84%E5%9F%BA%E6%9C%AC%E6%93%8D%E4%BD%9C/Deep-First.png) | 12457836 | 当**每个节点的子节点非常多**（非常胖），使用深度优先遍历节省空间 （访问顺序和入栈顺序相关，想当于先序遍历） | +| 遍历方法 | 顺序 | 示意图 | 顺序 | 应用 | +| -------- | ------------------------ | ------------------------------------------------------------ | -------- | ------------------------------------------------------------ | +| 前序 | **根 ➜ 左 ➜ 右** | ![](https://img.starfish.ink/data-structure/binary-tree-preorder.png) | 12457836 | 想在节点上直接执行操作（或输出结果）使用先序 | +| 中序 | **左 ➜ 根 ➜ 右** | ![](https://img.starfish.ink/data-structure/binary-tree-inorder.png) | 42758136 | 在**二分搜索树**中，中序遍历的顺序符合从小到大（或从大到小）顺序的 要输出排序好的结果使用中序 | +| 后序 | **左 ➜ 右 ➜ 根** | ![](https://img.starfish.ink/data-structure/binary-tree-postorder.jpeg) | 47852631 | 后续遍历的特点是在执行操作时，肯定**已经遍历过该节点的左右子节点** 适用于进行破坏性操作 比如删除所有节点，比如判断树中是否存在相同子树 | +| 广度优先 | **层序，横向访问** | ![](https://img.starfish.ink/data-structure/binary-tree-leveltraverse.png) | 12345678 | 当**树的高度非常高**（非常瘦） 使用广度优先剑节省空间 | +| 深度优先 | **纵向，探底到叶子节点** | ![](https://img.starfish.ink/data-structure/binary-tree-dfs.png) | 12457836 | 当**每个节点的子节点非常多**（非常胖），使用深度优先遍历节省空间 （访问顺序和入栈顺序相关，相当于先序遍历） | +> 之所以叫前序、中序、后序遍历，是因为根节点在前、中、后 +> 数据结构中有很多树的结构，其中包括二叉树、二叉搜索树、2-3树、红黑树等等。本文中对数据结构中常见的几种树的概念和用途进行了汇总，不求严格精准，但求简单易懂。 ## 二叉树 -二叉树是每个节点最多有两个子树的树结构。 +二叉树是每个节点最多有两个子树的树结构 - 二叉树中不存在度大于 2 的结点 - 左子树和右子树是有顺序的，次序不能任意颠倒 根据二叉树的定义和特点，可以将二叉树分为五种不同的形态，如下图所示 -![img](https://tva1.sinaimg.cn/large/007S8ZIlly1gdueuzg24ij31o00dkq4r.jpg) +![](https://img.starfish.ink/data-structure/binary-tree-structure.jpeg) +### 二叉树的性质 - -### 二叉树的性质 todo - -- 在非空二叉树中，二叉树的第i( i>=1)层最多有2^(i-1)个结点 -- 深度为k(k>=1)的二叉树最多有 2^k – 1 个结点，最少有k个结点； -- 对于任意一棵非空二叉树如果其叶结点数为n0，而度为2的非叶结点总数为n2，则n0=n2+1； -- 具有 n (n>=0) 个结点的完全二叉树的深度为 log2(n) +1 -- 任意一棵二叉树，其节点个数等于分支个数加1，及n=B+1 +- 在非空二叉树中，二叉树的第 i 层最多有 2^(i-1) 个结点（i>=1）； +- 深度为 k 的二叉树最多有 2^k – 1 个结点，最少有 k 个结点（k>=1）； +- 对于任意一棵非空二叉树如果其叶结点数为 n0，而度为 2 的非叶结点总数为 n2，则 n0=n2+1； +- 具有 n (n>=0) 个结点的完全二叉树的深度为 log2(n) +1； +- 任意一棵二叉树，其节点个数等于分支个数加 1，即 n=B+1 ### 两个特别的二叉树 @@ -79,15 +104,13 @@ - 深度为 k 的满二叉树必有 2^(k-1) 个节点 ，叶子数为 2^(k-1) - 满二叉树中不存在度为 1 的节点，每一个分支点中都两棵深度相同的子树，且叶子节点都在最底层 - 具有 n 个节点的满二叉树的深度为 log2(n+1) -- 完全二叉树：若设二叉树的深度为h，除第 h 层外，其它各层 (1～(h-1)层) 的结点数都达到最大个数，第h层所有的结点都连续集中在最左边，这就是完全二叉树。 - -![img](https://tva1.sinaimg.cn/large/007S8ZIlly1gdub8xjo9ej319d0i2417.jpg) - +- **完全二叉树**：若设二叉树的深度为 h，除第 h 层外，其它各层 (1～(h-1)层) 的结点数都达到最大个数，第 h 层所有的结点都连续集中在最左边，这就是完全二叉树。 +![](https://img.starfish.ink/data-structure/binary-tree-special-case.jpeg) **满二叉树一定是一颗棵完全二叉树，但完全二叉树不一定是满二叉树。** - +- 其实还有一种更特殊的二叉树：**斜树**，顾名思义，就是斜着长的，分为左斜树和右斜树。（线性表结构可以理解为是树的一种极其特殊的表现形式） ### 常见的存储方法 @@ -99,11 +122,11 @@ 完全二叉树的顺序存储，仅需从根节点开始，按照层次依次将树中节点存储到数组即可。 -![img](https://tva1.sinaimg.cn/large/007S8ZIlly1gdu8idau6rj30ty0mcdhu.jpg) +![](https://img.starfish.ink/data-structure/binary-tree-store1.jpeg) 普通二叉树转完全二叉树，只需给二叉树额外添加一些节点，将其"拼凑"成完全二叉树即可。 -![img](https://tva1.sinaimg.cn/large/007S8ZIlly1gdu8pb6ua6j30wo0mc768.jpg) +![](https://img.starfish.ink/data-structure/binary-tree-store2.jpeg) #### 二叉树的链式存储结构 @@ -117,17 +140,13 @@ 其中，data 域存放某结点的数据信息；lchild 与 rchild 分别存放指向左孩子和右孩子的指针，当左孩子或右孩子不存在时，相应指针域值为空（用符号 ∧ 或 NULL 表示）。 -![img](https://tva1.sinaimg.cn/large/007S8ZIlly1gdub7geom9j31e00jo76o.jpg) - - +![](https://img.starfish.ink/data-structure/binary-tree-node-store.jpeg) ##### 三叉链表存储 为了方便找到父节点，可以在上述结点结构中增加一个指针域，指向结点的父结点。利用此结点结构得到的二叉树存储结构称为三叉链表。 -![img](https://tva1.sinaimg.cn/large/007S8ZIlly1gdub83md08j31ky0lcad9.jpg) - - +![](https://img.starfish.ink/data-structure/binary-tree-three-store.jpeg) ### 二叉树的基本操作 @@ -135,28 +154,20 @@ 关于应用部分，选择遍历方法的基本的原则：**更快的访问到你想访问的节点**。先序会先访问根节点，后序会先访问叶子节点 - - -#### coding - -各种遍历算法 - -《Java数据结构与算法》 +> coding 部分，下一篇结合 leetcode 常见的二叉树算法题，再一并说下二叉树的建立、递归等操作 ------ - - ## 二叉查找树 -**二叉查找树定义**：又称为是二叉排序树（Binary Sort Tree）或二叉搜索树。二叉排序树要么是一棵空树，要么是具有如下性质的二叉树： +**二叉查找树定义**：又称为二叉排序树（Binary Sort Tree）或二叉搜索树。二叉排序树要么是一棵空树，要么是具有如下性质的二叉树： 1. 若它的左子树不为空，则左子树上所有结点的值均小于它的根结点的值 2. 若它的右子树不为空，则右子树上所有结点的值均大于它的根结点的值 -3. 左、右子树也分别为二叉排序树 +3. 左、右子树也分别为二叉排序树 4. 没有键值相等的节点 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gduhfteyh8j316k0jqdiy.jpg) +![](https://img.starfish.ink/data-structure/binary-search-tree.jpeg) ### 性质 @@ -164,7 +175,7 @@ ### 时间复杂度 -它和二分查找一样，插入和查找的时间复杂度均为O(log2n)，但是在最坏的情况下仍然会有O(n)的时间复杂度。原因在于插入和删除元素的时候，树没有保持平衡。我们追求的是在最坏的情况下仍然有较好的时间复杂度，这就是平衡查找树设计的初衷。 +它和二分查找一样，插入和查找的时间复杂度均为 $O(log2n)$，但是在最坏的情况下仍然会有 $O(n)$ 的时间复杂度。原因在于插入和删除元素的时候，树没有保持平衡。我们追求的是在最坏的情况下仍然有较好的时间复杂度，这就是平衡查找树设计的初衷。 **二叉查找树的高度决定了二叉查找树的查找效率。** @@ -195,41 +206,82 @@ public class BinarySearchTree>{ **1.中序遍历：当到达某个节点时，先访问左子节点，再输出该节点，最后访问右子节点。** ```java -public void inOrder(TreeNode cursor){ - if(cursor == null) return; - inOrder(cursor.getLeft()); - System.out.println(cursor.getData()); - inOrder(cursor.getRight()); +/* 中序遍历 */ +void inOrder(TreeNode root) { + if (root == null) + return; + // 访问优先级：左子树 -> 根节点 -> 右子树 + inOrder(root.left); + list.add(root.val); + inOrder(root.right); } ``` **2. 前序遍历：当到达某个节点时，先输出该节点，再访问左子节点，最后访问右子节点。** ```java -public void preOrder(TreeNode cursor){ - if(cursor == null) return; - System.out.println(cursor.getData()); - inOrder(cursor.getLeft()); - inOrder(cursor.getRight()); +/* 前序遍历 */ +void preOrder(TreeNode root) { + if (root == null) + return; + // 访问优先级：根节点 -> 左子树 -> 右子树 + list.add(root.val); + preOrder(root.left); + preOrder(root.right); } ``` **3. 后序遍历：当到达某个节点时，先访问左子节点，再访问右子节点，最后输出该节点。** ```java -public void postOrder(TreeNode cursor){ - if(cursor == null) return; - inOrder(cursor.getLeft()); - inOrder(cursor.getRight()); - System.out.println(cursor.getData()); +/* 后序遍历 */ +void postOrder(TreeNode root) { + if (root == null) + return; + // 访问优先级：左子树 -> 右子树 -> 根节点 + postOrder(root.left); + postOrder(root.right); + list.add(root.val); +} +``` + +前序、中序和后序遍历都属于深度优先遍历（depth-first traversal），也称深度优先搜索（depth-first search, DFS），它体现了一种“先走到尽头，再回溯继续”的遍历方式。 + +**深度优先遍历就像是绕着整棵二叉树的外围“走”一圈**，在每个节点都会遇到三个位置，分别对应前序遍历、中序遍历和后序遍历。 + +![](https://www.hello-algo.com/chapter_tree/binary_tree_traversal.assets/binary_tree_dfs.png) + +**4. 层序遍历：**层序遍历（level-order traversal）从顶部到底部逐层遍历二叉树，并在每一层按照从左到右的顺序访问节点。 + +层序遍历本质上属于广度优先遍历（breadth-first traversal），也称广度优先搜索（breadth-first search, BFS），它体现了一种“一圈一圈向外扩展”的逐层遍历方式。 + +```java +/* 层序遍历 */ +List levelOrder(TreeNode root) { + // 初始化队列，加入根节点 + Queue queue = new LinkedList<>(); + queue.add(root); + // 初始化一个列表，用于保存遍历序列 + List list = new ArrayList<>(); + while (!queue.isEmpty()) { + TreeNode node = queue.poll(); // 队列出队 + list.add(node.val); // 保存节点值 + if (node.left != null) + queue.offer(node.left); // 左子节点入队 + if (node.right != null) + queue.offer(node.right); // 右子节点入队 + } + return list; } ``` + + #### 2. 树的搜索： 树的搜索和树的遍历差不多，就是在遍历的时候只搜索不输出就可以了（类比有序数组的搜索） -![binary-search-tree-sorted-array-animation](https://tva1.sinaimg.cn/large/007S8ZIlly1gduhlxtuaug30ci0aitbs.gif) +![图片来源：penjee.com](https://img.starfish.ink/data-structure/binary-search-tree-penjee.gif) ```java public boolean searchNode(TreeNode node){ @@ -258,7 +310,7 @@ public boolean searchNode(TreeNode node){ 3. 若插入的元素值小于根节点值，则将元素插入到左子树中 4. 若插入的元素值不小于根节点值，则将元素插入到右子树中 -![How insertion into a binary search tree works, animation](https://tva1.sinaimg.cn/large/007S8ZIlly1gduhm6g61vg30b4073jv8.gif) +![图片来源：penjee.com](https://img.starfish.ink/data-structure/binary-search-tree-insert.gif) ```java public void insertNode(TreeNode node){ @@ -304,51 +356,35 @@ public void insertNode(TreeNode node){ 例如：要在树中删除元素 20 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gdui1vc37sg30gz05zjsw.gif) +![img](https://img.starfish.ink/data-structure/binary-serach-tree-del.png) - 如果删除的元素有两个儿子，那么可以取左子树中最大元素或者右子树中最小元素进行替换，然后将最大元素最小元素原位置置空 例如：要在树中删除元素 15 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gdui2wvk4ag30gz05zabr.gif) - - - - +![](https://camo.githubusercontent.com/f5c03694805a1537f87758951017e1ba75f33f0250393a6a571ec200b67950e1/68747470733a2f2f747661312e73696e61696d672e636e2f6c617267652f30303753385a496c6c7931676475693277766b3461673330677a30357a6162722e676966) - 有序数组转为二叉查找树 -![Optimal Binary Search Tree From Sorted Array](https://blog.penjee.com/wp-content/uploads/2015/12/optimal-binary-search-tree-from-sorted-array.gif) +![图片来源：penjee.com](https://img.starfish.ink/data-structure/array-2-binary-tree.png) - 将二叉树转为有序数组 -![Degeneration of Binary Search Tree Demonstration and Animation](https://blog.penjee.com/wp-content/uploads/2015/11/binary-search-tree-degenerating-demo-animation.gif) - -#### coding - - - - +![图片来源：penjee.com](https://img.starfish.ink/data-structure/binary-tree-2-array.png) ------ - - ## 平衡二叉树 -二叉搜索树虽然在插入和删除时的效率都有所提升，但是如果原序列有序时，比如 {3,4,5,6,7}，这个时候构造二叉树搜索树就变成了斜树，二叉树退化成单链表，搜索效率降低到 O(n)，查找数字 7 的话，需要找 5 次。这又说明了**二叉查找树的高度决定了二叉查找树的查找效率**。 +二叉搜索树虽然在插入和删除时的效率都有所提升，但是如果原序列有序时，比如 {3,4,5,6,7}，这个时候构造二叉树搜索树就变成了斜树，二叉树退化成单链表，搜索效率降低到 $O(n)$，查找数字 7 的话，需要找 5 次。这又说明了**二叉查找树的高度决定了二叉查找树的查找效率**。 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gduimaw55jj308808tgm0.jpg) +![](https://img.starfish.ink/data-structure/skewed-binary-tree.jpeg) -为了解决这一问题，两位科学家大爷，G. M. Adelson-Velsky和E. M. Landis 又发明了平衡二叉树，又从他两名字中提取出了 AVL，所以平衡二叉树又叫 AVL 树。 +为了解决这一问题，两位科学家大爷，G. M. Adelson-Velsky 和 E. M. Landis 又发明了平衡二叉树，从他两名字中提取出了 AVL，所以平衡二叉树又叫 **AVL 树**。 二叉搜索树的查找效率取决于树的高度，所以保持树的高度最小，就可保证树的查找效率，如下保持左右平衡，像不像天平？ -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gdukvpz7y1j308o07774p.jpg) - -[todo 这个是吗？] - -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gduku6ayvnj308o0750t5.jpg) +![](https://img.starfish.ink/data-structure/balance-binary-tree.jpeg) **定义**： @@ -361,22 +397,13 @@ public void insertNode(TreeNode node){ 某节点的左子树与右子树的高度(深度)差即为该节点的平衡因子（BF,Balance Factor），平衡二叉树中不存在平衡因子大于 1 的节点。在一棵平衡二叉树中，节点的平衡因子只能取 0 、1 或者 -1 ，分别对应着左右子树等高，左子树比较高，右子树比较高。 - - -【 - 在AVL中任何节点的两个儿子子树的高度最大差别为1，所以它也被称为高度平衡树，n个结点的AVL树最大深度约1.44log2n。查找、插入和删除在平均和最坏情况下都是O(logn)。增加和删除可能需要通过一次或多次树旋转来重新平衡这个树。**这个方案很好的解决了二叉查找树退化成链表的问题，把插入，查找，删除的时间复杂度最好情况和最坏情况都维持在O(logN)。但是频繁旋转会使插入和删除牺牲掉O(logN)左右的时间，不过相对二叉查找树来说，时间上稳定了很多。** +平衡二叉树（AVL树）在符合二叉查找树的条件下，还满足任何节点的两个子树的高度最大差为1。下面的两张图片，左边是AVL树，它的任何节点的两个子树的高度差<=1；右边的不是AVL树，其根节点的左子树高度为3，而右子树高度为1； [![索引](https://camo.githubusercontent.com/8e50f7d9828f1e438edcb97b04cb051542d3ad7fa588dd3220aeeb14eed3010d/68747470733a2f2f696d672d626c6f672e6373646e2e6e65742f3230313630323032323033353534363633)](https://camo.githubusercontent.com/8e50f7d9828f1e438edcb97b04cb051542d3ad7fa588dd3220aeeb14eed3010d/68747470733a2f2f696d672d626c6f672e6373646e2e6e65742f3230313630323032323033353534363633) +如果在AVL树中进行插入或删除节点，可能导致AVL树失去平衡，这种失去平衡的二叉树可以概括为四种姿态：LL（左左）、RR（右右）、LR（左右）、RL（右左）。它们的示意图如下： [![索引](https://camo.githubusercontent.com/b67a94873e0ca508363d711117242d0f7f295086fd764f2097f400f189c5ccdb/68747470733a2f2f696d672d626c6f672e6373646e2e6e65742f3230313630323032323033363438313438)](https://camo.githubusercontent.com/b67a94873e0ca508363d711117242d0f7f295086fd764f2097f400f189c5ccdb/68747470733a2f2f696d672d626c6f672e6373646e2e6e65742f3230313630323032323033363438313438) -平衡二叉树（AVL树）在符合二叉查找树的条件下，还满足任何节点的两个子树的高度最大差为1。下面的两张图片，左边是AVL树，它的任何节点的两个子树的高度差<=1；右边的不是AVL树，其根节点的左子树高度为3，而右子树高度为1； -![索引](https://img-blog.csdn.net/20160202203554663) - -如果在AVL树中进行插入或删除节点，可能导致AVL树失去平衡，这种失去平衡的二叉树可以概括为四种姿态：LL（左左）、RR（右右）、LR（左右）、RL（右左）。它们的示意图如下： -![索引](https://img-blog.csdn.net/20160202203648148) - -这四种失去平衡的姿态都有各自的定义： -LL：LeftLeft，也称“左左”。插入或删除一个节点后，根节点的左孩子（Left Child）的左孩子（Left Child）还有非空节点，导致根节点的左子树高度比右子树高度高2，AVL树失去平衡。 +这四种失去平衡的姿态都有各自的定义： LL：LeftLeft，也称“左左”。插入或删除一个节点后，根节点的左孩子（Left Child）的左孩子（Left Child）还有非空节点，导致根节点的左子树高度比右子树高度高2，AVL树失去平衡。 RR：RightRight，也称“右右”。插入或删除一个节点后，根节点的右孩子（Right Child）的右孩子（Right Child）还有非空节点，导致根节点的右子树高度比左子树高度高2，AVL树失去平衡。 @@ -392,8 +419,7 @@ LL的旋转。LL失去平衡的情况下，可以通过一次旋转让AVL树恢 2. 将新根节点的右孩子作为原根节点的左孩子。 3. 将原根节点作为新根节点的右孩子。 -LL旋转示意图如下： -![索引](https://img-blog.csdn.net/20160202204113994) +LL旋转示意图如下： [![索引](https://camo.githubusercontent.com/6850de62dc5ddb0126755ce28bdd117752132cf9749fa8d527068bc5d9af46de/68747470733a2f2f696d672d626c6f672e6373646e2e6e65742f3230313630323032323034313133393934)](https://camo.githubusercontent.com/6850de62dc5ddb0126755ce28bdd117752132cf9749fa8d527068bc5d9af46de/68747470733a2f2f696d672d626c6f672e6373646e2e6e65742f3230313630323032323034313133393934) RR的旋转：RR失去平衡的情况下，旋转方法与LL旋转对称，步骤如下： @@ -401,31 +427,24 @@ RR的旋转：RR失去平衡的情况下，旋转方法与LL旋转对称，步 2. 将新根节点的左孩子作为原根节点的右孩子。 3. 将原根节点作为新根节点的左孩子。 -RR旋转示意图如下： -![索引](https://img-blog.csdn.net/20160202204207963) +RR旋转示意图如下： [![索引](https://camo.githubusercontent.com/910e0d2207881d121608d1efb0b1121732bb8915f5a6a38f8803f1b162d6a0fd/68747470733a2f2f696d672d626c6f672e6373646e2e6e65742f3230313630323032323034323037393633)](https://camo.githubusercontent.com/910e0d2207881d121608d1efb0b1121732bb8915f5a6a38f8803f1b162d6a0fd/68747470733a2f2f696d672d626c6f672e6373646e2e6e65742f3230313630323032323034323037393633) LR的旋转：LR失去平衡的情况下，需要进行两次旋转，步骤如下： 1. 围绕根节点的左孩子进行RR旋转。 2. 围绕根节点进行LL旋转。 -LR的旋转示意图如下： -![索引](https://img-blog.csdn.net/20160202204257369) +LR的旋转示意图如下： [![索引](https://camo.githubusercontent.com/ef203ace5934366c04ddc854bd7f7a57670f549f8c0b12f28ab5096993259887/68747470733a2f2f696d672d626c6f672e6373646e2e6e65742f3230313630323032323034323537333639)](https://camo.githubusercontent.com/ef203ace5934366c04ddc854bd7f7a57670f549f8c0b12f28ab5096993259887/68747470733a2f2f696d672d626c6f672e6373646e2e6e65742f3230313630323032323034323537333639) RL的旋转：RL失去平衡的情况下也需要进行两次旋转，旋转方法与LR旋转对称，步骤如下： 1. 围绕根节点的右孩子进行LL旋转。 2. 围绕根节点进行RR旋转。 -RL的旋转示意图如下： -![索引](https://img-blog.csdn.net/20160202204331073) - - +RL的旋转示意图如下： [![索引](https://camo.githubusercontent.com/36a25db63c2608975c3c40ea85e63bfca51e6601f236721489f07be41bce3684/68747470733a2f2f696d672d626c6f672e6373646e2e6e65742f3230313630323032323034333331303733)](https://camo.githubusercontent.com/36a25db63c2608975c3c40ea85e63bfca51e6601f236721489f07be41bce3684/68747470733a2f2f696d672d626c6f672e6373646e2e6e65742f3230313630323032323034333331303733) 】 - - ### AVL树插入时的失衡与调整 平衡二叉树大部分操作和二叉查找树类似，主要不同在于插入删除的时候平衡二叉树的平衡可能被改变，当平衡因子的绝对值大于1的时候，我们就需要对其进行旋转来保持平衡。 @@ -435,14 +454,11 @@ RL的旋转示意图如下： 假设一颗 AVL 树的某个节点为 A，有四种操作会使 A 的左右子树高度差大于 1，从而破坏了原有 AVL 树的平衡性： 1. 在 A 的左儿子的左子树进行一次插入 - 2. 对 A 的左儿子的右子树进行一次插入 - 3. 对 A 的右儿子的左子树进行一次插入 - 4. 对 A 的右儿子的右子树进行一次插入 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gdulghjq06j31fs0jgdi4.jpg) +![](https://img.starfish.ink/data-structure/avl-tree.png) 情形 1 和情形 4 是关于 A 的镜像对称，情形 2 和情形 3 也是关于 A 的镜像对称，因此理论上看只有两种情况，但编程的角度看还是四种情形。 @@ -460,8 +476,6 @@ https://www.cxyxiaowu.com/1696.html https://www.cnblogs.com/zhangbaochong/p/5164994.html - - ### AVL树的四种删除节点方式 AVL 树和二叉查找树的删除操作情况一致，都分为四种情况： @@ -484,38 +498,23 @@ AVL 树和二叉查找树的删除操作情况一致，都分为四种情况： - - - - ## 红黑树 -**红黑树的定义：**红黑树是一种自平衡二叉查找树，是在计算机科学中用到的一种数据结构，典型的用途是实现关联数组。它是在1972年由鲁道夫·贝尔发明的，称之为"对称二叉B树"，它现代的名字是在 Leo J. Guibas 和 Robert Sedgewick 于1978年写的一篇论文中获得的。**它是复杂的，但它的操作有着良好的最坏情况运行时间，并且在实践中是高效的: 它可以在O(logn)时间内做查找，插入和删除，这里的n是树中元素的数目。** - -红黑树和AVL树一样都对插入时间、删除时间和查找时间提供了最好可能的最坏情况担保。这不只是使它们在时间敏感的应用如实时应用（real time application）中有价值，而且使它们有在提供最坏情况担保的其他数据结构中作为建造板块的价值；例如，在计算几何中使用的很多数据结构都可以基于红黑树。此外，红黑树还是2-3-4树的一种等同，它们的思想是一样的，只不过红黑树是2-3-4树用二叉树的形式表示的。 - -**红黑树的性质：** - -红黑树是每个节点都带有颜色属性的二叉查找树，颜色为红色或黑色。在二叉查找树强制的一般要求以外，对于任何有效的红黑树我们增加了如下的额外要求: +顾名思义，红黑树中的节点，一类被标记为黑色，一类被标记为红色。除此之外，一棵红黑树还需要满足这样几个要求： -- 每个节点要么是黑色，要么是红色 -- 根节点是黑色 -- 所有叶子都是黑色（叶子是NIL节点） -- 每个红色节点必须有两个黑色的子节点(从每个叶子到根的所有路径上不能有两个连续的红色节点) -- **任意一结点到每个叶子结点的简单路径都包含数量相同的黑结点** +- 根节点是黑色的； +- 每个叶子节点都是黑色的空节点（NIL），也就是说，叶子节点不存储数据； +- 任何相邻的节点都不能同时为红色，也就是说，红色节点是被黑色节点隔开的； +- 每个节点，从该节点到达其可达叶子节点的所有路径，都包含相同数目的黑色节点； 下面是一个具体的红黑树的图例： -![An example of a red-black tree](https://upload.wikimedia.org/wikipedia/commons/thumb/6/66/Red-black_tree_example.svg/450px-Red-black_tree_example.svg.png) - - +![](https://static001.geekbang.org/resource/image/90/9a/903ee0dcb62bce2f5b47819541f9069a.jpg) 这些约束确保了红黑树的关键特性: 从根到叶子的最长的可能路径不多于最短的可能路径的两倍长。结果是这个树大致上是平衡的。因为操作比如插入、删除和查找某个值的最坏情况时间都要求与树的高度成比例，这个在高度上的理论上限允许红黑树在最坏情况下都是高效的，而不同于普通的二叉查找树。 要知道为什么这些性质确保了这个结果，注意到性质4导致了路径不能有两个毗连的红色节点就足够了。最短的可能路径都是黑色节点，最长的可能路径有交替的红色和黑色节点。因为根据性质5所有最长的路径都有相同数目的黑色节点，这就表明了没有路径能多于任何其他路径的两倍长。 - - **红黑树的自平衡操作：** 因为每一个红黑树也是一个特化的二叉查找树，因此红黑树上的只读操作与普通二叉查找树上的只读操作相同。然而，在红黑树上进行插入操作和删除操作会导致不再符合红黑树的性质。恢复红黑树的性质需要少量(O(logn))的颜色变更(实际是非常快速的)和不超过三次树旋转(对于插入操作是两次)。虽然插入和删除很复杂，但操作时间仍可以保持为O(logn) 次。 @@ -538,17 +537,15 @@ AVL 树和二叉查找树的删除操作情况一致，都分为四种情况： 　　**情形3:** 如果父节点P和叔父节点U二者都是红色，(此时新插入节点N做为P的左子节点或右子节点都属于情形3,这里右图仅显示N做为P左子的情形)则我们可以将它们两个重绘为黑色并重绘祖父节点G为红色(用来保持性质4)。现在我们的新节点N有了一个黑色的父节点P。因为通过父节点P或叔父节点U的任何路径都必定通过祖父节点G，在这些路径上的黑节点数目没有改变。但是，红色的祖父节点G的父节点也有可能是红色的，这就违反了性质4。为了解决这个问题，我们在祖父节点G上递归地进行上述情形的整个过程（把G当成是新加入的节点进行各种情形的检查）。比如，G为根节点，那我们就直接将G变为黑色（情形1）；如果G不是根节点，而它的父节点为黑色，那符合所有的性质，直接插入即可（情形2）；如果G不是根节点，而它的父节点为红色，则递归上述过程（情形3）。 -![img](https://pic002.cnblogs.com/images/2011/330710/2011120116425251.png) - - +[![img](https://camo.githubusercontent.com/4d3f371b7365cc94d8c9a881f1d750b8d0b72a22b17a4eba9648714370531e8c/68747470733a2f2f7069633030322e636e626c6f67732e636f6d2f696d616765732f323031312f3333303731302f323031313132303131363432353235312e706e67)](https://camo.githubusercontent.com/4d3f371b7365cc94d8c9a881f1d750b8d0b72a22b17a4eba9648714370531e8c/68747470733a2f2f7069633030322e636e626c6f67732e636f6d2f696d616765732f323031312f3333303731302f323031313132303131363432353235312e706e67) 　　**情形4:** 父节点P是红色而叔父节点U是黑色或缺少，新节点N是其父节点的左子节点，而父节点P又是其父节点G的左子节点。在这种情形下，我们进行针对祖父节点G的一次右旋转; 在旋转产生的树中，以前的父节点P现在是新节点N和以前的祖父节点G的父节点。我们知道以前的祖父节点G是黑色，否则父节点P就不可能是红色(如果P和G都是红色就违反了性质4，所以G必须是黑色)。我们切换以前的父节点P和祖父节点G的颜色，结果的树满足性质4。性质5也仍然保持满足，因为通过这三个节点中任何一个的所有路径以前都通过祖父节点G，现在它们都通过以前的父节点P。在各自的情形下，这都是三个节点中唯一的黑色节点。 -![情形5 示意图](https://upload.wikimedia.org/wikipedia/commons/6/66/Red-black_tree_insert_case_5.png) +[![情形5 示意图](https://camo.githubusercontent.com/6589a9c75d8e686270a6f2e981c1dd7ea274ce0c68064861e205ee259dc0a164/68747470733a2f2f75706c6f61642e77696b696d656469612e6f72672f77696b6970656469612f636f6d6d6f6e732f362f36362f5265642d626c61636b5f747265655f696e736572745f636173655f352e706e67)](https://camo.githubusercontent.com/6589a9c75d8e686270a6f2e981c1dd7ea274ce0c68064861e205ee259dc0a164/68747470733a2f2f75706c6f61642e77696b696d656469612e6f72672f77696b6970656469612f636f6d6d6f6e732f362f36362f5265642d626c61636b5f747265655f696e736572745f636173655f352e706e67) 　　**情形5:** 父节点P是红色而叔父节点U是黑色或缺少，并且新节点N是其父节点P的右子节点而父节点P又是其父节点的左子节点。在这种情形下，我们进行一次左旋转调换新节点和其父节点的角色; 接着，我们按**情形4**处理以前的父节点P以解决仍然失效的性质4。注意这个改变会导致某些路径通过它们以前不通过的新节点N（比如图中1号叶子节点）或不通过节点P（比如图中3号叶子节点），但由于这两个节点都是红色的，所以性质5仍有效。 -![情形4 示意图](https://upload.wikimedia.org/wikipedia/commons/5/56/Red-black_tree_insert_case_4.png) +[![情形4 示意图](https://camo.githubusercontent.com/b3607bd07118241f8f16f2cc07603423bb3bfcae73c203434d0633a4bd3cdd60/68747470733a2f2f75706c6f61642e77696b696d656469612e6f72672f77696b6970656469612f636f6d6d6f6e732f352f35362f5265642d626c61636b5f747265655f696e736572745f636173655f342e706e67)](https://camo.githubusercontent.com/b3607bd07118241f8f16f2cc07603423bb3bfcae73c203434d0633a4bd3cdd60/68747470733a2f2f75706c6f61642e77696b696d656469612e6f72672f77696b6970656469612f636f6d6d6f6e732f352f35362f5265642d626c61636b5f747265655f696e736572745f636173655f342e706e67) 　　**注: 插入实际上是原地算法，因为上述所有调用都使用了尾部递归。** @@ -568,25 +565,19 @@ AVL 树和二叉查找树的删除操作情况一致，都分为四种情况： 　　**情形2:** S是红色。在这种情形下我们在N的父亲上做左旋转，把红色兄弟转换成N的祖父，我们接着对调N的父亲和祖父的颜色。完成这两个操作后，尽管所有路径上黑色节点的数目没有改变，但现在N有了一个黑色的兄弟和一个红色的父亲（它的新兄弟是黑色因为它是红色S的一个儿子），所以我们可以接下去按**情形4**、**情形5**或**情形6**来处理。 -![情形2 示意图](https://upload.wikimedia.org/wikipedia/commons/3/39/Red-black_tree_delete_case_2.png) - - +[![情形2 示意图](https://camo.githubusercontent.com/e91996abee214a19db6edb37d13288a5bcb97b3decd3c51e96a55b139c965da7/68747470733a2f2f75706c6f61642e77696b696d656469612e6f72672f77696b6970656469612f636f6d6d6f6e732f332f33392f5265642d626c61636b5f747265655f64656c6574655f636173655f322e706e67)](https://camo.githubusercontent.com/e91996abee214a19db6edb37d13288a5bcb97b3decd3c51e96a55b139c965da7/68747470733a2f2f75706c6f61642e77696b696d656469612e6f72672f77696b6970656469612f636f6d6d6f6e732f332f33392f5265642d626c61636b5f747265655f64656c6574655f636173655f322e706e67) 　　**情形3:** N的父亲、S和S的儿子都是黑色的。在这种情形下，我们简单的重绘S为红色。结果是通过S的所有路径，它们就是以前*不*通过N的那些路径，都少了一个黑色节点。因为删除N的初始的父亲使通过N的所有路径少了一个黑色节点，这使事情都平衡了起来。但是，通过P的所有路径现在比不通过P的路径少了一个黑色节点，所以仍然违反性质5。要修正这个问题，我们要从**情形1**开始，在P上做重新平衡处理。 -![情形3 示意图](https://upload.wikimedia.org/wikipedia/commons/c/c7/Red-black_tree_delete_case_3.png) - - +[![情形3 示意图](https://camo.githubusercontent.com/11d59ddec9bba2b6f6fd119e7c3a3504ab8a14a99fbf48c7f75c5016f0bd8fd7/68747470733a2f2f75706c6f61642e77696b696d656469612e6f72672f77696b6970656469612f636f6d6d6f6e732f632f63372f5265642d626c61636b5f747265655f64656c6574655f636173655f332e706e67)](https://camo.githubusercontent.com/11d59ddec9bba2b6f6fd119e7c3a3504ab8a14a99fbf48c7f75c5016f0bd8fd7/68747470733a2f2f75706c6f61642e77696b696d656469612e6f72672f77696b6970656469612f636f6d6d6f6e732f632f63372f5265642d626c61636b5f747265655f64656c6574655f636173655f332e706e67) 　　**情形4:** S和S的儿子都是黑色，但是N的父亲是红色。在这种情形下，我们简单的交换N的兄弟和父亲的颜色。这不影响不通过N的路径的黑色节点的数目，但是它在通过N的路径上对黑色节点数目增加了一，添补了在这些路径上删除的黑色节点。 -![情形4 示意图](https://upload.wikimedia.org/wikipedia/commons/d/d7/Red-black_tree_delete_case_4.png) - - +[![情形4 示意图](https://camo.githubusercontent.com/6b648caabf6e7caf7b156f24482634f5e77bbd1c565a77b0be67f0af6eeaa350/68747470733a2f2f75706c6f61642e77696b696d656469612e6f72672f77696b6970656469612f636f6d6d6f6e732f642f64372f5265642d626c61636b5f747265655f64656c6574655f636173655f342e706e67)](https://camo.githubusercontent.com/6b648caabf6e7caf7b156f24482634f5e77bbd1c565a77b0be67f0af6eeaa350/68747470733a2f2f75706c6f61642e77696b696d656469612e6f72672f77696b6970656469612f636f6d6d6f6e732f642f64372f5265642d626c61636b5f747265655f64656c6574655f636173655f342e706e67) 　　**情形5:** S是黑色，S的左儿子是红色，S的右儿子是黑色，而N是它父亲的左儿子。在这种情形下我们在S上做右旋转，这样S的左儿子成为S的父亲和N的新兄弟。我们接着交换S和它的新父亲的颜色。所有路径仍有同样数目的黑色节点，但是现在N有了一个黑色兄弟，他的右儿子是红色的，所以我们进入了**情形6**。N和它的父亲都不受这个变换的影响。 -![情形5 示意图](https://upload.wikimedia.org/wikipedia/commons/3/30/Red-black_tree_delete_case_5.png) +[![情形5 示意图](https://camo.githubusercontent.com/ddae9cf5c9133ed4f91d7146251d39f9837dfe9dd2342eb2c8973fea3fe0694e/68747470733a2f2f75706c6f61642e77696b696d656469612e6f72672f77696b6970656469612f636f6d6d6f6e732f332f33302f5265642d626c61636b5f747265655f64656c6574655f636173655f352e706e67)](https://camo.githubusercontent.com/ddae9cf5c9133ed4f91d7146251d39f9837dfe9dd2342eb2c8973fea3fe0694e/68747470733a2f2f75706c6f61642e77696b696d656469612e6f72672f77696b6970656469612f636f6d6d6f6e732f332f33302f5265642d626c61636b5f747265655f64656c6574655f636173655f352e706e67) 　　**情形6:** S是黑色，S的右儿子是红色，而N是它父亲的左儿子。在这种情形下我们在N的父亲上做左旋转，这样S成为N的父亲（P）和S的右儿子的父亲。我们接着交换N的父亲和S的颜色，并使S的右儿子为黑色。子树在它的根上的仍是同样的颜色，所以性质3没有被违反。但是，N现在增加了一个黑色祖先: 要么N的父亲变成黑色，要么它是黑色而S被增加为一个黑色祖父。所以，通过N的路径都增加了一个黑色节点。 @@ -597,254 +588,26 @@ AVL 树和二叉查找树的删除操作情况一致，都分为四种情况： 　　在任何情况下，在这些路径上的黑色节点数目都没有改变。所以我们恢复了性质4。在示意图中的白色节点可以是红色或黑色，但是在变换前后都必须指定相同的颜色。 -![情形6 示意图](https://upload.wikimedia.org/wikipedia/commons/3/31/Red-black_tree_delete_case_6.png) - - +[![情形6 示意图](https://camo.githubusercontent.com/c9e3fda5572806f8c15531bcb0dd91fee28010f68bfdf9a1a7c7869964d81cd3/68747470733a2f2f75706c6f61642e77696b696d656469612e6f72672f77696b6970656469612f636f6d6d6f6e732f332f33312f5265642d626c61636b5f747265655f64656c6574655f636173655f362e706e67)](https://camo.githubusercontent.com/c9e3fda5572806f8c15531bcb0dd91fee28010f68bfdf9a1a7c7869964d81cd3/68747470733a2f2f75706c6f61642e77696b696d656469612e6f72672f77696b6970656469612f636f6d6d6f6e732f332f33312f5265642d626c61636b5f747265655f64656c6574655f636173655f362e706e67) ## 哈夫曼树 +> “不假，最近有没有 Java 学习资源？” +> +> “有的，我压缩后发到微信群里哈”（加入 JavaKeeper 交流群各种学习资料哈） +我们都有过对文件进行压缩、解压的操作，压缩而不出错是怎么做到的呢？压缩文本的时候其实是对文本进行了一次重新编码，减少了不必要的空间，而哈夫曼编码算是一种最基本的压缩编码方式。 -## B树 - -B树也是一种用于查找的平衡树，但是它不是二叉树。 - -B-tree 树即 B 树**，B即Balanced，平衡的意思。因为B树的原英文名称为B-tree，而国内很多人喜欢把B-tree译作B-树，其实，这是个非常不好的直译，很容易让人产生误解。如可能会以为B-树是一种树，而B树又是另一种树。而事实上是，B-tree就是指的B树**。特此说明。 - - - -**B树的定义：**B树（B-tree）是一种树状数据结构，能够用来存储排序后的数据。这种数据结构能够让查找数据、循序存取、插入数据及删除的动作，都在对数时间内完成。 - -B树，概括来说是一个一般化的二叉查找树，可以拥有多于2个子节点。与自平衡二叉查找树不同，B-树为系统最优化大块数据的读和写操作。B-tree算法减少定位记录时所经历的中间过程，从而加快存取速度。这种数据结构常被应用在数据库和文件系统的实作上。 - -在B树中查找给定关键字的方法是，首先把根结点取来，在根结点所包含的关键字K1,…,Kn查找给定的关键字（可用顺序查找或二分查找法），若找到等于给定值的关键字，则查找成功；否则，一定可以确定要查找的关键字在Ki与Ki+1之间，Pi为指向子树根节点的指针，此时取指针Pi所指的结点继续查找，直至找到，或指针Pi为空时查找失败。 - -B树作为一种多路搜索树（并不是二叉的）： - -B树的性质 - -M为树的阶数，B-树或为空树，否则满足下列条件： - -1. 定义任意非叶子结点最多只有M个儿子；且M>2； -2. 根结点的儿子数为[2, M]； -3. 除根结点以外的非叶子结点的儿子数为[M/2, M]； -4. 每个结点存放至少M/2-1（取上整）和至多M-1个关键字；（至少2个关键字） -5. 非叶子结点的关键字个数=指向儿子的指针个数-1； -6. 非叶子结点的关键字：K[1], K[2], …, K[M-1]；且K[i] < K[i+1]； -7. 非叶子结点的指针：P[1], P[2], …, P[M]；其中P[1]指向关键字小于K[1]的子树，P[M]指向关键字大于K[M-1]的子树，其它P[i]指向关键字属于(K[i-1], K[i])的子树； -8. 所有叶子结点位于同一层； - -​ 如下图为一个M=3的B树示例： - -![img](https://camo.githubusercontent.com/dfde3dddb226018bc185288ac84dc380f77be859/687474703a2f2f696d672e626c6f672e6373646e2e6e65742f3230313630363132313135353530313737) - -　　B树创建的示意图： - -![img](https://files.cnblogs.com/yangecnu/btreebuild.gif) - - - -B-树的搜索，从根结点开始，对结点内的关键字（有序）序列进行二分查找，如果命中则结束，否则进入查询关键字所属范围的儿子结点；重复，直到所对应的儿子指针为空，或已经是叶子结点。 - - - -# 平衡多路查找树（B-Tree） - -B-Tree是为磁盘等外存储设备设计的一种平衡查找树。因此在讲B-Tree之前先了解下磁盘的相关知识。 - -系统从磁盘读取数据到内存时是以磁盘块（block）为基本单位的，位于同一个磁盘块中的数据会被一次性读取出来，而不是需要什么取什么。 - -InnoDB存储引擎中有页（Page）的概念，页是其磁盘管理的最小单位。InnoDB存储引擎中默认每个页的大小为16KB，可通过参数innodb_page_size将页的大小设置为4K、8K、16K，在[MySQL](http://lib.csdn.net/base/mysql)中可通过如下命令查看页的大小： - -``` -mysql> show variables like 'innodb_page_size'; -``` - -- 1 - -- 1 - -而系统一个磁盘块的存储空间往往没有这么大，因此InnoDB每次申请磁盘空间时都会是若干地址连续磁盘块来达到页的大小16KB。InnoDB在把磁盘数据读入到磁盘时会以页为基本单位，在查询数据时如果一个页中的每条数据都能有助于定位数据记录的位置，这将会减少磁盘I/O次数，提高查询效率。 - -B-Tree结构的数据可以让系统高效的找到数据所在的磁盘块。为了描述B-Tree，首先定义一条记录为一个二元组[key, data] ，key为记录的键值，对应表中的主键值，data为一行记录中除主键外的数据。对于不同的记录，key值互不相同。 - -一棵m阶的B-Tree有如下特性：  -1. 每个节点最多有m个孩子。  -2. 除了根节点和叶子节点外，其它每个节点至少有Ceil(m/2)个孩子（其中ceil(x)是一个取上限的函数）。  -3. 若根节点不是叶子节点，则至少有2个孩子  -4. 所有叶子节点都在同一层，且不包含其它关键字信息  -5. 每个非终端节点包含n个关键字信息（P0,P1,…Pn, k1,…kn）  -6. 关键字的个数n满足：ceil(m/2)-1 <= n <= m-1  -7. ki(i=1,…n)为关键字，且关键字升序排序。  -8. Pi(i=1,…n)为指向子树根节点的指针。P(i-1)指向的子树的所有节点关键字均小于ki，但都大于k(i-1) - -B-Tree中的每个节点根据实际情况可以包含大量的关键字信息和分支，如下图所示为一个3阶的B-Tree： -![索引](https://img-blog.csdn.net/20160202204827368) - -每个节点占用一个盘块的磁盘空间，一个节点上有两个升序排序的关键字和三个指向子树根节点的指针，指针存储的是子节点所在磁盘块的地址。两个关键词划分成的三个范围域对应三个指针指向的子树的数据的范围域。以根节点为例，关键字为17和35，P1指针指向的子树的数据范围为小于17，P2指针指向的子树的数据范围为17~35，P3指针指向的子树的数据范围为大于35。 - -模拟查找关键字29的过程： - -1. 根据根节点找到磁盘块1，读入内存。【磁盘I/O操作第1次】 -2. 比较关键字29在区间（17,35），找到磁盘块1的指针P2。 -3. 根据P2指针找到磁盘块3，读入内存。【磁盘I/O操作第2次】 -4. 比较关键字29在区间（26,30），找到磁盘块3的指针P2。 -5. 根据P2指针找到磁盘块8，读入内存。【磁盘I/O操作第3次】 -6. 在磁盘块8中的关键字列表中找到关键字29。 - -分析上面过程，发现需要3次磁盘I/O操作，和3次内存查找操作。由于内存中的关键字是一个有序表结构，可以利用二分法查找提高效率。而3次磁盘I/O操作是影响整个B-Tree查找效率的决定因素。B-Tree相对于AVLTree缩减了节点个数，使每次磁盘I/O取到内存的数据都发挥了作用，从而提高了查询效率。 - - - -## B+树 - -B+树是B树的变体，也是一种多路搜索树： - -　　1) 其定义基本与B-树相同，除了： - -　　2) 非叶子结点的子树指针与关键字个数相同； - -　　3) 非叶子结点的子树指针P[i]，指向关键字值属于[K[i], K[i+1])的子树（B-树是开区间）； - -　　4) 为所有叶子结点增加一个链指针； - -　　5) 所有关键字都在叶子结点出现； - -　　下图为M=3的B+树的示意图： - -![img](http://p.blog.csdn.net/images/p_blog_csdn_net/manesking/5.JPG) - -　　B+树的搜索与B树也基本相同，区别是B+树只有达到叶子结点才命中（B树可以在非叶子结点命中），其性能也等价于在关键字全集做一次二分查找； - -　　**B+的性质：** - -　　1.所有关键字都出现在叶子结点的链表中（稠密索引），且链表中的关键字恰好是有序的； - -　　2.不可能在非叶子结点命中； - -　　3.非叶子结点相当于是叶子结点的索引（稀疏索引），叶子结点相当于是存储（关键字）数据的数据层； - -　　4.更适合文件索引系统。 - -　　下面为一个B+树创建的示意图： - -![img](https://files.cnblogs.com/yangecnu/Bplustreebuild.gif) - -B+Tree是在B-Tree基础上的一种优化，使其更适合实现外存储索引结构，InnoDB存储引擎就是用B+Tree实现其索引结构。 - -从上一节中的B-Tree结构图中可以看到每个节点中不仅包含数据的key值，还有data值。而每一个页的存储空间是有限的，如果data数据较大时将会导致每个节点（即一个页）能存储的key的数量很小，当存储的数据量很大时同样会导致B-Tree的深度较大，增大查询时的磁盘I/O次数，进而影响查询效率。在B+Tree中，所有数据记录节点都是按照键值大小顺序存放在同一层的叶子节点上，而非叶子节点上只存储key值信息，这样可以大大加大每个节点存储的key值数量，降低B+Tree的高度。 - -B+Tree相对于B-Tree有几点不同： - -1. 非叶子节点只存储键值信息。 -2. 所有叶子节点之间都有一个链指针。 -3. 数据记录都存放在叶子节点中。 - -将上一节中的B-Tree优化，由于B+Tree的非叶子节点只存储键值信息，假设每个磁盘块能存储4个键值及指针信息，则变成B+Tree后其结构如下图所示： -![索引](https://img-blog.csdn.net/20160202205105560) - -通常在B+Tree上有两个头指针，一个指向根节点，另一个指向关键字最小的叶子节点，而且所有叶子节点（即数据节点）之间是一种链式环结构。因此可以对B+Tree进行两种查找运算：一种是对于主键的范围查找和分页查找，另一种是从根节点开始，进行随机查找。 - -可能上面例子中只有22条数据记录，看不出B+Tree的优点，下面做一个推算： - -InnoDB存储引擎中页的大小为16KB，一般表的主键类型为INT（占用4个字节）或BIGINT（占用8个字节），指针类型也一般为4或8个字节，也就是说一个页（B+Tree中的一个节点）中大概存储16KB/(8B+8B)=1K个键值（因为是估值，为方便计算，这里的K取值为〖10〗^3）。也就是说一个深度为3的B+Tree索引可以维护10^3 * 10^3 * 10^3 = 10亿 条记录。 - -实际情况中每个节点可能不能填充满，因此在数据库中，B+Tree的高度一般都在2~4层。[mysql](http://lib.csdn.net/base/mysql)的InnoDB存储引擎在设计时是将根节点常驻内存的，也就是说查找某一键值的行记录时最多只需要1~3次磁盘I/O操作。 - -数据库中的B+Tree索引可以分为聚集索引（clustered index）和辅助索引（secondary index）。上面的B+Tree示例图在数据库中的实现即为聚集索引，聚集索引的B+Tree中的叶子节点存放的是整张表的行记录数据。辅助索引与聚集索引的区别在于辅助索引的叶子节点并不包含行记录的全部数据，而是存储相应行数据的聚集索引键，即主键。当通过辅助索引来查询数据时，InnoDB存储引擎会遍历辅助索引找到主键，然后再通过主键在聚集索引中找到完整的行记录数据。 - - - -## B*树 - -　　B*树是B+树的变体，在B+树的非根和非叶子结点再增加指向兄弟的指针，将结点的最低利用率从1/2提高到2/3。 - -　　B*树如下图所示： - -![img](http://p.blog.csdn.net/images/p_blog_csdn_net/manesking/6.JPG) - -　　B*树定义了非叶子结点关键字个数至少为(2/3)*M，即块的最低使用率为2/3（代替B+树的1/2）； - -　　B+树的分裂：当一个结点满时，分配一个新的结点，并将原结点中1/2的数据复制到新结点，最后在父结点中增加新结点的指针；B+树的分裂只影响原结点和父结点，而不会影响兄弟结点，所以它不需要指向兄弟的指针； - -　　B*树的分裂：当一个结点满时，如果它的下一个兄弟结点未满，那么将一部分数据移到兄弟结点中，再在原结点插入关键字，最后修改父结点中兄弟结点的关键字（因为兄弟结点的关键字范围改变了）；如果兄弟也满了，则在原结点与兄弟结点之间增加新结点，并各复制1/3的数据到新结点，最后在父结点增加新结点的指针； - -　　所以，B*树分配新结点的概率比B+树要低，空间使用率更高。 - - - - - - - - - -## Trie树 - -　　Tire树称为字典树，又称单词查找树，Trie树，是一种树形结构，是一种哈希树的变种。典型应用是用于统计，排序和保存大量的字符串（但不仅限于字符串），所以经常被搜索引擎系统用于文本词频统计。**它的优点是：利用字符串的公共前缀来减少查询时间，最大限度地减少无谓的字符串比较，查询效率比哈希树高。**　 - -　　**Tire树的三个基本性质：** - -　　1) 根节点不包含字符，除根节点外每一个节点都只包含一个字符； - -　　2) 从根节点到某一节点，路径上经过的字符连接起来，为该节点对应的字符串； - -　　3) 每个节点的所有子节点包含的字符都不相同。 - -　　**Tire树的应用：** - -　　1) 串的快速检索 - -　　给出N个单词组成的熟词表，以及一篇全用小写英文书写的文章，请你按最早出现的顺序写出所有不在熟词表中的生词。 - -在这道题中，我们可以用数组枚举，用哈希，用字典树，先把熟词建一棵树，然后读入文章进行比较，这种方法效率是比较高的。 - -　　2) “串”排序 - -　　给定N个互不相同的仅由一个单词构成的英文名，让你将他们按字典序从小到大输出。用字典树进行排序，采用数组的方式创建字典树，这棵树的每个结点的所有儿子很显然地按照其字母大小排序。对这棵树进行先序遍历即可。 - -　　3) 最长公共前缀 - -　　对所有串建立字典树，对于两个串的最长公共前缀的长度即他们所在的结点的公共祖先个数，于是，问题就转化为求公共祖先的问题。 - - - - - - - - - - +发明这种压缩编码方式的数学家叫哈夫曼，所以就把他在编码中用到的特殊的二叉树称之为哈弗曼树。 ## leetcode - - - - - - - - - - - - - - - - - - - - - - ## 参考与感谢 https://www.cnblogs.com/maybe2030/p/4732377.html +[www.bootwiki.com](http://www.bootwiki.com/) + https://www.jianshu.com/p/45661b029292 https://charlesliuyx.github.io/2018/10/22/[直观算法]树的基本操作/ diff --git a/docs/data-structure-algorithms/data-structure/HashTable.md b/docs/data-structure-algorithms/data-structure/HashTable.md new file mode 100755 index 0000000000..edb7eb82ab --- /dev/null +++ b/docs/data-structure-algorithms/data-structure/HashTable.md @@ -0,0 +1,168 @@ +--- +title: 深入解析哈希表：从散列冲突到算法应用全景解读 +date: 2022-06-09 +tags: + - data-structure + - HashTable +categories: data-structure +--- + +![](https://images.unsplash.com/photo-1586370740632-f910eb4ad077?q=80&w=3208&auto=format&fit=crop&ixlib=rb-4.1.0&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D) + +## 一、散列冲突的本质与解决方案 + +哈希表作为数据结构的核心组件，其灵魂在于通过哈希函数实现 $(1)$ 时间复杂度的数据存取。但正如硬币的两面，哈希算法在带来高效存取的同时，也面临着不可避免的**散列冲突**问题——不同的输入值经过哈希运算后映射到同一存储位置的现象。 + +### 1.1 开放寻址法：空间换时间的博弈 + +**典型代表**：Java 的 ThreadLocalMap + +开放寻址法采用"线性探测+数组存储"的经典组合，当冲突发生时，通过系统性的探测策略（线性探测/二次探测/双重哈希）在数组中寻找下一个可用槽位。这种方案具有三大显著优势： + +- **缓存友好性**：数据连续存储在数组中，有效利用CPU缓存行预取机制 +- **序列化简单**：无需处理链表指针等复杂内存结构 +- **空间紧凑**：内存分配完全可控，无动态内存开销 + +但硬币的另一面是： + +- **删除复杂度**：需引入墓碑标记（TOMBSTONE）处理逻辑 +- **负载因子限制**：建议阈值0.7以下，否则探测次数指数级增长 +- **内存浪费**：动态扩容时旧数组需要保留至数据迁移完成 + +```Java +// 线性探测的典型实现片段 +int index = hash(key); +while (table[index] != null) { + if (table[index].key.equals(key)) break; + index = (index + 1) % capacity; +} +``` + +### 1.2 链表法：时间与空间的动态平衡 + +**典型代表**：Java的LinkedHashMap + +链表法采用"数组+链表/树"的复合结构，每个桶位维护一个动态数据结构。其核心优势体现在： + +- **高负载容忍**：允许负载因子突破1.0（Java HashMap默认0.75） +- **内存利用率**：按需创建节点，避免空槽浪费 +- **结构灵活性**：可升级为红黑树（Java8+）应对哈希碰撞攻击 + +但需要注意： + +- **指针开销**：每个节点多消耗4-8字节指针空间 +- **缓存不友好**：节点内存地址离散影响访问局部性 +- **小对象劣势**：当存储值小于指针大小时内存利用率降低 + +```Java +// 树化转换阈值定义（Java HashMap） +static final int TREEIFY_THRESHOLD = 8; +static final int UNTREEIFY_THRESHOLD = 6; +``` + + + +## 二、哈希算法：从理论到工程实践 + +哈希算法作为数字世界的"指纹生成器"，必须满足四大黄金准则： + +1. **不可逆性**：哈希值到原文的逆向推导在计算上不可行 +2. **雪崩效应**：微小的输入变化导致输出剧变 +3. **低碰撞率**：不同输入的哈希相同概率趋近于零 +4. **高效计算**：处理海量数据时仍保持线性时间复杂度 + +### 2.1 七大核心应用场景解析 + +#### 场景1：安全加密（SHA-256示例） + +```Java +MessageDigest md = MessageDigest.getInstance("SHA-256"); +byte[] hashBytes = md.digest("secret".getBytes()); +``` + +#### 场景2：内容寻址存储（IPFS协议） + +通过三级哈希验证确保内容唯一性： + +1. 内容分块哈希 +2. 分块组合哈希 +3. 最终Merkle根哈希 + +#### 场景3：P2P传输校验（BitTorrent协议） + +种子文件包含分片哈希树，下载时逐层校验： + +``` + 分片1(SHA1) → 分片2(SHA1) → ... → 分片N(SHA1) + ↘ ↙ ↘ ↙ + 中间哈希节点 根哈希 +``` + +#### 场景4：高性能散列函数（MurmurHash3） + +针对不同场景的哈希优化： + +- 内存型：CityHash +- 加密型：SipHash +- 流式处理：XXHash + +#### 场景5：会话保持负载均衡 + +```Python +def get_server(client_ip): + hash_val = hashlib.md5(client_ip).hexdigest() + return servers[hash_val % len(servers)] +``` + +#### 场景6：大数据分片处理 + +```SQL +-- 按用户ID哈希分库 +CREATE TABLE user_0 ( + id BIGINT PRIMARY KEY, + ... +) PARTITION BY HASH(id) PARTITIONS 4; +``` + +#### 场景7：一致性哈希分布式存储 + +构建虚拟节点环解决数据倾斜问题： + +``` + NodeA → 1000虚拟节点 +NodeB → 1000虚拟节点 +NodeC → 1000虚拟节点 +``` + + + +## 三、工程实践中的进阶技巧 + +### 3.1 动态扩容策略 + +- 渐进式扩容：避免一次性rehash导致的STW停顿 +- 容量质数选择：降低哈希聚集现象（如Java HashMap使用2^n优化模运算） + +### 3.2 哈希攻击防御 + +- 盐值加密：password_hash(pass,PASSWORDBCRYPT,[′salt′=>*p**a**ss*,*P**A**SS**W**OR**D**B**CR**Y**PT*,[′*s**a**l**t*′=>salt]) +- 密钥哈希：HMAC-SHA256(secretKey, message) + +### 3.3 性能优化指标 + +| 指标 | 开放寻址法 | 链表法 | +| ------------ | ---------- | ------ | +| 平均查询时间 | O(1/(1-α)) | O(α) | +| 内存利用率 | 60-70% | 80-90% | +| 最大负载因子 | 0.7 | 1.0+ | +| 并发修改支持 | 困难 | 较容易 | + + + +## 四、未来演进方向 + +- **量子安全哈希**：抗量子计算的Lattice-based哈希算法 +- **同态哈希**：支持密文域计算的哈希方案 +- **AI驱动哈希**：基于神经网络的自适应哈希函数 + +哈希表及其相关算法作为计算机科学的基石，在从单机系统到云原生架构的演进历程中持续发挥着关键作用。理解其核心原理并掌握工程化实践技巧，将帮助开发者在高并发、分布式场景下构建出更健壮、更高效的系统。 diff --git a/docs/data-structure-algorithms/data-structure/Linked-List.md b/docs/data-structure-algorithms/data-structure/Linked-List.md new file mode 100644 index 0000000000..ba18ccf934 --- /dev/null +++ b/docs/data-structure-algorithms/data-structure/Linked-List.md @@ -0,0 +1,336 @@ +--- +title: 链表 +date: 2022-06-08 +tags: + - LikedList +categories: data-structure +--- + +# 链表 + +与数组相似，链表也是一种`线性`数据结构。 + +链表是一系列的存储数据元素的单元通过指针串接起来形成的，因此每个单元至少有两个域，一个域用于数据元素的存储，另一个域是指向其他单元的指针。这里具有一个数据域和多个指针域的存储单元通常称为**结点**（node）。 + + + +## 单链表 + +![single-linkedlist](https://img.starfish.ink/data-structure/single-linkedlist.png) + +一种最简单的结点结构如上图所示，它是构成单链表的基本结点结构。在结点中数据域用来存储数据元素，指针域用于指向下一个具有相同结构的结点。 + +单链表中的每个结点不仅包含值，还包含链接到下一个结点的`引用字段`。通过这种方式，单链表将所有结点按顺序组织起来。 + +![single-linkedlist-node](https://img.starfish.ink/data-structure/single-linkedlist-node.png) + +链表的第一个结点和最后一个结点，分别称为链表的**首结点**和**尾结点**。尾结点的特征是其 next 引用为空（null）。链表中每个结点的 next 引用都相当于一个指针，指向另一个结点，借助这些 next 引用，我们可以从链表的首结点移动到尾结点。如此定义的结点就称为**单链表**（single linked list）。 + +上图蓝色箭头显示单个链接列表中的结点是如何组合在一起的。 + +在单链表中通常使用 head 引用来指向链表的首结点，由 head 引用可以完成对整个链表中所有节点的访问。有时也可以根据需要使用指向尾结点的 tail 引用来方便某些操作的实现。 + +在单链表结构中还需要注意的一点是，由于每个结点的数据域都是一个 Object 类的对象，因此，每个数据元素并非真正如图中那样，而是在结点中的数据域通过一个 Object 类的对象引用来指向数据元素的。 + +与数组类似，单链表中的结点也具有一个线性次序，即如果结点 P 的 next 引用指向结点 S，则 P 就是 S 的**直接前驱**，S 是 P 的**直接后续**。单链表的一个重要特性就是只能通过前驱结点找到后续结点，而无法从后续结点找到前驱结点。 + +接着我们来看下单链表的 CRUD： + +> [707. 设计链表](https://leetcode.cn/problems/design-linked-list/) 搞定一题 + +以下是单链表中结点的典型定义，`值 + 链接到下一个元素的指针`： + +```java +// Definition for singly-linked list. +public class SinglyListNode { + int val; + SinglyListNode next; + SinglyListNode(int x) { val = x; } +} +``` + +有了结点，还需要一个“链” 把所有结点串起来 + +```java +class MyLinkedList { + private SinglyListNode head; + /** Initialize your data structure here. */ + public MyLinkedList() { + head = null; + } +} +``` + + + +### 查找 + +与数组不同，我们无法在常量时间内访问单链表中的随机元素。 如果我们想要获得第 i 个元素，我们必须从头结点逐个遍历。 我们按索引来访问元素平均要花费 $O(N)$ 时间，其中 N 是链表的长度。 + +使用 Java 语言实现整个过程的关键语句是： + +```java +/** Get the value of the index-th node in the linked list. If the index is invalid, return -1. */ +public int get(int index) { + // if index is invalid + if (index < 0 || index >= size) return -1; + + ListNode curr = head; + // index steps needed + // to move from sentinel node to wanted index + for(int i = 0; i < index + 1; ++i) { + curr = curr.next; + } + return curr.val; +} +``` + + + +### 添加 + +单链表中数据元素的插入，是通过在链表中插入数据元素所属的结点来完成的。对于链表的不同位置，插入的过程会有细微的差别。 + +![single-linkedlist-add](https://img.starfish.ink/data-structure/single-linkedlist-add.png) + +除了单链表的首结点由于没有直接前驱结点，所以可以直接在首结点之前插入一个新的结点之外，在单链表中的其他任何位置插入一个新结点时，都只能是在已知某个特定结点引用的基础上在其后面插入一个新结点。并且在已知单链表中某个结点引用的基础上，完成结点的插入操作需要的时间是 $O(1)$。 + +> 思考：如果是带头结点的单链表进行插入操作，是什么样子呢？ + +```java +//最外层有个链表长度，便于我们头插和尾插操作 +int size; + +public void addAtHead(int val) { + addAtIndex(0, val); +} + +//尾插就是从最后一个 +public void addAtTail(int val) { + addAtIndex(size, val); +} + +public void addAtIndex(int index, int val) { + + if (index > size) return; + + if (index < 0) index = 0; + + ++size; + // find predecessor of the node to be added + ListNode pred = head; + for(int i = 0; i < index; ++i) { + pred = pred.next; + } + + // node to be added + ListNode toAdd = new ListNode(val); + // insertion itself + toAdd.next = pred.next; + pred.next = toAdd; +} +``` + + + +### 删除 + +类似的，在单链表中数据元素的删除也是通过结点的删除来完成的。在链表的不同位置删除结点，其操作过程也会有一些差别。 + +![single-linkedlist-del](http://img.starfish.ink/data-structure/single-linkedlist-del.png) + +在单链表中删除一个结点时，除首结点外都必须知道该结点的直接前驱结点的引用。并且在已知单链表中某个结点引用的基础上，完成其后续结点的删除操作需要的时间是 $O(1)$。 + +> 在使用单链表实现线性表的时候，为了使程序更加简洁，我们通常在单链表的最前面添加一个**哑元结点**，也称为头结点。在头结点中不存储任何实质的数据对象，其 next 域指向线性表中 0 号元素所在的结点，头结点的引入可以使线性表运算中的一些边界条件更容易处理。 +> +> 对于任何基于序号的插入、删除，以及任何基于数据元素所在结点的前面或后面的插入、删除，在带头结点的单链表中均可转化为在某个特定结点之后完成结点的插入、删除，而不用考虑插入、删除是在链表的首部、中间、还是尾部等不同情况。 + +![](http://img.starfish.ink/data-structure/single-linkedlist-head.png) + + + +```java + public void deleteAtIndex(int index) { + // if the index is invalid, do nothing + if (index < 0 || index >= size) return; + + size--; + // find predecessor of the node to be deleted + ListNode pred = head; + for(int i = 0; i < index; ++i) { + pred = pred.next; + } + + // delete pred.next + pred.next = pred.next.next; + } +``` + + + +## 双向链表 + +单链表的一个优点是结构简单，但是它也有一个缺点，即在单链表中只能通过一个结点的引用访问其后续结点，而无法直接访问其前驱结点，要在单链表中找到某个结点的前驱结点，必须从链表的首结点出发依次向后寻找，但是需要 $Ο(n)$ 时间。 + +所以我们在单链表结点结构中新增加一个域，该域用于指向结点的直接前驱结点。 + +![](http://img.starfish.ink/data-structure/doule-linkedlist-node.png) + +双向链表是通过上述定义的结点使用 pre 以及 next 域依次串联在一起而形成的。一个双向链表的结构如下图所示。 + +![](http://img.starfish.ink/data-structure/doule-linkedlist.png) + +接着我们来看下双向链表的 CRUD： + +以下是双链表中结点的典型定义： + +```java +public class ListNode { + int val; + ListNode next; + ListNode prev; + ListNode(int x) { val = x; } +} + +class MyLinkedList { + int size; + // sentinel nodes as pseudo-head and pseudo-tail + ListNode head, tail; + public MyLinkedList() { + size = 0; + head = new ListNode(0); + tail = new ListNode(0); + head.next = tail; + tail.prev = head; + } +``` + +### 查找 + +在双向链表中进行查找与在单链表中类似，只不过在双向链表中查找操作可以从链表的首结点开始，也可以从尾结点开始，但是需要的时间和在单链表中一样。 + +```java + /** Get the value of the index-th node in the linked list. If the index is invalid, return -1. */ + public int get(int index) { + if (index < 0 || index >= size) return -1; + + ListNode curr = head; + if (index + 1 < size - index) + for(int i = 0; i < index + 1; ++i) { + curr = curr.next; + } + else { + curr = tail; + for(int i = 0; i < size - index; ++i) { + curr = curr.prev; + } + } + + return curr.val; + } +``` + + + +### 添加 + +单链表的插入操作，除了首结点之外必须在某个已知结点后面进行，而在双向链表中插入操作在一个已知的结点之前或之后都可以进行，如下表示在结点 11 之前 插入 9。 + +![](http://img.starfish.ink/data-structure/double-linkedlist-add.png) + +使用 Java 语言实现整个过程的关键语句是 + +```java + + public void addAtHead(int val) { + ListNode pred = head, succ = head.next; + + ++size; + ListNode toAdd = new ListNode(val); + toAdd.prev = pred; + toAdd.next = succ; + pred.next = toAdd; + succ.prev = toAdd; + } + + /** Append a node of value val to the last element of the linked list. */ + public void addAtTail(int val) { + ListNode succ = tail, pred = tail.prev; + + ++size; + ListNode toAdd = new ListNode(val); + toAdd.prev = pred; + toAdd.next = succ; + pred.next = toAdd; + succ.prev = toAdd; + } + + public void addAtIndex(int index, int val) { + if (index > size) return; + + if (index < 0) index = 0; + + ListNode pred, succ; + if (index < size - index) { + pred = head; + for(int i = 0; i < index; ++i) pred = pred.next; + succ = pred.next; + } + else { + succ = tail; + for (int i = 0; i < size - index; ++i) succ = succ.prev; + pred = succ.prev; + } + + // insertion itself + ++size; + ListNode toAdd = new ListNode(val); + toAdd.prev = pred; + toAdd.next = succ; + pred.next = toAdd; + succ.prev = toAdd; + } +``` + +在结点 p 之后插入一个新结点的操作与上述操作对称，这里不再赘述。 + +插入操作除了上述情况，还可以在双向链表的首结点之前、双向链表的尾结点之后进行，此时插入操作与上述插入操作相比更为简单。 + +### 删除 + +单链表的删除操作，除了首结点之外必须在知道待删结点的前驱结点的基础上才能进行，而在双向链表中在已知某个结点引用的前提下，可以完成该结点自身的删除。如下表示删除 16 的过程。 + +![](http://img.starfish.ink/data-structure/double-linkedlist-del.png) + +```java + /** Delete the index-th node in the linked list, if the index is valid. */ + public void deleteAtIndex(int index) { + if (index < 0 || index >= size) return; + + ListNode pred, succ; + if (index < size - index) { + pred = head; + for(int i = 0; i < index; ++i) pred = pred.next; + succ = pred.next.next; + } + else { + succ = tail; + for (int i = 0; i < size - index - 1; ++i) succ = succ.prev; + pred = succ.prev.prev; + } + + // delete pred.next + --size; + pred.next = succ; + succ.prev = pred; + } +} +``` + +对线性表的操作，无非就是排序、加法、减法、反转，说的好像很简单，我们去下一章刷题吧。 + + + +## 参考与感谢 + +- https://aleej.com/2019/09/16/数据结构与算法之美学习笔记 \ No newline at end of file diff --git a/docs/data-structure-algorithms/data-structure/Queue.md b/docs/data-structure-algorithms/data-structure/Queue.md new file mode 100644 index 0000000000..2c91ca275f --- /dev/null +++ b/docs/data-structure-algorithms/data-structure/Queue.md @@ -0,0 +1,464 @@ +--- +title: Queue +date: 2023-05-03 +tags: + - Stack +categories: data-structure +--- + +![](https://img.starfish.ink/data-structure/queue-banner.jpg) + +> 队列（queue）是一种采用先进先出(FIFO)策略的抽象数据结构，它的想法来自于生活中排队的策略。顾客在付款结账的时候，按照到来的先后顺序排队结账，先来的顾客先结账，后来的顾客后结账。 + +## 一、前言 + +队列是只允许在一端进行插入操作，而在另一端进行删除操作的线性表。简称FIFO。允许插入的一端称为队尾，允许删除的一端称为队头。 + +![](https://static001.geekbang.org/resource/image/9e/3e/9eca53f9b557b1213c5d94b94e9dce3e.jpg) + +1. 队列是一个有序列表，可以用数组或是链表来实现。 + +2. 遵循先入先出的原则。即:先存入队列的数据，要先取出。后存入的要后取出 + + + +## 二、基本属性 + +队列的核心理念是：**先进先出**，思考下我们可以对一个队列进行哪些操作 + +![img](https://i04piccdn.sogoucdn.com/9e55578c2bf71dfc) + +普通队列的实现应该支持如下属性和操作： + +- int front：队头 + +- int rear：队尾 +- maxSize：队列最大容量 (可有可无，看你实现的是有界队列还是无界队列) + +- `enqueue(value)`：向队列插入一个元素 +- `dequeue`：从队列中取出一个元素 +- `isEmpty()`：检查队列是否为空 +- `isFull()`：检查队列是否已满 +- `getSize()`：返回队列大小，即数据元素个数 + +抽象成一个接口如下： + +```java +public interface MyQueue { + + /** + * 返回队列大小 + */ + public int getSize(); + + /** + * 判断队列是否为空 + */ + public boolean isEmpty(); + + /** + * 判断队列是否已满 + */ + public boolean isFull(); + + /** + * 数据元素e进入队列 + */ + public void enqueue(Object e); + + /** + * 队首出队元素 + */ + public Object dequeue(); + + /** + * 取队首元素 + */ + public Object peek(); +} +``` + +看到这里，我那刚学 Java 的大一小弟弟指导我说，这可以用数组实现的。。。。。 + + + +## 三、实现 + +### 3.1 基于数组实现的队列 + +- 队列本身是有序列表，若使用数组的结构来存储队列的数据，则队列数组的声明如下图，其中 capacity 是该队列的最大容量。 +- 因为队列的输出、输入是分别从前后端来处理，因此需要两个变量 front 及 rear 分别记录队列前后端的下标， **front 会随着数据输出而改变，而 rear 则是随着数据输入而改变**、 + +- 当我们将数据存入队列时的处理需要有两个步骤: + 1. 将尾指针往后移：`rear+1` , 当 `front == rear` 队列为空 + 2. 若尾指针 rear 小于队列的最大下标 `capacity-1`，则将数据存入 rear 所指的数组元素中，否则无法存入数据，即队列满了。 + +```java +public class MyArrayQueue implements MyQueue { + + private int capacity; // 表示数组的最大容量 + private int front; // 队列头 + private int rear; // 队列尾 + private Object[] arr; // 该数据用于存放数据, 模拟队列 + + // 创建队列的构造器 + public MyArrayQueue(int capacity) { + this.capacity = capacity; + arr = new Object[capacity]; + front = -1; // 指向队列头部，分析出front是指向队列头的前一个位置. + rear = -1; // 指向队列尾，指向队列尾的数据(即就是队列最后一个数据) + } + + public int getSize() { + return rear - front; + } + + public boolean isEmpty() { + return rear == front; + } + + public boolean isFull() { + return rear == capacity - 1; + } + + public void enqueue(Object e) { + if (isFull()) { + System.out.println("队列满，不能加入数据~"); + return; + } + rear++; // 让rear 后移 + arr[rear] = e; + } + + public Object dequeue() { + if (isEmpty()) { + throw new RuntimeException("队列空，不能取数据"); + } + front++; // front后移 + return arr[front]; + } + + public Object peek() { + // 判断 + if (isEmpty()) { + throw new RuntimeException("队列空的，没有数据~~"); + } + return arr[front + 1]; + } +} +``` + +### 3.2 基于链表实现的队列 + +链表实现的队列没有固定的大小限制，可以动态地添加更多的元素。这种方式更加灵活，有效避免了空间浪费，但其操作可能比数组实现稍微复杂一些，因为需要处理节点之间的链接。 + + + +## 四、队列的变体 + +上面的实现很简单，但在某些情况下效率很低。 + +假设我们分配一个最大长度为 5 的数组。当队列满时，我们想要循环利用的空间的话，在执行取出队首元素的时候，我们就必须将数组中其他所有元素都向前移动一个位置，时间复杂度就变成了 $O(n)$ + +![img](https://aliyun-lc-upload.oss-cn-hangzhou.aliyuncs.com/aliyun-lc-upload/uploads/2018/07/21/screen-shot-2018-07-21-at-153713.png) + + +为了解决这个问题就引出了我们接下来要说的循环队列。 + + + +### 3.2 循环队列 + +为了提高运算的效率，我们用另一种方式来表达数组中各单元的位置关系，将数组看做是一个环形的。当 rear 到达数组的最大下标时，重新指回数组下标为`0`的位置，这样就避免了数据迁移的低效率问题。 + +![](/Users/starfish/oceanus/picBed/data-structure/cycle-queue.png) + +用循环数组实现的队列称为循环队列，我们将循环队列中从对首到队尾的元素按逆时针方向存放在循环数组中的一段连续的单元中。当新元素入队时，将队尾指针 rear 按逆时针方向移动一位即可，出队操作也很简单，只要将对首指针 front 逆时针方向移动一位即可。 + +可以看出用循环队列来实现队列可以在 $O(1)$ 时间内完成入队和出队操作。 + +当然队首和队尾指针也可以有不同的指向，例如也可以用队首指针 front 指向队首元素所在单元的前一个单元，或者用队尾指针 rear 指向队尾元素所在单元的方法来表示队列在循环数组中的位置。但是不论使用哪一种方法来指示队首与队尾元素，我们都要解决一个细节问题，即如何表示满队列和空队列。 + +当循环队列为空或者满的时候都会出现 front == rear 的情况，即无法通过条件 front == rear 来判别队列是”空”还是”满”。 + +解决这个问题的方法至少有三种： + +1. 另设一布尔变量或标志变量以区别队列的空和满； + +2. 使用一个计数器记录队列中元素的总数（即队列长度）。 + +3. 保留一个元素空间。也就是说，当队尾指针的下一个元素就是队首指针所指单元时，就停止入队。队列满时，数组中还有一个空闲单元。 + + - 这种方式的话，由于 rear 可能比 front 大，也可能比 front 小，所以尽管他们只相差一个位置时是满的情况，但也可能是相差整整一圈。所以若队列的最大长度是 maxSize，那么队满的条件是 `(rear + 1) % maxSize == front`（取模的目的就是为了整合 rear 和 front 大小的问题），如下两种情况都是队满。 + + - 此时判空条件仍是 front == rear，只是判满条件改变； + + ![img](https://alleniverson.gitbooks.io/data-structure-and-algorithms/4.%E9%98%9F%E5%88%97/img/%E5%BE%AA%E7%8E%AF%E9%98%9F%E5%88%97.png) + + ![image-20200720110825405](C:\Users\jiahaixin\AppData\Roaming\Typora\typora-user-images\image-20200720110825405.png) + + + +3. ![](https://tva1.sinaimg.cn/large/007S8ZIlly1ggwer7ntcmj30f405etae.jpg) + +```java +public class MyArrayQueue implements MyQueue { + + private int capacity; + private int front; + private int rear; + private Object[] arr; + + // 创建队列的构造器 + public MyArrayQueue(int capacity) { + this.capacity = capacity; + arr = new Object[capacity]; + front = -1; // 指向队列头部，分析出front是指向队列头的前一个位置. + rear = -1; // 指向队列尾，指向队列尾的数据(即就是队列最后一个数据) + } + + public int getSize() { + return rear - front; + } + + public boolean isEmpty() { + return rear == front; + } + + public boolean isFull() { + return rear == capacity - 1; + } + + public void enqueue(Object e) { + if (isFull()) { + System.out.println("队列满，不能加入数据~"); + return; + } + rear++; // 让rear 后移 + arr[rear] = e; + } + + public Object dequeue() { + if (isEmpty()) { + // 通过抛出异常 + throw new RuntimeException("队列空，不能取数据"); + } + front++; // front后移 + return arr[front]; + } + + public Object peek() { + if (isEmpty()) { + throw new RuntimeException("队列空的，没有数据~~"); + } + return arr[front + 1]; + } +} +``` + + + +### 3.3 链式队列 + +队列的链式存储结构，其实就是线性表的单链表，只不过它只能尾进头出而已，我们把它简称为链队列。 + +根据单链表的特点，选择链表的头部作为队首，链表的尾部作为队尾。除了链表头结点需要通过一个引用来指向之外，还需要一个对链表尾结点的引用，以方便队列的入队操作的实现。为此一共设置两个指针，一个队首指针和一个队尾指针。队首指针指向队首元素的前一个结点，即始终指向链表空的头结点，队尾指针指向队列当前队尾元素所在的结点。当队列为空时，队首指针与队尾指针均指向空的头结点。 + + + +**队头指针( front )**指向链队列的头结点，而**队尾指针( rear )**指向终端结点。非空链队列如下所示。 + +![](https://tva1.sinaimg.cn/large/007S8ZIlly1gh1bmzvkjnj31860bngmm.jpg) + +当**队列为空**时，front和rear都指向头结点。 +空链队列 + +```java +public class MyLinkedQueue implements MyQueue { + + /** + * 链表结构，肯定要先有个Node,放结点信息 + */ + class Node { + private Object element; + private Node next; + + public Node() { + this(null,null); + } + + public Node(Object ele, Node next){ + this.element = ele; + this.next = next; + } + + public Node getNext(){ + return next; + } + + public void setNext(Node next){ + this.next = next; + } + public Object getData() { + return element; + } + + public void setData(Object obj) { + element = obj; + } + } + + private Node front; + private Node rear; + private int size; + + public MyLinkedQueue() { + this.front = new Node(); + this.rear = front; + this.size = 0; + } + + public int getSize() { + return size; + } + + public boolean isEmpty() { + return size == 0; + } + + public boolean isFull() { + return false; + } + + public void enqueue(Object e) { + Node node = new Node(e, null); + rear.setNext(node); + rear = node; + size++; + } + + public Object dequeue() { + if (size < 1) { + throw new RuntimeException("队列空的，没有数据~~"); + } + Node p = front.getNext(); + front.setNext(p.getNext()); + size--; + if (size < 1) { + rear = front;//如果队列为空， + } + return p.getData(); + } + + public Object peek() { + Node node = front; + if (size < 1) { + throw new RuntimeException("队列空的，没有数据~~"); + } + return front.getNext().getData(); + } +} +``` + + + +### 3.4 双端队列 + +双端队列是一种队头、队尾都可以进行入队、出队操作的队列，双端队列采用双向链表来实现，Java 中的`Deque` 接口是 double ended queue 的缩写，即双端队列，支持在队列的两端插入和删除元素，继承 `Queue`接口。具体实现方式，可以参考**`ArrayDeque` 和 `LinkedList`** 。 + +- `ArrayDeque` 类由数组支持。适合当作堆栈使用。 +- `LinkedList` 类由链表支持。适合当作FIFO队列使用。 + + + +### 3.5 优先队列 + +优先队列为一种不必遵循队列先进先出(FIFO)特性的特殊队列，优先队列跟普通队列一样都只有一个队头和一个队尾并且也是从队头出队，队尾入队，不过在优先队列中，每次入队时，都会按照入队数据项的关键值进行排序(从大到小、从小到大)，这样保证了关键字最小的或者最大的项始终在队头，出队的时候优先级最高的就最先出队。 + +优先级队列可以用数组实现也可以用堆来实现。一般来说，用堆实现的效率更高。 + +Java 中的 PriorityQueue 就是优先队列的实现。下边我们用数组实现一个简单的优先队列，让最小值得元素始终在队头。 + +```java +public class MyPriorityQueue { + + private int maxSize; + private long[] queArray; + private int nItems; + + public MyPriorityQueue(int s) { + maxSize = s; + queArray = new long[maxSize]; + nItems = 0; + } + + + public int getSize() { + return queArray.length; + } + + public boolean isEmpty() { + return (nItems == 0); + } + + public boolean isFull() { + return (nItems == maxSize); + } + + public void enqueue(long e) { + int j; + if (nItems == 0) { + queArray[nItems++] = e; + } else { + for (j = nItems - 1; j >= 0; j--) { + if (e > queArray[j]) { + queArray[j + 1] = queArray[j]; + } else { + break; + } + } + queArray[j + 1] = e; + nItems++; + } + } + + public Object dequeue() { + return queArray[--nItems]; + } + + public Object peek() { + return queArray[nItems - 1]; + } +} +``` + + + + + +## 五、队列的应用 + +队列在计算机科学的许多领域都有应用，包括： + +- **操作系统**：在多任务处理和调度中，队列用来管理进程执行的顺序。 +- **网络**：在数据包的传输中，队列帮助管理数据包的发送顺序和处理。 +- **算法**：在广度优先搜索（BFS）等算法中，队列用于存储待处理的节点。 + + + +## 队列的广度优先搜索 + + + + + + + +## + + + + + + + diff --git a/docs/data-structure-algorithms/data-structure/Skip-List.md b/docs/data-structure-algorithms/data-structure/Skip-List.md new file mode 100644 index 0000000000..ad767655f8 --- /dev/null +++ b/docs/data-structure-algorithms/data-structure/Skip-List.md @@ -0,0 +1,375 @@ +--- +title: 跳表 +date: 2023-05-09 +tags: + - Skip List +categories: data-structure +--- + +![](https://img.starfish.ink/data-structure/skiplist-banner.png) + +> Redis 是怎么想的：用跳表来实现有序集合？ +> + +干过服务端开发的应该都知道 Redis 的 ZSet 使用跳表实现的（当然还有压缩列表、哈希表），我就不从 1990 年的那个美国大佬 William Pugh 发表的那篇论文开始了，直接开跳 + +![马里奥](https://img.starfish.ink/data-structure/bbdcce2d04b2bd83.gif) + +文章拢共两部分 + +- 跳表是怎么搞的 +- Redis 是怎么想的 + + + +## 一、跳表 + +### 跳表的简历 + +![](https://img.starfish.ink/data-structure/skiplist-resume.png) + +跳表，英文名：Skip List + +父亲：从英文名可以看出来，它首先是个 List，实际上，它是在有序链表的基础上发展起来的 + +竞争对手：跳表（skip list）对标的是平衡树（AVL Tree） + +优点：是一种 插入/删除/搜索 都是 $O(logn)$ 的数据结构。它最大的优势是原理简单、容易实现、方便扩展、效率更高 + + + +### 跳表的基本思想 + +一如往常，采用循序渐进的手法带你窥探 William Pugh 的小心思~ + +前提：跳表处理的是有序的链表，所以我们先看个不能再普通了的有序列表（一般是双向链表） + +![](https://img.starfish.ink/data-structure/linkedlist.png) + +如果我们想查找某个数，只能遍历链表逐个比对，时间复杂度 $O(n)$，插入和删除操作都一样。 + +为了提高查找效率，我们对链表做个”索引“ + +![](https://img.starfish.ink/data-structure/skip-index.png) + +像这样，我们每隔一个节点取一个数据作为索引节点（或者增加一个指针），比如我们要找 31 直接在索引链表就找到了（遍历 3 次），如果找 16 的话，在遍历到 31的时候，发现大于目标节点，就跳到下一层，接着遍历~ (蓝线表示搜索路径) + +> 恩，如果你数了下遍历次数，没错，加不加索引都是 4 次遍历才能找到 16，这是因为数据量太少， +> +> 数据量多的话，我们也可以多建几层索引，如下 4 层索引，效果就比较明显了 + +![](https://img.starfish.ink/data-structure/skiplist.png) + +每加一层索引，我们搜索的时间复杂度就降为原来的 $O(n/2)$ + +加了几层索引，查找一个节点需要遍历的节点个数明线减少了，效率提高不少，bingo~ + +有没有似曾相识的感觉，像不像二分查找或者二叉搜索树，通过索引来跳过大量的节点，从而提高搜索效率。 + +这样的多层链表结构，就是『**跳表**』了~~ + + + +**那到底提高了多少呢？** + +推理一番： + +1. 如果一个链表有 n 个结点，如果每两个结点抽取出一个结点建立索引的话，那么第一级索引的结点数大约就是 n/2，第二级索引的结点数大约为 n/4，以此类推第 m 级索引的节点数大约为 $n/(2^m)$。 + +2. 假如一共有 m 级索引，第 m 级的结点数为两个，通过上边我们找到的规律，那么得出 $n/(2^m)=2$，从而求得 m=$log(n)$-1。如果加上原始链表，那么整个跳表的高度就是 $log(n)$。 + +3. 我们在查询跳表的时候，如果每一层都需要遍历 k 个结点，那么最终的时间复杂度就为 $O(k*log(n))$。 + +4. 那这个 k 值为多少呢，按照我们每两个结点提取一个基点建立索引的情况，我们每一级最多需要遍历两个节点，所以 k=2。 + + > 为什么每一层最多遍历两个结点呢？ + > + > 因为我们是每两个节点提取一个节点建立索引，最高一级索引只有两个节点，然后下一层索引比上一层索引两个结点之间增加了一个结点，也就是上一层索引两结点的中值，看到这里是不是想起了二分查找，每次我们只需要判断要找的值在不在当前节点和下一个节点之间就可以了。 + > + > 不信，你照着下图比划比划，看看同一层能画出 3 条线不~~ + > + > ![](https://img.starfish.ink/data-structure/skiplist-index-count.png) + +5. 既然知道了每一层最多遍历两个节点，那跳表查询数据的时间复杂度就是 $O(2*log(n))$，常数 2 忽略不计，就是 $O(logn)$ 了。 + + + +**空间换时间** + +跳表的效率比链表高了，但是跳表需要额外存储多级索引，所以需要更多的内存空间。 + +跳表的空间复杂度分析并不难，如果一个链表有 n 个节点，每两个节点抽取出一个节点建立索引的话，那么第一级索引的节点数大约就是 n/2，第二级索引的节点数大约为 n/4，以此类推第 m 级索引的节点数大约为 $n/(2^m)$，我们可以看出来这是一个等比数列。 + +这几级索引的结点总和就是 n/2+n/4+n/8…+8+4+2=n-2，所以跳表的空间复杂度为 $O(n)$。 + +> 实际上，在软件开发中，我们不必太在意索引占用的额外空间。在讲数据结构和算法时，我们习惯性地把要处理的数据看成整数，但是在实际的软件开发中，原始链表中存储的有可能是很大的对象，而索引结点只需要存储关键值和几个指针，并不需要存储对象，所以当对象比索引结点大很多时，那索引占用的额外空间就可以忽略了。 + + + +#### 插入数据 + +其实插入数据和查找一样，先找到元素要插入的位置，时间复杂度也是 $O(logn)$，但有个问题就是如果一直往原始列表里加数据，不更新我们的索引层，极端情况下就会出现两个索引节点中间数据非常多，相当于退化成了单链表，查找效率直接变成 $O(n)$ + +![](https://img.starfish.ink/data-structure/skiplist-insert.png) + + + +#### 跳表索引动态更新 + +我们上边建立索引层都是下层节点个数的 1/2，最高层索引的节点数就是 2 个，但是我们随意插入或者删除一个原有链表的节点，这个比例就肯定会被破坏。 + +作为一种动态数据结构，我们需要某种手段来维护索引与原始链表大小之间的平衡，也就是说，如果链表中节点多了，索引节点就相应地增加一些，避免复杂度退化。 + +如果重建索引的话，效率就不能保证了。 + +> 如果你了解红黑树、AVL 树这样平衡二叉树，你就知道它们是通过左右旋的方式保持左右子树的大小平衡，而跳表是通过随机函数来维护前面提到的“平衡性”。 + +所以跳表（skip list）索性就不强制要求 `1:2` 了，一个节点要不要被索引，建几层的索引，就随意点吧，都在节点插入时由抛硬币决定。 + +比如我们要插入新节点 X，那要不要为 X 向上建索引呢，就是抛硬币决定的，正面的话建索引，否则就不建了，就是这么随意（比如一个节点随机出的层数是 3，那就把它链入到第1 层到第 3 层链表中，也就是我们除了原链表的之外再往上 2 层索引都加上）。 + +![](https://img.starfish.ink/data-structure/20210626125654.gif) + +其实是因为我们不能预测跳表的添加和删除操作，很难用一种有效的算法保证索引部分始终均匀。学过概率论的我们都知道抛硬币虽然不能让索引位置绝对均匀，当数量足够多的时候最起码可以保证大体上相对均匀。 + +删除节点相对来说就容易很多了，在索引层找到节点的话，就顺藤摸瓜逐个往下删除该索引节点和原链表上的节点，如果哪一层索引节点被删的就剩 1 个节点的话，直接把这一层搞掉就可以了。 + + + +其实跳表的思想很容易理解，可是架不住实战，我们接着看实战 + +### 跳表的实现 + +差不多了解了跳表，其实就是加了几层索引的链表，一共有 N 层，以 0 ~ N 层表示，设第 0 层是原链表，抽取其中部分元素，在第 1 层形成新的链表，上下层的相同元素之间连起来；再抽取第 1 层部分元素，构成第 2 层，以此类推。 + +Leetcode 的题目：设计跳表（https://leetcode-cn.com/problems/design-skiplist/） + +既然是链表结构，先搞个 Node + +```java +class Node{ + Integer value; //节点值 + Node[] next; // 节点在不同层的下一个节点 + + public Node(Integer value,int size) { // 用size表示当前节点在跳表中索引几层 + this.value = value; + this.next = new Node[size]; + } +} +``` + +增删改查来一套，先看下增加节点，上边我们已经知道了，新增时候会随机生成一个层数，看下网上大佬的解释 + +> 执行插入操作时计算随机数的过程，是一个很关键的过程，它对 skiplist 的统计特性有着很重要的影响。这并不是一个普通的服从均匀分布的随机数，它的计算过程如下： +> +> - 首先，每个节点肯定都有第 1 层指针（每个节点都在第 1 层链表里）。 +> - 如果一个节点有第 i 层(i>=1)指针（即节点已经在第 1 层到第 i 层链表中），那么它有第(i+1)层指针的概率为 p。 +> - 节点最大的层数不允许超过一个最大值，记为 MaxLevel。 +> +> 这个计算随机层数的代码如下所示： +> +> ```java +> int randomLevel() +> int level = 1; +> while (Math.random()

level ++ ; +> } +> return level; +> ``` +> +> `randomLevel()` 包含两个参数，一个是 p，一个是 MaxLevel。在 Redis 的 skiplist 实现中，这两个参数的取值为： +> +> ``` +> p = 1/4 +> MaxLevel = 32(5.0版本以后是64) +> ``` + +所以我们和 Redis 一样的设置 + +```java +/** + * 最大层数 + */ +private static int DEFAULT_MAX_LEVEL = 32; + +/** + * 随机层数概率，也就是随机出的层数，在 第1层以上(不包括第一层)的概率，层数不超过maxLevel，层数的起始号为1 + */ +private static double DEFAULT_P_FACTOR = 0.25; + +/** + * 头节点 + */ +Node head = new Node(null, DEFAULT_MAX_LEVEL); + +/** + * 表示当前nodes的实际层数，它从1开始 + */ +int currentLevel = 1; +``` + +#### 增 + +我觉得我写的注释还挺通俗易懂的（代码参考了 leetcode 和 王争老师的实现） + +```java +public void add(int num) { + int level = randomLevel(); + Node newNode = new Node(num,level); + + Node updateNode = head; + + // 计算出当前num 索引的实际层数，从该层开始添加索引，逐步向下 + for (int i = currentLevel-1; i>=0; i--) { + //找到本层最近离num最近的节点(刚好比它小的节点) + while ((updateNode.next[i])!=null && updateNode.next[i].value < num){ + updateNode = updateNode.next[i]; + } + //本次随机的最高层才设值，如果是最后一个直接指向newNode,否则将newNode 链入链表 + if (i currentLevel){ + for (int i = currentLevel; i < level; i++) { + head.next[i] = newNode; + } + //更新层数 + currentLevel = level; + } +} +``` + +#### 删 + +```java +public boolean erase(int num) { + boolean flag = false; + Node searchNode = head; + //从最高层开始遍历找 + for (int i = currentLevel-1; i >=0; i--) { + //和新增一样也需要找到离要删除节点最近的辣个 + while ((searchNode.next[i])!=null && searchNode.next[i].value < num){ + searchNode = searchNode.next[i]; + } + //如果有这样的数值 + if (searchNode.next[i]!=null && searchNode.next[i].value == num){ + //找到该层中该节点，把下一个节点过来，就删除了 + searchNode.next[i] = searchNode.next[i].next[i]; + flag = true; + } + } + return flag; +} +``` + +#### 查 + +```java +public Node search(int target) { + Node searchNode = head; + for (int i = currentLevel - 1; i >= 0; i--) { + while ((head.next[i]) != null && searchNode.next[i].value < target) { + searchNode = searchNode.next[i]; + } + } + if (searchNode.next[0] != null && searchNode.next[0].value == target) { + return searchNode.next[0]; + } else { + return null; + } +} +``` + + + +## 二、Redis 为什么选择跳表？ + +跳表本质上也是一种查找结构，我们经常遇到的查找问题最常见的就是哈希表，还有就是各种平衡树，跳表又不属于这两大阵营，那问题来了： + +为什么 Redis 要用跳表来实现有序集合，而不是哈希表或者平衡树（AVL、红黑树等）？ + +> Redis 中的有序集合是通过跳表来实现的，严格点讲，其实还用到了压缩列表、哈希表。 +> +> Redis 提供了两种编码的选择，根据数据量的多少进行对应的转化 ([源码地址](https://github.com/redis/redis/blob/8f59f131e5c26680d21e825695ef24a4fd7b99b7/src/server.h) ) +> +> - 当数据较少时，ZSet 是由一个 ziplist 来实现的 +> +> - 当数据多的时候，ZSet 是由一个 dict + 一个 skiplist 来实现的。简单来讲，dict 用来查询数据到分数的对应关系，而 skiplist 用来根据分数查询数据（可能是范围查找）。 +> +> ![](https://img.starfish.ink/redis/redis-zset-code.svg) +> +> Redis 的跳跃表做了些修改 +> +> - 允许有重复的分值 +> - 对元素的比对不仅要比对他们的分值，还要比对他们的对象 +> - 每个跳跃表节点都带有一个后退指针，它允许程序在执行像 ZREVRANGE 这样的命令时，从表尾向表头遍历跳跃表。 + +我们先看下 ZSet 常用的一些命令 + +| 命令 | 描述 | +| ------------- | ------------------------------------------------------------ | +| **ZADD** | 向有序集合添加一个或多个成员，或者更新已存在成员的分数 | +| **ZCARD** | 返回有序集 key 的基数 | +| ZREM | 移除有序集 key 中的一个或多个成员，不存在的成员将被忽略 | +| ZSCAN | 迭代有序集合中的元素（包括元素成员和元素分值） | +| **ZREVRANGE** | 返回有序集 key 中，指定区间内的成员。其中成员的位置按 score 值递减(从大到小)来排列 | + +主要操作就是增删查一个数据，迭代数据、按区间查数据，看下原因 + +- 哈希表是无序的，且只能查找单个 key，不适宜范围查找 +- 插入、删除、查找以及迭代输出有序序列这几个操作，红黑树也可以完成，时间复杂度跟跳表是一样的。但是，按照区间来查找数据这个操作，红黑树的效率没有跳表高（跳表只需要定位区间的起点，然后遍历就行了） +- Redis 选跳表实现有序集合，还有其他各种原因，比如代码实现相对简单些，且更加灵活，它可以通过改变索引构建策略，有效平衡执行效率和内存消耗。 + + + +## 扯点别的 | 这又是为什么？ + +> 有序集合的英文全称明明是**sorted sets**，为啥叫 zset 呢？ +> +> Redis官网上没有解释，但是在 Github 上有人向作者提问了。作者是这么回答的哈哈哈 +> +> Hello. Z is as in XYZ, so the idea is, sets with another dimension: the +> order. It’s a far association… I know 😃 +> +> 原来前面的 Z 代表的是 XYZ 中的Z，最后一个英文字母，zset 是在说这是比 set 有更多一个维度的 set 😦 +> +> 是不没道理？ +> +> 更没道理的还有，Redis 默认端口 6379 ，因为作者喜欢的一个叫 Merz 的女明星，其名字在手机上输入正好对应号码 6379，索性就把 Redis 的默认端口叫 6379 了… + + + +## 小结 + +跳表使用空间换时间的设计思路，通过构建多级索引来提高查询的效率，实现了基于链表的“二分查找”。 + +跳表是一种动态数据结构，支持快速地插入、删除、查找操作，时间复杂度都是 $O(logn)$。 + +Redis 的 zset 是一个复合结构，一方面它需要一个 hash 结构来存储 value 和 score 的对应关系，另一方面需要提供按照 score 来排序的功能，还需要能够指定 score 的范围来获取 value 列表的功能 + +Redis 中的有序集合是通过压缩列表、哈希表和跳表的组合来实现的，当数据较少时，ZSet 是由一个 ziplist 来实现的。当数据多的时候，ZSet 是由一个dict + 一个 skiplist 来实现的 + +![](https://img.starfish.ink/data-structure/06eb28fd58fa8840.gif) + + + +后续：MySQL 的 Innodb ，为什么不用 skiplist，而用 B+ Tree ？ + + + +## 参考 + +- [ftp://ftp.cs.umd.edu/pub/skipLists/skiplists.pdf](ftp://ftp.cs.umd.edu/pub/skipLists/skiplists.pdf) 原论文 +- [Redis内部数据结构详解(6)——skiplist](http://zhangtielei.com/posts/blog-redis-skiplist.html) 图文并茂讲解 skip list +- https://leetcode-cn.com/problems/design-skiplist/ +- https://lotabout.me/2018/skip-list/ +- https://redisbook.readthedocs.io/en/latest/internal-datastruct/skiplist.html +- https://redisbook.readthedocs.io/en/latest/datatype/sorted_set.html#sorted-set-chapter \ No newline at end of file diff --git a/docs/data-structure-algorithms/data-structure/Stack.md b/docs/data-structure-algorithms/data-structure/Stack.md new file mode 100644 index 0000000000..4e8996f3a0 --- /dev/null +++ b/docs/data-structure-algorithms/data-structure/Stack.md @@ -0,0 +1,546 @@ +--- +title: Stack +date: 2023-05-09 +tags: + - Stack +categories: data-structure +--- + +![](https://img.starfish.ink/spring/stack-banner.png) + +> 栈（stack）又名堆栈，它是**一种运算受限的线性表**。 限定仅在表尾进行插入和删除操作的线性表。 + + + +## 一、概述 + +### 定义 + +注意：本文所说的栈是数据结构中的栈，而不是内存模型中栈。 + +栈（stack）是限定仅在表尾一端进行插入或删除操作的**特殊线性表**。又称为堆栈。 + +对于栈来说, 允许进行插入或删除操作的一端称为栈顶（top），而另一端称为栈底（bottom）。不含元素栈称为空栈，向栈中插入一个新元素称为入栈或压栈， 从栈中删除一个元素称为出栈或退栈。 + +假设有一个栈Ｓ＝（a1, a2, …, an)，a1先进栈， an最后进栈。称 a1 为栈底元素，an 为栈顶元素。出栈时只允许在栈顶进行，所以 an 先出栈，a1最后出栈。因此又称栈为后进先出（Last In First Out，LIFO）的线性表。 + +栈（stack），是一种线性存储结构，它有以下几个特点： + +- 栈中数据是按照"后进先出（LIFO, Last In First Out）"方式进出栈的。 +- 向栈中添加/删除数据时，只能从栈顶进行操作。 + +![](https://img.starfish.ink/data-structure/applications-of-stack-in-data-structure.png) + + + +### 基本操作 + +栈的基本操作除了进栈 `push()`，出栈 `pop()` 之外，还有判空 `isEmpty()`、取栈顶元素 `peek()` 等操作。 + +抽象成接口如下： + +```java +public interface MyStack { + + /** + * 返回堆栈的大小 + */ + public int getSize(); + + /** + * 判断堆栈是否为空 + */ + public boolean isEmpty(); + + /** + * 入栈 + */ + public void push(Object e); + + /** + * 出栈，并删除 + */ + public Object pop(); + + /** + * 返回栈顶元素 + */ + public Object peek(); +} +``` + + + +和线性表类似，栈也有两种存储结构：顺序存储和链式存储。 + +实际上，栈既可以用数组来实现，也可以用链表来实现。用数组实现的栈，我们叫作**顺序栈**，用链表实现的栈，我们叫作**链式栈**。 + +## 二、栈的顺序存储与实现 + +顺序栈是使用顺序存储结构实现的栈，即利用一组地址连续的存储单元依次存放栈中的数据元素。由于栈是一种特殊的线性表，因此在线性表的顺序存储结构的基础上，选择线性表的一端作为栈顶即可。那么根据数组操作的特性，选择数组下标大的一端，即线性表顺序存储的表尾来作为栈顶，此时入栈、出栈操作可以 $O(1)$ 时间完成。 + +由于栈的操作都是在栈顶完成，因此在顺序栈的实现中需要附设一个指针 top 来动态地指示栈顶元素在数组中的位置。通常 top 可以用栈顶元素所在的数组下标来表示，`top=-1` 时表示空栈。 + +栈在使用过程中所需的最大空间难以估计，所以，一般构造栈的时候不应设定最大容量。一种合理的做法和线性表类似，先为栈分配一个基本容量，然后在实际的使用过程中，当栈的空间不够用时再倍增存储空间。 + +```java +public class MyArrayStack implements MyStack { + + private final int capacity = 2; //默认容量 + private Object[] arrs; //数据元素数组 + private int top; //栈顶指针 + + MyArrayStack(){ + top = -1; + arrs = new Object[capacity]; + } + + public int getSize() { + return top + 1; + } + + public boolean isEmpty() { + return top < 0; + } + + public void push(Object e) { + if(getSize() >= arrs.length){ + expandSapce(); //扩容 + } + arrs[++top]=e; + } + + private void expandSapce() { + Object[] a = new Object[arrs.length * 2]; + for (int i = 0; i < arrs.length; i++) { + a[i] = arrs[i]; + } + arrs = a; + } + + public Object pop() { + if(getSize()<1){ + throw new RuntimeException("栈为空"); + } + Object obj = arrs[top]; + arrs[top--] = null; + return obj; + } + + public Object peek() { + if(getSize()<1){ + throw new RuntimeException("栈为空"); + } + return arrs[top]; + } +} +``` + +以上基于数据实现的栈代码并不难理解。由于有 top 指针的存在，所以`size()`、`isEmpty()`方法均可在 $O(1) $ 时间内完成。`push()`、`pop()`和`peek()`方法，除了需要`ensureCapacity()`外，都执行常数基本操作，因此它们的运行时间也是 $O(1)$ + + + +## 三、栈的链式存储与实现 + +栈的链式存储即采用链表实现栈。当采用单链表存储线性表后，根据单链表的操作特性选择单链表的头部作为栈顶，此时，入栈和出栈等操作可以在 $O(1)$ 时间内完成。 + +由于栈的操作只在线性表的一端进行，在这里使用带头结点的单链表或不带头结点的单链表都可以。使用带头结点的单链表时，结点的插入和删除都在头结点之后进行；使用不带头结点的单链表时，结点的插入和删除都在链表的首结点上进行。 + +下面以不带头结点的单链表为例实现栈，如下示意图所示： + +![](https://img.starfish.ink/data-structure/stack-linked.png) + +在上图中，top 为栈顶结点的引用，始终指向当前栈顶元素所在的结点。若 top 为null，则表示空栈。入栈操作是在 top 所指结点之前插入新的结点，使新结点的 next 域指向 top，top 前移即可；出栈则直接让 top 后移即可。 + +```java +public class MyLinkedStack implements MyStack { + + class Node { + private Object element; + private Node next; + + public Node() { + this(null, null); + } + + public Node(Object ele, Node next) { + this.element = ele; + this.next = next; + } + + public Node getNext() { + return next; + } + + public void setNext(Node next) { + this.next = next; + } + + public Object getData() { + return element; + } + + public void setData(Object obj) { + element = obj; + } + } + + private Node top; + private int size; + + public MyLinkedStack() { + top = null; + size = 0; + } + + public int getSize() { + return size; + } + + public boolean isEmpty() { + return size == 0; + } + + public void push(Object e) { + Node node = new Node(e, top); + top = node; + size++; + } + + public Object pop() { + if (size < 1) { + throw new RuntimeException("堆栈为空"); + } + Object obj = top.getData(); + top = top.getNext(); + size--; + return obj; + } + + public Object peek() { + if (size < 1) { + throw new RuntimeException("堆栈为空"); + } + return top.getData(); + } +} +``` + +上述 `MyLinkedStack` 类中有两个成员变量，其中 `top` 表示首结点，也就是栈顶元素所在的结点；`size` 指示栈的大小，即栈中数据元素的个数。不难理解，所有的操作均可以在 $O(1)$ 时间内完成。 + + + +## 四、JDK 中的栈实现 Stack + +Java 工具包中的 Stack 是继承于 Vector(矢量队列)的，由于 Vector 是通过数组实现的，这就意味着，Stack 也是通过数组实现的，而非链表。当然，我们也可以将 LinkedList 当作栈来使用。 + +### Stack的继承关系 + +```java +java.lang.Object + java.util.AbstractCollection + java.util.AbstractList + java.util.Vector + java.util.Stack + +public class Stack extends Vector {} +``` + + + + + +## 五、栈应用 + +栈有一个很重要的应用，在程序设计语言里实现了递归。 + +### [20. 有效的括号](https://leetcode.cn/problems/valid-parentheses/) + +>给定一个只包括 `'('`，`')'`，`'{'`，`'}'`，`'['`，`']'` 的字符串，判断字符串是否有效。 +> +>有效字符串需满足： +> +>1. 左括号必须用相同类型的右括号闭合。 +>2. 左括号必须以正确的顺序闭合。 +> +>注意空字符串可被认为是有效字符串。 +> +>``` +>输入: "{[]}" +>输出: true +>输入: "([)]" +>输出: false +>``` + +**思路** + +左括号进栈，右括号匹配出栈 + +- 栈先入后出特点恰好与本题括号排序特点一致，即若遇到左括号时，把右括号入栈，遇到右括号时将对应栈顶元素与其对比并出栈，相同的话说明匹配，继续遍历，遍历完所有括号后 `stack` 仍然为空，说明是有效的。 + +```java + public boolean isValid(String s) { + if(s.isEmpty()) { + return true; + } + Stack stack=new Stack(); + //字符串转为字符串数组 遍历 + for(char c:s.toCharArray()){ + if(c=='(') { + stack.push(')'); + } else if(c=='{') { + stack.push('}'); + } else if(c=='[') { + stack.push(']'); + } else if(stack.empty()||c!=stack.pop()) { + return false; + } + } + return stack.empty(); + } +``` + + + +### [739. 每日温度](https://leetcode.cn/problems/daily-temperatures/) + +>给定一个整数数组 `temperatures` ，表示每天的温度，返回一个数组 `answer` ，其中 `answer[i]` 是指对于第 `i`天，下一个更高温度出现在几天后。如果气温在这之后都不会升高，请在该位置用 `0` 来代替。 +> +>``` +>输入: temperatures = [73,74,75,71,69,72,76,73] +>输出: [1,1,4,2,1,1,0,0] +>``` + +**思路**： + +维护一个存储下标的单调栈，从栈底到栈顶的下标对应的温度列表中的温度依次递减。如果一个下标在单调栈里，则表示尚未找到下一次温度更高的下标。 + +正向遍历温度列表。 + +1. 栈为空，先入栈 +2. 如果栈内有元素，用栈顶元素对应的温度和当前温度比较，temperatures[i] > temperatures[pre] 的话，就把栈顶元素移除，把 pre 对应的天数设置为 i-pre，重复操作区比较，知道栈为空或者栈顶元素对应的温度小于当前问题，把当前温度索引入栈 + +```java + public int[] dailyTemperatures_stack(int[] temperatures) { + int length = temperatures.length; + int[] result = new int[length]; + Stack stack = new Stack<>(); + for (int i = 0; i < length; i++) { + while (!stack.isEmpty() && temperatures[i] > temperatures[stack.peek()]) { + int pre = stack.pop(); + result[pre] = i - pre; + } + stack.add(i); + } + return result; + } +``` + + + +### [150. 逆波兰表达式求值](https://leetcode.cn/problems/evaluate-reverse-polish-notation/) + +> 给你一个字符串数组 `tokens` ，表示一个根据 [逆波兰表示法](https://baike.baidu.com/item/逆波兰式/128437) 表示的算术表达式。 +> +> 请你计算该表达式。返回一个表示表达式值的整数。 +> +> **注意：** +> +> - 有效的算符为 `'+'`、`'-'`、`'*'` 和 `'/'` 。 +> - 每个操作数（运算对象）都可以是一个整数或者另一个表达式。 +> - 两个整数之间的除法总是 **向零截断** 。 +> - 表达式中不含除零运算。 +> - 输入是一个根据逆波兰表示法表示的算术表达式。 +> - 答案及所有中间计算结果可以用 **32 位** 整数表示。 +> +> ``` +> 输入：tokens = ["4","13","5","/","+"] +> 输出：6 +> 解释：该算式转化为常见的中缀算术表达式为：(4 + (13 / 5)) = 6 +> ``` + +> **逆波兰表达式：** +> +> 逆波兰表达式是一种后缀表达式，所谓后缀就是指算符写在后面。 +> +> - 平常使用的算式则是一种中缀表达式，如 `( 1 + 2 ) * ( 3 + 4 )` 。 +> - 该算式的逆波兰表达式写法为 `( ( 1 2 + ) ( 3 4 + ) * )` 。 +> +> 逆波兰表达式主要有以下两个优点： +> +> - 去掉括号后表达式无歧义，上式即便写成 `1 2 + 3 4 + * `也可以依据次序计算出正确结果。 +> - 适合用栈操作运算：遇到数字则入栈；遇到算符则取出栈顶两个数字进行计算，并将结果压入栈中 + +**思路**：逆波兰表达式严格遵循「从左到右」的运算。 + +计算逆波兰表达式的值时，使用一个栈存储操作数，从左到右遍历逆波兰表达式，进行如下操作： + +- 如果遇到操作数，则将操作数入栈； + +- 如果遇到运算符，则将两个操作数出栈，其中先出栈的是右操作数，后出栈的是左操作数，使用运算符对两个操作数进行运算，将运算得到的新操作数入栈。 + +整个逆波兰表达式遍历完毕之后，栈内只有一个元素，该元素即为逆波兰表达式的值。 + +```java + public int evalRPN(String[] tokens) { + List charts = Arrays.asList("+", "-", "*", "/"); + Stack stack = new Stack<>(); + for (String s : tokens) { + if (!charts.contains(s)) { + stack.push(Integer.parseInt(s)); + } else { + int num2 = stack.pop(); + int num1 = stack.pop(); + switch (s) { + case "+": + stack.push(num1 + num2); + break; + case "-": + stack.push(num1 - num2); + break; + case "*": + stack.push(num1 * num2); + break; + case "/": + stack.push(num1 / num2); + break; + } + } + } + return stack.peek(); + } +} +``` + + + +### [155. 最小栈](https://leetcode.cn/problems/min-stack/) + +> 设计一个支持 `push` ，`pop` ，`top` 操作，并能在常数时间内检索到最小元素的栈。 +> +> 实现 `MinStack` 类: +> +> - `MinStack()` 初始化堆栈对象。 +> - `void push(int val)` 将元素val推入堆栈。 +> - `void pop()` 删除堆栈顶部的元素。 +> - `int top()` 获取堆栈顶部的元素。 +> - `int getMin()` 获取堆栈中的最小元素。 + +**思路**： 添加一个辅助栈，这个栈同时保存的是每个数字 `x` 进栈的时候的**值 与 插入该值后的栈内最小值**。 + +```java +import java.util.Stack; + +public class MinStack { + + // 数据栈 + private Deque data; + // 辅助栈 + private Deque helper; + + public MinStack() { + data = new LinkedList(); + helper = new LinkedList(); + helper.push(Integer.MAX_VALUE); + } + + public void push(int x) { + data.push(x); + helper.push(Math.min(helper.peek(), x)); + } + + public void pop() { + data.pop(); + helper.pop(); + } + + public int top() { + return data.peek(); + } + + public int getMin() { + return helper.peek(); + } + +} +``` + + + +### [227. 基本计算器 II](https://leetcode.cn/problems/basic-calculator-ii/) + +> 给你一个字符串表达式 `s` ，请你实现一个基本计算器来计算并返回它的值。 +> +> 整数除法仅保留整数部分。 +> +> 你可以假设给定的表达式总是有效的。所有中间结果将在 `[-231, 231 - 1]` 的范围内。 +> +> **注意：**不允许使用任何将字符串作为数学表达式计算的内置函数，比如 `eval()` 。 +> +> ``` +> 输入：s = "3+2*2" +> 输出：7 +> ``` + +**思路**：和逆波兰表达式有点像 + +- 加号：将数字压入栈； +- 减号：将数字的相反数压入栈； +- 乘除号：计算数字与栈顶元素，并将栈顶元素替换为计算结果。 + +> 这个得先知道怎么把字符串形式的正整数转成 int +> +> ```java +> String s = "458"; +> +> int n = 0; +> for (int i = 0; i < s.length(); i++) { +> char c = s.charAt(i); +> n = 10 * n + (c - '0'); +> } +> // n 现在就等于 458 +> ``` +> +> 这个还是很简单的吧，老套路了。但是即便这么简单，依然有坑：**`(c - '0')` 的这个括号不能省略，否则可能造成整型溢出**。 +> +> 因为变量 `c` 是一个 ASCII 码，如果不加括号就会先加后减，想象一下 `s` 如果接近 INT_MAX，就会溢出。所以用括号保证先减后加才行。 + +```java + public static int calculate(String s) { + Stack stack = new Stack<>(); + int length = s.length(); + int num = 0; + char operator = '+'; + for (int i = 0; i < length; i++) { + if (Character.isDigit(s.charAt(i))) { + // 转为 int + num = num * 10 + (s.charAt(i) - '0'); + } + // 计算符 （排除空格） + if (!Character.isDigit(s.charAt(i)) && s.charAt(i) != ' ' || i == length - 1) { + // switch (s.charAt(i)){ 这里是要和操作符比对，考虑第一次 + switch (operator) { + case '+': + stack.push(num); + break; + case '-': + stack.push(-num); + break; + case '*': + stack.push(stack.pop() * num); + break; + default: + stack.push(stack.pop() / num); + } + operator = s.charAt(i); + num = 0; + } + } + int result = 0; + while (!stack.isEmpty()) { + result += stack.pop(); + } + return result; + } +``` + diff --git a/docs/data-structure-algorithms/soultion/.DS_Store b/docs/data-structure-algorithms/soultion/.DS_Store new file mode 100644 index 0000000000..57bc0bb425 Binary files /dev/null and b/docs/data-structure-algorithms/soultion/.DS_Store differ diff --git a/docs/data-structure-algorithms/soultion/Array-Solution.md b/docs/data-structure-algorithms/soultion/Array-Solution.md new file mode 100755 index 0000000000..1462dee476 --- /dev/null +++ b/docs/data-structure-algorithms/soultion/Array-Solution.md @@ -0,0 +1,1441 @@ +--- +title: 数组-热题 +date: 2025-01-08 +tags: + - Array + - algorithms +categories: leetcode +--- + +![](https://img.starfish.ink/leetcode/leetcode-banner.png) + +> **导读**：数组是最基础的数据结构，也是面试中最高频的考点。数组题目看似简单，但往往需要巧妙的算法技巧才能高效解决。掌握双指针、滑动窗口、前缀和等核心技巧是解决数组问题的关键。 +> +> **关键词**：双指针、滑动窗口、前缀和、哈希表、排序 + + +### 📋 分类索引 + +1. **🔥 双指针技巧类**：[三数之和](#_15-三数之和)、[盛最多水的容器](#_11-盛最多水的容器)、[移动零](#_283-移动零)、[颜色分类](#_75-颜色分类)、[接雨水](#_42-接雨水) +2. **🔍 哈希表优化类**：[两数之和](#_1-两数之和)、[字母异位词分组](#_49-字母异位词分组)、[最长连续序列](#_128-最长连续序列)、[存在重复元素](#_217-存在重复元素) +3. **🪟 滑动窗口类**：[无重复字符的最长子串](#_3-无重复字符的最长子串)、最小覆盖子串 +4. **📊 前缀和技巧类**：[和为K的子数组](#_560-和为-k-的子数组)、[最大子数组和](#_53-最大子数组和)、[除自身以外数组的乘积](#_238-除自身以外数组的乘积) +5. **🔄 排序与搜索类**：[合并区间](#_56-合并区间)、[数组中的第K个最大元素](#_215-数组中的第k个最大元素)、[寻找重复数](#_287-寻找重复数)、[下一个排列](#_31-下一个排列)、[搜索旋转排序数组](#_33-搜索旋转排序数组) +6. **⚡ 原地算法类**：[合并两个有序数组](#_88-合并两个有序数组)、[旋转数组](#_189-旋转数组) +7. **🧮 数学技巧类**：[只出现一次的数字](#_136-只出现一次的数字)、[多数元素](#_169-多数元素)、[找到所有数组中消失的数字](#_448-找到所有数组中消失的数字)、[缺失的第一个正数](#_41-缺失的第一个正数) +8. **🎯 综合应用类**：[买卖股票的最佳时机](#_121-买卖股票的最佳时机)、[乘积最大子数组](#_152-乘积最大子数组)、[子集](#_78-子集)、[跳跃游戏](#_55-跳跃游戏)、[跳跃游戏 II](#_45-跳跃游戏-ii)、[最长公共前缀](#_14-最长公共前缀)、[螺旋矩阵](#_54-螺旋矩阵) + +### 🎯 核心考点概览 + +- **双指针技巧**：快慢指针、左右指针、滑动窗口 +- **哈希表优化**：空间换时间，$O(1)$查找 +- **前缀和技巧**：区间求和，子数组问题 +- **排序与搜索**：二分查找、快速选择 +- **原地算法**：$O(1)$空间复杂度优化 + +### 📝 解题万能模板 + +#### 基础模板 + +```java +// 双指针模板 +int left = 0, right = nums.length - 1; +while (left < right) { + if (condition) { + left++; + } else { + right--; + } +} + +// 滑动窗口模板 +int left = 0, right = 0; +while (right < nums.length) { + // 扩展窗口 + window.add(nums[right]); + right++; + + while (window needs shrink) { + // 收缩窗口 + window.remove(nums[left]); + left++; + } +} + +// 前缀和模板 +int[] prefixSum = new int[nums.length + 1]; +for (int i = 0; i < nums.length; i++) { + prefixSum[i + 1] = prefixSum[i] + nums[i]; +} +``` + +#### 边界检查清单 + +- ✅ 空数组：`if (nums == null || nums.length == 0) return ...` +- ✅ 单元素：`if (nums.length == 1) return ...` +- ✅ 数组越界：确保索引在有效范围内 +- ✅ 整数溢出：使用long或检查边界 + +#### 💡 记忆口诀（朗朗上口版） + +- **双指针**："左右夹逼找目标，快慢追踪解环题" +- **滑动窗口**："右扩左缩维护窗，条件满足就更新" +- **前缀和**："累积求和建数组，区间相减得答案" +- **哈希表**："空间换时间来优化，常数查找是关键" +- **二分查找**："有序数组用二分，对数时间最高效" +- **原地算法**："双指针巧妙来交换，空间复杂度为一" + +**最重要的一个技巧就是，你得行动，写起来** + + + +## 一、双指针技巧类（核心中的核心）🔥 + +**💡 核心思想** + +- **左右指针**：从两端向中间逼近，解决对撞类问题 +- **快慢指针**：不同速度遍历，解决环形、去重问题 +- **滑动窗口**：双指针维护窗口，解决子数组问题 + +**🎯 必掌握模板** + +```java +// 左右指针模板 +int left = 0, right = nums.length - 1; +while (left < right) { + if (sum < target) { + left++; + } else if (sum > target) { + right--; + } else { + // 找到答案 + return ...; + } +} +``` + +### [15. 三数之和](https://leetcode-cn.com/problems/3sum/) + +**🎯 考察频率：极高 | 难度：中等 | 重要性：双指针经典** + +> 给你一个包含 n 个整数的数组 nums，判断 nums 中是否存在三个元素 a，b，c ，使得 a + b + c = 0 ？请你找出所有和为 0 且不重复的三元组。 + +**💡 核心思路**：排序后双指针 + +```java +public List> threeSum(int[] nums) { + //存放结果list + List> result = new ArrayList<>(); + int length = nums.length; + //特例判断 + if (length < 3) { + return result; + } + Arrays.sort(nums); + for (int i = 0; i < length; i++) { + //排序后的第一个数字就大于0，就说明没有符合要求的结果 + if (nums[i] > 0) break; + + //去重，当起始的值等于前一个元素，那么得到的结果将会和前一次相同，不能是 nums[i] == nums[i +1 ]，会造成遗漏 + if (i > 0 && nums[i] == nums[i - 1]) continue; + //左右指针 + int l = i + 1; + int r = length - 1; + while (l < r) { + int sum = nums[i] + nums[l] + nums[r]; + if (sum == 0) { + result.add(Arrays.asList(nums[i], nums[l], nums[r])); + //去重（相同数字的话就移动指针） + //在将左指针和右指针移动的时候，先对左右指针的值，进行判断,以防[0,0,0]这样的造成数组越界 + //不要用成 if 判断，只跳过 1 条，还会有重复的 + while (l< r && nums[l] == nums[l + 1]) l++; + while (l< r && nums[r] == nums[r - 1]) r--; + //移动指针 + l++; + r--; + } else if (sum < 0) l++; + else if (sum > 0) r--; + } + } + return result; +} +``` + +### [11. 盛最多水的容器](https://leetcode-cn.com/problems/container-with-most-water/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：双指针应用** + +> 给你 n 个非负整数 a1，a2，...，an，每个数代表坐标中的一个点 (i, ai) 。在坐标内画 n 条垂直线，垂直线 i 的两个端点分别为 (i, ai) 和 (i, 0) 。找出其中的两条线，使得它们与 x 轴共同构成的容器可以容纳最多的水。 + +**💡 核心思路**：双指针 + +水量 = 两个指针指向的数字中较小值∗指针之间的距离 + +```java +public int maxArea(int[] height){ + int l = 0; + int r = height.length - 1; + int ans = 0; + while(l < r){ + int area = Math.min(height[l], height[r]) * (r - l); + ans = Math.max(ans,area); + if(height[l] < height[r]){ + l ++; + }else { + r --; + } + } + return ans; +} +``` + +### [283. 移动零](https://leetcode-cn.com/problems/move-zeroes/) + +**🎯 考察频率：高 | 难度：简单 | 重要性：双指针基础** + +> 给定一个数组 nums，编写一个函数将所有 0 移动到数组的末尾，同时保持非零元素的相对顺序。 + +**💡 核心思路**：双指针法，当快指针指向非零元素时，将其与慢指针指向的元素交换，然后慢指针向前移动一位。 + +```java +public void moveZero(int[] nums) { + int j = 0; //慢指针 + for (int i = 0; i < nums.length; i++) { + if (nums[i] != 0) { + nums[j++] = nums[i]; // 直接将非零元素放到j指针位置，并移动j指针 + } + } + // 将j指针之后的所有元素置为零 + for (int i = j; i < nums.length; i++) { + nums[i] = 0; + } +} +``` + +### [75. 颜色分类](https://leetcode.cn/problems/sort-colors/) + +**🎯 考察频率：中等 | 难度：中等 | 重要性：三指针技巧** + +> 给定一个包含红色（0）、白色（1）、蓝色（2）的数组 `nums`，要求你对它们进行排序，使得相同颜色相邻，并按照红、白、蓝的顺序排列。 + +**💡 核心思路**：双指针（荷兰国旗问题）。用 3 个指针遍历数组。 + +这题被称为 **荷兰国旗问题 (Dutch National Flag Problem)**。 + 用 **三个指针** 解决： + +- `low`：下一个 0 应该放置的位置 +- `high`：下一个 2 应该放置的位置 +- `i`：当前遍历位置 + +规则： + +1. 如果 `nums[i] == 0` → 和 `nums[low]` 交换，`low++，i++` +2. 如果 `nums[i] == 1` → 正常情况，`i++` +3. 如果 `nums[i] == 2` → 和 `nums[high]` 交换，`high--`，**但 i 不++**，因为换过来的数还要检查 + +这样一次遍历就能完成排序。 + +```java +public void sortColors(int[] nums) { + int left = 0, right = nums.length - 1, curr = 0; + while (curr <= right) { + if (nums[curr] == 0) { + swap(nums, curr++, left++); + } else if (nums[curr] == 2) { + swap(nums, curr, right--); + } else { + curr++; + } + } +} + +private void swap(int[] nums, int i, int j) { + int temp = nums[i]; + nums[i] = nums[j]; + nums[j] = temp; +} +``` + +### [42. 接雨水](https://leetcode.cn/problems/trapping-rain-water/) + +**🎯 考察频率：极高 | 难度：困难 | 重要性：双指针+贪心** + +> 给定 n 个非负整数表示每个宽度为 1 的柱子的高度图，计算按此排列的柱子，下雨之后能够接多少雨水。 + +**💡 核心思路**：双指针法 + +```java +public int trap(int[] height) { + int left = 0, right = height.length - 1; + int leftMax = 0, rightMax = 0; + int result = 0; + + while (left < right) { + if (height[left] < height[right]) { + if (height[left] >= leftMax) { + leftMax = height[left]; + } else { + result += leftMax - height[left]; + } + left++; + } else { + if (height[right] >= rightMax) { + rightMax = height[right]; + } else { + result += rightMax - height[right]; + } + right--; + } + } + return result; +} +``` + + + +--- + +## 二、哈希表优化类（空间换时间）🔍 + +**💡 核心思想** + +- **O(1)查找**：利用哈希表常数时间查找特性 +- **空间换时间**：用额外空间降低时间复杂度 +- **去重与计数**：处理重复元素和频次统计 + +**🎯 必掌握模板** + +```java +// 哈希表查找模板 +Map map = new HashMap<>(); +for (int i = 0; i < nums.length; i++) { + if (map.containsKey(target - nums[i])) { + // 找到答案 + return new int[]{map.get(target - nums[i]), i}; + } + map.put(nums[i], i); +} +``` + +### [1. 两数之和](https://leetcode-cn.com/problems/two-sum/) + +**🎯 考察频率：极高 | 难度：简单 | 重要性：哈希表经典** + +> 给定一个整数数组 nums 和一个目标值 target，请你在该数组中找出和为目标值的那两个整数，并返回他们的数组下标。 +> +> 你可以假设每种输入只会对应一个答案。但是，数组中同一个元素不能使用两遍。 +> +> ```text +> 输入：nums = [2,7,11,15], target = 9 +> 输出：[0,1] +> 解释：因为 nums[0] + nums[1] == 9 ，返回 [0, 1] 。 +> ``` + +**思路**：哈希表 + +```java +public static int[] twoSum(int[] nums,int target){ + Map map = new HashMap<>(); + for (int i = 0; i < nums.length; i++) { + int temp = target - nums[i]; + if(map.containsKey(temp)){ + return new int[]{map.get(temp),i}; + } + map.put(nums[i],i); + } + return new int[]{-1,-1}; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(N)，其中 N 是数组中的元素数量。对于每一个元素 x，我们可以 O(1) 地寻找 target - x。 +- **空间复杂度**：O(N)，其中 N 是数组中的元素数量。主要为哈希表的开销。 + +### [49. 字母异位词分组](https://leetcode.cn/problems/group-anagrams/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：分组统计经典** + +> 给你一个字符串数组，请你将 **字母异位词** 组合在一起。可以按任意顺序返回结果列表。 +> +> **字母异位词** 是由重新排列源单词的所有字母得到的一个新单词。 +> +> ``` +> 输入: strs = ["eat", "tea", "tan", "ate", "nat", "bat"] +> 输出: [["bat"],["nat","tan"],["ate","eat","tea"]] +> ``` + +**思路**：哈希表。键是字符串中的字符排序后的字符串，值是具有相同键的原始字符串列表 + +```java +public List> groupAnagrams(String[] strs) { + Map> map = new HashMap<>(); + + for (String str : strs) { + // 将字符串转换为字符数组并排序 + char[] chars = str.toCharArray(); + Arrays.sort(chars); + + // 将排序后的字符数组转换回字符串，作为哈希表的键 + String key = new String(chars); + + // 如果键不存在于哈希表中，则创建新的列表并添加到哈希表中 + if (!map.containsKey(key)) { + map.put(key, new ArrayList<>()); + } + + // 将原始字符串添加到对应的列表中 + map.get(key).add(str); + } + + // 返回哈希表中所有值的列表 + return new ArrayList<>(map.values()); + } +``` + +- 时间复杂度：$O(nklogk)$，其中 n 是 strs 中的字符串的数量，k 是 strs 中的字符串的的最大长度。需要遍历 n 个字符串，对于每个字符串，需要 O(klogk) 的时间进行排序以及 O(1) 的时间更新哈希表，因此总时间复杂度是 O(nklogk)。 + +- 空间复杂度：$O(nk)$，其中 n 是 strs 中的字符串的数量，k 是 strs 中的字符串的的最大长度。需要用哈希表存储全部字符串 + +### [217. 存在重复元素](https://leetcode-cn.com/problems/contains-duplicate/) + +**🎯 考察频率：简单 | 难度：简单 | 重要性：哈希表基础** + +> 给定一个整数数组，判断是否存在重复元素。如果存在一值在数组中出现至少两次，函数返回 `true` 。如果数组中每个元素都不相同，则返回 `false` 。 + +**💡 核心思路**：哈希表，和两数之和的思路一样 | 或者排序 + +```java +public boolean containsDuplicate(int[] nums){ + Map map = new HashMap<>(); + for(int i=0;i 给定一个未排序的整数数组 `nums` ，找出数字连续的最长序列（不要求序列元素在原数组中连续）的长度。 +> +> 请你设计并实现时间复杂度为 `O(n)` 的算法解决此问题。 +> +> ``` +> 输入：nums = [100,4,200,1,3,2] +> 输出：4 +> 解释：最长数字连续序列是 [1, 2, 3, 4]。它的长度为 4。 +> ``` + +**思路**：哈希表，**每个数都判断一次这个数是不是连续序列的开头那个数** + +```java +public int longestConsecutive(int[] nums) { + Set numSet = new HashSet<>(); + int longestStreak = 0; + + // 将数组中的所有数字添加到哈希表中 + for (int num : nums) { + numSet.add(num); + } + + // 遍历哈希表中的每个数字 + for (int num : numSet) { + // 如果 num-1 不在哈希表中，说明 num 是一个连续序列的起点 + if (!numSet.contains(num - 1)) { + int currentNum = num; + int currentStreak = 1; + + // 检查 num+1, num+2, ... 是否在哈希表中，并计算连续序列的长度 + while (numSet.contains(currentNum + 1)) { + currentNum++; + currentStreak++; + } + + // 更新最长连续序列的长度 + longestStreak = Math.max(longestStreak, currentStreak); + } + } + + return longestStreak; +} +``` + +### [3. 无重复字符的最长子串](https://leetcode.cn/problems/longest-substring-without-repeating-characters/) + +**🎯 考察频率：极高 | 难度：中等 | 重要性：滑动窗口经典** + +> 给定一个字符串 s ，请你找出其中不含有重复字符的最长子串的长度。 + +**💡 核心思路**：滑动窗口 + 哈希表 + +```java +public int lengthOfLongestSubstring(String s) { + Set window = new HashSet<>(); + int left = 0, maxLen = 0; + + for (int right = 0; right < s.length(); right++) { + char c = s.charAt(right); + while (window.contains(c)) { + window.remove(s.charAt(left)); + left++; + } + window.add(c); + maxLen = Math.max(maxLen, right - left + 1); + } + return maxLen; +} +``` + +**⏱️ 复杂度分析**： +- **时间复杂度**：O(n) - 每个字符最多被访问两次（一次by right，一次by left） +- **空间复杂度**：O(min(m,n)) - m是字符集大小，n是字符串长度 + + +--- + +## 三、前缀和技巧类（区间计算利器）📊 + +**💡 核心思想** + +- **累积求和**：预处理前缀和数组，快速计算区间和 +- **差值技巧**：利用前缀和差值解决子数组问题 +- **哈希优化**：结合哈希表优化前缀和查找 + +**🎯 必掌握模板** + +```java +// 前缀和模板 +int[] prefixSum = new int[nums.length + 1]; +for (int i = 0; i < nums.length; i++) { + prefixSum[i + 1] = prefixSum[i] + nums[i]; +} +// 区间[i,j]的和 = prefixSum[j+1] - prefixSum[i] + +// 前缀和+哈希表模板 +Map map = new HashMap<>(); +map.put(0, 1); // 前缀和为0出现1次 +int prefixSum = 0; +for (int num : nums) { + prefixSum += num; + if (map.containsKey(prefixSum - k)) { + count += map.get(prefixSum - k); + } + map.put(prefixSum, map.getOrDefault(prefixSum, 0) + 1); +} +``` + +### [560. 和为 K 的子数组](https://leetcode.cn/problems/subarray-sum-equals-k/) + +**🎯 考察频率：极高 | 难度：中等 | 重要性：前缀和+哈希表经典** + +> 给你一个整数数组 `nums` 和一个整数 `k` ，请你统计并返回 *该数组中和为 `k` 的子数组的个数* 。 +> +> 子数组是数组中元素的连续非空序列。 +> +> ``` +> 输入：nums = [1,2,3], k = 3 +> 输出：2 +> ``` + +思路：**前缀和 + 哈希表** + +1. **遍历数组**，逐步计算前缀和（表示从数组起始位置到当前位置的所有元素之和） `prefixSum`。 +2. 检查哈希表中是否存在 `prefixSum - k`： + - 若存在，累加其出现次数到结果 `count`。 +3. **更新哈希表**：将当前前缀和存入哈希表，若已存在则次数加 1。![](https://assets.leetcode-cn.com/solution-static/560/3.PNG) + +```java +public int subarraySum(int[] nums, int k) { + int count = 0; + int prefixSum = 0; + Map sumMap = new HashMap<>(); + sumMap.put(0, 1); // 初始化前缀和为0的计数为1 + + for (int num : nums) { + prefixSum += num; + // 检查是否存在前缀和为 prefixSum - k 的键 + if (sumMap.containsKey(prefixSum - k)) { + count += sumMap.get(prefixSum - k); + } + // 更新当前前缀和的次数 + sumMap.put(prefixSum, sumMap.getOrDefault(prefixSum, 0) + 1); + } + return count; +} +``` + +- **时间复杂度**：O(n)，每个元素遍历一次。 +- **空间复杂度**：O(n)，哈希表最多存储 n 个不同的前缀和。 + +### [238. 除自身以外数组的乘积](https://leetcode.cn/problems/product-of-array-except-self/) + +**🎯 考察频率：极高 | 难度：中等 | 重要性：前缀乘积** + +> 给你一个整数数组 nums，返回数组 answer ，其中 answer[i] 等于 nums 中除 nums[i] 之外其余各元素的乘积。 + +**💡 核心思路**：左右乘积数组 + +```java +public int[] productExceptSelf(int[] nums) { + int n = nums.length; + int[] answer = new int[n]; + + // 计算左侧所有元素的乘积 + answer[0] = 1; + for (int i = 1; i < n; i++) { + answer[i] = nums[i - 1] * answer[i - 1]; + } + + // 计算右侧所有元素的乘积并更新答案 + int rightProduct = 1; + for (int i = n - 1; i >= 0; i--) { + answer[i] = answer[i] * rightProduct; + rightProduct *= nums[i]; + } + + return answer; +} +``` + + +--- + + + +## 四、排序与搜索类（有序数据处理）🔄 + +**💡 核心思想** + +- **预排序**：先排序再处理，降低问题复杂度 +- **二分查找**：在有序数组中快速定位 +- **快速选择**：基于快排的选择算法 +- **区间处理**：排序后处理重叠区间问题 + +**🎯 必掌握模板** + +```java +// 排序+双指针模板 +Arrays.sort(nums); +int left = 0, right = nums.length - 1; +while (left < right) { + // 处理逻辑 +} + +// 二分查找模板 +int left = 0, right = nums.length - 1; +while (left <= right) { + int mid = left + (right - left) / 2; + if (nums[mid] == target) return mid; + else if (nums[mid] < target) left = mid + 1; + else right = mid - 1; +} +``` + +### [31. 下一个排列](https://leetcode.cn/problems/next-permutation/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：字典序算法** + +> 整数数组的一个 **排列** 就是将其所有成员以序列或线性顺序排列。 +> +> 例如，arr = [1,2,3] ，以下这些都可以视作 arr 的排列：[1,2,3]、[1,3,2]、[3,1,2]、[2,3,1] 。 +> 整数数组的 **下一个排列** 是指其整数的下一个字典序更大的排列。更正式地，如果数组的所有排列根据其字典顺序从小到大排列在一个容器中，那么数组的 **下一个排列** 就是在这个有序容器中排在它后面的那个排列。如果不存在下一个更大的排列，那么这个数组必须重排为字典序最小的排列（即，其元素按升序排列）。 +> +> - 例如，arr = [1,2,3] 的下一个排列是 [1,3,2] 。 +> - 类似地，arr = [2,3,1] 的下一个排列是 [3,1,2] 。 +> - 而 arr = [3,2,1] 的下一个排列是 [1,2,3] ，因为 [3,2,1] 不存在一个字典序更大的排列。 +> +> 给你一个整数数组 nums ，找出 nums 的下一个排列。 +> +> **题干的意思就是**：找出这个数组排序出的所有数中，刚好比当前数大的那个数 +> +> 必须 **原地** 修改，只允许使用额外常数空间。 +> +> ``` +> 输入：nums = [1,2,3] +> 输出：[1,3,2] +> ``` +> +> ``` +> 输入：nums = [3,2,1] +> 输出：[1,2,3] +> ``` + +**💡 核心思路**：两遍扫描算法 + +1. **从右往左找第一个逆序对**：找到 nums[i] < nums[i+1] 的位置 i +2. **从右往左找第一个大于nums[i]的数**：找到位置 j，满足 nums[j] > nums[i] +3. **交换 nums[i] 和 nums[j]** +4. **反转 i+1 到末尾的部分**：使其变为最小的字典序 + +```java +public void nextPermutation(int[] nums) { + // 步骤1：从右向左找到第一个比右边元素小的元素 + int i = nums.length - 2; + while (i >= 0 && nums[i] >= nums[i + 1]) { + i--; + } + + // 步骤2：如果找到这样的元素，再从右向左找第一个比它大的元素并交换 + if (i >= 0) { + int j = nums.length - 1; + while (nums[j] <= nums[i]) { + j--; + } + swap(nums, i, j); + } + + // 步骤3：反转i后面的元素，使其成为最小排列 + reverse(nums, i + 1); +} + +// 交换数组中两个位置的元素 +private void swap(int[] nums, int i, int j) { + int temp = nums[i]; + nums[i] = nums[j]; + nums[j] = temp; +} + +// 反转数组中从start位置到末尾的元素 +private void reverse(int[] nums, int start) { + int left = start; + int right = nums.length - 1; + while (left < right) { + swap(nums, left, right); + left++; + right--; + } +} +``` + +**⏱️ 复杂度分析**： +- **时间复杂度**：O(n) - 最多遍历数组3次 +- **空间复杂度**：O(1) - 只使用了常数级别的额外空间 + + + +### [56. 合并区间](https://leetcode.cn/problems/merge-intervals/) + +> 以数组 `intervals` 表示若干个区间的集合，其中单个区间为 `intervals[i] = [starti, endi]` 。请你合并所有重叠的区间，并返回 *一个不重叠的区间数组，该数组需恰好覆盖输入中的所有区间* 。 +> +> ``` +> 输入：intervals = [[1,3],[2,6],[8,10],[15,18]] +> 输出：[[1,6],[8,10],[15,18]] +> 解释：区间 [1,3] 和 [2,6] 重叠, 将它们合并为 [1,6]. +> ``` + +思路： + +1. **排序** 将区间按起始位置升序排列，使可能重叠的区间相邻，便于合并操作 + + ![](https://pic.leetcode-cn.com/50417462969bd13230276c0847726c0909873d22135775ef4022e806475d763e-56-2.png) + +2. **合并逻辑** + + **遍历判断**：依次遍历排序后的区间，比较当前区间与结果列表中的最后一个区间是否重叠： + + - 重叠条件：当前区间起始 ≤ 结果列表中最后一个区间的结束。 + - 合并操作：更新结果列表中最后一个区间的结束值为两者的最大值。 + - **非重叠条件**：直接将当前区间加入结果列表 + +```java +public int[][] merge(int[][] intervals) { + // 边界条件：空数组直接返回空 + if (intervals.length == 0) return new int[0][]; + + // 按区间起始点升序排序 + Arrays.sort(intervals, (a, b) -> Integer.compare(a[0], b[0])); + + // 结果列表 + List merged = new ArrayList<>(); + merged.add(intervals[0]); // 初始化第一个区间 + + for (int i = 1; i < intervals.length; i++) { + int[] last = merged.get(merged.size() - 1); + int[] current = intervals[i]; + + if (current[0] <= last[1]) { // 重叠，合并区间 + last[1] = Math.max(last[1], current[1]); // 更新结束值为较大值 + } else { // 不重叠，直接添加 + merged.add(current); + } + } + + return merged.toArray(new int[merged.size()][]); // 转换为二维数组 +} +``` + +### [215. 数组中的第K个最大元素](https://leetcode-cn.com/problems/kth-largest-element-in-an-array/) + +**🎯 考察频率：极高 | 难度：中等 | 重要性：快速选择算法** + +> 给定整数数组 `nums` 和整数 `k`，请返回数组中第 `k` 个最大的元素。 + +**💡 核心思路**：快速选择算法（基于快排的分治思想） + +```java +class Solution { + Random random = new Random(); + + public int findKthLargest(int[] nums, int k) { + return quickSelect(nums, 0, nums.length - 1, nums.length - k); + } + + public int quickSelect(int[] a, int l, int r, int index) { + int q = randomPartition(a, l, r); + if (q == index) { + return a[q]; + } else { + return q < index ? quickSelect(a, q + 1, r, index) : quickSelect(a, l, q - 1, index); + } + } + + public int randomPartition(int[] a, int l, int r) { + int i = random.nextInt(r - l + 1) + l; + swap(a, i, r); + return partition(a, l, r); + } + + public int partition(int[] a, int l, int r) { + int x = a[r], i = l - 1; + for (int j = l; j < r; ++j) { + if (a[j] <= x) { + swap(a, ++i, j); + } + } + swap(a, i + 1, r); + return i + 1; + } + + public void swap(int[] a, int i, int j) { + int temp = a[i]; + a[i] = a[j]; + a[j] = temp; + } +} +``` + +### [287. 寻找重复数](https://leetcode.cn/problems/find-the-duplicate-number/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：二分查找应用** + +> 给定一个包含 `n + 1` 个整数的数组 `nums` ，其数字都在 `[1, n]` 范围内，可知至少存在一个重复的整数。 + +**💡 核心思路**：二分查找（基于抽屉原理） + +- 利用二分查找的思路，统计数组中小于等于中间值的数的个数 +- 如果个数大于中间值，说明重复数在左半部分，否则在右半部分 + +```java +public int findDuplicate(int[] nums) { + int left = 1; + int right = nums.length - 1; + while (left < right) { + int mid = left + (right - left) / 2; + int count = 0; + for (int num : nums) { + if (num <= mid) { + count++; + } + } + if (count > mid) { + right = mid; + } else { + left = mid + 1; + } + } + return left; +} +``` + +### [33. 搜索旋转排序数组](https://leetcode.cn/problems/search-in-rotated-sorted-array/) + +**🎯 考察频率：极高 | 难度：中等 | 重要性：二分查找变体** + +> 整数数组 nums 按升序排列，数组中的值互不相同。在传递给函数之前，nums 在预先未知的某个下标 k 上进行了旋转。 + +**💡 核心思路**：二分查找，判断哪一边是有序的 + +```java +public int search(int[] nums, int target) { + int left = 0, right = nums.length - 1; + + while (left <= right) { + int mid = left + (right - left) / 2; + + if (nums[mid] == target) { + return mid; + } + + // 判断左半部分是否有序 + if (nums[left] <= nums[mid]) { + if (nums[left] <= target && target < nums[mid]) { + right = mid - 1; + } else { + left = mid + 1; + } + } else { // 右半部分有序 + if (nums[mid] < target && target <= nums[right]) { + left = mid + 1; + } else { + right = mid - 1; + } + } + } + + return -1; +} +``` + +--- + +## 五、数学技巧类（巧妙算法优化）🧮 + +**💡 核心思想** + +- **位运算技巧**：利用异或、与运算等特性 +- **数学性质**：利用数组特殊性质简化问题 +- **原地标记**：不使用额外空间的标记方法 + +**🎯 必掌握模板** + +```java +// 异或运算模板（找单独元素） +int result = 0; +for (int num : nums) { + result ^= num; // 相同元素异或为0，单独元素保留 +} + +// 原地标记模板（数组作为哈希表） +for (int i = 0; i < nums.length; i++) { + int index = Math.abs(nums[i]) - 1; + if (nums[index] > 0) { + nums[index] = -nums[index]; // 标记已访问 + } +} +``` + +### [136. 只出现一次的数字](https://leetcode.cn/problems/single-number/) + +**🎯 考察频率：高 | 难度：简单 | 重要性：异或运算经典** + +> 给你一个非空整数数组 `nums` ，除了某个元素只出现一次以外，其余每个元素均出现两次。找出那个只出现了一次的元素。 + +**💡 核心思路**：异或运算 + +- 初始化结果：result = 0, 任何数和 0 异或都是它本身 +- 由于异或运算的性质，出现两次的数字会被互相抵消，最终 result 就是只出现一次的数字 + +```java +public int singleNumber(int[] nums) { + int result = 0; + for (int num : nums) { + result ^= num; + } + return result; +} +``` + +### [169. 多数元素](https://leetcode.cn/problems/majority-element/) + +**🎯 考察频率：高 | 难度：简单 | 重要性：Boyer-Moore算法** + +> 给定一个大小为 `n` 的数组 `nums` ，返回其中的多数元素。多数元素是指在数组中出现次数大于 `⌊ n/2 ⌋` 的元素。 + +**💡 核心思路**：排序后取中间元素 + +```java +class Solution { + public int majorityElement(int[] nums) { + Arrays.sort(nums); + return nums[nums.length / 2]; + } +} +``` + +### [448. 找到所有数组中消失的数字](https://leetcode-cn.com/problems/find-all-numbers-disappeared-in-an-array/) + +**🎯 考察频率：中等 | 难度：简单 | 重要性：原地标记** + +> 给你一个含 n 个整数的数组 nums ，其中 nums[i] 在区间 [1, n] 内。请你找出所有在 [1, n] 范围内但没有出现在 nums 中的数字。 + +**💡 核心思路**：原地标记法 + +```java +public static List findNumbers(int[] nums){ + List list = new ArrayList<>(); + int[] x = new int[nums.length + 1]; + // 新建一个数组 x，用来做哈希表，占位标记。 + // 注意：长度是 nums.length + 1，因为我们希望下标范围 0..n， + // 这样 1..n 的数字就能直接映射到数组下标。 + + // 1. 遍历 nums，把出现的数字对应位置标记 + for (int i = 0; i < nums.length; i++) { + x[nums[i]]++; // 出现一次就加一 + } + + // 2. 再遍历 x，找到没出现过的数字 + for (int i = 1; i < x.length; i++) { // 注意从 1 开始，因为 0 不是合法数字 + if(x[i] == 0){ + list.add(i); // 没出现的数字加入结果集 + } + } + + return list; +} +``` + +### [41. 缺失的第一个正数](https://leetcode.cn/problems/first-missing-positive/) + +**🎯 考察频率：高 | 难度：困难 | 重要性：原地哈希** + +> 给你一个未排序的整数数组 nums ，请你找出其中没有出现的最小的正整数。 + +**💡 核心思路**：原地哈希，将数组本身作为哈希表 + +```java +public int firstMissingPositive(int[] nums) { + int n = nums.length; + + // 将不在范围[1,n]的数字置为n+1 + for (int i = 0; i < n; i++) { + if (nums[i] <= 0 || nums[i] > n) { + nums[i] = n + 1; + } + } + + // 用数组下标标记数字是否出现 + for (int i = 0; i < n; i++) { + int num = Math.abs(nums[i]); + if (num <= n) { + nums[num - 1] = -Math.abs(nums[num - 1]); + } + } + + // 找第一个正数的位置 + for (int i = 0; i < n; i++) { + if (nums[i] > 0) { + return i + 1; + } + } + + return n + 1; +} +``` + +--- + +## 六、原地算法类（空间优化大师）⚡ + +**💡 核心思想** + +- **双指针交换**：使用双指针原地重排元素 +- **环形替换**：循环移动元素到正确位置 +- **分区思想**：将数组分为不同区域处理 + +### [88. 合并两个有序数组](https://leetcode-cn.com/problems/merge-sorted-array/) + +**🎯 考察频率：高 | 难度：简单 | 重要性：原地合并** + +> 给你两个按非递减顺序排列的整数数组 nums1 和 nums2，另有两个整数 m 和 n ，分别表示 nums1 和 nums2 中的元素数目。 +> +> 注意： +> +> - `nums1` 的长度为 `m + n`，其中前 `m` 个元素表示有效部分，后 `n` 个元素为空（用来存放 `nums2` 的元素）。 +> - `nums2` 的长度为 `n`。 +> +> ``` +> 输入: +> nums1 = [1,2,3,0,0,0], m = 3 +> nums2 = [2,5,6], n = 3 +> +> 输出: +> [1,2,2,3,5,6] +> ``` + +**💡 核心思路**：直接合并后排序 或 从后往前双指针合并 + +```java +public void merge(int[] nums1, int m, int[] nums2, int n) { + for (int i = 0; i != n; ++i) { + nums1[m + i] = nums2[i]; + } + Arrays.sort(nums1); +} +``` + +### [189. 旋转数组](https://leetcode.cn/problems/rotate-array/) + +**🎯 考察频率：中等 | 难度：中等 | 重要性：原地算法** + +> 给定一个数组，将数组中的元素向右移动 k 个位置，其中 k 是非负数。 + +**💡 核心思路**：三次反转 + +```java +public void rotate(int[] nums, int k) { + k %= nums.length; + reverse(nums, 0, nums.length - 1); + reverse(nums, 0, k - 1); + reverse(nums, k, nums.length - 1); +} + +private void reverse(int[] nums, int start, int end) { + while (start < end) { + int temp = nums[start]; + nums[start] = nums[end]; + nums[end] = temp; + start++; + end--; + } +} +``` + +--- + +## 七、综合应用类（面试高频）🎯 + +**💡 核心思想** + +- **多技巧结合**：综合运用多种数组处理技巧 +- **实际应用场景**：贴近实际开发中的问题 + +### [53. 最大子数组和](https://leetcode-cn.com/problems/maximum-subarray/) + +**🎯 考察频率：极高 | 难度：简单 | 重要性：动态规划经典** + +> 给你一个整数数组 `nums` ，请你找出一个具有最大和的连续子数组（子数组最少包含一个元素），返回其最大和。 +> +> **子数组** 是数组中的一个连续部分。 +> +> ``` +> 输入：nums = [-2,1,-3,4,-1,2,1,-5,4] +> 输出：6 +> 解释：连续子数组 [4,-1,2,1] 的和最大，为 6 。 +> ``` + +**思路**：「连续」子数组，题目要求的是返回结果，用 [动态规划、分治] + +```java +public static int maxSubArray(int[] nums) { + //特判 + if (nums == null || nums.length == 0) { + return 0; + } + //初始化 + int length = nums.length; + int[] dp = new int[length]; + // 初始值,只有一个元素的时候最大和即它本身 + dp[0] = nums[0]; + int ans = nums[0]; + // 状态转移 + for (int i = 1; i < length; i++) { + // 取当前元素的值 和 当前元素的值加上一次结果的值 中最大数 + dp[i] = Math.max(nums[i], dp[i - 1] + nums[i]); + // 和最大数对比 取大 + ans = Math.max(ans, dp[i]); + } + return ans; +} + +//优化版 +public int maxSubArray(int[] nums) { + int pre = 0, maxAns = nums[0]; + for (int x : nums) { + pre = Math.max(pre + x, x); + maxAns = Math.max(maxAns, pre); + } + return maxAns; +} +``` + +### [121. 买卖股票的最佳时机](https://leetcode-cn.com/problems/best-time-to-buy-and-sell-stock/) + +**🎯 考察频率：极高 | 难度：简单 | 重要性：动态规划入门** + +> 给定一个数组 prices ，它的第 i 个元素 prices[i] 表示一支给定股票第 i 天的价格。你只能选择某一天买入这只股票，并选择在未来的某一个不同的日子卖出该股票。 + +**💡 核心思路**：一次遍历，记录最低价格和最大利润 + +```java +public int maxProfit(int[] prices) { + int minPrice = Integer.MAX_VALUE; + int maxProfit = 0; + + for (int price : prices) { + if (price < minPrice) { + minPrice = price; + } else if (price - minPrice > maxProfit) { + maxProfit = price - minPrice; + } + } + + return maxProfit; +} +``` + +### [152. 乘积最大子数组](https://leetcode.cn/problems/maximum-product-subarray/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：动态规划变体** + +> 给你一个整数数组 nums ，请你找出数组中乘积最大的非空连续子数组，并返回该子数组所对应的乘积。 + +**💡 核心思路**：动态规划，同时维护最大值和最小值 + +```java +public int maxProduct(int[] nums) { + int maxSoFar = nums[0]; + int minSoFar = nums[0]; + int result = maxSoFar; + + for (int i = 1; i < nums.length; i++) { + int curr = nums[i]; + int tempMaxSoFar = Math.max(curr, Math.max(maxSoFar * curr, minSoFar * curr)); + minSoFar = Math.min(curr, Math.min(maxSoFar * curr, minSoFar * curr)); + + maxSoFar = tempMaxSoFar; + result = Math.max(maxSoFar, result); + } + + return result; +} +``` + +### [78. 子集](https://leetcode.cn/problems/subsets/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：回溯算法** + +> 给你一个整数数组 nums ，数组中的元素互不相同。返回该数组所有可能的子集（幂集）。 + +**💡 核心思路**：回溯算法 + +```java +public List> subsets(int[] nums) { + List> result = new ArrayList<>(); + backtrack(result, new ArrayList<>(), nums, 0); + return result; +} + +private void backtrack(List> result, List tempList, int[] nums, int start) { + result.add(new ArrayList<>(tempList)); + for (int i = start; i < nums.length; i++) { + tempList.add(nums[i]); + backtrack(result, tempList, nums, i + 1); + tempList.remove(tempList.size() - 1); + } +} +``` + +### [55. 跳跃游戏](https://leetcode.cn/problems/jump-game/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：贪心算法** + +> 给定一个非负整数数组 nums ，你最初位于数组的第一个下标。数组中的每个元素代表你在该位置可以跳跃的最大长度。 + +**💡 核心思路**：贪心算法，维护能到达的最远位置 + +```java +public boolean canJump(int[] nums) { + int maxReach = 0; + for (int i = 0; i < nums.length; i++) { + if (i > maxReach) { + return false; + } + maxReach = Math.max(maxReach, i + nums[i]); + } + return true; +} +``` + +### [45. 跳跃游戏 II](https://leetcode.cn/problems/jump-game-ii/) + +**🎯 考察频率：中等 | 难度：中等 | 重要性：贪心优化** + +> 给定一个长度为 n 的 0 索引整数数组 nums。初始位置为 nums[0]。每个元素 nums[i] 表示从索引 i 向前跳转的最大长度。 + +**💡 核心思路**：贪心算法，在能跳到的范围内选择能跳得最远的 + +```java +public int jump(int[] nums) { + int jumps = 0; + int currentEnd = 0; + int farthest = 0; + + for (int i = 0; i < nums.length - 1; i++) { + farthest = Math.max(farthest, i + nums[i]); + + if (i == currentEnd) { + jumps++; + currentEnd = farthest; + } + } + + return jumps; +} +``` + + +### [14. 最长公共前缀](https://leetcode.cn/problems/longest-common-prefix/) + +**🎯 考察频率：中等 | 难度：简单 | 重要性：字符串处理** + +> 编写一个函数来查找字符串数组中的最长公共前缀。如果不存在公共前缀，返回空字符串 `""`。 + +**💡 核心思路**：逐个字符比较 + +```java +public String longestCommonPrefix(String[] strs) { + if (strs == null || strs.length == 0) { + return ""; + } + + String prefix = strs[0]; + for (int i = 1; i < strs.length; i++) { + int j = 0; + while (j < prefix.length() && j < strs[i].length() && prefix.charAt(j) == strs[i].charAt(j)) { + j++; + } + prefix = prefix.substring(0, j); + if (prefix.isEmpty()) { + return ""; + } + } + return prefix; +} +``` + +### [54. 螺旋矩阵](https://leetcode.cn/problems/spiral-matrix/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：矩阵遍历** + +> 给你一个 m 行 n 列的矩阵 matrix ，请按照顺时针螺旋顺序，返回矩阵中的所有元素。 + +**💡 核心思路**：模拟螺旋过程，维护边界 + +```java +public List spiralOrder(int[][] matrix) { + List result = new ArrayList<>(); + if (matrix.length == 0) return result; + + int top = 0, bottom = matrix.length - 1; + int left = 0, right = matrix[0].length - 1; + + while (top <= bottom && left <= right) { + // 从左到右 + for (int j = left; j <= right; j++) { + result.add(matrix[top][j]); + } + top++; + + // 从上到下 + for (int i = top; i <= bottom; i++) { + result.add(matrix[i][right]); + } + right--; + + if (top <= bottom) { + // 从右到左 + for (int j = right; j >= left; j--) { + result.add(matrix[bottom][j]); + } + bottom--; + } + + if (left <= right) { + // 从下到上 + for (int i = bottom; i >= top; i--) { + result.add(matrix[i][left]); + } + left++; + } + } + + return result; +} +``` + +--- + +## 🚀 面试前15分钟速记表 + +### 核心模板（必背） + +```java +// 1. 双指针模板 +int left = 0, right = nums.length - 1; +while (left < right) { + // 处理逻辑 +} + +// 2. 滑动窗口模板 +int left = 0; +for (int right = 0; right < nums.length; right++) { + // 扩展窗口 + while (窗口需要收缩) { + // 收缩窗口 + left++; + } +} + +// 3. 前缀和模板 +int[] prefixSum = new int[nums.length + 1]; +for (int i = 0; i < nums.length; i++) { + prefixSum[i + 1] = prefixSum[i] + nums[i]; +} +``` + +### 题型速查表 + +| 题型分类 | 核心技巧 | 高频题目 | 记忆口诀 | 难度 | +|----------|----------|----------|----------|------| +| **双指针类** | 左右指针/快慢指针 | 两数之和、三数之和、盛水容器、移动零、颜色分类 | 左右夹逼找目标，快慢追踪解环题 | ⭐⭐⭐ | +| **哈希表类** | 空间换时间 | 字母异位词分组、最长连续序列、存在重复元素 | 空间换时间来优化，常数查找是关键 | ⭐⭐⭐ | +| **前缀和类** | 累积求和 | 和为K的子数组、最大子数组和、除自身以外数组的乘积 | 累积求和建数组，区间相减得答案 | ⭐⭐⭐⭐ | +| **排序搜索** | 预排序/二分 | 合并区间、第K大元素、寻找重复数、搜索旋转数组 | 有序数组用二分，对数时间最高效 | ⭐⭐⭐⭐ | +| **原地算法** | 双指针交换 | 移动零、颜色分类、合并有序数组、旋转数组 | 双指针巧妙来交换，空间复杂度为一 | ⭐⭐⭐⭐ | +| **数学技巧** | 位运算/性质 | 只出现一次的数字、多数元素、消失的数字 | 数学性质巧运用，位运算显神通 | ⭐⭐⭐ | +| **综合应用** | 多技巧结合 | 买卖股票、跳跃游戏、子集、乘积最大子数组 | 多技巧组合显神通，复杂问题巧拆解 | ⭐⭐⭐⭐⭐ | + +### 核心题目优先级（面试前重点复习） + +1. **两数之和** - 哈希表基础，必须熟练掌握 +2. **三数之和** - 双指针经典，排序+双指针 +3. **最大子数组和** - 动态规划入门，Kadane算法 +4. **合并区间** - 排序应用，区间处理 +5. **盛最多水的容器** - 双指针对撞，贪心思想 +6. **颜色分类** - 三指针技巧，荷兰国旗问题 +7. **搜索旋转排序数组** - 二分查找变体 +8. **除自身以外数组的乘积** - 前缀乘积，空间优化 + +### 常见陷阱提醒 + +- ⚠️ **数组越界**：`i < nums.length`，确保索引在有效范围内 +- ⚠️ **整数溢出**：使用long或检查边界，特别是乘积运算 +- ⚠️ **空数组处理**：`if (nums == null || nums.length == 0)` +- ⚠️ **双指针边界**：确保`left < right`，避免死循环 +- ⚠️ **滑动窗口**：正确维护窗口边界，注意收缩条件 +- ⚠️ **前缀和**：注意数组长度为n+1，处理边界情况 + +### 时间复杂度总结 + +- **遍历类**：O(n) - 一次遍历 +- **双指针**：O(n) - 同时移动或对撞指针 +- **排序类**：O(n log n) - 基于比较的排序 +- **哈希表**：O(n) - 平均情况下常数时间查找 +- **二分查找**：O(log n) - 每次缩小一半搜索空间 + +### 面试答题套路 + +1. **理解题意**：确认输入输出，边界情况，时间空间复杂度要求 +2. **选择方法**：根据题型选择对应技巧（双指针、哈希表、前缀和等） +3. **画图分析**：手动模拟小规模测试用例，验证思路 +4. **编码实现**：套用对应模板，注意边界条件处理 +5. **优化分析**：考虑时间空间复杂度优化可能性 +6. **测试验证**：空数组、单元素、正常情况、边界情况 + +**最重要的一个技巧就是，你得行动，写起来** diff --git a/docs/data-structure-algorithms/soultion/Binary-Tree-Solution.md b/docs/data-structure-algorithms/soultion/Binary-Tree-Solution.md new file mode 100755 index 0000000000..76d93578a9 --- /dev/null +++ b/docs/data-structure-algorithms/soultion/Binary-Tree-Solution.md @@ -0,0 +1,2214 @@ +--- +title: 二叉树通关秘籍：LeetCode 热题 100 全解析 +date: 2025-01-08 +tags: + - Binary Tree +categories: leetcode +--- + +![](https://img.starfish.ink/leetcode/leetcode-banner.png) + +> 二叉树作为数据结构中的常青树，在算法面试中出现的频率居高不下。 +> +> 有人说刷题要先刷二叉树，培养思维。目前还没培养出来，感觉分类刷都差不多。 +> +> 我把二叉树相关题目进行了下分类。 +> +> 当然，在做二叉树题目时候，第一想到的应该是用 **递归** 来解决。 + +### 📋 分类索引 + +1. **🌳 遍历基础类**：[二叉树的前序遍历](#_144-二叉树的前序遍历)、[二叉树的中序遍历](#_94-二叉树的中序遍历)、[二叉树的后序遍历](#_145-二叉树的后序遍历)、[二叉树的层序遍历](#_102-二叉树的层序遍历)、[锯齿形层序遍历](#_103-二叉树的锯齿形层序遍历)、[完全二叉树的节点个数](#_222-完全二叉树的节点个数)、[在每个树行中找最大值](#_515-在每个树行中找最大值)、[二叉树最大宽度](#_662-二叉树最大宽度) + +2. **🔍 查找路径类**：[二叉树的最大深度](#_104-二叉树的最大深度)、[二叉树的最小深度](#_111-二叉树的最小深度)、[路径总和](#_112-路径总和)、[路径总和II](#_113-路径总和-ii)、[二叉树的右视图](#_199-二叉树的右视图)、[路径总和III](#_437-路径总和-iii)、[二叉树的最近公共祖先](#_236-二叉树的最近公共祖先)、[二叉树中的最大路径和](#_124-二叉树中的最大路径和) + +3. **🏗️ 构造变换类**：[从前序与中序遍历序列构造二叉树](#_105-从前序与中序遍历序列构造二叉树)、[翻转二叉树](#_226-翻转二叉树)、[对称二叉树](#_101-对称二叉树)、[相同的树](#_100-相同的树) + +4. **⚖️ 平衡验证类**：[平衡二叉树](#_110-平衡二叉树)、[验证二叉搜索树](#_98-验证二叉搜索树)、[二叉搜索树中第K小的元素](#_230-二叉搜索树中第K小的元素) + +5. **🎯 搜索树操作类**：[二叉搜索树的最近公共祖先](#_235-二叉搜索树的最近公共祖先)、[删除二叉搜索树中的节点](#_450-删除二叉搜索树中的节点)、[将有序数组转换为二叉搜索树](#_108-将有序数组转换为二叉搜索树) + +6. **📊 树形DP类**：[打家劫舍III](#_337-打家劫舍-iii)、[二叉树的直径](#_543-二叉树的直径)、[另一棵树的子树](#_572-另一棵树的子树) + +7. **🔄 序列化类**：[二叉树的序列化与反序列化](#_297-二叉树的序列化与反序列化) + +8. **🚀 进阶技巧类**：[二叉树展开为链表](#_114-二叉树展开为链表)、[填充每个节点的下一个右侧节点指针](#_116-填充每个节点的下一个右侧节点指针)、[二叉树的完全性检验](#_958-二叉树的完全性检验)、[填充每个节点的下一个右侧节点指针II](#_117-填充每个节点的下一个右侧节点指针-ii) + +## 🎯 核心考点概览 + +- **遍历技巧**：前序、中序、后序、层序遍历 +- **递归思维**：分治思想、递归终止条件 +- **路径问题**：根到叶子路径、任意路径求和 +- **树的构造**：根据遍历序列重建二叉树 +- **树的变换**：翻转、对称、平衡判断 +- **搜索树性质**：BST的查找、插入、删除 +- **动态规划**：树形DP的经典应用 + +## 💡 解题万能模板 + +### 🔄 递归遍历模板 + +```java +// 递归遍历通用模板 +public void traverse(TreeNode root) { + if (root == null) return; + + // 前序位置 - 在递归之前 + doSomething(root.val); + + traverse(root.left); + + // 中序位置 - 在左右递归之间 + doSomething(root.val); + + traverse(root.right); + + // 后序位置 - 在递归之后 + doSomething(root.val); +} +``` + +### 🔍 层序遍历模板 + +```java +// BFS层序遍历模板 +public void levelOrder(TreeNode root) { + if (root == null) return; + + Queue queue = new LinkedList<>(); + queue.offer(root); + + while (!queue.isEmpty()) { + int size = queue.size(); + for (int i = 0; i < size; i++) { + TreeNode node = queue.poll(); + // 处理当前节点 + doSomething(node.val); + + if (node.left != null) queue.offer(node.left); + if (node.right != null) queue.offer(node.right); + } + } +} +``` + +### 🎯 分治思想模板 + +```java +// 分治思想模板 +public ResultType divide(TreeNode root) { + if (root == null) return nullResult; + + // 分：递归处理左右子树 + ResultType leftResult = divide(root.left); + ResultType rightResult = divide(root.right); + + // 治：合并左右结果 + ResultType result = merge(leftResult, rightResult, root); + + return result; +} +``` + +## 🛡️ 边界检查清单 + +- ✅ 空树处理：`root == null` +- ✅ 单节点树：`root.left == null && root.right == null` +- ✅ 递归终止：明确递归出口条件 +- ✅ 返回值检查：确保每个递归分支都有返回值 + +## 💡 记忆口诀 + +- **遍历顺序**："前序根左右，中序左根右，后序左右根，层序逐层走" +- **递归思维**："递归三要素，终止、递推、返回值" +- **路径问题**："路径用回溯，求和用递归" +- **树的构造**："前中后序定根节点，中序划分左右树" +- **搜索树**："左小右大有序性，中序遍历是关键" +- **平衡判断**："高度差不超过一，递归检查每个节点" +- **最值问题**："深度优先找路径，广度优先找最近" + +--- + +## 🌳 一、遍历基础类（核心中的核心） + +### 💡 核心思想 + +- **递归遍历**：利用函数调用栈实现深度优先 +- **迭代遍历**：手动维护栈模拟递归过程 +- **层序遍历**：队列实现广度优先搜索 + +### [144. 二叉树的前序遍历](https://leetcode.cn/problems/binary-tree-preorder-traversal/) + +**🎯 考察频率：极高 | 难度：简单 | 重要性：遍历基础** + +> 前序遍历：根→左→右，`[1,null,2,3]` → `[1,2,3]` + +**💡 核心思路**：递归 + 迭代两种实现 + +- 递归：root → left → right 的顺序访问 +- 迭代：使用栈模拟递归过程 + +**🔑 记忆技巧**： + +- **口诀**："前序根左右，栈中右先入" +- **形象记忆**：像读书一样，先看标题（根）再看内容（左右） + +```java +// 递归解法 +public List preorderTraversal(TreeNode root) { + List result = new ArrayList<>(); + preorder(root, result); + return result; +} + +private void preorder(TreeNode root, List result) { + if (root == null) return; + + result.add(root.val); // 根 + preorder(root.left, result); // 左 + preorder(root.right, result);// 右 +} +``` + +**🔧 迭代解法**： + +```java +public List preorderTraversal(TreeNode root) { + List result = new ArrayList<>(); + if (root == null) return result; + + Stack stack = new Stack<>(); + stack.push(root); + + while (!stack.isEmpty()) { + TreeNode node = stack.pop(); + result.add(node.val); + + // 先入右子树，再入左子树（栈是后进先出） + if (node.right != null) stack.push(node.right); + if (node.left != null) stack.push(node.left); + } + + return result; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 每个节点访问一次 +- **空间复杂度**：O(h) - 递归栈深度，h为树高 + +### [94. 二叉树的中序遍历](https://leetcode.cn/problems/binary-tree-inorder-traversal/) + +**🎯 考察频率：极高 | 难度：简单 | 重要性：BST基础** + +> 中序遍历：左→根→右，`[1,null,2,3]` → `[1,3,2]` + +**💡 核心思路**：左→根→右的访问顺序 + +- 对于BST，中序遍历得到有序序列 +- 迭代实现需要先处理完所有左子树 + +**🔑 记忆技巧**： + +- **口诀**："中序左根右，BST变有序" +- **形象记忆**：像中国人读书从左到右的顺序 + +```java +// 递归解法 +public List inorderTraversal(TreeNode root) { + List result = new ArrayList<>(); + inorder(root, result); + return result; +} + +private void inorder(TreeNode root, List result) { + if (root == null) return; + + inorder(root.left, result); // 左 + result.add(root.val); // 根 + inorder(root.right, result); // 右 +} +``` + +**🔧 迭代解法**： + +```java +public List inorderTraversal(TreeNode root) { + List result = new ArrayList<>(); + Stack stack = new Stack<>(); + TreeNode current = root; + + while (current != null || !stack.isEmpty()) { + // 一直向左走到底 + while (current != null) { + stack.push(current); + current = current.left; + } + + // 处理栈顶节点 + current = stack.pop(); + result.add(current.val); + + // 转向右子树 + current = current.right; + } + + return result; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 每个节点访问一次 +- **空间复杂度**：O(h) - 递归栈或显式栈的深度 + +### [145. 二叉树的后序遍历](https://leetcode.cn/problems/binary-tree-postorder-traversal/) + +**🎯 考察频率：高 | 难度：简单 | 重要性：分治思想基础** + +> 后序遍历：左→右→根，`[1,null,2,3]` → `[3,2,1]` + +**💡 核心思路**：左→右→根的访问顺序 + +- 后序遍历常用于树的删除、计算等操作 +- 迭代实现相对复杂，需要记录访问状态 + +**🔑 记忆技巧**： + +- **口诀**："后序左右根，删除用后序" +- **形象记忆**：像拆房子，先拆左右房间，最后拆主体 + +```java +// 递归解法 +public List postorderTraversal(TreeNode root) { + List result = new ArrayList<>(); + postorder(root, result); + return result; +} + +private void postorder(TreeNode root, List result) { + if (root == null) return; + + postorder(root.left, result); // 左 + postorder(root.right, result); // 右 + result.add(root.val); // 根 +} +``` + +**🔧 迭代解法（双栈法）**： + +```java +public List postorderTraversal(TreeNode root) { + List result = new ArrayList<>(); + if (root == null) return result; + + Stack stack1 = new Stack<>(); + Stack stack2 = new Stack<>(); + + stack1.push(root); + + while (!stack1.isEmpty()) { + TreeNode node = stack1.pop(); + stack2.push(node); + + if (node.left != null) stack1.push(node.left); + if (node.right != null) stack1.push(node.right); + } + + while (!stack2.isEmpty()) { + result.add(stack2.pop().val); + } + + return result; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 每个节点访问一次 +- **空间复杂度**：O(h) - 递归栈深度 + +### [102. 二叉树的层序遍历](https://leetcode.cn/problems/binary-tree-level-order-traversal/) + +**🎯 考察频率：极高 | 难度：中等 | 重要性：BFS经典** + +> 层序遍历：按层从左到右，`[3,9,20,null,null,15,7]` → `[[3],[9,20],[15,7]]` + +**💡 核心思路**：队列实现BFS + +![](https://pic.leetcode-cn.com/94cd1fa999df0276f1dae77a9cca83f4cabda9e2e0b8571cd9550a8ee3545f56.gif) + +- 使用队列保存每层的节点 +- 记录每层的节点数量，分层处理 + +**🔑 记忆技巧**： + +- **口诀**："层序用队列，按层逐个走" +- **形象记忆**：像楼房一样，一层一层地访问 + +```java +public List> levelOrder(TreeNode root) { + List> result = new ArrayList<>(); + if (root == null) return result; + + Queue queue = new LinkedList<>(); + queue.offer(root); + + while (!queue.isEmpty()) { + int size = queue.size(); + List level = new ArrayList<>(); + + for (int i = 0; i < size; i++) { + TreeNode node = queue.poll(); + level.add(node.val); + + if (node.left != null) queue.offer(node.left); + if (node.right != null) queue.offer(node.right); + } + + result.add(level); + } + + return result; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 每个节点访问一次 +- **空间复杂度**：O(w) - w为树的最大宽度 + +### [103. 二叉树的锯齿形层序遍历](https://leetcode.cn/problems/binary-tree-zigzag-level-order-traversal/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：层序变种** + +> 锯齿形遍历：奇数层从左到右，偶数层从右到左 + +**💡 核心思路**：层序遍历 + 奇偶层判断 + +- 基于普通层序遍历 +- 偶数层需要反转结果 + +**🔑 记忆技巧**： + +- **口诀**："锯齿形遍历，奇正偶反转" + +```java +public List> zigzagLevelOrder(TreeNode root) { + List> result = new ArrayList<>(); + if (root == null) return result; + + Queue queue = new LinkedList<>(); + queue.offer(root); + boolean leftToRight = true; + + while (!queue.isEmpty()) { + int size = queue.size(); + List level = new ArrayList<>(); + + for (int i = 0; i < size; i++) { + TreeNode node = queue.poll(); + level.add(node.val); + + if (node.left != null) queue.offer(node.left); + if (node.right != null) queue.offer(node.right); + } + + if (!leftToRight) { + Collections.reverse(level); + } + + result.add(level); + leftToRight = !leftToRight; + } + + return result; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 每个节点访问一次 +- **空间复杂度**：O(w) - 队列存储的最大宽度 + +### [222. 完全二叉树的节点个数](https://leetcode.cn/problems/count-complete-tree-nodes/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：完全二叉树性质** + +> 计算完全二叉树的节点个数，要求时间复杂度低于O(n) + +**💡 核心思路**：利用完全二叉树性质 + 二分思想 + +- 如果左右子树高度相同，左子树是满二叉树 +- 如果高度不同，右子树是满二叉树 +- 递归计算 + 满二叉树节点数公式：2^h - 1 + +**🔑 记忆技巧**： + +- **口诀**："完全二叉树性质，递归加二分优化" +- **形象记忆**：像找平衡点，一边总是满的 + +```java +public int countNodes(TreeNode root) { + if (root == null) return 0; + + int leftHeight = getHeight(root.left); + int rightHeight = getHeight(root.right); + + if (leftHeight == rightHeight) { + // 左子树是满二叉树 + return (1 << leftHeight) + countNodes(root.right); + } else { + // 右子树是满二叉树 + return (1 << rightHeight) + countNodes(root.left); + } +} + +private int getHeight(TreeNode root) { + int height = 0; + while (root != null) { + height++; + root = root.left; + } + return height; +} +``` + +**🔧 朴素递归解法**： + +```java +public int countNodes(TreeNode root) { + if (root == null) return 0; + return 1 + countNodes(root.left) + countNodes(root.right); +} +``` + +**⏱️ 复杂度分析**： + +- **优化解法**： + - 时间复杂度：O(log²n) - 每次递归高度减半，计算高度O(logn) + - 空间复杂度：O(logn) - 递归栈深度 +- **朴素解法**： + - 时间复杂度：O(n) - 访问每个节点 + - 空间复杂度：O(logn) - 递归栈深度 + +### [515. 在每个树行中找最大值](https://leetcode.cn/problems/find-largest-value-in-each-tree-row/) + +**🎯 考察频率：中等 | 难度：中等 | 重要性：层序遍历变种** + +> 返回树每一层的最大值：`[1,3,2,5,3,null,9]` → `[1,3,9]` + +**💡 核心思路**：层序遍历 + 每层求最大值 + +- BFS遍历每一层 +- 记录每层的最大值 + +**🔑 记忆技巧**： + +- **口诀**："层序遍历找最大，每层记录一个值" + +```java +public List largestValues(TreeNode root) { + List result = new ArrayList<>(); + if (root == null) return result; + + Queue queue = new LinkedList<>(); + queue.offer(root); + + while (!queue.isEmpty()) { + int size = queue.size(); + int maxVal = Integer.MIN_VALUE; + + for (int i = 0; i < size; i++) { + TreeNode node = queue.poll(); + maxVal = Math.max(maxVal, node.val); + + if (node.left != null) queue.offer(node.left); + if (node.right != null) queue.offer(node.right); + } + + result.add(maxVal); + } + + return result; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 访问每个节点一次 +- **空间复杂度**：O(w) - 队列存储的最大宽度 + +### [662. 二叉树最大宽度](https://leetcode.cn/problems/maximum-width-of-binary-tree/) + +**🎯 考察频率：中等 | 难度：中等 | 重要性：位置编码技巧** + +> 计算二叉树的最大宽度（两个端点间的距离） + +**💡 核心思路**：层序遍历 + 节点位置编码 + +- 给每个节点编号：根节点为1，左子节点为2*i，右子节点为2*i+1 +- 每层的宽度 = 最右位置 - 最左位置 + 1 + +**🔑 记忆技巧**： + +- **口诀**："位置编码层序遍历，左右相减计算宽度" + +```java +public int widthOfBinaryTree(TreeNode root) { + if (root == null) return 0; + + Queue nodeQueue = new LinkedList<>(); + Queue posQueue = new LinkedList<>(); + + nodeQueue.offer(root); + posQueue.offer(1); + + int maxWidth = 1; + + while (!nodeQueue.isEmpty()) { + int size = nodeQueue.size(); + int start = posQueue.peek(); + int end = start; + + for (int i = 0; i < size; i++) { + TreeNode node = nodeQueue.poll(); + int pos = posQueue.poll(); + end = pos; + + if (node.left != null) { + nodeQueue.offer(node.left); + posQueue.offer(2 * pos); + } + + if (node.right != null) { + nodeQueue.offer(node.right); + posQueue.offer(2 * pos + 1); + } + } + + maxWidth = Math.max(maxWidth, end - start + 1); + } + + return maxWidth; +} +``` + +**🔧 DFS解法**： + +```java +private int maxWidth = 0; + +public int widthOfBinaryTree(TreeNode root) { + maxWidth = 0; + List startPositions = new ArrayList<>(); + dfs(root, 0, 1, startPositions); + return maxWidth; +} + +private void dfs(TreeNode root, int depth, int pos, List startPositions) { + if (root == null) return; + + // 记录每层第一个节点的位置 + if (depth == startPositions.size()) { + startPositions.add(pos); + } + + // 计算当前层的宽度 + maxWidth = Math.max(maxWidth, pos - startPositions.get(depth) + 1); + + // 递归左右子树 + dfs(root.left, depth + 1, 2 * pos, startPositions); + dfs(root.right, depth + 1, 2 * pos + 1, startPositions); +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 访问每个节点一次 +- **空间复杂度**：O(w) - 队列存储的最大宽度 + +--- + +## 🔍 二、查找路径类 + +### 💡 核心思想 + +- **深度计算**：递归求解左右子树深度 +- **路径追踪**：回溯法记录路径 +- **路径和**：递归过程中累加计算 + +### [104. 二叉树的最大深度](https://leetcode.cn/problems/maximum-depth-of-binary-tree/) + +**🎯 考察频率：极高 | 难度：简单 | 重要性：递归入门** + +> 求二叉树的最大深度：`[3,9,20,null,null,15,7]` → `3` + +**💡 核心思路**：递归分治 + +- 树的深度 = max(左子树深度, 右子树深度) + 1 +- 空树深度为0 + +**🔑 记忆技巧**： + +- **口诀**："左右取最大，根节点加一" +- **形象记忆**：像测量房子高度，选择最高的那一侧 + +```java +public int maxDepth(TreeNode root) { + if (root == null) return 0; + + int leftDepth = maxDepth(root.left); + int rightDepth = maxDepth(root.right); + + return Math.max(leftDepth, rightDepth) + 1; +} +``` + +**🔧 BFS层序解法**： + +```java +public int maxDepth(TreeNode root) { + if (root == null) return 0; + + Queue queue = new LinkedList<>(); + queue.offer(root); + int depth = 0; + + while (!queue.isEmpty()) { + int size = queue.size(); + depth++; + + for (int i = 0; i < size; i++) { + TreeNode node = queue.poll(); + if (node.left != null) queue.offer(node.left); + if (node.right != null) queue.offer(node.right); + } + } + + return depth; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 访问每个节点 +- **空间复杂度**：O(h) - 递归栈深度 + +### [111. 二叉树的最小深度](https://leetcode.cn/problems/minimum-depth-of-binary-tree/) + +**🎯 考察频率：高 | 难度：简单 | 重要性：递归理解** + +> 求到叶子节点的最短路径长度：`[3,9,20,null,null,15,7]` → `2` + +**💡 核心思路**：递归 + 特殊情况处理 + +- 叶子节点：深度为1 +- 只有一个子树：取存在的子树深度 +- 有左右子树：取较小值 + +**🔑 记忆技巧**： + +- **口诀**："叶子节点深度一，单子树要特判，双子树取最小" + +```java +public int minDepth(TreeNode root) { + if (root == null) return 0; + + // 叶子节点 + if (root.left == null && root.right == null) { + return 1; + } + + // 只有右子树 + if (root.left == null) { + return minDepth(root.right) + 1; + } + + // 只有左子树 + if (root.right == null) { + return minDepth(root.left) + 1; + } + + // 有左右子树 + return Math.min(minDepth(root.left), minDepth(root.right)) + 1; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 最坏情况访问所有节点 +- **空间复杂度**：O(h) - 递归栈深度 + +### [112. 路径总和](https://leetcode.cn/problems/path-sum/) + +**🎯 考察频率：极高 | 难度：简单 | 重要性：路径递归** + +> 判断是否存在根到叶子路径和等于目标值：`sum = 22` + +**💡 核心思路**：递归 + 目标值递减 + +- 每次递归减去当前节点值 +- 到达叶子节点时检查目标值是否为节点值 + +**🔑 记忆技巧**： + +- **口诀**："目标递减到叶子，相等即为找到路径" + +```java +public boolean hasPathSum(TreeNode root, int targetSum) { + if (root == null) return false; + + // 叶子节点，检查路径和 + if (root.left == null && root.right == null) { + return targetSum == root.val; + } + + // 递归检查左右子树 + return hasPathSum(root.left, targetSum - root.val) || + hasPathSum(root.right, targetSum - root.val); +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 最坏情况访问所有节点 +- **空间复杂度**：O(h) - 递归栈深度 + +### [113. 路径总和 II](https://leetcode.cn/problems/path-sum-ii/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：回溯算法** + +> 找出所有根到叶子路径和等于目标值的路径 + +**💡 核心思路**：DFS + 回溯 + +- 使用列表记录当前路径 +- 到达叶子节点时检查并保存路径 +- 回溯时移除当前节点 + +**🔑 记忆技巧**： + +- **口诀**："DFS记录路径，回溯恢复状态" + +```java +public List> pathSum(TreeNode root, int targetSum) { + List> result = new ArrayList<>(); + List path = new ArrayList<>(); + dfs(root, targetSum, path, result); + return result; +} + +private void dfs(TreeNode root, int targetSum, List path, + List> result) { + if (root == null) return; + + // 添加当前节点到路径 + path.add(root.val); + + // 叶子节点，检查路径和 + if (root.left == null && root.right == null && targetSum == root.val) { + result.add(new ArrayList<>(path)); + } else { + // 递归左右子树 + dfs(root.left, targetSum - root.val, path, result); + dfs(root.right, targetSum - root.val, path, result); + } + + // 回溯：移除当前节点 + path.remove(path.size() - 1); +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n²) - 最坏情况每个节点都要复制路径 +- **空间复杂度**：O(h) - 递归栈和路径存储 + +### [199. 二叉树的右视图](https://leetcode.cn/problems/binary-tree-right-side-view/) + +**🎯 考察频率：极高 | 难度：中等 | 重要性：层序遍历应用** + +> 给定一个二叉树的 **根节点** `root`，想象自己站在它的右侧，按照从顶部到底部的顺序，返回从右侧所能看到的节点值。 +> +> ![img](https://assets.leetcode.com/uploads/2021/02/14/tree.jpg) +> +> `[1,2,3,null,5,null,4]` → `[1,3,4]` + +**💡 核心思路**： + +- 使用广度优先搜索（BFS）逐层遍历二叉树。 +- 对于每一层，记录最后一个节点（即最右侧的节点）的值。 +- 将每一层的最后一个节点的值加入结果列表。 + +**🔑 记忆技巧**： + +- **口诀**："层序遍历看右侧，每层最后即所见" +- **形象记忆**：像站在树的右边看，每层只能看到最右边的节点 + +```java +public List rightSideView(TreeNode root) { + List result = new ArrayList<>(); + if (root == null) return result; + + Queue queue = new LinkedList<>(); + queue.offer(root); + + while (!queue.isEmpty()) { + int size = queue.size(); + + for (int i = 0; i < size; i++) { + TreeNode node = queue.poll(); + + // 每层的最后一个节点加入结果 + if (i == size - 1) { + result.add(node.val); + } + + if (node.left != null) queue.offer(node.left); + if (node.right != null) queue.offer(node.right); + } + } + + return result; +} +``` + +**🔧 DFS解法**： + +```java +public List rightSideView(TreeNode root) { + List result = new ArrayList<>(); + dfs(root, 0, result); + return result; +} + +private void dfs(TreeNode root, int depth, List result) { + if (root == null) return; + + // 第一次到达这个深度，记录右视图 + if (depth == result.size()) { + result.add(root.val); + } + + // 先遍历右子树，再遍历左子树 + dfs(root.right, depth + 1, result); + dfs(root.left, depth + 1, result); +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 访问每个节点一次 +- **空间复杂度**：O(h) - 递归栈深度或队列空间 + +### [437. 路径总和 III](https://leetcode.cn/problems/path-sum-iii/) + +**🎯 考察频率：极高 | 难度：中等 | 重要性：前缀和在树中的应用** + +> 找出路径和等于目标值的路径数量（路径不需要从根节点开始，也不需要在叶子节点结束） + +**💡 核心思路**：前缀和 + DFS + +- 使用前缀和思想，记录从根到当前节点的路径和 +- 利用哈希表记录前缀和出现的次数 +- 如果当前前缀和减去目标值在哈希表中存在，说明找到了一条路径 + +**🔑 记忆技巧**： + +- **口诀**："前缀和思想用于树，哈希记录找路径" +- **形象记忆**：像数组前缀和一样，在树上也可以用同样的思想 + +```java +public int pathSum(TreeNode root, int targetSum) { + Map prefixSumCount = new HashMap<>(); + prefixSumCount.put(0L, 1); // 前缀和为0的路径有1条（空路径） + return dfs(root, 0L, targetSum, prefixSumCount); +} + +private int dfs(TreeNode root, long currentSum, int targetSum, + Map prefixSumCount) { + if (root == null) return 0; + + // 更新当前路径和 + currentSum += root.val; + + // 查看是否存在前缀和为 currentSum - targetSum 的路径 + int count = prefixSumCount.getOrDefault(currentSum - targetSum, 0); + + // 记录当前前缀和 + prefixSumCount.put(currentSum, + prefixSumCount.getOrDefault(currentSum, 0) + 1); + + // 递归左右子树 + count += dfs(root.left, currentSum, targetSum, prefixSumCount); + count += dfs(root.right, currentSum, targetSum, prefixSumCount); + + // 回溯：移除当前节点的贡献 + prefixSumCount.put(currentSum, prefixSumCount.get(currentSum) - 1); + + return count; +} +``` + +**🔧 暴力解法（理解用）**： + +```java +public int pathSum(TreeNode root, int targetSum) { + if (root == null) return 0; + + // 以当前节点为起点的路径数 + 左子树的路径数 + 右子树的路径数 + return pathSumFrom(root, targetSum) + + pathSum(root.left, targetSum) + + pathSum(root.right, targetSum); +} + +private int pathSumFrom(TreeNode root, long targetSum) { + if (root == null) return 0; + + int count = 0; + if (root.val == targetSum) count++; + + count += pathSumFrom(root.left, targetSum - root.val); + count += pathSumFrom(root.right, targetSum - root.val); + + return count; +} +``` + +**⏱️ 复杂度分析**： + +- **前缀和解法**： + - 时间复杂度：O(n) - 每个节点访问一次 + - 空间复杂度：O(h) - 递归栈和哈希表 +- **暴力解法**： + - 时间复杂度：O(n²) - 每个节点都要向下搜索 + - 空间复杂度：O(h) - 递归栈深度 + +### [236. 二叉树的最近公共祖先](https://leetcode.cn/problems/lowest-common-ancestor-of-a-binary-tree/) + +**🎯 考察频率：极高 | 难度：中等 | 重要性：树的经典问题** + +> 找到二叉树中两个节点的最近公共祖先 + +**💡 核心思路**：后序遍历 + 递归回溯 + +- 如果当前节点是p或q之一，返回当前节点 +- 递归查找左右子树 +- 如果左右子树都找到了节点，当前节点就是LCA +- 如果只有一边找到，返回找到的那一边 + +> - 两个节点的最近公共祖先其实就是这两个节点向根节点的「延长线」的交汇点 +> +> - **如果一个节点能够在它的左右子树中分别找到**`p`**和**`q`**，则该节点为**`LCA`**节点** +> +> - 若 root 是 p,q 的 最近公共祖先 ，则只可能为以下情况之一： +> +> - p 和 q 在 root 的子树中，且分列 root 的 异侧（即分别在左、右子树中）； +> - p=root ，且 q 在 root 的左或右子树中； +> - q=root ，且 p 在 root 的左或右子树中；![Picture2.png](https://pic.leetcode-cn.com/1599885247-mgYjRv-Picture2.png) + +**🔑 记忆技巧**： + +- **口诀**："后序遍历找祖先，左右都有即是它" +- **形象记忆**：像家族树找共同祖先，必须包含两个人的最小子树的根 + +```java +public TreeNode lowestCommonAncestor(TreeNode root, TreeNode p, TreeNode q) { + // 递归终止条件 + if (root == null || root == p || root == q) { + return root; + } + + // 递归查找左右子树 + TreeNode left = lowestCommonAncestor(root.left, p, q); + TreeNode right = lowestCommonAncestor(root.right, p, q); + + // 如果左右子树都找到了节点，当前节点就是LCA + if (left != null && right != null) { + return root; + } + + // 如果只有一边找到，返回找到的那一边 + return left != null ? left : right; +} +``` + +**🔧 存储父节点解法**： + +```java +public TreeNode lowestCommonAncestor(TreeNode root, TreeNode p, TreeNode q) { + Map parent = new HashMap<>(); + Stack stack = new Stack<>(); + + parent.put(root, null); + stack.push(root); + + // 构建父节点映射 + while (!parent.containsKey(p) || !parent.containsKey(q)) { + TreeNode node = stack.pop(); + + if (node.left != null) { + parent.put(node.left, node); + stack.push(node.left); + } + + if (node.right != null) { + parent.put(node.right, node); + stack.push(node.right); + } + } + + // 收集p的所有祖先 + Set ancestors = new HashSet<>(); + while (p != null) { + ancestors.add(p); + p = parent.get(p); + } + + // 找q的祖先中第一个在p的祖先集合中的 + while (!ancestors.contains(q)) { + q = parent.get(q); + } + + return q; +} +``` + +**⏱️ 复杂度分析**： + +- **递归解法**： + - 时间复杂度：O(n) - 最坏情况访问所有节点 + - 空间复杂度：O(h) - 递归栈深度 +- **存储父节点解法**： + - 时间复杂度：O(n) - 遍历所有节点 + - 空间复杂度：O(n) - 存储父节点映射 + +### [124. 二叉树中的最大路径和](https://leetcode.cn/problems/binary-tree-maximum-path-sum/) + +**🎯 考察频率：极高 | 难度：困难 | 重要性：树形DP经典** + +> 找任意节点到任意节点的最大路径和（可以不经过根节点） + +**💡 核心思路**：后序遍历 + 状态维护 + +- 对每个节点，考虑经过该节点的最大路径和 +- 返回值是从该节点向下的最大路径和 +- 全局变量记录所有可能路径的最大值 + +**🔑 记忆技巧**： + +- **口诀**："后序求贡献，全局记录最大值" + +```java +private int maxSum = Integer.MIN_VALUE; + +public int maxPathSum(TreeNode root) { + maxSum = Integer.MIN_VALUE; + maxGain(root); + return maxSum; +} + +private int maxGain(TreeNode node) { + if (node == null) return 0; + + // 递归计算左右子树的最大贡献值 + // 只有在贡献值大于0时才选择该子树 + int leftGain = Math.max(maxGain(node.left), 0); + int rightGain = Math.max(maxGain(node.right), 0); + + // 经过当前节点的最大路径和 + int pathSum = node.val + leftGain + rightGain; + + // 更新全局最大值 + maxSum = Math.max(maxSum, pathSum); + + // 返回从当前节点向下的最大路径和 + return node.val + Math.max(leftGain, rightGain); +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 每个节点访问一次 +- **空间复杂度**：O(h) - 递归栈深度 + +--- + +## 🏗️ 三、构造变换类 + +### 💡 核心思想 + +- **递归构造**：根据遍历序列的性质递归建树 +- **树的变换**：翻转、对称等操作 +- **结构比较**：递归比较两树的结构和值 + +### [105. 从前序与中序遍历序列构造二叉树](https://leetcode.cn/problems/construct-binary-tree-from-preorder-and-inorder-traversal/) + +**🎯 考察频率：极高 | 难度：中等 | 重要性：构造经典** + +> 根据前序和中序遍历结果构造唯一二叉树 + +**💡 核心思路**：分治递归 + +- 前序第一个元素是根节点 +- 在中序中找根节点位置，划分左右子树 +- 递归构造左右子树 + +**🔑 记忆技巧**： + +- **口诀**："前序定根节点，中序划左右" + +```java +public TreeNode buildTree(int[] preorder, int[] inorder) { + // 构建中序遍历的值到索引的映射 + Map inorderMap = new HashMap<>(); + for (int i = 0; i < inorder.length; i++) { + inorderMap.put(inorder[i], i); + } + + return build(preorder, 0, preorder.length - 1, + inorder, 0, inorder.length - 1, inorderMap); +} + +private TreeNode build(int[] preorder, int preStart, int preEnd, + int[] inorder, int inStart, int inEnd, + Map inorderMap) { + if (preStart > preEnd) return null; + + // 前序遍历第一个元素是根节点 + int rootVal = preorder[preStart]; + TreeNode root = new TreeNode(rootVal); + + // 在中序遍历中找到根节点的位置 + int rootIndex = inorderMap.get(rootVal); + + // 左子树的节点数量 + int leftSize = rootIndex - inStart; + + // 递归构造左右子树 + root.left = build(preorder, preStart + 1, preStart + leftSize, + inorder, inStart, rootIndex - 1, inorderMap); + root.right = build(preorder, preStart + leftSize + 1, preEnd, + inorder, rootIndex + 1, inEnd, inorderMap); + + return root; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 每个节点创建一次 +- **空间复杂度**：O(n) - 哈希表和递归栈 + +### [226. 翻转二叉树](https://leetcode.cn/problems/invert-binary-tree/) + +**🎯 考察频率：极高 | 难度：简单 | 重要性：基础操作** + +> 翻转二叉树：交换每个节点的左右子树 + +**💡 核心思路**：递归交换 + +- 递归翻转左右子树 +- 交换当前节点的左右子树 + +**🔑 记忆技巧**： + +- **口诀**："递归翻转左右子树，交换当前左右指针" + +```java +public TreeNode invertTree(TreeNode root) { + if (root == null) return null; + + // 递归翻转左右子树 + TreeNode left = invertTree(root.left); + TreeNode right = invertTree(root.right); + + // 交换左右子树 + root.left = right; + root.right = left; + + return root; +} +``` + +**🔧 迭代解法**： + +```java +public TreeNode invertTree(TreeNode root) { + if (root == null) return null; + + Queue queue = new LinkedList<>(); + queue.offer(root); + + while (!queue.isEmpty()) { + TreeNode node = queue.poll(); + + // 交换左右子树 + TreeNode temp = node.left; + node.left = node.right; + node.right = temp; + + if (node.left != null) queue.offer(node.left); + if (node.right != null) queue.offer(node.right); + } + + return root; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 访问每个节点 +- **空间复杂度**：O(h) - 递归栈深度 + +### [101. 对称二叉树](https://leetcode.cn/problems/symmetric-tree/) + +**🎯 考察频率：极高 | 难度：简单 | 重要性：结构判断** + +> 判断二叉树是否关于根节点对称 + +**💡 核心思路**：递归比较 + +- 比较左子树的左孩子和右子树的右孩子 +- 比较左子树的右孩子和右子树的左孩子 + +**🔑 记忆技巧**： + +- **口诀**："左左对右右，左右对右左" + +```java +public boolean isSymmetric(TreeNode root) { + if (root == null) return true; + return isSymmetricHelper(root.left, root.right); +} + +private boolean isSymmetricHelper(TreeNode left, TreeNode right) { + // 都为空，对称 + if (left == null && right == null) return true; + + // 一个为空一个不为空，不对称 + if (left == null || right == null) return false; + + // 值不同，不对称 + if (left.val != right.val) return false; + + // 递归检查 + return isSymmetricHelper(left.left, right.right) && + isSymmetricHelper(left.right, right.left); +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 最坏情况访问所有节点 +- **空间复杂度**：O(h) - 递归栈深度 + +### [100. 相同的树](https://leetcode.cn/problems/same-tree/) + +**🎯 考察频率：高 | 难度：简单 | 重要性：基础比较** + +> 判断两棵二叉树是否相同 + +**💡 核心思路**：递归比较 + +- 比较当前节点值 +- 递归比较左右子树 + +**🔑 记忆技巧**： + +- **口诀**："值相同且结构同，递归比较左右树" + +```java +public boolean isSameTree(TreeNode p, TreeNode q) { + // 都为空 + if (p == null && q == null) return true; + + // 一个为空一个不为空 + if (p == null || q == null) return false; + + // 值不同 + if (p.val != q.val) return false; + + // 递归比较左右子树 + return isSameTree(p.left, q.left) && isSameTree(p.right, q.right); +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(min(m,n)) - m,n分别为两树的节点数 +- **空间复杂度**：O(min(m,n)) - 递归栈深度 + +--- + +## ⚖️ 四、平衡验证类 + +### 💡 核心思想 + +- **平衡判断**：检查每个节点的左右子树高度差 +- **BST验证**：利用中序遍历或范围检查 +- **有序性利用**：BST的中序遍历是有序的 + +### [110. 平衡二叉树](https://leetcode.cn/problems/balanced-binary-tree/) + +**🎯 考察频率：极高 | 难度：简单 | 重要性：平衡树理解** + +> 判断二叉树是否为高度平衡的（每个节点的左右子树高度差不超过1） + +**💡 核心思路**：后序遍历 + 高度计算 + +- 先递归计算左右子树高度 +- 检查高度差是否超过1 +- 返回当前树的高度 + +**🔑 记忆技巧**： + +- **口诀**："后序算高度，高度差不超一" + +```java +public boolean isBalanced(TreeNode root) { + return height(root) >= 0; +} + +private int height(TreeNode root) { + if (root == null) return 0; + + int leftHeight = height(root.left); + int rightHeight = height(root.right); + + // 左右子树有一个不平衡，或当前节点不平衡 + if (leftHeight == -1 || rightHeight == -1 || + Math.abs(leftHeight - rightHeight) > 1) { + return -1; + } + + return Math.max(leftHeight, rightHeight) + 1; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 每个节点访问一次 +- **空间复杂度**：O(h) - 递归栈深度 + +### [98. 验证二叉搜索树](https://leetcode.cn/problems/validate-binary-search-tree/) + +**🎯 考察频率：极高 | 难度：中等 | 重要性：BST性质理解** + +> 判断二叉树是否为有效的二叉搜索树 + +**💡 核心思路**：范围检查或中序遍历 + +- 方法1：递归时维护节点值的有效范围 +- 方法2：中序遍历结果应该严格递增 + +**🔑 记忆技巧**： + +- **口诀**："范围递归或中序，严格递增是关键" + +```java +// 方法1：范围检查 +public boolean isValidBST(TreeNode root) { + return validate(root, null, null); +} + +private boolean validate(TreeNode node, Integer lower, Integer upper) { + if (node == null) return true; + + int val = node.val; + + // 检查当前节点是否在有效范围内 + if (lower != null && val <= lower) return false; + if (upper != null && val >= upper) return false; + + // 递归检查左右子树 + return validate(node.left, lower, val) && + validate(node.right, val, upper); +} +``` + +**🔧 中序遍历法**： + +```java +private Integer prev = null; + +public boolean isValidBST(TreeNode root) { + prev = null; + return inorder(root); +} + +private boolean inorder(TreeNode root) { + if (root == null) return true; + + // 检查左子树 + if (!inorder(root.left)) return false; + + // 检查当前节点 + if (prev != null && root.val <= prev) return false; + prev = root.val; + + // 检查右子树 + return inorder(root.right); +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 访问每个节点 +- **空间复杂度**：O(h) - 递归栈深度 + +### [230. 二叉搜索树中第K小的元素](https://leetcode.cn/problems/kth-smallest-element-in-a-bst/) + +**🎯 考察频率：极高 | 难度：中等 | 重要性：BST应用** + +> 找到二叉搜索树中第k小的元素 + +**💡 核心思路**：中序遍历 + +- BST的中序遍历结果是有序的 +- 遍历到第k个元素时返回 + +**🔑 记忆技巧**： + +- **口诀**："BST中序有序，第k个即答案" + +```java +private int count = 0; +private int result = 0; + +public int kthSmallest(TreeNode root, int k) { + count = 0; + inorder(root, k); + return result; +} + +private void inorder(TreeNode root, int k) { + if (root == null) return; + + inorder(root.left, k); + + count++; + if (count == k) { + result = root.val; + return; + } + + inorder(root.right, k); +} +``` + +**🔧 迭代解法**： + +```java +public int kthSmallest(TreeNode root, int k) { + Stack stack = new Stack<>(); + TreeNode current = root; + + while (current != null || !stack.isEmpty()) { + while (current != null) { + stack.push(current); + current = current.left; + } + + current = stack.pop(); + k--; + if (k == 0) return current.val; + + current = current.right; + } + + return -1; // 不会到达这里 +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(h + k) - h为树高，k为目标位置 +- **空间复杂度**：O(h) - 递归栈或显式栈 + +--- + +## 🎯 五、搜索树操作类 + +### 💡 核心思想 + +- **BST性质**：左小右大的有序性 +- **查找效率**：利用有序性快速定位 +- **递归操作**：插入、删除、查找的递归实现 + +### [235. 二叉搜索树的最近公共祖先](https://leetcode.cn/problems/lowest-common-ancestor-of-a-binary-search-tree/) + +**🎯 考察频率：极高 | 难度：简单 | 重要性：BST应用** + +> 找到BST中两个节点的最近公共祖先 + +**💡 核心思路**：利用BST性质 + +- 如果两个节点都在左子树，向左找 +- 如果两个节点都在右子树，向右找 +- 否则当前节点就是LCA + +**🔑 记忆技巧**： + +- **口诀**："同侧继续找，异侧即祖先" + +```java +public TreeNode lowestCommonAncestor(TreeNode root, TreeNode p, TreeNode q) { + if (root == null) return null; + + // 两个节点都在左子树 + if (p.val < root.val && q.val < root.val) { + return lowestCommonAncestor(root.left, p, q); + } + + // 两个节点都在右子树 + if (p.val > root.val && q.val > root.val) { + return lowestCommonAncestor(root.right, p, q); + } + + // 一个在左，一个在右，或者其中一个就是根节点 + return root; +} +``` + +**🔧 迭代解法**： + +```java +public TreeNode lowestCommonAncestor(TreeNode root, TreeNode p, TreeNode q) { + TreeNode current = root; + + while (current != null) { + if (p.val < current.val && q.val < current.val) { + current = current.left; + } else if (p.val > current.val && q.val > current.val) { + current = current.right; + } else { + return current; + } + } + + return null; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(h) - h为树的高度 +- **空间复杂度**：O(1) - 迭代版本为常数空间 + +### [450. 删除二叉搜索树中的节点](https://leetcode.cn/problems/delete-node-in-a-bst/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：BST操作** + +> 删除BST中的指定节点，保持BST性质 + +**💡 核心思路**：分情况讨论 + +- 叶子节点：直接删除 +- 只有一个子树：用子树替代 +- 有两个子树：用右子树最小值（或左子树最大值）替代 + +**🔑 记忆技巧**： + +- **口诀**："叶子直删，单子替代，双子找后继" + +```java +public TreeNode deleteNode(TreeNode root, int key) { + if (root == null) return null; + + if (key < root.val) { + // 在左子树中删除 + root.left = deleteNode(root.left, key); + } else if (key > root.val) { + // 在右子树中删除 + root.right = deleteNode(root.right, key); + } else { + // 找到要删除的节点 + if (root.left == null) { + return root.right; + } else if (root.right == null) { + return root.left; + } else { + // 有两个子树，找右子树的最小值 + TreeNode minNode = findMin(root.right); + root.val = minNode.val; + root.right = deleteNode(root.right, minNode.val); + } + } + + return root; +} + +private TreeNode findMin(TreeNode root) { + while (root.left != null) { + root = root.left; + } + return root; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(h) - h为树的高度 +- **空间复杂度**：O(h) - 递归栈深度 + +### [108. 将有序数组转换为二叉搜索树](https://leetcode.cn/problems/convert-sorted-array-to-binary-search-tree/) + +**🎯 考察频率：高 | 难度：简单 | 重要性：BST构造** + +> 将有序数组转换为高度平衡的BST + +**💡 核心思路**：分治递归 + +- 选择数组中间元素作为根节点 +- 递归构造左右子树 + +**🔑 记忆技巧**： + +- **口诀**："中间作根，左右递归" + +```java +public TreeNode sortedArrayToBST(int[] nums) { + return helper(nums, 0, nums.length - 1); +} + +private TreeNode helper(int[] nums, int left, int right) { + if (left > right) return null; + + // 选择中间位置作为根节点 + int mid = left + (right - left) / 2; + TreeNode root = new TreeNode(nums[mid]); + + // 递归构造左右子树 + root.left = helper(nums, left, mid - 1); + root.right = helper(nums, mid + 1, right); + + return root; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 每个元素访问一次 +- **空间复杂度**：O(logn) - 递归栈深度 + +--- + +## 📊 六、树形DP类 + +### 💡 核心思想 + +- **状态定义**：每个节点维护所需的状态信息 +- **状态转移**：根据子树状态计算当前状态 +- **最优子结构**：问题的最优解包含子问题的最优解 + +### [337. 打家劫舍 III](https://leetcode.cn/problems/house-robber-iii/) + +**🎯 考察频率：极高 | 难度：中等 | 重要性：树形DP经典** + +> 在二叉树上进行打家劫舍，相邻节点不能同时被选择 + +**💡 核心思路**：树形DP + +- 对每个节点维护两个状态：偷和不偷 +- 偷当前节点：不能偷子节点 +- 不偷当前节点：可以选择偷或不偷子节点 + +**🔑 记忆技巧**： + +- **口诀**："偷根不偷子，不偷根选子树最优" + +```java +public int rob(TreeNode root) { + int[] result = robHelper(root); + return Math.max(result[0], result[1]); +} + +// 返回数组：[不偷当前节点的最大值, 偷当前节点的最大值] +private int[] robHelper(TreeNode root) { + if (root == null) return new int[]{0, 0}; + + int[] left = robHelper(root.left); + int[] right = robHelper(root.right); + + // 不偷当前节点：可以选择偷或不偷子节点 + int notRob = Math.max(left[0], left[1]) + Math.max(right[0], right[1]); + + // 偷当前节点：不能偷子节点 + int rob = root.val + left[0] + right[0]; + + return new int[]{notRob, rob}; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 每个节点访问一次 +- **空间复杂度**：O(h) - 递归栈深度 + +### [543. 二叉树的直径](https://leetcode.cn/problems/diameter-of-binary-tree/) + +**🎯 考察频率：极高 | 难度：简单 | 重要性：路径长度计算** + +> 计算二叉树中任意两个节点路径长度的最大值 + +**💡 核心思路**：DFS + 路径计算 + +- 对每个节点，计算经过该节点的最长路径 +- 路径长度 = 左子树深度 + 右子树深度 +- 全局变量记录最大路径长度 + +**🔑 记忆技巧**： + +- **口诀**："左深加右深，全局记最大" + +```java +private int diameter = 0; + +public int diameterOfBinaryTree(TreeNode root) { + diameter = 0; + depth(root); + return diameter; +} + +private int depth(TreeNode root) { + if (root == null) return 0; + + int leftDepth = depth(root.left); + int rightDepth = depth(root.right); + + // 更新直径：经过当前节点的路径长度 + diameter = Math.max(diameter, leftDepth + rightDepth); + + // 返回当前节点的深度 + return Math.max(leftDepth, rightDepth) + 1; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 每个节点访问一次 +- **空间复杂度**：O(h) - 递归栈深度 + +### [572. 另一棵树的子树](https://leetcode.cn/problems/subtree-of-another-tree/) + +**🎯 考察频率：高 | 难度：简单 | 重要性：子树匹配** + +> 判断一棵树是否为另一棵树的子树 + +**💡 核心思路**：递归 + 树匹配 + +- 遍历主树的每个节点 +- 对每个节点检查是否与子树相同 + +**🔑 记忆技巧**： + +- **口诀**："遍历主树每个点，相同判断为子树" + +```java +public boolean isSubtree(TreeNode root, TreeNode subRoot) { + if (root == null) return false; + + // 检查当前节点为根的树是否与subRoot相同 + if (isSame(root, subRoot)) return true; + + // 递归检查左右子树 + return isSubtree(root.left, subRoot) || isSubtree(root.right, subRoot); +} + +private boolean isSame(TreeNode s, TreeNode t) { + if (s == null && t == null) return true; + if (s == null || t == null) return false; + if (s.val != t.val) return false; + + return isSame(s.left, t.left) && isSame(s.right, t.right); +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(m×n) - m,n分别为两树的节点数 +- **空间复杂度**：O(max(h1,h2)) - 递归栈深度 + +--- + +## 🔄 七、序列化类 + +### 💡 核心思想 + +- **序列化**：将树结构转换为字符串 +- **反序列化**：从字符串重建树结构 +- **编码策略**：前序、层序等不同的编码方式 + +### [297. 二叉树的序列化与反序列化](https://leetcode.cn/problems/serialize-and-deserialize-binary-tree/) + +**🎯 考察频率：极高 | 难度：困难 | 重要性：综合能力考察** + +> 设计算法来序列化和反序列化二叉树 + +**💡 核心思路**：前序遍历 + 递归构造 + +- 序列化：前序遍历，null用特殊符号表示 +- 反序列化：根据前序遍历的性质递归构造 + +**🔑 记忆技巧**： + +- **口诀**："前序序列化，递归反序列" + +```java +public class Codec { + private static final String NULL = "#"; + private static final String SEP = ","; + + // 序列化 + public String serialize(TreeNode root) { + StringBuilder sb = new StringBuilder(); + serialize(root, sb); + return sb.toString(); + } + + private void serialize(TreeNode root, StringBuilder sb) { + if (root == null) { + sb.append(NULL).append(SEP); + return; + } + + sb.append(root.val).append(SEP); + serialize(root.left, sb); + serialize(root.right, sb); + } + + // 反序列化 + public TreeNode deserialize(String data) { + Queue queue = new LinkedList<>(Arrays.asList(data.split(SEP))); + return deserialize(queue); + } + + private TreeNode deserialize(Queue queue) { + String val = queue.poll(); + if (NULL.equals(val)) return null; + + TreeNode root = new TreeNode(Integer.parseInt(val)); + root.left = deserialize(queue); + root.right = deserialize(queue); + + return root; + } +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 序列化和反序列化都是O(n) +- **空间复杂度**：O(n) - 存储序列化结果 + +--- + +## 🚀 八、进阶技巧类 + +### 💡 核心思想 + +- **原地操作**：在原树结构上进行变换 +- **指针操作**：巧妙使用指针连接节点 +- **空间优化**：常数空间完成复杂操作 + +### [114. 二叉树展开为链表](https://leetcode.cn/problems/flatten-binary-tree-to-linked-list/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：树变链表** + +> 将二叉树展开为单链表（按前序遍历顺序） + +**💡 核心思路**：后序遍历 + 原地操作 + +- 先处理右子树，再处理左子树 +- 将左子树插入到根节点和右子树之间 + +**🔑 记忆技巧**： + +- **口诀**："后序处理，左插右连" + +```java +public void flatten(TreeNode root) { + if (root == null) return; + + // 先递归展开左右子树 + flatten(root.left); + flatten(root.right); + + // 保存右子树 + TreeNode right = root.right; + + // 将左子树移到右边 + root.right = root.left; + root.left = null; + + // 找到当前右子树的末尾，连接原来的右子树 + TreeNode current = root; + while (current.right != null) { + current = current.right; + } + current.right = right; +} +``` + +**🔧 前序遍历解法**： + +```java +private TreeNode prev = null; + +public void flatten(TreeNode root) { + if (root == null) return; + + flatten(root.right); + flatten(root.left); + + root.right = prev; + root.left = null; + prev = root; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 每个节点访问一次 +- **空间复杂度**：O(h) - 递归栈深度 + +### [116. 填充每个节点的下一个右侧节点指针](https://leetcode.cn/problems/populating-next-right-pointers-in-each-node/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：层序连接** + +> 填充每个节点的next指针，指向其下一个右侧节点 + +**💡 核心思路**：层序遍历 + 指针连接 + +- 利用已建立的next指针遍历当前层 +- 连接下一层的节点 + +**🔑 记忆技巧**： + +- **口诀**："当前层遍历，下层建连接" + +```java +public Node connect(Node root) { + if (root == null) return null; + + Node leftmost = root; + + while (leftmost.left != null) { + Node head = leftmost; + + while (head != null) { + // 连接左右子节点 + head.left.next = head.right; + + // 连接右子节点和下一个节点的左子节点 + if (head.next != null) { + head.right.next = head.next.left; + } + + head = head.next; + } + + leftmost = leftmost.left; + } + + return root; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 每个节点访问一次 +- **空间复杂度**：O(1) - 常数空间 + +### [958. 二叉树的完全性检验](https://leetcode.cn/problems/check-completeness-of-a-binary-tree/) + +**🎯 考察频率：中等 | 难度：中等 | 重要性：完全二叉树判断** + +> 判断给定的二叉树是否是完全二叉树 + +**💡 核心思路**：层序遍历 + 空节点检查 + +- 完全二叉树的层序遍历中，一旦出现空节点，后面就不能再有非空节点 +- 使用BFS，遇到第一个空节点后，检查后续是否都为空 + +**🔑 记忆技巧**： + +- **口诀**："层序遍历遇空停，后续全空才完全" + +```java +public boolean isCompleteTree(TreeNode root) { + if (root == null) return true; + + Queue queue = new LinkedList<>(); + queue.offer(root); + boolean foundNull = false; + + while (!queue.isEmpty()) { + TreeNode node = queue.poll(); + + if (node == null) { + foundNull = true; + } else { + // 如果之前遇到过空节点，现在又遇到非空节点，不是完全二叉树 + if (foundNull) return false; + + queue.offer(node.left); + queue.offer(node.right); + } + } + + return true; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 最坏情况访问所有节点 +- **空间复杂度**：O(w) - 队列存储的最大宽度 + +### [117. 填充每个节点的下一个右侧节点指针 II](https://leetcode.cn/problems/populating-next-right-pointers-in-each-node-ii/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：任意二叉树的指针连接** + +> 填充每个节点的next指针，指向其下一个右侧节点（不是完全二叉树） + +**💡 核心思路**：层序遍历或利用已建立的next指针 + +- 方法1：层序遍历，逐层连接 +- 方法2：利用上层已建立的next指针遍历，连接下层 + +**🔑 记忆技巧**： + +- **口诀**："层序连接或上层带下层" + +```java +// 方法1：层序遍历 +public Node connect(Node root) { + if (root == null) return null; + + Queue queue = new LinkedList<>(); + queue.offer(root); + + while (!queue.isEmpty()) { + int size = queue.size(); + Node prev = null; + + for (int i = 0; i < size; i++) { + Node node = queue.poll(); + + if (prev != null) { + prev.next = node; + } + prev = node; + + if (node.left != null) queue.offer(node.left); + if (node.right != null) queue.offer(node.right); + } + } + + return root; +} +``` + +**🔧 O(1)空间解法**： + +```java +public Node connect(Node root) { + Node head = root; + + while (head != null) { + Node dummy = new Node(0); + Node tail = dummy; + + // 遍历当前层，连接下一层 + while (head != null) { + if (head.left != null) { + tail.next = head.left; + tail = tail.next; + } + if (head.right != null) { + tail.next = head.right; + tail = tail.next; + } + head = head.next; + } + + head = dummy.next; + } + + return root; +} +``` + +**⏱️ 复杂度分析**： + +- **层序遍历**： + - 时间复杂度：O(n) - 每个节点访问一次 + - 空间复杂度：O(w) - 队列空间 +- **O(1)空间解法**： + - 时间复杂度：O(n) - 每个节点访问一次 + - 空间复杂度：O(1) - 常数空间 + +--- + +## 🏆 面试前15分钟速记表 + +| 题型分类 | 核心技巧 | 高频题目 | 记忆口诀 | 难度 | +| -------------- | --------- | ------------------------------------------------------------ | ---------------------------------------------- | ----- | +| **遍历基础** | 递归+迭代 | 前序遍历、中序遍历、后序遍历、层序遍历、锯齿形遍历、完全二叉树的节点个数、在每个树行中找最大值、二叉树最大宽度 | 前序根左右，中序左根右，后序左右根，层序逐层走 | ⭐⭐⭐ | +| **查找路径** | DFS+回溯 | 最大深度、最小深度、路径总和、路径总和II、二叉树的右视图、路径总和III、二叉树的最近公共祖先、最大路径和 | 递归求深度，回溯记路径，路径和用DFS | ⭐⭐⭐⭐ | +| **构造变换** | 分治递归 | 从遍历序列构造树、翻转二叉树、对称二叉树、相同的树 | 前序定根节点，中序划左右，递归翻转左右子树 | ⭐⭐⭐⭐ | +| **平衡验证** | 性质检查 | 平衡二叉树、验证BST、BST中第K小元素 | 后序算高度，范围验证BST，中序遍历有序性 | ⭐⭐⭐⭐ | +| **搜索树操作** | BST性质 | BST的LCA、删除BST节点、有序数组转BST | 左小右大有序性，同侧继续找，异侧即祖先 | ⭐⭐⭐⭐ | +| **树形DP** | 状态转移 | 打家劫舍III、二叉树直径、另一棵树的子树 | 偷根不偷子，不偷根选最优，左深加右深 | ⭐⭐⭐⭐⭐ | +| **序列化** | 编码重建 | 二叉树序列化与反序列化 | 前序序列化，递归反序列 | ⭐⭐⭐⭐⭐ | +| **进阶技巧** | 原地操作 | 展开为链表、填充next指针、二叉树的完全性检验、填充next指针II | 后序处理左插右连，当前层遍历下层建连接 | ⭐⭐⭐⭐⭐ | + +### 按难度分级 + +- **⭐⭐⭐ 简单必会**：前序中序后序遍历、最大最小深度、翻转二叉树、对称二叉树、相同的树、平衡二叉树 +- **⭐⭐⭐⭐ 中等重点**：层序遍历、锯齿形遍历、路径总和II、从遍历序列构造树、验证BST、BST第K小元素、打家劫舍III、二叉树直径 +- **⭐⭐⭐⭐⭐ 困难经典**：最大路径和、序列化与反序列化、展开为链表 + +### 热题100核心优先级 + +1. **二叉树的中序遍历** - 递归和迭代两种写法 +2. **二叉树的层序遍历** - BFS模板题 +3. **二叉树的最大深度** - 递归入门 +4. **翻转二叉树** - 基础操作 +5. **对称二叉树** - 递归比较 +6. **路径总和** - DFS应用 +7. **从前序与中序构造二叉树** - 分治思想 +8. **验证二叉搜索树** - BST性质 +9. **二叉树中的最大路径和** - 树形DP经典 +10. **二叉树的序列化与反序列化** - 综合能力 + +### 热题100题目统计 + +- **总题目数**：38+道热题100二叉树题目 +- **新增重要题目**：LC199二叉树的右视图、LC437路径总和III、LC236二叉树的最近公共祖先、LC222完全二叉树的节点个数、LC515在每个树行中找最大值、LC662二叉树最大宽度、LC958二叉树的完全性检验、LC117填充每个节点的下一个右侧节点指针II +- **覆盖率**：100%覆盖热题100中的二叉树相关题目，并补充了重要的面试高频题 +- **核心算法**：递归、DFS、BFS、分治、动态规划、前缀和、位置编码 + +### 常见陷阱提醒 + +- ⚠️ **空指针**：`root == null` 的边界处理 +- ⚠️ **递归终止**：明确递归的出口条件 +- ⚠️ **返回值**：确保每个递归分支都有返回值 +- ⚠️ **全局变量**：多次调用时要重置全局变量 +- ⚠️ **BST性质**：注意是严格大于/小于，不能相等 + +### 解题步骤提醒 + +1. **确定递归函数的定义**：明确函数的输入输出 +2. **确定递归终止条件**：通常是`root == null` +3. **确定单层递归逻辑**：当前节点应该做什么 +4. **考虑返回值**：是否需要返回值来传递信息 + +--- + +*🎯 备注：本总结涵盖了二叉树算法的核心知识点，建议配合实际编程练习，重点掌握递归思维和分治思想。* + diff --git a/docs/data-structure-algorithms/soultion/DFS-Solution.md b/docs/data-structure-algorithms/soultion/DFS-Solution.md new file mode 100644 index 0000000000..2c423c6802 --- /dev/null +++ b/docs/data-structure-algorithms/soultion/DFS-Solution.md @@ -0,0 +1,1176 @@ +--- +title: DFS-热题 +date: 2025-04-08 +tags: + - DFS + - algorithms +categories: leetcode +--- + +![](https://img.starfish.ink/leetcode/leetcode-banner.png) + +> **导读**：深度优先搜索(DFS)是最重要的图论算法之一，也是面试中的高频考点。从简单的图遍历到复杂的回溯问题，DFS的应用无处不在。掌握DFS的核心思想和常见模板，是解决树、图、回溯等问题的关键。 +> +> **关键词**：深度优先搜索、回溯算法、树遍历、图遍历、递归 + +### 📋 分类索引 + +1. **🔥 基础DFS应用类（4题）**：[岛屿数量](#_200-岛屿数量)、[单词搜索](#_79-单词搜索)、[岛屿的最大面积](#_695-岛屿的最大面积)、[被围绕的区域](#_130-被围绕的区域) +2. **🌳 树的DFS应用类（9题）**：[二叉树的最大深度](#_104-二叉树的最大深度)、[验证二叉搜索树](#_98-验证二叉搜索树)、[二叉树的直径](#_543-二叉树的直径)、[对称二叉树](#_101-对称二叉树)、[二叉树的最小深度](#_111-二叉树的最小深度)、[路径总和III](#_437-路径总和-iii)、[把二叉搜索树转换为累加树](#_538-把二叉搜索树转换为累加树)、[二叉树中的最大路径和](#_124-二叉树中的最大路径和)、[从前序与中序遍历序列构造二叉树](#_105-从前序与中序遍历序列构造二叉树) +3. **🔄 回溯算法类（7题）**：[全排列](#_46-全排列)、[组合总和](#_39-组合总和)、[子集](#_78-子集)、[括号生成](#_22-括号生成)、[电话号码的字母组合](#_17-电话号码的字母组合)、[分割回文串](#_131-分割回文串)、[目标和](#_494-目标和) +4. **🔍 图论算法类（2题）**：[课程表II](#_210-课程表ii) +5. **🌟 高级技巧类（1题）**：[单词拆分](#_139-单词拆分) + +### 🎯 核心考点概览 + +- **基础DFS遍历**：图和矩阵的深度优先搜索，连通性检测 +- **树的DFS应用**：前中后序遍历，路径搜索，树的性质验证 +- **回溯算法**：全排列、组合、子集等搜索所有可能解 +- **图论应用**：环检测、拓扑排序、最长路径搜索 + +我们按分类，逐个攻破~ + +### 📝 必掌握模板 + +```java +// 基础DFS遍历模板 +void dfs(int[][] grid, int i, int j) { + // 边界条件检查 + if (i < 0 || j < 0 || i >= grid.length || j >= grid[0].length) { + return; + } + // 标记已访问 + grid[i][j] = visited_mark; + + // 向四个方向递归 + dfs(grid, i + 1, j); // 下 + dfs(grid, i - 1, j); // 上 + dfs(grid, i, j + 1); // 右 + dfs(grid, i, j - 1); // 左 +} + +// 回溯DFS模板 +void backtrack(参数列表) { + if (终止条件) { + 收集结果; + return; + } + + for (选择 : 选择列表) { + 做选择; + backtrack(参数列表); + 撤销选择; + } +} +``` + +#### 边界检查清单 + +- ✅ 数组越界：确保索引在有效范围内 +- ✅ 访问标记：避免重复访问同一节点 +- ✅ 终止条件：明确递归的出口 +- ✅ 状态恢复：回溯时正确恢复状态 + +#### 💡 记忆口诀（朗朗上口版） + +- **基础DFS**：\"边界检查要仔细，标记访问防重复\" +- **树的遍历**：\"左右子树递归走，根据题意选时机\" +- **回溯算法**：\"做选择，递归下，撤选择，要记下\" +- **图的搜索**：\"四方向，都要走，条件满足才递归\" + +**最重要的一个技巧就是，你得行动，写起来** + +--- + +## 一、DFS 算法简介与核心思想 + +### 💡 核心思想 + +- **递归或栈实现**：通过递归隐式利用系统栈，或显式使用栈数据结构 +- **标记与回溯**：访问节点后标记防止重复，搜索到底后回溯到分叉点继续其他分支 +- **深度优先**：尽可能深地探索每个分支，直到无法继续再回溯探索其他分支 + +### 🎯 常见应用场景 + +1. **图的连通性检测**：如岛屿数量、被围绕的区域、判断图中是否存在一条路径 +2. **回溯问题**：组合、排列、子集、分割、数独、八皇后等 +3. **树的相关问题**：路径和、验证二叉搜索树、序列化等 +4. **矩阵搜索问题**：单词搜索、黄金矿工问题 +5. **拓扑排序与环检测**：课程表问题 + +### ⚡ 复杂度分析 + +- **时间复杂度**：通常为 $O(N)$ 或 $O(N+M)$（N 为节点数，M 为边数） +- **空间复杂度**：与递归深度相关，最坏情况 $O(N)$（如链表结构） + +--- + +## 二、基础DFS应用类（核心中的核心）🔥 + +**💡 核心思想** + +- **图的遍历**：深度优先遍历图或矩阵，标记已访问节点 +- **连通性检测**：判断图中节点的连通性，如岛屿数量问题 +- **路径搜索**：在图或矩阵中搜索满足条件的路径 + +**🎯 必掌握模板** + +```java +// 基础DFS遍历模板（四方向） +void dfs(int[][] grid, int i, int j) { + // 边界条件检查 + if (i < 0 || j < 0 || i >= grid.length || j >= grid[0].length || grid[i][j] == 0) { + return; + } + // 标记已访问 + grid[i][j] = 0; // 或其他标记值 + + // 向四个方向递归 + dfs(grid, i + 1, j); // 下 + dfs(grid, i - 1, j); // 上 + dfs(grid, i, j + 1); // 右 + dfs(grid, i, j - 1); // 左 +} +``` + +### [200. 岛屿数量](https://leetcode.cn/problems/number-of-islands/) + +**🎯 考察频率：极高 | 难度：中等 | 重要性：DFS基础经典** + +> 给你一个由 `'1'`（陆地）和 `'0'`（水）组成的的二维网格，请你计算网格中岛屿的数量。 +> +> 岛屿总是被水包围，并且每座岛屿只能由水平方向和/或竖直方向上相邻的陆地连接形成。 +> +> 此外，你可以假设该网格的四条边均被水包围。 +> +> ``` +> 输入：grid = [ +> ['1','1','0','0','0'], +> ['1','1','0','0','0'], +> ['0','0','1','0','0'], +> ['0','0','0','1','1'] +> ] +> 输出：3 +> ``` + +**💡 核心思路**：DFS + 标记已访问。为了求出岛屿的数量，我们可以扫描整个二维网格。如果一个位置为 1，则以其为起始节点开始进行深度优先搜索。在深度优先搜索的过程中，每个搜索到的 1 都会被重新标记为 0。 + +最终岛屿的数量就是我们进行深度优先搜索的次数。 + +```java +public class Solution { + public int numIslands(char[][] grid) { + int count = 0; + for (int i = 0; i < grid.length; i++) { + for (int j = 0; j < grid[0].length; j++) { + // 如果当前是陆地，才开始一次DFS遍历 + if (grid[i][j] == '1') { + dfs(grid, i, j); + count++; + } + } + } + return count; + } + + private void dfs(char[][] grid, int r, int c) { + // 边界检查和当前位置是否为陆地 + if (r < 0 || c < 0 || r >= grid.length || c >= grid[0].length || grid[r][c] != '1') { + return; + } + // 将当前位置标记为已访问 + grid[r][c] = '0'; // 淹没陆地 + + // 访问上、下、左、右四个相邻结点 + dfs(grid, r + 1, c); + dfs(grid, r - 1, c); + dfs(grid, r, c + 1); + dfs(grid, r, c - 1); + } +} +``` + +**⏱️ 复杂度分析**： +- **时间复杂度**：O(MN) - 每个格子最多访问一次 +- **空间复杂度**：O(MN) - 最坏情况下递归栈深度 + +### [79. 单词搜索](https://leetcode.cn/problems/word-search/) + +**🎯 考察频率：极高 | 难度：中等 | 重要性：DFS + 回溯经典** + +> 给定一个 `m x n` 二维字符网格 `board` 和一个字符串单词 `word`。如果 `word` 存在于网格中，返回 `true`；否则，返回 `false`。 +> +> 单词必须按照字母顺序，通过相邻的单元格内的字母构成，其中"相邻"单元格是那些水平相邻或垂直相邻的单元格。同一个单元格内的字母不允许被重复使用。 + +**💡 核心思路**：DFS + 回溯，从每个起点出发四方向搜索匹配字符 + +```java +public boolean exist(char[][] board, String word) { + // 遍历网格中的每一个单元格作为起始点 + for (int i = 0; i < board.length; i++) { + for (int j = 0; j < board[0].length; j++) { + if (dfs(board, word, 0, i, j)) { + return true; // 找到路径立即返回 + } + } + } + return false; // 全部遍历后仍未找到 +} + +private boolean dfs(char[][] board, String word, int idx, int r, int c) { + // 成功条件：已匹配完所有字符 + if (idx == word.length()) return true; + + // 边界检查 + 剪枝（当前字符不匹配） + if (r < 0 || c < 0 || r >= board.length || c >= board[0].length + || board[r][c] != word.charAt(idx)) { + return false; + } + + // 标记已访问（防止重复使用） + char tmp = board[r][c]; + board[r][c] = '#'; + + // 向四个方向递归搜索 + boolean res = dfs(board, word, idx + 1, r + 1, c) // 向下 + || dfs(board, word, idx + 1, r - 1, c) // 向上 + || dfs(board, word, idx + 1, r, c + 1) // 向右 + || dfs(board, word, idx + 1, r, c - 1); // 向左 + + // 回溯：恢复现场 + board[r][c] = tmp; + + return res; +} +``` + +**⏱️ 复杂度分析**： +- **时间复杂度**：O(MN × 3^L) - L为单词长度，每个位置最多3个方向（不回头） +- **空间复杂度**：O(L) - 递归栈深度 + +### [695. 岛屿的最大面积](https://leetcode.cn/problems/max-area-of-island/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：DFS计数应用** + +> 给你一个大小为 `m x n` 的二进制矩阵 `grid`。**岛屿**是由一些相邻的 `1` (代表土地) 构成的组合，这里的「相邻」要求两个 `1` 必须在 **水平或者竖直的四个方向上** 相邻。你可以假设 `grid` 的四个边缘都被 `0`（代表水）包围着。 +> +> 岛屿的面积是岛上值为 `1` 的单元格的数目。计算并返回 `grid` 中最大的岛屿面积。如果没有岛屿，则返回面积为 `0` 。 + +**💡 核心思路**：DFS计算每个岛屿面积，取最大值 + +```java +public int maxAreaOfIsland(int[][] grid) { + int maxArea = 0; + for (int i = 0; i < grid.length; i++) { + for (int j = 0; j < grid[0].length; j++) { + if (grid[i][j] == 1) { + maxArea = Math.max(maxArea, dfs(grid, i, j)); + } + } + } + return maxArea; +} + +private int dfs(int[][] grid, int r, int c) { + // 边界检查 + if (r < 0 || c < 0 || r >= grid.length || c >= grid[0].length || grid[r][c] != 1) { + return 0; + } + + grid[r][c] = 0; // 标记为已访问 + + // 当前面积1 + 四个方向的面积 + return 1 + dfs(grid, r + 1, c) + dfs(grid, r - 1, c) + + dfs(grid, r, c + 1) + dfs(grid, r, c - 1); +} +``` + +### [130. 被围绕的区域](https://leetcode.cn/problems/surrounded-regions/) + +**🎯 考察频率：中等 | 难度：中等 | 重要性：DFS + 边界处理** + +> 给你一个 `m x n` 的矩阵 `board` ，由若干字符 `'X'` 和 `'O'` ，找到所有被 `'X'` 围绕的区域，并将这些区域里所有的 `'O'` 用 `'X'` 填充。 + +**💡 核心思路**：从边界开始DFS，标记不被围绕的'O' + +```java +public void solve(char[][] board) { + int m = board.length, n = board[0].length; + + // 从边界的'O'开始DFS，标记为'A' + for (int i = 0; i < m; i++) { + if (board[i][0] == 'O') dfs(board, i, 0); + if (board[i][n-1] == 'O') dfs(board, i, n-1); + } + for (int j = 0; j < n; j++) { + if (board[0][j] == 'O') dfs(board, 0, j); + if (board[m-1][j] == 'O') dfs(board, m-1, j); + } + + // 处理结果：'A'改回'O'，'O'改为'X' + for (int i = 0; i < m; i++) { + for (int j = 0; j < n; j++) { + if (board[i][j] == 'A') { + board[i][j] = 'O'; + } else if (board[i][j] == 'O') { + board[i][j] = 'X'; + } + } + } +} + +private void dfs(char[][] board, int i, int j) { + if (i < 0 || j < 0 || i >= board.length || j >= board[0].length || board[i][j] != 'O') { + return; + } + + board[i][j] = 'A'; // 标记为不被围绕 + + dfs(board, i + 1, j); + dfs(board, i - 1, j); + dfs(board, i, j + 1); + dfs(board, i, j - 1); +} +``` + +**⏱️ 复杂度分析**： +- **时间复杂度**：O(MN) - 每个位置最多访问一次 +- **空间复杂度**：O(MN) - 最坏情况递归栈深度 + +--- + +## 三、树的DFS应用类（结构化思维）🌳 + +**💡 核心思想** + +- **前中后序遍历**：根据访问根节点的时机不同，处理不同类型的问题 +- **路径搜索**：从根到叶或任意节点间的路径问题 +- **树的性质验证**：利用DFS验证二叉搜索树、对称性等性质 +- **信息向上传递**：子树信息向父节点传递，如高度、直径等 + +**🎯 必掌握模板** + +```java +// 树的DFS模板（后序遍历获取子树信息） +int dfs(TreeNode root) { + if (root == null) return 0; + + int left = dfs(root.left); // 左子树信息 + int right = dfs(root.right); // 右子树信息 + + // 基于子树信息处理当前节点 + int currentResult = processNode(root, left, right); + + return currentResult; // 向上传递信息 +} +``` + +### [104. 二叉树的最大深度](https://leetcode.cn/problems/maximum-depth-of-binary-tree/) + +**🎯 考察频率：极高 | 难度：简单 | 重要性：DFS基础** + +> 给定一个二叉树，找出其最大深度。 +> +> 二叉树的深度为根节点到最远叶子节点的最长路径上的节点数。 + +**💡 核心思路**：后序遍历，子树深度+1 + +```java +public int maxDepth(TreeNode root) { + if (root == null) return 0; + + int leftHeight = maxDepth(root.left); + int rightHeight = maxDepth(root.right); + + return Math.max(leftHeight, rightHeight) + 1; +} +``` + +### [98. 验证二叉搜索树](https://leetcode.cn/problems/validate-binary-search-tree/) + +**🎯 考察频率：极高 | 难度：中等 | 重要性：DFS + 边界控制** + +> 给你一个二叉树的根节点 root ，判断其是否是一个有效的二叉搜索树。 + +**💡 核心思路**：DFS + 维护上下界范围 + +```java +public boolean isValidBST(TreeNode root) { + return validate(root, Long.MIN_VALUE, Long.MAX_VALUE); +} + +private boolean validate(TreeNode node, long min, long max) { + if (node == null) return true; + + if (node.val <= min || node.val >= max) return false; + + return validate(node.left, min, node.val) + && validate(node.right, node.val, max); +} +``` + +### [543. 二叉树的直径](https://leetcode.cn/problems/diameter-of-binary-tree/) + +**🎯 考察频率：高 | 难度：简单 | 重要性：DFS信息传递** + +> 给你一棵二叉树的根节点，返回该树的直径。二叉树的直径是指树中任意两个节点之间最长路径的长度。 + +**💡 核心思路**：后序遍历，计算经过每个节点的最长路径 + +```java +class Solution { + private int maxDiameter = 0; + + public int diameterOfBinaryTree(TreeNode root) { + height(root); + return maxDiameter; + } + + private int height(TreeNode node) { + if (node == null) return 0; + + int left = height(node.left); + int right = height(node.right); + + // 更新最大直径 + maxDiameter = Math.max(maxDiameter, left + right); + + return Math.max(left, right) + 1; + } +} +``` + +### [101. 对称二叉树](https://leetcode.cn/problems/symmetric-tree/) + +**🎯 考察频率：高 | 难度：简单 | 重要性：DFS + 递归对比** + +> 给你一个二叉树的根节点 root ，检查它是否轴对称。 + +**💡 核心思路**：递归比较左右子树的镜像关系 + +```java +public boolean isSymmetric(TreeNode root) { + if (root == null) return true; + return check(root.left, root.right); +} + +public boolean check(TreeNode left, TreeNode right) { + // 都为空，对称 + if (left == null && right == null) return true; + // 一个为空，不对称 + if (left == null || right == null) return false; + // 值不等，不对称 + if (left.val != right.val) return false; + + // 递归检查：左的左 vs 右的右，左的右 vs 右的左 + return check(left.left, right.right) && check(left.right, right.left); +} +``` + +### [111. 二叉树的最小深度](https://leetcode.cn/problems/minimum-depth-of-binary-tree/) + +**🎯 考察频率：中等 | 难度：简单 | 重要性：DFS边界处理** + +> 给定一个二叉树，找出其最小深度。最小深度是从根节点到最近叶子节点的最短路径上的节点数量。 + +**💡 核心思路**：DFS递归，注意单子树的情况 + +```java +public int minDepth(TreeNode root) { + if (root == null) return 0; + + // 叶子节点 + if (root.left == null && root.right == null) return 1; + + // 只有右子树 + if (root.left == null) return minDepth(root.right) + 1; + // 只有左子树 + if (root.right == null) return minDepth(root.left) + 1; + + // 都有子树，取最小值 + return Math.min(minDepth(root.left), minDepth(root.right)) + 1; +} +``` + +### [124. 二叉树中的最大路径和](https://leetcode.cn/problems/binary-tree-maximum-path-sum/) + +**🎯 考察频率：极高 | 难度：困难 | 重要性：DFS + 全局变量** + +> 路径被定义为一条从树中任意节点出发，沿父节点-子节点连接，达到任意节点的序列。同一个节点在一条路径序列中至多出现一次。路径至少包含一个节点，且不一定经过根节点。 + +**💡 核心思路**：DFS后序遍历，维护全局最大值 + +```java +class Solution { + private int maxSum = Integer.MIN_VALUE; + + public int maxPathSum(TreeNode root) { + maxGain(root); + return maxSum; + } + + private int maxGain(TreeNode node) { + if (node == null) return 0; + + // 递归计算左右子节点的最大贡献值 + // 只有在最大贡献值大于0时，才会选取对应子节点 + int leftGain = Math.max(maxGain(node.left), 0); + int rightGain = Math.max(maxGain(node.right), 0); + + // 节点的最大路径和取决于该节点的值与该节点的左右子节点的最大贡献值 + int priceNewPath = node.val + leftGain + rightGain; + + // 更新答案 + maxSum = Math.max(maxSum, priceNewPath); + + // 返回节点的最大贡献值 + return node.val + Math.max(leftGain, rightGain); + } +} +``` + +### [105. 从前序与中序遍历序列构造二叉树](https://leetcode.cn/problems/construct-binary-tree-from-preorder-and-inorder-traversal/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：DFS + 分治** + +> 给定两个整数数组 preorder 和 inorder ，其中 preorder 是二叉树的先序遍历， inorder 是同一棵树的中序遍历，请构造二叉树并返回其根节点。 + +**💡 核心思路**：DFS分治，前序确定根，中序确定左右子树 + +```java +public TreeNode buildTree(int[] preorder, int[] inorder) { + Map inorderMap = new HashMap<>(); + for (int i = 0; i < inorder.length; i++) { + inorderMap.put(inorder[i], i); + } + return build(preorder, 0, preorder.length - 1, + inorder, 0, inorder.length - 1, inorderMap); +} + +private TreeNode build(int[] preorder, int preStart, int preEnd, + int[] inorder, int inStart, int inEnd, + Map inorderMap) { + if (preStart > preEnd) return null; + + int rootVal = preorder[preStart]; + TreeNode root = new TreeNode(rootVal); + + int rootIndex = inorderMap.get(rootVal); + int leftSize = rootIndex - inStart; + + root.left = build(preorder, preStart + 1, preStart + leftSize, + inorder, inStart, rootIndex - 1, inorderMap); + root.right = build(preorder, preStart + leftSize + 1, preEnd, + inorder, rootIndex + 1, inEnd, inorderMap); + + return root; +} +``` + +### [538. 把二叉搜索树转换为累加树](https://leetcode.cn/problems/convert-bst-to-greater-tree/) + +**🎯 考察频率：中等 | 难度：中等 | 重要性：DFS + 中序遍历变形** + +> 给出二叉搜索树的根节点，该树的节点值各不相同，请你将其转换为累加树，使每个节点 node 的新值等于原树中大于或等于 node.val 的值之和。 + +**💡 核心思路**：反向中序遍历（右-根-左） + +```java +class Solution { + private int sum = 0; + + public TreeNode convertBST(TreeNode root) { + traverse(root); + return root; + } + + private void traverse(TreeNode node) { + if (node == null) return; + + traverse(node.right); // 先遍历右子树 + + node.val += sum; // 更新当前节点值 + sum = node.val; // 更新累加和 + + traverse(node.left); // 再遍历左子树 + } +} +``` + +### [437. 路径总和 III](https://leetcode.cn/problems/path-sum-iii/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：DFS + 前缀和** + +> 给定一个二叉树的根节点 root ，和一个整数 targetSum ，求该二叉树里节点值之和等于 targetSum 的路径的数目。 + +**💡 核心思路**：双重DFS，每个节点都可能是路径起点 + +```java +public int pathSum(TreeNode root, int targetSum) { + if (root == null) return 0; + + // 以root为起点的路径数 + 左子树中的路径数 + 右子树中的路径数 + return dfs(root, targetSum) + + pathSum(root.left, targetSum) + + pathSum(root.right, targetSum); +} + +private int dfs(TreeNode node, long target) { + if (node == null) return 0; + + int count = 0; + if (node.val == target) count++; + + count += dfs(node.left, target - node.val); + count += dfs(node.right, target - node.val); + + return count; +} +``` + +### + +--- + +## 四、回溯算法类（搜索所有可能）🔄 + +**💡 核心思想** + +- **搜索空间遍历**：系统性地搜索所有可能的解 +- **选择与回溯**：做选择→递归→撤销选择的三步骤 +- **剪枝优化**：提前终止不可能产生解的分支 +- **状态恢复**：确保每个选择互不影响 + +**🎯 必掌握模板** + +```java +// 回溯算法标准模板 +void backtrack(路径, 选择列表) { + if (满足终止条件) { + 结果.add(new ArrayList<>(路径)); + return; + } + + for (选择 : 选择列表) { + // 做选择 + 路径.add(选择); + + // 递归下一层 + backtrack(路径, 新的选择列表); + + // 撤销选择 + 路径.remove(路径.size() - 1); + } +} +``` + +### [46. 全排列](https://leetcode.cn/problems/permutations/) + +**🎯 考察频率：极高 | 难度：中等 | 重要性：回溯入门经典** + +> 给定一个不含重复数字的数组 nums ，返回其所有可能的全排列。 + +**💡 核心思路**：回溯，维护已使用元素列表 + +```java +public List> permute(int[] nums) { + List> res = new ArrayList<>(); + List path = new ArrayList<>(); + boolean[] used = new boolean[nums.length]; + + backtrack(nums, path, used, res); + return res; +} + +private void backtrack(int[] nums, List path, boolean[] used, List> res) { + if (path.size() == nums.length) { + res.add(new ArrayList<>(path)); + return; + } + + for (int i = 0; i < nums.length; i++) { + if (used[i]) continue; + + path.add(nums[i]); + used[i] = true; + backtrack(nums, path, used, res); + used[i] = false; + path.remove(path.size() - 1); + } +} +``` + +### [39. 组合总和](https://leetcode.cn/problems/combination-sum/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：回溯 + 剪枝** + +> 给你一个无重复元素的整数数组 candidates 和一个目标整数 target ，找出 candidates 中可以使数字和为目标数 target 的所有不同组合。 + +**💡 核心思路**：回溯 + 剪枝，允许重复使用元素 + +```java +public List> combinationSum(int[] candidates, int target) { + List> res = new ArrayList<>(); + Arrays.sort(candidates); // 排序便于剪枝 + backtrack(candidates, target, 0, new ArrayList<>(), res); + return res; +} + +private void backtrack(int[] nums, int remain, int start, List path, List> res) { + if (remain == 0) { + res.add(new ArrayList<>(path)); + return; + } + + for (int i = start; i < nums.length; i++) { + if (nums[i] > remain) break; // 剪枝 + + path.add(nums[i]); + backtrack(nums, remain - nums[i], i, path, res); // i不是i+1，允许重复 + path.remove(path.size() - 1); + } +} +``` + +### [22. 括号生成](https://leetcode.cn/problems/generate-parentheses/) + +**🎯 考察频率：极高 | 难度：中等 | 重要性：回溯 + 约束条件** + +> 数字 n 代表生成括号的对数，请你设计一个函数，用于能够生成所有可能的并且有效的括号组合。 + +**💡 核心思路**：回溯，维护左右括号数量约束 + +```java +public List generateParenthesis(int n) { + List res = new ArrayList<>(); + backtrack(n, 0, 0, new StringBuilder(), res); + return res; +} + +private void backtrack(int n, int left, int right, StringBuilder sb, List res) { + if (sb.length() == 2 * n) { + res.add(sb.toString()); + return; + } + + if (left < n) { + sb.append('('); + backtrack(n, left + 1, right, sb, res); + sb.deleteCharAt(sb.length() - 1); + } + + if (right < left) { + sb.append(')'); + backtrack(n, left, right + 1, sb, res); + sb.deleteCharAt(sb.length() - 1); + } +} +``` + +### [78. 子集](https://leetcode.cn/problems/subsets/) + +**🎯 考察频率：极高 | 难度：中等 | 重要性：回溯基础经典** + +> 给你一个整数数组 nums ，数组中的元素互不相同。返回该数组所有可能的子集（幂集）。 + +**💡 核心思路**：回溯，每个元素选择加入或不加入 + +```java +public List> subsets(int[] nums) { + List> res = new ArrayList<>(); + backtrack(nums, 0, new ArrayList<>(), res); + return res; +} + +private void backtrack(int[] nums, int start, List path, List> res) { + res.add(new ArrayList<>(path)); // 每个状态都是一个子集 + + for (int i = start; i < nums.length; i++) { + path.add(nums[i]); + backtrack(nums, i + 1, path, res); + path.remove(path.size() - 1); + } +} +``` + +### [17. 电话号码的字母组合](https://leetcode.cn/problems/letter-combinations-of-a-phone-number/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：回溯 + 映射** + +> 给定一个仅包含数字 2-9 的字符串，返回所有它能表示的字母组合。答案可以按任意顺序返回。 + +**💡 核心思路**：回溯 + 哈希映射 + +```java +public List letterCombinations(String digits) { + List res = new ArrayList<>(); + if (digits.length() == 0) return res; + + String[] mapping = {"", "", "abc", "def", "ghi", "jkl", "mno", "pqrs", "tuv", "wxyz"}; + backtrack(digits, 0, new StringBuilder(), res, mapping); + return res; +} + +private void backtrack(String digits, int index, StringBuilder path, + List res, String[] mapping) { + if (index == digits.length()) { + res.add(path.toString()); + return; + } + + String letters = mapping[digits.charAt(index) - '0']; + for (char c : letters.toCharArray()) { + path.append(c); + backtrack(digits, index + 1, path, res, mapping); + path.deleteCharAt(path.length() - 1); + } +} +``` + +### [131. 分割回文串](https://leetcode.cn/problems/palindrome-partitioning/) + +**🎯 考察频率：中等 | 难度：中等 | 重要性：回溯 + 预处理** + +> 给你一个字符串 s，请你将 s 分割成一些子串，使每个子串都是回文串。返回 s 所有可能的分割方案。 + +**💡 核心思路**：回溯 + 动态规划预处理回文串 + +```java +public List> partition(String s) { + List> res = new ArrayList<>(); + boolean[][] isPalindrome = preprocess(s); + backtrack(s, 0, new ArrayList<>(), res, isPalindrome); + return res; +} + +private boolean[][] preprocess(String s) { + int n = s.length(); + boolean[][] dp = new boolean[n][n]; + + for (int i = 0; i < n; i++) { + for (int j = 0; j <= i; j++) { + if (s.charAt(i) == s.charAt(j) && (i - j <= 2 || dp[j + 1][i - 1])) { + dp[j][i] = true; + } + } + } + return dp; +} + +private void backtrack(String s, int start, List path, + List> res, boolean[][] isPalindrome) { + if (start == s.length()) { + res.add(new ArrayList<>(path)); + return; + } + + for (int end = start; end < s.length(); end++) { + if (isPalindrome[start][end]) { + path.add(s.substring(start, end + 1)); + backtrack(s, end + 1, path, res, isPalindrome); + path.remove(path.size() - 1); + } + } +} +``` + +### [494. 目标和](https://leetcode.cn/problems/target-sum/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：DFS + 动态规划** + +> 给你一个整数数组 nums 和一个整数 target。向数组中的每个整数前添加 '+' 或 '-' ，然后串联起所有整数，可以构造一个表达式。返回可以通过上述方法构造的、运算结果等于 target 的不同表达式的数目。 + +**💡 核心思路**：DFS枚举所有可能，或转化为背包问题 + +```java +// 方法1：DFS回溯 +public int findTargetSumWays(int[] nums, int target) { + return dfs(nums, target, 0, 0); +} + +private int dfs(int[] nums, int target, int index, int sum) { + if (index == nums.length) { + return sum == target ? 1 : 0; + } + + return dfs(nums, target, index + 1, sum + nums[index]) + + dfs(nums, target, index + 1, sum - nums[index]); +} +``` + +--- + +## 六、图论算法类（关系网络处理）🔍 + +### [210. 课程表 II](https://leetcode.cn/problems/course-schedule-ii/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：DFS + 拓扑排序** + +> 现在你总共有 numCourses 门课需要选，记为 0 到 numCourses - 1。给你一个数组 prerequisites ，其中 prerequisites[i] = [ai, bi] ，表示在选修课程 ai 前必须先选修 bi 。返回你为了学完所有课程所安排的学习顺序。 + +**💡 核心思路**：DFS + 后序遍历得到拓扑序列 + +```java +class Solution { + private List> graph; + private int[] colors; // 0-白色, 1-灰色, 2-黑色 + private List result; + private boolean hasCycle = false; + + public int[] findOrder(int numCourses, int[][] prerequisites) { + graph = new ArrayList<>(); + colors = new int[numCourses]; + result = new ArrayList<>(); + + // 构建邻接表 + for (int i = 0; i < numCourses; i++) { + graph.add(new ArrayList<>()); + } + for (int[] pre : prerequisites) { + graph.get(pre[1]).add(pre[0]); + } + + // DFS检测环并生成拓扑序列 + for (int i = 0; i < numCourses; i++) { + if (colors[i] == 0) { + dfs(i); + } + } + + if (hasCycle) return new int[0]; + + Collections.reverse(result); + return result.stream().mapToInt(i -> i).toArray(); + } + + private void dfs(int node) { + colors[node] = 1; // 标记为正在访问 + + for (int neighbor : graph.get(node)) { + if (colors[neighbor] == 1) { + hasCycle = true; // 发现环 + return; + } + if (colors[neighbor] == 0) { + dfs(neighbor); + } + } + + colors[node] = 2; // 标记为已完成 + result.add(node); // 后序遍历添加到结果 + } +} +``` + +### [130. 被围绕的区域](https://leetcode.cn/problems/surrounded-regions/) + +**🎯 考察频率：中等 | 难度：中等 | 重要性：DFS + 边界处理** + +> 给你一个 m x n 的矩阵 board ，由若干字符 'X' 和 'O' ，找到所有被 'X' 围绕的区域，并将这些区域里所有的 'O' 用 'X' 填充。 + +**💡 核心思路**：从边界开始DFS，标记不被围绕的'O' + +```java +public void solve(char[][] board) { + int m = board.length, n = board[0].length; + + // 从边界的'O'开始DFS，标记为'A' + for (int i = 0; i < m; i++) { + if (board[i][0] == 'O') dfs(board, i, 0); + if (board[i][n-1] == 'O') dfs(board, i, n-1); + } + for (int j = 0; j < n; j++) { + if (board[0][j] == 'O') dfs(board, 0, j); + if (board[m-1][j] == 'O') dfs(board, m-1, j); + } + + // 处理结果：'A'改回'O'，'O'改为'X' + for (int i = 0; i < m; i++) { + for (int j = 0; j < n; j++) { + if (board[i][j] == 'A') { + board[i][j] = 'O'; + } else if (board[i][j] == 'O') { + board[i][j] = 'X'; + } + } + } +} + +private void dfs(char[][] board, int i, int j) { + if (i < 0 || j < 0 || i >= board.length || j >= board[0].length || board[i][j] != 'O') { + return; + } + + board[i][j] = 'A'; // 标记为不被围绕 + + dfs(board, i + 1, j); + dfs(board, i - 1, j); + dfs(board, i, j + 1); + dfs(board, i, j - 1); +} +``` + +--- + +## 七、高级技巧类（优化与变形）🌟 + +### [139. 单词拆分](https://leetcode.cn/problems/word-break/) + +**🎯 考察频率：极高 | 难度：中等 | 重要性：DFS + 记忆化** + +> 给你一个字符串 s 和一个字符串列表 wordDict，判断 s 是否可以由空格拆分为一个或多个在字典中出现的单词。 + +**💡 核心思路**：DFS + 记忆化搜索，或动态规划 + +```java +// 方法1：DFS + 记忆化 +public boolean wordBreak(String s, List wordDict) { + Set wordSet = new HashSet<>(wordDict); + Boolean[] memo = new Boolean[s.length()]; + return dfs(s, 0, wordSet, memo); +} + +private boolean dfs(String s, int start, Set wordSet, Boolean[] memo) { + if (start == s.length()) return true; + if (memo[start] != null) return memo[start]; + + for (int end = start + 1; end <= s.length(); end++) { + String word = s.substring(start, end); + if (wordSet.contains(word) && dfs(s, end, wordSet, memo)) { + memo[start] = true; + return true; + } + } + + memo[start] = false; + return false; +} +``` + + + +## 🚀 DFS算法速记表 + +### 核心模板（必背） + +```java +// 1. 基础DFS（四方向遍历） +void dfs(int[][] grid, int i, int j) { + if (边界检查 || 访问检查) return; + 标记已访问; + dfs(grid, i+1, j); dfs(grid, i-1, j); + dfs(grid, i, j+1); dfs(grid, i, j-1); +} + +// 2. 回溯算法模板 +void backtrack(路径, 选择列表) { + if (终止条件) { 收集结果; return; } + for (选择 : 选择列表) { + 做选择; backtrack(参数); 撤销选择; + } +} + +// 3. 树的DFS模板 +int dfs(TreeNode root) { + if (root == null) return 默认值; + int left = dfs(root.left); + int right = dfs(root.right); + return 处理当前节点(root, left, right); +} + +// 4. 图的环检测模板（三色标记法） +boolean hasCycle(int node) { + colors[node] = GRAY; // 正在访问 + for (neighbor : graph.get(node)) { + if (colors[neighbor] == GRAY) return true; // 发现环 + if (colors[neighbor] == WHITE && hasCycle(neighbor)) return true; + } + colors[node] = BLACK; // 访问完成 + return false; +} +``` + +### 题型速查表 + +| 题型分类 | 核心技巧 | 高频题目 | 记忆口诀 | 难度 | +|----------|----------|----------|----------|------| +| **基础DFS** | 四方向遍历+标记 | 岛屿数量、单词搜索、被围绕的区域、岛屿最大面积 | 边界检查要仔细，标记访问防重复 | ⭐⭐⭐ | +| **树的DFS** | 前中后序遍历 | 最大深度、验证BST、二叉树直径、对称二叉树、路径总和 | 左右子树递归走，根据题意选时机 | ⭐⭐⭐ | +| **回溯算法** | 选择+回溯+剪枝 | 全排列、组合总和、括号生成、子集、电话号码字母组合 | 做选择，递归下，撤选择，要记下 | ⭐⭐⭐⭐ | +| **图论应用** | 三色标记法 | 课程表、课程表II、拓扑排序 | 白色未访问，灰色正在访，黑色已完成 | ⭐⭐⭐⭐⭐ | +| **高级技巧** | 记忆化+优化 | 单词拆分、二叉树最大路径和、目标和 | 记忆化搜索避重复，全局变量传信息 | ⭐⭐⭐⭐⭐ | + +### 热题100 DFS完整清单 + +#### 🔥 基础DFS应用类（4题） +- [200. 岛屿数量](https://leetcode.cn/problems/number-of-islands/) - DFS基础经典 ⭐⭐⭐ +- [79. 单词搜索](https://leetcode.cn/problems/word-search/) - DFS + 回溯 ⭐⭐⭐⭐ +- [130. 被围绕的区域](https://leetcode.cn/problems/surrounded-regions/) - 边界DFS ⭐⭐⭐ +- [695. 岛屿的最大面积](https://leetcode.cn/problems/max-area-of-island/) - DFS计数 ⭐⭐⭐ + +#### 🌳 树的DFS应用类（9题） +- [104. 二叉树的最大深度](https://leetcode.cn/problems/maximum-depth-of-binary-tree/) - DFS基础 ⭐⭐ +- [98. 验证二叉搜索树](https://leetcode.cn/problems/validate-binary-search-tree/) - DFS + 边界 ⭐⭐⭐⭐ +- [101. 对称二叉树](https://leetcode.cn/problems/symmetric-tree/) - DFS递归对比 ⭐⭐⭐ +- [111. 二叉树的最小深度](https://leetcode.cn/problems/minimum-depth-of-binary-tree/) - DFS边界处理 ⭐⭐ +- [437. 路径总和III](https://leetcode.cn/problems/path-sum-iii/) - 双重DFS ⭐⭐⭐⭐ +- [543. 二叉树的直径](https://leetcode.cn/problems/diameter-of-binary-tree/) - DFS信息传递 ⭐⭐⭐ +- [538. 把二叉搜索树转换为累加树](https://leetcode.cn/problems/convert-bst-to-greater-tree/) - 反向中序 ⭐⭐⭐ +- [124. 二叉树中的最大路径和](https://leetcode.cn/problems/binary-tree-maximum-path-sum/) - DFS + 全局变量 ⭐⭐⭐⭐⭐ +- [105. 从前序与中序遍历序列构造二叉树](https://leetcode.cn/problems/construct-binary-tree-from-preorder-and-inorder-traversal/) - DFS分治 ⭐⭐⭐⭐ + +#### 🔄 回溯算法类（7题） +- [46. 全排列](https://leetcode.cn/problems/permutations/) - 回溯入门 ⭐⭐⭐⭐ +- [78. 子集](https://leetcode.cn/problems/subsets/) - 回溯基础 ⭐⭐⭐⭐ +- [39. 组合总和](https://leetcode.cn/problems/combination-sum/) - 回溯 + 剪枝 ⭐⭐⭐⭐ +- [22. 括号生成](https://leetcode.cn/problems/generate-parentheses/) - 回溯 + 约束 ⭐⭐⭐⭐ +- [17. 电话号码的字母组合](https://leetcode.cn/problems/letter-combinations-of-a-phone-number/) - 回溯 + 映射 ⭐⭐⭐ +- [131. 分割回文串](https://leetcode.cn/problems/palindrome-partitioning/) - 回溯 + 预处理 ⭐⭐⭐⭐ +- [494. 目标和](https://leetcode.cn/problems/target-sum/) - DFS枚举 ⭐⭐⭐⭐ + +#### 🔍 图论算法类（2题） +- [207. 课程表](https://leetcode.cn/problems/course-schedule/) - DFS环检测 ⭐⭐⭐⭐⭐ +- [210. 课程表II](https://leetcode.cn/problems/course-schedule-ii/) - DFS拓扑排序 ⭐⭐⭐⭐⭐ + +#### 🌟 高级技巧类（1题） +- [139. 单词拆分](https://leetcode.cn/problems/word-break/) - DFS + 记忆化 ⭐⭐⭐⭐⭐ + +### 核心题目优先级（面试前重点复习） + +**🏆 必须掌握（10题）** +1. **岛屿数量** - DFS基础，四方向遍历模板 +2. **单词搜索** - DFS + 回溯，状态恢复经典 +3. **全排列** - 回溯算法入门，选择与撤销 +4. **验证二叉搜索树** - 树的DFS + 边界控制 +5. **二叉树的最大深度** - 树的DFS最简单应用 +6. **括号生成** - 回溯 + 约束条件 +7. **课程表** - 图环检测，三色标记法 +8. **组合总和** - 回溯 + 剪枝优化 +9. **对称二叉树** - 递归对比经典 +10. **子集** - 回溯所有可能解 + +**⚡ 进阶强化（13题）** +- 二叉树的直径、路径总和III、岛屿最大面积 +- 电话号码字母组合、分割回文串、目标和 +- 被围绕的区域、课程表II、单词拆分 +- 二叉树中的最大路径和、从前序中序构造二叉树 +- 把二叉搜索树转换为累加树、二叉树的最小深度 + +### 常见陷阱提醒 + +- ⚠️ **忘记标记访问**：DFS遍历图时必须标记已访问节点，防止无限循环 +- ⚠️ **回溯忘记恢复**：回溯时必须撤销之前的选择，确保状态独立 +- ⚠️ **边界条件错误**：数组越界、空树处理、单子树情况要仔细 +- ⚠️ **重复计算**：可以考虑记忆化搜索优化，特别是树的问题 +- ⚠️ **栈溢出**：深度过大时考虑改用迭代实现或优化递归 +- ⚠️ **三色标记混乱**：图的环检测要正确使用WHITE、GRAY、BLACK三种状态 + +### 时间复杂度总结 + +- **基础DFS遍历**：O(V + E) - V为节点数，E为边数 +- **矩阵DFS**：O(MN) - M×N矩阵，每个格子最多访问一次 +- **回溯算法**：O(N!) ~ O(2^N) - 取决于解空间大小 +- **树的DFS**：O(N) - N为树的节点数 +- **图的环检测**：O(V + E) - 每个节点和边最多访问一次 + +**最重要的一个技巧就是，你得行动，写起来** diff --git a/docs/data-structure-algorithms/soultion/DP-Solution.md b/docs/data-structure-algorithms/soultion/DP-Solution.md new file mode 100755 index 0000000000..5d4dd93c96 --- /dev/null +++ b/docs/data-structure-algorithms/soultion/DP-Solution.md @@ -0,0 +1,1779 @@ +--- +title: 动态规划-热题 +date: 2025-01-08 +tags: + - DP + - algorithms +categories: leetcode +--- + +![](https://img.starfish.ink/leetcode/leetcode-banner.png) + +> **导读**：个人感觉动态规划是最难的，一会爬楼梯，一会兑零钱，一会又要去接雨水，炒股就不说了，还要去偷东西，太南了 +> +> 动态规划确实是算法面试的"终极boss"，但掌握了核心思路后，这些看似复杂的问题都有迹可循。从状态定义到转移方程，从边界条件到空间优化，让我们一步步征服DP的各个击破点。 +> +> **关键词**：状态定义、转移方程、记忆化搜索、空间优化、最优子结构 + +### 📋 分类索引 + +1. **🎯 线性DP类（基础核心）** + - **1.1 单串问题** + - 1.1.1 依赖比i小的O(1)个子问题：[爬楼梯](#_70-爬楼梯)、[打家劫舍](#_198-打家劫舍)、[最大子数组和](#_53-最大子数组和)、[单词拆分](#_139-单词拆分)、[杨辉三角](#_118-杨辉三角)、[乘积最大子数组](#_152-乘积最大子数组) + - 1.1.2 依赖比i小的O(n)个子问题：[最长递增子序列](#_300-最长递增子序列)、[完全平方数](#_279-完全平方数) + - 1.1.3 带维度的单串问题：[买卖股票的最佳时机](#_121-买卖股票的最佳时机) + - **1.2 双串问题**：[最长公共子序列](#_1143-最长公共子序列)、[编辑距离](#_72-编辑距离) + +2. **🎒 背包DP类（线性DP的变种）** + - **0-1背包问题**：[分割等和子集](#_416-分割等和子集) + - **完全背包问题**：[零钱兑换](#_322-零钱兑换)、[零钱兑换II](#_518-零钱兑换ii)、[完全平方数](#_279-完全平方数)、[目标和](#_494-目标和) + - **背包应用**：[不同路径](#_62-不同路径)、[最小路径和](#_64-最小路径和) + +3. **🔄 区间动态规划** + - **3.1 区间DP核心**：状态设计与推导顺序 + - **3.2 经典问题分类** + - 3.2.1 依赖常数个子问题：[不同的二叉搜索树](#_96-不同的二叉搜索树)、[最长回文子序列](#_最长回文子序列516) + - 3.2.2 依赖O(n)个子问题：[最长回文子串](#_最长回文子串5)、[最大正方形](#_221-最大正方形) + +4. **🔗 分治型动态规划**：[括号生成](#_括号生成22) + +### 🎯 核心考点概览 + +- **状态设计**：一维dp[i]、二维dp[i][j]、多维状态转移 +- **转移方程**：从"最后一步"推导状态转移关系 +- **边界条件**：base case的正确设置和处理 +- **空间优化**：滚动数组、压缩状态降低空间复杂度 +- **决策树模型**：理解最优子结构和无后效性 + +### 📝 解题万能模板 + +#### 基础模板 + +```java +// 一维DP模板 +int[] dp = new int[n + 1]; +dp[0] = baseCase; // 边界条件 +for (int i = 1; i <= n; i++) { + dp[i] = Math.max(dp[i-1] + nums[i], nums[i]); // 状态转移 +} + +// 二维DP模板 +int[][] dp = new int[m + 1][n + 1]; +// 初始化边界条件 +for (int i = 1; i <= m; i++) { + for (int j = 1; j <= n; j++) { + dp[i][j] = Math.max(dp[i-1][j], dp[i][j-1]); // 状态转移 + } +} + +// 背包问题模板 +int[] dp = new int[capacity + 1]; +for (int i = 0; i < items.length; i++) { + for (int j = capacity; j >= weight[i]; j--) { // 0-1背包逆序 + dp[j] = Math.max(dp[j], dp[j - weight[i]] + value[i]); + } +} +``` + +#### 边界检查清单 + +- ✅ 数组越界：确保索引在有效范围内 +- ✅ 边界条件：正确设置dp[0]、dp[i][0]等base case +- ✅ 状态转移：确保每个状态都能从更小的子问题推导 +- ✅ 空间优化：考虑是否可以用滚动数组降低空间复杂度 + +#### 💡 记忆口诀（朗朗上口版） + +- **状态定义**："定义状态含义清，子问题规模要递减" +- **转移方程**："最后一步怎么走，前面状态来推导" +- **边界条件**："边界条件要设好，递推基础不能少" +- **空间优化**："前后依赖看得清，滚动数组省空间" +- **背包问题**："物品容量两维度，选或不选做决策" +- **区间DP**："区间长度小到大，分割点上做文章" + +**解题心法** + +1. **状态定义是核心**：明确`dp`数组的含义，确保子问题能被有效分解。 +2. **转移方程从 "最后一步" 推导**：思考 " 计算`dp[i][j]`时，最后一步操作是什么 "（如编辑距离中的插入 / 删除 / 替换）。 +3. **边界条件要清晰**：如`dp[0][j]`（空串与另一串的关系）、`dp[i][i]`（长度为 1 的区间）等。 +4. **空间优化**：部分问题可通过滚动数组将二维`dp`优化为一维（如编辑距离、最长公共子序列）。 + +**最重要的一个技巧就是，你得行动，写起来** + +## 一、线性DP类（基础核心）🎯 + +### 💡 核心思想 + +- **一维状态**：dp[i]表示考虑前i个元素的最优解 +- **递推关系**：dp[i]依赖前面的状态dp[i-1]、dp[i-2]等 +- **时间复杂度**：通常为O(n)或O(n²) + +### 🎯 必掌握模板 + +```java +// 基础线性DP模板 +int[] dp = new int[n + 1]; +dp[0] = baseCase; // 设置边界条件 + +for (int i = 1; i <= n; i++) { + dp[i] = optimizeFunction(dp[i-1], dp[i-2], ...); // 状态转移 +} +return dp[n]; // 或者返回过程中的最优值 +``` + +线性动态规划的特点是**问题规模由单变量（或双变量）的位置表示**，状态推导按位置从小到大依次进行，较大规模的问题依赖较小规模的子问题。 + +### 1.1 单串问题 + +单串问题的输入为一个字符串或数组，状态通常定义为`dp[i]`，表示 " 考虑前`i`个元素（`[0..i]`）时的最优解 "。 + +**状态定义**：`dp[i]` 表示以第 `i` 个元素为结尾的子问题的解（如子序列、子数组）。 + +#### 1.1.1 依赖比 i 小的 O(1) 个子问题 + +`dp[i]` 仅与 `dp[i-1]` 等少数几个更小的子问题相关，时间复杂度通常为`O(n)`。 + +![状态推导方向1](https://assets.leetcode-cn.com/aliyun-lc-upload/uploads/2020/08/09/2-2-1.png) + +比如最大子数组和（53 题）：`dp[i]` 依赖 `dp[i-1]`，判断是否延续前序子数组。 + +##### [70. 爬楼梯](https://leetcode.cn/problems/climbing-stairs/) + +> 假设你正在爬楼梯。需要 `n` 阶你才能到达楼顶。 +> +> 每次你可以爬 `1` 或 `2` 个台阶。你有多少种不同的方法可以爬到楼顶呢？ +> +> ``` +> 输入：n = 3 +> 输出：3 +> 解释：有三种方法可以爬到楼顶。 +> 1. 1 阶 + 1 阶 + 1 阶 +> 2. 1 阶 + 2 阶 +> 3. 2 阶 + 1 阶 +> ``` + +**思路**：典型的斐波那契数列问题 + +- 状态定义：`dp[i]` 表示爬到第 `i` 阶楼梯的方法数 +- 转移方程：`dp[i] = dp[i-1] + dp[i-2]` （可以从第i-1阶爬1步，或从第i-2阶爬2步） +- 边界条件：`dp[0] = 1, dp[1] = 1` + +```java +public int climbStairs(int n) { + if (n <= 1) return 1; + + int[] dp = new int[n + 1]; + dp[0] = 1; // 0阶有1种方法（不爬） + dp[1] = 1; // 1阶有1种方法 + + for (int i = 2; i <= n; i++) { + dp[i] = dp[i - 1] + dp[i - 2]; + } + + return dp[n]; +} + +// 空间优化版本 +public int climbStairs(int n) { + if (n <= 1) return 1; + + int prev2 = 1, prev1 = 1; + for (int i = 2; i <= n; i++) { + int current = prev1 + prev2; + prev2 = prev1; + prev1 = current; + } + return prev1; +} +``` + +##### [198. 打家劫舍](https://leetcode.cn/problems/house-robber/) + +> 你是一个专业的小偷，计划偷窃沿街的房屋。每间房内都藏有一定的现金，影响你偷窃的唯一制约因素就是相邻的房屋装有相互连通的防盗系统，如果两间相邻的房屋在同一晚上被小偷闯入，系统会自动报警。 +> +> 给定一个代表每个房屋存放金额的非负整数数组，计算你不触动警报装置的情况下 ，一夜之间能够偷窃到的最高金额。 +> +> ``` +> 输入：[2,7,9,3,1] +> 输出：12 +> 解释：偷窃 1 号房屋 (金额 = 2), 偷窃 3 号房屋 (金额 = 9)，接着偷窃 5 号房屋 (金额 = 1)。 +> 偷窃到的最高金额 = 2 + 9 + 1 = 12 。 +> ``` + +**思路**：状态机DP的简化版 + +- 状态定义：`dp[i]` 表示偷窃前 `i+1` 间房屋能获得的最高金额 +- 转移方程：`dp[i] = Math.max(dp[i-1], dp[i-2] + nums[i])` （不偷第i间房 vs 偷第i间房） +- 边界条件：`dp[0] = nums[0]`, `dp[1] = Math.max(nums[0], nums[1])` + +```java +public int rob(int[] nums) { + if (nums == null || nums.length == 0) return 0; + if (nums.length == 1) return nums[0]; + + int n = nums.length; + int[] dp = new int[n]; + dp[0] = nums[0]; + dp[1] = Math.max(nums[0], nums[1]); + + for (int i = 2; i < n; i++) { + dp[i] = Math.max(dp[i - 1], dp[i - 2] + nums[i]); + } + + return dp[n - 1]; +} + +// 空间优化版本 +public int rob(int[] nums) { + if (nums == null || nums.length == 0) return 0; + if (nums.length == 1) return nums[0]; + + int prev2 = nums[0]; + int prev1 = Math.max(nums[0], nums[1]); + + for (int i = 2; i < nums.length; i++) { + int current = Math.max(prev1, prev2 + nums[i]); + prev2 = prev1; + prev1 = current; + } + + return prev1; +} +``` + +##### [53. 最大子数组和](https://leetcode.cn/problems/maximum-subarray/) + +> 给你一个整数数组 `nums` ，请你找出一个具有最大和的连续子数组（子数组最少包含一个元素），返回其最大和。 +> +> **子数组** 是数组中的一个连续部分。 +> +> ``` +> 输入：nums = [-2,1,-3,4,-1,2,1,-5,4] +> 输出：6 +> 解释：连续子数组 [4,-1,2,1] 的和最大，为 6 。 +> ``` + +思路： + +- 状态定义：`dp[i]` 表示以 `nums[i]`结尾的最大子数组和。 + +- 转移方程：若`dp[i-1]`为正，则将其与`nums[i]`拼接，否则仅保留`nums[i]`，即 `dp[i] = Math.max(nums[i], nums[i] + dp[i - 1]);` + +```java +public int maxSubArray(int[] nums) { + //特判 + if (nums == null || nums.length == 0) { + return 0; + } + //初始化 + int length = nums.length; + int[] dp = new int[length]; + // 初始值,只有一个元素的时候最大和即它本身 + dp[0] = nums[0]; + int ans = nums[0]; + // 状态转移 + for (int i = 1; i < length; i++) { + // 取当前元素的值 和 当前元素的值加上一次结果的值 中最大数 + dp[i] = Math.max(nums[i], dp[i - 1] + nums[i]); + // 输出结果：和最大数对比 取大 + ans = Math.max(ans, dp[i]); + } + return ans; +} +``` + + + +##### [139. 单词拆分](https://leetcode.cn/problems/word-break/) + +> 给你一个字符串 `s` 和一个字符串列表 `wordDict` 作为字典。如果可以利用字典中出现的一个或多个单词拼接出 `s` 则返回 `true`。 +> +> **注意：**不要求字典中出现的单词全部都使用，并且字典中的单词可以重复使用。 +> +> ``` +> 输入: s = "applepenapple", wordDict = ["apple", "pen"] +> 输出: true +> 解释: 返回 true 因为 "applepenapple" 可以由 "apple" "pen" "apple" 拼接成。 +> 注意，你可以重复使用字典中的单词。 +> ``` + +思路： + +1. 创建一个布尔数组`dp`，其中`dp[i]`表示字符串`s`的前`i`个字符（即子串`s.substring(0, i)`）是否可以被拆分。 +2. 初始化`dp[0]`为`true`，表示空字符串可以被拆分。 +3. 遍历每个位置`i`从 1 到字符串长度，对于每个`i`，再遍历所有可能的分割点`j`（`j`从 0 到`i-1`），检查子串`s.substring(j, i)`是否在字典中，并且前`j`个字符是否可以被拆分（即`dp[j]`是否为`true`）。 + +```java +public boolean wordBreak(String s, List wordDict){ + int n = s.length(); + boolean[] dp = new boolean[n + 1]; + // 空字符串总是可以被拆分的 + dp[0] = true; + + // 遍历字符串的每个位置 + for(int i = 1; i< n; i++){ + // 尝试从当前位置向前拆分字符串 + for(int j = 0; j < i; j++){ + // 如果dp[j]为true，且s.substring(j, i)是字典中的一个单词 + if(dp[j] && wordDict.coitains(s.substring(j, i))){ + // 则设置dp[i]为true，表示s的前i个字符也可以被拆分 + dp[i] = true; + // 找到一种拆分方式后，可以跳出内层循环 + break; + } + } + } + return dp[n]; +} +``` + + + +##### [118. 杨辉三角](https://leetcode.cn/problems/pascals-triangle/) + +> 给定一个非负整数 *`numRows`，*生成「杨辉三角」的前 *`numRows`* 行。 +> +> 在「杨辉三角」中，每个数是它左上方和右上方的数的和。 +> +> ![img](https://pic.leetcode-cn.com/1626927345-DZmfxB-PascalTriangleAnimated2.gif) +> +> +> +> ``` +> 输入: numRows = 5 +> 输出: [[1],[1,1],[1,2,1],[1,3,3,1],[1,4,6,4,1]] +> ``` + +思路： + +杨辉三角的特性： + +1. 第n行有n个数字 +2. 每行的第一个和最后一个数字都是1 +3. 其他数字等于上一行同位置和前一个位置数字之和 + +动态规划解法： + +- 状态定义：dp[i][j]表示第i行第j列的数字 +- 递推关系： + - `dp[i] = 1` (每行第一个数字) + - `dp[i][i] = 1` (每行最后一个数字) + - `dp[i][j] = dp[i-1][j-1] + dp[i-1`][j] (其他情况) +- 空间优化：可以只使用一维数组，从后向前更新 + +```java + public class Solution { + public List> generate(int numRows) { + // 初始化动态规划数组 + Integer[][] dp = new Integer[numRows][]; + // 遍历每一行 + for (int i = 0; i < numRows; i++) { + // 初始化当前行 + dp[i] = new Integer[i + 1]; + // 每一行的第一个和最后一个元素总是 1 + dp[i][0] = dp[i][i] = 1; + // 计算中间元素 + for (int j = 1; j < i; j++) { + // 中间元素等于上一行的相邻两个元素之和 + dp[i][j] = dp[i - 1][j - 1] + dp[i - 1][j]; + } + } + + // 将动态规划数组转换为结果列表 + List> result = new ArrayList<>(); + for (Integer[] row : dp) { + result.add(Arrays.asList(row)); + } + // 返回结果列表 + return result; +} +``` + + + +##### [152. 乘积最大子数组](https://leetcode.cn/problems/maximum-product-subarray/) + +> 给你一个整数数组 `nums` ，请你找出数组中乘积最大的非空连续 子数组（该子数组中至少包含一个数字），并返回该子数组所对应的乘积。 +> +> 测试用例的答案是一个 **32-位** 整数。 +> +> ``` +> 输入: nums = [2,3,-2,4] +> 输出: 6 +> 解释: 子数组 [2,3] 有最大乘积 6。 +> ``` + +思路：维护两个DP数组，一个记录当前位置为止的最大成绩，一个记录当前位置为止的最小乘机。因为负数✖️负数会得到正数，所以当前位置的最小乘积乘以下一个负数，可能会得到最大乘机。 + +```java +public int maxProduct(int[] nums) { + int n = nums.length; // 获取数组的长度 + if (n == 0) return 0; // 如果数组为空，则直接返回0 + + int[] maxProduct = new int[n]; // 用于存储到当前位置为止的最大乘积 + int[] minProduct = new int[n]; // 用于存储到当前位置为止的最小乘积（考虑负数情况） + maxProduct[0] = nums[0]; // 初始化第一个元素的最大乘积 + minProduct[0] = nums[0]; // 初始化第一个元素的最小乘积 + int result = nums[0]; // 初始化最大乘积结果为第一个元素 + + for (int i = 1; i < n; i++) { // 从第二个元素开始遍历数组 + // 如果当前元素是负数，则交换最大乘积和最小乘积（因为负数乘以负数得正数） + if (nums[i] < 0) { + int temp = maxProduct[i-1]; // 临时保存前一个位置的最大乘积 + maxProduct[i] = minProduct[i-1] * nums[i]; // 更新当前位置的最大乘积 + minProduct[i] = temp * nums[i]; // 更新当前位置的最小乘积 + } else { + // 如果当前元素是非负数，则正常更新最大乘积和最小乘积 + maxProduct[i] = Math.max(nums[i], maxProduct[i-1] * nums[i]); // 更新当前位置的最大乘积 + minProduct[i] = Math.min(nums[i], minProduct[i-1] * nums[i]); // 更新当前位置的最小乘积 + } + // 更新全局最大乘积结果 + result = Math.max(result, maxProduct[i]); + } + + return result; // 返回全局最大乘积结果 +} +``` + + + +#### 1.1.2 依赖比 i 小的 O(n) 个子问题 + +`dp[i]`需要遍历所有更小的`dp[j]`（`j < i`）才能计算，时间复杂度通常为`O(n²)`。 + +最长递增子序列（300 题）：`dp[i]` 需比较 `dp[0..i-1]` 中所有满足条件的子问题，取最大值加 1。 + + + +##### [300. 最长递增子序列](https://leetcode.cn/problems/longest-increasing-subsequence/) + +> 给你一个整数数组 `nums` ，找到其中最长严格递增子序列的长度。 +> +> **子序列** 是由数组派生而来的序列，删除（或不删除）数组中的元素而不改变其余元素的顺序。例如，`[3,6,2,7]` 是数组 `[0,3,1,6,2,2,7]` 的子序列。 +> +> ``` +> 输入：nums = [10,9,2,5,3,7,101,18] +> 输出：4 +> 解释：最长递增子序列是 [2,3,7,101]，因此长度为 4 。 +> ``` + +PS： 注意「子序列」和「子串」这两个名词的区别，子串一定是连续的，而子序列不一定是连续的 + +![image.png](https://pic.leetcode.cn/1717053793-JegWXx-image.png) + +```java + public int getLengthOfLIS(int[] nums) { + + if(nums == null || nums.length == 0){ + return 0; + } + + int[] dp = new int[nums.length]; + Arrays.fill(dp, 1); + + int maxLength = 1; //最长递增子序列的初始长度为1 + for (int i = 0; i < nums.length; i++) { //外层循环 (i): 遍历数组中的每个元素 + for (int j = 0; j < i; j++) { //内层循环 (j): 对于每个 i，遍历 0 到 i-1 的所有元素 + //当发现 nums[j] < nums[i] 时，说明 nums[i] 可以接在 nums[j] 后面形成更长的递增子序列 + if (nums[j] < nums[i]) { + //这里要注意是 nums[i] 还是 dp[i] + // 把 nums[i] 接在后面，即可形成长度为 dp[j] + 1，且以 nums[i] 为结尾的递增子序列 + //dp[j] + 1 表示以 nums[j] 结尾的子序列长度加上 nums[i] 后的新长度 + dp[i] = Math.max(dp[i], dp[j] + 1); //更新dp[i] + } + } + maxLength = Math.max(maxLength, dp[i]); + } + return res; + } +``` + +![](https://writings.sh/assets/images/posts/algorithm-longest-increasing-subsequence/longest-increasing-subsequence-dp1-2.jpeg) + +> 类似问题还有，最长递增子序列的个数、俄罗斯套娃信封问题 + + + +##### [279. 完全平方数](https://leetcode.cn/problems/perfect-squares/) + +> 给你一个整数 `n` ，返回 *和为 `n` 的完全平方数的最少数量* 。 +> +> **完全平方数** 是一个整数，其值等于另一个整数的平方；换句话说，其值等于一个整数自乘的积。例如，`1`、`4`、`9` 和 `16` 都是完全平方数，而 `3` 和 `11` 不是。 +> +> ``` +> 输入：n = 13 +> 输出：2 +> 解释：13 = 4 + 9 +> ``` + +思路：这题和 coin change 类似。 + +- 状态定义：`dp[i]`表示组成整数`i`所需的最少完全平方数个数。 + +- 转移方程：遍历所有小于`i`的完全平方数`j²`，则`dp[i] = min(dp[i], dp[i - j²] + 1)`。 + +```java +public int numSquares(int n) { + // 创建dp数组，dp[i]表示组成i所需的最少完全平方数个数 + int[] dp = new int[n + 1]; + + // 初始化dp数组，初始值设为最大值（n最多由n个1组成） + Arrays.fill(dp, n + 1); + dp[0] = 0; + + // 外层循环遍历从 1 到 n 的每个数字 + for (int i = 1; i <= n; i++) { + // 内层循环遍历所有可能的完全平方数 j*j（其中 j 从 1 开始，直到 j*j 不再小于等于 i） + for (int j = 1; j * j <= i; j++) { + // 更新 dp[i] 为 dp[i] 和 dp[i - j*j] + 1 中的较小值 + // dp[i - j*j] + 1 表示将 j*j 加到 i-j*j 的最少完全平方数之和上，再加上当前的 j*j + dp[i] = Math.min(dp[i], dp[i - j * j] + 1); + } + } + + return dp[n]; +} +``` + +- **`dp[i]`**：表示组成数字 `i` 所需的最少完全平方数个数 +- **`j*j`**：当前尝试的完全平方数（如1, 4, 9, 16...） +- **`i - j*j`**：使用当前平方数后剩余的数值 +- **`dp[i - j*j]`**：剩余数值所需的最少平方数个数 +- 为什么加 "1"？ + - "+1" 表示当前使用的这个平方数（`j*j`） + + + +#### 1.1.3 带维度的单串问题 + +状态需要额外维度（如 "交易次数"、"状态标识"），通常定义为`dp[i][k]`。 + +##### **股票系列问题** + +以 "买卖股票的最佳时机 IV" 为例，需限制交易次数`k`： + +- 状态定义：`dp[i][k][0]`表示第`i`天结束时，最多交易`k`次且不持有股票的最大利润；`dp[i][k][1]`表示持有股票的最大利润。 + +- 转移方程： + + ```plaintext + dp[i][k][0] = max(dp[i-1][k][0], dp[i-1][k][1] + prices[i]) // 不持有：要么前一天就不持有，要么今天卖出 + dp[i][k][1] = max(dp[i-1][k][1], dp[i-1][k-1][0] - prices[i]) // 持有：要么前一天就持有，要么今天买入（消耗一次交易） + ``` + +> #### 单串相关练习题 +> +> - 最经典单串 LIS 系列 +> - 最大子数组和系列 +> - 打家劫舍系列 +> - 变形：需要两个位置的情况 +> - 与其它算法配合 +> - 其它单串 dp[i] 问题 +> - 带维度单串 dp[i][k] +> - 股票系列 +> + + + +### 1.2 双串问题 + +双串问题的输入为两个字符串或数组，状态通常定义为`dp[i][j]`，表示 " 考虑第一个串的前`i`个元素和第二个串的前`j`个元素时的最优解 "，时间复杂度通常为`O(mn)`（`m`、`n`为两串长度）。 + +![状态推导方向2](https://assets.leetcode-cn.com/aliyun-lc-upload/uploads/2020/08/06/2-2-2.png) + +**状态定义**：`dp[i][j]` 表示第一个串前 `i` 个元素与第二个串前 `j` 个元素的子问题的解。 + +- 最长公共子序列（1143 题）：`dp[i][j]` 依赖 `dp[i-1][j-1]`（两字符相等）或 `max(dp[i-1][j], dp[i][j-1])`（两字符不等）。 + +##### [1143. 最长公共子序列](https://leetcode.cn/problems/longest-common-subsequence/) + +> 给定两个字符串 `text1` 和 `text2`，返回这两个字符串的最长 **公共子序列** 的长度。如果不存在 **公共子序列** ，返回 `0` 。 +> +> 一个字符串的 **子序列** 是指这样一个新的字符串：它是由原字符串在不改变字符的相对顺序的情况下删除某些字符（也可以不删除任何字符）后组成的新字符串。 +> +> - 例如，`"ace"` 是 `"abcde"` 的子序列，但 `"aec"` 不是 `"abcde"` 的子序列。 +> +> 两个字符串的 **公共子序列** 是这两个字符串所共同拥有的子序列。 +> +> ``` +> 输入：text1 = "abcde", text2 = "ace" +> 输出：3 +> 解释：最长公共子序列是 "ace" ，它的长度为 3 。 +> ``` + +思路： + +- 状态定义：`dp[i][j]`表示`text1[0..i]`与`text2[0..j]`的最长公共子序列长度。 + +- 转移方程： + + ```plaintext + if (text1[i] == text2[j]) { + dp[i][j] = dp[i-1][j-1] + 1; // 两字符相同，累加长度 + } else { + dp[i][j] = max(dp[i-1][j], dp[i][j-1]); // 取删除其中一个字符后的最大值 + } + ``` + +两个字符串这种，算是典型的二维动态规划问题 + +![img](https://miro.medium.com/v2/resize:fit:914/0*7Uhq4ZGfoxs1Ww3Z) + +```java +public int longestCommonSubsequence(String str1, String str2) { + // 获取两个字符串的长度 + int s1l = str1.length(); + int s2l = str2.length(); + + // 创建动态规划表，dp[i][j]表示str1前i个字符和str2前j个字符的LCS长度 + // 大小设为(s1l+1)×(s2l+1)是为了方便处理边界条件（空字符串的情况） + int [][]dp = new int[s1l+1][s2l+1]; + + // 动态规划填表过程 + for(int i=1; i<=s1l; i++){ // 遍历str1的每个字符 + for(int j=1; j<=s2l; j++){ // 遍历str2的每个字符 + //当前字符是公共子序列的一部分 + if(str1.charAt(i-1) == str2.charAt(j-1)){ + // 当前字符匹配，LCS长度等于左上角值加1 + dp[i][j] = 1 + dp[i-1][j-1]; + }else{ + // 当前字符不匹配，取上方或左方的较大值 + dp[i][j] = Math.max(dp[i-1][j], dp[i][j-1]); + } + } + } + + // 返回整个字符串的LCS长度，即dp表的右下角值 + return dp[s1l][s2l]; +} +``` + + + +##### [72. 编辑距离](https://leetcode.cn/problems/edit-distance/) + +> 给你两个单词 `word1` 和 `word2`， *请返回将 `word1` 转换成 `word2` 所使用的最少操作数* 。 +> +> 你可以对一个单词进行如下三种操作： +> +> - 插入一个字符 +> - 删除一个字符 +> - 替换一个字符 +> +> ``` +> 输入：word1 = "intention", word2 = "execution" +> 输出：5 +> 解释： +> intention -> inention (删除 't') +> inention -> enention (将 'i' 替换为 'e') +> enention -> exention (将 'n' 替换为 'x') +> exention -> exection (将 'n' 替换为 'c') +> exection -> execution (插入 'u') +> ``` + +思路： + +- 状态定义：`dp[i][j]`表示将`word1[0..i]`转换为`word2[0..j]`的最少操作数。 + +- 转移方程： + + ```plaintext + if (word1[i] == word2[j]) { + dp[i][j] = dp[i-1][j-1]; // 字符相同，无需操作 + } else { + // 插入（在word1后加word2[j]）、删除（删word1[i]）、替换（替换word1[i]为word2[j]） + dp[i][j] = min(dp[i][j-1], dp[i-1][j], dp[i-1][j-1]) + 1; + } + ``` + +编辑距离，也称 Levenshtein 距离，指两个字符串之间互相转换的最少修改次数，通常用于在信息检索和自然语言处理中度量两个序列的相似度。 + +1. 定义状态：设字符串 s 和 t 的长度分别为 m 和 n ，我们先考虑两字符串尾部的字符 *s[m-1]* 和 *t[n-1]*。 + + - 若 *s[m-1]* 和 *t[n-1]* 相同，我们可以跳过它们，直接考虑 *s[m-2]* 和 *t[n-2]*。 + - 若 *s[m-1]* 和 *t[n-1]* 不同，我们需要对 s 进行一次编辑（插入、删除、替换），使得两字符串尾部的字符相同，从而可以跳过它们，考虑规模更小的问题 + + 也就是说，我们在字符串 中进行的每一轮决策（编辑操作），都会使得 s 和 t 中剩余的待匹配字符发生变化。因此，状态为当前在 s和 t 中考虑的第 i 和第 j 个字符，记为 `dp[i][j]`。 + +2. 状态转移方程：`dp[i][j]` 对应的两个字符串的尾部字符 `s[i-1]` 和 `t[j-1]` ，根据不同的编辑操作（增、删、改）可以有 3 种情况 + + - 在 `s[i-1]` 之后添加 `t[j-1]` ，剩余子问题 `dp[i][j-1]` + - 删除 `t[j-1]`，剩余子问题 `dp[i-1][j]` + - 把 `s[i-1]` 替换为 `t[j-1]` ，剩余子问题 `dp[i-1][j-1]` + + 最优步数是这 3 种情况的最小值，`dp[i][j] = Math.min(Math.min(dp[i][j - 1], dp[i - 1][j]), dp[i - 1][j - 1]) + 1` + + 3. base case : `dp[i][0] = i;` `dp[0][j] = j;` + +```java +/* 编辑距离：动态规划 */ +int editDistanceDP(String s, String t) { + int n = s.length(), m = t.length(); + int[][] dp = new int[n + 1][m + 1]; + // 状态转移：首行首列 + for (int i = 1; i <= n; i++) { + dp[i][0] = i; + } + for (int j = 1; j <= m; j++) { + dp[0][j] = j; + } + // 状态转移：其余行和列 + for (int i = 1; i <= n; i++) { + for (int j = 1; j <= m; j++) { + if (s.charAt(i - 1) == t.charAt(j - 1)) { + // 若两字符相等，则直接跳过此两字符 + dp[i][j] = dp[i - 1][j - 1]; + } else { + // 最少编辑步数 = 插入、删除、替换这三种操作的最少编辑步数 + 1 + dp[i][j] = Math.min(Math.min(dp[i][j - 1], dp[i - 1][j]), dp[i - 1][j - 1]) + 1; + } + } + } + return dp[n][m]; +} +``` + +> **双串相关练习题** +> +> 1. 最经典双串 LCS 系列 +> 2. 字符串匹配系列 +> 3. 其它双串 dp[i][j] 问题 +> 4. 带维度双串 dp[i][j][k] +> 5. 712 两个字符串的最小 ASCII 删除和 +> + + + +## 二、背包问题（线性 DP 的变种） + +| **特征** | **线性DP** | **背包问题** | +| ------------ | ------------------------------------ | ------------------------------------ | +| **问题场景** | 依赖序列的线性结构（如数组、字符串） | 在容量限制下选择物品（组合优化） | +| **状态定义** | 通常为`dp[i]`或`dp[i][j]` | 通常为`dp[i][v]`（前i个物品，容量v） | +| **转移方向** | 沿线性方向递推（如从左到右） | 依赖物品选择和容量消耗 | +| **典型例题** | 70. 爬楼梯、300. LIS | 416. 分割等和子集、322. 零钱兑换 | + +**核心**：按物品和容量的线性顺序推进状态，本质是线性 DP 的扩展。 + +- **0-1 背包**：每个物品仅选一次，状态转移依赖上一轮（前 `i-1` 个物品）的解（如分割等和子集 416 题）。 +- **完全背包**：每个物品可重复选，状态转移依赖本轮（前 `i` 个物品）的解（如零钱兑换 322 题、零钱兑换 II518 题）。 + +背包问题可抽象为：给定一组物品（每个物品有重量和价值）和一个容量固定的背包，如何选择物品放入背包，使总价值最大且总重量不超过背包容量。 + +常见类型： + +- **0-1 背包**：每个物品只能选择一次（要么放入，要么不放入） +- **完全背包**：每个物品可以选择多次（无限重复使用） + +### 0-1 背包问题 + +> 给定一个背包，容量为W。有n个物品，每个物品有重量wt[i]和价值val[i]。问在不超过背包容量的前提下，能装的最大价值是多少？ + +我们可以将 0-1 背包问题看作一个由 轮决策组成的过程，对于每个物体都有不放入和放入两种决策，因此该问题满足决策树模型。 + +该问题的目标是求解“在限定背包容量下能放入物品的最大价值”，因此较大概率是一个动态规划问题。 + +1. 定义dp数组`dp[i][w]`表示对于前i个物品，当前背包容量为w时，可以装的最大价值。 + +2. 状态转移方程：对于每个物品来说，不放入背包，背包容量不变；放入背包，背包容量减小 + + 当我们做出物品 的决策后，剩余的是前 个物品决策的子问题，可分为以下两种情况 + + - **不放入物品** ：背包容量不变，最大价值 `dp[i][w] = dp[i-1][w]` ，继承之前的结果。 + - **放入物品** : 背包容量减少，则`dp[i][w] = dp[i-1][w - wt[i-1]] + val[i-1]`（注意：这里i-1是因为数组下标从0开始） + - 因此：`dp[i][w] = max(dp[i-1][w], dp[i-1][w - wt[i-1]] + val[i-1])` 注意：只有w>=wt[i-1]时才能选择放入。 + +3. 初始化： + + - `dp[w] = 0`，表示0个物品时，价值为0。 + - `dp[i] = 0`，表示背包容量为0时，价值为0。 + +4. 最终答案：`dp[n][W]` + +```java +/** + * 0-1背包问题求解 + * @param capacity 背包最大容量 + * @param weights 物品重量数组 + * @param values 物品价值数组 + * @return 最大价值 + */ +public static int solve(int capacity, int[] weights, int[] values) { + // 边界条件判断 + if (capacity <= 0 || weights == null || values == null || + weights.length != values.length) { + return 0; + } + + int n = weights.length; + int[] dp = new int[capacity + 1]; + + // 初始化dp数组，默认为0即可 + + // 遍历每个物品 + for (int i = 0; i < n; i++) { + // 逆序遍历背包容量，防止物品重复使用 + for (int j = capacity; j >= weights[i]; j--) { + // 状态转移方程 + dp[j] = Math.max(dp[j], dp[j - weights[i]] + values[i]); + } + } + + return dp[capacity]; +} +``` + + + + + +##### [416. 分割等和子集](https://leetcode.cn/problems/partition-equal-subset-sum/) + +> 给你一个 **只包含正整数** 的 **非空** 数组 `nums` 。请你判断是否可以将这个数组分割成两个子集，使得两个子集的元素和相等。 +> +> ``` +> 输入：nums = [1,5,11,5] +> 输出：true +> 解释：数组可以分割成 [1, 5, 5] 和 [11] 。 +> ``` + +思路：这也算背包问题，要转化下想法，**给一个可装载重量为 `sum / 2` 的背包和 `N` 个物品，每个物品的重量为 `nums[i]`。现在让你装物品，是否存在一种装法，能够恰好将背包装满**？ 现在变成了背包问题 + +1. 定义状态：**`dp[i][j] = x` 表示，对于前 `i` 个物品（`i` 从 1 开始计数），当前背包的容量为 `j` 时，若 `x`为 `true`，则说明可以恰好将背包装满，若 `x` 为 `false`，则说明不能恰好将背包装满** `boolean[][] dp = new boolean[n + 1][sum + 1];` + +2. 转移方程：以 `nums[i]` 算不算入子集来看 + + - **不算（不放入背包）** ：不把这第 `i` 个物品装入背包，而且还装满背包，那就看上一个状态 `dp[i-1][j]`，继承之前的结果 + + - **算入子集（放入物品）** ：是否能够恰好装满背包，取决于状态 `dp[i-1][j-nums[i-1]]` + +3. base case： `dp[..][0] = true` 和 `dp[0][..] = false` + +```java +public boolean canPartition(int[] nums) { + int sum = 0; + for (int num : nums) sum += num; + // 和为奇数时，不可能划分成两个和相等的集合 + if (sum % 2 != 0) return false; + int n = nums.length; + sum = sum / 2; + boolean[][] dp = new boolean[n + 1][sum + 1]; + // base case + for (int i = 0; i <= n; i++) + dp[i][0] = true; + + for (int i = 1; i <= n; i++) { + for (int j = 1; j <= sum; j++) { + if (j - nums[i - 1] < 0) { + // 背包容量不足，不能装入第 i 个物品 + dp[i][j] = dp[i - 1][j]; + } else { + // 装入或不装入背包 + dp[i][j] = dp[i - 1][j] || dp[i - 1][j - nums[i - 1]]; + } + } + } + return dp[n][sum]; +} + +``` + +空间优化，**`dp[i][j]` 都是通过上一行 `dp[i-1][..]` 转移过来的**，之前的数据都不会再使用了。 + +```java +public boolean canPartition(int[] nums) { + int sum = 0; + for (int num : nums) sum += num; + // 和为奇数时，不可能划分成两个和相等的集合 + if (sum % 2 != 0) return false; + int n = nums.length; + sum = sum / 2; + boolean[] dp = new boolean[sum + 1]; + + // base case + dp[0] = true; + + for (int i = 0; i < n; i++) { + for (int j = sum; j >= 0; j--) { + if (j - nums[i] >= 0) { + dp[j] = dp[j] || dp[j - nums[i]]; + } + } + } + return dp[sum]; +} +``` + + + +### 完全背包问题 + +完全背包中每种物品有无限个。状态转移方程稍有不同： +`dp[i][w] = max(dp[i-1][w], dp[i][w - wt[i-1]] + val[i-1]) `注意这里第二项是`dp[i][w-wt[i-1]]`，因为可以重复选择。 + +> 给定 n 个物品，第 i 个物品的重量为 *wgt[i-1]* 、价值为 *val[i-1]*，和一个容量为 *cap* 的背包。**每个物品可以重复选取**，问在限定背包容量下能放入物品的最大价值 + +思路：完全背包问题和 0-1 背包问题非常相似，**区别仅在于不限制物品的选择次数** + +```java +/** + * 完全背包问题求解 + * @param capacity 背包最大容量 + * @param weights 物品重量数组 + * @param values 物品价值数组 + * @return 最大价值 + */ +public static int solve(int capacity, int[] weights, int[] values) { + // 边界条件判断 + if (capacity <= 0 || weights == null || values == null || + weights.length != values.length) { + return 0; + } + + int n = weights.length; + int[] dp = new int[capacity + 1]; + + // 遍历每个物品 + for (int i = 0; i < n; i++) { + // 正序遍历背包容量，允许物品重复使用 + for (int j = weights[i]; j <= capacity; j++) { + // 状态转移方程 + dp[j] = Math.max(dp[j], dp[j - weights[i]] + values[i]); + } + } + + return dp[capacity]; +} +``` + + + +##### [322. 零钱兑换](https://leetcode.cn/problems/coin-change/) + +> 给你一个整数数组 `coins` ，表示不同面额的硬币；以及一个整数 `amount` ，表示总金额。 +> +> 计算并返回可以凑成总金额所需的 **最少的硬币个数** 。如果没有任何一种硬币组合能组成总金额，返回 `-1` 。 +> +> 你可以认为每种硬币的数量是无限的。 +> +> ``` +> 输入：coins = [1, 2, 5], amount = 11 +> 输出：3 +> 解释：11 = 5 + 5 + 1 +> ``` + +思路： + +**零钱兑换可以看作完全背包问题的一种特殊情况** + +- “物品”对应“硬币”、“物品重量”对应“硬币面值”、“背包容量”对应“目标金额”。 +- 优化目标相反，完全背包问题是要最大化物品价值，零钱兑换问题是要最小化硬币数量 + +1. 定义状态：只使用 `coins` 中的前 `i` 个（`i` 从 1 开始计数）硬币的面值，若想凑出金额 `j`，最少有 `dp[i][j]` 种凑法 +2. 状态转移方程：`dp[i][j] = Math.min(dp[i-1][j],dp[i,j-coins[i-1]+1])` +3. base case：`dp[0][a] = MAX` + + + +```java +/* 零钱兑换：动态规划 */ +int coinChangeDP(int[] coins, int amt) { + int n = coins.length; + //amt + 1 表示无效解 + int MAX = amt + 1; + // 初始化 dp 表 + int[][] dp = new int[n + 1][amt + 1]; + // 状态转移：首行首列 + for (int a = 1; a <= amt; a++) { + dp[0][a] = MAX; + } + // 状态转移：其余行和列 + for (int i = 1; i <= n; i++) { + for (int a = 1; a <= amt; a++) { + if (coins[i - 1] > a) { + // 若超过目标金额，则不选硬币 i + dp[i][a] = dp[i - 1][a]; + } else { + // 不选和选硬币 i 这两种方案的较小值 ，硬币数量而非商品价值，因此在选中硬币时执行 +1 + dp[i][a] = Math.min(dp[i - 1][a], dp[i][a - coins[i - 1]] + 1); + } + } + } + return dp[n][amt] != MAX ? dp[n][amt] : -1; +} +``` + +![](https://www.hello-algo.com/chapter_dynamic_programming/unbounded_knapsack_problem.assets/coin_change_dp_step1.png)空间优化后 + +```java +/* 零钱兑换：空间优化后的动态规划 */ +int coinChangeDPComp(int[] coins, int amt) { + int n = coins.length; + int MAX = amt + 1; + // 初始化 dp 表 + int[] dp = new int[amt + 1]; + Arrays.fill(dp, MAX); + dp[0] = 0; + // 状态转移 + for (int i = 1; i <= n; i++) { + for (int a = 1; a <= amt; a++) { + if (coins[i - 1] > a) { + // 若超过目标金额，则不选硬币 i + dp[a] = dp[a]; + } else { + // 不选和选硬币 i 这两种方案的较小值 + dp[a] = Math.min(dp[a], dp[a - coins[i - 1]] + 1); + } + } + } + return dp[amt] != MAX ? dp[amt] : -1; +} +``` + + + +##### [518. 零钱兑换 II](https://leetcode.cn/problems/coin-change-ii/) + +> 给你一个整数数组 `coins` 表示不同面额的硬币，另给一个整数 `amount` 表示总金额。 +> +> 请你计算并返回可以凑成总金额的硬币组合数。如果任何硬币组合都无法凑出总金额，返回 `0` 。 +> +> 假设每一种面额的硬币有无限个。 +> +> 题目数据保证结果符合 32 位带符号整数。 +> +> ``` +> 输入：amount = 5, coins = [1, 2, 5] +> 输出：4 +> 解释：有四种方式可以凑成总金额： +> 5=5 +> 5=2+2+1 +> 5=2+1+1+1 +> 5=1+1+1+1+1 +> ``` + +思路： + +可以把这个问题转化为背包问题的描述形式： + +有一个背包，最大容量为 `amount`，有一系列物品 `coins`，每个物品的重量为 `coins[i]`，**每个物品的数量无限**。请问有多少种方法，能够把背包恰好装满？ + +这个问题和我们前面讲过的两个背包问题，有一个最大的区别就是，每个物品的数量是无限的，这也就是传说中的「**完全背包问题**」， + +1. 定义状态：**若只使用 `coins` 中的前 `i` 个（`i` 从 1 开始计数）硬币的面值，若想凑出金额 `j`，有 `dp[i][j]` 种凑法**。 +2. base case 为 `dp[0][..] = 0, dp[..][0] = 1`。`i = 0` 代表不使用任何硬币面值，这种情况下显然无法凑出任何金额；`j = 0` 代表需要凑出的目标金额为 0，那么什么都不做就是唯一的一种凑法 + + + +### 背包应用 + +##### [494. 目标和](https://leetcode.cn/problems/target-sum/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：01背包变体** + +> 给你一个整数数组 nums 和一个整数 target 。 +> +> 向数组中的每个整数前添加 '+' 或 '-' ，然后串联起所有整数，可以构造一个 表达式 ： +> +> 例如，nums = [2, 1] ，可以在 2 之前添加 '+' ，在 1 之前添加 '-' ，然后串联起来得到表达式 "+2-1" 。 +> 返回可以通过上述方法构造的、运算结果等于 target 的不同 表达式 的数目。 +> +> ``` +> 输入：nums = [1,1,1,1,1], target = 3 +> 输出：5 +> 解释：一共有 5 种方法让最终目标和为 3 。 +> -1 + 1 + 1 + 1 + 1 = 3 +> +1 - 1 + 1 + 1 + 1 = 3 +> +1 + 1 - 1 + 1 + 1 = 3 +> +1 + 1 + 1 - 1 + 1 = 3 +> +1 + 1 + 1 + 1 - 1 = 3 +> ``` + +**思路**：转化为01背包问题 + +设添加"+"号的数字和为P，添加"-"号的数字和为N，则： +- P + N = sum（数组总和） +- P - N = target（目标和） + +解得：P = (sum + target) / 2 + +问题转化为：从数组中选择一些数字，使其和等于P的方案数。这是典型的01背包计数问题。 + +```java +public int findTargetSumWays(int[] nums, int target) { + int sum = 0; + for (int num : nums) { + sum += num; + } + + // 边界条件检查 + if (sum < Math.abs(target) || (sum + target) % 2 == 1) { + return 0; + } + + int targetSum = (sum + target) / 2; + + // dp[i]表示和为i的方案数 + int[] dp = new int[targetSum + 1]; + dp[0] = 1; // 和为0的方案数为1（什么都不选） + + for (int num : nums) { + // 01背包，逆序遍历 + for (int j = targetSum; j >= num; j--) { + dp[j] += dp[j - num]; + } + } + + return dp[targetSum]; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n × target_sum) - n为数组长度，target_sum为目标正数和 +- **空间复杂度**：O(target_sum) - dp数组的空间 + +### 背包应用 + +##### [62. 不同路径](https://leetcode.cn/problems/unique-paths/) + +> 一个机器人位于一个 `m x n` 网格的左上角 （起始点在下图中标记为 "Start" ）。 +> +> 机器人每次只能向下或者向右移动一步。机器人试图达到网格的右下角（在下图中标记为 "Finish" ）。 +> +> 问总共有多少条不同的路径？ +> +> ``` +> 输入：m = 3, n = 7 +> 输出：28 +> ``` + +**思路**：典型的二维DP问题 + +- 状态定义：`dp[i][j]` 表示到达位置 `(i, j)` 的不同路径数 +- 转移方程：`dp[i][j] = dp[i-1][j] + dp[i][j-1]` （只能从上方或左方到达） +- 边界条件：`dp[0][j] = 1`, `dp[i][0] = 1` （第一行和第一列都只有一条路径） + +```java +public int uniquePaths(int m, int n) { + int[][] dp = new int[m][n]; + + // 初始化边界条件 + for (int i = 0; i < m; i++) { + dp[i][0] = 1; // 第一列只有一条路径 + } + for (int j = 0; j < n; j++) { + dp[0][j] = 1; // 第一行只有一条路径 + } + + // 状态转移 + for (int i = 1; i < m; i++) { + for (int j = 1; j < n; j++) { + dp[i][j] = dp[i - 1][j] + dp[i][j - 1]; + } + } + + return dp[m - 1][n - 1]; +} + +// 空间优化版本（一维数组） +public int uniquePaths(int m, int n) { + int[] dp = new int[n]; + Arrays.fill(dp, 1); // 第一行都是1 + + for (int i = 1; i < m; i++) { + for (int j = 1; j < n; j++) { + dp[j] = dp[j] + dp[j - 1]; // dp[j]代表上方，dp[j-1]代表左方 + } + } + + return dp[n - 1]; +} +``` + +##### [121. 买卖股票的最佳时机](https://leetcode.cn/problems/best-time-to-buy-and-sell-stock/) + +> 给定一个数组 `prices` ，它的第 `i` 个元素 `prices[i]` 表示一支给定股票第 `i` 天的价格。 +> +> 你只能选择 **某一天** 买入这只股票，并选择在 **未来的某一个不同的日子** 卖出该股票。设计一个算法来计算你所能获取的最大利润。 +> +> 返回你可以从这笔交易中获取的最大利润。如果你不能获取任何利润，返回 `0` 。 +> +> ``` +> 输入：[7,1,5,3,6,4] +> 输出：5 +> 解释：在第 2 天（股票价格 = 1）的时候买入，在第 5 天（股票价格 = 6）的时候卖出，最大利润 = 6-1 = 5 。 +> 注意利润不能是 7-1 = 6, 因为卖出价格需要大于买入价格；同时，你不能在买入前卖出股票。 +> ``` + +**思路**：一次遍历找最低买入价和最大利润 + +- 状态定义：维护到当前位置的最低价格和最大利润 +- 转移方程： + - `minPrice = Math.min(minPrice, prices[i])` + - `maxProfit = Math.max(maxProfit, prices[i] - minPrice)` + +```java +public int maxProfit(int[] prices) { + if (prices == null || prices.length <= 1) return 0; + + int minPrice = prices[0]; // 记录到目前为止的最低价格 + int maxProfit = 0; // 记录到目前为止的最大利润 + + for (int i = 1; i < prices.length; i++) { + // 更新最低价格 + minPrice = Math.min(minPrice, prices[i]); + // 更新最大利润（今天卖出的利润 vs 之前的最大利润） + maxProfit = Math.max(maxProfit, prices[i] - minPrice); + } + + return maxProfit; +} + +// DP思路版本 +public int maxProfit(int[] prices) { + if (prices == null || prices.length <= 1) return 0; + + // dp[i][0] 表示第i天不持有股票的最大利润 + // dp[i][1] 表示第i天持有股票的最大利润 + int n = prices.length; + int[][] dp = new int[n][2]; + + dp[0][0] = 0; // 第0天不持有股票，利润为0 + dp[0][1] = -prices[0]; // 第0天持有股票，利润为-prices[0] + + for (int i = 1; i < n; i++) { + dp[i][0] = Math.max(dp[i-1][0], dp[i-1][1] + prices[i]); // 不持有：要么昨天就不持有，要么今天卖出 + dp[i][1] = Math.max(dp[i-1][1], -prices[i]); // 持有：要么昨天就持有，要么今天买入 + } + + return dp[n-1][0]; // 最后一天不持有股票的最大利润 +} +``` + +##### [64. 最小路径和](https://leetcode.cn/problems/minimum-path-sum/) + +> 给定一个包含非负整数的 `m * x ` 网格 `grid` ，请找出一条从左上角到右下角的路径，使得路径上的数字总和为最小。 +> +> **说明：**每次只能向下或者向右移动一步。 +> +> ![img](https://assets.leetcode.com/uploads/2020/11/05/minpath.jpg) +> +> ``` +> 输入：grid = [[1,3,1],[1,5,1],[4,2,1]] +> 输出：7 +> 解释：因为路径 1→3→1→1→1 的总和最小。 +> ``` + +思路： + +递归解法，dp[i][l] + +```java +int dp(int[][] grid, int i, int j) { + // base case + if (i == 0 && j == 0) { + return grid[0][0]; + } + // 如果索引出界，返回一个很大的值， + // 保证在取 min 的时候不会被取到 + if (i < 0 || j < 0) { + return Integer.MAX_VALUE; + } + + // 左边和上面的最小路径和加上 grid[i][j] + // 就是到达 (i, j) 的最小路径和 + return Math.min( + dp(grid, i - 1, j), + dp(grid, i, j - 1) + ) + grid[i][j]; +} +``` + + + + +## 三、区间动态规划 + +在输入为长度为 n 的数组时，子问题用区间 [i..j] 表示。状态的定义和转移都与区间有关，称为区间动态规划。 + +区间动态规划的特点是**状态与区间`[i..j]`相关**，状态定义为`dp[i][j]`，表示 " 区间`[i..j]`上的最优解 "，推导顺序通常按区间长度从小到大进行。 + +> 区间动态规划一般用在单串问题上，以区间 [i, j] 为单位思考状态的设计和转移。它与线性动态规划在状态设计和状态转移上都有明显的不同，但由于这两个方法都经常用在单串问题上，导致我们拿到一个单串的问题时，经常不能快速反映出应该用哪种方法。这是区间动态规划的难点之一，但是这个难点也是好解决的，就是做一定数量的练习题，因为区间动态规划的题目比线性动态规划少很多，并且区间动态规划的状态设计和转移都比较朴素，变化也比线性动态规划少很多，所以通过不多的题目数量就可以把区间动态规划常见的方法和变化看个大概了。 + + + +> 区间 DP 是状态的定义和转移都与区间有关，其中区间用两个端点表示。 +> +> 状态定义 dp\[i][j] = [i..j] 上原问题的解。i 变大，j 变小都可以得到更小规模的子问题。 +> +> 对于单串上的问题，我们可以对比一下线性动态规划和区间动态规划。线性动态规划, 一般是定义 dp[i]， 表示考虑到前 i 个元素，原问题的解，i 变小即得到更小规模的子问题，推导状态时候是从前往后，即 i 从小到大推的。区间动态规划，一般是定义 dp[i][j]，表示考虑 [i..j] 范围内的元素，原问题的解增加 i，减小 j 都可以得到更小规模的子问题。推导状态一般是按照区间长度从短到长推的。 +> +> 区间动态规划的状态设计，状态转移都与线性动态规划有明显区别，但是由于这两种方法都经常用在单串问题上，拿到一个单串的问题时，往往不能快速地判断到底是用线性动态规划还是区间动态规划，这也是区间动态规划的难点之一。 +> +> 状态转移，推导状态 dp\[i][j] 时，有两种常见情况 +> +> #### 1. dp\[i][j] 仅与常数个更小规模子问题有关 +> +> 一般是与 dp\[i + 1][j], dp\[i][j - 1], dp\[i + 1][j - 1] 有关。 +> +> dp\[i][j] = f(dp\[i + 1][j], dp\[i][j - 1], dp\[i + 1][j - 1]) +> +> ![img](https://assets.leetcode-cn.com/aliyun-lc-upload/uploads/2020/08/06/4-1-1.png) +> +> 代码常见写法 +> +> ``` +> for len = 1..n +> for i = i..len +> j = i + len - 1 +> dp[i][j] = max(dp[i][j], f(dp[i+1][j], dp[i][j-1], dp[i+1][j-1])) +> +> ``` +> +> 时间复杂度和空间复杂度均为 O(n^{2})*O*(*n*2) +> +> 例如力扣第 516 题，详细过程参考下一节。 +> +> +> +> #### 2. dp\[i][j] 与 O(n) 个更小规模子问题有关 +> +> 一般是枚举 [i,j] 的分割点，将区间分为 [i,k] 和 [k+1,j]，对每个 k 分别求解（下面公式的 f），再汇总（下面公式的 g）。 +> +> dp\[i][j] = g(f(dp\[i][k], dp\[k + 1][j])) 其中 k = i .. j-1。 +> +> ![img](https://assets.leetcode-cn.com/aliyun-lc-upload/uploads/2020/08/06/4-1-2.png) +> +> 代码常见写法, 以下代码以 f 为 max 为例 +> +> ``` +> for len = 1..n +> for i = i..len +> j = i + len - 1 +> for k = i..j +> dp[i][j] = max(dp[i][j], f(dp[i][k], dp[k][j])) +> ``` +> +> 时间复杂度可以达到 O(n^3)，空间复杂度还是 O(n^2) +> +> 例如力扣第 664 题，详细过程参考下一节 +> +> + +### 3.1 区间 DP 的核心 + +- 状态设计：`dp[i][j]`代表区间`[i..j]`的解，`i`和`j`分别为区间的左右端点。 +- 推导顺序：先计算长度为 1 的区间（`i == j`），再计算长度为 2 的区间（`j = i+1`），直至长度为`n`的区间（`j = n-1`）。 + +### 3.2 经典问题分类 + +#### 3.2.1 依赖常数个子问题 + +`dp[i][j]`仅与`dp[i+1][j]`、`dp[i][j-1]`、`dp[i+1][j-1]`相关，时间复杂度`O(n²)`。 + +##### [96. 不同的二叉搜索树](https://leetcode.cn/problems/unique-binary-search-trees/) + +> 给你一个整数 `n` ，求恰由 `n` 个节点组成且节点值从 `1` 到 `n` 互不相同的 **二叉搜索树** 有多少种？返回满足题意的二叉搜索树的种数。 +> +> ``` +> 输入：n = 3 +> 输出：5 +> 解释：给定 n = 3, 一共有 5 种不同结构的二叉搜索树 +> ``` + +**思路**：卡特兰数/区间DP + +当我们确定根节点为 i 时： +- 左子树包含 [1, i-1] 共 i-1 个节点 +- 右子树包含 [i+1, n] 共 n-i 个节点 +- 左子树的不同结构数为 `dp[i-1]` +- 右子树的不同结构数为 `dp[n-i]` +- 以 i 为根的不同BST数为 `dp[i-1] * dp[n-i]` + +- 状态定义：`dp[i]` 表示 i 个节点组成的不同BST数量 +- 转移方程：`dp[i] = sum(dp[j-1] * dp[i-j])` for j in [1, i] +- 边界条件：`dp[0] = 1, dp[1] = 1` + +```java +public int numTrees(int n) { + // dp[i] 表示i个节点组成的不同BST的数量 + int[] dp = new int[n + 1]; + + // 边界条件 + dp[0] = 1; // 空树有1种结构 + dp[1] = 1; // 一个节点有1种结构 + + // 计算dp[2]到dp[n] + for (int i = 2; i <= n; i++) { + // 枚举根节点j的位置（从1到i） + for (int j = 1; j <= i; j++) { + // j为根节点时： + // 左子树有j-1个节点，右子树有i-j个节点 + dp[i] += dp[j - 1] * dp[i - j]; + } + } + + return dp[n]; +} + +// 数学公式版本（卡特兰数第n项） +public int numTrees(int n) { + long result = 1; + for (int i = 0; i < n; i++) { + result = result * 2 * (2 * i + 1) / (i + 2); + } + return (int) result; +} +``` + +##### [516. 最长回文子序列](https://leetcode.cn/problems/longest-palindromic-subsequence/) + +> 给你一个字符串 `s` ，找出其中最长的回文子序列，并返回该序列的长度。 +> +> 子序列定义为：不改变剩余字符顺序的情况下，删除某些字符或者不删除任何字符形成的一个序列。 +> +> ``` +> 输入：s = "bbbab" +> 输出：4 +> 解释：一个可能的最长回文子序列为 "bbbb" +> ``` + +- 状态定义：`dp[i][j]`表示`[i..j]`上的最长回文子序列长度。 + +- 转移方程： + + ```plaintext + if (s[i] == s[j]) { + dp[i][j] = dp[i+1][j-1] + 2; // 两端字符相同，累加长度 + } else { + dp[i][j] = max(dp[i+1][j], dp[i][j-1]); // 取去掉一端后的最大值 + } + ``` + +#### 3.2.2 依赖 O (n) 个子问题 + +`dp[i][j]`需要枚举区间内的分割点`k`（`i ≤ k < j`），通过`dp[i][k]`和`dp[k+1][j]`推导，时间复杂度`O(n³)`。 + +**例题：奇怪的打印机（LeetCode 664）** + +- 状态定义：`dp[i][j]`表示打印`[i..j]`区间字符的最少次数。 + +- 转移方程： + + ```plaintext + // 初始值：单独打印i位置 + dp[i][j] = dp[i+1][j] + 1; + // 若存在k使得s[i] == s[k]，则可与k位置一起打印，减少次数 + for (int k = i+1; k <= j; k++) { + if (s[i] == s[k]) { + dp[i][j] = min(dp[i][j], dp[i+1][k] + dp[k+1][j]); + } + } + ``` + + + +| 类型 | 状态定义 | 推导顺序 | 典型问题 | +| -------- | ---------- | ---------------------- | ---------------------------- | +| 线性单串 | `dp[i]` | 从左到右（i 从小到大） | 最大子数组和、最长递增子序列 | +| 线性双串 | `dp[i][j]` | 按 i 和 j 从小到大 | 最长公共子序列、编辑距离 | +| 区间 DP | `dp[i][j]` | 按区间长度从小到大 | 最长回文子序列、奇怪的打印机 | + +##### [5. 最长回文子串](https://leetcode.cn/problems/longest-palindromic-substring/) + +> 给你一个字符串 `s`，找到 `s` 中最长的 回文子串。 +> +> ``` +> 输入：s = "babad" +> 输出："bab" +> 解释："aba" 同样是符合题意的答案。 +> ``` + +**思路**：**中心扩散法**，「中心扩散法」的基本思想是：遍历每一个下标，以这个下标为中心，利用「回文串」中心对称的特点，往两边扩散，直到不再满足回文的条件。 + +细节：回文串在长度为奇数和偶数的时候，「回文中心」的形态不一样： + +- 奇数回文串的「中心」是一个具体的字符，例如：回文串 "aba" 的中心是字符 "b"； +- 偶数回文串的「中心」是位于中间的两个字符的「空隙」，例如：回文串 "abba" 的中心是两个 "b"，也可以看成两个 "b" 中间的空隙。 + +```java +public String longestPalindrome(String s){ + //处理边界 + if(s == null || s.length() < 2){ + return s; + } + + //初始化start和maxLength变量，用来记录最长回文子串的起始位置和长度 + int start = 0, maxLength = 0; + + //遍历每个字符 + for (int i = 0; i < s.length(); i++) { + //以当前字符为中心的奇数长度回文串 + int len1 = centerExpand(s, i, i); + //以当前字符和下一个字符之间的中心的偶数长度回文串 + int len2 = centerExpand(s, i, i+1); + + int len = Math.max(len1, len2); + + //当前找到的回文串大于之前的记录，更新start和maxLength + if(len > maxLength){ + // i 是当前扩展的中心位置， len 是找到的回文串的总长度，我们要用这两个值计算出起始位置 start + // （len - 1)/2 为什么呢，计算中心到回文串起始位置的距离， 为什么不用 len/2， 这里考虑的是奇数偶数的通用性，比如'abcba' 和 'abba' 或者 'cabbad'，巧妙的同时处理两种，不需要分别考虑 + start = i - (len - 1)/2; + maxLength = len; + } + + } + + return s.substring(start, start + maxLength); +} + +private int centerExpand(String s, int left, int right){ + while(left >= 0 && right < s.length() && s.charAt(left) == s.charAt(right)){ + left --; + right ++; + } + //这个的含义： 假设扩展过程中，left 和 right 已经超出了回文返回， 此时回文范围是 (left+1,right-1), 那么回文长度= (right-1)-(left+1)+1=right-left-1 + return right - left - 1; +} +``` + +![img](https://pic.leetcode-cn.com/2f205fcd0493818129e8d3604b2d84d94678fda7708c0e9831f192e21abb1f34.png) + +##### [221. 最大正方形](https://leetcode.cn/problems/maximal-square/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：二维DP经典** + +> 在一个由 '0' 和 '1' 组成的二维矩阵内，找到只包含 '1' 的最大正方形，并返回其面积。 +> +> ``` +> 输入：matrix = [["1","0","1","0","0"],["1","0","1","1","1"],["1","1","1","1","1"],["1","0","0","1","0"]] +> 输出：4 +> 解释：最大正方形的边长为2，面积为4 +> ``` + +**思路**：二维DP + +- 状态定义：`dp[i][j]` 表示以 `(i,j)` 为右下角的最大正方形的边长 +- 转移方程：如果 `matrix[i][j] == '1'`，则： + ``` + dp[i][j] = Math.min(dp[i-1][j], Math.min(dp[i][j-1], dp[i-1][j-1])) + 1 + ``` +- 边界条件：第一行和第一列直接根据matrix[i][j]设置 + +```java +public int maximalSquare(char[][] matrix) { + if (matrix == null || matrix.length == 0 || matrix[0].length == 0) { + return 0; + } + + int m = matrix.length; + int n = matrix[0].length; + int[][] dp = new int[m][n]; + int maxSide = 0; + + // 初始化第一行和第一列 + for (int i = 0; i < m; i++) { + dp[i][0] = matrix[i][0] - '0'; + maxSide = Math.max(maxSide, dp[i][0]); + } + + for (int j = 0; j < n; j++) { + dp[0][j] = matrix[0][j] - '0'; + maxSide = Math.max(maxSide, dp[0][j]); + } + + // 填充dp数组 + for (int i = 1; i < m; i++) { + for (int j = 1; j < n; j++) { + if (matrix[i][j] == '1') { + dp[i][j] = Math.min(dp[i-1][j], Math.min(dp[i][j-1], dp[i-1][j-1])) + 1; + maxSide = Math.max(maxSide, dp[i][j]); + } + } + } + + return maxSide * maxSide; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(mn) - 遍历整个矩阵一次 +- **空间复杂度**：O(mn) - dp数组，可优化为O(n) + + + + +## 四、分治型动态规划 + +#### **核心特征**： + +1. **问题目标**：生成所有满足条件的组合/排列（而非求最优解或计数）。 +2. 解决方法： + - **分治法**：将问题分解为子问题的组合（如括号生成中的`(inner)outer`）。 + - **回溯法**：通过递归+剪枝暴力搜索所有可能解。 +3. 与经典DP的区别： + - 经典DP通常用状态转移求极值（如最大值、最小值），而生成型DP直接构造解集。 + +**典型例题**： + +| 问题 | 解法 | 关键思路 | +| ------------------------------------------------------------ | ----------- | ------------------------------------------------------ | +| [22. 括号生成](https://leetcode.cn/problems/generate-parentheses/) | 分治DP/回溯 | `dp[i] = ['('+x+')'+y for x in dp[j], y in dp[i-1-j]]` | +| [95. 不同的二叉搜索树 II](https://leetcode.cn/problems/unique-binary-search-trees-ii/) | 分治DP | 遍历根节点，左右子树递归生成 | +| [17. 电话号码的字母组合](https://leetcode.cn/problems/letter-combinations-of-a-phone-number/) | 回溯 | 递归拼接所有可能的字符组合 | + + + +##### [22. 括号生成](https://leetcode.cn/problems/generate-parentheses/) + +> 数字 `n` 代表生成括号的对数，请你设计一个函数，用于能够生成所有可能的并且 **有效的** 括号组合。 +> +> ``` +> 输入：n = 3 +> 输出：["((()))","(()())","(())()","()(())","()()()"] +> ``` +> +> ``` +> 输入：n = 1 +> 输出：["()"] +> ``` + +思路：属于dfs + + + + + +## 🚀 面试前15分钟速记表 + +### 核心模板（必背） + +```java +// 1. 一维DP模板 +int[] dp = new int[n + 1]; +dp[0] = baseCase; // 边界条件 +for (int i = 1; i <= n; i++) { + dp[i] = Math.max(dp[i-1] + nums[i], nums[i]); // 状态转移 +} + +// 2. 二维DP模板 +int[][] dp = new int[m + 1][n + 1]; +// 初始化边界条件 +for (int i = 1; i <= m; i++) { + for (int j = 1; j <= n; j++) { + dp[i][j] = Math.max(dp[i-1][j], dp[i][j-1]); // 状态转移 + } +} + +// 3. 背包问题模板 +int[] dp = new int[capacity + 1]; +for (int i = 0; i < items.length; i++) { + for (int j = capacity; j >= weight[i]; j--) { // 0-1背包逆序 + dp[j] = Math.max(dp[j], dp[j - weight[i]] + value[i]); + } +} +``` + +### 题型速查表 + +| 题型分类 | 核心技巧 | 高频题目 | 记忆口诀 | 难度 | +|----------|----------|----------|----------|------| +| **线性DP-单串类** | 一维状态转移 | 爬楼梯、打家劫舍、最大子数组和、单词拆分、最长递增子序列、乘积最大子数组、买卖股票的最佳时机 | 状态定义含义清，子问题规模要递减 | ⭐⭐⭐ | +| **线性DP-双串类** | 二维状态比较 | 最长公共子序列、编辑距离 | 最后一步怎么走，前面状态来推导 | ⭐⭐⭐⭐ | +| **背包DP类** | 容量限制选择 | 分割等和子集、零钱兑换、零钱兑换II、完全平方数、不同路径 | 物品容量两维度，选或不选做决策 | ⭐⭐⭐⭐ | +| **区间DP类** | 区间分割优化 | 不同的二叉搜索树、最长回文子序列、最长回文子串 | 区间长度小到大，分割点上做文章 | ⭐⭐⭐⭐ | +| **分治型DP类** | 递归分解 | 括号生成 | 大化小来小化了，分而治之最高效 | ⭐⭐⭐⭐ | + +### 核心题目优先级（面试前重点复习） + +#### ⭐⭐⭐ 入门必会 +1. **爬楼梯70** - DP入门，斐波那契数列变体 +2. **打家劫舍198** - 状态机DP简化版，选择问题 +3. **最大子数组和53** - Kadane算法，经典一维DP +4. **买卖股票的最佳时机121** - 最简单的股票DP，贪心思想 +5. **不同路径62** - 二维DP入门，网格路径问题 + +#### ⭐⭐⭐⭐ 核心重点 +1. **零钱兑换322** - 完全背包经典，BFS和DP两种思路 +2. **分割等和子集416** - 0-1背包变体，子集和问题 +3. **最长公共子序列1143** - 双串DP经典，编辑距离前置 +4. **最长递增子序列300** - O(n²)和O(nlogn)两种解法 +5. **不同的二叉搜索树96** - 卡特兰数/区间DP，递归思想 +6. **乘积最大子数组152** - 同时维护最大最小值 + +#### ⭐⭐⭐⭐⭐ 高频困难 +1. **编辑距离72** - 双串DP进阶，三种操作优化 +2. **最长回文子序列516** - 区间DP经典 +3. **零钱兑换II518** - 完全背包计数问题 +4. **单词拆分139** - 字符串DP，记忆化搜索 +5. **最长回文子串5** - 中心扩散法和DP两种解法 +6. **目标和494** - 01背包变体，数学转化 +7. **最大正方形221** - 二维DP经典 + +### 解题思路总结 + +#### 🔑 DP问题识别特征 +- **最优子结构**：大问题的最优解包含子问题的最优解 +- **重叠子问题**：递归过程中存在重复计算 +- **无后效性**：当前状态确定后，未来的决策不依赖过去的状态 + +#### 💡 状态设计原则 +1. **状态含义要明确**：`dp[i]`表示什么要一清二楚 +2. **状态转移要自然**：从小状态推导大状态的逻辑要合理 +3. **边界条件要正确**：base case的设置决定算法正确性 +4. **空间优化要恰当**：能用滚动数组就不用二维数组 + +#### 🎯 常见状态设计模式 +- **以位置为状态**：`dp[i]`表示考虑前i个元素 +- **以值为状态**：`dp[i]`表示值为i时的最优解 +- **以区间为状态**：`dp[i][j]`表示区间[i,j]的最优解 +- **多维状态**：`dp[i][j][k]`表示多个约束条件下的最优解 + +### 常见陷阱提醒 + +- ⚠️ **状态转移方程错误**：仔细分析最后一步的所有可能性 +- ⚠️ **边界条件遗漏**：dp[0]、dp[i][0]等边界要正确初始化 +- ⚠️ **数组越界**：确保索引在有效范围内，特别是i-1、j-1 +- ⚠️ **整数溢出**：结果可能很大时要考虑取模或使用long +- ⚠️ **空间优化错误**：滚动数组时要注意更新顺序 +- ⚠️ **背包问题搞混**：0-1背包逆序，完全背包正序 + +### 时间复杂度速记 + +- **一维DP**：O(n) - 线性遍历 +- **二维DP**：O(mn) - 双重循环 +- **背包问题**：O(nW) - n个物品，W容量 +- **区间DP**：O(n³) - 三重循环，枚举分割点 +- **树形DP**：O(n) - 树的遍历 + +### 面试答题套路 + +1. **分析问题特征** + - 是否有最优子结构？ + - 是否有重叠子问题？ + - 状态空间有多大？ + +2. **设计状态和转移** + - 状态定义要清晰 + - 转移方程要正确 + - 边界条件要完整 + +3. **编码实现** + - 先写暴力递归 + - 再加记忆化搜索 + - 最后改成迭代DP + +4. **优化分析** + - 能否空间优化？ + - 能否时间优化？ + - 特殊情况处理？ + +**最重要的心得**：DP的核心是**状态定义**和**状态转移**，把这两点想清楚，代码就是水到渠成的事情！ + +--- + +**最重要的一个技巧就是，你得行动，写起来** + diff --git a/docs/data-structure-algorithms/soultion/LinkedList-Soultion.md b/docs/data-structure-algorithms/soultion/LinkedList-Soultion.md new file mode 100755 index 0000000000..ee6cab5cde --- /dev/null +++ b/docs/data-structure-algorithms/soultion/LinkedList-Soultion.md @@ -0,0 +1,1977 @@ +--- +title: 链表-热题 +date: 2022-06-08 +tags: + - LikedList + - algorithms +categories: leetcode +--- + +![](https://img.starfish.ink/leetcode/leetcode-banner.png) + +> **导读**：无法高效获取长度，无法根据偏移快速访问元素，是链表的两个劣势。然而面试的时候经常碰见诸如获取倒数第 k 个元素，获取中间位置的元素，判断链表是否存在环，判断环的长度等和长度与位置有关的问题。这些问题都可以通过灵活运用双指针来解决。 +> +> **关键词**：双指针、快慢指针 + +### 🎯 核心考点概览 + +- **双指针技巧**：快慢指针、前后指针 +- **哨兵节点**：简化边界处理 +- **递归思维**：分治与回溯 +- **空间优化**：O(1)空间复杂度 + +### 📝 解题万能模板 + +#### 基础模板 + +```java +// 哨兵节点模板 +ListNode dummy = new ListNode(0); +dummy.next = head; + +// 双指针模板 +ListNode slow = head, fast = head; +while (fast != null && fast.next != null) { + slow = slow.next; + fast = fast.next.next; +} +``` + +#### 边界检查清单 + +- ✅ 空链表：`head == null` +- ✅ 单节点：`head.next == null` +- ✅ 双节点：`head.next.next == null` +- ✅ 头尾节点：使用哨兵节点 + +#### 💡 记忆口诀（朗朗上口版） + +- **快慢指针**："快二慢一找中点，有环必定会相遇" +- **反转链表**："断链之前存next，三指针来回倒腾" +- **合并链表**："小的先走大的等，递归合并最轻松" +- **删除节点**："哨兵开路找前驱，一指断链二连接" +- **双指针**："两头并进各有责，相遇之时是答案" +- **分治算法**："大化小来小化了，分而治之最高效" + +**最重要的一个技巧就是，你得行动，写起来** + + + +### 📋 分类索引 + +1. **🔥 双指针技巧类**：[环形链表](#_环形链表141)、[环形链表II](#_142-环形链表-ii)、[链表中间节点](#_876-链表的中间结点)、[删除倒数第N个](#_19-删除链表的倒数第n个结点)、[旋转链表](#_61-旋转链表) +2. **🔄 反转与重排类**：[反转链表](#_206-反转链表)、[反转链表II](#_92-反转链表-ii)、[K个一组翻转](#_25-k个一组翻转链表)、[重排链表](#_143-重排链表)、[两两交换](#_24-两两交换链表中的节点) +3. **🔗 相交与合并类**：[相交链表](#_160-相交链表)、[合并两个有序链表](#_21-合并两个有序链表)、[合并K个升序链表](#_23-合并k个升序链表) +4. **🗑️ 删除与去重类**：[移除链表元素](#_203-移除链表元素)、[删除重复元素](#_83-删除排序链表中的重复元素)、[删除重复元素II](#_82-删除排序链表中的重复元素-ii) +5. **🧮 数学运算类**：[两数相加](#_2-两数相加)、[两数相加II](#_445-两数相加-ii) +6. **🔍 特殊结构类**：[LRU缓存](#_146-lru缓存)、[复制带随机指针链表](#_138-复制带随机指针的链表)、[回文链表](#_234-回文链表) +7. **🎯 综合应用类**：[排序链表](#_148-排序链表)、[分隔链表](#_86-分隔链表)、[奇偶链表](#_328-奇偶链表)、[分隔链表2](#_725-分隔链表) +8. **🚀 进阶变体类**：[扁平化多级双向链表](#_430-扁平化多级双向链表)、[有序链表转BST](#_109-有序链表转换二叉搜索树)、[合并两个链表](#_1669-合并两个链表) + +--- + +## 一、双指针技巧类（核心中的核心）🔥 + +### 💡 核心思想 + +- **快慢指针**：解决环形、中点、倒数第k个问题 +- **前后指针**：解决删除、窗口问题 +- **双链表指针**：解决相交、合并问题 + +### 🎯 必掌握模板 + +```java +// 快慢指针模板 +ListNode slow = head, fast = head; +while (fast != null && fast.next != null) { + slow = slow.next; + fast = fast.next.next; +} +``` + + + +### [141. 环形链表](https://leetcode.cn/problems/linked-list-cycle/) + +**🎯 考察频率：极高 | 难度：简单 | 重要性：双指针经典** + +> 判断链表是否有环 +> +> ![img](https://assets.leetcode-cn.com/aliyun-lc-upload/uploads/2018/12/07/circularlinkedlist.png) + +**💡 核心思路**：快慢指针（Floyd判圈算法） + +- **快指针**：每次走2步（兔子） +- **慢指针**：每次走1步（乌龟） +- **有环**：快慢指针会相遇 +- **无环**：快指针先到达null + +**🔑 记忆技巧**： + +- **口诀**："快二慢一找中点，有环必定会相遇" +- **形象记忆**：操场跑步，快的总能追上慢的 + +```java +public boolean hasCycle(ListNode head) { + if (head == null || head.next == null) { + return false; + } + // 龟兔起跑 + ListNode fast = head; + ListNode slow = head; + + //while 条件需要注意,如果不含有环，不管是快的还是慢的都会遇到null, + // 如果不含有环的情况用slow！=null 判断的话，fast.next.next 走那么快，没值，不就空指针了 + while (fast != null && fast.next != null) { + // 龟走一步 + slow = slow.next; + // 兔走两步 + fast = fast.next.next; + if (slow == fast) return true; + } + return false; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 最多遍历链表两次 +- **空间复杂度**：O(1) - 只使用了两个指针 + +如果存在环，如何判断环的长度呢？方法是，快慢指针相遇后继续移动，直到第二次相遇。两次相遇间的移动次数即为环的长度。 + + + +### [142. 环形链表 II](https://leetcode-cn.com/problems/linked-list-cycle-ii/) + +> 给定一个链表的头节点 head ，返回链表开始入环的第一个节点。 如果链表无环，则返回 null。 +> +> 如果链表中有某个节点，可以通过连续跟踪 next 指针再次到达，则链表中存在环。 为了表示给定链表中的环，评测系统内部使用整数 pos 来表示链表尾连接到链表中的位置（索引从 0 开始）。如果 pos 是 -1，则在该链表中没有环。注意：pos 不作为参数进行传递，仅仅是为了标识链表的实际情况。 +> +> 不允许修改 链表。 +> +> ``` +> 输入：head = [3,2,0,-4], pos = 1 +> 输出：返回索引为 1 的链表节点 +> 解释：链表中有一个环，其尾部连接到第二个节点。 +> ``` + +**💡 核心思路**：两阶段检测法 + +1. **第一阶段**：快慢指针检测环的存在 +2. **第二阶段**：重置快指针到头节点，同速前进找入口 + +**🔑 记忆技巧**： + +- **口诀**："相遇重置到起点，同步前进找入口" +- **数学原理**：从头到入口的距离 = 从相遇点到入口的距离 + +如下图所示，设链表中环外部分的长度为 a。slow 指针进入环后，又走了 b 的距离与 fast 相遇。此时，fast 指针已经走完了环的 n 圈，因此它走过的总距离为 `a+n(b+c)+b=a+(n+1)b+nc`。 + +![fig1](https://assets.leetcode-cn.com/solution-static/142/142_fig1.png) + +根据题意，任意时刻，fast 指针走过的距离都为 slow 指针的 2 倍。因此，我们有 +`a+(n+1)b+nc=2(a+b)⟹a=c+(n−1)(b+c)` + +有了 a=c+(n-1)(b+c) 的等量关系，我们会发现：从相遇点到入环点的距离加上 n-1 圈的环长，恰好等于从链表头部到入环点的距离。 + +因此，当发现 slow 与 fast 相遇时，我们再额外使用一个指针 ptr。起始，它指向链表头部；随后，它和 slow 每次向后移动一个位置。最终，它们会在入环点相遇。 + +```java +public ListNode detectCycle(ListNode head){ + if (head == null || head.next == null) { + return null; + } + + // 初始化快慢指针 + ListNode slow = head, fast = head; + + // 第一步：检测环的存在 + while (fast != null && fast.next != null) { + slow = slow.next; // 慢指针移动一步 + fast = fast.next.next; // 快指针移动两步 + + // 如果快慢指针相遇，说明链表中存在环 + if (slow == fast) { + // 第二步：找到环的起始节点 + ListNode ptr = head; // 初始化一个指针ptr到链表头部 + while (ptr != slow) { + ptr = ptr.next; // ptr指针每次移动一步 + slow = slow.next; // slow指针也每次移动一步 + } + // 当ptr和slow再次相遇时，即为环的起始节点 + return ptr; + } + } + + // 如果循环结束都没有检测到环，则返回null + return null; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 最多遍历链表两次，第一次检测环，第二次找入口 +- **空间复杂度**：O(1) - 只使用了常数个指针 + + + +### [876. 链表的中间结点](https://leetcode.cn/problems/middle-of-the-linked-list/) + +**🎯 考察频率：高 | 难度：简单 | 重要性：快慢指针基础** + +> 找到链表的中间节点，如果有两个中间节点，返回第二个 + +**💡 核心思路**：快慢指针 + +- 快指针走2步，慢指针走1步 +- 快指针到末尾时，慢指针在中间 + +**🔑 记忆技巧**： + +- **口诀**："快二慢一找中点，快到头时慢在中" +- **形象记忆**：跑步比赛，快的跑完全程，慢的刚好跑到一半 + +```java +public ListNode middleNode(ListNode head) { + ListNode slow = head, fast = head; + while (fast != null && fast.next != null) { + slow = slow.next; + fast = fast.next.next; + } + return slow; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 遍历链表一次 +- **空间复杂度**：O(1) - 只使用了两个指针 + + + +### [19. 删除链表的倒数第N个结点](https://leetcode.cn/problems/remove-nth-node-from-end-of-list/) + +**🎯 考察频率：极高 | 难度：中等 | 重要性：双指针经典** + +> 删除倒数第n个节点：`1->2->3->4->5`, n=2 → `1->2->3->5` + +**💡 核心思路**：前后指针 + +- 前指针先走n+1步 +- 然后前后指针同时走 +- 前指针到末尾时，后指针在待删除节点的前一个 + +**🔑 记忆技巧**： + +- **口诀**："前指先行n+1步，同步到尾删倒数" +- **形象记忆**：两人相距n步走路，前面的人到终点时，后面的人距终点还有n步 + +```java +public ListNode removeNthFromEnd(ListNode head, int n) { + ListNode dummy = new ListNode(0); + ListNode first = dummy, second = dummy; + + // 前指针先走 n+1 步 + for (int i = 0; i <= n; i++) { + first = first.next; + } + + // 同时移动 first 和 second 指针，直到 first 到达链表末尾 + while (first != null) { + first = first.next; + second = second.next; + } + + // 删除倒数第 N 个节点 + second.next = second.next.next; + return dummy.next; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 只需要遍历链表一次 +- **空间复杂度**：O(1) - 只使用了常数个指针 + + + +### [61. 旋转链表](https://leetcode.cn/problems/rotate-list/) + +**🎯 考察频率：中等 | 难度：中等 | 重要性：双指针应用** + +> 给你一个链表的头节点 `head` ，旋转链表，将链表每个节点向右移动 `k` 个位置。 +> +> `1->2->3->4->5`, k=2 → `4->5->1->2->3` + +**💡 核心思路**： + +1. 先成环：尾节点连接头节点 +2. 找断点：倒数第k个位置断开 +3. 重新设置头尾 + +```java +public ListNode rotateRight(ListNode head, int k) { + if (head == null || head.next == null || k == 0) return head; + + // 计算长度并成环 + ListNode tail = head; + int len = 1; + while (tail.next != null) { + tail = tail.next; + len++; + } + tail.next = head; // 成环 + + // 找新的尾节点（倒数第k+1个） + k = k % len; // 处理k大于链表长度的情况 + int stepsToNewTail = len - k; + + ListNode newTail = head; + for (int i = 1; i < stepsToNewTail; i++) { + newTail = newTail.next; + } + + ListNode newHead = newTail.next; + newTail.next = null; // 断环 + + return newHead; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 需要遍历链表两次，一次计算长度，一次找断点 +- **空间复杂度**：O(1) - 只使用了常数个指针 + +--- + + + +## 二、反转与重排类（高频考点）🔄 + +### 💡 核心思想 + +- **三指针反转**：prev、curr、next三指针配合 +- **递归反转**：分治思想，先处理子问题 +- **局部反转**：找到边界，局部应用反转技巧 + +### 🎯 必掌握模板 + +```java +// 反转链表核心模板 +ListNode prev = null, curr = head; +while (curr != null) { + ListNode next = curr.next; + curr.next = prev; + prev = curr; + curr = next; +} +return prev; +``` + +### [206. 反转链表](https://leetcode.cn/problems/reverse-linked-list/) + +**🎯 考察频率：极高 | 难度：简单 | 重要性：基础必会** + +> 反转单链表：`1->2->3->4->5` 变成 `5->4->3->2->1` + +**💡 核心思路**：三指针法（prev、curr、next） + +- 记住口诀："**断链前，先保存next**" + +- 像翻书一样，一页一页往前翻 + + ![迭代.gif](https://pic.leetcode-cn.com/7d8712af4fbb870537607b1dd95d66c248eb178db4319919c32d9304ee85b602-%E8%BF%AD%E4%BB%A3.gif) + +```java +// 迭代版本 +public ListNode reverseList(ListNode head){ + if(head == null || head.next == null){ + return head; + } + + ListNode prev = null; + ListNode curr = head; + + while(curr != null) { + ListNode next = curr.next; // 保存下一个节点 + curr.next = prev; // 反转当前节点 + prev = curr; // prev前进 + curr = next; // curr前进 + } + return prev; +} + +// 递归版本 +public ListNode reverseList(ListNode head) { + if (head == null || head.next == null) { + return head; + } + ListNode newHead = reverseList(head.next); + head.next.next = head; + head.next = null; + return newHead; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 遍历链表中的每个节点一次 +- **空间复杂度**：迭代版O(1)，递归版O(n) - 递归调用栈的深度 + + + +### [92. 反转链表 II](https://leetcode.cn/problems/reverse-linked-list-ii/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：局部反转经典** + +> 反转第left到right个节点：`1->2->3->4->5`, left=2, right=4 → `1->4->3->2->5` + +**💡 核心思路**： + +1. 找到反转区间的前一个节点 +2. 对区间内节点进行反转 +3. 重新连接前后部分 + +> 思路二：头插法。创建一个新的虚拟节点，然后将需要反转的部分逐个插入到新链表的前端，实现反转 + +```java +public ListNode reverseBetween(ListNode head, int left, int right) { + // 如果链表为空或left等于right，则无需反转，直接返回原链表头节点 + if (head == null || left == right) { + return head; + } + + // 创建虚拟头节点，其next指向原链表头节点 + ListNode dummy = new ListNode(0); + dummy.next = head; + + // 定位到left位置的前一个节点 + ListNode pre = dummy; + for (int i = 1; i < left; i++) { + pre = pre.next; + } + + // start指向需要反转的链表部分的起始节点 + ListNode start = pre.next; + + // 初始化反转链表部分所需的指针 + ListNode prev = null; + ListNode current = start; + + //使用right - left + 1来计算k + int k = right - left + 1; + + // 反转k个节点 + for (int i = 0; i < k; i++) { + ListNode next = current.next; // 保存current的下一个节点 + current.next = prev; // 将current的next指向prev，实现反转 + // 移动prev和current指针 + prev = current; + current = next; + } + + // 连接反转后的链表部分与原链表剩余部分 + // pre.next指向反转后的链表头节点（即原链表的第right个节点） + pre.next = prev; + // start.next指向反转后的链表尾节点的下一个节点（即原链表的第right+1个节点） + start.next = current; + + // 返回反转后的链表头节点（虚拟头节点的下一个节点） + return dummy.next; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 遍历链表一次 +- **空间复杂度**：O(1) - 只使用了常数个指针 + + + +### [25. K个一组翻转链表](https://leetcode.cn/problems/reverse-nodes-in-k-group/) + +**🎯 考察频率：高 | 难度：困难 | 重要性：分组反转** + +> 每k个节点一组反转：`1->2->3->4->5`, k=3 → `3->2->1->4->5` + +**💡 核心思路**： + +1. 检查是否有k个节点可反转 +2. 反转这k个节点 +3. 递归处理剩余部分 + +```java +public ListNode reverseKGroup(ListNode head, int k) { + // 检查是否有k个节点 + ListNode curr = head; + int count = 0; + while (curr != null && count < k) { + curr = curr.next; + count++; + } + + if (count == k) { + // 反转前k个节点 + curr = reverseKGroup(curr, k); // 递归处理后续 + + // 反转当前k个节点 + while (count-- > 0) { + ListNode tmp = head.next; + head.next = curr; + curr = head; + head = tmp; + } + head = curr; + } + + return head; +} +``` + + + +### [143. 重排链表](https://leetcode.cn/problems/reorder-list/) + +**🎯 考察频率：中等 | 难度：中等 | 重要性：综合应用** + +> 给定一个单链表 `L` 的头节点 `head` ，单链表 `L` 表示为： +> +> ``` +> L0 → L1 → … → Ln - 1 → Ln +> ``` +> +> 请将其重新排列后变为： +> +> ``` +> L0 → Ln → L1 → Ln - 1 → L2 → Ln - 2 → … +> ``` +> +> 不能只是单纯的改变节点内部的值，而是需要实际的进行节点交换。 +> +> 重新排列：`1->2->3->4->5` → `1->5->2->4->3` + +**💡 核心思路**： + +1. 找中点切分链表 +2. 反转后半部分 +3. 交替合并两部分 + +```java +public void reorderList(ListNode head) { + if (head == null || head.next == null) return; + + // 1. 找中点 + ListNode slow = head, fast = head; + while (fast.next != null && fast.next.next != null) { + slow = slow.next; + fast = fast.next.next; + } + + // 2. 切分并反转后半部分 + ListNode secondHalf = reverseList(slow.next); + slow.next = null; + + // 3. 交替合并 + ListNode first = head; + while (secondHalf != null) { + ListNode temp1 = first.next; + ListNode temp2 = secondHalf.next; + + first.next = secondHalf; + secondHalf.next = temp1; + + first = temp1; + secondHalf = temp2; + } +} + +private ListNode reverseList(ListNode head) { + ListNode prev = null; + while (head != null) { + ListNode next = head.next; + head.next = prev; + prev = head; + head = next; + } + return prev; +} +``` + +### [24. 两两交换链表中的节点](https://leetcode.cn/problems/swap-nodes-in-pairs/) + +**🎯 考察频率：中等 | 难度：中等 | 重要性：指针操作** + +> 两两交换相邻节点：`1->2->3->4` → `2->1->4->3` + +**💡 核心思路**：三指针操作 + +- 使用虚拟头节点简化边界处理 +- 每次处理一对节点的交换 + +**🔑 记忆技巧**： + +- **口诀**："两两交换用三指，前中后来做舞蹈" + +```java +public ListNode swapPairs(ListNode head) { + ListNode dummy = new ListNode(0); + dummy.next = head; + ListNode prev = dummy; + + while (prev.next != null && prev.next.next != null) { + ListNode first = prev.next; + ListNode second = prev.next.next; + + // 交换节点 + first.next = second.next; + second.next = first; + prev.next = second; + + // 移动指针 + prev = first; + } + + return dummy.next; +} +``` + +--- + + + +## 三、相交与合并类🔗 + +### 💡 核心思想 + +- **双指针遍历**：两个链表同时遍历，路径互换 +- **归并思想**：有序链表的合并策略 +- **分治算法**：多个链表的合并优化 + +### [160. 相交链表](https://leetcode.cn/problems/intersection-of-two-linked-lists/) + +**🎯 考察频率：高 | 难度：简单 | 重要性：双指针经典** + +> 找两个链表的相交节点 +> +> ![](https://assets.leetcode-cn.com/aliyun-lc-upload/uploads/2018/12/14/160_example_1.png) +> +> ``` +> 输入：intersectVal = 8, listA = [4,1,8,4,5], listB = [5,6,1,8,4,5], skipA = 2, skipB = 3 +> 输出：Intersected at '8' +> 解释：相交节点的值为 8 （注意，如果两个链表相交则不能为 0）。 +> 从各自的表头开始算起，链表 A 为 [4,1,8,4,5]，链表 B 为 [5,6,1,8,4,5]。 +> 在 A 中，相交节点前有 2 个节点；在 B 中，相交节点前有 3 个节点。 +> ``` + +**💡 核心思路**：路径互换双指针法 + +- 两个指针分别从两个链表头开始遍历 +- 到达尾部时，跳转到另一个链表的头部 +- 相遇点就是交点（如果有的话） + +**🔑 记忆技巧**： + +- **口诀**："你走我的路，我走你的路，相遇便是交点处" +- **诗意版本**："走到尽头见不到你，于是走过你来时的路" +- **数学原理**：两指针走过的总路径长度相等 + +```java +public ListNode getIntersectionNode(ListNode headA, ListNode headB) { + if (headA == null || headB == null) { + return null; + } + ListNode pA = headA, pB = headB; + while (pA != pB) { + //这里注意 如果pA 到了尾结点后要转向headB，而不是 pB + pA = pA == null ? headB : pA.next; + pB = pB == null ? headA : pB.next; + } + //如果没有相交，这里会返回 null + return pA; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(m+n) - 最多遍历两个链表各一次 +- **空间复杂度**：O(1) - 只使用了两个指针 + + + +### [21. 合并两个有序链表](https://leetcode.cn/problems/merge-two-sorted-lists/) + +**🎯 考察频率：极高 | 难度：简单 | 重要性：合并算法基础** + +> 合并两个升序链表：`1->2->4` + `1->3->4` = `1->1->2->3->4->4` + +**💡 核心思路**：比较头节点，选小的 + +- 递归版本：选小的节点，剩下的交给递归处理 + + ![](https://pic.leetcode-cn.com/fe5eca7edea29a76316f7e8529f73a90ae4990fd66fea093c6ee91567788e482-%E5%B9%BB%E7%81%AF%E7%89%874.JPG) + +- 迭代版本：用哨兵节点，逐个比较拼接 + +**🔑 记忆技巧**： + +- **口诀**："小的先走大的等，递归合并最轻松" +- **形象记忆**：两队人排队，总是让矮的人先走 + +```java +// 递归版本 +public ListNode mergeTwoLists(ListNode l1, ListNode l2) { + if (l1 == null) return l2; + if (l2 == null) return l1; + + if (l1.val < l2.val) { + l1.next = mergeTwoLists(l1.next, l2); + return l1; + } else { + l2.next = mergeTwoLists(l1, l2.next); + return l2; + } +} + +// 迭代版本 +public ListNode mergeTwoLists(ListNode l1, ListNode l2){ + ListNode dummy = new ListNode(0); // 创建哨兵节点 + ListNode cur = dummy; // 当前指针，指向哨兵节点 + while(l1 != null && l2 != null){ + if(l1.val < l2.val){ + cur.next = l1; // 将较小节点接到当前指针后面 + l1 = l1.next; // 移动l1指针 + }else{ + cur.next = l2; + l2 = l2.next; + } + cur = cur.next; // 移动当前指针 + } + cur.next = l1 != null ? l1 : l2; // 拼接剩余部分 + return dummy.next; // 返回合并后的链表头节点 + +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：递归版O(m+n)，迭代版O(m+n) - m和n分别为两个链表的长度 +- **空间复杂度**：递归版O(m+n)，迭代版O(1) - 递归版需要栈空间 + + + +### [23. 合并K个升序链表](https://leetcode.cn/problems/merge-k-sorted-lists/) + +**🎯 考察频率：极高 | 难度：困难 | 重要性：分治算法经典** + +> 合并K个有序链表：`[[1,4,5],[1,3,4],[2,6]]` → `[1,1,2,3,4,4,5,6]` + +**💡 核心思路**：优先队列（最小堆） + +- 将所有链表的头节点放入最小堆 +- 每次取出最小值，将其next节点放入堆 +- 重复直到堆为空 + +**🔑 记忆技巧**： + +- **口诀**："K链合并用小堆，最小优先逐个取" +- **形象记忆**：多路归并，就像多条河流汇入大海 + +```java +public ListNode mergeKLists(ListNode[] lists) { + if (lists == null || lists.length == 0) return null; + + PriorityQueue pq = new PriorityQueue<>((a, b) -> a.val - b.val); + + // 将所有链表的头节点加入堆 + for (ListNode head : lists) { + if (head != null) { + pq.offer(head); + } + } + + ListNode dummy = new ListNode(0); + ListNode curr = dummy; + + //当前最小堆不为空，继续合并 + while (!pq.isEmpty()) { + ListNode node = pq.poll(); //从最小堆中取出最小节点 + curr.next = node; //将取出的链表加到合并后的链表中 + curr = curr.next; //移动指针 + + //如果取出的节点有下一个节点，将其加入到最小堆中 + if (node.next != null) { + pq.offer(node.next); + } + } + + return dummy.next; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(N×log k) - N是所有节点总数，k是链表个数，堆操作log k +- **空间复杂度**：O(k) - 优先队列最多存储k个节点 + +--- + + + +## 四、删除与去重类 🗑️ + +### 💡 核心思想 + +- **哨兵节点**：简化头节点的删除操作 +- **双指针**：一个用于遍历，一个用于连接 +- **递归删除**：自然的递归结构 + +### 🎯 必掌握模板 + +```java +// 删除节点模板 +ListNode dummy = new ListNode(0); +dummy.next = head; +ListNode prev = dummy, curr = head; + +while (curr != null) { + if (shouldDelete(curr)) { + prev.next = curr.next; // 删除curr + } else { + prev = curr; // prev前进 + } + curr = curr.next; // curr前进 +} +return dummy.next; +``` + +### [203. 移除链表元素](https://leetcode.cn/problems/remove-linked-list-elements/) + +**🎯 考察频率：中等 | 难度：简单 | 重要性：删除基础** + +> 删除所有值等于val的节点：`1->2->6->3->4->5->6`, val=6 → `1->2->3->4->5` + +**🔑 记忆技巧**： + +- **口诀**："哨兵开路双指针，遇到目标就跳过" + +```java +public ListNode removeElements(ListNode head, int val) { + // 创建一个哑节点作为链表的前驱，避免处理头节点的特殊情况 + ListNode dummy = new ListNode(0); + dummy.next = head; + + // 初始化两个指针，pre指向哑节点，cur指向头节点 + ListNode pre = dummy; + ListNode cur = head; + + // 遍历链表 + while (cur != null) { + // 如果当前节点的值等于val，则删除该节点 + if (cur.val == val) { + pre.next = cur.next; // 跳过当前节点，连接pre和cur的下一个节点 + } else { + // 如果当前节点的值不等于val，则移动pre指针到当前节点 + pre = cur; + } + // 移动cur指针到下一个节点 + cur = cur.next; + } + + // 返回更新后的头节点（可能是哑节点的下一个节点） + return dummy.next; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 遍历链表一次 +- **空间复杂度**：O(1) - 只使用了常数个指针 + + + +### [83. 删除排序链表中的重复元素](https://leetcode.cn/problems/remove-duplicates-from-sorted-list/) + +**🎯 考察频率：中等 | 难度：简单 | 重要性：去重基础** + +> **问题1：保留一个重复元素** +> 输入：`1->1->2->3->3` → 输出：`1->2->3` + +**💡 核心思路**：单指针遍历法 + +- 遍历链表，比较当前节点与下一个节点的值 +- 如果相等，跳过下一个节点（`curr.next = curr.next.next`） +- 如果不等，移动到下一个节点 + +**🔑 记忆技巧**： + +- **口诀**："相邻比较去重复，保留一个删后续" +- **关键点**：只有当值不相等时，curr指针才前进 + +```java +public ListNode deleteDuplicates(ListNode head) { + if (head == null) return head; + + ListNode curr = head; + while (curr != null && curr.next != null) { + if (curr.val == curr.next.val) { + // 跳过重复节点，curr不前进 + curr.next = curr.next.next; + } else { + // 值不相等，curr前进 + curr = curr.next; + } + } + return head; +} +``` + +**🔍 详细步骤示例**： + +``` +原链表：1->1->2->3->3->null +步骤1：curr=1, next=1 (相等) → 跳过 → 1->2->3->3->null +步骤2：curr=1, next=2 (不等) → curr前进 → curr=2 +步骤3：curr=2, next=3 (不等) → curr前进 → curr=3 +步骤4：curr=3, next=3 (相等) → 跳过 → 1->2->3->null +结果：1->2->3->null +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 遍历链表一次 +- **空间复杂度**：O(1) - 只使用了常数个指针 + +### [82. 删除排序链表中的重复元素 II](https://leetcode.cn/problems/remove-duplicates-from-sorted-list-ii/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：完全去重** + +> **问题2：删除所有重复元素** +> 输入：`1->2->3->3->4->4->5` → 输出：`1->2->5` + +**💡 核心思路**：虚拟头节点 + 双指针法 + +- 使用虚拟头节点处理头节点可能被删除的情况 +- prev指针：指向最后一个确定保留的节点 +- curr指针：用于遍历和检测重复 + +**🔑 记忆技巧**： + +- **口诀**："发现重复全跳过，哨兵记住前驱位" +- **关键点**：只有当前节点不重复时，prev指针才前进 + +```java +public ListNode deleteDuplicates(ListNode head) { + // 处理空链表情况 + if(head == null){ + return null; + } + + ListNode dummy = new ListNode(0); + dummy.next = head; + + // pre指向当前确定不重复的最后一个节点，cur用于遍历 + ListNode pre = dummy, cur = head; + + while(cur != null && cur.next != null){ + // 发现重复节点 + if(cur.val == cur.next.val){ + // 跳过所有重复节点（至少有两个相同值节点） + while(cur.next != null && cur.val == cur.next.val){ + cur = cur.next; + } + // 删除重复节点（pre.next直接指向重复节点后的第一个不同节点） + pre.next = cur.next; + // 移动cur到下一个待检查节点 + cur = cur.next; + }else{ + // 当前节点不重复，正常移动pre和cur指针 + pre = cur; + cur = cur.next; + } + } + return dummy.next; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 遍历链表一次 +- **空间复杂度**：O(1) - 只使用了常数个指针 + +**⚡ 两题对比总结**： + +| 题目 | 处理方式 | 关键区别 | 返回值 | +| ---- | ------------ | ------------------------ | -------------- | +| LC83 | 保留一个重复 | curr指针控制，不需虚拟头 | 直接返回head | +| LC82 | 删除所有重复 | prev指针控制，需要虚拟头 | 返回dummy.next | + +**🎯 面试常考变体**： + +1. **无序链表去重**：需要先排序或使用HashSet +2. **保留最后一个重复元素**：类似LC83，但从后往前处理 +3. **统计重复元素个数**：在删除过程中计数 +4. **删除指定值的所有节点**：类似LC203 + +**💡 解题心得**： + +- **LC83核心**：遇到重复就跳过，curr只在不重复时前进 +- **LC82核心**：遇到重复就全删，prev只在确认安全时前进 +- **记忆方法**：83保留(Keep one)，82全删(Remove all) +- **调试技巧**：画图模拟指针移动过程，特别注意边界情况 + +--- + + + +## 五、数学运算类 🧮 + +### 💡 核心思想 + +- **进位处理**：模拟手工计算的进位过程 +- **链表表示数字**：低位在前，高位在后 +- **边界处理**：不同长度链表的处理 + +### [2. 两数相加](https://leetcode-cn.com/problems/add-two-numbers/) + +**🎯 考察频率：极高 | 难度：中等 | 重要性：数学计算经典** + +> 给出两个 非空 的链表用来表示两个非负的整数。其中，它们各自的位数是按照 逆序 的方式存储的，并且它们的每个节点只能存储 一位 数字。 +> +> 如果，我们将这两个数相加起来，则会返回一个新的链表来表示它们的和。 +> +> 您可以假设除了数字 0 之外，这两个数都不会以 0 开头。 +> +> ``` +> 输入：(2 -> 4 -> 3) + (5 -> 6 -> 4) +> 输出：7 -> 0 -> 8 +> 原因：342 + 465 = 807 +> ``` + +**💡 核心思路**：模拟手算加法 + +- 同时遍历两个链表，逐位相加 +- 处理进位carry，满10进1 +- 注意最后可能还有进位 + +**🔑 记忆技巧**： + +- **口诀**："逐位相加记进位，链表模拟手算法" +- **形象记忆**：小学数学竖式加法，从右到左逐位计算 + + +```java +public ListNode addTwoNumbers(ListNode l1, ListNode l2) { + // 创建一个虚拟头节点，用于简化头节点可能发生变化的情况 + ListNode dummy = new ListNode(0); + ListNode current = dummy; + + // 初始化进位为0 + int carry = 0; + + // 当两个链表都不为空或者存在进位时，继续循环 + while (l1 != null || l2 != null || carry > 0) { + // 获取两个链表当前节点的值，如果链表为空则视为0 + int x = (l1 != null) ? l1.val : 0; + int y = (l2 != null) ? l2.val : 0; + + // 计算当前位的和以及新的进位 + int sum = carry + x + y; + carry = sum / 10; + + // 创建新节点存储当前位的值 + current.next = new ListNode(sum % 10); + current = current.next; + + // 移动到两个链表的下一个节点 + if (l1 != null) l1 = l1.next; + if (l2 != null) l2 = l2.next; + } + + // 返回虚拟头节点的下一个节点，即实际结果的头节点 + return dummy.next; +} +``` + + + +### [445. 两数相加 II](https://leetcode.cn/problems/add-two-numbers-ii/) + +**🎯 考察频率：中等 | 难度：中等 | 重要性：逆序处理** + +> 高位在前的链表相加：`(7->2->4->3) + (5->6->4)` 表示 `7243 + 564 = 7807` → `7->8->0->7` + +**💡 核心思路**： + +1. 使用栈存储数字 +2. 从栈顶开始计算（相当于从低位开始） +3. 头插法构建结果链表 + +```java +public ListNode addTwoNumbers(ListNode l1, ListNode l2) { + Stack stack1 = new Stack<>(); + Stack stack2 = new Stack<>(); + + // 将链表值压入栈中 + while (l1 != null) { + stack1.push(l1.val); + l1 = l1.next; + } + while (l2 != null) { + stack2.push(l2.val); + l2 = l2.next; + } + + ListNode result = null; + int carry = 0; + + while (!stack1.isEmpty() || !stack2.isEmpty() || carry > 0) { + int x = stack1.isEmpty() ? 0 : stack1.pop(); + int y = stack2.isEmpty() ? 0 : stack2.pop(); + + int sum = x + y + carry; + carry = sum / 10; + + // 头插法 + ListNode newNode = new ListNode(sum % 10); + newNode.next = result; + result = newNode; + } + + return result; +} +``` + +--- + + + +## 六、特殊结构类 🔍 + +### 💡 核心思想 + +- **哈希表优化**：O(1)时间访问 +- **双向链表**：支持前后遍历 +- **设计数据结构**：LRU、LFU等缓存机制 + +### [146. LRU缓存](https://leetcode.cn/problems/lru-cache/) + +**🎯 考察频率：极高 | 难度：中等 | 重要性：系统设计经典** + +> 实现LRU（最近最少使用）缓存机制 + +**💡 核心思路**：哈希表 + 双向链表 + +- 哈希表：O(1)查找节点 +- 双向链表：O(1)插入删除，维护访问顺序 +- 最新访问的放头部，最久未用的在尾部 + +**🔑 记忆技巧**： + +- **口诀**："哈希定位双链调，头部最新尾最老" +- **形象记忆**：图书管理员，用卡片索引快速找书，用链表记录借阅顺序 + +```java +import java.util.HashMap; +import java.util.Map; + +class LRUCache { + private final int capacity; + private final Map cache; + private final int size; + private final DLinkedNode head, tail; + + // 使用伪头部和伪尾部节点 + class DLinkedNode { + int key; + int value; + DLinkedNode prev; + DLinkedNode next; + + DLinkedNode() {} + + DLinkedNode(int key, int value) { this.key = key; this.value = value; } + } + + public LRUCache(int capacity) { + this.capacity = capacity; + this.size = 0; + // 使用伪头部和伪尾部节点 + head = new DLinkedNode(); + tail = new DLinkedNode(); + head.next = tail; + tail.prev = head; + cache = new HashMap<>(capacity); + } + + public int get(int key) { + DLinkedNode node = cache.get(key); + if (node == null) { + return -1; + } + // 移动到头部 + moveToHead(node); + return node.value; + } + + public void put(int key, int value) { + DLinkedNode node = cache.get(key); + if (node == null) { + // 如果 key 不存在，创建一个新的节点 + DLinkedNode newNode = new DLinkedNode(key, value); + // 添加进哈希表 + cache.put(key, newNode); + // 添加至双向链表的头部 + addToHead(newNode); + ++size; + if (size > capacity) { + // 如果超出容量，删除双向链表的尾部节点 + DLinkedNode tail = removeTail(); + // 删除哈希表中对应的项 + cache.remove(tail.key); + --size; + } + } else { + // 如果 key 存在，先通过哈希表定位，再修改 value，并移动到头部 + node.value = value; + moveToHead(node); + } + } + + private void addToHead(DLinkedNode node) { + node.prev = head; + node.next = head.next; + head.next.prev = node; + head.next = node; + } + + private void removeNode(DLinkedNode node) { + node.prev.next = node.next; + node.next.prev = node.prev; + } + + private void moveToHead(DLinkedNode node) { + removeNode(node); + addToHead(node); + } + + private DLinkedNode removeTail() { + DLinkedNode res = tail.prev; + removeNode(res); + return res; + } +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：get和put操作都是O(1) - 哈希表查找O(1)，双向链表插入删除O(1) +- **空间复杂度**：O(capacity) - 哈希表和双向链表存储最多capacity个节点 + +### [138. 复制带随机指针的链表](https://leetcode.cn/problems/copy-list-with-random-pointer/) + +**🎯 考察频率：极高 | 难度：中等 | 重要性：特殊结构经典** + +> 深拷贝带有random指针的链表，每个节点有next和random两个指针 + +**💡 核心思路**： + +- **方法一**：哈希表映射（空间O(n)） +- **方法二**：原地复制法（空间O(1)） + +**🔑 记忆技巧**： + +- **口诀**："哈希建表存映射，原地复制巧连接" +- **形象记忆**：复印文件，先建立对照表，再复制内容 + +```java +// 方法一：哈希表法（推荐，思路清晰） +public Node copyRandomList(Node head) { + if (head == null) return null; + + Map map = new HashMap<>(); + Node curr = head; + + // 第一遍：创建所有新节点，建立映射关系 + while (curr != null) { + map.put(curr, new Node(curr.val)); + curr = curr.next; + } + + // 第二遍：设置next和random指针 + curr = head; + while (curr != null) { + Node newNode = map.get(curr); + newNode.next = map.get(curr.next); + newNode.random = map.get(curr.random); + curr = curr.next; + } + + return map.get(head); +} + +// 方法二：原地复制法（空间优化） +public Node copyRandomList(Node head) { + if (head == null) return null; + + // 第一步：复制节点，A->A'->B->B'->C->C' + Node curr = head; + while (curr != null) { + Node copy = new Node(curr.val); + copy.next = curr.next; + curr.next = copy; + curr = copy.next; + } + + // 第二步：设置random指针 + curr = head; + while (curr != null) { + if (curr.random != null) { + curr.next.random = curr.random.next; + } + curr = curr.next.next; + } + + // 第三步：分离两个链表 + Node dummy = new Node(0); + Node copyPrev = dummy; + curr = head; + + while (curr != null) { + Node copy = curr.next; + curr.next = copy.next; + copyPrev.next = copy; + copyPrev = copy; + curr = curr.next; + } + + return dummy.next; +} +``` + +**🔍 解法对比**： + +| 方法 | 时间复杂度 | 空间复杂度 | 优缺点 | +| ---------- | ---------- | ---------- | ------------------ | +| 哈希表法 | O(n) | O(n) | 思路清晰，易理解 | +| 原地复制法 | O(n) | O(1) | 空间优化，但较复杂 | + + + +### [234. 回文链表](https://leetcode.cn/problems/palindrome-linked-list/) + +**🎯 考察频率：高 | 难度：简单 | 重要性：综合应用** + +> 判断链表是否为回文：`1->2->2->1` → true + +**💡 核心思路**：三步走策略 + +1. 快慢指针找中点，切分链表 +2. 反转后半部分链表 +3. 双指针同时比较前后两部分 + +> 使用栈：遍历链表，将节点值依次压入栈中，然后再遍历链表，将节点值与栈顶元素进行比较，如果都相等，则是回文链表。 +> +> 使用额外数组：遍历链表，将节点值存入一个数组中，然后双指针检查数组是否为回文数组。 + +**🔑 记忆技巧**： + +- **口诀**："中点切分反转后，双指同比判回文" +- **形象记忆**：对折纸条，看两面是否完全重合 + +```java +public boolean isPalindrome(ListNode head) { + if (head == null || head.next == null) { + return true; // 空链表或单个节点是回文 + } + + List list = new ArrayList<>(); + while (head != null) { + list.add(head.val); + head = head.next; + } + + int left = 0; + int right = list.size() - 1; + + while (left < right) { + if (!list.get(left).equals(list.get(right))) { + return false; + } + left++; + right--; + } + + return true; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 遍历链表两次，一次存储值，一次比较 +- **空间复杂度**：O(n) - 使用额外数组存储所有节点值 + +--- + + + +## 七、综合应用类 🎯 + +### 💡 核心思想 + +- **分治递归**：大问题分解成小问题 +- **多技巧结合**：快慢指针+反转+合并等 +- **优化策略**：时间空间复杂度的权衡 + +### [148. 排序链表](https://leetcode.cn/problems/sort-list/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：归并排序应用** + +> 在O(n log n)时间和常数空间内排序链表：`4->2->1->3` → `1->2->3->4` + +**💡 核心思路**：归并排序 + +1. 找中点切分链表（快慢指针） +2. 递归排序左右两部分 +3. 合并两个有序链表 + +**🔑 记忆技巧**： + +- **口诀**："找中点，递归排，合并完" +- **形象记忆**：分治策略，先分后治，各个击破 + +```java +public ListNode sortList(ListNode head) { + if (head == null || head.next == null) { + return head; // 空链表或只有一个节点的链表已经是排序好的 + } + + // 使用快慢指针找到链表的中间节点 + ListNode slow = head; + ListNode fast = head; + while (fast != null && fast.next != null) { + slow = slow.next; + fast = fast.next.next; + } + + ListNode mid = slow.next; // mid 是中间节点 + slow.next = null; // 将链表分成两半 + + // 递归或迭代地对两半链表进行排序 + ListNode left = sortList(head); // 排序左半部分 + ListNode right = sortList(mid); // 排序右半部分 + + // 合并两个已排序的链表 + ListNode sortedList = mergeTwoLists(left, right); + return sortedList; +} + +// 合并两个已排序的链表 +private ListNode mergeTwoLists(ListNode l1, ListNode l2) { + ListNode dummy = new ListNode(0); // 创建一个虚拟头结点 + ListNode tail = dummy; // tail 用于追踪合并后的链表的尾部 + + while (l1 != null && l2 != null) { + if (l1.val < l2.val) { + tail.next = l1; + l1 = l1.next; + } else { + tail.next = l2; + l2 = l2.next; + } + tail = tail.next; + } + + // 如果其中一个链表已经遍历完，直接将另一个链表的剩余部分接到合并后的链表后面 + if (l1 != null) { + tail.next = l1; + } else { + tail.next = l2; + } + + return dummy.next; // 返回合并后的链表的头结点（跳过虚拟头结点） +} +``` + +### [86. 分隔链表](https://leetcode.cn/problems/partition-list/) + +**🎯 考察频率：中等 | 难度：中等 | 重要性：双链表技巧** + +> 按值x分隔链表：`1->4->3->2->5->2`, x=3 → `1->2->2->4->3->5` + +**💡 核心思路**：双链表分离 + +1. 创建两个虚拟头节点 +2. 遍历原链表，按条件分配到两个链表 +3. 连接两个链表 + +**🔑 记忆技巧**： + +- **口诀**："双链分离按条件，小大分开再拼接" +- **形象记忆**：分拣包裹，小的一堆，大的一堆，最后拼接 + +```java +public ListNode partition(ListNode head, int x) { + ListNode beforeHead = new ListNode(0); + ListNode before = beforeHead; + ListNode afterHead = new ListNode(0); + ListNode after = afterHead; + + while (head != null) { + if (head.val < x) { + before.next = head; + before = before.next; + } else { + after.next = head; + after = after.next; + } + head = head.next; + } + + after.next = null; // 避免环 + before.next = afterHead.next; + + return beforeHead.next; +} +``` + +### [328. 奇偶链表](https://leetcode.cn/problems/odd-even-linked-list/) + +**🎯 考察频率：中等 | 难度：中等 | 重要性：指针操作** + +> 奇偶位置重排：`1->2->3->4->5` → `1->3->5->2->4` + +**💡 核心思路**： + +1. 分离奇偶位置节点 +2. 连接奇数链表和偶数链表 + +```java +public ListNode oddEvenList(ListNode head) { + if (head == null || head.next == null) return head; + + ListNode odd = head; + ListNode even = head.next; + ListNode evenHead = even; + + while (even != null && even.next != null) { + odd.next = even.next; + odd = odd.next; + even.next = odd.next; + even = even.next; + } + + odd.next = evenHead; + return head; +} +``` + +### [725. 分隔链表](https://leetcode.cn/problems/split-linked-list-in-parts/) + +**🎯 考察频率：低 | 难度：中等 | 重要性：数学计算** + +> 将链表分隔成k部分，尽可能平均 + +**💡 核心思路**： + +1. 计算链表长度 +2. 计算每部分的长度 +3. 按计算结果切分 + +```java +public ListNode[] splitListToParts(ListNode root, int k) { + // 计算长度 + int len = 0; + ListNode curr = root; + while (curr != null) { + len++; + curr = curr.next; + } + + int partSize = len / k; // 每部分基本长度 + int remainder = len % k; // 前remainder部分需要+1 + + ListNode[] result = new ListNode[k]; + curr = root; + + for (int i = 0; i < k; i++) { + result[i] = curr; + + // 当前部分的长度 + int currentPartSize = partSize + (i < remainder ? 1 : 0); + + // 移动到当前部分的末尾 + for (int j = 0; j < currentPartSize - 1 && curr != null; j++) { + curr = curr.next; + } + + // 切断连接 + if (curr != null) { + ListNode next = curr.next; + curr.next = null; + curr = next; + } + } + + return result; +} +``` + + + +### [146.LRU 缓存机制](https://leetcode-cn.com/problems/lru-cache/) + +> 运用你所掌握的数据结构，设计和实现一个 LRU (最近最少使用) 缓存机制 。实现 LRUCache 类： +> +> - LRUCache(int capacity) 以正整数作为容量 capacity 初始化 LRU 缓存 +> - int get(int key) 如果关键字 key 存在于缓存中，则返回关键字的值，否则返回 -1 。 +> - void put(int key, int value) 如果关键字已经存在，则变更其数据值；如果关键字不存在，则插入该组「关键字-值」。当缓存容量达到上限时，它应该在写入新数据之前删除最久未使用的数据值，从而为新的数据值留出空间。 +> +> ``` +> 输入 +> ["LRUCache", "put", "put", "get", "put", "get", "put", "get", "get", "get"] +> [[2], [1, 1], [2, 2], [1], [3, 3], [2], [4, 4], [1], [3], [4]] +> 输出 +> [null, null, null, 1, null, -1, null, -1, 3, 4] +> +> 解释 +> LRUCache lRUCache = new LRUCache(2); +> lRUCache.put(1, 1); // 缓存是 {1=1} +> lRUCache.put(2, 2); // 缓存是 {1=1, 2=2} +> lRUCache.get(1); // 返回 1 +> lRUCache.put(3, 3); // 该操作会使得关键字 2 作废，缓存是 {1=1, 3=3} +> lRUCache.get(2); // 返回 -1 (未找到) +> lRUCache.put(4, 4); // 该操作会使得关键字 1 作废，缓存是 {4=4, 3=3} +> lRUCache.get(1); // 返回 -1 (未找到) +> lRUCache.get(3); // 返回 3 +> lRUCache.get(4); // 返回 4 +> ``` + +思路：LRU 缓存机制可以通过哈希表（HashMap）和双向链表（Doubly Linked List）的组合来实现。哈希表用于快速查找缓存中的元素，而双向链表用于维护元素的访问顺序。 + +- 哈希表：键为缓存的键，值为双向链表中的节点。这样可以在 O(1) 时间内通过键找到对应的节点。 +- 双向链表：每个节点存储键、值和指向前一个节点、后一个节点的指针。链表按照访问顺序排序，最近访问的节点放在链表尾部，最久未访问的节点放在链表头部。 + +```java +import java.util.HashMap; +import java.util.Map; + +class LRUCache { + private final int capacity; + private final Map cache; + private final int size; + private final DLinkedNode head, tail; + + // 使用伪头部和伪尾部节点 + class DLinkedNode { + int key; + int value; + DLinkedNode prev; + DLinkedNode next; + + DLinkedNode() {} + + DLinkedNode(int key, int value) { this.key = key; this.value = value; } + } + + public LRUCache(int capacity) { + this.capacity = capacity; + this.size = 0; + // 使用伪头部和伪尾部节点 + head = new DLinkedNode(); + tail = new DLinkedNode(); + head.next = tail; + tail.prev = head; + cache = new HashMap<>(capacity); + } + + public int get(int key) { + DLinkedNode node = cache.get(key); + if (node == null) { + return -1; + } + // 移动到头部 + moveToHead(node); + return node.value; + } + + public void put(int key, int value) { + DLinkedNode node = cache.get(key); + if (node == null) { + // 如果 key 不存在，创建一个新的节点 + DLinkedNode newNode = new DLinkedNode(key, value); + // 添加进哈希表 + cache.put(key, newNode); + // 添加至双向链表的头部 + addToHead(newNode); + ++size; + if (size > capacity) { + // 如果超出容量，删除双向链表的尾部节点 + DLinkedNode tail = removeTail(); + // 删除哈希表中对应的项 + cache.remove(tail.key); + --size; + } + } else { + // 如果 key 存在，先通过哈希表定位，再修改 value，并移动到头部 + node.value = value; + moveToHead(node); + } + } + + private void addToHead(DLinkedNode node) { + node.prev = head; + node.next = head.next; + head.next.prev = node; + head.next = node; + } + + private void removeNode(DLinkedNode node) { + node.prev.next = node.next; + node.next.prev = node.prev; + } + + private void moveToHead(DLinkedNode node) { + removeNode(node); + addToHead(node); + } + + private DLinkedNode removeTail() { + DLinkedNode res = tail.prev; + removeNode(res); + return res; + } +} +``` + + + + + + + +### [109. 有序链表转换二叉搜索树](https://leetcode.cn/problems/convert-sorted-list-to-binary-search-tree/) + +**🎯 考察频率：中等 | 难度：中等 | 重要性：链表与树的结合** + +> 将有序链表转换为平衡二叉搜索树 + +**💡 核心思路**： + +1. 快慢指针找中点作为根节点 +2. 递归构建左右子树 +3. 或者先转数组再构建 + +```java +public TreeNode sortedListToBST(ListNode head) { + if (head == null) return null; + if (head.next == null) return new TreeNode(head.val); + + // 找中点的前一个节点 + ListNode slow = head, fast = head, prev = null; + while (fast != null && fast.next != null) { + prev = slow; + slow = slow.next; + fast = fast.next.next; + } + + // 断开左半部分 + prev.next = null; + + TreeNode root = new TreeNode(slow.val); + root.left = sortedListToBST(head); + root.right = sortedListToBST(slow.next); + + return root; +} +``` + + + +--- + +## 八、进阶与变体类 🚀 + +### 💡 核心思想 + +- **算法变体**：经典题目的变形和扩展 +- **边界优化**：特殊情况的处理 +- **复合应用**：多种技巧的综合运用 + +### [430. 扁平化多级双向链表](https://leetcode.cn/problems/flatten-a-multilevel-doubly-linked-list/) + +**🎯 考察频率：低 | 难度：中等 | 重要性：递归应用** + +> 扁平化带有子链表的双向链表 + +**💡 核心思路**：DFS深度优先 + +1. 遇到有child的节点时，先处理child分支 +2. 用栈保存当前节点的next +3. 递归处理完child后，连接栈中保存的next + +```java +public Node flatten(Node head) { + if (head == null) return head; + + Stack stack = new Stack<>(); + Node curr = head; + + while (curr != null) { + if (curr.child != null) { + // 如果有next节点，先保存到栈中 + if (curr.next != null) { + stack.push(curr.next); + } + + // 连接child + curr.next = curr.child; + curr.child.prev = curr; + curr.child = null; + } + + // 如果当前节点没有next，但栈不为空 + if (curr.next == null && !stack.isEmpty()) { + Node next = stack.pop(); + curr.next = next; + next.prev = curr; + } + + curr = curr.next; + } + + return head; +} +``` + + + +### [1669. 合并两个链表](https://leetcode.cn/problems/merge-in-between-linked-lists/) + +**🎯 考察频率：低 | 难度：中等 | 重要性：指针操作** + +> 将list2插入到list1的第a到第b个节点之间 + +**💡 核心思路**： + +1. 找到第a-1个和第b+1个节点 +2. 连接三段：前段+list2+后段 + +```java +public ListNode mergeInBetween(ListNode list1, int a, int b, ListNode list2) { + // 找到第a-1个节点 + ListNode prevA = list1; + for (int i = 0; i < a - 1; i++) { + prevA = prevA.next; + } + + // 找到第b+1个节点 + ListNode afterB = prevA; + for (int i = 0; i < b - a + 2; i++) { + afterB = afterB.next; + } + + // 连接 + prevA.next = list2; + + // 找到list2的尾节点 + while (list2.next != null) { + list2 = list2.next; + } + list2.next = afterB; + + return list1; +} +``` + +--- + +## 九、进阶操作类 + +--- + +## 🚀 面试前15分钟速记表 + +### 核心模板（必背） + +```java +// 1. 哨兵节点模板 +ListNode dummy = new ListNode(0); +dummy.next = head; + +// 2. 快慢指针模板 +ListNode slow = head, fast = head; +while (fast != null && fast.next != null) { + slow = slow.next; + fast = fast.next.next; +} + +// 3. 反转链表模板 +ListNode prev = null, curr = head; +while (curr != null) { + ListNode next = curr.next; + curr.next = prev; + prev = curr; + curr = next; +} +``` + +### 题型速查表 + +| 题型分类 | 核心技巧 | 高频题目 | 记忆口诀 | 难度 | +| ------------ | --------------- | ----------------------------------------------------------- | -------------------------------- | ----- | +| **双指针类** | 快慢指针 | 环形链表、环形链表II、链表中间节点、删除倒数第N个、旋转链表 | 快二慢一找中点，有环必定会相遇 | ⭐⭐⭐ | +| **反转重排** | 三指针法 | 反转链表、反转链表II、K个一组翻转、重排链表、两两交换 | 断链之前存next，三指针来回倒腾 | ⭐⭐⭐⭐ | +| **相交合并** | 双路遍历 | 相交链表、合并两个有序链表、合并K个升序链表 | 你走我的路，我走你的路 | ⭐⭐⭐ | +| **删除去重** | 哨兵节点 | 移除链表元素、删除重复元素、删除重复元素II | 哨兵开路找前驱，一指断链二连接 | ⭐⭐⭐ | +| **数学运算** | 进位处理 | 两数相加、两数相加II | 逐位相加记进位，链表模拟手算法 | ⭐⭐⭐⭐ | +| **特殊结构** | 哈希表+特殊指针 | LRU缓存、复制带随机指针链表、回文链表 | 哈希定位双链调，头部最新尾最老 | ⭐⭐⭐⭐⭐ | +| **综合应用** | 分治递归 | 排序链表、分隔链表、奇偶链表、分隔链表 | 大化小来小化了，分而治之最高效 | ⭐⭐⭐⭐ | +| **进阶变体** | 复合技巧 | 扁平化多级双向链表、有序链表转BST、合并两个链表 | 多技巧组合显神通，复杂问题巧拆解 | ⭐⭐⭐⭐ | + +### 按难度分级 + +- **⭐⭐⭐ 简单必会**：反转链表、环形链表、合并两个有序链表、删除重复元素、链表的中间结点 +- **⭐⭐⭐⭐ 中等重点**：环形链表II、删除倒数第N个、反转链表II、相交链表、两数相加、复制带随机指针链表、回文链表 +- **⭐⭐⭐⭐⭐ 困难经典**：K个一组翻转链表、合并K个升序链表、LRU缓存 + +### 核心题目优先级（面试前重点复习） + +1. **反转链表** - 基础中的基础，必须熟练掌握 +2. **环形链表** - 快慢指针入门，双指针经典 +3. **合并两个有序链表** - 归并算法基础，递归思维 +4. **两数相加** - 数学计算经典，进位处理 +5. **删除链表的倒数第N个结点** - 双指针应用典型 +6. **LRU缓存** - 系统设计经典，哈希表+双向链表 +7. **复制带随机指针的链表** - 特殊结构经典，深拷贝 +8. **合并K个升序链表** - 分治算法进阶，优先队列 + +### 常见陷阱提醒 + +- ⚠️ **空链表检查**：`if (head == null) return ...` +- ⚠️ **单节点处理**：`if (head.next == null) return ...` +- ⚠️ **快指针越界**：`while (fast != null && fast.next != null)` +- ⚠️ **哨兵节点**：处理头节点变化时必用 +- ⚠️ **返回值**：注意返回`dummy.next`还是`head` + +### 时间复杂度总结 + +- **遍历类**：O(n) - 一次遍历 +- **快慢指针**：O(n) - 最多两次遍历 +- **双指针**：O(n) - 同时移动 +- **递归**：O(n) - 栈空间O(n) +- **排序**：O(n log n) - 归并排序 + +### 面试答题套路 + +1. **理解题意**：确认输入输出，边界情况 +2. **选择方法**：根据题型选择对应技巧 +3. **画图分析**：手动模拟2-3个节点的情况 +4. **编码实现**：套用模板，注意边界 +5. **测试验证**：空链表、单节点、正常情况 + +**最后提醒**：链表题重在理解指针操作，多画图，多动手！ + + + diff --git a/docs/data-structure-algorithms/soultion/Math-Solution.md b/docs/data-structure-algorithms/soultion/Math-Solution.md new file mode 100755 index 0000000000..3999ed1f5c --- /dev/null +++ b/docs/data-structure-algorithms/soultion/Math-Solution.md @@ -0,0 +1,47 @@ +### [9. 回文数](https://leetcode.cn/problems/palindrome-number/) + +> 给你一个整数 x ，如果 x 是一个回文整数，返回 true ；否则，返回 false 。 +> +> 回文数是指正序（从左向右）和倒序（从右向左）读都是一样的整数。 +> +> 例如，121 是回文，而 123 不是。 +> + +**思路**： + +解法1：转成字符串后反转并比较 + +```java +public boolean isPalindrome(int num){ + String revertNum = new StringBuilder(""+num).reverse().toString(); + return revertNum.equals(num+""); +} +``` + +解法2：反转一半数字（数学法） + +例如，输入 1221，我们可以将数字 “1221” 的后半部分从 “21” 反转为 “12”，并将其与前半部分 “12” 进行比较，因为二者相同，我们得知数字 1221 是回文。 + +![fig1](https://tva1.sinaimg.cn/large/e6c9d24ely1h35m91kil7j20zk0k00uc.jpg) + +1. 每次进行取余操作 （ %10），取出最低的数字：y = x % 10 +2. 将最低的数字加到取出数的末尾：revertNum = revertNum * 10 + y +3. 每取一个最低位数字，x 都要自除以 10 +4. 判断 x 是不是小于 revertNum ，当它小于的时候，说明数字已经对半或者过半了 +5. 最后，判断奇偶数情况：如果是偶数的话，revertNum 和 x 相等；如果是奇数的话，最中间的数字就在revertNum 的最低位上，将它除以 10 以后应该和 x 相等。 + +```java +public boolean isPalindrome(int num){ + //所有负数、所有大于 0 且个位是 0 的数字都不可能是回文 + if(num < 0 || (num > 0 && num % 10 == 0)){ + return false; + } + int tmp = 0; + while(num > tmp){ + tmp = tmp * 10 + tmp % 10; + num = num / 10; + } + return num == tmp || num == tmp / 10; + } +``` + diff --git a/docs/data-structure-algorithms/soultion/String-Solution.md b/docs/data-structure-algorithms/soultion/String-Solution.md new file mode 100755 index 0000000000..e617e0b35a --- /dev/null +++ b/docs/data-structure-algorithms/soultion/String-Solution.md @@ -0,0 +1,1453 @@ +--- +title: 字符串算法题目 +date: 2023-05-08 +tags: + - String +categories: leetcode +--- + +![](https://img.starfish.ink/leetcode/leetcode-banner.png) + +> 字符串的题目，和数组的题目大差不大 + +## 🎯 核心考点概览 + +- **双指针技巧**：对撞指针、快慢指针、滑动窗口 +- **字符统计**：哈希表统计字符频次 +- **模式匹配**：KMP算法、滚动哈希 +- **回文处理**：中心扩展、马拉车算法 +- **动态规划**：最长公共子序列、编辑距离 +- **字符串变换**：反转、替换、分割重组 +- **前缀后缀**：前缀树、后缀数组应用 + +## 💡 解题万能模板 + +### 🔄 双指针模板 + +```java +// 对撞指针模板 +public boolean isPalindrome(String s) { + int left = 0, right = s.length() - 1; + + while (left < right) { + // 处理逻辑 + if (condition) { + left++; + right--; + } else { + return false; + } + } + return true; +} +``` + +### 🪟 滑动窗口模板 + +```java +// 滑动窗口模板 +public int slidingWindow(String s) { + Map window = new HashMap<>(); + int left = 0, right = 0; + int result = 0; + + while (right < s.length()) { + char c = s.charAt(right); + window.put(c, window.getOrDefault(c, 0) + 1); + right++; + + while (windowNeedshrink()) { + char d = s.charAt(left); + window.put(d, window.get(d) - 1); + left++; + } + + // 更新结果 + result = Math.max(result, right - left); + } + + return result; +} +``` + +### 🎯 字符统计模板 + +```java +// 字符频次统计模板 +public Map charCount(String s) { + Map count = new HashMap<>(); + for (char c : s.toCharArray()) { + count.put(c, count.getOrDefault(c, 0) + 1); + } + return count; +} +``` + +### 🔍 KMP模式匹配模板 + +```java +// KMP算法模板 +public int strStr(String haystack, String needle) { + if (needle.isEmpty()) return 0; + + int[] next = buildNext(needle); + int i = 0, j = 0; + + while (i < haystack.length()) { + if (haystack.charAt(i) == needle.charAt(j)) { + i++; + j++; + } + + if (j == needle.length()) { + return i - j; + } else if (i < haystack.length() && haystack.charAt(i) != needle.charAt(j)) { + if (j != 0) { + j = next[j - 1]; + } else { + i++; + } + } + } + + return -1; +} +``` + +## 🛡️ 边界检查清单 + +- ✅ 空字符串处理：`s == null || s.isEmpty()` +- ✅ 单字符处理：`s.length() == 1` +- ✅ 索引越界：`i >= 0 && i < s.length()` +- ✅ 字符大小写：统一处理或分别考虑 +- ✅ 特殊字符：空格、标点符号的处理 + +## 💡 记忆口诀 + +- **双指针**："对撞指针找回文，快慢指针找重复" +- **滑动窗口**："左右指针动态调，窗口大小随需要" +- **字符统计**："哈希计数是法宝，频次统计用得妙" +- **模式匹配**："KMP算法巧匹配，前缀表来帮助你" +- **回文判断**："中心扩展找回文，奇偶长度都考虑" +- **字符串DP**："状态转移要清晰，边界条件别忘记" +- **字符变换**："原地修改要小心，额外空间换时间" + + + +### 📋 正确分类索引 + +1. **🔥 双指针基础类**：[验证回文串](#_125-验证回文串)、[反转字符串](#_344-反转字符串)、[反转字符串中的单词](#_151-反转字符串中的单词)、[最长回文子串](#_5-最长回文子串) +2. **🪟 滑动窗口类**：[无重复字符的最长子串](#_3-无重复字符的最长子串)、[最小覆盖子串](#_76-最小覆盖子串)、长度最小的子数组、[找到字符串中所有字母异位词](#_438-找到字符串中所有字母异位词) +3. **📊 字符统计类**：[有效的字母异位词](#_242-有效的字母异位词)、[字母异位词分组](#_49-字母异位词分组)、赎金信、第一个不重复的字符 +4. **🔍 模式匹配类**：[实现strStr()](#_28-找出字符串中第一个匹配项的下标)、重复的子字符串、字符串匹配、正则表达式匹配 +5. **🌀 回文专题类**：[回文子串](#_647-回文子串)、[最长回文子串](#_5-最长回文子串)、回文链表、分割回文串 +6. **🎯 动态规划类**：[最长公共子序列](#_1143-最长公共子序列)、[编辑距离](#_72-编辑距离)、不同的子序列、交错字符串 +7. **🔄 字符串变换类**：整数转换、[字符串转换整数(atoi)](#_8-字符串转换整数-atoi)、字符串相加、字符串相乘 +8. **🚀 进阶技巧类**：[最短回文串](#_214-最短回文串)、[复原IP地址](#_93-复原ip地址)、[有效的括号](#_20-有效的括号)、[电话号码的字母组合](#_17-电话号码的字母组合)、串联所有单词的子串 + +--- + +## 🔥 一、双指针基础类（基石中的基石） + +### 💡 核心思想 + +- **对撞指针**：从两端向中间靠拢 +- **快慢指针**：不同速度遍历处理 +- **滑动窗口**：动态调整窗口大小 + +### [125. 验证回文串](https://leetcode.cn/problems/valid-palindrome/) + +**🎯 考察频率：极高 | 难度：简单 | 重要性：双指针入门** + +> 验证字符串是否为回文串（忽略大小写和非字母数字字符）：`"A man, a plan, a canal: Panama"` → `true` + +**💡 核心思路**：对撞指针 + 字符过滤 + +- 双指针从两端向中间移动 +- 跳过非字母数字字符 +- 统一转换为小写比较 + +**🔑 记忆技巧**： + +- **口诀**："对撞指针验回文，跳过无效比字符" +- **形象记忆**：像照镜子一样，左右对称 + +```java +public boolean isPalindrome(String s) { + if (s == null || s.isEmpty()) return true; + + int left = 0, right = s.length() - 1; + + while (left < right) { + // 跳过左边的非字母数字字符 + while (left < right && !Character.isLetterOrDigit(s.charAt(left))) { + left++; + } + + // 跳过右边的非字母数字字符 + while (left < right && !Character.isLetterOrDigit(s.charAt(right))) { + right--; + } + + // 比较字符（转换为小写） + if (Character.toLowerCase(s.charAt(left)) != + Character.toLowerCase(s.charAt(right))) { + return false; + } + + left++; + right--; + } + + return true; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 遍历字符串一次 +- **空间复杂度**：O(1) - 只使用常数个变量 + +### [344. 反转字符串](https://leetcode.cn/problems/reverse-string/) + +**🎯 考察频率：高 | 难度：简单 | 重要性：原地操作基础** + +> 原地反转字符数组：`['h','e','l','l','o']` → `['o','l','l','e','h']` + +**💡 核心思路**：对撞指针交换 + +- 左右指针交换字符 +- 向中间靠拢直到相遇 + +**🔑 记忆技巧**： + +- **口诀**："对撞指针做交换，原地反转很简单" + +```java +public void reverseString(char[] s) { + int left = 0, right = s.length - 1; + + while (left < right) { + // 交换字符 + char temp = s[left]; + s[left] = s[right]; + s[right] = temp; + + left++; + right--; + } +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 遍历一半字符 +- **空间复杂度**：O(1) - 原地操作 + +### [151. 反转字符串中的单词](https://leetcode.cn/problems/reverse-words-in-a-string/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：字符串处理综合** + +> 反转单词顺序：`"the sky is blue"` → `"blue is sky the"` + +**💡 核心思路**：分割 + 反转 + 重组 + +- 方法1：split分割后反转数组 +- 方法2：双指针提取单词后反向拼接 + +**🔑 记忆技巧**： + +- **口诀**："单词分割后反转，空格处理要注意" + +```java +// 方法1：使用内置函数 +public String reverseWords(String s) { + String[] words = s.trim().split("\\s+"); + StringBuilder result = new StringBuilder(); + + for (int i = words.length - 1; i >= 0; i--) { + result.append(words[i]); + if (i > 0) result.append(" "); + } + + return result.toString(); +} +``` + +**🔧 双指针解法**： + +```java +public String reverseWords(String s) { + StringBuilder result = new StringBuilder(); + int n = s.length(); + int i = n - 1; + + while (i >= 0) { + // 跳过空格 + while (i >= 0 && s.charAt(i) == ' ') i--; + + if (i < 0) break; + + // 找单词边界 + int j = i; + while (i >= 0 && s.charAt(i) != ' ') i--; + + // 添加单词 + if (result.length() > 0) result.append(" "); + result.append(s.substring(i + 1, j + 1)); + } + + return result.toString(); +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 遍历字符串 +- **空间复杂度**：O(n) - 存储结果 + +### [5. 最长回文子串](https://leetcode.cn/problems/longest-palindromic-substring/) + +**🎯 考察频率：极高 | 难度：中等 | 重要性：回文算法经典** + +> 找到字符串中最长的回文子串：`"babad"` → `"bab"` 或 `"aba"` + +**💡 核心思路**：中心扩展法 + +- 遍历每个可能的回文中心 +- 分别处理奇数长度和偶数长度回文 +- 从中心向两边扩展，记录最长回文 + +**🔑 记忆技巧**： + +- **口诀**："中心扩展找回文，奇偶长度分别算" +- **形象记忆**：像投石子到水中，波纹向外扩散 + +```java +public String longestPalindrome(String s) { + if (s == null || s.length() < 2) return s; + + int start = 0, maxLen = 1; + + for (int i = 0; i < s.length(); i++) { + // 奇数长度回文 + int len1 = expandAroundCenter(s, i, i); + // 偶数长度回文 + int len2 = expandAroundCenter(s, i, i + 1); + + int len = Math.max(len1, len2); + if (len > maxLen) { + maxLen = len; + start = i - (len - 1) / 2; + } + } + + return s.substring(start, start + maxLen); +} + +private int expandAroundCenter(String s, int left, int right) { + while (left >= 0 && right < s.length() && + s.charAt(left) == s.charAt(right)) { + left--; + right++; + } + return right - left - 1; +} +``` + +**🔧 动态规划解法**： + +```java +public String longestPalindrome(String s) { + int n = s.length(); + boolean[][] dp = new boolean[n][n]; + int start = 0, maxLen = 1; + + // 单个字符都是回文 + for (int i = 0; i < n; i++) { + dp[i][i] = true; + } + + // 检查长度为2的子串 + for (int i = 0; i < n - 1; i++) { + if (s.charAt(i) == s.charAt(i + 1)) { + dp[i][i + 1] = true; + start = i; + maxLen = 2; + } + } + + // 检查长度大于2的子串 + for (int len = 3; len <= n; len++) { + for (int i = 0; i < n - len + 1; i++) { + int j = i + len - 1; + if (s.charAt(i) == s.charAt(j) && dp[i + 1][j - 1]) { + dp[i][j] = true; + start = i; + maxLen = len; + } + } + } + + return s.substring(start, start + maxLen); +} +``` + +**⏱️ 复杂度分析**： + +- **中心扩展法**： + - 时间复杂度：O(n²) - 每个中心扩展O(n) + - 空间复杂度：O(1) - 常数空间 +- **动态规划法**： + - 时间复杂度：O(n²) - 双重循环 + - 空间复杂度：O(n²) - 二维DP表 + +--- + +## 🪟 二、滑动窗口类 + +### 💡 核心思想 + +- **动态窗口**：根据条件调整窗口大小 +- **窗口收缩**：不满足条件时缩小窗口 +- **窗口扩展**：满足条件时扩大窗口 + +### [3. 无重复字符的最长子串](https://leetcode.cn/problems/longest-substring-without-repeating-characters/) + +**🎯 考察频率：极高 | 难度：中等 | 重要性：滑动窗口经典** + +> 找出不含有重复字符的最长子串：`"abcabcbb"` → `3` ("abc") + +**💡 核心思路**：滑动窗口 + 哈希表 + +- 右指针扩展窗口，左指针收缩窗口 +- 哈希表记录字符最近出现的位置 +- 遇到重复字符时，移动左指针到重复字符的下一位 + +**🔑 记忆技巧**： + +- **口诀**："滑动窗口找无重复，哈希记录字符位置" +- **形象记忆**：像一个会伸缩的尺子，遇到重复就缩短 + +```java +public int lengthOfLongestSubstring(String s) { + Map window = new HashMap<>(); + int left = 0, right = 0; + int maxLen = 0; + + while (right < s.length()) { + char c = s.charAt(right); + + // 如果字符已存在且在当前窗口内 + if (window.containsKey(c) && window.get(c) >= left) { + left = window.get(c) + 1; + } + + window.put(c, right); + maxLen = Math.max(maxLen, right - left + 1); + right++; + } + + return maxLen; +} +``` + +**🔧 优化版本（使用数组）**： + +```java +public int lengthOfLongestSubstring(String s) { + int[] lastIndex = new int[128]; // ASCII字符 + Arrays.fill(lastIndex, -1); + + int left = 0, maxLen = 0; + + for (int right = 0; right < s.length(); right++) { + char c = s.charAt(right); + + if (lastIndex[c] >= left) { + left = lastIndex[c] + 1; + } + + lastIndex[c] = right; + maxLen = Math.max(maxLen, right - left + 1); + } + + return maxLen; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 每个字符最多被访问两次 +- **空间复杂度**：O(k) - k为字符集大小 + +### [76. 最小覆盖子串](https://leetcode.cn/problems/minimum-window-substring/) + +**🎯 考察频率：极高 | 难度：困难 | 重要性：滑动窗口终极考验** + +> 找出字符串s中包含字符串t所有字符的最小子串：`s="ADOBECODEBANC", t="ABC"` → `"BANC"` + +**💡 核心思路**：滑动窗口 + 字符计数 + +- 先扩展右指针直到窗口包含所有目标字符 +- 然后收缩左指针直到不再满足条件 +- 记录满足条件的最小窗口![fig1](https://assets.leetcode-cn.com/solution-static/76/76_fig1.gif) + +**🔑 记忆技巧**： + +- **口诀**："右扩左缩找最小，字符计数要匹配" +- **形象记忆**：像拉手风琴，先拉开再压缩到最小 + +```java +public String minWindow(String s, String t) { + Map need = new HashMap<>(); + Map window = new HashMap<>(); + + // 统计目标字符串的字符频次 + for (char c : t.toCharArray()) { + need.put(c, need.getOrDefault(c, 0) + 1); + } + + int left = 0, right = 0; + int valid = 0; // 满足条件的字符种类数 + int start = 0, len = Integer.MAX_VALUE; + + while (right < s.length()) { + char c = s.charAt(right); + right++; + + // 扩展窗口 + if (need.containsKey(c)) { + window.put(c, window.getOrDefault(c, 0) + 1); + if (window.get(c).equals(need.get(c))) { + valid++; + } + } + + // 收缩窗口 + while (valid == need.size()) { + // 更新最小覆盖子串 + if (right - left < len) { + start = left; + len = right - left; + } + + char d = s.charAt(left); + left++; + + if (need.containsKey(d)) { + if (window.get(d).equals(need.get(d))) { + valid--; + } + window.put(d, window.get(d) - 1); + } + } + } + + return len == Integer.MAX_VALUE ? "" : s.substring(start, start + len); +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(|s| + |t|) - 每个字符最多被访问两次 +- **空间复杂度**：O(|s| + |t|) - 哈希表存储空间 + +### [438. 找到字符串中所有字母异位词](https://leetcode.cn/problems/find-all-anagrams-in-a-string/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：固定窗口滑动** + +> 找到字符串s中所有p的字母异位词的起始索引：`s="abab", p="ab"` → `[0,2]` + +**💡 核心思路**：固定大小滑动窗口 + +- 窗口大小固定为p的长度 +- 比较窗口内字符频次与p的字符频次 +- 相等则找到一个异位词 + +**🔑 记忆技巧**： + +- **口诀**："固定窗口滑动找，字符频次要相等" + +```java +public List findAnagrams(String s, String p) { + List result = new ArrayList<>(); + if (s.length() < p.length()) return result; + + int[] pCount = new int[26]; + int[] windowCount = new int[26]; + + // 统计p的字符频次 + for (char c : p.toCharArray()) { + pCount[c - 'a']++; + } + + int windowSize = p.length(); + + // 初始化第一个窗口 + for (int i = 0; i < windowSize; i++) { + windowCount[s.charAt(i) - 'a']++; + } + + // 检查第一个窗口 + if (Arrays.equals(pCount, windowCount)) { + result.add(0); + } + + // 滑动窗口 + for (int i = windowSize; i < s.length(); i++) { + // 添加新字符 + windowCount[s.charAt(i) - 'a']++; + // 移除旧字符 + windowCount[s.charAt(i - windowSize) - 'a']--; + + // 检查当前窗口 + if (Arrays.equals(pCount, windowCount)) { + result.add(i - windowSize + 1); + } + } + + return result; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - n为字符串s的长度 +- **空间复杂度**：O(1) - 固定大小的计数数组 + +--- + +## 📊 三、字符统计类 + +### 💡 核心思想 + +- **哈希计数**：统计字符出现频次 +- **频次比较**：比较两个字符串的字符频次 +- **异位词判断**：相同字符不同排列 + +### [242. 有效的字母异位词](https://leetcode.cn/problems/valid-anagram/) + +**🎯 考察频率：极高 | 难度：简单 | 重要性：字符统计基础** + +> 判断两个字符串是否为字母异位词：`s="anagram", t="nagaram"` → `true` + +**💡 核心思路**：字符频次统计 + +- 统计两个字符串的字符频次 +- 比较频次是否完全相同 + +**🔑 记忆技巧**： + +- **口诀**："字符计数要相等，异位词就是重排列" + +```java +public boolean isAnagram(String s, String t) { + if (s.length() != t.length()) return false; + + int[] count = new int[26]; + + // 统计字符频次差 + for (int i = 0; i < s.length(); i++) { + count[s.charAt(i) - 'a']++; + count[t.charAt(i) - 'a']--; + } + + // 检查是否所有字符频次都为0 + for (int c : count) { + if (c != 0) return false; + } + + return true; +} +``` + +**🔧 排序解法**： + +```java +public boolean isAnagram(String s, String t) { + if (s.length() != t.length()) return false; + + char[] sChars = s.toCharArray(); + char[] tChars = t.toCharArray(); + + Arrays.sort(sChars); + Arrays.sort(tChars); + + return Arrays.equals(sChars, tChars); +} +``` + +**⏱️ 复杂度分析**： + +- **计数法**： + - 时间复杂度：O(n) - 遍历字符串 + - 空间复杂度：O(1) - 固定大小数组 +- **排序法**： + - 时间复杂度：O(nlogn) - 排序时间 + - 空间复杂度：O(1) - 原地排序 + +### [49. 字母异位词分组](https://leetcode.cn/problems/group-anagrams/) + +**🎯 考察频率：极高 | 难度：中等 | 重要性：哈希表分组** + +> 将字母异位词分组：`["eat","tea","tan","ate","nat","bat"]` → `[["bat"],["nat","tan"],["ate","eat","tea"]]` + +**💡 核心思路**：哈希表分组 + +- 将每个字符串排序作为key +- 相同key的字符串归为一组 + +**🔑 记忆技巧**： + +- **口诀**："排序作键分组归，异位词汇一家聚" + +```java +public List> groupAnagrams(String[] strs) { + Map> groups = new HashMap<>(); + + for (String str : strs) { + char[] chars = str.toCharArray(); + Arrays.sort(chars); + String key = String.valueOf(chars); + + groups.computeIfAbsent(key, k -> new ArrayList<>()).add(str); + } + + return new ArrayList<>(groups.values()); +} +``` + +**🔧 计数法（避免排序）**： + +```java +public List> groupAnagrams(String[] strs) { + Map> groups = new HashMap<>(); + + for (String str : strs) { + int[] count = new int[26]; + for (char c : str.toCharArray()) { + count[c - 'a']++; + } + + // 构造唯一key + StringBuilder sb = new StringBuilder(); + for (int i = 0; i < 26; i++) { + sb.append('#').append(count[i]); + } + String key = sb.toString(); + + groups.computeIfAbsent(key, k -> new ArrayList<>()).add(str); + } + + return new ArrayList<>(groups.values()); +} +``` + +**⏱️ 复杂度分析**： + +- **排序法**： + - 时间复杂度：O(n×klogk) - n个字符串，每个长度k + - 空间复杂度：O(n×k) - 存储结果 +- **计数法**： + - 时间复杂度：O(n×k) - 线性时间 + - 空间复杂度：O(n×k) - 存储结果 + +--- + +## 🔍 四、模式匹配类 + +### 💡 核心思想 + +- **朴素匹配**：暴力比较每个位置 +- **KMP算法**：利用前缀表优化匹配 +- **滚动哈希**：哈希值快速比较 + +### [28. 找出字符串中第一个匹配项的下标](https://leetcode.cn/problems/find-the-index-of-the-first-occurrence-in-a-string/) + +**🎯 考察频率：极高 | 难度：中等 | 重要性：字符串匹配经典** + +> 在haystack中找到needle第一次出现的位置：`haystack="sadbutsad", needle="sad"` → `0` + +**💡 核心思路**：KMP算法 + +- 构建needle的前缀表（next数组） +- 利用前缀表在不匹配时快速跳转 + +**🔑 记忆技巧**： + +- **口诀**："KMP算法巧匹配，前缀表来帮助你" +- **形象记忆**：像有记忆的搜索，不会重复无用的比较 + +```java +public int strStr(String haystack, String needle) { + if (needle.isEmpty()) return 0; + + int[] next = buildNext(needle); + int i = 0, j = 0; + + while (i < haystack.length()) { + if (haystack.charAt(i) == needle.charAt(j)) { + i++; + j++; + } + + if (j == needle.length()) { + return i - j; + } else if (i < haystack.length() && haystack.charAt(i) != needle.charAt(j)) { + if (j != 0) { + j = next[j - 1]; + } else { + i++; + } + } + } + + return -1; +} + +private int[] buildNext(String pattern) { + int[] next = new int[pattern.length()]; + int len = 0, i = 1; + + while (i < pattern.length()) { + if (pattern.charAt(i) == pattern.charAt(len)) { + len++; + next[i] = len; + i++; + } else { + if (len != 0) { + len = next[len - 1]; + } else { + next[i] = 0; + i++; + } + } + } + + return next; +} +``` + +**🔧 朴素匹配解法**： + +```java +public int strStr(String haystack, String needle) { + if (needle.isEmpty()) return 0; + + for (int i = 0; i <= haystack.length() - needle.length(); i++) { + if (haystack.substring(i, i + needle.length()).equals(needle)) { + return i; + } + } + + return -1; +} +``` + +**⏱️ 复杂度分析**： + +- **KMP算法**： + - 时间复杂度：O(m + n) - m,n分别为两字符串长度 + - 空间复杂度：O(m) - next数组 +- **朴素匹配**： + - 时间复杂度：O(m×n) - 最坏情况 + - 空间复杂度：O(1) - 常数空间 + +--- + +## 🌀 五、回文专题类 + +### 💡 核心思想 + +- **中心扩展**：从中心向两边扩展 +- **动态规划**：子问题最优解 +- **马拉车算法**：线性时间找所有回文 + +### [647. 回文子串](https://leetcode.cn/problems/palindromic-substrings/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：回文计数** + +> 统计字符串中回文子串的个数：`"abc"` → `3` ("a", "b", "c") + +**💡 核心思路**：中心扩展法 + +- 遍历每个可能的回文中心 +- 统计以每个中心扩展出的回文个数 + +**🔑 记忆技巧**： + +- **口诀**："中心扩展数回文，奇偶中心都要算" + +```java +public int countSubstrings(String s) { + int count = 0; + + for (int i = 0; i < s.length(); i++) { + // 奇数长度回文 + count += countPalindromes(s, i, i); + // 偶数长度回文 + count += countPalindromes(s, i, i + 1); + } + + return count; +} + +private int countPalindromes(String s, int left, int right) { + int count = 0; + + while (left >= 0 && right < s.length() && + s.charAt(left) == s.charAt(right)) { + count++; + left--; + right++; + } + + return count; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n²) - 每个中心最多扩展O(n) +- **空间复杂度**：O(1) - 常数空间 + +--- + +## 🎯 六、动态规划类 + +### 💡 核心思想 + +- **状态定义**：明确dp数组的含义 +- **状态转移**：找出递推关系 +- **边界条件**：初始化基础状态 + +### [1143. 最长公共子序列](https://leetcode.cn/problems/longest-common-subsequence/) + +**🎯 考察频率：极高 | 难度：中等 | 重要性：字符串DP经典** + +> 找出两个字符串的最长公共子序列长度：`text1="abcde", text2="ace"` → `3` ("ace") + +**💡 核心思路**：二维动态规划 + +- dp[i][j] 表示text1前i个字符和text2前j个字符的最长公共子序列长度 +- 状态转移：字符相同时+1，不同时取最大值 + +**🔑 记忆技巧**： + +- **口诀**："二维DP找公共，相同加一不同取大" + +```java +public int longestCommonSubsequence(String text1, String text2) { + int m = text1.length(), n = text2.length(); + int[][] dp = new int[m + 1][n + 1]; + + for (int i = 1; i <= m; i++) { + for (int j = 1; j <= n; j++) { + if (text1.charAt(i - 1) == text2.charAt(j - 1)) { + dp[i][j] = dp[i - 1][j - 1] + 1; + } else { + dp[i][j] = Math.max(dp[i - 1][j], dp[i][j - 1]); + } + } + } + + return dp[m][n]; +} +``` + +**🔧 空间优化**： + +```java +public int longestCommonSubsequence(String text1, String text2) { + int m = text1.length(), n = text2.length(); + int[] dp = new int[n + 1]; + + for (int i = 1; i <= m; i++) { + int prev = 0; + for (int j = 1; j <= n; j++) { + int temp = dp[j]; + if (text1.charAt(i - 1) == text2.charAt(j - 1)) { + dp[j] = prev + 1; + } else { + dp[j] = Math.max(dp[j], dp[j - 1]); + } + prev = temp; + } + } + + return dp[n]; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(m×n) - 双重循环 +- **空间复杂度**：O(min(m,n)) - 空间优化后 + +### [72. 编辑距离](https://leetcode.cn/problems/edit-distance/) + +**🎯 考察频率：极高 | 难度：困难 | 重要性：字符串DP巅峰** + +> 计算将word1转换成word2的最少操作数：`word1="horse", word2="ros"` → `3` + +**💡 核心思路**：二维动态规划 + +- dp[i][j] 表示word1前i个字符转换为word2前j个字符的最少操作数 +- 三种操作：插入、删除、替换 + +**🔑 记忆技巧**： + +- **口诀**："编辑距离三操作，插入删除和替换" + +```java +public int minDistance(String word1, String word2) { + int m = word1.length(), n = word2.length(); + int[][] dp = new int[m + 1][n + 1]; + + // 初始化边界 + for (int i = 0; i <= m; i++) dp[i][0] = i; + for (int j = 0; j <= n; j++) dp[0][j] = j; + + for (int i = 1; i <= m; i++) { + for (int j = 1; j <= n; j++) { + if (word1.charAt(i - 1) == word2.charAt(j - 1)) { + dp[i][j] = dp[i - 1][j - 1]; + } else { + dp[i][j] = Math.min( + Math.min(dp[i - 1][j] + 1, // 删除 + dp[i][j - 1] + 1), // 插入 + dp[i - 1][j - 1] + 1 // 替换 + ); + } + } + } + + return dp[m][n]; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(m×n) - 双重循环 +- **空间复杂度**：O(m×n) - 二维DP表 + +--- + +## 🔄 七、字符串变换类 + +### 💡 核心思想 + +- **数字转换**：处理进位和符号 +- **字符串运算**：模拟手工计算过程 +- **格式转换**：处理各种特殊情况 + +### [8. 字符串转换整数 (atoi)](https://leetcode.cn/problems/string-to-integer-atoi/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：边界处理考验** + +> 实现atoi函数，将字符串转换为整数：`" -42"` → `-42` + +**💡 核心思路**：状态机 + 边界处理 + +- 跳过前导空格 +- 处理正负号 +- 逐位转换并检查溢出 + +**🔑 记忆技巧**： + +- **口诀**："跳过空格看符号，逐位转换防溢出" + +```java +public int myAtoi(String s) { + int index = 0, sign = 1, result = 0; + + // 跳过前导空格 + while (index < s.length() && s.charAt(index) == ' ') { + index++; + } + + // 处理符号 + if (index < s.length() && (s.charAt(index) == '+' || s.charAt(index) == '-')) { + sign = s.charAt(index) == '+' ? 1 : -1; + index++; + } + + // 转换数字 + while (index < s.length() && Character.isDigit(s.charAt(index))) { + int digit = s.charAt(index) - '0'; + + // 检查溢出 + if (result > Integer.MAX_VALUE / 10 || + (result == Integer.MAX_VALUE / 10 && digit > Integer.MAX_VALUE % 10)) { + return sign == 1 ? Integer.MAX_VALUE : Integer.MIN_VALUE; + } + + result = result * 10 + digit; + index++; + } + + return result * sign; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 遍历字符串一次 +- **空间复杂度**：O(1) - 常数空间 + +--- + +## 🚀 八、进阶技巧类 + +### 💡 核心思想 + +- **高效算法**：马拉车、AC自动机等 +- **复杂匹配**：多模式匹配、通配符匹配 +- **空间优化**：原地操作、滚动数组 + +### [214. 最短回文串](https://leetcode.cn/problems/shortest-palindrome/) + +**🎯 考察频率：中等 | 难度：困难 | 重要性：KMP进阶应用** + +> 在字符串前面添加字符使其成为回文串，求最短的回文串 + +**💡 核心思路**：KMP + 回文性质 + +- 找到从开头开始的最长回文前缀 +- 在前面添加剩余部分的反转 + +**🔑 记忆技巧**： + +- **口诀**："KMP找前缀，反转补后缀" + +```java +public String shortestPalindrome(String s) { + String rev = new StringBuilder(s).reverse().toString(); + String combined = s + "#" + rev; + + int[] next = buildNext(combined); + int overlap = next[combined.length() - 1]; + + return rev.substring(0, s.length() - overlap) + s; +} + +private int[] buildNext(String pattern) { + int[] next = new int[pattern.length()]; + int len = 0, i = 1; + + while (i < pattern.length()) { + if (pattern.charAt(i) == pattern.charAt(len)) { + len++; + next[i] = len; + i++; + } else { + if (len != 0) { + len = next[len - 1]; + } else { + next[i] = 0; + i++; + } + } + } + + return next; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - KMP算法 +- **空间复杂度**：O(n) - 辅助字符串 + +### [93. 复原IP地址](https://leetcode.cn/problems/restore-ip-addresses/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：回溯算法经典** + +> 将字符串分割成有效的IP地址：`"25525511135"` → `["255.255.11.135","255.255.111.35"]` + +**💡 核心思路**：回溯算法 + 剪枝 + +- 将字符串分成4段，每段代表IP的一个部分 +- 每段必须是0-255之间的有效数字 +- 不能有前导零（除了"0"本身） + +**🔑 记忆技巧**： + +- **口诀**："回溯分割四段IP，有效范围加剪枝" +- **形象记忆**：像切蛋糕一样，要切成4块合适大小的 + +```java +public List restoreIpAddresses(String s) { + List result = new ArrayList<>(); + if (s.length() < 4 || s.length() > 12) return result; + + backtrack(s, 0, new ArrayList<>(), result); + return result; +} + +private void backtrack(String s, int start, List path, List result) { + // 如果已经分成4段且用完所有字符 + if (path.size() == 4) { + if (start == s.length()) { + result.add(String.join(".", path)); + } + return; + } + + // 尝试不同长度的分割 + for (int len = 1; len <= 3 && start + len <= s.length(); len++) { + String segment = s.substring(start, start + len); + + // 检查是否为有效的IP段 + if (isValidSegment(segment)) { + path.add(segment); + backtrack(s, start + len, path, result); + path.remove(path.size() - 1); // 回溯 + } + } +} + +private boolean isValidSegment(String segment) { + // 检查长度 + if (segment.length() > 3) return false; + + // 检查前导零 + if (segment.length() > 1 && segment.charAt(0) == '0') return false; + + // 检查数值范围 + int num = Integer.parseInt(segment); + return num >= 0 && num <= 255; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(3^4) - 每段最多3种长度选择，共4段 +- **空间复杂度**：O(4) - 递归深度最多4层 + +### [20. 有效的括号](https://leetcode.cn/problems/valid-parentheses/) + +**🎯 考察频率：极高 | 难度：简单 | 重要性：栈的经典应用** + +> 判断括号字符串是否有效：`"()[]{}"` → `true`，`"([)]"` → `false` + +**💡 核心思路**：栈匹配 + +- 遇到左括号就入栈 +- 遇到右括号就检查栈顶是否为对应的左括号 +- 最后检查栈是否为空 + +**🔑 记忆技巧**： + +- **口诀**："左括号入栈，右括号配对，最后栈空才对" +- **形象记忆**：像穿衣服一样，先穿的后脱，后穿的先脱 + +```java +public boolean isValid(String s) { + Stack stack = new Stack<>(); + Map mapping = new HashMap<>(); + mapping.put(')', '('); + mapping.put('}', '{'); + mapping.put(']', '['); + + for (char c : s.toCharArray()) { + if (mapping.containsKey(c)) { + // 右括号 + if (stack.isEmpty() || stack.pop() != mapping.get(c)) { + return false; + } + } else { + // 左括号 + stack.push(c); + } + } + + return stack.isEmpty(); +} +``` + +**🔧 优化版本**： + +```java +public boolean isValid(String s) { + Stack stack = new Stack<>(); + + for (char c : s.toCharArray()) { + if (c == '(') { + stack.push(')'); + } else if (c == '{') { + stack.push('}'); + } else if (c == '[') { + stack.push(']'); + } else { + if (stack.isEmpty() || stack.pop() != c) { + return false; + } + } + } + + return stack.isEmpty(); +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(n) - 遍历字符串一次 +- **空间复杂度**：O(n) - 最坏情况栈存储所有左括号 + +### [17. 电话号码的字母组合](https://leetcode.cn/problems/letter-combinations-of-a-phone-number/) + +**🎯 考察频率：极高 | 难度：中等 | 重要性：回溯算法基础** + +> 给定数字字符串，返回所有可能的字母组合![img](https://assets.leetcode-cn.com/aliyun-lc-upload/uploads/2021/11/09/200px-telephone-keypad2svg.png) +> +> `"23"` → `["ad","ae","af","bd","be","bf","cd","ce","cf"]` + +**💡 核心思路**：回溯算法 + 映射表 + +- 建立数字到字母的映射关系 +- 对每个数字的每个字母进行递归组合 +- 到达字符串末尾时添加到结果中 + +**🔑 记忆技巧**： + +- **口诀**："数字映射字母表，回溯组合所有解" +- **形象记忆**：像老式手机键盘，每个数字对应几个字母 + +```java +public List letterCombinations(String digits) { + List result = new ArrayList<>(); + if (digits == null || digits.isEmpty()) return result; + + String[] mapping = { + "", // 0 + "", // 1 + "abc", // 2 + "def", // 3 + "ghi", // 4 + "jkl", // 5 + "mno", // 6 + "pqrs", // 7 + "tuv", // 8 + "wxyz" // 9 + }; + + backtrack(digits, 0, new StringBuilder(), result, mapping); + return result; +} + +private void backtrack(String digits, int index, StringBuilder path, + List result, String[] mapping) { + // 递归终止条件 + if (index == digits.length()) { + result.add(path.toString()); + return; + } + + // 获取当前数字对应的字母 + int digit = digits.charAt(index) - '0'; + String letters = mapping[digit]; + + // 尝试每个字母 + for (char letter : letters.toCharArray()) { + path.append(letter); + backtrack(digits, index + 1, path, result, mapping); + path.deleteCharAt(path.length() - 1); // 回溯 + } +} +``` + +**🔧 迭代解法**： + +```java +public List letterCombinations(String digits) { + if (digits == null || digits.isEmpty()) return new ArrayList<>(); + + String[] mapping = {"", "", "abc", "def", "ghi", "jkl", "mno", "pqrs", "tuv", "wxyz"}; + List result = new ArrayList<>(); + result.add(""); + + for (char digit : digits.toCharArray()) { + List temp = new ArrayList<>(); + String letters = mapping[digit - '0']; + + for (String combination : result) { + for (char letter : letters.toCharArray()) { + temp.add(combination + letter); + } + } + + result = temp; + } + + return result; +} +``` + +**⏱️ 复杂度分析**： + +- **时间复杂度**：O(3^m × 4^n) - m为对应3个字母的数字个数，n为对应4个字母的数字个数 +- **空间复杂度**：O(3^m × 4^n) - 存储所有组合结果 + +--- + +## 🏆 面试前15分钟速记表 + +| 题型分类 | 核心技巧 | 高频题目 | 记忆口诀 | 难度 | +| -------------- | -------- | ------------------------------------------------------------ | ------------------------------ | ----- | +| **双指针基础** | 对撞指针 | 验证回文串、反转字符串、反转字符串中的单词、最长回文子串 | 对撞指针找回文，快慢指针找重复 | ⭐⭐⭐ | +| **滑动窗口** | 动态窗口 | 无重复字符的最长子串、最小覆盖子串、找到字符串中所有字母异位词 | 左右指针动态调，窗口大小随需要 | ⭐⭐⭐⭐ | +| **字符统计** | 哈希计数 | 有效的字母异位词、字母异位词分组、赎金信、第一个不重复的字符 | 哈希计数是法宝，频次统计用得妙 | ⭐⭐⭐ | +| **模式匹配** | KMP算法 | 实现strStr()、重复的子字符串、字符串匹配、正则表达式匹配 | KMP算法巧匹配，前缀表来帮助你 | ⭐⭐⭐⭐ | +| **回文专题** | 中心扩展 | 回文子串、最长回文子串、回文链表、分割回文串 | 中心扩展找回文，奇偶长度都考虑 | ⭐⭐⭐⭐ | +| **字符串DP** | 状态转移 | 最长公共子序列、编辑距离、不同的子序列、交错字符串 | 状态转移要清晰，边界条件别忘记 | ⭐⭐⭐⭐⭐ | +| **字符串变换** | 边界处理 | 整数转换、字符串转换整数、字符串相加、字符串相乘 | 原地修改要小心，额外空间换时间 | ⭐⭐⭐⭐ | +| **进阶技巧** | 高效算法 | 最短回文串、复原IP地址、有效的括号、电话号码的字母组合、串联所有单词的子串 | 回溯分割巧组合，栈匹配括号对 | ⭐⭐⭐⭐⭐ | + +### 按难度分级 + +- **⭐⭐⭐ 简单必会**：验证回文串、反转字符串、有效的字母异位词、字母异位词分组、有效的括号 +- **⭐⭐⭐⭐ 中等重点**：无重复字符的最长子串、最长回文子串、实现strStr()、回文子串、最长公共子序列、字符串转换整数、复原IP地址、电话号码的字母组合 +- **⭐⭐⭐⭐⭐ 困难经典**：最小覆盖子串、编辑距离、正则表达式匹配、最短回文串 + +### 热题100核心优先级 + +1. **无重复字符的最长子串** - 滑动窗口模板 +2. **最长回文子串** - 中心扩展法 +3. **有效的字母异位词** - 字符统计基础 +4. **字母异位词分组** - 哈希表分组 +5. **验证回文串** - 双指针基础 +6. **反转字符串** - 原地操作 +7. **有效的括号** - 栈的经典应用 +8. **电话号码的字母组合** - 回溯算法入门 +9. **复原IP地址** - 回溯算法进阶 +10. **最小覆盖子串** - 滑动窗口进阶 + +### 热题100题目统计 + +- **总题目数**：48+道热题100字符串题目 +- **新增重要题目**：LC93复原IP地址、LC20有效的括号、LC17电话号码的字母组合 +- **覆盖率**：100%覆盖热题100中的字符串相关题目，并补充了重要的算法基础题 +- **核心算法**：双指针、滑动窗口、哈希表、KMP、动态规划、中心扩展、回溯算法、栈 + +### 常见陷阱提醒 + +- ⚠️ **空字符串**：`s == null || s.isEmpty()` 的边界处理 +- ⚠️ **字符大小写**：题目要求是否区分大小写 +- ⚠️ **索引越界**：`i >= 0 && i < s.length()` 边界检查 +- ⚠️ **字符编码**：ASCII vs Unicode，字符范围 +- ⚠️ **整数溢出**：字符串转数字时的溢出处理 +- ⚠️ **特殊字符**：空格、标点符号的处理策略 + +### 解题步骤提醒 + +1. **理解题意**：明确输入输出格式和边界条件 +2. **选择算法**：根据题目特点选择合适的算法 +3. **处理边界**：考虑空串、单字符等特殊情况 +4. **优化空间**：考虑是否可以原地操作或滚动数组 +5. **测试验证**：用示例和边界用例验证正确性 + +--- + +*🎯 备注：本总结涵盖了字符串算法的核心知识点，建议配合实际编程练习，重点掌握双指针、滑动窗口、KMP算法和字符串动态规划。* diff --git a/docs/data-structure-algorithms/soultion/stock-problems.md b/docs/data-structure-algorithms/soultion/stock-problems.md new file mode 100644 index 0000000000..b75e7106d2 --- /dev/null +++ b/docs/data-structure-algorithms/soultion/stock-problems.md @@ -0,0 +1,443 @@ +--- +title: 股票买卖问题-套路解题 +date: 2022-03-09 +tags: + - DP +categories: leetcode +aliases: + - DP-Solution +--- + +![](https://img.starfish.ink/leetcode/stock-profit-banner.png) + +> 刷 labuladong-东哥 的文章时，发现这么宝藏的一篇，真是一个套路解决所有股票问题（文章主要来自英文版 leetcode 的题解翻译） +> +> - https://labuladong.gitee.io/algo/1/12/ +> +> - https://leetcode.com/problems/best-time-to-buy-and-sell-stock-with-transaction-fee/discuss/108870/Most-consistent-ways-of-dealing-with-the-series-of-stock-problems + + + +leetcode 的股票问题目前总共有这么 6 题 + +- [121. 买卖股票的最佳时机（简单）](https://leetcode-cn.com/problems/best-time-to-buy-and-sell-stock/) + +- [122. 买卖股票的最佳时机 II（简单）](https://leetcode-cn.com/problems/best-time-to-buy-and-sell-stock-ii/) + +- [123. 买卖股票的最佳时机 III（困难）](https://leetcode-cn.com/problems/best-time-to-buy-and-sell-stock-iii/) + +- [188. 买卖股票的最佳时机 IV（困难）](https://leetcode-cn.com/problems/best-time-to-buy-and-sell-stock-iv/) + +- [309. 最佳买卖股票时机含冷冻期（中等）](https://leetcode-cn.com/problems/best-time-to-buy-and-sell-stock-with-cooldown/) + +- [714. 买卖股票的最佳时机含手续费（中等）](https://leetcode-cn.com/problems/best-time-to-buy-and-sell-stock-with-transaction-fee/) + + + +不管是按热题去刷还是按 DP 分类去刷，这几道题都是避不开的，本文介绍的方法算是一个极其通用的“公式”，我们只需要搞清楚每个题目的“特例”后，套“公式”真的可以闭眼写出来，废话不多说了，直接解释 + +- prices[i] 存储股票第 i 天的价格，长度为 `prices.length` + +- k 表示最大交易次数 + +- `dp[i][k]` 表示第 i 天，k 次交易的最大利润 + +- 特例：`dp[-1][k] = dp[i][0] = 0 `（没有股票或没有交易，就没有利润） + +我们可以有多少种操作呢？ 答案是三种：buy, sell, rest，分别表示买入、卖出、不操作（休息） + +那今天到底是哪个操作会是最大利润呢，求最值问题，想到用动态规划、max 函数 + +这里有一个隐藏的条件是，就是这三种操作有个先后顺序： + +- buy 必须是在没有持仓的情况下 +- sell 必须是在手里持仓为 1 的情况 + +![](https://img.starfish.ink/leetcode/stock-maxProfit.png) + +所以 `dp[i][k]` 可以拆成两部分，用三维数组来表示：`dp[i][k][0]` 和 `dp[i][k][1]` ，意思是手里没有股票和手里有股票的最大利润 + +先考虑特例： + +```java +dp[-1][k][0] = 0, dp[-1][k][1] = -Infinity +dp[i][0][0] = 0, dp[i][0][1] = -Infinity +``` + +> `dp[-1][k][0] = dp[i][0][0] = 0` 代表没有股票或者没有交易，又不持仓，肯定是没有利润 +> +> `dp[-1][k][1] = dp[i][0][1] = -Infinity` 没有股票或者没有交易，不可能持仓吧? + +递推关系（也就是 DP 中的状态转移方程） + +```java +dp[i][k][0] = max(dp[i-1][k][0],dp[i-1][k][1] + prices[i]); +dp[i][k][1] = max(dp[i-1][k][1],dp[i-1][k-1][0] - prices[i]); +``` + +> `dp[i][k][0]`，今天的持仓是 0 ，所以今天不能是买入，只能是卖出或者不操作 取最大值： +> +> - 昨天就不持有了，也就是 `dp[i-1][k][0]`，我今天才可能持仓为 0 +> - 不管是昨天卖了还是昨天没动都是不持有，如果昨天是卖了的话，为什么不是 k-1 呢，因为一次交易对应的是一对操作，买入 + 卖出，只有买入算是开启一次新交易，才会改变最大交易次数 +> +> - 今天卖出的话，昨天就必须持有才行，`dp[i-1][k][1]`，为什么还有个 `+ prices[i]`呢，最开始我有点迷糊，这怎么利润还算上股价了，这里是这么算的，我们第一天买入的话，利润就是 `-prices[i]` 了，所以最后如果卖出的话，要加上今日股价 +> +> `dp[i][k][1]`，今天的持仓是 1 ，所以今天不能卖出，只能是买入或者昨天就持有 取最大值： +> +> - 昨天就持有，最大交易次数 k，也就是 `dp[i-1][k][1]` +> - 今天买入的话，昨天就不能持有，而且要满足今日最大交易次数是 k 的限制，所以昨天是 k-1 次交易，也就是 `dp[i-1][k-1][0]`，再减去今日股价 + + + +### [121. 买卖股票的最佳时机 | k = 1](https://leetcode-cn.com/problems/best-time-to-buy-and-sell-stock/) + +> 给定一个数组 prices ，它的第 i 个元素 prices[i] 表示一支给定股票第 i 天的价格。 +> +> 你只能选择 某一天 买入这只股票，并选择在 未来的某一个不同的日子 卖出该股票。设计一个算法来计算你所能获取的最大利润。 +> +> 返回你可以从这笔交易中获取的最大利润。如果你不能获取任何利润，返回 0 。 +> +> ``` +> 输入：[7,1,5,3,6,4] +> 输出：5 +> 解释：在第 2 天（股票价格 = 1）的时候买入，在第 5 天（股票价格 = 6）的时候卖出，最大利润 = 6-1 = 5 。 +> 注意利润不能是 7-1 = 6, 因为卖出价格需要大于买入价格；同时，你不能在买入前卖出股票。 +> ``` + +Case 1，k = 1 + +``` +dp[i][1][0] = max(dp[i-1][1][0], dp[i-1][1][1] + prices[i]) +dp[i][1][1] = max(dp[i-1][1][1], dp[i-1][0][0] - prices[i]) + = max(dp[i-1][1][1], -prices[i]) +解释：k = 0 的 base case，所以 dp[i-1][0][0] = 0。 + +现在发现 k 都是 1，不会改变，即 k 对状态转移已经没有影响了。 +可以进行进一步化简去掉所有 k： +dp[i][0] = max(dp[i-1][0], dp[i-1][1] + prices[i]) +dp[i][1] = max(dp[i-1][1], -prices[i]) +``` + +套用公式，写出代码： + +```java +// 原始版本 +int maxProfit_k_1(int[] prices) { + int n = prices.length; + int[][] dp = new int[n][2]; + //特例 + dp[0][0] = 0; + dp[0][1] = -prices[0]; + + for(int i = 1;i 给定一个数组 prices ，其中 prices[i] 表示股票第 i 天的价格。 +> +> 在每一天，你可能会决定购买和/或出售股票。你在任何时候 最多 只能持有 一股 股票。你也可以购买它，然后在 同一天 出售。 +> 返回 你能获得的 最大 利润 。 +> +> ```java +> 输入: prices = [7,1,5,3,6,4] +> 输出: 7 +> 解释: 在第 2 天（股票价格 = 1）的时候买入，在第 3 天（股票价格 = 5）的时候卖出, 这笔交易所能获得利润 = 5-1 = 4 。 +>   随后，在第 4 天（股票价格 = 3）的时候买入，在第 5 天（股票价格 = 6）的时候卖出, 这笔交易所能获得利润 = 6-3 = 3 。 +> ``` + +题目中说可以随便交易，也就是 k 不限制了，当天买了当天就能卖（T+0），假设 k 无穷大，那就可以认为 k 和 k-1 是一样的，套公式 + +```java +dp[i][k][0] = max(dp[i-1][k][0], dp[i-1][k][1] + prices[i]) +dp[i][k][1] = max(dp[i-1][k][1], dp[i-1][k-1][0] - prices[i]) = max(dp[i-1][k][1], dp[i-1][k][0] - prices[i]) + +k 和 k-1 相同时，发现数组中的 k 其实没有变化，也就可以不记录 k 这个状态了 +dp[i][0] = max(dp[i-1][0], dp[i-1][1] + prices[i]) +dp[i][1] = max(dp[i-1][1], dp[i-1][0] - prices[i]) +``` + +直接上代码： + +```java +// 原始版本 +int maxProfit_k_inf(int[] prices) { + int n = prices.length; + int[][] dp = new int[n][2]; + dp[0][0] = 0; + //0天，利润是 -prices[0]，所以后边今日卖出的话要 +prices[i] + dp[0][1] = -prices[0]; + for (int i = 1; i < n; i++) { + dp[i][0] = Math.max(dp[i-1][0], dp[i-1][1] + prices[i]); + // 这一步是和k=1 不一样的地方 + dp[i][1] = Math.max(dp[i-1][1], dp[i-1][0] - prices[i]); + } + return dp[n - 1][0]; +} + +// 空间复杂度优化版本 +//每一天的状态只与前一天的状态有关，而与更早的状态都无关，因此我们不必存储这些无关的状态， +//只需要将 dp[i-1][0] 和 dp[i−1][1] 存放在两个变量中 +int maxProfit_k_inf(int[] prices) { + int n = prices.length; + int dp0 = 0, dp1 = -prices[0]; + for (int i = 0; i < n; i++) { + dp0 = Math.max(dp0, dp1 + prices[i]); + dp1 = Math.max(dp1, dp0 - prices[i]); + } + return dp0; +} +``` + + + +### [123. 买卖股票的最佳时机 III](https://leetcode-cn.com/problems/best-time-to-buy-and-sell-stock-iii/) | k = 2 + +> 给定一个数组，它的第 i 个元素是一支给定的股票在第 i 天的价格。 +> +> 设计一个算法来计算你所能获取的最大利润。你最多可以完成 两笔 交易。 +> +> 注意：你不能同时参与多笔交易（你必须在再次购买前出售掉之前的股票）。 +> +> ``` +> 输入：prices = [3,3,5,0,0,3,1,4] +> 输出：6 +> 解释：在第 4 天（股票价格 = 0）的时候买入，在第 6 天（股票价格 = 3）的时候卖出，这笔交易所能获得利润 = 3-0 = 3 。 +>   随后，在第 7 天（股票价格 = 1）的时候买入，在第 8 天 （股票价格 = 4）的时候卖出，这笔交易所能获得利润 = 4-1 = 3 。 +> ``` + +这道题是 k = 2，前两道我们都消除了 k ，换成了二维数组 + +这道题，我们不能消除 k，而且题目说最多两次交易，那我们要穷举出 1 次交易 和 2 次交易的结果，才能知道哪种收益最高 + +```java +public static int getMaxProfit(int[] prices){ + int max_k = 2; + int n = prices.length; + int dp[][][] = new int[n][3][2]; + + //特例 + // dp[0][1][0] = 0; + // dp[0][1][1] = - prices[0]; + // dp[0][2][0] = 0; + // dp[0][2][1] = - prices[0]; + + for (int i = 0; i < prices.length; i++) { + for (int k = 1; k <= max_k; k++) { + //特例 + if (i == 0) { + dp[0][k][0] = 0; + dp[0][k][1] = -prices[i]; + continue; + } + + dp[i][k][0] = Math.max(dp[i - 1][k][0], dp[i - 1][k][1] + prices[i]); + dp[i][k][1] = Math.max(dp[i - 1][k][1], dp[i - 1][k - 1][0] - prices[i]); + } + } + return dp[n - 1][max_k][0]; +} +``` + + + +### [188. 买卖股票的最佳时机 IV](https://leetcode-cn.com/problems/best-time-to-buy-and-sell-stock-iv/) | k 不限制 + +> 给定一个整数数组 prices ，它的第 i 个元素 prices[i] 是一支给定的股票在第 i 天的价格。 +> +> 设计一个算法来计算你所能获取的最大利润。你最多可以完成 k 笔交易。 +> +> 注意：你不能同时参与多笔交易（你必须在再次购买前出售掉之前的股票）。 +> +> ``` +> 输入：k = 2, prices = [3,2,6,5,0,3] +> 输出：7 +> 解释：在第 2 天 (股票价格 = 2) 的时候买入，在第 3 天 (股票价格 = 6) 的时候卖出, 这笔交易所能获得利润 = 6-2 = 4 。 +> 随后，在第 5 天 (股票价格 = 0) 的时候买入，在第 6 天 (股票价格 = 3) 的时候卖出, 这笔交易所能获得利润 = 3-0 = 3 。 +> ``` + +这个题目的 k 看似也是不限制，和 122 有点像，我们处理 122 的时候，把 k 看做无穷大，直接过滤了，我们看作是 T+0 交易，这道题是有限制的，不能同时参与多笔交易，其实就是 T+1 + +一次交易由买入和卖出构成，至少需要两天。如果 prices 数组长度 n，那其实买卖最多 n/2 次，也就是 k <= n/2 + +```java +public static int getMaxProfit(int max_k, int[] prices) { + int n = prices.length; + + if (n <= 1) { + return 0; + } + + //因为一次交易至少涉及两天，所以如果k大于总天数的一半，就直接取天数一半即可，多余的交易次数是无意义的 + max_k = Math.min(max_k, n / 2); + + int[][][] dp = new int[n][max_k + 1][2]; + //特例:第1天，不持有随便交易 0，持有的话就是 -prices[0] + for (int k = 0; k <= max_k; k++) { + dp[0][k][0] = 0; + dp[0][k][1] = -prices[0]; + } + + for (int i = 1; i < n; i++) { + for (int k = 1; k <= max_k; k++) { + dp[i][k][0] = Math.max(dp[i - 1][k][0], dp[i - 1][k][1] + prices[i]); + dp[i][k][1] = Math.max(dp[i - 1][k][1], dp[i - 1][k - 1][0] - prices[i]); + } + } + return dp[n - 1][max_k][0]; +} + +``` + + + +### [309. 最佳买卖股票时机含冷冻期](https://leetcode-cn.com/problems/best-time-to-buy-and-sell-stock-with-cooldown/) + +> 给定一个整数数组prices，其中第 prices[i] 表示第 i 天的股票价格 。 +> +> 设计一个算法计算出最大利润。在满足以下约束条件下，你可以尽可能地完成更多的交易（多次买卖一支股票）: +> +> 卖出股票后，你无法在第二天买入股票 (即冷冻期为 1 天)。 +> 注意：你不能同时参与多笔交易（你必须在再次购买前出售掉之前的股票）。 +> +> ``` +> 输入: prices = [1,2,3,0,2] +> 输出: 3 +> 解释: 对应的交易状态为: [买入, 卖出, 冷冻期, 买入, 卖出] +> ``` + +这里加了一个冷冻期的概念，sell 后不能 直接 buy，需要冷静冷静，也是不限次数 k，思路和 122 一样，加一个冷冻期的状态 + +拿出我们的万能公式再看一眼 + +``` +dp[i][k][0] = max(dp[i-1][k][0],dp[i-1][k][1] + prices[i]); +dp[i][k][1] = max(dp[i-1][k][1],dp[i-1][k-1][0] - prices[i]); +``` + + 因为有冷冻期的存在，如果我们不能在 i-1 天买入，要 buy 的话，必须等 1 天，也就是持仓公式中 `dp[i-1][k-1][0]` 买入的话，需要再往前1 天才能买入，即 `dp[i-2][k-1][0]` + +``` +dp[i][k][1] = max(dp[i-1][k][1],dp[i-2][k-1][0] - prices[i]); +``` + +剩下得就是套公式，和 122 一样了 + +```java +public static int maxProfit(int[] prices) { + int len = prices.length; + if (len < 2) { + return 0; + } + //特例 + int[][] dp = new int[len][2]; + dp[0][0] = 0; + dp[0][1] = -prices[0]; + for (int i = 1; i < len; i++) { + // 卖出状态 + dp[i][0] = Math.max(dp[i - 1][0], dp[i - 1][1] + prices[i]); + // 买入状态，比122 多加了一个if-else + if (i < 2) { + // 前三天不用考虑冷冻期的问题，因为不可能出现冷冻期 + dp[i][1] = Math.max(dp[i - 1][1], dp[i - 1][0] - prices[i]); + } else { + // 从第4天开始，买入考虑一天的冷冻期 + dp[i][1] = Math.max(dp[i - 1][1], dp[i - 2][0] - prices[i]); + } + } + return dp[len - 1][0]; +} +``` + + + +### [714. 买卖股票的最佳时机含手续费](https://leetcode-cn.com/problems/best-time-to-buy-and-sell-stock-with-transaction-fee/) + +> 给定一个整数数组 prices，其中 prices[i] 表示第 i 天的股票价格 ；整数 fee 代表了交易股票的手续费用。 +> +> 你可以无限次地完成交易，但是你每笔交易都需要付手续费。如果你已经购买了一个股票，在卖出它之前你就不能再继续购买股票了。 +> +> 返回获得利润的最大值。 +> +> 注意：这里的一笔交易指买入持有并卖出股票的整个过程，每笔交易你只需要为支付一次手续费。 +> +> ``` +> 输入：prices = [1, 3, 2, 8, 4, 9], fee = 2 +> 输出：8 +> 解释：能够达到的最大利润: +> 在此处买入 prices[0] = 1 +> 在此处卖出 prices[3] = 8 +> 在此处买入 prices[4] = 4 +> 在此处卖出 prices[5] = 9 +> 总利润: ((8 - 1) - 2) + ((9 - 4) - 2) = 8 +> ``` + +这个题，其实又是和 122 类似，k 不限制，只是多了一个手续费，这个手续费我们可以选择在买入时候交，也可以选择再卖出时候交，那通用公式就可以变成 + +``` +dp[i][k][0] = max(dp[i-1][k][0],dp[i-1][k][1] + prices[i]); +dp[i][k][1] = max(dp[i-1][k][1],dp[i-1][k-1][0] - prices[i] - free); //买入时候交 +``` + +或者卖出时候交 + +``` +dp[i][k][0] = max(dp[i-1][k][0],dp[i-1][k][1] + prices[i] - free); +dp[i][k][1] = max(dp[i-1][k][1],dp[i-1][k-1][0] - prices[i]); //买入时候交 +``` + +直接上代码吧 + +```java +public static int getMaxProfit(int[] prices, int fee) { + int n = prices.length; + int[][] dp = new int[n][2]; + + dp[0][0] = 0; + dp[0][1] = -prices[0]; + + for (int i = 1; i < n; i++) { + dp[i][0] = Math.max(dp[i - 1][0], dp[i - 1][1] + prices[i] - fee); + dp[i][1] = Math.max(dp[i - 1][1], dp[i - 1][0] - prices[i]); + } + return dp[n - 1][0]; +} +``` + diff --git "a/docs/data-structure-algorithms/soultion/\345\211\221\346\214\207offer.md" "b/docs/data-structure-algorithms/soultion/\345\211\221\346\214\207offer.md" new file mode 100755 index 0000000000..351f9e932a --- /dev/null +++ "b/docs/data-structure-algorithms/soultion/\345\211\221\346\214\207offer.md" @@ -0,0 +1,854 @@ +# 剑指Offer 热门题目汇总 + +> **导读**：剑指Offer是面试中的经典题目集合，涵盖了数据结构与算法的各个重要知识点。这些题目不仅考查基础知识，更注重解题思路和代码实现的优雅性。 +> +> **关键词**：数组、字符串、链表、二叉树、栈队列、动态规划、搜索回溯 + +## 数组相关 + +### [剑指 Offer 03. 数组中重复的数字](https://leetcode-cn.com/problems/shu-zu-zhong-zhong-fu-de-shu-zi-lcof/) + +**🎯 考察频率：高 | 难度：简单 | 重要性：数组基础** + +> 找出数组中重复的数字。在一个长度为 n 的数组 nums 里的所有数字都在 0～n-1 的范围内。 +> +> ``` +> 输入：[2, 3, 1, 0, 2, 5, 3] +> 输出：2 或 3 +> ``` + +**💡 核心思路**：利用数组索引，原地交换 + +由于数组中的数字都在0～n-1范围内，可以将每个数字放到对应的索引位置上。当发现某个位置上已经有正确的数字时，说明找到了重复数字。 + +```java +public int findRepeatNumber(int[] nums) { + int i = 0; + while (i < nums.length) { + // 如果当前位置的数字等于索引，继续下一个 + if (nums[i] == i) { + i++; + continue; + } + // 如果nums[i]位置上已经有正确的数字，找到重复 + if (nums[nums[i]] == nums[i]) { + return nums[i]; + } + // 否则交换到正确位置 + int temp = nums[i]; + nums[i] = nums[temp]; + nums[temp] = temp; + } + return -1; +} +``` + +**🔍 复杂度分析**： +- 时间复杂度：O(n) +- 空间复杂度：O(1) + +### [剑指 Offer 04. 二维数组中的查找](https://leetcode-cn.com/problems/er-wei-shu-zu-zhong-de-cha-zhao-lcof/) + +**🎯 考察频率：极高 | 难度：中等 | 重要性：二维数组经典** + +> 在一个 n * m 的二维数组中，每一行都按照从左到右递增的顺序排序，每一列都按照从上到下递增的顺序排序。 +> +> ``` +> 现有矩阵 matrix 如下： +> [ +> [1, 4, 7, 11, 15], +> [2, 5, 8, 12, 19], +> [3, 6, 9, 16, 22], +> [10, 13, 14, 17, 24], +> [18, 21, 23, 26, 30] +> ] +> 给定 target = 5，返回 true。 +> ``` + +**💡 核心思路**：从右上角开始搜索 + +利用矩阵的有序性质，从右上角开始： +- 如果当前元素等于target，找到答案 +- 如果当前元素大于target，向左移动（排除当前列） +- 如果当前元素小于target，向下移动（排除当前行） + +```java +public boolean findNumberIn2DArray(int[][] matrix, int target) { + if (matrix == null || matrix.length == 0 || matrix[0].length == 0) { + return false; + } + + int rows = matrix.length; + int cols = matrix[0].length; + + // 从右上角开始 + int row = 0; + int col = cols - 1; + + while (row < rows && col >= 0) { + if (matrix[row][col] == target) { + return true; + } else if (matrix[row][col] > target) { + col--; // 当前数字大于target，排除当前列 + } else { + row++; // 当前数字小于target，排除当前行 + } + } + + return false; +} +``` + +**🔍 复杂度分析**： +- 时间复杂度：O(m + n) +- 空间复杂度：O(1) + +### [剑指 Offer 45. 把数组排成最小的数](https://leetcode-cn.com/problems/ba-shu-zu-pai-cheng-zui-xiao-de-shu-lcof/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：自定义排序** + +> 输入一个非负整数数组，把数组里所有数字拼接起来排成一个数，打印能拼接出的所有数字中最小的一个。 +> +> ``` +> 输入: [10,2] +> 输出: "102" +> ``` +> +> ``` +> 输入: [3,30,34,5,9] +> 输出: "3033459" +> ``` + +**💡 核心思路**：自定义排序规则 + +关键在于定义比较规则：对于两个数字a和b，如果拼接后ab < ba，则a应该排在b前面。 + +```java +public String minNumber(int[] nums) { + String[] strs = new String[nums.length]; + for (int i = 0; i < nums.length; i++) { + strs[i] = String.valueOf(nums[i]); + } + + // 自定义排序规则 + Arrays.sort(strs, (x, y) -> (x + y).compareTo(y + x)); + + StringBuilder res = new StringBuilder(); + for (String s : strs) { + res.append(s); + } + + return res.toString(); +} +``` + +**🔍 复杂度分析**： +- 时间复杂度：O(n log n)，排序的时间复杂度 +- 空间复杂度：O(n)，存储字符串数组 + +## 字符串相关 + +### [剑指 Offer 05. 替换空格](https://leetcode-cn.com/problems/ti-huan-kong-ge-lcof/) + +**🎯 考察频率：中等 | 难度：简单 | 重要性：字符串基础** + +> 请实现一个函数，把字符串 s 中的每个空格替换成"%20"。 +> +> ``` +> 输入：s = "We are happy." +> 输出："We%20are%20happy." +> ``` + +**💡 核心思路**：StringBuilder或字符数组 + +```java +public String replaceSpace(String s) { + StringBuilder result = new StringBuilder(); + for (char c : s.toCharArray()) { + if (c == ' ') { + result.append("%20"); + } else { + result.append(c); + } + } + return result.toString(); +} +``` + +**🔍 复杂度分析**： +- 时间复杂度：O(n) +- 空间复杂度：O(n) + +### [剑指 Offer 58. 左旋转字符串](https://leetcode-cn.com/problems/zuo-xuan-zhuan-zi-fu-chuan-lcof/) + +**🎯 考察频率：中等 | 难度：简单 | 重要性：字符串操作** + +> 字符串的左旋转操作是把字符串前面的若干个字符转移到字符串的尾部。 +> +> ``` +> 输入: s = "abcdefg", k = 2 +> 输出: "cdefgab" +> ``` + +**💡 核心思路**：字符串拼接 + +```java +public String reverseLeftWords(String s, int n) { + return s.substring(n) + s.substring(0, n); +} +``` + +**🔍 复杂度分析**： +- 时间复杂度：O(n) +- 空间复杂度：O(n) + +## 链表相关 + +### [剑指 Offer 06. 从尾到头打印链表](https://leetcode-cn.com/problems/cong-wei-dao-tou-da-yin-lian-biao-lcof/) + +**🎯 考察频率：高 | 难度：简单 | 重要性：链表遍历** + +> 输入一个链表的头节点，从尾到头反过来返回每个节点的值（用数组返回）。 +> +> ``` +> 输入：head = [1,3,2] +> 输出：[2,3,1] +> ``` + +**💡 核心思路**：栈或递归 + +```java +// 方法1：使用栈 +public int[] reversePrint(ListNode head) { + Stack stack = new Stack<>(); + ListNode cur = head; + + // 遍历链表，入栈 + while (cur != null) { + stack.push(cur.val); + cur = cur.next; + } + + // 出栈到数组 + int[] result = new int[stack.size()]; + for (int i = 0; i < result.length; i++) { + result[i] = stack.pop(); + } + + return result; +} + +// 方法2：递归 +List tmp = new ArrayList<>(); +public int[] reversePrint(ListNode head) { + recur(head); + int[] res = new int[tmp.size()]; + for (int i = 0; i < res.length; i++) { + res[i] = tmp.get(i); + } + return res; +} + +void recur(ListNode head) { + if (head == null) return; + recur(head.next); + tmp.add(head.val); +} +``` + +**🔍 复杂度分析**： +- 时间复杂度：O(n) +- 空间复杂度：O(n) + +### [剑指 Offer 18. 删除链表的节点](https://leetcode-cn.com/problems/shan-chu-lian-biao-de-jie-dian-lcof/) + +**🎯 考察频率：高 | 难度：简单 | 重要性：链表操作** + +> 给定单向链表的头指针和一个要删除的节点的值，定义一个函数删除该节点。 +> +> ``` +> 输入: head = [4,5,1,9], val = 5 +> 输出: [4,1,9] +> ``` + +**💡 核心思路**：双指针，注意删除头节点的特殊情况 + +```java +public ListNode deleteNode(ListNode head, int val) { + // 删除头节点 + if (head.val == val) return head.next; + + ListNode pre = head, cur = head.next; + while (cur != null && cur.val != val) { + pre = cur; + cur = cur.next; + } + + if (cur != null) { + pre.next = cur.next; + } + + return head; +} +``` + +**🔍 复杂度分析**： +- 时间复杂度：O(n) +- 空间复杂度：O(1) + +### [剑指 Offer 22. 链表中倒数第k个节点](https://leetcode-cn.com/problems/lian-biao-zhong-dao-shu-di-kge-jie-dian-lcof/) + +**🎯 考察频率：极高 | 难度：简单 | 重要性：双指针经典** + +> 输入一个链表，输出该链表中倒数第k个节点。 +> +> ``` +> 给定一个链表: 1->2->3->4->5, 和 k = 2. +> 返回链表 4->5. +> ``` + +**💡 核心思路**：双指针，快慢指针 + +快指针先走k步，然后快慢指针同时移动，当快指针到达末尾时，慢指针正好指向倒数第k个节点。 + +```java +public ListNode getKthFromEnd(ListNode head, int k) { + ListNode former = head, latter = head; + + // 快指针先走k步 + for (int i = 0; i < k; i++) { + former = former.next; + } + + // 快慢指针同时移动 + while (former != null) { + former = former.next; + latter = latter.next; + } + + return latter; +} +``` + +**🔍 复杂度分析**： +- 时间复杂度：O(n) +- 空间复杂度：O(1) + +## 二叉树相关 + +### [剑指 Offer 07. 重建二叉树](https://leetcode-cn.com/problems/zhong-jian-er-cha-shu-lcof/) + +**🎯 考察频率：极高 | 难度：中等 | 重要性：树的构建** + +> 输入某二叉树的前序遍历和中序遍历的结果，请重建该二叉树。假设输入的前序遍历和中序遍历的结果中都不含重复的数字。 +> +> ``` +> 前序遍历 preorder = [3,9,20,15,7] +> 中序遍历 inorder = [9,3,15,20,7] +> +> 返回如下的二叉树： +> 3 +> / \ +> 9 20 +> / \ +> 15 7 +> ``` + +**💡 核心思路**：递归 + 哈希表优化 + +前序遍历的第一个节点是根节点，在中序遍历中找到这个根节点，就能确定左右子树。 + +```java +Map indexMap; + +public TreeNode buildTree(int[] preorder, int[] inorder) { + int n = preorder.length; + // 构造哈希映射快速定位根节点 + indexMap = new HashMap<>(); + for (int i = 0; i < n; i++) { + indexMap.put(inorder[i], i); + } + return myBuildTree(preorder, inorder, 0, n - 1, 0, n - 1); +} + +public TreeNode myBuildTree(int[] preorder, int[] inorder, + int preorder_left, int preorder_right, + int inorder_left, int inorder_right) { + if (preorder_left > preorder_right) { + return null; + } + + // 前序遍历中的第一个节点就是根节点 + int preorder_root = preorder_left; + // 在中序遍历中定位根节点 + int inorder_root = indexMap.get(preorder[preorder_root]); + + // 先把根节点建立出来 + TreeNode root = new TreeNode(preorder[preorder_root]); + // 得到左子树中的节点数目 + int size_left_subtree = inorder_root - inorder_left; + + // 递归地构造左子树，并连接到根节点 + root.left = myBuildTree(preorder, inorder, + preorder_left + 1, + preorder_left + size_left_subtree, + inorder_left, + inorder_root - 1); + + // 递归地构造右子树，并连接到根节点 + root.right = myBuildTree(preorder, inorder, + preorder_left + size_left_subtree + 1, + preorder_right, + inorder_root + 1, + inorder_right); + + return root; +} +``` + +**🔍 复杂度分析**： +- 时间复杂度：O(n) +- 空间复杂度：O(n) + +### [剑指 Offer 26. 树的子结构](https://leetcode-cn.com/problems/shu-de-zi-jie-gou-lcof/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：树的遍历** + +> 输入两棵二叉树A和B，判断B是不是A的子结构。 +> +> ``` +> 给定的树 A: +> 3 +> / \ +> 4 5 +> / \ +> 1 2 +> +> 给定的树 B: +> 4 +> / +> 1 +> +> 返回 true，因为 B 与 A 的一个子树拥有相同的结构和节点值。 +> ``` + +**💡 核心思路**：递归遍历 + 匹配判断 + +先在A中找到与B根节点相同的节点，然后判断以该节点为根的子树是否与B相同。 + +```java +public boolean isSubStructure(TreeNode A, TreeNode B) { + return (A != null && B != null) && (recur(A, B) || isSubStructure(A.left, B) || isSubStructure(A.right, B)); +} + +boolean recur(TreeNode A, TreeNode B) { + if (B == null) return true; + if (A == null || A.val != B.val) return false; + return recur(A.left, B.left) && recur(A.right, B.right); +} +``` + +**🔍 复杂度分析**： +- 时间复杂度：O(MN)，M、N分别为树A和树B的节点数量 +- 空间复杂度：O(M) + +### [剑指 Offer 32. 从上到下打印二叉树](https://leetcode-cn.com/problems/cong-shang-dao-xia-da-yin-er-cha-shu-lcof/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：BFS经典** + +> 从上到下打印出二叉树的每个节点，同一层的节点按照从左到右的顺序打印。 +> +> ``` +> 给定二叉树: [3,9,20,null,null,15,7], +> 3 +> / \ +> 9 20 +> / \ +> 15 7 +> +> 返回：[3,9,20,15,7] +> ``` + +**💡 核心思路**：广度优先遍历（BFS） + +使用队列实现BFS，逐层遍历二叉树。 + +```java +public int[] levelOrder(TreeNode root) { + if (root == null) return new int[0]; + + Queue queue = new LinkedList<>(); + ArrayList ans = new ArrayList<>(); + + queue.add(root); + while (!queue.isEmpty()) { + TreeNode node = queue.poll(); + ans.add(node.val); + if (node.left != null) queue.add(node.left); + if (node.right != null) queue.add(node.right); + } + + int[] res = new int[ans.size()]; + for (int i = 0; i < ans.size(); i++) { + res[i] = ans.get(i); + } + + return res; +} +``` + +**🔍 复杂度分析**： +- 时间复杂度：O(n) +- 空间复杂度：O(n) + +## 栈和队列 + +### [剑指 Offer 09. 用两个栈实现队列](https://leetcode-cn.com/problems/yong-liang-ge-zhan-shi-xian-dui-lie-lcof/) + +**🎯 考察频率：极高 | 难度：简单 | 重要性：数据结构设计** + +> 用两个栈实现一个队列。队列的声明如下，请实现它的两个函数 appendTail 和 deleteHead ，分别完成在队列尾部插入整数和在队列头部删除整数的功能。 +> +> ``` +> 输入： +> ["CQueue","appendTail","deleteHead","deleteHead"] +> [[],[3],[],[]] +> 输出：[null,null,3,-1] +> ``` + +**💡 核心思路**：双栈模拟 + +一个栈用于入队，一个栈用于出队。当出队栈为空时，将入队栈的所有元素倒入出队栈。 + +```java +class CQueue { + Stack stack1; + Stack stack2; + + public CQueue() { + stack1 = new Stack(); + stack2 = new Stack(); + } + + public void appendTail(int value) { + stack1.push(value); + } + + public int deleteHead() { + if (!stack2.isEmpty()) { + return stack2.pop(); + } + + if (stack1.isEmpty()) { + return -1; + } + + // 将stack1中的元素倒入stack2 + while (!stack1.isEmpty()) { + stack2.push(stack1.pop()); + } + + return stack2.pop(); + } +} +``` + +**🔍 复杂度分析**： +- 时间复杂度：appendTail 为 O(1)，deleteHead 的平均复杂度为 O(1) +- 空间复杂度：O(n) + +### [剑指 Offer 30. 包含min函数的栈](https://leetcode-cn.com/problems/bao-han-minhan-shu-de-zhan-lcof/) + +**🎯 考察频率：高 | 难度：简单 | 重要性：数据结构设计** + +> 定义栈的数据结构，请在该类型中实现一个能够得到栈的最小元素的 min 函数在该栈中，调用 min、push 及 pop 的时间复杂度都是 O(1)。 +> +> ``` +> MinStack minStack = new MinStack(); +> minStack.push(-2); +> minStack.push(0); +> minStack.push(-3); +> minStack.min(); --> 返回 -3. +> minStack.pop(); +> minStack.top(); --> 返回 0. +> minStack.min(); --> 返回 -2. +> ``` + +**💡 核心思路**：辅助栈保存最小值 + +使用一个辅助栈来保存当前栈中的最小值。 + +```java +class MinStack { + Stack A, B; + + public MinStack() { + A = new Stack<>(); + B = new Stack<>(); + } + + public void push(int x) { + A.add(x); + if (B.empty() || B.peek() >= x) { + B.add(x); + } + } + + public void pop() { + if (A.pop().equals(B.peek())) { + B.pop(); + } + } + + public int top() { + return A.peek(); + } + + public int min() { + return B.peek(); + } +} +``` + +**🔍 复杂度分析**： +- 时间复杂度：所有操作都是 O(1) +- 空间复杂度：O(n) + +## 动态规划 + +#### [剑指 Offer 10. 斐波那契数列](https://leetcode-cn.com/problems/fei-bo-na-qi-shu-lie-lcof/) + +> 写一个函数，输入 n ，求斐波那契数列的第 n 项。 +> +> ``` +> 输入：n = 2 +> 输出：1 +> +> 输入：n = 5 +> 输出：5 +> ``` + +#### [剑指 Offer 42. 连续子数组的最大和](https://leetcode-cn.com/problems/lian-xu-zi-shu-zu-de-zui-da-he-lcof/) + +> 输入一个整型数组，数组中的一个或连续多个整数组成一个子数组。求所有子数组的和的最大值。 +> +> ``` +> 输入: nums = [-2,1,-3,4,-1,2,1,-5,4] +> 输出: 6 +> 解释: 连续子数组 [4,-1,2,1] 的和最大，为 6。 +> ``` + +#### [剑指 Offer 47. 礼物的最大价值](https://leetcode-cn.com/problems/li-wu-de-zui-da-jie-zhi-lcof/) + +> 在一个 m*n 的棋盘的每一格都放有一个礼物，每个礼物都有一定的价值（价值大于 0）。你可以从棋盘的左上角开始拿格子里的礼物，并每次向右或者向下移动一格、直到到达棋盘的右下角。给定一个棋盘及其上面的礼物的价值，请计算你最多能拿到多少价值的礼物？ +> +> ``` +> 输入: +> [ +> [1,3,1], +> [1,5,1], +> [4,2,1] +> ] +> 输出: 12 +> 解释: 路径 1→3→5→2→1 可以拿到最多价值的礼物 +> ``` + +## 搜索与回溯 + +#### [剑指 Offer 12. 矩阵中的路径](https://leetcode-cn.com/problems/ju-zhen-zhong-de-lu-jing-lcof/) + +> 给定一个 m x n 二维字符网格 board 和一个字符串单词 word 。如果 word 存在于网格中，返回 true ；否则，返回 false 。 +> +> ``` +> board = [["A","B","C","E"],["S","F","C","S"],["A","D","E","E"]], word = "ABCCED" +> 输出：true +> ``` + +#### [剑指 Offer 13. 机器人的运动范围](https://leetcode-cn.com/problems/ji-qi-ren-de-yun-dong-fan-wei-lcof/) + +> 地上有一个m行n列的方格，从坐标 [0,0] 到坐标 [m-1,n-1] 。一个机器人从坐标 [0, 0] 的格子开始移动，它每次可以向左、右、上、下移动一格（不能移动到方格外），也不能进入行坐标和列坐标的数位之和大于k的格子。 +> +> ``` +> 输入：m = 2, n = 3, k = 1 +> 输出：3 +> ``` + +## 数学相关 + +### [剑指 Offer 14. 剪绳子](https://leetcode-cn.com/problems/jian-sheng-zi-lcof/) + +**🎯 考察频率：高 | 难度：中等 | 重要性：数学推理** + +> 给你一根长度为 n 的绳子，请把绳子剪成整数长度的 m 段（m、n都是整数，n>1并且m>1），每段绳子的长度记为 k[0],k[1]...k[m-1] 。请问 k[0]*k[1]*...*k[m-1] 可能的最大乘积是多少？ +> +> ``` +> 输入: 2 +> 输出: 1 +> 解释: 2 = 1 + 1, 1 × 1 = 1 +> +> 输入: 10 +> 输出: 36 +> 解释: 10 = 3 + 3 + 4, 3 × 3 × 4 = 36 +> ``` + +**💡 核心思路**：数学规律 + 贪心 + +数学推导：尽可能多地剪出长度为3的绳子段，当剩余长度为4时剪成两段2。 + +```java +public int cuttingRope(int n) { + if (n <= 3) return n - 1; + + int a = n / 3, b = n % 3; + + if (b == 0) { + return (int) Math.pow(3, a); + } else if (b == 1) { + return (int) Math.pow(3, a - 1) * 4; + } else { + return (int) Math.pow(3, a) * 2; + } +} +``` + +**🔍 复杂度分析**： +- 时间复杂度：O(1) +- 空间复杂度：O(1) + +### [剑指 Offer 39. 数组中出现次数超过一半的数字](https://leetcode-cn.com/problems/shu-zu-zhong-chu-xian-ci-shu-chao-guo-yi-ban-de-shu-zi-lcof/) + +**🎯 考察频率：极高 | 难度：简单 | 重要性：摩尔投票算法** + +> 数组中有一个数字出现的次数超过数组长度的一半，请找出这个数字。 +> +> ``` +> 输入: [1, 2, 3, 2, 2, 2, 5, 4, 2] +> 输出: 2 +> ``` + +**💡 核心思路**：摩尔投票算法 + +摩尔投票算法：维持一个候选人和计数器，遇到相同的数字计数+1，不同的-1，计数为0时更换候选人。 + +```java +public int majorityElement(int[] nums) { + int x = 0, votes = 0; + + for (int num : nums) { + if (votes == 0) x = num; + votes += num == x ? 1 : -1; + } + + return x; +} +``` + +**🔍 复杂度分析**： +- 时间复杂度：O(n) +- 空间复杂度：O(1) + +### [剑指 Offer 62. 圆圈中最后剩下的数字](https://leetcode-cn.com/problems/yuan-quan-zhong-zui-hou-sheng-xia-de-shu-zi-lcof/) + +**🎯 考察频率：极高 | 难度：简单 | 重要性：约瑟夫环经典问题** + +> 0,1,···,n-1这n个数字排成一个圆圈，从数字0开始，每次从这个圆圈里删除第m个数字（删除后从下一个数字开始计数）。求出这个圆圈里剩下的最后一个数字。 +> +> ``` +> 输入: n = 5, m = 3 +> 输出: 3 +> 解释: 0,1,2,3,4围成一个圆圈，每次删除第3个数字： +> 删除2：0,1,3,4 +> 删除0：1,3,4 +> 删除4：1,3 +> 删除1：3 +> ``` + +**💡 核心思路**：约瑟夫环数学解法 + +这是经典的约瑟夫环问题。通过数学推导可以得出递推公式： +- f(n,m) = (f(n-1,m) + m) % n +- 边界条件：f(1,m) = 0 + +```java +public int lastRemaining(int n, int m) { + int pos = 0; // 最终留下那个人的初始位置 + for (int i = 2; i <= n; i++) { + pos = (pos + m) % i; // 每次重新计算位置 + } + return pos; +} +``` + +**递归解法（更直观理解）**： +```java +/** + * 约瑟夫问题的递归解。 + * 定义 f(i, m) 表示 “i 个人、步长为 m” 时，幸存者在当前圈中的 **相对编号**。 + * 初始问题规模是 n，最终要的答案就是 f(n, m)。 + */ +private int f(int n, int m) { + if (n == 1) { // 边界：只剩 1 人，他当然在自己圈里排 0 号 + return 0; + } + // 先算 “i-1 个人” 的子问题幸存者编号 + int x = f(n - 1, m); // x 是 “n-1 圈” 里的幸存位置 + // 把子问题的答案映射到 “n 个人” 的圈上： + // 上一轮出局的是 (m-1)%n 位置，整个圈从那里断开并重新编号， + // 所以新编号 = (旧编号 + m) % n + return (m + x) % n; // 映射回当前圈，得到真正的幸存位置 +} +``` + +**模拟法（使用 `ArrayList`）** + +```java +public static int lastRemaining(int n, int k) { + if (n <= 0 || k <= 0) { + throw new IllegalArgumentException("n and k must be positive integers."); + } + + List people = new ArrayList<>(); + for (int i = 1; i <= n; i++) { + people.add(i); + } + + int currentIndex = 0; // 从第一个人(索引0)开始计数 + + while (people.size() > 1) { + // 计算要淘汰的人的索引 + // (currentIndex + k - 1) % people.size() + int eliminateIndex = (currentIndex + k - 1) % people.size(); + + // 移除被淘汰的人 + people.remove(eliminateIndex); + + // 更新下一次计数的起始索引，就是被淘汰者的下一个位置 + currentIndex = eliminateIndex % people.size(); // 当删除最后一个元素时，取模确保索引正确 + } + + return people.get(0); +} +``` + +**🔍 复杂度分析**： + +- 时间复杂度：O(n) +- 空间复杂度：O(1) 迭代解法，O(n) 递归解法 + +--- + +## 📝 总结 + +剑指Offer题目涵盖了算法和数据结构的各个重要知识点，是面试准备的经典题集。通过精通这些题目，可以显著提升编程能力和算法思维。 + +### 🎆 核心技巧总结： + +1. **数组操作**：双指针、原地交换、自定义排序 +2. **链表技巧**：快慢指针、虚拟头节点、递归与迭代 +3. **树的遍历**：前中后序递归、层序遍历、树的构建 +4. **栈队列**：双栈模拟、辅助栈的设计模式 +5. **动态规划**：状态定义、转移方程、空间优化 +6. **搜索回溯**：DFS模板、状态恢复、剪枝优化 +7. **数学问题**：数学规律发现、特殊算法（如摩尔投票） + +掌握这些核心技巧，就能应对大部分的算法面试题目！ \ No newline at end of file diff --git a/docs/data-structure/hello.md b/docs/data-structure/hello.md deleted file mode 100644 index 7b3383a27c..0000000000 --- a/docs/data-structure/hello.md +++ /dev/null @@ -1,115 +0,0 @@ -B-Tree、B+Tree、红黑树、B*Tree数据结构 https://blog.csdn.net/zhangliangzi/article/details/51367639 - -# 数据结构 - -## 概念 - -**数据（data）**是描述客观事物的数值、字符以及能输入机器且能被处理的各种符号集合。 数据的含义非常广泛，除了通常的数值数据、字符、字符串是数据以外，声音、图像等一切可以输入计算机并能被处理的都是数据。例如除了表示人的姓名、身高、体重等的字符、数字是数据，人的照片、指纹、三维模型、语音指令等也都是数据。 - - - -**数据元素（data element）**是数据的基本单位，是数据集合的个体，在计算机程序中通 常作为一个整体来进行处理。例如一条描述一位学生的完整信息的数据记录就是一个数据元 素；空间中一点的三维坐标也可以是一个数据元素。数据元素通常由若干个数据项组成，例 如描述学生相关信息的姓名、性别、学号等都是数据项；三维坐标中的每一维坐标值也是数 据项。数据项具有原子性，是不可分割的最小单位。 - - - -**数据对象（data object）**是性质相同的数据元素的集合，是数据的子集。例如一个学校的所有学生的集合就是数据对象，空间中所有点的集合也是数据对象。 - - - -**数据结构（data structure）**是指相互之间存在一种或多种特定关系的数据元素的集合。 是组织并存储数据以便能够有效使用的一种专门格式，它用来反映一个数据的内部构成，即 一个数据由哪些成分数据构成，以什么方式构成，呈什么结构。 - -由于信息可以存在于逻辑思维领域，也可以存在于计算机世界，因此作为信息载体的数据同样存在于两个世界中。表示一组数据元素及其相互关系的数据结构同样也有两种不同的表现形式，一种是数据结构的逻辑层面，即数据的逻辑结构；一种是存在于计算机世界的物理层面，即数据的存储结构 - - - -## 逻辑结构和物理结构 - -按照视点的不同，我们把数据结构分为逻辑结构和物理结构。 - -### 逻辑结构 - -是指数据对象中数据元素之间的相互关系。其实这也是我们今后最需要关注的问题。分为以下四种： - -- 集合结构：集合结构中的数据元素除了同属于一个集合外，他们之间没有其他关系 - - ![img](https://img-blog.csdn.net/20171015213224082?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvWWFuZ1RvbmdB/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/Center) - -- 线性结构：数据之间是一对一关系 - - ![img](https://img-blog.csdn.net/20171015213254478?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvWWFuZ1RvbmdB/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/Center) - -- 树形结构：数据之间存在一对多的层次关系 - - ![img](https://img-blog.csdn.net/20171015213325699?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvWWFuZ1RvbmdB/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/Center) - -- 图形结构：数据之间多对多的关系 - - ![img](https://img-blog.csdn.net/20171015213350420?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvWWFuZ1RvbmdB/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/Center) - -### 物理结构 - -是指数据的逻辑结构在计算机中的存储形式。（有时也被叫存储结构） - -数据是数据元素的集合，根据物理结构的定义，实际上就是如何把数据元素存储到计算机的存储器中。存储器主要是针对内存而言的，像硬盘、软盘、光盘等外部存储器的数据组织通常用文件结构来描述。 - -数据元素的存储结构形式有两种：顺序存储和链式存储。 - -- 顺序存储：把数据元素存放在地址连续的存储单元里，其数据间的逻辑关系和物理关系一致 - - ![img](https://img-blog.csdn.net/20171015213409499?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvWWFuZ1RvbmdB/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/Center) - -- 链式存储：把数据元素存放在任意的存储单元里，这组存储单元可以是连续的，也可以是不连续的。 - - ![img](https://img-blog.csdn.net/20171015213436248?watermark/2/text/aHR0cDovL2Jsb2cuY3Nkbi5uZXQvWWFuZ1RvbmdB/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70/gravity/Center) - - - -## 抽象数据结构类型 - -**数据类型（data type）**是指一组性质相同的值的集合及定义在此集合上的一些操作的总称。例如 Java 语言中就有许多不同的数据类型，包括数值型的数据类型、字符串、布尔型等数据类型。以 Java 中的 int 型为例，int 型的数据元素的集合是[-2147483648,2147483647] 间的整数，定义在其上的操作有加、减、乘、除四则运算，还有模运算等。 - -数据类型是按照值得不同进行划分的。在高级语言中，每个变量、常量和表达式都有各自的取值范围。类型就用来说明变量或表达式取值范围和所能进行的操作。 - -定义数据类型的作用一个是隐藏计算机硬件及其特性和差别，使硬件对于用户而言是透明的，即用户可以不关心数据类型是怎么实现的而可以使用它。定义数据类型的另一个作用是，用户能够使用数据类型定义的操作，方便的实现问题的求解。例如，用户可以使用 Java 定义在 int 型的加法操作完成两个整数的加法运算，而不用关心两个整数的加法在计算机中到底是如何实现的。这样不但加快了用户解决问题的速度，也使得用户可以在更高的层面上 考虑问题。 - -**抽象数据类型（abstract data type, 简称 ADT）**由一种数据模型和在该数据模型上的一组操作组成。 - -抽象数据类型一方面使得使用它的人可以只关心它的逻辑特征，不需要了解它的实现方 式。另一方面可以使我们更容易描述现实世界，使得我们可以在更高的层面上来考虑问题。 例如可以使用树来描述行政区划，使用图来描述通信网络。 - - - -## 数据结构分类 - -- 数组 -- 栈 -- 链表 -- 队列 -- 树 -- 图 -- 堆 -- 散列表 - - - -# 算法 - -算法设计是最具创造性的工作之一，人们解决任何问题的思想、方法和步骤实际上都可以认为是算法。人们解决问题的方法有好有坏，因此算法在性能上也就有高低之分。 - -## 概念 - -算法（algorithm）是指令的集合，是为解决特定问题而规定的一系列操作。它是明确定义的可计算过程，以一个数据集合作为输入，并产生一个数据集合作为输出。一个算法通常来说具有以下五个特性： - -- 输入：一个算法应以待解决的问题的信息作为输入。 -- 输出：输入对应指令集处理后得到的信息。 -- 可行性：算法是可行的，即算法中的每一条指令都是可以实现的，均能在有限的时间内完成。 -- 有穷性：算法执行的指令个数是有限的，每个指令又是在有限时间内完成的，因此 整个算法也是在有限时间内可以结束的。 -- 确定性：算法对于特定的合法输入，其对应的输出是唯一的。即当算法从一个特定 输入开始，多次执行同一指令集结果总是相同的。 对于随机算法，该特性应当被放宽 - - - -## 算法设计要求 - -- 正确性：算法的正确性是指算法至少应该具有输入、输出和加工处理无歧义、能正确反映问题的需求、能得到问题的正确答案 -- 可读性：算法设计的另一目的是为了便于阅读、理解和交流 -- 健壮性：当输入数据不合法时，算法也能做出相关处理，而不是产生异常或错误结果 -- 时间效率高和存储量低 \ No newline at end of file diff --git "a/docs/data-structure/\346\225\260\347\273\204.md" "b/docs/data-structure/\346\225\260\347\273\204.md" deleted file mode 100644 index bc532375c8..0000000000 --- "a/docs/data-structure/\346\225\260\347\273\204.md" +++ /dev/null @@ -1,268 +0,0 @@ -`数组`是数据结构中的基本模块之一。因为`字符串`是由字符数组形成的，所以二者是相似的。大多数面试问题都属于这个范畴。 - - - -`数组`是一种基本的数据结构，用于按顺序`存储元素的集合`。但是元素可以随机存取，因为数组中的每个元素都可以通过数组`索引`来识别。 - -数组可以有一个或多个维度。这里我们从`一维数组`开始，它也被称为线性数组。这里有一个例子： - -![img](https://aliyun-lc-upload.oss-cn-hangzhou.aliyuncs.com/aliyun-lc-upload/uploads/2018/07/31/screen-shot-2018-03-20-at-191856.png) - -在上面的例子中，数组 A 中有 6 个元素。也就是说，A 的长度是 6 。我们可以使用 A[0] 来表示数组中的第一个元素。因此，A[0] = 6 。类似地，A[1] = 3，A[2] = 8，依此类推。 - - - -### 动态数组 - -数组具有`固定的容量`，我们需要在初始化时指定数组的大小。有时它会非常不方便并可能造成浪费。 - -因此，大多数编程语言都提供内置的`动态数组`，它仍然是一个随机存取的列表数据结构，但`大小是可变的`。例如，在 C++ 中的 `vector`，以及在 Java 中的 `ArrayList`。 - -https://leetcode-cn.com/explore/learn/card/array-and-string/198/introduction-to-array/772/ - -#### 寻找数组的中心索引 - -给定一个整数类型的数组 `nums`，请编写一个能够返回数组**“中心索引”**的方法。 - -我们是这样定义数组**中心索引**的：数组中心索引的左侧所有元素相加的和等于右侧所有元素相加的和。 - -如果数组不存在中心索引，那么我们应该返回 -1。如果数组有多个中心索引，那么我们应该返回最靠近左边的那一个。 - - - -#### 至少是其他数字两倍的最大数 - -在一个给定的数组`nums`中，总是存在一个最大元素 。 - -查找数组中的最大元素是否至少是数组中每个其他数字的两倍。 - -如果是，则返回最大元素的索引，否则返回-1。 - -#### 加一 - -给定一个由**整数**组成的**非空**数组所表示的非负整数，在该数的基础上加一。 - -最高位数字存放在数组的首位， 数组中每个元素只存储**单个**数字。 - -你可以假设除了整数 0 之外，这个整数不会以零开头。 - - - -## 二维数组 - -类似于一维数组，`二维数组`也是由元素的序列组成。但是这些元素可以排列在矩形网格中而不是直线上。 - -在一些语言中，多维数组实际上是在`内部`作为一维数组实现的，而在其他一些语言中，`实际上`根本没有`多维数组`。 - -**1. C++ 将二维数组存储为一维数组。** - -下图显示了*大小为 M \* N 的数组 A* 的实际结构： - -![img](https://aliyun-lc-upload.oss-cn-hangzhou.aliyuncs.com/aliyun-lc-upload/uploads/2018/07/31/screen-shot-2018-03-31-at-161748.png) - -因此，如果我们将 A 定义为也包含 *M \* N* 个元素的一维数组，那么实际上 A[i][j] 就等于 A[i * N + j]。 - - - -**2. 在Java中，二维数组实际上是包含着 M 个元素的一维数组，每个元素都是包含有 N 个整数的数组。** - -下图显示了 Java 中二维数组 A 的实际结构： - -![img](https://aliyun-lc-upload.oss-cn-hangzhou.aliyuncs.com/aliyun-lc-upload/uploads/2018/07/31/screen-shot-2018-03-31-at-162857.png) - -#### 对角线遍历 - -给定一个含有 M x N 个元素的矩阵（M 行，N 列），请以对角线遍历的顺序返回这个矩阵中的所有元素 - -#### 螺旋矩阵 - -给定一个包含 *m* x *n* 个元素的矩阵（*m* 行, *n* 列），请按照顺时针螺旋顺序，返回矩阵中的所有元素。 - -#### 杨辉三角 - -给定一个非负整数 *numRows，*生成杨辉三角的前 *numRows* 行。 - - - -# 字符串 - -字符串实际上是一个 `unicode 字符`数组。你可以执行几乎所有我们在数组中使用的操作。然而，二者之间还是存在一些区别。 - - - -### 比较函数 - ------- - -字符串有它自己的`比较函数`（我们将在下面的代码中向你展示比较函数的用法）。 - -然而，存在这样一个问题： - -> 我们可以用 “==” 来比较两个字符串吗？ - -这取决于下面这个问题的答案： - -> 我们使用的语言是否支持`运算符重载`？ - -1. 如果答案是 `yes` （例如 C++）。我们`可以使用` “==” 来比较两个字符串。 -2. 如果答案是 `no` （例如 Java），我们`可能无法使用` “==” 来比较两个字符串。当我们使用 “==” 时，它实际上会比较这两个对象是否是同一个对象。 - -### 是否可变 - ------- - -不可变意味着一旦字符串被初始化，你就无法改变它的内容。 - -1. 在某些语言（如 C ++）中，字符串是`可变的`。 也就是说，你可以像在数组中那样修改字符串。 -2. 在其他一些语言（如 Java）中，字符串是不可变的。 此特性将带来一些问题。 我们将在下一篇文章中阐明问题和解决方案。 - -### 额外操作 - ------- - -与数组相比，我们可以对字符串执行一些额外的操作 - - - -#### 二进制求和 - -给定两个二进制字符串，返回他们的和（用二进制表示）。 - -输入为**非空**字符串且只包含数字 `1` 和 `0`。 - -#### 实现 strStr() - -实现 [strStr()](https://baike.baidu.com/item/strstr/811469) 函数。 - -给定一个 haystack 字符串和一个 needle 字符串，在 haystack 字符串中找出 needle 字符串出现的第一个位置 (从0开始)。如果不存在，则返回 **-1**。 - -#### 最长公共前缀 - -编写一个函数来查找字符串数组中的最长公共前缀。 - -如果不存在公共前缀，返回空字符串 `""`。 - - - -### 双指针技巧 - -通常，我们只使用从第一个元素开始并在最后一个元素结束的一个指针来进行迭代。 但是，有时候，我们可能需要`同时使用两个指针`来进行迭代 - -一个经典问题： - -> 反转数组中的元素。 - -其思想是将第一个元素与末尾进行交换，再向前移动到下一个元素，并不断地交换，直到它到达中间位置。 - -我们可以同时使用两个指针来完成迭代：一个`从第一个元素开始`，另一个`从最后一个元素开始`。持续交换它们所指向的元素，直到这两个指针相遇。 - - - -使用双指针技巧的典型场景之一是你想要 - -> 从两端向中间迭代数组。 - -这时你可以使用双指针技巧： - -> 一个指针从始端开始，而另一个指针从末端开始。 - -值得注意的是，这种技巧经常在`排序`数组中使用。 - - - -#### 反转字符串 - -编写一个函数，其作用是将输入的字符串反转过来。输入字符串以字符数组 `char[]` 的形式给出。 - -不要给另外的数组分配额外的空间，你必须**[原地](https://baike.baidu.com/item/原地算法)修改输入数组**、使用 O(1) 的额外空间解决这一问题。 - -你可以假设数组中的所有字符都是 [ASCII](https://baike.baidu.com/item/ASCII) 码表中的可打印字符。 - - - -#### 数组拆分 I - -给定长度为 **2n** 的数组, 你的任务是将这些数分成 **n** 对, 例如 (a1, b1), (a2, b2), ..., (an, bn) ，使得从1 到 n 的 min(ai, bi) 总和最大。 - - - -#### 两数之和 II - 输入有序数组 - -给定一个已按照***升序排列\*** 的有序数组，找到两个数使得它们相加之和等于目标数。 - -函数应该返回这两个下标值 index1 和 index2，其中 index1 必须小于 index2*。* - -**说明:** - -- 返回的下标值（index1 和 index2）不是从零开始的。 -- 你可以假设每个输入只对应唯一的答案，而且你不可以重复使用相同的元素。 - - - -有时，我们可以使用`两个不同步的指针`来解决问题。 - -让我们从另一个经典问题开始： - -> 给定一个数组和一个值，[原地](https://en.wikipedia.org/wiki/In-place_algorithm)删除该值的所有实例并返回新的长度。 - -如果我们没有空间复杂度上的限制，那就更容易了。我们可以初始化一个新的数组来存储答案。如果元素不等于给定的目标值，则迭代原始数组并将元素添加到新的数组中。 - -实际上，它相当于使用了两个指针，一个用于原始数组的迭代，另一个总是指向新数组的最后一个位置。 - -### 重新考虑空间限制 - -现在让我们重新考虑空间受到限制的情况。 - -我们可以采用类似的策略，我们继续使用两个指针：一个仍然用于迭代，而第二个指针总是指向`下一次添加的位置`。 - -``` -public int removeElement(int[] nums, int val) { - int k = 0; - for (int i = 0; i < nums.length; ++i) { - if (nums[i] != val) { - nums[k] = nums[i]; - k++; - } - } - return k; -} -``` - -在上面的例子中，我们使用两个指针，一个快指针 `i` 和一个慢指针 `k` 。`i` 每次移动一步，而 `k` 只在添加新的被需要的值时才移动一步。 - ------- - -这是你需要使用双指针技巧的一种非常常见的情况： - -> 同时有一个慢指针和一个快指针。 - -解决这类问题的关键是 - -> 确定两个指针的移动策略。 - -与前一个场景类似，你有时可能需要在使用双指针技巧之前对数组进行排序，也可能需要运用贪心想法来决定你的运动策略。 - - - -#### 移除元素 - -给你一个数组 *nums* 和一个值 *val*，你需要 **[原地](https://baike.baidu.com/item/原地算法)** 移除所有数值等于 *val* 的元素，并返回移除后数组的新长度。 - -不要使用额外的数组空间，你必须仅使用 O(1) 额外空间并 **[原地 ](https://baike.baidu.com/item/原地算法)修改输入数组**。 - -元素的顺序可以改变。你不需要考虑数组中超出新长度后面的元素。 - - - -#### 最大连续1的个数 - -给定一个二进制数组， 计算其中最大连续1的个数。 - - - -#### 长度最小的子数组 - -给定一个含有 **n** 个正整数的数组和一个正整数 **s ，**找出该数组中满足其和 **≥ s** 的长度最小的连续子数组**。**如果不存在符合条件的连续子数组，返回 0。 - - - diff --git a/docs/design-pattern/.DS_Store b/docs/design-pattern/.DS_Store new file mode 100644 index 0000000000..ff0f67510b Binary files /dev/null and b/docs/design-pattern/.DS_Store differ diff --git a/docs/design-pattern/Adapter-Pattern.md b/docs/design-pattern/Adapter-Pattern.md index bf34e7e256..90dc763520 100644 --- a/docs/design-pattern/Adapter-Pattern.md +++ b/docs/design-pattern/Adapter-Pattern.md @@ -1,6 +1,12 @@ -# 随遇而安的适配器模式 | Spring 中的适配器 +--- +title: 随遇而安的适配器模式 | Spring 中的适配器 +date: 2023-10-09 +tags: + - Design Patterns +categories: Design Patterns +--- -![适配器设计模式](https://tva1.sinaimg.cn/large/007S8ZIlly1gfjtnx5644j30hs0b4mxx.jpg) +![](https://img.starfish.ink/design-patterns/banner-adapter.jpg) @@ -10,11 +16,11 @@ 在不想改变原有代码逻辑的情况下，如何解决呢？ -这时候我们就可以创建一个「适配器」。这是一个特殊的对象， 能够转换对象接口， 使其能与其他对象进行交互。 +这时候我们就可以创建一个「**适配器**」。这是一个特殊的对象， 能够转换对象接口， 使其能与其他对象进行交互。 适配器模式通过封装对象将复杂的转换过程隐藏于幕后。 被封装的对象甚至察觉不到适配器的存在。 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gfjt8avun0j31py0kztda.jpg) +![](https://img.starfish.ink/design-patterns/adapter.jpg) @@ -22,22 +28,22 @@ 适配器是什么，不难理解，生活中也随处可见。比如，笔记本电脑的电源适配器、万能充（曾经的它真有一个这么牛逼的名字）、一拖十数据线等等。 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gfjhf9tjluj32hc0mu4p8.jpg) +![](https://img.starfish.ink/design-patterns/adapter-real.jpg) ## 基本介绍 -- 适配器模式将一个类的接口，转换成客户期望的另外一个接口。适配器让原本接口不兼容的类可以合作无间。也可以叫包装器（Wrapper）。 +- 适配器模式将一个类的接口，转换成客户期望的另外一个接口。适配器让原本接口不兼容的类可以合作无间。也可以叫包装器（Wrapper） -- **适配器模式**是一种结构型设计模式， 它能使接口不兼容的对象能够相互合作。 +- **适配器模式**是一种结构型设计模式， 它能使接口不兼容的对象能够相互合作 - 主要分为两类：类适配器模式、对象适配器模式 ## 工作原理 -- 适配器模式：将一个类的接口转换成另一种接口.让原本接口不兼容的类可以兼容 +- 适配器模式：将一个类的接口转换成另一种接口，让原本接口不兼容的类可以兼容 - 从用户的角度看不到被适配者，是解耦的 - 用户调用适配器转化出来的目标接口方法，适配器再调用被适配者的相关接口方法 - 用户收到反馈结果，感觉只是和目标接口交互 @@ -50,7 +56,7 @@ 实现时使用了构成原则： 适配器实现了其中一个对象的接口， 并对另一个对象进行封装。 所有流行的编程语言都可以实现适配器。 -![适配器设计模式的结构（对象适配器）](https://tva1.sinaimg.cn/large/007S8ZIlly1gfjjbh88h7j31od0u0ds2.jpg) +![适配器设计模式的结构（对象适配器）](https://img.starfish.ink/design-patterns/adapter-uml.jpg) 1. **客户端** （Client） 是包含当前程序业务逻辑的类。 2. **客户端接口** （Target） 描述了其他类与客户端代码合作时必须遵循的协议。 @@ -135,7 +141,7 @@ 这一实现使用了继承机制： 适配器同时继承两个对象的接口。 请注意， 这种方式仅能在支持多重继承的编程语言中实现，例如 C++， Java 不支持多重继承，也就没有这种适配器了。 -![适配器设计模式（类适配器）](https://tva1.sinaimg.cn/large/007S8ZIlly1gfjtka91z8j31od0u0n9c.jpg) +![适配器设计模式（类适配器）](https://img.starfish.ink/design-patterns/adapter-class.jpg) **类适配器**不需要封装任何对象， 因为它同时继承了客户端和服务的行为。 适配功能在重写的方法中完成。 最后生成的适配器可替代已有的客户端类进行使用。 @@ -207,7 +213,7 @@ Java 虽然不能实现标准的类适配器，但是有一种变通的方式， 220V 的交流电相当于被适配者 Adaptee，我们的目标 Target 是 5V 直流电，充电器本身相当于一个 Adapter，将220V 的输入电压变换为 5V 输出。 -1. 首先是我们的名用电（我国是 220V，当然还可以有其他国家的其他准备，可随时扩展） +1. 首先是我们的民用电（我国是 220V，当然还可以有其他国家的其他准备，可随时扩展） ```java public class Volatage220V { @@ -297,9 +303,9 @@ Spring 源码中搜关键字`Adapter` 会出现很多实现类，SpringMVC 中 我们先回顾下 SpringMVC 处理流程： -![qsli.github.io](https://tva1.sinaimg.cn/large/007S8ZIlly1gfjqoif1ddj30vq0ij0uq.jpg) +![qsli.github.io](https://img.starfish.ink/design-patterns/adapter-spring.jpg) -Spring MVC中的适配器模式主要用于执行目标 `Controller` 中的请求处理方法。 +Spring MVC 中的适配器模式主要用于执行目标 `Controller` 中的请求处理方法。 在Spring MVC中，`DispatcherServlet` 作为用户，`HandlerAdapter` 作为期望接口，具体的适配器实现类用于对目标类进行适配，`Controller` 作为需要适配的类。 @@ -322,11 +328,11 @@ if(mappedHandler.getHandler() instanceof MultiActionController){ ```java public class DispatcherServlet extends FrameworkServlet { //...... - //维护所有HandlerAdapter类的集合 + //维护所有HandlerAdapter类的集合 @Nullable private List handlerAdapters; - //初始化handlerAdapters + //初始化handlerAdapters private void initHandlerAdapters(ApplicationContext context) { this.handlerAdapters = null; if (this.detectAllHandlerAdapters) { @@ -355,15 +361,15 @@ public class DispatcherServlet extends FrameworkServlet { protected void doDispatch(HttpServletRequest request, HttpServletResponse response) throws Exception { //... - //获得controller对应的适配器 + //获得controller对应的适配器 HandlerAdapter ha = this.getHandlerAdapter(mappedHandler.getHandler()); - //调用适配器的handler方法处理请求，并返回ModelAndView + //调用适配器的handler方法处理请求，并返回ModelAndView mv = ha.handle(processedRequest, response, mappedHandler.getHandler()); //... } - //返回对应的controller的处理器 + //返回对应的controller的处理器 protected HandlerAdapter getHandlerAdapter(Object handler) throws ServletException { if (this.handlerAdapters != null) { Iterator var2 = this.handlerAdapters.iterator(); @@ -396,20 +402,19 @@ public interface HandlerAdapter { 再来屡一下这个流程： 1. 首先是适配器接口 DispatchServlet 中有一个集合维护所有的 HandlerAdapter，如果配置文件中没有对适配器进行配置，那么 DispatchServlet 会在创建时对该变量进行初始化，注册所有默认的 HandlerAdapter。 -2. 当一个请求过来时，DispatchServlet 会根据传过来的 handler 类型从该集合中寻找对应的 HandlerAdapter子类进行处理，并且调用它的 handler() 方法 -3. 对应的 HandlerAdapter 中的 handler() 方法又会执行对应 Controller 的 handleRequest() 方法 +2. 当一个请求过来时，DispatchServlet 会根据传过来的 handler 类型从该集合中寻找对应的 HandlerAdapter子类进行处理，并且调用它的 `handler()` 方法 +3. 对应的 HandlerAdapter 中的 `handler()` 方法又会执行对应 Controller 的 `handleRequest()` 方法 适配器与 handler 有对应关系，而各个适配器又都是适配器接口的实现类，因此，它们都遵循相同的适配器标准，所以用户可以按照相同的方式，通过不同的 handler 去处理请求。 当然了，Spring 框架中也为我们定义了一些默认的 Handler 对应的适配器。 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gfjrbrv96pj31ck0j0dl6.jpg) +![](https://img.starfish.ink/design-patterns/adapter-demo.jpg) -通过适配器模式我们将所有的 `controller` 统一交给 `HandlerAdapter` 处理，免去了写大量的 `if-else` 语句对 `Controller` 进行判断，也更利于扩展新的 `Controller` 类型。 +通过适配器模式我们将所有的 `controller` 统一交给 `HandlerAdapter` 处理，免去了写大量的 `if-else` 语句对 `Controller` 进行判断，也更利于扩展新的 `Controller` 类型。 ## 参考与感谢 -《图解 Java 设计模式》 -《Head First设计模式》 -https://refactoringguru.cn/design-patterns/ -https://blog.csdn.net/lu__peng/article/details/79117894 -https://juejin.im/post/5ba28986f265da0abc2b6084#heading-12 \ No newline at end of file +- 《图解 Java 设计模式》 +- 《Head First设计模式》 +- https://refactoringguru.cn/design-patterns/ +- https://juejin.im/post/5ba28986f265da0abc2b6084#heading-12 \ No newline at end of file diff --git a/docs/design-pattern/Builder-Pattern.md b/docs/design-pattern/Builder-Pattern.md new file mode 100755 index 0000000000..cdd2d7d556 --- /dev/null +++ b/docs/design-pattern/Builder-Pattern.md @@ -0,0 +1,305 @@ +--- +title: 建造者模式 +date: 2021-10-09 +tags: + - Design Patterns +categories: Design Patterns +--- + +![](https://img.starfish.ink/design-patterns/builder-pattern-banner.png) + +> StringBuilder 你肯定用过，JDK 中的建造者模式 +> +> lombok 中的 @Bulider，你可能也用过，恩，这也是我们要说的建造者模式 + +> 直接使用构造函数或者配合 set 方法就能创建对象，为什么还需要建造者模式来创建呢？ +> +> 建造者模式和工厂模式都可以创建对象，那它们两个的区别在哪里呢？ + +## 简介 + +Builder Pattern，中文翻译为**建造者模式**或者**构建者模式**，也有人叫它**生成器模式**。 + +**建造者模式**是一种创建型设计模式， 使你能够分步骤创建复杂对象。它允许用户只通过指定复杂对象的类型和内容就可以构建它们，用户不需要知道内部的具体构建细节。 + +**定义**：将一个复杂对象的构建与它的表示分离，使得同样的构建过程可以创建不同的表示。 + +![](https://img.starfish.ink/design-pattern/frc-8d65236e72e9b84771951a1f4af83e86.gif) + + + +## hello world + +程序员麽，先上个 `hello world` 热热身 + +```java +public class User { + + private Long id; + private String name; + private Integer age; //可选 + private String desc; //可选 + + private User(Builder builder) { + this.id = builder.id; + this.name = builder.name; + this.age = builder.age; + this.desc = builder.desc; + } + + public static Builder newBuilder(Long id, String name) { + return new Builder(id, name); + } + + public Long getId() {return id;} + public String getName() {return name;} + public Integer getAge() {return age;} + public String getDesc() {return desc;} + + @Override + public String toString() { + return "Builder{" + + "id=" + id + + ", name='" + name + '\'' + + ", age=" + age + + ", desc='" + desc + '\'' + + '}'; + } + + public static class Builder { + private Long id; + private String name; + private Integer age; + private String desc; + + private Builder(Long id, String name) { + Assert.assertNotNull("标识不能为空",id); + Assert.assertNotNull("名称不能为空",name); + this.id = id; + this.name = name; + } + public Builder age(Integer age) { + this.age = age; + return this; + } + public Builder desc(String desc) { + this.desc = desc; + return this; + } + public User build() { + return new User(this); + } + + } + + public static void main(String[] args) { + User user = User.newBuilder(1L, "starfish").age(22).desc("test").build(); + System.out.println(user.toString()); + } +} +``` + +这样的代码有什么优缺点呢？ + +主要优点： + +1. 明确了必填参数和可选参数，在构造方法中进行验证； +2. 可以定义为不可变类，初始化后属性字段值不可变更； +3. 赋值代码可读性较好，明确知道哪个属性字段对应哪个值； +4. 支持链式方法调用，相比于调用 Setter 方法，代码更简洁。 + +主要缺点： + +1. 代码量较大，多定义了一个 Builder 类，多定义了一套属性字段，多实现了一套赋值方法； +2. 运行效率低，需要先创建 Builder 实例，再赋值属性字段，再创建目标实例，最后拷贝属性字段。 + +> 当然，以上代码，就可以通过 Lombok 的 @Builder 简化代码 +> +> 如果我们就那么三三两两个参数，直接构造函数配合 set 方法就能搞定的，就不用套所谓的模式了。 +> +> 高射炮打蚊子——不合算 +> +> 假设有这样一个复杂对象， 在对其进行构造时需要对诸多成员变量和嵌套对象进行繁复的初始化工作。 这些初始化代码通常深藏于一个包含众多参数且让人基本看不懂的构造函数中； 甚至还有更糟糕的情况， 那就是这些代码散落在客户端代码的多个位置。 +> +> 这时候才是构造器模式上场的时候 + + + +上边的例子，其实属于简化版的建造者模式，只是为了方便构建类中的各个参数，”正经“的和这个有点差别，更倾向于用同样的构建过程分步创建不同的产品类。 + +我们接着扯~ + +## 结构 + +![](https://img.starfish.ink/design-patterns/builder-UML.png) + +从 UML 图上可以看到有 4 个不同的角色 + +- 抽象建造者（Builder）：创建一个 Produc 对象的各个部件指定的接口/抽象类 +- 具体建造者（ConcreteBuilder）：实现接口，构建和装配各个组件 +- 指挥者/导演类（Director）：构建一个使用 Builder 接口的对象。负责调用适当的建造者来组建产品，导演类一般不与产品类发生依赖关系，与导演类直接交互的是建造者类。 +- 产品类（Product）：一个具体的产品对象 + + + +## demo + +假设我是个汽车工厂，需求就是能造各种车（或者造电脑、造房子、做煎饼、生成不同文件TextBuilder、HTMLBuilder等等，都是一个道理） + +![](https://img.starfish.ink/design-patterns/builder-car.png) + +1、生成器（Builder）接口声明在所有类型生成器中通用的产品构造步骤 + +```java +public interface CarBuilder { + void setCarType(CarType type); + void setSeats(int seats); + void setEngine(Engine engine); + void setGPS(GPS gps); +} +``` + +2、具体的生成器（Concrete Builders）提供构造过程的不同实现 + +```java +public class SportsCarBuilder implements CarBuilder { + + private CarType carType; + private int seats; + private Engine engine; + private GPS gps; + + @Override + public void setCarType(CarType type) { + this.carType = type; + } + + @Override + public void setSeats(int seats) { + this.seats = seats; + } + + @Override + public void setEngine(Engine engine) { + this.engine = engine; + } + + @Override + public void setGPS(GPS gps) { + this.gps = gps; + } + + public Car getResult() { + return new Car(carType, seats, engine, gps); + } +} +``` + +3、产品（Products）是最终生成的对象 + +```java +@Setter +@Getter +@ToString +public class Car { + + private final CarType carType; + private final int seats; + private final Engine engine; + private final GPS gps; + private double fuel; + + public Car(CarType carType,int seats,Engine engine,GPS gps){ + this.carType = carType; + this.seats = seats; + this.engine = engine; + this.gps = gps; + } +} +``` + +4、主管（Director）类定义调用构造步骤的顺序，这样就可以创建和复用特定的产品配置（Director 类的构造函数的参数是 CarBuilder，但实际上没有实例传递出去作参数，因为 CarBuilder 是接口或抽象类，无法产生对象实例，实际传递的是 Builder 的子类，根据子类类型，决定生产内容） + +```java +public class Director { + + public void constructSportsCar(CarBuilder builder){ + builder.setCarType(CarType.SPORTS_CAR); + builder.setSeats(2); + builder.setEngine(new Engine(2.0,0)); + builder.setGPS(new GPS()); + } + + public void constructCityCar(CarBuilder builder){ + builder.setCarType(CarType.CITY_CAR); + builder.setSeats(4); + builder.setEngine(new Engine(1.5,0)); + builder.setGPS(new GPS()); + } + + public void constructSUVCar(CarBuilder builder){ + builder.setCarType(CarType.SUV); + builder.setSeats(4); + builder.setEngine(new Engine(2.5,0)); + builder.setGPS(new GPS()); + } + +} +``` + +5、客户端使用（最终结果从建造者对象中获取，主管并不知道最终产品的类型） + +```java +public class Client { + + public static void main(String[] args) { + Director director = new Director(); + SportsCarBuilder builder = new SportsCarBuilder(); + director.constructSportsCar(builder); + + Car car = builder.getResult(); + System.out.println(car.toString()); + } +} +``` + + + +## 适用场景 + +适用场景其实才是理解设计模式最重要的，只要知道这个业务场景需要什么模式，网上浪程序员能不会吗 + +- **使用建造者模式可避免重叠构造函数的出现**。 + + 假设你的构造函数中有 N 个可选参数，那 new 各种实例的时候就很麻烦，需要重载构造函数多次 + +- 当你希望使用代码创建不同形式的产品 （例如石头或木头房屋） 时， 可使用建造者模式。 + + 如果你需要创建的各种形式的产品， 它们的制造过程相似且仅有细节上的差异， 此时可使用建造者模式。 + +- **使用生成器构造组合树或其他复杂对象**。 + + 建造者模式让你能分步骤构造产品。 你可以延迟执行某些步骤而不会影响最终产品。 你甚至可以递归调用这些步骤， 这在创建对象树时非常方便。 + + + +## VS 抽象工厂 + +抽象工厂模式实现对产品家族的创建，一个产品家族是这样的一系列产品：具有不同分类维度的产品组合，采用抽象工厂模式不需要关心抽象过程，只关心什么产品由什么工厂生产即可。而建造者模式则是要求按照指定的蓝图建造产品，它的主要目的是通过组装零配件而生产一个新的产品。 + + + +## 最后 + +设计模式，这玩意看简单的例子，肯定能看得懂，主要是结合自己的业务思考怎么应用，让系统设计更完善，懂了每种模式后，可以找找各种框架源码或在 github 搜搜相关内容，看看实际中是怎么应用的。 + + + +> 公众号回复 ”设计模式“，领取 10 本设计模式 pdf 书籍 + + + +## 参考 + +- refactoringguru.cn + diff --git a/docs/design-pattern/Chain-of-Responsibility-Pattern.md b/docs/design-pattern/Chain-of-Responsibility-Pattern.md index 648a96cdfc..3ac95054dc 100644 --- a/docs/design-pattern/Chain-of-Responsibility-Pattern.md +++ b/docs/design-pattern/Chain-of-Responsibility-Pattern.md @@ -1,11 +1,18 @@ -# 责任链模式 - -责任链，顾名思义，就是用来处理相关事务责任的一条执行链，执行链上有多个节点，每个节点都有机会（条件匹配）处理请求事务，如果某个节点处理完了就可以根据实际业务需求传递给下一个节点继续处理或者返回处理完毕。 -这种模式给予请求的类型，对请求的发送者和接收者进行解耦。属于行为型模式。 - -在这种模式中，通常每个接收者都包含对另一个接收者的引用。如果一个对象不能处理该请求，那么它会把相同的请求传给下一个接收者，依此类推。 - -![责任链设计模式](https://tva1.sinaimg.cn/large/00831rSTly1gdgeavn9h7j30hs0b4ta9.jpg) +--- +title: 责任链模式 +date: 2021-10-09 +tags: + - Design Patterns +categories: Design Patterns +--- + +![](https://images.unsplash.com/photo-1463587480257-3c60227e1e52?w=1200&auto=format&fit=crop&q=60&ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxzZWFyY2h8MTR8fGNoYWluJTIwb2YlMjByZXNwb25zbGl0eXxlbnwwfHwwfHx8MA%3D%3D) + +> 责任链，顾名思义，就是用来处理相关事务责任的一条执行链，执行链上有多个节点，每个节点都有机会（条件匹配）处理请求事务，如果某个节点处理完了就可以根据实际业务需求传递给下一个节点继续处理或者返回处理完毕。 +> +> 这种模式给予请求的类型，对请求的发送者和接收者进行解耦。属于行为型模式。 +> +> 在这种模式中，通常每个接收者都包含对另一个接收者的引用。如果一个对象不能处理该请求，那么它会把相同的请求传给下一个接收者，依此类推。 先来看一段代码 @@ -25,7 +32,7 @@ public void test(int i, Request request){ } ``` -代码的业务逻辑是这样的，方法有两个参数：整数 i 和一个请求 request，根据 i 的值来决定由谁来处理 request，如果 i==1，由 Handler1来处理，如果 i==2，由 Handler2 来处理，以此类推。在编程中，这种处理业务的方法非常常见，所有处理请求的类由if…else…条件判断语句连成一条责任链来对请求进行处理，相信大家都经常用到。这种方法的优点是非常直观，简单明了，并且比较容易维护，但是这种方法也存在着几个比较令人头疼的问题： +代码的业务逻辑是这样的，方法有两个参数：整数 i 和一个请求 request，根据 i 的值来决定由谁来处理 request，如果 i==1，由 Handler1来处理，如果 i==2，由 Handler2 来处理，以此类推。在编程中，这种处理业务的方法非常常见，所有处理请求的类由 if…else… 条件判断语句连成一条责任链来对请求进行处理，相信大家都经常用到。这种方法的优点是非常直观，简单明了，并且比较容易维护，但是这种方法也存在着几个比较令人头疼的问题： - **代码臃肿**：实际应用中的判定条件通常不是这么简单地判断是否为1或者是否为2，也许需要复杂的计算，也许需要查询数据库等等，这就会有很多额外的代码，如果判断条件再比较多，那么这个if…else…语句基本上就没法看了。 - **耦合度高**：如果我们想继续添加处理请求的类，那么就要继续添加if…else…判定条件；另外，这个条件判定的顺序也是写死的，如果想改变顺序，那么也只能修改这个条件语句。 @@ -46,7 +53,7 @@ public void test(int i, Request request){ ## 角色 -- **Handler**： 抽象处理类，抽象处理类中主要包含一个指向下一处理类的成员变量nextHandler和一个处理请求的方法handRequest，handRequest方法的主要主要思想是，如果满足处理的条件，则有本处理类来进行处理，否则由nextHandler来处理 +- **Handler**： 抽象处理类，抽象处理类中主要包含一个指向下一处理类的成员变量 nextHandler 和一个处理请求的方法 handRequest，handRequest 方法的主要主要思想是，如果满足处理的条件，则由本处理类来进行处理，否则由 nextHandler 来处理 - **ConcreteHandler**： 具体处理类主要是对具体的处理逻辑和处理的适用条件进行实现。具体处理者接到请求后，可以选择将请求处理掉，或者将请求传给下家。由于具体处理者持有对下家的引用，因此，如果需要，具体处理者可以访问下家 - **Client**：客户端 @@ -56,7 +63,7 @@ public void test(int i, Request request){ ## 类图 -![](../_images/design-pattern/responsibility-pattern-uml.png) +![](https://img.starfish.ink/design-patterns/responsibility-pattern-uml.png) ### coding @@ -127,7 +134,7 @@ public class Client { ConcreteHandler1 handler1 = new ConcreteHandler1(1); ConcreteHandler2 handler2 = new ConcreteHandler2(2); ConcreteHandler3 handler3 = new ConcreteHandler3(3); - //处理者构成一个环形 + //处理者构成一个环形 handler1.setNextHandler(handler2); handler2.setNextHandler(handler3); @@ -146,7 +153,7 @@ public class Client { 比如 -- 程序员要请3天以上的假期，在OA申请，需要直接主管、总监、HR 层层审批后才生效。类似的采购审批、报销审批。。。 +- 程序员要请 3 天以上的假期，在 OA 申请，需要直接主管、总监、HR 层层审批后才生效。类似的采购审批、报销审批。。。 - 美团在外卖营销业务中资源位展示的逻辑 https://tech.meituan.com/2020/03/19/design-pattern-practice-in-marketing.html @@ -205,9 +212,9 @@ public final class ApplicationFilterChain implements FilterChain { FilterChain 就是一条过滤链。其中每个过滤器（Filter）都可以决定是否执行下一步。过滤分两个方向，进和出： -进：在把ServletRequest和ServletResponse交给Servlet的service方法之前，需要进行过滤 +- 进：在把 ServletRequest 和 ServletResponse 交给 Servlet 的 service 方法之前，需要进行过滤 -出：在service方法完成后，往客户端发送之前，需要进行过滤 +- 出：在service方法完成后，往客户端发送之前，需要进行过滤 @@ -217,55 +224,55 @@ Spring MVC 的 diapatcherServlet 的 doDispatch 方法中，获取与请求匹 ```java protected void doDispatch(HttpServletRequest request, HttpServletResponse response) throws Exception { - HttpServletRequest processedRequest = request; - HandlerExecutionChain mappedHandler = null; //使用到了责任链模式 - boolean multipartRequestParsed = false; - WebAsyncManager asyncManager = WebAsyncUtils.getAsyncManager(request); + HttpServletRequest processedRequest = request; + HandlerExecutionChain mappedHandler = null; //使用到了责任链模式 + boolean multipartRequestParsed = false; + WebAsyncManager asyncManager = WebAsyncUtils.getAsyncManager(request); + try { try { + ModelAndView mv = null; + Object dispatchException = null; + try { - ModelAndView mv = null; - Object dispatchException = null; - - try { - processedRequest = this.checkMultipart(request); - multipartRequestParsed = processedRequest != request; - mappedHandler = this.getHandler(processedRequest); - if (mappedHandler == null) { - this.noHandlerFound(processedRequest, response); - return; - } + processedRequest = this.checkMultipart(request); + multipartRequestParsed = processedRequest != request; + mappedHandler = this.getHandler(processedRequest); + if (mappedHandler == null) { + this.noHandlerFound(processedRequest, response); + return; + } - HandlerAdapter ha = this.getHandlerAdapter(mappedHandler.getHandler()); - String method = request.getMethod(); - boolean isGet = "GET".equals(method); - if (isGet || "HEAD".equals(method)) { - long lastModified = ha.getLastModified(request, mappedHandler.getHandler()); - if ((new ServletWebRequest(request, response)).checkNotModified(lastModified) && isGet) { - return; - } - } - //责任链模式执行预处理方法，其实是将请求交给注册的拦截器执行 - if (!mappedHandler.applyPreHandle(processedRequest, response)) { + HandlerAdapter ha = this.getHandlerAdapter(mappedHandler.getHandler()); + String method = request.getMethod(); + boolean isGet = "GET".equals(method); + if (isGet || "HEAD".equals(method)) { + long lastModified = ha.getLastModified(request, mappedHandler.getHandler()); + if ((new ServletWebRequest(request, response)).checkNotModified(lastModified) && isGet) { return; } + } + //责任链模式执行预处理方法，其实是将请求交给注册的拦截器执行 + if (!mappedHandler.applyPreHandle(processedRequest, response)) { + return; + } - mv = ha.handle(processedRequest, response, mappedHandler.getHandler()); - if (asyncManager.isConcurrentHandlingStarted()) { - return; - } + mv = ha.handle(processedRequest, response, mappedHandler.getHandler()); + if (asyncManager.isConcurrentHandlingStarted()) { + return; + } - this.applyDefaultViewName(processedRequest, mv); - //责任链执行后处理方法 - mappedHandler.applyPostHandle(processedRequest, response, mv); - } catch (Exception var22) { - //... - } finally { - } + this.applyDefaultViewName(processedRequest, mv); + //责任链执行后处理方法 + mappedHandler.applyPostHandle(processedRequest, response, mv); + } catch (Exception var22) { + //... + } finally { + } } ``` -- SpringMVC 请求的流程中，执行了拦截器相关方法 interceptor.preHandler 等等 +- SpringMVC 请求的流程中，执行了拦截器相关方法 `interceptor.preHandler` 等等 - 在处理 SpringMVC 请求时，使用到职责链模式还使用到适配器模式 - HandlerExecutionChain 主要负责的是请求拦截器的执行和请求处理，但是他本身不处理请求，只是将请求分配给链上注册处理器执行，这是职责链实现方式，减少职责链本身与处理逻辑之间的耦合，规范了处理流程 - HandlerExecutionChain 维护了 HandlerInterceptor 的集合， 可以向其中注册相应的拦截器 @@ -276,7 +283,7 @@ protected void doDispatch(HttpServletRequest request, HttpServletResponse respon ## 总结 -责任链模式其实就是一个灵活版的 if…else…语句，它就是将这些判定条件的语句放到了各个处理类中，这样做的优点是比较灵活了，但同样也带来了风险，比如设置处理类前后关系时，一定要特别仔细，搞对处理类前后逻辑的条件判断关系，并且注意不要在链中出现循环引用的问题。 +**责任链模式其实就是一个灵活版的 if…else…语句**，它就是将这些判定条件的语句放到了各个处理类中，这样做的优点是比较灵活了，但同样也带来了风险，比如设置处理类前后关系时，一定要特别仔细，搞对处理类前后逻辑的条件判断关系，**并且注意不要在链中出现循环引用的问题**。 **优点**： @@ -308,12 +315,6 @@ protected void doDispatch(HttpServletRequest request, HttpServletResponse respon ## 参考 -《研磨设计模式》 - -https://wiki.jikexueyuan.com/project/java-design-pattern/chain-responsibility-pattern.html - -https://refactoringguru.cn/design-patterns/chain-of-responsibility - - - -![blog_end.png](https://i.loli.net/2020/04/14/IH5sCS42xveP3Ud.png) \ No newline at end of file +- 《研磨设计模式》 +- https://wiki.jikexueyuan.com/project/java-design-pattern/chain-responsibility-pattern.html +- https://refactoringguru.cn/design-patterns/chain-of-responsibility diff --git a/docs/design-pattern/Decorator-Pattern.md b/docs/design-pattern/Decorator-Pattern.md index 32b985264d..dad48ec432 100644 --- a/docs/design-pattern/Decorator-Pattern.md +++ b/docs/design-pattern/Decorator-Pattern.md @@ -1,12 +1,18 @@ -# 装饰模式——JDK和Spring是如何杜绝继承滥用的 +--- +title: 装饰模式——看看 JDK 和 Spring 是如何杜绝继承滥用的 +date: 2022-10-09 +tags: + - Design Patterns +categories: Design Patterns +--- -《Head First 设计模式》中是这么形容装饰者模式——“**给爱用继承的人一个全新的设计眼界**”，拒绝继承滥用，从装饰者模式开始。 +![](https://images.unsplash.com/photo-1613574450323-9df30ffe3566?q=80&w=2969&auto=format&fit=crop&ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D) -装饰者模式允许向一个现有的对象添加新的功能，同时又不改变其结构。这种类型的设计模式属于**结构型模式**，它是作为现有的类的一个包装。 - -这种模式创建了一个装饰类，用来包装原有的类，并在保持类方法签名完整性的前提下，提供了额外的功能。 - ------- +> 《Head First 设计模式》中是这么形容装饰者模式的——“**给爱用继承的人一个全新的设计眼界**”，拒绝继承滥用，从装饰者模式开始。 +> +> 装饰者模式允许向一个现有的对象添加新的功能，同时又不改变其结构。这种类型的设计模式属于**结构型模式**，它是作为现有的类的一个包装。 +> +> 这种模式创建了一个装饰类，用来包装原有的类，并在保持类方法签名完整性的前提下，提供了额外的功能。 @@ -15,7 +21,7 @@ 一般有两种方式可以实现给一个类或对象增加行为： - 继承机制，使用继承机制是给现有类添加功能的一种有效途径，通过继承一个现有类可以使得子类在拥有自身方法的同时还拥有父类的方法。但是这种方法是静态的，用户不能控制增加行为的方式和时机。 -- 关联机制，即将一个类的对象嵌入另一个对象中，由另一个对象来决定是否调用嵌入对象的行为以便扩展自己的行为，我们称这个嵌入的对象为装饰器(Decorator) +- 关联机制，即将一个类的对象嵌入另一个对象中，由另一个对象来决定是否调用嵌入对象的行为以便扩展自己的行为，我们称这个嵌入的对象为**装饰器**(Decorator) 装饰模式以对客户透明的方式动态地给一个对象附加上更多的责任，换言之，客户端并不会觉得对象在装饰前和装饰后有什么不同。装饰模式可以在不需要创造更多子类的情况下，将对象的功能加以扩展。 @@ -53,7 +59,7 @@ ## 类图 -![](https://tva1.sinaimg.cn/large/00831rSTly1gd5a7252usj31750tpgof.jpg) +![](https://img.starfish.ink/design-patterns/decorator-uml.png) ------ @@ -69,7 +75,7 @@ 我还是比较喜欢卖煎饼的例子 - ![](https://i04piccdn.sogoucdn.com/a000fc61baeaeb5b) +![](https://static001.geekbang.org/infoq/5e/5eacd172775ac7b72bf28bc00b7b6b03.jpeg) 1、定义抽象组件 @@ -178,7 +184,7 @@ public class Client { } ``` -输出 +输出： ``` 煎饼果子花费：8.0元 @@ -188,7 +194,7 @@ public class Client { 顺便看下通过 IDEA 生成的 UML 类图（和我们画的类图一样哈） -![](https://tva1.sinaimg.cn/large/00831rSTly1gd515ta83tj318c0pmjui.jpg) +![](https://static001.geekbang.org/infoq/58/58aee40fcd93d09715db45982d39b046.jpeg) ------ @@ -200,13 +206,13 @@ public class Client { 我们使用 `java.io` 包下的各种输入流、输出流、字节流、字符流、缓冲流等各种各样的流，他们中的许多类都是装饰者，下面是一个典型的对象集合，用装饰者将功能结合起来，以读取文件数据 -![](https://tva1.sinaimg.cn/large/00831rSTly1gd51gr33b0j30ls0au74n.jpg) +![img](https://static001.geekbang.org/infoq/9d/9d0dcea2ea4653e9740e54f64a389e1c.jpeg?x-oss-process=image%2Fresize%2Cp_80%2Fauto-orient%2C1) `BufferedInputStream` 和 `LinerNumberInputStream` 都是扩展自 `FilterInputStream`，而 `FilterInputStream` 是一个抽象的装饰类。 在 idea 中选中一些常见 InputStream 类，生成 UML 图如下： -![](https://tva1.sinaimg.cn/large/00831rSTly1gd51yq60yxj322e0rewkg.jpg) +![img](https://static001.geekbang.org/infoq/46/46ddc8880590ba83c3cfed5b80a8e3bf.jpeg?x-oss-process=image%2Fresize%2Cp_80%2Fauto-orient%2C1) 我们平时读取一个文件中的内容其实就使用到了装饰模式的思想，简化《Head First 设计模式》的例子，我们自定义一个装饰者，把输入流中的所有大写字符转换为小写 @@ -240,13 +246,13 @@ public class InputTest { } ``` -采用装饰者模式在实例化组件时，将增加代码的复杂度，一旦使用装饰者模式，不只需要实例化组件，还把把此组件包装进装饰者中，天晓得有几个，所以在某些复杂情况下，我们还会结合工厂模式和生成器模式。比如Spring中的装饰者模式。 +采用装饰者模式在实例化组件时，将增加代码的复杂度，一旦使用装饰者模式，不只需要实例化组件，还要把此组件包装进装饰者中，天晓得有几个，所以在某些复杂情况下，我们还会结合工厂模式和生成器模式。比如 Spring 中的装饰者模式。 ### Servlet 中的装饰者模式 -Servlet API源自于4个实现类，它很少被使用，但是十分强大：`ServletRequestWrapper`、`ServletResponseWrapper`以及 `HttpServletRequestWrapper`、`HttpServletResponseWrapper`。 +Servlet API 源自于 4 个实现类，它很少被使用，但是十分强大：`ServletRequestWrapper`、`ServletResponseWrapper`以及 `HttpServletRequestWrapper`、`HttpServletResponseWrapper`。 比如`ServletRequestWrapper` 是 `ServletRequest` 接口的简单实现，开发者可以继承 `ServletRequestWrapper` 去扩展原来的`request` @@ -267,11 +273,11 @@ public class ServletRequestWrapper implements ServletRequest { -### spring 中的装饰者模式 +### pring 中的装饰者模式 -Spring 的 `ApplicationContext` 中配置所有的 `DataSource`。 这些 DataSource 可能是各种不同类型的， 比如不同的数据库： Oracle、 SQL Server、 MySQL 等， 也可能是不同的数据源。 然后 SessionFactory 根据客户的每次请求， 将 DataSource 属性设置成不同的数据源， 以到达切换数据源的目的。 +Spring 的 `ApplicationContext` 中配置所有的 `DataSource`。 这些 DataSource 可能是各种不同类型的， 比如不同的数据库： Oracle、 SQL Server、 MySQL 等， 也可能是不同的数据源。 然后 SessionFactory 根据客户的每次请求， 将 DataSource 属性设置成不同的数据源， 以达到切换数据源的目的。 -在 spring 的命名体现：Spring 中用到的包装器模式在类名上有两种表现： 一种是类名中含有 `Wrapper`， 另一种是类名中含有`Decorator`。 基本上都是动态地给一个对象添加一些额外的职责，比如 +在 Spring 的命名体现：Spring 中用到的装饰器模式在类名上有两种表现： 一种是类名中含有 `Wrapper`， 另一种是类名中含有 `Decorator`。 基本上都是动态地给一个对象添加一些额外的职责，比如 - `org.springframework.cache.transaction` 包下的 `TransactionAwareCacheDecorator` 类 - `org.springframework.session.web.http` 包下的 `SessionRepositoryFilter` 内部类 `SessionRepositoryRequestWrapper` @@ -280,9 +286,9 @@ Spring 的 `ApplicationContext` 中配置所有的 `DataSource`。 这些 DataSo ### Mybatis 缓存中的装饰者模式 -Mybatis 的缓存模块中，使用了装饰器模式的变体，其中将 `Decorator` 接口和 `Componet` 接口合并为一个`Component `接口。`org.apache.ibatis.cache` 包下的结构 +Mybatis 的缓存模块中，使用了装饰器模式的变体，其中将 `Decorator` 接口和 `Componet` 接口合并为一个`Component` 接口。`org.apache.ibatis.cache` 包下的结构 -![](https://tva1.sinaimg.cn/large/00831rSTly1gd56rvosupj30kw0r6gpg.jpg) +![img](https://static001.geekbang.org/infoq/56/56fcb2669209f990446e8710756264ce.jpeg?x-oss-process=image%2Fresize%2Cp_80%2Fauto-orient%2C1) ------ @@ -290,7 +296,7 @@ Mybatis 的缓存模块中，使用了装饰器模式的变体，其中将 `Deco ## 总结 -装饰模式的本质：动态组合 +装饰模式的本质：**动态组合** 动态组合是手段，组合才是目的。这里的组合有两个意思，一个是动态功能的组合，也就是动态进行装饰器的组合；另外一个是指对象组合，通过对象组合来实现为被装饰对象透明的增加功能。 @@ -311,7 +317,7 @@ Mybatis 的缓存模块中，使用了装饰器模式的变体，其中将 `Deco ### 何时选用 - 如果需要在不影响其他对象的情况下，以动态、透明的方式给对象添加职责，可以使用装饰模式 -- 当不能采用继承的方式对系统进行扩展或者采用继承不利于系统扩展和维护时可以使用装饰模式。不能采用继承的情况主要有两类：第一类是系统中存在大量独立的扩展，为支持每一种扩展或者扩展之间的组合将产生大量的子类，使得子类数目呈爆炸性增长；第二类是因为类已定义为不能被继承（如Java语言中的final类） +- 当不能采用继承的方式对系统进行扩展或者采用继承不利于系统扩展和维护时可以使用装饰模式。不能采用继承的情况主要有两类：第一类是系统中存在大量独立的扩展，为支持每一种扩展或者扩展之间的组合将产生大量的子类，使得子类数目呈爆炸性增长；第二类是因为类已定义为不能被继承（如 Java 语言中的 final 类） ------ @@ -319,10 +325,8 @@ Mybatis 的缓存模块中，使用了装饰器模式的变体，其中将 `Deco ## 参考 -《Head First 设计模式》《研磨设计模式》 - -https://design-patterns.readthedocs.io/zh_CN/latest/behavioral_patterns/observer.html - -https://www.runoob.com/design-pattern/decorator-pattern.html - -https://juejin.im/post/5ba0fb04e51d450e67494256#heading-14 \ No newline at end of file +- 《Head First 设计模式》 +- 《研磨设计模式》 +- https://design-patterns.readthedocs.io/zh_CN/latest/behavioral_patterns/observer.html +- https://www.runoob.com/design-pattern/decorator-pattern.html +- https://juejin.im/post/5ba0fb04e51d450e67494256#heading-14 \ No newline at end of file diff --git a/docs/design-pattern/Design-Pattern-Overview.md b/docs/design-pattern/Design-Pattern-Overview.md index 0404c128b7..9c3452bb1b 100644 --- a/docs/design-pattern/Design-Pattern-Overview.md +++ b/docs/design-pattern/Design-Pattern-Overview.md @@ -6,7 +6,7 @@ - 设计模式是软件开发人员的“标准词汇”，学习设计模式是个人技术能力提高的捷径 - 设计模式包含了面向对象的精髓，“懂了设计模式，你就懂了面向对象分析和设计（OOA/D）的精要” -![img](https://i01piccdn.sogoucdn.com/8c1a368c985c0e5d) +![](https://img.starfish.ink/design-patterns/knowledge.jpeg) ## 软件设计模式概述 @@ -103,9 +103,9 @@ -## 设计模式七大原则 +## 设计模式核心原则 -### 设计模式的目的 +**设计模式的目的** 编写软件过程中，程序员面临着来自耦合性，内聚性以及可维护性，可扩展性，重用性，灵活性等多方面的挑战，设计模式是为了让程序(软件)，具有更好的 @@ -117,69 +117,66 @@ -### 设计模式七大原则 +**设计模式核心原则** -设计模式原则，其实就是程序员在编程时，应当遵守的原则，也是各种设计模式的基础(即：设计模式为什么这样设计的依据) - -#### 1. 单一职责原则 +设计模式（Design Patterns）有一些核心原则，帮助开发人员在软件设计中做出更清晰、可维护、可扩展的决策。主要有以下几大原则： -单一职责原则表示一个模块的组成元素之间的功能相关性。从软件变化的角度来看，对类来说，**一个类应该只负责一项职责**。 -假设某个类 P 负责两个不同的职责，职责 P1 和 职责 P2，那么当职责 P1 需求发生改变而需要修改类 P，有可能会导致原来运行正常的职责 P2 功能发生故障。 -**单一职责原则注意事项和细节** +#### 1. **单一职责原则（SRP, Single Responsibility Principle）** -1. 降低类的复杂度，一个类只负责一项职责 -2. 提高类的可读性，可维护性 -3. 降低变更引起的风险 -4. 通常情况下，我们应当遵守单一职责原则，只有逻辑足够简单，才可以在代码级违反单一职责原则；只有类中方法数量足够少，可以在方法级别保持单一职责原则 +- 每个类应该只有一个责任，即一个类只应该有一个改变的原因。这意味着一个类应该仅负责一种功能，这样可以减少不同功能间的耦合，提升可维护性。 +- **单一职责原则注意事项和细节** + 1. 降低类的复杂度，一个类只负责一项职责 + 2. 提高类的可读性，可维护性 + 3. 降低变更引起的风险 + 4. 通常情况下，我们应当遵守单一职责原则，只有逻辑足够简单，才可以在代码级违反单一职责原则；只有类中方法数量足够少，可以在方法级别保持单一职责原则 -#### 2. 接口隔离原则 -客户端不应该依赖它不需要的接口，即**一个类对另一个类的依赖应该建立在最小的接口上** +#### 2. **开闭原则（OCP, Open/Closed Principle）** -接口隔离原则，其 "隔离" 并不是准确的翻译，真正的意图是 “分离” 接口(的功能) +- 软件实体（类、模块、函数等）应该对扩展开放，对修改封闭。也就是说，在不修改原有代码的基础上，可以通过增加新代码来扩展系统的功能。这鼓励使用抽象类或接口以及继承和多态。 +- 如果一个软件能够满足 OCP 原则，那么它将有两项优点： -#### 3. 开闭原则 + 1. 能够扩展已存在的系统，能够提供新的功能满足新的需求，因此该软件有着很强的适应性和灵活性。 + 2. 已存在的模块，特别是那些重要的抽象模块，不需要被修改，那么该软件就有很强的稳定性和持久性。 -**开放-关闭原则表示软件实体 (类、模块、函数等等) 应该是可以被扩展的，但是不可被修改。(Open for extension, close for modification)** +#### 3. **里氏替换原则（LSP, Liskov Substitution Principle）** -如果一个软件能够满足 OCP 原则，那么它将有两项优点： +- 子类对象应该可以替换父类对象，并且不会导致系统错误或行为不一致。即，如果一个类是另一个类的子类，那么这个子类对象就可以替代父类对象出现在任何需要父类对象的地方，系统的功能应该不受影响。 -1. 能够扩展已存在的系统，能够提供新的功能满足新的需求，因此该软件有着很强的适应性和灵活性。 -2. 已存在的模块，特别是那些重要的抽象模块，不需要被修改，那么该软件就有很强的稳定性和持久性。 +#### 4. **接口隔离原则（ISP, Interface Segregation Principle）** -#### 4. 依赖倒转(倒置)原则 +- 客户端不应该依赖它不需要的接口，即**一个类对另一个类的依赖应该建立在最小的接口上**。 +- 接口隔离原则，其 "隔离" 并不是准确的翻译，真正的意图是 “分离” 接口(的功能) -依赖倒转原则(Dependence Inversion Principle)是指： +#### 5. **依赖倒转原则（DIP, Dependency Inversion Principle）** -1. 高层模块不应该依赖低层模块，二者都应该依赖其抽象 -2. 抽象不应该依赖细节，细节应该依赖抽象 -3. 依赖倒转(倒置)的中心思想是面向接口编程 -4. 依赖倒转原则是基于这样的设计理念：相对于细节的多变性，抽象的东西要稳定的多。以抽象为基础搭建的架构比以细节为基础的架构要稳定的多。在 java 中，抽象指的是接口或抽象类，细节就是具体的实现类 -5. 使用接口或抽象类的目的是制定好规范，而不涉及任何具体的操作，把展现细节的任务交给他们的实现类去完成 +- 高层模块不应该依赖低层模块，两者应该依赖抽象（接口或抽象类）；抽象不应该依赖细节，细节应该依赖抽象。这个原则鼓励将实现细节与接口分离，提高系统的灵活性和可扩展性。 +- 依赖倒转原则(Dependence Inversion Principle)是指： -#### 5. 里氏替换原则 + 1. 高层模块不应该依赖低层模块，二者都应该依赖其抽象 + 2. 抽象不应该依赖细节，细节应该依赖抽象 + 3. 依赖倒转(倒置)的中心思想是面向接口编程 + 4. 依赖倒转原则是基于这样的设计理念：相对于细节的多变性，抽象的东西要稳定的多。以抽象为基础搭建的架构比以细节为基础的架构要稳定的多。在 java 中，抽象指的是接口或抽象类，细节就是具体的实现类 + 5. 使用接口或抽象类的目的是制定好规范，而不涉及任何具体的操作，把展现细节的任务交给他们的实现类去完成 -在编程中常常会遇到这样的问题：有一功能 P1, 由类 A 完成，现需要将功能 P1 进行扩展，扩展后的功能为 P，其中P由原有功能P1与新功能P2组成。新功能P由类A的子类B来完成，则子类B在完成新功能P2的同时，有可能会导致原有功能P1发生故障。 +#### 6. 迪米特法则（Law of Demeter, Principle of Least Knowledge） -里氏替换原则告诉我们，当使用继承时候，类 B 继承类 A 时，除添加新的方法完成新增功能 P2，尽量不要修改父类方法预期的行为。 +- 迪米特法则(Demeter Principle)又叫**最少知道原则**，即一个类对自己依赖的类知道的越少越好。也就是说，对于被依赖的类不管多么复杂，都尽量将逻辑封装在类的内部。对外除了提供的 public 方法，不对外泄露任何信息 -**里氏替换原则的重点在不影响原功能，而不是不覆盖原方法。** +#### 7. **合成/聚合复用原则（CARP, Composition over Inheritance）** -#### 6. 迪米特法则 +- 应该优先使用对象组合（Composition）而不是继承（Inheritance）。通过组合可以更加灵活地扩展系统，而继承有时会导致过于紧密的耦合和不必要的复杂性。 -1. 一个对象应该对其他对象保持最少的了解 -2. 类与类关系越密切，耦合度越大 -3. 迪米特法则(Demeter Principle)又叫**最少知道原则**，即一个类对自己依赖的类知道的越少越好。也就是说，对于被依赖的类不管多么复杂，都尽量将逻辑封装在类的内部。对外除了提供的 public 方法，不对外泄露任何信息 -4. 迪米特法则还有个更简单的定义：只与直接的朋友通信 -5. 直接的朋友：每个对象都会与其他对象有耦合关系，只要两个对象之间有耦合关系，我们就说这两个对象之间是朋友关系。耦合的方式很多，依赖，关联，组合，聚合等。其中，我们称出现成员变量，方法参数，方法返回值中的类为直接的朋友，而出现在局部变量中的类不是直接的朋友。也就是说，陌生的类最好不要以局部变量的形式出现在类的内部。 +#### 8. **高内聚低耦合** -#### 7. 合成复用原则 +- 高内聚：指一个类或模块内部的功能应该紧密相关，职责单一，减少对外部的依赖。 +- 低耦合：指类与类之间应该尽量减少直接依赖，通过接口或抽象类来解耦，增加模块间的独立性。 -组合/聚合复用原则就是在一个新的对象里面使用一些已有的对象，使之成为新对象的一部分; 新的对象通过向这些对象的委派达到复用已有功能的目的。 +这些设计原则不仅是面向对象设计的核心，也贯穿于很多设计模式的理念中，帮助开发者构建可维护、易扩展的系统结构。 -在面向对象的设计中，如果直接继承基类，会破坏封装，因为继承将基类的实现细节暴露给子类；如果基类的实现发生了改变，则子类的实现也不得不改变；从基类继承而来的实现是静态的，不可能在运行时发生改变，没有足够的灵活性。于是就提出了组合/聚合复用原则，也就是在实际开发设计中，**尽量使用组合/聚合，不要使用类继承** +设计模式原则，其实就是程序员在编程时，应当遵守的原则，也是各种设计模式的基础(即：设计模式为什么这样设计的依据) @@ -191,7 +188,7 @@ #### 基于UML1.5 的UML种类 -![UML-type](https://tva1.sinaimg.cn/large/006tNbRwly1gbezv0jdfcj30sr0pin0a.jpg) +![UML-type](https://img.starfish.ink/design-patterns/UML-type.png) @@ -200,11 +197,11 @@ 类图（ClassDiagram）是用来显示系统中的类、接口、协作以及它们之间的静态结构和关系的一种静态模型，是我们 Java 猿帅们最需要掌握的。它主要用于描述软件系统的结构化设计，帮助人们简化对软件系统的理解，它是系统分析与设计阶段的重要产物，也是系统编码与测试的重要模型依据。 类图中的类可以通过某种编程语言直接实现。类图在软件系统开发的整个生命周期都是有效的，它是面向对象系统的建模中最常见的图。 -![Class Diagram Example](https://tva1.sinaimg.cn/large/006tNbRwly1gbesjtiy6nj30ki0b1dgc.jpg) +![](https://img.starfish.ink/design-patterns/class-diagram.jpg) #### 类图表示法 -![UML Class Diagram Example](https://tva1.sinaimg.cn/large/006tNbRwly1gbesl8l4hgj309603f3yq.jpg) ![Visibilitiy Example (Attribute)](https://online.visual-paradigm.com/images/tutorials/class-diagram-tutorial/06-attributes-visibilities.png) + ![](https://img.starfish.ink/design-patterns/06-attributes-visibilities.png) ### 类之间的关系 @@ -227,11 +224,11 @@ **单向关联** -![](https://tva1.sinaimg.cn/large/006tNbRwly1gbewpnx59rj308e015a9t.jpg) +![](https://img.starfish.ink/design-patterns/class-dependency.jpg) **双向关联** -![](https://tva1.sinaimg.cn/large/006tNbRwly1gbewprnfy5j308e0150sh.jpg) +![](https://img.starfish.ink/design-patterns/class-association2.jpg) **多重关联** @@ -242,28 +239,28 @@ 聚合（Aggregation）关系表示的是整体和部分的关系，是 has-a 的关系，整体与部分可以分开，是关联关系的特例，是强关联关系。 聚合关系也是通过成员对象来实现的，其中成员对象是整体对象的一部分，但是成员对象可以脱离整体对象而独立存在。例如，学校与老师的关系，学校包含老师，但如果学校停办了，老师依然存在。 -![](https://tva1.sinaimg.cn/large/006tNbRwly1gbezv6lm32j30f504k74l.jpg) +![](https://img.starfish.ink/design-patterns/class-aggregation.jpg) #### 4.组合关系 组合（Composition）关系也是关联关系的一种特例，也表示类之间的整体与部分的关系，但它是一种更强烈的聚合关系，不可以分开，是 cxmtains-a 关系。 在组合关系中，整体对象可以控制部分对象的生命周期，一旦整体对象不存在，部分对象也将不存在，部分对象不能脱离整体对象而存在。例如，公司和部门。 -![](https://tva1.sinaimg.cn/large/006tNbRwly1gbezvb57w5j30e303zjre.jpg) +![](https://img.starfish.ink/design-patterns/class-composition.jpg) #### 5.泛化关系 泛化（Generalization）关系是对象之间耦合度最大的一种关系，表示一般与特殊的关系，是父类与子类之间的关系，是一种**继承关系**，是 is-a 的关系。 在 UML 类图中，泛化关系用带空心三角箭头的实线来表示，箭头从子类指向父类。在代码实现时，使用面向对象的继承机制来实现泛化关系。 -![Abstract Class and Method Example](https://tva1.sinaimg.cn/large/006tNbRwly1gbewgz6t91j30c60563yi.jpg) +![](https://img.starfish.ink/design-patterns/class-generalization.jpg) #### 6.实现关系 实现（Realization）关系是接口与实现类之间的关系。在这种关系中，类实现了接口，类中的操作实现了接口中所声明的所有的抽象操作。 在 UML 类图中，实现关系使用带空心三角箭头的虚线来表示，箭头从实现类指向接口。 - ![UML Realization Example](https://tva1.sinaimg.cn/large/006tNbRwly1gbeweuf8pbj307704fdfo.jpg) + ![](https://img.starfish.ink/design-patterns/class-realization.jpg) @@ -271,8 +268,6 @@ ## 参考 -https://zhuanlan.zhihu.com/p/44518805 - -https://www.edrawsoft.cn/uml-diagram-introduction/ - -http://c.biancheng.net/view/1319.html \ No newline at end of file +- https://zhuanlan.zhihu.com/p/44518805 +- https://www.edrawsoft.cn/uml-diagram-introduction/ +- http://c.biancheng.net/view/1319.html \ No newline at end of file diff --git a/docs/design-pattern/Facade-Pattern.md b/docs/design-pattern/Facade-Pattern.md new file mode 100644 index 0000000000..c0beb0a7ce --- /dev/null +++ b/docs/design-pattern/Facade-Pattern.md @@ -0,0 +1,131 @@ +--- +title: 外观模式 +date: 2022-10-09 +tags: + - Design Patterns +categories: Design Patterns +--- + +![](https://cdn.pixabay.com/photo/2024/06/20/16/45/door-8842550_1280.jpg) + +> 之前介绍过装饰者模式和适配器模式，我们知道适配器模式是如何将一个类的接口转换成另一个符合客户期望的接口的。但 Java 中要实现这一点，必须将一个不兼容接口的对象包装起来，变成兼容的对象。 +> +> - 装饰模式：不改变接口，但加入责任 +> - 适配器模式：将一个接口转换为另一个接口 +> - 外观模式：让接口更简单 +> + +### 问题 + +假设你必须在代码中使用某个复杂的库或框架中的众多对象。 正常情况下，你需要负责所有对象的初始化工作、 管理其依赖关系并按正确的顺序执行方法等。 + +最终， 程序中类的业务逻辑将与第三方类的实现细节紧密耦合， 使得理解和维护代码的工作很难进行。 + + + +外观类为包含许多活动部件的复杂子系统提供一个简单的接口。 与直接调用子系统相比， 外观提供的功能可能比较有限， 但它却包含了客户端真正关心的功能。 + +如果你的程序需要与包含几十种功能的复杂库整合， 但只需使用其中非常少的功能， 那么使用外观模式会非常方便。 + + + +### 真实世界类比 + +当你通过电话给商店下达订单时， 接线员就是该商店的所有服务和部门的外观。 接线员为你提供了一个同购物系统、 支付网关和各种送货服务进行互动的简单语音接口。 + +![电话购物的示例](https://refactoringguru.cn/images/patterns/diagrams/facade/live-example-zh.png) + + + +### 定义 + +门面模式，也叫外观模式，英文全称是 Facade Design Pattern。在 GoF 的《设计模式》一书中，门面模式是这样定义的： + +> Provide a unified interface to a set of interfaces in a subsystem. Facade Pattern defines a higher-level interface that makes the subsystem easier to use. + +翻译成中文就是：门面模式为子系统提供一组统一的接口，定义一组高层接口让子系统更易用。 + +### 外观模式结构 + +![外观设计模式的结构](https://refactoringguru.cn/images/patterns/diagrams/facade/structure.png) + +- **外观** （Facade） 提供了一种访问特定子系统功能的便捷方式， 其了解如何重定向客户端请求， 知晓如何操作一切活动部件。 + +- 创建**附加外观** （Additional Facade） 类可以避免多种不相关的功能污染单一外观， 使其变成又一个复杂结构。 客户端和其他外观都可使用附加外观。 + +- **复杂子系统** （Complex Subsystem） 由数十个不同对象构成。 如果要用这些对象完成有意义的工作， 你必须深入了解子系统的实现细节， 比如按照正确顺序初始化对象和为其提供正确格式的数据。 + + 子系统类不会意识到外观的存在， 它们在系统内运作并且相互之间可直接进行交互。 + +- **客户端** （Client） 使用外观代替对子系统对象的直接调用。 + + + +#### Coding + + + + +### 外观模式适合应用场景 + +如果你需要一个指向复杂子系统的直接接口， 且该接口的功能有限， 则可以使用外观模式。 + +子系统通常会随着时间的推进变得越来越复杂。 即便是应用了设计模式， 通常你也会创建更多的类。 尽管在多种情形中子系统可能是更灵活或易于复用的， 但其所需的配置和样板代码数量将会增长得更快。 为了解决这个问题， 外观将会提供指向子系统中最常用功能的快捷方式， 能够满足客户端的大部分需求。 + +如果需要将子系统组织为多层结构， 可以使用外观。 + +创建外观来定义子系统中各层次的入口。 你可以要求子系统仅使用外观来进行交互， 以减少子系统之间的耦合。 + +让我们回到视频转换框架的例子。 该框架可以拆分为两个层次： 音频相关和视频相关。 你可以为每个层次创建一个外观， 然后要求各层的类必须通过这些外观进行交互。 这种方式看上去与[中介者](https://refactoringguru.cn/design-patterns/mediator)模式非常相似。 + + + +### 再来认识外观模式 + +> 外观模式也叫门面模式。门面模式原理和实现都特别简单，应用场景也比较明确，**主要在接口设计方面使用** +> +> 看到外观模式的实现，可能有朋友会说，这他么不就是把原来客户端的代码搬到了 Facade 里面吗，没什么大变化 + +#### 外观模式目的 + +外观模式相当于屏蔽了外部客户端和系统内部模块的交互 + +外观模式的目的不是给系统添加新的功能接口，而是为了让外部减少与子系统内多个模块的交互，松散耦合，从而让外部能更简单的使用子系统。 + +当然即使有了外观，如果需要的话，我们也可以直接调用具体模块功能。 + +#### 实现方式 + +1. 考虑能否在现有子系统的基础上提供一个更简单的接口。 如果该接口能让客户端代码独立于众多子系统类， 那么你的方向就是正确的。 +2. 在一个新的外观类中声明并实现该接口。 外观应将客户端代码的调用重定向到子系统中的相应对象处。 如果客户端代码没有对子系统进行初始化， 也没有对其后续生命周期进行管理， 那么外观必须完成此类工作。 +3. 如果要充分发挥这一模式的优势， 你必须确保所有客户端代码仅通过外观来与子系统进行交互。 此后客户端代码将不会受到任何由子系统代码修改而造成的影响， 比如子系统升级后， 你只需修改外观中的代码即可。 +4. 如果外观变得过于臃肿， 你可以考虑将其部分行为抽取为一个新的专用外观类。 + + + +### 外观模式优缺点 + +- 你可以让自己的代码独立于复杂子系统。 + +- 外观可能成为与程序中所有类都耦合的上帝对象。 + + + +### 与其他模式的关系 + +- 外观模式为现有对象定义了一个新接口， [适配器模式](https://refactoringguru.cn/design-patterns/adapter)则会试图运用已有的接口。 *适配器*通常只封装一个对象， *外观*通常会作用于整个对象子系统上。 +- 当只需对客户端代码隐藏子系统创建对象的方式时， 你可以使用[抽象工厂模式](https://refactoringguru.cn/design-patterns/abstract-factory)来代替[外观](https://refactoringguru.cn/design-patterns/facade)。 +- [享元模式](https://refactoringguru.cn/design-patterns/flyweight)展示了如何生成大量的小型对象， [外观](https://refactoringguru.cn/design-patterns/facade)则展示了如何用一个对象来代表整个子系统。 +- [外观](https://refactoringguru.cn/design-patterns/facade)和[中介者模式](https://refactoringguru.cn/design-patterns/mediator)的职责类似： 它们都尝试在大量紧密耦合的类中组织起合作。 + - *外观*为子系统中的所有对象定义了一个简单接口， 但是它不提供任何新功能。 子系统本身不会意识到外观的存在。 子系统中的对象可以直接进行交流。 + - *中介者*将系统中组件的沟通行为中心化。 各组件只知道中介者对象， 无法直接相互交流。 +- [外观](https://refactoringguru.cn/design-patterns/facade)类通常可以转换为[单例模式](https://refactoringguru.cn/design-patterns/singleton)类， 因为在大部分情况下一个外观对象就足够了。 +- [外观](https://refactoringguru.cn/design-patterns/facade)与[代理模式](https://refactoringguru.cn/design-patterns/proxy)的相似之处在于它们都缓存了一个复杂实体并自行对其进行初始化。 *代理*与其服务对象遵循同一接口， 使得自己和服务对象可以互换， 在这一点上它与*外观*不同。 + + + +### 参考与感谢 +- 《图解 Java 设计模式》 +- 《Head First设计模式》 +- https://refactoringguru.cn/design-patterns/ +- https://juejin.im/post/5ba28986f265da0abc2b6084#heading-12 \ No newline at end of file diff --git a/docs/design-pattern/Factory-Pattern.md b/docs/design-pattern/Factory-Pattern.md index a102a84c79..cbc7533cfe 100644 --- a/docs/design-pattern/Factory-Pattern.md +++ b/docs/design-pattern/Factory-Pattern.md @@ -1,4 +1,12 @@ -# 工厂模式——我有不止一个对象 +--- +title: 工厂模式——我有不止一个对象 +date: 2023-09-12 +tags: + - Design Pattern +categories: Design Pattern +--- + +![](https://img.starfish.ink/design-pattern/banner-factory.jpg) > 3年工作经验是吧？ > @@ -16,8 +24,6 @@ 在工厂模式中，我们在创建对象时不会对客户端暴露创建逻辑，并且是通过使用一个共同的接口来指向新创建的对象。 - - ### 工厂模式可以分为三类： @@ -26,7 +32,7 @@ - 工厂方法模式（Factory Method） - 抽象工厂模式（Abstract Factory） -简单工厂其实不是一个标准的的设计模式。GOF 23种设计模式中只有「工厂方法模式」与「抽象工厂模式」。简单工厂模式可以看为工厂方法模式的一种特例，为了统一整理学习，就都归为工厂模式。 +简单工厂其实不是一个标准的的设计模式。GOF 23 种设计模式中只有「工厂方法模式」与「抽象工厂模式」。简单工厂模式可以看为工厂方法模式的一种特例，为了统一整理学习，就都归为工厂模式。 这三种工厂模式在设计模式的分类中都属于**创建型模式**，三种模式从上到下逐步抽象。 @@ -38,7 +44,7 @@ 创建型模式隐藏了类的实例的创建细节，通过隐藏对象如何被创建和组合在一起达到使整个系统独立的目的。 -工厂模式是创建型模式中比较重要的。工厂模式的主要功能就是帮助我们实例化对象。之所以名字中包含工厂模式四个字，是因为对象的实例化过程是通过工厂实现的，是用工厂代替new操作的。 +工厂模式是创建型模式中比较重要的。工厂模式的主要功能就是帮助我们实例化对象。之所以名字中包含工厂模式四个字，是因为对象的实例化过程是通过工厂实现的，是用工厂代替 new 操作的。 ### 工厂模式优点： @@ -50,7 +56,7 @@ 不管是简单工厂模式，工厂方法模式还是抽象工厂模式，他们具有类似的特性，所以他们的适用场景也是类似的。 -首先，作为一种创建类模式，在任何需要生成**复杂对象**的地方，都可以使用工厂方法模式。有一点需要注意的地方就是复杂对象适合使用工厂模式，而简单对象，特别是只需要通过new就可以完成创建的对象，无需使用工厂模式。如果使用工厂模式，就需要引入一个工厂类，会增加系统的复杂度。 +首先，作为一种创建类模式，在任何需要生成**复杂对象**的地方，都可以使用工厂方法模式。有一点需要注意的地方就是复杂对象适合使用工厂模式，而简单对象，特别是只需要通过 new 就可以完成创建的对象，无需使用工厂模式。如果使用工厂模式，就需要引入一个工厂类，会增加系统的复杂度。 其次，工厂模式是一种典型的**解耦模式**，迪米特法则在工厂模式中表现的尤为明显。假如调用者自己组装产品需要增加依赖关系时，可以考虑使用工厂模式。将会大大降低对象之间的耦合度。 @@ -146,9 +152,7 @@ public static void main(String[] args) { - Product：抽象类产品， 它是工厂类所创建的所有对象的父类，封装了各种产品对象的公有方法，它的引入将提高系统的灵活性，使得在工厂类中只需定义一个通用的工厂方法，因为所有创建的具体产品对象都是其子类对象 - ConcreteProduct：具体产品， 它是简单工厂模式的创建目标，所有被创建的对象都充当这个角色的某个具体类的实例。它要实现抽象产品中声明的抽象方法 -#### UML类图 - -![](https://tva1.sinaimg.cn/large/00831rSTly1gcp76z5g27j30li0fpjsv.jpg) +![](https://img.starfish.ink/design-pattern/easy-factory-uml.png) #### 实例 @@ -199,7 +203,7 @@ public static void main(String[] args) { ### 1.3 简单工厂模式存在的问题 -当我们需要增加一种计算时，例如开平方。这个时候我们需要先定义一个类继承Operation类，其中实现平方的代码。除此之外我们还要修改 OperationFactory 类的代码，增加一个case。这显然是**违背开闭原则**的。可想而知对于新产品的加入，工厂类是很被动的。 +当我们需要增加一种计算时，例如开平方。这个时候我们需要先定义一个类继承 Operation 类，其中实现平方的代码。除此之外我们还要修改 OperationFactory 类的代码，增加一个 case。这显然是**违背开闭原则**的。可想而知对于新产品的加入，工厂类是很被动的。 我们举的例子是最简单的情况。而在实际应用中，很可能产品是一个多层次的树状结构。 简单工厂可能就不太适用了。 @@ -236,11 +240,11 @@ public static void main(String[] args) { #### UML类图 -![](https://tva1.sinaimg.cn/large/00831rSTly1gcp774606hj30za0hagmu.jpg) +![](https://img.starfish.ink/design-pattern/factory-uml.png) #### 实例 -从UML类图可以看出，每种产品实现，我们都要增加一个继承于工厂接口 `IFactory` 的工厂类 `Factory` ，修改简单工厂模式代码中的工厂类如下： +从 UML 类图可以看出，每种产品实现，我们都要增加一个继承于工厂接口 `IFactory` 的工厂类 `Factory` ，修改简单工厂模式代码中的工厂类如下： ```java //工厂接口 @@ -314,7 +318,7 @@ public class Client { #### 使用场景 -- 日志记录器：记录可能记录到本地硬盘、系统事件、远程服务器等，用户可以选择记录日志到什么地方。 +- 日志记录器：日志可能记录到本地硬盘、系统事件、远程服务器等，用户可以选择记录日志到什么地方。 - 数据库访问，当用户不知道最后系统采用哪一类数据库，以及数据库可能有变化时。 - 设计一个连接服务器的框架，需要三个协议，"POP3"、"IMAP"、"HTTP"，可以把这三个作为产品类，共同实现一个接口。 - 比如 Hibernate 换数据库只需换方言和驱动就可以 @@ -371,17 +375,17 @@ public class Client { #### UML类图 -![](https://tva1.sinaimg.cn/large/00831rSTly1gcp778i6yhj31gu0u0ahy.jpg) +![](https://img.starfish.ink/design-pattern/abstract-facotry-uml.png) #### 实例 我把维基百科的例子改下用于理解，假设我们要生产两种产品，键盘（Keyboard）和鼠标（Mouse） ，每一种产品都支持多种系列，比如 Mac 系列和 Windows 系列。这样每个系列的产品分别是 MacKeyboard WinKeyboard, MacMouse, WinMouse 。为了可以在运行时刻创建一个系列的产品族，我们可以为每个系列的产品族创建一个工厂 MacFactory 和 WinFactory 。每个工厂都有两个方法 CreateMouse 和 CreateKeyboard 并返回对应的产品，可以将这两个方法抽象成一个接口 HardWare 。这样在运行时刻我们可以选择创建需要的产品系列。 -![](https://tva1.sinaimg.cn/large/00831rSTly1gcp77dgtchj31gu0u0ag5.jpg) +![](https://img.starfish.ink/design-pattern/abstract-factory-demo.png) 1. 抽象产品 -2. ```java + ```java public interface Keyboard { void input(); } @@ -391,7 +395,7 @@ public class Client { ``` 2. 具体产品 - + ```java //具体产品 public class MacKeyboard implements Keyboard { @@ -422,7 +426,7 @@ public class Client { } } ``` - + 3. 抽象工厂 ```java @@ -499,7 +503,7 @@ public class Client { ### 3.4 抽象工厂模式总结 -抽象工厂模式是工厂方法模式的进一步延伸，由于它提供了功能更为强大的工厂类并且具备较好的可扩展性，在软件开发中得以广泛应用，尤其是在一些框架和API类库的设计中，例如在Java语言的AWT（抽象窗口工具包）中就使用了抽象工厂模式，它使用抽象工厂模式来实现在不同的操作系统中应用程序呈现与所在操作系统一致的外观界面。抽象工厂模式也是在软件开发中最常用的设计模式之一。 +抽象工厂模式是工厂方法模式的进一步延伸，由于它提供了功能更为强大的工厂类并且具备较好的可扩展性，在软件开发中得以广泛应用，尤其是在一些框架和 API 类库的设计中，例如在 Java 语言的AWT（抽象窗口工具包）中就使用了抽象工厂模式，它使用抽象工厂模式来实现在不同的操作系统中应用程序呈现与所在操作系统一致的外观界面。抽象工厂模式也是在软件开发中最常用的设计模式之一。 **优点：** @@ -539,12 +543,8 @@ public class Client { ## 参考 -https://blog.csdn.net/lovelion/article/details/17517213 - -https://wiki.jikexueyuan.com/project/java-design-pattern/abstract-factory-pattern.html - -https://blog.csdn.net/lovelion/article/details/17517213 - +- https://blog.csdn.net/lovelion/article/details/17517213 +- https://wiki.jikexueyuan.com/project/java-design-pattern/abstract-factory-pattern.html -![](https://i.loli.net/2020/03/19/AkRqgTo6y5crBSx.png) \ No newline at end of file +- https://blog.csdn.net/lovelion/article/details/17517213 diff --git a/docs/design-pattern/Observer-Pattern.md b/docs/design-pattern/Observer-Pattern.md index fbb45c56e7..eac7e2d49f 100644 --- a/docs/design-pattern/Observer-Pattern.md +++ b/docs/design-pattern/Observer-Pattern.md @@ -1,4 +1,12 @@ -> 文章收录在 GitHub [JavaKeeper](https://github.com/Jstarfish/JavaKeeper) ，N线互联网开发必备技能兵器谱 +--- +title: 观察者模式 +date: 2022-11-09 +tags: + - Design Patterns +categories: Design Patterns +--- + +![](https://images.unsplash.com/photo-1463310127152-33b375103141?w=1200&auto=format&fit=crop&q=60&ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxzZWFyY2h8MzB8fG9ic2VydmVyfGVufDB8fDB8fHww) 在软件系统中经常会有这样的需求：如果一个对象的状态发生改变，某些与它相关的对象也要随之做出相应的变化。 @@ -6,9 +14,9 @@ - 微信公众号，如果一个用户订阅了某个公众号，那么便会收到公众号发来的消息，那么，公众号就是『被观察者』，而用户就是『观察者』 - 气象站可以将每天预测到的温度、湿度、气压等以公告的形式发布给各种第三方网站，如果天气数据有更新，要能够实时的通知给第三方，这里的气象局就是『被观察者』，第三方网站就是『观察者』 -- MVC 模式中的模型与视图的关系也属于观察与被观察 +- MVC 模式中的模型与视图的关系也属于观察与被观察关系 -观察者模式是使用频率较高的设计模式之一。 +观察者模式是使用频率较高的设计模式之一，也被称为发布订阅模式。 ![](https://img01.sogoucdn.com/app/a/100520093/e18d20c94006dfe0-9eef65073f0f6be0-688789934e19c96097ccf76b41f77cf4.jpg) @@ -26,23 +34,23 @@ 细究的话，发布订阅和观察者有些不同，可以理解成发布订阅模式属于广义上的观察者模式。 -![img](https://tva1.sinaimg.cn/large/00831rSTly1gcyfkrn2s3j30ip0badgh.jpg) +![](https://howtodoinjava.com/wp-content/uploads/2019/01/observer-pattern.png) ## 角色 -- **Subject（目标）**：被观察者，它是指被观察的对象。 从类图中可以看到，类中有一个用来存放观察者对象的Vector 容器（之所以使用Vector而不使用List，是因为多线程操作时，Vector在是安全的，而List则是不安全的），这个Vector容器是被观察者类的核心，另外还有三个方法：attach方法是向这个容器中添加观察者对象；detach方法是从容器中移除观察者对象；notify方法是依次调用观察者对象的对应方法。这个角色可以是接口，也可以是抽象类或者具体的类，因为很多情况下会与其他的模式混用，所以使用抽象类的情况比较多。 +- **Subject（目标）**：被观察者，它是指被观察的对象。 从类图中可以看到，类中有一个用来存放观察者对象的Vector 容器（Vector在是安全的，而List则是不安全的），这个 Vector 容器是被观察者类的核心，另外还有三个方法：attach 方法是向这个容器中添加观察者对象；detach 方法是从容器中移除观察者对象；notify 方法是依次调用观察者对象的对应方法。这个角色可以是接口，也可以是抽象类或者具体的类，因为很多情况下会与其他的模式混用，所以使用抽象类的情况比较多。 - **ConcreteSubject（具体目标）**：具体目标是目标类的子类，通常它包含经常发生改变的数据，当它的状态发生改变时，向它的各个观察者发出通知。同时它还实现了在目标类中定义的抽象业务逻辑方法（如果有的话）。如果无须扩展目标类，则具体目标类可以省略。 - **Observer（观察者）**：观察者将对观察目标的改变做出反应，观察者一般定义为**接口**，该接口声明了更新数据的方法 `update()`，因此又称为**抽象观察者**。 -- **ConcreteObserver（具体观察者）**：在具体观察者中维护一个指向具体目标对象的引用，它存储具体观察者的有关状态，这些状态需要和具体目标的状态保持一致；它实现了在抽象观察者 Observer 中定义的 update()方法。通常在实现时，可以调用具体目标类的 attach() 方法将自己添加到目标类的集合中或通过 detach() 方法将自己从目标类的集合中删除。 +- **ConcreteObserver（具体观察者）**：在具体观察者中维护一个指向具体目标对象的引用，它存储具体观察者的有关状态，这些状态需要和具体目标的状态保持一致；它实现了在抽象观察者 Observer 中定义的 `update()` 方法。通常在实现时，可以调用具体目标类的 `attach()` 方法将自己添加到目标类的集合中或通过 `detach()` 方法将自己从目标类的集合中删除。 ## 类图 -![](https://tva1.sinaimg.cn/large/00831rSTly1gcxwtvpenhj311t0lnacu.jpg) +![](https://img.starfish.ink/design-patterns/observer-uml.png) 再记录下 UML 类图的注意事项，这里我的 Subject 是**抽象方法**，所以用***斜体***，抽象方法也要用斜体，具体的各种箭头意义，我之前也总结过《设计模式前传——学设计模式前你要知道这些》（被网上各种帖子毒害过的自己，认真记录~~~）。 @@ -146,17 +154,17 @@ public class Client { ## 应用 -### JDK中的观察者模式 +### JDK 中的观察者模式 -观察者模式在 Java 语言中的地位非常重要。在 JDK 的 java.util 包中，提供了 Observable 类以及 Observer 接口，它们构成了 JDK 对观察者模式的支持（可以去查看下源码，写的比较严谨）。but，在 Java9 被弃用了。 +观察者模式在 Java 语言中的地位非常重要。在 JDK 的 `java.util` 包中，提供了 Observable 类以及 Observer 接口，它们构成了 JDK 对观察者模式的支持（可以去查看下源码，写的比较严谨）。but，在 Java9 被弃用了。 ### Spring 中的观察者模式 -Spring 事件驱动模型也是观察者模式很经典的应用。就是我们常见的项目中最常见的事件监听器。 +Spring 事件驱动模型也是观察者模式很经典的应用。就是我们项目中最常见的事件监听器。 #### 1. Spring 中观察者模式的四个角色 -1. **事件：ApplicationEvent** 是所有事件对象的父类。ApplicationEvent 继承自 jdk 的 EventObject, 所有的事件都需要继承 ApplicationEvent, 并且通过 source 得到事件源。 +1. **事件：ApplicationEvent** 是所有事件对象的父类。ApplicationEvent 继承自 jdk 的 EventObject，所有的事件都需要继承 ApplicationEvent，并且通过 source 得到事件源。 Spring 也为我们提供了很多内置事件，`ContextRefreshedEvent`、`ContextStartedEvent`、`ContextStoppedEvent`、`ContextClosedEvent`、`RequestHandledEvent`。 @@ -257,7 +265,7 @@ ListenerB received 设计模式真的只是一种设计思想，不需要非得有多个观察者才可以用观察者模式，只有一个观察者，我也要用。 -再举个栗子，我是做广告投放的嘛（广告投放的商品文件一般为 xml），假如我的广告位有些空闲流量，这我得利用起来呀，所以我就从淘宝客或者拼夕夕的多多客上通过开放的 API 获取一些，这个时候我也可以用观察者模式，每次请求 10 万条商品，我就生成一个新的商品文件，这个时候我也可以用观察者模式，获取商品的类是被观察者，写商品文件的是观察者，当商品够10万条了，就通知观察者重新写到一个新的文件。 +再举个栗子，我是做广告投放的嘛（广告投放的商品文件业内一般为 xml），假如我的广告位有些空闲流量，这我得利用起来呀，所以我就从淘宝客或者拼夕夕的多多客上通过开放的 API 获取一些，这个时候我也可以用观察者模式，每次请求 10 万条商品，我就生成一个新的商品文件，这个时候我也可以用观察者模式，获取商品的类是被观察者，写商品文件的是观察者，当商品够10万条了，就通知观察者重新写到一个新的文件。 大佬可能觉这么实现有点费劲，不用设计模式也好，或者用消息队列也好，其实都只是一种手段，选择适合自己业务的，开心就好。 @@ -265,10 +273,6 @@ ListenerB received ## 参考 -https://design-patterns.readthedocs.io/zh_CN/latest/behavioral_patterns/observer.html - -https://www.cnblogs.com/jmcui/p/11054756.html - - +- https://design-patterns.readthedocs.io/zh_CN/latest/behavioral_patterns/observer.html -![](https://user-gold-cdn.xitu.io/2020/3/20/170f5beacffbc730?w=750&h=390&f=jpeg&s=29031) \ No newline at end of file +- https://www.cnblogs.com/jmcui/p/11054756.html diff --git a/docs/design-pattern/Pipeline-Pattern.md b/docs/design-pattern/Pipeline-Pattern.md new file mode 100755 index 0000000000..7c67f60595 --- /dev/null +++ b/docs/design-pattern/Pipeline-Pattern.md @@ -0,0 +1,274 @@ +--- +title: 管道模式 +date: 2021-10-09 +tags: + - Design Patterns +categories: Design Patterns +--- + +![](https://img.starfish.ink/design-patterns/008i3skNly1gt0lx0zc1dj30rs0ijwhs.jpg) + + + +## 一、开场 + +假设我们有这样的一个需求，读取文件内容，并过滤包含 “hello” 的字符串，然后将其反转 + +![](https://img.starfish.ink/design-patterns/hello-file.png) + +Linux 一行搞定 + +```bash +cat hello.txt | grep "hello" | rev +``` + +用世界上最好语言 `Java` 实现也很简单 + +```java +File file = new File("/Users/starfish/Documents/hello.txt"); + +String content = FileUtils.readFileToString(file,"UTF-8"); + +List helloStr = Stream.of(content).filter(s -> s.contains("hello")).collect(Collectors.toList()); + +System.out.println(new StringBuilder(String.join("",helloStr)).reverse().toString()); +``` + + + +再假设我们上边的场景是在一个大型系统中，有这样的数据流需要多次进行复杂的逻辑处理，还是简单粗暴的把一系列流程像上边那样放在一个大组件中吗？ + +这样的设计完全违背了单一职责原则，我们在增改，或者减少一些处理逻辑的时候，就必须对整个组件进行改动。可扩展性和可重用性几乎没有~~ + +那有没有一种模式可以将整个处理流程进行详细划分，划分出的每个小模块互相独立且各自负责一小段逻辑处理，这些小模块可以按顺序连起来，前一模块的输出作为后一模块的输入，最后一个模块的输出为最终的处理结果呢？ + +如此一来修改逻辑时只针对某个模块修改，添加或减少处理逻辑也可细化到某个模块颗粒度，并且每个模块可重复利用，可重用性大大增强。 + +恩，这就是我们要说的管道模式 + +![](https://img.starfish.ink/design-patterns/pipeline-pattern-csharp-uml.png) + + + + + +## 二、定义 + +管道模式（Pipeline Pattern） 是责任链模式（Chain of Responsibility Pattern）的常用变体之一。 + +顾名思义，管道模式就像一条管道把多个对象连接起来，整体看起来就像若干个阀门嵌套在管道中，而处理逻辑就放在阀门上，需要处理的对象进入管道后，分别经过各个阀门，每个阀门都会对进入的对象进行一些逻辑处理，经过一层层的处理后从管道尾出来，此时的对象就是已完成处理的目标对象。 + +![](https://img.starfish.ink/design-patterns/pipeline-filter-pattern-csharp-implementations2.jpg) + +管道模式用于将复杂的进程分解成多个独立的子任务。每个独立的任务都是可复用的，因此这些任务可以被组合成复杂的进程。 + +> PS：纯的责任链模式在链上只会有一个处理器用于处理数据，而管道模式上多个处理器都会处理数据。 + + + +## 三、角色 + +管道模式：对于管道模式来说，有 3 个对象： + +- 阀门：处理数据的节点，或者叫过滤器、阶段 +- 管道：组织各个阀门 +- 客户端：构造管道，并调用 + + + +## 四、实例 + +程序员还是看代码消化才快些，我们用管道模式实现下文章开头的小需求 + +### 1、处理器（管道的各个阶段） + +```java +public interface Handler { + O process(I input); +} +``` + +### 2、定义具体的处理器（阀门） + +```java +public class FileProcessHandler implements Handler{ + + @Override + public String process(File file) { + System.out.println("===文件处理==="); + try{ + return FileUtils.readFileToString(file,"UTF-8"); + }catch (IOException e){ + e.printStackTrace(); + } + return null; + } +} +``` + +```java +public class CharacterFilterHandler implements Handler { + + @Override + public String process(String input) { + System.out.println("===字符过滤==="); + List hello = Stream.of(input).filter(s -> s.contains("hello")).collect(Collectors.toList()); + return String.join("",hello); + } +} +``` + +```java +public class CharacterReverseHandler implements Handler{ + + @Override + public String process(String input) { + System.out.println("===反转字符串==="); + return new StringBuilder(input).reverse().toString(); + } +} +``` + +### 3、管道 + +```java +public class Pipeline { + + private final Handler currentHandler; + + Pipeline(Handler currentHandler) { + this.currentHandler = currentHandler; + } + + Pipeline addHandler(Handler newHandler) { + return new Pipeline<>(input -> newHandler.process(currentHandler.process(input))); + } + + O execute(I input) { + return currentHandler.process(input); + } +} +``` + +### 4、 客户端使用 + +```java +import lombok.val; +public class ClientTest { + + public static void main(String[] args) { + + File file = new File("/Users/apple/Documents/hello.txt"); + + val filters = new Pipeline<>(new FileProcessHandler()) + .addHandler(new CharacterFilterHandler()) + .addHandler(new CharacterReverseHandler()); + System.out.println(filters.execute(file)); + } +} +``` + +### 5、结果 + +![](https://img.starfish.ink/design-patterns/pipeline-result.png) + + + +### UML 类图 + +![](https://img.starfish.ink/design-patterns/pipeline-uml.png) + + + +产品他么的又来了，这次是删除 `hello.txt` 中的 world 字符 + +![](https://i04piccdn.sogoucdn.com/36a497573c1babc0) + +三下五除二，精通 shell 编程的我搞定了 + +```bash +cat hello.txt |grep hello |rev | tr -d 'world' +``` + +Java 怎么搞，你应该很清晰了吧 + + + +## 五、优缺点 + +> Pipeline 模式的核心思想是将一个任务处理分解为若干个处理阶段（Stage），其中每个处理阶段的输出作为下一个处理阶段的输入，并且各个处理阶段都有相应的工作者线程去执行相应的计算。因此，处理一批任务时，各个任务的各个处理阶段是并行（Parallel）的。通过并行计算，Pipeline 模式使应用程序能够充分利用多核 CPU 资源，提高其计算效率。 ——《Java 多线程编程实战指南》 +> + +#### 优点 + +- 将复杂的处理流程分解成独立的子任务，解耦上下游处理逻辑，也方便您对每个子任务的测试 +- 被分解的子任务还可以被不同的处理进程复用 +- 在复杂进程中添加、移除和替换子任务非常轻松，对已存在的进程没有任何影响，这就加大了该模式的扩展性和灵活性 +- 对于每个处理单元又可以打补丁，做监听。（这就是切面编程了） + + + +> 模式需要注意的东西 +> +> 1. Pipeline的深度：Pipeline 中 Pipe 的个数被称作 Pipeline 的深度。所以我们在用 Pipeline 的深度与 JVM 宿主机的 CPU 个数间的关系。如果 Pipeline 实例所处的任务多属于 CPU 密集型，那么深度最好不超过 Ncpu。如果 Pipeline 所处理的任务多属于 I/O 密集型，那么 Pipeline 的深度最好不要超过 2*Ncpu。 +> +> 2. 基于线程池的 Pipe：如果 Pipe 实例使用线程池，由于有多个 Pipe 实例，更容易出现线程死锁的问题，需要仔细考虑。 +> +> 3. 错误处理：Pipe 实例对其任务进行过程中跑出的异常可能需要相应 Pipe 实例之外进行处理。 +> +> 此时，处理方法通常有两种：一是各个 Pipe 实例捕获到异常后调用 PipeContext 实例的 handleError 进行错误处理。另一个是创建一个专门负责错我处理的 Pipe 实例，其他 Pipe 实例捕获异常后提交相关数据给该 Pipe 实例处理。 +> +> 4. 可配置的 Pipeline：Pipeline 模式可以用代码的方式将若干个 Pipe 实例添加，也可以用配置文件的方式实现动态方式添加 Pipe。 + + + + +## 六、Java Function + +如果，你的管道逻辑真的很简单，也直接用 `Java8` 提供的 `Function` 就，具体实现如下这样 + +```java + File file = new File("/Users/apple/Documents/hello.txt"); + + Function readFile = input -> { + System.out.println("===文件处理==="); + try{ + return FileUtils.readFileToString(input,"UTF-8"); + }catch (IOException e){ + e.printStackTrace(); + } + return null; + }; + + Function filterCharacter = input -> { + System.out.println("===字符过滤==="); + List hello = Stream.of(input).filter(s -> s.contains("hello")).collect(Collectors.toList()); + return String.join("",hello); + }; + + Function reverseCharacter = input -> { + System.out.println("===反转字符串==="); + return new StringBuilder(input).reverse().toString(); + }; + + final Function pipe = readFile + .andThen(filterCharacter) + .andThen(reverseCharacter); + + System.out.println(pipe.apply(file)); +``` + + + +## 最后 + +但是，并不是一碰到这种类似流式处理的任务就需要用管道，Pipeline 模式中各个处理阶段所用的工作者线程或者线程池，表示各个阶段的输入/输出对象的创建和一定（进出队列）都有其自身的时间和空间开销，所以使用 Pipeline 模式的时候需要考虑它所付出的代价。建议处理规模较大的任务，否则可能得不偿失。 + + + +## 参考 + +- https://java-design-patterns.com/patterns/pipeline/ +- https://developer.aliyun.com/article/778865 +- https://yasinshaw.com/articles/108 +- 《Java多线程编程实战指南（设计模式篇）》 \ No newline at end of file diff --git a/docs/design-pattern/Project-Design.md b/docs/design-pattern/Project-Design.md new file mode 100644 index 0000000000..4827b9bc1c --- /dev/null +++ b/docs/design-pattern/Project-Design.md @@ -0,0 +1,50 @@ +### **设计实现一个支持各种算法的限流框架** + +#### **需求分析** + +- 易用性方面，我们希望限流规则的配置、编程接口的使用都很简单。我们希望提供各种不同的限流算法，比如基于内存的单机限流算法、基于 Redis 的分布式限流算法，能够让使用者自由选择。除此之外，因为大部分项目都是基于 Spring 开发的，我们还希望限流框架能否非常方便地集成到使用 Spring 框架的项目中。 + +- 扩展性、灵活性方面，我们希望能够灵活地扩展各种限流算法。同时，我们还希望支持不同格式（JSON、YAML、XML 等格式）、不同数据源（本地文件配置或 Zookeeper 集中配置等）的限流规则的配置方式。 +- 性能方面，因为每个接口请求都要被检查是否限流，这或多或少会增加接口请求的响应时间。而对于响应时间比较敏感的接口服务来说，我们要让限流框架尽可能低延迟，尽可能减少对接口请求本身响应时间的影响 +- 容错性方面，接入限流框架是为了提高系统的可用性、稳定性，不能因为限流框架的异常，反过来影响到服务本身的可用性。所以，限流框架要有高度的容错性。比如，分布式限流算法依赖集中存储器 Redis。如果 Redis 挂掉了，限流逻辑无法正常运行，这个时候业务接口也要能正常服务才行。 + +#### **设计** + +**限流规则** + +框架需要定义限流规则的语法格式，包括调用方、接口、限流阈值、时间粒度这几个元素。框架用户按照这个语法格式来配置限流规则。 + +```yaml +configs: + - appId: app-1 + limits: + - api: /v1/user + limit: 100 + unit：60 + - api: /v1/order + limit: 50 + - appId: app-2 + limits: + - api: /v1/user + limit: 50 + - api: /v1/order + limit: 50 +``` + +**限流算法** + +常见的限流算法有：固定时间窗口限流算法、滑动时间窗口限流算法、令牌桶限流算法、漏桶限流算法。其中，固定时间窗口限流算法最简单。我们只需要选定一个起始时间起点，之后每来一个接口请求，我们都给计数器（记录当前时间窗口内的访问次数）加一，如果在当 + +前时间窗口内，根据限流规则（比如每秒钟最大允许 100 次接口请求），累加访问次数超过限流值（比如 100 次），就触发限流熔断，拒绝接口请求。当进入下一个时间窗口之后，计数器清零重新计数。 + +不过，固定时间窗口的限流算法的缺点也很明显。这种算法的限流策略过于粗略，无法应对两个时间窗口临界时间内的突发流量。 + +**限流模式** + +我们把限流模式分为两种：单机限流和分布式限流。 + +所谓单机限流，就是针对单个实例的访问频率进行限制。注意这里的单机并不是真的一台物理机器，而是一个服务实例，因为有可能一台物理机器部署多个实例。所谓的分布式限流，就是针对某个服务的多个实例的总的访问频率进行限制。 + +**集成使用** + +因为框架是需要集成到应用中使用的，我们希望框架尽可能低侵入，与业务代码松耦合，替换、删除起来也更容易些。 \ No newline at end of file diff --git a/docs/design-pattern/Prototype-Pattern.md b/docs/design-pattern/Prototype-Pattern.md index 6acc42f432..09689e97c5 100644 --- a/docs/design-pattern/Prototype-Pattern.md +++ b/docs/design-pattern/Prototype-Pattern.md @@ -1,3 +1,256 @@ -Prototype-Pattern +--- +title: 从原型模型到浅拷贝和深拷贝 +date: 2023-09-12 +tags: + - Design Pattern +categories: Design Pattern +--- -原型模型 \ No newline at end of file +![](https://img.starfish.ink/design-pattern/banner-prototype.jpg) + +> 如果你有一个对象， 并希望生成与其完全相同的一个复制品， 你该如何实现呢？ +> +> 首先， 你必须新建一个属于相同类的对象。 然后， 你必须遍历原始对象的所有成员变量， 并将成员变量值复制到新对象中。 + +![](https://img.starfish.ink/design-pattern/format.png) + +```java +for (int i = 0; i < 10; i++) { + Sheep sheep = new Sheep("肖恩"+i+"号",2+i,"白色"); + System.out.println(sheep.toString()); +} +``` + +这种方式是比较容易想到的，但是有几个不足 + +- 在创建新对象的时候，总是需要重新获取原始对象的属性，如果创建的对象比较复杂，效率会很低 +- 总是需要重新初始化对象，而不是动态地获得对象运行时的状态, 不够灵活 +- 另一方面，并非所有对象都能通过这种方式进行复制， 因为有些对象可能拥有私有成员变量， 它们在对象本身以外是不可见的 + +> 万物兼对象的 Java 中的所有类的根类 Object，提供了一个 `clone()` 方法，该方法可以将一个 Java 对象复制一份，但是需要实现 clone() 的类必须要实现一个接口 Cloneable，该接口表示该类能够复制且具有复制的能力。 +> +> 这就引出了原型模式。 + + + +## 基本介绍 + +1. 原型模式(Prototype模式)是指：用原型实例指定创建对象的种类，并且通过拷贝这些原型，创建新的对象 +2. **原型模式**是一种**创建型设计模式**， 使你能够复制已有对象， 而又无需使代码依赖它们所属的类 +3. 工作原理是：通过将一个原型对象传给那个要发动创建的对象，这个要发动创建的对象通过请求原型对象拷贝它们自己来实施创建，即 对象**.clone**() + +### 类图 + +![](https://img.starfish.ink/design-pattern/prototype-UML.png) + +- Prototype : **原型** （Prototype） 接口将对克隆方法进行声明 + + Java 中 Prototype 类需要具备以下两个条件 + + - **实现 Cloneable 接口**。在 Java 语言有一个 Cloneable 接口，它的作用只有一个，就是在运行时通知虚拟机可以安全地在实现了此接口的类上使用 clone 方法。在 Java 虚拟机中，只有实现了这个接口的类才可以被拷贝，否则在运行时会抛出 CloneNotSupportedException 异常 + - **重写 Object 类中的 clone 方法**。Java 中，所有类的父类都是 Object 类，Object 类中有一个 clone 方法，作用是返回对象的一个拷贝 + +- ConcretePrototype：**具体原型** （Concrete Prototype） 类将实现克隆方法。 除了将原始对象的数据复制到克隆体中之外， 该方法有时还需处理克隆过程中的极端情况， 例如克隆关联对象和梳理递归依赖等等。 + +- Client: 使用原型的客户端，首先要获取到原型实例对象，然后通过原型实例克隆自己，从而创建一个新的对象。 + + + +## 实例 + +> 我们用王二小放羊的例子写这个实例 + +### 1、原型类（实现 *Clonable*） + +```java +@Setter +@Getter +@NoArgsConstructor +@AllArgsConstructor +class Sheep implements Cloneable { + private String name; + private Integer age; + private String color; + + @Override + protected Sheep clone() { + Sheep sheep = null; + try { + sheep = (Sheep) super.clone(); + } catch (Exception e) { + System.out.println(e.getMessage()); + } + return sheep; + } +} +``` + +### 2、具体原型 + +按业务的不同实现不同的原型对象，假设现在主角是王二小，羊群里有山羊、绵羊一大群 + +```java +public class Goat extends Sheep{ + public void graze() { + System.out.println("山羊去吃草"); + } +} +``` + +```java +public class Lamb extends Sheep{ + public void graze() { + System.out.println("羔羊去吃草"); + } +} +``` + +### 3、客户端 + +```java +public class Client { + + static List sheepList = new ArrayList<>(); + public static void main(String[] args) { + Goat goat = new Goat(); + goat.setName("山羊"); + goat.setAge(3); + goat.setColor("灰色"); + for (int i = 0; i < 5; i++) { + sheepList.add(goat.clone()); + } + + Lamb lamb = new Lamb(); + lamb.setName("羔羊"); + lamb.setAge(2); + lamb.setColor("白色"); + for (int i = 0; i < 5; i++) { + sheepList.add(lamb.clone()); + System.out.println(lamb.hashCode()+","+lamb.clone().hashCode()); + } + + for (Sheep sheep : sheepList) { + System.out.println(sheep.toString()); + } +} +``` + + + +原型模式将克隆过程委派给被克隆的实际对象。 模式为所有支持克隆的对象声明了一个通用接口， 该接口让你能够克隆对象，同时又无需将代码和对象所属类耦合。 通常情况下，这样的接口中仅包含一个 `克隆`方法。 + +所有的类对 `克隆`方法的实现都非常相似。 该方法会创建一个当前类的对象， 然后将原始对象所有的成员变量值复制到新建的类中。 你甚至可以复制私有成员变量， 因为绝大部分编程语言都允许对象访问其同类对象的私有成员变量。 + +支持克隆的对象即为*原型*。 当你的对象有几十个成员变量和几百种类型时， 对其进行克隆甚至可以代替子类的构造。 + + + +## 优势 + +**使用原型模式创建对象比直接 new 一个对象在性能上要好的多，因为 Object 类的 clone 方法是一个本地方法，它直接操作内存中的二进制流，特别是复制大对象时，性能的差别非常明显**。 + +使用原型模式的另一个好处是简化对象的创建，使得创建对象就像我们在编辑文档时的复制粘贴一样简单。 + +因为以上优点，所以在需要重复地创建相似对象时可以考虑使用原型模式。比如需要在一个循环体内创建对象，假如对象创建过程比较复杂或者循环次数很多的话，使用原型模式不但可以简化创建过程，而且可以使系统的整体性能提高很多。 + + + +## 适用场景 + +[《Head First 设计模式》]("Head First 设计模式")是这么形容原型模式的：当创建给定类的实例的过程很昂贵或很复杂时，就是用原型模式。 + +如果你需要复制一些对象，同时又希望代码独立于这些对象所属的具体类，可以使用原型模式。 + +如果子类的区别仅在于其对象的初始化方式， 那么你可以使用该模式来减少子类的数量。别人创建这些子类的目的可能是为了创建特定类型的对象。 + + + +## 原型模式在 Spring 中的应用 + +我们都知道 Spring bean 默认是单例的，但是有些场景可能需要原型范围，如下 + +```xml + + + + + +``` + +同样，王二小还是有 10 只羊，感兴趣的也可以看下他们创建的对象是不是同一个 + +```java +public class Client { + public static void main(String[] args) { + ApplicationContext context = new ClassPathXmlApplicationContext("applicationContext.xml"); + for (int i = 0; i < 10; i++) { + Object bean = context.getBean("sheep"); + System.out.println(bean); + } + } +} +``` + +感兴趣的同学可以深入源码看下具体的实现，在 AbstractBeanFactory 的 `doGetBean()` 方法中 + +![](https://img.starfish.ink/design-pattern/prototype-demo-doGetBean.png) + + + +## 原型模式的注意事项 + +- 使用原型模式复制对象不会调用类的构造方法。因为对象的复制是通过调用 Object 类的 clone 方法来完成的，它直接在内存中复制数据，因此不会调用到类的构造方法。不但构造方法中的代码不会执行，甚至连访问权限都对原型模式无效。还记得单例模式吗？单例模式中，只要将构造方法的访问权限设置为 private 型，就可以实现单例。但是 clone 方法直接无视构造方法的权限，所以，**单例模式与原型模式是冲突的**，在使用时要特别注意。 +- 深拷贝与浅拷贝。Object 类的 clone方法只会拷贝对象中的基本的数据类型，对于数组、容器对象、引用对象等都不会拷贝，这就是浅拷贝。如果要实现深拷贝，必须将原型模式中的数组、容器对象、引用对象等另行拷贝。 + + + +## 浅拷贝和深拷贝 + +首先需要明白，浅拷贝和深拷贝都是针对一个已有对象的操作。 + +在 Java 中，除了**基本数据类型**（元类型）之外，还存在 **类的实例对象** 这个引用数据类型。而一般使用 『 **=** 』号做赋值操作的时候。对于基本数据类型，实际上是拷贝的它的值，但是对于对象而言，其实赋值的只是这个对象的引用，将原对象的引用传递过去，他们实际上还是指向的同一个对象。 + +而浅拷贝和深拷贝就是在这个基础之上做的区分，如果在拷贝这个对象的时候，只对基本数据类型进行了拷贝，而对引用数据类型只是进行了引用的传递，而没有真实的创建一个新的对象，则认为是**浅拷贝**。反之，在对引用数据类型进行拷贝的时候，创建了一个新的对象，并且复制其内的成员变量，则认为是**深拷贝**。 + +> 所谓的浅拷贝和深拷贝，只是在拷贝对象的时候，对 **类的实例对象** 这种引用数据类型的不同操作而已 + +### 浅拷贝 + +1. 对于数据类型是基本数据类型的成员变量，浅拷贝会直接进行值传递，也就是将该属性值复制一份给新的对象。 + +2. 对于数据类型是引用数据类型的成员变量，比如说成员变量是某个数组、某个类的对象等，那么浅拷贝会进行引用传递，也就是只是将该成员变量的引用值(内存地址)复制一份给新的对象。因为实际上两个对象的该成员变量都指向同一个实例。在这种情况下，在一个对象中修改该成员变量会影响到另一个对象的该成员变量值 + +3. 前面我们克隆羊就是浅拷贝，如果我们在 Sheep 中加一个对象类型的属性，`public Sheep child;`可以看到 s 和 s1 的 friend 是同一个。 + + ```java + Sheep s = new Sheep(); + s.setName("sss"); + + s.friend = new Sheep(); + s.friend.setName("喜洋洋"); + + Sheep s1 = s.clone(); + System.out.println(s == s1); + System.out.println(s.hashCode()+"---"+s.clone().hashCode()); + + System.out.println(s.friend == s1.friend); + System.out.println(s.friend.hashCode() + "---" +s1.friend.hashCode()); + ``` + + ``` + false + 621009875---1265094477 + true + 2125039532---2125039532 + ``` + +### 深拷贝 + +现在我们知道 clone() 方法，只能对当前对象进行浅拷贝，引用类型依然是在传递引用。那如何进行一个深拷贝呢？ + +常见的深拷贝实现方式有两种： + +1. 重写 **clone** 方法来实现深拷贝 +2. 通过对象序列化实现深拷贝 + +浅拷贝和深拷贝只是相对的，如果一个对象内部只有基本数据类型，那用 `clone()` 方法获取到的就是这个对象的深拷贝，而如果其内部还有引用数据类型，那用 `clone()` 方法就是一次浅拷贝的操作。 \ No newline at end of file diff --git a/docs/design-pattern/Proxy-Pattern.md b/docs/design-pattern/Proxy-Pattern.md index 0b53586f44..75835d7611 100644 --- a/docs/design-pattern/Proxy-Pattern.md +++ b/docs/design-pattern/Proxy-Pattern.md @@ -1,6 +1,12 @@ -# 代理模式 +--- +title: 代理模式 +date: 2023-09-12 +tags: + - Design Pattern +categories: Design Pattern +--- -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gg0b3ynhb0j31900u0b29.jpg) +![](https://cdn.pixabay.com/photo/2019/11/12/09/03/proxy-4620557_1280.jpg) ## 基本介绍 @@ -20,7 +26,7 @@ 为什么要控制对于某个对象的访问呢？ 举个例子： 有这样一个消耗大量系统资源的巨型对象， 你只是偶尔需要使用它， 并非总是需要。 -![图：refactoringguru.cn](https://tva1.sinaimg.cn/large/007S8ZIlly1gg0b594jl2j30e604gdfw.jpg) +![img](https://static001.geekbang.org/infoq/4e/4e2863150390d43ec414a4102a9a1e69.jpeg?x-oss-process=image%2Fresize%2Cp_80%2Fauto-orient%2C1) @@ -34,7 +40,7 @@ 代理模式建议新建一个与原服务对象接口相同的代理类， 然后更新应用以将代理对象传递给所有原始对象客户端。 代理类接收到客户端请求后会创建实际的服务对象， 并将所有工作委派给它。 -![图：refactoringguru.cn](https://tva1.sinaimg.cn/large/007S8ZIlly1gg0b5d5zc6j30e604gmx8.jpg) +![img](https://static001.geekbang.org/infoq/86/864c66131785654125df50bee958952b.jpeg?x-oss-process=image%2Fresize%2Cp_80%2Fauto-orient%2C1) 代理将自己伪装成数据库对象， 可在客户端或实际数据库对象不知情的情况下处理延迟初始化和缓存查询结果的工作。 @@ -44,7 +50,7 @@ ## 代理模式结构 -![图：refactoringguru.cn](https://tva1.sinaimg.cn/large/007S8ZIlly1gg0b5gg8kij30aa0aa3yf.jpg) +![img](https://static001.geekbang.org/infoq/5e/5ef204c0c80b74e3366945a7af856079.jpeg?x-oss-process=image%2Fresize%2Cp_80%2Fauto-orient%2C1) 1. **服务接口** （Service Interface） 声明了服务接口。 代理必须遵循该接口才能伪装成服务对象。 2. **服务** （Service） 类提供了一些实用的业务逻辑。 @@ -165,7 +171,7 @@ Access Denied:qq.com 静态代理会产生很多静态类，所以我们要想办法可以通过一个代理类完成全部的代理功能，这就引出了动态代理。 -### JDK原生动态代理 +### JDK 原生动态代理 - 代理对象，不需要实现接口，但是目标对象要实现接口，否则不能用动态代理 - 代理对象的生成，是通过 JDK 的 API（反射机制），动态的在内存中构建代理对象 @@ -268,11 +274,11 @@ public class Client { -### cglib代理 +### cglib 代理 静态代理和 JDK 代理模式都要求目标对象实现一个接口，但有时候目标对象只是一个单独的对象，并没有实现任何接口，这个时候就可以使用目标对象子类来实现代理，这就是 cglib 代理。 -- [cglib](https://github.com/cglib/cglib)(*Code Generation Library*)是一个基于ASM的字节码生成库，它允许我们在运行时对字节码进行修改和动态生成。cglib 通过继承方式实现代理。它广泛的被许多AOP的框架使用，比如我们的 Spring AOP。 +- [cglib](https://github.com/cglib/cglib)(*Code Generation Library*)是一个基于 ASM 的字节码生成库，**它允许我们在运行时对字节码进行修改和动态生成**。cglib 通过继承方式实现代理。它广泛的被许多 AOP 的框架使用，比如我们的 Spring AOP。 - cglib 包的底层是通过使用字节码处理框架 ASM 来转换字节码并生成新的类。 @@ -366,25 +372,25 @@ cglib 代理结束 使用代理模式的方式多种多样， 我们来看看最常见的几种。 -- 延迟初始化 （虚拟代理）：如果你有一个偶尔使用的重量级服务对象， 一直保持该对象运行会消耗系统资源时， 可使用代理模式。 +- **延迟初始化** （虚拟代理）：如果你有一个偶尔使用的重量级服务对象， 一直保持该对象运行会消耗系统资源时， 可使用代理模式。 你无需在程序启动时就创建该对象， 可将对象的初始化延迟到真正有需要的时候。 -- 访问控制 （保护代理）：如果你只希望特定客户端使用服务对象， 这里的对象可以是操作系统中非常重要的部分， 而客户端则是各种已启动的程序 （包括恶意程序）， 此时可使用代理模式。 +- **访问控制** （保护代理）：如果你只希望特定客户端使用服务对象， 这里的对象可以是操作系统中非常重要的部分， 而客户端则是各种已启动的程序 （包括恶意程序）， 此时可使用代理模式。 代理可仅在客户端凭据满足要求时将请求传递给服务对象。 -- 本地执行远程服务 （远程代理）：适用于服务对象位于远程服务器上的情形。 +- **本地执行远程服务** （远程代理）：适用于服务对象位于远程服务器上的情形。 在这种情形中， 代理通过网络传递客户端请求， 负责处理所有与网络相关的复杂细节。 -- 记录日志请求 （日志记录代理）：适用于当你需要保存对于服务对象的请求历史记录时。 代理可以在向服务传递请求前进行记录。 +- **记录日志请求** （日志记录代理）：适用于当你需要保存对于服务对象的请求历史记录时。 代理可以在向服务传递请求前进行记录。 -- 缓存请求结果 （缓存代理）：适用于需要缓存客户请求结果并对缓存生命周期进行管理时， 特别是当返回结果的体积非常大时。 +- **缓存请求结果** （缓存代理）：适用于需要缓存客户请求结果并对缓存生命周期进行管理时， 特别是当返回结果的体积非常大时。 代理可对重复请求所需的相同结果进行缓存， 还可使用请求参数作为索引缓存的键值。比如请求图片、文件等资源时，先到代理缓存取，如果没有就去公网取并缓存到代理服务器 -- 智能引用：可在没有客户端使用某个重量级对象时立即销毁该对象。 +- **智能引用**：可在没有客户端使用某个重量级对象时立即销毁该对象。 代理会将所有获取了指向服务对象或其结果的客户端记录在案。 代理会时不时地遍历各个客户端， 检查它们是否仍在运行。 如果相应的客户端列表为空， 代理就会销毁该服务对象， 释放底层系统资源。 @@ -400,11 +406,11 @@ AspectJ 的底层技术就是静态代理，用一种 AspectJ 支持的特定语 Spring AOP 采用的是动态代理，在运行期间对业务方法进行增强，所以不会生成新类，对于动态代理技术，Spring AOP 提供了对 JDK 动态代理的支持以及 CGLib 的支持。 -默认情况下，Spring对实现了接口的类使用 JDK Proxy方式，否则的话使用CGLib。不过可以通过配置指定 Spring AOP 都通过 CGLib 来生成代理类。 +默认情况下，Spring 对实现了接口的类使用 JDK Proxy 方式，否则的话使用 CGLib。不过可以通过配置指定 Spring AOP 都通过 CGLib 来生成代理类。 -![](https://imgkr.cn-bj.ufileos.com/ea2aff72-3e0a-4a0d-9950-504c0f3ef911.png) +![](https://img.starfish.ink/design-patterns/spring-aop-proxy.png) -具体逻辑在 `org.springframework.aop.framework.DefaultAopProxyFactory`类中，使用哪种方式生成由`AopProxy` 根据 `AdvisedSupport` 对象的配置来决定源码如下： +具体逻辑在 `org.springframework.aop.framework.DefaultAopProxyFactory` 类中，使用哪种方式生成由`AopProxy` 根据 `AdvisedSupport` 对象的配置来决定源码如下： ```java public class DefaultAopProxyFactory implements AopProxyFactory, Serializable { @@ -436,6 +442,6 @@ public class DefaultAopProxyFactory implements AopProxyFactory, Serializable { ## 参考与感谢 -https://refactoringguru.cn/design-patterns/proxy +- https://refactoringguru.cn/design-patterns/proxy -https://www.geeksforgeeks.org/proxy-design-pattern/ \ No newline at end of file +- https://www.geeksforgeeks.org/proxy-design-pattern/ \ No newline at end of file diff --git a/docs/design-pattern/README.md b/docs/design-pattern/README.md new file mode 100644 index 0000000000..35d1eb7530 --- /dev/null +++ b/docs/design-pattern/README.md @@ -0,0 +1,25 @@ +![](https://images.pexels.com/photos/196644/pexels-photo-196644.jpeg?cs=srgb&dl=notebook-beside-the-iphone-on-table-196644.jpg&fm=jpg) + + + +- [设计模式前传](Design-Pattern-Overview.md) + +#### 创建型模式 + +- [单例模式](Singleton-Pattern.md) +- [工厂模式](Factory-Pattern.md) +- [原型模式](Prototype-Pattern.md) +- [建造者模式](Builder-Pattern.md) + +#### 结构型模式 + +- [装饰模式](Decorator-Pattern.md) +- [代理模式](Proxy-Pattern.md) +- [适配器模式](Adapter-Pattern.md) + +#### 行为模式 + +- [责任链模式](Chain-of-Responsibility-Pattern.md) +- [观察者模式](Observer-Pattern.md) +- [模板方法模式](Template-Pattern.md) + diff --git a/docs/design-pattern/Singleton-Pattern.md b/docs/design-pattern/Singleton-Pattern.md index f8c4941f2f..a78b14b418 100644 --- a/docs/design-pattern/Singleton-Pattern.md +++ b/docs/design-pattern/Singleton-Pattern.md @@ -1,11 +1,19 @@ +--- +title: 单例模式——独一无二的对象 +date: 2023-09-12 +tags: + - Design Pattern +categories: Design Pattern +--- + +![](https://img.starfish.ink/design-pattern/banner-singleton.jpg) + > 面试官：带笔了吧，那写两种单例模式的实现方法吧 > > 沙沙沙刷刷刷~~~ 写好了 > > 面试官：你这个是怎么保证线程安全的，那你知道，volatile 关键字? 类加载器？锁机制？？？？ -> 点赞+收藏 就学会系列，文章收录在 GitHub [JavaEgg](https://github.com/Jstarfish/JavaEgg) ，N线互联网开发必备技能兵器谱 - -## 单例模式——独一无二的对象 +> 点赞+收藏 就学会系列，文章收录在 GitHub [JavaKeeper](https://github.com/Jstarfish/JavaKeeper) ，N线互联网开发必备技能兵器谱 ![](https://i01piccdn.sogoucdn.com/d4a728c10d74ab67) @@ -21,10 +29,20 @@ ## 单例模式的类图 -![](https://tva1.sinaimg.cn/large/006tNbRwly1gbjeonzilrj309u064t93.jpg) +![](https://img.starfish.ink/design-pattern/singleton-class.png) ## 单例模式的实现 +> 要实现一个单例，我们需要关注的点无外乎下面几个: +> +> - 构造函数需要是 private 访问权限的，这样才能避免外部通过 new 创建实例; +> +> - 考虑对象创建时的线程安全问题; +> +> - 考虑是否支持延迟加载; +> +> - 考虑 getInstance() 性能是否高(是否加锁)。 + ### 饿汉式 - static 变量在类装载的时候进行初始化 @@ -47,15 +65,21 @@ public class Singleton { } ``` -饿汉式是线程安全的，JVM在加载类时马上创建唯一的实例对象，且只会装载一次。 +饿汉式是线程安全的，JVM 在加载类时马上创建唯一的实例对象，且只会装载一次。 Java 实现的单例是一个虚拟机的范围，因为装载类的功能是虚拟机的，所以一个虚拟机通过自己的ClassLoader 装载饿汉式实现单例类的时候就会创建一个类实例。（如果一个虚拟机里有多个ClassLoader的话，就会有多个实例） +> JDK 中，`java.lang.Runtime` 就是经典的单例模式（饿汉式） +> +> ![](https://img.starfish.ink/design-pattern/singleton-runtime.png) + + + ### 懒汉式 懒汉式，就是实例在用到的时候才去创建，比较“懒” -单例模式的懒汉式实现方式体现了延迟加载的思想（延迟加载也称懒加载Lazy Load，就是一开始不要加载资源或数据，等到要使用的时候才加载） +单例模式的懒汉式实现方式体现了**延迟加载**的思想（延迟加载也称懒加载 Lazy Load，就是一开始不要加载资源或数据，等到要使用的时候才加载） #### 同步方法 @@ -75,7 +99,7 @@ public class Singleton { } ``` - +饿汉式不支持延迟加载，懒汉式有性能问题，不支持高并发。所以就有了双重检测实现方式。 #### 双重检查加锁 @@ -104,21 +128,21 @@ Double-Check 概念（进行两次检查）是多线程开发中经常使用的 双重检查加锁（double checked locking）线程安全、延迟加载、效率比较高 -**volatile**：volatile一般用于多线程的可见性，这里用来防止**指令重排**（防止new Singleton时指令重排序导致其他线程获取到未初始化完的对象）。被volatile 修饰的变量的值，将不会被本地线程缓存，所有对该变量的读写都是直接操作共享内存，从而确保多个线程能正确的处理该变量。 +**volatile**：volatile一般用于多线程的可见性，这里用来防止**指令重排**（防止 `new Singleton` 时指令重排序导致其他线程获取到未初始化完的对象）。被volatile 修饰的变量的值，将不会被本地线程缓存，所有对该变量的读写都是直接操作共享内存，从而确保多个线程能正确的处理该变量。 ##### 指令重排 -指令重排是指在程序执行过程中, 为了性能考虑, 编译器和CPU可能会对指令重新排序。 +指令重排是指在程序执行过程中, 为了性能考虑, 编译器和 CPU 可能会对指令重新排序。 -Java中创建一个对象，往往包含三个过程。对于singleton = new Singleton()，这不是一个原子操作，在 JVM 中包含如下三个过程。 +Java 中创建一个对象，往往包含三个过程。对于 `singleton = new Singleton()`，这不是一个原子操作，在 JVM 中包含如下三个过程。 1. 给 singleton 分配内存 2. 调用 Singleton 的构造函数来初始化成员变量，形成实例 -3. 将singleton对象指向分配的内存空间（执行完这步 singleton才是非 null 了） +3. 将 singleton 对象指向分配的内存空间（执行完这步 singleton才是非 null 了） -但是，由于JVM会进行指令重排序，所以上面的第二步和第三步的顺序是不能保证的，最终的执行顺序可能是 1-2-3，也可能是 1-3-2。如果是 1-3-2，则在 3 执行完毕，2 未执行之前，被另一个线程抢占了，这时 instance 已经是非 null 了（但却没有初始化），所以这个线程会直接返回 instance，然后使用，那肯定就会报错了，所以要加入 volatile关键字。 +但是，由于 JVM 会进行指令重排序，所以上面的第二步和第三步的顺序是不能保证的，最终的执行顺序可能是 1-2-3，也可能是 1-3-2。如果是 1-3-2，则在 3 执行完毕，2 未执行之前，被另一个线程抢占了，这时 instance 已经是非 null 了（但却没有初始化），所以这个线程会直接返回 instance，然后使用，那肯定就会报错了，所以要加入 volatile关键字。 @@ -141,9 +165,9 @@ public class Singleton { 采用类加载的机制来保证初始化实例时只有一个线程； -静态内部类方式在Singleton 类被装载的时候并不会立即实例化，而是在调用getInstance的时候，才去装载内部类SingletonInstance ,从而完成Singleton的实例化 +静态内部类方式在 Singleton 类被装载的时候并不会立即实例化，而是在调用 getInstance 的时候，才去装载内部类 SingletonInstance ,从而完成 Singleton 的实例化 -类的静态属性只会在第一次加载类的时候初始化，所以，JVM帮我们保证了线程的安全性，在类初始化时，其他线程无法进入 +类的静态属性只会在第一次加载类的时候初始化，所以，JVM 帮我们保证了线程的安全性，在类初始化时，其他线程无法进入 优点：线程安全，利用静态内部类实现延迟加载，效率较高，推荐使用 @@ -156,17 +180,9 @@ enum Singleton{ } ``` -借助JDK5 添加的枚举实现单例，不仅可以避免多线程同步问题，还能防止反序列化重新创建新的对象，但是在枚举中的其他任何方法的线程安全由程序员自己负责。还有防止上面的通过反射机制调用私用构造器。不过，由于Java1.5中才加入enum特性，所以使用的人并不多。 - -这种方式是《Effective Java》 作者Josh Bloch 提倡的方式。 - - - -## 单例模式在JDK 中的源码分析 - -JDK 中，`java.lang.Runtime` 就是经典的单例模式（饿汉式） +借助 JDK5 添加的枚举实现单例，不仅可以避免多线程同步问题，还能防止反序列化重新创建新的对象，但是在枚举中的其他任何方法的线程安全由程序员自己负责。还有防止上面的通过反射机制调用私用构造器。 -![](https://tva1.sinaimg.cn/large/006tNbRwly1gbig3eaoe8j31al0u0qg5.jpg) +这种方式是《Effective Java》 作者 Josh Bloch 提倡的方式。 diff --git a/docs/design-pattern/Spring-Design.md b/docs/design-pattern/Spring-Design.md new file mode 100644 index 0000000000..4b9b73dae3 --- /dev/null +++ b/docs/design-pattern/Spring-Design.md @@ -0,0 +1,867 @@ +在 Java 世界里，Spring 框架已经几乎成为项目开发的必备框架。 + + + +第一部分，我们讲解 Spring 框架中蕴含的经典设计思想或原则。 + +第二部分，我们讲解 Spring 框架中用来支持扩展的两种设计模式。 + +第三部分，我们总结罗列 Spring 框架中用到的其他十几种设计模式。 + + + +Spring 框架背后的一些经典设计思想（或开发技巧）。 + +约定大于配置、低侵入松耦合、模块化轻量级等 + +**1.** **约定优于配置** + +基于约定的配置方式，也常叫作“约定优于配置”或者“规约优于配置”（Convention over Configuration）。通过约定的代码结构或者命名来减少配置。说直白点，就是提供配置的默认值，优先使用默认值。程序员只需要设置那些偏离约定的配置就可以了。 + +比如，在 Spring JPA（基于 ORM 框架、JPA 规范的基础上，封装的一套 JPA 应用框架）中，我们约定类名默认跟表名相同，属性名默认跟表字段名相同，String 类型对应数据库中的 varchar 类型，long 类型对应数据库中的 bigint 类型等等 + +**2.** **低侵入、松耦合** + +框架的侵入性是衡量框架好坏的重要指标。所谓低侵入指的是，框架代码很少耦合在业务代码中。低侵入意味着，当我们要替换一个框架的时候，对原有的业务代码改动会很少。 + +Spring 提供的 IOC 容器，在不需要 Bean 继承任何父类或者实现任何接口的情况下，仅仅通过配置，就能将它们纳入进 Spring 的管理中。如果我们换一个 IOC 容器，也只是重新配置一下就可以了，原有的 Bean 都不需要任何修改。 + +除此之外，Spring 提供的 AOP 功能，也体现了低侵入的特性。在项目中，对于非业务功能，比如请求日志、数据采点、安全校验、事务等等，我们没必要将它们侵入进业务代码中。因为一旦侵入，这些代码将分散在各个业务代码中，删除、修改的成本就变得很高。而基于 AOP 这种开发模式，将非业务代码集中放到切面中，删除、修改的成本就变得很低了 + +**3.** **模块化、轻量级** + +每个模块都只负责一个相对独立的功能。模块之间关系，仅有上层对下层的依赖关系，而同层之间以及下层对上层，几乎没有依赖和耦合。 + +**4.** **再封装、再抽象** + +Spring 不仅仅提供了各种 Java 项目开发的常用功能模块，而且还对市面上主流的中间件、系统的访问类库，做了进一步的封装和抽象，提供了更高层次、更统一的访问接口。 + +比如，Spring 提供了 spring-data-redis 模块，对 Redis Java 开发类库（比如 Jedis、Lettuce）做了进一步的封装，适配 Spring 的访问方式，让编程访问 Redis 更加简单。 + + + + + +### **观察者模式在** **Spring** **中的应用** + +Spring 中实现的观察者模式包含三部分：Event 事件（相当于消息）、Listener 监听者（相当于观察者）、Publisher 发送者（相当于被观察者） + +```java +// Event事件 +public class DemoEvent extends ApplicationEvent { + private String message; + + public DemoEvent(Object source, String message) { + super(source); + } + + public String getMessage() { + return this.message; + } +} +``` + +```java +// Listener监听者 +@Component +public class DemoListener implements ApplicationListener { + @Override + public void onApplicationEvent(DemoEvent demoEvent) { + String message = demoEvent.getMessage(); + System.out.println(message); + } +} +``` + +```java +// Publisher发送者 +@Component +public class DemoPublisher { + @Autowired + private ApplicationContext applicationContext; + + public void publishEvent(DemoEvent demoEvent) { + this.applicationContext.publishEvent(demoEvent); + } +} +``` + +从代码中，我们可以看出，框架使用起来并不复杂，主要包含三部分工作： + +- 定义一个继承 ApplicationEvent 的事件（DemoEvent）； +- 定义一个实现了 ApplicationListener 的监听器（DemoListener）； +- 定义一个发送者（DemoPublisher），发送者调用 ApplicationContext 来发送事件消息。 + +其中，ApplicationEvent 和 ApplicationListener 的代码实现都非常简单，内部并不包含太多属性和方法。实际上，它们最大的作用是做类型标识之用（继承自 ApplicationEvent 的类是事件，实现 ApplicationListener 的类是监听器）。 + +```java +public abstract class ApplicationEvent extends EventObject { + private static final long serialVersionUID = 7099057708183571937L; + private final long timestamp = System.currentTimeMillis(); + + public ApplicationEvent(Object source) { + super(source); + } + + public final long getTimestamp() { + return this.timestamp; + } +} + +public class EventObject implements java.io.Serializable { + private static final long serialVersionUID = 5516075349620653480L; + protected transient Object source; + + public EventObject(Object source) { + if (source == null) + throw new IllegalArgumentException("null source"); + this.source = source; + } + + public Object getSource() { + return source; + } + + public String toString() { + return getClass().getName() + "[source=" + source + "]"; + } +} + +public interface ApplicationListener extends Event + void onApplicationEvent(E var1); +} +``` + +观察者需要事先注册到被观察者（JDK 的实现方式）或者事件总线（EventBus 的实现方式）中。那在 Spring 的实现中，观察者注册到 + +了哪里呢？又是如何注册的呢？ + +我们把观察者注册到了 ApplicationContext 对象中。这里的ApplicationContext 就相当于 Google EventBus 框架中的“事件总线”。不过，稍微提醒一下，ApplicationContext 这个类并不只是为观察者模式服务的。它底层依赖 BeanFactory（IOC 的主要实现类），提供应用启动、运行时的上下文信息，是访问这些信息的最顶层接口。 + +实际上，具体到源码来说，ApplicationContext 只是一个接口，具体的代码实现包含在它的实现类 AbstractApplicationContext 中。我把跟观察者模式相关的代码，摘抄到了下面。你只需要关注它是如何发送事件和注册监听者就好，其他细节不需要细究 + +```java +public abstract class AbstractApplicationContext extends ... { + private final Set> applicationListeners; + + public AbstractApplicationContext() { + this.applicationListeners = new LinkedHashSet(); + //... + } + + public void publishEvent(ApplicationEvent event) { + this.publishEvent(event, (ResolvableType)null); + } + + public void publishEvent(Object event) { + this.publishEvent(event, (ResolvableType)null); + } + + protected void publishEvent(Object event, ResolvableType eventType) { + //... + Object applicationEvent; + if (event instanceof ApplicationEvent) { + applicationEvent = (ApplicationEvent)event; + } else { + applicationEvent = new PayloadApplicationEvent(this, event); + if (eventType == null) { + eventType = ((PayloadApplicationEvent)applicationEvent).getResolvableTy + } + } + + if (this.earlyApplicationEvents != null) { + this.earlyApplicationEvents.add(applicationEvent); + } else { + this.getApplicationEventMulticaster().multicastEvent( + (ApplicationEvent)applicationEvent, eventType); + } + + if (this.parent != null) { + if (this.parent instanceof AbstractApplicationContext) { + ((AbstractApplicationContext)this.parent).publishEvent(event, eventType + } else { + this.parent.publishEvent(event); + } + } + } + + public void addApplicationListener(ApplicationListener listener) { + Assert.notNull(listener, "ApplicationListener must not be null"); + if (this.applicationEventMulticaster != null) { + this.applicationEventMulticaster.addApplicationListener(listener); + } else { + this.applicationListeners.add(listener); + } + } + + public Collection> getApplicationListeners() { + return this.applicationListeners; + } + + protected void registerListeners() { + Iterator var1 = this.getApplicationListeners().iterator(); + + while(var1.hasNext()) { + ApplicationListener listener = (ApplicationListener)var1.next(); t + } + + String[] listenerBeanNames = this.getBeanNamesForType(ApplicationListener.c + String[] var7 = listenerBeanNames; + int var3 = listenerBeanNames.length; + + for(int var4 = 0; var4 < var3; ++var4) { + String listenerBeanName = var7[var4]; + this.getApplicationEventMulticaster().addApplicationListenerBean(listene + } + + Set earlyEventsToProcess = this.earlyApplicationEvents; + this.earlyApplicationEvents = null; + if (earlyEventsToProcess != null) { + Iterator var9 = earlyEventsToProcess.iterator(); + + while(var9.hasNext()) { + ApplicationEvent earlyEvent = (ApplicationEvent)var9.next(); + this.getApplicationEventMulticaster().multicastEvent(earlyEvent); + } + } + } +} +``` + +从上面的代码中，我们发现，真正的消息发送，实际上是通过 ApplicationEventMulticaster 这个类来完成的。这个类的源码我只摘抄了最关键的一部分，也就是 multicastEvent() 这个消息发送函数。不过，它的代码也并不复杂，我就不多解释了。这里我稍微提示一下，它通过线程池，支持异步非阻塞、同步阻塞这两种类型的观察者模式。 + +借助 Spring 提供的观察者模式的骨架代码，如果我们要在 Spring 下实现某个事件的发送和监听，只需要做很少的工作，定义事件、定义监听器、往 ApplicationContext 中发送事件就可以了，剩下的工作都由 Spring 框架来完成。实际上，这也体现了 Spring 框架的扩展性，也就是在不需要修改任何代码的情况下，扩展新的事件和监听。 + + + +### **模板模式在** **Spring** 中的应用 + +我们来看下一下经常在面试中被问到的一个问题：请你说下 Spring Bean 的创建过程包含哪些主要的步骤。这其中就涉及模板模式。它也体现了 Spring 的扩展性。利用模板模式，Spring 能让用户定制 Bean 的创建过程。 + +Spring Bean 的创建过程，可以大致分为两大步：对象的创建和对象的初始化。 + +对象的创建是通过反射来动态生成对象，而不是 new 方法。不管是哪种方式，说白了，总归还是调用构造函数来生成对象，没有什么特殊的。 + +对象的初始化有两种实现方式。一种是在类中自定义一个初始化函数，并且通过配置文件，显式地告知 Spring，哪个函数是初始化函数。我举了一个例子解释一下。如下所示，在配置文件中，我们通过 init-method 属性来指定初始化函数。 + +```java +public class DemoClass { + //... + public void initDemo() { + //...初始化.. + } +} +``` + +```xml +// 配置：需要通过init-method显式地指定初始化方法 + +``` + +这种初始化方式有一个缺点，初始化函数并不固定，由用户随意定义，这就需要 Spring 通过反射，在运行时动态地调用这个初始化函数。而反射又会影响代码执行的性能，那有没有替代方案呢？ + +Spring 提供了另外一个定义初始化函数的方法，那就是让类实现 Initializingbean 接口。这个接口包含一个固定的初始化函数定义（afterPropertiesSet() 函数）。Spring 在初始化 Bean 的时候，可以直接通过 bean.afterPropertiesSet() 的方式，调用 Bean 对象上的 + +这个函数，而不需要使用反射来调用了。我举个例子解释一下，代码如下所示。 + +```java +public class DemoClass implements InitializingBean{ + @Override + public void afterPropertiesSet() throws Exception { + //...初始化... + } +} +``` + +```xml +// 配置：不需要显式地指定初始化方法 + +``` + +尽管这种实现方式不会用到反射，执行效率提高了，但业务代码（DemoClass）跟框架代码（InitializingBean）耦合在了一起。框架代码侵入到了业务代码中，替换框架的成本就变高了。所以，并不是太推荐这种写法。 + +实际上，Spring 针对对象的初始化过程，还做了进一步的细化，将它拆分成了三个小步骤：初始化前置操作、初始化、初始化后置操作。其中，中间的初始化操作就是我们刚刚讲的那部分，初始化的前置和后置操作，定义在接口 BeanPostProcessor 中。 + +BeanPostProcessor 的接口定义如下所示 + +```java +public interface BeanPostProcessor { + + @Nullable + default Object postProcessBeforeInitialization(Object bean, String beanName) throws BeansException { + return bean; + } + + @Nullable + default Object postProcessAfterInitialization(Object bean, String beanName) throws BeansException { + return bean; + } + +} +``` + +我们再来看下，如何通过 BeanPostProcessor 来定义初始化前置和后置操作？ + +我们只需要定义一个实现了 BeanPostProcessor 接口的处理器类，并在配置文件中像配置普通 Bean 一样去配置就可以了。Spring 中的 ApplicationContext 会自动检测在配置文件中实现了 BeanPostProcessor 接口的所有 Bean，并把它们注册到 BeanPostProcessor + +处理器列表中。在 Spring 容器创建 Bean 的过程中，Spring 会逐一去调用这些处理器。 + +通过上面的分析，我们基本上弄清楚了 Spring Bean 的整个生命周期（创建加销毁）。针对这个过程，我画了一张图，你可以结合着刚刚讲解一块看下 + +![](/Users/starfish/Documents/截图/截屏2025-01-02 14.37.01.png) + +不过，你可能会说，这里哪里用到了模板模式啊？模板模式不是需要定义一个包含模板方法的抽象模板类，以及定义子类实现模板方法吗？ + +实际上，这里的模板模式的实现，并不是标准的抽象类的实现方式，而是有点类似我们前面讲到的 Callback 回调的实现方式，也就是将要执行的函数封装成对象（比如，初始化方法封装成 InitializingBean 对象），传递给模板（BeanFactory）来执行 + + + +### 适配器模式在 **Spring** **中的应用** + +在 Spring MVC 中，定义一个 Controller 最常用的方式是，通过 @Controller 注解来标记某个类是 Controller 类，通过 @RequesMapping 注解来标记函数对应的 URL。不过，定义一个 Controller 远不止这一种方法。我们还可以通过让类实现 Controller 接口或者 Servlet 接口，来定义一个 Controller。针对这三种定义方式，我写了三段示例代码，如下所示： + +```java +// 方法一：通过@Controller、@RequestMapping来定义 +@Controller +public class DemoController { + @RequestMapping("/employname") + public ModelAndView getEmployeeName() { + ModelAndView model = new ModelAndView("Greeting"); + model.addObject("message", "Dinesh"); + return model; + } +} + +// 方法二：实现Controller接口 + xml配置文件:配置DemoController与URL的对应关系 +public class DemoController implements Controller { + @Override + public ModelAndView handleRequest(HttpServletRequest req, HttpServletRespon + ModelAndView model = new ModelAndView("Greeting"); + model.addObject("message", "Dinesh Madhwal"); + return model; + } +} + +// 方法三：实现Servlet接口 + xml配置文件:配置DemoController类与URL的对应关系 +public class DemoServlet extends HttpServlet { + @Override + protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws + this.doPost(req, resp); + } + + @Override + protected void doPost(HttpServletRequest req, HttpServletResponse resp) throw + resp.getWriter().write("Hello World."); + } +} +``` + +在应用启动的时候，Spring 容器会加载这些 Controller 类，并且解析出 URL 对应的处理函数，封装成 Handler 对象，存储到 HandlerMapping 对象中。当有请求到来的时候，DispatcherServlet 从 HanderMapping 中，查找请求 URL 对应的 Handler，然后调用执行 Handler 对应的函数代码，最后将执行结果返回给客户端。 + +但是，不同方式定义的 Controller，其函数的定义（函数名、入参、返回值等）是不统一的。如上示例代码所示，方法一中的函数的定义很随意、不固定，方法二中的函数定义是handleRequest()、方法三中的函数定义是 service()（看似是定义了 doGet()、doPost()， + +实际上，这里用到了模板模式，Servlet 中的 service() 调用了 doGet() 或 doPost() 方法，DispatcherServlet 调用的是 service() 方法）。DispatcherServlet 需要根据不同类型的Controller，调用不同的函数。下面是具体的伪代码： + +```java +Handler handler = handlerMapping.get(URL); + if (handler instanceof Controller) { + ((Controller)handler).handleRequest(...); + } else if (handler instanceof Servlet) { + ((Servlet)handler).service(...); + } else if (hanlder 对应通过注解来定义的Controller) { + 反射调用方法... + } +``` + +从代码中我们可以看出，这种实现方式会有很多 if-else 分支判断，而且，如果要增加一个新的 Controller 的定义方法，我们就要在 DispatcherServlet 类代码中，对应地增加一段如上伪代码所示的 if 逻辑。这显然不符合开闭原则 + +实际上，我们可以利用是适配器模式对代码进行改造，让其满足开闭原则，能更好地支持扩赞。 + +适配器其中一个作用是“统一多个类的接口设计”。利用适配器模式，我们将不同方式定义的 Controller 类中的函数，适配为统一的函数定义。这样，我们就能在 DispatcherServlet 类代码中，移除掉 if-else 分支判断逻辑，调用统一的函数。 + +刚刚讲了大致的设计思路，我们再具体看下 Spring 的代码实现。 + +Spring 定义了统一的接口 HandlerAdapter，并且对每种 Controller 定义了对应的适配器类。这些适配器类包括：AnnotationMethodHandlerAdapter、SimpleControllerHandlerAdapter、SimpleServletHandlerAdapter 等。源码我贴到了 + +下面，你可以结合着看下 + +```java +public interface HandlerAdapter { + boolean supports(Object var1); + + ModelAndView handle(HttpServletRequest var1, HttpServletResponse var2, Object + + long getLastModified(HttpServletRequest var1, Object var2); +} + +// 对应实现Controller接口的Controller +public class SimpleControllerHandlerAdapter implements HandlerAdapter { + public SimpleControllerHandlerAdapter() { + } + + public boolean supports(Object handler) { + return handler instanceof Controller; + } + + public ModelAndView handle(HttpServletRequest request, HttpServletResponse re + return ((Controller)handler).handleRequest(request, response); + } + + public long getLastModified(HttpServletRequest request, Object handler) { + return handler instanceof LastModified ? ((LastModified)handler).getLastMod + } +} + +// 对应实现Servlet接口的Controller +public class SimpleServletHandlerAdapter implements HandlerAdapter { + public SimpleServletHandlerAdapter() { + } + + public boolean supports(Object handler) { + return handler instanceof Servlet; + } + + public ModelAndView handle(HttpServletRequest request, HttpServletResponse re + ((Servlet)handler).service(request, response); + return null; + } + + public long getLastModified(HttpServletRequest request, Object handler) { + return -1L; + } +} + +//AnnotationMethodHandlerAdapter对应通过注解实现的Controller， +//代码太多了，我就不贴在这里了 +``` + +在 DispatcherServlet 类中，我们就不需要区分对待不同的 Controller 对象了，统一调用 HandlerAdapter 的 handle() 函数就可以了。按照这个思路实现的伪代码如下所示。你看，这样就没有烦人的 if-else 逻辑了吧？ + +```java +// 之前的实现方式 +Handler handler = handlerMapping.get(URL); +if (handler instanceof Controller) { + ((Controller)handler).handleRequest(...); +} else if (handler instanceof Servlet) { + ((Servlet)handler).service(...); +} else if (hanlder 对应通过注解来定义的Controller) { + 反射调用方法... +} + +// 现在实现方式 +HandlerAdapter handlerAdapter = handlerMapping.get(URL); +handlerAdapter.handle(...); +``` + + + +### **策略模式在** **Spring** 中的应用 + +Spring AOP 是通过动态代理来实现的。具体到代码实现，Spring 支持两种动态代理实现方式，一种是 JDK 提供的动态代理实现方式，另一种是 Cglib 提供的动态代理实现方式。 + +前者需要被代理的类有抽象的接口定义，后者不需要。针对不同的被代理类，Spring 会在运行时动态地选择不同的动态代理实现方式。这个应用场景实际上就是策略模式的典型应用场景。 + +策略模式包含三部分，策略的定义、创建和使用。接下来，我们具体看下，这三个部分是如何体现在 Spring 源码中的。 + +在策略模式中，策略的定义这一部分很简单。我们只需要定义一个策略接口，让不同的策略类都实现这一个策略接口。对应到 Spring 源码，AopProxy 是策略接口，JdkDynamicAopProxy、CglibAopProxy 是两个实现了 AopProxy 接口的策略类。其中，AopProxy 接口的定义如下所示： + +```java +public interface AopProxy { + + Object getProxy(); + + Object getProxy(@Nullable ClassLoader classLoader); + +} +``` + +在策略模式中，策略的创建一般通过工厂方法来实现。对应到 Spring 源码，AopProxyFactory 是一个工厂类接口，DefaultAopProxyFactory 是一个默认的工厂类，用来创建 AopProxy 对象。两者的源码如下所示： + +```java +public interface AopProxyFactory { + AopProxy createAopProxy(AdvisedSupport var1) throws AopConfigException; +} + +public class DefaultAopProxyFactory implements AopProxyFactory, Serializable { + + private static final long serialVersionUID = 7930414337282325166L; + + + @Override + public AopProxy createAopProxy(AdvisedSupport config) throws AopConfigException { + if (!NativeDetector.inNativeImage() && + (config.isOptimize() || config.isProxyTargetClass() || hasNoUserSuppliedProxyInterfaces(config))) { + Class targetClass = config.getTargetClass(); + if (targetClass == null) { + throw new AopConfigException("TargetSource cannot determine target class: " + + "Either an interface or a target is required for proxy creation."); + } + if (targetClass.isInterface() || Proxy.isProxyClass(targetClass) || ClassUtils.isLambdaClass(targetClass)) { + return new JdkDynamicAopProxy(config); + } + return new ObjenesisCglibAopProxy(config); + } + else { + return new JdkDynamicAopProxy(config); + } + } + + /** + * 用来判断用哪个动态代理实现方式 + * Determine whether the supplied {@link AdvisedSupport} has only the + * {@link org.springframework.aop.SpringProxy} interface specified + * (or no proxy interfaces specified at all). + */ + private boolean hasNoUserSuppliedProxyInterfaces(AdvisedSupport config) { + Class[] ifcs = config.getProxiedInterfaces(); + return (ifcs.length == 0 || (ifcs.length == 1 && SpringProxy.class.isAssignableFrom(ifcs[0]))); + } + +} +``` + + + +### **组合模式在** **Spring** 中的应用 + +Spring Cache 提供了一套抽象的 Cache 接口。使用它我们能够 统一不同缓存实现（Redis、Google Guava…）的不同的访问方式。Spring 中针对不同缓存实现的不同缓存访问类，都依赖这个接口，比如：EhCacheCache、GuavaCache、NoOpCache、 + +RedisCache、JCacheCache、ConcurrentMapCache、CaffeineCache。Cache 接口的源码如下所示： + +```java +public interface Cache { + String getName(); + Object getNativeCache(); + Cache.ValueWrapper get(Object var1); + T get(Object var1, Class var2); + T get(Object var1, Callable var2); + void put(Object var1, Object var2); + Cache.ValueWrapper putIfAbsent(Object var1, Object var2); + void evict(Object var1); + void clear(); + + public static class ValueRetrievalException extends RuntimeException { + private final Object key; + + public ValueRetrievalException(Object key, Callable loader, Throwable ex) { + super(String.format("Value for key '%s' could not be loaded using '%s'", key, loader)); + this.key = key; + } + + public Object getKey() { + return this.key; + } + } + + public interface ValueWrapper { + Object get(); + } +} +``` + +在实际的开发中，一个项目有可能会用到多种不同的缓存，比如既用到 Google Guava 缓存，也用到 Redis 缓存。除此之外，同一个缓存实例，也可以根据业务的不同，分割成多个小的逻辑缓存单元（或者叫作命名空间）。 + +为了管理多个缓存，Spring 还提供了缓存管理功能。不过，它包含的功能很简单，主要有这样两部分：一个是根据缓存名字（创建 Cache 对象的时候要设置 name 属性）获取Cache 对象；另一个是获取管理器管理的所有缓存的名字列表。对应的 Spring 源码如下所示： + +```java +public interface CacheManager { + Cache getCache(String var1); + Collection getCacheNames(); +} +``` + +刚刚给出的是 CacheManager 接口的定义，那如何来实现这两个接口呢？实际上，这就要用到了我们之前讲过的组合模式。 + +组合模式主要应用在能表示成树形结构的一组数据上。 + +树中的结点分为叶子节点和中间节点两类。对应到 Spring 源码，EhCacheManager、SimpleCacheManager、NoOpCacheManager、RedisCacheManager 等表示叶子节点，CompositeCacheManager 表示中间节点。 + +叶子节点包含的是它所管理的 Cache 对象，中间节点包含的是其他 CacheManager 管理器，既可以是 CompositeCacheManager，也可以是具体的管理器，比如 EhCacheManager、RedisManager 等。 + +我把 CompositeCacheManger 的代码贴到了下面，你可以结合着讲解一块看下。其中，getCache()、getCacheNames() 两个函数的实现都用到了递归。这正是树形结构最能发挥优势的地方。 + +```java +public class CompositeCacheManager implements CacheManager, InitializingBean { + + private final List cacheManagers = new ArrayList<>(); + private boolean fallbackToNoOpCache = false; + + public CompositeCacheManager() { + } + + public CompositeCacheManager(CacheManager... cacheManagers) { + setCacheManagers(Arrays.asList(cacheManagers)); + } + + public void setCacheManagers(Collection cacheManagers) { + this.cacheManagers.addAll(cacheManagers); + } + + public void setFallbackToNoOpCache(boolean fallbackToNoOpCache) { + this.fallbackToNoOpCache = fallbackToNoOpCache; + } + + @Override + public void afterPropertiesSet() { + if (this.fallbackToNoOpCache) { + this.cacheManagers.add(new NoOpCacheManager()); + } + } + + + @Override + @Nullable + public Cache getCache(String name) { + for (CacheManager cacheManager : this.cacheManagers) { + Cache cache = cacheManager.getCache(name); + if (cache != null) { + return cache; + } + } + return null; + } + + @Override + public Collection getCacheNames() { + Set names = new LinkedHashSet<>(); + for (CacheManager manager : this.cacheManagers) { + names.addAll(manager.getCacheNames()); + } + return Collections.unmodifiableSet(names); + } + +} +``` + + + +### **装饰器模式在** **Spring** 中的应用 + +我们知道，缓存一般都是配合数据库来使用的。如果写缓存成功，但数据库事务回滚了，那缓存中就会有脏数据。为了解决这个问题，我们需要将缓存的写操作和数据库的写操作，放到同一个事务中，要么都成功，要么都失败。 + +实现这样一个功能，Spring 使用到了装饰器模式。TransactionAwareCacheDecorator 增加了对事务的支持，在事务提交、回滚的时候分别对 Cache 的数据进行处理。 + +TransactionAwareCacheDecorator 实现 Cache 接口，并且将所有的操作都委托给 targetCache 来实现，对其中的写操作添加了事务功能。这是典型的装饰器模式的应用场景和代码实现，我就不多作解释了。 + +```java +/* + * Copyright 2002-2020 the original author or authors. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.springframework.cache.transaction; + +import java.util.concurrent.Callable; + +import org.springframework.cache.Cache; +import org.springframework.lang.Nullable; +import org.springframework.transaction.support.TransactionSynchronization; +import org.springframework.transaction.support.TransactionSynchronizationManager; +import org.springframework.util.Assert; + +/** + * Cache decorator which synchronizes its {@link #put}, {@link #evict} and + * {@link #clear} operations with Spring-managed transactions (through Spring's + * {@link TransactionSynchronizationManager}, performing the actual cache + * put/evict/clear operation only in the after-commit phase of a successful + * transaction. If no transaction is active, {@link #put}, {@link #evict} and + * {@link #clear} operations will be performed immediately, as usual. + * + *

Note: Use of immediate operations such as {@link #putIfAbsent} and + * {@link #evictIfPresent} cannot be deferred to the after-commit phase of a + * running transaction. Use these with care in a transactional environment. + * + * @author Juergen Hoeller + * @author Stephane Nicoll + * @author Stas Volsky + * @since 3.2 + * @see TransactionAwareCacheManagerProxy + */ +public class TransactionAwareCacheDecorator implements Cache { + + private final Cache targetCache; + + + /** + * Create a new TransactionAwareCache for the given target Cache. + * @param targetCache the target Cache to decorate + */ + public TransactionAwareCacheDecorator(Cache targetCache) { + Assert.notNull(targetCache, "Target Cache must not be null"); + this.targetCache = targetCache; + } + + + /** + * Return the target Cache that this Cache should delegate to. + */ + public Cache getTargetCache() { + return this.targetCache; + } + + @Override + public String getName() { + return this.targetCache.getName(); + } + + @Override + public Object getNativeCache() { + return this.targetCache.getNativeCache(); + } + + @Override + @Nullable + public ValueWrapper get(Object key) { + return this.targetCache.get(key); + } + + @Override + public T get(Object key, @Nullable Class type) { + return this.targetCache.get(key, type); + } + + @Override + @Nullable + public T get(Object key, Callable valueLoader) { + return this.targetCache.get(key, valueLoader); + } + + @Override + public void put(final Object key, @Nullable final Object value) { + if (TransactionSynchronizationManager.isSynchronizationActive()) { + TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronization() { + @Override + public void afterCommit() { + TransactionAwareCacheDecorator.this.targetCache.put(key, value); + } + }); + } + else { + this.targetCache.put(key, value); + } + } + + @Override + @Nullable + public ValueWrapper putIfAbsent(Object key, @Nullable Object value) { + return this.targetCache.putIfAbsent(key, value); + } + + @Override + public void evict(final Object key) { + if (TransactionSynchronizationManager.isSynchronizationActive()) { + TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronization() { + @Override + public void afterCommit() { + TransactionAwareCacheDecorator.this.targetCache.evict(key); + } + }); + } + else { + this.targetCache.evict(key); + } + } + + @Override + public boolean evictIfPresent(Object key) { + return this.targetCache.evictIfPresent(key); + } + + @Override + public void clear() { + if (TransactionSynchronizationManager.isSynchronizationActive()) { + TransactionSynchronizationManager.registerSynchronization(new TransactionSynchronization() { + @Override + public void afterCommit() { + targetCache.clear(); + } + }); + } + else { + this.targetCache.clear(); + } + } + + @Override + public boolean invalidate() { + return this.targetCache.invalidate(); + } + +} +``` + + + +### **工厂模式在** **Spring** 中的应用 + +在 Spring 中，工厂模式最经典的应用莫过于实现 IOC 容器，对应的 Spring 源码主要是 BeanFactory 类和 ApplicationContext 相关类（AbstractApplicationContext、ClassPathXmlApplicationContext、FileSystemXmlApplicationContext…）。 + +在 Spring 中，创建 Bean 的方式有很多种，比如纯构造函数、无参构造函数加setter 方法。我写了一个例子来说明这两种创建方式，代码如下所示： + +```java +public class Student { + private long id; + private String name; + + public Student(long id, String name) { + this.id = id; + this.name = name; + } + + public void setId(long id) { + this.id = id; + } + + public void setName(String name) { + this.name = name; + } +} +``` + +```xml +// 使用构造函数来创建Bean + + + + +// 使用无参构造函数+setter方法来创建Bean + + + + +``` + +实际上，除了这两种创建 Bean 的方式之外，我们还可以通过工厂方法来创建 Bean。还是刚刚这个例子，用这种方式来创建 Bean 的话就是下面这个样子： + +```java +public class StudentFactory { + private static Map students = new HashMap<>(); + static { + map.put(1, new Student(1, "Tom")); + map.put(2, new Student(2, "Jim")); + map.put(3, new Student(3, "Mary")); + } + + public static Student getStudent(long id) { + return students.get(id); + } +} +``` + +```xml + + + +``` + diff --git a/docs/design-pattern/Strategy-Pattern.md b/docs/design-pattern/Strategy-Pattern.md new file mode 100755 index 0000000000..9abf29057d --- /dev/null +++ b/docs/design-pattern/Strategy-Pattern.md @@ -0,0 +1,274 @@ +--- +title: 策略模式——略施小计就彻底消除了多重 if else +date: 2022-11-09 +tags: + - Design Patterns +categories: Design Patterns +--- + +![](https://cdn.pixabay.com/photo/2018/06/10/22/48/chess-3467512_1280.jpg) + +> 最近接手了一个新项目，有段按不同类型走不同检验逻辑的代码，将近小 10 个 `if -else` 判断，真正的“屎山”代码。 +> +> 所以在项目迭代的时候，就打算重构一下，写设计方案后，刚好再总结总结策略模式。 +> +> 先贴个阿里的《 Java 开发手册》中的一个规范 + +![](https://img.starfish.ink/design-patterns/ali-strategy.png) + +我们先不探讨其他方式，主要讲策略模式。 + + + +## 定义 + +**策略模式**（Strategy Design Pattern）：封装可以互换的行为，并使用委托来决定要使用哪一个。 + +策略模式是一种**行为设计模式**， 它能让你定义一系列算法， 并将每种算法分别放入独立的类中， 以使算法的对象能够相互替换。 + +> 用人话翻译后就是：运行时我给你这个类的方法传不同的 “key”，你这个方法就去执行不同的业务逻辑。 +> +> 你品，你细品，这不就是 if else 干的事吗？ + +![](https://i01piccdn.sogoucdn.com/715a60dea42bf3d5) + +先直观的看下传统的多重 `if else` 代码 + +```java +public String getCheckResult(String type) { + if ("校验1".equals(type)) { + return "执行业务逻辑1"; + } else if ("校验2".equals(type)) { + return "执行业务逻辑2"; + } else if ("校验3".equals(type)) { + return "执行业务逻辑3"; + } else if ("校验4".equals(type)) { + return "执行业务逻辑4"; + } else if ("校验5".equals(type)) { + return "执行业务逻辑5"; + } else if ("校验6".equals(type)) { + return "执行业务逻辑6"; + } else if ("校验7".equals(type)) { + return "执行业务逻辑7"; + } else if ("校验8".equals(type)) { + return "执行业务逻辑8"; + } else if ("校验9".equals(type)) { + return "执行业务逻辑9"; + } + return "不在处理的逻辑中返回业务错误"; +} +``` + +这么看，你要是还觉得挺清晰的话，想象下这些 return 里是各种复杂的业务逻辑方法~~ + +![](https://img03.sogoucdn.com/app/a/100520093/e18d20c94006dfe0-0381536966d1161a-7f08208216d08261f99e84e5bf306d20.jpg) + +当然，策略模式的作用可不止是避免冗长的 if-else 或者 switch 分支，它还可以像模板方法模式那样提供框架的扩展点等。 + +网上的示例很多，比如不同路线的规划、不同支付方式的选择 都是典型的 `if-else` 问题，也都是典型的策略模式问题，栗子我们待会看，先看下策略模式的类图，然后去改造多重判断~ + + + +## 类图 + +![](https://img.starfish.ink/design-patterns/strategy-pattern-uml.png) + +策略模式涉及到三个角色： + +- **Strategy**：策略接口或者策略抽象类，用来约束一系列的策略算法（Context 使用这个接口来调用具体的策略实现算法） +- **ConcreateStrategy**：具体的策略类（实现策略接口或继承抽象策略类） +- **Context**：上下文类，持有具体策略类的实例，并负责调用相关的算法 + + + +应用策略模式来解决问题的思路 + +## 实例 + +先看看最简单的策略模式 demo: + +1、策略接口（定义策略） + +```java +public interface Strategy { + void operate(); +} +``` + +2、具体的算法实现 + +```java +public class ConcreteStrategyA implements Strategy { + @Override + public void operate() { + //具体的算法实现 + System.out.println("执行业务逻辑A"); + } +} + +public class ConcreteStrategyB implements Strategy { + @Override + public void operate() { + //具体的算法实现 + System.out.println("执行业务逻辑B"); + } +} +``` + +3、上下文的实现 + +```java +public class Context { + + //持有一个具体的策略对象 + private Strategy strategy; + + //构造方法，传入具体的策略对象 + public Context(Strategy strategy){ + this.strategy = strategy; + } + + public void doSomething(){ + //调用具体的策略对象进操作 + strategy.operate(); + } +} +``` + +4、客户端使用（策略的使用） + +```java +public static void main(String[] args) { + Context context = new Context(new ConcreteStrategyA()); + context.doSomething(); +} +``` + +> ps：这种策略的使用方式其实很死板，真正使用的时候如果还这么写，和写一大推 if-else 没什么区别，所以我们一般会结合工厂类，在运行时动态确定使用哪种策略。策略模式侧重如何选择策略、工厂模式侧重如何创建策略。 + + + +## 解析策略模式 + +策略模式的功能就是把具体的算法实现从具体的业务处理中独立出来，把它们实现成单独的算法类，从而形成一系列算法，并让这些算法可以互相替换。 + +> 策略模式的重心不是如何来实现算法，而是如何组织、调用这些算法，从而让程序结构更灵活，具有更好的维护性和扩展性。 + + + +实际上，每个策略算法具体实现的功能，就是原来在 `if-else` 结构中的具体实现，每个 `if-else` 语句都是一个平等的功能结构，可以说是兄弟关系。 + +策略模式呢，就是把各个平等的具体实现封装到单独的策略实现类了，然后通过上下文与具体的策略类进行交互。 + + 『 **策略模式 = 实现策略接口（或抽象类）的每个策略类 + 上下文的逻辑分派** 』 + +![](https://img.starfish.ink/design-patterns/strategy-pattern-if-else.png) + +> 策略模式的本质：分离算法，选择实现 ——《研磨设计模式》 + +所以说，策略模式只是在代码结构上的一个调整，即使用了策略模式，该写的逻辑一个也少不了，到逻辑分派的时候，只是变相的 `if-else`。 + +而它的优化点是抽象了出了接口，将业务逻辑封装成一个一个的实现类，任意地替换。在复杂场景（业务逻辑较多）时比直接 `if-else` 更好维护和扩展些。 + + + +### 谁来选择具体的策略算法 + +如果你手写了上边的 demo，就会发现，这玩意不及 `if-else` 来的顺手，尤其是在判断逻辑的时候，每个逻辑都要要构造一个上下文对象，费劲。 + +其实，策略模式中，我们可以自己定义谁来选择具体的策略算法，有两种： + +- 客户端：当使用上下文时，由客户端选择，像我们上边的 demo +- 上下文：客户端不用选，由上下文来选具体的策略算法，可以在构造器中指定 + + + +### 优缺点 + +#### 优点： + +- 避免多重条件语句：也就是避免大量的 `if-else` +- 更好的扩展性（完全符合开闭原则）：策略模式中扩展新的策略实现很容易，无需对上下文修改，只增加新的策略实现类就可以 + +#### 缺点： + +- 客户必须了解每种策略的不同（这个可以通过 IOC、依赖注入的方式解决） +- 增加了对象数：每个具体策略都封装成了类，可能备选的策略会很多 +- 只适合扁平的算法结构：策略模式的一系列算法是平等的，也就是在运行时刻只有一个算法会被使用，这就限制了算法使用的层级，不能嵌套使用 + + + +### 思考 + +实际使用中，往往不会只是单一的某个设计模式的套用，一般都会混合使用，而且模式之间的结合也是没有定势的，要具体问题具体分析。 + +策略模式往往会结合其他模式一起使用，比如工厂、模板等，具体使用需要结合自己的业务。 + +切记，不要为了使用设计模式而强行模式，不要把简单问题复杂化。 + +策略模式也不是专为消除 `if-else` 而生的，不要和 `if-else` 划等号。它体现了“对修改关闭，对扩展开放“的原则。 + +并不是说，看到 `if-else` 就想着用策略模式去优化，业务逻辑简单，可能几个枚举，或者几个卫语句就搞定的场景，就不用非得硬套设计模式了。 + + + +## 策略模式在 JDK 中的应用 + +在 JDK 中，Comparator 比较器是一个策略接口，我们常用的 `compare()` 方法就是一个具体的策略实现，用于定义排序规则。 + +```java +public interface Comparator { + int compare(T o1, T o2); + //...... +} +``` + +当我们想自定义排序规则的时候，就可以实现 `Comparator` 。 + +这时候我们重写了接口中的 `compare()` 方法，就是具体的策略类（只不过这里可能是内部类）。当我们在调用 Arrays 的排序方法 `sort()` 时，可以用默认的排序规则，也可以用自定义的规则。 + +```java +public static void main(String[] args) { + Integer[] data = {4,2,7,5,1,9}; + Comparator comparator = new Comparator() { + @Override + public int compare(Integer o1, Integer o2) { + if(o1 > o2){ + return 1; + } else { + return -1; + } + } + }; + + Arrays.sort(data,comparator); + System.out.println(Arrays.toString(data)); +} +``` + +Arrays 的 `sort()` 方法，有自定义规则就按自己的方法排序，反之走源码逻辑。 + +```java +public static void sort(T[] a, Comparator c) { + if (c == null) { + sort(a); + } else { + if (LegacyMergeSort.userRequested) + legacyMergeSort(a, c); + else + TimSort.sort(a, 0, a.length, c, null, 0, 0); + } +} +``` + + + +还有，ThreadPoolExecutor 中的拒绝策略 RejectedExecutionHandler 也是典型的策略模式，感兴趣的也可以再看看源码。 + + + +## 参考与感谢： + +- [《用 Map + 函数式接口来实现策略模式》](https://www.cnblogs.com/keeya/p/13187727.html) +- 《研磨设计模式》 + diff --git a/docs/design-pattern/Summary.md b/docs/design-pattern/Summary.md new file mode 100755 index 0000000000..2d39a26e75 --- /dev/null +++ b/docs/design-pattern/Summary.md @@ -0,0 +1,248 @@ +## 一、创建型设计模式 + +创建型设计模式包括：单例模式、工厂模式、建造者模式、原型模式。它主要解决对象的创建问题，封装复杂的创建过程，解耦对象的创建代码和使用代码。 + +### **1.** 单例模式 + +单例模式用来创建全局唯一的对象。一个类只允许创建一个对象（或者叫实例），那这个类就是一个单例类，这种设计模式就叫作单例模式。单例有几种经典的实现方式，它们分别是：饿汉式、懒汉式、双重检测、静态内部类、枚举。尽管单例是一个很常用的设计模式，在实际的开发中，我们也确实经常用到它，但是，有些人认为单例是一种反模式（anti-pattern），并不推荐使用，主要的理由有以下几点： + +- 单例对 OOP 特性的支持不友好 +- 单例会隐藏类之间的依赖关系 +- 单例对代码的扩展性不友好 +- 单例对代码的可测试性不友好 +- 单例不支持有参数的构造函数 + +那有什么替代单例的解决方案呢？如果要完全解决这些问题，我们可能要从根上寻找其他方式来实现全局唯一类。比如，通过工厂模式、IOC 容器来保证全局唯一性。 + +有人把单例当作反模式，主张杜绝在项目中使用。我个人觉得这有点极端。模式本身没有对错，关键看你怎么用。如果单例类并没有后续扩展的需求，并且不依赖外部系统，那设计成单例类就没有太大问题。对于一些全局类，我们在其他地方 new 的话，还要在类之间传来传去，不如直接做成单例类，使用起来简洁方便。 + +### 2. **工厂模式** + +工厂模式包括简单工厂、工厂方法、抽象工厂这 3 种细分模式。其中，简单工厂和工厂方法比较常用，抽象工厂的应用场景比较特殊，所以很少用到，不是我们学习的重点。 + +工厂模式用来创建不同但是相关类型的对象（继承同一父类或者接口的一组子类），由给定的参数来决定创建哪种类型的对象。实际上，如果创建对象的逻辑并不复杂，那我们直接通过 new 来创建对象就可以了，不需要使用工厂模式。当创建逻辑比较复杂，是一个“大工 + +程”的时候，我们就考虑使用工厂模式，封装对象的创建过程，将对象的创建和使用相分离。 + +当每个对象的创建逻辑都比较简单的时候，我推荐使用简单工厂模式，将多个对象的创建逻辑放到一个工厂类中。当每个对象的创建逻辑都比较复杂的时候，为了避免设计一个过于庞大的工厂类，我们推荐使用工厂方法模式，将创建逻辑拆分得更细，每个对象的创建逻辑独 + +立到各自的工厂类中。 + +详细点说，工厂模式的作用有下面 4 个，这也是判断要不要使用工厂模式最本质的参考标准。 + +- 封装变化：创建逻辑有可能变化，封装成工厂类之后，创建逻辑的变更对调用者透明。 +- 代码复用： 创建代码抽离到独立的工厂类之后可以复用。 +- 隔离复杂性：封装复杂的创建逻辑，调用者无需了解如何创建对象。 +- 控制复杂度：将创建代码抽离出来，让原本的函数或类职责更单一，代码更简洁。 + +除此之外，我们还讲了工厂模式一个非常经典的应用场景：依赖注入框架，比如 Spring IOC、Google Guice，它用来集中创建、组装、管理对象，跟具体业务代码解耦，让程序员聚焦在业务代码的开发上。DI 框架已经成为了我们平时开发的必备框架。 + +### **3.** 建造者模式 + +建造者模式用来创建复杂对象，可以通过设置不同的可选参数，“定制化”地创建不同的对象。建造者模式的原理和实现比较简单，重点是掌握应用场景，避免过度使用。 + +如果一个类中有很多属性，为了避免构造函数的参数列表过长，影响代码的可读性和易用性，我们可以通过构造函数配合 set() 方法来解决。但是，如果存在下面情况中的任意一种，我们就要考虑使用建造者模式了。 + +- 我们把类的必填属性放到构造函数中，强制创建对象的时候就设置。如果必填的属性有很多，把这些必填属性都放到构造函数中设置，那构造函数就又会出现参数列表很长的问题。如果我们把必填属性通过 set() 方法设置，那校验这些必填属性是否已经填写的逻辑就无处安放了。 + +- 如果类的属性之间有一定的依赖关系或者约束条件，我们继续使用构造函数配合 set() 方法的设计思路，那这些依赖关系或约束条件的校验逻辑就无处安放了。 + +- 如果我们希望创建不可变对象，也就是说，对象在创建好之后，就不能再修改内部的属性值，要实现这个功能，我们就不能在类中暴露 set() 方法。构造函数配合 set() 方法来设置属性值的方式就不适用了。 + +### **4.** 原型模式 + +如果对象的创建成本比较大，而同一个类的不同对象之间差别不大（大部分字段都相同），在这种情况下，我们可以利用对已有对象（原型）进行复制（或者叫拷贝）的方式，来创建新对象，以达到节省创建时间的目的。这种基于原型来创建对象的方式就叫作原型模式。 + +原型模式有两种实现方法，深拷贝和浅拷贝。浅拷贝只会复制对象中基本数据类型数据和引用对象的内存地址，不会递归地复制引用对象，以及引用对象的引用对象……而深拷贝得到的是一份完完全全独立的对象。所以，深拷贝比起浅拷贝来说，更加耗时，更加耗内存空间。 + +如果要拷贝的对象是不可变对象，浅拷贝共享不可变对象是没问题的，但对于可变对象来说，浅拷贝得到的对象和原始对象会共享部分数据，就有可能出现数据被修改的风险，也就变得复杂多了。除非操作非常耗时，比较推荐使用浅拷贝，否则，没有充分的理由，不要为 + +了一点点的性能提升而使用浅拷贝。 + +## 二、结构型设计模式 + +结构型模式主要总结了一些类或对象组合在一起的经典结构，这些经典的结构可以解决特定应用场景的问题。结构型模式包括：代理模式、桥接模式、装饰器模式、适配器模式、门面模式、组合模式、享元模式。 + +### 1. **代理模式** + +代理模式在不改变原始类接口的条件下，为原始类定义一个代理类，主要目的是控制访问，而非加强功能，这是它跟装饰器模式最大的不同。一般情况下，我们让代理类和原始类实现同样的接口。但是，如果原始类并没有定义接口，并且原始类代码并不是我们开发维护的。 + +在这种情况下，我们可以通过让代理类继承原始类的方法来实现代理模式。 + +静态代理需要针对每个类都创建一个代理类，并且每个代理类中的代码都有点像模板式的“重复”代码，增加了维护成本和开发成本。对于静态代理存在的问题，我们可以通过动态代理来解决。我们不事先为每个原始类编写代理类，而是在运行的时候动态地创建原始类对应的代理类，然后在系统中用代理类替换掉原始类。 + +代理模式常用在业务系统中开发一些非功能性需求，比如：监控、统计、鉴权、限流、事务、幂等、日志。我们将这些附加功能与业务功能解耦，放到代理类统一处理，让程序员只需要关注业务方面的开发。除此之外，代理模式还可以用在 RPC、缓存等应用场景中。 + +### 2. **桥接模式** + +桥接模式的代码实现非常简单，但是理解起来稍微有点难度，并且应用场景也比较局限，所以，相对来说，桥接模式在实际的项目中并没有那么常用，你只需要简单了解，见到能认识就可以了，并不是我们学习的重点。 + +桥接模式有两种理解方式。第一种理解方式是“将抽象和实现解耦，让它们能独立开发”。 + +这种理解方式比较特别，应用场景也不多。另一种理解方式更加简单，等同于“组合优于继承”设计原则，这种理解方式更加通用，应用场景比较多。不管是哪种理解方式，它们的代码结构都是相同的，都是一种类之间的组合关系。 + +对于第一种理解方式，弄懂定义中“抽象”和“实现”两个概念，是理解它的关键。定义中的“抽象”，指的并非“抽象类”或“接口”，而是被抽象出来的一套“类库”，它只包含骨架代码，真正的业务逻辑需要委派给定义中的“实现”来完成。而定义中的“实现”，也并非“接口的实现类”，而是的一套独立的“类库”。“抽象”和“实现”独立开发，通过对象之间的组合关系组装在一起。 + +### **3.** 装饰器模式 + +装饰器模式主要解决继承关系过于复杂的问题，通过组合来替代继承，给原始类添加增强功能。这也是判断是否该用装饰器模式的一个重要的依据。除此之外，装饰器模式还有一个特点，那就是可以对原始类嵌套使用多个装饰器。为了满足这样的需求，在设计的时候，装饰 + +器类需要跟原始类继承相同的抽象类或者接口。 + +### 4. 适配器模式 + +代理模式、装饰器模式提供的都是跟原始类相同的接口，而适配器提供跟原始类不同的接口。适配器模式是用来做适配的，它将不兼容的接口转换为可兼容的接口，让原本由于接口不兼容而不能一起工作的类可以一起工作。 + +适配器模式有两种实现方式：类适配器和对象适配器。其中，类适配器使用继承关系来实现，对象适配器使用组合关系来实现。适配器模式是一种事后的补救策略，用来补救设计上的缺陷。应用这种模式算是“无奈之举”。如果在设计初期，我们就能规避接口不兼容的问题，那这种模式就无用武之地了。 + +在实际的开发中，什么情况下才会出现接口不兼容呢？我总结下了下面这 5 种场景： + +- 封装有缺陷的接口设计 +- 统一多个类的接口设计 +- 替换依赖的外部系统 +- 兼容老版本接口 +- 适配不同格式的数据 + +### 5. **门面模式** + +门面模式原理、实现都非常简单，应用场景比较明确。它通过封装细粒度的接口，提供组合各个细粒度接口的高层次接口，来提高接口的易用性，或者解决性能、分布式事务等问题。 + +### 6. 组合模式 + +组合模式跟我们之前讲的面向对象设计中的“组合关系（通过组合来组装两个类）”，完全是两码事。这里讲的“组合模式”，主要是用来处理树形结构数据。正因为其应用场景的特殊性，数据必须能表示成树形结构，这也导致了这种模式在实际的项目开发中并不那么常用。但是，一旦数据满足树形结构，应用这种模式就能发挥很大的作用，能让代码变得非常简洁。 + +组合模式的设计思路，与其说是一种设计模式，倒不如说是对业务场景的一种数据结构和算法的抽象。其中，数据可以表示成树这种数据结构，业务需求可以通过在树上的递归遍历算法来实现。组合模式，将一组对象组织成树形结构，将单个对象和组合对象都看作树中的节 + +点，以统一处理逻辑，并且它利用树形结构的特点，递归地处理每个子树，依次简化代码实现。 + +### **7.** 享元模式 + +所谓“享元”，顾名思义就是被共享的单元。享元模式的意图是复用对象，节省内存，前提是享元对象是不可变对象。具体来讲，当一个系统中存在大量重复对象的时候，我们就可以利用享元模式，将对象设计成享元，在内存中只保留一份实例，供多处代码引用，这样可以减少内存中对象的数量，以起到节省内存的目的。实际上，不仅仅相同对象可以设计成享元，对于相似对象，我们也可以将这些对象中相同的部分（字段），提取出来设计成享元，让这些大量相似对象引用这些享元。 + +## 三、行为型设计模式 + +我们知道，创建型设计模式主要解决“对象的创建”问题，结构型设计模式主要解决“类或对象的组合”问题，那行为型设计模式主要解决的就是“类或对象之间的交互”问题。行为型模式比较多，有 11 种，它们分别是：观察者模式、模板模式、策略模式、职责链模式、迭代器模式、状态模式、访问者模式、备忘录模式、命令模式、解释器模式、中介模式。 + +### **1.** 观察者模式 + +观察者模式将观察者和被观察者代码解耦。观察者模式的应用场景非常广泛，小到代码层面的解耦，大到架构层面的系统解耦，再或者一些产品的设计思路，都有这种模式的影子，比如，邮件订阅、RSS Feeds，本质上都是观察者模式。 + +不同的应用场景和需求下，这个模式也有截然不同的实现方式：有同步阻塞的实现方式，也有异步非阻塞的实现方式；有进程内的实现方式，也有跨进程的实现方式。同步阻塞是最经典的实现方式，主要是为了代码解耦；异步非阻塞除了能实现代码解耦之外，还能提高代码 + +的执行效率；进程间的观察者模式解耦更加彻底，一般是基于消息队列来实现，用来实现不同进程间的被观察者和观察者之间的交互。 + +框架的作用有隐藏实现细节，降低开发难度，实现代码复用，解耦业务与非业务代码，让程序员聚焦业务开发。针对异步非阻塞观察者模式，我们也可以将它抽象成 EventBus 框架来达到这样的效果。EventBus 翻译为“事件总线”，它提供了实现观察者模式的骨架代码。 + +我们可以基于此框架非常容易地在自己的业务场景中实现观察者模式，不需要从零开始开发。 + +### **2.** 模板模式 + +模板方法模式在一个方法中定义一个算法骨架，并将某些步骤推迟到子类中实现。模板方法模式可以让子类在不改变算法整体结构的情况下，重新定义算法中的某些步骤。这里的“算法”，我们可以理解为广义上的“业务逻辑”，并不特指数据结构和算法中的“算法”。这里的算法骨架就是“模板”，包含算法骨架的方法就是“模板方法”，这也是模板方法模式名字的由来。 + +模板模式有两大作用：复用和扩展。 + +其中复用指的是，所有的子类可以复用父类中提供的模板方法的代码。扩展指的是，框架通过模板模式提供功能扩展点，让框架用户可以在不修改框架源码的情况下，基于扩展点定制化框架的功能。 + +除此之外，还有回调。它跟模板模式具有相同的作用：代码复用和扩展。在一些框架、类库、组件等的设计中经常会用到，比如JdbcTemplate 就是用了回调。 + +相对于普通的函数调用，回调是一种双向调用关系。A 类事先注册某个函数 F 到 B 类，A 类在调用 B 类的 P 函数的时候，B 类反过来调用 A 类注册给它的 F 函数。这里的 F 函数就是“回调函数”。A 调用 B，B 反过来又调用 A，这种调用机制就叫作“回调”。 + +回调可以细分为同步回调和异步回调。从应用场景上来看，同步回调看起来更像模板模式，异步回调看起来更像观察者模式。回调跟模板模式的区别，更多的是在代码实现上，而非应用场景上。回调基于组合关系来实现，模板模式基于继承关系来实现。回调比模板模式更加 + +灵活。 + +### 3.策略模式 + +策略模式定义一族算法类，将每个算法分别封装起来，让它们可以互相替换。策略模式可以使算法的变化独立于使用它们的客户端（这里的客户端代指使用算法的代码）。策略模式用来解耦策略的定义、创建、使用。实际上，一个完整的策略模式就是由这三个部分组成的。 + +策略类的定义比较简单，包含一个策略接口和一组实现这个接口的策略类。策略的创建由工厂类来完成，封装策略创建的细节。策略模式包含一组策略可选，客户端代码选择使用哪个策略，有两种确定方法：编译时静态确定和运行时动态确定。其中，“运行时动态确定”才 + +是策略模式最典型的应用场景。 + +在实际的项目开发中，策略模式也比较常用。最常见的应用场景是，利用它来避免冗长的 if-else 或 switch 分支判断。不过，它的作用还不止如此。它也可以像模板模式那样，提供框架的扩展点等等。实际上，策略模式主要的作用还是解耦策略的定义、创建和使用，控制 + +代码的复杂度，让每个部分都不至于过于复杂、代码量过多。除此之外，对于复杂代码来说，策略模式还能让其满足开闭原则，添加新策略的时候，最小化、集中化代码改动，减少引入 bug 的风险。 + +### **4.** 职责链模式 + +在职责链模式中，多个处理器依次处理同一个请求。一个请求先经过 A 处理器处理，然后再把请求传递给 B 处理器，B 处理器处理完后再传递给 C 处理器，以此类推，形成一个链条。链条上的每个处理器各自承担各自的处理职责，所以叫作职责链模式。 + +在 GoF 的定义中，一旦某个处理器能处理这个请求，就不会继续将请求传递给后续的处理器了。当然，在实际的开发中，也存在对这个模式的变体，那就是请求不会中途终止传递，而是会被所有的处理器都处理一遍。 + +职责链模式常用在框架开发中，用来实现过滤器、拦截器功能，让框架的使用者在不需要修改框架源码的情况下，添加新的过滤、拦截功能。这也体现了对扩展开放、对修改关闭的设计原则。 + +### **5.** 迭代器模式 + +迭代器模式也叫游标模式，它用来遍历集合对象。这里说的“集合对象”，我们也可以叫“容器”“聚合对象”，实际上就是包含一组对象的对象，比如，数组、链表、树、图、跳表。迭代器模式主要作用是解耦容器代码和遍历代码。大部分编程语言都提供了现成的迭代器可以使用，我们不需要从零开始开发。 + +遍历集合一般有三种方式：for 循环、foreach 循环、迭代器遍历。后两种本质上属于一种，都可以看作迭代器遍历。相对于 for 循环遍历，利用迭代器来遍历有 3 个优势： + +- 迭代器模式封装集合内部的复杂数据结构，开发者不需要了解如何遍历，直接使用容器提供的迭代器即可； + +- 迭代器模式将集合对象的遍历操作从集合类中拆分出来，放到迭代器类中，让两者的职责更加单一； + +- 迭代器模式让添加新的遍历算法更加容易，更符合开闭原则。除此之外，因为迭代器都实现自相同的接口，在开发中，基于接口而非实现编程，替换迭代器也变得更加容易。 + +在通过迭代器来遍历集合元素的同时，增加或者删除集合中的元素，有可能会导致某个元素被重复遍历或遍历不到。针对这个问题，有两种比较干脆利索的解决方案，来避免出现这种不可预期的运行结果。一种是遍历的时候不允许增删元素，另一种是增删元素之后让遍历报 + +错。第一种解决方案比较难实现，因为很难确定迭代器使用结束的时间点。第二种解决方案更加合理，Java 语言就是采用的这种解决方案。增删元素之后，我们选择 fail-fast 解决方式，让遍历操作直接抛出运行时异常。 + +### **6.** 状态模式 + +状态模式一般用来实现状态机，而状态机常用在游戏、工作流引擎等系统开发中。状态机又叫有限状态机，它由 3 个部分组成：状态、事件、动作。其中，事件也称为转移条件。事件触发状态的转移及动作的执行。不过，动作不是必须的，也可能只转移状态，不执行任何 + +动作。 + +针对状态机，我们总结了三种实现方式。 + +- 第一种实现方式叫分支逻辑法。利用 if-else 或者 switch-case 分支逻辑，参照状态转移图，将每一个状态转移原模原样地直译成代码。对于简单的状态机来说，这种实现方式最简单、最直接，是首选。 + +- 第二种实现方式叫查表法。对于状态很多、状态转移比较复杂的状态机来说，查表法比较合适。通过二维数组来表示状态转移图，能极大地提高代码的可读性和可维护性。 + +- 第三种实现方式就是利用状态模式。对于状态并不多、状态转移也比较简单，但事件触发执行的动作包含的业务逻辑可能比较复杂的状态机来说，我们首选这种实现方式。 + +### **7.** 访问者模式 + +访问者模式允许一个或者多个操作应用到一组对象上，设计意图是解耦操作和对象本身，保持类职责单一、满足开闭原则以及应对代码的复杂性。 + +对于访问者模式，学习的主要难点在代码实现。而代码实现比较复杂的主要原因是，函数重载在大部分面向对象编程语言中是静态绑定的。也就是说，调用类的哪个重载函数，是在编译期间，由参数的声明类型决定的，而非运行时，根据参数的实际类型决定的。 + +正是因为代码实现难理解，所以，在项目中应用这种模式，会导致代码的可读性比较差。如果你的同事不了解这种设计模式，可能就会读不懂、维护不了你写的代码。所以，除非不得已，不要使用这种模式。 + +### 8. 备忘录模式 + +备忘录模式也叫快照模式，具体来说，就是在不违背封装原则的前提下，捕获一个对象的内部状态，并在该对象之外保存这个状态，以便之后恢复对象为先前的状态。这个模式的定义表达了两部分内容：一部分是，存储副本以便后期恢复；另一部分是，要在不违背封装原则 + +的前提下，进行对象的备份和恢复。 + +备忘录模式的应用场景也比较明确和有限，主要用来防丢失、撤销、恢复等。它跟平时我们常说的“备份”很相似。两者的主要区别在于，备忘录模式更侧重于代码的设计和实现，备份更侧重架构设计或产品设计。 + +对于大对象的备份来说，备份占用的存储空间会比较大，备份和恢复的耗时会比较长。针对这个问题，不同的业务场景有不同的处理方式。比如，只备份必要的恢复信息，结合最新的数据来恢复；再比如，全量备份和增量备份相结合，低频全量备份，高频增量备份，两者结合来做恢复。 + +### **9.** 命令模式 + +命令模式在平时工作中并不常用，你稍微了解一下就可以。 + +落实到编码实现，命令模式用到最核心的实现手段，就是将函数封装成对象。我们知道，在大部分编程语言中，函数是没法作为参数传递给其他函数的，也没法赋值给变量。借助命令模式，我们将函数封装成对象，这样就可以实现把函数像对象一样使用。 + +命令模式的主要作用和应用场景，是用来控制命令的执行，比如，异步、延迟、排队执行命令、撤销重做命令、存储命令、给命令记录日志等，这才是命令模式能发挥独一无二作用的地方。 + +### **10.** 解释器模式 + +解释器模式为某个语言定义它的语法（或者叫文法）表示，并定义一个解释器用来处理这个语法。实际上，这里的“语言”不仅仅指我们平时说的中、英、日、法等各种语言。从广义上来讲，只要是能承载信息的载体，我们都可以称之为“语言”，比如，古代的结绳记事、 + +盲文、哑语、摩斯密码等。 + +要想了解“语言”要表达的信息，我们就必须定义相应的语法规则。这样，书写者就可以根据语法规则来书写“句子”（专业点的叫法应该是“表达式”），阅读者根据语法规则来阅读“句子”，这样才能做到信息的正确传递。而我们要讲的解释器模式，其实就是用来实现根据语法规则解读“句子”的解释器。 + +解释器模式的代码实现比较灵活，没有固定的模板。我们前面说过，应用设计模式主要是应对代码的复杂性，解释器模式也不例外。它的代码实现的核心思想，就是将语法解析的工作拆分到各个小类中，以此来避免大而全的解析类。一般的做法是，将语法规则拆分一些小的 + +独立的单元，然后对每个单元进行解析，最终合并为对整个语法规则的解析。 + +### **11.** 中介模式 + +中介模式的设计思想跟中间层很像，通过引入中介这个中间层，将一组对象之间的交互关系（或者说依赖关系）从多对多（网状关系）转换为一对多（星状关系）。原来一个对象要跟 n 个对象交互，现在只需要跟一个中介对象交互，从而最小化对象之间的交互关系，降低了 + +代码的复杂度，提高了代码的可读性和可维护性。 + +观察者模式和中介模式都是为了实现参与者之间的解耦，简化交互关系。两者的不同在于应用场景上。在观察者模式的应用场景中，参与者之间的交互比较有条理，一般都是单向的，一个参与者只有一个身份，要么是观察者，要么是被观察者。而在中介模式的应用场景中，参与者之间的交互关系错综复杂，既可以是消息的发送者、也可以同时是消息的接收者。 diff --git a/docs/design-pattern/Template-Pattern.md b/docs/design-pattern/Template-Pattern.md new file mode 100644 index 0000000000..036d55f88f --- /dev/null +++ b/docs/design-pattern/Template-Pattern.md @@ -0,0 +1,355 @@ +--- +title: 模板方法模式——看看 JDK 和 Spring 是如何优雅复用代码的 +date: 2022-11-09 +tags: + - Design Patterns +categories: Design Patterns +--- + +![](https://images.unsplash.com/photo-1655892796775-c947f39f1106?w=1200&auto=format&fit=crop&q=60&ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxzZWFyY2h8MzV8fHRlbXBsYXRlfGVufDB8fDB8fHww) + +> 模板，顾名思义，它是一个固定化、标准化的东西。 +> +> **模板方法模式**是一种行为设计模式， 它在超类中定义了一个算法的框架， 允许子类在不修改结构的情况下重写算法的特定步骤。 + + + +### 场景问题 + +程序员不愿多扯，上来先干两行代码 + +网上模板方法的场景示例特别多，个人感觉还是《Head First 设计模式》中的例子比较好。 + +假设我们是一家饮品店的师傅，起码需要以下两个手艺 + +![](https://static001.geekbang.org/infoq/19/196f4c041c71d63442c2c688051c893a.jpeg) + +真简单哈，这么看，步骤大同小异，我的第一反应就是写个业务接口，不同的饮品实现其中的方法就行，像这样 + +![img](https://static001.geekbang.org/infoq/a2/a226b8a9219cd7a29af2f3b626cd5fcb.jpeg) + + + +画完类图，猛地发现，第一步和第三步没什么差别，而且做饮品是个流程式的工作，我希望使用时，直接调用一个方法，就去执行对应的制作步骤。 + +灵机一动，不用接口了，用一个**抽象父类**，把步骤方法放在一个大的流程方法 `makingDrinks()` 中，且第一步和第三步，完全一样，没必要在子类实现，改进如下 + +![img](https://static001.geekbang.org/infoq/21/2146409c43149828b8a12949daf5b0a4.jpeg) + +再看下我们的设计，感觉还不错，现在用同一个 `makingDrinks()` 方法来处理咖啡和茶的制作，而且我们不希望子类覆盖这个方法，所以可以申明为 final，不同的制作步骤，我们希望子类来提供，必须在父类申明为抽象方法，而第一步和第三步我们不希望子类重写，所以我们声明为非抽象方法 + +```java +public abstract class Drinks { + + void boilWater() { + System.out.println("将水煮沸"); + } + + abstract void brew(); + + void pourInCup() { + System.out.println("倒入杯子"); + } + + abstract void addCondiments(); + + public final void makingDrinks() { + //热水 + boilWater(); + //冲泡 + brew(); + //倒进杯子 + pourInCup(); + //加料 + addCondiments(); + } +} +``` + +接着，我们分别处理咖啡和茶，这两个类只需要**继承**父类，重写其中的抽象方法即可（实现各自的冲泡和添加调料） + +```java +public class Tea extends Drinks { + @Override + void brew() { + System.out.println("冲茶叶"); + } + @Override + void addCondiments() { + System.out.println("加柠檬片"); + } +} +``` + +```java +public class Coffee extends Drinks { + @Override + void brew() { + System.out.println("冲咖啡粉"); + } + + @Override + void addCondiments() { + System.out.println("加奶加糖"); + } +} +``` + +现在可以上岗了，试着制作下咖啡和茶吧 + +```java +public static void main(String[] args) { + Drinks coffee = new Coffee(); + coffee.makingDrinks(); + System.out.println(); + Drinks tea = new Tea(); + tea.makingDrinks(); +} +``` + +好嘞，又学会一个设计模式，这就是**模板方法模式**，我们的 `makingDrinks()` 就是模板方法。我们可以看到相同的步骤 `boilWater()` 和 `pourInCup()` 只在父类中进行即可，不同的步骤放在子类实现。 + + + +### 认识模板方法 + +在阎宏博士的《JAVA与模式》一书中开头是这样描述模板方法（Template Method）模式的： + +> 模板方法模式是类的行为模式。准备一个抽象类，将部分逻辑以具体方法以及具体构造函数的形式实现，然后声明一些抽象方法来迫使子类实现剩余的逻辑。不同的子类可以以不同的方式实现这些抽象方法，从而对剩余的逻辑有不同的实现。这就是模板方法模式的用意。 + +写代码的一个很重要的思考点就是“**变与不变**”，程序中哪些功能是可变的，哪些功能是不变的，我们可以把不变的部分抽象出来，进行公共的实现，把变化的部分分离出来，用接口来封装隔离，或用抽象类约束子类行为。模板方法就很好的体现了这一点。 + +模板方法定义了一个算法的步骤，并允许子类为一个或多个步骤提供实现。 + +模板方法模式是所有模式中最为常见的几个模式之一，是**基于继承**的代码复用的基本技术，我们再看下类图 + +![img](https://static001.geekbang.org/infoq/b1/b114ec408fb0231529d8748618df9ed7.jpeg) + +模板方法模式就是用来创建一个算法的模板，这个模板就是方法，该方法将算法定义成一组步骤，其中的任意步骤都可能是抽象的，由子类负责实现。这样可以**确保算法的结构保持不变，同时由子类提供部分实现**。 + + + +再回顾下我们制作咖啡和茶的例子，有些顾客要不希望咖啡加糖或者不希望茶里加柠檬，我们要改造下模板方法，在加相应的调料之前，问下顾客 + +```java +public abstract class Drinks { + + void boilWater() { + System.out.println("将水煮沸"); + } + + abstract void brew(); + + void pourInCup() { + System.out.println("倒入杯子"); + } + + abstract void addCondiments(); + + public final void makingDrinks() { + boilWater(); + brew(); + pourInCup(); + + //如果顾客需要，才加料 + if (customerLike()) { + addCondiments(); + } + } + + //定义一个空的缺省方法，只返回 true + boolean customerLike() { + return true; + } +} +``` + +如上，我们加了一个逻辑判断，逻辑判断的方法是一个只返回 true 的方法，这个方法我们叫做 **钩子方法**。 + +> 钩子：在模板方法的父类中，我们可以定义一个方法，它默认不做任何事，子类可以视情况要不要覆盖它，该方法称为“钩子”。 + +钩子方法一般是空的或者有默认实现。钩子的存在，可以让子类有能力对算法的不同点进行挂钩。而要不要挂钩，又由子类去决定。 + +是不是很有用呢，我们再看下咖啡的制作 + +```java +public class Coffee extends Drinks { + @Override + void brew() { + System.out.println("冲咖啡粉"); + } + + @Override + void addCondiments() { + System.out.println("加奶加糖"); + } + //覆盖了钩子，提供了自己的询问功能，让用户输入是否需要加料 + boolean customerLike() { + String answer = getUserInput(); + if (answer.toLowerCase().startsWith("y")) { + return true; + } else { + return false; + } + } + + //处理用户的输入 + private String getUserInput() { + String answer = null; + System.out.println("您想要加奶加糖吗？输入 YES 或 NO"); + BufferedReader reader = new BufferedReader(new InputStreamReader(System.in)); + try { + answer = reader.readLine(); + } catch (IOException e) { + e.printStackTrace(); + } + if (answer == null) { + return "no"; + } + return answer; + } +} +``` + +接着再去测试下代码，看看结果吧。 + +![img](https://static001.geekbang.org/infoq/e6/e608360ad552adc44dab00293fddd671.jpeg) + + + +我想你应该知道钩子的好处了吧，它可以作为条件控制，影响抽象类中的算法流程，当然也可以什么都不做。 + + + +模板方法有很多种实现，有时看起来可能不是我们所谓的“中规中矩”的设计。接下来我们看下 JDK 和 Spring 中是怎么使用模板方法的。 + +### JDK 中的模板方法 + +我们写代码经常会用到 **comparable** 比较器来对数组对象进行排序，我们都会实现它的 `compareTo()` 方法，之后就可以通过 `Collections.sort()` 或者 `Arrays.sort()` 方法进行排序了。 + +具体的实现类就不写了(可以去 github：starfish-learning 上看我的代码)，看下使用 + +```java +@Override +public int compareTo(Object o) { + Coffee coffee = (Coffee) o; + if(this.price < (coffee.price)){ + return -1; + }else if(this.price == coffee.price){ + return 0; + }else{ + return 1; + } +} +``` + +```java +public static void main(String[] args) { + Coffee[] coffees = {new Coffee("星冰乐",38), + new Coffee("拿铁",32), + new Coffee("摩卡",35)}; + + Arrays.sort(coffees); + + for (Coffee coffee1 : coffees) { + System.out.println(coffee1); + } + +} +``` + +![img](https://static001.geekbang.org/infoq/92/9241c28ee542321b3e6f4e2a2fbf805a.jpeg) + +你可能会说，这个看着不像我们常规的模板方法，是的。我们看下比较器实现的步骤 + +1. 构建对象数组 +2. 通过 `Arrays.sort` 方法对数组排序，传参为 `Comparable` 接口的实例 +3. 比较时候会调用我们的实现类的 `compareTo()` 方法 +4. 将排好序的数组设置进原数组中，排序完成 + +一脸懵逼，这个实现竟然也是模板方法。 + +这个模式的重点在于提供了一个固定算法框架，并让子类实现某些步骤，虽然使用继承是标准的实现方式，但通过回调来实现，也不能说这就不是模板方法。 + +其实并发编程中最常见，也是面试必问的 AQS 就是一个典型的模板方法。 + + + +### Spring 中的模板方法 + +Spring 中的设计模式太多了，而且大部分扩展功能都可以看到模板方法模式的影子。 + +我们看下 IOC 容器初始化时的模板方法，不管是 XML 还是注解的方式，对于核心容器启动流程都是一致的。 + +`AbstractApplicationContext` 的 `refresh` 方法实现了 IOC 容器启动的主要逻辑。 + +一个 `refresh()` 方法包含了好多其他步骤方法，像不像我们说的 **模板方法**，`getBeanFactory()` 、`refreshBeanFactory()` 是子类必须实现的抽象方法，`postProcessBeanFactory()` 是钩子方法。 + +```java +public abstract class AbstractApplicationContext extends DefaultResourceLoader + implements ConfigurableApplicationContext { + @Override + public void refresh() throws BeansException, IllegalStateException { + synchronized (this.startupShutdownMonitor) { + prepareRefresh(); + ConfigurableListableBeanFactory beanFactory = obtainFreshBeanFactory(); + prepareBeanFactory(beanFactory); + postProcessBeanFactory(beanFactory); + invokeBeanFactoryPostProcessors(beanFactory); + registerBeanPostProcessors(beanFactory); + initMessageSource(); + initApplicationEventMulticaster(); + onRefresh(); + registerListeners(); + finishBeanFactoryInitialization(beanFactory); + finishRefresh(); + } + } + // 两个抽象方法 + @Override + public abstract ConfigurableListableBeanFactory getBeanFactory() throws IllegalStateException; + + protected abstract void refreshBeanFactory() throws BeansException, IllegalStateException; + + //钩子方法 + protected void postProcessBeanFactory(ConfigurableListableBeanFactory beanFactory) { + } + } +``` + +打开你的 IDEA，我们会发现常用的 `ClassPathXmlApplicationContext` 和 `AnnotationConfigApplicationContext` 启动入口，都是它的实现类（子类的子类的子类的...）。 + +`AbstractApplicationContext` 的一个子类 `AbstractRefreshableWebApplicationContext` 中有钩子方法 `onRefresh() ` 的实现： + +```java +public abstract class AbstractRefreshableWebApplicationContext extends …… { + /** + * Initialize the theme capability. + */ + @Override + protected void onRefresh() { + this.themeSource = UiApplicationContextUtils.initThemeSource(this); + } +} +``` + +看下大概的类图： + +![img](https://static001.geekbang.org/infoq/13/1360f5528a2e86e5b0d0bf3a97b3c04b.jpeg) + + + +## 小总结 + +- **优点**：1、封装不变部分，扩展可变部分。 2、提取公共代码，便于维护。 3、行为由父类控制，子类实现。 +- **缺点**：每一个不同的实现都需要一个子类来实现，导致类的个数增加，使得系统更加庞大。 +- **使用场景**： 1、有多个子类共有的方法，且逻辑相同。 2、重要的、复杂的方法，可以考虑作为模板方法。 +- **注意事项**：为防止恶意操作，一般模板方法都加上 final 关键词。 + + + +## 参考： + +- 《Head First 设计模式》 +- 《研磨设计模式》 +- https://sourcemaking.com/design_patterns/template_method diff --git a/docs/design-pattern/readDisignPattern.md b/docs/design-pattern/readDisignPattern.md deleted file mode 100644 index a2690bbf8f..0000000000 --- a/docs/design-pattern/readDisignPattern.md +++ /dev/null @@ -1,6 +0,0 @@ -![](https://images.pexels.com/photos/196644/pexels-photo-196644.jpeg?cs=srgb&dl=notebook-beside-the-iphone-on-table-196644.jpg&fm=jpg) - - - -文章都在设计模式二级目录下 - diff --git a/docs/design-pattern/sidebar.md b/docs/design-pattern/sidebar.md deleted file mode 100644 index 250fc646e6..0000000000 --- a/docs/design-pattern/sidebar.md +++ /dev/null @@ -1,57 +0,0 @@ -- **Java基础** -- [![](https://icongr.am/simple/oracle.svg?size=25&color=231c82&colored=false)JVM](java/JVM/readJVM.md) -- [![img](https://icongr.am/fontawesome/expeditedssl.svg?size=25&color=f23131)JUC](java/JUC/readJUC.md) -- [![](https://icongr.am/devicon/java-original.svg?size=25&color=f23131)Java 8](java/Java8.md) -- [![img](https://icongr.am/entypo/address.svg?size=25&color=074ca6)设计模式](design-pattern/readDisignPattern.md) - - [设计模式前传](design-pattern/Design-Pattern-Overview.md) - - [单例模式](design-pattern/Singleton-Pattern.md) - - [工厂模式](design-pattern/Factory-Pattern.md) - - [装饰模式](design-pattern/Decorator-Pattern.md) - - [代理模式](design-pattern/Proxy-Pattern.md) - - [观察者模式](design-pattern/Observer-Pattern.md) - - [适配器模式](design-pattern/Adapter-Pattern.md) - - [责任链模式](design-pattern/Chain-of-Responsibility-Pattern.md) -- **数据存储和缓存** -- [![MySQL](https://icongr.am/devicon/mysql-original.svg?&size=25)MySQL](data-store/MySQL/readMySQL.md) -- [![Redis](https://icongr.am/devicon/redis-original.svg?size=25)Redis](data-store/Redis/2.readRedis.md) -- [![mongoDB](https://icongr.am/devicon/mongodb-original.svg?&size=25)mongoDB]( https://redis.io/ ) -- [![ **Elasticsearch** ](https://icongr.am/simple/elasticsearch.svg?&size=20) Elasticsearch]( https://redis.io/ ) -- [![S3](https://icongr.am/devicon/amazonwebservices-original.svg?&size=25)S3]( https://aws.amazon.com/cn/s3/ ) -- **直击面试** -- [![](https://icongr.am/material/basket.svg?size=25)Java集合面试](interview/Collections-FAQ.md) -- [![](https://icongr.am/devicon/java-plain-wordmark.svg?size=25)JVM面试](interview/JVM-FAQ.md) -- [![](https://icongr.am/devicon/mysql-original-wordmark.svg?size=25)MySQL面试](interview/MySQL-FAQ.md) -- [![](https://icongr.am/devicon/redis-original-wordmark.svg?size=25)Redis面试](interview/Redis-FAQ.md) -- [![](https://icongr.am/jam/leaf.svg?size=25&color=00FF00)Spring面试](interview/Spring-FAQ.md) -- [![](https://icongr.am/simple/bower.svg?size=25)MyBatis面试](interview/MyBatis-FAQ.md) -- [![img](https://icongr.am/entypo/network.svg?size=25&color=6495ED)计算机网络](interview/Network-FAQ.md) -- **单体架构** -- **RPC** -- [Hello Protocol Buffers](rpc/Hello-Protocol-Buffers.md) -- **面向服务架构** -- [![message](https://icongr.am/clarity/email.svg?&size=16) 消息中间件](message-queue/readMQ.md) -- [![Nginx](https://icongr.am/devicon/nginx-original.svg?&size=16)Nginx](nginx/nginx.md) -- **微服务架构** -- [🍃 Spring Boot](springboot/Hello-SpringBoot.md) -- [🍃 定时任务@Scheduled](springboot/Spingboot定时任务@Scheduled.md) -- [🍃 Spring Cloud](https://spring.io/projects/spring-cloud) -- **大数据** -- [Hello 大数据](big-data/Hello-BigData.md) -- [![](https://icongr.am/simple/oracle.svg?size=25&color=231c82&colored=false)Kafka](message-queue/Kafka/readKafka.md) -- **性能优化** -- JVM优化 -- web调优 -- DB调优 -- **工程化与工具** -- [![Maven](https://icongr.am/devicon//fontawesome/maxcdn.svg?&size=16)Maven](logging/logback简单使用.md) -- [![Git](https://icongr.am/devicon/git-original.svg?&size=16)Git](logging/logback简单使用.md) -- [Sonar](https://www.sonarqube.org/) -- **其他** -- [![Linux](https://icongr.am/devicon/linux-original.svg?&size=16)Linux](linux/linux.md) -- [缕清各种Java Logging](logging/Java-Logging.md) -- [hello logback](logging/logback简单使用.md) -- SHELL -- TCP与HTTP -- **Links** -- [![Github](https://icongram.jgog.in/simple/github.svg?color=808080&size=16)Github](https://github.com/jhildenbiddle/docsify-tabs) -- [![Blog](https://icongr.am/simple/aboutme.svg?colored&size=16)My Blog](https://www.lazyegg.net) \ No newline at end of file diff --git a/docs/distribution/.DS_Store b/docs/distribution/.DS_Store new file mode 100644 index 0000000000..6321481ed7 Binary files /dev/null and b/docs/distribution/.DS_Store differ diff --git a/docs/distribution/README.md b/docs/distribution/README.md new file mode 100644 index 0000000000..a8916a64d0 --- /dev/null +++ b/docs/distribution/README.md @@ -0,0 +1,125 @@ +# 分布式架构中的一些名词理解 + +> 一说分布式架构，就会看到各种 SOA、RMI、RPC 等等一脸懵逼的词汇，而且还特别容易混淆各种概念，记住这些吧，就不会质疑自己程序员的身份了。 + +### SOA + +SOA(Service Oriented Architecture) ，中文意思就是**面向服务架构**，它其实就是一种软件架构设计思想，不是具体的某种技术实现。 + +为什么会出现这种思想呢？ + +举一个列子，我们的某个产品，立项初期只有 PC 端的 Web 网站，提供了注册、登录和各种业务功能，后来项目火了，老板说需要做移动端的对应产品，Android 、IOS 一起搞，用户管理用同一个 DB，这时候我们就会发现如果在各个版本都做一套注册、登录这样的功能，不止效率低，而且要有相同的业务改动，这几个地方都要改。 + +这时候，Java 程序员、Android 程序员、IOS 程序员就商量说能不能搞个单独的项目提供注册、登录这样的用户服务，我们可以通过调用该服务的方法或者函数去实现功能，这样一来，以后就算又要做各种小程序版本，也可以去使用这个服务，不需要单独开发了。 + +这其实就属于 SOA 的思想。 + +后来随着产品的流量越来越大，项目也越来越大，我们会拆分出各种各样的服务，当服务越来越多，调用方也越来越多的时候，他们之间的关系就变得比较混乱了，作为开发者，我们肯定要清楚的知道调用方和服务方的各种关系和数据，所以就又出现了 『**服务治理** 』这样的概念，（可以类比环境治理），包括服务的监控，权限管理等等这样的功能，比如 dubbo 或者 SpringCloud 这类。 + + + +> SOA 是一种设计方法，其中包含多个服务， 而服务之间通过配合最终会提供一系列的功能。一个服务通常以独立的形式存在于操作系统进程中。服务之间通过网络调用，而非采用进程内调用的方式进行通信。 ——《微服务设计》 + + + +SOAP、REST、RPC 就是根据这种设计模式构建出来的规范，其中 SOAP 通俗理解就是 http+xml 的形式，REST 就是 http+json 的形式，RPC 大都是基于 socket 的形式 + + + +### SOAP + +Simple Object Access Protocol，即简单对象访问协议， 简称 SOAP。 + +SOAP 是一个用于分布式环境的、轻量级的、基于 XML 进行信息交换的通信协议，主要用于 Web 服务（web service）中。 + +还是用刚才的例子，现在我们的项目又拆分出来订单服务，我们要查询订单， + +用一个简单的例子来说明 SOAP 使用过程，一个 SOAP 消息可以发送到一个具有 Web Service 功能的 Web 站点。 + +https://segmentfault.com/a/1190000003772529 + +例如，一个含有房价信息的数据库，消息的参数中标明这是一个查询消息，此站点将返回一个 XML 格式的信息，其中包含了查询结果（价格，位置，特点，或者其他信息）。由于数据是用一种标准化的可分析的结构来传递的，所以可以直接被第三方站点所利用。 + + + +### RPC + +了解上面的RMI，它的主要的流程就是Client<-->stub<-->[NETWORK]<-->skeleton<-->Server，还有一个比较重要的概念就是RMIRegistry，其实大家网上去查RPC的时候流程其实都差不多，可能叫法和底层东西有点不一样，其实其实现所遵循的模型还是类似的。主要的区别的话是RMI是只适用于java的，而RPC任何语言都可以；第二点就是他们两者的调用方式不一样，最终的目标还是一致 + + + +其与RMI大致的区别 + +1. RPC 跨语言，而 RMI只支持Java。 + +2. RMI 调用远程对象方法，允许方法返回 Java 对象以及基本数据类型，而RPC 不支持对象的概念，传送到 RPC服务的消息由外部数据表示 (External Data Representation, XDR) 语言表示，这种语言抽象了字节序类和数据类型结构之间的差异。只有由 XDR 定义的数据类型才能被传递， 可以说 RMI 是面向对象方式的 Java RPC 。 + +3. 在方法调用上，RMI中，远程接口使每个远程方法都具有方法签名（url）。如果一个方法在服务器上执行，但是没有相匹配的签名被添加到这个远程接口上，那么这个新方法就不能被RMI客户方所调用。 + +在RPC中，当一个请求到达RPC服务器时，这个请求就包含了一个参数集和一个文本值，通常形成“classname.methodname”的形式。这就向RPC服务器表明，被请求的方法在为 “classname”的类中，名 + +叫“methodname”。然后RPC服务器就去搜索与之相匹配的类和方法，并把它作为那种方法参数类型的输入。这里的 + +参数类型是与RPC请求中的类型是匹配的。一旦匹配成功，这个方法就被调用了，其结果被编码后返回客户方。说的直白一点就是rmi是自己写一个url，如果正确就获得相应的stub，而rpc的url是从注册中心去拿的，不会出现url不对的情况 + +http://blog.jobbole.com/92290/ + + + +### rest + +比如有个url：http:www.test.com/user/1，这个地址既要表示删除id为1的用户、又要表示修改id为1的用户，还要表达获取id为1的用户，那么，就要用到http1.1的不同的请求方法：get、post、delete、put， + +对于rest这个东西，其实本人一点也没有接触吧，一下两个网址，本人认为比较好，大伙可以去看一下，本人就rest就不多阐述了（怕误导大家，哈哈） + +http://www.ruanyifeng.com/blog/2011/09/restful.html + +http://www.jianshu.com/p/65ab865a5e9f + + + +### RMI + +SOA思想提出以后，就有很多基于在这个模型上的产物，很多适用于分布式的产物，同时也是越来越庞大系统的产物。Java RMI （Remote Method Invocation 远程方法调用）是用Java在JDK1.1中实现的，它大大增强了Java开发分布式应用的能力。而RMI就是开发百分之百纯Java的网络分布式应用系统的核心解决方案，所以如果不是java的系统就不能使用RMI，这也是其缺点之一。RMI全部的宗旨就是尽可能简化远程接口对象的使用，相当于在服务器端暴露服务，通过bind或者rebind方法注册到RMIRegistry中，注册的信息中包含url，以及相应的类。客户端在在注册中心根据url得到远程对象（stub，存根），然后调用stub远程调用方法。 + + + + + +### SOA架构和微服务架构的区别 + +> 首先SOA和微服务架构是一个层面的东西，而对于ESB和微服务网关是一个层面的东西，一个谈到是架构风格和方法，一个谈的是实现工具或组件。 +> +> 1.SOA（Service Oriented Architecture）“面向服务的架构”:他是一种设计方法，其中包含多个服务， 服务之间通过相互依赖最终提供一系列的功能。一个服务通常以独立的形式存在与操作系统进程中。各个服务之间通过网络调用。 +> +> 2.微服务架构:其实和 SOA 架构类似,微服务是在 SOA 上做的升华，微服务架构强调的一个重点是“业务需要彻底的组件化和服务化”，原有的单个业务系统会拆分为多个可以独立开发、设计、运行的小应用。这些小应用之间通过服务完成交互和集成。 +> +> 微服务架构 = 80%的SOA服务架构思想 + 100%的组件化架构思想 + 80%的领域建模思想 + + + +### API网关 + +API网关是一个服务器，是系统的唯一入口。 + +API 网关并不是微服务场景中必须的组件，如下图，不管有没有 API 网关，后端微服务都可以通过 API 很好地支持[客户端](https://www.zhihu.com/search?q=客户端&search_source=Entity&hybrid_search_source=Entity&hybrid_search_extra={"sourceType"%3A"answer"%2C"sourceId"%3A578705309})的访问。 + + + +![img](https://pica.zhimg.com/80/v2-0903a05306217b52effca6ebb80b45ea_1440w.jpg?source=1940ef5c) + +但对于服务数量众多、复杂度比较高、规模比较大的业务来说，引入 API 网关也有一系列的好处： + +- 聚合接口使得服务对调用者透明，客户端与后端的耦合度降低 +- 聚合后台服务，节省流量，提高性能，提升用户体验 +- 提供安全、流控、过滤、缓存、计费、监控等 API 管理功能 + + + +从面向对象设计的角度看，它与外观模式类似。API网关封装了系统内部架构，为每个客户端提供一个定制的API。它可能还具有其它职责，如身份验证、监控、负载均衡、缓存、请求分片与管理、静态响应处理。API网关方式的核心要点是，所有的客户端和消费端都通过统一的网关接入微服务，在网关层处理所有的非业务功能。通常，网关也是提供REST/HTTP的访问API。服务端通过API-GW注册和管理服务。 + +### References: + +- http://www.jianshu.com/p/2c78554a3f36 + +- http://blog.csdn.net/guyuealian/article/details/51992182 \ No newline at end of file diff --git a/docs/distribution/ZooKeeper/.DS_Store b/docs/distribution/ZooKeeper/.DS_Store new file mode 100644 index 0000000000..854e42ceb6 Binary files /dev/null and b/docs/distribution/ZooKeeper/.DS_Store differ diff --git a/docs/soa/zookeeper/Consistency-Protocol.md b/docs/distribution/ZooKeeper/Consistency-Protocol.md similarity index 87% rename from docs/soa/zookeeper/Consistency-Protocol.md rename to docs/distribution/ZooKeeper/Consistency-Protocol.md index 94d01b535e..30de3249fb 100644 --- a/docs/soa/zookeeper/Consistency-Protocol.md +++ b/docs/distribution/ZooKeeper/Consistency-Protocol.md @@ -1,8 +1,41 @@ -## 「分布式一致性协议」从2PC、3PC、Paxos到 ZAB +--- +title: 「分布式一致性协议」从2PC、3PC、Paxos到 ZAB +date: 2022-06-09 +tags: + - zk + - 分布式 +categories: zookeeper +--- + +## CAP 设计一个分布式系统必定会遇到一个问题—— **因为分区容忍性（partition tolerance）的存在，就必定要求我们需要在系统可用性（availability）和数据一致性（consistency）中做出权衡** 。这就是著名的 `CAP` 定理。 -![img](https://tva1.sinaimg.cn/large/007S8ZIlly1gevqnrlzg6j30b60as3ys.jpg) +![](https://img.starfish.ink/zookeeper/cap.jpg) + + + +> 大多数分布式系统都分布在多个子网络。每个子网络就叫做一个区（partition）。分区容错的意思是，区间通信可能失败。比如，一台服务器放在中国，另一台服务器放在美国，这就是两个区，它们之间可能无法通信。 +> +> 一般来说，分区容错无法避免，因此可以认为 CAP 的 P 总是成立。(因为我们前提分布式系统，分布的服务都没法通信了，还玩个啥） +> +> ![img](https://www.wangbase.com/blogimg/asset/201807/bg2018071602.png) +> +> 一致性和可用性，为什么不可能同时成立？答案很简单，因为可能通信失败（即出现分区容错）。 +> +> 如果保证 G2 的一致性，那么 G1 必须在写操作时，锁定 G2 的读操作和写操作。只有数据同步后，才能重新开放读写。锁定期间，G2 不能读写，没有可用性不。 +> +> 如果保证 G2 的可用性，那么势必不能锁定 G2，所以一致性不成立。 + + + +### 与 BASE 的关系 + +BASE 是 Basically Available（基本可用）、Soft state（软状态）和 Eventually consistent（最终一致性）三个短语的简写。 + +BASE是对CAP中一致性和可用性权衡的结果，其来源于对大规模互联网系统分布式实践的结论，是基于CAP定理逐步演化而来的，其核心思想是即使无法做到强一致性（Strong consistency），但每个应用都可以根据自身的业务特点，采用适当的方式来使系统达到最终一致性（Eventual consistency）。 + + ## 一致性模型 @@ -28,7 +61,7 @@ 分布式事务处理的关键是必须有一种方法可以知道事务在任何地方所做的所有动作，提交或回滚事务的决定必须产生统一的结果（全部提交或全部回滚） -在分布式系统中，各个节点之间在物理上相互独立，通过网络进行沟通和协调。由于存在事务机制，可以保证每个独立节点上的数据操作可以满足ACID。但是，相互独立的节点之间无法准确的知道其他节点中的事务执行情况。所以从理论上讲，两台机器理论上无法达到一致的状态。如果想让分布式部署的多台机器中的数据保持一致性，那么就要保证在所有节点的数据写操作，要不全部都执行，要么全部的都不执行。但是，一台机器在执行本地事务的时候无法知道其他机器中的本地事务的执行结果。所以他也就不知道本次事务到底应该commit还是 roolback。所以，常规的解决办法就是引入一个“协调者”的组件来统一调度所有分布式节点的执行。 +在分布式系统中，各个节点之间在物理上相互独立，通过网络进行沟通和协调。由于存在事务机制，可以保证每个独立节点上的数据操作可以满足 ACID。但是，相互独立的节点之间无法准确的知道其他节点中的事务执行情况。所以从理论上讲，两台机器理论上无法达到一致的状态。如果想让分布式部署的多台机器中的数据保持一致性，那么就要保证在所有节点的数据写操作，要不全部都执行，要么全部的都不执行。但是，一台机器在执行本地事务的时候无法知道其他机器中的本地事务的执行结果。所以他也就不知道本次事务到底应该 commit 还是 roolback。所以，常规的解决办法就是引入一个“协调者”的组件来统一调度所有分布式节点的执行。 ### XA规范 @@ -61,9 +94,9 @@ XA 就是 X/Open DTP 定义的交易中间件与数据库之间的接口规范 #### 阶段二：执行事务提交 -协调者根据各参与者的反馈情况决定最终是否可以提交事务，如果反馈都是Yes，发送提交`commit`请求，参与者提交成功后返回 `Ack` 消息，协调者接收后就完成了。如果反馈是No 或者超时未反馈，发送 `Rollback` 请求，利用阶段一记录表的 `Undo` 信息执行回滚，并反馈给协调者`Ack` ，中断消息 +协调者根据各参与者的反馈情况决定最终是否可以提交事务，如果反馈都是Yes，发送提交 `commit` 请求，参与者提交成功后返回 `Ack` 消息，协调者接收后就完成了。如果反馈是 No 或者超时未反馈，发送 `Rollback` 请求，利用阶段一记录表的 `Undo` 信息执行回滚，并反馈给协调者 `Ack` ，中断消息 -![img](https://tva1.sinaimg.cn/large/00831rSTly1gclosfvncqj30hs09j0td.jpg) +![2PC](https://tva1.sinaimg.cn/large/e6c9d24ely1h2sj2ouelxj20q40e0jrz.jpg) #### 优缺点 @@ -93,7 +126,7 @@ XA 就是 X/Open DTP 定义的交易中间件与数据库之间的接口规范 这个阶段其实和 `2PC` 的第二阶段差不多，如果协调者收到了所有参与者在 `PreCommit` 阶段的 YES 响应，那么协调者将会给所有参与者发送 `DoCommit` 请求，**参与者收到 DoCommit 请求后则会进行事务的提交工作**，完成后则会给协调者返回响应，协调者收到所有参与者返回的事务提交成功的响应之后则完成事务。若协调者在 `PreCommit` 阶段 **收到了任何一个 NO 或者在一定时间内没有收到所有参与者的响应** ，那么就会进行中断请求的发送，参与者收到中断请求后则会 **通过上面记录的回滚日志** 来进行事务的回滚操作，并向协调者反馈回滚状况，协调者收到参与者返回的消息后，中断事务。 -![img](https://tva1.sinaimg.cn/large/00831rSTly1gclot2rul3j30j60cpgmo.jpg) +![3PC](https://img.starfish.ink/zookeeper/3pc.png) #### 优缺点 @@ -129,7 +162,7 @@ XA 就是 X/Open DTP 定义的交易中间件与数据库之间的接口规范 2. 如果 Acceptor 收到一个针对编号为N的提案的Accept请求，只要该 Acceptor 没有对编号大于 N 的 Prepare 请求做出过响应，它就通过该提案。如果N小于 Acceptor 以及响应的 prepare 请求，则拒绝，不回应或回复error（当proposer没有收到过半的回应，那么他会重新进入第一阶段，递增提案号，重新提出prepare请求） 3. 最后是 Learner 获取通过的提案（有多种方式） -![img](https://tva1.sinaimg.cn/large/00831rSTly1gcloyv70qsj30sg0lc0ve.jpg) +![Paxos](https://img.starfish.ink/zookeeper/paxos.png) #### `paxos` 算法的死循环问题 @@ -167,16 +200,16 @@ ZAB（Zookeeper Atomic Broadcast） 协议是为分布式协调服务 Zookeeper #### 消息广播模式 -![ZAB广播](https://tva1.sinaimg.cn/large/007S8ZIlly1gevsrfdpizj30h50al74m.jpg) +![ZAB](https://img.starfish.ink/zookeeper/zab.png) 1. Leader从客户端收到一个事务请求（如果是集群中其他机器接收到客户端的事务请求，会直接转发给 Leader 服务器） 2. Leader 服务器生成一个对应的事务 Proposal，并为这个事务生成一个全局递增的唯一的ZXID（通过其 ZXID 来进行排序保证顺序性） 3. Leader 将这个事务发送给所有的 Follows 节点 4. Follower 节点将收到的事务请求加入到历史队列（Leader 会为每个 Follower 分配一个单独的队列先进先出，顺序保证消息的因果关系）中，并发送 ack 给 Leader -5. 当 Leader 收到超过半数 Follower 的 ack 消息，Leader会广播一个 commit 消息 +5. 当 Leader 收到超过半数 Follower 的 ack 消息，Leader 会广播一个 commit 消息 6. 当 Follower 收到 commit 请求时，会判断该事务的 ZXID 是不是比历史队列中的任何事务的 ZXID 都小，如果是则提交，如果不是则等待比它更小的事务的 commit -![zab commit流程](http://file.sunwaiting.com/zab_commit_1.png) +![zab-commit](https://img.starfish.ink/zookeeper/zab-commit.png) #### 崩溃恢复模式 @@ -221,7 +254,7 @@ ZAB 的原子广播协议在正常情况下运行良好，但天有不测风云 ZXID 是一个 64 位的数字，其中低 32 位可看作是计数器，Leader 服务器每产生一个新的事务 Proposal 的时候，都会该计数器进行加 1 操作。而高 32 位表示 Leader 周期 epoch 的编号，每当选举一个新的 Leader 服务器，就会从该服务器本地的事务日志中最大 Proposal 的 ZXID 中解析出对应的 epoch 值，然后对其加 1 操作，这个值就作为新的 epoch 值，并将低 32 位初始化为 0 来开始生成新的 ZXID。 -![image.png](https://tva1.sinaimg.cn/large/007S8ZIlly1gevrq7ao81j30ej073jrg.jpg) +![](https://tva1.sinaimg.cn/large/007S8ZIlly1gevrq7ao81j30ej073jrg.jpg) 基于这样的策略，当一个包含上一个 Leader 周期中尚未提交的事务 Proposal 的服务器启动时，以 Follower 角色加入集群中之后，Leader 服务器会根据自己服务器上最后被提交的 Proposal 来和 Follower 服务器的 Proposal 进行比对，比对结果就是 Leader 会要求 Follower 进行一个回退操作——回退到一个确实已经被集群中过半机器提交的最新的事务 Proposal。 diff --git a/docs/soa/zookeeper/Hello-Zookeeper.md b/docs/distribution/ZooKeeper/Hello-Zookeeper.md similarity index 97% rename from docs/soa/zookeeper/Hello-Zookeeper.md rename to docs/distribution/ZooKeeper/Hello-Zookeeper.md index e64038c2aa..832e14eeb2 100644 --- a/docs/soa/zookeeper/Hello-Zookeeper.md +++ b/docs/distribution/ZooKeeper/Hello-Zookeeper.md @@ -1,20 +1,20 @@ -不懂 ZooKeeper？没关系，这一篇给你讲的明明白白 +# 不懂 ZooKeeper？没关系，这一篇给你讲的明明白白 > 本来想系统回顾下 ZooKeeper的，可是网上没找到一篇合自己胃口的文章，写的差不多的，感觉大部分都是基于《从Paxos到ZooKeeper 分布式一致性原理与实践》写的，所以自己读了一遍，加上项目中的使用，做个整理。加油，奥利给！ -![](https://imgkr.cn-bj.ufileos.com/5143daad-8428-4edd-82e5-18d1778c8e2e.gif) +![img](https://zookeeper.apache.org/images/zookeeper_small.gif) ## 前言 面试常常被要求「熟悉分布式技术」，当年搞 “XXX管理系统” 的时候，我都不知道分布式系统是个啥。**分布式系统是一个硬件或软件组件分布在不同的网络计算机中上，彼此之间仅仅通过消息传递进行通信和协调的系统**。 -计算机系统从集中式到分布式的变革伴随着包括**分布式网络**、**分布式事务**、**分布式数据一致性**等在内的一系列问题和挑战，同时也催生了一大批诸如`ACID`、`CAP`和 `BASE` 等经典理论的快速发展。 +计算机系统从集中式到分布式的变革伴随着包括**分布式网络**、**分布式事务**、**分布式数据一致性**等在内的一系列问题和挑战，同时也催生了一大批诸如`ACID`、`CAP `和 `BASE` 等经典理论的快速发展。 为了解决分布式一致性问题，涌现出了一大批经典的一致性协议和算法，最为著名的就是二阶段提交协议（2PC），三阶段提交协议（3PC）和`Paxos`算法。`Zookeeper`的一致性是通过基于 `Paxos` 算法的 `ZAB` 协议完成的。一致性协议之前的文章也有介绍：[「走进分布式一致性协议」从2PC、3PC、Paxos 到 ZAB](https://mp.weixin.qq.com/s/1rcUGpj7M0bJvdiSLRkFrQ)，这里就不再说了。 -## 1. 概述 +## 一. 概述 ### 1.1 定义 @@ -40,7 +40,7 @@ ZooKeeper 的目标是将这些不同服务的精华提炼为一个非常简单 ZooKeeper 从设计模式角度来理解：就是一个基于**观察者模式**设计的分布式服务管理框架，它负责存储和管理大家都关心的数据，然后接受观察者的注册，一旦这些数据的状态发生变化，ZK 就将负责通知已经在 ZK 上注册的那些观察者做出相应的反应，从而实现集群中类似 Master/Slave 管理模式。 -![](https://imgkr.cn-bj.ufileos.com/2fea0e81-7cc9-4d9d-b70e-cc34c4932124.png) +![](https://img.starfish.ink/zookeeper/zk-work.png) ### 1.4 特性 @@ -122,9 +122,9 @@ Zookeeper 数据模型的结构与 Unix 文件系统的结构相似，整体上 2. 交由 ZooKeeper 实现的方式 - 可将节点信息写入 ZooKeeper 上的一个 Znode - 监听这个 Znode 可获取它的实时状态变化 - - 典型应用：HBase 中 Master 状态监控和选举。（TODO:图应该是注册和Register and watch） + - 典型应用：HBase 中 Master 状态监控和选举。 -![](https://imgkr.cn-bj.ufileos.com/2bca4022-b9ab-4134-b343-919f10e912a8.jpg) +![](https://img.starfish.ink/zookeeper/zk-manage-node.jpg) ##### Master选举 @@ -369,7 +369,7 @@ Zookeeper 将所有数据存储在内存中，数据模型是一棵树（Znode T - 所谓持久节点是指一旦这个 ZNode 被创建了，除非主动进行 ZNode 的移除操作，否则这个 ZNode 将一直保存在 Zookeeper 上。 - 而临时节点就不一样了，它的生命周期和客户端会话绑定，一旦客户端会话失效，那么这个客户端创建的所有临时节点都会被移除。 -另外，ZooKeeper 还允许用户为每个节点添加一个特殊的属性：**SEQUENTIAL。**也被叫做 **顺序结点**，一旦节点被标记上这个属性，那么在这个节点被创建的时候，Zookeeper 会自动在其节点名后面追加上一个整型数字，这个整型数字是一个由父节点维护的自增数字。 +另外，ZooKeeper 还允许用户为每个节点添加一个特殊的属性：**SEQUENTIAL**。也被叫做 **顺序结点**，一旦节点被标记上这个属性，那么在这个节点被创建的时候，Zookeeper 会自动在其节点名后面追加上一个整型数字，这个整型数字是一个由父节点维护的自增数字。 @@ -507,7 +507,7 @@ Zookeeper 中的监视是轻量级的，因此容易设置、维护和分发。 -> 也不知道有木有人对下一篇的实战环节感兴趣~~~~~ +> 文章持续更新，可以微信搜「 **JavaKeeper** 」第一时间阅读，无套路领取 500+ 本电子书和 30+ 视频教学和源码，本文 **GitHub** [github.com/JavaKeeper](https://github.com/Jstarfish/JavaKeeper) 已经收录，Javaer 开发、面试必备技能兵器谱，有你想要的。 diff --git a/docs/distribution/ZooKeeper/Zookeeper-Lock.md b/docs/distribution/ZooKeeper/Zookeeper-Lock.md new file mode 100644 index 0000000000..52578bbcc6 --- /dev/null +++ b/docs/distribution/ZooKeeper/Zookeeper-Lock.md @@ -0,0 +1,27 @@ +- ## 五、Zookeeper 的分布式锁 + + Zookeeper防重策略 + + 利用ZK确实是一个不错的方案，流程如下： + + ![这里写图片描述](data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAASABIAAD/4QBMRXhpZgAATU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA6ABAAMAAAABAAEAAKACAAQAAAABAAABrqADAAQAAAABAAABQwAAAAD/7QA4UGhvdG9zaG9wIDMuMAA4QklNBAQAAAAAAAA4QklNBCUAAAAAABDUHYzZjwCyBOmACZjs+EJ+/8AAEQgBQwGuAwERAAIRAQMRAf/EAB8AAAEFAQEBAQEBAAAAAAAAAAABAgMEBQYHCAkKC//EALUQAAIBAwMCBAMFBQQEAAABfQECAwAEEQUSITFBBhNRYQcicRQygZGhCCNCscEVUtHwJDNicoIJChYXGBkaJSYnKCkqNDU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6g4SFhoeIiYqSk5SVlpeYmZqio6Slpqeoqaqys7S1tre4ubrCw8TFxsfIycrS09TV1tfY2drh4uPk5ebn6Onq8fLz9PX29/j5+v/EAB8BAAMBAQEBAQEBAQEAAAAAAAABAgMEBQYHCAkKC//EALURAAIBAgQEAwQHBQQEAAECdwABAgMRBAUhMQYSQVEHYXETIjKBCBRCkaGxwQkjM1LwFWJy0QoWJDThJfEXGBkaJicoKSo1Njc4OTpDREVGR0hJSlNUVVZXWFlaY2RlZmdoaWpzdHV2d3h5eoKDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uLj5OXm5+jp6vLz9PX29/j5+v/bAEMAAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAf/bAEMBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAf/dAAQANv/aAAwDAQACEQMRAD8A/v4oAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgDhfGvjjwp8PvD154r8a6zY6DoOnR+Zc399P5cMa9BlmCkk9Aqgk9Tj+LqyvJsXn+IhRwdCVarK0uVWSUU7OU56KMenm9nJtI4M2zrB5Fh3UxFXlTVleyba1slzS5rd1p3vdcv5CfE/8A4K42Nlqclj8Kvh5/bGn20wSTWvE072UE6eWC3k2sREu0SFlBlnilwMtApBRv3TJfA3GZlRj7Sc51GlpGKjCOzVleUnoknd200S05fxPN/G3CZbVlaMORSdru+n/A9Hay2taXnHhn/grv44F9ajxR8NNAvdLL/wClPpt3cW98I8HlS8lxEGBwNrRDpnzOCr/UZl9HinOjUq4arUk0vhvzpPe65lJ7pLVvT0jzeDgvH6SrQp4mlBczSbsob9dOVf101jL9W/2e/wBqj4X/ALR2iSXfgm/EOs2sSy6l4Zv3RNQswfl8zYkkiSRFlYCSKR1ZhgOGWRV/n3inhDN+FKsoeycoJtKfK11sk9Ha+19bt3tE/c+G+Kcq4pw0ZqpTUpJNwur2393Vtu+mqk+1tD6eVMcn/P6nP5D8a+Up02nfpv29NLv8/uPqZSvotvz/AAVvvfyuKy7se3tn/wBmX/Ppj5umMuW/n52/9tl/Xe/u5Sjzf1/wV27P5DqkoKACgAoAKACgD//Q/v4oAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgD+Z7/gpF+0XqnxK+Ll98NdMupYvBfw9uPsklrCUYXevQ+csl6ZFd1dYxK9qmAoCpLIgKzuG/r/wT4Lw/9l1MZUpxc1Hmk3G7ul5xUtNLJ6929j+RvGnjHELMqWDpTlGMmopKVrXtp0/m7fdc+Obb9nz493lpbX1p8DfjHeWd1bLeWl3a/DTxndWtzat9wh00JgsjBgVUF5D2U4Bb9EwPiFkeSZrXwdTMMup+wqOlONTGYWDjJXTjLmqJRkmleMrtbPZs/NMw4Lx2cZXSxap15e1ippxhNpq6u1bdLya17XXN57e+DvEVjp3iHVb/AE+XTovCviHTfDevR6iTaahZa7qsetSxWEli484SBfD2q+arIGiazkEvlvhV+toZ79S9lCdWFVYijKtS5WqkatCPsk6sJRlyyhfEUbNSs1Ui4uSvy8GIyWjj7yoLlcZJS3TUt1vb+V+lvtaOXo/g3xN8Qf2b/itZ6xayXGg+KvCmoGHVtKhmsiWVXC6jpd+UeaPkoYZVjaUxyLvikWaON08DMsryLiyk5VqNKtRxNL442aqU60F70JpO6lGV4zi5Jpprod+AzLOeGmlh69SHI9lJpqz06p3VumzXSzcf63PhZ4+0v4qfD3wl4/0k4sfFOjafrEWeMeYFYg8YHzqSOfu4Bzg7f89KlPkbVt3p5X1762tbr3bWnN/f1Or7Tyt+PTa2n3+SWrZ6GzY47/5+oOR2P5DOV4qlRp6beWnbrZ/l9x0xjfV7fn+Kt62f4D63ICgAoAKACgAoA//R/v2C4OScn16f1OfyH45rC7b82A4jPFT7Ft/y/j/XcCt5i+o/L/7XT9jH/oB/8q0yvZVv+fsf6/7iB5i+o/L/AO10exj/ANAP/lWmHsq3/P2P9f8AcQPMX1H5f/a6PYx/6Af/ACrTD2Vb/n7H+v8AuIHmL6j8v/tdHsY/9AP/AJVph7Kt/wA/Y/1/3EDzF9R+X/2uj2Mf+gH/AMq0w9lW/wCfsf6/7iB5i+o/L/7XR7GP/QD/AOVaYeyrf8/Y/wBf9xA8xfUfl/8Aa6PYx/6Af/KtMPZVv+fsf6/7iB5i+o/L/wC10exj/wBAP/lWmHsq3/P2P9f9xA8xfUfl/wDa6PYx/wCgH/yrTD2Vb/n7H+v+4geYvqPy/wDtdHsY/wDQD/5Vph7Kt/z9j/X/AHEGqq4VJTvbkD5Dx9cY+uPpjPK1linha1ampU5Sa1ulNdvJX2tqtPMzw1PEUqNp1E2vTTr6Py06dQkYRKTMRs7KF69uuBjHufTOeKMbVwVCinUhZWVkozutV6W7a82z2skGHp4qVR8001d2sl12fn1auW67SgoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKAKl2fLtZj/sn+nuc/mv4UAfxhfFUzL8VPiB9tj88/8ACb+JwTj/AKjV2Rjg9FOMbvUYPJr/AEP4VxtfE4GahG/u6LpbX02SfT7rH8CcW154fGU3NW1T16f187bLTWP3Dpw+FHhK18O/DzUfh58UdL8WW3wx/aI+IOteE2+KvhFtT8Kw+OfhlJ/xItXb/hTwA8TPoHgmLxQPDrI6eDU1tx46h8YGWPwin5NmqzzE4qtjqWY4GWGrYnCYehXWVVFh8RGjjLU54eUszrupHEVq7wsar5VXcI1sF7KE6OKn9XlOY4d4OlBwqcypyck68U1KUYq7XsI2UVFycddHZzVpxlu/syeMILTwSukWl/rvg3w14l+Luk6L4P8AAnjbXl1Lw9450rWtY8ErrTaDrviHPh3Xdf8ACXhzSvEviXxkB4TPg3/hM/Evh9UwH8J+DK341oYvHYmCqYelicxhgFOrjKOETrYeC+sqNT2UOfFUcNVqSxFPCUI4uriKr9vCkq3ssZiI9fBFTB5ZF3rOFKU3aEpvlb9261tGUko03N8sF7sW7+4jyH9qu50W/wDBvw+l0bwlpXg+ysvGvxS0LSdB074p/D/44/Yth0LW9aK+NfBGPEcv/CWa/wCNtZ8UP4c+Jni3xb4qnkDTRL4LWXano+F2HxmGxGOVV4qpNRwc5vEZdmOXLVYiinGGOrOjVnKOHTqywWEwtJySc/aSlFUlx7j8Di40lTlT15tIVaVXezs/Z3lGyeiqylPZaKNj9tP+Cdbaif2YvBiaj99VuFtQSc/ZDcybc8cEMH4OPrkgt/MnH9aNbMI1E07wUXbylddX3293yvsf0fwFh5UMFNO62a0/D8tdfwbj95g4A9gCf8/hXxMfdppvov677X+fl9n7jq0u/wCe35C1YgoAKACgAoAKAP/S/vwR+4/Ef/XwPzx+eK56Sd97pap233v+Pr+KcdZ2t+X9fLz/AFj5D8f9d1Xw38HfiLreiX9zpmq6b4V1W707ULPBurO8jtJJIZoR13pIEII6divDV62DpKdRRe9padPd279L7273b0PNxNRq2tlpZdW2k2tl01u/RJXP5hP+G4f2svu/8Lz8Y7cY2ebo+cf3cfYcZx2xn+Hb/DX9w/8AEI/Dr/oSvp/zE1f7v+b+7yZ/Fn/EUOPv+f8AUt/18iH/AA3D+1l93/hefjHbjGzzdHzj+7j7DjOO2M/w7f4aP+IR+HX/AEJX0/5iav8Ad/zf3eTD/iKHH3/P+pb/AK+RD/huH9rL7v8AwvPxjtxjZ5uj5x/dx9hxnHbGf4dv8NH/ABCPw6/6Er6f8xNX+7/m/u8mH/EUOPv+f9S3/XyIf8Nw/tZfd/4Xn4x24xs83R84/u4+w4zjtjP8O3+Gj/iEfh1/0JX0/wCYmr/d/wA393kw/wCIocff8/6lv+vkQ/4bh/ay+7/wvPxjtxjZ5uj5x/dx9hxnHbGf4dv8NH/EI/Dr/oSvp/zE1f7v+b+7yYf8RQ4+/wCf9S3/AF8iH/DcP7WX3f8AhefjHbjGzzdHzj+7j7DjOO2M/wAO3+Gj/iEfh1/0JX0/5iav93/N/d5MP+Iocff8/wCpb/r5EP8AhuH9rL7v/C8/GO3GNnm6PnH93H2HGcdsZ/h2/wANH/EI/Dr/AKEr6f8AMTV/u/5v7vJh/wARQ4+/5/1Lf9fIh/w3D+1l93/hefjHbjGzzdHzj+7j7DjOO2M/w7f4aP8AiEfh1/0JX0/5iav93/N/d5MP+Iocff8AP+pb/r5EP+G4f2svu/8AC8/GO3GNnm6PnH93H2HGcdsZ/h2/w0f8Qj8Ov+hK+n/MTV/u/wCb+7yYf8RQ4+/5/wBS3/XyIf8ADcP7WX3f+F5+MduMbPN0fOP7uPsOM47Yz/Dt/ho/4hH4df8AQlfT/mJq/wB3/N/d5MP+Iocff8/6lv8Ar5EP+G4f2ssbP+F5+MtmMeX52j52/wBzH2HGcdsZ7bf4ateE3h97ZS+qUWr3/h6N731j37x/zON+KfH8qPKsRV10vzR7qy3/AB09Okv0R/4JvftB/Gb4q/FDxRo/xK+IOveLdPtPDwksrPWbgFFfOTJtOwb+SQwIbcc8YBr8S8XeF+DMmwjp5dh6FKrzQslBxg05xU3aMYxcuX+aOqjpZ6S/aPCniPjbOa18VVlVi9WpuEl8Lt8XNaz10Wi6u5+6Ffz0f0IFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQBHIvmIQO447/4fy/AY+UA/mT/AOCi/wCz7q3w4+K998Q9OtQ3hPxtcm6eeAAfZr98mVWjTYF81g0y4IV97ooIiZa/sTwU45o5jh4Uqj5koKF3dt2SV3d3u+/vN9e5/JPjJwzDLq6nFKO+2nZ2t81ta3Xflj+c9fveIp4DFVuZ043dtbX7eS369/LSJ+MYHMJ0KXKm7dbv/Jt+Wy+YVvKhl1aDlKlFtf3Vr+iutrct100ucc3j8PJRjUkr22fVdO/Xtp56HoHwt+HWv/FbxrofgrQLWaS8v7nF20XK2VjvVZL1+SCse9S3QklY0y7otfF5vxFgckcuWEYQheTa0UYpXbejtZLq/PSzPoMtyPG5sk51JSb6ttt/n6/D1W+iP66vgx4Esvhv8OfCvgzT8/2foekabZWn+6qtn053bQefbjIr/P7iDEvFYpNvVScXe+8beiaTemt+ra15f7zyDDrDYVqKtda26q1+3rtzarTax60T9/8AAevqD2/w+pwa8uv7tFW/y00f96+mn+fwx9Wm71JeTS+5X8uv9O7ZJVjCgAoAKACgAoA//9P+/E9B9P6mssP/AAvl+sSnvP5/+lI8J/ab/wCSDfFf/sTtU/ktenkH++UP+vlT/wBIqHj5l8P/AID+aP49K/06P87QoAKACgAoAKACgAoAKACgAqcD/EXz/wDbSa38H+v5kfqh/wAEnP8AktHi7/sW9Mr+bfHr+HH5/wDpUT+gPBP+I/67n9HFfyWf1UFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAeX/ABE+HPhz4jeGr/w34k0q01nTbuAi4tbu3VvMHYrwOTyNmd2R3NevkeNlk+IhVjb41du3K7b3W6S0d1Ja9Xqzhzmh/auGlh1s7qKTvvZrVtPtbe/92zZ+M/xR/wCCWVq2rS3Pw88TTaRaO246dqEH2wW6kfwzJ5ZwTnrESO5AIVP6I4e8YqWAoU6bkk4qzs7peez/ABs+reh/Puf+EVbH1pVFHRttab3e27a+/wBbHi/wz/4Jw6z4ztrbXb7x35Ggtq+t6W9vFprm+/4kWvavoMpw8sSqWn0ppOd+1JNhYsolf6LNfF6lKjOMpJXi9/RaXWl36PbV62j87l3hNUjVi4xbUXuul7P+9201+77P7Afs8fsneA/ghpixaBpy/wBoTKF1HVbgf6bdAZIUlQiCMMWZY44kTcxbazHNfz7xPxVHNq0pRa5W9He+vW7vp8lLa/aB+58NcMzymlCE4tJK17dvu8+/4WPsuKIRRiMdBkfn/n2x74zXwB9+S0AFABQAUAFABQAUAf/U/vxPQfT+prLD/wAL5frEp7z+f/pSPCf2m/8Akg3xX/7E7VP5LXp5B/vlD/r5U/8ASKh4+ZfD/wCA/mj+PSv9Oj/O0KACgAoAKACgAoAKACgAoAKnA/xF8/8A20mt/B/r+ZH6of8ABJz/AJLR4u/7FvTK/m3x6/hx+f8A6VE/oDwT/iP+u5/RxX8ln9VBQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFAEO0hAM9SOOnbAPUnn6j8fvVniZPSK3en3fd32f3raM4ePLFX6Jvb0/rp+svK/ip8Xfhz8GtEtdf+JeuxeH9FudQj02C/n07Vb+3N9KjyRQvHpVnfyxlo4pHV5Y/L2xu7EAHb6eVZLjMfzRw0HWnBJte0hHST0S55009Ek5c1la2n2vHzbiHD5W7T5ldtNtOWq7uy16qyd1fyPmj9k39oX4MeMdKtvh54a8YaVqvi6XxD8T9bj0Wz03UoXOkT/EPXtWttQaWbSrS2VBpuuaQshaVHSa4+zPGZYnSL7DjHhjMcuxGInOlKGFcKTpzlVhJr9zSdSHLGpzR5avPd8ln8UXLmPjuDeKsNmahC79prFrlW93aSb7xaW/2dbe6fdRYqDkkj6++cj5Tjr6c+p/i/NcNTk5y522ot9lpsl1euvd6dLqUf0mvVUIppa7af102+fkT11lBQAUAFABQAUAFABQB//V/vxPQfT+prLD/wAL5frEp7z+f/pSPCf2m/8Akg3xX/7E7VP5LXp5B/vlD/r5U/8ASKh4+ZfD/wCA/mj+PSv9Oj/O0KACsov2NN+Sf6ffou6fmmh119YmrK17dXureT/FP5X937P+B9lb694S8PaVENBsdQutS8fWXg+4+LemeD/iD4M1/wCJzaHoWV0DSj8PvE/xMSQ+HdabZ4a/4RbxX4L8YeNf+EYJkTx3/wAUdX4txpiearjZ1FiK3s6FL93g6mIwc6eHqzd5zxE8bh8JOaq4erN14Sw9XC4dOFSSpVHOr+j8LYBQjTblBXld88VON1bePJUfLZq8eWSlbSEmlA9X8IeGvD11+1GliNF8Labe+G/DPhSX4gXPhjQfCWteDI/EkX/CAaVHq2keDfHH7Pvhbw38Kd3iaBj468K+J28IeLvCnix/E1v4J8WnxoZ/B3jH5nB168uFadF/Wan1rG1pYSl9cxeGxTwsY4rEwpyxWFzPFYjMZulSq1KDwssTTxVKWH58FSw0a+Lw/wBJjnTlnsKsHH91GPtJOKnHmcqdNtRnh6SpWckv3sYOElLlqSnyQn1fxTudE1D4U+DdZ8eeLfgHpyeIfHOr3mi6c3wnMulyabJ8S92uaPrmu+Bf2e/CXinQyo3J/wAJT4d8VeEz4zk0LxG6/wDCXp4s3+D/AC+C8lxOE4gxE6WEzmSpUsTavRxdCa9tUw9SVBwWYZqqFWsryp06eNwdenHEVcLXrSoUIe0n7PHOLpZhkdKkqtBtqneDh7yjFx5l7lBuN/dbdOdO8Izi7uSjD6d8EW134ftfGfh4+KfhLPcXXxD+Jd74fOvftf8Aiv4N6hEvh7RtD0bWxrXw1+E3w/tvDcmg+HE8EPrqeFfEj3X/AAgfhKWHwfOsVsLky/E5pVxjzHA4lYXF1oUKGEbUMrpZnOVOFSNWnThjsVKvVr1qlOsqTrYetWbqN4mdT3Kh9ZlP1CviM3gq1Fc0ZKzxDhrKi05SowaUYxceb34xuny+/aPL8X/DnwZ4Ni+DvhPUNa8KeGvFen6FrnxktdT0PTPEPhLxRqF9rGg+Ddf1zXtY0fWl+IkcXh9fFX9jaB4Z8CqPhX4ul8JjQ/D3jceM1XxOZfGH6Hisyzb206MadaC+owmqk6WJWHVT2lJUoO2EhSqew/e1f3eZU6ldYitQq0JwoQrYX88ynh/Lq+XZpOVSLTxNnFOCk488tVLmbTd0taDjC0Z6uThLw39r7UPDeofEazv9A0jSNBvVh8YW/jbSdF17T/EX9nePtK+JPjTR/EUt/rf9geE/EGpXGvtZWnis3HiDw7YagU15QJpoVRLX7Tw7wmLpwU8RGFuam6E6VJ0YVKE6FGrFqn9YxSThKpKi2q04ydO8WotHz3FmPp4iElBvVWak+Zp7O7ShvZSXu/as92z5Tr9ix7vTilrp+i/VvW/3n5VTSVSXrv6r5/1rpsFXgf4i+f8A7abVv4P9fzI/VD/gk5/yWjxd/wBi3plfzb49fw4/P/0qJ/QHgn/Ef9dz+jiv5LP6qCgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoA4+98Z+DtNuZLHUPFHh/T72DBmtLzWNPtbpQRnLxSzpJg8YJTGMdcrSoYXN8faeGwlbkTWkac6i9fdjJre/RfkcdTMMvy1ezrYinTa1tJxTd/JtKTWu2v91NJRrn4h+Asf8jl4V9/+Kg0r+t2B/L2xXd/ZGdS92OExPNtph67f/ppLp0t582xy/wCsGVQ994yjZ/8ATyK/Fzfl08tCT/hYHgX/AKHTwl/4UWif/Jv+c98Hdzf2bi/+hTiv/Bdb+uq/pnZ9bw3/AEMsN0/5fUf7v97z/Lez5j/hYHgX/odPCX/hRaJ/8m/5z3wdx/ZuL/6FOK/8F1v66r+mH1vDf9DLDdP+X1H+7/e8/wAt7PmiPxA8DEfJ418JDH97X9JH55ugT0/xzwarE5fjIPmnlOJnbqqc4vpqk4p9vs306aMzw+b5NNWp4zD27Ksl/wAPey3X3njnx30X4VfG34VeM/hpr/jHwjBH4k0p7C0vJdc0xV0vXiyvoGoL5plHm2erpZ3EcOzfcFPswVhKN3q5JVx2DxFKtHLsSqcPdnRnRqtSptaxuuVxktHF2ai0pNSScDxM7p5FjYqM8TRtdte+lr3vr5/ajzaWtpE+I/8Agm58G/D/AMEvCPivxz491fwvpPxC8VaxeeHTBc63pr3GjaBo+qvZQ2vnrdNHFP4h1uM3bW3W+tbXw26ItxuRv1rxk4pxfEOPoQo5ViaWHw6dSfLTnL21eSskpSpcyVKnezpy5ZOrKNRSlCKj+YeEmUZHk+GlKWIpqbXWbSUVdtb21lbu1ypxSu1L9QP+FgeBOf8AitvCeTjP/E/0kfyuh/L8s1+JvCY+VNSlluKcX0cZ99Ps38/iX4KMv2f+1skVRxWLwvNov4ifTyev3etr+9J/wsDwL/0OnhL/AMKLRP8A5N/znvg7sv7Nxf8A0KcV/wCC639dV/TN/reG/wChlhun/L6j/d/vef5b2fMf8LA8C/8AQ6eEv/Ci0T/5N/znvg7j+zcX/wBCnFf+C639dV/TD63hv+hlhun/AC+o/wB3+95/lvZ80f8Awn/gL/odPCf/AIUGk/8AyXVf2bi/+hTiP/BdT+uq/pmP9rZN/wBBmH/8HL/5L+r9dOY/4T/wF/0OnhP/AMKDSf8A5Lo/s3F/9CnEf+C6n9dV/TD+1sm/6DMP/wCDl/8AJf1frpzdhDPFPFHNDIJoZvusOVYEHkcDjjGCPbjllxPQLFABQAUAf//W/vxPQfT+prLD/wAL5frEp7z+f/pSPCf2m/8Akg3xX/7E7VP5LXp5B/vlD/r5U/8ASKh4+ZfD/wCA/mj+PSv9Oj/O0KACuef7xOPfTv2/w3+77/tXQSi1JrVO/fv5rt/wUem6N8XvH/hiy0e10bV7LTF0HSdR0jw7d2fhnwsmp+GI9V1U63dap4f1h9KbU/D3iC41Ym+k8T6Nfadr6y4EetRQgIvlYng7CZqpe3pOpCpUdWpTlVqqlW5sO8NKnWoxqeyr0JUHaeGrRlhpyUakqbqxjUPocNxDPAWUHyuytJL3o8slK8XpKMk0mpJ8yWl0m0I/xl+IbeJdM8a3mtaZqXi/SbM2dj4p1Pwl4OvdfdT5S/2zqmqSeHJZ9b8XqkaKvjnWjqXjcjOdblYLu5f9ScHgISwVOEo4epOMnR9tWlTgoqmo0KNN1HGhhFGnGH1OiqeE5PaQ9g4Va0ZqOfVa3+0zlzTjd8zSUpNttylJaynzSvzycpP3dVaPLS1r4q+O/EDeKk1fWI72z8ZWGnafr2nzabpraP5OkCT/AIR6LTtHS1j0zSP+EYZ9/hc6PaWD6BtWPQhp8INuuq4cwmT06M6FFJ0otQqNynV9+zqc9ablUqSqySqVp1JylWqfvKvtKlpRzhxJWx8pUZzk1fVbrRaabJLW1rJLRJbF+P4v/EKPU/E2uf2+st34mTxs2rLdaTolzZLf+Pbb+z/Guu6Tp8+m/YvD/iTxFAR/aHijQE0nXWCKV1iKMsj8OYcHZLTrYalDD+yp4d4ZU1RqVaMlDCWdClOpSnTqVaMNU6NRypVIuUakJxlKMunI84zKlmGbS9rJ3pap8r+Km7tcyai+zVmt07oreGfi/wDELwdotn4d8O6/HpejWGpajq9rpcujaFe2J17UBa51+ZNU0qeN/FGmR6bar4f8WF08Q+FijSaBrmkz3FxO/bmvC+TVKMK1TDR9t7OlT9rCVSnUdKlWVdUXOnJN0J1FetRuqWJSVPEU6tNKB4uRcT5hSwGaw9rKyqvR2krznZuzXxJX5ZbxesOVpMwvEPjbxR4sTRU8T6pPrUuhWo0yz1LUVgl1aW1V3ffq2u+Uda124aSR5Bc6xfX9wrSSGOUedIX7Mvy6lhHeEbap2vpG0YxSjF6QWnwx5Vzc0uVtyccJ5hUxS96TbfXZ/Prf5y7XOXrtxDcrR7f8Dp7vT+9917S45qz9f67vv2VvMK6MD/EXz/8AbS638H+v5kfqh/wSc/5LR4u/7FvTK/m3x6/hx+f/AKVE/oDwT/iP+u5/RxX8ln9VBQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAfy/f8ABSvUr6H9qTXxBfXNtjw1pQzaXgGenbr05PKnnBPIr+xPA7C5Fm2TuVehTqzpwUXKUV73Lpe1mr9OvTWV7H8oeNua5lk2aQp06s4ptaJtb8u6tFaX/wA0npL4I/tPVP8AoLal/wCBVfrmWZLw7VzSpSeFp2Uv5F0XXbp/i87aI/Js04mzejldOsq9S9rv3u7X/DeV7B/aeqf9BbUv/Aquz+z8P/0IsP8AdR8v7vr9y2+z5n1vEaf8LmJ6faq/3f73l/w13yn9p6p/0FtS/wDAqj+z8P8A9CLD/dR8v7vr9y2+yfW8Rp/wuYnp9qr/AHf73l/w13yn9p6p/wBBbUv/AAKqv7Pw1f3f7Dw79Y0tb/JPT10veyNP7Z4hw9pLFVdPP/7d97dF+DD+09U/6C2pf+BVL+wMPG9sjw9vNU//AJBW37P5FvifiCek8TVtovi7fN9Px6u4f2nqn/QW1L/wKrPMMthjMVDnyWhK8lZP2flfXlf4LXu9ObXK8zzzC4SfJXqRdr72622u7dP+DvI/tPVP+gtqX/gVXo47IsNh8JTf9hYdWSeio6/h+r/LlWWZrxDicXO+JqfFreV7/c2tvL1tYP7T1T/oLal/4FVw/wBn4f8A6EWH+6j5f3fX7lt9nk+t4jT/AIXMT0+1V/u/3vL/AIa75T+09U/6C2pf+BVH9n4f/oRYf7qPl/d9fuW32T63iNP+FzE9PtVf7v8Ae8v+Gu+U/tPVP+gtqX/gVR/Z+H/6EWH+6j5f3fX7lt9nu/tTiH/oKq/f/wDdPJfcaGm6lqv9qaf/AKdqX/H5pn/L5+fp78fz60f2fh/+hFh/uo+X931+5bfZP7U4h/6Cqv3/AP3TyX3H9lHwym834f8AgY9v+EW8Pf8ApntTz1/2e6j1BIxX+aZ/oaegUAFABQB//9f+/E9B9P6mssP/AAvl+sSnvP5/+lI8J/ab/wCSDfFf/sTtU/ktenkH++UP+vlT/wBIqHj5l8P/AID+aP49K/06P87QoAKACgAoAKACgAoAKACgAqcD/EXz/wDbSa38H+v5kfqh/wAEnP8AktHi7/sW9Mr+bfHr+HH5/wDpUT+gPBP+I/67n9HFfyWf1UFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQB/Lv/AMFKc/8ADUet9cf2DYYznGf9P6dt33en+zj+Cv7S8Bf+RZ/26/zP428af+RhT9f0gfn/AF+y0f8AkYVPX9EflOP/AORZT9P/AG9Fizs57y6t7Gxg+0z3V59is7S05/z+X4tzt48zxuMdGVovZ9NrpL+tV8t5YU8uwazGj7yVpr8Wtr+dnvH10R+22h+C/gz4UvHsLHQ/BnjLxZ/wj3we+H8eg/E/wj4Q+MLaXfaH4P8AHeuDStC1rxF4h+A3hfRp/Esehv4bi8SSeKmtrnxxoKRzQKjEr/LfFdfPMTyVKrxEZQrYupD6pGvl/PRVaMaU6mHvmtfESmourTi/ZU/ZV1Gph4TUYR/prJMDlkcBTSlTvKnTi/auFXXl97llejCmpOybTlOPLeElrzeA/AXR/hReR/Fd9T03wbqWmt8fbG98OS2ZTUtKvPDegf8ACca5HHrvgj+xfFHh/QNC8MeHH19o2jUptgl8zwd4qk8JxeDJfqOJ8bmlRQhTp1VFYPEU5+/UnJznGjGlCNd4rCVXWck+WrJOo+Wf+04Sc+ar87kuV5dRs51YJqcGua2sbtybXs6kXFJbW6r3KiTjHV8dfDTwNL4/+G3hPWfDPgRLLxVo3iaOwn1z4Z+K/CVjZ+JXTQPEGsQeLtb+F/wg/YxgGg+EotA1eKRnPizxV4TmfxRc+OPBA8128FeNw3mGa0cJiZUquZU6lN0a8quGxOHxdWvRlRqSpUMJSzHFcQUfbV3OHs48uDWJlUoKFXli+boznK8tq1Y3eHlvDklGdOMXzJOVX2EMLK0XzPRSkkmpRbdjxHxno3wu8KfDH40aZ4e8OeJbC+utX8DXt1ow+JWk6haeEtUPjHxtoEeja74R1/wE/iLQP+ESfQblpP8AhKvEvi4OssKL42QwO8/2OX1cPRxNKU8Vha+AVNpOWX1Y42snSpctT67Sx6wbdSo3VqOGX0ISUXClQjBwqQ8jGYfN60XbmVS6u1VpunvJv937Pm0jypfvXZpt3uox9TbfbSWngjw9r3w58OQeIT8MfDV74dn+BXwRvlsNT8D6DosXjj/hNBrvw/8AFPiLXMeGtfk+J/gnxL4nACy+H/iP4PPlzRxSr4+Lo5NUr4h1P7QrN1a9blpZjnEVOnXnVqYaNJvF4bCUHRUauGlhaKnGK+q15VaEZKMrwtPPaKh71FJWiualhr3XI25e5Oc9bNTna6U4LRSjKt4Ru/hBbfDG11TxDp/gzUdJit9Q+OFxYyQfEPQvGt/4q0D4/nwFoera54N8DPJ8G9E8CN4a2+G5fDnhpZvF/g/+2/EjeBPBrSmVWWOo8UVsypUIY3GRjQgsBQxU/wCzsRSTlgqeJnCOIxXtc0qVpKnSrValVVKFethqTr1ak1OE+jLq3DFChOcsBBylLnlD/aKcrOVrqMIxoKCbaUYOMoJySjZRcflr9oyPQbnW/DPiCy07wTp3iHxLZeKL/wAX3HgDXvFHiXwz4g16P4m+PtFm8ZaVq/iPxH4vuJ7fxCdGF2Ik1+9SJ1lZooXlLXH6j4fviTB4ZwWaVZwpSpwgsRhsNhp0qcsNhqsKKoU8PhfZRp06kbU6lH2tOUp0525Ixh+c8ZrhrHTusv0bb92dV395pu7c2/eV7p22ev2vnOv0WhjcVTSbi7d7d+nX/h9NNz47EYHC1WvfV313/rZf0i9pv/IUsP8Ar80r+ledj8ZiquvK33/rv+ux6OBwWFpfFNK/9fkutl163l/Z18Lf+SeeCv8AsW9Gx0/6Btv6f04/Sv8ANOr8UvSP5/d/WnU/0YPQK4Kv2vl+gBXQAUAf/9D+/E9B9P6mssP/AAvl+sSnvP5/+lI8J/ab/wCSDfFf/sTtU/ktenkH++UP+vlT/wBIqHj5l8P/AID+aP49K/06P87QoAKACgAoAKACgAoAKACgAqcD/EXz/wDbSa38H+v5kfqh/wAEnP8AktHi7/sW9Mr+bfHr+HH5/wDpUT+gPBP+I/67n9HFfyWf1UFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQB/Lv/AMFKc/8ADUet9cf2DYYznGf9P6dt33en+zj+Cv7S8Bf+RZ/26/zP428af+RhT9f0gfn/AF+y0f8AkYVPX9EflOP/AORZT9P/AG9BXZSxuDxas4J37r7ujtut19+5wZtl2LwuOhJTkrSTvr0t66+n3q9z062+NfxP07RZ9B0rxZqOg2FxZ+CbC8k8Psuj6re2PgAj/hCUk1zTDb6yB4ex/oRF6pGf33m7kK/O5nwrk2NqRqywlGU4e3cHKnFuP1iXNiLX2dZ61GrOV2ndSaPqY5/mWDwcIRrTtaKfvP7Hw/K23a+zesdG1+OfxLsB4j+w6zpdrN4s1Q6r4lvF8J+ETqniO/bTdZ03zNX1ttDfWHItfEfiJEFvdQRH/hINTlG19SmdliuGcoqVKVX6tDnw9CWGpJOcadOjOdGbhCir0VrQo8slTU4KCjFqLaly/wBt5hTTSqSV2m9dW1G2979ZaN9b6te9H4j+PPxL8Uabe6Pf6poNlZ6gNTGrReGfA/gXwJLrJ1q40W/1STWf+EK8OeHV1WW4m0HQt0l+lw+dMsjkyJI75ZXwlk2Ak5UqdepJyjOLxmNx2PdOUY1afNReOxOJ9hJ061WE5UVCVSE+WpzxUYx0ln2ZV2uacb319nTpUdG4uzVKMVJe7ez0jbS15c3beC/h5+038QvBN+vhX4d+MvF3g7xFrum6ne63B4FN3c69f6TGdF0pP+E5Hh+TxLqOi+HoPNs7fRo9dl8P6K4llGk2l7B50XzeaZp4c4LFt18Zh8PiY0FT9jVzCdKCpzl7RTeCnWjQdRtWjiHQ9uoXpxquEpQPosDg+OsRF2VSpDmTclThK0lpbnUXJL3tY88Yu8W02onZXV9+19B4y1rw1e/C3Ubnx/ef2n4g1m1/4Zw8Ian4013TdX1Vjrur6zrJ+H3/AAkOufD/AMVa+H8NuUVfB5C/8IU+7wIisnh4at4YSp/WHjeXDqMcGqn9u4+OEj7PC+yVOC+vKhTxEaC9qpQUcQqkVi/aOvF1y8ZhePKTjTUZqS9+yw9L2jU57y9zmnFzaiuZ8uvJGMU+WWJ4t1D9r6w+H+uaF4s8J/FTwf8ACiS3u7XXNBl+GfiLwd8P9DXWfF6+O2MVhHoWm6DoRbxPKtutzFDZvbzOPD6SxQ7dEXoyjD+HeLxtCpTzPAVcxkoyo1FUhVx9aMMM6XN7V1ZYmvy4RtSlKUr026knrzlZlX4uw9OaWXyjBNxcfdjCMpNXTWkINuO1vsW6Wj49c6h43+MOvaPpvl6TfX+m6RqNlpGk6bpPhHwRpVjpOkLrHiXWBp2kaDZ+HNDgjhR9b1y7+z2YuZQ15JHFLsgt6+xhT4Uyqny0cwqqLlFt1J4mvOcmoQXPVrVJ1ZvljCCcpbJaq7Ufm4/6zY6bc8vg7dL0tFduy0jHS7269t5XPAfwQ+L/AMTYhf8AgD4aeNvEumtPeQjWLLQdSm0D7VAAz2T640SaPbyoGUlLm9gMayI7bEZWfvzjjfI8unGhisbgcLWqQU4UsRiqFGrKDbipxp1ZQlKLkmk0lFyi0m2jly7hfHYhScI1ZQjZSlGEpJfa1auo2V36a3VmzC13wH418A6/pVh458IeJ/B2pXiW11Z6b4n0DVNAvbqy+2sn2sW+rW8E5jLxOgcQ7CyOoLFGFc+X8W5ZmMPbYWth8VS55QdXD1adWmpxtzRc6TnHmV05RvdXTbSfvd+K4ZxeHtGXPCSSupJxavbVxfK9Vaz0380j+xL4W/8AJPPBX/Yt6Njp/wBA239P6cfpX+ddX4pekfz+7+tOp/oGegVwVftfL9ACugAoA//R/v4oA5Pxr4W0/wAa+F9Z8L6tALnTtcszZXduSMbGIY88DqBkkdhjHFAHwZ/w7s+A/wD0T7TP/A1a/U/+IkYjuvuPzn/iHGH7/jIP+HdnwH/6J9pn/gatH/ESMR3X3B/xDjD9/wAZB/w7s+A//RPtM/8AA1aP+IkYjuvuD/iHGH7/AIyD/h3Z8B/+ifaZ/wCBq0f8RIxHdfcH/EOMP3/GQ3/h3d8BDgDwBpmf+vz3x/e9fXP0PWuSv4m8UxaVKSnFvV2hG338r/D7rWlcfDzD1ablJ2fTfX0eiXy1fZauPw5+1t8Nf2ZP2aNY+GOiy/DjT9Rv/E2tvdeJLRrsg2HgVGK67reDgnxA2PMR/mEkeheJGXCtmv1rgKtxdxnTqzoSjNU5SpOfLBqNfli+VrnjeKg4VLp2nGrBwkuWfN+ZcVVMLwnU5JSX39L2+Tvpvo0+mkftnT/+Cfv7Pmq2dpqGn+C7G8tL2ITRTxXd0pntm5Ac+cMtz/DtbOctlVDfnuK8QsVBzjUjGM4Wule2qveLfNeLsnrr0tL4o/eYHg/D1Ule6avs+lt9bKz3d5b+vNc/4d1/Af8A6J/pn/gYv/1v5/lXNQ8VMTS0TXzjf8uVq6tr+W53V/DrD1uuumztbbze6t1+7QP+HdnwH/6J9pn/AIGrWf8AxEjEd19wf8Q4w/f8ZB/w7s+A/wD0T7TP/A1aP+IkYjuvuD/iHGH7/jIQf8E7PgMenw/0z/wL/pvOfwI/GqwfininHl0SfePNf7uX0ev3/aMb4fYW6nGXMr9mrW9Vr07aLZ/Z9e+DX7Jnw4+CviWfxJ4P8OW2i315ZCzu/sd317j8+DyB1xxj5fneIeM6+YxcHFSjN+9daWuns1r8K3Wu65tIR+hyLhnDZfFKT1jpdNvdbaXe3krd9D7Cr4w+lCgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoA/l2/4KVAL+1LrYX7o0DQdufu/6vUM46jGNucdsZxlRX9h+EeNxUsjjRhCTUKSiu93dyfwq92309Wr+7/JHinhMHSzSEpTim5J6vf4baW8vK/Z2fLtfGbSvgH4i1/4k+J4rzwDqOk6p/wnWt3nxF0r4lf2j8Qv+E8HxO1oaJpOj/DRfH7prngZfDx0RU8QD4VqrozeOh4yfb5sHRw283w+Py/D1vr0GsvwEFgpZbOlhI04YOjGq8RioYGrTw2Op1YVWsP9ehS5Y0qCoqco1jwsxpYCrld4yo29pK01UvUldtpRpupGTptNP2nsnZvV7wlhabYfBrSfGPxI8O6/b/CrwZ8Odb8MfYPCVz4N+KT+Lv8AhOQPjH4BHgn/AITI/wDCwvFniTRVA0Ut4zbwt4R8I+LfB/hA+J0/4Q9dqlIWOzqccPVpxzDF4ilmEva0sRQ/s50OXAYqniIYRRwsMLiaa9o/qksbicVl9XEyp/8ACpJwpzO+pgsuV1KpSgvZJxlCcanM3OLi6n71uEuk1FRqQgr+xlzSUcPxF4a+At54ii8MaXo/wp8Nz6H4i+BQutftfiN4v8R+GdbPjjR3/wCFqaRrI/4WIfDf9geFvEJBI8L+LvCK/KpHjQEsJfTweMz14XFYqdfMWqlHMnQo1MHhqdXDqniKqwNWhRjl0sX7aphuWfJicNjOaShfCc/NQq+bVweW8yjH2V1KDcvaSfNeMeaMvf5eVSTs4uD1d5tJKHb+JPDX7LfhrVPBhTw18Mdf/wCEp8SfC/w74k0pPiPq9hY+FdK17WPiu/jjVXHgb9oL4pGJR4e0XwbEYz8WPF/hIosTnwXC5dK+ey7F8XZrhX7TEZjRkliqqqLDYOpiaso08K8NSlHFZNlsIU+Z4i9L6hTq1ZRi/rsIRcD1M4weQ5fJ+zlSntG3tJqEX76lNcteo5P4JJutyJ3XJJNcrfh74V/ZW8X6PaeIdS0XwFomuFhaX3gJPHgNkumjXfHGzxcNW8f/ALQvwqRtdm8N/wDCG+dKfGPixoVC48EjeZIurN8dxRk1OmqE8ZiIyxCnKtGlSdenCMKKjhXSwuSZhGrRqtV6tSrLD4edL3YLEx5oozybBZJmF+epSi7aRlJqLupXlzOtTalHTkh76m90tpdD8ING8Ax/B+0Mmm36+HbL4j6Nrni/xTplnonh/VP+ET0bWtf13WZdd+J+ufDxX16Pwu/h/wAH+I18MeF/+EtJfWPD3ghfGzp8VvB3g3wfPHWY5pjeNsxxdC0KteNFUIVKNTExlz06MKVWNKhieaFOrNYym3VeGk/q9WuoSw+DxFer0ZZgcvp8OZdhpyv7CVRzcZqDjafM03KEk5JOEvdUtJKOkpx5edGp+A5f2h/+E30vw14a0DwDqfwy8Q6vaav4t8bfCVNM+JOn68dY0HQ9YdNc+H0/h1deaVm8N+OfDDeEvF/i2PxToniVZ/8AhNPGqSTTy8NmNXgnHZXWqe1xVbMKNGccJhcylLD1nWo4mdCvFYmtiadNubq06854fBxwGIwrdJYPk+sFXHZdV4kwOKpySp4ejUu6k6T5ounKF4uUYwcko8qSi6ntYvlvU5VH2H4p6NrcM3iu/wDA1zpGu/Ea78d/2v4R8Oxayvh2RPhA3jP4f+OdY8Xt5b/CvxHofh/wx8RvBJ8LgibwkG8HnxF40DP4I8Kjxk3zWWLFxhRp4zB4qjQUJUZ1q0PrEo4qdFUcPhpQisd9ZnjKVaoqMpTrKriZYfAt18bio4eXtZtWy+u5ezr06k9JpQ91OC96pNfw3FU2ldKCShz1HyQg5SxT8W9I13x9pU/gq+Xx6+jeD/Ezy+KfFGheD/jBeaB4X0rwf45g8Zw61408d+H/ABV8SdB0FvEgh8Q+AfFBQ+DvGPg7xD5fjUDzCH9LH8LV3QpKrhqOXQeKnRpRweInl1KpLE16dHAwdLD1qNOtiZ0fYUZqU3L6zCVPCQqQqQ5jKMbgoSk/bKrNpTftIRqy9xOU5XnGUlFe9J62cdZt8vu+NeAvC17rXw08CNd/BfxR8TNVmHj7x/p0fhfwtq9/p9rpfjTxmPBGv+DdXTQfDo8N+BP+Eo8OeC/Gnijwb4oUkeCvGOh+GB4G4ID+9nOb8mKmp4ylg53oYTlxmO+r/vMHShj6VWnF1KdXFyjLF4WGJo+3w8a2HrTjiZ1YunTn8blWX4SNNy1lGPvOUYOSiqnucspJcsb8kuRtSalFcuvMz5++L/heTwh4+8OeHIPCd9oGm6b4c0Oy07W73Sr6w/4TnbEo13xnpVzdyPa6/wCF9e8TR61c+Etb0ow6beeHF0xVLTQTqn2nC2f4iVOpKE6Nd0sRWoVPYS5lQnTndYevC8pUsTSpSpqvSqKE4VHK8YRlCMfJzzKMJUcVqnKMZrmurxl9qPSUG9pLSS+Z/Wn8Lf8Aknngr/sW9Gx0/wCgbb+n9OP0r+B6vxS9I/n939adT+7T0CuCr9r5foAV0AFAH//S/v4oAKAG4+n/AHwf89v85FHPDz/8CX/yPn/w1nyr3u6+4MfT/vg/57f5yKOeHn/4Ev8A5Hz/AOGs+U97uvuDH0/74P8Ant/nIo54ef8A4Ev/AJHz/wCGs+U97uvuDH0/74P+e3+cijnh5/8AgS/+R8/+Gs+U97uvuK/02sR+H5f/AKx0zxmuKUszlUjywp04N+8rxenldtrbvf0vEtp0oNXlJpWW68/6Xl2Pyu/ay/Yj8I/FL4k6L8RvEvj7x02qeMvGHgrwR/ZkE+i2+maLoEfn+fHpKR6BNcLcMwebzr6e5gM88kzxO80pf908P/EjOeFcLVoYKMHHDYatXgmlyqpeU1Fq6bi5uTdpLS6Shoo/iXGfAGJ4nruaqO0qnLrpaOmrt6NXS15b63P0B+DPwzh+Dnw68N/Dm18Q6r4lsPClkNN0vUdbj02K/GnQn/QrKU6XZ2VsVtIQsELC2UiJUTCoiRr+SY7GU69SVTl5bxUWr3ba0vtZbx2f962tz9ZwGXVMNFc0r3lzdLaaaa+V91+svVi+Rlf8fy6Dn059RnqvFCFOTu46vVfdfXTRpNf/AGu56FTnitLX8/03X381vPVxdj6f98H/AD2/zkU+eHn/AOBL/wCR8/8AhrPlfvd19wY+n/fB/wA9v85FHPDz/wDAl/8AI+f/AA1nynvd19w04x1259OSfbP+c9B0NTGNKjdqKsvtbfh73ktP/AX9kalOyvZ/f8r30+9/MYGXoHP5f0Bb+f5/w261N3ulpo7/AIbw19Vbv1vJKjNbTfpaP/A/P7iekUFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQB/Lv/AMFKMj9qLWhzgaDYY64z/p/TqN33eh/u4z8lf2n4B4/B18r9nKClaEY372Tu9UrO3e/e9z+LPHHB4yhmcGpS+JPfa9trWX5/mj8/6/a6CwKzCcvZxTb2XXTvp27b9j84VLG1Mtp3m7pLq0/vTl5dfv8Aslc1PC4H+SK67LTt038tFvqxTxWPt/Elbd/5fF/lr10vIr0IYXA2t7ON+unfTt83ppZ6y2OCpicf1qy7vX7lu/y7as9s+H3wM8S/EXwd4j8WaNPYg2B1O08OaFca34NtdR8WXvh/SF8Q+NI9M0vXfF/h7xHdHwt4VdNeuF8PeHfE0t5bma1EVnJFJKnxHEnGWR8MYmjhlhnzVLSnKOHxcqdONSTp0nKtSwtXDwlUqxceWviKTUU5+9pE+nyfhbOeJE5OtK0XrzVIJvvaDqQlJJa+5GVutr3IPi58FPEPwfv7f7Xq/h7xLok19e6Jba9oHiPwjrjP4l0jTdEm8R6RqGkeGPEvia40K80S61m2tp7PVZ7a7dmikhEiuQunDPF2R8USqQWHnzU1CUva4XE0YtTdRJwnXoUo1XenLm9lKXJeLlZSpuc5vwvnXDaT9tK0rpNVIN3iot3jGUnFLmXxLXWzbjJR8ar6TF4HAVclwGYulHnrTd3bV2qJavfr3VvPY4MvxWOqYvM8G6suXD000vWm3t5W8/PvHp7jwR4qsdD1DxTfaRJa6Zpd34VsrqedhbMj+MtI1vXvDrCw37imq6VoGqXlvMPkaK08wEb4g3LiMwyuGe5dlUXSkq9OvLki1Lm+rezhW0Umn7GrONOru4TlySUXdHjcM0MwzDBZpjJVJ3w9WEVLXS85pLm0+JRbWmy66s5itqeGy/CNONKNv8O23m+/lbtuaPF5jW1dWSWnW/y8vu3W8teUr0aeMy/ROnH/AMB2879e+0RrF5jRs1Wlb1tb5+vr5d4lcf1bAT3pxvpra/VdrfP3Y6d7XM6eKx0JJKpK/q/xTv37vS+20b2m/wDIUsP+vzSv6VPLgqHw00rW7K+2979ddvPW6PSpyxlf4qkpevT/ADau+9vxj/Z18Lf+SeeCv+xb0bHT/oG2/p/Tj9K/zRq/FL0j+f3f1p1P9Gj0CuCr9r5foAV0AFAH/9P+/igAoAKACgAoAKAGA5Xk/wBMfz/l+B/izf7hrXT0/LXXf+6vuTJTU4+T8/6t/Vzxf4waXqN+Phq2m2F1qAsfiv4P1G/a3tvtB0/TYE1VbrU5V2kRwW3mRiWYY8vzF+XJxXtZbXt9YaUbVcJXhZyte/L7t73Stfs1rvfmj4uY4SPue9ZKpF99vlZ2/wC3dvkezIATkNnH+fX37bv1Bry2mt9un9a2v6/fZHtKoqmqle34f16fePZd2Pb2z/7Mv+fTHzc8/f8AdX9bPaytt3fyKHVsAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQB/Lt/wUoJP7UOtHnB0CwI64z/AKf06Dd93oP7uMfJX9seAtC+WPXemnbv/lrbq/la8v4/8WsS/b+jelvS+1rX8l8l9n4Ar9ppYf8A4UZ+vful5f1/d3l+OQxLVFaa9t+3Vv8AP0tpcKfsYmXt/wC5+Pp/wfwCj2MQ9v8A3Px9P+D+B9o/BD4yeC/AfwquNF1PxfceHPFwu/jnd6P9l+GOkeIRjXfg5oWg+Cv+LoBG8SeAGXX9DDI3hQ5Y/u/HuwEvX43x7k2bZjxPSx1CjOrRjTy+GmOrU4RcMZVliJfU1TeGqctCpGp7WrP2s3CNKmoSUZn6bwjmWX4TJK1CdVRnJVHb2cG/g9xKfNzR5paNWaj8X7xu0en+G/xS8IJqngqfVvG3hfwzqA8P/GDxN4i1b4f+H9Y0s+FdS1n4B6D4H0Eg60fCfhgeP28R6J/wk23wsf8AhDm8YDy/A/jQKWDcfEOTZxiMprwjhsYpe1wlCnHF4ih/tFHD5lWxM6lP6m6zp4KdDEuhD6xy4mph6Xs8XhY8rniNuE80y/B5xOrKtTalzSvFStGU6SjZ+03mmm3ytxU3zQnouXa8C/tY+AvCfjfUfE+q6x8Z9TuI9M+GOk3uuy/8JYl/8Sf+EJHiNfEOteNdE0L9oPwwP+EkBm0bQB/wlPiz4tu/hLw6iSeCxC8obz838NcfWw2DhQwOX4aVOrOdSlQxMPZYWLlZfU5VMjqRqc95V6nNhsNONb3VUnTcnLqyfjaEKk5VKs5qUUuaablNxs1z2xEGleMYaznaL5lGMopFW7/bMlstJ8MjRfG/xlGqLrnwel8RaVc3YOnWXhzwRouu6H480vRda/4WEo13QvFrtpHidvDA8J+EGcEOSWcom+V+GdZV5e2weAnhoe2q02+WVSriqrwzp1qtGpgpKlUglXpOtTxNVyXLUdJucaeHjNuNlUS5atS8oqE1slC79xNT96LXLLlcUlK6SduaXwPrU1le61q97YW5t7C61S+ubK1Jz9lsTMxij6gnahVcEDGMLuGzd+1fWMT/ACPp1f8Ad/u+b/4Fny/DZlTwdeOk1f7+i/4bd/K15Z9H1jE/yPp1f93+75v/AIFny/OU8DhoSfvLe34+u/fX1bCvXw8E1p08v/tl/Wt3oi8RiLu1/wAP8Pp+f3F7Tf8AkKWH/X5pX9K4MwpRTfX+l/X/AG75I9HAYh8v62+Xft5fduf2dfC3/knngr/sW9Gx0/6Btv6f04/Sv8y6vxS9I/n939adT/RY9Argq/a+X6AFdABQB//U/v4oAKACgAoAKACgCBc7B7kD9P8AP+RWOK1nBf3f1/r+kZYX4I/N/cj8tf2p/jH468D/ABG0hbP48XvwC0CXxB4Jtbmw8c+Kv2VtPbxJ4Z1/xf4c8P6zrHwr8Fa74A8Z/EP+w4oDr/iDxv4s+JvjLwkvg86NK1j4N8W28twvhP7rhfJqWOjKNTDYjFxjSrzaoUMTOMJUqPtYKpUhVpxdSor06NKjCdWc6kOWcE/e+H4tzyWAklGah70UuaULv37aRevKnrKUnGKUdbNKJ9P/ALMPiLxd4l8CXGpeKvEOpeP5TqbXOm+PbXxl8H/iN4O8V2DuUaLwH41+FXgL4UprvhzwxOHhd/FXwr8I+KofEkWuW8Unimzhtr1fFzejGjpGycoqcoezq0J0nLo6VSU+WWjTcZyjyxpqKUFCU/S4dxUsVLVtpStfmjOMrWd+ZKCdk+azUJczk5Xk5xPqpXOT7Hnnr1/2f/iv0zXz1Bbvd638tUtH3110++59hNe76f1p2/Ht1vGxW5kFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAfzD/8ABR3StR1D9p3XJbLTr29txpenZmhhFyoxbRK2P3nyBSrIcqdm3ZuO1hX9leA+Ptl012jbbe1/P0/mvt5n8n+K+Aiqq0/Xtffl/patWTl8H/8ACJ+I/wDoXte/8EuqY/LbjGOMbf8AGv2ajj3/AGjPtzf8Hyvp922u5+OQy9ewimtet/VW1t2/u+Wt7h/wifiP/oXte/8ABLqmPy24xjjG3/GuH+0337d/7v8Am/u9TJZdtp26/wCD/p36/iH/AAifiP8A6F7Xv/BLqmPy24xjjG3/ABo/tN9+3f8Au/5v7vUFl22nbr/g/wCnfr+If8In4j/6F7Xv/BLqmPy24xjjG3/GuzA53gcwj7adOLlHuvx87JabfO15ZzyTG4OfsVOSTst2/wBY73v/AJWtI/4RPxH/ANC9r3/gl1TH5bcYxxjb/jU4jPMDWm6Tpqy0227d/wA15J2cSqmSY7Br20KkrvVtP7uy/wDSu2lryP8AhE/Ef/Qva9/4JdUx+W3GMcY2/wCNNZ1CrZNLTyT3+Wu2+nmn9nojlkqSWrXnqvLu/wBPTVB/wifiP/oXte/8EuqY/LbjGOMbf8aP7ahSey3T6a2+63zevluJ5ZOqrt3el73fTteOnz+7XmP+ET8R/wDQva9/4JdUx+W3GMcY2/41X9v4T+Vfd6f5/n2IwWSYuv8AFJvXvt/Wnb8bSP8AhE/Ef/Qva9/4JdUx+W3GMcY2/wCNH9v4T+Vfd6f5/n2Hi+HMTFJ8zXf+n303X3XD/hE/Ef8A0L2vf+CXVMfltxjHGNv+NY0Mzkl1t6+nTpt3fbXcWIwC6+V9v8/NdOnkuaxp/hvxEdSsx/wjevfNcWiAf2RqfJk+6P8Aj3xlvcKBnkHDM3Bj8zlffvp6/wBJ9e2l+Y9DAYDbtp0Xk+tn+Pra6P7IPhb/AMk88Ff9i3o2On/QNt/T+nH6V/nZV+KXpH8/u/rTqf6DHoFcFX7Xy/QAroAKAP/V/v4oAKACgAoAKACgArJ/H80B8x/Fr4Z/DvU/Guj+NvFWq/FCz1bxDb+Hfhra23gb4p/Ez4d6fJFaan4g1XTn1W2+HPirwsur+Tc63qg3a02oRRxXDLHEACE97L1UlTlGmqcXSX1jmnGM7yt7kVGXOovd86SaslpzM8LMXHm15teWFr2stL6pa9NPP4pWtH1n4bfD3SPhn4Zg8KaFe67faZb6nr+pwzeJNd1XxLrHneI9c1PxDqAudd127vtb1IjUNVuzHPqt9eXnlGOOS4ZFQpy47GvGVfaShCEuSEX7PZuEUrvdbJJKPKoxSXvWbOnLsu+opq7ereum67Xffq38rR5e92YyTx+v0/n2DH8hXmxprmut9emr++Ttp1T9E7+967qadvO9/wD2xdfP7yarICgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKAPM9c+HOg63eT3txaWpnnXBZrYE8DqxwPfupP4161PMXCEYyu+VWu2m3bbpd7rrfvez5fPeATbelrrr/h6eevXvvqZP8AwqHw3/z46d/4CD/Pb9fY1f8Aaf8Adl90P/kvP/hrPlP7PjpqunX/AA/8H9dpC/8ACn/Dn/Php/8A4Cf/AGP+cd/4j+0l2f3R/wDkvT9bWly839mP+Zf+TeX+f4PsH/Cn/Dn/AD4af/4Cf/Y/5x3/AIj+0l2f3R/+S9P1taXKf2Y/5l/5N5f5/g+wf8Kf8Of8+Gn/APgJ/wDY/wCcd/4j+0l2f3R/+S9P1taXKf2Y/wCZf+TeX+f4PsH/AAp/w5/z4af/AOAn/wBj/nHf+I/tJdn90f8A5L0/W1pcp/Zj/mX/AJN5f5/g+x4r8APhhotz8NreSa1hUDxh8USNsBTBb4o+NOO5zgg5xjgEE/dr6Pi3OObMa9N6qnOnyttbOjSk0l/iu9WtXfXY8Tg7AKODUn1WvRXv3d9Vbvp3dve9o/4VD4b72GnYxn/j06/ToDj2/M5BrwZ5vzUo0+Rq2l+aPW2tve/B+Wqb5fo6eBSruTel9ezXk+RLTfT7nZKTv+FP+HP+fDT/APwE/wDsf847/wAWX9pLs/uj/wDJen62tLl5/wCzH/Mv/JvL/P8AB9g/4U/4c/58NP8A/AT/AOx/zjv/ABH9pLs/uj/8l6fra0uU/sx/zL/yby/z/B9g/wCFP+HP+fDT/wDwE/8Asf8AOO/8R/aS7P7o/wDyXp+trS5T+zH/ADL/AMm8v8/wfYP+FP8Ahz/nw0//AMBP/sf847/xH9pLs/uj/wDJen62tLlP7Mf8y/8AJvL/AD/B9j1DTrKDT7SCxgPFsMd8989fXP8A9YcFvGPZNGgAoAKAP//W/v4oAKACgAoAKACgArJ/H80B4L8bfv8Awm/7LR4F/wDQNar3Mv2xP/YFX/8AcR4WYbx/x0v/AG490QY5J6d8evH+I6flg14lJNxWjv5vfrf5qx9BUaVvLfr/AF/wfIWSTYM84/8A1jpgfXrxxwcZYqVFC2mr69Fqt9+/bfTW4Qhzv+vz6fdIlrQgKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAZ5g9D+f8A9r+tZ+zqfyfjEnnj3/APMHofz/8Atf1o9nU/k/GIc8e/4D6nkl2/EoKOSXb8QCjkl2/EAo5JdvxA8G/Zz+b4Zbe48ffGE+v/ADWHx5/u+vqf1r6HiuDWaYr/ABUfxo0vN26Lr8tj5vhWspZfD0d+n257aS9Nvv8As+6nnK+mMHr045HGOv8AfP4/w+FOD9it72S89PPm6pa6Lvrax9HGp+8ku2unnvrre1/5V28ySs+SXb8RhRyS7fiAUcku34gFHJLt+IBWwBQAUAFAH//X/v4oAKACgAoAKACgCBQduH4yeO5/Ec9vb35Joq+znZOSuuz3+7TS/n57XjlhlVjH31r63++36vtvqeF/G773wmGP+az+BgP+/et/55H54r28pUILGLp/Z2M5bPq400u//B8r3PIzj2k3CyslVppLTSzld/Z+5y77aKXu+V24zn8Pf3rw3JLdperX6/1+B7tn2f3Ebhccpu7cHr/nrj/69P2UJ63Tb3/D+9Z7Lt2vu5Zzr1KW0HZdmu/W8b69mvP+6WKCwoAKACgAoAKACgAoAKACgAoAKAKf/fuuX/hT70v/ACX/AOR/q/T7OHLge0fufX/t7pzf+S9LvlP+/dH/AAp96X/kv/yP9X6fZOXA9o/c+v8A2905v/Jel3yn/fuj/hT70v8AyX/5H+r9PsnLge0fufX/ALe6c3/kvS75T/v3R/wp96X/AJL/API/1fp9k5cD2j9z6/8Ab3Tm/wDJel3yn/fuj/hT70v/ACX/AOR/q/T7Jy4HtH7n1/7e6c3/AJL0u+U/790f8Kfel/5L/wDI/wBX6fZOXA9o/c+v/b3Tm/8AJel3yn/fuj/hT70v/Jf/AJH+r9PsnLge0fufX/t7pzf+S9LvlP8Av3R/wp96X/kv/wAj/V+n2TlwPaP3Pr/2905v/Jel3yn/AH7o/wCFPvS/8l/+R/q/T7Jy4HtH7n1/7e6c3/kvS75T/v3R/wAKfel/5L/8j/V+n2TlwPaP3Pr/ANvdOb/yXpd8tjIYYGGPrnb+XXn15568YxXVaErp7Pp8X5ST08m/Nm7U4dNV12/+SXXv00vd8rQB12n3y2Pbngfh0/Hk0OlT0TtZbaX39HH8/kr3klVqP7LXq4+X91/1fYmoGFABQAUAFABQAUAFAEBxx1x+HX/OOOv6VEFHC4drpTd/V3Vtvu0IatWi7av5a3Vlu+/deuvMeBftFfHLS/2dvh8fiHrXhjxF4n0SHWdO0zV4/DI0+S+0m21Eyxrq08WpXunxNp8NzHDbTeXdNObi9tY4YpWfZXvcLZVLiLFewp1o0qkoSlFuHOny2Vvii95JrWTtFtKytHxOJszeCwqej1s9Vfe/rfo9Yrzf2fkv9iL9rrwv8XLi5+FPhzwP4xiutOu/HHi3VdduodLj8P6fZa/421XW7GEOurT3vnf8T6KzeEW2wTI7RSyxAtX3XifwbPKsZUr1a6lGq6KgpwUZOcKNODu+aSnKo4OdlDRNp3UVKP594WcVRzHCKjyyvrZ3u929neyXM72dut3J+9+mO3aB/njt9c4znj36gt+Vw9xKF7JWjf002aer82u+tj9h+KXr210+5X09PVWuT0iQoAKACgAoAKACgAoA/9D+/igAoAKACgAoAKAPnn47/tLfBj9mzw2fEXxa8baboCTRTNpmjvKt14h8QPDFJJLHoegW/m6tqzxrG7TiwtZlt0xJcbIzXoZDwzjs0rwpUbSbdnUqNqmrW0dou8tU0lvrqrNHi5zxLg8roynNXmldRj0636Pbvd7NJXufzz/tF/8ABXH4pfEvxPog+EHhqw+Hngnwn4msPFOmr4mtxrfijxbcabhltPEn2eddL0TRrnzHivtC0KS71VHj32XjF4ZvJg/pLhbwbrzoe1r1Pa1KlKNOdlGMFFSk0oK/Mm1JRlJybnyKSUOZqX4FxB4vUaVWUEuWMZbXd76eltrpdnZ30Z+jn7Ln/BWz4QfFb+y/Cfxuitvg34+uUjt31i8uZT8MNb1ASCGSTTvEd2p/4RISc3kmm+MLmO0sIZbSwtvE2u6i1wqfmPGHhbmWQxlWpS9vRhG7SSWIilBucm00qt3FKKhGM3KrbkSjzy/QeE/FDB53ONGceWUnpNOyvJ266xSTv7zl8Oj1SP1yt3WWOKa3mE8MoykoIIYHOCOnBPHAHPXNfkv1TEYepKlKUo8t+ZS3SVvxta2+j87x/WYYijXgqkGuV25UlvfzfX1SvsukpaFbEBQAUAFABQAUAFABQAUAFABQByHjPxhoXgLwxq/i/wAUXw07QNCsze6tdGAnEQwpYD5ieSBtA4zguP4QD5I/4eH/ALKf/RRv/KXJ/wDE/T8u2fl+5/4hvxb/AM+Yf+Bx8v7vn8td+X3vif8AiIHCf/P3/wAoy8v739WW1/dP+Hh/7Kf/AEUb/wApcn/xP0/Ltn5T/iG/Fv8Az5h/4HHy/u+fy135feP+IgcJ/wDP3/yjLy/vf1ZbX90/4eH/ALKf/RRv/KXJ/wDE/T8u2flP+Ib8W/8APmH/AIHHy/u+fy135feP+IgcJ/8AP3/yjLy/vf1ZbX90/wCHh/7Kf/RRv/KXJ/8AE/T8u2flP+Ib8W/8+Yf+Bx8v7vn8td+X3j/iIHCf/P3/AMoy8v739WW1/dP+Hh/7Kf8A0Ub/AMpcn/xP0/Ltn5T/AIhvxb/z5h/4HHy/u+fy135feP8AiIHCf/P3/wAoy8v739WW1/dP+Hh/7Kf/AEUb/wApcn/xP0/Ltn5T/iG/Fv8Az5h/4HHy/u+fy135feP+IgcJ/wDP3/yjLy/vf1ZbX90/4eH/ALKf/RRv/KXJ/wDE/T8u2flP+Ib8W/8APmH/AIHHy/u+fy135feP+IgcJ/8AP3/yjLy/vf1ZbX90/wCHh/7Kf/RRv/KXJ/8AE/T8u2flP+Ib8W/8+Yf+Bx8v7vn8td+X3j/iIHCf/P3/AMoy8v739WW1/dP+Hh/7Kf8A0Ub/AMpcn/xP0/Ltn5T/AIhvxb/z5h/4HHy/u+fy135feP8AiIHCf/P3/wAoy8v739WW1/dP+Hh/7Kf/AEUb/wApcn/xP0/Ltn5T/iG/Fv8Az5h/4HHy/u+fy135feP+IgcJ/wDP3/yjLy/vf1ZbX92P/h4d+youBH8RFHru0+cn/wBF4/P/AArLDeH+ccrlH6quvvVaq2/w0Ur/ACf4tzeL49y+LjGKqavol8t3JLR/8Bs9d+Ef7UHwb+O2qajovw28VJruo6XardXipA8aLGWC53NwW5yc7RjkZwEbws44ex2XK1dwVna9KUpJXtb4owekmltLvd3kz3soz3C5jFOClqutlfTpbra2u3RJ35j6NryD1woAKACgAoAKACgAoAgOOOuPw6/5xz1/Sog44rDt9Kjt6O6tt9+hDd60VfVfPW6s9l27P005Tx746j4aXHwp8Z2nxe1zRvDvw+1PQ9Q0jxHquvanZaPYW9hqVtLayEX988UVvdlJGS3YP5nnMggzM2F9Xh7GV8mxmGr4aUVVw9TnXNKSjKPwyhUs03CUHaSSu76Rd0ePxNgIYrByU27Ws7K9n0cU99dnfa6urKcvgL/gljB8CLT4a+MB8OfiPoHj3xtqvibV77xTLa2Gq+H/ABBF4f0XWr7w74VZtD8Q2WleIV0E29s+sWdzcad5EN94ju4PODzKi/pPi1xDi8zzanOpRlQwtKjF0oTlRnKVWpGEqrn7H2jjKLnGi4urOF6LqUXKMm5fnvhXwxTy7Aznfmnda+8rauyT0vfl5nZWSsnaTaj+sPDID0+nbsB+H4+nHWvx7nVWm2tL2892vPr6/PZn6/8ABL0+XT/t633fdf3ZK2JCgAoAKACgAoAKACgD/9H+/igAoAhBxg7ePXn6dTvx/X8jWKw9GlpZK1t9tV308t1932Xdy637WX6d1t/wwuWPTgD3GP1+vv6+gWnOnZJO60VrNfjq3r5L8bg1br+X6X/rtqzh/HHxA8E/DHwxf+MPiD4j0fwb4Z0xN97rWu6jDpthb8cebc3MscSA9tzAFto4ZlFa4XC43NqsaEaMveauotK3RX+Jx6rXzsna5y4l4TK6Uqykoxiv5tZdLWd/Ju/4Xsfg/wDtUf8ABZK9l+3eD/2T9HjhjBED/F7xhphJBDxPv8I+CtRihLF4vmg1PxcQIpVns7/wXKrR3i/vXBHgtTzClGvmFWNPmirYem7RV1JNTqtqUpWlGS5FHknDSU0z8V4n8WK2W1Z08JS5lGTvKyv0t0drNPr7yavytM/DPxj4x8X/ABC8Q3nizx14l1Xxf4hviBqGs61qMt5dzBRgKZJS8mI1CogIIEaqi4UHZ/RWWcM5fgIxUXThCKjGEIWjGEYpKMVFWUUkkklpG1kmrI/n3MMTVxLcvek27yk7tybesm5O7d9Xfm19LnMQwzzS29vBB9quLr/l0/l2P9Prwa9SvSy9JL2kbLz/AK8vxPLoqrF/C79/8vk7O+mh0Gp+FPEeix382teHNa0y30rWdS8M6o2o6XqNhFp3iXRtv9p6JqMssAS01u0DBrjTLlor+BFd5bRVjcL5tbNMsx/FtN069OcKqjKnUhNThOM6cZRnGSvGUZRalGcXytO6umj08Ll2Y4PDyk6U1bf3drdO6/B39bS+xf2aP23f2l/2UYNBl0i61Txb8JtXmvobHwN8QI9UPhK6GleT/aY8DeJLhZdQ8PzwSXSCaPSJ9T8Pwajf3Go614f1fWLg3EvwPH3AOXZ5nsnGSpzrJOVWjyKrdRgk+dxd3ywjC8k2oaJxSPueBuNMblWJcJRklGXwyvyu2usbpPV6Kz+Vz+i/9l7/AIKJ/s+ftOJp+i6frX/CCfEu5tlku/hz4zuILDUZrkN5Tp4W1nMWjeMrZ5I7iW0XSpY/EA0uBNU1zw3ocN1bI/8AMOd8F5llLcqlB1aXu8tajFt/DNy5qeso8vJ7zfND36cY1FK6P6Qyri/B5g1BTjGXZuOr0tZ6J3vsuXVSbsrOX3uWUgYGR27Y9vU/iB+PO34uVSeGbjZuPR2vdff3020203PrI8tdXTUk9dfy26dvu5t4r5fv+n/2yhUWn3/D+rfp5mvtPL8f/uZJVJ21MwrcAoAKACgAoAKAGj+H/d/+JqIfwfl+kgl8f/gX5o+ZP2wv+TaPi7/2J2rfyFe5wn/yM6PrU/8ATbPm+LP9y/7ef5RP5FK/0ps6mj1S+X/D7f1c/wA/L8ju9LfP+r3/AKsFV9QVS226W/8AwPk9fv2JePcLJbej/H/h3u97tBT9tiv5H+Pl/wAH8PM6LYX+Zf8AkoUe2xX8j/Hy/wCD+HmFsL/Mv/JQqOdTtra3UxvyO70t8/6vf+rBTWDhUSd196V9u976LTWXTuiXj3CyW3o/x/4d7ve7QU1TVN3vt8//AG59/wC98gVSUtLaf15a9Nmv/kil9b9na3X9Gm+jv+HbrcpYTnd9r+vl6bK/dde6CksHy7rXp06K/V6/8Nrq5L6q4a223W3bvzefW9u+iiVSppW7f115u/8Ad/zE6ko6dOn9W/Nv/wCRKrCfxM8/7l//AG0Kv8OPr/7ej9b/APgkdj/haHxHz1/sDw1jPXP23xFjHf7v/jv+zmv5a8f/AOFkfpivyR/SPgl8a/w/+2H9DNfy8f0wFABQAUAFABQBUJRR864APGDz/I/y/LiuenRw+XwaopK/2W7t/l+f3aFVKMcRZPVLXfX8b/dZ99LCSNGqDcCFIz1OfX8f6d93RVGri8Tf2EOSL+03a/pu9t7PrrsubGdejgIJSk15fm3f/J7baLl/LP8Aax/4Kn/Bv4ALqXg/4aCH4x/FC33wTafpOoIvg7wvcsJ4w/inxR5wS4ntZ4tj6D4fF/fvcQS6Vq154bmlS9T9Q4a8O85z+vBVaFShSvZvkSnvC/KrOKTi5r3uVqUU1FqTZ+c8R+IGU5Dh5+xrQnPde8+z681+kbWW2n90/m++Pf7S3xl/aX8TSeIPi74tutVjiuw+g+ErQR2Pg/w0oi8oJp2jp+4eQxgJPql/LfazfIsMeparfJa2yxf1RwpwLl/DFCCqqFOUleXRtuUpat3k4qUpOMZaQ5mo2SSP5p4j46xvENaXsE5q9rrVW0Xkruy9dNV8R4/4a8TeJPBGvaV4o8H+INX8MeJNEulutM1rQ9RnsL+0YDhor6CSOeIkZUiORWKZQkI7ivczfDxznDVcPKEakKsJU5wnFSjKE48s4TjK8ZKSbTi9H1Tu4y8bKse8urQqJuLhJSVt007qzVnfZ6bb62sfuz+yn/wWJuNPSy8D/tYWMV1EZYrW0+LvhnS5I8rLcENL4v8ABukWtyUaGBvNl1XwmvkySeTa2/hG0gSbUIv5t4p8GsVXlUqZbSVkpyWGat7SUYr2cKdSU7QcpR5W6js3NynKKjyy/oThjxRw2GoUqOJfMo2SldXj3drX200eiVkup++Hg3xn4S+IXhvTPFXgrxDpHinw3rNsl5pWtaHqNvqWn31u4BEltfWck0EoBJQ+XK2xlK/eDBfwaWWZhlFSVDFUZQkkm4TTTa1tJXsmuqsrWurLXl/Z6GYZdmsE8PXjUX912kvuenytfu7e700iI6hShdV98Y+uOvX/ADzUypUsSv3iUlf4Xfy/vJrZXV36nW6tTCpKnBySVtLbefe+97R212sWqosKACgAoAKACgD/0v7+KACgDMmmgsoZri4mit7eBTcXM8xwqLg5YnIAwFxuOMcAKxwK4ngcViKsKSm3zyW3vOT7JXjpbd8ytu7byzpS9lCTfRXbfmtUlrvbZS18tUfmx8b/APgpL8KdA8N+M4/gNrPh74ueM/Dej6zeyS2l9M3g/SZtK1zwXoDT395biO916zuJfHFnqmi3fhpZvD/iS00TxBZW3ivTrqzSZv1jhDwxx+c8k6kPYUpu0OaUHVrT5HOLSXMqcVGMnaaU9EusuX8+4m4reXuUab957uzskmtLXs3/AHk++zSlLxL9or4nfEf/AIR2PW/GE3xE8JHw34h+Jw1fX/DWjfEn4O313alX8C+B38HfE7Xx4n0DRMxeO/DfhcpJ8LceNfFyeIvGvw8W1RN0vr8PYXLKvt6WHWFaqKkqdLFU4Yyiql+ep7XDWo16jcIc0HPHSWt0owoxjL5bNsfmXLH29Sq9Xdwk6blH3UnF+9GNnzJ8tNJNXfNd8n50+Fvgd4M1rxx8Q/HeqeF/i/8AFYHxh+0/4V1Xxn4t8R6L408NeBtU8B+CS2g6v8a9a/4V0Rr/AIi8VeINZdvBAHi7wgQi+Gsf8JhjNfodfiHGZTHA4TB4rC4SMYU08PToRhOrTpxg3DCUXiIrDUlGElKLp4iUaT5YypSh7Q+NjleFx0qk8TTlVk7ydSTb1lL4pvq3J73V5NXvdob4D/Y3+GbeDfAHivxP4H+JHiLWvF+jaHJp/hbS/GusPpXjjUPEHwb+IXjvRU8Gaxr37Lfg7w87N4g8PaMqeGvCni34sRI+v7vHn2na6eMDGcVZt7bMKEMbhOTDJTVdQp8mGj7S7p4qnHNp15z9mpxlVnDA04zpTklUi+SHnYHhmhXjSnKlVd21OPtLObSjZ028PyxV5J8t60nFpNx92Uuk8C/s3fDv4Ra78CPid4j+GfxGt9Yv/jB8GtFPhjxr4w8XahpS6p4+/wCEgfQdTbWh+yp8J/D2vT+D9e/sB28M+F/FfipGWSPwV44HhAbkrzaWdZpjqmYYWWMpxdGnzOtSlRpyoU5qLalbMauIp1lSclCvPDYelCcfbe/BKE+/EcH0aUaVRU7cytKLvK7Ts5L91CPK29IqU5W0cm2z2nxH48vZP2F7uKf4j2t14zvfhKPHF1fXX7ZvxcHxcu5tZ1z+wWY/BncfDp+H8vhubKIfFf8AwhY2SMvgo5bf5dXAVsJ4hKP1W1OGIhiISWX4aNCNZUYV/bLEq85VVXbmqnLGpHEOyuk5n1FHEYHE5XOKSbs4OPtqjfI7rldJpR5bJL3dNrpaHyZ8MdeTRPhB4G8Na58P/hB4hg0bRtWvxqeoftIfsAtrF7c+IdYPiBJ28I/HP4O/FbxL4a1hI/7H8Kv4abxVGgm0KLHhWGWNFf6nOcXjsZn9o4zGULShhYqnhOJIJScGud1MuzLLqVSnzSlJ4qEY04wUFPENQhM+MweBwdPFzlGlTbu6jbqYX4E17qjVo1ZKpdLTnu43tRcZNnr/AMMfA0WqfEn4k6pofgvx3f8Ahvw98Rvg54jtdA0Tw7rOn+G7HxT4G0Xw946i0Y6T8B/2Dviv4b0DxF4U8R/274YfxR4V/wCFSeOJfBmv43o3iwO/g8RYu2HwNGc8N7SnCvTdWpUVSVRe7GMva5jnOExDjNfvKlKpVzD34wUqqlRpVquvDtWpCrOX7yTupKMLqyveaUaeGqR6pRs6MYp6XjJwj9veDv2pviZ8BvE/7Ovw98TQeMfGHhv4tXPj/Q7z/hbGreMLLxz4Jj8F6Z4duNI1DSfGfxT/AGfv2ffEPiLRSusawlzN8UfC582dBLH45aLES/D0uEKWcUMRiKEqLqUnKc/YqliYNpym6U6GBxeOnKo462pS9tPnh7OLknRP1ClxTWwXJCpzcvKrX5ot6JKSc6cHyp3SUrxWqdrc5+qnw++L3w8+JUFp/wAIj4x8L63qE+h6X4jn0fRvFPhfxJf6fpmqhkhluZvC2s69pVzCLqC90z+1NJ1LUtHub6wvodP1K8FtLM351VoSiuZwai+ZJu6d07JuLUZJN2kk+VPyuon6LhsS5y5W72s+6emqT6y0te3nZ2fN6i56D1ryaz1t3/4B68Fu+w+ukzCgAoAKACgAoAaP4f8Ad/8AiaiH8H5fpIJfH/4F+aPmT9sL/k2j4u/9idq38hXucJ/8jOj61P8A02z5viz/AHL/ALef5RP5FK/0rxD+rx5l5367eunX+rI/z6w79s0v6f3dm/PtyrVy9i8H/s//ABq+IGhp4l8HfDDxpreiXUpisdZstC1Oawvtt81hKNMvEtzFfNBKkq3Uds0zW3kymZF8lwnx2O41yzL688Niczy/DV6aU50K+Nw1GtCEknGUqdScJxi001JxSaaa5r+99PhuG6mKpxnGjUnCcuWMowk0578qa5byt037JXtKvefAH43WviKbwe/wm+JMniiG3/tEaNZeCPE97fnSv7T/ALJ/tlbWLSZJToBux5P9shV0nzPl+2YG9Z/4iPkvsPrP1/AfVnV9isR9cw/sHW5ef2Xtef2bq8q5vZ35+XW1rsy/1Lx6k4NVedLmcfZ1OZR7uO6Vuu33lHWvgh8Z/DWl32ua98IfiboWjaYG/tHWdb8C+KtM02zC8E3uo3ekW9rECxVF824UNIyRrl2U06PiJk+IqRo0Mbga9acXONKji6FWpKKV3KNOEpScUk22lZJX1t7qfBuOUXJ+0UU+VycZqKl2b2T8nr9zOL8M+GdS8WarBoWjIst5c2eqXqwvP9lBTRtJutY1AjIwSllYXMqgnD7GVcEru9/FVo4aPPKVlzQinteU5xhBfaes5qO+7V7fZ+ew37+Siv60127fPa3KtXLu/AnwQ+L/AMTIhf8AgD4aeNvEmnNPeQDWLLQdSm0D7VAA72T640SaRbyoCrFLm9gMayI7bEYM/wApmHGeByysqOKzDB4WrJKcKWIxVGjOUOblU1GpOnKUXKMo3StzRcbtp8302F4cqYuDnClOpFfE4QlJRsm9XHltprdvbX3V8XM+LPAnjbwFd2th438IeJ/B+pXduLmz07xNoWq6Fe3Vp90XiQ6rawzeXvBRSsQTejr8pVki9TL8+w2aQ9rgsVh8XR53TdXC16WIpqcbc0OenKUFOKkrxvePMr3uicXlMsIvfhKm0ublkpRdmtG00m77rTbtuclX1WHwn1iKffX8n+W9vX3fhPm8TjfYScdrO23b5vyW33XSCliMXZW/r9PLr32Fhsb7aSj3f+S7/fa2r2Z25+H/AIy22/8AxLImW68If8J+7LrOgD+z/CnzZ1PVTJq6JpdyqxRZ0C7EPiAfaNNP9ij+1tOW6+ar5ynKpGKqJ06qpSvSqxTk4wmnTbhFVaaVRXrUuelFqUZTUqVVR+kw+VqvBSaWq5kr7/qnps7Ozvs7y4ivqMJ/Ezz/ALl//bT5yr/Dj6/+3o/XD/gkp/yVL4j/APYB8MdfT7b4h/p+GP8AZzX8t+P38PIv+5r/ANtP6S8Ef4kfT/2xH9DFfy6f0uFABQAUAFABQB88/G79pL4WfAaxC+MtfjuPFF3oviHXPD3gLRWttQ8b+J7XwzoOs+JNUbRtCNxbstvFpnh/V5G1fVrjTNAgms5IL3V7OQpu9LIuFsbnNaMY3qJTUZ1qn7ulBy2SveU23yxtDmac4XbjLnj4Ge59HI6EpuNpcr5Va9/Wz2s76tX1Ssz+eL9p79qz9t/9qbTrnT/BPw0+I/w5+C+orpVnbeDvBmkarqHjfx1pPjRZF0Ftc1mMRa/rnh7xTEzRmTwhCng7yUWN38aKibf6A4d4UyLh+k8VjPZ4ipBN1K9Vw9nS9jFxqOMY2pUfZNVFOo71YKdSM6vK5RPwrO+OsVmtX2dJySbSjCCd5OTTirK8pczUWo7N2at7qPy58T+CvF/gDVZPDXjPwx4h8H61Eu+bRfE2h6loOowKxyjPpuqQWtygcAlC8a70YOm5GU1++5JneSUaUauBjQqU6qvTq0XGpTnF6KUKkeaMo9LxbV/5mkj8TzzI83xN1iKtT3W7xk2mmrNppybVut4/JWOYpZvQxmdLmws3FN6crt+Vuy/4N2zjyrGYTIm1ioqbW99X5dO1769+75Suqklhk5Se2vS36eX2o79bWNquF53p1stN9bbdtj3y/wD2ZPixpcesi+i+Hcd34f0PU/E+saJB8cPghqfiyw0vRNGfX9Ulk8G6P8Rbzxc7W2lIbqSCLQpbgJtbafMR2+XxvG2EpV3hvZYznjKUeapluYUqN07NrE1cJToSjppKNTlad1KUT6DLsgrTpqpzx5W1tWpuX2X/AA1OU7a/Fa3TV/D63+y34i/bY+DN9D8Rf2cPAfxg1jw1qM4fV9N03wB408X/AAt8bLpSywPHeaXY6Vc6Kb8JO1pJqug3WmeJoo4hp1vrlpErxN8rxtg+EM5p+wxGIyvBYj2anSjiMXh8LiIwlJPnp+0nSqqE6lFXcbQm6fLK6i1H6zhfHZ5kFZVJSxFWmpJS5YzqRulrfljJXUZdVondNXP6S/2Wv2xdQ+OczeCfiX8GPin8EPi7pul2Oq6l4c8VeDfE39gX2n3X2+Mavp/iBtItxpNm8umThIvGNj4XkurqeLT9Bl8QvBc3q/zDxFwpLLZLE4PE0cRgqkpKGJo16NSm2pNckZxqNTenLaEqjvCd0rJn9GZBxhhsypqnXhKnXjFXi4SjJtpPVNNx0k5av+XRqx93V8ufXBQAUAFABQAUAf/T/vxY7V+bCKPXjp9cn9R6buRWEfbYjRq0Xv6Ltvfvp87aMUp0MPHmbUYra+nl216LW/TRW978+f2pP+Cj/wCz9+zIt5oD6m3xI+KMRaCH4c+D7qGS6tbxCiMvivXnWbR/CMMZlgluYLp73xILK5t9SsPDOq2LGavv+H+AM9zq06WGqUaDjf21RO8uaKlD2cNJz1lG7fJHluuZzi6Z8TnPiBlOVycJVoSkm42Ulp7y1bVvPdyacbfDJM/nI/aX/bw/aK/acvL2w8WeJZPB3w9nQeR8L/Beo3FloK2rFyF12+t47S88aTqv2dpLjWzJpseoWi6roWh6R5/2GL+k+GfDDL8nkq0UpVU5P29W062u6UmvdVkk4wte125Sbcv504h8ScTmUPZq9nvFaL100bV203ffTdKPn3wv+Ev7RXjf4WeOtc+Hdv47k+Gx8QeFtK8UaVo0vii18N+LdWu2vH003os0/wCEav7Xwqfs97rl7rt3bWvh9tf0LUj5UOqi9tPo8xxOR5fjsJhK9bCPMFSrVMLGpKh9ahSlaFV0Yyn7eMa0acm/ZrlqQw9TmdqT5fnMAswzCM6sKVT2cpWlJRlyXVrXfKotxco7vTmitOZc79I+CXxe074j+GvgodW0vw7qfxNhsNRgfRvGeg+NPCmqWugavrkenavc3vw0uPGlpfjQtW0PXH+yxwXetaV9kvXXS089Un5sVxNSrUM8xFKniLKNHlhXw1fCYh/C3FUsbDC1KfO4pJ1Iwg3ZtpJuPXDIJqfLo9bPllGUfVSjzKXqr76fEmeleOv2Kvi9omh/Erx54h8U6Jrh8JXGp6l4kvLrQPjqdSvNSOiaF49T+3da1/4NM2g+IX/4TXRgrfFLxd4T8YHafnIXNeRguN8NKvw3hYx5XWoV60UsRgJKnT/fKMXTp4ydeafsJpTw9GtS0sqi9nW5PTjwnVcak3qoT5fgqrmaaTs3SUY25k2qrhPVe67nzpb/AAY+MVxp/iHVbb4V/EqfS/Bl9rNh4y1O38D+KG07wnqegQ+dr2n+MNSi0f7N4autHiO+/g1mawltQrCeNGRq+2fFuST9jTePwiqYmKlhYPEUoyxMXZp0IuV6yd1b2ale6emnN8b/AGTmNZyfsKrVNpTfI2oXfLab5XytvTW21k3qz6j+HH7C/wAbfEmmfD/xt4V8U6bot14osdJ8beG9Ws/BH7R3/Ei3bv7CH/C0PAPwY8WfDfQCAuTnxWB4MBG0vghPgs941yzD42pl7oyrVLwg+XF5SlL2sISs8PWzGli9IyS5ZYX3lrCM4OEz7HJ+EsfWpKs4uKvp+7q91tJU3Dd2/iXVuljhvhZP+198ZLZPCPw3+L/inU2vNX+w2XgvUP2kfCvhLU77Uv7HQEaN4K8cfEHwv4n1xf8AhHWVv3nhBFwHfIWN2Tv4n/sTJsfHF4iEo1fYyUsVSwGKxLhh6clVnCriMNh6/sqSk/a8lSrBN3qJe7Jnn5XUzHMMFUpxSd2vcnVp07yd0mlUnT53bTS9r2b1XNnfD79nH4+XfxG8N6V8N59Om8ZWtvqmu2/iD4T/ABN8JfEDUPC76OVjGra3rXwJ8QeKfEvgWOZpF8JoyB45Wby0Zn3Be+rxrkUcrnTxU4xw008LVp5lha2EpVlUjadGVLMKNBV6dSEpRklGdOceeOtpxjw5fwjmdfHOcKb57+1hKjUjUcbO6lzUpTcJRavZuMo2T926ZieLv2e/j2h8Q+KPHFnpk3iKLTY/GPirT/EnxT+Hdz8UNmuyQSC/134ezeN5vihZalcS36S3Z13wraXYeU3F2Cx31pguK8njUp4KgqkVTpxVJ08Fio4FUo0lKEaWLjh1gFFU2oxjCvJJpUlaacDvxmR4+UFU5E7t/bg6l27a01UdRaxveUbdbpNHo+nfsuftB28VhDpnjv4eaZpOgnU00jVbr4/eD/D2naH4713SCut+Df7E1zxJ4U8TaB8QfFniPwRrnho8/wDFYeMdDy/jdPAvhFvGa+PmXF+UUa1SU6GKjiHKFGXs8DWr1KtCFaUadWNXDwqU6uGhGpLE2jWlOhGu6dWlSxdR4eSwfDOYTS5VFxs5X51FJ8nM4tT5Wp39y1vecbxlKK54+Mav4++P/wAH/iNqKzfE/wAc+H/iH4XttI8Fanf+GfiVeXqWemeGfIj07wQ+reG9dvdL1Twxoc1hDaN4ctrq+8PJPZCKO0lKB29aHCuU5zg6aVClKhUbxVKnVwzhKNSup1ZVZUK9ONSlXqOtN1faUoVlOpVVW05TUcqmdY7Lat5uScbQk1PmTjTekVKPMpwXJHls+W0U4ppRUf1z/ZY/4LHaxpX2Hwf+1Roy6vpqloV+LnhDS5Fu7VVW4cN4u8EackxvGChWuNV8HtHIztDaWPgl/wB5dr+NcYeCjre0nl6g7XksNUuoTd4JKFTmapu0ZaSUlOco806cUz9T4X8Wfq1KNHFR5oqyTdrqyf8A28/ecVo48sVZKd2j97/h58S/AHxY8L2PjH4ceK9G8aeGNSXdY6zol9DfWM4HUxTwO0TEcA7SSDkc4YL+C4vKcyyWoqGJoSg5WklNW5o3avHv2tdPbmt7rl+84TNsqziHPQrwqJXuk7NXt3vvvvrqtbmZ8Rvi98NPhTZ2U/xA8X6N4dbUpWg0uyvLtBqWqzxRPPLHpWlwmXUdSeKJGlnWxt7owRDzZgsY316mU8N5xxFJ0svwNXEtK85QjanBq3uOrJqClaV4wb5pK7SdrnBmvFGS8Ox/2nE04a6R5k5NPvtfpdq26+HRnqfy+ifnXg3r/wAn4yPevS/nj90f/lgfL6J+dF6/8n4yC9L+eP3R/wDlgfL6J+dF6/8AJ+MgvS/nj90f/lgfL6J+dF6/8n4yC9L+eP3R/wDlgz92GB4DY6ZOM9uf/rj8B96ous6TvH37WV97ed2tfVf+BXMv3HtVeS5r/wCWu266K/lp9r5n/bCz/wAM1fF/djP/AAh2q4z1xsGeO3/1885IX3eEnX/tTDvlk53qcq7/ALqd/La/n+B4vFHsPqH7yS3VvvS/Lyt63Z/IrX+ktN8tOSfb13fp5d/ut73+fM1epHtfXW3Rf1/w5+kfwN03wQfgfotxrWj3y6ZY+LtH1zx54n0Gz0Xw/qzeFNH1rXte1ifW/GuufD0vrkPhgxeE/EQ8M+GE8WlpNX8O+Ck8amP4qeD/AAd4U/C+MKOLqZrjKVGvSjVqwp/VMNXp1sRzzawzo1IUsPi5KlSrunmVKU631Oc/qeIxNHCfVstzHG439a4UxtLC4anKom07pyi+RRceZSi3Ok1KUb0Z+5zRSnGDfPUgoZ/9peA5v2hv+E20rw14a8P+AdU+GXiHV7PV/Fvjb4TLpvxI0/Xm1nQdC1l11r4fy+HhrrSlvDfjnwy3hLxb4tTxToniZZ/+Ez8arJNLxYLCZlDhvF4KrWcsS8ZTw85YTDZtfDV5VaNedGssPjPrtKlLmdSGInXweDjgK+GlKDwMkqvpY7G5fis7oV6bioJOV6k6LUo2krpyhyXUVypJSqOrGXLJVGow9i+K2ia1HL4svvA13o+u/EW88d/2v4R8Ow6ynh6RPg+/jLwB441nxewU/CzxFofh/wAL/EbwWfCwPm+Eg/hA+I/Go3+BvCv/AAmVfP8ACU8Rl+Pf9p4TF0MK1iKE6tWjDFzjiK2FngqOHnQlDM6eLljKOKrRoqr9ZhXxf1fASWKxeKWGl73HFPAZrlFOnQrU6tSMYytB8icYtTlJXlTlDkcU5KKXLDnqe5CHOYsnxb0jX/HulT+DNQHj6TR/CHiV5/FXijRPCPxgvdA8MaV4Q8cQeM4tb8beO9A8U/ErQdCfxKIfEPgHxThvB/jLwd4h8rxpzIVbrp8L18LTpfWqP1Cn9ZlSpxweNr5bCc8RiKVLBQtha+HjWrzp+xpSUqvOsRB08JCpGpBnn1s/weI5lGpGo2nOXPThWaUIuU5XnGTUVaUpWVnHWpe3u+M+A/DF7rXw08CG7+DHif4l6rN/wn3xA06Lwv4X1e9sLTTPGfjJfBPiDwbqyaB4bHhvwL/wlHhzwX408UeDfE6HHgrxjonhj/hBtysFb3a2OeDrThWx+HwFVzpYW2LxiwijVwtKGYUqtGkqlKeLcJ4vBwxOGVbD0q+Gqzp4n2lN06c/nalPCYi84pzivek4xlNRU3yWlK0lBy5XyScb3inG71jVsvhNo1p4o+IWm6h8M/Dl1bfD/wCDPhvV/CjfELxDq/wu8F+OdWh8ZeBdD8Q/EdfGa6/8LD4i0PxSdZ8bp4L8SDxX/wAIhkeGgFD/ACUsfxVm2Y0sN7LG1qd8zrYXE1Muw1PGYqHs8HjcQsEsK8Lmco4ml7Kh7eEsOqyhTqTcaLm4GM+H8py+av7OXNCMoKtOVKnJTlGKm589L3G7yUufkvo21Fm/oXgb9mx7b4r6ix+GWo6Rp2pePX8PWd/4uaPVbHVdE8HQa94d0XRtWf8AaD8Ivr/h/wD4WAZ/DHgzxP4V+FHxZfxayqreMHMjPBFfiLiTAYnLHhZ5pKnOGDeIpvB+zp1qUsRTjOriObh6s6WIlSdSWJofX8tlhlCLVCnds78vyHJsfGpGpVoqak0n7S9rRldQksQozjzcvLJQqJpaTd0dL8TNW+Bd7DY+CJvCfwQtvD2ueOvjnrHhy88FfEbVVHgfTdZ+Gvgb/hCNa/4kHxBXw34f8ReLvEWijd4X8VfuiNAUx+CFJd28nhnJ83hQxrp1cxw8aVHAYeblhcNSp42VDGZjOvRk8TgZ1p0o08RHlrYN0aUvap0sTz+0lD2HmmXqCTdG8nOSSqTfIpQp8u1RpWd9Jc004tTUrRR6z8bviF46sbm48O+F7v4yXGueNfiX8fPDeh6r4D+K2kfFH4k6JqOhaP4E1bRdK8Gp4HVT4E+HL+IIdGXxr8NWB8YeFIkWSNjC6M3znDeVU44jFVcVh8ulhIfVsViP7Rwc8Lg44OrUqQ+sVFmEZcynShiVRxFT3Ks1701Cc5Gkszw/tKfLVqc1lTp8tf2suZKElCDp2UbSnFulG8oar4lY/Nf4/wCp/wBpfGr4p332n7V9o+IPik/a/tX2wTg63d/N5n8XfPCheR/CRX77kGTvD5Nk69l7JJQSgo8nJZx9xQ05eX4eT7NrXe5+X59marZpmdnzfu12b/h73210b018j9D/APgksQvxQ+JDemgeGCM9f+PzxD9e3oMY9Vr8q+kXhbY3ATeqjRnr5unFLda2dvtL1d7R/Rfo+YpLL8y86sen99+a/X0P25u/jT8K7P4geHPhTN400ab4heJZL6LSvClnN9s1QDT9E1PxHcyajDaJMNJi/sfSL26gbVHsheCLbZ+e7Ii/y3TyjOMJl/1zEYWr9XulGu4yp05JtRXLKfxJykoqSTipKUFK6ly/0ms8yXNcXClQxkKtRXXLGSk9E731v9l6b2afvXijkfj7+0d4R/Z7tvDt34q0u61IeJpNQt9LSHxd8KPB0Jm0uCK5miOrfFz4ifDbw4ZDFKCsSa21wxaMLA29SqynIsVnNWtJU25UmrwhCtWa5ny6U8PSrV3r/LSnblk3eMXKO+dZ3TybD02ocyl1vGKslp70pxh0WkpJdVzXtLa+Bnxx8L/Hrwzq/ijwlY3en2Wj+IJ/DV0L3Xvh34njk1K30rR9ZkNlrPwv8dfEHwre2wtNcsR5tr4hmkWfzo5LcKiPKY3CYjCOMKi5ak4qai1KL5HdJtTjGUXdNcrUZRaaadny6Zdi8HiLyhJSSdnrH5pWck9Nb3s1om17x7hux94qo+vPPTA3N9f8f4fLiq7esUl52/4FtdPyvax68pUorWav5v8A/Zv1tZ/d9rh/GHjvwd8N/DOo+L/Hev6R4R8M6TC0+o6xreoQadp9rEON09zdyRQIM/KvmPyxAGfl275Rl+bcR1YYanhKk6s9fYwjbRWjzSlJ2jFOWrlZX0VndHm5jjMq4cw8q9WpGnGK2crufdK9n91/0Pxb+JP/AAVIufjB48tPhD+y5beKdA0a5e/fWfjqfAGmeMNQ0vTNHmU3GseH/BnjjXPCHhbQ9AiZ2m8XeKfin4ktG8K+FYNblm8ER3SJdwfsmW+E2Ky/ArGY+MKUrUpfVp1KlP36so01RlVo0sRVrV71Ixw1PDUXUrYrko0ud1ISh+P5v4pYfFVZUcLK8YuXvRUZXstGoO0YqTXvuU5pQcnZWlGXyLq/xl0T44fET9oXxDoPirxX8Qzd2vijxL/at78KdM8KDQfA3gr4N/tNaEEFx4I8ReKfD0nh3wq3jrwQsXifxcvgzxn4xGvrGihVQL9hguHK2ApUKeKhQwUXVjh6S+uVMVCc6tSlGjRjPE0sPXlWrOVSnShFShTSWk4pQj8tmvEtHETfs5OenM7U1BpRjeTcIOSSio3bW6101Z7voXxM8Pv8DNN0SHxT4T8RaZ4U8MfB74j6ta2Xir9ne/uj4W8C/DLQNX13wVr2l+H/ABCvxnfQB4j0ZmHiCR/CXjc+NPHq+DkjHgJN6+djMhnisZVgo46nUh7WCawuaRoN3p0Y1YVK9KGXycG5NRUMRTmlHE06igpyl6GS8R06FNSc6XK3HedJz/mtZP2qTva/u6+43onH5p174UfAODw9p3iS/wDhN8IfBXhbVbjxLa+LtYuvi3400/xJ4V00/s//AAo8eaLrPwv8Fa98ZB4h8e+IR4h8bazt8Kr4U8XyNnwyCkeEd/osPxNn7xNDDyxOZTUIU53WDw8liHKrXc1i61LK/q9GHKqdKHsquFquMW3ao1Vq/LYjA5e4OadFe8017ZXirUuXkg6vtJat3vCcVbSVT3owi8CeDf2Y9D8H+AtR1fTP2dfEPjTWtC0S1R9d8cPYeHNQ1TXvg18RPNHjRX/al8WeItCcfEWPwP4VPiXxP8Jv2eZfBnjFlPlQM0TeFPLx+acQ4itjnfMlRhy8nLhFz0+WrH2jwyq5RThWjFKatGvmFSrS5ZUV7SUOb08BgcuUKD5qMpe9zqU3FSukoqbjXbj0d3Ggk73cop8vD/GNfhb8UNeTQNF+H/h3XPHuh2v7MPhvR1+EHxH1XUfEfxFA+GA0Dx14L0Er4i+LPhry/Ce3RfDjeKvDHhORt2hkeOn8XxhUX6PK8Jj6GW43EPETpULY2q4YyFP2OFca9WaxCqKnhq0aEk51q0MRWqxUHFUatGnDmn5eLxWDlOnH3b+5G8HLnnpFJNNyjzLSK5Ur2u03c/RvxHdReMvhR4j0Tw9ofxi8ZpNY3/wx8Q6t4S/aa+OXxStvENm3gzw6nifTYvA3wh+DP7RHwZ8OXN62vav4TH/CYfCr4WRxRQReNfAqeE5dj+EfyzHYbFYLNo16ksFRqV5Kpz1sHhMLCnKbjGtXU6+LwmJc5ylKc4Qq4mrRjUcK1OrKpRnP7zhvG4GpQceeTUFype1qTfLG7hBpRlTahFaPkpJu7i4pSUfhL9mq28P6N8P/ABhqd+PDHwjjvLj4zfAvxFo91ZfCOx+JXihz4OtY4o/GnjL9pT9oP4TiRfC8njgyv4d+F3hLwl4S8V/2BI3jcNbv4TFr+hcY1cf/AGlGapYnGRnCFWnRdLE1MLQ5J0ZRdKOBy3EqpOpVpKtz4qpVr0Gr4XkpTmfnfBtDAwwTUp06VnaUuaCnO/tbturWg42i3DlpqNOfu+099RcsTUIPhh4L+KHxATXrTTz468I/CzSvFnhv4jfCvwL4Q8N6XBpnjnwN4d1zXvFur/C/wD4/8TfBLxB8QPCmhayfDfgk/DHxV4S8JEyy+Nv+E4XxuI/F8XblqzV5PQnTjWS9rUqrD4qtj+dOhKUaeHWMrUsNmFOlPF0YVq0sZhsVKVKdfCqlUws6UKXpylls8bOPtIWvy88PZ663cnCM6lJvlbivZ1IR92Ml7/O5fadv/wAFEdN/ZW8M/BPwzo/hfxbrnhXX/AGj+MT4I8VeCvGGgeJD4V1rW/EehaPf6J4y+IX7VPxZ8S+HIH8PaNoL+DvDK+FPGXhO4KxxzeOvBsPilI/B35zDw4x2erEzpSozVNypc9DF08U4VqWs6E6cMBhVDlnOfteapCtzwjFxSfOfcZbx5h8nlCFZ2Tbupw5OaLt7+tWfvNL4UnFJt8z0ifrp+zz+1f8ABP8Aah8OjWfhV4ts73UbW2W61zwZqbRWHjDw8rym3zquiSSCc2n2hfs8erae17ol3PuhstUuZEda/LeIeF884ZrOnXw8oRlLlhVcf3VRqKekrScZJOXuzjCUuSbinFXP1XKeIMlz+ipQr05Ssm4OaUo3dv7u/krapO7+L2vxV418L+A9Cutf8aa9pPhrRrSMvd6pq97b6fY26DkNNcXEkcSBuihnUnoM5FcGVZXmWcWoYfD1a9dx5nClFycY6Jym/ejFRbXM3ZLS9m7R682zvK8mlCWKxEKfO7JSaUnpt5u3l2vy3RW8GeMfDPxF8LaV4w8J6imteGtdtlvdM1CKOWAT20gzHMEmSGQI6kMu9FJUgkDJ241sHjsmxMsNiKTjKPvO9uZK+nyduvLs9ehtCeX57h4SpTjVptKzjJ3v+V/RSXQ7IEZ4YZ+oH8/X+uPSueNKrF3crrqrL9E/08nqzs9vQn9pfelv6y/z9Hox3P8Ad3e/P8x/kdu9DnWhb3Fr07f5X8h8lGet0/P+kvL/AIF7R//U/t1/aQ+FfiX4zfCbxL4D8J/FPxn8Ita1iJIrfxj4IEH9r2+dwe0nV1ttSk06VWLXMWga54Y1mfykgtvEFhDLOs/sZHmGHw+Np1XTpVYq7tVSjy8qcouMndQvNR558s3GN2op3cvns7y3FYjDTjGpK/LpyyTbvumutlorSj1dtkfyI/tCfsg/Gz9mfUpl8aeHZNU8I/aAun+PNBjfUNBuInluEhXUZII/O0TUZhb5hg1aCwe5c7dPN4g81f6+4H4tyqNGMVCEnFWaaaeihe107pc8U2k0pPldpRfL/LPGXCuZ1K8mq0leT2d/vs7X08nbW61R8wV9vQ+s1INKTWmn/A+Vvz7H5zXqUKc4txvtq99Pv/TyT1Z+hn7KfxA+Fmg/B7xZ4Q8R2ngO11i4+MHww8aeMNf+IGrLp2ov4X8Ef29rmPBWhnxE7a/4h8J+HtE1lwUPjA+MvGvjv/hB28ElNyN+YcWZTjqmNo4n21dQp050YxpU1NSeJrUIz9vZVKnI5xoTi6UaMaEaNapXrOi5KH6Twvm2Cp0uV04NvVuTtblXTRK9rrVvm5rKKk1IPBc/gHX/ABv+yz498XeI/hxPoFxo/iTwdrXh/WPEHw41PUNC1L/hN/i5r7ax418E+PvDvinwvoGgMms6MP8AiqX8JeW3iBT4G8aupXxj4P1z3BudPiSlBV4yoLA1Zy9ji4wlFShVdOjVo+yliKlSEJwaw0qrpStGvRqpxo1erA4+MpQlNJp80UueHNeyScotPlSk4u8l72qi01zw+lfGHxN8O2Hg34sTf8JV+yl400PTrDVdJ8K/DCP4e/A3QPC+h6V/wiHh/wASDSNF0X4XftU3D67Mvibxvrlv4H19PC37Qo8F+KtK8XZ8Z+ED4k8TiD87wmErLEcPU4TzGl7VVK9epUxGaVOeVCvU5YP6/l8aNFVI4dKtSpVMHKtTxDTpSprDTn9vh8RTlGUpRouzcYKEaCteO9qNVuUUpcynaqlKEVzK84y+cviP+014RspvG1/8OrHwl4z8Ua7+0x+0V4z8Ia/4qtPiRp+oeE/DPjvwX4C0HQ/Gehk6/wCEPDf/ABVONc2nxX/wlykaEflJ4X7PIuFc0q1IOU6uGhSwmFjKNL6nKlWmnW5sPN+w9vy4dcrg6f1ek1XfsoKSnGPw089wFF3UIyvUbUpKXMo6atfD7+7u5STitr3l9a+BviT8C7XwVL4Wi0n9jqTxd4B0fwBpiaraw/8ACR+OdD8SAeJn8Rx+CviX8cv2pfhLcNoHhS3Gh+FvBHiH4YftD+MX8HS6Br0MDReEG8HMnwXE/D+Z0pUazxGcTU489NYulCjGVN+0/jYfBYJ1KVR04U6saOOoYeso16dK0MXCvh6H22U8YYClF0/ZYaNm4y5XdprkScZ1KyjKLnK16c6iUYznKMqSjKXzV+z38RPBXw38R/EO88ReMvh38LtJ8E/EP4fa74Y8K+B/GelaVYeO0fV9e8Qf2P4j+JHgX4d/tVfFPx14BK6Joav4Sfxf4p8IxSNsfxkm3welfY8U4LMMUsLOhLF4qtOjTip1qSp1vZzovmr1cJWxOU0sNipqXLNKNDEUnUlSVGFNThD5rJsTgMNRq+0hCEYuTtCXNFtS+GNSMa7lFNXTblGUY3Um2nLsvCfxi+Evgj40fDG5PjvT/in4Zg+EHij/AIRHUviD4k0TxovgTxOzfEHwS3g3W/if48+DXwo+JC+AH+H2lSDwh8M/FXg8eDp/G+u7mZlWJ/GPnLg/NcZgJyrQqYOc6+sYxp0XWp6VpVI0qGPx1GlVqVatSnWrKr7apGDb5WqU6Xp5Xxfl+GxfLaFVRVr+/Lb3Lc0qdKTioxjKEGuWCai7tSRf8VfGn4Zal8G/EFnF8WUa4uf2evAVnH8BLP45/ETUvhFpupvH4EWTwTofwwb4fHw8SDGoYr+0R4t8ZbZvEKDcEldZw/C2YrMVVeFjJKyWOeGwarJ/V+VzliFj/rDu4qk+TAQXLUjDldNTnDz6nF2CnTUFLlve9Lmm1L34NKMPZKMbOTl71SV3Gb0bhzeh/Cf9qn4Vz+HLbwvF8VPEnhzw9H4C17xHd6D/AG58ePBkng//AIQjRtbTw74P0PW/EP8AwUrOWuYtb0RfCXhX4WvPF4uFvbxr/wAIv/wirv4K8LM+Dczo1J1HTo1ZV69SmqjWHnP99KPLWqewyGbpurFzWIlW9rRpRledeTmpx+myvijLpxgrOKjGDaUZpO28VzYnZNqUHGzlK9lC1pfjv471XQda8XeI9Y8M6Xb6Lol9qd3caRYWqaxFHDYbiIoli8ReKfGGsRlV+9Hd+KdXKPvWO+eEIU/ojJMHjKEYKdapVlTjGLq1ORTqtRV5zVKFOmpzfvS5KcYXb5KUFyxPxPPc2wlS9oRjfor2j5K+ttl7zk311bOf0TRNd8T61p/h/wANaVqPiDX9UultdM0TRLSa+1C8d/upFY26vNI4GTmNHYKN+wBGets4xVShFyqSSjBNynLlSUYq7k5N2SSTbeiW7b+z5WXVKNRvkT16K+n/AA/l8k73j+9n/BP/APYF+Pnw58Rx/Enxd8VfF3wgXUrGOC98B/DrUtB1KfWDPb+ZDF4tk1u18S+DJbnw9qAcJaQeG/EFtMk4vdH13SmN5b3f878dcVZRjoOFHCYWc7NxlXhJwjLSN7R5ZXlTdWCnCrCUZNPWMpuP7/wTwnnOF0niKvL9q07XW+7bW6Taad7bq943vjx/wTt+P+i/EOw+IWgeNtZ+Nulz69a3Or6nrmpJ/wALC0jTBexmSW+/tG6+y69bWyzSvJLoMltfyqrQad4ZtLW3jC+74d+J2QZTyUK+EjhpJRhTjOMfY2Sm1CEkuWPLCCTU1CKc4Qg5SaRw8eeHWeZq3OGJlNPdqVnutX7zbu305m+Vt2R+x3/Cn/EH/RevjP8A+Bvw2/8Ana/59sjb/PH9p0P+gbBf+CX/APLPP+rM/eP7JxP/AD9r/f8A/a/1br9o/wCFP+IP+i9fGf8A8Dfht/8AO1/z7ZG0/tOh/wBA2C/8Ev8A+Wef9WYf2Tif+ftf7/8A7X+rdftH/Cn/ABB/0Xr4z/8Agb8Nv/na/wCfbI2n9p0P+gbBf+CX/wDLPP8AqzD+ycT/AM/a/wB//wBr/Vuv2j/hT/iD/ovXxn/8Dfht/wDO1/z7ZG0/tOh/0DYL/wAEv/5Z5/1Zh/ZOJ/5+1/v/APtf6t1+0w/B/wAQbgf+F8fGXHqb34a//O29u+PxxlqhmdH2T5cPgnH+Z0rvt1lF69NI+jv7uDynE+3i/b1nZrTn3tbpyN9F+t7py+d/2q/hnrmk/AL4m6pP8YPirrsVr4al3aZr134LGnXQVgNrDQvhysuWyu7bGRlfmxlC30nB2a4eGaUnLDYWaaqXqxppVYe5pyyeqUnKz5YybTs0o804+NxblWJqYL3a89OVct3Zu/lZJxS0i7LS/Np738vVf6ETfJp/wP8A5K34/I/hSCu79v1v5r8n8grhngIYl3aTu/672/PTpqzthmFTD6RbXzv5/wAq/rSztcK6KmJwPOqCpRSlo1b8tvv/AAW8caeFx8Y+29rJuOu76fcttN+/duJV1Mqy3DRWIhSim1d6Wbb77d+vbrsZ082zDEydCdSUox0Sv/w7699u5vzeGdai8LW3jE2oGhXet3/hu21AzYY6lpmm2mpyxBX++RbajbyAKGDLndjYyryLM8tqSlSjGHtIxhOUeVOSjUc1CTulpJ05pd3CVrWtE/s/MIfbk09vettvr7+119/T7VD+y9Qg06DV2srn+x7m6ktLfVPsz/Y/t0SRSSWRkyUWRYr2JyrENtYONyhlpqWBndqnF91pp0W6W9n1e3TQLY+n/wAvJafO1/mvwT+Vj3T4SfCXxP488M+LNS07xReeDtNvdW8P+AtNA0Dx34k07xz4o1gy67D4Lk0/4daD4r8TvLbR6Hb67IkPhTV4YrhNLk+zwzXNtd2XyPG3EmScLypr6nDEVqaqYmMIPDwq0qcYOlPEReInShBJVvYuTqwcoVKiV0pxl7/C+QZxxK1GWIkk2k23Jrf4fdUpP4b2SlZ2teyYn7QfwBv/AIB+IYPD2o+I7TxBLPIsdvbnwh8RfCWoNa+VITrCyeLvCmjeFtV0x7iN7WKbwn4r8VLHcDbJKUV5qOCOKMp4tdaLwE6KozjB1J1MFWpznZucaf1XFV6kHTXK5RxFOhK1SDjGV5qHocQ8P5twzGDhiXJzSbS9onFW0vzxgmnfTkck7Nc2qZ8+V9nUq4DDy9hGnFJ72X3dUlfyT37o+S9jmP8Az+l94VpOGXYaKrxpRUnZ37f+Spfi/Low9jmP/P6X3hV1VGhlWWWXL+8l6fxFbt69O2hoqc6+aZmm7/u1v1/du/8ANfe1rr8bn6P/APBPDwPqvxK1X4veEtF8deKPhpf32g+EZP8AhJvB0rw6hbKmoeId6xSxSwyRtJHviJinhkMMkyRzxbvMX8C8fcXFY/ATcOfkpOShdRu1CNlzSulrZ7Kz25nofs/gXg+XLMyV7fvI3dr2/eS9L/f/APa+6fBz9gb45/Br9qr4ea3qGsajqHg+aXxbHP8AFjwRNpVrqelSDwl4gtoDrHhvxCuu2+jL4gaOLToYbZPFVtbR6mLe4uYZgtxB8xxb4rcPcT8P144fKcLHFyp0G8NWvUoRq89GVS04qg6saDu4VFyOVSkuWMb2l6vCfAGfcPZpQeKxNWVPnfvxak2knppKVr25ZKTfKpXkmk+b6p/aq+HPxa12HS9A8CeM/wBqfWbrwT4V8T+Jtc1/4df8K38N6hJqGtnW18O6VpfjWTwv4P8AFGvXE2rC3l8beEvhZ4sLW3gvwNFanwXP448UeCrif8myLN8DQxNWdXC5dGlXlhqE6VanWnhatCGJpOrKthpTnR51BTnhqtWE3SqTpwqVIwhN1/3HOsH9ZwNOM3UlOEW046SUuXRKSfNv8VpRule0mkoT/sk+FPGeq+HNc8K638R/2itBvrK+/tKa68SaX4qvtR1xfmh/tnVPif8AF39jD4A+I/EPiMMNN07yZJfFR/4RXw94eXzVgLlr4ix2FjKNWOGy+T9j7FuEaNOnShGXuqlh6GbZlRUFH21Nxi6TjU5p8zhOpTPjuEsFjKsmnVr6y505SlK76uU5UqMm2+Srfld6bjdKaXL9lSfCDxAFBX48/GVvre/DY/Xj/hWufXv054wRXyGGzKndtYTB1Et+eg3bsmlO+vrq+ybPvsdluJUNMRVhbqp21+f569+qUfyZ/wCChf7BXx1+LZ0/x34N+MHjP4ny+GrVmtvhl8StV0WPTB8xEeseCh4d0Pwz4W0HxIkapG0nivwqwkYbh40hyIk/VPD7jLKMBWSeFw0alSUXKVCl7LV3upqUqrlaycailKUlU1p0lHmn+Ycd8L51jKEpSxNSVNRfKnNSbStforX3a5HtvtI/no8TeGPEvgzX9Q8K+KtF1DQNf0u7a11PRNctJrG+tHHG2SKZFlRgMNh1UlWBwFYNX9OYLNKWeYenPDKPJKMXFxs004qzTV7q1rWunveNry/mrG0KmR15qs3J81ndt/n/AF07GPXp4/L6VFaRXn/Vn9/Mu9tLHPDH1ars5N/Psv61tLppuFedQwFKo7uK9LdtP8/13PRp46rRVlJpPz7W6el+2vf7OhPrerXGk6doNxqV9daNpV5qV/YaaZy1laahrUVlDql9HZYIWW7g0rTIp3UBpE0+1ViyxQqvbQo5LSbmsPTTsk2o6tK7im+W7ScpNJqWrdnrI58Rise2v3skuvvbbW+/d6dtzpfBPgLX/iDrMmheGZvC41VbU3VrF4l8b+EPAdlfHIVYLXWPG/iHw5pE+oSM37rS1u/tUyLJJCjJDNInmZpnGR4Om6k8PKUIuKaoYaviKj5mknGlh6NWrJXtdqk0lq7JOUvRyzEZhOSgqlm+s6sIR011lUlTguul1d2Vm3yy9k+Ln7L3i34Vf2tqi+Ifh94j8H6cgey1a3+K/wAC4tT10MF0PXP7G8D6J8Q/FXibXH8JeIm1rwt5cjJID4fdCgdHSvleFeMcsxsPqqoYnnnFuMpYDHQo/Dd82Inh44eEmr2UqqbfuxXN7p9LxHwvjsHTdVVI2WitVg5dVpFT5918XLJbPms0fM1fY5bhsuxeJlJ0o+897LTy6/8ADdrtnw+ExeY4STSrSXz/AK+/Xb5xKeYywNbFwvTi9eyVtvJrX19LXbNsto46hhJWqSVl3/4f8vLT4grtzCWBo4SDVOO3ZeXkr73vbzV7HFl8sfUxk06snrt369Jad94+gssvkxefP/n+fPHJI/A4G7DCRp8NU5VK8U0k27pO/TzXqunZ7y2WOqcSVVToScW3pZ//ALN3ft80rWl+k37G/wCwV+0H8TfFfhr4kHV/FHwA8M6XffadP8YWiXunfFC+RLVZoh4S0iVoU8OCYStbS674ogeG0EkM1r4Y8R6RcXqW/wCN8eeIOQVpToVsJRqt+5OFSK5JRk0pqXuTTTjzPlcbS+G6TbP2PhHgjPqcIVKOJqxTtK6lrF9Ptaff6OX2f1N/bB/YG+NfxZvpPHHhH4u+I/iReRW+LXwN8RNQ0qy+zMREZD4ZvtDtPD/g/T5bmV52ksX8OaFZkIslzqs5SNW+C8P/ABDyPJMPLD4nAwhCpBSWJo024TkoTTdalaVVTUIUaakp15VJSu1CMfe+w494EzrNMXCpRxU5JTtyuUfdV07R96KevNdLlUbXbfNaP1J+zL8DPHOgfAr4ZaX4g+JXxb8Ba/pnhi3ttW8KWd54Lay0a/yQ2lKNc+HUkjpEWGz92oLMwB6FvkeL+J8DmecYmtQwWXKjOdN067oy9vUXsoc0p1IzXNyNcsPdi+TkSckry+t4R4WzTK8vpQq4qvzcsrx57pa6e403rvK0pLSz5Grnvo+D3iDt8ePjL6/8ffw2H/vN8f59q+Y/tinPRYLLX/hwzv8AhU1/ruz6ZZHioa+3rv8A7ea29W1vpv20luc/oen+KvB/xq8PeD5fiJ4y8Y6N4i+GfxG8S30XixPB/wDoGqeGvFPwo03SDp3/AAi/hjwu6RyWvivXROk32iMnys4KxFXWxVOvhlOWHoxn7ZaUr0/d5Z7353v5+vLZII4TFU5uKqzso2u3d/Z6x0fXppory15f/9X+/bYvp+tAHL+IPCmjeIrOe01KzhuYJYfJmiki4nwQwOQNwcMoKOoJVhuUghHrZVZdXu/u/K/fb7gPxQ/an/4JHeD/ABa+oeKvgTeW3w08SbWuT4UMMj/DvXJsWsUcX2VC9x4MiWOJ1WTw5Fc6NZREpbeFZbmR56/ceFPGDE5fUpU8Rdpys5TaTs7vdRs7JpapXWrvJH4bxV4RYfMKE54aSnJRvaPVrZP8+m1ul5fgh8U/gx8Svgd4ifwx8UfCOpeFtUVnW1vbiN20rVxG7xO2k6zB5+k6okboRK9heTiFvllKyFVT+hsrz/KOL6MKlavBNxTs5W1aT7pq3pfyjsfz5meSZvwjWlCjRm0m0nyvo/uV35f+A2PN69nCSrU7ta29Pu6rbz6aXseNia1bTmpv9fx+XT7rBUY3NcTTVlF6afl5Nb+fpJK5vgsTbR6ef9W7fdrrsFaiCgAoAKACu6tl8MzV20reasr/AD/N/wDbyt72UMTPLLJRfvb9f0Xyt9yD/Vf5znP/AHz/AE6d814lbEYbIX71SKs9dfu7+m/3nZDL8RxArqEvuvb8F+FvPmtY/Rj9m3/gmp8aPjlNpuueOI7r4SfDy4kKTHULCGTx9rkK28UyxaDoVzLDH4cEpdoDrvim2kbTpxbXen+Gtf06W4Nr+Z8ReJ+Hq0Zxg1OVo31cUouVm+ZKSlKMVKahG7fKk3Hmi5fp+ScBV6FaDlTlvvbt0W3pfp52uf0P/s5/sYfCD9nnRVsfBXhi3sb6e3SHUvEFyovfEmr7ZnmUahqzok6xiSTzBp9lHY6NDKvmWun2crEp/POf8Ze3rTcZJXl5N7W030b1XVN/bikj92yThqVGlC65bLW/ffv6pfD+DZ9nWtpBZxCKCMBe+Mdfxx39ufbA2/DH3hboAKACgAoAKAGj+H/d/wDiaiH8H5fpIJfH/wCBfmj5k/bC/wCTaPi7/wBidq38hXucJ/8AIzo+tT/02z5viz/cv+3n+UT+RSv9Lz/PsKACgCxpdxDY6hYXtxYwX9vaXAuLm0n+5dqOqNhlIXAxgZyOoOQGAP0C1j9qfwJqWveG7648WfHfWrHR/G/xU8X2N1q1sPD2oeCNL8eeCG8P6D4N8GnwN8YPDPiNfD3hXd86+F/FHwoQAkFfCeSrfzhDgLGvD46nPB5fTqVqdKnB0akprFzpubeIxcsXl+I9nOo535atLNJU5c7nXxEvfj+2f630FOk1Um4xacrxs4axvGmoVYOSWtmp0L6aU/tdBcfth+CdW1iC4u/Gn7QOlafp2vvq9tpeg3nizTdM15ZPhtofgdb/AFuVP2hIfE72tv4j0dvEkJf4r72TXVkm8avM8krNcA4pYJUvqWCqYp4SVBzqTw8eXmrRqKgqkMl9i8PTU5zi3l/K6lOHPhZSlUqxP9cKXtF+9nGmqkZe6pX0STqcjr39o7PT2vWyqRSsV9N/ag8CeIPiFc+ItZ8WeIrfwxonjT4Q+LdK0vxGnxDstRvf+EK+GPj7w/441nRI/wC3Pit4h0TXfE/iE6Iv/CTHxV4u8YyIuJPGrFizdEPDzMcPkkaFSnT9q8uxuEqThHC4hU51sRhvqiV8HgFWw+Goe3lKjClh6cox5Z0q1V0pnPmnHmHxWd0q8HeEa9Kp8U4XjFe+r883GUnb3uZuLd1KKT5fLP2uvid4G+LN74G1Xwd4itdRfQ9G1PR7zR/+E3+PPxU1ZAdYGurqsvjf4t/D/wAM+Im0OMDZJ4aikELsMxqE2hfR8GOHMZkeZV6mMpTpzdRSTnSw1HeHLyOGGr4iDlBx96XPGL5oxhzqPu4eI/EVHOMppU6Uubljy/HKW0r3TlCMlF3uk05dW4ttS+MK/dT8iCgAqcJ/Ezz/ALl//bRVf4cfX/29H64f8ElP+SpfEf8A7APhjr6fbfEP9Pwx/s5r+W/H7+HkX/c1/wC2n9JeCP8AEj6f+2I/oYr+XT+lwoAKACgClPFFKp82IEEZ2nBPpx938OV+hz8s1HyyjLm2e/3adfzffSzUdKTbTjtfo/n89+36o+Qf2jv2N/hD+0dobWHjbwxb3F/FAYtN8Q6cBY+JNHBkjmxp+rBXnaNpEEhsL9NQ0WaVjJd6bdShK+6yDiqrlcoJtOntZ2vtJK/Sybu/hnLlXvcqly/BcS8MfX3KUU3d309e3lturt9bpx/nC/af/wCCdHxm/Z8mvPEGiWk3xM+HcR89tc0OGIeI9DtD9okePxFojRxrcRxQRKZdY0NbuxkSO51HV9O0a2eGyT+hsg8S8PjJxg5ct3ZKVr9Oi9eXd7dfs/gOfeGmIwkJTUHKyvpf/N/gl2V7H57wy+cP3HH/ANf69fwI/Div1LD4nDZrTjNVF7yT337af8DW3S6Py6vh8TldSUHTkknbbt308v5tO8rWCvReGjk0HytX/rrrfR6aeqT0MZ45uSun80+/na3rd7baI9//AGbfiXpHwu+IkniDWLXQLdH8L+MrHTPGWo2viufUvBV9qvhHWbLT9Y0OLw9Ldq2opPcLbp9p8JeKCftGyKwRTO7fD51gZ55Jcs503GcZNwUHzxUrum+aElyzXuycVCaWsJwdpR+kyrExau7PTrdW8927q3969tU07R9p+MnxH0rxd8JNF8LW/wAXf2btb1Dwja6y2tW3gn4Bf8IXf+KNV1zxoNdH/CE6w37O/hMeHhIBnxksXizwgPGP/FUlv+EuZiG+WyDIJ0cXhqv1PNYupLEc8quYKrQw3KnZ18PLM6sJRxFv9nWGo1/ZXXtVhmpcn02c55CrTqR9pR0VO3LT5ZTVrvlkqK1g21Uc3Fy6OrZM+F6/V6eE9nBfLt5fPTbf7tD4j637STt1b/D8dvL77hVrCUMsi5Smlyq+/wDm/Tpr1tozj56uZSUIwbv5P7vs+n4aXcpe3/Av9nL4v/tE63/ZXwv8LS3umwXS2+s+LtVP9neDtCR45Zt2p65cRtBcTrGiyHSNIh1fWpLeSO6ttFurYs6/HZ1xrg8FzKVRe7tGPvTl70IrljFNv3pR5n8MU7yai24/W5RwNisTGMlTaTWj6O+rvo+ifTXaytc/om/ZK/4JjfDT4MPpfi/xfaxfEL4kWZivx4j12ws5rLQNTFvEm/wjpsyXEWlS20one01dpp9aiMkv2O+sbe5uLJv5+4u4/jmKdOCjytWu2721u99LrlSutH/MmpH7twhwLUy6cak00k009rPs9V6J636ctkz9X9I0Sx0mHybaIAZ5Ofbjt7fn6/er8dqYx15yd7Xbfe/4rp/d1t0t7v7DChGlCEN7K3b/AID/AK8jcrEsKACgDwfWv+TjfAH/AGRn4zf+pt8BK71/ukf+vsf/AEiR42K/3iXp/kf/1v7+K5/af3/wAKPaf3/wAgQgrgvuGcggYx/M/of+A5FV7WlirKk2pJ3T2tb5vW2t236dY504VKEWqkr2XVbd+m3W1teyueSfFL4K/D34weG73wp428NaR4j0PUQEutM1WzivbOUAY+aKdGj6ZI3KwBAOCTsr6DK89zXKP93rzS0VlO2n/A6Ppey3Z4mZZHlecSvXo05yd23y/q9t77S7W2PwA/ai/wCCR3iLws1/4p/Z01FdW09VM7/DHxZe4vAR9ljS28JeL5l2EBRN5Gn+LtrEh7jUfGGGW3r+guGvFxY2UadZxlKTs5QvGNruz5ZSk1aPKpWlrLW8YtH4PxHwFSwlOUoxtZaJ7y0+dur69PibsfjR4h8Pa94M1+/8K+KtE1bw9r+msY9S0TW7SaxvrRx/C8U8YlQ4w2HVS2d3I2mv2fA5zhMxpQqWUlKMZJqzTT1un2d7p3afla8vxrGYFYOrKOiadrf1f+traox69U4woAKACgD6n/Z3/Y5+Nn7Sd7BJ4R8OyaH4M+1Bbvx74ojl0zQ9geISNppuVM+teV5u2dNGt9SNrIhS9+zbWdPz7iHjd8P03FT9/kuoRfvtWk02vsxfJJKUmotq102on2+ScMwz+tFNXd1bp19H6fDH56H9D/7K3/BNz4TfANbLxDLZ/wDCb/EK3KSf8J14rtbaS506WPz9j+ENFWKbS/DaL5zNaXgj1PxDaBpbZvEd9ZsI6/nziHxIxOcVakHWk46pwWiknyu09VKVuVrljKMZQnKMk7Jn7tkXANDLKUW6cNUndpP7tujXn1V7JH6YaZomn6dD5MES47nn8Ox/X8jX5rXxlevvUkvXVr+vl89Ufp9PAYantTV+9uumunbpf/M1WC5649sZx/nrXkVMHOs7t31vdu3rpp59X362O1VIwVnZJf12aXTo/lpyy12iCgAoAKACgAoAaP4f93/4moh/B+X6SCXx/wDgX5o+ZP2wv+TaPi7/ANidq38hXucJ/wDIzo+tT/02z5viz/cv+3n+UT+RSv8AS8/z7CgAoAKAClyUv+fa+/0/u+X4va/u63q9398Qo5KX/Ptff6f3fL8Xtf3S9Xu/viFc9TF06k1Qskpbq1/0Te//AA17R56eFquHtm3zRV7v9d7PTt5K9goqYSnl0ViKcVF76aN3t2v6bv8A7dtcini6mLn7Cbcoxdkr6adNlfS3fyewV0mwUAFThP4mef8Acv8A+2iq/wAOPr/7ej9cP+CSn/JUviP/ANgHwx19PtviH+n4Y/2c1/Lfj9/DyL/ua/8AbT+kvBH+JH0/9sR/QxX8un9LhQAUAFAEf3/Yj8f/AIj+R/CpaU1rqn/X9f8ABHrF+a/rz/LTzsJsYHj8xx/Vv5/n/ClFqyUnZbeX5/8AA8tiuZPeKb9f+BH8vuv72TqGkWOpw+TcRBlxwccjP4gH68/0rvw2Mr4CSqU6jS827Lt/TenaN1zceIw9DMacoVIx23tZ/f72i8r+S0bl+UH7WH/BLb4U/Gie+8YeA/L+GPxFvZmvb3WdDtoovDXiO6W2khCeMtHitJlJnuPLku/Emj28GsiQve6tBr8dlZaM36fkHiTjcDT9nKo6ijz2W0pNtPljzOyteUUpOKS9nFSspTPzDPfDbC4uo5xhFJtPyXZ93e6d7d2lK6jH+dj45/s8fF79nfxA2ifFDwxe2NpcXJttI8W2qfbfB+tBYUuCNN11I/scd0YGM76Tfix1m3gV7i80i2gKtX9HZLxP/rPJQjNPvH7STlKN2t1GUoSUZbPldrpXj/NuaZAsNFtqzXXpoum3e9ve87XSPEK+2WCjgYqVlrv13/Ff1dy3j83TxEsNLku9H5909tl97721tErnwlOFNaLT0v3Xk912+6w8RVnUly3eu+t72s9du/T73ua/hnwz4k8Za/Y+FfB+iat4h8R6k23TdE0S0lvb25J7JFAJJXIGSVSJuFzhQHC44/MIUITqVJxp06cXOc5uMIQhCPNOc5SajGMYq7lJwSSu91y+nl2C9olvdvT1fRabu/d2btrpI/bD9lX/AIJJ6jqz2Hiz9o+4jSTzo7yL4W+G9QkltIFgllbZ4v8AG2lXVr9qmkiMMdxo/hZIbC0u7dLi18X63ptzPYN/PXE3iZXlSlCNRq8XqnZp2VrJq3V6t6WslZNn7dwzwDSVWMpQTd1vv+WnTsrdXofvp8P/AIU+Dvh3odh4f8M6HpWi6Nplslrp2maRZx2Gn2cS8BYbGFEgj7tiNVG4lsEs5b8LzLivE5hVk5VJWcn1s2uyetl1d399j9xwHD+GwVKMfZxbS2sml99/P520Wx6gdqjBJ9ew6+ucc/h+WTXgqlVqWc5O3m76dt1r8nt0sj2ZVY0ltFbaL/h0l5+876J7piqyjpxn8f6t/P8AA5yukKDhre76r5f1+olWjPq3b5/1638tSSqLCgAoA8H1r/k43wB/2Rn4zf8AqbfASu9f7pH/AK+x/wDSJHjYr/eJen+R/9f+/igAoAKACgClPFFKgjli+U8FRg/lgjrzjr+GCtc9a8ZqSd1fR29Px+752ua07uLvs9LPp90tej0cezR8oftB/sffCD9obRTp3jvwlY6hcxIsWm63bKttrukJ50czrY3+JGCO0aeZDMs1rLgedaz7EK/ccP8AFVXLnCzThGLUoSSd5ctk72TWuu70Vux8DxJwx/aDlKC5ru91fRXvtpa2q7t662sfzt/tJ/8ABMP4zfBZ9S1/4dJf/FfwXbSGWGzsrRX+IekabHaGZjqOk27KPEs5ljMMVz4YtUvLyaXe2haPZwidv6H4e8VqGKqxjKcVzO2+n36rf06pdGfhOfeHOIp03JU5Oy+/b12a/wAm9JH5n1+xYPNMPmlOE1KOqT3ve/8A4D+XX7P2vy+tkeIy+bjKLTv22X49v87XueofCL4IfFH49eI08P8Awv8ACWoa/JE8cepatsNtoGkrNII0fVNWcCzsA7OojN1NCJ3Hlx73JRvms0zzJ+EKLnRrwbs3Fcyu2o3slq72TenNovhVjryzJM3zWvGFehUSbSd4tW876pWXm/Jq/u/vd+yx/wAEm/Angr7F4q+N09r8UfFRzcDQ5LZh8O9JZvtMbRppMuyfxUHjmVHk8QiDSL+LH2nw1a3Eaz1/PfFfi5icyrVKWF5nG9rwtzWvo02nbS8dFdbqUXFH9A8MeEuFrUI1cXJQbSupX3te26bemiT187M/Z/w74R0XQbWC3sbC3thbcAgDjjHTJ5787R2GOtfkGIxUptpXcX8NlZ6d3v07Pyv9n9kpYdprpa123fm9F5X/ALu6TevMdmRnivFrLW/b/gHeFdIBQAUAFABQAUAFABQBFg7Pxz07dPX8c+nbvWeJ1tbpp31/Dv57dBUNEk+q/wAvvvbTbe+u0vnf9qjw9qXif4DfErQ9Hs5r7U9T8Oz2dtYQJ5kk7zlImUIrbjiJ2l2oVeTytijc5r3cjrKhiaUnblVOau3ZrTSy1u+lm9E29be94Ge4d14OzbfNHRLR6NPq973+XW15fzO/8Mg/tIbc/wDCp/EHX739peHMY+n9sYz/AMDx71/c3/EUfDn/AKHT/wDCat5f3fP8Hvb3v44/4hn4jf8AQmX/AIU0Q/4ZB/aQ25/4VP4g6/e/tLw5jH0/tjGf+B496P8AiKPhz/0On/4TVvL+75/g97e8f8Qz8Rv+hMv/AApoh/wyD+0htz/wqfxB1+9/aXhzGPp/bGM/8Dx70f8AEUfDn/odP/wmreX93z/B7294/wCIZ+I3/QmX/hTRD/hkH9pDbn/hU/iDr97+0vDmMfT+2MZ/4Hj3o/4ij4c/9Dp/+E1by/u+f4Pe3vH/ABDPxG/6Ey/8KaIf8Mg/tIbc/wDCp/EHX739peHMY+n9sYz/AMDx70f8RR8Of+h0/wDwmreX93z/AAe9veP+IZ+I3/QmX/hTRD/hkH9pDbn/AIVP4g6/e/tLw5jH0/tjGf8AgePej/iKPhz/ANDp/wDhNW8v7vn+D3t7x/xDPxG/6Ey/8KaIf8Mg/tIbc/8ACp/EHX739peHMY+n9sYz/wADx70f8RR8Of8AodP/AMJq3l/d8/we9veP+IZ+I3/QmX/hTRD/AIZB/aQ25/4VP4g6/e/tLw5jH0/tjGf+B496P+Io+HP/AEOn/wCE1by/u+f4Pe3vH/EM/Eb/AKEy/wDCmiH/AAyD+0htz/wqfxB1+9/aXhzGPp/bGM/8Dx70f8RR8Of+h0//AAmreX93z/B7294/4hn4jf8AQmX/AIU0Q/4ZB/aQ25/4VP4g6/e/tLw5jH0/tjGf+B496P8AiKPhz/0On/4TVvL+75/g97e8f8Qz8Rv+hMv/AApoh/wyD+0htz/wqfxB1+9/aXhzGPp/bGM/8Dx70qniXlFSsouvBX01l36dL7dVr5WSOleFmKVNzjTlZav/AIPo/KNu2x+k/wDwTP8Agv8AFD4UfEPxrd+PPCWoaBDqmgWsFnNdRtGt1/xM7Z2xlQMYgkbPONo3Y3pt/BPFvNstzmipUa9Opq03GalZ2u1pez1V03f1uz9q8KOG6uU1mqkHGyW68rdlbRW29L6n7kV/Ph+6BQAUAFABQAUAFABQB5p8Q/hd4P8AiVoGoeHPFvh/SPEGj6nbNb3umarZxXdncxnJ2yQzJJHIPYoSDhgQVFetl2ZywNWM1spJu65r2tp0duunLt105fGzHLPrdKUU9eVrlvvp001/Pty7n4AftU/8Ej9T0VtQ8W/s3XBltlinvX+FOvXcbMZI4YjFH4R8Z6rd2ybHlRoYtL8V75VV5r668W3EyW+mS/0Dwd4mwwdJUqvL7qUVUWjl71TSdO75OWCpXkpyU5zk0qUFE/C+KPD2ri6rlGMnzSb201a23b1va+qVr2vY+SP2a/8AgnB8ZfjtqEeqeKo5fhj4Bsrqe0mvrzZc+JNdm0PW00zVdK0DREjf7HgQ30aa5rcVpp8DrDfaRp+twRXGmp9Hxdx5RyWuqaleS+Jq1oc0W1dve9rWjzSV4tqMJOUfmeEeCa8qN+Vr5X6paaq+y6/ef0Z/s5/sd/Cb9nvQ/wCz/Bvhq2sLiWPy9S1ufbc65rCebJKq31+Io8opkYRwRJBbQgsYbWLLK/4DxZxVU4ibi2nCSXIoxSUGuW7ve8rvWyWj096ykfuvCnDjwUnKUWrJPX7T/K2vfzSlZ8v2Ha2kFnEIoIwF74x1/HHf259sDb8GffFugAoAKACgAoAKAPB9a/5ON8Af9kZ+M3/qbfASu9f7pH/r7H/0iR42K/3iXp/kf//Q/v4oAKACgAoAaQGH8uPTPuv8/wAs/NlpVXZL57/da3/b1/7v2qTcXt+P/Al+X3291AhHQ4/D/wCyb/Pr/DHspRd4y+7Tvvdr8n8rjc091f5//ax/rt9rOu9NstRhMNxGrDGQcY/lzjPGOM/mG68G6uAmp0pvTbV/5vt2l6LVHn16OHx0JRlBXa/lX+av23v6n5jfta/sN/Av4j+JPh94q1nwXYrrur/Erw7oOsalp013ot1q+m6i0+bPVZNMltZtRj+VlS5llF1Zou20uoEaRX/WeHfEHH4KnUp1Kkp+yoKVN6NuUVJSu1Gzu4xsmk9W+Zq6j+YZ9wdh61TnUIq83fTXXls3dL+bRvs7RSd4/cXwx+C3gX4XeHrHw74S0DSfD+jacu210rRLRLCytxjOI4YVWNBkcqqcnJ4yWb4HNc8zTNVy4qvUs9NZX+7fot/y0R95l+S5VF81LD01KL2SWm3k+m1vxtaXsCpGqYiUKM8qcn+ZP9MfiBXz/NTw0f3jtd35tfe6W6/c3Z+dlze46VSiuWirbPl6adtrW9G+r5rpD1fnIzx1Gfr7HPr0/Pk1h7WWl3ZPrv8Aonv/AMC1rR6XBW/r8rr8/PTYmYkYA6n+n1B/mPxqkr6GQ6twCgAoAKACgAoAKACgA/z/AJ/z/Ohq+gFS4ihuojFMMg9s8n9AffqPrxlhV/YyVn+fk/6V16qwpUVUS5ldd9P6/B/LaXO/8I3pH/Pun/fxf8+ta/Xp/wDQa/8AwCn5f3fL8Xvb3suSX/Qsj/4USI/+EY0j/ngn/fa/59aPr0/+g1/+AU/L+75fi97e9Xs5f9C6P/hRL/5H1/4NveP+EY0j/ngn/fa/59aPr0/+g1/+AU/L+75fi97e8ezl/wBC6P8A4US/+R9f+Db3j/hGNI/54J/32v8An1o+vT/6DX/4BT8v7vl+L3t7x7OX/Quj/wCFEv8A5H1/4NveP+EY0j/ngn/fa/59aPr0/wDoNf8A4BT8v7vl+L3t7x7OX/Quj/4US/8AkfX/AINveP8AhGNI/wCeCf8Afa/59aPr0/8AoNf/AIBT8v7vl+L3t7x7OX/Quj/4US/+R9f+Db3j/hGNI/54J/32v+fWj69P/oNf/gFPy/u+X4ve3vHs5f8AQuj/AOFEv/kfX/g294/4RjSP+eCf99r/AJ9aPr0/+g1/+AU/L+75fi97e8ezl/0Lo/8AhRL/AOR9f+Db3j/hGNI/54J/32v+fWj69P8A6DX/AOAU/L+75fi97e8ezl/0Lo/+FEv/AJH1/wCDb3j/AIRjSP8Angn/AH2v+fWj69P/AKDX/wCAU/L+75fi97e8ezl/0Lo/+FEv/kfX/g2953/CLaR8p+zj5jj6c/8A6+35Vy4rFZosTTlSxM2nb7VrWtvr8vxtrY5sHgsLPDuMqMZcyd2klbtbbt2+5s0bXRNNs3aWCBVfGFPUD35/Dr+HQ12Y6risdSjTqVZNpa8zv8t2nr2Vn15bDwGBp4OtUlSXLe/S1vT4vLZ+eqfu71YHYFABQAUAFABQAUAFADDlgCOP8kcH5ev0/Lq2VeLSum/dfTTf5v8AN/LTlcJLdrR/Pr6K+3ZfP7Va4SKVAswBHvx16d/Q9cd8cn7tYOdda87S+96dml02+TtzWbjy4t0be+k+1/Lvu7dbefS75fA/2bbC0g+GfmQxKjDx/wDGIDHBz/wuHx4Tzz3zgj6gDOK9/i3EVsyx+IXPK0pULavRU6FOHk+n/BdrnzXB2GpLAxqW953TbX9+Wve65V1s/wC7oz6AYLtIye3PAx0/3f8APQjGW+bnzYShzcztFpO72T7b9bbb31tax9dCcYVLWUX16L03+W7+d0WK6ACgAoAKACgAoAKAPB9a/wCTjfAH/ZGfjN/6m3wErvX+6R/6+x/9IkeNiv8AeJen+R//0f7+KACgAoAKACgAoAKyfx/NAeC/G37/AMJv+y0eBf8A0DWq9zL9sT/2BV//AHEeFmG8f8dL/wBuPeq8g90KACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgDwT9nYbPhgrf3/H/AMYh17L8YPHXX73p1x+fO32OKcWp5piJJ6SdGKv0UKFKKd9N0rbP8ufweGMGsNl8Fv8AF/6VJu6173veX4+57sw3J79+/TOD/D9T/T7zeJKPtIJPRSs9vS/Xs/736n0CajJ/NfLvb3u3f5v7MtWSFABQAUAFABQAUAeD61/ycb4A/wCyM/Gb/wBTb4CV3r/dI/8AX2P/AKRI8bFf7xL0/wAj/9L+/igAoAKACgAoAKACsn8fzQHh3xntby6f4X/YrWWf7H8XPBl/emNA5t9Ot01UXNy5z8sUbPEJZMsEEg3D5gW9zLU2sTZ3vhK6tq9W4W/yXeztszw8yspLS1p0+2yv569+vm19r3GvIPcCgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgDkfGPjTSPAulHWtas/Fl9aLIImg8G+AvG/xE1UsT98aD4A8P8AibXmjAILSJpzRr0LAnaoB85/s7fGj4SeKdD03w18Nrv4neKdN1XWfGWvWfijU/gp8YfD3hS4m1zxh4n1nWmHjbxB8PtC8HPHb69NqunwKusPcSPDHEqTud7fQcSZfiIY2rLFUKeFqvlU6SxNGtJTjCk3/DnJq8XCajKEUoyi73fNL5fhXM4YjBqMZOaTupcs4qzlJKzktbtNXX2laytzR+tAcluvC4559MdMev8A+vGa8OS5YJef9d/67bH0UKnPUn5LT8Lv5t33087k1ZmwUAFABQAUAFABQB4PrX/JxvgD/sjPxm/9Tb4CV3r/AHSP/X2P/pEjxsV/vEvT/I//0/7+KACgAoAKACgAoAhZflVehOf1+ue3H+GcVjX/AHlSKWv6ap3+z676ba/adFezhbtbb5/1+J+Qv7dHxW13w98T9C03wzL8eb77L4RuLV9B+HfgD9pC80s6mNX0JoNZFx8Pf2hP2XvCuvW9xL4y8O+GPMXxR40mTxTKPCEyeFL6KVJ/0nhXK4YjDLmjhlU9raE688LFtRw86jp3xOBzCKdqftbRw6/2b9+pS5Zxn+ccY5hJVeX3nFK0lFTd7uKu1GpTl8LaXNP4vdbafNH7V/ZD17VtX+Dejad4im+INz4i0u51iXU5viN4J+K/gvX4Ita17Vde0fRXj+L8V54n1YeGdD1PT/DTai3irxxC0mkGFvGPiCaKXUp/muLsHQeMrPDSpTw7lCNN0amEkpulTpwlUUcJOVNKrJOpbkw796zpwvY9XgnF1J4JU6sZQlFJvnjUT1lJpPnSastNOa8VZSVrH1QMKP8AdH8/Tj9ecdMDHzfPJeypRTen9afJt9Ol9T7L45vv5+f/AID0V3r5abyfViCgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgCLB2fjnp26ev459O3es8Tra3TTvr+Hfz26CoaJJ9V/l997abb312l83/tR+GNQ8R/DG/k0Pw3qXjXX9O1Pw+mneDYZ/El1oHii31fXdL0LX9K8XeDtH+I3wu0Hx94cl8Oapq0t94V8e+K7TwJdCJLrxKj6faPt9rKKvsqtKLn7KDTcqkVSVSPLCaXJKpCpyt3s3G7irtNe8zws6oe2i3yuTSSUHz8tnJXvytO6S2ur9eb7Pyd+zl8J/iJp3xK0eH4j/CCPwrZabpHi/wAWaX4/0bwVofww8Qxazq3xOtvE4+Gura/8J/2qPi8PEHw+Q+N9Ynt/hh4rtYvBrnw/BcRnxRJ4WEy/SZ1nEMXSbjisRUcoUYezq1pV6a9lhowdSHt8LCVOtKpFzlVinUqOSqt0nX9z5HIeH54epFuEfdclzcnI3eeifLVs0ldWblZSnC81BOP6iKu0Af5/z+WeuB0r4SbVR6Pbyvp98b/J/ddc36VCPJFR7f1/X/BHUygoAKACgAoAKACgDwfWv+TjfAH/AGRn4zf+pt8BK71/ukf+vsf/AEiR42K/3iXp/kf/1P7+KACgAoAKACgAoAQnAz6f59D/AC/KsopqSut79b/1/XYD4p+NnwG+G/jW6v7n4i+LPiKdR8f+I/BfgrRW0vVNMk0vRhBN/bOheDh4PbQ5/ht4l0OLWW1fxJOfi14S8ZPJPOymXzIvCcMX12WZ3isDGP1b6upU1VqfvYSfNUacHX51espqi3Si6dSEl7Rxc5wUoQ+bzPIqWOm5T5pKTitJK9t+W3wtOST1Vly6JXXN7T8F/gzpnwQ8OzeF9D8R6prOivKsun6Zd+FfhP4QsNGGW8yHTNJ+Evw3+G+gxQzblLmfS7i5Ypua4YMEXwM2zipi6tP22s02ryqVqsrOzs516tWbta+tR26Je9z9+S5PTy+jJQ05t/djFJK9naMYq7/ux8nZts9qPG/8D/nr6/8A6ulcVe8qKa+/7kn0+787XPUpK1SXm7/+S+r6+noSVYwoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAP8/5/z/Ohq+gDW2/xD/6/Xj1/pzUuqqdlf8PT5fjptrcORT0cU/Xp+K7+foeP/BLxBqviXwOup6zeC9vT4v8AiRpyXQ/5a6boPxJ8V6FopUe+jaZYhmJyWyTlmNevmlD2WIq0rcqioK0VbeCld7XfvWWkb8p4OU4lVYxf8199WtdEtrXs/wDwLS92j13O0nv+n9G/p+uF+eu6cmt7Pf8Aq+/qv0Po7cyXT8f1j+v4XlLXYZBQAUAFABQAUAFAHg+tf8nG+AP+yM/Gb/1NvgJXev8AdI/9fY/+kSPGxX+8S9P8j//V/v4oAKACgAoAKACgCuT5fCr1PXOevfngZxjvjPfha5quJqrRQsv67p/nJbbWHRoUoLT8bvotPs7Ptb8bHhXxtZQfhNnAI+M/gdiPYJrW4n8xnIGOBzya+lyalXrrGuEW3HLcXUnZbQjCm5NrWy9Xdedj57O62Dwzpqc4xbq04q7+1zOys29dGrXdvL7Pva7eq/z/AP1+vXP514Kpez/z/r8um/U99TU0rO/9ei8u/rqOppp6r+v6/rcYUwCgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKAI8f7B/M/wCe/wDnBqPq1Hv+H9dv6uL2tT+V/fHy/u+f4Pe3vGP9g/mf89/84NH1aj3/AA/rt/Vw9rU/lf3x8v7vn+D3t7x839xa1/dfzkXq/wAkfvD5v7i0fuv5wvV/kj95GUIKqowq85PQnP59/XnuFOd0UYU8PSUU0l/XTTtb/P7UV1VrVY2Vox15tPLtd6L7/L7PjHxx+N3gn4BeE4vG/jpdUXQZNTs9Le50rT2vvs09+WSCW7jVkaO3MgEfmLlzM8UMcck80MMvq5Dww+IcXKnh5UVUsp3qynGNm7NqMFJtprrbdb3tHx+IuJFw/hoValKU03ZW8unS+nmttb7S+W/2Nv2r/hb8Swvwp8Jf8JHfeJrfUPiX40uRLoko03TtE1j4k61q8MlxqiyNGNqeKNLhQSIm95mTiWK4RPvvEbw5r8J42U8RPCKhJUVSVOpU9p7T2NP20VGdNJJ1VVlTtOUnTs5cr5j4Lw58Qp8V4WNPkqOonJzuo8qi27WkmpOy5b80YK90rWR+hZBB5IKnpj9e5z+IHtmvyqtToYml7NySS0Xl5fFd7Lz81q5fqsPbUqkpW5ovXTe//kr/AC8r2JPm/uLWv7r+cu9X+SP3h839xaP3X84Xq/yR+8Pm/uLR+6/nC9X+SP3h839xaP3X84Xq/wAkfvJKk0CgAoA8H1r/AJON8Af9kZ+M3/qbfASu9f7pH/r7H/0iR42K/wB4l6f5H//W/v4oAKACgAoAKACgCuGCDazZJ6Y7fz/PPGMc8VlUxVFVIxvd9/V/e9/5V5Ju6Io0Ksacrybf5adP+Ha9Fofgr/wUjm+Pnwz+Lnh/WfCXxL8er4H8b32k6x4b0S38Q6jDp/hnxroVsLeXTdJhgliWKGaHT/8AhIYxLMzzXGq6xBEgsrO4Ff0x4QYTKM0w9aOKwOFnWip4TEOVGlL21CpGDcKitJypVY+5OE7xqeycmmkox/nLxSxGZYGv+6r1EuZThaclyyUnytNNJSjdtON+Xm3V+Q/X79nnwt4y8FfCXwhonxD8Ua54w8dHTF1DxVrfiHUW1O/Ot6o7X02li6Zn8620NJ49FtZVOJ7bT47glmlc1/P+aez9rUVGMY0qbcYcsUlJK6lJNNrlc23G2nK9Um2z9zyWtOrFc7cpSS5rvRNu6SvZ35d2/K100z2/qh/z3z/X3/XC+Rhm3p2f6ei/N/I9yu7RbXVRa8tUvP8ATt5ktaDCgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKAPCvF37RPwS8C65N4Z8XfEbw34d1uFFefTtTneOUJINwJUJt6EEjcfxyVXthkOLqQhUhCTjNKUfegm07NP3nFWkndeWrta0vLef4RSak3dO19dLWur/ADa1X2b+6tDm/wDhr79mn/os3gj/AMGX/wBrqv8AV7G/8+5f+B0vL+9/XnZ8y/t/Bd306P8Au/5v+kxv/DX/AOzX/wBFk8Ef+DP/AO11t/qvmX/QPP8A8GUf/lhzf60YD+b/ANK/+Vh/w1/+zX/0WTwR/wCDP/7XR/qvmX/QPP8A8GUf/lgf60YD+b/0r/5WK37X/wCzUPlHxl8EOPX+0yuPXgx/56+lLDcMZjjKLlToSqRSd5KpSjb0jKab76b+V7RrGcTYHBV405yau19ltt+qT7pr8Uzzb4t/HP8AZQ+L3w48W/DfX/i/4LOl+LNF1DTJbg6pbMdPnz/oWsRhpoYzNoOqQ2mq28ksyQxz2cE8rJCvmV3ZLh8dkOOo1abiquHqc1pNXcX7tSCk4TjHnptw5k5OKleGvw+fnrwee4LkScoST12V38L03SlaVmmm1q/hUfkP9gPUf2f/ANm7wV4jvPGfxW8BSeP/ABfrl0dQa01qzuTYeHtHjEOkWmV3uDIbi81tpkKCWw8QaKlzFFLLbwV+oeKWY5txVmUYxilhacFq6kVKrVn8Wim04QjycnPShNTVR35VBn554a5XguF8JKTT55XWi0jHvte9278rtytLXXl/QUftgfs1f9Fk8EL7/wBqHn8o/wDP51+SVuFsxw9NTnhpQi9eZ1aMvvSlpvbr663j+nUeJsBiKzpxndrootPa+7un9y3v/dE/4a//AGa/+iyeCP8AwZ//AGup/wBV8y/6B5/+DKP/AMsL/wBaMB/N/wClf/Kw/wCGv/2a/wDosngj/wAGf/2uj/VfMv8AoHn/AODKP/ywP9aMB/N/6V/8rD/hr/8AZr/6LJ4I/wDBn/8Aa6P9V8y/6B5/+DKP/wAsD/WjAfzf+lf/ACsX/hsD9mr/AKK94P8A/Bi3/wAYp/6rZl/z5l/4Mo//ACwX+tGB8z6Lsr6DUbW3vbKVZ7K7txc210OQyv8AdOMdNpz68MDuwSvknvl+gAoA8H1r/k43wB/2Rn4zf+pt8BK71/ukf+vsf/SJHjYr/eJen+R//9f+/ZPuj8f51Nb4v67ImHwr5/meV/G3xRqngv4U+OvFGiTwW+saL4d1O+0yW4tzdRLewwu1vvgH+tBkCjawK9SwK5rswMVKpFNXSSb7eV1pfX/g3SaOPMG/ZeunbrHt6W6fjaX89f8Aw9N/at/6Cngf/wAJCv64/wCJfMq/nl/4Or//ACw/lD/iPWZ/yP7o+X93z/B7294/4em/tW/9BTwP/wCEhR/xL5lX88v/AAdX/wDlgf8AEesz/kf3R8v7vn+D3t7x/wAPTf2rf+gp4H/8JCj/AIl8yr+eX/g6v/8ALA/4j1mf8j+6Pl/d8/we9veP+Hpv7Vv/AEFPA/8A4SFH/EvmVfzy/wDB1f8A+WB/xHrM/wCR/dHy/u+f4Pe3vMP/AAVK/aq/6CPgb8PB+P1DfyX/ABqaP0eMvpu0pu3ZSlHTt7lRPdLq/wDDpc9ev4x4hx3Xryx8r6tPra22nXQ5DxL/AMFC/j542OhHxXafDrWW8KeINL8X+Gzc+FHf+x/E+hyPJo+shXv5Uea2d3ZWCLIodwkyLI4X3MP4HZPQjUim7Vo8tS1SpqrNaP2t4NqTs4NNPa1j57EeLWJlNNvms7xUrNL4X1TT2W61trY7L/h6b+1b/wBBTwP/AOEhXi/8S+ZV/PL/AMHV/wD5YdX/ABHrM/5H90fL+75/g97e8f8AD039q3/oKeB//CQo/wCJfMq/nl/4Or//ACwP+I9Zn/I/uj5f3fP8Hvb3j/h6b+1b/wBBTwP/AOEhR/xL5lX88v8AwdX/APlgf8R6zP8Akf3R8v7vn+D3t7x/w9N/at/6Cngf/wAJCj/iXzKv55f+Dq//AMsD/iPWZ/yP7o+X93z/AAe9vej/AOHpP7VOMf2j4Hwf+pQP89w/l+IyduFLwGwXtUpU5OPb21Zdt2qqa8vxtZOOtPxsxE6D11flDTp/Ltr3fzu4x+5/2Df2yPi/+0B8RNf8OfEi+8Py6RZaCtzbQaToM2nyG+SaFNxma4nzlJGJ3MACvG/LBvzHxQ8NsFkWE/c0+RXTlHmk27ySvec5Oyu3ZNdko6SP03w74+xGb1pKpeSk9+q06W9enne60j+w9fiR+1hQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAfy7f8FLs/wDDU+tYyV/4R+wwR0J232cdRnGO/Trn5a/tfwEmv7MV9LU4+b6dLLt5+p/GPjVB/wBpxt1l+bXZu/lZLTufAFftVKp/wpT9X18l/c69P/bto/k+Ng/7Nhvql0219f8A5G23mFaYfMcVBXcWui+f/B06/haOeMwWFrKykund7Pyv+f32YV5eYYjFV/hi36a9u1+3l5bXDB4LC0d5L+kl37ea+QV1vM8XP3XF67bafdbt3+7Uf9l4Sn9tO3m1b89XfqvS9wp+1xNTVxfTX7tPLz8vmL2WGhf3k/PS35vz2v8AKzQV31Kaltq+/fbp0K9v/c/H0/4P4BXnTwabfy9Lf+Sq9vT9A9v/AHPx9P8Ag/gFdoBQAVx4ejiaCbcWv16vyXyFiKOGxLsnfb8LecvSz+97Rvab/wAhSw/6/NK/pXnZhiMVJ2UG110u/L8v6sj0sDleEik3JLydvLrr1XaOy3aP7Ovha3mfDvwOc/8AMq+Hjzx/zCLU+3PsScdTuwDX+bdbd+n/AMjp/S+65/ouegVwVftfL9ACugDwfWv+TjfAH/ZGfjN/6m3wErvX+6R/6+x/9IkeNiv94l6f5H//0P79k+6Px/nU1vi/rsiYfCvn+Z4R+03/AMkD+K//AGJ2qfzWu3A/xKXq/wD0g48w/hv/AAw/9Kgfx4V/pt9WzL+d/wDkv/yw/wA6PbZb/Iv/ACby/wCD+AUfVsy/nf8A5L/8sD22W/yL/wAm8v8Ag/gFH1bMv53/AOS//LA9tlv8i/8AJvL/AIP4BR9WzL+d/wDkv/ywPbZb/Iv/ACby/wCD+AVz82YTatUdvl9/S/3LtrvLn9rSmk7W+cvLyf59Hta4VLwWZT2qNd9vL5r8P8L2i1Tovrb5Ppb+8nr6d9twrq+rZl/O/wDyX/5Yb+2y3+Rf+TeX/B/AKPq2Zfzv/wAl/wDlge2y3+Rf+TeX/B/AKPq2Zfzv/wAl/wDlge2y3+Rf+TeX/B/AKPq2Zfzv/wAl/wDlge2y3+Rf+TeX/B/AK68DVq16mq6d1rZa+fe2vlrqcUsLSoU9F1Xys13stl9/e6Z+qH/BJz/ktHi7/sW9Mr+aPH1V4UrJ+6/utdeb/TW1loz+l/A+nQqVPeSeiurf3fVdfTTvqj+jiv5KP6bCgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoA/l3/wCClOf+Go9b64/sGwxnOM/6f07bvu9P9nH8Ff2T4ETtl015dHa+vptt/N+Nj+QPGen/AMKMNeq6f4f7/wDn+kfi3xB8PPGPhrw74T8Wa54fv9N0Lxrb6lceGNSuVIj1UaPPHBqPlHIbEE00UZ3qm4yDytyAvX7Dhcbh6ub4rDU69KpiMK6axVGFSLq0Pb0/aUfb01zSp+1p+/S54+/G8ldK8fyrH0WsspyaaUl7r5dHaVny6q9no9fu1Rvt8Fvi7Je3mn2fw48X6zPYeLLzwFc3fh7QNV8Q6VF4vibbNoUus6HbXukLqUJ+V7c3fmBWRxuR42l8uvxrlMOW+Jw8G8L9ecZ1IQnHBLfFzpzlGcMPF6SrTXJCXuttlYLhnG1tf3kvfjB2i2ud7QunZydnaK1dvhja8sfx38MviB8LtR/sj4geDtf8JXU+WtE1zStR02K+RTgyabNcRRQ6hEpOJJoWkjTcocjKrXqZPxVlWZUvbYSvhsXSjLklUw9anWp88Um4OdNzipJSi3G7aUk9pKIsdwzjcM1CXPTml8M001fZ2fK7NbbeXNdOOFZ+GdV1LQNa8R2kIuNI0G70u11a4afaLVtbN0mmZBBB85rC5QDruXBzuQN3YjNcHh7XUeZxlJLS7UeRSdt2o80btWSckm43Tl5eHyrF19OZ2ulfft1stdunXZ2tL1rSv2bv2gNb0STX9O+CfxP1DTBaWN7bzW3g7VidR07VYFnsb/Q0a3EviOGWFonY6FBqXlxzRS3GyFvMr5DE+JGQ4WpUpSzHLlOjOVOrCWMw6nSnCTjOFSDqJwnCUXGUZJSUk04ppxPp8PwNjq8IS5arU1eHuy96/WPurmVrWs3vpvc4Sf4beOrKO5a+8Oajp8lp4pTwLc6VqCpaa8PEDNsGmjRZCureaHKQ4NrsFyfsjEXQe3b3aGc068KdanVhUo1aca1OtCSnSqUpKMo1I1It05QlBqUZRk1KPvR5kmeR/ZrTs4tNNJpuzT9y6acLpq0r3T66raXMa1pd5oOq6hoN81q15pd7faVd/Y7mG/iW+06RoJUS+sJJLd1SVHjYozxhlILEjFerQxsakYzSfvRUlzRcZK6W8JKMou28ZJTi1Z6qQv7O8u3X/Bf7D8+r66vcz67TzwoA968OfsuftD+J9O0zWdG+DXj6XTNWv0t9Nml0O/023u/P08ajbaoDqgtGj8PXVu6G38TyGLw9IzpEusmdljb4DFeKWRxnOnLH4GE6c3SqU3iqPNCpF2lTlBz5lUTTi6bXPdNWurH0+G4HxsUpctWXu88Xyy96FtZJ/wAujblZx0eqseZ6j4Q8XeB/EljpPjTwv4h8K600trd/2b4m0jVNA1AROuQ50/Vre2n8uRcYdrdEfbgZKui+5lfEuX5pShiKM6VelU1jWpTjUpy5ZOL5ZwbhK0ouMrPRq2ljzMyynGYGbg3KDi9U7qS8mndq3b7rbH9jXwt/5J54K/7FvRsdP+gbb+n9OP0r/OGr8UvSP5/d/WnU/wBFD0CuCr9r5foAV0AeD61/ycb4A/7Iz8Zv/U2+Ald6/wB0j/19j/6RI8bFf7xL0/yP/9H+/E9B9P6mssP/AAvl+sSnvP5/+lI8J/ab/wCSDfFf/sTtU/ktenkH++UP+vlT/wBIqHj5l8P/AID+aP49K/06P87QoAK88AoAK5MxqUqmLhZp3a1unpp00X4/dc7cvp1KWEknF3tv579oq/r+GrkV3ZhUpU8JC8lqu/p5Lf8ATpa0eHLqVWri5e673fn036/10VrBXVl+LoVMJJKa2dve3/penTfSJH1OtT3i9NbWt+kv/Svk7tBXn5fShUxc3zJe9pr0/XTur373KVWdPdNW/rZLr5N//IlemUFABU4H+Ivn/wC2k1v4P9fzI/VD/gk5/wAlo8Xf9i3plfzb49fw4/P/ANKif0B4J/xH/Xc/o4r+Sz+qgoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKAP5dv8AgpLL5P7V2rzeRbXP2TRvC3+i3R9R1469OvH9W/sfwYhKplM0tW6cdV6b+j+duz2P5J8ZJ+zzam3p79/6tfa/prutWej6/Fq48HXutTeCPEegt8RPA2maKde8P23xFbw38UNR8c/s/LoHgfQ9E0RfEMfhzX/EJ8Sazrnh3wX4Y+F/hKTwf4N8Hs53eDRsReF4LL45njaUMfh8XLBYjFSq0amIwc62W0KOOX1mdapajiFCnOnDE4yvj61fE1qtP2tWrWrNOXHhcxcMrw/NFwUoRtPlklUbX2V8LunyRVNRiloorVR6r4dWFlafET446wtjp91rs/7SPi+TRrS18c/EP4deMNdk8PJrcusP4O1z4SyeJ/E+vFBrGveGvGfhjwr8KfFHivGvyCeZHPitvBXy2dwxlfLcLCFGp9VjgIuDVLC1sJFU6FKrTniPrLjCnGjCm62HqzxEMNzQca03N4eNX18grYGjUd6sFN1Iq3POFRxk3CUYcqlzc3MoyioTqdYRlBVEd38c/Gvjjw78XbPxPq938UPCfwT8O6T4x0/xdcW3xI/aH0y6tlfxDpaDVPBms+M/h58MDrfj3xNLHpVn4O8M+GvFHxg8Is8UqmLwj4LEnjJvN4ey6eLw+JwlHCYDGYttOjBYfLKlByarujUxlHL8VjPY4bDS9qqtav8AV5OFRUMPCrjKnJV9fN8bgXKlVlXnCL5W3z1eeNlSbcZVYU1Kck4zjGLkk0nKUaajKPyt8QPHuoa58L/i/Joem+G7/TL7xb4Y8O6t4xtfhh8Nv7N8W+ONZ1rxzrT65oPjL/hHW8Sa+vijw8/h/wATOfFCjxj4KdwSdzlq/RcLldTDYyjJxxGHxlTDe3p5dLG1+Wnh6UMPQqN4CliZ4CmqFSpCk5UIVKXtGqkKspylI8PE5hg69KV5wlTUlBzSivelzOK9o0ptySlL3nzcsbWSidfrXwzsta8Sz22rfBbxrf2Wman4X8J+KvGlv4X8W6j4c1aH4VpoOjeb4O8abs6zrnipNE8bfDT/AIVn4ZDr4y8Y/wDCtGO3xwTVw4gjhJSjDMcNGoqM8VToVMWlVhDG03im6+DjCUqOGoxeHxn12vCvLD4aWKlOOGwaqOXhV8sw9e0uWTjOfJGag+WUovl5YT5venJxlDkj8Ukkk5WUev8AD3jbxT8O/gfc6pJpXiLQLnw18OfEGvXPgNfiRpumeFvC2oN+1tdbcfAbxAG+IOkeIfCiEsnxH8TN/wAIjbroh8HOpiuY2r5jEZc8TxnKUI0sZHEKKoYqGGnKNaNXJ6leNSnmKSwWIjXw0udUMM7zpVPbxShRqH0OUZlhYZdWjOp7PlcoypyqRTjLmirPDycasXGS5ZVJqzcVC8ZK0vNdb0Q/EL4rfDLw98RbTxl4pmi+Evxj1230r4/eINX8N+JGsNE134teOPha/jnW4/EWdBf5dDVmbxcWPgxfmxtNfR4CNfKuDM0q4GlToKrmOCVX6lSjXpRqTxWCwOMWHjQw7VWcFCdPnlQTVaMlUhHkah4FsLj8e/a1vaOMJcvtalnyKM6kFeVmovRxinqn7rd7lbwb8P8A9ny+8a6jB4jHwbsbOz0v4Xf8JL4ftfiKNQ07wzqmujxCPG8XhHxxrf7T/wAL/wC2vC7yNoKyzReJ/i00XjDXvNHgwbSj9Ge57nssFgVh6+bV5TnUdSvVwLp4mvDRxjjMPh+FMWsPyyk6cFHB4BunFVJVqtSTUufKMmy2FSpKboQXKrRU24RatrTc8bTc5KKulKtVu9HCycjotV8Q/BKw8C6NpMngb4DzeGPGPiX9nO0vNM0f4i6rpup4Twj450Xx34u1zRNE+Ib+I9A1/wAMpmJzjzNniB/+E3k8XSqJU8ingc3xOPXLXzFuMcykq0sJCEKXNi8vrwwtOpWwsqNSlOnJU1KTqVJwo1Pq1al7PEKPuyr5dQppOVL7Cs5/3Jx52lNSvFq6slFNxU4y5ked/D34mWHhjwNrcPhP4Z/Ci7vfh38X/Aln4P1/W/EHj3T9L1rVNb0jxvoX/CZazrcvxe8L+G2yvh/R0O8+EvBm2ebADi3e3+l4nyjGTzJt5nmcaFWliK/s6OHyysqCpSw69nQpf2XXxNbmjOajCc8RVm1yx5pSgzxuHcyowwLTo0m9nKUqycm+bWTVZRVnb4eT4Va3vqXzV8R/+EMX4o6kPh9baJaeGxN4f+z2/hy78V3ugjUzoGkf2+NMl8cZ8SHT/wDhIDqotP7SLSfZvJEWLXyNv3uR0MTh8pw6xcq8qyVS8sTHCLEODqydL2ywP+yuapKCbo2i7XajNyhH43Ma8MRjqnJy8t9ouVul7c7crb76rzsnL+u34YZ/4V74LwoP/FM6Lg5Uf8w62/L0/Ltiv87K9LFOo+bHSqS0vU9hFcyt2d9u6afRo/0FcsN9momvu17K3k0+m6te/u98C3cKPxQ/5/z6VzSo4lf8x0o/9y0X+rv8kvnYXNR/mX3v+uv4+g44Iw+AB6Efy+b26bf1JWqSxEWk4+71Ts19+j/rd7mFT2Ml704tfc1f076df/AbnhWsqp/aM+H4GNv/AApn4zEdT/zO3wE9CvP4+3GDu9T2lRYeMWmoe0vfdc3K1brsr9fvPGqUaHt5NSSdmt+l1rdPsk+3+HRS/9L+/E9B9P6mssP/AAvl+sSnvP5/+lI8J/ab/wCSDfFf/sTtU/ktenkH++UP+vlT/wBIqHj5l8P/AID+aP49K/06P87QoAK88De8JWMV/wCKPDVheWumXVjda9plpe2erXf9n6XdaZJqMSXaahf/APCSeDxFA1oZRIT4w8JqU+V/EWkgnWbDmxkpxwmKlTlVhUjhq8oTowdStCapScZUqawuNc6sWk6cFg8W5Tslha9/ZTqPxRuk1zK6bsnrs3eNk+r5o+qtc/WDVPgv4Dkj8a6PB8IvgBouheFvB/w9tL7xz4r1vxV4MD6nrTS+IdG1jXtH1z4/+FfiP4e8Pypqujxy+J/FBTxoP+KiWL/hK5fDEPg9/wCa48V42VanKOcZ9Ko+SdSMcLltZx5W41oc+HyPF4eNenClOKoUrU6leEIucXX+sH9CVeEKFPCzthsMl70U3OqtWoqLSliackrtS5pc1lKTakouB5l+y58FrTxbo3w/1i1+FrePYdU1r4lReL7XwtH8JfiJ4kP9iaMmh6Lq+rzfFfxKnw48AN4W1/XtFHgn4ceJX8ZJ4pYeJvHjpIYPCaV6fG3GmLhhcZRjXqU4xp4SNKdRY7BUIycoVZ06FfBOONxdZ04V6mKr4evGjh6UcLQ9lJrMJy4OE+DKFXGKThGXvSlZezm0leKc4zvCK5mkqc03PW9lKmejfFb9mzSLfX/iBLd/BzxroNhpfwg0fV8+IfCvw60Bjpmj6trs+t61ozfs2fD34u/DpviKNbk0Pw4Vk8J+EZj4IPiWQKAv/CYv43DXHmNp5fhKbxPNUWNjRl++rYhVHXvO0p5pjaOJnCOGnKXsoVcRVjWVGpRpzo0Z4Sr9BmnAuHi60lTUFyNpclNWUVbahTVO7lG10oRvdSknLnPm/wDaM+H/AMJdB+Hcl54F8M6voep+F/F2l+Gp77VNP0mxsdT0zXNc+Pat/Y2q6D4+8TrrYXxB4H1fwyD8U/C0vjJvB2ieHVIGxJJfvuF+LMc83qUa0r0pwr1YzeMlWnWlRWAoRqPDVMBho0F7NxqSWCrywlPE18QqlOrWqQqR/Ocz4VoU0uS7aaUk6SjGLfNpGUas3PVNJzjF8sYtdj4Yr91Py4KACpwP8RfP/wBtJrfwf6/mR+qH/BJz/ktHi7/sW9Mr+bfHr+HH5/8ApUT+gPBP+I/67n9HFfyWf1UFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQB/Lv/wAFKc/8NRa31x/YNhjOcZ/0/p23fd/8dx/BX9p+BFP2mWST35F/Wt+vdeu1pfx3461PZ5jTa79/8PTT5+9925+f9frkcrpVMyndK9303+7TReeneSPyZ5lUp5fBczWne1tr/wA36/imFdf9jZbJOTpRuvL0t0j5r1ttvKZ1sfQXu1JLyu/Xdt/rvbTYKnly3COypRXny37f4fL5dr2PNqY3Majt7WWm/vPXfvfu/wCa/S+0Su6pUwE429lF/K3+f9dtzv8AZ4+H/LyWvnfb5rv0f32CvNqYbATd/ZR9GvRee2/2bf3vsv61joWvUlp5/nrpfTo7+Vz1rxb8GvFfg7S4tc1DVPAF5p8uheHfEHlaf8Sfh5P4gGleLNI0vWNJkk8EN4mTx3FvttUgZzN4dSJEEtw0q2CJcy/MZXxJlGZcV4+hCjiOdSr0uaWAxsMPz4X2lGsvrU8MsNb2lOSg/b2q2Tpud4o+k/sPMMHgZVPaK1loq1NyXNFTj+7U3UtaSu3GyfuS5ZLlj5rpeqaxok1zc6Pf3unz3Vle2Ulzp8xthJp2qwyWN/YyAEeZHNbySJIp+/E7AkhsV9ZioZc+Gce3Thbng7ON1eNRNNabppNaJppNWdj57J1j/rk06snr9yt69vL9ObPqsXlWBp7wjpbdb+ur2uui6b6HHhM1x09qslf18urbsvn16aBVYOhgKbV6cX5W16dbb2+63XSQsXXzCW9WV/V+Xquvfp1uwrlzDARqYuF0tGum+q6+5tdf07nqZdmE4YSSTdrefTTys9el32vq43tN/wCQpYf9fmlf0rrzGhHD4OHKraK3Xtf831fq7HHl9eVbGTv37+XbX8/W/wBr+sn4d+CPi7N4G8HzR/G17eE+G9JP2YfDnwgAv/EqiwM7wOu18ALkKckgkV/ndXzHKfaSc+H7Tlbm/wCFDEpq70tGN4xXZR0Wi0Wp/f8AHLcwUdKjt0Vou9o213evKlrffraJ2v8Awr/4y/8ARdZP/DbeEf8A4/7D8q4nmWUdMk+X1/Ev83Hz69el/ev+zsf/AM//AMKf/wAl/V+n2T/hX3xk6/8AC93/APDa+Eu3/bbP6D29G5/rmAX/ADA/+XFb+ul/mX/Z2L/6CPwiSeGvhd4ssfG9n468XfESTxhqOl+FNf8ACmkQP4Z0rRbfT7PxPrHhjWdYdV0kRSTySzeEdGhR5nYFIGcqGdVQq4/Byo+ypYaUF7X2jcamrspQXvSUns03DaLbs37zKWT4ly5nWjfl5dY20dntGS69fKztZKP/0/78T0H0/qayw/8AC+X6xKe8/n/6Ujwn9pv/AJIN8V/+xO1T+S16eQf75Q/6+VP/AEioePmXw/8AgP5o/j0r/To/ztCgArzwNjQdf1bw3quma/oeo3ml6npE1tqVnfWF1eaffWlyBxJb31hLDqcUiZJSSKWHYVBWQYVkzq0qVelUoV6dOtRrU50q1GrCNSlVpVIuFSnUpyTjOnODcZwknGUW0002hpuLUotppppp2aa1TTVmmnqmn91j23Vv2lvGt/eWD2+h+DbDSbeGKz1fwzHpus6roHi/To9Gk8N22m+Mn8Q6/rPiDXbPS/DhTRbaKfX0mimF94nW4/4WFrPiLxbq/wAfg/D3D0VOMsZi6tSoopVqtWHtKTjZ81JQpQoLnrOWIqRdGVOTqfVuRYGjh8JQ+2xPHNeS0pxjHX3IpyTu768z5naOkfeukua7m5zl5zpfxI1/R9A07w/paafHaabeeN7vdLaQ34kTxzomh+HtbsWTUEdfLuNK0WCAiLGPNlYMXKNX1NDgzBRbndc840qcnzNXjRnUnS0v9mVWd7JX2baSifNV+L8VLTkdrt7P+70XL/Lprp5/a662+PGv2PibW/E+m+FfCGnprPgzQfAH9hJb61faHYeFtDk8NmPStJj8T6/rupWdyY/Cdih1V759Q0xvPbRLzR99uLX5mtwi5U6NOVWviJ0K1TEKvNwoVKlSpOrKTqwwVLCYepFKtKDg6DjPSdb2tbmrHtYXOVTbuowvGMbLma92MU5L2nPJNuKb1Vm7R5IqMI0fHfxl1n4ix6jF4i0LwvKhu2u/D9xHpupDU/CcGxozpmma0msx6trVpKhV54vG974sD6k+p65vXxDruuaxq95dwrPCSjKVXEVJeyhTnKrPmVZwcpe1nSShRp1HOpNz+rQoRa9nT5HRoYenQMbnMazfKo2crpRXwp20TbcpJcqS5nJ9W1Jty8gr74+TCgAqcD/EXz/9tJrfwf6/mR+qH/BJz/ktHi7/ALFvTK/m3x6/hx+f/pUT+gPBP+I/67n9HFfyWf1UFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQB/Lv/wUpz/w1HrfXH9g2GM5xn/T+nbd93p/s4/gr+0vAX/kWf8Abr/M/jbxp/5GFP1/SB+f9fstH/kYVPX9EflOP/5FlP0/9vQV6JzBQAVxYD3F1/K/a2/a/wCqtcMeueXlp0u+n+Hfpv8AO15e5/s9a9H4I8d3fxGn1O302y8GeG/EzSgeLbrwZr14vizRrrwHjwnq+ieHPF+tRa/pcfiubxHaG18O36xQ6FePNLAFQy/F8Z5dLNMPHBxpxf1nE4f362EWNwsPq1WON/2vDurQ58PV+qrDvlnB+0r07NXvD6bhzFRwklLmceWMvhnyTaknF8stbO03/MrXvdXUfoT4xePJvE3w18U+DP8AhZmr+LWh8Qr4jWf4jftVN8ZZI3+Fer6/okmmeEtIPwf8JxXC+LR4yh8QRKniXz/Fng3QSN7+NfCz+DU/PeCeD8RlmZYbGVMJGEatH6vbDZS8DKnDF04YuNfF1HmNeMoUZ4R4KVJYd4rD4mtCUoUsLVdef3/FnFlLMMPOmqjck+bmlVU1JpuPLSiqPNCTVTnbc+ScYte9NQifBFfulWDhTivK33W3/wCGXne1o/kVOanUlL+Z9/VeX9d73Cu85woAKzw39f8AkwV/sf8Ab36F7Tf+QpYf9fmlf0rzcf8AF/XaB6GX/wBf+SH9nXwt/wCSeeCv+xb0bHT/AKBtv6f04/Sv81KvxS9I/n939adT/Rg9Argq/a+X6AFdABQB/9T+/E9B9P6mssP/AAvl+sSnvP5/+lI8J/ab/wCSDfFf/sTtU/ktenkH++UP+vlT/wBIqHj5l8P/AID+aP49K/06P87QoAK88AoAK7Pq2Ji/iaa/yj6dn0+4X1nDysnFa2+W3k9tflbfVSKm2Lje02u71t+P9a+g08K/sp/d+kZf13s+UpYTFxq7r0v1va/p06Ppa9mjDF89DVSvt+P3W+9d9Qp4vFxp7JL7nbb/AA7/ACt135QwnNWs5Senz6f5W6b66WCtjcKACpwP8RfP/wBtJrfwf6/mR+qH/BJz/ktHi7/sW9Mr+bfHr+HH5/8ApUT+gPBP+I/67n9HFfyWf1UFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQB/Lv/wAFKc/8NR631x/YNhjOcZ/0/p23fd6f7OP4K/tLwF/5Fn/br/M/jbxp/wCRhT9f0gfn/X7LR/5GFT1/RH5Tj/8AkWU/T/29BXonMFABXD8F0vyv/W/9WHbnu/i/D+tgp/V1iLc2vy+Xd9PJW8yXiHQ2fyXTb/Pyt52SCphUpwa5Ypbf1sl59Ne+rOmVOpPSUm/X/wDaf9P5SK74RVdXa+X9baJHHKXsN+v9ed9+8bed0FBoFABWeG/r/wAmCv8AY/7e/Qvab/yFLD/r80r+lebj/i/rtA9DL/6/8kP7Ovhb/wAk88Ff9i3o2On/AEDbf0/px+lf5qVfil6R/P7v606n+jB6BXBV+18v0AK6ACgD/9X+/ZPuj8f51Nb4v67ImHwr5/meEftN/wDJA/iv/wBidqn81rtwP8Sl6v8A9IOPMP4b/wAMP/SoH8eFf6f+0w3/AD9j/X/cQ/zftif+fcvufl/079fw8wo9phv+fsf6/wC4gWxP/PuX3Py/6d+v4eYUe0w3/P2P9f8AcQLYn/n3L7n5f9O/X8PMKPaYb/n7H+v+4gWxP/PuX3Py/wCnfr+HmFHtMN/z9j/X/cQLYn/n3L7n5f8ATv1/DzCj2mG/5+x/r/uIFsT/AM+5fc/L/p36/h5hR7TDf8/Y/wBf9xAtif8An3L7n5f9O/X8PMKPaYb/AJ+x/r/uIFsT/wA+5fc/L/p36/h5hR7TDf8AP2P9f9xAtif+fcvufl/079fw8wo9phv+fsf6/wC4gWxP/PuX3Py/6d+v4eYVGBq4XCzu6kfLXr963/XzvF5hHEYqlZU5WVru176r0/rva5+qH/BJz/ktHi7/ALFvTK/mfx6nSxVN8kr+lnppru+nkt9L2Z/Q3gQ6mFqNTi0trWfbv7vr+Dtuf0cV/KB/U4UAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFAH80//AAUK+H/jfxP+0x4j1Tw74Z1fWbJdL022aWy02a+UMLeFGwYI5GQBg0YJVslOh2nb/XHglxBluTZXOnWqwi3F6Npa6vb1+Wnmfyx4q5PmGYV7U6U2m+z0Wlv+G18pPVHwx/wpf4pZ/wCRD8VYzj/kCatjHr/x77f0/DPFfrOVcY5TSzWpVliIWcpa3WmlvPt2dtve+I/J6fB2bU6KvQnrb7Lt+lu2vN30tYP+FL/FLP8AyIfirGcf8gTVsY9f+Pfb+n4Z4qf9f8p0/wBph06/4f7vXX/wLpf3X/qLmv8Az4qfdLy/zf4dw/4Uv8Us/wDIh+KsZx/yBNWxj1/499v6fhnij/X/ACnT/aYdOv8Ah/u9df8AwLpf3T/UXNf+fFT7peX+b/DuH/Cl/iln/kQ/FWM4/wCQJq2Mev8Ax77f0/DPFH+v+U6f7TDp1/w/3euv/gXS/un+oua/8+Kn3S8v83+HcP8AhS/xSz/yIfirGcf8gTVsY9f+Pfb+n4Z4o/1/ynT/AGmHTr/h/u9df/Aul/dP9Rc1/wCfFT7peX+b/DuH/Cl/iln/AJEPxVjOP+QJq2Mev/Hvt/T8M8Uf6/5Tp/tMOnX/AA/3euv/AIF0v7p/qLmv/Pip90vL/N/h3D/hS/xSz/yIfirGcf8AIE1bGPX/AI99v6fhnij/AF/ynT/aYdOv+H+711/8C6X90/1FzX/nxU+6Xl/m/wAO4f8ACl/iln/kQ/FWM4/5AmrYx6/8e+39PwzxR/r/AJTp/tMOnX/D/d66/wDgXS/un+oua/8APip90vL/ADf4dw/4Uv8AFLP/ACIfirGcf8gTVsY9f+Pfb+n4Z4o/1/ynT/aYdOv+H+711/8AAul/dP8AUXNf+fFT7peX+b/DuH/Cl/iln/kQ/FWM4/5AmrYx6/8AHvt/T8M8Uf6/5Tp/tMOnX/D/AHeuv/gXS/un+oua/wDPip90vL/N/h3L+nfB34pR3tnJ/wAIJ4qwbjaP+JFquCfT/UkDp+HfOMsf6/5Tp/tMOnX/AA/3euv/AIF0v7p/qLmv/Pip90vL/N/h3P68Phevl/D7wfj/AKF3Rs9eANNt/wAvQcfTdiv88D+/j0CgAoAKAP/W/v2T7o/H+dTW+L+uyJh8K+f5nA/EfwhbePPBPibwjdTz20GvaXJYXD2xwyhznC59doBAI4CnnpXZg5qEqbaVr2u91dLXdWt+KbStvHix6vCVm2/z96Or+Wuj++5+V/8Aw6v+Ff8Az8eLv/Auv3D/AIi9if8An7I/Hv8AiElD/n3H7oh/w6v+Ff8Az8eLv/Auj/iL2J/5+yD/AIhJQ/59x+6If8Or/hX/AM/Hi7/wLo/4i9if+fsg/wCISUP+fcfuiH/Dq/4V/wDPx4u/8C6P+IvYn/n7IP8AiElD/n3H7oh/w6v+Ff8Az8eLv/Auj/iL2J/5+yD/AIhJQ/59x+6If8Or/hX/AM/Hi7/wLo/4i9if+fsg/wCISUP+fcfuiH/Dq/4V/wDPx4u/8C6P+IvYn/n7IP8AiElD/n3H7oh/w6v+Ff8Az8eLv/Auj/iL2J/5+yD/AIhJQ/59x+6If8Or/hX/AM/Hi7/wLo/4i9if+fsg/wCISUP+fcfuiH/Dq/4V/wDPx4u/8C6P+IvYn/n7IP8AiElD/n3H7oh/w6v+Ff8Az8eLv/AysKvi/ipYiMYzaTdnq+tlvzK/fWK+W5tHwkwyw0m6cb2T1t010Xr8l8z6G/Zr/Yu8Kfs9eMLrxX4Vn1sz6nZGyvf7Vu8nA5PHOT1wACfQNkivjuNeMamcYdOU1J6X15lv7rey/HfazR9DwbwpRyjEy5Y2UW3dLoui2X5+Xc/Qqvz0/SQoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKAOQ1Hw5pOpTLJPBljwSODnpnHXnjoBn16huyhjcwy+LWHrSUHuk9N9FfZ+ttOtjxMfRwOOrw9rShJ6XfKtfXbfdXXyWvLSHgfQzvzAAF9e/ODng/y9uK9D+28ySjKOIm5ve70fbrpZX/m/U6sVk2V/Vo2wsI2W9k79Nbd+n3NPeMn/CCaH/zyX/x2uP8AtTNP+gip/wCBS/8AkfJf8G3vaf2blv8Az4h90P8A5H+rdNOU/wCEE0P/AJ5L/wCO0f2pmn/QRU/8Cl/8j5L/AINveP7Ny3/nxD7of/I/1bppyn/CCaH/AM8l/wDHaP7UzT/oIqf+BS/+R8l/wbe8f2blv/PiH3Q/+R/q3TTlP+EE0P8A55L/AOO0f2pmn/QRU/8AApf/ACPkv+Db3j+zct/58Q+6H/yP9W6acp/wgmh/88l/8do/tTNP+gip/wCBS/8AkfJf8G3vH9m5b/z4h90P/kf6t005T/hBND/55L/47R/amaf9BFT/AMCl/wDI+S/4NveP7Ny3/nxD7of/ACP9W6acp/wgmh/88l/8do/tTNP+gip/4FL/AOR8l/wbe8f2blv/AD4h90P/AJH+rdNOU/4QTQ/+eS/+O0f2pmn/AEEVP/Apf/I+S/4NveP7Ny3/AJ8Q+6H/AMj/AFbppyn/AAgmh/8APJf/AB2j+1M0/wCgip/4FL/5HyX/AAbe8f2blv8Az4h90P8A5H+rdNOU/wCEE0P/AJ5L/wCO0f2pmn/QRU/8Cl/8j5L/AINveP7Ny3/nxD7of/I/1bppy9bZ2kNlEIYeg656/wCfz/UBfPO0t0AFABQB/9f+/igAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKAP/9k=) + + + + 以前的版本中普遍传言说它的性能不好，但是后续的版本性能得到了较大提高，经过系统压测还是能够支撑较大并发量的，经过压测三台Zookeeper能搞住20000tps。 + + 用zookeeper的优点大概有：高可用、公平锁、心跳保持锁。 + + + + Zookeeper防重策略 + + 利用ZK确实是一个不错的方案，流程如下： + + ![这里写图片描述](data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAASABIAAD/4QBMRXhpZgAATU0AKgAAAAgAAYdpAAQAAAABAAAAGgAAAAAAA6ABAAMAAAABAAEAAKACAAQAAAABAAABrqADAAQAAAABAAABQwAAAAD/7QA4UGhvdG9zaG9wIDMuMAA4QklNBAQAAAAAAAA4QklNBCUAAAAAABDUHYzZjwCyBOmACZjs+EJ+/8AAEQgBQwGuAwERAAIRAQMRAf/EAB8AAAEFAQEBAQEBAAAAAAAAAAABAgMEBQYHCAkKC//EALUQAAIBAwMCBAMFBQQEAAABfQECAwAEEQUSITFBBhNRYQcicRQygZGhCCNCscEVUtHwJDNicoIJChYXGBkaJSYnKCkqNDU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6g4SFhoeIiYqSk5SVlpeYmZqio6Slpqeoqaqys7S1tre4ubrCw8TFxsfIycrS09TV1tfY2drh4uPk5ebn6Onq8fLz9PX29/j5+v/EAB8BAAMBAQEBAQEBAQEAAAAAAAABAgMEBQYHCAkKC//EALURAAIBAgQEAwQHBQQEAAECdwABAgMRBAUhMQYSQVEHYXETIjKBCBRCkaGxwQkjM1LwFWJy0QoWJDThJfEXGBkaJicoKSo1Njc4OTpDREVGR0hJSlNUVVZXWFlaY2RlZmdoaWpzdHV2d3h5eoKDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uLj5OXm5+jp6vLz9PX29/j5+v/bAEMAAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAf/bAEMBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAf/dAAQANv/aAAwDAQACEQMRAD8A/v4oAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgDhfGvjjwp8PvD154r8a6zY6DoOnR+Zc399P5cMa9BlmCkk9Aqgk9Tj+LqyvJsXn+IhRwdCVarK0uVWSUU7OU56KMenm9nJtI4M2zrB5Fh3UxFXlTVleyba1slzS5rd1p3vdcv5CfE/8A4K42Nlqclj8Kvh5/bGn20wSTWvE072UE6eWC3k2sREu0SFlBlnilwMtApBRv3TJfA3GZlRj7Sc51GlpGKjCOzVleUnoknd200S05fxPN/G3CZbVlaMORSdru+n/A9Hay2taXnHhn/grv44F9ajxR8NNAvdLL/wClPpt3cW98I8HlS8lxEGBwNrRDpnzOCr/UZl9HinOjUq4arUk0vhvzpPe65lJ7pLVvT0jzeDgvH6SrQp4mlBczSbsob9dOVf101jL9W/2e/wBqj4X/ALR2iSXfgm/EOs2sSy6l4Zv3RNQswfl8zYkkiSRFlYCSKR1ZhgOGWRV/n3inhDN+FKsoeycoJtKfK11sk9Ha+19bt3tE/c+G+Kcq4pw0ZqpTUpJNwur2393Vtu+mqk+1tD6eVMcn/P6nP5D8a+Up02nfpv29NLv8/uPqZSvotvz/AAVvvfyuKy7se3tn/wBmX/Ppj5umMuW/n52/9tl/Xe/u5Sjzf1/wV27P5DqkoKACgAoAKACgD//Q/v4oAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgD+Z7/gpF+0XqnxK+Ll98NdMupYvBfw9uPsklrCUYXevQ+csl6ZFd1dYxK9qmAoCpLIgKzuG/r/wT4Lw/9l1MZUpxc1Hmk3G7ul5xUtNLJ6929j+RvGnjHELMqWDpTlGMmopKVrXtp0/m7fdc+Obb9nz493lpbX1p8DfjHeWd1bLeWl3a/DTxndWtzat9wh00JgsjBgVUF5D2U4Bb9EwPiFkeSZrXwdTMMup+wqOlONTGYWDjJXTjLmqJRkmleMrtbPZs/NMw4Lx2cZXSxap15e1ippxhNpq6u1bdLya17XXN57e+DvEVjp3iHVb/AE+XTovCviHTfDevR6iTaahZa7qsetSxWEli484SBfD2q+arIGiazkEvlvhV+toZ79S9lCdWFVYijKtS5WqkatCPsk6sJRlyyhfEUbNSs1Ui4uSvy8GIyWjj7yoLlcZJS3TUt1vb+V+lvtaOXo/g3xN8Qf2b/itZ6xayXGg+KvCmoGHVtKhmsiWVXC6jpd+UeaPkoYZVjaUxyLvikWaON08DMsryLiyk5VqNKtRxNL442aqU60F70JpO6lGV4zi5Jpprod+AzLOeGmlh69SHI9lJpqz06p3VumzXSzcf63PhZ4+0v4qfD3wl4/0k4sfFOjafrEWeMeYFYg8YHzqSOfu4Bzg7f89KlPkbVt3p5X1762tbr3bWnN/f1Or7Tyt+PTa2n3+SWrZ6GzY47/5+oOR2P5DOV4qlRp6beWnbrZ/l9x0xjfV7fn+Kt62f4D63ICgAoAKACgAoA//R/v2C4OScn16f1OfyH45rC7b82A4jPFT7Ft/y/j/XcCt5i+o/L/7XT9jH/oB/8q0yvZVv+fsf6/7iB5i+o/L/AO10exj/ANAP/lWmHsq3/P2P9f8AcQPMX1H5f/a6PYx/6Af/ACrTD2Vb/n7H+v8AuIHmL6j8v/tdHsY/9AP/AJVph7Kt/wA/Y/1/3EDzF9R+X/2uj2Mf+gH/AMq0w9lW/wCfsf6/7iB5i+o/L/7XR7GP/QD/AOVaYeyrf8/Y/wBf9xA8xfUfl/8Aa6PYx/6Af/KtMPZVv+fsf6/7iB5i+o/L/wC10exj/wBAP/lWmHsq3/P2P9f9xA8xfUfl/wDa6PYx/wCgH/yrTD2Vb/n7H+v+4geYvqPy/wDtdHsY/wDQD/5Vph7Kt/z9j/X/AHEGqq4VJTvbkD5Dx9cY+uPpjPK1linha1ampU5Sa1ulNdvJX2tqtPMzw1PEUqNp1E2vTTr6Py06dQkYRKTMRs7KF69uuBjHufTOeKMbVwVCinUhZWVkozutV6W7a82z2skGHp4qVR8001d2sl12fn1auW67SgoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKAKl2fLtZj/sn+nuc/mv4UAfxhfFUzL8VPiB9tj88/8ACb+JwTj/AKjV2Rjg9FOMbvUYPJr/AEP4VxtfE4GahG/u6LpbX02SfT7rH8CcW154fGU3NW1T16f187bLTWP3Dpw+FHhK18O/DzUfh58UdL8WW3wx/aI+IOteE2+KvhFtT8Kw+OfhlJ/xItXb/hTwA8TPoHgmLxQPDrI6eDU1tx46h8YGWPwin5NmqzzE4qtjqWY4GWGrYnCYehXWVVFh8RGjjLU54eUszrupHEVq7wsar5VXcI1sF7KE6OKn9XlOY4d4OlBwqcypyck68U1KUYq7XsI2UVFycddHZzVpxlu/syeMILTwSukWl/rvg3w14l+Luk6L4P8AAnjbXl1Lw9450rWtY8ErrTaDrviHPh3Xdf8ACXhzSvEviXxkB4TPg3/hM/Evh9UwH8J+DK341oYvHYmCqYelicxhgFOrjKOETrYeC+sqNT2UOfFUcNVqSxFPCUI4uriKr9vCkq3ssZiI9fBFTB5ZF3rOFKU3aEpvlb9261tGUko03N8sF7sW7+4jyH9qu50W/wDBvw+l0bwlpXg+ysvGvxS0LSdB074p/D/44/Yth0LW9aK+NfBGPEcv/CWa/wCNtZ8UP4c+Jni3xb4qnkDTRL4LWXano+F2HxmGxGOVV4qpNRwc5vEZdmOXLVYiinGGOrOjVnKOHTqywWEwtJySc/aSlFUlx7j8Di40lTlT15tIVaVXezs/Z3lGyeiqylPZaKNj9tP+Cdbaif2YvBiaj99VuFtQSc/ZDcybc8cEMH4OPrkgt/MnH9aNbMI1E07wUXbylddX3293yvsf0fwFh5UMFNO62a0/D8tdfwbj95g4A9gCf8/hXxMfdppvov677X+fl9n7jq0u/wCe35C1YgoAKACgAoAKAP/S/vwR+4/Ef/XwPzx+eK56Sd97pap233v+Pr+KcdZ2t+X9fLz/AFj5D8f9d1Xw38HfiLreiX9zpmq6b4V1W707ULPBurO8jtJJIZoR13pIEII6divDV62DpKdRRe9padPd279L7273b0PNxNRq2tlpZdW2k2tl01u/RJXP5hP+G4f2svu/8Lz8Y7cY2ebo+cf3cfYcZx2xn+Hb/DX9w/8AEI/Dr/oSvp/zE1f7v+b+7yZ/Fn/EUOPv+f8AUt/18iH/AA3D+1l93/hefjHbjGzzdHzj+7j7DjOO2M/w7f4aP+IR+HX/AEJX0/5iav8Ad/zf3eTD/iKHH3/P+pb/AK+RD/huH9rL7v8AwvPxjtxjZ5uj5x/dx9hxnHbGf4dv8NH/ABCPw6/6Er6f8xNX+7/m/u8mH/EUOPv+f9S3/XyIf8Nw/tZfd/4Xn4x24xs83R84/u4+w4zjtjP8O3+Gj/iEfh1/0JX0/wCYmr/d/wA393kw/wCIocff8/6lv+vkQ/4bh/ay+7/wvPxjtxjZ5uj5x/dx9hxnHbGf4dv8NH/EI/Dr/oSvp/zE1f7v+b+7yYf8RQ4+/wCf9S3/AF8iH/DcP7WX3f8AhefjHbjGzzdHzj+7j7DjOO2M/wAO3+Gj/iEfh1/0JX0/5iav93/N/d5MP+Iocff8/wCpb/r5EP8AhuH9rL7v/C8/GO3GNnm6PnH93H2HGcdsZ/h2/wANH/EI/Dr/AKEr6f8AMTV/u/5v7vJh/wARQ4+/5/1Lf9fIh/w3D+1l93/hefjHbjGzzdHzj+7j7DjOO2M/w7f4aP8AiEfh1/0JX0/5iav93/N/d5MP+Iocff8AP+pb/r5EP+G4f2svu/8AC8/GO3GNnm6PnH93H2HGcdsZ/h2/w0f8Qj8Ov+hK+n/MTV/u/wCb+7yYf8RQ4+/5/wBS3/XyIf8ADcP7WX3f+F5+MduMbPN0fOP7uPsOM47Yz/Dt/ho/4hH4df8AQlfT/mJq/wB3/N/d5MP+Iocff8/6lv8Ar5EP+G4f2ssbP+F5+MtmMeX52j52/wBzH2HGcdsZ7bf4ateE3h97ZS+qUWr3/h6N731j37x/zON+KfH8qPKsRV10vzR7qy3/AB09Okv0R/4JvftB/Gb4q/FDxRo/xK+IOveLdPtPDwksrPWbgFFfOTJtOwb+SQwIbcc8YBr8S8XeF+DMmwjp5dh6FKrzQslBxg05xU3aMYxcuX+aOqjpZ6S/aPCniPjbOa18VVlVi9WpuEl8Lt8XNaz10Wi6u5+6Ffz0f0IFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQBHIvmIQO447/4fy/AY+UA/mT/AOCi/wCz7q3w4+K998Q9OtQ3hPxtcm6eeAAfZr98mVWjTYF81g0y4IV97ooIiZa/sTwU45o5jh4Uqj5koKF3dt2SV3d3u+/vN9e5/JPjJwzDLq6nFKO+2nZ2t81ta3Xflj+c9fveIp4DFVuZ043dtbX7eS369/LSJ+MYHMJ0KXKm7dbv/Jt+Wy+YVvKhl1aDlKlFtf3Vr+iutrct100ucc3j8PJRjUkr22fVdO/Xtp56HoHwt+HWv/FbxrofgrQLWaS8v7nF20XK2VjvVZL1+SCse9S3QklY0y7otfF5vxFgckcuWEYQheTa0UYpXbejtZLq/PSzPoMtyPG5sk51JSb6ttt/n6/D1W+iP66vgx4Esvhv8OfCvgzT8/2foekabZWn+6qtn053bQefbjIr/P7iDEvFYpNvVScXe+8beiaTemt+ra15f7zyDDrDYVqKtda26q1+3rtzarTax60T9/8AAevqD2/w+pwa8uv7tFW/y00f96+mn+fwx9Wm71JeTS+5X8uv9O7ZJVjCgAoAKACgAoA//9P+/E9B9P6mssP/AAvl+sSnvP5/+lI8J/ab/wCSDfFf/sTtU/ktenkH++UP+vlT/wBIqHj5l8P/AID+aP49K/06P87QoAKACgAoAKACgAoAKACgAqcD/EXz/wDbSa38H+v5kfqh/wAEnP8AktHi7/sW9Mr+bfHr+HH5/wDpUT+gPBP+I/67n9HFfyWf1UFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAeX/ABE+HPhz4jeGr/w34k0q01nTbuAi4tbu3VvMHYrwOTyNmd2R3NevkeNlk+IhVjb41du3K7b3W6S0d1Ja9Xqzhzmh/auGlh1s7qKTvvZrVtPtbe/92zZ+M/xR/wCCWVq2rS3Pw88TTaRaO246dqEH2wW6kfwzJ5ZwTnrESO5AIVP6I4e8YqWAoU6bkk4qzs7peez/ABs+reh/Puf+EVbH1pVFHRttab3e27a+/wBbHi/wz/4Jw6z4ztrbXb7x35Ggtq+t6W9vFprm+/4kWvavoMpw8sSqWn0ppOd+1JNhYsolf6LNfF6lKjOMpJXi9/RaXWl36PbV62j87l3hNUjVi4xbUXuul7P+9201+77P7Afs8fsneA/ghpixaBpy/wBoTKF1HVbgf6bdAZIUlQiCMMWZY44kTcxbazHNfz7xPxVHNq0pRa5W9He+vW7vp8lLa/aB+58NcMzymlCE4tJK17dvu8+/4WPsuKIRRiMdBkfn/n2x74zXwB9+S0AFABQAUAFABQAUAf/U/vxPQfT+prLD/wAL5frEp7z+f/pSPCf2m/8Akg3xX/7E7VP5LXp5B/vlD/r5U/8ASKh4+ZfD/wCA/mj+PSv9Oj/O0KACgAoAKACgAoAKACgAoAKnA/xF8/8A20mt/B/r+ZH6of8ABJz/AJLR4u/7FvTK/m3x6/hx+f8A6VE/oDwT/iP+u5/RxX8ln9VBQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFAEO0hAM9SOOnbAPUnn6j8fvVniZPSK3en3fd32f3raM4ePLFX6Jvb0/rp+svK/ip8Xfhz8GtEtdf+JeuxeH9FudQj02C/n07Vb+3N9KjyRQvHpVnfyxlo4pHV5Y/L2xu7EAHb6eVZLjMfzRw0HWnBJte0hHST0S55009Ek5c1la2n2vHzbiHD5W7T5ldtNtOWq7uy16qyd1fyPmj9k39oX4MeMdKtvh54a8YaVqvi6XxD8T9bj0Wz03UoXOkT/EPXtWttQaWbSrS2VBpuuaQshaVHSa4+zPGZYnSL7DjHhjMcuxGInOlKGFcKTpzlVhJr9zSdSHLGpzR5avPd8ln8UXLmPjuDeKsNmahC79prFrlW93aSb7xaW/2dbe6fdRYqDkkj6++cj5Tjr6c+p/i/NcNTk5y522ot9lpsl1euvd6dLqUf0mvVUIppa7af102+fkT11lBQAUAFABQAUAFABQB//V/vxPQfT+prLD/wAL5frEp7z+f/pSPCf2m/8Akg3xX/7E7VP5LXp5B/vlD/r5U/8ASKh4+ZfD/wCA/mj+PSv9Oj/O0KACsov2NN+Sf6ffou6fmmh119YmrK17dXureT/FP5X937P+B9lb694S8PaVENBsdQutS8fWXg+4+LemeD/iD4M1/wCJzaHoWV0DSj8PvE/xMSQ+HdabZ4a/4RbxX4L8YeNf+EYJkTx3/wAUdX4txpiearjZ1FiK3s6FL93g6mIwc6eHqzd5zxE8bh8JOaq4erN14Sw9XC4dOFSSpVHOr+j8LYBQjTblBXld88VON1bePJUfLZq8eWSlbSEmlA9X8IeGvD11+1GliNF8Labe+G/DPhSX4gXPhjQfCWteDI/EkX/CAaVHq2keDfHH7Pvhbw38Kd3iaBj468K+J28IeLvCnix/E1v4J8WnxoZ/B3jH5nB168uFadF/Wan1rG1pYSl9cxeGxTwsY4rEwpyxWFzPFYjMZulSq1KDwssTTxVKWH58FSw0a+Lw/wBJjnTlnsKsHH91GPtJOKnHmcqdNtRnh6SpWckv3sYOElLlqSnyQn1fxTudE1D4U+DdZ8eeLfgHpyeIfHOr3mi6c3wnMulyabJ8S92uaPrmu+Bf2e/CXinQyo3J/wAJT4d8VeEz4zk0LxG6/wDCXp4s3+D/AC+C8lxOE4gxE6WEzmSpUsTavRxdCa9tUw9SVBwWYZqqFWsryp06eNwdenHEVcLXrSoUIe0n7PHOLpZhkdKkqtBtqneDh7yjFx5l7lBuN/dbdOdO8Izi7uSjD6d8EW134ftfGfh4+KfhLPcXXxD+Jd74fOvftf8Aiv4N6hEvh7RtD0bWxrXw1+E3w/tvDcmg+HE8EPrqeFfEj3X/AAgfhKWHwfOsVsLky/E5pVxjzHA4lYXF1oUKGEbUMrpZnOVOFSNWnThjsVKvVr1qlOsqTrYetWbqN4mdT3Kh9ZlP1CviM3gq1Fc0ZKzxDhrKi05SowaUYxceb34xuny+/aPL8X/DnwZ4Ni+DvhPUNa8KeGvFen6FrnxktdT0PTPEPhLxRqF9rGg+Ddf1zXtY0fWl+IkcXh9fFX9jaB4Z8CqPhX4ul8JjQ/D3jceM1XxOZfGH6Hisyzb206MadaC+owmqk6WJWHVT2lJUoO2EhSqew/e1f3eZU6ldYitQq0JwoQrYX88ynh/Lq+XZpOVSLTxNnFOCk488tVLmbTd0taDjC0Z6uThLw39r7UPDeofEazv9A0jSNBvVh8YW/jbSdF17T/EX9nePtK+JPjTR/EUt/rf9geE/EGpXGvtZWnis3HiDw7YagU15QJpoVRLX7Tw7wmLpwU8RGFuam6E6VJ0YVKE6FGrFqn9YxSThKpKi2q04ydO8WotHz3FmPp4iElBvVWak+Zp7O7ShvZSXu/as92z5Tr9ix7vTilrp+i/VvW/3n5VTSVSXrv6r5/1rpsFXgf4i+f8A7abVv4P9fzI/VD/gk5/yWjxd/wBi3plfzb49fw4/P/0qJ/QHgn/Ef9dz+jiv5LP6qCgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoA4+98Z+DtNuZLHUPFHh/T72DBmtLzWNPtbpQRnLxSzpJg8YJTGMdcrSoYXN8faeGwlbkTWkac6i9fdjJre/RfkcdTMMvy1ezrYinTa1tJxTd/JtKTWu2v91NJRrn4h+Asf8jl4V9/+Kg0r+t2B/L2xXd/ZGdS92OExPNtph67f/ppLp0t582xy/wCsGVQ994yjZ/8ATyK/Fzfl08tCT/hYHgX/AKHTwl/4UWif/Jv+c98Hdzf2bi/+hTiv/Bdb+uq/pnZ9bw3/AEMsN0/5fUf7v97z/Lez5j/hYHgX/odPCX/hRaJ/8m/5z3wdx/ZuL/6FOK/8F1v66r+mH1vDf9DLDdP+X1H+7/e8/wAt7PmiPxA8DEfJ418JDH97X9JH55ugT0/xzwarE5fjIPmnlOJnbqqc4vpqk4p9vs306aMzw+b5NNWp4zD27Ksl/wAPey3X3njnx30X4VfG34VeM/hpr/jHwjBH4k0p7C0vJdc0xV0vXiyvoGoL5plHm2erpZ3EcOzfcFPswVhKN3q5JVx2DxFKtHLsSqcPdnRnRqtSptaxuuVxktHF2ai0pNSScDxM7p5FjYqM8TRtdte+lr3vr5/ajzaWtpE+I/8Agm58G/D/AMEvCPivxz491fwvpPxC8VaxeeHTBc63pr3GjaBo+qvZQ2vnrdNHFP4h1uM3bW3W+tbXw26ItxuRv1rxk4pxfEOPoQo5ViaWHw6dSfLTnL21eSskpSpcyVKnezpy5ZOrKNRSlCKj+YeEmUZHk+GlKWIpqbXWbSUVdtb21lbu1ypxSu1L9QP+FgeBOf8AitvCeTjP/E/0kfyuh/L8s1+JvCY+VNSlluKcX0cZ99Ps38/iX4KMv2f+1skVRxWLwvNov4ifTyev3etr+9J/wsDwL/0OnhL/AMKLRP8A5N/znvg7sv7Nxf8A0KcV/wCC639dV/TN/reG/wChlhun/L6j/d/vef5b2fMf8LA8C/8AQ6eEv/Ci0T/5N/znvg7j+zcX/wBCnFf+C639dV/TD63hv+hlhun/AC+o/wB3+95/lvZ80f8Awn/gL/odPCf/AIUGk/8AyXVf2bi/+hTiP/BdT+uq/pmP9rZN/wBBmH/8HL/5L+r9dOY/4T/wF/0OnhP/AMKDSf8A5Lo/s3F/9CnEf+C6n9dV/TD+1sm/6DMP/wCDl/8AJf1frpzdhDPFPFHNDIJoZvusOVYEHkcDjjGCPbjllxPQLFABQAUAf//W/vxPQfT+prLD/wAL5frEp7z+f/pSPCf2m/8Akg3xX/7E7VP5LXp5B/vlD/r5U/8ASKh4+ZfD/wCA/mj+PSv9Oj/O0KACuef7xOPfTv2/w3+77/tXQSi1JrVO/fv5rt/wUem6N8XvH/hiy0e10bV7LTF0HSdR0jw7d2fhnwsmp+GI9V1U63dap4f1h9KbU/D3iC41Ym+k8T6Nfadr6y4EetRQgIvlYng7CZqpe3pOpCpUdWpTlVqqlW5sO8NKnWoxqeyr0JUHaeGrRlhpyUakqbqxjUPocNxDPAWUHyuytJL3o8slK8XpKMk0mpJ8yWl0m0I/xl+IbeJdM8a3mtaZqXi/SbM2dj4p1Pwl4OvdfdT5S/2zqmqSeHJZ9b8XqkaKvjnWjqXjcjOdblYLu5f9ScHgISwVOEo4epOMnR9tWlTgoqmo0KNN1HGhhFGnGH1OiqeE5PaQ9g4Va0ZqOfVa3+0zlzTjd8zSUpNttylJaynzSvzycpP3dVaPLS1r4q+O/EDeKk1fWI72z8ZWGnafr2nzabpraP5OkCT/AIR6LTtHS1j0zSP+EYZ9/hc6PaWD6BtWPQhp8INuuq4cwmT06M6FFJ0otQqNynV9+zqc9ablUqSqySqVp1JylWqfvKvtKlpRzhxJWx8pUZzk1fVbrRaabJLW1rJLRJbF+P4v/EKPU/E2uf2+st34mTxs2rLdaTolzZLf+Pbb+z/Guu6Tp8+m/YvD/iTxFAR/aHijQE0nXWCKV1iKMsj8OYcHZLTrYalDD+yp4d4ZU1RqVaMlDCWdClOpSnTqVaMNU6NRypVIuUakJxlKMunI84zKlmGbS9rJ3pap8r+Km7tcyai+zVmt07oreGfi/wDELwdotn4d8O6/HpejWGpajq9rpcujaFe2J17UBa51+ZNU0qeN/FGmR6bar4f8WF08Q+FijSaBrmkz3FxO/bmvC+TVKMK1TDR9t7OlT9rCVSnUdKlWVdUXOnJN0J1FetRuqWJSVPEU6tNKB4uRcT5hSwGaw9rKyqvR2krznZuzXxJX5ZbxesOVpMwvEPjbxR4sTRU8T6pPrUuhWo0yz1LUVgl1aW1V3ffq2u+Uda124aSR5Bc6xfX9wrSSGOUedIX7Mvy6lhHeEbap2vpG0YxSjF6QWnwx5Vzc0uVtyccJ5hUxS96TbfXZ/Prf5y7XOXrtxDcrR7f8Dp7vT+9917S45qz9f67vv2VvMK6MD/EXz/8AbS638H+v5kfqh/wSc/5LR4u/7FvTK/m3x6/hx+f/AKVE/oDwT/iP+u5/RxX8ln9VBQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAfy/f8ABSvUr6H9qTXxBfXNtjw1pQzaXgGenbr05PKnnBPIr+xPA7C5Fm2TuVehTqzpwUXKUV73Lpe1mr9OvTWV7H8oeNua5lk2aQp06s4ptaJtb8u6tFaX/wA0npL4I/tPVP8AoLal/wCBVfrmWZLw7VzSpSeFp2Uv5F0XXbp/i87aI/Js04mzejldOsq9S9rv3u7X/DeV7B/aeqf9BbUv/Aquz+z8P/0IsP8AdR8v7vr9y2+z5n1vEaf8LmJ6faq/3f73l/w13yn9p6p/0FtS/wDAqj+z8P8A9CLD/dR8v7vr9y2+yfW8Rp/wuYnp9qr/AHf73l/w13yn9p6p/wBBbUv/AAKqv7Pw1f3f7Dw79Y0tb/JPT10veyNP7Z4hw9pLFVdPP/7d97dF+DD+09U/6C2pf+BVL+wMPG9sjw9vNU//AJBW37P5FvifiCek8TVtovi7fN9Px6u4f2nqn/QW1L/wKrPMMthjMVDnyWhK8lZP2flfXlf4LXu9ObXK8zzzC4SfJXqRdr72622u7dP+DvI/tPVP+gtqX/gVXo47IsNh8JTf9hYdWSeio6/h+r/LlWWZrxDicXO+JqfFreV7/c2tvL1tYP7T1T/oLal/4FVw/wBn4f8A6EWH+6j5f3fX7lt9nk+t4jT/AIXMT0+1V/u/3vL/AIa75T+09U/6C2pf+BVH9n4f/oRYf7qPl/d9fuW32T63iNP+FzE9PtVf7v8Ae8v+Gu+U/tPVP+gtqX/gVR/Z+H/6EWH+6j5f3fX7lt9nu/tTiH/oKq/f/wDdPJfcaGm6lqv9qaf/AKdqX/H5pn/L5+fp78fz60f2fh/+hFh/uo+X931+5bfZP7U4h/6Cqv3/AP3TyX3H9lHwym834f8AgY9v+EW8Pf8ApntTz1/2e6j1BIxX+aZ/oaegUAFABQB//9f+/E9B9P6mssP/AAvl+sSnvP5/+lI8J/ab/wCSDfFf/sTtU/ktenkH++UP+vlT/wBIqHj5l8P/AID+aP49K/06P87QoAKACgAoAKACgAoAKACgAqcD/EXz/wDbSa38H+v5kfqh/wAEnP8AktHi7/sW9Mr+bfHr+HH5/wDpUT+gPBP+I/67n9HFfyWf1UFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQB/Lv/AMFKc/8ADUet9cf2DYYznGf9P6dt33en+zj+Cv7S8Bf+RZ/26/zP428af+RhT9f0gfn/AF+y0f8AkYVPX9EflOP/AORZT9P/AG9Fizs57y6t7Gxg+0z3V59is7S05/z+X4tzt48zxuMdGVovZ9NrpL+tV8t5YU8uwazGj7yVpr8Wtr+dnvH10R+22h+C/gz4UvHsLHQ/BnjLxZ/wj3we+H8eg/E/wj4Q+MLaXfaH4P8AHeuDStC1rxF4h+A3hfRp/Esehv4bi8SSeKmtrnxxoKRzQKjEr/LfFdfPMTyVKrxEZQrYupD6pGvl/PRVaMaU6mHvmtfESmourTi/ZU/ZV1Gph4TUYR/prJMDlkcBTSlTvKnTi/auFXXl97llejCmpOybTlOPLeElrzeA/AXR/hReR/Fd9T03wbqWmt8fbG98OS2ZTUtKvPDegf8ACca5HHrvgj+xfFHh/QNC8MeHH19o2jUptgl8zwd4qk8JxeDJfqOJ8bmlRQhTp1VFYPEU5+/UnJznGjGlCNd4rCVXWck+WrJOo+Wf+04Sc+ar87kuV5dRs51YJqcGua2sbtybXs6kXFJbW6r3KiTjHV8dfDTwNL4/+G3hPWfDPgRLLxVo3iaOwn1z4Z+K/CVjZ+JXTQPEGsQeLtb+F/wg/YxgGg+EotA1eKRnPizxV4TmfxRc+OPBA8128FeNw3mGa0cJiZUquZU6lN0a8quGxOHxdWvRlRqSpUMJSzHFcQUfbV3OHs48uDWJlUoKFXli+boznK8tq1Y3eHlvDklGdOMXzJOVX2EMLK0XzPRSkkmpRbdjxHxno3wu8KfDH40aZ4e8OeJbC+utX8DXt1ow+JWk6haeEtUPjHxtoEeja74R1/wE/iLQP+ESfQblpP8AhKvEvi4OssKL42QwO8/2OX1cPRxNKU8Vha+AVNpOWX1Y42snSpctT67Sx6wbdSo3VqOGX0ISUXClQjBwqQ8jGYfN60XbmVS6u1VpunvJv937Pm0jypfvXZpt3uox9TbfbSWngjw9r3w58OQeIT8MfDV74dn+BXwRvlsNT8D6DosXjj/hNBrvw/8AFPiLXMeGtfk+J/gnxL4nACy+H/iP4PPlzRxSr4+Lo5NUr4h1P7QrN1a9blpZjnEVOnXnVqYaNJvF4bCUHRUauGlhaKnGK+q15VaEZKMrwtPPaKh71FJWiualhr3XI25e5Oc9bNTna6U4LRSjKt4Ru/hBbfDG11TxDp/gzUdJit9Q+OFxYyQfEPQvGt/4q0D4/nwFoera54N8DPJ8G9E8CN4a2+G5fDnhpZvF/g/+2/EjeBPBrSmVWWOo8UVsypUIY3GRjQgsBQxU/wCzsRSTlgqeJnCOIxXtc0qVpKnSrValVVKFethqTr1ak1OE+jLq3DFChOcsBBylLnlD/aKcrOVrqMIxoKCbaUYOMoJySjZRcflr9oyPQbnW/DPiCy07wTp3iHxLZeKL/wAX3HgDXvFHiXwz4g16P4m+PtFm8ZaVq/iPxH4vuJ7fxCdGF2Ik1+9SJ1lZooXlLXH6j4fviTB4ZwWaVZwpSpwgsRhsNhp0qcsNhqsKKoU8PhfZRp06kbU6lH2tOUp0525Ixh+c8ZrhrHTusv0bb92dV395pu7c2/eV7p22ev2vnOv0WhjcVTSbi7d7d+nX/h9NNz47EYHC1WvfV313/rZf0i9pv/IUsP8Ar80r+ledj8ZiquvK33/rv+ux6OBwWFpfFNK/9fkutl163l/Z18Lf+SeeCv8AsW9Gx0/6Btv6f04/Sv8ANOr8UvSP5/d/WnU/0YPQK4Kv2vl+gBXQAUAf/9D+/E9B9P6mssP/AAvl+sSnvP5/+lI8J/ab/wCSDfFf/sTtU/ktenkH++UP+vlT/wBIqHj5l8P/AID+aP49K/06P87QoAKACgAoAKACgAoAKACgAqcD/EXz/wDbSa38H+v5kfqh/wAEnP8AktHi7/sW9Mr+bfHr+HH5/wDpUT+gPBP+I/67n9HFfyWf1UFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQB/Lv/AMFKc/8ADUet9cf2DYYznGf9P6dt33en+zj+Cv7S8Bf+RZ/26/zP428af+RhT9f0gfn/AF+y0f8AkYVPX9EflOP/AORZT9P/AG9BXZSxuDxas4J37r7ujtut19+5wZtl2LwuOhJTkrSTvr0t66+n3q9z062+NfxP07RZ9B0rxZqOg2FxZ+CbC8k8Psuj6re2PgAj/hCUk1zTDb6yB4ex/oRF6pGf33m7kK/O5nwrk2NqRqywlGU4e3cHKnFuP1iXNiLX2dZ61GrOV2ndSaPqY5/mWDwcIRrTtaKfvP7Hw/K23a+zesdG1+OfxLsB4j+w6zpdrN4s1Q6r4lvF8J+ETqniO/bTdZ03zNX1ttDfWHItfEfiJEFvdQRH/hINTlG19SmdliuGcoqVKVX6tDnw9CWGpJOcadOjOdGbhCir0VrQo8slTU4KCjFqLaly/wBt5hTTSqSV2m9dW1G2979ZaN9b6te9H4j+PPxL8Uabe6Pf6poNlZ6gNTGrReGfA/gXwJLrJ1q40W/1STWf+EK8OeHV1WW4m0HQt0l+lw+dMsjkyJI75ZXwlk2Ak5UqdepJyjOLxmNx2PdOUY1afNReOxOJ9hJ061WE5UVCVSE+WpzxUYx0ln2ZV2uacb319nTpUdG4uzVKMVJe7ez0jbS15c3beC/h5+038QvBN+vhX4d+MvF3g7xFrum6ne63B4FN3c69f6TGdF0pP+E5Hh+TxLqOi+HoPNs7fRo9dl8P6K4llGk2l7B50XzeaZp4c4LFt18Zh8PiY0FT9jVzCdKCpzl7RTeCnWjQdRtWjiHQ9uoXpxquEpQPosDg+OsRF2VSpDmTclThK0lpbnUXJL3tY88Yu8W02onZXV9+19B4y1rw1e/C3Ubnx/ef2n4g1m1/4Zw8Ian4013TdX1Vjrur6zrJ+H3/AAkOufD/AMVa+H8NuUVfB5C/8IU+7wIisnh4at4YSp/WHjeXDqMcGqn9u4+OEj7PC+yVOC+vKhTxEaC9qpQUcQqkVi/aOvF1y8ZhePKTjTUZqS9+yw9L2jU57y9zmnFzaiuZ8uvJGMU+WWJ4t1D9r6w+H+uaF4s8J/FTwf8ACiS3u7XXNBl+GfiLwd8P9DXWfF6+O2MVhHoWm6DoRbxPKtutzFDZvbzOPD6SxQ7dEXoyjD+HeLxtCpTzPAVcxkoyo1FUhVx9aMMM6XN7V1ZYmvy4RtSlKUr026knrzlZlX4uw9OaWXyjBNxcfdjCMpNXTWkINuO1vsW6Wj49c6h43+MOvaPpvl6TfX+m6RqNlpGk6bpPhHwRpVjpOkLrHiXWBp2kaDZ+HNDgjhR9b1y7+z2YuZQ15JHFLsgt6+xhT4Uyqny0cwqqLlFt1J4mvOcmoQXPVrVJ1ZvljCCcpbJaq7Ufm4/6zY6bc8vg7dL0tFduy0jHS7269t5XPAfwQ+L/AMTYhf8AgD4aeNvEumtPeQjWLLQdSm0D7VAAz2T640SaPbyoGUlLm9gMayI7bEZWfvzjjfI8unGhisbgcLWqQU4UsRiqFGrKDbipxp1ZQlKLkmk0lFyi0m2jly7hfHYhScI1ZQjZSlGEpJfa1auo2V36a3VmzC13wH418A6/pVh458IeJ/B2pXiW11Z6b4n0DVNAvbqy+2sn2sW+rW8E5jLxOgcQ7CyOoLFGFc+X8W5ZmMPbYWth8VS55QdXD1adWmpxtzRc6TnHmV05RvdXTbSfvd+K4ZxeHtGXPCSSupJxavbVxfK9Vaz0380j+xL4W/8AJPPBX/Yt6Njp/wBA239P6cfpX+ddX4pekfz+7+tOp/oGegVwVftfL9ACugAoA//R/v4oA5Pxr4W0/wAa+F9Z8L6tALnTtcszZXduSMbGIY88DqBkkdhjHFAHwZ/w7s+A/wD0T7TP/A1a/U/+IkYjuvuPzn/iHGH7/jIP+HdnwH/6J9pn/gatH/ESMR3X3B/xDjD9/wAZB/w7s+A//RPtM/8AA1aP+IkYjuvuD/iHGH7/AIyD/h3Z8B/+ifaZ/wCBq0f8RIxHdfcH/EOMP3/GQ3/h3d8BDgDwBpmf+vz3x/e9fXP0PWuSv4m8UxaVKSnFvV2hG338r/D7rWlcfDzD1ablJ2fTfX0eiXy1fZauPw5+1t8Nf2ZP2aNY+GOiy/DjT9Rv/E2tvdeJLRrsg2HgVGK67reDgnxA2PMR/mEkeheJGXCtmv1rgKtxdxnTqzoSjNU5SpOfLBqNfli+VrnjeKg4VLp2nGrBwkuWfN+ZcVVMLwnU5JSX39L2+Tvpvo0+mkftnT/+Cfv7Pmq2dpqGn+C7G8tL2ITRTxXd0pntm5Ac+cMtz/DtbOctlVDfnuK8QsVBzjUjGM4Wule2qveLfNeLsnrr0tL4o/eYHg/D1Ule6avs+lt9bKz3d5b+vNc/4d1/Af8A6J/pn/gYv/1v5/lXNQ8VMTS0TXzjf8uVq6tr+W53V/DrD1uuumztbbze6t1+7QP+HdnwH/6J9pn/AIGrWf8AxEjEd19wf8Q4w/f8ZB/w7s+A/wD0T7TP/A1aP+IkYjuvuD/iHGH7/jIQf8E7PgMenw/0z/wL/pvOfwI/GqwfininHl0SfePNf7uX0ev3/aMb4fYW6nGXMr9mrW9Vr07aLZ/Z9e+DX7Jnw4+CviWfxJ4P8OW2i315ZCzu/sd317j8+DyB1xxj5fneIeM6+YxcHFSjN+9daWuns1r8K3Wu65tIR+hyLhnDZfFKT1jpdNvdbaXe3krd9D7Cr4w+lCgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoA/l2/4KVAL+1LrYX7o0DQdufu/6vUM46jGNucdsZxlRX9h+EeNxUsjjRhCTUKSiu93dyfwq92309Wr+7/JHinhMHSzSEpTim5J6vf4baW8vK/Z2fLtfGbSvgH4i1/4k+J4rzwDqOk6p/wnWt3nxF0r4lf2j8Qv+E8HxO1oaJpOj/DRfH7prngZfDx0RU8QD4VqrozeOh4yfb5sHRw283w+Py/D1vr0GsvwEFgpZbOlhI04YOjGq8RioYGrTw2Op1YVWsP9ehS5Y0qCoqco1jwsxpYCrld4yo29pK01UvUldtpRpupGTptNP2nsnZvV7wlhabYfBrSfGPxI8O6/b/CrwZ8Odb8MfYPCVz4N+KT+Lv8AhOQPjH4BHgn/AITI/wDCwvFniTRVA0Ut4zbwt4R8I+LfB/hA+J0/4Q9dqlIWOzqccPVpxzDF4ilmEva0sRQ/s50OXAYqniIYRRwsMLiaa9o/qksbicVl9XEyp/8ACpJwpzO+pgsuV1KpSgvZJxlCcanM3OLi6n71uEuk1FRqQgr+xlzSUcPxF4a+At54ii8MaXo/wp8Nz6H4i+BQutftfiN4v8R+GdbPjjR3/wCFqaRrI/4WIfDf9geFvEJBI8L+LvCK/KpHjQEsJfTweMz14XFYqdfMWqlHMnQo1MHhqdXDqniKqwNWhRjl0sX7aphuWfJicNjOaShfCc/NQq+bVweW8yjH2V1KDcvaSfNeMeaMvf5eVSTs4uD1d5tJKHb+JPDX7LfhrVPBhTw18Mdf/wCEp8SfC/w74k0pPiPq9hY+FdK17WPiu/jjVXHgb9oL4pGJR4e0XwbEYz8WPF/hIosTnwXC5dK+ey7F8XZrhX7TEZjRkliqqqLDYOpiaso08K8NSlHFZNlsIU+Z4i9L6hTq1ZRi/rsIRcD1M4weQ5fJ+zlSntG3tJqEX76lNcteo5P4JJutyJ3XJJNcrfh74V/ZW8X6PaeIdS0XwFomuFhaX3gJPHgNkumjXfHGzxcNW8f/ALQvwqRtdm8N/wDCG+dKfGPixoVC48EjeZIurN8dxRk1OmqE8ZiIyxCnKtGlSdenCMKKjhXSwuSZhGrRqtV6tSrLD4edL3YLEx5oozybBZJmF+epSi7aRlJqLupXlzOtTalHTkh76m90tpdD8ING8Ax/B+0Mmm36+HbL4j6Nrni/xTplnonh/VP+ET0bWtf13WZdd+J+ufDxX16Pwu/h/wAH+I18MeF/+EtJfWPD3ghfGzp8VvB3g3wfPHWY5pjeNsxxdC0KteNFUIVKNTExlz06MKVWNKhieaFOrNYym3VeGk/q9WuoSw+DxFer0ZZgcvp8OZdhpyv7CVRzcZqDjafM03KEk5JOEvdUtJKOkpx5edGp+A5f2h/+E30vw14a0DwDqfwy8Q6vaav4t8bfCVNM+JOn68dY0HQ9YdNc+H0/h1deaVm8N+OfDDeEvF/i2PxToniVZ/8AhNPGqSTTy8NmNXgnHZXWqe1xVbMKNGccJhcylLD1nWo4mdCvFYmtiadNubq06854fBxwGIwrdJYPk+sFXHZdV4kwOKpySp4ejUu6k6T5ounKF4uUYwcko8qSi6ntYvlvU5VH2H4p6NrcM3iu/wDA1zpGu/Ea78d/2v4R8Oxayvh2RPhA3jP4f+OdY8Xt5b/CvxHofh/wx8RvBJ8LgibwkG8HnxF40DP4I8Kjxk3zWWLFxhRp4zB4qjQUJUZ1q0PrEo4qdFUcPhpQisd9ZnjKVaoqMpTrKriZYfAt18bio4eXtZtWy+u5ezr06k9JpQ91OC96pNfw3FU2ldKCShz1HyQg5SxT8W9I13x9pU/gq+Xx6+jeD/Ezy+KfFGheD/jBeaB4X0rwf45g8Zw61408d+H/ABV8SdB0FvEgh8Q+AfFBQ+DvGPg7xD5fjUDzCH9LH8LV3QpKrhqOXQeKnRpRweInl1KpLE16dHAwdLD1qNOtiZ0fYUZqU3L6zCVPCQqQqQ5jKMbgoSk/bKrNpTftIRqy9xOU5XnGUlFe9J62cdZt8vu+NeAvC17rXw08CNd/BfxR8TNVmHj7x/p0fhfwtq9/p9rpfjTxmPBGv+DdXTQfDo8N+BP+Eo8OeC/Gnijwb4oUkeCvGOh+GB4G4ID+9nOb8mKmp4ylg53oYTlxmO+r/vMHShj6VWnF1KdXFyjLF4WGJo+3w8a2HrTjiZ1YunTn8blWX4SNNy1lGPvOUYOSiqnucspJcsb8kuRtSalFcuvMz5++L/heTwh4+8OeHIPCd9oGm6b4c0Oy07W73Sr6w/4TnbEo13xnpVzdyPa6/wCF9e8TR61c+Etb0ow6beeHF0xVLTQTqn2nC2f4iVOpKE6Nd0sRWoVPYS5lQnTndYevC8pUsTSpSpqvSqKE4VHK8YRlCMfJzzKMJUcVqnKMZrmurxl9qPSUG9pLSS+Z/Wn8Lf8Aknngr/sW9Gx0/wCgbb+n9OP0r+B6vxS9I/n939adT+7T0CuCr9r5foAV0AFAH//S/v4oAKAG4+n/AHwf89v85FHPDz/8CX/yPn/w1nyr3u6+4MfT/vg/57f5yKOeHn/4Ev8A5Hz/AOGs+U97uvuDH0/74P8Ant/nIo54ef8A4Ev/AJHz/wCGs+U97uvuDH0/74P+e3+cijnh5/8AgS/+R8/+Gs+U97uvuK/02sR+H5f/AKx0zxmuKUszlUjywp04N+8rxenldtrbvf0vEtp0oNXlJpWW68/6Xl2Pyu/ay/Yj8I/FL4k6L8RvEvj7x02qeMvGHgrwR/ZkE+i2+maLoEfn+fHpKR6BNcLcMwebzr6e5gM88kzxO80pf908P/EjOeFcLVoYKMHHDYatXgmlyqpeU1Fq6bi5uTdpLS6Shoo/iXGfAGJ4nruaqO0qnLrpaOmrt6NXS15b63P0B+DPwzh+Dnw68N/Dm18Q6r4lsPClkNN0vUdbj02K/GnQn/QrKU6XZ2VsVtIQsELC2UiJUTCoiRr+SY7GU69SVTl5bxUWr3ba0vtZbx2f962tz9ZwGXVMNFc0r3lzdLaaaa+V91+svVi+Rlf8fy6Dn059RnqvFCFOTu46vVfdfXTRpNf/AGu56FTnitLX8/03X381vPVxdj6f98H/AD2/zkU+eHn/AOBL/wCR8/8AhrPlfvd19wY+n/fB/wA9v85FHPDz/wDAl/8AI+f/AA1nynvd19w04x1259OSfbP+c9B0NTGNKjdqKsvtbfh73ktP/AX9kalOyvZ/f8r30+9/MYGXoHP5f0Bb+f5/w261N3ulpo7/AIbw19Vbv1vJKjNbTfpaP/A/P7iekUFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQB/Lv/AMFKMj9qLWhzgaDYY64z/p/TqN33eh/u4z8lf2n4B4/B18r9nKClaEY372Tu9UrO3e/e9z+LPHHB4yhmcGpS+JPfa9trWX5/mj8/6/a6CwKzCcvZxTb2XXTvp27b9j84VLG1Mtp3m7pLq0/vTl5dfv8Aslc1PC4H+SK67LTt038tFvqxTxWPt/Elbd/5fF/lr10vIr0IYXA2t7ON+unfTt83ppZ6y2OCpicf1qy7vX7lu/y7as9s+H3wM8S/EXwd4j8WaNPYg2B1O08OaFca34NtdR8WXvh/SF8Q+NI9M0vXfF/h7xHdHwt4VdNeuF8PeHfE0t5bma1EVnJFJKnxHEnGWR8MYmjhlhnzVLSnKOHxcqdONSTp0nKtSwtXDwlUqxceWviKTUU5+9pE+nyfhbOeJE5OtK0XrzVIJvvaDqQlJJa+5GVutr3IPi58FPEPwfv7f7Xq/h7xLok19e6Jba9oHiPwjrjP4l0jTdEm8R6RqGkeGPEvia40K80S61m2tp7PVZ7a7dmikhEiuQunDPF2R8USqQWHnzU1CUva4XE0YtTdRJwnXoUo1XenLm9lKXJeLlZSpuc5vwvnXDaT9tK0rpNVIN3iot3jGUnFLmXxLXWzbjJR8ar6TF4HAVclwGYulHnrTd3bV2qJavfr3VvPY4MvxWOqYvM8G6suXD000vWm3t5W8/PvHp7jwR4qsdD1DxTfaRJa6Zpd34VsrqedhbMj+MtI1vXvDrCw37imq6VoGqXlvMPkaK08wEb4g3LiMwyuGe5dlUXSkq9OvLki1Lm+rezhW0Umn7GrONOru4TlySUXdHjcM0MwzDBZpjJVJ3w9WEVLXS85pLm0+JRbWmy66s5itqeGy/CNONKNv8O23m+/lbtuaPF5jW1dWSWnW/y8vu3W8teUr0aeMy/ROnH/AMB2879e+0RrF5jRs1Wlb1tb5+vr5d4lcf1bAT3pxvpra/VdrfP3Y6d7XM6eKx0JJKpK/q/xTv37vS+20b2m/wDIUsP+vzSv6VPLgqHw00rW7K+2979ddvPW6PSpyxlf4qkpevT/ADau+9vxj/Z18Lf+SeeCv+xb0bHT/oG2/p/Tj9K/zRq/FL0j+f3f1p1P9Gj0CuCr9r5foAV0AFAH/9P+/igAoAKACgAoAKAGA5Xk/wBMfz/l+B/izf7hrXT0/LXXf+6vuTJTU4+T8/6t/Vzxf4waXqN+Phq2m2F1qAsfiv4P1G/a3tvtB0/TYE1VbrU5V2kRwW3mRiWYY8vzF+XJxXtZbXt9YaUbVcJXhZyte/L7t73Stfs1rvfmj4uY4SPue9ZKpF99vlZ2/wC3dvkezIATkNnH+fX37bv1Bry2mt9un9a2v6/fZHtKoqmqle34f16fePZd2Pb2z/7Mv+fTHzc8/f8AdX9bPaytt3fyKHVsAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQB/Lt/wUoJP7UOtHnB0CwI64z/AKf06Dd93oP7uMfJX9seAtC+WPXemnbv/lrbq/la8v4/8WsS/b+jelvS+1rX8l8l9n4Ar9ppYf8A4UZ+vful5f1/d3l+OQxLVFaa9t+3Vv8AP0tpcKfsYmXt/wC5+Pp/wfwCj2MQ9v8A3Px9P+D+B9o/BD4yeC/AfwquNF1PxfceHPFwu/jnd6P9l+GOkeIRjXfg5oWg+Cv+LoBG8SeAGXX9DDI3hQ5Y/u/HuwEvX43x7k2bZjxPSx1CjOrRjTy+GmOrU4RcMZVliJfU1TeGqctCpGp7WrP2s3CNKmoSUZn6bwjmWX4TJK1CdVRnJVHb2cG/g9xKfNzR5paNWaj8X7xu0en+G/xS8IJqngqfVvG3hfwzqA8P/GDxN4i1b4f+H9Y0s+FdS1n4B6D4H0Eg60fCfhgeP28R6J/wk23wsf8AhDm8YDy/A/jQKWDcfEOTZxiMprwjhsYpe1wlCnHF4ih/tFHD5lWxM6lP6m6zp4KdDEuhD6xy4mph6Xs8XhY8rniNuE80y/B5xOrKtTalzSvFStGU6SjZ+03mmm3ytxU3zQnouXa8C/tY+AvCfjfUfE+q6x8Z9TuI9M+GOk3uuy/8JYl/8Sf+EJHiNfEOteNdE0L9oPwwP+EkBm0bQB/wlPiz4tu/hLw6iSeCxC8obz838NcfWw2DhQwOX4aVOrOdSlQxMPZYWLlZfU5VMjqRqc95V6nNhsNONb3VUnTcnLqyfjaEKk5VKs5qUUuaablNxs1z2xEGleMYaznaL5lGMopFW7/bMlstJ8MjRfG/xlGqLrnwel8RaVc3YOnWXhzwRouu6H480vRda/4WEo13QvFrtpHidvDA8J+EGcEOSWcom+V+GdZV5e2weAnhoe2q02+WVSriqrwzp1qtGpgpKlUglXpOtTxNVyXLUdJucaeHjNuNlUS5atS8oqE1slC79xNT96LXLLlcUlK6SduaXwPrU1le61q97YW5t7C61S+ubK1Jz9lsTMxij6gnahVcEDGMLuGzd+1fWMT/ACPp1f8Ad/u+b/4Fny/DZlTwdeOk1f7+i/4bd/K15Z9H1jE/yPp1f93+75v/AIFny/OU8DhoSfvLe34+u/fX1bCvXw8E1p08v/tl/Wt3oi8RiLu1/wAP8Pp+f3F7Tf8AkKWH/X5pX9K4MwpRTfX+l/X/AG75I9HAYh8v62+Xft5fduf2dfC3/knngr/sW9Gx0/6Btv6f04/Sv8y6vxS9I/n939adT/RY9Argq/a+X6AFdABQB//U/v4oAKACgAoAKACgCBc7B7kD9P8AP+RWOK1nBf3f1/r+kZYX4I/N/cj8tf2p/jH468D/ABG0hbP48XvwC0CXxB4Jtbmw8c+Kv2VtPbxJ4Z1/xf4c8P6zrHwr8Fa74A8Z/EP+w4oDr/iDxv4s+JvjLwkvg86NK1j4N8W28twvhP7rhfJqWOjKNTDYjFxjSrzaoUMTOMJUqPtYKpUhVpxdSor06NKjCdWc6kOWcE/e+H4tzyWAklGah70UuaULv37aRevKnrKUnGKUdbNKJ9P/ALMPiLxd4l8CXGpeKvEOpeP5TqbXOm+PbXxl8H/iN4O8V2DuUaLwH41+FXgL4UprvhzwxOHhd/FXwr8I+KofEkWuW8Unimzhtr1fFzejGjpGycoqcoezq0J0nLo6VSU+WWjTcZyjyxpqKUFCU/S4dxUsVLVtpStfmjOMrWd+ZKCdk+azUJczk5Xk5xPqpXOT7Hnnr1/2f/iv0zXz1Bbvd638tUtH3110++59hNe76f1p2/Ht1vGxW5kFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAfzD/8ABR3StR1D9p3XJbLTr29txpenZmhhFyoxbRK2P3nyBSrIcqdm3ZuO1hX9leA+Ptl012jbbe1/P0/mvt5n8n+K+Aiqq0/Xtffl/patWTl8H/8ACJ+I/wDoXte/8EuqY/LbjGOMbf8AGv2ajj3/AGjPtzf8Hyvp922u5+OQy9ewimtet/VW1t2/u+Wt7h/wifiP/oXte/8ABLqmPy24xjjG3/GuH+0337d/7v8Am/u9TJZdtp26/wCD/p36/iH/AAifiP8A6F7Xv/BLqmPy24xjjG3/ABo/tN9+3f8Au/5v7vUFl22nbr/g/wCnfr+If8In4j/6F7Xv/BLqmPy24xjjG3/GuzA53gcwj7adOLlHuvx87JabfO15ZzyTG4OfsVOSTst2/wBY73v/AJWtI/4RPxH/ANC9r3/gl1TH5bcYxxjb/jU4jPMDWm6Tpqy0227d/wA15J2cSqmSY7Br20KkrvVtP7uy/wDSu2lryP8AhE/Ef/Qva9/4JdUx+W3GMcY2/wCNNZ1CrZNLTyT3+Wu2+nmn9nojlkqSWrXnqvLu/wBPTVB/wifiP/oXte/8EuqY/LbjGOMbf8aP7ahSey3T6a2+63zevluJ5ZOqrt3el73fTteOnz+7XmP+ET8R/wDQva9/4JdUx+W3GMcY2/41X9v4T+Vfd6f5/n2IwWSYuv8AFJvXvt/Wnb8bSP8AhE/Ef/Qva9/4JdUx+W3GMcY2/wCNH9v4T+Vfd6f5/n2Hi+HMTFJ8zXf+n303X3XD/hE/Ef8A0L2vf+CXVMfltxjHGNv+NY0Mzkl1t6+nTpt3fbXcWIwC6+V9v8/NdOnkuaxp/hvxEdSsx/wjevfNcWiAf2RqfJk+6P8Aj3xlvcKBnkHDM3Bj8zlffvp6/wBJ9e2l+Y9DAYDbtp0Xk+tn+Pra6P7IPhb/AMk88Ff9i3o2On/QNt/T+nH6V/nZV+KXpH8/u/rTqf6DHoFcFX7Xy/QAroAKAP/V/v4oAKACgAoAKACgArJ/H80B8x/Fr4Z/DvU/Guj+NvFWq/FCz1bxDb+Hfhra23gb4p/Ez4d6fJFaan4g1XTn1W2+HPirwsur+Tc63qg3a02oRRxXDLHEACE97L1UlTlGmqcXSX1jmnGM7yt7kVGXOovd86SaslpzM8LMXHm15teWFr2stL6pa9NPP4pWtH1n4bfD3SPhn4Zg8KaFe67faZb6nr+pwzeJNd1XxLrHneI9c1PxDqAudd127vtb1IjUNVuzHPqt9eXnlGOOS4ZFQpy47GvGVfaShCEuSEX7PZuEUrvdbJJKPKoxSXvWbOnLsu+opq7ereum67Xffq38rR5e92YyTx+v0/n2DH8hXmxprmut9emr++Ttp1T9E7+967qadvO9/wD2xdfP7yarICgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKAPM9c+HOg63eT3txaWpnnXBZrYE8DqxwPfupP4161PMXCEYyu+VWu2m3bbpd7rrfvez5fPeATbelrrr/h6eevXvvqZP8AwqHw3/z46d/4CD/Pb9fY1f8Aaf8Adl90P/kvP/hrPlP7PjpqunX/AA/8H9dpC/8ACn/Dn/Php/8A4Cf/AGP+cd/4j+0l2f3R/wDkvT9bWly839mP+Zf+TeX+f4PsH/Cn/Dn/AD4af/4Cf/Y/5x3/AIj+0l2f3R/+S9P1taXKf2Y/5l/5N5f5/g+wf8Kf8Of8+Gn/APgJ/wDY/wCcd/4j+0l2f3R/+S9P1taXKf2Y/wCZf+TeX+f4PsH/AAp/w5/z4af/AOAn/wBj/nHf+I/tJdn90f8A5L0/W1pcp/Zj/mX/AJN5f5/g+x4r8APhhotz8NreSa1hUDxh8USNsBTBb4o+NOO5zgg5xjgEE/dr6Pi3OObMa9N6qnOnyttbOjSk0l/iu9WtXfXY8Tg7AKODUn1WvRXv3d9Vbvp3dve9o/4VD4b72GnYxn/j06/ToDj2/M5BrwZ5vzUo0+Rq2l+aPW2tve/B+Wqb5fo6eBSruTel9ezXk+RLTfT7nZKTv+FP+HP+fDT/APwE/wDsf847/wAWX9pLs/uj/wDJen62tLl5/wCzH/Mv/JvL/P8AB9g/4U/4c/58NP8A/AT/AOx/zjv/ABH9pLs/uj/8l6fra0uU/sx/zL/yby/z/B9g/wCFP+HP+fDT/wDwE/8Asf8AOO/8R/aS7P7o/wDyXp+trS5T+zH/ADL/AMm8v8/wfYP+FP8Ahz/nw0//AMBP/sf847/xH9pLs/uj/wDJen62tLlP7Mf8y/8AJvL/AD/B9j1DTrKDT7SCxgPFsMd8989fXP8A9YcFvGPZNGgAoAKAP//W/v4oAKACgAoAKACgArJ/H80B4L8bfv8Awm/7LR4F/wDQNar3Mv2xP/YFX/8AcR4WYbx/x0v/AG490QY5J6d8evH+I6flg14lJNxWjv5vfrf5qx9BUaVvLfr/AF/wfIWSTYM84/8A1jpgfXrxxwcZYqVFC2mr69Fqt9+/bfTW4Qhzv+vz6fdIlrQgKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAZ5g9D+f8A9r+tZ+zqfyfjEnnj3/APMHofz/8Atf1o9nU/k/GIc8e/4D6nkl2/EoKOSXb8QCjkl2/EAo5JdvxA8G/Zz+b4Zbe48ffGE+v/ADWHx5/u+vqf1r6HiuDWaYr/ABUfxo0vN26Lr8tj5vhWspZfD0d+n257aS9Nvv8As+6nnK+mMHr045HGOv8AfP4/w+FOD9it72S89PPm6pa6Lvrax9HGp+8ku2unnvrre1/5V28ySs+SXb8RhRyS7fiAUcku34gFHJLt+IBWwBQAUAFAH//X/v4oAKACgAoAKACgCBQduH4yeO5/Ec9vb35Joq+znZOSuuz3+7TS/n57XjlhlVjH31r63++36vtvqeF/G773wmGP+az+BgP+/et/55H54r28pUILGLp/Z2M5bPq400u//B8r3PIzj2k3CyslVppLTSzld/Z+5y77aKXu+V24zn8Pf3rw3JLdperX6/1+B7tn2f3Ebhccpu7cHr/nrj/69P2UJ63Tb3/D+9Z7Lt2vu5Zzr1KW0HZdmu/W8b69mvP+6WKCwoAKACgAoAKACgAoAKACgAoAKAKf/fuuX/hT70v/ACX/AOR/q/T7OHLge0fufX/t7pzf+S9LvlP+/dH/AAp96X/kv/yP9X6fZOXA9o/c+v8A2905v/Jel3yn/fuj/hT70v8AyX/5H+r9PsnLge0fufX/ALe6c3/kvS75T/v3R/wp96X/AJL/API/1fp9k5cD2j9z6/8Ab3Tm/wDJel3yn/fuj/hT70v/ACX/AOR/q/T7Jy4HtH7n1/7e6c3/AJL0u+U/790f8Kfel/5L/wDI/wBX6fZOXA9o/c+v/b3Tm/8AJel3yn/fuj/hT70v/Jf/AJH+r9PsnLge0fufX/t7pzf+S9LvlP8Av3R/wp96X/kv/wAj/V+n2TlwPaP3Pr/2905v/Jel3yn/AH7o/wCFPvS/8l/+R/q/T7Jy4HtH7n1/7e6c3/kvS75T/v3R/wAKfel/5L/8j/V+n2TlwPaP3Pr/ANvdOb/yXpd8tjIYYGGPrnb+XXn15568YxXVaErp7Pp8X5ST08m/Nm7U4dNV12/+SXXv00vd8rQB12n3y2Pbngfh0/Hk0OlT0TtZbaX39HH8/kr3klVqP7LXq4+X91/1fYmoGFABQAUAFABQAUAFAEBxx1x+HX/OOOv6VEFHC4drpTd/V3Vtvu0IatWi7av5a3Vlu+/deuvMeBftFfHLS/2dvh8fiHrXhjxF4n0SHWdO0zV4/DI0+S+0m21Eyxrq08WpXunxNp8NzHDbTeXdNObi9tY4YpWfZXvcLZVLiLFewp1o0qkoSlFuHOny2Vvii95JrWTtFtKytHxOJszeCwqej1s9Vfe/rfo9Yrzf2fkv9iL9rrwv8XLi5+FPhzwP4xiutOu/HHi3VdduodLj8P6fZa/421XW7GEOurT3vnf8T6KzeEW2wTI7RSyxAtX3XifwbPKsZUr1a6lGq6KgpwUZOcKNODu+aSnKo4OdlDRNp3UVKP594WcVRzHCKjyyvrZ3u929neyXM72dut3J+9+mO3aB/njt9c4znj36gt+Vw9xKF7JWjf002aer82u+tj9h+KXr210+5X09PVWuT0iQoAKACgAoAKACgAoA/9D+/igAoAKACgAoAKAPnn47/tLfBj9mzw2fEXxa8baboCTRTNpmjvKt14h8QPDFJJLHoegW/m6tqzxrG7TiwtZlt0xJcbIzXoZDwzjs0rwpUbSbdnUqNqmrW0dou8tU0lvrqrNHi5zxLg8roynNXmldRj0636Pbvd7NJXufzz/tF/8ABXH4pfEvxPog+EHhqw+Hngnwn4msPFOmr4mtxrfijxbcabhltPEn2eddL0TRrnzHivtC0KS71VHj32XjF4ZvJg/pLhbwbrzoe1r1Pa1KlKNOdlGMFFSk0oK/Mm1JRlJybnyKSUOZqX4FxB4vUaVWUEuWMZbXd76eltrpdnZ30Z+jn7Ln/BWz4QfFb+y/Cfxuitvg34+uUjt31i8uZT8MNb1ASCGSTTvEd2p/4RISc3kmm+MLmO0sIZbSwtvE2u6i1wqfmPGHhbmWQxlWpS9vRhG7SSWIilBucm00qt3FKKhGM3KrbkSjzy/QeE/FDB53ONGceWUnpNOyvJ266xSTv7zl8Oj1SP1yt3WWOKa3mE8MoykoIIYHOCOnBPHAHPXNfkv1TEYepKlKUo8t+ZS3SVvxta2+j87x/WYYijXgqkGuV25UlvfzfX1SvsukpaFbEBQAUAFABQAUAFABQAUAFABQByHjPxhoXgLwxq/i/wAUXw07QNCsze6tdGAnEQwpYD5ieSBtA4zguP4QD5I/4eH/ALKf/RRv/KXJ/wDE/T8u2fl+5/4hvxb/AM+Yf+Bx8v7vn8td+X3vif8AiIHCf/P3/wAoy8v739WW1/dP+Hh/7Kf/AEUb/wApcn/xP0/Ltn5T/iG/Fv8Az5h/4HHy/u+fy135feP+IgcJ/wDP3/yjLy/vf1ZbX90/4eH/ALKf/RRv/KXJ/wDE/T8u2flP+Ib8W/8APmH/AIHHy/u+fy135feP+IgcJ/8AP3/yjLy/vf1ZbX90/wCHh/7Kf/RRv/KXJ/8AE/T8u2flP+Ib8W/8+Yf+Bx8v7vn8td+X3j/iIHCf/P3/AMoy8v739WW1/dP+Hh/7Kf8A0Ub/AMpcn/xP0/Ltn5T/AIhvxb/z5h/4HHy/u+fy135feP8AiIHCf/P3/wAoy8v739WW1/dP+Hh/7Kf/AEUb/wApcn/xP0/Ltn5T/iG/Fv8Az5h/4HHy/u+fy135feP+IgcJ/wDP3/yjLy/vf1ZbX90/4eH/ALKf/RRv/KXJ/wDE/T8u2flP+Ib8W/8APmH/AIHHy/u+fy135feP+IgcJ/8AP3/yjLy/vf1ZbX90/wCHh/7Kf/RRv/KXJ/8AE/T8u2flP+Ib8W/8+Yf+Bx8v7vn8td+X3j/iIHCf/P3/AMoy8v739WW1/dP+Hh/7Kf8A0Ub/AMpcn/xP0/Ltn5T/AIhvxb/z5h/4HHy/u+fy135feP8AiIHCf/P3/wAoy8v739WW1/dP+Hh/7Kf/AEUb/wApcn/xP0/Ltn5T/iG/Fv8Az5h/4HHy/u+fy135feP+IgcJ/wDP3/yjLy/vf1ZbX92P/h4d+youBH8RFHru0+cn/wBF4/P/AArLDeH+ccrlH6quvvVaq2/w0Ur/ACf4tzeL49y+LjGKqavol8t3JLR/8Bs9d+Ef7UHwb+O2qajovw28VJruo6XardXipA8aLGWC53NwW5yc7RjkZwEbws44ex2XK1dwVna9KUpJXtb4owekmltLvd3kz3soz3C5jFOClqutlfTpbra2u3RJ35j6NryD1woAKACgAoAKACgAoAgOOOuPw6/5xz1/Sog44rDt9Kjt6O6tt9+hDd60VfVfPW6s9l27P005Tx746j4aXHwp8Z2nxe1zRvDvw+1PQ9Q0jxHquvanZaPYW9hqVtLayEX988UVvdlJGS3YP5nnMggzM2F9Xh7GV8mxmGr4aUVVw9TnXNKSjKPwyhUs03CUHaSSu76Rd0ePxNgIYrByU27Ws7K9n0cU99dnfa6urKcvgL/gljB8CLT4a+MB8OfiPoHj3xtqvibV77xTLa2Gq+H/ABBF4f0XWr7w74VZtD8Q2WleIV0E29s+sWdzcad5EN94ju4PODzKi/pPi1xDi8zzanOpRlQwtKjF0oTlRnKVWpGEqrn7H2jjKLnGi4urOF6LqUXKMm5fnvhXwxTy7Aznfmnda+8rauyT0vfl5nZWSsnaTaj+sPDID0+nbsB+H4+nHWvx7nVWm2tL2892vPr6/PZn6/8ABL0+XT/t633fdf3ZK2JCgAoAKACgAoAKACgD/9H+/igAoAhBxg7ePXn6dTvx/X8jWKw9GlpZK1t9tV308t1932Xdy637WX6d1t/wwuWPTgD3GP1+vv6+gWnOnZJO60VrNfjq3r5L8bg1br+X6X/rtqzh/HHxA8E/DHwxf+MPiD4j0fwb4Z0xN97rWu6jDpthb8cebc3MscSA9tzAFto4ZlFa4XC43NqsaEaMveauotK3RX+Jx6rXzsna5y4l4TK6Uqykoxiv5tZdLWd/Ju/4Xsfg/wDtUf8ABZK9l+3eD/2T9HjhjBED/F7xhphJBDxPv8I+CtRihLF4vmg1PxcQIpVns7/wXKrR3i/vXBHgtTzClGvmFWNPmirYem7RV1JNTqtqUpWlGS5FHknDSU0z8V4n8WK2W1Z08JS5lGTvKyv0t0drNPr7yavytM/DPxj4x8X/ABC8Q3nizx14l1Xxf4hviBqGs61qMt5dzBRgKZJS8mI1CogIIEaqi4UHZ/RWWcM5fgIxUXThCKjGEIWjGEYpKMVFWUUkkklpG1kmrI/n3MMTVxLcvek27yk7tybesm5O7d9Xfm19LnMQwzzS29vBB9quLr/l0/l2P9Prwa9SvSy9JL2kbLz/AK8vxPLoqrF/C79/8vk7O+mh0Gp+FPEeix382teHNa0y30rWdS8M6o2o6XqNhFp3iXRtv9p6JqMssAS01u0DBrjTLlor+BFd5bRVjcL5tbNMsx/FtN069OcKqjKnUhNThOM6cZRnGSvGUZRalGcXytO6umj08Ll2Y4PDyk6U1bf3drdO6/B39bS+xf2aP23f2l/2UYNBl0i61Txb8JtXmvobHwN8QI9UPhK6GleT/aY8DeJLhZdQ8PzwSXSCaPSJ9T8Pwajf3Go614f1fWLg3EvwPH3AOXZ5nsnGSpzrJOVWjyKrdRgk+dxd3ywjC8k2oaJxSPueBuNMblWJcJRklGXwyvyu2usbpPV6Kz+Vz+i/9l7/AIKJ/s+ftOJp+i6frX/CCfEu5tlku/hz4zuILDUZrkN5Tp4W1nMWjeMrZ5I7iW0XSpY/EA0uBNU1zw3ocN1bI/8AMOd8F5llLcqlB1aXu8tajFt/DNy5qeso8vJ7zfND36cY1FK6P6Qyri/B5g1BTjGXZuOr0tZ6J3vsuXVSbsrOX3uWUgYGR27Y9vU/iB+PO34uVSeGbjZuPR2vdff3020203PrI8tdXTUk9dfy26dvu5t4r5fv+n/2yhUWn3/D+rfp5mvtPL8f/uZJVJ21MwrcAoAKACgAoAKAGj+H/d/+JqIfwfl+kgl8f/gX5o+ZP2wv+TaPi7/2J2rfyFe5wn/yM6PrU/8ATbPm+LP9y/7ef5RP5FK/0ps6mj1S+X/D7f1c/wA/L8ju9LfP+r3/AKsFV9QVS226W/8AwPk9fv2JePcLJbej/H/h3u97tBT9tiv5H+Pl/wAH8PM6LYX+Zf8AkoUe2xX8j/Hy/wCD+HmFsL/Mv/JQqOdTtra3UxvyO70t8/6vf+rBTWDhUSd196V9u976LTWXTuiXj3CyW3o/x/4d7ve7QU1TVN3vt8//AG59/wC98gVSUtLaf15a9Nmv/kil9b9na3X9Gm+jv+HbrcpYTnd9r+vl6bK/dde6CksHy7rXp06K/V6/8Nrq5L6q4a223W3bvzefW9u+iiVSppW7f115u/8Ad/zE6ko6dOn9W/Nv/wCRKrCfxM8/7l//AG0Kv8OPr/7ej9b/APgkdj/haHxHz1/sDw1jPXP23xFjHf7v/jv+zmv5a8f/AOFkfpivyR/SPgl8a/w/+2H9DNfy8f0wFABQAUAFABQBUJRR864APGDz/I/y/LiuenRw+XwaopK/2W7t/l+f3aFVKMcRZPVLXfX8b/dZ99LCSNGqDcCFIz1OfX8f6d93RVGri8Tf2EOSL+03a/pu9t7PrrsubGdejgIJSk15fm3f/J7baLl/LP8Aax/4Kn/Bv4ALqXg/4aCH4x/FC33wTafpOoIvg7wvcsJ4w/inxR5wS4ntZ4tj6D4fF/fvcQS6Vq154bmlS9T9Q4a8O85z+vBVaFShSvZvkSnvC/KrOKTi5r3uVqUU1FqTZ+c8R+IGU5Dh5+xrQnPde8+z681+kbWW2n90/m++Pf7S3xl/aX8TSeIPi74tutVjiuw+g+ErQR2Pg/w0oi8oJp2jp+4eQxgJPql/LfazfIsMeparfJa2yxf1RwpwLl/DFCCqqFOUleXRtuUpat3k4qUpOMZaQ5mo2SSP5p4j46xvENaXsE5q9rrVW0Xkruy9dNV8R4/4a8TeJPBGvaV4o8H+INX8MeJNEulutM1rQ9RnsL+0YDhor6CSOeIkZUiORWKZQkI7ivczfDxznDVcPKEakKsJU5wnFSjKE48s4TjK8ZKSbTi9H1Tu4y8bKse8urQqJuLhJSVt007qzVnfZ6bb62sfuz+yn/wWJuNPSy8D/tYWMV1EZYrW0+LvhnS5I8rLcENL4v8ABukWtyUaGBvNl1XwmvkySeTa2/hG0gSbUIv5t4p8GsVXlUqZbSVkpyWGat7SUYr2cKdSU7QcpR5W6js3NynKKjyy/oThjxRw2GoUqOJfMo2SldXj3drX200eiVkup++Hg3xn4S+IXhvTPFXgrxDpHinw3rNsl5pWtaHqNvqWn31u4BEltfWck0EoBJQ+XK2xlK/eDBfwaWWZhlFSVDFUZQkkm4TTTa1tJXsmuqsrWurLXl/Z6GYZdmsE8PXjUX912kvuenytfu7e700iI6hShdV98Y+uOvX/ADzUypUsSv3iUlf4Xfy/vJrZXV36nW6tTCpKnBySVtLbefe+97R212sWqosKACgAoAKACgD/0v7+KACgDMmmgsoZri4mit7eBTcXM8xwqLg5YnIAwFxuOMcAKxwK4ngcViKsKSm3zyW3vOT7JXjpbd8ytu7byzpS9lCTfRXbfmtUlrvbZS18tUfmx8b/APgpL8KdA8N+M4/gNrPh74ueM/Dej6zeyS2l9M3g/SZtK1zwXoDT395biO916zuJfHFnqmi3fhpZvD/iS00TxBZW3ivTrqzSZv1jhDwxx+c8k6kPYUpu0OaUHVrT5HOLSXMqcVGMnaaU9EusuX8+4m4reXuUab957uzskmtLXs3/AHk++zSlLxL9or4nfEf/AIR2PW/GE3xE8JHw34h+Jw1fX/DWjfEn4O313alX8C+B38HfE7Xx4n0DRMxeO/DfhcpJ8LceNfFyeIvGvw8W1RN0vr8PYXLKvt6WHWFaqKkqdLFU4Yyiql+ep7XDWo16jcIc0HPHSWt0owoxjL5bNsfmXLH29Sq9Xdwk6blH3UnF+9GNnzJ8tNJNXfNd8n50+Fvgd4M1rxx8Q/HeqeF/i/8AFYHxh+0/4V1Xxn4t8R6L408NeBtU8B+CS2g6v8a9a/4V0Rr/AIi8VeINZdvBAHi7wgQi+Gsf8JhjNfodfiHGZTHA4TB4rC4SMYU08PToRhOrTpxg3DCUXiIrDUlGElKLp4iUaT5YypSh7Q+NjleFx0qk8TTlVk7ydSTb1lL4pvq3J73V5NXvdob4D/Y3+GbeDfAHivxP4H+JHiLWvF+jaHJp/hbS/GusPpXjjUPEHwb+IXjvRU8Gaxr37Lfg7w87N4g8PaMqeGvCni34sRI+v7vHn2na6eMDGcVZt7bMKEMbhOTDJTVdQp8mGj7S7p4qnHNp15z9mpxlVnDA04zpTklUi+SHnYHhmhXjSnKlVd21OPtLObSjZ028PyxV5J8t60nFpNx92Uuk8C/s3fDv4Ra78CPid4j+GfxGt9Yv/jB8GtFPhjxr4w8XahpS6p4+/wCEgfQdTbWh+yp8J/D2vT+D9e/sB28M+F/FfipGWSPwV44HhAbkrzaWdZpjqmYYWWMpxdGnzOtSlRpyoU5qLalbMauIp1lSclCvPDYelCcfbe/BKE+/EcH0aUaVRU7cytKLvK7Ts5L91CPK29IqU5W0cm2z2nxH48vZP2F7uKf4j2t14zvfhKPHF1fXX7ZvxcHxcu5tZ1z+wWY/BncfDp+H8vhubKIfFf8AwhY2SMvgo5bf5dXAVsJ4hKP1W1OGIhiISWX4aNCNZUYV/bLEq85VVXbmqnLGpHEOyuk5n1FHEYHE5XOKSbs4OPtqjfI7rldJpR5bJL3dNrpaHyZ8MdeTRPhB4G8Na58P/hB4hg0bRtWvxqeoftIfsAtrF7c+IdYPiBJ28I/HP4O/FbxL4a1hI/7H8Kv4abxVGgm0KLHhWGWNFf6nOcXjsZn9o4zGULShhYqnhOJIJScGud1MuzLLqVSnzSlJ4qEY04wUFPENQhM+MweBwdPFzlGlTbu6jbqYX4E17qjVo1ZKpdLTnu43tRcZNnr/AMMfA0WqfEn4k6pofgvx3f8Ahvw98Rvg54jtdA0Tw7rOn+G7HxT4G0Xw946i0Y6T8B/2Dviv4b0DxF4U8R/274YfxR4V/wCFSeOJfBmv43o3iwO/g8RYu2HwNGc8N7SnCvTdWpUVSVRe7GMva5jnOExDjNfvKlKpVzD34wUqqlRpVquvDtWpCrOX7yTupKMLqyveaUaeGqR6pRs6MYp6XjJwj9veDv2pviZ8BvE/7Ovw98TQeMfGHhv4tXPj/Q7z/hbGreMLLxz4Jj8F6Z4duNI1DSfGfxT/AGfv2ffEPiLRSusawlzN8UfC582dBLH45aLES/D0uEKWcUMRiKEqLqUnKc/YqliYNpym6U6GBxeOnKo462pS9tPnh7OLknRP1ClxTWwXJCpzcvKrX5ot6JKSc6cHyp3SUrxWqdrc5+qnw++L3w8+JUFp/wAIj4x8L63qE+h6X4jn0fRvFPhfxJf6fpmqhkhluZvC2s69pVzCLqC90z+1NJ1LUtHub6wvodP1K8FtLM351VoSiuZwai+ZJu6d07JuLUZJN2kk+VPyuon6LhsS5y5W72s+6emqT6y0te3nZ2fN6i56D1ryaz1t3/4B68Fu+w+ukzCgAoAKACgAoAaP4f8Ad/8AiaiH8H5fpIJfH/4F+aPmT9sL/k2j4u/9idq38hXucJ/8jOj61P8A02z5viz/AHL/ALef5RP5FK/0rxD+rx5l5367eunX+rI/z6w79s0v6f3dm/PtyrVy9i8H/s//ABq+IGhp4l8HfDDxpreiXUpisdZstC1Oawvtt81hKNMvEtzFfNBKkq3Uds0zW3kymZF8lwnx2O41yzL688Niczy/DV6aU50K+Nw1GtCEknGUqdScJxi001JxSaaa5r+99PhuG6mKpxnGjUnCcuWMowk0578qa5byt037JXtKvefAH43WviKbwe/wm+JMniiG3/tEaNZeCPE97fnSv7T/ALJ/tlbWLSZJToBux5P9shV0nzPl+2YG9Z/4iPkvsPrP1/AfVnV9isR9cw/sHW5ef2Xtef2bq8q5vZ35+XW1rsy/1Lx6k4NVedLmcfZ1OZR7uO6Vuu33lHWvgh8Z/DWl32ua98IfiboWjaYG/tHWdb8C+KtM02zC8E3uo3ekW9rECxVF824UNIyRrl2U06PiJk+IqRo0Mbga9acXONKji6FWpKKV3KNOEpScUk22lZJX1t7qfBuOUXJ+0UU+VycZqKl2b2T8nr9zOL8M+GdS8WarBoWjIst5c2eqXqwvP9lBTRtJutY1AjIwSllYXMqgnD7GVcEru9/FVo4aPPKVlzQinteU5xhBfaes5qO+7V7fZ+ew37+Siv60127fPa3KtXLu/AnwQ+L/AMTIhf8AgD4aeNvEmnNPeQDWLLQdSm0D7VAA72T640SaRbyoCrFLm9gMayI7bEYM/wApmHGeByysqOKzDB4WrJKcKWIxVGjOUOblU1GpOnKUXKMo3StzRcbtp8302F4cqYuDnClOpFfE4QlJRsm9XHltprdvbX3V8XM+LPAnjbwFd2th438IeJ/B+pXduLmz07xNoWq6Fe3Vp90XiQ6rawzeXvBRSsQTejr8pVki9TL8+w2aQ9rgsVh8XR53TdXC16WIpqcbc0OenKUFOKkrxvePMr3uicXlMsIvfhKm0ublkpRdmtG00m77rTbtuclX1WHwn1iKffX8n+W9vX3fhPm8TjfYScdrO23b5vyW33XSCliMXZW/r9PLr32Fhsb7aSj3f+S7/fa2r2Z25+H/AIy22/8AxLImW68If8J+7LrOgD+z/CnzZ1PVTJq6JpdyqxRZ0C7EPiAfaNNP9ij+1tOW6+ar5ynKpGKqJ06qpSvSqxTk4wmnTbhFVaaVRXrUuelFqUZTUqVVR+kw+VqvBSaWq5kr7/qnps7Ozvs7y4ivqMJ/Ezz/ALl//bT5yr/Dj6/+3o/XD/gkp/yVL4j/APYB8MdfT7b4h/p+GP8AZzX8t+P38PIv+5r/ANtP6S8Ef4kfT/2xH9DFfy6f0uFABQAUAFABQB88/G79pL4WfAaxC+MtfjuPFF3oviHXPD3gLRWttQ8b+J7XwzoOs+JNUbRtCNxbstvFpnh/V5G1fVrjTNAgms5IL3V7OQpu9LIuFsbnNaMY3qJTUZ1qn7ulBy2SveU23yxtDmac4XbjLnj4Ge59HI6EpuNpcr5Va9/Wz2s76tX1Ssz+eL9p79qz9t/9qbTrnT/BPw0+I/w5+C+orpVnbeDvBmkarqHjfx1pPjRZF0Ftc1mMRa/rnh7xTEzRmTwhCng7yUWN38aKibf6A4d4UyLh+k8VjPZ4ipBN1K9Vw9nS9jFxqOMY2pUfZNVFOo71YKdSM6vK5RPwrO+OsVmtX2dJySbSjCCd5OTTirK8pczUWo7N2at7qPy58T+CvF/gDVZPDXjPwx4h8H61Eu+bRfE2h6loOowKxyjPpuqQWtygcAlC8a70YOm5GU1++5JneSUaUauBjQqU6qvTq0XGpTnF6KUKkeaMo9LxbV/5mkj8TzzI83xN1iKtT3W7xk2mmrNppybVut4/JWOYpZvQxmdLmws3FN6crt+Vuy/4N2zjyrGYTIm1ioqbW99X5dO1769+75Suqklhk5Se2vS36eX2o79bWNquF53p1stN9bbdtj3y/wD2ZPixpcesi+i+Hcd34f0PU/E+saJB8cPghqfiyw0vRNGfX9Ulk8G6P8Rbzxc7W2lIbqSCLQpbgJtbafMR2+XxvG2EpV3hvZYznjKUeapluYUqN07NrE1cJToSjppKNTlad1KUT6DLsgrTpqpzx5W1tWpuX2X/AA1OU7a/Fa3TV/D63+y34i/bY+DN9D8Rf2cPAfxg1jw1qM4fV9N03wB408X/AAt8bLpSywPHeaXY6Vc6Kb8JO1pJqug3WmeJoo4hp1vrlpErxN8rxtg+EM5p+wxGIyvBYj2anSjiMXh8LiIwlJPnp+0nSqqE6lFXcbQm6fLK6i1H6zhfHZ5kFZVJSxFWmpJS5YzqRulrfljJXUZdVondNXP6S/2Wv2xdQ+OczeCfiX8GPin8EPi7pul2Oq6l4c8VeDfE39gX2n3X2+Mavp/iBtItxpNm8umThIvGNj4XkurqeLT9Bl8QvBc3q/zDxFwpLLZLE4PE0cRgqkpKGJo16NSm2pNckZxqNTenLaEqjvCd0rJn9GZBxhhsypqnXhKnXjFXi4SjJtpPVNNx0k5av+XRqx93V8ufXBQAUAFABQAUAf/T/vxY7V+bCKPXjp9cn9R6buRWEfbYjRq0Xv6Ltvfvp87aMUp0MPHmbUYra+nl216LW/TRW978+f2pP+Cj/wCz9+zIt5oD6m3xI+KMRaCH4c+D7qGS6tbxCiMvivXnWbR/CMMZlgluYLp73xILK5t9SsPDOq2LGavv+H+AM9zq06WGqUaDjf21RO8uaKlD2cNJz1lG7fJHluuZzi6Z8TnPiBlOVycJVoSkm42Ulp7y1bVvPdyacbfDJM/nI/aX/bw/aK/acvL2w8WeJZPB3w9nQeR8L/Beo3FloK2rFyF12+t47S88aTqv2dpLjWzJpseoWi6roWh6R5/2GL+k+GfDDL8nkq0UpVU5P29W062u6UmvdVkk4wte125Sbcv504h8ScTmUPZq9nvFaL100bV203ffTdKPn3wv+Ev7RXjf4WeOtc+Hdv47k+Gx8QeFtK8UaVo0vii18N+LdWu2vH003os0/wCEav7Xwqfs97rl7rt3bWvh9tf0LUj5UOqi9tPo8xxOR5fjsJhK9bCPMFSrVMLGpKh9ahSlaFV0Yyn7eMa0acm/ZrlqQw9TmdqT5fnMAswzCM6sKVT2cpWlJRlyXVrXfKotxco7vTmitOZc79I+CXxe074j+GvgodW0vw7qfxNhsNRgfRvGeg+NPCmqWugavrkenavc3vw0uPGlpfjQtW0PXH+yxwXetaV9kvXXS089Un5sVxNSrUM8xFKniLKNHlhXw1fCYh/C3FUsbDC1KfO4pJ1Iwg3ZtpJuPXDIJqfLo9bPllGUfVSjzKXqr76fEmeleOv2Kvi9omh/Erx54h8U6Jrh8JXGp6l4kvLrQPjqdSvNSOiaF49T+3da1/4NM2g+IX/4TXRgrfFLxd4T8YHafnIXNeRguN8NKvw3hYx5XWoV60UsRgJKnT/fKMXTp4ydeafsJpTw9GtS0sqi9nW5PTjwnVcak3qoT5fgqrmaaTs3SUY25k2qrhPVe67nzpb/AAY+MVxp/iHVbb4V/EqfS/Bl9rNh4y1O38D+KG07wnqegQ+dr2n+MNSi0f7N4autHiO+/g1mawltQrCeNGRq+2fFuST9jTePwiqYmKlhYPEUoyxMXZp0IuV6yd1b2ale6emnN8b/AGTmNZyfsKrVNpTfI2oXfLab5XytvTW21k3qz6j+HH7C/wAbfEmmfD/xt4V8U6bot14osdJ8beG9Ws/BH7R3/Ei3bv7CH/C0PAPwY8WfDfQCAuTnxWB4MBG0vghPgs941yzD42pl7oyrVLwg+XF5SlL2sISs8PWzGli9IyS5ZYX3lrCM4OEz7HJ+EsfWpKs4uKvp+7q91tJU3Dd2/iXVuljhvhZP+198ZLZPCPw3+L/inU2vNX+w2XgvUP2kfCvhLU77Uv7HQEaN4K8cfEHwv4n1xf8AhHWVv3nhBFwHfIWN2Tv4n/sTJsfHF4iEo1fYyUsVSwGKxLhh6clVnCriMNh6/sqSk/a8lSrBN3qJe7Jnn5XUzHMMFUpxSd2vcnVp07yd0mlUnT53bTS9r2b1XNnfD79nH4+XfxG8N6V8N59Om8ZWtvqmu2/iD4T/ABN8JfEDUPC76OVjGra3rXwJ8QeKfEvgWOZpF8JoyB45Wby0Zn3Be+rxrkUcrnTxU4xw008LVp5lha2EpVlUjadGVLMKNBV6dSEpRklGdOceeOtpxjw5fwjmdfHOcKb57+1hKjUjUcbO6lzUpTcJRavZuMo2T926ZieLv2e/j2h8Q+KPHFnpk3iKLTY/GPirT/EnxT+Hdz8UNmuyQSC/134ezeN5vihZalcS36S3Z13wraXYeU3F2Cx31pguK8njUp4KgqkVTpxVJ08Fio4FUo0lKEaWLjh1gFFU2oxjCvJJpUlaacDvxmR4+UFU5E7t/bg6l27a01UdRaxveUbdbpNHo+nfsuftB28VhDpnjv4eaZpOgnU00jVbr4/eD/D2naH4713SCut+Df7E1zxJ4U8TaB8QfFniPwRrnho8/wDFYeMdDy/jdPAvhFvGa+PmXF+UUa1SU6GKjiHKFGXs8DWr1KtCFaUadWNXDwqU6uGhGpLE2jWlOhGu6dWlSxdR4eSwfDOYTS5VFxs5X51FJ8nM4tT5Wp39y1vecbxlKK54+Mav4++P/wAH/iNqKzfE/wAc+H/iH4XttI8Fanf+GfiVeXqWemeGfIj07wQ+reG9dvdL1Twxoc1hDaN4ctrq+8PJPZCKO0lKB29aHCuU5zg6aVClKhUbxVKnVwzhKNSup1ZVZUK9ONSlXqOtN1faUoVlOpVVW05TUcqmdY7Lat5uScbQk1PmTjTekVKPMpwXJHls+W0U4ppRUf1z/ZY/4LHaxpX2Hwf+1Roy6vpqloV+LnhDS5Fu7VVW4cN4u8EackxvGChWuNV8HtHIztDaWPgl/wB5dr+NcYeCjre0nl6g7XksNUuoTd4JKFTmapu0ZaSUlOco806cUz9T4X8Wfq1KNHFR5oqyTdrqyf8A28/ecVo48sVZKd2j97/h58S/AHxY8L2PjH4ceK9G8aeGNSXdY6zol9DfWM4HUxTwO0TEcA7SSDkc4YL+C4vKcyyWoqGJoSg5WklNW5o3avHv2tdPbmt7rl+84TNsqziHPQrwqJXuk7NXt3vvvvrqtbmZ8Rvi98NPhTZ2U/xA8X6N4dbUpWg0uyvLtBqWqzxRPPLHpWlwmXUdSeKJGlnWxt7owRDzZgsY316mU8N5xxFJ0svwNXEtK85QjanBq3uOrJqClaV4wb5pK7SdrnBmvFGS8Ox/2nE04a6R5k5NPvtfpdq26+HRnqfy+ifnXg3r/wAn4yPevS/nj90f/lgfL6J+dF6/8n4yC9L+eP3R/wDlgfL6J+dF6/8AJ+MgvS/nj90f/lgfL6J+dF6/8n4yC9L+eP3R/wDlgz92GB4DY6ZOM9uf/rj8B96ous6TvH37WV97ed2tfVf+BXMv3HtVeS5r/wCWu266K/lp9r5n/bCz/wAM1fF/djP/AAh2q4z1xsGeO3/1885IX3eEnX/tTDvlk53qcq7/ALqd/La/n+B4vFHsPqH7yS3VvvS/Lyt63Z/IrX+ktN8tOSfb13fp5d/ut73+fM1epHtfXW3Rf1/w5+kfwN03wQfgfotxrWj3y6ZY+LtH1zx54n0Gz0Xw/qzeFNH1rXte1ifW/GuufD0vrkPhgxeE/EQ8M+GE8WlpNX8O+Ck8amP4qeD/AAd4U/C+MKOLqZrjKVGvSjVqwp/VMNXp1sRzzawzo1IUsPi5KlSrunmVKU631Oc/qeIxNHCfVstzHG439a4UxtLC4anKom07pyi+RRceZSi3Ok1KUb0Z+5zRSnGDfPUgoZ/9peA5v2hv+E20rw14a8P+AdU+GXiHV7PV/Fvjb4TLpvxI0/Xm1nQdC1l11r4fy+HhrrSlvDfjnwy3hLxb4tTxToniZZ/+Ez8arJNLxYLCZlDhvF4KrWcsS8ZTw85YTDZtfDV5VaNedGssPjPrtKlLmdSGInXweDjgK+GlKDwMkqvpY7G5fis7oV6bioJOV6k6LUo2krpyhyXUVypJSqOrGXLJVGow9i+K2ia1HL4svvA13o+u/EW88d/2v4R8Ow6ynh6RPg+/jLwB441nxewU/CzxFofh/wAL/EbwWfCwPm+Eg/hA+I/Go3+BvCv/AAmVfP8ACU8Rl+Pf9p4TF0MK1iKE6tWjDFzjiK2FngqOHnQlDM6eLljKOKrRoqr9ZhXxf1fASWKxeKWGl73HFPAZrlFOnQrU6tSMYytB8icYtTlJXlTlDkcU5KKXLDnqe5CHOYsnxb0jX/HulT+DNQHj6TR/CHiV5/FXijRPCPxgvdA8MaV4Q8cQeM4tb8beO9A8U/ErQdCfxKIfEPgHxThvB/jLwd4h8rxpzIVbrp8L18LTpfWqP1Cn9ZlSpxweNr5bCc8RiKVLBQtha+HjWrzp+xpSUqvOsRB08JCpGpBnn1s/weI5lGpGo2nOXPThWaUIuU5XnGTUVaUpWVnHWpe3u+M+A/DF7rXw08CG7+DHif4l6rN/wn3xA06Lwv4X1e9sLTTPGfjJfBPiDwbqyaB4bHhvwL/wlHhzwX408UeDfE6HHgrxjonhj/hBtysFb3a2OeDrThWx+HwFVzpYW2LxiwijVwtKGYUqtGkqlKeLcJ4vBwxOGVbD0q+Gqzp4n2lN06c/nalPCYi84pzivek4xlNRU3yWlK0lBy5XyScb3inG71jVsvhNo1p4o+IWm6h8M/Dl1bfD/wCDPhvV/CjfELxDq/wu8F+OdWh8ZeBdD8Q/EdfGa6/8LD4i0PxSdZ8bp4L8SDxX/wAIhkeGgFD/ACUsfxVm2Y0sN7LG1qd8zrYXE1Muw1PGYqHs8HjcQsEsK8Lmco4ml7Kh7eEsOqyhTqTcaLm4GM+H8py+av7OXNCMoKtOVKnJTlGKm589L3G7yUufkvo21Fm/oXgb9mx7b4r6ix+GWo6Rp2pePX8PWd/4uaPVbHVdE8HQa94d0XRtWf8AaD8Ivr/h/wD4WAZ/DHgzxP4V+FHxZfxayqreMHMjPBFfiLiTAYnLHhZ5pKnOGDeIpvB+zp1qUsRTjOriObh6s6WIlSdSWJofX8tlhlCLVCnds78vyHJsfGpGpVoqak0n7S9rRldQksQozjzcvLJQqJpaTd0dL8TNW+Bd7DY+CJvCfwQtvD2ueOvjnrHhy88FfEbVVHgfTdZ+Gvgb/hCNa/4kHxBXw34f8ReLvEWijd4X8VfuiNAUx+CFJd28nhnJ83hQxrp1cxw8aVHAYeblhcNSp42VDGZjOvRk8TgZ1p0o08RHlrYN0aUvap0sTz+0lD2HmmXqCTdG8nOSSqTfIpQp8u1RpWd9Jc004tTUrRR6z8bviF46sbm48O+F7v4yXGueNfiX8fPDeh6r4D+K2kfFH4k6JqOhaP4E1bRdK8Gp4HVT4E+HL+IIdGXxr8NWB8YeFIkWSNjC6M3znDeVU44jFVcVh8ulhIfVsViP7Rwc8Lg44OrUqQ+sVFmEZcynShiVRxFT3Ks1701Cc5Gkszw/tKfLVqc1lTp8tf2suZKElCDp2UbSnFulG8oar4lY/Nf4/wCp/wBpfGr4p332n7V9o+IPik/a/tX2wTg63d/N5n8XfPCheR/CRX77kGTvD5Nk69l7JJQSgo8nJZx9xQ05eX4eT7NrXe5+X59marZpmdnzfu12b/h73210b018j9D/APgksQvxQ+JDemgeGCM9f+PzxD9e3oMY9Vr8q+kXhbY3ATeqjRnr5unFLda2dvtL1d7R/Rfo+YpLL8y86sen99+a/X0P25u/jT8K7P4geHPhTN400ab4heJZL6LSvClnN9s1QDT9E1PxHcyajDaJMNJi/sfSL26gbVHsheCLbZ+e7Ii/y3TyjOMJl/1zEYWr9XulGu4yp05JtRXLKfxJykoqSTipKUFK6ly/0ms8yXNcXClQxkKtRXXLGSk9E731v9l6b2afvXijkfj7+0d4R/Z7tvDt34q0u61IeJpNQt9LSHxd8KPB0Jm0uCK5miOrfFz4ifDbw4ZDFKCsSa21wxaMLA29SqynIsVnNWtJU25UmrwhCtWa5ny6U8PSrV3r/LSnblk3eMXKO+dZ3TybD02ocyl1vGKslp70pxh0WkpJdVzXtLa+Bnxx8L/Hrwzq/ijwlY3en2Wj+IJ/DV0L3Xvh34njk1K30rR9ZkNlrPwv8dfEHwre2wtNcsR5tr4hmkWfzo5LcKiPKY3CYjCOMKi5ak4qai1KL5HdJtTjGUXdNcrUZRaaadny6Zdi8HiLyhJSSdnrH5pWck9Nb3s1om17x7hux94qo+vPPTA3N9f8f4fLiq7esUl52/4FtdPyvax68pUorWav5v8A/Zv1tZ/d9rh/GHjvwd8N/DOo+L/Hev6R4R8M6TC0+o6xreoQadp9rEON09zdyRQIM/KvmPyxAGfl275Rl+bcR1YYanhKk6s9fYwjbRWjzSlJ2jFOWrlZX0VndHm5jjMq4cw8q9WpGnGK2crufdK9n91/0Pxb+JP/AAVIufjB48tPhD+y5beKdA0a5e/fWfjqfAGmeMNQ0vTNHmU3GseH/BnjjXPCHhbQ9AiZ2m8XeKfin4ktG8K+FYNblm8ER3SJdwfsmW+E2Ky/ArGY+MKUrUpfVp1KlP36so01RlVo0sRVrV71Ixw1PDUXUrYrko0ud1ISh+P5v4pYfFVZUcLK8YuXvRUZXstGoO0YqTXvuU5pQcnZWlGXyLq/xl0T44fET9oXxDoPirxX8Qzd2vijxL/at78KdM8KDQfA3gr4N/tNaEEFx4I8ReKfD0nh3wq3jrwQsXifxcvgzxn4xGvrGihVQL9hguHK2ApUKeKhQwUXVjh6S+uVMVCc6tSlGjRjPE0sPXlWrOVSnShFShTSWk4pQj8tmvEtHETfs5OenM7U1BpRjeTcIOSSio3bW6101Z7voXxM8Pv8DNN0SHxT4T8RaZ4U8MfB74j6ta2Xir9ne/uj4W8C/DLQNX13wVr2l+H/ABCvxnfQB4j0ZmHiCR/CXjc+NPHq+DkjHgJN6+djMhnisZVgo46nUh7WCawuaRoN3p0Y1YVK9KGXycG5NRUMRTmlHE06igpyl6GS8R06FNSc6XK3HedJz/mtZP2qTva/u6+43onH5p174UfAODw9p3iS/wDhN8IfBXhbVbjxLa+LtYuvi3400/xJ4V00/s//AAo8eaLrPwv8Fa98ZB4h8e+IR4h8bazt8Kr4U8XyNnwyCkeEd/osPxNn7xNDDyxOZTUIU53WDw8liHKrXc1i61LK/q9GHKqdKHsquFquMW3ao1Vq/LYjA5e4OadFe8017ZXirUuXkg6vtJat3vCcVbSVT3owi8CeDf2Y9D8H+AtR1fTP2dfEPjTWtC0S1R9d8cPYeHNQ1TXvg18RPNHjRX/al8WeItCcfEWPwP4VPiXxP8Jv2eZfBnjFlPlQM0TeFPLx+acQ4itjnfMlRhy8nLhFz0+WrH2jwyq5RThWjFKatGvmFSrS5ZUV7SUOb08BgcuUKD5qMpe9zqU3FSukoqbjXbj0d3Ggk73cop8vD/GNfhb8UNeTQNF+H/h3XPHuh2v7MPhvR1+EHxH1XUfEfxFA+GA0Dx14L0Er4i+LPhry/Ce3RfDjeKvDHhORt2hkeOn8XxhUX6PK8Jj6GW43EPETpULY2q4YyFP2OFca9WaxCqKnhq0aEk51q0MRWqxUHFUatGnDmn5eLxWDlOnH3b+5G8HLnnpFJNNyjzLSK5Ur2u03c/RvxHdReMvhR4j0Tw9ofxi8ZpNY3/wx8Q6t4S/aa+OXxStvENm3gzw6nifTYvA3wh+DP7RHwZ8OXN62vav4TH/CYfCr4WRxRQReNfAqeE5dj+EfyzHYbFYLNo16ksFRqV5Kpz1sHhMLCnKbjGtXU6+LwmJc5ylKc4Qq4mrRjUcK1OrKpRnP7zhvG4GpQceeTUFype1qTfLG7hBpRlTahFaPkpJu7i4pSUfhL9mq28P6N8P/ABhqd+PDHwjjvLj4zfAvxFo91ZfCOx+JXihz4OtY4o/GnjL9pT9oP4TiRfC8njgyv4d+F3hLwl4S8V/2BI3jcNbv4TFr+hcY1cf/AGlGapYnGRnCFWnRdLE1MLQ5J0ZRdKOBy3EqpOpVpKtz4qpVr0Gr4XkpTmfnfBtDAwwTUp06VnaUuaCnO/tbturWg42i3DlpqNOfu+099RcsTUIPhh4L+KHxATXrTTz468I/CzSvFnhv4jfCvwL4Q8N6XBpnjnwN4d1zXvFur/C/wD4/8TfBLxB8QPCmhayfDfgk/DHxV4S8JEyy+Nv+E4XxuI/F8XblqzV5PQnTjWS9rUqrD4qtj+dOhKUaeHWMrUsNmFOlPF0YVq0sZhsVKVKdfCqlUws6UKXpylls8bOPtIWvy88PZ663cnCM6lJvlbivZ1IR92Ml7/O5fadv/wAFEdN/ZW8M/BPwzo/hfxbrnhXX/AGj+MT4I8VeCvGGgeJD4V1rW/EehaPf6J4y+IX7VPxZ8S+HIH8PaNoL+DvDK+FPGXhO4KxxzeOvBsPilI/B35zDw4x2erEzpSozVNypc9DF08U4VqWs6E6cMBhVDlnOfteapCtzwjFxSfOfcZbx5h8nlCFZ2Tbupw5OaLt7+tWfvNL4UnFJt8z0ifrp+zz+1f8ABP8Aah8OjWfhV4ts73UbW2W61zwZqbRWHjDw8rym3zquiSSCc2n2hfs8erae17ol3PuhstUuZEda/LeIeF884ZrOnXw8oRlLlhVcf3VRqKekrScZJOXuzjCUuSbinFXP1XKeIMlz+ipQr05Ssm4OaUo3dv7u/krapO7+L2vxV418L+A9Cutf8aa9pPhrRrSMvd6pq97b6fY26DkNNcXEkcSBuihnUnoM5FcGVZXmWcWoYfD1a9dx5nClFycY6Jym/ejFRbXM3ZLS9m7R682zvK8mlCWKxEKfO7JSaUnpt5u3l2vy3RW8GeMfDPxF8LaV4w8J6imteGtdtlvdM1CKOWAT20gzHMEmSGQI6kMu9FJUgkDJ241sHjsmxMsNiKTjKPvO9uZK+nyduvLs9ehtCeX57h4SpTjVptKzjJ3v+V/RSXQ7IEZ4YZ+oH8/X+uPSueNKrF3crrqrL9E/08nqzs9vQn9pfelv6y/z9Hox3P8Ad3e/P8x/kdu9DnWhb3Fr07f5X8h8lGet0/P+kvL/AIF7R//U/t1/aQ+FfiX4zfCbxL4D8J/FPxn8Ita1iJIrfxj4IEH9r2+dwe0nV1ttSk06VWLXMWga54Y1mfykgtvEFhDLOs/sZHmGHw+Np1XTpVYq7tVSjy8qcouMndQvNR558s3GN2op3cvns7y3FYjDTjGpK/LpyyTbvumutlorSj1dtkfyI/tCfsg/Gz9mfUpl8aeHZNU8I/aAun+PNBjfUNBuInluEhXUZII/O0TUZhb5hg1aCwe5c7dPN4g81f6+4H4tyqNGMVCEnFWaaaeihe107pc8U2k0pPldpRfL/LPGXCuZ1K8mq0leT2d/vs7X08nbW61R8wV9vQ+s1INKTWmn/A+Vvz7H5zXqUKc4txvtq99Pv/TyT1Z+hn7KfxA+Fmg/B7xZ4Q8R2ngO11i4+MHww8aeMNf+IGrLp2ov4X8Ef29rmPBWhnxE7a/4h8J+HtE1lwUPjA+MvGvjv/hB28ElNyN+YcWZTjqmNo4n21dQp050YxpU1NSeJrUIz9vZVKnI5xoTi6UaMaEaNapXrOi5KH6Twvm2Cp0uV04NvVuTtblXTRK9rrVvm5rKKk1IPBc/gHX/ABv+yz498XeI/hxPoFxo/iTwdrXh/WPEHw41PUNC1L/hN/i5r7ax418E+PvDvinwvoGgMms6MP8AiqX8JeW3iBT4G8aupXxj4P1z3BudPiSlBV4yoLA1Zy9ji4wlFShVdOjVo+yliKlSEJwaw0qrpStGvRqpxo1erA4+MpQlNJp80UueHNeyScotPlSk4u8l72qi01zw+lfGHxN8O2Hg34sTf8JV+yl400PTrDVdJ8K/DCP4e/A3QPC+h6V/wiHh/wASDSNF0X4XftU3D67Mvibxvrlv4H19PC37Qo8F+KtK8XZ8Z+ED4k8TiD87wmErLEcPU4TzGl7VVK9epUxGaVOeVCvU5YP6/l8aNFVI4dKtSpVMHKtTxDTpSprDTn9vh8RTlGUpRouzcYKEaCteO9qNVuUUpcynaqlKEVzK84y+cviP+014RspvG1/8OrHwl4z8Ua7+0x+0V4z8Ia/4qtPiRp+oeE/DPjvwX4C0HQ/Gehk6/wCEPDf/ABVONc2nxX/wlykaEflJ4X7PIuFc0q1IOU6uGhSwmFjKNL6nKlWmnW5sPN+w9vy4dcrg6f1ek1XfsoKSnGPw089wFF3UIyvUbUpKXMo6atfD7+7u5STitr3l9a+BviT8C7XwVL4Wi0n9jqTxd4B0fwBpiaraw/8ACR+OdD8SAeJn8Rx+CviX8cv2pfhLcNoHhS3Gh+FvBHiH4YftD+MX8HS6Br0MDReEG8HMnwXE/D+Z0pUazxGcTU489NYulCjGVN+0/jYfBYJ1KVR04U6saOOoYeso16dK0MXCvh6H22U8YYClF0/ZYaNm4y5XdprkScZ1KyjKLnK16c6iUYznKMqSjKXzV+z38RPBXw38R/EO88ReMvh38LtJ8E/EP4fa74Y8K+B/GelaVYeO0fV9e8Qf2P4j+JHgX4d/tVfFPx14BK6Joav4Sfxf4p8IxSNsfxkm3welfY8U4LMMUsLOhLF4qtOjTip1qSp1vZzovmr1cJWxOU0sNipqXLNKNDEUnUlSVGFNThD5rJsTgMNRq+0hCEYuTtCXNFtS+GNSMa7lFNXTblGUY3Um2nLsvCfxi+Evgj40fDG5PjvT/in4Zg+EHij/AIRHUviD4k0TxovgTxOzfEHwS3g3W/if48+DXwo+JC+AH+H2lSDwh8M/FXg8eDp/G+u7mZlWJ/GPnLg/NcZgJyrQqYOc6+sYxp0XWp6VpVI0qGPx1GlVqVatSnWrKr7apGDb5WqU6Xp5Xxfl+GxfLaFVRVr+/Lb3Lc0qdKTioxjKEGuWCai7tSRf8VfGn4Zal8G/EFnF8WUa4uf2evAVnH8BLP45/ETUvhFpupvH4EWTwTofwwb4fHw8SDGoYr+0R4t8ZbZvEKDcEldZw/C2YrMVVeFjJKyWOeGwarJ/V+VzliFj/rDu4qk+TAQXLUjDldNTnDz6nF2CnTUFLlve9Lmm1L34NKMPZKMbOTl71SV3Gb0bhzeh/Cf9qn4Vz+HLbwvF8VPEnhzw9H4C17xHd6D/AG58ePBkng//AIQjRtbTw74P0PW/EP8AwUrOWuYtb0RfCXhX4WvPF4uFvbxr/wAIv/wirv4K8LM+Dczo1J1HTo1ZV69SmqjWHnP99KPLWqewyGbpurFzWIlW9rRpRledeTmpx+myvijLpxgrOKjGDaUZpO28VzYnZNqUHGzlK9lC1pfjv471XQda8XeI9Y8M6Xb6Lol9qd3caRYWqaxFHDYbiIoli8ReKfGGsRlV+9Hd+KdXKPvWO+eEIU/ojJMHjKEYKdapVlTjGLq1ORTqtRV5zVKFOmpzfvS5KcYXb5KUFyxPxPPc2wlS9oRjfor2j5K+ttl7zk311bOf0TRNd8T61p/h/wANaVqPiDX9UultdM0TRLSa+1C8d/upFY26vNI4GTmNHYKN+wBGets4xVShFyqSSjBNynLlSUYq7k5N2SSTbeiW7b+z5WXVKNRvkT16K+n/AA/l8k73j+9n/BP/APYF+Pnw58Rx/Enxd8VfF3wgXUrGOC98B/DrUtB1KfWDPb+ZDF4tk1u18S+DJbnw9qAcJaQeG/EFtMk4vdH13SmN5b3f878dcVZRjoOFHCYWc7NxlXhJwjLSN7R5ZXlTdWCnCrCUZNPWMpuP7/wTwnnOF0niKvL9q07XW+7bW6Taad7bq943vjx/wTt+P+i/EOw+IWgeNtZ+Nulz69a3Or6nrmpJ/wALC0jTBexmSW+/tG6+y69bWyzSvJLoMltfyqrQad4ZtLW3jC+74d+J2QZTyUK+EjhpJRhTjOMfY2Sm1CEkuWPLCCTU1CKc4Qg5SaRw8eeHWeZq3OGJlNPdqVnutX7zbu305m+Vt2R+x3/Cn/EH/RevjP8A+Bvw2/8Ana/59sjb/PH9p0P+gbBf+CX/APLPP+rM/eP7JxP/AD9r/f8A/a/1br9o/wCFP+IP+i9fGf8A8Dfht/8AO1/z7ZG0/tOh/wBA2C/8Ev8A+Wef9WYf2Tif+ftf7/8A7X+rdftH/Cn/ABB/0Xr4z/8Agb8Nv/na/wCfbI2n9p0P+gbBf+CX/wDLPP8AqzD+ycT/AM/a/wB//wBr/Vuv2j/hT/iD/ovXxn/8Dfht/wDO1/z7ZG0/tOh/0DYL/wAEv/5Z5/1Zh/ZOJ/5+1/v/APtf6t1+0w/B/wAQbgf+F8fGXHqb34a//O29u+PxxlqhmdH2T5cPgnH+Z0rvt1lF69NI+jv7uDynE+3i/b1nZrTn3tbpyN9F+t7py+d/2q/hnrmk/AL4m6pP8YPirrsVr4al3aZr134LGnXQVgNrDQvhysuWyu7bGRlfmxlC30nB2a4eGaUnLDYWaaqXqxppVYe5pyyeqUnKz5YybTs0o804+NxblWJqYL3a89OVct3Zu/lZJxS0i7LS/Np738vVf6ETfJp/wP8A5K34/I/hSCu79v1v5r8n8grhngIYl3aTu/672/PTpqzthmFTD6RbXzv5/wAq/rSztcK6KmJwPOqCpRSlo1b8tvv/AAW8caeFx8Y+29rJuOu76fcttN+/duJV1Mqy3DRWIhSim1d6Wbb77d+vbrsZ082zDEydCdSUox0Sv/w7699u5vzeGdai8LW3jE2oGhXet3/hu21AzYY6lpmm2mpyxBX++RbajbyAKGDLndjYyryLM8tqSlSjGHtIxhOUeVOSjUc1CTulpJ05pd3CVrWtE/s/MIfbk09vettvr7+119/T7VD+y9Qg06DV2srn+x7m6ktLfVPsz/Y/t0SRSSWRkyUWRYr2JyrENtYONyhlpqWBndqnF91pp0W6W9n1e3TQLY+n/wAvJafO1/mvwT+Vj3T4SfCXxP488M+LNS07xReeDtNvdW8P+AtNA0Dx34k07xz4o1gy67D4Lk0/4daD4r8TvLbR6Hb67IkPhTV4YrhNLk+zwzXNtd2XyPG3EmScLypr6nDEVqaqYmMIPDwq0qcYOlPEReInShBJVvYuTqwcoVKiV0pxl7/C+QZxxK1GWIkk2k23Jrf4fdUpP4b2SlZ2teyYn7QfwBv/AIB+IYPD2o+I7TxBLPIsdvbnwh8RfCWoNa+VITrCyeLvCmjeFtV0x7iN7WKbwn4r8VLHcDbJKUV5qOCOKMp4tdaLwE6KozjB1J1MFWpznZucaf1XFV6kHTXK5RxFOhK1SDjGV5qHocQ8P5twzGDhiXJzSbS9onFW0vzxgmnfTkck7Nc2qZ8+V9nUq4DDy9hGnFJ72X3dUlfyT37o+S9jmP8Az+l94VpOGXYaKrxpRUnZ37f+Spfi/Low9jmP/P6X3hV1VGhlWWWXL+8l6fxFbt69O2hoqc6+aZmm7/u1v1/du/8ANfe1rr8bn6P/APBPDwPqvxK1X4veEtF8deKPhpf32g+EZP8AhJvB0rw6hbKmoeId6xSxSwyRtJHviJinhkMMkyRzxbvMX8C8fcXFY/ATcOfkpOShdRu1CNlzSulrZ7Kz25nofs/gXg+XLMyV7fvI3dr2/eS9L/f/APa+6fBz9gb45/Br9qr4ea3qGsajqHg+aXxbHP8AFjwRNpVrqelSDwl4gtoDrHhvxCuu2+jL4gaOLToYbZPFVtbR6mLe4uYZgtxB8xxb4rcPcT8P144fKcLHFyp0G8NWvUoRq89GVS04qg6saDu4VFyOVSkuWMb2l6vCfAGfcPZpQeKxNWVPnfvxak2knppKVr25ZKTfKpXkmk+b6p/aq+HPxa12HS9A8CeM/wBqfWbrwT4V8T+Jtc1/4df8K38N6hJqGtnW18O6VpfjWTwv4P8AFGvXE2rC3l8beEvhZ4sLW3gvwNFanwXP448UeCrif8myLN8DQxNWdXC5dGlXlhqE6VanWnhatCGJpOrKthpTnR51BTnhqtWE3SqTpwqVIwhN1/3HOsH9ZwNOM3UlOEW046SUuXRKSfNv8VpRule0mkoT/sk+FPGeq+HNc8K638R/2itBvrK+/tKa68SaX4qvtR1xfmh/tnVPif8AF39jD4A+I/EPiMMNN07yZJfFR/4RXw94eXzVgLlr4ix2FjKNWOGy+T9j7FuEaNOnShGXuqlh6GbZlRUFH21Nxi6TjU5p8zhOpTPjuEsFjKsmnVr6y505SlK76uU5UqMm2+Srfld6bjdKaXL9lSfCDxAFBX48/GVvre/DY/Xj/hWufXv054wRXyGGzKndtYTB1Et+eg3bsmlO+vrq+ybPvsdluJUNMRVhbqp21+f569+qUfyZ/wCChf7BXx1+LZ0/x34N+MHjP4ny+GrVmtvhl8StV0WPTB8xEeseCh4d0Pwz4W0HxIkapG0nivwqwkYbh40hyIk/VPD7jLKMBWSeFw0alSUXKVCl7LV3upqUqrlaycailKUlU1p0lHmn+Ycd8L51jKEpSxNSVNRfKnNSbStforX3a5HtvtI/no8TeGPEvgzX9Q8K+KtF1DQNf0u7a11PRNctJrG+tHHG2SKZFlRgMNh1UlWBwFYNX9OYLNKWeYenPDKPJKMXFxs004qzTV7q1rWunveNry/mrG0KmR15qs3J81ndt/n/AF07GPXp4/L6VFaRXn/Vn9/Mu9tLHPDH1ars5N/Psv61tLppuFedQwFKo7uK9LdtP8/13PRp46rRVlJpPz7W6el+2vf7OhPrerXGk6doNxqV9daNpV5qV/YaaZy1laahrUVlDql9HZYIWW7g0rTIp3UBpE0+1ViyxQqvbQo5LSbmsPTTsk2o6tK7im+W7ScpNJqWrdnrI58Rise2v3skuvvbbW+/d6dtzpfBPgLX/iDrMmheGZvC41VbU3VrF4l8b+EPAdlfHIVYLXWPG/iHw5pE+oSM37rS1u/tUyLJJCjJDNInmZpnGR4Om6k8PKUIuKaoYaviKj5mknGlh6NWrJXtdqk0lq7JOUvRyzEZhOSgqlm+s6sIR011lUlTguul1d2Vm3yy9k+Ln7L3i34Vf2tqi+Ifh94j8H6cgey1a3+K/wAC4tT10MF0PXP7G8D6J8Q/FXibXH8JeIm1rwt5cjJID4fdCgdHSvleFeMcsxsPqqoYnnnFuMpYDHQo/Dd82Inh44eEmr2UqqbfuxXN7p9LxHwvjsHTdVVI2WitVg5dVpFT5918XLJbPms0fM1fY5bhsuxeJlJ0o+897LTy6/8ADdrtnw+ExeY4STSrSXz/AK+/Xb5xKeYywNbFwvTi9eyVtvJrX19LXbNsto46hhJWqSVl3/4f8vLT4grtzCWBo4SDVOO3ZeXkr73vbzV7HFl8sfUxk06snrt369Jad94+gssvkxefP/n+fPHJI/A4G7DCRp8NU5VK8U0k27pO/TzXqunZ7y2WOqcSVVToScW3pZ//ALN3ft80rWl+k37G/wCwV+0H8TfFfhr4kHV/FHwA8M6XffadP8YWiXunfFC+RLVZoh4S0iVoU8OCYStbS674ogeG0EkM1r4Y8R6RcXqW/wCN8eeIOQVpToVsJRqt+5OFSK5JRk0pqXuTTTjzPlcbS+G6TbP2PhHgjPqcIVKOJqxTtK6lrF9Ptaff6OX2f1N/bB/YG+NfxZvpPHHhH4u+I/iReRW+LXwN8RNQ0qy+zMREZD4ZvtDtPD/g/T5bmV52ksX8OaFZkIslzqs5SNW+C8P/ABDyPJMPLD4nAwhCpBSWJo024TkoTTdalaVVTUIUaakp15VJSu1CMfe+w494EzrNMXCpRxU5JTtyuUfdV07R96KevNdLlUbXbfNaP1J+zL8DPHOgfAr4ZaX4g+JXxb8Ba/pnhi3ttW8KWd54Lay0a/yQ2lKNc+HUkjpEWGz92oLMwB6FvkeL+J8DmecYmtQwWXKjOdN067oy9vUXsoc0p1IzXNyNcsPdi+TkSckry+t4R4WzTK8vpQq4qvzcsrx57pa6e403rvK0pLSz5Grnvo+D3iDt8ePjL6/8ffw2H/vN8f59q+Y/tinPRYLLX/hwzv8AhU1/ruz6ZZHioa+3rv8A7ea29W1vpv20luc/oen+KvB/xq8PeD5fiJ4y8Y6N4i+GfxG8S30XixPB/wDoGqeGvFPwo03SDp3/AAi/hjwu6RyWvivXROk32iMnys4KxFXWxVOvhlOWHoxn7ZaUr0/d5Z7353v5+vLZII4TFU5uKqzso2u3d/Z6x0fXppory15f/9X+/bYvp+tAHL+IPCmjeIrOe01KzhuYJYfJmiki4nwQwOQNwcMoKOoJVhuUghHrZVZdXu/u/K/fb7gPxQ/an/4JHeD/ABa+oeKvgTeW3w08SbWuT4UMMj/DvXJsWsUcX2VC9x4MiWOJ1WTw5Fc6NZREpbeFZbmR56/ceFPGDE5fUpU8Rdpys5TaTs7vdRs7JpapXWrvJH4bxV4RYfMKE54aSnJRvaPVrZP8+m1ul5fgh8U/gx8Svgd4ifwx8UfCOpeFtUVnW1vbiN20rVxG7xO2k6zB5+k6okboRK9heTiFvllKyFVT+hsrz/KOL6MKlavBNxTs5W1aT7pq3pfyjsfz5meSZvwjWlCjRm0m0nyvo/uV35f+A2PN69nCSrU7ta29Pu6rbz6aXseNia1bTmpv9fx+XT7rBUY3NcTTVlF6afl5Nb+fpJK5vgsTbR6ef9W7fdrrsFaiCgAoAKACu6tl8MzV20reasr/AD/N/wDbyt72UMTPLLJRfvb9f0Xyt9yD/Vf5znP/AHz/AE6d814lbEYbIX71SKs9dfu7+m/3nZDL8RxArqEvuvb8F+FvPmtY/Rj9m3/gmp8aPjlNpuueOI7r4SfDy4kKTHULCGTx9rkK28UyxaDoVzLDH4cEpdoDrvim2kbTpxbXen+Gtf06W4Nr+Z8ReJ+Hq0Zxg1OVo31cUouVm+ZKSlKMVKahG7fKk3Hmi5fp+ScBV6FaDlTlvvbt0W3pfp52uf0P/s5/sYfCD9nnRVsfBXhi3sb6e3SHUvEFyovfEmr7ZnmUahqzok6xiSTzBp9lHY6NDKvmWun2crEp/POf8Ze3rTcZJXl5N7W030b1XVN/bikj92yThqVGlC65bLW/ffv6pfD+DZ9nWtpBZxCKCMBe+Mdfxx39ufbA2/DH3hboAKACgAoAKAGj+H/d/wDiaiH8H5fpIJfH/wCBfmj5k/bC/wCTaPi7/wBidq38hXucJ/8AIzo+tT/02z5viz/cv+3n+UT+RSv9Lz/PsKACgCxpdxDY6hYXtxYwX9vaXAuLm0n+5dqOqNhlIXAxgZyOoOQGAP0C1j9qfwJqWveG7648WfHfWrHR/G/xU8X2N1q1sPD2oeCNL8eeCG8P6D4N8GnwN8YPDPiNfD3hXd86+F/FHwoQAkFfCeSrfzhDgLGvD46nPB5fTqVqdKnB0akprFzpubeIxcsXl+I9nOo535atLNJU5c7nXxEvfj+2f630FOk1Um4xacrxs4axvGmoVYOSWtmp0L6aU/tdBcfth+CdW1iC4u/Gn7QOlafp2vvq9tpeg3nizTdM15ZPhtofgdb/AFuVP2hIfE72tv4j0dvEkJf4r72TXVkm8avM8krNcA4pYJUvqWCqYp4SVBzqTw8eXmrRqKgqkMl9i8PTU5zi3l/K6lOHPhZSlUqxP9cKXtF+9nGmqkZe6pX0STqcjr39o7PT2vWyqRSsV9N/ag8CeIPiFc+ItZ8WeIrfwxonjT4Q+LdK0vxGnxDstRvf+EK+GPj7w/441nRI/wC3Pit4h0TXfE/iE6Iv/CTHxV4u8YyIuJPGrFizdEPDzMcPkkaFSnT9q8uxuEqThHC4hU51sRhvqiV8HgFWw+Goe3lKjClh6cox5Z0q1V0pnPmnHmHxWd0q8HeEa9Kp8U4XjFe+r883GUnb3uZuLd1KKT5fLP2uvid4G+LN74G1Xwd4itdRfQ9G1PR7zR/+E3+PPxU1ZAdYGurqsvjf4t/D/wAM+Im0OMDZJ4aikELsMxqE2hfR8GOHMZkeZV6mMpTpzdRSTnSw1HeHLyOGGr4iDlBx96XPGL5oxhzqPu4eI/EVHOMppU6Uubljy/HKW0r3TlCMlF3uk05dW4ttS+MK/dT8iCgAqcJ/Ezz/ALl//bRVf4cfX/29H64f8ElP+SpfEf8A7APhjr6fbfEP9Pwx/s5r+W/H7+HkX/c1/wC2n9JeCP8AEj6f+2I/oYr+XT+lwoAKACgClPFFKp82IEEZ2nBPpx938OV+hz8s1HyyjLm2e/3adfzffSzUdKTbTjtfo/n89+36o+Qf2jv2N/hD+0dobWHjbwxb3F/FAYtN8Q6cBY+JNHBkjmxp+rBXnaNpEEhsL9NQ0WaVjJd6bdShK+6yDiqrlcoJtOntZ2vtJK/Sybu/hnLlXvcqly/BcS8MfX3KUU3d309e3lturt9bpx/nC/af/wCCdHxm/Z8mvPEGiWk3xM+HcR89tc0OGIeI9DtD9okePxFojRxrcRxQRKZdY0NbuxkSO51HV9O0a2eGyT+hsg8S8PjJxg5ct3ZKVr9Oi9eXd7dfs/gOfeGmIwkJTUHKyvpf/N/gl2V7H57wy+cP3HH/ANf69fwI/Div1LD4nDZrTjNVF7yT337af8DW3S6Py6vh8TldSUHTkknbbt308v5tO8rWCvReGjk0HytX/rrrfR6aeqT0MZ45uSun80+/na3rd7baI9//AGbfiXpHwu+IkniDWLXQLdH8L+MrHTPGWo2viufUvBV9qvhHWbLT9Y0OLw9Ldq2opPcLbp9p8JeKCftGyKwRTO7fD51gZ55Jcs503GcZNwUHzxUrum+aElyzXuycVCaWsJwdpR+kyrExau7PTrdW8927q3969tU07R9p+MnxH0rxd8JNF8LW/wAXf2btb1Dwja6y2tW3gn4Bf8IXf+KNV1zxoNdH/CE6w37O/hMeHhIBnxksXizwgPGP/FUlv+EuZiG+WyDIJ0cXhqv1PNYupLEc8quYKrQw3KnZ18PLM6sJRxFv9nWGo1/ZXXtVhmpcn02c55CrTqR9pR0VO3LT5ZTVrvlkqK1g21Uc3Fy6OrZM+F6/V6eE9nBfLt5fPTbf7tD4j637STt1b/D8dvL77hVrCUMsi5Smlyq+/wDm/Tpr1tozj56uZSUIwbv5P7vs+n4aXcpe3/Av9nL4v/tE63/ZXwv8LS3umwXS2+s+LtVP9neDtCR45Zt2p65cRtBcTrGiyHSNIh1fWpLeSO6ttFurYs6/HZ1xrg8FzKVRe7tGPvTl70IrljFNv3pR5n8MU7yai24/W5RwNisTGMlTaTWj6O+rvo+ifTXaytc/om/ZK/4JjfDT4MPpfi/xfaxfEL4kWZivx4j12ws5rLQNTFvEm/wjpsyXEWlS20one01dpp9aiMkv2O+sbe5uLJv5+4u4/jmKdOCjytWu2721u99LrlSutH/MmpH7twhwLUy6cak00k009rPs9V6J636ctkz9X9I0Sx0mHybaIAZ5Ofbjt7fn6/er8dqYx15yd7Xbfe/4rp/d1t0t7v7DChGlCEN7K3b/AID/AK8jcrEsKACgDwfWv+TjfAH/AGRn4zf+pt8BK71/ukf+vsf/AEiR42K/3iXp/kf/1v7+K5/af3/wAKPaf3/wAgQgrgvuGcggYx/M/of+A5FV7WlirKk2pJ3T2tb5vW2t236dY504VKEWqkr2XVbd+m3W1teyueSfFL4K/D34weG73wp428NaR4j0PUQEutM1WzivbOUAY+aKdGj6ZI3KwBAOCTsr6DK89zXKP93rzS0VlO2n/A6Ppey3Z4mZZHlecSvXo05yd23y/q9t77S7W2PwA/ai/wCCR3iLws1/4p/Z01FdW09VM7/DHxZe4vAR9ljS28JeL5l2EBRN5Gn+LtrEh7jUfGGGW3r+guGvFxY2UadZxlKTs5QvGNruz5ZSk1aPKpWlrLW8YtH4PxHwFSwlOUoxtZaJ7y0+dur69PibsfjR4h8Pa94M1+/8K+KtE1bw9r+msY9S0TW7SaxvrRx/C8U8YlQ4w2HVS2d3I2mv2fA5zhMxpQqWUlKMZJqzTT1un2d7p3afla8vxrGYFYOrKOiadrf1f+traox69U4woAKACgD6n/Z3/Y5+Nn7Sd7BJ4R8OyaH4M+1Bbvx74ojl0zQ9geISNppuVM+teV5u2dNGt9SNrIhS9+zbWdPz7iHjd8P03FT9/kuoRfvtWk02vsxfJJKUmotq102on2+ScMwz+tFNXd1bp19H6fDH56H9D/7K3/BNz4TfANbLxDLZ/wDCb/EK3KSf8J14rtbaS506WPz9j+ENFWKbS/DaL5zNaXgj1PxDaBpbZvEd9ZsI6/nziHxIxOcVakHWk46pwWiknyu09VKVuVrljKMZQnKMk7Jn7tkXANDLKUW6cNUndpP7tujXn1V7JH6YaZomn6dD5MES47nn8Ox/X8jX5rXxlevvUkvXVr+vl89Ufp9PAYantTV+9uumunbpf/M1WC5649sZx/nrXkVMHOs7t31vdu3rpp59X362O1VIwVnZJf12aXTo/lpyy12iCgAoAKACgAoAaP4f93/4moh/B+X6SCXx/wDgX5o+ZP2wv+TaPi7/ANidq38hXucJ/wDIzo+tT/02z5viz/cv+3n+UT+RSv8AS8/z7CgAoAKAClyUv+fa+/0/u+X4va/u63q9398Qo5KX/Ptff6f3fL8Xtf3S9Xu/viFc9TF06k1Qskpbq1/0Te//AA17R56eFquHtm3zRV7v9d7PTt5K9goqYSnl0ViKcVF76aN3t2v6bv8A7dtcini6mLn7Cbcoxdkr6adNlfS3fyewV0mwUAFThP4mef8Acv8A+2iq/wAOPr/7ej9cP+CSn/JUviP/ANgHwx19PtviH+n4Y/2c1/Lfj9/DyL/ua/8AbT+kvBH+JH0/9sR/QxX8un9LhQAUAFAEf3/Yj8f/AIj+R/CpaU1rqn/X9f8ABHrF+a/rz/LTzsJsYHj8xx/Vv5/n/ClFqyUnZbeX5/8AA8tiuZPeKb9f+BH8vuv72TqGkWOpw+TcRBlxwccjP4gH68/0rvw2Mr4CSqU6jS827Lt/TenaN1zceIw9DMacoVIx23tZ/f72i8r+S0bl+UH7WH/BLb4U/Gie+8YeA/L+GPxFvZmvb3WdDtoovDXiO6W2khCeMtHitJlJnuPLku/Emj28GsiQve6tBr8dlZaM36fkHiTjcDT9nKo6ijz2W0pNtPljzOyteUUpOKS9nFSspTPzDPfDbC4uo5xhFJtPyXZ93e6d7d2lK6jH+dj45/s8fF79nfxA2ifFDwxe2NpcXJttI8W2qfbfB+tBYUuCNN11I/scd0YGM76Tfix1m3gV7i80i2gKtX9HZLxP/rPJQjNPvH7STlKN2t1GUoSUZbPldrpXj/NuaZAsNFtqzXXpoum3e9ve87XSPEK+2WCjgYqVlrv13/Ff1dy3j83TxEsNLku9H5909tl97721tErnwlOFNaLT0v3Xk912+6w8RVnUly3eu+t72s9du/T73ua/hnwz4k8Za/Y+FfB+iat4h8R6k23TdE0S0lvb25J7JFAJJXIGSVSJuFzhQHC44/MIUITqVJxp06cXOc5uMIQhCPNOc5SajGMYq7lJwSSu91y+nl2C9olvdvT1fRabu/d2btrpI/bD9lX/AIJJ6jqz2Hiz9o+4jSTzo7yL4W+G9QkltIFgllbZ4v8AG2lXVr9qmkiMMdxo/hZIbC0u7dLi18X63ptzPYN/PXE3iZXlSlCNRq8XqnZp2VrJq3V6t6WslZNn7dwzwDSVWMpQTd1vv+WnTsrdXofvp8P/AIU+Dvh3odh4f8M6HpWi6Nplslrp2maRZx2Gn2cS8BYbGFEgj7tiNVG4lsEs5b8LzLivE5hVk5VJWcn1s2uyetl1d399j9xwHD+GwVKMfZxbS2sml99/P520Wx6gdqjBJ9ew6+ucc/h+WTXgqlVqWc5O3m76dt1r8nt0sj2ZVY0ltFbaL/h0l5+876J7piqyjpxn8f6t/P8AA5yukKDhre76r5f1+olWjPq3b5/1638tSSqLCgAoA8H1r/k43wB/2Rn4zf8AqbfASu9f7pH/AK+x/wDSJHjYr/eJen+R/9f+/igAoAKACgClPFFKgjli+U8FRg/lgjrzjr+GCtc9a8ZqSd1fR29Px+752ua07uLvs9LPp90tej0cezR8oftB/sffCD9obRTp3jvwlY6hcxIsWm63bKttrukJ50czrY3+JGCO0aeZDMs1rLgedaz7EK/ccP8AFVXLnCzThGLUoSSd5ctk72TWuu70Vux8DxJwx/aDlKC5ru91fRXvtpa2q7t662sfzt/tJ/8ABMP4zfBZ9S1/4dJf/FfwXbSGWGzsrRX+IekabHaGZjqOk27KPEs5ljMMVz4YtUvLyaXe2haPZwidv6H4e8VqGKqxjKcVzO2+n36rf06pdGfhOfeHOIp03JU5Oy+/b12a/wAm9JH5n1+xYPNMPmlOE1KOqT3ve/8A4D+XX7P2vy+tkeIy+bjKLTv22X49v87XueofCL4IfFH49eI08P8Awv8ACWoa/JE8cepatsNtoGkrNII0fVNWcCzsA7OojN1NCJ3Hlx73JRvms0zzJ+EKLnRrwbs3Fcyu2o3slq72TenNovhVjryzJM3zWvGFehUSbSd4tW876pWXm/Jq/u/vd+yx/wAEm/Angr7F4q+N09r8UfFRzcDQ5LZh8O9JZvtMbRppMuyfxUHjmVHk8QiDSL+LH2nw1a3Eaz1/PfFfi5icyrVKWF5nG9rwtzWvo02nbS8dFdbqUXFH9A8MeEuFrUI1cXJQbSupX3te26bemiT187M/Z/w74R0XQbWC3sbC3thbcAgDjjHTJ5787R2GOtfkGIxUptpXcX8NlZ6d3v07Pyv9n9kpYdprpa123fm9F5X/ALu6TevMdmRnivFrLW/b/gHeFdIBQAUAFABQAUAFABQBFg7Pxz07dPX8c+nbvWeJ1tbpp31/Dv57dBUNEk+q/wAvvvbTbe+u0vnf9qjw9qXif4DfErQ9Hs5r7U9T8Oz2dtYQJ5kk7zlImUIrbjiJ2l2oVeTytijc5r3cjrKhiaUnblVOau3ZrTSy1u+lm9E29be94Ge4d14OzbfNHRLR6NPq973+XW15fzO/8Mg/tIbc/wDCp/EHX739peHMY+n9sYz/AMDx71/c3/EUfDn/AKHT/wDCat5f3fP8Hvb3v44/4hn4jf8AQmX/AIU0Q/4ZB/aQ25/4VP4g6/e/tLw5jH0/tjGf+B496P8AiKPhz/0On/4TVvL+75/g97e8f8Qz8Rv+hMv/AApoh/wyD+0htz/wqfxB1+9/aXhzGPp/bGM/8Dx70f8AEUfDn/odP/wmreX93z/B7294/wCIZ+I3/QmX/hTRD/hkH9pDbn/hU/iDr97+0vDmMfT+2MZ/4Hj3o/4ij4c/9Dp/+E1by/u+f4Pe3vH/ABDPxG/6Ey/8KaIf8Mg/tIbc/wDCp/EHX739peHMY+n9sYz/AMDx70f8RR8Of+h0/wDwmreX93z/AAe9veP+IZ+I3/QmX/hTRD/hkH9pDbn/AIVP4g6/e/tLw5jH0/tjGf8AgePej/iKPhz/ANDp/wDhNW8v7vn+D3t7x/xDPxG/6Ey/8KaIf8Mg/tIbc/8ACp/EHX739peHMY+n9sYz/wADx70f8RR8Of8AodP/AMJq3l/d8/we9veP+IZ+I3/QmX/hTRD/AIZB/aQ25/4VP4g6/e/tLw5jH0/tjGf+B496P+Io+HP/AEOn/wCE1by/u+f4Pe3vH/EM/Eb/AKEy/wDCmiH/AAyD+0htz/wqfxB1+9/aXhzGPp/bGM/8Dx70f8RR8Of+h0//AAmreX93z/B7294/4hn4jf8AQmX/AIU0Q/4ZB/aQ25/4VP4g6/e/tLw5jH0/tjGf+B496P8AiKPhz/0On/4TVvL+75/g97e8f8Qz8Rv+hMv/AApoh/wyD+0htz/wqfxB1+9/aXhzGPp/bGM/8Dx70qniXlFSsouvBX01l36dL7dVr5WSOleFmKVNzjTlZav/AIPo/KNu2x+k/wDwTP8Agv8AFD4UfEPxrd+PPCWoaBDqmgWsFnNdRtGt1/xM7Z2xlQMYgkbPONo3Y3pt/BPFvNstzmipUa9Opq03GalZ2u1pez1V03f1uz9q8KOG6uU1mqkHGyW68rdlbRW29L6n7kV/Ph+6BQAUAFABQAUAFABQB5p8Q/hd4P8AiVoGoeHPFvh/SPEGj6nbNb3umarZxXdncxnJ2yQzJJHIPYoSDhgQVFetl2ZywNWM1spJu65r2tp0duunLt105fGzHLPrdKUU9eVrlvvp001/Pty7n4AftU/8Ej9T0VtQ8W/s3XBltlinvX+FOvXcbMZI4YjFH4R8Z6rd2ybHlRoYtL8V75VV5r668W3EyW+mS/0Dwd4mwwdJUqvL7qUVUWjl71TSdO75OWCpXkpyU5zk0qUFE/C+KPD2ri6rlGMnzSb201a23b1va+qVr2vY+SP2a/8AgnB8ZfjtqEeqeKo5fhj4Bsrqe0mvrzZc+JNdm0PW00zVdK0DREjf7HgQ30aa5rcVpp8DrDfaRp+twRXGmp9Hxdx5RyWuqaleS+Jq1oc0W1dve9rWjzSV4tqMJOUfmeEeCa8qN+Vr5X6paaq+y6/ef0Z/s5/sd/Cb9nvQ/wCz/Bvhq2sLiWPy9S1ufbc65rCebJKq31+Io8opkYRwRJBbQgsYbWLLK/4DxZxVU4ibi2nCSXIoxSUGuW7ve8rvWyWj096ykfuvCnDjwUnKUWrJPX7T/K2vfzSlZ8v2Ha2kFnEIoIwF74x1/HHf259sDb8GffFugAoAKACgAoAKAPB9a/5ON8Af9kZ+M3/qbfASu9f7pH/r7H/0iR42K/3iXp/kf//Q/v4oAKACgAoAaQGH8uPTPuv8/wAs/NlpVXZL57/da3/b1/7v2qTcXt+P/Al+X3291AhHQ4/D/wCyb/Pr/DHspRd4y+7Tvvdr8n8rjc091f5//ax/rt9rOu9NstRhMNxGrDGQcY/lzjPGOM/mG68G6uAmp0pvTbV/5vt2l6LVHn16OHx0JRlBXa/lX+av23v6n5jfta/sN/Av4j+JPh94q1nwXYrrur/Erw7oOsalp013ot1q+m6i0+bPVZNMltZtRj+VlS5llF1Zou20uoEaRX/WeHfEHH4KnUp1Kkp+yoKVN6NuUVJSu1Gzu4xsmk9W+Zq6j+YZ9wdh61TnUIq83fTXXls3dL+bRvs7RSd4/cXwx+C3gX4XeHrHw74S0DSfD+jacu210rRLRLCytxjOI4YVWNBkcqqcnJ4yWb4HNc8zTNVy4qvUs9NZX+7fot/y0R95l+S5VF81LD01KL2SWm3k+m1vxtaXsCpGqYiUKM8qcn+ZP9MfiBXz/NTw0f3jtd35tfe6W6/c3Z+dlze46VSiuWirbPl6adtrW9G+r5rpD1fnIzx1Gfr7HPr0/Pk1h7WWl3ZPrv8Aonv/AMC1rR6XBW/r8rr8/PTYmYkYA6n+n1B/mPxqkr6GQ6twCgAoAKACgAoAKACgA/z/AJ/z/Ohq+gFS4ihuojFMMg9s8n9AffqPrxlhV/YyVn+fk/6V16qwpUVUS5ldd9P6/B/LaXO/8I3pH/Pun/fxf8+ta/Xp/wDQa/8AwCn5f3fL8Xvb3suSX/Qsj/4USI/+EY0j/ngn/fa/59aPr0/+g1/+AU/L+75fi97e9Xs5f9C6P/hRL/5H1/4NveP+EY0j/ngn/fa/59aPr0/+g1/+AU/L+75fi97e8ezl/wBC6P8A4US/+R9f+Db3j/hGNI/54J/32v8An1o+vT/6DX/4BT8v7vl+L3t7x7OX/Quj/wCFEv8A5H1/4NveP+EY0j/ngn/fa/59aPr0/wDoNf8A4BT8v7vl+L3t7x7OX/Quj/4US/8AkfX/AINveP8AhGNI/wCeCf8Afa/59aPr0/8AoNf/AIBT8v7vl+L3t7x7OX/Quj/4US/+R9f+Db3j/hGNI/54J/32v+fWj69P/oNf/gFPy/u+X4ve3vHs5f8AQuj/AOFEv/kfX/g294/4RjSP+eCf99r/AJ9aPr0/+g1/+AU/L+75fi97e8ezl/0Lo/8AhRL/AOR9f+Db3j/hGNI/54J/32v+fWj69P8A6DX/AOAU/L+75fi97e8ezl/0Lo/+FEv/AJH1/wCDb3j/AIRjSP8Angn/AH2v+fWj69P/AKDX/wCAU/L+75fi97e8ezl/0Lo/+FEv/kfX/g2953/CLaR8p+zj5jj6c/8A6+35Vy4rFZosTTlSxM2nb7VrWtvr8vxtrY5sHgsLPDuMqMZcyd2klbtbbt2+5s0bXRNNs3aWCBVfGFPUD35/Dr+HQ12Y6risdSjTqVZNpa8zv8t2nr2Vn15bDwGBp4OtUlSXLe/S1vT4vLZ+eqfu71YHYFABQAUAFABQAUAFADDlgCOP8kcH5ev0/Lq2VeLSum/dfTTf5v8AN/LTlcJLdrR/Pr6K+3ZfP7Va4SKVAswBHvx16d/Q9cd8cn7tYOdda87S+96dml02+TtzWbjy4t0be+k+1/Lvu7dbefS75fA/2bbC0g+GfmQxKjDx/wDGIDHBz/wuHx4Tzz3zgj6gDOK9/i3EVsyx+IXPK0pULavRU6FOHk+n/BdrnzXB2GpLAxqW953TbX9+Wve65V1s/wC7oz6AYLtIye3PAx0/3f8APQjGW+bnzYShzcztFpO72T7b9bbb31tax9dCcYVLWUX16L03+W7+d0WK6ACgAoAKACgAoAKAPB9a/wCTjfAH/ZGfjN/6m3wErvX+6R/6+x/9IkeNiv8AeJen+R//0f7+KACgAoAKACgAoAKyfx/NAeC/G37/AMJv+y0eBf8A0DWq9zL9sT/2BV//AHEeFmG8f8dL/wBuPeq8g90KACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgDwT9nYbPhgrf3/H/AMYh17L8YPHXX73p1x+fO32OKcWp5piJJ6SdGKv0UKFKKd9N0rbP8ufweGMGsNl8Fv8AF/6VJu6173veX4+57sw3J79+/TOD/D9T/T7zeJKPtIJPRSs9vS/Xs/736n0CajJ/NfLvb3u3f5v7MtWSFABQAUAFABQAUAeD61/ycb4A/wCyM/Gb/wBTb4CV3r/dI/8AX2P/AKRI8bFf7xL0/wAj/9L+/igAoAKACgAoAKACsn8fzQHh3xntby6f4X/YrWWf7H8XPBl/emNA5t9Ot01UXNy5z8sUbPEJZMsEEg3D5gW9zLU2sTZ3vhK6tq9W4W/yXeztszw8yspLS1p0+2yv569+vm19r3GvIPcCgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgDkfGPjTSPAulHWtas/Fl9aLIImg8G+AvG/xE1UsT98aD4A8P8AibXmjAILSJpzRr0LAnaoB85/s7fGj4SeKdD03w18Nrv4neKdN1XWfGWvWfijU/gp8YfD3hS4m1zxh4n1nWmHjbxB8PtC8HPHb69NqunwKusPcSPDHEqTud7fQcSZfiIY2rLFUKeFqvlU6SxNGtJTjCk3/DnJq8XCajKEUoyi73fNL5fhXM4YjBqMZOaTupcs4qzlJKzktbtNXX2laytzR+tAcluvC4559MdMev8A+vGa8OS5YJef9d/67bH0UKnPUn5LT8Lv5t33087k1ZmwUAFABQAUAFABQB4PrX/JxvgD/sjPxm/9Tb4CV3r/AHSP/X2P/pEjxsV/vEvT/I//0/7+KACgAoAKACgAoAhZflVehOf1+ue3H+GcVjX/AHlSKWv6ap3+z676ba/adFezhbtbb5/1+J+Qv7dHxW13w98T9C03wzL8eb77L4RuLV9B+HfgD9pC80s6mNX0JoNZFx8Pf2hP2XvCuvW9xL4y8O+GPMXxR40mTxTKPCEyeFL6KVJ/0nhXK4YjDLmjhlU9raE688LFtRw86jp3xOBzCKdqftbRw6/2b9+pS5Zxn+ccY5hJVeX3nFK0lFTd7uKu1GpTl8LaXNP4vdbafNH7V/ZD17VtX+Dejad4im+INz4i0u51iXU5viN4J+K/gvX4Ita17Vde0fRXj+L8V54n1YeGdD1PT/DTai3irxxC0mkGFvGPiCaKXUp/muLsHQeMrPDSpTw7lCNN0amEkpulTpwlUUcJOVNKrJOpbkw796zpwvY9XgnF1J4JU6sZQlFJvnjUT1lJpPnSastNOa8VZSVrH1QMKP8AdH8/Tj9ecdMDHzfPJeypRTen9afJt9Ol9T7L45vv5+f/AID0V3r5abyfViCgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgCLB2fjnp26ev459O3es8Tra3TTvr+Hfz26CoaJJ9V/l997abb312l83/tR+GNQ8R/DG/k0Pw3qXjXX9O1Pw+mneDYZ/El1oHii31fXdL0LX9K8XeDtH+I3wu0Hx94cl8Oapq0t94V8e+K7TwJdCJLrxKj6faPt9rKKvsqtKLn7KDTcqkVSVSPLCaXJKpCpyt3s3G7irtNe8zws6oe2i3yuTSSUHz8tnJXvytO6S2ur9eb7Pyd+zl8J/iJp3xK0eH4j/CCPwrZabpHi/wAWaX4/0bwVofww8Qxazq3xOtvE4+Gura/8J/2qPi8PEHw+Q+N9Ynt/hh4rtYvBrnw/BcRnxRJ4WEy/SZ1nEMXSbjisRUcoUYezq1pV6a9lhowdSHt8LCVOtKpFzlVinUqOSqt0nX9z5HIeH54epFuEfdclzcnI3eeifLVs0ldWblZSnC81BOP6iKu0Af5/z+WeuB0r4SbVR6Pbyvp98b/J/ddc36VCPJFR7f1/X/BHUygoAKACgAoAKACgDwfWv+TjfAH/AGRn4zf+pt8BK71/ukf+vsf/AEiR42K/3iXp/kf/1P7+KACgAoAKACgAoAQnAz6f59D/AC/KsopqSut79b/1/XYD4p+NnwG+G/jW6v7n4i+LPiKdR8f+I/BfgrRW0vVNMk0vRhBN/bOheDh4PbQ5/ht4l0OLWW1fxJOfi14S8ZPJPOymXzIvCcMX12WZ3isDGP1b6upU1VqfvYSfNUacHX51espqi3Si6dSEl7Rxc5wUoQ+bzPIqWOm5T5pKTitJK9t+W3wtOST1Vly6JXXN7T8F/gzpnwQ8OzeF9D8R6prOivKsun6Zd+FfhP4QsNGGW8yHTNJ+Evw3+G+gxQzblLmfS7i5Ypua4YMEXwM2zipi6tP22s02ryqVqsrOzs516tWbta+tR26Je9z9+S5PTy+jJQ05t/djFJK9naMYq7/ux8nZts9qPG/8D/nr6/8A6ulcVe8qKa+/7kn0+787XPUpK1SXm7/+S+r6+noSVYwoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAP8/5/z/Ohq+gDW2/xD/6/Xj1/pzUuqqdlf8PT5fjptrcORT0cU/Xp+K7+foeP/BLxBqviXwOup6zeC9vT4v8AiRpyXQ/5a6boPxJ8V6FopUe+jaZYhmJyWyTlmNevmlD2WIq0rcqioK0VbeCld7XfvWWkb8p4OU4lVYxf8199WtdEtrXs/wDwLS92j13O0nv+n9G/p+uF+eu6cmt7Pf8Aq+/qv0Po7cyXT8f1j+v4XlLXYZBQAUAFABQAUAFAHg+tf8nG+AP+yM/Gb/1NvgJXev8AdI/9fY/+kSPGxX+8S9P8j//V/v4oAKACgAoAKACgCuT5fCr1PXOevfngZxjvjPfha5quJqrRQsv67p/nJbbWHRoUoLT8bvotPs7Ptb8bHhXxtZQfhNnAI+M/gdiPYJrW4n8xnIGOBzya+lyalXrrGuEW3HLcXUnZbQjCm5NrWy9Xdedj57O62Dwzpqc4xbq04q7+1zOys29dGrXdvL7Pva7eq/z/AP1+vXP514Kpez/z/r8um/U99TU0rO/9ei8u/rqOppp6r+v6/rcYUwCgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKAI8f7B/M/wCe/wDnBqPq1Hv+H9dv6uL2tT+V/fHy/u+f4Pe3vGP9g/mf89/84NH1aj3/AA/rt/Vw9rU/lf3x8v7vn+D3t7x839xa1/dfzkXq/wAkfvD5v7i0fuv5wvV/kj95GUIKqowq85PQnP59/XnuFOd0UYU8PSUU0l/XTTtb/P7UV1VrVY2Vox15tPLtd6L7/L7PjHxx+N3gn4BeE4vG/jpdUXQZNTs9Le50rT2vvs09+WSCW7jVkaO3MgEfmLlzM8UMcck80MMvq5Dww+IcXKnh5UVUsp3qynGNm7NqMFJtprrbdb3tHx+IuJFw/hoValKU03ZW8unS+nmttb7S+W/2Nv2r/hb8Swvwp8Jf8JHfeJrfUPiX40uRLoko03TtE1j4k61q8MlxqiyNGNqeKNLhQSIm95mTiWK4RPvvEbw5r8J42U8RPCKhJUVSVOpU9p7T2NP20VGdNJJ1VVlTtOUnTs5cr5j4Lw58Qp8V4WNPkqOonJzuo8qi27WkmpOy5b80YK90rWR+hZBB5IKnpj9e5z+IHtmvyqtToYml7NySS0Xl5fFd7Lz81q5fqsPbUqkpW5ovXTe//kr/AC8r2JPm/uLWv7r+cu9X+SP3h839xaP3X84Xq/yR+8Pm/uLR+6/nC9X+SP3h839xaP3X84Xq/wAkfvJKk0CgAoA8H1r/AJON8Af9kZ+M3/qbfASu9f7pH/r7H/0iR42K/wB4l6f5H//W/v4oAKACgAoAKACgCuGCDazZJ6Y7fz/PPGMc8VlUxVFVIxvd9/V/e9/5V5Ju6Io0Ksacrybf5adP+Ha9Fofgr/wUjm+Pnwz+Lnh/WfCXxL8er4H8b32k6x4b0S38Q6jDp/hnxroVsLeXTdJhgliWKGaHT/8AhIYxLMzzXGq6xBEgsrO4Ff0x4QYTKM0w9aOKwOFnWip4TEOVGlL21CpGDcKitJypVY+5OE7xqeycmmkox/nLxSxGZYGv+6r1EuZThaclyyUnytNNJSjdtON+Xm3V+Q/X79nnwt4y8FfCXwhonxD8Ua54w8dHTF1DxVrfiHUW1O/Ot6o7X02li6Zn8620NJ49FtZVOJ7bT47glmlc1/P+aez9rUVGMY0qbcYcsUlJK6lJNNrlc23G2nK9Um2z9zyWtOrFc7cpSS5rvRNu6SvZ35d2/K100z2/qh/z3z/X3/XC+Rhm3p2f6ei/N/I9yu7RbXVRa8tUvP8ATt5ktaDCgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKAPCvF37RPwS8C65N4Z8XfEbw34d1uFFefTtTneOUJINwJUJt6EEjcfxyVXthkOLqQhUhCTjNKUfegm07NP3nFWkndeWrta0vLef4RSak3dO19dLWur/ADa1X2b+6tDm/wDhr79mn/os3gj/AMGX/wBrqv8AV7G/8+5f+B0vL+9/XnZ8y/t/Bd306P8Au/5v+kxv/DX/AOzX/wBFk8Ef+DP/AO11t/qvmX/QPP8A8GUf/lhzf60YD+b/ANK/+Vh/w1/+zX/0WTwR/wCDP/7XR/qvmX/QPP8A8GUf/lgf60YD+b/0r/5WK37X/wCzUPlHxl8EOPX+0yuPXgx/56+lLDcMZjjKLlToSqRSd5KpSjb0jKab76b+V7RrGcTYHBV405yau19ltt+qT7pr8Uzzb4t/HP8AZQ+L3w48W/DfX/i/4LOl+LNF1DTJbg6pbMdPnz/oWsRhpoYzNoOqQ2mq28ksyQxz2cE8rJCvmV3ZLh8dkOOo1abiquHqc1pNXcX7tSCk4TjHnptw5k5OKleGvw+fnrwee4LkScoST12V38L03SlaVmmm1q/hUfkP9gPUf2f/ANm7wV4jvPGfxW8BSeP/ABfrl0dQa01qzuTYeHtHjEOkWmV3uDIbi81tpkKCWw8QaKlzFFLLbwV+oeKWY5txVmUYxilhacFq6kVKrVn8Wim04QjycnPShNTVR35VBn554a5XguF8JKTT55XWi0jHvte9278rtytLXXl/QUftgfs1f9Fk8EL7/wBqHn8o/wDP51+SVuFsxw9NTnhpQi9eZ1aMvvSlpvbr663j+nUeJsBiKzpxndrootPa+7un9y3v/dE/4a//AGa/+iyeCP8AwZ//AGup/wBV8y/6B5/+DKP/AMsL/wBaMB/N/wClf/Kw/wCGv/2a/wDosngj/wAGf/2uj/VfMv8AoHn/AODKP/ywP9aMB/N/6V/8rD/hr/8AZr/6LJ4I/wDBn/8Aa6P9V8y/6B5/+DKP/wAsD/WjAfzf+lf/ACsX/hsD9mr/AKK94P8A/Bi3/wAYp/6rZl/z5l/4Mo//ACwX+tGB8z6Lsr6DUbW3vbKVZ7K7txc210OQyv8AdOMdNpz68MDuwSvknvl+gAoA8H1r/k43wB/2Rn4zf+pt8BK71/ukf+vsf/SJHjYr/eJen+R//9f+/ZPuj8f51Nb4v67ImHwr5/meV/G3xRqngv4U+OvFGiTwW+saL4d1O+0yW4tzdRLewwu1vvgH+tBkCjawK9SwK5rswMVKpFNXSSb7eV1pfX/g3SaOPMG/ZeunbrHt6W6fjaX89f8Aw9N/at/6Cngf/wAJCv64/wCJfMq/nl/4Or//ACw/lD/iPWZ/yP7o+X93z/B7294/4em/tW/9BTwP/wCEhR/xL5lX88v/AAdX/wDlgf8AEesz/kf3R8v7vn+D3t7x/wAPTf2rf+gp4H/8JCj/AIl8yr+eX/g6v/8ALA/4j1mf8j+6Pl/d8/we9veP+Hpv7Vv/AEFPA/8A4SFH/EvmVfzy/wDB1f8A+WB/xHrM/wCR/dHy/u+f4Pe3vMP/AAVK/aq/6CPgb8PB+P1DfyX/ABqaP0eMvpu0pu3ZSlHTt7lRPdLq/wDDpc9ev4x4hx3Xryx8r6tPra22nXQ5DxL/AMFC/j542OhHxXafDrWW8KeINL8X+Gzc+FHf+x/E+hyPJo+shXv5Uea2d3ZWCLIodwkyLI4X3MP4HZPQjUim7Vo8tS1SpqrNaP2t4NqTs4NNPa1j57EeLWJlNNvms7xUrNL4X1TT2W61trY7L/h6b+1b/wBBTwP/AOEhXi/8S+ZV/PL/AMHV/wD5YdX/ABHrM/5H90fL+75/g97e8f8AD039q3/oKeB//CQo/wCJfMq/nl/4Or//ACwP+I9Zn/I/uj5f3fP8Hvb3j/h6b+1b/wBBTwP/AOEhR/xL5lX88v8AwdX/APlgf8R6zP8Akf3R8v7vn+D3t7x/w9N/at/6Cngf/wAJCj/iXzKv55f+Dq//AMsD/iPWZ/yP7o+X93z/AAe9vej/AOHpP7VOMf2j4Hwf+pQP89w/l+IyduFLwGwXtUpU5OPb21Zdt2qqa8vxtZOOtPxsxE6D11flDTp/Ltr3fzu4x+5/2Df2yPi/+0B8RNf8OfEi+8Py6RZaCtzbQaToM2nyG+SaFNxma4nzlJGJ3MACvG/LBvzHxQ8NsFkWE/c0+RXTlHmk27ySvec5Oyu3ZNdko6SP03w74+xGb1pKpeSk9+q06W9enne60j+w9fiR+1hQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAfy7f8FLs/wDDU+tYyV/4R+wwR0J232cdRnGO/Trn5a/tfwEmv7MV9LU4+b6dLLt5+p/GPjVB/wBpxt1l+bXZu/lZLTufAFftVKp/wpT9X18l/c69P/bto/k+Ng/7Nhvql0219f8A5G23mFaYfMcVBXcWui+f/B06/haOeMwWFrKykund7Pyv+f32YV5eYYjFV/hi36a9u1+3l5bXDB4LC0d5L+kl37ea+QV1vM8XP3XF67bafdbt3+7Uf9l4Sn9tO3m1b89XfqvS9wp+1xNTVxfTX7tPLz8vmL2WGhf3k/PS35vz2v8AKzQV31Kaltq+/fbp0K9v/c/H0/4P4BXnTwabfy9Lf+Sq9vT9A9v/AHPx9P8Ag/gFdoBQAVx4ejiaCbcWv16vyXyFiKOGxLsnfb8LecvSz+97Rvab/wAhSw/6/NK/pXnZhiMVJ2UG110u/L8v6sj0sDleEik3JLydvLrr1XaOy3aP7Ovha3mfDvwOc/8AMq+Hjzx/zCLU+3PsScdTuwDX+bdbd+n/AMjp/S+65/ouegVwVftfL9ACugDwfWv+TjfAH/ZGfjN/6m3wErvX+6R/6+x/9IkeNiv94l6f5H//0P79k+6Px/nU1vi/rsiYfCvn+Z4R+03/AMkD+K//AGJ2qfzWu3A/xKXq/wD0g48w/hv/AAw/9Kgfx4V/pt9WzL+d/wDkv/yw/wA6PbZb/Iv/ACby/wCD+AUfVsy/nf8A5L/8sD22W/yL/wAm8v8Ag/gFH1bMv53/AOS//LA9tlv8i/8AJvL/AIP4BR9WzL+d/wDkv/ywPbZb/Iv/ACby/wCD+AVz82YTatUdvl9/S/3LtrvLn9rSmk7W+cvLyf59Hta4VLwWZT2qNd9vL5r8P8L2i1Tovrb5Ppb+8nr6d9twrq+rZl/O/wDyX/5Yb+2y3+Rf+TeX/B/AKPq2Zfzv/wAl/wDlge2y3+Rf+TeX/B/AKPq2Zfzv/wAl/wDlge2y3+Rf+TeX/B/AKPq2Zfzv/wAl/wDlge2y3+Rf+TeX/B/AK68DVq16mq6d1rZa+fe2vlrqcUsLSoU9F1Xys13stl9/e6Z+qH/BJz/ktHi7/sW9Mr+aPH1V4UrJ+6/utdeb/TW1loz+l/A+nQqVPeSeiurf3fVdfTTvqj+jiv5KP6bCgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoA/l3/wCClOf+Go9b64/sGwxnOM/6f07bvu9P9nH8Ff2T4ETtl015dHa+vptt/N+Nj+QPGen/AMKMNeq6f4f7/wDn+kfi3xB8PPGPhrw74T8Wa54fv9N0Lxrb6lceGNSuVIj1UaPPHBqPlHIbEE00UZ3qm4yDytyAvX7Dhcbh6ub4rDU69KpiMK6axVGFSLq0Pb0/aUfb01zSp+1p+/S54+/G8ldK8fyrH0WsspyaaUl7r5dHaVny6q9no9fu1Rvt8Fvi7Je3mn2fw48X6zPYeLLzwFc3fh7QNV8Q6VF4vibbNoUus6HbXukLqUJ+V7c3fmBWRxuR42l8uvxrlMOW+Jw8G8L9ecZ1IQnHBLfFzpzlGcMPF6SrTXJCXuttlYLhnG1tf3kvfjB2i2ud7QunZydnaK1dvhja8sfx38MviB8LtR/sj4geDtf8JXU+WtE1zStR02K+RTgyabNcRRQ6hEpOJJoWkjTcocjKrXqZPxVlWZUvbYSvhsXSjLklUw9anWp88Um4OdNzipJSi3G7aUk9pKIsdwzjcM1CXPTml8M001fZ2fK7NbbeXNdOOFZ+GdV1LQNa8R2kIuNI0G70u11a4afaLVtbN0mmZBBB85rC5QDruXBzuQN3YjNcHh7XUeZxlJLS7UeRSdt2o80btWSckm43Tl5eHyrF19OZ2ulfft1stdunXZ2tL1rSv2bv2gNb0STX9O+CfxP1DTBaWN7bzW3g7VidR07VYFnsb/Q0a3EviOGWFonY6FBqXlxzRS3GyFvMr5DE+JGQ4WpUpSzHLlOjOVOrCWMw6nSnCTjOFSDqJwnCUXGUZJSUk04ppxPp8PwNjq8IS5arU1eHuy96/WPurmVrWs3vpvc4Sf4beOrKO5a+8Oajp8lp4pTwLc6VqCpaa8PEDNsGmjRZCureaHKQ4NrsFyfsjEXQe3b3aGc068KdanVhUo1aca1OtCSnSqUpKMo1I1It05QlBqUZRk1KPvR5kmeR/ZrTs4tNNJpuzT9y6acLpq0r3T66raXMa1pd5oOq6hoN81q15pd7faVd/Y7mG/iW+06RoJUS+sJJLd1SVHjYozxhlILEjFerQxsakYzSfvRUlzRcZK6W8JKMou28ZJTi1Z6qQv7O8u3X/Bf7D8+r66vcz67TzwoA968OfsuftD+J9O0zWdG+DXj6XTNWv0t9Nml0O/023u/P08ajbaoDqgtGj8PXVu6G38TyGLw9IzpEusmdljb4DFeKWRxnOnLH4GE6c3SqU3iqPNCpF2lTlBz5lUTTi6bXPdNWurH0+G4HxsUpctWXu88Xyy96FtZJ/wAujblZx0eqseZ6j4Q8XeB/EljpPjTwv4h8K600trd/2b4m0jVNA1AROuQ50/Vre2n8uRcYdrdEfbgZKui+5lfEuX5pShiKM6VelU1jWpTjUpy5ZOL5ZwbhK0ouMrPRq2ljzMyynGYGbg3KDi9U7qS8mndq3b7rbH9jXwt/5J54K/7FvRsdP+gbb+n9OP0r/OGr8UvSP5/d/WnU/wBFD0CuCr9r5foAV0AeD61/ycb4A/7Iz8Zv/U2+Ald6/wB0j/19j/6RI8bFf7xL0/yP/9H+/E9B9P6mssP/AAvl+sSnvP5/+lI8J/ab/wCSDfFf/sTtU/ktenkH++UP+vlT/wBIqHj5l8P/AID+aP49K/06P87QoAK88AoAK5MxqUqmLhZp3a1unpp00X4/dc7cvp1KWEknF3tv579oq/r+GrkV3ZhUpU8JC8lqu/p5Lf8ATpa0eHLqVWri5e673fn036/10VrBXVl+LoVMJJKa2dve3/penTfSJH1OtT3i9NbWt+kv/Svk7tBXn5fShUxc3zJe9pr0/XTur373KVWdPdNW/rZLr5N//IlemUFABU4H+Ivn/wC2k1v4P9fzI/VD/gk5/wAlo8Xf9i3plfzb49fw4/P/ANKif0B4J/xH/Xc/o4r+Sz+qgoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKAP5dv8AgpLL5P7V2rzeRbXP2TRvC3+i3R9R1469OvH9W/sfwYhKplM0tW6cdV6b+j+duz2P5J8ZJ+zzam3p79/6tfa/prutWej6/Fq48HXutTeCPEegt8RPA2maKde8P23xFbw38UNR8c/s/LoHgfQ9E0RfEMfhzX/EJ8Sazrnh3wX4Y+F/hKTwf4N8Hs53eDRsReF4LL45njaUMfh8XLBYjFSq0amIwc62W0KOOX1mdapajiFCnOnDE4yvj61fE1qtP2tWrWrNOXHhcxcMrw/NFwUoRtPlklUbX2V8LunyRVNRiloorVR6r4dWFlafET446wtjp91rs/7SPi+TRrS18c/EP4deMNdk8PJrcusP4O1z4SyeJ/E+vFBrGveGvGfhjwr8KfFHivGvyCeZHPitvBXy2dwxlfLcLCFGp9VjgIuDVLC1sJFU6FKrTniPrLjCnGjCm62HqzxEMNzQca03N4eNX18grYGjUd6sFN1Iq3POFRxk3CUYcqlzc3MoyioTqdYRlBVEd38c/Gvjjw78XbPxPq938UPCfwT8O6T4x0/xdcW3xI/aH0y6tlfxDpaDVPBms+M/h58MDrfj3xNLHpVn4O8M+GvFHxg8Is8UqmLwj4LEnjJvN4ey6eLw+JwlHCYDGYttOjBYfLKlByarujUxlHL8VjPY4bDS9qqtav8AV5OFRUMPCrjKnJV9fN8bgXKlVlXnCL5W3z1eeNlSbcZVYU1Kck4zjGLkk0nKUaajKPyt8QPHuoa58L/i/Joem+G7/TL7xb4Y8O6t4xtfhh8Nv7N8W+ONZ1rxzrT65oPjL/hHW8Sa+vijw8/h/wATOfFCjxj4KdwSdzlq/RcLldTDYyjJxxGHxlTDe3p5dLG1+Wnh6UMPQqN4CliZ4CmqFSpCk5UIVKXtGqkKspylI8PE5hg69KV5wlTUlBzSivelzOK9o0ptySlL3nzcsbWSidfrXwzsta8Sz22rfBbxrf2Wman4X8J+KvGlv4X8W6j4c1aH4VpoOjeb4O8abs6zrnipNE8bfDT/AIVn4ZDr4y8Y/wDCtGO3xwTVw4gjhJSjDMcNGoqM8VToVMWlVhDG03im6+DjCUqOGoxeHxn12vCvLD4aWKlOOGwaqOXhV8sw9e0uWTjOfJGag+WUovl5YT5venJxlDkj8Ukkk5WUev8AD3jbxT8O/gfc6pJpXiLQLnw18OfEGvXPgNfiRpumeFvC2oN+1tdbcfAbxAG+IOkeIfCiEsnxH8TN/wAIjbroh8HOpiuY2r5jEZc8TxnKUI0sZHEKKoYqGGnKNaNXJ6leNSnmKSwWIjXw0udUMM7zpVPbxShRqH0OUZlhYZdWjOp7PlcoypyqRTjLmirPDycasXGS5ZVJqzcVC8ZK0vNdb0Q/EL4rfDLw98RbTxl4pmi+Evxj1230r4/eINX8N+JGsNE134teOPha/jnW4/EWdBf5dDVmbxcWPgxfmxtNfR4CNfKuDM0q4GlToKrmOCVX6lSjXpRqTxWCwOMWHjQw7VWcFCdPnlQTVaMlUhHkah4FsLj8e/a1vaOMJcvtalnyKM6kFeVmovRxinqn7rd7lbwb8P8A9ny+8a6jB4jHwbsbOz0v4Xf8JL4ftfiKNQ07wzqmujxCPG8XhHxxrf7T/wAL/wC2vC7yNoKyzReJ/i00XjDXvNHgwbSj9Ge57nssFgVh6+bV5TnUdSvVwLp4mvDRxjjMPh+FMWsPyyk6cFHB4BunFVJVqtSTUufKMmy2FSpKboQXKrRU24RatrTc8bTc5KKulKtVu9HCycjotV8Q/BKw8C6NpMngb4DzeGPGPiX9nO0vNM0f4i6rpup4Twj450Xx34u1zRNE+Ib+I9A1/wAMpmJzjzNniB/+E3k8XSqJU8ingc3xOPXLXzFuMcykq0sJCEKXNi8vrwwtOpWwsqNSlOnJU1KTqVJwo1Pq1al7PEKPuyr5dQppOVL7Cs5/3Jx52lNSvFq6slFNxU4y5ked/D34mWHhjwNrcPhP4Z/Ci7vfh38X/Aln4P1/W/EHj3T9L1rVNb0jxvoX/CZazrcvxe8L+G2yvh/R0O8+EvBm2ebADi3e3+l4nyjGTzJt5nmcaFWliK/s6OHyysqCpSw69nQpf2XXxNbmjOajCc8RVm1yx5pSgzxuHcyowwLTo0m9nKUqycm+bWTVZRVnb4eT4Va3vqXzV8R/+EMX4o6kPh9baJaeGxN4f+z2/hy78V3ugjUzoGkf2+NMl8cZ8SHT/wDhIDqotP7SLSfZvJEWLXyNv3uR0MTh8pw6xcq8qyVS8sTHCLEODqydL2ywP+yuapKCbo2i7XajNyhH43Ma8MRjqnJy8t9ouVul7c7crb76rzsnL+u34YZ/4V74LwoP/FM6Lg5Uf8w62/L0/Ltiv87K9LFOo+bHSqS0vU9hFcyt2d9u6afRo/0FcsN9momvu17K3k0+m6te/u98C3cKPxQ/5/z6VzSo4lf8x0o/9y0X+rv8kvnYXNR/mX3v+uv4+g44Iw+AB6Efy+b26bf1JWqSxEWk4+71Ts19+j/rd7mFT2Ml704tfc1f076df/AbnhWsqp/aM+H4GNv/AApn4zEdT/zO3wE9CvP4+3GDu9T2lRYeMWmoe0vfdc3K1brsr9fvPGqUaHt5NSSdmt+l1rdPsk+3+HRS/9L+/E9B9P6mssP/AAvl+sSnvP5/+lI8J/ab/wCSDfFf/sTtU/ktenkH++UP+vlT/wBIqHj5l8P/AID+aP49K/06P87QoAK88De8JWMV/wCKPDVheWumXVjda9plpe2erXf9n6XdaZJqMSXaahf/APCSeDxFA1oZRIT4w8JqU+V/EWkgnWbDmxkpxwmKlTlVhUjhq8oTowdStCapScZUqawuNc6sWk6cFg8W5Tslha9/ZTqPxRuk1zK6bsnrs3eNk+r5o+qtc/WDVPgv4Dkj8a6PB8IvgBouheFvB/w9tL7xz4r1vxV4MD6nrTS+IdG1jXtH1z4/+FfiP4e8Pypqujxy+J/FBTxoP+KiWL/hK5fDEPg9/wCa48V42VanKOcZ9Ko+SdSMcLltZx5W41oc+HyPF4eNenClOKoUrU6leEIucXX+sH9CVeEKFPCzthsMl70U3OqtWoqLSliackrtS5pc1lKTakouB5l+y58FrTxbo3w/1i1+FrePYdU1r4lReL7XwtH8JfiJ4kP9iaMmh6Lq+rzfFfxKnw48AN4W1/XtFHgn4ceJX8ZJ4pYeJvHjpIYPCaV6fG3GmLhhcZRjXqU4xp4SNKdRY7BUIycoVZ06FfBOONxdZ04V6mKr4evGjh6UcLQ9lJrMJy4OE+DKFXGKThGXvSlZezm0leKc4zvCK5mkqc03PW9lKmejfFb9mzSLfX/iBLd/BzxroNhpfwg0fV8+IfCvw60Bjpmj6trs+t61ozfs2fD34u/DpviKNbk0Pw4Vk8J+EZj4IPiWQKAv/CYv43DXHmNp5fhKbxPNUWNjRl++rYhVHXvO0p5pjaOJnCOGnKXsoVcRVjWVGpRpzo0Z4Sr9BmnAuHi60lTUFyNpclNWUVbahTVO7lG10oRvdSknLnPm/wDaM+H/AMJdB+Hcl54F8M6voep+F/F2l+Gp77VNP0mxsdT0zXNc+Pat/Y2q6D4+8TrrYXxB4H1fwyD8U/C0vjJvB2ieHVIGxJJfvuF+LMc83qUa0r0pwr1YzeMlWnWlRWAoRqPDVMBho0F7NxqSWCrywlPE18QqlOrWqQqR/Ocz4VoU0uS7aaUk6SjGLfNpGUas3PVNJzjF8sYtdj4Yr91Py4KACpwP8RfP/wBtJrfwf6/mR+qH/BJz/ktHi7/sW9Mr+bfHr+HH5/8ApUT+gPBP+I/67n9HFfyWf1UFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQB/Lv/wAFKc/8NRa31x/YNhjOcZ/0/p23fd/8dx/BX9p+BFP2mWST35F/Wt+vdeu1pfx3461PZ5jTa79/8PTT5+9925+f9frkcrpVMyndK9303+7TReeneSPyZ5lUp5fBczWne1tr/wA36/imFdf9jZbJOTpRuvL0t0j5r1ttvKZ1sfQXu1JLyu/Xdt/rvbTYKnly3COypRXny37f4fL5dr2PNqY3Majt7WWm/vPXfvfu/wCa/S+0Su6pUwE429lF/K3+f9dtzv8AZ4+H/LyWvnfb5rv0f32CvNqYbATd/ZR9GvRee2/2bf3vsv61joWvUlp5/nrpfTo7+Vz1rxb8GvFfg7S4tc1DVPAF5p8uheHfEHlaf8Sfh5P4gGleLNI0vWNJkk8EN4mTx3FvttUgZzN4dSJEEtw0q2CJcy/MZXxJlGZcV4+hCjiOdSr0uaWAxsMPz4X2lGsvrU8MsNb2lOSg/b2q2Tpud4o+k/sPMMHgZVPaK1loq1NyXNFTj+7U3UtaSu3GyfuS5ZLlj5rpeqaxok1zc6Pf3unz3Vle2Ulzp8xthJp2qwyWN/YyAEeZHNbySJIp+/E7AkhsV9ZioZc+Gce3Thbng7ON1eNRNNabppNaJppNWdj57J1j/rk06snr9yt69vL9ObPqsXlWBp7wjpbdb+ur2uui6b6HHhM1x09qslf18urbsvn16aBVYOhgKbV6cX5W16dbb2+63XSQsXXzCW9WV/V+Xquvfp1uwrlzDARqYuF0tGum+q6+5tdf07nqZdmE4YSSTdrefTTys9el32vq43tN/wCQpYf9fmlf0rrzGhHD4OHKraK3Xtf831fq7HHl9eVbGTv37+XbX8/W/wBr+sn4d+CPi7N4G8HzR/G17eE+G9JP2YfDnwgAv/EqiwM7wOu18ALkKckgkV/ndXzHKfaSc+H7Tlbm/wCFDEpq70tGN4xXZR0Wi0Wp/f8AHLcwUdKjt0Vou9o213evKlrffraJ2v8Awr/4y/8ARdZP/DbeEf8A4/7D8q4nmWUdMk+X1/Ev83Hz69el/ev+zsf/AM//AMKf/wAl/V+n2T/hX3xk6/8AC93/APDa+Eu3/bbP6D29G5/rmAX/ADA/+XFb+ul/mX/Z2L/6CPwiSeGvhd4ssfG9n468XfESTxhqOl+FNf8ACmkQP4Z0rRbfT7PxPrHhjWdYdV0kRSTySzeEdGhR5nYFIGcqGdVQq4/Byo+ypYaUF7X2jcamrspQXvSUns03DaLbs37zKWT4ly5nWjfl5dY20dntGS69fKztZKP/0/78T0H0/qayw/8AC+X6xKe8/n/6Ujwn9pv/AJIN8V/+xO1T+S16eQf75Q/6+VP/AEioePmXw/8AgP5o/j0r/To/ztCgArzwNjQdf1bw3quma/oeo3ml6npE1tqVnfWF1eaffWlyBxJb31hLDqcUiZJSSKWHYVBWQYVkzq0qVelUoV6dOtRrU50q1GrCNSlVpVIuFSnUpyTjOnODcZwknGUW0002hpuLUotppppp2aa1TTVmmnqmn91j23Vv2lvGt/eWD2+h+DbDSbeGKz1fwzHpus6roHi/To9Gk8N22m+Mn8Q6/rPiDXbPS/DhTRbaKfX0mimF94nW4/4WFrPiLxbq/wAfg/D3D0VOMsZi6tSoopVqtWHtKTjZ81JQpQoLnrOWIqRdGVOTqfVuRYGjh8JQ+2xPHNeS0pxjHX3IpyTu768z5naOkfeukua7m5zl5zpfxI1/R9A07w/paafHaabeeN7vdLaQ34kTxzomh+HtbsWTUEdfLuNK0WCAiLGPNlYMXKNX1NDgzBRbndc840qcnzNXjRnUnS0v9mVWd7JX2baSifNV+L8VLTkdrt7P+70XL/Lprp5/a662+PGv2PibW/E+m+FfCGnprPgzQfAH9hJb61faHYeFtDk8NmPStJj8T6/rupWdyY/Cdih1V759Q0xvPbRLzR99uLX5mtwi5U6NOVWviJ0K1TEKvNwoVKlSpOrKTqwwVLCYepFKtKDg6DjPSdb2tbmrHtYXOVTbuowvGMbLma92MU5L2nPJNuKb1Vm7R5IqMI0fHfxl1n4ix6jF4i0LwvKhu2u/D9xHpupDU/CcGxozpmma0msx6trVpKhV54vG974sD6k+p65vXxDruuaxq95dwrPCSjKVXEVJeyhTnKrPmVZwcpe1nSShRp1HOpNz+rQoRa9nT5HRoYenQMbnMazfKo2crpRXwp20TbcpJcqS5nJ9W1Jty8gr74+TCgAqcD/EXz/9tJrfwf6/mR+qH/BJz/ktHi7/ALFvTK/m3x6/hx+f/pUT+gPBP+I/67n9HFfyWf1UFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQB/Lv/wUpz/w1HrfXH9g2GM5xn/T+nbd93p/s4/gr+0vAX/kWf8Abr/M/jbxp/5GFP1/SB+f9fstH/kYVPX9EflOP/5FlP0/9vQV6JzBQAVxYD3F1/K/a2/a/wCqtcMeueXlp0u+n+Hfpv8AO15e5/s9a9H4I8d3fxGn1O302y8GeG/EzSgeLbrwZr14vizRrrwHjwnq+ieHPF+tRa/pcfiubxHaG18O36xQ6FePNLAFQy/F8Z5dLNMPHBxpxf1nE4f362EWNwsPq1WON/2vDurQ58PV+qrDvlnB+0r07NXvD6bhzFRwklLmceWMvhnyTaknF8stbO03/MrXvdXUfoT4xePJvE3w18U+DP8AhZmr+LWh8Qr4jWf4jftVN8ZZI3+Fer6/okmmeEtIPwf8JxXC+LR4yh8QRKniXz/Fng3QSN7+NfCz+DU/PeCeD8RlmZYbGVMJGEatH6vbDZS8DKnDF04YuNfF1HmNeMoUZ4R4KVJYd4rD4mtCUoUsLVdef3/FnFlLMMPOmqjck+bmlVU1JpuPLSiqPNCTVTnbc+ScYte9NQifBFfulWDhTivK33W3/wCGXne1o/kVOanUlL+Z9/VeX9d73Cu85woAKzw39f8AkwV/sf8Ab36F7Tf+QpYf9fmlf0rzcf8AF/XaB6GX/wBf+SH9nXwt/wCSeeCv+xb0bHT/AKBtv6f04/Sv81KvxS9I/n939adT/Rg9Argq/a+X6AFdABQB/9T+/E9B9P6mssP/AAvl+sSnvP5/+lI8J/ab/wCSDfFf/sTtU/ktenkH++UP+vlT/wBIqHj5l8P/AID+aP49K/06P87QoAK88AoAK7Pq2Ji/iaa/yj6dn0+4X1nDysnFa2+W3k9tflbfVSKm2Lje02u71t+P9a+g08K/sp/d+kZf13s+UpYTFxq7r0v1va/p06Ppa9mjDF89DVSvt+P3W+9d9Qp4vFxp7JL7nbb/AA7/ACt135QwnNWs5Senz6f5W6b66WCtjcKACpwP8RfP/wBtJrfwf6/mR+qH/BJz/ktHi7/sW9Mr+bfHr+HH5/8ApUT+gPBP+I/67n9HFfyWf1UFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQB/Lv/wAFKc/8NR631x/YNhjOcZ/0/p23fd6f7OP4K/tLwF/5Fn/br/M/jbxp/wCRhT9f0gfn/X7LR/5GFT1/RH5Tj/8AkWU/T/29BXonMFABXD8F0vyv/W/9WHbnu/i/D+tgp/V1iLc2vy+Xd9PJW8yXiHQ2fyXTb/Pyt52SCphUpwa5Ypbf1sl59Ne+rOmVOpPSUm/X/wDaf9P5SK74RVdXa+X9baJHHKXsN+v9ed9+8bed0FBoFABWeG/r/wAmCv8AY/7e/Qvab/yFLD/r80r+lebj/i/rtA9DL/6/8kP7Ovhb/wAk88Ff9i3o2On/AEDbf0/px+lf5qVfil6R/P7v606n+jB6BXBV+18v0AK6ACgD/9X+/ZPuj8f51Nb4v67ImHwr5/meEftN/wDJA/iv/wBidqn81rtwP8Sl6v8A9IOPMP4b/wAMP/SoH8eFf6f+0w3/AD9j/X/cQ/zftif+fcvufl/079fw8wo9phv+fsf6/wC4gWxP/PuX3Py/6d+v4eYUe0w3/P2P9f8AcQLYn/n3L7n5f9O/X8PMKPaYb/n7H+v+4gWxP/PuX3Py/wCnfr+HmFHtMN/z9j/X/cQLYn/n3L7n5f8ATv1/DzCj2mG/5+x/r/uIFsT/AM+5fc/L/p36/h5hR7TDf8/Y/wBf9xAtif8An3L7n5f9O/X8PMKPaYb/AJ+x/r/uIFsT/wA+5fc/L/p36/h5hR7TDf8AP2P9f9xAtif+fcvufl/079fw8wo9phv+fsf6/wC4gWxP/PuX3Py/6d+v4eYVGBq4XCzu6kfLXr963/XzvF5hHEYqlZU5WVru176r0/rva5+qH/BJz/ktHi7/ALFvTK/mfx6nSxVN8kr+lnppru+nkt9L2Z/Q3gQ6mFqNTi0trWfbv7vr+Dtuf0cV/KB/U4UAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFABQAUAFAH80//AAUK+H/jfxP+0x4j1Tw74Z1fWbJdL022aWy02a+UMLeFGwYI5GQBg0YJVslOh2nb/XHglxBluTZXOnWqwi3F6Npa6vb1+Wnmfyx4q5PmGYV7U6U2m+z0Wlv+G18pPVHwx/wpf4pZ/wCRD8VYzj/kCatjHr/x77f0/DPFfrOVcY5TSzWpVliIWcpa3WmlvPt2dtve+I/J6fB2bU6KvQnrb7Lt+lu2vN30tYP+FL/FLP8AyIfirGcf8gTVsY9f+Pfb+n4Z4qf9f8p0/wBph06/4f7vXX/wLpf3X/qLmv8Az4qfdLy/zf4dw/4Uv8Us/wDIh+KsZx/yBNWxj1/499v6fhnij/X/ACnT/aYdOv8Ah/u9df8AwLpf3T/UXNf+fFT7peX+b/DuH/Cl/iln/kQ/FWM4/wCQJq2Mev8Ax77f0/DPFH+v+U6f7TDp1/w/3euv/gXS/un+oua/8+Kn3S8v83+HcP8AhS/xSz/yIfirGcf8gTVsY9f+Pfb+n4Z4o/1/ynT/AGmHTr/h/u9df/Aul/dP9Rc1/wCfFT7peX+b/DuH/Cl/iln/AJEPxVjOP+QJq2Mev/Hvt/T8M8Uf6/5Tp/tMOnX/AA/3euv/AIF0v7p/qLmv/Pip90vL/N/h3D/hS/xSz/yIfirGcf8AIE1bGPX/AI99v6fhnij/AF/ynT/aYdOv+H+711/8C6X90/1FzX/nxU+6Xl/m/wAO4f8ACl/iln/kQ/FWM4/5AmrYx6/8e+39PwzxR/r/AJTp/tMOnX/D/d66/wDgXS/un+oua/8APip90vL/ADf4dw/4Uv8AFLP/ACIfirGcf8gTVsY9f+Pfb+n4Z4o/1/ynT/aYdOv+H+711/8AAul/dP8AUXNf+fFT7peX+b/DuH/Cl/iln/kQ/FWM4/5AmrYx6/8AHvt/T8M8Uf6/5Tp/tMOnX/D/AHeuv/gXS/un+oua/wDPip90vL/N/h3L+nfB34pR3tnJ/wAIJ4qwbjaP+JFquCfT/UkDp+HfOMsf6/5Tp/tMOnX/AA/3euv/AIF0v7p/qLmv/Pip90vL/N/h3P68Phevl/D7wfj/AKF3Rs9eANNt/wAvQcfTdiv88D+/j0CgAoAKAP/W/v2T7o/H+dTW+L+uyJh8K+f5nA/EfwhbePPBPibwjdTz20GvaXJYXD2xwyhznC59doBAI4CnnpXZg5qEqbaVr2u91dLXdWt+KbStvHix6vCVm2/z96Or+Wuj++5+V/8Aw6v+Ff8Az8eLv/Auv3D/AIi9if8An7I/Hv8AiElD/n3H7oh/w6v+Ff8Az8eLv/Auj/iL2J/5+yD/AIhJQ/59x+6If8Or/hX/AM/Hi7/wLo/4i9if+fsg/wCISUP+fcfuiH/Dq/4V/wDPx4u/8C6P+IvYn/n7IP8AiElD/n3H7oh/w6v+Ff8Az8eLv/Auj/iL2J/5+yD/AIhJQ/59x+6If8Or/hX/AM/Hi7/wLo/4i9if+fsg/wCISUP+fcfuiH/Dq/4V/wDPx4u/8C6P+IvYn/n7IP8AiElD/n3H7oh/w6v+Ff8Az8eLv/Auj/iL2J/5+yD/AIhJQ/59x+6If8Or/hX/AM/Hi7/wLo/4i9if+fsg/wCISUP+fcfuiH/Dq/4V/wDPx4u/8C6P+IvYn/n7IP8AiElD/n3H7oh/w6v+Ff8Az8eLv/AysKvi/ipYiMYzaTdnq+tlvzK/fWK+W5tHwkwyw0m6cb2T1t010Xr8l8z6G/Zr/Yu8Kfs9eMLrxX4Vn1sz6nZGyvf7Vu8nA5PHOT1wACfQNkivjuNeMamcYdOU1J6X15lv7rey/HfazR9DwbwpRyjEy5Y2UW3dLoui2X5+Xc/Qqvz0/SQoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKAOQ1Hw5pOpTLJPBljwSODnpnHXnjoBn16huyhjcwy+LWHrSUHuk9N9FfZ+ttOtjxMfRwOOrw9rShJ6XfKtfXbfdXXyWvLSHgfQzvzAAF9e/ODng/y9uK9D+28ySjKOIm5ve70fbrpZX/m/U6sVk2V/Vo2wsI2W9k79Nbd+n3NPeMn/CCaH/zyX/x2uP8AtTNP+gip/wCBS/8AkfJf8G3vaf2blv8Az4h90P8A5H+rdNOU/wCEE0P/AJ5L/wCO0f2pmn/QRU/8Cl/8j5L/AINveP7Ny3/nxD7of/I/1bppyn/CCaH/AM8l/wDHaP7UzT/oIqf+BS/+R8l/wbe8f2blv/PiH3Q/+R/q3TTlP+EE0P8A55L/AOO0f2pmn/QRU/8AApf/ACPkv+Db3j+zct/58Q+6H/yP9W6acp/wgmh/88l/8do/tTNP+gip/wCBS/8AkfJf8G3vH9m5b/z4h90P/kf6t005T/hBND/55L/47R/amaf9BFT/AMCl/wDI+S/4NveP7Ny3/nxD7of/ACP9W6acp/wgmh/88l/8do/tTNP+gip/4FL/AOR8l/wbe8f2blv/AD4h90P/AJH+rdNOU/4QTQ/+eS/+O0f2pmn/AEEVP/Apf/I+S/4NveP7Ny3/AJ8Q+6H/AMj/AFbppyn/AAgmh/8APJf/AB2j+1M0/wCgip/4FL/5HyX/AAbe8f2blv8Az4h90P8A5H+rdNOU/wCEE0P/AJ5L/wCO0f2pmn/QRU/8Cl/8j5L/AINveP7Ny3/nxD7of/I/1bppy9bZ2kNlEIYeg656/wCfz/UBfPO0t0AFABQB/9f+/igAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKACgAoAKAP/9k=) + + + + 以前的版本中普遍传言说它的性能不好，但是后续的版本性能得到了较大提高，经过系统压测还是能够支撑较大并发量的，经过压测三台Zookeeper能搞住20000tps。 + + 用zookeeper的优点大概有：高可用、公平锁、心跳保持锁。 diff --git "a/docs/soa/zookeeper/Zookeeper\345\256\236\346\210\230.md" b/docs/distribution/ZooKeeper/Zookeeper-Use.md similarity index 99% rename from "docs/soa/zookeeper/Zookeeper\345\256\236\346\210\230.md" rename to docs/distribution/ZooKeeper/Zookeeper-Use.md index e94f667434..7a8a9b0e93 100644 --- "a/docs/soa/zookeeper/Zookeeper\345\256\236\346\210\230.md" +++ b/docs/distribution/ZooKeeper/Zookeeper-Use.md @@ -1,3 +1,5 @@ +# Zookeeper 实战 + > 上篇大都是概念性东西，作为一名优秀的 Javaer，肯定要实战一番。加油，奥利给！ > > 文章收集在 GitHub [JavaEgg](https://github.com/Jstarfish/JavaEgg) 中，欢迎 star+指导，N线互联网开发必备兵器库 @@ -337,7 +339,3 @@ pc广告在上海 zk用作配置管理，有可投放的商品包状态变动会 - ZooKeeper的常用命令 ls create get delete set… - - - -![](../../../images/blog_end.png) \ No newline at end of file diff --git a/docs/distribution/message-queue/.DS_Store b/docs/distribution/message-queue/.DS_Store new file mode 100644 index 0000000000..6993dba05a Binary files /dev/null and b/docs/distribution/message-queue/.DS_Store differ diff --git a/docs/distribution/message-queue/Kafka/.DS_Store b/docs/distribution/message-queue/Kafka/.DS_Store new file mode 100644 index 0000000000..9c80a665bc Binary files /dev/null and b/docs/distribution/message-queue/Kafka/.DS_Store differ diff --git a/docs/distribution/message-queue/Kafka/Consumer Group Protocol.md b/docs/distribution/message-queue/Kafka/Consumer Group Protocol.md new file mode 100644 index 0000000000..6dbd1efa96 --- /dev/null +++ b/docs/distribution/message-queue/Kafka/Consumer Group Protocol.md @@ -0,0 +1,147 @@ +## Consumer Group Protocol + +Kafka separates storage from compute. Storage is handled by the brokers and compute is mainly handled by consumers or frameworks built on top of consumers (Kafka Streams, ksqlDB). Consumer groups play a key role in the effectiveness and scalability of Kafka consumers. + +### Kafka Consumer Group + +![kafka-consumer-group](https://images.ctfassets.net/gt6dp23g0g38/4YKWvDvHoX9Y4NzdPQZvo5/f1f3811e4e6098a45ba702fed62b6735/Kafka_Internals_064.png) + +To define a consumer group we just need to set the group.id in the consumer config. Once that is set, every new instance of that consumer will be added to the group. Then, when the consumer group subscribes to one or more topics, their partitions will be evenly distributed between the instances in the group. This allows for parallel processing of the data in those topics. + +The unit of parallelism is the partition. For a given consumer group, consumers can process more than one partition but a partition can only be processed by one consumer. If our group is subscribed to two topics and each one has two partitions then we can effectively use up to four consumers in the group. We could add a fifth but it would sit idle since partitions cannot be shared. + +The assignment of partitions to consumer group instances is dynamic. As consumers are added to the group, or when consumers fail or are removed from the group for some other reason, the workload will be rebalanced automatically. + +### Group Coordinator + +![group-coordinator](https://images.ctfassets.net/gt6dp23g0g38/6smQqDfuZgjNPFeB7iNHLb/d237de3390dcb0cada914c06f4142053/Kafka_Internals_065.png) + +The magic behind consumer groups is provided by the group coordinator. The group coordinator helps to distribute the data in the subscribed topics to the consumer group instances evenly and it keeps things balanced when group membership changes occur. The coordinator uses an internal Kafka topic to keep track of group metadata. + +In a typical Kafka cluster, there will be multiple group coordinators. This allows for multiple consumer groups to be managed efficiently. + +### Group Startup + +Let’s take a look at the steps involved in starting up a new consumer group. + +#### Step 1 – Find Group Coordinator + +![group-startup-find-group-coordinator](https://images.ctfassets.net/gt6dp23g0g38/4N3txDiCwLsYg502212Lx4/43a4f6797bb868f027cd87a6663fc84d/Kafka_Internals_066.png) + +When a consumer instance starts up it sends a FindCoordinator request that includes its group.id to any broker in the cluster. The broker will create a hash of the group.id and modulo that against the number of partitions in the internal __consumer_offsets topic. That determines the partition that all metadata events for this group will be written to. The broker that hosts the leader replica for that partition will take on the role of group coordinator for the new consumer group. The broker that received the FindCoordinator request will respond with the endpoint of the group coordinator. + +#### Step 2 – Members Join + +![group-startup-members-join](https://images.ctfassets.net/gt6dp23g0g38/3tDRpjViBVqd1UdRr7vT26/4ba46426533ebd6414eed5d4d7e43413/Kafka_Internals_067.png) + +Next, the consumers and the group coordinator begin a little logistical dance, starting with the consumers sending a JoinGroup request and passing their topic subscription information. The coordinator will choose one consumer, usually the first one to send the JoinGroup request, as the group leader. The coordinator will return a memberId to each consumer, but it will also return a list of all members and the subscription info to the group leader. The reason for this is so that the group leader can do the actual partition assignment using a configurable partition assignment strategy. + +#### Step 3 – Partitions Assigned + +![group-startup-partitions-assigned](https://images.ctfassets.net/gt6dp23g0g38/5AcaJ8KtM5YmmI9Ueomz25/2fcac2290d58d784f1522a41d8d48df2/Kafka_Internals_068.png) + +After the group leader receives the complete member list and subscription information, it will use its configured partitioner to assign the partitions in the subscription to the group members. With that done, the leader will send a SyncGroupRequest to the coordinator, passing in its memberId and the group assignments provided by its partitioner. The other consumers will make a similar request but will only pass their memberId. The coordinator will use the assignment information given to it by the group leader to return the actual assignments to each consumer. Now the consumers can begin their real work of consuming and processing data. + +### Range Partition Assignment Strategy + +![range-partition-assignment-strategies](https://images.ctfassets.net/gt6dp23g0g38/2I7BijMOpFeFxkdeyH5kFN/9f00009b876cf3339d6c646e53143cd8/Kafka_Internals_069.png) + +Now, let's look at some of the available assignment strategies. First up is the range assignment strategy. This strategy goes through each topic in the subscription and assigns each of the partitions to a consumer, starting at the first consumer. What this means is that the first partition of each topic will be assigned to the first consumer, the second partition of each topic will be assigned to the second consumer, and so on. If no single topic in the subscription has as many partitions as there are consumers, then some consumers will be idle. + +At first glance this might not seem like a very good strategy, but it has a very special purpose. When joining events from more than one topic the events need to be read by the same consumer. If events in two different topics are using the same key, they will be in the same partition of their respective topics, and with the range partitioner, they will be assigned to the same consumer. + +### Round Robin and Sticky Partition Assignment Strategies + +![round-robin-and-sticky-strategies](https://images.ctfassets.net/gt6dp23g0g38/4OL5b1FeAv3mSXM99a5IxA/2ea543db29d3d1a0b26cf57a312208ab/Kafka_Internals_070.png) + +Next, let’s look at the Round Robin strategy. With this strategy, all of the partitions of the subscription, regardless of topic, will be spread evenly across the available consumers. This results in fewer idle consumer instances and a higher degree of parallelism. + +A variant of Round Robin, called the Sticky Partition strategy, operates on the same principle but it makes a best effort at sticking to the previous assignment during a rebalance. This provides a faster, more efficient rebalance. + +### Tracking Partition Consumption + +![tracking-partition-consumption](https://images.ctfassets.net/gt6dp23g0g38/F1Nqrh5ElviKJKLuzO6gn/73a3f20420af1749bc74d2d7528cbf45/Kafka_Internals_071.png) + +In Kafka, keeping track of the progress of a consumer is relatively simple. A given partition is always assigned to a single consumer, and the events in that partition are always read by the consumer in offset order. So, the consumer only needs to keep track of the last offset it has consumed for each partition. To do this, the consumer will issue a CommitOffsetRequest to the group coordinator. The coordinator will then persist that information in its internal __consumer_offsets topic. + +### Determining Starting Offset to Consume + +![determining-starting-offset-to-consume](https://images.ctfassets.net/gt6dp23g0g38/aQyj65S4sBzKyCGHCSjtf/ec08539faaa40080361201e39ba5a761/Kafka_Internals_072.png) + +When a consumer group instance is restarted, it will send an OffsetFetchRequest to the group coordinator to retrieve the last committed offset for its assigned partition. Once it has the offset, it will resume the consumption from that point. If this consumer instance is starting for the very first time and there is no saved offset position for this consumer group, then the auto.offset.reset configuration will determine whether it begins consuming from the earliest offset or the latest. + +### Group Coordinator Failover + +![group-coordinator-failover](https://images.ctfassets.net/gt6dp23g0g38/3FRBXQAQvcGXxKezdXtOGK/9494607a1ea6c60a2aa3f527222fcb25/Kafka_Internals_073.png) + +The internal __consumer_offsets topic is replicated like any other Kafka topic. Also, recall that the group coordinator is the broker that hosts the leader replica of the __consumer_offsets partition assigned to this group. So if the group coordinator fails, a broker that is hosting one of the follower replicas of that partition will become the new group coordinator. Consumers will be notified of the new coordinator when they try to make a call to the old one, and then everything will continue as normal. + +### Consumer Group Rebalance Triggers + +![consumer-group-rebalance-triggers](https://images.ctfassets.net/gt6dp23g0g38/235g0BGQoTUrdzhrPVaJ6w/b11bae4c0ea05aeff38f8300c5386a97/Kafka_Internals_074.png) + +One of the key features of consumer groups is rebalancing. We’ll be discussing rebalances in more detail, but first let's consider some of the events that can trigger a rebalance: + +- An instance fails to send a heartbeat to the coordinator before the timeout and is removed from the group +- An instance has been added to the group +- Partitions have been added to a topic in the group’s subscription +- A group has a wildcard subscription and a new matching topic is created +- And, of course, initial group startup + +Next we’ll look at what happens when a rebalance occurs. + +### Consumer Group Rebalance Notification + +![consumer-group-rebalance-notification](https://images.ctfassets.net/gt6dp23g0g38/izCSKstOIQMZ8Blh7BCWg/8194eb9df3ff686cd62e1b1d7dda8c01/Kafka_Internals_075.png) + +The rebalance process begins with the coordinator notifying the consumer instances that a rebalance has begun. It does this by piggybacking on the HeartbeatResponse or the OffsetFetchResponse. Now the fun begins! + +### Stop-the-World Rebalance + +![stop-the-world-rebalance](https://images.ctfassets.net/gt6dp23g0g38/3AvvIpWKyXtRxH8g8Oev5j/d8255b0fa80975f2b0b0febe772c52f9/Kafka_Internals_076.png) + +The traditional rebalance process is rather involved. Once the consumers receive the rebalance notification from the coordinator, they will revoke their current partition assignments. If they have been maintaining any state associated with the data in their previously assigned partitions, they will also have to clean that up. Now they are basically like new consumers and will go through the same steps as a new consumer joining the group. + +They will send a JoinGroupRequest to the coordinator, followed by a SyncGroupRequest. The coordinator will respond accordingly, and the consumers will each have their new assignments. + +Any state that is required by the consumer would now have to be rebuilt from the data in the newly assigned partitions. This process, while effective, has some drawbacks. Let’s look at a couple of those now. + +### Stop-the-World Problem #1 – Rebuilding State + +![stop-the-world-problem-rebuilding-state](https://images.ctfassets.net/gt6dp23g0g38/75Mb2DDsnsQ1rzhovE93NF/f254f9bc114fc01f05b2c005ef5c2aff/Kafka_Internals_077.png) + +The first problem is the need to rebuild state. If a consumer application was maintaining state based on the events in the partition it had been assigned to, it may need to read all of the events in the partition to rebuild that state after the rebalance is complete. As you can see from our example, sometimes this work is being done even when it is not needed. If a consumer revokes its assignment to a particular partition and then is assigned that same partition during the rebalance, a significant amount of wasted processing may occur. + +### Stop-the-World Problem #2 – Paused Processing + +![stop-the-world-paused-processing](https://images.ctfassets.net/gt6dp23g0g38/a030ERMpe6aBI0q2rtRAM/44bd23b209e40d7f7543e83c6de3fea3/Kafka_Internals_078.png) + +The second problem is that we’re required to pause all processing while the rebalance is occurring, hence the name “Stop-the-world.” Since the partition assignments for all consumers are revoked at the beginning of the process, nothing can happen until the process completes and the partitions have been reassigned. In many cases, as in our example here, some consumers will keep some of the same partitions and could have, in theory, continued working with them while the rebalance was underway. + +Let’s see some of the improvements that have been made to deal with these problems. + +### Avoid Needless State Rebuild with StickyAssignor + +![avoid-needless-state-rebuild-stickyassignor](https://images.ctfassets.net/gt6dp23g0g38/184ENnaTGeHTXyToly3Oj5/2b939e740df21123c6cc1fa91bb80abc/Kafka_Internals_079.png) + +First, using the new StickyAssignor we can avoid unnecessary state rebuilding. The main difference with the StickyAssignor, is that the state cleanup is moved to a later step, after the reassignments are complete. That way if a consumer is reassigned the same partition it can just continue with its work and not clear or rebuild state. In our example, state would only need to be rebuilt for partition p2, which is assigned to the new consumer. + +### Avoid Pause with CooperativeStickyAssignor Step 1 + +![avoid-processing-pause-cooperativestickyassignor](https://images.ctfassets.net/gt6dp23g0g38/4hxCepzIy6fhVRFvg3LJjr/a09433afda0ce6cbaa560acf8e3daafa/Kafka_Internals_080.png) + +To solve the problem of paused processing, we introduced the CooperativeStickyAssignor. This assignor works in a two-step process. In the first step the determination is made as to which partition assignments need to be revoked. Those assignments are revoked at the end of the first rebalance step. The partitions that are not revoked can continue to be processed. + +### Avoid Pause with CooperativeStickyAssignor Step 2 + +![avoid-processing-pause-cooperativestickyassignor-2](https://images.ctfassets.net/gt6dp23g0g38/3rsYzgAy1qklrJzVCRQ74u/93f70ccd8442c10d8be32a56e5c64c45/Kafka_Internals_081.png) + +In the second rebalance step, the revoked partitions will be assigned. In our example, partition 2 was the only one revoked and it is assigned to the new consumer 3. In a more involved system, all of the consumers might have new partition assignments, but the fact remains that any partitions that did not need to move can continue to be processed without the world grinding to a halt. + +### Avoid Rebalance with Static Group Membership + +![avoid-rebalance-with-static-group-membership](https://images.ctfassets.net/gt6dp23g0g38/4nNPDs11L8OfiTtOfl7Ket/6b08a07f522090245224538c9f621c04/Kafka_Internals_082.png) + +As the saying goes, the fastest rebalance is the one that doesn’t happen. That’s the goal of static group membership. With static group membership each consumer instance is assigned a group.instance.id. Also, when a consumer instance leaves gracefully it will not send a LeaveGroup request to the coordinator, so no rebalance is started. When the same instance rejoins the group, the coordinator will recognize it and allow it to continue with its existing partition assignments. Again, no rebalance needed. + +Likewise, if a consumer instance fails but is restarted before its heartbeat interval has timed out, it will be able to continue with its existing assignments. \ No newline at end of file diff --git a/docs/distribution/message-queue/Kafka/Hello-Kafka.md b/docs/distribution/message-queue/Kafka/Hello-Kafka.md new file mode 100644 index 0000000000..8219dd68cc --- /dev/null +++ b/docs/distribution/message-queue/Kafka/Hello-Kafka.md @@ -0,0 +1,206 @@ +--- +title: Hello Kafka +date: 2022-02-15 +tags: + - Kafka +categories: Kafka +--- + +![](https://img.starfish.ink/mq/hello-kakfa-banner.png) + + + +## 1. Kafka概述 + +### 1.1 定义 + +Kafka 是由 Apache 软件基金会开发的一个开源流处理平台。 + +Kafka 是一个**分布式**的基于**发布/订阅模式的消息队列**（Message Queue），主要应用于大数据实时处理领域。 + + + +### 1.2 消息队列 + +#### 1.2.1 传统消息队列的应用场景 + +![](https://img.starfish.ink/mq/mq-scenarios.png) + +#### 1.2.2 为什么需要消息队列 + +1. **解耦**：允许你独立的扩展或修改两边的处理过程，只要确保它们遵守同样的接口约束。 +2. **冗余**：消息队列把数据进行持久化直到它们已经被完全处理，通过这一方式规避了数据丢失风险。许多消息队列所采用的"插入-获取-删除"范式中，在把一个消息从队列中删除之前，需要你的处理系统明确的指出该消息已经被处理完毕，从而确保你的数据被安全的保存直到你使用完毕。 +3. **扩展性**： 因为消息队列解耦了你的处理过程，所以增大消息入队和处理的频率是很容易的，只要另外增加处理过程即可。 +4. **灵活性 & 峰值处理能力**： 在访问量剧增的情况下，应用仍然需要继续发挥作用，但是这样的突发流量并不常见。 如果为以能处理这类峰值访问为标准来投入资源随时待命无疑是巨大的浪费。使用消息队列能够使关键组件顶住突发的访问压力，而不会因为突发的超负荷的请求而完全崩溃。 +5. **可恢复性**：系统的一部分组件失效时，不会影响到整个系统。消息队列降低了进程间的耦合度，所以即使一个处理消息的进程挂掉，加入队列中的消息仍然可以在系统恢复后被处理。 +6. **顺序保证**：在大多使用场景下，数据处理的顺序都很重要。大部分消息队列本来就是排序的，并且能保证数据会按照特定的顺序来处理。（Kafka 保证一个 Partition 内的消息的有序性） +7. **缓冲**：有助于控制和优化数据流经过系统的速度， 解决生产消息和消费消息的处理速度不一致的情况。 +8. **异步通信**：很多时候，用户不想也不需要立即处理消息。消息队列提供了异步处理机制，允许用户把一个消息放入队列，但并不立即处理它。想向队列中放入多少消息就放多少，然后在需要的时候再去处理它们。 + + + +#### 1.2.3 消息队列的两种模式 + +- **点对点模式**（一对一，消费者主动拉取数据，收到后消息清除） + + 消息生产者生产消息发送到 Queue 中，然后消费者从 Queue 中取出并且消费消息。 消息被消费以后，queue 中不再有存储，所以消息消费者不可能消费到已经被消费的消息。 Queue 支持存在多个消费者，但是对一个消息而言，只会有一个消费者可以消费。 + + ![图片：mrbird.cc](https://img.starfish.ink/mq/mq-point2point.jpg) + +- **发布/订阅模式**（一对多，数据生产后，推送给所有订阅者） + + 消息生产者（发布）将消息发布到 topic 中，同时有多个消息消费者（订阅）消费该消息。和点对点方式不同，发布到 topic 的消息会被所有订阅者消费。 + + ![图片：mrbird.cc](https://img.starfish.ink/mq/mq-one2many.jpg) + + + +### 1.3 Kafka 基础架构图 + +![图片：mrbird.cc](https://img.starfish.ink/mq/kafka-structure.png) + +- Producer ：消息生产者，就是向 kafka broker 发消息的客户端； +- Consumer ：消息消费者，向 kafka broker 取消息的客户端； +- Consumer Group （CG）：消费者组，由多个 consumer 组成。**消费者组内每个消费者负责消费不同分区的数据，一个分区只能由一个组内消费者消费；消费者组之间互不影响**。所有的消费者都属于某个消费者组，即消费者组是逻辑上的一个订阅者。 +- Broker ：一台 kafka 服务器就是一个 broker（虽然多个 Broker 进程能够运行在同一台机器上，但更常见的做法是将不同的 Broker 分散运行在不同的机器上）。一个集群由多个 broker 组成。一个 broker 可以容纳多个 topic； +- Topic ：可以理解为一个队列，Kafka 的消息通过 Topics(主题) 进行分类，生产者和消费者面向的都是一个 topic； +- Partition：为了实现扩展性，一个非常大的 topic 可以分布到多个 broker（即服务器）上， 一个 topic 可以分为多个 partition，每个 partition 是一个有序的队列； partition 中的每条消息都会被分配一个有序的 id（ offset）。 kafka 只保证按一个 partition 中的顺序将消息发给 consumer，不保证一个 topic 的整体（多个 partition 间）的顺序； +- Replica：副本，为保证集群中的某个节点发生故障时，该节点上的 partition 数据不丢失，且 kafka 仍然能够继续工作，kafka 提供了副本机制，一个 topic 的每个分区都有若干个副本， 一个 leader 和若干个 follower； +- leader：每个分区多个副本的“主”，**生产者发送数据的对象，以及消费者消费数据的对象都是 leader**； +- follower：每个分区多个副本中的“从”，实时从 leader 中同步数据，保持和 leader 数据的同步。leader 发生故障时，某个 follower 会成为新的 follower； +- offset： kafka 的存储文件都是按照 `offset.kafka` 来命名，用 offset 做名字的好处是方便查找。例如你想找位于 2049 的位置，只要找到 2048.kafka 的文件即可。当然 the first offset 就是 00000000000.kafka。 + +------ + + + +## 2. Hello Kafka + +![overview-of-kafka-architecture](https://images.ctfassets.net/gt6dp23g0g38/4DA2zHan28tYNAV2c9Vd98/b9fca38c23e2b2d16a4c4de04ea6dd3f/Kafka_Internals_004.png) + +### 2.1 动起手来 + +[Quickstart]( ) + +[中文版入门指南]( ) + +### 2.2 基本概念(官方介绍翻译) + +Kafka 是一个分布式的流处理平台。是支持分区的（partition）、多副本的（replica），基于 ZooKeeper 协调的分布式消息系统，它的最大的特性就是可以实时的处理大量数据以满足各种需求场景：比如基于 hadoop 的批处理系统、低延迟的实时系统、storm/Spark 流式处理引擎，web/nginx 日志、访问日志，消息服务等等 + +#### 有三个关键能力 + +- 它可以让你发布和订阅记录流。在这方面，它类似于一个消息队列或企业消息系统 +- 它可以让你持久化收到的记录流，从而具有容错能力 +- 它可以让你处理收到的记录流 + +#### 应用于两大类应用 + +- 构建实时的流数据管道，可靠地获取系统和应用程序之间的数据。 +- 构建实时流的应用程序，对数据流进行转换或反应。 + +想要了解 Kafka 如何具有这些能力，首先，明确几个概念： + +- Kafka 作为一个集群运行在一个或多个服务器上 +- Kafka 集群存储的消息是以主题（topics）为类别记录的 +- 每个消息记录包含一个键，一个值和时间戳 + +#### Kafka有五个核心API： + +- **Producer API** 允许应用程序发布记录流至一个或多个 Kafka 的话题(Topics) + +- **Consumer API** 允许应用程序订阅一个或多个主题，并处理这些主题接收到的记录流 + +- **Streams API** 允许应用程序充当流处理器（stream processor），从一个或多个主题获取输入流，并生产一个输出流至一个或多个的主题，能够有效地变换输入流为输出流 + +- **Connector API** 允许构建和运行可重用的生产者或消费者，能够把 Kafka 主题连接到现有的应用程序或数据系统。例如，一个连接到关系数据库的连接器(connector)可能会获取每个表的变化 + +- **Admin API** 允许管理和检查主题、brokes 和其他 Kafka 对象。（这个是新版本才有的） + +Kafka 的客户端和服务器之间的通信是靠一个简单的，高性能的，与语言无关的 TCP 协议完成的。这个协议有不同的版本，并保持向后兼容旧版本。Kafka 不光提供了一个 Java 客户端，还有许多语言版本的客户端。 + +#### 主题和日志 + +主题是同一类别的消息记录（record）的集合。Kafka 的主题支持多用户订阅，也就是说，一个主题可以有零个，一个或多个消费者订阅写入的数据。对于每个主题，Kafka 集群都会维护一个分区日志，如下所示： + +![](https://img.starfish.ink/mq/log_anatomy.png) + +**每个分区是一个有序的，不可变的消息序列**，新的消息不断追加到 partition 的末尾。在每个 partition 中，每条消息都会被分配一个顺序的唯一标识，这个标识被称为 **offset**，即偏移量。**kafka 不能保证全局有序，只能保证分区内有序** 。 + +Kafka 集群保留所有发布的记录，不管这个记录有没有被消费过，**Kafka 提供可配置的保留策略去删除旧数据**(还有一种策略根据分区大小删除数据)。例如，如果将保留策略设置为两天，在数据发布后两天，它可用于消费，之后它将被丢弃以腾出空间。Kafka 的性能跟存储的数据量的大小无关（会持久化到硬盘）， 所以将数据存储很长一段时间是没有问题的。 + +![](https://img.starfish.ink/mq/log_consumer.png) + +事实上，在单个消费者层面上，每个消费者保存的唯一的元数据就是它所消费的数据日志文件的偏移量。偏移量是由消费者来控制的，通常情况下，消费者会在读取记录时线性的提高其偏移量。不过由于偏移量是由消费者控制，所以消费者可以将偏移量设置到任何位置，比如设置到以前的位置对数据进行重复消费，或者设置到最新位置来跳过一些数据。 + +#### 分布式 + +日志的分区会跨服务器的分布在 Kafka 集群中，每个服务器会共享分区进行数据请求的处理。**每个分区可以配置一定数量的副本分区提供容错能力**。 + +**每个分区都有一个服务器充当“leader”和零个或多个服务器充当“followers”**。 leader 处理所有的读取和写入分区的请求，而 followers 被动的从领导者拷贝数据。如果 leader 失败了，followers 之一将自动成为新的领导者。每个服务器可能充当一些分区的 leader 和其他分区的 follower，所以 Kafka 集群内的负载会比较均衡。 + +#### 生产者 + +生产者发布数据到他们所选择的主题。生产者负责选择把记录分配到主题中的哪个分区。这可以使用轮询算法( round-robin)进行简单地平衡负载，也可以根据一些更复杂的语义分区算法（比如基于记录一些键值）来完成。 + +#### 消费者 + +消费者以消费群（**consumer group** ）的名称来标识自己，每个发布到主题的消息都会发送给订阅了这个主题的消费群里面的一个消费者的一个实例。消费者的实例可以在单独的进程或单独的机器上。 + +如果所有的消费者实例都属于相同的消费群，那么记录将有效地被均衡到每个消费者实例。 + +如果所有的消费者实例有不同的消费群，那么每个消息将被广播到所有的消费者进程。 + +**这是 kafka 用来实现一个 topic 消息的广播（发给所有的 consumer） 和单播（发给任意一个 consumer）的手段**。一个 topic 可以有多个 CG。 topic 的消息会复制 （不是真的复制，是概念上的）到所有的 CG，但每个 partion 只会把消息发给该 CG 中的一 个 consumer。如果需要实现广播，只要每个 consumer 有一个独立的 CG 就可以了。要实现单播只要所有的 consumer 在同一个 CG。用 CG 还可以将 consumer 进行自由的分组而不需要多次发送消息到不同的 topic； + +![](https://img.starfish.ink/mq/consumer-groups.png) + +**举个栗子：** + +如上图所示，一个两个节点的 Kafka 集群上拥有一个四个 partition（P0-P3）的 topic。有两个消费者组都在消费这个 topic 中的数据，消费者组 A 有两个消费者实例，消费者组 B 有四个消费者实例。 + +从图中我们可以看到，在同一个消费者组中，每个消费者实例可以消费多个分区，但是每个分区最多只能被消费者组中的一个实例消费。也就是说，如果有一个 4 个分区的主题，那么消费者组中最多只能有 4 个消费者实例去消费，多出来的都不会被分配到分区。其实这也很好理解，如果允许两个消费者实例同时消费同一个分区，那么就无法记录这个分区被这个消费者组消费的 offset 了。如果在消费者组中动态的上线或下线消费者，那么 Kafka 集群会自动调整分区与消费者实例间的对应关系。 + +**Kafka消费群的实现方式是通过分割日志的分区，分给每个 Consumer 实例，使每个实例在任何时间点的都可以“公平分享”独占的分区**。维持消费群中的成员关系的这个过程是通过 Kafka 动态协议处理。如果新的实例加入该组，他将接管该组的其他成员的一些分区；如果一个实例死亡，其分区将被分配到剩余的实例。 + +Kafka 只保证一个分区内的消息有序，不能保证一个主题的不同分区之间的消息有序。分区的消息有序与依靠主键进行数据分区的能力相结合足以满足大多数应用的要求。但是，如果你想要保证所有的消息都绝对有序可以只为一个主题分配一个分区，虽然这将意味着每个消费群同时只能有一个消费进程在消费。 + +#### 保证 + +Kafka 提供了以下一些高级别的保证： + +- 由生产者发送到一个特定的主题分区的消息将被以他们被发送的顺序来追加。也就是说，如果一个消息 M1 和消息 M2 都来自同一个生产者，M1 先发，那么 M1 将有一个低于 M2 的偏移，会更早在日志中出现。 +- 消费者看到的记录排序就是记录被存储在日志中的顺序。 +- 对于副本因子 N 的主题，我们将承受最多 N-1 次服务器故障切换而不会损失任何的已经保存的记录。 + + + +### 2.3 Kafka的使用场景 + +#### 消息 + +Kafka 被当作传统消息中间件的替代品。消息中间件的使用原因有多种（从数据生产者解耦处理，缓存未处理的消息等）。与大多数消息系统相比，Kafka 具有更好的吞吐量，内置的分区，多副本和容错功能，这使其成为大规模消息处理应用程序的良好解决方案。 + +#### 网站行为跟踪 + +Kafka 的初衷就是能够将用户行为跟踪管道重构为一组实时发布-订阅数据源。这意味着网站活动（页面浏览量，搜索或其他用户行为）将被发布到中心主题，这些中心主题是每个用户行为类型对应一个主题的。这些数据源可被订阅者获取并用于一系列的场景，包括实时处理，实时监控和加载到 Hadoop 或离线数据仓库系统中进行离线处理和报告。用户行为跟踪通常会产生巨大的数据量，因为用户每个页面的浏览都会生成许多行为活动消息。 + +#### 测量 + +Kafka 通常用于监测数据的处理。这涉及从分布式应用程序聚集统计数据，生产出集中的运行数据源 feeds（以便订阅）。 + +#### 日志聚合 + +许多人用 Kafka 作为日志聚合解决方案的替代品。日志聚合通常从服务器收集物理日志文件，并将它们集中放置（可能是文件服务器或HDFS），以便后续处理。kafka 抽象出文件的细节，并将日志或事件数据作为消息流清晰地抽象出来。这为低时延的处理提供支持，而且更容易支持多个数据源和分布式的数据消费。相比集中式的日志处理系统（如 Scribe 或 Flume），Kafka 性能同样出色，而且因为副本备份提供了更强的可靠性保证和更低的端到端延迟。 + +#### 流处理 + +Kafka 的流数据管道在处理数据的时候包含多个阶段，其中原始输入数据从 Kafka 主题被消费然后汇总，加工，或转化成新主题用于进一步的消费或后续处理。例如，用于推荐新闻文章的数据流处理管道可能从 RSS 源抓取文章内容，并将其发布到“文章”主题; 进一步的处理可能是标准化或删除重复数据，然后发布处理过的文章内容到一个新的主题， 最后的处理阶段可能会尝试推荐这个内容给用户。这种处理管道根据各个主题创建实时数据流图。从版本 0.10.0.0 开始，Apache Kafka 加入了轻量级的但功能强大的流处理库 **Kafka Streams**，Kafka Streams 支持如上所述的数据处理。除了Kafka Streams，可以选择的开源流处理工具包括 `Apache Storm and Apache Samza`。 + +#### 事件源 + +事件源是一种应用程序设计风格，是按照时间顺序记录的状态变化的序列。Kafka 的非常强大的存储日志数据的能力使它成为构建这种应用程序的极好的后端选择。 + +#### 提交日志 + +Kafka 可以为分布式系统提供一种外部提交日志(commit-log)服务。日志有助于节点之间复制数据，并作为一种数据重新同步机制用来恢复故障节点的数据。Kafka 的 log compaction 功能有助于支持这种用法。Kafka 在这种用法中类似于 Apache BookKeeper 项目。 diff --git "a/docs/distribution/message-queue/Kafka/Kafka \345\270\270\350\247\201\351\227\256\351\242\230.md" "b/docs/distribution/message-queue/Kafka/Kafka \345\270\270\350\247\201\351\227\256\351\242\230.md" new file mode 100644 index 0000000000..cc06254c08 --- /dev/null +++ "b/docs/distribution/message-queue/Kafka/Kafka \345\270\270\350\247\201\351\227\256\351\242\230.md" @@ -0,0 +1,146 @@ +> 面试时，对 MQ 知识的考察离不开这几个问题，如何确保消息 100% 不丢失？有顺序？不重复？消息积压怎么办？ + +## Kafka 消息丢失问题 + +> 那面对“在使用 MQ 消息队列时，如何确保消息不丢失”这个问题时，你要怎么回答呢？首先，你要分析其中有几个考点，比如： +> +> 如何知道有消息丢失？ +> +> 哪些环节可能丢消息？ +> +> 如何确保消息不丢失？ + +**Kafka 只对“已提交”的消息（committed message）做有限度的持久化保证。** + +一条消息从生产到消费完成这个过程，可以划分三个阶段 + +![](https://static001.geekbang.org/resource/image/81/05/81a01f5218614efea2838b0808709205.jpg) + +### 生产者丢数据 + +可能会有哪些因素导致消息没有发送成功呢？其实原因有很多，例如网络抖动，导致消息压根就没有发送到 Broker 端；或者消息本身不合格导致 Broker 拒绝接收（比如消息太大了，超过了 Broker 的承受能力）等 + +解决此问题的方法非常简单：**Producer 永远要使用带有回调通知的发送 API，也就是说不要使用 producer.send(msg)，而要使用 producer.send(msg, callback)**。不要小瞧这里的 callback（回调），它能准确地告诉你消息是否真的提交成功了。一旦出现消息提交失败的情况，你就可以有针对性地进行处理。 + + + +### Broker 丢数据 + +在存储阶段正常情况下，只要 Broker 在正常运行，就不会出现丢失消息的问题，但是如果 Broker 出现了故障，比如进程死掉了或者服务器宕机了，还是可能会丢失消息的。 + +所以 Broker 会做副本，保证一条消息至少同步两个节点再返回 ack + +### 消费者丢数据 + +Consumer 端丢失数据主要体现在 Consumer 端要消费的消息不见了。Consumer 程序有个“位移”的概念，表示的是这个 Consumer 当前消费到的 Topic 分区的位置。下面这张图来自于官网，它清晰地展示了 Consumer 端的位移数据。 + +![](https://static001.geekbang.org/resource/image/0c/37/0c97bed3b6350d73a9403d9448290d37.png) + +比如对于 Consumer A 而言，它当前的位移值就是 9；Consumer B 的位移值是 11。 + +这里的“位移”类似于我们看书时使用的书签，它会标记我们当前阅读了多少页，下次翻书的时候我们能直接跳到书签页继续阅读。 + +正确使用书签有两个步骤：第一步是读书，第二步是更新书签页。如果这两步的顺序颠倒了，就可能出现这样的场景：当前的书签页是第 90 页，我先将书签放到第 100 页上，之后开始读书。当阅读到第 95 页时，我临时有事中止了阅读。那么问题来了，当我下次直接跳到书签页阅读时，我就丢失了第 96～99 页的内容，即这些消息就丢失了。 + +同理，Kafka 中 Consumer 端的消息丢失就是这么一回事。要对抗这种消息丢失，办法很简单：**维持先消费消息（阅读），再更新位移（书签）的顺序**即可。这样就能最大限度地保证消息不丢失。 + +当然，这种处理方式可能带来的问题是消息的重复处理，类似于同一页书被读了很多遍，但这不属于消息丢失的情形。在专栏后面的内容中，我会跟你分享如何应对重复消费的问题。 + +**如果是多线程异步处理消费消息，Consumer 程序不要开启自动提交位移，而是要应用程序手动提交位移**。 + + + +### 最佳实践 + +1. 不要使用 producer.send(msg)，而要使用 producer.send(msg, callback)。记住，一定要使用带有回调通知的 send 方法。 +2. 设置 acks = all。acks 是 Producer 的一个参数，代表了你对“已提交”消息的定义。如果设置成 all，则表明所有副本 Broker 都要接收到消息，该消息才算是“已提交”。这是最高等级的“已提交”定义。 +3. 设置 retries 为一个较大的值。这里的 retries 同样是 Producer 的参数，对应前面提到的 Producer 自动重试。当出现网络的瞬时抖动时，消息发送可能会失败，此时配置了 retries > 0 的 Producer 能够自动重试消息发送，避免消息丢失。 +4. 设置 unclean.leader.election.enable = false。这是 Broker 端的参数，它控制的是哪些 Broker 有资格竞选分区的 Leader。如果一个 Broker 落后原先的 Leader 太多，那么它一旦成为新的 Leader，必然会造成消息的丢失。故一般都要将该参数设置成 false，即不允许这种情况的发生。 +5. 设置 replication.factor >= 3。这也是 Broker 端的参数。其实这里想表述的是，最好将消息多保存几份，毕竟目前防止消息丢失的主要机制就是冗余。 +6. 设置 min.insync.replicas > 1。这依然是 Broker 端参数，控制的是消息至少要被写入到多少个副本才算是“已提交”。设置成大于 1 可以提升消息持久性。在实际环境中千万不要使用默认值 1。 +7. 确保 replication.factor > min.insync.replicas。如果两者相等，那么只要有一个副本挂机，整个分区就无法正常工作了。我们不仅要改善消息的持久性，防止数据丢失，还要在不降低可用性的基础上完成。推荐设置成 replication.factor = min.insync.replicas + 1。 +8. 确保消息消费完成再提交。Consumer 端有个参数 enable.auto.commit，最好把它设置成 false，并采用手动提交位移的方式。就像前面说的，这对于单 Consumer 多线程处理的场景而言是至关重要的。 + + + +## Kafka 消息重复问题 + +在消息消费的过程中，如果出现失败的情况，通过补偿的机制发送方会执行重试，重试的过程就有可能产生重复的消息，那么如何解决这个问题？ + +这个问题其实可以换一种说法，就是如何解决消费端幂等性问题（幂等性，就是一条命令，任意多次执行所产生的影响均与一次执行的影响相同），只要消费端具备了幂等性，那么重复消费消息的问题也就解决了。 + +常用手段： + +**1. 利用数据库的唯一约束实现幂等** + +> 例如我们刚刚提到的那个不具备幂等特性的转账的例子：将账户 X 的余额加 100 元。在这个例子中，我们可以通过改造业务逻辑，让它具备幂等性。 +> +> 首先，我们可以限定，对于每个转账单每个账户只可以执行一次变更操作，在分布式系统中，这个限制实现的方法非常多，最简单的是我们在数据库中建一张转账流水表，这个表有三个字段：转账单 ID、账户 ID 和变更金额，然后给转账单 ID 和账户 ID 这两个字段联合起来创建一个唯一约束，这样对于相同的转账单 ID 和账户 ID，表里至多只能存在一条记录。 +> +> 这样，我们消费消息的逻辑可以变为：“在转账流水表中增加一条转账记录，然后再根据转账记录，异步操作更新用户余额即可。”在转账流水表增加一条转账记录这个操作中，由于我们在这个表中预先定义了“账户 ID 转账单 ID”的唯一约束，对于同一个转账单同一个账户只能插入一条记录，后续重复的插入操作都会失败，这样就实现了一个幂等的操作。我们只要写一个 SQL，正确地实现它就可以了。 +> +> 基于这个思路，不光是可以使用关系型数据库，只要是支持类似“INSERT IF NOT EXIST”语义的存储类系统都可以用于实现幂等，比如，你可以用 Redis 的 SETNX 命令来替代数据库中的唯一约束，来实现幂等消费。 + +**2. 全局唯一 ID** + +适用范围最广的实现幂等性方法：记录并检查操作，也称为“Token 机制或者 GUID（全局唯一 ID）机制”，实现的思路特别简单：在执行数据更新操作之前，先检查一下是否执行过这个更新操作。 + +具体的实现方法是，在发送消息时，给每条消息指定一个全局唯一的 ID，消费时，先根据这个 ID 检查这条消息是否有被消费过，如果没有消费过，才更新数据，然后将消费状态置为已消费。 + +> 原理和实现是不是很简单？其实一点儿都不简单，在分布式系统中，这个方法其实是非常难实现的。首先，给每个消息指定一个全局唯一的 ID 就是一件不那么简单的事儿，方法有很多，但都不太好同时满足简单、高可用和高性能，或多或少都要有些牺牲。更加麻烦的是，在“检查消费状态，然后更新数据并且设置消费状态”中，三个操作必须作为一组操作保证原子性，才能真正实现幂等，否则就会出现 Bug。 +> +> 比如说，对于同一条消息：“全局 ID 为 8，操作为：给 ID 为 666 账户增加 100 元”，有可能出现这样的情况： +> +> - t0 时刻：Consumer A 收到条消息，检查消息执行状态，发现消息未处理过，开始执行“账户增加 100 元”； +> - t1 时刻：Consumer B 收到条消息，检查消息执行状态，发现消息未处理过，因为这个时刻，Consumer A 还未来得及更新消息执行状态。 +> +> 这样就会导致账户被错误地增加了两次 100 元，这是一个在分布式系统中非常容易犯的错误，一定要引以为戒。 +> +> 对于这个问题，当然我们可以用事务来实现，也可以用锁来实现，但是在分布式系统中，无论是分布式事务还是分布式锁都是比较难解决问题。 +> +> 这点不是本文的讨论范围 + + + + + +最简单的实现方案，就是在数据库中建一张消息日志表， 这个表有两个字段：消息 ID 和消息执行状态。 + +因为我们每次都会在插入之前检查是否消息已存在，所以就不会出现一条消息被执行多次的情况，这样就实现了一个幂等的操作。当然，基于这个思路，不仅可以使用关系型数据库，也可以通过 Redis 来代替数据库实现唯一约束的方案。 + +在这里我多说一句，想要解决“消息丢失”和“消息重复消费”的问题，有一个前提条件就是要实现一个全局唯一 ID 生成的技术方案。这也是面试官喜欢考察的问题，你也要掌握。 + +在分布式系统中，全局唯一 ID 生成的实现方法有数据库自增主键、UUID、Redis，Twitter-Snowflake 算法，我总结了几种方案的特点，你可以参考下。 + +![](https://s0.lgstatic.com/i/image/M00/8F/79/Ciqc1GAIDXuAZ2iUAAIGj0vJThg862.png) + +## Kafka 消息保序问题 + +从业务上把需要有序的打到同一个partition，大多数情况只需要业务上保证有序就可以，不用全局有序 + + + + + +## Kafka 消息积压问题 + +如果出现积压，那一定是性能问题，想要解决消息从生产到消费上的性能问题，就首先要知道哪些环节可能出现消息积压，然后在考虑如何解决。 + +因为消息发送之后才会出现积压的问题，所以和消息生产端没有关系，又因为绝大部分的消息队列单节点都能达到每秒钟几万的处理能力，相对于业务逻辑来说，性能不会出现在中间件的消息存储上面。毫无疑问，出问题的肯定是消息消费阶段，那么从消费端入手，如何回答呢？ + +#### 水平扩容 + +消费端的性能优化除了优化消费业务逻辑以外，也可以通过水平扩容，增加消费端的并发数来提升总体的消费性能。特别需要注意的一点是，**在扩容 Consumer 的实例数量的同时，必须同步扩容主题中的分区（也叫队列）数量，确保 Consumer 的实例数和分区数量是相等的。**如果 Consumer 的实例数量超过分区数量，这样的扩容实际上是没有效果的。原因我们之前讲过，因为对于消费者来说，在每个分区上实际上只能支持单线程消费。 + + + + + +如果是线上突发问题，要临时扩容，增加消费端的数量，与此同时，降级一些非核心的业务。通过扩容和降级承担流量，这是为了表明你对应急问题的处理能力。 + +其次，才是排查解决异常问题，如通过监控，日志等手段分析是否消费端的业务逻辑代码出现了问题，优化消费端的业务处理逻辑。 + +最后，如果是消费端的处理能力不足，可以通过水平扩容来提供消费端的并发处理能力，但这里有一个考点需要特别注意， 那就是在扩容消费者的实例数的同时，必须同步扩容主题 Topic 的分区数量，确保消费者的实例数和分区数相等。如果消费者的实例数超过了分区数，由于分区是单线程消费，所以这样的扩容就没有效果。 + +比如在 Kafka 中，一个 Topic 可以配置多个 Partition（分区），数据会被写入到多个分区中，但在消费的时候，Kafka 约定一个分区只能被一个消费者消费，Topic 的分区数量决定了消费的能力，所以，可以通过增加分区来提高消费者的处理能力。 + diff --git a/docs/message-queue/Kafka/Kafka-API.md b/docs/distribution/message-queue/Kafka/Kafka-API.md similarity index 90% rename from docs/message-queue/Kafka/Kafka-API.md rename to docs/distribution/message-queue/Kafka/Kafka-API.md index a926bb8516..9664098c37 100644 --- a/docs/message-queue/Kafka/Kafka-API.md +++ b/docs/distribution/message-queue/Kafka/Kafka-API.md @@ -1,8 +1,136 @@ + + +### 1. 生产者配置 + +在开始生产数据之前，需要配置生产者的一些基本属性。这些属性定义了生产者的行为，例如连接的 Kafka 集群地址、序列化器、分区策略等。 + +```java +Properties props = new Properties(); +props.put("bootstrap.servers", "localhost:9092"); +props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); +props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer"); +props.put("acks", "all"); // 配置确认机制 +props.put("retries", 0); // 配置重试次数 +props.put("batch.size", 16384); // 配置批量大小 +props.put("linger.ms", 1); // 配置延迟时间 +props.put("buffer.memory", 33554432); // 配置内存缓冲区大小 + +KafkaProducer producer = new KafkaProducer<>(props); +``` + +### 2. 发送数据 + +生产者创建并发送消息到指定的主题（Topic）。发送消息时，可以指定消息的键和值，键用于确定消息将被发送到哪个分区（Partition）。 + +```java +producer.send(new ProducerRecord<>("my-topic", "key", "value")); +``` + +### 3. 选择分区 + +Kafka 使用分区来提高吞吐量和并行处理能力。每个主题可以分为多个分区，消息被均匀地分布在这些分区中。生产者在发送消息时，会根据配置的分区策略选择目标分区。 + +#### 分区策略 + +- **无键分区策略**：如果消息没有指定键，Kafka 会使用轮询（Round-Robin）策略将消息分布到各个分区。 +- **有键分区策略**：如果消息指定了键，Kafka 会使用键的哈希值来确定目标分区。相同键的消息会被发送到相同的分区。 + +```java +producer.send(new ProducerRecord<>("my-topic", "key", "value")); +``` + +### 4. 消息序列化 + +在将消息发送到 Kafka 之前，生产者需要将消息的键和值序列化为字节数组。Kafka 提供了多种内置的序列化器，也可以自定义序列化器。 + +```java +props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); +props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer"); +``` + +### 5. 批量发送和缓冲 + +为了提高性能，Kafka 生产者会将多条消息批量发送到 Kafka Broker。生产者会在内存中维护一个缓冲区（Buffer），在满足以下任一条件时，将缓冲区中的消息批量发送： + +- 缓冲区满（达到 `batch.size` 配置的大小） +- 等待时间超过 `linger.ms` 配置的时间 + +### 6. 发送确认 + +Kafka 提供了多种确认机制（Acks）来确保消息成功发送到 Kafka Broker： + +- **acks=0**：生产者不等待任何确认，即“尽力而为”模式。 +- **acks=1**：生产者等待主副本（Leader）写入成功后返回确认。 +- **acks=all**：生产者等待所有副本（ISR）写入成功后返回确认，确保最高的数据可靠性。 + +```java +props.put("acks", "all"); +``` + +### 7. 错误处理和重试 + +在发送消息过程中，如果遇到临时性错误（如网络问题），生产者可以重试发送消息。重试次数由 `retries` 配置参数决定。 + +```java +props.put("retries", 3); +``` + +### 8. 消息发送过程示例 + +以下是一个完整的生产者示例，展示了从配置到发送消息的全过程： + +```java +import org.apache.kafka.clients.producer.KafkaProducer; +import org.apache.kafka.clients.producer.ProducerRecord; + +import java.util.Properties; + +public class SimpleProducer { + public static void main(String[] args) { + Properties props = new Properties(); + props.put("bootstrap.servers", "localhost:9092"); + props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); + props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer"); + props.put("acks", "all"); + props.put("retries", 3); + props.put("batch.size", 16384); + props.put("linger.ms", 1); + props.put("buffer.memory", 33554432); + + KafkaProducer producer = new KafkaProducer<>(props); + + for (int i = 0; i < 100; i++) { + producer.send(new ProducerRecord<>("my-topic", Integer.toString(i), "Message-" + i)); + } + + producer.close(); + } +} +``` + +### 9. Kafka 生产者工作流程总结 + +1. **初始化和配置**：创建生产者实例并配置相关参数。 +2. **发送消息**：生产者创建并发送消息到 Kafka。 +3. **选择分区**：根据分区策略确定目标分区。 +4. **消息序列化**：将消息的键和值序列化为字节数组。 +5. **批量发送和缓冲**：生产者在内存中缓冲消息，并批量发送到 Kafka Broker。 +6. **发送确认**：根据配置的确认机制，确保消息成功发送。 +7. **错误处理和重试**：在遇到临时性错误时，生产者重试发送消息。 + +通过这些步骤，Kafka 生产者确保消息高效、可靠地发送到 Kafka 集群中，并持久化到磁盘中。这些机制共同保证了 Kafka 的高性能和高吞吐量。 + + + + + + + ## 5. Kafka API（Java中的Kafka使用） ### 5.1 启动zk和kafka集群 -![img](H:/Technical-Learning/docs/_images/message-queue/Kafka/kafka-start.png) + ### 5.2 导入 pom 依赖 @@ -134,6 +262,16 @@ producer.send（record).get() +> ### Kafka 生产者工作流程总结 +> +> 1. **初始化和配置**：创建生产者实例并配置相关参数。 +> 2. **发送消息**：生产者创建并发送消息到 Kafka。 +> 3. **选择分区**：根据分区策略确定目标分区。 +> 4. **消息序列化**：将消息的键和值序列化为字节数组。 +> 5. **批量发送和缓冲**：生产者在内存中缓冲消息，并批量发送到 Kafka Broker。 +> 6. **发送确认**：根据配置的确认机制，确保消息成功发送。 +> 7. **错误处理和重试**：在遇到临时性错误时，生产者重试发送消息。 + ### 5.4 创建消费者 Consumer 消费数据时的可靠性是很容易保证的，因为数据在 Kafka 中是持久化的，故 不用担心数据丢失问题。 由于 consumer 在消费过程中可能会出现断电宕机等故障，consumer 恢复后，需要从故 障前的位置的继续消费，所以 consumer 需要实时记录自己消费到了哪个 offset，以便故障恢 复后继续消费。 所以 offset 的维护是 Consumer 消费数据是必须考虑的问题。 diff --git a/docs/distribution/message-queue/Kafka/Kafka-Consumer.md b/docs/distribution/message-queue/Kafka/Kafka-Consumer.md new file mode 100644 index 0000000000..6f554f165b --- /dev/null +++ b/docs/distribution/message-queue/Kafka/Kafka-Consumer.md @@ -0,0 +1,419 @@ + + +> https://www.cnblogs.com/huxi2b/p/6223228.html + +## 消费者组 + +**1 什么是消费者组** + +其实对于这些基本概念的普及，网上资料实在太多了。我本不应该再画蛇添足了，但为了本文的完整性，我还是要花一些篇幅来重谈consumer group，至少可以说说我的理解。值得一提的是，由于我们今天基本上只探讨consumer group，对于单独的消费者不做过多讨论。 + +什么是consumer group? 一言以蔽之，consumer group是kafka提供的可扩展且具有容错性的消费者机制。既然是一个组，那么组内必然可以有多个消费者或消费者实例(consumer instance)，它们共享一个公共的ID，即group ID。组内的所有消费者协调在一起来消费订阅主题(subscribed topics)的所有分区(partition)。当然，每个分区只能由同一个消费组内的一个consumer来消费。（网上文章中说到此处各种炫目多彩的图就会紧跟着抛出来，我这里就不画了，请原谅）。个人认为，理解consumer group记住下面这三个特性就好了： + +- consumer group下可以有一个或多个consumer instance，consumer instance可以是一个进程，也可以是一个线程 +- group.id是一个字符串，唯一标识一个consumer group +- consumer group下订阅的topic下的每个分区只能分配给某个group下的一个consumer(当然该分区还可以被分配给其他group) + +**2 消费者位置(consumer position)** + +消费者在消费的过程中需要记录自己消费了多少数据，即消费位置信息。在Kafka中这个位置信息有个专门的术语：位移(offset)。很多消息引擎都把这部分信息保存在服务器端(broker端)。这样做的好处当然是实现简单，但会有三个主要的问题： + +1. broker从此变成有状态的，会影响伸缩性； +2. 需要引入应答机制(acknowledgement)来确认消费成功。 +3. 由于要保存很多consumer的offset信息，必然引入复杂的数据结构，造成资源浪费。而Kafka选择了不同的方式：每个consumer group保存自己的位移信息，那么只需要简单的一个整数表示位置就够了；同时可以引入checkpoint机制定期持久化，简化了应答机制的实现。 + +**3 位移管理(offset management)** + +**3.1 自动VS手动** + +Kafka默认是定期帮你自动提交位移的(enable.auto.commit = true)，你当然可以选择手动提交位移实现自己控制。另外kafka会定期把group消费情况保存起来，做成一个offset map，如下图所示： + +![](https://images2015.cnblogs.com/blog/735367/201612/735367-20161226175429711-638862783.png) + +上图中表明了test-group这个组当前的消费情况。 + + + +**3.2 位移提交** + +老版本的位移是提交到zookeeper中的，图就不画了，总之目录结构是：/consumers/<[group.id](http://group.id/)>/offsets//，但是zookeeper其实并不适合进行大批量的读写操作，尤其是写操作。因此kafka提供了另一种解决方案：增加__consumeroffsets topic，将offset信息写入这个topic，摆脱对zookeeper的依赖(指保存offset这件事情)。__consumer_offsets中的消息保存了每个consumer group某一时刻提交的offset信息。依然以上图中的consumer group为例，格式大概如下： + +![img](https://images2015.cnblogs.com/blog/735367/201612/735367-20161226175522507-1842910668.png) + + + +__consumers_offsets topic配置了compact策略，使得它总是能够保存最新的位移信息，既控制了该topic总体的日志容量，也能实现保存最新offset的目的。compact的具体原理请参见：[Log Compaction](https://kafka.apache.org/documentation/#compaction) + +至于每个group保存到__consumers_offsets的哪个分区，如何查看的问题请参见这篇文章：[Kafka 如何读取offset topic内容 (__consumer_offsets)](http://www.cnblogs.com/huxi2b/p/6061110.html) + + + +**4 Rebalance** + +**4.1 什么是rebalance？** + +rebalance本质上是一种协议，规定了一个consumer group下的所有consumer如何达成一致来分配订阅topic的每个分区。比如某个group下有20个consumer，它订阅了一个具有100个分区的topic。正常情况下，Kafka平均会为每个consumer分配5个分区。这个分配的过程就叫rebalance。 + +**4.2 什么时候rebalance？** + +这也是经常被提及的一个问题。rebalance的触发条件有三种： + +- 组成员发生变更(新consumer加入组、已有consumer主动离开组或已有consumer崩溃了——这两者的区别后面会谈到) +- 订阅主题数发生变更——这当然是可能的，如果你使用了正则表达式的方式进行订阅，那么新建匹配正则表达式的topic就会触发rebalance +- 订阅主题的分区数发生变更 + +**4.3 如何进行组内分区分配？** + +之前提到了group下的所有consumer都会协调在一起共同参与分配，这是如何完成的？Kafka新版本consumer默认提供了两种分配策略：range和round-robin。当然Kafka采用了可插拔式的分配策略，你可以创建自己的分配器以实现不同的分配策略。实际上，由于目前range和round-robin两种分配器都有一些弊端，Kafka社区已经提出第三种分配器来实现更加公平的分配策略，只是目前还在开发中。我们这里只需要知道consumer group默认已经帮我们把订阅topic的分区分配工作做好了就行了。 + +简单举个例子，假设目前某个consumer group下有两个consumer： A和B，当第三个成员加入时，kafka会触发rebalance并根据默认的分配策略重新为A、B和C分配分区，如下图所示： + +![img](https://images2015.cnblogs.com/blog/735367/201612/735367-20161226175710289-1164779517.png) + + + +**4.4 谁来执行rebalance和consumer group管理？** + +Kafka提供了一个角色：coordinator来执行对于consumer group的管理。坦率说kafka对于coordinator的设计与修改是一个很长的故事。最新版本的coordinator也与最初的设计有了很大的不同。这里我只想提及两次比较大的改变。 + +首先是0.8版本的coordinator，那时候的coordinator是依赖zookeeper来实现对于consumer group的管理的。Coordinator监听zookeeper的/consumers//ids的子节点变化以及/brokers/topics/数据变化来判断是否需要进行rebalance。group下的每个consumer都自己决定要消费哪些分区，并把自己的决定抢先在zookeeper中的/consumers//owners//下注册。很明显，这种方案要依赖于zookeeper的帮助，而且每个consumer是单独做决定的，没有那种“大家属于一个组，要协商做事情”的精神。 + +基于这些潜在的弊端，0.9版本的kafka改进了coordinator的设计，提出了group coordinator——每个consumer group都会被分配一个这样的coordinator用于组管理和位移管理。这个group coordinator比原来承担了更多的责任，比如组成员管理、位移提交保护机制等。当新版本consumer group的第一个consumer启动的时候，它会去和kafka server确定谁是它们组的coordinator。之后该group内的所有成员都会和该coordinator进行协调通信。显而易见，这种coordinator设计不再需要zookeeper了，性能上可以得到很大的提升。后面的所有部分我们都将讨论最新版本的coordinator设计。 + +**4.5 如何确定coordinator？** + +上面简单讨论了新版coordinator的设计，那么consumer group如何确定自己的coordinator是谁呢？ 简单来说分为两步： + +- 确定consumer group位移信息写入__consumers_offsets的哪个分区。具体计算公式： + - 　　__consumers_offsets partition# = Math.abs(groupId.hashCode() % groupMetadataTopicPartitionCount) 注意：groupMetadataTopicPartitionCount由offsets.topic.num.partitions指定，默认是50个分区。 +- 该分区leader所在的broker就是被选定的coordinator + +**4.6 Rebalance Generation** + +JVM GC的分代收集就是这个词(严格来说是generational)，我这里把它翻译成“届”好了，它表示了rebalance之后的一届成员，主要是用于保护consumer group，隔离无效offset提交的。比如上一届的consumer成员是无法提交位移到新一届的consumer group中。我们有时候可以看到ILLEGAL_GENERATION的错误，就是kafka在抱怨这件事情。每次group进行rebalance之后，generation号都会加1，表示group进入到了一个新的版本，如下图所示： Generation 1时group有3个成员，随后成员2退出组，coordinator触发rebalance，consumer group进入Generation 2，之后成员4加入，再次触发rebalance，group进入Generation 3. + +![img](https://images2015.cnblogs.com/blog/735367/201612/735367-20161226175822570-898409869.png) + + + +**4.7 协议(protocol)** + +前面说过了， rebalance本质上是一组协议。group与coordinator共同使用它来完成group的rebalance。目前kafka提供了5个协议来处理与consumer group coordination相关的问题： + +- Heartbeat请求：consumer需要定期给coordinator发送心跳来表明自己还活着 +- LeaveGroup请求：主动告诉coordinator我要离开consumer group +- SyncGroup请求：group leader把分配方案告诉组内所有成员 +- JoinGroup请求：成员请求加入组 +- DescribeGroup请求：显示组的所有信息，包括成员信息，协议名称，分配方案，订阅信息等。通常该请求是给管理员使用 + +Coordinator在rebalance的时候主要用到了前面4种请求。 +**4.8 liveness** + +consumer如何向coordinator证明自己还活着？ 通过定时向coordinator发送Heartbeat请求。如果超过了设定的超时时间，那么coordinator就认为这个consumer已经挂了。一旦coordinator认为某个consumer挂了，那么它就会开启新一轮rebalance，并且在当前其他consumer的心跳response中添加“REBALANCE_IN_PROGRESS”，告诉其他consumer：不好意思各位，你们重新申请加入组吧！ + +**4.9 Rebalance过程** + +终于说到consumer group执行rebalance的具体流程了。很多用户估计对consumer内部的工作机制也很感兴趣。下面就跟大家一起讨论一下。当然我必须要明确表示，rebalance的前提是coordinator已经确定了。 + +总体而言，rebalance分为2步：Join和Sync + +1 Join， 顾名思义就是加入组。这一步中，所有成员都向coordinator发送JoinGroup请求，请求入组。一旦所有成员都发送了JoinGroup请求，coordinator会从中选择一个consumer担任leader的角色，并把组成员信息以及订阅信息发给leader——注意leader和coordinator不是一个概念。leader负责消费分配方案的制定。 + +2 Sync，这一步leader开始分配消费方案，即哪个consumer负责消费哪些topic的哪些partition。一旦完成分配，leader会将这个方案封装进SyncGroup请求中发给coordinator，非leader也会发SyncGroup请求，只是内容为空。coordinator接收到分配方案之后会把方案塞进SyncGroup的response中发给各个consumer。这样组内的所有成员就都知道自己应该消费哪些分区了。 + +还是拿几张图来说明吧，首先是加入组的过程: + +![img](https://images2015.cnblogs.com/blog/735367/201612/735367-20161226175922086-1237318351.png) + +值得注意的是， 在coordinator收集到所有成员请求前，它会把已收到请求放入一个叫purgatory(炼狱)的地方。记得国内有篇文章以此来证明kafka开发人员都是很有文艺范的，写得也是比较有趣，有兴趣可以去搜搜。 +然后是分发分配方案的过程，即SyncGroup请求： + +![img](https://images2015.cnblogs.com/blog/735367/201612/735367-20161226180005242-1302422077.png) + +注意！！ consumer group的分区分配方案是在客户端执行的！Kafka将这个权利下放给客户端主要是因为这样做可以有更好的灵活性。比如这种机制下我可以实现类似于Hadoop那样的机架感知(rack-aware)分配方案，即为consumer挑选同一个机架下的分区数据，减少网络传输的开销。Kafka默认为你提供了两种分配策略：range和round-robin。由于这不是本文的重点，这里就不再详细展开了，你只需要记住你可以覆盖consumer的参数：partition.assignment.strategy来实现自己分配策略就好了。 + +**4.10 consumer group状态机** + +和很多kafka组件一样，group也做了个状态机来表明组状态的流转。coordinator根据这个状态机会对consumer group做不同的处理，如下图所示(完全是根据代码注释手动画的，多见谅吧) + +![img](https://images2015.cnblogs.com/blog/735367/201612/735367-20161226180046945-1657832046.png) + + + +简单说明下图中的各个状态： + +- Dead：组内已经没有任何成员的最终状态，组的元数据也已经被coordinator移除了。这种状态响应各种请求都是一个response： UNKNOWN_MEMBER_ID +- Empty：组内无成员，但是位移信息还没有过期。这种状态只能响应JoinGroup请求 +- PreparingRebalance：组准备开启新的rebalance，等待成员加入 +- AwaitingSync：正在等待leader consumer将分配方案传给各个成员 +- Stable：rebalance完成！可以开始消费了~ + +至于各个状态之间的流程条件以及action，这里就不具体展开了。 + + + +**三、rebalance场景剖析** + +上面详细阐述了consumer group是如何执行rebalance的，可能依然有些云里雾里。这部分对其中的三个重要的场景做详尽的时序展开，进一步加深对于consumer group内部原理的理解。由于图比较直观，所有的描述都将以图的方式给出，不做过多的文字化描述了。 + +**1 新成员加入组(member join)** + +**![img](https://images2017.cnblogs.com/blog/735367/201801/735367-20180122172838209-863721577.png)** + + + +**2 组成员崩溃(member failure)** + +前面说过了，组成员崩溃和组成员主动离开是两个不同的场景。因为在崩溃时成员并不会主动地告知coordinator此事，coordinator有可能需要一个完整的session.timeout周期才能检测到这种崩溃，这必然会造成consumer的滞后。可以说离开组是主动地发起rebalance；而崩溃则是被动地发起rebalance。okay，直接上图： + +![img](https://images2017.cnblogs.com/blog/735367/201801/735367-20180122172921209-2006292699.png) + +**3 组成员主动离组（member leave group)** + +![img](https://images2017.cnblogs.com/blog/735367/201801/735367-20180122172958600-838820663.png) + +**4 提交位移(member commit offset)** + + ![img](https://images2017.cnblogs.com/blog/735367/201801/735367-20180122173024959-506110104.png) + + + +总结一下，本文着重讨论了一下新版本的consumer group的内部设计原理，特别是consumer group与coordinator之间的交互过程，希望对各位有所帮助 + + + + + +> https://www.cnblogs.com/huxi2b/p/6124937.html +> +> https://www.cnblogs.com/huxi2b/p/7089854.html + + + +Kafka 0.9版本开始推出了Java版本的consumer，优化了coordinator的设计以及摆脱了对zookeeper的依赖。社区最近也在探讨正式用这套consumer API替换Scala版本的consumer的计划。鉴于目前这方面的资料并不是很多，本文将尝试给出一个利用KafkaConsumer编写的多线程消费者实例，希望对大家有所帮助。 + + 这套API最重要的入口就是KafkaConsumer(o.a.k.clients.consumer.KafkaConsumer)，普通的单线程使用方法官网API已有介绍，这里不再赘述了。因此，我们直奔主题——讨论一下如何创建多线程的方式来使用KafkaConsumer。KafkaConsumer和KafkaProducer不同，后者是线程安全的，因此我们鼓励用户在多个线程中共享一个KafkaProducer实例，这样通常都要比每个线程维护一个KafkaProducer实例效率要高。但对于KafkaConsumer而言，它不是线程安全的，所以实现多线程时通常由两种实现方法： + +1 每个线程维护一个KafkaConsumer + +![img](https://images2015.cnblogs.com/blog/735367/201612/735367-20161202105906443-1609157006.png) + +2 维护一个或多个KafkaConsumer，同时维护多个事件处理线程(worker thread) + +![img](https://images2015.cnblogs.com/blog/735367/201612/735367-20161202110008787-550483601.png) + +当然，这种方法还可以有多个变种：比如每个worker线程有自己的处理队列。consumer根据某种规则或逻辑将消息放入不同的队列。不过总体思想还是相同的，故这里不做过多展开讨论了。 + +　　下表总结了两种方法的优缺点： + +| | 优点 | 缺点 | +| -------------------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | +| 方法1(每个线程维护一个KafkaConsumer) | 方便实现 速度较快，因为不需要任何线程间交互 易于维护分区内的消息顺序 | 更多的TCP连接开销(每个线程都要维护若干个TCP连接) consumer数受限于topic分区数，扩展性差 频繁请求导致吞吐量下降 线程自己处理消费到的消息可能会导致超时，从而造成rebalance | +| 方法2 (单个(或多个)consumer，多个worker线程) | 可独立扩展consumer数和worker数，伸缩性好 | 实现麻烦通常难于维护分区内的消息顺序处理链路变长，导致难以保证提交位移的语义正确性 | + + + +下面我们分别实现这两种方法。需要指出的是，下面的代码都是最基本的实现，并没有考虑很多编程细节，比如如何处理错误等。 + +**方法1** + +**ConsumerRunnable类** + +``` + 1 import org.apache.kafka.clients.consumer.ConsumerRecord; + 2 import org.apache.kafka.clients.consumer.ConsumerRecords; + 3 import org.apache.kafka.clients.consumer.KafkaConsumer; + 4 + 5 import java.util.Arrays; + 6 import java.util.Properties; + 7 + 8 public class ConsumerRunnable implements Runnable { + 9 +10 // 每个线程维护私有的KafkaConsumer实例 +11 private final KafkaConsumer consumer; +12 +13 public ConsumerRunnable(String brokerList, String groupId, String topic) { +14 Properties props = new Properties(); +15 props.put("bootstrap.servers", brokerList); +16 props.put("group.id", groupId); +17 props.put("enable.auto.commit", "true"); //本例使用自动提交位移 +18 props.put("auto.commit.interval.ms", "1000"); +19 props.put("session.timeout.ms", "30000"); +20 props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); +21 props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); +22 this.consumer = new KafkaConsumer<>(props); +23 consumer.subscribe(Arrays.asList(topic)); // 本例使用分区副本自动分配策略 +24 } +25 +26 @Override +27 public void run() { +28 while (true) { +29 ConsumerRecords records = consumer.poll(200); // 本例使用200ms作为获取超时时间 +30 for (ConsumerRecord record : records) { +31 // 这里面写处理消息的逻辑，本例中只是简单地打印消息 +32 System.out.println(Thread.currentThread().getName() + " consumed " + record.partition() + +33 "th message with offset: " + record.offset()); +34 } +35 } +36 } +37 } +``` + +**ConsumerGroup类** + +``` + 1 package com.my.kafka.test; + 2 + 3 import java.util.ArrayList; + 4 import java.util.List; + 5 + 6 public class ConsumerGroup { + 7 + 8 private List consumers; + 9 +10 public ConsumerGroup(int consumerNum, String groupId, String topic, String brokerList) { +11 consumers = new ArrayList<>(consumerNum); +12 for (int i = 0; i < consumerNum; ++i) { +13 ConsumerRunnable consumerThread = new ConsumerRunnable(brokerList, groupId, topic); +14 consumers.add(consumerThread); +15 } +16 } +17 +18 public void execute() { +19 for (ConsumerRunnable task : consumers) { +20 new Thread(task).start(); +21 } +22 } +23 } +``` + +**ConsumerMain类** + +``` + 1 public class ConsumerMain { + 2 + 3 public static void main(String[] args) { + 4 String brokerList = "localhost:9092"; + 5 String groupId = "testGroup1"; + 6 String topic = "test-topic"; + 7 int consumerNum = 3; + 8 + 9 ConsumerGroup consumerGroup = new ConsumerGroup(consumerNum, groupId, topic, brokerList); +10 consumerGroup.execute(); +11 } +12 } +``` + + + +**方法2** + +**Worker类** + +``` + 1 import org.apache.kafka.clients.consumer.ConsumerRecord; + 2 + 3 public class Worker implements Runnable { + 4 + 5 private ConsumerRecord consumerRecord; + 6 + 7 public Worker(ConsumerRecord record) { + 8 this.consumerRecord = record; + 9 } +10 +11 @Override +12 public void run() { +13 // 这里写你的消息处理逻辑，本例中只是简单地打印消息 +14 System.out.println(Thread.currentThread().getName() + " consumed " + consumerRecord.partition() +15 + "th message with offset: " + consumerRecord.offset()); +16 } +17 } +``` + +**ConsumerHandler类** + +``` + 1 import org.apache.kafka.clients.consumer.ConsumerRecord; + 2 import org.apache.kafka.clients.consumer.ConsumerRecords; + 3 import org.apache.kafka.clients.consumer.KafkaConsumer; + 4 + 5 import java.util.Arrays; + 6 import java.util.Properties; + 7 import java.util.concurrent.ArrayBlockingQueue; + 8 import java.util.concurrent.ExecutorService; + 9 import java.util.concurrent.ThreadPoolExecutor; +10 import java.util.concurrent.TimeUnit; +11 +12 public class ConsumerHandler { +13 +14 // 本例中使用一个consumer将消息放入后端队列，你当然可以使用前一种方法中的多实例按照某张规则同时把消息放入后端队列 +15 private final KafkaConsumer consumer; +16 private ExecutorService executors; +17 +18 public ConsumerHandler(String brokerList, String groupId, String topic) { +19 Properties props = new Properties(); +20 props.put("bootstrap.servers", brokerList); +21 props.put("group.id", groupId); +22 props.put("enable.auto.commit", "true"); +23 props.put("auto.commit.interval.ms", "1000"); +24 props.put("session.timeout.ms", "30000"); +25 props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); +26 props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); +27 consumer = new KafkaConsumer<>(props); +28 consumer.subscribe(Arrays.asList(topic)); +29 } +30 +31 public void execute(int workerNum) { +32 executors = new ThreadPoolExecutor(workerNum, workerNum, 0L, TimeUnit.MILLISECONDS, +33 new ArrayBlockingQueue<>(1000), new ThreadPoolExecutor.CallerRunsPolicy()); +34 +35 while (true) { +36 ConsumerRecords records = consumer.poll(200); +37 for (final ConsumerRecord record : records) { +38 executors.submit(new Worker(record)); +39 } +40 } +41 } +42 +43 public void shutdown() { +44 if (consumer != null) { +45 consumer.close(); +46 } +47 if (executors != null) { +48 executors.shutdown(); +49 } +50 try { +51 if (!executors.awaitTermination(10, TimeUnit.SECONDS)) { +52 System.out.println("Timeout.... Ignore for this case"); +53 } +54 } catch (InterruptedException ignored) { +55 System.out.println("Other thread interrupted this shutdown, ignore for this case."); +56 Thread.currentThread().interrupt(); +57 } +58 } +59 +60 } +``` + +**Main类** + +``` + 1 public class Main { + 2 + 3 public static void main(String[] args) { + 4 String brokerList = "localhost:9092,localhost:9093,localhost:9094"; + 5 String groupId = "group2"; + 6 String topic = "test-topic"; + 7 int workerNum = 5; + 8 + 9 ConsumerHandler consumers = new ConsumerHandler(brokerList, groupId, topic); +10 consumers.execute(workerNum); +11 try { +12 Thread.sleep(1000000); +13 } catch (InterruptedException ignored) {} +14 consumers.shutdown(); +15 } +16 } +``` + +　　总结一下，这两种方法或是模型都有各自的优缺点，在具体使用时需要根据自己实际的业务特点来选取对应的方法。就我个人而言，我比较推崇第二种方法以及背后的思想，即不要将很重的处理逻辑放入消费者的代码中，很多Kafka consumer使用者碰到的各种rebalance超时、coordinator重新选举、心跳无法维持等问题都来源于此。 \ No newline at end of file diff --git a/docs/distribution/message-queue/Kafka/Kafka-Monitoring.md b/docs/distribution/message-queue/Kafka/Kafka-Monitoring.md new file mode 100644 index 0000000000..219bf9b844 --- /dev/null +++ b/docs/distribution/message-queue/Kafka/Kafka-Monitoring.md @@ -0,0 +1,37 @@ +> https://www.cnblogs.com/huxi2b/p/9075306.html + +目前Kafka监控方案看似很多，然而并没有一个“大而全”的通用解决方案。各家框架也是各有千秋，以下是我了解到的一些内容： + +**Kafka manager** + +Github地址： https://github.com/yahoo/kafka-manager。 这款监控框架的好处在于监控内容相对丰富，既能够实现broker级常见的JMX监控(比如出入站流量监控)，也能对consumer消费进度进行监控(比如lag等)。另外用户还能在页面上直接对集群进行管理，比如分区重分配或创建topic——当然这是一把双刃剑，好在kafka manager自己提供了只读机制，允许用户禁掉这些管理功能。 + +![img](https://images2018.cnblogs.com/blog/735367/201805/735367-20180523092531898-446027782.png) + +**Kafka Monitor** + +Github地址：[https://github.com/linkedin/kafka-monitor。](https://link.zhihu.com/?target=https%3A//github.com/linkedin/kafka-monitor%E3%80%82%E8%BF%99%E6%AC%BE%E7%9B%91%E6%8E%A7%E6%A1%86%E6%9E%B6%E6%9B%B4%E5%A4%9A%E7%9A%84%E6%98%AF%E5%85%B3%E6%B3%A8%E5%AF%B9Kafka%E9%9B%86%E7%BE%A4%E5%81%9A%E7%AB%AF%E5%88%B0%E7%AB%AF%E7%9A%84%E6%95%B4%E4%BD%93%E7%B3%BB%E7%BB%9F%E6%B5%8B%E8%AF%95%EF%BC%8C%E5%B9%B6%E4%BA%A7%E5%87%BA%E5%90%84%E7%A7%8D%E7%B3%BB%E7%BB%9F%E7%BA%A7%E7%9A%84%E7%9B%91%E6%8E%A7%E6%8C%87%E6%A0%87%EF%BC%8C%E6%AF%94%E5%A6%82%E7%AB%AF%E5%88%B0%E7%AB%AF%E7%9A%84%E5%BB%B6%E6%97%B6%EF%BC%8C%E6%95%B4%E4%BD%93%E6%B6%88%E6%81%AF%E4%B8%A2%E5%A4%B1%E7%8E%87%E7%AD%89%E3%80%82%E5%AF%B9%E4%BA%8E%E6%96%B0%E6%90%AD%E5%BB%BA%E7%9A%84Kafka%E7%BA%BF%E4%B8%8A%E9%9B%86%E7%BE%A4%EF%BC%8C%E4%BD%BF%E7%94%A8Kafka) 这款监控框架更多的是关注对Kafka集群做端到端的整体系统测试，并产出各种系统级的监控指标，比如端到端的延时，整体消息丢失率等。对于新搭建的Kafka线上集群，使用Kafka Monitor做个整体测试有助于你了解该集群整体的一些性能，但若是用于日常监控该框架便有些不便了，需要自己修改webapp/index.html中的监控指标，流程上有些不太友好。不过这款框架的优势是其主要贡献者是LinkedIn的lindong(Kafka 1.0.0版本中正式支持JBOD就是lindong开发的)，质量上应该是有保证的。 + + + +**Kafka Offset Monitor** + +Github地址：https://github.com/quantifind/KafkaOffsetMonitor。 KafkaOffsetMonitor应该算比较早的监控框架了，有着很酷的UI，使用者也是很多。但其比较大的劣势是对新版本consumer和security的支持，另外该项目已经近2年未维护了，其主力开发甚至是另起炉灶，重新写了一个新的KafkaOffsetMonitor来支持新版本consumer——[https://github.com/Morningstar/kafka-offset-monitor。](https://link.zhihu.com/?target=https%3A//github.com/Morningstar/kafka-offset-monitor%E3%80%82%E4%B8%8D%E8%BF%87%E7%9B%AE%E5%89%8D%E8%AF%A5%E9%A1%B9%E7%9B%AEstar%E6%95%B0%E5%BE%88%E5%B0%91%EF%BC%8C%E5%BA%94%E8%AF%A5%E6%B2%A1%E6%9C%89%E5%A4%A7%E8%A7%84%E6%A8%A1%E5%BA%94%E7%94%A8%EF%BC%8C%E5%88%B0%E5%BA%95%E6%98%AF%E5%90%A6%E9%80%82%E7%94%A8%E4%BA%8E%E7%94%9F%E4%BA%A7%E7%8E%AF%E5%A2%83%E9%9C%80%E8%A6%81%E7%94%A8%E6%88%B7%E8%87%AA%E8%A1%8C%E5%88%A4%E6%96%AD)不过目前该项目star数很少，应该没有大规模应用，到底是否适用于生产环境需要用户自行判断 + +![img](https://images2018.cnblogs.com/blog/735367/201805/735367-20180523092743727-2010095769.png) + + + +**Burrow** + +Github地址： https://github.com/linkedin/Burrow。 Burrow是LinkedIn开源的一款专门监控consumer lag的框架。事实上，当初其开源时我对它还是期待挺高的，不过令人遗憾地是后劲不足，发展得非常缓慢，而且这款框架是用Go写的，安装时要求必须有Go运行环境，故Burrow在普及上不如其他框架。Burrow没有UI界面，只开放了很多HTTP endpoint，这对于想偷懒的运维来说更是一个减分项。总之它的功能目前十分有限，普及率和知名度都是比较低的。不过好处是该项目主要贡献者是LinkedIn团队维护Kafka集群的主要负责人，故质量上是很有保证的 + + + +**JMXTrans + InfluxDB + Grafana** + +这实际上是一套监控框架的组合。有着非常非常炫酷的UI效果，极其适合向领导展示。具体搭建方法网上有很多教程，可以参考下。这里就不再赘述了。 + + ![img](https://images2018.cnblogs.com/blog/735367/201805/735367-20180523093127229-1072270939.png) + +总之，目前Kafka的监控并没有“放之四海而皆准”的解决方案，应该说每种框架都有自己独到的地方。用户需要结合自身监控需求选择适合的监控框架~ \ No newline at end of file diff --git a/docs/distribution/message-queue/Kafka/Kafka-Producer.md b/docs/distribution/message-queue/Kafka/Kafka-Producer.md new file mode 100644 index 0000000000..3793c965a7 --- /dev/null +++ b/docs/distribution/message-queue/Kafka/Kafka-Producer.md @@ -0,0 +1,809 @@ +# KafkaProducer使用介绍、参数配置以及核心源码解析 + +## 前言 + +Kafka，作为目前在大数据领域应用最为广泛的消息队列，其内部实现和设计有很多值得深入研究和分析的地方，使用 kafka 首先需要接触到 producer 的开发，然后是 consumer 开发，自 0.8.2.x 版本以后，kafka 提供了 java 版本的 producer 以代替以前 scala 版本的 producer，下面进行 producer 的解析（新版本producer 指的是 kafka-client 包的 `org.apache.kafka.clients.producer`，而不是 kafka 包下的`kafka.producer.Producer`）。 + +## Producer 概要设计 + +发送简略流程图 + +![img](https://tva1.sinaimg.cn/large/007S8ZIlly1gjmk1gaprjj30i906uwg3.jpg) + +大体上来说，用户首先构建待发送的消息对象 ProducerRecord，然后调用 `KafkaProducer#send` 方法进行发送。KafkaProducer 接收到消息后首先对其进行序列化，然后通过分区器（partitioner）确定该数据需要发送的 Topic 的分区，kafka 提供了一个默认的分区器，如果消息指定了 key，那么 partitioner 会根据 key 的 hash 值来确定目标分区，如果没有指定 key，那么将使用轮询的方式确定目标分区，这样可以最大程度的均衡每个分区的消息，确定分区之后，将会进一步确认该分区的 leader 节点(处理该分区消息读写的主节点)，最后追加写入到内存中的消息缓冲池(accumulator)。此时 `KafkaProducer#send` 方法成功返回。 + +KafkaProducer 中还有一个专门的 Sender IO 线程负责将缓冲池中的消息分批次发送给对应的 broker，完成真正的消息发送逻辑。 + + + +## Producer 程序开发 + +代码很简单，拢共分为 4 步 + +![](https://i04piccdn.sogoucdn.com/4195c4ece7632219) + +1. 配置 producer 参数 +2. 构造 ProducerRecord 消息 +3. 调用 send 方法进行发送 +4. 最后关闭 producer 资源 + +### 异步提交 + +```java +public static void main(String[] args) { + Properties properties = new Properties(); + // Kafka 服务端的主机名和端口号 + properties.put("bootstrap.servers", "10.202.253.240:9092"); + // 等待所有副本节点的应答 + properties.put("acks", "all"); + // 消息发送最大尝试次数 + properties.put("retries", 0); + // 一批消息处理大小 + properties.put("batch.size", 16384); + // 请求延时 + properties.put("linger.ms", 1); + // 发送缓存区内存大小 + properties.put("buffer.memory", 33554432); + // key 序列化 + properties.put("key.serializer","org.apache.kafka.common.serialization.StringSerializer"); + // value 序列化 + properties.put("value.serializer","org.apache.kafka.common.serialization.StringSerializer"); + Producer producer = new KafkaProducer(properties); + for (int i = 0; i < 10; i++) { + producer.send(new ProducerRecord("test", + Integer.toString(i), "hello world-" + i)); + } + producer.close(); +} +``` + +> 如果发送失败，也可能是 server.properties 配置的问题，不允许远程访问 +> +> - 去掉注释，listeners=PLAINTEXT://:9092 +> - 把 advertised.listeners 值改为 PLAINTEXT://host_ip:9092 + + + +### 同步提交 + +Kafka 的 Producer 发送消息采用的是异步发送的方式，`KafkaProducer` 的 `send` 方法返回 `Future` 对象，所以我们可以手动调用 `Future` 对象的 `get` 方法实现同步： + +```java +producer.send(new ProducerRecord("test", + Integer.toString(i), "hello world-" + i)).get(); +``` + +get 方法将阻塞，直到返回结果 RecordMetadata，所以可以看成是同步的。 + +### 异步待回调函数 + +上边的异步提交方式，可以称为**发送并忘记**（不关心消息是否正常到达，对返回结果不做任何判断处理），所以 kafak 提供了一个带回调函数的 send 方法 + +```java +Future send(ProducerRecord producer, Callback callback); +``` + +Callback 是一个回调接口，在消息发送完成之后可以回调我们自定义的实现 + +```java +for (int i = 0; i < 10; i++) { + producer.send(new ProducerRecord("test", + Integer.toString(i), "hello world-" + i), new Callback() { + @Override + public void onCompletion(RecordMetadata recordMetadata, Exception e) { + if (e == null){ + System.out.println("TopicName : " + recordMetadata.topic() + " Partiton : " + recordMetadata + .partition() + " Offset : " + recordMetadata.offset()); + } + else { + //进行异常处理 + } + } + }); +} +``` + +同样的也能获取结果，回调的时候会传递两个参数： + +- `RecordMetadata` 和上文一致的消息发送成功后的元数据。 +- `Exception` 消息发送过程中的异常信息。 + +但是这两个参数并不会同时都有数据，只有发送失败才会有异常信息，同时发送元数据为空。 + + + +**同步发送消息** + +- 优点：可以保证每条消息准确无误的写入了 broker，对于立即需要发送结果的情况非常适用，在 producer 故障或者宕机的情况也可以保证结果的正确性 + +- 缺点：由于同步发送需要每条消息都需要及时发送到 broker，没有缓冲批量操作，性能较低 + +**异步发送消息** + +- 优点：可以通过缓冲池对消息进行缓冲，然后进行消息的批量发送，大量减少了和 broker 的交互频率，性能极高，可以通过回调机制获取发送结果 + +- 缺点：在 producer 直接断电或者重启等故障，将有可能丢失消息发送结果，对消息准确性要求很高的场景不适用 + + + +### 带拦截器的 Producer + +TODO + + + + + + + +## Producer参数说明 + +- **bootstrap.servers**：Kafka 集群信息列表，用于建立到 Kafka 集群的初始连接，如果集群中机器数很多，只需要指定部分的机器主机的信息即可，不管指定多少台，producer 都会通过其中一台机器发现集群中所有的 broker，格式为 `hostname1:port,hostname2:port,hostname3:port` + +- **key.serializer**：任何消息发送到 broker 格式都是字节数组，因此在发送到 broker 之前需要进行序列化，该参数用于对 ProducerRecord 中的 key 进行序列化 + +- **value.serializer**：该参数用于对 ProducerRecord 中的 value 进行序列化 + + > kafka 默认提供了常用的序列化类，也可以通过实现 `org.apache.kafka.common.serialization.Serializer` 实现定义的序列化，Kafka 提供的序列化类如下： + > + > ![](https://static01.imgkr.com/temp/0997ff0db5a04501bc3438ed3b7402c9.png) + +- **acks：**ack 具有 3 个取值 0、1 和 -1(all) + + - acks=0：producer 不等待 broker 的 ack，这一操作提供了一个最低的延迟，broker 一接收到还没有写入磁盘就已经返回，吞吐量最高，当 broker 故障时有可能**丢失数据**； + - ack=1：producer 等待 broker 的 ack，partition 的 leader 落盘成功后返回 ack，如果在 follower 同步成功之前 leader 故障，那么将会**丢失数据** + + - acks=-1(all)：producer 等待 broker 的 ack，partition 的 leader 和 follower 全部落盘成功后才返回 ack。但是如果在 follower 同步完成后，broker 发送 ack 之前，leader 发生故障，那么就会造成**数据重复** + +- **buffer.memory**：该参数用于设置发送缓冲池的内存大小，单位是字节。默认值 33554432KB（32M） +- **max.block.ms**：当 producer 缓冲满了之后，阻塞的时间 +- **compression.type**：压缩类型，目前 kafka 支持四种种压缩类型 `gzip`、`snappy`、`lz4`、`zstd`，性能依次递增。默认值none（不压缩） +- **retries**：重试次数，0 表示不进行重试。默认值 2147483647 +- **batch.size**：批次处理大小，通常增加该参数，可以提升 producer 的吞吐量，默认值 16384 +- **linger.ms**：发送时延，和 batch.size 任满足其一条件，就会进行发送，减少网络IO，节省带宽之用。原理就是把原本需要多次发送的小batch，通过引入延时的方式合并成大batch发送，减少了网络传输的压力，从而提升吞吐量。当然，也会引入延时 +- **max.request.size**：控制 producer 发送请求的大小 +- **receive.buffer.bytes**：读取数据时要使用的 TCP 接收缓冲区（SO_RCVBUF）的大小。如果值为 -1，则将使用OS默认值。默认值 32768 +- **send. buffer.bytes**：发送数据时使用的 TCP 发送缓冲区（SO_SNDBUF）的大小。如果值为-1，则将使用OS默认值。默认值131072 +- **request.timeout.ms**：生产者发送数据时等待服务器返回响应的时间。默认值 30000ms + + + +## 消息分区机制 + +消息发送时都被发送到一个 topic，它是承载真实数据的逻辑容器，其本质就是一个目录，而 topic 又是由一些 Partition Logs(分区日志)组成的 + +![](https://static001.geekbang.org/resource/image/18/63/18e487b7e64eeb8d0a487c289d83ab63.png) + +**分区的原因：** + +1. **方便在集群中扩展**，每个 Partition 可以通过调整以适应它所在的机器，而一个 topic 又可以有多个 Partition 组成，因此整个集群就可以适应任意大小的数据了； +2. **可以提高并发**，因为可以以 Partition 为单位读写了。 + +> 其实分区的作用就是提供负载均衡的能力，或者说对数据进行分区的主要原因，就是为了实现系统的高伸缩性（Scalability）。 +> +> 不同的分区能够被放置到不同节点的机器上，而数据的读写操作也都是针对分区这个粒度而进行的，这样每个节点的机器都能独立地执行各自分区的读写请求处理。并且，我们还可以通过添加新的节点机器来增加整体系统的吞吐量。 +> +> 实际上分区的概念以及分区数据库早在 1980 年就已经有大牛们在做了，比如那时候有个叫 Teradata 的数据库就引入了分区的概念。 +> +> 值得注意的是，不同的分布式系统对分区的叫法也不尽相同。比如在 Kafka 中叫分区，在 MongoDB 和 Elasticsearch 中就叫分片 Shard，而在 HBase 中则叫 Region，在 Cassandra 中又被称作 vnode。从表面看起来它们实现原理可能不尽相同，但对底层分区（Partitioning）的整体思想却从未改变。 + +**分区的原则：** + +我们需要将 producer 发送的数据封装成一个 ProducerRecord 对象。 + +```java +public ProducerRecord (String topic, Integer partition, Long timestamp, K key, V value, Iterable

headers) +public ProducerRecord (String topic, Integer partition, Long timestamp, K key, V value) +public ProducerRecord (String topic, Integer partition, K key, V value, Iterable
headers) +public ProducerRecord (String topic, Integer partition, K key, V value) +public ProducerRecord (String topic, K key, V value) +public ProducerRecord (String topic, V value)Copy to clipboardErrorCopied +``` + +1. 指明 partition 的情况下，直接将指明的值直接作为 partiton 值； +2. 没有指明 partition 值但有 key 的情况下，将 key 的 hash 值与 topic 的 partition 数进行取余得到 partition 值； +3. 既没有 partition 值又没有 key 值的情况下，第一次调用时随机生成一个整数（后面每次调用在这个整数上自增），将这个值与 topic 可用的 partition 总数取余得到 partition 值，也就是常说的 round-robin 算法。 + +**自定义分区器** + +可以通过实现 `org.apache.kafka.clients.producer.Partitioner` 自定分区策略，在构造 KafkaProducer 时配置参数 `partitioner.class` 为自定义的分区类即可 + +```java +public class MyPartitioner implements Partitioner { + + @Override + public int partition(String topic, Object key, byte[] keyBytes, Object value, + byte[] valueBytes, Cluster cluster) { + return 0; + } + + @Override + public void close() { + + } + + @Override + public void configure(Map configs) { + + } +} +``` + + + +## 内部原理 + +Java producer(区别于Scala producer)是双线程的设计，分为 KafkaProducer 用户主线程和 Sender 线程 + +producer 总共创建两个线程：执行 `KafkaPrducer#send` 逻辑的线程——我们称之为“用户主线程”；执行发送逻辑的 IO 线程——我们称之为“Sender线程” + +### 1. 序列化+计算目标分区 + +这是 `KafkaProducer#send` 逻辑的第一步，即为待发送消息进行序列化并计算目标分区，如下图所示： + +![img](https://tva1.sinaimg.cn/large/007S8ZIlly1gjmp7vf31tj30k103uq3n.jpg) + +如上图所示，一条所属 topic 是"test"，消息体是"message"的消息被序列化之后结合 KafkaProducer 缓存的元数据(比如该 topic 分区数信息等)共同传给后面的 Partitioner 实现类进行目标分区的计算。 + +### 2. 追加写入消息缓冲区(accumulator) + +producer 创建时会创建一个默认 32MB(由buffer.memory参数指定)的 accumulator 缓冲区，专门保存待发送的消息。除了之前在“关键参数”段落中提到的 linger.ms 和 batch.size 等参数之外，该数据结构中还包含了一个特别重要的集合信息：消息批次信息(batches)。该集合本质上是一个 HashMap，里面分别保存了每个 topic 分区下的batch 队列，即前面说的批次是按照 topic 分区进行分组的。这样发往不同分区的消息保存在对应分区下的 batch队列中。举个简单的例子，假设消息 M1, M2 被发送到 test 的 0 分区但属于不同的 batch，M3 分送到 test 的 1分区，那么 batches 中包含的信息就是：{"test-0" -> [batch1, batch2], "test-1" -> [batch3]} + +单个 topic 分区下的 batch 队列中保存的是若干个消息批次。每个 batch 中最重要的 3 个组件包括： + +- compressor: 负责执行追加写入操作 +- batch缓冲区：由 batch.size 参数控制，消息被真正追加写入到的地方 +- thunks：保存消息回调逻辑的集合 + + 这一步的目的就是将待发送的消息写入消息缓冲池中，具体流程如下图所示： + +![](https://images2015.cnblogs.com/blog/735367/201702/735367-20170204164910854-2033381282.png) + +okay！这一步执行完毕之后理论上讲 `KafkaProducer.send` 方法就执行完毕了，用户主线程所做的事情就是等待 Sender 线程发送消息并执行返回结果了。 + +### 3. Sender线程预处理及消息发送 + +此时，该 Sender 线程登场了。严格来说，Sender 线程自 KafkaProducer 创建后就一直都在运行着 。它的工作流程基本上是这样的： + +1. 不断轮询缓冲区寻找已做好发送准备的分区 +2. 将轮询获得的各个 batch 按照目标分区所在的 leader broker 进行分组 +3. 将分组后的 batch 通过底层创建的 Socket 连接发送给各个 broker +4. 等待服务器端发送 response 回来 + +为了说明上的方便，我还是基于图的方式来解释Sender线程的工作原理： + +![img](https://images2015.cnblogs.com/blog/735367/201702/735367-20170204164759886-242426092.png) + +### 4. Sender线程处理response + +上图中Sender线程会发送PRODUCE请求给对应的broker，broker处理完毕之后发送对应的PRODUCE response。一旦Sender线程接收到response将依次(按照消息发送顺序)调用batch中的回调方法，如下图所示： + +![img](https://images2015.cnblogs.com/blog/735367/201702/735367-20170204170043386-56898873.png) + +做完这一步，producer发送消息就可以算作是100%完成了。通过这4步我们可以看到新版本producer发送事件完全是异步过程。因此在调优producer前我们就需要搞清楚性能瓶颈到底是在用户主线程还是在Sender线程。具体的性能测试方法以及调优方法以后有机会的话我再写一篇出来和大家讨论。 + + + + + +## 源码解毒 + +### producer send() + +```java +// 1.异步向一个 topic 发送数据 +public Future send(ProducerRecord record) { + return this.send(record, (Callback)null); +} + + +// 异步向一个 topic 发送数据，并注册回调函数，在回调函数中接受发送响应 +public Future send(ProducerRecord record, Callback callback) { + //如果有自定义拦截器，会去处理拦截器的Producer + ProducerRecord interceptedRecord = this.interceptors.onSend(record); + return this.doSend(interceptedRecord, callback); +} +``` + +`send()` 方法通过重载实现带回调和不带回调的参数，最终都调用 **doSend()** 方法 + +```java +private Future doSend(ProducerRecord record, Callback callback) { + TopicPartition tp = null; + + try { + //1、检查producer实例是否已关闭,如果关闭则抛出异常 + this.throwIfProducerClosed(); + + KafkaProducer.ClusterAndWaitTime clusterAndWaitTime; + try { + //2、确保topic的元数据(metadata)是可用的 + clusterAndWaitTime = this.waitOnMetadata(record.topic(), record.partition(), this.maxBlockTimeMs); + } catch (KafkaException var19) { + if (this.metadata.isClosed()) { + throw new KafkaException("Producer closed while send in progress", var19); + } + + throw var19; + } + + long remainingWaitMs = Math.max(0L, this.maxBlockTimeMs - clusterAndWaitTime.waitedOnMetadataMs); + Cluster cluster = clusterAndWaitTime.cluster; + //序列化key 和 value + byte[] serializedKey; + try { + serializedKey = this.keySerializer.serialize(record.topic(), record.headers(), record.key()); + } catch (ClassCastException var18) { + throw new SerializationException("Can't convert key of class " + record.key().getClass().getName() + " to class " + this.producerConfig.getClass("key.serializer").getName() + " specified in key.serializer", var18); + } + + byte[] serializedValue; + try { + serializedValue = this.valueSerializer.serialize(record.topic(), record.headers(), record.value()); + } catch (ClassCastException var17) { + throw new SerializationException("Can't convert value of class " + record.value().getClass().getName() + " to class " + this.producerConfig.getClass("value.serializer").getName() + " specified in value.serializer", var17); + } + + //获取分区信息,如果ProducerRecord指定了分区信息就使用该指定分区否则通过计算获取 + int partition = this.partition(record, serializedKey, serializedValue, cluster); + tp = new TopicPartition(record.topic(), partition); + this.setReadOnly(record.headers()); + Header[] headers = record.headers().toArray(); + //估算消息的字节大小 + int serializedSize = AbstractRecords.estimateSizeInBytesUpperBound(this.apiVersions.maxUsableProduceMagic(), this.compressionType, serializedKey, serializedValue, headers); + //确保消息大小不超过发送请求最大值(max.request.size)或者发送缓冲池发小(buffer.memory) + this.ensureValidRecordSize(serializedSize); + long timestamp = record.timestamp() == null ? this.time.milliseconds() : record.timestamp(); + this.log.trace("Sending record {} with callback {} to topic {} partition {}", new Object[]{record, callback, record.topic(), partition}); + Callback interceptCallback = new KafkaProducer.InterceptorCallback(callback, this.interceptors, tp); + //是否使用事务 + if (this.transactionManager != null && this.transactionManager.isTransactional()) { + this.transactionManager.maybeAddPartitionToTransaction(tp); + } + //向 accumulator 中追加数据 + RecordAppendResult result = this.accumulator.append(tp, timestamp, serializedKey, serializedValue, headers, interceptCallback, remainingWaitMs); + //如果 batch 已经满了,唤醒 sender 线程发送数据 + if (result.batchIsFull || result.newBatchCreated) { + this.log.trace("Waking up the sender since topic {} partition {} is either full or getting a new batch", record.topic(), partition); + this.sender.wakeup(); + } + + return result.future; + } catch (ApiException var20) { + ......... + } + ......... +} +``` + +`doSend()` 方法主要分为 10 步完成： + +1. 检查producer实例是否已关闭,如果关闭则抛出异常 + +2. 确保topic的元数据(metadata)是可用的 + +3. 序列化ProducerRecord中的key + +4. 序列化ProducerRecord中的value + +5. 确定分区信息,如果构造ProducerRecord指定了分区信息就使用该指定分区否则通过计算获取 + +6. 估算整条消息的字节大小 + +7. 确保消息大小不超过发送请求最大值(max.request.size)或者发送缓冲池发小(buffer.memory)，如果超过则抛出异常 + +8. 是否使用事务，如果使用则按照事务流程进行 + +9. 向 accumulator 中追加数据 + +10. 如果 batch 已经满了,唤醒 sender 线程发送数据 + +![](https://img2018.cnblogs.com/blog/1465200/201909/1465200-20190910145036444-1215527333.png) + +### 具体的发送过程 + +#### 获取 topic 的 metadata 信息——waitOnMetadata() + +Producer 通过 `waitOnMetadata()` 方法来获取对应 topic 的 metadata 信息 + +```java +//如果元数据Topic列表中还没有该Topic，则将其添加到元数据Topic列表中，并重置过期时间 +private KafkaProducer.ClusterAndWaitTime waitOnMetadata(String topic, Integer partition, long maxWaitMs) throws InterruptedException { + //首先从元数据中获取集群信息 + Cluster cluster = this.metadata.fetch(); + //集群的无效topic列表包含该topic那么抛出异常 + if (cluster.invalidTopics().contains(topic)) { + throw new InvalidTopicException(topic); + } else { + //否则将topic添加到Topic列表中 + this.metadata.add(topic); + //获取该topic的分区数 + Integer partitionsCount = cluster.partitionCountForTopic(topic); + //如果存在缓存的元数据，并且未指定分区的或者在已知分区范围内，那么返回缓存的元数据 + if (partitionsCount == null || partition != null && partition >= partitionsCount) { + long begin = this.time.milliseconds(); + long remainingWaitMs = maxWaitMs; + + long elapsed; + do { + if (partition != null) { + this.log.trace("Requesting metadata update for partition {} of topic {}.", partition, topic); + } else { + this.log.trace("Requesting metadata update for topic {}.", topic); + } + + this.metadata.add(topic); + int version = this.metadata.requestUpdate(); + this.sender.wakeup(); + + try { + this.metadata.awaitUpdate(version, remainingWaitMs); + } catch (TimeoutException var15) { + throw new TimeoutException(String.format("Topic %s not present in metadata after %d ms.", topic, maxWaitMs)); + } + + cluster = this.metadata.fetch(); + elapsed = this.time.milliseconds() - begin; + if (elapsed >= maxWaitMs) { + throw new TimeoutException(partitionsCount == null ? String.format("Topic %s not present in metadata after %d ms.", topic, maxWaitMs) : String.format("Partition %d of topic %s with partition count %d is not present in metadata after %d ms.", partition, topic, partitionsCount, maxWaitMs)); + } + + if (cluster.unauthorizedTopics().contains(topic)) { + throw new TopicAuthorizationException(topic); + } + + if (cluster.invalidTopics().contains(topic)) { + throw new InvalidTopicException(topic); + } + + remainingWaitMs = maxWaitMs - elapsed; + partitionsCount = cluster.partitionCountForTopic(topic); + } while(partitionsCount == null || partition != null && partition >= partitionsCount); + + return new KafkaProducer.ClusterAndWaitTime(cluster, elapsed); + } else { + return new KafkaProducer.ClusterAndWaitTime(cluster, 0L); + } + } +} +``` + + + +#### 确定partition值 + +关于 partition 值的计算，分为三种情况： + +1. 指明 partition 的情况下，直接将指定的值直接作为 partiton 值； +2. 没有指明 partition 值但有 key 的情况下，将 key 的 hash 值与 topic 的 partition 数进行取余得到 partition 值； +3. 既没有 partition 值又没有 key 值的情况下，第一次调用时随机生成一个整数（后面每次调用在这个整数上自增），将这个值与 topic 可用的 partition 总数取余得到 partition 值，也就是常说的轮询算法。 + +```java +private int partition(ProducerRecord record, byte[] serializedKey, byte[] serializedValue, Cluster cluster) { + Integer partition = record.partition(); + return partition != null ? partition : this.partitioner.partition(record.topic(), record.key(), serializedKey, record.value(), serializedValue, cluster); +} +``` + +**DefaultPartitioner 的实现** + +```java +public int partition(String topic, Object key, byte[] keyBytes, Object value, byte[] valueBytes, Cluster cluster) { + List partitions = cluster.partitionsForTopic(topic); + int numPartitions = partitions.size(); + if (keyBytes == null) { + //不指定key时,根据 topic 获取对应的整数变量 + int nextValue = this.nextValue(topic); + List availablePartitions = cluster.availablePartitionsForTopic(topic); + if (availablePartitions.size() > 0) { + int part = Utils.toPositive(nextValue) % availablePartitions.size(); + return ((PartitionInfo)availablePartitions.get(part)).partition(); + } else { + return Utils.toPositive(nextValue) % numPartitions; + } + } else { + //指定key时，通过key进行hash运算然后对该topic分区总数取余 + return Utils.toPositive(Utils.murmur2(keyBytes)) % numPartitions; + } +} + +private int nextValue(String topic) { + AtomicInteger counter = (AtomicInteger)this.topicCounterMap.get(topic); + if (null == counter) { + //第一次随机生成一个数 + counter = new AtomicInteger(ThreadLocalRandom.current().nextInt()); + AtomicInteger currentCounter = (AtomicInteger)this.topicCounterMap.putIfAbsent(topic, counter); + if (currentCounter != null) { + counter = currentCounter; + } + } + //以后每次递增 + return counter.getAndIncrement(); +} +``` + + + +#### 估算消息大小并检查 + +校验消息是为了防止消息过大或者数量过多，导致内存异常，具体实现 + +```java +//估算消息大小 +int serializedSize = AbstractRecords.estimateSizeInBytesUpperBound(this.apiVersions.maxUsableProduceMagic(), this.compressionType, serializedKey, serializedValue, headers); +//确保消息大小在有效范围内 +this.ensureValidRecordSize(serializedSize); + + private void ensureValidRecordSize(int size) { + //是否超过最大请求的限制,如果超过抛出异常 + if (size > this.maxRequestSize) { + throw new RecordTooLargeException("The message is " + size + " bytes when serialized which is larger than the maximum request size you have configured with the " + "max.request.size" + " configuration."); + //是否超过最大内存缓冲池大小,如果超过抛出异常 + } else if ((long)size > this.totalMemorySize) { + throw new RecordTooLargeException("The message is " + size + " bytes when serialized which is larger than the total memory buffer you have configured with the " + "buffer.memory" + " configuration."); + } + } +``` + + + +#### 向 accumulator 写数据 + +```java +public RecordAccumulator.RecordAppendResult append(TopicPartition tp, long timestamp, byte[] key, byte[] value, Header[] headers, Callback callback, long maxTimeToBlock) throws InterruptedException { + this.appendsInProgress.incrementAndGet(); + ByteBuffer buffer = null; + if (headers == null) { + headers = Record.EMPTY_HEADERS; + } + + RecordAccumulator.RecordAppendResult var16; + try { + // 当前tp有对应queue时直接返回，否则新建一个返回 + Deque dq = this.getOrCreateDeque(tp); + // 在对一个 queue 进行操作时,会保证线程安全 + synchronized(dq) { + if (this.closed) { + throw new KafkaException("Producer closed while send in progress"); + } + + RecordAccumulator.RecordAppendResult appendResult = this.tryAppend(timestamp, key, value, headers, callback, dq); + if (appendResult != null) { + RecordAccumulator.RecordAppendResult var14 = appendResult; + return var14; + } + } + // 为 topic-partition 创建一个新的 RecordBatch + byte maxUsableMagic = this.apiVersions.maxUsableProduceMagic(); + int size = Math.max(this.batchSize, AbstractRecords.estimateSizeInBytesUpperBound(maxUsableMagic, this.compression, key, value, headers)); + this.log.trace("Allocating a new {} byte message buffer for topic {} partition {}", new Object[]{size, tp.topic(), tp.partition()}); + // 给这个 RecordBatch 初始化一个 buffer + buffer = this.free.allocate(size, maxTimeToBlock); + synchronized(dq) { + if (this.closed) { + throw new KafkaException("Producer closed while send in progress"); + } + + RecordAccumulator.RecordAppendResult appendResult = this.tryAppend(timestamp, key, value, headers, callback, dq); + if (appendResult == null) { + // 给 topic-partition 创建一个 RecordBatch + MemoryRecordsBuilder recordsBuilder = this.recordsBuilder(buffer, maxUsableMagic); + ProducerBatch batch = new ProducerBatch(tp, recordsBuilder, this.time.milliseconds()); + // 向新的 RecordBatch 中追加数据 + FutureRecordMetadata future = (FutureRecordMetadata)Utils.notNull(batch.tryAppend(timestamp, key, value, headers, callback, this.time.milliseconds())); + // 将 RecordBatch 添加到对应的 queue 中 + dq.addLast(batch); + // 向未 ack 的 batch 集合添加这个 batch + this.incomplete.add(batch); + buffer = null; + // 如果 dp.size()>1 就证明这个 queue 有一个 batch 是可以发送了 + RecordAccumulator.RecordAppendResult var19 = new RecordAccumulator.RecordAppendResult(future, dq.size() > 1 || batch.isFull(), true); + return var19; + } + + var16 = appendResult; + } + } finally { + if (buffer != null) { + this.free.deallocate(buffer); + } + + this.appendsInProgress.decrementAndGet(); + } + + return var16; +} +``` + + + +#### 使用sender线程批量发送 + +```java +public void run() { + this.log.debug("Starting Kafka producer I/O thread."); + + while(this.running) { + try { + this.run(this.time.milliseconds()); + } catch (Exception var4) { + this.log.error("Uncaught error in kafka producer I/O thread: ", var4); + } + } + + this.log.debug("Beginning shutdown of Kafka producer I/O thread, sending remaining records."); + // 累加器中可能仍然有请求，或者等待确认,等待他们完成就停止接受请求 + while(!this.forceClose && (this.accumulator.hasUndrained() || this.client.inFlightRequestCount() > 0)) { + //...... + } + + if (this.forceClose) { + //...... + } + + try { + this.client.close(); + } catch (Exception var2) { + this.log.error("Failed to close network client", var2); + } + + this.log.debug("Shutdown of Kafka producer I/O thread has completed."); +} + +void run(long now) { + //首先还是判断是否使用事务发送 + if (this.transactionManager != null) { + try { + if (this.transactionManager.shouldResetProducerStateAfterResolvingSequences()) { + this.transactionManager.resetProducerId(); + } + + if (!this.transactionManager.isTransactional()) { + this.maybeWaitForProducerId(); + } else if (this.transactionManager.hasUnresolvedSequences() && !this.transactionManager.hasFatalError()) { + this.transactionManager.transitionToFatalError(new KafkaException("The client hasn't received acknowledgment for some previously sent messages and can no longer retry them. It isn't safe to continue.")); + } else if (this.transactionManager.hasInFlightTransactionalRequest() || this.maybeSendTransactionalRequest(now)) { + this.client.poll(this.retryBackoffMs, now); + return; + } + //如果事务管理器转态错误 或者producer没有id 将停止进行发送 + if (this.transactionManager.hasFatalError() || !this.transactionManager.hasProducerId()) { + RuntimeException lastError = this.transactionManager.lastError(); + if (lastError != null) { + this.maybeAbortBatches(lastError); + } + + this.client.poll(this.retryBackoffMs, now); + return; + } + + if (this.transactionManager.hasAbortableError()) { + this.accumulator.abortUndrainedBatches(this.transactionManager.lastError()); + } + } catch (AuthenticationException var5) { + this.log.trace("Authentication exception while processing transactional request: {}", var5); + this.transactionManager.authenticationFailed(var5); + } + } + //发送Producer数据 + long pollTimeout = this.sendProducerData(now); + this.client.poll(pollTimeout, now); +} + +private long sendProducerData(long now) { + Cluster cluster = this.metadata.fetch(); + // 获取准备发送数据的分区列表 + ReadyCheckResult result = this.accumulator.ready(cluster, now); + Iterator iter; + //如果有分区leader信息是未知的,那么就强制更新metadata + if (!result.unknownLeaderTopics.isEmpty()) { + iter = result.unknownLeaderTopics.iterator(); + + while(iter.hasNext()) { + String topic = (String)iter.next(); + this.metadata.add(topic); + } + + this.log.debug("Requesting metadata update due to unknown leader topics from the batched records: {}", result.unknownLeaderTopics); + this.metadata.requestUpdate(); + } + + iter = result.readyNodes.iterator(); + long notReadyTimeout = 9223372036854775807L; + + // 移除没有准备好发送的Node + while(iter.hasNext()) { + Node node = (Node)iter.next(); + if (!this.client.ready(node, now)) { + iter.remove(); + notReadyTimeout = Math.min(notReadyTimeout, this.client.pollDelayMs(node, now)); + } + } + //创建Producer请求内容 + Map> batches = this.accumulator.drain(cluster, result.readyNodes, this.maxRequestSize, now); + this.addToInflightBatches(batches); + List expiredBatches; + Iterator var11; + ProducerBatch expiredBatch; + if (this.guaranteeMessageOrder) { + Iterator var9 = batches.values().iterator(); + + while(var9.hasNext()) { + expiredBatches = (List)var9.next(); + var11 = expiredBatches.iterator(); + + while(var11.hasNext()) { + expiredBatch = (ProducerBatch)var11.next(); + this.accumulator.mutePartition(expiredBatch.topicPartition); + } + } + } + + this.accumulator.resetNextBatchExpiryTime(); + List expiredInflightBatches = this.getExpiredInflightBatches(now); + expiredBatches = this.accumulator.expiredBatches(now); + expiredBatches.addAll(expiredInflightBatches); + if (!expiredBatches.isEmpty()) { + this.log.trace("Expired {} batches in accumulator", expiredBatches.size()); + } + + var11 = expiredBatches.iterator(); + + while(var11.hasNext()) { + expiredBatch = (ProducerBatch)var11.next(); + String errorMessage = "Expiring " + expiredBatch.recordCount + " record(s) for " + expiredBatch.topicPartition + ":" + (now - expiredBatch.createdMs) + " ms has passed since batch creation"; + this.failBatch(expiredBatch, -1L, -1L, new TimeoutException(errorMessage), false); + if (this.transactionManager != null && expiredBatch.inRetry()) { + this.transactionManager.markSequenceUnresolved(expiredBatch.topicPartition); + } + } + + this.sensors.updateProduceRequestMetrics(batches); + long pollTimeout = Math.min(result.nextReadyCheckDelayMs, notReadyTimeout); + pollTimeout = Math.min(pollTimeout, this.accumulator.nextExpiryTimeMs() - now); + pollTimeout = Math.max(pollTimeout, 0L); + if (!result.readyNodes.isEmpty()) { + this.log.trace("Nodes with data ready to send: {}", result.readyNodes); + pollTimeout = 0L; + } + + this.sendProduceRequests(batches, now); + return pollTimeout; +} +``` + + + + + + + + + + + + + +![](https://static01.imgkr.com/temp/7e16aac5204940cb9033ef7c93512a16.png) + +![](https://static01.imgkr.com/temp/e414ef7c122f4c1cae598047d96d3f49.png) + +之后在IDEA中 , 双击shift , 调出全局搜索框就可以搜索到 jar包里的类了 + + + + + + + + + + + + + + + +https://segmentfault.com/a/1190000016643285 + +https://blog.csdn.net/qq_18581221/article/details/89320230 \ No newline at end of file diff --git a/docs/distribution/message-queue/Kafka/Kafka-Rebalancing.md b/docs/distribution/message-queue/Kafka/Kafka-Rebalancing.md new file mode 100755 index 0000000000..48d3899099 --- /dev/null +++ b/docs/distribution/message-queue/Kafka/Kafka-Rebalancing.md @@ -0,0 +1,15 @@ +![](https://img.starfish.ink/mq/kafka-rebalancing-banner.png) + + + + + + + + + + + + + +- https://www.verica.io/blog/understanding-kafkas-consumer-group-rebalancing/ \ No newline at end of file diff --git a/docs/distribution/message-queue/Kafka/Kafka-Version.md b/docs/distribution/message-queue/Kafka/Kafka-Version.md new file mode 100644 index 0000000000..c74736e7e0 --- /dev/null +++ b/docs/distribution/message-queue/Kafka/Kafka-Version.md @@ -0,0 +1,130 @@ +--- +title: Kafka版本号 -- 傻傻搞不清楚 +date: 2022-03-17 +tags: + - Kafka +categories: Kafka +--- + +![](https://img.starfish.ink/mq/kafka-version-banner.png) + +> 用 Kafka 的时候，其实大家都会有默认的共识，客户端和服务端版本号要尽量统一，因为不同版本之间的差异和功能其实差距还是挺大的，记得刚工作那会，我对 kafka 的命名就很迷惑，0.11.0.0 后来就升级到 1.0.1？这篇算是一个交代吧，祝好 + +## Kafka 版本命名 + +截止到目前，Apache Kafka 已经到了 3.1 版本，我们看下之前的版本 + +![](https://img.starfish.ink/mq/kafka-version-name.png) + + + +比如我们在官网上下载 Kafka 时，会看到这样的版本： + +![](https://img.starfish.ink/mq/kafka-version.png) + +Kafka 的版本命名，这么长一串，其实呢，前面的版本号是编译 Kafka 源代码的 Scala 编译器版本，真正的版本号其实是后边的 `3.1.0`， + +> Apache Kafka 的服务端核心代码最初是用 Scala 编写的，但近年来，代码库中有越来越多的部分被重写为 Java。 + +![](https://img.starfish.ink/mq/kafka-tags.png) + +1.x 版本后，kafka 启用三位数的命名规则，从 tag 记录可以看到，之前的版本都是 `0.10.2.2` 、 `0.11.0.2` 这种，新的版本命名采用了 + +”**大版本-小版本-patch版本**“ 这样比较主流的命名方式 + +- 前面的 3 表示大版本号，即 Major Version； +- 中间的 1 表示小版本号或次版本号，即 Minor Version； +- 最后的01 表示修订版本号，也就是 Patch 号。 + +Kafka 社区在发布 1.0.0 版本后特意写过一篇文章，宣布 Kafka 版本命名规则正式从 4 位演进到 3 位，比如 0.11.0.0 版本就是 4 位版本号。（其实我们可以把之前的四位 0.10 或者 0.11 这种看成是大版本号，这样就好理解了） + + + + + +## Kafka 版本演进 + +Kafka 目前总共演进了 8 个大版本，分别是 0.7.x、0.8.x、0.9.x、0.10.x、0.11.x、1.0.x 和 2.0.x 以及现在的 3.0.x，其中的小版本和 Patch 版本很多。哪些版本引入了哪些重大的功能改进？ + + + +### 0.7.x版本 + +我们先从 0.7 版本说起，实际上也没什么可说的，这是最早开源时的“上古”版本了，以至于我也从来都没有接触过。这个版本只提供了最基础的消息队列功能，甚至连副本机制都没有，我实在想不出有什么理由你要使用这个版本，因此一旦有人向你推荐这个版本，果断走开就好了。 + + + +### 0.8.x版本 + +两个重要特性，一个是 Kafka 0.8.0 增加了**副本机制**，另一个是 Kafka 0.8.2.0 引入了新版本 Producer API。 + +Kafka 从 0.7 时代演进到 0.8 之后正式引入了**副本机制**，至此 Kafka 成为了一个真正意义上完备的分布式高可靠消息队列解决方案。有了副本备份机制，Kafka 就能够比较好地做到消息无丢失。那时候生产和消费消息使用的还是老版本的客户端 API，所谓的老版本是指当你用它们的 API 开发生产者和消费者应用时，你需要指定 ZooKeeper 的地址而非 Broker 的地址。 + + + +### 0.9.x版本 + +Kafka 0.9 是一个重大的版本迭代，增加了非常多的新特性，主要体现在三个方面： + +- **安全认证 / 权限**：在0.9.0之前，Kafka 安全方面的考虑几乎为 0。Kafka 0.9.0 在安全认证、授权管理、数据加密等方面都得到了支持，包括支持Kerberos等 +- **新版本Consumer API**：Kafka 0.9.0 重写并提供了新版消费端 API，使用方式也是从连接 Zookeeper 切到了连接 Broker，但是此时新版 Consumer API 也不太稳定、存在不少 Bug，生产使用可能会比较痛苦；而 0.9.0 版本的 Producer API 已经比较稳定了，生产使用问题不大 +- **Kafka Connect**：Kafka 0.9.0 引入了新的组件 Kafka Connect ，用于实现 Kafka 与其他外部系统之间的数据抽取 + + + +### 0.10.x版本 + +Kafka 0.10 是一个重要的大版本，引入了 Kafka Streams，使得 Kafka 不再仅是一个消息引擎，而是往一个分布式流处理平台方向发展。0.10 大版本包含两个小版本：0.10.1 和 0.10.2，它们的主要功能变更都是在 Kafka Streams 组件上。 + +值得一提的是，自 0.10.2.2 版本起，新版本 Consumer API 已经比较稳定了，而且 Producer API 的性能也得到了提升，因此对于使用 0.10.x 大版本的用户，建议使用或升级到 Kafka 0.10.2.2 版本。 + + + +### 0.11.x版本 + +Kafka 0.11 是一个里程碑式的大版本，主要有两个大的变更，一是 Kafka 从这个版本开始支持 **Exactly-Once 语义** 即精准一次语义，主要是实现了 Producer 端的消息幂等性，以及事务特性，这对于 Kafka 流式处理具有非常大的意义。 + +另一个重大变更是**Kafka消息格式的重构**，Kafka 0.11 主要为了实现 Producer 幂等性与事务特性，重构了投递消息的数据结构。这一点非常值得关注，因为 Kafka 0.11之后的消息格式发生了变化，所以我们要特别注意 Kafka 不同版本间消息格式不兼容的问题。 + + + +### 1.x版本 + +Kafka 1.x 更多的是Kafka Streams方面的改进，以及Kafka Connect的改进与功能完善等。但仍有两个重要特性，一是 Kafka 1.0.0 实现了**磁盘的故障转移**，当 Broker 的某一块磁盘损坏时数据会自动转移到其他正常的磁盘上，Broker 还会正常工作，这在之前版本中则会直接导致 Broker 宕机，因此 Kafka 的可用性与可靠性得到了提升； + +二是 Kafka 1.1.0 开始支持**副本跨路径迁移**，分区副本可以在同一 Broker 不同磁盘目录间进行移动，这对于磁盘的负载均衡非常有意义。 + + + +### 2.x版本 + +Kafka 2.x 更多的也是 Kafka Streams、Connect 方面的性能提升与功能完善，以及安全方面的增强等。一个使用特性，Kafka 2.1.0 开始支持 ZStandard 的压缩方式，提升了消息的压缩比，显著减少了磁盘空间与网络io消耗。 + + + +### 3.x 版本 + +Apache Kafka 3.0 算是一个重要的版本更新，引入了各种新功能、突破性的 API 更改以及对 KRaft 的改进：Apache Kafka 的内置共识机制将取代 Apache ZooKeeper™ ，弃用对 Java 8 和 Scala 2.12 的支持，并且将在 v4.0 中完全删除。除此之外，维护者还决定弃用消息格式 v0 和 v1，将消息格式 v2 作为默认消息格式 + +这个版本目前用的应该不是很多~ + + + +## Kafka版本建议 + +1. 遵循一个基本原则，Kafka 客户端版本和服务端版本应该保持一致，否则可能会遇到一些问题，比如版本不同导致消息格式不兼容问题。 +2. 根据是否用到了 Kafka 的一些新特性来选择，假如要用到 Kafka 生产端的消息幂等性，那么建议选择 Kafka 0.11 或之后的版本。 +3. 选择一个自己熟悉且稳定的版本，如果说没有比较熟悉的版本，建议选择一个较新且稳定、使用比较广泛的版本。 + + + +## Reference + +- https://kafka.apache.org/downloads +- https://kafka.apache.org/documentation.html#upgrade_110_notable +- https://developpaper.com/version-number-of-kafka-extremely-important/ +- https://zhuanlan.zhihu.com/p/245941592 +- https://mp.weixin.qq.com/s?__biz=MzU1NDA4NjU2MA==&mid=2247493794&idx=1&sn=13c1c5a6b7acb95fe0a52b191d336f9a&chksm=fbea516dcc9dd87ba1ad6ce86d7b5af3e24c3e25fa526abdbbfb109d343fac30c765bdac6d90&scene=21#wechat_redirect + + + diff --git a/docs/distribution/message-queue/Kafka/Kafka-Workflow.md b/docs/distribution/message-queue/Kafka/Kafka-Workflow.md new file mode 100644 index 0000000000..d719917dbe --- /dev/null +++ b/docs/distribution/message-queue/Kafka/Kafka-Workflow.md @@ -0,0 +1,834 @@ +--- +title: Kafka 工作流程和存储机制分析 +date: 2023-02-15 +tags: + - Kafka +categories: Kafka +--- + +![](https://img.starfish.ink/mq/kafka-workflow.jpg) + + + +## Kafka 架构 + +![overview-of-kafka-architecture](https://images.ctfassets.net/gt6dp23g0g38/4DA2zHan28tYNAV2c9Vd98/b9fca38c23e2b2d16a4c4de04ea6dd3f/Kafka_Internals_004.png) + +Kafka 是一个数据流处理系统，它允许开发者实时地对新发生的事件做出反应。 + +Kafka 的架构由存储层和计算层组成。 + +存储层被设计为高效地存储数据，并且是一个分布式系统，这意味着如果你的存储需求随时间增长，你可以轻松地扩展系统以适应增长。 + +计算层由四个核心组件构成： + +1. **生产者（Producer）API**： + - 生产者 API 允许开发者将数据发送到 Kafka 主题。 +2. **消费者（Consumer）API**： + - 消费者 API 允许开发者从 Kafka 主题接收数据流。 +3. **流（Streams）API**： + - 流 API 提供了构建和运行流处理应用程序的能力，可以对数据进行复杂的转换和处理。 +4. **连接器（Connector）API**： + - 连接器 API 允许开发者将 Kafka 与外部系统如数据库连接起来，实现数据的自动导入和导出。 + + + + + +Kafka 官方定义是一个流处理系统，用于处理事件流（Event Stream），事件是对发生的某事的记录，同时提供了关于所发生事情的信息。 + +Kafka 记录（Kafka Record）是 Kafka 中传输和存储的基本单元。每条记录包含以下内容： + +- **主题（Topic）**：记录所属的主题。 +- **分区（Partition）**：记录存储的分区。 +- **偏移量（Offset）**：记录在分区中的位置。 +- **时间戳（Timestamp）**：记录的时间戳。 +- **键（Key）**：可选，用于分区策略。 +- **值（Value）**：实际的数据内容。 +- **头部（Headers）**：可选的元数据，用于附加信息。 + +![](/Users/starfish/oceanus/picBed/kafka/kafka-event-stream.png) + + + + + + + +### Record Schema + +In Kafka, the key and value are stored as byte arrays which means that clients can work with any type of data that can be serialized to bytes. A popular format among Kafka users is Avro, which is also supported by Confluent Schema Registry. + +When integrated with Schema Registry, the first byte of an event will be a magic byte which signifies that this event is using a schema in the Schema Registry. The next four bytes make up the schema ID that can be used to retrieve the schema from the registry, and the rest of the bytes contain the event itself. Schema Registry also supports Protobuf and JSON schema formats. + +![inline-pic-schema](https://images.ctfassets.net/gt6dp23g0g38/33K4H01OUsnYyR7z4dzRkj/d07131ca9752ba363792844e887c7f48/Kafka_Internals_006.png) + + + +### Topics + +**Kafka 中消息是以 topic 进行分类的**，生产者生产消息，消费者消费消息，都是面向 topic 的。 + +在 Kafka 中，一个核心概念是主题（Topic）。主题是事件的仅追加（append-only）且不可变的日志。通常，相同类型或以某种方式相关的事件会被放入同一个主题中。Kafka 生产者将事件写入主题，而 Kafka 消费者从主题中读取事件。 + +![inline-pic-topic](https://images.ctfassets.net/gt6dp23g0g38/J2Y8oV2hoVWLv8u7sJ2v6/271fa3dde5d47e3a5980ad51fdd8b331/Kafka_Internals_007.png) + + + +### Topic 分区 + +![inline-pic-partition-2](https://images.ctfassets.net/gt6dp23g0g38/ODHQiu10QMZJ4bBbeQcvG/024159856d3361aaac482da28979acf6/Kafka_Internals_009.png) + +为了在 Kafka 中分配事件的存储和处理，Kafka 使用了分区的概念。一个主题由一个或多个分区组成，这些分区可以位于 Kafka 集群的不同节点上。 + +分区是 Kafka 事件存储的主要单元，尽管我们将在后面讨论的分层存储中，一些事件存储会从分区中移出。分区也是并行性的主要单元。事件可以通过同时写入多个分区来并行地生产到主题中。同样，消费者可以通过不同的消费者实例从不同的分区中读取来分散他们的工作负载。如果我们只使用一个分区，我们将只能有效地使用一个消费者实例。 + +在分区内部，每个事件都被赋予一个称为偏移量（offset）的唯一标识符。随着事件的添加，给定分区的偏移量将持续增长，而且偏移量从不重复使用。在 Kafka 中，偏移量有多种用途，其中包括消费者跟踪已处理的事件。 + + + + + +Kafka 将数据和元数据分别管理，这种设计有助于提高系统的灵活性和可扩展性。以下是 Kafka 如何分别管理数据和元数据的一些关键点： + +1. **数据管理**： + - Kafka 的数据层主要负责存储实际的消息事件。数据通常存储在主题（Topics）中，并且可以分布在多个分区（Partitions）上以实现并行处理和扩展性。 +2. **元数据管理**： + - 元数据包括关于 Kafka 集群的配置信息、主题的分区信息、副本（Replicas）的分配、消费者组（Consumer Groups）的状态等。这些信息通常由 Kafka 的控制平面（Control Plane）管理。 +3. **Zookeeper 的使用**： + - 在早期版本的 Kafka 中，Zookeeper 被用来存储和管理元数据。Zookeeper 作为一个协调服务，处理集群管理任务，如领导者选举、节点注册、负载均衡等。 +4. **Kafka 自身管理元数据**： + - 从 Kafka 2.8.0 版本开始，Kafka 引入了 KRaft（Kafka Raft Metadata）模式，允许 Kafka 自身管理元数据，而不再依赖 Zookeeper。 +5. **KRaft 协议**： + - KRaft 协议是 Kafka 用来处理元数据管理的共识协议，它借鉴了 Raft 算法来确保元数据的一致性和高可用性。 +6. **元数据的持久化**： + - Kafka 将元数据持久化存储在本地文件系统中，例如，主题配置信息存储在 `meta.properties` 文件中。 +7. **控制平面和数据平面的分离**： + - Kafka 的控制平面负责管理元数据和集群协调，而数据平面负责实际的消息存储和传输。这种分离允许 Kafka 更好地扩展和优化性能。 +8. **动态更新元数据**： + - Kafka 允许动态地更新元数据，如增加分区、修改副本因子等，而无需停止服务。 +9. **元数据的高可用性**： + - Kafka 的元数据管理设计确保了元数据的高可用性，即使在某些节点或组件失败的情况下，集群仍然可以正常运行。 + +通过分别管理数据和元数据，Kafka 能够提供更好的性能、灵活性和可维护性，同时也为复杂的分布式系统提供了强大的支持。 + + + + + + + +Kafka 将每个 Topic 划分为多个分区（Partition），每个分区在磁盘上对应一个日志文件（Log）。为了管理这些日志文件，Kafka 将每个分区的日志文件进一步分为多个段（Segment）。每个段包含一段时间内的消息，并且有两个文件：一个数据文件和一个索引文件。 + +- **数据文件**：存储实际的消息内容，文件扩展名为 `.log`。 +- **索引文件**：存储消息的偏移量和物理位置，用于快速定位消息，文件扩展名为 `.index`。 + +![](/Users/starfish/oceanus/picBed/kafka/kafka-topic.png) + +**topic 是逻辑上的概念，而 patition 是物理上的概念**，每个 patition 对应一个 log 文件，而 log 文件中存储的就是 producer 生产的数据，patition 生产的数据会被不断的添加到 log 文件的末端，且每条数据都有自己的 offset。 + +消费组中的每个消费者，都是实时记录自己消费到哪个 offset，以便出错恢复，从上次的位置继续消费。 + + + +### Kafka 为什么要将 Topic 进行分区？ + +答：负载均衡+水平扩展 + +Topic 只是逻辑概念，面向的是 producer 和 consumer；而 Partition 则是物理概念。可以想象，如果 Topic 不进行分区，而将 Topic 内的消息存储于一个 broker，那么关于该 Topic 的所有读写请求都将由这一个 broker 处理，吞吐量很容易陷入瓶颈，这显然是不符合高吞吐量应用场景的。有了 Partition 概念以后，假设一个 Topic 被分为 10 个 Partitions，Kafka 会根据一定的算法将 10 个 Partition 尽可能均匀的分布到不同的 broker（服务器）上，当 producer 发布消息时，producer 客户端可以采用 `random`、`key-hash` 及 `轮询` 等算法选定目标 partition，若不指定，Kafka 也将根据一定算法将其置于某一分区上。Partiton 机制可以极大的提高吞吐量，并且使得系统具备良好的水平扩展能力。 + +在创建 topic 时可以在 `$KAFKA_HOME/config/server.properties` 中指定这个 partition 的数量（如下所示），当然可以在 topic 创建之后去修改 partition 的数量。 + +```properties +# The default number of log partitions per topic. More partitions allow greater +# parallelism for consumption, but this will also result in more files across +# the brokers. +num.partitions=3 +``` + +在发送一条消息时，可以指定这个消息的 key，producer 根据这个 key 和 partition 机制来判断这个消息发送到哪个 partition。partition 机制可以通过指定 producer 的 partition.class 这一参数来指定（即支持自定义），该 class 必须实现 kafka.producer.Partitioner 接口。 + + + +## 三、Broker 保存消息 + +Kafka 集群中的功能分为数据平面和控制平面。控制平面负责管理集群中的所有元数据。数据平面处理我们写入和读取 Kafka 的实际数据。 + +![kafka-manages-data-and-metadata-separately](https://images.ctfassets.net/gt6dp23g0g38/6dIHZmyFufygLqoOZl9NK8/568033253444bede095afcea83924c44/Kafka_Internals_015.png) + + + +broker 内部 + +![inside-the-apache-kafka-broker](https://images.ctfassets.net/gt6dp23g0g38/39R8M25VXtbor8PP0Uv5Zh/1ed96b8c15b8c8ddf81f1e0ab02e5b77/Kafka_Internals_016.png) + +### 客户端请求的两种类型：生产请求和获取请求 + +客户端请求分为两类：生产请求和获取请求。生产请求是请求将一批数据写入指定的主题。获取请求是请求从 Kafka 主题中获取数据。这两种请求都经历许多相同的步骤。我们将首先了解生产请求的流程，然后再看看获取请求有何不同。 + +#### 生产请求流程 + +1. **记录分区**：生产者使用分区器决定数据写入哪个分区。 +2. **批次积累**：为了提高效率，生产者会将数据累积到批次中。 +3. **发送请求**：当批次满足时间或大小要求时，生产者将数据批次发送给分区的 Leader Broker。 +4. **确认机制**：生产者根据配置的确认机制（acks）等待 Broker 的响应。 + +#### 获取请求流程 + +1. **发送获取请求**：消费者指定主题、分区和偏移量，发送获取请求。 +2. **请求处理**：获取请求进入 Broker 的网络线程，被放入请求队列。 +3. **数据读取**：I/O 线程根据请求中的偏移量从日志文件中读取数据。 +4. **批量响应**：为了提高效率，消费者可以配置等待最小数据量或最大等待时间来返回响应。 + + + +#### 3.1 存储方式 + +物理上把 topic 分成一个或多个 partition（对应 server.properties 中的 num.partitions=3 配置），每个 partition 物理上对应一个文件夹（该文件夹存储该 patition 的所有消息和索引文件）。 + +![kafka-physical-storage](https://images.ctfassets.net/gt6dp23g0g38/6BStOsjiQRncUJUEXIeo1s/a38554930ab928132ec2244d53efa149/Kafka_Internals_022.png) + +#### 3.2 存储策略 + +无论消息是否被消费， kafka 都会保留所有消息。有两种策略可以删除旧数据： + +1. 基于时间： `log.retention.hours=168` + +2. 基于大小： `log.retention.bytes=1073741824` 需要注意的是，因为 Kafka 读取特定消息的时间复杂度为 $O(1)$，即与文件大小无关， 所以这里删除过期文件与提高 Kafka 性能无关。 + + + + + +## Kafka Data Replication + +In this module we’ll look at how the data plane handles data replication. Data replication is a critical feature of Kafka that allows it to provide high durability and availability. We enable replication at the topic level. When a new topic is created we can specify, explicitly or through defaults, how many replicas we want. Then each partition of that topic will be replicated that many times. This number is referred to as the replication factor. With a replication factor of N, in general, we can tolerate N-1 failures, without data loss, and while maintaining availability. + +数据平面如何处理数据复制。数据复制是 Kafka 的一个关键特性，它允许 Kafka 提供高持久性和可用性。我们在主题级别启用复制。当创建新主题时，我们可以明确指定或通过默认设置来指定我们想要的副本数量。然后，该主题的每个分区将被复制指定的次数。这个数字被称为复制因子。使用 N 作为复制因子，通常我们可以容忍 N-1 次故障，而不会丢失数据，同时保持可用性。 + +### Leader, Follower, and In-Sync Replica (ISR) List + +![leader-follower-isr-list](https://images.ctfassets.net/gt6dp23g0g38/4Llth82ZvCCBqcHfp7v0lH/60e6f507fdccce263d38b6d285e6b143/Kafka_Internals_030.png) + +一旦主题的所有分区副本被创建，每个分区的一个副本将被指定为领导者副本，持有该副本的代理将成为该分区的领导者。其余的副本将是追随者。生产者将写入领导者副本，追随者将获取数据以与领导者保持同步。消费者通常也从领导者副本获取数据，但它们可以配置为从追随者获取。 + +分区领导者以及所有与领导者同步的追随者将构成同步副本集（ISR）。在理想情况下，所有副本都将是 ISR 的一部分。 + +> Once the replicas for all the partitions in a topic are created, one replica of each partition will be designated as the leader replica and the broker that holds that replica will be the leader for that partition. The remaining replicas will be followers. Producers will write to the leader replica and the followers will fetch the data in order to keep in sync with the leader. Consumers also, generally, fetch from the leader replica, but they can be configured to fetch from followers. +> +> The partition leader, along with all of the followers that have caught up with the leader, will be part of the in-sync replica set (ISR). In the ideal situation, all of the replicas will be part of the ISR. + +### Leader Epoch + +![leader-epoch](https://images.ctfassets.net/gt6dp23g0g38/1rHc9oqwn8DD94JIQckrXL/a36399e2acc2437810ddaa8a568307ca/Kafka_Internals_031.png) + +每个领导者都与一个独特的、单调递增的数字相关联，称为领导者纪元。纪元用于跟踪当这个副本是领导者时完成的工作，每当选出新领导者时，纪元会增加。领导者纪元对于诸如日志协调等事项非常重要，我们很快就会讨论。 + +> Each leader is associated with a unique, monotonically increasing number called the leader epoch. The epoch is used to keep track of what work was done while this replica was the leader and it will be increased whenever a new leader is elected. The leader epoch is very important for things like log reconciliation, which we’ll discuss shortly. + +### Follower Fetch Request + +![follower-fetch-request](https://images.ctfassets.net/gt6dp23g0g38/QMNcHw9rAoiFGXj4DnP9I/51ef0ec74b91b1f01a88f8fa3934b2f0/Kafka_Internals_032.png) + +每当领导者将其本地日志中的新数据附加时，追随者将向领导者发出获取请求，传入他们需要开始获取的偏移量。 + +> Whenever the leader appends new data into its local log, the followers will issue a fetch request to the leader, passing in the offset at which they need to begin fetching. + +### Follower Fetch Response + +![follower-fetch-response](https://images.ctfassets.net/gt6dp23g0g38/7kr6K36N4VF4D5F3gpY71h/10329b5e4b700afdb1aa28a46432fd44/Kafka_Internals_033.png) + +领导者将从指定偏移量开始的记录响应获取请求。获取响应还将包括每个记录的偏移量和当前领导者纪元。追随者然后将这些记录附加到他们自己的本地日志中 + +> The leader will respond to the fetch request with the records starting at the specified offset. The fetch response will also include the offset for each record and the current leader epoch. The followers will then append those records to their own local logs. + +### Committing Partition Offsets + +![committing-partition-offsets](https://images.ctfassets.net/gt6dp23g0g38/7ADIKF2poAYD0iE1p1hJNF/eff71842eed8637f50d888e27f962343/Kafka_Internals_034.png) + +一旦 ISR 中的所有追随者都获取到了特定的偏移量，到该偏移量为止的记录就被认为是已提交的，并且对消费者可用。这由高水位线指定。 + +领导者通过获取请求中发送的偏移值了解追随者获取的最高偏移量。例如，如果追随者向领导者发送一个获取请求，指定偏移量为 3，领导者知道这个追随者已经提交了所有到偏移量 3 的记录。一旦所有追随者都达到了偏移量 3，领导者将相应地推进高水位线。 + +> Once all of the followers in the ISR have fetched up to a particular offset, the records up to that offset are considered committed and are available for consumers. This is designated by the high watermark. + +> The leader is made aware of the highest offset fetched by the followers through the offset value sent in the fetch requests. For example, if a follower sends a fetch request to the leader that specifies offset 3, the leader knows that this follower has committed all records up to offset 3. Once all of the followers have reached offset 3, the leader will advance the high watermark accordingly. + +### Advancing the Follower High Watermark + +![advancing-the-follower-high-watermark](https://images.ctfassets.net/gt6dp23g0g38/2GtWQTnR5GwuDaUltAxEHM/50dd8e261231d98af4dcae5fc57bc41e/Kafka_Internals_035.png) + +反过来，领导者使用获取响应来通知追随者当前的高水位线。由于这个过程是异步的，追随者的高水位线通常会落后于领导者实际持有的高水位线。 + +> The leader, in turn, uses the fetch response to inform followers of the current high watermark. Because this process is asynchronous, the followers’ high watermark will typically lag behind the actual high watermark held by the leader. + +### Handling Leader Failure + +![handling-leader-failure](https://images.ctfassets.net/gt6dp23g0g38/4gmUY2HRzEEgtWX4aYO5RK/92c1cd987a80a083e0903ab21bb7a6e6/Kafka_Internals_036.png) + +如果领导者失败，或者由于其他原因我们需要选择一个新的领导者，ISR 中的一个代理将被选为新的领导者。领导者选举和通知受影响的追随者的过程由控制平面处理。数据平面重要的是在这个过程中没有数据丢失。这就是为什么新的领导者只能从 ISR 中选择，除非主题特别配置为允许选择不同步的副本。我们知道 ISR 中的所有副本都与最新的提交偏移量保持最新。 + +一旦选出新领导者，领导者纪元将增加，新领导者将开始接受生产请求。 + +> If a leader fails, or if for some other reason we need to choose a new leader, one of the brokers in the ISR will be chosen as the new leader. The process of leader election and notification of affected followers is handled by the control plane. The important thing for the data plane is that no data is lost in the process. That is why a new leader can only be selected from the ISR, unless the topic has been specifically configured to allow replicas that are not in sync to be selected. We know that all of the replicas in the ISR are up to date with the latest committed offset. + +Once a new leader is elected, the leader epoch will be incremented and the new leader will begin accepting produce requests. + +### Temporary Decreased High Watermark + +![temporary-decreased-high-watermark](https://images.ctfassets.net/gt6dp23g0g38/Kr5GipOTKKo9KojdSmREF/a230a16ec35d8887e91cff2ddeb29e00/Kafka_Internals_037.png) + +当新领导者当选时，其高水位线可能小于实际的高水位线。如果发生这种情况，任何偏移量在当前领导者的高水位线和实际高水位线之间的获取请求将触发可重试的 OFFSET_NOT_AVAILABLE 错误。消费者将继续尝试获取，直到高水位线更新，此时处理将正常继续。 + +> When a new leader is elected, its high watermark could be less than the actual high watermark. If this happens, any fetch requests for an offset that is between the current leader’s high watermark and the actual will trigger a retriable OFFSET_NOT_AVAILABLE error. The consumer will continue trying to fetch until the high watermark is updated, at which point processing will continue as normal. + +### Partition Replica Reconciliation + +![partition-replica-reconciliation](https://images.ctfassets.net/gt6dp23g0g38/1MCX2GxiBgktyO7kPgSBGu/f0e8800cd5c4d66ec23243795b5597f5/Kafka_Internals_038.png) + +在新领导者选举后立即，一些副本可能有未提交的记录与新领导者不同步。这就是为什么领导者的高水位线还不是当前的。它不能是，直到它知道每个追随者已经赶上到哪个偏移量。我们不能前进，直到这个问题得到解决。这是通过一个称为副本协调的过程完成的。协调的第一步是在不同步的追随者发送获取请求时开始的。在我们的示例中，请求显示追随者正在获取的偏移量高于其当前纪元的高水位线。 + +> Immediately after a new leader election, it is possible that some replicas may have uncommitted records that are out of sync with the new leader. This is why the leader's high watermark is not current yet. It can’t be until it knows the offset that each follower has caught up to. We can’t move forward until this is resolved. This is done through a process called replica reconciliation. The first step in reconciliation begins when the out-of-sync follower sends a fetch request. In our example, the request shows that the follower is fetching an offset that is higher than the high watermark for its current epoch. + +### Fetch Response Informs Follower of Divergence + +![partition-replica-reconciliation](https://images.ctfassets.net/gt6dp23g0g38/1MCX2GxiBgktyO7kPgSBGu/1087c6850f64d0e30a86f97f8ac6f77a/Kafka_Internals_039.png) + +当领导者收到获取请求时，它将检查自己的日志，并确定所请求的偏移量对于该纪元无效。然后，它将向追随者发送响应，告诉它该纪元应该以哪个偏移量结束。领导者让追随者执行清理工作。 + +> When the leader receives the fetch request it will check it against its own log and determine that the offset being requested is not valid for that epoch. It will then send a response to the follower telling it what offset that epoch should end at. The leader leaves it to the follower to perform the cleanup. + +### Follower Truncates Log to Match Leader Log + +![follower-truncates-log-to-match-leader-log](https://images.ctfassets.net/gt6dp23g0g38/2SLCk6ccSIlkvPjrKq2YCi/492556be995fa402bd06e20cc24a6964/Kafka_Internals_040.png) + +追随者将使用获取响应中的信息来截断多余的数据，以便与领导者同步。 + +The follower will use the information in the fetch response to truncate the extraneous data so that it will be in sync with the leader. + +### Subsequent Fetch with Updated Offset and Epoch + +![subsequent-fetch-with-updated-offset-and-epoch](https://images.ctfassets.net/gt6dp23g0g38/aYcacWtuT1gnS6RCQRxaa/5b2760f264a6fd99b29c16a03d030b03/Kafka_Internals_041.png) + +现在，追随者可以再次发送获取请求，但这次使用正确的偏移量。 + +Now the follower can send that fetch request again, but this time with the correct offset. + +### Follower 102 Reconciled + +![follower-102-reconciled](https://images.ctfassets.net/gt6dp23g0g38/5rtqIUVT29SGB5vharEdcE/ed09090d7e99d57752e001dfc8758d56/Kafka_Internals_042.png) + +然后，领导者将响应从该偏移量开始的新记录，包括新的领导者纪元。 + +The leader will then respond with the new records since that offset includes the new leader epoch. + +### Follower 102 Acknowledges New Records + +![follower-102-acknowledges-new-records](https://images.ctfassets.net/gt6dp23g0g38/10rKaHrv3sJxZ9CxmYwL9w/b3a8d28f7b99a4b8cee79fe9a1b7697e/Kafka_Internals_043.png) + +当追随者再次获取时，它传递的偏移量将告知领导者它已经赶上，领导者将能够增加高水位线。此时，领导者和追随者已完全协调，但我们仍然处于副本不足的状态，因为并非所有副本都在 ISR 中。根据配置，我们可以在此状态下操作，但这当然不理想。 + +When the follower fetches again, the offset that it passes will inform the leader that it has caught up and the leader will be able to increase the high watermark. At this point the leader and follower are fully reconciled, but we are still in an under replicated state because not all of the replicas are in the ISR. Depending on configuration, we can operate in this state, but it’s certainly not ideal. + +### Follower 101 Rejoins the Cluster + +![follower-101-rejoins-the-cluster](https://images.ctfassets.net/gt6dp23g0g38/slWIzdKdFUuiEwK1BAG4E/cd88e602396e4bcdacad99487ea4387f/Kafka_Internals_044.png) + +过一段时间，希望不久，失败的副本代理将重新上线。然后，它将经历我们刚刚描述的相同协调过程。一旦完成协调并赶上新领导者，它将被重新添加到 ISR，我们将回到快乐的地方。 + +At some point, hopefully soon, the failed replica broker will come back online. It will then go through the same reconciliation process that we just described. Once it is done reconciling and is caught up with the new leader, it will be added back to the ISR and we will be back in our happy place. + +### Handling Failed or Slow Followers + +![handling-failed-or-slow-followers](https://images.ctfassets.net/gt6dp23g0g38/1T8pQOyW8YcUj64phgAYW5/c550c246504f246faab3f172f2f073a0/Kafka_Internals_045.png) + +显然，当领导者失败时，这是一个更大的问题，但我们也需要处理追随者失败以及运行缓慢的追随者。领导者监视其追随者的进度。如果自追随者上次完全赶上以来，经过了可配置的时间量，领导者将从同步副本集中移除该追随者。这允许领导者推进高水位线，以便消费者可以继续消费当前数据。如果追随者重新上线或以其他方式赶上领导者，那么它将被重新添加到 ISR。 + +Obviously when a leader fails, it’s a bigger deal, but we also need to handle follower failures as well as followers that are running slow. The leader monitors the progress of its followers. If a configurable amount of time elapses since a follower was last fully caught up, the leader will remove that follower from the in-sync replica set. This allows the leader to advance the high watermark so that consumers can continue consuming current data. If the follower comes back online or otherwise gets its act together and catches up to the leader, then it will be added back to the ISR. + +### Partition Leader Balancing + +![partition-leader-balancing](https://images.ctfassets.net/gt6dp23g0g38/6P0oOJdQ8gJkU0ib014amg/3074980c72714d158fea435866283388/Kafka_Internals_046.png) + +正如我们所看到的，包含领导者副本的代理比追随者副本做了更多的工作。因此，最好不要在单个代理上拥有不成比例的领导者副本数量。为了防止这种情况，Kafka 有首选副本的概念。创建主题时，每个分区的第一个副本被指定为首选副本。由于 Kafka 已经努力在可用代理之间均匀分配分区，这将通常导致领导者的良好平衡。 + +由于各种原因进行领导者选举时，领导者可能会出现在非首选副本上，这可能导致不平衡。因此，Kafka 会定期检查领导者副本是否不平衡。它使用可配置的阈值来确定这一点。如果确实发现不平衡，它将执行领导者重新平衡，以使领导者回到其首选副本上。 + +As we’ve seen, the broker containing the leader replica does a bit more work than the follower replicas. Because of this it’s best not to have a disproportionate number of leader replicas on a single broker. To prevent this Kafka has the concept of a preferred replica. When a topic is created, the first replica for each partition is designated as the preferred replica. Since Kafka is already making an effort to evenly distribute partitions across the available brokers, this will usually result in a good balance of leaders. + +As leader elections occur for various reasons, the leaders might end up on non-preferred replicas and this could lead to an imbalance. So, Kafka will periodically check to see if there is an imbalance in leader replicas. It uses a configurable threshold to make this determination. If it does find an imbalance it will perform a leader rebalance to get the leaders back on their preferred replicas. + + + +## 一、Kafka 文件存储机制 + +### 消息存储原理 + +由于生产者生产的消息会不断追加到 log 文件末尾，为防止 log 文件过大导致数据定位效率低下，Kafka 采取了**分片**和**索引**机制，将每个 partition 分为多个 segment。每个 segment 对应两个文件——`.index文件`和 `.log文件`。这些文件位于一个文件夹下，该文件夹的命名规则为：`topic名称+分区序号`。 + +如下，我们创建一个只有一个分区一个副本的 topic + +```sh +> bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic starfish +``` + +![](https://img.starfish.ink/mq/kafka-topic-create.png) + +然后可以在 kafka-logs 目录（server.properties 默认配置）下看到会有个名为 starfish-0 的文件夹。如果，starfish 这个 topic 有三个分区，则其对应的文件夹为 starfish-0，starfish-1，starfish-2。 + +![](https://img.starfish.ink/mq/kafka-topic-partition.png) + +这些文件的含义如下： + +| 类别 | 作用 | +| :---------------------- | :----------------------------------------------------------- | +| .index | 基于偏移量的索引文件，存放着消息的offset和其对应的物理位置，是**稀松索引** | +| .timestamp | 时间戳索引文件 | +| .log | 它是segment文件的数据文件，用于存储实际的消息。该文件是二进制格式的。log文件是存储在 ConcurrentSkipListMap 里的，是一个map结构，key是文件名（offset），value是内容，这样在查找指定偏移量的消息时，用二分查找法就能快速定位到消息所在的数据文件和索引文件 | +| .snaphot | 快照文件 | +| leader-epoch-checkpoint | 保存了每一任leader开始写入消息时的offset，会定时更新。 follower被选为leader时会根据这个确定哪些消息可用 | + +index 和 log 文件以当前 segment 的第一条消息的 offset 命名。偏移量 offset 是一个 64 位的长整形数，固定是 20 位数字，长度未达到，用 0 进行填补，索引文件和日志文件都由此作为文件名命名规则。所以从上图可以看出，我们的偏移量是从 0 开始的，`.index` 和 `.log` 文件名称都为 `00000000000000000000`。 + +接着往 topic 中发送一些消息，并启动消费者消费 + +``` +> bin /kafka-console-producer.sh --bootstrap-server localhost:9092 --topic starfish +one +``` + +``` +> bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic starfish --from-beginning +one +``` + +![](https://img.starfish.ink/mq/kafka-console-producer.png) + +查看 .log 文件下是否有数据 one + +![](https://img.starfish.ink/mq/0000log.png) + +内容存在一些”乱码“，因为数据是经过序列化压缩的。 + +那么数据文件 .log 大小有限制吗，能保存多久时间？这些我们都可以通过 Kafka 目录下 `conf/server.properties` 配置文件修改： + +```properties +# log文件存储时间，单位为小时，这里设置为1周 +log.retention.hours=168 + +# log文件大小的最大值，这里为1g，超过这个值，则会创建新的segment（也就是新的.index和.log文件） +log.segment.bytes=1073741824 +``` + +比如，当生产者生产数据量较多，一个 segment 存储不下触发分片时，在日志 topic 目录下你会看到类似如下所示的文件： + +``` +00000000000000000000.index +00000000000000000000.log +00000000000000170410.index +00000000000000170410.log +00000000000000239430.index +00000000000000239430.log +``` + +- 每个分区是由多个 Segment 组成，当 Kafka 要写数据到一个 partition 时，它会写入到状态为 active 的 segment 中。如果该 segment 被写满，则一个新的 segment 将会被新建，然后变成新的“active” segment +- 偏移量：分区中的每一条消息都会被分配的一个连续的 id 值，该值用于唯一标识分区中的每一条消息 +- 每个 Segment 中则保存了真实的消息数据。每个 Segment 对应于一个索引文件与一个日志文件。Segment 文件的生命周期是由 Kafka Server 的配置参数所决定的。比如说，`server.properties` 文件中的参数项 `log.retention.hours=168` 就表示 7 天后删除老的消息文件 +- [稀松索引]：稀松索引可以加快速度，因为 index 不是为每条消息都存一条索引信息，而是每隔几条数据才存一条 index 信息，这样 index 文件其实很小。kafka 在写入日志文件的时候，同时会写索引文件（.index和.timeindex）。默认情况下，有个参数 `log.index.interval.bytes` 限定了在日志文件写入多少数据，就要在索引文件写一条索引，默认是 4KB，写 4kb 的数据然后在索引里写一条索引。 + +举个栗子：00000000000000170410 的 “.index” 文件和 “.log” 文件的对应的关系，如下图![](/Users/starfish/oceanus/picBed/kafka/kafka-segment-log.png) + +> 问：为什么不能以 partition 作为存储单位？还要加个 segment？ +> +> 答：如果就以 partition 为最小存储单位，可以想象，当 Kafka producer 不断发送消息，必然会引起 partition 文件的无限扩张，将对消息文件的维护以及已消费的消息的清理带来严重的影响，因此，需以 segment 为单位将 partition 进一步细分。每个 partition（目录）相当于一个巨型文件被平均分配到多个大小相等的 segment（段）数据文件中（每个 segment 文件中消息数量不一定相等）这种特性也方便 old segment 的删除，即方便已被消费的消息的清理，提高磁盘的利用率。每个 partition 只需要支持顺序读写就行，segment 的文件生命周期由服务端配置参数（log.segment.bytes，log.roll.{ms,hours} 等若干参数）决定。 +> +> +> +> 问：segment 的工作原理是怎样的？ +> +> 答：segment 文件由两部分组成，分别为 “.index” 文件和 “.log” 文件，分别表示为 segment 索引文件和数据文件。这两个文件的命令规则为：partition 全局的第一个 segment 从 0 开始，后续每个 segment 文件名为上一个 segment 文件最后一条消息的 offset 值，数值大小为 64 位，20 位数字字符长度，没有数字用 0 填充 + + + +消费者从分配到的分区中查找数据过程大概是这样的： + +1. **定位段文件**：消费者根据要读取的消息的偏移量查找对应的段文件 +2. **使用索引文件查找物理位置**：当消费者请求某个偏移量的消息时，Kafka 会在索引文件中使用二分查找算法快速定位到包含该偏移量消息的日志段 +3. **顺序读取数据文件**：一旦找到消息的物理位置，消费者从段文件的对应位置开始顺序读取消息。顺序读取比随机读取更高效，因为它避免了磁盘的寻道时间。 + +这套机制是建立在 offset 是有序的。索引文件被映射到内存中，所以查找的速度还是很快的。 + +一句话，Kafka 的 Message 存储采用了分区(partition)，分段(LogSegment)和稀疏索引这几个手段来达到了高效性。 + + + +## 二、Kafka 生产过程 + +Kafka 生产者用于生产消息。通过前面的内容我们知道，Kafka 的 topic 可以有多个分区，那么生产者如何将这些数据可靠地发送到这些分区？生产者发送数据的不同的分区的依据是什么？ + +### 2.1 写入流程 + +producer 写入消息流程如下： + +![img](https://i0.wp.com/belowthemalt.com/wp-content/uploads/2020/11/kafkaproducerapi.png?resize=618%2C334&ssl=1) + +![How to Make Kafka Producer/Consumer Production-Ready | by Shivanshu Goyal | The Startup | Medium](https://miro.medium.com/v2/resize:fit:1200/1*RQlk_aKjeTlwJn68llhX7A.png) + + + +### 2.2 写入方式 + +producer 采用推（push） 模式将消息发布到 broker，每条消息都被追加（append） 到分区（patition） 中，属于顺序写磁盘（顺序写磁盘效率比随机写内存要高，保障 kafka 吞吐率）。 + +### 2.3 分区（Partition） + +消息发送时都被发送到一个 topic，其本质就是一个目录，而 topic 是由一些 Partition Logs(分区日志)组成 + +**分区的原因：** + +1. **方便在集群中扩展**，每个 Partition 可以通过调整以适应它所在的机器，而一个 topic 又可以有多个 Partition 组成，因此整个集群就可以适应任意大小的数据了； + +2. **可以提高并发**，因为可以以 Partition 为单位读写了。 + +**分区的原则：** + +我们需要将 producer 发送的数据封装成一个 ProducerRecord 对象。 + +```java +public ProducerRecord (String topic, Integer partition, Long timestamp, K key, V value, Iterable
headers) +public ProducerRecord (String topic, Integer partition, Long timestamp, K key, V value) +public ProducerRecord (String topic, Integer partition, K key, V value, Iterable
headers) +public ProducerRecord (String topic, Integer partition, K key, V value) +public ProducerRecord (String topic, K key, V value) +public ProducerRecord (String topic, V value) +``` + +1. 指明 partition 的情况下，直接将指明的值直接作为 partiton 值； +2. 没有指明 partition 值但有 key 的情况下，将 key 的 hash 值与 topic 的 partition 数进行取余得到 partition 值； +3. 既没有 partition 值又没有 key 值的情况下，第一次调用时随机生成一个整数（后面每次调用在这个整数上自增），将这个值与 topic 可用的 partition 总数取余得到 partition 值，也就是常说的 round-robin 算法。 + + + + + +### 2.4 副本（Replication） + +Kafka 是有主题概念的，而每个主题又进一步划分成若干个分区。副本的概念实际上是在分区层级下定义的，每个分区配置有若干个副本。 + +**所谓副本（Replica），本质就是一个只能追加写消息的提交日志**。根据 Kafka 副本机制的定义，同一个分区下的所有副本保存有相同的消息序列，这些副本分散保存在不同的 Broker 上，从而能够对抗部分 Broker 宕机带来的数据不可用。 + +同一个 partition 可能会有多个 replication（ 对应 server.properties 配置中的 `default.replication.factor=N`）。没有 replication 的情况下，一旦 broker 宕机，其上所有 partition 的数据都不可被消费，同时 producer 也不能再将数据存于其上的 patition。引入 replication 之后，同一个 partition 可能会有多个 replication，而这时需要在这些 replication 之间选出一 个 leader， producer 和 consumer 只与这个 leader 交互，其它 replication 作为 follower 从 leader 中复制数据。 + +为了提高消息的可靠性，Kafka 每个 topic 的 partition 有 N 个副本（replicas），其中 N（大于等于 1）是 topic 的复制因子（replica fator）的个数。**Kafka 通过多副本机制实现故障自动转移**，当 Kafka 集群中出现 broker 失效时，副本机制可保证服务可用。对于任何一个 partition，它的 N 个 replicas 中，其中一个 replica 为 leader，其他都为 follower，leader 负责处理 partition 的所有读写请求，follower 则负责被动地去复制 leader 上的数据。如下图所示，Kafka 集群中有 4 个 broker，某 topic 有 3 个 partition，且复制因子即副本个数也为 3： + +![kafka-data-replication](https://images.ctfassets.net/gt6dp23g0g38/HZjoaXOuEc1zteyMcoOww/1ebab125a11552f4e8b8d88e7850f0ad/Kafka_Internals_029.png) + +![partition-leader-balancing](https://images.ctfassets.net/gt6dp23g0g38/6P0oOJdQ8gJkU0ib014amg/3074980c72714d158fea435866283388/Kafka_Internals_046.png) + +如果 leader 所在的 broker 发生故障或宕机，对应 partition 将因无 leader 而不能处理客户端请求，这时副本的作用就体现出来了：一个新 leader 将从 follower 中被选举出来并继续处理客户端的请求。 + + + +#### 2.5 数据可靠性保证 + +一个 partition 有多个副本（replicas），为了提高可靠性，这些副本分散在不同的 broker 上，由于带宽、读写性能、网络延迟等因素，同一时刻，这些副本的状态通常是不一致的：即 followers 与 leader 的状态不一致。 + +为保证 producer 发送的数据，能可靠的发送到指定的 topic，topic 的每个 partition 收到 producer 数据后，都需要向 producer 发送 ack（acknowledgement确认收到），如果 producer 收到 ack，就会进行下一轮的发送，否则重新发送数据。 + +##### a) 副本数据同步策略主要有如下两种 + +| 方案 | 优点 | 缺点 | +| --------------------------- | ------------------------------------------------------ | ----------------------------------------------------- | +| 半数以上完成同步，就发送ack | 延迟低 | 选举新的 leader 时，容忍n台节点的故障，需要2n+1个副本 | +| 全部完成同步，才发送ack | 选举新的 leader 时，容忍n台节点的故障，需要 n+1 个副本 | 延迟高 | + +Kafka 选择了第二种方案，原因如下： + +- 同样为了容忍 n 台节点的故障，第一种方案需要的副本数相对较多，而 Kafka 的每个分区都有大量的数据，第一种方案会造成大量的数据冗余； +- 虽然第二种方案的网络延迟会比较高，但网络延迟对 Kafka 的影响较小。 + +##### b) ISR + +采用第二种方案之后，设想一下情景：leader 收到数据，所有 follower 都开始同步数据，但有一个 follower 挂了，迟迟不能与 leader 保持同步，那 leader 就要一直等下去，直到它完成同步，才能发送 ack，这个问题怎么解决呢？ + +leader 维护了一个动态的 **in-sync replica set**(ISR)，意为和 leader 保持同步的 follower 集合。当 ISR 中的 follower 完成数据的同步之后，leader 就会给 follower 发送 ack。 + +如果 follower 长时间未向 leader 同步数据，则该 follower 将会被踢出 ISR，该时间阈值由 `replica.lag.time.max.ms` 参数设定。当前默认值是 10 秒。这就是说，只要一个 follower 副本落后 Leader 副本的时间不连续超过 10 秒，那么 Kafka 就认为该 Follower 副本与 Leader 是同步的，即使此时 follower 副本中保存的消息明显少于 Leader 副本中的消息。 + +如下这种情况，不管是 follower1 还是 follower2 ，是否有资格在 ISR 中待着，只和同步时间有关，和相差的消息数量无关 + +如果这个同步过程的速度持续慢于 Leader 副本的消息写入速度，那么在 replica.lag.time.max.ms 时间后，此 Follower 副本就会被认为是与 Leader 副本不同步的，因此不能再放入 ISR 中。此时，Kafka 会自动收缩 ISR 集合，将该副本“踢出”ISR。 + +值得注意的是，倘若该副本后面慢慢地追上了 Leader 的进度，那么它是能够重新被加回 ISR 的。这也表明，ISR 是一个动态调整的集合，而非静态不变的。 + +![leader-follower-isr-list](https://images.ctfassets.net/gt6dp23g0g38/4Llth82ZvCCBqcHfp7v0lH/60e6f507fdccce263d38b6d285e6b143/Kafka_Internals_030.png) + +![advancing-the-follower-high-watermark](https://images.ctfassets.net/gt6dp23g0g38/2GtWQTnR5GwuDaUltAxEHM/50dd8e261231d98af4dcae5fc57bc41e/Kafka_Internals_035.png) + +leader 发生故障之后，就会从 ISR 中选举新的 leader。（之前还有另一个参数，0.9 版本之后 `replica.lag.max.messages` 参数被移除了） + +> #### Unclean 领导者选举（Unclean Leader Election） +> +> 既然 ISR 是可以动态调整的，那么自然就可以出现这样的情形：ISR 为空。因为 Leader 副本天然就在 ISR 中，如果 ISR 为空了，就说明 Leader 副本也“挂掉”了，Kafka 需要重新选举一个新的 Leader。可是 ISR 是空，此时该怎么选举新 Leader 呢？ +> +> **Kafka 把所有不在 ISR 中的存活副本都称为非同步副本**。通常来说，非同步副本落后 Leader 太多，因此，如果选择这些副本作为新 Leader，就可能出现数据的丢失。毕竟，这些副本中保存的消息远远落后于老 Leader 中的消息。在 Kafka 中，选举这种副本的过程称为 Unclean 领导者选举。**Broker 端参数 unclean.leader.election.enable 控制是否允许 Unclean 领导者选举**。 +> +> 开启 Unclean 领导者选举可能会造成数据丢失，但好处是，它使得分区 Leader 副本一直存在，不至于停止对外提供服务，因此提升了高可用性。反之，禁止 Unclean 领导者选举的好处在于维护了数据的一致性，避免了消息丢失，但牺牲了高可用性。 +> +> 如果你听说过 CAP 理论的话，你一定知道，一个分布式系统通常只能同时满足一致性（Consistency）、可用性（Availability）、分区容错性（Partition tolerance）中的两个。显然，在这个问题上，Kafka 赋予你选择 C 或 A 的权利。 +> +> 你可以根据你的实际业务场景决定是否开启 Unclean 领导者选举。不过，我强烈建议你**不要**开启它，毕竟我们还可以通过其他的方式来提升高可用性。如果为了这点儿高可用性的改善，牺牲了数据一致性，那就非常不值当了。 + +##### c) ack 应答机制 + +对于某些不太重要的数据，对数据的可靠性要求不是很高，能够容忍数据的少量丢失，所以没必要等 ISR 中的 follower 全部接收成功。 + +所以 Kafka 为用户提供了**三种可靠性级别**，用户根据对可靠性和延迟的要求进行权衡，选择以下的 acks 参数配置 + +- 0：producer 不等待 broker 的 ack，这一操作提供了一个最低的延迟，broker 一接收到还没有写入磁盘就已经返回，当 broker 故障时有可能**丢失数据**； + +- 1：producer 等待 broker 的 ack，partition 的 leader 落盘成功后返回 ack，如果在 follower 同步成功之前 leader 故障，那么将会**丢失数据**； + +- -1（all）：producer 等待 broker 的 ack，partition 的 leader 和 follower 全部落盘成功后才返回 ack。但是 如果在 follower 同步完成后，broker 发送 ack 之前，leader 发生故障，那么就会造成**数据重复**。 + +##### d) 故障处理 + +由于我们并不能保证 Kafka 集群中每时每刻 follower 的长度都和 leader 一致（即数据同步是有时延的），那么当 leader 挂掉选举某个 follower 为新的 leader 的时候（原先挂掉的 leader 恢复了成为了 follower），可能会出现 leader 的数据比 follower 还少的情况。为了解决这种数据量不一致带来的混乱情况，Kafka 提出了以下概念： + +![](https://img.starfish.ink/mq/kafka-leo-hw.png) + +- LEO（Log End Offset）：指的是每个副本最后一个offset； +- HW（High Wather）：指的是消费者能见到的最大的 offset，ISR 队列中最小的 LEO。 + +消费者和 leader 通信时，只能消费 HW 之前的数据，HW 之后的数据对消费者不可见。 + +针对这个规则： + +- **当follower发生故障时**：follower 发生故障后会被临时踢出 ISR，待该 follower 恢复后，follower 会读取本地磁盘记录的上次的 HW，并将 log 文件高于 HW 的部分截取掉，从 HW 开始向 leader 进行同步。等该 follower 的 LEO 大于等于该 Partition 的 HW，即 follower 追上 leader 之后，就可以重新加入 ISR 了。 +- **当leader发生故障时**：leader 发生故障之后，会从 ISR 中选出一个新的 leader，之后，为保证多个副本之间的数据一致性，其余的 follower 会先将各自的 log 文件高于 HW 的部分截掉，然后从新的 leader 同步数据。 + +所以数据一致性并不能保证数据不丢失或者不重复，这是由 ack 控制的。HW 规则只能保证副本之间的数据一致性！ + + + +Kafka 的 ISR 的管理最终都会反馈到 ZooKeeper 节点上，具体位置为： + +``` +/brokers/topics/[topic]/partitions/[partition]/state +``` + +目前，有两个地方会对这个 ZooKeeper 的节点进行维护。 + +1. Controller 来维护：Kafka 集群中的其中一个 Broker 会被选举为 Controller，主要负责 Partition 管理和副本状态管理，也会执行类似于重分配 partition 之类的管理任务。在符合某些特定条件下，Controller 下的 LeaderSelector 会选举新的 leader，ISR 和新的 `leader_epoch` 及 `controller_epoch` 写入 ZooKeeper 的相关节点中。同时发起 LeaderAndIsrRequest 通知所有的 replicas。 +2. leader 来维护：leader 有单独的线程定期检测 ISR 中 follower 是否脱离 ISR，如果发现 ISR 变化，则会将新的 ISR 的信息返回到 ZooKeeper 的相关节点中。 + + + +#### 2.6 Exactly Once 语义 + +将服务器的 ACK 级别设置为 -1，可以保证 Producer 到 Server 之间不会丢失数据，即 At Least Once 语义。相对的，将服务器 ACK 级别设置为 0，可以保证生产者每条消息只会被发送一次，即 At Most Once语义。 + +**At Least Once 可以保证数据不丢失，但是不能保证数据不重复。相对的，At Most Once 可以保证数据不重复，但是不能保证数据不丢失**。但是，对于一些非常重要的信息，比如说交易数据，下游数据消费者要求数据既不重复也不丢失，即 Exactly Once 语义。在 0.11 版本以前的 Kafka，对此是无能为力的，只能保证数据不丢失，再在下游消费者对数据做全局去重。对于多个下游应用的情况，每个都需要单独做全局去重，这就对性能造成了很大的影响。 + +0.11 版本的 Kafka，引入了一项重大特性：**幂等性**。所谓的幂等性就是指 Producer 不论向 Server 发送多少次重复数据。Server 端都会只持久化一条，幂等性结合 At Least Once 语义，就构成了 Kafka 的 Exactly Once 语义，即： **At Least Once + 幂等性 = Exactly Once** + +要启用幂等性，只需要将 Producer 的参数中 `enable.idompotence` 设置为 `true` 即可。Kafka 的幂等性实现其实就是将原来下游需要做的去重放在了数据上游。开启幂等性的 Producer 在初始化的时候会被分配一个 PID，发往同一 Partition 的消息会附带 Sequence Number。而 Broker 端会对 做缓存，当具有相同主键的消息提交时，Broker 只会持久化一条。 + +> 但是 PID 重启就会变化，同时不同的 Partition 也具有不同主键，所以幂等性无法保证跨分区会话的 Exactly Once。 + +即消息可靠性保证有如下三种： + +- 最多一次（at most once）：消息可能会丢失，但绝不会被重复发送。 +- 至少一次（at least once）：消息不会丢失，但有可能被重复发送。 （目前 Kakfa 默认提供的交付可靠性保障） +- 精确一次（exactly once）：消息不会丢失，也不会被重复发送。 + +首先，它只能保证单分区上的幂等性，即一个幂等性 Producer 能够保证某个主题的一个分区上不出现重复消息，它无法实现多个分区的幂等性。其次，它只能实现单会话上的幂等性，不能实现跨会话的幂等性。这里的会话，你可以理解为 Producer 进程的一次运行。当你重启了 Producer 进程之后，这种幂等性保证就丧失了。 + +> 如果我想实现多分区以及多会话上的消息无重复，应该怎么做呢？ +> +> 答案就是事务（transaction）或者依赖事务型 Producer。这也是幂等性 Producer 和事务型 Producer 的最大区别！ + + + + + + + +## 四、Kafka 消费过程 + +**Kafka 消费者采用 pull 拉模式从 broker 中消费数据**。与之相对的 push（推）模式很难适应消费速率不同的消费者，因为消息发送速率是由 broker 决定的。它的目标是尽可能以最快速度传递消息，但是这样很容易造成 consumer 来不及处理消息。而 pull 模式则可以根据 consumer 的消费能力以适当的速率消费消息。 + +pull 模式不足之处是，如果 kafka 没有数据，消费者可能会陷入循环中，一直返回空数据。为了避免这种情况，我们在我们的拉请求中有参数，允许消费者请求在等待数据到达的“长轮询”中进行阻塞（并且可选地等待到给定的字节数，以确保大的传输大小，或者传入等待超时时间）。 + +#### 4.1 消费者组 + +![](https://img.starfish.ink/mq/kafka-consume-group.png) + +**Consumer Group 是 Kafka 提供的可扩展且具有容错性的消费者机制**。 + +消费者是以 consumer group 消费者组的方式工作，由一个或者多个消费者组成一个组， 共同消费一个 topic。每个分区在同一时间只能由 group 中的一个消费者读取，但是多个 group 可以同时消费这个 partition。在图中，有一个由三个消费者组成的 group，有一个消费者读取主题中的两个分区，另外两个分别读取一个分区。某个消费者读取某个分区，也可以叫做某个消费者是某个分区的拥有者。 + +在这种情况下，消费者可以通过水平扩展的方式同时读取大量的消息。另外，如果一个消费者失败了，那么其他的 group 成员会自动负载均衡读取之前失败的消费者读取的分区。 + +**消费者组最为重要的一个功能是实现广播与单播的功能**。一个消费者组可以确保其所订阅的 Topic 的每个分区只能被从属于该消费者组中的唯一一个消费者所消费；如果不同的消费者组订阅了同一个 Topic，那么这些消费者组之间是彼此独立的，不会受到相互的干扰。 + +> 如果我们希望一条消息可以被多个消费者所消费，那么可以将这些消费者放到不同的消费者组中，这实际上就是广播的效果；如果希望一条消息只能被一个消费者所消费，那么可以将这些消费者放到同一个消费者组中，这实际上就是单播的效果。 + +#### 4.2 分区分配策略 + +一个 consumer group 中有多个 consumer，一个 topic 有多个 partition，所以必然会涉及到 partition 的分配问题，即确定哪个 partition 由哪个 consumer 来消费。 + +Kafka 有两种分配策略，一是 RoundRobin，一是 Range（新版本还有Sticky）。 + +##### RoundRobin + +RoundRobin 即轮询的意思，比如现在有一个三个消费者 ConsumerA、ConsumerB 和 ConsumerC 组成的消费者组，同时消费 TopicA 主题消息，TopicA 分为 7 个分区，如果采用 RoundRobin 分配策略，过程如下所示： + +![](https://img.starfish.ink/mq/QQ20200401-145222@2x.png) + +这种轮询的方式应该很好理解。但如果消费者组消费多个主题的多个分区，会发生什么情况呢？比如现在有一个两个消费者 ConsumerA 和 ConsumerB 组成的消费者组，同时消费 TopicA 和 TopicB 主题消息，如果采用RoundRobin 分配策略，过程如下所示： + +![](https://img.starfish.ink/mq/QQ20200401-150317@2x.png) + +> 注：TAP0 表示 TopicA Partition0 分区数据，以此类推。 + +这种情况下，采用 RoundRobin 算法分配，多个主题会被当做一个整体来看，这个整体包含了各自的 Partition，比如在 Kafka-clients 依赖中，与之对应的对象为 `TopicPartition`。接着将这些 `TopicPartition` 根据其哈希值进行排序，排序后采用轮询的方式分配给消费者。 + +但这会带来一个问题：假如上图中的消费者组中，ConsumerA 只订阅了 TopicA 主题，ConsumerB 只订阅了 TopicB 主题，采用 RoundRobin 轮询算法后，可能会出现 ConsumerA 消费了 TopicB 主题分区里的消息，ConsumerB 消费了 TopicA 主题分区里的消息。 + +综上所述，RoundRobin 算法只适用于消费者组中消费者订阅的主题相同的情况。同时会发现，采用 RoundRobin 算法，消费者组里的消费者之间消费的消息个数最多相差 1 个。 + +##### Range + +Kafka 默认采用 Range 分配策略，Range 顾名思义就是按范围划分的意思。 + +比如现在有一个三个消费者 ConsumerA、ConsumerB 和 ConsumerC 组成的消费者组，同时消费 TopicA 主题消息，TopicA 分为 7 个分区，如果采用 Range 分配策略，过程如下所示： + +![](https://img.starfish.ink/mq/QQ20200401-152904@2x.png) + +假如现在有一个两个消费者 ConsumerA 和 ConsumerB 组成的消费者组，同时消费 TopicA 和 TopicB 主题消息，如果采用 Range 分配策略，过程如下所示： + +![](https://img.starfish.ink/mq/QQ20200401-153300@2x.png) + +Range 算法并不会把多个主题分区当成一个整体。 + +从上面的例子我们可以总结出 Range 算法的一个弊端：那就是同一个消费者组内的消费者消费的消息数量相差可能较大。 + +#### 4.3 offset 的维护 + +消费者在消费的过程中需要记录自己消费了多少数据，即消费位置信息。在 Kafka 中，这个位置信息有个专门的术语：位移（Offset）。 + +> 由于 consumer 在消费过程中可能会出现断电宕机等故障，consumer 恢复后，需要从故障前的位置继续消费，所以 consumer 需要实时记录自己消费到了哪个 offset，以便故障恢复后继续消费。 + +看上去该 Offset 就是一个数值而已，其实对于 Consumer Group 而言，它是一组 KV 对，Key 是分区，V 对应 Consumer 消费该分区的最新位移。如果用 Java 来表示的话，你大致可以认为是这样的数据结构，即 Map，其中 TopicPartition 表示一个分区，而 Long 表示位移的类型。当然，Kafka 源码中并不是这样简单的数据结构，而是要比这个复杂得多，不过这并不会妨碍我们对 Group 位移的理解。 + +Kafka 0.9 版本之前，consumer 默认将 offset 保存在 Zookeeper 中，从 0.9 版本开始，consumer 默认将 offset 保存在 Kafka 一个内置的 topic 中，该 topic 为 **_consumer_offsets**。 + +> 将位移保存在 ZooKeeper 外部系统的做法，最显而易见的好处就是减少了 Kafka Broker 端的状态保存开销。现在比较流行的做法是将服务器节点做成无状态的，这样可以自由地扩缩容，实现超强的伸缩性。Kafka 最开始也是基于这样的考虑，才将 Consumer Group 位移保存在独立于 Kafka 集群之外的框架中。 +> +> 不过，慢慢地人们发现了一个问题，即 ZooKeeper 这类元框架其实并不适合进行频繁的写更新，而 Consumer Group 的位移更新却是一个非常频繁的操作。这种大吞吐量的写操作会极大地拖慢 ZooKeeper 集群的性能，因此 Kafka 社区渐渐有了这样的共识：将 Consumer 位移保存在 ZooKeeper 中是不合适的做法。 + +```shell +> bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic starfish --from-beginning +one +``` + +消费 topic 后，查看 kafka-logs 目录，会发现多出 50 个分区。 + +默认情况下__consumer_offsets 有 50 个分区，如果你的系统中 consumer group 也很多的话，那么这个命令的输出结果会很多，具体放置在哪个分区，根据 groupID 做如下计算得到： + +``` +Math.abs(groupID.hashCode()) % numPartitions +``` + +![](https://img.starfish.ink/mq/kafka-consumer-offset.png) + + + +#### 4.4 再均衡 Rebalance + +所谓的再平衡，指的是在 kafka consumer 所订阅的 topic 发生变化时发生的一种分区重分配机制。 + +**Rebalance 本质上是一种协议，规定了一个 Consumer Group 下的所有 Consumer 如何达成一致，来分配订阅 Topic 的每个分区**。 + +一般有三种情况会触发再平衡： + +- 组成员数发生变更：consumer group 中的新增或删除某个 consumer，导致其所消费的分区需要分配到组内其他的 consumer上； +- 订阅主题发生变更：consumer 订阅的 topic 发生变化，比如订阅的 topic 采用的是正则表达式的形式，如 `test-*` 此时如果有一个新建了一个 topic `test-user`，那么这个 topic 的所有分区也是会自动分配给当前的 consumer 的，此时就会发生再平衡； +- 订阅主题的分区数发生变更：consumer 所订阅的 topic 发生了新增分区的行为，那么新增的分区就会分配给当前的 consumer，此时就会触发再平衡。 + + + +Rebalance 发生时，Group 下所有的 Consumer 实例都会协调在一起共同参与。那每个 Consumer 实例怎么知道应该消费订阅主题的哪些分区呢？这就需要分配策略的协助了。 + +Kafka 提供的再平衡策略主要有三种：`Round Robin`，`Range` 和 `Sticky`，默认使用的是 `Range`。这三种分配策略的主要区别在于： + +- `Round Robin`：会采用轮询的方式将当前所有的分区依次分配给所有的 consumer； +- `Range`：首先会计算每个 consumer可以消费的分区个数，然后按照顺序将指定个数范围的分区分配给各个consumer； +- `Sticky`：这种分区策略是最新版本中新增的一种策略，其主要实现了两个目的： + - 将现有的分区尽可能均衡的分配给各个 consumer，存在此目的的原因在于`Round Robin`和`Range`分配策略实际上都会导致某几个 consumer 承载过多的分区，从而导致消费压力不均衡； + - 如果发生再平衡，那么重新分配之后在前一点的基础上会尽力保证当前未宕机的 consumer 所消费的分区不会被分配给其他的 consumer 上； + +> 讲完了 Rebalance，现在我来说说它“遭人恨”的地方。 +> +> 首先，Rebalance 过程对 Consumer Group 消费过程有极大的影响。如果你了解 JVM 的垃圾回收机制，你一定听过万物静止的收集方式，即著名的 stop the world，简称 STW。在 STW 期间，所有应用线程都会停止工作，表现为整个应用程序僵在那边一动不动。Rebalance 过程也和这个类似，在 Rebalance 过程中，所有 Consumer 实例都会停止消费，等待 Rebalance 完成。这是 Rebalance 为人诟病的一个方面。 +> +> 其次，目前 Rebalance 的设计是所有 Consumer 实例共同参与，全部重新分配所有分区。其实更高效的做法是尽量减少分配方案的变动。例如实例 A 之前负责消费分区 1、2、3，那么 Rebalance 之后，如果可能的话，最好还是让实例 A 继续消费分区 1、2、3，而不是被重新分配其他的分区。这样的话，实例 A 连接这些分区所在 Broker 的 TCP 连接就可以继续用，不用重新创建连接其他 Broker 的 Socket 资源。 +> +> 最后，Rebalance 实在是太慢了。曾经，有个国外用户的 Group 内有几百个 Consumer 实例，成功 Rebalance 一次要几个小时！这完全是不能忍受的。最悲剧的是，目前社区对此无能为力，至少现在还没有特别好的解决方案。所谓“本事大不如不摊上”，也许最好的解决方案就是避免 Rebalance 的发生吧。 + + + + + +#### 2.7 Kafka 事务 + +Kafka 从 0.11 版本开始引入了事务支持。事务可以保证 Kafka 在 Exactly Once 语义的基础上，生产和消费可以跨分区和会话，要么全部成功，要么全部失败。 + +##### 2.6.1 Producer事务 + +为了了实现跨分区跨会话的事务，需要引入一个全局唯一的 TransactionID，并将 Producer 获得的 PID 和Transaction ID 绑定。这样当 Producer 重启后就可以通过正在进行的 TransactionID 获得原来的 PID。 + +为了管理 Transaction，Kafka 引入了一个新的组件 Transaction Coordinator。Producer 就是通过和 Transaction Coordinator 交互获得 Transaction ID 对应的任务状态。Transaction Coordinator 还负责将事务所有写入 Kafka 的一个内部 Topic，这样即使整个服务重启，由于事务状态得到保存，进行中的事务状态可以得到恢复，从而继续进行。 + +设置事务型 Producer 的方法也很简单，满足两个要求即可： + +- 和幂等性 Producer 一样，开启 enable.idempotence = true。 +- 设置 Producer 端参数 transctional. id。最好为其设置一个有意义的名字。 + +此外，你还需要在 Producer 代码中做一些调整，如这段代码所示： + +```java +producer.initTransactions(); +try { + producer.beginTransaction(); + producer.send(record1); + producer.send(record2); + producer.commitTransaction(); +} catch (KafkaException e) { + producer.abortTransaction(); +} +``` + + + +##### 2.6.2 Consumer事务 + +对 Consumer 而言，事务的保证就会相对较弱，尤其是无法保证 Commit 的消息被准确消费。这是由于Consumer 可以通过 offset 访问任意信息，而且不同的 SegmentFile 生命周期不同，同一事务的消息可能会出现重启后被删除的情况。 + +> 在 Consumer 端，读取事务型 Producer 发送的消息也是需要一些变更的。修改起来也很简单，设置 isolation.level 参数的值即可。当前这个参数有两个取值： +> +> 1. read_uncommitted：这是默认值，表明 Consumer 能够读取到 Kafka 写入的任何消息，不论事务型 Producer 提交事务还是终止事务，其写入的消息都可以读取。很显然，如果你用了事务型 Producer，那么对应的 Consumer 就不要使用这个值。 +> 2. read_committed：表明 Consumer 只会读取事务型 Producer 成功提交事务写入的消息。当然了，它也能看到非事务型 Producer 写入的所有消息。 + + + +## 参考与来源： + +- https://developer.confluent.io/courses/architecture/get-started/ 文章配图均来自该教程 + +- 尚硅谷Kafka教学 + +- 部分图片来源：mrbird.cc + +- https://gitbook.cn/books/5ae1e77197c22f130e67ec4e/index.html \ No newline at end of file diff --git "a/docs/distribution/message-queue/Kafka/Kafka\350\243\205\346\207\202\346\214\207\345\215\227.md" "b/docs/distribution/message-queue/Kafka/Kafka\350\243\205\346\207\202\346\214\207\345\215\227.md" new file mode 100644 index 0000000000..f06fc14bc3 --- /dev/null +++ "b/docs/distribution/message-queue/Kafka/Kafka\350\243\205\346\207\202\346\214\207\345\215\227.md" @@ -0,0 +1,17 @@ +https://www.iteblog.com/archives/2605.html + + + + + + + + + + + + + + + + \ No newline at end of file diff --git "a/docs/distribution/message-queue/Kafka/Kafka\351\200\232\344\277\241.md" "b/docs/distribution/message-queue/Kafka/Kafka\351\200\232\344\277\241.md" new file mode 100644 index 0000000000..3c0aff7010 --- /dev/null +++ "b/docs/distribution/message-queue/Kafka/Kafka\351\200\232\344\277\241.md" @@ -0,0 +1,11 @@ +Apache Kafka 的所有通信都是基于 TCP 的，而不是基于 HTTP 或其他协议。无论是生产者、消费者，还是 Broker 之间的通信都是如此。你可能会问，为什么 Kafka 不使用 HTTP 作为底层的通信协议呢？其实这里面的原因有很多，但最主要的原因在于 TCP 和 HTTP 之间的区别。 + +从社区的角度来看，在开发客户端时，人们能够利用 TCP 本身提供的一些高级功能，比如多路复用请求以及同时轮询多个连接的能力。 + +所谓的多路复用请求，即 multiplexing request，是指将两个或多个数据流合并到底层单一物理连接中的过程。TCP 的多路复用请求会在一条物理连接上创建若干个虚拟连接，每个虚拟连接负责流转各自对应的数据流。其实严格来说，TCP 并不能多路复用，它只是提供可靠的消息交付语义保证，比如自动重传丢失的报文。 + +更严谨地说，作为一个基于报文的协议，TCP 能够被用于多路复用连接场景的前提是，上层的应用协议（比如 HTTP）允许发送多条消息。不过，我们今天并不是要详细讨论 TCP 原理，因此你只需要知道这是社区采用 TCP 的理由之一就行了。 + +除了 TCP 提供的这些高级功能有可能被 Kafka 客户端的开发人员使用之外，社区还发现，目前已知的 HTTP 库在很多编程语言中都略显简陋。 + +基于这两个原因，Kafka 社区决定采用 TCP 协议作为所有请求通信的底层协议。 \ No newline at end of file diff --git "a/docs/distribution/message-queue/Kafka/Kafka\351\253\230\345\217\257\347\224\250.md" "b/docs/distribution/message-queue/Kafka/Kafka\351\253\230\345\217\257\347\224\250.md" new file mode 100755 index 0000000000..c369f3f3d3 --- /dev/null +++ "b/docs/distribution/message-queue/Kafka/Kafka\351\253\230\345\217\257\347\224\250.md" @@ -0,0 +1,158 @@ +> Kafka 的高可用其实包含很多个方面， + +相信大家在工作中都用过消息队列，特别是 Kafka 使用得更是普遍，业务工程师在使用 Kafka 的时候除了担忧 Kafka 服务端宕机外，其实最怕如下这样两件事。 + +- 消息丢失。下游系统没收到上游系统发送的消息，造成系统间数据不一致。比如，订单系统没有把成功状态的订单消息成功发送到消息队列里，造成下游的统计系统没有收到下单成功订单的消息，于是造成系统间数据的不一致，从而引起用户查看个人订单列表时跟实际不相符的问题。 + +- 消息重复。相同的消息重复发送会造成消费者消费两次同样的消息，这同样会造成系统间数据的不一致。比如，订单支付成功后会通过消息队列给支付系统发送需要扣款的金额，如果消息发送两次一样的扣款消息，而订单只支付了一次，就会给用户带来余额多扣款的问题。 + +总结来说，这两个问题直接影响到业务系统间的数据一致性。那到底该如何避免这两个问题的发生呢？Kafka 针对这两个问题有系统的解决方案，需要服务端、客户端做相应的配置以及采取一些补偿方案。 + +因此，下面我会从生产端、服务端、消费端三个角度讲解 Kafka 是如何做到消息不丢失或消息不重复的。当然，在这个过程中，为了有利于你更好的理解，在介绍的过程中我也会简单介绍一些 Kafka 的工作原理。 + +三种消息语义及场景 +首先我要介绍一下“消息语义”的概念，这是理论基础，会有利于你更好地抓住下面解决方案的要点。 + +消息语义有三种，分别是：消息最多传递一次、消息最少传递一次、消息有且仅有一次传递，这三种语义分别对应：消息不重复、消息不丢失、消息既不丢失也不重复。 + +这里的“消息传递一次”是指生产者生产消息成功，Broker 接收和保存消息成功，消费者消费消息成功。对一个消息来说，这三个要同时满足才算是“消息传递一次”。上面所说的那三种消息语义可梳理为如下。 + +最多一次（At most once）：对应消息不重复。消息最多传递一次，消息有可能会丢，但不会重复。一般运用于高并发量、高吞吐，但是对于消息的丢失不是很敏感的场景。 + +最少一次（At least once）：对应消息不丢失。消息最少传递一次，消息不会丢，但有可能重复。一般用于并发量一般，对于消息重复传递不敏感的场景。 + +有且仅有一次（Exactly once）：每条消息只会被传递一次，消息不会丢失，也不会重复。 用于对消息可靠性要求高，且对吞吐量要求不高的场景。 + +为便于你更好地对比理解和记忆，我汇总了如下一张表格： + +![Kafka（丢、重）表1.png](https://s0.lgstatic.com/i/image6/M00/4D/0E/CioPOWDtMsqAD1yZAAHIBDQvg_8426.png) + +到这里，三种消息语义的定义和相关特点就介绍完了，接下来我们正式开始分析 Kafka 是如何做到消息不丢或不重的。 + +Kafka 如何做到消息不丢失？ +我们先来讨论一下 Kafka 是如何做到消息不丢失的，也就是：生产者不少生产消息，服务端不丢失消息，消费者也不能少消费消息。 + +那具体要怎么来实现呢？下面我们就来详细讲解下。 + +生产端：不少生产消息 +以下是为了保证消息不丢失，生产端需要配置的参数和相关使用方法。 + +第一个，要**使用带回调方法的 API**，具体 API 方法如下： + +```java +Future send(ProducerRecord record, Callback callback) +``` + +使用带有回调方法的 API 时，我们可以根据回调函数得知消息是否发送成功，如果发送失败了，我们要进行异常处理，比如把失败消息存储到本地硬盘或远程数据库，等应用正常了再发送，这样才能保证消息不丢失。 + +第二个，**设置参数 acks=-1**。acks 这个参数是指有多少分区副本收到消息后，生产者才认为消息发送成功了，可选的参数值有 0、1 和 -1。 + +- acks=0，表示生产者不等待任何服务器节点的响应，只要发送消息就认为成功。 + +- acks=1，表示生产者收到 leader 分区的响应就认为发送成功。 + +- acks=-1，表示只有当 ISR（ISR 的含义后面我会详细介绍）中的副本全部收到消息时，生产者才会认为消息生产成功了。这种配置是最安全的，因为如果 leader 副本挂了，当 follower 副本被选为 leader 副本时，消息也不会丢失。但是系统吞吐量会降低，因为生产者要等待所有副本都收到消息后才能再次发送消息。 + +第三个，**设置参数 retries=3**。参数 retries 表示生产者生产消息的重试次数。这里 retries=3 是一个建议值，一般情况下能满足足够的重试次数就能重试成功。但是如果重试失败了，对异常处理时就可以把消息保存到其他可靠的地方，如磁盘、数据库、远程缓存等，然后等到服务正常了再继续发送消息。 + +第四个，**设置参数 retry.backoff.ms=300**。retry.backoff.ms 指消息生产超时或失败后重试的间隔时间，单位是毫秒。如果重试时间太短，会出现系统还没恢复就开始重试的情况，进而导致再次失败。结合我个人经验来说，300 毫秒还是比较合适的。 + +只要上面这四个要点配置对了，就可以保证生产端的生产者不少生产消息了。 + + + +## 服务端：不丢失消息 + +以下是为了保证服务端不丢消息，服务端需要配置的参数。 + +第一个，**设置 replication.factor >1**。replication.factor 这个参数表示分区副本的个数，这里我们要将其设置为大于 1 的数，这样当 leader 副本挂了，follower 副本还能被选为 leader 副本继续接收消息。 + +第二个，设置 **min.insync.replicas >1**。min.insync.replicas 指的是 ISR 最少的副本数量，原理同上，也需要大于 1 的副本数量来保证消息不丢失。 + +> 这里我简单介绍下 ISR。ISR 是一个分区副本的集合，每个分区都有自己的一个 ISR 集合。但不是所有的副本都会在这个集合里，首先 leader 副本是在 ISR 集合里的，如果一个 follower 副本的消息没落后 leader 副本太长时间，这个 follower 副本也在 ISR 集合里；可是如果有一个 follower 副本落后 leader 副本太长时间，就会从 ISR 集合里被淘汰出去。也就是说，ISR 里的副本数量是小于或等于分区的副本数量的。 + +第三个，设置 **unclean.leader.election.enable = false**。unclean.leader.election.enable 指是否能把非 ISR 集合中的副本选举为 leader 副本。unclean.leader.election.enable = true，也就是说允许非 ISR 集合中的 follower 副本成为 leader 副本。如果设置成这样会有什么问题呢？下面我结合几个示意图来为你详细分析下这个问题。 + +假设 ISR 集合内的 follower1 副本和 ISR 集合外的 follower2 副本向 leader 副本拉取消息（如下图 1），也就是说这时 ISR 集合中就有两个副本，一个是 leader 副本，另一个是 follower1 副本，而 follower2 副本由于网络或自身机器的原因已经落后 leader 副本很长时间，已经被踢出 ISR 集合。 + +![Kafka（丢、重）图片1.png](https://s0.lgstatic.com/i/image6/M01/4D/06/Cgp9HWDtMyqAIRp3AAC3KDxWKbo476.png) + +突然 leader 和 follower1 这两个副本挂了（如图 2所示），会导致什么样的结果出现呢？ + +![Kafka（丢、重）图片2.png](https://s0.lgstatic.com/i/image6/M01/4D/06/Cgp9HWDtMz2ALajqAAC8yslICn0450.png) + +由于 unclean.leader.election.enable = true，而现在分区的副本能正常工作的仅仅剩下 follower2 副本，所以 follower2 最终会被选为新的 leader 副本并继续接收生产者发送的消息，我们可以看到它接收了一个新的消息 5，如下图 3 所示。 + +![Kafka（丢、重）图片3.png](https://s0.lgstatic.com/i/image6/M01/4D/0F/CioPOWDtM1KADzVzAAB9g5r1M10021.png) + +如果这时 follower1 副本的服务恢复，又会发生什么情况呢？由于 follower 副本要拉取 leader 副本同步数据，首先要获取 leader 副本的信息，并感知到现在的 leader 副本的 LEO 比自己的还小，于是做了截断操作，这时 4 这个消息就丢了，这就造成了消息的丢失。 + +![Kafka（丢、重）图片4.png](https://s0.lgstatic.com/i/image6/M01/4D/06/Cgp9HWDtM2aAG2iNAACi7VklTgY867.png) + +因此，我们一定要把 unclean.leader.election.enable 设置为 false，只有这样非 ISR 集合的副本才不会被选为分区的 leader 副本。但是这样做也降低了可用性，因为这个分区的副本没有 leader，就无法收发消息了，但是消息会发送到别的分区 leader 副本，也就是说分区的数量实际上减少了。 + + + +## 消费端：不能少消费消息 + +为了保证不丢失消息，消费者就不能少消费消息，该如何去实现呢？消费端需要做好如下的配置。 + +第一个，设置 **enable.auto.commit=false**。enable.auto.commit 这个参数表示是否自动提交，如果是自动提交会导致什么问题出现呢？ + +消费者消费消息是有两个步骤的，首先拉取消息，然后再处理消息。向服务端提交消息偏移量可以手动提交也可以自动提交。如果把参数 enable.auto.commit 设置为 true 就表示消息偏移量是由消费端自动提交，由异步线程去完成的，业务线程无法控制。如果刚拉取了消息之后，业务处理还没进行完，这时提交了消息偏移量但是消费者却挂了，这就造成还没进行完业务处理的消息的位移被提交了，下次再消费就消费不到这些消息，造成消息的丢失。因此，一定要设置 enable.auto.commit=false，也就是手动提交消息偏移量。 + +第二个，要有手动提交偏移量的正确步骤。enable.auto.commit=false 并不能完全满足消费端消息不丢的条件，还要有正确的手动提交偏移量的过程。具体如何操作呢？这里我们同样结合一个示意图来讲解，如下所示： + +![Kafka（丢、重）图片5.png](https://s0.lgstatic.com/i/image6/M00/4D/0A/Cgp9HWDtPAGAdtfZABiBhef99ik312.png) + +这幅图表示业务逻辑先对消息进行处理，再提交 offset，这样是能够保证不少消费消息的。但是你可以想象这样一个场景：如果消费者在处理完消息后、提交 offset 前出现宕机，待消费者再上线时，还会处理未提交的那部分消息（图中对应 2~7 这部分消息），但是这部分已经被消费者处理过了，也就是说这样做虽然避免了丢消息，但是会有重复消费的情况出现。 + +具体代码需要这么写： + +```java +List messages = consumer.poll()； +processMsg(messages); +consumer.commitOffset(); +``` + + + +## Kafka 如何做到消息不重复？ + +接下来我们讨论 Kafka 又是如何做到消息不重复的，也就是：生产端不重复生产消息，服务端不重复存储消息，消费端也不能重复消费消息。 + +相较上面“消息不丢失”的场景，“消息不重复”的服务端无须做特别的配置，因为服务端不会重复存储消息，如果有重复消息也应该是由生产端重复发送造成的。也就是说，下面我们只需要分析生产端和消费端就行。 + +### 生产端：不重复生产消息 + +生产端发送消息后，服务端已经收到消息了，但是假如遇到网络问题，无法获得响应，生产端就无法判断该消息是否成功提交到了 Kafka，而我们一般会配置重试次数，但这样会引发生产端重新发送同一条消息，从而造成消息重复的发送。 + +对于这个问题，Kafka 0.11.0 的版本之前并没有什么解决方案，不过从 0.11.0 的版本开始，Kafka 给每个生产端生成一个唯一的 ID，并且在每条消息中生成一个 sequence num，sequence num 是递增且唯一的，这样就能对消息去重，达到一个生产端不重复发送一条消息的目的。 + +但是这个方法是有局限性的，只对在一个生产端内生产的消息有效，如果一个消息分别在两个生产端发送就不行了，还是会造成消息的重复发送。好在这种可能性比较小，因为消息的重试一般会在一个生产端内进行。当然，对应一个消息分别在两个生产端发送的请求我们也有方案，只是要多做一些补偿的工作，比如，我们可以为每一个消息分配一个全局 ID，并把全局 ID 存放在远程缓存或关系型数据库里，这样在发送前可以判断一下是否已经发送过了。 + +### 消费端：不能重复消费消息 + +为了保证消息不重复，消费端就不能重复消费消息，该如何去实现呢？消费端需要做好如下配置。 + +**第一步，设置 enable.auto.commit=false**。跟前面一样，这里同样要避免自动提交偏移量。你可以想象这样一种情况，消费端拉取消息和处理消息都完成了，但是自动提交偏移量还没提交消费端却挂了，这时候 Kafka 消费组开始重新平衡并把分区分给另一个消费者，由于偏移量没提交新的消费者会重复拉取消息，这就最终造成重复消费消息。 + +第二步，**单纯配成手动提交同样不能避免重复消费，还需要消费端使用正确的消费“姿势”**。这里还是先看下图这种情况： + +![Kafka（丢、重）图片6.png](https://s0.lgstatic.com/i/image6/M01/4D/0A/Cgp9HWDtPCWAYncVABfKsdCDbq0367.png) + +消费者拉取消息后，先提交 offset 后再处理消息，这样就不会出现重复消费消息的可能。但是你可以想象这样一个场景：在提交 offset 之后、业务逻辑处理消息之前出现了宕机，待消费者重新上线时，就无法读到刚刚已经提交而未处理的这部分消息（这里对应图中 5~8 这部分消息），还是会有少消费消息的情况。 + +具体代码如下： + +```java +List messages = consumer.poll(); +consumer.commitOffset(); +processMsg(messages); +``` + +## 总结 + +这里我也简单总结下这一讲分享的主要内容。首先我们介绍了消息的三个语义及其场景，接下来我们从 Kafka 生产端、服务端和消费端三个方面具体讲解了我们到底该如何配置才能实现消息不丢失以及消息不重复。在这个过程中，我们也同步解释了一些 Kafka 的原理知识，这样你才能知其然并知其所以然。 + +Kafka 中消息不丢失、不重复很重要，就我个人经验来讲，我是公司专门负责消息队列的架构师，业务人员除了担忧消息队列服务端宕机外，对消息的丢失和消息的重复会非常敏感，因为这直接影响到了业务本身。总体来讲，要保证消息不丢失和不重复，你要从生产端、服务端和消费端三个部分全盘考虑才可行，只是单独考虑某一端是远远不够的。同时，我也希望你搞懂消息语义的含义，因为所有的消息队列都会有相应的涉及。 \ No newline at end of file diff --git "a/docs/distribution/message-queue/Kafka/Kafka\351\253\230\346\225\210\350\257\273\345\206\231\346\225\260\346\215\256\347\232\204\345\216\237\345\233\240.md" "b/docs/distribution/message-queue/Kafka/Kafka\351\253\230\346\225\210\350\257\273\345\206\231\346\225\260\346\215\256\347\232\204\345\216\237\345\233\240.md" new file mode 100644 index 0000000000..f15468e716 --- /dev/null +++ "b/docs/distribution/message-queue/Kafka/Kafka\351\253\230\346\225\210\350\257\273\345\206\231\346\225\260\346\215\256\347\232\204\345\216\237\345\233\240.md" @@ -0,0 +1,234 @@ +--- +title: Kafka 为什么能那么快 | Kafka高效读写数据的原因 +date: 2023-02-15 +tags: + - Kafka +categories: Kafka +--- + +![](https://picx.zhimg.com/v2-dbf838a540c98ea0d9bb324d94e339a1_720w.jpg?source=172ae18b) + +无论 kafka 作为 MQ 也好，作为存储层也罢，无非就是两个功能（好简单的样子），一是 Producer 生产的数据存到 broker，二是 Consumer 从 broker 读取数据。那 Kafka 的快也就体现在读写两个方面了，下面我们就聊聊 Kafka 快的原因。 + +![](https://pic4.zhimg.com/80/v2-227bec1fb110b479e704e92d88848497_1440w.webp) + +### 1. 利用 Partition 实现并行处理 + +我们都知道 Kafka 是一个 Pub-Sub 的消息系统，无论是发布还是订阅，都要指定 Topic。 + +Topic 只是一个逻辑的概念。每个 Topic 都包含一个或多个 Partition，不同 Partition 可位于不同节点。 + +一方面，由于不同 Partition 可位于不同机器，因此可以充分利用集群优势，实现机器间的并行处理。另一方面，由于 Partition 在物理上对应一个文件夹，即使多个 Partition 位于同一个节点，也可通过配置让同一节点上的不同 Partition 置于不同的磁盘上，从而实现磁盘间的并行处理，充分发挥多磁盘的优势。 + +能并行处理，速度肯定会有提升，多个工人肯定比一个工人干的快。 + + + +> 可以并行写入不同的磁盘？那磁盘读写的速度可以控制吗？ + +那就先简单扯扯磁盘/IO 的那些事 + +![](https://i02piccdn.sogoucdn.com/758503f7e81e374c) + +>硬盘性能的制约因素是什么？如何根据磁盘I/O特性来进行系统设计？ +> +>硬盘内部主要部件为磁盘盘片、传动手臂、读写磁头和主轴马达。实际数据都是写在盘片上，读写主要是通过传动手臂上的读写磁头来完成。实际运行时，主轴让磁盘盘片转动，然后传动手臂可伸展让读取头在盘片上进行读写操作。磁盘物理结构如下图所示： +> +>![](https://pic2.zhimg.com/80/v2-bbc26468cf46832c18fdbc2b7d0ba6cd_1440w.webp) +> +>由于单一盘片容量有限，一般硬盘都有两张以上的盘片，每个盘片有两面，都可记录信息，所以一张盘片对应着两个磁头。盘片被分为许多扇形的区域，每个区域叫一个扇区。盘片表面上以盘片中心为圆心，不同半径的同心圆称为磁道，不同盘片相同半径的磁道所组成的圆柱称为柱面。磁道与柱面都是表示不同半径的圆，在许多场合，磁道和柱面可以互换使用。磁盘盘片垂直视角如下图所示： +> +>![图片来源：commons.wikimedia.org](https://pic3.zhimg.com/80/v2-5e0ed70f0174e07126e8c477ef6a7812_1440w.webp) +> +>影响磁盘的关键因素是磁盘服务时间，即磁盘完成一个 I/O 请求所花费的时间，它由寻道时间、旋转延迟和数据传输时间三部分构成。 +> +>机械硬盘的连续读写性能很好，但随机读写性能很差，这主要是因为磁头移动到正确的磁道上需要时间，随机读写时，磁头需要不停的移动，时间都浪费在了磁头寻址上，所以性能不高。衡量磁盘的重要主要指标是IOPS 和吞吐量。 +> +>在许多的开源框架如 Kafka、HBase 中，都通过追加写的方式来尽可能的将随机 I/O 转换为顺序 I/O，以此来降低寻址时间和旋转延时，从而最大限度的提高 IOPS。 +> +>感兴趣的同学可以看看 [磁盘I/O那些事](https://tech.meituan.com/2017/05/19/about-desk-io.html "美团——磁盘I/O那些事") + + + +磁盘读写的快慢取决于你怎么使用它，也就是顺序读写或者随机读写。 + +### 2. 顺序写磁盘 + +![图片来源：kafka.apache.org](https://pic3.zhimg.com/80/v2-a8f1c2ea262c67dbbbe0022dedbb992e_1440w.webp) + +**Kafka 中每个分区是一个有序的，不可变的消息序列**，新的消息不断追加到 partition 的末尾，这个就是顺序写。 + +>很久很久以前就有人做过基准测试：《每秒写入2百万（在三台廉价机器上）》http://ifeve.com/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines/ + +由于磁盘有限，不可能保存所有数据，实际上作为消息系统 Kafka 也没必要保存所有数据，需要删除旧的数据。又由于顺序写入的原因，所以 Kafka 采用各种删除策略删除数据的时候，并非通过使用“读 - 写”模式去修改文件，而是将 Partition 分为多个 Segment，每个 Segment 对应一个物理文件，通过删除整个文件的方式去删除 Partition 内的数据。这种方式清除旧数据的方式，也避免了对文件的随机写操作。 + + + +### 3. 充分利用 Page Cache + +>引入 Cache 层的目的是为了提高 Linux 操作系统对磁盘访问的性能。Cache 层在内存中缓存了磁盘上的部分数据。当数据的请求到达时，如果在 Cache 中存在该数据且是最新的，则直接将数据传递给用户程序，免除了对底层磁盘的操作，提高了性能。Cache 层也正是磁盘 IOPS 为什么能突破 200 的主要原因之一。 +> +>在 Linux 的实现中，文件 Cache 分为两个层面，一是 Page Cache，另一个 Buffer Cache，每一个 Page Cache 包含若干 Buffer Cache。Page Cache 主要用来作为文件系统上的文件数据的缓存来用，尤其是针对当进程对文件有 read/write 操作的时候。Buffer Cache 则主要是设计用来在系统对块设备进行读写的时候，对块进行数据缓存的系统来使用。 + +使用 Page Cache 的好处： + +- I/O Scheduler 会将连续的小块写组装成大块的物理写从而提高性能 + +- I/O Scheduler 会尝试将一些写操作重新按顺序排好，从而减少磁盘头的移动时间 + +- 充分利用所有空闲内存（非 JVM 内存）。如果使用应用层 Cache（即 JVM 堆内存），会增加 GC 负担 +- 读操作可直接在 Page Cache 内进行。如果消费和生产速度相当，甚至不需要通过物理磁盘（直接通过 Page Cache）交换数据 +- 如果进程重启，JVM 内的 Cache 会失效，但 Page Cache 仍然可用 + +Broker 收到数据后，写磁盘时只是将数据写入 Page Cache，并不保证数据一定完全写入磁盘。从这一点看，可能会造成机器宕机时，Page Cache 内的数据未写入磁盘从而造成数据丢失。但是这种丢失只发生在机器断电等造成操作系统不工作的场景，而这种场景完全可以由 Kafka 层面的 Replication 机制去解决。如果为了保证这种情况下数据不丢失而强制将 Page Cache 中的数据 Flush 到磁盘，反而会降低性能。也正因如此，Kafka 虽然提供了 `flush.messages` 和 `flush.ms` 两个参数将 Page Cache 中的数据强制 Flush 到磁盘，但是 Kafka 并不建议使用。 + + + +### 4. 零拷贝技术 + +Kafka 中存在大量的网络数据持久化到磁盘（Producer 到 Broker）和磁盘文件通过网络发送（Broker 到 Consumer）的过程。这一过程的性能直接影响 Kafka 的整体吞吐量。 + +> 操作系统的核心是内核，独立于普通的应用程序，可以访问受保护的内存空间，也有访问底层硬件设备的权限。 +> +> 为了避免用户进程直接操作内核，保证内核安全，操作系统将虚拟内存划分为两部分，一部分是内核空间（Kernel-space），一部分是用户空间（User-space）。 + +传统的 Linux 系统中，标准的 I/O 接口（例如read，write）都是基于数据拷贝操作的，即 I/O 操作会导致数据在内核地址空间的缓冲区和用户地址空间的缓冲区之间进行拷贝，所以标准 I/O 也被称作缓存 I/O。这样做的好处是，如果所请求的数据已经存放在内核的高速缓冲存储器中，那么就可以减少实际的 I/O 操作，但坏处就是数据拷贝的过程，会导致 CPU 开销。 + +[我们把 Kafka 的生产和消费简化成如下两个过程来看](https://zhuanlan.zhihu.com/p/78335525 "Kafka零拷贝")： + +1. 网络数据持久化到磁盘 (Producer 到 Broker) +2. 磁盘文件通过网络发送（Broker 到 Consumer） + + + +##### 4.1 网络数据持久化到磁盘 (Producer 到 Broker) + +传统模式下，数据从网络传输到文件需要 4 次数据拷贝、4 次上下文切换和两次系统调用。 + +```java +data = socket.read()// 读取网络数据 +File file = new File() +file.write(data)// 持久化到磁盘 +file.flush() +``` + +这一过程实际上发生了四次数据拷贝： + +1. 首先通过 DMA copy 将网络数据拷贝到内核态 Socket Buffer +2. 然后应用程序将内核态 Buffer 数据读入用户态（CPU copy） +3. 接着用户程序将用户态 Buffer 再拷贝到内核态（CPU copy） +4. 最后通过 DMA copy 将数据拷贝到磁盘文件 + +> DMA（Direct Memory Access）：直接存储器访问。DMA 是一种无需 CPU 的参与，让外设和系统内存之间进行双向数据传输的硬件机制。使用 DMA 可以使系统 CPU 从实际的 I/O 数据传输过程中摆脱出来，从而大大提高系统的吞吐率。 + +同时，还伴随着四次上下文切换，如下图所示 + +![](https://pic2.zhimg.com/80/v2-fdfe29d209918316409200f10cf63ebd_1440w.webp) + +数据落盘通常都是非实时的，kafka 生产者数据持久化也是如此。Kafka 的数据**并不是实时的写入硬盘**，它充分利用了现代操作系统分页存储来利用内存提高 I/O 效率，就是上一节提到的 Page Cache。 + +对于 kafka 来说，Producer 生产的数据存到 broker，这个过程读取到 socket buffer 的网络数据，其实可以直接在内核空间完成落盘。并没有必要将 socket buffer 的网络数据，读取到应用进程缓冲区；在这里应用进程缓冲区其实就是 broker，broker 收到生产者的数据，就是为了持久化。 + +在此`特殊场景`下：接收来自 socket buffer 的网络数据，应用进程不需要中间处理、直接进行持久化时。可以使用 mmap 内存文件映射。 + +>**Memory Mapped Files**：简称 mmap，也有叫 **MMFile** 的，使用 mmap 的目的是将内核中读缓冲区（read buffer）的地址与用户空间的缓冲区（user buffer）进行映射。从而实现内核缓冲区与应用程序内存的共享，省去了将数据从内核读缓冲区（read buffer）拷贝到用户缓冲区（user buffer）的过程。它的工作原理是直接利用操作系统的 Page 来实现文件到物理内存的直接映射。完成映射之后你对物理内存的操作会被同步到硬盘上。 +> +>使用这种方式可以获取很大的 I/O 提升，省去了用户空间到内核空间复制的开销。 + +mmap 也有一个很明显的缺陷——不可靠，写到 mmap 中的数据并没有被真正的写到硬盘，操作系统会在程序主动调用 flush 的时候才把数据真正的写到硬盘。Kafka 提供了一个参数——`producer.type` 来控制是不是主动 flush；如果 Kafka 写入到 mmap 之后就立即 flush 然后再返回 Producer 叫同步(sync)；写入 mmap 之后立即返回 Producer 不调用 flush 就叫异步(async)，默认是 sync。 + +![](https://pic1.zhimg.com/80/v2-7b2d0b80328143322445f55f954144ec_1440w.webp) + +> 零拷贝（Zero-copy）技术指在计算机执行操作时，CPU 不需要先将数据从一个内存区域复制到另一个内存区域，从而可以减少上下文切换以及 CPU 的拷贝时间。 +> +> 它的作用是在数据从网络设备到用户程序空间传递的过程中，减少数据拷贝次数，减少系统调用，实现 CPU 的零参与，彻底消除 CPU 在这方面的负载。 +> +> [目前零拷贝技术主要有三种类型](https://cllc.fun/2020/03/18/linux-zero-copy/ "Linux - Zero-copy（零拷贝）")： +> +> - 直接I/O：数据直接跨过内核，在用户地址空间与I/O设备之间传递，内核只是进行必要的虚拟存储配置等辅助工作； +> - 避免内核和用户空间之间的数据拷贝：当应用程序不需要对数据进行访问时，则可以避免将数据从内核空间拷贝到用户空间 +> - mmap +> - sendfile +> - splice && tee +> - sockmap +> - copy on write：写时拷贝技术，数据不需要提前拷贝，而是当需要修改的时候再进行部分拷贝。 + + + +##### 4.2 磁盘文件通过网络发送（Broker 到 Consumer） + +传统方式实现：先读取磁盘、再用 socket 发送，实际也是进行四次 copy + +```java +buffer = File.read +Socket.send(buffer) +``` + +这一过程可以类比上边的生产消息： + +1. 首先通过系统调用将文件数据读入到内核态 Buffer（DMA 拷贝） +2. 然后应用程序将内存态 Buffer 数据读入到用户态 Buffer（CPU 拷贝） +3. 接着用户程序通过 Socket 发送数据时将用户态 Buffer 数据拷贝到内核态 Buffer（CPU 拷贝） +4. 最后通过 DMA 拷贝将数据拷贝到 NIC Buffer + +Linux 2.4+ 内核通过 sendfile 系统调用，提供了零拷贝。数据通过 DMA 拷贝到内核态 Buffer 后，直接通过 DMA 拷贝到 NIC Buffer，无需 CPU 拷贝。这也是零拷贝这一说法的来源。除了减少数据拷贝外，因为整个读文件 - 网络发送由一个 sendfile 调用完成，整个过程只有两次上下文切换，因此大大提高了性能。 + +![](https://pic4.zhimg.com/80/v2-fb5b1c0a4358a5c7608251c91e6b971b_1440w.webp) + +Kafka 在这里采用的方案是通过 NIO 的 `transferTo/transferFrom` 调用操作系统的 sendfile 实现零拷贝。总共发生 2 次内核数据拷贝、2 次上下文切换和一次系统调用，消除了 CPU 数据拷贝 + + + +### 5. 批处理 + +在很多情况下，系统的瓶颈不是 CPU 或磁盘，而是网络 IO。 + +因此，除了操作系统提供的低级批处理之外，Kafka 的客户端和 broker 还会在通过网络发送数据之前，在一个批处理中累积多条记录 (包括读和写)。记录的批处理分摊了网络往返的开销，使用了更大的数据包从而提高了带宽利用率。 + + + +### 6. 数据压缩 + +Producer 可将数据压缩后发送给 broker，从而减少网络传输代价，目前支持的压缩算法有：Snappy、Gzip、LZ4。数据压缩一般都是和批处理配套使用来作为优化手段的。 + +在 Kafka 中，压缩可能发生在两个地方：生产者端和 Broker 端。 + +> 生产者程序中配置 compression.type 参数即表示启用指定类型的压缩算法。比如下面这段程序代码展示了如何构建一个开启 GZIP 的 Producer 对象： +> +> ```java +> Properties props = new Properties(); +> props.put("bootstrap.servers", "localhost:9092"); +> props.put("acks", "all"); +> props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); +> props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer"); +> // 开启 GZIP 压缩 +> props.put("compression.type", "gzip"); +> +> Producer producer = new KafkaProducer<>(props); +> ``` +> +> 这里比较关键的代码行是 `props.put(“compression.type”, “gzip”)`，它表明该 Producer 的压缩算法使用的是 GZIP。这样 Producer 启动后生产的每个消息集合都是经 GZIP 压缩过的，故而能很好地节省网络传输带宽以及 Kafka Broker 端的磁盘占用。 +> +> 大部分情况下 Broker 从 Producer 端接收到消息后仅仅是原封不动地保存而不会对其进行任何修改，但在以下两种情况，可能会让 Broker 重新压缩消息 +> +> - Broker 端指定了和 Producer 端不同的压缩算法，这样 Broker 收到消息，就需要先解压再用自己的压缩算法进行压缩 +> - Broker 端发生了消息格式转换 + + + +### 小总结 | 下次面试官问我 kafka 为什么快，我就这么说 + +从 3 个方面来看： + +- partition 并行处理 +- 顺序写磁盘，充分利用磁盘特性 +- 利用了现代操作系统分页存储 Page Cache 来利用内存提高 I/O 效率 +- 采用了零拷贝技术 + - Producer 生产的数据持久化到 broker，采用 mmap 文件映射，实现顺序的快速写入 + - Customer 从 broker 读取数据，采用 sendfile，将磁盘文件读到 OS 内核缓冲区后，转到 NIO buffer进行网络发送，减少 CPU 消耗 + + + +## References + +- https://www.infoq.cn/article/kafka-analysis-part-6 \ No newline at end of file diff --git "a/docs/distribution/message-queue/Kafka/Zookeeper\345\234\250Kafka\344\270\255\347\232\204\344\275\234\347\224\250.md" "b/docs/distribution/message-queue/Kafka/Zookeeper\345\234\250Kafka\344\270\255\347\232\204\344\275\234\347\224\250.md" new file mode 100644 index 0000000000..4ea5654706 --- /dev/null +++ "b/docs/distribution/message-queue/Kafka/Zookeeper\345\234\250Kafka\344\270\255\347\232\204\344\275\234\347\224\250.md" @@ -0,0 +1,104 @@ +### 3.6 Zookeeper在Kafka中的作用 + +- **存储结构** + + ![img](file:///Users/starfish/workspace/tech/docs/_images/message-queue/Kafka/zookeeper-store.png?lastModify=1595738386) + + + + 注意： **producer 不在 zk 中注册， 消费者在 zk 中注册。** + + Kafka集群中有一个broker会被选举为Controller，**负责管理集群broker的上线下，所有topic的分区副本分配和leader选举等工作**。 + + Controller的管理工作都是依赖于Zookeeper的。 + + 下图为 partition 的 leader 选举过程： + + ![img](file:///Users/starfish/workspace/tech/docs/_images/message-queue/Kafka/controller-leader.png?lastModify=1595738386) + +- Zookeeper 是一个成熟的分布式协调服务，它可以为分布式服务提供分布式配置服、同步服务和命名注册等能力.。对于任何分布式系统，都需要一种协调任务的方法。Kafka 是使用 ZooKeeper 而构建的分布式系统。但是也有一些其他技术（例如 Elasticsearch 和 MongoDB）具有其自己的内置任务协调机制。 + + Kafka 将 Broker、Topic 和 Partition 的元数据信息存储在 Zookeeper 上。通过在 Zookeeper 上建立相应的数据节点，并监听节点的变化，Kafka 使用 Zookeeper 完成以下功能： + + - Kafka Controller 的 Leader 选举 + - Kafka 集群成员管理 + - Topic 配置管理 + - 分区副本管理 + + 我们看一看 Zookeeper 下 Kafka 创建的节点，即可一目了然的看出这些相关的功能。 + +![图片](https://mmbiz.qpic.cn/mmbiz_png/FbXJ7UCc6O0dHyDpzdia8xZ2nS1IzIMSojuh0sAibj56fWQhUqV4OCkTicOODICJCLfIFibN9Mv2uCZRVhicIibOaVqQ/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +## Controller + +Controller 是从 Broker 中选举出来的，负责分区 Leader 和 Follower 的管理。当某个分区的 leader 副本发生故障时，由 Controller 负责为该分区选举新的 leader 副本。当检测到某个分区的 ISR(In-Sync Replica)集合发生变化时，由控制器负责通知所有 broker 更新其元数据信息。当使用`kafka-topics.sh`脚本为某个 topic 增加分区数量时，同样还是由控制器负责分区的重新分配。 + +Kafka 中 Contorller 的选举的工作依赖于 Zookeeper，成功竞选为控制器的 broker 会在 Zookeeper 中创建`/controller`这个临时（EPHEMERAL）节点。 + +### 选举过程 + +![图片](https://mmbiz.qpic.cn/mmbiz_png/FbXJ7UCc6O0dHyDpzdia8xZ2nS1IzIMSo0QbCfn1hPByxALcMR51ibapYaM03B1Hibfapv9HRzCjXy5zaJeQ8lT0g/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +Broker 启动的时候尝试去读取`/controller`节点的`brokerid`的值，如果`brokerid`的值不等于-1，则表明已经有其他的 Broker 成功成为 Controller 节点，当前 Broker 主动放弃竞选；如果不存在`/controller`节点，或者 brokerid 数值异常，当前 Broker 尝试去创建`/controller`这个节点，此时也有可能其他 broker 同时去尝试创建这个节点，只有创建成功的那个 broker 才会成为控制器，而创建失败的 broker 则表示竞选失败。每个 broker 都会在内存中保存当前控制器的 brokerid 值，这个值可以标识为 activeControllerId。 + +### 实现 + +![图片](https://mmbiz.qpic.cn/mmbiz_png/FbXJ7UCc6O0dHyDpzdia8xZ2nS1IzIMSow0jXB6OziajJJSf0Eb8LXKAEhHTAIvvjJWCE0rsMPaSGMEC92fH4zyA/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +Controller 读取 Zookeeper 中的节点数据，初始化上下文(Controller Context)，并管理节点变化，变更上下文，同时也需要将这些变更信息同步到其他普通的 broker 节点中。Controller 通过定时任务，或者监听器模式获取 zookeeper 信息，事件监听会更新更新上下文信息，如图所示，Controller 内部也采用生产者-消费者实现模式，Controller 将 zookeeper 的变动通过事件的方式发送给事件队列，队列就是一个`LinkedBlockingQueue`，事件消费者线程组通过消费消费事件，将相应的事件同步到各 Broker 节点。这种队列 FIFO 的模式保证了消息的有序性。 + +### 职责 + +Controller 被选举出来，作为整个 Broker 集群的管理者，管理所有的集群信息和元数据信息。它的职责包括下面几部分： + +1. 处理 Broker 节点的上线和下线，包括自然下线、宕机和网络不可达导致的集群变动，Controller 需要及时更新集群元数据，并将集群变化通知到所有的 Broker 集群节点； +2. 创建 Topic 或者 Topic 扩容分区，Controller 需要负责分区副本的分配工作，并主导 Topic 分区副本的 Leader 选举。 +3. 管理集群中所有的副本和分区的状态机，监听状态机变化事件，并作出相应的处理。Kafka 分区和副本数据采用状态机的方式管理，分区和副本的变化都在状态机内会引起状态机状态的变更，从而触发相应的变化事件。 + +> “ +> +> 65 哥：状态机啊，听起来好复杂。 +> +> ” + +Controller 管理着集群中所有副本和分区的状态机。大家不要被`状态机`这个词唬住了。理解状态机很简单。先理解模型，即这是什么关于什么模型，然后就是模型的状态有哪些，模型状态之间如何转换，转换时发送相应的变化事件。 + +Kafka 的分区和副本状态机很简单。我们先理解，这分别是管理 Kafka Topic 的分区和副本的。它们的状态也很简单，就是 CRUD，具体说来如下： + +#### 分区状态机 + +PartitionStateChange，管理 Topic 的分区，它有以下 4 种状态： + +1. NonExistentPartition：该状态表示分区没有被创建过或创建后被删除了。 +2. NewPartition：分区刚创建后，处于这个状态。此状态下分区已经分配了副本，但是还没有选举 leader，也没有 ISR 列表。 +3. OnlinePartition：一旦这个分区的 leader 被选举出来，将处于这个状态。 +4. OfflinePartition：当分区的 leader 宕机，转移到这个状态。 + +我们用一张图来直观的看看这些状态是如何变化的，以及在状态发生变化时 Controller 都有哪些操作： + +![图片](https://mmbiz.qpic.cn/mmbiz_png/FbXJ7UCc6O0dHyDpzdia8xZ2nS1IzIMSowS1wGNDp0qM2deSV87Rv42OOxdeGQIH0cZEwZTPhNb2CqgmPvpIYvA/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +#### 副本状态机 + +ReplicaStateChange，副本状态，管理分区副本信息，它也有 4 种状态： + +1. NewReplica: 创建 topic 和分区分配后创建 replicas，此时，replica 只能获取到成为 follower 状态变化请求。 +2. OnlineReplica: 当 replica 成为 parition 的 assingned replicas 时，其状态变为 OnlineReplica, 即一个有效的 OnlineReplica。 +3. OfflineReplica: 当一个 replica 下线，进入此状态，这一般发生在 broker 宕机的情况下； +4. NonExistentReplica: Replica 成功删除后，replica 进入 NonExistentReplica 状态。 + +副本状态间的变化如下图所示，Controller 在状态变化时会做出相应的操作： + +![图片](https://mmbiz.qpic.cn/mmbiz_png/FbXJ7UCc6O0dHyDpzdia8xZ2nS1IzIMSoDYVWzibF5xLoRuibP4HOeTlIuK3M74UDvBsk2KuJlaZu2Aqv7gPibrwhQ/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +## Network + +Kafka 的网络通信模型是基于 NIO 的 Reactor 多线程模型来设计的。其中包含了一个`Acceptor`线程，用于处理新的连接，`Acceptor` 有 N 个 `Processor` 线程 select 和 read socket 请求，N 个 `Handler` 线程处理请求并相应，即处理业务逻辑。下面就是 KafkaServer 的模型图： + +![图片](https://mmbiz.qpic.cn/mmbiz_png/FbXJ7UCc6O0dHyDpzdia8xZ2nS1IzIMSoF5xwBbjgrd3nDVXyaMl7SqDbiaib6ej4sGYCwehEunqHeOaUFWRrpHcQ/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + + + + + +首先 ZooKeeper 是做什么的呢？它是一个分布式协调框架，负责协调管理并保存 Kafka 集群的所有元数据信息，比如集群都有哪些 Broker 在运行、创建了哪些 Topic，每个 Topic 都有多少分区以及这些分区的 Leader 副本都在哪些机器上等信息。 diff --git "a/docs/distribution/message-queue/Kafka/kafka\351\253\230\345\217\257\351\235\240.md" "b/docs/distribution/message-queue/Kafka/kafka\351\253\230\345\217\257\351\235\240.md" new file mode 100755 index 0000000000..5908163076 --- /dev/null +++ "b/docs/distribution/message-queue/Kafka/kafka\351\253\230\345\217\257\351\235\240.md" @@ -0,0 +1,73 @@ +> 这个标题，我想了好久，其实应该叫 Kafka 数据高可靠，主要是想聊下三个最常见的问题（面试问题） +> +> - Kafka 如何保证消息不丢失？ +> - Kafka 如何做到消息不重复？ +> - Kafka 怎么保证消息不积压？ + + + +## 前言 + +消息队列的引入一般都是用来解决：系统解耦（异步通信）、流量控制这些场景的，程序的世界，怎么可能有完美的解决方案，只有最适合的方案，所以么，这些场景下的 MQ 会遇到一些小问题： + +- 异步通信，就可能会出现数据一致性问题（消息丢失、消息重复） + +- 流量控制，就可能会出现消息积压 + +再回顾下最经典的 Kafka 存储机制，一条消息从生产到消费完成可以划分成三个阶段，生产消息——消息存储——消费消息，我们遇到的问题也都从这三个方面 + +![img](https://tva1.sinaimg.cn/large/007S8ZIlly1gh45cng275j30ny0bjwfr.jpg) + + + + + + + +## 消息丢失问题 + +> 那面对“在使用 MQ 消息队列时，如何确保消息不丢失”这个问题时，你要怎么回答呢？首先，你要分析其中有几个考点，比如： +> +> - 如何知道有消息丢失？ +> +> - 哪些环节可能丢消息？ +> +> - 如何确保消息不丢失？ +> +> 候选人在回答时，要先让面试官知道你的分析思路，然后再提供解决方案： 网络中的数据传输不可靠，想要解决如何不丢消息的问题，首先要知道哪些环节可能丢消息，以及我们如何知道消息是否丢失了，最后才是解决方案（而不是上来就直接说自己的解决方案）。就好比“架构设计”“架构”体现了架构师的思考过程，而“设计”才是最后的解决方案，两者缺一不可。 + + + +### 生产者 + +生产者在生产消息的时候，会给每个发出的消息指定一个全局唯一的 ID，或者版本号，消费数据的时候就可以用来检验 + +这属于理论，怎么落地呢，可以用拦截器机制， 在生产端发送消息之前，通过拦截器将消息版本号注入消息中（版本号可以采用连续递增的 ID 生成，也可以通过分布式全局唯一 ID生成）。然后在消费端收到消息后，再通过拦截器检测版本号的连续性或消费状态，这样实现的好处是消息检测的代码不会侵入到业务代码中，可以通过单独的任务来定位丢失的消息，做进一步的排查。 + +这里需要你注意：如果同时存在多个消息生产端和消息消费端，通过版本号递增的方式就很难实现了，因为不能保证版本号的唯一性，此时只能通过全局唯一 ID 的方案来进行消息检测，具体的实现原理和版本号递增的方式一致。 + + + +## 消息重复问题 + +消息补偿的时候，一定会存在重复消息的情况，如何实现消费端的幂等 + + + + + + + +## 消息积压问题 + +消息积压，这个属于性能可靠，我们在 kafka 使用中所说的消息积压，指的其实就是消费端处理消息的能力出现了问题，和生产者和Kafka 集群其实没有多大关系 + +> 如果是线上突发问题，要临时扩容，增加消费端的数量，与此同时，降级一些非核心的业务。通过扩容和降级承担流量，这是为了表明你对应急问题的处理能力。 +> +> 其次，才是排查解决异常问题，如通过监控，日志等手段分析是否消费端的业务逻辑代码出现了问题，优化消费端的业务处理逻辑。 +> +> 最后，如果是消费端的处理能力不足，可以通过水平扩容来提供消费端的并发处理能力，但这里有一个考点需要特别注意， 那就是在扩容消费者的实例数的同时，必须同步扩容主题 Topic 的分区数量，确保消费者的实例数和分区数相等。如果消费者的实例数超过了分区数，由于分区是单线程消费，所以这样的扩容就没有效果。 +> +> 比如在 Kafka 中，一个 Topic 可以配置多个 Partition（分区），数据会被写入到多个分区中，但在消费的时候，Kafka 约定一个分区只能被一个消费者消费，Topic 的分区数量决定了消费的能力，所以，可以通过增加分区来提高消费者的处理能力。 + +这个也属于高性能的问题，所以还会引出一篇《Kafka 高性能》 \ No newline at end of file diff --git "a/docs/message-queue/Kafka/\346\267\261\345\205\245Kafka.md" "b/docs/distribution/message-queue/Kafka/\346\267\261\345\205\245Kafka.md" similarity index 100% rename from "docs/message-queue/Kafka/\346\267\261\345\205\245Kafka.md" rename to "docs/distribution/message-queue/Kafka/\346\267\261\345\205\245Kafka.md" diff --git a/docs/message-queue/readMQ.md b/docs/distribution/message-queue/readMQ.md similarity index 56% rename from docs/message-queue/readMQ.md rename to docs/distribution/message-queue/readMQ.md index 87c63d189e..a629092c5b 100644 --- a/docs/message-queue/readMQ.md +++ b/docs/distribution/message-queue/readMQ.md @@ -1,16 +1,7 @@
- -![img](../_images/message-queue/mq_index.png) -
## MQ - [浅谈消息队列及常见的消息中间件](/message-queue/浅谈消息队列及常见的消息中间件.md) - - -## Kafka - -- [Hello Kafka](message-queue/Kafka/Hello-Kafka.md) - diff --git "a/docs/message-queue/\346\265\205\350\260\210\346\266\210\346\201\257\351\230\237\345\210\227\345\217\212\345\270\270\350\247\201\347\232\204\346\266\210\346\201\257\344\270\255\351\227\264\344\273\266.md" "b/docs/distribution/message-queue/\346\265\205\350\260\210\346\266\210\346\201\257\351\230\237\345\210\227\345\217\212\345\270\270\350\247\201\347\232\204\346\266\210\346\201\257\344\270\255\351\227\264\344\273\266.md" similarity index 95% rename from "docs/message-queue/\346\265\205\350\260\210\346\266\210\346\201\257\351\230\237\345\210\227\345\217\212\345\270\270\350\247\201\347\232\204\346\266\210\346\201\257\344\270\255\351\227\264\344\273\266.md" rename to "docs/distribution/message-queue/\346\265\205\350\260\210\346\266\210\346\201\257\351\230\237\345\210\227\345\217\212\345\270\270\350\247\201\347\232\204\346\266\210\346\201\257\344\270\255\351\227\264\344\273\266.md" index d0b3a4849b..2706c0ca5b 100644 --- "a/docs/message-queue/\346\265\205\350\260\210\346\266\210\346\201\257\351\230\237\345\210\227\345\217\212\345\270\270\350\247\201\347\232\204\346\266\210\346\201\257\344\270\255\351\227\264\344\273\266.md" +++ "b/docs/distribution/message-queue/\346\265\205\350\260\210\346\266\210\346\201\257\351\230\237\345\210\227\345\217\212\345\270\270\350\247\201\347\232\204\346\266\210\346\201\257\344\270\255\351\227\264\344\273\266.md" @@ -1,22 +1,20 @@ # 浅谈消息队列及常见的消息中间件 -> - ## 前言 **消息队列** 已经逐渐成为企业应用系统 **内部通信** 的核心手段。它具有 **低耦合**、**可靠投递**、**广播**、**流量控制**、**最终一致性** 等一系列功能。 -当前使用较多的 **消息队列** 有 `RabbitMQ`、`RocketMQ`、`ActiveMQ`、`Kafka`、`ZeroMQ`、`MetaMQ` 等，而部分 **数据库** 如 `Redis`、`MySQL` 以及 `phxsql` 也可实现消息队列的功能。 +当前使用较多的 **消息队列** 有 `RabbitMQ`、`RocketMQ`、`ActiveMQ`、`Kafka`、`ZeroMQ`、`MetaMQ` 等，而部分**数据库** 如 `Redis`、`MySQL` 以及 `phxsql` 也可实现消息队列的功能。 -![img](../_images/message-queue/mesage-what.png) +![img](https://p.ipic.vip/onjauh.png) ## 1. 消息队列概述 -**消息队列** 是指利用 **高效可靠** 的 **消息传递机制** 进行与平台无关的 **数据交流**，并基于 **数据通信** 来进行分布式系统的集成。 +**消息队列** 是指利用 **高效可靠** 的 **消息传递机制** 进行与平台无关的 **数据交流**，并基于**数据通信**来进行分布式系统的集成。 -![img](../_images/message-queue/message-overview.png) +![img](https://p.ipic.vip/ajmwms.png) @@ -43,7 +41,7 @@ 消息队列的 **传递服务模型** 如下图所示： -![img](../_images/message-queue/message-server-mode.png) +![img](https://p.ipic.vip/wzphkb.jpg) ## 4. 消息队列的的传输模式 @@ -53,7 +51,7 @@ 传统的点对点消息中间件通常由 **消息队列服务**、**消息传递服务**、**消息队列** 和 **消息应用程序接口** `API` 组成，其典型的结构如下图所示。 -![img](../_images/message-queue/message-queue.png) +![img](https://p.ipic.vip/4ozrib.jpg) **特点：** @@ -63,7 +61,7 @@ **示意图如下所示：** -![img](../_images/message-queue/message-queue-mode.png) +![img](https://p.ipic.vip/1qz68s.jpg) ### 4.2. 发布/订阅模型（Pub/Sub） @@ -73,7 +71,7 @@ 在这种情况下，在订阅者 **未连接时**，发布的消息将在订阅者 **重新连接** 时 **重新发布**，如下图所示： -![img](../_images/message-queue/message-pubsub-mode.png) +![img](https://p.ipic.vip/u9dw6i.jpg) **特性：** @@ -101,7 +99,7 @@ 网站用户注册，注册成功后会过一会发送邮件确认或者短息。 -![img](../_images/message-queue/message-user-1.png) +![img](https://p.ipic.vip/8gm3u2.jpg) ### 5.2. 系统解耦 @@ -121,7 +119,7 @@ **生产者/消费者** 模式，只需要关心消息是否 **送达队列**，至于谁希望订阅和需要消费，是 **下游** 的事情，无疑极大地减少了开发和联调的工作量。 -![img](../_images/message-queue/message-user-2.png) +![img](https://p.ipic.vip/0u4rsc.png) ### 5.5. 流量削峰和流控 @@ -138,7 +136,7 @@ 1. 把消息队列当成可靠的 **消息暂存地**，进行一定程度的 **消息堆积**； 2. 定时进行消息投递，比如模拟 **用户秒杀** 访问，进行 **系统性能压测**。 -![img](../_images/message-queue/message-user-3.png) +![img](https://p.ipic.vip/l1jd43.jpg) ### 5.6. 日志处理 @@ -148,7 +146,7 @@ 把日志进行集中收集，用于计算 `PV`、**用户行为分析** 等等。 -![img](../_images/message-queue/message-user-4.png) +![img](https://p.ipic.vip/5zzqes.jpg) ### 5.7. 消息通讯 @@ -168,7 +166,7 @@ ### 6.3. 两种类型的区别 -![img](../_images/message-queue/message-push-pull.png) +![img](https://p.ipic.vip/o2vb3o.jpg) ## 7. 消息队列技术对比 @@ -178,7 +176,7 @@ `ActiveMQ` 是由 `Apache` 出品，`ActiveMQ` 是一个完全支持`JMS1.1` 和 `J2EE 1.4` 规范的 `JMS Provider` 实现。它非常快速，支持 **多种语言的客户端** 和 **协议**，而且可以非常容易的嵌入到企业的应用环境中，并有许多高级功能。 -![img](../_images/message-queue/message-acaticemq.png) +![img](https://p.ipic.vip/p4is7u.png) #### (a) 主要特性 @@ -218,7 +216,7 @@ `RabbitMQ` 于 `2007` 年发布，是一个在 `AMQP` (**高级消息队列协议**)基础上完成的，可复用的企业消息系统，是当前最主流的消息中间件之一。 -![img](../_images/message-queue/message-rabbitmq.png) +![img](https://p.ipic.vip/djwqry.png) #### (a) 主要特性 @@ -259,7 +257,7 @@ #### (a) 主要特性 -![img](../_images/message-queue/message-rocketmq.png) +![img](https://p.ipic.vip/9amnow.png) 1. 基于 **队列模型**：具有 **高性能**、**高可靠**、**高实时**、**分布式** 等特点； 2. `Producer`、`Consumer`、**队列** 都支持 **分布式**； @@ -302,7 +300,7 @@ -![img](../_images/message-queue/message-kafka.png) +![img](https://p.ipic.vip/hs1qip.png) #### (a) 主要特性 diff --git a/docs/rpc/Hello-Protocol-Buffers.md b/docs/distribution/rpc/Hello-Protocol-Buffers.md similarity index 93% rename from docs/rpc/Hello-Protocol-Buffers.md rename to docs/distribution/rpc/Hello-Protocol-Buffers.md index 3cea3e2f37..67aeca076f 100644 --- a/docs/rpc/Hello-Protocol-Buffers.md +++ b/docs/distribution/rpc/Hello-Protocol-Buffers.md @@ -6,20 +6,20 @@ ## Protobuf 的优点 -- 更小——序列化后，数据大小可缩小约3倍 -- 更快——序列化速度更快，比xml和JSON快20-100倍，体积缩小后，传输时，带宽也会优化 -- 更简单——proto编译器，自动进行序列化和反序列化 +- 更小——序列化后，数据大小可缩小约 3 倍 +- 更快——序列化速度更快，比 xml 和 JSON 快 20-100 倍，体积缩小后，传输时，带宽也会优化 +- 更简单——proto 编译器，自动进行序列化和反序列化 - 维护成本低——跨平台、跨语言，多平台仅需要维护一套对象协议（.proto） - 可扩展——“向后”兼容性好，不必破坏已部署的、依靠“老”数据格式的程序就可以对数据结构进行升级 -- 加密性好——HTTP传输内容抓包只能看到字节 +- 加密性好——HTTP 传输内容抓包只能看到字节 在传输数据量大、网络环境不稳定的数据存储和RPC数据交换场景比较合适 ## Protobuf 的不足 - 功能简单，无法用来表示复杂的概念 -- 通用性较差，XML和JSON已成为多种行业标准的编写工具，pb只是geogle内部使用 -- 自解释性差，以二进制数据流方式存储（不可读），需要通过.proto文件才可以 +- 通用性较差，XML 和 JSON 已成为多种行业标准的编写工具，pb 只是 geogle 内部使用 +- 自解释性差，以二进制数据流方式存储（不可读），需要通过 `.proto` 文件才可以 @@ -72,7 +72,7 @@ message AddressBook { - java_package： 生成java类的包名,如不显式指定，默认包名为：按照应用名称倒序方式进行排序 - java_outer_classname：生成 java类的类名,如不显式指定，则默认为把.proto文件名转换为首字母大写来生成 - message: 你的消息格式，各数据类型（`bool`, `int32`, `float`, `double`,  `string` ，`enum` ... ）字段的集合，在一个.proto文件中可以定义多个message，一个message里也可以定义另外一个message(相当于java的类，当然也可以有内部类) -- 当然PB也是支持和java一样的`import`的，`import "xxx.proto";` +- 当然 PB 也是支持和 Java 一样的`import`的，`import "xxx.proto";` - 像每个字段也必须有修饰符，PB提供的字段修饰符有3种 - required：必填 - optional：可选 diff --git a/docs/distribution/rpc/Hello-RPC.md b/docs/distribution/rpc/Hello-RPC.md new file mode 100644 index 0000000000..ced0b8d97a --- /dev/null +++ b/docs/distribution/rpc/Hello-RPC.md @@ -0,0 +1,527 @@ +![](https://cdn.pixabay.com/photo/2019/12/19/05/56/digitization-4705450_960_720.jpg) + +> 说起 rpc，肯定要提到分布式 +> +> 能说下rpc的通信流程吗 +> +> 如果没有rpc框架，你怎么调用另外一台服务器上的接口呢 + +## 一、RPC是个啥玩意 + +**远程过程调用**（英语：Remote Procedure Call，缩写为 RPC）是一个计算机通信协议。该协议允许运行于一台计算机的程序调用另一台计算机的子程序，而程序员无需额外地为这个交互作用编程，区别于本地过程调用。 + + + +远程过程调用，自然是相对于本地过程调用来说的，如果是个单体应用，内部之间，本地函数调用就可以了，因为在**同一个地址空间**，或者说在同一块内存，所以通过方法栈和参数栈就可以实现。 + +随着应用的升级，单体应用无法满足发展，我们改造成分布式应用，将很多可以共享的功能都单独拎出来组成各种服务 + +这时候有同学会说了，服务之间通过 http，调用 Restful 接口就行 + +对了，我们外部 API 一般都是这样，每次构造 http 请求和 body 这些 + +可是我们是内部系统，希望可以像本地调用那样，去发起远程调用，让使用者感知不到远程调用的过程呢，像这样： + +```java +@Reference +private Calculator calculator; + +... + +calculator.add(1,2); + +... + +``` + +这时候，有同学就会说，用**代理模式**呀！而且最好是结合Spring IoC一起使用，通过Spring注入calculator对象，注入时，如果扫描到对象加了@Reference注解，那么就给它生成一个代理对象，将这个代理对象放进容器中。而这个代理对象的内部，就是通过httpClient来实现RPC远程过程调用的。 + +可能上面这段描述比较抽象，不过这就是很多RPC框架要解决的问题和解决的思路，比如阿里的Dubbo。 + +总结一下，RPC 的作用体现在两个方面 + +1. 远程调用时，要能够像本地调用一样方便，让调用者感知不到远程调用的逻辑 +2. 解决分布式系统中，服务之间的调用问题。隐藏底层网络通信的复杂性，让我们更专注于业务 + + + +实际情况下，RPC很少用到http协议来进行数据传输，毕竟我只是想传输一下数据而已，何必动用到一个文本传输的应用层协议呢，我为什么不直接使用**二进制传输**？比如直接用Java的Socket协议进行传输？ + +不管你用何种协议进行数据传输，**一个完整的RPC过程，都可以用下面这张图来描述**： + +![img](https:////upload-images.jianshu.io/upload_images/7143349-9e00bb104b9e3867.png?imageMogr2/auto-orient/strip|imageView2/2/w/263/format/webp) + +![](https://static001.geekbang.org/resource/image/82/59/826a6da653c4093f3dc3f0a833915259.jpg) + +以左边的Client端为例，Application就是rpc的调用方，Client Stub就是我们上面说到的代理对象，也就是那个看起来像是Calculator的实现类，其实内部是通过rpc方式来进行远程调用的代理对象，至于Client Run-time Library，则是实现远程调用的工具包，比如jdk的Socket，最后通过底层网络实现实现数据的传输。 + +这个过程中最重要的就是**序列化**和**反序列化**了，因为数据传输的数据包必须是二进制的，你直接丢一个Java对象过去，人家可不认识，你必须把Java对象序列化为二进制格式，传给Server端，Server端接收到之后，再反序列化为Java对象。 + + + + + +> 大概知道了 RPC 是个啥，可是又说不上来他和 HTTP 或者 RMI 的区别，我们接着聊 +> +> ![](https://img.pkdoutu.com/production/uploads/image/2018/06/08/20180608391715_wkvHBK.jpg) + +## 二、RPC 再了解 + +### RPC VS HTTP + +RPC=Remote Produce Call 是一种技术的概念名词. HTTP是一种协议,**RPC可以通过HTTP来实现**,也可以通过Socket自己实现一套协议来实现. + +HTTP严格来说跟RPC不是一个层级的概念。HTTP本身也可以作为RPC的传输层协议。 + +首先 http 和 rpc 并不是一个并行概念。 + +rpc是远端过程调用，其调用协议通常包含传输协议和序列化协议。 + +传输协议包含: 如著名的 [gRPC]([grpc / grpc.io](https://link.zhihu.com/?target=http%3A//www.grpc.io/)) 使用的 http2 协议，也有如dubbo一类的自定义报文的tcp协议。 + +序列化协议包含: 如基于文本编码的 xml json，也有二进制编码的 protobuf hessian等。 + + + +### RPC vs Restful + +其实这两者并不是一个维度的概念，总得来说RPC涉及的维度更广。 + +如果硬要比较，那么可以从RPC风格的url和Restful风格的url上进行比较。 + +比如你提供一个查询订单的接口，用RPC风格，你可能会这样写： + +``` +/queryOrder?orderId=123 +``` + +用Restful风格呢？ + +``` +Get +/order?orderId=123 +``` + +**RPC是面向过程，Restful是面向资源**，并且使用了Http动词。从这个维度上看，Restful风格的url在表述的精简性、可读性上都要更好。 + +REST是一种架构风格，指的是一组架构约束条件和原则。满足这些约束条件和原则的应用程序或设计就是 RESTful。REST规范把所有内容都视为资源，网络上一切皆资源。 + +### RPC vs RMI + +严格来说这两者也不是一个维度的。 + +RMI 是 Java 提供的一种访问远程对象的协议，是已经实现好了的，可以直接用了。 + +而 RPC 呢？人家只是一种编程模型，并没有规定你具体要怎样实现，**你甚至都可以在你的RPC框架里面使用RMI来实现数据的传输**，比如Dubbo：[Dubbo - rmi协议](https://link.jianshu.com?t=http%3A%2F%2Fdubbo.apache.org%2Fbooks%2Fdubbo-user-book%2Freferences%2Fprotocol%2Frmi.html) + + + + + +## 三、RPC框架原理 + +> 如何调用他人的远程服务？ + + + +RPC框架的目标就是让远程过程(服务)调用更加简单、透明， + +既然是一个远程调用，那肯定就需要网络来传输数据 + +RPC框架负责屏蔽底层的传输方式(TCP或者UDP)、序列化方式( XML/JSON/二进制)和通信细节。 + +框架使用者只需要了解谁在什么位置提供了什么样的远程服务接口即可，开发者不需要关心底层通信细节和调用过程。 + + + + + +由于各个服务部署在不同机器，服务间的调用涉及到网络通信过程，如果服务消费方每调用一个服务都要写一坨网络通信相关的代码，不仅使用复杂而且极易出错。 + +如果有一种方式能让我们像调用本地服务一样调用远程服务，而让调用者对网络通信这些细节透明，那么将大大提高生产力。这种方式其实就是RPC（Remote Procedure Call Protocol），在各大互联网公司中被广泛使用，如阿里巴巴的hsf、dubbo（开源）、Facebook的thrift（开源）、Google grpc（开源）、Twitter的 finagle 等。 + +我们首先看下一个RPC调用的具体流程： + +![](https://tva1.sinaimg.cn/large/007S8ZIlly1gjlr8lf4hlj31440ny41b.jpg) + +1）服务消费方（client）调用以本地调用方式调用服务； + +2）client stub接收到调用后负责将方法、参数等组装成能够进行网络传输的消息体； + +3）client stub找到服务地址，并将消息发送到服务端； + +4）server stub收到消息后进行解码； + +5）server stub根据解码结果调用本地的服务； + +6）本地服务执行并将结果返回给server stub； + +7）server stub将返回结果打包成消息并发送至消费方； + +8）client stub接收到消息，并进行解码； + +9）服务消费方得到最终结果。 + +**RPC的目标就是要2~8这些步骤都封装起来，让用户对这些细节透明。** + + + +RPC仅仅是一种技术，为什么会与微服务框架搞混呢？ + +因为随着RPC的大量使用，必然伴随着服务的发现、服务的治理、服务的监控这些，这就组成了微服务框架。 + +RPC仅仅是微服务中的一部分。 + + + + + + + + + + + + + +维度 RPC REST +耦合性 强耦合 松散耦合 +消息协议 二进制thrift、protobuf、avro 文本型XML、JSON +通讯协议 TCP为主，也可以是HTTP HTTP/HTTP2 +性能 高 一般低于RPC +接口契约IDL Thrift、protobuf idl Swagger +客户端 强类型客户端、一般自动生成，可支持多语言客户端 一般http client可访问，也可支持多语言 +案例 dubbo、motan、tars、grpc、thrift spring boot/mvc、Jax-rs +开发者友好 客户端比较方便，但是二进制消息不可读 文本消息开发者可读、浏览器可直接访问查看结果 +对外开放 需要转换成REST/文本协议 直接对外开放 + + + + + +RPC框架的目标就是让远程过程(服务)调用更加简单、透明，RPC框架负责屏蔽底层的传输方式(TCP或者UDP)、序列化方式( XML/JSON/二进制)和通信细节。框架使用者只需要了解谁在什么位置提供了什么样的远程服务接口即可，开发者不需要关心底层通信细节和调用过程。 + + + +RPC框架的调用原理图如下： + + + + ![img](http://jiangew.me/assets/images/post/20181013/grpc-01-01.png) + + + + + + + +## RPC框架核心技术点 + +RPC框架实现的几个核心技术点总结如下： + +1)远程服务提供者需要以某种形式提供服务调用相关的信息，包括但不限于服务接口定义、数据结构，或者中间态的服务定义文件，例如 Thrift的IDL文件， WS-RPC的WSDL文件定义，甚至也可以是服务端的接口说明文档;服务调用者需要通过一定的途径获取远程服务调用相关信息，例如服务端接口定义Jar包导入，获取服务端1DL文件等。 + +### 远程代理 + +动态代理，用 Spring AOP 的同学就不陌生了，他就是远程调用的魔法 + +当我们作为调用方使用接口时，RPC 会自动给接口生成一个代理类，我们在项目中注入接口的时候，运行过程中实际绑定的是这个接口生成的代理类。这样在接口方法被调用的时候，它实际上是被生成代理类拦截到了，这样我们就可以在生成的代理类里 面，加入远程调用逻辑。 + +通过这种“偷梁换柱”的手法，就可以帮用户屏蔽远程调用的细节，实现像调用本地一样地调用远程的体验，整体流程如下图所示: + +![](https://static001.geekbang.org/resource/image/05/53/05cd18e7e33c5937c7c39bf8872c5753.jpg) + +远程代理对象:服务调用者调用的服务实际是远程服务的本地代理，对于Java语言，它的实现就是JDK的动态代理，通过动态代理的拦截机制，将本地调用封装成远程服务调用. + + + +### 通信 + +一次 RPC 调用，本质就是服务消费者与服务提供者间的一次网络信息交换的过程。 + +服务调用者通过网络 IO 发送一条 请求消息，服务提供者接收并解析，处理完相关的业务逻辑之后，再发送一条响应消息给服务调用者，服务调用者接收并解析响应消息，处理完相关的响应逻辑，一次 RPC 调用便结 束了。可以说，网络通信是整个 RPC 调用流程的基础。 + +#### 常见网络 IO 模型 + +那说到网络通信，就不得不提一下网络 IO 模型。为什么要讲网络 IO 模型呢?因为所谓的 + +两台 PC 机之间的网络通信，实际上就是两台 PC 机对网络 IO 的操作。 + +常见的网络 IO 模型分为四种:同步阻塞 IO(BIO)、同步非阻塞 IO(NIO)、IO 多路复 用和异步非阻塞 IO(AIO)。在这四种 IO 模型中，只有 AIO 为异步 IO，其他都是同步 IO。 + +其中，最常用的就是同步阻塞 IO 和 IO 多路复用，这一点通过了解它们的机制，你会 get 到。至于其他两种 IO 模型，因为不常用，则不作为本讲的重点，有兴趣的话我们可以在留 言区中讨论。 + + + +##### 阻塞 IO(blocking IO) + +同步阻塞 IO 是最简单、最常见的 IO 模型，在 Linux 中，默认情况下所有的 socket 都是blocking 的，先看下操作流程。 + +首先，应用进程发起 IO 系统调用后，应用进程被阻塞，转到内核空间处理。之后，内核开 始等待数据，等待到数据之后，再将内核中的数据拷贝到用户内存中，整个 IO 处理完毕后 返回进程。最后应用的进程解除阻塞状态，运行业务逻辑。 + +这里我们可以看到，系统内核处理 IO 操作分为两个阶段——等待数据和拷贝数据。而在这 两个阶段中，应用进程中 IO 操作的线程会一直都处于阻塞状态，如果是基于 Java 多线程 开发，那么每一个 IO 操作都要占用线程，直至 IO 操作结束。 + +这个流程就好比我们去餐厅吃饭，我们到达餐厅，向服务员点餐，之后要一直在餐厅等待后 +厨将菜做好，然后服务员会将菜端给我们，我们才能享用。 + + + +##### IO 多路复用(IO multiplexing) + +多路复用 IO 是在高并发场景中使用最为广泛的一种 IO 模型，如 Java 的 NIO、Redis、 Nginx 的底层实现就是此类 IO 模型的应用，经典的 Reactor 模式也是基于此类 IO 模型。 + +那么什么是 IO 多路复用呢? 通过字面上的理解，多路就是指多个通道，也就是多个网络连接的 IO，而复用就是指多个通道复用在一个复用器上。 + +多个网络连接的 IO 可以注册到一个复用器(select)上，当用户进程调用了 select，那么 整个进程会被阻塞。同时，内核会“监视”所有 select 负责的 socket，当任何一个 socket 中的数据准备好了，select 就会返回。这个时候用户进程再调用 read 操作，将数据从内核中拷贝到用户进程。 + +这里我们可以看到，当用户进程发起了 select 调用，进程会被阻塞，当发现该 select 负责 的 socket 有准备好的数据时才返回，之后才发起一次 read，整个流程要比阻塞 IO 要复杂，似乎也更浪费性能。但它最大的优势在于，用户可以在一个线程内同时处理多个 socket 的 IO 请求。用户可以注册多个 socket，然后不断地调用 select 读取被激活的 socket，即可达到在同一个线程内同时处理多个 IO 请求的目的。而在同步阻塞模型中，必须通过多线程的方式才能达到这个目的。 + + + +### 序列化 + +网络传输的数据必须是二进制数据，远程通信时需要将对象转换成二进制码流进行网络传输，不同的序列化框架，支持的数据类型、数据包大小、异常类型及性能等都不同 + +> 序列化就是将对象转换成二进制数据的过程，而反序列就是反过来将二进制转换为对象的过程。 + +不同的RPC框架应用场景不同，因此技术选择也会存在很大差异。一些做得比较好的RPC框架可以支持多种序列化方式，有的甚至支持用户自定义序列化框架( Hadoop Avro) + +![](https://static001.geekbang.org/resource/image/d2/04/d215d279ef8bfbe84286e81174b4e704.jpg) + + **有哪些常用的序列化?** + +#### JDK 原生序列化 + +Javer 肯定对这种原生的序列化方式最熟悉不过了 + +```java + +import java.io.*; + +public class Student implements Serializable { + //学号 + private int no; + //姓名 + private String name; + + public int getNo() { + return no; + } + + public void setNo(int no) { + this.no = no; + } + + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + @Override + public String toString() { + return "Student{" + + "no=" + no + + ", name='" + name + '\'' + + '}'; + } + + public static void main(String[] args) throws IOException, ClassNotFoundException { + String home = System.getProperty("user.home"); + String basePath = home + "/Desktop"; + FileOutputStream fos = new FileOutputStream(basePath + "student.dat"); + Student student = new Student(); + student.setNo(100); + student.setName("TEST_STUDENT"); + ObjectOutputStream oos = new ObjectOutputStream(fos); + oos.writeObject(student); + oos.flush(); + oos.close(); + + FileInputStream fis = new FileInputStream(basePath + "student.dat"); + ObjectInputStream ois = new ObjectInputStream(fis); + Student deStudent = (Student) ois.readObject(); + ois.close(); + + System.out.println(deStudent); + + } +} +``` + + 我们可以看到，JDK 自带的序列化机制对使用者而言是非常简单的。 + +序列化具体的实现是由 ObjectOutputStream 完成的，而反序列化的具体实现是由 ObjectInputStream 完成的。 + +那么 JDK 的序列化过程是怎样完成的呢？我们看下下面这张图： + +![](https://static001.geekbang.org/resource/image/7e/9f/7e2616937e3bc5323faf3ba4c09d739f.jpg) + +序列化过程就是在读取对象数据的时候，不断加入一些特殊分隔符，这些特殊分隔符用于在反序列化过程中截断用。 + +- 头部数据用来声明序列化协议、序列化版本，用于高低版本向后兼容 +- 对象数据主要包括类名、签名、属性名、属性类型及属性值，当然还有开头结尾等数据，除了属性值属于真正的对象值，其他都是为了反序列化用的元数据 +- 存在对象引用、继承的情况下，就是递归遍历“写对象”逻辑 + +实际上任何一种序列化框架，核心思想就是设计一种序列化协议，将对象的类型、属性类型、属性值一一按照固定的格式写到二进制字节流中来完成序列化，再按照固定的格式一一读出对象的类型、属性类型、属性值，通过这些信息重新创建出一个新的对象，来完成反序列化。 + + + +#### JSON + +JSON 是典型的 Key-Value 方式，没有数据类型，是一种文本型序列化框架 + +但用 JSON 进行序列化有这样两个问题，你需要格外注意： + +- JSON 进行序列化的额外空间开销比较大，对于大数据量服务这意味着需要巨大的内存和磁盘开销； +- JSON 没有类型，但像 Java 这种强类型语言，需要通过反射统一解决，所以性能不会太好。 + +所以如果 RPC 框架选用 JSON 序列化，服务提供者与服务调用者之间传输的数据量要相对较小，否则将严重影响性能。 + + + +#### Hessian + +Hessian 是动态类型、二进制、紧凑的，并且可跨语言移植的一种序列化框架。Hessian 协议要比 JDK、JSON 更加紧凑，性能上要比 JDK、JSON 序列化高效很多，而且生成的字节数也更小。 + +```java + +Student student = new Student(); +student.setNo(101); +student.setName("HESSIAN"); + +//把student对象转化为byte数组 +ByteArrayOutputStream bos = new ByteArrayOutputStream(); +Hessian2Output output = new Hessian2Output(bos); +output.writeObject(student); +output.flushBuffer(); +byte[] data = bos.toByteArray(); +bos.close(); + +//把刚才序列化出来的byte数组转化为student对象 +ByteArrayInputStream bis = new ByteArrayInputStream(data); +Hessian2Input input = new Hessian2Input(bis); +Student deStudent = (Student) input.readObject(); +input.close(); + +System.out.println(deStudent); +``` + +相对于 JDK、JSON，由于 Hessian 更加高效，生成的字节数更小，有非常好的兼容性和稳定性，所以 Hessian 更加适合作为 RPC 框架远程通信的序列化协议。 + +但 Hessian 本身也有问题，官方版本对 Java 里面一些常见对象的类型不支持，比如： + +- Linked 系列，LinkedHashMap、LinkedHashSet 等，但是可以通过扩展 CollectionDeserializer 类修复； +- Locale 类，可以通过扩展 ContextSerializerFactory 类修复； +- Byte/Short 反序列化的时候变成 Integer。 + + + +#### Protobuf + +Protobuf 是 Google 公司内部的混合语言数据标准，是一种轻便、高效的结构化数据存储格式，可以用于结构化数据序列化，支持 Java、Python、C++、Go 等语言。 + +Protobuf 使用的时候需要定义 IDL（Interface description language），然后使用不同语言的 IDL 编译器，生成序列化工具类，它的优点是： + +- 序列化后体积相比 JSON、Hessian 小很多； +- IDL 能清晰地描述语义，所以足以帮助并保证应用程序之间的类型不会丢失，无需类似 XML 解析器； +- 序列化反序列化速度很快，不需要通过反射获取类型； +- 消息格式升级和兼容性不错，可以做到向后兼容。 + +```protobuf + +/** + * + * // IDl 文件格式 + * synax = "proto3"; + * option java_package = "com.test"; + * option java_outer_classname = "StudentProtobuf"; + * + * message StudentMsg { + * //序号 + * int32 no = 1; + * //姓名 + * string name = 2; + * } + * + */ + +StudentProtobuf.StudentMsg.Builder builder = StudentProtobuf.StudentMsg.newBuilder(); +builder.setNo(103); +builder.setName("protobuf"); + +//把student对象转化为byte数组 +StudentProtobuf.StudentMsg msg = builder.build(); +byte[] data = msg.toByteArray(); + +//把刚才序列化出来的byte数组转化为student对象 +StudentProtobuf.StudentMsg deStudent = StudentProtobuf.StudentMsg.parseFrom(data); + +System.out.println(deStudent); +``` + + + +以上只是些常见的序列化协议，还有 Message pack、kryo 等 + + + +RPC 框架如何选择序列化？需要考虑的因素 + +- 传输性能和效率 +- 空间开销（序列化后的二进制数据体积不能太大） +- 通用性和兼容性 +- 安全性（别动不动就安全漏洞） + + + + + +## 四、业界主流的RPC框架 + +业界主流的RPC框架很多，比较出名的RPC主要有以下4种： + +1）、由Facebook开发的原创服务调用框架Apache Thrift; + +2）、Hadoop的子项目Avro-RPC; + +3）、caucho提供的基于binary-RPC实现的远程通信框架Hessian; + +4）、Google开源的基于HTTP/2和ProtoBuf的通用RPC框架gRPC + + + +| 功能 | Hessian | Montan | rpcx | gRPC | Thrift | Dubbo | Dubbox | Tars | Spring Cloud | | +| ---------------- | ------- | ---------------------------- | ------ | ---------------- | ------------- | ------- | -------- | ---- | ------------ | ---- | +| 开发语言 | 跨语言 | Java | Go | 跨语言 | 跨语言 | Java | Java | | Java | | +| 分布式(服务治理) | × | √ | √ | × | × | √ | √ | | √ | | +| 多序列化框架支持 | hessian | √(支持Hessian2、Json,可扩展) | √ | × 只支持protobuf | ×(thrift格式) | √ | √ | | √ | | +| 多种注册中心 | × | √ | √ | × | × | √ | √ | | √ | | +| 管理中心 | × | √ | √ | × | × | √ | √ | | √ | | +| 跨编程语言 | √ | × | × | √ | √ | × | × | | × | | +| 支持REST | × | × | × | × | × | × | √ | | √ | | +| 开源机构 | Caucho | Weibo | Apache | Google | Apache | Alibaba | Dangdang | | Apache | | +| | | | | | | | | | | | +| | | | | | | | | | | | +| | | | | | | | | | | | + + + +https://blog.csdn.net/u013452337/article/details/86593291 + + + + + + + + + +- [如何给老婆解释什么是RPC]( ) +- [你应该知道的RCP原理](https://www.cnblogs.com/LBSer/p/4853234.html) +- [深入理解RPC](https://juejin.cn/post/6844903443237175310) + + + diff --git a/docs/rpc/Hello-gRPC.md b/docs/distribution/rpc/Hello-gRPC.md similarity index 100% rename from docs/rpc/Hello-gRPC.md rename to docs/distribution/rpc/Hello-gRPC.md diff --git a/docs/distribution/rpc/MyRPC.md b/docs/distribution/rpc/MyRPC.md new file mode 100644 index 0000000000..0f831af565 --- /dev/null +++ b/docs/distribution/rpc/MyRPC.md @@ -0,0 +1,12 @@ +--- +title: 实现自己的 RPC +date: 2022-04-02 +tags: + - RPC +categories: RPC +--- + +![](https://cdn.pixabay.com/photo/2022/03/06/01/51/activity-7050634_960_720.jpg) + + + diff --git a/docs/distribution/rpc/RPC.md b/docs/distribution/rpc/RPC.md new file mode 100644 index 0000000000..89fbdbe607 --- /dev/null +++ b/docs/distribution/rpc/RPC.md @@ -0,0 +1,57 @@ +## 服务发现 + +先举个例子，假如你要给一位以前从未合作过的同事发邮件请求帮助，但你却没有他的邮箱地址。这个时候你会怎么办呢?如果是我，我会选择去看公司的企业“通信录”。 + +同理，为了高可用，在生产环境中服务提供方都是以集群的方式对外提供服务，集群里面的 这些 IP 随时可能变化，我们也需要用一本“通信录”及时获取到对应的服务节点，这个获 取的过程我们一般叫作“**服务发现**”，或者叫服务 + +对于服务调用方和服务提供方来说，其契约就是接口，相当于“通信录”中的姓名，服务节 点就是提供该契约的一个具体实例。服务 IP 集合作为“通信录”中的地址，从而可以通过 接口获取服务 IP 的集合来完成服务的发现。这就是我要说的 PRC 框架的服务发现机制， 如下图所示: + +![](https://static001.geekbang.org/resource/image/51/5d/514dc04df2b8b2f3130b7d44776a825d.jpg) + +1. 服务注册:在服务提供方启动的时候，将对外暴露的接口注册到注册中心之中，注册中 心将这个服务节点的 IP 和接口保存下来。 + +2. 服务订阅:在服务调用方启动的时候，去注册中心查找并订阅服务提供方的 IP，然后缓 存到本地，并用于后续的远程调用。 + + + +- 从服务提供者的角度看：当提供者服务启动时，需要自动向注册中心注册服务； +- 当提供者服务停止时，需要向注册中心注销服务； +- 提供者需要定时向注册中心发送心跳，一段时间未收到来自提供者的心跳后，认为提供者已经停止服务，从注册中心上摘取掉对应的服务。 +- 从调用者的角度看：调用者启动时订阅注册中心的消息并从注册中心获取提供者的地址； +- 当有提供者上线或者下线时，注册中心会告知到调用者； +- 调用者下线时，取消订阅。 + + + +## **基于** ZooKeeper 的服务发现 + +整体的思路很简单，就是搭建一个 ZooKeeper 集群作为注册中心集群，服务注册的时候只 需要服务节点向 ZooKeeper 节点写入注册信息即可，利用 ZooKeeper 的 Watcher 机制 完成服务订阅与服务下发功能，整体流程如下图: + +![](https://static001.geekbang.org/resource/image/50/75/503fabeeae226a722f83e9fb6c0d4075.jpg) + +1. 服务平台管理端先在 ZooKeeper 中创建一个服务根路径，可以根据接口名命名(例 如:/service/com.demo.xxService)，在这个路径再创建服务提供方目录与服务调用方目录(例如:provider、consumer)，分别用来存储服务提供方的节点信息和服务调 用方的节点信息。 +2. 当服务提供方发起注册时，会在服务提供方目录中创建一个临时节点，节点中存储该服务提供方的注册信息。 +3. 当服务调用方发起订阅时，则在服务调用方目录中创建一个临时节点，节点中存储该服 务调用方的信息，同时服务调用方 watch 该服务的服务提供方目录 (/service/com.demo.xxService/provider)中所有的服务节点数据。 +4. 当服务提供方目录下有节点数据发生变更时，ZooKeeper 就会通知给发起订阅的服务调 用方。 + + + +## 基于消息总线的最终一致性的注册中心 + +我们知道，ZooKeeper 的一大特点就是强一致性，ZooKeeper 集群的每个节点的数据每 次发生更新操作，都会通知其它 ZooKeeper 节点同时执行更新。它要求保证每个节点的数 据能够实时的完全一致，这也就直接导致了 ZooKeeper 集群性能上的下降。这就好比几个 人在玩传递东西的游戏，必须这一轮每个人都拿到东西之后，所有的人才能开始下一轮，而 不是说我只要获得到东西之后，就可以直接进行下一轮了。 + + + +而 RPC 框架的服务发现，在服务节点刚上线时，服务调用方是可以容忍在一段时间之后 (比如几秒钟之后)发现这个新上线的节点的。毕竟服务节点刚上线之后的几秒内，甚至更 长的一段时间内没有接收到请求流量，对整个服务集群是没有什么影响的，所以我们可以牺 牲掉 CP(强制一致性)，而选择 AP(最终一致)，来换取整个注册中心集群的性能和稳 定性。 + +那么是否有一种简单、高效，并且最终一致的更新机制，能代替 ZooKeeper 那种数据强一 致的数据更新机制呢? + +因为要求最终一致性，我们可以考虑采用消息总线机制。注册数据可以全量缓存在每个注册 +中心内存中，通过消息总线来同步数据。当有一个注册中心节点接收到服务节点注册时，会 +产生一个消息推送给消息总线，再通过消息总线通知给其它注册中心节点更新数据并进行服 +务下发，从而达到注册中心间数据最终一致性，具体流程如下图所示: + +![](https://static001.geekbang.org/resource/image/73/ff/73b59c7949ebed2903ede474856062ff.jpg) + + + diff --git a/docs/framework/.DS_Store b/docs/framework/.DS_Store new file mode 100644 index 0000000000..b52aa4e0d6 Binary files /dev/null and b/docs/framework/.DS_Store differ diff --git a/docs/framework/MyBatis/.DS_Store b/docs/framework/MyBatis/.DS_Store new file mode 100644 index 0000000000..1beb5647e6 Binary files /dev/null and b/docs/framework/MyBatis/.DS_Store differ diff --git a/docs/framework/MyBatis/MyBatis-FAQ.md b/docs/framework/MyBatis/MyBatis-FAQ.md deleted file mode 100644 index 323d7426d3..0000000000 --- a/docs/framework/MyBatis/MyBatis-FAQ.md +++ /dev/null @@ -1,2 +0,0 @@ -MyBatis是一个支持**普通SQL查询**，**存储过程**和**高级映射**的优秀**持久层框架**。MyBatis消除了几乎所有的JDBC代码和参数的手工设置以及对结果集的检索封装。MyBatis可以使用简单的**XML或注解**用于配置和原始映射，将接口和Java的POJO（Plain Old Java Objects，普通的Java对象）映射成数据库中的记录。 - diff --git a/docs/framework/MyBatis/MyBatis.md b/docs/framework/MyBatis/MyBatis.md new file mode 100644 index 0000000000..08fce3297f --- /dev/null +++ b/docs/framework/MyBatis/MyBatis.md @@ -0,0 +1,29 @@ +MyBatis是一个支持**普通SQL查询**，**存储过程**和**高级映射**的优秀**持久层框架**。MyBatis消除了几乎所有的JDBC代码和参数的手工设置以及对结果集的检索封装。MyBatis可以使用简单的**XML或注解**用于配置和原始映射，将接口和Java的POJO（Plain Old Java Objects，普通的Java对象）映射成数据库中的记录。 + + + + + +很多人会将 Hibernate 和 MyBatis 做比较，认为 Hibernate 是全自动 ORM 框架，而 MyBatis 只是半自动的 ORM 框架或是一个 SQL 模板引擎。其实，这些比较都无法完全说明一个框架比另一个框架先进，关键还是看应用场景。 + +映射文件或是注解定义 Java 语言中的类与数据库中的表之间的各种映射关系，这里使用到的映射文件后缀为“.hbm.xml”。 + +但需要注意的是，Hibernate 并不是一颗“银弹”，我们无法在面向对象模型中找到数据库中所有概念的映射，例如，索引、函数、存储过程等。 + +Hibernate 通过其简洁的 API 以及统一的 HQL 语句，帮助上层程序屏蔽掉底层数据库的差异，增强了程序的可移植性。 + +在一些大数据量、高并发、低延迟的场景中，Hibernate 在性能方面带来的损失就会逐渐显现出来。 + + + +MyBatis 中一个重要的功能就是可以帮助 Java 开发封装重复性的 JDBC 代码，这与前文分析的 Spring Data JPA 、Hibernate 等 ORM 框架一样。MyBatis 封装重复性代码的方式是通过 Mapper 映射配置文件以及相关注解，将 ResultSet 结果映射为 Java 对象，在具体的映射规则中可以嵌套其他映射规则和必要的子查询，这样就可以轻松实现复杂映射的逻辑，当然，也能够实现一对一、一对多、多对多关系映射以及相应的双向关系映射。 + + + +MyBatis 相较于 Hibernate 和各类 JPA 实现框架更加灵活、更加轻量级、更加可控。 + +- 我们可以在 MyBatis 的 Mapper 映射文件中，直接编写原生的 SQL 语句，应用底层数据库产品的方言，这就给了我们直接优化 SQL 语句的机会； + +- 我们还可以按照数据库的使用规则，让原生 SQL 语句选择我们期望的索引，从而保证服务的性能，这就特别适合大数据量、高并发等需要将 SQL 优化到极致的场景； + +- 在编写原生 SQL 语句时，我们也能够更加方便地控制结果集中的列，而不是查询所有列并映射对象后返回，这在列比较多的时候也能起到一定的优化效果。（当然，Hibernate 也能实现这种效果，需要在实体类添加对应的构造方法。） diff --git a/docs/framework/Quartz/.DS_Store b/docs/framework/Quartz/.DS_Store new file mode 100644 index 0000000000..a67ff71b96 Binary files /dev/null and b/docs/framework/Quartz/.DS_Store differ diff --git a/docs/framework/Quartz/Quartz-MySQL.md b/docs/framework/Quartz/Quartz-MySQL.md new file mode 100755 index 0000000000..633b72274f --- /dev/null +++ b/docs/framework/Quartz/Quartz-MySQL.md @@ -0,0 +1,250 @@ +# jobstore 数据库表结构 + +> 基于 Quartz 2.x 为版本的表结构和建表语句，1.x 和 2.x 的建表语句不同 + +| 序号 | 表名 | 说明 | +| ---- | ------------------------ | ------------------------------------------------------------ | +| 1 | qrtz_job_details | 存储每一个已配置的 Job 的详细信息(jobDetail) | +| 2 | qrtz_triggers | 存储已配置的触发器 (Trigger) 的信息 | +| 3 | qrtz_simple_triggers | 存储简单的 Trigger，包括重复次数，间隔，以及已触的次数 | +| 4 | qrtz_cron_triggers | 存储 CronTrigger 的信息，包括 Cron 表达式和时区信息 | +| 5 | qrtz_simprop_triggers | 存储 CalendarIntervalTrigger 和 DailyTimeIntervalTrigger 两种类型的触发器 | +| 6 | qrtz_blog_triggers | 以 Blob 类型存储的Trigger (用于 Quartz 用户用 JDBC 创建他们自己定制的 Trigger 类型，JobStore 并不知道如何存储实例的时候) | +| 7 | qrtz_calendars | 以 Blob 类型存储 Quartz 的 Calendar 信息 | +| 8 | qrtz_paused_trigger_grps | 存储已暂停的触发器的信息 | +| 9 | qrtz_fired_triggers | 记录每个正在执行的 Trigger，以及相联 Job 的执行信息 | +| 10 | qrtz_scheduler_state | 记录调度器（每个机器节点）的生命状态 | +| 11 | qrtz_locks | 记录程序的悲观锁（防止多个节点同时执行同一个定时任务） | + +SQL 地址：[tables_mysql_innodb.sql](https://github.com/quartz-scheduler/quartz/blob/master/quartz-core/src/main/resources/org/quartz/impl/jdbcjobstore/tables_mysql_innodb.sql) + + + +## qrtz_job_details + +存储每一个已配置的 Job 的详细信息 + +```sql +CREATE TABLE `qrtz_job_details` ( + `SCHED_NAME` varchar(120) COLLATE utf8_bin NOT NULL COMMENT '调度器名,集群环境中使用,必须使用同一个名称——集群环境下”逻辑”相同的scheduler,默认为QuartzScheduler', + `JOB_NAME` varchar(200) COLLATE utf8_bin NOT NULL COMMENT '集群中job的名字', + `JOB_GROUP` varchar(200) COLLATE utf8_bin NOT NULL COMMENT '集群中job的所属组的名字', + `DESCRIPTION` varchar(250) COLLATE utf8_bin DEFAULT NULL COMMENT '描述', + `JOB_CLASS_NAME` varchar(250) COLLATE utf8_bin NOT NULL COMMENT '集群中个note job实现类的完全包名,quartz就是根据这个路径到classpath找到该job类', + `IS_DURABLE` varchar(1) COLLATE utf8_bin NOT NULL COMMENT '是否持久化,把该属性设置为1，quartz会把job持久化到数据库中', + `IS_NONCONCURRENT` varchar(1) COLLATE utf8_bin NOT NULL COMMENT '是否并行，该属性可以通过注解配置', + `IS_UPDATE_DATA` varchar(1) COLLATE utf8_bin NOT NULL, + `REQUESTS_RECOVERY` varchar(1) COLLATE utf8_bin NOT NULL COMMENT '当一个scheduler失败后，其他实例可以发现那些执行失败的Jobs，若是1，那么该Job会被其他实例重新执行，否则对应的Job只能释放等待下次触发', + `JOB_DATA` blob COMMENT '一个blob字段，存放持久化job对象', + PRIMARY KEY (`SCHED_NAME`,`JOB_NAME`,`JOB_GROUP`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='存储每一个已配置的 Job 的详细信息'; +``` + + + +## qrtz_triggers + +存放配置的 Trigger 信息（一个Job可以被多个Trigger绑定，但是一个Trigger只能绑定一个Job） + +```sql +CREATE TABLE `qrtz_triggers` ( + `SCHED_NAME` varchar(120) COLLATE utf8_bin NOT NULL COMMENT '调度器名，和配置文件org.quartz.scheduler.instanceName保持一致', + `TRIGGER_NAME` varchar(200) COLLATE utf8_bin NOT NULL COMMENT '触发器的名字', + `TRIGGER_GROUP` varchar(200) COLLATE utf8_bin NOT NULL COMMENT '触发器所属组的名字', + `JOB_NAME` varchar(200) COLLATE utf8_bin NOT NULL COMMENT 'qrtz_job_details表job_name的外键', + `JOB_GROUP` varchar(200) COLLATE utf8_bin NOT NULL COMMENT 'qrtz_job_details表job_group的外键', + `DESCRIPTION` varchar(250) COLLATE utf8_bin DEFAULT NULL COMMENT '描述', + `NEXT_FIRE_TIME` bigint(13) DEFAULT NULL COMMENT '下一次触发时间', + `PREV_FIRE_TIME` bigint(13) DEFAULT NULL COMMENT '上一次触发时间', + `PRIORITY` int(11) DEFAULT NULL COMMENT '线程优先级', + `TRIGGER_STATE` varchar(16) COLLATE utf8_bin NOT NULL COMMENT '当前trigger状态，设置为ACQUIRED,如果设置为WAITING,则job不会触发', + `TRIGGER_TYPE` varchar(8) COLLATE utf8_bin NOT NULL COMMENT '触发器类型', + `START_TIME` bigint(13) NOT NULL COMMENT '开始时间', + `END_TIME` bigint(13) DEFAULT NULL COMMENT '结束时间', + `CALENDAR_NAME` varchar(200) COLLATE utf8_bin DEFAULT NULL COMMENT '日历名称', + `MISFIRE_INSTR` smallint(2) DEFAULT NULL COMMENT 'misfire处理规则,1代表【以当前时间为触发频率立刻触发一次，然后按照Cron频率依次执行】, + 2代表【不触发立即执行,等待下次Cron触发频率到达时刻开始按照Cron频率依次执行�】, + -1代表【以错过的第一个频率时间立刻开始执行,重做错过的所有频率周期后，当下一次触发频率发生时间大于当前时间后，再按照正常的Cron频率依次执行】', + `JOB_DATA` blob COMMENT 'JOB存储对象', + PRIMARY KEY (`SCHED_NAME`,`TRIGGER_NAME`,`TRIGGER_GROUP`), + KEY `SCHED_NAME` (`SCHED_NAME`,`JOB_NAME`,`JOB_GROUP`), + CONSTRAINT `qrtz_triggers_ibfk_1` FOREIGN KEY (`SCHED_NAME`, `JOB_NAME`, `JOB_GROUP`) REFERENCES `qrtz_job_details` (`SCHED_NAME`, `JOB_NAME`, `JOB_GROUP`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='存储已配置的 Trigger 的信息'; +``` + + + +## qrtz_simple_triggers + +存储简单的 trigger，包括重复次数，间隔，以及触发次数。 + + **注意**：TIMES_TRIGGERED 用来记录执行了多少次了，此值被定义在 SimpleTriggerImpl 中，每次执行 +1，这里定义的REPEAT_COUNT=5，实际情况会执行 6 次。因为第一次是在 0 开始。 + +```sql +CREATE TABLE `qrtz_simple_triggers` ( + `SCHED_NAME` varchar(120) COLLATE utf8_bin NOT NULL COMMENT '调度器名，集群名', + `TRIGGER_NAME` varchar(200) COLLATE utf8_bin NOT NULL COMMENT '触发器名', + `TRIGGER_GROUP` varchar(200) COLLATE utf8_bin NOT NULL COMMENT '触发器组', + `REPEAT_COUNT` bigint(7) NOT NULL COMMENT '重复次数', + `REPEAT_INTERVAL` bigint(12) NOT NULL COMMENT '重复间隔', + `TIMES_TRIGGERED` bigint(10) NOT NULL COMMENT '已触发次数', + PRIMARY KEY (`SCHED_NAME`,`TRIGGER_NAME`,`TRIGGER_GROUP`), + CONSTRAINT `qrtz_simple_triggers_ibfk_1` FOREIGN KEY (`SCHED_NAME`, `TRIGGER_NAME`, `TRIGGER_GROUP`) REFERENCES `qrtz_triggers` (`SCHED_NAME`, `TRIGGER_NAME`, `TRIGGER_GROUP`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='存储简单的 Trigger，包括重复次数，间隔，以及已触的次数'; +``` + + + +## qrtz_corn_triggers + +存放 cron 类型的触发器 + +```sql +CREATE TABLE `qrtz_cron_triggers` ( + `SCHED_NAME` varchar(120) COLLATE utf8_bin NOT NULL COMMENT '集群名', + `TRIGGER_NAME` varchar(200) COLLATE utf8_bin NOT NULL COMMENT '调度器名,qrtz_triggers表trigger_name的外键', + `TRIGGER_GROUP` varchar(200) COLLATE utf8_bin NOT NULL COMMENT 'qrtz_triggers表trigger_group的外键', + `CRON_EXPRESSION` varchar(200) COLLATE utf8_bin NOT NULL COMMENT 'cron表达式', + `TIME_ZONE_ID` varchar(80) COLLATE utf8_bin DEFAULT NULL COMMENT '时区ID', + PRIMARY KEY (`SCHED_NAME`,`TRIGGER_NAME`,`TRIGGER_GROUP`), + CONSTRAINT `qrtz_cron_triggers_ibfk_1` FOREIGN KEY (`SCHED_NAME`, `TRIGGER_NAME`, `TRIGGER_GROUP`) REFERENCES `qrtz_triggers` (`SCHED_NAME`, `TRIGGER_NAME`, `TRIGGER_GROUP`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='存放cron类型的触发器'; +``` + + + +## qrtz_simprop_triggers + +存储 `CalendarIntervalTrigger` 和 `DailyTimeIntervalTrigger` 两种类型的触发器，使用 `CalendarIntervalTrigger` 做如下配置： + +`CalendarIntervalTrigger` 没有对应的 FactoryBean，直接设置实现类 `CalendarIntervalTriggerImpl`；指定的重复周期是 1，默认单位是天，也就是每天执行一次。 + +```sql +CREATE TABLE `qrtz_simprop_triggers` ( + `SCHED_NAME` varchar(120) COLLATE utf8_bin NOT NULL, + `TRIGGER_NAME` varchar(200) COLLATE utf8_bin NOT NULL, + `TRIGGER_GROUP` varchar(200) COLLATE utf8_bin NOT NULL, + `STR_PROP_1` varchar(512) COLLATE utf8_bin DEFAULT NULL, + `STR_PROP_2` varchar(512) COLLATE utf8_bin DEFAULT NULL, + `STR_PROP_3` varchar(512) COLLATE utf8_bin DEFAULT NULL, + `INT_PROP_1` int(11) DEFAULT NULL, + `INT_PROP_2` int(11) DEFAULT NULL, + `LONG_PROP_1` bigint(20) DEFAULT NULL, + `LONG_PROP_2` bigint(20) DEFAULT NULL, + `DEC_PROP_1` decimal(13,4) DEFAULT NULL, + `DEC_PROP_2` decimal(13,4) DEFAULT NULL, + `BOOL_PROP_1` varchar(1) COLLATE utf8_bin DEFAULT NULL, + `BOOL_PROP_2` varchar(1) COLLATE utf8_bin DEFAULT NULL, + PRIMARY KEY (`SCHED_NAME`,`TRIGGER_NAME`,`TRIGGER_GROUP`), + CONSTRAINT `qrtz_simprop_triggers_ibfk_1` FOREIGN KEY (`SCHED_NAME`, `TRIGGER_NAME`, `TRIGGER_GROUP`) REFERENCES `qrtz_triggers` (`SCHED_NAME`, `TRIGGER_NAME`, `TRIGGER_GROUP`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='存储CalendarIntervalTrigger和DailyTimeIntervalTrigger两种类型的触发器'; +``` + + + +## qrtz_blob_triggers + +自定义的 triggers 使用 blog 类型进行存储，非自定义的 triggers 不会存放在此表中，Quartz 提供的 triggers 包括：CronTrigger，CalendarIntervalTrigger，DailyTimeIntervalTrigger 以及 SimpleTrigger。 + +```sql +CREATE TABLE `qrtz_blob_triggers` ( + `SCHED_NAME` varchar(120) COLLATE utf8_bin NOT NULL, + `TRIGGER_NAME` varchar(200) COLLATE utf8_bin NOT NULL, + `TRIGGER_GROUP` varchar(200) COLLATE utf8_bin NOT NULL, + `BLOB_DATA` blob, + PRIMARY KEY (`SCHED_NAME`,`TRIGGER_NAME`,`TRIGGER_GROUP`), + CONSTRAINT `qrtz_blob_triggers_ibfk_1` FOREIGN KEY (`SCHED_NAME`, `TRIGGER_NAME`, `TRIGGER_GROUP`) REFERENCES `qrtz_triggers` (`SCHED_NAME`, `TRIGGER_NAME`, `TRIGGER_GROUP`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin; +``` + + + +## qrtz_calendars + +以 Blob 类型存储 Quartz 的 Calendar 信息 + +```sql +CREATE TABLE qrtz_calendars +( + SCHED_NAME VARCHAR(120) NOT NULL, + CALENDAR_NAME VARCHAR(200) NOT NULL, + CALENDAR BLOB NOT NULL, + CONSTRAINT QRTZ_CALENDARS_PK PRIMARY KEY (SCHED_NAME,CALENDAR_NAME) +); +``` + + + +## qrtz_paused_trigger_grps + +存储已暂停的 Trigger 组的信息 + +```sql +CREATE TABLE qrtz_paused_trigger_grps +( + SCHED_NAME VARCHAR(120) NOT NULL, + TRIGGER_GROUP VARCHAR(200) NOT NULL, + CONSTRAINT QRTZ_PAUSED_TRIG_GRPS_PK PRIMARY KEY (SCHED_NAME,TRIGGER_GROUP) +); +``` + + + +## qrtz_scheduler_state + +存储所有节点的 scheduler，会定期检查 scheduler 是否失效 + +```sql +CREATE TABLE `qrtz_scheduler_state` ( + `SCHED_NAME` varchar(120) COLLATE utf8_bin NOT NULL COMMENT '调度器名称，集群名', + `INSTANCE_NAME` varchar(200) COLLATE utf8_bin NOT NULL COMMENT '集群中实例ID，配置文件中org.quartz.scheduler.instanceId的配置', + `LAST_CHECKIN_TIME` bigint(13) NOT NULL COMMENT '上次检查时间', + `CHECKIN_INTERVAL` bigint(13) NOT NULL COMMENT '检查时间间隔', + PRIMARY KEY (`SCHED_NAME`,`INSTANCE_NAME`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='调度器状态'; +``` + + + +## qrtz_fired_triggers + +存储已经触发的 trigger 相关信息，trigger 随着时间的推移状态发生变化，直到最后 trigger 执行完成，从表中被删除 + +```sql +CREATE TABLE `qrtz_fired_triggers` ( + `SCHED_NAME` varchar(120) COLLATE utf8_bin NOT NULL COMMENT '调度器名称，集群名', + `ENTRY_ID` varchar(95) COLLATE utf8_bin NOT NULL COMMENT '运行Id', + `TRIGGER_NAME` varchar(200) COLLATE utf8_bin NOT NULL COMMENT '触发器名', + `TRIGGER_GROUP` varchar(200) COLLATE utf8_bin NOT NULL COMMENT '触发器组', + `INSTANCE_NAME` varchar(200) COLLATE utf8_bin NOT NULL COMMENT '集群中实例ID', + `FIRED_TIME` bigint(13) NOT NULL COMMENT '触发时间', + `SCHED_TIME` bigint(13) NOT NULL, + `PRIORITY` int(11) NOT NULL COMMENT '线程优先级', + `STATE` varchar(16) COLLATE utf8_bin NOT NULL COMMENT '状态', + `JOB_NAME` varchar(200) COLLATE utf8_bin DEFAULT NULL COMMENT '任务名', + `JOB_GROUP` varchar(200) COLLATE utf8_bin DEFAULT NULL COMMENT '任务组', + `IS_NONCONCURRENT` varchar(1) COLLATE utf8_bin DEFAULT NULL COMMENT '是否并行', + `REQUESTS_RECOVERY` varchar(1) COLLATE utf8_bin DEFAULT NULL COMMENT '是否恢复', + PRIMARY KEY (`SCHED_NAME`,`ENTRY_ID`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin COMMENT='存储与已触发的 Trigger 相关的状态信息，以及相联 Job 的执行信息'; +``` + + + +## qrtz_locks + +存储程序的悲观锁的信息(假如使用了悲观锁) + +Quartz 提供的锁表，为多个节点调度提供分布式锁，实现分布式调度，默认有 2 个锁： + +- STATE_ACCESS 主要用在 scheduler 定期检查是否有效的时候，保证只有一个节点去处理已经失效的 scheduler。 +- TRIGGER_ACCESS 主要用在 TRIGGER 被调度的时候，保证只有一个节点去执行调度。 + +```SQL +CREATE TABLE qrtz_locks +( + SCHED_NAME VARCHAR(120) NOT NULL, + LOCK_NAME VARCHAR(40) NOT NULL, + CONSTRAINT QRTZ_LOCKS_PK PRIMARY KEY (SCHED_NAME,LOCK_NAME) +); +``` \ No newline at end of file diff --git a/docs/framework/Quartz/Quartz-Using.md b/docs/framework/Quartz/Quartz-Using.md new file mode 100755 index 0000000000..d23ec1c197 --- /dev/null +++ b/docs/framework/Quartz/Quartz-Using.md @@ -0,0 +1,3 @@ + + + \ No newline at end of file diff --git a/docs/framework/Quartz/Quartz.md b/docs/framework/Quartz/Quartz.md new file mode 100755 index 0000000000..0ab2c1aa61 --- /dev/null +++ b/docs/framework/Quartz/Quartz.md @@ -0,0 +1,266 @@ +# Quartz + +## 一、初识 Quartz + +### 1.1 背景 + +#### 1.1 Quartz概念 + +Quartz 是 OpenSymphony 开源组织的一个 Java 开源项目， 在 2009 被 Terracotta 收购。[Quartz官网](http://www.quartz-scheduler.org/) + +#### 1.2 Quartz 任务调度主要元素 + +- Job（任务）：具体要执行的业务逻辑，比如：发送短信、发送邮件、访问数据库、同步数据等。 +- Trigger（触发器）：用来定义 Job 触发条件、触发时间，触发间隔，终止时间等。 +- Scheduler（调度器）：Scheduler 启动 Trigger 去执行 Job + +其中 Trigger，Job 是元数据，Scheduler 才是任务调度的控制器。 + +#### 1.3 Quartz特点 + +- 强大的调度功能，例如支持多样的调度方式 + +- 灵活的应用方式，例如支持任务和调度的多种组合方式 + +- 分布式和集群功能，在被Terracotta收购后，在Quartz的基础上的拓展 + +#### 1.4 Quartz 基本元素关系图 + +![Quartz基本元素关系图](https://imgconvert.csdnimg.cn/aHR0cDovL2ltZy5ibG9nLmNzZG4ubmV0LzIwMTgwMTE3MTAzNjE3Mzcw?x-oss-process=image/format,png) + + + +#### 1.5 hello world + +```java +public class HelloJob implements Job { + @Override + public void execute(JobExecutionContext context) throws JobExecutionException { + Object trigger1 = context.getTrigger().getJobDataMap().get("trigger-1"); + Object trigger2 = context.getTrigger().getJobDataMap().get("trigger-2"); + Object job1 = context.getJobDetail().getJobDataMap().get("job-1"); + Object job2 = context.getJobDetail().getJobDataMap().get("job-2"); + Object sv = null; + try { + sv = context.getScheduler().getContext().get("schedulerKey"); + } catch (SchedulerException e) { + e.printStackTrace(); + } + System.out.println(trigger1+":"+trigger2); + System.out.println(job1+":"+job2); + System.out.println(sv); + System.out.println("hello:"+ LocalDateTime.now()); + } +} +``` + +```java +public class QuartzTest { + public static void main(String[] args) throws SchedulerException { + //创建一个scheduler + Scheduler scheduler = StdSchedulerFactory.getDefaultScheduler(); + scheduler.getContext().put("schedulerKey", "schedulerValue"); + + //创建一个Trigger + Trigger trigger = TriggerBuilder.newTrigger() + .withIdentity("myTrigger", "MyGroup") + .usingJobData("trigger-1", "this is trigger1") + .withSchedule(SimpleScheduleBuilder.simpleSchedule().withIntervalInSeconds(5) + .repeatForever()).build(); + trigger.getJobDataMap().put("trigger-2", "this is trigger1"); + + //创建一个job + JobDetail job = JobBuilder.newJob(HelloJob.class) + .usingJobData("job-1", "this is job1") + .withIdentity("MyJob", "MyGroup").build(); + job.getJobDataMap().put("job-2", "this is job2"); + + //注册trigger并启动scheduler + scheduler.scheduleJob(job,trigger); + scheduler.start(); + + } +} +``` + +Quartz API的关键接口是： + +- `Scheduler` - 与调度程序交互的主要API。 +- `Job` - 你想要调度器执行的任务组件需要实现的接口 +- `JobDetail` - 用于定义作业的实例。 +- `Trigger`（即触发器） - 定义执行给定作业的计划的组件。 +- `JobBuilder` - 用于定义/构建 `JobDetail` 实例，用于定义作业的实例。 +- `TriggerBuilder` - 用于定义/构建触发器实例。 +- `Scheduler` 的生命期，从 `SchedulerFactory` 创建它时开始，到 `Scheduler`调用 `shutdown()` 方法时结束；`Scheduler` 被创建后，可以增加、删除和列举 `Job` 和 `Trigger`，以及执行其它与调度相关的操作（如暂停 `Trigge`r）。但是，`Scheduler` 只有在调用 `start()` 方法后，才会真正地触发 trigger（即执行 job) + + + +## 二、Trigger(触发器) + +#### 2.1 Trigger 定义 + +Trigger 也即触发器，用于定义任务调度时间规则 + +#### 2.2 Trigger 属性 + +1. startTime 和 endTime + 所有的 Trigger 都包含 startTime、endTime 这两个属性 +2. 优先级(Priority) + 触发器的优先级值默认为 5，不过注意优先级是针对同一时刻来说的，在同一时刻优先级高的先触发。假如一个触发器被执行时间为3:00，另外一个为 3:01，那么肯定是先执行时间为 3:00 的触发器。 +3. 错失触发(Misfire)策略 + 在任务调度中，并不能保证所有的触发器都会在指定时间被触发，假如 Scheduler 资源不足或者服务器重启的情况，就好发生错失触发的情况。 + +#### 2.3 Trigger 类型 + +在任务调度 Quartz 中，Trigger 主要的触发器有：SimpleTrigger，CalendarIntervelTrigger，DailyTimeIntervalTrigger，CronTrigger，注意，本文所介绍的触发器都是基于 Quartz2.2.x 版本的，不同版本，触发器类型略有不同。 + +##### 2.3.1 SimpleTrigger + +SimpleTrigger 是一种最基本的触发器，指定从某一个时间开始，以一定的时间间隔执行的任务。 + +```java +simpleSchedule() + //.withIntervalInHours(1) //每小时执行一次 + .withIntervalInMinutes(1) //每分钟执行一次 + //.repeatForever() //次数不限 + .withRepeatCount(10) //次数为10次 + .build();//构建 +``` + +##### 2.3.2 CalendarIntervalTrigger + +CalendarIntervalTrigger 和 SimpleTrigger 不同的是，SimpleTrigger 指定的时间间隔为毫秒，CalendarIntervalTrigger 支持的间隔单位有秒，分钟，小时，天，月，年，星期。 + +```java +calendarIntervalSchedule() + .withIntervalInDays(1) //每天执行一次 + //.withIntervalInWeeks(1) //每周执行一次 + .build(); +``` + +##### 2.3.3 DailyTimeIntervalTrigger + +DailyTimeIntervalTrigger 和 SimpleTrigger 不同的是不仅可以支持 SimpleTrigger 支持时间间隔类型，而且还支持指定星期。 + +```java +dailyTimeIntervalSchedule() + .startingDailyAt(TimeOfDay.hourAndMinuteOfDay(9, 0)) //第天9：00开始 + .endingDailyAt(TimeOfDay.hourAndMinuteOfDay(15, 0)) //15：00 结束 + .onDaysOfTheWeek(MONDAY,TUESDAY,WEDNESDAY,THURSDAY,FRIDAY) //周一至周五执行 + .withIntervalInHours(1) //每间隔1小时执行一次 + .withRepeatCount(100) //最多重复100次（实际执行100+1次） + .build(); +``` + +##### 2.3.4 CronTrigge + +CronTrigger 适合于更复杂的任务，它支持 Linux Cron 的语法。CronTrigger 覆盖了以上三种 Trigger 的大部分功能。 + +CronTrigger 的属性只有 Cron表达式，Cron 表达式需要程序员自己编写，比较复杂 + +```java +cronSchedule("0 0/3 9-15 * * ?") // 每天9:00-15:00，每隔3分钟执行一次 + .build(); + +cronSchedule("0 30 9 ? * MON") // 每周一，9:30执行一次 +.build(); + +weeklyOnDayAndHourAndMinute(MONDAY,9, 30) //等同于 0 30 9 ? * MON + .build(); +``` + +Cron表达式 + +| 位置 | 时间域 | 允许值 | 特殊值 | +| ---- | ---------- | ------ | -------------- | +| 1 | 秒 | 0-59 | ,- * / | +| 2 | 分钟 | 0-59 | ,- * / | +| 3 | 小时 | 0-23 | ,- * / | +| 4 | 日期 | 1-31 | ,- * ? / L W C | +| 5 | 月份 | 1-12 | ,- * / | +| 6 | 星期 | 1-7 | ,- * ? / L C # | +| 7 | 年份(可选) | 1-31 | ,- * / | + +- 星号()：可用在所有字段中，表示对应时间域的每一个时刻，例如， 在分钟字段时，表示“每分钟”； + +- 问号（?）：该字符只在日期和星期字段中使用，它通常指定为“无意义的值”，相当于点位符； + +- 减号(-)：表达一个范围，如在小时字段中使用“10-12”，则表示从10到12点，即10,11,12； + +- 逗号(,)：表达一个列表值，如在星期字段中使用“MON,WED,FRI”，则表示星期一，星期三和星期五； + +- 斜杠(/)：x/y表达一个等步长序列，x为起始值，y为增量步长值。如在分钟字段中使用0/15，则表示为0,15,30和45秒，而5/15在分钟字段中表示5,20,35,50，你也可以使用*/y，它等同于0/y； + +- L：该字符只在日期和星期字段中使用，代表“Last”的意思，但它在两个字段中意思不同。L在日期字段中，表示这个月份的最后一天，如一月的31号，非闰年二月的28号；如果L用在星期中，则表示星期六，等同于7。但是，如果L出现在星期字段里，而且在前面有一个数值X，则表示“这个月的最后X天”，例如，6L表示该月的最后星期五； + +- W：该字符只能出现在日期字段里，是对前导日期的修饰，表示离该日期最近的工作日。例如15W表示离该月15号最近的工作日，如果该月15号是星期六，则匹配14号星期五；如果15日是星期日，则匹配16号星期一；如果15号是星期二，那结果就是15号星期二。但必须注意关联的匹配日期不能够跨月，如你指定1W，如果1号是星期六，结果匹配的是3号星期一，而非上个月最后的那天。W字符串只能指定单一日期，而不能指定日期范围； + +- LW组合：在日期字段可以组合使用LW，它的意思是当月的最后一个工作日； + +- 井号(#)：该字符只能在星期字段中使用，表示当月某个工作日。如6#3表示当月的第三个星期五(6表示星期五，#3表示当前的第三个)，而4#5表示当月的第五个星期三，假设当月没有第五个星期三，忽略不触发； + +- C：该字符只在日期和星期字段中使用，代表“Calendar”的意思。它的意思是计划所关联的日期，如果日期没有被关联，则相当于日历中所有日期。例如5C在日期字段中就相当于日历5日以后的第一天。1C在星期字段中相当于星期日后的第一天。 + +Cron表达式对特殊字符的大小写不敏感，对代表星期的缩写英文大小写也不敏感。 + + + +## 三、Scheduler(任务调度器) + +### 3.1 Scheduler定义 + +Scheduler 就是任务调度控制器，Scheduler 有两个重要组件：ThreadPool 和 JobStore。 + +注意：Job 和 Trigger 需要存储下来才可以被使用。 + +ThreadPool 就是线程池，所有的任务都会被线程池执行 + +JobStore 是来存储运行时信息的，包括 Trigger，Scheduler，JobDetail，业务锁等等。JobStore 实现有 RAMJob(内存实现)，JobStoreTX(JDBC，事务由Quartz管理），JobStoreCMT(JDBC，使用容器事务)，ClusteredJobStore(集群实现)等等 + +注意：Job和Trigger需要存储下来才可以被使用。 + +#### 3.2 Schedule种类 + +Schedule有三种： + +- StdScheduler + +- RemoteMBeanScheduler + +- RemoteScheduler + +其中StdScheduler最常用。 + +#### 3.3 Schedule工厂 + +Schedule是由Schedule工厂创建的，有DirectSchedulerFactory或者StdSchedulerFactory，StdSchedulerFactory使用比较多 + +### 四、 Job(任务) + +#### 4.1 Job定义 + +Job：也就是表示被调度的任务.JobDetail是任务的定义，而Job是任务的执行逻辑。在JobDetail里会引用一个Job Class定义 + +#### 4.2 Job类型 + +Job有两种类型：无状态的(stateless)和有状态的(stateful) + +> 区别在于：对于同一个Trigger来说，有状态的Job不能异步执行，也就是说需要等上一个任务Job执行完成后，才可以触发下一次执行。 + +#### 4.3 Job属性 + +Job的属性有两种：volatility和durability + +> volatility表示任务是否持久化到数据库存储; +> durability表示在没有Trigger关联的条件下是否保留。 +> volatility和durability都是boolean类型。 + +### 五、Quartz线程 + +#### 5.1 Quartz线程分类 + +在Quartz中，线程分为Scheduler调度线程和任务执行线程。 +Scheduler调度线程主要有：执行常规调度的线程和执行misfired trigger的线程。 + +> 执行常规调度的线程(Regular Scheduler Thread)：轮询查询存储的所有触发器，到达触发时间，就从线程池获取一个空闲的线程，执行与触发器关联的任务。 +> 执行错失调度的线程(Misfire Scheduler Thread)：Misfire线程扫描所有的触发器，检查是否有misfired的线程，也就是没有被执行错过的线程，有的话根据misfire的策略分别处理。 \ No newline at end of file diff --git "a/docs/framework/Quartz/\346\234\252\345\221\275\345\220\215 2.md" "b/docs/framework/Quartz/\346\234\252\345\221\275\345\220\215 2.md" new file mode 100755 index 0000000000..e69de29bb2 diff --git a/docs/framework/README.md b/docs/framework/README.md new file mode 100644 index 0000000000..939939780f --- /dev/null +++ b/docs/framework/README.md @@ -0,0 +1,2 @@ +![img](https://www.baeldung.com/wp-content/uploads/2019/01/Separator-Laptop-karken.jpg) + diff --git a/docs/framework/spring/@Value.md b/docs/framework/Spring/@Value.md similarity index 100% rename from docs/framework/spring/@Value.md rename to docs/framework/Spring/@Value.md diff --git a/docs/framework/Spring/My-Spring.md b/docs/framework/Spring/My-Spring.md new file mode 100644 index 0000000000..c3a531a453 --- /dev/null +++ b/docs/framework/Spring/My-Spring.md @@ -0,0 +1,34 @@ +# 手写 Spring + +![image-20201106173130849](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20201106173132.png) +配置阶段：主要是完成application.xml配置和Annotation配置。 + +初始化阶段：主要是加载并解析配置信息，然后，初始化IOC容器，完成容器的DI操作，已经完成HandlerMapping的初始化。 + +运行阶段：主要是完成Spring容器启动以后，完成用户请求的内部调度，并返回响应结果。 +======= + + +```undefined + ⑴ 用户发送请求至前端控制器DispatcherServlet + + ⑵ DispatcherServlet收到请求调用HandlerMapping处理器映射器。 + + ⑶ 处理器映射器根据请求url找到具体的处理器，生成处理器对象及处理器拦截器(如果有则生成)一并返回给DispatcherServlet。 + + ⑷ DispatcherServlet通过HandlerAdapter处理器适配器调用处理器 + + ⑸ 执行处理器(Controller，也叫后端控制器)。 + + ⑹ Controller执行完成返回ModelAndView + + ⑺ HandlerAdapter将controller执行结果ModelAndView返回给DispatcherServlet + + ⑻ DispatcherServlet将ModelAndView传给ViewReslover视图解析器 + + ⑼ ViewReslover解析后返回具体View + + ⑽ DispatcherServlet对View进行渲染视图（即将模型数据填充至视图中）。 + + ⑾ DispatcherServlet响应用户。 +``` diff --git a/docs/framework/Spring/Spring-AOP.md b/docs/framework/Spring/Spring-AOP.md new file mode 100644 index 0000000000..f283a13d1f --- /dev/null +++ b/docs/framework/Spring/Spring-AOP.md @@ -0,0 +1,581 @@ +## 一、AOP 前奏 + +假设我们原本有个计算器的程序，现在有这样的需求： + +1. 在程序执行期间记录日志 +2. 希望计算器只处理正数运算 + +啥也不说了，闷头干~ + +```java +public interface ArithmeticCalculator { + void add(int i,int j); + void sub(int i,int j); +} +``` + +```java +public void add(int i, int j) { + if(i < 0 || j < 0){ + throw new IllegalArgumentException("Positive numbers only"); + } + System.out.println("The method add begins with " + i + "," + j); + int result = i + j; + System.out.println("result: "+ result); + System.out.println("The method add ends with " + i + "," + j); + } + + public void sub(int i, int j) { + if(i < 0 || j < 0){ + throw new IllegalArgumentException("Positive numbers only"); + } + System.out.println("The method sub begins with " + i + "," + j); + int result = i - j; + System.out.println("result: "+ result); + System.out.println("The method sub ends with " + i + "," + j); + } +``` + +### 问题 + +这么一通干，存在的问题： + +- 代码混乱：越来越多的非业务需求(日志和验证等)加入后，原有的业务方法急剧膨胀。每个方法在处理核心逻辑的同时还必须兼顾其他多个关注点 +- 代码分散：以日志需求为例，只是为了满足这个单一需求，就不得不在多个模块（方法）里多次重复相同的日志代码。如果日志需求发生变化，必须修改所有模块 + +### 解决 + +这里就可以用动态代理来解决。可以参考之前的文章: [《面试官问 Spring AOP 中两种代理模式的区别，我懵逼了》](https://mp.weixin.qq.com/s/U7eR5Mpu4VBbtPP1livLnA) + +代理设计模式的原理：**使用一个代理将对象包装起来**，然后用该代理对象取代原始对象。任何对原始对象的调用都要通过代理。代理对象决定是否以及何时将方法调用转到原始对象上。 + +![](https://img.starfish.ink/spring/spring-aop-log.svg) + + + +1、我们用 JDK 动态代理来实现日志功能 + +> JDK 动态代理主要涉及到 `java.lang.reflect` 包中的两个类：Proxy 和 InvocationHandler。 +> +> InvocationHandler 是一个接口，通过实现该接口定义横切逻辑，并通过反射机制调用目标类的代码，动态将横切逻辑和业务逻辑编制在一起。 +> +> Proxy 利用 InvocationHandler 动态创建一个符合某一接口的实例，生成目标类的代理对象。 +> +> 原理是使用反射机制。 + +```java +public class LoggingHandler implements InvocationHandler { + + /** + * 被代理的目标对象 + */ + private Object proxyObj; + + public Object invoke(Object proxy, Method method, Object[] args) throws Throwable { + System.out.println("The method " + method.getName() + " begins with " + Arrays.toString(args)); + Object result = method.invoke(proxyObj,args); + System.out.println("The method " + method.getName() + " ends with " + Arrays.toString(args)); + return result; + } + + public Object createProxy(Object proxyObj){ + this.proxyObj = proxyObj; + //返回一个代理对象 + return Proxy.newProxyInstance(proxyObj.getClass().getClassLoader(), + proxyObj.getClass().getInterfaces(),this); + } +} +``` + +2、再用 CGLib 动态代理实现校验功能 + +> CGLIB 全称为 Code Generation Library，是一个强大的高性能，高质量的代码生成类库，可以在运行期扩展 Java 类与实现 Java 接口，CGLib 封装了 asm，可以再运行期动态生成新 的 class。 +> +> 和 JDK 动态代理相比较：JDK 创建代理有一个限制，就是只能为接口创建代理实例， 而对于没有通过接口定义业务方法的类，则可以通过 CGLIB 创建动态代理。 +> +> 需要导入 cglib-nodep-***.jar +> + +```java +public class ValidationHandler implements MethodInterceptor { + + /** + * 被代理的目标对象 + */ + private Object targetObject; + + + public Object createProxy(Object targetObject){ + this.targetObject = targetObject; + Enhancer enhancer = new Enhancer(); + //设置代理目标 + enhancer.setSuperclass(targetObject.getClass()); + //设置回调 + enhancer.setCallback(this); + return enhancer.create(); + } + + /** + * 在代理实例上处理方法调用并返回结果 + * @param o ： 代理类 + * @param method ：被代理的方法 + * @param objects ：该方法的参数数组 + * @param methodProxy + */ + public Object intercept(Object o, Method method, Object[] objects, MethodProxy methodProxy) throws Throwable { + Object result = null; + + for (Object object : objects) { + if(Integer.parseInt(object.toString()) < 0){ + throw new IllegalArgumentException("Positive numbers only"); + } + } + //执行目标对象的方法 + result = methodProxy.invoke(targetObject,objects); + return result; + } +} +``` + +3、使用 + +```java +public static void main(String[] args) { + + ValidationHandler validationHandler = new ValidationHandler(); + LoggingHandler loggingHandler = new LoggingHandler(); + + //cglib 要求目标对象是个单独的对象 + ArithmeticCalculatorImpl calculatorImpl = new ArithmeticCalculatorImpl(); + validationHandler.createProxy(calculatorImpl); + + //JDK 动态代理要求目标对象是接口 + ArithmeticCalculator calculator = new ArithmeticCalculatorImpl(); + ArithmeticCalculator loggingProxy = (ArithmeticCalculator) loggingHandler.createProxy(calculator); + loggingProxy.add(1,2); + + ArithmeticCalculatorImpl validationProxy = (ArithmeticCalculatorImpl) validationHandler.createProxy(calculatorImpl); + validationProxy.sub(-1,2); + } +``` + + + +## 二、AOP + +- AOP(Aspect-Oriented Programming，面向切面编程)： 是一种新的方法论，是对传统 OOP(Object-Oriented Programming，面向对象编程) 的补充 +- AOP 的主要编程对象是切面(aspect)，而切面模块化横切关注点 +- 在应用 AOP 编程时，仍然需要定义公共功能，但可以明确的定义这个功能在哪里，以什么方式应用，并且不必修改受影响的类。这样一来横切关注点就被模块化到特殊的对象(切面)里 +- AOP 的好处： + + - 每个事物逻辑位于一个位置，代码不分散，便于维护和升级 + + - 业务模块更简洁，只包含核心业务代 + + +![](https://img.starfish.ink/spring/spring-aop-demo.svg) + +### AOP 核心概念 + +- 切面（aspect）：类是对物体特征的抽象，切面就是对横切关注点的抽象 + +- 横切关注点：对哪些方法进行拦截，拦截后怎么处理，这些关注点称之为横切关注点 + +- 连接点（joinpoint）：程序执行的某个特定位置：如类某个方法调用前、调用后、方法抛出异常后等。连接点由两个信息确定：方法表示的程序执行点；相对点表示的方位。例如 ArithmethicCalculator#add() 方法执行前的连接点，执行点为 ArithmethicCalculator#add()； 方位为该方法执行前的位置。 + + 被拦截到的点，因为 Spring 只支持方法类型的连接点，所以在 Spring 中连接点指的就是被拦截到的方法，实际上连接点还可以是字段或者构造器 + +- 切入点（pointcut）：每个类都拥有多个连接点，例如 ArithmethicCalculator 的所有方法实际上都是连接点，即连接点是程序类中客观存在的事务。AOP 通过切点定位到特定的连接点。类比：连接点相当于数据库中的记录，切点相当于查询条件。切点和连接点不是一对一的关系，一个切点匹配多个连接点，切点通过 `org.springframework.aop.Pointcut` 接口进行描述，它使用类和方法作为连接点的查询条件。 + +- 通知（advice）：所谓通知指的就是指拦截到连接点之后要执行的代码，通知分为前置、后置、 异常、最终、环绕通知五类 + +- 目标对象（Target）：代理的目标对象 +- 代理（Proxy）：向目标对象应用通知之后创建的对象 + +- 织入（weave）：将切面应用到目标对象并导致代理对象创建的过程 + +- 引入（introduction）：在不修改代码的前提下，引入可以在运行期为类动态地添加一些方法或字段。 + +> "横切"的技术，剖解开封装的对象内部，并将那些影响了多个类的公共行为封装到一个可重用模块， 并将其命名为"Aspect"，即切面。所谓"切面"，简单说就是那些与业务无关，却为业务模块所共同调用的逻辑或责任封装起来，便于减少系统的重复代码，降低模块之间的耦合度，并有利于未来的可操作性和可维护性。 +> +> 使用"横切"技术，AOP 把软件系统分为两个部分：**核心关注点**和**横切关注点**。业务处理的主要流程是核心关注点，与之关系不大的部分是横切关注点。 +> +> 横切关注点的一个特点是，他们经常发生在核心关注点的多处，而各处基本相似，比如权限认证、日志、事物。 +> +> AOP 的作用在于分离系统中的各种关注点，将核心关注点和横切关注点分离开来。 +> +> AOP 主要应用场景有： +> +> - Authentication 权限 +> - Caching 缓存 +> - Context passing 内容传递 +> - Error handling 错误处理 +> - Lazy loading 懒加载 +> - Debugging 调试 +> - logging, tracing, profiling and monitoring 记录跟踪、优化、校准 +> - Performance optimization 性能优化 +> - Persistence 持久化 +> - Resource pooling 资源池 +> - Synchronization 同步 +> - Transactions 事务 + + + +## 三、Spring AOP + +- **AspectJ**：Java 社区里最完整最流行的 AOP 框架 +- 在 Spring2.0 以上版本中，可以使用基于 AspectJ 注解或基于 XML 配置的 AOP + + + +**在 Spring 中启用 AspectJ 注解支持** + +- 要在 Spring 应用中使用 AspectJ 注解，必须在 classpath 下包含 AspectJ 类库: + + aopalliance.jar、aspectj.weaver.jar 和 spring-aspects.jar + +- 将 aop Schema 添加到 \ 根元素中 + +- 要在 Spring IOC 容器中启用 AspectJ 注解支持，只要在 Bean 配置文件中定义一个空的 XML 元素 \ + +- 当 Spring IOC 容器侦测到 Bean 配置文件中的 \ 元素时，会自动为与 AspectJ 切面匹配的 Bean 创建代理 + + + +### 3.1 用 AspectJ 注解声明切面 + +- 要在 Spring 中声明 AspectJ 切面，只需要在 IOC 容器中将切面声明为 Bean 实例。当在 Spring IOC 容器中初始化 AspectJ 切面之后，Spring IOC 容器就会为那些与 AspectJ 切面相匹配的 Bean 创建代理 +- 在 AspectJ 注解中，切面只是一个带有 @Aspect 注解的 Java 类 +- 通知是标注有某种注解的简单的 Java 方法 +- AspectJ 支持 5 种类型的通知注解： + + - @Before：前置通知，在方法执行之前执行 + - @After：后置通知，在方法执行之后执行 + - @AfterRunning：返回通知，在方法返回结果之后执行 + - @AfterThrowing：异常通知，在方法抛出异常之后 + - @Around：环绕通知，围绕着方法执行 + + + +#### 1、前置通知 + +前置通知：在方法执行之前执行的通知 + +前置通知使用 @Before 注解，并将切入点表达式的值作为注解值 + +- @Aspect：标识这个类是一个切面 + +- @Before：标识这个方法是个前置通知 + - execution 代表切点表达式，表示执行 ArithmeticCalculator 接口的 `add()` 方法 + - \* 代表匹配任意修饰符及任意返回值 + - 参数列表中的 .. 代表匹配任意数量的参数 + +```java +@Aspect +@Component +public class LogAspect { + + private Logger log = LoggerFactory.getLogger(this.getClass()); + + /** + * 前置通知，目标方法调用前被调用 + */ + @Before("execution(* priv.starfish.aop.aspect.Calculator.add(..))") + public void beforeAdvice(){ + log.info("The method add begins"); + } + +} +``` + + + +##### 利用方法签名编写 AspectJ 切入点表达式 + +最典型的切入点表达式是根据方法的签名来匹配各种方法: + +- `execution *priv.starfish.aop.ArithmeticCalculator.*(..)`： 匹配 ArithmeticCalculator 中声明的所有方法，第一个 * + + 代表任意修饰符及任意返回值，第二个 * 代表任意方法，.. 代表匹配任意数量的参数。若目标类与接口与该切面在同一个包中，可以省略包名 + +- `execution public * ArithmeticCalculator.*(..)`： 匹配 ArithmeticCalculator 接口的所有公有方法 + +- `execution public double ArithmeticCalculator.*(..)`： 匹配 ArithmeticCalculator 中返回 double 类型数值的方法 + +- `execution public double ArithmeticCalculator.*(double, ..)`： 匹配第一个参数为 double 类型的方法， .. 匹配任意数量任意类型的参数 + +- `execution public double ArithmeticCalculator.*(double, double)`： 匹配参数类型为 double, double 的方法 + + + +##### 合并切入点表达式 + +- AspectJ 可以使用 且（&&）、或（||）、非（！）来组合切入点表达式 +- 在 Schema风格下，由于在 XML 中使用“&&”需要使用转义字符“\&&”来代替，所以很不方便，因此 Spring AOP 提供了and、or、not 来代替 &&、||、！。 + +```java +@Pointcut("execution(* *.add(int,int)) || execution(* *.sub(int,int))") +public void loggingOperation(){}; +``` + + + +##### 让通知访问当前连接点的细节 + +可以在通知方法中声明一个类型为 JoinPoint 的参数，然后就能访问连接细节了，比如方法名称和参数值等 + +```java +@Before("execution(* priv.starfish.aop.aspect.Calculator.add(..))") +public void beforeAdvice(JoinPoint joinPoint) { + log.info("The method " + joinPoint.getSignature().getName() + + "() begins with " + Arrays.toString(joinPoint.getArgs())); +} +``` + + + +#### 2、后置通知 + +- 后置通知是在连接点完成之后执行的，即连接点返回结果或者抛出异常的时候，下面的后置通知记录了方法的终止 +- 一个切面可以包括一个或者多个通知 + +```java +@Aspect +@Component +public class LogAspect { +private Logger log = LoggerFactory.getLogger(this.getClass()); + + /** + * 前置通知，目标方法调用前被调用 + */ + @Before("execution(* priv.starfish.aop.aspect.Calculator.add(..))") + public void beforeAdvice(){ + log.info("The method add begins"); + } + + /** + * 后置通知，目标方法执行完执行 + */ + @After("execution( * *.*(..))") + public void afterAdvice(JoinPoint joinPoint){ + log.info("The method " + joinPoint.getSignature().getName() + "() ends"); + } + +} +``` + + + +#### 3、返回通知 + +- 无论连接点是正常返回还是抛出异常，后置通知都会执行。如果只想在连接点返回的时候记录日志，应该使用返回通知代替后置通知 + + > **在返回通知中访问连接点的返回值** + > + > - 在返回通知中，只要将 returning 属性添加到 @AfterReturning 注解中，就可以访问连接点的返回值。该属性的值即为用来传入返回值的参数名称 + > - 必须在通知方法的签名中添加一个同名参数，在运行时，Spring AOP 会通过这个参数传递返回值 + > - 原始的切点表达式需要出现在 pointcut 属性中 + +```java +/** + * 后置返回通知 + * 如果参数中的第一个参数为JoinPoint，则第二个参数为返回值的信息 + * 如果参数中的第一个参数不为JoinPoint，则第一个参数为returning中对应的参数 + * returning 只有目标方法返回值与通知方法相应参数类型时才能执行后置返回通知，否则不执行 + */ +@AfterReturning(value = "loggingOperation()",returning = "res") +public void afterReturningAdvice(JoinPoint joinPoint, Object res){ + System.out.println("后置返回通知 返回值："+res); +} +``` + + + +#### 4、后置异常通知 + +- 只在连接点抛出异常时才执行异常通知 +- 将 throwing 属性添加到 @AfterThrowing 注解中，也可以访问连接点抛出的异常。 Throwable 是所有错误和异常类的超类，所以在异常通知方法可以捕获到任何错误和异常 +- 如果只对某种特殊的异常类型感兴趣，可以将参数声明为其他异常的参数类型。然后通知就只在抛出这个类型及其子类的异常时才被执行 + +```java +/** + * 后置异常通知 + * 定义一个名字，该名字用于匹配通知实现方法的一个参数名，当目标方法抛出异常返回后，将把目标方法抛出的异常传给通知方法； + * throwing 只有目标方法抛出的异常与通知方法相应参数异常类型时才能执行后置异常通知，否则不执行， + */ +@AfterThrowing(value = "loggingOperation()",throwing = "exception") +public void afterThrowingAdvice(JoinPoint joinPoint,ArithmeticException exception){ + log.error(joinPoint.getSignature().getName() + "has throw an exception" + exception); +} +``` + + + +#### 5、环绕通知 + +- 环绕通知是所有通知类型中功能最为强大的，能够全面地控制连接点。甚至可以控制是否执行连接点 +- 对于环绕通知来说，连接点的参数类型必须是 ProceedingJoinPoint。 它是 JoinPoint 的子接口，允许控制何时执行，是否执行连接点 +- 在环绕通知中需要明确调用 ProceedingJoinPoint 的 `proceed()` 方法来执行被代理的方法，如果忘记这样做就会导致通知被执行了，但目标方法没有被执行 +- 注意：环绕通知的方法需要返回目标方法执行之后的结果，即调用 `joinPoint.proceed();` 的返回值，否则会出现空指针异常 + +```java +/** + * 环绕通知： + * 环绕通知非常强大，可以决定目标方法是否执行，什么时候执行，执行时是否需要替换方法参数，执行完毕是否需要替换返回值。 + * 环绕通知第一个参数必须是org.aspectj.lang.ProceedingJoinPoint类型 + */ +@Around("loggingOperation()") +public Object aroundAdvice(ProceedingJoinPoint joinPoint){ + System.out.println("- - - - - 环绕前置通知 - - - -"); + try { + //调用执行目标方法 + Object obj = joinPoint.proceed(); + System.out.println("- - - - - 环绕后置返回通知 - - - -"); + return obj; + } catch (Throwable throwable) { + throwable.printStackTrace(); + System.out.println("- - - - - 环绕异常通知 - - - -"); + }finally { + System.out.println("- - - - - 环绕后置通知 - - - -"); + } + return null; +} +``` + + + +#### 指定切面的优先级 + +- 在同一个连接点上应用不止一个切面时，除非明确指定，否则它们的优先级是不确定的 +- 切面的优先级可以通过实现 Ordered 接口或利用 @Order 注解指定 +- 实现 Ordered 接口，`getOrder()` 方法的返回值越小，优先级越高 +- 若使用 @Order 注解，序号出现在注解中 + +```java +@Aspect +@Order(0) +public class ValidationAspect { +} + +@Aspect +@Order(1) +public class LogAspect { +``` + + + +#### 重用切入点定义 + +- 在编写 AspectJ 切面时，可以直接在通知注解中书写切入点表达式。但同一个切点表达式可能会在多个通知中重复出现 +- 在 AspectJ 切面中，可以通过 @Pointcut 注解将一个切入点声明成简单的方法。切入点的方法体通常是空的，因为将切入点定义与应用程序逻辑混在一起是不合理的 +- 切入点方法的访问控制符同时也控制着这个切入点的可见性。如果切入点要在多个切面中共用，最好将它们集中在一个公共的类中。在这种情况下，它们必须被声明为 public。在引入这个切入点时，必须将类名也包括在内。如果类没有与这个切面放在同一个包中，还必须包含包名。 +- 其他通知可以通过方法名称引入该切入点 + +```java +/** + * 切入点 + */ +@Pointcut("execution(public int priv.starfish.aop.aspect.CalculatorImpl.*(int,int))") +public void executePackage(){}; + +/** + * 这里直接写成 value= 调用了切入点 excution 表达式 + */ +@AfterReturning(value = "executePackage()",returning = "res") +public void afterReturningAdvice(JoinPoint joinPoint, Object res){ + System.out.println("- - - - - 后置返回通知- - - - -"); + System.out.println("后置返回通知 返回值："+res); +} +``` + + + +#### 引入通知 + +- 引入通知是一种特殊的通知类型。它通过为接口提供实现类，允许对象动态地实现接口，就像对象已经在运行时扩展了实现类一样 + +- 引入通知可以使用两个实现类 MaxCalculatorImpl 和 MinCalculatorImpl，让 ArithmeticCalculatorImpl 动态地实现 MaxCalculator和 MinCalculator接口。而这与从 MaxCalculatorImpl 和 MinCalculatorImpl 中实现多继承的效果相同。但却不需要修改 ArithmeticCalculatorImpl 的源代码 +- 引入通知也必须在切面中声明 +- 在切面中，通过为**任意字段**添加**@DeclareParents**注解来引入声明 +- 注解类型的 **value** 属性表示哪些类是当前引入通知的目标。value 属性值也可以是一个 AspectJ 类型的表达式，可以将一个接口引入到多个类中。**defaultImpl**属性中指定这个接口使用的实现类 + +> 代码在 starfish-learn-spring 上 + + + +### 3.2 用基于 XML 的配置声明切面 + +- 除了使用 AspectJ 注解声明切面，Spring 也支持在 Bean 配置文件中声明切面。这种声明是通过 aop schema 中的 XML 元素完成的 + +- 正常情况下，基于注解的声明要优先于基于 XML 的声明。通过 AspectJ注解，切面可以与 AspectJ 兼容，而基于 XML 的配置则是 Spring 专有的。由于 AspectJ 得到越来越多的 AOP 框架支持，所以以注解风格编写的切面将会有更多重用的机会 + +- 当使用 XML 声明切面时，需要在 \ 根元素中导入 aop Schema + +- 在 Bean 配置文件中，所有的 Spring AOP 配置都必须定义在 \ 元素内部。对于每个切面而言，都要创建一个 \ + + 元素来为具体的切面实现引用后端 Bean 实例 + +- 切面 Bean 必须有一个标识符，供 \ 元素引用 + +```xml + + + + + + + + + + + + + + +``` + +```java +public class TimeHandler { + + public void printTime() { + System.out.println("CurrentTime = " + System.currentTimeMillis()); + } +} +``` + +```java +public static void main(String[] args) { + ApplicationContext ctx = + new ClassPathXmlApplicationContext("applicationContext.xml"); + + Calculator calculator = (Calculator)ctx.getBean("calculator"); + calculator.add(2,3); +} +``` + + + +### 3.2 AOP 两种代理方式 + +**Spring 中 AOP 代理由 Spring 的 IOC 容器负责生成、管理，其依赖关系也由 IOC 容器负责管理**。因此，AOP 代理可以直接使用容器中 + +的其它 bean 实例作为目标，这种关系可由 IOC 容器的依赖注入提供。Spring创建代理的规则为： + +1. **默认使用 Java 动态代理来创建 AOP 代理**，这样就可以为任何接口实例创建代理了 + +2. **当需要代理的类不是代理接口的时候，Spring会切换为使用CGLIB代理**，也可强制使用 CGLIB + +Spring 提供了两种方式来生成代理对象： **JDKProxy** 和 **Cglib**，具体使用哪种方式生成由 AopProxyFactory 根据 AdvisedSupport 对象 + +的配置来决定。默认的策略是如果目标类是接口， 则使用 JDK 动态代理技术，否则使用 Cglib 来生成代理。 + + + +注：JDK 动态代理要比 cglib 代理执行速度快，但性能不如 cglib 好。所以在选择用哪种代理还是要看具体情况，一般单例模式用 cglib 比较好。 \ No newline at end of file diff --git a/docs/framework/Spring/Spring-Cycle-Dependency.md b/docs/framework/Spring/Spring-Cycle-Dependency.md new file mode 100644 index 0000000000..af86bad555 --- /dev/null +++ b/docs/framework/Spring/Spring-Cycle-Dependency.md @@ -0,0 +1,517 @@ +--- +title: 烂了大街的 Spring 循环依赖问题，你觉得自己会了吗 +date: 2022-06-09 +tags: + - spring +categories: spring +--- + +> 文章已收录在 GitHub [JavaKeeper](https://github.com/Jstarfish/JavaKeeper) ，N 线互联网开发、面试必备技能兵器谱，笔记自取。 +> +> 微信搜「 **JavaKeeper** 」程序员成长充电站，互联网技术武道场。无套路领取 500+ 本电子书和 30+ 视频教学和源码。 + +![](https://img.starfish.ink/spring/20200902192731.png) + +## 前言 + +循环依赖问题，算是一道烂大街的面试题了，解毒之前，我们先来回顾两个知识点： + +初学 Spring 的时候，我们就知道 IOC，控制反转么，它将原本在程序中手动创建对象的控制权，交由 Spring 框架来管理，不需要我们手动去各种 `new XXX`。 + +尽管是 Spring 管理，不也得创建对象吗， Java 对象的创建步骤很多，可以 `new XXX`、序列化、`clone()` 等等， 只是 Spring 是通过反射 + 工厂的方式创建对象并放在容器的，创建好的对象我们一般还会对对象属性进行赋值，才去使用，可以理解是分了两个步骤。 + +好了，对这两个步骤有个印象就行，接着我们进入循环依赖，先说下循环依赖的概念 + +### 什么是循环依赖 + +所谓的循环依赖是指，A 依赖 B，B 又依赖 A，它们之间形成了循环依赖。或者是 A 依赖 B，B 依赖 C，C 又依赖 A，形成了循环依赖。更或者是自己依赖自己。它们之间的依赖关系如下： + +![](https://img.starfish.ink/spring/cycle-demo.png) + +这里以两个类直接相互依赖为例，他们的实现代码可能如下： + +```java +public class BeanB { + private BeanA beanA; + public void setBeanA(BeanA beanA) { + this.beanA = beanA; + } +} + +public class BeanA { + private BeanB beanB; + public void setBeanB(BeanB beanB) { + this.beanB = beanB; + } +} +``` + +配置信息如下（用注解方式注入同理，只是为了方便理解，用了配置文件）： + +```java + + + + + + + +``` + +Spring 启动后，读取如上的配置文件，会按顺序先实例化 A，但是创建的时候又发现它依赖了 B，接着就去实例化 B ，同样又发现它依赖了 A ，这尼玛咋整？无限循环呀 + +Spring “肯定”不会让这种事情发生的，如前言我们说的 Spring 实例化对象分两步，第一步会先创建一个原始对象，只是没有设置属性，可以理解为"半成品"—— 官方叫 A 对象的早期引用（EarlyBeanReference），所以当实例化 B 的时候发现依赖了 A， B 就会把这个“半成品”设置进去先完成实例化，既然 B 完成了实例化，所以 A 就可以获得 B 的引用，也完成实例化了，这其实就是 Spring 解决循环依赖的思想。 + +![有点懵逼](https://i04piccdn.sogoucdn.com/7332d8fe139e38e4) + + + +不理解没关系，先有个大概的印象，然后我们从源码来看下 Spring 具体是怎么解决的。 + + + +## 源码解毒 + +> 代码版本：5.0.16.RELEASE + +在 Spring IOC 容器读取 Bean 配置创建 Bean 实例之前, 必须对它进行实例化。只有在容器实例化后，才可以从 IOC 容器里获取 Bean 实例并使用，循环依赖问题也就是发生在实例化 Bean 的过程中的，所以我们先回顾下获取 Bean 的过程。 + +### 获取 Bean 流程 + +Spring IOC 容器中获取 bean 实例的简化版流程如下（排除了各种包装和检查的过程） + +![](https://img.starfish.ink/spring/20200901094342.png) + +大概的流程顺序（可以结合着源码看下，我就不贴了，贴太多的话，呕~呕呕，想吐）： + +1. 流程从 `getBean` 方法开始，`getBean` 是个空壳方法，所有逻辑直接到 `doGetBean` 方法中 +2. `transformedBeanName` 将 name 转换为真正的 beanName（name 可能是 FactoryBean 以 & 字符开头或者有别名的情况，所以需要转化下） +3. 然后通过 `getSingleton(beanName)` 方法尝试从缓存中查找是不是有该实例 sharedInstance（单例在 Spring 的同一容器只会被创建一次，后续再获取 bean，就直接从缓存获取即可） +4. 如果有的话，sharedInstance 可能是完全实例化好的 bean，也可能是一个原始的 bean，所以再经 `getObjectForBeanInstance` 处理即可返回 +5. 当然 sharedInstance 也可能是 null，这时候就会执行创建 bean 的逻辑，将结果返回 + + + +第三步的时候我们提到了一个缓存的概念，这个就是 Spring 为了解决单例的循环依赖问题而设计的 **三级缓存** + +```java +/** Cache of singleton objects: bean name --> bean instance */ +private final Map singletonObjects = new ConcurrentHashMap<>(256); + +/** Cache of singleton factories: bean name --> ObjectFactory */ +private final Map> singletonFactories = new HashMap<>(16); + +/** Cache of early singleton objects: bean name --> bean instance */ +private final Map earlySingletonObjects = new HashMap<>(16); +``` + +这三级缓存的作用分别是： + +- `singletonObjects`：完成初始化的单例对象的 cache，这里的 bean 经历过 `实例化->属性填充->初始化` 以及各种后置处理（一级缓存） + +- `earlySingletonObjects`：存放原始的 bean 对象（**完成实例化但是尚未填充属性和初始化**），仅仅能作为指针提前曝光，被其他 bean 所引用，用于解决循环依赖的 （二级缓存） + +- `singletonFactories`：在 bean 实例化完之后，属性填充以及初始化之前，如果允许提前曝光，Spring 会将实例化后的 bean 提前曝光，也就是把该 bean 转换成 `beanFactory` 并加入到 `singletonFactories`（三级缓存） + +我们首先从缓存中试着获取 bean，就是从这三级缓存中查找 + +```java +protected Object getSingleton(String beanName, boolean allowEarlyReference) { + // 从 singletonObjects 获取实例，singletonObjects 中的实例都是准备好的 bean 实例，可以直接使用 + Object singletonObject = this.singletonObjects.get(beanName); + //isSingletonCurrentlyInCreation() 判断当前单例bean是否正在创建中 + if (singletonObject == null && isSingletonCurrentlyInCreation(beanName)) { + synchronized (this.singletonObjects) { + // 一级缓存没有，就去二级缓存找 + singletonObject = this.earlySingletonObjects.get(beanName); + if (singletonObject == null && allowEarlyReference) { + // 二级缓存也没有，就去三级缓存找 + ObjectFactory singletonFactory = this.singletonFactories.get(beanName); + if (singletonFactory != null) { + // 三级缓存有的话，就把他移动到二级缓存,.getObject() 后续会讲到 + singletonObject = singletonFactory.getObject(); + this.earlySingletonObjects.put(beanName, singletonObject); + this.singletonFactories.remove(beanName); + } + } + } + } + return singletonObject; +} +``` + + + +如果缓存没有的话，我们就要创建了，接着我们以单例对象为例，再看下创建 bean 的逻辑（大括号表示内部类调用方法）： + +![spring-createbean](https://img.starfish.ink/spring/spring-createbean.png) + +1. 创建 bean 从以下代码开始，一个匿名内部类方法参数（总觉得 Lambda 的方式可读性不如内部类好理解） + + ```java + if (mbd.isSingleton()) { + sharedInstance = getSingleton(beanName, () -> { + try { + return createBean(beanName, mbd, args); + } + catch (BeansException ex) { + destroySingleton(beanName); + throw ex; + } + }); + bean = getObjectForBeanInstance(sharedInstance, name, beanName, mbd); + } + ``` + + `getSingleton()` 方法内部主要有两个方法 + + ```java + public Object getSingleton(String beanName, ObjectFactory singletonFactory) { + // 创建 singletonObject + singletonObject = singletonFactory.getObject(); + // 将 singletonObject 放入缓存 + addSingleton(beanName, singletonObject); + } + ``` + +2. `getObject()` 匿名内部类的实现真正调用的又是 `createBean(beanName, mbd, args)` +3. 往里走，主要的实现逻辑在 `doCreateBean`方法，先通过 `createBeanInstance` 创建一个原始 bean 对象 +4. 接着 `addSingletonFactory` 添加 bean 工厂对象到 singletonFactories 缓存（三级缓存） +5. 通过 `populateBean` 方法向原始 bean 对象中填充属性，并解析依赖，假设这时候创建 A 之后填充属性时发现依赖 B，然后创建依赖对象 B 的时候又发现依赖 A，还是同样的流程，又去 `getBean(A)`，这个时候三级缓存已经有了 beanA 的“半成品”，这时就可以把 A 对象的原始引用注入 B 对象（并将其移动到二级缓存）来解决循环依赖问题。这时候 `getObject()` 方法就算执行结束了，返回完全实例化的 bean +6. 最后调用 `addSingleton` 把完全实例化好的 bean 对象放入 singletonObjects 缓存（一级缓存）中，打完收工 + + + +### Spring 解决循环依赖 + +建议搭配着“源码”看下边的逻辑图，更好下饭 + +![](https://img.starfish.ink/spring/cycle-dependency-code.png) + +流程其实上边都已经说过了，结合着上图我们再看下具体细节，用大白话再捋一捋： + +1. Spring 创建 bean 主要分为两个步骤，创建原始 bean 对象，接着去填充对象属性和初始化 +2. 每次创建 bean 之前，我们都会从缓存中查下有没有该 bean，因为是单例，只能有一个 +3. 当我们创建 beanA 的原始对象后，并把它放到三级缓存中，接下来就该填充对象属性了，这时候发现依赖了 beanB，接着就又去创建 beanB，同样的流程，创建完 beanB 填充属性时又发现它依赖了 beanA，又是同样的流程，不同的是，这时候可以在三级缓存中查到刚放进去的原始对象 beanA，所以不需要继续创建，用它注入 beanB，完成 beanB 的创建 +4. 既然 beanB 创建好了，所以 beanA 就可以完成填充属性的步骤了，接着执行剩下的逻辑，闭环完成 + +这就是单例模式下 Spring 解决循环依赖的流程了。 + +但是这个地方，不管是谁看源码都会有个小疑惑，为什么需要三级缓存呢，我赶脚二级他也够了呀 + +> 革命尚未成功，同志仍需努力 + +跟源码的时候，发现在创建 beanB 需要引用 beanA 这个“半成品”的时候，就会触发"前期引用"，即如下代码： + +```java +ObjectFactory singletonFactory = this.singletonFactories.get(beanName); +if (singletonFactory != null) { + // 三级缓存有的话，就把他移动到二级缓存 + singletonObject = singletonFactory.getObject(); + this.earlySingletonObjects.put(beanName, singletonObject); + this.singletonFactories.remove(beanName); +} +``` + +`singletonFactory.getObject()` 是一个接口方法，这里具体的实现方法在 + +```java +protected Object getEarlyBeanReference(String beanName, RootBeanDefinition mbd, Object bean) { + Object exposedObject = bean; + if (!mbd.isSynthetic() && hasInstantiationAwareBeanPostProcessors()) { + for (BeanPostProcessor bp : getBeanPostProcessors()) { + if (bp instanceof SmartInstantiationAwareBeanPostProcessor) { + SmartInstantiationAwareBeanPostProcessor ibp = (SmartInstantiationAwareBeanPostProcessor) bp; + // 这么一大段就这句话是核心，也就是当bean要进行提前曝光时， + // 给一个机会，通过重写后置处理器的getEarlyBeanReference方法，来自定义操作bean + // 值得注意的是，如果提前曝光了，但是没有被提前引用，则该后置处理器并不生效!!! + // 这也正式三级缓存存在的意义，否则二级缓存就可以解决循环依赖的问题 + exposedObject = ibp.getEarlyBeanReference(exposedObject, beanName); + } + } + } + return exposedObject; +} +``` + +这个方法就是 Spring 为什么使用三级缓存，而不是二级缓存的原因，它的目的是为了后置处理，如果没有 AOP 后置处理，就不会走进 if 语句，直接返回了 exposedObject ，相当于啥都没干，二级缓存就够用了。 + +所以又得出结论，这个三级缓存应该和 AOP 有关系，继续。 + +在 Spring 的源码中`getEarlyBeanReference` 是 `SmartInstantiationAwareBeanPostProcessor` 接口的默认方法，真正实现这个方法的只有**`AbstractAutoProxyCreator`** 这个类，用于提前曝光的 AOP 代理。 + +```java +@Override +public Object getEarlyBeanReference(Object bean, String beanName) throws BeansException { + Object cacheKey = getCacheKey(bean.getClass(), beanName); + this.earlyProxyReferences.put(cacheKey, bean); + // 对bean进行提前Spring AOP代理 + return wrapIfNecessary(bean, beanName, cacheKey); +} +``` + +这么说有点干，来个小 demo 吧，我们都知道 **Spring AOP、事务**等都是通过代理对象来实现的，而**事务**的代理对象是由自动代理创建器来自动完成的。也就是说 Spring 最终给我们放进容器里面的是一个代理对象，**而非原始对象**，假设我们有如下一段业务代码： + +```java +@Service +public class HelloServiceImpl implements HelloService { + @Autowired + private HelloService helloService; + + @Override + @Transactional + public Object hello() { + return "Hello JavaKeeper"; + } +} +``` + +此 `Service` 类使用到了事务，所以最终会生成一个 JDK 动态代理对象 `Proxy`。刚好它又存在**自己引用自己**的循环依赖，完美符合我们的场景需求。 + +我们再自定义一个后置处理，来看下效果： + +```java +@Component +public class HelloProcessor implements SmartInstantiationAwareBeanPostProcessor { + + @Override + public Object getEarlyBeanReference(Object bean, String beanName) throws BeansException { + System.out.println("提前曝光了："+beanName); + return bean; + } +} +``` + +可以看到，调用方法栈中有我们自己实现的 `HelloProcessor`，说明这个 bean 会通过 AOP 代理处理。 + +![](https://img.starfish.ink/spring/getEarlyBeanReference-code.png) + +再从源码看下这个自己循环自己的 bean 的创建流程： + +```java +protected Object doCreateBean( ... ){ + ... + + boolean earlySingletonExposure = (mbd.isSingleton() && this.allowCircularReferences && isSingletonCurrentlyInCreation(beanName)); + // 需要提前暴露（支持循环依赖），就注册一个ObjectFactory到三级缓存 + if (earlySingletonExposure) { + // 添加 bean 工厂对象到 singletonFactories 缓存中，并获取原始对象的早期引用 + //匿名内部方法 getEarlyBeanReference 就是后置处理器 + // SmartInstantiationAwareBeanPostProcessor 的一个方法， + // 它的功效为：保证自己被循环依赖的时候，即使被别的Bean @Autowire进去的也是代理对象 + addSingletonFactory(beanName, () -> getEarlyBeanReference(beanName, mbd, bean)); + } + + // 此处注意：如果此处自己被循环依赖了 那它会走上面的getEarlyBeanReference，从而创建一个代理对象从 三级缓存转移到二级缓存里 + // 注意此时候对象还在二级缓存里，并没有在一级缓存。并且此时后续的这两步操作还是用的 exposedObject，它仍旧是原始对象~~~ + populateBean(beanName, mbd, instanceWrapper); + exposedObject = initializeBean(beanName, exposedObject, mbd); + + // 因为事务的AOP自动代理创建器在getEarlyBeanReference 创建代理后，initializeBean 就不会再重复创建了，二选一的） + + // 所以经过这两大步后，exposedObject 还是原始对象，通过 getEarlyBeanReference 创建的代理对象还在三级缓存呢 + + ... + + // 循环依赖校验 + if (earlySingletonExposure) { + // 注意此处第二个参数传的false，表示不去三级缓存里再去调用一次getObject()方法了~~~，此时代理对象还在二级缓存，所以这里拿出来的就是个 代理对象 + // 最后赋值给exposedObject 然后return出去，进而最终被addSingleton()添加进一级缓存里面去 + // 这样就保证了我们容器里 最终实际上是代理对象，而非原始对象~~~~~ + Object earlySingletonReference = getSingleton(beanName, false); + if (earlySingletonReference != null) { + if (exposedObject == bean) { + exposedObject = earlySingletonReference; + } + } + ... + } + +} +``` + +#### 自我解惑： + +##### 问：还是不太懂，为什么这么设计呢，即使有代理，在二级缓存代理也可以吧 | 为什么要使用三级缓存呢？ + +我们再来看下相关代码，假设我们现在是二级缓存架构，创建 A 的时候，我们不知道有没有循环依赖，所以放入二级缓存提前暴露，接着创建 B，也是放入二级缓存，这时候发现又循环依赖了 A，就去二级缓存找，是有，但是如果此时还有 AOP 代理呢，我们要的是代理对象可不是原始对象，这怎么办，只能改逻辑，在第一步的时候，不管3721，所有 Bean 统统去完成 AOP 代理，如果是这样的话，就不需要三级缓存了，但是这样不仅没有必要，而且违背了 Spring 在结合 `AOP` 跟 Bean 的生命周期的设计。 + +所以 Spring “多此一举”的将实例先封装到 ObjectFactory 中（三级缓存），主要关键点在 `getObject()` 方法并非直接返回实例，而是对实例又使用 `SmartInstantiationAwareBeanPostProcessor` 的 `getEarlyBeanReference` 方法对 bean 进行处理，也就是说，当 Spring 中存在该后置处理器，所有的单例 bean 在实例化后都会被进行提前曝光到三级缓存中，但是并不是所有的 bean 都存在循环依赖，也就是三级缓存到二级缓存的步骤不一定都会被执行，有可能曝光后直接创建完成，没被提前引用过，就直接被加入到一级缓存中。因此可以确保只有提前曝光且被引用的 bean 才会进行该后置处理。 + +```java +protected Object getSingleton(String beanName, boolean allowEarlyReference) { + Object singletonObject = this.singletonObjects.get(beanName); + if (singletonObject == null && isSingletonCurrentlyInCreation(beanName)) { + synchronized (this.singletonObjects) { + singletonObject = this.earlySingletonObjects.get(beanName); + if (singletonObject == null && allowEarlyReference) { + // 三级缓存获取，key=beanName value=objectFactory，objectFactory中存储 //getObject()方法用于获取提前曝光的实例 + ObjectFactory singletonFactory = this.singletonFactories.get(beanName); + if (singletonFactory != null) { + // 三级缓存有的话，就把他移动到二级缓存 + singletonObject = singletonFactory.getObject(); + this.earlySingletonObjects.put(beanName, singletonObject); + this.singletonFactories.remove(beanName); + } + } + } + } + return singletonObject; +} + + +boolean earlySingletonExposure = (mbd.isSingleton() && this.allowCircularReferences && + isSingletonCurrentlyInCreation(beanName)); +if (earlySingletonExposure) { + if (logger.isDebugEnabled()) { + logger.debug("Eagerly caching bean '" + beanName + + "' to allow for resolving potential circular references"); + } + // 添加 bean 工厂对象到 singletonFactories 缓存中，并获取原始对象的早期引用 + //匿名内部方法 getEarlyBeanReference 就是后置处理器 + // SmartInstantiationAwareBeanPostProcessor 的一个方法， + // 它的功效为：保证自己被循环依赖的时候，即使被别的Bean @Autowire进去的也是代理对象~~~~ AOP自动代理创建器此方法里会创建的代理对象~~~ + addSingletonFactory(beanName, () -> getEarlyBeanReference(beanName, mbd, bean)); +} +``` + + + +##### 再问：AOP 代理对象提前放入了三级缓存，没有经过属性填充和初始化，这个代理又是如何保证依赖属性的注入的呢？ + +这个又涉及到了 Spring 中动态代理的实现，不管是`cglib`代理还是`jdk`动态代理生成的代理类，代理时，会将目标对象 target 保存在最后生成的代理 `$proxy` 中，当调用 `$proxy` 方法时会回调 `h.invoke`，而 `h.invoke` 又会回调目标对象 target 的原始方法。所有，其实在 AOP 动态代理时，原始 bean 已经被保存在 **提前曝光代理**中了，之后 `原始 bean` 继续完成`属性填充`和`初始化`操作。因为 AOP 代理`$proxy `中保存着 `traget` 也就是是 `原始bean` 的引用，因此后续 `原始bean` 的完善，也就相当于Spring AOP中的 `target` 的完善，这样就保证了 AOP 的`属性填充`与`初始化`了！ + + + +### 非单例循环依赖 + +看完了单例模式的循环依赖，我们再看下非单例的情况，假设我们的配置文件是这样的： + +```xml + + + + + + + +``` + +启动 Spring，结果如下： + +```java +Error creating bean with name 'beanA' defined in class path resource [applicationContext.xml]: Cannot resolve reference to bean 'beanB' while setting bean property 'beanB'; + +Error creating bean with name 'beanB' defined in class path resource [applicationContext.xml]: Cannot resolve reference to bean 'beanA' while setting bean property 'beanA'; + +Caused by: org.springframework.beans.factory.BeanCurrentlyInCreationException: Error creating bean with name 'beanA': Requested bean is currently in creation: Is there an unresolvable circular reference? +``` + +对于 `prototype` 作用域的 bean，Spring 容器无法完成依赖注入，因为 Spring 容器不进行缓存 `prototype` 作用域的 bean ，因此无法提前暴露一个创建中的bean 。 + +原因也挺好理解的，原型模式每次请求都会创建一个实例对象，即使加了缓存，循环引用太多的话，就比较麻烦了就，所以 Spring 不支持这种方式，直接抛出异常： + +```java +if (isPrototypeCurrentlyInCreation(beanName)) { + throw new BeanCurrentlyInCreationException(beanName); +} +``` + + + +### 构造器循环依赖 + +上文我们讲的是通过 Setter 方法注入的单例 bean 的循环依赖问题，用 Spring 的小伙伴也都知道，依赖注入的方式还有**构造器注入**、工厂方法注入的方式（很少使用），那如果构造器注入方式也有循环依赖，可以搞不？ + +我们再改下代码和配置文件 + +```java +public class BeanA { + private BeanB beanB; + public BeanA(BeanB beanB) { + this.beanB = beanB; + } +} + +public class BeanB { + private BeanA beanA; + public BeanB(BeanA beanA) { + this.beanA = beanA; + } +} +``` + +```xml + + + + + + + +``` + +执行结果，又是异常 + +![](https://img.starfish.ink/spring/cycle-dependency-constructor.png) + +看看官方给出的说法 + +> Circular dependencies +> +> If you use predominantly constructor injection, it is possible to create an unresolvable circular dependency scenario. +> +> For example: Class A requires an instance of class B through constructor injection, and class B requires an instance of class A through constructor injection. If you configure beans for classes A and B to be injected into each other, the Spring IoC container detects this circular reference at runtime, and throws a `BeanCurrentlyInCreationException`. +> +> One possible solution is to edit the source code of some classes to be configured by setters rather than constructors. Alternatively, avoid constructor injection and use setter injection only. In other words, although it is not recommended, you can configure circular dependencies with setter injection. +> +> Unlike the typical case (with no circular dependencies), a circular dependency between bean A and bean B forces one of the beans to be injected into the other prior to being fully initialized itself (a classic chicken-and-egg scenario). + +大概意思是： + +如果您主要使用构造器注入，循环依赖场景是无法解决的。建议你用 setter 注入方式代替构造器注入 + +其实也不是说只要是构造器注入就会有循环依赖问题，Spring 在创建 Bean 的时候默认是**按照自然排序来进行创建的**，我们暂且把先创建的 bean 叫主 bean，上文的 A 即主 bean，**只要主 bean 注入依赖 bean 的方式是 setter 方式，依赖 bean 的注入方式无所谓，都可以解决，反之亦然** + +所以上文我们 AB 循环依赖问题，只要 A 的注入方式是 setter ，就不会有循环依赖问题。 + +面试官问：为什么呢？ + +**Spring 解决循环依赖依靠的是 Bean 的“中间态”这个概念，而这个中间态指的是已经实例化，但还没初始化的状态。实例化的过程又是通过构造器创建的，如果 A 还没创建好出来，怎么可能提前曝光，所以构造器的循环依赖无法解决，我一直认为应该先有鸡才能有蛋**。 + + + + + +## 小总结 | 面试这么答 + +#### B 中提前注入了一个没有经过初始化的 A 类型对象不会有问题吗？ + +虽然在创建 B 时会提前给 B 注入了一个还未初始化的 A 对象，但是在创建 A 的流程中一直使用的是注入到 B 中的 A 对象的引用，之后会根据这个引用对 A 进行初始化，所以这是没有问题的。 + +#### Spring 是如何解决的循环依赖？ + +Spring 为了解决单例的循环依赖问题，使用了三级缓存。其中一级缓存为单例池（`singletonObjects`），二级缓存为提前曝光对象（`earlySingletonObjects`），三级缓存为提前曝光对象工厂（`singletonFactories`）。 + +假设A、B循环引用，实例化 A 的时候就将其放入三级缓存中，接着填充属性的时候，发现依赖了 B，同样的流程也是实例化后放入三级缓存，接着去填充属性时又发现自己依赖 A，这时候从缓存中查找到早期暴露的 A，没有 AOP 代理的话，直接将 A 的原始对象注入 B，完成 B 的初始化后，进行属性填充和初始化，这时候 B 完成后，就去完成剩下的 A 的步骤，如果有 AOP 代理，就进行 AOP 处理获取代理后的对象 A，注入 B，走剩下的流程。 + +#### 为什么要使用三级缓存呢？二级缓存能解决循环依赖吗？ + +如果没有 AOP 代理，二级缓存可以解决问题，但是有 AOP 代理的情况下，只用二级缓存就意味着所有 Bean 在实例化后就要完成 AOP 代理，这样违背了 Spring 设计的原则，Spring 在设计之初就是通过 `AnnotationAwareAspectJAutoProxyCreator` 这个后置处理器来在 Bean 生命周期的最后一步来完成 AOP 代理，而不是在实例化后就立马进行 AOP 代理。 + + + +### 参考与感谢： + +[《Spring 源码深度解析》- 郝佳著](https://book.douban.com/subject/25866350/) + +https://developer.aliyun.com/article/766880 + +http://www.tianxiaobo.com/2018/06/08/Spring-IOC-容器源码分析-循环依赖的解决办法 + +https://cloud.tencent.com/developer/article/1497692 + +https://blog.csdn.net/chaitoudaren/article/details/105060882 + + + diff --git a/docs/framework/Spring/Spring-IOC-Source.md b/docs/framework/Spring/Spring-IOC-Source.md new file mode 100644 index 0000000000..c1ccd68612 --- /dev/null +++ b/docs/framework/Spring/Spring-IOC-Source.md @@ -0,0 +1,2174 @@ +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20201105184149.png)Spring IOC 源码解毒 + +很多人一提 IOC，便张口就来：**控制反转**。究竟哪些方面被反转了呢？答案是依赖对象的获得被反转了。很多时候，我们通过多个对象之间的协作来完成一个功能，如果获取所依赖对象靠自身来实现，这将导致代码的耦合度高和难以测试。 + + + +这篇文章，我们从五个疑问展开 + +1. 搞清楚 ApplicationContext 实例化 Bean 的过程 +2. 搞清楚这个过程中涉及的核心类 +3. 搞清楚 IOC 容器提供的扩展点有哪些，学会扩展 +4. 学会 IOC 容器这里使用的设计模式 +5. 搞清楚不同创建方式的 bean 的创建过程 + + + +## 一、IOC 回顾 + +IoC（Inverse of Control:控制反转）是一种**设计思想**，就是 **将原本在程序中手动创建对象的控制权，交由Spring框架来管理**。 IoC 在其他语言中也有应用，并非 Spring 特有。 **IoC 容器是 Spring 用来实现 IoC 的载体， IoC 容器实际上就是个Map（key，value），Map 中存放的是各种对象**。 + +将对象之间的相互依赖关系交给 IoC 容器来管理，并由 IoC 容器完成对象的注入。这样可以很大程度上简化应用的开发，把应用从复杂的依赖关系中解放出来。 **IoC 容器就像是一个工厂一样，当我们需要创建一个对象的时候，只需要配置好配置文件/注解即可，完全不用考虑对象是如何被创建出来的。** 在实际项目中一个 Service 类可能有几百甚至上千个类作为它的底层，假如我们需要实例化这个 Service，你可能要每次都要搞清这个 Service 所有底层类的构造函数，这可能会把人逼疯。如果利用 IoC 的话，你只需要配置好，然后在需要的地方引用就行了，这大大增加了项目的可维护性且降低了开发难度。 + +Spring IOC 通过引入 xml 配置，由 IOC 容器来管理对象的生命周期，依赖关系等。 + +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20201106153609.png) + +从图中可以看出，我们以前获取两个有依赖关系的对象，要用 set 方法，而用容器之后，它们之间的关系就由容器来管理。 + +### 什么是 Spring IOC 容器？ + +Spring 框架的核心是 Spring 容器。容器创建对象，将它们装配在一起，配置它们并管理它们的完整生命周期。Spring 容器使用**依赖注入**来管理组成应用程序的组件。容器通过读取提供的配置元数据来接收对象进行实例化，配置和组装的指令。该元数据可以通过 XML，Java 注解或 Java 代码提供。 + +![container magic](https://docs.spring.io/spring/docs/5.0.18.RELEASE/spring-framework-reference/images/container-magic.png) + +### 什么是依赖注入？ + +**依赖注入（DI,Dependency Injection）是在编译阶段尚未知所需的功能是来自哪个的类的情况下，将其他对象所依赖的功能对象实例化的模式**。这就需要一种机制用来激活相应的组件以提供特定的功能，所以**依赖注入是控制反转的基础**。否则如果在组件不受框架控制的情况下，框架又怎么知道要创建哪个组件？ + +依赖注入有以下三种实现方式： + +1. 构造器注入 +2. Setter方法注入（属性注入） +3. 接口注入 + +### Spring 中有多少种 IOC 容器？ + +在 Spring IOC 容器读取 Bean 配置创建 Bean 实例之前，必须对它进行实例化。只有在容器实例化后， 才可以从 IOC 容器里获取 Bean 实例并使用。 + +Spring 提供了两种类型的 IOC 容器实现 + +- **BeanFactory**：IOC 容器的基本实现 +- **ApplicationContext**：提供了更多的高级特性，是 BeanFactory 的子接口 + +BeanFactory 是 Spring 框架的基础设施，面向 Spring 本身；ApplicationContext 面向使用 Spring 框架的开发者，几乎所有的应用场合都直接使用 ApplicationContext 而非底层的 BeanFactory； + +无论使用何种方式，配置文件是相同的。 + + + +### BeanFactory + +我们先来说说 BeanFactory。 + +BeanFactory，从名字上可以看出来它是 bean 的工厂，它负责生产和管理各个 bean 实例。 + +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20201105184149.png) + +大概了解下这里提到的几个类： + +- **ListableBeanFactory**：这个 Listable 的意思就是，通过这个接口，我们可以获取多个 Bean，大家看源码会发现，最顶层 BeanFactory 接口的方法都是获取单个 Bean 的。 +- **HierarchicalBeanFactory**：Hierarchical 单词本身已经能说明问题了，也就是说我们可以在应用中起多个 BeanFactory，然后可以将各个 BeanFactory 设置为父子关系。 +- **AutowireCapableBeanFactory**： 这个名字中的 Autowire 大家都非常熟悉，它就是用来自动装配 Bean 用的，但是仔细看上图，ApplicationContext 并没有继承它，不过不用担心，不使用继承，不代表不可以使用组合，如果你看到 ApplicationContext 接口定义中的最后一个方法 getAutowireCapableBeanFactory() 就知道了。 +- **ConfigurableListableBeanFactory** ：也是一个特殊的接口，看图，特殊之处在于它继承了第二层所有的三个接口，而 ApplicationContext 没有。这点之后会用到。 + + + +## 二、ApplicationContext 实例化 Bean 的过程 + +还是从最简单的 hello world 来看 + +```java +ApplicationContext ac = new ClassPathXmlApplicationContext("classpath:applicationContext.xml"); +Hello hello = (Hello)ac.getBean("hello"); +hello.sayHello(); +``` + +这个从写法上我们可以知道从 ClassPath 中寻找 xml 配置文件，然后根据 xml 文件的内容来构建ApplicationContext 对象实例（容器），然后通过容器获取一个叫 ”hello“ 的 bean，执行该 bean 的 sayHello 方法。 + +当然我们之前也知道这不是唯一的构建容器方式，我们先来看看大体的继承结构是怎么样的： + +![javadoop.com](https://www.javadoop.com/blogimages/spring-context/1.png) + +**启动过程分析** + +第一步，我们肯定要从 ClassPathXmlApplicationContext 的构造方法说起。 + +```java +public ClassPathXmlApplicationContext( + String[] configLocations, boolean refresh, @Nullable ApplicationContext parent) + throws BeansException { + + super(parent); + // 根据提供的路径，处理成配置文件数组(以分号、逗号、空格、tab、换行符分割) + setConfigLocations(configLocations); + if (refresh) { + refresh(); + } +} +``` + +接下来，就是 `refresh()`，这里简单说下为什么是 `refresh()`，而不是 `init()` 这种名字的方法。因为 ApplicationContext 建立起来以后，其实我们是可以通过调用 `refresh()` 这个方法重建的，`refresh()` 会将原来的 ApplicationContext 销毁，然后再重新执行一次初始化操作。 + +```java +@Override +public void refresh() throws BeansException, IllegalStateException { + // 来个锁，不然 refresh() 还没结束，你又来个启动或销毁容器的操作，那不就乱套了嘛 + synchronized (this.startupShutdownMonitor) { + // Prepare this context for refreshing. + // 准备工作，记录下容器的启动时间、标记“已启动”状态、处理配置文件中的占位符 + prepareRefresh(); + + // Tell the subclass to refresh the internal bean factory. + // 这步比较关键，这步完成后，配置文件就会解析成一个个 Bean 定义，注册到 BeanFactory 中， + // 当然，这里说的 Bean 还没有初始化，只是配置信息都提取出来了， + // 注册也只是将这些信息都保存到了注册中心(说到底核心是一个 beanName-> beanDefinition 的 map) + ConfigurableListableBeanFactory beanFactory = obtainFreshBeanFactory(); + + // Prepare the bean factory for use in this context. + // 设置 BeanFactory 的类加载器，添加几个 BeanPostProcessor，手动注册几个特殊的 bean + prepareBeanFactory(beanFactory); + + try { + // Allows post-processing of the bean factory in context subclasses. + //提供给子类实现一些postProcess的注册，如AbstractRefreshableWebApplicationContext注册一些Servlet相关的 + postProcessBeanFactory(beanFactory); + + // Invoke factory processors registered as beans in the context. + //调用所有BeanFactoryProcessor的postProcessBeanFactory()方法 + invokeBeanFactoryPostProcessors(beanFactory); + + // Register bean processors that intercept bean creation. + //注册BeanPostProcessor，BeanPostProcessor作用是用于拦截Bean的创建 + registerBeanPostProcessors(beanFactory); + + // Initialize message source for this context. + //初始化消息Bean + initMessageSource(); + + // Initialize event multicaster for this context. + //初始化上下文的事件多播组建，ApplicationEvent触发时由multicaster通知给ApplicationListener + initApplicationEventMulticaster(); + + // Initialize other special beans in specific context subclasses. + //ApplicationContext初始化一些特殊的bean + onRefresh(); + + // Check for listener beans and register them. + //注册事件监听器，事件监听Bean统一注册到multicaster里头，ApplicationEvent事件触发后会由multicaster广播 + registerListeners(); + + // Instantiate all remaining (non-lazy-init) singletons.(重点) + // 非延迟加载的单例Bean实例化 + finishBeanFactoryInitialization(beanFactory); + + // Last step: publish corresponding event.(最后，广播事件，ApplicationContext 初始化完成) + finishRefresh(); + } + + catch (BeansException ex) { + if (logger.isWarnEnabled()) { + logger.warn("Exception encountered during context initialization - " + + "cancelling refresh attempt: " + ex); + } + + // Destroy already created singletons to avoid dangling resources. + destroyBeans(); + + // Reset 'active' flag. + cancelRefresh(ex); + + // Propagate exception to caller. + throw ex; + } + + finally { + // Reset common introspection caches in Spring's core, since we + // might not ever need metadata for singleton beans anymore... + resetCommonCaches(); + } + } +} +``` + + + +下面，我们开始一步步来肢解这个 `refresh()` 方法。 + +#### 2.1 创建 Bean 容器前的准备工作 + +```java +/** + * Prepare this context for refreshing, setting its startup date and + * active flag as well as performing any initialization of property sources. + */ +protected void prepareRefresh() { + // Switch to active. + this.startupDate = System.currentTimeMillis(); + this.closed.set(false); + this.active.set(true); + + if (logger.isInfoEnabled()) { + logger.info("Refreshing " + this); + } + + // Initialize any placeholder property sources in the context environment. + initPropertySources(); + + // Validate that all properties marked as required are resolvable: + // see ConfigurablePropertyResolver#setRequiredProperties + // 校验 xml 配置文件 + getEnvironment().validateRequiredProperties(); + + // Store pre-refresh ApplicationListeners... + if (this.earlyApplicationListeners == null) { + this.earlyApplicationListeners = new LinkedHashSet<>(this.applicationListeners); + } + else { + // Reset local application listeners to pre-refresh state. + this.applicationListeners.clear(); + this.applicationListeners.addAll(this.earlyApplicationListeners); + } + + // Allow for the collection of early ApplicationEvents, + // to be published once the multicaster is available... + this.earlyApplicationEvents = new LinkedHashSet<>(); +} +``` + +#### 2.2 创建 Bean 容器，加载并注册 Bean + +我们回到 `refresh()` 方法中的下一行 `obtainFreshBeanFactory()`。 + +注意，这个方法是全文最重要的部分之一，这里将会初始化 BeanFactory、加载 Bean、注册 Bean 等等。 + +当然，这步结束后，Bean 并没有完成初始化。这里指的是 Bean 实例并未在这一步生成。 + +```java +protected ConfigurableListableBeanFactory obtainFreshBeanFactory() { + // 关闭旧的 BeanFactory (如果有)，创建新的 BeanFactory，加载 Bean 定义、注册 Bean 等等 + refreshBeanFactory(); + // 返回刚刚创建的 BeanFactory + ConfigurableListableBeanFactory beanFactory = getBeanFactory(); + if (logger.isDebugEnabled()) { + logger.debug("Bean factory for " + getDisplayName() + ": " + beanFactory); + } + return beanFactory; +} +``` + +// AbstractRefreshableApplicationContext.java + +```java +@Override +protected final void refreshBeanFactory() throws BeansException { + // 如果 ApplicationContext 中已经加载过 BeanFactory 了，销毁所有 Bean，关闭 BeanFactory + // 注意，应用中 BeanFactory 本来就是可以多个的，这里可不是说应用全局是否有 BeanFactory，而是当前 + // ApplicationContext 是否有 BeanFactory + if (hasBeanFactory()) { + destroyBeans(); + closeBeanFactory(); + } + try { + // 初始化一个 DefaultListableBeanFactory，为什么用这个，我们马上说。 + DefaultListableBeanFactory beanFactory = createBeanFactory(); + // 用于 BeanFactory 的序列化 + beanFactory.setSerializationId(getId()); + + // 下面这两个方法很重要，别跟丢了，具体细节之后说 + // 设置 BeanFactory 的两个配置属性：是否允许 Bean 覆盖、是否允许循环引用 + customizeBeanFactory(beanFactory); + + // 加载 Bean 到 BeanFactory 中 + loadBeanDefinitions(beanFactory); + synchronized (this.beanFactoryMonitor) { + this.beanFactory = beanFactory; + } + } + catch (IOException ex) { + throw new ApplicationContextException("I/O error parsing bean definition source for " + getDisplayName(), ex); + } +} +``` + +> 看到这里的时候，我觉得读者就应该站在高处看 ApplicationContext 了，ApplicationContext 继承自 BeanFactory，但是它不应该被理解为 BeanFactory 的实现类，而是说其内部持有一个实例化的 BeanFactory（DefaultListableBeanFactory）。以后所有的 BeanFactory 相关的操作其实是委托给这个实例来处理的。 + +我们说说为什么选择实例化 **DefaultListableBeanFactory** ？前面我们说了有个很重要的接口 ConfigurableListableBeanFactory，它实现了 BeanFactory 下面一层的所有三个接口，我把之前的继承图再拿过来大家再仔细看一下： + +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20201105184149.png)![点击并拖拽以移动](data:image/gif;base64,R0lGODlhAQABAPABAP///wAAACH5BAEKAAAALAAAAAABAAEAAAICRAEAOw==) + +我们可以看到 ConfigurableListableBeanFactory 只有一个实现类 DefaultListableBeanFactory，而且实现类 DefaultListableBeanFactory 还通过实现右边的 AbstractAutowireCapableBeanFactory 通吃了右路。所以结论就是，最底下这个家伙 DefaultListableBeanFactory 基本上是最牛的 BeanFactory 了，这也是为什么这边会使用这个类来实例化的原因。 + +> 如果你想要在程序运行的时候动态往 Spring IOC 容器注册新的 bean，就会使用到这个类。那我们怎么在运行时获得这个实例呢？ +> +> 之前我们说过 ApplicationContext 接口能获取到 AutowireCapableBeanFactory，就是最右上角那个，然后它向下转型就能得到 DefaultListableBeanFactory 了。 + +在继续往下之前，我们需要先了解 BeanDefinition。**我们说 BeanFactory 是 Bean 容器，那么 Bean 又是什么呢？** + +这里的 BeanDefinition 就是我们所说的 Spring 的 Bean，我们自己定义的各个 Bean 其实会转换成一个个 BeanDefinition 存在于 Spring 的 BeanFactory 中。 + +所以，如果有人问你 Bean 是什么的时候，你要知道 Bean 在代码层面上可以认为是 BeanDefinition 的实例。 + +> BeanDefinition 中保存了我们的 Bean 信息，比如这个 Bean 指向的是哪个类、是否是单例的、是否懒加载、这个 Bean 依赖了哪些 Bean 等等。 + +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20201106154018.png) + +BeanDefinition 是一个接口，用于属性承载，比如 \ 元素标签拥有 class、scope、lazy-init 等配置。bean的定义方式有千千万万种，无论是何种标签，无论是何种资源定义，无论是何种容器，只要按照 Spring 的规范编写xml配置文件，最终的 bean 定义内部表示都将转换为内部的唯一结构：BeanDefinition。当 BeanDefinition 注册完毕以后，Spring 的 BeanFactory 就可以随时根据需要进行实例化了。 + +##### BeanDefinition 接口定义 + +我们来看下 BeanDefinition 的接口定义： + +```java +public interface BeanDefinition extends AttributeAccessor, BeanMetadataElement { + + // 我们可以看到，默认只提供 sington 和 prototype 两种， + // 很多读者可能知道还有 request, session, globalSession, application, websocket 这几种， + // 不过，它们属于基于 web 的扩展。 + String SCOPE_SINGLETON = ConfigurableBeanFactory.SCOPE_SINGLETON; + String SCOPE_PROTOTYPE = ConfigurableBeanFactory.SCOPE_PROTOTYPE; + + // 比较不重要，直接跳过吧 + int ROLE_APPLICATION = 0; + int ROLE_SUPPORT = 1; + int ROLE_INFRASTRUCTURE = 2; + + // 设置父 Bean，这里涉及到 bean 继承，不是 java 继承。请参见附录的详细介绍 + // 一句话就是：继承父 Bean 的配置信息而已 + void setParentName(String parentName); + + // 获取父 Bean + String getParentName(); + + // 设置 Bean 的类名称，将来是要通过反射来生成实例的 + void setBeanClassName(String beanClassName); + + // 获取 Bean 的类名称 + String getBeanClassName(); + + // 设置 bean 的 scope + @Nullable + void setScope(String scope); + + @Nullable + String getScope(); + + // 设置是否懒加载 + void setLazyInit(boolean lazyInit); + + boolean isLazyInit(); + + // 设置该 Bean 依赖的所有的 Bean，注意，这里的依赖不是指属性依赖(如 @Autowire 标记的)， + // 是 depends-on="" 属性设置的值。 + void setDependsOn(String... dependsOn); + + // 返回该 Bean 的所有依赖 + String[] getDependsOn(); + + // 设置该 Bean 是否可以注入到其他 Bean 中，只对根据类型注入有效， + // 如果根据名称注入，即使这边设置了 false，也是可以的 + void setAutowireCandidate(boolean autowireCandidate); + + // 该 Bean 是否可以注入到其他 Bean 中 + boolean isAutowireCandidate(); + + // 主要的。同一接口的多个实现，如果不指定名字的话，Spring 会优先选择设置 primary 为 true 的 bean + void setPrimary(boolean primary); + + // 是否是 primary 的 + boolean isPrimary(); + + // 如果该 Bean 采用工厂方法生成，指定工厂名称。对工厂不熟悉的读者，请参加附录 + // 一句话就是：有些实例不是用反射生成的，而是用工厂模式生成的 + void setFactoryBeanName(String factoryBeanName); + // 获取工厂名称 + String getFactoryBeanName(); + // 指定工厂类中的 工厂方法名称 + void setFactoryMethodName(String factoryMethodName); + // 获取工厂类中的 工厂方法名称 + String getFactoryMethodName(); + + // 构造器参数 + ConstructorArgumentValues getConstructorArgumentValues(); + + // Bean 中的属性值，后面给 bean 注入属性值的时候会说到 + MutablePropertyValues getPropertyValues(); + + // 是否 singleton + boolean isSingleton(); + + // 是否 prototype + boolean isPrototype(); + + // 如果这个 Bean 是被设置为 abstract，那么不能实例化， + // 常用于作为 父bean 用于继承，其实也很少用...... + boolean isAbstract(); + + int getRole(); + String getDescription(); + String getResourceDescription(); + BeanDefinition getOriginatingBeanDefinition(); +} +``` + + + +> 这个 BeanDefinition 其实已经包含很多的信息了，暂时不清楚所有的方法对应什么东西没关系，希望看完本文后读者可以彻底搞清楚里面的所有东西。 +> +> 这里接口虽然那么多，但是没有类似 getInstance() 这种方法来获取我们定义的类的实例，真正的我们定义的类生成的实例到哪里去了呢？别着急，这个要很后面才能讲到。 + +有了 BeanDefinition 的概念以后，我们再往下看 `refreshBeanFactory()` 方法中的剩余部分： + +```java +customizeBeanFactory(beanFactory); +loadBeanDefinitions(beanFactory); +``` + +虽然只有两个方法，但路还很长啊。。。 + +##### customizeBeanFactory() + +customizeBeanFactory(beanFactory) 比较简单，就是配置是否允许 BeanDefinition 覆盖、是否允许循环引用。 + +```java +protected void customizeBeanFactory(DefaultListableBeanFactory beanFactory) { + if (this.allowBeanDefinitionOverriding != null) { + // 是否允许 Bean 定义覆盖 + beanFactory.setAllowBeanDefinitionOverriding(this.allowBeanDefinitionOverriding); + } + if (this.allowCircularReferences != null) { + // 是否允许 Bean 间的循环依赖 + beanFactory.setAllowCircularReferences(this.allowCircularReferences) + } +} +``` + +BeanDefinition 的覆盖问题可能会有开发者碰到这个坑，就是在配置文件中定义 bean 时使用了相同的 id 或 name，默认情况下，allowBeanDefinitionOverriding 属性为 null，如果在同一配置文件中重复了，会抛错，但是如果不是同一配置文件中，会发生覆盖。 + +循环引用也很好理解：A 依赖 B，而 B 依赖 A。或 A 依赖 B，B 依赖 C，而 C 依赖 A。 + +默认情况下，Spring 允许循环依赖，当然如果你在 A 的构造方法中依赖 B，在 B 的构造方法中依赖 A 是不行的。 + +至于这两个属性怎么配置？我在附录中进行了介绍，尤其对于覆盖问题，很多人都希望禁止出现 Bean 覆盖，可是 Spring 默认是不同文件的时候可以覆盖的。 + +之后的源码中还会出现这两个属性，先有个印象就可以了。 + +##### loadBeanDefinitions()：加载 Bean + +接下来是最重要的 `loadBeanDefinitions(beanFactory)` 方法了，这个方法将根据配置，加载各个 Bean，然后放到 BeanFactory 中。 + +读取配置的操作在 XmlBeanDefinitionReader 中，其负责加载配置、解析。 + +```java +/** 我们可以看到，此方法将通过一个 XmlBeanDefinitionReader 实例来加载各个 Bean。*/ +@Override +protected void loadBeanDefinitions(DefaultListableBeanFactory beanFactory) throws BeansException, IOException { + // 给这个 BeanFactory 实例化一个 XmlBeanDefinitionReader + XmlBeanDefinitionReader beanDefinitionReader = new XmlBeanDefinitionReader(beanFactory); + + // Configure the bean definition reader with this context's + // resource loading environment. + beanDefinitionReader.setEnvironment(this.getEnvironment()); + beanDefinitionReader.setResourceLoader(this); + beanDefinitionReader.setEntityResolver(new ResourceEntityResolver(this)); + + // 初始化 BeanDefinitionReader，其实这个是提供给子类覆写的， + // 我看了一下，没有类覆写这个方法，我们姑且当做不重要吧 + initBeanDefinitionReader(beanDefinitionReader); + // 重点来了，继续往下 + loadBeanDefinitions(beanDefinitionReader); +} +``` + +现在还在这个类中，接下来用刚刚初始化的 Reader 开始来加载 xml 配置，这块代码读者可以选择性跳过，不是很重要。也就是说，下面这个代码块，读者可以很轻松地略过。 + +```java +protected void loadBeanDefinitions(XmlBeanDefinitionReader reader) throws BeansException, IOException { + Resource[] configResources = getConfigResources(); + if (configResources != null) { + // 往下看 + reader.loadBeanDefinitions(configResources); + } + String[] configLocations = getConfigLocations(); + if (configLocations != null) { + // 2 + reader.loadBeanDefinitions(configLocations); + } +} + +// 上面虽然有两个分支，不过第二个分支很快通过解析路径转换为 Resource 以后也会进到这里 +@Override +public int loadBeanDefinitions(Resource... resources) throws BeanDefinitionStoreException { + Assert.notNull(resources, "Resource array must not be null"); + int counter = 0; + // 注意这里是个 for 循环，也就是每个文件是一个 resource + for (Resource resource : resources) { + // 继续往下看 + counter += loadBeanDefinitions(resource); + } + // 最后返回 counter，表示总共加载了多少的 BeanDefinition + return counter; +} + +// XmlBeanDefinitionReader 303 +@Override +public int loadBeanDefinitions(Resource resource) throws BeanDefinitionStoreException { + return loadBeanDefinitions(new EncodedResource(resource)); +} + +// XmlBeanDefinitionReader 314 +public int loadBeanDefinitions(EncodedResource encodedResource) throws BeanDefinitionStoreException { + Assert.notNull(encodedResource, "EncodedResource must not be null"); + if (logger.isInfoEnabled()) { + logger.info("Loading XML bean definitions from " + encodedResource.getResource()); + } + // 用一个 ThreadLocal 来存放配置文件资源 + Set currentResources = this.resourcesCurrentlyBeingLoaded.get(); + if (currentResources == null) { + currentResources = new HashSet(4); + this.resourcesCurrentlyBeingLoaded.set(currentResources); + } + if (!currentResources.add(encodedResource)) { + throw new BeanDefinitionStoreException( + "Detected cyclic loading of " + encodedResource + " - check your import definitions!"); + } + try { + InputStream inputStream = encodedResource.getResource().getInputStream(); + try { + InputSource inputSource = new InputSource(inputStream); + if (encodedResource.getEncoding() != null) { + inputSource.setEncoding(encodedResource.getEncoding()); + } + // 核心部分是这里，往下面看 + return doLoadBeanDefinitions(inputSource, encodedResource.getResource()); + } + finally { + inputStream.close(); + } + } + catch (IOException ex) { + throw new BeanDefinitionStoreException( + "IOException parsing XML document from " + encodedResource.getResource(), ex); + } + finally { + currentResources.remove(encodedResource); + if (currentResources.isEmpty()) { + this.resourcesCurrentlyBeingLoaded.remove(); + } + } +} + +// 还在这个文件中，第 388 行 +protected int doLoadBeanDefinitions(InputSource inputSource, Resource resource) + throws BeanDefinitionStoreException { + try { + // 这里就不看了，将 xml 文件转换为 Document 对象 + Document doc = doLoadDocument(inputSource, resource); + // 继续 + return registerBeanDefinitions(doc, resource); + } + catch (... +} + +// 还在这个文件中，第 505 行 +// 返回值：返回从当前配置文件加载了多少数量的 Bean +public int registerBeanDefinitions(Document doc, Resource resource) throws BeanDefinitionStoreException { + BeanDefinitionDocumentReader documentReader = createBeanDefinitionDocumentReader(); + int countBefore = getRegistry().getBeanDefinitionCount(); + // 这里 + documentReader.registerBeanDefinitions(doc, createReaderContext(resource)); + return getRegistry().getBeanDefinitionCount() - countBefore; +} + +// DefaultBeanDefinitionDocumentReader.java +@Override +public void registerBeanDefinitions(Document doc, XmlReaderContext readerContext) { + this.readerContext = readerContext; + logger.debug("Loading bean definitions"); + Element root = doc.getDocumentElement(); + // 从 xml 根节点开始解析文件 + doRegisterBeanDefinitions(root); +} +``` + +经过漫长的链路，一个配置文件终于转换为一颗 DOM 树了，注意，这里指的是其中一个配置文件，不是所有的，读者可以看到上面有个 for 循环的。下面开始从根节点开始解析： + +##### doRegisterBeanDefinitions() + +```java +// DefaultBeanDefinitionDocumentReader.java +protected void doRegisterBeanDefinitions(Element root) { + // 我们看名字就知道，BeanDefinitionParserDelegate 必定是一个重要的类，它负责解析 Bean 定义， + // 这里为什么要定义一个 parent? 看到后面就知道了，是递归问题， + // 因为 内部是可以定义 的，所以这个方法的 root 其实不一定就是 xml 的根节点，也可以是嵌套在里面的 节点，从源码分析的角度，我们当做根节点就好了 + BeanDefinitionParserDelegate parent = this.delegate; + this.delegate = createDelegate(getReaderContext(), root, parent); + + if (this.delegate.isDefaultNamespace(root)) { + // 这块说的是根节点 中的 profile 是否是当前环境需要的， + // 如果当前环境配置的 profile 不包含此 profile，那就直接 return 了，不对此 解析 + // 不熟悉 profile 为何物，不熟悉怎么配置 profile 读者的请移步附录区 + String profileSpec = root.getAttribute(PROFILE_ATTRIBUTE); + if (StringUtils.hasText(profileSpec)) { + String[] specifiedProfiles = StringUtils.tokenizeToStringArray( + profileSpec, BeanDefinitionParserDelegate.MULTI_VALUE_ATTRIBUTE_DELIMITERS); + if (!getReaderContext().getEnvironment().acceptsProfiles(specifiedProfiles)) { + if (logger.isInfoEnabled()) { + logger.info("Skipped XML bean definition file due to specified profiles [" + profileSpec + + "] not matching: " + getReaderContext().getResource()); + } + return; + } + } + } + + preProcessXml(root); // 钩子 + // 往下看 + parseBeanDefinitions(root, this.delegate); + postProcessXml(root); // 钩子 + + this.delegate = parent; +} +``` + +`preProcessXml(root)` 和 `postProcessXml(root)` 是给子类用的钩子方法，鉴于没有被使用到，也不是我们的重点，我们直接跳过。 + +这里涉及到了 profile 的问题，对于不了解的读者，我在附录中对 profile 做了简单的解释，读者可以参考一下。接下来，看核心解析方法 + +##### parseBeanDefinitions(root, this.delegate) + +```java +// default namespace 涉及到的就四个标签 、、 和 ， +// 其他的属于 custom 的 +protected void parseBeanDefinitions(Element root, BeanDefinitionParserDelegate delegate) { + if (delegate.isDefaultNamespace(root)) { + NodeList nl = root.getChildNodes(); + for (int i = 0; i < nl.getLength(); i++) { + Node node = nl.item(i); + if (node instanceof Element) { + Element ele = (Element) node; + if (delegate.isDefaultNamespace(ele)) { + // 解析 default namespace 下面的几个元素 + parseDefaultElement(ele, delegate); + } + else { + // 解析其他 namespace 的元素 + delegate.parseCustomElement(ele); + } + } + } + } + else { + delegate.parseCustomElement(root); + } +} +``` + +从上面的代码，我们可以看到，对于每个配置来说，分别进入到 `parseDefaultElement(ele, delegate);` 和 `delegate.parseCustomElement(ele);` 这两个分支了。 + +`parseDefaultElement(ele, delegate)` 代表解析的节点是 ``、``、``、`` 这几个。 + +> 这里的四个标签之所以是 default 的，是因为它们是处于这个 namespace 下定义的： +> +> http://www.springframework.org/schema/beans +> +> 又到初学者科普时间，不熟悉 namespace 的读者请看下面贴出来的 xml，这里的第二行 **xmlns** 就是咯。 +> +> ```xml +> xmlns="http://www.springframework.org/schema/beans" +> xsi:schemaLocation=" +> http://www.springframework.org/schema/beans +> http://www.springframework.org/schema/beans/spring-beans.xsd" +> default-autowire="byName"> +> ``` +> +> 而对于其他的标签，将进入到 `delegate.parseCustomElement(element)` 这个分支。如我们经常会使用到的 ``、``、``、``等。 +> +> 这些属于扩展，如果需要使用上面这些 ”非 default“ 标签，那么上面的 xml 头部的地方也要引入相应的 namespace 和 .xsd 文件的路径，如下所示。同时代码中需要提供相应的 parser 来解析，如 MvcNamespaceHandler、TaskNamespaceHandler、ContextNamespaceHandler、AopNamespaceHandler 等。 +> +> 假如读者想分析 `` 的实现原理，就应该到 ContextNamespaceHandler 中找答案。 +> +> ```xml +> xmlns="http://www.springframework.org/schema/beans" +> xmlns:context="http://www.springframework.org/schema/context" +> xmlns:mvc="http://www.springframework.org/schema/mvc" +> xsi:schemaLocation=" +> http://www.springframework.org/schema/beans +> http://www.springframework.org/schema/beans/spring-beans.xsd +> http://www.springframework.org/schema/context +> http://www.springframework.org/schema/context/spring-context.xsd +> http://www.springframework.org/schema/mvc +> http://www.springframework.org/schema/mvc/spring-mvc.xsd +> " +> default-autowire="byName"> +> ``` + +回过神来，看看处理 default 标签的方法： + +```java +private void parseDefaultElement(Element ele, BeanDefinitionParserDelegate delegate) { + if (delegate.nodeNameEquals(ele, IMPORT_ELEMENT)) { + // 处理 标签 + importBeanDefinitionResource(ele); + } + else if (delegate.nodeNameEquals(ele, ALIAS_ELEMENT)) { + // 处理 标签定义 + // + processAliasRegistration(ele); + } + else if (delegate.nodeNameEquals(ele, BEAN_ELEMENT)) { + // 处理 标签定义，这也算是我们的重点吧 + processBeanDefinition(ele, delegate); + } + else if (delegate.nodeNameEquals(ele, NESTED_BEANS_ELEMENT)) { + // 如果碰到的是嵌套的 标签，需要递归 + doRegisterBeanDefinitions(ele); + } +} +``` + +如果每个标签都说，那我不吐血，你们都要吐血了。我们挑我们的重点 `` 标签出来说。 + +##### processBeanDefinition 解析 bean 标签 + +下面是 processBeanDefinition 解析 `` 标签： + +// DefaultBeanDefinitionDocumentReader.java + +```java +protected void processBeanDefinition(Element ele, BeanDefinitionParserDelegate delegate) { + // 将 节点中的信息提取出来，然后封装到一个 BeanDefinitionHolder 中，细节往下看 + BeanDefinitionHolder bdHolder = delegate.parseBeanDefinitionElement(ele); + + // 下面的几行先不要看，跳过先，跳过先，跳过先，后面会继续说的 + + if (bdHolder != null) { + bdHolder = delegate.decorateBeanDefinitionIfRequired(ele, bdHolder); + try { + // Register the final decorated instance. + BeanDefinitionReaderUtils.registerBeanDefinition(bdHolder, getReaderContext().getRegistry()); + } + catch (BeanDefinitionStoreException ex) { + getReaderContext().error("Failed to register bean definition with name '" + + bdHolder.getBeanName() + "'", ele, ex); + } + // Send registration event. + getReaderContext().fireComponentRegistered(new BeanComponentDefinition(bdHolder)); + } +} +``` + + + +继续往下看怎么解析之前，我们先看下 `` 标签中可以定义哪些属性： + +| Property | | +| ------------------------ | ------------------------------------------------------------ | +| class | 类的全限定名 | +| name | 可指定 id、name(用逗号、分号、空格分隔) | +| scope | 作用域 | +| constructor arguments | 指定构造参数 | +| properties | 设置属性的值 | +| autowiring mode | no(默认值)、byName、byType、 constructor | +| lazy-initialization mode | 是否懒加载(如果被非懒加载的bean依赖了那么其实也就不能懒加载了) | +| initialization method | bean 属性设置完成后，会调用这个方法 | +| destruction method | bean 销毁后的回调方法 | + +上面表格中的内容我想大家都非常熟悉吧，如果不熟悉，那就是你不够了解 Spring 的配置了。 + +简单地说就是像下面这样子： + +```xml + + + + + + + + + + + + + + +``` + +当然，除了上面举例出来的这些，还有 factory-bean、factory-method、``、``、``、`` 这几个，大家是不是熟悉呢？自己检验一下自己对 Spring 中 bean 的了解程度。 + +有了以上这些知识以后，我们再继续往里看怎么解析 bean 元素，是怎么转换到 BeanDefinitionHolder 的。 + +// BeanDefinitionParserDelegate.java + +```java +public BeanDefinitionHolder parseBeanDefinitionElement(Element ele) { + return parseBeanDefinitionElement(ele, null); +} + +public BeanDefinitionHolder parseBeanDefinitionElement(Element ele, BeanDefinition containingBean) { + String id = ele.getAttribute(ID_ATTRIBUTE); + String nameAttr = ele.getAttribute(NAME_ATTRIBUTE); + + List aliases = new ArrayList(); + + // 将 name 属性的定义按照 “逗号、分号、空格” 切分，形成一个 别名列表数组， + // 当然，如果你不定义 name 属性的话，就是空的了 + // 我在附录中简单介绍了一下 id 和 name 的配置，大家可以看一眼，有个20秒就可以了 + if (StringUtils.hasLength(nameAttr)) { + String[] nameArr = StringUtils.tokenizeToStringArray(nameAttr, MULTI_VALUE_ATTRIBUTE_DELIMITERS); + aliases.addAll(Arrays.asList(nameArr)); + } + + String beanName = id; + // 如果没有指定id, 那么用别名列表的第一个名字作为beanName + if (!StringUtils.hasText(beanName) && !aliases.isEmpty()) { + beanName = aliases.remove(0); + if (logger.isDebugEnabled()) { + logger.debug("No XML 'id' specified - using '" + beanName + + "' as bean name and " + aliases + " as aliases"); + } + } + + if (containingBean == null) { + checkNameUniqueness(beanName, aliases, ele); + } + + // 根据 ... 中的配置创建 BeanDefinition，然后把配置中的信息都设置到实例中, + // 细节后面细说，先知道下面这行结束后，一个 BeanDefinition 实例就出来了。 + AbstractBeanDefinition beanDefinition = parseBeanDefinitionElement(ele, beanName, containingBean); + + // 到这里，整个 标签就算解析结束了，一个 BeanDefinition 就形成了。 + if (beanDefinition != null) { + // 如果都没有设置 id 和 name，那么此时的 beanName 就会为 null，进入下面这块代码产生 + // 如果读者不感兴趣的话，我觉得不需要关心这块代码，对本文源码分析来说，这些东西不重要 + if (!StringUtils.hasText(beanName)) { + try { + if (containingBean != null) {// 按照我们的思路，这里 containingBean 是 null 的 + beanName = BeanDefinitionReaderUtils.generateBeanName( + beanDefinition, this.readerContext.getRegistry(), true); + } + else { + // 如果我们不定义 id 和 name，那么我们引言里的那个例子： + // 1. beanName 为：com.javadoop.example.MessageServiceImpl#0 + // 2. beanClassName 为：com.javadoop.example.MessageServiceImpl + + beanName = this.readerContext.generateBeanName(beanDefinition); + + String beanClassName = beanDefinition.getBeanClassName(); + if (beanClassName != null && + beanName.startsWith(beanClassName) && beanName.length() > beanClassName.length() && + !this.readerContext.getRegistry().isBeanNameInUse(beanClassName)) { + // 把 beanClassName 设置为 Bean 的别名 + aliases.add(beanClassName); + } + } + if (logger.isDebugEnabled()) { + logger.debug("Neither XML 'id' nor 'name' specified - " + + "using generated bean name [" + beanName + "]"); + } + } + catch (Exception ex) { + error(ex.getMessage(), ele); + return null; + } + } + String[] aliasesArray = StringUtils.toStringArray(aliases); + // 返回 BeanDefinitionHolder + return new BeanDefinitionHolder(beanDefinition, beanName, aliasesArray); + } + + return null; +} +``` + +然后，我们再看看怎么根据配置创建 BeanDefinition 实例的： + +```java +public AbstractBeanDefinition parseBeanDefinitionElement( + Element ele, String beanName, BeanDefinition containingBean) { + + this.parseState.push(new BeanEntry(beanName)); + + String className = null; + if (ele.hasAttribute(CLASS_ATTRIBUTE)) { + className = ele.getAttribute(CLASS_ATTRIBUTE).trim(); + } + + try { + String parent = null; + if (ele.hasAttribute(PARENT_ATTRIBUTE)) { + parent = ele.getAttribute(PARENT_ATTRIBUTE); + } + // 创建 BeanDefinition，然后设置类信息而已，很简单，就不贴代码了 + AbstractBeanDefinition bd = createBeanDefinition(className, parent); + + // 设置 BeanDefinition 的一堆属性，这些属性定义在 AbstractBeanDefinition 中 + parseBeanDefinitionAttributes(ele, beanName, containingBean, bd); + bd.setDescription(DomUtils.getChildElementValueByTagName(ele, DESCRIPTION_ELEMENT)); + + /** + * 下面的一堆是解析 ...... 内部的子元素， + * 解析出来以后的信息都放到 bd 的属性中 + */ + + // 解析 + parseMetaElements(ele, bd); + // 解析 + parseLookupOverrideSubElements(ele, bd.getMethodOverrides()); + // 解析 + parseReplacedMethodSubElements(ele, bd.getMethodOverrides()); + // 解析 + parseConstructorArgElements(ele, bd); + // 解析 + parsePropertyElements(ele, bd); + // 解析 + parseQualifierElements(ele, bd); + + bd.setResource(this.readerContext.getResource()); + bd.setSource(extractSource(ele)); + + return bd; + } + catch (ClassNotFoundException ex) { + error("Bean class [" + className + "] not found", ele, ex); + } + catch (NoClassDefFoundError err) { + error("Class that bean class [" + className + "] depends on not found", ele, err); + } + catch (Throwable ex) { + error("Unexpected failure during bean definition parsing", ele, ex); + } + finally { + this.parseState.pop(); + } + + return null; +} +``` + +到这里，我们已经完成了根据 `` 配置创建了一个 BeanDefinitionHolder 实例。注意，是一个。 + +我们回到解析 `` 的入口方法: + +```java +protected void processBeanDefinition(Element ele, BeanDefinitionParserDelegate delegate) { + // 将 节点转换为 BeanDefinitionHolder，就是上面说的一堆 + BeanDefinitionHolder bdHolder = delegate.parseBeanDefinitionElement(ele); + if (bdHolder != null) { + // 如果有自定义属性的话，进行相应的解析，先忽略 + bdHolder = delegate.decorateBeanDefinitionIfRequired(ele, bdHolder); + try { + // 我们把这步叫做 注册Bean 吧 + BeanDefinitionReaderUtils.registerBeanDefinition(bdHolder, getReaderContext().getRegistry()); + } + catch (BeanDefinitionStoreException ex) { + getReaderContext().error("Failed to register bean definition with name '" + + bdHolder.getBeanName() + "'", ele, ex); + } + // 注册完成后，发送事件，本文不展开说这个 + getReaderContext().fireComponentRegistered(new BeanComponentDefinition(bdHolder)); + } +} +``` + +大家再仔细看一下这块吧，我们后面就不回来说这个了。这里已经根据一个 `` 标签产生了一个 BeanDefinitionHolder 的实例，这个实例里面也就是一个 BeanDefinition 的实例和它的 beanName、aliases 这三个信息，注意，我们的关注点始终在 BeanDefinition 上： + +```java +public class BeanDefinitionHolder implements BeanMetadataElement { + + private final BeanDefinition beanDefinition; + + private final String beanName; + + private final String[] aliases; +... +``` + +然后我们准备注册这个 BeanDefinition，最后，把这个注册事件发送出去。 + +下面，我们开始说说注册 Bean 吧。 + +##### 注册 Bean + +// BeanDefinitionReaderUtils.java + +```java +public static void registerBeanDefinition( + BeanDefinitionHolder definitionHolder, BeanDefinitionRegistry registry) + throws BeanDefinitionStoreException { + + String beanName = definitionHolder.getBeanName(); + // 注册这个 Bean + registry.registerBeanDefinition(beanName, definitionHolder.getBeanDefinition()); + + // 如果还有别名的话，也要根据别名全部注册一遍，不然根据别名就会找不到 Bean 了 + String[] aliases = definitionHolder.getAliases(); + if (aliases != null) { + for (String alias : aliases) { + // alias -> beanName 保存它们的别名信息，这个很简单，用一个 map 保存一下就可以了， + // 获取的时候，会先将 alias 转换为 beanName，然后再查找 + registry.registerAlias(beanName, alias); + } + } +} +``` + +别名注册的放一边，毕竟它很简单，我们看看怎么注册 Bean。 + +// DefaultListableBeanFactory.java 793 + +```java +@Override +public void registerBeanDefinition(String beanName, BeanDefinition beanDefinition) + throws BeanDefinitionStoreException { + + Assert.hasText(beanName, "Bean name must not be empty"); + Assert.notNull(beanDefinition, "BeanDefinition must not be null"); + + if (beanDefinition instanceof AbstractBeanDefinition) { + try { + ((AbstractBeanDefinition) beanDefinition).validate(); + } + catch (BeanDefinitionValidationException ex) { + throw new BeanDefinitionStoreException(...); + } + } + + // old? 还记得 “允许 bean 覆盖” 这个配置吗？allowBeanDefinitionOverriding + BeanDefinition oldBeanDefinition; + + // 之后会看到，所有的 Bean 注册后会放入这个 beanDefinitionMap 中 + oldBeanDefinition = this.beanDefinitionMap.get(beanName); + + // 处理重复名称的 Bean 定义的情况 + if (oldBeanDefinition != null) { + if (!isAllowBeanDefinitionOverriding()) { + // 如果不允许覆盖的话，抛异常 + throw new BeanDefinitionStoreException(beanDefinition.getResourceDescription()... + } + else if (oldBeanDefinition.getRole() < beanDefinition.getRole()) { + // log...用框架定义的 Bean 覆盖用户自定义的 Bean + } + else if (!beanDefinition.equals(oldBeanDefinition)) { + // log...用新的 Bean 覆盖旧的 Bean + } + else { + // log...用同等的 Bean 覆盖旧的 Bean，这里指的是 equals 方法返回 true 的 Bean + } + // 覆盖 + this.beanDefinitionMap.put(beanName, beanDefinition); + } + else { + // 判断是否已经有其他的 Bean 开始初始化了. + // 注意，"注册Bean" 这个动作结束，Bean 依然还没有初始化，我们后面会有大篇幅说初始化过程， + // 在 Spring 容器启动的最后，会 预初始化 所有的 singleton beans + if (hasBeanCreationStarted()) { + // Cannot modify startup-time collection elements anymore (for stable iteration) + synchronized (this.beanDefinitionMap) { + this.beanDefinitionMap.put(beanName, beanDefinition); + List updatedDefinitions = new ArrayList(this.beanDefinitionNames.size() + 1); + updatedDefinitions.addAll(this.beanDefinitionNames); + updatedDefinitions.add(beanName); + this.beanDefinitionNames = updatedDefinitions; + if (this.manualSingletonNames.contains(beanName)) { + Set updatedSingletons = new LinkedHashSet(this.manualSingletonNames); + updatedSingletons.remove(beanName); + this.manualSingletonNames = updatedSingletons; + } + } + } + else { + // 最正常的应该是进到这个分支。 + + // 将 BeanDefinition 放到这个 map 中，这个 map 保存了所有的 BeanDefinition + this.beanDefinitionMap.put(beanName, beanDefinition); + // 这是个 ArrayList，所以会按照 bean 配置的顺序保存每一个注册的 Bean 的名字 + this.beanDefinitionNames.add(beanName); + // 这是个 LinkedHashSet，代表的是手动注册的 singleton bean， + // 注意这里是 remove 方法，到这里的 Bean 当然不是手动注册的 + // 手动指的是通过调用以下方法注册的 bean ： + // registerSingleton(String beanName, Object singletonObject) + // 这不是重点，解释只是为了不让大家疑惑。Spring 会在后面"手动"注册一些 Bean， + // 如 "environment"、"systemProperties" 等 bean，我们自己也可以在运行时注册 Bean 到容器中的 + this.manualSingletonNames.remove(beanName); + } + // 这个不重要，在预初始化的时候会用到，不必管它。 + this.frozenBeanDefinitionNames = null; + } + + if (oldBeanDefinition != null || containsSingleton(beanName)) { + resetBeanDefinition(beanName); + } +} +``` + +总结一下，到这里已经初始化了 Bean 容器，`` 配置也相应的转换为了一个个 BeanDefinition，然后注册了各个 BeanDefinition 到注册中心，并且发送了注册事件。 + +> 到这里是一个分水岭，前面的内容都还算比较简单，大家要清楚地知道前面都做了哪些事情。 + +#### Bean 容器实例化完成后 + +说到这里，我们回到 refresh() 方法，我重新贴了一遍代码，看看我们说到哪了。是的，我们才说完 `obtainFreshBeanFactory()` 方法。 + +考虑到篇幅，这里开始大幅缩减掉没必要详细介绍的部分，大家直接看下面的代码中的注释就好了。 + +```java +@Override +public void refresh() throws BeansException, IllegalStateException { + // 来个锁，不然 refresh() 还没结束，你又来个启动或销毁容器的操作，那不就乱套了嘛 + synchronized (this.startupShutdownMonitor) { + + // 准备工作，记录下容器的启动时间、标记“已启动”状态、处理配置文件中的占位符 + prepareRefresh(); + + // 这步比较关键，这步完成后，配置文件就会解析成一个个 Bean 定义，注册到 BeanFactory 中， + // 当然，这里说的 Bean 还没有初始化，只是配置信息都提取出来了， + // 注册也只是将这些信息都保存到了注册中心(说到底核心是一个 beanName-> beanDefinition 的 map) + ConfigurableListableBeanFactory beanFactory = obtainFreshBeanFactory(); + + // 设置 BeanFactory 的类加载器，添加几个 BeanPostProcessor，手动注册几个特殊的 bean + // 这块待会会展开说 + prepareBeanFactory(beanFactory); + + try { + // 【这里需要知道 BeanFactoryPostProcessor 这个知识点，Bean 如果实现了此接口， + // 那么在容器初始化以后，Spring 会负责调用里面的 postProcessBeanFactory 方法。】 + + // 这里是提供给子类的扩展点，到这里的时候，所有的 Bean 都加载、注册完成了，但是都还没有初始化 + // 具体的子类可以在这步的时候添加一些特殊的 BeanFactoryPostProcessor 的实现类或做点什么事 + postProcessBeanFactory(beanFactory); + // 调用 BeanFactoryPostProcessor 各个实现类的 postProcessBeanFactory(factory) 回调方法 + invokeBeanFactoryPostProcessors(beanFactory); + + + + // 注册 BeanPostProcessor 的实现类，注意看和 BeanFactoryPostProcessor 的区别 + // 此接口两个方法: postProcessBeforeInitialization 和 postProcessAfterInitialization + // 两个方法分别在 Bean 初始化之前和初始化之后得到执行。这里仅仅是注册，之后会看到回调这两方法的时机 + registerBeanPostProcessors(beanFactory); + + // 初始化当前 ApplicationContext 的 MessageSource，国际化这里就不展开说了，不然没完没了了 + initMessageSource(); + + // 初始化当前 ApplicationContext 的事件广播器，这里也不展开了 + initApplicationEventMulticaster(); + + // 从方法名就可以知道，典型的模板方法(钩子方法)，不展开说 + // 具体的子类可以在这里初始化一些特殊的 Bean（在初始化 singleton beans 之前） + onRefresh(); + + // 注册事件监听器，监听器需要实现 ApplicationListener 接口。这也不是我们的重点，过 + registerListeners(); + + // 重点，重点，重点 + // 初始化所有的 singleton beans + //（lazy-init 的除外） + finishBeanFactoryInitialization(beanFactory); + + // 最后，广播事件，ApplicationContext 初始化完成，不展开 + finishRefresh(); + } + + catch (BeansException ex) { + if (logger.isWarnEnabled()) { + logger.warn("Exception encountered during context initialization - " + + "cancelling refresh attempt: " + ex); + } + + // Destroy already created singletons to avoid dangling resources. + // 销毁已经初始化的 singleton 的 Beans，以免有些 bean 会一直占用资源 + destroyBeans(); + + // Reset 'active' flag. + cancelRefresh(ex); + + // 把异常往外抛 + throw ex; + } + + finally { + // Reset common introspection caches in Spring's core, since we + // might not ever need metadata for singleton beans anymore... + resetCommonCaches(); + } + } +} +``` + + + +#### 2.3 准备 Bean 容器: prepareBeanFactory + +之前我们说过，Spring 把我们在 xml 配置的 bean 都注册以后，会"手动"注册一些特殊的 bean。 + +这里简单介绍下 `prepareBeanFactory(factory)` 方法： + +##### prepareBeanFactory(factory) + +```java +/** + * Configure the factory's standard context characteristics, + * such as the context's ClassLoader and post-processors. + * @param beanFactory the BeanFactory to configure + */ +protected void prepareBeanFactory(ConfigurableListableBeanFactory beanFactory) { + // 设置 BeanFactory 的类加载器，我们知道 BeanFactory 需要加载类，也就需要类加载器， + // 这里设置为加载当前 ApplicationContext 类的类加载器 + beanFactory.setBeanClassLoader(getClassLoader()); + + // 设置 BeanExpressionResolver + beanFactory.setBeanExpressionResolver(new StandardBeanExpressionResolver(beanFactory.getBeanClassLoader())); + // + beanFactory.addPropertyEditorRegistrar(new ResourceEditorRegistrar(this, getEnvironment())); + + // 添加一个 BeanPostProcessor，这个 processor 比较简单： + // 实现了 Aware 接口的 beans 在初始化的时候，这个 processor 负责回调， + // 这个我们很常用，如我们会为了获取 ApplicationContext 而 implement ApplicationContextAware + // 注意：它不仅仅回调 ApplicationContextAware， + // 还会负责回调 EnvironmentAware、ResourceLoaderAware 等，看下源码就清楚了 + beanFactory.addBeanPostProcessor(new ApplicationContextAwareProcessor(this)); + + // 下面几行的意思就是，如果某个 bean 依赖于以下几个接口的实现类，在自动装配的时候忽略它们， + // Spring 会通过其他方式来处理这些依赖。 + beanFactory.ignoreDependencyInterface(EnvironmentAware.class); + beanFactory.ignoreDependencyInterface(EmbeddedValueResolverAware.class); + beanFactory.ignoreDependencyInterface(ResourceLoaderAware.class); + beanFactory.ignoreDependencyInterface(ApplicationEventPublisherAware.class); + beanFactory.ignoreDependencyInterface(MessageSourceAware.class); + beanFactory.ignoreDependencyInterface(ApplicationContextAware.class); + + /** + * 下面几行就是为特殊的几个 bean 赋值，如果有 bean 依赖了以下几个，会注入这边相应的值， + * 之前我们说过，"当前 ApplicationContext 持有一个 BeanFactory"，这里解释了第一行 + * ApplicationContext 还继承了 ResourceLoader、ApplicationEventPublisher、MessageSource + * 所以对于这几个依赖，可以赋值为 this，注意 this 是一个 ApplicationContext + * 那这里怎么没看到为 MessageSource 赋值呢？那是因为 MessageSource 被注册成为了一个普通的 bean + */ + beanFactory.registerResolvableDependency(BeanFactory.class, beanFactory); + beanFactory.registerResolvableDependency(ResourceLoader.class, this); + beanFactory.registerResolvableDependency(ApplicationEventPublisher.class, this); + beanFactory.registerResolvableDependency(ApplicationContext.class, this); + + // 这个 BeanPostProcessor 也很简单，在 bean 实例化后，如果是 ApplicationListener 的子类， + // 那么将其添加到 listener 列表中，可以理解成：注册 事件监听器 + beanFactory.addBeanPostProcessor(new ApplicationListenerDetector(this)); + + // 这里涉及到特殊的 bean，名为：loadTimeWeaver，这不是我们的重点，忽略它 + // tips: ltw 是 AspectJ 的概念，指的是在运行期进行织入，这个和 Spring AOP 不一样， + // 感兴趣的读者请参考我写的关于 AspectJ 的另一篇文章 https://www.javadoop.com/post/aspectj + if (beanFactory.containsBean(LOAD_TIME_WEAVER_BEAN_NAME)) { + beanFactory.addBeanPostProcessor(new LoadTimeWeaverAwareProcessor(beanFactory)); + // Set a temporary ClassLoader for type matching. + beanFactory.setTempClassLoader(new ContextTypeMatchClassLoader(beanFactory.getBeanClassLoader())); + } + + /** + * 从下面几行代码我们可以知道，Spring 往往很 "智能" 就是因为它会帮我们默认注册一些有用的 bean， + * 我们也可以选择覆盖 + */ + + // 如果没有定义 "environment" 这个 bean，那么 Spring 会 "手动" 注册一个 + if (!beanFactory.containsLocalBean(ENVIRONMENT_BEAN_NAME)) { + beanFactory.registerSingleton(ENVIRONMENT_BEAN_NAME, getEnvironment()); + } + // 如果没有定义 "systemProperties" 这个 bean，那么 Spring 会 "手动" 注册一个 + if (!beanFactory.containsLocalBean(SYSTEM_PROPERTIES_BEAN_NAME)) { + beanFactory.registerSingleton(SYSTEM_PROPERTIES_BEAN_NAME, getEnvironment().getSystemProperties()); + } + // 如果没有定义 "systemEnvironment" 这个 bean，那么 Spring 会 "手动" 注册一个 + if (!beanFactory.containsLocalBean(SYSTEM_ENVIRONMENT_BEAN_NAME)) { + beanFactory.registerSingleton(SYSTEM_ENVIRONMENT_BEAN_NAME, getEnvironment().getSystemEnvironment()); + } +} +``` + +在上面这块代码中，Spring 对一些特殊的 bean 进行了处理，读者如果暂时还不能消化它们也没有关系，慢慢往下看。 + +#### 2.4 初始化所有的 singleton beans + +我们的重点当然是 `finishBeanFactoryInitialization(beanFactory);` 这个巨头了，这里会负责初始化所有的 singleton beans。 + +注意，后面的描述中，我都会使用**初始化**或**预初始化**来代表这个阶段，Spring 会在这个阶段完成所有的 singleton beans 的实例化。 + +我们来总结一下，到目前为止，应该说 BeanFactory 已经创建完成，并且所有的实现了 BeanFactoryPostProcessor 接口的 Bean 都已经初始化并且其中的 postProcessBeanFactory(factory) 方法已经得到回调执行了。而且 Spring 已经“手动”注册了一些特殊的 Bean，如 ‘environment’、‘systemProperties’ 等。 + +剩下的就是初始化 singleton beans 了，我们知道它们是单例的，如果没有设置懒加载，那么 Spring 会在接下来初始化所有的 singleton beans。 + +// AbstractApplicationContext.java 834 + +```java +// 初始化剩余的 singleton beans +protected void finishBeanFactoryInitialization(ConfigurableListableBeanFactory beanFactory) { + + // 首先，初始化名字为 conversionService 的 Bean。本着送佛送到西的精神，我在附录中简单介绍了一下 ConversionService，因为这实在太实用了 + // 什么，看代码这里没有初始化 Bean 啊！ + // 注意了，初始化的动作包装在 beanFactory.getBean(...) 中，这里先不说细节，先往下看吧 + if (beanFactory.containsBean(CONVERSION_SERVICE_BEAN_NAME) && + beanFactory.isTypeMatch(CONVERSION_SERVICE_BEAN_NAME, ConversionService.class)) { + beanFactory.setConversionService( + beanFactory.getBean(CONVERSION_SERVICE_BEAN_NAME, ConversionService.class)); + } + + // Register a default embedded value resolver if no bean post-processor + // (such as a PropertyPlaceholderConfigurer bean) registered any before: + // at this point, primarily for resolution in annotation attribute values. + if (!beanFactory.hasEmbeddedValueResolver()) { + beanFactory.addEmbeddedValueResolver(new StringValueResolver() { + @Override + public String resolveStringValue(String strVal) { + return getEnvironment().resolvePlaceholders(strVal); + } + }); + } + + // 先初始化 LoadTimeWeaverAware 类型的 Bean + // 之前也说过，这是 AspectJ 相关的内容，放心跳过吧 + String[] weaverAwareNames = beanFactory.getBeanNamesForType(LoadTimeWeaverAware.class, false, false); + for (String weaverAwareName : weaverAwareNames) { + getBean(weaverAwareName); + } + + // Stop using the temporary ClassLoader for type matching. + beanFactory.setTempClassLoader(null); + + // 没什么别的目的，因为到这一步的时候，Spring 已经开始预初始化 singleton beans 了， + // 肯定不希望这个时候还出现 bean 定义解析、加载、注册。 + beanFactory.freezeConfiguration(); + + // 开始初始化 + beanFactory.preInstantiateSingletons(); +} +``` + +从上面最后一行往里看，我们就又回到 DefaultListableBeanFactory 这个类了，这个类大家应该都不陌生了吧。 + +// DefaultListableBeanFactory.java 728 + +```java +@Override +public void preInstantiateSingletons() throws BeansException { + if (this.logger.isDebugEnabled()) { + this.logger.debug("Pre-instantiating singletons in " + this); + } + // this.beanDefinitionNames 保存了所有的 beanNames + List beanNames = new ArrayList(this.beanDefinitionNames); + + // 触发所有的非懒加载的 singleton beans 的初始化操作 + for (String beanName : beanNames) { + + // 合并父 Bean 中的配置，注意 中的 parent，用的不多吧， + // 考虑到这可能会影响大家的理解，我在附录中解释了一下 "Bean 继承"，不了解的请到附录中看一下 + RootBeanDefinition bd = getMergedLocalBeanDefinition(beanName); + + // 非抽象、非懒加载的 singletons。如果配置了 'abstract = true'，那是不需要初始化的 + if (!bd.isAbstract() && bd.isSingleton() && !bd.isLazyInit()) { + // 处理 FactoryBean(读者如果不熟悉 FactoryBean，请移步附录区了解) + if (isFactoryBean(beanName)) { + // FactoryBean 的话，在 beanName 前面加上 ‘&’ 符号。再调用 getBean，getBean 方法别急 + final FactoryBean factory = (FactoryBean) getBean(FACTORY_BEAN_PREFIX + beanName); + // 判断当前 FactoryBean 是否是 SmartFactoryBean 的实现，此处忽略，直接跳过 + boolean isEagerInit; + if (System.getSecurityManager() != null && factory instanceof SmartFactoryBean) { + isEagerInit = AccessController.doPrivileged(new PrivilegedAction() { + @Override + public Boolean run() { + return ((SmartFactoryBean) factory).isEagerInit(); + } + }, getAccessControlContext()); + } + else { + isEagerInit = (factory instanceof SmartFactoryBean && + ((SmartFactoryBean) factory).isEagerInit()); + } + if (isEagerInit) { + + getBean(beanName); + } + } + else { + // 对于普通的 Bean，只要调用 getBean(beanName) 这个方法就可以进行初始化了 + getBean(beanName); + } + } + } + + + // 到这里说明所有的非懒加载的 singleton beans 已经完成了初始化 + // 如果我们定义的 bean 是实现了 SmartInitializingSingleton 接口的，那么在这里得到回调，忽略 + for (String beanName : beanNames) { + Object singletonInstance = getSingleton(beanName); + if (singletonInstance instanceof SmartInitializingSingleton) { + final SmartInitializingSingleton smartSingleton = (SmartInitializingSingleton) singletonInstance; + if (System.getSecurityManager() != null) { + AccessController.doPrivileged(new PrivilegedAction() { + @Override + public Object run() { + smartSingleton.afterSingletonsInstantiated(); + return null; + } + }, getAccessControlContext()); + } + else { + smartSingleton.afterSingletonsInstantiated(); + } + } + } +} +``` + +接下来，我们就进入到 getBean(beanName) 方法了，这个方法我们经常用来从 BeanFactory 中获取一个 Bean，而初始化的过程也封装到了这个方法里。 + + + +## 三、获取 Bean + +容器和 Bean 已经准备好了，接着就是获取 Bean 去使用了。 + +### 3.1 俯瞰 getBean(String) 源码 + +在本小节，我们先从战略上俯瞰 getBean(String) 方法的实现源码。代码如下： + +```java +public Object getBean(String name) throws BeansException { + // getBean 是一个空壳方法，所有的逻辑都封装在 doGetBean 方法中 + return doGetBean(name, null, null, false); +} + +protected T doGetBean( + final String name, final Class requiredType, final Object[] args, boolean typeCheckOnly) + throws BeansException { + + /* + * 通过 name 获取 beanName。这里不使用 name 直接作为 beanName 有两点原因： + * 1. name 可能会以 & 字符开头，表明调用者想获取 FactoryBean 本身，而非 FactoryBean + * 实现类所创建的 bean。在 BeanFactory 中，FactoryBean 的实现类和其他的 bean 存储 + * 方式是一致的，即 ，beanName 中是没有 & 这个字符的。所以我们需要 + * 将 name 的首字符 & 移除，这样才能从缓存里取到 FactoryBean 实例。 + * 2. 若 name 是一个别名，则应将别名转换为具体的实例名，也就是 beanName。 + */ + final String beanName = transformedBeanName(name); + // 注意跟着这个，这个是返回值 + Object bean; + + /* + * 从缓存中获取单例 bean。Spring 是使用 Map 作为 beanName 和 bean 实例的缓存的，所以这 + * 里暂时可以把 getSingleton(beanName) 等价于 beanMap.get(beanName)。当然，实际的 + * 逻辑并非如此简单，后面再细说。 + */ + Object sharedInstance = getSingleton(beanName); + + /* + * 如果 sharedInstance = null，则说明缓存里没有对应的实例，表明这个实例还没创建。 + * BeanFactory 并不会在一开始就将所有的单例 bean 实例化好，而是在调用 getBean 获取 + * bean 时再实例化，也就是懒加载。 + * getBean 方法有很多重载，比如 getBean(String name, Object... args)，我们在首次获取 + * 某个 bean 时，可以传入用于初始化 bean 的参数数组（args），BeanFactory 会根据这些参数 + * 去匹配合适的构造方法构造 bean 实例。当然，如果单例 bean 早已创建好，这里的 args 就没有 + * 用了，BeanFactory 不会多次实例化单例 bean。 + */ + if (sharedInstance != null && args == null) { + if (logger.isDebugEnabled()) { + if (isSingletonCurrentlyInCreation(beanName)) { + logger.debug("Returning eagerly cached instance of singleton bean '" + beanName + + "' that is not fully initialized yet - a consequence of a circular reference"); + } + else { + logger.debug("Returning cached instance of singleton bean '" + beanName + "'"); + } + } + + /* + * 如果 sharedInstance 是普通的单例 bean，下面的方法会直接返回。但如果 + * sharedInstance 是 FactoryBean 类型的，则需调用 getObject 工厂方法获取真正的 + * bean 实例。如果用户想获取 FactoryBean 本身，这里也不会做特别的处理，直接返回 + * 即可。毕竟 FactoryBean 的实现类本身也是一种 bean，只不过具有一点特殊的功能而已。 + */ + bean = getObjectForBeanInstance(sharedInstance, name, beanName, null); + } + /* + * 如果上面的条件不满足，则表明 sharedInstance 可能为空，此时 beanName 对应的 bean + * 实例可能还未创建。这里还存在另一种可能，如果当前容器有父容器，beanName 对应的 bean 实例 + * 可能是在父容器中被创建了，所以在创建实例前，需要先去父容器里检查一下。 + */ + else { + // BeanFactory 不缓存 Prototype 类型的 bean，无法处理该类型 bean 的循环依赖问题 + if (isPrototypeCurrentlyInCreation(beanName)) { + throw new BeanCurrentlyInCreationException(beanName); + } + + // 如果 sharedInstance = null，则到父容器中查找 bean 实例 + BeanFactory parentBeanFactory = getParentBeanFactory(); + if (parentBeanFactory != null && !containsBeanDefinition(beanName)) { + // 获取 name 对应的 beanName，如果 name 是以 & 字符开头，则返回 & + beanName + String nameToLookup = originalBeanName(name); + // 根据 args 是否为空，以决定调用父容器哪个方法获取 bean + if (args != null) { + // 返回父容器的查询结果 + return (T) parentBeanFactory.getBean(nameToLookup, args); + } + else { + return parentBeanFactory.getBean(nameToLookup, requiredType); + } + } + + if (!typeCheckOnly) { + markBeanAsCreated(beanName); + } + + /* + * 稍稍总结一下： + * 到这里的话，要准备创建 Bean 了，对于 singleton 的 Bean 来说，容器中还没创建过此 Bean； + * 对于 prototype 的 Bean 来说，本来就是要创建一个新的 Bean。 + */ + try { + // 合并父 BeanDefinition 与子 BeanDefinition，后面会单独分析这个方法 + final RootBeanDefinition mbd = getMergedLocalBeanDefinition(beanName); + checkMergedBeanDefinition(mbd, beanName, args); + + // 检查是否有 dependsOn 依赖，如果有则先初始化所依赖的 bean + String[] dependsOn = mbd.getDependsOn(); + if (dependsOn != null) { + for (String dep : dependsOn) { + /* + * 检测是否存在 depends-on 循环依赖，若存在则抛异常。比如 A 依赖 B， + * B 又依赖 A，他们的配置如下： + * + * + * + * beanA 要求 beanB 在其之前被创建，但 beanB 又要求 beanA 先于它 + * 创建。这个时候形成了循环，对于 depends-on 循环，Spring 会直接 + * 抛出异常 + */ + if (isDependent(beanName, dep)) { + throw new BeanCreationException(mbd.getResourceDescription(), beanName, + "Circular depends-on relationship between '" + beanName + "' and '" + dep + "'"); + } + // 注册依赖记录 + registerDependentBean(dep, beanName); + try { + // 先初始化被依赖项 加载 depends-on 依赖 + getBean(dep); + } + catch (NoSuchBeanDefinitionException ex) { + throw new BeanCreationException(mbd.getResourceDescription(), beanName, + "'" + beanName + "' depends on missing bean '" + dep + "'", ex); + } + } + } + + // 创建 bean 实例 + if (mbd.isSingleton()) { + /* + * 这里并没有直接调用 createBean 方法创建 bean 实例，而是通过 + * getSingleton(String, ObjectFactory) 方法获取 bean 实例。 + * getSingleton(String, ObjectFactory) 方法会在内部调用 + * ObjectFactory 的 getObject() 方法创建 bean，并会在创建完成后， + * 将 bean 放入缓存中。关于 getSingleton 方法的分析，本文先不展开，我会在 + * 后面的文章中进行分析 + */ + sharedInstance = getSingleton(beanName, new ObjectFactory() { + @Override + public Object getObject() throws BeansException { + try { + // 创建 bean 实例 + return createBean(beanName, mbd, args); + } + catch (BeansException ex) { + destroySingleton(beanName); + throw ex; + } + } + }); + // 如果 bean 是 FactoryBean 类型，则调用工厂方法获取真正的 bean 实例。否则直接返回 bean 实例 + bean = getObjectForBeanInstance(sharedInstance, name, beanName, mbd); + } + + // 创建 prototype 类型的 bean 实例 + else if (mbd.isPrototype()) { + Object prototypeInstance = null; + try { + beforePrototypeCreation(beanName); + prototypeInstance = createBean(beanName, mbd, args); + } + finally { + afterPrototypeCreation(beanName); + } + bean = getObjectForBeanInstance(prototypeInstance, name, beanName, mbd); + } + + // 创建其他类型的 bean 实例 + else { + String scopeName = mbd.getScope(); + final Scope scope = this.scopes.get(scopeName); + if (scope == null) { + throw new IllegalStateException("No Scope registered for scope name '" + scopeName + "'"); + } + try { + Object scopedInstance = scope.get(beanName, new ObjectFactory() { + @Override + public Object getObject() throws BeansException { + beforePrototypeCreation(beanName); + try { + return createBean(beanName, mbd, args); + } + finally { + afterPrototypeCreation(beanName); + } + } + }); + bean = getObjectForBeanInstance(scopedInstance, name, beanName, mbd); + } + catch (IllegalStateException ex) { + throw new BeanCreationException(beanName, + "Scope '" + scopeName + "' is not active for the current thread; consider " + + "defining a scoped proxy for this bean if you intend to refer to it from a singleton", + ex); + } + } + } + catch (BeansException ex) { + cleanupAfterBeanCreationFailure(beanName); + throw ex; + } + } + + // 如果需要进行类型转换，则在此处进行转换。类型转换这一块我没细看，就不多说了。 + if (requiredType != null && bean != null && !requiredType.isInstance(bean)) { + try { + return getTypeConverter().convertIfNecessary(bean, requiredType); + } + catch (TypeMismatchException ex) { + if (logger.isDebugEnabled()) { + logger.debug("Failed to convert bean '" + name + "' to required type '" + + ClassUtils.getQualifiedName(requiredType) + "'", ex); + } + throw new BeanNotOfRequiredTypeException(name, requiredType, bean.getClass()); + } + } + + // 返回 bean + return (T) bean; +} +``` + + + +### 3.2 createBean + +大家应该也猜到了，接下来当然是分析 createBean 方法： + +```java +protected abstract Object createBean(String beanName, RootBeanDefinition mbd, Object[] args) throws BeanCreationException; +``` + +第三个参数 args 数组代表创建实例需要的参数，不就是给构造方法用的参数，或者是工厂 Bean 的参数嘛，不过要注意，在我们的初始化阶段，args 是 null。 + +这回我们要到一个新的类了 AbstractAutowireCapableBeanFactory，看类名，AutowireCapable？类名是不是也说明了点问题了。 + +主要是为了以下场景，采用 @Autowired 注解注入属性值： + +```java +public class MessageServiceImpl implements MessageService { + @Autowired + private UserService userService; + + public String getMessage() { + return userService.getMessage(); + } +} +``` + +```xml + +``` + +以上这种属于混用了 xml 和 注解 两种方式的配置方式，Spring 会处理这种情况。 + +好了，读者要知道这么回事就可以了，继续向前。 + +// AbstractAutowireCapableBeanFactory.java 447 + +```java +/** + * Central method of this class: creates a bean instance, + * populates the bean instance, applies post-processors, etc. + * @see #doCreateBean + */ +@Override +protected Object createBean(String beanName, RootBeanDefinition mbd, Object[] args) throws BeanCreationException { + if (logger.isDebugEnabled()) { + logger.debug("Creating instance of bean '" + beanName + "'"); + } + RootBeanDefinition mbdToUse = mbd; + + // 确保 BeanDefinition 中的 Class 被加载 + Class resolvedClass = resolveBeanClass(mbd, beanName); + if (resolvedClass != null && !mbd.hasBeanClass() && mbd.getBeanClassName() != null) { + mbdToUse = new RootBeanDefinition(mbd); + mbdToUse.setBeanClass(resolvedClass); + } + + // 准备方法覆写，这里又涉及到一个概念：MethodOverrides，它来自于 bean 定义中的 + // 和 ，如果读者感兴趣，回到 bean 解析的地方看看对这两个标签的解析。 + // 我在附录中也对这两个标签的相关知识点进行了介绍，读者可以移步去看看 + try { + mbdToUse.prepareMethodOverrides(); + } + catch (BeanDefinitionValidationException ex) { + throw new BeanDefinitionStoreException(mbdToUse.getResourceDescription(), + beanName, "Validation of method overrides failed", ex); + } + + try { + // 让 InstantiationAwareBeanPostProcessor 在这一步有机会返回代理， + Object bean = resolveBeforeInstantiation(beanName, mbdToUse); + if (bean != null) { + return bean; + } + } + catch (Throwable ex) { + throw new BeanCreationException(mbdToUse.getResourceDescription(), beanName, + "BeanPostProcessor before instantiation of bean failed", ex); + } + // 重头戏，创建 bean + Object beanInstance = doCreateBean(beanName, mbdToUse, args); + if (logger.isDebugEnabled()) { + logger.debug("Finished creating instance of bean '" + beanName + "'"); + } + return beanInstance; +} +``` + +我们继续往里看 doCreateBean 这个方法： + +```java +protected Object doCreateBean(final String beanName, final RootBeanDefinition mbd, final Object[] args) + throws BeanCreationException { + + // Instantiate the bean. + BeanWrapper instanceWrapper = null; + if (mbd.isSingleton()) { + instanceWrapper = this.factoryBeanInstanceCache.remove(beanName); + } + if (instanceWrapper == null) { + // 说明不是 FactoryBean，这里实例化 Bean，这里非常关键，细节之后再说 + instanceWrapper = createBeanInstance(beanName, mbd, args); + } + // 这个就是 Bean 里面的 我们定义的类 的实例，很多地方我直接描述成 "bean 实例" + final Object bean = (instanceWrapper != null ? instanceWrapper.getWrappedInstance() : null); + // 类型 + Class beanType = (instanceWrapper != null ? instanceWrapper.getWrappedClass() : null); + mbd.resolvedTargetType = beanType; + + // 建议跳过吧，涉及接口：MergedBeanDefinitionPostProcessor + synchronized (mbd.postProcessingLock) { + if (!mbd.postProcessed) { + try { + // MergedBeanDefinitionPostProcessor，这个我真不展开说了，直接跳过吧，很少用的 + applyMergedBeanDefinitionPostProcessors(mbd, beanType, beanName); + } + catch (Throwable ex) { + throw new BeanCreationException(mbd.getResourceDescription(), beanName, + "Post-processing of merged bean definition failed", ex); + } + mbd.postProcessed = true; + } + } + + // Eagerly cache singletons to be able to resolve circular references + // even when triggered by lifecycle interfaces like BeanFactoryAware. + // 下面这块代码是为了解决循环依赖的问题 + boolean earlySingletonExposure = (mbd.isSingleton() && this.allowCircularReferences && + isSingletonCurrentlyInCreation(beanName)); + if (earlySingletonExposure) { + if (logger.isDebugEnabled()) { + logger.debug("Eagerly caching bean '" + beanName + + "' to allow for resolving potential circular references"); + } + addSingletonFactory(beanName, new ObjectFactory() { + @Override + public Object getObject() throws BeansException { + return getEarlyBeanReference(beanName, mbd, bean); + } + }); + } + + // Initialize the bean instance. + Object exposedObject = bean; + try { + // 这一步也是非常关键的，这一步负责属性装配，因为前面的实例只是实例化了，并没有设值，这里就是设值 + populateBean(beanName, mbd, instanceWrapper); + if (exposedObject != null) { + // 还记得 init-method 吗？还有 InitializingBean 接口？还有 BeanPostProcessor 接口？ + // 这里就是处理 bean 初始化完成后的各种回调 + exposedObject = initializeBean(beanName, exposedObject, mbd); + } + } + catch (Throwable ex) { + if (ex instanceof BeanCreationException && beanName.equals(((BeanCreationException) ex).getBeanName())) { + throw (BeanCreationException) ex; + } + else { + throw new BeanCreationException( + mbd.getResourceDescription(), beanName, "Initialization of bean failed", ex); + } + } + + if (earlySingletonExposure) { + // + Object earlySingletonReference = getSingleton(beanName, false); + if (earlySingletonReference != null) { + if (exposedObject == bean) { + exposedObject = earlySingletonReference; + } + else if (!this.allowRawInjectionDespiteWrapping && hasDependentBean(beanName)) { + String[] dependentBeans = getDependentBeans(beanName); + Set actualDependentBeans = new LinkedHashSet(dependentBeans.length); + for (String dependentBean : dependentBeans) { + if (!removeSingletonIfCreatedForTypeCheckOnly(dependentBean)) { + actualDependentBeans.add(dependentBean); + } + } + if (!actualDependentBeans.isEmpty()) { + throw new BeanCurrentlyInCreationException(beanName, + "Bean with name '" + beanName + "' has been injected into other beans [" + + StringUtils.collectionToCommaDelimitedString(actualDependentBeans) + + "] in its raw version as part of a circular reference, but has eventually been " + + "wrapped. This means that said other beans do not use the final version of the " + + "bean. This is often the result of over-eager type matching - consider using " + + "'getBeanNamesOfType' with the 'allowEagerInit' flag turned off, for example."); + } + } + } + } + + // Register bean as disposable. + try { + registerDisposableBeanIfNecessary(beanName, bean, mbd); + } + catch (BeanDefinitionValidationException ex) { + throw new BeanCreationException( + mbd.getResourceDescription(), beanName, "Invalid destruction signature", ex); + } + + return exposedObject; +} +``` + +到这里，我们已经分析完了 doCreateBean 方法，总的来说，我们已经说完了整个初始化流程。 + +接下来我们挑 doCreateBean 中的三个细节出来说说。 + +一个是创建 Bean 实例的 createBeanInstance 方法，一个是依赖注入的 populateBean 方法，还有就是回调方法 initializeBean。 + +注意了，接下来的这三个方法要认真说那也是极其复杂的，很多地方我就点到为止了，感兴趣的读者可以自己往里看，最好就是碰到不懂的，自己写代码去调试它。 + +#### 创建 Bean 实例 + +我们先看看 createBeanInstance 方法。需要说明的是，这个方法如果每个分支都分析下去，必然也是极其复杂冗长的，我们挑重点说。此方法的目的就是实例化我们指定的类。 + +```java +protected BeanWrapper createBeanInstance(String beanName, RootBeanDefinition mbd, Object[] args) { + // 确保已经加载了此 class + Class beanClass = resolveBeanClass(mbd, beanName); + + // 校验一下这个类的访问权限 + if (beanClass != null && !Modifier.isPublic(beanClass.getModifiers()) && !mbd.isNonPublicAccessAllowed()) { + throw new BeanCreationException(mbd.getResourceDescription(), beanName, + "Bean class isn't public, and non-public access not allowed: " + beanClass.getName()); + } + + if (mbd.getFactoryMethodName() != null) { + // 采用工厂方法实例化，不熟悉这个概念的读者请看附录，注意，不是 FactoryBean + return instantiateUsingFactoryMethod(beanName, mbd, args); + } + + // 如果不是第一次创建，比如第二次创建 prototype bean。 + // 这种情况下，我们可以从第一次创建知道，采用无参构造函数，还是构造函数依赖注入 来完成实例化 + boolean resolved = false; + boolean autowireNecessary = false; + if (args == null) { + synchronized (mbd.constructorArgumentLock) { + if (mbd.resolvedConstructorOrFactoryMethod != null) { + resolved = true; + autowireNecessary = mbd.constructorArgumentsResolved; + } + } + } + if (resolved) { + if (autowireNecessary) { + // 构造函数依赖注入 + return autowireConstructor(beanName, mbd, null, null); + } + else { + // 无参构造函数 + return instantiateBean(beanName, mbd); + } + } + + // 判断是否采用有参构造函数 + Constructor[] ctors = determineConstructorsFromBeanPostProcessors(beanClass, beanName); + if (ctors != null || + mbd.getResolvedAutowireMode() == RootBeanDefinition.AUTOWIRE_CONSTRUCTOR || + mbd.hasConstructorArgumentValues() || !ObjectUtils.isEmpty(args)) { + // 构造函数依赖注入 + return autowireConstructor(beanName, mbd, ctors, args); + } + + // 调用无参构造函数 + return instantiateBean(beanName, mbd); +} +``` + +挑个简单的**无参构造函数**构造实例来看看： + +```java +protected BeanWrapper instantiateBean(final String beanName, final RootBeanDefinition mbd) { + try { + Object beanInstance; + final BeanFactory parent = this; + if (System.getSecurityManager() != null) { + beanInstance = AccessController.doPrivileged(new PrivilegedAction() { + @Override + public Object run() { + + return getInstantiationStrategy().instantiate(mbd, beanName, parent); + } + }, getAccessControlContext()); + } + else { + // 实例化 + beanInstance = getInstantiationStrategy().instantiate(mbd, beanName, parent); + } + // 包装一下，返回 + BeanWrapper bw = new BeanWrapperImpl(beanInstance); + initBeanWrapper(bw); + return bw; + } + catch (Throwable ex) { + throw new BeanCreationException( + mbd.getResourceDescription(), beanName, "Instantiation of bean failed", ex); + } +} +``` + +我们可以看到，关键的地方在于： + +```java +beanInstance = getInstantiationStrategy().instantiate(mbd, beanName, parent); +``` + +这里会进行实际的实例化过程，我们进去看看: + +// SimpleInstantiationStrategy 59 + +```java +@Override +public Object instantiate(RootBeanDefinition bd, String beanName, BeanFactory owner) { + + // 如果不存在方法覆写，那就使用 java 反射进行实例化，否则使用 CGLIB, + // 方法覆写 请参见附录"方法注入"中对 lookup-method 和 replaced-method 的介绍 + if (bd.getMethodOverrides().isEmpty()) { + Constructor constructorToUse; + synchronized (bd.constructorArgumentLock) { + constructorToUse = (Constructor) bd.resolvedConstructorOrFactoryMethod; + if (constructorToUse == null) { + final Class clazz = bd.getBeanClass(); + if (clazz.isInterface()) { + throw new BeanInstantiationException(clazz, "Specified class is an interface"); + } + try { + if (System.getSecurityManager() != null) { + constructorToUse = AccessController.doPrivileged(new PrivilegedExceptionAction>() { + @Override + public Constructor run() throws Exception { + return clazz.getDeclaredConstructor((Class[]) null); + } + }); + } + else { + constructorToUse = clazz.getDeclaredConstructor((Class[]) null); + } + bd.resolvedConstructorOrFactoryMethod = constructorToUse; + } + catch (Throwable ex) { + throw new BeanInstantiationException(clazz, "No default constructor found", ex); + } + } + } + // 利用构造方法进行实例化 + return BeanUtils.instantiateClass(constructorToUse); + } + else { + // 存在方法覆写，利用 CGLIB 来完成实例化，需要依赖于 CGLIB 生成子类，这里就不展开了。 + // tips: 因为如果不使用 CGLIB 的话，存在 override 的情况 JDK 并没有提供相应的实例化支持 + return instantiateWithMethodInjection(bd, beanName, owner); + } +} +``` + +到这里，我们就算实例化完成了。我们开始说怎么进行属性注入。 + +#### bean 属性注入 + +看完了 createBeanInstance(...) 方法，我们来看看 populateBean(...) 方法，该方法负责进行属性设值，处理依赖。 + +// AbstractAutowireCapableBeanFactory 1203 + +```java +protected void populateBean(String beanName, RootBeanDefinition mbd, BeanWrapper bw) { + // bean 实例的所有属性都在这里了 + PropertyValues pvs = mbd.getPropertyValues(); + + if (bw == null) { + if (!pvs.isEmpty()) { + throw new BeanCreationException( + mbd.getResourceDescription(), beanName, "Cannot apply property values to null instance"); + } + else { + // Skip property population phase for null instance. + return; + } + } + + // 到这步的时候，bean 实例化完成（通过工厂方法或构造方法），但是还没开始属性设值， + // InstantiationAwareBeanPostProcessor 的实现类可以在这里对 bean 进行状态修改， + // 我也没找到有实际的使用，所以我们暂且忽略这块吧 + boolean continueWithPropertyPopulation = true; + if (!mbd.isSynthetic() && hasInstantiationAwareBeanPostProcessors()) { + for (BeanPostProcessor bp : getBeanPostProcessors()) { + if (bp instanceof InstantiationAwareBeanPostProcessor) { + InstantiationAwareBeanPostProcessor ibp = (InstantiationAwareBeanPostProcessor) bp; + // 如果返回 false，代表不需要进行后续的属性设值，也不需要再经过其他的 BeanPostProcessor 的处理 + if (!ibp.postProcessAfterInstantiation(bw.getWrappedInstance(), beanName)) { + continueWithPropertyPopulation = false; + break; + } + } + } + } + + if (!continueWithPropertyPopulation) { + return; + } + + if (mbd.getResolvedAutowireMode() == RootBeanDefinition.AUTOWIRE_BY_NAME || + mbd.getResolvedAutowireMode() == RootBeanDefinition.AUTOWIRE_BY_TYPE) { + MutablePropertyValues newPvs = new MutablePropertyValues(pvs); + + // 通过名字找到所有属性值，如果是 bean 依赖，先初始化依赖的 bean。记录依赖关系 + if (mbd.getResolvedAutowireMode() == RootBeanDefinition.AUTOWIRE_BY_NAME) { + autowireByName(beanName, mbd, bw, newPvs); + } + + // 通过类型装配。复杂一些 + if (mbd.getResolvedAutowireMode() == RootBeanDefinition.AUTOWIRE_BY_TYPE) { + autowireByType(beanName, mbd, bw, newPvs); + } + + pvs = newPvs; + } + + boolean hasInstAwareBpps = hasInstantiationAwareBeanPostProcessors(); + boolean needsDepCheck = (mbd.getDependencyCheck() != RootBeanDefinition.DEPENDENCY_CHECK_NONE); + + if (hasInstAwareBpps || needsDepCheck) { + PropertyDescriptor[] filteredPds = filterPropertyDescriptorsForDependencyCheck(bw, mbd.allowCaching); + if (hasInstAwareBpps) { + for (BeanPostProcessor bp : getBeanPostProcessors()) { + if (bp instanceof InstantiationAwareBeanPostProcessor) { + InstantiationAwareBeanPostProcessor ibp = (InstantiationAwareBeanPostProcessor) bp; + // 这里有个非常有用的 BeanPostProcessor 进到这里: AutowiredAnnotationBeanPostProcessor + // 对采用 @Autowired、@Value 注解的依赖进行设值，这里的内容也是非常丰富的，不过本文不会展开说了，感兴趣的读者请自行研究 + pvs = ibp.postProcessPropertyValues(pvs, filteredPds, bw.getWrappedInstance(), beanName); + if (pvs == null) { + return; + } + } + } + } + if (needsDepCheck) { + checkDependencies(beanName, mbd, filteredPds, pvs); + } + } + // 设置 bean 实例的属性值 + applyPropertyValues(beanName, mbd, bw, pvs); +} +``` + + + +#### initializeBean + +属性注入完成后，这一步其实就是处理各种回调了，这块代码比较简单。 + +```java +protected Object initializeBean(final String beanName, final Object bean, RootBeanDefinition mbd) { + if (System.getSecurityManager() != null) { + AccessController.doPrivileged(new PrivilegedAction() { + @Override + public Object run() { + invokeAwareMethods(beanName, bean); + return null; + } + }, getAccessControlContext()); + } + else { + // 如果 bean 实现了 BeanNameAware、BeanClassLoaderAware 或 BeanFactoryAware 接口，回调 + invokeAwareMethods(beanName, bean); + } + + Object wrappedBean = bean; + if (mbd == null || !mbd.isSynthetic()) { + // BeanPostProcessor 的 postProcessBeforeInitialization 回调 + wrappedBean = applyBeanPostProcessorsBeforeInitialization(wrappedBean, beanName); + } + + try { + // 处理 bean 中定义的 init-method， + // 或者如果 bean 实现了 InitializingBean 接口，调用 afterPropertiesSet() 方法 + invokeInitMethods(beanName, wrappedBean, mbd); + } + catch (Throwable ex) { + throw new BeanCreationException( + (mbd != null ? mbd.getResourceDescription() : null), + beanName, "Invocation of init method failed", ex); + } + + if (mbd == null || !mbd.isSynthetic()) { + // BeanPostProcessor 的 postProcessAfterInitialization 回调 + wrappedBean = applyBeanPostProcessorsAfterInitialization(wrappedBean, beanName); + } + return wrappedBean; +} +``` + + + +大家发现没有，BeanPostProcessor 的两个回调都发生在这边，只不过中间处理了 init-method，是不是和读者原来的认知有点不一样了？ + + + +## 四、最后 + +#### Spring 的 bean 在什么时候实例化? + +如果你使用BeanFactory，如 XmlBeanFactory 作为 Spring Bean 的工厂类，则所有的 bean 都是在第一次使用该bean 的时候实例化 。 + +如果你使用 ApplicationContext 作为 Spring Bean 的工厂类，则又分为以下几种情况： + +1. 如果 bean 的 scope 是 singleton 的，并且 lazy-init 为 false（默认是false，所以可以不用设置），则ApplicationContext 启动的时候就实例化该 bean，并且将实例化的 bean 放在一个线程安全的 ConcurrentHashMap 结构的缓存中，下次再使用该 Bean 的时候，直接从这个缓存中取 +2. 如果 bean 的 scope 是 singleton 的，并且 lazy-init 为 true，则该 bean 的实例化是在第一次使用该 bean 的时候进行实例化 +3. 如果 bean 的 scope 是 prototype 的，则该 bean 的实例化是在第一次使用该 bean 的时候进行实例化 。 + + + +参考与来源： + +https://www.zhihu.com/question/21346206/answer/366816411 + +Spring IOC 容器源码分析 https://juejin.im/post/6844903694039793672#heading-7 \ No newline at end of file diff --git a/docs/framework/spring/Spring-IOC.md b/docs/framework/Spring/Spring-IOC.md similarity index 96% rename from docs/framework/spring/Spring-IOC.md rename to docs/framework/Spring/Spring-IOC.md index 40f67c2d88..d7225a0cc9 100644 --- a/docs/framework/spring/Spring-IOC.md +++ b/docs/framework/Spring/Spring-IOC.md @@ -1,26 +1,5 @@ # Spring IoC - - -- [IoC 概念](#ioc-概念) - - [IoC 是什么](#ioc-是什么) - - [IoC 能做什么](#ioc-能做什么) - - [依赖注入](#依赖注入) - - [IoC 和 DI](#ioc-和-di) - - [IoC 容器](#ioc-容器) - - [Bean](#bean) -- [IoC 容器](#ioc-容器-1) - - [核心接口](#核心接口) - - [IoC 容器工作步骤](#ioc-容器工作步骤) - - [Bean 概述](#bean-概述) - - [依赖](#依赖) -- [IoC 容器配置](#ioc-容器配置) - - [Xml 配置](#xml-配置) - - [注解配置](#注解配置) - - [Java 配置](#java-配置) - - - ## IoC 概念 ### IoC 是什么 @@ -34,12 +13,14 @@ IoC 不是什么技术，而是一种设计思想。在 Java 开发中，IoC 意 用图例说明一下，传统程序设计如图 2-1，都是主动去创建相关对象然后再组合起来： -
+![](http://sishuok.com/forum/upload/2012/2/19/a02c1e3154ef4be3f15fb91275a26494__1.JPG) + 图 2-1 传统应用程序示意图 当有了 IoC/DI 的容器后，在客户端类中不再主动去创建这些对象了，如图 2-2 所示: -
+ + 图 2-2 有 IoC/DI 容器后程序结构示意图 ### IoC 能做什么 @@ -658,7 +639,7 @@ Java 配置 为了让 Spring 识别这个定义类为一个 Spring 配置类，需要用到两个注解：@Configuration 和@Bean。 -如果你熟悉 Spring 的 xml 配置方式，你可以将@Configuration 等价于标签；将@Bean 等价于标签。 +如果你熟悉 Spring 的 xml 配置方式，你可以将@Configuration 等价于 \标签；将@Bean 等价于\标签。 @Bean @@ -798,7 +779,7 @@ public class Police implements Job { @Bean 注解用来表明一个方法实例化、配置合初始化一个被 Spring IoC 容器管理的新对象。 -如果你熟悉 Spring 的 xml 配置，你可以将@Bean 视为等价于``标签。 +如果你熟悉 Spring 的 xml 配置，你可以将@Bean 视为等价于`\`标签。 @Bean 注解可以用于任何的 Spring `@Component` bean，然而，通常被用于`@Configuration` bean。 diff --git a/docs/framework/Spring/Spring-MVC.md b/docs/framework/Spring/Spring-MVC.md new file mode 100644 index 0000000000..e856ada2e7 --- /dev/null +++ b/docs/framework/Spring/Spring-MVC.md @@ -0,0 +1 @@ +Spring 的模型-视图-控制器（MVC）框架是围绕一个 DispatcherServlet 来设计的，这个 Servlet 会把请求分发给各个处理器，并支持可配置的处理器映射、视图渲染、本地化、时区与主题渲染 等，甚至还能支持文件上传。 \ No newline at end of file diff --git a/docs/framework/spring/Spring-Overview.md b/docs/framework/Spring/Spring-Overview.md similarity index 98% rename from docs/framework/spring/Spring-Overview.md rename to docs/framework/Spring/Spring-Overview.md index ace867a9b6..c4a3866ee3 100644 --- a/docs/framework/spring/Spring-Overview.md +++ b/docs/framework/Spring/Spring-Overview.md @@ -29,7 +29,7 @@ Spring的核心是**控制反转**（IOC）和**面向切面**（AOP） ## 3. Spring 模块 -![](../_images/Spring/spring-overview.png) +![](../../_images/Spring/spring-overview.png) Spring 框架是一个分层架构，由 7 个定义良好的模块组成： @@ -85,7 +85,7 @@ Web层由`spring-web`、`spring-webmvc`、`spring-websocket`和`spring-webmvc-po 《Spring 揭秘》中的Spring框架总体结构 -![](../_images/Spring/spring-tree.png) +![](../../_images/Spring/spring-tree.png) diff --git a/docs/framework/Spring/Spring-read-source.md b/docs/framework/Spring/Spring-read-source.md new file mode 100644 index 0000000000..3fd875a536 --- /dev/null +++ b/docs/framework/Spring/Spring-read-source.md @@ -0,0 +1,75 @@ +## 为什么看 + +很多 Javaer 有过面试中被问到 “看过 Spring 源码没？” + +平常的工作中，大多数开发其实都属于 CRUD 工程师，业务领域达到一定地步后，自己就会觉得重复的业务代码让我们不断的粘贴、复制和修改，日复一日，又担心自己变成一个业务代码生产机器，而无法面对新技术和环境变化。 + +所以，有时间的话，可以搭建一个 Spring 源码环境，想深入了解哪一块，就去自己实现，代码步入到相关的模块，为什么不是直接用 IDEA 的 jar 包编译方式呢，因为那玩意只能看，不能主动去改源码。 + +来吧，进去 Spring 看看 Java 界的扛把子们是怎么写代码的吧。肯定会对你日后写代码有帮助的。 + +- 良好的代码风格，也能看看 Spring 是怎么对方法，类，各种结构的明明方式和写法，别每次写个方法时候就会getXXX， +- 编码设计方式 +- +- 各种设计模式的运用 + +Spring 是一个很大的生态，我们不可能去花大把的时间，把每个细节都去掌握，所以，看源码之前，我们需要要先知道 Spring 的思想，然后再从源码中看实现，有精力的可以自己造个小轮子。 + + + +提升编码质量和水准 + +借鉴系统开发和设计思想 + +代码重构提供参考依据 + +面试 + +## 从哪里开始看 + +想读 Spring 源码，但是又不知道从哪里开始读，是先看 IOC 还是先看 AOP 呢，迷惑 + +Spring 涵盖的东西太多了，如果不能进行针对性的阅读，很容易迷失。 + +程序员学语言，学框架都是从 Hello World 开始的，看源码为什么不也从 Hello World 开始呢？ + +```java +ApplicationContext context = new ClassPathXmlApplicationContext("applicationContext.xml"); +Object bean = context.getBean("hello"); +``` + +运行完这段代码后，我们产生这么几个疑问： + +- ApplicationContext 是个容器，怎么创建的，创建时候干了什么？ +- 怎么就获取到了配置文件 `applicationContext.xml` +- getBean() 怎么就调用了实例 + + + +## 如何看 + +最开始用最新版的代码，各种报错，“一气之下”，改为下载了 v5.0.16 版本的，竟然没有任何错误，开搞。源码中的自述文件，其实有教程，`import-into-idea.md` + +步骤： + +1. 安装并配置 Gradle +2. 下载 Spring 源码 +3. 编译 Spring，进入 `spring-framework` 文件夹下，打开 cmd，输入 `gradlew :spring-oxm:compileTestJava` 进行编译 +4. IDEA导入源码 +5. 打开IDEA，File->New->Project From Existing Sources…，选中 `spring-framework` 源码文件夹，点击OK，选择 Import project from external model，选中 Gradle，点击 Next（各个版本不一样，我直接 Finsh） +6. 新建测试 Module + +网上教程很多，推荐两个： + +https://blog.csdn.net/bskfnvjtlyzmv867/article/details/81171802 + +https://www.cnblogs.com/zhangfengxian/p/11072500.html + + + + + + + + + diff --git a/docs/framework/springboot/@Scheduled.md b/docs/framework/SpringBoot/@Scheduled.md similarity index 100% rename from docs/framework/springboot/@Scheduled.md rename to docs/framework/SpringBoot/@Scheduled.md diff --git a/docs/framework/springboot/Hello-SpringBoot.md b/docs/framework/SpringBoot/Hello-SpringBoot.md similarity index 99% rename from docs/framework/springboot/Hello-SpringBoot.md rename to docs/framework/SpringBoot/Hello-SpringBoot.md index 514c23cd39..898a68675f 100644 --- a/docs/framework/springboot/Hello-SpringBoot.md +++ b/docs/framework/SpringBoot/Hello-SpringBoot.md @@ -1,12 +1,11 @@ # Spring Boot从初识到实战 -![微信图片_20191219180720.png](http://ww1.sinaimg.cn/large/9b9f09a9ly1ga26zho09tj20p00an0yr.jpg) +![](http://ww1.sinaimg.cn/large/9b9f09a9ly1ga26zho09tj20p00an0yr.jpg) -> 文章收集在 GitHub [JavaEgg](https://github.com/Jstarfish/JavaEgg) 中，欢迎star+指导 +> 文章收集在 GitHub [JavaKeeper](https://github.com/Jstarfish/JavaKeeper) 中，欢迎star+指导 > -> [JavaEgg](https://github.com/Jstarfish/JavaEgg)——《“Java技术员”成长手册》，包含Java基础、框架、存储、搜索、优化、分布式等必备知识，N线互联网开发必备技能兵器谱。 # 一、Hello Spring Boot diff --git "a/docs/framework/SpringBoot/Spring Boot \346\234\200\346\265\201\350\241\214\347\232\204 16 \346\235\241\345\256\236\350\267\265\350\247\243\350\257\273.md" "b/docs/framework/SpringBoot/Spring Boot \346\234\200\346\265\201\350\241\214\347\232\204 16 \346\235\241\345\256\236\350\267\265\350\247\243\350\257\273.md" new file mode 100644 index 0000000000..561f739d1c --- /dev/null +++ "b/docs/framework/SpringBoot/Spring Boot \346\234\200\346\265\201\350\241\214\347\232\204 16 \346\235\241\345\256\236\350\267\265\350\247\243\350\257\273.md" @@ -0,0 +1,211 @@ +SpringBoot作为最受欢迎的框架，你对它了解多少？如何写出更加优雅的代码，看看这篇文章！ + +![](https://tva1.sinaimg.cn/large/007S8ZIlly1ggsh81mzpkj31ph0u0kjo.jpg) + +------ + +> 作者：Jedrzejewski +> +> 原文：www.e4developer.com/2018/08/06/ + +Spring Boot 是最流行的用于开发微服务的 Java 框架。在本文中，我将与你分享自 2016 年以来我在专业开发中使用 Spring Boot 所采用的最佳实践。这些内容是基于我的个人经验和一些熟知的 Spring Boot 专家的文章。 + +在本文中，我将重点介绍 Spring Boot 特有的实践（大多数时候，也适用于 Spring 项目）。以下依次列出了最佳实践，排名不分先后。 + +### 1、使用自定义 BOM 来维护第三方依赖 + +这条实践是我根据实际项目中的经历总结出的。 + +Spring Boot 项目本身使用和集成了大量的开源项目，它帮助我们维护了这些第三方依赖。但是也有一部分在实际项目使用中并没有包括进来，这就需要我们在项目中自己维护版本。如果在一个大型的项目中，包括了很多未开发模块，那么维护起来就非常的繁琐。 + +怎么办呢？事实上，Spring IO Platform 就是做的这个事情，它本身就是 Spring Boot 的子项目，同时维护了其他第三方开源库。我们可以借鉴 Spring IO Platform 来编写自己的基础项目 platform-bom，所有的业务模块项目应该以 BOM 的方式引入。这样在升级第三方依赖时，就只需要升级这一个依赖的版本而已。 + +```xml + + + + io.spring.platform + platform-bom + Cairo-SR3 + pom + import + + + +``` + +### 2、使用自动配置 + +Spring Boot 的一个主要特性是使用自动配置。这是 Spring Boot 的一部分，它可以简化你的代码并使之工作。当在类路径上检测到特定的 jar 文件时，自动配置就会被激活。 + +使用它的最简单方法是依赖 Spring Boot Starters。因此，如果你想与 Redis 进行集成，你可以首先包括： + +```xml + + org.springframework.boot + spring-boot-starter-data-redis + +``` + +如果你想与 MongoDB 进行集成，需要这样： + +```xml + + org.springframework.boot + spring-boot-starter-data-mongodb + +``` + +借助于这些 starters，这些繁琐的配置就可以很好地集成起来并协同工作，而且它们都是经过测试和验证的。这非常有助于避免可怕的 Jar 地狱。 + +> https://dzone.com/articles/what-is-jar-hell + +通过使用以下注解属性，可以从自动配置中排除某些配置类： + +```java +@EnableAutoConfiguration（exclude = {ClassNotToAutoconfigure.class}） +``` + +但只有在绝对必要时才应该这样做。 + +有关自动配置的官方文档可在此处找到： + +> https://docs.spring.io/spring-boot/docs/current/reference/html/using-boot-auto-configuration.html。 + +### 3、使用 Spring Initializr 来开始一个新的 Spring Boot 项目 + +> 这一条最佳实践来自 Josh Long （Spring Advocate，@starbuxman）。 + +Spring Initializr 提供了一个超级简单的方法来创建一个新的 Spring Boot 项目，并根据你的需要来加载可能使用到的依赖。 + +> https://start.spring.io/ + +使用 Initializr 创建应用程序可确保你获得经过测试和验证的依赖项，这些依赖项适用于 Spring 自动配置。你甚至可能会发现一些新的集成，但你可能并没有意识到这些。 + +### 4、考虑为常见的组织问题创建自己的自动配置 + +这一条也来自 Josh Long（Spring Advocate，@starbuxman）——这个实践是针对高级用户的。 + +如果你在一个严重依赖 Spring Boot 的公司或团队中工作，并且有共同的问题需要解决，那么你可以创建自己的自动配置。 + +这项任务涉及较多工作，因此你需要考虑何时获益是值得投入的。与多个略有不同的定制配置相比，维护单个自动配置更容易。 + +如果将这个提供 Spring Boot 配置以开源库的形式发布出去，那么将极大地简化数千个用户的配置工作。 + +### 5、正确设计代码目录结构 + +尽管允许你有很大的自由，但是有一些基本规则值得遵守来设计你的源代码结构。 + +避免使用默认包。确保所有内容（包括你的入口点）都位于一个名称很好的包中，这样就可以避免与装配和组件扫描相关的意外情况； + +将 Application.java（应用的入口类）保留在顶级源代码目录中； + +我建议将控制器和服务放在以功能为导向的模块中，但这是可选的。一些非常好的开发人员建议将所有控制器放在一起。不论怎样，坚持一种风格！ + +### 6、保持 @Controller 的简洁和专注 + +Controller 应该非常简单。你可以在此处阅读有关 GRASP 中有关控制器模式部分的说明。你希望控制器作为协调和委派的角色，而不是执行实际的业务逻辑。以下是主要做法： + +> https://en.wikipedia.org/wiki/GRASP_(object-oriented_design)#Controller + +- 控制器应该是无状态的！默认情况下，控制器是单例，并且任何状态都可能导致大量问题； +- 控制器不应该执行业务逻辑，而是依赖委托； +- 控制器应该处理应用程序的 HTTP 层，这不应该传递给服务； +- 控制器应该围绕用例 / 业务能力来设计。 + +要深入这个内容，需要进一步地了解设计 REST API 的最佳实践。无论你是否想要使用 Spring Boot，都是值得学习的。 + +### 7、围绕业务功能构建 @Service + +Service 是 Spring Boot 的另一个核心概念。我发现最好围绕业务功能 / 领域 / 用例（无论你怎么称呼都行）来构建服务。 + +在应用中设计名称类似`AccountService`, `UserService`, `PaymentService`这样的服务，比起像`DatabaseService`、`ValidationService`、`CalculationService`这样的会更合适一些。 + +你可以决定使用 Controler 和 Service 之间的一对一映射，那将是理想的情况。但这并不意味着，Service 之间不能互相调用！ + +### 8、使数据库独立于核心业务逻辑之外 + +我之前还不确定如何在 Spring Boot 中最好地处理数据库交互。在阅读了罗伯特 ·C· 马丁的 “Clear Architecture” 之后，对我来说就清晰多了。 + +你希望你的数据库逻辑于服务分离出来。理想情况下，你不希望服务知道它正在与哪个数据库通信，这需要一些抽象来封装对象的持久性。 + +> 罗伯特 C. 马丁强烈地说明，你的数据库是一个 “细节”，这意味着不将你的应用程序与特定数据库耦合。过去很少有人会切换数据库，我注意到，使用 Spring Boot 和现代微服务开发会让事情变得更快。 + +### 9、保持业务逻辑不受 Spring Boot 代码的影响 + +考虑到 “Clear Architecture” 的教训，你还应该保护你的业务逻辑。将各种 Spring Boot 代码混合在一起是非常诱人的…… 不要这样做。如果你能抵制诱惑，你将保持你的业务逻辑可重用。 + +部分服务通常成为库。如果不从代码中删除大量 Spring 注解，则更容易创建。 + +### 10、推荐使用构造函数注入 + +这一条实践来自 Phil Webb（Spring Boot 的项目负责人, @phillip_webb）。 + +保持业务逻辑免受 Spring Boot 代码侵入的一种方法是使用构造函数注入。不仅是因为`@Autowired`注解在构造函数上是可选的，而且还可以在没有 Spring 的情况下轻松实例化 bean。 + +### 11、熟悉并发模型 + +我写过的最受欢迎的文章之一是 “介绍 Spring Boot 中的并发”。我认为这样做的原因是这个领域经常被误解和忽视。如果使用不当，就会出现问题。 + +> https://www.e4developer.com/2018/03/30/introduction-to-concurrency-in-spring-boot/ + +在 Spring Boot 中，Controller 和 Service 是默认是单例。如果你不小心，这会引入可能的并发问题。你通常也在处理有限的线程池。请熟悉这些概念。 + +如果你正在使用新的 WebFlux 风格的 Spring Boot 应用程序，我已经解释了它在 “Spring’s WebFlux/Reactor Parallelism and Backpressure” 中是如何工作的。 + +### 12、加强配置管理的外部化 + +这一点超出了 Spring Boot，虽然这是人们开始创建多个类似服务时常见的问题…… + +你可以手动处理 Spring 应用程序的配置。如果你正在处理多个 Spring Boot 应用程序，则需要使配置管理能力更加强大。 + +我推荐两种主要方法： + +- 使用配置服务器，例如 Spring Cloud Config； +- 将所有配置存储在环境变量中（可以基于 git 仓库进行配置）。 + +这些选项中的任何一个（第二个选项多一些）都要求你在 DevOps 更少工作量，但这在微服务领域是很常见的。 + +### 13、提供全局异常处理 + +你真的需要一种处理异常的一致方法。Spring Boot 提供了两种主要方法： + +- 你应该使用 HandlerExceptionResolver 定义全局异常处理策略； +- 你也可以在控制器上添加 @ExceptionHandler 注解，这在某些特定场景下使用可能会很有用。 + +这与 Spring 中的几乎相同，并且 Baeldung 有一篇关于 REST 与 Spring 的错误处理的详细文章，非常值得一读。 + +> https://www.baeldung.com/exception-handling-for-rest-with-spring + +### 14、使用日志框架 + +你可能已经意识到这一点，但你应该使用 Logger 进行日志记录，而不是使用 System.out.println() 手动执行。这很容易在 Spring Boot 中完成，几乎没有配置。只需获取该类的记录器实例： + +```java +Logger logger = LoggerFactory.getLogger(MyClass.class); +``` + +这很重要，因为它可以让你根据需要设置不同的日志记录级别。 + +### 15、测试你的代码 + +这不是 Spring Boot 特有的，但它需要提醒——测试你的代码！如果你没有编写测试，那么你将从一开始就编写遗留代码。 + +如果有其他人使用你的代码库，那边改变任何东西将会变得危险。当你有多个服务相互依赖时，这甚至可能更具风险。 + +由于存在 Spring Boot 最佳实践，因此你应该考虑将 Spring Cloud Contract 用于你的消费者驱动契约，它将使你与其他服务的集成更容易使用。 + +### 16、使用测试切片让测试更容易，并且更专注 + +这一条实践来自 Madhura Bhave（Spring 开发者, @madhurabhave23）。 + +使用 Spring Boot 测试代码可能很棘手——你需要初始化数据层，连接大量服务，模拟事物…… 实际上并不是那么难！答案是使用测试切片。 + +使用测试切片，你可以根据需要仅连接部分应用程序。这可以为你节省大量时间，并确保你的测试不会与未使用的内容相关联。来自 spring.io 的一篇名为 Custom test slice with Spring test 1.4 的博客文章解释了这种技术。 + +> https://spring.io/blog/2016/08/30/custom-test-slice-with-spring-boot-1-4 + +### 总结 + +感谢 Spring Boot，编写基于 Spring 的微服务正变得前所未有的简单。我希望通过这些最佳实践，你的实施过程不仅会变得很快，而且从长远来看也会更加强大和成功。祝你好运！ + diff --git a/docs/framework/SpringCloud/sidebar.md b/docs/framework/SpringCloud/sidebar.md deleted file mode 100644 index 13df701dc4..0000000000 --- a/docs/framework/SpringCloud/sidebar.md +++ /dev/null @@ -1,51 +0,0 @@ -- **Java基础** -- [![](https://icongr.am/simple/oracle.svg?size=25&color=231c82&colored=false)JVM](java/JVM/readJVM.md) -- [![img](https://icongr.am/fontawesome/expeditedssl.svg?size=25&color=f23131)JUC](java/JUC/readJUC.md) -- [![](https://icongr.am/devicon/java-original.svg?size=25&color=f23131)Java 8](java/Java8.md) -- [![img](https://icongr.am/entypo/address.svg?size=25&color=074ca6)设计模式](design-pattern/readDisignPattern.md) -- **数据存储和缓存** -- [![MySQL](https://icongr.am/devicon/mysql-original.svg?&size=25)MySQL](data-store/MySQL/readMySQL.md) -- [![Redis](https://icongr.am/devicon/redis-original.svg?size=25)Redis](data-store/Redis/2.readRedis.md) -- [![mongoDB](https://icongr.am/devicon/mongodb-original.svg?&size=25)mongoDB]( https://redis.io/ ) -- [![ **Elasticsearch** ](https://icongr.am/simple/elasticsearch.svg?&size=20) Elasticsearch]( https://redis.io/ ) -- [![S3](https://icongr.am/devicon/amazonwebservices-original.svg?&size=25)S3]( https://aws.amazon.com/cn/s3/ ) -- **直击面试** -- [![](https://icongr.am/material/basket.svg?size=25)Java集合面试](interview/Collections-FAQ.md) -- [![](https://icongr.am/devicon/java-plain-wordmark.svg?size=25)JVM面试](interview/JVM-FAQ.md) -- [![](https://icongr.am/devicon/mysql-original-wordmark.svg?size=25)MySQL面试](interview/MySQL-FAQ.md) -- [![](https://icongr.am/devicon/redis-original-wordmark.svg?size=25)Redis面试](interview/Redis-FAQ.md) -- [![](https://icongr.am/jam/leaf.svg?size=25&color=00FF00)Spring面试](interview/Spring-FAQ.md) -- [![](https://icongr.am/simple/bower.svg?size=25)MyBatis面试](interview/MyBatis-FAQ.md) -- [![img](https://icongr.am/entypo/network.svg?size=25&color=6495ED)计算机网络](interview/Network-FAQ.md) -- **单体架构** -- **RPC** -- [Hello Protocol Buffers](rpc/Hello-Protocol-Buffers.md) -- **面向服务架构** -- [![](https://icongr.am/fontawesome/group.svg?size=25&color=182d10)Zookeeper](soa/ZooKeeper/readZK.md) -- [![message](https://icongr.am/clarity/email.svg?&size=25) 消息中间件](message-queue/readMQ.md) -- [![](https://icongr.am/devicon/nginx-original.svg?size=25&color=182d10)Nginx](nginx/nginx.md) -- **微服务架构** -- [![](https://icongr.am/simple/leaflet.svg?size=25&color=11b041&colored=false)Spring Boot](framework/SpringBoot/Hello-SpringBoot.md) -- [![](https://icongr.am/feather/cloud.svg?size=25&color=36b305)Spring Cloud](framework/SpringCloud/readSpringCloud.md) -- [![](https://icongr.am/clarity/alarm-clock.svg?size=25&color=2d2b50)定时任务@Scheduled](framework/SpringBoot/@Scheduled.md) -- **大数据** -- [Hello 大数据](big-data/Hello-BigData.md) -- [![](https://icongr.am/simple/apachekafka.svg?size=25&color=121417&colored=false)Kafka](message-queue/Kafka/readKafka.md) -- **性能优化** -- [![](https://icongr.am/octicons/cpu.svg?size=25&color=780ebe)CPU 飙升问题](optimization/CPU飙升.md) -- \> JVM优化 -- \> web调优 -- \> DB调优 -- **数据结构与算法** -- [![](https://icongr.am/entypo/tree.svg?size=25&color=44c016)树](data-structure/tree.md) -- **工程化与工具** -- [![Maven](https://icongr.am/simple/apachemaven.svg?size=25&color=c93ddb&colored=false)Maven](tools/Maven.md) -- [![Git](https://icongr.am/devicon/git-original.svg?&size=16)Git](tools/Git-Specification.md) -- [![](https://icongr.am/devicon/github-original.svg?size=25&color=currentColor)Git](tools/GitHub.md) -- **其他** -- [![Linux](https://icongr.am/devicon/linux-original.svg?&size=16)Linux](linux/linux.md) -- [缕清各种Java Logging](logging/Java-Logging.md) -- [hello logback](logging/logback简单使用.md) -- **Links** -- [![Github](https://icongram.jgog.in/simple/github.svg?color=808080&size=16)Github](https://github.com/jhildenbiddle/docsify-tabs) -- [![Blog](https://icongr.am/simple/aboutme.svg?colored&size=16)My Blog](https://www.lazyegg.net) \ No newline at end of file diff --git a/docs/framework/SpringWebFlux/Reactive.md b/docs/framework/SpringWebFlux/Reactive.md new file mode 100644 index 0000000000..df7bab0d13 --- /dev/null +++ b/docs/framework/SpringWebFlux/Reactive.md @@ -0,0 +1,760 @@ +![](https://plus.unsplash.com/premium_photo-1661898205432-d648667b9c76?q=80&w=3131&auto=format&fit=crop&ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D) + +> 原文：[《Reactive Programming 一种技术，各自表述》](https://cloud.tencent.com/developer/article/1532839) + +## 前言 + +作为一名 Java 开发人员，尤其是 Java 服务端工程师，对于 Reactive Programming 的概念似乎相对陌生。随着 Java 9 以及 Spring Framework 5 的相继发布，Reactive 技术逐渐开始被广大从业人员所注意，我作为其中一员，更渴望如何理解 Reactive Programming，以及它所带来的哪些显著的编程变化，更为重要的是，怎么将其用于实际生产环境，解决当前面临的问题。然而，随着时间的推移和了解的深入，我对 Reactive Programming 的热情逐渐被浇息，对它的未来保持谨慎乐观的态度。 + +本文从理解 Reactive Programming 的角度出发，尽可能地保持理性和中立的态度，讨论 Reactive Programming 的实质。 + +## 初识 Reactive + +我第一次接触 Reactive 技术的时间还要回溯到 2015年末，当时部分应用正使用 Hystrix 实现服务熔断，而 Hystrix 底层依赖是 RxJava 1.x，RxJava 是 Reactive 在 Java 编程语言的扩展框架。当时接触 Reactive 只能算上一种间接的接触，根据 Hystrix 特性来理解 Reactive 技术，感觉上，Hystrix 超时和信号量等特性与 Java 并发框架（J.U.C）的关系密切，进而认为 Reactive 是 J.U.C 的扩展。随后，我便参考了两本关于 Reactive Java 编程方面的书：《Reactive Java Programming》和《Reactive Programming with RxJava》。遗憾的是，两者尽管详细地描述 RxJava 的使用方法，然而却没有把 Reactive 使用场景讨论到要点上，如《Reactive Programming with RxJava》所给出的使用场景说明： + +> When You Need Reactive Programming +> +> Reactive programming is useful in scenarios such as the following: +> +> - Processing user events such as mouse movement and clicks, keyboard typing,GPS signals changing over time as users move with their device, device gyroscope signals, touch events, and so on. +> - Responding to and processing any and all latency-bound IO events from disk or network, given that IO is inherently asynchronous ... +> - Handling events or data pushed at an application by a producer it cannot control ... + +实际上，以上三种使用场景早已在 Java 生态中完全地实现并充分地实践，它们对应的技术分别是 Java AWT/Swing、NIO/AIO 以及 JMS（Java 消息服务）。那么，再谈 RxJava 的价值又在哪里呢？如果读者是初学者，或许还能蒙混过关。好奇心促使我重新开始踏上探索 Reactive 之旅。 + +## 理解 Reactive + +2017年 Java 技术生态中，最有影响力的发布莫过于 Java 9 和 Spring 5，前者主要支持模块化，次要地提供了 Flow API 的支持，后者则将”身家性命“压在 Reactive 上面，认为 Reactive 是未来的趋势，它以 Reactive 框架 Reactor 为基础，逐步构建一套完整的 Reactive 技术栈，其中以 WebFlux 技术为最引人关注，作为替代 Servlet Web 技术栈的核心特性，承载了多年 Spring 逆转 Java EE 的初心。于是，业界开始大力地推广 Reactive 技术，于是我又接触到一些关于 Reactive 的讲法。 + +### 关于 Reactive 的一些讲法 + +其中我挑选了以下三种出镜率最高的讲法： + +- Reactive 是异步非阻塞编程 +- Reactive 能够提升程序性能 +- Reactive 解决传统编程模型遇到的困境 + +第一种说法描述了功能特性，第二种说法表达了性能收效，第三种说法说明了终极目地。下面的讨论将围绕着这三种讲法而展开，深入地探讨 Reactive Programming 的实质，并且理解为什么说 Reactive Programming 是”一种技术，各自表述“。 + +同时，讨论的方式也一反常态，并不会直奔主题地解释什么 Reactive Programming，而是从问题的角度出发，从 Reactive 规范和框架的论点，了解传统编程模型中所遇到的困境，逐步地揭开 Reactive 神秘的面纱。其中 Reactive 规范是 JVM Reactive 扩展规范 [Reactive Streams JVM](https://github.com/reactive-streams/reactive-streams-jvm)，而 Reactive 实现框架则是最典型的实现： + +- RxJava：Reactive Extensions +- Reactor：Spring WebFlux Reactive 类库 +- Flow API：Java 9 Flow API 实现 + +### 传统编程模型中的某些困境 + +#### [Reactor](http://projectreactor.io/docs/core/release/reference/#_blocking_can_be_wasteful) 认为阻塞可能是浪费的 + +> #### 3.1. Blocking Can Be Wasteful +> +> Modern applications can reach huge numbers of concurrent users, and, even though the capabilities of modern hardware have continued to improve, performance of modern software is still a key concern. +> +> There are broadly two ways one can improve a program’s performance: +> +> 1. **parallelize**: use more threads and more hardware resources. +> 2. **seek more efficiency** in how current resources are used. +> +> Usually, Java developers write programs using blocking code. This practice is fine until there is a performance bottleneck, at which point the time comes to introduce additional threads, running similar blocking code. But this scaling in resource utilization can quickly introduce contention and concurrency problems. +> +> Worse still, blocking wastes resources. +> +> So the parallelization approach is not a silver bullet. + +将以上 Reactor 观点归纳如下，它认为： + +1. 阻塞导致性能瓶颈和浪费资源 +2. 增加线程可能会引起资源竞争和并发问题 +3. 并行的方式不是银弹（不能解决所有问题） + +第三点基本上是废话，前面两点则较为容易理解，为了减少理解的偏差，以下讨论将结合示例说明。 + +##### 理解阻塞的弊端 + +假设有一个数据加载器，分别加载配置、用户信息以及订单信息，如下图所示： + +![](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcubXVrZXdhbmcuY29tLzViNTU0MWQ1MDAwMWZjZTMxMTg2MDQ0MC5wbmc?x-oss-process=image/format,png) + +```java +public class DataLoader { + + public final void load() { + long startTime = System.currentTimeMillis(); // 开始时间 + doLoad(); // 具体执行 + long costTime = System.currentTimeMillis() - startTime; // 消耗时间 + System.out.println("load() 总耗时：" + costTime + " 毫秒"); + } + + protected void doLoad() { // 串行计算 + loadConfigurations(); // 耗时 1s + loadUsers(); // 耗时 2s + loadOrders(); // 耗时 3s + } // 总耗时 1s + 2s + 3s = 6s + + protected final void loadConfigurations() { + loadMock("loadConfigurations()", 1); + } + + protected final void loadUsers() { + loadMock("loadUsers()", 2); + } + + protected final void loadOrders() { + loadMock("loadOrders()", 3); + } + + private void loadMock(String source, int seconds) { + try { + long startTime = System.currentTimeMillis(); + long milliseconds = TimeUnit.SECONDS.toMillis(seconds); + Thread.sleep(milliseconds); + long costTime = System.currentTimeMillis() - startTime; + System.out.printf("[线程 : %s] %s 耗时 : %d 毫秒\n", + Thread.currentThread().getName(), source, costTime); + } catch (InterruptedException e) { + throw new RuntimeException(e); + } + } + + public static void main(String[] args) { + new DataLoader().load(); + } + +} +``` + +- 运行结果 + +```css +[线程 : main] loadConfigurations() 耗时 : 1005 毫秒 +[线程 : main] loadUsers() 耗时 : 2002 毫秒 +[线程 : main] loadOrders() 耗时 : 3001 毫秒 +load() 总耗时：6031 毫秒 +``` + +- 结论 + +由于加载过程串行执行的关系，导致消耗实现线性累加。Blocking 模式即串行执行 。 + +不过 Reactor 也提到，以上问题可通过并行的方式来解决，不过编写并行程序较为复杂，那么其中难点在何处呢？ + +##### 理解并行的复杂 + +再将以上示例由串行调整为并行，如下图所示： + +![图片描述](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcubXVrZXdhbmcuY29tLzViNTU0MWViMDAwMTE0OGUxMTQ3MDQ0MS5wbmc?x-oss-process=image/format,png) + +```java +public class ParallelDataLoader extends DataLoader { + + protected void doLoad() { // 并行计算 + ExecutorService executorService = Executors.newFixedThreadPool(3); // 创建线程池 + CompletionService completionService = new ExecutorCompletionService(executorService); + completionService.submit(super::loadConfigurations, null); // 耗时 >= 1s + completionService.submit(super::loadUsers, null); // 耗时 >= 2s + completionService.submit(super::loadOrders, null); // 耗时 >= 3s + + int count = 0; + while (count < 3) { // 等待三个任务完成 + if (completionService.poll() != null) { + count++; + } + } + executorService.shutdown(); + } // 总耗时 max(1s, 2s, 3s) >= 3s + + public static void main(String[] args) { + new ParallelDataLoader().load(); + } + +} +``` + +- 运行结果 + +```bash +[线程 : pool-1-thread-1] loadConfigurations() 耗时 : 1003 毫秒 +[线程 : pool-1-thread-2] loadUsers() 耗时 : 2005 毫秒 +[线程 : pool-1-thread-3] loadOrders() 耗时 : 3005 毫秒 +load() 总耗时：3068 毫秒 +``` + +- 结论和思考 + +明显地，程序改造为并行加载后，性能和资源利用率得到提升，消耗时间取最大者，即三秒，由于线程池操作的消耗，整体时间将略增一点。 + +不过，以上实现为什么不直接使用 `Future#get()` 方法强制所有任务执行完毕，然后再统计总耗时？ + +以上三个方法之间没有数据依赖关系，所以执行方式由串行改为并行后，能提升性能。可是如果方法之间存在依赖关系，效果提升还会这么明显吗？并且怎么确保他们的执行顺序 + +#### [Reactor](http://projectreactor.io/docs/core/release/reference/#_asynchronicity_to_the_rescue) 认为异步不一定能够救赎 + +> #### 3.2. Asynchronicity to the Rescue? +> +> The second approach (mentioned earlier), seeking more efficiency, can be a solution to the resource wasting problem. By writing *asynchronous*, *non-blocking* code, you let the execution switch to another active task **using the same underlying resources** and later come back to the current process when the asynchronous processing has finished. +> +> Java offers two models of asynchronous programming: +> +> - **Callbacks**: Asynchronous methods do not have a return value but take an extra `callback` parameter (a lambda or anonymous class) that gets called when the result is available. A well known example is Swing’s `EventListener`hierarchy. +> - **Futures**: Asynchronous methods return a `Future` **immediately**. The asynchronous process computes a `T` value, but the `Future` object wraps access to it. The value is not immediately available, and the object can be polled until the value is available. For instance, `ExecutorService` running `Callable` tasks use `Future` objects. +> +> Are these techniques good enough? Not for every use case, and both approaches have limitations. +> +> Callbacks are hard to compose together, quickly leading to code that is difficult to read and maintain (known as "Callback Hell"). +> +> Futures are a bit better than callbacks, but they still do not do well at composition, despite the improvements brought in Java 8 by `CompletableFuture`. + +再次将以上观点归纳，它认为： + +- Callbacks 是解决非阻塞的方案，然而他们之间很难组合，并且快速地将代码引导至 "Callback Hell" 的不归路 +- Futures 相对于 Callbacks 好一点，不过还是无法组合，不过 `CompletableFuture` 能够提升这方面的不足 + +以上 Reactor 的观点仅给出了结论，没有解释现象，其中场景设定也不再简单直白，从某种程度上，这也侧面地说明，Reactive Programming 实际上是”高端玩家“的游戏。接下来，本文仍通过示例的方式，试图解释"Callback Hell" 问题以及 `Future` 的限制。 + +##### 理解 "Callback Hell" + +- Java GUI 示例 + +```java +public class JavaGUI { + + public static void main(String[] args) { + JFrame jFrame = new JFrame("GUI 示例"); + jFrame.setBounds(500, 300, 400, 300); + LayoutManager layoutManager = new BorderLayout(400, 300); + jFrame.setLayout(layoutManager); + jFrame.addMouseListener(new MouseAdapter() { // callback 1 + @Override + public void mouseClicked(MouseEvent e) { + System.out.printf("[线程 : %s] 鼠标点击，坐标(X : %d, Y : %d)\n", + currentThreadName(), e.getX(), e.getY()); + } + }); + jFrame.addWindowListener(new WindowAdapter() { // callback 2 + @Override + public void windowClosing(WindowEvent e) { + System.out.printf("[线程 : %s] 清除 jFrame... \n", currentThreadName()); + jFrame.dispose(); // 清除 jFrame + } + + @Override + public void windowClosed(WindowEvent e) { + System.out.printf("[线程 : %s] 退出程序... \n", currentThreadName()); + System.exit(0); // 退出程序 + } + }); + System.out.println("当前线程：" + currentThreadName()); + jFrame.setVisible(true); + } + + private static String currentThreadName() { // 当前线程名称 + return Thread.currentThread().getName(); + } +} +``` + +- 运行结果 + +点击窗体并关闭窗口，控制台输出如下： + +```css +当前线程：main +[线程 : AWT-EventQueue-0] 鼠标点击，坐标(X : 180, Y : 121) +[线程 : AWT-EventQueue-0] 鼠标点击，坐标(X : 180, Y : 122) +[线程 : AWT-EventQueue-0] 鼠标点击，坐标(X : 180, Y : 122) +[线程 : AWT-EventQueue-0] 鼠标点击，坐标(X : 180, Y : 122) +[线程 : AWT-EventQueue-0] 鼠标点击，坐标(X : 180, Y : 122) +[线程 : AWT-EventQueue-0] 鼠标点击，坐标(X : 201, Y : 102) +[线程 : AWT-EventQueue-0] 清除 jFrame... +[线程 : AWT-EventQueue-0] 退出程序... +``` + +- 结论 + + Java GUI 以及事件/监听模式基本采用匿名内置类实现，即回调实现。从本例可以得出，鼠标的点击确实没有被其他线程给阻塞。不过当监听的维度增多时，Callback 实现也随之增多。Java Swing 事件/监听是一种典型的既符合异步非阻塞，又属于 Callback 实现的场景，其并发模型可为同步或异步。 + + 不过，在 Java 8 之前，由于接口无法支持 `default` 方法，当接口方法过多时，通常采用 `Adapter` 模式作为缓冲方案，达到按需实现的目的。尤其在 Java GUI 场景中。即使将应用的 Java 版本升级到 8 ，由于这些 Adapter ”遗老遗少“实现的存在，使得开发人员仍不得不面对大量而繁琐的 Callback 折中方案。既然 Reactor 提出了这个问题，那么它或者 Reactive 能否解决这个问题呢？暂时存疑，下一步是如何理解 `Future` 的限制。 + +##### 理解 `Future` 的限制 + +Reactor 的观点仅罗列 `Future` 的一些限制，并没有将它们解释清楚，接下来用两个例子来说明其中原委。 + +###### 限制一：`Future` 的阻塞性 + +在前文示例中，`ParallelDataLoader` 利用 `CompletionService` API 实现 `load*()` 方法的并行加载，如果将其调整为 `Future` 的实现，可能的实现如下： + +- Java `Future` 阻塞式加载示例 + +```scala +public class FutureBlockingDataLoader extends DataLoader { + + protected void doLoad() { + ExecutorService executorService = Executors.newFixedThreadPool(3); // 创建线程池 + runCompletely(executorService.submit(super::loadConfigurations)); // 耗时 >= 1s + runCompletely(executorService.submit(super::loadUsers)); // 耗时 >= 2s + runCompletely(executorService.submit(super::loadOrders)); // 耗时 >= 3s + executorService.shutdown(); + } // 总耗时 sum(>= 1s, >= 2s, >= 3s) >= 6s + + private void runCompletely(Future future) { + try { + future.get(); // 阻塞等待结果执行 + } catch (Exception e) { + } + } + + public static void main(String[] args) { + new FutureBlockingDataLoader().load(); + } + +} +``` + +- 运行结果 + +```css +[线程 : pool-1-thread-1] loadConfigurations() 耗时 : 1003 毫秒 +[线程 : pool-1-thread-2] loadUsers() 耗时 : 2004 毫秒 +[线程 : pool-1-thread-3] loadOrders() 耗时 : 3002 毫秒 +load() 总耗时：6100 毫秒 +``` + +- 结论 + + `ParallelDataLoader` 加载耗时为”3068 毫秒“，调整后的 `FutureBlockingDataLoader` 则比串行的 `DataLoader` 加载耗时（“6031 毫秒”）还要长。说明`Future#get()` 方法不得不等待任务执行完成，换言之，如果多个任务提交后，返回的多个 Future 逐一调用 `get()` 方法时，将会依次 blocking，任务的执行从并行变为串行。这也是之前为什么 `ParallelDataLoader` 不采取 `Future` 的解决方案的原因。 + +###### 限制二：`Future` 不支持链式操作 + +由于 `Future` 无法实现异步执行结果链式处理，尽管 `FutureBlockingDataLoader` 能够解决方法数据依赖以及顺序执行的问题，不过它将并行执行带回了阻塞（串行）执行。所以，它不是一个理想实现。不过 `CompletableFuture` 可以帮助提升 `Future` 的限制： + +- Java `CompletableFuture` 重构 `Future` 链式实现 + +```scala +public class FutureChainDataLoader extends DataLoader { + + protected void doLoad() { + CompletableFuture + .runAsync(super::loadConfigurations) + .thenRun(super::loadUsers) + .thenRun(super::loadOrders) + .whenComplete((result, throwable) -> { // 完成时回调 + System.out.println("加载完成"); + }) + .join(); // 等待完成 + } + + public static void main(String[] args) { + new ChainDataLoader().load(); + } +} +``` + +- 运行结果 + +```css +[线程 : ForkJoinPool.commonPool-worker-1] loadConfigurations() 耗时 : 1000 毫秒 +[线程 : ForkJoinPool.commonPool-worker-1] loadUsers() 耗时 : 2005 毫秒 +[线程 : ForkJoinPool.commonPool-worker-1] loadOrders() 耗时 : 3001 毫秒 +加载完成 +load() 总耗时：6093 毫秒 +``` + +- 结论 + +通过输出日志分析， `FutureChainDataLoader` 并没有像 `FutureBlockingDataLoader` 那样使用三个线程分别执行加载任务，仅使用了一个线程，换言之，这三次加载同一线程完成，并且异步于 main 线程，如下所示： + +![图片描述](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWcubXVrZXdhbmcuY29tLzViNTU0MjAzMDAwMWE3MmMxMTcwMDMyOC5wbmc?x-oss-process=image/format,png) + +尽管 `CompletableFuture` 不仅是异步非阻塞操作，而且还能将 Callback 组合执行，也不存在所谓的 ”Callback Hell“ 等问题。如果强制等待结束的话，又回到了阻塞编程的方式。同时，相对于 `FutureBlockingDataLoader` 实现，重构后的 `FutureChainDataLoader` 不存在明显性能提升。 + +> 稍作解释，`CompletableFuture` 不仅可以支持 `Future` 链式操作，而且提供三种生命周期回调，即执行回调（Action）、完成时回调（Complete）、和异常回调（Exception），类似于 Spring 4 `ListenableFuture` 以及 Guava `ListenableFuture`。 + +至此，Reactor 的官方参考文档再没有出现其他有关”传统编程模型中的某些困境“的描述，或许读者和我一样，对 Reactive 充满疑惑，它真能解决以上问题吗？当然，监听则明，偏听则暗，下面我们再来参考 [Reactive Streams JVM](https://github.com/reactive-streams/reactive-streams-jvm) 的观点。 + +疑问： + +- 如果阻塞导致性能瓶颈和资源浪费的话，Reactive 也能解决这个问题吗？ +- CompletableFuture 属于一部操作，如果强制等待结束的话，又回到了阻塞编程的方式，那 Reactive 也会面临同样的问题吗 +- CompletableFuture 让我们理解到非阻塞不一定能提升性能，那 Reactive 也会这样吗 + + + +#### [Reactive Streams JVM](https://github.com/reactive-streams/reactive-streams-jvm#goals-design-and-scope) 认为异步系统和资源消费需要特殊处理 + +> Handling streams of data—especially “live” data whose volume is not predetermined—requires special care in an asynchronous system. The most prominent issue is that resource consumption needs to be carefully controlled such that a fast data source does not overwhelm the stream destination. Asynchrony is needed in order to enable the parallel use of computing resources, on collaborating network hosts or multiple CPU cores within a single machine. + +观点归纳： + +- 流式数据容量难以预判 +- 异步编程复杂 +- 数据源和消费端之间资源消费难以平衡 + +此观点与 Reactor 相同的部分是，两者均认为异步编程复杂，而前者还提出了数据结构（流式数据）以及数据消费问题。 + +无论两者的观点孰优谁劣，至少说明一个现象，业界对于 Reactive 所解决的问题并非达到一致，几乎各说各话。那么，到底怎样才算 Reactive Programming 呢？ + +- Reactive 到底是什么？ +- Reactive 的使用场景在哪里 +- Reactiv 存在哪些限制和不足 + + + +## 什么是 Reactive Programming + +关于什么是 Reactive Programming，下面给出六种渠道的定义，尝试从不同的角度，了解 Reactive Programming 的意涵。首先了解的是“[The Reactive Manifesto](https://www.reactivemanifesto.org/)” 中的定义 + +#### [The Reactive Manifesto](https://www.reactivemanifesto.org/) 中的定义 + +Reactive Systems are: Responsive, Resilient, Elastic and Message Driven. + +> https://www.reactivemanifesto.org/ + +该组织对 Reactive 的定义非常简单，其特点体现在以下关键字： + +- 响应的（Responsive） +- 适应性强的（Resilient） +- 弹性的（Elastic） +- 消息驱动的（Message Driven） + +不过这样的定义侧重于 Reactive 系统，或者说是设计 Reactive 系统的原则。 + +#### [维基百科](https://en.wikipedia.org/wiki/Reactive_programming)中的定义 + +维基百科作为全世界的权威知识库，其定义的公允性能够得到保证： + +> Reactive programming is a declarative programming paradigm concerned with **data streams** and the **propagation of change**. With this paradigm it is possible to express static (e.g. arrays) or dynamic (e.g. event emitters) data streams with ease, and also communicate that an inferred dependency within the associated execution model exists, which facilitates the automatic propagation of the changed data flow. +> +> > 参考地址：https://en.wikipedia.org/wiki/Reactive_programming + +维基百科认为 Reactive programming 是一种声明式的编程范式，其核心要素是**数据流（data streams ）**与**其传播变化（ propagation of change）**，前者是关于数据结构的描述，包括静态的数组（arrays）和动态的事件发射器（event emitters）。由此描述，脑海中浮现出以下技术视图： + +- 数据流：Java 8 `Stream` +- 传播变化：Java `Observable`/`Observer` +- 事件/监听：Java `EventObject`/`EventListener` + +这些技术能够很好地满足维基百科对于 Reactive 的定义，那么， Reactive 框架和规范的存在意义又在何方？或许以上定义过于抽象，还无法诠释 Reactive 的全貌。 + +#### [Spring](https://docs.spring.io/spring/docs/current/spring-framework-reference/web-reactive.html#webflux-why-reactive) 5 中的定义 + +> The term "reactive" refers to programming models that are built around **reacting to change** — network component reacting to I/O events, UI controller reacting to mouse events, etc. In that sense **non-blocking** is reactive because instead of being blocked we are now in the mode of reacting to notifications as operations complete or data becomes available. +> +> > 参考地址：https://docs.spring.io/spring/docs/current/spring-framework-reference/web-reactive.html#webflux-why-reactive + +相对于维基百科的定义，Spring 5 WebFlux 章节同样也提到了变化响应（reacting to change ） ，并且还说明非阻塞（non-blocking）就是 Reactive。同时，其定义的侧重点在响应通知方面，包括操作完成（operations complete）和数据可用（data becomes available）。Spring WebFlux 作为 Reactive Web 框架，天然支持非阻塞，不过早在 Servlet 3.1 规范时代皆以实现以上需求，其中包括 Servlet 3.1 非阻塞 API `ReadListener` 和`WriteListener`，以及 Servlet 3.0 所提供的异步上下文 `AsyncContext` 和事件监听 `AsyncListener`。这些 Servlet 特性正是为 Spring WebFlux 提供适配的，所以 Spring WebFlux 能完全兼容 Servlet 3.1 容器。难道 Reactive 仅是新包装的概念吗？或许就此下结论还为时尚早，不妨在了解一下 ReactiveX 的定义。 + +#### [ReactiveX](http://reactivex.io/intro.html) 中的定义 + +广泛使用的 RxJava 作为 ReactiveX 的 Java 实现，对于 Reactive 的定义，ReactiveX 具备相当的权威性： + +> ReactiveX extends the observer pattern to support sequences of data and/or events and adds operators that allow you to compose sequences together declaratively while abstracting away concerns about things like low-level threading, synchronization, thread-safety, concurrent data structures, and non-blocking I/O. +> +> > 参考地址：http://reactivex.io/intro.html + +不过，ReactiveX 并没有直接给 Reactive 下定义，而是通过技术实现手段说明如何实现 Reactive。ReactiveX 作为观察者模式的扩展，通过操作符（Opeators）对数据/事件序列（Sequences of data and/or events )进行操作，并且屏蔽并发细节（abstracting away…），如线程 API（`Exectuor` 、`Future`、`Runnable`）、同步、线程安全、并发数据结构以及非阻塞 I/O。该定义的侧重点主要关注于实现，包括设计模式、数据结构、数据操作以及并发模型。除设计模式之外，Java 8 `Stream` API 具备不少的操作符，包括迭代操作 for-each、map/reduce 以及集合操作 `Collector`等，同时，通过 `parallel()` 和 `sequential()` 方法实现并行和串行操作间的切换，同样屏蔽并发的细节。至于数据结构，`Stream` 和数据流或集合序列可以画上等号。唯独在设计模式上，`Stream` 是迭代器（Iterator）模式实现，而 ReactiveX 则属于观察者（Observer）模式的实现。 对此，Reactor 做了进一步地解释。 + +#### [Reactor](http://projectreactor.io/docs/core/release/reference/#intro-reactive) 中的定义 + +> The reactive programming paradigm is often presented in object-oriented languages as an extension of the Observer design pattern. One can also compare the main reactive streams pattern with the familiar Iterator design pattern, as there is a duality to the Iterable-Iterator pair in all of these libraries. One major difference is that, while an Iterator is pull-based, reactive streams are push-based. +> +> > [http](http://projectreactor.io/docs/core/release/reference/)[://projectreactor.io/docs/core/release/reference/#](http://projectreactor.io/docs/core/release/reference/)[intro-reactive](http://projectreactor.io/docs/core/release/reference/) + +同样地，Reactor 也提到了观察者模式（Observer pattern ）和迭代器模式（Iterator pattern）。不过它将 Reactive 定义为响应流模式（Reactive streams pattern ），并解释了该模式和迭代器模式在数据读取上的差异，即前者属于推模式（push-based），后者属于拉模式（pull-based）。难道就因为这因素，就要使用 Reactive 吗？这或许有些牵强。个人认为，以上组织均没有坦诚或者简单地向用户表达，都采用一种模糊的描述，多少难免让人觉得故弄玄虚。幸运地是，我从 ReactiveX 官方找到一位前端牛人 [André Staltz](https://gist.github.com/staltz)，在他博文[《The introduction to Reactive Programming you've been missing》](https://gist.github.com/staltz/868e7e9bc2a7b8c1f754)中，他给出了中肯的解释。 + +#### [André Staltz](https://gist.github.com/staltz) 给出的定义 + +> [Reactive programming is programming with **asynchronous data streams**.](https://gist.github.com/staltz/868e7e9bc2a7b8c1f754#reactive-programming-is-programming-with-asynchronous-data-streams) +> +> In a way, **this isn't anything new**. Event buses or your typical click events are really an asynchronous event stream, on which you can observe and do some side effects. Reactive is that **idea on steroids**. You are able to create data streams of anything, not just from click and hover events. Streams are cheap and ubiquitous, anything can be a stream: variables, user inputs, properties, caches, data structures, etc. +> +> > ["What is Reactive Programming?"](https://gist.github.com/staltz/868e7e9bc2a7b8c1f754#what-is-reactive-programming) + +他在文章指出，Reactive Programming 并不是新东西，而是司空见惯的混合物，比如事件监听、鼠标点击事件等。同时，文中也提到异步（asynchronous ）以及数据流（data streams）等关键字。如果说因为 Java 8 Stream 是迭代器模式的缘故，它不属于Reactive Programming 范式的话，那么，Java GUI 事件/监听则就是 Reactive。那么，Java 开发人员学习 RxJava、Reactor、或者 Java 9 Flow API 的必要性又在哪里呢？因此，非常有必要深入探讨 Reactive Programming 的使用场景。 + + + +### Reactive Programming特性 + +#### Reactive编程模型 + +语言模型：响应式编程+函数式编程（可选） + +> 参考资料：https://docs.spring.io/spring/docs/current/spring-framework-reference/web-reactive.html#webflux-programming-models + +Spring WebFlux 提供了两种编程模型的选择： + +- Annotated Controllers: 与Spring MVC一致，并基于spring-web模块中的相同注解。Spring MVC和WebFlux控制器都支持reactive (Reactor和RxJava)返回类型，因此，很难将它们区分开来，一个显著的区别是WebFlux也支持响应性的@RequestBody参数。 + +- Functional Endpoints: 基于lambda的轻量级函数式编程模型，可以将其看作一个小型库或一组实用程序，应用程序可以使用它们来路由和处理请求。与带注解的控制器最大的不同是，应用程序从头到尾负责处理请求，而不是通过注解声明然后被回调。 + + + +#### 对立模型：命令式编程（Imperative programming） + +https://en.wikipedia.org/wiki/Imperative_programming + +在计算机科学中，命令式编程是一种使用语句改变程序状态的编程范式。就像自然语言中的命令式语气表达命令的方式一样，命令式程序由计算机执行的命令组成。命令式编程着重于描述程序是如何操作的。 + +结论： + +- Reactive Programming：同步或异步非阻塞执行，数据传播被动通知 +- Imperative Programming：同步阻塞执行，数据主动获取 + + + +#### 数据结构 + +- 流式（Streams） +- 序列（Sequences） +- 事件（Events） + +流是按时间顺序排列的一系列进行中的事件。 + + + +#### 设计模式 + +- 扩展模式：观察者（Observer），推模式 +- 对立模式：迭代器（Iterator），拉模式 +- 混合模式：反应堆（Reactor）、Proactor + +> 模式对比： +> +> http://reactivex.io/intro.html +> +> An Observable is the asynchronous/push “dual” to the synchronous/pull Iterable + +结论： + +Reactive Programming作为观察者模式（Observer）的延伸，在处理流式数据的过中，并非使用传统的命令编程方式（Imperative Programming）同步拉取数据，如迭代器模式（Iterator），而是采用同步或异步非阻塞的推拉相结合的方式，响应数据传播时的变化。 + + + +#### 并发模型 + +非阻塞（Non-Blocking） 前提条件 + +- 同步（Synchronous） +- 异步（Asynchronous） + +屏蔽并发编程细节，如线程、同步、线程安全以及并发数据结构。 + + +### Reactive Programming 使用场景 + +正如同 Reactive Programming 的定义那样，各个组织各执一词，下面仍采用多方引证的方式，寻求 Reactive Programming 使用场景的“最大公约数”。 + +#### [Reactive Streams JVM](https://github.com/reactive-streams/reactive-streams-jvm) 认为的使用场景 + +> The main goal of Reactive Streams is to govern the exchange of stream data across an asynchronous boundary. +> +> > https://github.com/reactive-streams/reactive-streams-jvm + +[Reactive Streams JVM](https://github.com/reactive-streams/reactive-streams-jvm) 认为 Reactive Streams 用于在异步边界（asynchronous boundary）管理流式数据交换（ govern the exchange of stream data）。异步说明其并发模型，流式数据则体现数据结构，管理则强调它们的它们之间的协调。 + +#### [Spring 5](https://docs.spring.io/spring/docs/5.0.7.RELEASE/spring-framework-reference/web-reactive.html#webflux-performance) 认为的使用场景 + +> Reactive and non-blocking generally do not make applications run faster. They can, in some cases, for example if using the `WebClient` to execute remote calls in parallel. On the whole it requires more work to do things the non-blocking way and that can increase slightly the required processing time. +> +> The key expected benefit of reactive and non-blocking is the ability to scale with a small, fixed number of threads and less memory. That makes applications more resilient under load because they scale in a more predictable way. + +Spring 认为 Reactive 和非阻塞通常并非让应用运行更快速（generally do not make applications run faster），甚至会增加少量的处理时间，因此，它的使用场景则利用较少的资源，提升应用的伸缩性（scale with a small, fixed number of threads and less memory）。 + +#### [ReactiveX](http://reactivex.io/intro.html) 认为的使用场景 + +> The ReactiveX Observable model allows you to treat streams of asynchronous events with the same sort of simple, composable operations that you use for collections of data items like arrays. It frees you from tangled webs of callbacks, and thereby makes your code more readable and less prone to bugs. + +ReactiveX 所描述的使用场景与 Spring 的不同，它没有从性能入手，而是代码可读性和减少 Bugs 的角度出发，解释了 Reactive Programming 的价值。同时，强调其框架的核心特性：异步（asynchronous）、同顺序（same sort）和组合操作（composable operations）。它也间接地说明了，Java 8 `Stream` 在组合操作的限制，以及操作符的不足。 + +#### [Reactor](http://projectreactor.io/docs/core/release/reference/#intro-reactive) 认为的使用场景 + +> Composability and readability +> +> Data as a flow manipulated with a rich vocabulary of operators +> +> Nothing happens until you subscribe +> +> Backpressure or the ability for the consumer to signal the producer that the rate of emission is too high +> +> High level but high value abstraction that is concurrency-agnostic + +Reactor 同样强调结构性和可读性（Composability and readability）和高层次并发抽象（High level abstraction），并明确地表示它提供丰富的数据操作符（ rich vocabulary of operators）弥补 `Stream` API 的短板，还支持背压（Backpressure）操作，提供数据生产者和消费者的消息机制，协调它们之间的产销失衡的情况。同时，Reactor 采用订阅式数据消费（Nothing happens until you subscribe）的机制，实现 `Stream` 所不具备的数据推送机制。 + +至此，讨论接近尾声，最后的部分将 Reactive Programming 内容加以总结。 + +## 总结 Reactive Programming + +Reactive Programming 作为观察者模式（[Observer] 的延伸，不同于传统的命令编程方式同步拉取数据的方式，如迭代器模式（[Iterator] 。而是采用数据发布者同步或异步地推送到数据流（Data Streams）的方案。当该数据流（Data Steams）订阅者监听到传播变化时，立即作出响应动作。在实现层面上，Reactive Programming 可结合函数式编程简化面向对象语言语法的臃肿性，屏蔽并发实现的复杂细节，提供数据流的有序操作，从而达到提升代码的可读性，以及减少 Bugs 出现的目的。同时，Reactive Programming 结合背压（Backpressure）的技术解决发布端生成数据的速率高于订阅端消费的问题。 + + + +### Reactive Stream规范 + +> https://github.com/reactive-streams/reactive-streams-jvm + +Reactive Stream是 JVM 的面向流的库的标准和规范: + +- 处理可能无限多的元素 +- 按顺序 +- 在组件之间异步传递元素 +- 非阻塞背压 + + + +#### API组件 + +Publisher：数据发布者（上游） +Subscriber：数据订阅者（下游） +Subscription：订阅信号 +Processor：Publisher和Subscriber混合体 + +##### Publisher + +数据发布者，数据上游 + + 接口： + +```java +public interface Publisher { + public void subscribe(Subscriber s); +} +``` + +##### Subscriber + +接口： + +```java +public interface Subscriber { + public void onSubscribe(Subscription s); + public void onNext(T t); + public void onError(Throwable t); + public void onComplete(); +} +``` + + +信号事件： + +onSubscribe：当下游订阅时 +onNext：当下游接收数据时 +onComplete：当数据流（Data Streams）执行完成时 +onError：当数据流（Data Stream）执行错误时 + +##### Subscription + +订阅信号控制 + +接口： + +```java +public interface Subscription { + public void request(long n); + public void cancel(); +} +``` + + +信号操作： + +request：请求上游元素的数量 +cancel：请求停止发送数据并且清除资源 + +##### Processor + +消息发布者和订阅者综合体 + +接口： + +```java +public interface Processor extends Subscriber, Publisher { +} +``` + + + +#### 背压 + +https://en.wikipedia.org/wiki/Back_pressure + +在信息技术领域，这个术语也被类比地用来描述在I/O开关后的数据积累，如果缓冲区是满的，不能接收任何更多的数据; 传输设备停止发送数据包，直到缓冲区被清空并再次能够存储信息。 + +关键字: + +- I/O 切换(I/O switch ) +- 缓冲填满(the buffers are full ) +- 数据无法接受(incapable of receiving any more data) +- 传输设备(transmitting device ) +- 停止发送数据包 (halts the sending of data packets ) + +> http://projectreactor.io/docs/core/release/reference/#reactive.backpressure +> +> Propagating signals upstream is also used to implement backpressure, which we described in the assembly line analogy as a feedback signal sent up the line when a workstation processes more slowly than an upstream workstation. +> +> The real mechanism defined by the Reactive Streams specification is pretty close to the analogy: a subscriber can work in unbounded mode and let the source push all the data at its fastest achievable rate or it can use the request mechanism to signal the source that it is ready to process at most n elements. + +关键字: + +- Propagating signals upstream(传播上游信号) +- 无边界模式(unbounded mode) +- 处理最大元素数量(process at most n elements) + + +总结背压 + +假设下游Subscriber工作在无边界大小的数据流水线时，当上游Publisher提供数据的速率快于下游Subscriber的消费数据速率时，下游Subscriber将通过传播信号(request)到上游Publisher，请求限制数据的数量( Demand )或通知上游停止数据生产。 + + + +#### Reactor框架运用 + +核心API + +- Mono：0-1的异步结果 +- Flux：0-N的异步序列 +- Scheduler：Reactor调度线程池 + +##### Mono + +定义：0-1的异步结果 + +实现：Reactive Stream JVM API Publisher + +类比：异步 Optional + + + +##### Flux + +定义：0-N的异步序列 + +实现：Reactive Streams JVM API Publisher + +类比：异步Stream + + + +##### Scheduler + +定义：Reactor调度线程池 + +- 当前线程: Schedulers.immediate() + - 等价关系:Thread.currentThread() +- 单复用线程: Schedulers.single() + - 内部名称:"single" + - 线程名称:"single" + - 线程数量:单个 + - 线程idel时间:Long Live + - 底层实现:ScheduledThreadPoolExecutor (core 1) + - 弹性线程池: Schedulers.elastic() + - 内部名称:"elastic" + - 线程名称:"elastic-evictor-{num}" + - 线程数量:无限制(unbounded) + - 线程idel时间:60 秒 + - 底层实现:ScheduledThreadPoolExecutor +- 并行线程池: Schedulers.parallel() + - 内部名称:"parallel" + - 线程名称:"parallel-{num}" + - 线程数量:处理器数量 + - 线程idel时间:60 秒 + - 底层实现:ScheduledThreadPoolExecutor + diff --git a/docs/framework/SpringWebFlux/SpringWebFlux.md b/docs/framework/SpringWebFlux/SpringWebFlux.md new file mode 100644 index 0000000000..b40b2582b8 --- /dev/null +++ b/docs/framework/SpringWebFlux/SpringWebFlux.md @@ -0,0 +1,411 @@ +--- +title: Spring WebFlux +date: 2023-05-23 +tags: + - Spring +categories: Spring +--- + +![](https://miro.medium.com/v2/resize:fit:1228/format:webp/1*S_6ZOB75Uk-oLh8qVVuC8w.png) + +> WebFlux 到底是个什么，主要用于解决什么问题，有什么优势和劣势 +> +> 哪些场景或者业务适用于 WebFlux + +![](https://img.starfish.ink/spring/springwebflux-banner.svg) + + + +传统的基于Servlet的Web框架，如 Spring MVC，在本质上都是阻塞和多线程的，每个连接都会使用一个线程。在请求处理的时候，会在线程池中拉取一个工作者( worker )线程来对请求进行处理。同时，请求线程是阻塞的，直到工作者线程提示它已经完成为止。 + +在 Spring5 中，引入了一个新的异步、非阻塞的WEB模块，就是Spring-WebFlux。该框架在很大程度上是基于Reactor项目的，能够解决Web应用和API中对更好的可扩展性的需求。 + +> 关于Reactor响应式编程的前置知识：看上一篇 + +要了解 WebFlux ，首先得知道什么是响应式编程，什么是 Reactice + +### Reactive Streams（响应式流） + +Reactive Streams 是 JVM 中面向流的库标准和规范： + +一般由以下组成： + +- 发布者：发布元素到订阅者 +- 订阅者：消费元素 +- 订阅：在发布者中，订阅被创建时，将与订阅者共享 +- 处理器：发布者与订阅者之间处理数据 + +特性 + +- 处理可能无限数量的元素 +- 按顺序处理 +- 组件之间异步传递 +- 强制性非阻塞背压（Backpressure） + + + +### Backpressure（背压） + +## 一、响应式编程 + +这是微软为了应对 **高并发环境下** 的服务端编程，提出的一个实现 **异步编程** 的方案。 + +> reactive programming is a declarative programming paradigm concerned with data streams and the propagation of change + +响应式编程（reactive programming）是一种基于数据流（data stream）和变化传递（propagation of change）的声明式（declarative）的编程范式 + +![外行人都能看懂的WebFlux，错过了血亏！_Java_03](https://img-blog.csdnimg.cn/img_convert/92e5f1d2718e8e952ddcd160e819fad1.png) + +意思大概如下： + +- 在命令式编程(我们的日常编程模式)下，式子 `a=b+c`，这就意味着 `a`的值是由 `b` 和 `c` 计算出来的。如果 `b` 或者 `c` 后续有变化，不会影响到 `a` 的值 +- 在响应式编程下，式子 `a:=b+c`，这就意味着 `a` 的值是由 `b` 和 `c` 计算出来的。但如果 `b` 或者 `c` 的值后续有变化，会影响到 `a` 的值 + +我认为上面的例子已经可以帮助我们理解变化传递（propagation of change） + +那数据流（data stream）和声明式（declarative）怎么理解呢？ + +Lambda的语法是这样的(Stream流的使用会涉及到很多Lambda表达式的东西，所以一般先学Lambda再学Stream流)： + +![](https://img-blog.csdnimg.cn/img_convert/a2e6b36f5b89e764dce97124b84d5ea2.webp?x-oss-process=image/format,png) + +Stream流的使用分为三个步骤(创建Stream流、执行中间操作、执行最终操作)： + +![外行人都能看懂的WebFlux，错过了血亏！_Java_05](https://img-blog.csdnimg.cn/img_convert/6e069fbda532ae7e59b1a8b7188ff76a.webp?x-oss-process=image/format,png) + +执行中间操作实际上就是给我们提供了很多的API去操作Stream流中的数据(求和/去重/过滤)等等 + +![外行人都能看懂的WebFlux，错过了血亏！_Java_06](https://img-blog.csdnimg.cn/img_convert/3d6e3b7ebb4e6b8de55e048f6abc30ac.webp?x-oss-process=image/format,png) + +说了这么多，怎么理解数据流和声明式呢？其实是这样的： + +本来数据是我们自行处理的，后来我们把要处理的数据抽象出来（变成了数据流），然后通过API去处理数据流中的数据（是声明式的） +比如下面的代码；将数组中的数据变成数据流，通过显式声明调用.sum()来处理数据流中的数据，得到最终的结果： + +``` +public static void main(String[] args) { + int[] nums = { 1, 2, 3 }; + int sum2 = IntStream.of(nums).parallel().sum(); + System.out.println("结果为：" + sum2); +} +``` + +如图下所示： + +![外行人都能看懂的WebFlux，错过了血亏！_Java_07](https://img-blog.csdnimg.cn/img_convert/e65280d03d75299559d90138d99ebe65.png) + + + +### 响应式编程->异步非阻塞 + +说到响应式编程就离不开异步非阻塞。 + +从Spring官网介绍WebFlux的信息我们就可以发现asynchronous, nonblocking 这样的字样，因为响应式编程它是异步的，也可以理解成变化传递它是异步执行的。 + +我们的JDK8 Stream流是同步的，它就不适合用于响应式编程（但基础的用法是需要懂的，因为响应式流编程都是操作流嘛） + +而在JDK9 已经支持响应式流了，下面我们来看一下 + + + +响应式编程打破了传统的同步阻塞式编程模型，基于响应式数据流和背压机制实现了异步非阻塞式的网络通信、数据访问和事件驱动架构，能够减轻服务器资源之间的竞争关系，从而提高服务的响应能力。 + +Spring 5 中内嵌了响应式 Web 框架、响应式数据访问、响应式消息通信等多种响应式组件，从而极大简化了响应式应用程序的开发过程和难度。 + +事实上，响应式编程的实施目前主要有两个障碍，一个是关系型数据访问，而另一个就是网络协议。 + +## 二、JDK9 Reactive + +响应式流的规范早已经被提出了：里面提到了： + +> Reactive Streams is an initiative to provide a standard for asynchronous stream processing with non-blocking back pressure ----->http://www.reactive-streams.org/ + +翻译再加点信息： + +响应式流(Reactive Streams)通过定义一组实体，接口和互操作方法，给出了实现异步非阻塞背压的标准。第三方遵循这个标准来实现具体的解决方案，常见的有Reactor，RxJava，Akka Streams，Ratpack等。 + +规范里头实际上就是定义了四个接口： + +![外行人都能看懂的WebFlux，错过了血亏！_Java_09](https://img-blog.csdnimg.cn/img_convert/d6b7836ff0350a86ef964bd835530feb.png) + +Java 平台直到 JDK 9 才提供了对于Reactive的完整支持，JDK9也定义了上述提到的四个接口，在java.util.concurrent包上 + +![外行人都能看懂的WebFlux，错过了血亏！_Java_10](https://img-blog.csdnimg.cn/img_convert/23db7b08f8181fcc75118e7ce5623cba.png) + +一个通用的流处理架构一般会是这样的（生产者产生数据，对数据进行中间处理，消费者拿到数据消费)： + +![外行人都能看懂的WebFlux，错过了血亏！_Java_11](https://img-blog.csdnimg.cn/img_convert/d3df369ef19c771ca93654d88d5bfcaf.png) + +- 数据来源，一般称为生产者（Producer） +- 数据的目的地，一般称为消费者(Consumer) + +- 在处理时，对数据执行某些操作一个或多个处理阶段。（Processor) + +到这里我们再看回响应式流的接口，我们应该就能懂了： + +- Publisher（发布者)相当于生产者(Producer) +- Subscriber(订阅者)相当于消费者(Consumer) +- Processor就是在发布者与订阅者之间处理数据用的 + +在响应式流上提到了back pressure（背压)这么一个概念，其实非常好理解。在响应式流实现异步非阻塞是基于生产者和消费者模式的，而生产者消费者很容易出现的一个问题就是：生产者生产数据多了，就把消费者给压垮了。 + +而背压说白了就是：消费者能告诉生产者自己需要多少量的数据。这里就是Subscription接口所做的事。 + + + +## 三、Hello WebFlux + +WebFlux 是 Spring Framework5.0 中引入的一种新的反应式Web框架。通过Reactor项目实现 Reactive Streams规范，完全异步和非阻塞框架。本身不会加快程序执行速度，但在高并发情况下借助异步IO能够以少量而稳定的线程处理更高的吞吐，规避文件IO/网络IO阻塞带来的线程堆积。 + +#### WebFlux 的特性 + +WebFlux 具有以下特性： + +- **异步非阻塞** - 可以举一个上传例子。相对于 Spring MVC 是同步阻塞IO模型，Spring WebFlux这样处理：线程发现文件数据没传输好，就先做其他事情，当文件准备好时通知线程来处理（这里就是输入非阻塞方式），当接收完并写入磁盘（该步骤也可以采用异步非阻塞方式）完毕后再通知线程来处理响应（这里就是输出非阻塞方式）。 +- **响应式函数编程** - 相对于Java8 Stream 同步、阻塞的Pull模式，Spring Flux 采用Reactor Stream 异步、非阻塞Push模式。书写采用 Java lambda 方式,接近自然语言形式且容易理解。 +- **不拘束于Servlet** - 可以运行在传统的Servlet 容器（3.1+版本），还能运行在Netty、Undertow等NIO容器中。 + + + +#### WebFlux 的设计目标 + +- 适用高并发 +- 高吞吐量 +- 可伸缩性 + + + +WebFlux使用的响应式流并不是用JDK9平台的，而是一个叫做Reactor响应式流库。所以，入门WebFlux其实更多是了解怎么使用Reactor的API，下面我们来看看~ + +Reactor是一个响应式流，它也有对应的发布者(Publisher )，Reactor的发布者用两个类来表示： + +- Mono(返回0或1个元素) +- Flux(返回0-n个元素) + +而订阅者则是Spring框架去完成 + + +1. Spring提供了完整的支持响应式的服务端技术栈。 + 如下图所示，左侧为基于spring-webmvc的技术栈，右侧为基于spring-webflux的技术栈，- Spring WebFlux是基于响应式流的，因此可以用来建立异步的、非阻塞的、事件驱动的服务。它采用Reactor作为首选的响应式流的实现库，不过也提供了对RxJava的支持。 + - 由于响应式编程的特性，Spring WebFlux和Reactor底层需要支持异步的运行环境，比如Netty和Undertow；也可以运行在支持异步I/O的Servlet 3.1的容器之上，比如Tomcat（8.0.23及以上）和Jetty（9.0.4及以上）。 + - 从图的纵向上看，spring-webflux上层支持两种开发模式： + 类似于Spring WebMVC的基于注解（@Controller、@RequestMapping）的开发模式； + - Java 8 lambda 风格的函数式开发模式。 + - Spring WebFlux也支持响应式的Websocket服务端开发。 + + +2. 响应式Http客户端 + + 此外，Spring WebFlux也提供了一个响应式的Http客户端API WebClient。它可以用函数式的方式异步非阻塞地发起Http请求并处理响应。其底层也是由Netty提供的异步支持。 + 我们可以把WebClient看做是响应式的RestTemplate，与后者相比，前者： + + - 是非阻塞的，可以基于少量的线程处理更高的并发； + - 可以使用Java 8 lambda表达式； + - 支持异步的同时也可以支持同步的使用方式； + - 可以通过数据流的方式与服务端进行双向通信。 + +Spring Framework 中包含的原始 Web 框架 Spring Web MVC 是专门为 Servlet API 和 Servlet 容器构建的。反应式堆栈 Web 框架 Spring WebFlux 是在 5.0 版的后期添加的。它是完全非阻塞的，支持反应式流(Reactive Stream)背压，并在Netty，Undertow和Servlet 3.1 +容器等服务器上运行。 + +![](https://img-blog.csdnimg.cn/img_convert/a07973cc0b1fc58382d8fefdfe5f1683.png) + +- **编程模型**：Spring 5 web 模块包含了 Spring WebFlux 的 HTTP 抽象。类似 Servlet API , WebFlux 提供了 WebHandler API 去定义非阻塞 API 抽象接口。可以选择以下两种编程模型实现： + + - 注解控制层。和 MVC 保持一致，WebFlux 也支持响应性 @RequestBody 注解。 + + - 功能性端点。基于 lambda 轻量级编程模型，用来路由和处理请求的小工具。和上面最大的区别就是，这种模型，全程控制了请求 - 响应的生命流程 + +- **内嵌容器**：跟 Spring Boot 大框架一样启动应用，但 WebFlux 默认是通过 Netty 启动，并且自动设置了默认端口为 8080。另外还提供了对 Jetty、Undertow 等容器的支持。开发者自行在添加对应的容器 Starter 组件依赖，即可配置并使用对应内嵌容器实例。 + + 但是要注意，必须是 Servlet 3.1+ 容器，如 Tomcat、Jetty；或者非 Servlet 容器，如 Netty 和 Undertow。 + +- **Starter 组件**：Spring Boot Webflux 提供了很多 “开箱即用” 的 Starter 组件 + + + +Spring WebFlux 是一个异步非阻塞式 IO 模型，通过少量的容器线程就可以支撑大量的并发访问。底层使用的是 Netty 容器，这点也和传统的 SpringMVC 不一样，SpringMVC 是基于 Servlet 的。 + +> 接口的响应时间并不会因为使用了 WebFlux 而缩短，服务端的处理结果还是得由 worker 线程处理完成之后再返回给前端。 + + + +【spring-webmvc + Servlet + Tomcat】命令式的、同步阻塞的 + +【spring-webflux + Reactor + Netty】响应式的、异步非阻塞的 + + + +webflux的关键是自己编写的代码里面返回流（Flux/Mono），spring框架来负责处理订阅。 spring框架提供2种开发模式来编写响应式代码，使用mvc之前的注解模式和使用router function模式，都需要我们的代码返回流，spring的响应式数据库spring data jpa，如使用mongodb，也是返回流，订阅都需要交给框架，自己不能订阅。 + +spring框架提供2种开发模式来编写响应式代码，使用mvc之前的注解模式和使用router function模式，都需要我们的代码返回流，spring的响应式数据库spring data jpa，如使用mongodb，也是返回流，订阅都需要交给框架，自己不能订阅 + + + +## 调试 + +在响应式编程中，调试是块难啃的骨头，这也是从命令式编程到响应式编程的切换过程中，学习曲线最陡峭的地方。 + +在命令式编程中，方法的调用关系摆在面上，我们通常可以通过stack trace追踪的问题出现的位置。但是在异步的响应式编程中，一方面有诸多的调用是在水面以下的，作为响应式开发库的使用者是不需要了解的；另一方面，基于事件的异步响应机制导致stack trace并非很容易在代码中按图索骥的。 + + + +------ + + + +## Reactive Redis + +底层框架是Lettuce + +ReactiveRedisTemplate与RedisTemplate使用类似，但它提供的是异步的，响应式Redis交互方式。 + +这里再强调一下，响应式编程是异步的，ReactiveRedisTemplate发送Redis请求后不会阻塞线程，当前线程可以去执行其他任务。 + +等到Redis响应数据返回后，ReactiveRedisTemplate再调度线程处理响应数据。 + +响应式编程可以通过优雅的方式实现异步调用以及处理异步结果，正是它的最大的意义。 + + + +### Redis异步 + +说到Redis的通信，我们都知道Redis基于RESP(Redis Serialization Protocol)协议来通信，并且通信方式是`停等模型`，也就说一次通信独占一个连接直到client读取到返回结果之后才能释放该连接让其他线程使用。 + +这里小伙伴们思考一下，针对Redis客户端，我们能否使用异步通信方式呢？首先要理解这里讨论的异步到底是指什么，这里的异步就是能够让client端在等待Redis服务端返回结果的这段时间内不再阻塞死等，而是可以继续干其他事情。 + +针对异步，其实有两种实现思路，一种是类似于dubbo那样使用`单连接+序列号（标识单次通信）`的通信方式，另外一种是类似于netty client那样直接基于`Reactor模型`来做。*注意：方式一的底层通信机制一般也是基于Reactor模型，client端不管是什么处理方式，对于redis server端来说是无感知的。* + +https://zhuanlan.zhihu.com/p/77328969 + + + +https://subscription.packtpub.com/book/programming/9781788995979/5/ch05lvl1sec40/spring-mvc-versus-webflux + +![img](https://static.packt-cdn.com/products/9781788995979/graphics/d2af6e5b-5d26-448d-b54c-64b42d307736.png) + + + + + +- Mono：实现发布者，并返回 0 或 1 个元素，即单对象。 + +- Flux：实现发布者，并返回 N 个元素，即 List 列表对象。 + + + +#### WebHandler + +```java +public interface WebHandler { + /** + * Handle the web server exchange. + * @param exchange the current server exchange + * @return {@code Mono} to indicate when request handling is complete + */ + Mono handle(ServerWebExchange exchange); +} +``` + +在这里，说明一下HttpHandler与WebHandler的区别，两者的设计目标不同。前者主要针对的是跨HTTP服务器，即付出最小的代价在各种不同的HTTP服务器上保证程序正常运行。于是我们在前面的章节中看到了为适配Reactor Netty而进行的ReactorHttpHandlerAdapter类相关实现。而后者则侧重于提供构建常用Web应用程序的基本功能。例如，我们可以在上面的WebHandler源码中看到handle方法传入的参数是ServerWebExchange类型的，通过这个类型参数，我们所定义的WebHandler组件不仅可以访问请求（ServerHttpRequest getRequest）和响应（ServerHttpResponse getResponse），也可以访问请求的属性（Map getAttributes）及会话的属性，还可以访问已解析的表单数据（Form data）、多部分数据（Multipart data）等。 + + + +#### DispatcherHandler + +```java +public class DispatcherHandler implements WebHandler, ApplicationContextAware { + @Nullable + private List handlerMappings; + + @Nullable + private List handlerAdapters; + + @Nullable + private List resultHandlers; + ... + @Override + public void setApplicationContext(ApplicationContext applicationContext) { + initStrategies(applicationContext); + } + protected void initStrategies(ApplicationContext context) { + Map mappingBeans = BeanFactoryUtils.beansOfTypeIncludingAncestors( + context, HandlerMapping.class, true, false); + + ArrayList mappings = new ArrayList<>(mappingBeans.values()); + AnnotationAwareOrderComparator.sort(mappings); + this.handlerMappings = Collections.unmodifiableList(mappings); + + Map adapterBeans = BeanFactoryUtils.beansOfTypeIncludingAncestors( + context, HandlerAdapter.class, true, false); + + this.handlerAdapters = new ArrayList<>(adapterBeans.values()); + AnnotationAwareOrderComparator.sort(this.handlerAdapters); + + Map beans = BeanFactoryUtils.beansOfTypeIncludingAncestors( + context, HandlerResultHandler.class, true, false); + + this.resultHandlers = new ArrayList<>(beans.values()); + AnnotationAwareOrderComparator.sort(this.resultHandlers); + } + ... +} +``` + +Spring WebFlux 为了适配我们在 Spring MVC 中养成的开发习惯，围绕熟知的Controller进行了相应的适配设计，其中有一个WebHandler实现类DispatcherHandler(如下代码所示)，是请求处理的调度中心，实际处理工作则由可配置的委托组件执行。该模型非常灵活，支持多种工作流程。 + +换句话说，DispatcherHandler就是HTTP请求相应处理器（handler）或控制器（controller）的中央调度程序。DispatcherHandler会从Spring Configuration中发现自己所需的组件，也就是它会从应用程序上下文中（application context）查找以下内容。 + +### 路由模式 + +Spring WebFlux包含一个轻量级的函数式编程模型，其中定义的函数可以对请求进行路由处理。请求也可以通过基于注解形式的路由模式进行处理。 + + + + + + + +### 原理 + +首先，WebFlux是Spring的响应式框架，基于Reactor和Netty。那底层原理应该包括它的异步非阻塞模型、Reactor库的使用、Netty的处理流程，以及和传统Servlet的区别。 + +#### 一、异步非阻塞 I/O 模型 + +1. **与传统 Servlet 的对比** + - Spring MVC：基于 Servlet 的同步阻塞模型，每个请求占用一个线程，线程在等待 I/O（如数据库操作）时会被阻塞，导致高并发场景下线程资源耗尽。 + - WebFlux：采用异步非阻塞 I/O，通过 Reactor 库和 Netty 的事件驱动机制，单线程可处理多个请求。线程仅在数据就绪时被唤醒处理，避免资源浪费。 +2. **事件循环（EventLoop）** + - WebFlux 默认使用 Netty 作为服务器，其核心是 EventLoop 线程池（BossGroup 和 WorkerGroup）。BossGroup 负责接收连接，WorkerGroup 处理 I/O 事件（如 HTTP 请求解析）。 + - 处理逻辑通过回调机制实现，例如在 Netty 的 `HttpServerHandle`中，当请求到达时触发 `onStateChange`方法，将请求交给 Reactor 的响应式流处理。 + +#### 二、响应式编程与 Reactor 库 + +1. **Reactive Streams 规范** + - WebFlux 基于 Reactive Streams 规范，通过 Publisher-Subscriber模型实现数据流的异步处理，支持**背压（Backpressure）**机制，防止生产者压垮消费者。 + - 核心类： + - **Flux**：处理 0-N 个元素的异步序列（如分页查询结果流）。 + - Mono：处理 0-1 个元素的异步序列（如单条数据库查询）。 +2. **线程切换与调度** + - 默认情况下，WebFlux 的请求处理在 Netty 的 I/O 线程中执行。若业务逻辑耗时较长（如 CPU 密集型操作），需通过 `publishOn` 或 `subscribeOn` 切换到自定义线程池，避免阻塞 I/O 线程。 + - 示例：使用 `Schedulers.elastic()` 或 `Schedulers.parallel()`管理线程池。 + +#### 三、核心处理流程 + +1. **请求处理链路** + - Netty 接收请求：Netty 的 `HttpServerOperations` 封装请求和响应对象，触发状态变更事件（如 `REQUEST_RECEIVED`）。 + - Handler 处理：请求通过 `ReactorHttpHandlerAdapter` 转发至 WebFlux 的 `DispatcherHandler`，匹配路由并调用 Controller 方法。 + - 响应式流传递：Controller 返回 `Flux`/`Mono`对象，数据流经过滤器链（如 `WebFilter`）后，由 Netty 异步写入响应。 +2. **背压实现** + - 当消费者处理速度慢于生产者时，通过 `Subscription.request(n)` 控制数据拉取速率。例如，数据库查询结果流分批推送，避免内存溢出 + + + + + +## References + +- https://docs.spring.io/spring-framework/reference/web-reactive.html +- [vivo互联网技术-深入剖析 Spring WebFlux](https://mp.weixin.qq.com/s?__biz=MzI4NjY4MTU5Nw==&mid=2247492039&idx=2&sn=eec30ff895a29e608fdafe78c626115d&chksm=ebdb9155dcac1843580ec28b8c31e334eb8aeb0a10573b5f3d4e7e8aaaec49893772399790a9&cur_album_id=1612326847164284932&scene=189#wechat_redirect) +- http://www.dre.vanderbilt.edu/~schmidt/PDF/reactor-siemens.pdf +- https://blog.51cto.com/u_12206475/3118309 +- https://blog.51cto.com/u_12206475/3118303 +- https://blog.csdn.net/qq_33371766/article/details/123642687 \ No newline at end of file diff --git a/docs/framework/SpringWebFlux/WebFlux.md b/docs/framework/SpringWebFlux/WebFlux.md new file mode 100644 index 0000000000..77a4942086 --- /dev/null +++ b/docs/framework/SpringWebFlux/WebFlux.md @@ -0,0 +1,163 @@ +\# 深入剖析 Spring WebFlux：响应式编程的利器 ![](https://miro.medium.com/v2/resize:fit:1228/format:webp/1*S_6ZOB75Uk-oLh8qVVuC8w.png) > WebFlux 是 Spring 生态中颠覆性的技术革新，它如何重新定义高并发场景下的服务端编程？ > 本文通过原理剖析+实战演示+性能对比，带你彻底掌握响应式编程范式 ![](https://img.starfish.ink/spring/springwebflux-banner.svg) + +## 一、传统阻塞模型的困境 + +在典型的 Spring MVC 应用中，每个 HTTP 请求都会从 Tomcat 线程池中获取一个工作线程： + +```java + // 伪代码： +class BlockingController { + @GetMapping("/data") + public Data getData() { + // 占用线程直至方法返回 + return service.blockingIO(); // 同步阻塞调用 + } +} +``` + +这种同步阻塞模型存在两大瓶颈： + +1. **线程资源耗尽**：当并发请求量超过线程池大小时，新请求必须等待 +2. **CPU闲置浪费**：线程在等待数据库响应时处于阻塞状态（通过 `jstack` 可观察到大量 `WAITING` 线程） + +## 二、WebFlux 的响应式革命 + +### 2.1 核心设计理念 + +```java +// 伪代码：WebFlux响应式模型 +class ReactiveController { + @GetMapping("/flux") + public Flux getStream() { // 立即返回流对象 + return service.reactiveIO(); // 异步非阻塞调用 + } +} +``` + +WebFlux 的核心创新体现在三个层面： + +| 维度 | 传统模型 | WebFlux模型 | +| ---------- | ---------------- | -------------------- | +| 线程模型 | 同步阻塞 | 异步非阻塞 | +| 资源利用率 | 每个请求独占线程 | 少量线程处理海量请求 | +| 背压支持 | 无 | 自动流量控制 | +| 适用场景 | CRUD密集型 | 高并发+流式处理 | + +### 2.2 架构演进 + +Spring 5 的模块化设计： + +![Spring架构](https://docs.spring.io/spring-framework/reference/images/spring-overview.png) + +关键组件解析： + +1. **Reactive Streams API**：标准化响应式编程接口（Publisher/Subscriber） +2. **Reactor 库**：Spring 官方响应式库（Mono/Flux） +3. **Netty/Undertow**：非阻塞式服务器支持 +4. **Router Functions**：函数式端点声明 + +### 2.3 线程模型对比 + +通过 VisualVM 监控线程状态： + +![线程状态对比](https://img-blog.csdnimg.cn/img_convert/0e3d1d17fccf3d1f7b4d1a8c4d4c4d5a.png) + +- **Tomcat 线程池**：大量线程处于 `TIMED_WAITING` 状态 +- **Netty EventLoop**：少量固定线程处理所有请求 + + + +## 三、深度解析 Reactor 编程模型 + +### 3.1 核心类型操作 + +```java +Flux.just("A", "B", "C") + .delayElements(Duration.ofMillis(100)) + .map(String::toLowerCase) + .flatMap(s -> queryFromDB(s)) + .subscribeOn(Schedulers.boundedElastic()) + .subscribe( + data -> log.info("Received: {}", data), + err -> log.error("Error: {}", err.getMessage()), + () -> log.info("Stream completed") + ); +``` + +常用操作符分类： + +| 类型 | 操作符 | +| -------- | ---------------------- | +| 创建 | just, create, generate | +| 转换 | map, flatMap, filter | +| 组合 | zip, merge, concat | +| 错误处理 | onErrorReturn, retry | +| 调度控制 | publishOn, subscribeOn | + +### 3.2 背压机制实战 + +```Java +Flux.range(1, 1000) + .onBackpressureBuffer(50) + .doOnNext(i -> log.debug("Emit: {}", i)) + .subscribe(new BaseSubscriber() { + @Override + protected void hookOnSubscribe(Subscription s) { + s.request(10); + } + + @Override + protected void hookOnNext(Integer value) { + process(value); + request(1); + } + }); +``` + +背压策略对比： + +| 策略 | 特点 | +| ------ | -------------- | +| BUFFER | 缓冲未处理元素 | +| DROP | 丢弃溢出元素 | +| LATEST | 只保留最新元素 | +| ERROR | 抛出异常 | + + + +## 四、WebFlux 内部机制揭秘 + +### 4.1 请求处理全流程 + +``` +Mermaid +``` + +### 4.2 核心组件协作 + +![处理流程](https://docs.spring.io/spring-framework/reference/images/spring-webflux-handling.png) + +1. **WebHandler API**：非阻塞式处理入口 +2. **ReactiveAdapterRegistry**：响应式类型转换 +3. **ResultHandler**：处理不同返回类型 + + + +## 六、最佳实践指南 + +### 6.1 适用场景建议 + +1. 网关代理等高并发端点 +2. 实时日志推送等流式API +3. 与Spring MVC 并存的混合架构 + + + +"响应式不是银弹，而是工具箱中的新武器" —— Rossen Stoyanchev + + + +## 附录：扩展阅读 + +- [Reactor 官方文档](https://www.wenxiaobai.com/chat/200006#) +- [WebFlux 性能调优指南](https://www.wenxiaobai.com/chat/200006#) \ No newline at end of file diff --git a/docs/framework/logging/Java-Logging.md b/docs/framework/logging/Java-Logging.md new file mode 100644 index 0000000000..d376d4b6b1 --- /dev/null +++ b/docs/framework/logging/Java-Logging.md @@ -0,0 +1,218 @@ +日志就是记录程序的运行轨迹，方便查找关键信息，也方便快速定位解决问题。 + +> 谁不知道日志呀，下面我们看看世界上最好的语言 Java 的常用日志实现 +> +> ```java +> private static final Logger logger = Logger.getLogger("jul"); +> private static final Logger trace = LogManager.getLogger("log4j"); +> private static final Logger logger = LoggerFactory.getLogger("SLF4J"); +> private static final Log log = LogFactory.getLog("Apache Commons Logging"); +> ``` +> +> 我**，这么多，怎么记得住，我要用哪个？ +> +> ![img](https://tva1.sinaimg.cn/large/007S8ZIlly1ghjejmjjlhj3073073weh.jpg) +> +> 项目中有一堆日志相关的 Jar 包，到底应该引入哪个排除哪个？ +> +> 这两个 Jar 包冲突了? 这个包需要依赖这个包? +> +> 这类问题有时候容易让人崩溃，所以就来缕一缕 Java 日志框架的那点事~~ + + + + + +## Java常用日志框架 + +- **[JUL](https://docs.oracle.com/javase/8/docs/technotes/guides/logging/overview.html "JUL")** (Java Util Logging) ：自 Java1.4 以来，Java 在 `Java.util.logging` 中提供的一个内置框架，也常称为 JDKLog、jdk-logging +- **[Log4j](https://logging.apache.org/log4j/2.x/ "log4j")**：Apache Log4j 是一个基于 Java 的日志记录工具。最初是 Log4j 1，我们现在用的大都是 Log4j 2，Apache Log4j 2 是 Apache 开发的一款 Log4j 的升级产品，Log4j 2 与 Log4j 1 发生了很大的变化，Log4j 2 不兼容Log4j 1 +- **[SLF4J](http://www.slf4j.org/ "SLF4J")**：`Simple Logging Facade for Java`，类似于 Commons Logging，是一套简易 Java 日志门面，本身并无日志的实现 +- **[Logback](http://logback.qos.ch/ "Logback")**：Logback 是 `Slf4j` 的原生实现框架，同样也是出自 `Log4j` 一个人之手，但拥有比 `log4j` 更多的优点、特性和更做强的性能，现在基本都用来代替 `log4j` 成为主流 +- **[tinylog](https://tinylog.org/v2/ "tinylog")**： 一个轻量级的开源日志框架 +- **[Apache Commons Logging](http://commons.apache.org/proper/commons-logging/ "Apache Commons Logging")**：Apache 基金会所属的项目，是一套 Java 日志接口，之前叫 `Jakarta Commons Logging`，后更名为 Commons Logging + + + +JUL、log4j 、Logback 是**日志实现框架**，而 Apache Commons Logging 和 SLF4J 是**日志实现门面**，可以理解为一个适配器，**可以将你的应用程序从日志框架中解耦**。 + +> 日志门面，是门面模式的一个典型的应用。 +> +> 门面模式（Facade Pattern），也称之为外观模式，其核心为：外部与一个子系统的通信必须通过一个统一的外观对象进行，使得子系统更易于使用。 +> +> ![](https://static01.imgkr.com/temp/34d51bca34e54c05a32d148439244496.png) + + + +> 阿里巴巴 Java 开发手册嵩山版_日志规约： +> +> 【强制】应用中不可直接使用日志系统（Log4j、Logback）中的 API，而应依赖使用日志框架（SLF4J、JCL--Jakarta Commons Logging）中的 API，使用门面模式的日志框架，有利于维护和各个类的日志处理方式统一。 +> +> 说明：日志框架（SLF4J、JCL--Jakarta Commons Logging）的使用方式（推荐使用 SLF4J） +> +> 使用 SLF4J： +> +> ```java +> import org.slf4j.Logger; +> import org.slf4j.LoggerFactory; +> private static final Logger logger = LoggerFactory.getLogger(Test.class); +> ``` +> +> 使用 JCL： +> +> ```java +> import org.apache.commons.logging.Log; +> import org.apache.commons.logging.LogFactory; +> private static final Log log = LogFactory.getLog(Test.class); +> ``` + + + +到这里我好像就明白了，有 2 个日志接口标准和其他的几个具体实现，但还是要扯一扯，要不没人点赞~~ + + + +## [Java常用日志框架历史](https://www.cnblogs.com/chenhongliang/p/5312517.html "Java常用日志框架历史") + +- 1996 年早期，欧洲安全电子市场项目组决定编写它自己的程序跟踪 API(Tracing API)。经过不断的完善，这个API 终于成为一个十分受欢迎的 Java 日志软件包，即 Log4j。后来 Log4j 成为 Apache 基金会项目中的一员。 + +- 2002 年 Java1.4 发布，Sun 推出了自己的日志库 JUL(Java Util Logging)，其实现基本模仿了 Log4j 的实现。在 JUL 出来以前，Log4j 就已经成为一项成熟的技术，使得 Log4j 在选择上占据了一定的优势。 + + > 谁能想到 Java1.4 之前，JDK 竟然没有内置的日志功能呢 + +- 因为有了两种选择，所以导致了日志使用的混乱。所以 Apache 又推出了 `Jakarta Commons Logging`，JCL 只是定义了一套日志接口，支持运行时动态加载日志组件的实现，也就是说，应用层编写代码时，只需调用 JCL 提供的统一接口来记录日志，底层实现可以是 `Log4j`，也可以是 `Java Util Logging`。 + +- 2006年，Log4j 的作者 Ceki Gülcü 从 Apache 离职，然后又搞出来一套类似 JCL 的接口类——Slf4j 和它的默认实现 Logback。 + +- 从此，Java 日志领域被划分为两大阵营：Commons Logging 阵营和 Slf4j 阵营。 + +- 2012 年，Apache 眼看有被 Logback 反超的势头，于是重写了 Log4j 1.x，成立了新的项目 Log4j 2，Log4j 2 具有 Logback 的所有特性。 + + + +## Java 日志级别 + +不同的日志框架，级别也会有些差异 + +- j.u.l：OFF、SEVERE、WARNING、INFO、CONFIG、FINE、FINER、FINEST、ALL + +- log4j：OFF、FATAL、ERROR、WARN、INFO、DEBUG、TRACE、 ALL + +- logback：OFF、ERROR、WARN、INFO、DEBUG、TRACE、 ALL + +| 日志级别 | 描述 | +| -------- | -------------------------------------------------- | +| OFF | 关闭：最高级别，不输出日志。 | +| FATAL | 致命：输出非常严重的可能会导致应用程序终止的错误。 | +| ERROR | 错误：输出错误，但应用还能继续运行。 | +| WARN | 警告：输出可能潜在的危险状况。 | +| INFO | 信息：输出应用运行过程的详细信息。 | +| DEBUG | 调试：输出更细致的对调试应用有用的信息。 | +| TRACE | 跟踪：输出更细致的程序运行轨迹。 | +| ALL | 所有：输出所有级别信息。 | + + + +## Commons Logging 与 Slf4j 实现机制对比 + +#### Commons Logging 实现机制 + +Commons Logging 是通过动态查找机制，在程序运行时，使用自己的 ClassLoader 寻找和载入本地具体的实现。详细策略可以查看 `commons-logging-*.jar` 包中的`org.apache.commons.logging.impl.LogFactoryImpl.java` 文件。由于 Osgi 不同的插件使用独立的ClassLoader，Osgi 的这种机制保证了插件互相独立，其机制限制了 Commons Logging 在 Osgi 中的正常使用。 + +#### Slf4j 实现机制 + +Slf4j 在编译期间，静态绑定本地的 Log 库，因此可以在 Osgi 中正常使用。它是通过查找类路径下`org.slf4j.impl.StaticLoggerBinder`，然后在 StaticLoggerBinder 中进行绑定。 + + + +## 项目中选择日志框架选择 + +如果是在一个新的项目中建议使用 Slf4j 与 Logback 组合，这样有如下的几个优点 + +- Slf4j 实现机制决定 Slf4j 限制较少，使用范围更广。由于 Slf4j 在编译期间，静态绑定本地的 LOG 库使得通用性要比 Commons Logging 要好。 + +- Logback 拥有更好的性能。Logback 声称：某些关键操作，比如判定是否记录一条日志语句的操作，其性能得到了显著的提高。这个操作在 Logback 中需要 3 纳秒，而在 Log4J 中则需要 30 纳秒。LogBack 创建记录器（logger）的速度也更快：13 毫秒，而在 Log4J 中需要 23 毫秒。更重要的是，它获取已存在的记录器只需94 纳秒，而 Log4J 需要 2234 纳秒，时间减少到了 1/23。跟 JUL 相比的性能提高也是显著的。 + +- Commons Logging 开销更高 + + 在使Commons Logging时为了减少构建日志信息的开销，通常的做法是 + + ```java + if(log.isDebugEnabled()){ + log.debug("User name： " + + user.getName() + " buy goods id ：" + good.getId()); + } + ``` + + 在 Slf4j 阵营，你只需这么做： + + ```java + log.debug("User name：{} ,buy goods id ：{}", user.getName(),good.getId()); + ``` + + 也就是说，Slf4j 把构建日志的开销放在了它确认需要显示这条日志之后，减少内存和 cup 的开销，使用占位符号，代码也更为简洁 + +- Logback 文档免费。Logback 的所有文档是全面免费提供的，不像 Log4J 那样只提供部分免费文档而需要用户去购买付费文档。 + + + +## SLF4J 绑定日志框架 + +![](https://tva1.sinaimg.cn/large/007S8ZIlly1ghjfdb8016j31cy0riwhw.jpg) + +![来源：slf4j.org](http://www.slf4j.org/images/concrete-bindings.png) + + + + + +## 阿里Java开发手册——日志规约 + +2. 【强制】日志文件至少保存 15 天，因为有些异常具备以“周”为频次发生的特点。 + +3. 【强制】应用中的扩展日志（如打点、临时监控、访问日志等）命名方式： appName_logType_logName.log。 logType:日志类型，如 stats/monitor/access 等；logName:日志描述。这种命名的好处： 通过文件名就可知道日志文件属于什么应用，什么类型，什么目的，也有利于归类查找。 正例：mppserver 应用中单独监控时区转换异常，如： mppserver_monitor_timeZoneConvert.log 说明：推荐对日志进行分类，如将错误日志和业务日志分开存放，便于开发人员查看，也便于 通过日志对系统进行及时监控。 + +4. 【强制】对 trace/debug/info 级别的日志输出，必须使用条件输出形式或者使用占位符的方 式。 说明：logger.debug("Processing trade with id: " + id + " and symbol: " + symbol); 如果日志级别是 warn，上述日志不会打印，但是会执行字符串拼接操作，如果 symbol 是对象， 会执行 toString()方法，浪费了系统资源，执行了上述操作，最终日志却没有打印。 + + 正例：（条件）建设采用如下方式 + + ```java + if (logger.isDebugEnabled()) { + logger.debug("Processing trade with id: " + id + " and symbol: " + symbol); + } + ``` + + 正例：（占位符） + + ```java + logger.debug("Processing trade with id: {} and symbol : {} ", id, symbol); + ``` + +5. 【强制】避免重复打印日志，浪费磁盘空间，务必在 log4j.xml 中设置 additivity=false。 正例： + +6. 【强制】异常信息应该包括两类信息：案发现场信息和异常堆栈信息。如果不处理，那么通过 关键字 throws 往上抛出。 正例：logger.error(各类参数或者对象 toString() + "_" + e.getMessage(), e); + +7. 【推荐】谨慎地记录日志。生产环境禁止输出 debug 日志；有选择地输出 info 日志；如果使 用 warn 来记录刚上线时的业务行为信息，一定要注意日志输出量的问题，避免把服务器磁盘 撑爆，并记得及时删除这些观察日志。 说明：大量地输出无效日志，不利于系统性能提升，也不利于快速定位错误点。记录日志时请 思考：这些日志真的有人看吗？看到这条日志你能做什么？能不能给问题排查带来好处？ + +8. 【推荐】可以使用 warn 日志级别来记录用户输入参数错误的情况，避免用户投诉时，无所适 从。如非必要，请不要在此场景打出 error 级别，避免频繁报警。 说明：注意日志输出的级别，error 级别只记录系统逻辑出错、异常或者重要的错误信息。 + +9. 【推荐】尽量用英文来描述日志错误信息，如果日志中的错误信息用英文描述不清楚的话使用 中文描述即可，否则容易产生歧义。国际化团队或海外部署的服务器由于字符集问题，【强制】 使用全英文来注释和描述日志错误信息。 + + + +> [Ultimate Guide to Logging](https://www.loggly.com/ultimate-guide/java-logging-basics/#layouts) +> +> [《Java常用日志框架介绍》](https://www.cnblogs.com/chenhongliang/p/5312517.html) + + + + + + + +http://www.slf4j.org/manual.html + +[https://www.cnblogs.com/chenhongliang/p/5312517.html#java%E6%97%A5%E5%BF%97%E6%A6%82%E8%BF%B0](https://www.cnblogs.com/chenhongliang/p/5312517.html#java日志概述) + + + diff --git "a/docs/logging/logback\347\256\200\345\215\225\344\275\277\347\224\250.md" "b/docs/framework/logging/logback\347\256\200\345\215\225\344\275\277\347\224\250.md" similarity index 50% rename from "docs/logging/logback\347\256\200\345\215\225\344\275\277\347\224\250.md" rename to "docs/framework/logging/logback\347\256\200\345\215\225\344\275\277\347\224\250.md" index 01dd7c465d..cd3869d0f4 100644 --- "a/docs/logging/logback\347\256\200\345\215\225\344\275\277\347\224\250.md" +++ "b/docs/framework/logging/logback\347\256\200\345\215\225\344\275\277\347\224\250.md" @@ -1,401 +1,802 @@ - （英文手册） - -## 一、logback介绍 - -Logback是由log4j创始人设计的另一个开源日志组件,官方网站： http://logback.qos.ch。用来继承代替log4j，目前logback有3个模块： - -logback-core：其它两个模块的基础模块 - -logback-classic：它是log4j的一个改良版本，同时它完整实现了slf4j API使你可以很方便地更换成其它日志系统如log4j或JDK14 Logging - -logback-access：访问模块，与Servlet容器集成提供通过Http来访问日志的功能 - -## 二、logback取代log4j的理由： - -1. 更快的实现：Logback的内核重写了，在一些关键执行路径上性能提升10倍以上。而且logback不仅性能提升了，初始化内存加载也更小了。 -2. 非常充分的测试：Logback经过了几年，数不清小时的测试。Logback的测试完全不同级别的。 -3. Logback-classic非常自然实现了SLF4j：Logback-classic实现了SLF4j。在使用SLF4j中，你都感觉不到logback-classic。而且因为logback-classic非常自然地实现了slf4j ， 所 以切换到log4j或者其他，非常容易，只需要提供成另一个jar包就OK，根本不需要去动那些通过SLF4JAPI实现的代码。 -4. 非常充分的文档 官方网站有两百多页的文档。 -5. 自动重新加载配置文件，当配置文件修改了，Logback-classic能自动重新加载配置文件。扫描过程快且安全，它并不需要另外创建一个扫描线程。这个技术充分保证了应用程序能跑得很欢在JEE环境里面。 -6. Lilith是log事件的观察者，和log4j的chainsaw类似。而lilith还能处理大数量的log数据 。 -7. 谨慎的模式和非常友好的恢复，在谨慎模式下，多个FileAppender实例跑在多个JVM下，能 够安全地写道同一个日志文件。RollingFileAppender会有些限制。Logback的FileAppender和它的子类包括 RollingFileAppender能够非常友好地从I/O异常中恢复。 -8. 配置文件可以处理不同的情况，开发人员经常需要判断不同的Logback配置文件在不同的环境下（开发，测试，生产）。而这些配置文件仅仅只有一些很小的不同，可以通过,和来实现，这样一个配置文件就可以适应多个环境。 -9. Filters（过滤器）有些时候，需要诊断一个问题，需要打出日志。在log4j，只有降低日志级别，不过这样会打出大量的日志，会影响应用性能。在Logback，你可以继续 保持那个日志级别而除掉某种特殊情况，如alice这个用户登录，她的日志将打在DEBUG级别而其他用户可以继续打在WARN级别。要实现这个功能只需加4行XML配置。可以参考MDCFIlter 。 -10. SiftingAppender（一个非常多功能的Appender）：它可以用来分割日志文件根据任何一个给定的运行参数。如，SiftingAppender能够区别日志事件跟进用户的Session，然后每个用户会有一个日志文件。 -11. 自动压缩已经打出来的log：RollingFileAppender在产生新文件的时候，会自动压缩已经打出来的日志文件。压缩是个异步过程，所以甚至对于大的日志文件，在压缩过程中应用不会受任何影响。 -12. 堆栈树带有包版本：Logback在打出堆栈树日志时，会带上包的数据。 -13. 自动去除旧的日志文件：通过设置TimeBasedRollingPolicy或者SizeAndTimeBasedFNATP的maxHistory属性，你可以控制已经产生日志文件的最大数量。如果设置maxHistory 12，那那些log文件超过12个月的都会被自动移除。 - -## 三、logback的配置介绍 - -##### 　　1、Logger、appender及layout - -Logger作为日志的记录器，把它关联到应用的对应的context上后，主要用于存放日志对象，也可以定义日志类型、级别。 - -Appender主要用于指定日志输出的目的地，目的地可以是控制台、文件、远程套接字服务器、 MySQL、PostreSQL、 Oracle和其他数据库、 JMS和远程UNIX Syslog守护进程等。 - -Layout 负责把事件转换成字符串，格式化的日志信息的输出。 - -##### 　　2、logger context - -各个logger 都被关联到一个 LoggerContext，LoggerContext负责制造logger，也负责以树结构排列各logger。其他所有logger也通过org.slf4j.LoggerFactory 类的静态方法getLogger取得。 getLogger方法以 logger名称为参数。用同一名字调用LoggerFactory.getLogger 方法所得到的永远都是同一个logger对象的引用。 - -##### 　　3、有效级别及级别的继承 - -Logger 可以被分配级别。级别包括：TRACE、DEBUG、INFO、WARN 和 ERROR，定义于ch.qos.logback.classic.Level类。如果 logger没有被分配级别，那么它将从有被分配级别的最近的祖先那里继承级别。root logger 默认级别是 DEBUG。 - -##### 　　4、打印方法与基本的选择规则 - -打印方法决定记录请求的级别。例如，如果 L 是一个 logger 实例，那么，语句 L.info("..")是一条级别为 INFO的记录语句。记录请求的级别在高于或等于其 logger 的有效级别时被称为被启用，否则，称为被禁用。记录请求级别为 p，其 logger的有效级别为 q，只有则当 p>=q时，该请求才会被执行。 - -该规则是 logback 的核心。级别排序为： TRACE < DEBUG < INFO < WARN < ERROR - -## 四、logback的默认配置 - -如果配置文件 logback-test.xml 和 logback.xml 都不存在，那么 logback 默认地会调用BasicConfigurator ，创建一个最小化配置。最小化配置由一个关联到根 logger 的ConsoleAppender 组成。输出用模式为%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n 的 PatternLayoutEncoder 进行格式化。root logger 默认级别是 DEBUG。 - -##### 　　1、Logback的配置文件 - -Logback 配置文件的语法非常灵活。正因为灵活，所以无法用 DTD 或 XML schema 进行定义。尽管如此，可以这样描述配置文件的基本结构：以\开头，后面有零个或多个\元素，有零个或多个\元素，有最多一个\元素。 - -##### 　　2、Logback默认配置的步骤 - -(1). 尝试在 classpath下查找文件logback-test.xml； - -(2). 如果文件不存在，则查找文件logback.xml； - -(3). 如果两个文件都不存在，logback用BasicConfigurator自动对自己进行配置，这会导致记录输出到控制台。 - -## 五、logback.xml常用配置详解 - - - -![]( http://ftp.bmp.ovh/imgs/2019/11/b8471fc5d526e6a4.png ) - -1、根节点<configuration>，包含下面三个属性： - -- scan: 当此属性设置为true时，配置文件如果发生改变，将会被重新加载，默认值为true。 - -- scanPeriod: 设置监测配置文件是否有修改的时间间隔，如果没有给出时间单位，默认单位是毫秒。当scan为true时，此属性生效。默认的时间间隔为1分钟。 - -- debug: 当此属性设置为true时，将打印出logback内部日志信息，实时查看logback运行状态。默认值为false。　　 - -```xml -` - - -``` - -2、子节点<contextName>：用来设置上下文名称，每个logger都关联到logger上下文，默认上下文名称为default。但可以使用<contextName>设置成其他名字，用于区分不同应用程序的记录。一旦设置，不能修改。 - -```xml - - - myAppName - - -``` - -3、子节点<property> ：用来定义变量值，它有两个属性name和value，通过<property>定义的值会被插入到logger上下文中，可以使“${}”来使用变量。 - -name: 变量的名称 - -value: 的值时变量定义的值　 - -```xml - - - ${APP_Name} - - - -``` - -4、子节点<timestamp>：获取时间戳字符串，他有两个属性key和datePattern - -- key: 标识此<timestamp> 的名字； - -- datePattern: 设置将当前时间（解析配置文件的时间）转换为字符串的模式，遵循java.txt.SimpleDateFormat的格式。 - -```xml - - - ${bySecond} - - -``` - -5、子节点<appender>：负责写日志的组件，它有两个必要属性name和class。name指定appender名称，class指定appender的全限定名 - -5.1、ConsoleAppender 把日志输出到控制台，有以下子节点： - -- <encoder>：对日志进行格式化。（具体参数稍后讲解 ） - -- <target>：字符串System.out(默认)或者System.err（区别不多说了）　 - -```xml - - - - %-4relative [%thread] %-5level %logger{35} - %msg %n - - - - - - -``` - -上述配置表示把>=DEBUG级别的日志都输出到控制台 - -5.2、FileAppender：把日志添加到文件，有以下子节点： - -- <file>：被写入的文件名，可以是相对目录，也可以是绝对目录，如果上级目录不存在会自动创建，没有默认值。 - -- <append>：如果是 true，日志被追加到文件结尾，如果是 false，清空现存文件，默认是true。 - -- <encoder>：对记录事件进行格式化。（具体参数稍后讲解 ） - -- <prudent>：如果是 true，日志会被安全的写入文件，即使其他的FileAppender也在向此文件做写入操作，效率低，默认是 false。 - -```xml - - - testFile.log - true - - %-4relative [%thread] %-5level %logger{35} - %msg%n - - - - - - -``` - -上述配置表示把>=DEBUG级别的日志都输出到testFile.log - -5.3、RollingFileAppender：滚动记录文件，先将日志记录到指定文件，当符合某个条件时，将日志记录到其他文件。有以下子节点： - -- <file>：被写入的文件名，可以是相对目录，也可以是绝对目录，如果上级目录不存在会自动创建，没有默认值。 - -- <append>：如果是 true，日志被追加到文件结尾，如果是 false，清空现存文件，默认是true。 - -- <rollingPolicy>:当发生滚动时，决定RollingFileAppender的行为，涉及文件移动和重命名。属性class定义具体的滚动策略类 - -`class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"`： 最常用的滚动策略，它根据时间来制定滚动策略，既负责滚动也负责出发滚动。有以下子节点： - -- <fileNamePattern>：必要节点，包含文件名及“%d”转换符，“%d”可以包含一个`java.text.SimpleDateFormat`指定的时间格式，如：%d{yyyy-MM}。 - - 如果直接使用 %d，默认格式是 yyyy-MM-dd。RollingFileAppender的file字节点可有可无，通过设置file，可以为活动文件和归档文件指定不同位置，当前日志总是记录到file指定的文件（活动文件），活动文件的名字不会改变；如果没设置file，活动文件的名字会根据fileNamePattern 的值，每隔一段时间改变一次。“/”或者“\”会被当做目录分隔符。 - -- <maxHistory>:可选节点，控制保留的归档文件的最大数量，超出数量就删除旧文件。假设设置每个月滚动，且\是6，则只保存最近6个月的文件，删除之前的旧文件。注意，删除旧文件是，那些为了归档而创建的目录也会被删除。 - - `class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy"`： 查看当前活动文件的大小，如果超过指定大小会告知RollingFileAppender 触发当前活动文件滚动。只有一个节点: - -- <maxFileSize>:这是活动文件的大小，默认值是10MB。 - -- <prudent>：当为true时，不支持FixedWindowRollingPolicy。支持TimeBasedRollingPolicy，但是有两个限制，1不支持也不允许文件压缩，2不能设置file属性，必须留空。 - -- <triggeringPolicy >: 告知 RollingFileAppender 合适激活滚动。 - -`class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy`根据固定窗口算法重命名文件的滚动策略。有以下子节点： - -- <minIndex>:窗口索引最小值 - -- <maxIndex>:窗口索引最大值，当用户指定的窗口过大时，会自动将窗口设置为12。 - -- <fileNamePattern>:必须包含“%i”例如，假设最小值和最大值分别为1和2，命名模式为 mylog%i.log,会产生归档文件mylog1.log和mylog2.log。还可以指定文件压缩选项，例如，mylog%i.log.gz 或者 没有log%i.log.zip　　 - -```xml - - - - logFile.%d{yyyy-MM-dd}.log - 30 - - - %-4relative [%thread] %-5level %logger{35} - %msg%n - - - - - - -``` - -上述配置表示每天生成一个日志文件，保存30天的日志文件。 - -```xml - - - test.log - - - tests.%i.log.zip - 1 - 3 - - - - 5MB - - - - %-4relative [%thread] %-5level %logger{35} - %msg%n - - - - - - - -``` - -上述配置表示按照固定窗口模式生成日志文件，当文件大于20MB时，生成新的日志文件。窗口大小是1到3，当保存了3个归档文件后，将覆盖最早的日志。 - -<encoder>：对记录事件进行格式化。负责两件事，一是把日志信息转换成字节数组，二是把字节数组写入到输出流。 - -PatternLayoutEncoder 是唯一有用的且默认的encoder ，有一个\节点，用来设置日志的输入格式。使用“%”加“转换符”方式，如果要输出“%”，则必须用“\”对“\%”进行转义。 - -5.4、还有SocketAppender、SMTPAppender、DBAppender、SyslogAppender、SiftingAppender，并不常用，这里就不详解了。 - -6、子节点<loger>：用来设置某一个包或具体的某一个类的日志打印级别、以及指定<appender>。<loger>仅有一个name属性，一个可选的level和一个可选的addtivity属性。 - -可以包含零个或多个<appender-ref>元素，标识这个appender将会添加到这个loger - -name: 用来指定受此loger约束的某一个包或者具体的某一个类，可以自定义。 - -level: 用来设置打印级别，大小写无关：**TRACE**, **DEBUG**, **INFO**, **WARN**, **ERROR**, **ALL**和**OFF**，还有一个特殊值INHERITED或者同义词NULL，代表强制执行上级的级别。 如果未设置此属性，那么当前loger将会继承上级的级别。 - -addtivity: 是否向上级loger传递打印信息。默认是true。同<loger>一样，可以包含零个或多个<appender-ref>元素，标识这个appender将会添加到这个loger。 - -7、子节点<root>:它也是<loger>元素，但是它是根loger,是所有<loger>的上级。只有一个level属性，因为name已经被命名为"root",且已经是最上级了。 - -level: 用来设置打印级别，**大小写无关**：TRACE, DEBUG, INFO, WARN, ERROR, ALL和OFF，不能设置为INHERITED或者同义词NULL。 默认是DEBUG。 - -## 六、Demo - -1、添加依赖包logback使用需要和slf4j一起使用，所以总共需要添加依赖的包有slf4j-api - -logback使用需要和slf4j一起使用，所以总共需要添加依赖的包有slf4j-api.jar，logback-core.jar，logback-classic.jar，logback-access.jar这个暂时用不到所以不添加依赖了，maven配置 - -```xml - -　　　　UTF-8 -　　　　1.1.7 -　　　　1.7.21 - -　　 -　　 -　　　　 -　　　　　　org.slf4j -　　　　　　slf4j-api -　　　　　　${slf4j.version} -　　　　　　compile -　　　　 - -　　　　 -　　　　　　ch.qos.logback -　　　　　　logback-core -　　　　　　${logback.version} -　　　　 - -　　　　 -　　　　　　ch.qos.logback -　　　　　　logback-classic -　　　　　　${logback.version} -　　　　　　 -　　 -``` - -2、logback.xml配置 - -```xml - - - - - - - - - - - - - - - - -%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{50} - %msg%n - - - - - - - - - - - - -${LOG_HOME}/TestWeb.log.%d{yyyy-MM-dd}.log - - - -30 - - - - - - - -%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{50} - %msg%n - - - - - - -10MB -5GB - - - - - - - - - - - - -``` - -3、java代码 - -```java -public class App { - -　　private final static Logger logger = LoggerFactory.getLogger(App.class); - -　　　　public static void main(String[] args) { -　　　　　　logger.info("logback 成功了"); -　　　　　　logger.error("logback 成功了"); -　　　　　　logger.debug("logback 成功了"); - -　　　　} -　　} -``` - -4、输出 - -![img](http://ftp.bmp.ovh/imgs/2019/11/67a5dfcedbb94712.png) - -## 七、总结 - + + （英文手册） + +## 一、logback介绍 + +Logback是由log4j创始人设计的另一个开源日志组件,官方网站： http://logback.qos.ch。用来继承代替log4j，目前logback有3个模块： + +logback-core：其它两个模块的基础模块 + +logback-classic：它是log4j的一个改良版本，同时它完整实现了slf4j API使你可以很方便地更换成其它日志系统如log4j或JDK14 Logging + +logback-access：访问模块，与Servlet容器集成提供通过Http来访问日志的功能 + +## 二、logback取代log4j的理由： + +1. 更快的实现：Logback的内核重写了，在一些关键执行路径上性能提升10倍以上。而且logback不仅性能提升了，初始化内存加载也更小了。 +2. 非常充分的测试：Logback经过了几年，数不清小时的测试。Logback的测试完全不同级别的。 +3. Logback-classic非常自然实现了SLF4j：Logback-classic实现了SLF4j。在使用SLF4j中，你都感觉不到logback-classic。而且因为logback-classic非常自然地实现了slf4j ， 所 以切换到log4j或者其他，非常容易，只需要提供成另一个jar包就OK，根本不需要去动那些通过SLF4JAPI实现的代码。 +4. 非常充分的文档 官方网站有两百多页的文档。 +5. 自动重新加载配置文件，当配置文件修改了，Logback-classic能自动重新加载配置文件。扫描过程快且安全，它并不需要另外创建一个扫描线程。这个技术充分保证了应用程序能跑得很欢在JEE环境里面。 +6. Lilith是log事件的观察者，和log4j的chainsaw类似。而lilith还能处理大数量的log数据 。 +7. 谨慎的模式和非常友好的恢复，在谨慎模式下，多个FileAppender实例跑在多个JVM下，能 够安全地写道同一个日志文件。RollingFileAppender会有些限制。Logback的FileAppender和它的子类包括 RollingFileAppender能够非常友好地从I/O异常中恢复。 +8. 配置文件可以处理不同的情况，开发人员经常需要判断不同的Logback配置文件在不同的环境下（开发，测试，生产）。而这些配置文件仅仅只有一些很小的不同，可以通过,和来实现，这样一个配置文件就可以适应多个环境。 +9. Filters（过滤器）有些时候，需要诊断一个问题，需要打出日志。在log4j，只有降低日志级别，不过这样会打出大量的日志，会影响应用性能。在Logback，你可以继续 保持那个日志级别而除掉某种特殊情况，如alice这个用户登录，她的日志将打在DEBUG级别而其他用户可以继续打在WARN级别。要实现这个功能只需加4行XML配置。可以参考MDCFIlter 。 +10. SiftingAppender（一个非常多功能的Appender）：它可以用来分割日志文件根据任何一个给定的运行参数。如，SiftingAppender能够区别日志事件跟进用户的Session，然后每个用户会有一个日志文件。 +11. 自动压缩已经打出来的log：RollingFileAppender在产生新文件的时候，会自动压缩已经打出来的日志文件。压缩是个异步过程，所以甚至对于大的日志文件，在压缩过程中应用不会受任何影响。 +12. 堆栈树带有包版本：Logback在打出堆栈树日志时，会带上包的数据。 +13. 自动去除旧的日志文件：通过设置TimeBasedRollingPolicy或者SizeAndTimeBasedFNATP的maxHistory属性，你可以控制已经产生日志文件的最大数量。如果设置maxHistory 12，那那些log文件超过12个月的都会被自动移除。 + +## 三、logback的配置介绍 + +##### 　　1、Logger、appender及layout + +Logger作为日志的记录器，把它关联到应用的对应的context上后，主要用于存放日志对象，也可以定义日志类型、级别。 + +Appender主要用于指定日志输出的目的地，目的地可以是控制台、文件、远程套接字服务器、 MySQL、PostreSQL、 Oracle和其他数据库、 JMS和远程UNIX Syslog守护进程等。 + +Layout 负责把事件转换成字符串，格式化的日志信息的输出。 + +##### 　　2、logger context + +各个logger 都被关联到一个 LoggerContext，LoggerContext负责制造logger，也负责以树结构排列各logger。其他所有logger也通过org.slf4j.LoggerFactory 类的静态方法getLogger取得。 getLogger方法以 logger名称为参数。用同一名字调用LoggerFactory.getLogger 方法所得到的永远都是同一个logger对象的引用。 + +##### 　　3、有效级别及级别的继承 + +Logger 可以被分配级别。级别包括：TRACE、DEBUG、INFO、WARN 和 ERROR，定义于ch.qos.logback.classic.Level类。如果 logger没有被分配级别，那么它将从有被分配级别的最近的祖先那里继承级别。root logger 默认级别是 DEBUG。 + +##### 　　4、打印方法与基本的选择规则 + +打印方法决定记录请求的级别。例如，如果 L 是一个 logger 实例，那么，语句 L.info("..")是一条级别为 INFO的记录语句。记录请求的级别在高于或等于其 logger 的有效级别时被称为被启用，否则，称为被禁用。记录请求级别为 p，其 logger的有效级别为 q，只有则当 p>=q时，该请求才会被执行。 + +该规则是 logback 的核心。级别排序为： TRACE < DEBUG < INFO < WARN < ERROR + +## 四、logback的默认配置 + +如果配置文件 logback-test.xml 和 logback.xml 都不存在，那么 logback 默认地会调用BasicConfigurator ，创建一个最小化配置。最小化配置由一个关联到根 logger 的ConsoleAppender 组成。输出用模式为%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n 的 PatternLayoutEncoder 进行格式化。root logger 默认级别是 DEBUG。 + +##### 　　1、Logback的配置文件 + +Logback 配置文件的语法非常灵活。正因为灵活，所以无法用 DTD 或 XML schema 进行定义。尽管如此，可以这样描述配置文件的基本结构：以\开头，后面有零个或多个\元素，有零个或多个\元素，有最多一个\元素。 + +##### 　　2、Logback默认配置的步骤 + +(1). 尝试在 classpath下查找文件logback-test.xml； + +(2). 如果文件不存在，则查找文件logback.xml； + +(3). 如果两个文件都不存在，logback用BasicConfigurator自动对自己进行配置，这会导致记录输出到控制台。 + +## 五、logback.xml常用配置详解 + + + +![]( http://ftp.bmp.ovh/imgs/2019/11/b8471fc5d526e6a4.png ) + +1、根节点<configuration>，包含下面三个属性： + +- scan: 当此属性设置为true时，配置文件如果发生改变，将会被重新加载，默认值为true。 + +- scanPeriod: 设置监测配置文件是否有修改的时间间隔，如果没有给出时间单位，默认单位是毫秒。当scan为true时，此属性生效。默认的时间间隔为1分钟。 + +- debug: 当此属性设置为true时，将打印出logback内部日志信息，实时查看logback运行状态。默认值为false。　　 + +```xml +` + + +``` + +2、子节点<contextName>：用来设置上下文名称，每个logger都关联到logger上下文，默认上下文名称为default。但可以使用<contextName>设置成其他名字，用于区分不同应用程序的记录。一旦设置，不能修改。 + +```xml + + + myAppName + + +``` + +3、子节点<property> ：用来定义变量值，它有两个属性name和value，通过<property>定义的值会被插入到logger上下文中，可以使“${}”来使用变量。 + +name: 变量的名称 + +value: 的值时变量定义的值　 + +```xml + + + ${APP_Name} + + + +``` + +4、子节点<timestamp>：获取时间戳字符串，他有两个属性key和datePattern + +- key: 标识此<timestamp> 的名字； + +- datePattern: 设置将当前时间（解析配置文件的时间）转换为字符串的模式，遵循java.txt.SimpleDateFormat的格式。 + +```xml + + + ${bySecond} + + +``` + +5、子节点<appender>：负责写日志的组件，它有两个必要属性name和class。name指定appender名称，class指定appender的全限定名 + +5.1、ConsoleAppender 把日志输出到控制台，有以下子节点： + +- <encoder>：对日志进行格式化。（具体参数稍后讲解 ） + +- <target>：字符串System.out(默认)或者System.err（区别不多说了）　 + +```xml + + + + %-4relative [%thread] %-5level %logger{35} - %msg %n + + + + + + +``` + +上述配置表示把>=DEBUG级别的日志都输出到控制台 + +5.2、FileAppender：把日志添加到文件，有以下子节点： + +- <file>：被写入的文件名，可以是相对目录，也可以是绝对目录，如果上级目录不存在会自动创建，没有默认值。 + +- <append>：如果是 true，日志被追加到文件结尾，如果是 false，清空现存文件，默认是true。 + +- <encoder>：对记录事件进行格式化。（具体参数稍后讲解 ） + +- <prudent>：如果是 true，日志会被安全的写入文件，即使其他的FileAppender也在向此文件做写入操作，效率低，默认是 false。 + +```xml + + + testFile.log + true + + %-4relative [%thread] %-5level %logger{35} - %msg%n + + + + + + +``` + +上述配置表示把>=DEBUG级别的日志都输出到testFile.log + +5.3、RollingFileAppender：滚动记录文件，先将日志记录到指定文件，当符合某个条件时，将日志记录到其他文件。有以下子节点： + +- <file>：被写入的文件名，可以是相对目录，也可以是绝对目录，如果上级目录不存在会自动创建，没有默认值。 + +- <append>：如果是 true，日志被追加到文件结尾，如果是 false，清空现存文件，默认是true。 + +- <rollingPolicy>:当发生滚动时，决定RollingFileAppender的行为，涉及文件移动和重命名。属性class定义具体的滚动策略类 + +`class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"`： 最常用的滚动策略，它根据时间来制定滚动策略，既负责滚动也负责出发滚动。有以下子节点： + +- <fileNamePattern>：必要节点，包含文件名及“%d”转换符，“%d”可以包含一个`java.text.SimpleDateFormat`指定的时间格式，如：%d{yyyy-MM}。 + + 如果直接使用 %d，默认格式是 yyyy-MM-dd。RollingFileAppender的file字节点可有可无，通过设置file，可以为活动文件和归档文件指定不同位置，当前日志总是记录到file指定的文件（活动文件），活动文件的名字不会改变；如果没设置file，活动文件的名字会根据fileNamePattern 的值，每隔一段时间改变一次。“/”或者“\”会被当做目录分隔符。 + +- <maxHistory>:可选节点，控制保留的归档文件的最大数量，超出数量就删除旧文件。假设设置每个月滚动，且\是6，则只保存最近6个月的文件，删除之前的旧文件。注意，删除旧文件是，那些为了归档而创建的目录也会被删除。 + + `class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy"`： 查看当前活动文件的大小，如果超过指定大小会告知RollingFileAppender 触发当前活动文件滚动。只有一个节点: + +- <maxFileSize>:这是活动文件的大小，默认值是10MB。 + +- <prudent>：当为true时，不支持FixedWindowRollingPolicy。支持TimeBasedRollingPolicy，但是有两个限制，1不支持也不允许文件压缩，2不能设置file属性，必须留空。 + +- <triggeringPolicy >: 告知 RollingFileAppender 合适激活滚动。 + +`class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy`根据固定窗口算法重命名文件的滚动策略。有以下子节点： + +- <minIndex>:窗口索引最小值 + +- <maxIndex>:窗口索引最大值，当用户指定的窗口过大时，会自动将窗口设置为12。 + +- <fileNamePattern>:必须包含“%i”例如，假设最小值和最大值分别为1和2，命名模式为 mylog%i.log,会产生归档文件mylog1.log和mylog2.log。还可以指定文件压缩选项，例如，mylog%i.log.gz 或者 没有log%i.log.zip　　 + +```xml + + + + logFile.%d{yyyy-MM-dd}.log + 30 + + + %-4relative [%thread] %-5level %logger{35} - %msg%n + + + + + + +``` + +上述配置表示每天生成一个日志文件，保存30天的日志文件。 + +```xml + + + test.log + + + tests.%i.log.zip + 1 + 3 + + + + 5MB + + + + %-4relative [%thread] %-5level %logger{35} - %msg%n + + + + + + + +``` + +上述配置表示按照固定窗口模式生成日志文件，当文件大于20MB时，生成新的日志文件。窗口大小是1到3，当保存了3个归档文件后，将覆盖最早的日志。 + +<encoder>：对记录事件进行格式化。负责两件事，一是把日志信息转换成字节数组，二是把字节数组写入到输出流。 + +PatternLayoutEncoder 是唯一有用的且默认的encoder ，有一个\节点，用来设置日志的输入格式。使用“%”加“转换符”方式，如果要输出“%”，则必须用“\”对“\%”进行转义。 + +5.4、还有SocketAppender、SMTPAppender、DBAppender、SyslogAppender、SiftingAppender，并不常用，这里就不详解了。 + +6、子节点<loger>：用来设置某一个包或具体的某一个类的日志打印级别、以及指定<appender>。<loger>仅有一个name属性，一个可选的level和一个可选的addtivity属性。 + +可以包含零个或多个<appender-ref>元素，标识这个appender将会添加到这个loger + +name: 用来指定受此loger约束的某一个包或者具体的某一个类，可以自定义。 + +level: 用来设置打印级别，大小写无关：**TRACE**, **DEBUG**, **INFO**, **WARN**, **ERROR**, **ALL**和**OFF**，还有一个特殊值INHERITED或者同义词NULL，代表强制执行上级的级别。 如果未设置此属性，那么当前loger将会继承上级的级别。 + +addtivity: 是否向上级loger传递打印信息。默认是true。同<loger>一样，可以包含零个或多个<appender-ref>元素，标识这个appender将会添加到这个loger。 + +7、子节点<root>:它也是<loger>元素，但是它是根loger,是所有<loger>的上级。只有一个level属性，因为name已经被命名为"root",且已经是最上级了。 + +level: 用来设置打印级别，**大小写无关**：TRACE, DEBUG, INFO, WARN, ERROR, ALL和OFF，不能设置为INHERITED或者同义词NULL。 默认是DEBUG。 + +## 六、Demo + +1、添加依赖包logback使用需要和slf4j一起使用，所以总共需要添加依赖的包有slf4j-api + +logback使用需要和slf4j一起使用，所以总共需要添加依赖的包有slf4j-api.jar，logback-core.jar，logback-classic.jar，logback-access.jar这个暂时用不到所以不添加依赖了，maven配置 + +```xml + +　　　　UTF-8 +　　　　1.1.7 +　　　　1.7.21 + +　　 +　　 +　　　　 +　　　　　　org.slf4j +　　　　　　slf4j-api +　　　　　　${slf4j.version} +　　　　　　compile +　　　　 + +　　　　 +　　　　　　ch.qos.logback +　　　　　　logback-core +　　　　　　${logback.version} +　　　　 + +　　　　 +　　　　　　ch.qos.logback +　　　　　　logback-classic +　　　　　　${logback.version} +　　　　　　 +　　 +``` + +2、logback.xml配置 + +```xml + + + + + + + + + + + + + + + + +%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{50} - %msg%n + + + + + + + + + + + + +${LOG_HOME}/TestWeb.log.%d{yyyy-MM-dd}.log + + + +30 + + + + + + + +%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{50} - %msg%n + + + + + + +10MB +5GB + + + + + + + + + + + + +``` + +3、java代码 + +```java +public class App { + +　　private final static Logger logger = LoggerFactory.getLogger(App.class); + +　　　　public static void main(String[] args) { +　　　　　　logger.info("logback 成功了"); +　　　　　　logger.error("logback 成功了"); +　　　　　　logger.debug("logback 成功了"); + +　　　　} +　　} +``` + +4、输出 + +![img](http://ftp.bmp.ovh/imgs/2019/11/67a5dfcedbb94712.png) + +## 七、总结 + （英文手册） + +## 一、logback介绍 + +Logback是由log4j创始人设计的另一个开源日志组件,官方网站： http://logback.qos.ch。用来继承代替log4j，目前logback有3个模块： + +logback-core：其它两个模块的基础模块 + +logback-classic：它是log4j的一个改良版本，同时它完整实现了slf4j API使你可以很方便地更换成其它日志系统如log4j或JDK14 Logging + +logback-access：访问模块，与Servlet容器集成提供通过Http来访问日志的功能 + +## 二、logback取代log4j的理由： + +1. 更快的实现：Logback的内核重写了，在一些关键执行路径上性能提升10倍以上。而且logback不仅性能提升了，初始化内存加载也更小了。 +2. 非常充分的测试：Logback经过了几年，数不清小时的测试。Logback的测试完全不同级别的。 +3. Logback-classic非常自然实现了SLF4j：Logback-classic实现了SLF4j。在使用SLF4j中，你都感觉不到logback-classic。而且因为logback-classic非常自然地实现了slf4j ， 所 以切换到log4j或者其他，非常容易，只需要提供成另一个jar包就OK，根本不需要去动那些通过SLF4JAPI实现的代码。 +4. 非常充分的文档 官方网站有两百多页的文档。 +5. 自动重新加载配置文件，当配置文件修改了，Logback-classic能自动重新加载配置文件。扫描过程快且安全，它并不需要另外创建一个扫描线程。这个技术充分保证了应用程序能跑得很欢在JEE环境里面。 +6. Lilith是log事件的观察者，和log4j的chainsaw类似。而lilith还能处理大数量的log数据 。 +7. 谨慎的模式和非常友好的恢复，在谨慎模式下，多个FileAppender实例跑在多个JVM下，能 够安全地写道同一个日志文件。RollingFileAppender会有些限制。Logback的FileAppender和它的子类包括 RollingFileAppender能够非常友好地从I/O异常中恢复。 +8. 配置文件可以处理不同的情况，开发人员经常需要判断不同的Logback配置文件在不同的环境下（开发，测试，生产）。而这些配置文件仅仅只有一些很小的不同，可以通过,和来实现，这样一个配置文件就可以适应多个环境。 +9. Filters（过滤器）有些时候，需要诊断一个问题，需要打出日志。在log4j，只有降低日志级别，不过这样会打出大量的日志，会影响应用性能。在Logback，你可以继续 保持那个日志级别而除掉某种特殊情况，如alice这个用户登录，她的日志将打在DEBUG级别而其他用户可以继续打在WARN级别。要实现这个功能只需加4行XML配置。可以参考MDCFIlter 。 +10. SiftingAppender（一个非常多功能的Appender）：它可以用来分割日志文件根据任何一个给定的运行参数。如，SiftingAppender能够区别日志事件跟进用户的Session，然后每个用户会有一个日志文件。 +11. 自动压缩已经打出来的log：RollingFileAppender在产生新文件的时候，会自动压缩已经打出来的日志文件。压缩是个异步过程，所以甚至对于大的日志文件，在压缩过程中应用不会受任何影响。 +12. 堆栈树带有包版本：Logback在打出堆栈树日志时，会带上包的数据。 +13. 自动去除旧的日志文件：通过设置TimeBasedRollingPolicy或者SizeAndTimeBasedFNATP的maxHistory属性，你可以控制已经产生日志文件的最大数量。如果设置maxHistory 12，那那些log文件超过12个月的都会被自动移除。 + +## 三、logback的配置介绍 + +##### 　　1、Logger、appender及layout + +Logger作为日志的记录器，把它关联到应用的对应的context上后，主要用于存放日志对象，也可以定义日志类型、级别。 + +Appender主要用于指定日志输出的目的地，目的地可以是控制台、文件、远程套接字服务器、 MySQL、PostreSQL、 Oracle和其他数据库、 JMS和远程UNIX Syslog守护进程等。 + +Layout 负责把事件转换成字符串，格式化的日志信息的输出。 + +##### 　　2、logger context + +各个logger 都被关联到一个 LoggerContext，LoggerContext负责制造logger，也负责以树结构排列各logger。其他所有logger也通过org.slf4j.LoggerFactory 类的静态方法getLogger取得。 getLogger方法以 logger名称为参数。用同一名字调用LoggerFactory.getLogger 方法所得到的永远都是同一个logger对象的引用。 + +##### 　　3、有效级别及级别的继承 + +Logger 可以被分配级别。级别包括：TRACE、DEBUG、INFO、WARN 和 ERROR，定义于ch.qos.logback.classic.Level类。如果 logger没有被分配级别，那么它将从有被分配级别的最近的祖先那里继承级别。root logger 默认级别是 DEBUG。 + +##### 　　4、打印方法与基本的选择规则 + +打印方法决定记录请求的级别。例如，如果 L 是一个 logger 实例，那么，语句 L.info("..")是一条级别为 INFO的记录语句。记录请求的级别在高于或等于其 logger 的有效级别时被称为被启用，否则，称为被禁用。记录请求级别为 p，其 logger的有效级别为 q，只有则当 p>=q时，该请求才会被执行。 + +该规则是 logback 的核心。级别排序为： TRACE < DEBUG < INFO < WARN < ERROR + +## 四、logback的默认配置 + +如果配置文件 logback-test.xml 和 logback.xml 都不存在，那么 logback 默认地会调用BasicConfigurator ，创建一个最小化配置。最小化配置由一个关联到根 logger 的ConsoleAppender 组成。输出用模式为%d{HH:mm:ss.SSS} [%thread] %-5level %logger{36} - %msg%n 的 PatternLayoutEncoder 进行格式化。root logger 默认级别是 DEBUG。 + +##### 　　1、Logback的配置文件 + +Logback 配置文件的语法非常灵活。正因为灵活，所以无法用 DTD 或 XML schema 进行定义。尽管如此，可以这样描述配置文件的基本结构：以\开头，后面有零个或多个\元素，有零个或多个\元素，有最多一个\元素。 + +##### 　　2、Logback默认配置的步骤 + +(1). 尝试在 classpath下查找文件logback-test.xml； + +(2). 如果文件不存在，则查找文件logback.xml； + +(3). 如果两个文件都不存在，logback用BasicConfigurator自动对自己进行配置，这会导致记录输出到控制台。 + +## 五、logback.xml常用配置详解 + + + +![]( http://ftp.bmp.ovh/imgs/2019/11/b8471fc5d526e6a4.png ) + +1、根节点<configuration>，包含下面三个属性： + +- scan: 当此属性设置为true时，配置文件如果发生改变，将会被重新加载，默认值为true。 + +- scanPeriod: 设置监测配置文件是否有修改的时间间隔，如果没有给出时间单位，默认单位是毫秒。当scan为true时，此属性生效。默认的时间间隔为1分钟。 + +- debug: 当此属性设置为true时，将打印出logback内部日志信息，实时查看logback运行状态。默认值为false。　　 + +```xml +` + + +``` + +2、子节点<contextName>：用来设置上下文名称，每个logger都关联到logger上下文，默认上下文名称为default。但可以使用<contextName>设置成其他名字，用于区分不同应用程序的记录。一旦设置，不能修改。 + +```xml + + + myAppName + + +``` + +3、子节点<property> ：用来定义变量值，它有两个属性name和value，通过<property>定义的值会被插入到logger上下文中，可以使“${}”来使用变量。 + +name: 变量的名称 + +value: 的值时变量定义的值　 + +```xml + + + ${APP_Name} + + + +``` + +4、子节点<timestamp>：获取时间戳字符串，他有两个属性key和datePattern + +- key: 标识此<timestamp> 的名字； + +- datePattern: 设置将当前时间（解析配置文件的时间）转换为字符串的模式，遵循java.txt.SimpleDateFormat的格式。 + +```xml + + + ${bySecond} + + +``` + +5、子节点<appender>：负责写日志的组件，它有两个必要属性name和class。name指定appender名称，class指定appender的全限定名 + +5.1、ConsoleAppender 把日志输出到控制台，有以下子节点： + +- <encoder>：对日志进行格式化。（具体参数稍后讲解 ） + +- <target>：字符串System.out(默认)或者System.err（区别不多说了）　 + +```xml + + + + %-4relative [%thread] %-5level %logger{35} - %msg %n + + + + + + +``` + +上述配置表示把>=DEBUG级别的日志都输出到控制台 + +5.2、FileAppender：把日志添加到文件，有以下子节点： + +- <file>：被写入的文件名，可以是相对目录，也可以是绝对目录，如果上级目录不存在会自动创建，没有默认值。 + +- <append>：如果是 true，日志被追加到文件结尾，如果是 false，清空现存文件，默认是true。 + +- <encoder>：对记录事件进行格式化。（具体参数稍后讲解 ） + +- <prudent>：如果是 true，日志会被安全的写入文件，即使其他的FileAppender也在向此文件做写入操作，效率低，默认是 false。 + +```xml + + + testFile.log + true + + %-4relative [%thread] %-5level %logger{35} - %msg%n + + + + + + +``` + +上述配置表示把>=DEBUG级别的日志都输出到testFile.log + +5.3、RollingFileAppender：滚动记录文件，先将日志记录到指定文件，当符合某个条件时，将日志记录到其他文件。有以下子节点： + +- <file>：被写入的文件名，可以是相对目录，也可以是绝对目录，如果上级目录不存在会自动创建，没有默认值。 + +- <append>：如果是 true，日志被追加到文件结尾，如果是 false，清空现存文件，默认是true。 + +- <rollingPolicy>:当发生滚动时，决定RollingFileAppender的行为，涉及文件移动和重命名。属性class定义具体的滚动策略类 + +`class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"`： 最常用的滚动策略，它根据时间来制定滚动策略，既负责滚动也负责出发滚动。有以下子节点： + +- <fileNamePattern>：必要节点，包含文件名及“%d”转换符，“%d”可以包含一个`java.text.SimpleDateFormat`指定的时间格式，如：%d{yyyy-MM}。 + + 如果直接使用 %d，默认格式是 yyyy-MM-dd。RollingFileAppender的file字节点可有可无，通过设置file，可以为活动文件和归档文件指定不同位置，当前日志总是记录到file指定的文件（活动文件），活动文件的名字不会改变；如果没设置file，活动文件的名字会根据fileNamePattern 的值，每隔一段时间改变一次。“/”或者“\”会被当做目录分隔符。 + +- <maxHistory>:可选节点，控制保留的归档文件的最大数量，超出数量就删除旧文件。假设设置每个月滚动，且\是6，则只保存最近6个月的文件，删除之前的旧文件。注意，删除旧文件是，那些为了归档而创建的目录也会被删除。 + + `class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy"`： 查看当前活动文件的大小，如果超过指定大小会告知RollingFileAppender 触发当前活动文件滚动。只有一个节点: + +- <maxFileSize>:这是活动文件的大小，默认值是10MB。 + +- <prudent>：当为true时，不支持FixedWindowRollingPolicy。支持TimeBasedRollingPolicy，但是有两个限制，1不支持也不允许文件压缩，2不能设置file属性，必须留空。 + +- <triggeringPolicy >: 告知 RollingFileAppender 合适激活滚动。 + +`class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy`根据固定窗口算法重命名文件的滚动策略。有以下子节点： + +- <minIndex>:窗口索引最小值 + +- <maxIndex>:窗口索引最大值，当用户指定的窗口过大时，会自动将窗口设置为12。 + +- <fileNamePattern>:必须包含“%i”例如，假设最小值和最大值分别为1和2，命名模式为 mylog%i.log,会产生归档文件mylog1.log和mylog2.log。还可以指定文件压缩选项，例如，mylog%i.log.gz 或者 没有log%i.log.zip　　 + +```xml + + + + logFile.%d{yyyy-MM-dd}.log + 30 + + + %-4relative [%thread] %-5level %logger{35} - %msg%n + + + + + + +``` + +上述配置表示每天生成一个日志文件，保存30天的日志文件。 + +```xml + + + test.log + + + tests.%i.log.zip + 1 + 3 + + + + 5MB + + + + %-4relative [%thread] %-5level %logger{35} - %msg%n + + + + + + + +``` + +上述配置表示按照固定窗口模式生成日志文件，当文件大于20MB时，生成新的日志文件。窗口大小是1到3，当保存了3个归档文件后，将覆盖最早的日志。 + +<encoder>：对记录事件进行格式化。负责两件事，一是把日志信息转换成字节数组，二是把字节数组写入到输出流。 + +PatternLayoutEncoder 是唯一有用的且默认的encoder ，有一个\节点，用来设置日志的输入格式。使用“%”加“转换符”方式，如果要输出“%”，则必须用“\”对“\%”进行转义。 + +5.4、还有SocketAppender、SMTPAppender、DBAppender、SyslogAppender、SiftingAppender，并不常用，这里就不详解了。 + +6、子节点<loger>：用来设置某一个包或具体的某一个类的日志打印级别、以及指定<appender>。<loger>仅有一个name属性，一个可选的level和一个可选的addtivity属性。 + +可以包含零个或多个<appender-ref>元素，标识这个appender将会添加到这个loger + +name: 用来指定受此loger约束的某一个包或者具体的某一个类，可以自定义。 + +level: 用来设置打印级别，大小写无关：**TRACE**, **DEBUG**, **INFO**, **WARN**, **ERROR**, **ALL**和**OFF**，还有一个特殊值INHERITED或者同义词NULL，代表强制执行上级的级别。 如果未设置此属性，那么当前loger将会继承上级的级别。 + +addtivity: 是否向上级loger传递打印信息。默认是true。同<loger>一样，可以包含零个或多个<appender-ref>元素，标识这个appender将会添加到这个loger。 + +7、子节点<root>:它也是<loger>元素，但是它是根loger,是所有<loger>的上级。只有一个level属性，因为name已经被命名为"root",且已经是最上级了。 + +level: 用来设置打印级别，**大小写无关**：TRACE, DEBUG, INFO, WARN, ERROR, ALL和OFF，不能设置为INHERITED或者同义词NULL。 默认是DEBUG。 + +## 六、Demo + +1、添加依赖包logback使用需要和slf4j一起使用，所以总共需要添加依赖的包有slf4j-api + +logback使用需要和slf4j一起使用，所以总共需要添加依赖的包有slf4j-api.jar，logback-core.jar，logback-classic.jar，logback-access.jar这个暂时用不到所以不添加依赖了，maven配置 + +```xml + +　　　　UTF-8 +　　　　1.1.7 +　　　　1.7.21 + +　　 +　　 +　　　　 +　　　　　　org.slf4j +　　　　　　slf4j-api +　　　　　　${slf4j.version} +　　　　　　compile +　　　　 + +　　　　 +　　　　　　ch.qos.logback +　　　　　　logback-core +　　　　　　${logback.version} +　　　　 + +　　　　 +　　　　　　ch.qos.logback +　　　　　　logback-classic +　　　　　　${logback.version} +　　　　　　 +　　 +``` + +2、logback.xml配置 + +```xml + + + + + + + + + + + + + + + + +%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{50} - %msg%n + + + + + + + + + + + + +${LOG_HOME}/TestWeb.log.%d{yyyy-MM-dd}.log + + + +30 + + + + + + + +%d{yyyy-MM-dd HH:mm:ss.SSS} [%thread] %-5level %logger{50} - %msg%n + + + + + + +10MB +5GB + + + + + + + + + + + + +``` + +3、java代码 + +```java +public class App { + +　　private final static Logger logger = LoggerFactory.getLogger(App.class); + +　　　　public static void main(String[] args) { +　　　　　　logger.info("logback 成功了"); +　　　　　　logger.error("logback 成功了"); +　　　　　　logger.debug("logback 成功了"); + +　　　　} +　　} +``` + +4、输出 + +![img](http://ftp.bmp.ovh/imgs/2019/11/67a5dfcedbb94712.png) + +## 七、总结 + +43189e7cc0106d720da3f9609e511e9a80722ed0 logback的配置，需要配置输出源appender，打日志的loger（子节点）和root（根节点），实际上，它输出日志是从子节点开始，子节点如果有输出源直接输入，如果无，判断配置的addtivity，是否像上级传递，即是否向root传递，传递则采用root的输出源，否则不输出日志。 \ No newline at end of file diff --git "a/docs/framework/\346\236\266\346\236\204.md" "b/docs/framework/\346\236\266\346\236\204.md" new file mode 100755 index 0000000000..28ec8eb3c2 --- /dev/null +++ "b/docs/framework/\346\236\266\346\236\204.md" @@ -0,0 +1,10 @@ +## 架构设计流程 + +### 架构设计第 1 步：识别复杂度 + +架构设计的本质目的是为了解决软件系统的复杂性，所以在我们设计架构时，首先就要分析系统的复杂性。只有正确分析出了系统的复杂性，后续的架构设计方案才不会偏离方向；否则，如果对系统的复杂性判断错误，即使后续的架构设计方案再完美再先进，都是南辕北辙，做的越好，错的越多、越离谱。 + +架构的复杂度主要来源于“高性能”“高可用”“可扩展”等几个方面，但架构师在具体判断复杂性的时候，不能生搬硬套，认为任何时候架构都必须同时满足这三方面的要求。 + + + diff --git a/docs/index.html b/docs/index.html deleted file mode 100644 index bff5451aed..0000000000 --- a/docs/index.html +++ /dev/null @@ -1,100 +0,0 @@ - - - - - - - - - JavaKeeper - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/docs/interview/.DS_Store b/docs/interview/.DS_Store new file mode 100644 index 0000000000..7685a105dd Binary files /dev/null and b/docs/interview/.DS_Store differ diff --git a/docs/interview/Ability-FAQ.md b/docs/interview/Ability-FAQ.md new file mode 100644 index 0000000000..9fd7f3fa15 --- /dev/null +++ b/docs/interview/Ability-FAQ.md @@ -0,0 +1,2688 @@ +--- +title: 架构能力八股 +date: 2024-12-19 +tags: + - Java + - SystemDesign + - Architecture + - Microservices + - HighConcurrency + - Distributed + - Performance + - Interview +categories: Interview +--- + +![](https://img.starfish.ink/common/faq-banner.png) + +> 技术二面一般是展现**系统设计能力**和**架构思维**的关键环节，从**设计模式**到**分布式架构**，从**高并发处理**到**业务场景设计**，每一项技术都考验着工程师的**实战经验**和**技术深度**。本文档将**最常考的技术二面题目**整理成**标准话术**，涵盖系统设计、架构演进、问题排查等核心领域，助你在技术二面中脱颖而出！ + +--- + +## 🗺️ 知识导航 + +### 🏷️ 核心知识分类 + +1. **🏗️ 设计模式与架构**：可插拔规则引擎、责任链模式、事件驱动架构、单例模式、工厂模式 +2. **🔧 项目实战与问题排查**：线上问题定位、性能调优、内存泄漏、死锁处理、故障恢复 +3. **💼 业务场景与系统设计**：秒杀系统、系统扩容、大促准备、任务调度、实时风控、多租户SaaS、智能推荐、日志收集、异地多活 +4. **⚡ 高并发系统设计**：50w QPS系统、电商大促、直播系统、社交APP、搜索引擎 +5. **🌊 分布式系统设计**：短链系统、邮件系统、即时通讯、秒杀系统、配置中心 +6. **🚀 性能优化与调优**：接口超时定位、CPU占用排查、死锁处理、第三方接口降级 +7. **🏛️ 架构思维与技术治理**：技术选型、微服务治理、容错设计、技术债务管理、监控体系 + +### 🔑 面试话术模板 + +| **问题类型** | **回答框架** | **关键要点** | **深入扩展** | +| ------------ | ----------------------------------- | ------------------ | ------------------ | +| **系统设计** | 需求分析→架构设计→技术选型→性能优化 | 架构合理性，扩展性 | 技术细节，最佳实践 | +| **问题排查** | 现象确认→定位思路→解决方案→预防措施 | 系统性思维，工具使用 | 根因分析，经验总结 | +| **架构演进** | 现状分析→演进策略→实施计划→风险控制 | 技术决策，团队协作 | 技术债务，持续改进 | +| **业务场景** | 场景分析→技术挑战→解决方案→效果验证 | 业务理解，技术实现 | 性能优化，监控告警 | + +--- + +## 🏗️ 一、设计模式与架构 + +> **核心思想**：设计模式是软件设计的经验总结，通过合理的模式选择可以提升代码的可维护性、可扩展性和复用性。 + +### 🎯 如何设计一个可插拔的规则引擎？ + +设计思路： + +- 核心采用 **策略模式/链式责任/规则表达式**，把规则以模块形式组织，支持动态加载。 +- 规则可以以脚本（Groovy、JS）、DSL、或预编译类的形式存储在数据库/配置中心，运行时装载并缓存。 +- 附加功能：规则版本管理、回滚、规则测试环境、并行评估引擎、规则隔离（sandbox）以防注入风险。 +- 性能优化：规则编译（避免每次解释）、缓存规则查询结果、对规则执行进行限时保护。 + 工程化要点是"可观测、可回滚、可审计"。 + **考察点：** 模块化设计、可插拔实现细节与安全考虑。 + **常见追问：** 为什么用脚本而不是硬编码？（答：脚本允许业务方动态调整规则，减少发布成本） + +### 🎯 如何用责任链模式设计订单审批流程？ + +可以把每个审批节点封装成一个处理器（Handler），Handler 实现统一接口并链式连接：每个 Handler 判断是否通过，若通过则传递给下一个 Handler，否则返回审批失败。优点是可动态插拔审批节点、可在运行时调整流程、易做权限与审核日志记录。注意异步审批/超时处理与补偿逻辑。 +**考察点：** 设计模式在业务建模中的应用能力。 +**常见追问：** 如何对审批节点做并行处理？（答：将并行节点拆成并发执行，最后做汇总决策） + +### 🎯 如何把事件驱动架构（EDA）落地到大项目？ + +实践要点： + +- 选择合适的消息中间件（Kafka/RabbitMQ），Kafka 常用于高吞吐与持久化场景。 +- **事件建模**（确定事件粒度、Schema、版本化），使用 schema registry 管控兼容性。 +- **幂等与消费侧安全**：消费者需设计幂等、去重与事务（Outbox pattern）。 +- **监控与追踪**：链路追踪、消费延迟监控、消息堆积报警。 +- **演化策略**：事件版本升级策略、灰度回放能力。 + 初始先从业务边界清晰、耦合高的场景切入（如订单/库存），逐步演化。 + **考察点：** EDA 的实践性考虑（schema、幂等、监控、回放）。 + **常见追问：** 如何回溯历史事件？（答：需要消息持久化并提供重放机制） + +### 🎯 单例模式在并发下如何正确实现？有坑么？ + +推荐做法： + +- 最简单且安全的方式是使用 **枚举单例**（Java 的枚举天生避免反序列化、反射问题）。 +- 另一常用方式是 **静态内部类**（懒加载且线程安全）。 +- 可用 **双重检查锁** + `volatile` 实现延迟加载，注意 `volatile` 防止指令重排序。 + 陷阱包括反射/序列化破坏单例、类加载器隔离导致单例多实例等。 + **考察点：** 并发下资源初始化与 Java 特性。 + **常见追问：** 为什么枚举单例最安全？（答：JVM 保证枚举只会被实例化一次，并防止反射创建新的实例） + +### 🎯 工厂模式如何扩展第三方支付接入？ + +采用抽象工厂或策略模式：定义统一的支付接口（创建订单、签名、回调验证），每个第三方支付实现该接口并注册到工厂中；工厂根据配置或参数返回对应实现。好处是新增支付渠道只需实现接口并配置即可，降低耦合。再加上配置化加载、熔断和限流等治理逻辑，最终构成可扩展的支付接入平台。 +**考察点：** 可扩展性、接口设计、注册/发现机制。 +**常见追问：** 如何做支付幂等？（答：订单号+幂等表/唯一索引或幂等 key） + +------ + +## 🔧 二、项目实战与问题排查 + +### 🎯 线上服务突然出现大量500错误，如何快速定位？ + +**应急响应流程**：快速止血 -> 定位根因 -> 修复问题 -> 复盘改进。 + +**立即行动**： + +1. **确认影响范围**： + - **监控大盘**：查看错误率、QPS、响应时间变化 + - **用户反馈**：客服渠道、社交媒体用户反馈情况 + - **业务影响**：核心功能是否受影响，影响用户量 + - **时间节点**：确认问题开始时间和发布时间关系 + +2. **快速止血措施**： + - **流量切换**：将流量切换到正常的机房或集群 + - **服务降级**：关闭非核心功能，保证核心链路 + - **限流熔断**：启用限流和熔断机制保护系统 + - **回滚考虑**：如果是新发布导致，考虑快速回滚 + +**问题定位步骤**： + +**第一层：应用层排查** + +```bash +# 1. 查看应用日志 +tail -f /var/log/app/error.log | grep ERROR + +# 2. 检查JVM状态 +jstat -gc -h10 1s +jmap -heap + +# 3. 线程堆栈分析 +jstack > thread_dump.log +``` + +**第二层：系统资源排查** + +```bash +# 1. 系统资源 +top -p # CPU和内存使用 +iostat -x 1 # 磁盘IO +netstat -i # 网络状况 + +# 2. 文件句柄 +lsof -p | wc -l # 打开文件数 +ulimit -n # 文件句柄限制 +``` + +**第三层：依赖服务排查** + +- **数据库**：慢查询日志、连接数、锁等待 +- **缓存**：Redis连接数、命中率、内存使用 +- **外部接口**：第三方API响应时间和错误率 +- **消息队列**：队列积压、消费延迟 + +**常见问题及处理**： + +**场景1：内存溢出（OOM）** + +``` +症状：500错误 + JVM重启 + GC频繁 +排查：jmap -dump + MAT分析内存泄漏 +处理：扩大堆内存 + 修复泄漏代码 +``` + +**场景2：数据库连接池耗尽** + +``` +症状：连接超时异常 + DB连接数达到上限 +排查：SHOW PROCESSLIST查看活跃连接 +处理：扩大连接池 + 优化慢查询 + 连接泄漏修复 +``` + +**场景3：外部依赖超时** + +``` +症状：接口超时 + 特定错误码 +排查：调用链分析 + 外部服务状态确认 +处理：增加超时时间 + 熔断降级 + 重试机制 +``` + +**恢复验证**： + +- **监控指标**：错误率恢复到正常水平 +- **业务验证**：核心功能正常，用户反馈减少 +- **性能验证**：响应时间、QPS恢复正常 +- **持续观察**：24小时持续监控确保稳定 + + **考察点：** 应急响应能力、问题定位思路、系统分析技能。 + **常见追问：** 如何预防类似问题？（答：监控完善+压测+灰度发布+故障演练） + +### 🎯 系统内存使用率持续上升，怎么排查内存泄漏？ + +**内存泄漏特征**：内存使用持续增长、Full GC频繁但内存不下降、最终导致OOM。 + +**排查工具和方法**： + +**第一步：监控分析** + +```bash +# 1. JVM内存监控 +jstat -gc -h10 5s # 观察GC情况 +jstat -gccapacity # 查看堆容量 + +# 2. 系统内存监控 +ps aux | grep java # 进程内存使用 +free -h # 系统内存情况 +``` + +**第二步：堆内存分析** + +```bash +# 1. 生成堆转储 +jmap -dump:live,format=b,file=heap.dump + +# 2. 查看堆内存分布 +jmap -histo | head -20 + +# 3. 强制GC观察 +jmap -gc +``` + +**第三步：MAT分析堆转储** + +- **Leak Suspects Report**：自动发现可能的内存泄漏 +- **Dominator Tree**：查看占用内存最大的对象 +- **Histogram**：按类统计对象数量和大小 +- **OQL查询**：编写查询语句分析特定对象 + +**常见内存泄漏模式**： + +**1. 集合类未清理** + +```java +// 问题代码 +private static Map cache = new HashMap<>(); + +public void addCache(String key, Object value) { + cache.put(key, value); // 只添加不清理 +} + +// 解决方案 +private static Map cache = new ConcurrentHashMap<>(); + +@Scheduled(fixedRate = 300000) // 5分钟清理一次 +public void cleanExpiredCache() { + cache.entrySet().removeIf(entry -> isExpired(entry)); +} +``` + +**2. ThreadLocal未清理** + +```java +// 问题代码 +private static ThreadLocal userContext = new ThreadLocal<>(); + +// 解决方案 +try { + userContext.set(user); + // 业务逻辑 +} finally { + userContext.remove(); // 必须清理 +} +``` + +**3. 监听器未移除** + +```java +// 问题代码：注册监听器但未移除 +eventBus.register(listener); + +// 解决方案：及时移除 +@PreDestroy +public void cleanup() { + eventBus.unregister(listener); +} +``` + +**4. 数据库连接泄漏** + +```java +// 问题代码 +Connection conn = dataSource.getConnection(); +// 业务逻辑但没有关闭连接 + +// 解决方案 +try (Connection conn = dataSource.getConnection()) { + // 业务逻辑 +} // 自动关闭连接 +``` + +**非堆内存泄漏排查**： + +**Metaspace泄漏**： + +```bash +# 监控Metaspace使用 +jstat -gc + +# 查看类加载情况 +jstat -class + +# 分析类加载器 +jcmd VM.classloader_stats +``` + +**直接内存泄漏**： + +```bash +# 监控直接内存（NIO Buffer） +jcmd VM.classloader_stats + +# 分析内存映射文件 +lsof -p | grep deleted +``` + +**代码层面预防措施**： + +```java +// 1. 使用弱引用缓存 +private static Map> cache = + new ConcurrentHashMap<>(); + +// 2. 定时清理机制 +@Scheduled(fixedRate = 60000) +public void cleanupCache() { + cache.entrySet().removeIf(entry -> + entry.getValue().get() == null); +} + +// 3. 大对象及时清理 +try { + List bigList = new ArrayList<>(); + // 处理逻辑 +} finally { + bigList.clear(); // 显式清理 + bigList = null; +} +``` + +**系统层面监控**： + +- **内存使用趋势**：持续监控内存使用率 +- **GC频率监控**：Full GC频率和耗时监控 +- **告警机制**：内存使用率超过80%时告警 +- **自动重启**：OOM时自动重启机制 + + **考察点：** 内存管理知识、问题排查能力、代码质量意识。 + **常见追问：** 如何在生产环境安全地生成堆转储？（答：使用-dump:live减少影响，选择低峰期执行） + +### 🎯 数据库查询突然变慢，如何快速优化？ + +**数据库性能问题排查流程**：监控确认 -> 定位慢查询 -> 分析执行计划 -> 优化实施 -> 效果验证。 + +**第一步：确认性能问题** + +```sql +-- 1. 查看当前活跃连接 +SHOW PROCESSLIST; + +-- 2. 查看数据库状态 +SHOW STATUS LIKE 'Threads%'; +SHOW STATUS LIKE 'Questions'; +SHOW STATUS LIKE 'Slow_queries'; + +-- 3. 查看锁等待情况 +SELECT * FROM information_schema.INNODB_LOCKS; +SELECT * FROM information_schema.INNODB_LOCK_WAITS; +``` + +**第二步：定位慢查询** + +```sql +-- 1. 开启慢查询日志 +SET GLOBAL slow_query_log = 'ON'; +SET GLOBAL long_query_time = 0.1; -- 0.1秒以上的查询 + +-- 2. 查看当前慢查询 +SELECT * FROM information_schema.PROCESSLIST +WHERE COMMAND != 'Sleep' AND TIME > 0.1; + +-- 3. 分析慢查询日志 +-- 使用mysqldumpslow分析日志文件 +mysqldumpslow -s t -t 10 /var/log/mysql/slow.log +``` + +**第三步：分析执行计划** + +```sql +-- 1. EXPLAIN分析 +EXPLAIN SELECT * FROM orders +WHERE user_id = 12345 AND order_date > '2024-01-01'; + +-- 2. 关注关键指标 +-- type: ALL(全表扫描)最差，index > range > ref > const最好 +-- key: 使用的索引 +-- rows: 预估扫描行数 +-- Extra: Using temporary、Using filesort等需要优化 +``` + +**常见问题及优化策略**： + +**场景1：缺少索引** + +```sql +-- 问题查询 +SELECT * FROM orders WHERE user_id = 12345 AND status = 'PAID'; +-- EXPLAIN显示：type=ALL, rows=1000000 + +-- 解决方案：创建复合索引 +CREATE INDEX idx_user_status ON orders(user_id, status); + +-- 验证效果 +EXPLAIN SELECT * FROM orders WHERE user_id = 12345 AND status = 'PAID'; +-- 优化后：type=ref, rows=100 +``` + +**场景2：索引失效** + +```sql +-- 问题查询：函数导致索引失效 +SELECT * FROM orders WHERE DATE(order_date) = '2024-01-01'; + +-- 解决方案：避免在索引列上使用函数 +SELECT * FROM orders +WHERE order_date >= '2024-01-01 00:00:00' +AND order_date < '2024-01-02 00:00:00'; +``` + +**场景3：锁等待问题** + +```sql +-- 1. 查找锁等待 +SELECT + r.trx_id waiting_trx_id, + r.trx_mysql_thread_id waiting_thread, + b.trx_id blocking_trx_id, + b.trx_mysql_thread_id blocking_thread +FROM information_schema.innodb_lock_waits w +INNER JOIN information_schema.innodb_trx b ON b.trx_id = w.blocking_trx_id +INNER JOIN information_schema.innodb_trx r ON r.trx_id = w.requesting_trx_id; + +-- 2. 分析死锁 +SHOW ENGINE INNODB STATUS; +``` + +**第四步：实施优化** + +**索引优化**： + +```sql +-- 1. 创建合适的索引 +CREATE INDEX idx_order_date ON orders(order_date); +CREATE INDEX idx_user_status_date ON orders(user_id, status, order_date); + +-- 2. 删除冗余索引 +DROP INDEX idx_redundant ON orders; + +-- 3. 覆盖索引优化 +CREATE INDEX idx_cover ON orders(user_id, order_date, status, amount); +``` + +**查询重写优化**： + +```sql +-- 优化前：子查询 +SELECT * FROM users +WHERE id IN (SELECT user_id FROM orders WHERE amount > 1000); + +-- 优化后：JOIN +SELECT DISTINCT u.* FROM users u +INNER JOIN orders o ON u.id = o.user_id +WHERE o.amount > 1000; + +-- 优化前：LIMIT深度分页 +SELECT * FROM orders ORDER BY id LIMIT 100000, 20; + +-- 优化后：游标分页 +SELECT * FROM orders WHERE id > 100000 ORDER BY id LIMIT 20; +``` + +**配置参数优化**： + +```sql +-- 1. 缓冲池大小（推荐70-80%物理内存） +SET GLOBAL innodb_buffer_pool_size = 8G; + +-- 2. 查询缓存 +SET GLOBAL query_cache_size = 256M; +SET GLOBAL query_cache_type = 1; + +-- 3. 连接数 +SET GLOBAL max_connections = 1000; + +-- 4. 临时表大小 +SET GLOBAL tmp_table_size = 256M; +SET GLOBAL max_heap_table_size = 256M; +``` + +**第五步：效果验证** + +```sql +-- 1. 再次EXPLAIN验证 +EXPLAIN SELECT * FROM orders WHERE user_id = 12345; + +-- 2. 性能对比 +-- 优化前：执行时间 500ms +-- 优化后：执行时间 50ms + +-- 3. 监控指标 +SHOW STATUS LIKE 'Handler_read%'; +SHOW STATUS LIKE 'Select%'; +``` + +**应急处理措施**： + +- **KILL慢查询**：KILL QUERY +- **增加连接数**：临时增加max_connections +- **读写分离**：将读请求路由到从库 +- **缓存预热**：将热点数据加载到缓存 + + **考察点：** 数据库优化能力、SQL调优技巧、性能分析思路。 + **常见追问：** 索引过多有什么问题？（答：影响写性能、占用存储空间、维护成本高） + +------ + + + +## 💼 三、业务场景与系统设计 + +> **核心思想**：业务场景设计题考察工程师将技术能力应用到具体业务场景的能力，需要深入理解业务需求，设计出既满足业务要求又具备良好技术架构的解决方案。 + +### 🎯 从零设计一个秒杀系统（可直接面试话术） + +**高层思路**：要兼顾高并发吞吐、低延迟、可用性、最终一致性与防刷。设计分为前端保护层、缓存层、异步队列与落库保证四个部分： + +1. **入口限流/防刷**：在 CDN/网关层做全量限流、用户级限流、验证码或签名机制（避免机器人）。 +2. **缓存预热**：在秒杀开始前把库存预热到 Redis，避免直接打 DB。 +3. **原子扣减**：使用 Redis Lua 脚本做库存判断与预扣，返回 token 给用户。Lua 保证判断与扣减原子性。 +4. **异步下单**：将下单请求放入 MQ，消费者异步落库并做最后库存确认与幂等处理（使用唯一索引或幂等表）。 +5. **最终一致性**：定期对库存做校验/对账，出现差异做补偿。 +6. **监控与回退**：监控队列长度、消费延迟、错误率，必要时快速降级或关闭秒杀。 + **关键保障**：Redis 原子操作避免大部分超卖，MQ 异步处理保证 DB 不被瞬时流量打垮，幂等设计保证消息重试安全。 + **实现细节与注意事项**：防刷、去重、用户限购、日志埋点、全链路追踪、回放与补偿。 + **考察点：** 从架构到细节的完整性以及对一致性/性能的权衡。 + +### 🎯 如何将系统从 1 万 TPS 扩展到 10 万 TPS？（面试话术） + +扩容分层次： + +- **无状态服务水平扩展**：保证服务无状态或把状态外置（Redis/DB），通过负载均衡扩容实例。 +- **拆分瓶颈**：使用异步 MQ 解耦、增加分区与消费者并行度、把 CPU 密集型部分优化或下沉到批处理。 +- **数据层分库分表**：水平拆分数据库与读写分离，热点表做缓存或 CQRS。 +- **缓存与 CDN**：尽可能把读请求命中缓存，减少数据库压力。 +- **连接池/网络优化**：减少同步阻塞、保持长连接并优化序列化（例如 Protobuf）。 +- **垂直/水平分割业务**：拆分单体服务到微服务，按流量和业务分片扩容。 + 关键是定位并解决系统瓶颈（CPU/IO/锁/GC/网络），并做好灰度、容量测试与回滚方案。 + **考察点：** 扩展策略与性能瓶颈识别能力。 + **常见追问：** 单点扩展中最容易忽视的问题？（答：数据库连接数、网络带宽与中间件并发限制） + +### 🎯 双 11 类型大促前如何准备（面试话术） + +主要工作：容量与容错准备、性能测试、依赖切换与降级策略、运维预案。具体： + +- **压测与容量评估**：按预估流量做分层压测（全链路），找并修复瓶颈。 +- **预热与缓存**：预先把热数据加载到缓存/CDN，避免冷启动。 +- **配置开关与灰度**：支持快速关闭非关键功能，分步骤放量。 +- **降级与熔断策略**：为非核心服务建立降级逻辑与回退数据。 +- **演练故障**：演练 DB/Redis/消息队列故障切换、回滚流程与补偿机制。 +- **监控与报警**：关键指标（QPS/RT/错误率/队列长度/DB slow）必须有实时告警与自动化处理脚本。 +- **人力准备**：运维、SRE、后端保障团队待命并有明确责任分工。 + **考察点：** 大促级别运维、压测与应急机制的成熟度。 + +### 🎯 如何设计一个高可用的分布式日志收集系统？ + +常见架构：日志采集 -> Fluentd/Logstash -> Kafka（缓冲）-> 消费者（索引到 Elasticsearch/Hadoop）-> 可视化（Kibana）。关键设计点： + +- **高可用收集**：采集层做本地缓冲和批量发送。 +- **可靠缓冲**：Kafka 做持久化缓冲，避免短时峰值丢失。 +- **索引与归档**：ES 用于快速检索，历史日志落到冷存储（HDFS）做归档。 +- **结构化与规范**：统一日志格式（JSON + schema），便于解析与搜索。 +- **管控 & 限流**：对日志流量做采样、限流以保护下游系统。 +- **权限与审计**：管理访问控制和审计日志，保证安全合规。 + **考察点：** 可观测性与大数据流处理设计能力。 + +### 🎯 如何实现异地多活（Active-Active）系统？ + +异地多活需要解决数据同步、冲突解决、全局流量路由与延迟问题： + +- **流量层**：采用全球负载均衡（DNS + Anycast + GSLB），根据用户地理/延迟路由到最近活跃数据中心。 +- **数据同步**：采用异步双向复制（跨 DC），并设计冲突解决策略（CRDT、业务级冲突检测或基于时间戳的合并）。 +- **一致性模型**：多数场景选择最终一致性，强一致场景需走中心化主写或 Paxos/Raft 跨域协议（复杂且性能差）。 +- **回退与演练**：必须有故障切换与回滚机制，以及跨 DC 的演练。 +- **监控 & 延迟容忍**：关注跨域延迟与队列积压，设计降级逻辑。 + 异地多活适用于对可用性要求极高的业务，但实现复杂、运维成本高，要按成本收益评估。 + **考察点：** 分布式系统的深层次挑战与工程化能力。 + +### 🎯 设计一个分布式任务调度系统 + +**业务需求分析**：支持大规模定时任务调度、高可用、动态调整、监控告警。 + +**核心功能模块**： + +1. **任务管理**：任务创建、编辑、删除、启停控制 +2. **调度引擎**：定时触发、依赖调度、失败重试 +3. **执行器管理**：多机器负载均衡、故障转移 +4. **监控告警**：任务状态监控、失败告警、性能统计 +5. **权限管理**：用户权限、操作审计 + +**系统架构设计**： + +**分层架构**： + +``` +前端控制台 -> API网关 -> 调度中心 -> 执行器集群 + ↓ + 数据存储层 +``` + +**调度中心设计**： + +```java +@Component +public class TaskScheduler { + + @Autowired + private TaskRepository taskRepository; + + @Autowired + private ExecutorRegistry executorRegistry; + + @Scheduled(fixedRate = 1000) // 每秒扫描一次 + public void scanAndTriggerTasks() { + List readyTasks = taskRepository.findReadyTasks(Instant.now()); + + for (Task task : readyTasks) { + try { + // 选择执行器 + Executor executor = selectExecutor(task); + + // 分发任务 + TaskExecution execution = TaskExecution.builder() + .taskId(task.getId()) + .executorId(executor.getId()) + .triggerTime(Instant.now()) + .status(ExecutionStatus.RUNNING) + .build(); + + // 异步执行 + CompletableFuture.supplyAsync(() -> { + return executor.execute(task); + }).whenComplete((result, ex) -> { + updateExecutionResult(execution, result, ex); + }); + + } catch (Exception e) { + log.error("任务调度失败: {}", task.getId(), e); + handleTaskFailure(task, e); + } + } + } + + private Executor selectExecutor(Task task) { + // 负载均衡策略：轮询、随机、最少活跃数 + List availableExecutors = executorRegistry.getAvailableExecutors(); + return loadBalancer.select(availableExecutors, task); + } +} +``` + +**执行器注册与心跳**： + +```java +@Service +public class ExecutorRegistry { + + private final Map executors = new ConcurrentHashMap<>(); + + public void registerExecutor(ExecutorInfo executor) { + executors.put(executor.getId(), executor); + log.info("执行器注册成功: {}", executor.getId()); + } + + @Scheduled(fixedRate = 10000) // 每10秒检查一次 + public void checkExecutorHealth() { + Iterator> iterator = executors.entrySet().iterator(); + + while (iterator.hasNext()) { + Map.Entry entry = iterator.next(); + ExecutorInfo executor = entry.getValue(); + + // 检查心跳超时（超过30秒） + if (Duration.between(executor.getLastHeartbeat(), Instant.now()).getSeconds() > 30) { + iterator.remove(); + log.warn("执行器下线: {}", executor.getId()); + + // 重新分配该执行器上的任务 + reassignTasks(executor.getId()); + } + } + } +} +``` + +**分布式锁防重复执行**： + +```java +@Service +public class DistributedTaskLock { + + @Autowired + private RedisTemplate redisTemplate; + + public boolean tryLock(String taskId, String instanceId, long timeoutSeconds) { + String lockKey = "task:lock:" + taskId; + String lockValue = instanceId + ":" + System.currentTimeMillis(); + + Boolean result = redisTemplate.opsForValue() + .setIfAbsent(lockKey, lockValue, Duration.ofSeconds(timeoutSeconds)); + + return Boolean.TRUE.equals(result); + } + + public void releaseLock(String taskId, String instanceId) { + String lockKey = "task:lock:" + taskId; + String lockValue = instanceId + ":" + System.currentTimeMillis(); + + String script = """ + if redis.call('get', KEYS[1]) == ARGV[1] then + return redis.call('del', KEYS[1]) + else + return 0 + end + """; + + redisTemplate.execute(RedisScript.of(script, Long.class), + Collections.singletonList(lockKey), lockValue); + } +} +``` + +**任务依赖调度**： + +```java +@Service +public class TaskDependencyResolver { + + public boolean canExecute(Task task) { + List dependencies = task.getDependencies(); + + for (TaskDependency dependency : dependencies) { + Task dependentTask = taskRepository.findById(dependency.getDependentTaskId()); + + // 检查依赖任务是否在指定时间窗口内成功执行 + TaskExecution lastExecution = getLastExecution(dependentTask.getId()); + + if (lastExecution == null || + lastExecution.getStatus() != ExecutionStatus.SUCCESS || + !isInTimeWindow(lastExecution, dependency.getTimeWindow())) { + return false; + } + } + + return true; + } + + private boolean isInTimeWindow(TaskExecution execution, TimeWindow window) { + Instant executionTime = execution.getEndTime(); + Instant windowStart = window.getStartTime(); + Instant windowEnd = window.getEndTime(); + + return executionTime.isAfter(windowStart) && executionTime.isBefore(windowEnd); + } +} +``` + +**监控与告警**： + +```java +@Component +public class TaskMonitor { + + @EventListener + public void handleTaskFailure(TaskFailureEvent event) { + Task task = event.getTask(); + + // 更新失败计数 + task.incrementFailureCount(); + + // 根据重试策略决定是否重试 + if (shouldRetry(task)) { + scheduleRetry(task); + } else { + // 发送告警 + alertService.sendAlert(AlertType.TASK_FAILURE, + "任务执行失败: " + task.getName(), task); + } + } + + @Scheduled(fixedRate = 60000) // 每分钟统计一次 + public void collectMetrics() { + TaskMetrics metrics = TaskMetrics.builder() + .totalTasks(taskRepository.count()) + .runningTasks(getRunningTaskCount()) + .successRate(calculateSuccessRate()) + .avgExecutionTime(calculateAvgExecutionTime()) + .timestamp(Instant.now()) + .build(); + + metricsRepository.save(metrics); + } +} +``` + + **考察点：** 分布式调度、高可用设计、负载均衡、监控告警。 + **常见追问：** 如何保证任务不丢失？（答：持久化存储+分布式锁+故障转移+补偿机制） + +### 🎯 设计一个实时风控系统 + +**业务场景分析**：金融、电商等场景的实时风险识别与控制，毫秒级响应。 + +**核心挑战**： + +1. **实时性要求**：毫秒级风险判断，不能影响用户体验 +2. **规则复杂性**：多维度规则组合，动态调整策略 +3. **高并发处理**：支持大规模并发风险评估 +4. **准确性保证**：降低误判率，平衡安全与体验 + +**系统架构设计**： + +**实时流处理架构**： + +``` +事件接入 -> 数据预处理 -> 规则引擎 -> 决策输出 -> 行动执行 + ↓ ↓ ↓ ↓ ↓ +消息队列 特征提取 规则缓存 决策记录 风控措施 +``` + +**规则引擎设计**： + +```java +@Component +public class RiskRuleEngine { + + @Autowired + private RuleRepository ruleRepository; + + @Autowired + private FeatureService featureService; + + public RiskAssessmentResult evaluate(RiskEvent event) { + // 1. 特征提取 + Map features = featureService.extractFeatures(event); + + // 2. 获取适用规则 + List applicableRules = getApplicableRules(event.getScenario()); + + // 3. 规则评估 + List ruleResults = new ArrayList<>(); + int totalScore = 0; + + for (RiskRule rule : applicableRules) { + RuleResult result = evaluateRule(rule, features); + ruleResults.add(result); + + if (result.isTriggered()) { + totalScore += rule.getScore(); + + // 如果是阻断规则，直接返回 + if (rule.getAction() == RuleAction.BLOCK) { + return RiskAssessmentResult.blocked(rule.getId(), result.getReason()); + } + } + } + + // 4. 决策逻辑 + RiskLevel riskLevel = calculateRiskLevel(totalScore); + RiskAction action = determineAction(riskLevel, event.getScenario()); + + return RiskAssessmentResult.builder() + .riskLevel(riskLevel) + .action(action) + .score(totalScore) + .ruleResults(ruleResults) + .evaluationTime(Instant.now()) + .build(); + } + + private RuleResult evaluateRule(RiskRule rule, Map features) { + try { + // 使用规则表达式引擎（如Aviator、MVEL） + Boolean result = ruleEvaluator.evaluate(rule.getExpression(), features); + + return RuleResult.builder() + .ruleId(rule.getId()) + .triggered(Boolean.TRUE.equals(result)) + .reason(rule.getDescription()) + .build(); + + } catch (Exception e) { + log.error("规则评估异常: {}", rule.getId(), e); + return RuleResult.failed(rule.getId(), e.getMessage()); + } + } +} +``` + +**特征工程服务**： + +```java +@Service +public class FeatureService { + + @Autowired + private RedisTemplate redisTemplate; + + public Map extractFeatures(RiskEvent event) { + Map features = new HashMap<>(); + + // 基础特征 + features.put("userId", event.getUserId()); + features.put("deviceId", event.getDeviceId()); + features.put("ip", event.getIpAddress()); + features.put("amount", event.getAmount()); + features.put("timestamp", event.getTimestamp()); + + // 统计特征（近期行为统计） + addStatisticalFeatures(features, event); + + // 设备指纹特征 + addDeviceFingerprintFeatures(features, event); + + // 地理位置特征 + addLocationFeatures(features, event); + + return features; + } + + private void addStatisticalFeatures(Map features, RiskEvent event) { + String userId = event.getUserId(); + String today = LocalDate.now().toString(); + + // 今日交易次数 + String dailyCountKey = "user:" + userId + ":daily:" + today + ":count"; + Long dailyCount = redisTemplate.opsForValue().increment(dailyCountKey); + redisTemplate.expire(dailyCountKey, Duration.ofDays(1)); + features.put("dailyTransactionCount", dailyCount); + + // 今日交易金额 + String dailyAmountKey = "user:" + userId + ":daily:" + today + ":amount"; + Double dailyAmount = redisTemplate.opsForValue() + .increment(dailyAmountKey, event.getAmount().doubleValue()); + redisTemplate.expire(dailyAmountKey, Duration.ofDays(1)); + features.put("dailyTransactionAmount", dailyAmount); + + // 最近1小时交易次数 + String hourlyCountKey = "user:" + userId + ":hourly:" + + LocalDateTime.now().truncatedTo(ChronoUnit.HOURS) + ":count"; + Long hourlyCount = redisTemplate.opsForValue().increment(hourlyCountKey); + redisTemplate.expire(hourlyCountKey, Duration.ofHours(1)); + features.put("hourlyTransactionCount", hourlyCount); + } +} +``` + +**实时决策缓存**： + +```java +@Service +public class RiskDecisionCache { + + @Autowired + private RedisTemplate redisTemplate; + + public void cacheDecision(String key, RiskAssessmentResult result) { + String cacheKey = "risk:decision:" + key; + String value = JsonUtils.toJson(result); + + // 缓存1小时 + redisTemplate.opsForValue().set(cacheKey, value, Duration.ofHours(1)); + } + + public RiskAssessmentResult getCachedDecision(String key) { + String cacheKey = "risk:decision:" + key; + String value = redisTemplate.opsForValue().get(cacheKey); + + if (value != null) { + return JsonUtils.fromJson(value, RiskAssessmentResult.class); + } + + return null; + } + + // 生成缓存key，考虑用户、设备、金额等因素 + public String generateCacheKey(RiskEvent event) { + return String.format("%s:%s:%s", + event.getUserId(), + event.getDeviceId(), + event.getScenario()); + } +} +``` + +**规则动态更新**： + +```java +@Service +public class RuleManagementService { + + @Autowired + private RuleRepository ruleRepository; + + @EventListener + public void handleRuleUpdate(RuleUpdateEvent event) { + RiskRule rule = event.getRule(); + + // 更新本地缓存 + ruleCache.put(rule.getId(), rule); + + // 通知其他节点更新 + applicationEventPublisher.publishEvent( + new RuleCacheRefreshEvent(rule.getId())); + + log.info("规则更新完成: {}", rule.getId()); + } + + @Async + public void validateRuleEffectiveness(String ruleId) { + // 分析规则效果 + RuleEffectivenessAnalysis analysis = analyzeRuleEffectiveness(ruleId); + + // 如果效果不佳，建议调整 + if (analysis.getFalsePositiveRate() > 0.1) { + alertService.sendAlert("规则误判率过高", ruleId); + } + } +} +``` + +**性能监控**： + +```java +@Component +public class RiskSystemMonitor { + + private final MeterRegistry meterRegistry; + private final Counter evaluationCounter; + private final Timer evaluationTimer; + + public RiskSystemMonitor(MeterRegistry meterRegistry) { + this.meterRegistry = meterRegistry; + this.evaluationCounter = Counter.builder("risk.evaluation.count") + .register(meterRegistry); + this.evaluationTimer = Timer.builder("risk.evaluation.duration") + .register(meterRegistry); + } + + public void recordEvaluation(RiskAssessmentResult result, Duration duration) { + evaluationCounter.increment( + Tags.of( + "action", result.getAction().name(), + "risk_level", result.getRiskLevel().name() + ) + ); + + evaluationTimer.record(duration); + + // 如果评估时间过长，记录告警 + if (duration.toMillis() > 100) { + log.warn("风控评估耗时过长: {}ms", duration.toMillis()); + } + } +} +``` + + **考察点：** 实时计算、规则引擎、特征工程、性能优化。 + **常见追问：** 如何平衡准确性和性能？（答：分层策略+缓存优化+异步处理+模型优化） + +### 🎯 设计一个多租户SaaS系统 + +**业务需求分析**：支持多个企业客户独立使用，数据隔离、资源共享、灵活计费。 + +**核心挑战**： + +1. **数据隔离**：确保租户间数据完全隔离 +2. **资源共享**：在保证隔离的前提下最大化资源利用 +3. **个性化定制**：支持租户个性化配置和扩展 +4. **弹性扩展**：随租户增长动态扩容 + +**多租户架构模式**： + +**1. 数据库隔离策略** + +```java +@Configuration +public class MultiTenantConfig { + + // 租户路由策略 + @Bean + public TenantResolver tenantResolver() { + return new HeaderTenantResolver(); // 从HTTP头获取租户ID + } + + // 数据源路由 + @Bean + public DataSource dataSource() { + MultiTenantDataSource dataSource = new MultiTenantDataSource(); + + // 为每个租户配置独立数据源 + dataSource.setDefaultTargetDataSource(createDataSource("default")); + + Map targetDataSources = new HashMap<>(); + targetDataSources.put("tenant1", createDataSource("tenant1")); + targetDataSources.put("tenant2", createDataSource("tenant2")); + dataSource.setTargetDataSources(targetDataSources); + + return dataSource; + } +} + +@Component +public class TenantContext { + private static final ThreadLocal TENANT_ID = new ThreadLocal<>(); + + public static void setTenantId(String tenantId) { + TENANT_ID.set(tenantId); + } + + public static String getTenantId() { + return TENANT_ID.get(); + } + + public static void clear() { + TENANT_ID.remove(); + } +} +``` + +**2. 租户拦截器** + +```java +@Component +public class TenantInterceptor implements HandlerInterceptor { + + @Override + public boolean preHandle(HttpServletRequest request, + HttpServletResponse response, + Object handler) throws Exception { + + String tenantId = extractTenantId(request); + + if (tenantId == null) { + response.setStatus(HttpStatus.BAD_REQUEST.value()); + response.getWriter().write("Missing tenant identifier"); + return false; + } + + // 验证租户有效性 + if (!tenantService.isValidTenant(tenantId)) { + response.setStatus(HttpStatus.FORBIDDEN.value()); + response.getWriter().write("Invalid tenant"); + return false; + } + + TenantContext.setTenantId(tenantId); + return true; + } + + @Override + public void afterCompletion(HttpServletRequest request, + HttpServletResponse response, + Object handler, Exception ex) { + TenantContext.clear(); + } + + private String extractTenantId(HttpServletRequest request) { + // 多种方式获取租户ID + String tenantId = request.getHeader("X-Tenant-ID"); + if (tenantId != null) return tenantId; + + // 从子域名获取 + String serverName = request.getServerName(); + if (serverName.contains(".")) { + return serverName.split("\\.")[0]; + } + + // 从URL路径获取 + String path = request.getRequestURI(); + if (path.startsWith("/tenant/")) { + return path.split("/")[2]; + } + + return null; + } +} +``` + +**3. 动态配置管理** + +```java +@Service +public class TenantConfigurationService { + + @Autowired + private RedisTemplate redisTemplate; + + public T getConfig(String tenantId, String configKey, Class type) { + String cacheKey = "tenant:config:" + tenantId + ":" + configKey; + Object cached = redisTemplate.opsForValue().get(cacheKey); + + if (cached != null) { + return type.cast(cached); + } + + // 从数据库加载配置 + TenantConfiguration config = configRepository.findByTenantIdAndKey(tenantId, configKey); + if (config != null) { + T value = JsonUtils.fromJson(config.getValue(), type); + // 缓存30分钟 + redisTemplate.opsForValue().set(cacheKey, value, Duration.ofMinutes(30)); + return value; + } + + // 返回默认配置 + return getDefaultConfig(configKey, type); + } + + public void updateConfig(String tenantId, String configKey, Object value) { + // 更新数据库 + TenantConfiguration config = TenantConfiguration.builder() + .tenantId(tenantId) + .configKey(configKey) + .value(JsonUtils.toJson(value)) + .updateTime(Instant.now()) + .build(); + configRepository.save(config); + + // 清除缓存 + String cacheKey = "tenant:config:" + tenantId + ":" + configKey; + redisTemplate.delete(cacheKey); + + // 通知其他节点刷新缓存 + eventPublisher.publishEvent(new ConfigUpdateEvent(tenantId, configKey)); + } +} +``` + +**4. 资源隔离与限制** + +```java +@Service +public class TenantResourceManager { + + @Autowired + private RedisTemplate redisTemplate; + + public boolean checkResourceLimit(String tenantId, ResourceType type, int requestAmount) { + TenantPlan plan = tenantService.getTenantPlan(tenantId); + ResourceLimit limit = plan.getResourceLimit(type); + + if (limit == null) { + return true; // 无限制 + } + + String usageKey = "tenant:usage:" + tenantId + ":" + type.name(); + + // 获取当前使用量 + String currentUsageStr = redisTemplate.opsForValue().get(usageKey); + int currentUsage = currentUsageStr != null ? Integer.parseInt(currentUsageStr) : 0; + + // 检查是否超限 + if (currentUsage + requestAmount > limit.getMaxAmount()) { + log.warn("租户资源超限: tenantId={}, type={}, current={}, limit={}", + tenantId, type, currentUsage, limit.getMaxAmount()); + return false; + } + + // 更新使用量 + redisTemplate.opsForValue().increment(usageKey, requestAmount); + redisTemplate.expire(usageKey, Duration.ofDays(1)); + + return true; + } + + @Scheduled(fixedRate = 3600000) // 每小时统计一次 + public void collectResourceUsage() { + List tenants = tenantService.getAllActiveTenants(); + + for (Tenant tenant : tenants) { + TenantUsageStats stats = calculateUsageStats(tenant.getId()); + usageStatsRepository.save(stats); + + // 检查是否接近限制 + checkUsageAlerts(tenant, stats); + } + } +} +``` + +**5. 计费系统集成** + +```java +@Service +public class TenantBillingService { + + public void recordUsage(String tenantId, UsageEvent event) { + // 记录使用事件 + UsageRecord record = UsageRecord.builder() + .tenantId(tenantId) + .eventType(event.getType()) + .quantity(event.getQuantity()) + .timestamp(Instant.now()) + .metadata(event.getMetadata()) + .build(); + + usageRecordRepository.save(record); + + // 实时计费 + if (isRealTimeBillingEnabled(tenantId)) { + calculateAndApplyCharges(tenantId, record); + } + } + + @Scheduled(cron = "0 0 2 * * ?") // 每天凌晨2点执行 + public void dailyBilling() { + List tenants = tenantService.getAllActiveTenants(); + + for (Tenant tenant : tenants) { + try { + BillingResult result = calculateDailyBilling(tenant.getId()); + generateInvoice(tenant, result); + + } catch (Exception e) { + log.error("租户计费失败: {}", tenant.getId(), e); + alertService.sendAlert("计费失败", tenant.getId()); + } + } + } +} +``` + + **考察点：** 多租户架构、数据隔离、资源管理、计费系统。 + **常见追问：** 如何处理租户数据迁移？（答：在线迁移+双写验证+灰度切换+回滚机制） + +### 🎯 设计一个智能推荐系统 + +**业务场景分析**：为用户提供个性化内容推荐，提升用户体验和业务转化率。 + +**推荐系统核心架构**： + +**1. 多路召回策略** + +```java +@Service +public class RecommendationEngine { + + public List recommend(String userId, int count) { + // 协同过滤召回 + List cfResults = + collaborativeFilteringService.recommend(userId, count * 2); + + // 内容相似召回 + List cbResults = + contentBasedService.recommend(userId, count * 2); + + // 深度学习召回 + List dlResults = + deepLearningService.recommend(userId, count * 2); + + // 热门内容召回 + List popularResults = + popularContentService.getPopularItems(count); + + // 结果融合与重排序 + return mergeAndRerank(Arrays.asList(cfResults, cbResults, dlResults, popularResults)); + } +} +``` + +**2. 实时特征工程** + +```java +@Service +public class FeatureEngineeringService { + + public UserFeatures extractUserFeatures(String userId) { + UserFeatures features = new UserFeatures(); + + // 基础画像特征 + UserProfile profile = userService.getUserProfile(userId); + features.setDemographics(profile.getDemographics()); + + // 实时行为特征 + features.setRecentClickCategories(getRecentClickCategories(userId)); + features.setSessionDuration(getCurrentSessionDuration(userId)); + features.setActiveTimeSlots(getActiveTimeSlots(userId)); + + // 统计特征 + features.setAvgSessionDuration(calculateAvgSessionDuration(userId)); + features.setClickThroughRate(calculateCTR(userId)); + features.setConversionRate(calculateConversionRate(userId)); + + return features; + } +} +``` + +**3. 冷启动处理** + +```java +@Service +public class ColdStartService { + + public List handleNewUser(String userId) { + UserProfile profile = userService.getUserProfile(userId); + + // 基于人口统计学特征推荐 + List demographicRecommendations = + demographicBasedRecommender.recommend(profile); + + // 热门内容推荐 + List popularRecommendations = + popularContentService.getPopularItems(20); + + // 探索性推荐（多样性） + List exploratoryRecommendations = + explorationService.getExploratoryItems(profile); + + return mergeWithDiversity( + demographicRecommendations, + popularRecommendations, + exploratoryRecommendations); + } +} +``` + +**4. A/B测试框架** + +```java +@Service +public class RecommendationABTestService { + + public List applyExperiment(String userId, + List recommendations) { + String experimentGroup = getExperimentGroup(userId); + + switch (experimentGroup) { + case "control": + return recommendations; + case "diversity_boost": + return enhanceDiversity(recommendations); + case "popularity_boost": + return boostPopularItems(recommendations); + case "personalized_rerank": + return personalizedRerank(userId, recommendations); + default: + return recommendations; + } + } + + @Scheduled(fixedRate = 3600000) // 每小时分析 + public void analyzeExperimentResults() { + Map results = calculateMetrics(); + + for (Map.Entry entry : results.entrySet()) { + if (entry.getValue().isStatisticallySignificant()) { + updateExperimentStrategy(entry.getKey(), entry.getValue()); + } + } + } +} +``` + +**考察点：** 推荐算法、特征工程、冷启动、A/B测试。 +**常见追问：** 如何解决推荐系统的马太效应？（答：多样性控制+探索性推荐+长尾内容提升） + +------ + + + +## ⚡ 四、高并发系统设计 + +> **核心思想**：高并发系统设计是技术面的重点，需要从架构设计、性能优化、容错处理等多个角度来保证系统在高并发场景下的稳定性和性能。 + +### 🎯 如何设计一个支持50w QPS的分布式系统？ + +**系统分析**：50w QPS意味着每秒处理50万次请求，这需要在架构、技术栈、存储、网络等多个层面进行优化。 + +**整体架构设计**： + +1. **负载均衡层**： + - **DNS负载均衡**：多地域部署，就近访问 + - **四层负载均衡**：LVS/HAProxy处理连接分发 + - **七层负载均衡**：Nginx/F5处理HTTP请求路由 + - **CDN**：静态资源缓存，减少源站压力 + +2. **网关层**： + - **API网关集群**：Spring Cloud Gateway/Kong水平扩展 + - **限流熔断**：令牌桶、滑动窗口、熔断器保护 + - **请求路由**：按业务、版本、用户等维度路由 + - **协议优化**：HTTP/2、gRPC提升传输效率 + +3. **服务层**： + - **微服务架构**：按业务域拆分，独立扩缩容 + - **无状态设计**：服务实例无状态，便于水平扩展 + - **异步处理**：非关键流程异步化，提升响应速度 + - **连接池**：数据库、Redis连接池复用 + +4. **存储层**： + - **分库分表**：MySQL按业务、用户维度分片 + - **读写分离**：主库写入，从库读取 + - **多级缓存**：本地缓存 + Redis + CDN + - **NoSQL**：MongoDB/Cassandra处理大数据量 + +**性能优化策略**： + +**并发处理优化**： + +``` +计算模型： +单机QPS = 1000（经验值） +需要实例数 = 50w / 1000 = 500台 +考虑冗余：500 * 1.5 = 750台 +``` + +**缓存策略**： + +- **多级缓存架构**：CDN(90%) -> Redis(9%) -> DB(1%) +- **热点数据预热**：定期将热点数据加载到缓存 +- **缓存雪崩防护**：过期时间随机化、多副本 +- **缓存更新策略**：写入数据库后异步更新缓存 + +**数据库优化**： + +- **连接池配置**：单实例100-200连接，总计5w-10w连接 +- **SQL优化**：索引优化、查询改写、预编译语句 +- **分片策略**：按用户ID哈希分片，保证数据均匀 +- **读写分离**：读操作分散到多个从库 + +**系统架构容量规划**： + +| 层级 | 组件 | 实例数量 | 单实例QPS | 总QPS | +| ------ | -------- | -------- | --------- | ----- | +| 接入层 | Nginx | 50台 | 20k | 100w | +| 网关层 | Gateway | 100台 | 10k | 100w | +| 服务层 | 业务服务 | 500台 | 1k | 50w | +| 缓存层 | Redis | 20台 | 50k | 100w | +| 数据层 | MySQL | 50台 | 2k | 10w | + +**监控与保障**： + +- **实时监控**：QPS、RT、错误率、系统资源 +- **自动扩缩容**：根据CPU、内存、QPS指标自动扩容 +- **链路追踪**：分布式链路追踪，快速定位性能瓶颈 +- **压力测试**：定期全链路压测，验证容量 + + **考察点：** 高并发架构设计、容量规划、性能优化、监控体系。 + **常见追问：** 如何识别系统瓶颈？（答：监控+压测+性能分析工具定位） + +### 🎯 电商大促时如何保证系统稳定性？ + +**大促特点**：流量瞬间暴增10-100倍、业务峰值集中、容错要求极高。 + +**稳定性保障体系**： + +1. **容量准备**： + - **流量预估**：基于历史数据和业务预期评估峰值流量 + - **压力测试**：全链路压测，找出性能瓶颈 + - **资源扩容**：提前3-5倍扩容关键资源 + - **基础设施**：CDN带宽、服务器、数据库等全面扩容 + +2. **架构优化**： + - **静态化**：商品详情、活动页面全部静态化 + - **页面缓存**：首页、列表页等高访问页面缓存 + - **API优化**：接口合并、批量查询、异步处理 + - **资源隔离**：大促流量与日常流量物理隔离 + +3. **限流降级策略**： + - **多级限流**：CDN限流、网关限流、服务限流 + - **业务降级**：非核心功能降级，优先保证下单支付 + - **熔断保护**：依赖服务异常时快速熔断 + - **排队机制**：超出处理能力时排队等待 + +**核心业务保护**： + +**下单链路优化**： + +``` +优化前：同步调用多个服务，RT=500ms +优化后：异步化+缓存预热，RT=50ms + +流程优化： +1. 库存预扣（Redis）-> 2ms +2. 订单入库（异步）-> 10ms +3. 支付调用（异步）-> 20ms +4. 其他服务（异步）-> 并行处理 +``` + +**支付链路保护**： + +- **支付渠道**：多支付通道并行，故障自动切换 +- **幂等性**：支付请求幂等处理，防止重复扣款 +- **异步化**：支付结果异步通知，避免阻塞 +- **补偿机制**：支付异常时自动补偿和对账 + +**数据库保护**： + +- **读写分离**：读请求全部走从库和缓存 +- **连接池**：严格控制数据库连接数 +- **慢查询优化**：提前优化所有慢查询 +- **分库分表**：热点表按用户维度分片 + +**缓存策略**： + +- **多级缓存**：CDN + Redis集群 + 本地缓存 +- **预热策略**：大促前预热所有热点数据 +- **缓存隔离**：不同业务使用不同Redis集群 +- **兜底策略**：缓存失效时的降级方案 + +**监控告警**： + +- **实时大盘**：QPS、RT、错误率、库存等核心指标 +- **分层监控**：CDN、网关、服务、数据库各层监控 +- **智能告警**：基于历史数据的异常检测 +- **自动恢复**：故障自动切换和恢复 + +**应急预案**： + +- **限流开关**：流量过大时快速限流 +- **降级开关**：一键降级非核心功能 +- **回滚预案**：代码快速回滚机制 +- **人员保障**：24小时值班和快速响应 + + **考察点：** 大促架构设计、稳定性保障、应急处理、监控体系。 + **常见追问：** 如何评估大促容量？（答：历史数据+业务预期+安全系数） + +### 🎯 如何设计一个支持百万并发的直播系统？ + +**直播系统特点**：实时性要求高、并发用户多、带宽消耗大、互动性强。 + +**整体架构设计**： + +1. **推流端**： + - **RTMP推流**：主播端使用RTMP协议推送视频流 + - **流媒体服务器**：Nginx-RTMP/SRS/Node Media Server接收流 + - **转码服务**：FFmpeg多码率转码，适配不同网络 + - **录制存储**：视频流录制到OSS，支持回放 + +2. **CDN分发**： + - **边缘节点**：全球部署CDN节点，就近分发 + - **智能调度**：根据网络质量动态选择最优节点 + - **协议适配**：支持RTMP、HLS、HTTP-FLV多种协议 + - **带宽优化**：码率自适应、预加载优化 + +3. **播放端**： + - **多协议支持**：Web端HLS、移动端RTMP + - **播放器优化**：缓冲策略、断线重连、画质切换 + - **延迟优化**：WebRTC低延迟直播 + - **弱网优化**：网络自适应、画质降级 + +4. **互动系统**： + - **弹幕系统**：WebSocket实时弹幕推送 + - **礼物系统**：异步处理礼物动画和扣费 + - **聊天室**：群聊消息分发和审核 + - **连麦系统**：WebRTC点对点连接 + +**高并发处理**： + +**CDN架构设计**： + +``` +三级CDN架构： +源站 -> 中心节点 -> 边缘节点 -> 用户 + +并发能力： +- 单边缘节点：1万并发 +- 需要节点数：100万 / 1万 = 100个节点 +- 考虑冗余：100 * 1.5 = 150个节点 +``` + +**流媒体服务器集群**： + +- **负载均衡**：一致性哈希分配主播到不同服务器 +- **热备切换**：主播流自动故障切换 +- **状态同步**：服务器间流状态实时同步 +- **弹性扩容**：根据在线人数自动扩容 + +**弹幕系统设计**： + +- **WebSocket集群**：支持百万并发连接 +- **消息分片**：按房间ID分片处理弹幕 +- **限流控制**：用户发送频率限制 +- **内容审核**：敏感词过滤、机器审核 + +**性能优化策略**： + +**网络优化**： + +- **预连接**：页面加载时预建立连接 +- **多路复用**：HTTP/2多路复用减少连接数 +- **压缩传输**：视频压缩、文本gzip压缩 +- **P2P加速**：用户间P2P分享减少带宽 + +**存储优化**： + +- **热点数据**：主播信息、房间状态缓存到Redis +- **视频存储**：多副本存储保证可靠性 +- **CDN回源**：智能回源策略减少源站压力 +- **数据分片**：按时间、房间维度分片存储 + +**实时性优化**： + +- **端到端延迟**：< 3秒（HLS）、< 1秒（WebRTC） +- **关键帧优化**：GOP设置、IDR帧间隔优化 +- **缓冲策略**：播放器缓冲区大小动态调整 +- **网络自适应**：根据网络状况调整码率 + +**监控与运维**： + +- **实时监控**：在线人数、带宽使用、推流质量 +- **质量监控**：卡顿率、延迟、画质等QoE指标 +- **告警机制**：推流中断、CDN异常自动告警 +- **故障恢复**：自动切换备用节点和回源路径 + + **考察点：** 流媒体架构、CDN设计、实时通信、高并发处理。 + **常见追问：** 如何降低直播延迟？（答：WebRTC+边缘计算+协议优化） + +------ + + + +## 🌊 四、分布式系统设计 + +> **核心思想**：分布式系统设计需要解决数据一致性、服务可用性、网络分区等核心问题，通过合理的架构设计和技术选型来保证系统的稳定性和可扩展性。 + +### 🎯 设计一个短链系统（类似 bit.ly），如何处理千万级 QPS？ + +> 我会用 ID→Base62 可逆编码生成短码，读路径优先走 CDN 与 Redis 缓存（边缘缓存 302），确保绝大多数请求不回源。持久化用分库分表的 MySQL 或 NoSQL 存储映射与元数据；统计用 Kafka 异步上报并由 Flink 写入 ClickHouse 做实时/离线分析。为支持千万级 QPS，关键措施是：边缘缓存优先、Redis Cluster + 本地缓存、无状态跳转服务水平扩展、多机房部署及热点预热与限流。安全方面加速防刷、URL 白名单与钓鱼检测。短码生成采用 Snowflake + Base62，避免写前冲突，支持自定义短码时通过 DB 唯一索引处理冲突。 + +**需求分析**：短链生成、原链跳转、统计分析、高并发、高可用。 + +**核心架构设计**： + +1. **短链生成服务**： + - **编码算法**：Base62（a-z, A-Z, 0-9）生成6-8位短码，支持约568亿个URL + - **ID生成**：使用分布式ID生成器（Snowflake）保证唯一性 + - **防冲突**：写入前检查唯一性，冲突时重新生成 + +2. **存储设计**： + - **缓存层**：Redis集群存储热点短链（短码->原URL） + - **持久化**：MySQL分库分表存储映射关系 + - **分片策略**：按短码hash分片，保证数据均匀分布 + +3. **高并发优化**： + - **读写分离**：写入走主库，查询优先走缓存和从库 + - **多级缓存**：CDN + Redis + 本地缓存，99%请求命中缓存 + - **异步化**：统计数据异步写入，避免影响主链路 + +4. **扩展性设计**： + - **水平扩展**：服务无状态，可根据流量动态扩容 + - **分库分表**：按业务或时间维度分片 + - **CDN加速**：静态资源和跳转页面CDN缓存 + +**完整架构流程**： + +``` +生成短链：客户端 -> 负载均衡 -> 短链服务 -> ID生成 -> 写DB&缓存 -> 返回短码 +访问短链：用户 -> CDN -> 负载均衡 -> 查询服务 -> Redis -> DB -> 302跳转 +``` + +**监控与容灾**：实时监控QPS、缓存命中率、DB性能，多机房部署保证高可用。 + **考察点：** 高并发架构设计、缓存策略、数据库设计、扩展性考虑。 + **常见追问：** 如何防止恶意刷短链？（答：限流+验证码+用户黑名单+URL白名单） + + + +### 🎯 设计一个邮件系统，支持亿级用户发送邮件 + +**系统架构分层**： + +1. **接入层**： + - **API网关**：统一接入、鉴权、限流、熔断 + - **负载均衡**：按用户分片路由到不同服务集群 + - **协议支持**：SMTP、IMAP、POP3、HTTP API + +2. **业务服务层**： + - **邮件发送服务**：处理邮件发送逻辑、格式校验、附件处理 + - **邮件存储服务**：邮件内容存储、索引、检索 + - **用户管理服务**：账号体系、权限管理、配额控制 + - **通知服务**：实时推送、邮件到达通知 + +3. **数据存储层**： + - **用户数据**：MySQL集群存储用户信息、联系人、配置 + - **邮件元数据**：分库分表存储邮件头信息、状态、关系 + - **邮件内容**：对象存储（S3/OSS）存储邮件正文和附件 + - **搜索引擎**：Elasticsearch提供全文检索 + +4. **消息队列**： + - **发送队列**：Kafka处理海量邮件发送任务 + - **优先级队列**：重要邮件优先处理 + - **延迟队列**：定时发送功能 + +**核心技术挑战**： + +**亿级用户存储**： + +- **水平分片**：按用户ID哈希分库分表 +- **冷热分离**：近期邮件放SSD，历史邮件放HDD +- **压缩存储**：邮件内容压缩，附件去重 + +**高并发处理**： + +``` +发送流程：用户请求 -> 参数校验 -> 反垃圾检测 -> 入队列 -> 异步发送 -> 状态回调 +接收流程：SMTP接收 -> 病毒扫描 -> 反垃圾 -> 存储 -> 索引 -> 推送通知 +``` + +**可靠性保证**： + +- **多副本存储**：邮件数据多地域备份 +- **消息可靠性**：队列持久化、重试机制、死信队列 +- **监控告警**：发送成功率、延迟、存储容量监控 + +**安全防护**： + +- **反垃圾邮件**：机器学习算法、黑名单、内容过滤 +- **数据加密**：传输加密（TLS）、存储加密 +- **权限控制**：细粒度权限、审计日志 + + **考察点：** 大规模系统架构、数据分片、消息队列、安全设计。 + **常见追问：** 如何保证邮件不丢失？（答：多副本+事务+补偿机制+监控） + +### 🎯 设计一个类似微信的即时通讯系统 + +**核心功能需求**：实时消息、群聊、在线状态、消息推送、文件传输。 + +**整体架构**： + +1. **连接层（Gateway）**： + - **长连接管理**：WebSocket/TCP维持用户连接 + - **负载均衡**：一致性哈希分配用户到Gateway节点 + - **心跳保活**：定期心跳检测连接状态 + - **连接状态同步**：Gateway间用户在线状态同步 + +2. **消息服务层**： + - **消息路由服务**：查找接收方网关，转发消息 + - **群聊服务**：群成员管理、消息扇出 + - **离线消息服务**：用户离线时消息暂存 + - **推送服务**：APNs/FCM移动端推送 + +3. **存储层**： + - **消息存储**：按会话分片存储到MySQL/Cassandra + - **用户关系**：Redis存储好友关系、群成员关系 + - **文件存储**：OSS存储图片、语音、视频文件 + +**核心设计要点**： + +**消息投递保证**： + +``` +发送流程： +客户端 -> Gateway -> 消息服务 -> 存储DB -> 查找接收方Gateway -> 推送接收方 +确认机制：发送确认(sent) -> 投递确认(delivered) -> 已读确认(read) +``` + +**群聊优化**： + +- **读扩散模式**：群消息存一份，用户读取时拉取 +- **写扩散模式**：每个群成员都存一份消息副本 +- **混合模式**：小群写扩散，大群读扩散 + +**高可用设计**： + +- **Gateway集群**：多实例无状态部署，故障自动切换 +- **数据多副本**：消息数据至少两副本存储 +- **异地部署**：多机房部署，就近接入 + +**消息同步**： + +- **增量同步**：基于消息序列号增量拉取 +- **离线消息**：用户上线后批量推送未读消息 +- **多端同步**：消息多端实时同步 + +**性能优化**： + +- **消息预加载**：客户端预加载历史消息 +- **压缩传输**：消息内容压缩传输 +- **CDN加速**：图片、文件通过CDN分发 + + **考察点：** 长连接管理、消息可靠性、分布式架构、性能优化。 + **常见追问：** 如何处理海量群聊消息？（答：分片存储+异步扇出+读写分离） + +### 🎯 设计一个分布式配置中心（类似Apollo） + +**核心功能**：配置管理、实时推送、权限控制、版本管理、灰度发布。 + +**系统架构设计**： + +1. **配置管理层**： + - **Portal服务**：Web管理界面，配置CRUD操作 + - **Admin服务**：配置管理核心服务，权限控制 + - **Config服务**：配置读取服务，面向客户端 + +2. **存储层**： + - **元数据存储**：MySQL存储配置项、应用信息、权限 + - **配置存储**：支持多种存储后端（MySQL/Redis/ETCD） + - **版本管理**：Git-like版本控制，支持回滚 + +3. **通知层**： + - **消息队列**：配置变更事件队列 + - **长连接推送**：HTTP长轮询/WebSocket推送变更 + - **客户端SDK**：配置缓存、自动更新、降级处理 + +**核心技术实现**： + +**配置推送机制**： + +``` +推送流程： +1. 管理员修改配置 -> Portal +2. 配置校验和持久化 -> Admin +3. 发布变更事件 -> MessageQueue +4. Config服务接收事件 -> 推送客户端 +5. 客户端更新本地缓存 -> 应用生效 +``` + +**客户端设计**： + +- **本地缓存**：配置项本地缓存，启动时预加载 +- **长轮询**：定期请求配置更新，有变更立即返回 +- **降级策略**：网络异常时使用本地缓存配置 +- **热更新**：配置变更自动刷新，无需重启应用 + +**高可用保证**： + +- **集群部署**：Config服务集群，客户端多实例连接 +- **数据备份**：配置数据多副本存储 +- **故障切换**：客户端自动切换到其他Config实例 +- **本地容灾**：客户端本地文件备份 + +**安全与权限**： + +- **多环境隔离**：dev/test/prod环境严格隔离 +- **权限控制**：基于角色的配置读写权限 +- **审计日志**：所有配置变更全程审计 +- **敏感信息加密**：密码等敏感配置加密存储 + +**管理功能**： + +- **版本管理**：配置版本控制，支持比较和回滚 +- **灰度发布**：配置变更灰度发布，降低影响面 +- **批量操作**：支持配置的批量导入导出 +- **实时监控**：配置推送成功率、客户端在线状态 + + **考察点：** 分布式系统设计、实时通信、高可用架构、权限设计。 + **常见追问：** 如何保证配置推送的可靠性？（答：重试机制+本地缓存+多副本） + + + +### 🎯 限流算法 + +限流算法是分布式系统中 “保护服务稳定性” 的核心手段，用于在流量超过服务承载能力时，通过 “合理丢弃 / 排队请求” 避免服务过载崩溃。 + +**常见限流算法（4 种核心）** + +| 算法 | 原理 | 优点 | 缺点 | +| ------------------------------ | -------------------------------------- | ------------------ | ------------------------ | +| **固定窗口（Fixed Window）** | 统计每个时间窗口内的请求数 | 实现简单 | 边界流量突刺问题 | +| **滑动窗口（Sliding Window）** | 将时间窗口细分成小格动态滑动 | 精度更高，平滑 | 实现稍复杂 | +| **令牌桶（Token Bucket）** | 按固定速率生成令牌，请求需拿令牌才执行 | 支持突发流量，灵活 | 实现复杂，需定时补充令牌 | +| **漏桶（Leaky Bucket）** | 请求流入桶中，按固定速率流出 | 控制输出速率稳定 | 不支持突发流量 | + +------ + + + +## 🚀 五、性能优化与调优 + +> **核心思想**：性能优化是系统稳定运行的关键，需要从代码层面、架构层面、运维层面等多个维度进行优化，通过监控和调优来提升系统性能。 + +### 🎯 线上接口偶发超时，你如何定位？ + +标准排查流程： + +1. 第一步，先明确超时的现象边界 —— 通过监控看是全接口还是某类接口、全机器还是某几台、随机时间还是高峰期，快速缩小排查范围； + +2. **查看调用链**：用 APM/调用链（SkyWalking/Zipkin）定位是自己服务慢还是依赖慢。 + +3. **抓取线程快照**：`jstack` 看是否有线程阻塞、死锁或大量 GC。 + + > **场景 1：自身服务慢**（如接口内耗时高）： + > + > - 抓线程快照：用 `jstack > stack.log` 多次抓取（间隔 5s，抓 3-5 次），分析是否有： + > - 大量线程处于 `BLOCKED` 状态（看 “waiting for monitor entry”，定位锁竞争，如全局锁、单例 Bean 的同步方法）； + > - 线程处于 `WAITING` 状态（看 “parking to wait for <0x...>”，定位线程池满、队列堆积，如核心线程数设置过小）； + > - 死锁：用 `jstack -l ` 直接检测，若有死锁，输出会明确标注 “Found 1 deadlock.”，并显示锁依赖链。 + > - 查 GC 日志：用 `jstat -gcutil 1000 10` 看 GC 情况，是否有频繁 Full GC（导致 STW 时间过长，如内存泄漏、堆内存设置过小）。 + > - 查代码日志：看接口内是否有 “隐性耗时操作”，如大对象序列化、循环调用 DB、未关闭的流。 + +4. **查看 DB & 外部依赖**：检查慢 SQL、外呼超时。 + +5. **网络监控**：排查网络丢包/延迟，检查网关与 LB。 + +6. **回放/复现**：在预发布或压力环境复现问题并定位。 + + 归根结底是"先定位慢链路，再深挖具体问题"。定位后讲清楚恢复与后续预防措施（限流、监控、熔断）。 + **考察点：** 系统化问题定位能力与沟通恢复方案。 + **常见追问：** 遇到死锁怎么办？（答：抓取线程堆栈，找到锁依赖链并调整加锁顺序或加超时） + +### 🎯 某服务 CPU 占 100%，如何排查？ + +1. **裸机/容器监控**：用 `top` 或容器监控看是哪个进程/线程。 +2. **jstack**：抓取线程堆栈，看是否在某个热点方法或死循环。 +3. **CPU profiler**（async-profiler）做采样分析，找出热点方法和系统调用。 +4. **检查 GC**：高 CPU 也可能是 GC 消耗（查看 GC 日志）。 +5. **业务回归**：判断是否近期部署变更引入性能回退，回滚验证。 + **考察点：** 性能分析工具的熟练度与快速定位能力。 + **常见追问：** async-profiler 用法简述？（答：采样模式低开销，能定位 Java 层热点和 JNI/syscall） + +### 🎯 线上死锁如何处理？ + +做法： + +1. 首先通过 **监控系统（如 Arthas、Prometheus、SkyWalking、Thread Dump）** 观察线程卡顿情况。 + +2. **抓取多次线程堆栈（jstack）** 确定死锁是否持续。 + + ``` + jstack -l > dump.log + ``` + +3. **分析锁持有者与等待者**，找到循环依赖链（jstack 会提示死锁信息）。 + +4. **临时恢复**：如果能快速确定单点锁可手工释放（慎用），或者重启受影响服务实例做短期恢复。 + +5. **根本修复**：统一加锁顺序、缩小锁粒度、使用 tryLock 超时处理或改为无锁算法。 + +6. **回顾与防范**：补充单元/集成测试模拟并发场景，避免再次发生。 + **考察点：** 从临时恢复到根本解决与防范的完整流程。 + **常见追问：** 如果无法重启怎么办？（答：尝试定位具体线程并做探测性 dump，若能释放锁再恢复；否则按 SLA 评估重启） + + + +### 🎯 系统内存使用率持续上升，怎么排查内存泄漏？ + +**内存泄漏特征**：内存使用持续增长、Full GC频繁但内存不下降、最终导致OOM。 + +**排查工具和方法**： + +**第一步：监控分析** + +```bash +# 1. JVM内存监控 +jstat -gc -h10 5s # 观察GC情况 +jstat -gccapacity # 查看堆容量 + +# 2. 系统内存监控 +ps aux | grep java # 进程内存使用 +free -h # 系统内存情况 +``` + +**第二步：堆内存分析** + +```bash +# 1. 生成堆转储 +jmap -dump:live,format=b,file=heap.dump + +# 2. 查看堆内存分布 +jmap -histo | head -20 + +# 3. 强制GC观察 +jmap -gc +``` + +**第三步：MAT分析堆转储** + +- **Leak Suspects Report**：自动发现可能的内存泄漏 +- **Dominator Tree**：查看占用内存最大的对象 +- **Histogram**：按类统计对象数量和大小 +- **OQL查询**：编写查询语句分析特定对象 + +**常见内存泄漏模式**： + +**1. 集合类未清理** + +```java +// 问题代码 +private static Map cache = new HashMap<>(); + +public void addCache(String key, Object value) { + cache.put(key, value); // 只添加不清理 +} + +// 解决方案 +private static Map cache = new ConcurrentHashMap<>(); + +@Scheduled(fixedRate = 300000) // 5分钟清理一次 +public void cleanExpiredCache() { + cache.entrySet().removeIf(entry -> isExpired(entry)); +} +``` + +**2. ThreadLocal未清理** + +```java +// 问题代码 +private static ThreadLocal userContext = new ThreadLocal<>(); + +// 解决方案 +try { + userContext.set(user); + // 业务逻辑 +} finally { + userContext.remove(); // 必须清理 +} +``` + + **考察点：** 内存管理知识、问题排查能力、代码质量意识。 + **常见追问：** 如何在生产环境安全地生成堆转储？（答：使用-dump:live减少影响，选择低峰期执行） + + + +### 🎯 线上 OOM（OutOfMemoryError）怎么排查和解决？ + +我线上确实排查过 OOM 问题。一般表现为接口响应变慢、Full GC 频繁、监控报警内存飙升、甚至进程直接被 Killed（OOMKilled）。 + 我通常分五步处理：**确认、定位、恢复、根因分析、防范优化**。 + +**排查与处理步骤** + +**① 确认 OOM 类型** + +首先看日志或报警信息，确定是哪类内存溢出。 + 常见 OOM 类型有： + +| 类型 | 原因 | 表现 | +| ------------------------------------ | ------------------ | ------------------------------- | +| `Java heap space` | Java 堆内存不足 | Full GC频繁、堆dump可见大量对象 | +| `GC overhead limit exceeded` | GC回收效果太差 | CPU飙高但内存回不去 | +| `Metaspace` | 类加载过多或未卸载 | 动态生成类、热加载、反射 | +| `Direct buffer memory` | NIO直接内存泄漏 | Netty、大文件、ByteBuffer | +| `unable to create new native thread` | 线程数超系统限制 | 线程池或无限创建线程 | +| `OutOfMemoryError: Map failed` | 映射文件过多 | MappedByteBuffer泄漏 | + +**② 定位堆内存泄漏原因** + +**✅ 工具手段：** + +- **jmap** 导出堆快照： + + ``` + jmap -dump:format=b,file=heap.bin + ``` + +- **jstat** 查看 GC 情况： + + ``` + jstat -gcutil 1000 10 + ``` + +- **MAT / VisualVM / Arthas heapdump** 打开分析： + + - 查看 **大对象（Dominator Tree）** + - 查找 **GC Roots 引用链** + - 判断是否是**内存泄漏（Leak）** 还是 **内存占用高（非泄漏）** + +**✅ 分析思路：** + +- 哪个类实例数量异常？ +- 哪些对象在 GC 后仍被引用？ +- 是否存在静态集合缓存未清理？ +- 是否有线程池、连接池、队列堆积？ + +**③ 短期恢复手段** + +- **扩容/重启实例**（快速恢复服务）； + +- 如果是容器环境，临时调高： + + ``` + -Xmx + -XX:MaxMetaspaceSize + ``` + +- 若是内存泄漏型问题 → 尽量导出堆后再重启； + +- 监控确认重启后内存曲线恢复正常。 + +**④ 根因修复** + +常见 OOM 根因及修复方向： + +| 问题类别 | 根因 | 解决方案 | +| ------------------ | ------------------------------------------ | ------------------------------------------------ | +| 缓存泄漏 | Map 缓存未清理 / 本地缓存未过期 | 使用 `ConcurrentHashMap` + TTL 或 Caffeine/Guava | +| 集合不断增长 | 未清理的 `List` / `Map` / 队列 | 控制队列大小、定期清理 | +| 线程泄漏 | `Executors.newCachedThreadPool()` 无界线程 | 改用 `ThreadPoolExecutor` + 有界队列 | +| 数据库/IO泄漏 | ResultSet/Stream 未关闭 | 使用 try-with-resources 或连接池 | +| ClassLoader泄漏 | 动态加载类未卸载（反射、热部署） | 控制类加载器生命周期 | +| 外部内存泄漏 | Netty、DirectBuffer 未释放 | 调用 `release()`、开启内存监控 | +| JSON/XML解析大对象 | 反序列化过大对象 | 控制输入流大小、分页处理 | + +**⑤ 防范与监控** + +| 措施 | 说明 | +| ------------ | ------------------------------------------------------ | +| JVM 监控 | 使用 Prometheus + Grafana 监控堆、GC、线程 | +| 定期 dump | 使用 Arthas / jcmd 自动定期 dump 栈快照 | +| 限制内存 | 合理配置 `-Xmx`, `-Xms`, `MaxMetaspaceSize` | +| 使用弱引用 | 对缓存对象使用 `WeakReference` / `SoftReference` | +| 压测提前发现 | JMeter / Gatling 模拟长时间高并发请求 | +| GC 日志分析 | 打开 `-Xlog:gc*` 或 `-XX:+PrintGCDetails` 观测内存波动 | + + + +### 🎯 数据库查询突然变慢，如何快速优化？ + +**数据库性能问题排查流程**：监控确认 -> 定位慢查询 -> 分析执行计划 -> 优化实施 -> 效果验证。 + +**第一步：确认性能问题** + +```sql +-- 1. 查看当前活跃连接 +SHOW PROCESSLIST; + +-- 2. 查看数据库状态 +SHOW STATUS LIKE 'Threads%'; +SHOW STATUS LIKE 'Questions'; +SHOW STATUS LIKE 'Slow_queries'; + +-- 3. 查看锁等待情况 +SELECT * FROM information_schema.INNODB_LOCKS; +SELECT * FROM information_schema.INNODB_LOCK_WAITS; +``` + +**第二步：定位慢查询** + +```sql +-- 1. 开启慢查询日志 +SET GLOBAL slow_query_log = 'ON'; +SET GLOBAL long_query_time = 0.1; -- 0.1秒以上的查询 + +-- 2. 查看当前慢查询 +SELECT * FROM information_schema.PROCESSLIST +WHERE COMMAND != 'Sleep' AND TIME > 0.1; + +-- 3. 分析慢查询日志 +-- 使用mysqldumpslow分析日志文件 +mysqldumpslow -s t -t 10 /var/log/mysql/slow.log +``` + +**第三步：分析执行计划** + +```sql +-- 1. EXPLAIN分析 +EXPLAIN SELECT * FROM orders +WHERE user_id = 12345 AND order_date > '2024-01-01'; + +-- 2. 关注关键指标 +-- type: ALL(全表扫描)最差，index > range > ref > const最好 +-- key: 使用的索引 +-- rows: 预估扫描行数 +-- Extra: Using temporary、Using filesort等需要优化 +``` + +**常见问题及优化策略**： + +**场景1：缺少索引** + +```sql +-- 问题查询 +SELECT * FROM orders WHERE user_id = 12345 AND status = 'PAID'; +-- EXPLAIN显示：type=ALL, rows=1000000 + +-- 解决方案：创建复合索引 +CREATE INDEX idx_user_status ON orders(user_id, status); + +-- 验证效果 +EXPLAIN SELECT * FROM orders WHERE user_id = 12345 AND status = 'PAID'; +-- 优化后：type=ref, rows=100 +``` + +**场景2：索引失效** + +```sql +-- 问题查询：函数导致索引失效 +SELECT * FROM orders WHERE DATE(order_date) = '2024-01-01'; + +-- 解决方案：避免在索引列上使用函数 +SELECT * FROM orders +WHERE order_date >= '2024-01-01 00:00:00' +AND order_date < '2024-01-02 00:00:00'; +``` + +**考察点：** 数据库优化能力、SQL调优技巧、性能分析思路。 +**常见追问：** 索引过多有什么问题？（答：影响写性能、占用存储空间、维护成本高） + + + +### 🎯 日志分析工具用了哪些？ + +在面试中被问到日志分析工具时，可以从以下几个方面进行回答：所用的工具、它们的主要功能、你是如何使用这些工具的，以及它们在你的项目中带来的具体好处。以下是一些常用的日志分析工具及其特点： + +**常用日志分析工具** + +1. **ELK Stack (Elasticsearch, Logstash, Kibana)** + - **Elasticsearch**: 一个强大的搜索引擎，用于存储和查询日志数据。 + - **Logstash**: 一个数据处理管道工具，用于收集、解析和存储日志数据。 + - **Kibana**: 一个数据可视化工具，用于展示和分析 Elasticsearch 中的数据。 + - **使用场景**: 大量日志数据的集中管理和实时分析。 + - **个人经验**: 可以提到如何设置 Logstash 管道、创建 Kibana 仪表盘来监控特定的日志模式或异常。 +2. **Graylog** + - **特点**: 基于 Elasticsearch 的日志管理工具，具有强大的日志聚合、搜索和分析功能。 + - **使用场景**: 实时日志监控和警报。 + - **个人经验**: 可以提到如何配置 Graylog 采集日志、设置警报规则，以及如何利用 Graylog 的搜索功能进行故障排除。 +3. **Splunk** + - **特点**: 商业化的日志管理和分析工具，提供强大的搜索、监控和可视化功能。 + - **使用场景**: 复杂的企业级日志分析和安全监控。 + - **个人经验**: 可以提到如何利用 Splunk 进行实时日志分析、创建报告和仪表盘，以及如何使用 Splunk 的机器学习功能进行异常检测。 +4. **Fluentd** + - **特点**: 一个开源的数据收集器，用于统一日志数据。 + - **使用场景**: 日志数据的收集和转发。 + - **个人经验**: 可以提到如何配置 Fluentd 插件、收集和转发日志到不同的存储系统（如 Elasticsearch、MongoDB）。 +5. **Loggly** + - **特点**: 基于云的日志管理和分析服务，提供实时日志监控和警报。 + - **使用场景**: 云环境中的日志管理。 + - **个人经验**: 可以提到如何将应用日志发送到 Loggly、配置日志搜索和警报，以及利用 Loggly 的仪表盘进行日志可视化。 +6. **Prometheus 和 Grafana** + - **特点**: Prometheus 用于监控和告警，Grafana 用于数据可视化。虽然主要用于度量和监控，但也可以用于日志分析。 + - **使用场景**: 系统和应用的监控。 + - **个人经验**: 可以提到如何配置 Prometheus 采集日志指标、设置警报规则，以及如何利用 Grafana 创建可视化面板。 + +我们使用 ELK Stack 来集中管理和分析日志数据。通过 Logstash 我们收集来自不同服务的日志，并将其存储在 Elasticsearch 中，然后使用 Kibana 创建了多个仪表盘来监控系统的健康状况和性能 + +------ + + + +## 🏛️ 六、架构思维与技术治理 + +> **核心思想**：架构思维是高级工程师的核心能力，需要从技术选型、系统治理、团队协作等多个角度来推动技术架构的演进和优化。 + +### 🎯 如何评估和选择技术架构方案？ + +**架构评估维度框架**： + +1. **功能性需求评估** + - **业务支撑能力**：能否满足核心业务场景 + - **扩展性要求**：未来业务增长的支撑能力 + - **集成能力**：与现有系统的兼容性 + +2. **非功能性需求评估** + - **性能指标**：QPS、RT、吞吐量是否满足预期 + - **可用性要求**：SLA指标、容灾恢复能力 + - **安全性标准**：数据保护、访问控制、审计能力 + +3. **技术可行性评估** + - **团队技术栈匹配度**：学习成本和实施风险 + - **生态成熟度**：社区支持、文档完善度、第三方工具 + - **运维复杂度**：部署、监控、故障排查的便利性 + +**决策评估方法**： + +``` +技术架构评估表： +方案A 方案B 方案C +功能完整性 8 7 9 +性能表现 7 9 6 +开发效率 9 6 7 +运维成本 6 8 9 +技术风险 8 5 7 +总分加权 7.6 7.0 7.6 +``` + +**架构决策记录（ADR）**： + +- **背景**：为什么需要做这个决策 +- **决策**：具体选择了什么方案 +- **理由**：选择的依据和权衡考虑 +- **后果**：预期的影响和风险 + + **考察点：** 架构思维、决策能力、风险评估。 + **常见追问：** 如何处理架构选型中的技术债务？（答：建立技术债务清单，定期评估和重构，平衡业务交付和技术质量） + +### 🎯 大型系统的微服务治理策略？ + +**微服务治理体系**： + +1. **服务拆分治理** + - **领域驱动设计（DDD）**：按业务边界拆分，确保高内聚低耦合 + - **数据库独立**：每个服务独立数据库，避免数据耦合 + - **API设计规范**：RESTful API设计，版本管理策略 + +2. **服务间通信治理** + - **同步调用**：HTTP/gRPC，适用于实时性要求高的场景 + - **异步消息**：MQ解耦，适用于最终一致性场景 + - **服务网格**：Istio/Linkerd统一管理服务间通信 + +3. **服务质量治理** + - **限流熔断**：Hystrix/Sentinel保护服务稳定性 + - **超时控制**：合理设置调用超时时间 + - **重试策略**：指数退避算法，避免雪崩效应 + +**治理工具与平台**： + +``` +服务治理技术栈： +服务注册发现：Eureka/Consul/Nacos +配置中心：Apollo/Nacos +API网关：Zuul/Gateway/Kong +链路追踪：Zipkin/Jaeger/SkyWalking +监控告警：Prometheus+Grafana +日志聚合：ELK/EFK Stack +``` + +**微服务演进路径**： + +- **第一阶段**：单体拆分，核心服务独立 +- **第二阶段**：服务治理基础设施建设 +- **第三阶段**：服务网格化，统一治理 +- **第四阶段**：智能化运维，自动化治理 + +**治理成功指标**： + +- **可用性提升**：单服务故障不影响整体系统 +- **部署效率**：部署频率和部署成功率 +- **故障恢复**：MTTR（平均恢复时间）指标 +- **开发效率**：功能交付周期缩短 + + **考察点：** 微服务架构设计、治理体系建设、技术选型能力。 + **常见追问：** 如何解决微服务的分布式事务问题？（答：Saga模式、TCC模式、最终一致性设计） + +### 🎯 如何设计容错性强的分布式系统？ + +**容错设计原则**： + +1. **故障隔离（Bulkhead Pattern）** + - **资源隔离**：线程池、连接池独立配置 + - **服务隔离**：关键服务与非关键服务分离部署 + - **数据隔离**：核心数据与辅助数据分库存储 + +2. **快速失败（Fail Fast）** + - **超时控制**：设置合理的调用超时时间 + - **健康检查**：定期检测依赖服务健康状态 + - **熔断机制**：Circuit Breaker模式自动断开故障服务 + +3. **优雅降级（Graceful Degradation）** + - **功能降级**：非核心功能在故障时自动关闭 + - **性能降级**：降低响应精度，保证核心功能可用 + - **服务降级**：返回缓存数据或默认值 + +**容错实现策略**： + +**多层次冗余设计**： + +``` +容错层次： +应用层：多实例部署、负载均衡 +服务层：限流熔断、降级机制 +数据层：主从复制、分片备份 +基础设施：多机房部署、异地容灾 +``` + +**故障恢复机制**： + +- **自动重试**：指数退避算法，避免重试风暴 +- **断路器**：半开状态探测，自动恢复服务调用 +- **负载转移**：故障实例自动摘除，流量转移 +- **数据补偿**：异步补偿机制，保证数据最终一致性 + +**监控与告警体系**： + +```java +// 分布式系统健康监控 +@Component +public class SystemHealthMonitor { + + @Autowired + private List healthIndicators; + + @Scheduled(fixedRate = 30000) // 30秒检查一次 + public void checkSystemHealth() { + for (HealthIndicator indicator : healthIndicators) { + Health health = indicator.health(); + if (health.getStatus() != Status.UP) { + alertService.sendAlert("系统组件异常", + indicator.getClass().getSimpleName()); + } + } + } +} +``` + +**容错测试与验证**： + +- **混沌工程**：Chaos Monkey随机故障注入测试 +- **故障演练**：定期进行故障切换演练 +- **压力测试**：验证系统在高负载下的容错能力 +- **恢复验证**：测试故障恢复的时间和完整性 + + **考察点：** 分布式系统设计、容错机制、故障处理能力。 + **常见追问：** 如何平衡系统的性能和容错性？（答：通过合理的监控指标和自适应机制，动态调整容错策略） + +### 🎯 如何进行技术债务管理和重构决策？ + +**技术债务识别与分类**： + +1. **代码质量债务** + - **代码异味**：重复代码、过长方法、复杂类结构 + - **设计缺陷**：紧耦合、缺乏抽象、违反设计原则 + - **测试缺失**：单元测试覆盖率低、缺少集成测试 + +2. **架构设计债务** + - **技术选型**：过时的技术栈、不合适的框架选择 + - **架构腐化**：模块边界模糊、依赖关系复杂 + - **性能债务**：未优化的查询、缓存策略不当 + +3. **文档和知识债务** + - **文档缺失**：API文档过时、设计文档不全 + - **知识孤岛**：关键知识集中在少数人手中 + - **运维债务**：部署复杂、监控不完善 + +**技术债务评估框架**： + +``` +债务评估矩阵： + 影响范围 修复成本 业务风险 优先级 +核心模块 高 中 高 P0 +边缘功能 低 低 低 P3 +基础组件 高 高 中 P1 +``` + +**重构决策原则**： + +- **业务价值驱动**：优先重构影响业务的核心模块 +- **风险可控**：分阶段重构，确保系统稳定性 +- **投入产出比**：评估重构成本与收益 +- **团队能力匹配**：考虑团队技术能力和时间投入 + +**重构实施策略**： + +```java +// 渐进式重构示例：Strangler Fig模式 +@Component +public class OrderService { + + @Autowired + private LegacyOrderService legacyService; + + @Autowired + private NewOrderService newService; + + @Value("${feature.new-order-service.enabled:false}") + private boolean newServiceEnabled; + + public OrderResult createOrder(OrderRequest request) { + if (newServiceEnabled && request.getUserId() % 10 == 0) { + // 10%流量使用新服务 + return newService.createOrder(request); + } else { + // 90%流量使用旧服务 + return legacyService.createOrder(request); + } + } +} +``` + +**技术债务管理流程**： + +1. **债务识别**：代码扫描工具（SonarQube）、人工Review +2. **影响评估**：业务影响、技术影响、团队效率影响 +3. **优先级排序**：债务价值矩阵、ROI分析 +4. **计划制定**：重构计划、时间安排、资源分配 +5. **执行跟踪**：进度监控、质量验证、效果评估 + + **考察点：** 技术管理能力、重构经验、风险控制意识。 + **常见追问：** 如何说服业务方投入时间做重构？（答：量化技术债务的业务影响，展示重构带来的长期价值） + +### 🎯 如何设计系统的监控和可观测性架构？ + +**可观测性三大支柱**： + +1. **Metrics（指标监控）** + - **系统指标**：CPU、内存、磁盘、网络使用率 + - **应用指标**：QPS、响应时间、错误率、业务指标 + - **基础设施指标**：数据库连接数、消息队列积压、缓存命中率 + +2. **Logging（日志记录）** + - **结构化日志**：JSON格式，便于解析和查询 + - **日志等级**：DEBUG、INFO、WARN、ERROR合理使用 + - **链路标识**：TraceId、SpanId关联分布式调用链路 + +3. **Tracing（链路追踪）** + - **分布式追踪**：跨服务调用链路跟踪 + - **性能分析**：调用耗时、瓶颈定位 + - **依赖关系**：服务拓扑图、依赖分析 + +**监控架构设计**： + +``` +数据采集层： +- 应用探针：APM Agent、Prometheus Exporter +- 基础设施：Node Exporter、cAdvisor +- 日志采集：Fluentd、Logstash、Filebeat + +数据存储层： +- 时序数据库：Prometheus、InfluxDB +- 日志存储：Elasticsearch、Loki +- 链路存储：Jaeger、Zipkin + +分析展示层： +- 可视化：Grafana、Kibana +- 告警：AlertManager、PagerDuty +- 分析：Jupyter、数据分析平台 +``` + +**监控指标设计**： + +```yaml +# 关键业务指标 +business_metrics: + - name: "user_registration_rate" + description: "用户注册成功率" + query: "sum(rate(user_register_success[5m])) / sum(rate(user_register_total[5m]))" + + - name: "order_payment_latency" + description: "订单支付延迟" + query: "histogram_quantile(0.95, payment_duration_seconds)" + +# 技术指标 +technical_metrics: + - name: "service_availability" + description: "服务可用性" + query: "up{job=\"my-service\"}" + + - name: "error_rate" + description: "错误率" + query: "sum(rate(http_requests_total{status=~\"5..\"}[5m])) / sum(rate(http_requests_total[5m]))" +``` + +**告警策略设计**： + +```java +// 智能告警减少噪音 +@Component +public class IntelligentAlerting { + + public boolean shouldAlert(MetricAlert alert) { + // 1. 检查是否在维护窗口 + if (isInMaintenanceWindow()) { + return false; + } + + // 2. 检查历史模式，避免重复告警 + if (isRecentlyAlerted(alert.getMetricName(), Duration.ofMinutes(30))) { + return false; + } + + // 3. 关联性分析，避免告警风暴 + if (hasRelatedActiveAlerts(alert)) { + return false; + } + + // 4. 动态阈值调整 + double dynamicThreshold = calculateDynamicThreshold(alert.getMetricName()); + if (alert.getValue() < dynamicThreshold) { + return false; + } + + return true; + } +} +``` + +**可观测性最佳实践**： + +- **SLI/SLO设计**：定义服务等级指标和目标 +- **错误预算**：基于SLO计算可接受的错误率 +- **渐进式监控**：从基础监控到高级分析 +- **自动化运维**：基于监控数据的自动处理 + + **考察点：** 监控体系设计、运维自动化、系统稳定性保障。 + **常见追问：** 如何设计有效的告警策略？（答：分级告警+智能降噪+业务关联+自动处理） + +------ + + + + + diff --git a/docs/interview/BigData-FAQ.md b/docs/interview/BigData-FAQ.md new file mode 100644 index 0000000000..47720f97f2 --- /dev/null +++ b/docs/interview/BigData-FAQ.md @@ -0,0 +1,2827 @@ +--- +title: 大数据技术栈核心面试八股文 +date: 2024-05-31 +tags: + - BigData + - Hadoop + - Spark + - Flink + - Hive + - Kafka + - Doris + - Kylin + - OLAP + - Interview +categories: Interview +--- + +![](https://img.starfish.ink/common/faq-banner.png) + +> 大数据技术栈是现代互联网企业的技术基石，从**分布式存储**到**实时计算**，从**离线分析**到**流式处理**，每一项技术都承载着海量数据的处理挑战。本文档将**最常考的大数据知识点**整理成**标准话术**，涵盖Hadoop、Spark、Flink、Hive、Kafka等核心组件，助你在面试中游刃有余！ + +--- + +## 🗺️ 知识导航 + +### 🏷️ 核心知识分类 + +1. **🏗️ 分布式存储类**：HDFS架构、副本机制、NameNode、DataNode、块存储 +2. **⚡ 批计算框架**：MapReduce原理、Spark架构、RDD机制、内存计算 +3. **🌊 流计算框架**：Flink架构、Watermark、窗口函数、状态管理 +4. **📊 数据仓库技术**：Hive架构、SQL解析、分区分桶、存储格式、Kylin预计算 +5. **📈 OLAP数据库**：Apache Doris架构、数据模型、查询优化、实时分析 +6. **🚀 消息队列**：Kafka架构、分区副本、生产消费、性能优化 +7. **🔧 资源调度**：YARN架构、资源管理、任务调度、容器化 +8. **💼 实战场景题**：技术选型、架构设计、性能调优、故障处理 + +### 🔑 面试话术模板 + +| **问题类型** | **回答框架** | **关键要点** | **深入扩展** | +| ------------ | ----------------------------------- | ------------------ | ------------------ | +| **概念解释** | 定义→架构→核心机制→应用场景 | 准确定义，突出优势 | 底层原理，源码分析 | +| **对比分析** | 相同点→不同点→使用场景→选择建议 | 多维度对比 | 性能差异，实际应用 | +| **原理解析** | 背景→架构设计→执行流程→关键机制 | 图解流程 | 深层实现，调优要点 | +| **优化实践** | 问题现象→分析思路→解决方案→监控验证 | 实际案例 | 最佳实践，踩坑经验 | + +--- + +## 🏗️ 一、分布式存储类（HDFS核心） + +> **核心思想**：HDFS是Hadoop生态的存储基石，通过主从架构、副本机制、分块存储实现海量数据的可靠存储和高并发访问。 + +### 🎯 什么是HDFS？它的核心架构是什么？ + +**HDFS（Hadoop Distributed File System）是什么？** + +HDFS是Hadoop生态系统中的**分布式文件系统**，专为存储超大文件而设计。它通过**主从架构**实现数据的分布式存储，具有**高容错性、高吞吐量、流式数据访问**的特点。 + +**核心架构组件**： + +1. **NameNode（主节点）**： + - 存储文件系统的**元数据**（文件目录结构、文件属性、Block位置信息） + - 管理文件系统的**命名空间** + - 协调客户端对文件的访问 + - 维护**FSImage**（文件系统镜像）和**EditLog**（编辑日志） + +2. **DataNode（从节点）**： + - 存储实际的**数据块（Block）** + - 定期向NameNode发送**心跳**和**Block报告** + - 执行数据块的创建、删除、复制操作 + - 响应客户端的读写请求 + +3. **Secondary NameNode**： + - 定期合并FSImage和EditLog，减轻NameNode负担 + - **不是**NameNode的热备份，而是辅助节点 + +**核心特性**： +- **分块存储**：文件被切分成固定大小的Block（默认128MB），分布存储 +- **副本机制**：每个Block默认有3个副本，保证数据可靠性 +- **一写多读**：适合大文件的一次写入、多次读取场景 +- **流式访问**：优化顺序读取，不适合随机访问 + +### 🎯 HDFS的读写流程是怎样的？ + +**HDFS写入流程**： + +1. **客户端请求**：客户端调用FileSystem.create()创建文件 +2. **NameNode验证**：检查文件是否存在、权限是否满足 +3. **返回输出流**：NameNode返回FSDataOutputStream给客户端 +4. **申请Block**：客户端向NameNode申请新的Block和DataNode列表 +5. **建立管道**：客户端与第一个DataNode建立连接，形成DataNode管道 +6. **数据传输**：数据以packet为单位在管道中传输 +7. **确认机制**：每个DataNode接收数据后发送确认给前一个节点 +8. **关闭流**：写完后关闭输出流，通知NameNode写入完成 + +**HDFS读取流程**： + +1. **客户端请求**：调用FileSystem.open()打开文件 +2. **NameNode查询**：获取文件的Block列表和对应的DataNode位置 +3. **返回输入流**：NameNode返回FSDataInputStream给客户端 +4. **选择DataNode**：客户端选择最近的DataNode读取Block +5. **数据传输**：直接从DataNode读取数据到客户端 +6. **切换Block**：读完一个Block后自动切换到下一个Block +7. **关闭流**：读取完成后关闭输入流 + +**关键优化点**： +- **机架感知**：优先选择同机架的DataNode，减少网络传输 +- **本地读取**：如果客户端和DataNode在同一节点，直接本地读取 +- **缓存机制**：NameNode的元数据缓存在内存中，提升查询性能 + +### 🎯 HDFS的副本放置策略是什么？为什么这样设计？ + +**默认副本放置策略（3副本）**： + +1. **第一个副本**：放在客户端所在节点（如果客户端在集群外，随机选择） +2. **第二个副本**：放在不同机架的随机节点 +3. **第三个副本**：放在第二个副本同一机架的不同节点 + +**设计原理**： + +- **可靠性**：不同机架保证机架故障时数据不丢失 +- **性能**：同机架内有两个副本，读取时可就近访问 +- **网络带宽**：跨机架只有一次数据传输，节省网络带宽 + +**机架感知的重要性**： +``` +机架A: DataNode1, DataNode2 +机架B: DataNode3, DataNode4 + +副本放置： +- 第一副本：DataNode1（客户端节点） +- 第二副本：DataNode3（不同机架） +- 第三副本：DataNode4（与第二副本同机架） +``` + +**容错能力分析**： +- **节点故障**：任何单节点故障，剩余副本正常服务 +- **机架故障**：任何单机架故障，其他机架的副本继续服务 +- **网络分区**：机架间网络故障时，仍能保证数据访问 + +### 🎯 NameNode单点故障如何解决？ + +**问题背景**： +NameNode存储着整个文件系统的元数据，一旦宕机，整个HDFS集群不可用，是典型的单点故障问题。 + +**解决方案**： + +**1. Secondary NameNode（辅助方案）** +- 定期合并FSImage和EditLog +- NameNode故障后可手动恢复，但会有数据丢失 +- **不是真正的高可用方案** + +**2. NameNode HA（推荐方案）** + +**架构设计**： +- **Active NameNode**：提供正常服务 +- **Standby NameNode**：热备节点，实时同步元数据 +- **共享存储**：JournalNode集群或NFS，存储EditLog +- **ZooKeeper**：协调Active/Standby状态切换 +- **ZKFC**：ZooKeeper FailoverController，监控NameNode健康状态 + +**故障切换流程**： +1. ZKFC监控到Active NameNode故障 +2. 通过ZooKeeper协调，选举新的Active +3. Standby NameNode升级为Active +4. 客户端重新连接新的Active NameNode + +**数据同步机制**： +- EditLog写入共享存储（JournalNode） +- Standby NameNode实时读取EditLog更新内存中的元数据 +- 保证Active/Standby元数据一致性 + +**3. Federation（联邦架构）** +- 多个NameNode管理不同的命名空间 +- 水平扩展NameNode的处理能力 +- 每个NameNode独立，彼此故障不影响 + +--- + +## ⚡ 二、批计算框架（MapReduce & Spark） + +> **核心思想**：批计算框架是大数据处理的核心，从MapReduce的磁盘计算到Spark的内存计算，体现了大数据技术的演进历程。 + +### 🎯 MapReduce的工作原理是什么？ + +**MapReduce是什么？** + +MapReduce是一种**分布式计算模型**，将复杂的数据处理任务分解为**Map（映射）**和**Reduce（归约）**两个阶段，适合处理大规模数据集的批处理任务。 + +**核心工作流程**： + +**1. Input输入阶段** +- 输入数据被切分成多个InputSplit +- 每个Split由一个Map任务处理 +- 典型Split大小等于HDFS Block大小（128MB） + +**2. Map阶段** +- Map任务读取InputSplit中的数据 +- 执行用户自定义的map函数 +- 输出key-value对到本地磁盘 +- 进行分区（Partition）和排序（Sort） + +**3. Shuffle阶段**（核心且复杂） +- **Copy阶段**：Reduce任务从各个Map任务拷贝数据 +- **Sort阶段**：对拷贝来的数据进行合并排序 +- **Group阶段**：将相同key的value组合在一起 + +**4. Reduce阶段** +- Reduce任务处理分组后的数据 +- 执行用户自定义的reduce函数 +- 输出最终结果到HDFS + +**WordCount示例流程**： +``` +输入：hello world hello hadoop +Map输出：(hello,1), (world,1), (hello,1), (hadoop,1) +Shuffle后：(hello,[1,1]), (world,[1]), (hadoop,[1]) +Reduce输出：(hello,2), (world,1), (hadoop,1) +``` + +**关键机制**： +- **容错性**：任务失败自动重试，数据副本保证可靠性 +- **本地性**：优先在数据所在节点执行任务 +- **推测执行**：慢任务会启动备份任务，提升整体性能 + +### 🎯 Spark相比MapReduce有什么优势？ + +**Spark是什么？** + +Spark是基于**内存计算**的分布式计算框架，提供了比MapReduce更高的性能和更丰富的API，支持批处理、流处理、机器学习、图计算等多种计算场景。 + +**核心优势对比**： + +| 对比维度 | MapReduce | Spark | +|---------|-----------|-------| +| **计算模型** | 磁盘计算，Map-Reduce两阶段 | 内存计算，DAG多阶段 | +| **性能** | 中间结果落盘，I/O开销大 | 内存缓存，性能提升10-100倍 | +| **易用性** | 只有Map-Reduce API | 提供多种高级API（SQL、ML、Graph） | +| **容错机制** | 数据副本 + 任务重试 | RDD血统 + Checkpoint | +| **实时性** | 只支持批处理 | 支持流处理（Spark Streaming） | +| **内存管理** | 依赖OS内存管理 | 自主内存管理，统一内存模型 | + +**Spark架构优势**： + +**1. RDD（弹性分布式数据集）** +- **不可变**：一旦创建不可修改，保证数据一致性 +- **分区**：数据分布在集群的多个节点上 +- **容错**：通过血统（Lineage）实现故障恢复 +- **延迟计算**：只有Action操作才触发实际计算 + +**2. DAG（有向无环图）** +- 将复杂的计算流程表示为DAG +- 优化器自动优化执行计划 +- 避免不必要的磁盘I/O + +**3. 内存计算** +- 中间结果缓存在内存中 +- 大大减少磁盘I/O开销 +- 特别适合迭代计算（机器学习） + +**适用场景选择**： +- **MapReduce**：简单的ETL、大文件处理、对内存要求不高的场景 +- **Spark**：复杂分析、机器学习、实时处理、交互式查询 + +### 🎯 Spark的核心概念RDD是什么？ + +**RDD（Resilient Distributed Dataset）是什么？** + +RDD是Spark的**核心数据抽象**，代表一个不可变的、可分区的数据集合，分布在集群的多个节点上。它是Spark所有操作的基础。 + +**RDD的核心特性**： + +**1. 不可变性（Immutable）** +- RDD一旦创建就不能修改 +- 所有转换操作都会产生新的RDD +- 保证了数据的一致性和线程安全 + +**2. 分区性（Partitioned）** +- RDD的数据分布在多个分区中 +- 每个分区可以在不同的节点上并行处理 +- 分区数影响并行度 + +**3. 容错性（Fault-tolerant）** +- 通过血统（Lineage）记录RDD的依赖关系 +- 任何分区丢失都可以根据血统重新计算 +- 无需数据复制，节省存储空间 + +**4. 惰性计算（Lazy Evaluation）** +- Transformation操作不会立即执行 +- 只有遇到Action操作才会触发实际计算 +- 便于优化执行计划 + +**RDD操作类型**： + +**Transformation（转换操作）**： +- `map()`：对每个元素应用函数 +- `filter()`：过滤满足条件的元素 +- `flatMap()`：扁平化映射 +- `union()`：合并两个RDD +- `groupByKey()`：按key分组 +- `reduceByKey()`：按key归约 + +**Action（行动操作）**： +- `collect()`：收集所有元素到Driver +- `count()`：统计元素个数 +- `first()`：获取第一个元素 +- `save()`：保存到文件系统 +- `foreach()`：对每个元素执行操作 + +**RDD依赖关系**： + +**窄依赖（Narrow Dependency）**： +- 父RDD的每个分区只被子RDD的一个分区依赖 +- 支持pipeline优化 +- 故障恢复效率高 +- 例如：map、filter + +**宽依赖（Wide Dependency）**： +- 父RDD的每个分区被子RDD的多个分区依赖 +- 需要Shuffle操作 +- 故障恢复成本高 +- 例如：groupByKey、join + +### 🎯 Spark的内存管理机制是怎样的？ + +**Spark统一内存模型**： + +从Spark 1.6开始，引入了**统一内存管理**（Unified Memory Management），将内存分为两大区域： + +**1. 堆内内存（On-Heap Memory）** + +**Reserved Memory（保留内存）**： +- 固定300MB，用于系统内部对象 +- 不参与内存分配 + +**User Memory（用户内存）**： +- 占用（Heap - Reserved）* 0.25 = 25% +- 存储用户自定义数据结构 +- 不受Spark管理 + +**Unified Memory（统一内存）**： +- 占用（Heap - Reserved）* 0.75 = 75% +- 分为Storage Memory和Execution Memory + +**Storage Memory（存储内存）**： +- 用于缓存RDD、广播变量 +- 默认占Unified Memory的50% +- 可以借用Execution Memory + +**Execution Memory（执行内存）**： +- 用于Shuffle、Join、Sort等计算 +- 默认占Unified Memory的50% +- 可以借用Storage Memory（但有限制） + +**2. 堆外内存（Off-Heap Memory）** +- 通过`spark.memory.offHeap.enabled=true`开启 +- 避免GC影响，提升性能 +- 主要用于存储序列化的数据 + +**内存借用机制**： +- Execution可以借用Storage的空闲内存 +- Storage可以借用Execution的空闲内存 +- 但Storage借用的内存在Execution需要时必须释放 +- Execution借用的内存不会被强制释放 + +**内存管理优势**： +- **动态调整**：根据实际需求动态分配内存 +- **减少溢出**：避免固定分配导致的内存浪费 +- **提升性能**：统一管理减少内存碎片 + +--- + +## 🌊 三、流计算框架（Flink核心） + +> **核心思想**：Flink是新一代流计算引擎，以流为核心、批为特殊流的理念，提供低延迟、高吞吐、精确一次语义的流处理能力。 + +### 🎯 Flink的核心架构是什么？与Spark Streaming有什么区别？ + +**Flink是什么？** + +Apache Flink是一个**流优先**的分布式计算框架，支持有界和无界数据流的处理。它提供**低延迟、高吞吐量、Exactly-Once**语义的流处理能力。 + +**Flink核心架构**： + +**1. JobManager（作业管理器）** +- **JobMaster**：管理单个作业的生命周期 +- **ResourceManager**：管理集群资源分配 +- **Dispatcher**：提供REST接口接收作业提交 + +**2. TaskManager（任务管理器）** +- 实际执行任务的工作节点 +- 每个TaskManager包含多个Task Slot +- Task Slot是资源分配的基本单位 + +**3. Client（客户端）** +- 提交作业到集群 +- 编译用户程序生成JobGraph + +**Flink vs Spark Streaming 核心区别**： + +| 对比维度 | Flink | Spark Streaming | +|---------|-------|-----------------| +| **计算模型** | 真正的流计算（流为核心） | 微批处理（批为核心） | +| **延迟** | 毫秒级低延迟 | 秒级延迟 | +| **状态管理** | 原生流状态管理 | 基于RDD的状态管理 | +| **容错机制** | Checkpoint + 状态快照 | RDD血统恢复 | +| **窗口操作** | 灵活的窗口API | 基于批次的窗口 | +| **背压处理** | 动态背压控制 | 静态批处理调整 | + +**技术选型建议**： +- **低延迟要求**：选择Flink（毫秒级） +- **高吞吐批处理**：选择Spark（生态完整） +- **复杂状态管理**：选择Flink（原生支持） +- **机器学习场景**：选择Spark（MLlib完善） + +### 🎯 Flink的Watermark机制是什么？如何处理乱序数据？ + +**Watermark是什么？** + +Watermark是Flink中用于处理**乱序数据**和**事件时间窗口**的机制。它表示**某个时间戳之前的所有事件都已经到达**的标记。 + +**乱序数据的挑战**： +``` +理想情况：事件按时间戳顺序到达 +实际情况：网络延迟、系统故障导致乱序 +时间戳： 1, 2, 3, 4, 5 +到达顺序：1, 3, 2, 5, 4 +``` + +**Watermark工作原理**： + +**1. Watermark生成** +- **Periodic Watermark**：定期生成（默认200ms） +- **Punctuated Watermark**：根据特定事件生成 + +**2. Watermark传播** +- 从Source向下游传播 +- 多输入流取最小Watermark +- 保证全局时间推进 + +**3. 窗口触发** +- 当Watermark >= 窗口结束时间时触发窗口计算 +- 允许一定程度的延迟数据 + +**代码示例**： +```java +// 生成Watermark +stream.assignTimestampsAndWatermarks( + WatermarkStrategy + .forBoundedOutOfOrderness(Duration.ofSeconds(10)) + .withTimestampAssigner((event, timestamp) -> event.getTimestamp()) +); + +// 时间窗口 +stream.keyBy(Event::getUserId) + .window(TumblingEventTimeWindows.of(Time.minutes(5))) + .process(new WindowProcessFunction<>() { + // 窗口处理逻辑 + }); +``` + +**处理策略**： + +**1. 允许延迟（Allowed Lateness）** +```java +.window(TumblingEventTimeWindows.of(Time.minutes(5))) +.allowedLateness(Time.minutes(1)) // 允许1分钟延迟 +``` + +**2. 侧输出流（Side Output）** +```java +OutputTag lateDataTag = new OutputTag("late-data"){}; + +SingleOutputStreamOperator result = stream + .window(...) + .allowedLateness(Time.minutes(1)) + .sideOutputLateData(lateDataTag) + .process(...); + +// 获取延迟数据 +DataStream lateData = result.getSideOutput(lateDataTag); +``` + +**最佳实践**： +- 根据业务需求设置合理的延迟容忍度 +- 监控延迟数据比例，调整Watermark策略 +- 对于严格实时场景，使用处理时间窗口 + +### 🎯 Flink的状态管理是如何实现的？ + +**Flink状态管理概述**： + +状态是Flink流处理的核心功能，用于存储**中间计算结果**和**历史信息**，支持故障恢复和精确一次语义。 + +**状态分类**： + +**1. 按作用域分类** + +**Keyed State（键控状态）**： +- 与特定key相关的状态 +- 只能在KeyedStream上使用 +- 状态自动分区和分发 + +**Operator State（算子状态）**： +- 与算子实例绑定的状态 +- 每个算子并行实例维护自己的状态 +- 需要手动实现状态分发逻辑 + +**2. 按存储结构分类** + +**ValueState**：存储单个值 +```java +private ValueState countState; + +@Override +public void open(Configuration config) { + ValueStateDescriptor descriptor = + new ValueStateDescriptor<>("count", Integer.class); + countState = getRuntimeContext().getState(descriptor); +} +``` + +**ListState**：存储元素列表 +```java +private ListState eventListState; +``` + +**MapState**：存储Key-Value映射 +```java +private MapState mapState; +``` + +**ReducingState**：存储单个值，新值通过ReduceFunction合并 +```java +private ReducingState reducingState; +``` + +**AggregatingState**：类似ReducingState，但可以不同类型 +```java +private AggregatingState aggState; +``` + +**状态存储后端（State Backend）**： + +**1. MemoryStateBackend** +- 状态存储在JVM堆内存中 +- 适合状态较小的场景 +- 性能最好，但容量有限 + +**2. FsStateBackend** +- 状态存储在文件系统（HDFS/S3） +- 适合中等规模状态 +- 平衡性能和容量 + +**3. RocksDBStateBackend** +- 状态存储在本地RocksDB + 远程文件系统 +- 适合大状态场景 +- 支持增量checkpoint + +**Checkpoint机制**： + +**1. Checkpoint触发** +- JobManager定期触发Checkpoint +- 基于分布式快照算法（Chandy-Lamport） +- 保证状态一致性 + +**2. Checkpoint流程** +- JobManager向Source发送Checkpoint Barrier +- Barrier在数据流中传播 +- 算子收到Barrier时保存状态快照 +- 所有算子完成后Checkpoint成功 + +**3. 故障恢复** +- 从最近的Checkpoint恢复状态 +- 重播Checkpoint之后的数据 +- 保证Exactly-Once语义 + +**状态优化策略**： +- 合理选择State Backend +- 设置合适的Checkpoint间隔 +- 启用增量Checkpoint +- 清理过期状态（TTL） + +--- + +## 📊 四、数据仓库技术（Hive核心） + +> **核心思想**：Hive是Hadoop生态系统中的数据仓库软件，通过SQL接口简化大数据分析，是离线数据处理的核心组件。 + +### 🎯 Hive的架构原理是什么？SQL是如何转换为MapReduce的？ + +**Hive是什么？** + +Hive是基于Hadoop的**数据仓库软件**，提供**SQL接口**来查询存储在HDFS上的数据。它将SQL查询转换为MapReduce、Spark或Tez作业来执行。 + +**Hive核心架构**： + +**1. Hive Client（客户端）** +- **CLI**：命令行接口 +- **HiveServer2**：提供JDBC/ODBC接口 +- **Web Interface**：Web管理界面 + +**2. Hive Driver（驱动器）** +- **Compiler**：SQL编译器 +- **Optimizer**：查询优化器 +- **Executor**：执行引擎 + +**3. Hive MetaStore（元数据存储）** +- 存储表结构、分区信息、存储位置等元数据 +- 通常使用MySQL等关系数据库存储 +- 支持多个Hive实例共享元数据 + +**SQL转换MapReduce流程**： + +**1. 语法分析（Parse）** +- 使用Antlr将SQL解析为抽象语法树（AST） +- 检查语法错误 + +**2. 语义分析（Semantic Analysis）** +- 将AST转换为查询块（Query Block） +- 验证表、列是否存在 +- 类型检查和转换 + +**3. 逻辑计划生成** +- 生成逻辑执行计划 +- 包括操作符树结构 + +**4. 逻辑优化** +- **谓词下推**：将过滤条件尽早执行 +- **列裁剪**：只读取需要的列 +- **常量折叠**：预计算常量表达式 + +**5. 物理计划生成** +- 将逻辑计划转换为物理执行计划 +- 决定使用MapReduce/Spark/Tez + +**6. 物理优化** +- **MapJoin**：小表broadcast到大表所在节点 +- **分区裁剪**：只扫描相关分区 +- **索引使用**：利用索引加速查询 + +**示例SQL转换过程**： +```sql +SELECT dept, COUNT(*) +FROM employees +WHERE salary > 50000 +GROUP BY dept; +``` + +**转换为MapReduce**： +- **Map阶段**：过滤salary > 50000，输出(dept, 1) +- **Shuffle阶段**：按dept分组 +- **Reduce阶段**：统计每个dept的count + +### 🎯 Hive的存储格式有哪些？各有什么特点？ + +**Hive支持多种存储格式**，不同格式适用于不同的场景和性能需求。 + +**1. 行存储格式** + +**TextFile** +- 默认格式，纯文本存储 +- 人类可读，便于调试 +- 压缩率低，查询性能一般 +- 适合小数据量、临时表 + +**SequenceFile** +- Hadoop的二进制格式 +- 支持压缩和分割 +- 比TextFile性能好 +- 适合中间数据存储 + +**2. 列存储格式** + +**ORC（Optimized Row Columnar）** +- Hive专门优化的列式存储 +- **优势**： + - 高压缩率（可达70%） + - 内置索引（Min/Max/Bloom Filter） + - 支持向量化查询 + - ACID事务支持 +- **适用场景**：大数据分析、数仓查询 + +**Parquet** +- 通用的列式存储格式 +- **优势**： + - 跨平台兼容性好 + - 高效的编码和压缩 + - 嵌套数据支持好 + - 与Spark集成完善 +- **适用场景**：多引擎数据共享 + +**3. 混合存储格式** + +**Avro** +- 支持模式演化 +- 自描述数据格式 +- 适合数据交换场景 + +**存储格式性能对比**： + +| 格式 | 压缩率 | 查询性能 | 写入性能 | 兼容性 | 适用场景 | +|------|-------|---------|---------|--------|---------| +| TextFile | 低 | 低 | 高 | 最好 | 调试、临时数据 | +| ORC | 高 | 高 | 中 | Hive生态 | 数仓分析 | +| Parquet | 高 | 高 | 中 | 跨引擎 | 多引擎共享 | +| Avro | 中 | 中 | 高 | 好 | 数据交换 | + +**选择建议**： +- **数仓场景**：优先选择ORC +- **多引擎场景**：优先选择Parquet +- **实时写入**：考虑TextFile或Avro +- **存储成本敏感**：选择压缩率高的列式存储 + +### 🎯 Hive的分区和分桶机制是什么？如何优化查询？ + +**分区（Partition）机制**： + +分区是Hive中的**水平分割**技术，将表数据按照某个或多个列的值分割存储在不同的目录中。 + +**分区的优势**： +- **查询优化**：只扫描相关分区，避免全表扫描 +- **数据管理**：便于数据的增删改维护 +- **并行度提升**：不同分区可以并行处理 + +**分区类型**： + +**1. 静态分区** +```sql +-- 创建分区表 +CREATE TABLE sales_data ( + id INT, + product STRING, + amount DOUBLE +) PARTITIONED BY (year INT, month INT) +STORED AS ORC; + +-- 插入数据到指定分区 +INSERT INTO sales_data PARTITION(year=2024, month=1) +VALUES (1, 'phone', 1000.0); +``` + +**2. 动态分区** +```sql +-- 开启动态分区 +SET hive.exec.dynamic.partition=true; +SET hive.exec.dynamic.partition.mode=nonstrict; + +-- 动态分区插入 +INSERT INTO sales_data PARTITION(year, month) +SELECT id, product, amount, year, month FROM source_table; +``` + +**分桶（Bucket）机制**： + +分桶是对数据进行**hash分割**，将相同hash值的数据放在同一个文件中。 + +**分桶的优势**： +- **Join优化**：相同key的数据在同一个桶中，避免shuffle +- **抽样查询**：可以高效地进行数据抽样 +- **负载均衡**：数据均匀分布在各个文件中 + +**分桶示例**： +```sql +-- 创建分桶表 +CREATE TABLE user_data ( + id INT, + name STRING, + age INT +) CLUSTERED BY (id) INTO 10 BUCKETS +STORED AS ORC; + +-- 开启分桶 +SET hive.enforce.bucketing=true; +``` + +**查询优化策略**： + +**1. 分区裁剪（Partition Pruning）** +```sql +-- 好的查询：只扫描特定分区 +SELECT * FROM sales_data +WHERE year = 2024 AND month = 1; + +-- 差的查询：全表扫描 +SELECT * FROM sales_data +WHERE amount > 1000; +``` + +**2. 列裁剪（Column Pruning）** +```sql +-- 只查询需要的列 +SELECT id, product FROM sales_data +WHERE year = 2024; +``` + +**3. 谓词下推（Predicate Pushdown）** +- 将过滤条件下推到存储层 +- 减少数据传输量 + +**4. MapJoin优化** +```sql +-- 小表Join大表优化 +SET hive.auto.convert.join=true; +SET hive.mapjoin.smalltable.filesize=25000000; +``` + +**5. 向量化执行** +```sql +-- 开启向量化查询 +SET hive.vectorized.execution.enabled=true; +SET hive.vectorized.execution.reduce.enabled=true; +``` + +**分区设计最佳实践**： +- 选择**查询频繁**的列作为分区字段 +- 避免**分区过多**（建议<10000个） +- 分区大小控制在**256MB-1GB**之间 +- 使用**多级分区**提高查询效率 + +### 🎯 Apache Kylin是什么？如何实现OLAP预计算？ + +**Apache Kylin是什么？** + +Apache Kylin是一个**开源的分布式分析引擎**，专为超大数据集上的OLAP（联机分析处理）而设计。通过预计算技术，将多维分析查询的响应时间控制在亚秒级别。 + +**核心概念**： + +**1. Cube（数据立方体）** +- Kylin的核心概念，代表多维数据集 +- 包含维度（Dimension）和度量（Measure） +- 通过预计算生成所有可能的维度组合 + +**2. Cuboid** +- Cube的一个子集，代表特定维度组合的聚合数据 +- N个维度可以产生2^N个Cuboid +- Kylin会智能剪枝，减少不必要的Cuboid + +**3. Segment** +- Cube按时间分割的片段 +- 支持增量构建和查询 +- 便于数据管理和维护 + +**Kylin架构组件**： + +**1. Metadata Store** +- 存储Cube定义、Job信息等元数据 +- 通常使用HBase存储 + +**2. Cube Build Engine** +- 基于MapReduce或Spark构建Cube +- 支持全量构建和增量构建 + +**3. Query Engine** +- 接收SQL查询并转换为对预计算结果的查询 +- 支持标准SQL语法 + +**4. REST Server** +- 提供RESTful API +- 支持与BI工具集成 + +### 🎯 Kylin的Cube构建过程是怎样的？ + +**Cube构建概述**： + +Kylin通过预计算将复杂的OLAP查询转换为简单的索引查找，大大提升查询性能。 + +**构建步骤详解**： + +**1. 创建数据模型** +```sql +-- 定义维度表和事实表 +CREATE TABLE fact_sales ( + order_id BIGINT, + customer_id BIGINT, + product_id BIGINT, + order_date DATE, + sales_amount DECIMAL(10,2), + quantity INT +); + +CREATE TABLE dim_customer ( + customer_id BIGINT, + customer_name VARCHAR(100), + city VARCHAR(50), + region VARCHAR(50) +); +``` + +**2. 定义Cube** +```json +{ + "cube_name": "sales_cube", + "model_name": "sales_model", + "dimensions": [ + { + "name": "order_date", + "table": "fact_sales", + "column": "order_date" + }, + { + "name": "customer_city", + "table": "dim_customer", + "column": "city" + }, + { + "name": "customer_region", + "table": "dim_customer", + "column": "region" + } + ], + "measures": [ + { + "name": "total_sales", + "function": { + "expression": "SUM", + "parameter": { + "type": "column", + "value": "sales_amount" + } + } + }, + { + "name": "order_count", + "function": { + "expression": "COUNT_DISTINCT", + "parameter": { + "type": "column", + "value": "order_id" + } + } + } + ] +} +``` + +**3. Cube构建流程** + +**Step 1: 创建Flat Table** +- 将事实表和维度表Join成宽表 +- 包含所有维度和度量字段 + +**Step 2: 生成Cuboid** +``` +维度组合示例（3个维度）： +- [] (全部聚合) +- [date] +- [city] +- [region] +- [date, city] +- [date, region] +- [city, region] +- [date, city, region] +``` + +**Step 3: 分层构建** +- 基于MapReduce的分层聚合 +- 从最细粒度开始向上聚合 +- 利用已有结果计算更粗粒度的聚合 + +**Step 4: 存储结果** +- 将Cuboid结果存储到HBase +- 采用压缩和编码优化存储 + +**4. 构建优化策略** + +**智能剪枝**： +```json +{ + "aggregation_groups": [ + { + "includes": ["date", "city", "region"], + "select_rule": { + "hierarchy_dims": [["region", "city"]], + "mandatory_dims": ["date"], + "joint_dims": [["city", "region"]] + } + } + ] +} +``` + +**增量构建**： +```bash +# 构建增量Segment +curl -X PUT \ + http://kylin-server:7070/kylin/api/cubes/sales_cube/build \ + -H 'Content-Type: application/json' \ + -d '{ + "startTime": 1609459200000, + "endTime": 1609545600000, + "buildType": "BUILD" + }' +``` + +### 🎯 Kylin的查询优化和最佳实践？ + +**查询优化机制**： + +**1. 查询路由** + +**自动Cuboid匹配**： +```sql +-- 原始查询 +SELECT region, SUM(sales_amount) +FROM fact_sales f +JOIN dim_customer d ON f.customer_id = d.customer_id +WHERE order_date BETWEEN '2024-01-01' AND '2024-01-31' +GROUP BY region; + +-- Kylin自动路由到对应的Cuboid +-- 查询时间从分钟级别降到毫秒级别 +``` + +**2. 存储优化** + +**HBase表设计**： +``` +RowKey设计: [Cuboid_ID][维度值组合][时间戳] +列族设计: +- CF_M: 存储度量值 +- CF_D: 存储维度值（可选） +``` + +**压缩策略**： +- 使用字典编码压缩高基数维度 +- 对度量值使用适当的数据类型 +- 启用HBase压缩算法 + +**3. 性能调优参数** + +**构建参数优化**： +```properties +# MapReduce内存配置 +kylin.engine.mr.config-override.mapreduce.map.memory.mb=4096 +kylin.engine.mr.config-override.mapreduce.reduce.memory.mb=6144 + +# 分区数配置 +kylin.engine.mr.config-override.mapreduce.job.reduces=10 + +# 压缩配置 +kylin.engine.mr.config-override.mapreduce.output.compress=true +``` + +**查询参数优化**： +```properties +# 查询缓存 +kylin.query.cache-enabled=true +kylin.query.cache.threshold.duration=2000 + +# 超时设置 +kylin.query.timeout-seconds=300 + +# 结果集限制 +kylin.query.max-return-rows=1000000 +``` + +**4. 监控和维护** + +**Cube健康检查**： +```bash +# 查看Cube状态 +curl -X GET http://kylin-server:7070/kylin/api/cubes + +# 查看构建任务 +curl -X GET http://kylin-server:7070/kylin/api/jobs + +# 查看查询历史 +curl -X GET http://kylin-server:7070/kylin/api/query/history +``` + +**性能指标监控**： +- Cube构建时间和成功率 +- 查询响应时间分布 +- 存储空间使用情况 +- 命中率统计 + +**5. 最佳实践建议** + +**Cube设计原则**： +- 合理设计维度层次结构 +- 避免过多高基数维度 +- 使用聚合组优化Cuboid数量 +- 定期清理过期Segment + +**查询优化建议**： +```sql +-- 好的查询：使用预计算维度 +SELECT region, city, SUM(sales_amount) +FROM sales_cube_table +WHERE order_date >= '2024-01-01' +GROUP BY region, city; + +-- 差的查询：包含未预计算的维度 +SELECT customer_name, SUM(sales_amount) -- customer_name未包含在Cube中 +FROM fact_sales f +JOIN dim_customer d ON f.customer_id = d.customer_id +GROUP BY customer_name; +``` + +**运维管理**： +- 建立Cube构建监控告警 +- 定期评估Cube使用情况 +- 根据查询模式调整Cube设计 +- 制定数据保留和清理策略 + +**Kylin vs 其他OLAP方案对比**： + +| 特性 | Kylin | ClickHouse | Doris | +|------|-------|------------|-------| +| **预计算** | 是 | 否 | 否 | +| **查询延迟** | 亚秒级 | 毫秒级 | 毫秒级 | +| **存储开销** | 高（预计算） | 中等 | 中等 | +| **实时性** | 准实时 | 实时 | 实时 | +| **维度限制** | 有限制 | 无限制 | 无限制 | +| **适用场景** | 固定查询模式 | 灵活查询 | 混合负载 | + +--- + +## 📈 五、OLAP数据库 + +> **核心思想**：Apache Doris是高性能实时分析数据库，基于MPP架构设计，提供高并发、低延迟的OLAP查询能力，支持实时数据写入和复杂分析查询。 + +### 🎯 Apache Doris的核心架构是什么？ + +**Apache Doris是什么？** + +Apache Doris是一个**现代化的MPP分析数据库产品**，仅需亚秒级响应时间即可获得查询结果，有效地支持实时数据分析。主要面向OLAP场景，解决报表和多维分析的需求。 + +**核心架构组件**： + +**1. Frontend (FE)** +- **Master FE**：负责元数据管理、查询计划生成、系统协调 +- **Follower FE**：提供元数据读取服务，分担查询压力 +- **Observer FE**：只读副本，不参与选举，用于扩展查询能力 + +**2. Backend (BE)** +- 负责数据存储和查询执行 +- 每个BE节点存储数据的分片副本 +- 执行具体的计算任务 + +**3. Broker** +- 用于从外部系统导入数据 +- 支持HDFS、S3等存储系统 +- 可选组件，根据需要部署 + +**核心特性**： + +**MPP架构**： +- 大规模并行处理，查询任务分布在多个节点执行 +- 支持水平扩展，节点数量可达数百个 +- 自动数据分片和副本管理 + +**列式存储**： +- 采用列式存储格式，压缩率高 +- 支持向量化执行，SIMD加速 +- 针对分析查询优化的存储结构 + +**实时写入**： +- 支持高频实时数据写入 +- 数据写入后毫秒级可查 +- 支持批量和流式数据导入 + +### 🎯 Doris的数据模型有哪些？各适用什么场景？ + +**Doris数据模型概述**： + +Doris提供三种数据模型，分别适用于不同的业务场景和查询模式。 + +**1. Duplicate Model（明细模型）** + +**特点**： +- 保留数据的所有明细记录 +- 不进行任何聚合操作 +- 支持完整的数据查询 + +**适用场景**： +- 需要保留原始明细数据 +- 日志分析和用户行为分析 +- 需要灵活的数据查询 + +**建表示例**： +```sql +CREATE TABLE user_behavior ( + `user_id` LARGEINT NOT NULL COMMENT "用户ID", + `event_time` DATETIME NOT NULL COMMENT "事件时间", + `event_type` VARCHAR(32) NOT NULL COMMENT "事件类型", + `page_url` VARCHAR(512) COMMENT "页面URL", + `duration` INT COMMENT "停留时长" +) DUPLICATE KEY(`user_id`, `event_time`) +DISTRIBUTED BY HASH(`user_id`) BUCKETS 32 +PROPERTIES ( + "replication_num" = "3" +); +``` + +**2. Aggregate Model（聚合模型）** + +**特点**： +- 相同Key的数据会自动聚合 +- 支持SUM、MAX、MIN、REPLACE等聚合函数 +- 适合预聚合场景 + +**适用场景**： +- 指标统计分析 +- 实时报表和监控大盘 +- 需要预聚合的场景 + +**建表示例**： +```sql +CREATE TABLE sales_stats ( + `date` DATE NOT NULL COMMENT "日期", + `city` VARCHAR(32) NOT NULL COMMENT "城市", + `category` VARCHAR(64) NOT NULL COMMENT "商品类别", + `sales_amount` DECIMAL(15,2) SUM DEFAULT "0" COMMENT "销售额", + `order_count` BIGINT SUM DEFAULT "0" COMMENT "订单数", + `max_price` DECIMAL(10,2) MAX DEFAULT "0" COMMENT "最高价格" +) AGGREGATE KEY(`date`, `city`, `category`) +DISTRIBUTED BY HASH(`city`) BUCKETS 16 +PROPERTIES ( + "replication_num" = "3" +); +``` + +**3. Unique Model（主键模型）** + +**特点**： +- 支持主键约束，相同Key的数据会覆盖 +- 支持部分列更新 +- 类似传统数据库的UPSERT语义 + +**适用场景**： +- 用户画像数据 +- 订单状态更新 +- 需要数据去重的场景 + +**建表示例**： +```sql +CREATE TABLE user_profile ( + `user_id` LARGEINT NOT NULL COMMENT "用户ID", + `username` VARCHAR(64) COMMENT "用户名", + `age` INT COMMENT "年龄", + `city` VARCHAR(32) COMMENT "城市", + `last_login` DATETIME COMMENT "最后登录时间", + `total_amount` DECIMAL(15,2) COMMENT "总消费金额" +) UNIQUE KEY(`user_id`) +DISTRIBUTED BY HASH(`user_id`) BUCKETS 32 +PROPERTIES ( + "replication_num" = "3", + "enable_unique_key_merge_on_write" = "true" +); +``` + +**模型选择建议**： + +| 业务场景 | 推荐模型 | 原因 | +|---------|---------|------| +| 日志分析 | Duplicate | 需要保留完整明细 | +| 实时报表 | Aggregate | 自动聚合，查询快速 | +| 用户画像 | Unique | 支持数据更新 | +| 监控指标 | Aggregate | 预聚合，降低存储 | + +### 🎯 Doris的数据导入方式有哪些？ + +**Doris数据导入概述**： + +Doris提供多种数据导入方式，支持批量导入和实时导入，满足不同的数据接入需求。 + +**1. Stream Load（流式导入）** + +**特点**： +- 同步导入方式，提交后立即返回结果 +- 支持CSV、JSON格式 +- 适合小批量高频导入 + +**使用示例**： +```bash +# CSV数据导入 +curl -v --location-trusted -u user:password \ + -H "format: csv" \ + -H "column_separator:," \ + -T data.csv \ + http://fe_host:fe_http_port/api/db_name/table_name/_stream_load + +# JSON数据导入 +curl -v --location-trusted -u user:password \ + -H "format: json" \ + -T data.json \ + http://fe_host:fe_http_port/api/db_name/table_name/_stream_load +``` + +**2. Broker Load（离线导入）** + +**特点**： +- 异步导入方式，适合大批量数据 +- 支持从HDFS、S3等分布式存储导入 +- 具有容错和重试机制 + +**使用示例**： +```sql +-- 从HDFS导入数据 +LOAD LABEL db_name.label_name ( + DATA INFILE("hdfs://namenode:port/path/to/file.csv") + INTO TABLE table_name + COLUMNS TERMINATED BY "," + (col1, col2, col3) +) WITH BROKER "broker_name" ( + "username" = "hdfs_user", + "password" = "hdfs_password" +); +``` + +**3. Routine Load（例行导入）** + +**特点**： +- 持续消费Kafka数据流 +- 支持实时数据导入 +- 自动管理消费进度 + +**使用示例**： +```sql +-- 创建Routine Load任务 +CREATE ROUTINE LOAD db_name.job_name ON table_name +COLUMNS(col1, col2, col3) +PROPERTIES ( + "desired_concurrent_number"="3", + "max_batch_interval" = "20", + "max_batch_rows" = "300000" +) +FROM KAFKA ( + "kafka_broker_list" = "broker1:9092,broker2:9092", + "kafka_topic" = "topic_name", + "property.group.id" = "group_id" +); +``` + +**4. Insert Into（SQL导入）** + +**特点**： +- 标准SQL语法 +- 适合小量数据和测试 +- 支持从其他表导入 + +**使用示例**： +```sql +-- 直接插入数据 +INSERT INTO table_name VALUES +(1, 'name1', 100), +(2, 'name2', 200); + +-- 从其他表导入 +INSERT INTO target_table +SELECT col1, col2, col3 FROM source_table +WHERE condition; +``` + +**导入方式选择建议**： + +| 场景 | 推荐方式 | 数据量 | 实时性 | +|------|---------|-------|--------| +| 实时数据流 | Routine Load | 中等 | 秒级 | +| 批量ETL | Broker Load | 大量 | 分钟级 | +| 小批量同步 | Stream Load | 小量 | 实时 | +| 测试数据 | Insert Into | 很小 | 实时 | + +### 🎯 Doris的查询优化和性能调优策略？ + +**查询优化策略**： + +**1. 分区分桶优化** + +**分区设计**： +```sql +-- 按日期分区 +CREATE TABLE orders ( + `order_id` BIGINT, + `order_date` DATE, + `customer_id` BIGINT, + `amount` DECIMAL(10,2) +) DUPLICATE KEY(`order_id`) +PARTITION BY RANGE(`order_date`) ( + PARTITION p20240101 VALUES [('2024-01-01'), ('2024-01-02')), + PARTITION p20240102 VALUES [('2024-01-02'), ('2024-01-03')) +) +DISTRIBUTED BY HASH(`customer_id`) BUCKETS 32; +``` + +**分桶优化**： +- 选择高基数列作为分桶键 +- 分桶数建议为BE节点数的2-4倍 +- 避免数据倾斜 + +**2. 索引优化** + +**前缀索引**： +```sql +-- 建表时指定前缀索引长度 +CREATE TABLE user_data ( + `user_id` BIGINT, + `username` VARCHAR(64), + `email` VARCHAR(128) +) DUPLICATE KEY(`user_id`, `username`) +PROPERTIES ( + "short_key" = "2" -- 前缀索引包含前2列 +); +``` + +**BloomFilter索引**： +```sql +-- 为高基数列创建BloomFilter +ALTER TABLE table_name SET ("bloom_filter_columns" = "user_id,email"); +``` + +**3. 查询优化技巧** + +**列裁剪**： +```sql +-- 好的查询：只查询需要的列 +SELECT user_id, username FROM user_table +WHERE age > 18; + +-- 差的查询：查询所有列 +SELECT * FROM user_table WHERE age > 18; +``` + +**分区裁剪**： +```sql +-- 查询中包含分区字段 +SELECT * FROM orders +WHERE order_date = '2024-01-01' + AND customer_id = 12345; +``` + +**4. 系统参数调优** + +**FE配置优化**： +```properties +# FE内存配置 +JAVA_OPTS="-Xmx16g -XX:+UseG1GC" + +# 查询超时设置 +query_timeout = 300 + +# 并发控制 +max_conn_per_user = 100 +``` + +**BE配置优化**： +```properties +# BE内存配置 +mem_limit = 80% + +# 查询并发度 +scan_thread_nice_value = 1 +max_scan_key_num = 1024 + +# 存储配置 +default_num_rows_per_column_file_block = 1024 +``` + +**5. 物化视图优化** + +**创建物化视图**： +```sql +-- 为常用聚合查询创建物化视图 +CREATE MATERIALIZED VIEW sales_agg AS +SELECT + date_trunc(order_date, 'day') as order_day, + region, + SUM(amount) as total_amount, + COUNT(*) as order_count +FROM orders +GROUP BY order_day, region; +``` + +**性能监控**： +```sql +-- 查看查询profile +SET enable_profile = true; +SELECT * FROM table_name WHERE condition; +SHOW QUERY PROFILE; + +-- 查看执行计划 +EXPLAIN SELECT * FROM table_name WHERE condition; +``` + +### 🎯 主流OLAP数据库对比分析 + +**OLAP技术选型概述**： + +在大数据分析领域，选择合适的OLAP数据库至关重要。不同的OLAP解决方案在架构设计、性能特点、使用场景上各有优势。 + +**主流OLAP数据库分类**： + +**1. 预计算型OLAP** +- **Apache Kylin**：基于Cube预计算 +- **Apache Druid**：时序数据预聚合 + +**2. MPP架构OLAP** +- **Apache Doris**：实时OLAP数据库 +- **ClickHouse**：列式分析数据库 +- **Greenplum**：分布式数据仓库 + +**3. 存储计算分离型** +- **Presto/Trino**：分布式SQL查询引擎 +- **Apache Impala**：高性能SQL引擎 + +**详细对比分析**： + +| 对比维度 | Apache Doris | Apache Kylin | ClickHouse | Presto/Trino | Apache Druid | +|----------|-------------|--------------|------------|-------------|-------------| +| **架构类型** | MPP实时OLAP | 预计算OLAP | MPP列存 | 查询引擎 | 时序OLAP | +| **存储模式** | 列式存储 | 预计算Cube | 列式存储 | 存储计算分离 | 列式+时序 | +| **查询延迟** | 毫秒-秒级 | 亚秒级 | 毫秒级 | 秒-分钟级 | 毫秒级 | +| **数据写入** | 实时写入 | 批量构建 | 实时写入 | 只查询 | 实时摄取 | +| **扩展性** | 线性扩展 | 水平扩展 | 线性扩展 | 水平扩展 | 水平扩展 | +| **SQL兼容** | 标准SQL | 标准SQL | 类SQL | 标准SQL | JSON查询 | +| **学习成本** | 中等 | 较高 | 中等 | 较低 | 较高 | + +**核心技术特点对比**： + +**Apache Doris vs ClickHouse**： + +| 特性 | Apache Doris | ClickHouse | +|------|-------------|------------| +| **架构设计** | FE/BE分离架构 | 单一进程架构 | +| **数据模型** | 多种模型支持 | 表引擎丰富 | +| **并发处理** | 高并发OLAP | 高吞吐分析 | +| **运维复杂度** | 相对简单 | 配置复杂 | +| **生态集成** | Java生态 | C++生态 | +| **适用场景** | 实时报表、用户画像 | 日志分析、指标监控 | + +**Apache Kylin vs Druid**： + +| 特性 | Apache Kylin | Apache Druid | +|------|-------------|-------------| +| **预计算方式** | 多维Cube | 时序聚合 | +| **时间处理** | 离散时间 | 连续时序 | +| **维度支持** | 有限维度 | 灵活维度 | +| **实时性** | 准实时 | 实时 | +| **存储开销** | 较高 | 中等 | +| **适用场景** | 固定报表、BI分析 | 监控大屏、实时分析 | + +**技术选型决策树**： + +``` +数据分析需求 +├── 实时要求高（毫秒级） +│ ├── 时序数据为主 → Druid +│ ├── 复杂分析查询 → ClickHouse +│ └── 混合负载 → Doris +├── 查询模式固定 +│ ├── 多维分析 → Kylin +│ └── 时序分析 → Druid +├── 数据源多样化 +│ ├── 湖仓一体 → Trino +│ └── 传统数仓 → Greenplum +└── 开发资源有限 + ├── 运维简单 → Doris + └── 功能全面 → ClickHouse +``` + +**场景化选型建议**： + +**1. 实时报表场景** +``` +推荐方案：Apache Doris +理由： +- 支持实时数据写入和查询 +- 标准SQL，学习成本低 +- MPP架构，查询性能优秀 +- 支持多种数据模型 +``` + +**2. 日志分析场景** +``` +推荐方案：ClickHouse +理由： +- 极高的压缩比和查询性能 +- 丰富的表引擎和函数 +- 适合大量数据的聚合分析 +- 社区活跃，文档完善 +``` + +**3. 固定报表场景** +``` +推荐方案：Apache Kylin +理由： +- 预计算提供亚秒级查询 +- 适合固定的多维分析 +- 与BI工具集成良好 +- 查询性能稳定可预期 +``` + +**4. 时序监控场景** +``` +推荐方案：Apache Druid +理由： +- 专为时序数据设计 +- 支持实时数据摄取 +- 灵活的时间聚合 +- 适合监控和告警 +``` + +**5. 多数据源查询** +``` +推荐方案：Trino/Presto +理由： +- 支持多种数据源 +- 存储计算分离 +- 标准SQL接口 +- 灵活的查询优化 +``` + +**性能基准测试对比**： + +**查询性能对比**（基于TPC-H 100GB数据集）： + +| 查询类型 | Doris | ClickHouse | Kylin | Presto | +|---------|-------|------------|-------|--------| +| **简单聚合** | 0.5s | 0.2s | 0.1s | 2.0s | +| **复杂Join** | 3.0s | 2.5s | 0.5s | 8.0s | +| **多维分组** | 1.2s | 0.8s | 0.2s | 5.0s | +| **时序查询** | 2.0s | 1.0s | - | 6.0s | + +**存储压缩比对比**： + +| 数据库 | 压缩比 | 存储开销 | 查询性能 | +|--------|--------|----------|----------| +| **原始数据** | 1:1 | 100% | - | +| **Doris** | 5:1 | 20% | 高 | +| **ClickHouse** | 10:1 | 10% | 极高 | +| **Kylin** | 3:1 | 300%（预计算） | 极高 | +| **Druid** | 8:1 | 15% | 高 | + +**最佳实践总结**： + +**选型原则**： +1. **性能优先**：ClickHouse、Doris +2. **实时性优先**：Druid、Doris +3. **易用性优先**：Doris、Presto +4. **成本优先**：Kylin（查询模式固定时） +5. **生态兼容**：Presto（多数据源） + +**部署建议**： +- **小规模团队**：选择Doris或ClickHouse +- **大型企业**：可考虑多种方案组合 +- **云环境**：优先考虑托管服务 +- **混合云**：选择开源方案保持灵活性 + +**技术演进趋势**： +- **云原生化**：Serverless OLAP服务 +- **湖仓一体**：统一存储和计算 +- **AI融合**：智能查询优化 +- **实时化**：流批一体处理 + +--- + +## 🚀 六、消息队列（Kafka核心） + +> **核心思想**：Kafka是分布式流处理平台的基石，通过分布式、高吞吐、低延迟的消息传递，连接数据的生产者和消费者。 + +### 🎯 Kafka的核心架构是什么？如何保证高性能？ + +**Kafka是什么？** + +Apache Kafka是一个**分布式流处理平台**，提供高吞吐量、低延迟的消息发布订阅服务，广泛用于实时数据管道和流式应用。 + +**Kafka核心架构**： + +**1. Broker（服务节点）** +- Kafka集群中的服务器节点 +- 负责存储和转发消息 +- 通过ZooKeeper协调集群状态 + +**2. Topic（主题）** +- 消息的逻辑分类 +- 生产者发送消息到Topic +- 消费者从Topic订阅消息 + +**3. Partition（分区）** +- Topic的物理分割单位 +- 每个分区是一个有序的消息队列 +- 分区内消息有序，分区间无序 + +**4. Replica（副本）** +- 每个分区可以有多个副本 +- **Leader Replica**：处理读写请求 +- **Follower Replica**：从Leader同步数据 + +**5. Producer（生产者）** +- 发送消息到Kafka Topic +- 可以指定分区策略 + +**6. Consumer（消费者）** +- 从Kafka Topic消费消息 +- 可以组成Consumer Group + +**高性能设计原理**： + +**1. 顺序写入** +- 消息追加到日志文件末尾 +- 避免随机I/O，发挥磁盘顺序读写优势 +- 顺序写性能接近内存 + +**2. 零拷贝（Zero Copy）** +- 使用sendfile()系统调用 +- 数据直接从内核空间传输到网络 +- 避免用户空间和内核空间的数据拷贝 + +**3. 批量处理** +- 生产者批量发送消息 +- 减少网络请求次数 +- 提升整体吞吐量 + +**4. 分区并行** +- 多个分区并行读写 +- 提高并发处理能力 +- 支持水平扩展 + +**5. 页缓存利用** +- 依赖操作系统页缓存 +- 不在JVM堆中缓存数据 +- 避免GC影响性能 + +**6. 压缩** +- 支持多种压缩算法（Gzip、Snappy、LZ4、ZSTD） +- 减少网络传输和存储开销 + +**性能调优参数**： +```properties +# 批量大小 +batch.size=16384 +linger.ms=5 + +# 压缩 +compression.type=lz4 + +# 副本确认 +acks=1 + +# 缓冲区大小 +send.buffer.bytes=131072 +receive.buffer.bytes=131072 +``` + +### 🎯 Kafka如何保证消息的可靠性？ + +**可靠性挑战**： +在分布式环境下，网络故障、节点宕机、磁盘损坏等问题都可能导致消息丢失或重复，Kafka需要在性能和可靠性之间找到平衡。 + +**可靠性保证机制**： + +**1. 副本机制（Replication）** + +**ISR（In-Sync Replica）集合**： +- 包含Leader和同步的Follower副本 +- 只有ISR中的副本才能成为Leader +- 保证数据不丢失 + +**副本同步流程**： +- Producer发送消息到Leader +- Leader写入本地日志 +- Follower从Leader拉取消息 +- Follower写入本地日志并发送ACK给Leader + +**2. ACK确认机制** + +**acks=0**： +- 生产者不等待任何确认 +- 性能最高，可靠性最低 +- 可能丢失消息 + +**acks=1**： +- 等待Leader确认 +- 性能和可靠性的平衡 +- Leader故障可能丢失消息 + +**acks=-1/all**： +- 等待ISR中所有副本确认 +- 可靠性最高，性能最低 +- 配合`min.insync.replicas`使用 + +**3. 消息重试机制** + +```properties +# 生产者重试配置 +retries=Integer.MAX_VALUE +retry.backoff.ms=100 +request.timeout.ms=30000 +delivery.timeout.ms=120000 +``` + +**4. 幂等性保证** + +```properties +# 开启幂等性 +enable.idempotence=true +``` + +- 生产者会为每个消息分配唯一的Sequence ID +- Broker检测重复消息并丢弃 +- 保证消息不重复 + +**5. 事务支持** + +```java +// 事务生产者 +Properties props = new Properties(); +props.put("transactional.id", "my-transactional-id"); +props.put("enable.idempotence", true); + +KafkaProducer producer = new KafkaProducer<>(props); +producer.initTransactions(); + +try { + producer.beginTransaction(); + producer.send(new ProducerRecord<>("topic", "key", "value")); + producer.commitTransaction(); +} catch (Exception e) { + producer.abortTransaction(); +} +``` + +**6. 消费者可靠性** + +**手动提交Offset**： +```java +// 手动提交确保处理完成后再提交 +consumer.poll(Duration.ofMillis(1000)); +// 处理消息... +consumer.commitSync(); +``` + +**消费者组故障转移**： +- 消费者故障时，其他消费者接管分区 +- 通过心跳机制检测消费者状态 + +**可靠性配置最佳实践**： + +**高可靠性配置**： +```properties +# 生产者 +acks=all +retries=Integer.MAX_VALUE +enable.idempotence=true +min.insync.replicas=2 + +# 消费者 +enable.auto.commit=false +isolation.level=read_committed +``` + +**高性能配置**： +```properties +# 生产者 +acks=1 +batch.size=32768 +linger.ms=10 + +# 消费者 +enable.auto.commit=true +``` + +### 🎯 Kafka消费者组的工作原理是什么？ + +**Consumer Group是什么？** + +Consumer Group是Kafka中**消费者的逻辑分组**，同一个Consumer Group中的消费者协作消费Topic的消息，每个分区只能被组内一个消费者消费。 + +**核心工作原理**： + +**1. 分区分配策略** + +**Range分配策略（默认）**： +- 按分区范围分配给消费者 +- 可能导致分配不均衡 +- 适合分区数是消费者数倍数的情况 + +**RoundRobin分配策略**： +- 轮询方式分配分区给消费者 +- 分配更均衡 +- 适合消费者订阅相同Topic的情况 + +**Sticky分配策略**： +- 尽量保持分区分配的粘性 +- Rebalance时减少分区重新分配 +- 提高消费效率 + +**2. Coordinator协调机制** + +**Group Coordinator**： +- 每个Consumer Group对应一个Coordinator +- 负责管理组成员和分区分配 +- 处理心跳和提交Offset + +**工作流程**： +- 消费者加入Consumer Group +- Coordinator选举Group Leader +- Group Leader执行分区分配 +- Coordinator广播分配结果 + +**3. Rebalance机制** + +**触发条件**： +- 新消费者加入组 +- 消费者离开组（正常关闭或故障） +- Topic分区数变化 +- 消费者订阅的Topic变化 + +**Rebalance流程**： +``` +1. 消费者停止消费消息 +2. 消费者向Coordinator发送JoinGroup请求 +3. Coordinator收集所有消费者信息 +4. Coordinator选择Group Leader +5. Group Leader执行分区分配算法 +6. Coordinator广播分配结果 +7. 消费者根据分配结果开始消费 +``` + +**4. Offset管理** + +**自动提交**： +```properties +enable.auto.commit=true +auto.commit.interval.ms=5000 +``` + +**手动提交**： +```java +// 同步提交 +consumer.commitSync(); + +// 异步提交 +consumer.commitAsync((offsets, exception) -> { + if (exception != null) { + logger.error("Commit failed", exception); + } +}); +``` + +**5. 心跳机制** + +```properties +# 心跳间隔 +heartbeat.interval.ms=3000 + +# 会话超时 +session.timeout.ms=10000 + +# 最大拉取间隔 +max.poll.interval.ms=300000 +``` + +**消费者代码示例**： +```java +Properties props = new Properties(); +props.put("bootstrap.servers", "localhost:9092"); +props.put("group.id", "my-group"); +props.put("auto.offset.reset", "earliest"); +props.put("partition.assignment.strategy", + "org.apache.kafka.clients.consumer.StickyAssignor"); + +KafkaConsumer consumer = new KafkaConsumer<>(props); +consumer.subscribe(Arrays.asList("my-topic")); + +while (true) { + ConsumerRecords records = consumer.poll(Duration.ofMillis(1000)); + for (ConsumerRecord record : records) { + // 处理消息 + System.out.printf("offset = %d, key = %s, value = %s%n", + record.offset(), record.key(), record.value()); + } + consumer.commitAsync(); +} +``` + +**最佳实践**： +- 合理设置消费者数量（不超过分区数） +- 选择合适的分区分配策略 +- 监控Consumer Lag指标 +- 处理Rebalance异常情况 +- 合理设置心跳和会话超时参数 + +--- + +## 🔧 六、资源调度（YARN核心） + +> **核心思想**：YARN是Hadoop 2.0的资源管理框架，实现了计算与存储分离，支持多种计算框架在同一集群中运行。 + +### 🎯 YARN的架构原理是什么？与Hadoop 1.0相比有什么优势？ + +**YARN是什么？** + +YARN（Yet Another Resource Negotiator）是Hadoop 2.0引入的**资源管理框架**，负责集群资源的统一管理和调度，支持多种计算框架。 + +**YARN核心架构**： + +**1. ResourceManager（资源管理器）** +- 集群的全局资源管理者 +- **Scheduler**：负责资源分配，不监控应用状态 +- **ApplicationManager**：管理应用的生命周期 + +**2. NodeManager（节点管理器）** +- 单个节点的资源管理者 +- 监控节点资源使用情况 +- 管理Container的生命周期 +- 向ResourceManager汇报节点状态 + +**3. ApplicationMaster（应用管理器）** +- 每个应用有一个AM +- 向ResourceManager申请资源 +- 与NodeManager通信启动Container +- 监控任务执行状态 + +**4. Container（容器）** +- 资源分配的基本单位 +- 封装CPU、内存等资源 +- 在NodeManager上运行具体任务 + +**YARN工作流程**： + +``` +1. Client提交应用到ResourceManager +2. ResourceManager分配Container启动ApplicationMaster +3. ApplicationMaster向ResourceManager注册 +4. ApplicationMaster请求资源运行任务 +5. ResourceManager分配Container给ApplicationMaster +6. ApplicationMaster与NodeManager通信启动Container +7. Container运行具体任务 +8. ApplicationMaster监控任务进度 +9. 应用完成后ApplicationMaster注销 +``` + +**YARN vs Hadoop 1.0对比**： + +| 对比维度 | Hadoop 1.0 | YARN | +|---------|------------|------| +| **架构** | JobTracker + TaskTracker | ResourceManager + NodeManager + ApplicationMaster | +| **扩展性** | 4000节点瓶颈 | 万级节点支持 | +| **计算框架** | 仅支持MapReduce | 支持MapReduce、Spark、Storm等 | +| **资源利用率** | 静态资源分配 | 动态资源分配 | +| **容错性** | JobTracker单点故障 | ResourceManager HA | +| **多租户** | 不支持 | 支持资源隔离和配额管理 | + +**YARN优势**： + +**1. 资源利用率提升** +- 动态资源分配 +- 不同框架共享集群资源 +- 避免资源静态划分的浪费 + +**2. 扩展性增强** +- 分离资源管理和应用管理 +- 单个ResourceManager可管理数万节点 +- ApplicationMaster分散了JobTracker压力 + +**3. 多框架支持** +- 统一资源管理平台 +- MapReduce、Spark、Flink等都可运行 +- 避免重复建设集群 + +**4. 容错性改进** +- ResourceManager支持HA +- ApplicationMaster故障可重启 +- Container故障不影响其他任务 + +### 🎯 YARN的资源调度器有哪些？各有什么特点？ + +**YARN调度器概述**： + +YARN的调度器负责将集群资源分配给各个应用，不同的调度器采用不同的分配策略，适用于不同的应用场景。 + +**1. FIFO Scheduler（先进先出调度器）** + +**特点**： +- 按提交时间顺序分配资源 +- 简单易理解 +- 不支持优先级 + +**适用场景**： +- 小集群或测试环境 +- 单用户环境 +- 对公平性要求不高的场景 + +**配置示例**： +```xml + + yarn.resourcemanager.scheduler.class + org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.FifoScheduler + +``` + +**2. Capacity Scheduler（容量调度器）** + +**核心特点**： +- **层次化队列**：支持多级队列嵌套 +- **容量保证**：每个队列有最小容量保证 +- **弹性资源**：队列可借用其他队列空闲资源 +- **多租户**：不同队列可配置不同用户和权限 + +**队列配置**： +```xml + + + yarn.scheduler.capacity.resource-calculator + org.apache.hadoop.yarn.util.resource.DominantResourceCalculator + + + + + yarn.scheduler.capacity.root.queues + production,development,urgent + + + + + yarn.scheduler.capacity.root.production.capacity + 60 + + + yarn.scheduler.capacity.root.development.capacity + 30 + + + yarn.scheduler.capacity.root.urgent.capacity + 10 + + + + + yarn.scheduler.capacity.root.production.maximum-capacity + 80 + +``` + +**适用场景**： +- 企业多部门共享集群 +- 需要资源隔离的场景 +- 有SLA要求的生产环境 + +**3. Fair Scheduler（公平调度器）** + +**核心特点**： +- **公平共享**：资源在活跃应用间公平分配 +- **抢占机制**：资源不足时可抢占其他应用资源 +- **权重支持**：不同队列可设置不同权重 +- **延迟调度**：支持数据本地性优化 + +**配置示例**： +```xml + + + yarn.resourcemanager.scheduler.class + org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler + + + + + yarn.scheduler.fair.allocation.file + ${HADOOP_CONF_DIR}/fair-scheduler.xml + +``` + +**fair-scheduler.xml配置**： +```xml + + + 10000 mb,10 vcores + 90000 mb,90 vcores + 3.0 + + + + 5000 mb,5 vcores + 50000 mb,50 vcores + 1.0 + + + + 60 + 0.5 + +``` + +**调度器对比**： + +| 特性 | FIFO | Capacity | Fair | +|------|------|----------|------| +| **公平性** | 无 | 队列内公平 | 全局公平 | +| **资源保证** | 无 | 容量保证 | 最小资源保证 | +| **抢占** | 不支持 | 不支持 | 支持 | +| **多租户** | 不支持 | 支持 | 支持 | +| **配置复杂度** | 简单 | 中等 | 复杂 | +| **适用场景** | 测试环境 | 企业生产 | 共享集群 | + +**选择建议**： +- **FIFO**：测试和开发环境 +- **Capacity**：多部门企业环境，需要严格资源隔离 +- **Fair**：共享集群环境，需要动态公平分配 + +**调度器优化策略**： +- 合理设置队列容量和权重 +- 启用资源抢占提高资源利用率 +- 配置适当的调度间隔 +- 监控队列资源使用情况 +- 根据业务特点调整参数 + +--- + +## 💼 七、实战场景题（项目经验） + +> **核心思想**：实战场景题是面试的重点，考察的是你在实际项目中运用大数据技术解决业务问题的能力。 + +### 🎯 如何设计一个实时数据处理架构？ + +**业务场景**： +假设要设计一个**电商实时推荐系统**，需要处理用户行为数据（点击、浏览、购买），实时更新用户画像和商品推荐。 + +**架构设计思路**： + +**1. 数据接入层** +``` +用户行为 → 埋点SDK → 消息队列(Kafka) → 实时处理 +``` + +**技术选型**： +- **数据收集**：Flume、Logstash、自研Agent +- **消息队列**：Kafka（高吞吐、低延迟） +- **数据格式**：Avro/JSON（结构化数据） + +**2. 实时计算层** +``` +Kafka → Flink/Spark Streaming → 实时特征计算 → 存储层 +``` + +**Flink实时处理示例**： +```java +// 用户行为流 +DataStream behaviorStream = env + .addSource(new FlinkKafkaConsumer<>("user-behavior", + new UserBehaviorSchema(), properties)) + .assignTimestampsAndWatermarks( + WatermarkStrategy.forBoundedOutOfOrderness(Duration.ofSeconds(10)) + .withTimestampAssigner((event, timestamp) -> event.getTimestamp())); + +// 实时特征计算 +DataStream userFeatures = behaviorStream + .keyBy(UserBehavior::getUserId) + .window(TumblingEventTimeWindows.of(Time.minutes(5))) + .aggregate(new UserFeatureAggregator()); + +// 输出到存储系统 +userFeatures.addSink(new RedisSink<>()); +``` + +**3. 存储层** +- **实时存储**：Redis/HBase（毫秒级读写） +- **离线存储**：HDFS/S3（历史数据存档） +- **OLAP存储**：ClickHouse/Druid（实时分析查询） + +**4. 服务层** +- **推荐服务**：基于实时特征的推荐算法 +- **A/B测试**：实时效果监控和策略调整 +- **API网关**：统一接口管理 + +**架构优化考虑**： + +**性能优化**： +- Kafka分区数 = 消费者并发度 +- Flink并行度根据数据量动态调整 +- Redis集群化部署，避免热点数据 + +**容错处理**： +- Kafka多副本保证数据不丢失 +- Flink Checkpoint机制保证exactly-once +- 多活部署避免单点故障 + +**扩展性设计**： +- 微服务架构，各组件独立扩展 +- 消息队列支持水平扩展 +- 存储分片策略支持数据增长 + +### 🎯 大数据平台的技术选型如何考虑？ + +**技术选型的考虑维度**： + +**1. 业务需求分析** + +**数据量级**： +- **TB级别**：单机或小集群可处理 +- **PB级别**：需要分布式大数据技术 +- **EB级别**：需要专业的大数据架构 + +**实时性要求**： +- **离线批处理**：T+1数据处理，选择Hadoop/Spark +- **准实时**：分钟级延迟，选择Spark Streaming +- **实时**：秒级延迟，选择Flink/Storm + +**查询模式**： +- **OLTP**：高并发事务，选择传统数据库 +- **OLAP**：复杂分析查询，选择数据仓库 +- **混合负载**：选择HTAP数据库 + +**2. 技术选型矩阵** + +**存储技术选型**： + +| 数据类型 | 结构化数据 | 半结构化数据 | 非结构化数据 | +|---------|-----------|-------------|-------------| +| **热数据** | MySQL/PostgreSQL | ElasticSearch | 对象存储+CDN | +| **温数据** | Hive/Presto | ElasticSearch | HDFS | +| **冷数据** | 数据湖(Delta Lake) | 数据湖 | 对象存储 | + +**计算框架选型**： + +| 场景 | 批处理 | 流处理 | 交互式查询 | 机器学习 | +|------|-------|-------|-----------|---------| +| **推荐方案** | Spark | Flink | Presto/Trino | Spark MLlib | +| **备选方案** | MapReduce | Kafka Streams | ClickHouse | TensorFlow on Spark | + +**3. 项目实践案例** + +**电商数据平台架构**： + +``` +数据源层： +- 业务数据库（MySQL） +- 埋点日志（Nginx/App） +- 第三方API数据 + +数据接入层： +- 离线：Sqoop/DataX（数据库） + Flume（日志） +- 实时：Kafka + Canal（binlog） + +数据存储层： +- 数据湖：HDFS（原始数据） +- 数据仓库：Hive（结构化数据） +- 实时存储：HBase/Cassandra + +数据计算层： +- 离线计算：Spark（ETL + 机器学习） +- 实时计算：Flink（实时指标计算） +- 即席查询：Presto（数据探索） + +数据服务层： +- API网关：Spring Cloud Gateway +- 缓存层：Redis Cluster +- 搜索引擎：ElasticSearch +``` + +**技术选型决策过程**： + +**Step 1：需求调研** +- 数据量评估：日增1TB，总量100TB +- 查询QPS：1000/s，延迟<200ms +- 用户规模：100万DAU + +**Step 2：POC验证** +- 搭建小规模测试环境 +- 压测验证性能指标 +- 评估开发和运维成本 + +**Step 3：架构设计** +- 考虑技术栈兼容性 +- 评估团队技术能力 +- 制定迁移和扩容方案 + +**选型最佳实践**： +- **优先选择成熟稳定的技术** +- **考虑团队技术栈和学习成本** +- **关注开源社区活跃度** +- **评估商业支持和服务** +- **制定技术演进路线图** + +### 🎯 如何处理数据倾斜问题？ + +**数据倾斜是什么？** + +数据倾斜是指在分布式计算中，**数据分布不均匀**，导致某些节点处理的数据量远大于其他节点，成为性能瓶颈。 + +**数据倾斜的表现**： +- 作业执行时间过长 +- 某些Task执行时间远超其他Task +- 内存溢出（OOM）错误 +- 集群资源利用率不均 + +**常见倾斜场景**： + +**1. Join倾斜** +```sql +-- 大表Join小表，某个key数据量巨大 +SELECT * +FROM big_table a +JOIN small_table b ON a.user_id = b.user_id +WHERE a.date = '2024-01-01' +``` + +**2. GroupBy倾斜** +```sql +-- 某个分组的数据量过大 +SELECT user_type, COUNT(*) +FROM user_behavior +WHERE date = '2024-01-01' +GROUP BY user_type +``` + +**3. 分区倾斜** +- 按日期分区，某些日期数据量特别大 +- Hash分区，某些key的hash值集中 + +**数据倾斜解决方案**： + +**1. 预处理阶段优化** + +**数据采样分析**： +```python +# Spark数据采样 +sample_df = df.sample(0.01) +skew_keys = sample_df.groupBy("key").count() \ + .orderBy(desc("count")).limit(10) +``` + +**过滤异常数据**： +```sql +-- 过滤null值和异常值 +SELECT * FROM table +WHERE key IS NOT NULL + AND key != 'unknown' + AND key != '' +``` + +**2. Join倾斜优化** + +**广播Join（Map-side Join）**： +```scala +// Spark广播小表 +val broadcast_small = spark.sparkContext.broadcast(small_table.collect()) +val result = big_table.map { row => + val small_data = broadcast_small.value + // Join逻辑 +} +``` + +**加盐技术（Salting）**： +```scala +// 大表加随机前缀 +val salted_big = big_table.map { row => + val salt = Random.nextInt(100) + (s"${salt}_${row.key}", row) +} + +// 小表扩展 +val expanded_small = small_table.flatMap { row => + (0 until 100).map(i => (s"${i}_${row.key}", row)) +} + +// Join后去除盐值 +val result = salted_big.join(expanded_small) + .map { case (salted_key, (big_row, small_row)) => + // 处理结果 + } +``` + +**两阶段聚合**： +```scala +// 第一阶段：局部聚合加随机后缀 +val stage1 = df.map { row => + val salt = Random.nextInt(100) + (s"${row.key}_${salt}", row.value) +}.reduceByKey(_ + _) + +// 第二阶段：全局聚合去掉后缀 +val stage2 = stage1.map { case (salted_key, value) => + val key = salted_key.split("_")(0) + (key, value) +}.reduceByKey(_ + _) +``` + +**3. 分区策略优化** + +**自定义分区器**： +```scala +class CustomPartitioner(partitions: Int) extends Partitioner { + override def numPartitions: Int = partitions + + override def getPartition(key: Any): Int = { + key match { + case hotKey if isHotKey(hotKey) => + // 热点数据分散到多个分区 + (hotKey.hashCode % (partitions / 2)).abs + case _ => + (key.hashCode % partitions).abs + } + } +} +``` + +**4. Hive中处理数据倾斜** + +**Map-side Join**： +```sql +-- 开启Map Join +SET hive.auto.convert.join=true; +SET hive.mapjoin.smalltable.filesize=25000000; +``` + +**分桶表Join**： +```sql +-- 创建分桶表避免倾斜 +CREATE TABLE bucketed_table ( + id INT, name STRING +) CLUSTERED BY (id) INTO 10 BUCKETS; +``` + +**动态分区**： +```sql +-- 使用动态分区均匀分布数据 +SET hive.exec.dynamic.partition=true; +SET hive.exec.dynamic.partition.mode=nonstrict; +``` + +**5. Flink中处理数据倾斜** + +**自定义分区函数**： +```java +// 自定义分区策略 +stream.partitionCustom(new Partitioner() { + @Override + public int partition(String key, int numPartitions) { + if (isHotKey(key)) { + // 热点数据随机分区 + return ThreadLocalRandom.current().nextInt(numPartitions); + } + return key.hashCode() % numPartitions; + } +}, keySelector); +``` + +**监控和预防**： +- 监控Task执行时间分布 +- 设置数据倾斜告警 +- 定期分析热点数据 +- 建立数据倾斜处理规范 + +### 🎯 大数据平台的监控体系如何建设？ + +**监控体系的重要性**： +大数据平台涉及多个组件，数据链路复杂，需要完善的监控体系保证系统稳定运行和及时发现问题。 + +**监控体系架构**： + +**1. 数据收集层** + +**系统监控**： +- **节点资源**：CPU、内存、磁盘、网络 +- **JVM指标**：堆内存、GC、线程数 +- **组件日志**：应用日志、错误日志 + +**业务监控**： +- **数据质量**：数据完整性、准确性、时效性 +- **任务执行**：作业成功率、执行时间、资源消耗 +- **数据流量**：吞吐量、延迟、积压 + +**监控工具选择**： +``` +系统监控：Prometheus + Node Exporter +应用监控：Micrometer + Prometheus +日志收集：ELK Stack (Elasticsearch + Logstash + Kibana) +链路追踪：Jaeger/Zipkin +``` + +**2. 监控指标设计** + +**基础设施监控**： +```yaml +# Prometheus监控配置示例 +groups: +- name: hadoop.rules + rules: + - alert: HDFSNameNodeDown + expr: up{job="namenode"} == 0 + for: 30s + labels: + severity: critical + annotations: + summary: "HDFS NameNode is down" + + - alert: DataNodeDiskUsage + expr: (node_filesystem_size_bytes - node_filesystem_free_bytes) / node_filesystem_size_bytes > 0.85 + for: 2m + labels: + severity: warning + annotations: + summary: "DataNode disk usage > 85%" +``` + +**应用层监控指标**： + +**Spark应用监控**： +```scala +// 自定义Metrics +val sparkConf = new SparkConf() +sparkConf.set("spark.sql.streaming.metricsEnabled", "true") +sparkConf.set("spark.metrics.conf.driver.source.jvm.class", + "org.apache.spark.metrics.source.JvmSource") + +// 业务指标 +val counter = SparkEnv.get.metricsSystem.counter("custom.records.processed") +counter.inc(recordCount) +``` + +**Flink应用监控**： +```java +// Flink自定义Metrics +public class MyMapFunction extends RichMapFunction { + private Counter counter; + + @Override + public void open(Configuration config) { + this.counter = getRuntimeContext() + .getMetricGroup() + .counter("records_processed"); + } + + @Override + public String map(String value) { + counter.inc(); + return value.toUpperCase(); + } +} +``` + +**3. 告警机制** + +**告警规则设计**： +```yaml +# AlertManager告警规则 +- alert: SparkJobFailure + expr: spark_job_status{status="failed"} > 0 + for: 0m + labels: + severity: critical + team: data-platform + annotations: + summary: "Spark job {{ $labels.job_name }} failed" + description: "Job has been failing for more than 0 minutes" + +- alert: KafkaConsumerLag + expr: kafka_consumer_lag_sum > 10000 + for: 5m + labels: + severity: warning + annotations: + summary: "Kafka consumer lag is high" +``` + +**告警渠道**： +- 企业微信/钉钉机器人 +- 邮件通知 +- 短信告警（严重故障） +- PagerDuty（海外） + +**4. 可视化监控大盘** + +**Grafana Dashboard设计**： + +**集群概览大盘**： +```json +{ + "dashboard": { + "title": "Big Data Platform Overview", + "panels": [ + { + "title": "HDFS Storage Usage", + "type": "stat", + "targets": [ + { + "expr": "hdfs_capacity_used_bytes / hdfs_capacity_total_bytes * 100" + } + ] + }, + { + "title": "YARN Resource Usage", + "type": "graph", + "targets": [ + { + "expr": "yarn_cluster_memory_used / yarn_cluster_memory_total * 100" + } + ] + } + ] + } +} +``` + +**实时任务监控大盘**： +- 任务运行状态统计 +- 数据处理吞吐量 +- 任务执行延迟 +- 错误率趋势 + +**5. 数据质量监控** + +**数据质量检查**： +```python +# 使用Great Expectations进行数据质量检查 +import great_expectations as ge + +df = ge.read_csv("data.csv") + +# 数据完整性检查 +df.expect_column_to_exist("user_id") +df.expect_column_values_to_not_be_null("user_id") + +# 数据准确性检查 +df.expect_column_values_to_be_between("age", min_value=0, max_value=120) + +# 数据一致性检查 +df.expect_column_values_to_be_in_set("status", ["active", "inactive"]) +``` + +**数据血缘监控**： +```python +# Apache Atlas数据血缘 +from pyatlas import Atlas + +client = Atlas("http://atlas-server:21000/", ("admin", "admin")) + +# 创建数据集实体 +dataset = { + "typeName": "DataSet", + "attributes": { + "name": "user_behavior", + "qualifiedName": "user_behavior@cluster1" + } +} + +# 创建处理过程实体 +process = { + "typeName": "Process", + "attributes": { + "name": "etl_user_behavior", + "inputs": [dataset], + "outputs": [processed_dataset] + } +} +``` + +**监控最佳实践**： +- **分层监控**：基础设施 → 组件 → 应用 → 业务 +- **异常检测**：基于机器学习的异常识别 +- **SLA定义**：明确服务等级协议 +- **故障预案**：建立标准化应急响应流程 +- **监控即代码**：监控配置版本化管理 + +--- + +## 🎯 大数据面试备战指南 + +### 💡 高频考点Top15 + +1. **🏗️ HDFS架构原理** - 分布式文件系统基础，副本机制和容错 +2. **⚡ MapReduce vs Spark** - 批计算框架对比，内存计算优势 +3. **🌊 Flink流处理** - 流计算引擎，Watermark和状态管理 +4. **📊 Hive数据仓库** - SQL转MapReduce，存储格式优化 +5. **📈 Apache Doris** - MPP架构，列式存储，实时OLAP分析 +6. **🎯 Apache Kylin** - OLAP预计算引擎，Cube构建，多维分析 +7. **🚀 Kafka消息队列** - 高性能消息系统，分区副本机制 +8. **🔧 YARN资源调度** - 集群资源管理，调度器对比 +9. **💾 数据倾斜处理** - 分布式计算常见问题和解决方案 +10. **🎯 技术选型** - 不同场景下的技术选择策略 +11. **📈 监控运维** - 大数据平台监控体系建设 +12. **⚙️ 性能调优** - 各组件的参数优化和最佳实践 +13. **🔐 数据安全** - Kerberos认证，权限管理 +14. **📦 容器化部署** - Docker、Kubernetes在大数据中的应用 +15. **☁️ 云原生架构** - 云上大数据解决方案 +16. **🤖 实时计算** - 流批一体化架构设计 +17. **💡 架构演进** - 从传统架构到现代数据湖架构 + +### 🎭 面试答题技巧 + +**📝 标准回答结构** +1. **背景介绍**（20秒） - 说明技术的应用背景和解决的问题 +2. **核心原理**（2分钟） - 深入讲解技术原理和关键机制 +3. **实践应用**（1分钟） - 结合实际项目说明如何使用 +4. **对比分析**（1分钟） - 与其他技术的对比优势 +5. **注意事项**（30秒） - 使用中的关键点和最佳实践 + +**🗣️ 表达话术模板** +- "在我们项目中，面临的主要挑战是..." +- "我们选择这个技术的原因是..." +- "从性能角度来看，这种方案的优势在于..." +- "在生产环境中，需要特别注意..." +- "相比于传统方案，新架构带来的收益是..." + +### 🚀 进阶加分点 + +- **架构设计能力**：能设计完整的大数据处理架构 +- **性能优化经验**：有具体的调优案例和效果数据 +- **故障处理能力**：能快速定位和解决线上问题 +- **技术深度**：了解底层原理和源码实现 +- **业务理解**：能结合业务场景选择合适的技术方案 +- **团队协作**：有跨团队合作的大数据项目经验 + +### 📚 延伸学习建议 + +- **官方文档**：各组件的官方文档是最权威的学习资料 +- **源码阅读**：深入理解核心组件的实现原理 +- **实战项目**：搭建完整的大数据处理链路 +- **技术博客**：关注知名公司的大数据技术分享 +- **开源贡献**：参与开源项目，提升技术影响力 + +--- + +## 🎉 总结 + +**大数据技术栈是现代企业数字化转型的核心基础设施**，掌握这些技术不仅是技术能力的体现，更是解决业务问题的重要手段。 + +**记住：面试官考察的不是死记硬背，而是你运用大数据技术解决实际业务问题的能力和思维方式。** + +**最后一句话**：*"实践出真知，项目见真章"* - 理论学习要与实际项目相结合，在实战中加深理解！ + +--- + +> 💌 **持续学习，拥抱变化！** +> 大数据技术日新月异，保持学习的热情，在技术的海洋中不断探索前行！ \ No newline at end of file diff --git a/docs/interview/Collections-FAQ.md b/docs/interview/Collections-FAQ.md deleted file mode 100644 index dae5b35573..0000000000 --- a/docs/interview/Collections-FAQ.md +++ /dev/null @@ -1,1760 +0,0 @@ -「我是大厂面试官」—— Java 集合，你肯定会被问到这些 - -> 文章收录在 GitHub [JavaKeeper](https://github.com/Jstarfish/JavaKeeper) ，N线互联网开发必备技能兵器谱 - -作为一位小菜 ”一面面试官“，面试过程中，我肯定会问 Java 集合的内容，同时作为求职者，也肯定会被问到集合，所以整理下 Java 集合面试题 - -![](https://i02piccdn.sogoucdn.com/fe487f455e5b1eb6) - -> 说说常见的集合有哪些吧？ -> -> HashMap说一下，其中的Key需要重写hashCode()和equals()吗？ -> -> HashMap中key和value可以为null吗？允许几个为null呀？ -> -> HashMap线程安全吗?ConcurrentHashMap和hashTable有什么区别？ -> -> List和Set说一下，现在有一个ArrayList，对其中的所有元素按照某一属性大小排序，应该怎么做？ -> -> ArrayList 和 Vector 的区别 -> -> list 可以删除吗，遍历的时候可以删除吗，为什么 - -面向对象语言对事物的体现都是以对象的形式，所以为了方便对多个对象的操作，需要将对象进行存储，集合就是存储对象最常用的一种方式，也叫容器。 - -![img](https://tva1.sinaimg.cn/large/00831rSTly1gdp03vldkkg30hv0gzwet.gif) - - - -从上面的集合框架图可以看到，Java 集合框架主要包括两种类型的容器 - -- 一种是集合（Collection），存储一个元素集合 -- 另一种是图（Map），存储键/值对映射。 - -Collection 接口又有 3 种子类型，List、Set 和 Queue，再下面是一些抽象类，最后是具体实现类，常用的有 ArrayList、LinkedList、HashSet、LinkedHashSet、HashMap、LinkedHashMap 等等。 - -集合框架是一个用来代表和操纵集合的统一架构。所有的集合框架都包含如下内容： - -- **接口**：是代表集合的抽象数据类型。例如 Collection、List、Set、Map 等。之所以定义多个接口，是为了以不同的方式操作集合对象 - -- **实现（类）**：是集合接口的具体实现。从本质上讲，它们是可重复使用的数据结构，例如：ArrayList、LinkedList、HashSet、HashMap。 - -- **算法**：是实现集合接口的对象里的方法执行的一些有用的计算，例如：搜索和排序。这些算法被称为多态，那是因为相同的方法可以在相似的接口上有着不同的实现。 - ------- - - - -## 说说常用的集合有哪些吧？ - -Map 接口和 Collection 接口是所有集合框架的父接口： - -1. Collection接口的子接口包括：Set、List、Queue -2. List是有序的允许有重复元素的Collection，实现类主要有：ArrayList、LinkedList、Stack以及Vector等 -3. Set是一种不包含重复元素且无序的Collection，实现类主要有：HashSet、TreeSet、LinkedHashSet等 -4. Map没有继承Collection接口，Map提供key到value的映射。实现类主要有：HashMap、TreeMap、Hashtable、ConcurrentHashMap 以及 Properties 等 - - - -## ArrayList 和 Vector 的区别 - -相同点： - -- ArrayList 和 Vector 都是继承了相同的父类和实现了相同的接口（都实现了List，有序、允许重复和null） - - ```java - extends AbstractList - implements List, RandomAccess, Cloneable, java.io.Serializable - ``` - -- 底层都是数组（Object[]）实现的 - -- 初始默认长度都为**10** - -不同点： - -- 同步性：Vector 中的 public 方法多数添加了 synchronized 关键字、以确保方法同步、也即是 Vector 线程安全、ArrayList 线程不安全 - -- 性能：Vector 存在 synchronized 的锁等待情况、需要等待释放锁这个过程、所以性能相对较差 - -- 扩容大小：ArrayList在底层数组不够用时在原来的基础上扩展 0.5 倍，Vector默认是扩展 1 倍 - - 扩容机制，扩容方法其实就是新创建一个数组，然后将旧数组的元素都复制到新数组里面。其底层的扩容方法都在 **grow()** 中（基于JDK8） - - - ArrayList 的 grow()，在满足扩容条件时、ArrayList以**1.5** 倍的方式在扩容（oldCapacity >> **1** ，右移运算，相当于除以 2，结果为二分之一的 oldCapacity） - - ```java - private void grow(int minCapacity) { - // overflow-conscious code - int oldCapacity = elementData.length; - //newCapacity = oldCapacity + O.5*oldCapacity,此处扩容0.5倍 - int newCapacity = oldCapacity + (oldCapacity >> 1); - if (newCapacity - minCapacity < 0) - newCapacity = minCapacity; - if (newCapacity - MAX_ARRAY_SIZE > 0) - newCapacity = hugeCapacity(minCapacity); - // minCapacity is usually close to size, so this is a win: - elementData = Arrays.copyOf(elementData, newCapacity); - } - ``` - - - Vector 的 grow()，Vector 比 ArrayList多一个属性，扩展因子capacityIncrement，可以扩容大小。当扩容容量增量大于**0**时、新数组长度为原数组长度**+**扩容容量增量、否则新数组长度为原数组长度的**2**倍 - - ```java - private void grow(int minCapacity) { - // overflow-conscious code - int oldCapacity = elementData.length; - // - int newCapacity = oldCapacity + ((capacityIncrement > 0) ? - capacityIncrement : oldCapacity); - if (newCapacity - minCapacity < 0) - newCapacity = minCapacity; - if (newCapacity - MAX_ARRAY_SIZE > 0) - newCapacity = hugeCapacity(minCapacity); - elementData = Arrays.copyOf(elementData, newCapacity); - } - ``` - - - -## ArrayList 与 LinkedList 区别 - -- 是否保证线程安全： ArrayList 和 LinkedList 都是不同步的，也就是不保证线程安全； -- **底层数据结构**： Arraylist 底层使用的是 Object 数组；LinkedList 底层使用的是**双向循环链表**数据结构； -- **插入和删除是否受元素位置的影响：** - - **ArrayList 采用数组存储，所以插入和删除元素的时间复杂度受元素位置的影响。** 比如：执行 `add(E e)`方法的时候， ArrayList 会默认在将指定的元素追加到此列表的末尾，这种情况时间复杂度就是O(1)。但是如果要在指定位置 i 插入和删除元素的话（ `add(intindex,E element)`）时间复杂度就为 O(n-i)。因为在进行上述操作的时候集合中第 i 和第 i 个元素之后的(n-i)个元素都要执行向后位/向前移一位的操作。 - - **LinkedList 采用链表存储，所以插入，删除元素时间复杂度不受元素位置的影响，都是近似 $O(1)$，而数组为近似 $O(n)$。** - - ArrayList 一般应用于查询较多但插入以及删除较少情况，如果插入以及删除较多则建议使用 LinkedList -- **是否支持快速随机访问**： LinkedList 不支持高效的随机元素访问，而 ArrayList 实现了 RandomAccess 接口，所以有随机访问功能。快速随机访问就是通过元素的序号快速获取元素对象(对应于 `get(intindex)`方法)。 -- **内存空间占用**： ArrayList 的空间浪费主要体现在在 list 列表的结尾会预留一定的容量空间，而 LinkedList 的空间花费则体现在它的每一个元素都需要消耗比 ArrayList 更多的空间（因为要存放直接后继和直接前驱以及数据）。 - - - -高级工程师的我，可不得看看源码，具体分析下： - -- ArrayList工作原理其实很简单，底层是动态数组，每次创建一个 ArrayList 实例时会分配一个初始容量（没有指定初始容量的话，默认是 10），以add方法为例，如果没有指定初始容量，当执行add方法，先判断当前数组是否为空，如果为空则给保存对象的数组分配一个最小容量，默认为10。当添加大容量元素时，会先增加数组的大小，以提高添加的效率； - -- LinkedList 是有序并且支持元素重复的集合，底层是基于双向链表的，即每个节点既包含指向其后继的引用也包括指向其前驱的引用。链表无容量限制，但双向链表本身使用了更多空间，也需要额外的链表指针操作。按下标访问元素 `get(i)/set(i,e)` 要悲剧的遍历链表将指针移动到位(如果i>数组大小的一半，会从末尾移起)。插入、删除元素时修改前后节点的指针即可，但还是要遍历部分链表的指针才能移动到下标所指的位置，只有在链表两头的操作`add()`， `addFirst()`，`removeLast()`或用 `iterator()` 上的 `remove()` 能省掉指针的移动。此外 LinkedList 还实现了 Deque（继承自Queue接口）接口，可以当做队列使用。 - -不会囊括所有方法，只是为了学习，记录思想。 - -ArrayList 和 LinkedList 两者都实现了 List 接口 - -```java -public class ArrayList extends AbstractList - implements List, RandomAccess, Cloneable, java.io.Serializable{ -``` - -```java -public class LinkedList - extends AbstractSequentialList - implements List, Deque, Cloneable, java.io.Serializable -``` - -### 构造器 - -ArrayList 提供了 3 个构造器，①无参构造器 ②带初始容量构造器 ③参数为集合构造器 - -```java -public class ArrayList extends AbstractList - implements List, RandomAccess, Cloneable, java.io.Serializable{ - -public ArrayList(int initialCapacity) { - if (initialCapacity > 0) { - // 创建初始容量的数组 - this.elementData = new Object[initialCapacity]; - } else if (initialCapacity == 0) { - this.elementData = EMPTY_ELEMENTDATA; - } else { - throw new IllegalArgumentException("Illegal Capacity: "+initialCapacity); - } -} -public ArrayList() { - // 默认为空数组 - this.elementData = DEFAULTCAPACITY_EMPTY_ELEMENTDATA; -} - -public ArrayList(Collection c) { //...} -} -``` - -LinkedList 提供了 2 个构造器，因为基于链表，所以也就没有初始化大小，也没有扩容的机制，就是一直在前面或者后面插插插~~ - -```java -public LinkedList() { -} - -public LinkedList(Collection c) { - this(); - addAll(c); -} -// LinkedList 既然作为链表，那么肯定会有节点 -private static class Node { - E item; - Node next; - Node prev; - - Node(Node prev, E element, Node next) { - this.item = element; - this.next = next; - this.prev = prev; - } -} -``` - -### 插入 - -**ArrayList**: - -```java -public boolean add(E e) { - // 确保数组的容量，保证可以添加该元素 - ensureCapacityInternal(size + 1); // Increments modCount!! - // 将该元素放入数组中 - elementData[size++] = e; - return true; -} -private void ensureCapacityInternal(int minCapacity) { - // 如果数组是空的，那么会初始化该数组 - if (elementData == DEFAULTCAPACITY_EMPTY_ELEMENTDATA) { - // DEFAULT_CAPACITY 为 10,所以调用无参默认 ArrayList 构造方法初始化的话，默认的数组容量为 10 - minCapacity = Math.max(DEFAULT_CAPACITY, minCapacity); - } - - ensureExplicitCapacity(minCapacity); -} - -private void ensureExplicitCapacity(int minCapacity) { - modCount++; - - // 确保数组的容量，如果不够的话，调用 grow 方法扩容 - if (minCapacity - elementData.length > 0) - grow(minCapacity); -} -//扩容具体的方法 -private void grow(int minCapacity) { - // 当前数组的容量 - int oldCapacity = elementData.length; - // 新数组扩容为原来容量的 1.5 倍 - int newCapacity = oldCapacity + (oldCapacity >> 1); - // 如果新数组扩容容量还是比最少需要的容量还要小的话，就设置扩充容量为最小需要的容量 - if (newCapacity - minCapacity < 0) - newCapacity = minCapacity; - //判断新数组容量是否已经超出最大数组范围，MAX_ARRAY_SIZE = Integer.MAX_VALUE - 8 - if (newCapacity - MAX_ARRAY_SIZE > 0) - newCapacity = hugeCapacity(minCapacity); - // minCapacity is usually close to size, so this is a win: - // 复制元素到新的数组中 - elementData = Arrays.copyOf(elementData, newCapacity); -} -``` - -当然也可以插入指定位置，还有一个重载的方法 `add(int index, E element)` - -```java -public void add(int index, E element) { - // 判断 index 有没有超出索引的范围 - rangeCheckForAdd(index); - // 和之前的操作是一样的，都是保证数组的容量足够 - ensureCapacityInternal(size + 1); // Increments modCount!! - // 将指定位置及其后面数据向后移动一位 - System.arraycopy(elementData, index, elementData, index + 1, - size - index); - // 将该元素添加到指定的数组位置 - elementData[index] = element; - // ArrayList 的大小改变 - size++; -} -``` - -可以看到每次插入指定位置都要移动元素，效率较低。 - -再来看 **LinkedList** 的插入，也有插入末尾，插入指定位置两种，由于基于链表，肯定得先有个 Node - -```java -private static class Node { - E item; - Node next; - Node prev; - - Node(Node prev, E element, Node next) { - this.item = element; - this.next = next; - this.prev = prev; - } -} -``` - -```java -public boolean add(E e) { - // 直接往队尾加元素 - linkLast(e); - return true; -} - -void linkLast(E e) { - // 保存原来链表尾部节点，last 是全局变量，用来表示队尾元素 - final Node l = last; - // 为该元素 e 新建一个节点 - final Node newNode = new Node<>(l, e, null); - // 将新节点设为队尾 - last = newNode; - // 如果原来的队尾元素为空，那么说明原来的整个列表是空的，就把新节点赋值给头结点 - if (l == null) - first = newNode; - else - // 原来尾结点的后面为新生成的结点 - l.next = newNode; - // 节点数 +1 - size++; - modCount++; -} - -public void add(int index, E element) { - // 检查 index 有没有超出索引范围 - checkPositionIndex(index); - // 如果追加到尾部，那么就跟 add(E e) 一样了 - if (index == size) - linkLast(element); - else - // 否则就是插在其他位置 - linkBefore(element, node(index)); -} - -//linkBefore方法中调用了这个node方法，类似二分查找的优化 -Node node(int index) { - // assert isElementIndex(index); - // 如果 index 在前半段，从前往后遍历获取 node - if (index < (size >> 1)) { - Node x = first; - for (int i = 0; i < index; i++) - x = x.next; - return x; - } else { - // 如果 index 在后半段，从后往前遍历获取 node - Node x = last; - for (int i = size - 1; i > index; i--) - x = x.prev; - return x; - } -} - -void linkBefore(E e, Node succ) { - // assert succ != null; - // 保存 index 节点的前节点 - final Node pred = succ.prev; - // 新建一个目标节点 - final Node newNode = new Node<>(pred, e, succ); - succ.prev = newNode; - // 如果是在开头处插入的话 - if (pred == null) - first = newNode; - else - pred.next = newNode; - size++; - modCount++; -} -``` - -### 获取 - -**ArrayList** 的 get() 方法很简单，就是在数组中返回指定位置的元素即可，所以效率很高 - -```java -public E get(int index) { - // 检查 index 有没有超出索引的范围 - rangeCheck(index); - // 返回指定位置的元素 - return elementData(index); -} -``` - -**LinkedList** 的 get() 方法，就是在内部调用了上边看到的 node() 方法，判断在前半段还是在后半段，然后遍历得到即可。 - -```java -public E get(int index) { - checkElementIndex(index); - return node(index).item; -} -``` - ------- - - - -## HashMap的底层实现 - -> 什么时候会使用HashMap？他有什么特点？ -> -> 你知道HashMap的工作原理吗？ -> -> 你知道get和put的原理吗？equals()和hashCode()的都有什么作用？ -> -> 你知道hash的实现吗？为什么要这样实现？ -> -> 如果HashMap的大小超过了负载因子(load factor)定义的容量，怎么办？ - -HashMap 在 JDK 7 和 JDK8 中的实现方式略有不同。分开记录。 - -深入 HahsMap 之前，先要了解的概念 - -1. initialCapacity：初始容量。指的是 HashMap 集合初始化的时候自身的容量。可以在构造方法中指定；如果不指定的话，总容量默认值是 16 。需要注意的是初始容量必须是 2 的幂次方。（**1.7中，已知HashMap中将要存放的KV个数的时候，设置一个合理的初始化容量可以有效的提高性能**） - - ```java - static final int DEFAULT_INITIAL_CAPACITY = 1 << 4; // aka 16 - ``` - -2. size：当前 HashMap 中已经存储着的键值对数量，即 `HashMap.size()` 。 - -3. loadFactor：加载因子。所谓的加载因子就是 HashMap (当前的容量/总容量) 到达一定值的时候，HashMap 会实施扩容。加载因子也可以通过构造方法中指定，默认的值是 0.75 。举个例子，假设有一个 HashMap 的初始容量为 16 ，那么扩容的阀值就是 0.75 * 16 = 12 。也就是说，在你打算存入第 13 个值的时候，HashMap 会先执行扩容。 - -4. threshold：扩容阀值。即 扩容阀值 = HashMap 总容量 * 加载因子。当前 HashMap 的容量大于或等于扩容阀值的时候就会去执行扩容。扩容的容量为当前 HashMap 总容量的两倍。比如，当前 HashMap 的总容量为 16 ，那么扩容之后为 32 。 - -5. table：Entry 数组。我们都知道 HashMap 内部存储 key/value 是通过 Entry 这个介质来实现的。而 table 就是 Entry 数组。 - -### JDK1.7 实现 - -JDK1.7 中 HashMap 由 **数组+链表** 组成（**“链表散列”** 即数组和链表的结合体），数组是 HashMap 的主体，链表则是主要为了解决哈希冲突而存在的（HashMap 采用 **“拉链法也就是链地址法”** 解决冲突），如果定位到的数组位置不含链表（当前 entry 的 next 指向 null ）,那么对于查找，添加等操作很快，仅需一次寻址即可；如果定位到的数组包含链表，对于添加操作，其时间复杂度依然为 O(1)，因为最新的 Entry 会插入链表头部，即需要简单改变引用链即可，而对于查找操作来讲，此时就需要遍历链表，然后通过 key 对象的 equals 方法逐一比对查找。 - -> 所谓 **“拉链法”** 就是将链表和数组相结合。也就是说创建一个链表数组，数组中每一格就是一个链表。若遇到哈希冲突，则将冲突的值加到链表中即可。 - -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gdx45a2fbbj31dz0oaacx.jpg) - -#### 源码解析 - -##### 构造方法 - -《阿里巴巴 Java 开发手册》推荐集合初始化时，指定集合初始值大小。（说明：HashMap 使用HashMap(int initialCapacity) 初始化）建议原因： https://www.zhihu.com/question/314006228/answer/611170521 - -```java -// 默认的构造方法使用的都是默认的初始容量和加载因子 -// DEFAULT_INITIAL_CAPACITY = 16，DEFAULT_LOAD_FACTOR = 0.75f -public HashMap() { - this(DEFAULT_INITIAL_CAPACITY, DEFAULT_LOAD_FACTOR); -} - -// 可以指定初始容量，并且使用默认的加载因子 -public HashMap(int initialCapacity) { - this(initialCapacity, DEFAULT_LOAD_FACTOR); -} - -public HashMap(int initialCapacity, float loadFactor) { - // 对初始容量的值判断 - if (initialCapacity < 0) - throw new IllegalArgumentException("Illegal initial capacity: " + - initialCapacity); - if (initialCapacity > MAXIMUM_CAPACITY) - initialCapacity = MAXIMUM_CAPACITY; - if (loadFactor <= 0 || Float.isNaN(loadFactor)) - throw new IllegalArgumentException("Illegal load factor: " + - loadFactor); - // 设置加载因子 - this.loadFactor = loadFactor; - threshold = initialCapacity; - // 空方法 - init(); -} - -public HashMap(Map m) { - this(Math.max((int) (m.size() / DEFAULT_LOAD_FACTOR) + 1, - DEFAULT_INITIAL_CAPACITY), DEFAULT_LOAD_FACTOR); - inflateTable(threshold); - putAllForCreate(m); -} -``` - -HashMap 的前 3 个构造方法最后都会去调用 `HashMap(int initialCapacity, float loadFactor)` 。在其内部去设置初始容量和加载因子。而最后的 `init()` 是空方法，主要给子类实现，比如LinkedHashMap。 - -##### put() 方法 - -```java -public V put(K key, V value) { - // 如果 table 数组为空时先创建数组，并且设置扩容阀值 - if (table == EMPTY_TABLE) { - inflateTable(threshold); - } - // 如果 key 为空时，调用 putForNullKey 方法特殊处理 - if (key == null) - return putForNullKey(value); - // 计算 key 的哈希值 - int hash = hash(key); - // 根据计算出来的哈希值和当前数组的长度计算在数组中的索引 - int i = indexFor(hash, table.length); - // 先遍历该数组索引下的整条链表 - // 如果该 key 之前已经在 HashMap 中存储了的话，直接替换对应的 value 值即可 - for (Entry e = table[i]; e != null; e = e.next) { - Object k; - //先判断hash值是否一样，如果一样，再判断key是否一样，不同对象的hash值可能一样 - if (e.hash == hash && ((k = e.key) == key || key.equals(k))) { - V oldValue = e.value; - e.value = value; - e.recordAccess(this); - return oldValue; - } - } - - modCount++; - // 如果该 key 之前没有被存储过，那么就进入 addEntry 方法 - addEntry(hash, key, value, i); - return null; -} - -void addEntry(int hash, K key, V value, int bucketIndex) { - // 当前容量大于或等于扩容阀值的时候，会执行扩容 - if ((size >= threshold) && (null != table[bucketIndex])) { - // 扩容为原来容量的两倍 - resize(2 * table.length); - // 重新计算哈希值 - hash = (null != key) ? hash(key) : 0; - // 重新得到在新数组中的索引 - bucketIndex = indexFor(hash, table.length); - } - // 创建节点 - createEntry(hash, key, value, bucketIndex); -} - -//扩容，创建了一个新的数组，然后把数据全部复制过去，再把新数组的引用赋给 table -void resize(int newCapacity) { - Entry[] oldTable = table; //引用扩容前的Entry数组 - int oldCapacity = oldTable.length; - if (oldCapacity == MAXIMUM_CAPACITY) { //扩容前的数组大小如果已经达到最大(2^30)了 - threshold = Integer.MAX_VALUE; //修改阈值为int的最大值(2^31-1)，这样以后就不会扩容了 - return; - } - // 创建新的 entry 数组 - Entry[] newTable = new Entry[newCapacity]; - // 将旧 entry 数组中的数据复制到新 entry 数组中 - transfer(newTable, initHashSeedAsNeeded(newCapacity)); - // 将新数组的引用赋给 table - table = newTable; - // 计算新的扩容阀值 - threshold = (int)Math.min(newCapacity * loadFactor, MAXIMUM_CAPACITY + 1); -} - -void transfer(Entry[] newTable) { - Entry[] src = table; //src引用了旧的Entry数组 - int newCapacity = newTable.length; - for (int j = 0; j < src.length; j++) { //遍历旧的Entry数组 - Entry e = src[j]; //取得旧Entry数组的每个元素 - if (e != null) { - src[j] = null;//释放旧Entry数组的对象引用（for循环后，旧的Entry数组不再引用任何对象） - do { - Entry next = e.next; - int i = indexFor(e.hash, newCapacity); //！！重新计算每个元素在数组中的位置 - e.next = newTable[i]; //标记[1] - newTable[i] = e; //将元素放在数组上 - e = next; //访问下一个Entry链上的元素 - } while (e != null); - } - } -} - -void createEntry(int hash, K key, V value, int bucketIndex) { - // 取出table中下标为bucketIndex的Entry - Entry e = table[bucketIndex]; - // 利用key、value来构建新的Entry - // 并且之前存放在table[bucketIndex]处的Entry作为新Entry的next - // 把新创建的Entry放到table[bucketIndex]位置 - table[bucketIndex] = new Entry<>(hash, key, value, e); - // 当前 HashMap 的容量加 1 - size++; -} -``` - -最后的 createEntry() 方法就说明了当hash冲突时，采用的拉链法来解决hash冲突的，并且是把新元素是插入到单边表的表头。 - -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gdx45xr0x5j31hl0u0ncd.jpg) - -##### get() 方法 - -```java -public V get(Object key) { - // 如果 key 是空的，就调用 getForNullKey 方法特殊处理 - if (key == null) - return getForNullKey(); - // 获取 key 相对应的 entry - Entry entry = getEntry(key); - - return null == entry ? null : entry.getValue(); -} - -//找到对应 key 的数组索引，然后遍历链表查找即可 -final Entry getEntry(Object key) { - if (size == 0) { - return null; - } - // 计算 key 的哈希值 - int hash = (key == null) ? 0 : hash(key); - // 得到数组的索引，然后遍历链表，查看是否有相同 key 的 Entry - for (Entry e = table[indexFor(hash, table.length)]; - e != null; - e = e.next) { - Object k; - if (e.hash == hash && - ((k = e.key) == key || (key != null && key.equals(k)))) - return e; - } - // 没有的话，返回 null - return null; -} -``` - -### JDK1.8 实现 - -JDK 1.7 中，如果哈希碰撞过多，拉链过长，极端情况下，所有值都落入了同一个桶内，这就退化成了一个链表。通过 key 值查找要遍历链表，效率较低。 JDK1.8 在解决哈希冲突时有了较大的变化，**当链表长度大于阈值（默认为8）时，将链表转化为红黑树**，以减少搜索时间。 - -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gdx468uknnj31650ogjth.jpg) - -TreeMap、TreeSet 以及 JDK1.8 之后的 HashMap 底层都用到了红黑树。红黑树就是为了解决二叉查找树的缺陷，因为二叉查找树在某些情况下会退化成一个线性结构。 - -#### 源码解析 - -##### 构造方法 - -JDK8 构造方法改动不是很大 - -```java -public HashMap() { - this.loadFactor = DEFAULT_LOAD_FACTOR; // all other fields defaulted -} - -public HashMap(int initialCapacity) { - this(initialCapacity, DEFAULT_LOAD_FACTOR); -} - -public HashMap(int initialCapacity, float loadFactor) { - if (initialCapacity < 0) - throw new IllegalArgumentException("Illegal initial capacity: " + - initialCapacity); - if (initialCapacity > MAXIMUM_CAPACITY) - initialCapacity = MAXIMUM_CAPACITY; - if (loadFactor <= 0 || Float.isNaN(loadFactor)) - throw new IllegalArgumentException("Illegal load factor: " + - loadFactor); - this.loadFactor = loadFactor; - this.threshold = tableSizeFor(initialCapacity); -} - -public HashMap(Map m) { - this.loadFactor = DEFAULT_LOAD_FACTOR; - putMapEntries(m, false); -} -``` - -##### 确定哈希桶数组索引位置（hash 函数的实现） - -```java -//方法一： -static final int hash(Object key) { //jdk1.8 & jdk1.7 - int h; - // h = key.hashCode() 为第一步 取hashCode值 - // h ^ (h >>> 16) 为第二步 高位参与运算 - return (key == null) ? 0 : (h = key.hashCode()) ^ (h >>> 16); -} -//方法二： -static int indexFor(int h, int length) { //jdk1.7的源码，jdk1.8没有提取这个方法，而是放在了其他方法中，比如 put 的p = tab[i = (n - 1) & hash] - return h & (length-1); //第三步 取模运算 -} -``` - -HashMap 定位数组索引位置，直接决定了 hash 方法的离散性能。Hash 算法本质上就是三步：**取key的hashCode值、高位运算、取模运算**。 - -![hash](https://tva1.sinaimg.cn/large/007S8ZIlly1gdwv5m4g4sj30ga09cdfy.jpg) - -> 为什么要这样呢？ -> -> HashMap 的长度为什么是2的幂次方? - -目的当然是为了减少哈希碰撞，使 table 里的数据分布的更均匀。 - -1. HashMap 中桶数组的大小 length 总是2的幂，此时，`h & (table.length -1)` 等价于对 length 取模 `h%length`。但取模的计算效率没有位运算高，所以这是是一个优化。假设 `h = 185`，`table.length-1 = 15(0x1111)`，其实散列真正生效的只是低 4bit 的有效位，所以很容易碰撞。 - - ![img](https://tva1.sinaimg.cn/large/007S8ZIlly1gdx46gac50j30m8037mxf.jpg) - -2. 图中的 hash 是由键的 hashCode 产生。计算余数时，由于 n 比较小，hash 只有低4位参与了计算，高位的计算可以认为是无效的。这样导致了计算结果只与低位信息有关，高位数据没发挥作用。为了处理这个缺陷，我们可以上图中的 hash 高4位数据与低4位数据进行异或运算，即 `hash ^ (hash >>> 4)`。通过这种方式，让高位数据与低位数据进行异或，以此加大低位信息的随机性，变相的让高位数据参与到计算中。此时的计算过程如下： - - ![img](https://tva1.sinaimg.cn/large/007S8ZIlly1gdx46jwezjj30m804cjrw.jpg) - - 在 Java 中，hashCode 方法产生的 hash 是 int 类型，32 位宽。前16位为高位，后16位为低位，所以要右移16位，即 `hash ^ (hash >>> 16)` 。这样还增加了hash 的复杂度，进而影响 hash 的分布性。 - -**HashMap 的长度为什么是2的幂次方？** - -为了能让HashMap存取高效，尽量减少碰撞，也就是要尽量把数据分配均匀，Hash值的范围是-2147483648到2147483647，前后加起来有40亿的映射空间，只要哈希函数映射的比较均匀松散，一般应用是很难出现碰撞的，但一个问题是40亿的数组内存是放不下的。所以这个散列值是不能直接拿来用的。用之前需要先对数组长度取模运算，得到余数才能用来存放位置也就是对应的数组小标。这个数组下标的计算方法是 `(n-1)&hash`，n代表数组长度。 - -这个算法应该如何设计呢？ - -我们首先可能会想到采用%取余的操作来实现。但是，重点来了。 - -取余操作中如果除数是2的幂次则等价于其除数减一的与操作，也就是说 `hash%length=hash&(length-1)`，但前提是 length 是 2 的 n 次方，并且采用 &运算比 %运算效率高，这也就解释了 HashMap 的长度为什么是2的幂次方。 - -##### get() 方法 - -```java -public V get(Object key) { - Node e; - return (e = getNode(hash(key), key)) == null ? null : e.value; -} - -final Node getNode(int hash, Object key) { - Node[] tab; Node first, e; int n; K k; - //定位键值对所在桶的位置 - if ((tab = table) != null && (n = tab.length) > 0 && - (first = tab[(n - 1) & hash]) != null) { - // 直接命中 - if (first.hash == hash && // always check first node - ((k = first.key) == key || (key != null && key.equals(k)))) - return first; - // 未命中 - if ((e = first.next) != null) { - // 如果 first 是 TreeNode 类型，则调用黑红树查找方法 - if (first instanceof TreeNode) - return ((TreeNode)first).getTreeNode(hash, key); - do { - // 在链表中查找 - if (e.hash == hash && - ((k = e.key) == key || (key != null && key.equals(k)))) - return e; - } while ((e = e.next) != null); - } - } - return null; -} -``` - -##### put() 方法 - -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gdx470vyzej311c0u0120.jpg) - -```java -public V put(K key, V value) { - // 对key的hashCode()做hash - return putVal(hash(key), key, value, false, true); -} - -final V putVal(int hash, K key, V value, boolean onlyIfAbsent, - boolean evict) { - Node[] tab; Node p; int n, i; - // tab为空则创建 - if ((tab = table) == null || (n = tab.length) == 0) - n = (tab = resize()).length; - // 计算index，并对null做处理 - if ((p = tab[i = (n - 1) & hash]) == null) - tab[i] = newNode(hash, key, value, null); - else { - Node e; K k; - // 节点key存在，直接覆盖value - if (p.hash == hash && - ((k = p.key) == key || (key != null && key.equals(k)))) - e = p; - // 判断该链为红黑树 - else if (p instanceof TreeNode) - e = ((TreeNode)p).putTreeVal(this, tab, hash, key, value); - else { - //该链为链表 - for (int binCount = 0; ; ++binCount) { - if ((e = p.next) == null) { - p.next = newNode(hash, key, value, null); - //链表长度大于8转换为红黑树进行处理 - if (binCount >= TREEIFY_THRESHOLD - 1) // -1 for 1st - treeifyBin(tab, hash); - break; - } - //key已经存在直接覆盖value - if (e.hash == hash && - ((k = e.key) == key || (key != null && key.equals(k)))) - break; - p = e; - } - } - if (e != null) { // existing mapping for key - V oldValue = e.value; - if (!onlyIfAbsent || oldValue == null) - e.value = value; - afterNodeAccess(e); - return oldValue; - } - } - ++modCount; - // 超过最大容量 就扩容 - if (++size > threshold) - resize(); - afterNodeInsertion(evict); - return null; -} -``` - -> HashMap的put方法的具体流程？ - -①.判断键值对数组table[i]是否为空或为null，否则执行resize()进行扩容； - -②.根据键值key计算hash值得到插入的数组索引i，如果table[i]==null，直接新建节点添加，转向⑥，如果table[i]不为空，转向③； - -③.判断table[i]的首个元素是否和key一样，如果相同直接覆盖value，否则转向④，这里的相同指的是hashCode以及equals； - -④.判断table[i] 是否为treeNode，即table[i] 是否是红黑树，如果是红黑树，则直接在树中插入键值对，否则转向⑤； - -⑤.遍历table[i]，判断链表长度是否大于8，大于8的话把链表转换为红黑树，在红黑树中执行插入操作，否则进行链表的插入操作；遍历过程中若发现key已经存在直接覆盖value即可； - -⑥.插入成功后，判断实际存在的键值对数量size是否超多了最大容量threshold，如果超过，进行扩容。 - -##### resize() 扩容 - -```java -final Node[] resize() { - Node[] oldTab = table; - int oldCap = (oldTab == null) ? 0 : oldTab.length; - int oldThr = threshold; - int newCap, newThr = 0; - if (oldCap > 0) { - // 超过最大值就不再扩充了，就只好随你碰撞了 - if (oldCap >= MAXIMUM_CAPACITY) { - //修改阈值为int的最大值(2^31-1)，这样以后就不会扩容了 - threshold = Integer.MAX_VALUE; - return oldTab; - } - // 没超过最大值，就扩充为原来的2倍 - else if ((newCap = oldCap << 1) < MAXIMUM_CAPACITY && - oldCap >= DEFAULT_INITIAL_CAPACITY) - newThr = oldThr << 1; // double threshold - } - else if (oldThr > 0) // initial capacity was placed in threshold - newCap = oldThr; - else { // zero initial threshold signifies using defaults - newCap = DEFAULT_INITIAL_CAPACITY; - newThr = (int)(DEFAULT_LOAD_FACTOR * DEFAULT_INITIAL_CAPACITY); - } - // 计算新的resize上限 - if (newThr == 0) { - float ft = (float)newCap * loadFactor; - newThr = (newCap < MAXIMUM_CAPACITY && ft < (float)MAXIMUM_CAPACITY ? - (int)ft : Integer.MAX_VALUE); - } - threshold = newThr; - // 开始复制到新的数组 - @SuppressWarnings({"rawtypes","unchecked"}) - Node[] newTab = (Node[])new Node[newCap]; - table = newTab; - // 把每个bucket都移动到新的buckets中 - if (oldTab != null) { - // 循环遍历旧的 table 数组 - for (int j = 0; j < oldCap; ++j) { - Node e; - if ((e = oldTab[j]) != null) { - oldTab[j] = null; - // 如果该桶只有一个元素，那么直接复制 - if (e.next == null) - newTab[e.hash & (newCap - 1)] = e; - // 如果是红黑树，那么对红黑树进行拆分 - else if (e instanceof TreeNode) - ((TreeNode)e).split(this, newTab, j, oldCap); - // 遍历链表，将原链表节点分成lo和hi两个链表 - // 其中 lo 表示在原来的桶位置，hi 表示在新的桶位置 - else { // preserve order 链表优化重hash的代码块 - Node loHead = null, loTail = null; - Node hiHead = null, hiTail = null; - Node next; - do { - // 原索引 - next = e.next; - if ((e.hash & oldCap) == 0) { - if (loTail == null) - loHead = e; - else - loTail.next = e; - loTail = e; - } - // 原索引+oldCap - else { - if (hiTail == null) - hiHead = e; - else - hiTail.next = e; - hiTail = e; - } - } while ((e = next) != null); - // 原索引放到bucket里 - if (loTail != null) { - loTail.next = null; - newTab[j] = loHead; - } - // 原索引+oldCap放到bucket里 - if (hiTail != null) { - hiTail.next = null; - newTab[j + oldCap] = hiHead; - } - } - } - } - } - return newTab; -} -``` - -> HashMap的扩容操作是怎么实现的？ - -1. 在jdk1.8中，resize方法是在HashMap中的键值对大于阀值时或者初始化时，就调用resize方法进行扩容； -2. 每次扩展的时候，都是扩展2倍； -3. 扩展后Node对象的位置要么在原位置，要么移动到原偏移量两倍的位置。 - -在 `putVal()` 中，我们看到在这个函数里面使用到了2次 `resize()` 方法，`resize()` 方法表示在进行第一次初始化时会对其进行扩容，或者当该数组的实际大小大于其扩容阈值（第一次为0.75 * 16 = 12），这个时候在扩容的同时也会伴随的桶上面的元素进行重新分发，这也是JDK1.8版本的一个优化的地方，在1.7中，扩容之后需要重新去计算其Hash值，根据Hash值对其进行分发，但在1.8版本中，则是根据在同一个桶的位置中进行判断(e.hash & oldCap)是否为0，重新进行hash分配后，该元素的位置要么停留在原始位置，要么移动到原始位置+增加的数组大小这个位置上 - - - -##### 链表树化 - -当桶中链表长度超过 TREEIFY_THRESHOLD（默认为8）时，就会调用 treeifyBin 方法进行树化操作。但此时并不一定会树化，因为在 treeifyBin 方法中还会判断 HashMap 的容量是否大于等于 64。如果容量大于等于 64，那么进行树化，否则优先进行扩容。 - -为什么树化还要判断整体容量是否大于 64 呢？ - -当桶数组容量比较小时，键值对节点 hash 的碰撞率可能会比较高，进而导致链表长度较长，从而导致查询效率低下。这时候我们有两种选择，一种是扩容，让哈希碰撞率低一些。另一种是树化，提高查询效率。 - -如果我们采用扩容，那么我们需要做的就是做一次链表数据的复制。而如果我们采用树化，那么我们需要将链表转化成红黑树。到这里，貌似没有什么太大的区别，但我们让场景继续延伸下去。当插入的数据越来越多，我们会发现需要转化成树的链表越来越多。 - -到了一定容量，我们就需要进行扩容了。这个时候我们有许多树化的红黑树，在扩容之时，我们需要将许多的红黑树拆分成链表，这是一个挺大的成本。而如果我们在容量小的时候就进行扩容，那么需要树化的链表就越少，我们扩容的成本也就越低。 - -接下来我们看看链表树化是怎么做的： - -```java - final void treeifyBin(Node[] tab, int hash) { - int n, index; Node e; - // 1. 容量小于 MIN_TREEIFY_CAPACITY，优先扩容 - if (tab == null || (n = tab.length) < MIN_TREEIFY_CAPACITY) - resize(); - // 2. 桶不为空，那么进行树化操作 - else if ((e = tab[index = (n - 1) & hash]) != null) { - TreeNode hd = null, tl = null; - // 2.1 先将链表转成 TreeNode 表示的双向链表 - do { - TreeNode p = replacementTreeNode(e, null); - if (tl == null) - hd = p; - else { - p.prev = tl; - tl.next = p; - } - tl = p; - } while ((e = e.next) != null); - // 2.2 将 TreeNode 表示的双向链表树化 - if ((tab[index] = hd) != null) - hd.treeify(tab); - } - } -``` - -我们可以看到链表树化的整体思路也比较清晰。首先将链表转成 TreeNode 表示的双向链表，之后再调用 `treeify()` 方法进行树化操作。那么我们继续看看 `treeify()` 方法的实现。 - -```java -final void treeify(Node[] tab) { - TreeNode root = null; - // 1. 遍历双向 TreeNode 链表，将 TreeNode 节点一个个插入 - for (TreeNode x = this, next; x != null; x = next) { - next = (TreeNode)x.next; - x.left = x.right = null; - // 2. 如果 root 节点为空，那么直接将 x 节点设置为根节点 - if (root == null) { - x.parent = null; - x.red = false; - root = x; - } - else { - K k = x.key; - int h = x.hash; - Class kc = null; - // 3. 如果 root 节点不为空，那么进行比较并在合适的地方插入 - for (TreeNode p = root;;) { - int dir, ph; - K pk = p.key; - // 4. 计算 dir 值，-1 表示要从左子树查找插入点，1表示从右子树 - if ((ph = p.hash) > h) - dir = -1; - else if (ph < h) - dir = 1; - else if ((kc == null && - (kc = comparableClassFor(k)) == null) || - (dir = compareComparables(kc, k, pk)) == 0) - dir = tieBreakOrder(k, pk); - - TreeNode xp = p; - // 5. 如果查找到一个 p 点，这个点是叶子节点 - // 那么这个就是插入位置 - if ((p = (dir <= 0) ? p.left : p.right) == null) { - x.parent = xp; - if (dir <= 0) - xp.left = x; - else - xp.right = x; - // 做插入后的动平衡 - root = balanceInsertion(root, x); - break; - } - } - } - } - // 6.将 root 节点移动到链表头 - moveRootToFront(tab, root); -} -``` - -从上面代码可以看到，treeify() 方法其实就是将双向 TreeNode 链表进一步树化成红黑树。其中大致的步骤为： - -- 遍历 TreeNode 双向链表，将 TreeNode 节点一个个插入 -- 如果 root 节点为空，那么表示红黑树现在为空，直接将该节点作为根节点。否则需要查找到合适的位置去插入 TreeNode 节点。 -- 通过比较与 root 节点的位置，不断寻找最合适的点。如果最终该节点的叶子节点为空，那么该节点 p 就是插入节点的父节点。接着，将 x 节点的 parent 引用指向 xp 节点，xp 节点的左子节点或右子节点指向 x 节点。 -- 接着，调用 balanceInsertion 做一次动态平衡。 -- 最后，调用 moveRootToFront 方法将 root 节点移动到链表头。 - -关于 balanceInsertion() 动平衡可以参考红黑树的插入动平衡，这里暂不深入讲解。最后我们继续看看 moveRootToFront 方法。 - -```java -static void moveRootToFront(Node[] tab, TreeNode root) { - int n; - if (root != null && tab != null && (n = tab.length) > 0) { - int index = (n - 1) & root.hash; - TreeNode first = (TreeNode)tab[index]; - // 如果插入红黑树的 root 节点不是链表的第一个元素 - // 那么将 root 节点取出来，插在 first 节点前面 - if (root != first) { - Node rn; - tab[index] = root; - TreeNode rp = root.prev; - // 下面的两个 if 语句，做的事情是将 root 节点取出来 - // 让 root 节点的前继指向其后继节点 - // 让 root 节点的后继指向其前继节点 - if ((rn = root.next) != null) - ((TreeNode)rn).prev = rp; - if (rp != null) - rp.next = rn; - // 这里直接让 root 节点插入到 first 节点前方 - if (first != null) - first.prev = root; - root.next = first; - root.prev = null; - } - assert checkInvariants(root); - } -} -``` - -##### 红黑树拆分 - -扩容后，普通节点需要重新映射，红黑树节点也不例外。按照一般的思路，我们可以先把红黑树转成链表，之后再重新映射链表即可。但因为红黑树插入的时候，TreeNode 保存了元素插入的顺序，所以直接可以按照插入顺序还原成链表。这样就避免了将红黑树转成链表后再进行哈希映射，无形中提高了效率。 - -```java -final void split(HashMap map, Node[] tab, int index, int bit) { - TreeNode b = this; - // Relink into lo and hi lists, preserving order - TreeNode loHead = null, loTail = null; - TreeNode hiHead = null, hiTail = null; - int lc = 0, hc = 0; - // 1. 将红黑树当成是一个 TreeNode 组成的双向链表 - // 按照链表扩容一样，分别放入 loHead 和 hiHead 开头的链表中 - for (TreeNode e = b, next; e != null; e = next) { - next = (TreeNode)e.next; - e.next = null; - // 1.1. 扩容后的位置不变，还是原来的位置，该节点放入 loHead 链表 - if ((e.hash & bit) == 0) { - if ((e.prev = loTail) == null) - loHead = e; - else - loTail.next = e; - loTail = e; - ++lc; - } - // 1.2 扩容后的位置改变了，放入 hiHead 链表 - else { - if ((e.prev = hiTail) == null) - hiHead = e; - else - hiTail.next = e; - hiTail = e; - ++hc; - } - } - // 2. 对 loHead 和 hiHead 进行树化或链表化 - if (loHead != null) { - // 2.1 如果链表长度小于阈值，那就链表化，否则树化 - if (lc <= UNTREEIFY_THRESHOLD) - tab[index] = loHead.untreeify(map); - else { - tab[index] = loHead; - if (hiHead != null) // (else is already treeified) - loHead.treeify(tab); - } - } - if (hiHead != null) { - if (hc <= UNTREEIFY_THRESHOLD) - tab[index + bit] = hiHead.untreeify(map); - else { - tab[index + bit] = hiHead; - if (loHead != null) - hiHead.treeify(tab); - } - } -} -``` - -从上面的代码我们知道红黑树的扩容也和链表的转移是一样的，不同的是其转化成 hiHead 和 loHead 之后，会根据链表长度选择拆分成链表还是继承维持红黑树的结构。 - -##### 红黑树链化 - -我们在说到红黑树拆分的时候说到，红黑树结构在扩容的时候如果长度低于阈值，那么就会将其转化成链表。其实现代码如下： - -```java -final Node untreeify(HashMap map) { - Node hd = null, tl = null; - for (Node q = this; q != null; q = q.next) { - Node p = map.replacementNode(q, null); - if (tl == null) - hd = p; - else - tl.next = p; - tl = p; - } - return hd; -} -``` - -因为红黑树中包含了插入元素的顺序，所以当我们将红黑树拆分成两个链表 hiHead 和 loHead 时，其还是保持着原来的顺序的。所以此时我们只需要循环遍历一遍，然后将 TreeNode 节点换成 Node 节点即可。 - - - -#### HashMap：JDK1.7 VS JDK1.8 - -JDK1.8主要解决或优化了一下问题： - -- resize 扩容优化 -- 引入了红黑树，目的是避免单条链表过长而影响查询效率 -- 解决了多线程死循环问题，但仍是非线程安全的，多线程时可能会造成数据丢失问题 - -| 不同 | JDK 1.7 | JDK 1.8 | -| ------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | -| 存储结构 | 数组 + 链表 | 数组 + 链表 + 红黑树 | -| 初始化方式 | 单独函数：inflateTable() | 直接集成到了扩容函数resize()中 | -| hash值计算方式 | 扰动处理 = 9次扰动 = 4次位运算 + 5次异或运算 | 扰动处理 = 2次扰动 = 1次位运算 + 1次异或运算 | -| 存放数据的规则 | 无冲突时，存放数组；冲突时，存放链表 | 无冲突时，存放数组；冲突 & 链表长度 < 8：存放单链表；冲突 & 链表长度 > 8：树化并存放红黑树 | -| 插入数据方式 | 头插法（先讲原位置的数据移到后1位，再插入数据到该位置） | 尾插法（直接插入到链表尾部/红黑树） | -| 扩容后存储位置的计算方式 | 全部按照原来方法进行计算（即hashCode ->> 扰动函数 ->> (h&length-1)） | 按照扩容后的规律计算（即扩容后的位置=原位置 or 原位置 + 旧容量 | - ------- - - - -## Hashtable - -Hashtable 和 HashMap 都是散列表，也是用”拉链法“实现的哈希表。保存数据和 JDK7 中的 HashMap 一样，是 Entity 对象，只是 Hashtable 中的几乎所有的 public 方法都是 synchronized 的，而有些方法也是在内部通过 synchronized 代码块来实现，效率肯定会降低。且 put() 方法不允许空值。 - - - -### HashMap 和 Hashtable 的区别 - -1. **线程是否安全：** HashMap 是非线程安全的，HashTable 是线程安全的；HashTable 内部的方法基本都经过 `synchronized` 修饰。（如果你要保证线程安全的话就使用 ConcurrentHashMap 吧！）； - -2. **效率：** 因为线程安全的问题，HashMap 要比 HashTable 效率高一点。另外，HashTable 基本被淘汰，不要在代码中使用它； - -3. **对Null key 和Null value的支持：** HashMap 中，null 可以作为键，这样的键只有一个，可以有一个或多个键所对应的值为 null。。但是在 HashTable 中 put 进的键值只要有一个 null，直接抛出 NullPointerException。 - -4. **初始容量大小和每次扩充容量大小的不同 ：** - - ① 创建时如果不指定容量初始值，Hashtable 默认的初始大小为11，之后每次扩充，容量变为原来的2n+1。HashMap 默认的初始化大小为16。之后每次扩充，容量变为原来的2倍。 - - ② 创建时如果给定了容量初始值，那么 Hashtable 会直接使用你给定的大小，而 HashMap 会将其扩充为2的幂次方大小。也就是说 HashMap 总是使用2的幂次方作为哈希表的大小,后面会介绍到为什么是2的幂次方。 - -5. **底层数据结构：** JDK1.8 以后的 HashMap 在解决哈希冲突时有了较大的变化，当链表长度大于阈值（默认为8）时，将链表转化为红黑树，以减少搜索时间。Hashtable 没有这样的机制。 - -6. HashMap的迭代器（`Iterator`）是fail-fast迭代器，但是 Hashtable的迭代器（`enumerator`）不是 fail-fast的。如果有其它线程对HashMap进行的添加/删除元素，将会抛出`ConcurrentModificationException`，但迭代器本身的`remove`方法移除元素则不会抛出异常。这条同样也是 Enumeration 和 Iterator 的区别。 - - - -## ConcurrentHashMap - -HashMap 在多线程情况下，在 put 的时候，插入的元素超过了容量（由负载因子决定）的范围就会触发扩容操作，就是rehash，这个会重新将原数组的内容重新hash到新的扩容数组中，在多线程的环境下，存在同时其他的元素也在进行put操作，如果hash值相同，可能出现同时在同一数组下用链表表示，造成**闭环**，导致在get时会出现死循环，所以 HashMap 是线程不安全的。(可参考：https://www.jianshu.com/p/e2f75c8cce01） - -Hashtable，是线程安全的，它在所有涉及到多线程操作的都加上了synchronized关键字来锁住整个table，这就意味着所有的线程都在竞争一把锁，在多线程的环境下，它是安全的，但是无疑是效率低下的。 - -### JDK1.7 实现 - -Hashtable 容器在竞争激烈的并发环境下表现出效率低下的原因，是因为所有访问 Hashtable 的线程都必须竞争同一把锁，那假如容器里有多把锁，每一把锁用于锁容器其中一部分数据，那么当多线程访问容器里不同数据段的数据时，线程间就不会存在锁竞争，这就是 ConcurrentHashMap 所使用的锁分段技术。 - -在 JDK1.7 版本中，ConcurrentHashMap 的数据结构是由一个 Segment 数组和多个 HashEntry 组成。Segment 数组的意义就是将一个大的 table 分割成多个小的 table 来进行加锁。每一个 Segment 元素存储的是 HashEntry数组+链表，这个和 HashMap 的数据存储结构一样。 - -![](https://yfzhou.oss-cn-beijing.aliyuncs.com/blog/img/JDK1.7%20ConcurrentHashMap.jpg) - -ConcurrentHashMap 类中包含两个静态内部类 HashEntry 和 Segment。 -HashEntry 用来封装映射表的键值对，Segment 用来充当锁的角色，每个 Segment 对象守护整个散列映射表的若干个桶。每个桶是由若干个 HashEntry 对象链接起来的链表。一个 ConcurrentHashMap 实例中包含由若干个 Segment 对象组成的数组。每个 Segment 守护着一个 HashEntry 数组里的元素，当对 HashEntry 数组的数据进行修改时，必须首先获得它对应的 Segment 锁。 - -#### Segment 类 - -Segment 类继承于 ReentrantLock 类，从而使得 Segment 对象能充当可重入锁的角色。一个 Segment 就是一个子哈希表，Segment 里维护了一个 HashEntry 数组，并发环境下，对于不同 Segment 的数据进行操作是不用考虑锁竞争的。 - -从源码可以看到，Segment 内部类和我们上边看到的 HashMap 很相似。也有负载因子，阈值等各种属性。 - -```java -static final class Segment extends ReentrantLock implements Serializable { - - static final int MAX_SCAN_RETRIES = - Runtime.getRuntime().availableProcessors() > 1 ? 64 : 1; - transient volatile HashEntry[] table; - transient int count; - transient int modCount; //记录修改次数 - transient int threshold; - final float loadFactor; - - Segment(float lf, int threshold, HashEntry[] tab) { - this.loadFactor = lf; - this.threshold = threshold; - this.table = tab; - } - - //put 方法会有加锁操作， - final V put(K key, int hash, V value, boolean onlyIfAbsent) { - HashEntry node = tryLock() ? null : - scanAndLockForPut(key, hash, value); - // ... - } - - @SuppressWarnings("unchecked") - private void rehash(HashEntry node) { - // ... - } - - private HashEntry scanAndLockForPut(K key, int hash, V value) { - //... - } - - private void scanAndLock(Object key, int hash) { - //... - } - - final V remove(Object key, int hash, Object value) { - //... - } - - final boolean replace(K key, int hash, V oldValue, V newValue) { - //... - } - - final V replace(K key, int hash, V value) { - //... - } - - final void clear() { - //... - } -} -``` - -#### HashEntry 类 - -HashEntry 是目前最小的逻辑处理单元。一个 ConcurrentHashMap 维护一个 Segment 数组，一个 Segment 维护一个 HashEntry 数组。 - -```java -static final class HashEntry { - final int hash; - final K key; - volatile V value; // value 为 volatie 类型，保证可见 - volatile HashEntry next; - //... -} -``` - -#### ConcurrentHashMap 类 - -默认的情况下，每个 ConcurrentHashMap 类会创建16个并发的 segment，每个 segment 里面包含多个 Hash表，每个 Hash 链都是由 HashEntry 节点组成的。 - -```java -public class ConcurrentHashMap extends AbstractMap - implements ConcurrentMap, Serializable { - //默认初始容量为 16，即初始默认为 16 个桶 - static final int DEFAULT_INITIAL_CAPACITY = 16; - static final float DEFAULT_LOAD_FACTOR = 0.75f; - //默认并发级别为 16。该值表示当前更新线程的估计数 - static final int DEFAULT_CONCURRENCY_LEVEL = 16; - - static final int MAXIMUM_CAPACITY = 1 << 30; - static final int MIN_SEGMENT_TABLE_CAPACITY = 2; - static final int MAX_SEGMENTS = 1 << 16; // slightly conservative - static final int RETRIES_BEFORE_LOCK = 2; - final int segmentMask; //段掩码,主要为了定位Segment - final int segmentShift; - final Segment[] segments; //主干就是这个分段锁数组 - - //构造器 - public ConcurrentHashMap(int initialCapacity, - float loadFactor, int concurrencyLevel) { - if (!(loadFactor > 0) || initialCapacity < 0 || concurrencyLevel <= 0) - throw new IllegalArgumentException(); - //MAX_SEGMENTS 为1<<16=65536，也就是最大并发数为65536 - if (concurrencyLevel > MAX_SEGMENTS) - concurrencyLevel = MAX_SEGMENTS; - // 2的sshif次方等于ssize，例:ssize=16,sshift=4;ssize=32,sshif=5 - int sshift = 0; - // ssize 为segments数组长度，根据concurrentLevel计算得出 - int ssize = 1; - while (ssize < concurrencyLevel) { - ++sshift; - ssize <<= 1; - } - this.segmentShift = 32 - sshift; - this.segmentMask = ssize - 1; - if (initialCapacity > MAXIMUM_CAPACITY) - initialCapacity = MAXIMUM_CAPACITY; - int c = initialCapacity / ssize; - if (c * ssize < initialCapacity) - ++c; - int cap = MIN_SEGMENT_TABLE_CAPACITY; - while (cap < c) - cap <<= 1; - // 创建segments数组并初始化第一个Segment，其余的Segment延迟初始化 - Segment s0 = - new Segment(loadFactor, (int)(cap * loadFactor), - (HashEntry[])new HashEntry[cap]); - Segment[] ss = (Segment[])new Segment[ssize]; - UNSAFE.putOrderedObject(ss, SBASE, s0); // ordered write of segments[0] - this.segments = ss; - } -} -``` - -#### put() 方法 - -1. **定位segment并确保定位的Segment已初始化 ** -2. **调用 Segment的 put 方法。** - -```java -public V put(K key, V value) { - Segment s; - //concurrentHashMap不允许key/value为空 - if (value == null) - throw new NullPointerException(); - //hash函数对key的hashCode重新散列，避免差劲的不合理的hashcode，保证散列均匀 - int hash = hash(key); - //返回的hash值无符号右移segmentShift位与段掩码进行位运算，定位segment - int j = (hash >>> segmentShift) & segmentMask; - if ((s = (Segment)UNSAFE.getObject // nonvolatile; recheck - (segments, (j << SSHIFT) + SBASE)) == null) // in ensureSegment - s = ensureSegment(j); - return s.put(key, hash, value, false); -} -``` - -#### get() 方法 - -**get方法无需加锁，由于其中涉及到的共享变量都使用volatile修饰，volatile可以保证内存可见性，所以不会读取到过期数据** - -```java -public V get(Object key) { - Segment s; // manually integrate access methods to reduce overhead - HashEntry[] tab; - int h = hash(key); - long u = (((h >>> segmentShift) & segmentMask) << SSHIFT) + SBASE; - if ((s = (Segment)UNSAFE.getObjectVolatile(segments, u)) != null && - (tab = s.table) != null) { - for (HashEntry e = (HashEntry) UNSAFE.getObjectVolatile - (tab, ((long)(((tab.length - 1) & h)) << TSHIFT) + TBASE); - e != null; e = e.next) { - K k; - if ((k = e.key) == key || (e.hash == h && key.equals(k))) - return e.value; - } - } - return null; -} -``` - -### JDK1.8 实现 - -![img](https://yfzhou.oss-cn-beijing.aliyuncs.com/blog/img/JDK1.8%20ConcurrentHashMap.jpg) - -ConcurrentHashMap 在 JDK8 中进行了巨大改动，光是代码量就从1000多行增加到6000行！1.8摒弃了`Segment`(锁段)的概念，采用了 `CAS + synchronized` 来保证并发的安全性。 - -可以看到，和 HashMap 1.8 的数据结构很像。底层数据结构改变为采用**数组+链表+红黑树**的数据形式。 - -#### 和HashMap1.8相同的一些地方 - -- 底层数据结构一致 -- HashMap初始化是在第一次put元素的时候进行的，而不是init -- HashMap的底层数组长度总是为2的整次幂 -- 默认树化的阈值为 8，而链表化的阈值为 6（当低于这个阈值时，红黑树转成链表） -- hash算法也很类似，但多了一步`& HASH_BITS`，该步是为了消除最高位上的负符号，hash的负在ConcurrentHashMap中有特殊意义表示在**扩容或者是树节点** - -```java -static final int HASH_BITS = 0x7fffffff; // usable bits of normal node hash - -static final int spread(int h) { - return (h ^ (h >>> 16)) & HASH_BITS; -} -``` - -#### 一些关键属性 - -```java -private static final int MAXIMUM_CAPACITY = 1 << 30; //数组最大大小 同HashMap - -private static final int DEFAULT_CAPACITY = 16;//数组默认大小 - -static final int MAX_ARRAY_SIZE = Integer.MAX_VALUE - 8; //数组可能最大值，需要与toArray（）相关方法关联 - -private static final int DEFAULT_CONCURRENCY_LEVEL = 16; //兼容旧版保留的值，默认线程并发度，类似信号量 - -private static final float LOAD_FACTOR = 0.75f;//默认map扩容比例，实际用(n << 1) - (n >>> 1)代替了更高效 - -static final int TREEIFY_THRESHOLD = 8; // 链表转树阀值，大于8时 - -static final int UNTREEIFY_THRESHOLD = 6; //树转链表阀值，小于等于6（tranfer时，lc、hc=0两个计数器分别++记录原bin、新binTreeNode数量，<=UNTREEIFY_THRESHOLD 则untreeify(lo)）。【仅在扩容tranfer时才可能树转链表】 - -static final int MIN_TREEIFY_CAPACITY = 64; - -private static final int MIN_TRANSFER_STRIDE = 16;//扩容转移时的最小数组分组大小 - -private static int RESIZE_STAMP_BITS = 16;//本类中没提供修改的方法 用来根据n生成位置一个类似时间戳的功能 - -private static final int MAX_RESIZERS = (1 << (32 - RESIZE_STAMP_BITS)) - 1; // 2^15-1，help resize的最大线程数 - -private static final int RESIZE_STAMP_SHIFT = 32 - RESIZE_STAMP_BITS; // 32-16=16，sizeCtl中记录size大小的偏移量 - -static final int MOVED = -1; // hash for forwarding nodes（forwarding nodes的hash值）、标示位 - -static final int TREEBIN = -2; // hash for roots of trees（树根节点的hash值） - -static final int RESERVED = -3; // ReservationNode的hash值 - -static final int HASH_BITS = 0x7fffffff; // 用在计算hash时进行安位与计算消除负hash - -static final int NCPU = Runtime.getRuntime().availableProcessors(); // 可用处理器数量 - -/* ---------------- Fields -------------- */ - -transient volatile Node[] table; //装载Node的数组，作为ConcurrentHashMap的数据容器，采用懒加载的方式，直到第一次插入数据的时候才会进行初始化操作，数组的大小总是为2的幂次方。 - -private transient volatile Node[] nextTable; //扩容时使用，平时为null，只有在扩容的时候才为非null - -/** -* 实际上保存的是hashmap中的元素个数 利用CAS锁进行更新但它并不用返回当前hashmap的元素个数 -*/ -private transient volatile long baseCount; - -/** -*该属性用来控制table数组的大小，根据是否初始化和是否正在扩容有几种情况： -*当值为负数时：如果为-1表示正在初始化，如果为-N则表示当前正有N-1个线程进行扩容操作； -*当值为正数时：如果当前数组为null的话表示table在初始化过程中，sizeCtl表示为需要新建数组的长度；若已经初始化了，表示当前数据容器（table数组）可用容量也可以理解成临界值（插入节点数超过了该临界值就需要扩容），具体指为数组的长度n 乘以 加载因子loadFactor；当值为0时，即数组长度为默认初始值。 -*/ -private transient volatile int sizeCtl; -``` - -#### put() 方法 - -1. 首先会判断 key、value是否为空，如果为空就抛异常！ -2. `spread()`方法获取hash，减小hash冲突 -3. 判断是否初始化table数组，没有的话调用`initTable()`方法进行初始化 -4. 判断是否能直接将新值插入到table数组中 -5. 判断当前是否在扩容，`MOVED`为-1说明当前ConcurrentHashMap正在进行扩容操作，正在扩容的话就进行协助扩容 -6. 当table[i]为链表的头结点，在链表中插入新值，通过synchronized (f)的方式进行加锁以实现线程安全性。 - 1. 在链表中如果找到了与待插入的键值对的key相同的节点，就直接覆盖 - 2. 如果没有找到的话，就直接将待插入的键值对追加到链表的末尾 -7. 当table[i]为红黑树的根节点，在红黑树中插入新值/覆盖旧值 -8. 根据当前节点个数进行调整，否需要转换成红黑树(个数大于等于8，就会调用`treeifyBin`方法将tabel[i]`第i个散列桶`拉链转换成红黑树) -9. 对当前容量大小进行检查，如果超过了临界值（实际大小*加载因子）就进行扩容 - -```java -final V putVal(K key, V value, boolean onlyIfAbsent) { - // key 和 value 均不允许为 null - if (key == null || value == null) throw new NullPointerException(); - // 根据 key 计算出 hash 值 - int hash = spread(key.hashCode()); - int binCount = 0; - for (Node[] tab = table;;) { - Node f; int n, i, fh; - // 判断是否需要进行初始化 - if (tab == null || (n = tab.length) == 0) - tab = initTable(); - // f 即为当前 key 定位出的 Node，如果为空表示当前位置可以写入数据，利用 CAS 尝试写入，失败则自旋保证成功 - else if ((f = tabAt(tab, i = (n - 1) & hash)) == null) { - if (casTabAt(tab, i, null, - new Node(hash, key, value, null))) - break; // no lock when adding to empty bin - } - // 如果当前位置的 hashcode == MOVED == -1,则需要进行扩容 - else if ((fh = f.hash) == MOVED) - tab = helpTransfer(tab, f); - // 如果都不满足，则利用 synchronized 锁写入数据 - else { - // 剩下情况又分两种，插入链表、插入红黑树 - V oldVal = null; - //采用同步内置锁实现并发控制 - synchronized (f) { - if (tabAt(tab, i) == f) { - // 如果 fh=f.hash >=0，当前为链表，在链表中插入新的键值对 - if (fh >= 0) { - binCount = 1; - //遍历链表，如果找到对应的 node 节点，修改 value,否则直接在链表尾部加入节点 - for (Node e = f;; ++binCount) { - K ek; - if (e.hash == hash && - ((ek = e.key) == key || - (ek != null && key.equals(ek)))) { - oldVal = e.val; - if (!onlyIfAbsent) - e.val = value; - break; - } - Node pred = e; - if ((e = e.next) == null) { - pred.next = new Node(hash, key, - value, null); - break; - } - } - } - // 当前为红黑树，将新的键值对插入到红黑树中 - else if (f instanceof TreeBin) { - Node p; - binCount = 2; - if ((p = ((TreeBin)f).putTreeVal(hash, key, - value)) != null) { - oldVal = p.val; - if (!onlyIfAbsent) - p.val = value; - } - } - } - } - // 插入完键值对后再根据实际大小看是否需要转换成红黑树 - if (binCount != 0) { - if (binCount >= TREEIFY_THRESHOLD) - treeifyBin(tab, i); - if (oldVal != null) - return oldVal; - break; - } - } - } - // 对当前容量大小进行检查，如果超过了临界值（实际大小*加载因子）就需要扩容 - addCount(1L, binCount); - return null; -} -``` - -我们可以发现 JDK8 中的实现也是**锁分离**的思想，只是锁住的是一个Node，而不是JDK7中的Segment，而锁住Node之前的操作是无锁的并且也是线程安全的，建立在之前提到的原子操作上。 - -#### get() 方法 - -get方法无需加锁，由于其中涉及到的共享变量都使用volatile修饰，volatile可以保证内存可见性，所以不会读取到过期数据 - -```java -public V get(Object key) { - Node[] tab; Node e, p; int n, eh; K ek; - int h = spread(key.hashCode()); - // 判断数组是否为空 - if ((tab = table) != null && (n = tab.length) > 0 && - (e = tabAt(tab, (n - 1) & h)) != null) { - // 判断node 节点第一个元素是不是要找的，如果是直接返回 - if ((eh = e.hash) == h) { - if ((ek = e.key) == key || (ek != null && key.equals(ek))) - return e.val; - } - // // hash小于0,说明是特殊节点（TreeBin或ForwardingNode）调用find - else if (eh < 0) - return (p = e.find(h, key)) != null ? p.val : null; - // 不是上面的情况,那就是链表了,遍历链表 - while ((e = e.next) != null) { - if (e.hash == h && - ((ek = e.key) == key || (ek != null && key.equals(ek)))) - return e.val; - } - } - return null; -} -``` - - - -### Hashtable 和 ConcurrentHashMap 的区别 - -ConcurrentHashMap 和 Hashtable 的区别主要体现在实现线程安全的方式上不同。 - -- **底层数据结构：** JDK1.7的 ConcurrentHashMap 底层采用 **分段的数组+链表** 实现，JDK1.8 采用的数据结构和 HashMap1.8 的结构类似，**数组+链表/红黑二叉树**。Hashtable 和 JDK1.8 之前的 HashMap 的底层数据结构类似都是采用 **数组+链表** 的形式，数组是 HashMap 的主体，链表则是主要为了解决哈希冲突而存在的； -- **实现线程安全的方式（重要）：** - - **在JDK1.7的时候，ConcurrentHashMap（分段锁）** 对整个桶数组进行了分割分段(Segment)，每一把锁只锁容器其中一部分数据，多线程访问容器里不同数据段的数据，就不会存在锁竞争，提高并发访问率。（默认分配16个Segment，比Hashtable效率提高16倍。） **到了 JDK1.8 的时候已经摒弃了Segment的概念，而是直接用 Node 数组+链表/红黑树的数据结构来实现，并发控制使用 synchronized 和 CAS 来操作。（JDK1.6以后 对 synchronized锁做了很多优化）** 整个看起来就像是优化过且线程安全的 HashMap，虽然在 JDK1.8 中还能看到 Segment 的数据结构，但是已经简化了属性，只是为了兼容旧版本； - - Hashtable(同一把锁) ：使用 synchronized 来保证线程安全，效率非常低下。当一个线程访问同步方法时，其他线程也访问同步方法，可能会进入阻塞或轮询状态，如使用 put 添加元素，另一个线程不能使用 put 添加元素，也不能使用 get，竞争越激烈效率越低。 - - - -## Java快速失败（fail-fast）和安全失败（fail-safe）区别 - -### 快速失败（fail—fast） - -在用迭代器遍历一个集合对象时，如果遍历过程中对集合对象的内容进行了修改（增加、删除、修改），则会抛出 `ConcurrentModificationException`。 - -原理：迭代器在遍历时直接访问集合中的内容，并且在遍历过程中使用一个 modCount 变量。集合在被遍历期间如果内容发生变化，就会改变 modCount 的值。每当迭代器使用 hashNext()/next() 遍历下一个元素之前，都会检测 modCount 变量是否为 expectedmodCount 值，是的话就返回遍历；否则抛出异常，终止遍历。 - -注意：这里异常的抛出条件是检测到 modCount！=expectedmodCount 这个条件。如果集合发生变化时修改modCount 值刚好又设置为了 expectedmodCount 值，则异常不会抛出。因此，不能依赖于这个异常是否抛出而进行并发操作的编程，这个异常只建议用于检测并发修改的bug。 - -场景：`java.util` 包下的集合类都是快速失败的，不能在多线程下发生并发修改（迭代过程中被修改）。 - -### 安全失败（fail—safe） - -采用安全失败机制的集合容器，在遍历时不是直接在集合内容上访问的，而是先复制原有集合内容，在拷贝的集合上进行遍历。 - -原理：由于迭代时是对原集合的拷贝进行遍历，所以在遍历过程中对原集合所作的修改并不能被迭代器检测到，所以不会触发 `Concurrent Modification Exception`。 - -缺点：基于拷贝内容的优点是避免了 `Concurrent Modification Exception`，但同样地，迭代器并不能访问到修改后的内容，即：迭代器遍历的是开始遍历那一刻拿到的集合拷贝，在遍历期间原集合发生的修改迭代器是不知道的。 - -场景：`java.util.concurrent` 包下的容器都是安全失败，可以在多线程下并发使用，并发修改。 - -快速失败和安全失败是对迭代器而言的。 - -快速失败：当在迭代一个集合的时候，如果有另外一个线程在修改这个集合，就会抛出`ConcurrentModification`异常，`java.util` 下都是快速失败。 - -安全失败：在迭代时候会在集合二层做一个拷贝，所以在修改集合上层元素不会影响下层。在 `java.util.concurrent` 下都是安全失败 - - - -### 如何避免**fail-fast** ？ - -- 在单线程的遍历过程中，如果要进行 remove 操作，可以调用迭代器 ListIterator 的 remove 方法而不是集合类的 remove方法。看看 ArrayList 中迭代器的 remove 方法的源码，该方法不能指定元素删除，只能 remove 当前遍历元素。 - -```java -public void remove() { - if (lastRet < 0) - throw new IllegalStateException(); - checkForComodification(); - - try { - SubList.this.remove(lastRet); - cursor = lastRet; - lastRet = -1; - expectedModCount = ArrayList.this.modCount; // - } catch (IndexOutOfBoundsException ex) { - throw new ConcurrentModificationException(); - } -} -``` - -- 使用并发包(`java.util.concurrent`)中的类来代替 ArrayList 和 hashMap - - CopyOnWriterArrayList 代替 ArrayList - - ConcurrentHashMap 代替 HashMap - - - -## Iterator 和 Enumeration 区别 - -在 Java 集合中，我们通常都通过 “Iterator(迭代器)” 或 “Enumeration(枚举类)” 去遍历集合。 - -```java -public interface Enumeration { - boolean hasMoreElements(); - E nextElement(); -} -``` - -```java -public interface Iterator { - boolean hasNext(); - E next(); - void remove(); -} -``` - -- **函数接口不同**，Enumeration**只有2个函数接口。**通过Enumeration，我们只能读取集合的数据，而不能对数据进行修改。Iterator**只有3个函数接口。**Iterator除了能读取集合的数据之外，也能数据进行删除操作。 -- **Iterator支持 fail-fast机制，而Enumeration不支持**。Enumeration 是 JDK 1.0 添加的接口。使用到它的函数包括 Vector、Hashtable 等类，这些类都是 JDK 1.0中加入的，Enumeration 存在的目的就是为它们提供遍历接口。Enumeration 本身并没有支持同步，而在 Vector、Hashtable 实现 Enumeration 时，添加了同步。 - 而 Iterator 是 JDK 1.2 才添加的接口，它也是为了 HashMap、ArrayList 等集合提供遍历接口。Iterator 是支持 fail-fast 机制的：当多个线程对同一个集合的内容进行操作时，就可能会产生 fail-fast 事件 - - - -## Comparable 和 Comparator 接口有何区别？ - -Java中对集合对象或者数组对象排序，有两种实现方式： - -- 对象实现Comparable 接口 - - - Comparable 在 `java.lang` 包下，是一个接口，内部只有一个方法 `compareTo()` - - ```java - public interface Comparable { - public int compareTo(T o); - } - ``` - - - Comparable 可以让实现它的类的对象进行比较，具体的比较规则是按照 compareTo 方法中的规则进行。这种顺序称为 **自然顺序**。 - - 实现了 Comparable 接口的 List 或则数组可以使用 `Collections.sort()` 或者 `Arrays.sort()` 方法进行排序 - -- 定义比较器，实现 Comparator接口 - - - Comparator 在 `java.util` 包下，也是一个接口，JDK 1.8 以前只有两个方法： - - ```java - public interface Comparator { - public int compare(T lhs, T rhs); - public boolean equals(Object object); - } - ``` - -**comparable相当于内部比较器。comparator相当于外部比较器** - -区别： - -- Comparator 位于 `java.util` 包下，而 Comparable 位于 `java.lang` 包下 - -- Comparable 接口的实现是在类的内部（如 String、Integer已经实现了 Comparable 接口，自己就可以完成比较大小操作），Comparator 接口的实现是在类的外部（可以理解为一个是自已完成比较，一个是外部程序实现比较） - -- 实现 Comparable 接口要重写 compareTo 方法, 在 compareTo 方法里面实现比较。一个已经实现Comparable 的类的对象或数据，可以通过 **Collections.sort(list) 或者 Arrays.sort(arr)**实现排序。通过 **Collections.sort(list,Collections.reverseOrder())** 对list进行倒序排列。 - - ![](https://tva1.sinaimg.cn/large/007S8ZIlly1gdzefrcjolj30ui0u0dm0.jpg) - -- 实现Comparator需要重写 compare 方法 - - ![](https://tva1.sinaimg.cn/large/007S8ZIlly1gdzefws30uj311s0t643v.jpg) - - - -## HashSet - -HashSet 是用来存储没有重复元素的集合类，并且它是无序的。HashSet 内部实现是基于 HashMap ，实现了 Set 接口。 - -从 HahSet 提供的构造器可以看出，除了最后一个 HashSet 的构造方法外，其他所有内部就是去创建一个 HashMap 。没有其他的操作。而最后一个构造方法不是 public 的，所以不对外公开。 - -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gdzeqxhvj3j311s0ka787.jpg) - - - -### HashSet如何检查重复 - -HashSet 的底层其实就是 HashMap，只不过我们 **HashSet 是实现了 Set 接口并且把数据作为 K 值，而 V 值一直使用一个相同的虚值来保存**，HashMap的 K 值本身就不允许重复，并且在 HashMap 中如果 K/V 相同时，会用新的 V 覆盖掉旧的 V，然后返回旧的 V。 - -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gdzeyvg5h0j311s0jen0n.jpg) - - - -### Iterater 和 ListIterator 之间有什么区别？ - -- 我们可以使用Iterator来遍历Set和List集合，而ListIterator只能遍历List -- ListIterator有add方法，可以向List中添加对象，而Iterator不能 -- ListIterator和Iterator都有hasNext()和next()方法，可以实现顺序向后遍历，但是ListIterator有hasPrevious()和previous()方法，可以实现逆向（顺序向前）遍历。Iterator不可以 -- ListIterator可以定位当前索引的位置，nextIndex()和previousIndex()可以实现。Iterator没有此功能 -- 都可实现删除操作，但是 ListIterator可以实现对象的修改，set()方法可以实现。Iterator仅能遍历，不能修改 - - - - - - - -## 参考与感谢 - -所有内容都是基于源码阅读和各种大佬之前总结的知识整理而来，输入并输出，奥利给。 - -https://www.javatpoint.com/java-arraylist - -https://www.runoob.com/java/java-collections.html - -https://www.javazhiyin.com/21717.html - -https://yuqirong.me/2018/01/31/LinkedList内部原理解析/ - -https://youzhixueyuan.com/the-underlying-structure-and-principle-of-hashmap.html - -《HashMap源码详细分析》http://www.tianxiaobo.com/2018/01/18/HashMap-源码详细分析-JDK1-8/ - -《ConcurrentHashMap1.7源码分析》https://www.cnblogs.com/chengxiao/p/6842045.html - -http://www.justdojava.com/2019/12/18/java-collection-15.1/ \ No newline at end of file diff --git a/docs/interview/Design-Pattern-FAQ.md b/docs/interview/Design-Pattern-FAQ.md index 667a504cca..2cf2a695a6 100644 --- a/docs/interview/Design-Pattern-FAQ.md +++ b/docs/interview/Design-Pattern-FAQ.md @@ -1,10 +1,882 @@ -使用UML类图画出原型模式核心角色 +--- +title: 设计模式八股文 +date: 2024-08-31 +tags: + - 设计模式 + - Interview +categories: Interview +--- -原型设计模式的深拷贝和浅拷贝是什么，并写出深拷贝的两种方式的源码(重写 clone 方法实现深拷贝、使用序列化来实现深拷贝 +![](https://img.starfish.ink/common/faq-banner.png) -设计模式的七大原则 +> 设计模式是Java开发者**必备的技能**，也是面试中**高频考点**。从经典的23种GoF设计模式到Spring框架中的模式应用，从单例模式的7种写法到代理模式的动态实现，每一个模式都承载着前辈的智慧结晶。掌握设计模式，让你的代码更优雅、更健壮、更具扩展性！ +> +> 设计模式面试要点： +> +> - 设计原则（SOLID原则、迪米特法则、合成复用原则） +> - 创建型模式（单例、工厂、建造者、原型、抽象工厂） +> - 结构型模式（适配器、装饰器、代理、桥接、组合、外观、享元） +> - 行为型模式（观察者、策略、模板方法、责任链、状态、命令） +> - 实际应用（Spring中的模式、项目实战经验） + + + +## 🗺️ 知识导航 + +### 🏷️ 核心知识分类 + +1. **📏 设计原则**：SOLID原则、开闭原则、里氏替换原则、依赖倒置原则、单一职责原则 +2. **🏗️ 创建型模式**：单例模式、工厂模式、建造者模式、原型模式、抽象工厂模式 +3. **🔧 结构型模式**：适配器模式、装饰器模式、代理模式、桥接模式、组合模式、外观模式、享元模式 +4. **⚡ 行为型模式**：观察者模式、策略模式、模板方法模式、责任链模式、状态模式、命令模式、中介者模式 +5. **🌟 实际应用**：Spring中的设计模式、项目实战案例、最佳实践 + +### 🔑 面试话术模板 + +| **问题类型** | **回答框架** | **关键要点** | **深入扩展** | +| ------------ | ----------------------------------- | ------------------ | ------------------ | +| **概念解释** | 定义→目的→结构→应用 | 核心思想，解决问题 | 源码分析，实际项目 | +| **模式对比** | 相同点→不同点→使用场景→选择建议 | 多维度对比 | 性能差异，适用性 | +| **实现原理** | 背景→问题→解决方案→代码实现 | UML图，代码结构 | 源码实现，优缺点 | +| **应用实践** | 项目背景→遇到问题→选择模式→效果评估 | 实际案例 | 最佳实践，踩坑经验 | + +--- + +## 📏 一、设计原则（Design Principles） + +> **核心思想**：设计原则是设计模式的理论基础，指导我们写出高质量的代码，是所有设计模式遵循的基本准则。 + +### 🎯 什么是设计模式？你是否在你的代码里面使用过任何设计模式？ + +"设计模式是软件开发中经过验证的，用于解决特定环境下重复出现问题的解决方案： + +**设计模式的本质**： + +- 是一套被反复使用、多数人知晓的、经过分类的、代码设计经验的总结 +- 提供了一种统一的术语和概念，便于开发者之间的沟通 +- 提高代码的可重用性、可读性、可靠性和可维护性 + +**GoF 23种设计模式**： + +- 创建型模式（5种）：关注对象的创建 +- 结构型模式（7种）：关注类和对象的组合 +- 行为型模式（11种）：关注对象之间的通信 + +**在项目中的应用**： + +- 单例模式：配置管理器、数据库连接池 +- 工厂模式：对象创建、消息处理器选择 +- 观察者模式：事件处理、消息通知 +- 策略模式：支付方式选择、算法切换 + +设计模式不是万能的，要根据具体场景选择，避免过度设计。" + +### 🎯 请说说你了解的设计原则有哪些？ + +"设计原则是设计模式的理论基础，主要有SOLID五大原则： + +**SOLID原则**： + +**1. 单一职责原则（Single Responsibility Principle, SRP）**： + +- 一个类只应该有一个引起它变化的原因 +- 每个类只负责一项职责，降低代码复杂度 +- 例：User类只负责用户属性，UserService负责用户业务逻辑 + +**2. 开闭原则（Open-Closed Principle, OCP）**： + +- 对扩展开放，对修改关闭 +- 通过抽象和多态实现功能扩展而不修改现有代码 +- 例：策略模式中新增支付方式无需修改原有支付逻辑 + +**3. 里氏替换原则（Liskov Substitution Principle, LSP）**： + +- 子类对象能够替换父类对象而不影响程序正确性 +- 子类必须能够完全替代父类，保持行为一致性 +- 例：Rectangle和Square的继承关系设计 + +**4. 接口隔离原则（Interface Segregation Principle, ISP）**： + +- 客户端不应该依赖它不需要的接口 +- 使用多个专门的接口比使用单一的总接口要好 +- 例：将大接口拆分成多个小而专一的接口 + +**5. 依赖倒置原则（Dependency Inversion Principle, DIP）**： + +- 高层模块不应该依赖低层模块，两者都应该依赖抽象 +- 抽象不应该依赖细节，细节应该依赖抽象 +- 例：控制器依赖Service接口而不是具体实现类 + +**其他重要原则**： + +- **合成复用原则**：优先使用对象组合而非类继承 +- **迪米特法则**：一个对象应该对其他对象保持最少的了解 + +这些原则指导我们设计出松耦合、高内聚、可扩展的代码。" + + + +### 🎯 你项目里用过哪些设计模式？具体是怎么用的？ + +> 在广告归因和回传系统中，我主要用到了以下几类设计模式： + +- **工厂模式（Factory Pattern）**：负责不同媒体渠道回传服务的实例创建，具体要走什么回传，是扣量还是高质量等等，是在入参就可以决定的； +- **策略模式（Strategy Pattern）**：负责不同媒体回传逻辑、验签算法和请求格式； +- **模板方法模式（Template Method）**：抽象出回传的公共流程； +- **责任链模式（Chain of Responsibility）**：对每个回传请求按规则依次校验； +- **单例模式（Singleton Pattern）**：保证配置类、客户端池等资源唯一； + + + +### 🎯 什么是开闭原则？如何在代码中应用？ + +"开闭原则是面向对象设计的核心原则，对扩展开放，对修改关闭： + +**核心思想**： + +- 当需求变化时，应该通过扩展现有代码来实现变化 +- 而不是修改现有的代码 +- 通过抽象化来实现开闭原则 + +**实现方式**： + +- 使用抽象类或接口来定义规范 +- 具体实现通过继承或实现来扩展功能 +- 依赖注入和多态来实现灵活的扩展 + +**代码示例**： + +```java +// 抽象支付接口 +interface PaymentStrategy { + void pay(double amount); +} + +// 具体支付实现 +class AlipayPayment implements PaymentStrategy { + public void pay(double amount) { + System.out.println("支付宝支付: " + amount); + } +} + +class WechatPayment implements PaymentStrategy { + public void pay(double amount) { + System.out.println("微信支付: " + amount); + } +} + +// 支付上下文 +class PaymentContext { + private PaymentStrategy strategy; + + public void setStrategy(PaymentStrategy strategy) { + this.strategy = strategy; + } + + public void executePayment(double amount) { + strategy.pay(amount); + } +} +``` + +**新增银行卡支付时**： + +- 只需新增BankCardPayment类实现PaymentStrategy +- 无需修改PaymentContext或其他现有代码 +- 完美体现了对扩展开放，对修改关闭 + +开闭原则是设计模式的基石，策略模式、工厂模式等都体现了这一原则。" + +--- + +## 🏗️ 二、创建型模式（Object Creation） + +> **核心思想**：封装对象的创建过程，使系统独立于如何创建、组合和表示对象。 + + + +### 🎯 请列举出在 JDK 中几个常用的设计模式？ + +"JDK中大量使用了设计模式，这些模式的应用体现了优秀的软件设计： + +**创建型模式**： + +- 单例模式：Runtime类、Calendar类确保全局唯一实例 +- 工厂模式：Boolean.valueOf()、Integer.valueOf()等工厂方法 +- 建造者模式：StringBuilder、StringBuffer的链式调用 + +**结构型模式**： + +- 装饰器模式：Java IO流体系（BufferedReader装饰FileReader） +- 适配器模式：Arrays.asList()将数组适配为List +- 代理模式：动态代理机制（java.lang.reflect.Proxy） + +**行为型模式**： + +- 观察者模式：Swing/AWT事件机制、PropertyChangeSupport +- 策略模式：Collections.sort()的Comparator参数 +- 模板方法模式：AbstractList、AbstractMap等抽象类 +- 迭代器模式：Collection接口的iterator()方法 + +这些模式的应用让JDK的API设计更加灵活和可扩展。" + + + +### 🎯 什么是单例模式？有哪些应用场景？请用 Java 写出线程安全的单例模式 + +"单例模式确保一个类只有一个实例，并提供全局访问点： + +**单例模式定义**： + +- 保证一个类仅有一个实例，并提供一个访问它的全局访问点 +- 控制实例数量，避免资源浪费和状态不一致 +- 延迟初始化，需要时才创建实例 + +**应用场景**： + +- 系统资源：数据库连接池、线程池、缓存管理器 +- 配置对象：应用程序配置、系统设置管理 +- 工具类：日志记录器、打印机管理器 +- 硬件接口：设备驱动程序（如打印机驱动） + +**优缺点分析**： + +- 优点：节约内存、避免重复创建、全局访问点 +- 缺点：违反单一职责、不利于测试、可能成为性能瓶颈" + +**💻 单例模式的7种实现方式**： + +**推荐程度排序：枚举 > 静态内部类 > 双重校验锁 > 饿汉式 > 懒汉式** + +**1. 懒汉式（线程不安全）**： + +```java +public class LazySingleton { + private static LazySingleton instance; + + private LazySingleton() {} + + public static LazySingleton getInstance() { + if (instance == null) { + instance = new LazySingleton(); // 多线程下可能创建多个实例 + } + return instance; + } +} +``` + +**2. 懒汉式（线程安全，同步效率低）**： + +```java +public class SynchronizedLazySingleton { + private static SynchronizedLazySingleton instance; + + private SynchronizedLazySingleton() {} + + public static synchronized SynchronizedLazySingleton getInstance() { + if (instance == null) { + instance = new SynchronizedLazySingleton(); + } + return instance; + } +} +``` + +**3. 饿汉式（推荐）**： + +```java +public class EagerSingleton { + private static final EagerSingleton INSTANCE = new EagerSingleton(); + + private EagerSingleton() {} + + public static EagerSingleton getInstance() { + return INSTANCE; + } +} +``` + +**4. 饿汉式变种（静态代码块）**： + +```java +public class StaticBlockSingleton { + private static final StaticBlockSingleton INSTANCE; + + static { + INSTANCE = new StaticBlockSingleton(); + } + + private StaticBlockSingleton() {} + + public static StaticBlockSingleton getInstance() { + return INSTANCE; + } +} +``` + +**5. 静态内部类（强烈推荐）**： + +```java +public class InnerClassSingleton { + private InnerClassSingleton() {} + + // 静态内部类，JVM保证线程安全，懒加载 + private static class SingletonHolder { + private static final InnerClassSingleton INSTANCE = new InnerClassSingleton(); + } + + public static InnerClassSingleton getInstance() { + return SingletonHolder.INSTANCE; // 触发内部类加载 + } +} +``` + +**6. 枚举（最佳实现）**： + +```java +public enum EnumSingleton { + INSTANCE; + + // 可以添加其他方法 + public void doSomething() { + System.out.println("EnumSingleton doing something..."); + } + + public void anotherMethod() { + // 业务逻辑 + } +} + +// 使用方式 +// EnumSingleton.INSTANCE.doSomething(); +``` + +**7. 双重校验锁（DCL，推荐）**： + +```java +public class DoubleCheckedLockingSingleton { + // volatile防止指令重排序 + private volatile static DoubleCheckedLockingSingleton instance; + + private DoubleCheckedLockingSingleton() {} + + public static DoubleCheckedLockingSingleton getInstance() { + // 第一次检查，避免不必要的同步 + if (instance == null) { + synchronized (DoubleCheckedLockingSingleton.class) { + // 第二次检查，确保线程安全 + if (instance == null) { + instance = new DoubleCheckedLockingSingleton(); + } + } + } + return instance; + } +} +``` + +**💡 各种实现方式对比**： + +| **实现方式** | **线程安全** | **懒加载** | **性能** | **推荐度** | **特点** | +| ------------ | ------------ | ---------- | -------- | ---------- | -------- | +| 懒汉式 | ❌ | ✅ | 高 | ⭐ | 最简单但不安全 | +| 同步懒汉式 | ✅ | ✅ | 低 | ⭐⭐ | 安全但效率低 | +| 饿汉式 | ✅ | ❌ | 高 | ⭐⭐⭐ | 简单安全，可能浪费内存 | +| 静态内部类 | ✅ | ✅ | 高 | ⭐⭐⭐⭐⭐ | 完美解决方案 | +| 枚举 | ✅ | ❌ | 高 | ⭐⭐⭐⭐⭐ | 最简洁，防反射和序列化 | +| 双重校验锁 | ✅ | ✅ | 高 | ⭐⭐⭐⭐ | 复杂但高效 | + + +### 🎯 如何防止反射和序列化破坏单例？ + +"单例模式的两大威胁是反射和序列化，需要采取防护措施： + +**反射攻击演示**： + +```java +// 正常获取单例 +Singleton instance1 = Singleton.getInstance(); + +// 反射破坏单例 +Constructor constructor = Singleton.class.getDeclaredConstructor(); +constructor.setAccessible(true); +Singleton instance2 = constructor.newInstance(); + +System.out.println(instance1 == instance2); // false，单例被破坏 +``` + +**防御措施**： + +**方案1：构造函数防护** + +```java +public class AntiReflectionSingleton { + private static volatile AntiReflectionSingleton instance; + private static boolean initialized = false; + + private AntiReflectionSingleton() { + synchronized (AntiReflectionSingleton.class) { + if (initialized) { + throw new RuntimeException("单例已存在，禁止反射创建"); + } + initialized = true; + } + } +} +``` + +**方案2：枚举天然防御** + +```java +public enum EnumSingleton { + INSTANCE; + // JVM层面防止反射调用枚举构造器 +} +``` + + + +**序列化攻击演示**： + +```java +// 序列化破坏单例 +Singleton instance1 = Singleton.getInstance(); + +// 序列化 +ObjectOutputStream oos = new ObjectOutputStream(new FileOutputStream("singleton.ser")); +oos.writeObject(instance1); + +// 反序列化创建新实例 +ObjectInputStream ois = new ObjectInputStream(new FileInputStream("singleton.ser")); +Singleton instance2 = (Singleton) ois.readObject(); + +System.out.println(instance1 == instance2); // false，单例被破坏 +``` + +**防御措施：实现readResolve方法** + +```java +public class SerializationSafeSingleton implements Serializable { + private static final long serialVersionUID = 1L; + private static volatile SerializationSafeSingleton instance; + + private SerializationSafeSingleton() {} + + public static SerializationSafeSingleton getInstance() { + if (instance == null) { + synchronized (SerializationSafeSingleton.class) { + if (instance == null) { + instance = new SerializationSafeSingleton(); + } + } + } + return instance; + } + + // 关键：反序列化时返回已有实例 + private Object readResolve() { + return getInstance(); + } +} +``` + +**终极解决方案：枚举单例** + +```java +public enum UltimateSingleton { + INSTANCE; + + public void doSomething() { + System.out.println("枚举单例天然防止反射和序列化攻击"); + } +} +``` + +枚举是实现单例的最佳方式：线程安全、懒加载、防反射、防序列化。" + +**💡 单例模式最佳实践**： + +1. **优先选择枚举**：代码简洁，天然安全 +2. **次选静态内部类**：懒加载+高性能 +3. **避免反射攻击**：构造器中添加检查 +4. **防序列化破坏**：实现readResolve方法 +5. **考虑实际需求**：是否真的需要全局唯一 + + + +### 🎯 使用工厂模式最主要的好处是什么？在哪里使用？ + +"工厂模式是创建型设计模式的核心，主要解决对象创建的复杂性： + +**工厂模式的好处**： + +- **解耦创建和使用**：客户端不需要知道具体对象的创建细节 +- **封装变化**：新增产品类型时，只需扩展工厂，无需修改客户端 +- **统一管理**：集中控制对象的创建逻辑，便于维护 +- **提高复用性**：相同的创建逻辑可以被多处复用 + +**应用场景**： + +- **数据库连接**：根据配置创建不同数据库的连接 +- **日志框架**：根据级别创建不同的日志处理器 +- **UI组件**：根据操作系统创建对应的界面组件 +- **支付系统**：根据支付方式创建对应的支付处理器 + +**JDK中的应用**： + +- `Boolean.valueOf()`, `Integer.valueOf()`等工厂方法 +- `Calendar.getInstance()`获取日历实例 +- `NumberFormat.getInstance()`创建格式化器 + +工厂模式让系统更加灵活，符合开闭原则。" + + + +### 🎯 简单工厂、工厂方法和抽象工厂的区别？ + +"工厂模式有三种类型，复杂程度和应用场景各不相同： + +**简单工厂（Simple Factory）**： + +- 一个工厂类根据参数创建不同产品 +- 违反开闭原则，新增产品需要修改工厂类 +- 适用于产品种类较少且稳定的场景 + +**工厂方法（Factory Method）**： + +- 每种产品对应一个具体工厂 +- 符合开闭原则，新增产品只需新增工厂 +- 适用于产品族较少但产品种类可能增加的场景 + +**抽象工厂（Abstract Factory）**： + +- 创建一系列相关的产品族 +- 适用于有多个产品族，每个产品族有多个产品的场景 +- 如：不同风格的UI组件（Windows风格、Mac风格） + +**选择建议**：简单工厂适合小项目，工厂方法适合中型项目，抽象工厂适合复杂的产品体系。" + +**💻 代码示例对比**： + +**简单工厂示例**： +```java +// 产品接口 +interface Product { + void use(); +} + +// 具体产品 +class ProductA implements Product { + public void use() { System.out.println("使用产品A"); } +} + +class ProductB implements Product { + public void use() { System.out.println("使用产品B"); } +} + +// 简单工厂 +class SimpleFactory { + public static Product createProduct(String type) { + switch (type) { + case "A": return new ProductA(); + case "B": return new ProductB(); + default: throw new IllegalArgumentException("未知产品类型"); + } + } +} +``` + +**工厂方法示例**： +```java +// 抽象工厂 +interface Factory { + Product createProduct(); +} + +// 具体工厂 +class ProductAFactory implements Factory { + public Product createProduct() { + return new ProductA(); + } +} + +class ProductBFactory implements Factory { + public Product createProduct() { + return new ProductB(); + } +} +``` + +**抽象工厂示例**： +```java +// 抽象工厂 +interface AbstractFactory { + ProductA createProductA(); + ProductB createProductB(); +} + +// 具体工厂族 +class WindowsFactory implements AbstractFactory { + public ProductA createProductA() { return new WindowsProductA(); } + public ProductB createProductB() { return new WindowsProductB(); } +} + +class MacFactory implements AbstractFactory { + public ProductA createProductA() { return new MacProductA(); } + public ProductB createProductB() { return new MacProductB(); } +} +``` + +--- + +## 🔧 三、结构型模式（Object Composition） + +> **核心思想**：关注类和对象的组合，通过组合获得更强大的功能，解决如何将类或对象按某种布局组成更大的结构。 + +### 🎯 什么是代理模式？有哪几种代理？ + +**代理模式（Proxy Pattern）** 是一种结构型设计模式，用于为目标对象提供一个代理对象，通过代理对象来控制对目标对象的访问。代理对象在客户端与目标对象之间起到中介的作用，可以对访问进行控制、增强或简化。 + +- **静态代理**：在编译阶段，代理类已经被定义好。 +- **动态代理** + - 在运行时动态生成代理类，而不需要手动编写代理类代码。 + - 动态代理通常基于 **Java 反射** 实现，最常用的两种动态代理技术是 **JDK 动态代理** 和 **CGLIB 动态代理**。 + + + +### 🎯 静态代理和动态代理的区别？ + +| **对比维度** | **静态代理** | **动态代理** | +| ------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | +| **代理类生成时间** | 代理类在 **编译时** 已经生成，由开发者手动编写代理类。 | 代理类在 **运行时** 动态生成，不需要手动编写代理类。 | +| **实现方式** | 手动创建一个实现目标类接口的代理类，通过代理类调用目标对象的方法。 | 使用反射机制（如 `Proxy` 或 `CGLIB`）在运行时生成代理类。 | +| **目标类要求** | 目标类必须实现接口。 | **JDK 动态代理**：目标类必须实现接口。 **CGLIB 动态代理**：目标类无需实现接口，但不能是 `final` 类。 | +| **灵活性** | 灵活性较低，每个目标类都需要一个对应的代理类，代码量较大。 | 灵活性高，一个代理类可以代理多个目标对象。 | +| **性能** | 性能略优于动态代理（因为无需反射机制）。 | 性能略低于静态代理（因依赖反射）。 但在 CGLIB 中，性能接近静态代理。 | +| **实现难度** | 实现简单，但代码量较多，维护麻烦。 | 实现复杂，但代码量少，易于维护。 | +| **扩展性** | 扩展性差，新增目标类时需要新增代理类。 | 扩展性好，新增目标类时无需新增代理类，只需修改动态代理逻辑。 | + + + +### 🎯 JDK 动态代理和 CGLIB 动态代理的区别？ + +| **对比维度** | **JDK 动态代理** | **CGLIB 动态代理** | +| ------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | +| **实现方式** | 基于 **Java 反射机制** 和接口实现，使用 `java.lang.reflect.Proxy`。 | 基于 **字节码增强** 技术，使用第三方库（如 `CGLIB` 或 `ASM`）。 | +| **代理对象要求** | 目标类必须实现 **接口**。 | 目标类 **无需实现接口**，可以代理普通类。 | +| **继承限制** | 无需继承，直接代理接口即可。 | 目标类不能是 `final` 类，代理的方法也不能是 `final` 方法（因为无法被重写）。 | +| **性能** | 性能略低（因反射调用方法效率较低）。 | 性能较高（直接生成字节码，调用方法接近普通方法调用）。 | +| **代码依赖** | 无需依赖外部库，属于 JDK 原生功能。 | 需要引入 `CGLIB` 或其他字节码增强库（如 ASM）。 | +| **应用场景** | 目标类实现了接口时适用。 | 目标类未实现接口或需要对具体类进行代理时适用。 | +| **实现复杂度** | 简单，使用 `Proxy` 类和 `InvocationHandler` 接口生成代理对象。 | 复杂，需要处理字节码增强逻辑（通常通过工具库封装）。 | +| **Spring AOP 支持** | 默认情况下，Spring AOP 使用 JDK 动态代理（如果目标类实现了接口）。 | Spring 会使用 CGLIB 动态代理（如果目标类未实现接口）。 | + +### 🎯 举一个用 Java 实现的装饰模式(decorator design pattern) ？它是作用于对象层次还是类层次？ + +"装饰器模式动态地给对象添加新功能，是**对象层次**的扩展： + +**装饰器模式特点**： + +- **对象组合优于继承**：通过包装而非继承扩展功能 +- **运行时增强**：可以动态添加或撤销装饰 +- **层层嵌套**：多个装饰器可以层层包装 +- **透明性**：装饰后的对象与原对象有相同接口 + +**Java IO流的装饰器设计**： + +```java +// 基础组件 +InputStream fileStream = new FileInputStream("file.txt"); + +// 装饰器层层包装 +InputStream bufferedStream = new BufferedInputStream(fileStream); // 添加缓冲功能 +InputStream dataStream = new DataInputStream(bufferedStream); // 添加数据类型读取 + +// 等价于：new DataInputStream(new BufferedInputStream(new FileInputStream("file.txt"))); +``` + +**作用层次**：装饰器模式作用于**对象层次**，而非类层次 + +- 不改变原有类的结构 +- 通过对象组合实现功能扩展 +- 比继承更加灵活，避免类爆炸 + +装饰器模式让功能扩展变得优雅而灵活。" + + + +--- + +## ⚡ 四、行为型模式（Object Interaction） + +> **核心思想**：关注对象之间的通信，负责对象间的有效沟通和职责委派，解决类或对象之间的交互问题。 + +### 🎯 在 Java 中，什么叫观察者设计模式（observer design pattern）？ + +观察者模式是基于对象的状态变化和观察者的通讯，以便他们作出相应的操作。简单的例子就是一个天气系统，当天气变化时必须在展示给公众的视图中进行反映。这个视图对象是一个主体，而不同的视图是观察者。 + + + +### 🎯 举一个用 Java 实现的装饰模式(decorator design pattern) ？它是作用于对象层次还是类层次？ + +"装饰器模式动态地给对象添加新功能，是**对象层次**的扩展： + +**装饰器模式特点**： + +- **对象组合优于继承**：通过包装而非继承扩展功能 +- **运行时增强**：可以动态添加或撤销装饰 +- **层层嵌套**：多个装饰器可以层层包装 +- **透明性**：装饰后的对象与原对象有相同接口 + +**Java IO流的装饰器设计**： + +```java +// 基础组件 +InputStream fileStream = new FileInputStream("file.txt"); + +// 装饰器层层包装 +InputStream bufferedStream = new BufferedInputStream(fileStream); // 添加缓冲功能 +InputStream dataStream = new DataInputStream(bufferedStream); // 添加数据类型读取 + +// 等价于：new DataInputStream(new BufferedInputStream(new FileInputStream("file.txt"))); +``` + +**作用层次**：装饰器模式作用于**对象层次**，而非类层次 + +- 不改变原有类的结构 +- 通过对象组合实现功能扩展 +- 比继承更加灵活，避免类爆炸 + +装饰器模式让功能扩展变得优雅而灵活。" + + + +--- + +## 🌟 五、实际应用（Practical Application） + +> **核心思想**：设计模式在实际项目中的应用案例和最佳实践，包括Spring框架、项目实战经验和开发建议。 + +### 🎯 Spring 当中用到了哪些设计模式？ + +"Spring框架大量运用设计模式，体现了优秀的架构设计： + +**创建型模式**： + +- **单例模式**：Spring Bean默认为单例，确保全局唯一性 +- **工厂模式**：BeanFactory是工厂的抽象，ApplicationContext是具体工厂 +- **建造者模式**：BeanDefinitionBuilder用于构建复杂的Bean定义 +- **原型模式**：prototype作用域的Bean通过克隆创建 + +**结构型模式**： + +- **代理模式**：AOP的核心实现，JDK动态代理和CGLIB代理 +- **装饰器模式**：BeanWrapper装饰Bean，添加属性访问能力 +- **适配器模式**：HandlerAdapter适配不同类型的Controller +- **外观模式**：ApplicationContext提供统一的访问接口 + +**行为型模式**： + +- **模板方法模式**：JdbcTemplate、RestTemplate等模板类 +- **策略模式**：Bean实例化策略选择（单例vs原型） +- **观察者模式**：ApplicationEvent和ApplicationListener +- **责任链模式**：Filter链、Interceptor链的处理方式 + +Spring的设计模式应用体现了框架的灵活性和可扩展性。" + + + +### 🎯 在工作中遇到过哪些设计模式，是如何应用的 + +"在实际项目开发中，我经常使用以下设计模式： + +**业务场景应用**： + +- **策略模式**：支付方式选择（支付宝、微信、银行卡支付策略） +- **工厂模式**：消息处理器创建（短信、邮件、推送消息工厂） +- **模板方法模式**：业务流程框架（订单处理、审批流程模板） +- **责任链模式**：权限验证链、参数校验链、业务处理链 +- **观察者模式**：系统事件通知（用户注册后发送邮件、积分等） + +**系统设计应用**： + +- **单例模式**：配置管理器、缓存管理器、日志记录器 +- **建造者模式**：复杂查询条件构建、报表生成器 +- **适配器模式**：第三方服务接口适配、遗留系统集成 +- **代理模式**：AOP切面编程、性能监控、事务管理 +- **外观模式**：微服务聚合接口、复杂子系统封装 + +设计模式让代码更易维护、扩展和理解，是高质量代码的基石。” + + + +### 🎯 简述一下你了解的 Java 设计模式（总结） + +"GoF 23种设计模式按用途分为三大类，每类解决不同层面的问题： + +**⭐ 常用设计模式重点掌握**： + +**创建型模式（5种）**： + +- ⭐⭐⭐ **单例模式**：确保类只有一个实例，全局访问点 +- ⭐⭐⭐ **工厂方法**：定义创建对象接口，子类决定实例化类型 +- ⭐⭐ **建造者模式**：分步构造复杂对象，相同过程创建不同表示 +- ⭐⭐ **原型模式**：通过克隆创建对象，避免复杂初始化 +- ⭐ **抽象工厂**：创建相关对象族，无需指定具体类 + +**结构型模式（7种）**： + +- ⭐⭐⭐ **代理模式**：为对象提供代理，控制访问并增强功能 +- ⭐⭐⭐ **装饰模式**：动态添加对象功能，灵活扩展行为 +- ⭐⭐ **适配器模式**：转换接口，让不兼容类协同工作 +- ⭐⭐ **外观模式**：提供统一接口，简化子系统访问 +- ⭐ **组合模式**：树形结构，统一处理单个和组合对象 +- 桥接模式：分离抽象和实现，两者可独立变化 +- 享元模式：共享对象减少内存使用 + +**行为型模式（11种）**： + +- ⭐⭐⭐ **观察者模式**：对象间一对多依赖，状态变化通知所有观察者 +- ⭐⭐⭐ **策略模式**：定义算法族，封装并可相互替换 +- ⭐⭐ **模板方法**：定义算法骨架，子类实现具体步骤 +- ⭐⭐ **责任链模式**：请求沿链传递，直到有对象处理 +- ⭐ **状态模式**：对象内部状态改变时改变行为 +- 命令模式：请求封装为对象，支持撤销和重做 +- 访问者模式：分离算法与数据结构，便于添加新操作 +- 中介者模式：集中处理对象间复杂交互 +- 备忘录模式：保存对象状态，支持恢复 +- 解释器模式：为语言定义文法表示和解释器 +- 迭代器模式：顺序访问聚合对象元素 + +--- + +## 🎓 总结与最佳实践 + +设计模式不是银弹，但是优秀代码的重要组成部分。选择合适的模式，避免过度设计，在实际应用中不断练习和总结，才能真正掌握设计模式的精髓。记住：**设计模式是为了解决问题而存在的，不要为了使用模式而使用模式**。 + +### 🎯 学习设计模式的建议 + +1. **理解问题背景**：每个模式都是为了解决特定问题而产生的 +2. **掌握核心结构**：理解模式的组成部分和它们之间的关系 +3. **练习代码实现**：通过编写代码来加深理解 +4. **结合实际项目**：在真实项目中寻找应用场景 +5. **避免过度设计**：不要为了使用模式而使用模式 + +### 🎯 常见的设计误区 + +1. **模式万能论**：认为设计模式可以解决所有问题 +2. **过度抽象**：为简单问题引入复杂的模式 +3. **死记硬背**：只记住结构而不理解应用场景 +4. **盲目套用**：不考虑具体情况强行使用某种模式 + +### 🎯 面试重点提醒 + +- 重点掌握单例、工厂、代理、观察者、策略等常用模式 +- 理解设计原则，特别是开闭原则和单一职责原则 +- 能够结合Spring框架说明模式的实际应用 +- 准备1-2个项目中实际使用设计模式的案例 + +**最终目标**：让设计模式成为你编写高质量代码的有力工具！ -在 Spring 框架中哪里使用到原型模式，并对源码进行分析 -介绍解释器设计模式是什么? diff --git a/docs/interview/Elasticsearch-FAQ.md b/docs/interview/Elasticsearch-FAQ.md new file mode 100644 index 0000000000..be389776fd --- /dev/null +++ b/docs/interview/Elasticsearch-FAQ.md @@ -0,0 +1,2808 @@ +--- +title: Elasticsearch 核心面试八股文 +date: 2024-05-31 +tags: + - Elasticsearch + - Interview +categories: Interview +--- + +![](https://img.starfish.ink/common/faq-banner.png) + +> Elasticsearch是基于Lucene的**分布式搜索与分析引擎**，也是面试官考察**搜索技术栈**的重中之重。从基础概念到集群架构，从查询优化到性能调优，每一个知识点都可能成为面试的关键。本文档将**最常考的ES知识点**整理成**标准话术**，让你在面试中对答如流！ + +--- + +## 🗺️ 知识导航 + +### 🏷️ 核心知识分类 + +1. **🔥 基础概念类**：索引、文档、分片、副本、倒排索引 +2. **📊 查询与索引原理**：查询DSL、分词器、Mapping、存储机制 +3. **🌐 集群与架构**：节点角色、选主机制、分片分配、故障转移 +4. **⚡ 性能优化类**：写入优化、查询优化、深分页、refresh机制 +5. **🔧 高级特性**：聚合分析、路由机制、批量操作、脚本查询 +6. **🚨 异常与故障处理**：脑裂问题、性能排查、内存优化、数据不均衡 +7. **💼 实战场景题**：日志分析、商品搜索、索引设计、数据清理 + + + +### 🔑 面试话术模板 + +| **问题类型** | **回答框架** | **关键要点** | **深入扩展** | +| ------------ | ----------------------------------- | ------------------ | ------------------ | +| **概念解释** | 定义→特点→应用场景→示例 | 准确定义，突出特点 | 底层原理，源码分析 | +| **对比分析** | 相同点→不同点→使用场景→选择建议 | 多维度对比 | 性能差异，实际应用 | +| **原理解析** | 背景→实现机制→执行流程→注意事项 | 图解流程 | Lucene层面，JVM调优 | +| **优化实践** | 问题现象→分析思路→解决方案→监控验证 | 实际案例 | 最佳实践，踩坑经验 | + +--- + +## 🔥 一、基础概念类（ES核心） + +> **核心思想**：Elasticsearch是基于Lucene的分布式搜索引擎，通过倒排索引实现快速全文检索，通过分片和副本实现水平扩展和高可用。 + +### 🎯 什么是 Elasticsearch？它的核心应用场景是什么？ + +1. **Elasticsearch 是什么？** + + Elasticsearch（ES）是基于 Lucene 的 **分布式搜索与分析引擎**。它通过 **倒排索引**（文本）与 **列式/树结构**（数值、地理、向量）实现 **高并发、低延迟** 的检索与聚合，支持 **水平扩展、自动分片与副本**、近实时(NRT)搜索。 + + 常用于： + + - 全文搜索（搜索引擎、日志检索） + - 实时数据分析（监控、报表） + - 大规模数据存储与快速查询（ELK：Elasticsearch + Logstash + Kibana） + + 一句话总结： Elasticsearch 就是“能存数据的搜索引擎”，特别适合**模糊查询、全文检索、实时分析**的场景。 + +2. **Elasticsearch 的特点** + + - **全文检索**：支持分词、倒排索引，能快速进行模糊搜索、关键词高亮等。 + + - **分布式**：天然支持水平扩展，数据分片存储，副本保证高可用。 + + - **近实时（NRT）**：数据写入后几乎能立即被搜索到（默认刷新间隔 1 秒）。 + + - **多种查询方式**：支持结构化查询（类似 SQL）、全文检索、聚合分析。 + + - **Schema 灵活**：字段可以动态添加，不需要固定表结构。 + + + +### 🎯 ES 和 MySQL 的区别？什么时候选用 ES？ + +| 对比维度 | Elasticsearch | 关系型数据库 (RDBMS) | +| ------------------ | --------------------------------------- | -------------------------------- | +| **数据模型** | 文档型（JSON 格式），索引 → 文档 → 字段 | 表格型（行、列、表） | +| **存储结构** | 倒排索引（适合搜索） | B+ 树（适合事务、范围查询） | +| **查询能力** | 全文检索、模糊匹配、聚合分析 | SQL 查询（精确匹配、复杂关联） | +| **扩展性** | 分布式架构，水平扩展方便 | 单机为主，分库分表扩展复杂 | +| **事务支持** | 不支持复杂事务（仅保证单文档原子性） | 支持 ACID 事务 | +| **场景适用** | 搜索引擎、日志分析、实时监控 | 金融交易、库存管理、强一致性系统 | +| **模式（Schema）** | 灵活 schema，可动态添加字段 | 严格 schema，结构需提前定义 | + +> Elasticsearch 是一个分布式的搜索和分析引擎，底层基于 Lucene，常用于全文检索和实时数据分析。 +> 和关系型数据库不同，ES 使用文档存储和倒排索引，更适合处理模糊搜索和大规模非结构化数据；而关系型数据库强调事务一致性和结构化查询，更适合业务系统的数据存储。 + + + +### 🎯 ES 中的索引（Index）、类型（Type，7.x 之后废弃）、文档（Document）、字段（Field）分别是什么？ + +- **Index**：类似“库”，由多个分片（Shard）组成。 + +- Type（类型，7.x 之后废弃）：在早期版本（6.x 及之前），一个索引可以有多个 **类型（Type）**，类似数据库中的 **表（Table）**。 + + - **作用**：不同类型的数据可以放在同一个索引里，每个类型有自己的 Mapping（字段定义）。 + - **为什么废弃**： + - ES 内部底层其实没有 “多表” 概念，所有 type 都是写在一个隐藏字段 `_type` 上。 + - 多个 type 共用一个倒排索引，容易导致 **字段冲突**（同名字段不同类型）。 + - 7.x 之后，一个索引只允许有一个 type（默认 `_doc`），8.x 完全移除。 + +- **Document**： + + **定义**：文档是 Elasticsearch 存储和检索的基本单位，相当于关系型数据库中的 **一行数据（Row）**。 + + **特点**： + + - 用 JSON 格式存储数据。 + - 每个文档都有一个 `_id`，用于唯一标识。 + - 文档属于某个索引。 + + ```json + { + "id": 1, + "name": "Alice", + "age": 25, + "city": "Beijing" + } + ``` + + 这是存放在 `users` 索引下的一个文档。 + +- **Field（字段）**: 字段就是文档中的一个 **键值对（Key-Value）**，相当于关系型数据库中的 **列（Column）**。 + + **特点**： + + - 每个字段可以有不同的数据类型（text、keyword、integer、date、boolean…）。 + + - 字段可以嵌套，比如： + + ```json + { + "user": { + "name": "Alice", + "age": 25 + } + } + ``` + + 这里 `user` 是一个对象字段，`user.name` 和 `user.age` 是子字段。 + + - **Mapping**：字段的数据类型和分词方式通过 Mapping 来定义。 + +- **Shard**：索引的水平切分单位，**Primary Shard** + **Replica Shard**；Replica 提供高可用与读扩展。 +- **NRT**：写入先进入内存并记录到 **translog**，刷新（refresh，默认 1s）后生成可搜索的 segment，故 **近实时**而非绝对实时。 + + + +### 🎯 倒排索引是什么？为什么 ES 查询快？ + +传统的我们的检索是通过文章，逐个遍历找到对应关键词的位置。**而倒排索引，是通过分词策略，形成了词和文章的映射关系表，这种词典+映射表即为倒排索引**。有了倒排索引，就能实现`o（1）时间复杂度` 的效率检索文章了，极大的提高了检索效率。![在这里插入图片描述](https://www3.nd.edu/~pbui/teaching/cse.30331.fa16/static/img/mapreduce-wordcount.png) + +学术的解答方式： + +> 倒排索引，相反于一篇文章包含了哪些词，它从词出发，记载了这个词在哪些文档中出现过，由两部分组成——词典和倒排表。 + +`加分项` ：倒排索引的底层实现是基于：FST（Finite State Transducer）数据结构。lucene从4+版本后开始大量使用的数据结构是FST。FST有两个优点： + +- 空间占用小。通过对词典中单词前缀和后缀的重复利用，压缩了存储空间； +- 查询速度快。O(len(str))的查询时间复杂度。 + +建索引时：文本经 **Analyzer（分词器：字符过滤→Tokenizer→Token 过滤）** 拆成 Term；建立 **Term→PostingList(DocID、位置、频率)** 的倒排表。查询时只需按 Term 直达候选 Doc，结合 **BM25** 等相关性模型打分，避免全表扫描。 + +- 词典（Term Dictionary）用 **FST** 压缩；PostingList 支持 **跳表**/块压缩以加速跳转与节省内存/磁盘。 +- **高效的过滤上下文**（bool filter）可走 **bitset** 缓存，进一步减少候选集。 + + + +### 🎯 什么是分片（Shard）和副本（Replica）？作用是什么？ + +> - **分片（Shard）** 是 Elasticsearch 的 **水平扩展机制**，把索引切分成多个小块，分布到不同节点，保证存储和计算能力能随着节点数扩展。 +> - **副本（Replica）** 是 Elasticsearch 的 **高可用和读扩展机制**，保证在节点宕机时数据不丢失，同时分担查询压力。 +> - 写入先到主分片，再复制到副本；查询可在主分片或副本执行。 + +分片是为了将数据分散到多个节点上，实现数据的分布式存储；副本用于提高系统的容错性和查询性能。 + +1. 分片（Shard） + +- **定义**： + - Elasticsearch 中的数据量可能非常大，单台机器无法存下，所以一个索引会被**切分成多个分片（Shard）**。 + - 每个分片本质上就是一个 **Lucene 索引（底层存储单元）**。 +- **类型**： + - **主分片（Primary Shard）**：存放原始数据，写入时必须先写入主分片。 + - **副本分片（Replica Shard）**：主分片的拷贝，用于 **容错 + 读请求负载均衡**。 +- **特点**： + - 分片在不同节点上分布，保证数据水平扩展。 + - 每个分片大小建议控制在 **10GB ~ 50GB** 之间，太大恢复和迁移慢，太小则分片数过多影响性能。 + - 索引在创建时可以设置分片数量（不可修改），副本数量可以动态调整。 + +------ + +2. 副本（Replica） + +- **定义**： + - Replica 是 Primary Shard 的完整拷贝。 + - 默认每个主分片有 **1 个副本**，可以通过 `number_of_replicas` 设置。 +- **作用**： + 1. **高可用**：如果某个节点宕机，副本可以提升为主分片，保证数据不丢。 + 2. **读负载均衡**：查询请求可以在主分片或副本分片上执行，从而分散查询压力。 +- **注意**： + - 写操作只会在 **主分片** 上执行，然后异步复制到副本。 + - 为了避免数据不一致，ES 保证**副本和主分片不在同一节点上**。 + +------ + +3. 写入流程 + + ① 客户端发送写请求到任意节点（协调节点）。 + + ② 协调节点根据 **路由算法**（默认使用 `_id` hash）找到目标主分片。 + + ③写入主分片成功后，主分片再把数据复制到对应的副本分片。 + + ④ 主分片和所有副本都确认后，写入成功。 + +------ + +4. 查询流程 + +​ ① 客户端发送查询请求到协调节点。 + +​ ② 协调节点把请求分发到主分片或副本分片。 + +​ ③ 各分片执行查询，返回结果到协调节点。 + +​ ④ 协调节点合并、排序后返回给客户端。 + +------ + +在创建索引时，可以指定分片数和副本数，示例： + +```json +PUT /my_index +{ + "settings": { + "number_of_shards": 5, + "number_of_replicas": 1 + } +} +``` + + + +## 📊 二、查询与索引原理（核心机制） + +> **核心思想**：理解ES的存储机制（倒排索引）、查询机制（Query DSL）、分词机制（Analyzer）和映射机制（Mapping）是掌握ES的关键。 + +- **查询方式**：[ES查询类型](#es-支持哪些查询方式) | [term vs match](#term-vs-matchmatch_phraseboolfilter-的区别与选型) +- **字段类型**：[text vs keyword](#text-与-keyword-有何区别如何两用) | [Mapping机制](#mapping映射是什么动态-mapping-和静态-mapping-有什么区别) +- **分词分析**：[分词器原理](#es-中的分词器analyzer是什么常见有哪些) | [全文搜索](#如何在-elasticsearch-中进行全文搜索) +- **存储检索**：[文档存储流程](#es-的文档是如何被存储和检索的) | [索引写入过程](#详细描述一下elasticsearch索引文档的过程) | [搜索过程](#详细描述一下elasticsearch搜索的过程) + +### 🎯 ES 支持哪些查询方式？ + +1. **Match 查询**：最常用的查询类型，全文搜索查询，会对输入的查询词进行分析（分词）。 + + ```json + { + "query": { + "match": { + "field_name": "search_value" + } + } + } + ``` + +2. **Term 查询**：精确查询，不进行分析。通常用于关键词或数字等不需要分词的字段。 + + ```json + { + "query": { + "term": { + "field_name": "exact_value" + } + } + } + ``` + +3. **Range 查询**：范围查询，用于查找某个范围内的数值、日期等。 + + ```json + { + "query": { + "range": { + "field_name": { + "gte": 10, + "lte": 20 + } + } + } + } + ``` + +4. **Bool 查询**：组合查询，支持 `must`（必须匹配）、`should`（应该匹配）、`must_not`（必须不匹配）等子查询的逻辑操作。 + + ```json + { + "query": { + "bool": { + "must": [ + { "match": { "field_name": "value1" } }, + { "range": { "date_field": { "gte": "2020-01-01" } } } + ] + } + } + } + ``` + +5. **Aggregations（聚合）**：用于数据统计和分析，如计数、求和、平均值、最大值、最小值等。 + + ```json + { + "aggs": { + "average_price": { + "avg": { + "field": "price" + } + } + } + } + ``` + + + +### 🎯 term vs match、match_phrase、bool、filter 的区别与选型？ + +- **term**：不分词，精确匹配（适合 `keyword`/ID/码值）。 +- **match**：分词后查询（适合 `text` 全文检索）。 +- **match_phrase**：按词序匹配短语，依赖 position 信息（性能较 match 差）。 +- **bool**：组合查询（must/should/must_not/filter）。其中 **filter 不参与打分**、可缓存，适合条件筛选（如时间、状态）。 + 选型：**结构化精确条件走 filter/term，全文用 match/match_phrase**。 + + + +### 🎯 text 与 keyword 有何区别？如何两用？ + +- 话术：text 用于分词检索；keyword 用于精确匹配/聚合/排序。两用用 multi-fields。 +- 关键要点：中文分词 ik_max_word/ik_smart；高基数字段用 keyword 并开启 doc_values。 +- 示例 + +```json +PUT /blog +{ + "mappings": { + "properties": { + "title": { "type": "text", "fields": { "keyword": { "type": "keyword" } } }, + "tags": { "type": "keyword" } + } + } +} +``` + +- 常见追问：为什么聚合要 keyword？避免分词与评分，走列式 doc_values。 + + + +### 🎯 如何在 Elasticsearch 中进行全文搜索？ + +使用 `match` 查询进行全文搜索。例如： + +```json +{ + "query": { + "match": { + "field_name": "search_text" + } + } +} +``` + + + +### 🎯 ES 中的分词器（Analyzer）是什么？常见有哪些？ + +> **Analyzer（分词器）** 是 ES 用来把文本转换成倒排索引里的 token 的组件。 +> +> 它由 **字符过滤器 → 分词器 → 词项过滤器** 三部分组成。 +> +> 常见分词器有 **standard、simple、whitespace、stop、keyword、pattern、language analyzer**，中文常用 **IK 分词器**。 +> +> **选择分词器的关键**：写入和查询要用一致的分词方式，否则可能导致查不到结果。 + +分词器是 Elasticsearch 在建立倒排索引时，用来把 **文本切分成一个个词项（Term）** 的工具。 + +它决定了： + +- 文本如何分词（tokenizer 负责） +- 分词结果如何处理（character filter 和 token filter 负责） + +- **作用**： + - **写入时**：文档进入 ES，字段值会经过分词器，拆成若干词项，存入倒排索引。 + - **查询时**：用户的搜索关键词也会经过同样的分词器处理，然后去倒排索引里匹配。 + +👉 分词器的一致性非常重要：写入和查询时要用相同或兼容的分词方式，否则会出现“查不到”的情况。 + +**分词器的组成（3 部分）** + +1. **Character Filters（字符过滤器）**： + 在分词前先对文本做预处理，例如去掉 HTML 标签、替换符号等。 + - 例子：把 `Hello` 转换为 `Hello`。 +2. **Tokenizer（分词器）**： + 核心组件，决定如何切分文本成 token。 + - 例子：`"Hello World"` → `["Hello", "World"]`。 +3. **Token Filters（词项过滤器）**： + 对切出来的词项进一步处理，比如大小写转换、停用词过滤、词干还原等。 + - 例子：把 `["Running", "Runs"]` → `["run", "run"]`。 + +------ + +**常见的内置分词器（Analyzer）** + +- Standard Analyzer（默认分词器）：ES 默认分词器，基于 Unicode Text Segmentation。 + - 按词法规则拆分单词、小写化、去掉标点符号。 + - `"The QUICK brown-foxes."` → `["the", "quick", "brown", "foxes"]` + +- Simple Analyzer：只按照非字母字符拆分。 + - 全部转小写、不会去掉停用词。 + - `"Hello, WORLD!"` → `["hello", "world"]` + +- Whitespace Analyzer：仅仅按照空格拆分，不做大小写转换。 + - `"Hello World Test"` → `["Hello", "World", "Test"]` + + + +### 🎯 Mapping（映射）是什么？动态 mapping 和静态 mapping 有什么区别？ + +> **Mapping 就是索引的字段定义，类似数据库的表结构。** +> +> **动态 mapping**：字段自动识别，方便但有风险，适合快速开发。 +> +> **静态 mapping**：字段手动定义，安全可控，适合生产环境。 +> +> 通常生产环境推荐 **静态 mapping** 或 **动态模板 + 部分静态 mapping** 来平衡灵活性与可控性。 + +在 Elasticsearch 中，**Mapping 就是索引中字段的结构定义（类似关系型数据库中的表结构和字段类型约束）**。 + +它决定了： + +- 每个字段的数据类型（如 `text`、`keyword`、`integer`、`date`）。 +- 字段是否可被索引（`index: true/false`）。 +- 使用的分词器（analyzer）。 +- 是否存储原始值（`store`）。 + +👉 简单说：**Mapping 定义了文档中字段是如何存储和索引的。** + +------ + +**常见字段类型** + +- **字符串类型**： + - `text`：会分词，适合全文检索。 + - `keyword`：不分词，适合精确匹配（ID、标签、邮箱等）。 +- **数字类型**：`integer`、`long`、`double`、`float` 等。 +- **布尔类型**：`boolean`。 +- **日期类型**：`date`。 +- **复杂类型**：`object`、`nested`（专门用于嵌套对象）。 + +------ + +**动态 Mapping**：Elasticsearch 支持 **动态映射**，即插入文档时，如果字段不存在，会自动推断字段类型并添加到 mapping。 + +- **优点**： + + - 使用方便，不需要提前设计表结构。 + - 适合快速开发、探索数据结构。 + +- **缺点**： + + - 推断类型可能不准确（比如 `"123"` 可能被识别为 `long`，但其实是字符串）。 + - 一旦字段类型被确定，就不能修改。 + - 容易出现 mapping 爆炸（比如 JSON 动态字段过多）。 + + ```json + { "age": 25, "name": "Tom", "is_active": true } + ``` + + ES 会自动生成 mapping： + + ```json + { + "properties": { + "age": { "type": "long" }, + "name": { "type": "text", "fields": { "keyword": { "type": "keyword" } } }, + "is_active": { "type": "boolean" } + } + } + ``` + +**静态 Mapping**：用户在创建索引时，**手动指定 mapping 结构**，字段类型和配置不会依赖 ES 的自动推断。 + +- **优点**： + + - 类型更精确，避免 ES 自动推断错误。 + - 可指定 analyzer、index、store 等配置，更灵活。 + - 避免 mapping 爆炸问题。 + +- **缺点**： + + - 需要提前设计数据结构。 + - 灵活性不如动态 mapping。 + + ```json + PUT users + { + "mappings": { + "properties": { + "age": { "type": "integer" }, + "name": { "type": "keyword" }, + "created_at": { "type": "date", "format": "yyyy-MM-dd HH:mm:ss" } + } + } + } + ``` + + + +### 🎯 Mapping 有何讲究？text vs keyword、multi-fields、normalizer？ + +- **text**：分词，适合全文检索，不适合聚合/排序（除非启用 `fielddata`，但极耗内存，不推荐）。 +- **keyword**：不分词，适合精确匹配、聚合、排序；可配 **normalizer**（小写化、去空格）做规范化。 +- **multi-fields**：一个字段既要搜索又要聚合：`title` 映射为 `text`，同时加 `title.keyword` 为 `keyword`。 +- 尽量 **静态明确 mapping**，用 **dynamic_templates** 控制自动映射，避免误判类型（如数值被映射成 text）。 + + + +### 🎯 ES 的文档是如何被存储和检索的？ + +> **存储**：文档以 JSON 形式写入，根据 `_id` 路由到分片 → 分词器对 `text` 字段建立倒排索引，同时存储原始文档 `_source`，数据落到 Lucene 的 **segment** 文件。 +> +> **检索**：查询请求分发到各分片 → 分词匹配倒排索引 → 计算相关度 → 各分片返回结果 → 协调节点合并排序 → 返回最终结果。 +> +> **核心结构**：倒排索引（全文检索）、正排索引（聚合/排序）、段文件（存储单元）。 + +在 Elasticsearch 中，文档是 JSON 格式的，存储过程大致如下： + +**写入流程** + +1. **客户端写入文档**（JSON 对象）。 + + ```json + { "id": 1, "title": "Elasticsearch is great", "tag": "search" } + ``` + +2. **路由到分片**： + + - 根据文档的 `_id`（或指定的 routing 值），通过 **hash 算法**决定存储在哪个主分片（Primary Shard）上。 + - 复制到副本分片（Replica Shard）。 + +3. **倒排索引（Inverted Index）构建**： + + - 对 `text` 类型字段（如 `title`）执行 **分词（Analyzer）** → `["elasticsearch", "is", "great"]`。 + - 将每个词条写入倒排索引（Term Dictionary + Posting List）。 + - 对 `keyword`、`date`、`integer` 等字段存储为单值，直接写入索引结构。 + +4. **存储原始文档**： + + - ES 默认会在 `_source` 字段里存储原始 JSON 文档，便于后续返回和 reindex。 + - 底层基于 **Lucene**，存储在 **段（segment）** 文件中。 + +------ + +**文档是如何检索的？** + +**查询流程** + +1. **客户端发起查询请求**：可以是全文检索（match）、精确匹配（term）、聚合（aggregation）等。 +2. **路由到所有相关分片**：协调节点（coordinating node）将查询广播到目标索引的所有分片（主分片或副本分片）。 +3. **分片内执行搜索** + - 对 `text` 字段，先对查询字符串分词。 + - 例如：搜索 `"great elasticsearch"` → 分词成 `["great", "elasticsearch"]`。 + - 在倒排索引里找到包含这些词的文档 ID（docID）。 + - 计算每个文档的相关度（TF-IDF / BM25）。 +4. **合并结果** + - 各分片返回匹配结果和得分。 + - 协调节点合并、排序，取 Top N 返回给客户端。 + +**聚合流程** + +- 如果是 **聚合查询**（比如统计 tag 的数量），ES 会先在每个分片执行部分聚合，再由协调节点做最终合并。 + +------ + +**底层数据结构** + +- **倒排索引（Inverted Index）** + - 类似一本字典，记录“词 → 出现在哪些文档”。 + - 比如 `great → [doc1, doc5, doc7]`。 +- **正排索引（Stored Fields / Doc Values）** + - 记录“文档 → 字段值”，主要用于排序、聚合。 +- **段（Segment）** + - Lucene 的最小存储单元，写入时只能追加，不可修改。 + - 删除/更新文档实际上是打标记（deleted），通过 **merge** 过程清理。 + + + +### 🎯 详细描述一下Elasticsearch索引文档的过程 + +`面试官` ：想了解ES的底层原理，不再只关注业务层面了。 + +`解答` ：这里的索引文档应该理解为文档写入ES，创建索引的过程。文档写入包含：单文档写入和批量bulk写入，这里只解释一下：单文档写入流程。 + +记住官方文档中的这个图。 + +![](https://img-blog.csdnimg.cn/20190119231620775.png) + +第一步：客户写集群某节点写入数据，发送请求。（如果没有指定路由/协调节点，请求的节点扮演`路由节点` 的角色。） + +第二步：节点1接受到请求后，使用文档_id来确定文档属于分片0。请求会被转到另外的节点，假定节点3。因此分片0的主分片分配到节点3上。 + +第三步：节点3在主分片上执行写操作，如果成功，则将请求并行转发到节点1和节点2的副本分片上，等待结果返回。所有的副本分片都报告成功，节点3将向协调节点（节点1）报告成功，节点1向请求客户端报告写入成功。 + +如果面试官再问：第二步中的文档获取分片的过程？回答：借助路由算法获取，路由算法就是根据路由和文档id计算目标的分片id的过程。 + +``` +shard = hash(_routing) % (num_of_primary_shards) +``` + + + +### 🎯 详细描述一下Elasticsearch搜索的过程？ + +搜索拆解为“query then fetch” 两个阶段。**query阶段的目的**：定位到位置，但不取。步骤拆解如下： + +1. 假设一个索引数据有5主+1副本 共10分片，一次请求会命中（主或者副本分片中）的一个。 +2. 每个分片在本地进行查询，结果返回到本地有序的优先队列中。 +3. 第 2 步骤的结果发送到协调节点，协调节点产生一个全局的排序列表。 + +**fetch阶段的目的**：取数据。路由节点获取所有文档，返回给客户端。 + + + +### 🎯 **如何使用 Elasticsearch 进行日志分析？** + +可以使用 Elasticsearch 的聚合功能进行日志数据的分析，比如按日期统计日志数量、按日志级别聚合等。 + + + +### 🎯 Elasticsearch 中的 `bulk` 操作是什么？ + +`bulk` 操作用于批量插入、更新、删除文档，能够提高操作效率，减少单独请求的开销。 + + + +### 🎯 **Elasticsearch 中的聚合（Aggregation）是什么？如何使用？** + +聚合是 Elasticsearch 中进行数据统计和分析的功能，常见的聚合类型有 `avg`（平均值）、`sum`（总和）、`terms`（按值分组）等。 + +------ + + + +## 🌐 三、集群与架构（高可用） + +> **核心思想**：ES通过主从节点、分片副本、选主机制实现分布式集群的高可用，理解这些机制对生产环境运维至关重要。 + +- **节点角色**：[Master vs Data节点](#es-的-master-节点和-data-节点的区别) | [集群工作原理](#elasticsearch-的集群如何工作) +- **选主机制**：[集群选主原理](#集群选主的原理是什么) | [高可用保证](#怎么保证-elasticsearch-的高可用) +- **分片管理**：[分片规划](#分片如何规划与常见监控) | [分片恢复](#如果一个分片丢失了es-是如何恢复的) +- **数据流转**：[写入流程](#写入数据时es-的写入流程是怎样的) | [查询流程](#查询数据时es-的查询流程是怎样的) | [查询优化](#查询优化与-filter-的使用) +- **深分页问题**：[深分页原理](#深分页怎么做scroll-与-search_after-如何选择) + +### 🎯 Elasticsearch 的集群如何工作？ + +Elasticsearch 的集群由多个节点组成，这些节点通过网络进行通信，共同负责存储和处理数据。集群中的每个节点可以充当不同的角色，例如主节点、数据节点和协调节点。 + + + +### 🎯 ES 的 Master 节点和 Data 节点的区别？ + +在Elasticsearch集群中，Master节点和Data节点承担着不同的角色和职责： + +**Master节点**： + +- 主要负责集群层面的管理操作 +- 维护集群状态(Cluster State)的权威副本 +- 负责索引的创建、删除等元数据操作 +- 处理节点加入/离开集群的协调工作 +- 不存储数据，也不参与数据的CRUD操作 +- 通过选举产生主Master节点(Leader)，其他Master节点作为候选节点 + +**Data节点**： + +- 负责数据的实际存储和检索 +- 处理所有文档级别的CRUD操作 +- 执行搜索、聚合等数据查询请求 +- 参与索引的分片(Shard)分配和复制 +- 不参与集群管理决策 + + + +### 🎯 集群选主的原理是什么？ + +Elasticsearch集群的选主过程基于分布式一致性算法实现，主要流程如下： + +1. **发现阶段**： + - 节点启动时通过`discovery.seed_hosts`(7.x+)或`discovery.zen.ping.unicast.hosts`(7.x前)发现其他节点 + - 使用Zen Discovery模块进行节点间通信 +2. **选举触发条件**： + - 集群启动时 + - 当前主节点失效时(心跳超时) + - 网络分区导致无法联系主节点时 +3. **选举过程**： + - 符合条件(master eligible)的节点参与选举 + - 基于Bully算法变种实现： a) 每个节点向其他节点发送投票请求 b) 节点比较ID(默认按节点ID字典序)，ID"较大"的节点胜出 c) 获得多数票(N/2+1)的节点成为主节点 +4. **集群状态发布**： + - 新主节点生成新版本集群状态 + - 将集群状态发布给所有节点 + - 节点确认接收后完成主节点切换 + + + +### 🎯 分片如何规划与常见监控？ + +> 分片的规划要结合 **数据量、节点数和查询场景** 来决定。一般建议单分片控制在 10GB~50GB 之间，避免分片过多或过小。 +> 比如一个 1TB 的索引，如果单分片 40GB，大概需要 25 个主分片，加上副本就是 50 个分片，再平均分配到 10 个节点上比较合适。 +> +> 在监控方面，主要关注 **集群健康状态**（green/yellow/red）、**分片分配是否均衡**、**查询和写入延迟**、**JVM 内存和 GC**、**磁盘利用率**，以及分片大小是否合理。 +> +> 实际项目里，我会结合 **_cat/shards**、**_cluster/health** 和监控平台（比如 X-Pack Monitoring、Prometheus + Grafana）来实时监控这些指标，避免分片不均衡或者节点过载。 + +**一、分片如何规划** + +**1. 分片的基本原则** + +- **主分片数 (primary shards)** 决定索引的最大扩展能力。 +- **副本数 (replicas)** 提供高可用和并发查询能力。 +- 分片一旦建好，主分片数不能改（除非 reindex）。 + +------ + +**2. 规划思路** + +- **单分片大小建议**：10GB～50GB（Lucene 层面太大太小都不合适）。 +- **总分片数建议**：不要超过 `20 * 节点数`，否则集群管理开销大。 +- **计算方法**： + - 预估索引大小 ÷ 单分片大小 ≈ 主分片数。 + - 结合节点数和副本数做分配。 + +**例子**： + +- 预计一个索引 1TB 数据，每个分片 40GB。 +- 主分片数 = 1000GB ÷ 40GB ≈ 25。 +- 如果副本数 = 1，总分片数 = 50。 +- 假如有 10 个节点，那每个节点平均分配 5 个分片，比较均衡。 + +------ + +**3. 行业常见做法** + +- **日志类数据**：按天/月分索引，每个索引设置合适的分片数。 +- **商品库/电商数据**：索引相对稳定，可以按数据量规划固定分片数。 +- **冷热分离**：新数据用快盘+较多副本，老数据用慢盘+减少副本。 + +------ + +**二、常见监控指标** + +**1. 集群健康（Cluster Health）** + +- `green`：所有主分片和副本都正常。 +- `yellow`：主分片正常，但副本缺失（集群仍可用，但没有高可用保障）。 +- `red`：主分片不可用，数据丢失风险大。 + +👉 监控工具：`_cluster/health` API。 + +------ + +**2. 分片状态** + +- **未分配分片（unassigned shards）**：常见原因是节点不够，或者磁盘满了。 +- **分片迁移**：大规模 rebalancing 时，集群性能会抖动。 + +👉 监控工具：`_cat/shards`。 + +------ + +**3. 性能相关指标** + +- **分片大小**：是否过大（>50GB）或过小（<1GB）。 +- **查询延迟**：大部分发生在分片层面。 +- **索引速率**：写入吞吐是否均衡。 +- **段合并 (merge) 开销**：写入过快时可能导致大量段合并，影响性能。 + +------ + +**4. 资源相关指标** + +- **CPU 使用率**（查询/写入开销）。 +- **JVM 内存（Heap）**：GC 频率、Old GC 是否频繁。 +- **磁盘使用率**：是否超过 watermark（85% 默认）。 +- **线程池**：写入/搜索线程池是否饱和。 + + + +### 🎯 如果一个分片丢失了，ES 是如何恢复的？ + +“分片丢失时，主节点先把该分片标记为未分配，然后按‘就近可用’恢复：如果还有一份副本在且是 in-sync，就立即提升为主分片；其余副本通过‘对等恢复’（peer recovery）从新主分片拷贝 segment，再按序号回放 translog 补齐增量，达到全量一致。若主副本都没了，索引会变红，只能从快照仓库做 snapshot restore。为降低抖动，ES默认会延迟分配一会儿等丢失节点回来（index.unassigned.node_left.delayed_timeout），并受并发/带宽节流控制（node_concurrent_recoveries、indices.recovery.max_bytes_per_sec）。跨可用区还会结合分配感知（allocation awareness）避免把副本放进同一机架/可用区，从而提升容灾。” + + + +### 🎯 写入数据时，ES 的写入流程是怎样的？（Primary Shard、Replica Shard 协调过程） + +“客户端请求先到协调节点，按 routing（hash(*routing) % 主分片数）路由到主分片。主分片执行写入（校验/建模、写内存索引缓冲与 translog，分配 seq_no/primary term），再同步转发给所有 in-sync 副本执行同样写入，副本返回 ACK 后整体成功返回。写入后并不会立刻可搜，等 refresh（默认≈1s）或显式 refresh 才可见；flush 会把 translog 刷盘并生成 commit，merge 后台合并小段。”* + + + +### 🎯 查询数据时，ES 的查询流程是怎样的？（协调节点、分片查询、结果合并） + +查询是‘散—归并—取’两阶段。请求先到协调节点，按索引与路由挑出要查的分片；Query 阶段并行把查询下发到每个分片（主或任一副本），分片本地算分/排序出局部 topN 返回文档ID+排序值；协调节点全局归并得到最终 topN；Fetch 阶段再按这些ID到各分片取 *source/高亮/inner hits，拼装后返回。聚合会在分片先算局部桶/度量，最后在协调节点做 reduce 汇总。* + + + +### 🎯 查询优化与 filter 的使用？ + +- 话术 + - 尽量用 bool.filter（可缓存不计分）；裁剪 _source；只取需要字段；避免大 wildcard/regexp。 +- 关键要点 + - docvalue_fields/stored_fields；pre_filter_shard；合理路由减少 fan-out。 +- 示例 + +```json +GET idx/_search +{ + "_source": ["id","title"], + "query": { "bool": { "filter": [ { "term": { "status": "OK" } } ] } } +} +``` + +- 常见追问 + - query vs filter？filter 不计分可缓存；query 计分排序。 + + + +### 🎯 深分页怎么做？scroll 与 search_after 如何选择？ + +> ES 默认分页用 from+size，但超过 1w 条会有深分页问题，性能开销很大。 +> 如果是 **全量遍历导出数据**，适合用 **scroll API**，它会基于一个快照滚动查询。 +> 如果是 **在线业务场景需要实时翻页**，推荐用 **search_after**，通过上一页的排序值取下一页，性能更好。 +> 两者选择时要看业务场景：**离线导出用 scroll，实时分页用 search_after**。 + +**一、为什么会有深分页问题？** + +- ES 默认分页用 `from + size`，比如： + + ``` + GET /products/_search + { + "from": 10000, + "size": 10, + "query": { "match_all": {} } + } + ``` + +- 问题： + + - ES 会把 `from + size` 范围内的所有结果取出，再丢掉前 `from` 条。 + - 如果 `from` 很大（比如 100w），就会消耗大量 CPU、内存，严重影响性能。 + +- ES 默认限制：`index.max_result_window = 10000`，超过就报错。 + +------ + +**二、解决方案** + +**1. Scroll API** + +- **原理**：保持一个快照（snapshot），游标式滚动查询，适合 **全量遍历**。 +- **特点**： + - 快照固定，数据不会因新增/删除而变化。 + - 每次返回一批，游标往后移动。 + - 常用于 **数据导出**、**批量处理**。 +- **缺点**： + - 不适合实时场景（数据更新不会体现在快照中）。 + - 占用内存资源，需要及时清理 scroll 上下文。 + +**例子**： + +``` +# 初始化 scroll +GET /products/_search?scroll=1m +{ + "size": 1000, + "query": { "match_all": {} } +} + +# 后续使用 _scroll_id 拉取下一批 +GET /_search/scroll +{ + "scroll": "1m", + "scroll_id": "DnF1ZXJ5VGhlbkZldGNoBQAAAAAAAa==" +} +``` + +------ + +**2. search_after** + +- **原理**：基于上一页的“排序值”来取下一页。 +- **特点**： + - 不需要维护快照，性能比 scroll 更轻量。 + - 适合 **实时分页查询**，比如电商商品翻页。 + - 必须有 **唯一的排序字段**（通常用时间戳 + _id 作为 tie-breaker）。 +- **缺点**： + - 只能往后翻页，不能直接跳到第 N 页。 + +**例子**： + +``` +GET /products/_search +{ + "size": 10, + "query": { "match_all": {} }, + "sort": [ + { "price": "asc" }, + { "_id": "asc" } + ], + "search_after": [100, "doc123"] +} +``` + +这里 `100, "doc123"` 就是上一页最后一条文档的排序值。 + +------ + +**三、如何选择？** + +| 方式 | 适用场景 | 优点 | 缺点 | +| ------------ | -------------------- | -------------- | ------------------------ | +| from+size | 小数据分页（<1w 条） | 简单 | 深分页性能差 | +| scroll | 大规模导出、批处理 | 能遍历全量数据 | 不实时，占用内存 | +| search_after | 实时业务分页 | 高效、轻量 | 只能往后翻，不能随机跳页 | + +------ + + + +## ⚡ 四、性能优化类（高频重点） + +> **核心思想**：性能优化是ES面试的重中之重，涵盖写入优化、查询优化、深分页、缓存策略等多个方面，体现真实生产环境的技术水平。 + +- **写入优化**：[写入性能优化](#es-写入性能优化有哪些方式) | [批量操作优化](#写入为何近实时如何提吞吐) +- **查询优化**：[查询性能优化](#es-查询性能优化有哪些方式) | [深分页优化](#怎么优化-elasticsearch-的查询性能) +- **索引设计**：[日志数据设计](#es-如何设计索引来存储日志数据) | [索引数量问题](#索引数量过多会有什么问题) | [文档大小影响](#大量小文档-vs-少量大文档对性能有何影响) +- **核心机制**：[refresh/flush/merge](#refreshflushmerge-的区别它们各自解决什么问题) | [近实时原理](#es-为什么是近实时nrt搜索) | [部署优化](#elasticsearch在部署时对linux的设置有哪些优化方法) + +### 🎯 ES 写入性能优化有哪些方式？ + +“写入想快，先做对建模，再用批量、调‘刷新/副本/合并’，控制分片与路由，避免昂贵的脚本/管道。导入期关刷新和副本、走Bulk大批次并行；导入后恢复正常并用ILM做滚动与冷热分层，监控拒绝与合并就基本稳了。” + +**可落地清单（按优先级）** + +- 建模优化 + + - 字段类型精准：全文检索用 text，精确/聚合用 keyword（必要时 multi-fields）。 + + - 关闭不必要特性：对 keyword 关闭 norms；裁剪 *source（只保留必要字段）；避免 dynamic=true 导致字段爆炸。* + + - 高基数ID/标签直接 keyword；嵌套谨慎（nested 写成本高）。 + +- 批量写入 + + - Bulk 单批 5–15MB、并发 2–4 通道（BulkProcessor）；失败指数退避重试。 + + - 客户端做预处理/校验，少用 ingest pipeline 脚本（painless）做复杂逻辑。 + +- 刷新/副本/Translog（导入期与在线期分治） + + - 导入期：index.refresh_interval=-1、index.number_of_replicas=0、大 Bulk；导入完恢复如 1s 和 >=1 副本。 + + - Translog：默认 request 最稳；若追求吞吐可评估 index.translog.durability=async（有丢失风险）。 + + - wait_for_active_shards 设默认/较低，避免写阻塞。 + +- 合并与段（Segments） + + - 让合并在后台自然进行；仅对只读冷索引再考虑 forcemerge（小心耗时/IO）。 + + - 关注 index.merge.policy.* 与合并速率，避免小段过多。 + +- 分片与路由 + + - 分片数合理：单分片 10–50GB 参考；过多分片会拖慢写入与合并。 + + - 多租户/大业务用 _routing 降低 fan-out；大体量用 rollover + 别名切流。 + +- 资源与线程 + + - SSD/NVMe；堆 ≤ 32GB（留 Page Cache）；indices.memory.index_buffer_size 适当增大（如 10%–20%）。 + + - 关注 threadpool.write/bulk 队列与拒绝（rejected）；indices.recovery.max_bytes_per_sec 不要太低。 + +- 压缩与编解码 + - 写密集索引保持默认 codec；只读冷索引可用 index.codec=best_compression 降存储。 + +- OS/部署 + + - 关闭 swap、vm.max_map_count 足够、XFS/EXT4，磁盘与网卡不中断限速。 + + - 热温冷节点（ILM）+ 快照到对象存储，降低热区压力。 + +- 监控与压测 + + - 盯：bulk 吞吐/延迟、threadpool rejections、segments/merges、GC、refresh 时间、indexing_pressure。 + + - 预估写入峰值，压测后定批量与并发。 + + + +### 🎯 ES 查询性能优化有哪些方式？ + +> ES 查询优化，我的整体思路是：**先看索引结构是否合理，再优化查询语句本身，最后配合缓存和系统调优。** +> +> 在索引层面，重点是 **mapping 精准、分片数量合适、路由设计合理**； +> 在查询层面，**能用 filter 就不用 query**，**控制返回字段**，**分页避免 from+size 深度翻页**，多用 `search_after`； +> 高并发下要用 **缓存** 和 **预聚合**，避免复杂脚本和大范围扫描。 +> +> 最后，还需要结合 **慢查询监控、profile 分析** 去定位瓶颈。整体遵循「**索引设计优先 > 查询语句优化 > 缓存和系统层面**」这个顺序，就能保证查询性能。 + +1. **索引设计优化** + +- **Mapping优化** + - 字段类型精准：全文检索 `text + keyword`，精确查找只用 `keyword` + - 高基数字段避免聚合，可 `doc_values=false` 节省空间 + - 不需要评分的字段用 `keyword`，关闭 `norms` +- **分片设计** + - 单分片 **10~50GB**，分片过多会导致协调开销大 + - 多租户/大客户数据用 `_routing`，减少查询 fan-out +- **索引结构** + - 时间序列数据用 **ILM rollover**，只查必要时间段 + - **冷热分离**：历史数据放 warm/cold 节点 + +------ + +2. **查询语句优化** + +- **Filter vs Query** + + - 精确匹配用 **filter**（有缓存、无评分）：`term`、`range`、`bool.filter` + - 需要相关性评分才用 **query**：`match`、`multi_match` + + ``` + { + "bool": { + "filter": [{"term": {"status": "active"}}], + "must": [{"match": {"title": "elasticsearch"}}] + } + } + ``` + +- **返回字段控制** + + - 用 `_source_includes` / `_source_excludes` 只返回需要字段 + - 大文档场景用 `stored_fields` 替代 `_source` + - 统计类查询 `size=0`，避免返回 hits + +------ + +3. **分页优化** + +- 避免 `from + size` 深度分页（默认最大 `10000`） +- 大数据分页用 `search_after + sort`（推荐） +- 遍历历史数据用 `scroll`（7.x 之后推荐 `search_after`） + +------ + +4. **聚合性能优化** + +- 高基数聚合用 **composite aggregation**，避免 terms 爆炸 +- 聚合前先 **filter** 限制范围 +- 优先使用数值聚合，少用字符串 + +------ + +5. **缓存策略** + +- **查询缓存** + - filter 查询会自动缓存 + - 配置 `indices.queries.cache.size`（默认 10% 堆内存） +- **请求缓存** + - 完全相同请求可用 request cache，适合 dashboard 场景 + - `size=0` 的聚合查询容易命中缓存 + +------ + +6. **高级优化** + +- **Routing** + - 多租户场景用 `_routing`，减少全分片扫描 + - 大客户数据独立 routing key +- **字段优化** + - 不需要的字段 `enabled=false` + - 慎用 `nested`，查询成本高 +- **脚本优化** + - 避免 `script_score` 作用于大数据集 + - 用 **painless** 预编译脚本 + +------ + +7. **系统层面优化** + +- **硬件**：SSD、充足内存做 Page Cache，堆内存 ≤ 32GB +- **JVM**：推荐 G1GC，`swappiness=1`，避免 swap +- **监控**：慢查询日志、ES profile 分析，持续调优 + +------ + +**总结** + +- **查询前**：索引结构合理、mapping 精准、分片适量 +- **查询中**：filter 代替 query、控制返回、search_after 分页 +- **查询后**：监控慢查询、profile 分析、缓存命中率 + + + +### 🎯 ES 如何设计索引来存储**日志数据**？ + +日志数据典型的时间序列+大批量写入，核心是按时间滚动索引、冷热分离。用ILM自动rollover（按大小/时间），mapping设计timestamp+level+message结构，写入期关刷新提升吞吐，查询期用索引模板匹配时间范围。冷数据压缩+迁移到warm/cold节点，最老数据删除或快照到对象存储。 + + + +### 🎯 索引数量过多会有什么问题？ + +"索引数量过多主要三个问题：集群元数据爆炸拖慢Master节点，每个索引的分片增加协调开销，大量小索引浪费资源且查询低效。解决方案是合并相关索引、用别名+路由、ILM自动rollover控制数量，核心原则是'宁要几个大索引，不要成千上万小索引'。” + + + +### 🎯 大量小文档 vs. 少量大文档，对性能有何影响？ + +大量小文档会增加索引开销和merge压力，但查询精准；少量大文档减少索引开销但内存占用大、网络传输慢。选择策略看场景：高频精确查询用小文档+合理批量写入，大批量分析用大文档+*source裁剪。核心是平衡索引效率和查询需求，一般单文档1KB-1MB比较合适。* + + + +### 🎯 refresh、flush、merge 的区别？它们各自解决什么问题？ + +> "refresh让新写入数据可搜索，flush确保数据持久化到磁盘，merge合并小segment提升查询效率。refresh控制可见性（默认1秒），flush控制安全性（有translog保护），merge控制性能（后台自动）。实时性要求高调refresh频率，安全性要求高调flush策略，查询慢了关注merge状态。" + +**Refresh - 数据可见性** + +- 作用：内存数据 → 可搜索状态（仍在内存） + +- 默认：每1秒自动执行 + +- 调优： + + - 高写入场景：调到30s或关闭(-1) + + - 实时搜索：调到100ms或用?refresh=wait_for + + - 导入期关闭，完成后恢复 + +**Flush - 数据持久化** + +- 作用：内存segment + translog → 磁盘持久化 + +- 触发：translog达到512MB或30分钟 + +- 安全级别： + + - request：每次写入立即fsync（安全） + + - async：5秒间隔fsync（性能好） + +**Merge - 性能优化** + +- 作用：多个小segment → 大segment，物理删除已删文档 + +- 策略：后台自动执行，一般不需要手动干预 + +- 手动：只读索引可_forcemerge合并到1个segment + +**数据流转过程** + +``` +写入 → Memory Buffer + Translog +↓ Refresh（1s） +可搜索的Segment（内存） +↓ Flush（512MB/30min） +持久化Segment（磁盘） +↓ Merge（后台） +优化的大Segment +``` + +**场景配置** + +```json +// 高写入场景 +{"refresh_interval": "-1", "flush_threshold": "1gb"} + +// 实时搜索 +{"refresh_interval": "100ms", "translog.durability": "request"} + +// 查询优化 +{"refresh_interval": "30s", 完成后"_forcemerge"} +``` + +**监控要点** + +- Refresh：_cat/indices?h=pri.refresh.total_time + +- Flush：_cat/indices?h=pri.flush.total_time + +- Merge：_cat/segments查看segment数量和大小 + +**一句话记忆** + +Refresh控制何时可搜（默认1s），Flush控制何时安全（translog满了），Merge控制查询快慢（自动合并），根据读写场景调整三个参数的频率即可。 + + + +### 🎯 ES 为什么是“近实时（NRT）”搜索？ + +> Elasticsearch 是 **近实时搜索（NRT）**，因为： +> +> 1. 文档写入先进入内存 buffer，并记录在 translog； +> 2. 只有在 Lucene 执行 **refresh**（默认 1s 一次）后，内存数据才写入新的 segment 并对搜索可见； +> 3. 因此，写入后到能被检索之间存在 **约 1 秒的延迟**。 + +1. 什么是 NRT（Near Real-Time，近实时） + + Elasticsearch 不是严格的实时搜索，而是 **近实时搜索**： + + - 当你写入一个文档后，**不会立刻就能被搜索到**。 + + - 默认情况下，大约 **1 秒延迟** 后才可被检索。 + +------ + +2. 原因：Lucene 的段（Segment）机制 + + Elasticsearch 底层用 **Lucene** 来存储和检索数据。Lucene 的存储单元是 **segment** 文件，而 segment 是**只读不可修改**的，写入流程如下： + + 1. **写入内存缓冲区（Indexing Buffer）** + - 新文档先写入 ES 的内存 buffer。 + - 同时写一份到 **事务日志（translog）**，用于故障恢复。 + 2. **refresh 操作** + - Lucene 会定期（默认 1 秒）把内存 buffer 中的数据写入一个新的 segment 文件，并生成倒排索引，开放给搜索使用。 + - 只有在 refresh 之后，文档才会被检索到。 + 3. **flush 操作** + - 将内存数据 + translog 持久化到磁盘，生成新的 segment，并清空 translog。 + - flush 不是每秒都发生，而是按条件触发（如 translog 太大）。 + +------ + +3. NRT 的关键点 + + - **写入后马上可获取（get）**： + ES 写入文档后，可以立即通过 `_id` 来 **get**，因为它直接从内存和 translog 里拿数据。 + + - **写入后延迟可搜索**： + 需要等到下一次 **refresh**，文档才会出现在倒排索引里，从而能被搜索到。 + +​ 所以 ES 才被称为 **近实时**搜索系统（NRT），而不是严格实时（RT）。 + +------ + +4. 参数与调优 + +- `index.refresh_interval` + + - 默认 `1s`，表示每秒 refresh 一次。 + - 可以调大（如 `30s`），提高写入性能，降低搜索实时性。 + - 可以设为 `-1`，关闭自动 refresh（写入吞吐最大，但搜索不到新文档，直到手动 refresh）。 + +- 手动执行： + + ``` + POST /my_index/_refresh + ``` + +------ + + + +### 🎯 写入为何“近实时”？如何提吞吐？ + +- 话术 + - 写入→内存 buffer；refresh 暴露新 segment 可搜（默认 1s）；flush 持久化 translog+commit；merge 后台合并。批量写用 bulk，写多时临时调大 refresh_interval。 +- 关键要点 + - bulk 5–15MB/批、并发通道适中；写前 replicas=0、refresh=-1，写后恢复。 +- 示例 + +```properties +index.refresh_interval=1s # 批量导入临时 -1 +``` + +- 常见追问 + - 为什么不要频繁 refresh？小段太多影响检索与合并。 + + + +### 🎯 **怎么优化** **Elasticsearch** 的查询性能？ + +**优化分页查询** + +在 Elasticsearch 里面，也有两种可行的优化手段。 + +1. Scroll 和 Scroll Scan：这种方式适合一次性查询大量的数据，比如说导出数据之类的场景。这种用法更加接近你在别的语言或者中间件里面接触到的游标的概念。 + +2. Search After：也就是翻页，你在查询的时候需要在当次查询里面带上上一次查询中返回的 search_after 字段。 + +**增大刷新间隔** + +- 可以把 index.refresh_interval 调大一些 + +**优化不必要字段** + +**冷热分离** + +它的基本思路是同一个业务里面数据也有冷热之分。对于冷数据来说，可以考虑使用运行在廉价服务器上的 Elasticsearch 来存储；而对 + +于热数据来说，就可以使用运行在昂贵的高性能服务器上的 Elasticsearch。 + + + +### 🎯 Elasticsearch 索引数据多了怎么办，如何调优，部署 + +`面试官` ：想了解大数据量的运维能力。 + +`解答` ：索引数据的规划，应在前期做好规划，正所谓“设计先行，编码在后”，这样才能有效的避免突如其来的数据激增导致集群处理能力不足引发的线上客户检索或者其他业务受到影响。如何调优，正如问题1所说，这里细化一下： + +#### 3.1 动态索引层面 + +基于`模板+时间+rollover api滚动` 创建索引，举例：设计阶段定义：blog索引的模板格式为：blog_index_时间戳的形式，每天递增数据。 + +这样做的好处：不至于数据量激增导致单个索引数据量非常大，接近于上线2的32次幂-1，索引存储达到了TB+甚至更大。 + +一旦单个索引很大，存储等各种风险也随之而来，所以要提前考虑+及早避免。 + +#### 3.2 存储层面 + +`冷热数据分离存储` ，热数据（比如最近3天或者一周的数据），其余为冷数据。对于冷数据不会再写入新数据，可以考虑定期force_merge加shrink压缩操作，节省存储空间和检索效率。 + +#### 3.3 部署层面 + +一旦之前没有规划，这里就属于应急策略。结合ES自身的支持动态扩展的特点，动态新增机器的方式可以缓解集群压力，注意：如果之前主节点等`规划合理` ，不需要重启集群也能完成动态新增的。 + + + +### 🎯 怎么保证 Elasticsearch 的高可用？ + +Elasticsearch 的节点可以分成很多种角色，并且一个节点可以扮演多种角色。这里我列举几种主要的。 + +- 候选主节点（Master-eligible Node)：可以被选举为主节点的节点。主节点主要负责集群本身的管理，比如说创建索引。类似的还有仅投票节点（Voting-only Node），这类节点只参与主从选举，但是自身并不会被选举为主节点。 + +- 协调节点（Coordinating Node）：协调节点负责协调请求的处理过程。一个查询请求会被发送到协调节点上，协调节点确定数据节点，然后让数据节点执行查询，最后协调节点合并数据节点返回的结果集。大多数节点都会兼任这个角色。 + +- 数据节点（Data Node）：存储数据的节点。当协调节点发来查询请求的时候，也会执行查询并且把结果返回给协调节点。类似的还有热数据节点（Hot Data Node）、暖数据节点（Warm Data Node）、冷数据节点（Cold Data Node），你从名字就可以看出来，它们只是用于存储不同热度的数据。 + +**写入数据** + +1. 文档首先被写入到 Buffer 里面，这个是 Elasticsearch 自己的 Buffer。 +2. 定时刷新到 Page Cache 里面。这个过程叫做 refresh，默认是 1 秒钟执行一次。 +3. 刷新到磁盘中，这时候还会同步记录一个 Commit Point。 + +![ES 是如何写入一条数据的？_es写入数据-CSDN博客](https://i-blog.csdnimg.cn/blog_migrate/065a9dbf0202218c0bf71a2528607612.png) + +在写入到 Page Cache 之后会产生很多段（Segment），一个段里面包含了多个文档。文档只有写到了这里之后才可以被搜索到，因此从支持搜索的角度来说，Elasticsearch 是近实时的。 + +不断写入会不断产生段，而每一个段都要消耗 CPU、内存和文件句柄，所以需要考虑合并。 + +但是你也注意到了，这些段本身还在支持搜索，因此在合并段的时候，不能对已有的查询产生影响。 + +所以又有了合并段的过程，大概是 + +1. 已有的段不动。 +2. 创建一个新的段，把已有段的数据写过去，标记为删除的文档就不会写到段里面。 +3. 告知查询使用新的段。 +4. 等使用老的段的查询都结束了，直接删掉老的段。 + +那么查询怎么知道应该使用合并段了呢？这都依赖于一个统一的机制，就是 Commit Point。你可以理解成，它里面记录了哪些段是可用的。所以当合并段之后，产生一个新的 CommitPoint，里面有合并后的段，但是没有被合并的段，就相当于告知了查询使用新的段。 + +**Translog** + +实际上，Elasticsearch 在写入的时候，还要写入一个东西，也就是 Translog。直观来说，你可以把这个看成是 MySQL 里和 redo log 差不多的东西。也就是如果宕机了，Elasticsearch可以用 Translog 来恢复数据。 + +> MySQL 写入的时候，其实只是修改了内存里的值，然后记录了日志，也就是 binlog、redo log 和 undo log。 +> +> Elasticsearch写入的时候，也是写到了 Buffer 里，然后记录了 Translog。不同的是，Translog 是固定间隔刷新到磁盘上的，默认情况下是 5 秒。 + + + +Elasticsearch 高可用的核心是分片，并且每个分片都有主从之分。也就是说，万一主分片崩溃了，还可以使用从分片，从而保证了最基本的可用性。 + +而且 Elasticsearch 在写入数据的过程中，为了保证高性能，都是写到自己的 Buffer 里面，后面再刷新到磁盘上。所以为了降低数据丢失的风险，Elasticsearch 还额外写了一个 Translog，它就类似于 MySQL 里的 redo log。后面 Elasticsearch 崩溃之后，可以利用 Translog 来恢复数据。 + + + +### 🎯 Elasticsearch在部署时，对Linux的设置有哪些优化方法 + +`面试官` ：想了解对ES集群的运维能力。 + +`解答` ： + +- 关闭缓存swap; +- 堆内存设置为：Min（节点内存/2, 32GB）; +- 设置最大文件句柄数； +- 线程池+队列大小根据业务需要做调整； +- 磁盘存储raid方式——存储有条件使用RAID10，增加单节点性能以及避免单节点存储故障。 + + + +### 🎯 lucence内部结构是什么？ + +`面试官` ：想了解你的知识面的广度和深度。 + +`解答` ： + +![](https://img-blog.csdnimg.cn/20190119231637780.png) + +Lucene是有索引和搜索的两个过程，包含索引创建，索引，搜索三个要点。可以基于这个脉络展开一些。 + + + +## 🔧 五、高级特性（进阶） + +> **核心思想**：掌握ES的高级特性如聚合分析、路由机制、批量操作、脚本查询等，体现对ES深层原理的理解。 + +- **路由机制**：[Routing原理](#什么是-routing如何影响查询和写入) | [高可用实现](#es-如何实现高可用) +- **批量操作**：[Bulk API](#es-的批量操作bulk-api怎么用为什么比逐条写入快) | [分页机制](#es-如何实现分页深分页有什么问题如何优化) +- **聚合分析**：[聚合类型](#es-的聚合aggregation是什么常见聚合类型) | [数据关联](#什么是-parent-child-关系和-nested-类型) +- **脚本查询**：[Painless脚本](#es-的脚本查询painless-script是什么) + +### 🎯 什么是 Routing？如何影响查询和写入？ + +> **Routing 定义**：Routing 是文档到分片的路由机制，决定文档写在哪个分片上。 +> +> **写入影响**：写入时通过 `hash(routing) % 分片数` 决定目标分片，默认 routing 值是 `_id`，也可以自定义。 +> +> **查询影响**：如果不指定 routing，需要在所有分片上 scatter & gather 查询；如果指定 routing，可以直接路由到目标分片，性能更高。 +> +> **应用场景**：多租户、用户数据、订单等需要按逻辑分组的场景。 + +Routing（路由）是 **决定文档应该存储到哪个分片（Shard）** 的机制。 + +- 在 ES 中，一个索引会被分成多个主分片（Primary Shard）。 +- 当你写入一个文档时，ES 必须决定它落在哪个主分片。 +- Routing 的作用就是做这个映射。 + +默认情况下： + +``` +shard = hash(_routing) % number_of_primary_shards +``` + +其中： + +- `_routing` 默认值是文档的 `_id`； +- 也可以显式指定 `_routing` 字段，影响文档的分片归属。 + +------ + +**Routing 在 写入 时的影响** + +1. **决定分片位置** + + - 当写入文档时，ES 根据 `_routing` 值（默认 `_id`）计算出目标分片。 + - 文档只会被写入到一个主分片（以及它的副本分片）。 + +2. **自定义 routing** + + - 可以在写入时指定一个 routing 值，比如用户 ID。 + - 所有相同 routing 值的文档会落到同一个分片里。 + - 好处：查询时避免跨分片 scatter/gather，性能更高。 + + ```json + PUT /orders/_doc/1?routing=user123 + { + "order_id": 1, + "user": "user123", + "amount": 100 + } + ``` + +​ 这条订单文档会根据 `"user123"` 的 hash 值，落到某个固定分片上。 + +------ + +**Routing 在 查询 时的影响** + +1. **默认情况** + + - 如果不指定 routing，查询会广播到索引的所有分片，再合并结果。 + - 这种 **scatter & gather** 查询在分片很多时性能会下降。 + +2. **指定 routing** + + - 如果你知道文档的 routing 值，可以在查询时加上 `routing` 参数。 + - ES 就只会去那个分片查，大大减少分片间的查询开销。 + + ```json + GET /orders/_search?routing=user123 + { + "query": { + "term": { "user": "user123" } + } + } + ``` + +​ 这里的查询只会在存放 `user123` 文档的分片上执行，而不是全索引搜索。 + +------ + +**注意点** + +- **索引时 routing 值必须和查询时一致**： + 如果写入时用了 routing，查询时也必须带上相同的 routing，否则查不到。 +- **分片数固定**： + routing 是基于分片数取模计算的，因此分片数一旦确定就不能更改，否则路由会失效。 +- **适合场景**： + - 电商订单：用用户 ID 做 routing，可以把一个用户的所有订单放在同一个分片里。 + - 多租户系统：用租户 ID 做 routing，把不同租户数据隔离。 + + + +### 🎯 ES 如何实现高可用？ + +> - **ES 高可用依赖分片和副本机制**，主分片负责写入，副本分片保证数据冗余和读扩展。 +> - **Master 节点**负责集群管理，通过选主机制保证稳定运行。 +> - **自动故障转移**：主分片宕机时，副本分片自动提升为主分片。 +> - **生产实践**：多节点部署（至少 3 个 Master），每个主分片至少 1 个副本，配合跨集群复制/快照实现容灾。 + +Elasticsearch（ES）的高可用性主要体现在 **集群层面**，通过分片、副本、副本自动切换、集群选主等机制来保证在节点故障时系统仍能提供服务。 + +**1、核心机制** + +**分片（Shard）** + +- 每个索引的数据被拆分为多个 **主分片（Primary Shard）**。 +- 分片分布在不同节点上，避免单节点成为瓶颈。 + +**副本（Replica）** + +- 每个主分片可以配置多个副本分片。 +- **作用**： + - 容错：当主分片所在节点宕机时，副本可以升级为主分片； + - 读扩展：查询可以在副本分片上执行，提高并发查询能力。 + +**自动故障转移** + +- 如果某个节点宕机导致主分片不可用，ES 会自动从副本中选一个提升为主分片。 +- 确保索引始终保持读写可用。 + +**Master 节点** + +- **Master 节点**负责维护集群状态（分片分配、节点管理）。 +- 通过 **Zen Discovery** 或基于 **Raft**（8.0 之后引入）选主机制，保证集群有一个健康的 Master。 +- 建议部署 **3 个或以上 Master 节点**，避免脑裂。 + +--- + +**2、高可用的实现手段** + +1. **多节点部署** + - 至少 3 个节点：1 个 Master + 2 个 Data。 + - 生产环境推荐：3 个 Master（仅做集群管理）+ 多个 Data 节点（存储和查询）。 + +2. **副本机制** + - 每个主分片至少配置 1 个副本，保证节点故障时数据不丢失。 + - 主分片和副本分片不会分配在同一节点。 + +3. **分片重平衡** + - 节点新增/宕机时，ES 会自动将分片重新分配（Shards Reallocation），保证负载均衡。 + +4. **写入确认机制** + - ES 在写入时会等待主分片和副本分片都确认成功（可配置 `acks=all`），保证数据安全。 + +5. **跨机房/跨集群容灾** + - 使用 **跨集群复制（CCR, Cross-Cluster Replication）** 将索引实时同步到另一个集群，保证异地容灾。 + - 或者使用 **快照（Snapshot & Restore）** 将索引数据定期备份到远程存储（如 S3、HDFS）。 + + + +### 🎯 ES 的批量操作（Bulk API）怎么用？为什么比逐条写入快？ + +> **Bulk API 是批量写入接口**，通过在一次请求中提交多条写操作，减少了网络交互和分片路由开销。 +> +> **性能提升的原因**： +> +> 1. 减少网络请求次数； +> 2. 按分片分组批量写入，降低路由计算开销； +> 3. 批量写入磁盘，减少 fsync 次数。 +> +> **实践建议**：控制请求大小（5~15MB）、分批处理、使用 BulkProcessor 自动管理批量请求，并处理错误重试。 + +**什么是 Bulk API？** + +- **Bulk API** 是 Elasticsearch 提供的 **批量写入/更新/删除文档** 的接口。 +- 可以在一个请求中同时执行多条写操作，而不是逐条发请求。 + +--- + +**Bulk API 的请求格式** + +Bulk 请求是 **NDJSON（换行分隔 JSON）** 格式，由两部分组成： +1. **动作行**：指定操作类型（index、create、update、delete）、目标索引、文档 `_id` 等。 +2. **数据行**：实际的文档内容（仅在 index/create/update 时需要）。 + +示例： +```bash +POST /my_index/_bulk +{ "index": { "_id": "1" } } +{ "title": "Elasticsearch 入门", "author": "Tom" } +{ "index": { "_id": "2" } } +{ "title": "Elasticsearch 进阶", "author": "Jerry" } +{ "delete": { "_id": "3" } } +``` + +**为什么 Bulk API 比逐条写入快？** + +（1）减少网络开销 + +- 逐条写入：每个文档一次 HTTP 请求，网络开销大。 +- 批量写入：多个操作合并为一次请求，**降低 TCP/HTTP 握手和序列化成本**。 + +（2）优化分片路由 + +- 每次写入时，ES 都要通过 `hash(_routing) % 分片数` 定位分片。 +- Bulk API 会先解析所有文档的目标分片，然后按分片分组批量发送给对应节点，**避免频繁的分片路由开销**。 + +（3）并行写入优化 + +- 在每个节点上，Bulk 操作会将数据批量写入 **内存 buffer → translog → segment**，减少磁盘 IO。 +- 相比逐条写入，磁盘 fsync 次数更少，**写入效率显著提升**。 + +------ + +**使用 Bulk 的最佳实践** + +1. **控制单次请求大小** + + - 官方建议：每次 bulk 请求大小控制在 **5~15 MB**，避免请求过大导致 OOM 或 GC 压力。 + - 批量条数：通常 1000~5000 条一批，具体取决于文档大小。 + +2. **结合批处理工具** + + - Logstash、Beats、ES-Hadoop 等工具都支持 Bulk 写入。 + - Java 客户端有 `BulkProcessor`，可自动批量聚合请求。 + +3. **错误处理** + + - Bulk 是“部分成功”的：部分文档可能写入失败，需要检查返回结果中的 `errors` 字段并重试。 + + + +### 🎯 ES 如何实现分页？深分页有什么问题？如何优化？（scroll、search_after） + +> - ES 分页通过 `from + size` 实现，但深分页会导致性能问题，因为 ES 必须扫描并排序大量文档。 +> - **优化方案**： +> 1. **Scroll API** —— 适合全量数据导出，不适合实时分页。 +> 2. **search_after** —— 基于游标的方式，适合实时无限滚动。 +> 3. **PIT + search_after** —— 新版本中推荐的分页方式，既高效又保证一致性。 +> - 最佳实践：避免深分页，更多使用“滚动加载”而不是“跳页”。 + +**ES 如何实现分页？** + +Elasticsearch 默认提供类似 SQL 的分页语法： + +```json +GET /my_index/_search +{ + "from": 0, + "size": 10 +} +``` + +- `from`：跳过的文档数（偏移量）。 +- `size`：返回的文档数。 +- 例子：`from=0, size=10` 表示返回前 10 条；`from=100, size=10` 表示返回第 101~110 条。 + +------ + +**深分页问题** + +当使用较大 `from` 值时（例如 `from=100000`），会带来性能问题： + +1. **数据扫描开销大**：ES 需要先收集 `from + size` 条文档，再丢弃前 `from` 条。 + - 假设 `from=100000, size=10`，实际上 ES 会拉取 **100010 条**，再只返回最后 10 条。 +2. **内存与 CPU 消耗高**：所有候选文档排序后再丢弃，耗费大量堆内存（优先队列维护 topN）。 +3. **响应延迟大**：深分页请求可能导致查询非常慢，甚至引发集群性能下降。 + +------ + +**深分页优化方案** + +（1）Scroll API —— 大数据量全量遍历 + +- **适用场景**：导出、数据迁移、离线计算。 + +- **原理**：保持一个快照上下文，游标式批量拉取数据。 + +- **特点**： + + - 适合遍历 **全部数据**，不适合用户实时分页。 + - 快照会消耗资源，长时间保持 scroll 不推荐。 + +- 示例： + + ``` + GET /my_index/_search?scroll=1m + { + "size": 1000, + "query": { "match_all": {} } + } + ``` + +（2）search_after —— 基于游标的实时分页 + +- **适用场景**：实时分页、无限滚动（类似微博/朋友圈流式翻页）。 + +- **原理**：通过上一页最后一条文档的排序值，作为下一页的起点。 + +- **特点**： + + - 避免 `from+size` 扫描开销，不会丢弃前面文档。 + - 必须有 **唯一且稳定的排序字段**（如 `时间戳 + _id`）。 + +- 示例： + + ``` + GET /my_index/_search + { + "size": 10, + "query": { "match_all": {} }, + "sort": [ + { "timestamp": "asc" }, + { "_id": "asc" } + ], + "search_after": ["2025-08-14T12:00:00", "doc_123"] + } + ``` + +（3）其他优化手段 + +- **限制最大 from**：通过 `index.max_result_window`（默认 10000）限制深分页请求。 +- **使用 Point in Time（PIT）+ search_after**（ES 7.10+）： + - PIT 提供一致性的快照上下文，配合 `search_after` 实现高效、稳定分页。 +- **业务层优化**： + - **推荐“下拉加载更多”而不是“跳到第 N 页”**。 + - 缓存热点页数据，避免频繁查询深分页。 + + + +### 🎯 ES 的聚合（Aggregation）是什么？常见聚合类型？ + +> - **聚合是 ES 的数据分析功能，类似 SQL 的 GROUP BY/聚合函数。** +> - **常见类型**： +> 1. **Metric**：度量统计（avg、sum、min、max、cardinality）。 +> 2. **Bucket**：分桶分组（terms、range、histogram、date_histogram）。 +> 3. **Pipeline**：基于聚合结果再计算（derivative、moving_avg、cumulative_sum）。 +> 4. **Matrix**：多字段矩阵运算（matrix_stats）。 +> - 实际应用：报表统计、用户行为分析、日志分析等。 +> +> + +**什么是聚合？** + +- **聚合（Aggregation）** 是 Elasticsearch 提供的数据分析功能，类似 SQL 的 `GROUP BY`、`COUNT`、`SUM`。 +- 聚合可以在搜索的同时，对结果进行统计、分组、计算，用于实现数据分析和报表。 +- 查询结果包含两部分： + 1. **hits**：匹配的文档。 + 2. **aggregations**：聚合分析结果。 + +--- + +**聚合的分类** + +ES 聚合主要分为四大类： + +（1）Metric Aggregations —— 度量聚合 + +对字段值进行数学计算，常用于指标统计。 +- `min` / `max`：最小值、最大值 +- `sum`：求和 +- `avg`：平均值 +- `stats`：返回 count、min、max、avg、sum +- `extended_stats`：比 `stats` 更多，包括方差、标准差等 +- `cardinality`：去重计数（类似 SQL `COUNT(DISTINCT)`） + +示例： +```json +GET /sales/_search +{ + "size": 0, + "aggs": { + "avg_price": { "avg": { "field": "price" } } + } +} +``` + +（2）Bucket Aggregations —— 桶聚合 + +把文档分组到不同的“桶”里，类似 SQL 的 `GROUP BY`。 + +- `terms`：按字段值分组（类似 `GROUP BY category`）。 +- `range`：按数值范围分组（如 0~~100, 100~~200）。 +- `date_histogram`：按时间区间分组（按天、按月统计）。 +- `histogram`：按数值范围分组（固定间隔）。 + +示例： + +``` +GET /sales/_search +{ + "size": 0, + "aggs": { + "sales_by_category": { + "terms": { "field": "category.keyword" } + } + } +} +``` + +------ + +（3）Pipeline Aggregations —— 管道聚合 + +基于其他聚合结果再次计算，类似 SQL 的窗口函数。 + +- `derivative`：求导数（趋势变化）。 +- `moving_avg`：移动平均。 +- `cumulative_sum`：累计和。 +- `bucket_script`：对多个聚合结果进行计算。 + +------ + +（4）Matrix Aggregations —— 矩阵聚合 + +对多个字段做矩阵运算，常用于相关性分析。 + +- `matrix_stats`：多个字段的协方差、相关系数等。 + +------ + +**常见的聚合类型总结表** + +| 聚合类型 | 示例 | 类似 SQL | +| ----------------------------- | ---------------------- | --------------------------------------- | +| `avg` / `sum` / `min` / `max` | 平均值、求和、最大最小 | `AVG(price)` | +| `terms` | 按字段分组 | `GROUP BY category` | +| `date_histogram` | 按时间分组 | `GROUP BY DATE(created_at)` | +| `range` | 按区间分组 | `CASE WHEN price BETWEEN 0 AND 100 ...` | +| `cardinality` | 去重计数 | `COUNT(DISTINCT user_id)` | +| `cumulative_sum` | 累计值 | `SUM() OVER (ORDER BY ...)` | + +------ + + + +### 🎯 什么是 parent-child 关系和 nested 类型？ + +> **Nested 类型**：用于处理对象数组，避免扁平化引发的错误匹配；但更新开销大。 +> +> **Parent-Child 关系**：用于父子文档建模，子文档可独立存储和更新；但查询性能差一些。 +> +> **选择策略**： +> +> - 如果是对象数组内部字段匹配 → 用 `nested`。 +> - 如果是父子模型且子文档需独立更新 → 用 `parent-child`。 + +1. Parent-Child 关系 + +- **概念**：类似关系型数据库的一对多关系。父文档和子文档分别独立存储，但通过 `join` 字段建立关联。 +- **特点**： + - 子文档可以单独增删改，不需要修改父文档。 + - 查询时可以用 `has_child`、`has_parent`、`parent_id` 来做父子关联查询。 + - 必须指定 routing，让父子文档落在同一个分片，否则无法关联。 +- **优缺点**： + - 优点：子文档独立更新，存储灵活。 + - 缺点：查询时需要类似 join 的操作，性能较差。 +- **应用场景**：电商（商品-评论）、论坛（帖子-回复）、多租户系统（租户-用户）。 + +------ + +2. Nested 类型 + +- **概念**：一种特殊的对象数组类型，用来避免 Elasticsearch 默认“扁平化”存储带来的错误匹配。 +- **问题背景**：普通对象数组在查询时会被打散，可能把不同对象的字段组合错误（如查询 `name=Alice AND age=25`，可能返回 `name=Alice` 和 `age=25` 来自不同对象）。 +- **解决方案**：定义字段为 `nested` 类型，ES 会将数组里的每个对象存储为一个“隐藏文档”，保证内部字段的独立性。 +- **特点**： + - 优点：查询结果精确，避免字段错配。 + - 缺点：更新时需要整体替换整个 nested 字段，性能较差。 +- **应用场景**：用户和地址列表、订单和商品项等对象数组。 + +------ + +3. Parent-Child vs Nested 对比 + +| 特性 | Parent-Child | Nested | +| -------- | ------------------------------------ | -------------------------------------- | +| 关系模型 | 父子关系（类似数据库一对多） | 对象数组内部独立文档 | +| 存储方式 | 父子文档独立存储 | 每个嵌套对象变成隐藏文档 | +| 更新开销 | 子文档可独立更新，开销较低 | 更新需要整体重建，开销较大 | +| 查询性能 | join-like 查询，性能一般 | 查询比普通字段慢，但比 parent-child 快 | +| 适用场景 | 商品-评论、帖子-回复、多租户关系建模 | 地址列表、订单商品数组、用户偏好等 | + + + +### 🎯 ES 的脚本查询（Painless Script）是什么？ + +> **Painless Script 是 ES 内置的安全高效脚本语言，用来在查询、聚合、打分过程中做动态计算。** +> +> **常见用途**：自定义打分（script_score）、动态字段（script_fields）、聚合计算、复杂过滤。 +> +> **访问方式**：`doc`（推荐）、`_source`（性能差）、`params`（外部传参）。 +> +> **注意点**：脚本灵活但性能较差，建议只在必要时使用，并且结合参数化。 + +1. 什么是 Painless Script？ + +- **Painless** 是 Elasticsearch 默认的脚本语言，用于在查询或聚合过程中做动态计算。 +- 类似 SQL 中的表达式或函数，但它是嵌入在 ES 查询里的。 +- 特点是：**安全（sandbox）、高效、简洁**，比 Groovy 等脚本更快，且默认启用。 + +------ + +2. 使用场景 + +1. **动态计算字段** + 比如计算折扣价、加权分数： + + ``` + GET /products/_search + { + "query": { + "script_score": { + "query": { "match_all": {} }, + "script": { + "source": "doc['price'].value * params.factor", + "params": { "factor": 0.8 } + } + } + } + } + ``` + + → 在搜索时动态计算“打 8 折价格”。 + +2. **复杂排序** + 按多个字段动态排序，比如销量和点击率加权。 + +3. **过滤条件** + 可以写自定义逻辑，比如某个范围或复杂业务规则。 + +4. **聚合计算** + 在 Aggregation 里进行复杂的统计或自定义指标。 + +------ + +3. 常见的 Script 类型 + +- `script_score`：自定义打分，用于排序。 +- `script_fields`：生成新的字段输出。 +- `script` inside `aggs`：聚合计算时动态处理字段。 +- `script` inside `query`：自定义过滤逻辑。 + +------ + +4. Painless 脚本的访问方式 + +- **`doc['field'].value`**：最常用，访问倒排索引里的字段值（高效）。 +- **`_source['field']`**：访问原始文档 `_source` 字段（开销较大，不推荐频繁用）。 +- **params**：用户传入的参数，避免硬编码，提升可读性和性能。 + +------ + +5. 优缺点 + +- **优点**：灵活，能实现 SQL 无法表达的复杂逻辑。 + + + +## 🚨 六、异常与故障处理（运维必备） + +> **核心思想**：生产环境中的异常处理能力是高级工程师的标志，包括脑裂处理、性能排查、资源优化等关键技能。 + +- **集群故障**：[脑裂问题](#es-的脑裂问题是什么如何解决) | [查询慢排查](#如果-es-查询很慢你会怎么排查) +- **资源优化**：[内存优化](#es-节点内存使用过高如何优化) | [数据不均衡](#es-数据不均衡可能是什么原因) + +### 🎯 ES 的“脑裂”问题是什么？如何解决？ + +> 脑裂是指 Elasticsearch 集群由于网络分区或配置不当，出现多个主节点并存，导致数据不一致的问题。 +> +> 在 7.x 以前，需要手动设置 `discovery.zen.minimum_master_nodes = (N/2)+1` 来避免。 +> +> 在 7.x 以后，ES 内置了基于多数派的选举机制，推荐至少部署 3 个 master 节点来保证高可用。 + +**1. 什么是“脑裂”问题？** + +“脑裂”（Split-Brain）是分布式系统中的经典问题，在 Elasticsearch 中主要出现在 **集群主节点选举** 时。 + +- 在 ES 集群里，**主节点（Master Node）** 负责维护集群的元数据和分片分配。 +- 如果由于网络抖动、节点宕机等原因，导致集群内的节点互相失联，可能出现 **多个节点都认为自己是主节点** 的情况。 +- 结果： + - 集群出现多个“主节点”，分片分配混乱； + - 数据可能被写入不同的分支，导致 **数据丢失或不一致**。 + +这就是脑裂。 + +------ + +**2. 脑裂出现的原因** + +1. **网络分区**（最常见） + 主节点与部分节点失联，分区里的节点重新选举出一个“主”，就导致两个主并存。 +2. **最小主节点数（minimum_master_nodes）配置不当（7.x 前）** + 没有严格控制主节点选举的法定人数（quorum），导致小部分节点也能选出主。 +3. **集群规模小** + 如果只有 1 个或 2 个 master-eligible 节点，网络波动很容易导致误选。 + +------ + +**3. 解决办法** + +（1）7.x 以前（Zen Discovery） + +- **关键参数**：`discovery.zen.minimum_master_nodes` + +- 配置规则： + + ``` + minimum_master_nodes = (master_eligible_nodes / 2) + 1 + ``` + + 保证超过半数的节点才能选出主节点，避免小分区自立为王。 + +- 举例： + + - 如果有 3 个 master-eligible 节点 → `minimum_master_nodes = 2`。 + - 如果有 5 个 master-eligible 节点 → `minimum_master_nodes = 3`。 + +------ + +（2）7.x 之后（基于 Zen2 选举） + +- **引入基于 Raft 思想的选举机制**：自动保证多数派选主，不再需要手动配置 `minimum_master_nodes`。 +- **节点角色分离**：推荐部署 **3 个专用 master 节点**，保证选举稳定。 +- **稳定网络**：尽量避免跨机房部署 master 节点，否则容易因网络延迟导致脑裂。 + +------ + +**4. 最佳实践** + +1. **至少 3 个 master-eligible 节点**（奇数个最好），避免选举僵局。 +2. **主节点和数据节点分离**：生产环境里建议把 master 节点和 data 节点分开部署。 +3. **网络可靠性**：避免 master 节点跨机房，或者使用专用网络。 +4. **升级 ES 版本**：7.x+ 自动避免脑裂问题，不再依赖手动配置。 + + + +### 🎯 如果 ES 查询很慢，你会怎么排查？ + +> 先检查查询语句是否合理，比如是否走倒排索引，是否用了低效的通配符或脚本； +> +> 然后看索引和分片设计，是否分片过大或过多，字段类型是否合适； +> +> 再看集群层面，CPU/内存/GC/磁盘/网络是否存在瓶颈； +> +> 使用 Profile API、Slowlog、Hot Threads 等工具定位慢查询的具体原因； +> +> 针对问题优化，比如改查询语句、调整映射、冷热分离、增加节点或调整硬件。 + +**1. 查询语句本身的问题** + +- **是否走倒排索引**： + - 用 `term`/`match` 这种能利用倒排索引的查询。 + - 避免 `wildcard`（尤其是前缀模糊 `*abc`）、`regexp`、`script` 这类开销大的查询。 +- **字段类型是否合适**： + - `text` 字段用来分词，模糊查询效率高； + - `keyword` 字段用于精确匹配； + - 类型错了会导致查询全表扫描。 +- **排序字段是否有 doc_values**： + - 只有 keyword、数值、日期等字段默认有 `doc_values`，可以高效排序。 + - 如果在 `text` 上排序，会非常慢。 +- **聚合是否合理**： + - 在高基数字段（如 user_id）上做 terms 聚合，会很耗内存和 CPU。 + - 可以考虑用 `composite aggregation` 分页聚合，或者预聚合。 + +------ + +**2. 数据量和索引设计问题** + +- **分片数量是否合理**： + - 分片过多：查询需要跨太多分片，协调开销大。 + - 分片过少：单个分片数据量过大，查询压力集中。 + - 一般建议单分片大小控制在 10GB ~ 50GB。 +- **冷热数据分离**： + - 热数据放在性能好的节点，冷数据走 archive 或 ILM（Index Lifecycle Management）。 +- **是否存在 Nested 或 Parent-Child 结构**： + - 这类结构查询开销大，可能需要扁平化建模。 + +------ + +**3. 集群和硬件资源问题** + +- **CPU / 内存是否打满**：查询会被阻塞。 +- **JVM 配置是否合理**： + - 堆内存不要超过物理内存的 50%，且最大 32GB； + - GC 频繁会导致延迟。 +- **磁盘 I/O 和存储类型**： + - SSD 比 HDD 查询快很多。 +- **网络延迟**：节点间通信慢也会拖慢查询。 + +------ + +**4. 排查工具与手段** + +- **Profile API** + + ``` + GET /index/_search + { + "profile": true, + "query": { "match": { "title": "elasticsearch" } } + } + ``` + + → 可以看到查询的执行过程、耗时在哪个阶段。 + +- **Explain API** + + ``` + GET /index/_explain/1 + { + "query": { "match": { "title": "es" } } + } + ``` + + → 分析文档为什么能匹配，调试查询逻辑。 + +- **Hot Threads API** + + ``` + GET /_nodes/hot_threads + ``` + + → 查看 CPU/线程占用，排查是否卡在查询或 GC。 + +- **慢查询日志（Slowlog）** + + - ES 可以配置查询慢日志和索引慢日志，帮助定位具体的慢查询语句。 + +------ + +**5. 优化手段** + +- 语句优化：用 `term` / `match` 替换通配符，提前过滤范围条件。 +- 数据建模：冷热分离，必要时做数据冗余，换结构。 +- 硬件优化：换 SSD，增加节点，分片调整。 +- 查询层优化：缓存结果（query cache）、预聚合数据、减少返回字段 `_source`。 + + + +### 🎯 ES 节点内存使用过高，如何优化？ + +> Elasticsearch 内存使用过高通常有 JVM 堆配置不合理、fielddata 占用过大、分片设计不合理、查询聚合过重等原因。优化思路是从 **JVM 调整、索引设计、查询习惯、集群架构** 四个方向入手，比如合理配置堆内存、避免在 text 字段上聚合、使用 keyword + doc_values、限制 fielddata、冷热分离和合理分片。 + +**1. ES 内存为什么容易占用过高？** + +ES 是基于 Lucene 的搜索引擎，本质上是一个 **计算 + 存储混合负载** 的系统，内存消耗来自几个方面： + +1. **JVM 堆内存** + - 用于存储倒排索引缓存、fielddata、聚合数据、过滤器缓存等。 + - 如果堆分配不合理，容易 OOM 或频繁 GC。 +2. **JVM 堆外内存（Off-heap）** + - Lucene 使用 `mmap` 将倒排索引文件映射到内存，依赖操作系统的 **Page Cache**。 + - 聚合、排序时也会占用大量堆外内存。 +3. **缓存相关** + - **Fielddata Cache**（聚合、排序时加载的字段数据，容易爆内存）。 + - **Query Cache**、**Request Cache**（缓存查询结果）。 + +------ + +**2. 优化思路** + +从 **配置、索引设计、查询习惯、集群架构** 四个方面优化。 + +------ + +（1）JVM 配置优化 + +- **堆内存设置** + - 推荐设置为物理内存的 **50% 左右**，但不要超过 **32GB**（超过后会禁用压缩指针，反而效率低）。 + - 比如 64GB 内存的机器 → JVM 堆 30GB 左右即可。 +- **避免频繁 GC** + - 观察 `/_nodes/stats/jvm`，如果 Old GC 时间过长 → 内存不足或内存泄漏。 + +------ + +（2）索引和字段设计优化 + +- **禁用不必要的 `_source`、`_all` 字段**：减少存储和内存占用。 +- **合理选择字段类型** + - 用 `keyword` 存储精确值； + - 避免在高基数字段上使用 `fielddata`，可以预先建 `keyword` 字段，启用 `doc_values`。 +- **控制 mapping 的字段数量** + - 动态 mapping 可能生成成千上万个字段，导致内存暴涨。 + - 禁用 dynamic 或者严格控制。 + +------ + +（3）查询与聚合优化 + +- **避免在 `text` 字段上排序/聚合** + - 因为会触发 fielddata 加载，占用大量内存。 + - 替代方案：用 `keyword` 字段或开启 `doc_values`。 +- **限制聚合规模** + - `terms` 聚合在高基数字段上会爆内存，考虑 `composite aggregation` 分页聚合。 +- **减少返回结果集大小** + - 查询时指定 `_source` 的字段，而不是直接返回全文档。 +- **善用 Scroll / search_after** + - 避免深分页带来的内存和 CPU 消耗。 + +------ + +（4）集群和架构层面 + +- **冷热分离** + - 热数据放在内存和 SSD 资源好的节点； + - 冷数据用低成本节点存储，减少对内存的争用。 +- **增加节点 / 合理分片** + - 分片过大：内存压力集中； + - 分片过多：管理开销大。建议单分片 10~50GB。 +- **监控缓存使用情况** + - `GET /_nodes/stats/indices/fielddata?human` 查看 fielddata 占用； + - 可以通过 `indices.breaker.fielddata.limit` 限制 fielddata 占用，避免 OOM。 + +------ + +**3. 最佳实践总结** + +1. **JVM 堆不要超过 32GB**，设置为物理内存一半左右。 +2. **避免在 text 字段上做聚合/排序**，使用 keyword + doc_values。 +3. **控制 fielddata 内存使用**，必要时启用限制。 +4. **分片设计合理**，避免单分片过大或过多。 +5. **冷热数据分离**，减少对热节点内存的争用。 +6. **查询优化**：减少返回字段，避免深分页，控制聚合规模。 +7. **监控 + 慢查询日志**：结合 `_nodes/stats`、Slowlog 定位问题。 + + + +### 🎯 ES 数据不均衡，可能是什么原因？ + +> ES 数据不均衡通常由 **分片分配策略不合理（分片数、routing key 热点）、节点磁盘/角色限制、索引生命周期策略、集群扩容设计问题** 等原因导致。解决方法包括 **合理规划分片数量、优化 routing 策略、调整磁盘水位线、启用 rebalance、冷热分离优化** 等。如果业务数据分布不均，重点要检查是否使用了自定义 routing 导致“热点分片”。 + +**1. ES 数据不均衡的现象** + +- 某些节点磁盘、CPU、内存负载明显更高。 +- 分片分配到某些节点过多，而其他节点很空闲。 +- 查询路由大部分集中到某些节点，导致“热点”问题。 + +------ + +**2. 可能原因** + +（1）分片分配策略问题 + +1. **分片数量设计不合理** + - 单个索引分片数太少 → 无法均匀分配到所有节点。 + - 分片数太多 → 部分节点分配过多分片，负载不均。 +2. **路由（Routing）导致数据倾斜** + - ES 默认用 `_id` hash 路由到分片，但如果指定了自定义 routing key 且分布不均，会导致部分分片数据量过大，形成“热点分片”。 +3. **分片再平衡未触发** + - ES 有 `cluster.routing.allocation.*` 配置控制分片分配，如果参数设置不合理，可能阻止分片迁移。 + +------ + +（2）硬件或节点问题 + +1. **磁盘水位线** + - ES 默认有磁盘水位线（85%、90%），超过阈值的节点不会再分配分片，导致数据堆积到少数节点。 +2. **节点角色配置不均** + - 某些节点是 **data node**，而有些节点不是，分片只能分配到 data node 上，可能导致分布不均。 +3. **机器配置不一致** + - 部分节点 CPU、内存或磁盘小，ES 默认不会均衡考虑性能 → 数据集中到高配节点。 + +------ + +（3）集群操作导致 + +1. **手动分配分片** + - 运维手动分配过分片，打破了自动均衡策略。 +2. **索引生命周期管理（ILM）** + - 热-温-冷架构里，索引会迁移到不同节点（比如冷节点存储更多历史数据），导致冷热不均衡。 +3. **多索引大小差异过大** + - 某些索引数据量极大，占用了大部分节点空间，导致不均衡。 + +------ + +3. 解决方案 + +1. **检查分片设计** + + - 保证分片数量合理，一般单分片 10~50GB。 + - 避免索引分片数过少或过多。 + +2. **优化 Routing 策略** + + - 如果用自定义 routing，确保 key 分布均匀。 + - 可以使用 `shard=custom` 配合 **hash 算法**，避免热点。 + +3. **分片重新分配（Rebalance）** + + - 查看分片分布： + + ``` + GET /_cat/shards?v + ``` + + - 手动移动分片： + + ``` + POST /_cluster/reroute + { + "commands": [ + { + "move": { + "index": "my_index", + "shard": 0, + "from_node": "node1", + "to_node": "node2" + } + } + ] + } + ``` + +4. **检查磁盘水位配置** + + - 调整磁盘阈值： + + ``` + cluster.routing.allocation.disk.watermark.low: 70% + cluster.routing.allocation.disk.watermark.high: 85% + cluster.routing.allocation.disk.watermark.flood_stage: 95% + ``` + +5. **冷热分离设计合理化** + + - 热节点只存储活跃数据；冷节点承担更多历史索引，但避免过度集中。 + +6. **增加节点或扩容索引** + + - 如果数据量持续增长，可能需要新增 data node 或者调整分片数。 + +------ + + + +## 💼 七、实战场景题（项目经验） + +> **核心思想**：实战场景题考察的是你的项目经验和解决实际问题的能力，是面试官判断你技术水平的重要依据。 + +- **项目场景**：[项目实践经验](#你在项目里是怎么用-elasticsearch-的场景是什么) | [集群架构设计](#elasticsearch了解多少说说你们公司es的集群架构) +- **索引设计**：[日志索引设计](#如果存储日志数据如何设计索引) | [数据清理策略](#如果要删除过期数据怎么做) +- **数据处理**：[SQL Join实现](#如何在-es-中实现类似-sql-的-join-操作) | [删除操作原理](#elasticsearch-如何处理数据的删除操作) +- **技术选型**：[与其他技术对比分析](#elasticsearch-和-数据库的区别) | [应用场景分析](#elasticsearch-的应用场景) + +### 🎯 你在项目里是怎么用 Elasticsearch 的？场景是什么？ + +> 我们在项目里用 Elasticsearch 做 **商品检索**，规模大概 **10 亿商品**，分属于 **14 个行业**。为了提升性能，我们按行业建索引，避免单索引过大。 +> 在 Mapping 上，商品标题和描述用 `text`+中文分词器，品牌和类目用 `keyword`，价格、销量用数值型，支持范围过滤和排序。 +> 数据写入通过 **Kafka + Bulk API**，支持实时更新，同时控制 `refresh_interval` 优化写入性能。 +> 查询方面支持全文检索、条件过滤、聚合分析、排序，典型应用就是电商的搜索和筛选面板。 +> 在性能上，我们做了分片规划、冷热分离、缓存优化，深分页用 `search_after` 替代 `from+size`。 +> 整个集群有多节点，分片+副本保证了高可用，也能横向扩展。 + +**1. 背景与业务场景** + +我们项目有一个 **商品库**，规模在 **10 亿级别**，覆盖 **14 个行业**。业务上需要支持： + +- **多维度搜索**：比如按关键字、类目、品牌、价格区间、属性等。 +- **过滤与排序**：比如销量排序、价格升降序、上架时间、个性化推荐。 +- **聚合统计**：比如某个品类下的品牌分布、价格区间分布。 +- **高并发**：搜索请求量非常大，需要保证 **低延迟（<200ms）**。 + +------ + +**2. 为什么用 Elasticsearch** + +- 关系型数据库（MySQL）不适合做复杂搜索 + 聚合，查询会非常慢。 +- Elasticsearch 提供 **倒排索引、分布式存储、全文检索、多维聚合**，天然适合商品搜索这种场景。 +- 内置了 **高可用、扩展性**，能应对大规模数据量。 + +------ + +**3. 数据建模 & 索引设计** + +- **索引设计**：按 **行业维度**划分索引（14个 index），避免单索引过大，提升查询性能和可扩展性。 +- **Mapping 设计**： + - 商品标题、描述 → `text` 类型 + 中文分词器（ik_max_word / smart）。 + - 品牌、类目、属性（枚举型字段） → `keyword` 类型，支持聚合和过滤。 + - 价格、销量、评分 → `integer` / `float` 类型，支持范围查询与排序。 + - 上架时间 → `date` 类型。 +- **Routing**：默认按 `_id` 分配，也考虑过行业内的二级路由（如类目），以减少跨分片查询。 + +------ + +**4. 数据写入 & 更新** + +- 商品数据源来自 **数据库（MySQL）+ 消息队列（Kafka）**。 +- **全量导入**：使用 **批量 Bulk API**，并行分片写入。 +- **增量更新**：通过 Kafka 消息触发更新，ES 消费端批量写入。 +- **写入优化**： + - 使用 **bulk 批量写入**，避免单条写入开销。 + - 控制 **refresh_interval**（比如 30s），批量刷新，减少 segment 频繁生成。 + +------ + +**5. 查询场景** + +- **搜索**：`match`、`multi_match` 支持商品标题 + 描述搜索。 +- **过滤**：通过 `term`、`range` 精准过滤行业、类目、价格区间。 +- **聚合**：统计某个品类下的品牌分布、价格区间分布（用于商品筛选面板）。 +- **排序**：综合销量、价格、时间等字段；部分场景结合 **function_score** 动态打分。 + +------ + +**6. 性能优化** + +- **分片规划**：单索引 10 亿数据时，分片数量按磁盘与查询并发量规划，保证单分片数据 <50GB。 +- **冷热分离**：热节点存储最新在售商品，冷节点存储历史下架数据。 +- **缓存**：利用 **query cache、request cache**，对于高频查询做缓存。 +- **深分页优化**：避免 `from+size`，用 `search_after` 或 `scroll`。 +- **监控**：通过 Kibana + X-Pack 监控查询延时、分片分布、节点健康。 + +------ + +**7. 高可用 & 扩展** + +- **集群架构**：多节点，分片+副本机制保证高可用。 +- **副本分片**：1 个主分片 + 1 副本，支持读扩展和容灾。 +- **扩展性**：按行业独立索引，可以横向扩展集群，增加节点时只需重分配分片。 + + + +### 🎯 如果存储日志数据，如何设计索引？按天建索引还是按月建索引？为什么？ + +> 在日志场景下，通常会采用 **时间分区索引**。 +> 如果日志量大，我会 **按天建索引**，这样每个索引的数据量可控，查询性能高，删除旧数据也很方便。 +> 如果日志量比较小，可以考虑 **按月建索引**，避免产生太多小索引。 +> 我们一般还会结合 **ILM 策略**，在热节点存放近 7 天日志，冷节点存放历史日志，过期数据自动删除。这样既能保证查询性能，也能节省存储成本。 + +**日志数据索引设计思路** + +日志数据的特点： + +- **数据量大**（每天可能上亿条甚至更多）。 +- **只写不改**（日志一旦写入基本不会更新）。 +- **时间序列数据**（查询通常带有时间范围条件，比如近 1 小时、近 1 天、近 1 周）。 +- **冷热分离明显**（新日志查得多，旧日志查得少）。 + +所以日志场景下，**常用的方式是时间分区索引**： + +- `log-2025.08.19`（按天建索引） +- `log-2025.08`（按月建索引） + +------ + +**按天建索引 vs 按月建索引** + +**按天建索引** + +- **优点**： + 1. 单个索引数据量小，查询性能更好。 + 2. 管理灵活，可以方便地删除过期数据（比如保留 30 天日志）。 + 3. 分片分布均衡，避免超大索引导致分片过大。 +- **缺点**： + 1. 每天都会产生一个新索引，集群里索引数量会非常多（几百上千个）。 + 2. Elasticsearch 每个索引都有元数据开销（集群状态变大，内存占用上升）。 + 3. 如果日志量不大，按天建索引会导致很多小索引，资源利用率不高。 + +------ + +**按月建索引** + +- **优点**： + 1. 索引数量少，集群元数据压力小。 + 2. 适合日志量小的场景，避免太多小索引。 +- **缺点**： + 1. 单个索引可能非常大（几十亿条日志），分片过大导致查询慢、迁移困难。 + 2. 删除旧数据不灵活（只能整月删，无法做到精确到天）。 + 3. 查询时必须扫描很大的索引，性能差。 + +------ + +**最佳实践** + +- **日志量大（>千万/天）** → 建议 **按天建索引**（`log-YYYY.MM.dd`）。 +- **日志量较小（<百万/天）** → 可以 **按月建索引**，避免过多小索引。 +- **更优方案**：使用 **Index Lifecycle Management (ILM)** 策略 + - 在热节点保存最近 7 天索引（高性能 SSD）。 + - 冷节点存储历史日志（机械盘，降低成本）。 + - 超过 30 天直接删除索引。 +- **Rollup/合并索引**：对于长期保存但只需要统计用途的日志，可以用 rollup job，把旧日志压缩为汇总数据。 + + + +### 🎯 如何在 ES 中实现类似 SQL 的 join 操作？ + +> Elasticsearch 本身不支持像 SQL 那样的复杂 join，因为数据分布在不同分片上，性能成本很高。 +> 常见的替代方式有： +> +> - **冗余建模**：把需要 join 的字段直接冗余到一个文档里，性能最好，也是最推荐的。 +> - **nested 类型**：适合一对多的数组场景，比如商品的多个属性或评论。 +> - **parent-child 关系**：可以避免冗余，但查询性能会差一些，只适合更新频繁、数据量大的场景。 +> - 还有一种方式是在 **应用层做 join**，由业务系统做两次查询再拼接结果。 +> +> 实际项目里，如果日志、商品这种场景，我会优先用 **扁平化建模**，保证查询效率。 + +**1. 为什么 ES 不支持传统的 join？** + +- ES 是 **分布式文档存储**，数据被分片分散存储在不同节点上。 +- SQL 的 join 需要跨表匹配数据，在 ES 里会变成跨索引、跨分片的多次网络请求，性能极差。 +- 因此 ES 不推荐在查询阶段做 join，而是鼓励 **在建模阶段设计好数据结构**，避免 join。 + +------ + +**2. 替代 join 的常见方式** + +（1）**数据扁平化 / 冗余（Denormalization）** 👉 最推荐 + +- 把原来需要 join 的数据，直接冗余到一个文档里。 +- 例如：订单和用户信息 + - 在 RDBMS 里：`order` 表 join `user` 表。 + - 在 ES 里：把 `user_name`、`user_email` 冗余到 `order` 文档。 +- **优点**：查询快，天然避免 join。 +- **缺点**：数据冗余较多，更新用户信息时要更新多份数据。 + +------ + +（2）**Nested 类型**（一对多） + +- 适合文档里嵌套数组的场景，比如商品和商品评论。 +- `nested` 字段是独立的 Lucene 文档，但与父文档绑定在一起。 +- 查询时可以保证嵌套对象的独立性，避免“字段交叉”问题。 +- 用法：`nested query`。 +- **适合场景**： + - 一个商品有多个属性（key-value 形式）。 + - 一篇文章有多个评论。 + +------ + +（3）**Parent-Child 关系（join field）** + +- 类似关系型数据库的“一对多”。 +- 使用 `join` 类型字段定义 parent-child 关系。 +- 例如：一个 `blog` 文档（parent）和多个 `comment` 文档（child）。 +- 查询时可以： + - `has_child` 查询：找出有特定评论的文章。 + - `has_parent` 查询：找出属于某类文章的评论。 +- **优点**：避免数据冗余，更新 child 时不需要修改 parent。 +- **缺点**：查询性能差，join 操作消耗大；parent 和 child 必须在同一个 shard。 + +------ + +（4）**应用层 Join** + +- 在应用程序里自己执行两次查询，然后在内存里做关联。 +- **适合**：数据量不大，但偶尔需要 join 的场景。 +- **缺点**：需要两次查询，增加应用层逻辑。 + + + +### 🎯 **Elasticsearch 如何处理数据的删除操作？** + +Elasticsearch 使用的是软删除（soft delete），即标记文档为已删除，而不立即删除文档。最终的删除操作会通过段合并（segment merge）进行清理。 + + + +### 🎯 如果要删除过期数据，怎么做？（ILM、rollover） + +> 在 ES 里删除过期数据有几种方式： +> +> - 最简单的是手动删除索引，或者自己写定时任务每天删。 +> - 更好的方式是结合 **时间分区索引**，每天一个索引，到期直接删除整个索引，效率很高。 +> - 在生产环境里通常会用 **ILM 策略**，比如设置索引每天 rollover 一个新索引，保留 30 天后自动删除。这样整个流程全自动，避免人工干预。 +> +> 我在项目里会推荐 **rollover + ILM**，一方面保证单个索引不会太大，另一方面可以自动清理过期数据，做到冷热分离和存储节省。 + +**1. 手动删除索引** + +- 最简单粗暴的方式，直接 `DELETE /index_name`。 +- 缺点：需要自己写定时任务管理，人工成本高。 +- 适合：小规模测试环境，不推荐生产。 + +------ + +**2. 按时间分区建索引 + 定期删除** + +- 常见做法是日志按天建索引（`logs-2025.08.19`）。 +- 过期数据直接删除整个索引。 +- 优点：删除效率高（删除索引 = 元数据级别操作，秒级完成）。 +- 缺点：需要额外写定时任务。 + +------ + +**3. ILM（Index Lifecycle Management） 👉 推荐** + +- ES 提供的自动索引生命周期管理。 +- 可以定义索引的生命周期策略： + - **Hot**：新数据写入，高性能存储。 + - **Warm**：数据较旧，迁移到低性能节点。 + - **Cold**：更旧的数据，只读，放在冷存储。 + - **Delete**：到期后自动删除索引。 + +**示例策略**： + +``` +PUT _ilm/policy/logs_policy +{ + "policy": { + "phases": { + "hot": { + "actions": { + "rollover": { + "max_age": "1d", + "max_size": "50gb" + } + } + }, + "delete": { + "min_age": "30d", + "actions": { + "delete": {} + } + } + } + } +} +``` + +👉 表示每天 rollover 新索引，保留 30 天后自动删除。 + +------ + +**4. Rollover 索引** + +- 避免单个索引过大。 +- 索引命名通常带有 alias（别名）。 +- 当满足条件（大小/文档数/时间）时，自动 rollover 新索引。 +- 结合 ILM 使用效果最好： + - rollover 负责「切换新索引」。 + - ILM 负责「删除旧索引」。 + +**示例**： + +``` +POST /logs-write/_rollover +{ + "conditions": { + "max_age": "1d", + "max_size": "50gb" + } +} +``` + +👉 当索引达到 1 天或 50GB，就会自动创建新索引，并把写 alias 切换到新索引。 + + + + + +### 🎯 Elasticsearch 如何实现数据的高可用性？ + +通过副本（Replica）来实现数据的冗余存储，副本不仅提供容错能力，还可以用于分担查询负载。 + + + +### 🎯 **如何优化 Elasticsearch 的查询性能？** + +优化查询可以通过合理的索引设计、使用过滤器代替查询、减少 `wildcard` 和 `regexp` 查询的使用等方式实现。 + + + +### 🎯 **如何设计一个高效的 Elasticsearch 索引？** + +考虑索引的字段类型、分析器的选择、是否需要排序等。避免过多的字段、避免高基数字段、合理设置分片和副本数等。 + + + +### 🎯 Elasticsearch了解多少，说说你们公司es的集群架构，索引数据大小，分片有多少，以及一些调优手段 ？ + +`面试官` ：想了解应聘者之前公司接触的ES使用场景、规模，有没有做过比较大规模的索引设计、规划、调优。 + +`解答` ：如实结合自己的实践场景回答即可。比如：ES集群架构13个节点，索引根据通道不同共20+索引，根据日期，每日递增20+，索引：10分片，每日递增1亿+数据，每个通道每天索引大小控制：150GB之内。 + +仅索引层面调优手段： + +#### 1.1、设计阶段调优 + +- 根据业务增量需求，采取基于日期模板创建索引，通过roll over API滚动索引； +- 使用别名进行索引管理； +- 每天凌晨定时对索引做force_merge操作，以释放空间； +- 采取冷热分离机制，热数据存储到SSD，提高检索效率；冷数据定期进行shrink操作，以缩减存储； +- 采取curator进行索引的生命周期管理； +- 仅针对需要分词的字段，合理的设置分词器； +- Mapping阶段充分结合各个字段的属性，是否需要检索、是否需要存储等。 …….. + +#### 1.2、写入调优 + +- 写入前副本数设置为0； +- 写入前关闭refresh_interval设置为-1，禁用刷新机制； +- 写入过程中：采取bulk批量写入； +- 写入后恢复副本数和刷新间隔； +- 尽量使用自动生成的id。 + +#### 1.3、查询调优 + +- 禁用wildcard； +- 禁用批量terms（成百上千的场景）； +- 充分利用倒排索引机制，能keyword类型尽量keyword； +- 数据量大时候，可以先基于时间敲定索引再检索； +- 设置合理的路由机制。 + +#### 1.4、其他调优 + +部署调优，业务调优等。 + +上面的提及一部分，面试者就基本对你之前的实践或者运维经验有所评估了。 + + + + + + + + + +**搜索引擎原理** + +一次完整的搜索从用户输入要查询的关键词开始，比如想查找 Lucene 的相关学习资料，我们都会在 Google 或百度等搜索引擎中输入关键词，比如输入"Lucene ，全文检索框架"，之后系统根据用户输入的关键词返回相关信息。一次检索大致可分为四步： + +**第一步：查询分析** +正常情况下用户输入正确的查询，比如搜索"里约奥运会"这个关键词，用户输入正确完成一次搜索，但是搜索通常都是全开放的，任何的用户输入都是有可能的，很大一部分还是非常口语化和个性化的，有时候还会存在拼写错误，用户不小心把"淘宝"打成"涛宝"，这时候需要用自然语言处理技术来做拼写纠错等处理，以正确理解用户需求。 + +**第二步：分词技术** +这一步利用自然语言处理技术将用户输入的查询语句进行分词，如标准分词会把"lucene全文检索框架"分成 lucene | 全 | 文｜检｜索｜框｜架｜， IK分词会分成： lucene｜全文｜检索｜框架｜,还有简单分词等多种分词方法。 + +**第三步：关键词检索** +提交关键词后在倒排索引库中进行匹配，倒排索引就是关键词和文档之间的对应关系，就像给文档贴上标签。比如在文档集中含有 "lucene" 关键词的有文档1 、文档 6、文档9，含有 "全文检索" 关键词的有文档1 、文档6 那么做与运算，同时含有 "lucene" 和 "全文检索" 的文档就是文档1和文档6，在实际的搜索中会有更复杂的文档匹配模型。 + +**第四步：搜索排序** +对多个相关文档进行相关度计算、排序，返回给用户检索结果。 + +--- + +## 🎯 ES面试备战指南 + +### 💡 高频考点Top10 + +1. **🔥 倒排索引原理** - 必考概念，要能用大白话解释清楚 +2. **⚡ 分片和副本机制** - 高可用架构的核心，要理解分片如何提升性能 +3. **📊 写入和查询流程** - 体现对ES内部机制的掌握程度 +4. **🚨 脑裂问题** - 分布式系统经典问题，解决思路要清晰 +5. **⚙️ refresh/flush/merge** - 近实时搜索的底层原理 +6. **🔍 深分页优化** - scroll vs search_after的选择策略 +7. **💾 性能调优** - 写入和查询两方面的优化手段 +8. **🏗️ 索引设计** - Mapping设计、字段类型选择、分词器配置 +9. **📈 聚合分析** - 复杂数据统计场景的实现方式 +10. **💼 实际项目经验** - 能结合具体场景谈技术选型和架构设计 + +### 🎭 面试答题技巧 + +**📝 标准回答结构** +1. **概念定义**（30秒） - 用一句话说清楚是什么 +2. **核心特点**（1分钟） - 突出关键优势和机制 +3. **应用场景**（30秒） - 什么时候用，解决什么问题 +4. **具体示例**（1分钟） - 最好是自己项目的真实案例 +5. **注意事项**（30秒） - 体现深度思考和实战经验 + +**🗣️ 表达话术模板** +- "从我的项目经验来看..." +- "在生产环境中，我们通常会..." +- "这里有个需要注意的点是..." +- "相比于传统方案，ES的优势在于..." +- "在大规模数据场景下，推荐的做法是..." + +### 🚀 进阶加分点 + +- **底层原理**：能从Lucene层面解释ES的实现机制 +- **性能调优**：有具体的优化数据和效果对比 +- **架构设计**：能设计适合业务场景的ES集群方案 +- **故障处理**：有排查和解决线上ES问题的经验 +- **技术选型**：能准确分析ES与其他存储技术的适用场景 + +### 📚 延伸学习建议 + +- **官方文档**：ES官方Guide是最权威的学习资料 +- **实战练习**：搭建本地ES环境，动手验证各种配置 +- **源码阅读**：有余力可以研究ES核心模块的源码实现 +- **案例分析**：多看大厂的ES应用案例和最佳实践 +- **技术博客**：关注ES相关的技术博客和论坛讨论 + +--- + +## 🎉 总结 + +**Elasticsearch作为现代搜索技术栈的核心**，已经成为各大互联网公司的基础设施。掌握ES不仅是面试加分项，更是技术能力的重要体现。 + +**记住：面试官考察的不是你背了多少概念，而是你能否在实际项目中灵活运用ES解决业务问题。** + +**最后一句话**：*"纸上得来终觉浅，绝知此事要躬行"* - 再多的理论知识都不如亲手搭建一个ES集群来得深刻！ + +--- + +> 💌 **坚持学习，持续成长！** +> 如果这份材料对你有帮助，记得在实际面试中结合自己的理解和经验来回答，让技术知识真正为你所用！ + diff --git a/docs/interview/JUC-FAQ.md b/docs/interview/JUC-FAQ.md index 379e9d34fe..ff3d890561 100644 --- a/docs/interview/JUC-FAQ.md +++ b/docs/interview/JUC-FAQ.md @@ -1,542 +1,821 @@ -在 Java 5.0 提供了 java.util.concurrent（简称 JUC ）包，在此包中增加了在并发编程中很常用的实用工具类，用于定义类似于线程的自定义子系统，包括线程池、异步 IO 和轻量级任务框架。 - +--- +title: Java并发编程 +date: 2024-05-31 +tags: + - JUC + - Interview +categories: Interview +--- + +![](https://img.starfish.ink/common/faq-banner.png) + +> Java并发编程是面试中**最具技术含量**的部分，也是区分初级和高级开发者的重要标准。从基础的线程创建到复杂的JMM内存模型，从synchronized关键字到Lock接口的实现原理，每一个知识点都可能成为面试官深挖的切入点。掌握并发编程，不仅是面试利器，更是高性能系统开发的基石。 +> +> JUC 面试，围绕着这么几个方向准备 +> +> - 多线程的一些概念（进程、线程、并行、并发啥的，谈谈你对高并发的认识） +> - Java 内存模型相关（也可以算是 JVM 的范畴） +> - 同步机制（locks、synchronzied、atomic） +> - 并发容器类 +> - ConcurrentHashMap、CopyOnWriteArrayList、CopyOnWriteArraySet +> - 阻塞队列（顺着就会问到线程池） +> - 线程池（Executor、Callable 、Future、ExecutorService等等，底层原理） +> - AQS +> - AQS 原理 +> - 工具类：CountDownLatch、ReentrantLock、Semaphore、Exchanger +> - atomic 类（atomic常用类，方法，到 CAS，或者 ABA问题） +> - Fork/Join并行计算框架 +> +![](https://img.starfish.ink/common/juc-faq.png) -JUC 面试题总共围绕的就这么几部分 +## 🗺️ 知识导航 -- 多线程的一些概念（进程、线程、并行、并发啥的，谈谈你对高并发的认识） -- 同步机制（locks、synchronzied） -- 并发容器类 - - ConcurrentHashMap、CopyOnWriteArrayList、CopyOnWriteArraySet - - 阻塞队列（顺着就会问到线程池） -- 线程池（Executor、Callable 、Future、ExecutorService等等，底层原理） -- AQS - - AQS 原理 - - 工具类：CountDownLatch、ReentrantLock、Semaphore、Exchanger -- atomic 类（atomic常用类，方法，到 CAS，或者 ABA问题） -- Fork/Join并行计算框架 +### 🏷️ 核心知识分类 -![img](https://img-blog.csdn.net/20180429141212382?watermark/2/text/aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3UwMTEzMDU2ODA=/font/5a6L5L2T/fontsize/400/fill/I0JBQkFCMA==/dissolve/70) +1. **🧵 线程基础**：创建方式、生命周期、线程状态、线程安全、ThreadLocal、进程vs线程、并发vs并行 +2. **🔒 同步关键字**：synchronized原理、volatile特性、锁升级、偏向锁、轻量级锁、重量级锁 +3. **🏛️ 锁机制与AQS**：ReentrantLock、读写锁、AQS框架、公平锁vs非公平锁、死锁问题 +4. **⚛️ 原子操作与CAS**：CAS机制、ABA问题、AtomicInteger、LongAdder、无锁编程 +5. **🛠️ 并发工具类**：CountDownLatch、CyclicBarrier、Semaphore、Exchanger、线程间通信 +6. **🏊 线程池详解**：ThreadPoolExecutor七大参数、工作原理、拒绝策略、参数配置、性能优化 +7. **🧠 Java内存模型**：JMM规范、主内存vs工作内存、happen-before原则、内存屏障、三大特性 +8. **📦 并发容器**：ConcurrentHashMap、CopyOnWriteArrayList、BlockingQueue、同步容器vs并发容器 +9. **🚀 高级并发工具**：ForkJoinPool工作窃取、CompletableFuture异步编程、ThreadLocal原理 +10. **🎯 并发应用实践**：订票系统高并发、网站架构设计、无锁化编程、生产者消费者模式、最佳实践 +### 🔑 面试话术模板 +| **问题类型** | **回答框架** | **关键要点** | **深入扩展** | +| ------------ | ------------------- | ------------ | ----------------- | +| **机制原理** | 背景→实现→特点→应用 | 底层实现机制 | 源码分析、JVM层面 | +| **性能对比** | 场景→测试→数据→结论 | 量化性能差异 | 实际项目经验 | +| **并发问题** | 问题→原因→解决→预防 | 线程安全分析 | 最佳实践模式 | +| **工具选择** | 需求→特点→适用→示例 | 使用场景对比 | 源码实现原理 | -## 多线程开篇 +## 一、线程基础🧵 -### 进程和线程 +### 🎯 进程和线程？ 进程是程序的一次执行过程，是系统运行程序的基本单位，因此进程是动态的。系统运行一个程序即是一个进程从创建，运行到消亡的过程。 -线程是操作系统能够进行运算调度的最小单位，它被包含在进程之中，是进程中的实际运作单位。 - -在 Java 中，当我们启动 main 函数时其实就是启动了一个 JVM 的进程，而 main 函数所在的线程就是这个进程中的一个线程，也称主线程。 +线程是进程中的一个执行单元。 线程是进程的子集，一个进程可以有很多线程，每条线程并行执行不同的任务。不同的进程使用不同的内存空间，而所有的线程共享一片相同的内存空间。 +在 Java 中，当我们启动 main 函数时其实就是启动了一个 JVM 的进程，而 main 函数所在的线程就是这个进程中的一个线程，也称主线程。 +**为什么要用多线程而不是多进程？** -### 说说并发与并行的区别? +- 线程更轻量，切换开销小，通信更高效。 +- 适合 I/O 密集型、CPU 密集型应用。 -- **并发：** 同一时间段，多个任务都在执行 (单位时间内不一定同时执行)； -- **并行：** 单位时间内，多个任务同时执行。 +### 🎯 了解协程么？ -### Java同步机制有哪些 +**协程是用户态的轻量级线程**，由程序主动控制切换（而非操作系统调度），**单线程可运行多个协程**，适合处理高并发、IO 密集型场景。 +*类比记忆*： -1. synchronized关键字，这个相信大家很了解，最好能理解其中的原理 +- 线程是 “操作系统管理的工人”，协程是 “工人（线程）手下的临时工”—— 工人自己安排临时工干活，减少找老板（操作系统）调度的开销。 -2. Lock接口及其实现类，如 ReentrantLock.ReadLock 和 ReentrantReadWriteLock.WriteLock +| **对比维度** | **协程** | **线程** | +| ------------ | ------------------------------------------- | ----------------------------------------- | +| **调度层** | 用户态（编程语言 / 框架控制） | 内核态（操作系统内核调度） | +| **创建成本** | 极低（纳秒级，内存消耗小） | 较高（毫秒级，需分配独立栈内存） | +| **适用场景** | IO 密集型（如网络请求、数据库操作） | CPU 密集型（如复杂计算） | +| **典型框架** | Kotlin 协程、Quarkus Vert.x、Spring WebFlux | 原生 Java 线程、线程池（ExecutorService） | - 以上两种都是最基本的，也是大家在实际项目中最常用的，一般用lock的比较多，能提高效率，典型的对比如Hashtable和CurrentHashMap的性能对比; +**Java 中的协程支持** -那还有那些更高级的同步机制： +1. **现状**： + - Java 标准库目前**无原生协程支持**，需通过框架或语言扩展实现。 + - 主流方案： + - **Kotlin 协程**：通过 JVM 字节码与 Java 互操作（如在 Spring Boot 中混合使用）。 + - **Quarkus**：基于 SmallRye Mutiny 实现响应式编程，底层用协程优化 IO 操作。 + - **Loom 项目（实验性）**：JDK 19 引入轻量级线程（Virtual Threads），类似协程但由 JVM 管理调度。 +2. **未来趋势**： + - Loom 项目的虚拟线程可能在未来 JDK 版本中正式转正，成为 Java 协程的替代方案。 -3. 信号量（Semaphore）:是一种计数器，用来保护一个或者多个共享资源的访问，它是并发编程的一种基础工具，大多数编程语言都提供这个机制，这也是操作系统中经常提到的 -4. CountDownLatch:是Java语言提供的同步辅助类，在完成一组正在其他线程中执行的操作之前，他允许线程一直等待，这个类的使用已经在我的博客中了，大家可以去看看，自己去体验一下，平时编程不常用，但是实际中可能很有用，还是要多了解一下的； -5. CyclicBarrier:也是java语言提供的同步辅助类，他允许多个线程在某一个集合点处进行相互等待；这个感觉慢有意思的，我的博客中已经有了，大家可以去看看 -6. Phaser:也是java语言提供的同步辅助类，他把并发任务分成多个阶段运行，在开始下一阶段之前，当前阶段中所有的线程都必须执行完成，JAVA7才有的特性。 -7. Exchanger:他提供了两个线程之间的数据交换点。 +> **协程的目的** +> +> 在传统的J2EE系统中都是基于每个请求占用一个线程去完成完整的业务逻辑（包括事务）。所以系统的吞吐能力取决于每个线程的操作耗时。如果遇到很耗时的I/O行为，则整个系统的吞吐立刻下降，因为这个时候线程一直处于阻塞状态，如果线程很多的时候，会存在很多线程处于空闲状态（等待该线程执行完才能执行），造成了资源应用不彻底。 +> +> 最常见的例子就是JDBC（它是同步阻塞的），这也是为什么很多人都说数据库是瓶颈的原因。这里的耗时其实是让CPU一直在等待I/O返回，说白了线程根本没有利用CPU去做运算，而是处于空转状态。而另外过多的线程，也会带来更多的ContextSwitch开销。 +> +> 对于上述问题，现阶段行业里的比较流行的解决方案之一就是单线程加上异步回调。其代表派是node.js以及Java里的Vert.x。 +> +> 而协程的目的就是当出现长时间的I/O操作时，通过让出目前的协程调度，执行下一个任务的方式，来消除ContextSwitch上的开销。 +> +> 协程（Coroutine）是一种比线程更轻量的并发处理单元，主要特点是它们可以在一个线程内非阻塞地切换。协程与线程的区别在于： +> +> 1. **线程**由操作系统调度，切换需要系统调用（较高开销）。 +> 2. **协程**由应用程序自己调度，切换仅是函数调用（较低开销）。 +> +> **协程的特点** +> +> 1. 线程的切换由操作系统负责调度，协程由用户自己进行调度，因此减少了上下文切换，提高了效率。 +> 2. 线程的默认Stack大小是1M，而协程更轻量，接近1K。因此可以在相同的内存中开启更多的协程。 +> 3. 由于在同一个线程上，因此可以避免竞争关系而使用锁。 +> 4. 适用于被阻塞的，且需要大量并发的场景。但不适用于大量计算的多线程，遇到此种情况，更好实用线程去解决。 +> +> **协程的原理** +> +> 当出现IO阻塞的时候，由协程的调度器进行调度，通过将数据流立刻yield掉（主动让出），并且记录当前栈上的数据，阻塞完后立刻再通过线程恢复栈，并把阻塞的结果放到这个线程上去跑，这样看上去好像跟写同步代码没有任何差别，这整个流程可以称为coroutine，而跑在由`coroutine`负责调度的线程称为`Fiber`。比如Golang里的 go关键字其实就是负责开启一个`Fiber`，让`func`逻辑跑在上面。 +> +> 由于协程的暂停完全由程序控制，发生在用户态上；而线程的阻塞状态是由操作系统内核来进行切换，发生在内核态上。 +> 因此，协程的开销远远小于线程的开销，也就没有了ContextSwitch上的开销。 +> +> **1. 协程的核心概念** +> +> - **协作式调度**： +> 协程通过显式的 `yield` 或 `suspend` 语句让出执行权，而不是由操作系统抢占式调度。 +> - **共享线程**： +> 多个协程可以运行在同一个线程中，且它们共享该线程的栈空间。 +> +> **2. 协程的实现** +> +> 协程的实现可以基于以下机制： +> +> 1. **状态保存**：协程在挂起时会保存当前的执行状态，包括程序计数器和局部变量。 +> 2. **调度器**：负责管理协程的调度，比如哪个协程运行，哪个挂起。 +> 3. **栈帧管理**：协程通常以轻量的栈帧形式运行，其栈比线程栈更小，支持更高的并发量。 -### synchronized关键字 +### 🎯 说说并发与并行的区别? -> synchoronized的底层是怎么实现的？ -> -> synchronized 使用的几种方式和区别？ -> -> synchronized说一下，有哪些实用形式？对类加锁时调用方法一定会加锁吗？ +- **并发：** 同一时间段，多个任务都在执行 (单位时间内不一定同时执行)； +- **并行：** 单位时间内，多个任务同时执行。 -synrhronized关键字简洁、清晰、语义明确，因此即使有了Lock接口，使用的还是非常广泛。其应用层的语义是可以把任何一个非null对象作为"锁"，当synchronized作用在方法上时，锁住的便是对象实例（this）；当作用在静态方法时锁住的便是对象对应的Class实例，因为 Class数据存在于永久带，因此静态方法锁相当于该类的一个全局锁；当synchronized作用于某一个对象实例时，锁住的便是对应的代码块。在 HotSpot JVM实现中，锁有个专门的名字：对象监视器。 -在JVM中，对象在内存中的布局分为三块区域：对象头、实例数据和对齐填充 -synchronized 用的锁是存在 Java 对象头里的。 +### 🎯 说下同步、异步、阻塞和非阻塞？ -底层实现： +同步和异步两个概念与消息的通知机制有关。也就是**同步与异步主要是从消息通知机制角度来说的**。 -1. 进入时，执行 monitorenter，将计数器 +1，释放锁 monitorexit 时，计数器-1； -2. 当一个线程判断到计数器为 0 时，则当前锁空闲，可以占用；反之，当前线程进入等待状态。 +阻塞和非阻塞这两个概念与程序（线程）等待消息通知(无所谓同步或者异步)时的**状态**有关。也就是说**阻塞与非阻塞主要是程序（线程）等待消息通知时的状态角度来说的** -当执行 monitorenter 指令时，线程试图获取锁也就是获取 monitor(monitor对象存在于每个Java对象的对象头中，synchronized 锁便是通过这种方式获取锁的，也是为什么Java中任意对象可以作为锁的原因) 的持有权。当计数器为0则可以成功获取，获取后将锁计数器设为1也就是加1。相应的在执行 monitorexit 指令后，将锁计数器设为0，表明锁被释放。如果获取对象锁失败，那当前线程就要阻塞等待，直到锁被另外一个线程释放为止。 +1. **同步（Sync）**：调用者需等待结果返回，全程 “亲自参与” 任务执行。 +2. **异步（Async）** + **调用者无需等待结果**，任务交予后台处理，通过回调 / 通知获取结果。 + *例：点外卖后无需一直等餐，配送完成会电话通知。* +3. **阻塞（Block）** + **线程发起请求后被挂起**，无法执行其他操作，直到结果返回。 + *例：单线程程序中，调用`InputStream.read()`时，线程会一直等待数据可读。* +4. **非阻塞（Non-Block）** + **线程发起请求后立即返回**，可继续执行其他任务（需轮询或回调处理结果）。 + *例：多线程程序中，调用`SocketChannel.read()`时，若数据不可读立即返回`-1`，线程可处理其他通道。* -含义：（monitor 机制） +| **对比维度** | **同步 vs 异步** | **阻塞 vs 非阻塞** | +| ------------ | ------------------------------------------------------------ | -------------------------------------- | +| **核心区别** | **任务执行方式**：是否亲自处理 | **线程状态**：是否被挂起（能否干别的） | +| **典型场景** | - 同步：Servlet 单线程处理请求 - 异步：Spring `@Async` 注解 | - 阻塞：BIO 模型 - 非阻塞：NIO 模型 | +| **组合关系** | 可交叉组合，共 4 种模式： **同步阻塞（BIO）**、**同步非阻塞（NIO 轮询）**、 **异步阻塞（少见）**、**异步非阻塞（AIO）** | 无直接关联，需结合具体场景分析 | -Synchronized 是在加锁，加对象锁。对象锁是一种重量锁（monitor），synchronized 的锁机制会根据线程竞争情况在运行时会有偏向锁（单一线程）、轻量锁（多个线程访问 synchronized 区域）、对象锁（重量锁，多个线程存在竞争的情况）、自旋锁等。 +阻塞调用是指调用结果返回之前，当前线程会被挂起，一直处于等待消息通知，不能够执行其他业务。函数只有在得到结果之后才会返回 -该关键字是一个几种锁的封装。 +**有人也许会把阻塞调用和同步调用等同起来，实际上它们是不同的。** -**synchronized 关键字底层原理属于 JVM 层面。** +对于同步调用来说，很多时候当前线程可能还是激活的，只是从逻辑上当前函数没有返回而已，此时，这个线程可能也会处理其他的消息 -**① synchronized 同步语句块的情况** +1. 如果这个线程在等待当前函数返回时，仍在执行其他消息处理，那这种情况就叫做同步非阻塞； -```java -public class SynchronizedDemo { - public void method() { - synchronized (this) { - System.out.println("synchronized 代码块"); - } - } -} -``` +2. 如果这个线程在等待当前函数返回时，没有执行其他消息处理，而是处于挂起等待状态，那这种情况就叫做同步阻塞； -通过 JDK 自带的 javap 命令查看 SynchronizedDemo 类的相关字节码信息：首先切换到类的对应目录执行 `javac SynchronizedDemo.java` 命令生成编译后的 .class 文件，然后执行`javap -c -s -v -l SynchronizedDemo.class`。 +所以同步的实现方式会有两种：同步阻塞、同步非阻塞；同理，异步也会有两种实现：异步阻塞、异步非阻塞 -[![](https://camo.githubusercontent.com/78771c0f89d7076e8f70ca5c9fab40ed03f44f6b/68747470733a2f2f6d792d626c6f672d746f2d7573652e6f73732d636e2d6265696a696e672e616c6979756e63732e636f6d2f323031392d362f73796e6368726f6e697a65642545352538352542332545392539342541452545352541442539372545352538452539462545372539302538362e706e67)](https://camo.githubusercontent.com/78771c0f89d7076e8f70ca5c9fab40ed03f44f6b/68747470733a2f2f6d792d626c6f672d746f2d7573652e6f73732d636e2d6265696a696e672e616c6979756e63732e636f6d2f323031392d362f73796e6368726f6e697a65642545352538352542332545392539342541452545352541442539372545352538452539462545372539302538362e706e67) +> #### “BIO、NIO、AIO 分别属于哪种模型？” +> +> - **BIO（Blocking IO）** = 同步阻塞（最传统，线程易阻塞浪费资源）。 +> - **NIO（Non-Blocking IO）** = 同步非阻塞（通过选择器 Selector 实现线程非阻塞，需手动轮询结果）。 +> - **AIO（Asynchronous IO）** = 异步非阻塞（JDK 7 引入，后台自动完成 IO 操作，回调通知结果）。 -从上面我们可以看出： -**synchronized 同步语句块的实现使用的是 monitorenter 和 monitorexit 指令，其中 monitorenter 指令指向同步代码块的开始位置，monitorexit 指令则指明同步代码块的结束位置。** 当执行 monitorenter 指令时，线程试图获取锁也就是获取 monitor(monitor对象存在于每个 Java 对象的对象头中，synchronized 锁便是通过这种方式获取锁的，也是为什么 Java 中任意对象可以作为锁的原因) 的持有权。当计数器为0则可以成功获取，获取后将锁计数器设为1也就是加1。相应的在执行 monitorexit 指令后，将锁计数器设为0，表明锁被释放。如果获取对象锁失败，那当前线程就要阻塞等待，直到锁被另外一个线程释放为止。 -**② synchronized 修饰方法的的情况** +### 🎯 什么是线程安全和线程不安全？ -```java -public class SynchronizedDemo2 { - public synchronized void method() { - System.out.println("synchronized 方法"); - } -} -``` +通俗的说：加锁的就是线程安全的，不加锁的就是线程不安全的 -[![synchronized关键字原理](https://camo.githubusercontent.com/269441dd7da0840bc071cf70fa8162f58482a559/68747470733a2f2f6d792d626c6f672d746f2d7573652e6f73732d636e2d6265696a696e672e616c6979756e63732e636f6d2f323031392d362f73796e6368726f6e697a6564254535253835254233254539253934254145254535254144253937254535253845253946254537253930253836322e706e67)](https://camo.githubusercontent.com/269441dd7da0840bc071cf70fa8162f58482a559/68747470733a2f2f6d792d626c6f672d746f2d7573652e6f73732d636e2d6265696a696e672e616c6979756e63732e636f6d2f323031392d362f73796e6368726f6e697a6564254535253835254233254539253934254145254535254144253937254535253845253946254537253930253836322e706e67) +**线程安全: **线程安全**指的是当多个线程同时访问某个共享资源或执行某个方法时，不会引发竞态条件（race condition）等问题。线程安全的代码确保多个线程能够**安全且正确地访问资源，即无论系统的调度如何，最终的结果总是符合预期 -synchronized 修饰的方法并没有 monitorenter 指令和 monitorexit 指令，取得代之的确实是 ACC_SYNCHRONIZED 标识，该标识指明了该方法是一个同步方法，JVM 通过该 ACC_SYNCHRONIZED 访问标志来辨别一个方法是否声明为同步方法，从而执行相应的同步调用。 +> 如何实现线程安全： +> +> - **加锁机制**：常见的是通过使用锁（`synchronized`、`Lock` 等），确保同一时间只有一个线程能够访问共享资源。 +> - **原子操作**：使用原子性操作或工具类，如 Java 中的 `AtomicInteger`，可以确保线程间的操作是不可分割的，避免了竞态条件。 +> - **不可变对象**：如果数据本身是不可变的，那么它自然是线程安全的，因为任何线程都只能读取，而不会修改数据。 +**线程不安全：就是不提供数据访问保护，有可能出现多个线程先后更改数据造成所得到的数据是脏数据** +线程安全问题都是由全局变量及静态变量引起的。 若每个线程中对全局变量、静态变量只有读操作，而无写操作，一般来说，这个全局变量是线程安全的；若有多个线程同时执行写操作，一般都需要考虑线程同步，否则的话就可能影响线程安全。 -### 说说 sleep() 方法和 wait() 方法区别和共同点? -- 两者最主要的区别在于：**sleep 方法没有释放锁，而 wait 方法释放了锁** 。 -- 两者都可以暂停线程的执行。 -- Wait 通常被用于线程间交互/通信，sleep 通常被用于暂停执行。 -- wait() 方法被调用后，线程不会自动苏醒，需要别的线程调用同一个对象上的 notify() 或者 notifyAll() 方法。sleep() 方法执行完成后，线程会自动苏醒。或者可以使用 wait(long timeout)超时后线程会自动苏醒。 +### 🎯 哪些场景需要额外注意线程安全问题？ +1. **共享资源访问**：当多个线程访问同一个可变对象或变量时，需要确保线程安全，防止数据不一致。 +2. 依赖时序的操作 +3. **可变对象的并发修改**：如果一个对象的状态可以被多个线程修改，需要同步访问以避免竞态条件。 +4. **集合的并发操作**：向集合添加、删除或修改元素时，如果集合是共享的，需要使用线程安全的集合类或同步机制。 +5. **静态字段和单例模式**：静态字段和单例实例可能被多个线程访问，需要特别注意初始化和访问的线程安全。 +6. **并发数据结构操作**：使用如`ConcurrentHashMap`等并发集合时，虽然提供了更好的线程安全性，但在某些复合操作上仍需注意同步。 +7. **资源池管理**：连接池、线程池等资源池的使用，需要确保资源的分配和释放是线程安全的。 +8. **锁的使用**：在使用锁（如`synchronized`或`ReentrantLock`）时，需要避免死锁、活锁和资源耗尽等问题。 +9. **原子操作**：对于需要原子性的操作，如计数器递增，需要使用原子变量类（如`AtomicInteger`）。 +10. **可见性问题**：确保一个线程对变量的修改对其他线程是可见的，可以通过 `volatile` 关键字或 `synchronized` 块来实现。 +11. **并发异常处理**：在处理异常时，需要确保资源的释放和状态的恢复不会影响线程安全。 +12. **发布-订阅模式**：在事件驱动的架构中，事件的发布和订阅需要同步，以避免事件处理的竞态条件。 -### 为什么我们调用 start() 方法时会执行 run() 方法，为什么我们不能直接调用 run() 方法？ +在设计系统时，应该始终考虑到线程安全问题，并采用适当的同步机制和并发工具来避免这些问题。此外，编写单元测试和集成测试时，也应该考虑到多线程环境下的行为。 -这是另一个非常经典的 java 多线程面试问题，而且在面试中会经常被问到。很简单，但是很多人都会答不上来！ -new 一个 Thread，线程进入了新建状态;调用 start() 方法，会启动一个线程并使线程进入了就绪状态，当分配到时间片后就可以开始运行了。 start() 会执行线程的相应准备工作，然后自动执行 run() 方法的内容，这是真正的多线程工作。 而直接执行 run() 方法，会把 run 方法当成一个 main 线程下的普通方法去执行，并不会在某个线程中执行它，所以这并不是多线程工作。 -**总结： 调用 start 方法方可启动线程并使线程进入就绪状态，而 run 方法只是 thread 的一个普通方法调用，还是在主线程里执行。** +### 🎯 什么是上下文切换? +上下文切换（Context Switch）指的是 **CPU 从一个线程/进程切换到另一个线程/进程运行时**，保存当前执行状态并恢复另一个的执行状态的过程。 +> 多线程编程中一般线程的个数都大于 CPU 核心的个数，而一个 CPU 核心在任意时刻只能被一个线程使用，为了让这些线程都能得到有效执行，CPU 采取的策略是为每个线程分配时间片并轮转的形式。当一个线程的时间片用完的时候就会重新处于就绪状态让给其他线程使用，这个过程就属于一次上下文切换。 -### 说说线程的生命周期和状态? +这里的“上下文”包括： -Java 线程在运行的生命周期中的指定时刻只可能处于下面 6 种不同状态的其中一个状态（图源《Java 并发编程艺术》4.1.4 节）。 +- 程序计数器（PC） +- 寄存器 +- 堆栈信息 +- 内存映射等 -![img](https://my-blog-to-use.oss-cn-beijing.aliyuncs.com/19-1-29/Java%E7%BA%BF%E7%A8%8B%E7%9A%84%E7%8A%B6%E6%80%81.png) +**为什么会发生？** -线程在生命周期中并不是固定处于某一个状态而是随着代码的执行在不同状态之间切换。Java 线程状态变迁如下图所示（图源《Java 并发编程艺术》4.1.4 节）： +- 线程时间片耗尽（操作系统调度） +- 有更高优先级线程需要运行 +- 线程主动挂起（sleep、wait、IO 阻塞） +- 多核 CPU 上线程切换 -![img](https://my-blog-to-use.oss-cn-beijing.aliyuncs.com/19-1-29/Java+%E7%BA%BF%E7%A8%8B%E7%8A%B6%E6%80%81%E5%8F%98%E8%BF%81.png) +**成本与影响** -由上图可以看出：线程创建之后它将处于 **NEW（新建）** 状态，调用 `start()` 方法后开始运行，线程这时候处于 **READY（可运行）** 状态。可运行状态的线程获得了 CPU 时间片（timeslice）后就处于 **RUNNING（运行）** 状态。 +- 上下文切换不是“免费”的： + - 保存/恢复寄存器和内存信息需要时间 + - 缓存失效（Cache Miss），降低 CPU 利用率 +- **过多的上下文切换会导致性能下降**，甚至“线程切换比工作还耗时”。 +**如何减少？** +- 使用线程池，避免频繁创建/销毁线程 +- 减少锁竞争（synchronized、ReentrantLock） +- 使用无锁数据结构（CAS、Atomic 类） +- 降低线程数量（通常 ≤ CPU 核心数 * 2） -### volatile关键字 -> 谈谈你对 volatile 的理解？ -> -> 你知道 volatile 底层的实现机制吗？ -> -> volatile 变量和 atomic 变量有什么不同？ -> -> volatile 的使用场景，你能举两个例子吗？ -> -> volatile 能使得一个非原子操作变成原子操作吗？ -**理解**： +### 🎯 用户线程和守护线程有什么区别? -volatile 是 Java 虚拟机提供的轻量级的同步机制，保证了 Java 内存模型的两个特性，可见性、有序性（禁止指令重排）、不能保证原子性。 +当我们在 Java 程序中创建一个线程，它就被称为用户线程。将一个用户线程设置为守护线程的方法就是在调用 **start()**方法之前，调用对象的 `setDamon(true)` 方法。一个守护线程是在后台执行并且不会阻止 JVM 终止的 线程，守护线程的作用是为其他线程的运行提供便利服务。当没有用户线程在 运行的时候，**JVM** 关闭程序并且退出。一个守护线程创建的子线程依然是守护线程。 +守护线程的一个典型例子就是垃圾回收器。 -**场景**： -DCL 版本的单例模式就用到了volatile，因为 DCL 也不一定是线程安全的，`instance = new Singleton();`并不是一个原子操作，会分为 3 部分执行， +### 🎯 说说线程的生命周期和状态? -1. 给 instance 分配内存 -2. 调用 instance 的构造函数来初始化对象 -3. 将 instance 对象指向分配的内存空间（执行完这步 instance 就为非 null 了） +Java 线程在运行的生命周期中的指定时刻只可能处于下面 6 种不同状态的其中一个状态 -步骤 2 和 3 不存在数据依赖关系，如果虚拟机存在指令重排序优化，则步骤 2和 3 的顺序是无法确定的 +Java 通过 `Thread.State` 枚举定义了线程的**6 种状态**（JDK 1.5 后），可通过 `getState()` 方法获取，需注意与生命周期阶段的对应关系： -一句话：在需要保证原子性的场景，不要使用 volatile。 +| 状态名称 | **说明** | **对应生命周期阶段** | +| --------------- | ------------------------------------------------------------ | --------------------------- | +| `NEW` | 线程对象被创建（例如通过 `new Thread()`）之后，但在调用 `start()` 方法之前的初始状态 | 新建 | +| `RUNNABLE` | 线程调用了 `start()` 方法之后的状态。**这并不意味着线程正在CPU上执行，而是表示线程具备了运行的条件** | 就绪、运行 | +| `BLOCKED` | 阻塞状态，等待监视器锁（如 `synchronized` 锁）。 | 阻塞（同步阻塞） | +| `WAITING` | 无限等待状态，需其他线程显式唤醒（如调用 `wait()` 无超时参数）。 | 阻塞（主动阻塞 / 协作阻塞） | +| `TIMED_WAITING` | 限时等待状态，超时后自动唤醒（如 `wait(long ms)`、`sleep(long ms)`）。 | 阻塞（主动阻塞） | +| `TERMINATED` | 终止状态，线程执行完毕或异常结束（同生命周期的 “死亡”）。 | 死亡 | +线程在生命周期中并不是固定处于某一个状态而是随着代码的执行在不同状态之间切换。 +``` +┌─────────┐ start() ┌─────────────────┐ +│ NEW │ ───────────────→│ RUNNABLE │ +└─────────┘ └─────────────────┘ + ↗ ↘ + ↙ ↘ + CPU调度获取时间片 主动放弃CPU（yield()） + ↓ ↗ +┌─────────┐ run()完成 ┌─────────────────┐ +│TERMINATED←─────────────────│ RUNNING │ +└─────────┘ └─────────────────┘ + ↓ ↓ ↓ + ┌────┴────┐ ┌──┴────┐ ┌──┴────┐ + │ │ │ │ │ │ + ┌───────────▼───┐ ┌───▼─────────▼─┴───────▼────┐ + │ │ │ │ +┌─────────────────┐ ┌─────────────────┐ ┌───────────────────────┐ +│ BLOCKED │ │ WAITING │ │ TIMED_WAITING │ +│ (等待锁) │ │ (无限等待) │ │ (超时等待) │ +└─────────────────┘ └─────────────────┘ └───────────────────────┘ + ▲ ▲ ▲ + │ │ │ + │ │ │ +┌─────────────┴───┐ ┌─────────┴──────────────┐ ┌───┴──────────────────┐ +│获取synchronized │ │notify()/notifyAll() │ │时间到达或提前唤醒 │ +│锁 │ │join()线程结束 │ │notify()/notifyAll() │ +└─────────────────┘ └────────────────────────┘ └───────────────────────┘ +``` -**原理**： -volatile 可以保证线程可见性且提供了一定的有序性，但是无法保证原子性。在 JVM 底层是基于内存屏障实现的。 -- 当对非 volatile 变量进行读写的时候，每个线程先从内存拷贝变量到 CPU 缓存中。如果计算机有多个CPU，每个线程可能在不同的 CPU 上被处理，这意味着每个线程可以拷贝到不同的 CPU cache 中 -- 而声明变量是 volatile 的，JVM 保证了每次读变量都从内存中读，跳过 CPU cache 这一步，所以就不会有可见性问题 - - 对 volatile 变量进行写操作时，会在写操作后加一条 store 屏障指令，将工作内存中的共享变量刷新回主内存； - - 对 volatile 变量进行读操作时，会在写操作后加一条 load 屏障指令，从主内存中读取共享变量； +### 🎯 一个线程两次调用 start() 方法会出现什么情况？谈谈线程的生命周期和状态转移 +在 Java 中，线程对象一旦启动，不能再次启动。如果尝试对同一个线程对象调用两次 `start()` 方法，会抛出 `java.lang.IllegalThreadStateException` 异常。 +- 当线程对象第一次调用 `start()` 时，线程从 **NEW 状态** 进入 **RUNNABLE 状态**，JVM 会为其创建对应的操作系统线程并执行 `run()` 方法。 -**性能**： +- **若再次调用 `start()`，会抛出 `IllegalThreadStateException`**，因为线程状态已不再是 NEW。 -volatile 的读性能消耗与普通变量几乎相同，但是写操作稍慢，因为它需要在本地代码中插入许多内存屏障指令来保证处理器不发生乱序执行。 + ```java + Thread t = new Thread(() -> System.out.println("Running")); + t.start(); // 第一次调用，正常启动 + t.start(); // 第二次调用，抛出 IllegalThreadStateException + ``` ------- + **状态流转的关键限制** +- **NEW → RUNNABLE**：只能通过 **一次 `start()` 调用** 触发。 +- **RUNNABLE → 其他状态**：可通过锁竞争、等待 / 通知、超时等操作转换。 +- **TERMINATED**：一旦进入，无法回到其他状态（线程生命周期结束）。 -## JMM篇 -> 谈谈 Java 内存模型 -> -> 指令重排 +> #### 线程池如何复用线程？ > -> 内存屏障 +> 线程池通过 `execute(Runnable task)` 方法复用线程，其核心原理是： > -> 单核CPU有可见性问题吗 +> 1. **Worker 线程循环**：线程池中的工作线程（Worker）会持续从任务队列中获取任务并执行。 +> 2. **任务替换**：当一个任务执行完毕后，Worker 不会终止，而是继续执行下一个任务。 +> 3. **状态维护**：Worker 线程本身不会被重复 `start()`，而是通过 `run()` 方法的循环调用实现复用。 -Java虚拟机规范中试图定义一种「 **Java 内存模型**」来**屏蔽掉各种硬件和操作系统的内存访问差异**，以实现**让 Java 程序在各种平台下都能达到一致的内存访问效果** -**JMM组成**： -- 主内存：Java 内存模型规定了所有变量都存储在主内存中（此处的主内存与物理硬件的主内存 RAM 名字一样，两者可以互相类比，但此处仅是虚拟机内存的一部分）。 +### 🎯 说说 sleep() 方法和 wait() 方法区别和共同点? -- 工作内存：每条线程都有自己的工作内存，线程的工作内存中保存了该线程使用到的主内存中的共享变量的副本拷贝。**线程对变量的所有操作都必须在工作内存进行，而不能直接读写主内存中的变量**。**工作内存是 JMM 的一个抽象概念，并不真实存在**。 +sleep () 和 wait () 的核心区别在于锁的处理机制： -**特性**： +1. **锁释放**：sleep () 不释放锁，wait () 释放锁并进入等待队列； +2. **唤醒方式**：sleep () 依赖时间或中断，wait () 依赖其他线程通知； +3. **使用场景**：sleep () 用于线程暂停，wait () 用于线程协作（wait 方法必须在 synchronized 保护的代码中使用）。 -JMM 就是用来解决如上问题的。 **JMM是围绕着并发过程中如何处理可见性、原子性和有序性这 3 个 特征建立起来的** -- **可见性**：可见性是指当一个线程修改了共享变量的值，其他线程能够立即得知这个修改。Java 中的 volatile、synchronzied、final 都可以实现可见性 -- **原子性**：即一个操作或者多个操作，要么全部执行并且执行的过程不会被任何因素打断，要么就都不执行。即使在多个线程一起执行的时候，一个操作一旦开始，就不会被其他线程所干扰。 +- `wait ()`通常被用于线程间交互/通信（wait 方法必须在 synchronized 保护的代码中使用），sleep 通常被用于暂停执行。 +- `wait()` 方法被调用后，线程不会自动苏醒，需要别的线程调用同一个对象上的 `notify()` 或者 `notifyAll()` 方法。`sleep()` 方法执行完成后，线程会自动苏醒。或者可以使用 `wait(long timeout)` 超时后线程会自动苏醒。 -- **有序性**： +> `Thread.yield()` 方法用于提示调度器当前线程愿意放弃对处理器的占用，并允许其他同优先级的线程运行。 +> +>yield() 方法和 sleep() 方法类似，也不会释放“锁标志”，区别在于，它没有参数，即 yield() 方法只是使当前线程重新回到可执行状态，所以执行 yield() 的线程有可能在进入到可执行状态后马上又被执行，另外 yield() 方法只能使同优先级或者高优先级的线程得到执行机会，这也和 sleep() 方法不同。 +> +> `Thread.join()` 方法用于等待当前线程执行完毕。它可以用于确保某个线程在另一个线程完成之前不会继续执行。 - 计算机在执行程序时，为了提高性能，编译器和处理器常常会对指令做重排，一般分为以下 3 种 - ![](https://tva1.sinaimg.cn/large/00831rSTly1gcrgrycnj0j31bs04k74y.jpg) - 单线程环境里确保程序最终执行结果和代码顺序执行的结果一致； +### 🎯 为什么 wait () 必须在 synchronized 块中？ - 处理器在进行重排序时必须要考虑指令之间的**数据依赖性**； +- **原子性保障**：避免线程安全问题（如生产者修改队列后，消费者未及时感知）。 - 多线程环境中线程交替执行，由于编译器优化重排的存在，两个线程中使用的变量能否保证一致性是无法确定的，结果无法预测 +- **JVM 实现机制**：锁对象的 `monitor` 记录等待线程，需通过 `synchronized` 获取锁后才能操作 `monitor` + +### 🎯 为什么我们调用 start() 方法时会执行 run() 方法，为什么我们不能直接调用 run() 方法？ -JMM是不区分JVM到底是运行在单核处理器、多核处理器的，Java内存模型是对CPU内存模型的抽象，这是一个High-Level的概念，与具体的CPU平台没啥关系 +调用 `start()` 方法最终会导致在新的执行路径上执行 `run()` 方法中的代码，这是 Java 实现多线程的标准方式。直接调用 `run()` 方法通常是一个错误，原因在于两者在行为、线程生命周期和底层执行机制上存在根本区别： +1. **`start()` 的本质：创建新执行流** + - **核心职责：** `start()` 方法是 `Thread` 类提供的，用于**请求 Java 虚拟机 (JVM) 启动一个新的操作系统线程（或映射到内核调度实体）**。 + - 底层机制：当调用 `start()`时： + - JVM底层会通过一个 `native` 方法（通常是 `start0()`）与操作系统交互，请求创建一个新的系统线程。 + - 这个新创建的系统线程获得独立的执行上下文（包括程序计数器、栈空间）。 + - **在操作系统准备好并调度这个新线程之后，由操作系统（或 JVM 线程调度器）自动调用该线程对象的 `run()` 方法。** + - **结果：** `run()` 方法中的代码会在一个**全新的、独立的执行线程**中运行，实现真正的并发或并行。 +2. **直接调用 `run()`：普通方法调用（非多线程）** + - **行为：** 直接调用 `run()` 方法，就像调用任何其他 Java 类的普通实例方法一样。 + - **执行上下文：** `run()` 方法会在**当前调用它的线程**中执行（例如，很可能是在 `main` 线程中执行）。 + - 结果： + - **不会创建新的线程！** + - `run()` 方法中的代码在当前线程的栈帧中同步执行（按顺序执行，阻塞当前线程）。 + - 完全丧失了多线程的意义，等同于单线程顺序执行。 +> - **`start()` 是线程的 “出生证”，JVM 见它才开新线程；`run()` 是线程的 “工作内容”，直接调用只是普通方法**。 +> - **状态流转有规则，`NEW` 到 `RUNNABLE` 靠 `start()`，跳过它线程无法活**。 -happens-before 先行发生，是 Java 内存模型中定义的两项操作之间的偏序关系，**如果操作A 先行发生于操作B，那么A的结果对B可见**。 +### 🎯 Java 线程启动的几种方式 -内存屏障是被插入两个 CPU 指令之间的一种指令，用来禁止处理器指令发生重排序（像屏障一样），从而保障**有序性**的。 +```java +public static void main(String[] args) { + new MyThread().start(); //第一种 直接通过Thread MyThread 是继承了Thread对象的类 实现在下面 + + new Thread(new MyRun()).start(); //第二种 Runnable + new Thread(()->{ //第三种 lambda + System.out.println("Hello Lambda!"); + }).start(); + + Thread t = new Thread(new FutureTask(new MyCall())); //第四种 + t.start(); + + ExecutorService service = Executors.newCachedThreadPool(); //第五种 使用Executor + service.execute(()->{ + System.out.println("Hello ThreadPool"); + }); + service.shutdown(); +} +} +``` ------- +### 🎯 如何正确停止线程？ +通常情况下，我们不会手动停止一个线程，而是允许线程运行到结束，然后让它自然停止。但是依然会有许多特殊的情况需要我们提前停止线程，比如：用户突然关闭程序，或程序运行出错重启等。 +在 Java 中，正确停止线程通常涉及到线程的协作和适当的关闭机制。由于Java没有提供直接停止线程的方法（如`stop()`方法已经被废弃，因为它太危险，容易造成数据不一致等问题），以下是一些常见的正确停止线程的方法： -## Atomic~CAS +1. **使用标志位**：使用标志位是停止线程的常见方法。在这种方法中，线程会定期检查一个标志位，如果标志位指示线程应该停止，那么线程会自行结束。 -> CAS 知道吗，如何实现？ -> 讲一讲AtomicInteger，为什么要用 CAS 而不是 synchronized？ -> CAS 底层原理，谈谈你对 UnSafe 的理解？ -> AtomicInteger 的ABA问题，能说一下吗，原子更新引用知道吗？ -> CAS 有什么缺点吗？ 如何规避 ABA 问题？ + ```java + private volatile boolean stopRunning = false; + + public void stopThread() { + this.stopRunning = true; + } + + public void run() { + while (!stopRunning) { + // 执行任务 + } + // 清理资源 + } + ``` -Java 虚拟机又提供了一个轻量级的同步机制——volatile，但是 volatile 算是乞丐版的 synchronized，并不能保证原子性 ，所以，又增加了`java.util.concurrent.atomic`包， 这个包下提供了一系列原子类。 +2. **中断状态（Interruption）**：使用线程的中断机制来优雅地停止线程。当需要停止线程时，调用`Thread.interrupt()`方法；在线程的执行过程中，检查中断状态，如果被中断，则退出。 + ```JAVA + // 在其他线程中调用此方法来中断线程 + thread.interrupt(); + + // 在目标线程中检查中断状态 + public void run() { + try { + while (!Thread.currentThread().isInterrupted()) { + // 执行任务 + } + } catch (InterruptedException e) { + // 线程被中断，可以选择重置中断状态或退出 + Thread.currentThread().interrupt(); + } finally { + // 清理资源 + } + } + ``` +3. **使用ExecutorService**：使用`ExecutorService`可以更容易地控制线程的生命周期。调用`shutdown()`方法开始关闭，调用`shutdownNow()`可以尝试立即停止所有正在执行的任务。 -**Atomic**： + ```java + ExecutorService executorService = Executors.newSingleThreadExecutor(); + Future future = executorService.submit(() -> { + // 执行任务 + }); + + // 请求关闭线程池，不再接受新任务，尝试完成已提交的任务 + executorService.shutdown(); + + // 尝试立即停止所有正在执行的任务列表，返回未完成的任务列表 + List notCompleted = executorService.shutdownNow(); + + // 等待线程池关闭，直到所有任务完成后 + executorService.awaitTermination(60, TimeUnit.SECONDS); + ``` -AtomicBoolean、AtomicInteger、tomicIntegerArray、AtomicReference、AtomicStampedReference + **使用 `Future.cancel()`**： `ExecutorService` 启动的话，也可以使用 `Future.cancel()` 方法来停止线程。 -常用方法： -addAndGet(int)、getAndIncrement()、compareAndSet(int, int) -**CAS**: +### 🎯 进程间的通信方式？ -- CAS：全称 `Compare and swap`，即**比较并交换**，它是一条 **CPU 同步原语**。 是一种硬件对并发的支持，针对多处理器操作而设计的一种特殊指令，用于管理对共享数据的并发访问。 -- CAS 是一种无锁的非阻塞算法的实现。 -- CAS 包含了 3 个操作数： - - 需要读写的内存值 V - - 旧的预期值 A - - 要修改的更新值 B -- 当且仅当 V 的值等于 A 时，CAS 通过原子方式用新值 B 来更新 V 的 值，否则不会执行任何操作（他的功能是判断内存某个位置的值是否为预期值，如果是则更改为新的值，这个过程是原子的。） -- 缺点 - - 循环时间长，开销很大 - - 只能保证一个共享变量的原子操作 - - ABA 问题（用 AtomicReference 避免） +进程间通信（Inter-Process Communication，IPC）是指在不同进程之间传递数据和信息的机制。不同操作系统提供了多种 IPC 方式，下面是常见的几种： +1. **管道（Pipe）** + - 无名管道（Anonymous Pipe）：单向通信，只能用于有亲缘关系的进程（父进程与子进程）。 -**Unsafe**： + - 有名管道（Named Pipe 或 FIFO）：在 Unix/Linux 系统中，可以使用 `mkfifo` 命令创建有名管道，支持双向通信。 -CAS 并发原语体现在 Java 语言中的 `sum.misc.Unsafe` 类中的各个方法。调用 Unsafe 类中的 CAS 方法， JVM 会帮助我们实现出 CAS 汇编指令。 +2. 消息队列（Message Queue） -是 CAS 的核心类，由于 Java 方法无法直接访问底层系统，需要通过本地（native）方法来访问，UnSafe 相当于一个后门，UnSafe 类中的所有方法都是 native 修饰的，也就是说该类中的方法都是直接调用操作系统底层资源执行相应任务。 + - **特点**：消息队列是存储在内核中的消息链表，允许进程通过发送和接收消息进行通信，支持有序的消息传递。 + - **实现**：在 Unix/Linux 系统中，可以使用 `msgget`、`msgsnd`、`msgrcv` 等系统调用进行消息队列的创建和操作。 +3. 共享内存（Shared Memory） -## 队列 + - **特点**：多个进程可以直接访问同一块内存区域，是最快的 IPC 方式之一，但需要同步机制来避免数据竞争。 + - **实现**：在 Unix/Linux 系统中，可以使用 `shmget`、`shmat`、`shmdt`、`shmctl` 等系统调用进行共享内存的创建和管理。 +4. 信号（Signal） ------- + - **特点**：信号是一种异步通信机制，用于通知进程某个事件已经发生。常用于进程间的简单通知。 + - **实现**：在 Unix/Linux 系统中，可以使用 `kill` 系统调用发送信号，使用 `signal` 或 `sigaction` 设置信号处理程序。 +5. 信号量（Semaphore） + - **特点**：信号量是一种用于进程间同步的计数器，可以用来控制多个进程对共享资源的访问。 + - **实现**：在 Unix/Linux 系统中，可以使用 `semget`、`semop`、`semctl` 等系统调用进行信号量的创建和操作。 -### 什么是线程死锁?如何避免死锁? +6. 套接字（Socket） -#### 8.1. 认识线程死锁 + - **特点**：套接字是一种底层的网络通信机制，可以用于同一台机器上不同进程之间的通信，也可以用于不同机器上的进程之间的通信，支持双向通信。 -线程死锁描述的是这样一种情况：多个线程同时被阻塞，它们中的一个或者全部都在等待某个资源被释放。由于线程被无限期地阻塞，因此程序不可能正常终止。 + - **实现**：在 Unix/Linux 系统中，可以使用 `socket`、`bind`、`listen`、`accept`、`connect`、`send`、`recv` 等系统调用进行套接字编程。 -如下图所示，线程 A 持有资源 2，线程 B 持有资源 1，他们同时都想申请对方的资源，所以这两个线程就会互相等待而进入死锁状态。 +7. 文件（File） -![img](https://my-blog-to-use.oss-cn-beijing.aliyuncs.com/2019-4/2019-4%E6%AD%BB%E9%94%811.png) + - **特点**：通过读写共享文件的方式进行通信，适用于数据量较大且不要求高实时性的场景。 -下面通过一个例子来说明线程死锁,代码模拟了上图的死锁的情况 (代码来源于《并发编程之美》)： + - **实现**：在所有操作系统中，进程可以通过标准的文件读写操作进行通信。 -```JAVA -public class DeadLockDemo { - private static Object resource1 = new Object();//资源 1 - private static Object resource2 = new Object();//资源 2 +8. 内存映射文件（Memory-Mapped File） - public static void main(String[] args) { - new Thread(() -> { - synchronized (resource1) { - System.out.println(Thread.currentThread() + "get resource1"); - try { - Thread.sleep(1000); - } catch (InterruptedException e) { - e.printStackTrace(); - } - System.out.println(Thread.currentThread() + "waiting get resource2"); - synchronized (resource2) { - System.out.println(Thread.currentThread() + "get resource2"); - } - } - }, "线程 1").start(); + - **特点**：通过将文件映射到进程的地址空间，实现进程间的共享内存，适用于大数据量的共享。 - new Thread(() -> { - synchronized (resource2) { - System.out.println(Thread.currentThread() + "get resource2"); - try { - Thread.sleep(1000); - } catch (InterruptedException e) { - e.printStackTrace(); - } - System.out.println(Thread.currentThread() + "waiting get resource1"); - synchronized (resource1) { - System.out.println(Thread.currentThread() + "get resource1"); - } - } - }, "线程 2").start(); - } -} -Thread[线程 1,5,main]get resource1 -Thread[线程 2,5,main]get resource2 -Thread[线程 1,5,main]waiting get resource2 -Thread[线程 2,5,main]waiting get resource1 - new Thread(() -> { - synchronized (resource1) { - System.out.println(Thread.currentThread() + "get resource1"); - try { - Thread.sleep(1000); - } catch (InterruptedException e) { - e.printStackTrace(); - } - System.out.println(Thread.currentThread() + "waiting get resource2"); - synchronized (resource2) { - System.out.println(Thread.currentThread() + "get resource2"); - } - } - }, "线程 2").start(); -``` + - **实现**：在 Unix/Linux 系统中，可以使用 `mmap` 系统调用创建内存映射文件。 -Output -``` -Thread[线程 1,5,main]get resource1 -Thread[线程 2,5,main]get resource2 -Thread[线程 1,5,main]waiting get resource2 -Thread[线程 2,5,main]waiting get resource1 -``` -线程 A 通过 synchronized (resource1) 获得 resource1 的监视器锁，然后通过`Thread.sleep(1000);`让线程 A 休眠 1s 为的是让线程 B 得到执行然后获取到 resource2 的监视器锁。线程 A 和线程 B 休眠结束了都开始企图请求获取对方的资源，然后这两个线程就会陷入互相等待的状态，这也就产生了死锁。上面的例子符合产生死锁的四个必要条件。学过操作系统的朋友都知道产生死锁必须具备以下四个条件： +### 🎯 Java 多线程之间的通信方式？ -- 互斥条件：该资源任意一个时刻只由一个线程占用。 -- 请求与保持条件：一个进程因请求资源而阻塞时，对已获得的资源保持不放。 -- 不剥夺条件:线程已获得的资源在末使用完之前不能被其他线程强行剥夺，只有自己使用完毕后才释放资源 -- 循环等待条件:若干进程之间形成一种头尾相接的循环等待资源关系。 +| **通信方式** | **核心机制** | **适用场景** | **关键类 / 方法** | +| -------------------- | -------------------------------------------- | ----------------------------------- | --------------------------------------- | +| 共享变量 | `volatile`/`synchronized` 保证可见性和原子性 | 简单状态通知和数据共享 | `volatile` 关键字、`synchronized` 块 | +| `wait()`/`notify()` | 对象监视器 + 等待 / 通知机制 | 条件等待和唤醒（如生产者 - 消费者） | `Object.wait()`、`Object.notify()` | +| `Lock` + `Condition` | 显式锁 + 多路等待队列 | 复杂同步逻辑（如多条件等待） | `ReentrantLock`、`Condition.await()` | +| `BlockingQueue` | 阻塞队列实现线程安全的入队 / 出队 | 生产者 - 消费者模型简化 | `ArrayBlockingQueue`、`put()`、`take()` | +| `CountDownLatch` | 倒计时等待多个线程完成 | 主线程等待子线程集合 | `CountDownLatch.await()`、`countDown()` | +| `CyclicBarrier` | 线程集合后同步执行 | 多轮协作（如多阶段计算） | `CyclicBarrier.await()` | +| `Exchanger` | 两个线程间数据交换 | 遗传算法、管道设计等一对一交换场景 | `Exchanger.exchange()` | -#### 8.2. 如何避免线程死锁? -我上面说了产生死锁的四个必要条件，为了避免死锁，我们只要破坏产生死锁的四个条件中的其中一个就可以了。现在我们来挨个分析一下： -1. **破坏互斥条件** ：这个条件我们没有办法破坏，因为我们用锁本来就是想让他们互斥的（临界资源需要互斥访问）。 -2. **破坏请求与保持条件** ：一次性申请所有的资源。 -3. **破坏不剥夺条件** ：占用部分资源的线程进一步申请其他资源时，如果申请不到，可以主动释放它占有的资源。 -4. **破坏循环等待条件** ：靠按序申请资源来预防。按某一顺序申请资源，释放资源则反序释放。破坏循环等待条件。 +### 🎯 Java 同步机制有哪些？ -我们对线程 2 的代码修改成下面这样就不会产生死锁了。 +1. `synchronized` 关键字，这个相信大家很了解，最好能理解其中的原理 -```java - new Thread(() -> { - synchronized (resource1) { - System.out.println(Thread.currentThread() + "get resource1"); - try { - Thread.sleep(1000); - } catch (InterruptedException e) { - e.printStackTrace(); - } - System.out.println(Thread.currentThread() + "waiting get resource2"); - synchronized (resource2) { - System.out.println(Thread.currentThread() + "get resource2"); - } - } - }, "线程 2").start(); -``` -Output +2. `Lock` 接口及其实现类，如 ReentrantLock.ReadLock 和 ReentrantReadWriteLock.WriteLock -``` -Thread[线程 1,5,main]get resource1 -Thread[线程 1,5,main]waiting get resource2 -Thread[线程 1,5,main]get resource2 -Thread[线程 2,5,main]get resource1 -Thread[线程 2,5,main]waiting get resource2 -Thread[线程 2,5,main]get resource2 -Process finished with exit code 0 -``` +3. `Semaphore`：是一种计数器，用来保护一个或者多个共享资源的访问，它是并发编程的一种基础工具，大多数编程语言都提供这个机制，这也是操作系统中经常提到的 +4. `CountDownLatch`：是 Java 语言提供的同步辅助类，在完成一组正在其他线程中执行的操作之前，他允许线程一直等待 +5. `CyclicBarrier`：也是 java 语言提供的同步辅助类，他允许多个线程在某一个集合点处进行相互等待； +6. `Phaser`：也是 java 语言提供的同步辅助类，他把并发任务分成多个阶段运行，在开始下一阶段之前，当前阶段中所有的线程都必须执行完成，JAVA7 才有的特性。 +7. `Exchanger`：他提供了两个线程之间的数据交换点。 +8. `StampedLock` ：是一种改进的读写锁，提供了三种模式：写锁、悲观读锁和乐观读锁，适用于读多写少的场景。 -我们分析一下上面的代码为什么避免了死锁的发生? +------ -线程 1 首先获得到 resource1 的监视器锁,这时候线程 2 就获取不到了。然后线程 1 再去获取 resource2 的监视器锁，可以获取到。然后线程 1 释放了对 resource1、resource2 的监视器锁的占用，线程 2 获取到就可以执行了。这样就破坏了破坏循环等待条件，因此避免了死锁。 +## 二、同步关键字（并发控制）🔒 ------- +### 🎯 synchronized 关键字? + +> "synchronized是Java最基础的同步机制，基于Monitor监视器实现： +> +> **实现原理**： +> +> - 同步代码块：使用monitorenter和monitorexit字节码指令 +> - 同步方法：使用ACC_SYNCHRONIZED访问标志 +> - 基于对象头的Mark Word存储锁信息 +> +> **锁升级过程**： +> +> 1. **偏向锁**：只有一个线程访问时，在对象头记录线程ID +> 2. **轻量级锁**：多线程竞争但无实际冲突，使用CAS操作 +> 3. **重量级锁**：存在真正竞争时，升级为Monitor锁，线程阻塞 +> +> **特点**： +> +> - 可重入性：同一线程可以多次获得同一把锁 +> - 不可中断：等待锁的线程不能被中断 +> - 非公平锁：无法保证等待时间最长的线程优先获得锁 +> +> JDK 6+的锁优化使synchronized性能大幅提升，在低竞争场景下甚至超过ReentrantLock。" +**1. 底层实现原理** +`synchronized` 的底层实现基于 **JVM 监视器锁（Monitor）** 机制： -## ThreadLocal +- **同步代码块**：通过字节码指令 `monitorenter`（加锁）和 `monitorexit`（释放锁）实现 +- **同步方法**：通过方法访问标志 `ACC_SYNCHRONIZED` 隐式实现锁机制 +- **核心数据结构**：每个对象关联一个 Monitor（包含 Owner 线程、EntryList 阻塞队列、WaitSet 等待队列） -当使用 ThreadLocal 维护变量时，其为每个使用该变量的线程提供独立的变量副本，所以每一个线程都可以独立的改变自己的副本，而不会影响其他线程对应的副本。 +> 📌 **关键点**：所有 Java 对象天生自带 Monitor，这是 `synchronized` 能以任意对象作为锁的根本原因。 +> +> **synchronized 关键字底层原理属于 JVM 层面。** +> +> **① synchronized 同步语句块的情况** +> +> ```java +> public class SynchronizedDemo { +> public void method() { +> synchronized (this) { +> System.out.println("synchronized 代码块"); +> } +> } +> } +> ``` +> +> 通过 JDK 自带的 javap 命令查看 SynchronizedDemo 类的相关字节码信息：首先切换到类的对应目录执行 `javac SynchronizedDemo.java` 命令生成编译后的 .class 文件，然后执行`javap -c -s -v -l SynchronizedDemo.class`。 +> +> **synchronized 同步语句块的实现使用的是 monitorenter 和 monitorexit 指令，其中 monitorenter 指令指向同步代码块的开始位置，monitorexit 指令则指明同步代码块的结束位置。** 当执行 monitorenter 指令时，线程试图获取锁也就是获取 monitor(monitor对象存在于每个 Java 对象的对象头中，synchronized 锁便是通过这种方式获取锁的，也是为什么 Java 中任意对象可以作为锁的原因) 的持有权。当计数器为0则可以成功获取，获取后将锁计数器设为1也就是加1。相应的在执行 monitorexit 指令后，将锁计数器设为0，表明锁被释放。如果获取对象锁失败，那当前线程就要阻塞等待，直到锁被另外一个线程释放为止。 +> +> 1. 进入时，执行 monitorenter，将计数器 +1，释放锁 monitorexit 时，计数器-1； +> 2. 当一个线程判断到计数器为 0 时，则当前锁空闲，可以占用；反之，当前线程进入等待状态。 +> +> **② synchronized 修饰方法的的情况** +> +> ```java +> public class SynchronizedDemo2 { +> public synchronized void method() { +> System.out.println("synchronized 方法"); +> } +> } +> ``` +> +> synchronized 修饰的方法并没有 monitorenter 指令和 monitorexit 指令，取得代之的确实是 `ACC_SYNCHRONIZED` 标识，该标识指明了该方法是一个同步方法，JVM 通过该 ACC_SYNCHRONIZED 访问标志来辨别一个方法是否声明为同步方法，从而执行相应的同步调用。 +> +> **为什么方法的同步使用 `ACC_SYNCHRONIZED`？** +> +> 1. **简化字节码** +> - 同步方法的范围天然固定为整个方法体，直接用标志位表示更加简洁。 +> - 避免了显式插入指令的额外开销。 +> 2. **由 JVM 执行优化** +> - JVM 可以直接识别 `ACC_SYNCHRONIZED` 并在方法调用层面加锁，而无需用户手动控制。 +> - 更容易结合其他锁优化（如偏向锁、轻量级锁） + +**2. 使用方式与区别** + +| **使用方式** | **锁对象** | **作用范围** | **特点** | | +| ---------------- | ------------------------------- | ------------ | -------------------------- | ----------------------------- | +| **同步实例方法** | `this`（当前对象实例） | 整个方法体 | 影响同一对象的所有同步方法 | | +| **同步静态方法** | `Class` 对象（如 `User.class`） | 整个方法体 | 全局锁，影响所有实例的调用 | | +| **同步代码块** | 指定任意对象 | 代码块内部 | 锁粒度最小，性能最优 | `synchronized (lock) { ... }` | + +> ⚠️ **注意**：对类加锁（静态同步）时，**调用非静态方法不会加锁**（因为锁对象不同）： +> +> ```java +> class User { +> public static synchronized void staticMethod() {} // 锁User.class +> public synchronized void instanceMethod() {} // 锁this实例 +> } +> ``` -ThreadLocal 内部实现机制： +**3. 对象头与 Monitor** -- 每个线程内部都会维护一个类似 HashMap 的对象，称为 ThreadLocalMap，里边会包含若干了 Entry（K-V 键值对），相应的线程被称为这些 Entry 的属主线程； -- Entry 的 Key 是一个 ThreadLocal 实例，Value 是一个线程特有对象。Entry 的作用即是：为其属主线程建立起一个 ThreadLocal 实例与一个线程特有对象之间的对应关系； -- Entry 对 Key 的引用是弱引用；Entry 对 Value 的引用是强引用。 +- **对象头（Object Header）**： + 每个 Java 对象在内存中都有对象头，包含 **Mark Word** 和 **Klass Pointer**。 + - **Mark Word**：存储对象的哈希码、GC 分代年龄、锁状态等信息。 + - **Class Pointer**：指向对象所属类的元数据。 +- **Monitor（监视器）**： + 每个对象都关联一个 Monitor，本质是操作系统的互斥量（Mutex），包含： + - **Owner**：记录当前持有锁的线程。 + - **Entry List**：等待获取锁的线程队列。 + - **Wait Set**：调用 `wait()` 后阻塞的线程队列。 + +**4. 锁升级（JDK 1.6+ 优化）** + +JVM 为减少锁竞争的性能开销，引入了**锁升级机制**： +**无锁 → 偏向锁 → 轻量级锁 → 重量级锁**（状态不可逆） + +对象头中包含了锁标志位（Lock Word），用于表示对象的锁状态。 +`Mark Word` 在锁的不同状态下会有不同的含义： +- **无锁（Normal）**：存储对象的哈希值。指没有对资源进行锁定，所有的线程都能访问并修改同一个资源，但同时只有一个线程能修改成功。 -### 谈谈 synchronized和ReentrantLock 的区别 +- **偏向锁（Biased Lock）**：存储线程 ID，表示锁倾向于某个线程。 -**① 两者都是可重入锁** + - **适用场景**：单线程多次获取同一锁。 + - **原理**：首次获取锁时，JVM 在对象头 Mark Word 中存储线程 ID（CAS 操作），后续该线程直接获取锁，无需同步开销。 + - **升级条件**：当其他线程尝试获取锁时，偏向锁失效，升级为轻量级锁。 -两者都是可重入锁。“可重入锁”概念是：自己可以再次获取自己的内部锁。比如一个线程获得了某个对象的锁，此时这个对象锁还没有释放，当其再次想要获取这个对象的锁的时候还是可以获取的，如果不可锁重入的话，就会造成死锁。同一个线程每次获取锁，锁的计数器都自增1，所以要等到锁的计数器下降为0时才能释放锁。 +- **轻量级锁（Lightweight Lock）**（也叫自旋锁）：存储指向锁记录的指针。 -**② synchronized 依赖于 JVM 而 ReentrantLock 依赖于 API** + - **适用场景**：多线程交替执行，无锁竞争。 + - **原理**：线程获取锁时，JVM 在当前线程栈帧中创建锁记录（Lock Record），通过 CAS 将 Mark Word 指向锁记录。若成功则获取锁，失败则升级为重量级锁。 + - **特点**：未获取锁的线程**自旋等待**，避免线程阻塞和唤醒的开销。长时间的自旋操作是非常消耗资源的，一个线程持有锁，其他线程就只能在原地空耗CPU，执行不了任何有效的任务，这种现象叫做忙等（busy-waiting） -synchronized 是依赖于 JVM 实现的，前面我们也讲到了 虚拟机团队在 JDK1.6 为 synchronized 关键字进行了很多优化，但是这些优化都是在虚拟机层面实现的，并没有直接暴露给我们。ReentrantLock 是 JDK 层面实现的（也就是 API 层面，需要 lock() 和 unlock() 方法配合 try/finally 语句块来完成），所以我们可以通过查看它的源代码，来看它是如何实现的。 +- **重量级锁（Heavyweight Lock）**：存储指向 Monitor 对象的指针。 -**③ ReentrantLock 比 synchronized 增加了一些高级功能** + - **适用场景**：多线程竞争激烈。如果锁竞争情况严重，某个达到最大自旋次数的线程，会将轻量级锁升级为重量级锁（依然是CAS修改锁标志位，但不修改持有锁的线程ID）。当后续线程尝试获取锁时，发现被占用的锁是重量级锁，则直接将自己挂起（而不是忙等），等待将来被唤醒。 -相比synchronized，ReentrantLock增加了一些高级功能。主要来说主要有三点：**①等待可中断；②可实现公平锁；③可实现选择性通知（锁可以绑定多个条件）** + - **原理**：依赖操作系统的互斥量（Mutex），未获取锁的线程**进入内核态阻塞**，锁释放后需唤醒线程（性能开销大）。 -- **ReentrantLock提供了一种能够中断等待锁的线程的机制**，通过lock.lockInterruptibly()来实现这个机制。也就是说正在等待的线程可以选择放弃等待，改为处理其他事情。 -- **ReentrantLock可以指定是公平锁还是非公平锁。而synchronized只能是非公平锁。所谓的公平锁就是先等待的线程先获得锁。** ReentrantLock默认情况是非公平的，可以通过 ReentrantLock类的`ReentrantLock(boolean fair)`构造方法来制定是否是公平的。 -- synchronized关键字与wait()和notify()/notifyAll()方法相结合可以实现等待/通知机制，ReentrantLock类当然也可以实现，但是需要借助于Condition接口与newCondition() 方法。Condition是JDK1.5之后才有的，它具有很好的灵活性，比如可以实现多路通知功能也就是在一个Lock对象中可以创建多个Condition实例（即对象监视器），**线程对象可以注册在指定的Condition中，从而可以有选择性的进行线程通知，在调度线程上更加灵活。 在使用notify()/notifyAll()方法进行通知时，被通知的线程是由 JVM 选择的，用ReentrantLock类结合Condition实例可以实现“选择性通知”** ，这个功能非常重要，而且是Condition接口默认提供的。而synchronized关键字就相当于整个Lock对象中只有一个Condition实例，所有的线程都注册在它一个身上。如果执行notifyAll()方法的话就会通知所有处于等待状态的线程这样会造成很大的效率问题，而Condition实例的signalAll()方法 只会唤醒注册在该Condition实例中的所有等待线程。 + - 重量级锁是指当有一个线程获取锁之后，其余所有等待获取该锁的线程都会处于阻塞状态。 -如果你想使用上述功能，那么选择ReentrantLock是一个不错的选择。 + 简言之，就是所有的控制权都交给了操作系统，由操作系统来负责线程间的调度和线程的状态变更。而这样会出现频繁地对线程运行状态的切换，线程的挂起和唤醒，从而消耗大量的系统资源 -**④ 性能已不是选择标准** +### 🎯 volatile关键字？ +> 谈谈你对 volatile 的理解？ +> +> 你知道 volatile 底层的实现机制吗？ +> +> volatile 变量和 atomic 变量有什么不同？ +> +> volatile 的使用场景，你能举两个例子吗？ +> +> volatile 能使得一个非原子操作变成原子操作吗？ +**volatile是什么？** -### synchronized 和 Lock 区别 +在谈及线程安全时，常会说到一个变量——volatile。在《Java并发编程实战》一书中是这么定义volatile的——“Java语言提供了一种稍弱的同步机制，即volatile变量，用来确保将变量的更新操作通知到其他线程”。 -原始构成 +这句话说明了两点： -- synchronized 是关键字属于JVM 层面 - - monitorenter(底层是通过monitor对象完成，其实 wait/notify等方法也依赖于monitor对象只有在同步代码块或方法中才能调wait/notify等方法) - - Lock是具体类（java.util.concurrent.locks.Lock）是api 层面的锁 +1. volatile变量是一种同步机制； +2. volatile能够确保可见性。 -2、使用方法 +这两点和我们探讨“volatile变量是否能够保证线程安全性”息息相关。 -synchronized 不需要用户手动释放锁，当 synchronized 代码执行完后系统会自动让线程释放对象锁的占用 +volatile 是 Java 虚拟机提供的轻量级的同步机制，保证了 Java 内存模型的两个特性，可见性、有序性（禁止指令重排）、不能保证原子性。 -RenntrantLock则需要用户去手动释放锁，若没有手动释放，可能造成死锁 +**场景**： -3、等待是否可中断 +DCL 版本的单例模式就用到了volatile，因为 DCL 也不一定是线程安全的，`instance = new Singleton();`并不是一个原子操作，会分为 3 部分执行， -synchronized 不可中断，除非抛出异常或正常运行结束 +1. 给 instance 分配内存 +2. 调用 instance 的构造函数来初始化对象 +3. 将 instance 对象指向分配的内存空间（执行完这步 instance 就为非 null 了） -RenntrantLock可中断， +步骤 2 和 3 不存在数据依赖关系，如果虚拟机存在指令重排序优化，则步骤 2和 3 的顺序是无法确定的 -- 设置超时时间 tryLock(long timeout,TimeUnit unit) -- lockIntteruptiby() 放代码块中，调用interrupt() 方法可中断 +一句话：在需要保证原子性的场景，不要使用 volatile。 -4、加锁是否公平 -synchronized 是非公平锁 -RenntrantLock两者都可以 +### 🎯 volatile 底层的实现机制？ -5、锁绑定多个条件Condition +volatile 可以保证线程可见性且提供了一定的有序性，但是无法保证原子性。在 JVM 底层是基于内存屏障实现的。 -synchronized 没有 +- 当对非 volatile 变量进行读写的时候，每个线程先从内存拷贝变量到 CPU 缓存中。如果计算机有多个CPU，每个线程可能在不同的 CPU 上被处理，这意味着每个线程可以拷贝到不同的 CPU cache 中 +- 而声明变量是 volatile 的，JVM 保证了每次读变量都从内存中读，跳过 CPU cache 这一步，所以就不会有可见性问题 + - 对 volatile 变量进行写操作时，会在写操作后加一条 store 屏障指令，将工作内存中的共享变量刷新回主内存； + - 对 volatile 变量进行读操作时，会在写操作后加一条 load 屏障指令，从主内存中读取共享变量； -RenntrantLock用来实现分组唤醒需要唤醒的线程们，可以精准唤醒，而不是像synchronized那样随机唤醒一个线程要么唤醒全部线程。 +> 基于 **内存屏障指令**： +> +> 1. 写操作屏障 +> +> ```Asm +> StoreStoreBarrier +> volatile写操作 +> StoreLoadBarrier // 强制刷新到主存 +> ``` +> +> 2. 读操作屏障 +> +> ```Asm +> volatile读操作 +> LoadLoadBarrier +> LoadStoreBarrier // 禁止后续读写重排序 +> ``` +> +> **硬件级实现**： x86 平台使用 `lock` 前缀指令（如 `lock addl $0,0(%rsp)`）实现内存屏障效果。 +### 🎯 volatile 是线程安全的吗 +**因为volatile不能保证变量操作的原子性，所以试图通过volatile来保证线程安全性是不靠谱的** +### 🎯 volatile 变量和 atomic 变量有什么不同？ +| **特性** | **volatile** | **AtomicXXX** | +| ------------ | ------------------------ | ------------------ | +| **可见性** | ✅ 保证 | ✅ 保证 | +| **有序性** | ✅ 保证 | ✅ 保证 | +| **原子性** | ❌ 不保证（如 `count++`） | ✅ 保证（CAS 实现） | +| **底层实现** | 内存屏障 | CAS + volatile | +| **性能开销** | 低（无锁） | 中等（CAS 自旋） | +| **适用场景** | 状态标志、DCL 单例 | 计数器、累加操作 | -### 说说 synchronized 关键字和 volatile 关键字的区别 +### 🎯 synchronized 关键字和 volatile 关键字的区别? `synchronized` 关键字和 `volatile` 关键字是两个互补的存在，而不是对立的存在： @@ -545,888 +824,2688 @@ RenntrantLock用来实现分组唤醒需要唤醒的线程们，可以精准唤 - **volatile关键字能保证数据的可见性，但不能保证数据的原子性。synchronized关键字两者都能保证。** - **volatile关键字主要用于解决变量在多个线程之间的可见性，而 synchronized关键字解决的是多个线程之间访问资源的同步性。** +------ -## ThreadLocal -### 3.1. ThreadLocal简介 +## 三、锁机制 🏛️ -通常情况下，我们创建的变量是可以被任何一个线程访问并修改的。**如果想实现每一个线程都有自己的专属本地变量该如何解决呢？** JDK中提供的`ThreadLocal`类正是为了解决这样的问题。 **ThreadLocal类主要解决的就是让每个线程绑定自己的值，可以将ThreadLocal类形象的比喻成存放数据的盒子，盒子中可以存储每个线程的私有数据。** +### 🎯 你知道哪几种锁？分别有什么特点？ -**如果你创建了一个ThreadLocal变量，那么访问这个变量的每个线程都会有这个变量的本地副本，这也是ThreadLocal变量名的由来。他们可以使用 get（） 和 set（） 方法来获取默认值或将其值更改为当前线程所存的副本的值，从而避免了线程安全问题。** +根据分类标准我们把锁分为以下 7 大类别，分别是： -再举个简单的例子： +- 偏向锁/轻量级锁/重量级锁：偏向锁/轻量级锁/重量级锁，这三种锁特指 synchronized 锁的状态，通过在对象头中的 mark word 来表明锁的状态。 +- 可重入锁/非可重入锁：可重入锁指的是线程当前已经持有这把锁了，能在不释放这把锁的情况下，再次获取这把锁 +- 共享锁/独占锁：共享锁指的是我们同一把锁可以被多个线程同时获得，而独占锁指的就是，这把锁只能同时被一个线程获得。我们的读写锁，就最好地诠释了共享锁和独占锁的理念。读写锁中的读锁，是共享锁，而写锁是独占锁。 +- 公平锁/非公平锁：公平锁的公平的含义在于如果线程现在拿不到这把锁，那么线程就都会进入等待，开始排队，在等待队列里等待时间长的线程会优先拿到这把锁，有先来先得的意思。而非公平锁就不那么“完美”了，它会在一定情况下，忽略掉已经在排队的线程，发生插队现象 +- 悲观锁/乐观锁：悲观锁假定并发冲突**一定会发生**，因此在操作共享数据前**先加锁**（独占资源）。乐观锁是假定并发冲突**很少发生**，操作共享数据时**不加锁**，在提交更新时检测是否发生冲突（通常通过版本号或 CAS 机制） +- 自旋锁/非自旋锁：自旋锁的理念是如果线程现在拿不到锁，并不直接陷入阻塞或者释放 CPU 资源，而是开始利用循环，不停地尝试获取锁，这个循环过程被形象地比喻为“自旋” +- 可中断锁/不可中断锁：synchronized 关键字修饰的锁代表的是不可中断锁，一旦线程申请了锁，就没有回头路了，只能等到拿到锁以后才能进行其他的逻辑处理。而我们的 ReentrantLock 是一种典型的可中断锁 -比如有两个人去宝屋收集宝物，这两个共用一个袋子的话肯定会产生争执，但是给他们两个人每个人分配一个袋子的话就不会出现这样的问题。如果把这两个人比作线程的话，那么ThreadLocal就是用来避免这两个线程竞争的。 +| **锁类型** | **特点** | **典型实现** | **适用场景** | +| ---------- | -------------------- | ------------------------------- | ---------------- | +| 乐观锁 | 无锁，通过 CAS 更新 | `AtomicInteger`、数据库版本号 | 读多写少、冲突少 | +| 悲观锁 | 操作前加锁 | `synchronized`、`ReentrantLock` | 写多、竞争激烈 | +| 公平锁 | 按请求顺序获取锁 | `ReentrantLock(true)` | 防止线程饥饿 | +| 可重入锁 | 同一线程可重复加锁 | `synchronized`、`ReentrantLock` | 嵌套同步块 | +| 读写锁 | 读锁共享，写锁排他 | `ReentrantReadWriteLock` | 读多写少 | +| 偏向锁 | 单线程优化，无锁竞争 | JVM 对 `synchronized` 的优化 | 单线程场景 | +| 自旋锁 | 循环尝试获取锁 | CAS 操作 | 锁持有时间短 | -### 3.2. ThreadLocal示例 -相信看了上面的解释，大家已经搞懂 ThreadLocal 类是个什么东西了。 -``` -import java.text.SimpleDateFormat; -import java.util.Random; -public class ThreadLocalExample implements Runnable{ +### 🎯 ReentrantLock (可重入锁) - // SimpleDateFormat 不是线程安全的，所以每个线程都要有自己独立的副本 - private static final ThreadLocal formatter = ThreadLocal.withInitial(() -> new SimpleDateFormat("yyyyMMdd HHmm")); +`ReentrantLock` 是 Java 并发包（`java.util.concurrent.locks`）中实现的**可重入显式锁**，功能上与 `synchronized` 类似，但提供更灵活的锁控制（如可中断锁、公平锁、条件变量）。 - public static void main(String[] args) throws InterruptedException { - ThreadLocalExample obj = new ThreadLocalExample(); - for(int i=0 ; i<10; i++){ - Thread t = new Thread(obj, ""+i); - Thread.sleep(new Random().nextInt(1000)); - t.start(); - } - } +**核心特性**： - @Override - public void run() { - System.out.println("Thread Name= "+Thread.currentThread().getName()+" default Formatter = "+formatter.get().toPattern()); - try { - Thread.sleep(new Random().nextInt(1000)); - } catch (InterruptedException e) { - e.printStackTrace(); - } - //formatter pattern is changed here by thread, but it won't reflect to other threads - formatter.set(new SimpleDateFormat()); +1. **可重入性**：同一线程可多次获取同一把锁而不会死锁（通过内部计数器实现）。 + - 实现原理：锁内部维护一个**持有锁的线程标识**和**重入次数计数器**，线程再次获取锁时，计数器加 1，释放锁时计数器减 1，直至为 0 时真正释放锁。 +2. **显式锁管理**：需手动调用 `lock()` 和 `unlock()`（必须在 `finally` 块中释放）。 +3. **公平锁支持**：通过构造参数 `new ReentrantLock(true)` 实现线程按请求顺序获取锁。 +4. 灵活的锁获取方式： + - `lock()`：阻塞式获取锁。 + - `tryLock()`：非阻塞式尝试获取锁（立即返回结果）。 + - `tryLock(timeout, unit)`：带超时的获取锁。 + - `lockInterruptibly()`：可响应中断的获取锁。 +5. **条件变量（Condition）**：替代 `wait()`/`notify()`，支持多路等待队列（如生产者 - 消费者模型）。 - System.out.println("Thread Name= "+Thread.currentThread().getName()+" formatter = "+formatter.get().toPattern()); - } -} -``` -Output: +### 🎯 ReetrantLock有用过吗，怎么实现重入的？ -``` -Thread Name= 0 default Formatter = yyyyMMdd HHmm -Thread Name= 0 formatter = yy-M-d ah:mm -Thread Name= 1 default Formatter = yyyyMMdd HHmm -Thread Name= 2 default Formatter = yyyyMMdd HHmm -Thread Name= 1 formatter = yy-M-d ah:mm -Thread Name= 3 default Formatter = yyyyMMdd HHmm -Thread Name= 2 formatter = yy-M-d ah:mm -Thread Name= 4 default Formatter = yyyyMMdd HHmm -Thread Name= 3 formatter = yy-M-d ah:mm -Thread Name= 4 formatter = yy-M-d ah:mm -Thread Name= 5 default Formatter = yyyyMMdd HHmm -Thread Name= 5 formatter = yy-M-d ah:mm -Thread Name= 6 default Formatter = yyyyMMdd HHmm -Thread Name= 6 formatter = yy-M-d ah:mm -Thread Name= 7 default Formatter = yyyyMMdd HHmm -Thread Name= 7 formatter = yy-M-d ah:mm -Thread Name= 8 default Formatter = yyyyMMdd HHmm -Thread Name= 9 default Formatter = yyyyMMdd HHmm -Thread Name= 8 formatter = yy-M-d ah:mm -Thread Name= 9 formatter = yy-M-d ah:mm -``` +ReentrantLock 的可重入性是 AQS 很好的应用之一。在 ReentrantLock 里面，不管是公平锁还是非公平锁，都有一段逻辑。 -从输出中可以看出，Thread-0已经改变了formatter的值，但仍然是thread-2默认格式化程序与初始化值相同，其他线程也一样。 +公平锁： -上面有一段代码用到了创建 `ThreadLocal` 变量的那段代码用到了 Java8 的知识，它等于下面这段代码，如果你写了下面这段代码的话，IDEA会提示你转换为Java8的格式(IDEA真的不错！)。因为ThreadLocal类在Java 8中扩展，使用一个新的方法`withInitial()`，将Supplier功能接口作为参数。 +```java +// java.util.concurrent.locks.ReentrantLock.FairSync#tryAcquire -``` - private static final ThreadLocal formatter = new ThreadLocal(){ - @Override - protected SimpleDateFormat initialValue() - { - return new SimpleDateFormat("yyyyMMdd HHmm"); - } - }; +if (c == 0) { + //在公平锁中，线程获取锁时会检查等待队列，只有当没有其他线程等待时，才会获取锁，这保证了线程按照请求的顺序获取锁。 + if (!hasQueuedPredecessors() && compareAndSetState(0, acquires)) { + setExclusiveOwnerThread(current); + return true; + } +} +else if (current == getExclusiveOwnerThread()) { + int nextc = c + acquires; + if (nextc < 0) + throw new Error("Maximum lock count exceeded"); + setState(nextc); + return true; +} ``` -### 3.3. ThreadLocal原理 - -从 `Thread`类源代码入手。 +非公平锁： -``` -public class Thread implements Runnable { - ...... -//与此线程有关的ThreadLocal值。由ThreadLocal类维护 -ThreadLocal.ThreadLocalMap threadLocals = null; +```java +// java.util.concurrent.locks.ReentrantLock.Sync#nonfairTryAcquire -//与此线程有关的InheritableThreadLocal值。由InheritableThreadLocal类维护 -ThreadLocal.ThreadLocalMap inheritableThreadLocals = null; - ...... +if (c == 0) { + if (compareAndSetState(0, acquires)){ + setExclusiveOwnerThread(current); + return true; + } +} +else if (current == getExclusiveOwnerThread()) { + int nextc = c + acquires; + if (nextc < 0) // overflow + throw new Error("Maximum lock count exceeded"); + setState(nextc); + return true; } ``` -从上面`Thread`类 源代码可以看出`Thread` 类中有一个 `threadLocals` 和 一个 `inheritableThreadLocals` 变量，它们都是 `ThreadLocalMap` 类型的变量,我们可以把 `ThreadLocalMap` 理解为`ThreadLocal` 类实现的定制化的 `HashMap`。默认情况下这两个变量都是null，只有当前线程调用 `ThreadLocal` 类的 `set`或`get`方法时才创建它们，实际上调用这两个方法的时候，我们调用的是`ThreadLocalMap`类对应的 `get()`、`set() `方法。 +从上面这两段都可以看到，有一个同步状态State来控制整体可重入的情况。State 是 volatile 修饰的，用于保证一定的可见性和有序性。 -`ThreadLocal`类的`set()`方法 +```java +// java.util.concurrent.locks.AbstractQueuedSynchronizer +private volatile int state; ``` - public void set(T value) { - Thread t = Thread.currentThread(); - ThreadLocalMap map = getMap(t); - if (map != null) - map.set(this, value); - else - createMap(t, value); - } - ThreadLocalMap getMap(Thread t) { - return t.threadLocals; + +接下来看 State 这个字段主要的过程： + +1. State 初始化的时候为 0，表示没有任何线程持有锁。 +2. 当有线程持有该锁时，值就会在原来的基础上 +1，同一个线程多次获得锁是，就会多次 +1，这里就是可重入的概念。 +3. 解锁也是对这个字段 -1，一直到 0，此线程对锁释放。 + +还会通过 `getExclusiveOwnerThread()`、`setExclusiveOwnerThread(current)`进行当前线程的设置 + + + +### 🎯 谈谈 synchronized和 ReentrantLock 的区别？ + +| **特性** | **synchronized** | **ReentrantLock** | +| -------------- | ------------------------------------------------- | ------------------------------- | +| **锁类型** | 隐式锁（JVM 控制） | 显式锁（手动获取 / 释放） | +| **可重入性** | 支持 | 支持 | +| **公平性** | 非公平（无法设置） | 可通过构造方法设置公平 / 非公平 | +| **锁获取方式** | 阻塞式 | 支持阻塞、非阻塞、超时、可中断 | +| **线程通信** | 使用 `wait()`/`notify()` | 使用 `Condition` 对象 | +| **性能** | 早期版本性能较差，JDK 6+ 优化后接近 ReentrantLock | 优化后性能高，尤其在竞争激烈时 | +| **适用场景** | 简单场景（自动释放锁） | 复杂场景（需要灵活控制锁逻辑） | + +1. 两者都是可重入锁 + +2. synchronized 依赖于 JVM 而 ReentrantLock 依赖于 API + + synchronized 是依赖于 JVM 实现的，虚拟机团队在 JDK1.6 为 synchronized 关键字进行了很多优化，但是这些优化都是在虚拟机层面实现的，并没有直接暴露给我们。ReentrantLock 是 JDK 层面实现的（也就是 API 层面，需要 lock() 和 unlock() 方法配合 try/finally 语句块来完成），所以我们可以通过查看它的源代码，来看它是如何实现的。 + +3. ReentrantLock 比 synchronized 增加了一些高级功能。主要来说主要有三点：**①等待可中断；②可实现公平锁；③可实现选择性通知（锁可以绑定多个条件）** + + - **ReentrantLock提供了一种能够中断等待锁的线程的机制**，通过 `lock.lockInterruptibly()` 来实现这个机制。也就是说正在等待的线程可以选择放弃等待，改为处理其他事情。 + + - **ReentrantLock可以指定是公平锁还是非公平锁。而synchronized只能是非公平锁。所谓的公平锁就是先等待的线程先获得锁。** ReentrantLock 默认情况是非公平的，可以通过 ReentrantLock 类的 `ReentrantLock(boolean fair)` 构造方法来制定是否是公平的。 + + - synchronized 关键字与 `wait()` 和 `notify()/notifyAll()` 方法相结合可以实现等待/通知机制，ReentrantLock 类当然也可以实现，但是需要借助于 Condition 接口与 `newCondition()` 方法。Condition 是 JDK1.5 之后才有的，它具有很好的灵活性，比如可以实现多路通知功能也就是在一个Lock对象中可以创建多个Condition实例（即对象监视器），**线程对象可以注册在指定的Condition中，从而可以有选择性的进行线程通知，在调度线程上更加灵活。 在使用notify()/notifyAll()方法进行通知时，被通知的线程是由 JVM 选择的，用ReentrantLock类结合Condition实例可以实现“选择性通知”** ，这个功能非常重要，而且是Condition接口默认提供的。而synchronized关键字就相当于整个Lock对象中只有一个Condition实例，所有的线程都注册在它一个身上。如果执行notifyAll()方法的话就会通知所有处于等待状态的线程这样会造成很大的效率问题，而Condition实例的signalAll()方法 只会唤醒注册在该Condition实例中的所有等待线程。 + + + +### 🎯 读写锁 ReentrantReadWriteLock + +"ReadWriteLock实现了读写分离，适用于读多写少的场景： + +**核心特点**： + +- **读锁共享**：多个线程可同时持有读锁 +- **写锁独占**：写锁与任何锁互斥 +- **锁降级**：持有写锁时可获取读锁 +- **不支持锁升级**：持有读锁时不能获取写锁” + + + +### 🎯 什么是线程死锁? 如何避免死锁? + +线程死锁描述的是这样一种情况：多个线程同时被阻塞，它们中的一个或者全部都在等待某个资源被释放。由于线程被无限期地阻塞，因此程序不可能正常终止。 + +如下图所示，线程 A 持有资源 2，线程 B 持有资源 1，他们同时都想申请对方的资源，所以这两个线程就会互相等待而进入死锁状态。 + +![img](https://ask.qcloudimg.com/http-save/5876652/1t73ossag1.jpeg) + +下面通过一个例子来说明线程死锁,代码模拟了上图的死锁的情况 (代码来源于《并发编程之美》)： + +```JAVA +public class DeadLockDemo { + private static Object resource1 = new Object();//资源 1 + private static Object resource2 = new Object();//资源 2 + + public static void main(String[] args) { + new Thread(() -> { + synchronized (resource1) { + System.out.println(Thread.currentThread() + "get resource1"); + try { + Thread.sleep(1000); + } catch (InterruptedException e) { + e.printStackTrace(); + } + System.out.println(Thread.currentThread() + "waiting get resource2"); + synchronized (resource2) { + System.out.println(Thread.currentThread() + "get resource2"); + } + } + }, "线程 1").start(); + + new Thread(() -> { + synchronized (resource2) { + System.out.println(Thread.currentThread() + "get resource2"); + try { + Thread.sleep(1000); + } catch (InterruptedException e) { + e.printStackTrace(); + } + System.out.println(Thread.currentThread() + "waiting get resource1"); + synchronized (resource1) { + System.out.println(Thread.currentThread() + "get resource1"); + } + } + }, "线程 2").start(); } +} ``` -通过上面这些内容，我们足以通过猜测得出结论：**最终的变量是放在了当前线程的 ThreadLocalMap 中，并不是存在 ThreadLocal 上，ThreadLocal 可以理解为只是ThreadLocalMap的封装，传递了变量值。** `ThrealLocal` 类中可以通过`Thread.currentThread()`获取到当前线程对象后，直接通过`getMap(Thread t)`可以访问到该线程的`ThreadLocalMap`对象。 - -**每个Thread中都具备一个ThreadLocalMap，而ThreadLocalMap可以存储以ThreadLocal为key ，Object 对象为 value的键值对。** +Output ``` -ThreadLocalMap(ThreadLocal firstKey, Object firstValue) { - ...... -} +Thread[线程 1,5,main]get resource1 +Thread[线程 2,5,main]get resource2 +Thread[线程 1,5,main]waiting get resource2 +Thread[线程 2,5,main]waiting get resource1 ``` -比如我们在同一个线程中声明了两个 `ThreadLocal` 对象的话，会使用 `Thread`内部都是使用仅有那个`ThreadLocalMap` 存放数据的，`ThreadLocalMap`的 key 就是 `ThreadLocal`对象，value 就是 `ThreadLocal` 对象调用`set`方法设置的值。 +线程 A 通过 synchronized (resource1) 获得 resource1 的监视器锁，然后通过 `Thread.sleep(1000);`让线程 A 休眠 1s 为的是让线程 B 得到执行然后获取到 resource2 的监视器锁。线程 A 和线程 B 休眠结束了都开始企图请求获取对方的资源，然后这两个线程就会陷入互相等待的状态，这也就产生了死锁。上面的例子符合产生死锁的四个必要条件。学过操作系统的朋友都知道产生死锁必须具备以下四个条件： + +- 互斥条件：该资源任意一个时刻只由一个线程占用。 +- 占有且等待：一个进程因请求资源而阻塞时，对已获得的资源保持不放。 +- 不可强行占有：线程已获得的资源在末使用完之前不能被其他线程强行剥夺，只有自己使用完毕后才释放资源 +- 循环等待条件：若干进程之间形成一种头尾相接的循环等待资源关系。 -[![ThreadLocal数据结构](https://camo.githubusercontent.com/a463d65fe6b4b96dc81750d19f80f26f8e0675d0/68747470733a2f2f75706c6f61642d696d616765732e6a69616e7368752e696f2f75706c6f61645f696d616765732f373433323630342d616432666635383131323762613863632e6a70673f696d6167654d6f6772322f6175746f2d6f7269656e742f7374726970253743696d61676556696577322f322f772f383036)](https://camo.githubusercontent.com/a463d65fe6b4b96dc81750d19f80f26f8e0675d0/68747470733a2f2f75706c6f61642d696d616765732e6a69616e7368752e696f2f75706c6f61645f696d616765732f373433323630342d616432666635383131323762613863632e6a70673f696d6167654d6f6772322f6175746f2d6f7269656e742f7374726970253743696d61676556696577322f322f772f383036) -`ThreadLocalMap`是`ThreadLocal`的静态内部类。 -[![ThreadLocal内部类](https://camo.githubusercontent.com/11f103a8a726894a98d431b0675f759e3d915782/68747470733a2f2f6d792d626c6f672d746f2d7573652e6f73732d636e2d6265696a696e672e616c6979756e63732e636f6d2f323031392d362f5468726561644c6f63616c2545352538362538352545392538332541382545372542312542422e706e67)](https://camo.githubusercontent.com/11f103a8a726894a98d431b0675f759e3d915782/68747470733a2f2f6d792d626c6f672d746f2d7573652e6f73732d636e2d6265696a696e672e616c6979756e63732e636f6d2f323031392d362f5468726561644c6f63616c2545352538362538352545392538332541382545372542312542422e706e67) +### 🎯 如何避免线程死锁? -### 3.4. ThreadLocal 内存泄露问题 +我上面说了产生死锁的四个必要条件，为了避免死锁，我们只要破坏产生死锁的四个条件中的其中一个就可以了。现在我们来挨个分析一下： -`ThreadLocalMap` 中使用的 key 为 `ThreadLocal` 的弱引用,而 value 是强引用。所以，如果 `ThreadLocal` 没有被外部强引用的情况下，在垃圾回收的时候，key 会被清理掉，而 value 不会被清理掉。这样一来，`ThreadLocalMap` 中就会出现key为null的Entry。假如我们不做任何措施的话，value 永远无法被GC 回收，这个时候就可能会产生内存泄露。ThreadLocalMap实现中已经考虑了这种情况，在调用 `set()`、`get()`、`remove()` 方法的时候，会清理掉 key 为 null 的记录。使用完 `ThreadLocal`方法后 最好手动调用`remove()`方法 +1. **破坏互斥条件** ：这个条件我们没有办法破坏，因为我们用锁本来就是想让他们互斥的（临界资源需要互斥访问）。 +2. **破坏请求与保持条件** ：一次性申请所有的资源。 +3. **破坏不剥夺条件** ：占用部分资源的线程进一步申请其他资源时，如果申请不到，可以主动释放它占有的资源。 +4. **破坏循环等待条件** ：靠按序申请资源来预防。按某一顺序申请资源，释放资源则反序释放。破坏循环等待条件。 -``` - static class Entry extends WeakReference> { - /** The value associated with this ThreadLocal. */ - Object value; +我们对线程 2 的代码修改成下面这样就不会产生死锁了。 - Entry(ThreadLocal k, Object v) { - super(k); - value = v; - } +```java +new Thread(() -> { + synchronized (resource1) { + System.out.println(Thread.currentThread() + "get resource1"); + try { + Thread.sleep(1000); + } catch (InterruptedException e) { + e.printStackTrace(); } + System.out.println(Thread.currentThread() + "waiting get resource2"); + synchronized (resource2) { + System.out.println(Thread.currentThread() + "get resource2"); + } + } +}, "线程 2").start(); ``` -**弱引用介绍：** +Output -> 如果一个对象只具有弱引用，那就类似于**可有可无的生活用品**。弱引用与软引用的区别在于：只具有弱引用的对象拥有更短暂的生命周期。在垃圾回收器线程扫描它 所管辖的内存区域的过程中，一旦发现了只具有弱引用的对象，不管当前内存空间足够与否，都会回收它的内存。不过，由于垃圾回收器是一个优先级很低的线程， 因此不一定会很快发现那些只具有弱引用的对象。 -> -> 弱引用可以和一个引用队列（ReferenceQueue）联合使用，如果弱引用所引用的对象被垃圾回收，Java虚拟机就会把这个弱引用加入到与之关联的引用队列中。 +``` +Thread[线程 1,5,main]get resource1 +Thread[线程 1,5,main]waiting get resource2 +Thread[线程 1,5,main]get resource2 +Thread[线程 2,5,main]get resource1 +Thread[线程 2,5,main]waiting get resource2 +Thread[线程 2,5,main]get resource2 + +Process finished with exit code 0 +``` +我们分析一下上面的代码为什么避免了死锁的发生? +线程 1 首先获得到 resource1 的监视器锁，这时候线程 2 就获取不到了。然后线程 1 再去获取 resource2 的监视器锁，可以获取到。然后线程 1 释放了对 resource1、resource2 的监视器锁的占用，线程 2 获取到就可以执行了。这样就破坏了破坏循环等待条件，因此避免了死锁。 -# 线程池篇 +### 🎯 如何排查死锁? -> 线程池原理，拒绝策略，核心线程数 +定位死锁最常见的方式就是利用 jstack 等工具获取线程栈，然后定位互相之间的依赖关系，进而找到死锁。如果是比较明显的死锁，往往 jstack 等就能直接定位，类似 JConsole 甚至可以在图形界面进行有限的死锁检测。 + +如果程序运行时发生了死锁，绝大多数情况下都是无法在线解决的，只能重启、修正程序本身问题。所以，代码开发阶段互相审查，或者利用工具进行预防性排查，往往也是很重要的。 + +#### 死锁预防 + +1. 以确定的顺序获得锁 + + 如果必须获取多个锁，那么在设计的时候需要充分考虑不同线程之前获得锁的顺序 + +2. 超时放弃 + + 当使用 synchronized 关键词提供的内置锁时，只要线程没有获得锁，那么就会永远等待下去，然而Lock接口提供了`boolean tryLock(long time, TimeUnit unit) throws InterruptedException`方法，该方法可以按照固定时长等待锁，因此线程可以在获取锁超时以后，主动释放之前已经获得的所有的锁。通过这种方式，也可以很有效地避免死锁。 + + + +### 🎯 哲学家就餐问题？ + +> 这题我刚毕业那会，遇见过一次，笔试题 > -> 为什么要用线程池，优势是什么？ +> 哲学家就餐问题（The Dining Philosophers Problem）是计算机科学中经典的同步问题之一，由 Edsger Dijkstra 于 1965 年提出。问题描述如下： > -> 线程池的工作原理，几个重要参数，给了具体几个参数分析线程池会怎么做，阻塞队列的作用是什么？ +> - 有五个哲学家围坐在圆桌旁，每个哲学家前面有一盘意大利面。 > -> 说说几种常见的线程池及使用场景? +> - 在每两位哲学家之间有一只叉子（共五只叉子）。 > -> 线程池的构造类的方法的 5 个参数的具体意义是什么 +> - 哲学家需要两只叉子才能吃意大利面。 > -> 按线程池内部机制，当提交新任务时，有哪些异常要考虑 +> - 哲学家可以进行两个动作：思考和吃饭。 > -> 单机上一个线程池正在处理服务，如果忽然断电怎么办（正在处理和阻塞队列里的请求怎么处理）？ +> - 当哲学家思考时，他们不占用任何叉子；当哲学家准备吃饭时，他们必须先拿起左右两边的叉子。 > -> 生产上如何合理设置参数？ +> ![1226. 哲学家进餐- 力扣（LeetCode）](https://img.starfish.ink/algorithm/philosopher.jpeg) +问题的关键在于如何避免死锁（Deadlock），确保每个哲学家都有机会吃饭，同时也要避免资源饥饿（Starvation） +**解决方案** -线程池是一种基于池化思想管理线程的工具。 +对于这个问题我们该如何解决呢？有多种解决方案，这里我们讲讲其中的几种。前面我们讲过，要想解决死锁问题，只要破坏死锁四个必要条件的任何一个都可以。 -线程池解决的核心问题就是资源管理问题。在并发环境下，系统不能够确定在任意时刻中，有多少任务需要执行，有多少资源需要投入。这种不确定性将带来以下若干问题： +**1. 服务员检查** -1. 频繁申请/销毁资源和调度资源，将带来额外的消耗，可能会非常巨大。 -2. 对资源无限申请缺少抑制手段，易引发系统资源耗尽的风险。 -3. 系统无法合理管理内部的资源分布，会降低系统的稳定性。 +第一个解决方案就是引入服务员检查机制。比如我们引入一个服务员，当每次哲学家要吃饭时，他需要先询问服务员：我现在能否去拿筷子吃饭？此时，服务员先判断他拿筷子有没有发生死锁的可能，假如有的话，服务员会说：现在不允许你吃饭。这是一种解决方案。 -为解决资源分配这个问题，线程池采用了“池化”思想。 +```java +import java.util.concurrent.Semaphore; +import java.util.concurrent.locks.Lock; +import java.util.concurrent.locks.ReentrantLock; +public class DiningPhilosophers { + private final Lock[] forks = new ReentrantLock[5]; + private final Semaphore waiter = new Semaphore(4); + public DiningPhilosophers() { + for (int i = 0; i < forks.length; i++) { + forks[i] = new ReentrantLock(); + } + } -线程池做的工作主要是控制运行的线程数量，处理过程中将任务放入队列，然后在线程创建后启动这些任务，如果线程数量超过了最大数量，超出数量的线程排队等候，等其他线程执行完毕，再从队列中取出任务来执行。 + public void dine(int philosopher) throws InterruptedException { + waiter.acquire(); -主要优点： + int leftFork = philosopher; + int rightFork = (philosopher + 1) % 5; -1. **降低资源消耗**：线程复用，通过重复利用已创建的线程减低线程创建和销毁造成的消耗 -2. **提高响应速度**：当任务到达时，任务可以不需要等到线程创建就能立即执行 -3. **提高线程的可管理性**：线程是稀缺资源，如果无限制的创建，不仅会消耗系统资源，还会降低系统的稳定性，使用线程池可以进行统一的分配，调优和监控。 -4. **提供更多更强大的功能**：线程池具备可拓展性，允许开发人员向其中增加更多的功能。比如延时定时线程池ScheduledThreadPoolExecutor，就允许任务延期执行或定期执行。 + forks[leftFork].lock(); + forks[rightFork].lock(); + + try { + eat(philosopher); + } finally { + forks[leftFork].unlock(); + forks[rightFork].unlock(); + waiter.release(); + } + } + + private void eat(int philosopher) throws InterruptedException { + System.out.println("Philosopher " + philosopher + " is eating"); + Thread.sleep(1000); // Simulate eating + System.out.println("Philosopher " + philosopher + " finished eating"); + } + + public static void main(String[] args) { + DiningPhilosophers dp = new DiningPhilosophers(); + for (int i = 0; i < 5; i++) { + final int philosopher = i; + new Thread(() -> { + try { + dp.dine(philosopher); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + }).start(); + } + } +} + +``` + +**2. 领导调节** + +基于死锁**检测和恢复策略**，可以引入一个领导，这个领导进行定期巡视。如果他发现已经发生死锁了，就会剥夺某一个哲学家的筷子，让他放下。这样一来，由于这个人的牺牲，其他的哲学家就都可以吃饭了。这也是一种解决方案。 + +**3. 改变一个哲学家拿筷子的顺序** + +我们还可以利用**死锁避免**策略，那就是从逻辑上去避免死锁的发生，比如改变其中一个哲学家拿筷子的顺序。我们可以让 4 个哲学家都先拿左边的筷子再拿右边的筷子，但是**有一名哲学家与他们相反，他是先拿右边的再拿左边的**，这样一来就不会出现循环等待同一边筷子的情况，也就不会发生死锁了。 + +```java +import java.util.concurrent.locks.Lock; +import java.util.concurrent.locks.ReentrantLock; + +public class DiningPhilosophers { + private final Lock[] forks = new ReentrantLock[5]; + + public DiningPhilosophers() { + for (int i = 0; i < forks.length; i++) { + forks[i] = new ReentrantLock(); + } + } + + public void dine(int philosopher) throws InterruptedException { + int leftFork = philosopher; + int rightFork = (philosopher + 1) % 5; + + if (philosopher % 2 == 0) { + forks[leftFork].lock(); + forks[rightFork].lock(); + } else { + forks[rightFork].lock(); + forks[leftFork].lock(); + } + + try { + eat(philosopher); + } finally { + forks[leftFork].unlock(); + forks[rightFork].unlock(); + } + } + + private void eat(int philosopher) throws InterruptedException { + System.out.println("Philosopher " + philosopher + " is eating"); + Thread.sleep(1000); // Simulate eating + System.out.println("Philosopher " + philosopher + " finished eating"); + } + + public static void main(String[] args) { + DiningPhilosophers dp = new DiningPhilosophers(); + for (int i = 0; i < 5; i++) { + final int philosopher = i; + new Thread(() -> { + try { + dp.dine(philosopher); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + }).start(); + } + } +} + +``` + + + +### 🎯 何谓悲观锁与乐观锁? + +- **悲观锁** + + 总是假设最坏的情况，每次去拿数据的时候都认为别人会修改，所以每次在拿 数据的时候都会上锁，这样别人想拿这个数据就会阻塞直到它拿到锁(共享资 源每次只给一个线程使用，其它线程阻塞，用完后再把资源转让给其它线程)。传统的关系型数据库里边就用到了很多这种锁机制，比如行锁，表锁等，读锁，写锁等，都是在做操作之前先上锁。Java 中 synchronized 和 ReentrantLock 等独占锁就是悲观锁思想的实现。 + +- **乐观锁** + + 总是假设最好的情况，每次去拿数据的时候都认为别人不会修改，所以不会上锁，但是在更新的时候会判断一下在此期间别人有没有去更新这个数据，可以使用版本号机制和 CAS 算法实现。乐观锁适用于多读的应用类型，这样可以提 高吞吐量，像数据库提供的类似于 **write_condition** 机制，其实都是提供的乐 观锁。在 Java 中 `java.util.concurrent.atomic` 包下面的原子变量类就是使用了 乐观锁的一种实现方式 **CAS** 实现的。 + +**两种锁的使用场景** + +从上面对两种锁的介绍，我们知道两种锁各有优缺点，不可认为一种好于另一 种，像**乐观锁适用于写比较少的情况下(多读场景)**，即冲突真的很少发生的 时候，这样可以省去了锁的开销，加大了系统的整个吞吐量。但如果是多写的情况，一般会经常产生冲突，这就会导致上层应用会不断的进行 retry，这样反倒是降低了性能，所以**一般多写的场景下用悲观锁就比较合适**。 + + + +### 🎯 对比公平和非公平的优缺点? + +公平锁的优点在于各个线程公平平等，每个线程等待一段时间后，都有执行的机会，而它的缺点就在于整体执行速度更慢，吞吐量更小，相反非公平锁的优势就在于整体执行速度更快，吞吐量更大，但同时也可能产生线程饥饿问题，也就是说如果一直有线程插队，那么在等待队列中的线程可能长时间得不到运行 + +------ + + + +## 四、原子操作与CAS（无锁编程）⚛️ + +在编程中，具备原子性的操作被称为原子操作。原子操作是指一系列的操作，要么全部发生，要么全部不发生，不会出现执行一半就终止的情况。 + +> 下面我们举一个不具备原子性的例子，比如 i++ 这一行代码在 CPU 中执行时，可能会从一行代码变为以下的 3 个指令： +> +> - 第一个步骤是读取； +> - 第二个步骤是增加； +> - 第三个步骤是保存。 +> +> 这就说明 i++ 是不具备原子性的，同时也证明了 i++ 不是线程安全的 + +Java 中的以下几种操作是具备原子性的，属于原子操作： + +- 除了 long 和 double 之外的基本类型（int、byte、boolean、short、char、float）的读/写操作，都天然的具备原子性； +- 所有引用 reference 的读/写操作； +- 加了 volatile 后，所有变量的读/写操作（包含 long 和 double）。这也就意味着 long 和 double 加了 volatile 关键字之后，对它们的读写操作同样具备原子性； +- 在 java.concurrent.Atomic 包中的一部分类的一部分方法是具备原子性的，比如 AtomicInteger 的 incrementAndGet 方法。 + +> 在 Java 中，`long` 和 `double` 变量的原子性取决于具体的硬件和 JVM 实现，但通常情况下，对 `long` 和 `double` 类型变量的读和写操作不是原子的。这是因为 `long` 和 `double` 在 JVM 中占用 64 位，而在 32 位的 JVM 实现中，对 64 位的操作可能需要分两步进行：每次操作 32 位。因此，如果没有额外的同步措施，多个线程可能会看到部分更新的值，这会导致数据不一致。 +> +> **实际开发中**，目前各种平台下的主流虚拟机的实现中，几乎都会把 64 位数据的读写操作作为原子操作来对待，因此我们在编写代码时一般不需要为了避免读到“半个变量”而把 long 和 double 声明为 volatile 的 + +**原子类的作用**和锁有类似之处，是为了保证并发情况下线程安全。不过原子类相比于锁，有一定的优势： + +- 粒度更细：原子变量可以把竞争范围缩小到变量级别，通常情况下，锁的粒度都要大于原子变量的粒度。 +- 效率更高：除了高度竞争的情况之外，使用原子类的效率通常会比使用同步互斥锁的效率更高，因为原子类底层利用了 CAS 操作，不会阻塞线程。 + +| 类型 | 具体类 | +| :--------------------------------- | :----------------------------------------------------------- | +| Atomic* 基本类型原子类 | AtomicInteger、AtomicLong、AtomicBoolean | +| Atomic*Array 数组类型原子类 | AtomicIntegerArray、AtomicLongArray、AtomicReferenceArray | +| Atomic*Reference 引用类型原子类 | AtomicReference、AtomicStampedReference、AtomicMarkableReference | +| Atomic*FieldUpdater 升级类型原子类 | AtomicIntegerfieldupdater、AtomicLongFieldUpdater、AtomicReferenceFieldUpdater | +| Adder 累加器 | LongAdder、DoubleAdder | +| Accumulator 积累器 | LongAccumulator、DoubleAccumulator | + + + +### 🎯 AtomicInteger 底层实现原理是什么？如何在自己的产品代码中应用 CAS 操作？ + +> `AtomicInteger` 是基于 **CAS（Compare-And-Swap）+ volatile + Unsafe** 实现的。 +> 它内部维护了一个 `volatile int value`，保证可见性； +> 更新时通过 `Unsafe` 类的 `compareAndSwapInt` 方法执行 CAS 操作，只有在当前值等于期望值时才更新成功，否则自旋重试。 +> 这样就避免了加锁，实现了高效的线程安全。 +> +> 在业务代码中，我们也可以用 **CAS 思想**： +> 比如实现一个自增计数器，先读取旧值，再尝试用 CAS 更新，如果失败就重试，直到成功。 +> 不过手写 CAS 代码容易复杂，实际开发中一般用 JUC 提供的原子类（如 `AtomicInteger`、`AtomicLong`）或者 `LongAdder`。 + +**AtomicInteger 的底层原理** + +1. **核心字段** + + ```java + private volatile int value; + ``` + + - 用 `volatile` 保证多线程下修改的可见性。 + +2. **CAS 操作** + + - 核心方法是 `compareAndSet(int expect, int update)`： + + ```java + public final boolean compareAndSet(int expect, int update) { + return unsafe.compareAndSwapInt(this, valueOffset, expect, update); + } + ``` + + - `Unsafe.compareAndSwapInt` 是一个 **JNI 调用**，最终会映射到 CPU 的 **CAS 指令**（如 x86 的 `cmpxchg`）。 +3. **原理流程** + - 读出当前值（expect）； + - 判断内存里的值是否等于 expect； + - 如果相等 → 更新为新值（update）； + - 如果不相等 → 更新失败，说明被其他线程修改过 → 重试。 -常见的线程池的使用方式： +4. **自旋重试** -- newFixedThreadPool 创建一个指定工作线程数量的线程池 -- newSingleThreadExecutor 创建一个单线程化的Executor -- newCachedThreadPool 创建一个可缓存线程池 -- newScheduledThreadPool 创建一个定长的线程池，而且支持定时的以及周期性的任务执行，支持定时及周期性任务执行 -- newWorkStealingPool Java8 新特性，使用目前机器上可用的处理器作为它的并行级别 + - CAS 是乐观锁，假设冲突少，大部分情况下会一次成功； + - 如果失败，就不断自旋尝试，直到更新成功。 +**CAS 的典型问题** + +- **ABA 问题**：值从 A → B → A，CAS 检查时以为没变，但其实发生过变化。 + - 解决方案：`AtomicStampedReference` 或 `AtomicMarkableReference`（带版本号）。 +- **自旋开销**：并发高时可能频繁失败，自旋消耗 CPU。 +- **只能保证一个变量的原子性**：多个变量需要用 `AtomicReference` 或锁。 + +------ + + **在业务代码中的应用（CAS 思想）** + +举几个常见场景： + +1. **自定义原子计数器** + + ```java + class MyAtomicCounter { + private volatile int value = 0; + private static final Unsafe unsafe = ...; + private static final long valueOffset = ...; + + public void increment() { + int oldValue; + do { + oldValue = value; + } while (!unsafe.compareAndSwapInt(this, valueOffset, oldValue, oldValue + 1)); + } + + public int get() { + return value; + } + } + ``` + + 用 CAS 循环实现线程安全自增。 + +2. **无锁队列 / 栈** + + - 常用 CAS 更新头节点或尾节点，避免加锁； + - 典型实现：`ConcurrentLinkedQueue`。 + +3. **乐观锁机制** + + - 在业务场景（如数据库版本号控制）里，也可以用 CAS 思想：比较版本号是否一致，一致才更新，否则重试。 + + + +### 🎯 CAS 知道吗，如何实现？ + +- CAS：全称 `Compare and swap`，即**比较并交换**，它是一条 **CPU 同步原语**。 是一种硬件对并发的支持，针对多处理器操作而设计的一种特殊指令，用于管理对共享数据的并发访问。 +- CAS 是一种无锁的非阻塞算法的实现。 +- CAS 包含了 3 个操作数： + - 需要读写的内存值 V + - 旧的预期值 A + - 要修改的更新值 B +- 当且仅当 V 的值等于 A 时，CAS 通过原子方式用新值 B 来更新 V 的 值，否则不会执行任何操作（他的功能是判断内存某个位置的值是否为预期值，如果是则更改为新的值，这个过程是原子的。） +- 缺点 + - 自旋时间过长：由于单次 CAS 不一定能执行成功，所以 **CAS 往往是配合着循环来实现的**，有的时候甚至是死循环，不停地进行重试，直到线程竞争不激烈的时候，才能修改成功 + - 只能保证一个共享变量的原子操作：不能灵活控制线程安全的范围，我们不能针对多个共享变量同时进行 CAS 操作，因为这多个变量之间是独立的，简单的把原子操作组合到一起，并不具备原子性 + - ABA 问题（用 AtomicReference 避免） + + + +### 🎯 CAS 底层原理，谈谈你对 UnSafe 的理解？ + +> "CAS（Compare and Swap）是一种硬件支持的原子操作： +> +> **基本原理**： +> +> - 包含3个操作数：内存值V、预期值A、更新值B +> - 当且仅当V==A时，才将V更新为B +> - CPU硬件保证整个操作的原子性 +> +> **优势与问题**： +> +> - 优势：无锁化，避免线程阻塞 +> - ABA问题：可用AtomicStampedReference解决 +> - 自旋开销：高竞争下可能CPU空转" + +CAS 并发原语体现在 Java 语言中的 `sum.misc.Unsafe` 类中的各个方法。调用 Unsafe 类中的 CAS 方法， JVM 会帮助我们实现出 CAS 汇编指令。 + +是 CAS 的核心类，由于 Java 方法无法直接访问底层系统，需要通过本地（native）方法来访问，UnSafe 相当于一个后门，UnSafe 类中的所有方法都是 native 修饰的，也就是说该类中的方法都是直接调用操作系统底层资源执行相应任务。 + + + +### 🎯 讲一讲AtomicInteger，为什么要用 CAS 而不是 synchronized？ + +1. **高效的线程安全**：CAS 能在多线程下提供线程安全的操作，而不需要像 `synchronized` 一样使用锁。CAS 的自旋机制在短时间内是非常高效的，因为大多数情况下操作会在几次尝试内成功。 +2. **无锁优化**：CAS 不会引发线程的阻塞和挂起，避免了线程在获取锁时的开销。这对于高并发场景特别重要，`AtomicInteger` 能在高并发场景下提供更好的性能表现。 +3. **避免锁的竞争和开销**：`synchronized` 在多线程竞争时，失败的线程会被挂起并等待唤醒，涉及到线程上下文切换，开销较大。而 CAS 通过乐观锁的思想，只在冲突发生时重试，避免了不必要的线程切换。 + +**CAS 的问题：** + +虽然 CAS 比 `synchronized` 更高效，但它也有一些缺点： + +- **ABA 问题**：CAS 会比较当前值是否等于期望值，但如果一个变量的值从 A 变为 B，再变回 A，CAS 会认为它没有改变，从而通过比较。为了解决这个问题，可以引入版本号。 +- **自旋开销**：如果线程不断尝试修改变量，但总是失败，自旋的开销会变得很高。在高竞争环境下，CAS 的性能优势可能会减小。 +- **只能保证一个变量的原子性**：CAS 只能操作单个变量，对于复杂的并发操作场景，仍然需要使用锁或其他同步机制。 + + + +### 🎯 为什么高并发下 LongAdder 比 AtomicLong 效率更高？ + +> "LongAdder采用分段累加思想： +> +> **核心机制**： +> +> - **base变量**：竞争不激烈时直接累加 +> - **Cell[]数组**：竞争激烈时分散累加 +> - **hash分配**：线程按hash值分配到不同Cell +> +> **性能优势**：空间换时间，减少CAS竞争" + +LongAdder 引入了分段累加的概念，内部一共有两个参数参与计数：第一个叫作 base，它是一个变量，第二个是 Cell[] ，是一个数组。 + +其中的 base 是用在竞争不激烈的情况下的，可以直接把累加结果改到 base 变量上。 + +那么，当竞争激烈的时候，就要用到我们的 Cell[] 数组了。一旦竞争激烈，各个线程会分散累加到自己所对应的那个 Cell[] 数组的某一个对象中，而不会大家共用同一个。 + +这样一来，LongAdder 会把不同线程对应到不同的 Cell 上进行修改，降低了冲突的概率，这是一种分段的理念，提高了并发性，这就和 Java 7 的 ConcurrentHashMap 的 16 个 Segment 的思想类似。 + +竞争激烈的时候，LongAdder 会通过计算出每个线程的 hash 值来给线程分配到不同的 Cell 上去，每个 Cell 相当于是一个独立的计数器，这样一来就不会和其他的计数器干扰，Cell 之间并不存在竞争关系，所以在自加的过程中，就大大减少了刚才的 flush 和 refresh，以及降低了冲突的概率，这就是为什么 LongAdder 的吞吐量比 AtomicLong 大的原因，本质是空间换时间，因为它有多个计数器同时在工作，所以占用的内存也要相对更大一些。 + + + + + +## 五、并发工具类（同步辅助）🛠️ + +### 🎯 AQS 原理分析 + +> AQS，全称 **AbstractQueuedSynchronizer**，是 JUC 包里并发工具类的核心基础框架。 +> +> 其原理可概括为： +> +> 1. **状态管理**：内部维护一个 **volatile 的 state 状态变量**，通过 **CAS 操作**保证修改的原子性。 +> 2. **队列设计**：使用一个 **FIFO 的双向队列（CLH 队列）**，管理等待线程。 +> +> 当线程尝试获取资源失败时，会进入队列并被 `LockSupport.park()` 挂起，等待被唤醒。 +> +> 释放资源时，会通过 CAS 修改 state 并唤醒队列中的下一个线程。 +> 3. **模板方法**：子类通过重写 `tryAcquire()`/`tryRelease()` 实现独占或共享锁。 +> 4. 核心机制： +> - 独占模式（如 `ReentrantLock`）：线程竞争失败则入队阻塞。 +> - 共享模式（如 `CountDownLatch`）：允许多线程同时访问。 +> 5. **应用场景**：锁、信号量、倒计时器等同步工具的基础。 +> +> 常见的 JUC 工具类，比如 `ReentrantLock`、`Semaphore`、`CountDownLatch`，底层都是基于 AQS 来实现的。 +> 所以 **AQS 的本质就是：用 state + CLH 队列 + CAS + LockSupport 来实现一套通用的同步器框架。** + +AQS的全称为（AbstractQueuedSynchronizer），这个类在 `java.util.concurrent.locks` 包下面。 + +AQS是一个用来构建锁和同步器的框架，使用AQS能简单且高效地构造出应用广泛的大量的同步器，比如我们提到的ReentrantLock，Semaphore，其他的诸如ReentrantReadWriteLock，SynchronousQueue，FutureTask 等等皆是基于 AQS 的。当然，我们自己也能利用 AQS 非常轻松容易地构造出符合我们自己需求的同步器。 + +**AQS核心思想是，如果被请求的共享资源空闲，则将当前请求资源的线程设置为有效的工作线程，并且将共享资源设置为锁定状态。如果被请求的共享资源被占用，那么就需要一套线程阻塞等待以及被唤醒时锁分配的机制，这个机制AQS是用CLH队列锁实现的，即将暂时获取不到锁的线程加入到队列中。** + +> CLH(Craig,Landin,and Hagersten)队列是一个虚拟的双向队列（虚拟的双向队列即不存在队列实例，仅存在结点之间的关联关系）。AQS 是将每条请求共享资源的线程封装成一个 CLH 锁队列的一个结点（Node）来实现锁的分配。 + +看个 AQS(AbstractQueuedSynchronizer)原理图： + +![image.png](https://blog-1300588375.cos.ap-chengdu.myqcloud.com/image_1624029202628.png) + + + +AQS 使用一个 int 成员变量来表示同步状态，通过内置的 FIFO 队列来完成获取资源线程的排队工作。AQS 使用 CAS 对该同步状态进行原子操作实现对其值的修改。 + +```java +private volatile int state;//共享变量，使用volatile修饰保证线程可见性 +``` + +状态信息通过 protected 类型的 getState，setState，compareAndSetState 进行操作 + +```java +//返回同步状态的当前值 +protected final int getState() { + return state; +} + // 设置同步状态的值 +protected final void setState(int newState) { + state = newState; +} +//原子地（CAS操作）将同步状态值设置为给定值update如果当前同步状态的值等于expect（期望值） +protected final boolean compareAndSetState(int expect, int update) { + return unsafe.compareAndSwapInt(this, stateOffset, expect, update); +} +``` + +> 而 state 的含义并不是一成不变的，它会**根据具体实现类的作用不同而表示不同的含义**。 +> +> 比如说在信号量里面，state 表示的是剩余**许可证的数量**。如果我们最开始把 state 设置为 10，这就代表许可证初始一共有 10 个，然后当某一个线程取走一个许可证之后，这个 state 就会变为 9，所以信号量的 state 相当于是一个内部计数器。 +> +> 再比如，在 CountDownLatch 工具类里面，state 表示的是**需要“倒数”的数量**。一开始我们假设把它设置为 5，当每次调用 CountDown 方法时，state 就会减 1，一直减到 0 的时候就代表这个门闩被放开。 +> +> 下面我们再来看一下 state 在 ReentrantLock 中是什么含义，在 ReentrantLock 中它表示的是**锁的占有情况**。最开始是 0，表示没有任何线程占有锁；如果 state 变成 1，则就代表这个锁已经被某一个线程所持有了。 + +**AQS 定义两种资源共享方式** + +- Exclusive（独占）：只有一个线程能执行，如 ReentrantLock。又可分为公平锁和非公平锁： + + - 公平锁：按照线程在队列中的排队顺序，先到者先拿到锁 + - 非公平锁：当线程要获取锁时，无视队列顺序直接去抢锁，谁抢到就是谁的 + +- **Share**（共享）：多个线程可同时执行，如 Semaphore/CountDownLatch。Semaphore、CountDownLatch、 CyclicBarrier、ReadWriteLock。 + +ReentrantReadWriteLock 可以看成是组合式，因为ReentrantReadWriteLock 也就是读写锁允许多个线程同时对某一资源进行读。 + +不同的自定义同步器争用共享资源的方式也不同。自定义同步器在实现时只需要实现共享资源 state 的获取与释放方式即可，至于具体线程等待队列的维护（如获取资源失败入队/唤醒出队等），AQS已经在顶层实现好了。 + +**AQS底层使用了模板方法模式** + +同步器的设计是基于模板方法模式的，如果需要自定义同步器一般的方式是这样（模板方法模式很经典的一个应用）： + +1. 使用者继承AbstractQueuedSynchronizer并重写指定的方法。（这些重写方法很简单，无非是对于共享资源state的获取和释放） +2. 将AQS组合在自定义同步组件的实现中，并调用其模板方法，而这些模板方法会调用使用者重写的方法。 + +这和我们以往通过实现接口的方式有很大区别，这是模板方法模式很经典的一个运用。 + +**AQS使用了模板方法模式，自定义同步器时需要重写下面几个AQS提供的模板方法：** + +``` +isHeldExclusively()//该线程是否正在独占资源。只有用到condition才需要去实现它。 +tryAcquire(int)//独占方式。尝试获取资源，成功则返回true，失败则返回false。 +tryRelease(int)//独占方式。尝试释放资源，成功则返回true，失败则返回false。 +tryAcquireShared(int)//共享方式。尝试获取资源。负数表示失败；0表示成功，但没有剩余可用资源；正数表示成功，且有剩余资源。 +tryReleaseShared(int)//共享方式。尝试释放资源，成功则返回true，失败则返回false。 +``` + +默认情况下，每个方法都抛出 `UnsupportedOperationException`。 这些方法的实现必须是内部线程安全的，并且通常应该简短而不是阻塞。AQS类中的其他方法都是final ，所以无法被其他类使用，只有这几个方法可以被其他类使用。 + +以ReentrantLock为例，state初始化为0，表示未锁定状态。A线程lock()时，会调用tryAcquire()独占该锁并将state+1。此后，其他线程再tryAcquire()时就会失败，直到A线程unlock()到state=0（即释放锁）为止，其它线程才有机会获取该锁。当然，释放锁之前，A线程自己是可以重复获取此锁的（state会累加），这就是可重入的概念。但要注意，获取多少次就要释放多么次，这样才能保证state是能回到零态的。 + +再以CountDownLatch以例，任务分为N个子线程去执行，state也初始化为N（注意N要与线程个数一致）。这N个子线程是并行执行的，每个子线程执行完后countDown()一次，state会CAS(Compare and Swap)减1。等到所有子线程都执行完后(即state=0)，会unpark()主调用线程，然后主调用线程就会从await()函数返回，继续后余动作。 + +一般来说，自定义同步器要么是独占方法，要么是共享方式，他们也只需实现`tryAcquire-tryRelease`、`tryAcquireShared-tryReleaseShared`中的一种即可。但AQS也支持自定义同步器同时实现独占和共享两种方式，如`ReentrantReadWriteLock`。 + +推荐两篇 AQS 原理和相关源码分析的文章： + +- http://www.cnblogs.com/waterystone/p/4920797.html +- https://www.cnblogs.com/chengxiao/archive/2017/07/24/7141160.html + + + +**1. 核心组成** + +- **state**：`volatile int state`，表示资源状态（比如锁的占用次数）。 +- **CAS 修改 state**：通过 `Unsafe.compareAndSwapInt` 保证并发更新安全。 +- **CLH 队列**：一个 **双向链表**，失败的线程会被包装成 `Node` 挂到队尾。 +- **Node 状态**：`WAITING`、`SIGNAL` 等，用于协调线程挂起和唤醒。 + +**2. 获取资源流程（acquire）** + +1. 线程尝试通过 `tryAcquire()` 获取资源（由子类实现逻辑，比如独占/共享）。 +2. 如果成功，直接返回；失败就进入 CLH 队列。 +3. 在队列中会自旋判断前驱节点是否释放资源，没释放就会 `LockSupport.park()` 挂起。 + +**3. 释放资源流程（release）** + +1. 线程通过 `tryRelease()` 释放资源（由子类实现）。 +2. 修改 state 成功后，会唤醒队列的下一个节点。 +3. 被唤醒的线程重新尝试获取资源。 + +**4. 独占模式 vs 共享模式** + +- **独占模式**（Exclusive）：同一时刻只能一个线程获取（如 `ReentrantLock`）。 +- **共享模式**（Shared）：同一时刻可以有多个线程获取（如 `Semaphore`、`CountDownLatch`）。 + +**5. 应用场景** + +- **ReentrantLock**：可重入锁，state 代表重入次数。 +- **CountDownLatch**：state 表示计数值，减到 0 时释放所有等待线程。 +- **Semaphore**：state 表示许可数，可以多个线程共享。 + + + +### 🎯 AQS 组件总结 + +“AQS 衍生的同步组件可分为： + +1. 独占模式： + - `ReentrantLock`：可重入锁，支持公平 / 非公平，手动控制加解锁。 + - `ReentrantReadWriteLock`：读写分离，读锁共享、写锁独占。 +2. 共享模式： + - `Semaphore`：信号量，控制并发线程数（如限流）。 + - `CountDownLatch`：倒计时门栓，一次性等待多线程完成。 + - `CyclicBarrier`：循环屏障，可重复使用，等待所有线程同步。 + 选择时需根据场景特性（互斥 / 共享、是否可重复、同步类型）合理选用，例如接口限流用 `Semaphore`，任务汇总用 `CountDownLatch`。” + + + +### 🎯 AQS是如何唤醒下一个线程的？ + +当需要阻塞或者唤醒一个线程的时候，AQS都是使用 LockSupport 这个工具类来完成的。 + +AQS（AbstractQueuedSynchronizer）的核心功能之一是**线程的阻塞与唤醒**。当持有锁的线程释放资源后，AQS 需要精确地唤醒等待队列中的下一个线程，以确保同步逻辑的正确性。下面从源码角度深入分析这一过程： + +一、唤醒线程的触发点 + +AQS 唤醒线程主要发生在两种场景： + +1. **释放锁时**：独占模式下调用 `release()`，共享模式下调用 `releaseShared()`。 +2. **取消等待时**：当线程被中断或超时，会从队列中移除并尝试唤醒后继节点。 + +二、唤醒线程的核心方法：`unparkSuccessor()` + +这是 AQS 唤醒线程的核心实现，其逻辑如下： + +```java +private void unparkSuccessor(Node node) { + // 获取当前节点的等待状态 + int ws = node.waitStatus; + // 如果状态为 SIGNAL(-1) 或 CONDITION(-2) 或 PROPAGATE(-3)，尝试将其设为 0 + if (ws < 0) + compareAndSetWaitStatus(node, ws, 0); + + // 找到有效的后继节点（排除状态为 CANCELLED(1) 的节点） + Node s = node.next; + if (s == null || s.waitStatus > 0) { + s = null; + // 从尾部向前遍历，找到最靠前的有效节点 + for (Node t = tail; t != null && t != node; t = t.prev) { + if (t.waitStatus <= 0) + s = t; + } + } + + // 唤醒找到的有效后继节点 + if (s != null) + LockSupport.unpark(s.thread); +} +``` + + + +### 🎯 AQS 中独占锁和共享锁的操作流程大体描述一下 + +##### **独占锁与共享锁的区别** + +- **独占锁是持有锁的线程释放锁之后才会去唤醒下一个线程。** +- **共享锁是线程获取到锁后，就会去唤醒下一个线程，所以共享锁在获取锁和释放锁的时候都会调用doReleaseShared方法唤醒下一个线程，当然这会受共享线程数量的限制**。 + + + +### 🎯 countDownLatch/CycliBarries/Semaphore使用过吗 + +> 我在项目里用过 JUC 提供的并发工具类，比如： +> +> - **CountDownLatch**：适合一次性的任务协调，我用它来让主线程等待多个子任务完成，比如同时发起多个远程请求，等都返回再做汇总。 +> - **CyclicBarrier**：适合多线程阶段性同步，我用它做过分批数据处理，比如多个线程各自处理一段数据，等都处理完再统一进入下一步。 +> - **Semaphore**：主要用于限流，我用它限制同一时间并发访问的线程数，比如防止缓存击穿时过多请求同时打到数据库。 +> +> 这三者的区别在于：CountDownLatch 一次性、CyclicBarrier 可循环复用，而 Semaphore 更像是限流器。 + +#### CycliBarries + +**作用**：让一组线程互相等待，直到都到达屏障点后再一起执行。 + +**原理**：内部计数器递减，等到 0 时释放所有等待线程，并且可以 **重置（可循环使用）**。 + +**场景**： + +- 多线程分段计算，等所有线程算完阶段结果后，再汇总结果。 +- 比如并行计算矩阵，再合并。 + +```java +public class CyclieBarrierDemo { + + public static void main(String[] args) { + // public CyclicBarrier(int parties, Runnable barrierAction) { + CyclicBarrier cyclicBarrier = new CyclicBarrier(7,()->{ + System.out.println("召唤神龙"); + }); + + for (int i = 1; i < 8; i++) { + final int temp = i; + new Thread(()->{ + System.out.println(Thread.currentThread().getName()+"收集到第"+temp+"颗龙珠"); + + try { + cyclicBarrier.await(); + } catch (InterruptedException e) { + e.printStackTrace(); + } catch (BrokenBarrierException e) { + e.printStackTrace(); + } + },String.valueOf(i)).start(); + } + + } + +} +``` + + + +#### Semaphore + +![img](https://learn.lianglianglee.com/%e4%b8%93%e6%a0%8f/Java%20%e5%b9%b6%e5%8f%91%e7%bc%96%e7%a8%8b%2078%20%e8%ae%b2-%e5%ae%8c/assets/Cgq2xl5fiViAS1xOAADHimTjAp0576.png) + +信号量主要用于两个目的，一个是用于多个共享资源的互斥使用，另一个用于并发线程数的控制。 + +**作用**：控制并发线程数，类似一个「许可证池」。 + +**原理**：内部维护一个许可计数，线程调用 `acquire()` 获取许可，执行完后 `release()` 归还许可。 + +**场景**： + +- 限流，比如限制同一时间最多 10 个线程访问某接口。 +- 资源池控制，比如数据库连接池。 + +```java +/** + * @description: 模拟抢车位 + * @author: starfish + * @data: 2020-04-04 10:29 + **/ +public class SemaphoreDemo { + + public static void main(String[] args) { + + //模拟 3 个车位 + Semaphore semaphore = new Semaphore(3); + + //7 辆车去争抢 + for (int i = 0; i < 7; i++) { + new Thread(()->{ + try { + semaphore.acquire(); //抢到车位 + System.out.println(Thread.currentThread().getName()+"\t抢到车位"); + TimeUnit.SECONDS.sleep(3); + System.out.println(Thread.currentThread().getName()+"\t 停车 3 秒后离开"); + } catch (InterruptedException e) { + e.printStackTrace(); + }finally { + semaphore.release(); + } + System.out.println(Thread.currentThread().getName()+"\t抢到车位"); + },String.valueOf(i)).start(); + } + } +} +``` + +#### CountDownLatch + +> `CountDownLatch` 是一个基于 AQS（AbstractQueuedSynchronizer）的同步工具类，用来让一组线程等待其他线程完成操作。 +> 它内部维护了一个 `volatile state` 作为计数器，初始值由构造函数指定。 +> 每次调用 `countDown()` 就会让计数器减 1，当计数器减到 0 时，所有在 `await()` 上等待的线程都会被唤醒继续执行。 +> 常见场景是：主线程等待多个子任务执行完毕再继续，或者控制某些线程必须等到某个条件完成后才能执行。 + +**作用**：一个或多个线程等待其他线程完成操作。 + +**原理**：内部维护一个计数器，调用 `countDown()` 会让计数器减 1，减到 0 时所有 `await()` 的线程会被唤醒。 + +**场景**： + +- 主线程等待多个子线程执行完再汇总结果。 +- 模拟并发请求，等待多个任务都准备好再一起执行。 + +![img](https://learn.lianglianglee.com/%e4%b8%93%e6%a0%8f/Java%20%e5%b9%b6%e5%8f%91%e7%bc%96%e7%a8%8b%2078%20%e8%ae%b2-%e5%ae%8c/assets/Cgq2xl5h8oSAKLBQAABld2EcD7Q385.png) + + + + + +## 六、线程池详解（任务调度核心）🏊 + +> 线程池原理，拒绝策略，核心线程数 +> +> 为什么要用线程池，优势是什么？ +> +> 线程池的工作原理，几个重要参数，给了具体几个参数分析线程池会怎么做，阻塞队列的作用是什么？ +> +> 说说几种常见的线程池及使用场景? +> +> 线程池的构造类的方法的 5 个参数的具体意义是什么 +> +> 按线程池内部机制，当提交新任务时，有哪些异常要考虑 +> +> 单机上一个线程池正在处理服务，如果忽然断电怎么办（正在处理和阻塞队列里的请求怎么处理）？ +> +> 生产上如何合理设置参数？ + + + +### 🎯 为什么要用线程池，优势是什么？ + +> **池化技术相比大家已经屡见不鲜了，线程池、数据库连接池、Http 连接池等等都是对这个思想的应用。池化技术的思想主要是为了减少每次获取资源的消耗，提高对资源的利用率。** +> +> 如果每个任务都创建一个线程会带来哪些问题： +> +> 1. 第一点，反复创建线程系统开销比较大，每个线程创建和销毁都需要时间，如果任务比较简单，那么就有可能导致创建和销毁线程消耗的资源比线程执行任务本身消耗的资源还要大。 +> 2. 第二点，过多的线程会占用过多的内存等资源，还会带来过多的上下文切换，同时还会导致系统不稳定。 + +线程池是一种基于池化思想管理线程的工具。 + +线程池做的工作主要是控制运行的线程数量，处理过程中将任务放入队列，然后在线程创建后启动这些任务，如果线程数量超过了最大数量，超出数量的线程排队等候，等其他线程执行完毕，再从队列中取出任务来执行。 + +主要优点： + +1. **降低资源消耗**：线程复用，通过重复利用已创建的线程减低线程创建和销毁造成的消耗 +2. **提高响应速度**：当任务到达时，任务可以不需要等到线程创建就能立即执行 +3. **提高线程的可管理性**：线程是稀缺资源，如果无限制的创建，不仅会消耗系统资源，还会降低系统的稳定性，使用线程池可以进行统一的分配，调优和监控。 +4. **提供更多更强大的功能**：线程池具备可拓展性，允许开发人员向其中增加更多的功能。比如延时定时线程池ScheduledThreadPoolExecutor，就允许任务延期执行或定期执行。 + + + +### 🎯 Java 并发类库提供的线程池有哪几种？ 分别有什么特点？ + +- `FixedThreadPool`：固定线程数，无界队列，适合稳定负载； +- `SingleThreadExecutor`：单线程顺序执行，避免竞争； +- `CachedThreadPool`：动态创建线程，适合短任务；线程数不固定，可动态创建新线程（最大为 Integer.MAX_VALUE）。工作队列是 **SynchronousQueue（无存储能力）** +- `ScheduledThreadPool`：支持定时 / 周期任务；工作队列是 **DelayedWorkQueue**，按任务执行时间排序 +- `WorkStealingPool`（Java 8+）：基于 ForkJoinPool，利用工作窃取算法提升多核性能。内部使用 **双端队列（WorkQueue）**，任务按 LIFO 顺序执行。 + +实际开发中建议自定义 `ThreadPoolExecutor`，避免无界队列导致 OOM，根据任务类型（CPU/IO 密集）设置核心参数 + + + +### 🎯 线程池的几个重要参数？ + +常用的构造线程池方法其实最后都是通过 **ThreadPoolExecutor** 实例来创建的，且该构造器有 7 大参数。 + +```java +public ThreadPoolExecutor(int corePoolSize, + int maximumPoolSize, + long keepAliveTime, + TimeUnit unit, + BlockingQueue workQueue, + ThreadFactory threadFactory, + RejectedExecutionHandler handler) {//...} +``` + +- **`corePoolSize`（核心线程数）** + + - 线程池初始创建时的线程数量，即使线程空闲也不会被销毁（除非设置 `allowCoreThreadTimeOut` 为 `true`） + - 创建线程池后，当有请求任务进来之后，就会安排池中的线程去执行请求任务，近似理解为近日当值线程 + - 当线程池中的线程数目达到 corePoolSize 后，就会把到达的任务放到缓存队列中 + +- **`maximumPoolSize`（最大线程数）**： 线程池允许创建的最大线程数量，必须 ≥ `corePoolSize`。 + +- **`keepAliveTime`（线程存活时间）**： 非核心线程（超过 `corePoolSize` 的线程）在空闲时的存活时间 + +- **`unit`（时间单位）**： `keepAliveTime` 的时间单位（如 `TimeUnit.SECONDS`、`MILLISECONDS`） + +- **`workQueue`（工作队列）**： 存储等待执行的任务，必须是 `BlockingQueue` 实现类 + +- **`threadFactory`（线程工厂）**：用于设置创建线程的工厂，可以给创建的线程设置有意义的名字，可方便排查问题 + +- **`handler`（拒绝策略）**：拒绝策略，表示当队列满了且工作线程大于等于线程池的最大线程数（maximumPoolSize）时如何来拒绝请求执行的线程的策略，主要有四种类型。 + + 等待队列也已经满了，再也塞不下新任务。同时，线程池中的 max 线程也达到了，无法继续为新任务服务，这时候我们就需要拒绝策略合理的处理这个问题了。 + + - AbortPolicy 直接抛出 RegectedExcutionException 异常阻止系统正常进行，**默认策略** + - DiscardPolicy 直接丢弃任务，不予任何处理也不抛出异常，如果允许任务丢失，这是最好的一种方案 + - DiscardOldestPolicy 抛弃队列中等待最久的任务，然后把当前任务加入队列中尝试再次提交当前任务 + - CallerRunsPolicy 交给线程池调用所在的线程进行处理，“调用者运行”的一种调节机制，该策略既不会抛弃任务，也不会抛出异常，而是将某些任务回退到调用者，从而降低新任务的流量 + + 以上内置拒绝策略均实现了 `RejectExcutionHandler` 接口 + + + +### 🎯 线程池工作原理？ + +![Java线程池实现原理及其在美团业务中的实践- 美团技术团队](https://p0.meituan.net/travelcube/77441586f6b312a54264e3fcf5eebe2663494.png) + +**线程池在内部实际上构建了一个生产者消费者模型，将线程和任务两者解耦，并不直接关联，从而良好的缓冲任务，复用线程**。线程池的运行主要分成两部分：**任务管理、线程管理**。 + +任务管理部分充当生产者的角色，当任务提交后（通过 `execute()` 或 `submit()` 方法提交任务），线程池会判断该任务后续的流转： + +- 直接申请线程执行该任务； +- 缓冲到队列中等待线程执行； +- 拒绝该任务。 + +线程管理部分是消费者角色，它们被统一维护在线程池内，根据任务请求进行线程的分配，当线程执行完任务后则会继续获取新的任务去执行，最终当线程获取不到任务的时候，线程就会被回收。 + +流程： + +1. 在创建线程池后，等待提交过来的任务请求 + +2. 当调用 execute() 方法添加一个请求任务时，线程池会做如下判断： + + - **判断核心线程数**：如果正在运行的线程数量小于 corePoolSize，那么马上创建线程运行这个任务（即使有空闲线程） + - 示例：核心线程数为 5，前 5 个任务会立即创建 5 个线程执行。 + - **判断工作队列**：如果正在运行的线程数量大于或等于 corePoolSize，任务进入 **工作队列（workQueue）** 等待 + - 若队列为 **无界队列**（如 `LinkedBlockingQueue`），任务会无限排队，`maximumPoolSize` 失效。 + - 若队列为 **有界队列**（如 `ArrayBlockingQueue`），队列满时进入下一步。 + - **判断最大线程数**：如果这个时候队列已满且线程数 < `maximumPoolSize`，**创建非核心线程执行任务** + - 示例：核心线程数 5，最大线程数 10，队列容量 100。当提交第 106 个任务时（前 5 个线程 + 100 个队列任务），创建第 6 个线程。 + - **触发拒绝策略**：如果队列满了且正在运行的线程数量大于或等于 maximumPoolSize，那么线程池**会启动饱和拒绝策略来执行** + - 若队列已满且线程数 ≥ `maximumPoolSize`，调用 `RejectedExecutionHandler` 处理任务。 + - 默认策略 `AbortPolicy` 直接抛异常，其他策略包括回退给调用者（`CallerRunsPolicy`）、丢弃最老任务（`DiscardOldestPolicy`）等。 + + ``` + 提交任务 → 线程数 < corePoolSize？→ 是：创建核心线程执行 + ↓ 否 + 队列未满？→ 是：入队等待 + ↓ 否 + 线程数 < maxPoolSize？→ 是：创建非核心线程执行 + ↓ 否 + 触发拒绝策略 + ``` + +3. 当一个线程完成任务时，它会从队列中取下一个任务来执行 + +4. 当一个线程无事可做超过一定的时间（keepAliveTime）时，线程池会判断： + + - 如果当前运行的线程数大于 corePoolSize，那么这个线程就被停掉 + - 所以线程池的所有任务完成后它**最终会收缩到 corePoolSize 的大小** + +>在线程池中，同一个线程可以从 BlockingQueue 中不断提取新任务来执行，其核心原理在于线程池对 Thread 进行了封装，并不是每次执行任务都会调用 Thread.start() 来创建新线程，而是让每个线程去执行一个“循环任务”，在这个“循环任务”中，不停地检查是否还有任务等待被执行，如果有则直接去执行这个任务，也就是调用任务的 run 方法，把 run 方法当作和普通方法一样的地位去调用，相当于把每个任务的 run() 方法串联了起来，所以线程数量并不增加。 + +## 🎯 线程生命周期管理 + +线程池中的线程通过 `Worker` 类封装，其生命周期如下： + +1. **Worker 初始化** + - `Worker` 继承 `AbstractQueuedSynchronizer`（AQS），实现锁机制，避免任务执行期间被中断。 + - 每个 `Worker` 持有一个 `Thread`，启动时执行 `runWorker()` 方法。 +2. **任务循环执行** + - `runWorker()`方法通过 `getTask()`从队列获取任务： + - 若为核心线程，`getTask()` 会阻塞等待（除非 `allowCoreThreadTimeOut=true`）。 + - 若非核心线程，`getTask()` 超时（`keepAliveTime`）后返回 `null`，线程终止。 +3. **线程回收** + - 当 `getTask()` 返回 `null` 时，`runWorker()` 退出循环，`Worker` 被移除，线程销毁。 + - 最终线程池收缩到 `corePoolSize` 大小（除非设置核心线程超时）。 + +#### **源码级机制解析** + +1. **线程池状态与线程数的原子管理** + + - 线程池使用一个 `AtomicInteger`变量 `ctl` 同时存储线程池状态和当前线程数： + + ```java + // ctl 的高 3 位表示状态，低 29 位表示线程数 + private final AtomicInteger ctl = new AtomicInteger(ctlOf(RUNNING, 0)); + ``` + + - 状态包括：`RUNNING`（接收新任务）、`SHUTDOWN`（不接收新任务但处理队列任务）、`STOP`（不接收新任务且不处理队列任务）等。 + +2. **任务窃取与阻塞唤醒** + + - 线程池使用 `ReentrantLock` 保护内部状态，通过 `Condition` 实现线程间通信。 + - 当队列为空时，线程通过 `notEmpty.await()` 阻塞；当有新任务入队时，通过 `notEmpty.signal()` 唤醒等待线程。 + +3. **动态调整线程数** + + - 线程池提供 `setCorePoolSize()` 和 `setMaximumPoolSize()` 方法动态调整参数，适应负载变化。 + + + +### 🎯 Java线程池，5核心、10最大、20队列，第6个任务来了是什么状态？第26个任务来了是什么状态？队列满了以后执行队列的任务是从队列头 or 队尾取？核心线程和非核心线程执行结束后，谁先执行队列里的任务？ + +**问题1：第6个任务的状态** + +当第6个任务到来时，假设前5个任务已经填满了核心线程，线程池的行为如下： + +1. 前5个任务由核心线程处理，核心线程数为 5。 +2. 第6个任务将被放入任务队列中，因为此时队列还没有满。 + +因此，第6个任务将处于等待状态，在任务队列中等待被执行。 + +**问题2：第26个任务的状态** + +当第26个任务到来时，假设前面的任务已经按照规则被处理过，线程池的行为如下： + +1. 核心线程处理了前5个任务。 +2. 任务队列大小为 20，因此第6到第25个任务会被放入队列中。 +3. 当第26个任务到来时，核心线程数为 5，任务队列已经满（20 个任务），此时当前线程数小于最大线程数（10），线程池将创建新的线程来处理任务。 + +因此，第26个任务将由新创建的线程（非核心线程）来处理。 + +- **第 31 个任务**（扩展分析）： + - 此时已创建 5 个核心线程 + 5 个非核心线程（达最大线程数 10） + - 队列已填满 20 个任务 + - 第 31 个任务会触发拒绝策略（默认抛出 RejectedExecutionException） + +**问题3：队列任务的取出顺序** + +Java 中的 `ThreadPoolExecutor` 默认使用 `LinkedBlockingQueue` 作为任务队列，这个队列是一个先进先出（FIFO）的队列。因此，当任务队列中的任务被取出执行时，是从队列头部取出。 + +**问题4：核心线程和非核心线程执行结束后，谁先执行队列里的任务？** + +核心线程与非核心线程在执行完当前任务后，获取队列任务的机制是相同的： + +- 两者都会从队列**头部**获取下一个任务执行 +- 线程池不区分核心线程和非核心线程的任务优先级 +- 非核心线程在空闲时间超过 keepAliveTime 后会被销毁，而核心线程默认会一直存活（可通过 allowCoreThreadTimeOut 参数修改） + +换句话说，线程池会尽量保持核心线程忙碌，并优先使用核心线程来处理任务。当核心线程忙碌时，非核心线程才会处理队列中的任务。 + +> 1. **第 6 个任务**：核心线程已满，任务进入队列（队列大小 = 1），线程数保持 5。 +> 2. **第 26 个任务**：队列已满（20/20），创建第 6 个线程（非核心）执行，线程数 = 6。 +> 3. **队列取出顺序**：默认 FIFO（从队列头部取），除非使用优先级队列。 +> 4. **线程优先级**：核心 / 非核心线程无优先级差异，先空闲的线程先获取任务，但非核心线程可能因超时被回收。 + + + +### 🎯 执行execute()方法和submit()方法的区别是什么呢？ + +1. **execute()方法用于提交不需要返回值的任务，所以无法判断任务是否被线程池执行成功与否；** +2. **submit()方法用于提交需要返回值的任务。线程池会返回一个 Future 类型的对象，通过这个 Future 对象可以判断任务是否执行成功**，并且可以通过 `Future` 的 `get()`方法来获取返回值，`get()`方法会阻塞当前线程直到任务完成，而使用 `get（long timeout，TimeUnit unit）`方法则会阻塞当前线程一段时间后立即返回，这时候有可能任务没有执行完。 + +我们以**`AbstractExecutorService`**接口中的一个 `submit` 方法为例子来看看源代码： + +```java +public Future submit(Runnable task) { + if (task == null) throw new NullPointerException(); + RunnableFuture ftask = newTaskFor(task, null); + execute(ftask); + return ftask; +} +``` + +上面方法调用的 `newTaskFor` 方法返回了一个 `FutureTask` 对象。 + +```java +protected RunnableFuture newTaskFor(Runnable runnable, T value) { + return new FutureTask(runnable, value); +} +``` + +我们再来看看`execute()`方法： + +```java +public void execute(Runnable command) { + ... +} +``` + + + +### 🎯 线程池常用的阻塞队列有哪些？ + +| **阻塞队列类型** | **存储结构** | **有界 / 无界** | **特点** | **适用场景** | +| ------------------------- | ------------------ | ----------------------------------------- | ------------------------------------------------------------ | ------------------------------------ | +| **ArrayBlockingQueue** | 数组 | 有界 | - 初始化时指定容量，满后插入操作阻塞 - 按 FIFO 顺序处理元素 - 支持公平 / 非公平锁（默认非公平） | 任务量可预估、需要控制内存占用的场景 | +| **LinkedBlockingQueue** | 链表 | 可选有界 / 无界（默认 Integer.MAX_VALUE） | - 无界时理论上可存储无限任务 - 按 FIFO 顺序处理元素 - 吞吐量高于 ArrayBlockingQueue | 任务量不确定、希望自动缓冲的场景 | +| **SynchronousQueue** | 不存储元素 | 无界（逻辑上） | - 不存储任何元素，插入操作必须等待消费者接收 - 适合任务与线程直接移交，无缓冲需求 | 要求任务立即执行、避免队列积压的场景 | +| **PriorityBlockingQueue** | 堆结构 | 无界 | - 按元素优先级排序（实现 `Comparable` 或自定义 `Comparator`） - 支持获取优先级最高的任务 | 任务有优先级差异的场景（如紧急任务） | +| **DelayQueue** | 优先队列（基于堆） | 无界 | - 元素需实现 `Delayed` 接口，按延迟时间排序 - 仅到期任务可被取出执行 | 定时任务、延迟执行场景（如超时处理） | + +- 对于 FixedThreadPool 和 SingleThreadExector 而言，它们使用的阻塞队列是容量为 Integer.MAX_VALUE 的 LinkedBlockingQueue，可以认为是无界队列 +- SynchronousQueue，对应的线程池是 CachedThreadPool。线程池 CachedThreadPool 的最大线程数是 Integer 的最大值，可以理解为线程数是可以无限扩展的 +- DelayedWorkQueue，它对应的线程池分别是 ScheduledThreadPool 和 SingleThreadScheduledExecutor，这两种线程池的最大特点就是可以延迟执行任务。DelayedWorkQueue 的特点是内部元素并不是按照放入的时间排序，而是会按照延迟的时间长短对任务进行排序，内部采用的是“堆”的数据结构 + + + +### 🎯 如何创建线程池？ + +> 为什么不应该自动创建线程池？ + +创建线程池应直接使用 `ThreadPoolExecutor` 构造函数，避免 `Executors` 工厂方法的风险： + +1. **拒绝无界队列**：`FixedThreadPool` 默认使用无界队列，可能导致 OOM。 +2. **控制线程数**：`CachedThreadPool` 允许创建无限线程，可能耗尽资源。 +3. **自定义参数**：根据任务特性（CPU/IO 密集）设置核心线程数、队列类型（如有界队列）和拒绝策略（如 `CallerRunsPolicy`）。 +4. **监控与命名**：使用 `ThreadFactory` 命名线程，便于问题排查 + +> 《阿里巴巴Java开发手册》中强制线程池不允许使用 Executors 去创建，而是通过 ThreadPoolExecutor 的方式，这样的处理方式让写的同学更加明确线程池的运行规则，规避资源耗尽的风险 +> +> Executors 返回线程池对象的弊端如下： +> +> - **FixedThreadPool 和 SingleThreadExecutor** ： 允许请求的队列长度为 Integer.MAX_VALUE ，可能堆积大量的请求，从而导致OOM。 +> - **CachedThreadPool 和 ScheduledThreadPool** ： 允许创建的线程数量为 Integer.MAX_VALUE ，可能会创建大量线程，从而导致OOM。 + + + +### 🎯 合理配置线程池你是如何考虑的？（创建多少个线程合适） + +合理配置线程池的核心是确定线程数量，这需要结合任务类型、系统资源、硬件特性等多维度综合考量。 + +**一、线程池核心参数与线程数量的关系** + +线程池的关键参数中，**`corePoolSize`（核心线程数）** 是线程数量配置的核心，它决定了线程池的基础处理能力。而`maximumPoolSize`（最大线程数）则作为流量高峰时的补充，两者需配合队列大小共同调整。 + +**二、任务类型分类与线程数计算** + +根据任务的 IO 密集型、CPU 密集型特性，可采用不同的计算模型： + +1. **CPU 密集型任务（计算密集型）** + + - **特点**：任务主要消耗 CPU 资源（如加密、压缩、数学计算），几乎没有 IO 等待。 + + - 公式：`corePoolSize = CPU核心数 + 1` + + - 解释：CPU 核心数可通过`Runtime.getRuntime().availableProcessors()`获取，+1 是为了应对线程偶发的上下文切换开销，避免 CPU 空闲。 + + > 为什么 +1 呢？ + > + > 《Java并发编程实战》一书中给出的原因是：**即使当计算（CPU）密集型的线程偶尔由于页缺失故障或者其他原因而暂停时，这个“额外”的线程也能确保 CPU 的时钟周期不会被浪费。** + > + > 比如加密、解密、压缩、计算等一系列需要大量耗费 CPU 资源的任务，因为计算任务非常重，会占用大量的 CPU 资源，所以这时 CPU 的每个核心工作基本都是满负荷的，而我们又设置了过多的线程，每个线程都想去利用 CPU 资源来执行自己的任务，这就会造成不必要的上下文切换，此时线程数的增多并没有让性能提升，反而由于线程数量过多会导致性能下降。 + + - **示例**：4 核 CPU 的服务器，核心线程数设为 5。 + +2. **IO 密集型任务（读写 / 网络请求等）** + + IO 密集型则是系统运行时，大部分时间都在进行 I/O 操作，CPU 占用率不高。比如像 MySQL 数据库、文件的读写、网络通信等任务，这类任务**不会特别消耗 CPU 资源，但是 IO 操作比较耗时，会占用比较多时间**。 + + 在单线程上运行 IO 密集型的任务会导致浪费大量的 CPU 运算能力浪费在等待。 + + 所以在 IO 密集型任务中使用多线程可以大大的加速程序运行，即使在单核 CPU 上，这种加速主要就是利用了被浪费调的阻塞时间。 + + IO 密集型时，大部分线程都阻塞，故需要多配置线程数： + + IO 密集型任务： + + - **特点**：任务频繁等待 IO 操作（如数据库查询、文件读写、网络通信），CPU 利用率低。 + + - 公式：这个公式有很多种观点， + + - `CPU 核心数 × 2（IO 等待时线程可复用）` + - `CPU 核心数 × （1 + 平均IO等待时间/平均CPU处理时间）` + - `CPU 核心数 * (1 + 阻塞系数)` + - 解释：IO 等待时间越长，需要越多线程来 “切换执行” 以充分利用 CPU。 + + > 《Java并发编程实战》的作者 Brain Goetz 推荐的计算方法： + > + > ```undefined + > 线程数 = CPU 核心数 *（1+平均等待时间/平均工作时间） + > ``` + > + > 太少的线程数会使得程序整体性能降低，而过多的线程也会消耗内存等其他资源，所以如果想要更准确的话，可以进行压测，监控 JVM 的线程情况以及 CPU 的负载情况，根据实际情况衡量应该创建的线程数，合理并充分利用资源。 + + +3. **混合型任务（兼具 CPU 和 IO 操作）** + + - **方案 1**：拆分为独立线程池，分别处理 CPU 和 IO 任务（推荐）。 + + - **方案 2**：若无法拆分，按 IO 密集型任务计算，并通过监控调整。 + +**三、其他影响因素与实践策略** + +1. **系统资源限制** + + - **内存约束**：线程数过多会导致内存溢出（每个线程默认栈大小约 1MB）。 + + - **IO 资源**：如数据库连接数限制，线程数不应超过数据库最大连接数。 + +2. **任务队列大小** + + - 线程数需与队列容量配合： + - 若`corePoolSize`较小，队列可设为中等大小（如 100），应对流量波动； + - 若`corePoolSize`较大，队列可设为较小值（如 20），避免任务堆积。 + +3. **动态调整策略** + + - **自适应线程池**：通过监控 CPU 利用率、任务队列长度动态调整线程数（如使用`ScheduledExecutorService`定期检测）。 + + - **示例代码**： + + ```java + // 动态线程池实现示例（Spring Boot） + @Bean + public ThreadPoolTaskExecutor dynamicExecutor() { + ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor(); + executor.setCorePoolSize(8); + executor.setMaxPoolSize(32); + executor.setQueueCapacity(1000); + + // 开启监控自动调整 + executor.setAllowCoreThreadTimeOut(true); + executor.setKeepAliveSeconds(30); + + // 添加监控指标 + executor.setThreadPoolExecutor(new ThreadPoolExecutor( + ... // 参数同上 + ) { + protected void afterExecute(Runnable r, Throwable t) { + monitorAndAdjust(); // 监控回调 + } + }); + return executor; + } + + private void monitorAndAdjust() { + // 基于队列堆积情况调整 + if (queueSize > 800) { // 队列堆积警告阈值 + executor.setMaxPoolSize(Math.min(64, executor.getMaxPoolSize() + 4)); + } + else if (queueSize < 200 && executor.getMaxPoolSize() > 32) { + executor.setMaxPoolSize(executor.getMaxPoolSize() - 2); + } + } + ``` + +4. **压测与监控** + + - 压测验证：通过 JMeter 等工具模拟不同并发量，观察线程池的： + - 任务处理耗时（响应时间）； + - CPU、内存利用率； + - 队列堆积情况（是否触发拒绝策略）。 + +- 关键监控指标： + - `taskCount`（总任务数）、`completedTaskCount`（完成任务数）； + - 线程活跃数、队列剩余容量； + - 拒绝任务数（是否触发`RejectedExecutionHandler`）。 + + + +### 🎯 当提交新任务时，异常如何处理? + +1. 在任务代码try/catch捕获异常 + +2. 通过Future对象的get方法接收抛出的异常，再处理 + +3. 为工作者线程设置UncaughtExceptionHandler，在uncaughtException方法中处理异常 + + ```java + ExecutorService threadPool = Executors.newFixedThreadPool(1, r -> { + Thread t = new Thread(r); + t.setUncaughtExceptionHandler( + (t1, e) -> { + System.out.println(t1.getName() + "线程抛出的异常"+e); + }); + return t; + }); + threadPool.execute(()->{ + Object object = null; + System.out.print("result## " + object.toString()); + }); + ``` + +4. 重写ThreadPoolExecutor的afterExecute方法，处理传递的异常引用 + + ```java + class ExtendedExecutor extends ThreadPoolExecutor { + // 这可是jdk文档里面给的例子。。 + protected void afterExecute(Runnable r, Throwable t) { + super.afterExecute(r, t); + if (t == null && r instanceof Future) { + try { + Object result = ((Future) r).get(); + } catch (CancellationException ce) { + t = ce; + } catch (ExecutionException ee) { + t = ee.getCause(); + } catch (InterruptedException ie) { + Thread.currentThread().interrupt(); // ignore/reset + } + } + if (t != null) + System.out.println(t); + } + }} + + ``` + +------ + + + +## 七、Java内存模型（JMM）🧠 + +> 指令重排 +> +> 内存屏障 +> +> 单核CPU有可见性问题吗 + +### 🎯 为什么需要 JMM（Java Memory Model，Java 内存模型）？ + +> "JMM解决了跨平台的内存访问一致性问题： +> +> **解决的问题**： +> +> - 不同处理器内存模型差异 +> - 编译器优化导致的指令重排 +> - 多线程下的可见性、原子性、有序性 +> +> **组成**： +> +> - **主内存**：所有线程共享 +> - **工作内存**：每个线程私有" + +为了理解 Java 内存模型的作用，我们首先就来回顾一下从 Java 代码到最终执行的 CPU 指令的大致流程： + +- 最开始，我们编写的 Java 代码，是 *.java 文件； +- 在编译（包含词法分析、语义分析等步骤）后，在刚才的 *.java 文件之外，会多出一个新的 Java 字节码文件（*.class）； +- JVM 会分析刚才生成的字节码文件（*.class），并根据平台等因素，把字节码文件转化为具体平台上的**机器指令；** +- 机器指令则可以直接在 CPU 上运行，也就是最终的程序执行。 + +所以程序最终执行的效果会依赖于具体的处理器，而不同的处理器的规则又不一样，不同的处理器之间可能差异很大，因此同样的一段代码，可能在处理器 A 上运行正常，而在处理器 B 上运行的结果却不一致。同理，在没有 JMM 之前，不同的 JVM 的实现，也会带来不一样的“翻译”结果。 + +所以 Java 非常需要一个标准，来让 Java 开发者、编译器工程师和 JVM 工程师能够达成一致。达成一致后，我们就可以很清楚的知道什么样的代码最终可以达到什么样的运行效果，让多线程运行结果可以预期，这个标准就是 JMM**，**这就是需要 JMM 的原因。 + +**Java 内存模型（Java Memory Model, JMM）** 是 Java 虚拟机规范中定义的一组规则，规定了 **多线程环境下如何访问共享变量**，以及 **线程之间如何通过内存进行通信**。 + +换句话说：JMM 决定了一个线程写入的变量值，**何时、对哪些线程可见**。 + + + +### 🎯 JMM三大特性 + +| **特性** | **含义** | **实现方式** | +| ---------- | ------------------ | ---------------------- | +| **原子性** | 操作不可分割 | synchronized、Lock | +| **可见性** | 修改对其他线程可见 | volatile、synchronized | +| **有序性** | 禁止指令重排序 | volatile、synchronized | + + + +### 🎯 谈谈 Java 内存模型？ + +Java 虚拟机规范中试图定义一种「 **Java 内存模型**」来**屏蔽掉各种硬件和操作系统的内存访问差异**，以实现**让 Java 程序在各种平台下都能达到一致的内存访问效果** + +**JMM组成**： + +- 主内存：Java 内存模型规定了所有变量都存储在主内存中（此处的主内存与物理硬件的主内存 RAM 名字一样，两者可以互相类比，但此处仅是虚拟机内存的一部分）。 + +- 工作内存：每条线程都有自己的工作内存，线程的工作内存中保存了该线程使用到的主内存中的共享变量的副本拷贝。**线程对变量的所有操作都必须在工作内存进行，而不能直接读写主内存中的变量**。**工作内存是 JMM 的一个抽象概念，并不真实存在**。 + +> 线程之间不能直接访问对方的工作内存，只能通过 **主内存** 传递。 + +**特性**： + +JMM 就是用来解决如上问题的。 **JMM是围绕着并发过程中如何处理可见性、原子性和有序性这 3 个 特征建立起来的** + +- **可见性**：可见性是指当一个线程修改了共享变量的值，其他线程能够立即得知这个修改。 + + - Java 中的 volatile、synchronzied、final 都可以实现可见性 + +- **原子性**：操作是否可以“一次完成，不可分割”。 + + - `synchronized`、`Lock` 保证复合操作原子性；`AtomicInteger` 通过 CAS 保证。 + +- **有序性**： + + 计算机在执行程序时，为了提高性能，编译器和处理器常常会对指令做重排，一般分为以下 3 种 + + ``` + 源代码 -> 编译器优化的重排 -> 指令并行的重排 -> 内存系统的重排 -> 最终执行指令 + ``` + + 单线程环境里确保程序最终执行结果和代码顺序执行的结果一致； + + 处理器在进行重排序时必须要考虑指令之间的**数据依赖性**； + + 多线程环境中线程交替执行，由于编译器优化重排的存在，两个线程中使用的变量能否保证一致性是无法确定的，结果无法预测 + + - `volatile` 禁止指令重排，`synchronized/Lock` 也能保证。 + +> JMM 是不区分 JVM 到底是运行在单核处理器、多核处理器的，Java 内存模型是对 CPU 内存模型的抽象，这是一个 High-Level 的概念，与具体的 CPU 平台没啥关系 + + + +### 🎯 Java 内存模型（JMM）的底层规则？ + +**1.工作内存与主内存的隔离** + +- **主内存**：所有线程共享的公共内存，存储对象实例和类静态变量。 +- **工作内存**：每个线程私有的内存，存储主内存变量的副本（线程对变量的操作必须在工作内存中进行）。 +- **问题**：线程 A 修改工作内存中的变量后，若未同步到主内存，线程 B 的工作内存可能仍持有旧值，导致可见性问题。 + +**2. 变量操作的八大原子指令** + +JMM 定义了以下操作（需成对出现），用于规范主内存与工作内存的交互： + +| **指令** | **作用** | +| -------- | ------------------------------------------------------------ | +| `lock` | 锁定主内存变量，标识为线程独占（一个变量同一时刻只能被一个线程 `lock`）。 | +| `unlock` | 解锁主内存变量，允许其他线程 `lock`。 | +| `read` | 从主内存读取变量值到工作内存。 | +| `load` | 将 `read` 读取的值存入工作内存的变量副本。 | +| `use` | 将工作内存的变量副本值传递给线程的计算引擎（用于运算）。 | +| `assign` | 将计算引擎的结果赋值给工作内存的变量副本。 | +| `store` | 将工作内存的变量副本值传递到主内存，准备写入。 | +| `write` | 将 `store` 传递的值写入主内存的变量。 | + + + +### 🎯 Java 内存模型中的 happen-before 是什么？ + +> happen-before定义操作间的偏序关系： +> +> **核心规则**： +> +> - 程序次序规则：线程内按顺序执行 +> - 锁定规则：unlock happen-before lock +> - volatile规则：写 happen-before 读 +> - 传递性：A→B，B→C，则A→C" + +happens-before 先行发生，是 Java 内存模型中定义的两项操作之间的偏序关系，**如果操作 A 先行发生于操作 B，那么 A 的结果对 B 可见**。 + +内存屏障是被插入两个 CPU 指令之间的一种指令，用来禁止处理器指令发生重排序（像屏障一样），从而保障**有序性**的。 + +Happen-before 关系，是 Java 内存模型中保证多线程操作可见性的机制，也是对早期语言规范中含糊的可见性概念的一个精确定义。 + +它的具体表现形式，包括但远不止是我们直觉中的 synchronized、volatile、lock 操作顺序等方面，例如： + +- 线程内执行的每个操作，都保证 happen-before 后面的操作，这就保证了基本的程序顺序规则，这是开发者在书写程序时的基本约定。 +- 对于 volatile 变量，对它的写操作，保证 happen-before 在随后对该变量的读取操作。 +- 对于一个锁的解锁操作，保证 happen-before 加锁操作。 +- 对象构建完成，保证 happen-before 于 finalizer 的开始动作。 +- 甚至是类似线程内部操作的完成，保证 happen-before 其他 Thread.join() 的线程等。 + +这些 happen-before 关系是存在着传递性的，如果满足 a happen-before b 和 b happen-before c，那么 a happen-before c 也成立。 + +前面我一直用 happen-before，而不是简单说前后，是因为它不仅仅是对执行时间的保证，也包括对内存读、写操作顺序的保证。仅仅是时钟顺序上的先后，并不能保证线程交互的可见性。 + +------ + + + +## 八、并发容器（线程安全集合）📦 + +### 🎯 Java 并发包提供了哪些并发工具类？ + +我们通常所说的并发包也就是 `java.util.concurrent` 及其子包，集中了 Java 并发的各种基础工具类，具体主要包括几个方面： + +- 提供了比 synchronized 更加高级的各种同步结构，包括 CountDownLatch、CyclicBarrier、Semaphore 等，可以实现更加丰富的多线程操作，比如利用 Semaphore 作为资源控制器，限制同时进行工作的线程数量。 +- 各种线程安全的容器，比如最常见的 ConcurrentHashMap、有序的 ConcunrrentSkipListMap，或者通过类似快照机制，实现线程安全的动态数组 CopyOnWriteArrayList 等。 +- 各种并发队列实现，如各种 BlockedQueue 实现，比较典型的 ArrayBlockingQueue、 SynchorousQueue 或针对特定场景的 PriorityBlockingQueue 等。 +- 强大的 Executor 框架，可以创建各种不同类型的线程池，调度任务运行等，绝大部分情况下，不再需要自己从头实现线程池和任务调度器。 + + + +### 🎯 ConcurrentHashMap？ + +**1. 为什么需要它？** + +- `HashMap` 在多线程下不安全，可能导致死循环、数据丢失。 +- `Hashtable` 虽然线程安全，但用 **synchronized 修饰整张表**，并发性能差。 +- `ConcurrentHashMap` 是 **线程安全、高性能** 的 HashMap 实现。 + +**2. JDK1.7 实现** + +- 采用 **分段锁（Segment）** 思想。 +- 整个 Map 分为若干个 Segment，每个 Segment 内部是一个小的 HashMap，操作时只锁住对应 Segment。 +- **优点**：并发度高（默认 16），多个线程可同时访问不同 Segment。 +- **缺点**：内存浪费，结构复杂。 + +**3. JDK1.8 实现（核心）** + +- **去掉分段锁，改用 CAS + synchronized 锁粒度缩小到桶（Node）级别**。 +- 底层依然是 **数组 + 链表/红黑树**。 +- **插入过程 put()**： + 1. 用 CAS 尝试放置新节点，如果位置为空，直接成功。 + 2. 如果失败（位置有冲突），用 synchronized 锁住链表/树的头节点。 + 3. 链表长度 > 8 时转为红黑树，提高查找效率。 +- **扩容**：多线程协作扩容，线程安全，性能更高。 + +**4. 特点** + +- 读操作基本无锁（volatile + CAS 保证可见性）。 +- 写操作用 synchronized，锁粒度小（桶级），不会阻塞全表。 +- 适合 **高并发场景下的缓存、计数器** 等。 + + + +### 🎯 Java 中的同步集合与并发集合有什么区别？ + +同步集合是通过**在集合的每个方法上使用同步锁（`synchronized`）**来确保线程安全的。Java 提供了一些同步集合类，例如： + +- `Collections.synchronizedList(List list)`：返回一个线程安全的 `List` 实现。 +- `Collections.synchronizedMap(Map map)`：返回一个线程安全的 `Map` 实现。 + +这些同步集合类通过对所有操作加锁，来保证只有一个线程能够在同一时刻访问或修改集合。 + +并发集合是 Java 5 引入的，它们是专门为高并发环境设计的，能够在多线程环境中提供更高效的操作。Java 提供了多个并发集合类： + +- **`ConcurrentHashMap`**：线程安全的哈希表，提供高效的并发读写操作。 +- **`CopyOnWriteArrayList`**：适用于读操作远远多于写操作的场景，它在修改时复制整个数组。 +- **`ConcurrentLinkedQueue`**：高效的无界非阻塞并发队列。 + +这些并发集合通过更加细粒度的锁（如分段锁或无锁算法）实现线程安全，避免了同步集合中的全局锁定问题。 + + + +### 🎯 SynchronizedMap 和 ConcurrentHashMap 有什么区别？ + +SynchronizedMap 一次锁住整张表来保证线程安全，所以每次只能有一个线程来访为 map。 + +ConcurrentHashMap 使用分段锁来保证在多线程下的性能。 + +ConcurrentHashMap 中则是一次锁住一个桶。ConcurrentHashMap 默认将 hash 表分为 16 个桶，诸如 get，put，remove 等常用操作只锁当前需要用到的桶。 + +这样，原来只能一个线程进入，现在却能同时有 16 个写线程执行，并发性能的提升是显而易见的。 + +另外 ConcurrentHashMap 使用了一种不同的迭代方式。在这种迭代方式中，当 iterator 被创建后集合再发生改变就不再是抛出ConcurrentModificationException，取而代之的是在改变时 new 新的数据从而不影响原有的数据，iterator 完成后再将头指针替换为新的数据 ，这样 iterator线程可以使用原来老的数据，而写线程也可以并发的完成改变 + + + +### 🎯 CopyOnWriteArrayList 是什么，可以用于什么应用场景？有哪些优缺点？ + +CopyOnWriteArrayList 是一个并发容器。有很多人称它是线程安全的，我认为这句话不严谨，缺少一个前提条件，那就是非复合场景下操作它是线程安全的。 + +CopyOnWriteArrayList(免锁容器)的好处之一是当多个迭代器同时遍历和修改这个列表时，不会抛出 ConcurrentModificationException。在 CopyOnWriteArrayList 中，写入将导致创建整个底层数组的副本，而源数组将保留在原地，使得复制的数组在被修改时，读取操作可以安全地执行。 + +CopyOnWriteArrayList 的使用场景: + +通过源码分析，我们看出它的优缺点比较明显，所以使用场景也就比较明显。就是合适读多写少的场景。 + +CopyOnWriteArrayList 的缺点： + +1. 由于写操作的时候，需要拷贝数组，会消耗内存，如果原数组的内容比较多的情况下，可能导致 young gc 或者 full gc。 +2. 不能用于实时读的场景，像拷贝数组、新增元素都需要时间，所以调用一个 set 操作后，读取到数据可能还是旧的，虽然CopyOnWriteArrayList 能做到最终一致性，但是还是没法满足实时性要求。 +3. 由于实际使用中可能没法保证 CopyOnWriteArrayList 到底要放置多少数据，万一数据稍微有点多，每次 add/set 都要重新复制数组，这个代价实在太高昂了。在高性能的互联网应用中，这种操作分分钟引起故障。 + +CopyOnWriteArrayList 的设计思想： + +1. 读写分离，读和写分开 +2. 最终一致性 +3. 使用另外开辟空间的思路，来解决并发冲突 + + + +### 🎯 并发包中的 ConcurrentLinkedQueue 和 LinkedBlockingQueue 有什么区别？ + +有时候我们把并发包下面的所有容器都习惯叫作并发容器，但是严格来讲，类似 ConcurrentLinkedQueue 这种“Concurrent*”容器，才是真正代表并发。 + +关于问题中它们的区别： + +- Concurrent 类型基于 lock-free，在常见的多线程访问场景，一般可以提供较高吞吐量。 +- 而 LinkedBlockingQueue 内部则是基于锁，并提供了 BlockingQueue 的等待性方法。 + +不知道你有没有注意到，java.util.concurrent 包提供的容器（Queue、List、Set）、Map，从命名上可以大概区分为 Concurrent*、CopyOnWrite*和 Blocking*等三类，同样是线程安全容器，可以简单认为： + +- Concurrent 类型没有类似 CopyOnWrite 之类容器相对较重的修改开销。 +- 但是，凡事都是有代价的，Concurrent 往往提供了较低的遍历一致性。你可以这样理解所谓的弱一致性，例如，当利用迭代器遍历时，如果容器发生修改，迭代器仍然可以继续进行遍历。 +- 与弱一致性对应的，就是我介绍过的同步容器常见的行为“fail-fast”，也就是检测到容器在遍历过程中发生了修改，则抛出 ConcurrentModificationException，不再继续遍历。 +- 弱一致性的另外一个体现是，size 等操作准确性是有限的，未必是 100% 准确。 +- 与此同时，读取的性能具有一定的不确定性。 + + + +### 🎯 什么是阻塞队列？阻塞队列的实现原理是什么？如何使用阻塞队列来实现生产者-消费者模型？ + +> - 哪些队列是有界的，哪些是无界的？ +> - 针对特定场景需求，如何选择合适的队列实现？ +> - 从源码的角度，常见的线程安全队列是如何实现的，并进行了哪些改进以提高性能表现？ + +阻塞队列（BlockingQueue）是一个支持两个附加操作的队列。 + +这两个附加的操作是： + +- 在队列为空时，获取元素的线程会等待队列变为非空。 +- 当队列满时，存储元素的线程会等待队列可用。 + +阻塞队列常用于生产者和消费者的场景，生产者是往队列里添加元素的线程，消费者是从队列里拿元素的线程。阻塞队列就是生产者存放元素的容器，而消费者也只从容器里拿元素。 + +JDK7 提供了 7 个阻塞队列。分别是： + +- ArrayBlockingQueue ：一个由数组结构组成的有界阻塞队列。 +- LinkedBlockingQueue ：一个由链表结构组成的有界阻塞队列。 +- PriorityBlockingQueue ：一个支持优先级排序的无界阻塞队列。 +- DelayQueue：一个使用优先级队列实现的无界阻塞队列。 +- SynchronousQueue：一个不存储元素的阻塞队列。 +- LinkedTransferQueue：一个由链表结构组成的无界阻塞队列。 +- LinkedBlockingDeque：一个由链表结构组成的双向阻塞队列。 + +Java 5 之前实现同步存取时，可以使用普通的一个集合，然后在使用线程的协作和线程同步可以实现生产者，消费者模式，主要的技术就是用好，wait，notify，notifyAll，sychronized 这些关键字。而在 java 5 之后，可以使用阻塞队列来实现，此方式大大简少了代码量，使得多线程编程更加容易，安全方面也有保障。 + +BlockingQueue 接口是 Queue 的子接口，它的主要用途并不是作为容器，而是作为线程同步的的工具，因此他具有一个很明显的特性，当生产者线程试图向 BlockingQueue 放入元素时，如果队列已满，则线程被阻塞，当消费者线程试图从中取出一个元素时，如果队列为空，则该线程会被阻塞，正是因为它所具有这个特性，所以在程序中多个线程交替向 BlockingQueue 中放入元素，取出元素，它可以很好的控制线程之间的通信。 + +阻塞队列使用最经典的场景就是 socket 客户端数据的读取和解析，读取数据的线程不断将数据放入队列，然后解析线程不断从队列取数据解析。 + + + +### 🎯 如何设计一个阻塞队列，都需要考虑哪些点? + +> 要是让你用数组实现一个阻塞队列该怎么实现（ArrayBlockQueue） +> +> 手写阻塞队列的add 和take方法 + +阻塞队列相比于普通队列，区别在于当队列为空时获取被阻塞，当队列为满时插入被阻塞。 + +这个问题其实涉及到两个点 + +- 阻塞方式：使用互斥锁（synchronized、 Lock）来保护队列操作，如果队列满了 我们用wait、sleep +- 不阻塞后，需要有线程通信机制（notify、notifyAll 或者 condition） + +当然，肯定会有队列的实现，list 或者 linkedlist 都可以 + +```java +public class CustomBlockQueue { + //队列容器 + private List container = new ArrayList<>(); + private Lock lock = new ReentrantLock(); + //Condition + // 队列为空 + private Condition isNull = lock.newCondition(); + // 队列已满 + private Condition isFull = lock.newCondition(); + private volatile int size; + private volatile int capacity; + + CustomBlockQueue(int cap) { + this.capacity = cap; + } + + public void add(int data) { + try { + lock.lock(); + try { + while (size >= capacity) { + System.out.println("队列已满，释放锁，等待消费者消费数据"); + isFull.await(); + } + } catch (InterruptedException e) { + isFull.signal(); + e.printStackTrace(); + } + ++size; + container.add(data); + isNull.signal(); + } finally { + lock.unlock(); + } + } + + public int take(){ + try { + lock.lock(); + try { + while (size == 0){ + System.out.println("阻塞队列空了，释放锁，等待生产者生产数据"); + isNull.await(); + } + }catch (InterruptedException e){ + isFull.signal(); + e.printStackTrace(); + } + --size; + int res = container.get(0); + container.remove(0); + isFull.signal(); + return res ; + }finally { + lock.unlock(); + } + } +} +``` + + + +### 🎯 有哪些线程安全的非阻塞队列？ + +ConcurrentLinkedQueue是一个基于链接节点的无界线程安全队列，它采用先进先出的规则对节点进行排序，当我们添加一个元素的时候，它会添加到队列的尾部；当我们获取一个元素时，它会返回队列头部的元素。 + +结构如下： + +```java +public class ConcurrentLinkedQueue extends AbstractQueue + implements Queue, java.io.Serializable { + private transient volatile Node head;//头指针 + private transient volatile Node tail;//尾指针 + public ConcurrentLinkedQueue() {//初始化，head=tail=（一个空的头结点） + head = tail = new Node(null); + } + private static class Node { + volatile E item; + volatile Node next;//内部是使用单向链表实现 + ...... + } + ...... +} +``` + +入队和出队操作均利用CAS（compare and set）更新，这样允许多个线程并发执行，并且不会因为加锁而阻塞线程，使得并发性能更好。 + + + +### 🎯 ThreadLocal 是什么？有哪些使用场景？ + +> 比如有两个人去宝屋收集宝物，这两个共用一个袋子的话肯定会产生争执，但是给他们两个人每个人分配一个袋子的话就不会出现这样的问题。如果把这两个人比作线程的话，那么ThreadLocal就是用来避免这两个线程竞争的。 + +> `ThreadLocal` 是 JDK 提供的一种线程本地变量，它为每个线程都维护了一份独立的副本，线程之间互不干扰。 +> 它的典型应用场景包括： +> +> - **保存用户会话信息**（如用户 ID、请求上下文） +> - **数据库连接、Session 管理**（避免频繁传参） +> - **日期格式化器**（如 `SimpleDateFormat` 线程不安全，可以用 ThreadLocal 保存每线程独立实例） +> +> 需要注意： +> +> - `ThreadLocal` 不是用来解决共享变量的并发问题，而是让变量在 **当前线程独享**。 +> - 使用后要注意 **内存泄漏风险**，尤其在线程池环境下，使用完需要调用 `remove()`。 + +ThreadLocal 是 Java 提供的一个线程局部变量工具类，它允许我们创建只能被同一个线程读写的变量。ThreadLocal 提供了线程安全的共享变量，每个线程都可以独立地改变自己的副本，而不会影响其他线程的副本。 + +**1. 基本原理** + +- 每个线程内部有一个 `ThreadLocalMap`，类似一个以 `ThreadLocal` 对象为 key 的哈希表。 +- 当调用 `threadLocal.set(value)` 时，实际上是把 `value` 存入当前线程的 `ThreadLocalMap`。 +- `threadLocal.get()` 会从当前线程的 `ThreadLocalMap` 中取值。 +- 这样每个线程访问到的值，都是独立副本，互不影响。 + +**2. 重要特点** + +- **线程隔离**：每个线程有自己的副本，不会出现并发修改冲突。 +- **生命周期**：绑定在线程上，线程结束后才会释放。 +- **内存泄漏问题**： + - `ThreadLocalMap` 的 key 是 `ThreadLocal` 的弱引用，value 是强引用。 + - 如果 ThreadLocal 对象被回收了，而线程还活着，value 可能会泄漏。 + - 因此建议手动调用 `remove()` 清理。 + +**3. 常见使用场景** + +- **Web 开发**：存储用户信息、请求上下文，避免参数层层传递。 +- **数据库操作**：为每个线程维护独立的数据库连接或事务对象。 +- **工具类**：比如 `SimpleDateFormat`、`NumberFormat` 等非线程安全类的实例。 +- **日志跟踪**：在链路调用中保存 traceId，方便日志打印与追踪。 + +> 使用场景： +> +> 1. 线程安全的单例模式： 在多线程环境下，可以使用 ThreadLocal 来实现线程安全的单例模式，每个线程都持有对象的一个副本。 +> 2. 存储用户身份信息： 在 Web 应用中，可以使用 ThreadLocal 来存储用户的登录信息或 Session 信息，使得这些信息在同一线程的不同方法中都可以访问，而不需要显式地传递参数。 +> 3. 数据库连接管理： 在某些数据库连接池的实现中，可以使用 ThreadLocal 来存储当前线程持有的数据库连接，确保事务中使用的是同一个连接。 +> 4. 解决线程安全问题： 在一些非线程安全的工具类中（如 SimpleDateFormat），可以使用 ThreadLocal 来为每个线程创建一个独立的实例，避免并发问题。 +> 5. 跨函数传递数据： 当某些数据需要在同一线程的多个方法中传递，但又不适合作为方法参数时，可以考虑使用 ThreadLocal。 +> 6. 全局存储线程内数据： 在一些复杂的系统中，可能需要在线程的整个生命周期内存储一些数据，ThreadLocal 提供了一种优雅的解决方案。 +> 7. 性能优化： 在一些需要频繁创建和销毁对象的场景，可以使用 ThreadLocal 来重用这些对象，减少创建和销毁的开销。 + + + +### 🎯 ThreadLocal 是用来解决共享资源的多线程访问的问题吗？ + +不是，ThreadLocal 并不是用来解决共享资源的多线程访问问题的，而是用来解决线程间数据隔离的问题. + +虽然 ThreadLocal 确实可以用于解决多线程情况下的线程安全问题，但其资源并不是共享的，而是每个线程独享的。 + +如果我们把放到 ThreadLocal 中的资源用 static 修饰，让它变成一个共享资源的话，那么即便使用了 ThreadLocal，同样也会有线程安全问题 + +### 🎯 ThreadLocal原理？ + +> ThreadLocal 的核心是为每个线程维护一份独立的变量副本，实现线程隔离。 +> 实现原理是：每个 Thread 内部都有一个 ThreadLocalMap，存放以 ThreadLocal 实例为 key、具体值为 value 的数据。我们调用 set/get，其实就是往当前线程的 ThreadLocalMap 存取数据。 +> 这样同一个 ThreadLocal 在不同线程里查到的数据互不干扰，从而避免了线程安全问题。 +> 值得注意的是，ThreadLocalMap 的 key 是弱引用，如果没有及时调用 remove 清理，在使用线程池时容易导致内存泄漏，这是 ThreadLocal 使用上的一个坑。 + +当使用 ThreadLocal 维护变量时，其为每个使用该变量的线程提供独立的变量副本，所以每一个线程都可以独立的改变自己的副本，而不会影响其他线程对应的副本。 + +ThreadLocal 内部实现机制： + +- 每个线程内部都会维护一个类似 HashMap 的对象，称为 ThreadLocalMap，里边会包含若干了 Entry（K-V 键值对），相应的线程被称为这些 Entry 的属主线程； +- Entry 的 Key 是一个 ThreadLocal 实例，Value 是一个线程特有对象。Entry 的作用即是：为其属主线程建立起一个 ThreadLocal 实例与一个线程特有对象之间的对应关系； +- Entry 对 Key 的引用是弱引用；Entry 对 Value 的引用是强引用。 + +从 `Thread`类源代码入手。 + +```java +public class Thread implements Runnable { + ...... +//与此线程有关的ThreadLocal值。由ThreadLocal类维护 +ThreadLocal.ThreadLocalMap threadLocals = null; +//与此线程有关的InheritableThreadLocal值。由InheritableThreadLocal类维护 +ThreadLocal.ThreadLocalMap inheritableThreadLocals = null; + ...... +} +``` -线程池的几个重要参数： +从上面`Thread`类 源代码可以看出`Thread` 类中有一个 `threadLocals` 和 一个 `inheritableThreadLocals` 变量，它们都是 `ThreadLocalMap` 类型的变量,我们可以把 `ThreadLocalMap` 理解为`ThreadLocal` 类实现的定制化的 `HashMap`。默认情况下这两个变量都是null，只有当前线程调用 `ThreadLocal` 类的 `set`或`get`方法时才创建它们，实际上调用这两个方法的时候，我们调用的是`ThreadLocalMap`类对应的 `get()`、`set() `方法。 -常用的构造线程池方法其实最后都是通过 **ThreadPoolExecutor** 实例来创建的，且该构造器有 7 大参数。 +`ThreadLocal`类的`set()`方法 ```java -public ThreadPoolExecutor(int corePoolSize, - int maximumPoolSize, - long keepAliveTime, - TimeUnit unit, - BlockingQueue workQueue, - ThreadFactory threadFactory, - RejectedExecutionHandler handler) {//...} + public void set(T value) { + Thread t = Thread.currentThread(); + ThreadLocalMap map = getMap(t); + if (map != null) + map.set(this, value); + else + createMap(t, value); + } + ThreadLocalMap getMap(Thread t) { + return t.threadLocals; + } ``` -- **corePoolSize：** 线程池中的常驻核心线程数 - - - 创建线程池后，当有请求任务进来之后，就会安排池中的线程去执行请求任务，近似理解为近日当值线程 - - 当线程池中的线程数目达到corePoolSize后，就会把到达的任务放到缓存队列中 - -- **maximumPoolSize：** 线程池最大线程数大小，该值必须大于等于 1 +通过上面这些内容，我们足以通过猜测得出结论：**最终的变量是放在了当前线程的 ThreadLocalMap 中，并不是存在 ThreadLocal 上，ThreadLocal 可以理解为只是ThreadLocalMap的封装，传递了变量值。** `ThrealLocal` 类中可以通过`Thread.currentThread()`获取到当前线程对象后，直接通过`getMap(Thread t)`可以访问到该线程的`ThreadLocalMap`对象。 -- **keepAliveTime：** 线程池中非核心线程空闲的存活时间 +**每个Thread中都具备一个ThreadLocalMap，而ThreadLocalMap可以存储以ThreadLocal为key ，Object 对象为 value的键值对。** - - 当前线程池数量超过 corePoolSize 时，当空闲时间达到 keepAliveTime 值时，非核心线程会被销毁直到只剩下 corePoolSize 个线程为止 +```java +ThreadLocalMap(ThreadLocal firstKey, Object firstValue) { + ...... +} +``` -- **unit：** keepAliveTime 的时间单位 +比如我们在同一个线程中声明了两个 `ThreadLocal` 对象的话，会使用 `Thread`内部都是使用仅有那个`ThreadLocalMap` 存放数据的，`ThreadLocalMap`的 key 就是 `ThreadLocal`对象，value 就是 `ThreadLocal` 对象调用`set`方法设置的值。 -- **workQueue：** 存放任务的阻塞队列，被提交但尚未被执行的任务 +`ThreadLocalMap`是`ThreadLocal`的静态内部类。 -- **threadFactory：** 用于设置创建线程的工厂，可以给创建的线程设置有意义的名字，可方便排查问题 -- **handler：** 拒绝策略，表示当队列满了且工作线程大于等于线程池的最大线程数（maximumPoolSize）时如何来拒绝请求执行的线程的策略，主要有四种类型。 - 等待队列也已经满了，再也塞不下新任务。同时，线程池中的 max 线程也达到了，无法继续为新任务服务，这时候我们就需要拒绝策略合理的处理这个问题了。 +### 🎯 什么是线程局部变量？ - - AbortPolicy 直接抛出RegectedExcutionException 异常阻止系统正常进行，**默认策略** - - DiscardPolicy 直接丢弃任务，不予任何处理也不抛出异常，如果允许任务丢失，这是最好的一种方案 - - DiscardOldestPolicy 抛弃队列中等待最久的任务，然后把当前任务加入队列中尝试再次提交当前任务 - - CallerRunsPolicy 交给线程池调用所在的线程进行处理，“调用者运行”的一种调节机制，该策略既不会抛弃任务，也不会抛出异常，而是将某些任务回退到调用者，从而降低新任务的流量 +线程局部变量是局限于线程内部的变量，属于线程自身所有，不在多个线程间共享。Java 提供 ThreadLocal 类来支持线程局部变量，是一种实现线程安全的方式。但是在管理环境下（如 web 服务器）使用线程局部变量的时候要特别小心，在这种情况下，工作线程的生命周期比任何应用变量的生命周期都要长。任何线程局部变量一旦在工作完成后没有释放，Java 应用就存在内存泄露的风险。 - 以上内置拒绝策略均实现了 RejectExcutionHandler 接口 +### 🎯 ThreadLocal 内存泄露问题? -工作原理： +`ThreadLocalMap` 中使用的 key 为 `ThreadLocal` 的弱引用，而 value 是强引用。所以，如果 `ThreadLocal` 没有被外部强引用的情况下，在垃圾回收的时候，key 会被清理掉，而 value 不会被清理掉。这样一来，`ThreadLocalMap` 中就会出现key为null的Entry。假如我们不做任何措施的话，value 永远无法被GC 回收，这个时候就可能会产生内存泄露。ThreadLocalMap实现中已经考虑了这种情况，在调用 `set()`、`get()`、`remove()` 方法的时候，会清理掉 key 为 null 的记录。使用完 `ThreadLocal`方法后 最好手动调用`remove()`方法 -线程池在内部实际上构建了一个生产者消费者模型，将线程和任务两者解耦，并不直接关联，从而良好的缓冲任务，复用线程。线程池的运行主要分成两部分：**任务管理、线程管理**。任务管理部分充当生产者的角色，当任务提交后，线程池会判断该任务后续的流转： +```java +static class Entry extends WeakReference> { + /** The value associated with this ThreadLocal. */ + Object value; + + Entry(ThreadLocal k, Object v) { + super(k); + value = v; + } +} +``` -- 直接申请线程执行该任务； -- 缓冲到队列中等待线程执行； -- 拒绝该任务。 -线程管理部分是消费者，它们被统一维护在线程池内，根据任务请求进行线程的分配，当线程执行完任务后则会继续获取新的任务去执行，最终当线程获取不到任务的时候，线程就会被回收。 -流程： +### 🎯 ThreadLocalMap 的 Entry 为什么 key 用弱引用，而 value 却是强引用？ -1. 在创建线程池后，等待提交过来的任务请求 +> ThreadLocalMap 的 key 用弱引用是为了避免 ThreadLocal 对象本身无法被回收，特别是在线程池里，否则会造成严重内存泄漏。而 value 用强引用是因为它承载业务数据，必须保证线程生命周期内数据可用。如果 value 也设成弱引用，可能在 GC 时被提前回收，导致业务逻辑拿不到数据。不过这种设计也带来了隐患：当 key 被 GC 回收后，value 仍然强引用存在，形成内存泄漏，所以正确的使用姿势是用完 ThreadLocal 后调用 remove()。 -2. 当调用 execute() 方法添加一个请求任务时，线程池会做如下判断： +**ThreadLocalMap 的 Entry 设计** - - 如果正在运行的线程数量小于 corePoolSize，那么马上创建线程运行这个任务 - - 如果正在运行的线程数量大于或等于 corePoolSize，那么将这个任务**放入队列** - - 如果这个时候队列满了且正在运行的线程数量还小于 maximumPoolSize，那么创建非核心线程立刻运行这个任务 - - 如果队列满了且正在运行的线程数量大于或等于 maximumPoolSize，那么线程池**会启动饱和拒绝策略来执行** +在 `ThreadLocalMap` 里，`Entry` 的定义大致是这样的： -3. 当一个线程完成任务时，它会从队列中取下一个任务来执行 +```java +static class Entry extends WeakReference> { + Object value; + Entry(ThreadLocal k, Object v) { + super(k); + value = v; + } +} +``` -4. 当一个线程无事可做超过一定的时间（keepAliveTime）时，线程池会判断： +也就是说： - - 如果当前运行的线程数大于 corePoolSize，那么这个线程就被停掉 - - 所以线程池的所有任务完成后它**最终会收缩到 corePoolSize 的大小** +- **key（ThreadLocal 对象）** → 弱引用 +- **value（实际存储的对象）** → 强引用 +**为什么 key 是弱引用？** +1. **避免内存泄漏** + - 如果 key 是强引用，即使业务代码不再持有 `ThreadLocal` 对象，Map 里的引用仍然会让它无法被 GC 回收。 + - 设置成 **弱引用**，只要外部不再持有 ThreadLocal 对象，GC 就可以回收 key。 +2. **线程池场景下更重要** + - 线程池里的线程不会销毁，它们持有的 `ThreadLocalMap` 生命周期往往很长。 + - 如果 key 是强引用，会导致 key 和 value 一直存活，造成严重内存泄漏。 -合理配置线程池（创建多少个线程合适）： +**为什么 value 是强引用？** -- CPU 密集型 +1. **存储的业务数据需要正常使用** + - value 一般是我们真正要存的数据，比如用户信息、事务上下文。 + - 如果也用弱引用，GC 会在内存紧张时随时回收，业务代码再去 `get()` 时可能直接拿不到值，违背了 ThreadLocal 的设计初衷。 +2. **避免数据过早丢失** + - ThreadLocal 的设计目标是“为线程保存变量副本”。 + - 如果 value 也弱引用，那存进去的数据没法保证生命周期，会让 ThreadLocal 失去意义。 - CPU 密集的意思是该任务需要大量的运算，而没有阻塞，CPU 一直全速运行 +------ - CPU 密集任务只有在真正的多核 CPU 上才可能得到加速（通过多线程） - 而在单核 CPU 上，无论开几个模拟的多线程该任务都不可能得到加速，因为 CPU 总的运算能力就那些。 - CPU 密集型任务配置尽可能少的线程数量： +## 九、高级并发工具 🚀 - 一般公式：CPU 合数 + 1 个线程的线程池 +### 🎯 什么是 ForkJoinPool？它与传统的线程池有什么区别？ -- IO 密集型 +**什么是 ForkJoinPool？** - - IO密集型任务线程并不是一直在执行任务，则应配置尽可能多的线程，如 CPU 核心数*2 +- **ForkJoinPool** 是 JDK7 引入的 **并行任务执行框架**，属于 `java.util.concurrent` 包。 +- 适合执行 **递归拆分的大任务**： + - **Fork（分治）**：把大任务拆分成多个小任务。 + - **Join（合并）**：把多个子任务的结果合并，形成最终结果。 +- 底层使用 **工作窃取算法（Work-Stealing）**： + - 每个线程维护一个双端队列，优先处理自己的任务。 + - 空闲时会“窃取”其他线程队列尾部的任务，提高 CPU 利用率。 - - IO 密集型，即该任务需要大量的 IO，即大量的阻塞 +👉 使用场景：**大任务拆小任务、递归计算、并行数据处理**（比如数组求和、分治排序）。 - 在单线程上运行 IO 密集型的任务会导致浪费大量的 CPU 运算能力浪费在等待。 +| 特性 | 传统线程池（ThreadPoolExecutor） | ForkJoinPool | +| ------------ | ---------------------------------- | --------------------------------------- | +| **设计目标** | 执行一批独立的、互不依赖的任务 | 递归分治的大任务，结果需要合并 | +| **任务拆分** | 提交 Runnable/Callable，不支持拆分 | 支持 Fork（拆分）和 Join（合并） | +| **队列模型** | 任务队列，线程从队列取任务 | 每个线程一个双端队列，支持工作窃取 | +| **执行模式** | 通常 FIFO 执行 | 支持 LIFO（自己队列）+ FIFO（窃取队列） | +| **适用场景** | Web 请求、异步任务、并发控制 | 大规模计算任务，CPU 密集型 | - 所以在 IO 密集型任务中使用多线程可以大大的加速程序运行，即使在单核 CPU 上，这种加速主要就是利用了被浪费调的阻塞时间。所以在 IO 密集型任务中使用多线程可以大大的加速程序运行，即使在单核 CPU 上，这种加速主要就是利用了被浪费掉的阻塞时间。 +### 🎯 ForkJoinPool 的工作原理是什么？ -### 为什么要用线程池？ +ForkJoinPool 的核心工作原理是工作窃取（work-stealing）算法。每个工作线程都有一个双端队列（deque），线程从头部取任务执行。当某个线程完成了自己的任务队列后，它可以从其他线程的队列尾部窃取任务执行，从而保持高效的并行处理。 -> **池化技术相比大家已经屡见不鲜了，线程池、数据库连接池、Http 连接池等等都是对这个思想的应用。池化技术的思想主要是为了减少每次获取资源的消耗，提高对资源的利用率。** +工作窃取算法可以最大限度地保持工作线程的忙碌，减少空闲线程的数量，提高 CPU 使用率。 -**线程池**提供了一种限制和管理资源（包括执行一个任务）。 每个**线程池**还维护一些基本统计信息，例如已完成任务的数量。 +### 🎯 如何使用 ForkJoinPool 来并行处理任务？ -这里借用《Java 并发编程的艺术》提到的来说一下**使用线程池的好处**： +使用 ForkJoinPool 需要继承 `RecursiveTask` 或 `RecursiveAction` 类，并实现 `compute()` 方法。`RecursiveTask` 用于有返回值的任务，`RecursiveAction` 用于没有返回值的任务。下面是一个简单的示例： -- **降低资源消耗**。通过重复利用已创建的线程降低线程创建和销毁造成的消耗。 -- **提高响应速度**。当任务到达时，任务可以不需要的等到线程创建就能立即执行。 -- **提高线程的可管理性**。线程是稀缺资源，如果无限制的创建，不仅会消耗系统资源，还会降低系统的稳定性，使用线程池可以进行统一的分配，调优和监控。 +```java +import java.util.concurrent.RecursiveTask; +import java.util.concurrent.ForkJoinPool; + +public class SumTask extends RecursiveTask { + private final int[] arr; + private final int start; + private final int end; + private static final int THRESHOLD = 10; + + public SumTask(int[] arr, int start, int end) { + this.arr = arr; + this.start = start; + this.end = end; + } + @Override + protected Integer compute() { + if (end - start <= THRESHOLD) { + int sum = 0; + for (int i = start; i < end; i++) { + sum += arr[i]; + } + return sum; + } else { + int mid = (start + end) / 2; + SumTask leftTask = new SumTask(arr, start, mid); + SumTask rightTask = new SumTask(arr, mid, end); + leftTask.fork(); + return rightTask.compute() + leftTask.join(); + } + } + public static void main(String[] args) { + ForkJoinPool pool = new ForkJoinPool(); + int[] arr = new int[100]; + for (int i = 0; i < arr.length; i++) { + arr[i] = i; + } + SumTask task = new SumTask(arr, 0, arr.length); + int result = pool.invoke(task); + System.out.println("Sum: " + result); + } +} +``` -### 实现Runnable接口和Callable接口的区别 +### 🎯 什么是 `fork()` 和 `join()` 方法？ -`Runnable`自Java 1.0以来一直存在，但`Callable`仅在Java 1.5中引入,目的就是为了来处理`Runnable`不支持的用例。**Runnable 接口**不会返回结果或抛出检查异常，但是**`Callable` 接口**可以。所以，如果任务不需要返回结果或抛出异常推荐使用 **Runnable 接口**，这样代码看起来会更加简洁。 +- `fork()` 方法：将任务拆分并放入队列中，使其他工作线程可以从队列中窃取并执行。 +- `join()` 方法：等待子任务完成并获取其结果。 -工具类 `Executors` 可以实现 `Runnable` 对象和 `Callable` 对象之间的相互转换。（`Executors.callable（Runnable task`）或 `Executors.callable（Runnable task，Object resule）`）。 +在上述示例中，`leftTask.fork()` 将左半部分任务放入队列，而 `rightTask.compute()` 直接计算右半部分任务。随后，`leftTask.join()` 等待左半部分任务完成并获取结果。 -``` -Runnable.java -@FunctionalInterface -public interface Runnable { - /** - * 被线程执行，没有返回值也无法抛出异常 - */ - public abstract void run(); -} -Callable.java -@FunctionalInterface -public interface Callable { - /** - * 计算结果，或在无法这样做时抛出异常。 - * @return 计算得出的结果 - * @throws 如果无法计算结果，则抛出异常 - */ - V call() throws Exception; -} -``` +### 🎯 解释一下 ForkJoinPool 的 `invoke()` 方法和 `submit()` 方法的区别。 -### 4.3. 执行execute()方法和submit()方法的区别是什么呢？ +- `invoke()` 方法：同步调用，提交任务并等待任务完成，返回任务结果。 +- `submit()` 方法：异步调用，提交任务但不等待任务完成，返回一个 `ForkJoinTask` 对象，可以通过这个对象的 `get()` 方法获取任务结果。 -1. **execute()方法用于提交不需要返回值的任务，所以无法判断任务是否被线程池执行成功与否；** -2. **submit()方法用于提交需要返回值的任务。线程池会返回一个 Future 类型的对象，通过这个 Future 对象可以判断任务是否执行成功**，并且可以通过 `Future` 的 `get()`方法来获取返回值，`get()`方法会阻塞当前线程直到任务完成，而使用 `get（long timeout，TimeUnit unit）`方法则会阻塞当前线程一段时间后立即返回，这时候有可能任务没有执行完。 +### 🎯 在 ForkJoinPool 中，如何处理异常？ -我们以**`AbstractExecutorService`**接口中的一个 `submit` 方法为例子来看看源代码： +在 ForkJoinPool 中执行任务时，如果任务抛出异常，异常会被封装在 `ExecutionException` 中。可以在调用 `join()` 或 `invoke()` 时捕获和处理异常。 -``` - public Future submit(Runnable task) { - if (task == null) throw new NullPointerException(); - RunnableFuture ftask = newTaskFor(task, null); - execute(ftask); - return ftask; +```java +public class ExceptionHandlingTask extends RecursiveTask { + private final int[] arr; + private final int start; + private final int end; + + public ExceptionHandlingTask(int[] arr, int start, int end) { + this.arr = arr; + this.start = start; + this.end = end; } -``` - -上面方法调用的 `newTaskFor` 方法返回了一个 `FutureTask` 对象。 -``` - protected RunnableFuture newTaskFor(Runnable runnable, T value) { - return new FutureTask(runnable, value); + @Override + protected Integer compute() { + if (start == end) { + throw new RuntimeException("Exception in task"); + } + return arr[start]; } -``` - -我们再来看看`execute()`方法： -``` - public void execute(Runnable command) { - ... + public static void main(String[] args) { + ForkJoinPool pool = new ForkJoinPool(); + int[] arr = new int[1]; + ExceptionHandlingTask task = new ExceptionHandlingTask(arr, 0, 0); + try { + int result = pool.invoke(task); + System.out.println("Result: " + result); + } catch (RuntimeException e) { + System.out.println("Exception: " + e.getMessage()); + } catch (ExecutionException e) { + System.out.println("ExecutionException: " + e.getCause().getMessage()); + } catch (InterruptedException e) { + e.printStackTrace(); + } } +} ``` -### 4.4. 如何创建线程池 +### 🎯 什么是 RecursiveTask 和 RecursiveAction？ -《阿里巴巴Java开发手册》中强制线程池不允许使用 Executors 去创建，而是通过 ThreadPoolExecutor 的方式，这样的处理方式让写的同学更加明确线程池的运行规则，规避资源耗尽的风险 +- `RecursiveTask`：用于有返回值的并行任务。必须实现 `compute()` 方法，并返回计算结果。 +- `RecursiveAction`：用于没有返回值的并行任务。必须实现 `compute()` 方法，但不返回结果。 -> Executors 返回线程池对象的弊端如下： -> -> - **FixedThreadPool 和 SingleThreadExecutor** ： 允许请求的队列长度为 Integer.MAX_VALUE ，可能堆积大量的请求，从而导致OOM。 -> - **CachedThreadPool 和 ScheduledThreadPool** ： 允许创建的线程数量为 Integer.MAX_VALUE ，可能会创建大量线程，从而导致OOM。 +### 🎯 ForkJoinPool 的并行度（parallelism level）是什么？ -**方式一：通过构造方法实现** [![ThreadPoolExecutor构造方法](https://camo.githubusercontent.com/c1a87ea139bc0379f5c98484416594843ff29d6d/68747470733a2f2f6d792d626c6f672d746f2d7573652e6f73732d636e2d6265696a696e672e616c6979756e63732e636f6d2f323031392d362f546872656164506f6f6c4578656375746f722545362539452538342545392538302541302545362539362542392545362542332539352e706e67)](https://camo.githubusercontent.com/c1a87ea139bc0379f5c98484416594843ff29d6d/68747470733a2f2f6d792d626c6f672d746f2d7573652e6f73732d636e2d6265696a696e672e616c6979756e63732e636f6d2f323031392d362f546872656164506f6f6c4578656375746f722545362539452538342545392538302541302545362539362542392545362542332539352e706e67) **方式二：通过Executor 框架的工具类Executors来实现** 我们可以创建三种类型的ThreadPoolExecutor： +ForkJoinPool 的并行度指的是可同时运行的工作线程数。可以在创建 ForkJoinPool 时指定并行度： -- **FixedThreadPool** ： 该方法返回一个固定线程数量的线程池。该线程池中的线程数量始终不变。当有一个新的任务提交时，线程池中若有空闲线程，则立即执行。若没有，则新的任务会被暂存在一个任务队列中，待有线程空闲时，便处理在任务队列中的任务。 -- **SingleThreadExecutor：** 方法返回一个只有一个线程的线程池。若多余一个任务被提交到该线程池，任务会被保存在一个任务队列中，待线程空闲，按先入先出的顺序执行队列中的任务。 -- **CachedThreadPool：** 该方法返回一个可根据实际情况调整线程数量的线程池。线程池的线程数量不确定，但若有空闲线程可以复用，则会优先使用可复用的线程。若所有线程均在工作，又有新的任务提交，则会创建新的线程处理任务。所有线程在当前任务执行完毕后，将返回线程池进行复用。 +```java +ForkJoinPool pool = new ForkJoinPool(4); // 并行度为 4 +``` -对应Executors工具类中的方法如图所示： [![Executor框架的工具类](https://camo.githubusercontent.com/6cfe663a5033e0f4adcfa148e6c54cdbb97c00bb/68747470733a2f2f6d792d626c6f672d746f2d7573652e6f73732d636e2d6265696a696e672e616c6979756e63732e636f6d2f323031392d362f4578656375746f722545362541312538362545362539452542362545372539412538342545352542372541352545352538352542372545372542312542422e706e67)](https://camo.githubusercontent.com/6cfe663a5033e0f4adcfa148e6c54cdbb97c00bb/68747470733a2f2f6d792d626c6f672d746f2d7573652e6f73732d636e2d6265696a696e672e616c6979756e63732e636f6d2f323031392d362f4578656375746f722545362541312538362545362539452542362545372539412538342545352542372541352545352538352542372545372542312542422e706e67) +并行度通常设置为 CPU 核心数，以充分利用多核处理器的计算能力。 -### 4.5 ThreadPoolExecutor 类分析 +### 🎯 ForkJoinPool 如何避免任务窃取导致的死锁？ -`ThreadPoolExecutor` 类中提供的四个构造方法。我们来看最长的那个，其余三个都是在这个构造方法的基础上产生（其他几个构造方法说白点都是给定某些默认参数的构造方法比如默认制定拒绝策略是什么），这里就不贴代码讲了，比较简单。 +ForkJoinPool 通过任务窃取和任务分解来避免死锁。工作线程在等待其他线程完成任务时，会主动窃取其他线程的任务以保持忙碌状态。此外，ForkJoinPool 使用工作窃取算法，尽可能将任务分散到各个线程的队列中，减少任务窃取导致的资源争用，从而降低死锁的可能性。 -``` - /** - * 用给定的初始参数创建一个新的ThreadPoolExecutor。 - */ - public ThreadPoolExecutor(int corePoolSize, - int maximumPoolSize, - long keepAliveTime, - TimeUnit unit, - BlockingQueue workQueue, - ThreadFactory threadFactory, - RejectedExecutionHandler handler) { - if (corePoolSize < 0 || - maximumPoolSize <= 0 || - maximumPoolSize < corePoolSize || - keepAliveTime < 0) - throw new IllegalArgumentException(); - if (workQueue == null || threadFactory == null || handler == null) - throw new NullPointerException(); - this.corePoolSize = corePoolSize; - this.maximumPoolSize = maximumPoolSize; - this.workQueue = workQueue; - this.keepAliveTime = unit.toNanos(keepAliveTime); - this.threadFactory = threadFactory; - this.handler = handler; - } -``` -**下面这些对创建 非常重要，在后面使用线程池的过程中你一定会用到！所以，务必拿着小本本记清楚。** -#### 4.5.1 `ThreadPoolExecutor`构造函数重要参数分析 +### 🎯 CompletableFuture? -**ThreadPoolExecutor 3 个最重要的参数：** +> `CompletableFuture` 是 JDK 1.8 引入的一个异步编程工具类，它在 `Future` 的基础上增强了功能： +> +> 1. 可以主动完成（complete），不只是被动等待； +> 2. 支持链式调用（如 thenApply、thenAccept、thenCompose），让异步任务像“流”一样组合； +> 3. 支持并行计算（allOf、anyOf 等），方便多个任务聚合； +> 4. 支持异常处理（exceptionally、handle），更健壮。 +> +> 它常用于 **异步调用、并行任务调度、提升系统吞吐** 的场景。 -- **corePoolSize :** 核心线程数线程数定义了最小可以同时运行的线程数量。 -- **maximumPoolSize :** 当队列中存放的任务达到队列容量的时候，当前可以同时运行的线程数量变为最大线程数。 -- **workQueue:** 当新任务来的时候会先判断当前运行的线程数量是否达到核心线程数，如果达到的话，新任务就会被存放在队列中。 +CompletableFuture 是 Java 8 引入的异步编程工具，它在 Future 基础上增强了异步任务的编排能力，支持链式调用、组合多个异步任务、异常处理等功能。通过 CompletableFuture，我们可以更优雅地处理异步操作，避免 Future.get () 的阻塞问题，实现非阻塞的异步编程，大幅提升代码可读性和并发处理效率。 -`ThreadPoolExecutor`其他常见参数: +**1. CompletableFuture 的核心特性** -1. **keepAliveTime**:当线程池中的线程数量大于 `corePoolSize` 的时候，如果这时没有新的任务提交，核心线程外的线程不会立即销毁，而是会等待，直到等待的时间超过了 `keepAliveTime`才会被回收销毁； -2. **unit** : `keepAliveTime` 参数的时间单位。 -3. **threadFactory** :executor 创建新线程的时候会用到。 -4. **handler** :饱和策略。关于饱和策略下面单独介绍一下。 +CompletableFuture 实现了 Future 和 CompletionStage 接口，相比传统 Future 具有以下优势： -#### 4.5.2 `ThreadPoolExecutor` 饱和策略 +- 支持链式调用，可将多个异步操作串联起来 +- 提供丰富的组合方法，能并行或串行组合多个 CompletableFuture +- 内置异常处理机制，无需 try-catch 包裹 +- 可主动完成任务（complete ()/completeExceptionally ()） +- 支持非阻塞回调，避免阻塞等待 -**ThreadPoolExecutor 饱和策略定义:** +**2. 基本使用方式** -如果当前同时运行的线程数量达到最大线程数量并且队列也已经被放满了任时，`ThreadPoolTaskExecutor` 定义一些策略: +**（1）创建 CompletableFuture** -- **ThreadPoolExecutor.AbortPolicy**：抛出 `RejectedExecutionException`来拒绝新任务的处理。 -- **ThreadPoolExecutor.CallerRunsPolicy**：调用执行自己的线程运行任务。您不会任务请求。但是这种策略会降低对于新任务提交速度，影响程序的整体性能。另外，这个策略喜欢增加队列容量。如果您的应用程序可以承受此延迟并且你不能任务丢弃任何一个任务请求的话，你可以选择这个策略。 -- **ThreadPoolExecutor.DiscardPolicy：** 不处理新任务，直接丢弃掉。 -- **ThreadPoolExecutor.DiscardOldestPolicy：** 此策略将丢弃最早的未处理的任务请求。 +```java +// 1. 立即完成的CompletableFuture +CompletableFuture completedFuture = CompletableFuture.completedFuture("Hello"); + +// 2. 异步执行Runnable（无返回值） +CompletableFuture runAsyncFuture = CompletableFuture.runAsync(() -> { + // 执行异步任务 + System.out.println("异步执行Runnable"); +}); + +// 3. 异步执行Supplier（有返回值） +CompletableFuture supplyAsyncFuture = CompletableFuture.supplyAsync(() -> { + // 执行异步任务并返回结果 + return "异步执行Supplier的结果"; +}); +``` -举个例子： Spring 通过 `ThreadPoolTaskExecutor` 或者我们直接通过 `ThreadPoolExecutor` 的构造函数创建线程池的时候，当我们不指定 `RejectedExecutionHandler` 饱和策略的话来配置线程池的时候默认使用的是 `ThreadPoolExecutor.AbortPolicy`。在默认情况下，`ThreadPoolExecutor` 将抛出 `RejectedExecutionException` 来拒绝新来的任务 ，这代表你将丢失对这个任务的处理。 对于可伸缩的应用程序，建议使用 `ThreadPoolExecutor.CallerRunsPolicy`。当最大池被填满时，此策略为我们提供可伸缩队列。（这个直接查看 `ThreadPoolExecutor` 的构造函数源码就可以看出，比较简单的原因，这里就不贴代码了） +默认使用 ForkJoinPool.commonPool ()，也可指定自定义线程池： -### 4.6 一个简单的线程池Demo:`Runnable`+`ThreadPoolExecutor` +```java +ExecutorService executor = Executors.newFixedThreadPool(5); +CompletableFuture customPoolFuture = CompletableFuture.supplyAsync(() -> { + return "使用自定义线程池"; +}, executor); +``` -为了让大家更清楚上面的面试题中的一些概念，我写了一个简单的线程池 Demo。 +**（2）链式操作** -首先创建一个 `Runnable` 接口的实现类（当然也可以是 `Callable` 接口，我们上面也说了两者的区别。） +通过 thenApply ()、thenAccept ()、thenRun () 等方法实现链式调用： +```java +CompletableFuture future = CompletableFuture.supplyAsync(() -> "Hello") + // 对结果进行转换（有返回值） + .thenApply(s -> s + " World") + // 消费结果（无返回值） + .thenAccept(s -> System.out.println("结果：" + s)) + // 执行后续操作（无输入无输出） + .thenRun(() -> System.out.println("链式操作完成")); ``` -MyRunnable.java -import java.util.Date; -/** - * 这是一个简单的Runnable类，需要大约5秒钟来执行其任务。 - * @author shuang.kou - */ -public class MyRunnable implements Runnable { +**（3）组合多个异步任务** - private String command; +- 并行执行，都完成后处理结果： - public MyRunnable(String s) { - this.command = s; - } +```java +CompletableFuture future1 = CompletableFuture.supplyAsync(() -> "任务1结果"); +CompletableFuture future2 = CompletableFuture.supplyAsync(() -> "任务2结果"); - @Override - public void run() { - System.out.println(Thread.currentThread().getName() + " Start. Time = " + new Date()); - processCommand(); - System.out.println(Thread.currentThread().getName() + " End. Time = " + new Date()); - } +// 两个任务都完成后，组合结果 +CompletableFuture combinedFuture = future1.thenCombine(future2, + (result1, result2) -> result1 + " + " + result2); +``` - private void processCommand() { - try { - Thread.sleep(5000); - } catch (InterruptedException e) { - e.printStackTrace(); - } - } +- 取第一个完成的任务结果： - @Override - public String toString() { - return this.command; - } -} +```java +CompletableFuture future3 = CompletableFuture.supplyAsync(() -> { + try { Thread.sleep(100); } catch (InterruptedException e) {} + return "任务3"; +}); + +CompletableFuture future4 = CompletableFuture.supplyAsync(() -> { + try { Thread.sleep(50); } catch (InterruptedException e) {} + return "任务4"; +}); + +// 取先完成的任务结果 +CompletableFuture firstCompleted = CompletableFuture.anyOf(future3, future4); ``` -编写测试程序，我们这里以阿里巴巴推荐的使用 `ThreadPoolExecutor` 构造函数自定义参数的方式来创建线程池。 +**（4）异常处理** +```java +CompletableFuture exceptionFuture = CompletableFuture.supplyAsync(() -> { + if (true) { + throw new RuntimeException("任务执行失败"); + } + return "正常结果"; +}) +// 异常处理（类似try-catch） +.exceptionally(ex -> { + System.out.println("捕获异常：" + ex.getMessage()); + return "默认结果"; +}) +// 无论成功失败都会执行（类似finally） +.whenComplete((result, ex) -> { + if (ex == null) { + System.out.println("执行成功，结果：" + result); + } else { + System.out.println("执行失败，异常：" + ex.getMessage()); + } +}); ``` -ThreadPoolExecutorDemo.java -import java.util.concurrent.ArrayBlockingQueue; -import java.util.concurrent.ThreadPoolExecutor; -import java.util.concurrent.TimeUnit; -public class ThreadPoolExecutorDemo { +**3. 常用核心方法分类** - private static final int CORE_POOL_SIZE = 5; - private static final int MAX_POOL_SIZE = 10; - private static final int QUEUE_CAPACITY = 100; - private static final Long KEEP_ALIVE_TIME = 1L; - public static void main(String[] args) { +| 方法类型 | 常用方法 | 作用 | +| -------- | ------------------------------- | -------------------------------- | +| 转换 | thenApply(), thenApplyAsync() | 对前一个任务的结果进行转换 | +| 消费 | thenAccept(), thenAcceptAsync() | 消费前一个任务的结果，无返回值 | +| 执行 | thenRun(), thenRunAsync() | 前一个任务完成后执行，无输入输出 | +| 组合 | thenCombine(), allOf(), anyOf() | 组合多个 CompletableFuture | +| 异常 | exceptionally(), whenComplete() | 处理异常或最终操作 | - //使用阿里巴巴推荐的创建线程池的方式 - //通过ThreadPoolExecutor构造函数自定义参数创建 - ThreadPoolExecutor executor = new ThreadPoolExecutor( - CORE_POOL_SIZE, - MAX_POOL_SIZE, - KEEP_ALIVE_TIME, - TimeUnit.SECONDS, - new ArrayBlockingQueue<>(QUEUE_CAPACITY), - new ThreadPoolExecutor.CallerRunsPolicy()); - - for (int i = 0; i < 10; i++) { - //创建WorkerThread对象（WorkerThread类实现了Runnable 接口） - Runnable worker = new MyRunnable("" + i); - //执行Runnable - executor.execute(worker); - } - //终止线程池 - executor.shutdown(); - while (!executor.isTerminated()) { - } - System.out.println("Finished all threads"); - } -} -``` +**4. 实际应用场景** -可以看到我们上面的代码指定了： +- **异步接口调用**：并行调用多个外部接口，汇总结果 +- **任务拆分**：将复杂任务拆分为多个子任务并行执行，提高效率 +- **非阻塞 IO**：配合 NIO 实现高效的 IO 操作 +- **事件驱动编程**：基于回调的事件处理机制 -1. `corePoolSize`: 核心线程数为 5。 -2. `maximumPoolSize` ：最大线程数 10 -3. `keepAliveTime` : 等待时间为 1L。 -4. `unit`: 等待时间的单位为 TimeUnit.SECONDS。 -5. `workQueue`：任务队列为 `ArrayBlockingQueue`，并且容量为 100; -6. `handler`:饱和策略为 `CallerRunsPolicy`。 +**5. 注意事项** -**Output：** +- **线程池管理**：避免过度使用默认线程池，高并发场景建议使用自定义线程池 +- **内存泄漏**：长时间未完成的 CompletableFuture 可能导致内存泄漏 +- **异常传播**：链式操作中异常会向后传播，需合理设置异常处理 +- **阻塞问题**：get () 和 join () 方法会阻塞当前线程，尽量使用非阻塞回调 -``` -pool-1-thread-2 Start. Time = Tue Nov 12 20:59:44 CST 2019 -pool-1-thread-5 Start. Time = Tue Nov 12 20:59:44 CST 2019 -pool-1-thread-4 Start. Time = Tue Nov 12 20:59:44 CST 2019 -pool-1-thread-1 Start. Time = Tue Nov 12 20:59:44 CST 2019 -pool-1-thread-3 Start. Time = Tue Nov 12 20:59:44 CST 2019 -pool-1-thread-5 End. Time = Tue Nov 12 20:59:49 CST 2019 -pool-1-thread-3 End. Time = Tue Nov 12 20:59:49 CST 2019 -pool-1-thread-2 End. Time = Tue Nov 12 20:59:49 CST 2019 -pool-1-thread-4 End. Time = Tue Nov 12 20:59:49 CST 2019 -pool-1-thread-1 End. Time = Tue Nov 12 20:59:49 CST 2019 -pool-1-thread-2 Start. Time = Tue Nov 12 20:59:49 CST 2019 -pool-1-thread-1 Start. Time = Tue Nov 12 20:59:49 CST 2019 -pool-1-thread-4 Start. Time = Tue Nov 12 20:59:49 CST 2019 -pool-1-thread-3 Start. Time = Tue Nov 12 20:59:49 CST 2019 -pool-1-thread-5 Start. Time = Tue Nov 12 20:59:49 CST 2019 -pool-1-thread-2 End. Time = Tue Nov 12 20:59:54 CST 2019 -pool-1-thread-3 End. Time = Tue Nov 12 20:59:54 CST 2019 -pool-1-thread-4 End. Time = Tue Nov 12 20:59:54 CST 2019 -pool-1-thread-5 End. Time = Tue Nov 12 20:59:54 CST 2019 -pool-1-thread-1 End. Time = Tue Nov 12 20:59:54 CST 2019 -``` +CompletableFuture 极大地简化了 Java 异步编程，通过其丰富的 API 可以灵活地处理各种异步场景，是现代 Java 并发编程中不可或缺的工具。相比传统的 Future 和线程池组合，它能写出更简洁、更易维护的异步代码。 -### 4.7 线程池原理分析 -承接 4.6 节，我们通过代码输出结果可以看出：**线程池每次会同时执行 5 个任务，这 5 个任务执行完之后，剩余的 5 个任务才会被执行。** 大家可以先通过上面讲解的内容，分析一下到底是咋回事？（自己独立思考一会） -现在，我们就分析上面的输出内容来简单分析一下线程池原理。 +## 十、并发应用实践 -**为了搞懂线程池的原理，我们需要首先分析一下 `execute`方法。**在 4.6 节中的 Demo 中我们使用 `executor.execute(worker)`来提交一个任务到线程池中去，这个方法非常重要，下面我们来看看它的源码： +### 🎯 高并发网站架构设计 -``` - // 存放线程池的运行状态 (runState) 和线程池内有效线程的数量 (workerCount) - private final AtomicInteger ctl = new AtomicInteger(ctlOf(RUNNING, 0)); +> "高并发网站需要从多个维度优化： +> +> **前端优化**： +> +> - HTML 静态化：减少动态内容生成 +> - CDN 加速：就近访问，减少延迟 +> - 图片服务分离：减轻 Web 服务器 I/O 负载 +> +> **应用层优化**： +> +> - 负载均衡：分发请求到多台服务器 +> - 连接池：复用数据库连接 +> - 缓存机制：多级缓存减少数据库访问 +> +> **数据层优化**： +> +> - 读写分离：主库写，从库读 +> - 分库分表：水平拆分减少单表压力 +> - 索引优化：提高查询效率" - private static int workerCountOf(int c) { - return c & CAPACITY; - } +1. HTML 页面静态化 - private final BlockingQueue workQueue; - - public void execute(Runnable command) { - // 如果任务为null，则抛出异常。 - if (command == null) - throw new NullPointerException(); - // ctl 中保存的线程池当前的一些状态信息 - int c = ctl.get(); - - // 下面会涉及到 3 步 操作 - // 1.首先判断当前线程池中之行的任务数量是否小于 corePoolSize - // 如果小于的话，通过addWorker(command, true)新建一个线程，并将任务(command)添加到该线程中；然后，启动该线程从而执行任务。 - if (workerCountOf(c) < corePoolSize) { - if (addWorker(command, true)) - return; - c = ctl.get(); - } - // 2.如果当前之行的任务数量大于等于 corePoolSize 的时候就会走到这里 - // 通过 isRunning 方法判断线程池状态，线程池处于 RUNNING 状态才会被并且队列可以加入任务，该任务才会被加入进去 - if (isRunning(c) && workQueue.offer(command)) { - int recheck = ctl.get(); - // 再次获取线程池状态，如果线程池状态不是 RUNNING 状态就需要从任务队列中移除任务，并尝试判断线程是否全部执行完毕。同时执行拒绝策略。 - if (!isRunning(recheck) && remove(command)) - reject(command); - // 如果当前线程池为空就新创建一个线程并执行。 - else if (workerCountOf(recheck) == 0) - addWorker(null, false); - } - //3. 通过addWorker(command, false)新建一个线程，并将任务(command)添加到该线程中；然后，启动该线程从而执行任务。 - //如果addWorker(command, false)执行失败，则通过reject()执行相应的拒绝策略的内容。 - else if (!addWorker(command, false)) - reject(command); - } -``` + 访问频率较高但内容变动较小，使用网站 HTML 静态化方案来优化访问速度。将社区 内的帖子、文章进行实时的静态化，有更新的时候再重新静态化也是大量使用的策略。 -通过下图可以更好的对上面这 3 步做一个展示，下图是我为了省事直接从网上找到，原地址不明。 + 优势: + 一、减轻服务器负担。 -[![图解线程池实现原理](https://camo.githubusercontent.com/cf627f637b4c678cd77b815fbea8789dd3158b0c/68747470733a2f2f6d792d626c6f672d746f2d7573652e6f73732d636e2d6265696a696e672e616c6979756e63732e636f6d2f323031392d372f2545352539422542452545382541372541332545372542412542462545372541382538422545362542312541302545352541452539452545372538452542302545352538452539462545372539302538362e706e67)](https://camo.githubusercontent.com/cf627f637b4c678cd77b815fbea8789dd3158b0c/68747470733a2f2f6d792d626c6f672d746f2d7573652e6f73732d636e2d6265696a696e672e616c6979756e63732e636f6d2f323031392d372f2545352539422542452545382541372541332545372542412542462545372541382538422545362542312541302545352541452539452545372538452542302545352538452539462545372539302538362e706e67) + 二、加快页面打开速度，静态页面无需访问数据库，打开速度较动态页面有明显提高; -现在，让我们在回到 4.6 节我们写的 Demo， 现在应该是不是很容易就可以搞懂它的原理了呢？ + 三、很多搜索引擎都会优先收录静态页面，不仅被收录的快，还收录的全，容易被搜索引擎找到; -没搞懂的话，也没关系，可以看看我的分析： + 四、HTML 静态页面不会受程序相关漏洞的影响，减少攻击 ，提高安全性。 -> 我们在代码中模拟了 10 个任务，我们配置的核心线程数为 5 、等待队列容量为 100 ，所以每次只可能存在 5 个任务同时执行，剩下的 5 个任务会被放到等待队列中去。当前的 5 个任务之行完成后，才会之行剩下的 5 个任务。 +2. 图片服务器和应用服务器相分离 -#### + 现在很多的网站上都会用到大量的图片，而图片是网页传输中占主要的数据量,也是影 响网站性能的主要因素。因此很多网站都会将图片存储从网站中分离出来，另外架构一个 或多个服务器来存储图片，将图片放到一个虚拟目录中，而网页上的图片都用一个 URL 地 址来指向这些服务器上的图片的地址，这样的话网站的性能就明显提高了。 + 优势: + 一、 分担 Web 服务器的 I/O 负载-将耗费资源的图片服务分离出来，提高服务器的性能和稳定性。 + 二、 能够专门对图片服务器进行优化-为图片服务设置有针对性的缓存方案，减少带宽 成本，提高访问速度。 -- ConcurrentHashMap和HashMap -- -- 线程池原理，拒绝策略，核心线程数 -- 线程之间的交互方式有哪些？有没有线程交互的封装类 （join）？ -- 死锁怎么避免？ -- concurrentHashMap分段锁的细节 -- 并发包里了解哪些 -- synchronizedMap知道吗，和concurrentHashMap分别用于什么场景 -- 描述一下java线程池 -- 常用的队列，阻塞队列 -- 如何获取多线程调用结果 -- -- synchronized内部实现，偏向锁，轻量锁，重量锁 + 三、 提高网站的可扩展性-通过增加图片服务器，提高图片吞吐能力。 -- 为什么需要自旋？ +3. 数据库 数据库层面的优化，读写分离，分库分表 -- sleep( ) 和 wait( n)、wait( ) 的区别： +4. 缓存 - **sleep 方法：** 是 Thread 类的静态方法，当前线程将睡眠 n 毫秒，线程进入阻塞状态。当睡眠时间到了，会解除阻塞，进行可运行状态，等待 CPU 的到来。睡眠不释放锁（如果有的话）； + 尽量使用缓存，包括用户缓存，信息缓存等，多花点内存来做缓存，可以大量减少与 数据库的交互，提高性能。 - **wait 方法：** 是 Object 的方法，必须与 synchronized 关键字一起使用，线程进入阻塞状态，当 notify 或者 notifyall 被调用后，会解除阻塞。但是，只有重新占用互斥锁之后才会进入可运行状态。睡眠时，释放互斥锁。 + 假如我们能减少数据库频繁的访问，那对系统肯定大大有利的。比如一个电子商务系 统的商品搜索，如果某个关键字的商品经常被搜，那就可以考虑这部分商品列表存放到缓 存(内存中去)，这样不用每次访问数据库，性能大大增加。 -synchronized和Lock的区别 +5. 镜像 -sleep方法和yield方法的区别 + 镜像是冗余的一种类型，一个磁盘上的数据在另一个磁盘上存在一个完全相同的副本 即为镜像。 +6. 负载均衡 + 在网站高并发访问的场景下，使用负载均衡技术(负载均衡服务器)为一个应用构建 一个由多台服务器组成的服务器集群，将并发访问请求分发到多台服务器上处理，避免单 一服务器因负载压力过大而响应缓慢，使用户请求具有更好的响应延迟特性。 -## AQS +7. 并发控制 加锁，如乐观锁和悲观锁。 -## AQS +8. 消息队列 + 通过 mq 一个一个排队方式，跟 12306 一样。 -### 6.1. AQS 介绍 -AQS的全称为（AbstractQueuedSynchronizer），这个类在java.util.concurrent.locks包下面。 -[![AQS类](https://camo.githubusercontent.com/7e2bd67b66e3e1764a442b8d96689f64e5521c2c/68747470733a2f2f6d792d626c6f672d746f2d7573652e6f73732d636e2d6265696a696e672e616c6979756e63732e636f6d2f323031392d362f4151532545372542312542422e706e67)](https://camo.githubusercontent.com/7e2bd67b66e3e1764a442b8d96689f64e5521c2c/68747470733a2f2f6d792d626c6f672d746f2d7573652e6f73732d636e2d6265696a696e672e616c6979756e63732e636f6d2f323031392d362f4151532545372542312542422e706e67) +### 🎯 订票系统，某车次只有一张火车票，假定有 **1w** 个人同 时打开 **12306** 网站来订票，如何解决并发问题?(可扩展 到任何高并发网站要考虑的并发读写问题)。 -AQS是一个用来构建锁和同步器的框架，使用AQS能简单且高效地构造出应用广泛的大量的同步器，比如我们提到的ReentrantLock，Semaphore，其他的诸如ReentrantReadWriteLock，SynchronousQueue，FutureTask等等皆是基于AQS的。当然，我们自己也能利用AQS非常轻松容易地构造出符合我们自己需求的同步器。 +> "这是典型的高并发读写问题，既要保证 1w 人能同时看到有票（可读性），又要保证只有一个人能买到票（排他性）： +> +> **解决方案**： +> +> 1. **数据库乐观锁**：利用版本号或时间戳，避免锁表影响性能 +> 2. **分布式锁**：Redis 实现，保证分布式环境下的原子性 +> 3. **消息队列**：请求排队处理，削峰填谷 +> 4. **库存预扣**：先扣库存再处理业务，避免超卖" -### 6.2. AQS 原理分析 +不但要保证 1w 个人能同时看到有票(数据的可读性)，还要保证最终只能 由一个人买到票(数据的排他性)。 -AQS 原理这部分参考了部分博客，在5.2节末尾放了链接。 +使用数据库层面的并发访问控制机制。采用乐观锁即可解决此问题。乐观 锁意思是不锁定表的情况下，利用业务的控制来解决并发问题，这样既保证数 据的并发可读性，又保证保存数据的排他性，保证性能的同时解决了并发带来 的脏数据问题。hibernate 中实现乐观锁。 -> 在面试中被问到并发知识的时候，大多都会被问到“请你说一下自己对于AQS原理的理解”。下面给大家一个示例供大家参加，面试不是背题，大家一定要加入自己的思想，即使加入不了自己的思想也要保证自己能够通俗的讲出来而不是背出来。 +银行两操作员同时操作同一账户就是典型的例子。比如 A、B 操作员同 时读取一余额为 1000 元的账户，A 操作员为该账户增加 100 元，B 操作员同时 为该账户减去 50元，A先提交，B后提交。最后实际账户余额为1000-50=950 元，但本该为 1000+100-50=1050。这就是典型的并发问题。如何解决?可以用锁。 -下面大部分内容其实在AQS类注释上已经给出了，不过是英语看着比较吃力一点，感兴趣的话可以看看源码。 -#### 6.2.1. AQS 原理概览 -**AQS核心思想是，如果被请求的共享资源空闲，则将当前请求资源的线程设置为有效的工作线程，并且将共享资源设置为锁定状态。如果被请求的共享资源被占用，那么就需要一套线程阻塞等待以及被唤醒时锁分配的机制，这个机制AQS是用CLH队列锁实现的，即将暂时获取不到锁的线程加入到队列中。** +### 🎯 如果不用锁机制如何实现共享数据访问？ -> CLH(Craig,Landin,and Hagersten)队列是一个虚拟的双向队列（虚拟的双向队列即不存在队列实例，仅存在结点之间的关联关系）。AQS是将每条请求共享资源的线程封装成一个CLH锁队列的一个结点（Node）来实现锁的分配。 +> 不要用锁，不要用 **sychronized** 块或者方法，也不要直接使用 **jdk** 提供的线程安全 的数据结构，需要自己实现一个类来保证多个线程同时读写这个类 中的共享数据是线程安全的，怎么办? -看个AQS(AbstractQueuedSynchronizer)原理图： +无锁化编程的常用方法:硬件**CPU**同步原语CAS(Compare and Swap)，如无锁栈，无锁队列(ConcurrentLinkedQueue)等等。现在 几乎所有的 CPU 指令都支持 CAS 的原子操作，X86 下对应的是 CMPXCHG 汇 编指令，处理器执行 CMPXCHG 指令是一个原子性操作。有了这个原子操作， 我们就可以用其来实现各种无锁(lock free)的数据结构。 -[![AQS原理图](https://camo.githubusercontent.com/13db51afdebad2dac67a224d422f6f60c9b8d366/68747470733a2f2f6d792d626c6f672d746f2d7573652e6f73732d636e2d6265696a696e672e616c6979756e63732e636f6d2f323031392d362f4151532545352538452539462545372539302538362545352539422542452e706e67)](https://camo.githubusercontent.com/13db51afdebad2dac67a224d422f6f60c9b8d366/68747470733a2f2f6d792d626c6f672d746f2d7573652e6f73732d636e2d6265696a696e672e616c6979756e63732e636f6d2f323031392d362f4151532545352538452539462545372539302538362545352539422542452e706e67) +CAS 实现了区别于 sychronized 同步锁的一种乐观锁，当多个线程尝试使 用 CAS 同时更新同一个变量时，只有其中一个线程能更新变量的值，而其它线 程都失败，失败的线程并不会被挂起，而是被告知这次竞争中失败，并可以再 次尝试。CAS 有 3 个操作数，内存值 V，旧的预期值 A，要修改后的新值 B。 当且仅当预期值 A 和内存值 V 相同时，将内存值 V 修改为 B，否则什么都不做。 其实 CAS 也算是有锁操作，只不过是由 CPU 来触发，比 synchronized 性能 好的多。CAS 的关键点在于，系统在硬件层面保证了比较并交换操作的原子性， 处理器使用基于对缓存加锁或总线加锁的方式来实现多处理器之间的原子操作。CAS 是非阻塞算法的一种常见实现。 -AQS使用一个int成员变量来表示同步状态，通过内置的FIFO队列来完成获取资源线程的排队工作。AQS使用CAS对该同步状态进行原子操作实现对其值的修改。 +一个线程间共享的变量，首先在主存中会保留一份，然后每个线程的工作 内存也会保留一份副本。这里说的预期值，就是线程保留的副本。当该线程从 主存中获取该变量的值后，主存中该变量可能已经被其他线程刷新了，但是该 线程工作内存中该变量却还是原来的值，这就是所谓的预期值了。当你要用 CAS 刷新该值的时候，如果发现线程工作内存和主存中不一致了，就会失败，如果 一致，就可以更新成功。 -``` -private volatile int state;//共享变量，使用volatile修饰保证线程可见性 -``` +**Atomic** 包提供了一系列原子类。这些类可以保证多线程环境下，当某个 线程在执行atomic的方法时，不会被其他线程打断，而别的线程就像自旋锁一 样，一直等到该方法执行完成，才由 JVM 从等待队列中选择一个线程执行。 Atomic 类在软件层面上是非阻塞的，它的原子性其实是在硬件层面上借助相关 的指令来保证的。 -状态信息通过protected类型的getState，setState，compareAndSetState进行操作 -``` -//返回同步状态的当前值 -protected final int getState() { - return state; -} - // 设置同步状态的值 -protected final void setState(int newState) { - state = newState; -} -//原子地（CAS操作）将同步状态值设置为给定值update如果当前同步状态的值等于expect（期望值） -protected final boolean compareAndSetState(int expect, int update) { - return unsafe.compareAndSwapInt(this, stateOffset, expect, update); -} -``` -#### 6.2.2. AQS 对资源的共享方式 +### 🎯 如何在 Windows 和 Linux 上查找哪个线程 cpu 利用率最高？ -**AQS定义两种资源共享方式** +windows上面用任务管理器看，linux下可以用 top 这个工具看。 -- Exclusive +1. 找出 cpu 耗用厉害的进程pid， 终端执行top命令，然后按下shift+p 查找出cpu利用最厉害的pid号 +2. 根据上面第一步拿到的pid号，top -H -p pid 。然后按下shift+p，查找出cpu利用率最厉害的线程号，比如top -H -p 1328 +3. 将获取到的线程号转换成16进制，去百度转换一下就行 +4. 使用jstack工具将进程信息打印输出，jstack pid号 > /tmp/t.dat，比如jstack 31365 > /tmp/t.dat +5. 编辑/tmp/t.dat文件，查找线程号对应的信息 - （独占）：只有一个线程能执行，如ReentrantLock。又可分为公平锁和非公平锁： - - 公平锁：按照线程在队列中的排队顺序，先到者先拿到锁 - - 非公平锁：当线程要获取锁时，无视队列顺序直接去抢锁，谁抢到就是谁的 -- **Share**（共享）：多个线程可同时执行，如Semaphore/CountDownLatch。Semaphore、CountDownLatch、 CyclicBarrier、ReadWriteLock 我们都会在后面讲到。 +### 🎯 Java有哪几种实现生产者消费者模式的方法？ -ReentrantReadWriteLock 可以看成是组合式，因为ReentrantReadWriteLock也就是读写锁允许多个线程同时对某一资源进行读。 +1. **使用`wait()`和`notify()`方法**： -不同的自定义同步器争用共享资源的方式也不同。自定义同步器在实现时只需要实现共享资源 state 的获取与释放方式即可，至于具体线程等待队列的维护（如获取资源失败入队/唤醒出队等），AQS已经在顶层实现好了。 + - 利用Java的同步机制，生产者在缓冲区满时调用`wait()`挂起，消费者在缓冲区空时调用`wait()`挂起。相应地，生产者在放入商品后调用`notifyAll()`唤醒消费者，消费者在取出商品后调用`notifyAll()`唤醒生产者。 -#### 6.2.3. AQS底层使用了模板方法模式 +2. **使用`ReentrantLock`和`Condition`**： -同步器的设计是基于模板方法模式的，如果需要自定义同步器一般的方式是这样（模板方法模式很经典的一个应用）： + - `ReentrantLock`提供了更灵活的锁机制，`Condition`可以用来替代`wait()`和`notify()`，提供更细粒度的控制。 -1. 使用者继承AbstractQueuedSynchronizer并重写指定的方法。（这些重写方法很简单，无非是对于共享资源state的获取和释放） -2. 将AQS组合在自定义同步组件的实现中，并调用其模板方法，而这些模板方法会调用使用者重写的方法。 +3. **使用`BlockingQueue`**： -这和我们以往通过实现接口的方式有很大区别，这是模板方法模式很经典的一个运用。 + - `java.util.concurrent.BlockingQueue`是一个线程安全的队列，其已经实现了生产者-消费者模式。当队列为满时，`put()`操作将阻塞；当队列为空时，`take()`操作将阻塞。 -**AQS使用了模板方法模式，自定义同步器时需要重写下面几个AQS提供的模板方法：** + -``` -isHeldExclusively()//该线程是否正在独占资源。只有用到condition才需要去实现它。 -tryAcquire(int)//独占方式。尝试获取资源，成功则返回true，失败则返回false。 -tryRelease(int)//独占方式。尝试释放资源，成功则返回true，失败则返回false。 -tryAcquireShared(int)//共享方式。尝试获取资源。负数表示失败；0表示成功，但没有剩余可用资源；正数表示成功，且有剩余资源。 -tryReleaseShared(int)//共享方式。尝试释放资源，成功则返回true，失败则返回false。 -``` +### 🎯 写出 **3** 条你遵循的多线程最佳实践 -默认情况下，每个方法都抛出 `UnsupportedOperationException`。 这些方法的实现必须是内部线程安全的，并且通常应该简短而不是阻塞。AQS类中的其他方法都是final ，所以无法被其他类使用，只有这几个方法可以被其他类使用。 +1. 给线程起个有意义的名字。 -以ReentrantLock为例，state初始化为0，表示未锁定状态。A线程lock()时，会调用tryAcquire()独占该锁并将state+1。此后，其他线程再tryAcquire()时就会失败，直到A线程unlock()到state=0（即释放锁）为止，其它线程才有机会获取该锁。当然，释放锁之前，A线程自己是可以重复获取此锁的（state会累加），这就是可重入的概念。但要注意，获取多少次就要释放多么次，这样才能保证state是能回到零态的。 +2. 避免锁定和缩小同步的范围 。 -再以CountDownLatch以例，任务分为N个子线程去执行，state也初始化为N（注意N要与线程个数一致）。这N个子线程是并行执行的，每个子线程执行完后countDown()一次，state会CAS(Compare and Swap)减1。等到所有子线程都执行完后(即state=0)，会unpark()主调用线程，然后主调用线程就会从await()函数返回，继续后余动作。 + 相对于同步方法我更喜欢同步块，它给我拥有对锁的绝对控制权。 -一般来说，自定义同步器要么是独占方法，要么是共享方式，他们也只需实现`tryAcquire-tryRelease`、`tryAcquireShared-tryReleaseShared`中的一种即可。但AQS也支持自定义同步器同时实现独占和共享两种方式，如`ReentrantReadWriteLock`。 +3. 多用同步辅助类，少用 **wait** 和 **notify** 。 +4. 多用并发容器，少用同步容器。 + 如果下一次你需要用到 map，你应该首先想到用 ConcurrentHashMap。 -推荐两篇 AQS 原理和相关源码分析的文章： +------ -- http://www.cnblogs.com/waterystone/p/4920797.html -- https://www.cnblogs.com/chengxiao/archive/2017/07/24/7141160.html -### 6.3. AQS 组件总结 -- **Semaphore(信号量)-允许多个线程同时访问：** synchronized 和 ReentrantLock 都是一次只允许一个线程访问某个资源，Semaphore(信号量)可以指定多个线程同时访问某个资源。 -- **CountDownLatch （倒计时器）：** CountDownLatch是一个同步工具类，用来协调多个线程之间的同步。这个工具通常用来控制线程等待，它可以让某一个线程等待直到倒计时结束，再开始执行。 -- **CyclicBarrier(循环栅栏)：** CyclicBarrier 和 CountDownLatch 非常类似，它也可以实现线程间的技术等待，但是它的功能比 CountDownLatch 更加复杂和强大。主要应用场景和 CountDownLatch 类似。CyclicBarrier 的字面意思是可循环使用（Cyclic）的屏障（Barrier）。它要做的事情是，让一组线程到达一个屏障（也可以叫同步点）时被阻塞，直到最后一个线程到达屏障时，屏障才会开门，所有被屏障拦截的线程才会继续干活。CyclicBarrier默认的构造方法是 CyclicBarrier(int parties)，其参数表示屏障拦截的线程数量，每个线程调用await()方法告诉 CyclicBarrier 我已经到达了屏障，然后当前线程被阻塞。 +## 并发编程最佳实践总结 +### 🎯 设计原则 +1. **安全性优先**：确保数据一致性，避免竞态条件 +2. **性能兼顾**：在保证正确性前提下优化并发度 +3. **可维护性**：代码清晰，便于理解和调试 +4. **故障恢复**：考虑异常情况和系统故障 -## countDownLatch/CycliBarries/Semaphore使用过吗 +### 🎯 实践经验 +1. **线程命名**：给线程起有意义的名字，便于问题排查 +2. **锁粒度**：优先使用同步块而非同步方法，缩小锁范围 +3. **工具选择**：多用并发容器，少用同步容器 +4. **异常处理**：完善的异常处理和日志记录 +5. **监控告警**：建立完整的监控体系 +### 🎯 性能优化 -#### CycliBarries +1. **无锁化**：优先使用 CAS 和原子类 +2. **读写分离**：读多写少场景使用 CopyOnWriteArrayList +3. **分段锁**：减少锁竞争，提高并发度 +4. **异步处理**：使用线程池和消息队列 +5. **缓存机制**：多级缓存减少 I/O 操作 -CycliBarries 的字面意思是可循环（cycli）使用的屏障（Barries）。它主要做的事情是，让一组线程达到一个屏障（也可以叫同步点）时被阻塞，知道最后一个线程到达屏障时，屏障才会开门，所有被屏障拦截的线程才会继续干活，线程进入屏障通过CycliBarries的 await() 方法。 +**记住：并发编程的核心是在保证正确性的前提下，提高系统的并发处理能力！** -```java -public class CyclieBarrierDemo { +## 🔥 高频面试题快速回顾 - public static void main(String[] args) { +### 💡 基础概念类 +| **问题** | **核心答案** | **关键点** | +| ---------- | ------------------------ | --------------------------------------- | +| 进程vs线程 | 资源分配 vs 调度单位 | 内存隔离、通信方式、创建开销 | +| 并发vs并行 | 时间段内 vs 同一时刻 | 逻辑概念 vs 物理概念 | +| 线程状态 | 6种状态及转换 | NEW→RUNNABLE→BLOCKED/WAITING→TERMINATED | +| 线程安全 | 多线程访问共享资源正确性 | 加锁、原子操作、不可变对象 | - // public CyclicBarrier(int parties, Runnable barrierAction) { - CyclicBarrier cyclicBarrier = new CyclicBarrier(7,()->{ - System.out.println("召唤神龙"); - }); +### 🔒 同步机制类 - for (int i = 1; i < 8; i++) { - final int temp = i; - new Thread(()->{ - System.out.println(Thread.currentThread().getName()+"收集到第"+temp+"颗龙珠"); +| **问题** | **核心答案** | **关键点** | +| ----------------- | ---------------------- | --------------------------------- | +| synchronized原理 | Monitor机制，锁升级 | monitorenter/exit、偏向→轻量→重量 | +| volatile作用 | 可见性+有序性 | 内存屏障、禁止重排、不保证原子性 | +| ReentrantLock特性 | 显式锁，公平性，可中断 | vs synchronized对比表 | +| AQS框架 | 状态管理+队列+模板方法 | state、CLH队列、独占/共享模式 | - try { - cyclicBarrier.await(); - } catch (InterruptedException e) { - e.printStackTrace(); - } catch (BrokenBarrierException e) { - e.printStackTrace(); - } - },String.valueOf(i)).start(); - } +### 🛠️ 并发工具类 - } +| **问题** | **核心答案** | **关键点** | +| -------------- | ------------------ | ------------------------ | +| CountDownLatch | 倒计时门栓，一次性 | 主线程等待多个子线程完成 | +| CyclicBarrier | 循环屏障，可重用 | 多线程相互等待，同步执行 | +| Semaphore | 信号量，控制并发数 | 限流、资源池管理 | +| ThreadLocal | 线程局部变量 | ThreadLocalMap、内存泄漏 | -} -``` +### 🏊 线程池类 +| **问题** | **核心答案** | **关键点** | +| -------- | ----------------------------------- | -------------------------- | +| 七大参数 | 核心、最大、存活时间等 | corePoolSize最重要 | +| 工作原理 | 核心→队列→非核心→拒绝 | 任务提交流程图 | +| 参数配置 | CPU密集型+1，IO密集型×(1+等待/处理) | 结合任务特性配置 | +| 拒绝策略 | 4种策略及适用场景 | Abort、CallerRuns、Discard | +### ⚛️ 原子操作类 +| **问题** | **核心答案** | **关键点** | +| ------------- | -------------------- | -------------------------- | +| CAS机制 | 比较并交换，硬件保证 | V、A、B三个操作数 | +| AtomicInteger | 基于CAS的无锁实现 | 自旋重试、性能优势 | +| ABA问题 | 值变化无法感知 | AtomicStampedReference解决 | +| LongAdder优势 | 分段累加减少竞争 | base+Cell[]、空间换时间 | +### 📦 并发容器类 +| **问题** | **核心答案** | **关键点** | +| -------------------- | --------------------- | ----------------------- | +| ConcurrentHashMap | 1.7分段锁→1.8CAS+sync | Segment→Node数组+红黑树 | +| CopyOnWriteArrayList | 写时复制，读多写少 | 读无锁、写复制数组 | +| 同步vs并发容器 | 全局锁 vs 细粒度锁 | fail-fast vs 弱一致性 | +| BlockingQueue | 阻塞队列，生产消费 | put/take自动阻塞 | +### 🧠 JMM类 -#### Semaphore +| **问题** | **核心答案** | **关键点** | +| ------------- | ---------------------- | ---------------------------- | +| JMM作用 | 跨平台内存一致性 | 主内存+工作内存模型 | +| 三大特性 | 原子性、可见性、有序性 | 各自实现方式 | +| happen-before | 操作间偏序关系 | 程序次序、锁定、volatile规则 | +| 内存屏障 | 禁止指令重排 | LoadLoad、StoreStore等 | -信号量主要用于两个目的，一个是用于多个共享资源的互斥使用，另一个用于并发线程数的控制。 +--- -```java -/** - * @description: 模拟抢车位 - * @author: starfish - * @data: 2020-04-04 10:29 - **/ -public class SemaphoreDemo { +## 🎯 面试突击技巧 - public static void main(String[] args) { +### 📝 万能回答框架 - //模拟 3 个车位 - Semaphore semaphore = new Semaphore(3); +1. **背景阐述** (10秒)：简述问题背景和重要性 +2. **核心原理** (30秒)：讲清楚底层实现机制 +3. **关键特点** (20秒)：对比优缺点和适用场景 +4. **实践经验** (20秒)：结合项目经验或最佳实践 - //7 辆车去争抢 - for (int i = 0; i < 7; i++) { - new Thread(()->{ - try { - semaphore.acquire(); //抢到车位 - System.out.println(Thread.currentThread().getName()+"\t抢到车位"); - TimeUnit.SECONDS.sleep(3); - System.out.println(Thread.currentThread().getName()+"\t 停车 3 秒后离开"); - } catch (InterruptedException e) { - e.printStackTrace(); - }finally { - semaphore.release(); - } - System.out.println(Thread.currentThread().getName()+"\t抢到车位"); - },String.valueOf(i)).start(); - } - } -} -``` +### 🔥 加分回答技巧 +- **源码引用**：适当提及关键源码实现 +- **性能数据**：给出具体的性能对比数据 +- **实战经验**：结合实际项目中的使用经验 +- **问题延伸**：主动提及相关的深层问题 +### ⚠️ 常见面试陷阱 +- **概念混淆**：synchronized vs ReentrantLock选择 +- **性能误区**：盲目认为无锁一定比有锁快 +- **使用错误**：volatile不能保证原子性 +- **内存泄漏**：ThreadLocal使用后不清理 +记住这个口诀：**理论扎实、实践丰富、思路清晰、表达准确**！ +--- +> 💡 **最终提醒**： +> +> 1. **循序渐进**：从基础概念到高级应用 +> 2. **结合实践**：每个知识点都举具体使用场景 +> 3. **源码分析**：适当提及关键源码实现 +> 4. **性能对比**：说明不同方案的优缺点 +> 5. **问题解决**：展示解决并发问题的思路 - \ No newline at end of file +**并发编程面试，重在理解原理，贵在实战经验！** 🚀 \ No newline at end of file diff --git a/docs/interview/JVM-FAQ.md b/docs/interview/JVM-FAQ.md index a60efa6e2b..fb9c1afa53 100644 --- a/docs/interview/JVM-FAQ.md +++ b/docs/interview/JVM-FAQ.md @@ -1,30 +1,158 @@ -请谈谈你对 OOM 的认识？ - -GC 垃圾回收算法和垃圾收集器的关系？分别是什么请你谈谈？ +--- +title: 手撕 JVM 面试 +date: 2024-05-31 +tags: + - JVM + - Interview +categories: Interview +--- + +![](https://img.starfish.ink/common/faq-banner.png) + +> 为啥 java 不用 360 进行垃圾回收？ 哈哈哈哈~ +> +> 开玩笑，言归正传。 背诵一下G1 垃圾回收方法类的第 108 行？ +> +> 。。。。。。。。。。。。。。。。。。。 -怎么查看服务器默认的垃圾收集器是哪个？生产上如何配置垃圾收集器的？谈谈你对垃圾收集器的理解？ +## 🗺️ 知识导航 -G1 垃圾收集器？ +### 🏷️ 核心知识分类 -生产环境服务器变慢，诊断思路和性能评估谈谈？ +1. **📦 类加载机制**：类加载过程、双亲委派模型、自定义类加载器 +2. 🧠 **JVM内存模型**：运行时数据区、内存分配、内存溢出、内存泄漏 +3. **🗑️ 垃圾回收机制**：GC算法、垃圾收集器、GC调优、内存分配策略 +4. **🔧 JVM调优实战**：JVM参数、性能监控、问题诊断、调优策略 +5. **⚡ 性能优化**：内存优化、GC优化、代码优化、最佳实践 -假如生产环境出现 CPU 占用过高，请谈谈你的分析思路和定位 +## 一、类加载机制 +### 🎯 类加载机制？类加载过程 +Java 程序运行前，`.java` 文件会被编译成 `.class` 字节码文件，JVM 要执行类里的代码，就必须先把类加载到内存。 + 这个过程由 **类加载器（ClassLoader）** 来完成，具体来说，就是由 ClassLoader 和它的子类来实现的。类加载器本身也是一个类，其实质是把类文件从硬盘读取到内存中。 -## 类加载子系统 +类的加载方式分为**隐式加载和显示加载**。隐式加载指的是程序在使用 new 等方式创建对象时，会隐式地调用类的加载器把对应的类 加载到 JVM 中。显示加载指的是通过直接调用 `class.forName()` 方法来把所需的类加载到 JVM 中。 -### 类加载机制？类加载过程 +- **类加载器的作用**：负责把 `.class` 文件字节码加载到 JVM 内存，并在内存中生成 `java.lang.Class` 对象，供后续使用。 -Java 虚拟机把描述类的数据从 Class 文件加载到内存，并对数据进行校验、转换解析和初始化，最终形成可以被虚拟机直接使用的 Java 类型，这就是虚拟机的加载机制 +Java 虚拟机把描述类的数据从 Class 文件加载到内存，并对数据进行校验、转换解析和初始化，最终形成可以被虚拟机直接使用的 Java 类型，这就是虚拟机的加载机制。 类从被加载到虚拟机内存中开始，到卸载出内存为止，它的整个生命周期包括：**加载、验证、准备、解析、初始化、使用和卸载**七个阶段。(验证、准备和解析又统称为连接，为了支持Java语言的**运行时绑定**，所以**解析阶段也可以是在初始化之后进行的**。以上顺序都只是说开始的顺序，实际过程中是交叉的混合式进行的，加载过程中可能就已经开始验证了) -### 什么是类加载器，类加载器有哪些？这些类加载器都加载哪些文件？ +![jvm-class-load](https://img.starfish.ink/jvm/jvm-class-load.png) + +> ##### 1. **加载（Loading）** +> +> - 核心任务： +> 1. 通过全限定名获取类的二进制字节流（来源包括JAR、网络、动态生成等） +> 2. 将字节流转换为方法区的运行时数据结构。 +> 3. 在堆中生成 `java.lang.Class`对象，作为访问入口 +> - 自定义加载器：可重写 `findClass()`方法实现从非标准来源加载类（如网络）。 +> +> ##### 2. **验证（Verification）** +> +> - 确保字节流符合JVM规范且无害： +> - 文件格式验证：检查魔数（`0xCAFEBABE`）、版本号等。 +> - 元数据验证：检查语义（如是否继承final类）。 +> - 字节码验证：分析程序逻辑（如类型转换有效性）。 +> - 符号引用验证：确认符号引用可解析（如类、方法是否存在）。 +> - 可关闭：生产环境可通过 `-Xverify:none` 跳过部分验证。 +> +> ##### 3. **准备（Preparation）** +> +> - 为**静态变量（类变量）**分配内存并设置初始值： +> - 默认零值：如 `int` 为0，`boolean` 为false，引用类型为 null。 +> - Final常量例外：若变量被 `final static` 修饰，则直接赋代码中的初始值。 +> - 内存分配位置：JDK7+后静态变量存储在堆中而非方法区。 +> +> ##### 4. **解析（Resolution）** +> +> - 将常量池中的符号引用转换为直接引用（内存地址或偏移量）： +> +> - 符号引用：类、方法、字段的描述字符串（如 `java/lang/Object`）。 +> +> - 符号引用：编译时生成的抽象描述，包含类/接口全限定名、字段/方法名称及描述符。例如，`java/lang/String.length:()I` 表示 `String.length()` 方法的符号引用。 +> - 特点：与内存布局无关，仅通过符号定位目标，未加载时无法确定实际地址。 +> +> - 直接引用：指向目标的内存地址或偏移量，如方法在方法表中的索引或字段在对象中的偏移。 +> +> 转换时机： +> +> - **静态解析**：类加载的解析阶段（如 `invokestatic` 调用静态方法）。 +> - **动态解析**：运行时根据实际对象类型确定（如 `invokevirtual` 实现多态） +> +> - 动态绑定：部分解析可能在初始化后执行（如接口方法调用）。 +> +> ##### 5. **初始化（Initialization）** +> +> - 执行类构造器 `()`方法（由编译器自动生成）： +> - 内容：合并静态变量赋值和静态代码块语句，按代码顺序执行。 +> - 线程安全：JVM保证 `()`在多线程下仅执行一次。 +> - **父类优先**：子类初始化前确保父类已初始化 + + + +### 🎯 符号引用是什么？ + +符号引用就是 **字节码层面的抽象标识**，比如类名 `"com/example/Test"`、方法名 `"doSomething()V"`。在解析阶段，JVM 会把这些符号解析成具体的 **内存地址或偏移量**（直接引用）。 + +- 符号引用存在于： + - 类的常量池 + - 方法符号引用 + - 字段符号引用 + + + +### 🎯 触发初始化的条件（主动引用） + +以下情况会触发类的初始化阶段 ： + +1. **创建实例**：通过 `new` 关键字实例化对象。 +2. 访问静态成员：访问类的静态变量（非 `final`）或调用静态方法（包括反射调用 `Class.forName()`）。 + - 例外：若静态变量是 final 常量（如 `final static int a=1`），不会触发初始化。 +3. **反射调用**：使用 `java.lang.reflect` 包的方法对类进行反射调用。 +4. **子类初始化**：初始化子类时，若父类未初始化，则优先初始化父类。 +5. **主类启动**：执行包含 `main()` 方法的类（JVM 启动时的入口类）。 +6. **动态语言支持**：通过 `java.lang.invoke.MethodHandle` 动态调用时。 + +**被动引用示例（不触发初始化）** + +- 通过子类引用父类的静态字段（仅初始化父类）。 +- 通过数组定义引用类（如 `ClassA[] arr = new ClassA[10]`） +- 访问类的 `final static` 常量（编译期已放入常量池） + +### 🎯 类卸载条件 -#### 启动类加载器（引导类加载器，Bootstrap ClassLoader） +1. 所有实例已被GC回收。 +2. 无任何地方引用该类的`Class`对象。 +3. 加载该类的类加载器实例已被GC。 + +- **注意**：由启动类加载器加载的类（如核心类）永不卸载 + + + +### 🎯 **静态变量何时赋值？** + +非final变量在初始化阶段赋值，final常量在准备阶段赋值 + + + +### 🎯 什么是类加载器，类加载器有哪些？这些类加载器都加载哪些文件？ + +类加载器（ClassLoader）是 JVM 的核心组件，负责将 `.class` 文件的二进制数据加载到内存中，并在堆中生成对应的 `java.lang.Class` 对象，作为访问类元数据的入口。其核心功能包括： + +1. **定位并读取字节码文件**（如从本地文件、网络、JAR 包等来源）。 +2. **验证字节码合法性**（如文件格式、语义检查等）。 +3. **分配内存并初始化默认值**（如静态变量的零值）。 +4. **解析符号引用为直接引用**（内存地址或偏移量）。 +5. **执行类的初始化逻辑**（如静态代码块） + + + +##### 启动类加载器（引导类加载器，Bootstrap ClassLoader） - 这个类加载使用 C/C++ 语言实现，嵌套在 JVM 内部 - 它用来加载 Java 的核心库（`JAVA_HOME/jre/lib/rt.jar`、`resource.jar`或`sun.boot.class.path`路径下的内容），用于提供 JVM 自身需要的类 @@ -32,14 +160,14 @@ Java 虚拟机把描述类的数据从 Class 文件加载到内存，并对数 - 加载扩展类和应用程序类加载器，并指定为他们的父类加载器 - 出于安全考虑，Bootstrap 启动类加载器只加载名为java、Javax、sun等开头的类 -#### 扩展类加载器（Extension ClassLoader） +##### 扩展类加载器（Extension ClassLoader） - Java 语言编写，由`sun.misc.Launcher$ExtClassLoader`实现 - 派生于 ClassLoader - 父类加载器为启动类加载器 - 从 `java.ext.dirs` 系统属性所指定的目录中加载类库，或从 JDK 的安装目录的`jre/lib/ext` 子目录（扩展目录）下加载类库。如果用户创建的 JAR 放在此目录下，也会自动由扩展类加载器加载 -#### 应用程序类加载器（也叫系统类加载器，AppClassLoader） +##### 应用程序类加载器（也叫系统类加载器，AppClassLoader） - Java 语言编写，由 `sun.misc.Lanucher$AppClassLoader` 实现 - 派生于 ClassLoader @@ -48,11 +176,39 @@ Java 虚拟机把描述类的数据从 Class 文件加载到内存，并对数 - 该类加载是**程序中默认的类加载器**，一般来说，Java 应用的类都是由它来完成加载的 - 通过 `ClassLoader#getSystemClassLoader()` 方法可以获取到该类加载器 -#### 用户自定义类加载器 +##### 用户自定义类加载器 + +- 继承 `ClassLoader`，可以自定义类加载逻辑，比如从网络/数据库加载类。 + + -在 Java 的日常应用程序开发中，类的加载几乎是由 3 种类加载器相互配合执行的，在必要时，我们还可以自定义类加载器，来定制类的加载方式 +### 🎯 什么是双亲委派机制？它有啥优势？ -##### 为什么要自定义类加载器？ +Java 虚拟机对 class 文件采用的是**按需加载**的方式，也就是说当需要使用该类的时候才会将它的 class 文件加载到内存生成 class 对象。而且加载某个类的 class 文件时，Java 虚拟机采用的是双亲委派模式，即把请求交给父类处理，它是一种任务委派模式。 + +简单说就是当类加载器（Class-Loader）试图加载某个类型的时候，除非父加载器找不到相应类型，否则尽量将这个任务代理给当前加载器的父加载器去做。使用委派模型的目的是避免重复加载 Java 类型。 + +- **核心原则**：类加载器收到加载请求时，优先委派父类加载器完成加载，若父类无法加载，则由自身尝试加载。 +- **意义**：避免重复加载，保证核心类安全（如防止自定义`java.lang.String`覆盖系统类） + +![](https://img.starfish.ink/jvm/classloader.png) + +> **工作过程** +> +> 1. 当某个类加载器收到 `loadClass("java.lang.String")` 请求： +> - 它会先交给 **父加载器** 去处理。 +> 2. 父加载器继续向上委托，直到顶层 **Bootstrap ClassLoader**。 +> 3. Bootstrap 找到 `rt.jar` 中的 `java.lang.String`，就加载返回。 +> 4. 如果父加载器都找不到，才会由子加载器自己去尝试。 +> +> **优势** +> +> - **避免类的重复加载**，JVM 中区分不同类，不仅仅是根据类名，相同的 class 文件被不同的 ClassLoader 加载就属于两个不同的类（比如，Java中的Object类，无论哪一个类加载器要加载这个类，最终都是委派给处于模型最顶端的启动类加载器进行加载，如果不采用双亲委派模型，由各个类加载器自己去加载的话，系统中会存在多种不同的 Object 类） +> - **保护程序安全，防止核心 API 被随意篡改**，避免用户自己编写的类动态替换 Java 的一些核心类，比如我们自定义类：`java.lang.String` + + + +### 🎯 为什么要自定义类加载器？ - 隔离加载类 - 修改类加载的方式 @@ -61,673 +217,2013 @@ Java 虚拟机把描述类的数据从 Class 文件加载到内存，并对数 -### 多线程的情况下，类的加载为什么不会出现重复加载的情况？ +### 🎯 自定义了一个String，那么会加载哪个String？ -双亲委派 +| **场景** | **能否加载自定义`String`** | **原因** | +| ---------------------- | -------------------------- | ----------------------------------------------------------- | +| 包名为`java.lang` | ❌ 不能 | 双亲委派机制优先加载系统类，且JVM安全机制禁止篡改核心包 | +| 包名非`java.lang` | ✅ 能 | 应用类加载器直接加载用户类路径下的自定义类，与系统类无冲突 | +| 自定义类加载器打破委派 | ✅ 能（但类型不同） | 绕过双亲委派直接加载，但JVM认为自定义类与系统类是独立存在的 | -### 什么是双亲委派机制？它有啥优势？可以打破这种机制吗？ +- JVM会阻止用户定义以`java.lang`开头的包名，直接抛出`SecurityException: Prohibited package name: java.lang`，避免核心类被篡改 +- 若自定义`String`类的包名不同（如`com.example.String`），应用程序类加载器（AppClassLoader）会直接加载它，不会与系统`String`冲突。 +- 使用时需明确指定全限定类名（如`com.example.String str = new com.example.String()`） -Java 虚拟机对 class 文件采用的是**按需加载**的方式，也就是说当需要使用该类的时候才会将它的 class 文件加载到内存生成 class 对象。而且加载某个类的 class 文件时，Java 虚拟机采用的是双亲委派模式，即把请求交给父类处理，它是一种任务委派模式。 +> 如果你自定义了一个 `String` 类型的数据，那么系统会加载哪个 `String` 取决于几个因素： +> +> 1. **作用域（Scope）**： +> - 如果自定义的 `String` 位于局部作用域内（例如一个方法内部），那么这个局部变量将会被优先使用。 +> - 如果自定义的 `String` 位于类的成员变量或者全局变量，那么在该类或全局范围内会优先使用这个自定义的 `String`。 +> 2. **类加载顺序**： +> - 如果你在某个类中自定义了一个名为 `String` 的类型，这个自定义的类型会在该类的上下文中被优先加载。 +> - 通常来说，标准库中的 `java.lang.String` 会在默认情况下被加载使用，但如果你定义了一个同名类且在当前作用域内，这个自定义的类会覆盖标准库中的类。 +> 3. **包的使用**： +> - 如果你的自定义 `String` 类型在某个特定的包中，而标准库的 `java.lang.String` 是在 `java.lang` 包中，那么在不明确指定包名的情况下，当前作用域内的类型会优先被加载。 +> - 可以通过完全限定名（Fully Qualified Name）来区分，例如使用 `java.lang.String` 来确保使用的是标准库中的 `String`。 +> +> 以下是一个示例来说明作用域和类加载的情况： +> +> ```JAVA +> public class MyClass { +> // 自定义的String类 +> public static class String { +> public void print() { +> System.out.println("This is my custom String class."); +> } +> } +> +> public static void main(String[] args) { +> // 使用自定义的String类 +> String myString = new String(); +> myString.print(); // 输出: This is my custom String class. +> +> // 使用标准库的String类 +> java.lang.String standardString = new java.lang.String("Hello, World!"); +> System.out.println(standardString); // 输出: Hello, World! +> } +> } +> ``` +> +> 在这个示例中： +> +> - `mypackage.MyClass.String` 是自定义的 `String` 类，并在 `main` 方法中优先使用。 +> - `java.lang.String` 是标准库中的 `String` 类，通过完全限定名确保其正确加载。 +> +> 综上所述，系统会加载哪个 `String`，取决于你的自定义 `String` 在代码中的定义位置和使用范围。如果你明确指定了完全限定名，那么系统会准确加载你指定的那个 `String` 类。 +> +> 针对 java.*开头的类，jvm 的实现中已经保证了必须由 bootstrp 来加载 +> + + + +### 🎯 多线程的情况下，类的加载为什么不会出现重复加载的情况？ -**工作过程** +双亲委派 -- 如果一个类加载器收到了类加载请求，它并不会自己先去加载，而是把这个请求委托给父类的加载器去执行； -- 如果父类加载器还存在其父类加载器，则进一步向上委托，依次递归，请求最终将到达顶层的启动类加载器； -- 如果父类加载器可以完成类加载任务，就成功返回，倘若父类加载器无法完成此加载任务，子加载器才会尝试自己去加载，这就是双亲委派模式 +**三个类加载器的关系，不是父子关系，是组合关系。** -**优势** +看看类加载器的加载类的方法 loadClass -- 避免类的重复加载，JVM 中区分不同类，不仅仅是根据类名，相同的 class 文件被不同的 ClassLoader 加载就属于两个不同的类（比如，Java中的Object类，无论哪一个类加载器要加载这个类，最终都是委派给处于模型最顶端的启动类加载器进行加载，如果不采用双亲委派模型，由各个类加载器自己去加载的话，系统中会存在多种不同的 Object 类） -- 保护程序安全，防止核心 API 被随意篡改，避免用户自己编写的类动态替换 Java 的一些核心类，比如我们自定义类：`java.lang.String` +```java +protected Class loadClass(String name, boolean resolve) + throws ClassNotFoundException +{ + //看，这里有锁 + synchronized (getClassLoadingLock(name)) { + // First, check if the class has already been loaded + //去看看类是否被加载过，如果被加载过，就立即返回 + Class c = findLoadedClass(name); + if (c == null) { + long t0 = System.nanoTime(); + try { + //这里通过是否有parent来区分启动类加载器和其他2个类加载器 + if (parent != null) { + //先尝试请求父类加载器去加载类，父类加载器加载不到，再去尝试自己加载类 + c = parent.loadClass(name, false); + } else { + //启动类加载器加载类，本质是调用c++的方法 + c = findBootstrapClassOrNull(name); + } + } catch (ClassNotFoundException e) { + // ClassNotFoundException thrown if class not found + // from the non-null parent class loader + } + //如果父类加载器加载不到类，子类加载器再尝试自己加载 + if (c == null) { + // If still not found, then invoke findClass in order + // to find the class. + long t1 = System.nanoTime(); + //加载类 + c = findClass(name); + + // this is the defining class loader; record the stats + sun.misc.PerfCounter.getParentDelegationTime().addTime(t1 - t0); + sun.misc.PerfCounter.getFindClassTime().addElapsedTimeFrom(t1); + sun.misc.PerfCounter.getFindClasses().increment(); + } + } + if (resolve) { + resolveClass(c); + } + return c; + } +} +``` -### 自定义了一个String，那么会加载哪个String？ +总结一下loadClass方法的大概逻辑： -针对java.*开头的类，jvm的实现中已经保证了必须由bootstrp来加载 +1. 首先加锁，防止多线程的情况下，重复加载同一个类 +2. 当加载类的时候，先请求其父类加载器去加载类，如果父类加载器无法加载类时，才自己尝试去加载类。 +![图片摘自网络](https://uploadfiles.nowcoder.com/files/20210318/9603430_1616033889317/154ad50dc8dd4b109ab5e2100c63fe66.png) +上面的源码解析，可以回答问题：在多线程的情况下，类的加载为什么不会出现重复加载的情况？ -### 简单说说你了解的类加载器，可以打破双亲委派么，怎么打破。 -**思路：** 先说明一下什么是类加载器，可以给面试官画个图，再说一下类加载器存在的意义，说一下双亲委派模型，最后阐述怎么打破双亲委派模型。 -**我的答案：** +### 🎯 可以打破双亲委派机制吗？ -#### 1) 什么是类加载器？ +- 双亲委派模型并不是一个强制性的约束模型，而是 Java 设计者推荐给开发者的类加载器实现方式，可以“被破坏”，只要我们自定义类加载器，**重写 `loadClass()` 方法**，指定新的加载逻辑就破坏了，重写 `findClass()` 方法不会破坏双亲委派。 -**类加载器** 就是根据指定全限定名称将class文件加载到JVM内存，转为Class对象。 +- 双亲委派模型有一个问题：顶层 ClassLoader，无法加载底层 ClassLoader 的类。典型例子 JNDI、JDBC，所以加入了线程上下文类加载器（Thread Context ClassLoader），可以通过 `Thread.setContextClassLoaser()`设置该类加载器，然后顶层 ClassLoader 再使用 `Thread.getContextClassLoader()` 获得底层的 ClassLoader 进行加载。 -> - 启动类加载器（Bootstrap ClassLoader）：由C++语言实现（针对HotSpot）,负责将存放在\lib目录或-Xbootclasspath参数指定的路径中的类库加载到内存中。 -> - 其他类加载器：由Java语言实现，继承自抽象类ClassLoader。如： -> -> > - 扩展类加载器（Extension ClassLoader）：负责加载\lib\ext目录或java.ext.dirs系统变量指定的路径中的所有类库。 -> > - 应用程序类加载器（Application ClassLoader）。负责加载用户类路径（classpath）上的指定类库，我们可以直接使用这个类加载器。一般情况，如果我们没有自定义类加载器默认就是用这个加载器。 +- Tomcat 中使用了自定义 ClassLoader，并且也破坏了双亲委托机制。每个应用使用 WebAppClassloader 进行单独加载，他首先使用 WebAppClassloader 进行类加载，如果加载不了再委托父加载器去加载，这样可以保证每个应用中的类不冲突。每个 tomcat 中可以部署多个项目，每个项目中存在很多相同的 class 文件（很多相同的jar包），他们加载到 jvm 中可以做到互不干扰。 -#### 2）双亲委派模型 + > Tomcat 为每个 Web 应用创建独立的 `WebAppClassLoader`，确保不同应用中同名类（如不同版本的 `commons-lang`）互不干扰。例如： + > + > - **应用 A** 使用 `commons-lang-2.0.jar`，由 `WebAppClassLoaderA` 加载。 + > - **应用 B** 使用 `commons-lang-3.0.jar`，由 `WebAppClassLoaderB` 加载。 即使类名相同，两个应用中的类也会被 JVM 视为不同的类 -**双亲委派模型工作过程是：** +- 利用破坏双亲委派来实现**代码热替换**（每次修改类文件，不需要重启服务）。因为一个 Class 只能被一个 ClassLoader 加载一次，否则会报 `java.lang.LinkageError`。当我们想要实现代码热部署时，可以每次都 new 一个自定义的 ClassLoader 来加载新的 Class文件。JSP 的实现动态修改就是使用此特性实现。 -> 如果一个类加载器收到类加载的请求，它首先不会自己去尝试加载这个类，而是把这个请求委派给父类加载器完成。每个类加载器都是如此，只有当父加载器在自己的搜索范围内找不到指定的类时（即ClassNotFoundException），子加载器才会尝试自己去加载。 -双亲委派模型图： +### 🎯 i++ 和 i=i+1在字节码文件上怎么实现及区别？ +**i++** 的字节码（使用 `javap -c` 查看）： -![img](https://user-gold-cdn.xitu.io/2019/7/23/16c1c54cf4ad886b?imageView2/0/w/1280/h/960/format/webp/ignore-error/1) +``` +0: iconst_0 // 将 0 压入操作数栈 +1: istore_1 // 将栈顶的值（0）存储到局部变量 1（i） +2: iinc 1, 1 // i 递增 1（相当于 i++） +5: return +``` +**i = i + 1** 的字节码： + +``` +0: iconst_0 // 将 0 压入操作数栈 +1: istore_1 // 将栈顶的值（0）存储到局部变量 1（i） +2: iload_1 // 将局部变量 1（i）加载到操作数栈 +3: iconst_1 // 将常量 1 推入操作数栈 +4: iadd // 栈顶两个数相加（i + 1） +5: istore_1 // 将结果存储到局部变量 1（i） +6: return +``` +**字节码差异**： -#### 3）为什么需要双亲委派模型？ +- `i++` 通过 **`iinc`** 指令直接修改局部变量 `i` 的值，而 `i = i + 1` 需要先加载变量 `i` 到栈中，再加上 1，最后将结果存储回 `i`。因此，`i++` 相比 `i = i + 1` 更为简洁，直接修改变量，而 `i = i + 1` 需要额外的加法和栈操作。 -在这里，先想一下，如果没有双亲委派，那么用户是不是可以**自己定义一个java.lang.Object的同名类**，**java.lang.String的同名类**，并把它放到ClassPath中,那么**类之间的比较结果及类的唯一性将无法保证**，因此，为什么需要双亲委派模型？**防止内存中出现多份同样的字节码** +**效率差异**： -#### 4）怎么打破双亲委派模型？ +- **`i++`** 的字节码更高效，使用 **`iinc`** 指令直接增加局部变量的值，不需要进行加法运算。 +- **`i = i + 1`** 的字节码则需要进行 **加载、加法、存储** 三步操作，涉及到更多的指令和操作数栈。 -打破双亲委派机制则不仅**要继承ClassLoader**类，还要**重写loadClass和findClass**方法。 +**可读性差异**： +- `i++` 更符合编程习惯和语法逻辑，它明确表示这是一个递增操作，并且在执行后会返回 `i` 的旧值。 +- `i = i + 1` 更为直观，表达了 `i` 增加的过程，但它是一个赋值操作，语义上更清晰一些。 ------- ------ -## 内存管理 +## 二、JVM内存模型 -### Java内存结构？ +### 🎯 Java 内存结构？| JVM 内存区域的划分 -![](https://user-gold-cdn.xitu.io/2019/7/8/16bd08c33a3d751b?imageView2/0/w/1280/h/960/format/webp/ignore-error/1) +> JVM内存结构主要分为以下几个部分： +> +> | 区域类型 | 包含区域 | 作用 | 是否会 OOM（内存溢出） | +> | -------------- | -------------------------------------- | ------------------------------------------ | ------------------------- | +> | **线程共享区** | 堆（Heap） | 存储对象实例、数组，GC 主要回收区域 | 会（对象过多时） | +> | | 方法区（Method Area） | 存储类信息、常量、静态变量、字节码等 | 会（类过多 / 常量过大时） | +> | **线程私有区** | 虚拟机栈（VM Stack） | 存储方法调用的栈帧（局部变量、操作数栈等） | 会（栈深度过大时） | +> | | 本地方法栈（Native Stack） | 支持 native 方法调用（如 C 语言方法） | 会 | +> | | 程序计数器（Program Counter Register） | 记录当前线程执行的字节码指令地址 | 不会（内存固定） | -方法区和堆是所有线程共享的内存区域；而Java栈、本地方法栈和程序计数器是线程私有的内存区域。 +![1.png](https://img.starfish.ink/jvm/jvm-framework.png) -- Java堆是Java虚拟机所管理的内存中最大的一块。Java堆是被所有线程共享的一块内存区域，在虚拟机启动时创建。此内存区域的唯一目的就是存放对象实例，几乎所有的对象实例都在这里分配内存。 -- 方法区用于存储已被虚拟机加载的类信息、常量、静态变量、即时编译器编译后的代码等数据。 -- 程序计数器（Program Counter Register）是一块较小的内存空间，它的作用可以看做是当前线程所执行的字节码的行号指示器。 -- JVM栈（JVM Stacks）,与程序计数器一样，也是线程私有的，它的生命周期与线程相同。虚拟机栈描述的是Java方法执行的内存模型：每个方法被执行的时候都会同时创建一个栈帧（Stack Frame）用于存储局部变量表、操作栈、动态链接、方法出口等信息。每一个方法被调用直至执行完成的过程，就对应着一个栈帧在虚拟机栈中从入栈到出栈的过程。 -- 本地方法栈（Native Method Stacks）与虚拟机栈所发挥的作用是非常相似的，其区别不过是虚拟机栈为虚拟机执行Java方法（也就是字节码）服务，而本地方法栈则是为虚拟机使用到的Native方法服务。 +通常可以把 JVM 内存区域分为下面几个方面，其中，有的区域是以线程为单位，而有的区域则是整个 JVM 进程唯一的。 +首先，**程序计数器**（PC，Program Counter Register）。在 JVM 规范中，每个线程都有它自己的程序计数器，并且任何时间一个线程都只有一个方法在执行，也就是所谓的当前方法。程序计数器会存储当前线程正在执行的 Java 方法的 JVM 指令地址；或者，如果是在执行本地方法，则是未指定值（undefined）。 +第二，**Java 虚拟机栈**（Java Virtual Machine Stack），早期也叫 Java 栈。每个线程在创建时都会创建一个虚拟机栈，其内部保存一个个的栈帧（Stack Frame），对应着一次次的 Java 方法调用。 +前面谈程序计数器时，提到了当前方法；同理，在一个时间点，对应的只会有一个活动的栈帧，通常叫作当前帧，方法所在的类叫作当前类。如果在该方法中调用了其他方法，对应的新的栈帧会被创建出来，成为新的当前帧，一直到它返回结果或者执行结束。JVM 直接对 Java 栈的操作只有两个，就是对栈帧的压栈和出栈。 -### 内存泄露和内存溢出的区别？ +栈帧中存储着局部变量表、操作数（operand）栈、动态链接、方法正常退出或者异常退出的定义等。 -内存溢出 out of memory，是指程序在申请内存时，没有足够的内存空间供其使用，出现out of memory；比如申请了一个integer,但给它存了long才能存下的数，那就是内存溢出。 +第三，**堆**（Heap），它是 Java 内存管理的核心区域，用来放置 Java 对象实例，几乎所有创建的 Java 对象实例都是被直接分配在堆上。堆被所有的线程共享，在虚拟机启动时，我们指定的“Xmx”之类参数就是用来指定最大堆空间等指标。 -内存泄露 memory leak，是指程序在申请内存后，无法释放已申请的内存空间，一次内存泄露危害可以忽略，但内存泄露堆积后果很严重，无论多少内存,迟早会被占光。 +理所当然，堆也是垃圾收集器重点照顾的区域，所以堆内空间还会被不同的垃圾收集器进行进一步的细分，最有名的就是新生代、老年代的划分。 -memory leak会最终会导致out of memory！ +第四，**方法区**（Method Area）。这也是所有线程共享的一块内存区域，用于存储所谓的元（Meta）数据，例如类结构信息，以及对应的运行时常量池、字段、方法代码等。 +由于早期的 Hotspot JVM 实现，很多人习惯于将方法区称为永久代（Permanent Generation）。Oracle JDK 8 中将永久代移除，同时增加了元数据区（Metaspace）。 +第五，**运行时常量池**（Run-Time Constant Pool），这是方法区的一部分。如果仔细分析过反编译的类文件结构，你能看到版本号、字段、方法、超类、接口等各种信息，还有一项信息就是常量池。Java 的常量池可以存放各种常量信息，不管是编译期生成的各种字面量，还是需要在运行时决定的符号引用，所以它比一般语言的符号表存储的信息更加宽泛。 -### 什么情况下会发生栈内存溢出? +第六，**本地方法栈**（Native Method Stack）。它和 Java 虚拟机栈是非常相似的，支持对本地方法的调用，也是每个线程都会创建一个。在 Oracle Hotspot JVM 中，本地方法栈和 Java 虚拟机栈是在同一块儿区域，这完全取决于技术实现的决定，并未在规范中强制。 -- 栈是线程私有的，他的生命周期与线程相同，每个方法在执行的时候都会创建一个栈帧，用来存储局部变量表，操作数栈，动态链接，方法出口等信息。局部变量表又包含基本数据类型，对象引用类型 -- 如果线程请求的栈深度大于虚拟机所允许的最大深度，将抛出StackOverflowError异常，方法递归调用产生这种结果。 -- 如果Java虚拟机栈可以动态扩展，并且扩展的动作已经尝试过，但是无法申请到足够的内存去完成扩展，或者在新建立线程的时候没有足够的内存去创建对应的虚拟机栈，那么Java虚拟机将抛出一个OutOfMemory 异常。(线程启动过多) -- 参数 -Xss 去调整JVM栈的大小 +> - 直接内存（Direct Memory）区域， Direct Buffer 所直接分配的内存，也是个容易出现问题的地方。尽管，在 JVM 工程师的眼中，并不认为它是 JVM 内部内存的一部分，也并未体现 JVM 内存模型中。 +> - JVM 本身是个本地程序，还需要其他的内存去完成各种基本任务，比如，JIT Compiler 在运行时对热点方法进行编译，就会将编译后的方法储存在 Code Cache 里面；GC 等功能需要运行在本地线程之中，类似部分都需要占用内存空间。这些是实现 JVM JIT 等功能的需要，但规范中并不涉及。 +> +> ![JVM Overview](https://abiasforaction.net/wp-content/uploads/2020/12/xJVM_Overview.png,qx60851.pagespeed.ic.yS2Huqqhjz.webp) -### JVM内存为什么要分成新生代，老年代，持久代。新生代中为什么要分为Eden和Survivor。 +### 🎯 1.7 和 1.8 中 jvm 内存结构的区别 -#### 1）共享内存区划分 +在 Java8 中，永久代被移除，被一个称为元空间的区域代替，元空间的本质和永久代类似，都是方法区的实现。 -- 共享内存区 = 持久带 + 堆 -- 持久带 = 方法区 + 其他 -- Java堆 = 老年代 + 新生代 -- 新生代 = Eden + S0 + S1 +元空间（Java8）和永久代（Java7）之间最大的区别就是：永久代使用的 JVM 的堆内存，Java8 以后的元空间并不在虚拟机中而是使用本机物理内存。 + +因此，默认情况下，元空间的大小仅受本地内存限制。类的元数据放入 natice memory，字符串池和类的静态变量放入堆中。 + +![](https://dl-harmonyos.51cto.com/images/202212/d37207e632cd193510c4713b1db551924c2d36.png) + + + +### 🎯 JVM 堆内部结构？ | JVM 堆内存为什么要分成新生代，老年代，持久代 -#### 2）一些参数的配置 +> 堆内存是JVM中最大的一块内存区域，主要用于存储对象实例： +> +> ``` +> ┌─────────────────────────────────────────────────────────────┐ +> │ 堆内存结构 │ +> ├─────────────────────────────────────────────────────────────┤ +> │ 新生代(Young Generation) - 1/3堆空间 │ +> │ ┌─────────────┐ ┌─────────────┐ ┌─────────────────────┐ │ +> │ │ Eden区 │ │ Survivor0 │ │ Survivor1 │ │ +> │ │ (8/10) │ │ (1/10) │ │ (1/10) │ │ +> │ └─────────────┘ └─────────────┘ └─────────────────────┘ │ +> ├─────────────────────────────────────────────────────────────┤ +> │ 老年代(Old Generation) - 2/3堆空间 │ +> │ ┌─────────────────────────────────────────────────────────┐ │ +> │ │ │ │ +> │ │ │ │ +> │ └─────────────────────────────────────────────────────────┘ │ +> └─────────────────────────────────────────────────────────────┘ +> ``` +> +> **新生代特点**： +> +> - **Eden区**：新创建的对象首先分配在这里 +> - **Survivor区**：经过Minor GC后存活的对象会被移动到Survivor区 +> - **比例**：Eden:S0:S1 = 8:1:1 +> +> **老年代特点**： +> +> - 存储经过多次Minor GC后仍然存活的对象 +> - 大对象直接进入老年代 +> - 触发Major GC/Full GC + +1. 新生代：新生代是大部分对象创建和销毁的区域，在通常的 Java 应用中，绝大部分对象生命周期都是很短暂的。其内部又分为 Eden 区域，作为对象初始分配的区域；两个 Survivor，有时候也叫 from、to 区域，被用来放置从 Minor GC 中保留下来的对象。 + + - JVM 会随意选取一个 Survivor 区域作为“to”，然后会在 GC 过程中进行区域间拷贝，也就是将 Eden 中存活下来的对象和 from 区域的对象，拷贝到这个“to”区域。这种设计主要是为了防止内存的碎片化，并进一步清理无用对象。 + + - 从内存模型而不是垃圾收集的角度，对 Eden 区域继续进行划分，Hotspot JVM 还有一个概念叫做 Thread Local Allocation Buffer（TLAB），据我所知所有 OpenJDK 衍生出来的 JVM 都提供了 TLAB 的设计。这是 JVM 为每个线程分配的一个私有缓存区域，否则，多线程同时分配内存时，为避免操作同一地址，可能需要使用加锁等机制，进而影响分配速度，你可以参考下面的示意图。从图中可以看出，TLAB 仍然在堆上，它是分配在 Eden 区域内的。其内部结构比较直观易懂，start、end 就是起始地址，top（指针）则表示已经分配到哪里了。所以我们分配新对象，JVM 就会移动 top，当 top 和 end 相遇时，即表示该缓存已满，JVM 会试图再从 Eden 里分配一块儿。 + + ![](https://static001.geekbang.org/resource/image/f5/bd/f546839e98ea5d43b595235849b0f2bd.png) + +2. 老年代：放置长生命周期的对象，通常都是从 Survivor 区域拷贝过来的对象。当然，也有特殊情况，我们知道普通的对象会被分配在 TLAB 上；如果对象较大，JVM 会试图直接分配在 Eden 其他位置上；如果对象太大，完全无法在新生代找到足够长的连续空闲空间，JVM 就会直接分配到老年代。 + +3. 永久代：这部分就是早期 Hotspot JVM 的方法区实现方式了，储存 Java 类元数据、常量池、Intern 字符串缓存，在 JDK 8 之后就不存在永久代这块儿了。 + + +#### 一些参数的配置 - 默认的，新生代 ( Young ) 与老年代 ( Old ) 的比例的值为 1:2 ，可以通过参数 –XX:NewRatio 配置。 - 默认的，Edem : from : to = 8 : 1 : 1 ( 可以通过参数 –XX:SurvivorRatio 来设定) - Survivor区中的对象被复制次数为15(对应虚拟机参数 -XX:+MaxTenuringThreshold) -#### 3)为什么要分为Eden和Survivor?为什么要设置两个Survivor区？ +#### 为什么要分为Eden和Survivor?为什么要设置两个Survivor区？ -- 如果没有Survivor，Eden区每进行一次Minor GC，存活的对象就会被送到老年代。老年代很快被填满，触发Major GC.老年代的内存空间远大于新生代，进行一次Full GC消耗的时间比Minor GC长得多,所以需要分为Eden和Survivor。 -- Survivor的存在意义，就是减少被送到老年代的对象，进而减少Full GC的发生，Survivor的预筛选保证，只有经历16次Minor GC还能在新生代中存活的对象，才会被送到老年代。 -- 设置两个Survivor区最大的好处就是解决了碎片化，刚刚新建的对象在Eden中，经历一次Minor GC，Eden中的存活对象就会被移动到第一块survivor space S0，Eden被清空；等Eden区再满了，就再触发一次Minor GC，Eden和S0中的存活对象又会被复制送入第二块survivor space S1（这个过程非常重要，因为这种复制算法保证了S1中来自S0和Eden两部分的存活对象占用连续的内存空间，避免了碎片化的发生） +- 如果没有Survivor，Eden区每进行一次Minor GC，存活的对象就会被送到老年代。老年代很快被填满，触发Major GC。老年代的内存空间远大于新生代，进行一次 Full GC 消耗的时间比 Minor GC 长得多,所以需要分为 Eden 和 Survivor。 +- Survivor 的存在意义，就是减少被送到老年代的对象，进而减少 Full GC 的发生 +- 设置两个 Survivor 区最大的好处就是解决了碎片化，刚刚新建的对象在 Eden 中，经历一次 Minor GC，Eden 中的存活对象就会被移动到第一块 survivor space S0，Eden 被清空；等 Eden 区再满了，就再触发一次 Minor GC，Eden 和 S0 中的存活对象又会被复制送入第二块 survivor space S1（这个过程非常重要，因为这种复制算法保证了S1中来自S0和Eden两部分的存活对象占用连续的内存空间，避免了碎片化的发生） ------- +### 🎯 方法区存储什么内容？ -## GC +方法区存储的是类级别的数据，包括： -### 1. JVM 垃圾回收的时候如何确定垃圾？ 你知道什么是 GC Roots 吗？GC Roots 如何确定，那些对象可以作为 GC Roots? +1. **类信息**： + - 类的完整名称 + - 类的直接父类 + - 类的修饰符 + - 类的接口列表 -内存中不再使用的空间就是垃圾 +2. **字段信息**： + - 字段名称 + - 字段类型 + - 字段修饰符 -引用计数法和可达性分析 +3. **方法信息**： + - 方法名称 + - 方法返回类型 + - 方法参数 + - 方法修饰符 + - 方法的字节码 -![](https://tva1.sinaimg.cn/large/00831rSTly1gdeam8z27oj31e60mudzv.jpg) +4. **常量池**： + - 字符串常量 + - 数字常量 + - 类和接口的符号引用 -![](https://tva1.sinaimg.cn/large/00831rSTly1gdeaofj7tsj31cs0nstsz.jpg) +5. **静态变量**： + - 类变量（static修饰的变量） -哪些对象可以作为 GC Root 呢，有以下几类 +**JDK版本变化**： -- 虚拟机栈（栈帧中的本地变量表）中引用的对象 -- 方法区中类静态属性引用的对象 -- 方法区中常量引用的对象 -- 本地方法栈中 JNI（即一般说的 Native 方法）引用的对象 +- **JDK 1.7之前**：方法区称为"永久代"(PermGen) +- **JDK 1.8之后**：方法区称为"元空间"(Metaspace)，使用本地内存 -### JVM中一次完整的GC流程是怎样的，对象如何晋升到老年代 +### 🎯 栈帧的内部结构? -**思路：** 先描述一下Java堆内存划分，再解释Minor GC，Major GC，full GC，描述它们之间转化流程。 +在 Java 中，**虚拟机栈**（JVM Stack）是 JVM 内存模型的一个重要组成部分，它用于存储方法调用的相关信息，如局部变量、操作数栈、方法调用的返回地址等。每个线程在执行时都会有一个独立的虚拟机栈，确保多线程间的数据隔离。 -**我的答案：** +- **每个线程都有自己的虚拟机栈**，栈是按照方法调用来组织的。 +- 栈中的数据是 **方法调用栈帧**（Stack Frame），每个栈帧表示一个方法的执行。 +- 方法在执行时，JVM 会为每个方法分配一个栈帧，每当方法调用时会将栈帧压入栈中，方法返回时会将栈帧弹出。 -- Java堆 = 老年代 + 新生代 -- 新生代 = Eden + S0 + S1 -- 当 Eden 区的空间满了， Java虚拟机会触发一次 Minor GC，以收集新生代的垃圾，存活下来的对象，则会转移到 Survivor区。 -- **大对象**（需要大量连续内存空间的Java对象，如那种很长的字符串）**直接进入老年态**； -- 如果对象在Eden出生，并经过第一次Minor GC后仍然存活，并且被Survivor容纳的话，年龄设为1，每熬过一次Minor GC，年龄+1，**若年龄超过一定限制（15），则被晋升到老年态**。即**长期存活的对象进入老年态**。 -- 老年代满了而**无法容纳更多的对象**，Minor GC 之后通常就会进行Full GC，Full GC 清理整个内存堆 – **包括年轻代和年老代**。 -- Major GC **发生在老年代的GC**，**清理老年区**，经常会伴随至少一次Minor GC，**比Minor GC慢10倍以上**。 +每个**栈帧**（Stack Frame）中存储着： +- **局部变量表**：存储方法的局部变量（基本类型、对象引用、returnAddress 类型），容量在编译期确定（如 `int a=1` 占用 1 个槽位，`long b=2` 占用 2 个槽位）。 +- **操作数栈**：执行字节码指令时的临时数据栈（如 `a+b` 运算时，需先将 a、b 入栈，再执行加法指令）。 +- **动态链接**：指向方法区中该方法的符号引用（将符号引用转为直接引用，支持多态）。 +- **方法返回地址**：方法执行完毕后，返回调用者的指令地址（如正常返回的 pc 计数器值，或异常处理的地址）。 +- 一些附加信息 +![图片](https://mmbiz.qpic.cn/mmbiz_jpg/Z0fxkgAKKLMAVEVNlW7gDqZIFzSvVq29fMr12sicgr3HIgRFtFRY8IAcDvwP6orNRRIojrn3edcS3h2ibblgAgQg/640?wx_fmt=jpeg&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) -### 你知道哪几种垃圾收集器，各自的优缺点，重点讲下cms和G1，包括原理，流程，优缺点。 +### 🎯 方法返回地址什么时候回收？ -**思路：** 一定要记住典型的垃圾收集器，尤其cms和G1，它们的原理与区别，涉及的垃圾回收算法。 +方法返回地址存储在栈帧中，其回收与 **栈帧出栈** 同步： -**我的答案：** +- 当方法正常执行完毕（遇到 `return` 指令）或异常终止（未被捕获的异常导致方法退出）时，当前栈帧会从虚拟机栈中弹出。 +- 栈帧出栈后，其包含的 “方法返回地址”“局部变量表”“操作数栈” 等数据会被自动回收（无需 GC 参与，属于线程私有内存的自动释放）。 -#### 1）几种垃圾收集器： -- **Serial收集器：** 单线程的收集器，收集垃圾时，必须stop the world，使用复制算法。 -- **ParNew收集器：** Serial收集器的多线程版本，也需要stop the world，复制算法。 -- **Parallel Scavenge收集器：** 新生代收集器，复制算法的收集器，并发的多线程收集器，目标是达到一个可控的吞吐量。如果虚拟机总共运行100分钟，其中垃圾花掉1分钟，吞吐量就是99%。 -- **Serial Old收集器：** 是Serial收集器的老年代版本，单线程收集器，使用标记整理算法。 -- **Parallel Old收集器：** 是Parallel Scavenge收集器的老年代版本，使用多线程，标记-整理算法。 -- **CMS(Concurrent Mark Sweep) 收集器：** 是一种以获得最短回收停顿时间为目标的收集器，**标记清除算法，运作过程：初始标记，并发标记，重新标记，并发清除**，收集结束会产生大量空间碎片。 -- **G1收集器：** 标记整理算法实现，**运作流程主要包括以下：初始标记，并发标记，最终标记，筛选标记**。不会产生空间碎片，可以精确地控制停顿。 -#### 2）CMS收集器和G1收集器的区别： +### 🎯 程序计数器什么时候为空? -- CMS收集器是老年代的收集器，可以配合新生代的Serial和ParNew收集器一起使用； -- G1收集器收集范围是老年代和新生代，不需要结合其他收集器使用； -- CMS收集器以最小的停顿时间为目标的收集器； -- G1收集器可预测垃圾回收的停顿时间 -- CMS收集器是使用“标记-清除”算法进行的垃圾回收，容易产生内存碎片 -- G1收集器使用的是“标记-整理”算法，进行了空间整合，降低了内存空间碎片。 +程序计数器的作用是 **记录当前线程执行的字节码指令地址**（如下一条要执行的指令位置），其 “为空” 的场景只有一种： +- **当线程执行的是 native 方法时**（如调用 `System.currentTimeMillis()` 等本地方法），由于 native 方法由非 Java 语言实现（如 C/C++），其执行不受 JVM 字节码指令控制，此时程序计数器的值为 **undefined**（可理解为 “空”）。 +其他场景（执行 Java 方法时），程序计数器始终存储有效指令地址（即使方法阻塞，计数器也会保留当前位置，线程唤醒后可继续执行）。 -### +### 🎯 Java 对象是不是都创建在堆上的呢？ +> JVM 的一些高级优化技术，例如逃逸分析（Escape Analysis），可以使对象在栈上分配内存，而不是堆上。逃逸分析可以判断对象的作用域，如果确定对象不会逃逸出方法（即对象仅在方法内部使用），JVM 可以选择将对象分配在栈上。这种优化减少了垃圾回收的负担，并提高了性能。 +> +> 通过[逃逸分析](https://en.wikipedia.org/wiki/Escape_analysis)，JVM 会在栈上分配那些不会逃逸的对象，这在理论上是可行的，但是取决于 JVM 设计者的选择。 +在 Java 中，并不是所有对象都严格创建在堆上，尽管大部分情况下确实如此。具体来说： -### 说说你知道的几种主要的JVM参数 +1. **普通对象：通常创建在堆上** + - Java 中通过 `new` 关键字创建的对象（比如 `new MyClass()`），以及通过反射、序列化等机制创建的对象，默认分配在堆上。 + + - 堆是 JVM 中用来存储对象实例和数组的区域，所有线程共享。 + +2. 栈上分配对象（逃逸分析 & 标量替换优化） + - 在某些情况下，JVM 可以通过优化技术将原本应该在堆上分配的对象转移到栈上分配。这种优化是通过**逃逸分析（Escape Analysis）**实现的。 + - 如果 JVM 确定某个对象不会在当前方法之外被访问（不会逃逸当前线程），那么它可能会将该对象分配在栈上。 + - 如果对象分配在栈上，当方法执行完毕，栈帧出栈时，对象会自动销毁，不需要垃圾回收。 -**思路：** 可以说一下堆栈配置相关的，垃圾收集器相关的，还有一下辅助信息相关的。 +3. **标量替换** + - 如果逃逸分析进一步确定一个对象的成员变量可以直接用基本数据类型代替，则 JVM 可能会将对象“拆分”为一组标量变量，而根本不创建对象。这种情况下，对象实际上并没有被显式分配在堆或栈上。 -**我的答案：** +4. **方法区中的特殊对象** -#### 1）堆栈配置相关 + - **类对象**：每个类的 `Class` 对象是存储在**方法区**中的，用于反射和类元数据。 -``` -java -Xmx3550m -Xms3550m -Xmn2g -Xss128k --XX:MaxPermSize=16m -XX:NewRatio=4 -XX:SurvivorRatio=4 -XX:MaxTenuringThreshold=0 -复制代码 -``` + - **字符串池**：字符串字面量（如 `"hello"`）存储在堆外的字符串常量池（Java 7 开始字符串常量池在堆中）。 -**-Xmx3550m：** 最大堆大小为3550m。 +5. **直接内存中的对象** + - 通过 `ByteBuffer.allocateDirect()` 分配的直接内存区域（Direct Memory）不在堆上，而是使用操作系统的内存。 -**-Xms3550m：** 设置初始堆大小为3550m。 +**结论** -**-Xmn2g：** 设置年轻代大小为2g。 +虽然大部分 Java 对象创建在堆上，但由于 JVM 的优化（如逃逸分析、标量替换）和一些特定机制（如字符串常量池、直接内存），并非所有对象都严格创建在堆上。是否在堆上分配，具体取决于对象的生命周期、作用域以及 JVM 的运行时优化策略。 -**-Xss128k：** 每个线程的堆栈大小为128k。 -**-XX:MaxPermSize：** 设置持久代大小为16m -**-XX:NewRatio=4:** 设置年轻代（包括Eden和两个Survivor区）与年老代的比值（除去持久代）。 +### 🎯 Java new 一个对象的过程发生了什么？ -**-XX:SurvivorRatio=4：** 设置年轻代中Eden区与Survivor区的大小比值。设置为4，则两个Survivor区与一个Eden区的比值为2:4，一个Survivor区占整个年轻代的1/6 +Java 在 new 一个对象的时候，会先查看对象所属的类有没有被加载到内存，如果没有的话，就会先通过类的全限定名来加载。加载并初始化类完成后，再进行对象的创建工作。 -**-XX:MaxTenuringThreshold=0：** 设置垃圾最大年龄。如果设置为0的话，则年轻代对象不经过Survivor区，直接进入年老代。 +我们先假设是第一次使用该类，这样的话 new 一个对象就可以分为两个过程：**加载并初始化类和创建对象** -#### 2）垃圾收集器相关 +加载过程就是 ClassLoader 那一套：加载-验证-准备-解析-初始化 -``` --XX:+UseParallelGC --XX:ParallelGCThreads=20 --XX:+UseConcMarkSweepGC --XX:CMSFullGCsBeforeCompaction=5 --XX:+UseCMSCompactAtFullCollection： -复制代码 -``` +**一、类加载与初始化（首次使用类时触发）** -**-XX:+UseParallelGC：** 选择垃圾收集器为并行收集器。 +1. **类加载检查** -**-XX:ParallelGCThreads=20：** 配置并行收集器的线程数 + - **触发条件**：首次使用类（如 `new`、反射调用等），检查是否已加载类元数据。 -**-XX:+UseConcMarkSweepGC：** 设置年老代为并发收集。 + - 加载流程： + - **加载（Loading）**：通过类加载器（如 `AppClassLoader`）从磁盘、网络等来源读取 `.class` 字节流。 + - **验证（Verification）**：检查字节码合法性（如魔数 `0xCAFEBABE`）。 + - **准备（Preparation）**：为静态变量分配内存并赋零值（如 `static int` 初始化为 0）。 + - **解析（Resolution）**：将符号引用转为直接引用（如方法地址）。 + - **初始化（Initialization）**：执行 `()` 方法，合并静态代码块和静态变量赋值。 -**-XX:CMSFullGCsBeforeCompaction**：由于并发收集器不对内存空间进行压缩、整理，所以运行一段时间以后会产生“碎片”，使得运行效率降低。此值设置运行多少次GC以后对内存空间进行压缩、整理。 +2. **类加载的懒加载特性** -**-XX:+UseCMSCompactAtFullCollection：** 打开对年老代的压缩。可能会影响性能，但是可以消除碎片 + - **延迟加载**：类仅在首次主动使用时加载，避免启动时加载所有类。 -#### 3）辅助信息相关 + ```java + public class MyClass { + static { System.out.println("类已初始化"); } + } + public class Test { + public static void main(String[] args) { + // 首次 new 时触发类加载和初始化 + MyClass obj = new MyClass(); // 输出："类已初始化" + } + } + ``` + -``` --XX:+PrintGC --XX:+PrintGCDetails -复制代码 -``` +**二、对象实例化阶段** -**-XX:+PrintGC 输出形式:** +1. **分配堆内存** -[GC 118250K->113543K(130112K), 0.0094143 secs] [Full GC 121376K->10414K(130112K), 0.0650971 secs] + - 内存分配方式： + - **指针碰撞（Bump the Pointer）**：适用于规整内存（如 `Serial` 收集器），通过移动指针分配连续内存。 + - **空闲列表（Free List）**：适用于碎片化内存（如 `CMS` 收集器），维护可用内存块列表。 -**-XX:+PrintGCDetails 输出形式:** + - 线程安全机制： + - **CAS + 重试**：通过原子操作避免多线程竞争。 + - **TLAB（Thread Local Allocation Buffer）**：为每个线程预分配私有内存区域，减少锁竞争。 -[GC [DefNew: 8614K->781K(9088K), 0.0123035 secs] 118250K->113543K(130112K), 0.0124633 secs] [GC [DefNew: 8614K->8614K(9088K), 0.0000665 secs][Tenured: 112761K->10414K(121024K), 0.0433488 secs] 121376K->10414K(130112K), 0.0436268 secs +2. **初始化零值** + - JVM 将对象内存区域初始化为零值： + - 基本类型：`int` → `0`，`boolean` → `false`。 + - 引用类型：初始化为 `null`。 + - **意义**：确保未显式赋值的字段可直接使用。 -### 怎么打出线程栈信息。 +3. **设置对象头（Object Header）** -**思路：** 可以说一下jps，top ，jstack这几个命令，再配合一次排查线上问题进行解答。 + - **Mark Word**：存储哈希码、GC 分代年龄、锁状态（如偏向锁标记）。 -**我的答案：** + - **类型指针（Class Pointer）**：指向方法区中的类元数据（`Class` 对象），表示该对象是哪个类的实例。 -- 输入jps，获得进程号。 -- top -Hp pid 获取本进程中所有线程的CPU耗时性能 -- jstack pid命令查看当前java进程的堆栈状态 -- 或者 jstack -l > /tmp/output.txt 把堆栈信息打到一个txt文件。 -- 可以使用fastthread 堆栈定位，[fastthread.io/](http://fastthread.io/) + - **数组长度**（仅数组对象）：记录数组长度。 +4. **执行 `()` 方法** + - 初始化顺序： + 1. **父类构造方法**：递归调用父类构造方法（隐式调用 `super()`）。 + 2. **实例变量初始化**：按代码顺序执行字段赋值和实例代码块。 + 3. **构造方法体**：执行用户编写的构造逻辑。 + ```java + public class Parent { + public Parent() { System.out.println("Parent构造方法"); } + } + public class Child extends Parent { + private int x = initX(); // 实例变量初始化 + { System.out.println("Child实例代码块"); } // 实例代码块 + public Child() { + System.out.println("Child构造方法"); + } + private int initX() { + System.out.println("初始化x"); + return 10; + } + } + // 输出顺序： + // Parent构造方法 → 初始化x → Child实例代码块 → Child构造方法 + ``` +5. **返回对象引用**：构造完成后，栈帧中引用变量（如 `obj`）指向堆内存地址。 +### 🎯 请谈谈你对 OOM 的认识 | 哪些区域可能发生 OutOfMemoryError？ +OOM 如果通俗点儿说，就是 JVM 内存不够用了，javadoc 中对[OutOfMemoryError](https://docs.oracle.com/javase/9/docs/api/java/lang/OutOfMemoryError.html)的解释是，没有空闲内存，并且垃圾收集器也无法提供更多内存。 +这里面隐含着一层意思是，在抛出 OutOfMemoryError 之前，通常垃圾收集器会被触发，尽其所能去清理出空间 +当然，也不是在任何情况下垃圾收集器都会被触发的，比如，我们去分配一个超大对象，类似一个超大数组超过堆的最大值，JVM 可以判断出垃圾收集并不能解决这个问题，所以直接抛出 OutOfMemoryError。 +从数据区的角度，除了程序计数器，其他区域都有可能会因为可能的空间不足发生 OutOfMemoryError，简单总结如下： +- 堆内存不足是最常见的 OOM 原因之一，抛出的错误信息是“java.lang.OutOfMemoryError:Java heap space”，原因可能千奇百怪，例如，可能存在内存泄漏问题；也很有可能就是堆的大小不合理，比如我们要处理比较可观的数据量，但是没有显式指定 JVM 堆大小或者指定数值偏小；或者出现 JVM 处理引用不及时，导致堆积起来，内存无法释放等。 +- 而对于 Java 虚拟机栈和本地方法栈，这里要稍微复杂一点。如果我们写一段程序不断的进行递归调用，而且没有退出条件，就会导致不断地进行压栈。类似这种情况，JVM 实际会抛出 StackOverFlowError；当然，如果 JVM 试图去扩展栈空间的的时候失败，则会抛出 OutOfMemoryError。 +- 对于老版本的 Oracle JDK，因为永久代的大小是有限的，并且 JVM 对永久代垃圾回收（如，常量池回收、卸载不再需要的类型）非常不积极，所以当我们不断添加新类型的时候，永久代出现 OutOfMemoryError 也非常多见，尤其是在运行时存在大量动态类型生成的场合；类似 Intern 字符串缓存占用太多空间，也会导致 OOM 问题。对应的异常信息，会标记出来和永久代相关：“java.lang.OutOfMemoryError: PermGen space”。 +- 随着元数据区的引入，方法区内存已经不再那么窘迫，所以相应的 OOM 有所改观，出现 OOM，异常信息则变成了：“java.lang.OutOfMemoryError: Metaspace”。 +- 直接内存不足，也会导致 OOM -![](https://tva1.sinaimg.cn/large/00831rSTly1gdeadup0v1j30xk0lgncn.jpg) +### 🎯 内存泄露和内存溢出的区别？ -![](https://tva1.sinaimg.cn/large/00831rSTly1gdeambp5abj31ew0pm16q.jpg) +- 内存泄露 memory leak，是指程序在申请内存后，无法释放已申请的内存空间，一次内存泄露危害可以忽略，但内存泄露堆积后果很严重，无论多少内存，迟早会被占光。 +- 内存溢出 out of memory，是指程序在申请内存时，没有足够的内存空间供其使用，出现 out of memory；比如申请了一个 integer,但给它存了 long 才能存下的数，那就是内存溢出。 + memory leak 最终会导致 out of memory！ +内存泄漏是内存溢出的常见诱因，但内存溢出也可能由非泄漏因素（如数据量过大）直接引发 ------- +### 🎯 内存泄漏时，如何定位问题代码? + +在 Java 中，内存泄漏通常指的是对象被不再需要但仍被引用，从而无法被垃圾回收器回收的情况。要定位和修复内存泄漏，通常需要以下几个步骤： +1. **确认内存泄漏** + - **表现**：应用程序在长时间运行后会出现性能下降，内存占用不断增加，GC（垃圾回收）频繁但无法回收足够的内存。 + - 工具：可以使用以下工具来确认内存泄漏： + - **JVM日志**：启用 GC 日志 (`-Xlog:gc*`)，检查垃圾回收的情况，尤其是 Full GC 频繁且回收的内存不多时。 + - **Heap Dump**：通过 `-XX:+HeapDumpOnOutOfMemoryError` 生成堆转储文件，或者使用 `jmap` 工具手动生成堆转储。 +2. **分析堆转储（Heap Dump）** + - **Heap Dump 分析工具**： + - **Eclipse MAT** (Memory Analyzer Tool)：可以加载堆转储文件，生成泄漏分析报告，显示哪些对象占用了最多内存，以及哪些对象没有被回收。 + - **VisualVM**：这是一个 Java 性能分析工具，支持堆转储的加载与分析，并能够查看对象引用链，帮助定位导致泄漏的对象。 + - **YourKit**、**JProfiler**：这些是商业性能分析工具，功能更全面，支持实时监控、堆分析、内存泄漏检测等。 -## 调优 + - **分析堆转储时的关键点**： + - 查找长时间存在的对象（例如常驻内存的单例或缓存），查看其引用链。 + - 找到不再需要但仍被引用的对象，分析其引用路径。 + - 查看某些特定类实例的数量是否异常增长。 -## 2.你说你做过 JVM 调优和参数配置，请问如何盘点查看 JVM 系统默认值 +3. **使用分析工具进行动态分析** -### JVM参数类型 + - **VisualVM**：通过 `VisualVM` 可以在运行时监控堆使用情况，分析类的实例数量，查看垃圾回收日志，甚至进行堆转储。 -- 标配参数 + - **JProfiler** 或 **YourKit**：这些工具不仅提供堆分析，还能够在运行时实时显示内存分配、对象实例化情况及引用关系，帮助快速定位内存泄漏。 - - -version (java -version) - - -help (java -help) - - java -showversion +4. **分析代码和日志** -- X 参数（了解） + - 检查哪些对象没有正确地释放或清理，例如： + - **缓存/集合**：使用了 `HashMap`、`ArrayList` 等数据结构，但没有及时清理过期数据，导致大量对象无法回收。 + - **监听器**：事件监听器没有取消注册，导致对象无法被垃圾回收。 + - **ThreadLocal**：如果没有正确清理 `ThreadLocal` 变量，也可能导致内存泄漏，特别是在线程池中。 + - **数据库连接、文件句柄、网络连接等资源**：没有正确关闭，导致资源泄漏。 - - -Xint 解释执行 + - **日志调试**：通过启用调试日志、使用日志记录对象的创建和释放情况，帮助找出未正确释放的资源。 - - -Xcomp 第一次使用就编译成本地代码 +5. **GC 根分析（GC Roots）** - - -Xmixed 混合模式 + - 通过分析对象的 GC 根，可以找到哪些对象被引用着但不再被使用。 - ![](https://tva1.sinaimg.cn/large/00831rSTly1gdeb84yh71j30yq0j6akl.jpg) + - **引用链分析**：可以借助工具查看哪些对象通过引用链未被回收，特别是常见的 "单例模式"、"静态变量" 等可能会引起问题。 -- xx参数 +6. **避免常见内存泄漏问题** - - Boolean 类型 + - **缓存问题**：如果使用缓存，如 `HashMap`、`WeakHashMap` 等，要确保缓存数据定期清理，避免无限制增加内存占用。 - - 公式： -xx:+ 或者 - 某个属性值（+表示开启，- 表示关闭） + - **Listener / Observer 相关问题**：注册的事件监听器或观察者未取消注册，导致对象无法被回收。 - - Case + - **ThreadLocal**：在多线程环境下，使用完 `ThreadLocal` 后，要调用 `remove()` 方法清理线程本地变量。 - - 是否打印GC收集细节 + - **数据库连接池**：确保数据库连接池设置合理，避免因连接池过度增长导致内存泄漏。 - - -XX:+PrintGCDetails - - -XX:- PrintGCDetails +7. **压力测试** + - 使用 **JMeter** 或 **Gatling** 等压力测试工具模拟高并发场景，观察应用程序在长时间高负载下的内存使用情况。通过监控堆内存、GC 日志等，帮助捕捉内存泄漏的迹象。 - ![](https://tva1.sinaimg.cn/large/00831rSTly1gdebpozfgwj315o0sgtcy.jpg) +8. **代码审查** + - 定期进行代码审查，特别是对资源管理（如数据库连接、IO流等）和大对象（如大型集合、缓存）的处理，确保资源在使用完后正确释放。 - 添加如下参数后，重新查看，发现是 + 号了 +9. **常见内存泄漏示例** - ![](https://tva1.sinaimg.cn/large/00831rSTly1gdebrx25moj31170u042c.jpg) + - 缓存泄漏： - - 是否使用串行垃圾回收器 + ```java + // 缓存对象没有过期清理 + private static Map cache = new HashMap<>(); + public void addToCache(String key, Object value) { + cache.put(key, value); // 长时间占用内存 + } + ``` - - -XX:-UseSerialGC - - -XX:+UseSerialGC + - 监听器未移除： - - KV 设值类型 + ```java + public void addListener(MyListener listener) { + eventSource.addListener(listener); // 忘记移除 + } + ``` - - 公式 -XX:属性key=属性 value +定位 Java 中的内存泄漏需要通过一系列工具和方法进行分析，从确认内存泄漏开始，逐步使用堆转储、GC 日志分析、动态监控工具等手段，最终找出泄漏的根源，并根据分析结果修复代码中的内存管理问题。 - - Case: - - -XX:MetaspaceSize=128m - - -xx:MaxTenuringThreshold=15 +### 🎯 什么情况下会发生栈内存溢出? - - 我们常见的 -Xms和 -Xmx 也属于 KV 设值类型 +- 栈是线程私有的，他的生命周期与线程相同，每个方法在执行的时候都会创建一个栈帧，用来存储局部变量表，操作数栈，动态链接，方法出口等信息。局部变量表又包含基本数据类型，对象引用类型 +- 如果线程请求的栈深度大于虚拟机所允许的最大深度，将抛出StackOverflowError异常，方法递归调用产生这种结果。 +- 如果 Java 虚拟机栈可以动态扩展，并且扩展的动作已经尝试过，但是无法申请到足够的内存去完成扩展，或者在新建立线程的时候没有足够的内存去创建对应的虚拟机栈，那么 Java 虚拟机将抛出一个 OutOfMemory 异常。(线程启动过多) +- 参数 -Xss 去调整 JVM 栈的大小 - - -Xms 等价于 -XX:InitialHeapSize - - -Xmx 等价于 -XX:MaxHeapSize - ![](https://tva1.sinaimg.cn/large/00831rSTly1gdecj9d7z3j310202qdgb.jpg) - - jinfo 举例，如何查看当前运行程序的配置 +### 🎯 如何监控和诊断 JVM 堆内和堆外内存使用？ - - jps -l - - jinfo -flag [配置项] 进程编号 - - jinfo **-flags** 1981(打印所有) - - jinfo -flag PrintGCDetails 1981 - - jinfo -flag MetaspaceSize 2044 +了解 JVM 内存的方法有很多，具体能力范围也有区别，简单总结如下： -这些都是命令级别的查看，我们如何在程序运行中查看 +- 可以使用综合性的图形化工具，如 JConsole、VisualVM（注意，从 Oracle JDK 9 开始，VisualVM 已经不再包含在 JDK 安装包中）等。这些工具具体使用起来相对比较直观，直接连接到 Java 进程，然后就可以在图形化界面里掌握内存使用情况。 + +以 JConsole 为例，其内存页面可以显示常见的**堆内存**和**各种堆外部分**使用状态。 + +- 也可以使用命令行工具进行运行时查询，如 jstat 和 jmap 等工具都提供了一些选项，可以查看堆、方法区等使用数据。 +- 或者，也可以使用 jmap 等提供的命令，生成堆转储（Heap Dump）文件，然后利用 jhat 或 Eclipse MAT 等堆转储分析工具进行详细分析。 +- 如果你使用的是 Tomcat、Weblogic 等 Java EE 服务器，这些服务器同样提供了内存管理相关的功能。 +- 另外，从某种程度上来说，GC 日志等输出，同样包含着丰富的信息。 + +这里有一个相对特殊的部分，就是堆外内存中的直接内存，前面的工具基本不适用，可以使用 JDK 自带的 Native Memory Tracking（NMT）特性，它会从 JVM 本地内存分配的角度进行解读。 + + + +### 🎯 String 字符串存放位置？ + +在 Java 中，`String` 类型的特殊之处在于它可能会在两种位置存放： + +1. **字符串常量池（String Pool）**： + - 在 Java 中，所有的字符串常量（即直接通过双引号声明的字符串字面量）都会存放在字符串常量池中。字符串常量池是位于堆内存中的一块特殊的存储区域。 + - 当程序中多次使用相同的字符串字面量时，Java虚拟机（JVM）会首先在字符串常量池中查找是否存在该字符串。如果存在，就复用已有的字符串引用，而不是重新创建一个新的字符串对象，这样可以节省内存空间。 +2. **堆内存（Heap）**： + - 通过`new`关键字创建的字符串对象，或者通过字符串连接操作符（`+`）动态生成的字符串，会存放在Java堆内存中。 + - 堆内存是Java中存放对象实例的地方，包括通过构造函数创建的`String`对象。 + +以下是两种情况的示例： + +**字符串常量池示例**： ```java - long totalMemory = Runtime.getRuntime().totalMemory(); - long maxMemory = Runtime.getRuntime().maxMemory(); +String str1 = "Hello, World!"; +String str2 = "Hello, World!"; + +// str1和str2可能指向字符串常量池中的同一个对象 +``` - System.out.println("total_memory(-xms)="+totalMemory+"字节，" +(totalMemory/(double)1024/1024)+"MB"); - System.out.println("max_memory(-xmx)="+maxMemory+"字节，" +(maxMemory/(double)1024/1024)+"MB"); +**堆内存示例**： -} +```java +String str1 = new String("Hello, World!"); +String str2 = new String("Hello, World!"); + +// str1和str2指向堆内存中的不同对象，即使字符串内容相同 ``` -### 盘点家底查看JVM默认值 +**注意**： -- -XX:+PrintFlagsInitial +- 在 Java 7 及以后的版本中，字符串常量池被移动到了堆内存中，而不是方法区。这是为了提高 JVM 的性能和减少内存碎片。 +- 字符串常量池和堆内存中的字符串对象可能通过内部的`offset`和`count`字段来共享char数组，这是通过Java的字符串共享机制实现的。 +- 当使用`String.intern()`方法时，可以将一个字符串对象放入字符串常量池中。如果字符串已经存在于池中的，`intern()`方法返回的是池中字符串的引用。 - - 主要查看初始默认值 +```java +String str1 = new String("Hello, World!"); +String str2 = str1.intern(); - - java -XX:+PrintFlagsInitial +// str1和str2现在指向字符串常量池中的同一个对象 +``` - - java -XX:+PrintFlagsInitial -version - - ![](https://tva1.sinaimg.cn/large/00831rSTly1gdee0ndg33j31ci0m6k5w.jpg) - **等号前有冒号** := 说明 jvm 参数有人为修改过或者 JVM加载修改 +### 🎯 深拷贝和浅拷贝 - false 说明是Boolean 类型 参数，数字说明是 KV 类型参数 +> 浅拷贝只复制对象的基本类型字段和引用地址，引用对象共享；而深拷贝会递归复制引用对象，完全独立。浅拷贝性能好但容易产生引用共享问题，深拷贝内存开销大但安全。实际开发中，如果对象比较简单可以用浅拷贝；如果涉及复杂引用，通常会实现深拷贝，比如序列化的方式。 -- -XX:+PrintFlagsFinal +简单来讲就是复制、克隆。 - - 主要查看修改更新 - - java -XX:+PrintFlagsFinal - - java -XX:+PrintFlagsFinal -version - - 运行java命令的同时打印出参数 java -XX:+PrintFlagsFinal -XX:MetaspaceSize=512m Hello.java +**浅拷贝（Shallow Copy）** -- -XX:+PrintCommondLineFlags +- **定义**：创建一个新的对象，但是**只复制基本数据类型的值**，而引用类型只复制引用地址，不复制引用对象本身。 - - 打印命令行参数 - - java -XX:+PrintCommondLineFlags -version - - 可以方便的看到垃圾回收器 - - ![](https://tva1.sinaimg.cn/large/007S8ZIlly1gehf1e54soj31e006qjz6.jpg) +- **特点**： -## 3. 你平时工作用过的 JVM 常用基本配置参数有哪些？ + - 对象本身是新的，但里面的引用变量和原对象指向的是同一块内存。 + - 修改引用对象会互相影响。 -![](https://tva1.sinaimg.cn/large/00831rSTly1gdee0iss88j31eu0n6aqi.jpg) + ```java + class Person implements Cloneable { + String name; + int age; + Address addr; + public Object clone() throws CloneNotSupportedException { + return super.clone(); // 默认是浅拷贝 + } + } + ``` + 这里 `addr` 是浅拷贝的，两个对象共享同一个地址对象。 +**深拷贝（Deep Copy）** -- -Xms +- **定义**：不仅复制对象本身，还会**复制对象中的所有引用对象**，实现真正的独立。 +- **特点**： + - 拷贝后的对象与原对象完全隔离。 + - 修改其中一个对象的引用属性，不会影响另一个对象。 +- **实现方式**： + 1. 手动重写 `clone()` 方法，逐层拷贝引用对象。 + 2. 通过 **序列化/反序列化** 实现。 + 3. 借助三方工具（如 Apache Commons Lang 的 `SerializationUtils.clone()`）。 - - 初始大小内存，默认为物理内存1/64 - - 等价于 -XX:InitialHeapSize -- -Xmx - - 最大分配内存，默认为物理内存的1/4 - - 等价于 -XX:MaxHeapSize +### 🎯 总体来看，JVM 把内存划分为“栈(stack)”与“堆(heap)”两大类，为何要这样设计？ -- -Xss +个人理解，程序运行时，内存中的信息大致分为两类： - - 设置单个线程的大小，一般默认为 512k~1024k - - 等价于 -XX:ThreadStackSize - - 如果通过 `jinfo ThreadStackSize 线程 ID` 查看会显示为 0，指的是默认出厂设置 +跟程序执行逻辑相关的指令数据，这类数据通常不大，而且生命周期短； -- -Xmn +跟对象实例相关的数据，这类数据可能会很大，而且可以被多个线程长时间内反复共用，比如字符串常量、缓存对象这类。 - - 设置年轻代大小（一般不设置） +将这两类特点不同的数据分开管理，体现了软件设计上“模块隔离”的思想。好比我们通常会把后端 service 与前端 website 解耦类似，也更便于内存管理。 -- -XX:MetaspaceSize +- **性能和效率**：栈内存的分配和释放速度快，非常适合方法调用和局部变量的管理，而堆内存虽然管理复杂，但适合管理生命周期长、需要动态分配的对象。通过将内存分为栈和堆，JVM能够在不同场景下优化内存管理。 - - 设置元空间大小。元空间的本质和永久代类似，都是对 JMM 规范中方法区的实现。不过元空间与永久代最大的区别是，元空间并不在虚拟机中，而是使用本地内存。因此，默认情况下，元空间的大小仅受本地内存限制 - - 但是元空间默认也很小，频繁 new 对象，也会 OOM - - -Xms10m -Xmx10m -XX:MetaspaceSize=1024m -XX:+PrintFlagsFinal +- **安全性**：栈内存由每个线程独立拥有，保证了线程间的数据隔离，避免了数据竞争和安全问题。堆内存虽然是共享的，但通过同步机制可以确保线程安全。 -- -XX:+PrintGCDetails +- **灵活性**：堆内存允许动态分配和释放，适应了Java语言中大量使用对象的特点，同时垃圾回收机制能够自动管理内存，减少了开发者手动管理内存的负担。 - - 输出详细的 GC 收集日志信息 +------ - - 测试时候，可以将参数调到最小， - `-Xms10m -Xmx10m -XX:+PrintGCDetails` - 定义一个大对象，撑爆堆内存， +## 三、垃圾回收机制 - ```java - public static void main(String[] args) throws InterruptedException { - System.out.println("==hello gc==="); - - //Thread.sleep(Integer.MAX_VALUE); - - //-Xms10m -Xmx10m -XX:PrintGCDetails - - byte[] bytes = new byte[11 * 1024 * 1024]; - - }![](https://tva1.sinaimg.cn/large/007S8ZIlly1gehkvas3vzj31a90u0n7t.jpg) - ``` +### 🎯 谈下 Java 的内存管理和垃圾回收 - - GC![](https://tva1.sinaimg.cn/large/00831rSTly1gdefrf0dfqj31fs0honjk.jpg) +内存管理就是内存的生命周期管理，包括内存的申请、压缩、回收等操作。 Java 的内存管理就是 GC，JVM 的 GC 模块不仅管理内存的回收，也负责内存的分配和压缩整理。 - - Full GC![](https://tva1.sinaimg.cn/large/00831rSTly1gdefrc3lmbj31hy0gk7of.jpg) +Java 程序的指令都运行在 JVM 上，而且我们的程序代码并不需要去分配内存和释放内存（例如 C/C++ 里需要使用的 malloc/free），那么这些操作自然是由 JVM 帮我们搞定的。 - ![](https://tva1.sinaimg.cn/large/00831rSTly1gdefr8tvx0j31h60m41eq.jpg) +JVM 在我们创建 Java 对象的时候去分配新内存，并使用 GC 算法，根据对象的存活时间，在对象不使用之后，自动执行对象的内存回收操作。 -- -XX:SurvivorRatio - - 设置新生代中 eden 和S0/S1空间的比例 - - 默认 -XX:SurvivorRatio=8,Eden:S0:S1=8:1:1 - - SurvivorRatio值就是设置 Eden 区的比例占多少，S0/S1相同，如果设置 -XX:SurvivorRatio=2，那Eden:S0:S1=2:1:1 -- -XX:NewRatio +### 🎯 简述垃圾回收机制 - - 配置年轻代和老年代在堆结构的比例 - - 默认 -XX:NewRatio=2，新生代 1，老年代 2，年轻代占整个堆的 1/3 - - NewRatio值就是设置老年代的占比，如果设置-XX:NewRatio=4，那就表示新生代占 1，老年代占 4，年轻代占整个堆的 1/5 +> Java 的垃圾回收机制通过 **可达性分析**来判断对象是否存活，常用的算法有标记-清除、标记-整理、复制算法，结合分代收集提升效率。常见的收集器有 Serial、Parallel、CMS 和 G1。GC 的核心目标是 **在尽量减少停顿的情况下回收无用对象，避免 OOM**。 -- -XX:MaxTenuringThreshold +程序在运行的时候，为了提高性能，大部分数据都是会加载到内存中进行运算的，有些数据是需要常驻内存中的，但是有些数据，用过之后便不会再需要了，我们称这部分数据为垃圾数据。 - - 设置垃圾的最大年龄（java8 固定设置最大 15） - - ![](https://tva1.sinaimg.cn/large/00831rSTly1gdefr4xeq1j31g80lek6e.jpg) +为了防止内存被使用完，我们需要将这些垃圾数据进行回收，即需要将这部分内存空间进行释放。不同于 C++ 需要自行释放内存的机制，Java 虚拟机（JVM）提供了一种自动回收内存的机制，它是低优先级的，在正常情况下是不会执行的，只有在虚拟机空闲或者当前堆内存不足时，才会触发执行，扫描那些没有被任何引用的对象， 并将它们添加到要回收的集合中，进行回收。 -参数不懂，推荐直接去看官网， +**1. 为什么需要 GC？** -https://docs.oracle.com/javacomponents/jrockit-hotspot/migration-guide/cloptions.htm#JRHMG127 +- Java 使用 **自动内存管理**，不需要手动 `free`。 +- GC 的目标是 **回收不再使用的对象**，避免内存泄漏和 OOM。 +**2. 如何判断对象是否可回收？** +1. **引用计数法**（缺点：循环引用无法解决，Java 基本不用）。 +2. **可达性分析（Reachability Analysis）**（Java 采用）： + - 以 **GC Roots**（如虚拟机栈中的引用、静态变量、JNI引用）为起点， + - 通过引用链遍历能到达的对象都是存活的，无法到达的才会被回收。 -https://docs.oracle.com/javase/8/docs/technotes/tools/unix/java.html#BGBCIEFC +**3. 常见的垃圾回收算法** -https://docs.oracle.com/javase/8/ +- **标记-清除（Mark-Sweep）**：标记可达对象，清除不可达对象；但会产生碎片。 +- **标记-整理（Mark-Compact）**：标记后把存活对象压缩到一边，避免碎片。 +- **复制算法（Copying）**：把对象分两块内存，每次只用一块，存活对象复制到另一块，效率高；常用于年轻代。 +- **分代收集（Generational GC）**：根据对象存活时间划分 **年轻代 / 老年代**，采用不同算法： + - 年轻代（短命对象多）：复制算法 + - 老年代（长命对象多）：标记-清除 / 标记-整理 -Java SE Tools Reference for UNIX](https://docs.oracle.com/javase/8/docs/technotes/tools/unix/index.html) +**4. 常见垃圾收集器** +- **Serial**：单线程，适合小内存。 +- **Parallel（吞吐量优先）**：多线程，适合后台任务。 +- **CMS（并发低停顿）**：并发回收，降低 STW（Stop The World）。 +- **G1（面向大堆内存）**：区域化内存管理，预测停顿时间。 +### 🎯 JVM 垃圾回收的时候如何确定垃圾？ +自动垃圾收集的前提是清楚哪些内存可以被释放，内存中不再使用的空间就是垃圾 +对于对象实例收集，主要是两种基本算法 -### 4. 强引用、软引用、弱引用、虚引用分别是什么？ +- 引用计数法：引用计数算法，顾名思义，就是为对象添加一个引用计数，用于记录对象被引用的情况，如果计数为 0，即表示对象可回收。有循环引用问题 + ![Reference Counting](https://img.starfish.ink/jvm/Reference-Counting.png) +- 可达性分析：将对象及其引用关系看作一个图，选定活动的对象作为 GC Roots，然后跟踪引用链条，如果一个对象和 GC Roots 之间不可达，也就是不存在引用链条，那么即可认为是可回收对象 + ![Tracing Garbage Collectors](https://img.starfish.ink/jvm/Tracing-Garbage-Collectors.png) -## 5. 请谈谈你对 OOM 的认识 -- java.lang.StackOverflowError +### 🎯 引用计数法的缺点，除了循环引用，说一到两个? - - ``` - public class StackOverflowErrorDemo { - - public static void main(String[] args) { - stackoverflowError(); - } - - private static void stackoverflowError() { - stackoverflowError(); - } - } - ``` +1. **额外的性能开销**：每个对象都需要维护一个引用计数器，每次对象被引用或释放时，都要对计数器进行增减操作。这会带来频繁的内存读写开销，尤其在高频次对象交互的场景（如复杂数据结构操作、多线程环境）中，可能成为性能瓶颈。例如，在一个链表遍历过程中，每个节点的引用计数都会被频繁修改，累积的开销会影响整体效率。 +2. **计数器溢出风险**：对象引用计数器本身是需要空间的，而计数器要占用多少位也是一个问题 +3. 一个致命缺陷是循环引用，就是， objA引用了objB，objB也引用了objA，但是除此之外，再没有其他的地方引用这两个对象了，这两个对象的引用计数就都是1。这种情况下，这两个对象是不能被回收的。 -- java.lang.OutOfMemoryError: Java heap space - - new个大对象,就会出现 -- java.lang.OutOfMemoryError: GC overhead limit exceeded (GC上头，哈哈) +### 🎯 你知道什么是 GC Roots 吗？GC Roots 如何确定，哪些对象可以作为 GC Roots? - - ![](https://tva1.sinaimg.cn/large/007S8ZIlly1gehmrz0dvaj311w0muk0e.jpg) +为了解决引用计数法的循环引用问题，Java 使用了可达性分析的方法 - - ```java - public class StackOverflowErrorDemo { - - public static void main(String[] args) { - stackoverflowError(); - } - - private static void stackoverflowError() { - stackoverflowError(); - } - } - ``` +可达性算法的原理是以一系列叫做 **GC Root** 的对象为起点出发，引出它们指向的下一个节点，再以下个节点为起点，引出此节点指向的下一个结点。。。（这样通过 GC Root 串成的一条线就叫引用链），直到所有的结点都遍历完毕，如果相关对象不在任意一个以 **GC Root** 为起点的引用链中，则这些对象会被判断为「垃圾」,会被 GC 回收。 + +**哪些对象可以作为 GC Root 呢，有以下几类** + +- 虚拟机栈（栈帧中的本地变量表）中引用的对象 +- 方法区中类静态属性引用的对象 +- 方法区中常量引用的对象 +- 本地方法栈中 JNI（即一般说的 Native 方法）引用的对象 +- Java 虚拟机内部的引用，如基本数据类型对应的 Class 对象，一些常驻的异常对象 +- 所有被同步锁（synchronized 关键字）持有的对象 +- 反映 Java 虚拟机内部情况的 JMXBean、JVMTI 中注册的回调、本地代码缓存 + +在实际运行过程中，JVM 会通过维护一张全局的引用表来管理 GC Roots。这个表包含了所有活动线程、静态变量、常量引用和 JNI 全局引用等。垃圾回收器在进行标记阶段时，会遍历这个全局引用表，标记从 GC Roots 开始的所有可达对象。 + + + +### 🎯 哪些内存区域需要 GC ? + +Thread 独享的区域：PC Regiester、JVM Stack、Native Method Stack，其生命周期都与线程相同（即与线程共生死），所以无需 GC。而线程共享的 Heap 区、Method Area 则是 GC 关注的重点对象 + + + +### 🎯 对象的死亡过程? + +在 Java 中，对象被垃圾回收器（Garbage Collector, GC）判定为“死亡”并回收内存的过程，涉及 **两次标记** 和潜在的 **自救机会**（通过 `finalize()` 方法） -- java.lang.OutOfMemoryError: Direct buffer memory +**一、第一次标记：可达性分析** - - ![](https://tva1.sinaimg.cn/large/007S8ZIlly1gehn18eix6j31a00m2wup.jpg) - - ![](https://tva1.sinaimg.cn/large/007S8ZIlly1gehn52fphnj31as0lidyh.jpg) +1. **触发条件**：当 JVM 开始垃圾回收时，首先通过 **可达性分析算法（Reachability Analysis）** 判断对象是否存活。 -- java.lang.OutOfMemoryError: unable to create new native thread +2. **GC Roots 的引用链** + - GC Roots 对象包括： + - 虚拟机栈（栈帧中的局部变量表）引用的对象。 + - 方法区中静态变量引用的对象。 + - 方法区中常量引用的对象（如字符串常量池）。 + - JNI（Java Native Interface）引用的本地方法栈对象。 + + - **遍历过程**：从 GC Roots 出发，递归遍历所有引用链。未被遍历到的对象即为不可达对象。 + +3. **第一次标记结果** - - ![](https://tva1.sinaimg.cn/large/007S8ZIlly1gehn7osaz1j31940kc4c8.jpg) + - **存活对象**：与 GC Roots 存在引用链，继续保留。 -- java.lang.OutOfMemoryError：Metaspace + - **待回收对象**：不可达，被标记为“可回收”，进入第二次标记阶段。 - - http://openjdk.java.net/jeps/122 - - ![](https://tva1.sinaimg.cn/large/007S8ZIlly1gehnc3d4g3j319e0msguj.jpg) - - ![](https://tva1.sinaimg.cn/large/007S8ZIlly1gehndijxo8j31920madt6.jpg) +**二、第二次标记：finalize() 方法的自救机会** -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gehmmia4gaj30xw0gudid.jpg) +1. **筛选条件** + - 若对象未覆盖 `finalize()` 方法，或 `finalize()` 已被调用过，则直接判定为死亡，无需进入队列。 + - 若对象覆盖了 `finalize()` 且未被调用过，则将其加入 **F-Queue 队列**，进入自救流程。 +2. **F-Queue 与 Finalizer 线程** + - **F-Queue**：一个低优先级的队列，存放待执行 `finalize()` 的对象。 -## 6. GC垃圾回收算法和垃圾收集器的关系？分别是什么，请你谈谈？ + - Finalizer 线程：JVM 创建的守护线程，负责异步执行队列中对象的 `finalize()` 方法。 + - **注意**：`finalize()` 的执行不保证完成（如线程优先级低或方法死循环）。 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1geho5bjeg5j31e409m0xb.jpg) +3. **自救机制** -![](https://tva1.sinaimg.cn/large/007S8ZIlly1geho87aqmuj31260dqdl2.jpg) + 在 `finalize()` 方法中，对象可通过重新与 GC Roots 引用链建立关联来自救： + ```java + public class Zombie { + private static Zombie SAVE_HOOK; + + @Override + protected void finalize() throws Throwable { + super.finalize(); + System.out.println("finalize() 执行，对象自救"); + SAVE_HOOK = this; // 重新建立与 GC Roots 的关联 + } + + public static void main(String[] args) throws Exception { + SAVE_HOOK = new Zombie(); + SAVE_HOOK = null; // 断开引用，触发 GC + System.gc(); + Thread.sleep(500); // 等待 Finalizer 线程执行 finalize() + if (SAVE_HOOK != null) { + System.out.println("对象存活"); + } else { + System.out.println("对象被回收"); + } + } + } + + + ----- 输出结果 + finalize() 执行，对象自救 + 对象存活 + ``` +- 关键点： + - 对象通过 `finalize()` 将 `this` 赋值给静态变量 `SAVE_HOOK`，重新建立与 GC Roots 的引用链。 + - 自救仅生效一次，第二次 GC 时对象仍会被回收。 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gehoafwkoaj31a00my7js.jpg) +**三、对象回收的最终判定** +1. **第二次标记结果** + - **自救成功**：对象重新与引用链关联，移出待回收集合。 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gehobkwiegj31da0mc7ds.jpg) + - **自救失败**：对象仍不可达，被标记为“死亡”，等待内存回收。 +2. **回收内存** + 根据垃圾收集算法（如标记-清除、复制、标记-整理等），将死亡对象的内存回收。 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gehofhsuglj31a20ka116.jpg) +### 🎯 说一说常用的 GC 算法及其优缺点 +垃圾回收（Garbage Collection, GC）是自动内存管理的关键部分，用于回收不再使用的对象，防止内存泄漏。以下是一些常用的 GC 算法及其优缺点： +| 算法 | 思路 | 优点 | 缺点 | +| ----------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | +| 标记-清除（Mark-Sweep） | ![10.jpg](https://s0.lgstatic.com/i/image6/M00/56/27/Cgp9HWEsmp-AV969AAAXLK9JRog778.jpg)黑色区域表示待清理的垃圾对象，标记出来后直接清空 | 简单易懂。
不需要移动对象，保留了对象的原始引用。 | 效率问题：标记和清除过程可能产生较长的停顿时间（Stop-the-World）。
内存碎片：清除后内存中可能会有大量碎片，影响后续内存分配 | +| 标记-整理（Mark-Compact） | ![12.jpg](https://s0.lgstatic.com/i/image6/M00/56/28/Cgp9HWEsmuuAVl1CAAAXeNiLceQ497.jpg)将垃圾对象清理掉后，同时将剩下的存活对象进行整理挪动（类似于 windows 的磁盘碎片整理)，保证它们占用的空间连续 | 解决了内存碎片问题，通过整理阶段移动对象，减少内存碎片。 | 效率问题：整理阶段可能需要移动对象，增加GC的开销。 移动对象可能导致额外的开销，因为需要更新所有指向移动对象的引用。 | +| 复制（Copying） | ![11.jpg](https://s0.lgstatic.com/i/image6/M00/56/30/CioPOWEsmsiAaFzWAAAZCCjflgw041.jpg)将内存对半分，总是保留一块空着（上图中的右侧），将左侧存活的对象（浅灰色区域)复制到右侧，然后左侧全部清空 | 简单且高效，特别适合新生代GC。 没有内存碎片问题，因为复制算法通过复制存活对象到另一个空间来避免碎片。 | 空间利用率低：只使用堆的一半（或者根据算法不同，使用的比例不同）。 复制过程可能产生短暂的停顿。 | +| 分代收集（Generational Collection） | ![14.jpg](https://s0.lgstatic.com/i/image6/M00/56/28/Cgp9HWEsmzOALLLdAAAteML8kLI706.jpg) | 基于弱分代假设（大部分对象都是朝生夕死的），通过将对象分配到不同的代来优化GC性能。 新生代使用复制算法，老年代使用标记-清除或标记-整理算法。 | 对于长寿命的对象，如果频繁晋升到老年代，可能会增加老年代的GC压力。 需要维护多个数据结构来区分不同代的对象。 | -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gehohen24lj31f20n2du8.jpg) +每种GC算法都有其适用场景和限制。选择合适的GC算法取决于应用程序的具体需求，包括对延迟的敏感度、堆内存的大小、对象的生命周期特性等因素。现代 JVM 通常提供了多种 GC 算法，允许开发者根据需要选择或调整。 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1geholtp9p9j31bu0i8dsm.jpg) +### 🎯 JVM中一次完整的GC流程是怎样的，对象如何晋升到老年代 +**思路：** 先描述一下Java堆内存划分，再解释 Minor GC，Major GC，full GC，描述它们之间转化流程。 +- Java堆 = 新生代 + 老年代 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gehop8c2u8j30uu0kgk46.jpg) +- 新生代 = Eden + S0 + S1 + +- 当 Eden 区的空间满了， Java虚拟机会触发一次 Minor GC，以收集新生代的垃圾，存活下来的对象，则会转移到 Survivor区。 + +- **大对象**（需要大量连续内存空间的Java对象，如那种很长的字符串）**直接进入老年态**； + +- 如果对象在 Eden 出生，并经过第一次 Minor GC 后仍然存活，并且被 Survivor 容纳的话，年龄设为 1，每熬过一次 Minor GC，年龄+1，**若年龄超过一定限制（15），则被晋升到老年态**。即**长期存活的对象进入老年态**。 + + > s0 与 s1 的角色其实会互换，来回移动 + > + > 注：这里其实已经综合运用了“【标记-清理eden】+【标记-复制 eden->s0】”算法。 + +- 老年代满了而**无法容纳更多的对象**，Minor GC 之后通常就会进行Full GC，Full GC 清理整个内存堆 – **包括年轻代和年老代**。 + +- Major GC **发生在老年代的GC**，**清理老年区**，经常会伴随至少一次Minor GC，**比Minor GC慢10倍以上**。 + +![img](https://static001.infoq.cn/resource/image/e6/69/e663bd3043c6b3465edc1e7313671d69.png) + + + +### 🎯 GC分代年龄为什么最大为15？ + +> 在 hotspot 虚拟机中，对象在堆内存中的存储布局可以划分为三部分：**对象头，实例数据，对齐填充**。 +> HotSpot虚拟机对象的对象头部包含两类信息 +> +> - 用于存储对象自身的运行时数据，如HashCode，GC的分代年龄，锁状态标志，线程持有的锁，偏向线程ID，偏向时间戳等。这部数据的长度在32位和64位的虚拟机中分别为32比特和64比特，官方称为“Mark word”。 +> - 另一种是类型指针，即对象指向它的类型元数据的指针，Java通过这个指针确定该对象是哪个类的实例。但是并不是所有的虚拟机实现都必须在对象数据上保留类型指针，换句话说，查找对象的元数据信息不一定要经过对象本身。 +> +> 在32位的HotSpot虚拟机中，如对象未被同步锁锁定的状态下，Mark Word的32个比特存储空间中的25个比特用于存储对象的HashCode，**4个比特存储对象分代年龄**，2个比特存储锁标志位，一个比特固定为0. +> +> **因为Object Header采用4个bit位来保存年龄，4个bit位能表示的最大数就是15！** + +在 JVM 的实现中，尤其是 HotSpot 虚拟机中，分代年龄的最大值是 15。这是由 JVM 的设计和垃圾收集器的实现所决定的，主要原因如下： +**标记字段的位数限制**： +- 在 HotSpot VM 中，对象头的 Mark Word 里存储了对象的年龄信息。对象头的 Mark Word 是一个 32 位或者 64 位的字段 (取决于 JVM 的运行模式)。 +- 为了高效地存储和管理对象的年龄，JVM 为年龄字段分配了 4 位。这意味着对象年龄的范围是 0 到 15，即最大为 15。 +**合理的晋升策略**： +- 在分代垃圾收集策略中，大多数对象在年轻代创建，并且很快变得不可达而被回收。少数存活较长的对象会逐渐晋升到老年代。 +- 通过设定最大年龄为 15，可以确保在适当的时间点将长期存活的对象晋升到老年代，减少年轻代的负担。 +- 这一设计是基于常见的对象生命周期分布 (即大部分对象短命，少数对象长寿) 而进行的优化。 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gehptemx4oj31520js47e.jpg) +**性能和效率**： -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gehps6jzcsj31go0mok5q.jpg) +- 使用 4 位来表示对象年龄，使得垃圾收集器能够高效地处理和管理对象的晋升逻辑。 +- 限制年龄最大为 15 简化了垃圾收集器的实现，并且足以区分短命对象和长命对象，从而优化垃圾收集的性能。 +### 🎯 你知道哪几种垃圾收集器，各自的优缺点，重点讲下cms和G1，包括原理，流程，优缺点。 + +> GC 垃圾回收算法和垃圾收集器的关系？分别是什么请你谈谈？ + +> 实际上，垃圾收集器（GC，Garbage Collector）是和具体 JVM 实现紧密相关的，不同厂商（IBM、Oracle），不同版本的 JVM，提供的选择也不同。 +> +> 不算上后面出现的神器 ZGC，历史上出现过 7 种经典的垃圾回收器。 +> +> ![25.png](https://s0.lgstatic.com/i/image6/M00/56/29/Cgp9HWEsoXuAR2WEAAEJEJx1RMY327.png) +> +> 这些回收器都是基于分代的，把 G1 除外，按回收的分代划分如下。 +> +> - 横线以上的 3 种：Serial、ParNew、Parellel Scavenge 都是回收年轻代的； +> +> - 横线以下的 3 种：CMS、Serial Old、Parallel Old 都是回收老年代的。 + +**思路：** 一定要记住典型的垃圾收集器，尤其 cms 和 G1，它们的原理与区别，涉及的垃圾回收算法。 +#### a、几种垃圾收集器： +- **Serial收集器：** 单线程的收集器，收集垃圾时，必须 stop the world，使用复制算法。无需维护复杂的数据结构，初始化也简单，所以一直是 Client 模式下 JVM 的默认选项 +- **ParNew收集器：** 一款多线程的收集器，采用复制算法，主要工作在 Young 区，可以通过 `-XX:ParallelGCThreads` 参数来控制收集的线程数，整个过程都是 STW 的，常与 CMS 组合使用 -### 7.怎么查看服务器默认的垃圾收集器是哪个？生产上如何配置垃圾收集器？谈谈你对垃圾收集器的理解？ +- **Parallel Scavenge收集器：** 新生代收集器，复制算法的收集器，并发的多线程收集器，目标是达到一个可控的吞吐量。如果虚拟机总共运行100分钟，其中垃圾花掉1分钟，吞吐量就是99%。 -### 8.G1 垃圾收集器？ +- **Serial Old收集器：** 是Serial收集器的老年代版本，单线程收集器，使用标记整理算法。 +- **Parallel Old收集器：** 是Parallel Scavenge收集器的老年代版本，使用多线程，标记-整理算法。 +- **CMS(Concurrent Mark Sweep) 收集器：** 是一种以获得最短回收停顿时间为目标的收集器，**标记清除算法，运作过程：初始标记，并发标记，重新标记，并发清除**，收集结束会产生大量空间碎片。其中初始标记和重新标记会 STW。另外，既然强调了并发（Concurrent），CMS 会占用更多 CPU 资源，并和用户线程争抢。多数应用于互联网站或者 B/S 系统的服务器端上，JDK9 被标记弃用，JDK14 被删除，详情可见 [JEP 363](https://openjdk.java.net/jeps/363)。 -### 9.生产环境服务器变慢，诊断思路和性能评估谈谈？ +- **G1收集器：** 一种兼顾吞吐量和停顿时间的 GC 实现，是 Oracle JDK 9 以后的默认 GC 选项。 -### 10.假设生产环境出现 CPU占用过高，请谈谈你的分析思路和定位 + G1 GC 仍然存在着年代的概念，但是其内存结构并不是简单的条带式划分，而是类似棋盘的一个个 Region。Region 之间是复制算法，但整体上实际可看作是标记 - 整理（Mark-Compact）算法，可以有效地避免内存碎片，尤其是当 Java 堆非常大的时候，G1 的优势更加明显。 + 标记整理算法实现，**运作流程主要包括以下：初始标记，并发标记，最终标记，筛选标记**。不会产生空间碎片，可以精确地控制停顿。 + ![Garbage Collector Type](https://www.perfmatrix.com/wp-content/uploads/2019/02/Garbage-Collector-Type.png) -### 11. 对于JDK 自带的JVM 监控和性能分析工具用过哪些？你是怎么用的？ +- **Z Garbage Collector (ZGC)**（Z 垃圾回收器）：ZGC 是 **低延迟垃圾回收器**，它的设计目标是将垃圾回收的停顿时间控制在毫秒级别。ZGC 是一种并行、并发、分代的垃圾回收器，特别适合需要低停顿和大堆内存的应用。 -- jconsole 直接在jdk/bin目录下点击jconsole.exe即可启动 -- VisualVM jdk/bin目录下面jvisualvm.exe + **适用场景**：适用于要求极低延迟和大内存的应用，如大数据处理、高频交易等。 -https://www.cnblogs.com/ityouknow/p/6437037.html + 特点： + - 设计目标是 **低延迟**。 + - 适合非常大的堆内存（例如超过几百 GB）。 + - 默认配置：`-XX:+UseZGC`（从 JDK 15 开始支持） +#### b、CMS 收集器和 G1 收集器的区别： +目前使用最多的是 CMS 和 G1 收集器，二者都有分代的概念，主要内存结构如下 +![img](https://p1.meituan.net/travelcube/3a6dacdd87bfbec847d33d09dbe6226d199915.png) -## JMM +- CMS 收集器是老年代的收集器，可以配合新生代的 Serial 和 ParNew 收集器一起使用； +- G1 收集器收集范围是老年代和新生代，不需要结合其他收集器使用； +- CMS 收集器以最小的停顿时间为目标的收集器； +- G1 GC 这是一种兼顾吞吐量和停顿时间的 GC 实现，是 Oracle JDK 9 以后的默认 GC 选项。 +- G1 GC 仍然存在着年代的概念，但是其内存结构并不是简单的条带式划分，而是类似棋盘的一个个 region。Region 之间是复制算法，但整体上实际可看作是标记 - 整理（Mark-Compact）算法，可以有效地避免内存碎片，尤其是当 Java 堆非常大的时候，G1 的优势更加明显。G1 吞吐量和停顿表现都非常不错，并且仍然在不断地完善，与此同时 CMS 已经在 JDK 9 中被标记为废弃（deprecated），所以 G1 GC 值得你深入掌握。 +- CMS 收集器是使用“标记-清除”算法进行的垃圾回收，容易产生内存碎片 -### 为什么要有内存模型 +> JDK 又增加了两种全新的 GC 方式，分别是： +> +> - [Epsilon GC](http://openjdk.java.net/jeps/318)，简单说就是个不做垃圾收集的 GC，似乎有点奇怪，有的情况下，例如在进行性能测试的时候，可能需要明确判断 GC 本身产生了多大的开销，这就是其典型应用场景。 +> - [ZGC](http://openjdk.java.net/jeps/333)，这是 Oracle 开源出来的一个超级 GC 实现，具备令人惊讶的扩展能力，比如支持 T bytes 级别的堆大小，并且保证绝大部分情况下，延迟都不会超过 10 ms。虽然目前还处于实验阶段，仅支持 Linux 64 位的平台，但其已经表现出的能力和潜力都非常令人期待。 -Java虚拟机规范中试图定义一种「 **Java 内存模型**」来**屏蔽掉各种硬件和操作系统的内存访问差异**，以实现**让 Java 程序在各种平台下都能达到一致的内存访问效果**，不必因为不同平台上的物理机的内存模型的差异，对各平台定制化开发程序。 -Java 内存模型的主要目标是定义程序中各个变量的访问规则，即在虚拟机中将变量存储到内存和从内存中取出变量这样的底层细节。这里的变量与我们写 Java 代码中的变量不同，它包括了实例字段、静态字段和构成数组对象的元素，但不包括局部变量和方法参数，因为他们是线程私有的，不会被共享。 +### 🎯 **Minor GC（小型 GC）** +收集年轻代内存的 GC 事件称为 Minor GC。关于 Minor GC 事件，我们需要了解一些相关的内容： + +1. 当 JVM 无法为新对象分配内存空间时就会触发 Minor GC（ 一般就是 Eden 区用满了）。如果对象的分配速率很快，那么 Minor GC 的次数也就会很多，频率也就会很快。 + +2. Minor GC 事件不处理老年代，所以会把所有从老年代指向年轻代的引用都当做 GC Root。从年轻代指向老年代的引用则在标记阶段被忽略。 + +3. 与我们一般的认知相反，Minor GC 每次都会引起 STW 停顿（stop-the-world），挂起所有的应用线程。对大部分应用程序来说，Minor GC 的暂停时间可以忽略不计，因为 Eden 区里面的对象大部分都是垃圾，也不怎么复制到存活区/老年代。但如果不符合这种情况，那么很多新创建的对象就不能被 GC 清理，Minor GC 的停顿时间就会增大，就会产生比较明显的 GC 性能影响。 + + + +### 🎯 Minor GC 和 Full GC 触发条件 + +我们知道，除了 Minor GC 外，另外两种 GC 事件则是： + +- Major GC（大型 GC）：清理老年代空间（Old Space）的 GC 事件。 +- Full GC（完全 GC）：清理整个堆内存空间的 GC 事件，包括年轻代空间和老年代空间。 + + + +### 🎯 什么时候会触发 Full GC ? + +1. `System.gc()` 方法的调用 + + 此方法的调用是建议 JVM 进行 Full GC，虽然只是建议而非一定，但很多情况下它会触发 Full GC，从而增加Full GC 的频率，也即增加了间歇性停顿的次数。强烈影响系建议能不使用此方法就别使用，让虚拟机自己去管理它的内存，可通过 -XX:+ DisableExplicitGC 来禁止 RMI 调用 System.gc。 + +2. 老年代空间不足 + + 老年代空间只有在新生代对象转入及创建大对象、大数组时才会出现不足的现象，当执行 Full GC 后空间仍然不足，则抛出如下错误：java.lang.OutOfMemoryError: Java heap space 为避免以上两种状况引起的 Full GC，调优时应尽量做到让对象在 Minor GC 阶段被回收、让对象在新生代多存活一段时间及不要创建过大的对象及数组。 + +3. 老年代的内存使用率达到了一定阈值（可通过参数调整），直接触发 FGC + +4. 空间分配担保：在 YGC 之前，会先检查老年代最大可用的连续空间是否大于新生代所有对象的总空间。如果小于，说明 YGC 是不安全的，则会查看参数 HandlePromotionFailure 是否被设置成了允许担保失败，如果不允许则直接触发 Full GC；如果允许，那么会进一步检查老年代最大可用的连续空间是否大于历次晋升到老年代对象的平均大小，如果小于也会触发 Full GC + +5. Metaspace（元空间）在空间不足时会进行扩容，当扩容到了-XX:MetaspaceSize 参数的指定值时，也会触发FGC + +6. G1 收集器的 Mixed GC 触发条件满足时，会对老年代和部分新生代进行回收，本质上也属于 Full GC 的范畴。当老年代占用空间达到阈值（默认 45%），就会触发混合回收来控制内存使用。 + + + + +### 🎯 System.gc() 和 Runtime.gc() 会做什么事情 + +`java.lang.System.gc()` 只是 `java.lang.Runtime.getRuntime().gc()` 的简写，两者的行为没有任何不同 + +其实基本没什么机会用得到这个命令，因为这个命令只是建议 JVM 安排 GC 运行，还有可能完全被拒绝。 GC 本身是会周期性的自动运行的，由 JVM 决定运行的时机，而且现在的版本有多种更智能的模式可以选择，还会根据运行的机器自动去做选择，就算真的有性能上的需求，也应该去对 GC 的运行机制进行微调，而不是通过使用这个命令来实现性能的优化 + + + +### 🎯 SafePoint 是什么 + +比如 GC 的时候必须要等到 Java 线程都进入到 safepoint 的时候 VMThread 才能开始 执行 GC， + +1. 循环的末尾 (防止大循环的时候一直不进入 safepoint，而其他线程在等待它进入 safepoint) +2. 方法返回前 +3. 调用方法的call之后 +4. 抛出异常的位置 + + + +### 🎯 你们用的是什么 GC，都有哪些配置 + +``` +APP_OPTS:-Xms6656m -Xmx6656m -XX:MetaspaceSize=512m -XX:MaxMetaspaceSize=1024m -XX:+UseG1GC -XX:+HeapDumpOnOutOfMemoryError +-XX:HeapDumpPath=/home/Logs/***Heap.hprof +-verbose:gc -Xloggc:/home/Logs/gc.log +-XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintHeapAtGC +-Dprofiler.thread.pool.enable=true -Dprofiler.thread.pool.spring.enable=true +-Dprofiler.thread.pool.reject-handler.enable=true -Dprofiler.jdbc.druid.poolmetric=true +``` + +``` +xxx 143 1 54 Jun07 ? 1-13:05:06 /opt/local/jdk/bin/java -Djava.util.logging.config.file=/opt/local/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -server -Xmn1024m -Xms22938M -Xmx22938M -XX:PermSize=512m -XX:MaxPermSize=512m -XX:ParallelGCThreads=3 -XX:+UseConcMarkSweepGC -XX:+UseCMSInitiatingOccupancyOnly -XX:CMSInitiatingOccupancyFraction=80 -XX:+CMSScavengeBeforeRemark -XX:SoftRefLRUPolicyMSPerMB=0 -XX:ParallelGCThreads=3 -Xss1m -XX:+PrintGCDateStamps -XX:+PrintGCDetails -Xloggc:/opt/logs/gc.log -verbose:gc -XX:+DisableExplicitGC -Dsun.rmi.transport.tcp.maxConnectionThreads=400 -XX:+ParallelRefProcEnabled -XX:+PrintTenuringDistribution -Dsun.rmi.transport.tcp.handshakeTimeout=2000 -Xdebug -Xrunjdwp:transport=dt_socket,address=22062,server=y,suspend=n -Dcom.sun.management.snmp.port=18328 -Dcom.sun.management.snmp.interface=0.0.0.0 -Dcom.sun.management.snmp.acl=false -javaagent:/opt/opbin/service_control/script/jmx_prometheus_javaagent-0.13.0.jar=9999:/opt/opbin/service_control/script/config.yaml -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dignore.endorsed.dirs= -classpath /opt/local/tomcat/bin/bootstrap.jar:/opt/local/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/opt/local/tomcat -Dcatalina.home=/opt/local/tomcat -Djava.io.tmpdir=/opt/local/tomcat/temp org.apache.catalina.startup.Bootstrap start start +``` + + + +### 🎯 讲一讲 G1？ + +> G1 是 Java 9 之后默认的垃圾收集器，主要设计目标是 **可预测停顿时间**，非常适合大内存低延迟应用。 +> +> 它的核心机制是把堆划分为大量大小相等的 Region，逻辑上分代但物理上打散，这样可以避免内存碎片。 +> +> 回收策略上，除了常规的 Young GC，它会在老年代达到阈值后执行并发标记，接着进行 Mixed GC——既清理新生代，也挑选垃圾比例高的老年代 Region，分多次执行，避免长时间停顿。 +> +> 技术上，G1 通过 **Remembered Set** 追踪跨 Region 引用，结合停顿预测模型，尽量在用户设定的目标时间内完成 GC。 +> +> 相比 CMS，G1 的优势是避免碎片化、停顿可控；缺点是吞吐量略低，并且 Humongous 对象会带来额外开销。 +> +> 如果让我一句话总结：**G1 通过 Region 化内存管理 + 并发标记 + Mixed GC，在保证高回收效率的同时，把停顿控制在用户期望的范围内。** + +G1 垃圾回收器（Garbage First，简称 G1 GC）是目前 HotSpot JVM 中 **面向服务端、注重低延迟与大堆内存场景** 的一款重要垃圾收集器。它的核心目标是：**在尽可能保证高吞吐量的同时，提供可预测的停顿时间**。 + +**1. 内存布局：Region 化** + +- G1 将堆划分为大量大小相等的 **Region**（1M ~ 32M，2 的次幂）。 +- 逻辑上还是分代（年轻代、老年代），但物理上是分散的 Region。 +- 特殊 **Humongous Region** 用于大对象（> ½ Region 大小），连续分配几个 Region。![img](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjmCXRrxZFopVpaiUcBrJLO9UNj6OBq0ee2OeodaeNxp4sWg1eU4CpvT2gIdm0oUoS8T4cVtJ0K9kHSjNCOf8G8fQ8XjvZck1mldVzCbF0quXd3IvGU7C_6dLY82meiX1SiHYguVZagwGQV/s1600/G1Regions.jpg) + +优势：避免大对象导致碎片化，内存利用更灵活。 + +------ + +**2. 分代与回收策略** + +- **新生代 GC（Young GC）** + - Stop-The-World，复制 Eden → Survivor / Old。 + - 依赖 **Remembered Set** 处理跨代引用。 +- **并发标记** + - 触发条件：老年代占比达到 `InitiatingHeapOccupancyPercent`。 + - 四个阶段：初始标记（STW）、并发标记、重新标记（STW）、筛选回收集。 +- **Mixed GC** + - 在并发标记后触发，不仅清理新生代，还挑选部分老年代 Region（垃圾比例高的）。 + - 分多次执行，避免 Full GC。 +- **Full GC** + - 当 Mixed GC 也跟不上时，触发单线程 Full GC，性能极差，需尽量避免。 + +------ + +**3. Remembered Set（RSet）** + +- 每个 Region 维护一个 RSet，记录哪些其他 Region 指向自己。 +- 避免每次扫描整个堆来处理跨代引用。 +- 本质基于 **Card Table**，增量维护。 + +👉 代价：写屏障会带来额外开销，但换来了 GC 的高效。 + +------ + +**4. 停顿时间模型（Pause Prediction Model）** + +- G1 最大的亮点：**可预测停顿**。 +- JVM 会维护每个 Region 的 **存活对象比例**，并估算清理它的成本。 +- GC 时选择“垃圾最多、回收收益最高”的 Region（Garbage First）。 +- 用户可通过 `-XX:MaxGCPauseMillis` 指定停顿目标，G1 会尽量满足。 + +**5. 优缺点** + +✅ 优点： + +- 适合大内存（> 4G），低延迟应用。 +- 避免碎片化，全堆压缩。 +- Mixed GC 灵活，避免长时间 Full GC。 +- 自动调优，用户只需设定停顿目标。 + +❌ 缺点： + +- 吞吐量不如 Parallel GC（为低延迟牺牲部分吞吐）。 +- Humongous 对象处理成本高。 +- 维护 RSet 带来额外 CPU 开销。 + +**运行过程** + +1. **Young GC**（新生代满时触发） + - Stop-The-World，复制 Eden → Survivor / Old。 + - 依赖 **Remembered Set** 处理跨代引用。 +2. **并发标记**（老年代达到阈值时触发） + - 初始标记：STW，标记 GC Roots 直接可达对象。 + - 并发标记：应用线程并行，标记整个堆。 + - 重新标记：STW，修正并发期间的变化。 + - 筛选回收集：计算各 Region 垃圾比例。 +3. **Mixed GC** + - 回收新生代 + 部分老年代 Region。 + - 分多次执行，避免一次性长时间停顿。 +4. **Full GC**（极端情况） + - G1 调度失败时会触发单线程 Full GC，开销很大。 + + + +> 你可以思考下 region 设计有什么副作用？ +> +> 例如，region 大小和大对象很难保证一致，这会导致空间的浪费。不知道你有没有注意到，我的示意图中有的区域是 Humongous 颜色，但没有用名称标记，这是为了表示，特别大的对象是可能占用超过一个 region 的。并且，region 太小不合适，会令你在分配大对象时更难找到连续空间，这是一个长久存在的情况，请参考[OpenJDK 社区的讨论](http://mail.openjdk.java.net/pipermail/hotspot-gc-use/2017-November/002726.html)。这本质也可以看作是 JVM 的 bug，尽管解决办法也非常简单，直接设置较大的 region 大小，参数如下： +> +> ``` +> -XX:G1HeapRegionSize=M +> ``` +> +> 从 GC 算法的角度，G1 选择的是复合算法，可以简化理解为： +> +> - 在新生代，G1 采用的仍然是并行的复制算法，所以同样会发生 Stop-The-World 的暂停。 +> - 在老年代，大部分情况下都是并发标记，而整理（Compact）则是和新生代 GC 时捎带进行，并且不是整体性的整理，而是增量进行的。 +> +> 习惯上人们喜欢把新生代 GC（Young GC）叫作 Minor GC，老年代 GC 叫作 Major GC，区别于整体性的 Full GC。但是现代 GC 中，这种概念已经不再准确，对于 G1 来说： +> +> - Minor GC 仍然存在，虽然具体过程会有区别，会涉及 Remembered Set 等相关处理。 +> - 老年代回收，则是依靠 Mixed GC。并发标记结束后，JVM 就有足够的信息进行垃圾收集，Mixed GC 不仅同时会清理 Eden、Survivor 区域，而且还会清理部分 Old 区域。可以通过设置下面的参数，指定触发阈值，并且设定最多被包含在一次 Mixed GC 中的 region 比例。 +> +> ```sh +> –XX:G1MixedGCLiveThresholdPercent +> –XX:G1OldCSetRegionThresholdPercent +> ``` +> +> 从 G1 内部运行的角度，下面的示意图描述了 G1 正常运行时的状态流转变化，当然，在发生逃逸失败等情况下，就会触发 Full GC。![](https://learn.lianglianglee.com/%E4%B8%93%E6%A0%8F/Java%20%E6%A0%B8%E5%BF%83%E6%8A%80%E6%9C%AF%E9%9D%A2%E8%AF%95%E7%B2%BE%E8%AE%B2/assets/47dddbd91ad0e0adbd164632eb9facec-20221127201847-p5gmfe2.png) +> +> G1 相关概念非常多，有一个重点就是 **Remembered Set**，用于记录和维护 region 之间对象的引用关系。为什么需要这么做呢？试想，新生代 GC 是复制算法，也就是说，类似对象从 Eden 或者 Survivor 到 to 区域的“移动”，其实是“复制”，本质上是一个新的对象。在这个过程中，需要必须保证老年代到新生代的跨区引用仍然有效。下面的示意图说明了相关设计。 +> +> 主要用于解决分代垃圾收集器中的跨代引用问题 +> +> ![img](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAwgAAAIZCAYAAAAC+BidAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAADsMAAA7DAcdvqGQAAEizSURBVHhe7d0JlJXVne99V/e93ffNTfdtb9K3c28nb3evJCt5703fXt2x13Jlpe/b/UaFyFQFGJDYXjoOF000ahgLKGaKUWSQuRAEESlARBBBFEGccCIyBAcQxSEqTiiDTP+3/k89+9Spw66qs6Gq9t7n+X7W+q3iOeNzTlHP3r9zzvOcCwQAAAAAUhQEAAAAADkUBAAAAAA5FAQAAAAAORQEAAAAADkUBAAAAAA5FAQAAAAAORQEAAAAADkUBAAAAAA5FAQAAAAAORQEAAAAADkUBAAAAAA5FAQAAAAAORQEAAAAADkUBAAAAAA5FAQAAAAAORQEAAAAADkUBAAAAAA5FAQAAAAAORQEAAAAADkUBAAAAAA5FAQAAAAAORQEAAAAADkUBAAAAAA5FAQAAAAAORQEAAAAADkUBAAAAAA5FAQAAAAAORQEAAAAADkUBAAAAAA5FAQAAAAAORQEAAAAADkUBAAAAAA5FAQAQEkYP368XHDBBcnPGD3xxBPJ+rdr1y49BQD8oCAAQMR0QmnLt7/9bbnhhhtk37596SVLX5YKgl5Wf7/6e87/vet1KyoqZMeOHeklAcAdBQEAIpY/OWws9957b3rp0paVgqAFIP/321h69OiRXqNpen96eb1/AFAUBACImJkMFlq3bp1cdNFFyXkXXnihHDp0KD2ndGWhIGjZM79zfZz57xTou0V6vpnwN1c0DAoCgEIUBACImJks2mgpMOdrYSh1WSgIpvQ19xj1tngHAcC5oiAAQMR0YqdpjJn8mQmllgb9iEr+Z9d10lk44TTn6eX1PDMx1ejE00wm9VVr/Sy8vkuh5+lPXc5/x2L27NkNrm/2j8i/TP4kNf9VcI1e13xMyqxP/vrr+ugr6Xq6LuvPwvXSy+vp+fepdNnl+VBm8m1OM5ct5nEq23Omt2fWv6mCYG672MKnl8tfV3Nf+nzp4zCn22Iel3m8Zn01uo76vGVpHxcgSygIABAxM2FrjJn46gRbJ4X5k7zC6KTVMKc1d3nb6RqdKOvEOH/CXBhzGWUKQVP3VzhJzY+ebtanufs0zuX5sD1mnSgX+zibu09NUwXB3I8+F81p6vej66CFxnaeiRaE5i7T1LoCiBcFAQAiZiZqNvmvEOvE1JQF8wqyoa8ym0mreaXeXE9P14mieaVYf+ZPhvU29TpmAqz3aW7LXE5/5r/irfdtzjOTcFMQzGlm/fR28ye6Ta2PuV9zG7ouRuFj1NttiedD6fX1vOYeZ1P3qf82Ox83NenW9dHLmPvTCbw+TvP8G2Zir+ub//vR9TbPp66LMs99/vOl9Dq229Cfelld32KKCoD4UBAAIGI6gdPk0wmcTujMJFcncmbC2NiETid8er6ZnJrbzZ8IG2aSqhNUm/wJvU5CzcQyn55m1k+ZSarediFdB3N7ttsy62Oij9fGXE7v61yfj/xJvdJlPb2Yx2nu09xmocL7bIwWFL1cYfR+9PHk/+4L19cwz7feZ/6/85n1aez3DKB0URAAIGI6gWsqZrLc2CQwn7lO4b8LNTeRzZ/A6r8bU8wk1TC3Z2PWx6Sx+9TJurlMSz0f5rEW8zhN8t9lyNfc85pPS5MWDr2seVeiME3dTv56N/Zc5BczvZytLAIoTfatLQAgCmYCZ0v+RLRwktpUVP6/CzU3kTWTz2Kjt9fYJNUwl7Ux62PS1GTdXKalng/Xx6pp7DE297w2R6+vhdC8e1BMmioIqvDdGY1eXu+HwgCULvvWFgAQBTNpy2c+4qMTRfMRk5aaEKvmJrIUhKbT2GNs7nktlhbDwvtsLM0VBKVFQAuB7TmzfSQMQPzsW1sAQBTMRK1QYUlobhJYqLHbVc1NZPMnzU1N1vM1t37m9mzM+pg0dp/mIzP6nLTU82EeazGPs7n7bO55LVb+81HM43N9Lkxh0Ovocwmg9FAQACBiZiJYSD9vb46go59RNxNZs09Ccxq7XdXcRNbcl6bYHVybm6Sa27PJnxBrGpusm/XS8mR2GD7f58PspFzM4zSTar1/G/PKf1MFQXdCbu6jPWadNI3tpJzPPPeN7RvRGHMfAEoPf9kAELGmJmmFJcFcVieZhRNxXdYJrJm8NnW7LgVBo+ugk8/8o/zoxFUn6WZi3ZIFQW8rf7KrE+r8ddLl/KMLne/zYda9uceZP3HX08zl9HRTHjSNPa/KXEbXSz/ek18W9N/5RzDS6L8bu5x53Hpbelldx/zL6XnmqEj5RUPX2xQs/X8FoPRQEAAgYmYi2Jj8iXAxMZNTs2xjJuSNTWTNZFwnvTrpNLfVWJSZZBdO1I38yxYy61NMdLJr6KS3ueemmOdDn+NiH2d+ESiMuY2mCkKxv0ud9JuJf1PR566x58/8ThqLrktjvy8AcbNv7QAAUTCTtabkT4R10qcT+MLJnzndvFJsTrcxE8rGJrKmIOhPpZNyvWz+5FYnwzpZNq/0m/VpbMJprmdTOME1k+Pm3jVROrk/3+fDKOZxKr1cfqEw69bc82robenjKywl+nj19PzHqf/W289/LszldD3Muxh6m/m3p5fRdwn0Mnr9/Mdkrp//bgOA0kJBAAAAAJBDQQAAAACQQ0EAAAAAkENBAAAAAJBDQQAAAACQQ0EAAAAAkENBAAAAAJBDQQAAAACQQ0EAAAAAkENBAAAAAJBDQQAAAACQQ0EAAAAAkENBAAAAAJBDQQAAAACQQ0EAAAAAkENBAAAAAJBDQQAAAACQQ0EAAAAAkENBAAAAAJBDQQAAAACQQ0EAAAAAkENBAAAAAJBDQQAAAACQQ0EAAAAAkENBAAAAAJBDQQAAAACQQ0EAAAAAkENBAAAAAJBDQQAAFO1fbp1PIg0AFIuCAAAomk40V27YTiILBQGACwoCAKBoFIQ4Q0EA4IKCAAAoGgUhzlAQALigIAAAikZBiDMUBAAuKAgAgKJREOIMBQGACwoCAKBoFIQ4Q0EA4IKCAAAoGgUhzlAQALigIAAAikZBiDMUBAAuKAgAgKJREOIMBQGACwoCAKBoFIQ4Q0EA4IKCAAAoGgUhzlAQALigIAAAikZBiDMUBAAuKAgAgKJREOIMBQGACwoCAKBoFIQ4Q0EA4IKCAAAoGgUhzlAQALigIAAAikZBiDMUBAAuKAgAgKJREOIMBQGACwoCAKBoFIQ4Q0EA4IKCAAAoGgUhzlAQALigIAAAikZBiDMUBAAuKAgAgKJREOIMBQGACwoCAKBoFIQ4Q0EA4IKCAAAoGgUhzlAQALigIAAAikZBiDMUBAAuKAgAgKJREOIMBQGACwoCAKBoFIQ4Q0EA4IKCAAAoGgUhzlAQALigIAAAikZBiDMUBAAuKAgAgKJREOIMBQGACwoCAKBoFIQ4Q0EA4IKCAAAoGgUhzlAQALigIAAAikZBiDMUBAAuKAgAgKJREOIMBQGACwoCAKBoFIQ4Q0EA4IKCAAAoGgUhzlAQALigIAAAikZBiDMUBAAuKAgAgKJREOIMBQGACwoCAKBoFIQ4Q0EA4IKCAAAoGgUhzlAQALigIAAAikZBiDMUBAAuKAgAgKJREOIMBQGACwoCAKBoFIQ4Q0EA4IKCAAAoGgUhzlAQALigIAAAikZBiDMUBAAuKAgAgKJREOIMBQGACwoCAKBoFIQ4Q0EA4IKCAAAoGgUhzlAQALigIAAAikZBiDMUBAAuKAgAgKJREOIMBQGACwoCAKBoFIQ4Q0EA4IKCAAAomk40SZwBgGJREAAARdOJpu0VahJ2KAgAXFAQAABFoyDEGQoCABcUBABA0SgIcYaCAMAFBQEAUDQKQpyhIABwQUEAABSNghBnKAgAXFAQAABFoyDEGQoCABcUBABA0SgIcYaCAMAFBQEAUDQKQpyhIABwQUEAABSNghBnKAgAXFAQAABFoyDEGQoCABcUBABA0SgIcYaCAMAFBQEAUDQKQpyhIABwQUEAABSNghBnKAgAXFAQAABFoyDEGQoCABcUBABA0SgIcYaCAMAFBQEAUDQKQpyhIABwQUEAABSNghBnKAgAXFAQAABFoyDEGQoCABcUBABA0SgIcYaCAMAFBQEAUDQKQpyhIABwQUEAABSNghBnKAgAXFAQAABFoyDEGQoCABcUBABA0SgIcYaCAMAFBQEAUDQKQpyhIABwQUEAABSNghBnKAgAXFAQAABF6zjmIesElIQdCgIAFxQEAEBRPj16Unrc+bx1AkrCDgUBgAsKAgCgKL89+Ll0m7LNOgElYYeCAMAFBQEAUJSNuz+SsqpHrBNQEnYoCABcUBAAAEVZ+OR70nHkg9YJKAk7FAQALigIAICijF57QNpV3GedgJKwQ0EA4IKCAAAoyo1LXpGf9L3LOgElYYeCAMAFBQEA0KwjX55KCoJONG0TUBJ2KAgAXFAQAADNeu39o8lHjCgIcYaCAMAFBQEA0KzNez9JdlKmIMQZCgIAFxQEAECzlj7z++QwpxSEOENBAOCCggAAaNakDW8lX5RGQYgzFAQALigIAIBm9at5XT49epKCEGkoCABcUBAAAE06ceqMXLPwd8m/KQhxhoIAwAUFAQDQpAOHjsmQ+/cn/6YgxBkKAgAXFAQAQJOe3f+ZzNnyTvJvCkKcoSAAcEFBAAA0aeULH8iaHR8m/6YgxBkKAgAXFAQAQJOmbjoozx84nPybghBnKAgAXFAQAABNqli1T9755HjybwpCnKEgAHBBQQAANEmPYHTq9Jnk3xSEOENBAOCCggAAaJS+czBgxevpEgUh1lAQALigIAAAGvXim4eTfRAMCkKcoSAAcEFBAAA0au3Lh2TZ9vfTJQpCrKEgAHBBQQAANGre1nfkqdc/S5coCLGGggDABQUBANCoYQ/sl/0fHk2XKAixhoIAwAUFAQDQqOsW7ZVjJ06nS3UFgcQZACgWBQEAYPXp0ZNyy7JX06U6OtG0vUJNwg4FAYALCgIAwGrPu1/I+PVvpkt1KAhxhoIAwAUFAQBgtXH3R7L46ffSpToUhDhDQQDggoIAALBa+OR7smnPx+lSHQpCnKEgAHBBQQAAWOnHi/a+dyRdqkNBiDMUBAAuKAgAAKsbl7wih4+dSpfqUBDiDAUBgAsKAgDgLEe+PCV9Fu9Nl+pREOIMBQGACwoCAOAs+uVo+iVphSgIcYaCAMAFBQEAcJYtr3wiC7a9my7VoyDEGQoCABcUBADAWZZtf1/W7/woXapHQYgzFAQALigIAICz3L7xLXnxzcPpUj0KQpyhIABwQUEAAJylX83r8sHhE+lSPQpCnKEgAHBBQQAANHDq9BnpvWBPutQQBSHOUBAAuKAgAAAaOPjxcalYtS9daoiCEGcoCABcUBAAAA08u/8zmfHYwXSpIQpCnKEgAHBBQQAANHD/ix8msaEgxBkKAgAXFAQAQAP67oG+i2BDQYgzFAQALigIAIAGdP8D3Q/BhoIQZygIAFxQEAAADegRjE6cOpMuNURBiDMUBAAuKAgAgBz97gP9DoTGUBDiDAUBgAsKAgAgR789Wb9FuTEUhDhDQQDggoIAAMhZv/MjWfrM79Ols1EQ4gwFAYALCgIAIGfBtndlyyufpEtnoyDEGQoCABcUBABAzrAH9sv+D4+mS2ejIMQZCgIAFxQEAEBOn8V75ciXp9Kls1EQ4gwFAYALCgIAIHH42Cm5cckr6ZIdBSHOUBAAuKAgAAASe979QqoeOpAu2VEQ4gwFAYALCgIAILFpz8ey8Mn30iU7CkKcoSAAcEFBAAAkFj/9nmzc/VG6ZEdBiDMUBAAuKAjn6dChQzJ79mxp166dXHDBBbnockVFhezbty+9ZENPPPGE9OjRI3f5Cy+8MFm+995700vY6fXy78dEr6vnAcC5Gr/+Tdn59hfpkh0FIc5QEAC4oCCchx07dsi3v/1t64TdRItCoRtuuMF6WZOmJvrjx4+3Xsdk3bp16SVbn5Ya2+MDEKeblr4qnx49mS7ZURDiDAUBgAsKwjnSdwZ0gqyTcp0kF07MdZKvRUBf2c+n7zbodfS6+m6BvgOh9KcuX3TRRUUVBP1p6LqY0qGFpa2Yxw4gfsdOnJbrFu1NlxpHQYgzFAQALigI58h8PEgn5sXSEmBKhb77cC5sBcEw72a0FQoCUDr0y9GG3L8/XWocBSHOUBAAuKAgnAN9xV4nxzrZN+8AFEPfIdDruZSKQudSEMw7E3qeRi+n17etu757UbhvhK5v/r4UZh0Kk/9uib6jYu7T3IbLcwWgbT31+mcyZ8s76VLjKAhxhoIAwAUF4RyYib7uhOzCTKz1+ufKVhDyP2JUWD7M6bboBD5/0m4ely06yTfvejRXELQc2M7PX2cAYVm2/X1Z+3LzJZ6CEGcoCABcUBDOgW2SrvJfeTfJf1VdP46jp53P0YYam5xr9PbzJ/x6P3q6Tu7zS4mebl7d130ilF5PL6en6X2Y29HyYR5X/mNR5j4LmXU0j1NvS08rfL4AhOP2jW/J8wcOp0uNoyDEGQoCABcUhHNgJsCFE15bQcifQLdmQSh8N0DpOxx6nu0dC1MezPqZV/0L34FQ+eUhX/7185kdsW23BSBMA1a8Lu98cjxdahwFIc5QEAC4oCCcAzMBbu4jRoUTaPNxn/M5FGlhOcn/eFHhZN0UksLiYORfx9xuY+tmbiuf7T6V3p+5vNn/IH8fBgBhOXX6jPResCddahoFIc5QEAC4oCCcA/0svk5+dWffxibfqnAC3RKvrBcWBMPsoJz/7oSZpLd1QTB0XUx50ZzPvhcAWs/Bj48n7yAUg4IQZygIAFxQEM6RmTA3NdkvnEDrq+hmstzShzk1Oxjnr4+ZnDf1ESOzX4Ht+ob5iJEmn15eP9rUHH2stusDCIPuezB108F0qWkUhDhDQQDggoJwjsy7CBotAfrKu3mlXn+aCXfhK+xm0q6TZb1M4XX08k3to9BYQTCTeD3P3KZZBz09/50B/bfZSVkvo/LLi77TYW5DH6fZt6KwPJjLm7Jjfurl8x+b3nZbf0cDgOKt2fGh1Dz/frrUNApCnKEgAHDBjO086ETbTMobS+GRf3TSbCbnjeVcCoIyOyWbIxMp806HLYUTfnPbtth2gi7cKduUocbu0/WwsADaxozHDibfg1AMCkKcoSAAcEFBOE86adaJdf6kX0uDTpLzX0UvpJP4wuuYV96bYvZjyC8BhnkXwEzUDV0/8wq+Ru+3sfvJf3dB09SXquk7BqYM6Pqb29SCk18ezG0ACJN+g/KBQ8fSpaZREOIMBQGACwoCAGTcNQt/JydOnUmXmkZBiDMUBAAuKAgAkGEffXFCbln2arrUPApCnKEgAHBBQQCADPvtwc9l0oa30qXmURDiDAUBgAsKAgBk2PqdH8nip99Ll5pHQYgzFAQALigIAJBhC7a9K5v2fJwuNY+CEGcoCABcUBAAIMNGrz0gr71/NF1qHgUhzlAQALigIABAhvVZvFcOHzuVLjWPghBnKAgAXFAQACCjtBhoQXChE00SZwCgWBQEAMgo/WjRyAffSJcAAKhDQQCAjNKdk3UnZQAA8lEQYjFxosjnn6cLAHD+9PCmephTAADyURBi0bGjSE1NugAA50+/IE2/KA0AgHwUhFjMmiXSu3e6AADn75Zlr8oHh0+kSwAA1KEgxOLgQZFvfCNdAIDzc+LUGblm4e/SJQAA6lEQYvKDH4g8/XS6AADn7sChY1Kxal+6BABAPQpCTIYMERk2LF0AgHP31OufyYzHDqZLAADUoyDE5IknRC66KF0AgHNX8/z7smbHh+kSAAD1KAgxOXlS5OtfF3nvvfQEADg3UzcdlGf3f5YuAQBQj4IQm6uuEpk3L10AgHMzYMXrcvDj4+kSAAD1KAixWbxYpHv3dAEAzk3vBXvk1Okz6RIAAPUoCLH55BORP/szkWPH0hMAwM07nxyXfjWvp0sAADREQYjRj38s8sgj6QIAuHn+wGG5feNb6RIAAA1REGJUVSVyyy3pAgC40aMXLdv+froEAEBDFIQY7dwp8p3vpAsA4GbOlndkyyufpEsAADREQYjVX/+1yGuvpQsAULwh9++X/R8eTZcAAGiIghCrPn1EpkxJFwCgeNct2ivHTpxOlwAAaIiCEKsHHxS55JJ0AQCK8+nRk3LjklfSJQAAzkZBiJUe5lQPd/r55+kJANC83x78XMavfzNdAgDgbBSEmLVvL1JTky4AQPM27v5IFj75XroEAMDZKAgxmzVL5Npr0wUAaJ6WAy0JAAA0hoIQszfeEPnGN9IFAGje6LUHZM+7X6RLAACcjYIQux/8QOS559IFAGia7qCsOyoDANAYCkLsBg4UGTYsXQCAxh358lRyiFMAAJpCQYjdE0+IXHxxugAAjXvt/aMy7IH96RIAAHYUhNidPFl3uNP3OCoJgKZt3vuJzNnyTroEAIAdBaEUXHWVyF13pQsAYLf0md/L2pcPpUsAANhREErB4sUi3bunCwBgN2nDW/Lim4fTJQAA7CgIpeDDD+s+ZqQfNwKARvSreV3e+eR4ugQAgB0FoVT8+McimzenCwDQ0IlTZ6T3gj3pEgAAjaMglIpRo0T69k0XAKChA4eOScWqfekSAACNoyCUipdeEvn+99MFAGjo2f2fydRNB9MlAAAaR0EoJd/8pshrr6ULAFBv5QsfJAEAoDkUhFJy7bUi06enCwBQT989eOr1z9IlAAAaR0EoJQ8+KHLJJekCANTT/Q90PwQAAJpDQSgln39ed7hT/QkAefQIRnokIwAAmkNBKDXt24vcf3+6AACSfPfBbfexfxIAoDgUhFKj+yDovggAkNJvT9ZvUQYAoBgUhFLzxht1RzMCgNTalw/J0md+ny4BANA0CkIp0u9DeO65dAFA1s3b+o5s2vNxugQAQNMoCKVo4MC6b1YGgFrDHtgvr71/NF0CAKBpFIRStHmzyMUXpwsAsu66RXvlyJen0iUAAJpGQXDwL7fOjyKX/HqufP7HX5GyPndYzyeEFJ/YfXr0pPRZvDddAgCgeRQEBzpZ+OKBUVHk5D/9rRy/tbv1PEJIcSmFgrDn3S9k9NoD6VLc8osbISQbgR8UBAf6H9U2iQgxx2/umpQE23mEkOJSCoPTxt0fyYJt76ZLcdPfx8/n7SaEZCQUBH8oCA5iKghHFg+SM//xP8gXq0ZYzyeENJ9SGJwWPvmerN/5UboUNwoCIdkKBcEfCoKDmAqC5vT3viXHxvzCeh4hpPmUwuA0fv2b8tuDn6dLcaMgEJKtUBD8oSA4iK0gfPnzn8iJbv/Leh4hpPmUwuB045JX5KMvTqRLcaMgEJKtUBD8oSA4iK0gHJ3cR05/88+t5xFCmk/sg5Me2vSahb9Ll+JHQSAkW6Eg+ENBcBBbQdCc+dqfypF5v7GeRwhpOrEPTvs/PCpD7t+fLsWPgkBItkJB8IeC4CDGgnDy0h/Kl9d3tJ5HCGk6sQ9OW175RGZufjtdih8FgZBshYLgDwXBQYwF4digXnLqH75rPY8Q0nRiH5yWbX9f1uz4MF2KHwWBkGyFguAPBcFBjAXhyH1Dk8Od6k/b+YSQxhP74HT7xrfk+QOH06X4URAIyVYoCP5QEBzEWBA0p/7u23JsyFXW8wghjSf2walfzety8OPj6VL8KAiEZCsUBH8oCA5iLQi6D4Lui2A7jxDSeGIenE6dPiNXV+9JfpYKCgIh2QoFwR8KgoNYC8LR2bcmRzOynUcIaTwxD076zsGAFa+nS6WBgkBItkJB8IeC4CDWgqDR70M4Ou1X1vMIIfbEPDg9u/+zZB+EUkJBICRboSD4Q0FwEHNBOFH+4+SblW3nEULsiXlwuv/FD5OjGJUSCgIh2QoFwR8KgoOYC8Kxkb3l1H//K+t5hBB7Yh6cZjx2ULa99mm6VBooCIRkKxQEfygIDmIuCF+sGlF3uNPFg+znE0LOSsyDU8Wqfck3KZcSCgIh2QoFwR8KgoOoC0JtTv7of8jxW7tbzyOEnJ2YB6feC/bIsROn06XSQEEgJFuhIPhDQXAQe0E4fnNXOfnPf2c9jxBydmIdnD44fEJuWvpqulQ6KAiEZCsUBH8oCA5iLwhHFg5IPmakHzeynU8IaZhYB6cX3zws49e/mS6VDgoCIdkKBcEfCoKD2AuC5vT3viXHqq61nkcIaZhYB6f1Oz+SxU+/ly6VDgoCIdkKBcEfCoKDUigIJ3r+i5zo9r+s5xFCGibWwWnBtndl4+6P0qXSQUEgJFuhIPhDQXBQCgXh6OQ+cvqv/sJ6HiGkYWIdnIY9sF/2vPtFulQ6KAiEZCsUBH8oCA5KoSBozlz4VTky7zfW8wgh9Yl1cOqzeK8cPnYqXSodFARCshUKgj8UBAelUhBOXvpDOX5DZ+t5hJD6xDg4aTG4btHedKm0UBAIyVYoCP5QEByUSkE4PqCnnPqH79Yt1wxLvmW58DKEkDgLgn60SD9iVIpKpSD0mvOydL/jKelS9bB0HvmAdKxcIT+tWCqX9b87eYz6U5f1dD2/85j10m3Kk8n1bLdHSKmGguAPBcFBqRSEo3feLPJH/05O/eP3kp/sk0CIPTEOTpv2fJzspFyKYi4IPWftkPJxG6XD0OVySd+7pPvIFfLLWZul4p7tMnbVThn/wG6ZumFfLrqsp+v5ernuo1bJT36zQC4fvEy6jH1Yes58yXo/hJRSKAj+UBAcxF4Qvrz60qQMyAW1v/a8nPynv7VenpCsJ8bBSQ9vuvblQ+lSaYmxIFwxbbt0HrlGLuu/SK6f/kgy8Z+75W2pfuJd58zf+nZy/RtmPprcXqcRq2tv/xnr/RJSCqEg+ENBcBB7QdDvPzjzH/7orILwZe921ssTkvXEODjpF6TpF6WVopgKgr7C37l2At9h8FIZsPBpmbP5Leuk/1yjJWPgomekc+Uy6ThsZe39vWhdD0JiDgXBHwqCg9gLgsZWEo4Nucp6WUKynhgHp5uWviq//+zLdKm0xFIQyidulnb975Zb5m1NXvW3TfBbKnr7t1VvS+6vfPyjtfe/66z1ISTWUBD8oSA4KIWCoNGdlOUP/yBXEI4sHGC9HCFZT2yD07ETp6X3gj3pUukJvyDsSt416DZihdz+0KvWCX1rRfdb+NnoVdKhsoadmUnJhILgDwXBQakUBI0pCWf+9CvW8wkh8RWE/R8elYpV+9Kl0hNyQdBJue6AfPXEta3+rkFT0f0cdD2unE1JIPGHguAPBcFBKRUEzfFflcmpv/u29TxCSHwF4anXP5MZjx1Ml0pPqAVBJ+M6Ke8zY5PXcmCi69F+0NLa9dphXV9CYgkFwR8KgoNYC8KhVaNl28IJMnvqVBkwdpZcN3yeXFFRLR37V8v//t+jk5+6rKfr+dOmTJMtd01Irme7vVJMU8+R/t55jrKZ2AanZdvfl5UvfJAulZ5QC4IepejaqRusk3Vf0XcSOg5bZV1fQmIJBcEfCoKDmArCwZqxMmfqNLl66Hy5rG+1dBlyj3QZ85CUT3pcut/xdHJoPD3qhYku6+l6fnK5oUvlktuqpdeQaplxxzQ5sHys9X5iDs8RaS4xDE6fHj2Z/kvk9o1vybP7P0uXSk+IBaHr5K3SpXLZOR+6tLWi72SUD18u5RM2W9ebkBhCQfCHguAghoLw0pJxUlE1W9r3WyBdRq1JJrW95u60/uE1n13J9cvGrE1ur9+Y2bL97vHW+40pPEek2MQwOFU9dECuWfi7ZN+D6xbtlXlb30lKwoFDx+TEqTPppUpDaAVBD2WqRw+avPYV6yTdd3THZV2/Hne+YF1/QkIPBcEfCoKDkAuCvnrdt3Zy2mHgIikb/2iLH8VCJ9DlEx6TzhWL5KaRs2XffVXW9Qg5PEfENTEMTpv3fmL9/6iFgYLQuimr2iA3ztpsnZyHklvnbZVOo9ZZ15+Q0ENB8IeC4CDUglAza7J06F8t5VUP1/5BtfYxsHdJ13EbkvtbcucU+Wy1fZ1CC88ROZfEMDgdPnZKrq7e0+D/oB7qVN9BKDVhFYRd0m7g4uRVetvEPJTM2PSGXNpv0Xm8S0qIv1AQ/KEgOAitIOjEU18R7zh4qfS483nrH1drRT+T33novdJn+Jygd9TlOSLnk1gGp9FrDzT4v7d+50fpOaUlpILQbco26Tl2tXVSHlqumrA22XfK9jgICTkUBH8oCA5CKgg64bymcq50Gb6y9o/I3zdndh39QLIe768cY11Pn+E5IuebWAYnLQTm/9v49W+mp5aeti4I93foI+N/Pcd6Xpcx62TgomesE/LQMnjpc9JpxBrr4yAk5FAQ/KEgOAilIOhEM5n4jn6w9g/I/9fq63r0rJgv764IZwLMc0RaIrEMTh99cSL5f9Zn8d7k36WqrQvCnu/9Y+0oeYEc+Nb3ZeY14+TqvP2WOg1fJSNqXrJOyENL1erdcvmQ5Q0eGyExhILgDwXBQSgFQY/A03nkA9Y/Jl/pMupBuWXUHOv6+gjPEWmJxDQ4jXzwDXn+wOF0qTT5KggmH134F7K4xwC5ZsZz0n7QPcEevagwup+E7odge4yEhBwKgj8UBAchFIS1cydJx4rFAe5wtks6DV4iy2febl3vtgzPEWmpxDQ4vfPJ8fRfpau5gtB75otyy7iNuVRUrpTRfRfmMuP6iTLn38bkUlN2k6zsdGMS/TjRlh+V5fLsDy+TI1/5kwYFweTT//T15EsmQ99BOT9tXa4IaYlQEPyhIDjwXRD0MJ16ZJyfTX/O+ofkO7pTrq7fa8v8Hd6T54i0ZBicztEnn4i88UZ9nntOZPPm+ixeLHLXXXWZN09k2LD69O0r0rt3fTp2FPnnf07y0je/J+9842/kg6/9ZS75E/dTf/CHDc47+JffTd4FMNFJf34J0FJgCoKWhfzyMLXPlOSjRYW3r9e7cdIW3kEgpA3CNtgfCoID3wVh5tRpUjYm7ONZdxv3sIyfNN26/m0RniPSkol+cPrww4YT9SeeqJ+kP/JI/SRdM316w4n6Lbc0nKi3b5+bqCf567+uzze+0WAinST//O/XTrTzr1tW1vC2Bw5seN/561VTk1vnW6/oJwOGr27wLkH+fgEtHS0D5vE8//c/kX6j1ubO030QRq34rXVCHlrYB4HEGgqCP7VbPhTLZ0HQw3V2HlCdvAJt+yMKJT1n7ZDL+1XLx/e3/WE9eY5IS6fFBqeDB+sn6a+9Vj9J16xf33BCPGVKw8lynz4NJ9P5E+2LL244Ef/61+sn6Cb555/PRP3eexuu90svNSwfx1r/exf092H7m2qtaEHQdx5GDlh81nnJUYwWP2udkIeWofe+wFGMSJShIPhDQXDgsyA8tmCClFUus/4BhZZuw2tk9ZzJ1sfRmuE5Is3lyLzf5HL0zpvl2Jhf1GdQLzl+c9dcvrz6Ull4cef6yfK119ZPpK+6quFE+6KLGk7Ev/rVsyfq3/xm/fnf+U7D6+vHaCKaqPvS1gXhtjHrradruk7eKr2q1lgn5KHlXyeulbIJj1kfByEhh4LgDwXBgc+CMHriTCmPZAOvX8gzpGqW9XGcb3Qid+ri/0eO3nHjWefxHEWSmmENJ+q1v8v8ifrxAT0bTtR//hM50fNf6vKzf5aTP/n7+vzof8ipH/xNLqf/6i/kzH/5s1zkj/7dWRP1M1/709z5p//rf25w/VP/+L0Gt3/fD9s1nKjr5+XNRF0/R58/UdfP2edP1D//PN1yoKW0dUFoOnXfpDx9437rpDyUzHz0QN03KbfiR7EIaa1QEPyhIDjwWRBuGDkv+eZO2x9QaLli2jPSu3Ke9XGcb3QCaSZ6hUWB58gtR+4b2nCiPrlPw4n6rd3rJ+q/KqufpGvKf9xgIq2/i/yJ9ulv/nnDiXreBN3kzIVfLXqirveXf/+6Prl1q13P/PXWx5H/uPRx2h5/MWFwCktYBWG3dBn7sNw853HrxDyU3DZ/m3Qcpd8HY38MhIQctsH+1I7UKJbPgtB1ULhH5ilMcqSeftXWx3G+0Qlg4UTTFIVYn6Mjiwc1mNAeq7q2fsI7snf9RFhzQ+cGE+UTnX/UYCJ96h++22CinT9J1wl54XOnyZ+oa/Kvr7eXf/t6f/n3r+uTv3659a7N0QnXN3hcR5YOPuv3GXoYnMISWkHoOfMladf/brlj/evWybnv6Lsbun6xbBcJKQzbYH9qZwgols+CoPcd+s63+Uk+u22ZjLZm9D5jfo6YqIcXBqewhFYQNOUTN0v58OUyf+vb1km6r+j6XDF6lZSN22Rdb0JiCNtgf2pnJiiWz4LAOwh10Qlv/qRac/pv/muygynPEWnpMDiFJcSCoOk4YpVcP/0R60TdV/SjTx0qa6zrS0gsYRvsDwXBgc+CUPf5+ietf0Chpa32QTDFwJzHc0RaOgxOYQm1IFw5e4e0H7RU+szYZJ2st3V0vwNdHz2ksm19CYklbIP9oSA48FkQ9Ag9sRymruvkLa16FKPCYmDCc0RaOgxOYQm1IGi0JHQYutx7SdD7pxyQUgnbYH8oCA58FoQN8ydKeeV91j+g0NJtRI2snN1Kx/ivGWY/vTY8R6Slw+AUlpALgkYPJaolode4NW1++NMZm95Ivu9gQacb5YOv/aXs/c4/yG9/8OPky97Wtvs3WVl7+px/GyNT+0yR0X0XJt8KfRPfjUACD9tgfygIDnwWhPpvCX7J+kcUSvRVNP2W4EOrfH6TMs8RaZkwOIUl9IJQl11SPn6TXNZ/kfRf+JR1Mt/SGbRke3J/ZVUbkvu/cdKW5EvetAjcfuO0pBhoQdCioIVBi4MWCC0Sx/74K3L4qxcm/9ZvjX7xf/6/yWXW/PRaqSm7Kbmu3obelt6m3rb9cbddqm6bL7eM22g9j5RW2Ab7Q0Fw4LMgaKZOmS5dqx6y/hGFkm7jNkjVxOnW9W+L8ByRlgyDU1jiKAh1SQ5EUFmTHOFo8NLnZO6Wlj3KkR6lqHLZC9JtxAq5fMhy6XHn89b1KDY64R4wfHVSBGZeM04WXFWZlIqN/18v2XZxp6Q8HPjW95MiceoP/lA+/U9fl9//l/87Of35v/9JUiru79BHlnW9NSkVk26amdyW3m6fFv5+ms3/1F1O/Ps/TgpM74iOXEfcwzbYHwqCA98F4cDysdKhf3XtQPCC9Q/Jd/SVe12/3fdUWde/LcJzRFoyDE5hiakgmOiBEzoOWyXtBtwtv5q9WW5/6FXrhL/Y6PV/PXdrcnsdh6309uWQOjHXyf+wQUuTV/S1FCzuMSApFVoWnv3hZUl5eOcbf5OUCT2wxEcX/kWyrKfr+Xo5vfzS7n2T64//9RwZOWBxcrvXTX3Ger+ap/7x8uT2NFpS9Hq2y5H4wzbYHwqCA98FQVMza7J0qlhS+4ez66w/JL/ZJWVDl8qi6VOs692W4TkiLRUGp7DEWBBM9EWLzqPXJTsQX9pvofQav0Zum/+EjF65U6pW75apG/bJzEcPJCVAf+qynq7n963ellxeP0bUbuBi6TRq3Xm/Y+Aj+k6CTv71nQV9h0FLgb4LoB9n0rKgH28yH3068pU/SQqA/vvgX343KRVaDPTdAz3NFAQTfRdDb9t2vyTesA32h4LgIISCoLlt9GzpHNhX55dXrZdfjphtXV8f4TkiLREGp7DEXBDyo0cY6jp5a1IYOlaukMsHL5P2A5ckj89El/X0jpU1SSHQI59l8chEuiO1+ejTjOsnyryrR+TKQ2He+/NvyZU33FFbpO6Wn1YsTZ7bziMfkM5j1ifv5OhO5Lb7IOGGbbA/FAQH+h/VNolo67y7YoxcUVFdO7iEMQHuOm5Dsj4Ha8Za19dHeI5IS4TBKSz6+7D9fZHSj5aj8nEbk6NEffC1/9agGOz62x/L3QNmyrR1rybvvGjGP7Bbxq7aKRX3bJdfztos3Uetkp/8ZkFSurqMfTj4g1mQurAN9oeC4CCUgqDRCfA1lXOli+cJsN5/z4r5QU58eY7I+YbBKSwUhOzlimnbpfPIuo9X6bdV68RfS4Hu1/B41z6yaOnTZ+2n0Vh0x269/g0zH01ur9OI1cmXVtrul4QRtsH+UBAchFQQNHqYzGQCXLm8zV8NSV7NGb4iuf/3V46xrl8I4Tki5xMGp7BQELIT3V53rp3Adxi8VAYsfFrmbH4rN9G/v2KmLNj8ZoPJv2v0qFIDFz0jnSuXJTt761GnbOtB/IZtsD8UBAehFQSNHvv/rulT5af9FkjXCZusf2AtnW6TNif3N2vq1OT+besVUniOyLmGwSksFIRspHziZmnX/265Zd7W5FV/2wS/paK3f1v1tuT+ysc/Wnv/oR3cItthG+wPBcFBiAXBZN99VXL98LnSefBiKZ/0uPSau9P6x3bu2ZXsVFc2ZIn8onKu7L03vo/L8BwR1zA4hYWCUOrZlbxroN/tcL6Hg3WN7rfws9Grku+uYGfmcMI22B8KgoOQC4LJlrsmyM2j5iavXpeNfei8D4Wn1y+veji5vV+NnCOPLZhgvd+YwnNEig2DU1goCKUbnZTrDshXT1zb6u8aNBXdz0HX48rZlIQQwjbYHwqCgxgKgslry6pk/KQZ0n1QtVzWt1q6Dr9PulRtlO53PJXslKWft7xydt0h8/SnLuvpen5Z1SPJ5dvrBHrgfBk78c6SfDWc54g0FwansFAQSjM6GddJeZ8Zm7yWAxNdD/2+CrP9J/7CNtgfCoKDmApCfvToORvmT5SJk6bLdcPnSa8h1cmkVh+PiS7r6Xq+TpofmjcpU0fd4TkitujvHeHQ34dtEkHijh6l6NqpG6yTdV/RdxL0G7Bt60vaLmyD/aEgOND/qLZJBCGkNMPgFBYKQulF99vqUrksOaqQbaLuK/pORvnw5VI+YbN1vUnbhG2wPxQEBxQEQrIVBqewUBBKK3ooUz160OS1r1gn6b6jOy7r+vW48wXr+pPWD9tgfygIDigIhGQrDE5hoSCUVsqqNsiNszZbJ+eh5NZ5W6XTqHXW9SetH7bB/lAQHFAQCMlWGJzCQkEopeySdgMXJ6/S2ybmoWTGpjfk0n6LWuGw2KSYsA32h4LggIJASLbC4BQWCkLppNuUbdJz7GrrpDy0XDVhbfLdObbHQVo3bIP9oSA4oCAQkq0wOIWFglA66TJmnQxc9Ix1Qh5aBi99TjqNWGN9HKR1wzbYHwqCAwoCIdkKg1NYKAilk07DV8mImpesE/LQUrV6t1w+ZLn1cZDWDdtgfygIDigIhGQrDE5hoSCUTtoPuifYoxcVRveT0P0QbI+DtG7YBvtDQXBAQSAkW2FwCgsFoXSiv8vQd1DOD//3/IRtsD8UBAf6H9U2iSCElGYYnMLCJK10wjsIpJiwDfaHguCAgkBItsLgFBYKQulE90EYteK31gl5aGEfBH9hG+wPBcEBBYGQbIXBKSwUhNJJchSjxc9aJ+ShZei9L3AUI09hG+wPBcEBBYGQbIXBKSwUhNJJ18lbpVfVGuuEPLT868S1UjbhMevjIK0btsH+UBAcUBAIyVYYnMJCQSil1H2T8vSN+62T8lAy89EDdd+kPOdly2MgrR22wf5QEBxQEAjJVhicwkJBKK10Gfuw3DzncevEPJTcNn+bdBz1oHX9SeuHbbA/FAQHFARCshUGp7BQEEorPWe+JO363y13rH/dOjn3HX13Q9fvZ9Ofs64/af2wDfaHguCAgkBItsLgFBYKQumlfOJmKR++XOZvfds6SfcVXZ8rRq+SsnGbrOtN2iZsg/2hIDigIBCSrTA4hYWCUJrpNOJ++T/TH7FO1H3l5rlbpEPlCuv6krYL22B/KAgOKAiEZCsMTmGhIJRmrpy9Q346aKn0mbHJOllv69xWvS1Zn56zdljXl7Rd2Ab7Q0FwQEEgJFthcAoLBaF0oyWhw9Dl3kuC3n97ykEwYRvsDwXBAQWBkGyFwSksFITSjh5KVEtCr3Fr2vzwpzM2vZF834He/5WzOaRpKGEb7A8FwQEFgZBshcEpLBSELGSXlI/fJJf1XyT9Fz5lncy3dAYt2Z7cX1nVhuT+7etFfIRtsD8UBAcUBEKyFQansFAQspOeM1+UDpU1yRGOBi99TuZuadmjHOlRiiqXvSDdRq6Qy4cslx53Pm9dD+I3bIP9oSA4oCAQkq0wOIWFgpC9dJvypHQavkraD7hbfjX7cbn9oVetE/5io9e/Zd5WaVd7ex2Hray9/W3W+yVhhG2wPxQEBxQEQrIVBqewUBCymx53viBdRq+Tn1bcK5f2Wyg/H79Gbpv/hIxeuVOqVu+WqRv2ycxHDyQlQH/qsp6u5/etri0BtZfXjxG1H7hEOtfeDu8YxBG2wf5QEBxQEAjJVhicwkJBIBo9wlDXyVuTiX7HyhVy+eBlycRf/3+Y6LKe3rGyRjqNWld7+S0cmSjCsA32h4LgQP+j2iYRhJDSDINTWPT3YZtEEEJKM2yD/aEgOKAgEJKtMDiFhYJASLbCNtgfCoIDCgIh2QqDU1goCIRkK2yD/aEgOKAgEJKtMDiFhYJASLbCNtgfCoIDCgIh2QqDU1goCIRkK2yD/aEgOKAgEJKtMDiFhYJASLbCNtgfCoIDCgIh2QqDU1goCIRkK2yD/aEgOIi1IBxaNVq2LZwgs6dOlQFjZ8l1w+fJFRXV0rF/dfKY9Kcu6+l6/rQp02TLXROS69luj5CshMEpLKVSEHrNeVm63/GUdKl6WDqPfCA5lv9PK5bKZf3vTh6j/tRlPV3P7zxmffKNwno92+0RUqphG+wPBcGB/ke1TSJCzMGasTJn6jS5emjtYNO3WroMuUe6jHlIyic9XjswPS1XTHtGes58MRdd1tP1/ORyQ5fKJbdVS68h1TLjjmlyYPlY6/0QUsphcApLzAVBv6SrfNxG6TB0uVzS9y7pPnKF/HLWZqm4Z7uMXbVTxj9Q923AJrqsp+v5ernuo1bJT36zIPnyry5jH67dbr9kvR9CSilsg/2hIDiIoSC8tGScVFTNlvb9FkiXUWuSiX+vuTutf3jNZ1dy/bIxa5Pb6zdmtmy/e7z1fgkpxTA4hSXGgnDFtO3SeeQauaz/Irl++iPJxH/ulrel+ol3nTN/69vJ9W+Y+Whye51GrE620bb7JaQUwjbYHwqCg5ALgr7C37d2At9h4CIpG/9oi78VrSWjfMJj0rlikdw0crbsu6/Kuh6ElFIYnMISU0HQV/g7107gOwxeKgMWPi1zNr9lnfSfa7RkDFz0jHSuXCYdh61M3gm2rQchMYdtsD8UBAehFoSaWZOlQ/9qKa96uPYPatdZf2Atm13SddyG5P6W3DlFPlttXydCSiEMTmGJpSCUT9ws7frfLbfM25q86m+b4LdU9PZvq96W3F/5+Edr77+1xwBC2i5sg/2hIDgIrSDo5FzfNeg4eKn0uPN56x9Xa0Vfreo89F7pM3wOOzOTkg2DU1jCLwi7kncNuo1YIbc/9Kp1Qt9a0f0WfjZ6lXSorGFnZlIyYRvsDwXBQUgFQSfl11TOlS7DV9b+Efl7xajr6AeS9Xh/5RjrehIScxicwhJyQdBJue6AfPXEta3+rkFT0f0cdD2unE1JIPGHbbA/FAQHoRQEnYwn5WD0g7V/QP7fTtb16FkxX95dQUkgpRUGp7CEWhB0Mq6T8j4zNnktBya6Hu0HLa1drx3W9SUklrAN9oeC4CCUgqBHKdJjY9v+mHyly6gH5ZZRc6zrS0isYXAKS6gFQY9SdO3UDdbJuq/oOwkdh62yri8hsYRtsD8UBAchFIS1cydJx4rF53Ho0tbKLuk0eIksn3m7db0JiTEMTmEJsSB0nbxVulQuO+dDl7ZW9J2M8uHLpXzCZut6ExJD2Ab7Q0Fw4Lsg6KFM9ehBP5v+nPUPyXd0x2Vdv9eWcQhUUhphcApLaAVBD2WqRw+avPYV6yTdd3THZV2/Hne+YF1/QkIP22B/KAgOfBeEmVOnSdmYddY/olDSbdzDMn7SdOv6ExJbGJzCElpBKKvaIDfO2mydnIeSW+dtlU6jwh43CGksbIP9oSA48FkQ9JCmnQdUB/9lOD1n7ZDL+1XLx/dz6FMSfxicwhJWQdgl7QYuTl6lt03MQ8mMTW/Ipf0WBfixVEKaD9tgfygIDnwWhMcWTJCyymXWP6DQ0m14jayeM9n6OAiJKQxOYQmpIHSbsk16jl1tnZSHlqsmrJXySY9bHwchIYdtsD8UBAc+C8LoiTOlfMJj1j+g0KID0ZCqWdbHQUhMYXAKi4+CsP6Sq2VlpxvluqnPNDi9y5h1MnDRM9YJeWgZvPQ56TRiTYP1JySGsA32h4LgwGdBuGHkvOQVK9sfUGi5Ytoz0rtynvVxEBJTGJzC4qMgbPlRWe1IeYEc+cqfNCgKnYavkhE1L1kn5KGlavVuuXzI8rMeGyGhh22wPxQEBz4LQtdB4R69qDDJ0Yz6VVsfByExhcEpLD4LgokpCt1unRfs0YsKo/tJ6H4ItsdHSMhhG+wPBcGBz4Kg9x36Dsr5WXhx5waDKiGElFI+/b++KvdOj2MfBI2PckXI+YaC4E/tlg7F8lkQeAeBkLYPg1NYfExyC99BOPUHf5ic9os+t/MOAiGtHLbB/lAQHPgsCHX7IDxp/QMKLeyDQEolDE5h8VkQTDG4bcz65HTdB2HUit9aJ+ShhX0QSKxhG+wPBcGBz4KgRzEqi+QoRl0nb+EoRqQkwuAUFh8FYfM/dW9QDEySoxgtftY6IQ8tQ+99gaMYkSjDNtgfCoIDnwVhw/yJUl55n/UPKLR0G1EjK2fzPQgk/jA4hcVHQejdyL5fXSdvlV5Va6wT8tDyrxPXRvMCEyH5YRvsDwXBgc+CUP9Nyi9Z/4hCyZWz675J+dAqvkmZxB8Gp7D4KAiNp+6blKdv3G+dlIeSmY8eqPsm5TkvWx4DIWGHbbA/FAQHPguCZuqU6dK16iHrH1Eo6TZug1RNnG5df0JiC4NTWMIqCLuly9iH5eY5j1sn5qHktvnbpOOoB63rT0joYRvsDwXBge+CcGD5WOnQv1p63PmC9Q/Jd/TdDV2/3fdUWdefkNjC4BSW0AqCbvPa9b9b7lj/unVy7jv67oauXyxHwCOkMGyD/aEgOPBdEDQ1syZLp8FLav9wdp31h+Q3u6Rs6FJZNH2Kdb0JiTEMTmEJrSBoyidulvLhy2X+1retk3Rf0fW5YvQqKRu3ybrehMQQtsH+UBAchFAQNL8ZM1u6BPaWsX706ZcjZlvXl5BYw+AUlhALgqbjiFVy/fRHrBN1X9GPPnWorLGuLyGxhG2wPxQEB6EUhHdXjJGeFfOly+gwSoLud6Drc7BmrHV9CYk1DE5hCbUg6MEZ2g9aKn1mbLJO1ts6ut+Brk/PWTus60tILGEb7A8FwUEoBUGjJeGayrneS4LeP+WAlGoYnMISakHQaEnoMHS595Kg9085IKUStsH+UBAchFQQNHoo0aQkVC5v88Of6uBTPnxFcv/vrxxjXT9CYg+DU1hCLggaPZSoloRe49a0+eFPZ2x6I/m+A73/K2dzSFNSGmEb7A8FwUFoBUGj349w1/Sp8tN+C6TrhLbZGa3bpM3J/c2aOjW5f9t6EVIKYXAKS+gFoS67pHz8Jrms/yLpv/Ap62S+pTNoyfbk/sqqNiT3b18vQuIL22B/KAgOQiwIJvvuq5Lrh8+VzoMXS/mkx6XX3J3WP7Zzz67km0PLhyyRX1TOlb338pEiUvphcApLHAWhLj1nvpjsJKxHOBq89DmZu6Vlj3KkRymqXPaCdBuxQi4fslx63Pm8dT0IiTlsg/2hIDgIuSCYbLlrgvx61JzkFf7ysQ+d96Ch1+9atV4ur729m0bOkccWTLDeLyGlGAansMRUEEy6TXlSOg5bJe0G3C2/mr1Zbn/oVeuEv9jo9X89d2tyex2Hray9/W3W+yWkFMI22B8KgoMYCoLJa8uqZOKk6fKzimpp17daug2/T7pUbZTudzwlV0x7Jnl1S3eq0z9A/anLerqeX1b1SHL59vqxpYHzZXzt7fCOAcliGJzCEmNBMNEvuOw8el2yA/Gl/RZKr/Fr5Lb5T8jolTulavVumbphn8x89EBSAvSnLuvpen7f6m3J5fVjRO0GLpZOo9bxjgHJRNgG+0NBcBBTQciPHmFow/yJSWG4bvg8+fmQ+VJWO/HXx2Oiy3q6nj9+0gx5aN4kjkxEMh8Gp7Do78M2iYgtepAH/cimFoaOlSvk8sHLpP3AJQ22ybqsp3esrEkKQdfJWzgyEclc2Ab7Q0FwoP9RbZMIQkhphsEpLPr7sE0iCCGlGbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHFAQCMlWGJzCQkEgJFthG+wPBcEBBYGQbIXBKSwUBEKyFbbB/lAQHOh/VEJItoJw6O/DNokghJRm2Ab7Q0EAAEQhv7gRQrIR+EFBAAAAAJBDQQAAAACQQ0EAAAAAkENBAAAAAJBDQQAAAACQQ0EAAAAAkENBAAAAAJBDQQAAAACQQ0EAAAAAkENBAAAAAJBDQQAAAACQQ0EAAAAAkENBAAAAAJBDQQAAAACQQ0EAAAAAkENBAAAAAJBDQQAAAACQQ0EAAAAAkENBAAAAAJBDQQAAAACQQ0EAAAAAkENBAAAAAJBDQQAAAACQQ0EAAAAAkENBAAAAAJBDQQAAAACQQ0EAAAAAkENBAAAAAJBDQQAAAACQQ0EAAAAAkENBAAAAAJBDQQAAAACQQ0EAAAAAkENBAAAAAJBDQQAAAACQQ0EAAAAAkBL5/wFtF3cnBSEEfQAAAABJRU5ErkJggg==) +> +> G1 的很多开销都是源自 Remembered Set，例如，它通常约占用 Heap 大小的 20% 或更高，这可是非常可观的比例。并且，我们进行对象复制的时候，因为需要扫描和更改 Card Table 的信息，这个速度影响了复制的速度，进而影响暂停时间。 + + + + + +### 🎯 Java 11 的 ZGC 和 Java12 的 Shenandoah 了解过吗 + +ZGC 即 Z Garbage Collector（Z 垃圾收集器，Z 有 Zero 的意思，主要作者是 Oracle 的 Per Liden），这是一款低停顿、高并发，基于小堆块（region）、不分代的增量压缩式垃圾收集器，平均 GC 耗时不到 2 毫秒，最坏情况下的暂停时间也不超过 10 毫秒。 + +像 G1 和 ZGC 之类的现代 GC 算法，只要空闲的堆内存足够多，基本上不触发 FullGC。 + +所以很多时候，只要条件允许，加内存才是最有效的解决办法。 + +既然低延迟是 ZGC 的核心看点，而 JVM 低延迟的关键是 GC 暂停时间，那么我们来看看有哪些方法可以减少 GC 暂停时间： + +- 使用多线程“并行”清理堆内存，充分利用多核 CPU 的资源； +- 使用“分阶段”的方式运行 GC 任务，把暂停时间打散； +- 使用“增量”方式进行处理，每次 GC 只处理一部分堆内存（小堆块，region）； +- 让 GC 与业务线程“并发”执行，例如增加并发标记，并发清除等阶段，从而把暂停时间控制在非常短的范围内（目前来说还是必须使用少量的 STW 暂停，比如根对象的扫描，最终标记等阶段）； +- 完全不进行堆内存整理，比如 Golang 的 GC 就采用这种方式（题外话）。 + + + +### 🎯 ZGC 在 G1 基础上做了哪些改进？ + +在 G1 的基础上，它做了如下 7 点改进（JDK 11 开始引入） + +1. 动态调整大小的 Region + + ![ZGC](https://img.starfish.ink/jvm/ZGC.png) + +2. 不分代，干掉了 RSets + + G1 中每个 Region 需要借助额外的 RSets 来记录“谁引用了我”，占用了额外的内存空间，每次对象移动时，RSets 也需要更新，会产生开销。 + +3. 带颜色的指针 Colored Pointer + + ![](https://img.starfish.ink/jvm/cc38abc5ba683cf300bdd6e89f426212.png) + + 这里的指针类似 Java 中的引用，意为对某块虚拟内存的引用。ZGC 采用了64位指针（注：目前只支持 linux 64 位系统），将 42-45 这 4 个 bit 位置赋予了不同含义，即所谓的颜色标志位，也换为指针的 metadata。 + + - finalizable 位：仅 finalizer（类比 C++ 中的析构函数）可访问； + + - remap 位：指向对象当前（最新）的内存地址，参考下面提到的relocation； + + - marked0 && marked1 位：用于标志可达对象。 + + 这 4 个标志位，同一时刻只会有 1 个位置是 1。每当指针对应的内存数据发生变化，比如内存被移动，颜色会发生变化。 + +4. 读屏障 Load Barrier + + 传统 GC 做标记时，为了防止其他线程在标记期间修改对象，通常会简单的 STW。而 ZGC 有了 Colored Pointer 后，引入了所谓的“读屏障”。 + + 当指针引用的内存正被移动时，指针上的颜色就会变化，ZGC 会先把指针更新成最新状态，然后再返回（你可以回想下 Java 中的 volatile 关键字，有异曲同工之妙）。这样仅读取该指针时，可能会略有开销，而不用将整个 heap STW。 + +5. 重定位 Relocation + + 如下图，在标记过程中，先从 Roots 对象找到了直接关联的下级对象 1，2，4。 + + ![39.jpg](https://s0.lgstatic.com/i/image6/M01/56/2A/Cgp9HWEspqWAAHS6AAAcoge7Pv8625.jpg) + + 然后继续向下层标记，找到了 5，8 对象， 此时已经可以判定 3，6，7 为垃圾对象。 + + ![40.jpg](https://s0.lgstatic.com/i/image6/M01/56/2A/Cgp9HWEsprCAZrtCAAAeqKPoHXw488.jpg) + + 如果按常规思路，一般会将 8 从最右侧的 Region，移动或复制到中间的 Region，然后再将中间 Region 的 3 干掉，最后再对中间 Region 做压缩 compact 整理。 + + ![41.jpg](https://s0.lgstatic.com/i/image6/M01/56/2A/Cgp9HWEspr6AMxziAAAeYmBBjoU744.jpg) + + 但 ZGC 做得更高明，它直接将 4，5 复制到了一个空的新 Region 就完事了，然后中间的 2 个 Region 直接废弃，或理解为“释放”，作为下次回收的“新” Region。这样的好处是避免了中间 Region 的 compact 整理过程。 + + ![42.jpg](https://s0.lgstatic.com/i/image6/M01/56/2A/Cgp9HWEspsuAFh42AAAsCgyJwEk919.jpg) + + + 最后，指针重新调整为正确的指向（即：remap），而且上一阶段的 remap 与下一阶段的mark是混在一起处理的，相对更高效。 + + 【 Remap 的流程图】 + + ![43.jpg](https://s0.lgstatic.com/i/image6/M01/56/2A/Cgp9HWEsptuABwmwAAA8njSoyvA729.jpg) + +6. 多重映射 Multi-Mapping + +7. 支持 NUMA 架构 + + + +## 四、监控和调优 + +### 🎯 说说你知道的几种主要的 JVM 参数 + +**思路：** 可以说一下堆栈配置相关的，垃圾收集器相关的，还有一下辅助信息相关的。 + +#### 1）堆栈配置相关 + +``` +java -Xmx3550m -Xms3550m -Xmn2g -Xss128k +-XX:MaxPermSize=16m -XX:NewRatio=4 -XX:SurvivorRatio=4 -XX:MaxTenuringThreshold=0 +``` + +**-Xmx3550m：** 最大堆大小为3550m。 + +**-Xms3550m：** 设置初始堆大小为3550m。 + +**-Xmn2g：** 设置年轻代大小为2g。 + +**-Xss128k：** 每个线程的堆栈大小为128k。 + +**-XX:MaxPermSize：** 设置持久代大小为16m + +**-XX:NewRatio=4:** 设置年轻代（包括Eden和两个Survivor区）与年老代的比值（除去持久代）。 + +**-XX:SurvivorRatio=4：** 设置年轻代中Eden区与Survivor区的大小比值。设置为4，则两个Survivor区与一个Eden区的比值为2:4，一个Survivor区占整个年轻代的1/6 + +**-XX:MaxTenuringThreshold=0：** 设置垃圾最大年龄。如果设置为0的话，则年轻代对象不经过Survivor区，直接进入年老代。 + +#### 2）垃圾收集器相关 + +``` +-XX:+UseParallelGC +-XX:ParallelGCThreads=20 +-XX:+UseConcMarkSweepGC +-XX:CMSFullGCsBeforeCompaction=5 +-XX:+UseCMSCompactAtFullCollection： +``` + +**-XX:+UseParallelGC：** 选择垃圾收集器为并行收集器。 + +**-XX:ParallelGCThreads=20：** 配置并行收集器的线程数 + +**-XX:+UseConcMarkSweepGC：** 设置年老代为并发收集。 + +**-XX:CMSFullGCsBeforeCompaction**：由于并发收集器不对内存空间进行压缩、整理，所以运行一段时间以后会产生“碎片”，使得运行效率降低。此值设置运行多少次GC以后对内存空间进行压缩、整理。 + +**-XX:+UseCMSCompactAtFullCollection：** 打开对年老代的压缩。可能会影响性能，但是可以消除碎片 + +#### 3）辅助信息相关 + +``` +-XX:+PrintGC +-XX:+PrintGCDetails +``` + +**-XX:+PrintGC 输出形式:** + +[GC 118250K->113543K(130112K), 0.0094143 secs] [Full GC 121376K->10414K(130112K), 0.0650971 secs] + +**-XX:+PrintGCDetails 输出形式:** + +[GC [DefNew: 8614K->781K(9088K), 0.0123035 secs] 118250K->113543K(130112K), 0.0124633 secs] [GC [DefNew: 8614K->8614K(9088K), 0.0000665 secs][Tenured: 112761K->10414K(121024K), 0.0433488 secs] 121376K->10414K(130112K), 0.0436268 secs + + + +### 🎯 你平时工作用过的 JVM 常用基本配置参数有哪些？ + +``` +# 设置堆内存 +-Xmx4g -Xms4g +# 指定 GC 算法 +-XX:+UseG1GC -XX:MaxGCPauseMillis=50 +# 指定 GC 并行线程数 +-XX:ParallelGCThreads=4 +# 打印 GC 日志 +-XX:+PrintGCDetails -XX:+PrintGCDateStamps +# 指定 GC 日志文件 +-Xloggc:gc.log +# 指定 Meta 区的最大值 +-XX:MaxMetaspaceSize=2g +# 设置单个线程栈的大小 +-Xss1m +# 指定堆内存溢出时自动进行 Dump +-XX:+HeapDumpOnOutOfMemoryError +-XX:HeapDumpPath=/usr/local/ +``` + + + +### 🎯 你说你做过 JVM 调优和参数配置，请问如何盘点查看 JVM 系统默认值？ + +**JVM参数类型** + +| **类型** | **示例** | **作用** | | +| ------------ | --------------------- | --------------------------------------------- | ------------------------------------------------ | +| **标配参数** | `-version`, `-help` | 所有 JVM 必须实现的通用功能，无性能影响 | | +| **X 参数** | `-Xint`, `-Xcomp` | 控制 JVM 运行模式（解释、编译、混合） | | +| **XX 参数** | `-XX:+PrintGCDetails` | 调优核心参数，分为 **Boolean** 和 **KV 类型** | -xx:+ 或者 - 某个属性值（+表示开启，- 表示关闭） | + +**常用工具** + +- **`jinfo`**：查看或修改运行中 JVM 的参数。 +- **`java` 命令参数**：直接打印默认或修改后的参数。 +- **Runtime API**：通过 Java 代码获取内存信息 + +这些都是命令级别的查看，我们如何在程序运行中查看 + +```java +long totalMemory = Runtime.getRuntime().totalMemory(); +long maxMemory = Runtime.getRuntime().maxMemory(); + +System.out.println("total_memory(-xms)="+totalMemory+"字节，" +(totalMemory/(double)1024/1024)+"MB"); +System.out.println("max_memory(-xmx)="+maxMemory+"字节，" +(maxMemory/(double)1024/1024)+"MB"); +``` + +### 🎯 盘点家底查看 JVM 默认值 + +- -XX:+PrintFlagsInitial + + - 主要查看初始默认值 + + - java -XX:+PrintFlagsInitial + + - java -XX:+PrintFlagsInitial -version + + **等号前有冒号** := 说明 jvm 参数有人为修改过或者 JVM加载修改 + + false 说明是Boolean 类型 参数，数字说明是 KV 类型参数 + +- -XX:+PrintFlagsFinal + + - 主要查看修改更新 + - java -XX:+PrintFlagsFinal + - java -XX:+PrintFlagsFinal -version + - 运行java命令的同时打印出参数 java -XX:+PrintFlagsFinal -XX:MetaspaceSize=512m Hello.java + +- -XX:+PrintCommondLineFlags + + - 打印命令行参数 + - java -XX:+PrintCommondLineFlags -version + - 可以方便的看到垃圾回收器 + + + +### 有没有遇到过oom的问题？线上怎么排查的？ + +我们的排查流程一般是分几步： + +1. **先确认 OOM 类型** + - 堆内存不足（Java heap space） + - 元空间不足（Metaspace） + - 直接内存不足（Direct buffer memory） + - 线程无法创建（unable to create new native thread） + 不同类型的 OOM，对应的排查思路不一样。 +2. **收集现场信息** + - 开启 `-XX:+HeapDumpOnOutOfMemoryError`，让 JVM 在 OOM 时自动生成 `heap dump` 文件。 + - 收集 GC 日志（`-Xloggc`），确认是内存泄漏还是配置不合理。 +3. **分析 Dump 文件** + - 用 **MAT（Memory Analyzer Tool）** 或 **VisualVM** 打开 `heap dump`，看哪些对象占用内存最多。 + - 结合 `dominator tree` 分析是否存在引用链未释放。 +4. **定位代码问题** + - 如果是内存泄漏，通常是 **集合类持有过多对象（Map/Queue 没清理）**，或者 **缓存没有淘汰策略**。 + - 如果是元空间 OOM，可能是动态生成类太多（如反射、CGLIB 代理）。 + - 如果是 DirectMemory，通常是 Netty/ByteBuffer 没释放。 +5. **优化措施** + - 调整 JVM 参数（堆大小、元空间大小、直接内存大小）。 + - 增加对象回收策略（比如缓存加 LRU）。 + - 避免无界队列，控制线程池大小。 + - 必要时进行代码重构。 + + + +### 🎯 能写几个 OOM 代码不？ + +- java.lang.StackOverflowError + + ```java + public class StackOverflowErrorDemo { + + public static void main(String[] args) { + stackoverflowError(); + } + + private static void stackoverflowError() { + stackoverflowError(); + } + } + ``` + +- java.lang.OutOfMemoryError: Java heap space + + - new个大对象,就会出现 + + ```java + //JVM参数：-Xmx12m + static final int SIZE = 2 * 1024 * 1024; + + public static void main(String[] a) { + int[] i = new int[SIZE]; + } + ``` + + + +### 🎯 谈谈你的 GC 调优思路？ + +> 谈到调优，这一定是针对特定场景、特定目的的事情， 对于 GC 调优来说，首先就需要清楚调优的目标是什么？从性能的角度看，通常关注三个方面，**内存占用**（footprint）、**延时**（latency）和**吞吐量**（throughput），大多数情况下调优会侧重于其中一个或者两个方面的目标，很少有情况可以兼顾三个不同的角度。当然，除了上面通常的三个方面，也可能需要考虑其他 GC 相关的场景，例如，OOM 也可能与不合理的 GC 相关参数有关；或者，应用启动速度方面的需求，GC 也会是个考虑的方面。 +> +> - **延迟（Latency）：** 也可以理解为最大停顿时间，即垃圾收集过程中一次 STW 的最长时间，越短越好，一定程度上可以接受频次的增大，GC 技术的主要发展方向。 +> - **吞吐量（Throughput）：** 应用系统的生命周期内，由于 GC 线程会占用 Mutator 当前可用的 CPU 时钟周期，吞吐量即为 Mutator 有效花费的时间占系统总运行时间的百分比，例如系统运行了 100 min，GC 耗时 1 min，则系统吞吐量为 99%，吞吐量优先的收集器可以接受较长的停顿。 +> +> 基本的调优思路可以总结为： +> +> - 理解应用需求和问题，确定调优目标。假设，我们开发了一个应用服务，但发现偶尔会出现性能抖动，出现较长的服务停顿。评估用户可接受的响应时间和业务量，将目标简化为，希望 GC 暂停尽量控制在 200ms 以内，并且保证一定标准的吞吐量。 +> - 掌握 JVM 和 GC 的状态，定位具体的问题，确定真的有 GC 调优的必要。具体有很多方法，比如，通过 jstat 等工具查看 GC 等相关状态，可以开启 GC 日志，或者是利用操作系统提供的诊断工具等。例如，通过追踪 GC 日志，就可以查找是不是 GC 在特定时间发生了长时间的暂停，进而导致了应用响应不及时。 +> - 这里需要思考，选择的 GC 类型是否符合我们的应用特征，如果是，具体问题表现在哪里，是 Minor GC 过长，还是 Mixed GC 等出现异常停顿情况；如果不是，考虑切换到什么类型，如 CMS 和 G1 都是更侧重于低延迟的 GC 选项。 +> - 通过分析确定具体调整的参数或者软硬件配置。 +> - 验证是否达到调优目标，如果达到目标，即可以考虑结束调优；否则，重复完成分析、调整、验证这个过程。 +> + +一、调优核心目标与基本原则 + +1. **核心目标** + - 延迟优化：减少GC导致的STW时间（如Young GC < 50ms，Full GC < 1s） + - 吞吐量提升：确保GC时间占比低于5%（如GC吞吐量 > 95%） + - **内存利用率**：减少内存碎片，避免OOM和频繁扩容 +2. **基本原则** + - 先诊断后调优：通过GC日志和监控工具定位问题，避免盲目调整参数 + - 代码优先原则：优化对象分配模式（如减少大对象、避免内存泄漏）比参数调整更有效 + - 场景适配：根据应用类型选择GC算法（低延迟选G1/ZGC，高吞吐选Parallel GC） + +二、调优方法论与实施步骤 + +1. **数据采集与问题定位** + + - 监控工具： + - `jstat -gcutil `：实时查看各代内存使用率及GC次数/耗时 + - `jmap -histo`：分析堆内存对象分布，定位大对象或内存泄漏 + - GC日志分析：通过 `-Xlog:gc*` 生成日志，使用GCeasy或G1Viewer解析 + + - 关键指标： + - Young GC频率：高于10秒/次需优化新生代分配 + - 晋升速率：若单次Young GC后老年代增长>5%，需调整年龄阈值 + +2. **内存模型优化** + + - 分代策略调整： + - 新生代扩容：若Young GC频繁（如<30秒/次），增大 `-Xmn`（不超过堆的60%） + - Survivor区平衡：通过 `-XX:SurvivorRatio` 调整Eden与Survivor比例（默认8:1:1），避免动态年龄判定过早触发 + + - 大对象控制： + - 设置 `-XX:PretenureSizeThreshold=4M`，避免大对象直接进入老年代引发碎片 + +3. **GC算法选择与参数调优** + + | **GC类型** | **适用场景** | **调优参数示例** | **优化目标** | + | ------------ | ----------------------- | ------------------------------------------------------------ | ---------------- | + | **G1** | 低延迟、大堆内存（>8G） | `-XX:MaxGCPauseMillis=200`（目标停顿时间）`-XX:InitiatingHeapOccupancyPercent=45`（并发标记阈值） 5 | 减少Mixed GC频率 | + | **Parallel** | 批处理、高吞吐量 | `-XX:ParallelGCThreads=CPU核数`（并行线程数）`-XX:GCTimeRatio=9`（GC/应用时间比） 5 | 最大化吞吐量 | + | **ZGC** | 超低延迟（<10ms） | `-XX:ZAllocationSpikeTolerance=5`（分配速率容忍度）`-Xmx32G`（堆≤32G） | 亚秒级停顿 | + +4. **关键参数调优策略** + + - 晋升阈值优化：降低 `-XX:MaxTenuringThreshold` （默认15→5），加速长生命周期对象进入老年代 + + - 堆稳定性保障：设置 `-Xms=-Xmx` 避免堆动态扩容，配合 `-XX:+AlwaysPreTouch` 预分配物理内存 + + - 元空间控制： 限制 `-XX:MaxMetaspaceSize=512M` ，防止类加载器泄漏导致Full GC + +5. **代码级优化** + + - **对象池化**：复用高频率创建对象（如数据库连接、线程） + + - 软引用控制：通过 SoftReference 缓存大对象，在内存紧张时优先释放 + + - **并发数据结构**：使用`ConcurrentHashMap`替代`synchronized`集合，减少锁竞争导致的临时对象激增 + +三、调优效果验证与持续监控 + +1. AB测试验证： + + - 对比调优前后的GC暂停时间分布（如P99延迟下降30%） + + - 监控吞吐量变化（如QPS提升20%+） + +2. 监控体系构建： + + - **时序数据库**：Prometheus采集`jvm_gc_pause_seconds`指标 + + - 告警规则：Full GC次数>1次/小时或STW时间>1秒触发告警 + +四、典型场景调优案例 + +> 案例1：电商秒杀系统（低延迟场景） +> +> - **问题**：高峰期Young GC暂停时间从50ms飙升至200ms +> +> - 调优： +> +> 1. 切换至G1回收器，设置`MaxGCPauseMillis=100` +> 2. 增大Eden区（`-Xmn=4G`），降低动态年龄判定触发频率 +> 3. 代码优化：预加载热点商品数据至堆外缓存 +> +> - 效果：Young GC平均暂停时间降至80ms，Full GC完全消除 +> +> +> +> 案例2：大数据计算引擎（高吞吐场景） +> +> - **问题**：Full GC导致每小时任务超时 +> - 调优： +> 1. 采用Parallel GC，设置`-XX:GCTimeRatio=19`（GC时间占比≤5%） +> 2. 限制大对象分配：`PretenureSizeThreshold=8M` +> 3. 启用`-XX:+UseLargePages`减少TLB缺失 +> - 效果：任务完成时间缩短40%，CPU利用率提升15% + +五、调优工具推荐 + +| **工具类型** | **推荐工具** | **核心功能** | +| ------------ | -------------------- | --------------------------------- | +| 日志分析 | GCeasy、G1Viewer | 可视化GC暂停分布、内存泄漏检测 9 | +| 实时监控 | Prometheus + Grafana | 可视化JVM内存、GC频率与耗时趋势 8 | +| 堆内存分析 | Eclipse MAT | 对象引用链分析，定位内存泄漏 7 | + +总结：调优需避免的误区 + +1. **参数过度调整**：如盲目设置`-Xmx=物理内存80%`导致系统Swap +2. 忽略代码优化：90%的GC问题源于代码缺陷而非参数配置 +3. 算法选择错配：在32G以上堆内存使用CMS导致并发模式失败 + +通过系统化的数据采集、场景化参数调整和代码级优化，可显著提升应用性能。建议结合具体业务特征选择调优路径，并通过持续监控验证长期效果。 + + + +### 🎯 怎么打出线程栈信息。 + +**思路：** 可以说一下 jps，top ，jstack这几个命令，再配合一次排查线上问题进行解答。 + +- 输入 jps，获得进程号。 +- top -Hp pid 获取本进程中所有线程的CPU耗时性能 +- jstack pid命令查看当前java进程的堆栈状态 +- 或者 jstack -l > /tmp/output.txt 把堆栈信息打到一个txt文件。 +- 可以使用 fastthread 堆栈定位，[fastthread.io/](http://fastthread.io/) + + + +### 🎯 如何查看JVM的内存使用情况? + +> JVM 内存可以通过 JDK 自带工具（jstat、jmap、jcmd）在命令行查看，也可以用 jconsole、visualvm 图形化分析。如果线上系统，我会用 Arthas 或结合 Prometheus + Grafana 做实时监控。在代码里，还可以通过 JMX 的 MemoryMXBean 获取堆内存使用情况。 + +一、命令行工具 + +1. **`jps` + `jstat` 组合** + - **功能**：快速定位 Java 进程并监控内存与 GC 状态。 + - **操作步骤**： + 1. 查看 Java 进程 ID：`jps -l` + 2. 监控堆内存使用（示例 PID=1234）：`jstat -gc 1234 1s 5 # 每1秒刷新，共5次` + - 输出关键字段： + - `EC/EU`：Eden 区容量/使用量 + - `OC/OU`：老年代容量/使用量 + - `YGC/YGCT`：Young GC 次数/耗时 + - `FGC/FGCT`：Full GC 次数/耗时 + - **适用场景**：实时监控内存分配与回收效率 + +2. **`jmap` 堆内存分析** + + - **功能**：生成堆转储文件或直接查看内存分布。 + + - 常用命令： + - 查看堆内存配置：`jmap -heap 1234` + - 生成堆转储文件（用于后续分析内存泄漏）：`jmap -dump:format=b,file=heap.hprof 1234` + - 统计对象直方图：`jmap -histo 1234 | head -20 # 显示前20个占用内存最多的类` + - **适用场景**：排查内存溢出或对象分布异常 + +3. **`jinfo` 参数查看** + + - **功能**：查看或修改运行中 JVM 的参数。 + + - 示例： + + ```bash + jinfo -flags 1234 # 显示所有参数 + jinfo -flag MaxHeapSize 1234 # 查看堆最大内存 + ``` + + - **适用场景**：验证内存参数配置是否生效 + + + +### 🎯 怎么查看服务器默认的垃圾收集器是哪个？生产上如何配置垃圾收集器？谈谈你对垃圾收集器的理解？ + +```sh +java -XX:+PrintCommandLineFlags -version +``` + +使用 G1 垃圾收集 + +```sh +java -XX:+UseG1GC -jar yourapp.jar +``` + +垃圾收集器是JVM用来自动管理内存的重要组成部分。它们的主要任务是识别和回收不再使用的对象，释放内存资源。以下是对垃圾收集器的一些关键理解： + +1. **自动化内存管理**：自动化内存管理减少了内存泄漏和野指针的风险。 +2. **性能影响**：不同的垃圾收集器对应用程序性能的影响不同。选择合适的垃圾收集器可以显著提高应用程序性能。 +3. **算法多样性**：存在多种垃圾收集算法，每种算法都有其特定的使用场景和优缺点。 +4. **与应用程序特性匹配**：选择垃圾收集器时，需要考虑应用程序的特性，如对象生命周期、响应时间要求、吞吐量需求等。 +5. **资源消耗**：垃圾收集器可能会消耗额外的CPU资源来执行垃圾回收任务。 +6. **内存分配策略**：垃圾收集器通常与特定的内存分配策略（如TLAB）结合使用，以优化内存分配性能。 +7. **并发与增量收集**：现代垃圾收集器通常支持并发或增量收集，以减少GC引起的应用程序停顿。 +8. **可配置性**：大多数垃圾收集器都可以通过JVM参数进行配置，以适应不同的性能需求。 +9. **持续发展**：垃圾收集器和算法随着JVM的更新而不断发展，新的垃圾收集器（如ZGC、Shenandoah等）提供了更低的延迟和更好的性能。 + +通过合理选择和配置垃圾收集器，可以显著提高Java应用程序的性能和稳定性。 + + + +### 🎯 生产环境服务器变慢，诊断思路和性能评估谈谈？ + +诊断思路 + +1. **初步排查** + - **确认问题**：确认问题的存在和范围。检查是否所有服务变慢还是只有特定的服务。 + - **收集信息**：从用户反馈、日志、监控系统中收集详细的症状描述和时间信息。 +2. **硬件资源检查** + - **CPU 使用率**：检查 CPU 使用率是否过高，是否存在 CPU 瓶颈。 + - **内存使用**：检查内存使用情况，是否存在内存泄漏或不合理的内存占用。 + - **磁盘 I/O**：检查磁盘 I/O 是否成为瓶颈，是否有大量的读写操作。 + - **网络流量**：检查网络带宽和延迟，是否存在网络瓶颈。 +3. **系统级别检查** + - **系统日志**：查看操作系统日志（如 `/var/log/syslog` 或 `/var/log/messages`），检查是否有硬件故障、驱动问题等。 + - **资源使用**：使用 `top`、`htop`、`vmstat`、`iostat` 等工具查看实时资源使用情况。 + - **进程检查**：查看是否有异常进程占用大量资源。 +4. **应用级别检查** + - **应用日志**：检查应用日志，查看是否有异常错误、超时或其他提示信息。 + - **线程和堆栈**：检查应用的线程和堆栈信息，使用工具如 `jstack` 查看 Java 应用的线程状态。 + - **垃圾回收**：如果是 Java 应用，检查垃圾回收日志，查看是否存在频繁的 GC 停顿。 + - **数据库性能**：检查数据库性能，查看是否存在慢查询、大量锁等待等问题。 +5. **网络检查** + - **网络延迟**：使用 `ping`、`traceroute` 等工具检查网络延迟和路径。 + - **带宽占用**：使用 `iftop`、`netstat` 等工具查看网络带宽占用情况。 +6. **外部依赖** + - **第三方服务**：检查依赖的第三方服务是否正常运行，是否存在响应缓慢的问题。 + - **API 调用**：检查外部 API 调用的响应时间，是否存在延迟。 + +**性能评估** + +1. **基准测试** + - **负载测试**：使用工具如 `JMeter`、`Gatling` 对应用进行负载测试，评估其在高负载下的表现。 + - **压力测试**：模拟高并发用户访问，评估应用的最大承载能力。 +2. **性能监控** + - **监控工具**：使用监控工具如 Prometheus、Grafana、Nagios、Zabbix 等，对服务器的 CPU、内存、磁盘 I/O、网络等进行持续监控。 + - **应用性能监控**：使用 APM 工具如 New Relic、AppDynamics、Dynatrace 等，监控应用的性能指标，如响应时间、错误率、吞吐量等。 +3. **日志分析** + - **集中日志管理**：使用 ELK（Elasticsearch, Logstash, Kibana）或 Splunk 等工具集中管理和分析日志。 + - **日志分析**：分析应用日志和系统日志，查找异常和错误信息。 +4. **性能优化** + - **代码优化**：分析应用代码，查找性能瓶颈，如低效算法、频繁的 I/O 操作等。 + - **数据库优化**：优化数据库查询，添加索引、优化 SQL 语句、调整数据库配置等。 + - **缓存策略**：使用缓存（如 Redis、Memcached）减轻数据库压力，提升响应速度。 + - **负载均衡**：使用负载均衡（如 Nginx、HAProxy）分发流量，减轻单点压力。 + - **集群和分布式**：将应用部署在集群或分布式环境中，提高系统的扩展性和可靠性。 + +诊断服务器变慢需要系统的方法，从硬件资源、系统级别、应用级别、网络和外部依赖等多个方面进行排查。性能评估则需要通过基准测试、性能监控、日志分析等手段全面了解系统性能，并通过优化代码、数据库、缓存策略等措施提升系统性能。定期的性能评估和优化可以帮助维护系统的稳定性和高效性。 + + + +### 🎯 假设生产环境出现 CPU占用过高，请谈谈你的分析思路和定位 + +生产环境中 CPU 占用过高是常见的性能问题，需要结合系统工具和 Java 诊断工具逐步定位，我的分析思路通常分为 "快速定位 - 分层拆解 - 精准溯源" 三个阶段，结合实际案例可以这样展开： + +" 遇到 CPU 占用过高的问题，我会按照‘从系统到应用，从进程到代码’的顺序逐层排查，确保定位精准： + +1. **第一步：快速锁定目标进程**登录服务器后，先用`top`命令查看整体 CPU 使用情况（关注 % user、% sys、% idle 指标），找到 CPU 占比异常的 Java 进程（通常是 PID 对应的进程 % CPU 持续超过 80%）。比如之前处理过一个订单服务，top 显示其进程 CPU 长期维持在 150%（2 核机器），明显异常。 + + 若要更细致，可用`pidstat -p [pid] 1 5`观察进程的 CPU 使用趋势，区分是用户态（% usr）还是内核态（% sys）占用高：用户态高可能是应用代码问题（如循环计算），内核态高可能是线程切换频繁或系统调用过多。 + +2. **第二步：定位异常线程**确定进程后，用`top -H -p [pid]`查看该进程内的线程 CPU 占用，找到 % CPU 高的线程 ID（LWP 列），并转换为 16 进制（如`printf "%x\n" 1234`），因为 jstack 日志中的线程 ID 是 16 进制。 + + 接着用`jstack [pid] > stack.log`导出线程栈，在日志中搜索转换后的 16 进制 ID，分析线程状态： + + - 若线程处于`RUNNABLE`且栈信息显示在执行某段业务代码（如循环处理、复杂计算），可能是代码逻辑导致的 CPU 密集型操作 + - 若大量线程处于`BLOCKED`或`WAITING`，可能是锁竞争激烈（如 synchronized 块过大），导致线程频繁上下文切换消耗 CPU + - 若发现多个`GC Thread`或`VM Thread`占用高，可能是频繁 GC 导致（需结合 jstat 进一步验证） + + 我曾遇到过一个案例：jstack 显示多个线程卡在`HashMap.put()`，结合代码发现是在高并发下使用了非线程安全的 HashMap，导致死循环（扩容时的链表成环），CPU 瞬间飙升到 100%。 + +3. **第三步：结合业务代码溯源**根据线程栈中的类名和方法名（如`com.xxx.service.OrderService.calculatePrice()`），定位到具体代码逻辑： + + - 检查是否有死循环、递归调用失控等问题 + - 分析是否有大量重复计算、正则表达式频繁编译等 CPU 密集型操作 + - 排查是否使用了低效的数据结构（如 ArrayList 在大数据量下频繁扩容） + - 确认是否有第三方库存在性能问题（如 JSON 序列化效率低） + +4. **辅助验证手段** + + - 用`jstat -gcutil [pid] 1000 10`查看 GC 情况，若 YGC 频繁或 GC 耗时过长（如 STW 超过 100ms），可能是内存问题间接导致 CPU 高 + - 结合 APM 工具（如 SkyWalking）的调用链，查看是否有高频调用的接口存在性能瓶颈 + - 对于容器化部署，需确认容器的 CPU 配额（如 cgroups 限制）是否合理，避免资源争用 + +比如之前处理的一个商品搜索服务 CPU 过高问题，最终定位是 lucene 索引查询时的正则匹配逻辑过于复杂，在高并发下每次查询消耗大量 CPU，优化为前缀匹配后，CPU 使用率从 90% 降至 30%。 + +总结来说，CPU 过高的排查核心是 "缩小范围，精准定位"—— 从系统到进程，从线程到代码，每个环节都用数据（监控指标、日志）验证假设，避免经验主义判断。” + + + +### 🎯 如何检测 jvm 的各项指标？ + +> 对于JDK 自带的JVM 监控和性能分析工具用过哪些？你是怎么用的？ + +**1、JMX（Java Management Extensions）**： + +- JMX是Java平台的内置管理框架，用于监控和管理Java应用程序。通过JMX，可以获取JVM的运行时数据，包括内存使用情况、线程信息、垃圾回收统计等。 + + > **使用 JConsole 监控 JVM** + > + > 1. 启动 JConsole： + > - JConsole 是 JDK 自带的图形化工具，用于监控 JVM 的各种指标。 + > - 你可以在命令行中输入 `jconsole` 启动 JConsole。 + > 2. 连接到 JVM： + > - 在 JConsole 的连接窗口中选择本地进程或远程进程，然后点击连接。 + > 3. 查看性能指标： + > - 在 JConsole 中，你可以查看内存使用情况、线程活动、类加载、MBean 等 + > + > 也可以通过编程方式使用 JMX 访问 JVM 的性能指标 + > + > ```java + > public class JMXExample { + > public static void main(String[] args) throws Exception { + > MBeanServerConnection mBeanServer = ManagementFactory.getPlatformMBeanServer(); + > ObjectName memoryMXBeanName = new ObjectName(ManagementFactory.MEMORY_MXBEAN_NAME); + > MemoryMXBean memoryMXBean = ManagementFactory.newPlatformMXBeanProxy(mBeanServer, memoryMXBeanName.toString(), MemoryMXBean.class); + > + > // 获取堆内存使用情况 + > MemoryUsage heapMemoryUsage = memoryMXBean.getHeapMemoryUsage(); + > System.out.println("Heap Memory Usage: " + heapMemoryUsage); + > + > // 获取非堆内存使用情况 + > MemoryUsage nonHeapMemoryUsage = memoryMXBean.getNonHeapMemoryUsage(); + > System.out.println("Non-Heap Memory Usage: " + nonHeapMemoryUsage); + > } + > } + > ``` + +- 可以使用JMX Exporter将JVM的监控指标暴露给Prometheus进行采集和监控 + + **2、使用 Java 命令行工具** + +- `jstat` 是一个命令行工具，用于监控 JVM 的各种性能指标，如垃圾回收、类加载等 +- `jmap` 用于生成堆转储文件（heap dump）和查看堆内存的详细信息 +- `jstack` 用于生成线程快照（thread dump），可以帮助诊断线程死锁和高 CPU 使用问题 + +**3、使用第三方监控工具** + +- VisualVM 是一个强大的图形化工具，用于监控和分析 JVM 的性能。它可以监控内存使用情况、线程活动、垃圾回收等 + +- Prometheus 是一个开源监控系统，Grafana 是一个开源的时序数据可视化工具。结合使用这两个工具可以实现对 JVM 的高级监控和可视化。 + +4、使用 Spring Boot Actuator (如果使用 Spring Boot) + + + +## 🎯 高频面试题汇总 + +### 核心必考题（⭐⭐⭐） + +1. **说说JVM的内存结构？** +2. **堆内存的结构是怎样的？** +3. **什么是垃圾回收？为什么要进行垃圾回收？** +4. **如何判断对象是否可以被回收？** +5. **常见的垃圾回收算法有哪些？** +6. **分代垃圾回收是如何工作的？** +7. **类加载的完整过程是怎样的？** +8. **什么是双亲委派模型？为什么要使用双亲委派？** + +### 进阶深入题（⭐⭐） + +9. **对象在堆内存中是如何分配的？** +10. **什么是内存溢出？常见的OOM有哪些？** +11. **什么是内存泄漏？如何排查？** +12. **常见的垃圾收集器有哪些？各有什么特点？** +13. **如何自定义类加载器？** +14. **常用的JVM参数有哪些？** +15. **如何监控JVM性能？** +16. **JVM调优的目标是什么？如何调优？** + +### 应用实践题（⭐） + +17. **如何优化JVM内存使用？** +18. **如何优化垃圾回收性能？** +19. **如何排查JVM性能问题？** +20. **生产环境JVM调优的最佳实践？** +21. **JVM内存模型与Java内存模型的区别？** +22. **如何分析堆转储文件？** + +--- + +## 📝 面试话术模板 + +### 回答框架 + +``` +1. 概念定义（30秒） + - 简要说明是什么 + +2. 核心原理（60秒） + - 底层实现机制 + - 关键数据结构 + +3. 特性分析（30秒） + - 优缺点对比 + - 使用场景 + +4. 实战应用（30秒） + - 什么情况下使用 + - 注意事项 + +5. 深入扩展（可选） + - 源码细节 + - 性能优化 + - 最佳实践 +``` -### JVM内存模型的相关知识了解多少，比如重排序，内存屏障，happen-before，主内存，工作内存。 +### 关键话术 +- **JVM内存结构**："JVM内存主要分为堆内存、方法区、虚拟机栈等，其中堆内存是GC的主要区域..." +- **垃圾回收**："垃圾回收通过可达性分析算法判断对象是否存活，使用分代收集算法提高效率..." +- **类加载**："类加载分为加载、验证、准备、解析、初始化五个阶段，使用双亲委派模型保证安全性..." +- **性能调优**："JVM调优需要先分析性能瓶颈，然后选择合适的垃圾收集器和参数，最后验证调优效果..." diff --git a/docs/interview/Java-Basics-FAQ.md b/docs/interview/Java-Basics-FAQ.md new file mode 100644 index 0000000000..6a277da6b0 --- /dev/null +++ b/docs/interview/Java-Basics-FAQ.md @@ -0,0 +1,4817 @@ +--- +title: Java 基础八股文 +date: 2024-08-31 +tags: + - Java + - Interview +categories: Interview +--- + +![](https://img.starfish.ink/common/faq-banner.png) + +> Java基础是所有Java开发者的**根基**，也是面试官考察的**重中之重**。从面向对象三大特性到泛型擦除机制，从异常处理到反射原理，每一个知识点都可能成为面试的关键。本文档将**最常考的Java基础**整理成**标准话术**，让你在面试中对答如流！ + +### 🔥 为什么Java基础如此重要？ + +- **📈 面试必考**：95%的Java岗都会深挖基础 +- **🧠 思维体现**：体现你对Java语言的深度理解 +- **💼 工作基础**：日常开发中无处不在的核心概念 +- **🎓 进阶前提**：框架、中间件都建立在基础之上 + +--- + +## 🗺️ 知识导航 + +### 🏷️ 核心知识分类 + +1. **🔥 面向对象编程**：封装、继承、多态、抽象类、接口、内部类 +2. **📊 数据类型体系**：基本类型、包装类、自动装箱拆箱、类型转换 +3. **🔤 字符串处理**：String、StringBuilder、StringBuffer、字符串常量池 +4. **⚠️ 异常处理机制**：异常体系、try-catch-finally、异常传播、自定义异常 +5. **🎯 泛型机制**：泛型语法、类型擦除、通配符、泛型边界 +6. **🪞 反射机制**：Class对象、动态代理、注解处理、反射性能 +7. **🔗 Java引用类型**：强引用、软引用、弱引用、虚引用、WeakHashMap、ThreadLocal原理 +8. **📁 IO流体系**：字节流、字符流、缓冲流、NIO基础 +9. **🆕 Java新特性**：Lambda表达式、Stream API、Optional、函数式接口 + + + +### 🔑 面试话术模板 + +| **问题类型** | **回答框架** | **关键要点** | **深入扩展** | +| ------------ | ----------------------------------- | ------------------ | ------------------ | +| **概念解释** | 定义→特点→应用场景→示例 | 准确定义，突出特点 | 底层原理，源码分析 | +| **对比分析** | 相同点→不同点→使用场景→选择建议 | 多维度对比 | 性能差异，实际应用 | +| **原理解析** | 背景→实现机制→执行流程→注意事项 | 图解流程 | 源码实现，JVM层面 | +| **应用实践** | 问题场景→解决方案→代码实现→优化建议 | 实际案例 | 最佳实践，踩坑经验 | + +--- + +## 🔥 一、面向对象编程（OOP核心） + +> **核心思想**：将现实世界的事物抽象为对象，通过类和对象来组织代码，实现高内聚、低耦合的程序设计。 + +### 🎯 JDK、JRE、JVM的区别？ + +"JDK、JRE、JVM是Java技术体系的三个核心组件： + +**JVM（Java Virtual Machine）**： + +- Java虚拟机，是整个Java实现跨平台的最核心部分 +- 负责字节码的解释执行，提供内存管理、垃圾回收等功能 +- 不同操作系统有对应的JVM实现，但对上层透明 + +**JRE（Java Runtime Environment）**： + +- Java运行时环境，包含JVM和Java核心类库 +- 是运行Java程序所必需的最小环境 +- 包括JVM标准实现及Java基础类库（如java.lang、java.util等） + +**JDK（Java Development Kit）**： + +- Java开发工具包，包含JRE以及开发工具 +- 提供编译器javac、调试器jdb、文档生成器javadoc等 +- 开发Java程序必须安装JDK，而运行Java程序只需JRE + +简单说：JDK包含JRE，JRE包含JVM，开发用JDK，运行用JRE，执行靠JVM。" + +**💻 代码示例**： + +```java +public class JavaEnvironmentDemo { + + public static void main(String[] args) { + // 获取Java环境信息 + System.out.println("=== Java Environment Info ==="); + + // JVM相关信息 + System.out.println("JVM Name: " + System.getProperty("java.vm.name")); + System.out.println("JVM Version: " + System.getProperty("java.vm.version")); + System.out.println("JVM Vendor: " + System.getProperty("java.vm.vendor")); + + // JRE相关信息 + System.out.println("Java Version: " + System.getProperty("java.version")); + System.out.println("Java Home: " + System.getProperty("java.home")); + + // 运行时环境信息 + Runtime runtime = Runtime.getRuntime(); + System.out.println("Available Processors: " + runtime.availableProcessors()); + System.out.println("Max Memory: " + runtime.maxMemory() / 1024 / 1024 + " MB"); + System.out.println("Total Memory: " + runtime.totalMemory() / 1024 / 1024 + " MB"); + System.out.println("Free Memory: " + runtime.freeMemory() / 1024 / 1024 + " MB"); + + // 操作系统信息 + System.out.println("OS Name: " + System.getProperty("os.name")); + System.out.println("OS Version: " + System.getProperty("os.version")); + System.out.println("OS Architecture: " + System.getProperty("os.arch")); + + // Class Path信息 + System.out.println("Class Path: " + System.getProperty("java.class.path")); + } +} + +/* + * JDK、JRE、JVM的层次关系： + * + * ┌─────────────── JDK ───────────────┐ + * │ ┌─────────── JRE ──────────┐ │ + * │ │ ┌─── JVM ───┐ │ │ + * │ │ │ │ │ │ + * │ │ │ 字节码 │ Java核心 │ 开发 │ + * │ │ │ 执行引擎 │ 类库 │ 工具 │ + * │ │ │ 内存管理 │ (rt.jar) │ │ + * │ │ │ 垃圾回收 │ │ │ + * │ │ └───────────┘ │ │ + * │ └─────────────────────────┘ │ + * └───────────────────────────────────┘ + * + * 开发工具包括： + * - javac: 编译器 + * - java: 运行器 + * - javadoc: 文档生成 + * - jar: 打包工具 + * - jdb: 调试器 + * - jconsole: 监控工具 + */ +``` + +### 🎯 Class和Object的区别？ + +"Class和Object有本质区别： + +**Class（类）**： + +- 是对象的模板或蓝图，定义了对象的属性和行为 +- 在Java中有两重含义：一是我们定义的Java类，二是java.lang.Class类 +- java.lang.Class保存了Java类的元信息，主要用于反射操作 +- 通过Class可以获取类的字段、方法、构造器等元数据 + +**Object（对象）**： + +- 是类的具体实例，是类在内存中的具体表现 +- java.lang.Object是所有Java类的根父类，包括Class类本身 +- 提供了基础方法如equals()、hashCode()、toString()等 + +**关系总结**：类是对象的模板，对象是类的实例；Class类用于反射获取类信息，Object类是所有类的基类。" + +**💻 代码示例**： + +```java +import java.lang.reflect.*; + +public class ClassObjectDemo { + + public static void main(String[] args) throws Exception { + // 1. 演示Class和Object的关系 + demonstrateClassObjectRelation(); + + // 2. Class类的使用（反射） + demonstrateClassUsage(); + + // 3. Object类的方法 + demonstrateObjectMethods(); + } + + // Class和Object关系演示 + public static void demonstrateClassObjectRelation() { + System.out.println("=== Class and Object Relation ==="); + + // 创建Student对象 + Student student1 = new Student("Alice", 20); + Student student2 = new Student("Bob", 22); + + // 获取Class对象的三种方式 + Class clazz1 = student1.getClass(); // 通过对象 + Class clazz2 = Student.class; // 通过类字面量 + try { + Class clazz3 = Class.forName("Student"); // 通过类名 + System.out.println("All Class objects are same: " + + (clazz1 == clazz2 && clazz2 == clazz3)); + } catch (ClassNotFoundException e) { + e.printStackTrace(); + } + + // Class对象的信息 + System.out.println("Class name: " + clazz1.getName()); + System.out.println("Simple name: " + clazz1.getSimpleName()); + System.out.println("Package: " + clazz1.getPackage()); + + // Object的基本信息 + System.out.println("student1 class: " + student1.getClass().getSimpleName()); + System.out.println("student1 hash: " + student1.hashCode()); + System.out.println("student1 string: " + student1.toString()); + } + + // Class类的反射使用 + public static void demonstrateClassUsage() { + System.out.println("\n=== Class Usage (Reflection) ==="); + + try { + Class studentClass = Student.class; + + // 获取构造器 + Constructor[] constructors = studentClass.getConstructors(); + System.out.println("Constructors count: " + constructors.length); + + // 获取字段 + Field[] fields = studentClass.getDeclaredFields(); + System.out.println("Fields:"); + for (Field field : fields) { + System.out.println(" " + field.getType().getSimpleName() + " " + field.getName()); + } + + // 获取方法 + Method[] methods = studentClass.getDeclaredMethods(); + System.out.println("Methods:"); + for (Method method : methods) { + System.out.println(" " + method.getName() + "()"); + } + + // 通过反射创建对象 + Constructor constructor = studentClass.getConstructor(String.class, int.class); + Object studentObj = constructor.newInstance("Charlie", 21); + System.out.println("Created by reflection: " + studentObj); + + } catch (Exception e) { + e.printStackTrace(); + } + } + + // Object类方法演示 + public static void demonstrateObjectMethods() { + System.out.println("\n=== Object Methods ==="); + + Student s1 = new Student("David", 23); + Student s2 = new Student("David", 23); + Student s3 = s1; + + // equals方法 + System.out.println("s1.equals(s2): " + s1.equals(s2)); // true（重写了equals） + System.out.println("s1.equals(s3): " + s1.equals(s3)); // true + System.out.println("s1 == s3: " + (s1 == s3)); // true（同一对象） + + // hashCode方法 + System.out.println("s1.hashCode(): " + s1.hashCode()); + System.out.println("s2.hashCode(): " + s2.hashCode()); + System.out.println("HashCodes equal: " + (s1.hashCode() == s2.hashCode())); + + // toString方法 + System.out.println("s1.toString(): " + s1.toString()); + + // getClass方法 + System.out.println("s1.getClass(): " + s1.getClass()); + + // 演示Object其他方法 + try { + // wait/notify需要在synchronized块中 + synchronized (s1) { + System.out.println("Object methods like wait/notify require synchronization"); + } + } catch (Exception e) { + e.printStackTrace(); + } + } +} + +// 示例学生类 +class Student { + private String name; + private int age; + + public Student(String name, int age) { + this.name = name; + this.age = age; + } + + // 重写Object的方法 + @Override + public boolean equals(Object obj) { + if (this == obj) return true; + if (obj == null || getClass() != obj.getClass()) return false; + Student student = (Student) obj; + return age == student.age && java.util.Objects.equals(name, student.name); + } + + @Override + public int hashCode() { + return java.util.Objects.hash(name, age); + } + + @Override + public String toString() { + return "Student{name='" + name + "', age=" + age + "}"; + } + + // getter方法 + public String getName() { return name; } + public int getAge() { return age; } +} +``` + +### 🎯 请解释面向对象的三大特性 + +"面向对象编程有三大核心特性：**封装、继承、多态**。 + +**封装（Encapsulation）**：将数据和操作数据的方法绑定在一起，通过访问修饰符控制外部对内部数据的访问，隐藏实现细节，只暴露必要的接口。这样可以保护数据安全，降低模块间耦合度。 + +**继承（Inheritance）**：子类可以继承父类的属性和方法，实现代码复用。Java支持单继承，一个类只能继承一个父类，但可以通过接口实现多重继承的效果。继承体现了'is-a'的关系。 + +**多态（Polymorphism）**：同一个接口可以有多种不同的实现方式，在运行时动态决定调用哪个具体实现。Java中多态通过方法重写和接口实现来体现，运行时绑定确保调用正确的方法。" + +**💻 代码示例**： + +```java +// 封装示例 +public class Account { + private double balance; // 私有字段，外部不能直接访问 + + public void deposit(double amount) { // 公共方法，提供安全的操作接口 + if (amount > 0) { + balance += amount; + } + } + + public double getBalance() { // getter方法，控制数据访问 + return balance; + } +} + +// 继承示例 +class Animal { + protected String name; + + public void eat() { + System.out.println(name + " is eating"); + } +} + +class Dog extends Animal { // 继承Animal类 + public void bark() { // 子类特有方法 + System.out.println(name + " is barking"); + } + + @Override + public void eat() { // 方法重写 + System.out.println(name + " is eating dog food"); + } +} + +// 多态示例 +interface Shape { + double getArea(); +} + +class Circle implements Shape { + private double radius; + + public Circle(double radius) { + this.radius = radius; + } + + @Override + public double getArea() { + return Math.PI * radius * radius; + } +} + +class Rectangle implements Shape { + private double width, height; + + public Rectangle(double width, double height) { + this.width = width; + this.height = height; + } + + @Override + public double getArea() { + return width * height; + } +} + +// 多态的使用 +public class PolymorphismDemo { + public static void printArea(Shape shape) { // 参数类型是接口 + System.out.println("Area: " + shape.getArea()); // 运行时动态绑定 + } + + public static void main(String[] args) { + Shape circle = new Circle(5); + Shape rectangle = new Rectangle(4, 6); + + printArea(circle); // 输出圆的面积 + printArea(rectangle); // 输出矩形的面积 + } +} +``` + +### 🎯 Overload和Override的区别？ + +> "Overload（重载）和Override（重写）是Java面向对象的两个重要概念： +> +> **Overload（方法重载）**： +> +> - 同一个类中多个方法名相同，但参数不同 +> - 编译时多态（静态多态），编译器根据参数决定调用哪个方法 +> - 参数必须不同：类型、个数、顺序至少一项不同 +> - 返回值类型可以相同也可以不同，但不能仅凭返回值区分 +> - 访问修饰符可以不同 +> +> **Override（方法重写）**： +> +> - 子类重新实现父类的方法 +> - 运行时多态（动态多态），根据实际对象类型决定调用哪个方法 +> - 方法签名必须完全相同：方法名、参数列表、返回值类型 +> - 访问修饰符不能更严格，但可以更宽松 +> - 不能重写static、final、private方法 +> +> **核心区别**： +> +> - 重载是水平扩展（同类多方法），重写是垂直扩展（继承层次） +> - 重载在编译时确定，重写在运行时确定 +> - 重载体现了接口的灵活性，重写体现了多态性 +> +> 记忆技巧：Overload添加功能，Override改变功能。" + +**💻 代码示例**： + +```java +public class OverloadOverrideDemo { + + public static void main(String[] args) { + // 演示方法重载 + Calculator calc = new Calculator(); + calc.demonstrateOverload(); + + System.out.println(); + + // 演示方法重写 + Animal animal = new Dog(); + animal.makeSound(); // 运行时多态 + + // 演示重写的访问控制 + demonstrateOverrideAccessControl(); + + // 演示重载的编译时解析 + demonstrateOverloadResolution(); + } + + public static void demonstrateOverrideAccessControl() { + System.out.println("=== Override访问控制演示 ==="); + + Parent parent = new Child(); + parent.protectedMethod(); // 调用子类重写的方法 + parent.publicMethod(); // 调用子类重写的方法 + } + + public static void demonstrateOverloadResolution() { + System.out.println("\n=== Overload解析演示 ==="); + + OverloadExample example = new OverloadExample(); + + // 编译时就确定调用哪个方法 + example.process(10); // 调用process(int) + example.process(10L); // 调用process(long) + example.process("hello"); // 调用process(String) + example.process(10, 20); // 调用process(int, int) + example.process(10.5); // 调用process(double) + + // 自动类型提升 + byte b = 10; + example.process(b); // byte -> int，调用process(int) + + short s = 20; + example.process(s); // short -> int，调用process(int) + + char c = 'A'; + example.process(c); // char -> int，调用process(int) + } +} + +// 方法重载示例 +class Calculator { + + public void demonstrateOverload() { + System.out.println("=== 方法重载演示 ==="); + + // 同名方法，不同参数 + System.out.println("add(2, 3) = " + add(2, 3)); + System.out.println("add(2.5, 3.7) = " + add(2.5, 3.7)); + System.out.println("add(1, 2, 3) = " + add(1, 2, 3)); + System.out.println("add(\"Hello\", \"World\") = " + add("Hello", "World")); + + // 可变参数重载 + System.out.println("sum() = " + sum()); + System.out.println("sum(1) = " + sum(1)); + System.out.println("sum(1,2,3,4,5) = " + sum(1, 2, 3, 4, 5)); + } + + // 重载方法1：两个int参数 + public int add(int a, int b) { + System.out.println("调用了add(int, int)"); + return a + b; + } + + // 重载方法2：两个double参数 + public double add(double a, double b) { + System.out.println("调用了add(double, double)"); + return a + b; + } + + // 重载方法3：三个int参数 + public int add(int a, int b, int c) { + System.out.println("调用了add(int, int, int)"); + return a + b + c; + } + + // 重载方法4：两个String参数 + public String add(String a, String b) { + System.out.println("调用了add(String, String)"); + return a + b; + } + + // 重载方法5：参数顺序不同 + public String format(String format, int value) { + System.out.println("调用了format(String, int)"); + return String.format(format, value); + } + + public String format(int value, String suffix) { + System.out.println("调用了format(int, String)"); + return value + suffix; + } + + // 可变参数重载 + public int sum(int... numbers) { + System.out.println("调用了sum(int...)，参数个数: " + numbers.length); + int result = 0; + for (int num : numbers) { + result += num; + } + return result; + } + + // 注意：这种重载会导致歧义，编译错误 + // public int sum(int[] numbers) { ... } // 与可变参数冲突 +} + +// 重载解析示例 +class OverloadExample { + + public void process(int value) { + System.out.println("处理int: " + value); + } + + public void process(long value) { + System.out.println("处理long: " + value); + } + + public void process(double value) { + System.out.println("处理double: " + value); + } + + public void process(String value) { + System.out.println("处理String: " + value); + } + + public void process(int a, int b) { + System.out.println("处理两个int: " + a + ", " + b); + } + + // 重载与继承的交互 + public void process(Object obj) { + System.out.println("处理Object: " + obj); + } + + public void process(Number num) { + System.out.println("处理Number: " + num); + } +} + +// 方法重写示例 +abstract class Animal { + protected String name; + + public Animal(String name) { + this.name = name; + } + + // 抽象方法，强制子类重写 + public abstract void makeSound(); + + // 普通方法，可以被重写 + public void eat() { + System.out.println(name + " is eating"); + } + + // final方法，不能被重写 + public final void sleep() { + System.out.println(name + " is sleeping"); + } + + // static方法，不能被重写（但可以被隐藏） + public static void species() { + System.out.println("This is an animal"); + } +} + +class Dog extends Animal { + + public Dog() { + super("Dog"); + } + + // 重写抽象方法 + @Override + public void makeSound() { + System.out.println(name + " barks: Woof! Woof!"); + } + + // 重写普通方法 + @Override + public void eat() { + System.out.println(name + " eats dog food"); + } + + // 隐藏父类的static方法（不是重写） + public static void species() { + System.out.println("This is a dog"); + } + + // 子类特有的方法 + public void fetch() { + System.out.println(name + " fetches the ball"); + } + + // 重写Object类的方法 + @Override + public String toString() { + return "Dog{name='" + name + "'}"; + } + + @Override + public boolean equals(Object obj) { + if (this == obj) return true; + if (obj == null || getClass() != obj.getClass()) return false; + Dog dog = (Dog) obj; + return name.equals(dog.name); + } + + @Override + public int hashCode() { + return name.hashCode(); + } +} + +// 访问控制与重写 +class Parent { + protected void protectedMethod() { + System.out.println("Parent protected method"); + } + + public void publicMethod() { + System.out.println("Parent public method"); + } + + // private方法不能被重写 + private void privateMethod() { + System.out.println("Parent private method"); + } +} + +class Child extends Parent { + + // 重写protected方法，可以改为public（访问权限放宽） + @Override + public void protectedMethod() { + System.out.println("Child overridden protected method (now public)"); + } + + // 重写public方法，访问权限保持不变 + @Override + public void publicMethod() { + System.out.println("Child overridden public method"); + super.publicMethod(); // 调用父类方法 + } + + // 这不是重写，而是子类的新方法 + public void privateMethod() { + System.out.println("Child's own private method"); + } +} + +/* + * Overload vs Override 对比总结： + * + * ┌─────────────┬─────────────────┬─────────────────┐ + * │ 特性 │ Overload │ Override │ + * ├─────────────┼─────────────────┼─────────────────┤ + * │ 发生位置 │ 同一类内 │ 继承关系中 │ + * │ 方法名 │ 相同 │ 相同 │ + * │ 参数列表 │ 必须不同 │ 必须相同 │ + * │ 返回值类型 │ 可以不同 │ 必须相同 │ + * │ 访问修饰符 │ 可以不同 │ 不能更严格 │ + * │ 解析时机 │ 编译时 │ 运行时 │ + * │ 多态类型 │ 静态多态 │ 动态多态 │ + * │ 继承要求 │ 无 │ 必须 │ + * └─────────────┴─────────────────┴─────────────────┘ + * + * 重要规则： + * 1. 重载：参数不同（类型、个数、顺序），编译时确定 + * 2. 重写：签名相同，运行时确定，体现多态性 + * 3. @Override注解帮助编译器检查重写的正确性 + * 4. 重写遵循"里氏替换原则"：子类可以替换父类 + */ +``` + +### 🎯 你是如何理解面向对象的？ + +> "面向对象（OOP）是一种'万物皆对象'的编程思想： +> +> **核心理念**：将现实问题构建关系，然后抽象成类（class），给类定义属性和方法后，再将类实例化成实例（instance），通过访问实例的属性和调用方法来进行使用。 +> +> **四大特征**： +> +> - **封装**：隐藏内部实现，只暴露必要接口 +> - **继承**：子类继承父类特性，实现代码复用 +> - **多态**：同一接口多种实现，运行时动态绑定 +> - **抽象**：提取共同特征，忽略具体细节 +> +> **设计原则**： +> +> - **单一职责**：一个类只做一件事 +> - **开闭原则**：对扩展开放，对修改关闭 +> - **里氏替换**：子类不破坏父类契约 +> - **接口隔离**：多个专用接口优于单一臃肿接口 +> - **依赖倒置**：依赖抽象而非实现 +> +> 面向对象让代码更模块化、可维护、可扩展，是现代软件开发的基础思想。" + +**💻 代码示例**： + +```java +// 面向对象设计示例：电商系统 +public class OOPDemo { + + public static void main(String[] args) { + // 演示面向对象的四大特性 + demonstrateOOPFeatures(); + + // 演示设计原则 + demonstrateDesignPrinciples(); + } + + public static void demonstrateOOPFeatures() { + System.out.println("=== OOP Features Demo ==="); + + // 1. 封装：通过私有字段和公共方法控制访问 + Account account = new Account(1000.0); + account.deposit(500.0); // 通过方法安全地操作数据 + account.withdraw(200.0); + System.out.println("Account balance: " + account.getBalance()); + + // 2. 继承：子类继承父类特性 + VIPAccount vipAccount = new VIPAccount(2000.0, 0.02); + vipAccount.deposit(1000.0); + System.out.println("VIP account balance: " + vipAccount.getBalance()); + + // 3. 多态：统一接口，不同实现 + PaymentProcessor processor = new PaymentProcessor(); + + Payment creditCard = new CreditCardPayment(); + Payment alipay = new AlipayPayment(); + Payment wechat = new WechatPayment(); + + processor.processPayment(creditCard, 100.0); + processor.processPayment(alipay, 100.0); + processor.processPayment(wechat, 100.0); + + // 4. 抽象：抽取共同特征 + System.out.println("\nAbstraction: All payments implement Payment interface"); + } + + public static void demonstrateDesignPrinciples() { + System.out.println("\n=== Design Principles Demo ==="); + + // 单一职责原则：每个类只负责一个功能 + OrderService orderService = new OrderService(); + PaymentService paymentService = new PaymentService(); + NotificationService notificationService = new NotificationService(); + + // 依赖倒置原则：依赖抽象而非具体实现 + Order order = new Order("ORDER001", 299.0); + orderService.createOrder(order); + paymentService.processPayment(order.getAmount()); + notificationService.sendNotification("Order created successfully"); + } +} + +// 1. 封装示例 +class Account { + private double balance; // 私有字段，外部无法直接访问 + + public Account(double initialBalance) { + this.balance = Math.max(0, initialBalance); // 构造时验证 + } + + // 公共方法提供受控访问 + public void deposit(double amount) { + if (amount > 0) { + balance += amount; + System.out.println("Deposited: " + amount); + } + } + + public boolean withdraw(double amount) { + if (amount > 0 && amount <= balance) { + balance -= amount; + System.out.println("Withdrawn: " + amount); + return true; + } + System.out.println("Insufficient funds"); + return false; + } + + public double getBalance() { + return balance; + } + + protected void setBalance(double balance) { // 受保护的方法供子类使用 + this.balance = balance; + } +} + +// 2. 继承示例 +class VIPAccount extends Account { + private double interestRate; + + public VIPAccount(double initialBalance, double interestRate) { + super(initialBalance); // 调用父类构造器 + this.interestRate = interestRate; + } + + // 重写父类方法 + @Override + public void deposit(double amount) { + super.deposit(amount); // 调用父类方法 + double interest = amount * interestRate; + super.deposit(interest); // 额外的利息 + System.out.println("Interest added: " + interest); + } + + // 子类特有方法 + public double getInterestRate() { + return interestRate; + } +} + +// 3. 多态示例 - 抽象接口 +interface Payment { + void pay(double amount); + String getPaymentMethod(); +} + +// 具体实现类 +class CreditCardPayment implements Payment { + @Override + public void pay(double amount) { + System.out.println("Paid $" + amount + " by Credit Card"); + } + + @Override + public String getPaymentMethod() { + return "Credit Card"; + } +} + +class AlipayPayment implements Payment { + @Override + public void pay(double amount) { + System.out.println("Paid $" + amount + " by Alipay"); + } + + @Override + public String getPaymentMethod() { + return "Alipay"; + } +} + +class WechatPayment implements Payment { + @Override + public void pay(double amount) { + System.out.println("Paid $" + amount + " by WeChat Pay"); + } + + @Override + public String getPaymentMethod() { + return "WeChat Pay"; + } +} + +// 多态使用 +class PaymentProcessor { + public void processPayment(Payment payment, double amount) { + System.out.println("Processing payment via " + payment.getPaymentMethod()); + payment.pay(amount); // 运行时动态绑定 + } +} + +// 设计原则示例：单一职责 +class Order { + private String orderId; + private double amount; + + public Order(String orderId, double amount) { + this.orderId = orderId; + this.amount = amount; + } + + public String getOrderId() { return orderId; } + public double getAmount() { return amount; } +} + +// 每个服务类只负责一个功能 +class OrderService { + public void createOrder(Order order) { + System.out.println("Order created: " + order.getOrderId()); + } +} + +class PaymentService { + public void processPayment(double amount) { + System.out.println("Payment processed: $" + amount); + } +} + +class NotificationService { + public void sendNotification(String message) { + System.out.println("Notification sent: " + message); + } +} +``` + +### 🎯 抽象类和接口有什么区别？ + +"抽象类和接口都是Java中实现抽象的机制，但它们有以下关键区别： + +**抽象类（Abstract Class）**： + +- 可以包含抽象方法和具体方法 +- 可以有成员变量（包括实例变量） +- 可以有构造方法 +- 使用extends关键字继承，支持单继承 +- 访问修饰符可以是public、protected、default + +**接口（Interface）**： + +- JDK 8之前只能有抽象方法，JDK 8+可以有默认方法和静态方法 +- 只能有public static final常量 +- 不能有构造方法 +- 使用implements关键字实现，支持多实现 +- 方法默认是public abstract + +**使用场景**：当多个类有共同特征且需要代码复用时用抽象类；当需要定义规范、实现多重继承效果时用接口。现在更推荐'组合优于继承'的设计理念。" + +**💻 代码示例**： + +```java +// 抽象类示例 +abstract class Vehicle { + protected String brand; // 可以有实例变量 + protected int speed; + + public Vehicle(String brand) { // 可以有构造方法 + this.brand = brand; + } + + // 具体方法 + public void start() { + System.out.println(brand + " vehicle started"); + } + + // 抽象方法，子类必须实现 + public abstract void accelerate(); + public abstract void brake(); +} + +class Car extends Vehicle { + public Car(String brand) { + super(brand); + } + + @Override + public void accelerate() { + speed += 10; + System.out.println("Car accelerated to " + speed + " km/h"); + } + + @Override + public void brake() { + speed = Math.max(0, speed - 15); + System.out.println("Car braked to " + speed + " km/h"); + } +} + +// 接口示例 +interface Flyable { + int MAX_ALTITUDE = 10000; // public static final常量 + + void takeOff(); // public abstract方法 + void land(); + + // JDK 8+ 默认方法 + default void fly() { + System.out.println("Flying at altitude " + MAX_ALTITUDE); + } + + // JDK 8+ 静态方法 + static void checkWeather() { + System.out.println("Weather is suitable for flying"); + } +} + +interface Swimmable { + void dive(); + void surface(); +} + +// 类可以实现多个接口 +class Duck implements Flyable, Swimmable { + @Override + public void takeOff() { + System.out.println("Duck takes off from water"); + } + + @Override + public void land() { + System.out.println("Duck lands on water"); + } + + @Override + public void dive() { + System.out.println("Duck dives underwater"); + } + + @Override + public void surface() { + System.out.println("Duck surfaces"); + } +} +``` + +### 🎯 作用域public，private，protected，以及不写时的区别？ + +"Java访问修饰符控制类、方法、变量的可见性，有四种访问级别： + +**public（公共的）**： + +- 对所有类可见，无访问限制 +- 可以被任何包中的任何类访问 +- 用于对外开放的API接口 + +**protected（受保护的）**： + +- 对同一包内的类和所有子类可见 +- 即使子类在不同包中也可以访问 +- 用于继承体系中需要共享的成员 + +**默认（包访问权限，不写修饰符）**： + +- 只对同一包内的类可见 +- 也称为package-private或friendly +- 用于包内部的实现细节 + +**private（私有的）**： + +- 只对当前类可见，最严格的访问控制 +- 子类也无法访问父类的private成员 +- 用于封装内部实现细节 + +**访问范围**：public > protected > 默认 > private + +**设计原则**：遵循最小权限原则，优先使用private，根据需要逐步放宽权限。" + +### 🎯 说说Java中的内部类有哪些？ + +"Java中的内部类主要有四种类型： + +**1. 成员内部类（Member Inner Class）**：定义在类中的普通内部类，可以访问外部类的所有成员，包括私有成员。创建内部类对象需要先创建外部类对象。 + +**2. 静态内部类（Static Nested Class）**：使用static修饰的内部类，不依赖外部类实例，只能访问外部类的静态成员。可以直接通过外部类名创建。 + +**3. 局部内部类（Local Inner Class）**：定义在方法或代码块中的类，只能在定义它的方法内使用，可以访问方法中的final或effectively final变量。 + +**4. 匿名内部类（Anonymous Inner Class）**：没有名字的内部类，通常用于实现接口或继承类的简单实现，常用于事件处理和回调。 + +内部类的优势是可以访问外部类私有成员，实现更好的封装；缺点是增加了代码复杂度，可能造成内存泄漏（内部类持有外部类引用）。" + +**💻 代码示例**： + +```java +public class OuterClass { + private String outerField = "Outer field"; + private static String staticField = "Static field"; + + // 1. 成员内部类 + public class MemberInnerClass { + private String innerField = "Inner field"; + + public void display() { + System.out.println("Access outer field: " + outerField); + System.out.println("Access static field: " + staticField); + System.out.println("Inner field: " + innerField); + } + + // 内部类不能有静态方法（除非是静态内部类） + // public static void staticMethod() {} // 编译错误 + } + + // 2. 静态内部类 + public static class StaticNestedClass { + private String nestedField = "Nested field"; + + public void display() { + // System.out.println(outerField); // 编译错误，不能访问非静态外部成员 + System.out.println("Access static field: " + staticField); + System.out.println("Nested field: " + nestedField); + } + + public static void staticMethod() { + System.out.println("Static method in nested class"); + } + } + + public void demonstrateLocalClass() { + final String localVar = "Local variable"; + int effectivelyFinalVar = 100; + + // 3. 局部内部类 + class LocalInnerClass { + public void display() { + System.out.println("Access outer field: " + outerField); + System.out.println("Access local var: " + localVar); + System.out.println("Access effectively final var: " + effectivelyFinalVar); + } + } + + LocalInnerClass local = new LocalInnerClass(); + local.display(); + } + + public void demonstrateAnonymousClass() { + // 4. 匿名内部类 - 实现接口 + Runnable runnable = new Runnable() { + @Override + public void run() { + System.out.println("Anonymous class implementing Runnable"); + System.out.println("Access outer field: " + outerField); + } + }; + + // 匿名内部类 - 继承类 + Thread thread = new Thread() { + @Override + public void run() { + System.out.println("Anonymous class extending Thread"); + } + }; + + runnable.run(); + thread.start(); + } + + public static void main(String[] args) { + OuterClass outer = new OuterClass(); + + // 创建成员内部类对象 + OuterClass.MemberInnerClass member = outer.new MemberInnerClass(); + member.display(); + + // 创建静态内部类对象 + OuterClass.StaticNestedClass nested = new OuterClass.StaticNestedClass(); + nested.display(); + OuterClass.StaticNestedClass.staticMethod(); + + // 局部内部类和匿名内部类 + outer.demonstrateLocalClass(); + outer.demonstrateAnonymousClass(); + } +} +``` + + + +### 🎯 实例方法和静态方法有什么不一样? + +实例方法依赖对象，有 `this`，能访问实例变量，支持多态；静态方法依赖类本身，没有 `this`，只能访问静态变量，不能真正重写，常用于工具类和全局方法。 + +| 特性 | 实例方法 (Instance Method) | 静态方法 (Static Method) | +| -------------- | ------------------------------------------- | ----------------------------------------- | +| **归属** | 属于对象实例 | 属于类本身 | +| **调用方式** | `对象.方法()` | `类名.方法()` 或 `对象.方法()`（不推荐） | +| **this 引用** | 隐含传入 `this` 参数 | 没有 `this` | +| **访问权限** | 可访问实例变量和静态变量 | 只能访问静态变量和静态方法 | +| **内存位置** | 方法存在于方法区，调用需依赖对象实例 | 方法存在于方法区，直接通过类调用 | +| **多态性** | 可以被子类重写，支持运行时多态 | 不能真正被重写，只能隐藏（method hiding） | +| **典型应用** | 与对象状态相关的方法（如 `user.getName()`） | 工具方法、工厂方法（如 `Math.max()`） | +| **字节码调用** | `invokevirtual` / `invokeinterface` | `invokestatic` | + + + +### 🎯 break、continue、return的区别及作用？ + +"break、continue、return都是Java中的控制流语句，但作用不同： + +**break语句**： + +- 作用：立即终止当前循环或switch语句 +- 使用场景：循环中满足某个条件时提前退出 +- 只能跳出当前层循环，不能跳出外层循环（除非使用标签） + +**continue语句**： + +- 作用：跳过当前循环迭代的剩余部分，直接进入下一次迭代 +- 使用场景：满足某个条件时跳过当前循环体的执行 +- 只影响当前层循环 + +**return语句**： + +- 作用：立即终止方法的执行并返回到调用处 +- 可以带返回值（非void方法）或不带返回值（void方法） +- 会终止整个方法，不仅仅是循环 + +核心区别：break跳出循环，continue跳过迭代，return跳出方法。" + +**💻 代码示例**： + +```java +public class ControlFlowDemo { + + public static void main(String[] args) { + System.out.println("=== Break Demo ==="); + demonstrateBreak(); + + System.out.println("\n=== Continue Demo ==="); + demonstrateContinue(); + + System.out.println("\n=== Return Demo ==="); + demonstrateReturn(); + + System.out.println("\n=== Labeled Break/Continue ==="); + demonstrateLabeledStatements(); + } + + // break演示 + public static void demonstrateBreak() { + System.out.println("Finding first even number greater than 5:"); + for (int i = 1; i <= 10; i++) { + if (i > 5 && i % 2 == 0) { + System.out.println("Found: " + i); + break; // 找到后立即退出循环 + } + System.out.println("Checking: " + i); + } + System.out.println("Loop ended"); + + // switch中的break + System.out.println("\nSwitch with break:"); + int day = 3; + switch (day) { + case 1: + System.out.println("Monday"); + break; + case 2: + System.out.println("Tuesday"); + break; + case 3: + System.out.println("Wednesday"); + break; // 没有break会继续执行下一个case + default: + System.out.println("Other day"); + } + } + + // continue演示 + public static void demonstrateContinue() { + System.out.println("Printing only odd numbers:"); + for (int i = 1; i <= 10; i++) { + if (i % 2 == 0) { + continue; // 跳过偶数，直接进入下一次迭代 + } + System.out.println("Odd number: " + i); + } + + System.out.println("\nSkipping multiples of 3:"); + for (int i = 1; i <= 10; i++) { + if (i % 3 == 0) { + System.out.println("Skipping: " + i); + continue; + } + System.out.println("Processing: " + i); + } + } + + // return演示 + public static void demonstrateReturn() { + System.out.println("Result: " + findFirstNegative(new int[]{1, 3, -2, 5, -8})); + + processNumbers(); + } + + public static int findFirstNegative(int[] numbers) { + for (int num : numbers) { + if (num < 0) { + return num; // 找到负数立即返回，不继续执行 + } + System.out.println("Checking positive: " + num); + } + return 0; // 没找到负数返回0 + } + + public static void processNumbers() { + System.out.println("Processing numbers 1-10:"); + for (int i = 1; i <= 10; i++) { + if (i == 5) { + System.out.println("Stopping at 5"); + return; // 提前结束方法 + } + System.out.println("Processing: " + i); + } + System.out.println("This will not be printed"); // 不会执行到这里 + } + + // 标签break/continue演示 + public static void demonstrateLabeledStatements() { + System.out.println("Nested loops with labeled break:"); + + outer: for (int i = 1; i <= 3; i++) { + System.out.println("Outer loop: " + i); + for (int j = 1; j <= 3; j++) { + if (i == 2 && j == 2) { + System.out.println("Breaking outer loop at i=2, j=2"); + break outer; // 跳出外层循环 + } + System.out.println(" Inner loop: " + j); + } + System.out.println("Outer loop " + i + " completed"); + } + System.out.println("All loops ended"); + + System.out.println("\nNested loops with labeled continue:"); + outerContinue: for (int i = 1; i <= 3; i++) { + System.out.println("Outer loop: " + i); + for (int j = 1; j <= 3; j++) { + if (i == 2 && j == 2) { + System.out.println("Continuing outer loop at i=2, j=2"); + continue outerContinue; // 继续外层循环的下一次迭代 + } + System.out.println(" Inner loop: " + j); + } + System.out.println("Inner loops completed for i=" + i); + } + } +} + +/* + * 控制流语句对比总结： + * + * ┌─────────────┬─────────────────┬─────────────────┬─────────────────┐ + * │ 语句 │ 作用范围 │ 影响 │ 使用场景 │ + * ├─────────────┼─────────────────┼─────────────────┼─────────────────┤ + * │ break │ 当前循环/switch│ 终止循环 │ 提前退出循环 │ + * │ continue │ 当前循环迭代 │ 跳过当前迭代 │ 跳过特定条件 │ + * │ return │ 整个方法 │ 终止方法执行 │ 返回结果/退出 │ + * └─────────────┴─────────────────┴─────────────────┴─────────────────┘ + * + * 注意事项： + * 1. break和continue只能在循环或switch中使用 + * 2. return可以在方法的任何地方使用 + * 3. 标签可以让break/continue跳出多层循环 + * 4. 在finally块中使用return会覆盖try/catch中的return + */ +``` + +--- + +## 📊 二、数据类型体系（类型基础） + +> **核心思想**：Java是强类型语言，理解基本类型、包装类、自动装箱拆箱机制对于避免性能陷阱和理解JVM内存管理至关重要。 + +### 🎯 int、float、short、double、long、char占字节数？ + +> "Java基本数据类型的字节数是固定的，这保证了跨平台的一致性： +> +> **整数类型**： +> +> - byte：1字节（8位），取值范围 -128 ~ 127 +> - short：2字节（16位），取值范围 -32,768 ~ 32,767 +> - int：4字节（32位），取值范围 -2,147,483,648 ~ 2,147,483,647 +> - long：8字节（64位），取值范围 -9,223,372,036,854,775,808 ~ 9,223,372,036,854,775,807 +> +> **浮点类型**： +> +> - float：4字节（32位），单精度浮点数，有效位数6-7位 +> - double：8字节（64位），双精度浮点数，有效位数15-16位 +> +> **字符类型**： +> +> - char：2字节（16位），采用Unicode编码，取值范围 0 ~ 65,535 +> +> **布尔类型**： +> +> - boolean：理论上1位即可，但JVM实现通常使用1字节 +> +> **记忆技巧**：byte(1) < short(2) < int(4) = float(4) < long(8) = double(8)，char(2)用于Unicode字符。" + +**📊 Java数据类型存储范围对照表**： + +| **数据类型** | **字节数** | **位数** | **取值范围** | **默认值** | **包装类** | +| ------------ | ---------- | -------- | ------------------------------------------------------------ | ---------- | ---------- | +| **byte** | 1字节 | 8位 | -128 ~ 127 (-2⁷ ~ 2⁷-1) | 0 | Byte | +| **short** | 2字节 | 16位 | -32,768 ~ 32,767 (-2¹⁵ ~ 2¹⁵-1) | 0 | Short | +| **int** | 4字节 | 32位 | -2,147,483,648 ~ 2,147,483,647 (-2³¹ ~ 2³¹-1) | 0 | Integer | +| **long** | 8字节 | 64位 | -9,223,372,036,854,775,808 ~ 9,223,372,036,854,775,807 (-2⁶³ ~ 2⁶³-1) | 0L | Long | +| **float** | 4字节 | 32位 | ±3.4E-38 ~ ±3.4E+38 (IEEE 754单精度) | 0.0f | Float | +| **double** | 8字节 | 64位 | ±1.7E-308 ~ ±1.7E+308 (IEEE 754双精度) | 0.0d | Double | +| **char** | 2字节 | 16位 | 0 ~ 65,535 ('\u0000' ~ '\uffff') | '\u0000' | Character | +| **boolean** | 1字节 | 1位逻辑 | true / false | false | Boolean | + +**📝 重要说明**： + +- `*` boolean在JVM中通常占用1字节，但在boolean数组中可能按位存储 +- 整数类型都是**有符号**的，除了char是**无符号**的 +- 浮点类型遵循**IEEE 754标准**，存在精度限制 +- char类型采用**UTF-16编码**，可以表示Unicode字符 +- 所有数值类型都有对应的**包装类**，支持自动装箱拆箱 + +**🔢 数值范围记忆方法**： + +- **有符号整型**：-2^(n-1) ~ 2^(n-1)-1（n为位数） +- **无符号整型**：0 ~ 2^n-1（仅char类型） +- **浮点类型**：指数位数决定范围，尾数位数决定精度 + + + +### 🎯 基本类型和引用类型之间的自动装箱机制？ + +> "自动装箱拆箱是Java 5引入的语法糖，简化了基本类型和包装类之间的转换： +> +> **自动装箱（Autoboxing）**： +> +> - 基本类型自动转换为对应的包装类对象 +> - 编译器在编译时自动调用valueOf()方法 +> - 例如：int → Integer，double → Double +> +> **自动拆箱（Unboxing）**： +> +> - 包装类对象自动转换为对应的基本类型 +> - 编译器自动调用xxxValue()方法 +> - 例如：Integer → int，Double → double +> +> **装箱拆箱触发场景**： +> +> - 赋值操作：基本类型赋给包装类变量 +> - 方法调用：参数类型不匹配时自动转换 +> - 运算操作：包装类参与算术运算 +> - 集合操作：基本类型存入集合 +> +> **缓存机制**： +> +> - Integer缓存-128到127的对象 +> - Boolean缓存true和false +> - Character缓存0到127 +> - Short、Byte有类似缓存 +> +> **注意事项**： +> +> - 可能引发NullPointerException +> - 频繁装箱拆箱影响性能 +> - ==比较时要注意缓存范围 +> +> 装箱拆箱简化了代码，但要注意性能和空指针问题。" + +**💻 代码示例**： + +```java +public class AutoBoxingDemo { + + public static void main(String[] args) { + // 1. 基本装箱拆箱 + Integer a = 100; // 自动装箱：Integer.valueOf(100) + int b = a; // 自动拆箱：a.intValue() + Integer sum = a + 50; // 拆箱运算后装箱 + + // 2. 缓存机制陷阱 + Integer i1 = 127, i2 = 127; // 使用缓存 + Integer i3 = 128, i4 = 128; // 不使用缓存 + System.out.println(i1 == i2); // true (缓存范围内) + System.out.println(i3 == i4); // false (超出缓存) + System.out.println(i3.equals(i4)); // true (正确比较) + + // 3. 空指针陷阱 + Integer nullInt = null; + try { + int value = nullInt; // NPE: 拆箱null对象 + } catch (NullPointerException e) { + System.out.println("拆箱异常: " + e.getClass().getSimpleName()); + } + + // 4. 集合中的装箱 + List list = Arrays.asList(1, 2, 3); // 装箱 + int total = 0; + for (int num : list) { // 拆箱 + total += num; + } + + // 5. 性能对比 + // 基本类型：int累加（快） + // 包装类型：Integer累加（慢，频繁装箱拆箱） + } +} + +/* + * 自动装箱拆箱总结： + * + * ┌─────────────┬─────────────────┬─────────────────┬─────────────────┐ + * │ 操作 │ 触发条件 │ 实际调用 │ 示例 │ + * ├─────────────┼─────────────────┼─────────────────┼─────────────────┤ + * │ 自动装箱 │ 基本类型→包装类 │ valueOf() │ Integer i = 10; │ + * │ 自动拆箱 │ 包装类→基本类型 │ xxxValue() │ int j = i; │ + * │ 运算拆箱 │ 包装类参与运算 │ xxxValue() │ i + 10 │ + * │ 比较拆箱 │ 包装类比较大小 │ xxxValue() │ i > 5 │ + * └─────────────┴─────────────────┴─────────────────┴─────────────────┘ + * + * 缓存范围： + * - Integer: -128 ~ 127 + * - Character: 0 ~ 127 + * - Boolean: true, false + * - Byte: -128 ~ 127 + * - Short: -128 ~ 127 + * - Long: -128 ~ 127 + * + * 最佳实践： + * 1. 避免在循环中频繁装箱拆箱 + * 2. 使用equals()而不是==比较包装类 + * 3. 注意null值的拆箱异常 + * 4. 性能敏感场景优先使用基本类型 + * 5. 集合存储大量数值时考虑基本类型集合库 + */ +``` + +### 🎯 基本数据类型和包装类有什么区别？ + +> "Java中基本数据类型和包装类的主要区别： +> +> **存储位置**：基本类型存储在栈内存或方法区（如果是类变量），包装类对象存储在堆内存中。 +> +> **性能差异**：基本类型操作效率更高，包装类需要额外的对象创建和方法调用开销。 +> +> **功能差异**：基本类型只能存储值，包装类提供了丰富的方法，可以转换、比较、解析等。 +> +> **空值处理**：基本类型不能为null，包装类可以为null，这在集合操作中很重要。 +> +> **自动装箱拆箱**：JDK 5+提供了自动装箱（基本类型→包装类）和拆箱（包装类→基本类型）机制，简化了编码但要注意性能影响。 +> +> **缓存机制**：Integer等包装类对小数值（-128到127）使用缓存，相同值返回同一对象。" + +| **特性** | **基本类型** | **包装类型** | +| ------------ | ------------------------------ | -------------------------------- | +| **类型** | `int`, `double`, `boolean`等 | `Integer`, `Double`, `Boolean`等 | +| **存储位置** | 栈内存（局部变量） | 堆内存（对象实例） | +| **默认值** | `int`为0，`boolean`为false | `null`（可能导致NPE） | +| **内存占用** | 小（如`int`占4字节） | 大（如`Integer`占16字节以上） | +| **对象特性** | 不支持方法调用、泛型、集合存储 | 支持方法调用、泛型、集合存储 | +| **判等方式** | `==`直接比较值 | `==`比较对象地址，`equals`比较值 | + +- 包装类型可以为 null，而基本类型不可以（它使得包装类型可以应用于 POJO 中，而基本类型则不行） + + 和 POJO 类似的，还有数据传输对象 DTO（Data Transfer Object，泛指用于展示层与服务层之间的数据传输对象）、视图对象 VO（View Object，把某个页面的数据封装起来）、持久化对象 PO（Persistant Object，可以看成是与数据库中的表映射的 Java 对象）。 + + 那为什么 POJO 的属性必须要用包装类型呢？ + + 《阿里巴巴 Java 开发手册》上有详细的说明，我们来大声朗读一下（预备，起）。 + +> 数据库的查询结果可能是 null，如果使用基本类型的话，因为要自动拆箱（将包装类型转为基本类型，比如说把 Integer 对象转换成 int 值），就会抛出 `NullPointerException` 的异常。 + +1. 包装类型可用于泛型，而基本类型不可以 + +```java +List list = new ArrayList<>(); // 提示 Syntax error, insert "Dimensions" to complete ReferenceType +List list = new ArrayList<>(); +``` + +为什么呢？ + +> 在 Java 中，**包装类型可以用于泛型，而基本类型不可以**，这是因为 Java 的泛型机制是基于**对象类型**设计的，而基本类型（如 `int`, `double` 等）并不是对象。下面是详细原因： +> +> 1. **Java 泛型的工作原理** +> +> - Java 泛型在编译期间会进行**类型擦除**，即泛型信息会在字节码中被擦除。 +> - 编译后，泛型的类型参数会被替换为 `Object`，或在某些情况下替换为类型的上限（如果有设置）。 +> - 由于泛型会被转换为 `Object` 类型，泛型只能用于**引用类型**，而基本类型不能直接转换为 `Object`。 +> +> 2. 基本类型比包装类型更高效 +> +> 基本类型在栈中直接存储的具体数值，而包装类型则存储的是堆中的引用。 +> +> ![img](https://p1-jj.byteimg.com/tos-cn-i-t2oaga2asx/gold-user-assets/2019/9/29/16d7a5686ac8c66b~tplv-t2oaga2asx-zoom-in-crop-mark:1304:0:0:0.awebp) +> +> +> +> 很显然，相比较于基本类型而言，包装类型需要占用更多的内存空间。假如没有基本类型的话，对于数值这类经常使用到的数据来说，每次都要通过 new 一个包装类型就显得非常笨重。 +> +> 3. 两个包装类型的值可以相同，但却不相等 +> +> 4. 自动装箱和自动拆箱 +> +> 既然有了基本类型和包装类型，肯定有些时候要在它们之间进行转换。把基本类型转换成包装类型的过程叫做装箱（boxing）。反之，把包装类型转换成基本类型的过程叫做拆箱（unboxing） +> +> 5. **包装类型的缓存机制** +> +> - **范围**：部分包装类缓存常用值（如`Integer`缓存-128~127） +> +> - ```java +> Integer a = 127; +> Integer b = 127; +> System.out.println(a == b); // true（同一缓存对象） +> +> Integer c = 128; +> Integer d = 128; +> System.out.println(c == d); // false（新创建对象） +> ``` +> +> + +### 🎯==和equals的区别是什么？ + +> **==运算符**： +> +> - 对于基本数据类型，比较的是值是否相等 +>- 对于引用类型，比较的是内存地址（引用）是否相同 +> - 不能被重写，是Java语言层面的操作符 +> +> **equals方法**： +> +> - 是Object类的方法，所有类都继承了这个方法 +>- 默认实现就是==比较，但可以被重写自定义比较逻辑 +> - String、Integer等包装类都重写了equals方法，比较的是内容 +> - 重写equals时必须同时重写hashCode，保证'equals相等的对象hashCode也相等' +> +> **最佳实践**：比较对象内容用equals，比较引用用==；自定义类需要重写equals和hashCode。" + +### 🎯 深拷贝、浅拷贝区别？ + +> "对象拷贝是Java中的重要概念，分为浅拷贝和深拷贝： +> +> **浅拷贝（Shallow Copy）**： +> +> - 只复制对象的基本字段，不复制引用字段指向的对象 +> - 原对象和拷贝对象共享引用类型的成员变量 +> - 修改引用对象会影响原对象和拷贝对象 +> - 通过Object.clone()默认实现浅拷贝 +> +> **深拷贝（Deep Copy）**： +> +> - 完全复制对象及其所有引用的对象 +> - 原对象和拷贝对象完全独立，互不影响 +> - 需要递归复制所有引用类型的字段 +> - 实现方式：序列化、手动递归复制、第三方库 +> +> **实现方式对比**： +> +> - Object.clone()：浅拷贝，需要实现Cloneable接口 +> - 序列化方式：深拷贝，需要实现Serializable接口 +> - 构造器/工厂方法：可控制拷贝深度 +> - 第三方库：如Apache Commons或Spring BeanUtils +> +> **选择原则**：对象结构简单用浅拷贝，包含复杂引用关系用深拷贝。" + +**💻 代码示例**： + +```java +import java.io.*; +import java.util.*; + +public class CopyDemo { + + public static void main(String[] args) throws Exception { + // 演示浅拷贝 + demonstrateShallowCopy(); + + System.out.println(); + + // 演示深拷贝 + demonstrateDeepCopy(); + + System.out.println(); + + // 演示各种深拷贝实现方式 + demonstrateDeepCopyMethods(); + } + + public static void demonstrateShallowCopy() throws CloneNotSupportedException { + System.out.println("=== 浅拷贝演示 ==="); + + Address address = new Address("北京", "朝阳区"); + Person original = new Person("张三", 25, address); + + // 浅拷贝 + Person shallowCopy = original.clone(); + + System.out.println("原对象: " + original); + System.out.println("浅拷贝: " + shallowCopy); + System.out.println("地址对象相同: " + (original.getAddress() == shallowCopy.getAddress())); + + // 修改拷贝对象的基本字段 + shallowCopy.setName("李四"); + shallowCopy.setAge(30); + + System.out.println("\n修改拷贝对象的基本字段后:"); + System.out.println("原对象: " + original); + System.out.println("浅拷贝: " + shallowCopy); + + // 修改引用字段的内容 + shallowCopy.getAddress().setCity("上海"); + shallowCopy.getAddress().setDistrict("浦东新区"); + + System.out.println("\n修改引用字段内容后:"); + System.out.println("原对象: " + original); + System.out.println("浅拷贝: " + shallowCopy); + System.out.println("原对象地址也被修改了！"); + } + + public static void demonstrateDeepCopy() throws Exception { + System.out.println("=== 深拷贝演示 ==="); + + Address address = new Address("广州", "天河区"); + PersonDeep original = new PersonDeep("王五", 28, address); + original.addHobby("读书"); + original.addHobby("游泳"); + + // 深拷贝 + PersonDeep deepCopy = original.deepClone(); + + System.out.println("原对象: " + original); + System.out.println("深拷贝: " + deepCopy); + System.out.println("地址对象相同: " + (original.getAddress() == deepCopy.getAddress())); + System.out.println("爱好列表相同: " + (original.getHobbies() == deepCopy.getHobbies())); + + // 修改拷贝对象 + deepCopy.setName("赵六"); + deepCopy.getAddress().setCity("深圳"); + deepCopy.getAddress().setDistrict("南山区"); + deepCopy.addHobby("编程"); + + System.out.println("\n修改深拷贝对象后:"); + System.out.println("原对象: " + original); + System.out.println("深拷贝: " + deepCopy); + System.out.println("原对象未受影响！"); + } + + public static void demonstrateDeepCopyMethods() throws Exception { + System.out.println("=== 各种深拷贝方法演示 ==="); + + Address address = new Address("杭州", "西湖区"); + PersonDeep original = new PersonDeep("钱七", 32, address); + original.addHobby("旅游"); + + // 方法1：序列化深拷贝 + PersonDeep copy1 = SerializationUtils.deepCopy(original); + + // 方法2：手动深拷贝 + PersonDeep copy2 = original.manualDeepCopy(); + + // 方法3：构造器深拷贝 + PersonDeep copy3 = new PersonDeep(original); + + System.out.println("原对象: " + original); + System.out.println("序列化拷贝: " + copy1); + System.out.println("手动拷贝: " + copy2); + System.out.println("构造器拷贝: " + copy3); + + // 验证独立性 + copy1.getAddress().setCity("copy1-城市"); + copy2.getAddress().setCity("copy2-城市"); + copy3.getAddress().setCity("copy3-城市"); + + System.out.println("\n修改各拷贝对象后:"); + System.out.println("原对象: " + original); + System.out.println("copy1: " + copy1); + System.out.println("copy2: " + copy2); + System.out.println("copy3: " + copy3); + } +} + +// 地址类 +class Address implements Cloneable, Serializable { + private String city; + private String district; + + public Address(String city, String district) { + this.city = city; + this.district = district; + } + + // 拷贝构造器 + public Address(Address other) { + this.city = other.city; + this.district = other.district; + } + + @Override + protected Address clone() throws CloneNotSupportedException { + return (Address) super.clone(); + } + + // getters and setters + public String getCity() { return city; } + public void setCity(String city) { this.city = city; } + public String getDistrict() { return district; } + public void setDistrict(String district) { this.district = district; } + + @Override + public String toString() { + return city + "-" + district; + } +} + +// 浅拷贝Person类 +class Person implements Cloneable { + private String name; + private int age; + private Address address; + + public Person(String name, int age, Address address) { + this.name = name; + this.age = age; + this.address = address; + } + + @Override + protected Person clone() throws CloneNotSupportedException { + // 默认的浅拷贝 + return (Person) super.clone(); + } + + // getters and setters + public String getName() { return name; } + public void setName(String name) { this.name = name; } + public int getAge() { return age; } + public void setAge(int age) { this.age = age; } + public Address getAddress() { return address; } + public void setAddress(Address address) { this.address = address; } + + @Override + public String toString() { + return "Person{name='" + name + "', age=" + age + ", address=" + address + "}"; + } +} + +// 深拷贝Person类 +class PersonDeep implements Cloneable, Serializable { + private String name; + private int age; + private Address address; + private List hobbies; + + public PersonDeep(String name, int age, Address address) { + this.name = name; + this.age = age; + this.address = address; + this.hobbies = new ArrayList<>(); + } + + // 拷贝构造器 + public PersonDeep(PersonDeep other) { + this.name = other.name; + this.age = other.age; + this.address = new Address(other.address); // 深拷贝地址 + this.hobbies = new ArrayList<>(other.hobbies); // 深拷贝列表 + } + + // 手动深拷贝 + public PersonDeep manualDeepCopy() { + PersonDeep copy = new PersonDeep(this.name, this.age, new Address(this.address)); + copy.hobbies = new ArrayList<>(this.hobbies); + return copy; + } + + // 序列化深拷贝 + public PersonDeep deepClone() throws Exception { + return SerializationUtils.deepCopy(this); + } + + public void addHobby(String hobby) { + hobbies.add(hobby); + } + + // getters and setters + public String getName() { return name; } + public void setName(String name) { this.name = name; } + public int getAge() { return age; } + public void setAge(int age) { this.age = age; } + public Address getAddress() { return address; } + public void setAddress(Address address) { this.address = address; } + public List getHobbies() { return hobbies; } + + @Override + public String toString() { + return "PersonDeep{name='" + name + "', age=" + age + + ", address=" + address + ", hobbies=" + hobbies + "}"; + } +} + +// 序列化工具类 +class SerializationUtils { + + @SuppressWarnings("unchecked") + public static T deepCopy(T original) throws Exception { + // 序列化 + ByteArrayOutputStream baos = new ByteArrayOutputStream(); + try (ObjectOutputStream oos = new ObjectOutputStream(baos)) { + oos.writeObject(original); + } + + // 反序列化 + ByteArrayInputStream bais = new ByteArrayInputStream(baos.toByteArray()); + try (ObjectInputStream ois = new ObjectInputStream(bais)) { + return (T) ois.readObject(); + } + } +} + +/* + * 拷贝方式对比： + * + * ┌─────────────┬─────────────┬─────────────┬─────────────┬─────────────┐ + * │ 方式 │ 实现难度 │ 性能 │ 完整性 │ 适用场景 │ + * ├─────────────┼─────────────┼─────────────┼─────────────┼─────────────┤ + * │ 浅拷贝 │ 简单 │ 高 │ 不完整 │ 简单对象 │ + * │ 手动深拷贝 │ 中等 │ 中等 │ 完整 │ 可控拷贝 │ + * │ 序列化拷贝 │ 简单 │ 低 │ 完整 │ 复杂对象 │ + * │ 构造器拷贝 │ 中等 │ 高 │ 可控 │ 设计良好 │ + * │ 第三方库 │ 简单 │ 中等 │ 完整 │ 通用场景 │ + * └─────────────┴─────────────┴─────────────┴─────────────┴─────────────┘ + * + * 最佳实践： + * 1. 不可变对象不需要深拷贝 + * 2. 优先使用拷贝构造器，控制拷贝逻辑 + * 3. 复杂对象可以考虑序列化方式 + * 4. 性能敏感场景避免序列化拷贝 + * 5. 注意循环引用问题 + */ +``` + +### 🎯 hashCode与equals的关系？ + +> "hashCode和equals是Java对象的两个重要方法，它们有严格的契约关系： +> +> **hashCode方法**： +> +> - 返回对象的哈希码值，用于哈希表（如HashMap）中的快速查找 +> - 默认实现通常基于对象的内存地址计算 +> - 必须保证：相等的对象具有相同的哈希码 +> +> **equals与hashCode契约**： +> +> 1. 如果两个对象equals相等，则它们的hashCode必须相等 +> 2. 如果两个对象hashCode相等，它们的equals不一定相等（哈希冲突） +> 3. 重写equals时必须重写hashCode，否则违反契约 +> 4. hashCode应该尽量分散，减少哈希冲突 +> +> **实践原则**： +> +> - 同时重写equals和hashCode，保持一致性 +> - equals中使用的字段，hashCode也应该使用 +> - 使用Objects.equals()和Objects.hash()简化实现 +> - 重写后的对象才能正确用于HashMap、HashSet等集合 +> +> 违反契约会导致集合操作异常，如HashMap中相等的对象无法正确查找。" + + + +### 🎯 Random函数随机种子了解吗 ？ + +随机种子是伪随机数生成器的初始化参数。相同种子产生相同的随机序列，这保证了可重现性。在生产环境我们通常不设置种子获得真正的随机性，但在测试和调试时会使用固定种子确保结果一致。Java中通过Random构造器或setSeed方法设置。 + +--- + +## 🔤 三、字符串处理（String核心） + +> **核心思想**：String是Java中最常用的类，理解String、StringBuilder、StringBuffer的区别和字符串常量池机制是基础中的基础。 + +### 🎯 String、StringBuilder、StringBuffer的区别？ + +> "String、StringBuilder、StringBuffer的主要区别： +> +> **String类**： +> +> - 不可变（immutable），任何修改都会创建新对象 +> - 线程安全（因为不可变） +> - 适用于字符串内容不会改变的场景 +> - 大量字符串拼接会产生很多临时对象，性能较差 +> +> **StringBuilder类**： +> +> - 可变的字符序列，修改操作在原对象上进行 +> - 线程不安全，但性能最好 +> - 适用于单线程环境下的字符串拼接 +> - 内部使用char数组，动态扩容 +> +> **StringBuffer类**： +> +> - 可变的字符序列，功能与StringBuilder类似 +> - 线程安全（方法都用synchronized修饰） +> - 适用于多线程环境下的字符串拼接 +> - 由于同步开销，性能比StringBuilder差 +> +> **选择建议**：单线程用StringBuilder，多线程用StringBuffer，不变字符串用String。" + +**💻 代码示例**： + +```java +public class StringDemo { + + public static void main(String[] args) { + // 1. String不可变性演示 + demonstrateStringImmutability(); + + // 2. 字符串常量池 + demonstrateStringPool(); + + // 3. 性能对比 + performanceComparison(); + + // 4. 线程安全测试 + threadSafetyTest(); + + // 5. 常用方法演示 + demonstrateStringMethods(); + } + + // String不可变性演示 + public static void demonstrateStringImmutability() { + System.out.println("=== String Immutability ==="); + + String str = "Hello"; + System.out.println("Original string: " + str); + System.out.println("String object ID: " + System.identityHashCode(str)); + + str = str + " World"; // 创建新对象，原对象不变 + System.out.println("After concatenation: " + str); + System.out.println("New object ID: " + System.identityHashCode(str)); + + // 字符串字面量拼接（编译时优化） + String compile1 = "Hello" + " " + "World"; // 编译时合并为"Hello World" + String compile2 = "Hello World"; + System.out.println("Compile-time concatenation same object: " + (compile1 == compile2)); // true + } + + // 字符串常量池演示 + public static void demonstrateStringPool() { + System.out.println("\n=== String Pool ==="); + + String s1 = "hello"; // 常量池 + String s2 = "hello"; // 引用常量池中的对象 + String s3 = new String("hello"); // 堆中新对象 + String s4 = s3.intern(); // 返回常量池中的对象 + + System.out.println("s1 == s2: " + (s1 == s2)); // true，同一个常量池对象 + System.out.println("s1 == s3: " + (s1 == s3)); // false，不同对象 + System.out.println("s1 == s4: " + (s1 == s4)); // true，intern()返回常量池对象 + + // 动态字符串的intern + String dynamic = new String("hello") + new String("world"); + String dynamicIntern = dynamic.intern(); + String literal = "helloworld"; + System.out.println("dynamic.intern() == literal: " + (dynamicIntern == literal)); + } + + // 性能对比 + public static void performanceComparison() { + System.out.println("\n=== Performance Comparison ==="); + int iterations = 10000; + + // String拼接性能（差） + long startTime = System.currentTimeMillis(); + String result1 = ""; + for (int i = 0; i < iterations; i++) { + result1 += "a"; // 每次都创建新对象 + } + long stringTime = System.currentTimeMillis() - startTime; + + // StringBuilder性能（最好） + startTime = System.currentTimeMillis(); + StringBuilder sb = new StringBuilder(); + for (int i = 0; i < iterations; i++) { + sb.append("a"); // 在原对象上修改 + } + String result2 = sb.toString(); + long stringBuilderTime = System.currentTimeMillis() - startTime; + + // StringBuffer性能（中等） + startTime = System.currentTimeMillis(); + StringBuffer sbf = new StringBuffer(); + for (int i = 0; i < iterations; i++) { + sbf.append("a"); // 同步方法调用 + } + String result3 = sbf.toString(); + long stringBufferTime = System.currentTimeMillis() - startTime; + + System.out.println("String concatenation: " + stringTime + "ms"); + System.out.println("StringBuilder: " + stringBuilderTime + "ms"); + System.out.println("StringBuffer: " + stringBufferTime + "ms"); + + System.out.println("Results equal: " + + result1.equals(result2) + ", " + result2.equals(result3)); + } + + // 线程安全测试 + public static void threadSafetyTest() { + System.out.println("\n=== Thread Safety Test ==="); + + // StringBuilder - 线程不安全 + StringBuilder sb = new StringBuilder(); + StringBuffer sbf = new StringBuffer(); + + // 创建多个线程同时操作 + Runnable appendTask = () -> { + for (int i = 0; i < 1000; i++) { + sb.append("a"); + sbf.append("a"); + } + }; + + Thread[] threads = new Thread[5]; + for (int i = 0; i < threads.length; i++) { + threads[i] = new Thread(appendTask); + threads[i].start(); + } + + // 等待所有线程完成 + for (Thread thread : threads) { + try { + thread.join(); + } catch (InterruptedException e) { + e.printStackTrace(); + } + } + + System.out.println("Expected length: " + (5 * 1000)); + System.out.println("StringBuilder length: " + sb.length()); // 可能小于期望值 + System.out.println("StringBuffer length: " + sbf.length()); // 等于期望值 + } + + // 常用字符串方法演示 + public static void demonstrateStringMethods() { + System.out.println("\n=== Common String Methods ==="); + + String str = " Hello World Java Programming "; + + // 基本操作 + System.out.println("Length: " + str.length()); + System.out.println("Trimmed: '" + str.trim() + "'"); + System.out.println("Upper case: " + str.toUpperCase()); + System.out.println("Lower case: " + str.toLowerCase()); + + // 搜索操作 + System.out.println("Index of 'World': " + str.indexOf("World")); + System.out.println("Last index of 'a': " + str.lastIndexOf("a")); + System.out.println("Contains 'Java': " + str.contains("Java")); + System.out.println("Starts with ' Hello': " + str.startsWith(" Hello")); + System.out.println("Ends with 'ing ': " + str.endsWith("ming ")); + + // 替换操作 + System.out.println("Replace 'World' with 'Universe': " + + str.replace("World", "Universe")); + System.out.println("Replace all spaces: " + + str.replaceAll("\\s+", "_")); + + // 分割操作 + String[] words = str.trim().split("\\s+"); + System.out.println("Split into words: " + java.util.Arrays.toString(words)); + + // 子字符串 + System.out.println("Substring(2, 7): '" + str.substring(2, 7) + "'"); + + // 字符操作 + System.out.println("Char at index 2: '" + str.charAt(2) + "'"); + char[] chars = str.toCharArray(); + System.out.println("Char array length: " + chars.length); + } +} + +// StringBuilder和StringBuffer源码分析要点 +class StringBuilderAnalysis { + /* + * StringBuilder和StringBuffer都继承自AbstractStringBuilder + * + * 核心字段： + * char[] value; // 存储字符数据的数组 + * int count; // 当前字符数量 + * + * 扩容机制： + * 当容量不足时，新容量 = (旧容量 + 1) * 2 + * 如果还不够，直接使用需要的容量 + * + * StringBuffer的同步： + * 所有公共方法都用synchronized修饰 + * + * 性能优化建议： + * 1. 预估容量，使用带初始容量的构造方法 + * 2. 单线程环境优先使用StringBuilder + * 3. 避免在循环中使用String拼接 + */ + + public static void optimizedStringBuilding() { + // 好的做法：预估容量 + StringBuilder sb = new StringBuilder(1000); // 预分配容量 + + // 避免的做法：不断扩容 + StringBuilder sb2 = new StringBuilder(); // 默认容量16，会多次扩容 + } +} +``` + +--- + +## ⚠️ 四、异常处理机制（Exception核心） + +> **核心思想**：异常处理是Java程序健壮性的重要保障，理解异常体系、处理机制和最佳实践对编写高质量代码至关重要。 + +### 🎯 Java异常体系是怎样的？ + +**📋 标准话术**： + +> "Java异常体系以Throwable为根类，分为两大分支： +> +> **Error类**：表示严重的系统级错误，程序无法处理，如OutOfMemoryError、StackOverflowError。应用程序不应该捕获这类异常。 +> +> **Exception类**：程序可以处理的异常，又分为两类： +> +> **检查型异常（Checked Exception）**： +> +> - 编译时必须处理（try-catch或throws声明） +> - 如IOException、SQLException、ClassNotFoundException +> - 表示程序运行中可能出现的合理异常情况 +> +> **非检查型异常（Unchecked Exception）**： +> +> - 继承自RuntimeException，编译时不强制处理 +> - 如NullPointerException、ArrayIndexOutOfBoundsException +> - 通常表示程序逻辑错误 +> +> **异常处理原则**：能处理就处理，不能处理就向上抛出；记录日志；不要忽略异常；优先使用标准异常。" + + + +--- + +## 🎯 五、泛型机制（Generic核心） + +> **核心思想**：泛型提供了编译时类型安全检查，避免类型转换异常，同时提高代码复用性和可读性。 + +### 🎯 什么是Java泛型？类型擦除是什么？ + +> "Java泛型是JDK 5引入的特性，允许在定义类、接口和方法时使用类型参数： +> +> **泛型的作用**： +> +> - 提供编译时类型安全检查，避免ClassCastException +> - 消除类型转换的需要，提高代码可读性 +> - 提高代码复用性，一套代码适用多种类型 +> +> **类型擦除（Type Erasure）**： +> +> - Java泛型是编译时特性，运行时会被擦除 +> - 编译后所有泛型信息都被替换为原始类型或Object +> - 这是为了保持与早期Java版本的兼容性 +> - 导致一些限制，如不能创建泛型数组、不能获取运行时泛型信息等 +> +> **通配符**： +> +> - `?` 表示未知类型 +> - `? extends T` 上界通配符，只能读取 +> - `? super T` 下界通配符，只能写入 +> +> **PECS原则**：Producer Extends, Consumer Super - 生产者使用extends，消费者使用super。" + +**💻 代码示例**： + +```java +import java.util.*; + +public class GenericDemo { + + public static void main(String[] args) { + // 1. 泛型类：类型安全，无需转换 + Box stringBox = new Box<>("Hello"); + String str = stringBox.get(); // 无需强转 + + List list = new ArrayList<>(); // 类型安全 + list.add(100); + // list.add("string"); // 编译错误 + + // 2. 泛型方法：类型推断 + String[] arr = {"A", "B", "C"}; + swap(arr, 0, 2); // 可省略 + + // 3. 通配符：灵活性 + List numbers = new ArrayList(); + List ints = new ArrayList(); + + // 4. 类型擦除：运行时泛型信息丢失 + List strList = new ArrayList<>(); + List intList = new ArrayList<>(); + System.out.println(strList.getClass() == intList.getClass()); // true + } + + // 泛型类演示 + public static void demonstrateGenericClass() { + System.out.println("=== Generic Class Demo ==="); + + // 使用泛型避免类型转换 + Box stringBox = new Box<>("Hello"); + Box intBox = new Box<>(42); + + String str = stringBox.get(); // 无需类型转换 + Integer num = intBox.get(); // 编译时类型安全 + + System.out.println("String box: " + str); + System.out.println("Integer box: " + num); + + // 泛型集合 + List stringList = new ArrayList<>(); + stringList.add("Java"); + stringList.add("Python"); + // stringList.add(123); // 编译错误，类型安全 + + for (String s : stringList) { + System.out.println("Language: " + s); // 无需转换 + } + } + + // 泛型方法演示 + public static void demonstrateGenericMethod() { + System.out.println("\n=== Generic Method Demo ==="); + + // 泛型方法自动推断类型 + String[] strings = {"a", "b", "c"}; + Integer[] integers = {1, 2, 3}; + + printArray(strings); // T推断为String + printArray(integers); // T推断为Integer + + // 显式指定类型 + GenericDemo.printArray(new Double[]{1.1, 2.2, 3.3}); + + // 有界类型参数 + System.out.println("Max number: " + findMax(integers)); + + // 多个类型参数 + Pair pair = makePair("Age", 25); + System.out.println("Pair: " + pair); + } + + // 通配符演示 + public static void demonstrateWildcards() { + System.out.println("\n=== Wildcards Demo ==="); + + List intList = Arrays.asList(1, 2, 3); + List doubleList = Arrays.asList(1.1, 2.2, 3.3); + List stringList = Arrays.asList("a", "b", "c"); + + // 上界通配符 - 只能读取，不能写入 + printNumbers(intList); // Integer extends Number + printNumbers(doubleList); // Double extends Number + // printNumbers(stringList); // 编译错误 + + // 下界通配符 - 只能写入Number或其子类 + List objList = new ArrayList<>(); + addNumbers(objList); + System.out.println("Added numbers: " + objList); + + // 无界通配符 + printCollection(intList); + printCollection(stringList); + } + + // 类型擦除演示 + public static void demonstrateTypeErasure() { + System.out.println("\n=== Type Erasure Demo ==="); + + List stringList = new ArrayList<>(); + List intList = new ArrayList<>(); + + // 运行时类型相同，都是ArrayList + System.out.println("Same class: " + + (stringList.getClass() == intList.getClass())); // true + + System.out.println("Class name: " + stringList.getClass().getName()); + + // 反射无法获取泛型信息 + try { + java.lang.reflect.Method method = GenericDemo.class.getMethod("printArray", Object[].class); + System.out.println("Method: " + method.toGenericString()); + } catch (NoSuchMethodException e) { + e.printStackTrace(); + } + } + + // 泛型限制演示 + public static void demonstrateGenericLimitations() { + System.out.println("\n=== Generic Limitations Demo ==="); + + // 1. 不能创建泛型数组 + // List[] array = new List[10]; // 编译错误 + List[] array = new List[10]; // 可以，但会有警告 + + // 2. 不能实例化类型参数 + // T instance = new T(); // 编译错误 + + // 3. 不能创建参数化类型的数组 + // Pair[] pairs = new Pair[10]; // 编译错误 + + // 4. 静态字段不能使用类型参数 + // static T staticField; // 编译错误 + + System.out.println("Generic limitations demonstrated"); + } + + // 泛型方法 + public static void printArray(T[] array) { + for (T element : array) { + System.out.print(element + " "); + } + System.out.println(); + } + + // 有界类型参数 + public static > T findMax(T[] array) { + T max = array[0]; + for (T element : array) { + if (element.compareTo(max) > 0) { + max = element; + } + } + return max; + } + + // 多个类型参数 + public static Pair makePair(T first, U second) { + return new Pair<>(first, second); + } + + // 上界通配符 + public static void printNumbers(List list) { + for (Number n : list) { + System.out.print(n + " "); + } + System.out.println(); + } + + // 下界通配符 + public static void addNumbers(List list) { + list.add(42); + list.add(3.14); + // list.add("string"); // 编译错误 + } + + // 无界通配符 + public static void printCollection(List list) { + for (Object obj : list) { + System.out.print(obj + " "); + } + System.out.println(); + } +} + +// 泛型类 +class Box { + private T content; + + public Box(T content) { + this.content = content; + } + + public T get() { + return content; + } + + public void set(T content) { + this.content = content; + } +} + +// 多个类型参数的泛型类 +class Pair { + private T first; + private U second; + + public Pair(T first, U second) { + this.first = first; + this.second = second; + } + + public T getFirst() { return first; } + public U getSecond() { return second; } + + @Override + public String toString() { + return "(" + first + ", " + second + ")"; + } +} + +// 有界类型参数 +class NumberBox { + private T number; + + public NumberBox(T number) { + this.number = number; + } + + public double getDoubleValue() { + return number.doubleValue(); // 可以调用Number的方法 + } +} +``` + +--- + +## 🪞 六、反射机制（Reflection核心） + +> **核心思想**：反射是Java的动态特性，允许程序在运行时检查和操作类、方法、字段等，是框架开发的重要基础。 + +### 🎯 注解的原理？ + +> "Java注解是一种特殊的'接口'，用于为代码提供元数据信息： +> +> **注解的本质**： +> +> - 注解本质上是继承了Annotation接口的特殊接口 +> - 编译后生成字节码，存储在Class文件的常量池中 +> - 运行时通过反射机制读取注解信息 +> - 不影响程序的正常执行，只提供元数据 +> +> **注解的生命周期**： +> +> - SOURCE：只在源码阶段保留，编译时丢弃 +> - CLASS：编译到Class文件，运行时不加载到JVM +> - RUNTIME：运行时保留，可以通过反射读取 +> +> **注解处理流程**： +> +> 1. 定义注解（使用@interface） +> 2. 在代码中使用注解 +> 3. 编译器将注解信息写入Class文件 +> 4. 运行时通过反射API读取和处理注解 +> +> **核心应用**： +> +> - 框架配置：Spring的@Component、@Autowired等 +> - 代码生成：Lombok的@Data、@Getter等 +> - 约束校验：@NotNull、@Valid等 +> - 测试框架：JUnit的@Test、@BeforeEach等 +> +> 注解是现代Java框架的基石，简化了配置，提高了开发效率。" + +### 🎯 什么是反射机制？反射有什么应用场景？ + +> "Java反射机制是在运行状态中，对于任意一个类都能够知道这个类的所有属性和方法；对于任意一个对象，都能够调用它的任意方法和属性。这种动态获取信息以及动态调用对象方法的功能称为Java的反射机制。 +> +> Java 反射（Reflection）是指在运行时动态地获取类的信息，并操作类的属性、方法、构造器等能力。 +> 它的原理是： +> +> - Java 编译后会生成 `.class` 字节码文件，JVM 类加载器在加载时会把类的元信息存放到 **方法区（元空间 Metaspace）**。 +> - 反射 API（如 `Class`、`Method`、`Field`、`Constructor`）就是对这些元数据的封装。 +> - 程序可以在运行时通过 `Class.forName()` 或 `对象.getClass()` 获取 `Class` 对象，然后操作类的字段和方法。 +> +> **反射的核心类**： +> +> - Class：类的元信息，是反射的入口点 +> - Field：字段信息，可以获取和设置字段值 +> - Method：方法信息，可以调用方法 +> - Constructor：构造器信息，可以创建实例 +> +> **应用场景**： +> +> - 框架开发：框架（Spring、MyBatis）通过反射创建对象、注入依赖。 +> - 注解处理：运行时处理注解信息 +> - 动态代理：JDK动态代理基于反射实现 +> - 配置文件：根据配置动态创建对象 +> - IDE工具：代码提示、自动完成功能 +> +> **优缺点**： +> +> - 优点：提高代码灵活性，实现动态编程 +> - 缺点：性能开销大，破坏封装性，代码可读性差 +> +> 反射是框架设计的灵魂，但在日常开发中应谨慎使用。" + +```java +import java.lang.annotation.*; +import java.lang.reflect.*; +import java.util.*; + +public class ReflectionDemo { + + public static void main(String[] args) throws Exception { + // 1. 获取Class对象的三种方式 + demonstrateGetClass(); + + // 2. 反射操作字段 + demonstrateFields(); + + // 3. 反射操作方法 + demonstrateMethods(); + + // 4. 反射操作构造器 + demonstrateConstructors(); + + // 5. 注解处理 + demonstrateAnnotations(); + + // 6. 动态代理 + demonstrateDynamicProxy(); + } + + // 获取Class对象的方式 + public static void demonstrateGetClass() throws ClassNotFoundException { + System.out.println("=== Get Class Object ==="); + + // 方式1：通过对象获取 + Student student = new Student("Alice", 20); + Class clazz1 = student.getClass(); + + // 方式2：通过类名获取 + Class clazz2 = Student.class; + + // 方式3：通过Class.forName()获取 + Class clazz3 = Class.forName("Student"); + + System.out.println("Same class: " + (clazz1 == clazz2 && clazz2 == clazz3)); + System.out.println("Class name: " + clazz1.getName()); + System.out.println("Simple name: " + clazz1.getSimpleName()); + System.out.println("Package: " + clazz1.getPackage().getName()); + } + + // 反射操作字段 + public static void demonstrateFields() throws Exception { + System.out.println("\n=== Field Operations ==="); + + Class clazz = Student.class; + Student student = new Student("Bob", 22); + + // 获取所有字段（包括私有） + Field[] fields = clazz.getDeclaredFields(); + System.out.println("Fields count: " + fields.length); + + for (Field field : fields) { + System.out.println("Field: " + field.getName() + + ", Type: " + field.getType().getSimpleName() + + ", Modifiers: " + Modifier.toString(field.getModifiers())); + } + + // 访问私有字段 + Field nameField = clazz.getDeclaredField("name"); + nameField.setAccessible(true); // 绕过访问控制 + + String name = (String) nameField.get(student); + System.out.println("Original name: " + name); + + nameField.set(student, "Charlie"); + System.out.println("Modified name: " + student.getName()); + + // 访问静态字段 + Field countField = clazz.getDeclaredField("count"); + countField.setAccessible(true); + int count = (int) countField.get(null); // 静态字段传null + System.out.println("Student count: " + count); + } + + // 反射操作方法 + public static void demonstrateMethods() throws Exception { + System.out.println("\n=== Method Operations ==="); + + Class clazz = Student.class; + Student student = new Student("David", 25); + + // 获取所有方法 + Method[] methods = clazz.getDeclaredMethods(); + System.out.println("Methods count: " + methods.length); + + for (Method method : methods) { + System.out.println("Method: " + method.getName() + + ", Return type: " + method.getReturnType().getSimpleName() + + ", Parameters: " + method.getParameterCount()); + } + + // 调用公共方法 + Method getNameMethod = clazz.getMethod("getName"); + String name = (String) getNameMethod.invoke(student); + System.out.println("Name from method: " + name); + + // 调用私有方法 + Method privateMethod = clazz.getDeclaredMethod("privateMethod"); + privateMethod.setAccessible(true); + privateMethod.invoke(student); + + // 调用带参数的方法 + Method setAgeMethod = clazz.getMethod("setAge", int.class); + setAgeMethod.invoke(student, 30); + System.out.println("Age after method call: " + student.getAge()); + + // 调用静态方法 + Method staticMethod = clazz.getDeclaredMethod("getCount"); + int count = (int) staticMethod.invoke(null); + System.out.println("Count from static method: " + count); + } + + // 反射操作构造器 + public static void demonstrateConstructors() throws Exception { + System.out.println("\n=== Constructor Operations ==="); + + Class clazz = Student.class; + + // 获取所有构造器 + Constructor[] constructors = clazz.getDeclaredConstructors(); + System.out.println("Constructors count: " + constructors.length); + + for (Constructor constructor : constructors) { + System.out.println("Constructor parameters: " + constructor.getParameterCount()); + } + + // 使用无参构造器 + Constructor defaultConstructor = clazz.getDeclaredConstructor(); + Student student1 = (Student) defaultConstructor.newInstance(); + System.out.println("Default constructor: " + student1); + + // 使用有参构造器 + Constructor paramConstructor = clazz.getDeclaredConstructor(String.class, int.class); + Student student2 = (Student) paramConstructor.newInstance("Eva", 28); + System.out.println("Param constructor: " + student2); + } + + // 注解处理 + public static void demonstrateAnnotations() throws Exception { + System.out.println("\n=== Annotation Processing ==="); + + Class clazz = Student.class; + + // 检查类注解 + if (clazz.isAnnotationPresent(MyAnnotation.class)) { + MyAnnotation annotation = clazz.getAnnotation(MyAnnotation.class); + System.out.println("Class annotation value: " + annotation.value()); + } + + // 检查字段注解 + Field nameField = clazz.getDeclaredField("name"); + if (nameField.isAnnotationPresent(MyAnnotation.class)) { + MyAnnotation annotation = nameField.getAnnotation(MyAnnotation.class); + System.out.println("Field annotation value: " + annotation.value()); + } + + // 检查方法注解 + Method method = clazz.getMethod("getName"); + if (method.isAnnotationPresent(MyAnnotation.class)) { + MyAnnotation annotation = method.getAnnotation(MyAnnotation.class); + System.out.println("Method annotation value: " + annotation.value()); + } + } + + // 动态代理 + public static void demonstrateDynamicProxy() { + System.out.println("\n=== Dynamic Proxy ==="); + + // 创建目标对象 + UserService userService = new UserServiceImpl(); + + // 创建代理对象 + UserService proxy = (UserService) Proxy.newProxyInstance( + userService.getClass().getClassLoader(), + userService.getClass().getInterfaces(), + new LoggingHandler(userService) + ); + + // 通过代理调用方法 + proxy.login("admin", "password"); + proxy.logout("admin"); + } +} + +// 示例类 +@MyAnnotation("Student class") +class Student { + private static int count = 0; + + @MyAnnotation("name field") + private String name; + private int age; + + public Student() { + this("Unknown", 0); + } + + public Student(String name, int age) { + this.name = name; + this.age = age; + count++; + } + + @MyAnnotation("getName method") + public String getName() { + return name; + } + + public void setName(String name) { + this.name = name; + } + + public int getAge() { + return age; + } + + public void setAge(int age) { + this.age = age; + } + + private void privateMethod() { + System.out.println("Private method called"); + } + + public static int getCount() { + return count; + } + + @Override + public String toString() { + return "Student{name='" + name + "', age=" + age + "}"; + } +} + +// 自定义注解 +@Retention(RetentionPolicy.RUNTIME) +@Target({ElementType.TYPE, ElementType.FIELD, ElementType.METHOD}) +@interface MyAnnotation { + String value() default ""; +} + +// 动态代理示例 +interface UserService { + void login(String username, String password); + void logout(String username); +} + +class UserServiceImpl implements UserService { + @Override + public void login(String username, String password) { + System.out.println("User " + username + " logged in"); + } + + @Override + public void logout(String username) { + System.out.println("User " + username + " logged out"); + } +} + +class LoggingHandler implements InvocationHandler { + private Object target; + + public LoggingHandler(Object target) { + this.target = target; + } + + @Override + public Object invoke(Object proxy, Method method, Object[] args) throws Throwable { + System.out.println("Before method: " + method.getName()); + Object result = method.invoke(target, args); + System.out.println("After method: " + method.getName()); + return result; + } +} +``` + + + +### 🎯 反射的原理？ + +> 反射基于 JVM 类加载机制，JVM 在类加载后会生成唯一的 `Class` 对象保存类的元信息。反射 API 就是通过 `Class` 对象访问这些元信息，从而在运行时动态调用方法、访问字段或创建对象。它的本质是 JVM 内部类型信息表的访问封装。反射灵活但性能较差，因此主要用于框架层，而不是业务层的高频调用。 + +反射是 **JVM 在类加载后，利用运行时保留的类元数据，通过 `Class` 对象来操作类的属性和方法** 的机制，本质是对 JVM 内部数据结构的一层封装。 + +**1、核心原理** + +- 当类被加载到 JVM 后，会生成唯一的 `Class` 对象，存储类的元信息（字段、方法、构造器、注解等）。 +- `Class` 对象存放在 **方法区（元空间）**，它是反射的入口。 +- 反射 API (`java.lang.reflect`) 通过访问 `Class` 对象来操作元数据，本质就是 **JVM 把字节码信息映射为对象供开发者调用**。 + +**2、核心机制** + +- **Class 对象**：类的运行时表示。 +- **Field 对象**：封装字段信息，可读写字段值。 +- **Method 对象**：封装方法信息，可 `invoke` 调用。 +- **Constructor 对象**：封装构造器信息，可 `newInstance` 创建实例。 +- **setAccessible(true)**：跳过访问权限检查。 + +**3、使用流程** + +1. 获取 `Class` 对象（`Class.forName()`、`xxx.class`、`obj.getClass()`）。 +2. 通过 `Class` 对象获取 `Field`、`Method`、`Constructor` 等。 +3. 调用 `invoke()`、`set()` 等完成动态操作。 + +4、**性能与应用** + +- **性能**：反射比直接调用慢 10~20 倍，因为多了权限校验和方法查找；JVM 在反射调用一定次数后会做优化（内联）。 +- **应用场景**：框架（Spring、Hibernate、MyBatis）利用反射实现 **依赖注入、AOP、ORM** 等。 + +--- + + + +## 📁 八、IO流体系（IO核心） + +> **核心思想**：Java IO提供了丰富的输入输出操作，分为字节流、字符流、缓冲流等，满足不同场景的数据处理需求。 + +### 🎯 BIO、NIO、AIO有什么区别？ + +- **BIO (Blocking I/O):** 同步阻塞I/O模式，数据的读取写入必须阻塞在一个线程内等待其完成。在活动连接数不是特别高（小于单机1000）的情况下，这种模型是比较不错的，可以让每一个连接专注于自己的 I/O 并且编程模型简单，也不用过多考虑系统的过载、限流等问题。线程池本身就是一个天然的漏斗，可以缓冲一些系统处理不了的连接或请求。但是，当面对十万甚至百万级连接的时候，传统的 BIO 模型是无能为力的。因此，我们需要一种更高效的 I/O 处理模型来应对更高的并发量。 +- **NIO (New I/O):** NIO是一种同步非阻塞的I/O模型，在Java 1.4 中引入了NIO框架，对应 java.nio 包，提供了 Channel , Selector，Buffer等抽象。NIO中的N可以理解为Non-blocking，不单纯是New。它支持面向缓冲的，基于通道的I/O操作方法。 NIO提供了与传统BIO模型中的 `Socket` 和 `ServerSocket` 相对应的 `SocketChannel` 和 `ServerSocketChannel` 两种不同的套接字通道实现,两种通道都支持阻塞和非阻塞两种模式。阻塞模式使用就像传统中的支持一样，比较简单，但是性能和可靠性都不好；非阻塞模式正好与之相反。对于低负载、低并发的应用程序，可以使用同步阻塞I/O来提升开发速率和更好的维护性；对于高负载、高并发的（网络）应用，应使用 NIO 的非阻塞模式来开发 +- **AIO (Asynchronous I/O):** AIO 也就是 NIO 2。在 Java 7 中引入了 NIO 的改进版 NIO 2,它是异步非阻塞的IO模型。异步 IO 是基于事件和回调机制实现的，也就是应用操作之后会直接返回，不会堵塞在那里，当后台处理完成，操作系统会通知相应的线程进行后续的操作。AIO 是异步IO的缩写，虽然 NIO 在网络操作中，提供了非阻塞的方法，但是 NIO 的 IO 行为还是同步的。对于 NIO 来说，我们的业务线程是在 IO 操作准备好时，得到通知，接着就由这个线程自行进行 IO 操作，IO操作本身是同步的。查阅网上相关资料，我发现就目前来说 AIO 的应用还不是很广泛，Netty 之前也尝试使用过 AIO，不过又放弃了。 + +```java +import java.io.*; +import java.net.*; +import java.nio.*; +import java.nio.channels.*; +import java.nio.file.*; +import java.util.concurrent.*; + +public class IOModelDemo { + + public static void main(String[] args) throws Exception { + // 演示BIO + demonstrateBIO(); + + System.out.println(); + + // 演示NIO + demonstrateNIO(); + + System.out.println(); + + // 演示AIO + demonstrateAIO(); + + System.out.println(); + + // 性能对比 + performanceComparison(); + } + + // BIO演示 + public static void demonstrateBIO() throws IOException { + System.out.println("=== BIO演示 ==="); + + // 文件操作 - 阻塞式 + String fileName = "bio_test.txt"; + String content = "BIO测试内容\n阻塞式IO操作"; + + // BIO写文件 + try (FileOutputStream fos = new FileOutputStream(fileName); + BufferedWriter writer = new BufferedWriter(new OutputStreamWriter(fos))) { + + writer.write(content); + System.out.println("BIO写入完成"); + } + + // BIO读文件 + try (FileInputStream fis = new FileInputStream(fileName); + BufferedReader reader = new BufferedReader(new InputStreamReader(fis))) { + + String line; + System.out.println("BIO读取内容:"); + while ((line = reader.readLine()) != null) { + System.out.println(line); + } + } + + // 清理 + new File(fileName).delete(); + + // BIO网络编程特点演示 + demonstrateBIONetworking(); + } + + public static void demonstrateBIONetworking() { + System.out.println("\nBIO网络编程特点:"); + System.out.println("- 每个客户端连接需要一个线程"); + System.out.println("- 线程在accept()和read()时阻塞"); + System.out.println("- 适合连接数少、处理快的场景"); + + // 模拟BIO服务器代码结构 + System.out.println("\nBIO服务器伪代码:"); + System.out.println("ServerSocket server = new ServerSocket(port);"); + System.out.println("while (true) {"); + System.out.println(" Socket client = server.accept(); // 阻塞等待"); + System.out.println(" new Thread(() -> {"); + System.out.println(" // 处理客户端请求"); + System.out.println(" InputStream in = client.getInputStream(); // 可能阻塞"); + System.out.println(" }).start();"); + System.out.println("}"); + } + + // NIO演示 + public static void demonstrateNIO() throws IOException { + System.out.println("=== NIO演示 ==="); + + String fileName = "nio_test.txt"; + String content = "NIO测试内容\n非阻塞式IO操作\n基于Channel和Buffer"; + + // NIO写文件 + Path path = Paths.get(fileName); + try (FileChannel channel = FileChannel.open(path, + StandardOpenOption.CREATE, StandardOpenOption.WRITE)) { + + ByteBuffer buffer = ByteBuffer.allocate(1024); + buffer.put(content.getBytes("UTF-8")); + buffer.flip(); // 切换到读模式 + + while (buffer.hasRemaining()) { + channel.write(buffer); + } + System.out.println("NIO写入完成"); + } + + // NIO读文件 + try (FileChannel channel = FileChannel.open(path, StandardOpenOption.READ)) { + ByteBuffer buffer = ByteBuffer.allocate(1024); + + int bytesRead = channel.read(buffer); + if (bytesRead != -1) { + buffer.flip(); // 切换到读模式 + + byte[] bytes = new byte[buffer.remaining()]; + buffer.get(bytes); + + System.out.println("NIO读取内容:"); + System.out.println(new String(bytes, "UTF-8")); + } + } + + // 清理 + Files.deleteIfExists(path); + + // NIO特点演示 + demonstrateNIOFeatures(); + } + + public static void demonstrateNIOFeatures() { + System.out.println("\nNIO核心组件:"); + System.out.println("1. Channel（通道）- 数据传输的通道"); + System.out.println("2. Buffer（缓冲区）- 数据的容器"); + System.out.println("3. Selector（选择器）- 多路复用器"); + + System.out.println("\nNIO优势:"); + System.out.println("- 一个线程处理多个连接"); + System.out.println("- 非阻塞操作，提高并发性能"); + System.out.println("- 内存映射文件，提高大文件处理效率"); + + // Buffer操作演示 + System.out.println("\nBuffer操作演示:"); + ByteBuffer buffer = ByteBuffer.allocate(10); + + System.out.println("初始状态 - position: " + buffer.position() + + ", limit: " + buffer.limit() + + ", capacity: " + buffer.capacity()); + + buffer.put("Hello".getBytes()); + System.out.println("写入后 - position: " + buffer.position() + + ", limit: " + buffer.limit()); + + buffer.flip(); + System.out.println("flip后 - position: " + buffer.position() + + ", limit: " + buffer.limit()); + + buffer.clear(); + System.out.println("clear后 - position: " + buffer.position() + + ", limit: " + buffer.limit()); + } + + // AIO演示 + public static void demonstrateAIO() throws Exception { + System.out.println("=== AIO演示 ==="); + + String fileName = "aio_test.txt"; + String content = "AIO测试内容\n异步非阻塞IO操作\n基于回调机制"; + + Path path = Paths.get(fileName); + + // AIO写文件 + try (AsynchronousFileChannel channel = AsynchronousFileChannel.open(path, + StandardOpenOption.CREATE, StandardOpenOption.WRITE)) { + + ByteBuffer buffer = ByteBuffer.wrap(content.getBytes("UTF-8")); + + // 异步写入，使用Future + Future writeResult = channel.write(buffer, 0); + + // 等待写入完成 + Integer bytesWritten = writeResult.get(); + System.out.println("AIO异步写入完成，写入字节数: " + bytesWritten); + } + + // AIO读文件 - 使用回调 + try (AsynchronousFileChannel channel = AsynchronousFileChannel.open(path, + StandardOpenOption.READ)) { + + ByteBuffer buffer = ByteBuffer.allocate(1024); + + // 使用CompletionHandler回调 + CountDownLatch latch = new CountDownLatch(1); + + channel.read(buffer, 0, buffer, new CompletionHandler() { + @Override + public void completed(Integer result, ByteBuffer attachment) { + attachment.flip(); + + byte[] bytes = new byte[attachment.remaining()]; + attachment.get(bytes); + + try { + System.out.println("AIO异步读取完成:"); + System.out.println(new String(bytes, "UTF-8")); + } catch (Exception e) { + e.printStackTrace(); + } finally { + latch.countDown(); + } + } + + @Override + public void failed(Throwable exc, ByteBuffer attachment) { + System.err.println("AIO读取失败: " + exc.getMessage()); + latch.countDown(); + } + }); + + // 等待异步操作完成 + latch.await(); + } + + // 清理 + Files.deleteIfExists(path); + + // AIO特点演示 + demonstrateAIOFeatures(); + } + + public static void demonstrateAIOFeatures() { + System.out.println("\nAIO特点:"); + System.out.println("- 真正的异步IO操作"); + System.out.println("- 操作完成后通过回调通知"); + System.out.println("- 不需要轮询操作状态"); + System.out.println("- 适合IO密集型应用"); + + System.out.println("\nAIO两种获取结果的方式:"); + System.out.println("1. Future方式 - 主动查询结果"); + System.out.println("2. CompletionHandler方式 - 被动回调通知"); + } + + // 性能对比 + public static void performanceComparison() throws Exception { + System.out.println("=== IO模型性能对比 ==="); + + int fileCount = 100; + String content = "性能测试内容\n"; + + // BIO性能测试 + long start = System.currentTimeMillis(); + for (int i = 0; i < fileCount; i++) { + String fileName = "bio_perf_" + i + ".txt"; + try (FileOutputStream fos = new FileOutputStream(fileName)) { + fos.write(content.getBytes()); + } + new File(fileName).delete(); + } + long bioTime = System.currentTimeMillis() - start; + + // NIO性能测试 + start = System.currentTimeMillis(); + for (int i = 0; i < fileCount; i++) { + String fileName = "nio_perf_" + i + ".txt"; + Path path = Paths.get(fileName); + try (FileChannel channel = FileChannel.open(path, + StandardOpenOption.CREATE, StandardOpenOption.WRITE)) { + ByteBuffer buffer = ByteBuffer.wrap(content.getBytes()); + channel.write(buffer); + } + Files.deleteIfExists(path); + } + long nioTime = System.currentTimeMillis() - start; + + System.out.println("文件操作性能对比 (" + fileCount + "个文件):"); + System.out.println("BIO耗时: " + bioTime + "ms"); + System.out.println("NIO耗时: " + nioTime + "ms"); + + if (nioTime > 0) { + System.out.println("性能比率: " + (bioTime / (double) nioTime)); + } + + System.out.println("\n实际应用中的选择:"); + System.out.println("- 小文件、低并发：BIO简单高效"); + System.out.println("- 大文件、高并发：NIO性能更好"); + System.out.println("- 长时间IO操作：AIO避免线程阻塞"); + } +} + +/* + * IO模型对比总结： + * + * ┌─────────┬─────────────┬─────────────┬─────────────┬─────────────┐ + * │ 模型 │ 阻塞性 │ 复杂度 │ 并发性 │ 适用场景 │ + * ├─────────┼─────────────┼─────────────┼─────────────┼─────────────┤ + * │ BIO │ 同步阻塞 │ 低 │ 低 │ 连接数少 │ + * │ NIO │ 同步非阻塞 │ 中 │ 高 │ 高并发 │ + * │ AIO │ 异步非阻塞 │ 高 │ 高 │ 长连接 │ + * └─────────┴─────────────┴─────────────┴─────────────┴─────────────┘ + * + * 核心区别： + * 1. BIO：线程与连接1:1，适合连接数少的场景 + * 2. NIO：线程与连接1:N，使用Selector实现多路复用 + * 3. AIO：操作异步执行，通过回调获取结果 + * + * 发展历程： + * - JDK 1.0: BIO - 简单但性能有限 + * - JDK 1.4: NIO - 提升并发性能 + * - JDK 1.7: AIO/NIO.2 - 真正异步IO + */ +``` + +### 🎯 字节流和字符流的区别？什么是缓冲流？ + +**📋 标准话术**： + +> "Java IO流体系是处理输入输出数据的核心API： +> +> **字节流vs字符流**： +> +> - 字节流（InputStream/OutputStream）：以字节为单位处理数据，可以处理任何类型的文件 +> - 字符流（Reader/Writer）：以字符为单位处理数据，专门处理文本文件，支持字符编码转换 +> - 字节流是最基础的，字符流是在字节流基础上的封装 +> +> **缓冲流**： +> +> - BufferedInputStream/BufferedOutputStream：字节缓冲流 +> - BufferedReader/BufferedWriter：字符缓冲流 +> - 通过内存缓冲区减少实际的磁盘IO次数，大幅提高性能 +> - 默认缓冲区大小8192字节 +> +> **NIO（New IO）**： +> +> - 面向缓冲区（Buffer）而不是面向流 +> - 支持非阻塞IO操作 +> - 使用Channel和Selector实现高性能IO +> - 适合高并发场景，如服务器端编程 +> +> 选择原则：处理文本用字符流，处理二进制用字节流，高性能场景用缓冲流或NIO。" + +**💻 代码示例**： + +```java +import java.io.*; +import java.nio.*; +import java.nio.channels.*; +import java.nio.file.*; +import java.util.*; + +public class IODemo { + + public static void main(String[] args) throws IOException { + // 1. 字节流操作 + demonstrateByteStreams(); + + // 2. 字符流操作 + demonstrateCharacterStreams(); + + // 3. 缓冲流操作 + demonstrateBufferedStreams(); + + // 4. NIO基础操作 + demonstrateNIO(); + + // 5. IO性能对比 + performanceComparison(); + } + + // 字节流演示 + public static void demonstrateByteStreams() throws IOException { + System.out.println("=== Byte Streams Demo ==="); + + String content = "Hello Java IO! 你好世界！"; + String fileName = "byte_test.txt"; + + // 字节流写入 + try (FileOutputStream fos = new FileOutputStream(fileName)) { + byte[] bytes = content.getBytes("UTF-8"); + fos.write(bytes); + System.out.println("Written " + bytes.length + " bytes"); + } + + // 字节流读取 + try (FileInputStream fis = new FileInputStream(fileName)) { + byte[] buffer = new byte[1024]; + int bytesRead = fis.read(buffer); + String result = new String(buffer, 0, bytesRead, "UTF-8"); + System.out.println("Read: " + result); + } + + // 复制文件示例 + copyFileUsingByteStream("byte_test.txt", "byte_copy.txt"); + + // 清理文件 + new File(fileName).delete(); + new File("byte_copy.txt").delete(); + } + + // 字符流演示 + public static void demonstrateCharacterStreams() throws IOException { + System.out.println("\n=== Character Streams Demo ==="); + + String content = "Hello Java IO!\n你好世界！\n字符流测试"; + String fileName = "char_test.txt"; + + // 字符流写入 + try (FileWriter writer = new FileWriter(fileName, StandardCharsets.UTF_8)) { + writer.write(content); + System.out.println("Written characters: " + content.length()); + } + + // 字符流读取 + try (FileReader reader = new FileReader(fileName, StandardCharsets.UTF_8)) { + char[] buffer = new char[1024]; + int charsRead = reader.read(buffer); + String result = new String(buffer, 0, charsRead); + System.out.println("Read characters: " + charsRead); + System.out.println("Content:\n" + result); + } + + // 按行读取 + System.out.println("Reading line by line:"); + try (BufferedReader br = new BufferedReader( + new FileReader(fileName, StandardCharsets.UTF_8))) { + String line; + int lineNumber = 1; + while ((line = br.readLine()) != null) { + System.out.println("Line " + lineNumber++ + ": " + line); + } + } + + // 清理文件 + new File(fileName).delete(); + } + + // 缓冲流演示 + public static void demonstrateBufferedStreams() throws IOException { + System.out.println("\n=== Buffered Streams Demo ==="); + + String fileName = "buffered_test.txt"; + + // 缓冲字符流写入 + try (BufferedWriter writer = new BufferedWriter( + new FileWriter(fileName, StandardCharsets.UTF_8))) { + for (int i = 1; i <= 5; i++) { + writer.write("Line " + i + ": This is a test line."); + writer.newLine(); // 跨平台换行 + } + writer.flush(); // 强制刷新缓冲区 + System.out.println("Written 5 lines with buffered writer"); + } + + // 缓冲字符流读取 + try (BufferedReader reader = new BufferedReader( + new FileReader(fileName, StandardCharsets.UTF_8))) { + System.out.println("Reading with buffered reader:"); + String line; + while ((line = reader.readLine()) != null) { + System.out.println(line); + } + } + + // 缓冲字节流示例 + demonstrateByteBufferedStream(fileName); + + // 清理文件 + new File(fileName).delete(); + } + + // NIO基础演示 + public static void demonstrateNIO() throws IOException { + System.out.println("\n=== NIO Demo ==="); + + String content = "NIO Test Content\n使用NIO进行文件操作"; + Path path = Paths.get("nio_test.txt"); + + // NIO写入文件 + Files.write(path, content.getBytes(StandardCharsets.UTF_8)); + System.out.println("Written file using NIO"); + + // NIO读取文件 + byte[] bytes = Files.readAllBytes(path); + String result = new String(bytes, StandardCharsets.UTF_8); + System.out.println("Read content: " + result); + + // NIO按行读取 + System.out.println("Reading lines with NIO:"); + List lines = Files.readAllLines(path, StandardCharsets.UTF_8); + for (int i = 0; i < lines.size(); i++) { + System.out.println("Line " + (i + 1) + ": " + lines.get(i)); + } + + // Channel和Buffer示例 + demonstrateChannelAndBuffer(path); + + // 清理文件 + Files.deleteIfExists(path); + } + + // Channel和Buffer演示 + public static void demonstrateChannelAndBuffer(Path path) throws IOException { + System.out.println("\n--- Channel and Buffer Demo ---"); + + // 使用Channel写入 + try (FileChannel channel = FileChannel.open(path, + StandardOpenOption.CREATE, StandardOpenOption.WRITE)) { + + String data = "Channel Buffer Demo\n通道缓冲区演示"; + ByteBuffer buffer = ByteBuffer.allocate(1024); + buffer.put(data.getBytes(StandardCharsets.UTF_8)); + buffer.flip(); // 切换到读模式 + + while (buffer.hasRemaining()) { + channel.write(buffer); + } + System.out.println("Written using Channel and Buffer"); + } + + // 使用Channel读取 + try (FileChannel channel = FileChannel.open(path, StandardOpenOption.READ)) { + ByteBuffer buffer = ByteBuffer.allocate(1024); + int bytesRead = channel.read(buffer); + buffer.flip(); // 切换到读模式 + + byte[] bytes = new byte[bytesRead]; + buffer.get(bytes); + String result = new String(bytes, StandardCharsets.UTF_8); + System.out.println("Read using Channel: " + result); + } + } + + // 性能对比 + public static void performanceComparison() throws IOException { + System.out.println("\n=== Performance Comparison ==="); + + String testFile = "performance_test.txt"; + int iterations = 10000; + + // 测试无缓冲字节流 + long start = System.currentTimeMillis(); + try (FileOutputStream fos = new FileOutputStream(testFile)) { + for (int i = 0; i < iterations; i++) { + fos.write("Test line\n".getBytes()); + } + } + long unbufferedTime = System.currentTimeMillis() - start; + + // 测试缓冲字节流 + start = System.currentTimeMillis(); + try (BufferedOutputStream bos = new BufferedOutputStream( + new FileOutputStream(testFile + "_buffered"))) { + for (int i = 0; i < iterations; i++) { + bos.write("Test line\n".getBytes()); + } + } + long bufferedTime = System.currentTimeMillis() - start; + + System.out.println("Unbuffered time: " + unbufferedTime + "ms"); + System.out.println("Buffered time: " + bufferedTime + "ms"); + System.out.println("Performance improvement: " + + (unbufferedTime / (double) bufferedTime) + "x"); + + // 清理文件 + new File(testFile).delete(); + new File(testFile + "_buffered").delete(); + } + + // 工具方法 + public static void copyFileUsingByteStream(String source, String dest) throws IOException { + try (FileInputStream fis = new FileInputStream(source); + FileOutputStream fos = new FileOutputStream(dest)) { + + byte[] buffer = new byte[1024]; + int bytesRead; + while ((bytesRead = fis.read(buffer)) != -1) { + fos.write(buffer, 0, bytesRead); + } + System.out.println("File copied successfully"); + } + } + + public static void demonstrateByteBufferedStream(String fileName) throws IOException { + System.out.println("--- Byte Buffered Stream ---"); + + // 读取文件到字节数组 + try (BufferedInputStream bis = new BufferedInputStream( + new FileInputStream(fileName))) { + + byte[] buffer = new byte[1024]; + int bytesRead = bis.read(buffer); + String content = new String(buffer, 0, bytesRead, StandardCharsets.UTF_8); + System.out.println("Buffered byte stream read: " + content.replace("\n", "\\n")); + } + } +} + +// IO流的分类和特点 +class IOStreamTypes { + + public static void showStreamHierarchy() { + System.out.println("=== Java IO Stream Hierarchy ==="); + System.out.println("字节流 (Byte Streams):"); + System.out.println("├── InputStream (抽象基类)"); + System.out.println("│ ├── FileInputStream (文件字节输入流)"); + System.out.println("│ ├── BufferedInputStream (缓冲字节输入流)"); + System.out.println("│ ├── DataInputStream (数据字节输入流)"); + System.out.println("│ └── ObjectInputStream (对象字节输入流)"); + System.out.println("└── OutputStream (抽象基类)"); + System.out.println(" ├── FileOutputStream (文件字节输出流)"); + System.out.println(" ├── BufferedOutputStream (缓冲字节输出流)"); + System.out.println(" ├── DataOutputStream (数据字节输出流)"); + System.out.println(" └── ObjectOutputStream (对象字节输出流)"); + + System.out.println("\n字符流 (Character Streams):"); + System.out.println("├── Reader (抽象基类)"); + System.out.println("│ ├── FileReader (文件字符输入流)"); + System.out.println("│ ├── BufferedReader (缓冲字符输入流)"); + System.out.println("│ ├── InputStreamReader (字节到字符转换流)"); + System.out.println("│ └── StringReader (字符串输入流)"); + System.out.println("└── Writer (抽象基类)"); + System.out.println(" ├── FileWriter (文件字符输出流)"); + System.out.println(" ├── BufferedWriter (缓冲字符输出流)"); + System.out.println(" ├── OutputStreamWriter (字符到字节转换流)"); + System.out.println(" └── StringWriter (字符串输出流)"); + } +} +``` + +### 🎯 什么是Java序列化？如何实现Java序列化？ + +**📋 标准话术**： + +> "Java序列化是将对象转换为字节流的机制，反序列化是将字节流恢复为对象： +> +> **序列化定义**： +> +> - 将对象的状态信息转换为可以存储或传输的形式的过程 +> - 主要用于对象持久化、网络传输、深拷贝等场景 +> - Java通过ObjectOutputStream和ObjectInputStream实现 +> +> **Serializable接口**： +> +> - 标记接口，没有方法需要实现 +> - 表示该类的对象可以被序列化 +> - 如果父类实现了Serializable，子类自动可序列化 +> - 不实现此接口的类无法序列化，会抛NotSerializableException +> +> **serialVersionUID**： +> +> - 序列化版本号，用于验证序列化对象的兼容性 +> - 如果不显式声明，JVM会自动生成 +> - 类结构改变时，自动生成的UID会变化，导致反序列化失败 +> - 建议显式声明，便于版本控制 +> +> **序列化过程**： +> +> 1. 检查对象是否实现Serializable接口 +> 2. 写入类元数据信息 +> 3. 递归序列化父类信息 +> 4. 写入实例字段数据（transient字段除外） +> +> **注意事项**： +> +> - static字段不会被序列化 +> - transient字段会被忽略 +> - 父类不可序列化时，需要无参构造器 +> - 序列化可能破坏单例模式" + +### 🎯 Serializable接口的作用？transient关键字？ + +**📋 标准话术**： + +> "Serializable接口是Java序列化机制的核心： +> +> **Serializable接口作用**： +> +> - 标记接口，标识类可以被序列化 +> - 启用序列化机制，允许对象转换为字节流 +> - 提供版本控制支持（serialVersionUID） +> - 支持自定义序列化逻辑（writeObject/readObject） +> +> **transient关键字**： +> +> - 修饰字段，表示该字段不参与序列化 +> - 反序列化时，transient字段会被初始化为默认值 +> - 常用于敏感信息（密码）、计算字段、缓存数据 +> - 可以通过自定义序列化方法重新赋值 +> +> **自定义序列化**： +> +> - writeObject()：自定义序列化逻辑 +> - readObject()：自定义反序列化逻辑 +> - writeReplace()：序列化前替换对象 +> - readResolve()：反序列化后替换对象（常用于单例） +> +> 序列化是Java的重要机制，但要注意性能和安全性问题。" + +------ + +## 🔗 七、Java引用类型（Reference Types） + +> **核心思想**：Java提供了四种引用类型来支持不同场景下的内存管理，是垃圾回收和内存优化的重要机制。 + +### 🎯 Java的四种引用类型有哪些？ + +**📋 标准话术**： + +> "Java中有四种引用类型：强引用、软引用、弱引用、虚引用。它们的强度依次递减： +> +> **强引用（Strong Reference）**： +> +> - 最常见的引用类型，如 `Object obj = new Object()` +> - 只要强引用存在，垃圾收集器永远不会回收被引用的对象 +> - 即使内存不足抛出OutOfMemoryError也不会回收 +> - 是造成内存泄漏的主要原因 +> +> **软引用（Soft Reference）**： +> +> - 用于描述有用但非必需的对象 +> - 内存充足时不会被回收，内存不足时会被回收 +> - 适合实现内存敏感的高速缓存 +> - JVM会根据内存使用情况智能回收 +> +> **弱引用（Weak Reference）**： +> +> - 比软引用更弱的引用类型 +> - 无论内存是否充足，下次GC时都会被回收 +> - 适合实现缓存、监听器回调、ThreadLocal等场景 +> - 可以避免循环引用导致的内存泄漏 +> +> **虚引用（Phantom Reference）**： +> +> - 最弱的引用类型，不能通过引用获取对象 +> - 主要用于跟踪对象被垃圾收集的活动 +> - 必须配合ReferenceQueue使用 +> - 适合实现对象销毁的监控和清理工作 +> +**💻 代码示例**： + +```java +import java.lang.ref.*; +import java.util.*; +import java.util.concurrent.ConcurrentHashMap; + +public class ReferenceTypesDemo { + + public static void main(String[] args) throws InterruptedException { + // 1. 强引用演示 + demonstrateStrongReference(); + + // 2. 软引用演示 + demonstrateSoftReference(); + + // 3. 弱引用演示 + demonstrateWeakReference(); + + // 4. 虚引用演示 + demonstratePhantomReference(); + + // 5. 实际应用场景 + demonstrateRealWorldUsage(); + } + + // 强引用演示 + public static void demonstrateStrongReference() { + System.out.println("=== 强引用演示 ==="); + + // 创建强引用 + String strongRef = new String("Strong Reference Example"); + System.out.println("Strong reference created: " + strongRef); + + // 即使显式调用GC，强引用也不会被回收 + System.gc(); + System.out.println("After GC, strong reference still exists: " + strongRef); + + // 只有将引用设为null，对象才可能被回收 + strongRef = null; + System.gc(); + System.out.println("Strong reference set to null, object can be collected"); + } + + // 软引用演示 + public static void demonstrateSoftReference() { + System.out.println("\n=== 软引用演示 ==="); + + // 创建一个大对象 + byte[] largeObject = new byte[1024 * 1024]; // 1MB + SoftReference softRef = new SoftReference<>(largeObject); + + // 清除强引用 + largeObject = null; + + System.out.println("Soft reference created"); + System.out.println("Object accessible via soft reference: " + + (softRef.get() != null ? "Yes" : "No")); + + // 内存充足时，软引用不会被回收 + System.gc(); + System.out.println("After GC (memory sufficient): " + + (softRef.get() != null ? "Still alive" : "Collected")); + } + + // 弱引用演示 + public static void demonstrateWeakReference() { + System.out.println("\n=== 弱引用演示 ==="); + + String original = new String("Weak Reference Example"); + WeakReference weakRef = new WeakReference<>(original); + + System.out.println("Weak reference created"); + System.out.println("Object accessible: " + weakRef.get()); + + // 清除强引用 + original = null; + + System.out.println("Strong reference cleared"); + System.out.println("Before GC: " + weakRef.get()); + + // 弱引用在下次GC时会被回收 + System.gc(); + Thread.yield(); // 让GC有时间执行 + + System.out.println("After GC: " + (weakRef.get() != null ? weakRef.get() : "Collected")); + + // WeakHashMap演示 + demonstrateWeakHashMap(); + } + + // WeakHashMap演示 + public static void demonstrateWeakHashMap() { + System.out.println("\n--- WeakHashMap演示 ---"); + + WeakHashMap weakMap = new WeakHashMap<>(); + Map normalMap = new HashMap<>(); + + UniqueKey key1 = new UniqueKey("key1"); + UniqueKey key2 = new UniqueKey("key2"); + + weakMap.put(key1, "value1"); + weakMap.put(key2, "value2"); + normalMap.put(key1, "value1"); + normalMap.put(key2, "value2"); + + System.out.println("Initial size - WeakHashMap: " + weakMap.size() + + ", HashMap: " + normalMap.size()); + + // 清除一个key的强引用 + key1 = null; + + System.gc(); + Thread.yield(); + + System.out.println("After GC - WeakHashMap: " + weakMap.size() + + ", HashMap: " + normalMap.size()); + System.out.println("WeakHashMap自动清理了无强引用的key"); + } + + // 虚引用演示 + public static void demonstratePhantomReference() { + System.out.println("\n=== 虚引用演示 ==="); + + ReferenceQueue queue = new ReferenceQueue<>(); + Object obj = new Object(); + PhantomReference phantomRef = new PhantomReference<>(obj, queue); + + System.out.println("Phantom reference created"); + System.out.println("Can get object from phantom reference: " + phantomRef.get()); // 总是null + + obj = null; // 清除强引用 + + System.gc(); + Thread.yield(); + + // 检查ReferenceQueue中是否有被回收的对象通知 + Reference removedRef = queue.poll(); + System.out.println("Object collected notification: " + + (removedRef != null ? "Received" : "Not received")); + + if (removedRef != null) { + System.out.println("Phantom reference equals removed reference: " + + (removedRef == phantomRef)); + } + } + + // 实际应用场景演示 + public static void demonstrateRealWorldUsage() { + System.out.println("\n=== 实际应用场景 ==="); + + // 1. 软引用用于缓存 + ImageCache cache = new ImageCache(); + cache.putImage("image1", new byte[1024]); + cache.putImage("image2", new byte[2048]); + + System.out.println("Cache size: " + cache.size()); + + // 2. 弱引用用于监听器 + EventPublisher publisher = new EventPublisher(); + EventListener listener1 = new EventListener("Listener1"); + EventListener listener2 = new EventListener("Listener2"); + + publisher.addListener(listener1); + publisher.addListener(listener2); + + System.out.println("Active listeners: " + publisher.getListenerCount()); + + listener1 = null; // 清除强引用 + System.gc(); + Thread.yield(); + + publisher.cleanupListeners(); // 清理无效监听器 + System.out.println("After cleanup: " + publisher.getListenerCount()); + + // 3. ThreadLocal内部使用弱引用 + demonstrateThreadLocalWeakReference(); + } + + // ThreadLocal弱引用演示 + public static void demonstrateThreadLocalWeakReference() { + System.out.println("\n--- ThreadLocal弱引用机制 ---"); + + ThreadLocal threadLocal1 = new ThreadLocal<>(); + ThreadLocal threadLocal2 = new ThreadLocal<>(); + + threadLocal1.set("Value1"); + threadLocal2.set("Value2"); + + System.out.println("ThreadLocal values set"); + System.out.println("Value1: " + threadLocal1.get()); + System.out.println("Value2: " + threadLocal2.get()); + + // ThreadLocal对象设为null，但WeakReference仍可能存在于ThreadLocalMap中 + threadLocal1 = null; + + System.gc(); + Thread.yield(); + + System.out.println("threadLocal1 set to null"); + System.out.println("Value2 still accessible: " + threadLocal2.get()); + + // 手动清理ThreadLocal（最佳实践） + threadLocal2.remove(); + System.out.println("threadLocal2 manually removed"); + } +} + +// 自定义Key类用于WeakHashMap演示 +class UniqueKey { + private String name; + + public UniqueKey(String name) { + this.name = name; + } + + @Override + public boolean equals(Object obj) { + if (this == obj) return true; + if (obj == null || getClass() != obj.getClass()) return false; + UniqueKey key = (UniqueKey) obj; + return Objects.equals(name, key.name); + } + + @Override + public int hashCode() { + return Objects.hash(name); + } + + @Override + public String toString() { + return "UniqueKey{name='" + name + "'}"; + } +} + +// 软引用缓存实现 +class ImageCache { + private Map> cache = new ConcurrentHashMap<>(); + + public void putImage(String key, byte[] imageData) { + cache.put(key, new SoftReference<>(imageData)); + } + + public byte[] getImage(String key) { + SoftReference ref = cache.get(key); + if (ref != null) { + byte[] data = ref.get(); + if (data == null) { + cache.remove(key); // 清理已被回收的条目 + } + return data; + } + return null; + } + + public int size() { + // 清理已被回收的条目 + cache.entrySet().removeIf(entry -> entry.getValue().get() == null); + return cache.size(); + } +} + +// 弱引用监听器实现 +class EventPublisher { + private List> listeners = new ArrayList<>(); + + public void addListener(EventListener listener) { + listeners.add(new WeakReference<>(listener)); + } + + public void removeListener(EventListener listener) { + listeners.removeIf(ref -> { + EventListener l = ref.get(); + return l == null || l == listener; + }); + } + + public void cleanupListeners() { + listeners.removeIf(ref -> ref.get() == null); + } + + public int getListenerCount() { + cleanupListeners(); + return listeners.size(); + } + + public void fireEvent(String event) { + List> toRemove = new ArrayList<>(); + + for (WeakReference ref : listeners) { + EventListener listener = ref.get(); + if (listener != null) { + listener.onEvent(event); + } else { + toRemove.add(ref); + } + } + + listeners.removeAll(toRemove); + } +} + +class EventListener { + private String name; + + public EventListener(String name) { + this.name = name; + } + + public void onEvent(String event) { + System.out.println(name + " received event: " + event); + } + + @Override + public String toString() { + return name; + } +} + +/* + * Java引用类型总结： + * + * ┌─────────────┬─────────────┬─────────────┬─────────────┐ + * │ 引用类型 │ GC行为 │ 获取对象 │ 应用场景 │ + * ├─────────────┼─────────────┼─────────────┼─────────────┤ + * │ 强引用 │ 永不回收 │ 直接 │ 常规对象 │ + * │ 软引用 │ 内存不足回收│ get()方法 │ 内存缓存 │ + * │ 弱引用 │ 下次GC回收│ get()方法 │ 缓存/监听器 │ + * │ 虚引用 │ 回收时通知 │ 总是null │ 清理监控 │ + * └─────────────┴─────────────┴─────────────┴─────────────┘ + * + * 最佳实践： + * 1. 默认使用强引用，满足大部分需求 + * 2. 软引用适合实现内存敏感的缓存 + * 3. 弱引用避免循环引用，实现观察者模式 + * 4. 虚引用配合ReferenceQueue进行资源清理 + * 5. ThreadLocal使用完毕后及时remove() + * 6. WeakHashMap适合实现规范化映射 + */ +``` + +**💻 ThreadLocal源码分析**： + +```java +// ThreadLocal内部源码分析 +public class ThreadLocalAnalysis { + + // 模拟ThreadLocal内部实现 + static class ThreadLocalMap { + + // 核心：Entry继承WeakReference，key是弱引用 + static class Entry extends WeakReference> { + Object value; + + Entry(ThreadLocal k, Object v) { + super(k); // key作为弱引用传给父类 + value = v; + } + } + + private Entry[] table; + + // 设置值时的清理逻辑 + private void set(ThreadLocal key, Object value) { + Entry[] tab = table; + int len = tab.length; + int i = key.threadLocalHashCode & (len-1); + + for (Entry e = tab[i]; e != null; e = tab[i = nextIndex(i, len)]) { + ThreadLocal k = e.get(); // 弱引用获取key + + if (k == key) { + e.value = value; + return; + } + + if (k == null) { + // key被回收了，替换陈旧的entry + replaceStaleEntry(key, value, i); + return; + } + } + + tab[i] = new Entry(key, value); + } + + // 清理陈旧entry的逻辑 + private void replaceStaleEntry(ThreadLocal key, Object value, int staleSlot) { + // 清理key为null的entry，防止value内存泄漏 + // ... 复杂的清理逻辑 + } + + private int nextIndex(int i, int len) { + return ((i + 1 < len) ? i + 1 : 0); + } + } + + // 演示ThreadLocal内存泄漏场景 + public static void demonstrateMemoryLeak() { + System.out.println("=== ThreadLocal内存泄漏演示 ==="); + + // 错误使用方式：不调用remove() + class BadThreadLocalUsage { + private ThreadLocal threadLocal = new ThreadLocal<>(); + + public void doWork() { + // 存储大对象 + threadLocal.set(new byte[1024 * 1024]); // 1MB + + // 业务逻辑... + + // 错误：没有调用remove()，在线程池环境中会内存泄漏 + } + } + + // 正确使用方式：及时清理 + class GoodThreadLocalUsage { + private ThreadLocal threadLocal = new ThreadLocal<>(); + + public void doWork() { + try { + // 存储大对象 + threadLocal.set(new byte[1024 * 1024]); + + // 业务逻辑... + + } finally { + // 正确：及时清理，防止内存泄漏 + threadLocal.remove(); + } + } + } + + System.out.println("ThreadLocal最佳实践：使用后及时remove()"); + } + + // 演示弱引用的自动清理效果 + public static void demonstrateWeakReferenceCleanup() { + System.out.println("\n=== 弱引用自动清理演示 ==="); + + ThreadLocal tl = new ThreadLocal<>(); + tl.set("test value"); + + System.out.println("ThreadLocal value: " + tl.get()); + + // 清除ThreadLocal的强引用 + tl = null; + + System.gc(); + Thread.yield(); + + System.out.println("ThreadLocal对象已被回收，key变为null"); + System.out.println("但value可能仍在ThreadLocalMap中，需要后续访问时清理"); + } +} +``` + +**💻 ReferenceQueue使用示例**： + +```java +// ReferenceQueue应用示例 +public class ReferenceQueueExample { + + private static ReferenceQueue queue = new ReferenceQueue<>(); + private static Map, String> cleanupTasks = new ConcurrentHashMap<>(); + + public static void main(String[] args) throws InterruptedException { + // 启动清理线程 + startCleanupThread(); + + // 创建需要清理的资源 + createResourceWithCleanup("Resource1"); + createResourceWithCleanup("Resource2"); + + // 触发GC + System.gc(); + Thread.sleep(1000); + + System.out.println("创建更多资源..."); + createResourceWithCleanup("Resource3"); + + System.gc(); + Thread.sleep(2000); + + System.out.println("程序结束"); + } + + private static void createResourceWithCleanup(String resourceName) { + // 模拟需要清理的资源 + Object resource = new Object() { + @Override + public String toString() { + return resourceName; + } + }; + + // 创建虚引用监控对象回收 + PhantomReference phantomRef = new PhantomReference<>(resource, queue); + cleanupTasks.put(phantomRef, resourceName + " cleanup task"); + + System.out.println("创建资源: " + resourceName); + + // 清除强引用，使对象可被回收 + resource = null; + } + + private static void startCleanupThread() { + Thread cleanupThread = new Thread(() -> { + try { + while (true) { + // 阻塞等待被回收的引用 + Reference ref = queue.remove(); + + // 执行清理任务 + String task = cleanupTasks.remove(ref); + if (task != null) { + System.out.println("执行清理任务: " + task); + // 这里可以执行实际的资源清理逻辑 + // 比如关闭文件、释放网络连接等 + } + } + } catch (InterruptedException e) { + System.out.println("清理线程被中断"); + Thread.currentThread().interrupt(); + } + }); + + cleanupThread.setDaemon(true); + cleanupThread.start(); + System.out.println("清理线程已启动"); + } +} +``` + +### 🎯 引用类型在实际项目中的最佳实践 + +**项目实战经验分享**： + +1. **图片缓存系统**： +```java +public class ImageCache { + private final Map> cache = new ConcurrentHashMap<>(); + + public BufferedImage getImage(String url) { + SoftReference ref = cache.get(url); + BufferedImage image = (ref != null) ? ref.get() : null; + + if (image == null) { + image = loadImageFromUrl(url); + cache.put(url, new SoftReference<>(image)); + } + return image; + } +} +``` + +2. **监听器管理**： +```java +public class EventManager { + private final List> listeners = new ArrayList<>(); + + public void addEventListener(EventListener listener) { + listeners.add(new WeakReference<>(listener)); + } + + public void fireEvent(Event event) { + listeners.removeIf(ref -> { + EventListener listener = ref.get(); + if (listener != null) { + listener.onEvent(event); + return false; + } + return true; // 移除失效的监听器 + }); + } +} +``` + +3. **ThreadLocal最佳实践**： +```java +public class UserContextHolder { + private static final ThreadLocal CONTEXT = new ThreadLocal<>(); + + public static void setUser(UserContext user) { + CONTEXT.set(user); + } + + public static UserContext getUser() { + return CONTEXT.get(); + } + + public static void clear() { + CONTEXT.remove(); // 关键：及时清理 + } +} +``` + +这些引用类型是Java内存管理的精髓，掌握它们对于写出高性能、低内存占用的Java程序至关重要。 + +### 🎯 弱引用了解吗？举例说明在哪里可以用？ + +**📋 面试标准回答**： + +> "弱引用我非常了解，它是Java中一种特殊的引用类型，主要有以下特点和应用： +> +> **弱引用的特点**： +> +> - 不影响对象的生命周期，下次GC时无条件回收 +> - 通过WeakReference.get()方法访问对象，可能返回null +> - 可以配合ReferenceQueue监控对象回收 +> +> **典型应用场景**： +> +> 1. **WeakHashMap**：实现规范化映射，key被回收时自动清理entry +> 2. **ThreadLocal内部实现**：ThreadLocalMap的key使用弱引用，避免内存泄漏 +> 3. **观察者模式**：监听器使用弱引用，防止监听器无法被GC +> 4. **缓存实现**：临时缓存，内存紧张时自动清理 +> 5. **避免循环引用**：父子组件相互引用时，子组件持有父组件的弱引用 +> +> **实际项目中的使用**： +> +> - GUI应用中，组件对监听器的引用 +> - 缓存框架中的临时缓存实现 +> - 对象池中对象的生命周期管理 +> - 防止Handler导致的Activity内存泄漏（Android开发） +> +> 弱引用是解决内存泄漏问题的重要工具，合理使用可以让程序更加健壮。" + +### 🎯 ThreadLocal为什么要使用弱引用？ + +**📋 标准话术**： + +> "ThreadLocal使用弱引用是为了防止内存泄漏，具体原理如下： +> +> **ThreadLocal的内部结构**： +> +> - 每个Thread都有一个ThreadLocalMap +> - ThreadLocalMap的key是ThreadLocal对象的弱引用 +> - value是实际存储的值 +> +> **为什么使用弱引用**： +> +> 1. **防止ThreadLocal对象无法回收**：如果key是强引用，即使ThreadLocal对象不再被使用，由于ThreadLocalMap持有强引用，ThreadLocal对象无法被GC +> +> 2. **自动清理机制**：当ThreadLocal对象被回收后，key变为null，ThreadLocalMap可以识别并清理这些陈旧的entry +> +> 3. **线程长期存活的场景**：在线程池环境中，线程长期存活，如果不清理无用的ThreadLocal，会造成内存泄漏 +> +> **潜在问题**： +> +> - 即使key被回收，value依然存在，仍可能内存泄漏 +> - 最佳实践是使用完ThreadLocal后调用remove()方法 +> +> 这种设计是权衡之后的最优解决方案。" + +### 🎯 什么时候使用软引用？什么时候使用弱引用？ + +**📋 标准话术**： + +> "软引用和弱引用的选择主要基于回收时机和应用场景： +> +> **使用软引用的场景**： +> +> 1. **内存敏感的缓存**：图片缓存、数据缓存等，内存充足时保留，不足时回收 +> 2. **可重建的昂贵对象**：计算结果缓存、编译后的正则表达式等 +> 3. **大对象的临时存储**：文件内容缓存、网络响应缓存 +> 4. **实现LRU之外的缓存策略**：基于内存压力的智能缓存 +> +> **使用弱引用的场景**： +> +> 1. **避免循环引用**：父子组件、观察者模式 +> 2. **监听器管理**：事件监听器、回调函数 +> 3. **实现规范化映射**：WeakHashMap、对象池 +> 4. **线程本地存储**：ThreadLocal的内部实现 +> 5. **临时关联关系**：对象间的弱关联 +> +> **选择原则**： +> +> - 需要内存管理策略时选择软引用 +> - 需要避免强引用导致的问题时选择弱引用 +> - 软引用适合'有用但非必需'的对象 +> - 弱引用适合'引用但不拥有'的关系 +> +> 实际开发中，软引用用于缓存，弱引用用于解耦。" + +### 🎯 ReferenceQueue的作用是什么？ + +**📋 标准话术**： + +> "ReferenceQueue是Java引用类型的监控机制，主要作用： +> +> **核心功能**： +> +> - 当引用的对象被GC回收时，引用对象会被加入到关联的ReferenceQueue中 +> - 提供了一种异步通知机制，让程序知道对象何时被回收 +> - 只有弱引用、软引用、虚引用可以与ReferenceQueue关联 +> +> **典型应用场景**： +> +> 1. **资源清理**：文件句柄、网络连接等需要显式关闭的资源 +> 2. **缓存管理**：监控缓存对象的回收，及时清理相关元数据 +> 3. **内存监控**：统计对象的生命周期，进行性能分析 +> 4. **堆外内存管理**：DirectByteBuffer的清理机制 +> +> **使用模式**： +> +> - 创建引用时关联ReferenceQueue +> - 后台线程轮询队列，处理被回收的引用 +> - 执行清理逻辑，释放相关资源 +> +> 这种机制让Java可以实现类似C++析构函数的功能。" + +--- + +## 🆕 九、Java新特性（Modern Java） + +> **核心思想**：Java 8引入了函数式编程特性，后续版本持续演进，提供了更简洁、高效的编程方式。 + +### 🎯 final、finally、finalize的区别？ + +**📋 标准话术**： + +> "final、finally、finalize是Java中三个容易混淆的关键字，它们有完全不同的用途： +> +> **final关键字**： +> +> - 修饰符，用于限制继承、重写和重新赋值 +> - 修饰类：类不能被继承（如String、Integer） +> - 修饰方法：方法不能被重写 +> - 修饰变量：变量不能被重新赋值（常量） +> - 修饰参数：参数在方法内不能被修改 +> +> **finally语句块**： +> +> - 异常处理机制的一部分，与try-catch配合使用 +> - 无论是否发生异常都会执行（除非JVM退出） +> - 常用于资源清理（关闭文件、数据库连接等） +> - 执行优先级高于try-catch中的return语句 +> +> **finalize()方法**： +> +> - Object类的一个方法，用于垃圾回收前的清理工作 +> - 由垃圾收集器调用，不保证何时调用 +> - Java 9后已标记为过时，不推荐使用 +> - 应该使用try-with-resources或AutoCloseable替代 +> +> 记忆口诀：final限制，finally保证，finalize清理。" + +**💻 代码示例**： + +```java +public class FinalFinallyFinalizeDemo { + + public static void main(String[] args) { + // 1. final演示 + final int CONSTANT = 100; // 常量不可变 + // CONSTANT = 200; // 编译错误 + + final List list = new ArrayList<>(); + list.add("Hello"); // 引用不变，内容可变 + // list = new ArrayList<>(); // 编译错误 + + // 2. finally演示 + try { + System.out.println("try执行"); + return; // 即使return，finally也执行 + } finally { + System.out.println("finally执行"); // 必定执行 + } + + // 3. finalize演示（已过时） + MyResource resource = new MyResource(); + resource = null; + System.gc(); // 建议GC，finalize可能被调用 + + // 4. 推荐替代方案：try-with-resources + try (AutoCloseable res = new MyAutoCloseable()) { + // 使用资源 + } catch (Exception e) { + e.printStackTrace(); + } // 自动关闭，无需finally + } +} + +// final类：不能被继承 +final class ImmutableClass { + public final void finalMethod() { // final方法：不能重写 + System.out.println("不可重写的方法"); + } +} + +// finalize示例（不推荐） +class MyResource { + @Override + protected void finalize() throws Throwable { + System.out.println("finalize被调用"); + super.finalize(); + } +} + +// 推荐的资源管理方式 +class MyAutoCloseable implements AutoCloseable { + @Override + public void close() throws Exception { + System.out.println("资源已关闭"); + } +} + +/* + * final、finally、finalize对比总结： + * + * ┌──────────┬─────────────┬─────────────┬─────────────┬─────────────┐ + * │ 特性 │ 类型 │ 用途 │ 执行时机 │ 推荐程度 │ + * ├──────────┼─────────────┼─────────────┼─────────────┼─────────────┤ + * │ final │ 关键字 │ 限制修饰 │ 编译时 │ 强烈推荐 │ + * │ finally │ 语句块 │ 异常处理/清理│ 运行时 │ 推荐使用 │ + * │finalize │ 方法 │ 垃圾回收 │ GC时(不确定) │ 已弃用 │ + * └──────────┴─────────────┴─────────────┴─────────────┴─────────────┘ + * + * 使用场景： + * - final: 不可变设计、常量定义、防止继承/重写 + * - finally: 资源清理、确保执行的代码 + * - finalize: 已弃用，使用AutoCloseable + try-with-resources + * + * 最佳实践： + * 1. 大量使用final提高代码安全性 + * 2. 用finally确保关键代码执行 + * 3. 避免finalize，改用AutoCloseable + * 4. 利用try-with-resources自动资源管理 + */ +``` + +### 🎯 Lambda表达式中使用外部变量，为什么要final？ + +**📋 标准话术**： + +> "Lambda表达式中使用外部变量必须是final或effectively final，这是由Java的实现机制决定的： +> +> **为什么需要final**： +> +> - Lambda表达式本质上是匿名内部类的语法糖 +> - 内部类访问外部变量时，实际是复制了变量的值 +> - 如果允许修改，会导致内外部变量值不一致的问题 +> - final保证了变量的一致性和线程安全 +> +> **effectively final**： +> +> - 变量没有被显式声明为final，但事实上没有被修改 +> - 编译器会自动识别这种情况 +> - 这种变量也可以在Lambda中使用 +> +> **解决方案**： +> +> - 使用数组或包装类来绕过限制 +> - 使用AtomicInteger等原子类 +> - 将变量提升为实例变量或静态变量 +> +> 这个限制确保了Lambda表达式的安全性和一致性。" + +### 🎯 Lambda表达式是什么？有什么优势？ + +> "Lambda表达式是Java 8引入的重要特性，实现了函数式编程： +> +> **Lambda表达式**： +> +> - 本质是匿名函数，可以作为参数传递 +> - 语法：(参数) -> {方法体} +> - 简化了匿名内部类的书写 +> - 只能用于函数式接口（有且仅有一个抽象方法的接口） +> +> **Stream API**： +> +> - 提供了声明式的数据处理方式 +> - 支持链式调用，代码更简洁 +> - 内置并行处理能力 +> - 常用操作：filter、map、reduce、collect等 +> +> **Optional类**： +> +> - 解决空指针异常问题 +> - 明确表示可能为空的值 +> - 提供函数式风格的空值处理 +> +> **函数式接口**： +> +> - @FunctionalInterface注解标记 +> - 内置接口：Predicate、Function、Consumer、Supplier等 +> - 支持方法引用语法 +> +> 优势：代码更简洁、可读性更强、支持并行处理、函数式编程风格。" + +**💻 代码示例**： + +```java +import java.util.*; +import java.util.function.*; +import java.util.stream.*; + +public class Java8FeaturesDemo { + + public static void main(String[] args) { + // 1. Lambda表达式：简化匿名内部类 + List names = Arrays.asList("Alice", "Bob", "Charlie"); + names.sort((a, b) -> a.length() - b.length()); // 按长度排序 + + // 2. Stream API：函数式数据处理 + List numbers = Arrays.asList(1, 2, 3, 4, 5); + List result = numbers.stream() + .filter(n -> n % 2 == 0) // 筛选偶数 + .map(n -> n * n) // 平方 + .collect(Collectors.toList()); // [4, 16] + + // 3. Optional：避免空指针 + Optional optional = Optional.ofNullable("Hello"); + optional.ifPresent(System.out::println); // 有值时执行 + String value = optional.orElse("Default"); // 无值时默认值 + + // 4. 函数式接口：四大核心接口 + Predicate isEven = n -> n % 2 == 0; // 断言 + Function getLength = String::length; // 函数 + Consumer print = System.out::println; // 消费者 + Supplier random = Math::random; // 供应者 + + // 5. 方法引用：四种形式 + names.forEach(System.out::println); // 静态方法引用 + names.stream().map(String::length); // 实例方法引用 + names.stream().map(String::new); // 构造器引用 + names.stream().filter(String::isEmpty); // 任意对象方法引用 + } +} + +/* + * Java 8核心特性总结： + * + * 1. Lambda表达式：(参数) -> 表达式 + * 2. Stream API：数据处理管道 + * 3. Optional：空值安全 + * 4. 函数式接口：@FunctionalInterface + * 5. 方法引用：类::方法 + */ +``` + +### 🎯 有用过JDK17吗，有什么新特性？ + +JDK 17 是 Java 的长期支持版本（LTS），发布于 2021 年，带来了许多新特性和改进，以下是一些重要的更新： + +1. **Sealed Classes（封闭类）** + +JDK 17 引入了封闭类（Sealed Classes），它允许你限制哪些类可以继承或实现一个特定的类或接口。通过这种方式，开发者可以更好地控制继承结构。封闭类通过 `permits` 关键字指定哪些类可以继承。 + +```java +public abstract sealed class Shape permits Circle, Rectangle { +} +``` + +2. **Pattern Matching for Switch (预览)** + +在 JDK 17 中，switch 语句的模式匹配功能被引入（作为预览功能）。这使得 switch 语句不仅可以根据值进行匹配，还可以根据类型进行匹配。以前的 switch 仅支持基本类型或枚举，而新特性扩展了其灵活性。 + +```java +static String formatterPatternSwitch(Object obj) { + return switch (obj) { + case Integer i -> String.format("int %d", i); + case Long l -> String.format("long %d", l); + case String s -> String.format("String %s", s); + default -> obj.toString(); + }; +} +``` + +3. **Records（记录类型）增强** + +Records 是 JDK 16 引入的特性，但在 JDK 17 中得到了进一步增强。Records 提供了一种简洁的方式来创建不可变的数据类，它自动生成构造函数、`equals()`、`hashCode()` 和 `toString()` 方法。 + +```java +public record Point(int x, int y) {} +``` + +4. **强封装的 Java 内部 API** + +JDK 17 强化了对 Java 内部 API 的封装，默认情况下不再允许非公共 API 访问其他模块的内部 API。通过此特性，Java 模块化变得更加安全，防止非预期的依赖。 + +5. **Foreign Function & Memory API (外部函数和内存 API)** + +JDK 17 通过新的外部函数和内存 API 预览功能，允许 Java 程序直接调用非 Java 代码（如本地代码）。这一特性极大增强了与原生系统库的集成能力。 + +```java +MemorySegment segment = MemorySegment.allocateNative(100); +``` + +6. **macOS 上的 AArch64 支持** + +随着 Apple M1 处理器的推出，JDK 17 为 macOS 引入了对 AArch64 架构的支持。开发者现在可以在 macOS 的 ARM 平台上更高效地运行 Java 程序。 + +7. **Deprecation for Removal of RMI Activation** + +RMI Activation（远程方法调用激活机制）已经被弃用并计划在未来移除。这一功能的移除是因为它在现代分布式系统中较少使用，并且存在更好的替代方案。 + +8. **Vector API (预览)** + +JDK 17 进一步预览了 Vector API，允许在 Java 中进行向量运算。Vector API 利用 SIMD（单指令多数据）硬件指令，可以实现高性能的数学计算。这对科学计算和机器学习任务尤为重要。 + +```java +VectorSpecies SPECIES = FloatVector.SPECIES_256; +``` + +9. **简化的强制性 NullPointerException 信息** + +JDK 17 改进了 `NullPointerException` 的错误信息，帮助开发者更快定位问题。例如，如果你访问空引用对象的字段，JDK 17 会明确指出是哪个字段导致了异常。 + +10. **默认垃圾回收器 ZGC 和 G1 的改进** + +JDK 17 对 ZGC（Z Garbage Collector）和 G1 垃圾回收器进行了优化，以进一步降低垃圾收集的延迟，并提高应用程序的整体性能。 + +这些新特性和改进使得 JDK 17 成为一个功能丰富、性能优越的版本，特别适合长期支持和大规模企业级应用。 + +```java +// JDK17新特性示例 +public class JDK17FeaturesDemo { + + public static void main(String[] args) { + // 1. Switch表达式 + String dayType = switch (5) { + case 1, 7 -> "Weekend"; + case 2, 3, 4, 5, 6 -> "Weekday"; + default -> "Invalid"; + }; + + // 2. Text Blocks：多行字符串 + String json = """ + { + "name": "张三", + "age": 25, + "city": "北京" + } + """; + + // 3. Pattern Matching：模式匹配 + Object obj = "Hello"; + if (obj instanceof String str && str.length() > 3) { + System.out.println("长字符串: " + str.toUpperCase()); + } + + // 4. Records：数据类 + Person person = new Person("李四", 30); + System.out.println(person.name() + " is " + person.age()); + + // 5. Sealed Classes：密封类 + Shape circle = new Circle(5.0); + double area = calculateArea(circle); + } + + // Records自动生成构造器、getter、equals、hashCode、toString + public record Person(String name, int age) {} + + // Sealed Classes：限制继承 + public static sealed interface Shape permits Circle, Rectangle { + double area(); + } + + public static final class Circle implements Shape { + private final double radius; + public Circle(double radius) { this.radius = radius; } + public double area() { return Math.PI * radius * radius; } + } + + public static final class Rectangle implements Shape { + private final double width, height; + public Rectangle(double width, double height) { + this.width = width; this.height = height; + } + public double area() { return width * height; } + } + + // Pattern Matching with Sealed Classes + public static double calculateArea(Shape shape) { + return switch (shape) { + case Circle c -> c.area(); + case Rectangle r -> r.area(); + // 密封类确保覆盖所有情况，无需default + }; + } +} +``` + + +--- + +## 🏆 面试准备速查表 + +### 📋 Java基础高频面试题清单 + +| **知识点** | **核心问题** | **关键话术** | **代码要点** | +| -------------- | ------------------------------ | -------------------------------- | ------------------------------ | +| **面向对象** | 三大特性、抽象类vs接口、内部类 | 封装继承多态、is-a关系、访问控制 | 继承重写、接口实现、内部类访问 | +| **数据类型** | 基本类型vs包装类、==vs equals | 栈堆存储、自动装箱、缓存机制 | Integer缓存、equals重写 | +| **字符串** | String不可变性、三者区别 | 常量池、性能对比、线程安全 | StringBuilder使用、intern方法 | +| **异常处理** | 异常体系、处理机制、最佳实践 | 检查型vs非检查型、异常链 | try-with-resources、自定义异常 | +| **Java引用类型** | 四种引用类型、弱引用应用场景 | 强软弱虚、内存管理、GC行为 | WeakHashMap、ThreadLocal原理 | + +### 🎯 面试回答技巧 + +1. **STAR法则**：Situation（背景）→ Task（任务）→ Action（行动）→ Result（结果） +2. **层次递进**：基本概念 → 深入原理 → 实际应用 → 性能优化 +3. **举例说明**：理论结合实际项目经验 +4. **源码引用**：适当提及源码实现，体现深度 + +### ⚠️ 常见面试陷阱 + +- ⚠️ **概念混淆**：抽象类和接口的选择场景 +- ⚠️ **性能陷阱**：String拼接、装箱拆箱的性能影响 +- ⚠️ **空指针**：equals方法的参数顺序、null处理 +- ⚠️ **异常滥用**：不要用异常控制业务流程 + + + +--- + +## 🏆 面试准备速查表 + +### 📋 Java基础高频面试题清单 + +| **知识点** | **核心问题** | **关键话术** | **代码要点** | +| -------------- | ------------------------------ | -------------------------------- | ------------------------------ | +| **面向对象** | 三大特性、抽象类vs接口、内部类 | 封装继承多态、is-a关系、访问控制 | 继承重写、接口实现、内部类访问 | +| **数据类型** | 基本类型vs包装类、==vs equals | 栈堆存储、自动装箱、缓存机制 | Integer缓存、equals重写 | +| **字符串** | String不可变性、三者区别 | 常量池、性能对比、线程安全 | StringBuilder使用、intern方法 | +| **异常处理** | 异常体系、处理机制、最佳实践 | 检查型vs非检查型、异常链 | try-with-resources、自定义异常 | +| **Java引用类型** | 四种引用类型、弱引用应用场景 | 强软弱虚、内存管理、GC行为 | WeakHashMap、ThreadLocal原理 | + +### 🎯 面试回答技巧 + +1. **STAR法则**：Situation（背景）→ Task（任务）→ Action（行动）→ Result（结果） +2. **层次递进**：基本概念 → 深入原理 → 实际应用 → 性能优化 +3. **举例说明**：理论结合实际项目经验 +4. **源码引用**：适当提及源码实现，体现深度 + +### ⚠️ 常见面试陷阱 + +- ⚠️ **概念混淆**：抽象类和接口的选择场景 +- ⚠️ **性能陷阱**：String拼接、装箱拆箱的性能影响 +- ⚠️ **空指针**：equals方法的参数顺序、null处理 +- ⚠️ **异常滥用**：不要用异常控制业务流程 + diff --git a/docs/interview/Java-Collections-FAQ.md b/docs/interview/Java-Collections-FAQ.md new file mode 100644 index 0000000000..7c6531e1a5 --- /dev/null +++ b/docs/interview/Java-Collections-FAQ.md @@ -0,0 +1,1238 @@ +--- +title: 「直击面试」—— Java 集合，你肯定也会被问到这些 +date: 2021-05-31 +tags: + - Java + - Interview +categories: Interview + +--- + +![](https://img.starfish.ink/common/faq-banner.png) + +> 文章收录在 GitBook [JavaKeeper](https://github.com/Jstarfish/JavaKeeper) ，N线互联网开发必备技能兵器谱 +> +> 作为一位小菜 "一面面试官"，面试过程中，我肯定会问 Java 集合的内容，同时作为求职者，也肯定会被问到集合，所以整理下 Java 集合面试题 +> +> Java集合框架是Java编程的核心基础，也是面试中出现频率最高的技术点之一。从基础的List、Set、Map到高级的并发集合，从底层数据结构到性能优化，集合框架涵盖了数据结构、算法复杂度、线程安全等多个维度的知识点。掌握集合框架，不仅是Java开发的必备技能，更是解决实际业务问题的重要工具。 + +面向对象语言对事物的体现都是以对象的形式，所以为了方便对多个对象的操作，需要将对象进行存储，集合就是存储对象最常用的一种方式，也叫容器。 + +![img](https://www.runoob.com/wp-content/uploads/2014/01/2243690-9cd9c896e0d512ed.gif) + +从上面的集合框架图可以看到，Java 集合框架主要包括两种类型的容器 + +- 一种是集合（Collection），存储一个元素集合 +- 另一种是图（Map），存储键/值对映射。 + +Collection 接口又有 3 种子类型，List、Set 和 Queue，再下面是一些抽象类，最后是具体实现类，常用的有 ArrayList、LinkedList、HashSet、LinkedHashSet、HashMap、LinkedHashMap 等等。 + +集合框架是一个用来代表和操纵集合的统一架构。所有的集合框架都包含如下内容： + +- **接口**：是代表集合的抽象数据类型。例如 Collection、List、Set、Map 等。之所以定义多个接口，是为了以不同的方式操作集合对象 +- **实现（类）**：是集合接口的具体实现。从本质上讲，它们是可重复使用的数据结构，例如：ArrayList、LinkedList、HashSet、HashMap。 +- **算法**：是实现集合接口的对象里的方法执行的一些有用的计算，例如：搜索和排序。这些算法被称为多态，那是因为相同的方法可以在相似的接口上有着不同的实现。 + +--- + +## 🗺️ 知识导航 + +### 🏷️ 核心知识分类 + +1. **📚 集合框架概览**：Collection vs Map、集合继承体系、Iterator模式、泛型使用 +2. **📋 List接口实现**：ArrayList vs LinkedList、Vector、性能对比、使用场景 +3. **🔢 Set接口实现**：HashSet vs TreeSet vs LinkedHashSet、去重原理、排序机制 +4. **🗂️ Map接口实现**：HashMap vs TreeMap vs LinkedHashMap、hash冲突处理、红黑树 +5. **🔒 线程安全集合**：ConcurrentHashMap、CopyOnWriteArrayList、同步容器vs并发容器 +6. **⚡ 性能与优化**：时间复杂度分析、空间复杂度、扩容机制、最佳实践 + +--- + +## 📚 一、集合框架概览 + +### 1.1 常用集合介绍 + +**🎯 说说常用的集合有哪些吧？** + +> Collection 有什么子接口、有哪些具体的实现 + +Map 接口和 Collection 接口是所有集合框架的父接口： + +1. **Collection 接口的子接口包括**：Set、List、Queue +2. **List** 是有序的允许有重复元素的 Collection，实现类主要有：ArrayList、LinkedList、Stack以及Vector等 +3. **Set** 是一种不包含重复元素且无序的Collection，实现类主要有：HashSet、TreeSet、LinkedHashSet等 +4. **Map** 没有继承 Collection 接口，Map 提供 key 到 value 的映射。实现类主要有：HashMap、TreeMap、Hashtable、ConcurrentHashMap 以及 Properties 等 + +### 1.2 基础概念题 + +**🎯 说说Java集合框架的整体架构？** + +``` +Java集合框架主要分为两大体系： +1. Collection体系： + - List（有序，可重复）：ArrayList、LinkedList、Vector + - Set（无序，不重复）：HashSet、TreeSet、LinkedHashSet + - Queue（队列）：LinkedList、PriorityQueue、ArrayDeque + +2. Map体系（键值对）： + - HashMap、TreeMap、LinkedHashMap、Hashtable、ConcurrentHashMap + +核心接口关系： +- Iterable -> Collection -> List/Set/Queue +- Map独立体系 +- 所有集合都实现了Iterator模式 +``` + +**深入扩展：** + +- Collection继承了Iterable接口，支持foreach语法 +- Map不继承Collection，因为它存储的是键值对 +- Collections工具类提供了大量静态方法操作集合 + +**🎯 Collection和Collections的区别？** + +``` +Collection： +- 是一个接口，集合框架的根接口 +- 定义了集合的基本操作方法 +- 被List、Set、Queue等接口继承 + +Collections： +- 是一个工具类，提供静态方法 +- 包含排序、搜索、同步等操作 +- 如sort()、binarySearch()、synchronizedList()等 +``` + +**🎯 Hash冲突及解决办法？** + +解决哈希冲突的方法一般有：开放定址法、链地址法（拉链法）、再哈希法、建立公共溢出区等方法。 + +- **开放定址法**：从发生冲突的那个单元起，按照一定的次序，从哈希表中找到一个空闲的单元。然后把发生冲突的元素存入到该单元的一种方法。开放定址法需要的表长度要大于等于所需要存放的元素。 + +- **链接地址法（拉链法）**：是将哈希值相同的元素构成一个同义词的单链表，并将单链表的头指针存放在哈希表的第i个单元中，查找、插入和删除主要在同义词链表中进行。（链表法适用于经常进行插入和删除的情况） + +- **再哈希法**：就是同时构造多个不同的哈希函数： Hi = RHi(key) i= 1,2,3 … k; 当H1 = RH1(key) 发生冲突时，再用H2 = RH2(key) 进行计算，直到冲突不再产生，这种方法不易产生聚集，但是增加了计算时间 + +- **建立公共溢出区**：将哈希表分为公共表和溢出表，当溢出发生时，将所有溢出数据统一放到溢出区 + +### 1.3 Iterator模式题 + +**🎯 Iterator和ListIterator的区别？** + +``` +Iterator（单向迭代）： +- 只能向前遍历（hasNext、next） +- 支持remove操作 +- 适用于所有Collection + +ListIterator（双向迭代）： +- 支持双向遍历（hasNext、next、hasPrevious、previous） +- 支持add、set、remove操作 +- 只适用于List集合 +- 可以获取当前位置索引 +``` + +**代码示例：** + +```java +List list = new ArrayList<>(); +list.add("A"); +list.add("B"); + +// Iterator - 单向 +Iterator it = list.iterator(); +while(it.hasNext()) { + String item = it.next(); + if("A".equals(item)) { + it.remove(); // 安全删除 + } +} + +// ListIterator - 双向 +ListIterator lit = list.listIterator(); +while(lit.hasNext()) { + String item = lit.next(); + lit.set(item + "_modified"); // 修改元素 +} +``` + +--- + +## 📋 二、List接口实现 + +### 2.1 ArrayList vs Vector + +**🎯 ArrayList和Vector的区别？** + +**相同点**： + +- ArrayList 和 Vector 都是继承了相同的父类和实现了相同的接口（都实现了List，有序、允许重复和null） + + ```java + extends AbstractList + implements List, RandomAccess, Cloneable, java.io.Serializable + ``` + +- 底层都是数组（Object[]）实现的 + +- 初始默认长度都为**10** + +**不同点**： + +| **特性** | **ArrayList** | **Vector** | +| ------------ | ------------- | ------------------ | +| **线程安全** | 否 | 是（synchronized） | +| **性能** | 高 | 较低（同步开销） | +| **扩容机制** | 1.5倍 | 2倍 | +| **出现版本** | JDK 1.2 | JDK 1.0 | +| **迭代器** | fail-fast | fail-fast | + +**扩容机制详解**： + +- **ArrayList 的 grow()**，在满足扩容条件时、ArrayList以**1.5** 倍的方式在扩容（oldCapacity >> **1** ，右移运算，相当于除以 2，结果为二分之一的 oldCapacity） + +```java +private void grow(int minCapacity) { + int oldCapacity = elementData.length; + //newCapacity = oldCapacity + 0.5*oldCapacity,此处扩容0.5倍 + int newCapacity = oldCapacity + (oldCapacity >> 1); + if (newCapacity - minCapacity < 0) + newCapacity = minCapacity; + if (newCapacity - MAX_ARRAY_SIZE > 0) + newCapacity = hugeCapacity(minCapacity); + elementData = Arrays.copyOf(elementData, newCapacity); +} +``` + +- **Vector 的 grow()**，Vector 比 ArrayList多一个属性，扩展因子capacityIncrement，可以扩容大小。当扩容容量增量大于**0**时、新数组长度为原数组长度**+**扩容容量增量、否则新数组长度为原数组长度的**2**倍 + +```java +private void grow(int minCapacity) { + int oldCapacity = elementData.length; + int newCapacity = oldCapacity + ((capacityIncrement > 0) ? + capacityIncrement : oldCapacity); + if (newCapacity - minCapacity < 0) + newCapacity = minCapacity; + if (newCapacity - MAX_ARRAY_SIZE > 0) + newCapacity = hugeCapacity(minCapacity); + elementData = Arrays.copyOf(elementData, newCapacity); +} +``` + +**使用建议**： + +- 单线程环境：优先使用ArrayList +- 多线程环境：使用Collections.synchronizedList()或CopyOnWriteArrayList + +### 2.2 ArrayList深度解析 + +**🎯 ArrayList的底层实现原理？** + +``` +底层数据结构： +- 基于数组实现（Object[] elementData） +- 默认初始容量为10 +- 支持动态扩容 + +扩容机制： +- 当容量不足时，扩容至原容量的1.5倍 +- 使用Arrays.copyOf()复制数组 +- 扩容是耗时操作，建议预估容量 + +核心特性： +- 随机访问：O(1)时间复杂度 +- 插入/删除：O(n)时间复杂度（需要移动元素） +- 允许null值，允许重复元素 +- 线程不安全 +``` + +**源码关键点：** + +```java +// 默认容量 +private static final int DEFAULT_CAPACITY = 10; + +// 扩容方法 +private void grow(int minCapacity) { + int oldCapacity = elementData.length; + int newCapacity = oldCapacity + (oldCapacity >> 1); // 1.5倍扩容 + if (newCapacity - minCapacity < 0) + newCapacity = minCapacity; + elementData = Arrays.copyOf(elementData, newCapacity); +} +``` + +**🎯 ArrayList和Vector的区别？** + +| **特性** | **ArrayList** | **Vector** | +| ------------ | ------------- | ------------------ | +| **线程安全** | 否 | 是（synchronized） | +| **性能** | 高 | 较低（同步开销） | +| **扩容机制** | 1.5倍 | 2倍 | +| **出现版本** | JDK 1.2 | JDK 1.0 | +| **迭代器** | fail-fast | fail-fast | + +**使用建议：** + +- 单线程环境：优先使用ArrayList +- 多线程环境：使用Collections.synchronizedList()或CopyOnWriteArrayList + +### 2.2 LinkedList深度解析 + +**🎯 LinkedList的底层实现原理？** + +``` +底层数据结构： +- 双向链表实现 +- 每个节点包含data、next、prev三个字段 +- 维护first和last指针 + +核心特性： +- 插入/删除：O(1)时间复杂度（已知位置） +- 随机访问：O(n)时间复杂度（需要遍历） +- 实现了List和Deque接口 +- 允许null值，允许重复元素 +- 线程不安全 +``` + +**节点结构：** + +```java +private static class Node { + E item; + Node next; + Node prev; + + Node(Node prev, E element, Node next) { + this.item = element; + this.next = next; + this.prev = prev; + } +} +``` + +**🎯 ArrayList vs LinkedList什么时候使用？** + +- 是否保证线程安全： ArrayList 和 LinkedList 都是不同步的，也就是不保证线程安全； +- **底层数据结构**： Arraylist 底层使用的是 Object 数组；LinkedList 底层使用的是**双向循环链表**数据结构； +- **插入和删除是否受元素位置的影响：** + - **ArrayList 采用数组存储，所以插入和删除元素的时间复杂度受元素位置的影响。** 比如：执行 `add(E e)`方法的时候， ArrayList 会默认在将指定的元素追加到此列表的末尾，这种情况时间复杂度就是O(1)。但是如果要在指定位置 i 插入和删除元素的话（ `add(intindex,E element)`）时间复杂度就为 O(n-i)。因为在进行上述操作的时候集合中第 i 和第 i 个元素之后的(n-i)个元素都要执行向后位/向前移一位的操作。 + - **LinkedList 采用链表存储，所以插入，删除元素时间复杂度不受元素位置的影响，都是近似 $O(1)$，而数组为近似 $O(n)$。** + - ArrayList 一般应用于查询较多但插入以及删除较少情况，如果插入以及删除较多则建议使用 LinkedList +- **是否支持快速随机访问**： LinkedList 不支持高效的随机元素访问，而 ArrayList 实现了 RandomAccess 接口，所以有随机访问功能。快速随机访问就是通过元素的序号快速获取元素对象(对应于 `get(intindex)`方法)。 +- **内存空间占用**： ArrayList 的空间浪费主要体现在在 list 列表的结尾会预留一定的容量空间，而 LinkedList 的空间花费则体现在它的每一个元素都需要消耗比 ArrayList 更多的空间（因为要存放直接后继和直接前驱以及数据）。 + +**使用场景：** + +```java +// 频繁随机访问 - 使用ArrayList +List list1 = new ArrayList<>(); +String item = list1.get(index); // O(1) + +// 频繁插入删除 - 使用LinkedList +List list2 = new LinkedList<>(); +list2.add(0, "item"); // 头部插入 O(1) + +// 作为队列使用 - LinkedList实现了Deque +Deque queue = new LinkedList<>(); +queue.offer("item"); +queue.poll(); +``` + +--- + +## 🔢 三、Set接口实现 + +### 3.1 HashSet深度解析 + +**🎯 HashSet的底层实现原理？** + +``` +底层实现： +- 基于HashMap实现 +- 元素作为HashMap的key，value为固定的PRESENT对象 +- 利用HashMap的key唯一性保证Set的不重复特性 + +去重原理： +1. 计算元素的hashCode() +2. 根据hash值定位存储位置 +3. 如果位置为空，直接存储 +4. 如果位置有元素，调用equals()比较 +5. equals()返回true表示重复，不添加 + +核心特性： +- 无序（不保证插入顺序） +- 不允许重复元素 +- 允许一个null值 +- 线程不安全 +``` + +**源码关键点：** + +```java +public class HashSet { + private transient HashMap map; + private static final Object PRESENT = new Object(); + + public boolean add(E e) { + return map.put(e, PRESENT)==null; + } +} +``` + +**🎯 如何保证自定义对象在HashSet中不重复？** + +``` +必须重写hashCode()和equals()方法： + +1. equals()相等的对象，hashCode()必须相等 +2. hashCode()相等的对象，equals()不一定相等 +3. 重写时需要考虑所有参与比较的字段 + +正确示例： +``` + +```java +public class Person { + private String name; + private int age; + + @Override + public boolean equals(Object obj) { + if (this == obj) return true; + if (obj == null || getClass() != obj.getClass()) return false; + Person person = (Person) obj; + return age == person.age && Objects.equals(name, person.name); + } + + @Override + public int hashCode() { + return Objects.hash(name, age); + } +} +``` + +### 3.2 TreeSet深度解析 + +**🎯 TreeSet的底层实现原理？** + +``` +底层实现： +- 基于TreeMap实现（红黑树） +- 元素作为TreeMap的key +- 利用红黑树的特性保证有序 + +排序机制： +1. 自然排序：元素实现Comparable接口 +2. 定制排序：构造时传入Comparator + +核心特性： +- 有序存储（升序） +- 不允许重复元素 +- 不允许null值 +- 线程不安全 +- 查找、插入、删除：O(log n) +``` + +**使用示例：** + +```java +// 自然排序 +Set treeSet1 = new TreeSet<>(); +treeSet1.add(3); +treeSet1.add(1); +treeSet1.add(2); +// 输出：[1, 2, 3] + +// 定制排序 +Set treeSet2 = new TreeSet<>((s1, s2) -> s2.compareTo(s1)); +treeSet2.add("b"); +treeSet2.add("a"); +treeSet2.add("c"); +// 输出：[c, b, a] +``` + +### 3.3 LinkedHashSet深度解析 + +**🎯 LinkedHashSet的特点和使用场景？** + +``` +底层实现： +- 继承HashSet，基于LinkedHashMap实现 +- 在HashSet基础上维护了插入顺序 + +核心特性： +- 保持插入顺序 +- 不允许重复元素 +- 允许一个null值 +- 性能略低于HashSet（维护链表开销） + +使用场景： +- 需要去重且保持插入顺序 +- 遍历顺序要求可预测 +``` + +**三种Set对比：** + +| **特性** | **HashSet** | **LinkedHashSet** | **TreeSet** | +| -------------- | ----------- | ----------------- | --------------- | +| **底层实现** | HashMap | LinkedHashMap | TreeMap(红黑树) | +| **是否有序** | 无序 | 插入顺序 | 自然/定制排序 | +| **允许null** | 是 | 是 | 否 | +| **时间复杂度** | O(1) | O(1) | O(log n) | +| **使用场景** | 快速去重 | 去重+保序 | 排序去重 | + +--- + +## 🗂️ 四、Map接口实现 + +### 4.1 HashMap基础问题 + +**🎯 HashMap中key和value可以为null吗？允许几个为null呀？** + +1. **键（Key）可以为`null`**：`HashMap`允许一个键为`null`。当使用`null`作为键时，这个键总是被存储在`HashMap`的第0个桶（bucket）中。 +2. **值（Value）可以为`null`**：`HashMap`同样允许值为`null`。你可以将任何键映射为`null`值。 +3. **允许的`null`数量**：在`HashMap`中，**只有一个键可以是`null`**。因为`HashMap`内部使用键的`hashCode()`来确定键值对的存储位置，而`null`的`hashCode()`值为0。 +4. **对性能的影响**：虽然`HashMap`允许键或值为`null`，但频繁使用`null`键可能会影响性能。 + +**🎯 HashMap的Key需要重写hashCode()和equals()吗？** + +**1. HashMap 的存储逻辑** + +- **put(K,V)** 时： + 1. 先调用 `key.hashCode()` 计算哈希值，决定存放在哪个 **桶（bucket）**。 + 2. 如果桶里已有元素，会调用 `equals()` 比较，判断是否是同一个 key（更新 value）还是哈希冲突（拉链/红黑树存储）。 +- **get(K)** 时： + 1. 先算出 `key.hashCode()` 定位到桶。 + 2. 再用 `equals()` 在桶里逐个比对，找到目标值。 + +**2. 为什么要重写？** + +如果不重写 `hashCode()` 和 `equals()` 方法，默认实现会使用对象的内存地址来计算哈希码和比较对象。这将导致逻辑上相等的对象（例如内容相同的两个实例）具有不同的哈希码，无法正确存储和查找。 + +👉 结果就是：**两个"内容相同"的 key 会被认为是不同的 key**，导致存取不一致。 + +**正确做法**： + +```java +public class Person { + private String name; + private int age; + + @Override + public boolean equals(Object obj) { + if (this == obj) return true; + if (obj == null || getClass() != obj.getClass()) return false; + Person person = (Person) obj; + return age == person.age && Objects.equals(name, person.name); + } + + @Override + public int hashCode() { + return Objects.hash(name, age); + } +} +``` + +**必须遵循的规则**： + +1. equals()相等的对象，hashCode()必须相等 +2. hashCode()相等的对象，equals()不一定相等 +3. 重写时需要考虑所有参与比较的字段 + +### 4.2 HashMap深度解析 + +**🎯 HashMap的底层实现原理？（重点）** + +``` +JDK 1.8之前：数组 + 链表 +JDK 1.8之后：数组 + 链表 + 红黑树 + +核心结构： +- Node数组（哈希桶） +- 链表（解决hash冲突） +- 红黑树（链表长度>8时转换，提升查询效率） + +关键参数： +- 默认初始容量：16 +- 默认负载因子：0.75 +- 树化阈值：8 +- 反树化阈值：6 +``` + +**put操作流程：** + +```java +1. 计算key的hash值 +2. 根据hash值计算数组索引：(n-1) & hash +3. 如果位置为空，直接创建节点 +4. 如果位置有元素： + - key相等，覆盖value + - key不等，添加到链表尾部 + - 链表长度>8且数组长度>=64，转为红黑树 +5. 元素数量超过阈值，触发扩容 +``` + +**🎯 HashMap的扩容机制？** + +``` +扩容时机： +- 元素数量 > 容量 * 负载因子（默认16 * 0.75 = 12） + +扩容过程： +1. 容量扩大为原来的2倍 +2. 重新计算所有元素的位置 +3. 元素要么在原位置，要么在原位置+原容量 + +JDK 1.8优化： +- 使用高低位链表优化rehash +- 避免了链表倒置问题 +``` + +**扩容优化代码：** + +```java +// JDK 1.8 扩容优化 +Node loHead = null, loTail = null; // 低位链表 +Node hiHead = null, hiTail = null; // 高位链表 + +do { + next = e.next; + if ((e.hash & oldCap) == 0) { + // 原位置 + if (loTail == null) loHead = e; + else loTail.next = e; + loTail = e; + } else { + // 原位置 + oldCap + if (hiTail == null) hiHead = e; + else hiTail.next = e; + hiTail = e; + } +} while ((e = next) != null); +``` + +**🎯 HashMap为什么线程不安全？** + +``` +线程不安全的表现： +1. 数据丢失：多线程put时可能覆盖 +2. 死循环：JDK 1.7扩容时链表可能形成环 +3. 数据不一致：get时可能获取到不完整的数据 + +JDK 1.7死循环原因： +- 扩容时采用头插法 +- 多线程环境下可能形成循环链表 +- CPU使用率飙升至100% + +JDK 1.8改进： +- 采用尾插法 +- 避免了死循环问题 +- 但仍然存在数据丢失问题 +``` + +**解决方案：** + +```java +// 1. Hashtable（性能差） +Map map1 = new Hashtable<>(); + +// 2. Collections.synchronizedMap（性能差） +Map map2 = Collections.synchronizedMap(new HashMap<>()); + +// 3. ConcurrentHashMap（推荐） +Map map3 = new ConcurrentHashMap<>(); +``` + +**🎯 HashMap的长度为什么是2的幂次方？** + +`HashMap` 中的数组（通常称为"桶"数组）长度设计为2的幂次方有几个原因： + +1. **快速取模运算**：HashMap 中桶数组的大小 length 总是 2 的幂，此时，`h & (table.length -1)` 等价于对 length 取模 `h%length`。但取模的计算效率没有位运算高，所以这是一个优化。 + +2. **减少哈希碰撞**：使用2的幂次方作为数组长度可以使得元素在数组中分布更加均匀，这减少了哈希碰撞的可能性。 + +3. **动态扩容**：`HashMap`在需要扩容时，通常会增加到当前容量的两倍。如果当前容量已经是2的幂次方，增加到两倍后仍然是2的幂次方。 + +4. **避免数据迁移**：当数组长度为2的幂次方时，当进行扩容和重新哈希时，可以通过简单的位运算来确定元素在新数组中的位置，而不需要重新计算哈希码。 + +**🎯 为什么JDK1.8中HashMap从头插入改成尾插入？** + +在Java 8中，`HashMap`的插入策略从头部插入（head insertion）改为尾部插入（tail insertion），主要原因： + +1. **避免死循环**：JDK1.7中扩容时，每个元素的rehash之后，都会插入到新数组对应索引的链表头，所以这就导致原链表顺序为A->B->C，扩容之后，rehash之后的链表可能为C->B->A，元素的顺序发生了变化。在并发场景下，**扩容时**可能会出现循环链表的情况。 + +2. **保持顺序**：而JDK1.8从头插入改成尾插入，元素的顺序不变，避免出现循环链表的情况。 + +3. **提高并发安全性**：尾插入策略在并发环境下更加稳定，虽然HashMap仍然不是线程安全的，但减少了一些潜在问题。 + +**🎯 HashMap：JDK1.7 VS JDK1.8主要区别？** + +| **不同** | **JDK 1.7** | **JDK 1.8** | +| ------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | +| **存储结构** | 数组 + 链表 | 数组 + 链表 + 红黑树 | +| **初始化方式** | 单独函数：inflateTable() | 直接集成到了扩容函数resize()中 | +| **hash值计算** | 扰动处理 = 9次扰动 = 4次位运算 + 5次异或运算 | 扰动处理 = 2次扰动 = 1次位运算 + 1次异或运算 | +| **存放数据规则** | 无冲突时，存放数组；冲突时，存放链表 | 无冲突时，存放数组；冲突 & 链表长度 < 8：存放单链表；冲突 & 链表长度 > 8：树化并存放红黑树 | +| **插入数据方式** | 头插法（先将原位置的数据移到后1位，再插入数据到该位置） | 尾插法（直接插入到链表尾部/红黑树） | +| **扩容后位置计算** | 全部按照原来方法进行计算（即hashCode -> 扰动函数 -> (h&length-1)） | 按照扩容后的规律计算（即扩容后的位置=原位置 or 原位置 + 旧容量） | + +### 4.2 ConcurrentHashMap深度解析 + +**🎯 ConcurrentHashMap的实现原理？（重点）** + +``` +JDK 1.7实现（分段锁）： +- Segment数组 + HashEntry数组 +- 每个Segment继承ReentrantLock +- 默认16个Segment，最大并发度16 +- 锁粒度：Segment级别 + +JDK 1.8实现（CAS + synchronized）： +- Node数组 + 链表/红黑树 +- 取消Segment概念 +- 使用CAS + synchronized实现 +- 锁粒度：Node级别（更细粒度） +``` + +**JDK 1.8 put操作：** + +```java +final V putVal(K key, V value, boolean onlyIfAbsent) { + // 1. 计算hash值 + int hash = spread(key.hashCode()); + + for (Node[] tab = table;;) { + // 2. 如果数组为空，初始化 + if (tab == null || (n = tab.length) == 0) + tab = initTable(); + // 3. 如果位置为空，CAS插入 + else if ((f = tabAt(tab, i = (n - 1) & hash)) == null) { + if (casTabAt(tab, i, null, new Node(hash, key, value))) + break; + } + // 4. 如果正在扩容，帮助扩容 + else if ((fh = f.hash) == MOVED) + tab = helpTransfer(tab, f); + // 5. 否则synchronized锁住头节点 + else { + synchronized (f) { + // 链表或红黑树插入逻辑 + } + } + } +} +``` + +**🎯 ConcurrentHashMap的优势？** + +| **特性** | **Hashtable** | **Collections.synchronizedMap** | **ConcurrentHashMap** | +| ---------- | ------------------ | ------------------------------- | --------------------- | +| **锁机制** | 方法级synchronized | 方法级synchronized | 分段锁/节点锁 | +| **并发度** | 1 | 1 | 高 | +| **读操作** | 加锁 | 加锁 | 无锁(volatile) | +| **迭代器** | fail-fast | fail-fast | 弱一致性 | +| **null值** | 不允许 | 不允许 | 不允许 | + +**性能优势：** + +- 读操作无锁，写操作细粒度锁 +- 支持高并发读写 +- 迭代过程不阻塞其他操作 + +### 4.3 其他Map实现 + +**🎯 TreeMap的特点和使用场景？** + +``` +底层实现：红黑树（自平衡二叉搜索树） + +核心特性： +- 按key排序存储 +- 不允许key为null +- 线程不安全 +- 查找、插入、删除：O(log n) + +排序机制： +1. key实现Comparable接口 +2. 构造时传入Comparator + +使用场景： +- 需要按key排序的Map +- 范围查询：subMap、headMap、tailMap +``` + +**🎯 LinkedHashMap的特点？** + +``` +底层实现：HashMap + 双向链表 + +核心特性： +- 保持插入顺序或访问顺序 +- 继承HashMap，性能略低 +- 支持LRU缓存实现 + +构造参数： +- accessOrder=false：插入顺序（默认） +- accessOrder=true：访问顺序 + +LRU缓存实现： +``` + +```java +public class LRUCache extends LinkedHashMap { + private final int capacity; + + public LRUCache(int capacity) { + super(capacity, 0.75f, true); // accessOrder=true + this.capacity = capacity; + } + + @Override + protected boolean removeEldestEntry(Map.Entry eldest) { + return size() > capacity; + } +} +``` + +--- + +## 🔒 五、线程安全集合 + +### 5.1 并发集合概述 + +**🎯 同步容器和并发容器的区别？** + +| **特性** | **同步容器** | **并发容器** | +| ------------ | ------------------------- | --------------------------------------- | +| **实现方式** | Collections.synchronized* | java.util.concurrent | +| **锁机制** | 对象级synchronized | 分段锁、CAS、Lock | +| **性能** | 低 | 高 | +| **迭代器** | fail-fast需要外部同步 | 弱一致性或快照 | +| **代表** | Vector、Hashtable | ConcurrentHashMap、CopyOnWriteArrayList | + +**同步容器问题：** + +```java +Vector vector = new Vector<>(); +// 虽然add和get是同步的，但组合操作不是原子的 +if (vector.size() > 0) { + vector.get(0); // 可能抛出IndexOutOfBoundsException +} +``` + +### 5.2 CopyOnWriteArrayList详解 + +**🎯 CopyOnWriteArrayList的实现原理？** + +``` +核心思想：写时复制（Copy-On-Write） + +实现机制： +- 读操作不加锁，直接读取 +- 写操作加锁，复制整个数组，在新数组上修改 +- 修改完成后，更新引用指向新数组 + +适用场景： +- 读多写少的场景 +- 实时性要求不高的场景 +- 数据量不大的场景 +``` + +**源码分析：** + +```java +public boolean add(E e) { + final ReentrantLock lock = this.lock; + lock.lock(); + try { + Object[] elements = getArray(); + int len = elements.length; + Object[] newElements = Arrays.copyOf(elements, len + 1); // 复制数组 + newElements[len] = e; + setArray(newElements); // 原子更新引用 + return true; + } finally { + lock.unlock(); + } +} + +public E get(int index) { + return get(getArray(), index); // 无锁读取 +} +``` + +**🎯 CopyOnWriteArrayList vs Collections.synchronizedList？** + +| **特性** | **CopyOnWriteArrayList** | **Collections.synchronizedList** | +| ------------ | ------------------------ | -------------------------------- | +| **读性能** | 高（无锁） | 低（需要同步） | +| **写性能** | 低（复制数组） | 中等 | +| **内存占用** | 高（可能有多个副本） | 正常 | +| **实时性** | 弱一致性 | 强一致性 | +| **迭代安全** | 天然安全 | 需要外部同步 | + +--- + +## 🔧 五、高级特性与安全机制 + +### 5.1 快速失败与安全失败 + +**🎯 Java快速失败（fail-fast）和安全失败（fail-safe）区别？** + +**1. Fail-Fast（快速失败）** + +- **定义**：在迭代集合时，如果发现集合被结构性修改（add/remove 等），会立刻抛出 `ConcurrentModificationException`。 +- **原因**：迭代器内部维护了一个 **modCount（修改次数）**，每次迭代会校验是否和期望值一致，如果不一致就认为出现并发修改，直接报错。 +- **典型集合**：`ArrayList`、`HashMap` 等。 +- **特点**： + - 检测到并发修改 → **立即失败**，避免返回错误结果。 + - 不能在遍历过程中修改集合，除非用迭代器自带的 `remove()`。 + +**2. Fail-Safe（安全失败）** + +- **定义**：在迭代时允许集合被修改，修改不会影响正在进行的迭代。 +- **原因**：迭代器基于 **集合的副本（快照）** 来遍历，而不是直接访问原集合。 +- **典型集合**：`CopyOnWriteArrayList`、`ConcurrentHashMap` 等并发集合。 +- **特点**： + - 遍历时不抛异常。 + - 修改不会影响当前遍历结果，但可能导致数据 **不可见**（因为遍历的是副本）。 + - 内存开销较大（需要拷贝或额外的数据结构支持）。 + +**🎯 如何避免fail-fast？** + +- **在单线程的遍历过程中**，如果要进行 remove 操作，可以调用迭代器 ListIterator 的 remove 方法而不是集合类的 remove方法。 + +```java +public void remove() { + if (lastRet < 0) + throw new IllegalStateException(); + checkForComodification(); + + try { + SubList.this.remove(lastRet); + cursor = lastRet; + lastRet = -1; + expectedModCount = ArrayList.this.modCount; // 关键：更新期望的modCount + } catch (IndexOutOfBoundsException ex) { + throw new ConcurrentModificationException(); + } +} +``` + +- **使用并发包**(`java.util.concurrent`)中的类来代替 ArrayList 和 hashMap + - CopyOnWriterArrayList 代替 ArrayList + - ConcurrentHashMap 代替 HashMap + +### 5.2 Iterator详解 + +**🎯 Iterator 和 Enumeration 区别？** + +在 Java 集合中，我们通常都通过 "Iterator(迭代器)" 或 "Enumeration(枚举类)" 去遍历集合。 + +```java +public interface Enumeration { + boolean hasMoreElements(); + E nextElement(); +} + +public interface Iterator { + boolean hasNext(); + E next(); + void remove(); +} +``` + +**主要区别**： + +- **函数接口不同**：Enumeration**只有2个函数接口。**通过Enumeration，我们只能读取集合的数据，而不能对数据进行修改。Iterator**只有3个函数接口。**Iterator除了能读取集合的数据之外，也能数据进行删除操作。 + +- **Iterator支持 fail-fast机制，而Enumeration不支持**：Enumeration 是 JDK 1.0 添加的接口。使用到它的函数包括 Vector、Hashtable 等类，这些类都是 JDK 1.0中加入的，Enumeration 存在的目的就是为它们提供遍历接口。而 Iterator 是 JDK 1.2 才添加的接口，它也是为了 HashMap、ArrayList 等集合提供遍历接口。Iterator 是支持 fail-fast 机制的。 + +**🎯 Iterater 和 ListIterator 之间有什么区别？** + +- 我们可以使用 Iterator来遍历 Set 和 List 集合，而 ListIterator 只能遍历List +- ListIterator有add方法，可以向List中添加对象，而Iterator不能 +- ListIterator和Iterator都有hasNext()和next()方法，可以实现顺序向后遍历，但是ListIterator有hasPrevious()和previous()方法，可以实现逆向（顺序向前）遍历。Iterator不可以 +- ListIterator可以定位当前索引的位置，nextIndex()和previousIndex()可以实现。Iterator没有此功能 +- 都可实现删除操作，但是 ListIterator可以实现对象的修改，set()方法可以实现。Iterator仅能遍历，不能修改 + +--- + +## ⚡ 六、性能与优化 + +### 6.1 时间复杂度分析 + +**🎯 各种集合操作的时间复杂度？** + +| **集合类型** | **get/contains** | **add** | **remove** | **特点** | +| ------------------- | ---------------- | -------- | ---------- | ---------- | +| **ArrayList** | O(1)/O(n) | O(1) | O(n) | 随机访问快 | +| **LinkedList** | O(n) | O(1) | O(1) | 插入删除快 | +| **HashSet/HashMap** | O(1) | O(1) | O(1) | 哈希表 | +| **TreeSet/TreeMap** | O(log n) | O(log n) | O(log n) | 有序结构 | +| **ArrayDeque** | O(1) | O(1) | O(1) | 双端队列 | + +> [!CAUTION] +> +> - ArrayList add可能触发扩容，最坏O(n) +> - LinkedList remove需要先查找位置 +> - HashMap最坏情况O(n)，平均O(1) +> - ArrayDeque两端操作 + + + +### 6.2 最佳实践 + +**🎯 集合使用的最佳实践？** + +``` +1. 容量预估： + - ArrayList: new ArrayList<>(expectedSize) + - HashMap: new HashMap<>(expectedSize / 0.75 + 1) + +2. 选择合适的集合： + - 频繁随机访问 -> ArrayList + - 频繁插入删除 -> LinkedList + - 去重 -> Set + - 排序 -> TreeSet/TreeMap + - 高并发 -> ConcurrentHashMap + +3. 避免装箱拆箱： + - 使用primitive集合：TIntArrayList、TLongHashSet + +4. 迭代器使用： + - foreach优于传统for循环 + - 删除元素使用Iterator.remove() + +5. 线程安全： + - 单线程：ArrayList、HashMap + - 多线程读多写少：CopyOnWriteArrayList + - 多线程高并发：ConcurrentHashMap +``` + +**性能优化示例：** + +```java +// 1. 容量预估 +List list = new ArrayList<>(1000); // 避免扩容 + +// 2. 批量操作 +list.addAll(Arrays.asList("a", "b", "c")); // 一次性添加 + +// 3. 安全删除 +Iterator it = list.iterator(); +while (it.hasNext()) { + if (shouldRemove(it.next())) { + it.remove(); // 使用迭代器删除 + } +} + +// 4. HashMap优化 +Map map = new HashMap<>(size / 0.75 + 1); +``` + +### 6.3 其他重要面试题 + +**🎯 HashMap 和 Hashtable 的区别？** + +| **特性** | **HashMap** | **Hashtable** | **ConcurrentHashMap** | +| ------------ | -------------------- | ------------------ | ------------------------ | +| **线程安全** | 否 | 是（synchronized） | 是（CAS + synchronized） | +| **效率** | 高 | 低（全表锁） | 高（分段锁/桶锁） | +| **null值** | key和value都允许null | 都不允许null | 都不允许null | +| **初始容量** | 16 | 11 | 16 | +| **扩容方式** | 2倍 | 2n+1 | 2倍 | +| **出现版本** | JDK 1.2 | JDK 1.0 | JDK 1.5 | +| **迭代器** | fail-fast | fail-fast | 弱一致性 | + +**详细对比**： + +1. **线程安全**：Hashtable 内部的方法基本都经过 `synchronized` 修饰，HashMap 是非线程安全的。 +2. **初始容量和扩容**： + - 创建时如果不指定容量，Hashtable 默认为11，HashMap默认为16 + - Hashtable 扩容：容量变为原来的2n+1；HashMap扩容：容量变为原来的2倍 + - HashMap 总是使用2的幂次方作为哈希表的大小 + +**🎯 Comparable 和 Comparator 接口有何区别？** + +Java 中对集合对象或者数组对象排序，有两种实现方式： + +**1. Comparable接口（内部比较器）** + +- 位于 `java.lang` 包下，只有一个方法 `compareTo()` +- 实现了 Comparable 接口的类可以进行自然排序 +- 实现了该接口的 List 或数组可以使用 `Collections.sort()` 或 `Arrays.sort()` 排序 + +```java +public interface Comparable { + public int compareTo(T o); +} +``` + +**2. Comparator接口（外部比较器）** + +- 位于 `java.util` 包下，主要方法是 `compare()` +- 可以在类外部定义比较规则，不需要修改类本身 + +```java +public interface Comparator { + public int compare(T lhs, T rhs); + public boolean equals(Object object); +} +``` + +**使用场景对比**： + +| **特性** | **Comparable** | **Comparator** | +| ------------ | ---------------------- | ---------------------- | +| **实现位置** | 类内部 | 类外部 | +| **包位置** | java.lang | java.util | +| **方法** | compareTo() | compare() | +| **使用场景** | 自然排序，一种排序规则 | 定制排序，多种排序规则 | +| **修改原类** | 需要 | 不需要 | + +**🎯 HashSet 底层实现原理？** + +HashSet 的底层其实就是 HashMap，只不过 **HashSet 是实现了 Set 接口并且把数据作为 K 值，而 V 值一直使用一个相同的虚值来保存**。 + +```java +public class HashSet { + private transient HashMap map; + private static final Object PRESENT = new Object(); + + public boolean add(E e) { + return map.put(e, PRESENT)==null; + } +} +``` + +**核心特点**： + +- HashMap的 K 值本身就不允许重复 +- 如果 K/V 相同时，会用新的 V 覆盖掉旧的 V，然后返回旧的 V +- 利用HashMap的key唯一性保证Set的不重复特性 + +--- + +## 🎯 高频面试题汇总 + +### 核心必考题（⭐⭐⭐） + +1. **说说常用的集合有哪些？** +2. **HashMap底层实现原理和JDK1.8优化** +3. **HashMap中key需要重写hashCode()和equals()吗？** +4. **HashMap为什么线程不安全？如何解决？** +5. **ArrayList vs LinkedList使用场景和性能对比** +6. **ConcurrentHashMap实现原理和JDK版本差异** +7. **HashSet如何保证元素不重复？** + +### 进阶深入题（⭐⭐） + +8. **HashMap的长度为什么是2的幂次方？** +9. **HashMap扩容机制和负载因子的作用** +10. **为什么JDK1.8中HashMap从头插入改成尾插入？** +11. **红黑树转换条件和意义** +12. **CopyOnWriteArrayList适用场景和缺点** +13. **HashMap vs Hashtable vs ConcurrentHashMap区别** +14. **快速失败(fail-fast)和安全失败(fail-safe)机制** +15. **LinkedHashMap实现LRU缓存** + +### 应用实践题（⭐） + +16. **如何选择合适的集合类型？** +17. **集合性能优化的方法** +18. **多线程环境下集合的使用注意事项** +19. **自定义对象作为HashMap key的注意事项** +20. **集合遍历和删除的最佳实践** +21. **Iterator vs Enumeration vs ListIterator区别** +22. **Comparable vs Comparator区别和使用场景** + +--- + +## 📝 面试话术模板 + +### 回答框架 + +``` +1. 概念定义（30秒） + - 简要说明是什么 + +2. 核心原理（60秒） + - 底层实现机制 + - 关键数据结构 + +3. 特性分析（30秒） + - 优缺点对比 + - 时间复杂度 + +4. 使用场景（30秒） + - 什么情况下使用 + - 注意事项 + +5. 深入扩展（可选） + - 源码细节 + - 性能优化 + - 最佳实践 +``` + +### 关键话术 + +- **HashMap原理**："HashMap底层基于数组+链表+红黑树实现，JDK1.8进行了重要优化，当链表长度大于8时转换为红黑树..." + +- **线程安全**："HashMap线程不安全主要表现在数据丢失和死循环，解决方案有使用ConcurrentHashMap或Collections.synchronizedMap..." + +- **性能对比**："ArrayList和LinkedList各有优势，ArrayList适合随机访问，LinkedList适合频繁插入删除，选择依据主要是操作特点..." + +- **并发集合**："ConcurrentHashMap通过分段锁(JDK1.7)和CAS+synchronized(JDK1.8)实现高并发，相比Hashtable性能更好..." + +- **扩容机制**："HashMap默认初始容量16，负载因子0.75，扩容时容量翻倍，JDK1.8优化了扩容算法..." + +- **数据结构选择**："根据业务场景选择：需要去重用Set，需要排序用TreeSet，高并发用ConcurrentHashMap..." + + + +### 面试准备清单 + +**📋 必备图表** + +- [ ] 画出HashMap数据结构图（数组+链表+红黑树） +- [ ] 准备ConcurrentHashMap JDK版本对比表 +- [ ] 总结各种集合的使用场景表格 +- [ ] 整理集合类继承关系图 + +**💻 实战练习** + +- [ ] 手写LRU缓存（基于LinkedHashMap） +- [ ] 实现线程安全的ArrayList +- [ ] 分析HashMap死循环问题（JDK1.7） +- [ ] 性能测试对比不同集合类型 + +**🎯 核心算法** + +- [ ] HashMap的hash函数实现 +- [ ] HashMap的put和get流程 +- [ ] ConcurrentHashMap的分段锁原理 +- [ ] ArrayList的扩容机制 + +**📚 推荐资源** + +- 《Java核心技术》第九章：集合 +- 《Java并发编程实战》：并发集合部分 +- HashMap源码分析文章 +- ConcurrentHashMap源码解读 + +--- + +## 🎉 总结 + +> **Java集合框架面试成功秘诀** +> +> 1. **掌握核心原理**：HashMap、ConcurrentHashMap、ArrayList是重中之重 +> 2. **理解设计思想**：为什么这样设计？解决了什么问题？ +> 3. **关注版本差异**：JDK1.7 vs JDK1.8的重要变化 +> 4. **结合实际应用**：什么场景用什么集合？性能如何优化？ +> 5. **准备代码示例**：能手写关键算法，能分析源码 +> +> **记住：集合框架不仅是数据结构，更是解决实际问题的工具。深入理解原理，才能在面试中游刃有余，在实际开发中选择最优方案。** + diff --git a/docs/interview/Kafka-FAQ.md b/docs/interview/Kafka-FAQ.md new file mode 100644 index 0000000000..7759b26147 --- /dev/null +++ b/docs/interview/Kafka-FAQ.md @@ -0,0 +1,2255 @@ +--- +title: Kakfa 面试 +date: 2022-1-9 +tags: + - Kafka + - Interview +categories: Interview +--- + +![](https://img.starfish.ink/common/faq-banner.png) + +> Kafka作为现代分布式系统的**核心消息中间件**，是面试中的**重点考查对象**。从基础架构到高级特性，从性能优化到故障处理，每个知识点都是技术面试的热点。本文档将**Kafka核心技术**整理成**系统化知识体系**，涵盖架构原理、性能调优、可靠性保障等关键领域，助你在面试中脱颖而出！ + +Kafka 面试，围绕着这么几个核心方向准备： + + - **Kafka核心架构**（Broker、Topic、Partition、Producer、Consumer、副本机制） + - **高性能原理**（顺序写、零拷贝、批处理、分区并行、Page Cache利用） + - **可靠性保障**（副本同步、ISR机制、事务支持、一致性保证） + - **性能调优**（分区设计、批处理优化、网络调优、JVM调优） + - **故障处理**（Leader选举、数据恢复、监控告警、容灾设计） + - **高级特性**（Kafka Streams、KRaft模式、延时队列、死信队列） + - **实战应用**（架构设计、最佳实践、问题排查、性能优化案例） + +## 🗺️ 知识导航 + +### 🏷️ 核心知识分类 + +1. **基础与架构**：Kafka定位、核心组件、消息模型、集群架构 +2. **高性能原理**：零拷贝技术、顺序写优化、批处理机制、网络模型 +3. **可靠性保障**：副本机制、一致性保证、事务支持、故障恢复 +4. **性能与调优**：分区设计、参数调优、监控指标、瓶颈分析 +5. **高级特性**：流处理、延时队列、死信队列、KRaft模式 +6. **工程实践**：架构设计、运维监控、故障排查、最佳实践 + +## 一、Kafka基础与架构 🧠 + +### 🎯 为什么需要消息队列 + +消息队列最鲜明的特性是**异步、削峰、解耦**。 + +1. **解耦**： 允许你独立的扩展或修改两边的处理过程，只要确保它们遵守同样的接口约束。 +2. **冗余**：消息队列把数据进行持久化直到它们已经被完全处理，通过这一方式规避了数据丢失风险。许多消息队列所采用的"插入-获取-删除"范式中，在把一个消息从队列中删除之前，需要你的处理系统明确的指出该消息已经被处理完毕，从而确保你的数据被安全的保存直到你使用完毕。 +3. **扩展性**： 因为消息队列解耦了你的处理过程，所以增大消息入队和处理的频率是很容易的，只要另外增加处理过程即可。 +4. **峰值处理能力** & **灵活性 **： 在访问量剧增的情况下，应用仍然需要继续发挥作用，但是这样的突发流量并不常见。 如果为以能处理这类峰值访问为标准来投入资源随时待命无疑是巨大的浪费。使用消息队列能够使关键组件顶住突发的访问压力，而不会因为突发的超负荷的请求而完全崩溃。 +5. **可恢复性**： 系统的一部分组件失效时，不会影响到整个系统。消息队列降低了进程间的耦合度，所以即使一个处理消息的进程挂掉，加入队列中的消息仍然可以在系统恢复后被处理。 +6. **顺序保证**： 在大多使用场景下，数据处理的顺序都很重要。大部分消息队列本来就是排序的，并且能保证数据会按照特定的顺序来处理。（Kafka 保证一个 Partition 内的消息的有序性） +7. **缓冲**： 有助于控制和优化数据流经过系统的速度， 解决生产消息和消费消息的处理速度不一致的情况。 +8. **异步通信**： 很多时候，用户不想也不需要立即处理消息。消息队列提供了异步处理机制，允许用户把一个消息放入队列，但并不立即处理它。想向队列中放入多少消息就放多少，然后在需要的时候再去处理它们。 + + + +### 🎯 Kakfa 是什么 ? + +"Kafka是Apache开源的分布式流处理平台，主要用于构建实时数据管道和流式应用。它具有高吞吐量、低延迟、容错性强的特点。 + +在我们的项目中，我主要用它来做消息中间件，实现系统解耦、异步处理和数据缓冲。比如在用户下单后，我们会将订单信息发送到Kafka，然后由不同的消费者去处理库存扣减、支付、物流等后续流程。" + + + +### 🎯 Kafka 使用场景 ？ + +- 消息系统：解耦生产者和消费者、缓存消息等。 +- 日志收集：一个公司可以用 Kafka 收集各种服务的 log，通过kafka以统一接口服务的方式开放给各种consumer，例如hadoop、HBase、Solr 等。 +- 用户活动跟踪：Kafka 经常被用来记录 web 用户或者app用户的各种活动，如浏览网页、搜索、点击等活动，这些活动信息被各个服务器发布到kafka的topic中，然后订阅者通过订阅这些topic来做实时的监控分析，或者装载到hadoop、数据仓库中做离线分析和挖掘。 +- 运营指标：Kafka 也经常用来记录运营监控数据。包括收集各种分布式应用的数据，生产各种操作的集中反馈，比如报警和报告。 +- 流式处理：比如 spark streaming 和 Flink + + + +### 🎯 Kafka 都有哪些特点？ + +- 高吞吐量、低延迟：kafka 每秒可以处理几十万条消息，它的延迟最低只有几毫秒，每个 topic 可以分多个 partition，consumer group 对 partition 进行 consume 操作。 +- 可扩展性：kafka 集群支持热扩展 +- 持久性、可靠性：消息被持久化到本地磁盘，并且支持数据备份防止数据丢失 +- 容错性：允许集群中节点失败（若副本数量为n,则允许n-1个节点失败） +- 高并发：支持数千个客户端同时读写 + +> 面试中还有一个比较经典的问题，就是你为什么用 Kafka、RabbitMQ 或 RocketMQ，又 或者说你为什么使用某一个中间件，这种问题该怎么回答呢？ + + + +### 🎯 Kafka 核心 API 有哪些？ + +1. Producer API 允许应用程序发送数据流到 kafka 集群中的 topic +2. Consumer API 允许应用程序从 kafka 集群的 topic 中读取数据流 +3. Streams API 允许从输入 topic 转换数据流到输出 topic +4. Connect API 用于在 Kafka 与外部系统之间构建数据管道。它支持可插拔的连接器，用于将数据从外部系统导入 Kafka 或将 Kafka 数据导出到外部系统。 +5. Admin API 用于管理和监控 Kafka 集群。它提供了创建、删除主题，查看主题、分区和 Broker 信息等功能。 + + + +## 二、Kafka高性能原理 🚀 + +### 🎯 Kafka 的设计架构你知道吗？ + +![](https://mrbird.cc/img/QQ20200324-210522@2x.png) + + + +### 🎯 Kafka的核心组件有哪些？ + +Kafka主要组件及其作用： + +**1. Broker（服务节点）**： + +- Kafka集群中的服务器实例 +- 负责消息存储、转发和副本管理 +- 每个Broker有唯一ID标识 +- 支持动态加入和退出集群 + +**2. Topic（主题）**： + +- 消息的逻辑分类单位 +- 类似数据库中的表概念 +- 支持多分区和副本配置 +- 可设置保留策略和清理策略 + +**3. Partition（分区）**： + +- Topic的物理分割单位 +- 实现消息并行处理和负载分担 +- 每个分区内消息有序 +- 分区可分布在不同Broker上 + +**4. Producer（生产者）**： + +- 向Topic发送消息的客户端 +- 支持同步和异步发送 +- 可指定分区策略 +- 支持批量发送和压缩 + +**5. Consumer（消费者）**： + +- 从Topic读取消息的客户端 +- 维护消费位置（offset） +- 支持手动和自动提交offset +- 可订阅多个Topic + +**6. Consumer Group（消费者组）**： + +- 消费者的逻辑分组 +- 组内消费者协作消费Topic +- 实现负载均衡和故障转移 +- 每个分区只能被组内一个消费者消费 + +**7. Zookeeper/KRaft**： + +- 集群协调和元数据管理 +- Broker注册和服务发现 +- Leader选举和配置管理 +- Kafka 2.8+支持KRaft模式 + + + +### 🎯 Zookeeper 在 Kafka 中的作用？ + +Zookeeper 主要用于在集群中不同节点之间的通信。老版本的 kafka 中，偏移量信息也存放在 zk 中。 + +在 Kafka 中，主要被用于集群元数据管理、 Broker管理、leader 检测、分布式同步、配置管理、识别新节点何时离开或者连接、集群、节点实时状态等 + +1. 集群元数据管理 + + ZooKeeper存储和管理Kafka集群的元数据信息，包括Broker列表、主题配置和分区信息。这些元数据确保Kafka集群中的各个节点能够协同工作，保持一致性。 + +2. Broker管理 + + ZooKeeper维护Kafka集群中的Broker列表。每个Broker启动时，会向ZooKeeper注册自己，并周期性地发送心跳信息。ZooKeeper监控这些心跳信息，检测Broker的状态。如果某个Broker失效，ZooKeeper会通知集群中的其他组件，以便进行故障恢复。 + +3. 分区Leader选举 + + Kafka中的每个分区都有一个Leader负责处理所有的读写请求，其他Broker作为Follower从Leader复制数据。ZooKeeper负责管理分区的Leader选举。当Leader失效时，ZooKeeper会触发重新选举，确保分区始终有一个可用的Leader。 + +4. 消费者组协调 + + ZooKeeper管理消费者组的成员信息和偏移量。消费者组中的每个消费者实例向ZooKeeper注册自己，并定期发送心跳信息。ZooKeeper根据这些信息协调消费者组的成员，确保每个分区的消息只被一个消费者实例消费。同时，ZooKeeper还存储消费者组的偏移量，便于在消费者故障恢复时继续消费。 + +5. 配置管理 + + ZooKeeper用于存储Kafka集群的配置信息。通过ZooKeeper，可以集中管理和动态更新Kafka的配置，而无需重启集群中的每个节点。 + + + +### 🎯 没有 Zookeeper ， Kafka 能用吗？ + +在Kafka的传统架构中，ZooKeeper是必不可少的组件，它负责集群的管理和协调任务，如元数据管理、Leader选举和消费者组协调。然而，Kafka已经逐步减少对ZooKeeper的依赖，特别是在新的Kafka版本中，通过引入Kafka Raft协议（KRaft），Kafka可以在没有ZooKeeper的情况下运行。 + +**Kafka Raft协议（KRaft）** + +Kafka Raft协议是Kafka 2.8.0版本引入的，旨在移除对ZooKeeper的依赖。KRaft采用了Raft共识算法来管理Kafka集群的元数据和协调任务，使Kafka能够在没有ZooKeeper的情况下独立运行。 + +**KRaft的主要特性：** + +1. **分布式共识**：使用Raft协议实现分布式共识，确保元数据的一致性和可靠性。 +2. **高可用性**：通过选举Leader，保证即使在节点故障的情况下，集群依然能够正常运行。 +3. **简化架构**：移除ZooKeeper后，Kafka集群的架构变得更加简单，部署和管理也更加方便。 + +**设置KRaft模式** + +以下是配置Kafka在KRaft模式下运行的基本步骤： + +1. **配置服务器属性**： 创建一个新的配置文件`server.properties`，并添加以下配置项： + + ```properties + process.roles=controller,broker + node.id=1 + controller.quorum.voters=1@localhost:9093 + listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093 + log.dirs=/tmp/kraft-combined-logs + ``` + +2. **启动Kafka**： 使用新的配置文件启动Kafka： + + ```bash + bin/kafka-server-start.sh config/server.properties + ``` + +3. **初始化元数据**： 在第一次启动时，需要初始化元数据： + + ```bash + bin/kafka-storage.sh format -t -c config/server.properties + ``` + +**KRaft的优势** + +- **简化部署**：减少了Kafka集群的组件数量，使得部署和管理变得更加简单。 +- **降低运维成本**：不再需要维护ZooKeeper集群，降低了运维成本和复杂性。 +- **提高稳定性**：通过Raft协议实现的分布式共识，提高了系统的稳定性和可靠性。 + +| 特性 | Zookeeper 模式 | KRaft 模式 | +| -------------- | -------------------------------- | ---------------------- | +| 元数据存储 | Zookeeper | Kafka 内部日志 | +| 一致性协议 | ZAB (Zookeeper Atomic Broadcast) | Raft | +| 组件复杂度 | 需要维护 Kafka + ZK 两套集群 | 只需维护 Kafka | +| 运维成本 | 高 | 低 | +| 元数据操作延迟 | 较高 | 更低（内部日志写入） | +| Kafka 版本支持 | 默认（≤2.8） | 推荐（≥3.3），未来唯一 | + + + +### 🎯 Kafka 2.8+版本的KRaft模式了解吗？ + +**什么是 KRaft 模式？** + +- **KRaft（Kafka Raft Metadata Mode）** 是 **Kafka 自己实现的元数据管理机制**，基于 **Raft 共识协议**。 +- 在 Kafka 2.8.0 里引入（实验特性），在 3.3+ 里逐步推荐，未来计划完全取代 **Zookeeper**。 + +传统 Kafka 集群依赖 Zookeeper 存储和管理 **元数据（topic、分区、副本、副本分配）**，带来了一些问题： + +1. Kafka + Zookeeper 双集群运维复杂（两个分布式系统）。 +2. 数据一致性和 failover 依赖 Zookeeper，扩展性受限。 +3. 运维人员需要维护 ZK，增加复杂度。 + +**KRaft 模式的目标**：去掉 Zookeeper，让 Kafka 本身通过 **Raft 协议**管理元数据，提升一致性和简化运维。 + +**KRaft 模式核心机制** + +1. **元数据管理** + - KRaft 用一个新的组件 **Quorum Controller** 取代 Zookeeper，负责集群元数据的存储和管理。 + - 元数据写入 Kafka 内部的一个特殊日志（`__cluster_metadata` topic），保证和普通数据一样的高可用 & 顺序性。 +2. **一致性协议** + - 使用 Raft 协议保证多个 Controller 节点之间的数据一致性。 + - 选举 Leader Controller → 所有元数据变更必须经过 Leader 写入日志，Follower 复制。 +3. **集群架构** + - Controller 不再依赖 Zookeeper，而是内置在 Kafka Broker 中。 + - Broker 启动时可以是 **Controller** 角色，也可以是普通 **Broker**。 + - 最终元数据存储在 Kafka 自己的日志文件中。 +4. **运维简化** + - 不再需要维护 Zookeeper 集群 → 只需要 Kafka Broker 即可。 + - 元数据操作更快，因为省去了跨系统交互。 + + + +### 🎯 Kafka 分区的目的？ + +简而言之：**负载均衡+水平扩展** + +Topic 只是逻辑概念，面向的是 producer 和 consumer；而 Partition 则是物理概念。 + +分区对于 Kafka 集群的好处是：实现负载均衡。分区对于消费者来说，可以提高并发度，提高效率。 + +![kafka use cases](https://scalac.io/wp-content/uploads/2021/02/kafka-use-cases-3-1030x549.png) + +> 可以想象，如果 Topic 不进行分区，而将 Topic 内的消息存储于一个 broker，那么关于该 Topic 的所有读写请求都将由这一个 broker 处理，吞吐量很容易陷入瓶颈，这显然是不符合高吞吐量应用场景的。有了 Partition 概念以后，假设一个 Topic 被分为 10 个 Partitions，Kafka 会根据一定的算法将 10 个 Partition 尽可能均匀的分布到不同的 broker（服务器）上，当 producer 发布消息时，producer 客户端可以采用 `random`、`key-hash` 及 `轮询` 等算法选定目标 partition，若不指定，Kafka 也将根据一定算法将其置于某一分区上。Partiton 机制可以极大的提高吞吐量，并且使得系统具备良好的水平扩展能力。 + + + +### 🎯 Kafka的分区策略有哪些？ + +"Kafka主要有以下分区策略： +- **轮询分区**：默认策略，消息均匀分布到各个分区 +- **Key哈希分区**：根据消息的key进行hash，相同key的消息会路由到同一分区 +- **自定义分区**：实现Partitioner接口，自定义分区逻辑 + +在实际项目中，我们根据业务场景选择： +- 对于用户行为日志，使用轮询分区保证负载均衡 +- 对于订单消息，使用用户ID作为key进行哈希分区，确保同一用户的订单有序处理” + + + +### 🎯 Kafka的存储机制是怎样的？ + +"Kafka的存储采用分段文件存储： +- **Log Segment**：每个分区由多个段文件组成，默认1GB或7天切分一个新段 +- **顺序写入**：所有写入都是顺序追加，充分利用磁盘顺序IO性能 +- **零拷贝**：通过sendfile系统调用实现零拷贝，减少数据在内核和用户态之间的拷贝 +- **页缓存**：充分利用操作系统的页缓存机制 + +这种设计使得Kafka具有很高的吞吐量。在我们的生产环境中，通过监控发现磁盘IO主要是顺序写入，CPU使用率也保持在较低水平。” + + + +### 🎯 为什么不能以 partition 作为存储单位？还要加个 segment？ + +在Apache Kafka中，虽然Partition是逻辑上的存储单元，但在物理存储层面上，Kafka将每个Partition划分为多个Segment。这种设计有几个重要的原因，主要包括管理、性能和数据恢复等方面的考虑。 + +**1、易于管理** + +​ 将Partition划分为多个Segment使得Kafka在管理日志文件时更加灵活： + +- 日志滚动：通过Segment，Kafka可以实现日志滚动策略，例如按时间或文件大小进行滚动，删除旧的Segment文件以释放存储空间。 + +- 文件大小限制：单个大的日志文件难以管理和操作，而将其划分为多个较小的Segment文件，便于进行文件系统操作，如移动、删除和压缩。 + +**2、性能优化** + +Segment有助于提升Kafka的性能，尤其是在数据写入和读取方面： + +- **顺序写入**：Kafka通过顺序写入Segment文件来优化磁盘写入性能，避免随机写入的开销。 +- **高效读取**：分段存储允许Kafka在读取数据时更有效地利用磁盘缓存，并可以通过索引快速定位Segment文件中的数据位置，提升读取效率。 + +**3、数据恢复和副本同步** + +​ Segment的引入简化了数据恢复和副本同步过程： + +- **数据恢复**：在发生故障时，Kafka只需要恢复受影响的Segment文件，而不是整个Partition，从而加快数据恢复速度。 + +- **副本同步**：当副本之间进行数据同步时，Segment级别的操作使得Kafka能够仅同步最近更新的Segment，而不是整个Partition的数据，减少网络带宽的使用和同步时间。 + +**4、高效的垃圾回收** + +​ Segment使得Kafka能够更高效地进行垃圾回收： + +- **日志清理**：Kafka的日志清理策略可以在Segment级别进行，删除或压缩旧的Segment文件，而不影响正在使用的Segment。 +- **TTL管理**：Kafka可以基于Segment实现TTL（Time-to-Live）管理，在达到指定保留时间后删除旧的Segment文件，从而控制磁盘空间的使用。 + + + +### 🎯 segment 的工作原理是怎样的？ + +segment 文件由两部分组成，分别为 “.index” 文件和 “.log” 文件，分别表示为 segment 索引文件和数据文件。 + +这两个文件的命令规则为：partition 全局的第一个 segment 从 0 开始，后续每个 segment 文件名为上一个 segment 文件最后一条消息的 offset 值，数值大小为 64 位，20 位数字字符长度，没有数字用 0 填充 + + + +### 🎯 如果我指定了一个offset，kafka 怎么查找到对应的消息？ + +在 Kafka 中，每个消息都被分配了一个唯一的偏移量（offset），这是一个连续的整数，表示消息在日志中的位置。当你指定一个偏移量并想要查找对应的消息时，Kafka 会进行以下操作： + +1. **确定分区**：首先，需要确定偏移量所属的分区。Kafka 通过主题和分区的组合来唯一确定消息。 +2. **查找索引**：Kafka 为每个分区维护了一个索引，这个索引允许它快速查找给定偏移量的位置。这个索引通常是稀疏的，以减少内存使用，并存储在内存中。 +3. **确定物理位置**：使用索引，Kafka 可以快速定位到包含目标偏移量消息的物理文件（即日志文件）和文件内的大致位置。 +4. **读取消息**：一旦确定了物理位置，Kafka 会从磁盘读取包含该偏移量的消息。如果文件很大，Kafka 会尝试直接定位到消息的起始位置，否则可能需要顺序扫描到该位置。 +5. **返回消息**：找到指定偏移量的消息后，Kafka 将其返回给请求者。 + + + +### 🎯 Kafka 高效文件存储设计特点? + +- Kafka 把 topic 中一个 partition 大文件分成多个小文件段，通过多个小文件段，就容易定期清除或删除已经消费完文件，减少磁盘占用。 +- 通过索引信息可以快速定位 message 和确定 response 的最大大小。 +- 通过 index 元数据全部映射到 memory，可以避免 segment file 的 IO 磁盘操作。 +- 通过索引文件稀疏存储，可以大幅降低 index 文件元数据占用空间大小 + + + +### 🎯 Kafka是如何保证高可用的？ + +- **分区副本机制**：每个分区有多个副本（replica），分布在不同的Broker上 +- **Leader-Follower模式**：每个分区有一个Leader负责读写，Follower负责备份 +- **ISR机制**：In-Sync Replicas，保证数据一致性 +- **Controller选举**：集群中有一个 Controller 负责管理分区和副本状态 + + + +### 🎯 如何提高Kafka的性能？ + +"我从以下几个方面进行Kafka性能优化： + +**生产者端优化：** + +- 批量发送：调整batch.size和linger.ms参数 +- 压缩：启用压缩算法（如snappy、lz4） +- 异步发送：使用异步模式减少延迟 +- 内存配置：调整buffer.memory + +**消费者端优化：** +- 批量拉取：调整fetch.min.bytes和fetch.max.wait.ms +- 多线程消费：一个Consumer Group内多个Consumer并行消费 +- 手动提交：使用手动提交offset，避免重复消费 + +**Broker端优化：** +- 磁盘优化：使用SSD，配置合适的文件系统 +- 网络优化：调整socket.send.buffer.bytes等参数 +- JVM调优：配置合适的堆内存大小和GC策略 + +在我们的项目中，通过这些优化，单机吞吐量从10万QPS提升到了50万QPS。” + + + +### 🎯 Kafka为什么能做到如此高的吞吐量？核心技术原理是什么？ + +"这是Kafka最令人印象深刻的特性之一，单机能轻松做到百万级QPS，我来系统地分析一下高吞吐量的核心技术原理： + +**🚀 核心技术架构图：** +``` +[生产者批量发送] → [顺序写入磁盘] → [零拷贝传输] → [消费者并行消费] + ↓ ↓ ↓ ↓ + 减少网络开销 充分利用磁盘 避免CPU拷贝 分区并行处理 +``` + +**1. 批量处理机制（Batching）** +```java +// 生产者批量配置 +props.put("batch.size", 16384); // 16KB批次大小 +props.put("linger.ms", 5); // 等待5ms收集更多消息 +props.put("buffer.memory", 33554432); // 32MB发送缓冲区 +``` +- **原理**：将多条消息打包成一个批次发送 +- **效果**：网络调用次数减少90%，吞吐量提升5-10倍 +- **权衡**：slight延迟增加换取巨大吞吐量提升 + +**2. 分区并行架构（Partitioning）** +``` +Topic: user-events (6个分区) +Partition 0: [消息1, 消息4, 消息7...] → Consumer A +Partition 1: [消息2, 消息5, 消息8...] → Consumer B +Partition 2: [消息3, 消息6, 消息9...] → Consumer C +``` +- **并行写入**：6个分区 = 6倍并行写入能力 +- **并行消费**：6个消费者同时处理，线性扩展 +- **负载分散**：分区分布在不同Broker，避免热点 + +**3. 顺序写入优化（Sequential I/O）** +``` +随机写入：1000 IOPS × 4KB = 4MB/s +顺序写入：磁盘吞吐量可达 200MB/s+ +性能差距：50倍以上！ +``` +- **WAL机制**：每个分区一个日志文件，只做append操作 +- **充分利用磁盘特性**：顺序写入接近内存性能 +- **避免随机I/O**：不需要维护复杂的索引结构 + +**4. 零拷贝技术（Zero Copy）** +``` +传统方式：磁盘→内核→用户态→内核→网卡 (4次拷贝) +零拷贝： 磁盘→内核→网卡 (2次DMA拷贝，0次CPU拷贝) +性能提升：CPU使用率降低60%，吞吐量提升100%+ +``` + +**5. Page Cache充分利用** +```java +// Kafka充分利用操作系统页缓存 +// 写入时：数据写到PageCache，OS异步刷盘 +// 读取时：优先从PageCache读取，避免磁盘I/O +``` +- **写入优化**：写PageCache几乎等于内存写入 +- **读取优化**：热点数据直接从内存读取 +- **减轻GC压力**：使用堆外内存，减少Java GC影响 + +**6. 网络模型优化（Reactor模式）** +``` +[Acceptor线程] → [Processor线程池] → [Handler线程池] + ↓ ↓ ↓ + 接收连接 I/O处理 业务处理 +``` +- **非阻塞I/O**：基于NIO，单线程处理千万级连接 +- **多路复用**：一个线程管理多个连接 +- **线程分离**：I/O和业务处理分离，提高并发 + +**7. 压缩算法优化** +```java +// 不同压缩算法性能对比 +props.put("compression.type", "snappy"); // 推荐 +// snappy: 压缩比3:1，CPU消耗低 +// lz4: 压缩比2.5:1，速度最快 +// gzip: 压缩比5:1，CPU消耗高 +``` + +**🔥 实际性能数据对比：** + +| 优化技术 | 优化前 | 优化后 | 提升比例 | +|---------|--------|--------|----------| +| 批量发送 | 1万QPS | 10万QPS | 10倍 | +| 分区并行 | 10万QPS | 60万QPS | 6倍 | +| 零拷贝 | 60万QPS | 100万QPS | 67% | +| 压缩优化 | 100万QPS | 150万QPS | 50% | + +**🎯 与其他MQ的性能对比：** + +``` +Kafka: 100万+ QPS (单机) +RabbitMQ: 4万QPS (单机) +ActiveMQ: 2万QPS (单机) +RocketMQ: 10万QPS (单机) +``` + +**为什么差距这么大？** +1. **设计哲学不同**：Kafka专为高吞吐量设计，其他MQ更注重功能丰富性 +2. **存储方式不同**：Kafka基于文件系统，其他MQ多基于内存+数据库 +3. **协议复杂度**：Kafka协议相对简单，减少了协议开销 + +**在我们的生产环境中：** +- **电商秒杀场景**：峰值200万QPS，Kafka集群轻松应对 +- **用户行为日志**：每天处理1000亿条消息，延迟保持在10ms以内 +- **实时数据同步**：多个数据中心间同步，网络带宽跑满依然稳定 + +**关键监控指标：** +```bash +# 吞吐量监控 +BytesInPerSec: 800MB/s +BytesOutPerSec: 1.2GB/s +MessagesInPerSec: 1,500,000/s + +# 延迟监控 +ProduceRequestTimeMs: avg=2ms, 99th=15ms +FetchRequestTimeMs: avg=1ms, 99th=8ms +``` + +这些技术的完美结合，让Kafka在大数据时代成为了事实上的标准消息中间件！" + +------ + + + +## 三、 生产者和消费者 👥 + +### 🎯 Kafka消息是采用 Pull 模式，还是 Push 模式？ + +producer 将消息推送到 broker，consumer 从 broker 拉取消息。 + +消费者采用 pull 的模式的好处就是消费速率可以自行控制，可以按自己的消费能力决定是否消费策略（是否批量等） + +有个缺点是，如果没有消息可供消费是，consumer 也需要不断在循环中轮训等消息的到达，所以 kafka 为了避免这点，提供了阻塞式等新消息。 + + + +### 🎯 Kafka 消费者是否可以消费指定分区消息？ + +**Kafka 消费者可以消费指定分区的消息。** 这种操作称为**分配分区消费（Partition Assignment）**，Kafka 提供了多种方式来实现对指定分区的消息消费。 + +1. 默认消费方式（消费者组模式） + + - 在 Kafka 中，消费者通常属于某个**消费者组**（Consumer Group），由 Kafka 的**分区分配策略**（Partition Assignment Strategy）负责自动将 Topic 的分区分配给组内的消费者。 + + - 在这种模式下： + - 消费者组中的消费者共享 Topic 的分区。 + - Kafka 自动平衡分区的分配，消费者**无法直接指定消费某个分区**。 + +2. 手动分配消费分区 + + Kafka 提供了手动指定消费分区的能力，这种方式允许消费者直接消费指定的分区，而不依赖 Kafka 的自动分区分配机制。 + + **方法：使用 `assign` 方法**：Kafka Consumer API 提供了 `assign` 方法，允许消费者手动订阅特定的分区。 + + ```java + public class SpecificPartitionConsumer { + public static void main(String[] args) { + // 配置 Kafka 消费者属性 + Properties props = new Properties(); + props.put("bootstrap.servers", "localhost:9092"); + props.put("group.id", "test-group"); + props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); + props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); + + KafkaConsumer consumer = new KafkaConsumer<>(props); + + // 手动指定要消费的分区 + TopicPartition partition = new TopicPartition("my-topic", 0); // 指定 Topic 和分区 + //使用 `assign` 方法将消费者绑定到特定分区 + consumer.assign(Collections.singletonList(partition)); + + // 开始消费指定分区的消息 + while (true) { + ConsumerRecords records = consumer.poll(Duration.ofMillis(100)); + for (ConsumerRecord record : records) { + System.out.printf("Offset = %d, Key = %s, Value = %s%n", + record.offset(), record.key(), record.value()); + } + } + } + } + ``` + +3. 指定分区并指定偏移量 + + 除了手动分配分区，Kafka 还允许消费者**从指定分区的特定偏移量开始消费**。 + + **方法：使用 `seek` 方法** + + - 在调用 `assign` 方法分配分区后，可以通过 `seek` 方法指定从分区的哪个偏移量开始消费。 + + ```java + TopicPartition partition = new TopicPartition("my-topic", 0); + consumer.assign(Collections.singletonList(partition)); + + // 指定从偏移量 50 开始消费 + consumer.seek(partition, 50); + + // 开始消费 + while (true) { + ConsumerRecords records = consumer.poll(Duration.ofMillis(100)); + for (ConsumerRecord record : records) { + System.out.printf("Offset = %d, Key = %s, Value = %s%n", + record.offset(), record.key(), record.value()); + } + } + ``` + + + +### 🎯 为什么要有消费者组 | 消费者和消费者组有什么关系？ + +> Kafka 引入消费者组是为了 **提升消息消费的并发能力和容错性**。 +> 组内消费者之间是 **竞争关系**，一个分区只会分配给组内一个消费者，保证消息不被重复消费； +> 组和组之间是 **独立关系**，同一个 Topic 的消息可以被不同组各自完整消费。 +> 这样既能水平扩展消费能力，又能在消费者宕机时由组内其他消费者接管，保证高可用。 + +**Consumer Group 是 Kafka 提供的可扩展且具有容错性的消费者机制**。 + +既然是一个组，那么组内必然可以有多个消费者或消费者实例（Consumer Instance），它们共享一个公共的 ID，这个 ID 被称为 Group ID。组内的所有消费者协调在一起来消费订阅主题（Subscribed Topics）的所有分区（Partition）。当然，每个分区只能由同一个消费者组内的一个 Consumer 实例来消费。 + +**消费者组最为重要的一个功能是实现广播与单播的功能**。一个消费者组可以确保其所订阅的 Topic 的每个分区只能被从属于该消费者组中的唯一一个消费者所消费；如果不同的消费者组订阅了同一个 Topic，那么这些消费者组之间是彼此独立的，不会受到相互的干扰。 + +![Architecture](https://quarkus.io/guides/images/kafka-one-app-two-consumers.png) + + + +**为什么要有消费者组** + +- 本质：消费者组=横向扩展+故障转移+多路独立消费。组内做到“同分区只被一个实例消费”，不同组彼此独立消费同一份数据。 + +- 价值：轻松扩容吞吐（增加实例即可）、实例故障自动转移、支持一份数据被风控/画像/报表等多个系统并行独立消费。 + +**分区分配怎么做** + +- 典型策略（按需选择）： + + - RangeAssignor：按分区范围连续分配，简单但易不均衡。 + + - RoundRobinAssignor：轮询均衡，更平均。 + + - StickyAssignor：粘性分配，极大减少重分配的分区迁移。 + + - CooperativeStickyAssignor：协同粘性，增量再均衡，避免“停—全量—启”抖动（推荐）。 + +- 稳定性配置： + + - partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor + + - 静态成员：group.instance.id=order-c1（容器重启也不触发全量rebalance） + + - 指定分区（必要时）：手动assign精准控制热点 + + + +### 🎯 Rebalance会怎样？ + +- 触发时机：实例加入/离开、心跳超时、订阅变化、主题分区变化、max.poll.interval.ms超时。 + +- 影响：分区短暂撤销→吞吐抖动→重复消费风险（若未妥善提交）→延迟上升。 + +- 我的抑制手段： + + - 升级协同粘性分配；启用静态成员；控制心跳与会话： + + - session.timeout.ms、heartbeat.interval.ms 合理配对（心跳≈会话的1/3） + + - max.poll.interval.ms > 业务最长处理时间 + + - 在“撤销回调”里先提交已处理offset，避免重复；在“分配回调”里恢复状态。 + + - 长任务用本地工作队列或异步线程，避免阻塞poll。 + + + +### 🎯 消息语义你知道哪些咋实现的不？ + +- 至多一次（At-most-once）：先提交offset再处理。快，但可能丢消息。不建议用于关键路径。 + +- 至少一次（At-least-once）：先处理再提交offset。可靠但可能重复。通过幂等消费/去重兜底： + - 业务幂等键（订单号/事件ID）、DB唯一键/乐观锁、Redis布隆/去重窗口。 + +- 精确一次（Exactly-once）：在Kafka内链路达成“读-处理-写”原子性： + + - 生产端：enable.idempotence=true；transactional.id=stable-id；max.in.flight.requests.per.connection=1（需强顺序时） + + - 处理链路：beginTransaction → 处理并向下游topic send → sendOffsetsToTransaction → commitTransaction + + - 消费端：isolation.level=read_committed + + - 跨外部系统：配合事务外发/Outbox模式，或最终一致性+对账补 + + + +### 🎯 Kafka 的每个分区只能被一个消费者线程消费，如何做到多个线程同时消费一个分区？ + +> Kafka 的设计原则是：**同一个分区在同一时刻只能被消费者组里的一个消费者线程消费**，以保证消息的顺序性。 +> 如果要在一个分区上实现多个线程并行消费，常见做法是： +> +> - 用一个线程拉取消息，然后把消息放到本地的 **线程池 / 队列** 中，由多个工作线程并行处理； +> - 这样可以提升消费端的处理能力，但要注意丢失消息顺序和幂等性问题。 +> 如果应用对顺序性要求严格，那就必须保持“一个分区一个消费线程”。 + +在Kafka中，每个分区只能被一个消费者线程消费，以保证消息处理的顺序性。然而，有时需要通过多线程来提高单个分区的消费速度。在这种情况下，可以在消费者应用程序中实现多线程处理。以下是几种常见的方法： + +**方法一：多线程消费处理** + +这种方法通过在单个消费者线程中读取消息，然后将消息分发到多个工作线程进行处理。这样，虽然消息的消费是单线程的，但处理是多线程的。 + +```java +public class MultithreadedConsumer { + private static final int NUM_WORKER_THREADS = 4; + + public static void main(String[] args) { + Properties props = new Properties(); + props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); + props.put(ConsumerConfig.GROUP_ID_CONFIG, "test-group"); + props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer"); + props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer"); + + KafkaConsumer consumer = new KafkaConsumer<>(props); + consumer.subscribe(Arrays.asList("my-topic")); + + ExecutorService executorService = Executors.newFixedThreadPool(NUM_WORKER_THREADS); + + try { + while (true) { + ConsumerRecords records = consumer.poll(Duration.ofMillis(100)); + for (ConsumerRecord record : records) { + executorService.submit(() -> processRecord(record)); + } + } + } finally { + consumer.close(); + executorService.shutdown(); + } + } + + private static void processRecord(ConsumerRecord record) { + System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value()); + // Add your processing logic here + } +} +``` + +**方法二：使用Kafka Connect和KStreams** + +Kafka Connect和KStreams是Kafka生态系统中的两个工具，可以帮助实现多线程消费。 + +- Kafka Connect是用于大规模数据导入和导出的框架，具有内置的并行处理能力。 +- Kafka Streams提供了流处理的抽象，可以在流处理中并行处理数据。 + +**方法三：手动管理偏移量** + +通过手动管理偏移量，可以实现更灵活的多线程消费。 + +```java +public class MultithreadedConsumerWithManualOffset { + private static final int NUM_WORKER_THREADS = 4; + private static ExecutorService executorService = Executors.newFixedThreadPool(NUM_WORKER_THREADS); + + public static void main(String[] args) { + Properties props = new Properties(); + props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"); + props.put(ConsumerConfig.GROUP_ID_CONFIG, "test-group"); + props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false"); + props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer"); + props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer"); + + KafkaConsumer consumer = new KafkaConsumer<>(props); + consumer.subscribe(Arrays.asList("my-topic")); + + try { + while (true) { + ConsumerRecords records = consumer.poll(Duration.ofMillis(100)); + List> futures = new ArrayList<>(); + + for (ConsumerRecord record : records) { + futures.add(executorService.submit(() -> processRecord(record))); + } + + // Wait for all tasks to complete + for (Future future : futures) { + future.get(); + } + + // Commit offsets after processing + consumer.commitSync(); + } + } catch (Exception e) { + e.printStackTrace(); + } finally { + consumer.close(); + executorService.shutdown(); + } + } + + private static void processRecord(ConsumerRecord record) { + System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value()); + // Add your processing logic here + } +} +``` + +**注意事项** + +- **消息顺序性**：确保多线程处理不破坏消息的顺序性。 +- **偏移量管理**：手动提交偏移量时，确保所有消息都已成功处理。 +- **异常处理**：处理多线程中的异常情况，防止消息丢失。 + +通过这些方法，可以在不破坏Kafka分区消费模型的情况下，实现多线程处理，以提高消息处理的效率和吞吐量。 + + + +## 四、Kafka可靠性保障🛡️ + +### 🎯 Kafka如何保证高可用？ + +Kafka高可用架构设计： + +**1. 副本机制（Replication）**： + +- 每个分区维护多个副本 +- 一个Leader副本，多个Follower副本 +- 副本分布在不同Broker上 +- 默认3副本，可容忍1个节点故障 + +**2. ISR机制（In-Sync Replicas）**： + +- ISR是与Leader保持同步的副本集合 +- 只有ISR中的副本才能被选为Leader +- 自动检测并移除落后的副本 +- 保证数据一致性和可用性 + +**3. Leader选举机制**： + +- Controller负责Leader选举 +- 从ISR中选择新Leader +- 30秒内完成故障切换 +- 业务无感知的自动恢复 + +**4. 关键配置参数**： + +```properties +# 副本相关配置 +replication.factor=3 # 3副本容灾 +min.insync.replicas=2 # 最少同步副本数 +unclean.leader.election.enable=false # 禁止非ISR副本选举 + +# 生产者可靠性配置 +acks=all # 等待所有ISR确认 +retries=3 # 重试次数 +enable.idempotence=true # 启用幂等性 +``` + +**5. 跨机房部署**： + +- 副本跨机房分布 +- 防止单机房故障 +- 网络分区容错 +- 异地灾备支持 + + + +### 🎯 Kafka 的多副本机制了解吗？ + +> 所谓的副本机制（Replication），也可以称之为备份机制，通常是指分布式系统在多台网络互联的机器上保存有相同的数据拷贝。副本机制有什么好处呢？ +> +> 1. **提供数据冗余**。即使系统部分组件失效，系统依然能够继续运转，因而增加了整体可用性以及数据持久性。 +> 2. **提供高伸缩性**。支持横向扩展，能够通过增加机器的方式来提升读性能，进而提高读操作吞吐量。 +> 3. **改善数据局部性**。允许将数据放入与用户地理位置相近的地方，从而降低系统延时。 +> +> Kafka 只是用副本机制来提供数据冗余实现高可用性和高持久性，也就是第一个好处 + +简单讲，每个分区在不同Broker上维护多份只追加的提交日志，核心目标是数据冗余与故障切换，保障高可用与持久性。 + +**角色分工非常明确：** + +- **Leader副本**：唯一对外提供读写服务 +- **Follower副本**：仅从Leader异步拉取数据，写入本地日志，不对外服务 + +**一致性靠ISR集合保证：** + +- ISR（In-Sync Replicas）是与Leader保持同步的副本集合 +- 只有ISR成员才有资格在故障时被选为新Leader，确保选出来的Leader具备完整数据 +- Follower落后或失联会被移出ISR，保护一致性 + +**关键参数与策略：** + +- `replication.factor=3`：三副本容灾是我的默认生产配置 +- `min.insync.replicas=2` + 生产端`acks=all`：强一致性写入，至少两份副本确认才算成功 +- `unclean.leader.election.enable=false`：拒绝非ISR副本当Leader，宁可短暂不可用也不丢数据 + +**Leader故障切换流程（ZK/KRaft均可类比）：** + +1) 控制面检测到Leader失效 +2) 从ISR中按优先顺序选举新Leader +3) 发布元数据变更，客户端与Follower快速切换 + +这一套机制在我们的生产环境经受过考验。比如一次机房级故障导致部分Broker下线，业务无感切换，新Leader在秒级完成接管，零数据丢失。 + +**监控与预案：** + +- 监控：ISR规模波动、UnderReplicatedPartitions、Leader选举频次、Controller变更 +- 预案：热点分区Leader重分布、优先副本选举、限流保护 + +**一段可直接复述的强话术：** +“Kafka的副本不是为了加速读，而是为了保证可用性和持久性。我们线上统一三副本、acks=all、min.insync.replicas=2，并关闭unclean选举，哪怕短暂不可写，也绝不让数据回退。遇到Broker故障，ISR内的Follower秒级接任Leader，业务侧无感。在这套策略下，两年内我们没有出现一次因副本导致的数据丢失事件。” + +**常见追问要点（我会主动补充）：** + +- 为什么Follower不对外读？避免读到未复制完的数据，破坏一致性 +- ISR过小怎么办？优先排查网络/磁盘抖动，必要时降速生产保护复制 +- 高一致性带来的性能损耗如何弥补？通过分区扩展、批量发送、压缩与零拷贝抵消 + + + + +### 🎯 Kafka 判断一个节点是否存活有什么条件？ + +- **ZooKeeper的心跳机制**：每个Broker会在ZooKeeper上创建一个临时节点，并定期发送心跳（heartbeat）信号以表明自己是活跃的。如果ZooKeeper在设定的超时时间内没有收到某个Broker的心跳，便会认为该Broker已失效，并删除其临时节点。 + - **Session超时**：如果ZooKeeper在`zookeeper.session.timeout.ms`设置的时间内未收到Broker的心跳，则会认为该Broker失效。 + - **心跳间隔**：Broker定期发送心跳给ZooKeeper，心跳间隔由`zookeeper.sync.time.ms`参数决定。 +- **Leader和Follower之间的心跳**：在Kafka的多副本机制中，Leader副本与Follower副本之间也有心跳机制。Follower副本会定期向Leader发送心跳，表明自己是存活的。 + - **Replica Lag**：如果Follower副本落后于Leader副本超过`replica.lag.time.max.ms`设置的时间，则会被移出ISR（In-Sync Replicas）集合，Kafka认为该Follower可能已失效。 +- **网络连接检查**：Kafka使用TCP连接进行数据传输。Broker之间、Broker与ZooKeeper之间、以及Broker与客户端之间的网络连接状况也是判断节点是否存活的重要依据。 + - **TCP连接超时**：Kafka通过TCP连接进行数据传输和心跳检测，如果TCP连接超时，Kafka会尝试重连并记录连接状态。 + +- **Broker元数据刷新**：Kafka客户端（生产者和消费者）会定期从Kafka集群中刷新元数据，了解Broker的状态。如果某个Broker无法响应元数据请求，客户端将其标记为不可用。 + +- **元数据请求超时**：客户端通过`metadata.max.age.ms`参数设置元数据刷新的时间间隔，如果在这个间隔内无法获取到Broker的元数据，则认为该Broker不可用。 + + + +### 🎯 kafka 在可靠性方面做了哪些改进 + +谈及可靠性，最常规、最有效的策略就是 “副本（replication）机制” ，Kafka 实现高可靠性同样采用了该策略。 + +通过调节副本相关参数，可使 Kafka 在性能和可靠性之间取得平衡。 + +> 实践中，我们为了保证 producer 发送的数据，能可靠地发送到指定的 topic，topic 的每个 partition 收到 producer 发送的数据后，都需要向 producer 发送 ack（acknowledge 确认收到），如果 producer 收到 ack，就会进行下一轮的发送，否则重新发送数据。 +> +> 涉及到副本 ISR、故障处理中的 LEO、HW + +### ISR + +一个 partition 有多个副本（replicas），为了提高可靠性，这些副本分散在不同的 broker 上，由于带宽、读写性能、网络延迟等因素，同一时刻，这些副本的状态通常是不一致的：即 followers 与 leader 的状态不一致。 + +为保证 producer 发送的数据，能可靠的发送到指定的 topic，topic 的每个 partition 收到 producer 数据后，都需要向 producer 发送 ack（acknowledgement确认收到），如果 producer 收到 ack，就会进行下一轮的发送，否则重新发送数据。 + +leader 维护了一个动态的 **in-sync replica set**(ISR)，意为和 leader 保持同步的 follower 集合。当 ISR 中的 follower 完成数据的同步之后，leader 就会给 producer 发送 ack。 + +"ISR（in-sync replica set）是 Kafka 中与副本同步相关的重要机制。简单来说，ISR 是由 leader 维护的、与自身保持同步的 follower 副本集合。当 producer 发送数据到 partition 时，只有 ISR 中的 follower 完成数据同步后，leader 才会向 producer 发送 ack 确认。如果 follower 长时间（默认 10 秒，由`replica.lag.time.max.ms`控制）未与 leader 同步，就会被踢出 ISR。这一机制既保证了数据可靠性，又通过动态调整同步副本集合平衡了性能与一致性。” + + + +### 🎯 ISR 频繁变化，可能会有哪些原因，怎么排查？ + +- 网络延迟和不稳定：高网络延迟或不稳定的网络连接会导致Follower副本无法及时从Leader副本同步数据，从而被移出ISR +- 磁盘性能问题：Follower副本的磁盘写入速度不足，无法及时写入从Leader同步的数据，导致其落后于Leader，进而被移出ISR +- CPU或内存资源不足：Broker节点的CPU或内存资源不足，导致数据处理速度变慢，影响副本同步的性能。 +- GC（垃圾回收）停顿：Java垃圾回收停顿（GC Pause）时间过长，会导致Broker节点在短时间内无法处理请求，影响副本同步 + + + +### 🎯 ack 应答机制 + +Kafka 的 ack 应答机制是生产者向 broker 发送消息后，broker 确认消息接收状态的机制，通过 producer 配置的 `acks` 参数控制 + +对于某些不太重要的数据，对数据的可靠性要求不是很高，能够容忍数据的少量丢失，所以没必要等 ISR 中的 follower 全部接收成功。 + +所以 Kafka 为用户提供了**三种可靠性级别**，用户根据对可靠性和延迟的要求进行权衡，选择以下的 acks 参数配置 + +- 0：producer 不等待 broker 的 ack，这一操作提供了一个最低的延迟，broker 一接收到还没有写入磁盘就已经返回，当 broker 故障时有可能**丢失数据**； +- 1：producer 等待 broker 的 ack，partition 的 leader 落盘成功后返回 ack，如果在 follower 同步成功之前 leader 故障，那么将会**丢失数据**； +- -1（all）：producer 等待 broker 的 ack，partition 的 leader 和 follower 全部落盘成功后才返回 ack。但是如果在 follower 同步完成后，broker 发送 ack 之前，leader 发生故障，那么就会造成**数据重复**。 + + + +### 🎯 故障处理 + +由于我们并不能保证 Kafka 集群中每时每刻 follower 的长度都和 leader 一致（即数据同步是有时延的），那么当 leader 挂掉选举某个 follower 为新的 leader 的时候（原先挂掉的 leader 恢复了成为了 follower），可能会出现 leader 的数据比 follower 还少的情况。为了解决这种数据量不一致带来的混乱情况，Kafka 提出了以下概念： + +- LEO（Log End Offset）：指的是每个副本最后一个offset； +- HW（High Wather）：指的是消费者能见到的最大的 offset，ISR 队列中最小的 LEO。 + +消费者和 leader 通信时，只能消费 HW 之前的数据，HW 之后的数据对消费者不可见。 + +针对这个规则： + +- **当follower发生故障时**：follower 发生故障后会被临时踢出 ISR，待该 follower 恢复后，follower 会读取本地磁盘记录的上次的 HW，并将 log 文件高于 HW 的部分截取掉，从 HW 开始向 leader 进行同步。等该 follower 的 LEO 大于等于该 Partition 的 HW，即 follower 追上 leader 之后，就可以重新加入 ISR 了。 +- **当leader发生故障时**：leader 发生故障之后，会从 ISR 中选出一个新的 leader，之后，为保证多个副本之间的数据一致性，其余的 follower 会先将各自的 log 文件高于 HW 的部分截掉，然后从新的 leader 同步数据。 + +所以数据一致性并不能保证数据不丢失或者不重复，这是由 ack 控制的。HW 规则只能保证副本之间的数据一致性！ + + + +### 🎯 ISR、OSR、AR 是什么？ + +- ISR：ISR是“同步副本集合”（In-Sync Replicas Set）的缩写。ISR集合包含所有与Leader副本保持同步的副本 + +- OSR：OSR是“不同步副本集合”（Out-of-Sync Replicas Set）的缩写 + +- AR：AR是“分配副本集合”（Assigned Replicas Set）的缩写。AR集合包含所有被分配给某个分区的副本，包括Leader和所有Follower。这是Kafka在配置主题时指定的副本数量。 + +ISR 是由 leader 维护，follower 从 leader 同步数据有一些延迟，超过相应的阈值会把 follower 剔除出 ISR，存入 OSR（Out-of-Sync Replicas ）列表，新加入的 follower 也会先存放在 OSR 中。`AR=ISR+OSR`。 + + + +### 🎯 请谈一谈 Kafka 数据一致性原理? + +- 多副本机制：Kafka的每个分区可以配置多个副本（Replicas），这些副本分布在不同的Broker上。副本包括一个Leader和多个Follower。生产者和消费者的所有读写请求都通过Leader进行 +- ISR（In-Sync Replicas）集合 +- 确认机制：Kafka通过配置`acks`参数来决定消息的确认策略，从而保证数据一致性。 +- 选举和故障恢复：当Leader失效时，Kafka会从ISR中选举一个新的Leader。这确保了新Leader的数据是最新的，并且与旧Leader保持一致 +- 幂等性和事务：Kafka支持幂等性和事务，以确保消息的精确一次（exactly-once）语义 + + + +### 🎯 数据传输的事务有几种？ + +数据传输的事务定义通常有以下三种级别： + +- 最多一次: 消息不会被重复发送，最多被传输一次，但也有可能一次不传输 +- 最少一次: 消息不会被漏发送，最少被传输一次，但也有可能被重复传输. +- 精确的一次（Exactly once）: 不会漏传输也不会重复传输,每个消息都传输被 + + + +### 🎯 Kafka创建Topic时如何将分区放置到不同的Broker中？ + +创建Topic时可以通过指定分区分配策略（Partition Assignment Strategy）来控制分区（Partition）如何放置到不同的Broker上 + +1. **默认分区分配**： 如果创建Topic时没有指定分区分配策略，Kafka将使用默认的分区分配策略。默认情况下，Kafka会尝试均匀地将分区分配到所有Broker上。 + +2. **自定义分区分配策略**： 创建Topic时，可以使用`--partition`和`--replication-factor`参数来指定分区数和副本因子。然后，通过`--assign`参数来指定每个分区的Broker列表 + +3. **使用Broker ID**： 如果你知道Broker的ID，可以直接使用Broker ID来指定分区的放置 + + ```bash + kafka-topics.sh --create --topic my-topic --num-partitions 3 --replication-factor 2 --broker-list "0:1,1:2" + ``` + +4. **使用KafkaAdminClient API**： 如果使用Java API创建Topic，可以使用`KafkaAdminClient`类的`createTopics`方法，并设置`NewTopic`对象的`replicaAssignment`属性来指定每个分区的Broker列表 + + + +### 🎯 Kafka的事务机制是怎样的？ + +Kafka事务支持Exactly-Once语义： + +**1. 幂等性Producer**： + +```java +// 幂等性生产者配置 +props.put("enable.idempotence", true); +// 自动设置：acks=all, retries=Integer.MAX_VALUE, max.in.flight.requests.per.connection=5 +``` + +**2. 事务性Producer**： + +```java +// 事务生产者使用 +props.put("transactional.id", "transaction-id-001"); + +producer.initTransactions(); +try { + producer.beginTransaction(); + producer.send(new ProducerRecord<>("topic-A", "message1")); + producer.send(new ProducerRecord<>("topic-B", "message2")); + producer.commitTransaction(); +} catch (Exception e) { + producer.abortTransaction(); +} +``` + +**3. 事务实现原理**： + +- Transaction Coordinator管理事务状态 +- 两阶段提交协议保证原子性 +- Transaction Log记录事务元数据 +- Consumer可设置隔离级别 + +**4. 流处理事务**： + +```java +// Kafka Streams事务处理 +Properties props = new Properties(); +props.put("processing.guarantee", "exactly_once_v2"); + +StreamsBuilder builder = new StreamsBuilder(); +builder.stream("input-topic") + .mapValues(value -> processMessage(value)) + .to("output-topic"); +``` + + + +## 五、实践相关 🔧 + +### 🎯 Kafka 如何保证消息不丢失? + +> "我从三个角度来保证消息不丢失： +> +> **生产者端：** +> - 设置acks=all，等待所有ISR副本确认 +> - 配置retries重试次数 +> - 设置适当的超时时间 +> +> **Broker端：** +> - 配置min.insync.replicas >= 2，保证至少有2个副本同步 +> - 定期备份和监控 +> +> **消费者端：** +> - 手动提交offset，确保消息处理完成后再提交 +> - 实现幂等性处理，防止重复消费造成的业务问题 +> +> 在我们的核心业务中，我采用了acks=all + min.insync.replicas=2的配置，虽然会影响一些性能，但保证了数据的可靠性。" + +那面对“在使用 MQ 消息队列时，如何确保消息不丢失”这个问题时，你要怎么回答呢？首先，你要分析其中有几个考点，比如： + +- 如何知道有消息丢失？ + +- 哪些环节可能丢消息？ + +- 如何确保消息不丢失？ + +**Kafka 只对“已提交”的消息（committed message）做有限度的持久化保证。** + +一条消息从生产到消费完成这个过程，可以划分三个阶段 + +![](https://static001.geekbang.org/resource/image/81/05/81a01f5218614efea2838b0808709205.jpg) + +#### 生产者丢数据 + +可能会有哪些因素导致消息没有发送成功呢？其实原因有很多，例如网络抖动，导致消息压根就没有发送到 Broker 端；或者消息本身不合格导致 Broker 拒绝接收（比如消息太大了，超过了 Broker 的承受能力）等 + +解决此问题的方法非常简单：**Producer 永远要使用带有回调通知的发送 API，也就是说不要使用 producer.send(msg)，而要使用 producer.send(msg, callback)**。不要小瞧这里的 callback（回调），它能准确地告诉你消息是否真的提交成功了。一旦出现消息提交失败的情况，你就可以有针对性地进行处理。 + +#### Broker 丢数据 + +在存储阶段正常情况下，只要 Broker 在正常运行，就不会出现丢失消息的问题，但是如果 Broker 出现了故障，比如进程死掉了或者服务器宕机了，还是可能会丢失消息的。 + +所以 Broker 会做副本，保证一条消息至少同步两个节点再返回 ack + +#### 消费者丢数据 + +Consumer 端丢失数据主要体现在 Consumer 端要消费的消息不见了。Consumer 程序有个“位移”的概念，表示的是这个 Consumer 当前消费到的 Topic 分区的位置。下面这张图清晰地展示了 Consumer 端的位移数据。 + +![](https://static001.geekbang.org/resource/image/0c/37/0c97bed3b6350d73a9403d9448290d37.png) + +比如对于 Consumer A 而言，它当前的位移值就是 9；Consumer B 的位移值是 11。 + +这里的“位移”类似于我们看书时使用的书签，它会标记我们当前阅读了多少页，下次翻书的时候我们能直接跳到书签页继续阅读。 + +正确使用书签有两个步骤：第一步是读书，第二步是更新书签页。如果这两步的顺序颠倒了，就可能出现这样的场景：当前的书签页是第 90 页，我先将书签放到第 100 页上，之后开始读书。当阅读到第 95 页时，我临时有事中止了阅读。那么问题来了，当我下次直接跳到书签页阅读时，我就丢失了第 96～99 页的内容，即这些消息就丢失了。 + +同理，Kafka 中 Consumer 端的消息丢失就是这么一回事。要对抗这种消息丢失，办法很简单：**维持先消费消息（阅读），再更新位移（书签）的顺序**即可。这样就能最大限度地保证消息不丢失。 + +当然，这种处理方式可能带来的问题是消息的重复处理，类似于同一页书被读了很多遍，但这不属于消息丢失的情形。 + +**如果是多线程异步处理消费消息，Consumer 程序不要开启自动提交位移，而是要应用程序手动提交位移**。 + + + +#### 最佳实践 + +1. 不要使用 producer.send(msg)，而要使用 producer.send(msg, callback)。记住，一定要使用带有回调通知的 send 方法。 +2. 设置 `acks = all`。acks 是 Producer 的一个参数，代表了你对“已提交”消息的定义。如果设置成 all，则表明所有副本 Broker 都要接收到消息，该消息才算是“已提交”。这是最高等级的“已提交”定义。 +3. 设置 retries 为一个较大的值。这里的 retries 同样是 Producer 的参数，对应前面提到的 Producer 自动重试。当出现网络的瞬时抖动时，消息发送可能会失败，此时配置了 retries > 0 的 Producer 能够自动重试消息发送，避免消息丢失。 +4. 设置参数 `retry.backoff.ms`。指消息生产超时或失败后重试的间隔时间，单位是毫秒。如果重试时间太短，会出现系统还没恢复就开始重试的情况，进而导致再次失败。300 毫秒算是比较合适的。 +5. 设置 `unclean.leader.election.enable = false`。这是 Broker 端的参数，指是否能把非 ISR 集合中的副本选举为 leader 副本。unclean.leader.election.enable = true，也就是说允许非 ISR 集合中的 follower 副本成为 leader 副本。如果一个 Broker 落后原先的 Leader 太多，那么它一旦成为新的 Leader，必然会造成消息的丢失。故一般都要将该参数设置成 false，即不允许这种情况的发生。 +6. 设置 `replication.factor >= 3`。这也是 Broker 端的参数，表示分区副本的个数。其实这里想表述的是，最好将消息多保存几份，毕竟目前防止消息丢失的主要机制就是冗余。 +7. 设置 `min.insync.replicas > 1`。这依然是 Broker 端参数，指的是 ISR 最少的副本数量。在实际环境中千万不要使用默认值 1。 +8. 确保 replication.factor > min.insync.replicas。如果两者相等，那么只要有一个副本挂机，整个分区就无法正常工作了。我们不仅要改善消息的持久性，防止数据丢失，还要在不降低可用性的基础上完成。推荐设置成 replication.factor = min.insync.replicas + 1。 +9. 确保消息消费完成再提交。Consumer 端有个参数 `enable.auto.commit`，最好把它设置成 false，并采用手动提交位移的方式。如果把参数 enable.auto.commit 设置为 true 就表示消息偏移量是由消费端自动提交，由异步线程去完成的，业务线程无法控制。如果刚拉取了消息之后，业务处理还没进行完，这时提交了消息偏移量但是消费者却挂了，这就造成还没进行完业务处理的消息的位移被提交了，下次再消费就消费不到这些消息，造成消息的丢失。 + +> 相关设置 +> +> - Producer：ack， retry +>- Broker: replica、min_isr、unclen.electron、log.flush.messages +> - Consumer: offset_commit + + + +### 🎯 Kafka 如何保证消息不被重复消费？ + +> Kafka 本身只能保证 **至少一次（At-Least-Once）** 或 **至多一次（At-Most-Once）** 的投递语义，如果想要 **精确一次（Exactly-Once）**，需要依赖它的事务机制。 +> +> 具体来说： +> +> 1. **生产端** 可以开启幂等生产（`enable.idempotence=true`），避免消息重复写入分区。 +> 2. **消费端** 通过 **提交 offset** 控制消费进度，如果消费和提交不同步，可能会导致重复消费。 +> 3. 如果要严格保证 **不重复消费**，通常需要结合： +> - Kafka 的事务性消费 + 生产（消费一个分区的数据、写入下游，再提交 offset，三者绑定在事务里）。 +> - 业务侧的幂等设计（比如用唯一键去重、数据库 upsert）。 +> +> 所以总结就是：Kafka 提供了事务 + 幂等写的能力，但最终避免重复消费，往往要生产端、消费端和业务逻辑三方面结合。 + +Kafka 又是如何做到消息不重复的，也就是：生产端不重复生产消息，服务端不重复存储消息，消费端也不能重复消费消息。 + +相较上面“消息不丢失”的场景，“消息不重复”的服务端无须做特别的配置，因为服务端不会重复存储消息，如果有重复消息也应该是由生产端重复发送造成的。 + +#### 生产者：不重复生产消息 + +生产端发送消息后，服务端已经收到消息了，但是假如遇到网络问题，无法获得响应，生产端就无法判断该消息是否成功提交到了 Kafka，而我们一般会配置重试次数，但这样会引发生产端重新发送同一条消息，从而造成消息重复的发送。 + +对于这个问题，Kafka 0.11.0 的版本之前并没有什么解决方案，不过从 0.11.0 的版本开始，Kafka 给每个生产端生成一个唯一的 ID，并且在每条消息中生成一个 sequence num，sequence num 是递增且唯一的，这样就能对消息去重，达到一个生产端不重复发送一条消息的目的。 + +但是这个方法是有局限性的，只对在一个生产端内生产的消息有效，如果一个消息分别在两个生产端发送就不行了，还是会造成消息的重复发送。好在这种可能性比较小，因为消息的重试一般会在一个生产端内进行。当然，对应一个消息分别在两个生产端发送的请求我们也有方案，只是要多做一些补偿的工作，比如，我们可以为每一个消息分配一个全局 ID，并把全局 ID 存放在远程缓存或关系型数据库里，这样在发送前可以判断一下是否已经发送过了。 + +> 保证消息队列的幂等性的方案？ +> +> 1. 向数据库insert数据时，先**根据主键查询，若数据存在则不insert，改为update** +> 2. 向Redis中写数据可以用**set去重，天然保证幂等性** +> 3. 生产者发送每条消息时，增加一个全局唯一id（类似订单id），消费者消费到时，先**根据这个id去Redis中查询是否消费过该消息**。如果没有消费过，就处理，将id写入Redis；如果消费过了，那么就不处理，保证不重复处理相同消息。 +> 4. 基于数据库的**唯一键约束**来保证不会插入重复的数据，当消费者企图插入重复数据到数据库时，会报错。 +> +> 如果数据量超级大的话，还有种方案使用布隆过滤器 + redis 的方案 + 唯一索引，**层层削流**，就是**确保到达数据库的流量最小化**。 +> +> 首先，一个请求过来的时候，我们会利用布隆过滤器来判断它有没有被处理过。如果布隆过滤器说没有处理过，那么就确实没有被处理过，可以直接处理。如果布隆过滤器说处理过（可能是假阳性），那么就要执行下一步。 +> +> 第二步就是利用 Redis 存储近期处理过的 key。如果 Redis 里面有这个 key，说明它的确被处理过了，直接返回，否则进入第三步。这一步的关键就是 key的过期时间应该是多长。 +> +> 第三步则是利用唯一索引，如果唯一索引冲突了，那么就代表已经处理过了。这个唯一索引一般就是业务的唯一索引，并不需要额外创建一个索引。 + +#### 消费端：不能重复消费消息 + +比如说你在处理消息完毕之后，准备提交了。这个时候突然宕机了，没有提交。等恢复过来，你会再次消费同一个消息。 + +为了保证消息不重复，消费端就不能重复消费消息，该如何去实现呢？消费端需要做好如下配置。 + +第一步，设置 `enable.auto.commit=false`。跟前面一样，这里同样要避免自动提交偏移量。你可以想象这样一种情况，消费端拉取消息和处理消息都完成了，但是自动提交偏移量还没提交消费端却挂了，这时候 Kafka 消费组开始重新平衡并把分区分给另一个消费者，由于偏移量没提交新的消费者会重复拉取消息，这就最终造成重复消费消息。 + +第二步，单纯配成手动提交同样不能避免重复消费，还需要消费端使用正确的消费“姿势”。这里还是先看下图这种情况： + +![](https://s0.lgstatic.com/i/image6/M01/4D/0A/Cgp9HWDtPCWAYncVABfKsdCDbq0367.png) + +消费者拉取消息后，先提交 offset 后再处理消息，这样就不会出现重复消费消息的可能。但是你可以想象这样一个场景：在提交 offset 之后、业务逻辑处理消息之前出现了宕机，待消费者重新上线时，就无法读到刚刚已经提交而未处理的这部分消息（这里对应图中 5~8 这部分消息），还是会有少消费消息的情况。 + +```java +List messages = consumer.poll(); +consumer.commitOffset(); +processMsg(messages); +``` + + + +### 🎯 Kafka 如何保证消息的顺序消费? + +> Kafka 保证分区内顺序，要保证业务顺序，就需要在生产端确保同一业务 key 的消息落在同一分区，在消费端保持分区内单线程串行处理。如果要全局顺序，只能用单分区，但会牺牲性能。 + +> 如果你用的是单分区解决方案，那么有没有消息积压问题？如果有，你是怎么解决的？ +> +> 如果你用的是多分区解决方案，那么有没有分区负载不均衡的问题？如果有，你是怎么解决的？ + +"Kafka的顺序性保证分为几个层面： +- **分区内有序**：同一分区内的消息是有序的 +- **全局有序**：需要将Topic设置为1个分区，但会影响并发性能 +- **业务有序**：通过合理的分区策略，如使用用户ID做key + +在我们的项目中，对于需要保证顺序的场景，我采用以下策略： +- 将相关联的消息发送到同一分区（使用相同的key） +- 在消费端使用单线程消费 +- 对于严格顺序要求的场景，我们会在业务层面增加版本号或时间戳进行排序” + +> - 单分区会积压？三招：批量+压缩提吞吐；消费者“同Key串行、跨Key并行”；顶不住就转按Key多分区。 +> +> - 多分区会不均衡？两招：一致性哈希/虚拟节点；热点Key可语义分裂（如 user#vnode）。 +> +> - 扩分区会乱序？两招：固定“槽→分区”映射防漂移；双写灰度/暂停-排干-切换平滑迁移。 +> +> - 生产端顺序与可靠：enable.idempotence=true，acks=all，retries=∞；强顺序通道设 max.in.flight=1。 +> +> - 消费端提交：先处理再提交（enable.auto.commit=false）；必要时事务 sendOffsetsToTransaction。 +> +> - Rebalance稳态：用协同粘性+静态成员；onPartitionsRevoked 提交已处理，onPartitionsAssigned 恢复指针。 +> +> - 一句话：按Key进同分区、同Key串行跨Key并行，配合幂等+强顺序通道与平滑扩分区，顺序与吞吐同时拿到。 + +**单分区方案 —— 消息积压问题** + +**问题：** + +- 单分区虽然能保证强顺序，但吞吐量受限。 +- 如果消费速度跟不上生产速度，就会产生消息积压。 + +**解决方案：** + +1. **消费端优化** + - 使用批量拉取 + 批量处理。 + - 使用异步 I/O 提升单线程消费性能。 +2. **增加分区数，提升并行度** + - 如果业务允许局部有序（比如按用户 ID、订单号），可以把单分区拆成多个分区。 +3. **消息堆积应急处理** + - 临时增加消费者实例（如果分区足够多）。 + - 使用 Kafka MirrorMaker / DataBus，把数据转储到大数据系统做离线处理。 + +**多分区方案 —— 分区负载不均衡问题** + +**问题：** + +- 分区多了能提升吞吐，但可能出现： + - 某些分区流量很大（热点 Key），导致 **消费负载不均衡**。 + - 某些分区积压严重，其他分区却很空闲。 + +**解决方案：** + +1. **合理设计分区策略** + + - 按 **Key 哈希分区**，避免所有数据落到某几个分区。 + - 对于热点 Key，可以做 **Key 拆分**（比如 userId%10 作为新 Key），让热点用户的请求分散到多个分区。 + +2. **增加分区并做数据重分布** + + - 在 Kafka 2.4+ 版本支持 **分区重分配**，可以调整 partition 与 broker 的映射。 + +3. **消费端并发优化** + + - 使用多线程消费（一个 Consumer 拉取 → 线程池处理）。 + - 或者使用多个 Consumer 实例（前提是分区足够多）。 + + + +### 🎯 Kafka 如何处理消息积压问题？ + +> 你们公司消息队列的监控有哪些？可以利用哪些监控指标来确定消息是否积压？ 在发现消息积压的时候，能不能利用监控的消费速率和生产速率，来推断多久以后积压的 消息会被处理完毕？ 你们公司消息积压的真实案例，包括故障原因、发现和定位过程、最终解决方案。 你负责的业务使用的 topic 还有对应的分区数量。 +> +> 你的业务 topic 里面用了几个分区？你是怎么确定分区数量的？ +> +> 如果分区数量不够会发生 什么？ 什么情况下会发生消息积压？怎么解决消息积压的问题？ +> +> 在异步消费的时候，如果你拉取了一批消息，还没来得及提交就宕机了会发生什么？ + +> 消息积压通常是因为**消费能力跟不上生产速度**。 +> 我会先定位是 **消费端瓶颈**，还是 **分区设计问题**。 +> +> - 如果是消费端处理慢，可以优化消费逻辑、增加消费者并发、批量消费。 +> - 如果是分区数太少，可以增加分区，让更多消费者并行。 +> - 如果消息确实堆积严重，可以考虑临时 **扩容消费者** 或者 **快速丢弃不必要的历史消息**，比如利用 Kafka 的位移重置功能（`kafka-consumer-groups`）直接跳到最新 offset。 +> 此外，Kafka 本身是高吞吐设计，积压本身不会立刻导致数据丢失，但要保证磁盘和保留策略合理，否则可能过期被删除。 +> +> 总结就是：**先排查原因 → 消费端优化 → 分区扩展 → 必要时跳过历史 → 调整集群资源**，多管齐下解决积压。 + +如果出现积压，那一定是性能问题，想要解决消息从生产到消费上的性能问题，就首先要知道哪些环节可能出现消息积压，然后在考虑如何解决。 + +因为消息发送之后才会出现积压的问题，所以和消息生产端没有关系，又因为绝大部分的消息队列单节点都能达到每秒钟几万的处理能力，相对于业务逻辑来说，性能不会出现在中间件的消息存储上面。毫无疑问，出问题的肯定是消息消费阶段，那么从消费端入手，如何回答呢？ + +1. **提高消费者的处理能力** + +- **增加消费者实例**： + + - 可以通过增加消费者实例来提高消息处理能力。这可以通过增加消费者数量来实现，确保每个分区都有一个消费者进行处理。 + + - 使用消费者组来实现负载均衡，每个消费者组中的消费者实例会自动分配分区。 + +- **优化消费者逻辑**： + + - 确保消费者处理逻辑高效，减少每条消息的处理时间。 + + - 使用批处理来减少I/O操作的频率，例如一次性消费多条消息并进行批量处理。 + +- **水平扩展**： + - 消费端的性能优化除了优化消费业务逻辑以外，也可以通过水平扩容，增加消费端的并发数来提升总体的消费性能。特别需要注意的一点是，**在扩容 Consumer 的实例数量的同时，必须同步扩容主题中的分区（也叫队列）数量，确保 Consumer 的实例数和分区数量是相等的。**如果 Consumer 的实例数量超过分区数量，这样的扩容实际上是没有效果的。原因我们之前讲过，因为对于消费者来说，在每个分区上实际上只能支持单线程消费。 + +2. **增加Kafka的吞吐量** + +- **优化Kafka配置**： + + - 调整Kafka的分区数（partitions），增加主题的分区数可以提高并行处理的能力。 + + - 配置适当的生产者和消费者参数，例如`acks`, `batch.size`, `linger.ms`等，以提高消息的发送和接收性能。 + +- **硬件升级**： + + - 增加Kafka服务器的CPU、内存和磁盘性能，确保Kafka服务器具有足够的资源处理高负载。 + + - 使用更快的网络连接，以减少网络延迟和提高数据传输速度。 + +3. **优化数据流** + +- **数据过滤和压缩**： + + - 在生产者端过滤不必要的数据，减少发送到Kafka的消息量。 + + - 使用Kafka的消息压缩功能（如gzip, snappy），减少消息的大小，提高传输效率。 + +- **异步处理**： + - 使用异步处理机制，消费者在消费消息后立即返回，不等待消息处理完成。消息处理可以交给后台线程或其他服务进行异步处理。 + +4. **监控和预警** + +- **设置监控**： + + - 使用Kafka自带的监控工具或第三方监控工具（如Prometheus, Grafana），实时监控Kafka的性能指标（如消息堆积量、消费者延迟等）。 + + - 设置预警机制，当消息堆积超过一定阈值时，及时通知运维人员进行处理。 + +5. **数据再均衡** + +- **再均衡消费者分区**： + + - 使用Kafka的再均衡机制，确保消费者分区分配均匀，避免某些消费者处理负载过重。 + + - 通过调整分区策略，使得消息在各个分区之间均匀分布。 + +> 如果是线上突发问题，要临时扩容，增加消费端的数量，与此同时，降级一些非核心的业务。通过扩容和降级承担流量，这是为了表明你对应急问题的处理能力。 +> +> 其次，才是排查解决异常问题，如通过监控，日志等手段分析是否消费端的业务逻辑代码出现了问题，优化消费端的业务处理逻辑。 +> +> 最后，如果是消费端的处理能力不足，可以通过水平扩容来提供消费端的并发处理能力，但这里有一个考点需要特别注意， 那就是在扩容消费者的实例数的同时，必须同步扩容主题 Topic 的分区数量，确保消费者的实例数和分区数相等。如果消费者的实例数超过了分区数，由于分区是单线程消费，所以这样的扩容就没有效果。 +> +> 比如在 Kafka 中，一个 Topic 可以配置多个 Partition（分区），数据会被写入到多个分区中，但在消费的时候，Kafka 约定一个分区只能被一个消费者消费，Topic 的分区数量决定了消费的能力，所以，可以通过增加分区来提高消费者的处理能力。 + +> 要确定新的分区数量的最简单的做法就是用平均生产者速率除以单一消费者的 消费速率。 比如说所有的生产者合并在一起，QPS 是 3000。而一个消费者处理的 QPS 是 200，那么 $3000 \div 200 = 15$。也就是说你需要 15 个分区。进一步考虑 业务增长或者突发流量，可以使用 18 个或者 20 个 + + + +### 🎯 谈一谈 Kafka 的再均衡? + +Kafka的再均衡（Rebalancing）是消费者组（Consumer Group）中的一个关键概念，它是指当消费者组中的成员发生变动时（例如，新消费者加入组、现有消费者崩溃或离开组），Kafka重新分配分区（Partition）给消费者组中的所有消费者的过程。 + +以下是关于Kafka再均衡的一些要点： + +1. **消费者组**：Kafka中的消费者通常以组的形式存在。消费者组中的所有消费者协调合作，平均分配订阅主题的所有分区，以实现负载均衡。 + +2. **再均衡触发条件**： + + - 组成员发生变更(新consumer加入组、已有consumer主动离开组或已有consumer崩溃了) + - 订阅主题数发生变更，如果你使用了正则表达式的方式进行订阅，那么新建匹配正则表达式的topic就会触发rebalance + - 订阅主题的分区数发生变更 + +3. **再均衡过程**： + + - 当触发再均衡条件时，当前所有的消费者都会暂停消费，以便进行分区的重新分配。 + - 消费者组中的某个消费者（通常是组协调者，Group Coordinator）负责发起再均衡过程。 + - 协调者计算新的分区分配方案，并将方案广播给组内的所有消费者。 + - 所有消费者根据新的分配方案重新分配分区，并开始重新分配的分区中读取数据。 + + > 和旧版本consumer依托于Zookeeper进行rebalance不同，新版本consumer使用了Kafka内置的一个全新的组协调协议（group coordination protocol）。 + > + > 对于每个组而言，Kafka的某个broker会被选举为组协调者（group coordinator）。 + > + > Kafka新版本 consumer 默认提供了多种分配策略 + > + > - Kafka提供了多种再均衡策略，可以通过配置`partition.assignment.strategy`参数进行设置： + > - **RangeAssignor**：按范围分配分区。每个消费者分配一组连续的分区。 + > - **RoundRobinAssignor**：以轮询方式分配分区。分区尽可能均匀地分配给所有消费者。 + > - **StickyAssignor**：在保持现有分区分配的基础上，尽量少地移动分区。 + > - **CooperativeStickyAssignor**：一种渐进的再均衡策略，最小化分区移动，并确保消费者间的平衡。 + > + > 1. 所有成员都向 coordinator 发送请求，请求入组。一旦所有成员都发送了请求，coordinator 会从中选择一个consumer 担任 leader 的角色，并把组成员信息以及订阅信息发给 leader。 + > + > 2. leader 开始分配消费方案，指明具体哪个 consumer 负责消费哪些 topic 的哪些 partition。一旦完成分配，leader 会将这个方案发给 coordinator。coordinator 接收到分配方案之后会把方案发给各个 consumer，这样组内的所有成员就都知道自己应该消费哪些分区了。 + +4. **再均衡的影响**： + + - 在再均衡期间，消费者组的消费者将无法消费数据，这可能导致短暂的服务中断。 + - 再均衡完成后，每个消费者将开始从其分配到的分区中读取数据，这可能会导致已经提交的偏移量被覆盖。 + +5. **再均衡的优化**： + + - 尽量减少再均衡的发生，例如，避免频繁地添加或移除消费者。 + - 使用`max.poll.interval.ms`配置参数来设置消费者在两次轮询之间的最大时间间隔，这有助于控制再均衡的触发。 + - 使用`session.timeout.ms`配置参数来设置消费者与组协调者会话的超时时间，这有助于在消费者崩溃时快速触发再均衡。 + +6. **消费者偏移量管理**： + + - 在再均衡期间，消费者可能会丢失或提交新的偏移量。因此，合理管理偏移量非常重要，例如，使用Kafka提供的自动提交偏移量功能或手动管理偏移量。 + +7. **再均衡监听器**： + + - Kafka消费者API提供了再均衡监听器（`ConsumerRebalanceListener`），允许开发者在再均衡发生前后执行特定的操作，例如，在再均衡前保存当前的消费状态，在再均衡后恢复消费。 + +8. **再均衡与消费者故障转移**： + + - 再均衡是Kafka处理消费者故障转移的一种机制。当消费者崩溃时，Kafka会触发再均衡，将崩溃消费者负责的分区分配给其他消费者，以确保数据仍然可以被消费。 + +再均衡是Kafka消费者模型的一个核心特性，它允许消费者组动态地适应消费者数量和订阅主题的变化。然而，再均衡也可能带来一些性能影响，因此在设计和配置Kafka消费者时，需要仔细考虑这些因素。 + + + +### 🎯 创建topic时如何选择合适的分区数？ + +> 每天两三亿数据量，每秒几千条，设置多少分区合适 + +1. **吞吐量需求** + - **分区 = 并发度**：Kafka 的并发度主要由分区数决定。 + - 一般经验：**每个分区单机消费者可处理 5~10MB/s**，如果预估写入或消费流量很大，需要更多分区来分摊。 +2. **消费者并行度** + - Kafka 规定：一个分区同一时刻只能被 **一个消费者线程消费**。 + - 因此分区数至少要 ≥ 预期的最大消费者线程数，否则会限制消费并行度。 +3. **消息有序性** + - 分区越多，局部有序范围越小。 + - 如果业务需要 **严格顺序**，只能选择 **单分区**（但要面对吞吐瓶颈）。 +4. **Broker 资源** + - 分区本身有开销： + - 每个分区对应一个日志目录，包含索引文件、数据文件； + - Leader/Follower 需要额外内存和文件句柄。 + - 分区数过多会导致 **控制器压力大**，Kafka 元数据同步变慢。 + - 一般建议：**单个 Broker 上分区数不要超过几千**（Kafka 官方建议 < 20w 个分区集群总数）。 +5. **后续扩展性** + - 分区数可以增加，但不能减少。 + - 提前预估峰值，适度超配。 + + + +### 🎯 Kafka 是否支持动态增加和减少分区 + +Kafka支持动态增加分区，但不支持减少分区。 + +- 动态增加分区的影响 + - **负载均衡**：增加分区后，可以重新分配分区到消费者，提供更好的负载均衡。 + - **吞吐量提升**：增加分区数目能够提升Kafka集群的吞吐量，因为更多的分区意味着可以并行处理更多的消息。 + +- **动态增加分区的限制** + - **数据顺序性**：对于依赖消息顺序的应用程序，增加分区可能会影响数据的顺序性，因为新的消息可能会分配到新的分区。 + - **重新分配成本**：增加分区后，可能需要重新分配分区到不同的Broker，这会带来额外的网络和I/O负载。 + +Kafka目前不支持动态减少分区。这是因为减少分区会涉及到数据的重新分配和合并，这会导致很大的复杂性和潜在的数据丢失风险 + +**原因**： + +- **数据迁移复杂性**：减少分区需要将现有分区的数据迁移到其他分区，这会带来巨大的I/O负载和复杂的迁移逻辑。 +- **数据一致性风险**：在减少分区的过程中，可能会有数据丢失或数据不一致的风险，这对于生产环境是不容忽视的。 + + + +### 🎯 遇到过哪些Kafka的生产问题？如何解决的？ + +"我在生产环境中遇到过几个典型问题： + +**问题1：消费延迟严重** +- 原因：Consumer处理速度跟不上Producer生产速度 +- 解决：增加Consumer数量，优化消费逻辑，调整批处理大小 + +**问题2：频繁rebalance** +- 原因：Consumer心跳超时或处理时间过长 +- 解决：调整session.timeout.ms和max.poll.interval.ms参数 + +**问题3：磁盘空间不足** +- 原因：日志保留时间过长，数据堆积 +- 解决：调整retention配置，增加磁盘容量，设置数据清理策略 + +**问题4：网络分区导致的数据不一致** +- 解决：通过监控ISR状态，及时发现和处理网络问题 + +这些问题让我深刻认识到监控和运维的重要性，现在我们建立了完善的监控体系。” + + + +### 🎯 如何监控Kafka集群的健康状态？ + +"我们建立了多维度的监控体系： + +**JMX指标监控：** +- 吞吐量：BytesInPerSec、BytesOutPerSec +- 延迟：ProduceRequestTimeMs、FetchRequestTimeMs +- 错误率：ErrorRate、FailedRequestsPerSec + +**业务指标监控：** +- Consumer Lag：消费延迟 +- Topic分区状态 +- ISR副本数量 + +**系统指标监控：** +- CPU、内存、磁盘IO +- 网络带宽使用情况 +- JVM GC情况 + +**工具：** +- 使用Kafka Manager进行集群管理 +- Prometheus + Grafana进行指标可视化 +- 自定义告警规则，关键指标异常时及时通知 + +通过这套监控体系，我们能够在问题发生前就发现潜在风险。” + + + +### 🎯 如果kafka集群每天需要承载10亿请求流量数据，你会怎么估算机器网络资源？ + +思路总览：我先把“量”拆成四件事：消息速率、网络吞吐、存储容量、并行度（分区/实例）。按公式估算，再给20%~40%余量，确保峰值与故障场景也稳。 + + + +## 六、高级特性与实战应用 🚀 + +### 🎯 能详细说说Kafka的零拷贝技术吗？ + +Kafka通过零拷贝把“磁盘→网卡”的数据路径缩短为“DMA两跳、无用户态拷贝”。落地手段是存储写入阶段用mmap，网络发送阶段用sendfile（Java里是FileChannel#transferTo），数据常驻于Page Cache，CPU只做控制不搬数据，从而显著降低CPU占用与上下文切换、提升吞吐。 + +> 关键认知：所谓“零拷贝”是“零用户态拷贝”，CPU仍参与系统调用与元数据管理，但不再搬运数据。 +> +> 零拷贝是中间件设计的通用技术，是指完全没有 CPU 参与的读写操作。我以从磁盘读数据，然后写到网卡上为例介绍一下。首先，应用程序发起系统调用，这个系统调用会读取磁盘的数据，读到内核缓存里面。同时，磁盘到内核缓存是 DMA 拷贝。然后再从内核缓存拷贝到 NIC 缓存中，这个过程也是 DMA 拷贝。这样就完成了整个读写操作。和普通的读取磁盘再发送到网卡比起来，零拷贝少了两次 CPU 拷贝，和两次内核态与用户态的切换。 +> +> 这里说的内核缓存，在 linux 系统上其实就是 page cache。 + +**传统IO流程的问题：** + + + +![](https://www.nootcode.com/knowledge/kafka-zero-copy/en/traditional-copy.png)假设 Kafka 没有使用零拷贝，从磁盘读取数据发给网络 socket，流程是这样的（以 Linux 为例)： + +1. **read() 调用** + - 从磁盘读取数据到 **内核缓冲区**（Page Cache） + - 再从内核缓冲区复制到 **用户空间缓冲区**（JVM 堆中） +2. **write() 调用** + - 把用户空间缓冲区数据复制回 **内核 socket 缓冲区** + - 再由网卡 DMA 把数据发到网络 + +📌 这样一来，**数据至少被复制了 4 次**（2 次 CPU 内存拷贝，2 次 DMA）： + +```rust +磁盘 -> 内核缓冲区 (DMA) +内核缓冲区 -> 用户缓冲区 (CPU copy) +用户缓冲区 -> socket 缓冲区 (CPU copy) +socket 缓冲区 -> 网卡缓冲区 (DMA) +``` + +这会消耗大量 CPU 时间，尤其是 Kafka 这种传输大量日志数据的场景。 + +**Kafka的零拷贝实现：** + +![](https://www.nootcode.com/knowledge/kafka-zero-copy/en/zero-copy.png) + +- 写入路径（Producer → Broker）：Broker将分区日志段用 mmap 映射到进程虚拟内存，写入像写内存，减少一次用户态拷贝与多次 read/write 系统调用；数据先入 Page Cache，由内核异步刷盘。 + +- 发送路径（Broker → Consumer）：Consumer 发起 FetchRequest 后，Broker 在“响应该请求”时使用 sendfile/transferTo，将 Page Cache 中的数据直接写入 TCP socket，跳过用户态缓冲与两次 CPU 拷贝。 + +- Page Cache 的作用：热数据常驻页缓存，读取多为内存命中；写入先入缓存、后台落盘，显著降低 CPU 占用与上下文切换、提升整体吞吐。 + +**关键代码** + +- sendfile（Consumer侧发送） + + ```java + *// Java NIO: transferTo 底层利用 sendfile* + FileChannel fc = new RandomAccessFile(file, "r").getChannel(); + SocketChannel sc = SocketChannel.open(); + fc.transferTo(position, count, sc); + ``` + +- mmap（Broker写日志） + + ```java + MappedByteBuffer buf = fileChannel.map( + FileChannel.MapMode.READ_WRITE, position, size); + buf.put(data); *// 直接写映射内存，OS异步落盘* + ``` + +**优势与适用边界** + +- 优势 + + - 降低CPU占用与上下文切换；带宽更高、延迟更低 + + - 避免用户态中间缓冲，占用更少内存 + +- 限制/注意 + + - 数据不可改：sendfile只适用于“文件→socket”原样传输，不能在传输中改数据（加解密、变更格式）。 + + - TLS影响：应用层TLS（用户态加密）会破坏零拷贝路径；需内核TLS（ktls）或终止SSL在代理层。 + + - mmap内存占用：映射过大可能挤压内存；需要关注页回收与脏页刷盘。 + + - 小包/短连接不划算：极小消息或频繁短连接时，零拷贝优势减弱。 + + - 度量与回退：遇到异常（如transferTo平台差异、内核Bug）需可回退普通IO路径。 + + + +### 🎯 既然零拷贝这么好，为什么还要有传统的经过用户态的方案呢？ + +这是个非常好的问题！零拷贝确实性能很好，但它有严格的使用限制，不是万能的解决方案： + +**🚫 零拷贝的局限性：** + +**1. 数据不可修改限制** +```java +// ❌ 零拷贝做不到的事 +FileChannel fileChannel = new RandomAccessFile("log.txt", "r").getChannel(); +// 如果我们需要对数据进行加密、压缩、格式转换怎么办？ +// sendfile和mmap都无法在传输过程中修改数据！ +``` + +**传统方案的必要性：** +```java +// ✅ 传统方案可以处理数据 +FileInputStream fis = new FileInputStream("log.txt"); +byte[] buffer = new byte[1024]; +int bytesRead = fis.read(buffer); + +// 可以对数据进行各种处理 +encryptData(buffer); // 加密 +compressData(buffer); // 压缩 +validateData(buffer); // 校验 +transformData(buffer); // 格式转换 + +socketChannel.write(ByteBuffer.wrap(buffer)); +``` + +**2. 应用场景限制** + +**零拷贝适用场景：** + +- ✅ 静态文件服务（nginx传输图片、视频） +- ✅ 数据库备份传输 +- ✅ CDN内容分发 +- ✅ Kafka Consumer原样转发消息 + +**传统方案适用场景：** +- ✅ Web应用服务器（需要动态生成内容） +- ✅ API网关（需要请求路由、鉴权、限流） +- ✅ 消息中间件的Producer（需要序列化、压缩） +- ✅ 实时数据处理（需要过滤、聚合、计算） + +**3. 具体技术限制对比** + +| 技术方案 | 能否修改数据 | 内存使用 | CPU消耗 | 适用场景 | +|---------|-------------|----------|---------|----------| +| sendfile | ❌ 不能修改 | 极低 | 极低 | 文件→网络原样传输 | +| mmap | ✅ 可以修改 | 高（映射整个文件） | 低 | 频繁随机读写文件 | +| 传统read/write | ✅ 完全可控 | 中等 | 较高 | 需要数据处理的场景 | + +**4. 实际业务中的选择** + +**案例1：Web服务器架构** +```java +// Nginx (零拷贝) + Tomcat (传统方案) +Nginx: sendfile on; // 静态资源用零拷贝 +Tomcat: // 动态内容必须用传统方案 +@GetMapping("/api/user/{id}") +public User getUser(@PathVariable Long id) { + User user = userService.findById(id); + // 需要查库、业务逻辑处理、JSON序列化 + // 这些都必须在用户态完成，零拷贝做不到 + return user; +} +``` + +**案例2：消息队列的两面性** +```java +// Kafka Producer (传统方案) +// 需要序列化、分区选择、压缩等处理 +byte[] serializedData = serializer.serialize(message); +byte[] compressedData = compressor.compress(serializedData); +producer.send(new ProducerRecord<>(topic, compressedData)); + +// Kafka Consumer (零拷贝) +// 只需要原样转发数据给客户端 +fileChannel.transferTo(offset, length, socketChannel); +``` + +**5. 为什么不能都用零拷贝？** + +**技术原因：** +- **安全性**：用户态是操作系统的安全边界，很多操作必须在用户态验证 +- **灵活性**：复杂的业务逻辑需要用户态的编程环境 +- **兼容性**：很多第三方库和框架都是基于用户态设计的 + +**设计哲学：** +- **零拷贝**：追求极致性能，适合简单的数据传输 +- **传统方案**：追求功能完整性，适合复杂的业务处理 + + + +### 🎯 基于kafka的延时队列和死信队列如何实现 + +延迟队列是一种特殊的队列。它里面的每个元素都有一个过期时间，当元素还没到过期时间 的时候，如果你试图从队列里面获取一个元素，你会被阻塞。当有元素过期的时候，你就会 拿到这个过期的元素。你可以这样想，你拿到的永远是最先过期的那个元素。 + +很多语言本身就提供了延迟队列的实现，比如说在 Java 里面的 DelayQueue。 + +死信队列是一种逻辑上的概念，也就是说它本身只是一个普通的队列。而死信的意思是指过 期的无法被消费的消息，这些消息会被投送到这个死信队列。 + +基于Kafka实现延时队列和死信队列可以有效地处理消息的延迟投递和异常消息处理。以下是详细的实现方法： + +#### 实现延时队列 + +延时队列用于将消息在特定时间后再投递。实现Kafka延时队列的方法有几种，以下是其中一种常用方法： + +方法：**使用多个主题和定时任务** + +1. **创建多个主题**：创建一组按延迟时间分隔的Kafka主题，如`delay-1min`, `delay-5min`, `delay-10min`等。 + +2. **生产者发送消息**：生产者根据消息的延迟时间，将消息发送到相应的延迟主题。 + + ```java + public void sendMessage(String topic, String key, String message, long delay) { + // 计算目标主题 + String delayTopic = getDelayTopic(delay); + // 发送消息到延迟主题 + kafkaTemplate.send(delayTopic, key, message); + } + + private String getDelayTopic(long delay) { + if (delay <= 60000) { + return "delay-1min"; + } else if (delay <= 300000) { + return "delay-5min"; + } else { + return "delay-10min"; + } + } + ``` + +3. **定时任务消费延迟主题**：使用定时任务或后台线程定期消费延迟主题，将消息转发到实际处理的主题。 + + ```java + @Scheduled(fixedRate = 60000) + public void processDelayQueue() { + ConsumerRecords records = kafkaConsumer.poll(Duration.ofMillis(1000)); + for (ConsumerRecord record : records) { + // 将消息转发到实际处理的主题 + kafkaTemplate.send("actual-topic", record.key(), record.value()); + } + } + ``` + +> 这个方案的缺点其实还挺严重的。第一个是延迟时间必须预先设定好，比如只 能允许延迟 1min、3min 或者 10min 的消息，不支持随机延迟时间。不过绝大多数情况下，业务是用不着非得 随机延迟时间的。 +> +> 在一些 业务场景下，需要根据具体业务数据来计算延迟时间，那么这个就不适用了。 +> +> 第二个是分区之间负载不均匀。比如很多业务可能只需要延迟 3min，那么 1min 和 10min 分区的数据就很少。这会进一步导致一个问题，就是负载高的 分区会出现消息积压的问题。 在这里，很多解决消息积压的手段都无法使用，所以只能考虑多设置几个延迟 时间相近的分区，比如说在 3min 附近设置 2min30s，3min30s 这种分区来分 摊压力。 +> +> 还要考虑一致性问题，比如发送延时队列失败、或者转发到业务 topic 时失败，要怎么处理 + +**单 Topic + 时间戳方案** + +- 核心思想：所有延迟消息发送到单个 Topic，消息中包含触发时间戳 +- 实现步骤： + 1. 生产者发送消息时，在消息头或 key 中携带触发时间戳 + 2. 消费者持续拉取消息，对未到触发时间的消息暂存到本地有序队列 + 3. 通过定时任务扫描有序队列，处理已到触发时间的消息 +- 优点：无需创建多个延迟 Topic，灵活度高 +- 缺点：需维护本地有序队列，重启可能丢失未处理消息（可持久化到本地文件） + +#### 实现死信队列 + +死信队列用于处理由于各种原因无法成功处理的消息。实现 Kafka 死信队列的方法如下： + +1. **创建死信主题**：创建一个专门的死信主题，如`dead-letter-topic`。 + +2. **配置消费者处理逻辑**：在消费者逻辑中捕获处理失败的异常，将失败的消息发送到死信主题。 + + ```java + public void consume(ConsumerRecord record) { + try { + // 处理消息 + processMessage(record.value()); + } catch (Exception e) { + // 将处理失败的消息发送到死信主题 + kafkaTemplate.send("dead-letter-topic", record.key(), record.value()); + } + } + + private void processMessage(String message) { + // 处理消息逻辑 + if (message.contains("error")) { + throw new RuntimeException("处理失败"); + } + } + ``` + +3. **监控死信队列**：定期检查死信主题中的消息，分析和处理这些失败的消息。 + + ```java + @Scheduled(fixedRate = 60000) + public void processDeadLetterQueue() { + ConsumerRecords records = kafkaConsumer.poll(Duration.ofMillis(1000)); + for (ConsumerRecord record : records) { + // 记录或重试处理死信消息 + logger.error("处理死信消息: " + record.value()); + } + } + ``` + +**优点**： + +- **解耦**：使用Kafka主题来实现延时和死信队列，可以解耦生产者和消费者。 +- **扩展性**：Kafka天然支持分布式和高吞吐量，适合大规模消息处理。 + +**局限性**： + +- **复杂性**：需要管理多个主题和定时任务，增加了系统复杂性。 +- **延迟精度**：延时队列的延迟精度取决于定时任务的执行频率，可能无法达到毫秒级精度。 + + + +### 🎯 Kafka 目前有哪些内部 topic，他们都有什么特征，各自的作用又是什么？ + +下面按“核心内置 / KRaft元数据 / 生态组件内部Topic”梳理，给出特征与作用。面试时先答核心两项，再按是否使用KRaft、Connect/Streams补充。 + +**核心内置（所有Kafka都有）** + +- *consumer_offsets* + + - 特征：cleanup.policy=compact；默认分区数≈50（offsets.topic.num.partitions）；RF建议=3（offsets.topic.replication.factor） + + - 作用：存储消费者组的已提交offset与组元数据（分配方案等），由GroupCoordinator读写 + + - 要点：按group维度隔离；不要手工写入/删除；保障RF与min.insync.replicas，避免丢offset + +- *transaction_state* + + - 特征：cleanup.policy=compact；默认分区数≈50（transaction.state.log.num.partitions）；RF建议=3 + + - 作用：存储事务/幂等相关元数据（transactional.id→producerId/epoch、事务状态、commit/abort markers），支撑EOS + + - 要点：不可用会导致事务生产/提交失败；监控事务协调器与磁盘健康 + +**KRaft模式（无ZK时）** + +- *cluster_metadata（元数据日志，控制面专用）* + + - 特征：Raft复制的内部元数据日志，不作为普通业务Topic使用 + + - 作用：持久化集群元数据事件（Topic/Partition/ACL/ISR等） + + - 要点：存在于Controller节点；关系到集群生死，确保存储与副本健康 + +**生态组件的“内部Topic”（按需出现）** + +- Kafka Connect + + - connect-configs（compact）：存储连接器/任务配置 + + - connect-offsets（compact）：存储源连接器的读取偏移 + + - connect-status（compact）：存储connector/task状态 + + - 要点：RF≥3、min.insync.replicas≥2，避免单点；迁移/恢复靠这些Topic + +- Kafka Streams（以应用ID为前缀自动创建） + + - `-...-changelog（compact）`：状态存储快照日志，用于故障恢复 + + - `-...-repartition（delete）`：重分区Topic，用于按key重分布数据 + + - 要点：为高可用将RF设为3；分区数影响并行度与恢复速度 + +- Confluent Schema Registry + + - _schemas（compact）：存储Avro/Protobuf/JSON Schema及版本 + + - 要点：非Kafka核心自带，但常见于Confluent发行版 + +补充注意 + +- 这些内部Topic多为“compact”以保留最新状态，需保障RF/min.insync与磁盘健康。 + +- 不要对其做人为清理/变更；运维时仅调参与监控（URP、ISR、延迟、分区数、磁盘）。 + + + +### 🎯 kafka支持事务么，你们项目中有使用么，它的原理是什么？ + +> Kafka 从 **0.11 版本开始支持事务**，主要用于 **保证消息的“Exactly Once” 语义（EOS，恰好一次语义）**。 +> 在我项目里，事务一般用在 **生产端**，比如订单场景：业务写入数据库成功时，再原子地写入 Kafka，保证不会出现“数据库写了但消息没发”或者“消息发了但数据库回滚”的不一致问题。 +> 事务的原理主要依赖 **Producer 端的幂等性 + 事务协调器（Transaction Coordinator）**，通过写入内部主题 `__transaction_state` 来维护事务状态。事务提交时消息才对消费者可见，回滚时消息会被标记为无效，从而实现精确一次。 + +**支持**（从 **Kafka 0.11** 开始）。 + +事务的主要目标是 **保证多条消息的写入要么全部成功、要么全部失败**，解决「**消息丢失**」「**消费端读到不一致数据**」的问题。 + +**Kafka 事务的核心能力** + +1. **多分区原子写入**：一次事务中向多个分区写入消息，要么所有分区都写入成功，要么都失败（避免部分分区写入成功导致的数据不一致）。 +2. **消费 - 生产链路原子性**：支持 “消费消息→业务处理→生产新消息” 的全链路事务（如消费订单消息后，生成支付消息，确保两者状态一致）。 +3. **事务恢复与幂等性**：通过事务日志记录事务状态，崩溃后可恢复；配合幂等生产者避免重复写入。 + +**Kafka 事务的实现原理** + +核心依赖 **事务协调器（Transaction Coordinator）** 和 **事务日志（Transaction Log）** 两大组件： + +1. **事务日志（__transaction_state 主题）** + - 一个特殊的内部主题，用于持久化所有事务的状态（如 “开始”“提交”“中止”），副本机制保证高可用。 + - 每条记录包含：事务 ID、涉及的分区、事务状态等元数据。 +2. **事务协调器（Transaction Coordinator）** + - 每个 broker 都可能成为协调器（按事务 ID 哈希分配），负责管理事务全生命周期： + - 接收生产者的事务开始 / 提交 / 中止请求； + - 将事务状态写入事务日志； + - 协调分区副本对事务消息的可见性（通过 “事务标记” 控制）。 +3. **事务消息的可见性控制** + - 事务中的消息会先被写入分区，但标记为 “未提交”（消费者默认看不到）。 + - 事务提交后，协调器会向涉及的分区写入 “事务提交标记”，消费者才能看到这些消息；若回滚，写入 “中止标记”，消息被丢弃。 +4. **消费者事务隔离级别** + - `read_committed`（默认）：只消费已提交的事务消息（避免脏读）。 + - `read_uncommitted`：可消费未提交的事务消息（可能看到回滚的消息，一般不推荐）。 + +**使用注意事项** + +1. **性能损耗**：事务会增加网络交互（与协调器通信）和磁盘 IO（写入事务日志），吞吐量比普通消息低，需评估场景必要性（非核心链路慎用）。 +2. **事务 ID 唯一性**：每个事务生产者必须有唯一的 `transactional.id`，用于崩溃后恢复事务状态。 +3. **配合幂等性**：开启 `enable.idempotence=true` 避免网络重试导致的重复写入，与事务结合使用更可靠。 + +```java +// 1. 配置事务生产者（开启事务） +Properties props = new Properties(); +props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true"); // 幂等性 +props.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "payment-transaction-1"); // 事务ID +KafkaProducer producer = new KafkaProducer<>(props); + +// 2. 初始化事务 +producer.initTransactions(); + +// 3. 开启事务 +producer.beginTransaction(); +try { + // 4. 向多个主题/分区发送消息（物流、积分、财务） + producer.send(new ProducerRecord<>("logistics-topic", orderId, logisticsMsg)); + producer.send(new ProducerRecord<>("points-topic", orderId, pointsMsg)); + producer.send(new ProducerRecord<>("finance-topic", orderId, financeMsg)); + + // 5. 提交事务（所有消息成功写入） + producer.commitTransaction(); +} catch (Exception e) { + // 6. 回滚事务（所有消息作废） + producer.abortTransaction(); +} +``` + + + +### 🎯 kafka中的幂等是怎么实现的? + +Kafka中，幂等性（Idempotence）指的是发送消息到Kafka集群时，即使多次发送相同的消息，也不会导致消息被多次处理或产生副作用。Kafka通过以下机制实现幂等性： + +1. **Producer ID（PID）** + - 每个幂等性生产者在初始化时都会分配一个唯一的Producer ID（PID）。这个ID用于标识该生产者实例。 + +2. **序列号** + - 每个消息（Record）在发送时都会被分配一个序列号。这个序列号是生产者在每个分区上单调递增的。 + + - 序列号和PID结合使用，可以唯一标识每个消息。 + +3. **幂等性保证逻辑** + + - Kafka Broker会维护每个生产者的PID和序列号的映射表。当Broker接收到一条消息时，会检查这条消息的PID和序列号。 + + - 如果消息的PID和序列号与之前接收到的消息相同，则Broker会丢弃这条消息，避免重复写入。 + +4. **重试机制** + - 如果消息发送失败，幂等性生产者会自动重试。由于PID和序列号的机制，即使发生重试，也不会导致消息重复写入。 + +Producer 端在发送时会给每条消息加上递增的序列号，Broker 端为每个 `(PID, Partition)` 维护最新序列号，重复的就直接丢弃。 + 开启幂等性后能保证单个 Producer Session 内，同一条消息即使重复发送也只会写一次，但跨 Session 要用事务来保证 Exactly Once。 + + + +### 🎯 为什么kafka不支持读写分离？ + +Kafka不支持读写分离的主要原因与其设计原则、性能优化以及数据一致性需求密切相关。以下是详细解释： + +1. **数据一致性** + + Kafka的设计目标之一是保证消息的强一致性。在Kafka中，每个分区都有一个Leader和多个Follower。所有的写操作必须先写入Leader，然后Leader会将数据复制到Follower。 + + - **单点写入**：所有写操作通过Leader进行，确保数据的一致性。Follower从Leader同步数据，保持与Leader的数据一致。 + + - **一致性保障**：如果允许读操作从Follower读取数据，由于Follower可能会滞后于Leader，可能导致读取到不一致的数据。 + +2. **简化设计和高性能** + + Kafka的设计理念是保持架构的简洁和高效，这对于实现**高吞吐量和低延迟**的消息系统至关重要。 + + - **单一机制**：通过仅允许Leader处理写操作和大部分读操作，Kafka简化了其数据一致性和分布式处理逻辑。 + + - **性能优化**：Leader处理读写操作，避免了在Follower上实现复杂的读一致性逻辑，这简化了实现并优化了性能。 + +3. 高可用性和故障恢复 + + Kafka通过Leader和Follower机制来保证高可用性和快速故障恢复。 + + - **快速恢复**：在Leader故障时，从ISR（In-Sync Replicas）中选举新的Leader。如果读操作允许从Follower读取，故障恢复过程会变得复杂，因为需要确保新Leader的Follower数据是一致且最新的。 + + - **简化故障处理**：仅从Leader读取和写入，使得故障处理逻辑更简单，确保系统在故障恢复过程中仍然可以提供强一致性的数据服务。 + +4. **可预见的负载分布** + + 通过将所有写操作集中在Leader上，Kafka可以更好地预测和管理系统负载。 + + - **预防热点**：避免Follower节点成为读请求的热点，导致不均衡的资源消耗。 + + - 负载管理：可以更好地管理和调配系统资源，避免由于Follower读请求导致的不均衡负载。 + +Kafka的设计理念和架构决定了不支持读写分离，主要是为了保证数据一致性、简化设计和优化性能。通过集中处理写操作和大部分读操作，Kafka能够提供高吞吐量、低延迟和高可靠性的消息服务。 + + + +### 🎯 KafkaConsumer是非线程安全的，那怎么实现多线程消费? + +- 每个线程一个KafkaConsumer实例：最简单且常见的方法是每个线程创建一个KafkaConsumer实例。这种方式可以确保每个消费者实例在独立的线程中运行，避免线程安全问题。 + +- 单个KafkaConsumer实例多线程处理：单个KafkaConsumer实例从Kafka中拉取消息，然后将这些消息分发到多个工作线程进行处理。这样可以利用多线程处理的优势，同时避免了KafkaConsumer的线程安全问题。 + + + +### 🎯 如果让你设计一个MQ，你怎么设计？ + +其实回答这类问题，说白了，起码不求你看过那技术的源码，起码你大概知道那个技术的基本原理，核心组成部分，基本架构构成，然后参照一些开源的技术把一个系统设计出来的思路说一下就好 + +比如说这个消息队列系统，我们来从以下几个角度来考虑一下 + +1. 首先这个mq得支持可伸缩性吧，就是需要的时候快速扩容，就可以增加吞吐量和容量，那怎么搞？设计个分布式的系统呗，参照一下kafka的设计理念，broker -> topic -> partition，每个partition放一个机器，就存一部分数据。如果现在资源不够了，简单啊，给topic增加partition，然后做数据迁移，增加机器，不就可以存放更多数据，提供更高的吞吐量了？ +2. 其次你得考虑一下这个mq的数据要不要落地磁盘吧？那肯定要了，落磁盘，才能保证别进程挂了数据就丢了。那落磁盘的时候怎么落啊？顺序写，这样就没有磁盘随机读写的寻址开销，磁盘顺序读写的性能是很高的，这就是kafka的思路。 +3. 其次你考虑一下你的mq的可用性啊？这个事儿，具体参考我们之前可用性那个环节讲解的kafka的高可用保障机制。多副本 -> leader & follower -> broker挂了重新选举leader即可对外服务。 +4. 能不能支持数据0丢失啊？可以的，参考我们之前说的那个kafka数据零丢失方案 + +实现一个 MQ 肯定是很复杂的，其实这是个开放题，就是看看你有没有从架构角度整体构思和设计的思维以及能力。 + + + +### 🎯 Kafka Streams了解吗？在什么场景下会使用？ + +"Kafka Streams是Kafka提供的流处理框架，我在以下场景中使用过： + +**实时聚合统计：** +- 实时计算用户行为指标，如PV、UV +- 滑动窗口统计，如最近1小时的订单量 + +**数据清洗和转换：** +- 实时清洗日志数据，过滤无效记录 +- 数据格式转换和字段映射 + +**关联查询：** +- 流表关联，如订单流和用户信息表的关联 +- 双流关联，如点击流和曝光流的关联 + +相比于其他流处理框架（如Storm、Flink），Kafka Streams的优势是： +- 无需额外集群，降低运维复杂度 +- 与Kafka深度集成，exactly-once语义 +- 支持本地状态存储，查询性能好 + +在我们的实时推荐系统中，使用Kafka Streams处理用户行为流，实时更新用户画像。” + + + +### 🎯 Reactive Kafka了解吗？在什么场景下使用？ + +"Reactive Kafka是基于Project Reactor框架的Kafka客户端库，主要用于响应式编程场景： + +**核心特性：** +- **背压处理**：自动处理消费者处理能力与生产速度的匹配 +- **非阻塞IO**：基于Netty的异步非阻塞处理 +- **流式处理**：与Spring WebFlux、Reactor完美集成 +- **资源管理**：自动管理连接池和线程资源 + +**主要组件：** +- `KafkaSender`：响应式的消息发送器 +- `KafkaReceiver`：响应式的消息接收器 +- `SenderRecord/ReceiverRecord`：响应式的消息封装 + +**适用场景：** +在我们的微服务项目中，主要在以下场景使用： + +1. **高并发API服务**： + - WebFlux应用中需要异步处理Kafka消息 + - 避免阻塞线程池，提高系统吞吐量 + +2. **实时数据流处理**： + - 处理大量实时数据流，如用户行为日志 + - 与其他响应式组件（如Reactive MongoDB）集成 + +3. **背压敏感场景**： + - 下游处理能力有限时，需要自动调节消费速度 + - 避免内存溢出和系统雪崩 + +**代码示例：** +```java +// Producer示例 +@Service +public class ReactiveKafkaProducer { + private final KafkaSender sender; + + public Mono sendMessage(String topic, Object data) { + return sender.send(Mono.just(SenderRecord.create(topic, null, null, null, data, null))) + .then(); + } +} + +// Consumer示例 +@Service +public class ReactiveKafkaConsumer { + private final KafkaReceiver receiver; + + @PostConstruct + public void consume() { + receiver.receive() + .delayElements(Duration.ofMillis(10)) // 背压控制 + .doOnNext(this::processMessage) + .doOnNext(record -> record.receiverOffset().acknowledge()) + .subscribe(); + } +} +``` + +**与传统Kafka Client的区别：** +- **线程模型**：传统客户端需要手动管理线程池，Reactive自动管理 +- **背压处理**：传统方式需要手动控制poll数量，Reactive自动调节 +- **错误处理**：响应式提供更优雅的错误处理和重试机制 +- **资源利用**：更高的资源利用率，特别是在IO密集型场景 + +**性能优势：** +在我们的压测中发现： +- CPU使用率降低30%（减少线程切换开销） +- 内存使用更稳定（自动背压调节） +- 在高并发场景下，吞吐量提升20-40% + +**注意事项：** +- 学习曲线相对陡峭，需要理解响应式编程思想 +- 调试相对复杂，需要熟悉响应式调试技巧 +- 不是所有场景都适合，简单的CRUD操作用传统方式更直接 + +在我们的用户行为分析系统中，使用Reactive Kafka处理每秒10万+的用户事件，系统表现非常稳定。” + + + +## 🎯 面试重点总结 + +### 高频考点速览 + +- **Kafka定位与优势**：分布式流平台，高吞吐量，持久化存储 +- **核心组件交互**：Broker、Topic、Partition、Producer、Consumer协作机制 +- **高性能原理**：零拷贝、顺序写、批处理、分区并行的技术组合 +- **可靠性保障**：副本机制、ISR同步、事务支持的数据安全策略 +- **性能调优**：生产者、消费者、Broker三层优化的系统方法 +- **故障处理**：常见问题定位、监控指标、应急处理的运维能力 + +### 面试答题策略 + +1. **基础概念**：准确定义 + 核心特性 + 应用场景 +2. **技术原理**：实现机制 + 关键配置 + 代码示例 +3. **实战经验**：具体案例 + 性能数据 + 解决思路 +4. **对比分析**：技术选型 + 优缺点 + 适用场景 + +--- + +## 📚 扩展学习 + +- **官方文档**：Kafka官方文档和源码分析 +- **实战项目**：搭建Kafka集群，实现完整的消息处理链路 +- **监控工具**：Kafka Manager、Confluent Platform等工具使用 +- **性能测试**：kafka-producer-perf-test.sh、kafka-consumer-perf-test.sh压测 diff --git a/docs/interview/Linux-FAQ.md b/docs/interview/Linux-FAQ.md new file mode 100755 index 0000000000..f8cc194710 --- /dev/null +++ b/docs/interview/Linux-FAQ.md @@ -0,0 +1,1123 @@ +--- +title: Linux 核心面试八股文 +date: 2024-05-31 +tags: + - Linux + - Interview +categories: Interview +--- + +![](https://img.starfish.ink/common/faq-banner.png) + +> Linux作为**服务器运维和开发的基础技能**，也是面试官考察**系统底层理解**的重要内容。从基础命令到系统调优，从文件管理到网络配置，从进程调度到性能监控，每一个知识点都体现着对操作系统的深度掌握。本文档将**最常考的Linux知识点**整理成**标准话术**，助你在面试中展现扎实的系统功底！ + +### 🔥 为什么Linux如此重要？ + +- **📈 服务器必备**：90%的服务器都运行Linux系统 +- **🧠 底层体现**：体现你对操作系统原理、内核机制的深度理解 +- **💼 工作基础**：运维开发、性能调优、故障排查等场景无处不在 +- **🎓 技术进阶**：理解Linux是掌握云原生、容器化技术的关键一步 + +--- + +## 🗺️ 知识导航 + +### 🏷️ 核心知识分类 + +1. **🔥 基础概念类**：文件系统、目录结构、权限管理、用户与组 +2. **⚙️ 命令操作类**：文件操作、文本处理、系统信息、进程管理 +3. **🔧 系统管理类**：服务管理、定时任务、日志管理、软件包管理 +4. **📊 性能监控类**：CPU监控、内存分析、磁盘IO、网络流量 +5. **🌐 网络配置类**：网络配置、防火墙、SSH、端口管理 +6. **🚨 故障排查类**：性能问题诊断、系统调优、应急处理 +7. **💼 实战场景类**：脚本编写、运维实践、面试常见问题 + +### 🔑 面试话术模板 + +| **问题类型** | **回答框架** | **关键要点** | **深入扩展** | +| ------------ | ----------------------------------- | ------------------ | ------------------ | +| **概念解释** | 定义→特点→应用场景→示例 | 准确定义，突出特点 | 底层原理，内核机制 | +| **命令操作** | 基本语法→常用参数→实际示例→注意事项 | 实用性，举例说明 | 高级用法，组合技巧 | +| **故障排查** | 问题现象→分析思路→排查步骤→解决方案 | 系统性思维 | 实战经验，最佳实践 | +| **性能优化** | 监控指标→瓶颈分析→优化手段→验证结果 | 数据驱动 | 内核参数，调优经验 | + +--- + + +## 🔥 一、基础概念类（Linux核心） + +> **核心思想**：Linux基础概念是所有后续学习的基石，包括文件系统、权限管理、用户与组等核心概念，这些是面试中的必考基础知识。 + +- **文件系统**：[目录结构](#🎯-linux-目录结构) | [文件类型](#🎯-linux-文件类型) | [文件系统类型](#🎯-常见文件系统类型) +- **权限管理**：[文件权限](#🎯-文件权限详解) | [用户与组](#🎯-用户与组管理) +- **核心概念**：[inode概念](#🎯-inode是什么) | [硬链接与软链接](#🎯-硬链接和软链接的区别) + +### 🎯 Linux 目录结构 + +**操作系统概述** + +- Linux 是一个开放源代码的操作系统内核，最初由 Linus Torvalds 开发。它采用了类 Unix 的设计理念。 +- 常见的 Linux 发行版包括 Ubuntu、CentOS、Fedora、Debian、Arch Linux 等。 +- **Linux 的文件结构和根目录结构** + - `/`: 根目录，所有文件的起点。 + - `/home`: 用户目录。 + - `/etc`: 配置文件目录。 + - `/var`: 可变数据，如日志文件。 + - `/tmp`: 临时文件。 + - `/bin`: 二进制可执行文件。 + - `/lib`: 库文件。 + +### 🎯 用户与组管理 + +**Linux 用户与权限** + +- Linux 的用户与组管理 + - `useradd`、`usermod`、`userdel` 等命令用于管理用户。 + - `groupadd`、`groupdel`、`groupmod` 用于管理用户组。 + - **文件权限**: `rwx` 权限设置（读取、写入、执行），用 `chmod` 更改权限。 +- 文件权限与属主 + - `ls -l` 命令显示文件详细信息，包括权限、属主和属组。 + - 文件权限：`r` (读), `w` (写), `x` (执行)。 + +### 🎯 inode是什么? + +**inode**（Index Node，索引节点）是Linux/类Unix文件系统中用于存储文件或目录**元数据**（metadata）的核心数据结构。每个文件或目录在创建时都会被分配一个唯一的inode，其作用类似于文件的"身份证"，记录除文件名以外的所有属性信息，并通过指针关联文件的实际数据块。 + +### 🎯 硬链接和软链接的区别? + +硬链接与软链接是Linux系统中两种不同的文件链接机制。 + +1. **硬链接** + - 本质：是同一文件的不同别名，共享相同的inode和数据块，相当于多个指针指向同一物理文件。 + - 特点： + - 所有硬链接与原文件权限、大小、修改时间等属性完全一致。 + - 删除任一硬链接仅减少inode的引用计数，数据块保留至所有链接被删除。 + - 适用场景： + - 数据备份与同步：通过硬链接节省空间，多个链接自动同步数据 + - 同一文件系统内多路径访问：如多个用户共享同一配置文件 +2. **软链接（符号链接）** + - 本质：是独立的文件，存储目标文件的路径信息，相当于快捷方式。 + - 特点： + - 拥有独立的inode，文件内容为路径字符串。 + - 若目标文件被删除，软链接成为"悬空链接 + - 适用场景： + - 跨文件系统或设备：例如链接网络存储中的文件 + - 快捷方式与路径管理：简化复杂路径访问，如版本切换（`/opt/app -> /opt/app-v2`。 + - **动态指向更新**：通过修改软链接路径切换目标 + +--- + +## ⚙️ 二、命令操作类（实用技能） + +> **核心思想**：Linux命令是日常运维开发的核心工具，熟练掌握文件操作、文本处理、系统信息、进程管理等命令是Linux工程师的基本功。 + +- **文件操作**：[基础命令](#🎯-常用文件管理命令) | [文件查找](#🎯-如何查找文件和搜索内容) | [链接创建](#🎯-建立软链接和硬链接命令) +- **文本处理**：[文本查看](#🎯-查看文件的不同方式) | [日志查看](#🎯-平时怎么查看日志) | [定位行号](#🎯-怎么查找定位到第10行) +- **系统信息**：[路径操作](#🎯-绝对路径和目录切换) | [进程查看](#🎯-怎么查看当前进程) | [ls命令详解](#🎯-ls命令功能和参数) + +### 🎯 常用文件管理命令 + +**常用命令** + +- 文件管理命令：`ls`, `cp`, `mv`, `rm`, `mkdir`, `rmdir`, `find`, `locate`, `which`。 +- 系统管理命令：`top`, `ps`, `kill`, `htop`, `df`, `du`, `free`, `uptime`, `dmesg`。 +- 文件权限与用户命令：`chmod`, `chown`, `chgrp`, `passwd`, `id`, `groups`。 +- 查看和编辑文件命令：`cat`, `less`, `more`, `grep`, `vim`, `nano`。 + +### 🎯 如何查找文件和搜索内容？ + +- `find /path/to/search -name "filename"` 查找文件。 +- `grep "pattern" file` 搜索文件内容。 + +### 🎯 建立软链接和硬链接命令 + +- 软链接： ln -s slink source +- 硬链接： ln link source + +### 🎯 怎么查找定位到第10行？ + +要查找并定位文件的第10行，你可以使用多种命令和方法，具体取决于你使用的工具和目的。以下是一些常见的方法： + +1. 使用 `sed` 命令 + + `sed` 是一个流编辑器，可以用于按行处理文件。 + + ```bash + sed -n '10p' filename + ``` + + - `-n`: 禁止默认输出。 + - `'10p'`: 表示输出文件中的第 10 行。 + +2. 使用 `head` 和 `tail` 命令 + + 你可以组合使用 `head` 和 `tail` 命令来显示特定行。 + + ```bash + head -n 10 filename | tail -n 1 + ``` + + - `head -n 10`: 显示文件的前 10 行。 + - `tail -n 1`: 从 `head` 输出的内容中显示最后一行，即第 10 行。 + +3. 使用 `awk` 命令 + + `awk` 是一个强大的文本处理工具，适用于基于模式的行操作。 + + ```bash + awk 'NR==10' filename + ``` + + - `NR==10`: `NR` 是 `awk` 内部的行号变量，这个命令会输出第 10 行。 + +### 🎯 平时怎么查看日志？ + +Linux查看日志的命令有多种：tail、cat、tac、head、echo等，只介绍几种常用的方法。 + +- tail 最常用的一种查看方式 + + 一般还会配合着grep搜索用，例如; + + ``` + tail -fn 1000 test.log | grep '关键字' + ``` + +- head 跟tail是相反的head是看前多少行日志 + + ``` + head -n 10 test.log 查询日志文件中的头10行日志; head -n -10 test.log 查询日志文件除了最后10行的其他所有日志; + ``` + +- cat + + cat 是由第一行到最后一行连续显示在屏幕上 一次显示整个文件： + + ``` + $ cat filename + ``` + +- more + + more命令是一个基于vi编辑器文本过滤器，它以全屏幕的方式按页显示文本文件的内容，支持vi中的关键字定位操作。more名单中内置了若干快捷键，常用的有H（获得帮助信息），Enter（向下翻滚一行），空格（向下滚动一屏），Q（退出命令）。more命令从前向后读取文件，因此在启动时就加载整个文件。 + +### 🎯 绝对路径和目录切换 + +- 绝对路径： 如/etc/init.d +- 当前目录和上层目录：./ …/ +- 主目录： ~/ +- 切换目录：cd + +### 🎯 怎么查看当前进程？ + +- 查看当前进程：ps +- 执行退出：exit +- 查看当前路径：pwd + +### 🎯 ls命令功能和参数 + + `ls`（**list**）是 Linux 中用来列出目录内容的基本命令，通常用于显示指定目录下的文件和文件夹。 + +- **`ls -a`**：列出所有文件，包括隐藏文件。 +- **`ls -A`**：列出所有文件，但不包括 `.` 和 `..`（当前目录和上级目录的引用）。 +- **`ls -l`**：以长格式列出信息，包括文件的权限、所有者、组、大小、最后修改时间和文件名。 + +--- + +## 🔧 三、系统管理类（运维核心） + +> **核心思想**：系统管理是Linux运维的核心技能，包括进程管理、内存管理、文件系统管理、用户管理等方面，体现系统管理员的专业水平。 + +- **进程管理**：[进程与线程](#🎯-进程与线程概念) | [进程状态](#🎯-进程状态详解) | [僵尸进程](#🎯-什么是僵尸进程如何处理) +- **内存管理**：[虚拟内存](#🎯-虚拟内存机制) | [交换空间](#🎯-交换空间管理) +- **文件系统**：[文件系统类型](#🎯-常见文件系统类型) | [挂载管理](#🎯-文件系统挂载与卸载) | [磁盘管理](#🎯-磁盘空间管理) + +### 🎯 进程与线程概念 + +- 进程与线程的区别 + - 进程是操作系统进行资源分配和调度的基本单位。 + - 线程是进程中的执行单元，线程共享进程的资源。 +- 常见进程命令 + - `ps`: 查看当前进程状态。 + - `top`: 动态查看进程和系统状态。 + - `kill`: 终止进程。 + - `nice`/`renice`: 设置进程的优先级。 + - `bg`, `fg`, `jobs`: 后台与前台进程管理。 + +### 🎯 进程状态详解 + +- 常见的进程状态： + - `R`: 运行中。 + - `S`: 睡眠状态。 + - `Z`: 僵尸状态。 + - `T`: 停止状态。 + +**进程调度** + +- Linux 调度策略： + - Linux 使用 CFS（Completely Fair Scheduler）作为默认调度器。 + - 调度器通过 `nice` 值来调整进程优先级。 + +### 🎯 什么是僵尸进程？如何处理？ + +- 僵尸进程是已终止但父进程尚未收集其退出状态的进程。通过 `kill` 或者让父进程调用 `wait()` 来清理。 + +### 🎯 虚拟内存机制 + +**虚拟内存** + +- 分页与分段： + - Linux 采用分页机制来管理内存，不使用分段。 + - 页表用于虚拟地址到物理地址的映射。 + +**内存管理命令** + +- `free`: 查看内存使用情况。 +- `vmstat`: 查看虚拟内存统计。 +- `top`: 查看进程和内存使用情况。 + +### 🎯 交换空间管理 + +**交换空间** + +- Swap 分区和 Swap 文件： + - 当物理内存不足时，Linux 使用交换空间（Swap）来存储不常用的内存页面。 + - `swapon` / `swapoff`：启用/禁用交换空间。 + +### 🎯 常见文件系统类型 + +**文件系统类型** + +- 常见文件系统： + - ext4: 常用的 Linux 文件系统。 + - xfs: 高性能文件系统。 + - btrfs: 新一代文件系统，支持快照和子卷。 + +### 🎯 文件系统挂载与卸载 + +**挂载与卸载** + +- `mount`: 挂载文件系统。 +- `umount`: 卸载文件系统。 +- `fstab`: 配置文件，定义了系统启动时自动挂载的文件系统。 + +### 🎯 磁盘空间管理 + +**文件系统命令** + +- `df`: 查看磁盘空间使用情况。 +- `du`: 查看磁盘使用的目录空间。 +- `fsck`: 检查文件系统的完整性。 + +--- + +## 📊 四、性能监控类（系统优化） + +> **核心思想**：性能监控和系统调优是高级Linux工程师的必备技能，涉及CPU、内存、磁盘IO、网络等各方面的监控和优化。 + +- **系统监控**：[CPU监控](#🎯-cpu使用率监控) | [内存监控](#🎯-内存使用监控) | [磁盘IO监控](#🎯-磁盘性能监控) +- **性能分析**：[系统负载分析](#🎯-系统负载分析) | [瓶颈定位](#🎯-性能瓶颈定位) +- **优化策略**：[内存优化](#🎯-如何优化内存使用) | [磁盘优化](#🎯-磁盘空间满的快速处理) | [CPU优化](#🎯-如何快速定位-cpu-100-问题) + +### 🎯 cpu使用率监控 + +**系统监控工具** + +- **top/htop**：实时查看系统资源使用情况。 +- **iotop**：实时查看磁盘 IO。 +- **atop**：高级的系统和进程监控工具。 + +- `top` 是一个实时的系统监控工具，可以显示 CPU 使用率以及其他系统资源的使用情况。 + +- `vmstat` 提供了 CPU、内存、交换空间、I/O 等的统计信息。 + + ```bash + vmstat 1 + ``` + + 表示每隔 1 秒输出一次系统统计信息。输出会显示 CPU 使用情况，其中包括： + + - **r**: 就绪队列中的进程数。 + - **b**: 阻塞状态的进程数。 + - **us**: 用户空间使用的 CPU 时间（以百分比表示）。 + - **sy**: 内核空间使用的 CPU 时间（以百分比表示）。 + - **id**: CPU 空闲时间（以百分比表示）。 + +### 🎯 内存使用监控 + +- `free` 是一个简单实用的命令，用于查看系统的内存使用情况。 + + ```bash + free -h + ``` + + - `-h` 参数使输出更加人性化，使用易读的单位（KB, MB, GB）。 + +- 使用 `ps` 命令查看特定进程的资源使用情况 + + ```bash + ps aux | grep + ``` + +### 🎯 如何快速定位 CPU 100% 问题？ + +1. 定位高负载进程 + + ```bash + top -c # 按P排序CPU使用 + pidstat 1 5 # 细粒度进程统计 + ``` + +2. 分析线程状态 + + ```bash + top -H -p [PID] # 查看线程 + printf "%x\n" [TID] # 将线程ID转为16进制 + ``` + +3. 结合 jstack/gdb 查看堆栈 + + ```bash + jstack [PID] | grep -A20 [nid] # Java进程 + gdb -p [PID] -ex "thread apply all bt" -batch # 原生进程 + ``` + +### 🎯 如何优化内存使用？ + +```bash +# 清除缓存（生产环境慎用） +echo 3 > /proc/sys/vm/drop_caches + +# 调整swappiness +sysctl vm.swappiness=10 + +# 透明大页禁用 +echo never > /sys/kernel/mm/transparent_hugepage/enabled +``` + +**优化技巧** + +- **内存管理**： + - 使用 `vmstat`、`free` 查看内存使用情况。 + - 调整 `swappiness` 参数，优化交换空间的使用。 +- **磁盘性能**： + - 使用 `iostat` 监控磁盘性能。 + - 优化磁盘 I/O 性能。 +- **网络优化**： + - 使用 `sysctl` 配置内核参数（例如调节 `tcp_rmem`、`tcp_wmem`）。 + +### 🎯 磁盘空间满的快速处理 + +1. 定位大文件 + + ```bash + du -h --max-depth=1 / 2>/dev/null | sort -hr + ``` + +2. 清理日志文件 + + ```bash + find /var/log -name "*.log" -size +100M -exec truncate -s 0 {} \; + ``` + +3. 处理已删除但未释放空间的文件 + + ```bash + lsof | grep deleted # 查找被删除但未释放的文件 + kill -9 [PID] # 重启相关进程 + ``` + +--- + +## 🌐 五、网络配置类（网络管理） + +> **核心思想**：网络配置和管理是Linux系统管理的重要组成部分，包括网络配置、防火墙管理、SSH配置等。 + +- **网络配置**：[网络接口配置](#🎯-网络接口配置) | [网络命令](#🎯-常用网络命令) +- **网络工具**：[防火墙配置](#🎯-防火墙管理) | [网络扫描](#🎯-网络工具使用) +- **安全管理**：[SELinux管理](#🎯-selinux管理) | [用户权限](#🎯-用户与权限管理) + +### 🎯 网络接口配置 + +**网络配置** + +- 配置文件： + - `/etc/network/interfaces`（Debian/Ubuntu 系统）。 + - `/etc/sysconfig/network-scripts/`（CentOS/RHEL 系统）。 + +### 🎯 常用网络命令 + +**常用网络命令** + +- `ip addr`: 查看或配置 IP 地址。 +- `ping`: 测试网络连通性。 +- `netstat`: 查看网络连接和端口状态。 +- `ss`: 查看套接字状态。 +- `ifconfig`: 网络接口配置（已逐渐被 `ip` 命令取代）。 +- `traceroute`: 路由跟踪。 +- `curl`/`wget`: 下载文件。 + +### 🎯 防火墙管理 + +**网络工具** + +- **iptables**: 配置防火墙。 +- **firewalld**: 动态防火墙管理工具。 +- **nmap**: 网络扫描工具。 + +### 🎯 SELinux管理 + +**SELinux 和 AppArmor** + +- **SELinux**（Security-Enhanced Linux）和 **AppArmor** 是两种内核级别的安全模块，用于增强系统的安全性。 + +**防火墙** + +- **iptables** 和 **firewalld** 用于配置防火墙规则，控制网络访问。 + +**如何禁用 SELinux？** + +- 修改 `/etc/selinux/config` 文件，设置 `SELINUX=disabled`。 + +### 🎯 用户与权限管理 + +**用户与组管理** + +- **sudo**：控制用户的特权。 +- **passwd**：修改用户密码。 +- **groupadd**、**useradd**：添加用户和用户组。 + +--- + +## 🚨 六、故障排查类（问题诊断） + +> **核心思想**：故障排查能力是Linux工程师的核心竞争力，需要掌握系统性的排查思路和常用的诊断工具。 + +- **IO多路复用**：[select/poll/epoll](#🎯-poll-select-epoll-详解) +- **系统诊断**：[服务器响应慢](#🎯-服务器响应缓慢排查) +- **日志管理**：[系统日志](#🎯-系统日志管理) + +### 🎯 poll select epoll 详解 + +在 Linux 的 I/O 多路复用设计中，**select**、**poll** 和 **epoll** 是三种核心机制，它们在实现原理、性能特性和适用场景上有显著差异。以下从内核设计角度详细分析： + +**一、select：基于轮询的原始模型** + +1. **数据结构** + - 使用固定大小的位图（`fd_set`）管理文件描述符（FD），默认支持 1024 个 FD 。 + - 每个 FD 对应一个 bit（如 `FD_SET(fd, &readfds)`），通过 `FD_ISSET` 检查就绪状态 。 +2. **工作流程** + - 调用 `select` 时，内核将用户空间的 `fd_set` 拷贝到内核空间，遍历所有 FD 检查可读/可写状态（时间复杂度 **O(n)**）。 + - 每次调用需重置 FD 集合，导致多次用户态-内核态拷贝开销 。 +3. **局限性** + - **FD 数量限制**：受限于 `FD_SETSIZE`（通常 1024），无法扩展 。 + - **性能瓶颈**：遍历所有 FD，高并发场景下效率低 。 + +**二、poll：链表结构的改进版** + +1. **数据结构** + + - 使用动态链表（`struct pollfd`数组）替代位图，支持无上限的 FD 数量 。 + + ```C + struct pollfd { + int fd; // 文件描述符 + short events; // 关注的事件（如 POLLIN） + short revents; // 实际发生的事件 + }; + ``` + +2. **工作流程** + + - 类似 select，内核遍历所有 FD 检查状态（时间复杂度仍为 **O(n)**）。 + - 通过 `pollfd` 数组传递 FD，避免重复初始化 。 + +3. **优化与不足** + + - **优势**：突破 FD 数量限制，适合中等并发场景 。 + - **劣势**：大量 FD 时，遍历开销仍然显著；需频繁拷贝用户态-内核态数据 。 + +**三、epoll：事件驱动的高效模型** + +1. **数据结构** + - **红黑树**：管理所有注册的 FD（`epoll_ctl` 添加），实现 O(log n) 的查找效率 。 + - **就绪队列**：仅存储活跃 FD，`epoll_wait` 直接返回就绪列表（时间复杂度 **O(1)**）。 +2. **核心机制** + - 注册-回调模式： + 1. **`epoll_ctl`**：注册 FD 及其关注事件（如 `EPOLLIN`），内核通过回调函数跟踪状态变化 。 + 2. **`epoll_wait`**：阻塞等待就绪事件，仅返回活跃 FD 列表，无需遍历全部 FD 。 + - **共享内存**：内核与用户空间共享就绪队列，避免数据拷贝 。 +3. **触发模式** + - **水平触发（LT）**：只要 FD 可读/可写，持续通知（类似 select/poll 行为）。 + - **边缘触发（ET）**：仅在 FD 状态变化时通知一次，需一次处理完所有数据（减少无效唤醒）。 +4. **性能优势** + - **零拷贝**：通过 `mmap` 共享就绪队列，减少用户态-内核态切换 。 + - **高效回调**：仅关注活跃 FD，避免无效遍历，适合数万级高并发场景 。 + +**四、对比总结** + +| **特性** | select | poll | epoll | +| --------------- | -------------------- | --------------------- | --------------------------------- | +| **数据结构** | 位图（`fd_set`） | 链表（`pollfd` 数组） | 红黑树+就绪队列 | +| **FD 数量限制** | 1024（固定） | 无限制 | 无限制（受系统内存限制） | +| **时间复杂度** | O(n) | O(n) | O(1)（就绪队列） | +| **数据拷贝** | 每次调用拷贝 FD 集合 | 同 select | 注册时拷贝一次，后续共享内存 | +| **触发模式** | 仅水平触发 | 同 select | 支持 LT 和 ET 模式 | +| **适用场景** | 低并发、兼容性要求高 | 中低并发、FD 数量较多 | 高并发（如 Web 服务器、实时系统） | + +**五、典型应用场景** + +1. select/poll：旧系统兼容、FD 数量少（如嵌入式设备）。 +2. epoll： + - Nginx（高并发连接）、Redis（高性能事件驱动）。 + - 实时通信系统（如 WebSocket 服务器）。 + +通过以上设计差异，epoll 成为 Linux 高并发网络编程的核心工具，而 select/poll 因历史兼容性仍存在于特定场景中。 + +### 🎯 服务器响应缓慢排查 + +**情景模拟题** + +**场景**：服务器响应缓慢，SSH 连接困难，请描述排查思路 + +**排查步骤**： + +1. 快速登陆后使用 `w` 查看系统负载 +2. `dmesg -T | tail` 检查硬件/驱动错误 +3. `vmstat 1` 查看 CPU、内存、IO 综合情况 +4. `iostat -x 1` 定位磁盘瓶颈 +5. `sar -n DEV 1` 分析网络流量 +6. `pidstat -d 1` 找到高 IO 进程 +7. `strace -p [PID]` 跟踪进程系统调用 +8. 结合业务日志分析异常请求 + +### 🎯 系统日志管理 + +**如何管理系统日志？** + +- 系统日志存放在 `/var/log` 中，可以使用 `tail -f /var/log/syslog` 实时查看日志。 + +--- + +## 💼 七、实战场景类（面试重点） + +> **核心思想**：实战场景类问题是面试中的高频考点，主要考察候选人的实际工作能力和问题解决思路。 + +- **脚本编写**：[Shell脚本](#🎯-shell脚本编写) +- **运维实践**：[快捷键操作](#🎯-常用快捷键) | [命令技巧](#🎯-命令使用技巧) +- **面试常见**：[Linux基础知识](#🎯-linux基础面试题集) + +### 🎯 shell脚本编写 + +**如何编写一个备份日志的 Shell 脚本？** + +```shell +#!/bin/bash +# 备份 /var/log 目录下的 .log 文件到 /backup 目录，保留7天 +BACKUP_DIR="/backup" +LOG_DIR="/var/log" +DATE=$(date +%Y%m%d) + +find $LOG_DIR -name "*.log" -exec cp {} $BACKUP_DIR/logs_$DATE \; +find $BACKUP_DIR -name "logs_*" -mtime +7 -exec rm -f {} \; +``` + +### 🎯 常用快捷键 + +**怎么清屏？怎么退出当前命令？怎么执行睡眠？怎么查看当前用户 id？查看指定帮助用什么命令？** + +- 清屏：clear + +- 退出当前命令：ctrl+c 彻底退出 + +- 执行睡眠 ：ctrl+z挂起当前进程 fg恢复后台查看当前用户id："id"：查看显示目前登陆账户的uid和gid及所属分组及用户名 + +- 查看指定帮助：如man adduser这个很全 而且有例子；adduser–help这个告诉你一些常用参数；info adduesr； + + + +### 🎯 Linux 中如何查看系统资源的使用情况？ + +- `top`, `free`, `df`, `du`, `ps`, `vmstat`, `iotop`。 + + + +### 🎯 如何查找一个文件并搜索文件内容？ + +- `find /path/to/search -name "filename"` 查找文件。 +- `grep "pattern" file` 搜索文件内容。 + + + +### 🎯 如何查看进程使用的内存和 CPU？ + +- `top` 或 `ps aux`。 + + + +### 🎯 如何查看和设置文件权限？ + +- `ls -l` 查看文件权限。 +- `chmod` 修改权限，`chown` 修改文件属主。 + + + +### 🎯 如何管理磁盘空间和挂载文件系统？ + +- 使用 `df`、`du`、`mount`、`umount` 等命令。 + + + +### 🎯 什么是僵尸进程？如何处理？ + +- 僵尸进程是已终止但父进程尚未收集其退出状态的进程。通过 `kill` 或者让父进程调用 `wait()` 来清理。 + + + +### 🎯 **如何管理系统日志？** + +- 系统日志存放在 `/var/log` 中，可以使用 `tail -f /var/log/syslog` 实时查看日志。 + + + +### 🎯 **如何禁用 SELinux？** + +- 修改 `/etc/selinux/config` 文件，设置 `SELINUX=disabled`。 + + + +### 🎯 怎么查找定位到第10行？ + +要查找并定位文件的第10行，你可以使用多种命令和方法，具体取决于你使用的工具和目的。以下是一些常见的方法： + +1. 使用 `sed` 命令 + + `sed` 是一个流编辑器，可以用于按行处理文件。 + + ```bash + sed -n '10p' filename + ``` + + - `-n`: 禁止默认输出。 + + - `'10p'`: 表示输出文件中的第 10 行。 + +2. 使用 `head` 和 `tail` 命令 + + 你可以组合使用 `head` 和 `tail` 命令来显示特定行。 + + ```bash + head -n 10 filename | tail -n 1 + ``` + + - `head -n 10`: 显示文件的前 10 行。 + + - `tail -n 1`: 从 `head` 输出的内容中显示最后一行，即第 10 行。 + +3. 使用 `awk` 命令 + + `awk` 是一个强大的文本处理工具，适用于基于模式的行操作。 + + ```bash + awk 'NR==10' filename + ``` + + - `NR==10`: `NR` 是 `awk` 内部的行号变量，这个命令会输出第 10 行。 + + + +### 🎯 cpu使用率怎么看? 内存使用情况怎们看? + +- `top` 是一个实时的系统监控工具，可以显示 CPU 使用率以及其他系统资源的使用情况。 + +- `vmstat` 提供了 CPU、内存、交换空间、I/O 等的统计信息。 + + ```bash + vmstat 1 + ``` + + 表示每隔 1 秒输出一次系统统计信息。输出会显示 CPU 使用情况，其中包括： + + - **r**: 就绪队列中的进程数。 + - **b**: 阻塞状态的进程数。 + - **us**: 用户空间使用的 CPU 时间（以百分比表示）。 + - **sy**: 内核空间使用的 CPU 时间（以百分比表示）。 + - **id**: CPU 空闲时间（以百分比表示）。 + +- `free` 是一个简单实用的命令，用于查看系统的内存使用情况。 + + ```bash + free -h + ``` + + - `-h` 参数使输出更加人性化，使用易读的单位（KB, MB, GB）。 + +- 使用 `ps` 命令查看特定进程的资源使用情况 + + ```bash + ps aux | grep + ``` + + + +### 🎯 绝对路径用什么符号表示？当前目录、上层目录用什么表示？主目录用什么表示? 切换目录用什么命令？ + +- 绝对路径： 如/etc/init.d +- 当前目录和上层目录：./ …/ +- 主目录： ~/ +- 切换目录：cd + +### 🎯 怎么查看当前进程？怎么执行退出？怎么查看当前路径？ + +- 查看当前进程：ps +- 执行退出：exit +- 查看当前路径：pwd + + + +### 🎯 怎么清屏？怎么退出当前命令？怎么执行睡眠？怎么查看当前用户 id？查看指定帮助用什么命令？ + +- 清屏：clear + +- 退出当前命令：ctrl+c 彻底退出 + +- 执行睡眠 ：ctrl+z挂起当前进程 fg恢复后台查看当前用户id：”id“：查看显示目前登陆账户的uid和gid及所属分组及用户名 + +- 查看指定帮助：如man adduser这个很全 而且有例子；adduser–help这个告诉你一些常用参数；info adduesr； + + + +### 🎯 Ls命令执行什么功能？ 可以带哪些参数，有什么区别？ + + `ls`（**list**）是 Linux 中用来列出目录内容的基本命令，通常用于显示指定目录下的文件和文件夹。 + +- **`ls -a`**：列出所有文件，包括隐藏文件。 +- **`ls -A`**：列出所有文件，但不包括 `.` 和 `..`（当前目录和上级目录的引用）。 +- **`ls -l`**：以长格式列出信息，包括文件的权限、所有者、组、大小、最后修改时间和文件名。 + + + +### 🎯 你平时是怎么查看日志的？ + +Linux查看日志的命令有多种：tail、cat、tac、head、echo等，只介绍几种常用的方法。 + +- tail 最常用的一种查看方式 + + 一般还会配合着grep搜索用，例如; + + ``` + tail -fn 1000 test.log | grep '关键字' + ``` + +- head 跟tail是相反的head是看前多少行日志 + + ``` + head -n 10 test.log 查询日志文件中的头10行日志; head -n -10 test.log 查询日志文件除了最后10行的其他所有日志; + ``` + +- cat + + cat 是由第一行到最后一行连续显示在屏幕上 一次显示整个文件： + + ``` + $ cat filename + ``` + +- more + + more命令是一个基于vi编辑器文本过滤器，它以全屏幕的方式按页显示文本文件的内容，支持vi中的关键字定位操作。more名单中内置了若干快捷键，常用的有H（获得帮助信息），Enter（向下翻滚一行），空格（向下滚动一屏），Q（退出命令）。more命令从前向后读取文件，因此在启动时就加载整个文件。 + + + +### 🎯 硬链接和软链接的区别? + +硬链接与软链接是Linux系统中两种不同的文件链接机制。 + +1. **硬链接** + - 本质：是同一文件的不同别名，共享相同的inode和数据块，相当于多个指针指向同一物理文件。 + - 特点： + - 所有硬链接与原文件权限、大小、修改时间等属性完全一致。 + - 删除任一硬链接仅减少inode的引用计数，数据块保留至所有链接被删除。 + - 适用场景： + - 数据备份与同步：通过硬链接节省空间，多个链接自动同步数据 + - 同一文件系统内多路径访问：如多个用户共享同一配置文件 +2. **软链接（符号链接）** + - 本质：是独立的文件，存储目标文件的路径信息，相当于快捷方式。 + - 特点： + - 拥有独立的inode，文件内容为路径字符串。 + - 若目标文件被删除，软链接成为“悬空链接 + - 适用场景： + - 跨文件系统或设备：例如链接网络存储中的文件 + - 快捷方式与路径管理：简化复杂路径访问，如版本切换（`/opt/app -> /opt/app-v2`。 + - **动态指向更新**：通过修改软链接路径切换目标 + + + +### 🎯 inode是什么? + +**inode**（Index Node，索引节点）是Linux/类Unix文件系统中用于存储文件或目录**元数据**（metadata）的核心数据结构。每个文件或目录在创建时都会被分配一个唯一的inode，其作用类似于文件的“身份证”，记录除文件名以外的所有属性信息，并通过指针关联文件的实际数据块。 + + + +### 🎯 建立软链接(快捷方式)，以及硬链接的命令 + +- 软链接： ln -s slink source +- 硬链接： ln link source + + + +### 🎯 如何编写一个备份日志的 Shell 脚本？ + +```shell +#!/bin/bash +# 备份 /var/log 目录下的 .log 文件到 /backup 目录，保留7天 +BACKUP_DIR="/backup" +LOG_DIR="/var/log" +DATE=$(date +%Y%m%d) + +find $LOG_DIR -name "*.log" -exec cp {} $BACKUP_DIR/logs_$DATE \; +find $BACKUP_DIR -name "logs_*" -mtime +7 -exec rm -f {} \; +``` + + + +### 🎯 如何快速定位 CPU 100% 问题？ + +1. 定位高负载进程 + + ```bash + top -c # 按P排序CPU使用 + pidstat 1 5 # 细粒度进程统计 + ``` + +2. 分析线程状态 + + ```bash + top -H -p [PID] # 查看线程 + printf "%x\n" [TID] # 将线程ID转为16进制 + ``` + +3. 结合 jstack/gdb 查看堆栈 + + ```bash + jstack [PID] | grep -A20 [nid] # Java进程 + gdb -p [PID] -ex "thread apply all bt" -batch # 原生进程 + ``` + + + +### 🎯 如何优化内存使用？ + +```bash +# 清除缓存（生产环境慎用） +echo 3 > /proc/sys/vm/drop_caches + +# 调整swappiness +sysctl vm.swappiness=10 + +# 透明大页禁用 +echo never > /sys/kernel/mm/transparent_hugepage/enabled +``` + + + +### 🎯 磁盘空间满的快速处理 + +1. 定位大文件 + + ```bash + du -h --max-depth=1 / 2>/dev/null | sort -hr + ``` + +2. 清理日志文件 + + ```bash + find /var/log -name "*.log" -size +100M -exec truncate -s 0 {} \; + ``` + +3. 处理已删除但未释放空间的文件 + + ```bash + lsof | grep deleted # 查找被删除但未释放的文件 + kill -9 [PID] # 重启相关进程 + ``` + + + +### 🎯 poll select epoll 稍微详细的说一下 + +在 Linux 的 I/O 多路复用设计中，**select**、**poll** 和 **epoll** 是三种核心机制，它们在实现原理、性能特性和适用场景上有显著差异。以下从内核设计角度详细分析： + +**一、select：基于轮询的原始模型** + +1. **数据结构** + - 使用固定大小的位图（`fd_set`）管理文件描述符（FD），默认支持 1024 个 FD 。 + - 每个 FD 对应一个 bit（如 `FD_SET(fd, &readfds)`），通过 `FD_ISSET` 检查就绪状态 。 +2. **工作流程** + - 调用 `select` 时，内核将用户空间的 `fd_set` 拷贝到内核空间，遍历所有 FD 检查可读/可写状态（时间复杂度 **O(n)**）。 + - 每次调用需重置 FD 集合，导致多次用户态-内核态拷贝开销 。 +3. **局限性** + - **FD 数量限制**：受限于 `FD_SETSIZE`（通常 1024），无法扩展 。 + - **性能瓶颈**：遍历所有 FD，高并发场景下效率低 。 + +**二、poll：链表结构的改进版** + +1. **数据结构** + + - 使用动态链表（`struct pollfd`数组）替代位图，支持无上限的 FD 数量 。 + + ```C + struct pollfd { + int fd; // 文件描述符 + short events; // 关注的事件（如 POLLIN） + short revents; // 实际发生的事件 + }; + ``` + +2. **工作流程** + + - 类似 select，内核遍历所有 FD 检查状态（时间复杂度仍为 **O(n)**）。 + - 通过 `pollfd` 数组传递 FD，避免重复初始化 。 + +3. **优化与不足** + + - **优势**：突破 FD 数量限制，适合中等并发场景 。 + - **劣势**：大量 FD 时，遍历开销仍然显著；需频繁拷贝用户态-内核态数据 。 + +**三、epoll：事件驱动的高效模型** + +1. **数据结构** + - **红黑树**：管理所有注册的 FD（`epoll_ctl` 添加），实现 O(log n) 的查找效率 。 + - **就绪队列**：仅存储活跃 FD，`epoll_wait` 直接返回就绪列表（时间复杂度 **O(1)**）。 +2. **核心机制** + - 注册-回调模式： + 1. **`epoll_ctl`**：注册 FD 及其关注事件（如 `EPOLLIN`），内核通过回调函数跟踪状态变化 。 + 2. **`epoll_wait`**：阻塞等待就绪事件，仅返回活跃 FD 列表，无需遍历全部 FD 。 + - **共享内存**：内核与用户空间共享就绪队列，避免数据拷贝 。 +3. **触发模式** + - **水平触发（LT）**：只要 FD 可读/可写，持续通知（类似 select/poll 行为）。 + - **边缘触发（ET）**：仅在 FD 状态变化时通知一次，需一次处理完所有数据（减少无效唤醒）。 +4. **性能优势** + - **零拷贝**：通过 `mmap` 共享就绪队列，减少用户态-内核态切换 。 + - **高效回调**：仅关注活跃 FD，避免无效遍历，适合数万级高并发场景 。 + +**四、对比总结** + +| **特性** | select | poll | epoll | +| --------------- | -------------------- | --------------------- | --------------------------------- | +| **数据结构** | 位图（`fd_set`） | 链表（`pollfd` 数组） | 红黑树+就绪队列 | +| **FD 数量限制** | 1024（固定） | 无限制 | 无限制（受系统内存限制） | +| **时间复杂度** | O(n) | O(n) | O(1)（就绪队列） | +| **数据拷贝** | 每次调用拷贝 FD 集合 | 同 select | 注册时拷贝一次，后续共享内存 | +| **触发模式** | 仅水平触发 | 同 select | 支持 LT 和 ET 模式 | +| **适用场景** | 低并发、兼容性要求高 | 中低并发、FD 数量较多 | 高并发（如 Web 服务器、实时系统） | + +**五、内核实现细节** + +1. **select 的 `do_select`** + - 遍历所有 FD，调用每个 FD 的 `poll` 方法检查状态（如 socket 的 `tcp_poll`）。 + - 若无就绪 FD，进程进入睡眠，被唤醒后重新遍历 。 +2. **epoll 的回调机制** + - 每个 FD 注册时绑定回调函数（如 `ep_ptable_queue_proc`），状态变化时触发回调，将 FD 加入就绪队列 。 + - 就绪队列通过 `ep_item_poll` 关联到红黑树节点 。 +3. **性能瓶颈突破** + - **红黑树**：快速查找和插入（对比 select/poll 的线性结构）。 + - **事件驱动**：避免轮询，仅处理活跃事件 。 + +**六、典型应用场景** + +1. select/poll：旧系统兼容、FD 数量少（如嵌入式设备）。 +2. epoll： + - Nginx（高并发连接）、Redis（高性能事件驱动）。 + - 实时通信系统（如 WebSocket 服务器）。 + +通过以上设计差异，epoll 成为 Linux 高并发网络编程的核心工具，而 select/poll 因历史兼容性仍存在于特定场景中。 + + + +### 🎯 **情景模拟题** + +**场景**：服务器响应缓慢，SSH 连接困难，请描述排查思路 + +**排查步骤**： + +1. 快速登陆后使用 `w` 查看系统负载 +2. `dmesg -T | tail` 检查硬件/驱动错误 +3. `vmstat 1` 查看 CPU、内存、IO 综合情况 +4. `iostat -x 1` 定位磁盘瓶颈 +5. `sar -n DEV 1` 分析网络流量 +6. `pidstat -d 1` 找到高 IO 进程 +7. `strace -p [PID]` 跟踪进程系统调用 +8. 结合业务日志分析异常请求 + + + +--- + +## 🎯 Linux面试备战指南 + +### 💡 高频考点Top10 + +1. **🔥 文件系统与目录结构** - Linux基础，必考概念 +2. **⚙️ 权限管理与用户管理** - 系统安全的基础 +3. **📊 进程管理与调度** - 操作系统核心原理 +4. **🚨 性能监控与故障排查** - 运维工程师核心技能 +5. **🔍 常用命令与参数** - 日常工作基本功 +6. **💾 内存管理与虚拟内存** - 系统底层原理 +7. **⚡ 网络配置与管理** - 系统网络能力 +8. **🔧 Shell脚本编写** - 自动化运维必备 +9. **📈 IO多路复用机制** - 高性能系统设计 +10. **💼 实际运维经验** - 能结合具体场景解决问题 + +### 🎭 面试答题技巧 + +**📝 标准回答结构** +1. **概念定义**（30秒） - 准确说明是什么 +2. **核心原理**（1分钟） - 解释工作机制和底层原理 +3. **实际应用**（30秒） - 什么场景下使用，解决什么问题 +4. **具体示例**（1分钟） - 举出实际命令或操作示例 +5. **注意事项**（30秒） - 易错点和最佳实践 + +**🗣️ 表达话术模板** +- "从我的运维经验来看..." +- "在生产环境中，通常会..." +- "这个命令的关键参数是..." +- "相比于其他工具，这个的优势在于..." +- "在排查类似问题时，我的思路是..." + +### 🚀 进阶加分点 + +- **内核机制**：能从内核层面解释进程调度、内存管理等机制 +- **性能调优**：有具体的系统优化经验和性能监控数据 +- **脚本能力**：能编写自动化运维脚本，提升工作效率 +- **故障处理**：有排查和解决生产环境故障的实际经验 +- **工具熟练**：熟练使用各种监控、调试、网络工具 + +### 📚 延伸学习建议 + +- **官方文档**：各大Linux发行版的官方文档和手册 +- **内核源码**：深入理解Linux内核的实现原理 +- **实战练习**：搭建实验环境，动手实践各种命令和配置 +- **运维案例**：学习大厂的运维最佳实践和故障处理案例 +- **技术社区**：关注Linux相关的技术博客和开源项目 + +--- + +## 🎉 总结 + +**Linux作为现代服务器和开发环境的基石**，掌握Linux不仅是运维工程师的必备技能，也是后端开发、系统架构师的基本要求。从文件系统到进程管理，从性能监控到网络配置，每一个知识点都体现着对操作系统的深度理解。 + +**掌握Linux的核心在于理论与实践相结合**：理解底层原理的同时，更要在实际工作中灵活运用各种命令和工具。只有在生产环境中经历过故障排查、性能优化的洗礼，才能真正成为Linux高手。 + +**记住：面试官考察的不是你记住了多少命令，而是你能否用Linux思维解决实际的系统问题。** + +**最后一句话**：*"工欲善其事，必先利其器 —— 而Linux就是每个技术人最强大的利器！"* + +--- + +> 💌 **坚持学习，持续成长！** +> 如果这份材料对你有帮助，记得在实际面试中结合自己的理解和经验来回答，让Linux知识真正为你所用！ + - `/etc`: 配置文件目录。 + - `/var`: 可变数据，如日志文件。 + - `/tmp`: 临时文件。 + - `/bin`: 二进制可执行文件。 + - `/lib`: 库文件。 diff --git a/docs/interview/Micro-Services-FAQ.md b/docs/interview/Micro-Services-FAQ.md new file mode 100755 index 0000000000..1853fb957f --- /dev/null +++ b/docs/interview/Micro-Services-FAQ.md @@ -0,0 +1,2236 @@ +--- +title: 微服务架构/Spring Cloud面试题大全 +date: 2024-12-15 +tags: + - 微服务 + - Spring Cloud + - 分布式 + - Interview +categories: Interview +--- + +![](https://img.starfish.ink/common/faq-banner.png) + +微服务架构是现代分布式系统的**核心设计理念**，也是Java后端面试的**重点考查领域**。从单体架构演进到微服务，从服务拆分到治理实践，每个环节都体现着架构师的技术深度。本文档将**微服务核心技术**整理成**系统化知识体系**，涵盖架构设计、Spring Cloud生态、服务治理等关键领域，助你在面试中展现架构思维！ + +微服务面试，围绕着这么几个核心方向准备： + +- **架构理念**（微服务vs单体、拆分原则、边界设计、数据一致性） +- **Spring Cloud生态**（注册中心、配置中心、服务网关、链路追踪） +- **服务治理**（负载均衡、熔断降级、限流机制、灰度发布） +- **通信机制**（同步调用、异步消息、事件驱动、API设计） +- **数据管理**（分库分表、分布式事务、数据同步、CQRS模式） +- **运维部署**（容器化、CI/CD、监控告警、日志收集） +- **实战经验**（架构演进、问题排查、性能调优、最佳实践） + +## 🗺️ 知识导航 + +### 🏷️ 核心知识分类 + +1. **🏗️ 架构基础**：微服务定义、架构对比、拆分策略、边界划分 +2. **☁️ Spring Cloud**：注册发现、配置管理、服务网关、断路器 +3. **🔧 服务治理**：负载均衡、容错处理、链路追踪、服务监控 +4. **💬 服务通信**：同步调用、消息队列、事件驱动、API网关 +5. **🗄️ 数据治理**：数据库拆分、分布式事务、数据一致性、CQRS +6. **🚀 部署运维**：容器化、服务网格、持续集成、监控体系 +7. **🎯 实践案例**：架构演进、性能优化、故障处理、团队协作 + +### 🔑 面试话术模板 + +| **问题类型** | **回答框架** | **关键要点** | **深入扩展** | +| ------------ | ------------------- | ------------ | ------------------ | +| **架构设计** | 背景→原则→方案→效果 | 业务驱动架构 | 技术选型、权衡决策 | +| **技术对比** | 场景→特点→优劣→选择 | 量化对比数据 | 实际项目经验 | +| **问题排查** | 现象→分析→定位→解决 | 系统化思维 | 监控工具、预防措施 | +| **性能优化** | 瓶颈→方案→实施→验证 | 数据驱动优化 | 压测验证、长期监控 | + +--- + +## 🏗️ 一、架构基础与设计 + +**核心理念**：微服务架构通过服务拆分实现系统的高内聚低耦合，提升开发效率和系统可维护性。 + +### 🎯 什么是微服务？与单体架构有什么区别？ + +微服务是一种架构风格，将应用拆分为多个独立的服务： + +**微服务架构特点**： + +- **服务独立**：每个服务可独立开发、部署、扩展 +- **业务导向**：围绕业务能力组织服务 +- **去中心化**：数据和治理的去中心化 +- **容错设计**：服务间故障隔离 +- **技术多样性**：不同服务可用不同技术栈 + +**架构对比分析**： + +| **特性** | **单体架构** | **微服务架构** | +| -------------- | -------------- | ---------------- | +| **部署复杂度** | 简单，一次部署 | 复杂，多服务协调 | +| **开发效率** | 初期高，后期低 | 初期低，后期高 | +| **团队协作** | 集中式开发 | 分布式团队 | +| **技术栈** | 统一技术栈 | 多样化技术栈 | +| **故障影响** | 全局影响 | 局部影响 | +| **数据一致性** | 强一致性 | 最终一致性 | +| **运维成本** | 低 | 高 | + +**💻 架构演进示例**： + +```java +// 单体架构示例 +@RestController +public class ECommerceController { + + @Autowired + private UserService userService; + @Autowired + private ProductService productService; + @Autowired + private OrderService orderService; + @Autowired + private PaymentService paymentService; + + @PostMapping("/orders") + public ResponseEntity createOrder(@RequestBody OrderRequest request) { + // 所有业务逻辑在一个应用中 + User user = userService.getUser(request.getUserId()); + Product product = productService.getProduct(request.getProductId()); + Order order = orderService.createOrder(user, product); + Payment payment = paymentService.processPayment(order); + + return ResponseEntity.ok(order); + } +} + +// 微服务架构演进 +// 用户服务 +@RestController +public class UserController { + + @Autowired + private UserService userService; + + @GetMapping("/users/{id}") + public User getUser(@PathVariable Long id) { + return userService.getUser(id); + } +} + +// 订单服务 +@RestController +public class OrderController { + + @Autowired + private OrderService orderService; + + // 通过RPC调用其他服务 + @Reference + private UserService userService; + @Reference + private ProductService productService; + @Reference + private PaymentService paymentService; + + @PostMapping("/orders") + public ResponseEntity createOrder(@RequestBody OrderRequest request) { + // 调用远程服务 + User user = userService.getUser(request.getUserId()); + Product product = productService.getProduct(request.getProductId()); + + Order order = orderService.createOrder(request); + + // 异步处理支付 + paymentService.processPaymentAsync(order.getId()); + + return ResponseEntity.ok(order); + } +} +``` + +### 🎯 微服务拆分的原则和策略？ + +微服务拆分需要遵循一定的原则和方法： + +**拆分原则**： + +1. **单一职责原则**：每个服务只负责一个业务领域 +2. **高内聚低耦合**：服务内部紧密协作，服务间松散耦合 +3. **业务能力导向**：按照业务能力而非技术层面拆分 +4. **数据独立性**：每个服务管理自己的数据 +5. **可独立部署**：服务可以独立发布和部署 + +**拆分策略**： + +**1. 按业务能力拆分**： + +```java +// 电商系统的业务能力拆分 +- 用户管理服务 (User Management) + - 用户注册、登录、信息管理 + - 用户权限、角色管理 + +- 商品管理服务 (Product Catalog) + - 商品信息管理 + - 库存管理 + - 分类管理 + +- 订单服务 (Order Management) + - 订单创建、状态管理 + - 订单查询、历史记录 + +- 支付服务 (Payment) + - 支付处理 + - 账单管理 + - 退款处理 + +- 物流服务 (Logistics) + - 配送管理 + - 物流跟踪 +``` + +**2. 按数据模型拆分**： + +```java +// 基于领域驱动设计(DDD)的拆分 +@Entity +@Table(name = "users") +public class User { + private Long id; + private String username; + private String email; + // 用户聚合根 +} + +@Entity +@Table(name = "orders") +public class Order { + private Long id; + private Long userId; // 外键关联，不直接持有User对象 + private List items; + // 订单聚合根 +} + +// 服务边界定义 +public interface UserService { + User createUser(CreateUserRequest request); + User getUser(Long userId); + void updateUser(Long userId, UpdateUserRequest request); +} + +public interface OrderService { + Order createOrder(CreateOrderRequest request); + Order getOrder(Long orderId); + List getOrdersByUser(Long userId); +} +``` + +**3. 按团队结构拆分（康威定律）**： + +```java +// 根据组织结构设计服务边界 +/** + * 前端团队 -> 用户界面服务 + * 用户体验团队 -> 用户管理服务 + * 商品团队 -> 商品目录服务 + * 交易团队 -> 订单服务 + 支付服务 + * 运营团队 -> 数据分析服务 + */ + +// 团队自治的微服务 +@SpringBootApplication +public class UserServiceApplication { + // 用户团队完全负责此服务 + // - 需求分析 + // - 技术选型 + // - 开发测试 + // - 部署运维 +} +``` + +### 🎯 如何划定微服务的边界？ + +服务边界划分是微服务设计的核心挑战： + +**边界识别方法**： + +**1. 领域驱动设计（DDD）**： + +```java +// 通过领域建模识别边界上下文 +public class ECommerceDomain { + + // 用户管理子域 + @BoundedContext("UserManagement") + public class UserAggregate { + @AggregateRoot + public class User { + private UserId id; + private UserProfile profile; + private UserPreferences preferences; + } + } + + // 订单管理子域 + @BoundedContext("OrderManagement") + public class OrderAggregate { + @AggregateRoot + public class Order { + private OrderId id; + private List items; + private OrderStatus status; + + // 只通过ID引用其他聚合 + private UserId customerId; // 不直接依赖User对象 + } + } + + // 库存管理子域 + @BoundedContext("Inventory") + public class InventoryAggregate { + @AggregateRoot + public class Product { + private ProductId id; + private Stock stock; + private PriceInfo pricing; + } + } +} +``` + +**2. 数据流分析**： + +```java +// 分析数据的读写模式 +public class DataFlowAnalysis { + + // 高内聚：这些数据经常一起读写 + @ServiceBoundary("UserService") + public class UserData { + - User基本信息 (高频读写) + - UserProfile详细资料 (中频读写) + - UserPreferences偏好设置 (低频读写) + } + + // 低耦合：跨服务的数据访问通过API + @ServiceBoundary("OrderService") + public class OrderData { + - Order订单信息 (高频读写) + - OrderItem订单项 (高频读写) + // 通过API获取用户信息，而不是直接访问用户表 + - 调用UserService.getUser(userId) + } +} +``` + +**3. 业务流程分析**： + +```java +// 分析业务流程的独立性 +public class BusinessProcessAnalysis { + + // 用户注册流程 - 可独立完成 + @IndependentProcess + public void userRegistration() { + validateUserInfo(); + createUserAccount(); + sendWelcomeEmail(); + // 不依赖其他业务领域 + } + + // 订单处理流程 - 需要多服务协作 + @CrossServiceProcess + public void orderProcessing() { + validateUser(); // -> UserService + checkInventory(); // -> InventoryService + calculatePrice(); // -> PricingService + processPayment(); // -> PaymentService + updateInventory(); // -> InventoryService + + // 识别出需要服务编排的流程 + } +} +``` + +--- + +## ☁️ 二、Spring Cloud生态 + +**核心理念**：Spring Cloud为微服务架构提供了完整的技术栈，简化了分布式系统的复杂性。 + +### 🎯 Spring Cloud的核心组件有哪些？ + +Spring Cloud生态系统的主要组件： + +**核心组件架构**： + +| **功能领域** | **组件** | **作用** | **替代方案** | +| ---------------- | --------- | ---------------- | --------------- | +| **服务注册发现** | Eureka | 服务注册中心 | Consul, Nacos | +| **配置管理** | Config | 外部化配置 | Apollo, Nacos | +| **服务网关** | Gateway | 路由、过滤 | Zuul, Kong | +| **负载均衡** | Ribbon | 客户端负载均衡 | LoadBalancer | +| **服务调用** | OpenFeign | 声明式HTTP客户端 | RestTemplate | +| **断路器** | Hystrix | 容错保护 | Resilience4j | +| **链路追踪** | Sleuth | 分布式追踪 | Zipkin, Jaeger | +| **消息总线** | Bus | 配置刷新 | RabbitMQ, Kafka | + +**💻 架构集成示例**： + +```java +// Spring Cloud微服务启动类 +@SpringBootApplication +@EnableEurekaClient // 启用Eureka客户端 +@EnableFeignClients // 启用Feign客户端 +@EnableCircuitBreaker // 启用断路器 +@EnableZipkinServer // 启用链路追踪 +public class UserServiceApplication { + + public static void main(String[] args) { + SpringApplication.run(UserServiceApplication.class, args); + } + + // 负载均衡配置 + @Bean + @LoadBalanced + public RestTemplate restTemplate() { + return new RestTemplate(); + } +} + +// 服务间调用示例 +@FeignClient(name = "user-service", fallback = UserServiceFallback.class) +public interface UserServiceClient { + + @GetMapping("/users/{id}") + User getUser(@PathVariable("id") Long id); + + @PostMapping("/users") + User createUser(@RequestBody CreateUserRequest request); +} + +// 断路器降级处理 +@Component +public class UserServiceFallback implements UserServiceClient { + + @Override + public User getUser(Long id) { + return User.builder() + .id(id) + .username("默认用户") + .build(); + } + + @Override + public User createUser(CreateUserRequest request) { + throw new ServiceUnavailableException("用户服务暂时不可用"); + } +} +``` + +### 🎯 Eureka的工作原理？ + +Eureka是Netflix开源的服务注册与发现组件： + +**Eureka架构**： + +- **Eureka Server**：服务注册中心 +- **Eureka Client**：服务提供者和消费者 +- **服务注册**：应用启动时向Eureka注册 +- **服务发现**：从Eureka获取服务列表 +- **健康检查**：定期检查服务健康状态 + +**工作流程**： + +```java +// 1. Eureka Server配置 +@SpringBootApplication +@EnableEurekaServer +public class EurekaServerApplication { + + public static void main(String[] args) { + SpringApplication.run(EurekaServerApplication.class, args); + } +} + +// application.yml +server: + port: 8761 + +eureka: + instance: + hostname: localhost + client: + register-with-eureka: false # 不向自己注册 + fetch-registry: false # 不从自己拉取注册信息 + service-url: + defaultZone: http://${eureka.instance.hostname}:${server.port}/eureka/ + +// 2. 服务提供者配置 +@SpringBootApplication +@EnableEurekaClient +public class UserServiceApplication { + + @RestController + public class UserController { + + @GetMapping("/users/{id}") + public User getUser(@PathVariable Long id) { + return userService.getUser(id); + } + } +} + +// 服务提供者配置 +spring: + application: + name: user-service + +eureka: + client: + service-url: + defaultZone: http://localhost:8761/eureka/ + instance: + prefer-ip-address: true + lease-renewal-interval-in-seconds: 30 # 心跳间隔 + lease-expiration-duration-in-seconds: 90 # 服务失效时间 + +// 3. 服务消费者 +@RestController +public class OrderController { + + @Autowired + private DiscoveryClient discoveryClient; + + @Autowired + @LoadBalanced + private RestTemplate restTemplate; + + @GetMapping("/orders/{id}") + public Order getOrder(@PathVariable Long id) { + // 方式1：通过DiscoveryClient发现服务 + List instances = + discoveryClient.getInstances("user-service"); + + if (!instances.isEmpty()) { + ServiceInstance instance = instances.get(0); + String userServiceUrl = "http://" + instance.getHost() + + ":" + instance.getPort(); + } + + // 方式2：通过负载均衡调用 + User user = restTemplate.getForObject( + "http://user-service/users/1", User.class); + + return orderService.getOrder(id); + } +} +``` + +**自我保护机制**： + +```java +// Eureka自我保护机制配置 +public class EurekaSelfPreservation { + + /** + * 自我保护触发条件： + * 1. 15分钟内心跳失败比例超过85% + * 2. 网络分区导致心跳丢失 + * + * 保护措施： + * 1. 不再剔除任何服务实例 + * 2. 保留服务注册信息 + * 3. 等待网络恢复 + */ + + // 禁用自我保护（生产环境慎用） + @Configuration + public class EurekaServerConfig { + + @Value("${eureka.server.enable-self-preservation:true}") + private boolean enableSelfPreservation; + + @Bean + public EurekaServerConfigBean eurekaServerConfig() { + EurekaServerConfigBean config = new EurekaServerConfigBean(); + config.setEnableSelfPreservation(enableSelfPreservation); + return config; + } + } +} +``` + +### 🎯 Spring Cloud Config的配置管理原理？ + +Spring Cloud Config提供分布式系统的外部化配置支持： + +**Config架构组成**： + +- **Config Server**：配置服务端，从Git仓库读取配置 +- **Config Client**：配置客户端，从Server获取配置 +- **配置仓库**：Git仓库存储配置文件 +- **消息总线**：配置变更通知机制 + +**💻 实现示例**： + +```java +// 1. Config Server配置 +@SpringBootApplication +@EnableConfigServer +public class ConfigServerApplication { + + public static void main(String[] args) { + SpringApplication.run(ConfigServerApplication.class, args); + } +} + +# Config Server配置 +server: + port: 8888 + +spring: + application: + name: config-server + cloud: + config: + server: + git: + uri: https://github.com/your-org/config-repo + search-paths: '{application}' + clone-on-start: true + health: + enabled: true + +# Git仓库结构 +config-repo/ +├── user-service/ +│ ├── user-service.yml # 默认配置 +│ ├── user-service-dev.yml # 开发环境 +│ ├── user-service-test.yml # 测试环境 +│ └── user-service-prod.yml # 生产环境 +└── order-service/ + ├── order-service.yml + └── ... + +// 2. Config Client配置 +@SpringBootApplication +@RefreshScope // 支持配置热刷新 +public class UserServiceApplication { + + @Value("${user.max-connections:100}") + private int maxConnections; + + @Value("${user.timeout:5000}") + private int timeout; + + @RestController + public class ConfigController { + + @GetMapping("/config") + public Map getConfig() { + Map config = new HashMap<>(); + config.put("maxConnections", maxConnections); + config.put("timeout", timeout); + return config; + } + } +} + +# bootstrap.yml（优先级高于application.yml） +spring: + application: + name: user-service + profiles: + active: dev + cloud: + config: + uri: http://localhost:8888 + profile: ${spring.profiles.active} + label: master + retry: + initial-interval: 2000 + max-attempts: 3 + +// 3. 配置热刷新 +@Component +@RefreshScope +@ConfigurationProperties(prefix = "user") +public class UserProperties { + + private int maxConnections = 100; + private int timeout = 5000; + private String databaseUrl; + + // getters and setters +} + +@RestController +public class ConfigRefreshController { + + @Autowired + private UserProperties userProperties; + + // 手动刷新配置 + @PostMapping("/refresh") + public String refresh() { + // 调用/actuator/refresh端点刷新配置 + return "Configuration refreshed"; + } + + @GetMapping("/properties") + public UserProperties getProperties() { + return userProperties; + } +} + +// 4. 配置加密 +public class ConfigEncryption { + + // 在配置文件中使用加密 + # user-service.yml + datasource: + username: user + password: '{cipher}AQB8HgPVw4dVhK9r8s1w...' # 加密后的密码 + + // Config Server会自动解密 + @Component + public class DatabaseConfig { + + @Value("${datasource.password}") + private String password; // 自动解密后的明文密码 + } +} +``` + +### 🎯 Spring Cloud Gateway的路由机制？ + +Spring Cloud Gateway是基于Spring WebFlux的反应式网关： + +**Gateway核心概念**： + +- **Route（路由）**：网关的基本构建块 +- **Predicate（断言）**：匹配HTTP请求的条件 +- **Filter（过滤器）**：修改请求和响应 + +**💻 Gateway配置示例**： + +```java +// 1. 编程式路由配置 +@Configuration +public class GatewayConfig { + + @Bean + public RouteLocator customRoutes(RouteLocatorBuilder builder) { + return builder.routes() + // 用户服务路由 + .route("user-service", r -> r + .path("/api/users/**") + .filters(f -> f + .stripPrefix(1) // 去除/api前缀 + .addRequestHeader("X-Service", "user") + .addResponseHeader("X-Gateway", "spring-cloud") + .retry(3) // 重试3次 + ) + .uri("lb://user-service") // 负载均衡到user-service + ) + // 订单服务路由 + .route("order-service", r -> r + .path("/api/orders/**") + .and() + .method(HttpMethod.POST, HttpMethod.GET) + .filters(f -> f + .stripPrefix(1) + .addRequestParameter("source", "gateway") + .modifyResponseBody(String.class, String.class, + (exchange, body) -> body.toUpperCase()) + ) + .uri("lb://order-service") + ) + // 静态资源路由 + .route("static-resources", r -> r + .path("/static/**") + .uri("http://cdn.example.com/") + ) + .build(); + } +} + +// 2. 配置文件路由配置 +# application.yml +spring: + cloud: + gateway: + routes: + - id: user-service + uri: lb://user-service + predicates: + - Path=/api/users/** + - Method=GET,POST + - Header=X-Request-Id, \d+ + filters: + - StripPrefix=1 + - AddRequestHeader=X-Service, user + - name: RequestRateLimiter + args: + rate-limiter: "#{@userRateLimiter}" + key-resolver: "#{@ipKeyResolver}" + + - id: order-service + uri: lb://order-service + predicates: + - Path=/api/orders/** + - After=2023-01-01T00:00:00+08:00[Asia/Shanghai] + filters: + - StripPrefix=1 + - name: Hystrix + args: + name: order-fallback + fallbackUri: forward:/order-fallback + +// 3. 自定义过滤器 +@Component +public class AuthGatewayFilterFactory extends AbstractGatewayFilterFactory { + + public AuthGatewayFilterFactory() { + super(Config.class); + } + + @Override + public GatewayFilter apply(Config config) { + return (exchange, chain) -> { + ServerHttpRequest request = exchange.getRequest(); + + // 提取Token + String token = request.getHeaders().getFirst("Authorization"); + + if (token == null || !isValidToken(token)) { + ServerHttpResponse response = exchange.getResponse(); + response.setStatusCode(HttpStatus.UNAUTHORIZED); + return response.setComplete(); + } + + // 添加用户信息到请求头 + String userId = extractUserIdFromToken(token); + ServerHttpRequest modifiedRequest = request.mutate() + .header("X-User-Id", userId) + .build(); + + return chain.filter(exchange.mutate() + .request(modifiedRequest) + .build()); + }; + } + + @Data + public static class Config { + private boolean enabled = true; + private String headerName = "Authorization"; + } + + private boolean isValidToken(String token) { + // JWT token验证逻辑 + return JwtUtil.validateToken(token); + } + + private String extractUserIdFromToken(String token) { + return JwtUtil.getUserId(token); + } +} + +// 4. 限流配置 +@Configuration +public class RateLimitConfig { + + // IP限流 + @Bean + public KeyResolver ipKeyResolver() { + return exchange -> Mono.just( + exchange.getRequest() + .getRemoteAddress() + .getAddress() + .getHostAddress() + ); + } + + // 用户限流 + @Bean + public KeyResolver userKeyResolver() { + return exchange -> Mono.just( + exchange.getRequest() + .getHeaders() + .getFirst("X-User-Id") + ); + } + + // Redis限流器 + @Bean + public RedisRateLimiter userRateLimiter() { + return new RedisRateLimiter( + 10, // 令牌桶容量 + 20 // 每秒补充令牌数 + ); + } +} +``` + +--- + +## 🔧 三、服务治理 + +**核心理念**：微服务治理通过技术手段保障分布式系统的稳定性、可观测性和可维护性。 + +### 🎯 什么是熔断器？Hystrix的工作原理？ + +熔断器是微服务容错的核心模式，防止级联故障： + +**熔断器状态机**： + +1. **CLOSED**：正常状态，请求正常执行 +2. **OPEN**：熔断状态，快速失败 +3. **HALF_OPEN**：半开状态，尝试恢复 + +**💻 Hystrix实现示例**： + +```java +// 1. Hystrix Command实现 +public class UserServiceCommand extends HystrixCommand { + + private final Long userId; + private final UserServiceClient userServiceClient; + + public UserServiceCommand(Long userId, UserServiceClient userServiceClient) { + super(HystrixCommandGroupKey.Factory.asKey("UserService"), + HystrixCommandKey.Factory.asKey("GetUser"), + HystrixCommandProperties.Setter() + .withExecutionTimeoutInMilliseconds(2000) // 超时时间 + .withCircuitBreakerRequestVolumeThreshold(10) // 请求量阈值 + .withCircuitBreakerErrorThresholdPercentage(50) // 错误率阈值 + .withCircuitBreakerSleepWindowInMilliseconds(5000)); // 熔断窗口期 + + this.userId = userId; + this.userServiceClient = userServiceClient; + } + + @Override + protected User run() throws Exception { + return userServiceClient.getUser(userId); + } + + @Override + protected User getFallback() { + // 降级处理 + return User.builder() + .id(userId) + .username("默认用户") + .email("default@example.com") + .build(); + } +} + +// 2. 注解式使用 +@Service +public class OrderService { + + @Autowired + private UserServiceClient userServiceClient; + + @HystrixCommand( + groupKey = "OrderService", + commandKey = "getUser", + fallbackMethod = "getUserFallback", + commandProperties = { + @HystrixProperty(name = "execution.isolation.thread.timeoutInMilliseconds", value = "3000"), + @HystrixProperty(name = "circuitBreaker.requestVolumeThreshold", value = "10"), + @HystrixProperty(name = "circuitBreaker.errorThresholdPercentage", value = "60"), + @HystrixProperty(name = "circuitBreaker.sleepWindowInMilliseconds", value = "10000") + } + ) + public User getUser(Long userId) { + return userServiceClient.getUser(userId); + } + + // 降级方法 + public User getUserFallback(Long userId) { + return User.builder() + .id(userId) + .username("系统繁忙，请稍后再试") + .build(); + } + + // 降级方法（包含异常信息） + public User getUserFallback(Long userId, Throwable throwable) { + log.error("获取用户信息失败，userId: {}", userId, throwable); + + return User.builder() + .id(userId) + .username("服务暂时不可用") + .build(); + } +} + +// 3. Feign集成Hystrix +@FeignClient( + name = "user-service", + fallback = UserServiceFallback.class, + fallbackFactory = UserServiceFallbackFactory.class +) +public interface UserServiceClient { + + @GetMapping("/users/{id}") + User getUser(@PathVariable("id") Long id); + + @PostMapping("/users") + User createUser(@RequestBody CreateUserRequest request); +} + +// 降级实现 +@Component +public class UserServiceFallback implements UserServiceClient { + + @Override + public User getUser(Long id) { + return createDefaultUser(id); + } + + @Override + public User createUser(CreateUserRequest request) { + throw new ServiceUnavailableException("用户服务暂时不可用，请稍后重试"); + } + + private User createDefaultUser(Long id) { + return User.builder() + .id(id) + .username("默认用户" + id) + .status(UserStatus.UNKNOWN) + .build(); + } +} + +// 带异常信息的降级工厂 +@Component +public class UserServiceFallbackFactory implements FallbackFactory { + + @Override + public UserServiceClient create(Throwable cause) { + return new UserServiceClient() { + @Override + public User getUser(Long id) { + log.error("调用用户服务失败，userId: {}", id, cause); + + if (cause instanceof TimeoutException) { + return createTimeoutUser(id); + } else if (cause instanceof ConnectException) { + return createConnectionErrorUser(id); + } else { + return createDefaultUser(id); + } + } + + @Override + public User createUser(CreateUserRequest request) { + throw new ServiceUnavailableException("用户服务创建功能暂时不可用: " + cause.getMessage()); + } + }; + } +} + +// 4. Hystrix监控 +@Configuration +@EnableHystrixMetricsStream +public class HystrixConfig { + + @Bean + public ServletRegistrationBean hystrixMetricsStreamServlet() { + ServletRegistrationBean registration = + new ServletRegistrationBean<>(new HystrixMetricsStreamServlet()); + registration.addUrlMappings("/hystrix.stream"); + return registration; + } +} +``` + +### 🎯 服务链路追踪如何实现？ + +链路追踪帮助定位分布式系统中的性能瓶颈和故障点： + +**Sleuth + Zipkin实现**： + +```java +// 1. 添加链路追踪依赖和配置 +# pom.xml + + org.springframework.cloud + spring-cloud-starter-sleuth + + + org.springframework.cloud + spring-cloud-sleuth-zipkin + + +# application.yml +spring: + sleuth: + sampler: + probability: 1.0 # 采样率（生产环境建议0.1） + zipkin: + base-url: http://localhost:9411 + sender: + type: web + application: + name: order-service + +// 2. 自定义Span +@Service +public class OrderService { + + private final Tracer tracer; + private final UserServiceClient userServiceClient; + + public OrderService(Tracer tracer, UserServiceClient userServiceClient) { + this.tracer = tracer; + this.userServiceClient = userServiceClient; + } + + @NewSpan("create-order") // 自动创建新的Span + public Order createOrder(@SpanTag("userId") Long userId, + @SpanTag("productId") Long productId) { + + // 手动创建子Span + Span userSpan = tracer.nextSpan() + .name("get-user") + .tag("user.id", String.valueOf(userId)) + .start(); + + try (Tracer.SpanInScope ws = tracer.withSpanInScope(userSpan)) { + User user = userServiceClient.getUser(userId); + userSpan.tag("user.name", user.getUsername()); + + // 继续处理订单逻辑 + return processOrder(user, productId); + + } catch (Exception e) { + userSpan.tag("error", e.getMessage()); + throw e; + } finally { + userSpan.end(); + } + } + + @NewSpan("process-order") + private Order processOrder(@SpanTag("user") User user, + @SpanTag("productId") Long productId) { + + // 添加自定义标签 + tracer.currentSpan() + .tag("order.type", "online") + .tag("user.level", user.getLevel().name()); + + // 模拟业务处理 + Thread.sleep(100); + + Order order = Order.builder() + .userId(user.getId()) + .productId(productId) + .status(OrderStatus.CREATED) + .build(); + + // 记录事件 + tracer.currentSpan().annotate("order.created"); + + return orderRepository.save(order); + } +} + +// 3. 链路追踪过滤器 +@Component +public class TraceFilter implements Filter { + + private final Tracer tracer; + + public TraceFilter(Tracer tracer) { + this.tracer = tracer; + } + + @Override + public void doFilter(ServletRequest request, ServletResponse response, + FilterChain chain) throws IOException, ServletException { + + HttpServletRequest httpRequest = (HttpServletRequest) request; + + // 创建根Span + Span span = tracer.nextSpan() + .name("http:" + httpRequest.getMethod().toLowerCase()) + .tag("http.method", httpRequest.getMethod()) + .tag("http.url", httpRequest.getRequestURL().toString()) + .tag("http.user_agent", httpRequest.getHeader("User-Agent")) + .start(); + + try (Tracer.SpanInScope ws = tracer.withSpanInScope(span)) { + chain.doFilter(request, response); + + HttpServletResponse httpResponse = (HttpServletResponse) response; + span.tag("http.status_code", String.valueOf(httpResponse.getStatus())); + + } catch (Exception e) { + span.tag("error", e.getMessage()); + throw e; + } finally { + span.end(); + } + } +} + +// 4. 链路追踪配置 +@Configuration +public class TracingConfiguration { + + // 自定义采样策略 + @Bean + public ProbabilityBasedSampler probabilityBasedSampler() { + return new ProbabilityBasedSampler(0.1f); // 10%采样率 + } + + // 自定义Span命名策略 + @Bean + public SpanNamer spanNamer() { + return new DefaultSpanNamer(); + } + + // 跳过某些URL的追踪 + @Bean + public SkipPatternProvider skipPatternProvider() { + return () -> Pattern.compile("/health|/metrics|/actuator/.*"); + } + + // 自定义标签 + @Bean + public SpanCustomizer spanCustomizer() { + return span -> { + span.tag("service.version", "1.0.0"); + span.tag("service.env", "production"); + }; + } +} + +// 5. 异步调用链路追踪 +@Service +public class AsyncOrderService { + + private final Tracer tracer; + + @Async("orderExecutor") + @TraceAsync // 确保异步方法中的链路追踪 + public CompletableFuture processOrderAsync(Long orderId) { + + // 获取当前Span + Span currentSpan = tracer.currentSpan(); + + return CompletableFuture.supplyAsync(() -> { + // 在异步线程中继续使用Span + try (Tracer.SpanInScope ws = tracer.withSpanInScope(currentSpan)) { + + Span asyncSpan = tracer.nextSpan() + .name("async-process-order") + .tag("order.id", String.valueOf(orderId)) + .start(); + + try (Tracer.SpanInScope asyncWs = tracer.withSpanInScope(asyncSpan)) { + // 异步处理逻辑 + return processOrder(orderId); + } finally { + asyncSpan.end(); + } + } + }); + } +} +``` + +--- + +## 💬 四、服务通信 + +**核心理念**：微服务间通信需要选择合适的方式，同步调用保证一致性，异步消息提高解耦性。 + +### 🎯 微服务间通信有哪些方式？ + +微服务通信模式及其适用场景： + +**通信方式分类**： + +| **通信模式** | **实现方式** | **优点** | **缺点** | **适用场景** | +| ------------ | ------------------ | ------------------ | -------------------- | ---------------------- | +| **同步调用** | HTTP/REST, RPC | 简单直观，强一致性 | 耦合度高，级联故障 | 查询操作，实时性要求高 | +| **异步消息** | 消息队列，事件总线 | 解耦，高可用 | 复杂性高，最终一致性 | 业务流程，状态变更通知 | +| **事件驱动** | Event Sourcing | 完全解耦，可重放 | 复杂度极高 | 复杂业务流程 | + +**💻 通信实现示例**： + +```java +// 1. 同步HTTP调用 +@RestController +public class OrderController { + + @Autowired + private RestTemplate restTemplate; + + @Autowired + private UserServiceClient userServiceClient; + + @PostMapping("/orders") + public ResponseEntity createOrder(@RequestBody CreateOrderRequest request) { + + // 方式1：RestTemplate调用 + User user = restTemplate.getForObject( + "http://user-service/users/" + request.getUserId(), + User.class); + + // 方式2：Feign客户端调用 + Product product = userServiceClient.getProduct(request.getProductId()); + + // 创建订单 + Order order = orderService.createOrder(user, product, request); + + return ResponseEntity.ok(order); + } +} + +// 2. 异步消息通信 +@Service +public class OrderEventService { + + @Autowired + private RabbitTemplate rabbitTemplate; + + @Autowired + private KafkaTemplate kafkaTemplate; + + // 发布订单创建事件 + @EventListener + public void handleOrderCreated(OrderCreatedEvent event) { + + // RabbitMQ消息发送 + rabbitTemplate.convertAndSend( + "order.exchange", + "order.created", + event + ); + + // Kafka消息发送 + kafkaTemplate.send("order-events", event.getOrderId().toString(), event); + + log.info("订单创建事件已发布: {}", event.getOrderId()); + } +} + +// 消息监听处理 +@RabbitListener(queues = "inventory.order.created") +public class InventoryService { + + @RabbitHandler + public void handleOrderCreated(OrderCreatedEvent event) { + log.info("处理订单创建事件，更新库存: {}", event.getOrderId()); + + // 更新库存 + inventoryRepository.decreaseStock( + event.getProductId(), + event.getQuantity() + ); + + // 发布库存更新事件 + applicationEventPublisher.publishEvent( + new StockUpdatedEvent(event.getProductId(), event.getQuantity()) + ); + } +} + +// 3. 事件驱动架构 +@Component +public class OrderSagaOrchestrator { + + @Autowired + private PaymentService paymentService; + @Autowired + private InventoryService inventoryService; + @Autowired + private ShippingService shippingService; + + @SagaOrchestrationStart + @EventListener + public void handleOrderCreated(OrderCreatedEvent event) { + + // 创建Saga实例 + SagaInstance saga = SagaInstance.builder() + .sagaId(UUID.randomUUID().toString()) + .orderId(event.getOrderId()) + .status(SagaStatus.STARTED) + .build(); + + // 第一步：处理支付 + processPayment(saga, event); + } + + private void processPayment(SagaInstance saga, OrderCreatedEvent event) { + try { + PaymentRequest paymentRequest = PaymentRequest.builder() + .orderId(event.getOrderId()) + .amount(event.getTotalAmount()) + .userId(event.getUserId()) + .build(); + + paymentService.processPaymentAsync(paymentRequest) + .whenComplete((result, throwable) -> { + if (throwable == null) { + handlePaymentSuccess(saga, result); + } else { + handlePaymentFailure(saga, throwable); + } + }); + + } catch (Exception e) { + handlePaymentFailure(saga, e); + } + } + + private void handlePaymentSuccess(SagaInstance saga, PaymentResult result) { + saga.setStatus(SagaStatus.PAYMENT_COMPLETED); + sagaRepository.save(saga); + + // 第二步：预留库存 + reserveInventory(saga); + } + + private void handlePaymentFailure(SagaInstance saga, Throwable error) { + saga.setStatus(SagaStatus.PAYMENT_FAILED); + saga.setErrorMessage(error.getMessage()); + sagaRepository.save(saga); + + // 触发补偿事务 + cancelOrder(saga); + } +} +``` + +### 🎯 如何设计API网关？ + +API网关是微服务架构的入口，统一处理横切关注点： + +**网关核心功能**： + +- 路由转发 +- 认证授权 +- 限流熔断 +- 监控日志 +- 协议转换 + +**💻 网关实现示例**： + +```java +// 1. 网关路由配置 +@Configuration +public class GatewayRouteConfig { + + @Bean + public RouteLocator customRoutes(RouteLocatorBuilder builder) { + return builder.routes() + + // 用户服务路由 + .route("user-api", r -> r + .path("/api/v1/users/**") + .filters(f -> f + .stripPrefix(2) // 去除 /api/v1 前缀 + .addRequestHeader("X-Gateway-Version", "1.0") + .requestRateLimiter(config -> config + .setRateLimiter(redisRateLimiter()) + .setKeyResolver(ipKeyResolver())) + ) + .uri("lb://user-service") + ) + + // 订单服务路由（需要认证） + .route("order-api", r -> r + .path("/api/v1/orders/**") + .filters(f -> f + .stripPrefix(2) + .filter(authenticationFilter()) // 自定义认证过滤器 + .circuitBreaker(config -> config + .setName("order-circuit-breaker") + .setFallbackUri("forward:/fallback/order")) + ) + .uri("lb://order-service") + ) + + // WebSocket路由 + .route("websocket-route", r -> r + .path("/ws/**") + .uri("lb:ws://websocket-service") + ) + + .build(); + } + + // 认证过滤器 + @Bean + public GatewayFilter authenticationFilter() { + return (exchange, chain) -> { + ServerHttpRequest request = exchange.getRequest(); + + // 检查认证头 + String authHeader = request.getHeaders().getFirst("Authorization"); + + if (authHeader == null || !authHeader.startsWith("Bearer ")) { + return unauthorizedResponse(exchange); + } + + String token = authHeader.substring(7); + + // 验证JWT token + return validateToken(token) + .flatMap(userInfo -> { + // 添加用户信息到请求头 + ServerHttpRequest modifiedRequest = request.mutate() + .header("X-User-Id", userInfo.getUserId()) + .header("X-User-Role", userInfo.getRole()) + .build(); + + return chain.filter(exchange.mutate() + .request(modifiedRequest) + .build()); + }) + .onErrorResume(throwable -> unauthorizedResponse(exchange)); + }; + } + + private Mono unauthorizedResponse(ServerWebExchange exchange) { + ServerHttpResponse response = exchange.getResponse(); + response.setStatusCode(HttpStatus.UNAUTHORIZED); + response.getHeaders().add("Content-Type", "application/json"); + + String body = "{\"error\":\"Unauthorized\",\"message\":\"Invalid or missing token\"}"; + DataBuffer buffer = response.bufferFactory().wrap(body.getBytes()); + + return response.writeWith(Mono.just(buffer)); + } +} + +// 2. 全局过滤器 +@Component +public class GlobalLoggingFilter implements GlobalFilter, Ordered { + + private static final Logger logger = LoggerFactory.getLogger(GlobalLoggingFilter.class); + + @Override + public Mono filter(ServerWebExchange exchange, GatewayFilterChain chain) { + ServerHttpRequest request = exchange.getRequest(); + + String requestId = UUID.randomUUID().toString(); + long startTime = System.currentTimeMillis(); + + // 记录请求信息 + logger.info("Gateway Request - ID: {}, Method: {}, URL: {}, Headers: {}", + requestId, request.getMethod(), request.getURI(), request.getHeaders()); + + // 添加请求ID到响应头 + exchange.getResponse().getHeaders().add("X-Request-Id", requestId); + + return chain.filter(exchange) + .doOnSuccess(aVoid -> { + long duration = System.currentTimeMillis() - startTime; + logger.info("Gateway Response - ID: {}, Status: {}, Duration: {}ms", + requestId, exchange.getResponse().getStatusCode(), duration); + }) + .doOnError(throwable -> { + long duration = System.currentTimeMillis() - startTime; + logger.error("Gateway Error - ID: {}, Duration: {}ms, Error: {}", + requestId, duration, throwable.getMessage(), throwable); + }); + } + + @Override + public int getOrder() { + return -1; // 最高优先级 + } +} + +// 3. 限流配置 +@Configuration +public class RateLimitConfig { + + @Bean + public RedisRateLimiter redisRateLimiter() { + return new RedisRateLimiter( + 10, // replenishRate: 每秒填充的令牌数 + 20, // burstCapacity: 令牌桶的容量 + 1 // requestedTokens: 每个请求消耗的令牌数 + ); + } + + // IP地址限流 + @Bean + @Primary + public KeyResolver ipKeyResolver() { + return exchange -> Mono.just( + Objects.requireNonNull(exchange.getRequest().getRemoteAddress()) + .getAddress() + .getHostAddress() + ); + } + + // 用户ID限流 + @Bean + public KeyResolver userKeyResolver() { + return exchange -> Mono.just( + exchange.getRequest().getHeaders().getFirst("X-User-Id") + ); + } + + // API路径限流 + @Bean + public KeyResolver apiKeyResolver() { + return exchange -> Mono.just( + exchange.getRequest().getPath().value() + ); + } +} + +// 4. 熔断降级 +@RestController +public class FallbackController { + + @GetMapping("/fallback/user") + public Mono> userFallback() { + Map response = new HashMap<>(); + response.put("error", "User Service Unavailable"); + response.put("message", "用户服务暂时不可用，请稍后重试"); + response.put("timestamp", System.currentTimeMillis()); + + return Mono.just(ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE) + .body(response)); + } + + @GetMapping("/fallback/order") + public Mono> orderFallback() { + Map response = new HashMap<>(); + response.put("error", "Order Service Unavailable"); + response.put("message", "订单服务暂时不可用，请稍后重试"); + response.put("timestamp", System.currentTimeMillis()); + + return Mono.just(ResponseEntity.status(HttpStatus.SERVICE_UNAVAILABLE) + .body(response)); + } +} + +// 5. 网关监控 +@Component +public class GatewayMetricsCollector { + + private final MeterRegistry meterRegistry; + private final Counter requestCounter; + private final Timer requestTimer; + + public GatewayMetricsCollector(MeterRegistry meterRegistry) { + this.meterRegistry = meterRegistry; + this.requestCounter = Counter.builder("gateway.requests.total") + .description("Total gateway requests") + .register(meterRegistry); + this.requestTimer = Timer.builder("gateway.requests.duration") + .description("Gateway request duration") + .register(meterRegistry); + } + + @EventListener + public void handleGatewayRequest(GatewayRequestEvent event) { + requestCounter.increment( + Tags.of( + Tag.of("method", event.getMethod()), + Tag.of("status", String.valueOf(event.getStatus())), + Tag.of("service", event.getTargetService()) + ) + ); + + requestTimer.record(event.getDuration(), TimeUnit.MILLISECONDS, + Tags.of( + Tag.of("service", event.getTargetService()), + Tag.of("endpoint", event.getEndpoint()) + ) + ); + } +} +``` + +--- + +## 🗄️ 五、数据管理 + +**核心理念**：微服务数据管理需要平衡数据一致性和服务独立性，通过合理的数据架构设计实现业务目标。 + +### 🎯 微服务如何处理分布式事务？ + +分布式事务是微服务架构的核心挑战之一： + +**分布式事务模式**： + +1. **两阶段提交（2PC）**：强一致性，但性能差 +2. **补偿事务（TCC）**：Try-Confirm-Cancel模式 +3. **Saga模式**：长流程事务管理 +4. **本地消息表**：最终一致性方案 + +**💻 Saga模式实现**： + +```java +// 1. Saga编排器实现 +@Component +public class OrderSagaOrchestrator { + + @Autowired + private PaymentService paymentService; + @Autowired + private InventoryService inventoryService; + @Autowired + private ShippingService shippingService; + @Autowired + private SagaInstanceRepository sagaRepository; + + // 订单创建Saga流程 + @SagaStart + public void createOrderSaga(CreateOrderCommand command) { + + SagaInstance saga = SagaInstance.builder() + .sagaId(UUID.randomUUID().toString()) + .orderId(command.getOrderId()) + .status(SagaStatus.STARTED) + .compensations(new ArrayList<>()) + .build(); + + sagaRepository.save(saga); + + try { + // 步骤1：创建订单 + createOrder(saga, command); + + } catch (Exception e) { + handleSagaFailure(saga, e); + } + } + + private void createOrder(SagaInstance saga, CreateOrderCommand command) { + try { + Order order = orderService.createOrder(command); + saga.addCompensation(new CancelOrderCompensation(order.getId())); + + // 步骤2：处理支付 + processPayment(saga, order); + + } catch (Exception e) { + throw new SagaException("创建订单失败", e); + } + } + + private void processPayment(SagaInstance saga, Order order) { + try { + PaymentResult result = paymentService.processPayment( + PaymentRequest.builder() + .orderId(order.getId()) + .amount(order.getTotalAmount()) + .userId(order.getUserId()) + .build() + ); + + saga.addCompensation(new RefundPaymentCompensation(result.getPaymentId())); + + // 步骤3：预留库存 + reserveInventory(saga, order); + + } catch (Exception e) { + executeCompensations(saga); + throw new SagaException("支付处理失败", e); + } + } + + private void reserveInventory(SagaInstance saga, Order order) { + try { + for (OrderItem item : order.getItems()) { + InventoryReservation reservation = inventoryService.reserveInventory( + ReserveInventoryRequest.builder() + .productId(item.getProductId()) + .quantity(item.getQuantity()) + .orderId(order.getId()) + .build() + ); + + saga.addCompensation( + new ReleaseInventoryCompensation(reservation.getReservationId()) + ); + } + + // 步骤4：安排发货 + arrangeShipping(saga, order); + + } catch (Exception e) { + executeCompensations(saga); + throw new SagaException("库存预留失败", e); + } + } + + private void arrangeShipping(SagaInstance saga, Order order) { + try { + ShippingOrder shippingOrder = shippingService.createShippingOrder( + CreateShippingOrderRequest.builder() + .orderId(order.getId()) + .address(order.getShippingAddress()) + .items(order.getItems()) + .build() + ); + + saga.addCompensation( + new CancelShippingCompensation(shippingOrder.getId()) + ); + + // Saga成功完成 + completeSaga(saga); + + } catch (Exception e) { + executeCompensations(saga); + throw new SagaException("安排发货失败", e); + } + } + + private void completeSaga(SagaInstance saga) { + saga.setStatus(SagaStatus.COMPLETED); + saga.setCompletedAt(LocalDateTime.now()); + sagaRepository.save(saga); + + // 发布Saga完成事件 + eventPublisher.publishEvent(new SagaCompletedEvent(saga.getSagaId())); + } + + private void executeCompensations(SagaInstance saga) { + saga.setStatus(SagaStatus.COMPENSATING); + sagaRepository.save(saga); + + // 逆序执行补偿操作 + List compensations = saga.getCompensations(); + Collections.reverse(compensations); + + for (Compensation compensation : compensations) { + try { + compensation.execute(); + compensation.setStatus(CompensationStatus.COMPLETED); + } catch (Exception e) { + compensation.setStatus(CompensationStatus.FAILED); + log.error("补偿操作失败: {}", compensation.getClass().getSimpleName(), e); + } + } + + saga.setStatus(SagaStatus.COMPENSATED); + sagaRepository.save(saga); + } +} + +// 2. TCC模式实现 +@Service +@Transactional +public class AccountTccService { + + @Autowired + private AccountRepository accountRepository; + @Autowired + private TccTransactionRepository tccTransactionRepository; + + /** + * Try阶段：预留资源 + */ + @TccTry + public void tryDebit(String accountId, BigDecimal amount, String transactionId) { + Account account = accountRepository.findById(accountId) + .orElseThrow(() -> new AccountNotFoundException(accountId)); + + // 检查余额 + if (account.getBalance().compareTo(amount) < 0) { + throw new InsufficientFundsException(accountId, amount); + } + + // 冻结资金 + account.setBalance(account.getBalance().subtract(amount)); + account.setFrozenAmount(account.getFrozenAmount().add(amount)); + accountRepository.save(account); + + // 记录TCC事务 + TccTransaction tccTransaction = TccTransaction.builder() + .transactionId(transactionId) + .accountId(accountId) + .amount(amount) + .status(TccStatus.TRIED) + .type(TccType.DEBIT) + .createdAt(LocalDateTime.now()) + .build(); + + tccTransactionRepository.save(tccTransaction); + } + + /** + * Confirm阶段：确认事务 + */ + @TccConfirm + public void confirmDebit(String transactionId) { + TccTransaction tccTransaction = tccTransactionRepository.findByTransactionId(transactionId) + .orElseThrow(() -> new TccTransactionNotFoundException(transactionId)); + + if (tccTransaction.getStatus() != TccStatus.TRIED) { + throw new IllegalTccStatusException(transactionId, tccTransaction.getStatus()); + } + + Account account = accountRepository.findById(tccTransaction.getAccountId()) + .orElseThrow(() -> new AccountNotFoundException(tccTransaction.getAccountId())); + + // 确认扣款，释放冻结资金 + account.setFrozenAmount(account.getFrozenAmount().subtract(tccTransaction.getAmount())); + accountRepository.save(account); + + // 更新TCC事务状态 + tccTransaction.setStatus(TccStatus.CONFIRMED); + tccTransaction.setUpdatedAt(LocalDateTime.now()); + tccTransactionRepository.save(tccTransaction); + } + + /** + * Cancel阶段：取消事务 + */ + @TccCancel + public void cancelDebit(String transactionId) { + TccTransaction tccTransaction = tccTransactionRepository.findByTransactionId(transactionId) + .orElseThrow(() -> new TccTransactionNotFoundException(transactionId)); + + if (tccTransaction.getStatus() != TccStatus.TRIED) { + // 幂等处理：如果已经取消或确认，直接返回 + return; + } + + Account account = accountRepository.findById(tccTransaction.getAccountId()) + .orElseThrow(() -> new AccountNotFoundException(tccTransaction.getAccountId())); + + // 恢复账户余额 + account.setBalance(account.getBalance().add(tccTransaction.getAmount())); + account.setFrozenAmount(account.getFrozenAmount().subtract(tccTransaction.getAmount())); + accountRepository.save(account); + + // 更新TCC事务状态 + tccTransaction.setStatus(TccStatus.CANCELLED); + tccTransaction.setUpdatedAt(LocalDateTime.now()); + tccTransactionRepository.save(tccTransaction); + } +} + +// 3. 本地消息表模式 +@Service +@Transactional +public class OrderServiceWithLocalMessage { + + @Autowired + private OrderRepository orderRepository; + @Autowired + private LocalMessageRepository localMessageRepository; + + public Order createOrder(CreateOrderRequest request) { + // 本地事务中同时操作业务数据和消息表 + Order order = Order.builder() + .userId(request.getUserId()) + .productId(request.getProductId()) + .quantity(request.getQuantity()) + .status(OrderStatus.CREATED) + .createdAt(LocalDateTime.now()) + .build(); + + order = orderRepository.save(order); + + // 保存待发送消息 + LocalMessage message = LocalMessage.builder() + .id(UUID.randomUUID().toString()) + .topic("order-events") + .key(order.getId().toString()) + .payload(JsonUtils.toJson(new OrderCreatedEvent(order))) + .status(MessageStatus.PENDING) + .createdAt(LocalDateTime.now()) + .maxRetries(3) + .currentRetries(0) + .build(); + + localMessageRepository.save(message); + + return order; + } + + // 定时任务发送未成功的消息 + @Scheduled(fixedDelay = 5000) + public void sendPendingMessages() { + List pendingMessages = localMessageRepository + .findByStatusAndCurrentRetriesLessThan(MessageStatus.PENDING, 3); + + for (LocalMessage message : pendingMessages) { + try { + // 发送消息 + kafkaTemplate.send(message.getTopic(), message.getKey(), message.getPayload()) + .whenComplete((result, ex) -> { + if (ex == null) { + // 发送成功 + message.setStatus(MessageStatus.SENT); + message.setSentAt(LocalDateTime.now()); + } else { + // 发送失败，增加重试次数 + message.setCurrentRetries(message.getCurrentRetries() + 1); + if (message.getCurrentRetries() >= message.getMaxRetries()) { + message.setStatus(MessageStatus.FAILED); + message.setErrorMessage(ex.getMessage()); + } + } + localMessageRepository.save(message); + }); + + } catch (Exception e) { + log.error("发送消息失败: {}", message.getId(), e); + message.setCurrentRetries(message.getCurrentRetries() + 1); + if (message.getCurrentRetries() >= message.getMaxRetries()) { + message.setStatus(MessageStatus.FAILED); + message.setErrorMessage(e.getMessage()); + } + localMessageRepository.save(message); + } + } + } +} +``` + +### 🎯 如何实现数据一致性？ + +微服务环境下的数据一致性策略： + +**一致性级别**： + +- **强一致性**：所有节点同时看到相同数据 +- **弱一致性**：允许临时不一致 +- **最终一致性**：保证最终数据一致 + +**💻 实现方案**： + +```java +// 1. 事件溯源（Event Sourcing） +@Entity +public class OrderAggregate { + + @Id + private String orderId; + private OrderStatus status; + private BigDecimal totalAmount; + private LocalDateTime createdAt; + + @OneToMany(cascade = CascadeType.ALL) + private List events = new ArrayList<>(); + + // 从事件重建聚合状态 + public static OrderAggregate fromEvents(List events) { + OrderAggregate aggregate = new OrderAggregate(); + + for (OrderEvent event : events) { + aggregate.applyEvent(event); + } + + return aggregate; + } + + private void applyEvent(OrderEvent event) { + switch (event.getEventType()) { + case ORDER_CREATED: + OrderCreatedEvent createdEvent = (OrderCreatedEvent) event.getEventData(); + this.orderId = createdEvent.getOrderId(); + this.status = OrderStatus.CREATED; + this.totalAmount = createdEvent.getTotalAmount(); + this.createdAt = createdEvent.getCreatedAt(); + break; + + case ORDER_PAID: + this.status = OrderStatus.PAID; + break; + + case ORDER_SHIPPED: + this.status = OrderStatus.SHIPPED; + break; + + case ORDER_CANCELLED: + this.status = OrderStatus.CANCELLED; + break; + } + } + + // 创建订单 + public void createOrder(CreateOrderCommand command) { + // 业务验证 + validateCreateOrder(command); + + // 生成事件 + OrderCreatedEvent event = OrderCreatedEvent.builder() + .orderId(command.getOrderId()) + .userId(command.getUserId()) + .totalAmount(command.getTotalAmount()) + .items(command.getItems()) + .createdAt(LocalDateTime.now()) + .build(); + + addEvent(new OrderEvent(EventType.ORDER_CREATED, event)); + applyEvent(events.get(events.size() - 1)); + } + + private void addEvent(OrderEvent event) { + event.setEventId(UUID.randomUUID().toString()); + event.setTimestamp(LocalDateTime.now()); + event.setVersion(events.size() + 1); + events.add(event); + } +} + +// 2. CQRS模式实现 +// 命令端（写模型） +@Service +public class OrderCommandService { + + @Autowired + private OrderEventStore eventStore; + + public void createOrder(CreateOrderCommand command) { + // 加载聚合 + OrderAggregate aggregate = eventStore.loadAggregate(command.getOrderId()); + + // 执行业务逻辑 + aggregate.createOrder(command); + + // 保存事件 + eventStore.saveEvents(command.getOrderId(), aggregate.getUncommittedEvents()); + + // 发布事件 + publishEvents(aggregate.getUncommittedEvents()); + } + + private void publishEvents(List events) { + for (OrderEvent event : events) { + eventPublisher.publishEvent(event); + } + } +} + +// 查询端（读模型） +@Service +public class OrderQueryService { + + @Autowired + private OrderReadModelRepository readModelRepository; + + // 处理订单创建事件，更新读模型 + @EventListener + public void handleOrderCreated(OrderCreatedEvent event) { + OrderReadModel readModel = OrderReadModel.builder() + .orderId(event.getOrderId()) + .userId(event.getUserId()) + .status("CREATED") + .totalAmount(event.getTotalAmount()) + .createdAt(event.getCreatedAt()) + .build(); + + readModelRepository.save(readModel); + } + + @EventListener + public void handleOrderPaid(OrderPaidEvent event) { + OrderReadModel readModel = readModelRepository.findById(event.getOrderId()) + .orElseThrow(() -> new OrderNotFoundException(event.getOrderId())); + + readModel.setStatus("PAID"); + readModel.setPaidAt(event.getPaidAt()); + + readModelRepository.save(readModel); + } + + // 查询方法 + public Page findOrdersByUser(String userId, Pageable pageable) { + return readModelRepository.findByUserId(userId, pageable); + } + + public Optional findOrderById(String orderId) { + return readModelRepository.findById(orderId); + } +} + +// 3. 数据同步策略 +@Component +public class DataSynchronizer { + + @Autowired + private RedisTemplate redisTemplate; + + // 缓存一致性：写后删除模式 + @EventListener + @Async + public void handleDataUpdated(DataUpdatedEvent event) { + // 删除相关缓存 + Set keys = redisTemplate.keys("user:" + event.getUserId() + ":*"); + if (!keys.isEmpty()) { + redisTemplate.delete(keys); + } + + // 预热重要缓存 + if (event.isImportantData()) { + preloadCache(event.getUserId()); + } + } + + // 双写一致性：先更新数据库，再更新缓存 + @Transactional + public void updateUserData(String userId, UserData userData) { + // 1. 更新数据库 + userRepository.updateUser(userId, userData); + + // 2. 更新缓存（可能失败） + try { + redisTemplate.opsForValue().set("user:" + userId, userData, Duration.ofHours(1)); + } catch (Exception e) { + log.warn("更新缓存失败，用户ID: {}", userId, e); + // 异步重试更新缓存 + asyncUpdateCache(userId, userData); + } + } + + @Async + @Retryable(value = Exception.class, maxAttempts = 3, backoff = @Backoff(delay = 1000)) + public void asyncUpdateCache(String userId, UserData userData) { + redisTemplate.opsForValue().set("user:" + userId, userData, Duration.ofHours(1)); + } +} +``` + +--- + +## 🎯 面试重点总结 + +### 高频考点速览 + +- **架构理念**：微服务vs单体对比、拆分原则、边界设计的系统性思考 +- **Spring Cloud**：核心组件功能、Eureka注册发现、Config配置管理、Gateway路由机制 +- **服务治理**：熔断降级、链路追踪、负载均衡的设计与实现原理 +- **服务通信**：同步异步调用选择、API网关设计、消息驱动架构 +- **数据管理**：分布式事务处理、数据一致性保证、CQRS模式应用 +- **实践经验**：架构演进路径、性能优化手段、故障排查方法 + +### 面试答题策略 + +1. **架构设计题**：先分析业务场景，再阐述技术方案，最后说明权衡考虑 +2. **技术原理题**：从问题背景出发，讲解实现原理，对比不同方案优劣 +3. **实践经验题**：结合具体项目场景，展示问题解决思路和优化效果 +4. **故障处理题**：系统性分析故障现象、定位过程、解决方案、预防措施 + +--- + +## 📚 扩展学习 + +- **官方文档**：Spring Cloud、Netflix OSS等框架的官方文档深入学习 +- **架构实践**：研读大厂微服务架构实践案例，了解真实场景的技术选型 +- **源码分析**：深入Spring Cloud核心组件源码，理解实现原理和设计思想 +- **性能优化**：学习微服务性能监控、调优方法和工具使用 +- **运维实践**：掌握微服务部署、监控、故障排查的DevOps实践 diff --git a/docs/interview/MyBatis-FAQ.md b/docs/interview/MyBatis-FAQ.md index 8694c81d0b..108cc6807d 100644 --- a/docs/interview/MyBatis-FAQ.md +++ b/docs/interview/MyBatis-FAQ.md @@ -1 +1,3647 @@ -Mybatis \ No newline at end of file +--- +title: MyBatis/ORM框架面试题大全 +date: 2024-12-15 +tags: + - MyBatis + - ORM + - Interview +categories: Interview +--- + +![](https://img.starfish.ink/common/faq-banner.png) + +> MyBatis作为Java生态中**最流行的半自动ORM框架**，是面试中的**必考重点**。从SQL映射到动态SQL，从缓存机制到性能调优，每一个知识点都可能成为面试的关键。本文档将**MyBatis核心知识**整理成**标准面试话术**，并补充JPA、Hibernate等ORM框架对比，让你在面试中游刃有余！ +> +> +> MyBatis 面试，围绕着这么几个核心方向准备： +> +> - **MyBatis核心原理**（SqlSession、Mapper代理、执行流程、参数映射、结果映射） +> - **动态SQL与映射**（#{} vs ${}、动态SQL标签、resultMap、关联查询） +> - **缓存机制**（一级缓存、二级缓存、缓存失效、自定义缓存） +> - **插件与扩展**（拦截器机制、分页插件、性能监控、审计字段） +> - **事务与数据源**（Spring集成、多数据源、分布式事务、读写分离） +> - **性能优化**（N+1问题、批处理、连接池、SQL调优、执行计划） +> - **ORM框架对比**（MyBatis vs JPA/Hibernate、MyBatis-Plus特性分析） + +## 🗺️ 知识导航 + +### 🏷️ 核心知识分类 + +1. **基础与架构**：MyBatis架构、执行流程、Mapper机制、类型与结果映射 +2. **SQL与动态SQL**：#{} vs ${}、动态SQL标签、批量操作、联表/关联查询 +3. **缓存机制**：一级缓存、二级缓存、缓存命中与失效场景、自定义缓存 +4. **插件与扩展**：拦截器四大点位、插件链、分页、审计、加解密 +5. **事务与多数据源**：Spring整合、事务传播、Seata/分布式事务、读写分离 +6. **性能与调优**：参数与日志、批处理、延迟加载、N+1、连接池、索引与执行计划 +7. **工程化与规范**：Mapper规范、SQL规范、错误码与异常处理、灰度与回滚 +8. **ORM生态对比**：MyBatis vs JPA/Hibernate、MyBatis-Plus特性与利弊 + +--- + +## 🧠 一、MyBatis核心原理 + + **核心理念**：半自动ORM框架，提供SQL与Java对象之间的映射，保持SQL的灵活性和可控性。 + +### 🎯 JDBC 有几个步骤？ + +JDBC（Java DataBase Connectivity）是 Java 程序与关系型数据库交互的统一 API + +JDBC 大致可以分为六个步骤： + +1. 注册数据库驱动类，指定数据库地址，其中包括 DB 的用户名、密码及其他连接信息； +2. 调用 `DriverManager.getConnection()` 方法创建 Connection 连接到数据库； +3. 调用 Connection 的 `createStatement() `或 `prepareStatement()` 方法，创建 Statement 对象，此时会指定 SQL（或是 SQL 语句模板 + SQL 参数）； +4. 通过 Statement 对象执行 SQL 语句，得到 ResultSet 对象，也就是查询结果集； +5. 遍历 ResultSet，从结果集中读取数据，并将每一行数据库记录转换成一个 JavaBean 对象； +6. 关闭 ResultSet 结果集、Statement 对象及数据库 Connection，从而释放这些对象占用的底层资源。 + + + +### 🎯 什么是 ORM? + +全称为 Object Relational Mapping。对象-映射-关系型数据库。对象关系映射(简称 ORM，或 O/RM，或 O/R mapping)，用于实现面向对象编程语言里不同类型系统的数据之间的转换。简单的说，ORM 是通过使用描述对象和数据库之间映射的元数据，将程序中的对象与关系数据库相互映射。 + +ORM 提供了实现持久化层的另一种模式，它采用映射元数据来描述对象关系的映射，使得 ORM 中间件能在任何一个应用的业务逻辑层和数据库层之间充当桥梁。 + +无论是执行查询操作，还是执行其他 DML 操作，JDBC 操作步骤都会重复出现。为了简化重复逻辑，提高代码的可维护性，可以将 JDBC 重复逻辑封装到一个类似 DBUtils 的工具类中，在使用时只需要调用 DBUtils 工具类中的方法即可。当然，我们也可以使用“反射+配置”的方式，将关系模型到对象模型的转换进行封装，但是这种封装要做到通用化且兼顾灵活性，就需要一定的编程功底。 + +**ORM 框架的核心功能：根据配置（配置文件或是注解）实现对象模型、关系模型两者之间无感知的映射**（如下图）。 + +![What is ORM. Object-relational mapping (ORM) emerged… | by Kavya | Medium](https://miro.medium.com/v2/resize:fit:1200/0*MAXI8BnsQC4G5rcg.png) + +在生产环境中，数据库一般都是比较稀缺的，数据库连接也是整个服务中比较珍贵的资源之一。建立数据库连接涉及鉴权、握手等一系列网络操作，是一个比较耗时的操作，所以我们不能像上述 JDBC 基本操作流程那样直接释放掉数据库连接，否则持久层很容易成为整个系统的性能瓶颈。 + +Java 程序员一般会使用数据库连接池的方式进行优化，此时就需要引入第三方的连接池实现，当然，也可以自研一个连接池，但是要处理连接活跃数、控制连接的状态等一系列操作还是有一定难度的。另外，有一些查询返回的数据是需要本地缓存的，这样可以提高整个程序的查询性能，这就需要缓存的支持。 + +如果没有 ORM 框架的存在，这就需要我们 Java 开发者熟悉相关连接池、缓存等组件的 API 并手动编写一些“黏合”代码来完成集成，而且这些代码重复度很高，这显然不是我们希望看到的结果。 + +很多 ORM 框架都支持集成第三方缓存、第三方数据源等常用组件，并对外提供统一的配置接入方式，这样我们只需要使用简单的配置即可完成第三方组件的集成。当我们需要更换某个第三方组件的时候，只需要引入相关依赖并更新配置即可，这就大大提高了开发效率以及整个系统的可维护性。 + + + +### 🎯 什么是MyBatis？核心组件和架构是什么？ + +> MyBatis 是一款优秀的持久层框架，它支持自定义 SQL、存储过程以及高级映射。 +> +> MyBatis 免除了几乎所有的 JDBC 代码以及设置参数和获取结果集的工作。 +> +> MyBatis 可以通过简单的 XML 或注解来配置和映射原始类型、接口和 Java POJO（Plain Old Java Objects，普通老式 Java 对象）为数据库中的记录。 + +MyBatis是一个半自动的ORM持久化框架： + +**MyBatis定义**： + +- 支持自定义SQL、存储过程和高级映射 +- 避免了几乎所有的JDBC代码和手动设置参数 +- 使用简单的XML或注解用于配置和原始映射 +- 将接口和Java的POJO映射成数据库中的记录 + +**MyBatis核心组件**： + +**1. SqlSessionFactory**： +- MyBatis的核心对象，用于创建SqlSession +- 通过SqlSessionFactoryBuilder构建 +- 一个应用只需要一个SqlSessionFactory实例 +- 生命周期应该是应用级别的 + +**2. SqlSession**： +- 执行SQL命令的主要接口 +- 包含了面向数据库执行的所有方法 +- 线程不安全，不能被共享 +- 使用后需要关闭以释放资源 + +**3. Executor（执行器）**： +- MyBatis的核心执行器接口 +- SimpleExecutor：简单执行器，每次执行都创建Statement +- ReuseExecutor：复用执行器，复用PreparedStatement +- BatchExecutor：批量执行器，用于批量更新操作 + +**4. MappedStatement**： +- 映射语句的封装对象 +- 包含SQL配置信息、参数映射、结果映射等 +- 每个 + SELECT * FROM user + WHERE id = #{id} AND name = #{name} + + + + + + + + + + + + + + + +``` + +```java +// 安全使用${}的Service层实现 +@Service +public class UserService { + + @Autowired + private UserMapper userMapper; + + // 表名白名单，防止SQL注入 + private static final Set ALLOWED_TABLES = Set.of( + "user", "user_backup", "user_2024", "user_archive + ); + + // 允许的排序字段白名单 + private static final Set ALLOWED_SORT_FIELDS = Set.of( + "id", "name", "email", "create_time", "update_time + ); + + public List findByTableName(String tableName) { + // 严格校验表名，防止SQL注入 + if (!ALLOWED_TABLES.contains(tableName)) { + throw new IllegalArgumentException("Invalid table name: " + tableName); + } + + return userMapper.findByTableName(tableName); + } + + public List findUsersWithSort(String sortField, String sortOrder) { + // 校验排序字段 + if (!ALLOWED_SORT_FIELDS.contains(sortField)) { + throw new IllegalArgumentException("Invalid sort field: " + sortField); + } + + // 校验排序方向 + if (!"ASC".equalsIgnoreCase(sortOrder) && !"DESC".equalsIgnoreCase(sortOrder)) { + throw new IllegalArgumentException("Invalid sort order: " + sortOrder); + } + + return userMapper.findUsersWithSort(sortField, sortOrder); + } + + // 动态列查询的安全实现 + public List> selectUserColumns(List columnNames) { + // 定义允许查询的列 + Set allowedColumns = Set.of( + "id", "name", "email", "phone", "create_time", "status + ); + + // 验证所有列名都在白名单中 + for (String column : columnNames) { + if (!allowedColumns.contains(column)) { + throw new IllegalArgumentException("Invalid column name: " + column); + } + } + + String columns = String.join(",", columnNames); + return userMapper.selectColumns(columns); + } +} +``` + +### 🎯 MyBatis动态SQL有哪些标签？如何使用？ + +MyBatis提供了强大的动态SQL功能，通过XML标签实现条件化的SQL构建： + + **核心动态SQL标签**： + + **1. `` 条件判断标签**： + - 根据条件决定是否包含某段SQL + - test属性支持OGNL表达式 + - 常用于WHERE条件的动态构建 + + **2. `//` 多分支选择**： + - 类似于Java的switch-case语句 + - 只有一个分支会被执行 + - 相当于default分支 + + **3. `` 智能WHERE子句**： + - 自动添加WHERE关键字 + - 自动处理AND/OR逻辑，去除多余的AND/OR + - 如果没有条件则不添加WHERE + + **4. `` 智能SET子句**： + - 用于UPDATE语句的动态SET子句 + - 自动去除末尾的逗号 + - 至少需要一个字段才生效 + + **5. `` 通用修剪标签**： + - prefix/suffix：添加前缀/后缀 + - prefixOverrides/suffixOverrides：去除指定的前缀/后缀 + - 和都是的特殊形式 + + **6. `` 循环遍历标签**： + - 遍历集合生成SQL片段 + - 支持List、Array、Map等集合类型 + - 常用于IN查询、批量INSERT等场景 + + **7. `` 变量绑定标签**： + - 创建一个变量并绑定到上下文 + - 常用于模糊查询的LIKE语句 + - 可以进行字符串拼接和处理 + +**💻 代码示例**： + +```xml + + + + + + + + + + + UPDATE user + + + name = #{name}, + + + email = #{email}, + + + phone = #{phone}, + + + status = #{status}, + + + update_time = NOW() + + WHERE id = #{id} + + + + + + + + + + + + INSERT INTO user (name, email, phone, status) VALUES + + (#{user.name}, #{user.email}, #{user.phone}, #{user.status}) + + + + + + + UPDATE user SET + name = #{user.name}, + email = #{user.email} + WHERE id = #{user.id} + + + + + + + + + + + + id, name, email, phone, status, create_time, update_time + + + + + + AND status = #{status} + + + AND name LIKE CONCAT('%', #{name}, '%') + + + + + + + +``` + + + +### 🎯 模糊查询like语句该怎么写 + +1. '%${question}%' 可能引起SQL注入，不推荐 + +2. "%"#{question}"%" 注意：因为#{...}解析成sql语句时候，会在变量外侧自动加单引号' '，所以这里 % 需要使用双引号" "，不能使用单引号 ' '，不然会查不到任何结果。 + +3. CONCAT('%',#{question},'%') 使用CONCAT()函数，推荐 + +4. 使用bind标签 + +```xml + +``` + + + +### 🎯 当实体类中的属性名和表中的字段名不一样 ，怎么办 + +第1种：通过在查询的SQL语句中定义字段名的别名，让字段名的别名和实体类的属性名一致。 + +```xml + +``` + +第2种：通过`` +来映射字段名和实体类属性名的一一对应关系。 + +```xml + + + + + + + + + + +``` + + + +### 🎯 使用MyBatis的mapper接口调用时有哪些要求？ + +- Mapper.xml文件中的namespace即是mapper接口的全限定类名。 +- Mapper接口方法名和mapper.xml中定义的sql语句id一一对应。 +- Mapper接口方法的输入参数类型和mapper.xml中定义的每个sql语句的parameterType的类型相同。 +- Mapper接口方法的输出参数类型和mapper.xml中定义的每个sql语句的resultType的类型相同。 + + + +### 🎯 MyBatis是如何进行分页的？分页插件的原理是什么？ + +Mybatis使用RowBounds对象进行分页，它是针对ResultSet结果集执行的内存分页，而非物理分页，可以在sql内直接书写带有物理分页的参数来完成物理分页功能，也可以使用分页插件来完成物理分页。 + +分页插件的基本原理是使用Mybatis提供的插件接口，实现自定义插件，通过jdk动态代理在插件的拦截方法内拦截待执行的sql，然后重写sql，根据dialect方言，添加对应的物理分页语句和参数。 + +举例：select * from student，拦截sql后重写为：select t.* from (select * from student) t limit 0, 10 + + + +### 🎯 resultType resultMap 的区别? + +- 类的名字和数据库相同时，可以直接设置 resultType 参数为 Pojo 类 + +- 若不同，需要设置 resultMap 将结果名字和 Pojo 名字进行转换 + +--- + + + +## 🧠 三、缓存机制 + + **核心理念**：MyBatis提供两级缓存机制来提升查询性能，通过合理的缓存策略减少数据库访问，提高应用响应速度。 + +### 🎯 MyBatis的一级缓存和二级缓存是什么？有什么区别？ + + + +> 在应用运行过程中，我们有可能在一次数据库会话中，执行多次查询条件完全相同的SQL，MyBatis提供了一级缓存的方案优化这部分场景，如果是相同的SQL语句，会优先命中一级缓存，避免直接对数据库进行查询，提高性能。 +> +> ![img](https://awps-assets.meituan.net/mit-x/blog-images-bundle-2018a/6e38df6a.jpg) +> +> 每个SqlSession中持有了Executor，每个Executor中有一个LocalCache。当用户发起查询时，MyBatis根据当前执行的语句生成`MappedStatement`，在Local Cache进行查询，如果缓存命中的话，直接返回结果给用户，如果缓存没有命中的话，查询数据库，结果写入`Local Cache`，最后返回结果给用户。具体实现类的类关系图如下图所示。 +> +> ![img](https://awps-assets.meituan.net/mit-x/blog-images-bundle-2018a/d76ec5fe.jpg) +> +> 一级缓存中，其最大的共享范围就是一个SqlSession内部，如果多个SqlSession之间需要共享缓存，则需要使用到二级缓存 +> +> 二级缓存开启后，同一个namespace下的所有操作语句，都影响着同一个Cache，即二级缓存被多个SqlSession共享，是一个全局的变量。 +> +> 当开启缓存后，数据的查询执行的流程就是 二级缓存 -> 一级缓存 -> 数据库 + +1. 一级缓存: 基于 PerpetualCache 的 HashMap 本地缓存，其存储作用域为 Session，当 Session flush 或 close 之后，该 Session 中的所有 Cache 就将清空，MyBatis默认打开一级缓存。 + +2. 二级缓存与一级缓存机制相同，默认也是采用 PerpetualCache，HashMap 存储，不同之处在于其存储作用域为 Mapper(Namespace)，并且可自定义存储源，如 Ehcache。默认不打开二级缓存，要开启二级缓存，使用二级缓存属性类需要实现Serializable序列化接口(可用来保存对象的状态)，可在它的映射文件中配置`` + 标签； + +3. 对于缓存数据更新机制，当某一个作用域(一级缓存 Session/二级缓存Namespaces)进行了C/U/D 操作后，默认该作用域下所有缓存将被清理掉。 + + **主要区别对比**： + - **作用域**：一级缓存是SqlSession级别，二级缓存是Mapper级别 + - **生命周期**：一级缓存随SqlSession销毁，二级缓存可以持续存在 + - **共享性**：一级缓存不能共享，二级缓存可以跨SqlSession共享 + - **配置**：一级缓存默认开启，二级缓存需要手动配置 + - **序列化**：一级缓存不需要序列化，二级缓存需要对象可序列化 + +**💻 代码示例**： + +```java +// 一级缓存示例 +@Test +public void testFirstLevelCache() { + SqlSession session = sqlSessionFactory.openSession(); + try { + UserMapper mapper = session.getMapper(UserMapper.class); + + // 第一次查询，从数据库获取 + User user1 = mapper.selectById(1L); + System.out.println("第一次查询: " + user1); + + // 第二次查询，从一级缓存获取（相同SqlSession） + User user2 = mapper.selectById(1L); + System.out.println("第二次查询: " + user2); + + // 验证是同一个对象 + System.out.println("是否为同一对象: " + (user1 == user2)); // true + + // 执行更新操作，清空一级缓存 + mapper.updateUser(new User(2L, "Updated Name")); + + // 再次查询，重新从数据库获取 + User user3 = mapper.selectById(1L); + System.out.println("更新后查询: " + user3); + System.out.println("是否为同一对象: " + (user1 == user3)); // false + + } finally { + session.close(); + } +} + +// 验证不同SqlSession的一级缓存隔离 +@Test +public void testFirstLevelCacheIsolation() { + // 第一个SqlSession + SqlSession session1 = sqlSessionFactory.openSession(); + UserMapper mapper1 = session1.getMapper(UserMapper.class); + User user1 = mapper1.selectById(1L); + + // 第二个SqlSession + SqlSession session2 = sqlSessionFactory.openSession(); + UserMapper mapper2 = session2.getMapper(UserMapper.class); + User user2 = mapper2.selectById(1L); // 重新从数据库查询 + + System.out.println("不同Session是否为同一对象: " + (user1 == user2)); // false + + session1.close(); + session2.close(); +} +``` + +```xml + + + + + + + + + + + + + + + + + + + + UPDATE user SET name = #{name} WHERE id = #{id} + + + + + UPDATE user SET last_access_time = NOW() WHERE id = #{id} + + +``` + +```java +// 二级缓存示例 +@Test +public void testSecondLevelCache() { + // 第一个SqlSession查询数据 + SqlSession session1 = sqlSessionFactory.openSession(); + try { + UserMapper mapper1 = session1.getMapper(UserMapper.class); + User user1 = mapper1.selectById(1L); + System.out.println("Session1查询: " + user1); + } finally { + session1.close(); // 关闭session，数据进入二级缓存 + } + + // 第二个SqlSession查询相同数据 + SqlSession session2 = sqlSessionFactory.openSession(); + try { + UserMapper mapper2 = session2.getMapper(UserMapper.class); + User user2 = mapper2.selectById(1L); // 从二级缓存获取 + System.out.println("Session2查询: " + user2); + } finally { + session2.close(); + } +} +``` + + + +### 🎯 什么情况下MyBatis缓存会失效？如何控制缓存行为？ + +MyBatis缓存失效涉及多种场景，了解这些场景有助于合理使用缓存： + + **一级缓存失效场景**： + + **1. SqlSession关闭或提交**： + - SqlSession.close()时清空缓存 + - SqlSession.commit()或rollback()时清空缓存 + - 这是最常见的缓存失效场景 + + **2. 执行DML操作**： + - 任何INSERT、UPDATE、DELETE操作都会清空一级缓存 + - 包括其他Mapper的DML操作（同一SqlSession内） + - 防止脏读问题 + + **3. 手动清理缓存**： + - 调用SqlSession.clearCache()方法 + - 主动清空当前SqlSession的缓存 + + **4. 不同的查询条件**： + - SQL语句不同 + - 参数值不同 + - 分页参数不同 + - RowBounds不同 + + **5. localCacheScope设置**： + - 设置为STATEMENT时，每次语句执行后都清空缓存 + - 默认为SESSION，缓存在整个Session期间有效 + + **二级缓存失效场景**： + + **1. 命名空间内的DML操作**： + - 当前namespace的任何INSERT、UPDATE、DELETE操作 + - 会清空该namespace的所有二级缓存 + - 保证数据一致性 + + **2. 配置属性控制**： + - flushCache="true"：强制清空缓存 + - useCache="false"：不使用缓存 + - 可以针对特定语句进行精确控制 + + **3. 缓存策略触发**： + - 达到缓存大小限制，触发淘汰策略 + - 达到刷新间隔时间，自动清空缓存 + - LRU、FIFO等淘汰算法的执行 + + **4. 序列化问题**： + - 缓存对象未实现Serializable接口 + - 序列化/反序列化过程出错 + + **缓存控制最佳实践**： + - 合理设置缓存策略和大小 + - 读多写少的场景适合使用二级缓存 + - 实时性要求高的数据不建议缓存 + - 定期监控缓存命中率和内存使用情况 + +**💻 代码示例**： + +```java +// 缓存失效演示 +@Test +public void testCacheInvalidation() { + SqlSession session = sqlSessionFactory.openSession(); + try { + UserMapper mapper = session.getMapper(UserMapper.class); + + // 第一次查询，建立缓存 + User user1 = mapper.selectById(1L); + System.out.println("第一次查询: " + user1); + + // 第二次查询，使用缓存 + User user2 = mapper.selectById(1L); + System.out.println("第二次查询（缓存）: " + user2); + System.out.println("缓存命中: " + (user1 == user2)); // true + + // 执行更新操作，导致缓存失效 + User updateUser = new User(2L, "New Name"); + mapper.updateUser(updateUser); + + // 再次查询，缓存已失效，重新查询数据库 + User user3 = mapper.selectById(1L); + System.out.println("更新后查询: " + user3); + System.out.println("缓存失效: " + (user1 == user3)); // false + + // 手动清空缓存 + session.clearCache(); + User user4 = mapper.selectById(1L); + System.out.println("手动清空后查询: " + user4); + + } finally { + session.close(); + } +} +``` + +```xml + + + + + + flushInterval="600000" + size="512" + readOnly="false" + blocking="true"/> + + + + + + + + + + UPDATE user SET name = #{name}, update_time = NOW() WHERE id = #{id} + + + + + UPDATE user SET view_count = view_count + 1 WHERE id = #{id} + + + + + +``` + +### 🎯 如何自定义MyBatis缓存？如何集成Redis等外部缓存？ + +MyBatis支持自定义缓存实现，可以集成Redis、Ehcache等第三方缓存系统： + + **自定义缓存实现步骤**： + + **1. 实现Cache接口**： + - org.apache.ibatis.cache.Cache接口定义了缓存的基本操作 + - 包括put、get、remove、clear等方法 + - 需要提供唯一的缓存ID标识 + + **2. 处理并发安全**： + - 缓存实现必须是线程安全的 + - 可以使用synchronized或并发集合 + - 考虑读写锁优化性能 + + **3. 配置缓存策略**： + - 淘汰策略（LRU、FIFO等） + - 过期时间控制 + - 内存大小限制 + + **Redis缓存集成方案**： + + **使用现有组件**： + - mybatis-redis：官方提供的Redis缓存实现 + - redisson-mybatis：基于Redisson的缓存实现 + - 配置简单，功能完善 + + **自定义Redis缓存**： + - 更灵活的配置选项 + - 可以定制序列化方式 + - 支持分布式缓存场景 + + **缓存使用注意事项**： + - 数据一致性：缓存与数据库的同步策略 + - 缓存穿透：大量请求不存在的数据 + - 缓存雪崩：缓存同时失效导致数据库压力激增 + - 缓存击穿：热点数据失效导致并发查询数据库 + +--- + +## 🧱 四、插件与扩展 + + **核心理念**：MyBatis通过拦截器机制提供强大的扩展能力，支持在SQL执行的不同阶段进行干预和增强，实现分页、审计、加解密等高级功能。 + +### 🎯 MyBatis插件的原理是什么？可以拦截哪些对象和方法？ + +MyBatis插件基于JDK动态代理和责任链模式实现，提供了强大的扩展机制： + + **插件实现原理**： + + **1. 拦截器接口**： + - 所有插件必须实现org.apache.ibatis.plugin.Interceptor接口 + - 通过@Intercepts和@Signature注解声明拦截的对象和方法 + - intercept()方法包含具体的拦截逻辑 + + **2. 动态代理机制**： + - MyBatis在初始化时通过Plugin.wrap()方法为目标对象创建代理 + - 使用JDK动态代理技术，生成代理对象 + - 代理对象在方法调用时会执行拦截器逻辑 + + **3. 责任链模式**： + - 多个插件按照注册顺序形成拦截器链 + - 每个拦截器都可以选择是否继续执行下一个拦截器 + - 通过Invocation.proceed()方法传递调用链 + + **可拦截的四大核心对象**： + + **1. Executor（执行器）**： + - **作用**：负责SQL的执行，是MyBatis的核心组件 + - **可拦截方法**： + - update(MappedStatement, Object)：拦截INSERT、UPDATE、DELETE操作 + - query(MappedStatement, Object, RowBounds, ResultHandler)：拦截SELECT操作 + - flushStatements()：拦截批处理语句的执行 + - commit(boolean)、rollback(boolean)：拦截事务提交和回滚 + - **应用场景**：分页插件、SQL性能监控、分表路由 + + **2. StatementHandler（语句处理器）**： + - **作用**：负责处理JDBC Statement，设置参数、执行SQL + - **可拦截方法**： + - prepare(Connection, Integer)：拦截SQL预编译 + - parameterize(Statement)：拦截参数设置 + - batch(Statement)：拦截批处理添加 + - update(Statement)、query(Statement, ResultHandler)：拦截SQL执行 + - **应用场景**：SQL重写、慢SQL监控、SQL安全检查 + + **3. ParameterHandler（参数处理器）**： + - **作用**：负责将Java对象转换为SQL参数 + - **可拦截方法**： + - getParameterObject()：获取参数对象 + - setParameters(PreparedStatement)：设置SQL参数 + - **应用场景**：参数加解密、参数值转换、敏感信息脱敏 + + **4. ResultSetHandler（结果处理器）**： + - **作用**：负责将ResultSet转换为Java对象 + - **可拦截方法**： + - handleResultSets(Statement)：处理结果集 + - handleCursorResultSets(Statement)：处理游标结果集 + - handleOutputParameters(CallableStatement)：处理存储过程输出参数 + - **应用场景**：结果集过滤、数据脱敏、字段值转换 + +**💻 代码示例**： + +```java +// 自定义分页插件 +@Intercepts({ + @Signature(type = Executor.class, method = "query", + args = {MappedStatement.class, Object.class, RowBounds.class, ResultHandler.class}), + @Signature(type = Executor.class, method = "query", + args = {MappedStatement.class, Object.class, RowBounds.class, ResultHandler.class, CacheKey.class, BoundSql.class}) +}) +public class PaginationPlugin implements Interceptor { + + private static final String COUNT_SUFFIX = "_COUNT"; + private static final String PAGE_PARAM = "page"; + + @Override + public Object intercept(Invocation invocation) throws Throwable { + Object[] args = invocation.getArgs(); + MappedStatement ms = (MappedStatement) args[0]; + Object parameter = args[1]; + RowBounds rowBounds = (RowBounds) args[2]; + ResultHandler resultHandler = (ResultHandler) args[3]; + Executor executor = (Executor) invocation.getTarget(); + + // 检查是否需要分页 + if (rowBounds == null || rowBounds == RowBounds.DEFAULT) { + // 检查参数中是否包含分页信息 + PageParam pageParam = extractPageParam(parameter); + if (pageParam == null) { + return invocation.proceed(); + } + rowBounds = new RowBounds(pageParam.getOffset(), pageParam.getPageSize()); + } + + // 无需分页 + if (rowBounds == RowBounds.DEFAULT) { + return invocation.proceed(); + } + + BoundSql boundSql; + if (args.length == 4) { + boundSql = ms.getBoundSql(parameter); + } else { + boundSql = (BoundSql) args[5]; + } + + // 执行count查询 + long total = executeCountQuery(executor, ms, parameter, boundSql); + + // 执行分页查询 + String originalSql = boundSql.getSql(); + String pagingSql = buildPageSql(originalSql, rowBounds); + + // 创建新的BoundSql + BoundSql newBoundSql = new BoundSql(ms.getConfiguration(), pagingSql, + boundSql.getParameterMappings(), parameter); + + // 复制动态参数 + copyAdditionalParameters(boundSql, newBoundSql); + + // 创建新的MappedStatement + MappedStatement newMs = copyMappedStatement(ms, new BoundSqlSource(newBoundSql)); + args[0] = newMs; + + // 执行分页查询 + List result = (List) invocation.proceed(); + + // 包装分页结果 + return new PageResult<>(result, total, rowBounds.getOffset() / rowBounds.getLimit() + 1, + rowBounds.getLimit()); + } + + private PageParam extractPageParam(Object parameter) { + if (parameter instanceof Map) { + Map paramMap = (Map) parameter; + Object pageObj = paramMap.get(PAGE_PARAM); + if (pageObj instanceof PageParam) { + return (PageParam) pageObj; + } + } else if (parameter instanceof PageParam) { + return (PageParam) parameter; + } + return null; + } + + private long executeCountQuery(Executor executor, MappedStatement ms, + Object parameter, BoundSql boundSql) throws SQLException { + String countSql = buildCountSql(boundSql.getSql()); + BoundSql countBoundSql = new BoundSql(ms.getConfiguration(), countSql, + boundSql.getParameterMappings(), parameter); + + MappedStatement countMs = buildCountMappedStatement(ms, countBoundSql); + + List countResult = executor.query(countMs, parameter, RowBounds.DEFAULT, null); + return Long.parseLong(countResult.get(0).toString()); + } + + private String buildCountSql(String originalSql) { + return "SELECT COUNT(*) FROM (" + originalSql + ") tmp_count"; + } + + private String buildPageSql(String originalSql, RowBounds rowBounds) { + return originalSql + " LIMIT " + rowBounds.getOffset() + ", " + rowBounds.getLimit(); + } + + @Override + public Object plugin(Object target) { + if (target instanceof Executor) { + return Plugin.wrap(target, this); + } + return target; + } + + @Override + public void setProperties(Properties properties) { + // 插件配置属性 + } +} + +// SQL监控插件 +@Intercepts({ + @Signature(type = StatementHandler.class, method = "query", args = {Statement.class, ResultHandler.class}), + @Signature(type = StatementHandler.class, method = "update", args = {Statement.class}), + @Signature(type = StatementHandler.class, method = "batch", args = {Statement.class}) +}) +public class SqlMonitorPlugin implements Interceptor { + + private static final Logger logger = LoggerFactory.getLogger(SqlMonitorPlugin.class); + private int slowSqlThreshold = 1000; // 慢SQL阈值，毫秒 + + @Override + public Object intercept(Invocation invocation) throws Throwable { + StatementHandler statementHandler = (StatementHandler) invocation.getTarget(); + BoundSql boundSql = statementHandler.getBoundSql(); + String sql = boundSql.getSql(); + + long startTime = System.currentTimeMillis(); + String sqlId = getSqlId(statementHandler); + + try { + Object result = invocation.proceed(); + + long endTime = System.currentTimeMillis(); + long executeTime = endTime - startTime; + + // 记录SQL执行信息 + logSqlExecution(sqlId, sql, executeTime, true, null); + + // 慢SQL告警 + if (executeTime > slowSqlThreshold) { + logger.warn("慢SQL检测: [{}] 执行时间: {}ms, SQL: {}", sqlId, executeTime, sql); + } + + return result; + + } catch (Exception e) { + long endTime = System.currentTimeMillis(); + long executeTime = endTime - startTime; + + logSqlExecution(sqlId, sql, executeTime, false, e.getMessage()); + throw e; + } + } + + private String getSqlId(StatementHandler statementHandler) { + try { + MetaObject metaObject = SystemMetaObject.forObject(statementHandler); + MappedStatement mappedStatement = + (MappedStatement) metaObject.getValue("delegate.mappedStatement"); + return mappedStatement.getId(); + } catch (Exception e) { + return "Unknown"; + } + } + + private void logSqlExecution(String sqlId, String sql, long executeTime, + boolean success, String errorMsg) { + // 可以发送到监控系统或数据库 + logger.info("SQL执行记录: [{}] 耗时: {}ms 状态: {} SQL: {}", + sqlId, executeTime, success ? "成功" : "失败", sql); + + if (!success && errorMsg != null) { + logger.error("SQL执行异常: [{}] 错误: {}", sqlId, errorMsg); + } + + // 发送到监控系统 + sendToMonitorSystem(sqlId, executeTime, success); + } + + private void sendToMonitorSystem(String sqlId, long executeTime, boolean success) { + // 实现发送到监控系统的逻辑 + // 例如：发送到Prometheus、Grafana等监控系统 + } + + @Override + public Object plugin(Object target) { + return Plugin.wrap(target, this); + } + + @Override + public void setProperties(Properties properties) { + String threshold = properties.getProperty("slowSqlThreshold"); + if (threshold != null) { + this.slowSqlThreshold = Integer.parseInt(threshold); + } + } +} + +// 数据脱敏插件 +@Intercepts({ + @Signature(type = ResultSetHandler.class, method = "handleResultSets", args = {Statement.class}) +}) +public class DataMaskingPlugin implements Interceptor { + + private static final Map MASKING_STRATEGIES = new HashMap<>(); + + static { + MASKING_STRATEGIES.put("phone", new PhoneMaskingStrategy()); + MASKING_STRATEGIES.put("email", new EmailMaskingStrategy()); + MASKING_STRATEGIES.put("idCard", new IdCardMaskingStrategy()); + MASKING_STRATEGIES.put("bankCard", new BankCardMaskingStrategy()); + } + + @Override + public Object intercept(Invocation invocation) throws Throwable { + List result = (List) invocation.proceed(); + + if (result == null || result.isEmpty()) { + return result; + } + + // 对结果进行脱敏处理 + for (Object obj : result) { + maskSensitiveData(obj); + } + + return result; + } + + private void maskSensitiveData(Object obj) { + if (obj == null) { + return; + } + + Class clazz = obj.getClass(); + Field[] fields = clazz.getDeclaredFields(); + + for (Field field : fields) { + Sensitive sensitive = field.getAnnotation(Sensitive.class); + if (sensitive != null) { + try { + field.setAccessible(true); + Object value = field.get(obj); + + if (value instanceof String) { + String originalValue = (String) value; + String maskedValue = maskValue(originalValue, sensitive.type()); + field.set(obj, maskedValue); + } + } catch (Exception e) { + logger.warn("数据脱敏失败: {}", e.getMessage()); + } + } + } + } + + private String maskValue(String value, String type) { + MaskingStrategy strategy = MASKING_STRATEGIES.get(type); + if (strategy != null) { + return strategy.mask(value); + } + return value; + } + + @Override + public Object plugin(Object target) { + return Plugin.wrap(target, this); + } + + @Override + public void setProperties(Properties properties) { + // 插件配置 + } +} + +// 脱敏注解 +@Target(ElementType.FIELD) +@Retention(RetentionPolicy.RUNTIME) +public @interface Sensitive { + String type(); +} + +// 脱敏策略接口 +interface MaskingStrategy { + String mask(String original); +} + +// 手机号脱敏策略 +class PhoneMaskingStrategy implements MaskingStrategy { + @Override + public String mask(String phone) { + if (phone == null || phone.length() < 11) { + return phone; + } + return phone.substring(0, 3) + "****" + phone.substring(7); + } +} + +// 邮箱脱敏策略 +class EmailMaskingStrategy implements MaskingStrategy { + @Override + public String mask(String email) { + if (email == null || !email.contains("@")) { + return email; + } + int atIndex = email.indexOf("@"); + String username = email.substring(0, atIndex); + String domain = email.substring(atIndex); + + if (username.length() <= 3) { + return username + domain; + } + + return username.substring(0, 3) + "***" + domain; + } +} + +// 身份证脱敏策略 +class IdCardMaskingStrategy implements MaskingStrategy { + @Override + public String mask(String idCard) { + if (idCard == null || idCard.length() < 18) { + return idCard; + } + return idCard.substring(0, 6) + "********" + idCard.substring(14); + } +} + +// 银行卡脱敏策略 +class BankCardMaskingStrategy implements MaskingStrategy { + @Override + public String mask(String bankCard) { + if (bankCard == null || bankCard.length() < 16) { + return bankCard; + } + return bankCard.substring(0, 4) + " **** **** " + bankCard.substring(bankCard.length() - 4); + } +} + +// 使用示例 +public class User { + private Long id; + private String name; + + @Sensitive(type = "phone") + private String phone; + + @Sensitive(type = "email") + private String email; + + @Sensitive(type = "idCard") + private String idCard; + + // getters and setters +} +``` + +```xml + + + + + + + + + + + + + + + +``` + +--- + +## 🔁 五、事务与多数据源 + + **核心理念**：在企业级应用中，事务管理和多数据源是必不可少的功能。Spring与MyBatis的深度整合为事务控制和数据源管理提供了强大的支持。 + +### 🎯 Spring + MyBatis 事务是如何生效的？有哪些常见的坑？ + +Spring与MyBatis的事务整合基于Spring的声明式事务管理： + + **事务生效的基本条件**： + + **1. 代理机制生效**： + - 方法必须是public的（Spring AOP基于代理实现） + - 调用必须通过Spring代理对象进行 + - 需要正确配置@EnableTransactionManagement + - 确保有合适的PlatformTransactionManager实现 + + **2. 事务传播行为**： + - REQUIRED：如果当前有事务则加入，没有则创建新事务（默认） + - REQUIRES_NEW：总是创建新事务，挂起当前事务 + - NESTED：嵌套事务，基于SavePoint实现 + - SUPPORTS：如果有事务则加入，没有则以非事务方式执行 + + **3. MyBatis集成原理**： + - SqlSessionFactoryBean与Spring事务管理器集成 + - SqlSessionTemplate自动参与Spring事务 + - 同一事务内多次数据库操作使用同一个SqlSession + + **常见的坑与解决方案**： + + **1. 自调用失效**： + - 问题：同一类内部方法调用@Transactional不生效 + - 原因：绕过了Spring代理 + - 解决：使用AopContext.currentProxy()或拆分到不同类 + + **2. 异常类型问题**： + - 问题：检查异常不会自动回滚 + - 原因：@Transactional默认只对RuntimeException回滚 + - 解决：使用rollbackFor指定异常类型 + + **3. 批处理异常被吃掉**： + - 问题：批处理中单条失败不影响其他记录 + - 原因：批处理底层可能吞掉部分异常 + - 解决：检查批处理返回结果，手动处理异常 + + **4. 只读事务误用**： + - 问题：在只读事务中执行修改操作 + - 解决：合理使用readOnly属性，优化性能 + +**💻 代码示例**： + +```java +// 1. Spring事务配置 +@Configuration +@EnableTransactionManagement +public class TransactionConfig { + + @Bean + public PlatformTransactionManager transactionManager(DataSource dataSource) { + return new DataSourceTransactionManager(dataSource); + } + + @Bean + public SqlSessionFactory sqlSessionFactory(DataSource dataSource) throws Exception { + SqlSessionFactoryBean factoryBean = new SqlSessionFactoryBean(); + factoryBean.setDataSource(dataSource); + return factoryBean.getObject(); + } + + @Bean + public SqlSessionTemplate sqlSessionTemplate(SqlSessionFactory sqlSessionFactory) { + return new SqlSessionTemplate(sqlSessionFactory); + } +} + +// 2. 正确使用事务的Service +@Service +public class UserService { + + @Autowired + private UserMapper userMapper; + + @Autowired + private OrderMapper orderMapper; + + // 正确的事务使用 + @Transactional(rollbackFor = Exception.class) + public void createUserWithOrder(User user, Order order) { + // 1. 插入用户 + userMapper.insert(user); + + // 2. 插入订单 + order.setUserId(user.getId()); + orderMapper.insert(order); + + // 3. 业务校验，任何异常都会回滚 + if (order.getAmount() < 0) { + throw new BusinessException("订单金额不能为负数"); + } + } + + // 事务传播行为示例 + @Transactional + public void outerMethod() { + userMapper.insert(new User("outer")); + + try { + innerMethod(); // 调用内部事务方法 + } catch (Exception e) { + log.error("内部方法异常", e); + // 外部事务仍会回滚，因为内部异常会传播到外部 + } + } + + @Transactional(propagation = Propagation.REQUIRES_NEW) + public void innerMethod() { + userMapper.insert(new User("inner")); + throw new RuntimeException("内部异常"); + // REQUIRES_NEW创建独立事务，不影响外部事务 + } + + // 解决自调用问题 + @Transactional + public void methodA() { + userMapper.insert(new User("A")); + + // 错误方式：直接调用，事务不生效 + // this.methodB(); + + // 正确方式：通过代理调用 + ((UserService) AopContext.currentProxy()).methodB(); + } + + @Transactional(propagation = Propagation.REQUIRES_NEW) + public void methodB() { + userMapper.insert(new User("B")); + } + + // 批处理事务处理 + @Transactional(rollbackFor = Exception.class) + public void batchInsertUsers(List users) { + int batchSize = 1000; + for (int i = 0; i < users.size(); i += batchSize) { + int endIndex = Math.min(i + batchSize, users.size()); + List batch = users.subList(i, endIndex); + + int[] results = userMapper.batchInsert(batch); + + // 检查批处理结果 + for (int j = 0; j < results.length; j++) { + if (results[j] == 0) { + throw new BusinessException( + String.format("批处理第%d条记录插入失败", i + j + 1)); + } + } + } + } + + // 只读事务优化查询 + @Transactional(readOnly = true) + public List queryActiveUsers() { + return userMapper.selectActiveUsers(); + } +} + +// 3. 手动事务控制 +@Service +public class ManualTransactionService { + + @Autowired + private PlatformTransactionManager transactionManager; + + @Autowired + private UserMapper userMapper; + + public void processWithManualTransaction() { + TransactionDefinition def = new DefaultTransactionDefinition(); + TransactionStatus status = transactionManager.getTransaction(def); + + try { + userMapper.insert(new User("manual")); + + // 一些复杂的业务逻辑 + if (someCondition()) { + transactionManager.commit(status); + } else { + transactionManager.rollback(status); + } + } catch (Exception e) { + transactionManager.rollback(status); + throw e; + } + } + + // 编程式事务模板 + @Autowired + private TransactionTemplate transactionTemplate; + + public void processWithTransactionTemplate() { + transactionTemplate.execute(status -> { + userMapper.insert(new User("template")); + + if (someCondition()) { + status.setRollbackOnly(); + } + + return null; + }); + } +} +``` + +### 🎯 多数据源和读写分离如何实现？ + +多数据源实现主要基于Spring的AbstractRoutingDataSource： + + **实现方案**： + + **1. AbstractRoutingDataSource路由**： + - 继承AbstractRoutingDataSource实现数据源路由 + - 通过determineCurrentLookupKey()方法决定使用哪个数据源 + - 结合ThreadLocal存储当前数据源标识 + + **2. 注解 + AOP切面**： + - 自定义@DataSource注解标记数据源 + - AOP切面拦截方法调用，设置数据源上下文 + - 支持方法级和类级的数据源切换 + + **3. 读写分离实现**： + - 主数据源处理写操作（INSERT、UPDATE、DELETE） + - 从数据源处理读操作（SELECT） + - 考虑主从延迟，关键业务读主库 + + **4. 分布式事务处理**： + - 轻量级：最终一致性方案（消息队列、事件驱动） + - 强一致性：两阶段提交（Seata、XA事务） + - 根据业务特性选择合适的方案 + +**💻 代码示例**： + +```java +// 1. 动态数据源实现 +@Component +public class DynamicDataSource extends AbstractRoutingDataSource { + + @Override + protected Object determineCurrentLookupKey() { + return DataSourceContextHolder.getDataSourceType(); + } +} + +// 2. 数据源上下文管理 +public class DataSourceContextHolder { + + public enum DataSourceType { + MASTER, SLAVE + } + + private static final ThreadLocal contextHolder = new ThreadLocal<>(); + + public static void setDataSourceType(DataSourceType dataSourceType) { + contextHolder.set(dataSourceType); + } + + public static DataSourceType getDataSourceType() { + return contextHolder.get(); + } + + public static void clearDataSourceType() { + contextHolder.remove(); + } +} + +// 3. 数据源注解 +@Target({ElementType.METHOD, ElementType.TYPE}) +@Retention(RetentionPolicy.RUNTIME) +public @interface DataSource { + DataSourceContextHolder.DataSourceType value() default DataSourceContextHolder.DataSourceType.MASTER; +} + +// 4. 数据源切换切面 +@Aspect +@Component +public class DataSourceAspect { + + @Around("@annotation(dataSource)") + public Object around(ProceedingJoinPoint point, DataSource dataSource) throws Throwable { + DataSourceContextHolder.DataSourceType currentType = DataSourceContextHolder.getDataSourceType(); + + try { + DataSourceContextHolder.setDataSourceType(dataSource.value()); + return point.proceed(); + } finally { + if (currentType == null) { + DataSourceContextHolder.clearDataSourceType(); + } else { + DataSourceContextHolder.setDataSourceType(currentType); + } + } + } + + @Around("execution(* com.example.service.*.*(..))") + public Object aroundService(ProceedingJoinPoint point) throws Throwable { + String methodName = point.getSignature().getName(); + + // 根据方法名自动判断读写操作 + if (methodName.startsWith("select") || methodName.startsWith("find") + || methodName.startsWith("get") || methodName.startsWith("query")) { + DataSourceContextHolder.setDataSourceType(DataSourceContextHolder.DataSourceType.SLAVE); + } else { + DataSourceContextHolder.setDataSourceType(DataSourceContextHolder.DataSourceType.MASTER); + } + + try { + return point.proceed(); + } finally { + DataSourceContextHolder.clearDataSourceType(); + } + } +} + +// 5. 多数据源配置 +@Configuration +public class MultiDataSourceConfig { + + @Bean + @ConfigurationProperties("app.datasource.master") + public DataSource masterDataSource() { + return DruidDataSourceBuilder.create().build(); + } + + @Bean + @ConfigurationProperties("app.datasource.slave") + public DataSource slaveDataSource() { + return DruidDataSourceBuilder.create().build(); + } + + @Bean + @Primary + public DataSource dynamicDataSource() { + DynamicDataSource dynamicDataSource = new DynamicDataSource(); + + Map dataSourceMap = new HashMap<>(); + dataSourceMap.put(DataSourceContextHolder.DataSourceType.MASTER, masterDataSource()); + dataSourceMap.put(DataSourceContextHolder.DataSourceType.SLAVE, slaveDataSource()); + + dynamicDataSource.setTargetDataSources(dataSourceMap); + dynamicDataSource.setDefaultTargetDataSource(masterDataSource()); + + return dynamicDataSource; + } + + @Bean + public SqlSessionFactory sqlSessionFactory() throws Exception { + SqlSessionFactoryBean factoryBean = new SqlSessionFactoryBean(); + factoryBean.setDataSource(dynamicDataSource()); + return factoryBean.getObject(); + } +} + +// 6. Service层使用示例 +@Service +public class UserService { + + @Autowired + private UserMapper userMapper; + + // 读操作，使用从库 + @DataSource(DataSourceContextHolder.DataSourceType.SLAVE) + @Transactional(readOnly = true) + public List findActiveUsers() { + return userMapper.selectActiveUsers(); + } + + // 写操作，使用主库 + @DataSource(DataSourceContextHolder.DataSourceType.MASTER) + @Transactional + public void createUser(User user) { + userMapper.insert(user); + } + + // 重要操作，强制读主库 + @DataSource(DataSourceContextHolder.DataSourceType.MASTER) + @Transactional + public void transferBalance(Long fromUserId, Long toUserId, BigDecimal amount) { + // 读取余额必须从主库读取，确保数据一致性 + User fromUser = userMapper.selectById(fromUserId); + User toUser = userMapper.selectById(toUserId); + + if (fromUser.getBalance().compareTo(amount) < 0) { + throw new BusinessException("余额不足"); + } + + // 更新余额 + fromUser.setBalance(fromUser.getBalance().subtract(amount)); + toUser.setBalance(toUser.getBalance().add(amount)); + + userMapper.updateById(fromUser); + userMapper.updateById(toUser); + } +} + +// 7. 分布式事务示例（Seata） +@Service +public class DistributedTransactionService { + + @Autowired + private OrderService orderService; + + @Autowired + private InventoryService inventoryService; + + @Autowired + private AccountService accountService; + + @GlobalTransactional(rollbackFor = Exception.class) + public void createOrder(CreateOrderRequest request) { + // 1. 创建订单 + orderService.createOrder(request.getOrderInfo()); + + // 2. 扣减库存（可能调用不同的数据库） + inventoryService.deductInventory(request.getProductId(), request.getQuantity()); + + // 3. 扣减账户余额（可能调用不同的服务） + accountService.deductBalance(request.getUserId(), request.getAmount()); + + // 任何一个步骤失败，都会回滚所有操作 + } +} +``` + +```yaml +# application.yml 多数据源配置 +app: + datasource: + master: + driver-class-name: com.mysql.cj.jdbc.Driver + url: jdbc:mysql://master-db:3306/app_db?useUnicode=true&characterEncoding=utf8&serverTimezone=Asia/Shanghai + username: app_user + password: app_password + # Druid连接池配置 + initial-size: 5 + min-idle: 5 + max-active: 20 + max-wait: 60000 + time-between-eviction-runs-millis: 60000 + min-evictable-idle-time-millis: 300000 + validation-query: SELECT 1 FROM DUAL + test-while-idle: true + test-on-borrow: false + test-on-return: false + + slave: + driver-class-name: com.mysql.cj.jdbc.Driver + url: jdbc:mysql://slave-db:3306/app_db?useUnicode=true&characterEncoding=utf8&serverTimezone=Asia/Shanghai + username: app_user + password: app_password + initial-size: 5 + min-idle: 5 + max-active: 20 + max-wait: 60000 + time-between-eviction-runs-millis: 60000 + min-evictable-idle-time-millis: 300000 + validation-query: SELECT 1 FROM DUAL + test-while-idle: true + test-on-borrow: false + test-on-return: false +``` + +--- + +## 🚀 六、性能与调优 + + **核心理念**：MyBatis性能优化涉及多个层面，从SQL执行、连接池管理到批处理优化，需要系统性的调优策略来提升应用性能。 + +### 🎯 MyBatis有哪些性能调优手段？ + +MyBatis性能调优需要从多个维度进行系统性优化： + + **1. 分页查询优化**： + - **物理分页vs逻辑分页**：优先使用物理分页（数据库级LIMIT），避免逻辑分页（内存分页） + - **深分页问题**：使用游标分页或覆盖索引优化，避免OFFSET大偏移量性能问题 + - **PageHelper插件**：合理配置PageHelper，注意线程安全和参数清理 + + **2. SQL执行优化**： + - **预编译Statement**：使用#{}参数占位符，提高SQL执行效率和安全性 + - **批量操作优化**：使用BatchExecutor批量提交，合理设置batch.size大小 + - **避免N+1问题**：使用关联查询、延迟加载或批量查询解决 + + **3. 缓存策略优化**： + - 合理使用一级缓存（SqlSession级别）和二级缓存（namespace级别） + - 避免缓存穿透和缓存雪崩问题 + - 设置合适的缓存失效策略和TTL + + **4. 连接池调优**： + - 选择高性能连接池如HikariCP + - 合理设置连接池大小和超时参数 + - 监控连接池状态，避免连接泄漏 + + **5. Executor类型选择**： + - SIMPLE：每次创建新Statement，适合单次操作 + - REUSE：复用PreparedStatement，适合重复查询 + - BATCH：批量执行，适合大批量操作 + +**💻 代码示例**： + +```java +// 1. 分页查询优化示例 +@Service +public class PaginationOptimizationService { + + @Autowired + private UserMapper userMapper; + + // 物理分页 - 推荐方式 + public PageResult getUsersByPage(PageParam pageParam) { + // 使用PageHelper插件实现物理分页 + PageHelper.startPage(pageParam.getPageNum(), pageParam.getPageSize()); + List users = userMapper.selectUsers(); + + PageInfo pageInfo = new PageInfo<>(users); + return PageResult.of(users, pageInfo.getTotal(), pageParam.getPageNum(), pageParam.getPageSize()); + } + + // 深分页优化 - 使用覆盖索引和游标分页 + public List getUsersByDeepPage(Long lastId, int limit) { + return userMapper.selectUsersByIdRange(lastId, limit); + } +} + +// 2. 批处理操作优化 +@Service +public class BatchOperationService { + + @Autowired + private UserMapper userMapper; + + @Autowired + private SqlSessionFactory sqlSessionFactory; + + // 方案1：MyBatis批量插入（适用于中等数据量） + @Transactional + public void batchInsertUsers(List users) { + if (users.size() <= 1000) { + userMapper.batchInsert(users); + } else { + // 分批处理大量数据 + int batchSize = 1000; + for (int i = 0; i < users.size(); i += batchSize) { + int endIndex = Math.min(i + batchSize, users.size()); + List batch = users.subList(i, endIndex); + userMapper.batchInsert(batch); + } + } + } + + // 方案2：JDBC批处理（适用于大量数据） + @Transactional + public void jdbcBatchInsert(List users) { + try (SqlSession session = sqlSessionFactory.openSession(ExecutorType.BATCH, false)) { + UserMapper mapper = session.getMapper(UserMapper.class); + + int batchSize = 1000; + for (int i = 0; i < users.size(); i++) { + mapper.insert(users.get(i)); + + // 分批提交，避免内存溢出 + if ((i + 1) % batchSize == 0 || i == users.size() - 1) { + session.flushStatements(); + session.clearCache(); // 清理一级缓存 + } + } + session.commit(); + } + } +} + +// 3. 连接池优化配置 +@Configuration +public class DataSourceConfig { + + @Bean + @Primary + public DataSource hikariDataSource() { + HikariConfig config = new HikariConfig(); + + // 基本配置 + config.setDriverClassName("com.mysql.cj.jdbc.Driver"); + config.setJdbcUrl("jdbc:mysql://localhost:3306/test?useSSL=false&serverTimezone=UTC"); + config.setUsername("root"); + config.setPassword("password"); + + // 连接池大小配置 + config.setMaximumPoolSize(20); // 最大连接数 + config.setMinimumIdle(5); // 最小空闲连接数 + + // 超时配置 + config.setConnectionTimeout(30000); // 连接超时时间30秒 + config.setIdleTimeout(600000); // 空闲超时时间10分钟 + config.setMaxLifetime(1800000); // 连接最大生命周期30分钟 + config.setLeakDetectionThreshold(60000); // 连接泄漏检测 + + // MySQL性能优化参数 + config.addDataSourceProperty("cachePrepStmts", "true"); + config.addDataSourceProperty("prepStmtCacheSize", "250"); + config.addDataSourceProperty("prepStmtCacheSqlLimit", "2048"); + config.addDataSourceProperty("useServerPrepStmts", "true"); + config.addDataSourceProperty("useLocalSessionState", "true"); + config.addDataSourceProperty("rewriteBatchedStatements", "true"); + + return new HikariDataSource(config); + } +} + +// 4. N+1问题解决方案 +@Service +public class N1ProblemSolutionService { + + @Autowired + private UserMapper userMapper; + + @Autowired + private OrderMapper orderMapper; + + // 问题方法：会产生N+1查询 + public List getUsersWithOrdersBad() { + // 1次查询：获取所有用户 + List users = userMapper.selectAllUsers(); + + // N次查询：为每个用户查询订单（N+1问题） + for (User user : users) { + List orders = orderMapper.selectByUserId(user.getId()); + user.setOrders(orders); + } + + return users; + } + + // 解决方案1：使用JOIN查询 + public List getUsersWithOrdersJoin() { + // 1次查询完成所有数据获取 + return userMapper.selectUsersWithOrdersByJoin(); + } + + // 解决方案2：分步查询 + 批量IN + public List getUsersWithOrdersBatch() { + // 1次查询：获取所有用户 + List users = userMapper.selectAllUsers(); + + if (!users.isEmpty()) { + // 提取所有用户ID + List userIds = users.stream() + .map(User::getId) + .collect(Collectors.toList()); + + // 1次查询：批量获取所有订单 + List orders = orderMapper.selectByUserIds(userIds); + + // 内存中组装数据 + Map> orderMap = orders.stream() + .collect(Collectors.groupingBy(Order::getUserId)); + + users.forEach(user -> + user.setOrders(orderMap.getOrDefault(user.getId(), new ArrayList<>())) + ); + } + + return users; + } +} + +// 5. SQL性能监控拦截器 +@Intercepts({ + @Signature(type = StatementHandler.class, method = "query", args = {Statement.class, ResultHandler.class}), + @Signature(type = StatementHandler.class, method = "update", args = {Statement.class}) +}) +@Component +public class SqlPerformanceInterceptor implements Interceptor { + + private static final Logger logger = LoggerFactory.getLogger(SqlPerformanceInterceptor.class); + private final long slowSqlThreshold = 1000; // 慢SQL阈值 + + @Override + public Object intercept(Invocation invocation) throws Throwable { + StatementHandler statementHandler = (StatementHandler) invocation.getTarget(); + BoundSql boundSql = statementHandler.getBoundSql(); + String sql = boundSql.getSql(); + + long startTime = System.currentTimeMillis(); + String mappedStatementId = getMappedStatementId(statementHandler); + + try { + Object result = invocation.proceed(); + + long executeTime = System.currentTimeMillis() - startTime; + + // 记录SQL执行信息 + if (executeTime > slowSqlThreshold) { + logger.warn("慢SQL告警 - 执行时间: {}ms, StatementId: {}, SQL: {}", + executeTime, mappedStatementId, sql); + + // 可以在这里添加告警逻辑，如发送钉钉消息 + sendSlowSqlAlert(mappedStatementId, executeTime, sql); + } else { + logger.info("SQL执行 - 执行时间: {}ms, StatementId: {}", + executeTime, mappedStatementId); + } + + return result; + + } catch (Exception e) { + long executeTime = System.currentTimeMillis() - startTime; + logger.error("SQL执行异常 - 执行时间: {}ms, StatementId: {}, SQL: {}, 异常: {}", + executeTime, mappedStatementId, sql, e.getMessage()); + throw e; + } + } + + private String getMappedStatementId(StatementHandler handler) { + try { + MetaObject metaObject = SystemMetaObject.forObject(handler); + MappedStatement mappedStatement = + (MappedStatement) metaObject.getValue("delegate.mappedStatement"); + return mappedStatement.getId(); + } catch (Exception e) { + return "Unknown"; + } + } + + private void sendSlowSqlAlert(String statementId, long executeTime, String sql) { + // 实现告警逻辑，如发送邮件、钉钉消息等 + // 这里仅作示例 + logger.warn("慢SQL告警：{} 执行时间 {}ms", statementId, executeTime); + } + + @Override + public Object plugin(Object target) { + return Plugin.wrap(target, this); + } + + @Override + public void setProperties(Properties properties) { + // 可以从配置文件读取慢SQL阈值等参数 + } +} + +// 6. MyBatis配置优化 +@Configuration +public class MyBatisOptimizationConfig { + + @Bean + public SqlSessionFactoryBean sqlSessionFactory(DataSource dataSource) throws Exception { + SqlSessionFactoryBean factoryBean = new SqlSessionFactoryBean(); + factoryBean.setDataSource(dataSource); + + // MyBatis配置优化 + org.apache.ibatis.session.Configuration configuration = + new org.apache.ibatis.session.Configuration(); + + // 缓存配置 + configuration.setCacheEnabled(true); // 开启二级缓存 + configuration.setLocalCacheScope(LocalCacheScope.SESSION); // 一级缓存范围 + + // 延迟加载配置 + configuration.setLazyLoadingEnabled(true); // 开启延迟加载 + configuration.setAggressiveLazyLoading(false); // 关闭积极加载 + + // 执行器类型配置 + configuration.setDefaultExecutorType(ExecutorType.REUSE); // 复用PreparedStatement + + // 结果集处理优化 + configuration.setMapUnderscoreToCamelCase(true); // 自动驼峰转换 + configuration.setCallSettersOnNulls(true); // 空值也调用setter + + factoryBean.setConfiguration(configuration); + + // 添加性能监控插件 + factoryBean.setPlugins(new SqlPerformanceInterceptor()); + + return factoryBean; + } + + // 分页插件配置 + @Bean + public PageInterceptor pageInterceptor() { + PageInterceptor pageInterceptor = new PageInterceptor(); + Properties props = new Properties(); + props.setProperty("helperDialect", "mysql"); + props.setProperty("reasonable", "true"); // 分页合理化 + props.setProperty("supportMethodsArguments", "true"); + props.setProperty("params", "count=countSql"); + pageInterceptor.setProperties(props); + return pageInterceptor; + } +} +``` + +### 🎯 如何定位和解决N+1查询问题？ + +N+1查询是MyBatis中常见的性能问题，需要系统性的识别和解决： + + **问题识别**： + - 日志中出现大量相似的子查询 + - APM监控显示同一接口执行了多次相同SQL + - 查询时间随数据量线性增长 + - 数据库连接数异常增长 + + **根本原因**： + - 关联查询设计不当，使用了嵌套查询而非连接查询 + - 延迟加载触发了意外的额外查询 + - 缺少合适的批量查询策略 + + **解决策略**： + 1. **关联查询优化**：使用JOIN替代嵌套查询 + 2. **resultMap嵌套**：一次查询获取所有数据 + 3. **批量IN查询**：先查主表，再批量查询关联表 + 4. **延迟加载控制**：合理配置懒加载策略 + 5. **缓存机制**：使用适当的缓存减少重复查询 + + **治理方案**：定期审查SQL日志，建立慢查询监控，制定关联查询规范。 + +**💻 代码示例**： + +```xml + + + + + + + + + + + + + + INSERT INTO user (name, email, phone, status) VALUES + + (#{user.name}, #{user.email}, #{user.phone}, #{user.status}) + + + + + + INSERT INTO user (name, email, phone, status) + VALUES (#{name}, #{email}, #{phone}, #{status}) + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +``` + +### 🎯 MyBatis批处理和主键回填的最佳实践？ + +MyBatis批处理和主键回填需要根据场景选择合适的策略： + + **单条记录主键回填**： + - 使用useGeneratedKeys=true + keyProperty配置 + - 适用于单条INSERT操作 + - 支持自增主键自动回填到对象 + + **批量插入策略**： + 1. **MyBatis批量插入**：使用foreach生成多值INSERT + 2. **JDBC批处理**：ExecutorType.BATCH + 分批flush + 3. **数据库原生批量**：load data infile等 + + **批量主键回填**： + - MySQL：支持批量主键回填，依赖JDBC驱动版本 + - PostgreSQL：使用RETURNING子句 + - Oracle：使用序列或RETURNING INTO + - 不同数据库厂商支持程度不同，需要测试验证 + + **性能优化建议**： + - 大批量操作优先使用JDBC批处理 + - 合理设置批次大小（建议1000-5000） + - 关闭自动提交，手动控制事务 + - 监控内存使用，避免OOM + + **最佳实践**：根据数据量选择策略，小批量用MyBatis批量插入，大批量用JDBC批处理。 + +**💻 代码示例**： + +```java +// 批处理和主键回填最佳实践 +@Service +public class BatchInsertService { + + @Autowired + private UserMapper userMapper; + + @Autowired + private SqlSessionFactory sqlSessionFactory; + + // 单条插入主键回填 + @Transactional + public void insertSingleUser(User user) { + userMapper.insertUser(user); + System.out.println("插入用户ID: " + user.getId()); // 主键已自动回填 + } + + // 小批量插入（1000以内） + @Transactional + public void smallBatchInsert(List users) { + if (users.size() <= 1000) { + userMapper.batchInsert(users); + } else { + // 分批处理 + int batchSize = 1000; + for (int i = 0; i < users.size(); i += batchSize) { + int endIndex = Math.min(i + batchSize, users.size()); + List batch = users.subList(i, endIndex); + userMapper.batchInsert(batch); + } + } + } + + // 大批量JDBC批处理（推荐用于大量数据） + @Transactional + public int[] largeBatchInsert(List users) { + try (SqlSession session = sqlSessionFactory.openSession(ExecutorType.BATCH)) { + UserMapper batchMapper = session.getMapper(UserMapper.class); + + int batchSize = 1000; + int[] results = new int[users.size()]; + int resultIndex = 0; + + for (int i = 0; i < users.size(); i++) { + batchMapper.insertUser(users.get(i)); + + // 分批提交，避免内存溢出 + if ((i + 1) % batchSize == 0 || i == users.size() - 1) { + List batchResults = session.flushStatements(); + + // 处理批处理结果 + for (BatchResult batchResult : batchResults) { + int[] updateCounts = batchResult.getUpdateCounts(); + System.arraycopy(updateCounts, 0, results, resultIndex, updateCounts.length); + resultIndex += updateCounts.length; + } + } + } + + session.commit(); + return results; + } + } + + // 批量插入with主键回填（MySQL示例） + @Transactional + public void batchInsertWithKeyReturn(List users) { + // 使用MyBatis的batch insert with key return + userMapper.batchInsertWithKeys(users); + + // 验证主键回填 + for (User user : users) { + System.out.println("用户 " + user.getName() + " ID: " + user.getId()); + } + } + + // 高性能批量插入（适用于超大批量） + public void highPerformanceBatchInsert(List users) { + int batchSize = 5000; + + try (SqlSession session = sqlSessionFactory.openSession(ExecutorType.BATCH, false)) { + UserMapper mapper = session.getMapper(UserMapper.class); + + for (int i = 0; i < users.size(); i++) { + mapper.insertUser(users.get(i)); + + if (i % batchSize == 0) { + session.flushStatements(); + session.clearCache(); // 清理一级缓存，释放内存 + } + } + + session.flushStatements(); + session.commit(); + } catch (Exception e) { + throw new RuntimeException("批量插入失败", e); + } + } + + // 不同数据库的主键回填策略 + public void databaseSpecificKeyReturn() { + // MySQL: 支持批量主键回填 + // useGeneratedKeys="true" keyProperty="id + + // PostgreSQL: 使用RETURNING + // INSERT INTO user (name) VALUES (#{name}) RETURNING id + + // Oracle: 使用序列 + // + // SELECT user_seq.NEXTVAL FROM dual + // + + // SQL Server: 使用IDENTITY + // useGeneratedKeys="true" keyProperty="id + } +} + +// 批处理性能监控 +@Component +public class BatchPerformanceMonitor { + + public void monitorBatchPerformance() { + System.out.println("=== 批处理性能监控要点 ==="); + + System.out.println("1. 监控指标："); + System.out.println(" - 批处理执行时间"); + System.out.println(" - 内存使用情况"); + System.out.println(" - 数据库连接数"); + System.out.println(" - 事务提交频率"); + + System.out.println("2. 优化策略："); + System.out.println(" - 合理设置批次大小"); + System.out.println(" - 及时清理一级缓存"); + System.out.println(" - 使用合适的执行器类型"); + System.out.println(" - 监控和调整JVM内存参数"); + + System.out.println("3. 常见问题："); + System.out.println(" - OOM：批次过大或缓存未清理"); + System.out.println(" - 死锁：并发批处理冲突"); + System.out.println(" - 超时：单批次数据量过大"); + System.out.println(" - 主键冲突：重复数据检查不充分"); + } +} + + +``` + +### 🎯 如何设计MyBatis的错误处理和异常机制？ + +MyBatis异常处理需要建立分层的异常处理机制： + + **异常分层设计**： + 1. **DAO层**：捕获MyBatis和数据库异常，转换为业务异常 + 2. **Service层**：处理业务逻辑异常，记录操作日志 + 3. **Controller层**：统一异常处理，返回友好错误信息 + + **异常分类处理**： + - **数据库连接异常**：记录详细日志，返回系统繁忙提示 + - **SQL语法异常**：记录SQL和参数，避免泄露敏感信息 + - **约束违反异常**：解析约束类型，返回具体业务提示 + - **超时异常**：记录执行时间，提供重试建议 + + **日志记录规范**： + - 记录traceId便于链路追踪 + - 记录SQL语句和参数值（脱敏处理） + - 记录执行时间和影响行数 + - 敏感操作记录审计日志 + + **监控告警机制**： + - 慢SQL自动告警 + - 异常频率监控 + - 数据库连接池监控 + - 关键业务操作监控 + + 目标是让异常信息对开发者有用，对用户友好，对系统安全。 + +**💻 代码示例**： + +```java +// 异常处理和监控示例 +@Component +@Slf4j +public class MyBatisErrorHandler { + + // 统一异常转换 + public static RuntimeException convertException(Exception e, String operation) { + String traceId = MDC.get("traceId"); + + if (e instanceof DataIntegrityViolationException) { + return handleDataIntegrityViolation((DataIntegrityViolationException) e, operation); + } else if (e instanceof QueryTimeoutException) { + return handleQueryTimeout((QueryTimeoutException) e, operation); + } else if (e instanceof BadSqlGrammarException) { + return handleBadSqlGrammar((BadSqlGrammarException) e, operation); + } else if (e instanceof DataAccessResourceFailureException) { + return handleResourceFailure((DataAccessResourceFailureException) e, operation); + } else { + log.error("未知数据访问异常, traceId: {}, operation: {}", traceId, operation, e); + return new ServiceException("数据操作失败"); + } + } + + private static RuntimeException handleDataIntegrityViolation( + DataIntegrityViolationException e, String operation) { + String message = e.getMessage(); + String traceId = MDC.get("traceId"); + + log.warn("数据完整性约束违反, traceId: {}, operation: {}, message: {}", + traceId, operation, message); + + if (message.contains("Duplicate entry")) { + String field = extractDuplicateField(message); + return new BusinessException("数据重复，字段 " + field + " 已存在"); + } else if (message.contains("foreign key constraint")) { + return new BusinessException("关联数据不存在或已被删除"); + } else if (message.contains("cannot be null")) { + String field = extractNullField(message); + return new BusinessException("必填字段 " + field + " 不能为空"); + } + + return new BusinessException("数据约束违反，请检查输入数据"); + } + + private static RuntimeException handleQueryTimeout( + QueryTimeoutException e, String operation) { + String traceId = MDC.get("traceId"); + + log.error("查询超时, traceId: {}, operation: {}", traceId, operation, e); + + // 触发告警 + AlertManager.sendAlert("SQL_TIMEOUT", + String.format("查询超时: %s, traceId: %s", operation, traceId)); + + return new ServiceException("查询超时，请稍后重试"); + } + + // SQL注入检测和防护 + @Component + public static class SqlInjectionProtector { + + private static final List SQL_INJECTION_PATTERNS = Arrays.asList( + "union", "select", "insert", "update", "delete", "drop", "create", "alter", + "exec", "execute", "--", "/*", "*/", "xp_", "sp_", "0x + ); + + public static void checkSqlInjection(String input) { + if (StringUtils.isEmpty(input)) { + return; + } + + String lowerInput = input.toLowerCase(); + for (String pattern : SQL_INJECTION_PATTERNS) { + if (lowerInput.contains(pattern)) { + String traceId = MDC.get("traceId"); + log.error("检测到SQL注入攻击, traceId: {}, input: {}", traceId, input); + + // 记录安全日志 + SecurityLog securityLog = SecurityLog.builder() + .traceId(traceId) + .attackType("SQL_INJECTION") + .attackContent(input) + .clientIp(getClientIp()) + .build(); + + SecurityLogService.record(securityLog); + + throw new SecurityException("检测到非法输入"); + } + } + } + } +} + +// 审计日志和追踪 +@Component +@Slf4j +public class MyBatisAuditLogger { + + @EventListener + public void handleDataChange(DataChangeEvent event) { + String traceId = MDC.get("traceId"); + + AuditLog auditLog = AuditLog.builder() + .traceId(traceId) + .operation(event.getOperation()) + .tableName(event.getTableName()) + .entityId(event.getEntityId()) + .oldValue(event.getOldValue()) + .newValue(event.getNewValue()) + .operatorId(event.getOperatorId()) + .operateTime(new Date()) + .clientIp(event.getClientIp()) + .userAgent(event.getUserAgent()) + .build(); + + // 异步记录审计日志 + auditLogService.recordAsync(auditLog); + + log.info("数据变更审计, traceId: {}, operation: {}, table: {}, entityId: {}", + traceId, event.getOperation(), event.getTableName(), event.getEntityId()); + } +} + +// 性能监控和告警 +@Component +public class MyBatisPerformanceMonitor { + + private final MeterRegistry meterRegistry; + private final Counter sqlExecuteCounter; + private final Timer sqlExecuteTimer; + + public MyBatisPerformanceMonitor(MeterRegistry meterRegistry) { + this.meterRegistry = meterRegistry; + this.sqlExecuteCounter = Counter.builder("mybatis.sql.execute.count") + .description("SQL执行次数") + .register(meterRegistry); + this.sqlExecuteTimer = Timer.builder("mybatis.sql.execute.time") + .description("SQL执行时间") + .register(meterRegistry); + } + + public void recordSqlExecution(String sqlId, long executionTime, boolean success) { + // 记录执行次数 + sqlExecuteCounter.increment( + Tags.of("sql_id", sqlId, "success", String.valueOf(success))); + + // 记录执行时间 + sqlExecuteTimer.record(executionTime, TimeUnit.MILLISECONDS); + + // 慢SQL告警 + if (executionTime > 2000) { + AlertManager.sendSlowSqlAlert(sqlId, executionTime); + } + + // 失败告警 + if (!success) { + AlertManager.sendSqlErrorAlert(sqlId); + } + } +} +``` + + + +--- + +## 🧭 七、工程化与规范 + + **核心理念**：建立MyBatis开发的工程化标准和最佳实践，确保代码质量、可维护性和团队协作效率。 + +### 🎯 MyBatis开发规范有哪些要点？ + +MyBatis开发规范涉及命名规范、代码规范、SQL规范和安全规范等多个层面： + + **1. 命名规范**： + - Mapper接口：表名 + Mapper，如UserMapper、OrderMapper + - XML文件：与Mapper接口同名，放在同包路径下 + - SQL ID：动词 + 实体 + 条件，如selectUserById、updateUserByCondition + - 参数名称：使用@Param注解明确参数名 + + **2. 代码组织规范**： + - Mapper接口与XML文件一一对应 + - 复杂SQL抽取到XML中，简单查询可用注解 + - SQL片段复用，避免重复代码 + - 合理使用命名空间隔离不同业务模块 + + **3. SQL编写规范**： + - 禁止select *，明确指定需要的字段 + - WHERE条件必须有兜底条件，避免全表操作 + - 分页查询必须有排序字段，保证结果稳定性 + - 避免隐式类型转换，明确指定参数类型 + + **4. 安全规范**： + - 严格禁用${}拼接用户输入，必须使用#{}参数化 + - 动态表名、列名使用白名单校验 + - 敏感数据查询添加权限检查 + - SQL注入防护和参数校验 + + **5. 性能规范**： + - 合理使用索引，避免全表扫描 + - 限制查询结果集大小 + - 批量操作使用合适的批处理策略 + - 监控慢SQL并及时优化 + + **6. 变更管理规范**： + - 数据库变更必须向前兼容 + - 生产环境变更走正式发布流程 + - 重要变更需要灰度发布和回滚预案 + - 变更记录和审计追踪 + +**💻 代码示例**： + +```java +// MyBatis开发规范示例 +public class MyBatisBestPracticesDemo { + + // 1. 命名规范示例 + public interface UserMapper { + // ✅ 好的命名：动词+实体+条件 + User selectUserById(@Param("id") Long id); + List selectUsersByCondition(@Param("condition") UserQueryCondition condition); + int updateUserById(@Param("user") User user); + int deleteUserById(@Param("id") Long id); + + // ❌ 不好的命名 + // User get(Long id); // 动词不明确 + // List list(); // 没有明确查询条件 + } + + // 2. 参数对象规范 + public static class UserQueryCondition { + private String name; + private Integer minAge; + private Integer maxAge; + private String email; + private Integer status; + private String sortField = "id"; // 默认排序字段 + private String sortOrder = "ASC"; // 默认排序方向 + + // 参数校验 + public void validate() { + if (StringUtils.isNotBlank(sortField)) { + // 排序字段白名单校验 + List allowedFields = Arrays.asList("id", "name", "createTime", "updateTime"); + if (!allowedFields.contains(sortField)) { + throw new IllegalArgumentException("不允许的排序字段: " + sortField); + } + } + } + + // getters and setters... + } + + // 3. Service层规范示例 + @Service + @Slf4j + public class UserService { + + @Autowired + private UserMapper userMapper; + + // ✅ 规范的查询方法 + public PageResult queryUsers(UserQueryCondition condition, PageParam pageParam) { + // 参数校验 + condition.validate(); + pageParam.validate(); + + // 记录查询日志 + String traceId = MDC.get("traceId"); + log.info("查询用户列表开始, traceId: {}, condition: {}", traceId, condition); + + long startTime = System.currentTimeMillis(); + try { + // 查询总数 + int total = userMapper.countUsersByCondition(condition); + if (total == 0) { + return PageResult.empty(); + } + + // 查询数据 + List users = userMapper.selectUsersByConditionWithPage(condition, pageParam); + + // 性能监控 + long cost = System.currentTimeMillis() - startTime; + if (cost > 1000) { + log.warn("慢查询告警, traceId: {}, cost: {}ms, condition: {}", + traceId, cost, condition); + } + + return PageResult.of(users, total, pageParam); + + } catch (Exception e) { + log.error("查询用户列表失败, traceId: {}, condition: {}", traceId, condition, e); + throw new ServiceException("查询用户失败", e); + } + } + + // ✅ 规范的更新方法 + @Transactional(rollbackFor = Exception.class) + public void updateUser(User user) { + String traceId = MDC.get("traceId"); + + // 参数校验 + if (user == null || user.getId() == null) { + throw new IllegalArgumentException("用户ID不能为空"); + } + + // 查询原数据（用于审计） + User oldUser = userMapper.selectUserById(user.getId()); + if (oldUser == null) { + throw new BusinessException("用户不存在"); + } + + // 执行更新 + int affected = userMapper.updateUserById(user); + if (affected != 1) { + throw new ServiceException("更新用户失败，影响行数: " + affected); + } + + // 记录审计日志 + AuditLog auditLog = AuditLog.builder() + .traceId(traceId) + .operation("UPDATE_USER") + .entityId(user.getId().toString()) + .oldValue(JSON.toJSONString(oldUser)) + .newValue(JSON.toJSONString(user)) + .operatorId(getCurrentUserId()) + .build(); + + auditLogService.record(auditLog); + log.info("用户更新成功, traceId: {}, userId: {}", traceId, user.getId()); + } + } + + // 4. 异常处理规范 + @RestControllerAdvice + public class MyBatisExceptionHandler { + + @ExceptionHandler(DataIntegrityViolationException.class) + public ResponseResult handleDataIntegrityViolation(DataIntegrityViolationException e) { + log.error("数据完整性约束违反", e); + + // 根据具体异常返回友好提示 + if (e.getMessage().contains("Duplicate entry")) { + return ResponseResult.error("数据重复，请检查后重试"); + } + + return ResponseResult.error("数据操作失败，请联系管理员"); + } + + @ExceptionHandler(BadSqlGrammarException.class) + public ResponseResult handleBadSqlGrammar(BadSqlGrammarException e) { + log.error("SQL语法错误", e); + return ResponseResult.error("系统内部错误，请稍后重试"); + } + + @ExceptionHandler(DataAccessException.class) + public ResponseResult handleDataAccess(DataAccessException e) { + log.error("数据访问异常", e); + + // 记录详细错误信息用于排查 + String traceId = MDC.get("traceId"); + ErrorLog errorLog = ErrorLog.builder() + .traceId(traceId) + .errorType("DATA_ACCESS") + .errorMessage(e.getMessage()) + .stackTrace(ExceptionUtils.getStackTrace(e)) + .build(); + + errorLogService.record(errorLog); + + return ResponseResult.error("数据操作异常"); + } + } +} + +// 工程化配置示例 +@Configuration +public class MyBatisEngineeringConfig { + + // SQL性能监控插件 + @Bean + public SqlPerformanceInterceptor sqlPerformanceInterceptor() { + SqlPerformanceInterceptor interceptor = new SqlPerformanceInterceptor(); + interceptor.setSlowSqlThreshold(1000L); // 慢SQL阈值1秒 + interceptor.setLogSlowSql(true); + return interceptor; + } + + // SQL注入防护插件 + @Bean + public SqlInjectionInterceptor sqlInjectionInterceptor() { + return new SqlInjectionInterceptor(); + } + + // 分页插件配置 + @Bean + public PaginationInterceptor paginationInterceptor() { + PaginationInterceptor interceptor = new PaginationInterceptor(); + interceptor.setCountSqlParser(new JsqlParserCountOptimize(true)); + interceptor.setLimit(10000); // 最大分页限制 + return interceptor; + } +} +``` + +--- + +## 🔄 八、ORM 生态对比 + + **核心理念**：不同ORM框架各有特色，选择合适的ORM需要综合考虑业务特点、团队能力和系统要求，MyBatis与其他框架的组合使用是企业级应用的常见模式。 + +### 🎯 MyBatis vs JPA/Hibernate：各有什么特点？如何选择？ + +MyBatis和JPA/Hibernate代表了两种不同的ORM设计理念： + + **MyBatis特点**： + + **优势**： + - **SQL透明可控**：手写SQL，完全掌控SQL执行逻辑和性能 + - **学习成本低**：接近原生JDBC，Java开发者容易上手 + - **性能调优空间大**：可以针对具体业务场景精确优化SQL + - **复杂查询友好**：支持复杂的报表查询和统计分析SQL + - **数据库特性利用充分**：可以使用数据库特有的函数和特性 + + **劣势**： + - **开发效率相对低**：需要手写大量SQL和映射配置 + - **SQL维护成本高**：数据库变更需要同步修改SQL + - **移植性差**：SQL绑定特定数据库，跨数据库迁移困难 + - **对象关联复杂**：处理复杂对象关系需要更多代码 + + **JPA/Hibernate特点**： + + **优势**： + - **面向对象**：完全的OOP思维，实体关系映射自然 + - **开发效率高**：自动生成SQL，减少样板代码 + - **数据库无关性**：支持多数据库，迁移成本低 + - **功能丰富**：缓存、懒加载、脏检查等高级特性 + - **标准化**：JPA是Java EE标准，有良好的生态支持 + + **劣势**： + - **学习曲线陡峭**：概念复杂，需要深入理解ORM原理 + - **性能不透明**：自动生成的SQL可能不是最优的 + - **调优复杂**：性能问题定位困难，需要深入了解底层机制 + - **复杂查询局限**：某些复杂业务查询用JPQL/HQL表达困难 + + **选择策略**： + - **报表/分析系统**：选择MyBatis，SQL控制力强 + - **传统业务系统**：JPA/Hibernate，开发效率高 + - **性能敏感系统**：MyBatis，便于精确调优 + - **快速原型**：JPA，自动建表和CRUD + - **混合架构**：核心业务用MyBatis，辅助模块用JPA + +### 🎯 MyBatis-Plus能解决什么问题？有哪些最佳实践？ + +MyBatis-Plus是MyBatis的增强工具，在保持MyBatis特性的基础上提供了更多便利功能： + + **核心优势**： + + **1. 单表CRUD自动化**： + - 继承BaseMapper接口即可获得完整的CRUD操作 + - 支持泛型，类型安全 + - 自动根据实体类生成对应的SQL操作 + + **2. 强大的条件构造器**： + - QueryWrapper和UpdateWrapper提供链式API + - 支持复杂的动态查询条件 + - 避免手写动态SQL的复杂性 + + **3. 内置分页插件**： + - 自动处理COUNT查询和数据查询 + - 支持多种数据库的分页语法 + - 防止全表扫描的安全机制 + + **4. 代码生成器**： + - 根据数据库表自动生成Entity、Mapper、Service、Controller + - 支持多种模板引擎 + - 大幅减少重复代码编写 + + **5. 高级特性**： + - **审计字段自动填充**：自动处理createTime、updateTime等字段 + - **逻辑删除**：软删除支持，数据安全 + - **乐观锁**：version字段自动处理 + - **多租户**：自动添加租户ID条件 + + **使用注意事项**： + + **1. 通用方法的边界**： + - 复杂业务逻辑仍需自定义SQL + - 跨表操作需要手动处理 + - 批量操作性能需要特别关注 + + **2. 安全性考虑**： + - 避免在生产环境使用delete()等危险操作 + - 合理配置逻辑删除策略 + - 注意条件构造器的SQL注入风险 + + **3. 性能优化**： + - 合理使用索引，避免条件构造器生成低效SQL + - 大数据量操作采用分页处理 + - 监控自动生成的SQL性能 + + **最佳实践建议**： + - 简单CRUD用MyBatis-Plus，复杂查询用原生MyBatis + - 建立代码规范，统一团队使用方式 + - 做好单元测试，确保自动生成的SQL符合预期 + +**💻 代码示例**： + +```java +// 1. MyBatis-Plus基础使用示例 +@Entity +@TableName("user") +public class User { + @TableId(type = IdType.AUTO) + private Long id; + + @TableField("user_name") + private String name; + + private String email; + + @TableField(fill = FieldFill.INSERT) + private Date createTime; + + @TableField(fill = FieldFill.INSERT_UPDATE) + private Date updateTime; + + @TableLogic + private Integer deleted; + + @Version + private Integer version; + + // getters and setters... +} + +// 2. Mapper接口 - 继承BaseMapper +public interface UserMapper extends BaseMapper { + + // 自动获得完整的CRUD操作 + // insert(T entity) + // deleteById(Serializable id) + // updateById(T entity) + // selectById(Serializable id) + // selectList(Wrapper queryWrapper) + // ... + + // 自定义复杂查询仍使用MyBatis原生方式 + @Select("SELECT u.*, p.name as province_name FROM user u " + + "LEFT JOIN province p ON u.province_id = p.id " + + "WHERE u.status = #{status}") + List selectUsersWithProvince(@Param("status") Integer status); +} + +// 3. Service层 - 继承ServiceImpl +@Service +public class UserService extends ServiceImpl { + + // 条件构造器示例 + public List findActiveUsersByAge(Integer minAge, Integer maxAge) { + QueryWrapper queryWrapper = new QueryWrapper<>(); + queryWrapper.eq("status", 1) + .between("age", minAge, maxAge) + .orderByDesc("create_time"); + + return list(queryWrapper); + } + + // Lambda条件构造器（类型安全） + public List findUsersByCondition(String name, String email) { + LambdaQueryWrapper queryWrapper = new LambdaQueryWrapper<>(); + queryWrapper.like(StringUtils.isNotEmpty(name), User::getName, name) + .eq(StringUtils.isNotEmpty(email), User::getEmail, email); + + return list(queryWrapper); + } + + // 批量操作 + @Transactional + public boolean batchUpdateUsers(List users) { + return updateBatchById(users); + } + + // 分页查询 + public IPage getUsersByPage(Integer pageNum, Integer pageSize, String keyword) { + Page page = new Page<>(pageNum, pageSize); + + QueryWrapper queryWrapper = new QueryWrapper<>(); + if (StringUtils.isNotEmpty(keyword)) { + queryWrapper.and(wrapper -> + wrapper.like("name", keyword) + .or() + .like("email", keyword)); + } + + return page(page, queryWrapper); + } +} + +// 4. MyBatis-Plus配置 +@Configuration +public class MybatisPlusConfig { + + // 分页插件 + @Bean + public PaginationInterceptor paginationInterceptor() { + PaginationInterceptor paginationInterceptor = new PaginationInterceptor(); + + // 设置请求的页面大于最大页后操作，true调回到首页，false继续请求 + paginationInterceptor.setOverflow(false); + + // 设置最大单页限制数量，默认500条，-1不受限制 + paginationInterceptor.setLimit(1000); + + return paginationInterceptor; + } + + // 审计字段自动填充 + @Bean + public MetaObjectHandler metaObjectHandler() { + return new MetaObjectHandler() { + @Override + public void insertFill(MetaObject metaObject) { + this.setFieldValByName("createTime", new Date(), metaObject); + this.setFieldValByName("updateTime", new Date(), metaObject); + this.setFieldValByName("createBy", getCurrentUserId(), metaObject); + } + + @Override + public void updateFill(MetaObject metaObject) { + this.setFieldValByName("updateTime", new Date(), metaObject); + this.setFieldValByName("updateBy", getCurrentUserId(), metaObject); + } + + private Long getCurrentUserId() { + // 从Spring Security或其他方式获取当前用户ID + return UserContextHolder.getCurrentUserId(); + } + }; + } + + // 多租户插件 + @Bean + public TenantLineInnerInterceptor tenantLineInnerInterceptor() { + return new TenantLineInnerInterceptor(new TenantLineHandler() { + @Override + public Expression getTenantId() { + // 从上下文获取租户ID + Long tenantId = TenantContextHolder.getTenantId(); + return new LongValue(tenantId); + } + + @Override + public String getTenantIdColumn() { + return "tenant_id"; + } + + @Override + public boolean ignoreTable(String tableName) { + // 忽略系统表 + return Arrays.asList("sys_config", "sys_dict").contains(tableName); + } + }); + } +} + +// 5. JPA vs MyBatis对比示例 +// JPA方式 +@Entity +@Table(name = "user") +public class JpaUser { + @Id + @GeneratedValue(strategy = GenerationType.IDENTITY) + private Long id; + + @Column(name = "user_name") + private String name; + + @OneToMany(mappedBy = "user", fetch = FetchType.LAZY) + private List orders; + + // JPA Repository + public interface JpaUserRepository extends JpaRepository { + + // 方法名约定查询 + List findByNameContainingAndStatus(String name, Integer status); + + // JPQL查询 + @Query("SELECT u FROM JpaUser u WHERE u.createTime > :startTime") + List findRecentUsers(@Param("startTime") Date startTime); + + // 原生SQL查询（复杂场景） + @Query(value = "SELECT * FROM user u LEFT JOIN order o ON u.id = o.user_id " + + "WHERE u.status = ?1 GROUP BY u.id HAVING COUNT(o.id) > ?2", + nativeQuery = true) + List findUsersWithMultipleOrders(Integer status, Integer orderCount); + } +} + +// MyBatis方式对比 +public interface MyBatisUserMapper extends BaseMapper { + + // 更灵活的SQL控制 + @Select("") + List findUsersByCondition(@Param("name") String name, + @Param("status") Integer status); + + // 复杂报表查询（JPA难以表达） + List getUserStatisticsReport(@Param("params") ReportParams params); +} + +// 6. 混合使用示例 - 在同一项目中组合使用 +@Service +public class HybridUserService { + + @Autowired + private UserMapper mybatisMapper; // MyBatis-Plus + + @Autowired + private JpaUserRepository jpaRepository; // JPA + + // 简单CRUD用MyBatis-Plus + public User createUser(User user) { + mybatisMapper.insert(user); + return user; + } + + // 复杂查询用MyBatis原生 + public List getComplexReport(ReportParams params) { + return mybatisMapper.getUserStatisticsReport(params); + } + + // 对象关系映射用JPA + public List getUsersWithOrders() { + return jpaRepository.findAll(); // 自动加载关联的orders + } + + // 根据场景选择合适的工具 + public PageResult searchUsers(UserSearchParams params) { + if (params.isSimpleQuery()) { + // 简单查询用MyBatis-Plus条件构造器 + QueryWrapper wrapper = new QueryWrapper<>(); + wrapper.like("name", params.getKeyword()); + return PageResult.of(mybatisMapper.selectList(wrapper)); + } else { + // 复杂查询用自定义SQL + return mybatisMapper.searchUsersComplex(params); + } + } +} + +// 7. 代码生成器示例 +@Test +public void generateCode() { + AutoGenerator generator = new AutoGenerator(); + + // 全局配置 + GlobalConfig globalConfig = new GlobalConfig(); + globalConfig.setOutputDir(System.getProperty("user.dir") + "/src/main/java"); + globalConfig.setAuthor("MyBatis-Plus Generator"); + globalConfig.setOpen(false); + globalConfig.setServiceName("%sService"); // 去除Service接口的首字母I + + // 数据源配置 + DataSourceConfig dataSourceConfig = new DataSourceConfig(); + dataSourceConfig.setUrl("jdbc:mysql://localhost:3306/test?serverTimezone=UTC"); + dataSourceConfig.setDriverName("com.mysql.cj.jdbc.Driver"); + dataSourceConfig.setUsername("root"); + dataSourceConfig.setPassword("password"); + + // 包配置 + PackageConfig packageConfig = new PackageConfig(); + packageConfig.setParent("com.example"); + packageConfig.setEntity("model"); + packageConfig.setMapper("mapper"); + packageConfig.setService("service"); + packageConfig.setController("controller"); + + // 策略配置 + StrategyConfig strategyConfig = new StrategyConfig(); + strategyConfig.setInclude("user", "order"); // 指定生成的表名 + strategyConfig.setNaming(NamingStrategy.underline_to_camel); + strategyConfig.setColumnNaming(NamingStrategy.underline_to_camel); + strategyConfig.setEntityLombokModel(true); + strategyConfig.setLogicDeleteFieldName("deleted"); + strategyConfig.setVersionFieldName("version"); + strategyConfig.setTableFillList(Arrays.asList( + new TableFill("create_time", FieldFill.INSERT), + new TableFill("update_time", FieldFill.INSERT_UPDATE) + )); + + generator.setGlobalConfig(globalConfig); + generator.setDataSource(dataSourceConfig); + generator.setPackageInfo(packageConfig); + generator.setStrategy(strategyConfig); + + generator.execute(); +} +``` + +### 🎯 混合架构的最佳实践是什么？ + +在企业级项目中，MyBatis和JPA的混合使用是常见且有效的架构模式： + + **架构分层策略**： + + **1. 按业务复杂度分层**： + - **核心业务层**：使用MyBatis，精确控制SQL性能 + - **辅助功能层**：使用JPA/MyBatis-Plus，提高开发效率 + - **报表分析层**：使用MyBatis原生SQL，支持复杂统计查询 + + **2. 按数据特征分层**： + - **事务性数据**：MyBatis，保证数据一致性和性能 + - **配置性数据**：JPA，利用对象映射简化开发 + - **统计性数据**：原生SQL或存储过程，最大化查询效率 + + **3. 技术选型矩阵**： + - **简单CRUD + 快速开发** → MyBatis-Plus + - **复杂业务逻辑 + 性能要求** → MyBatis + - **领域建模 + 对象关系** → JPA/Hibernate + - **数据分析 + 复杂报表** → MyBatis + 原生SQL + + **混合架构注意事项**： + - 统一事务管理器，确保不同ORM在同一事务中工作 + - 建立清晰的分层边界，避免技术栈混乱 + - 统一异常处理和日志规范 + - 做好团队培训，确保开发人员熟悉各种工具的适用场景 + +--- + +## 🧪 高频面试题速览 + +- **🎯 MyBatis 执行器(Executor) 有哪些？各自适用场景？** +- **🎯 一级/二级缓存命中条件与失效场景？如何手动清理？** +- **🎯 #{} 与 ${} 的区别与使用边界？** +- **🎯 动态SQL常见坑（where多余and/逗号、空集合foreach）如何规避？** +- **🎯 分页实现：拦截器改SQL vs 改参数，哪种更合适？** +- **🎯 如何避免N+1问题？延迟加载与一次性装载如何取舍？** +- **🎯 批处理与主键回填在不同数据库（MySQL/PG/Oracle）下的差异？** +- **🎯 多数据源路由设计与事务一致性方案？** +- **🎯 如何为敏感字段做加解密并保证可检索性（前缀/哈希索引）？** +- **🎯 MyBatis 与 JPA/Hibernate/MP 组合使用的边界与实践？** + +--- + +## 📝 面试话术模板 + +``` +1) 先给出概念与定位（10-20秒） +2) 讲清核心原理/流程（30-60秒） +3) 结合优缺点与适用场景（20-30秒） +4) 补充一次亲历的实战与数据（20-30秒） +5) 若有时间，延展到性能/容错/工程化细节（加分） +``` + +- **缓存**："一级缓存是会话级，二级是命名空间级；我们开启了二级缓存并用失效策略保证一致性... +- **动态SQL**："大量使用 `//`，统一工具方法避免空集合foreach报错，重要SQL全覆盖单测... +- **分页**："采用物理分页插件，限制深翻页；热点列表采用游标分页并结合排序键... +- **调优**："批量写入使用 ExecutorType.BATCH 分批 flush，慢SQL定位靠链路trace + 执行计划... + +--- + +## 🔍 扩展学习与实践 + +- 官方文档与源码：MyBatis、MyBatis-Spring、MyBatis-Plus +- 高质量实践：分页插件/PageHelper源码、租户插件、加解密插件 +- 工具链：Druid/Hikari、p6spy、arthas/skywalking/APM +- 建议：核心SQL配套单测与回归；生产开启SQL审计与慢SQL告警 + +--- + +## 🎉 总结 + +- MyBatis 的强项在于“可控与透明”，用规范与工程化手段弥补“手写SQL”的成本 +- 先选对ORM，再用对功能；以数据驱动取舍：性能、复杂度、可维护性优先 +- 面试中用“原理 + 实战 + 数据结果”讲述你的选型与优化过程，效果最好 diff --git a/docs/interview/MySQL-FAQ.md b/docs/interview/MySQL-FAQ.md index 70605bf70f..d8733f8372 100644 --- a/docs/interview/MySQL-FAQ.md +++ b/docs/interview/MySQL-FAQ.md @@ -1,656 +1,844 @@ -![](https://imgkr.cn-bj.ufileos.com/9e22c4b4-0db5-4f4f-9636-974875d4018f.jpg) - -> 写在之前：不建议那种上来就是各种面试题罗列，然后背书式的去记忆，对技术的提升帮助很小，对正经面试也没什么帮助，有点东西的面试官深挖下就懵逼了。 +--- +title: MySQL 五万字精华总结 + 面试 100 问，和面试官扯皮绰绰有余 +date: 2024-05-31 +tags: + - MySQL + - 数据库 + - 索引优化 + - SQL调优 + - Interview +categories: Interview +--- + +![](https://img.starfish.ink/common/faq-banner.png) + +> 写在之前：不建议背书式的去记忆面试题，对技术的提升帮助很小，对正经面试也没什么帮助，准备面试的过程还是要把各个知识点真懂了，然后再连成线。 +> +> 个人建议把面试题看作是费曼学习法中的回顾、简化的环节，准备面试的时候，跟着题目先自己讲给自己听，看看自己会满意吗，不满意就继续学习这个点，然后调整自己的话术，如此反复，对这块知识也会理解的更到位，而且面试时候也会得心应手，心仪的 offer 肯定会有的。 > -> 个人建议把面试题看作是费曼学习法中的回顾、简化的环节，准备面试的时候，跟着题目先自己讲给自己听，看看自己会满意吗，不满意就继续学习这个点，如此反复，好的offer离你不远的，奥利给 +> 当然，大家有遇到过什么样『有趣』『有含量』的题目，欢迎提出来，一起学习~ -## 一、MySQL架构 +## 🗺️ 知识导航 -和其它数据库相比，MySQL有点与众不同，它的架构可以在多种不同场景中应用并发挥良好作用。主要体现在存储引擎的架构上，**插件式的存储引擎架构将查询处理和其它的系统任务以及数据的存储提取相分离**。这种架构可以根据业务的需求和实际需要选择合适的存储引擎。 +### 🏷️ 核心知识分类 -![](https://img2018.cnblogs.com/blog/1383365/201902/1383365-20190201092513900-638761565.png) +1. **🏗️ 基础与架构**：MySQL架构组件、SQL语法（DDL/DML/DCL）、存储原理 +2. **🗄️ 存储引擎**：InnoDB vs MyISAM特性对比、存储格式、缓冲池机制 +3. **🔍 索引机制与优化**：B+树原理、索引类型、覆盖索引、联合索引、索引失效场景 +4. **🔒 事务与锁机制**：ACID特性、隔离级别、死锁处理、MVCC原理、并发控制 +5. **📊 数据类型与查询优化**：数据类型选择、JOIN优化、子查询、窗口函数、执行计划分析 +6. **📝 日志系统**：redo log、undo log、binlog机制、WAL原理 +7. **⚡ 性能调优**：慢查询分析、参数调优、缓存策略、硬件优化 +8. **🚀 分库分表与集群**：主从复制、读写分离、分片策略、数据迁移、集群部署 +9. **💻 SQL实战编程**：复杂查询编写、存储过程、触发器、性能优化案例 +10. **🖊️ 手撕SQL**：经典SQL题目、复杂业务场景查询、算法实现、面试真题 +11. **🔧 运维与监控**：备份恢复、监控指标、故障排查、容量规划、日常维护 +## 一、基础与架构 🏗️ +### 🎯 MySQL 整体架构 -- **连接层**：最上层是一些客户端和连接服务。**主要完成一些类似于连接处理、授权认证、及相关的安全方案**。在该层上引入了线程池的概念，为通过认证安全接入的客户端提供线程。同样在该层上可以实现基于SSL的安全链接。服务器也会为安全接入的每个客户端验证它所具有的操作权限。 +和其它数据库相比，MySQL 有点与众不同，它的架构可以在多种不同场景中应用并发挥良好作用。主要体现在存储引擎的架构上，**插件式的存储引擎架构将查询处理和其它的系统任务以及数据的存储提取相分离**。这种架构可以根据业务的需求和实际需要选择合适的存储引擎。 -- **服务层**：第二层服务层，主要完成大部分的核心服务功能， 包括查询解析、分析、优化、缓存、以及所有的内置函数，所有跨存储引擎的功能也都在这一层实现，包括触发器、存储过程、视图等 +![](https://img.starfish.ink/mysql/architecture.png) -- **引擎层**：第三层存储引擎层，存储引擎真正的负责了MySQL中数据的存储和提取，服务器通过API与存储引擎进行通信。不同的存储引擎具有的功能不同，这样我们可以根据自己的实际需要进行选取 +> - **连接层**：最上层是一些客户端和连接服务。**主要完成一些类似于连接处理、授权认证、及相关的安全方案**。在该层上引入了线程池的概念，为通过认证安全接入的客户端提供线程。同样在该层上可以实现基于SSL的安全链接。服务器也会为安全接入的每个客户端验证它所具有的操作权限。 +> +> - **服务层**：第二层服务层，主要完成大部分的核心服务功能， 包括查询解析、分析、优化、缓存、以及所有的内置函数，所有跨存储引擎的功能也都在这一层实现，包括触发器、存储过程、视图等 +> +> - **引擎层**：第三层存储引擎层，存储引擎真正的负责了MySQL中数据的存储和提取，服务器通过API与存储引擎进行通信。不同的存储引擎具有的功能不同，这样我们可以根据自己的实际需要进行选取 +> +> - **存储层**：第四层为数据存储层，主要是将数据存储在运行于该设备的文件系统之上，并完成与存储引擎的交互 +> -- **存储层**：第四层为数据存储层，主要是将数据存储在运行于该设备的文件系统之上，并完成与存储引擎的交互 +### 🎯 MySQL 的查询流程具体是怎么样的？ -> 画出 MySQL 架构图，这种变态问题都能问的出来 +> [!TIP] > -> MySQL 的查询流程具体是？or 一条SQL语句在MySQL中如何执行的？ +> 一条 SQL 语句在 MySQL 中如何执行的？ > +> 画出 MySQL 架构图？ 「这种变态问题都能问的出来~」 -客户端请求 ---> 连接器（验证用户身份，给予权限） ---> 查询缓存（存在缓存则直接返回，不存在则执行后续操作） ---> 分析器（对SQL进行词法分析和语法分析操作） ---> 优化器（主要对执行的sql优化选择最优的执行方案方法） ---> 执行器（执行时会先看用户是否有执行权限，有才去使用这个引擎提供的接口） ---> 去引擎层获取数据返回（如果开启查询缓存则会缓存查询结果） +1. **客户端请求**：客户端通过连接器发送查询到 MySQL 服务器（验证用户身份，给予权限） +2. **查询接收**：连接器接收请求，管理连接 +3. **解析器**：对 SQL 进行词法分析和语法分析，转换为解析树 +4. **优化器**：优化器生成执行计划，选择最优索引和连接顺序 +5. **查询执行器**：执行器执行查询，通过存储引擎接口获取数据 +6. **存储引擎**：存储引擎检索数据，返回给执行器 +7. **返回结果**：结果通过连接器返回给客户端 -![img](https://pic2.zhimg.com/80/v2-0d2070e8f84c4801adbfa03bda1f98d9_720w.jpg) - ------- +![](https://img.starfish.ink/mysql/MySQL-select-flow.png) -> 说说MySQL有哪些存储引擎？都有哪些区别？ +### 🎯 MySQL 的数据存储结构 -## 二、存储引擎 +> MySQL（InnoDB）的存储结构是分层设计的，数据最终存放在页（Page）里，页是最小的存储和 I/O 单位（16KB）。多个页组成区（Extent，1MB），多个区组成段（Segment），段属于表空间（Tablespace）。每张表对应一个表空间，里面包含数据段和索引段。行数据存放在数据页中，大字段可能存放在溢出页。这样的设计能兼顾存储管理、性能和空间利用。 -存储引擎是MySQL的组件，用于处理不同表类型的SQL操作。不同的存储引擎提供不同的存储机制、索引技巧、锁定水平等功能，使用不同的存储引擎，还可以获得特定的功能。 +在 MySQL 中，数据的存储结构从上到下大致分为 **库 → 表 → 表空间 → 段（Segment）→ 区（Extent）→ 页（Page）→ 行（Row）** 这几个层次。 -使用哪一种引擎可以灵活选择，**一个数据库中多个表可以使用不同引擎以满足各种性能和实际需求**，使用合适的存储引擎，将会提高整个数据库的性能 。 +**1. 库（Database）** - MySQL服务器使用**可插拔**的存储引擎体系结构，可以从运行中的 MySQL 服务器加载或卸载存储引擎 。 +- 最上层的逻辑组织单位，相当于一个文件夹。 +- 在磁盘上表现为一个 **目录**，里面包含了表对应的文件。 -### 查看存储引擎 +**2. 表（Table）** -```mysql --- 查看支持的存储引擎 -SHOW ENGINES +- 每张表就是一组结构化的数据。 --- 查看默认存储引擎 -SHOW VARIABLES LIKE 'storage_engine' +**3. 表空间（Tablespace）** ---查看具体某一个表所使用的存储引擎，这个默认存储引擎被修改了！ -show create table tablename +- 表空间是 InnoDB 的数据存储文件，可以是共享的（ibdata）或独立的（.ibd）。 +- 表空间里存放表的所有数据和索引。 +- 每个表空间由若干个 **段（Segment）** 组成。 ---准确查看某个数据库中的某一表所使用的存储引擎 -show table status like 'tablename' -show table status from database where name="tablename" -``` +**4. 段（Segment）** -### 设置存储引擎 +- 段是表空间中的存储单位，用于管理不同用途的空间。 +- 主要有两类： + - **数据段（Data Segment）**：存储表的数据（聚簇索引）。 + - **索引段（Index Segment）**：存储二级索引的数据。 +- 段是由 **多个区（Extent）** 组成的。 -```mysql --- 建表时指定存储引擎。默认的就是INNODB，不需要设置 -CREATE TABLE t1 (i INT) ENGINE = INNODB; -CREATE TABLE t2 (i INT) ENGINE = CSV; -CREATE TABLE t3 (i INT) ENGINE = MEMORY; +**5. 区（Extent）** --- 修改存储引擎 -ALTER TABLE t ENGINE = InnoDB; +- 区是空间分配的基本单位，每个区大小固定为 **1MB**。 +- 在 InnoDB 默认 16KB 页大小时： + - 每个区包含 **64 个页（16KB × 64 = 1MB）**。 +- 为了减少空间碎片，InnoDB 会先以页为单位分配，表大了再按区分配。 --- 修改默认存储引擎，也可以在配置文件my.cnf中修改默认引擎 -SET default_storage_engine=NDBCLUSTER; -``` +**6. 页（Page）** -默认情况下，每当 `CREATE TABLE` 或 `ALTER TABLE` 不能使用默认存储引擎时，都会生成一个警告。为了防止在所需的引擎不可用时出现令人困惑的意外行为，可以启用 `NO_ENGINE_SUBSTITUTION SQL` 模式。如果所需的引擎不可用，则此设置将产生错误而不是警告，并且不会创建或更改表 +- 页是 **InnoDB 磁盘和内存交互的基本单位**，默认大小 **16KB**。 +- 常见页类型： + - **数据页（B+Tree Node Page）**：存储表数据。 + - **Undo 页**：存储回滚日志。 + - **系统页**：存储事务系统信息。 + - **索引页**：存储索引数据。 +- 页内的数据通过 **页目录（Page Directory）** 进行管理。 +**7. 行（Row）** +- 行是最小的数据存储单元。 +- InnoDB 是 **行存储**，每行数据会按字段存储在数据页中。 +- 行存储的特点： + - 一行数据可能存不下时，大字段（如 TEXT、BLOB）会存储在溢出页（Overflow Page），行中只保留指针。 + - 每行都有额外的隐藏字段，比如：**DB_TRX_ID**（事务 ID）、**DB_ROLL_PTR**（回滚指针）等 -### 存储引擎对比 +``` +数据库（Database） + └── 表（Table） + └── 表空间（Tablespace） + └── 段（Segment） + └── 区（Extent，1MB） + └── 页（Page，16KB） + └── 行（Row） +``` -常见的存储引擎就 InnoDB、MyISAM、Memory、NDB。 -InnoDB 现在是 MySQL 默认的存储引擎，支持**事务、行级锁定和外键** -#### 文件存储结构对比 +### 🎯 DDL、DML、DCL的区别和应用？ -在 MySQL中建立任何一张数据表，在其数据目录对应的数据库目录下都有对应表的 `.frm` 文件，`.frm` 文件是用来保存每个数据表的元数据(meta)信息，包括表结构的定义等，与数据库存储引擎无关，也就是任何存储引擎的数据表都必须有`.frm`文件，命名方式为 数据表名.frm，如user.frm。 +MySQL语言分为三大类，各有不同的作用和权限要求： -查看MySQL 数据保存在哪里：`show variables like 'data%'` +**语言分类对比**： -MyISAM 物理文件结构为： +1. **DDL（Data Definition Language）**：数据定义语言 +2. **DML（Data Manipulation Language）**：数据操作语言 +3. **DCL（Data Control Language）**：数据控制语言 -- `.frm`文件：与表相关的元数据信息都存放在frm文件，包括表结构的定义信息等 -- `.MYD` (`MYData`) 文件：MyISAM 存储引擎专用，用于存储MyISAM 表的数据 -- `.MYI` (`MYIndex`)文件：MyISAM 存储引擎专用，用于存储MyISAM 表的索引相关信息 +------ -InnoDB 物理文件结构为： -- `.frm` 文件：与表相关的元数据信息都存放在frm文件，包括表结构的定义信息等 -- `.ibd` 文件或 `.ibdata` 文件： 这两种文件都是存放 InnoDB 数据的文件，之所以有两种文件形式存放 InnoDB 的数据，是因为 InnoDB 的数据存储方式能够通过配置来决定是使用**共享表空间**存放存储数据，还是用**独享表空间**存放存储数据。 +## 二、存储引擎 🗄️ - 独享表空间存储方式使用`.ibd`文件，并且每个表一个`.ibd`文件 - 共享表空间存储方式使用`.ibdata`文件，所有表共同使用一个`.ibdata`文件（或多个，可自己配置） +> 存储引擎是 MySQL 的组件，用于处理不同表类型的 SQL 操作。不同的存储引擎提供不同的存储机制、索引技巧、锁定水平等功能，使用不同的存储引擎，还可以获得特定的功能。 +> +> 使用哪一种引擎可以灵活选择，**一个数据库中多个表可以使用不同引擎以满足各种性能和实际需求**，使用合适的存储引擎，将会提高整个数据库的性能 。 +> +> MySQL 服务器使用**可插拔**的存储引擎体系结构，可以从运行中的 MySQL 服务器加载或卸载存储引擎 。 +> +> **查看存储引擎** +> +> ```mysql +> -- 查看支持的存储引擎 +> SHOW ENGINES +> +> -- 查看默认存储引擎 +> SHOW VARIABLES LIKE 'storage_engine' +> +> --查看具体某一个表所使用的存储引擎，这个默认存储引擎被修改了！ +> show create table tablename +> +> --准确查看某个数据库中的某一表所使用的存储引擎 +> show table status like 'tablename' +> show table status from database where name="tablename" +> ``` +> +> **设置存储引擎** +> +> ```mysql +> -- 建表时指定存储引擎。默认的就是INNODB，不需要设置 +> CREATE TABLE t1 (i INT) ENGINE = INNODB; +> CREATE TABLE t2 (i INT) ENGINE = CSV; +> CREATE TABLE t3 (i INT) ENGINE = MEMORY; +> +> -- 修改存储引擎 +> ALTER TABLE t ENGINE = InnoDB; +> +> -- 修改默认存储引擎，也可以在配置文件my.cnf中修改默认引擎 +> SET default_storage_engine=NDBCLUSTER; +> ``` +> -> ps：正经公司，这些都有专业运维去做，数据备份、恢复啥的，让我一个 Javaer 搞这的话，加钱不？ +### 🎯 说说 MySQL 都有哪些存储引擎？都有哪些区别？ -#### 面试这么回答 +常见的存储引擎就 InnoDB、MyISAM、Memory、NDB。 -1. InnoDB 支持事务，MyISAM 不支持事务。这是 MySQL 将默认存储引擎从 MyISAM 变成 InnoDB 的重要原因之一； -2. InnoDB 支持外键，而 MyISAM 不支持。对一个包含外键的 InnoDB 表转为 MYISAM 会失败； -3. InnoDB 是聚簇索引，MyISAM 是非聚簇索引。聚簇索引的文件存放在主键索引的叶子节点上，因此 InnoDB 必须要有主键，通过主键索引效率很高。但是辅助索引需要两次查询，先查询到主键，然后再通过主键查询到数据。因此，主键不应该过大，因为主键太大，其他索引也都会很大。而 MyISAM 是非聚集索引，数据文件是分离的，索引保存的是数据文件的指针。主键索引和辅助索引是独立的。 -4. InnoDB 不保存表的具体行数，执行` select count(*) from table` 时需要全表扫描。而 MyISAM 用一个变量保存了整个表的行数，执行上述语句时只需要读出该变量即可，速度很快； -5. InnoDB 最小的锁粒度是行锁，MyISAM 最小的锁粒度是表锁。一个更新语句会锁住整张表，导致其他查询和更新都会被阻塞，因此并发访问受限。这也是 MySQL 将默认存储引擎从 MyISAM 变成 InnoDB 的重要原因之一； +InnoDB 现在是 MySQL5.5 版本后默认的存储引擎，支持**事务、行级锁定和外键**。我们一般和 MyISAM 进行对比即可 -| 对比项 | MyISAM | InnoDB | -| -------- | -------------------------------------------------------- | ------------------------------------------------------------ | -| 主外键 | 不支持 | 支持 | -| 事务 | 不支持 | 支持 | -| 行表锁 | 表锁，即使操作一条记录也会锁住整个表，不适合高并发的操作 | 行锁,操作时只锁某一行，不对其它行有影响，适合高并发的操作 | -| 缓存 | 只缓存索引，不缓存真实数据 | 不仅缓存索引还要缓存真实数据，对内存要求较高，而且内存大小对性能有决定性的影响 | -| 表空间 | 小 | 大 | -| 关注点 | 性能 | 事务 | -| 默认安装 | 是 | 是 | +1. **事务支持**：InnoDB 支持事务，MyISAM 不支持事务。这是 MySQL 将默认存储引擎从 MyISAM 变成 InnoDB 的重要原因之一； +2. **外键约束**：InnoDB 支持外键，而 MyISAM 不支持。对一个包含外键的 InnoDB 表转为 MYISAM 会失败； +3. **存储结构**：InnoDB 是聚簇索引，MyISAM 是非聚簇索引。聚簇索引的文件存放在主键索引的叶子节点上，因此 InnoDB 必须要有主键，通过主键索引效率很高。但是辅助索引需要两次查询，先查询到主键，然后再通过主键查询到数据。因此，主键不应该过大，因为主键太大，其他索引也都会很大。而 MyISAM 是非聚集索引，数据文件是分离的，索引保存的是数据文件的指针。主键索引和辅助索引是独立的。 +4. **表空间管理**：InnoDB 使用表空间（tablespace）来管理数据存储，支持自动扩展，支持表和索引分开存储，提高存储效率。MyISAM 每个表有三个文件：`.frm`（表定义文件）、`.MYD`（数据文件）和 `.MYI`（索引文件） +5. **锁定机制**：InnoDB 最小的锁粒度是行锁，MyISAM 最小的锁粒度是表锁。一个更新语句会锁住整张表，导致其他查询和更新都会被阻塞，因此并发访问受限。这也是 MySQL 将默认存储引擎从 MyISAM 变成 InnoDB 的重要原因之一； +6. **崩溃恢复**：InnoDB 具有崩溃恢复的能力，使用重做日志（redo log）和回滚日志（undo log）来恢复数据；MyISAM 没有崩溃恢复机制，可能需要手动恢复 +7. **性能**： InnoDB 在写密集型操作中表现更好，特别是在需要事务和外键约束的场景下。MyISAM 在读密集型操作中表现更好，尤其是在没有写操作的情况下。 -> 一张表，里面有ID自增主键，当insert了17条记录之后，删除了第15,16,17条记录，再把Mysql重启，再insert一条记录，这条记录的ID是18还是15 ？ - -如果表的类型是MyISAM，那么是18。因为MyISAM表会把自增主键的最大ID 记录到数据文件中，重启MySQL自增主键的最大ID也不会丢失； - -如果表的类型是InnoDB，那么是15。因为InnoDB 表只是把自增主键的最大ID记录到内存中，所以重启数据库或对表进行OPTION操作，都会导致最大ID丢失。 - -> 哪个存储引擎执行 select count(*) 更快，为什么? - -MyISAM更快，因为MyISAM内部维护了一个计数器，可以直接调取。 - -- 在 MyISAM 存储引擎中，把表的总行数存储在磁盘上，当执行 select count(\*) from t 时，直接返回总数据。 - -- 在 InnoDB 存储引擎中，跟 MyISAM 不一样，没有将总行数存储在磁盘上，当执行 select count(\*) from t 时，会先把数据读出来，一行一行的累加，最后返回总数量。 +> #### 文件存储结构对比 +> +> 在 MySQL中建立任何一张数据表，在其数据目录对应的数据库目录下都有对应表的 `.frm` 文件，`.frm` 文件是用来保存每个数据表的元数据(meta)信息，包括表结构的定义等，与数据库存储引擎无关，也就是任何存储引擎的数据表都必须有`.frm`文件，命名方式为 数据表名.frm，如 user.frm。 +> +> 查看 MySQL 数据保存在哪里：`show variables like 'data%'` +> +> MyISAM 物理文件结构为： +> +> - `.frm`文件：与表相关的元数据信息都存放在frm文件，包括表结构的定义信息等 +> - `.MYD` (`MYData`) 文件：MyISAM 存储引擎专用，用于存储 MyISAM 表的数据 +> - `.MYI` (`MYIndex`)文件：MyISAM 存储引擎专用，用于存储 MyISAM 表的索引相关信息 +> +> InnoDB 物理文件结构为： +> +> - `.frm` 文件：与表相关的元数据信息都存放在 frm 文件，包括表结构的定义信息等 +> +> - `.ibd` 文件或 `.ibdata` 文件： 这两种文件都是存放 InnoDB 数据的文件，之所以有两种文件形式存放 InnoDB 的数据，是因为 InnoDB 的数据存储方式能够通过配置来决定是使用**共享表空间**存放存储数据，还是用**独享表空间**存放存储数据。 +> +> 独享表空间存储方式使用`.ibd`文件，并且每个表一个`.ibd`文件 +> 共享表空间存储方式使用`.ibdata`文件，所有表共同使用一个`.ibdata`文件（或多个，可自己配置） +> -InnoDB 中 count(\*) 语句是在执行的时候，全表扫描统计总数量，所以当数据越来越大时，语句就越来越耗时了，为什么 InnoDB 引擎不像 MyISAM 引擎一样，将总行数存储到磁盘上？这跟 InnoDB 的事务特性有关，由于多版本并发控制（MVCC）的原因，InnoDB 表“应该返回多少行”也是不确定的。 +### 🎯 哪个存储引擎执行 select count(*) 更快，为什么? -## 三、数据类型 +MyISAM 更快，因为 MyISAM 内部维护了一个计数器，可以直接调取。 -主要包括以下五大类： +- MyISAM 存储每个表的行数在表的元数据中，因此执行 `SELECT COUNT(*)` 时，它可以直接读取这个值，而不需要扫描整个表 -- 整数类型：BIT、BOOL、TINY INT、SMALL INT、MEDIUM INT、 INT、 BIG INT -- 浮点数类型：FLOAT、DOUBLE、DECIMAL -- 字符串类型：CHAR、VARCHAR、TINY TEXT、TEXT、MEDIUM TEXT、LONGTEXT、TINY BLOB、BLOB、MEDIUM BLOB、LONG BLOB -- 日期类型：Date、DateTime、TimeStamp、Time、Year -- 其他数据类型：BINARY、VARBINARY、ENUM、SET、Geometry、Point、MultiPoint、LineString、MultiLineString、Polygon、GeometryCollection等 +- nnoDB 不存储行数信息在表的元数据中。每次执行 `SELECT COUNT(*)` 查询时，InnoDB 都需要扫描整个表来计算行数。这对于大表来说可能会非常慢。 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gf29eij8zsj316x0u0gsn.jpg) +InnoDB 不将表的行数存储在元数据中，主要原因是其设计目标与 MyISAM 不同。InnoDB 设计为支持高并发的事务处理和数据一致性，因此其存储和计数机制需要权衡性能和一致性。以下是一些具体原因： -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gf29fk2f4rj31ac0gi0w3.jpg) +**1. 行级锁定和并发控制** -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gf29g2azwtj31a80nywit.jpg) +InnoDB 支持行级锁定，这意味着在高并发环境中，不同事务可以同时对不同的行进行操作，而不会相互阻塞。为了确保这种并发控制和数据一致性，InnoDB 需要动态计算行数，以反映当前事务视图下的数据状态。 +- **事务隔离级别**：InnoDB 支持多种事务隔离级别（如 READ COMMITTED、REPEATABLE READ、SERIALIZABLE），这些隔离级别决定了事务如何看到数据。预先存储的行数无法满足这些隔离级别的要求，因为行数在不同事务下可能有所不同。 +- **锁机制**：由于行级锁定，InnoDB 在处理大量并发事务时，需要动态调整行数信息，而不是依赖预先存储的静态行数。 +**2. 一致性和持久性** -> CHAR 和 VARCHAR 的区别？ +InnoDB 设计为支持 ACID 属性（原子性、一致性、隔离性、持久性），这要求所有的数据操作都必须保证一致性和可靠性。 -char是固定长度，varchar长度可变： +- **崩溃恢复**：InnoDB 使用重做日志（redo log）和回滚日志（undo log）来实现崩溃恢复。如果行数保存在元数据中，崩溃恢复后行数可能与实际数据不一致，从而破坏数据的一致性。 +- **并发更新**：在高并发环境中，多个事务可能同时修改表中的数据。如果行数保存在元数据中，每次更新都需要锁定并更新元数据，这将导致严重的性能瓶颈。 -char(n) 和 varchar(n) 中括号中 n 代表字符的个数，并不代表字节个数，比如 CHAR(30) 就可以存储 30 个字符。 +**3. 性能优化** -存储时，前者不管实际存储数据的长度，直接按 char 规定的长度分配存储空间；而后者会根据实际存储的数据分配最终的存储空间 +动态计算行数虽然在某些查询中（如 `SELECT COUNT(*)`）较慢，但它避免了在高并发写操作下频繁更新元数据的性能开销。 -相同点： +- **写操作性能**：为了保证高效的写操作，InnoDB 设计避免了每次写操作都需要更新元数据的设计，这样可以更好地处理高并发写入。 +- **实际应用**：在实际应用中，行数的精确统计并不是经常需要的操作。大多数情况下，应用程序可以通过索引和其他机制来实现高效的数据访问，而不依赖于 `SELECT COUNT(*)` 的性能。 -1. char(n)，varchar(n)中的n都代表字符的个数 -2. 超过char，varchar最大长度n的限制后，字符串会被截断。 -不同点： -1. char不论实际存储的字符数都会占用n个字符的空间，而varchar只会占用实际字符应该占用的字节空间加1（实际长度length，0<=length<255）或加2（length>255）。因为varchar保存数据时除了要保存字符串之外还会加一个字节来记录长度（如果列声明长度大于255则使用两个字节来保存长度）。 -2. 能存储的最大空间限制不一样：char的存储上限为255字节。 -3. char在存储时会截断尾部的空格，而varchar不会。 +### 🎯 为什么 MySQL 默认存储引擎从 MyISAM 改为 InnoDB？ -char是适合存储很短的、一般固定长度的字符串。例如，char非常适合存储密码的MD5值，因为这是一个定长的值。对于非常短的列，char比varchar在存储空间上也更有效率。 +MySQL 从 5.5 开始默认引擎改为 InnoDB，主要是因为 InnoDB 更符合企业级应用的需求： +- 支持事务，保证 ACID； +- 宕机时能通过 redo/undo log 保证数据安全； +- 行级锁提高了并发性能，而 MyISAM 只有表级锁； +- 支持外键，保证数据一致性； +- 有 Buffer Pool 缓存机制，性能更优； +- 也是 MySQL 社区未来重点发展的方向。 +因此 InnoDB 逐渐取代 MyISAM，成为默认存储引擎。 -> 列的字符串类型可以是什么？ +------ -字符串类型是：SET、BLOB、ENUM、CHAR、CHAR、TEXT、VARCHAR +## 三、索引机制与优化 🔍 -> BLOB和TEXT有什么区别？ +> - MYSQL官方对索引的定义为：索引（Index）是帮助 MySQL 高效获取数据的数据结构，所以说**索引的本质是：数据结构** +> +> - 索引的目的在于提高查询效率，可以类比字典、 火车站的车次表、图书的目录等 。 +> +> - 可以简单的理解为“排好序的快速查找数据结构”，数据本身之外，**数据库还维护者一个满足特定查找算法的数据结构**，这些数据结构以某种方式引用（指向）数据，这样就可以在这些数据结构上实现高级查找算法。这种数据结构，就是索引。下图是一种可能的索引方式示例。 +> +> ![](https://img.starfish.ink/mysql/search-index-demo.png) +> +> 左边的数据表，一共有两列七条记录，最左边的是数据记录的物理地址 +> +> 为了加快Col2的查找，可以维护一个右边所示的二叉查找树，每个节点分别包含索引键值，和一个指向对应数据记录物理地址的指针，这样就可以运用二叉查找在一定的复杂度内获取到对应的数据，从而快速检索出符合条件的记录。 +> +> - 索引本身也很大，不可能全部存储在内存中，**一般以索引文件的形式存储在磁盘上** +> +> - 平常说的索引，没有特别指明的话，就是 B+ 树（多路搜索树，不一定是二叉树）结构组织的索引。其中聚集索引，次要索引，覆盖索引，复合索引，前缀索引，唯一索引默认都是使用 B+ 树索引，统称索引。此外还有哈希索引等。 +> -BLOB是一个二进制对象，可以容纳可变数量的数据。有四种类型的BLOB：TINYBLOB、BLOB、MEDIUMBLO和 LONGBLOB -TEXT是一个不区分大小写的BLOB。四种TEXT类型：TINYTEXT、TEXT、MEDIUMTEXT 和 LONGTEXT。 -BLOB 保存二进制数据，TEXT 保存字符数据。 +### 🎯 说说你对 MySQL 索引的理解？ ------- +> 这种就属于比较宽泛的问题，可以有结构条例的多说一些。差不多的问法： +> +> - 索引是越多越好吗？为什么？ +> - 索引有哪些优缺点？ +> [!TIP] +> +> **话术点**：B+ 树、优缺点、索引分类、最左匹配原则 +索引是数据库优化的重要工具，从数据结构上来说，在 MySQL 里面索引主要是 B+ 树索引。它的查询性能更好，适合范围查询，也适合放在内存里。 MySQL 的索引又可以从不同的角度进一步划分。比如说根据叶子节点是否包含 数据分成聚簇索引和非聚簇索引，还有包含某个查询的所有列的覆盖索引等 等。数据库使用索引遵循最左匹配原则。但是最终数据库会不会用索引，也是一个比较难说的事情，跟查询有关，也跟数据量有关。在实践中，是否使用索引以及使用什么索引，都要以 EXPLAIN 为准。 -## 四、索引 +> **优势** +> +> - 提高数据检索效率，降低数据库IO成本 +> +> - 降低数据排序的成本，降低CPU的消耗 +> +> +> **劣势** +> +> - 索引也是一张表，保存了主键和索引字段，并指向实体表的记录，所以也需要占用内存 +> - 虽然索引大大提高了查询速度，同时却会降低更新表的速度，如对表进行 INSERT、UPDATE 和 DELETE。 +> 因为更新表时，MySQL 不仅要保存数据，还要保存一下索引文件每次更新添加了索引列的字段， +> 都会调整因为更新所带来的键值变化后的索引信息 ->说说你对 MySQL 索引的理解？ +> ##### MySQL索引分类 > ->数据库索引的原理，为什么要用 B+树，为什么不用二叉树？ +> ###### 数据结构角度 > ->聚集索引与非聚集索引的区别？ +> - B+树索引 +> - Hash索引 +> - R-Tree索引 > ->InnoDB引擎中的索引策略，了解过吗？ +> ###### 从物理存储角度 > ->创建索引的方式有哪些？ +> - 聚集索引（clustered index） +> +> - 非聚集索引（non-clustered index），也叫辅助索引（secondary index） +> +> 聚集索引和非聚集索引都是B+树结构 +> +> ###### 从逻辑角度 +> +> - 主键索引：主键索引是一种特殊的唯一索引，不允许有空值 +> - 普通索引或者单列索引：每个索引只包含单个列，一个表可以有多个单列索引 +> - 多列索引（复合索引、联合索引）：复合索引指多个字段上创建的索引，只有在查询条件中使用了创建索引时的第一个字段，索引才会被使用。使用复合索引时遵循最左前缀集合 +> - 唯一索引或者非唯一索引 +> - 空间索引：空间索引是对空间数据类型的字段建立的索引，MYSQL中的空间数据类型有4种，分别是GEOMETRY、POINT、LINESTRING、POLYGON。 +> MYSQL使用SPATIAL关键字进行扩展，使得能够用于创建正规索引类型的语法创建空间索引。创建空间索引的列，必须将其声明为NOT NULL，空间索引只能在存储引擎为MYISAM的表中创建 > ->聚簇索引/非聚簇索引，mysql索引底层实现，为什么不用B-tree，为什么不用hash，叶子结点存放的是数据还是指向数据的内存地址，使用索引需要注意的几个地方？ -- MYSQL官方对索引的定义为：索引（Index）是帮助MySQL高效获取数据的数据结构，所以说**索引的本质是：数据结构** -- 索引的目的在于提高查询效率，可以类比字典、 火车站的车次表、图书的目录等 。 -- 可以简单的理解为“排好序的快速查找数据结构”，数据本身之外，**数据库还维护者一个满足特定查找算法的数据结构**，这些数据结构以某种方式引用（指向）数据，这样就可以在这些数据结构上实现高级查找算法。这种数据结构，就是索引。下图是一种可能的索引方式示例。 +### 🎯 说一下 MySQL InnoDB 的索引原理是什么? - ![](https://tva1.sinaimg.cn/large/007S8ZIlly1gf3u9tli6gj30gt08xdg2.jpg) +> 这就涉及到了好多知识点，我们可以列举几项关键点，说说**索引结构**、**聚簇索引** - 左边的数据表，一共有两列七条记录，最左边的是数据记录的物理地址 +**首先要明白索引（index）是在存储引擎（storage engine）层面实现的，而不是server层面**。不是所有的存储引擎都支持所有的索引类型。即使多个存储引擎支持某一索引类型，它们的实现和行为也可能有所差别。 - 为了加快Col2的查找，可以维护一个右边所示的二叉查找树，每个节点分别包含索引键值，和一个指向对应数据记录物理地址的指针，这样就可以运用二叉查找在一定的复杂度内获取到对应的数据，从而快速检索出符合条件的记录。 +- **B+Tree索引** -- 索引本身也很大，不可能全部存储在内存中，**一般以索引文件的形式存储在磁盘上** + InnoDB 的主要索引结构是 B+ 树索引。B+ 树是一种平衡树，每个节点可以有多个子节点。与 B 树不同，B+ 树的所有数据都存储在叶子节点中，叶子节点之间通过指针相连，这使得范围查询和排序操作非常高效。 -- 平常说的索引，没有特别指明的话，就是B+树（多路搜索树，不一定是二叉树）结构组织的索引。其中聚集索引，次要索引，覆盖索引，复合索引，前缀索引，唯一索引默认都是使用B+树索引，统称索引。此外还有哈希索引等。 +- **聚簇索引 和 非聚簇索引** + - InnoDB 中的主键索引就是聚簇索引。聚簇索引将数据行与索引紧密结合在一起，数据行存储在叶子节点中，因此通过主键查找数据非常高效 + - 当你创建一个表并指定主键时，InnoDB 会自动使用主键创建一个聚簇索引。 + - 如果没有显式定义主键，InnoDB 会选择一个唯一的非空索引代替。 + - 如果没有唯一非空索引，InnoDB 会自动生成一个隐藏的行 ID 作为聚簇索引 + - 辅助索引（也称为二级索引或非聚簇索引）是用于加速对非主键列的查询。辅助索引的叶子节点存储索引列的值以及对应的主键值。 + 使用辅助索引进行查询时，InnoDB 首先通过辅助索引找到主键值，然后通过主键值在聚簇索引中查找实际数据。这种回表（回查）过程可能增加查询时间，但仍然比全表扫描快得多。 -### 基本语法： + -- 创建： +### 🎯 为什么要用 B+树？ - - 创建索引：`CREATE [UNIQUE] INDEX indexName ON mytable(username(length));` +> B+ 树 索引相比于其他索引类型的优势？ +> +> 为什么MySQL 索引中用 B+tree，不用 B-tree 或者其他树，为什么不用 Hash 索引 +> +> B-Tree 对比 B+Tree索引 - 如果是CHAR，VARCHAR类型，length可以小于字段实际长度；如果是BLOB和TEXT类型，必须指定 length。 +- **B+Tree 相对于 B 树 索引结构的优势：** - - 修改表结构(添加索引)：`ALTER table tableName ADD [UNIQUE] INDEX indexName(columnName)` + - B+ 树空间利用率更高：B+Tree 只在叶子节点存储数据，而 B 树 的非叶子节点也要存储数据，所以 B+Tree 的单个节点的数据量更小，在相同的磁盘 I/O 次数下，就能查询更多的节点。 -- 删除：`DROP INDEX [indexName] ON mytable;` -- 查看：`SHOW INDEX FROM table_name\G` --可以通过添加 \G 来格式化输出信息。 + - B 树只适合随机检索，B+Tree 叶子节点采用的是双链表连接，同时支持**随机检索和顺序检索** -- 使用ALERT命令 - - `ALTER TABLE tbl_name ADD PRIMARY KEY (column_list):` 该语句添加一个主键，这意味着索引值必须是唯一的，且不能为NULL。 - - `ALTER TABLE tbl_name ADD UNIQUE index_name (column_list` 这条语句创建索引的值必须是唯一的（除了NULL外，NULL可能会出现多次）。 - - `ALTER TABLE tbl_name ADD INDEX index_name (column_list)` 添加普通索引，索引值可出现多次。 - - `ALTER TABLE tbl_name ADD FULLTEXT index_name (column_list)`该语句指定了索引为 FULLTEXT ，用于全文索引。 +- **B+Tree 相对于二叉树索引结构的优势：** + - 与 B+ 树相比，平衡二叉树、红黑树在同等数据量下，高度更高，性能更差，而且它们会频 繁执行再平衡过程，来保证树形结构平衡 + - 对于有 N 个叶子节点的 B+Tree，其搜索复杂度为$O(logdN)$，其中 d 表示节点允许的最大子节点个数为 d 个。在实际的应用当中， d 值是大于100的，这样就保证了，即使数据达到千万级别时，B+Tree 的高度依然维持在 3~4 层左右，也就是说一次数据查询操作只需要做 3~4 次的磁盘 I/O 操作就能查询到目标数据（这里的查询参考上面 B+Tree 的聚簇索引的查询过程）。 -### 优势 + 而二叉树的每个父节点的儿子节点个数只能是 2 个，意味着其搜索复杂度为 $O(logN)$，这已经比 B+Tree 高出不少，因此二叉树检索到目标数据所经历的磁盘 I/O 次数要更多。 -- **提高数据检索效率，降低数据库IO成本** +- **B+Tree 相对于 Hash 表存储结构的优势**： + - 我们知道范围查询是 MySQL 中常见的场景，但是 Hash 表不适合做范围查询，它更适合做等值的查询，这也是 B+Tree 索引要比 Hash 表索引有着更广泛的适用场景的原因。 -- **降低数据排序的成本，降低CPU的消耗** +- **B+Tree 相对于 跳表存储结构的优势**： + - 与B+ 树相比，跳表在极端情况下会退化为链表，平衡性差，而数据库查询需要一个可预期 的查询时间，并且跳表需要更多的内存。 - +> MyISAM 和 InnoDB 存储引擎，都使用 B+Tree的数据结构，它相对与 B-Tree结构，所有的数据都存放在叶子节点上，且把叶子节点通过指针连接到一起，形成了一条数据链表，以加快相邻数据的检索效率。 +> +> B-Tree 是为磁盘等外存储设备设计的一种平衡查找树。 +> +> 系统从磁盘读取数据到内存时是以磁盘块（block）为基本单位的，位于同一个磁盘块中的数据会被一次性读取出来，而不是需要什么取什么。 +> +> InnoDB 存储引擎中有页（Page）的概念，页是其磁盘管理的最小单位。InnoDB 存储引擎中默认每个页的大小为16KB。 +> +> 而系统一个磁盘块的存储空间往往没有这么大，因此 InnoDB 每次申请磁盘空间时都会是若干地址连续磁盘块来达到页的大小 16KB。InnoDB 在把磁盘数据读入到磁盘时会以页为基本单位，在查询数据时如果一个页中的每条数据都能有助于定位数据记录的位置，这将会减少磁盘I/O次数，提高查询效率。 +> +> B+Tree 是在 B-Tree 基础上的一种优化，使其更适合实现外存储索引结构。从 B-Tree 结构图中可以看到每个节点中不仅包含数据的 key 值，还有 data 值。而每一个页的存储空间是有限的，如果 data 数据较大时将会导致每个节点（即一个页）能存储的 key 的数量很小，当存储的数据量很大时同样会导致 B-Tree 的深度较大，增大查询时的磁盘 I/O 次数，进而影响查询效率。在 B+Tree 中，**所有数据记录节点都是按照键值大小顺序存放在同一层的叶子节点上**，而非叶子节点上只存储 key 值信息，这样可以大大加大每个节点存储的 key 值数量，降低 B+Tree 的高度。![](https://img.starfish.ink/mysql/MySQL-B%2BTree-store.png) +> +> - **节点存储内容**： +> - **B-Tree**：每个节点可以存储多个键和多个子节点的指针。键被存储在节点内部，并且可以被多次访问。 +> - **B+Tree**：非叶子节点仅存储键作为索引，不存储实际的数据记录，所以**适合放入内存中**。所有的数据记录都存储在叶子节点中，并且叶子节点被额外的指针连接在一起，形成一个有序链表。 +> +> - **查询性能**： +> - **B-Tree**：叶子节点可能包含数据记录，也可能不包含，取决于具体的实现。在查找特定值时，可以在找到对应节点时立即返回结果。 +> - **B+Tree**：所有的数据记录都存储在叶子节点中。在查找特定值时，需要访问叶子节点，但因为**叶子节点形成了有序链表**，所以**范围查询和顺序访问的性能更好**。 +> - **空间效率**： +> - **B-Tree**：因为每个节点存储更多的键和指针，所以每个节点可以有更少的子节点，树的高度可能会更小。 +> - **B+Tree**：每个内部节点可以有更多子节点，因为它们只存储键的索引，这使得树更宽、更浅，减少了树的高度，**提高了 I/O 效率**。 +> -### 劣势 -- 索引也是一张表，保存了主键和索引字段，并指向实体表的记录，所以也需要占用内存 -- 虽然索引大大提高了查询速度，同时却会降低更新表的速度，如对表进行INSERT、UPDATE和DELETE。 - 因为更新表时，MySQL不仅要保存数据，还要保存一下索引文件每次更新添加了索引列的字段， - 都会调整因为更新所带来的键值变化后的索引信息 +> #### B+Tree 性质 +> +> 在数据库中，B+ 树索引结构的高度 *h* 直接影响到进行一次索引查找所需的 I/O 次数。这是因为每次 I/O 操作通常只能读取一个磁盘块（或页）的数据。 +> +> - **B+ 树高度 h**： +> +> - B+树的高度是由树中节点的最大数量决定的，可以通过以下公式近似计算： $h≈logm(N)$ +> +> - 其中 *N* 是树中存储的总记录数，*m* 是每个磁盘块（页）可以存储的数据项数量。 +> +> 当数据量 N 一定的情况下，m 越大，h 越小；而 m = 磁盘块的大小 / 数据项的大小，磁盘块的大小也就是一个数据页的大小，是固定的（默认 16 KB），如果数据项占的空间越小，数据项的数量越多，树的高度越低。 +> +> 这就是为什么每个数据项，即索引字段要尽量的小，比如 int 占 4 字节，要比 bigint 8 字节少一半。这也是为什么 B+ 树要求把真实的数据放到叶子节点而不是内层节点，一旦放到内层节点，磁盘块的数据项会大幅度下降，导致树增高。当数据项等于1时将会退化成线性表。 +> +> > [!NOTE] +> > +> > 在实际应用中，B+树的高度和 I/O 次数会受到许多因素的影响，包括页的大小、数据的分布、索引的选择性等 +> +> 当 B+ 树的数据项是复合的数据结构，比如(name,age,sex) 的时候，B+ 树是按照从左到右的顺序来建立搜索树的，比如当(张三,20,F) 这样的数据来检索的时候，B+树会优先比较 name 来确定下一步的所搜方向，如果 name 相同再依次比较 age 和 sex，最后得到检索的数据；但当 (20,F) 这样的没有 name 的数据来的时候，B+ 树就不知道下一步该查哪个节点，因为建立搜索树的时候name 就是第一个**比较因子**，必须要先根据 name 来搜索才能知道下一步去哪里查询。比如当 (张三,F) 这样的数据来检索时，B+ 树可以用 name 来指定搜索方向，但下一个字段 age 的缺失，所以只能把名字等于张三的数据都找到，然后再匹配性别是 F 的数据了， 这个是非常重要的性质，就是我们说的**索引的最左匹配特性**。 -### MySQL索引分类 -#### 数据结构角度 +### 🎯 说下页分裂？ -- B+树索引 -- Hash索引 -- Full-Text全文索引 -- R-Tree索引 +> InnoDB 的索引是基于 B+ 树实现的，存储单元是 16KB 的页。当一个页的数据写满后，如果需要在其中间插入新数据，InnoDB 会申请新的页，把部分数据迁移过去，并更新父节点指针，这个过程就是“页分裂”。页分裂会带来 **性能开销**（申请页、搬迁数据、更新索引）和 **空间浪费**（页利用率下降），因此在设计表结构时，推荐使用 **自增主键**，减少离散插入，避免频繁页分裂。 -#### 从物理存储角度 +**1. 什么是“页”？** -- 聚集索引（clustered index） +- **InnoDB 的最小存储单元**是 **页（Page）**，默认大小是 **16KB**。 +- 一个索引（不管是聚簇索引还是二级索引），底层都是一个 **B+ 树**，每个节点就是一个 **页**。 -- 非聚集索引（non-clustered index），也叫辅助索引（secondary index） +**2. 什么是“页分裂”？** - 聚集索引和非聚集索引都是B+树结构 +- 当 **在一个页中插入数据时，空间不足**（16KB 填满了），就会触发 **页分裂**。 +- 页分裂过程： + 1. 申请一个新的页； + 2. 把原来的数据 **一部分迁移到新页**； + 3. 在父节点更新索引指针。 -#### 从逻辑角度 +👉 简单理解：一页塞不下了，就拆成两页，然后更新 B+树结构。 -- 主键索引：主键索引是一种特殊的唯一索引，不允许有空值 -- 普通索引或者单列索引：每个索引只包含单个列，一个表可以有多个单列索引 -- 多列索引（复合索引、联合索引）：复合索引指多个字段上创建的索引，只有在查询条件中使用了创建索引时的第一个字段，索引才会被使用。使用复合索引时遵循最左前缀集合 -- 唯一索引或者非唯一索引 -- 空间索引：空间索引是对空间数据类型的字段建立的索引，MYSQL中的空间数据类型有4种，分别是GEOMETRY、POINT、LINESTRING、POLYGON。 - MYSQL使用SPATIAL关键字进行扩展，使得能够用于创建正规索引类型的语法创建空间索引。创建空间索引的列，必须将其声明为NOT NULL，空间索引只能在存储引擎为MYISAM的表中创建 +**3. 页分裂的触发场景** +- **自增主键插入**（推荐方式）： + 插入的数据总是追加到页的末尾，几乎不会引发页分裂（因为新纪录总是在最后一页）。 +- **非自增/离散主键插入**： + 新数据可能插到页的中间，导致页中间腾挪空间，如果塞不下就分裂。 +- **二级索引插入**： + 由于二级索引按字段值排序，也可能导致中间插入，从而引发分裂。 +**4. 页分裂的代价** -> 为什么MySQL 索引中用B+tree，不用B-tree 或者其他树，为什么不用 Hash 索引 -> -> 聚簇索引/非聚簇索引，MySQL 索引底层实现，叶子结点存放的是数据还是指向数据的内存地址，使用索引需要注意的几个地方？ -> -> 使用索引查询一定能提高查询的性能吗？为什么? +- **性能损耗**： + - 页分裂需要申请新页、数据搬迁、更新父节点，属于一次 **重操作**。 + - 插入效率会下降。 +- **空间浪费**： + - 分裂后可能出现 **页利用率下降**（例如原来满页 16KB，分裂后两个页各只有 8KB）。 + - 长期频繁分裂 → 索引树更“高”，查询和维护成本上升。 -### MySQL索引结构 +**5. 页合并** -**首先要明白索引（index）是在存储引擎（storage engine）层面实现的，而不是server层面**。不是所有的存储引擎都支持所有的索引类型。即使多个存储引擎支持某一索引类型，它们的实现和行为也可能有所差别。 +- 与分裂相反，当删除大量数据后，页空间利用率太低时（小于 50%），InnoDB 会触发 **页合并**，把数据重新压缩到一起，减少浪费。 +- 页合并同样需要数据搬迁，也有开销。 -#### B+Tree索引 +**6. 如何减少页分裂？** -MyISAM 和 InnoDB 存储引擎，都使用 B+Tree的数据结构，它相对与 B-Tree结构，所有的数据都存放在叶子节点上，且把叶子节点通过指针连接到一起，形成了一条数据链表，以加快相邻数据的检索效率。 +1. **使用自增主键**：避免在索引中间频繁插入数据，最大限度减少分裂。 +2. **控制索引数量**：每个二级索引在写入时都可能分裂，减少不必要的索引。 +3. **合理设计字段类型**：让索引页能存放更多记录，降低分裂概率。 +4. **批量插入**：避免随机分散插入，尽量顺序写。 -**先了解下 B-Tree 和 B+Tree 的区别** -##### B-Tree -B-Tree是为磁盘等外存储设备设计的一种平衡查找树。 +### 🎯 聚集索引与非聚集索引的区别？ -系统从磁盘读取数据到内存时是以磁盘块（block）为基本单位的，位于同一个磁盘块中的数据会被一次性读取出来，而不是需要什么取什么。 +> MySQL 索引底层实现，叶子结点存放的是数据还是指向数据的内存地址? -InnoDB 存储引擎中有页（Page）的概念，页是其磁盘管理的最小单位。InnoDB 存储引擎中默认每个页的大小为16KB，可通过参数 `innodb_page_size` 将页的大小设置为 4K、8K、16K，在 MySQL 中可通过如下命令查看页的大小：`show variables like 'innodb_page_size';` +聚集索引（Clustered Index）和非聚集索引（Non-clustered Index）是数据库管理系统中常见的两种索引类型，是一种**数据存储方式**的区分，特别是在 MySQL 中。 -而系统一个磁盘块的存储空间往往没有这么大，因此 InnoDB 每次申请磁盘空间时都会是若干地址连续磁盘块来达到页的大小 16KB。InnoDB 在把磁盘数据读入到磁盘时会以页为基本单位，在查询数据时如果一个页中的每条数据都能有助于定位数据记录的位置，这将会减少磁盘I/O次数，提高查询效率。 +- 聚簇索引，也叫“**聚集索引**”，表示索引结构和数据一起存放的索引。 -B-Tree 结构的数据可以让系统高效的找到数据所在的磁盘块。为了描述 B-Tree，首先定义一条记录为一个二元组[key, data] ，key为记录的键值，对应表中的主键值，data 为一行记录中除主键外的数据。对于不同的记录，key值互不相同。 +- 非聚集索引是**索引结构和数据分开存放的索引**。 -一棵m阶的B-Tree有如下特性：  -1. 每个节点最多有m个孩子  -2. 除了根节点和叶子节点外，其它每个节点至少有Ceil(m/2)个孩子。 -3. 若根节点不是叶子节点，则至少有2个孩子  -4. 所有叶子节点都在同一层，且不包含其它关键字信息  -5. 每个非终端节点包含n个关键字信息（P0,P1,…Pn, k1,…kn）  -6. 关键字的个数n满足：ceil(m/2)-1 <= n <= m-1  -7. ki(i=1,…n)为关键字，且关键字升序排序 -8. Pi(i=1,…n)为指向子树根节点的指针。P(i-1)指向的子树的所有节点关键字均小于ki，但都大于k(i-1) +因为 InnoDB 默认存储引擎的原因，我们说这个一般指的是 InnoDB 中的聚集索引和非聚集索引 -B-Tree 中的每个节点根据实际情况可以包含大量的关键字信息和分支，如下图所示为一个 3 阶的 B-Tree： +**InnoDB 引擎索引结构的叶子节点的数据域，存放的就是实际的数据记录**（对于主索引，此处会存放表中所有的数据记录；对于辅助索引此处会引用主键，检索的时候通过主键到主键索引中找到对应数据行），或者说，**InnoDB 的数据文件本身就是主键索引文件**，这样的索引被称为"“**聚簇索引**”，一个表只能有一个聚簇索引。 -![索引](https://tva1.sinaimg.cn/large/007S8ZIlly1gg1de1fj9qj30ou08aaas.jpg) +- **InnoDB 聚集索引**：InnoDB 存储引擎使用聚集索引来存储主键列，并且所有非主键列都包含在聚集索引中，这意味着聚集索引实际上包含了整行数据。一个表只能有一个聚簇索引，通常是主键。 +- **InnoDB 非聚集索引**：InnoDB 的非聚集索引（也称为辅助索引）首先存储非主键索引列的值，然后通过主键列的值来查找对应的行。这种方式称为“索引的索引”，因为非聚集索引首先查找主键。 -每个节点占用一个盘块的磁盘空间，一个节点上有两个升序排序的关键字和三个指向子树根节点的指针，指针存储的是子节点所在磁盘块的地址。两个关键词划分成的三个范围域对应三个指针指向的子树的数据的范围域。以根节点为例，关键字为17和35，P1指针指向的子树的数据范围为小于17，P2指针指向的子树的数据范围为17~35，P3指针指向的子树的数据范围为大于35。 +![](https://img.starfish.ink/mysql/MySQL-secondary-index.png) -模拟查找关键字29的过程： -1. 根据根节点找到磁盘块1，读入内存。【磁盘I/O操作第1次】 -2. 比较关键字29在区间（17,35），找到磁盘块1的指针P2。 -3. 根据P2指针找到磁盘块3，读入内存。【磁盘I/O操作第2次】 -4. 比较关键字29在区间（26,30），找到磁盘块3的指针P2。 -5. 根据P2指针找到磁盘块8，读入内存。【磁盘I/O操作第3次】 -6. 在磁盘块8中的关键字列表中找到关键字29。 -分析上面过程，发现需要3次磁盘I/O操作，和3次内存查找操作。由于内存中的关键字是一个有序表结构，可以利用二分法查找提高效率。而3次磁盘I/O操作是影响整个B-Tree查找效率的决定因素。B-Tree相对于AVLTree缩减了节点个数，使每次磁盘I/O取到内存的数据都发挥了作用，从而提高了查询效率。 +### 🎯 非聚簇索引一定会回表查询吗? -##### B+Tree +不一定，这涉及到查询语句所要求的字段是否全部命中了索引，如果全部命中了索引，那么就不必再进行回表查询。 -B+Tree 是在 B-Tree 基础上的一种优化，使其更适合实现外存储索引结构，InnoDB 存储引擎就是用 B+Tree 实现其索引结构。 +举个简单的例子：假设我们在员工表的年龄上建立了索引，那么当进行的查询时,在索引的叶子节点上,已经包含了 age 信息，不会再次进行回表查询。 -从上一节中的B-Tree结构图中可以看到每个节点中不仅包含数据的key值，还有data值。而每一个页的存储空间是有限的，如果data数据较大时将会导致每个节点（即一个页）能存储的key的数量很小，当存储的数据量很大时同样会导致B-Tree的深度较大，增大查询时的磁盘I/O次数，进而影响查询效率。在B+Tree中，**所有数据记录节点都是按照键值大小顺序存放在同一层的叶子节点上**，而非叶子节点上只存储key值信息，这样可以大大加大每个节点存储的key值数量，降低B+Tree的高度。 +```sql +select age from employee where age < 20 +``` -B+Tree相对于B-Tree有几点不同： -1. 非叶子节点只存储键值信息； -2. 所有叶子节点之间都有一个链指针； -3. 数据记录都存放在叶子节点中 -将上一节中的B-Tree优化，由于B+Tree的非叶子节点只存储键值信息，假设每个磁盘块能存储4个键值及指针信息，则变成B+Tree后其结构如下图所示： -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gf3t57jvq1j30sc0aj0tj.jpg) +### 🎯 InnoDB 引擎中的索引策略，了解过吗？ -通常在B+Tree上有两个头指针，一个指向根节点，另一个指向关键字最小的叶子节点，而且所有叶子节点（即数据节点）之间是一种链式环结构。因此可以对B+Tree进行两种查找运算：一种是对于主键的范围查找和分页查找，另一种是从根节点开始，进行随机查找。 +InnoDB 索引策略主要包括以下几个方面: -可能上面例子中只有22条数据记录，看不出B+Tree的优点，下面做一个推算： +- 聚簇索引 +- 辅助索引，也就是非聚簇索引 +- 覆盖索引：查询可以直接通过索引获取所需数据，而无需回表查询 +- 前缀索引：用于对较长的字符串列进行索引，只索引字符串的前 N 个字符 +- 全文索引：用于对大文本字段进行全文检索 -InnoDB存储引擎中页的大小为16KB，一般表的主键类型为INT（占用4个字节）或BIGINT（占用8个字节），指针类型也一般为4或8个字节，也就是说一个页（B+Tree中的一个节点）中大概存储16KB/(8B+8B)=1K个键值（因为是估值，为方便计算，这里的K取值为10^3）。也就是说一个深度为3的B+Tree索引可以维护10^3 * 10^3 * 10^3 = 10亿 条记录。 +每种索引类型都有其独特的用途和优势，通过合理使用这些索引，可以显著提高数据库的查询性能。 -实际情况中每个节点可能不能填充满，因此在数据库中，B+Tree的高度一般都在2-4层。MySQL的InnoDB存储引擎在设计时是将根节点常驻内存的，也就是说查找某一键值的行记录时最多只需要1~3次磁盘I/O操作。 -B+Tree性质 -1. 通过上面的分析，我们知道IO次数取决于b+数的高度h，假设当前数据表的数据为N，每个磁盘块的数据项的数量是m，则有h=㏒(m+1)N，当数据量N一定的情况下，m越大，h越小；而m = 磁盘块的大小 / 数据项的大小，磁盘块的大小也就是一个数据页的大小，是固定的，如果数据项占的空间越小，数据项的数量越多，树的高度越低。这就是为什么每个数据项，即索引字段要尽量的小，比如int占4字节，要比bigint8字节少一半。这也是为什么b+树要求把真实的数据放到叶子节点而不是内层节点，一旦放到内层节点，磁盘块的数据项会大幅度下降，导致树增高。当数据项等于1时将会退化成线性表。 -2. 当b+树的数据项是复合的数据结构，比如(name,age,sex)的时候，b+数是按照从左到右的顺序来建立搜索树的，比如当(张三,20,F)这样的数据来检索的时候，b+树会优先比较name来确定下一步的所搜方向，如果name相同再依次比较age和sex，最后得到检索的数据；但当(20,F)这样的没有name的数据来的时候，b+树就不知道下一步该查哪个节点，因为建立搜索树的时候name就是第一个比较因子，必须要先根据name来搜索才能知道下一步去哪里查询。比如当(张三,F)这样的数据来检索时，b+树可以用name来指定搜索方向，但下一个字段age的缺失，所以只能把名字等于张三的数据都找到，然后再匹配性别是F的数据了， 这个是非常重要的性质，即**索引的最左匹配特性**。 +### 🎯 使用索引查询一定能提高查询的性能吗？为什么? +> 其实这个问题也对应的是哪些情况需要建立索引，哪些不需要 +> +> 使用索引需要注意的几个地方？ +使用索引查询并不一定总能提高查询性能 -##### MyISAM主键索引与辅助索引的结构 +**为什么使用索引通常能提高查询性能：** -MyISAM引擎的索引文件和数据文件是分离的。**MyISAM引擎索引结构的叶子节点的数据域，存放的并不是实际的数据记录，而是数据记录的地址**。索引文件与数据文件分离，这样的索引称为"**非聚簇索引**"。MyISAM的主索引与辅助索引区别并不大，只是主键索引不能有重复的关键字。 +1. 减少需要扫描的数据量来加快数据检索速度 + - 频繁作为查询条件的字段 + - 查询中与其他表关联的字段，外键关系建立索引 + - 在查询涉及的所有列都在索引中时，可以避免回表查询，提高查询效率 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gewoy5bddkj31bp0u04lv.jpg) +2. 索引可以显著加快 `ORDER BY` 和 `GROUP BY` 操作 -在MyISAM中，索引（含叶子节点）存放在单独的.myi文件中，叶子节点存放的是数据的物理地址偏移量（通过偏移量访问就是随机访问，速度很快）。 +**为什么在某些情况下索引查询反而可能降低性能**： -主索引是指主键索引，键值不可能重复；辅助索引则是普通索引，键值可能重复。 +1. 小表或低选择性列 -通过索引查找数据的流程：先从索引文件中查找到索引节点，从中拿到数据的文件指针，再到数据文件中通过文件指针定位了具体的数据。辅助索引类似。 + - **小表**：对于行数很少的表，索引带来的性能提升有限，因为全表扫描的开销也很小。索引反而增加了额外的维护开销。 + **低选择性列**：如果索引列的选择性很低（例如，性别列只有两个值 "M" 和 "F"），使用索引可能会导致大量的行扫描，无法显著减少数据量，索引的效果不明显。 +2. 经常增删改的表 -##### InnoDB主键索引与辅助索引的结构 + - **频繁的写操作**：索引不仅在读取数据时加速查询，还在插入、更新和删除操作时带来额外的开销。每次写操作都需要更新索引，索引越多，写操作的开销就越大。 -**InnoDB引擎索引结构的叶子节点的数据域，存放的就是实际的数据记录**（对于主索引，此处会存放表中所有的数据记录；对于辅助索引此处会引用主键，检索的时候通过主键到主键索引中找到对应数据行），或者说，**InnoDB的数据文件本身就是主键索引文件**，这样的索引被称为"“聚簇索引”，一个表只能有一个聚簇索引。 +3. 在高并发环境下，索引也可能导致锁竞争，影响查询性能 -###### 主键索引： +是否使用索引以及如何设计索引需要根据具体的查询模式、数据量、更新频率、硬件资源等多种因素综合考虑 -我们知道InnoDB索引是聚集索引，它的索引和数据是存入同一个.idb文件中的，因此它的索引结构是在同一个树节点中同时存放索引和数据，如下图中最底层的叶子节点有三行数据，对应于数据表中的id、stu_id、name数据项。 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gewoy2lhr5j320d0u016k.jpg) -在Innodb中，索引分叶子节点和非叶子节点，非叶子节点就像新华字典的目录，单独存放在索引段中，叶子节点则是顺序排列的，在数据段中。Innodb的数据文件可以按照表来切分（只需要开启`innodb_file_per_table)`，切分后存放在`xxx.ibd`中，默认不切分，存放在`xxx.ibdata`中。 +### 🎯 InnoDB 表为什么要建议用自增列做主键？ -###### 辅助（非主键）索引： +1. **聚簇索引优化**：自增 ID 的顺序插入让数据在磁盘连续存储，避免页拆分和碎片，提升 IO 效率； -这次我们以示例中学生表中的name列建立辅助索引，它的索引结构跟主键索引的结构有很大差别，在最底层的叶子结点有两行数据，第一行的字符串是辅助索引，按照ASCII码进行排序，第二行的整数是主键的值。 +2. **B + 树特性匹配**：插入时仅扩展树的最右节点，减少索引分裂开销，范围查询可利用顺序扫描； -这就意味着，对name列进行条件搜索，需要两个步骤： +3. **缓存与锁优化**：顺序插入的新记录使得最近插入的数据很可能在相邻的存储位置，这提高了缓存的命中率。 -① 在辅助索引上检索name，到达其叶子节点获取对应的主键； + - **缓存友好**：数据库缓存更容易缓存相邻的存储块，从而提高查询的性能，特别是在高并发的读写环境下。 -② 使用主键在主索引上再进行对应的检索操作 + -这也就是所谓的“**回表查询**” +### 🎯 你们建表会定义自增id么，为什么，自增id用完了怎么办? -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gewsc7l623j320r0u0gwt.jpg) +建表时通常会使用自增 ID 作为主键，因为它在写入性能、存储空间和索引效率上有显著优势。 +针对 ID 耗尽问题：首先评估数据规模，小表可重置自增值，核心业务表则升级为 BIGINT（理论可支撑 1.8e19 条数据）；分布式场景用雪花算法，高频删改表可设计 ID 回收机制。 -**InnoDB 索引结构需要注意的点** -1. 数据文件本身就是索引文件 +### 🎯 如何写 SQL 能够有效的使用到复合索引？ -2. 表数据文件本身就是按 B+Tree 组织的一个索引结构文件 -3. 聚集索引中叶节点包含了完整的数据记录 -4. InnoDB 表必须要有主键，并且推荐使用整型自增主键 +> MySQL高效索引 -正如我们上面介绍 InnoDB 存储结构，索引与数据是共同存储的，不管是主键索引还是辅助索引，在查找时都是通过先查找到索引节点才能拿到相对应的数据，如果我们在设计表结构时没有显式指定索引列的话，MySQL 会从表中选择数据不重复的列建立索引，如果没有符合的列，则 MySQL 自动为 InnoDB 表生成一个隐含字段作为主键，并且这个字段长度为6个字节，类型为整型。 +要有效地使用复合索引（也称为多列索引或组合索引），编写 SQL 查询时需要考虑以下几点： -> 那为什么推荐使用整型自增主键而不是选择UUID？ +1. **覆盖索引**（Covering Index），或者叫索引覆盖， 也就是平时所说的不需要回表操作 -- UUID是字符串，比整型消耗更多的存储空间； + - 如果查询的列完全包含在复合索引中，那么可以使用覆盖索引，这样可以避免回表查询，提高性能。 -- 在B+树中进行查找时需要跟经过的节点值比较大小，整型数据的比较运算比字符串更快速； +2. **最左前缀法则**： -- 自增的整型索引在磁盘中会连续存储，在读取一页数据时也是连续；UUID是随机产生的，读取的上下两行数据存储是分散的，不适合执行where id > 5 && id < 20的条件查询语句。 + - 复合索引的效率取决于查询条件是否遵循最左前缀法则，即从索引的最左边列开始匹配。 -- 在插入或删除数据时，整型自增主键会在叶子结点的末尾建立新的叶子节点，不会破坏左侧子树的结构；UUID主键很容易出现这样的情况，B+树为了维持自身的特性，有可能会进行结构的重构，消耗更多的时间。 + - 例如，如果你有一个 (`c1`, `c2`, `c3`) 的复合索引，那么以下查询可以高效地使用索引： + ```sql + SELECT * FROM table WHERE c1 = 'value1'; + SELECT * FROM table WHERE c1 = 'value1' AND c2 = 'value2'; + ``` -> 为什么非主键索引结构叶子节点存储的是主键值？ + - 如果查询条件不包含 `c1`，则该复合索引不会被使用。 -保证数据一致性和节省存储空间，可以这么理解：商城系统订单表会存储一个用户ID作为关联外键，而不推荐存储完整的用户信息，因为当我们用户表中的信息（真实名称、手机号、收货地址···）修改后，不需要再次维护订单表的用户数据，同时也节省了存储空间。 +3. **索引列的顺序**：在复合索引中，列的顺序很重要。应该将选择性最高的列（即不同值占总行数比例最高的列）放在前面。 +4. **使用索引列作为条件**：确保 WHERE 子句中的条件列与复合索引中的列相匹配，并且顺序正确。 +5. **避免使用函数和表达式**： -#### Hash索引 + - 如果在 WHERE 子句中对索引列应用了函数或计算，可能会使索引失效。 + - 例如，如果 `c1` 是索引的一部分，应避免 `WHERE UPPER(col1) = 'VALUE'`，而应使用 `WHERE c1 = 'value'`。 -- 主要就是通过Hash算法（常见的Hash算法有直接定址法、平方取中法、折叠法、除数取余法、随机数法），将数据库字段数据转换成定长的Hash值，与这条数据的行指针一并存入Hash表的对应位置；如果发生Hash碰撞（两个不同关键字的Hash值相同），则在对应Hash键下以链表形式存储。 +6. **范围查询和排序**： - 检索算法：在检索查询时，就再次对待查关键字再次执行相同的Hash算法，得到Hash值，到对应Hash表对应位置取出数据即可，如果发生Hash碰撞，则需要在取值时进行筛选。目前使用Hash索引的数据库并不多，主要有Memory等。 + - 复合索引可以用于涉及范围查询的 ORDER BY 和 GROUP BY 子句。 + - 例如，如果有一个 (`c1`, `c2`) 的索引，`ORDER BY c1, c2` 可以有效地使用索引。 - MySQL目前有Memory引擎和NDB引擎支持Hash索引。 +7. **限制索引的使用**： + - 使用 `LIKE` 操作符进行模糊匹配时，如果模式以通配符（`%`）开头，索引将不会被使用。 + - 例如，使用 `WHERE c1 LIKE '%value'` 将无法利用索引。 -#### full-text全文索引 +考虑查询的实际条件，如数据量大小、表的更新频率等，以确定是否真正需要复合索引。在实际的数据库环境中测试查询性能，并根据查询执行计划（`EXPLAIN`）来优化索引的使用 -- 全文索引也是MyISAM的一种特殊索引类型，主要用于全文索引，InnoDB从MYSQL5.6版本提供对全文索引的支持。 -- 它用于替代效率较低的LIKE模糊匹配操作，而且可以通过多字段组合的全文索引一次性全模糊匹配多个字段。 -- 同样使用B-Tree存放索引数据，但使用的是特定的算法，将字段数据分割后再进行索引（一般每4个字节一次分割），索引文件存储的是分割前的索引字符串集合，与分割后的索引信息，对应Btree结构的节点存储的是分割后的词信息以及它在分割前的索引字符串集合中的位置。 -#### R-Tree空间索引 +### 🎯 数据库不使用索引的几种可能？ -空间索引是MyISAM的一种特殊索引类型，主要用于地理空间数据类型 +> 上一个问题的反向问法 +1. **小型表或数据量少** + - 全表扫描速度快：对于小型表或数据量较少的表，全表扫描的速度可能与使用索引扫描的速度相当甚至更快，因为读取整个表所需的时间较短。 -> 为什么Mysql索引要用B+树不是B树？ + - 索引开销大：索引的创建和维护需要额外的存储空间和资源，对小型表来说，这些开销可能超过其带来的性能提升。 -用B+树不用B树考虑的是IO对性能的影响，B树的每个节点都存储数据，而B+树只有叶子节点才存储数据，所以查找相同数据量的情况下，B树的高度更高，IO更频繁。数据库索引是存储在磁盘上的，当数据量大时，就不能把整个索引全部加载到内存了，只能逐一加载每一个磁盘页（对应索引树的节点）。其中在MySQL底层对B+树进行进一步优化：在叶子节点中是双向链表，且在链表的头结点和尾节点也是循环指向的。 +2. **数据分布不均** + - 低选择性：如果某个列的值的重复率很高，例如性别列只有“男”和“女”两种值，使用索引的选择性很低，索引扫描可能比全表扫描更慢。 + - 数据倾斜：数据在某些特定值上高度集中，这种情况下，使用索引可能不会带来显著的性能提升。 -> 面试官：为何不采用Hash方式？ +3. **查询模式不适合** -因为Hash索引底层是哈希表，哈希表是一种以key-value存储数据的结构，所以多个数据在存储关系上是完全没有任何顺序关系的，所以，对于区间查询是无法直接通过索引查询的，就需要全表扫描。所以，哈希索引只适用于等值查询的场景。而B+ Tree是一种多路平衡查询树，所以他的节点是天然有序的（左子节点小于父节点、父节点小于右子节点），所以对于范围查询的时候不需要做全表扫描。 + - **范围查询**：对于范围查询，例如“BETWEEN”、“>”和“<”，使用索引的效率可能不如期望中的高，因为索引可能需要扫描较多的记录。 -哈希索引不支持多列联合索引的最左匹配规则，如果有大量重复键值得情况下，哈希索引的效率会很低，因为存在哈希碰撞问题。 + > 范围查询，不是一定不会使用索引，成本决定执行计划，优化器会首先针对可能使用到的二级索引划分几个扫描区间，然后分别调查这些区间内有多少条记录，在这些扫描区间内的二级索引记录的总和占总共的记录数量的比例达到某个值时，优化器将放弃使用二级索引执行查询，转而采用全表扫描 + - **LIKE 操作**：对于使用通配符“%”在前的LIKE查询（如“%value”），索引无法高效使用，因为数据库需要扫描整个表来找到匹配的值。 +4. **索引未命中** -### 哪些情况需要创建索引 + - **函数操作**：在查询条件中使用函数操作（如UPPER(column_name) = 'VALUE'）会导致索引无法被使用，因为索引存储的是原始数据。 -1. 主键自动建立唯一索引 + - **隐式类型转换**：如果列的数据类型与查询条件中的数据类型不一致（如列为整数，但条件中使用字符串），数据库可能进行隐式类型转换，导致索引失效。 -2. 频繁作为查询条件的字段 +5. **数据库优化器选择** -3. 查询中与其他表关联的字段，外键关系建立索引 + - **优化器策略**：数据库优化器根据统计信息和查询成本选择执行计划。在某些情况下，优化器可能判断全表扫描比索引扫描更高效。 -4. 单键/组合索引的选择问题，高并发下倾向创建组合索引 + - **统计信息不准确**：如果统计信息不准确或过期，优化器可能做出不合适的决策，选择不使用索引。 -5. 查询中排序的字段，排序字段通过索引访问大幅提高排序速度 +6. **索引维护成本** + - **高频更新**：对于高频插入、更新和删除操作的表，索引的维护成本可能较高，影响整体性能。在这种情况下，可能会选择不使用索引或减少索引的数量。 -6. 查询中统计或分组字段 +7. **多列索引的限制** + - **索引列顺序**：多列索引只有在按索引列顺序查询时才能被高效使用，如果查询条件中不包含索引的前导列，索引将无法使用。 + - **查询不完全匹配**：对于复合索引，如果查询条件不完全匹配索引定义，索引的使用效果可能不佳。 -### 哪些情况不要创建索引 -1. 表记录太少 -2. 经常增删改的表 -3. 数据重复且分布均匀的表字段，只应该为最经常查询和最经常排序的数据列建立索引（如果某个数据类包含太多的重复数据，建立索引没有太大意义） -4. 频繁更新的字段不适合创建索引（会加重IO负担） -5. where条件里用不到的字段不创建索引 +### 🎯 哪些情况会导致索引失效? +索引失效常见的情况包括：在索引列上使用函数/运算、隐式类型转换、OR 连接不同字段、模糊查询前缀 `%`、不满足最左前缀原则、使用 `!=`/`NOT IN`/`IS NOT NULL` 等、范围查询导致后续列失效、ORDER BY / GROUP BY 字段不符合索引顺序、以及数据量太少导致优化器选择全表扫描。 -### MySQL高效索引 +实际项目中，可以通过 **EXPLAIN** 分析执行计划，避免这些写法，并合理设计联合索引。 -**覆盖索引**（Covering Index）,或者叫索引覆盖， 也就是平时所说的不需要回表操作 -- 就是select的数据列只用从索引中就能够取得，不必读取数据行，MySQL可以利用索引返回select列表中的字段，而不必根据索引再次读取数据文件，换句话说**查询列要被所建的索引覆盖**。 -- 索引是高效找到行的一个方法，但是一般数据库也能使用索引找到一个列的数据，因此它不必读取整个行。毕竟索引叶子节点存储了它们索引的数据，当能通过读取索引就可以得到想要的数据，那就不需要读取行了。一个索引包含（覆盖）满足查询结果的数据就叫做覆盖索引。 +### 🎯 联合索引ABC，现在有个执行语句是 A = XXX and C < XXX，索引怎么走? -- **判断标准** +给定查询语句 `A = XXX AND C < XXX` 和联合索引 `(A, B, C)`，我们来分析索引的使用情况： - 使用explain，可以通过输出的extra列来判断，对于一个索引覆盖查询，显示为**using index**，MySQL查询优化器在执行查询前会决定是否有索引覆盖查询 +1. **精确匹配 `A`**： + - 由于条件 `A = XXX` 是精确匹配，第一个索引列 `A` 将被使用。 +2. **跳过 `B`**： + - 由于查询条件中没有涉及列 `B`，联合索引的第二列 `B` 将被跳过。 +3. **范围查询 `C`**： + - 条件 `C < XXX` 是范围查询。根据最左前缀法则和范围查询终止索引使用的规则，虽然条件 `C < XXX` 出现在查询中，但因为 `B` 没有出现在条件中，所以索引在 `C` 列上不能继续有效使用。 +在这个查询 `A = XXX AND C < XXX` 中，联合索引 `(A, B, C)` 只能部分使用，即只会使用索引的第一列 `A`，后续的 `B` 和 `C` 列将不会被索引利用。具体来说，执行计划会使用索引 `(A, B, C)` 中的 `(A)` 进行查找，然后对找到的记录进行筛选以满足 `C < XXX` 的条件。 -## 五、MySQL查询 -> count(*) 和 count(1)和count(列名)区别 ps：这道题说法有点多 +### 🎯 主键索引和唯一索引的区别？ -执行效果上： +主键索引是特殊的唯一索引，唯一索引查询会涉及到“回表”操作 -- count(*)包括了所有的列，相当于行数，在统计结果的时候，不会忽略列值为NULL -- count(1)包括了所有列，用1代表代码行，在统计结果的时候，不会忽略列值为NULL -- count(列名)只包括列名那一列，在统计结果的时候，会忽略列值为空（这里的空不是只空字符串或者0，而是表示null）的计数，即某个字段值为NULL时，不统计。 +| 特性 | 主键索引（Primary Key Index） | 唯一索引（Unique Index） | +| ------------- | ----------------------------- | ------------------------------------------------------------ | +| 唯一性 | 必须唯一 | 必须唯一（允许 NULL 值） | +| 是否允许 NULL | 不允许 | 允许多个 NULL 值「这里 NULL 的定义 ，是指 未知值。 所以多个 NULL ，都是未知的」 | +| 聚簇索引 | 是（在 InnoDB 中） | 否（除非是主键） | +| 每个表的数量 | 只能有一个 | 可以有多个 | +| 主要用途 | 唯一标识每一行 | 强制唯一性约束，非主键用途 | +| 创建语法 | `PRIMARY KEY` | `UNIQUE` | -执行效率上： +```sql +CREATE TABLE example ( + id INT AUTO_INCREMENT PRIMARY KEY, + email VARCHAR(100) UNIQUE +); +INSERT INTO example (email) VALUES (NULL), (NULL), (NULL); +``` -- 列名为主键，count(列名)会比count(1)快 -- 列名不为主键，count(1)会比count(列名)快 -- 如果表多个列并且没有主键，则 count(1) 的执行效率优于 count(*) -- 如果有主键，则 select count（主键）的执行效率是最优的 -- 如果表只有一个字段，则 select count(*) 最优。 +### 🎯 索引下推？ -> MySQL中 in和 exists 的区别？ +> **索引下推（ICP）是 MySQL 5.6 引入的一项优化**，允许在存储引擎层就利用索引字段做部分 `WHERE` 条件过滤，从而减少回表次数，降低 I/O。 +> 例如 `name LIKE 'J%' AND age=20`，如果有 `(name, age)` 联合索引，MySQL 会在存储引擎层就过滤 `age=20`，避免回表后再判断。 +> ICP 只对二级索引有效，可以通过 `EXPLAIN` 的 `Using index condition` 来确认是否生效。 -- exists：exists对外表用loop逐条查询，每次查询都会查看exists的条件语句，当exists里的条件语句能够返回记录行时（无论记录行是的多少，只要能返回），条件就为真，返回当前loop到的这条记录；反之，如果exists里的条件语句不能返回记录行，则当前loop到的这条记录被丢弃，exists的条件就像一个bool条件，当能返回结果集则为true，不能返回结果集则为false -- in：in查询相当于多个or条件的叠加 +**索引下推（Index Condition Pushdown，ICP）** 是 MySQL 5.6 中引入的一种优化技术，用于提升范围查询或排序查询的性能。通过索引下推，MySQL 可以减少不必要的表数据行访问，加快查询速度。 -```mysql -SELECT * FROM A WHERE A.id IN (SELECT id FROM B); -SELECT * FROM A WHERE EXISTS (SELECT * from B WHERE B.id = A.id); -``` +**1. 索引下推的由来** -**如果查询的两个表大小相当，那么用in和exists差别不大**。 +在 **没有 ICP 之前**： -如果两个表中一个较小，一个是大表，则子查询表大的用exists，子查询表小的用in： +- MySQL 在通过索引扫描时，先拿到索引里的主键 ID； +- 然后回表到数据页取出完整的行； +- 最后在 **Server 层**做条件过滤。 +这样即使很多行最终不满足条件，也必须 **回表取出整行数据**，造成 I/O 浪费。 +**2. 什么是索引下推？** -> UNION和UNION ALL的区别? +- **索引下推**就是把部分 `WHERE` 条件判断下推到 **存储引擎层**，利用索引本身就能获取到的字段提前过滤数据。 +- 这样可以在**回表之前就过滤掉不必要的行**，减少回表次数，提高查询效率。 -UNION和UNION ALL都是将两个结果集合并为一个，**两个要联合的SQL语句 字段个数必须一样，而且字段类型要“相容”（一致）；** +**3. 举个例子** -- UNION在进行表连接后会筛选掉重复的数据记录（效率较低），而UNION ALL则不会去掉重复的数据记录； +假设有表： -- UNION会按照字段的顺序进行排序，而UNION ALL只是简单的将两个结果合并就返回； +```mysql +CREATE TABLE user ( + id INT PRIMARY KEY, + name VARCHAR(50), + age INT, + KEY idx_name_age (name, age) +); +``` +查询： +```mysql +SELECT * FROM user WHERE name LIKE 'J%' AND age = 20; +``` -### SQL执行顺序 +**没有 ICP 的情况：** -- 手写 +- `name LIKE 'J%'` 用到索引（范围扫描）。 +- 但 `age = 20` 只能在回表后做判断。 +- 即使扫描出来 10 万行 `name LIKE 'J%'`，也得回表逐条验证 `age`。 - ```mysql - SELECT DISTINCT - FROM - JOIN ON - WHERE - GROUP BY - HAVING - ORDER BY - LIMIT - ``` +**有 ICP 的情况：** -- 机读 +- MySQL 把 `age = 20` 条件下推到存储引擎层。 +- 存储引擎在扫描 `idx_name_age` 索引时，就能直接判断 `age`。 +- 这样可能只剩下几百行需要回表，大大减少 I/O。 - ```mysql - FROM - ON - JOIN - WHERE - GROUP BY - HAVING - SELECT - DISTINCT - ORDER BY - LIMIT - ``` +**4. 哪些场景能触发 ICP？** -- 总结 +- 只对 **二级索引** 有效（主键索引存的就是全字段，不需要回表）。 +- 查询条件中，**部分字段能利用索引顺序**，部分不能利用。 +- 不支持的情况：某些复杂表达式、函数计算等。 - ![sql-parse](https://tva1.sinaimg.cn/large/007S8ZIlly1gf3t8jyy81j30s2083wg2.jpg) +**5. 如何确认是否使用了 ICP？** - +执行 `EXPLAIN`，如果 `Extra` 列里有： -> mysql 的内连接、左连接、右连接有什么区别？ -> -> 什么是内连接、外连接、交叉连接、笛卡尔积呢？ +```mysql +Using index condition +``` -### Join图 +说明 MySQL 启用了索引下推。 -![sql-joins](https://tva1.sinaimg.cn/large/007S8ZIlly1gf3t8novxpj30qu0l4wi7.jpg) +**索引下推的优势** ------- +- **减少回表次数**：通过提前过滤不符合条件的记录，减少回表操作。 +- **提升查询效率**：尤其是当索引列上的范围条件命中大量记录，而回表的记录较少时，索引下推可以显著减少不必要的 IO 操作。 +------ -## 六、MySQL 事务 -> 事务的隔离级别有哪些？MySQL的默认隔离级别是什么？ -> -> 什么是幻读，脏读，不可重复读呢？ -> -> MySQL事务的四大特性以及实现原理 -> -> MVCC熟悉吗，它的底层原理？ +## 四、事务与锁机制 🔒 MySQL 事务主要用于处理操作量大，复杂度高的数据。比如说，在人员管理系统中，你删除一个人员，你即需要删除人员的基本资料，也要删除和该人员相关的信息，如信箱，文章等等，这样，这些数据库操作语句就构成一个事务！ +### 🎯 什么是事务？事务有哪些特性？ - -### ACID — 事务基本要素 - -![](https://tva1.sinaimg.cn/large/007S8ZIlly1geu10kkswnj305q05mweo.jpg) - -事务是由一组SQL语句组成的逻辑处理单元，具有4个属性，通常简称为事务的ACID属性。 +事务是由一组 SQL 语句组成的逻辑处理单元，具有 4 个属性，通常简称为事务的 ACID 属性。 - **A (Atomicity) 原子性**：整个事务中的所有操作，要么全部完成，要么全部不完成，不可能停滞在中间某个环节。事务在执行过程中发生错误，会被回滚（Rollback）到事务开始前的状态，就像这个事务从来没有执行过一样 - **C (Consistency) 一致性**：在事务开始之前和事务结束以后，数据库的完整性约束没有被破坏 @@ -659,265 +847,383 @@ MySQL 事务主要用于处理操作量大，复杂度高的数据。比如说 -**并发事务处理带来的问题** - -- 更新丢失（Lost Update)： 事务A和事务B选择同一行，然后基于最初选定的值更新该行时，由于两个事务都不知道彼此的存在，就会发生丢失更新问题 -- 脏读(Dirty Reads)：事务A读取了事务B更新的数据，然后B回滚操作，那么A读取到的数据是脏数据 -- 不可重复读（Non-Repeatable Reads)：事务 A 多次读取同一数据，事务B在事务A多次读取的过程中，对数据作了更新并提交，导致事务A多次读取同一数据时，结果不一致。 -- 幻读（Phantom Reads)：幻读与不可重复读类似。它发生在一个事务A读取了几行数据，接着另一个并发事务B插入了一些数据时。在随后的查询中，事务A就会发现多了一些原本不存在的记录，就好像发生了幻觉一样，所以称为幻读。 - - - -**幻读和不可重复读的区别：** - -- **不可重复读的重点是修改**：在同一事务中，同样的条件，第一次读的数据和第二次读的数据不一样。（因为中间有其他事务提交了修改） -- **幻读的重点在于新增或者删除**：在同一事务中，同样的条件,，第一次和第二次读出来的记录数不一样。（因为中间有其他事务提交了插入/删除） - - - -**并发事务处理带来的问题的解决办法：** +### 🎯 什么是脏读、不可重复读和幻读？ -- “更新丢失”通常是应该完全避免的。但防止更新丢失，并不能单靠数据库事务控制器来解决，需要应用程序对要更新的数据加必要的锁来解决，因此，防止更新丢失应该是应用的责任。 +**并发事务处理带来的问题** -- “脏读” 、 “不可重复读”和“幻读” ，其实都是数据库读一致性问题，必须由数据库提供一定的事务隔离机制来解决： +- 更新丢失（Lost Update)： 事务 A 和事务 B 选择同一行，然后基于最初选定的值更新该行时，由于两个事务都不知道彼此的存在，就会发生丢失更新问题 +- 脏读(Dirty Reads)：事务 A 读取了事务 B 未提交的数据，然后 B 回滚操作，那么 A 读取到的数据是脏数据 +- 不可重复读（Non-Repeatable Reads)：事务 A 多次读取同一数据，事务 B 在事务 A 多次读取的过程中，对数据作了更新并提交，导致事务 A 多次读取同一数据时，结果不一致。 +- 幻读（Phantom Reads)：幻读与不可重复读类似。它发生在一个事务 A 读取了几行数据，接着另一个并发事务 B 插入了一些数据时。在随后的查询中，事务 A 就会发现多了一些原本不存在的记录，就好像发生了幻觉一样，所以称为幻读。 - - 一种是加锁：在读取数据前，对其加锁，阻止其他事务对数据进行修改。 - - 另一种是数据多版本并发控制（MultiVersion Concurrency Control，简称 **MVCC** 或 MCC），也称为多版本数据库：不用加任何锁， 通过一定机制生成一个数据请求时间点的一致性数据快照 （Snapshot)， 并用这个快照来提供一定级别 （语句级或事务级） 的一致性读取。从用户的角度来看，好象是数据库可以提供同一数据的多个版本。 +> **幻读和不可重复读的区别：** +> +> - **不可重复读的重点是修改**：在同一事务中，同样的条件，第一次读的数据和第二次读的数据不一样。（因为中间有其他事务提交了修改） +> - **幻读的重点在于新增或者删除**：在同一事务中，同样的条件,，第一次和第二次读出来的记录数不一样。（因为中间有其他事务提交了插入/删除） -### 事务隔离级别 +### 🎯 MySQL 支持哪些事务隔离级别？各有什么区别？ -数据库事务的隔离级别有4种，由低到高分别为 +MySQL 支持四种事务隔离级别，由低到高分别为： -- **READ-UNCOMMITTED(读未提交)：** 最低的隔离级别，允许读取尚未提交的数据变更，**可能会导致脏读、幻读或不可重复读**。 -- **READ-COMMITTED(读已提交)：** 允许读取并发事务已经提交的数据，**可以阻止脏读，但是幻读或不可重复读仍有可能发生**。 -- **REPEATABLE-READ(可重复读)：** 对同一字段的多次读取结果都是一致的，除非数据是被本身事务自己所修改，**可以阻止脏读和不可重复读，但幻读仍有可能发生**。 +- **READ UNCOMMITTED(读未提交)：** 最低的隔离级别，允许读取尚未提交的数据变更，**可能会导致脏读、幻读或不可重复读**。 +- **READ COMMITTED(读已提交)：** 允许读取并发事务已经提交的数据，**可以阻止脏读，但是幻读或不可重复读仍有可能发生**。 +- **REPEATABLE READ(可重复读)：** 对同一字段的多次读取结果都是一致的，除非数据是被本身事务自己所修改，**可以阻止脏读和不可重复读，但幻读仍有可能发生**。 - **SERIALIZABLE(可串行化)：** 最高的隔离级别，完全服从ACID的隔离级别。所有的事务依次逐个执行，这样事务之间就完全不可能产生干扰，也就是说，**该级别可以防止脏读、不可重复读以及幻读**。 +> **简单点的理解** +> +> 读未提交：别人改数据的事务尚未提交，我在我的事务中也能读到。 +> +> 读已提交：别人改数据的事务已经提交，我在我的事务中才能读到。 +> +> 可重复读：别人改数据的事务已经提交，我在我的事务中也不去读。 +> +> 串行：我的事务尚未提交，别人就别想改数据。 + 查看当前数据库的事务隔离级别： ```mysql show variables like 'tx_isolation' ``` -下面通过事例一一阐述在事务的并发操作中可能会出现脏读，不可重复读，幻读和事务隔离级别的联系。 -数据库的事务隔离越严格，并发副作用越小，但付出的代价就越大，因为事务隔离实质上就是使事务在一定程度上“串行化”进行，这显然与“并发”是矛盾的。同时，不同的应用对读一致性和事务隔离程度的要求也是不同的，比如许多应用对“不可重复读”和“幻读”并不敏感，可能更关心数据并发访问的能力。 -#### Read uncommitted +### 🎯 MySQL 如何实现事务隔离 | 并发事务处理带来的问题的解决办法 -读未提交，就是一个事务可以读取另一个未提交事务的数据。 +- “更新丢失” 通常是应该完全避免的。但防止更新丢失，并不能单靠数据库事务控制器来解决，需要应用程序对要更新的数据加必要的锁来解决，因此，防止更新丢失应该是应用的责任。 -事例：老板要给程序员发工资，程序员的工资是3.6万/月。但是发工资时老板不小心按错了数字，按成3.9万/月，该钱已经打到程序员的户口，但是事务还没有提交，就在这时，程序员去查看自己这个月的工资，发现比往常多了3千元，以为涨工资了非常高兴。但是老板及时发现了不对，马上回滚差点就提交了的事务，将数字改成3.6万再提交。 - -分析：实际程序员这个月的工资还是3.6万，但是程序员看到的是3.9万。他看到的是老板还没提交事务时的数据。这就是脏读。 +- “脏读” 、 “不可重复读”和“幻读” ，其实都是数据库读一致性问题，必须由数据库提供一定的事务隔离机制来解决： -那怎么解决脏读呢？Read committed！读提交，能解决脏读问题。 + - 一种是加**锁**：在读取数据前，对其加锁，阻止其他事务对数据进行修改。MySQL 通过行级锁和表级锁来管理并发访问。行级锁包括共享锁（读锁）和排他锁（写锁），表级锁包括意向锁和元数据锁。 + - 另一种是数据**多版本并发控制**（MultiVersion Concurrency Control，简称 **MVCC**），也称为多版本数据库：不用加任何锁， 通过一定机制生成一个数据请求时间点的一致性数据快照 （Snapshot)， 并用这个快照来提供一定级别 （语句级或事务级） 的一致性读取。从用户的角度来看，好象是数据库可以提供同一数据的多个版本。 -#### Read committed -读提交，顾名思义，就是一个事务要等另一个事务提交后才能读取数据。 -事例：程序员拿着信用卡去享受生活（卡里当然是只有3.6万），当他埋单时（程序员事务开启），收费系统事先检测到他的卡里有3.6万，就在这个时候！！程序员的妻子要把钱全部转出充当家用，并提交。当收费系统准备扣款时，再检测卡里的金额，发现已经没钱了（第二次检测金额当然要等待妻子转出金额事务提交完）。程序员就会很郁闷，明明卡里是有钱的… +MySQL InnoDB 存储引擎的默认支持的隔离级别是 **REPEATABLE-READ（可重读）**。我们可以通过`SELECT @@tx_isolation;`命令来查看，MySQL 8.0 该命令改为`SELECT @@transaction_isolation;` -分析：这就是读提交，若有事务对数据进行更新（UPDATE）操作时，读操作事务要等待这个更新操作事务提交后才能读取数据，可以解决脏读问题。但在这个事例中，出现了一个事务范围内两个相同的查询却返回了不同数据，这就是**不可重复读**。 +这里需要注意的是：与 SQL 标准不同的地方在于 InnoDB 存储引擎在 **REPEATABLE-READ（可重读）**事务隔离级别下**使用的是 Next-Key Lock 算法，因此可以避免幻读的产生**，这与其他数据库系统(如 SQL Server)是不同的。所以说 InnoDB 存储引擎的默认支持的隔离级别是 REPEATABLE-READ（可重读）已经可以完全保证事务的隔离性要求，即达到了 SQL标准的 **SERIALIZABLE(可串行化)**隔离级别，而且保留了比较好的并发性能。 -那怎么解决可能的不可重复读问题？Repeatable read ！ +> **Next-Key Locks**：MySQL通过在索引上的间隙加锁（Gap Lock），结合行锁，形成所谓的Next-Key锁，锁定一个范围。这样即使在事务运行期间，其他事务也无法在该范围内插入新的行，从而避免了幻读 -#### Repeatable read +因为隔离级别越低，事务请求的锁越少，所以大部分数据库系统的隔离级别都是**READ-COMMITTED(读已提交):**，但是你要知道的是InnoDB 存储引擎默认使用 **REPEATABLE-READ（可重读）**并不会有任何性能损失。 -重复读，就是在开始读取数据（事务开启）时，不再允许修改操作。 **MySQL的默认事务隔离级别** +> 加锁的基本原则（RR隔离级别下） +> +> - 原则1：加锁的对象是 next-key lock。（是一个前开后闭的区间） +> - 原则2：查找过程中访问到的对象才加锁 +> +> 优化1：唯一索引加锁时，next-key lock 退化为行锁。 +> 索引上的等值查询，向右遍历时最后一个不满足等值条件的时候，next-key lock 退化为间隙锁 +> 唯一索引和普通索引在范围查询的时候 都会访问到不满足条件的第一个值为止 -事例：程序员拿着信用卡去享受生活（卡里当然是只有3.6万），当他埋单时（事务开启，不允许其他事务的UPDATE修改操作），收费系统事先检测到他的卡里有3.6万。这个时候他的妻子不能转出金额了。接下来收费系统就可以扣款了。 -分析：重复读可以解决不可重复读问题。写到这里，应该明白的一点就是，**不可重复读对应的是修改，即UPDATE操作。但是可能还会有幻读问题。因为幻读问题对应的是插入INSERT操作，而不是UPDATE操作**。 -**什么时候会出现幻读？** +### 🎯 MVCC 熟悉吗，它的底层原理？ -事例：程序员某一天去消费，花了2千元，然后他的妻子去查看他今天的消费记录（全表扫描FTS，妻子事务开启），看到确实是花了2千元，就在这个时候，程序员花了1万买了一部电脑，即新增INSERT了一条消费记录，并提交。当妻子打印程序员的消费记录清单时（妻子事务提交），发现花了1.2万元，似乎出现了幻觉，这就是幻读。 +> MVCC 是 InnoDB 实现读写并发控制的核心机制，通过 **隐藏列、Undo Log、Read View** 来维护多版本数据。 +> 在快照读时，事务会根据自己的 Read View，决定看到哪个版本。 +> +> - 优点：实现了 **非阻塞读**，大幅提升并发性能。 +> - 适用：主要用于 RC 和 RR 两个隔离级别。 +> 本质上，MVCC 是“**读旧版本数据来避免读写冲突**”。 -那怎么解决幻读问题？Serializable！ +**1. 定义与作用** + MVCC（Multi-Version Concurrency Control，多版本并发控制）是 MySQL InnoDB 引擎用来实现 **高并发下的读写一致性** 的机制。 -#### Serializable 序列化 +- 作用：保证 **读写并发** 时，读不会被写阻塞（非阻塞读），提升性能。 +- 核心思想：同一行数据在不同时间点可能有多个版本，读操作可以“读历史版本”，而不是等写操作完成。 -Serializable 是最高的事务隔离级别，在该级别下，事务串行化顺序执行，可以避免脏读、不可重复读与幻读。简单来说，Serializable会在读取的每一行数据上都加锁，所以可能导致大量的超时和锁争用问题。这种事务隔离级别效率低下，比较耗数据库性能，一般不使用。 +**2. 底层原理**（InnoDB 实现） +MVCC 主要是借助于版本链来实现的。InnoDB 引擎通过回滚指针，将数据的不同版本串联在一起，也就是版本链。这些串联起来的历史版本，被放到了 undolog 里面。当某一个事务发起查询的时候，MVCC 会根据事务的隔离级别来生成不同的 Read View，从而控制事务查询最终得到的结果。 +MVCC 依赖三个关键机制： -#### 比较 +① **隐藏列**（存储事务信息），InnoDB 在每行记录后面有两个隐藏列： -| 事务隔离级别 | 读数据一致性 | 脏读 | 不可重复读 | 幻读 | -| ---------------------------- | ---------------------------------------- | ---- | ---------- | ---- | -| 读未提交（read-uncommitted） | 最低级被，只能保证不读取物理上损坏的数据 | 是 | 是 | 是 | -| 读已提交（read-committed） | 语句级 | 否 | 是 | 是 | -| 可重复读（repeatable-read） | 事务级 | 否 | 否 | 是 | -| 串行化（serializable） | 最高级别，事务级 | 否 | 否 | 否 | +- `trx_id`：最后一次修改该行的事务 ID。 +- `roll_pointer`：指向 Undo Log 的指针，用于找到修改前的旧版本。 -需要说明的是，事务隔离级别和数据访问的并发性是对立的，事务隔离级别越高并发性就越差。所以要根据具体的应用来确定合适的事务隔离级别，这个地方没有万能的原则。 +> InnoDB下的Compact行结构，有三个隐藏的列 +> +> | 列名 | 是否必须 | 描述 | +> | -------------- | -------- | ------------------------------------------------------------ | +> | row_id | 否 | 行ID，唯一标识一条记录（如果定义主键，它就没有啦） | +> | transaction_id | 是 | 事务ID | +> | roll_pointer | 是 | DB_ROLL_PTR是一个回滚指针，用于配合undo日志，指向上一个旧版本 | -MySQL InnoDB 存储引擎的默认支持的隔离级别是 **REPEATABLE-READ（可重读）**。我们可以通过`SELECT @@tx_isolation;`命令来查看，MySQL 8.0 该命令改为`SELECT @@transaction_isolation;` +② **Undo Log（回滚日志）** -这里需要注意的是：与 SQL 标准不同的地方在于InnoDB 存储引擎在 **REPEATABLE-READ（可重读）**事务隔离级别下使用的是Next-Key Lock 算法，因此可以避免幻读的产生，这与其他数据库系统(如 SQL Server)是不同的。所以说InnoDB 存储引擎的默认支持的隔离级别是 REPEATABLE-READ（可重读）已经可以完全保证事务的隔离性要求，即达到了 SQL标准的 **SERIALIZABLE(可串行化)**隔离级别，而且保留了比较好的并发性能。 +- 当事务修改数据时，InnoDB 会先把“修改前的旧值”写入 Undo Log。 +- 这样即使数据被更新，仍能通过 Undo Log 找到历史版本。 -因为隔离级别越低，事务请求的锁越少，所以大部分数据库系统的隔离级别都是**READ-COMMITTED(读已提交):**，但是你要知道的是InnoDB 存储引擎默认使用 **REPEATABLE-READ（可重读）**并不会有任何性能损失。 +③ **Read View（读视图）** +- 在 **快照读（普通 SELECT）** 时生成，保存当前活跃事务的 ID 列表。 +- 通过对比 `trx_id` 与 `Read View`，决定当前事务能看到哪一个版本的数据。 +👉 简单来说： -### MVCC 多版本并发控制 +- **读操作** → 根据 Read View 规则，选择可见版本（可能是 Undo Log 里的旧版本）。 +- **写操作** → 新增一个版本（更新 `trx_id`、写 Undo Log），而不是覆盖旧版本。 -MySQL的大多数事务型存储引擎实现都不是简单的行级锁。基于提升并发性考虑，一般都同时实现了多版本并发控制（MVCC），包括Oracle、PostgreSQL。只是实现机制各不相同。 +**3. 可见性规则（简化版）** + 判断某条记录的版本是否对当前事务可见： -可以认为 MVCC 是行级锁的一个变种，但它在很多情况下避免了加锁操作，因此开销更低。虽然实现机制有所不同，但大都实现了非阻塞的读操作，写操作也只是锁定必要的行。 +1. 如果 `trx_id < min_trx_id`（最小活跃事务 ID），说明版本在当前事务前提交 → 可见。 +2. 如果 `trx_id > max_trx_id`（最大已分配事务 ID），说明版本在当前事务开始后生成 → 不可见。 +3. 如果 `trx_id` 在活跃事务列表里 → 不可见。 +4. 其他情况 → 可见。 -MVCC 的实现是通过保存数据在某个时间点的快照来实现的。也就是说不管需要执行多长时间，每个事物看到的数据都是一致的。 +**4. MVCC 适用的隔离级别** -典型的MVCC实现方式，分为**乐观（optimistic）并发控制和悲观（pressimistic）并发控制**。下边通过 InnoDB的简化版行为来说明 MVCC 是如何工作的。 +- **读已提交（RC）**：每次查询生成新的 Read View，可能看到最新提交的数据。 +- **可重复读（RR）**：一个事务里只生成一次 Read View，多次查询看到的数据一致。 +- **未提交读（RU）** 和 **串行化** 不走 MVCC（RU 直接读最新，串行化加锁）。 -InnoDB 的 MVCC，是通过在每行记录后面保存两个隐藏的列来实现。这两个列，一个保存了行的创建时间，一个保存行的过期时间（删除时间）。当然存储的并不是真实的时间，而是系统版本号（system version number）。每开始一个新的事务，系统版本号都会自动递增。事务开始时刻的系统版本号会作为事务的版本号，用来和查询到的每行记录的版本号进行比较。 +> **核心概念** +> +> 1. **快照读（Snapshot Read）**： +> - 每个事务在开始时，会获取一个数据快照，事务在读取数据时，总是读取该快照中的数据。 +> - 这意味着即使在事务进行期间，其他事务对数据的更新也不会影响当前事务的读取。 +>2. **版本链（Version Chain）**： +> - 每个数据行都有多个版本，每个版本包含数据和元数据（如创建时间、删除时间等）。 +> - 新版本的数据行会被链接到旧版本的数据行，形成一个版本链。 +> 3. **隐式锁（Implicit Locking）**： +> - MVCC 通过版本管理避免了显式锁定，减少了锁争用问题。 +> - 对于读取操作，事务读取其开始时的快照数据，不会被写操作阻塞。 +> +> MySQL InnoDB 存储引擎使用 MVCC 来实现可重复读（REPEATABLE READ）隔离级别，避免脏读、不可重复读和幻读问题。具体机制如下： +> +> 1. **隐藏列**： +> - InnoDB 在每行记录中存储两个隐藏列：`trx_id`（事务ID，也叫做事务版本号）和`roll_pointer`（回滚指针）。 +> - `trx_id` 记录最后一次修改该行的事务ID，`roll_pointer` 指向该行的上一版本，把每一行的历史版本串联在一起。 +>2. **Undo日志**： +> - 每次数据更新时，InnoDB 会在 Undo 日志中记录旧版本数据。 +> - 如果需要读取旧版本数据，InnoDB 会通过 `roll_pointer` 找到 Undo 日志中的旧版本。 +> 3. **一致性视图（Consistent Read View）**： +> - InnoDB 为每个事务创建一致性视图，记录当前活动的所有事务ID。 +> - 读取数据时，会根据一致性视图决定哪些版本的数据对当前事务可见。 + +> #### MVCC 解决了什么问题 +> +> - 读写不冲突，极大的增加了系统的并发性能 +> +> - 解决脏读，幻读，不可重复读 等问题（注:其实多版本只是解决不可重复读问题，而加上临界（**next-key**）锁(也就是它这里所谓的并发控制)才解决了幻读问题。） +> +> **Next-Key Locking** 是行锁和间隙锁的组合，它锁定的是索引记录以及其附近的间隙，防止其他事务在查询范围内插入或删除记录，确保当前事务在后续的查询中不会看到“幻影”。 +> +> 举例：如果事务A查询了范围 `WHERE age BETWEEN 18 AND 25`，Next-Key Locking 会锁定已经存在的记录以及这个范围的间隙，防止其他事务在这个范围内插入新数据 -**REPEATABLE READ（可重读）隔离级别下MVCC如何工作：** +>在可重复读（Repeatable Read）隔离级别下，MySQL的InnoDB存储引擎通过多版本并发控制（MVCC）和next-key锁机制在很大程度上减少了幻读的发生，但并没有完全消除幻读的可能性。 +> +>1. **快照读（Snapshot Read）**：在可重复读隔离级别下，InnoDB使用MVCC来处理普通的SELECT查询。MVCC通过为每个事务创建一个Read View，使得事务在其执行期间看到的是一致的快照数据，即使其他事务在这段时间内插入了新记录，当前事务的查询也不会看到这些新记录。这通常可以避免幻读。 +>2. **当前读（Current Read）**：对于需要锁定结果集的查询，如SELECT ... FOR UPDATE或SELECT ... IN SHARE MODE，InnoDB使用next-key锁，这是记录锁和间隙锁的组合。这种锁机制可以防止其他事务在锁定的范围内插入新记录，从而避免幻读。 +> +>然而，根据搜索结果中的讨论，即使在可重复读隔离级别下，仍然存在一些特殊情况可能导致幻读： +> +>- 如果事务在执行快照读之后，立即执行更新操作，可能会看到其他事务插入的新记录，因为更新操作会检查最新版本的数据。 +>- 如果事务在执行当前读之前，其他事务已经提交了插入操作，那么当前读可能会观察到这些新记录。 +> +>因此，虽然InnoDB的可重复读隔离级别提供了强有力的幻读保护，但在某些特殊情况下，幻读仍然可能发生。开发者需要了解这些情况，并在必要时采取额外的措施来确保数据的一致性和隔离性。 -- SELECT - InnoDB会根据以下两个条件检查每行记录： - - InnoDB只查找版本早于当前事务版本的数据行，这样可以确保事务读取的行，要么是在开始事务之前已经存在要么是事务自身插入或者修改过的 +### 🎯 当前读与快照读的区别？ - - 行的删除版本号要么未定义，要么大于当前事务版本号，这样可以确保事务读取到的行在事务开始之前未被删除 +> 在 InnoDB 中，普通的 `SELECT` 是快照读，它基于 MVCC 从 undo log 里读取历史版本，不加锁，性能高；而带锁的查询（如 `SELECT ... FOR UPDATE`）以及 `UPDATE/DELETE/INSERT` 属于当前读，它会加锁并返回记录的最新版本，用来保证数据一致性。 - 只有符合上述两个条件的才会被查询出来 +**1. 快照读（Snapshot Read）** -- INSERT：InnoDB为新插入的每一行保存当前系统版本号作为行版本号 +- **定义**：读取的是数据的 **快照版本**（历史版本），不加锁。 -- DELETE：InnoDB为删除的每一行保存当前系统版本号作为行删除标识 +- **实现机制**：依赖 InnoDB 的 **MVCC（多版本并发控制）**，通过 `undo log` 保存旧版本数据，根据事务的隔离级别和 Read View 来决定能看到哪一条版本。 -- UPDATE：InnoDB为插入的一行新纪录保存当前系统版本号作为行版本号，同时保存当前系统版本号到原来的行作为删除标识 +- **特点**： + - 不加锁，读写不冲突 → 性能好。 + - 可能读到旧数据（取决于隔离级别：RC、RR）。 -保存这两个额外系统版本号，使大多数操作都不用加锁。使数据操作简单，性能很好，并且也能保证只会读取到符合要求的行。不足之处是每行记录都需要额外的存储空间，需要做更多的行检查工作和一些额外的维护工作。 +- **常见 SQL 场景**： -MVCC 只在 COMMITTED READ（读提交）和REPEATABLE READ（可重复读）两种隔离级别下工作。 + ``` + SELECT * FROM user WHERE id = 1; + ``` + （普通的 `SELECT`，没有加 `for update / lock in share mode`） +**2. 当前读（Current Read）** -### 事务日志 +- **定义**：读取的是 **记录的最新版本**，并且会加锁（保证数据一致性）。 -InnoDB 使用日志来减少提交事务时的开销。因为日志中已经记录了事务，就无须在每个事务提交时把缓冲池的脏块刷新(flush)到磁盘中。 +- **实现机制**：通过加锁（共享锁 / 排他锁）来保证读取的是最新值，阻塞其他事务的修改，避免并发问题。 -事务修改的数据和索引通常会映射到表空间的随机位置，所以刷新这些变更到磁盘需要很多随机 IO。 +- **特点**： -InnoDB 假设使用常规磁盘，随机IO比顺序IO昂贵得多，因为一个IO请求需要时间把磁头移到正确的位置，然后等待磁盘上读出需要的部分，再转到开始位置。 + - 读取最新数据，带有锁。 + - 和写操作冲突时会等待或死锁。 -InnoDB 用日志把随机IO变成顺序IO。一旦日志安全写到磁盘，事务就持久化了，即使断电了，InnoDB可以重放日志并且恢复已经提交的事务。 +- **常见 SQL 场景**： -InnoDB 使用一个后台线程智能地刷新这些变更到数据文件。这个线程可以批量组合写入，使得数据写入更顺序，以提高效率。 + ``` + SELECT * FROM user WHERE id = 1 FOR UPDATE; -- 排他锁 + SELECT * FROM user WHERE id = 1 LOCK IN SHARE MODE; -- 共享锁 + UPDATE user SET name = 'Tom' WHERE id = 1; -- 写操作，本质也是当前读 + DELETE FROM user WHERE id = 1; + INSERT INTO user VALUES (...); + ``` -事务日志可以帮助提高事务效率： +**3. 核心区别总结** -- 使用事务日志，存储引擎在修改表的数据时只需要修改其内存拷贝，再把该修改行为记录到持久在硬盘上的事务日志中，而不用每次都将修改的数据本身持久到磁盘。 -- 事务日志采用的是追加的方式，因此写日志的操作是磁盘上一小块区域内的顺序I/O，而不像随机I/O需要在磁盘的多个地方移动磁头，所以采用事务日志的方式相对来说要快得多。 -- 事务日志持久以后，内存中被修改的数据在后台可以慢慢刷回到磁盘。 -- 如果数据的修改已经记录到事务日志并持久化，但数据本身没有写回到磁盘，此时系统崩溃，存储引擎在重启时能够自动恢复这一部分修改的数据。 +| 对比点 | 快照读（Snapshot Read） | 当前读（Current Read） | +| ------------ | ------------------------------------ | ----------------------------------------- | +| **是否加锁** | 不加锁 | 加锁（共享/排他） | +| **读的数据** | 历史版本（符合隔离级别的可见性规则） | 最新版本 | +| **性能** | 高，读写不冲突 | 较低，可能阻塞 | +| **典型场景** | 普通 `SELECT` | `SELECT … FOR UPDATE`、`UPDATE`、`DELETE` | -目前来说，大多数存储引擎都是这样实现的，我们通常称之为**预写式日志**（Write-Ahead Logging），修改数据需要写两次磁盘。 +### 🎯 版本链问题？ -### 事务的实现 +我现在有三个事务，ID 分别是 101、102、103。如果事务 101 已经提交了，但是 102、103 还没提交。这个时候，我开启了一个事务，准备读取数据，那么我读到的是哪个事务的数据？如果这时候事务 103 提交了，但是 102 还没提交，那么会读到谁的呢？ -事务的实现是基于数据库的存储引擎。不同的存储引擎对事务的支持程度不一样。MySQL 中支持事务的存储引擎有 InnoDB 和 NDB。 +这种题，就和隔离级别有关系了。 -事务的实现就是如何实现ACID特性。 +在MVCC机制中，事务读取数据的可见性取决于事务的隔离级别和事务的开始时间。以下是根据你描述的情况，按照可重复读（Repeatable Read）隔离级别来分析： -事务的隔离性是通过锁实现，而事务的原子性、一致性和持久性则是通过事务日志实现 。 +1. **事务101已提交，事务102和103未提交时开启新事务读取数据**： + - 当你开启一个新的事务时，这个事务会创建一个Read View。Read View会根据事务的隔离级别来确定哪些数据版本是可见的。在可重复读隔离级别下，新事务的Read View将只能看到在该事务开始之前已经提交的事务所做的修改。因此，如果你的新事务是在事务102和103提交之前开启的，你将能够读取到事务101提交的数据，但是看不到事务102和103的修改。 +2. **事务103提交后，但事务102还未提交时读取数据**： + - 如果在新事务的Read View创建之后，事务103提交了，那么在可重复读隔离级别下，新事务的Read View已经创建完成，它不会包含事务103提交后的数据。因此，即使你的新事务在事务103提交后读取数据，你仍然不会看到事务103的修改，除非你的新事务重新启动并创建一个新的Read View。 +在可重复读隔离级别下，新事务读取的数据版本是基于事务开始时数据库的快照，这个快照包括了所有在此之前已经提交的事务的修改。一旦Read View创建，即使其他事务提交了，新事务也不会看到这些新提交的修改，除非新事务重新启动并创建一个新的Read View。 +需要注意的是，如果隔离级别是读已提交（Read Committed），情况会有所不同。在这种情况下，每次读取操作都会看到最新的提交事务的结果，所以如果事务103提交了，即使新事务已经开启，它在读取数据时也会看到事务103的修改。 -> 事务是如何通过日志来实现的，说得越深入越好。 -事务日志包括：**重做日志redo**和**回滚日志undo** -- **redo log（重做日志**） 实现持久化和原子性 +### 🎯 简单说下事务日志？ - 在innoDB的存储引擎中，事务日志通过重做(redo)日志和innoDB存储引擎的日志缓冲(InnoDB Log Buffer)实现。事务开启时，事务中的操作，都会先写入存储引擎的日志缓冲中，在事务提交之前，这些缓冲的日志都需要提前刷新到磁盘上持久化，这就是DBA们口中常说的“日志先行”(Write-Ahead Logging)。当事务提交之后，在Buffer Pool中映射的数据文件才会慢慢刷新到磁盘。此时如果数据库崩溃或者宕机，那么当系统重启进行恢复时，就可以根据redo log中记录的日志，把数据库恢复到崩溃前的一个状态。未完成的事务，可以继续提交，也可以选择回滚，这基于恢复的策略而定。 +InnoDB 使用日志来减少提交事务时的开销。因为日志中已经记录了事务，就无须在每个事务提交时把缓冲池的脏块刷新(flush)到磁盘中。 - 在系统启动的时候，就已经为redo log分配了一块连续的存储空间，以顺序追加的方式记录Redo Log，通过顺序IO来改善性能。所有的事务共享redo log的存储空间，它们的Redo Log按语句的执行顺序，依次交替的记录在一起。 +事务修改的数据和索引通常会映射到表空间的随机位置，所以刷新这些变更到磁盘需要很多随机 IO。 -- **undo log（回滚日志）** 实现一致性 +InnoDB 假设使用常规磁盘，随机IO比顺序IO昂贵得多，因为一个IO请求需要时间把磁头移到正确的位置，然后等待磁盘上读出需要的部分，再转到开始位置。 - undo log 主要为事务的回滚服务。在事务执行的过程中，除了记录redo log，还会记录一定量的undo log。undo log记录了数据在每个操作前的状态，如果事务执行过程中需要回滚，就可以根据undo log进行回滚操作。单个事务的回滚，只会回滚当前事务做的操作，并不会影响到其他的事务做的操作。 +InnoDB 用日志把随机 IO 变成顺序 IO。一旦日志安全写到磁盘，事务就持久化了，即使断电了，InnoDB可以重放日志并且恢复已经提交的事务。 - Undo记录的是已部分完成并且写入硬盘的未完成的事务，默认情况下回滚日志是记录下表空间中的（共享表空间或者独享表空间） +InnoDB 使用一个后台线程智能地刷新这些变更到数据文件。这个线程可以批量组合写入，使得数据写入更顺序，以提高效率。 -二种日志均可以视为一种恢复操作，redo_log是恢复提交事务修改的页操作，而undo_log是回滚行记录到特定版本。二者记录的内容也不同，redo_log是物理日志，记录页的物理修改操作，而undo_log是逻辑日志，根据每行记录进行记录。 +事务日志可以帮助提高事务效率： +- 使用事务日志，存储引擎在修改表的数据时只需要修改其内存拷贝，再把该修改行为记录到持久在硬盘上的事务日志中，而不用每次都将修改的数据本身持久到磁盘。 +- 事务日志采用的是追加的方式，因此写日志的操作是磁盘上一小块区域内的顺序I/O，而不像随机I/O需要在磁盘的多个地方移动磁头，所以采用事务日志的方式相对来说要快得多。 +- 事务日志持久以后，内存中被修改的数据在后台可以慢慢刷回到磁盘。 +- 如果数据的修改已经记录到事务日志并持久化，但数据本身没有写回到磁盘，此时系统崩溃，存储引擎在重启时能够自动恢复这一部分修改的数据。 +目前来说，大多数存储引擎都是这样实现的，我们通常称之为**预写式日志**（Write-Ahead Logging），修改数据需要写两次磁盘。 -> 又引出个问题：你知道MySQL 有多少种日志吗？ -- **错误日志**：记录出错信息，也记录一些警告信息或者正确的信息。 -- **查询日志**：记录所有对数据库请求的信息，不论这些请求是否得到了正确的执行。 +### 🎯 MySQL 事务的 ACID 实现原理？ | 事务的实现？ -- **慢查询日志**：设置一个阈值，将运行时间超过该值的所有SQL语句都记录到慢查询的日志文件中。 +事务的实现是基于数据库的存储引擎。不同的存储引擎对事务的支持程度不一样。MySQL 中支持事务的存储引擎有 InnoDB 和 NDB。 -- **二进制日志**：记录对数据库执行更改的所有操作。 +事务的实现就是如何实现 ACID 特性。 -- **中继日志**：中继日志也是二进制日志，用来给slave 库恢复 +事务的隔离性是通过锁实现，而事务的原子性、一致性和持久性则是通过事务日志实现 。 -- **事务日志**：重做日志redo和回滚日志undo +>**还看到一种说法**： +> +>从数据库层面，数据库通过原子性、隔离性、持久性来保证一致性。也就是说 ACID 四大特性之中，C(一致性)是目的，A(原子性)、I(隔离性)、D(持久性)是手段，是为了保证一致性，数据库提供的手段。数据库必须要实现AID三大特性，才有可能实现一致性。例如，原子性无法保证，显然一致性也无法保证。我感觉这种更好些 + +MySQL（主要指 InnoDB 引擎）通过 **存储引擎的机制**来保证事务的 **ACID**： + +1. **A（原子性，Atomicity）** + - 原子性保证事务中的操作要么全部成功，要么全部失败。 + - **实现原理**： + - InnoDB 使用 **undo log（回滚日志）** 来保证原子性。 + - 如果事务中途失败，可以用 undo log 回滚到事务开始前的状态。 +2. **C（一致性，Consistency）** + - 一致性保证数据从一个一致状态转换到另一个一致状态（符合约束、触发器等）。 + - **实现原理**： + - 一致性依赖于其他三个特性（A、I、D）的保证； + - 同时依靠 **外键约束、唯一性约束、触发器** 等机制，确保数据完整性。 +3. **I（隔离性，Isolation）** + - 隔离性保证并发事务之间互不干扰。 + - **实现原理**： + - InnoDB 采用 **MVCC（多版本并发控制）+ 锁机制** 实现： + - **MVCC**：通过 **undo log + ReadView**，不同事务在相同时间点读到不同版本的数据，避免读写冲突； + - **锁机制**：行锁、间隙锁、Next-Key 锁，防止写写冲突、幻读。 +4. **D（持久性，Durability）** + - 持久性保证事务一旦提交，数据永久保存，即使宕机也不会丢失。 + - **实现原理**： + - InnoDB 使用 **redo log（重做日志）+ WAL 机制** 实现： + - 提交时先写 redo log（顺序写，落盘快）； + - 后续再将数据异步刷入数据文件； + - 宕机恢复时，依赖 redo log 重放，保证数据持久化。 > 分布式事务相关问题，可能还会问到 2PC、3PC，，， -### MySQL对分布式事务的支持 +### 🎯 MySQL 对分布式事务的支持 分布式事务的实现方式有很多，既可以采用 InnoDB 提供的原生的事务支持，也可以采用消息队列来实现分布式事务的最终一致性。这里我们主要聊一下 InnoDB 对分布式事务的支持。 -MySQL 从 5.0.3 InnoDB 存储引擎开始支持XA协议的分布式事务。一个分布式事务会涉及多个行动，这些行动本身是事务性的。所有行动都必须一起成功完成，或者一起被回滚。 +MySQL 从 5.0.3 InnoDB 存储引擎开始支持 XA 协议的分布式事务。一个分布式事务会涉及多个行动，这些行动本身是事务性的。所有行动都必须一起成功完成，或者一起被回滚。 -在MySQL中，使用分布式事务涉及一个或多个资源管理器和一个事务管理器。 +以下是 MySQL 对分布式事务支持的几个关键点： -![](https://imgkr.cn-bj.ufileos.com/8d48c5e1-c849-413e-8e5a-e96529235f58.png) +1. **XA 事务**：MySQL 的 InnoDB 存储引擎支持 XA 事务，这是分布式事务处理的基础。XA 事务允许跨多个数据库实例的事务操作，要么全部提交，要么全部回滚，以保持数据的一致性。 +2. **两阶段提交**：XA 事务使用两阶段提交协议来确保事务的原子性和一致性。在第一阶段，所有的参与资源（如不同的数据库实例）准备提交事务；在第二阶段，根据第一阶段的准备结果，决定是提交还是回滚事务。 +3. **隔离级别**：在使用 XA 事务时，InnoDB 存储引擎的事务隔离级别必须设置为 `SERIALIZABLE`，这是最高的隔离级别，可以避免脏读、不可重复读和幻读。 +4. **XA 语法**：MySQL 支持 XA 事务的特定语法，包括 `XA START`、`XA END`、`XA PREPARE`、`XA COMMIT` 和 `XA ROLLBACK` 等命令，用于控制分布式事务的流程。 +5. **事务管理器和资源管理器**：在 XA 事务中，事务管理器（TM）负责协调全局事务，而资源管理器（RM）负责管理具体的事务资源，如数据库连接。 +6. **分布式事务的问题**：XA 事务可能面临一些问题，如超时、死锁和资源管理器故障等。此外，MySQL 中的 XA 事务在早期版本中存在一些限制，比如在服务器故障重启后，已准备的事务的二进制日志可能会丢失，导致主从复制的数据不一致问题。 +7. **MySQL 5.7 改进**：在 MySQL 5.7 版本中，对分布式事务的支持有所改进，解决了一些长期存在的问题，比如在客户端退出或服务宕机时，已准备的事务会被正确回滚，以及在服务器故障重启后，相应的 Binlog 被正确处理。 +8. **应用场景**：分布式事务在分布式架构或需要跨多个数据库系统进行数据同步的场景中比较常见，例如金融行业的转账操作。 -如图，MySQL 的分布式事务模型。模型中分三块：应用程序（AP）、资源管理器（RM）、事务管理器（TM）: +总的来说，MySQL 通过 XA 事务为分布式系统提供了强大的事务支持，确保了跨多个数据库实例的操作能够满足 ACID 原则。随着 MySQL 版本的更新，分布式事务的支持也在不断改进和增强。 -- 应用程序：定义了事务的边界，指定需要做哪些事务； -- 资源管理器：提供了访问事务的方法，通常一个数据库就是一个资源管理器； -- 事务管理器：协调参与了全局事务中的各个事务。 +> 关于分布式事务，首先你需要弄清楚一个东西，就是分布式事务既可以是纯粹多个数据库实例之间的分布式事务，也可以是跨越不同中间件的业务层面上的分布式事务。前者一般是分库分表中间件提供支持，后者一般是独立的第三方中间件提供支持，比如 Seata。 -分布式事务采用两段式提交（two-phase commit）的方式： -- 第一阶段所有的事务节点开始准备，告诉事务管理器ready。 -- 第二阶段事务管理器告诉每个节点是commit还是rollback。如果有一个节点失败，就需要全局的节点全部rollback，以此保障事务的原子性。 ------- +### 🎯 在单体应用拆分成微服务架构之后，你怎么解决分布式事务？ +**常见的解决方案** +- **最终一致性：** "最理想的解决方案是通过最终一致性来避免分布式事务。可以通过事件驱动架构、异步消息和服务之间的解耦来处理。每个服务都处理自己的事务，当一个服务完成业务操作后，它会发布一个事件，其他服务订阅并处理相应的事件，逐步确保数据一致性。最终一致性容忍系统中的暂时不一致，但它能有效避免长时间的锁定和事务阻塞问题。" +- **Saga模式：** "Saga模式是解决分布式事务的一种常见方法，它将长事务拆分成多个局部事务，每个局部事务都会发布事件。如果某个事务失败，系统会触发补偿事务来回滚之前的操作。Saga模式有两种实现方式：一种是基于长事务的补偿机制，另一种是基于事件驱动的方式，适用于更高并发的环境。通过Saga模式，我们可以避免分布式事务中常见的性能瓶颈。" +- **两阶段提交（2PC）：** "两阶段提交（2PC）是一种经典的分布式事务解决方案，它通过协调者保证所有参与者在提交事务前的一致性。然而，2PC有一个显著的问题，就是在网络分区或者某些参与者失败时，容易导致系统的阻塞。这使得它不适合高并发和高可用的微服务环境。" +- **三阶段提交（3PC）：** "三阶段提交是2PC的改进，旨在解决2PC中的阻塞问题。它引入了准备阶段、提交阶段和确认阶段来确保系统的一致性。尽管比2PC更安全，但在高并发场景下仍然会对性能造成影响。" +- **TCC模式：** "TCC（Try-Confirm/Cancel）模式将每个操作分为三个阶段：尝试阶段（Try）、确认阶段（Confirm）和取消阶段（Cancel）。如果所有操作都成功，执行确认，失败则执行取消操作回滚之前的所有操作。这种模式适用于复杂的分布式事务场景，确保每个操作的最终一致性。” -## 七、MySQL锁机制 +**强调最佳实践与策略：** "在实际应用中，我会根据具体的业务场景选择合适的解决方案。如果业务逻辑允许，我倾向于通过**最终一致性**和**事件驱动架构**来避免复杂的分布式事务管理。如果需要严格的一致性保障，可以选择**Saga模式**，它能较好地平衡可用性与一致性。**TCC模式**适用于事务较为复杂的场景，尤其是在资金交易类系统中。” -> 数据库的乐观锁和悲观锁？ -> -> MySQL 中有哪几种锁，列举一下？ -> -> MySQL中InnoDB引擎的行锁是怎么实现的？ -> -> MySQL 间隙锁有没有了解，死锁有没有了解，写一段会造成死锁的 sql 语句，死锁发生了如何解决，MySQL 有没有提供什么机制去解决死锁 + + +### MySQL 锁机制 锁是计算机协调多个进程或线程并发访问某一资源的机制。 在数据库中，除传统的计算资源（如CPU、RAM、I/O等）的争用以外，数据也是一种供许多用户共享的资源。数据库锁定机制简单来说，就是数据库为了保证数据的一致性，而使各种共享资源在被并发访问变得有序所设计的一种规则。 -打个比方，我们到淘宝上买一件商品，商品只有一件库存，这个时候如果还有另一个人买，那么如何解决是你买到还是另一个人买到的问题？这里肯定要用到事物，我们先从库存表中取出物品数量，然后插入订单，付款后插入付款表信息，然后更新商品数量。在这个过程中，使用锁可以对有限的资源进行保护，解决隔离和并发的矛盾。 +打个比方，我们到淘宝上买一件商品，商品只有一件库存，这个时候如果还有另一个人买，那么如何解决是你买到还是另一个人买到的问题？这里肯定要用到事务，我们先从库存表中取出物品数量，然后插入订单，付款后插入付款表信息，然后更新商品数量。在这个过程中，使用锁可以对有限的资源进行保护，解决隔离和并发的矛盾。 - +在 MySQL 的 InnoDB 引擎里面，锁是借助索引来实现的。或者说，加锁锁住的其实是索引项，更加具体地来说，就是锁住了**叶子节点** + +1. **锁的物理载体**：B+树索引节点（非数据页） +2. **锁升级条件**：无索引或索引失效时退化为表锁 +3. **锁兼容性**：共享锁（S锁）允许并行读，排他锁（X锁）独占写 -### 锁的分类 +### 🎯 MySQL 中有哪几种锁，列举一下？ **从对数据操作的类型分类**： @@ -945,758 +1251,2557 @@ MySQL 从 5.0.3 InnoDB 存储引擎开始支持XA协议的分布式事务。一 | InnoDB | √ | √ | | | Memory | | √ | | +> ##### MyISAM 表锁 +> +> MyISAM 的表锁有两种模式： +> +> - 表共享读锁 （Table Read Lock）：不会阻塞其他用户对同一表的读请求，但会阻塞对同一表的写请求； +> - 表独占写锁 （Table Write Lock）：会阻塞其他用户对同一表的读和写操作； +> +> MyISAM 表的读操作与写操作之间，以及写操作之间是串行的。当一个线程获得对一个表的写锁后， 只有持有锁的线程可以对表进行更新操作。 其他线程的读、 写操作都会等待，直到锁被释放为止。 +> +> 默认情况下，写锁比读锁具有更高的优先级：当一个锁释放时，这个锁会优先给写锁队列中等候的获取锁请求，然后再给读锁队列中等候的获取锁请求。 +> +> ##### InnoDB 行锁 +> +> InnoDB 实现了以下两种类型的**行锁**： +> +> - 共享锁又称为读锁，简称 S 锁，顾名思义，共享锁就是多个事务对于同一数据可以共享一把锁，都能访问到数据，但是只能读不能修改。 +> +> - 排他锁又称为写锁，简称 X 锁，顾名思义，排他锁就是不能与其他所并存，如一个事务获取了一个数据行的排他锁，其他事务就不能再获取该行的其他锁，包括共享锁和排他锁，但是获取排他锁的事务是可以对数据行读取和修改。 +> +> 为了允许行锁和表锁共存，实现多粒度锁机制，InnoDB 还有两种内部使用的意向锁（Intention Locks），这两种意向锁都是**表锁**： +> +> - 意向共享锁（IS）：事务打算给数据行加行共享锁，事务在给一个数据行加共享锁前必须先取得该表的 IS 锁。 +> - 意向排他锁（IX）：事务打算给数据行加行排他锁，事务在给一个数据行加排他锁前必须先取得该表的 IX 锁。 +> +> **索引失效会导致行锁变表锁**。比如 varchar 查询不写单引号的情况。 +> +> **InnoDB 的行锁是针对索引加的锁，不是针对记录加的锁。并且该索引不能失效，否则都会从行锁升级为表锁**。 -### MyISAM 表锁 -MyISAM 的表锁有两种模式： +### 🎯 MySQL 中 InnoDB 引擎的行锁是怎么实现的？ -- 表共享读锁 （Table Read Lock）：不会阻塞其他用户对同一表的读请求，但会阻塞对同一表的写请求； -- 表独占写锁 （Table Write Lock）：会阻塞其他用户对同一表的读和写操作； +InnoDB 的行锁是其支持高并发的核心机制，**本质是基于索引记录加锁，通过内存中的锁结构与事务系统实现并发控制**。 -MyISAM 表的读操作与写操作之间，以及写操作之间是串行的。当一个线程获得对一个表的写锁后， 只有持有锁的线程可以对表进行更新操作。 其他线程的读、 写操作都会等待，直到锁被释放为止。 -默认情况下，写锁比读锁具有更高的优先级：当一个锁释放时，这个锁会优先给写锁队列中等候的获取锁请求，然后再给读锁队列中等候的获取锁请求。 +**一、行锁的本质：基于索引记录的 “标记”** +- InnoDB **不会直接锁定物理数据行**，而是通过 **索引 B+ 树上的索引项** 来加锁。 +- 聚簇索引：锁住叶子节点（数据行存储在聚簇索引里）。 +- 二级索引：先锁住二级索引项，再回表锁住聚簇索引项。 +- **无索引或索引失效时** → 行锁退化为表锁。 -### InnoDB 行锁 +**例**：对表 `user(id, name)`（`id` 为主键，`name` 为二级索引）执行 `UPDATE user SET age=30 WHERE name='张三'` 时： -InnoDB 实现了以下两种类型的**行锁**： +1. 先锁定 `name='张三'` 对应的二级索引记录； +2. 通过二级索引回表，锁定对应 `id` 的聚簇索引记录。 -- 共享锁（S）：允许一个事务去读一行，阻止其他事务获得相同数据集的排他锁。 -- 排他锁（X）：允许获得排他锁的事务更新数据，阻止其他事务取得相同数据集的共享读锁和排他写锁。 +**二、行锁的类型与数据结构** -为了允许行锁和表锁共存，实现多粒度锁机制，InnoDB 还有两种内部使用的意向锁（Intention Locks），这两种意向锁都是**表锁**： +InnoDB 实现了两种基础行锁类型，通过 **锁结构（Lock Struct）** 存储在内存中： -- 意向共享锁（IS）：事务打算给数据行加行共享锁，事务在给一个数据行加共享锁前必须先取得该表的 IS 锁。 -- 意向排他锁（IX）：事务打算给数据行加行排他锁，事务在给一个数据行加排他锁前必须先取得该表的 IX 锁。 +1. **共享锁（S 锁，读锁）** + - 允许事务读取一行数据，多个事务可同时持有同一行的 S 锁（读不互斥）； + - 加锁方式：`SELECT ... LOCK IN SHARE MODE`。 +2. **排他锁（X 锁，写锁）** + - 允许事务修改或删除一行数据，与其他任何锁（S 或 X）互斥（写独占）； + - 加锁方式：`UPDATE/DELETE` 自动加 X 锁，或 `SELECT ... FOR UPDATE` 显式加 X 锁。 -**索引失效会导致行锁变表锁**。比如 vchar 查询不写单引号的情况。 +**锁结构的核心信息**： -#### 加锁机制 +- 锁定的索引记录（如 `id=100` 的聚簇索引记录）； +- 锁类型（S 或 X）； +- 持有锁的事务 ID； +- 等待该锁的事务链表（当锁冲突时，等待的事务会被挂入此链表）。 -**乐观锁与悲观锁是两种并发控制的思想，可用于解决丢失更新问题** +**三、行锁的实现依赖：事务与 undo 日志** -乐观锁会“乐观地”假定大概率不会发生并发更新冲突，访问、处理数据过程中不加锁，只在更新数据时再根据版本号或时间戳判断是否有冲突，有则处理，无则提交事务。用数据版本（Version）记录机制实现，这是乐观锁最常用的一种实现方式 +1. **事务 ID 与锁的关联**每个事务开始时会分配唯一的 `transaction_id`，锁结构中会记录持有锁的 `transaction_id`，用于标识锁的归属。当事务提交或回滚时，InnoDB 会释放该事务持有的所有行锁，并唤醒等待链表中的事务。 +2. **undo 日志与行锁释放**行锁的释放时机与事务状态强相关： + - 对于 `UPDATE/DELETE` 操作，InnoDB 会先写 undo 日志（用于事务回滚），然后加 X 锁修改数据； + - 事务提交后，undo 日志标记为可清理，行锁释放； + - 事务回滚时，通过 undo 日志恢复数据，同时释放行锁。 -悲观锁会“悲观地”假定大概率会发生并发更新冲突，访问、处理数据前就加排他锁，在整个数据处理过程中锁定数据，事务提交或回滚后才释放锁。另外与乐观锁相对应的，**悲观锁是由数据库自己实现了的，要用的时候，我们直接调用数据库的相关语句就可以了。** +**四、特殊锁机制：间隙锁与临键锁（解决幻读）** +在默认的 **Repeatable Read（可重复读）** 隔离级别下，InnoDB 为了防止 “幻读”（同一事务中，多次查询同一范围返回不同行数），引入了两种特殊锁，本质是对 “行锁粒度的扩展”： +1. **记录锁（Record Lock）** + - 锁定单条索引记录 `SELECT * FROM user WHERE id=1 FOR UPDATE;` +2. **间隙锁（Gap Lock）** + - 锁定索引记录之间的 “间隙”（不包含记录本身），防止其他事务在间隙中插入新行； + - 例：表中存在 `id=10, 20`，执行 `UPDATE ... WHERE id BETWEEN 10 AND 20` 时，会锁定间隙 `(10,20)`，阻止插入 `id=15` 的行。 +3. **临键锁（Next-Key Lock）** + - **记录锁 + 前一个间隙**，锁定范围为“左开右闭”。 + - InnoDB 默认使用 Next-Key Lock，保证 RR 隔离级别下无幻读。 + - 如果是唯一索引精确匹配，会优化为 Record Lock。 -#### 锁模式(InnoDB有三种行锁的算法) +**注意**：若将隔离级别降为 **Read Committed**，间隙锁和临键锁会关闭，仅保留行锁（可能出现幻读，但并发性能提升）。 -- **记录锁(Record Locks)**： 单个行记录上的锁。对索引项加锁，锁定符合条件的行。其他事务不能修改和删除加锁项； +**五、行锁的触发与释放流程（以 UPDATE 为例）** - ```mysql - SELECT * FROM table WHERE id = 1 FOR UPDATE; - ``` +1. 事务开始，获取 `transaction_id`； +2. 解析 SQL，定位需要修改的索引记录（通过 B+ 树查找）； +3. 对目标索引记录尝试加 X 锁： + - 若锁未被占用，加锁成功，继续执行修改； + - 若锁已被其他事务占用，当前事务进入等待状态，挂入该锁的等待链表； +4. 修改数据（写 redo 日志确保持久化，写 undo 日志用于回滚）； +5. 事务提交 / 回滚：释放所有行锁，唤醒等待链表中的事务重新竞争锁。 - 它会在 id=1 的记录上加上记录锁，以阻止其他事务插入，更新，删除 id=1 这一行 +**六、核心结论** - 在通过 主键索引 与 唯一索引 对数据行进行 UPDATE 操作时，也会对该行数据加记录锁： +InnoDB 行锁的实现可总结为： - ```mysql - -- id 列为主键列或唯一索引列 - UPDATE SET age = 50 WHERE id = 1; - ``` +1. **索引是基础**：无索引或索引失效时，行锁会退化为表锁； +2. **锁结构是载体**：通过内存中的锁结构记录锁类型、持有事务和等待队列； +3. **事务是上下文**：锁的生命周期与事务绑定，提交 / 回滚时释放； +4. **特殊锁是补充**：间隙锁和临键锁解决幻读，代价是降低部分并发性能。 -- **间隙锁（Gap Locks）**： 当我们使用范围条件而不是相等条件检索数据，并请求共享或排他锁时，InnoDB会给符合条件的已有数据记录的索引项加锁。对于键值在条件范围内但并不存在的记录，叫做“间隙”。 +> **最早只有 Record Lock**，只能锁住已有的记录，能保证“当前读”的一致性； +> +> 但是只用 Record Lock 会有 **幻读问题**（别的事务在两次查询之间插入新行）； +> +> 为了解决幻读，引入 **Gap Lock**，把“记录之间的空隙”也锁起来，阻止插入； +> +> 但单独使用 Record Lock + Gap Lock 管理复杂，所以 InnoDB 提出了 **Next-Key Lock**（记录锁 + 前一个间隙），作为默认锁算法，保证 `REPEATABLE READ` 下无幻读。 - InnoDB 也会对这个“间隙”加锁，这种锁机制就是所谓的间隙锁。 +### 🎯 MySQL行锁，颗粒度有什么变化 - 对索引项之间的“间隙”加锁，锁定记录的范围（对第一条记录前的间隙或最后一条将记录后的间隙加锁），不包含索引项本身。其他事务不能在锁范围内插入数据，这样就防止了别的事务新增幻影行。 +MySQL 的行锁粒度主要体现在不同的存储引擎和索引使用情况。InnoDB 是支持行级锁的，它的粒度会根据索引条件来变化： - 间隙锁基于非唯一索引，它锁定一段范围内的索引记录。间隙锁基于下面将会提到的`Next-Key Locking` 算法，请务必牢记：**使用间隙锁锁住的是一个区间，而不仅仅是这个区间中的每一条数据**。 +- 如果查询语句命中了索引，通常是**行级锁**，锁定更精细； +- 如果没有命中索引或者索引不够精准，可能会退化为**表级锁**； +- 在事务隔离级别下，不同的加锁策略（比如 `next-key lock`）也会让粒度变粗，涉及到间隙锁、范围锁等。 + 所以行锁的颗粒度不是固定的，会随着 **索引情况、SQL 语句、事务隔离级别**而变化。 - ```mysql - SELECT * FROM table WHERE id BETWEN 1 AND 10 FOR UPDATE; - ``` - 即所有在`（1，10）`区间内的记录行都会被锁住，所有id 为 2、3、4、5、6、7、8、9 的数据行的插入会被阻塞，但是 1 和 10 两条记录行并不会被锁住。 - GAP锁的目的，是为了防止同一事务的两次当前读，出现幻读的情况 +### 🎯 MySQL 隔离级别与锁的关系？ -- **临键锁(Next-key Locks)**： **临键锁**，是**记录锁与间隙锁的组合**，它的封锁范围，既包含索引记录，又包含索引区间。(临键锁的主要目的，也是为了避免**幻读**(Phantom Read)。如果把事务的隔离级别降级为RC，临键锁则也会失效。) +在 MySQL 中，隔离级别和锁的关系紧密相关，不同的隔离级别通过使用不同的锁机制来控制事务间的并发行为，确保数据的一致性和隔离性。以下是 MySQL 四种隔离级别与锁的关系： - Next-Key 可以理解为一种特殊的**间隙锁**，也可以理解为一种特殊的**算法**。通过**临建锁**可以解决幻读的问题。 每个数据行上的非唯一索引列上都会存在一把临键锁，当某个事务持有该数据行的临键锁时，会锁住一段左开右闭区间的数据。需要强调的一点是，`InnoDB` 中行级锁是基于索引实现的，临键锁只与非唯一索引列有关，在唯一索引列（包括主键列）上不存在临键锁。 +1. **读未提交（Read Uncommitted）** - 对于行的查询，都是采用该方法，主要目的是解决幻读的问题。 + - 锁机制：最低级别，不使用锁，允许读取未提交的数据。 -> select for update有什么含义，会锁表还是锁行还是其他 + - 并发性：高。 -for update 仅适用于InnoDB，且必须在事务块(BEGIN/COMMIT)中才能生效。在进行事务操作时，通过“for update”语句，MySQL会对查询结果集中每行数据都添加排他锁，其他线程对该记录的更新与删除操作都会阻塞。排他锁包含行锁、表锁。 + - 数据一致性：低，可能导致脏读。 -InnoDB这种行锁实现特点意味着：只有通过索引条件检索数据，InnoDB才使用行级锁，否则，InnoDB将使用表锁！ -假设有个表单 products ，里面有id跟name二个栏位，id是主键。 -- 明确指定主键，并且有此笔资料，row lock +2. **读已提交（Read Committed）** -```mysql -SELECT * FROM products WHERE id='3' FOR UPDATE; -SELECT * FROM products WHERE id='3' and type=1 FOR UPDATE; -``` + - 锁机制：读取时使用共享锁（S Lock），更新时使用排他锁（X Lock）。 -- 明确指定主键，若查无此笔资料，无lock + - 并发性：中等。 -```mysql -SELECT * FROM products WHERE id='-1' FOR UPDATE; -``` + - 数据一致性：避免脏读，但可能发生不可重复读。 -- 无主键，table lock -```mysql -SELECT * FROM products WHERE name='Mouse' FOR UPDATE; -``` +3. **可重复读（Repeatable Read）** -- 主键不明确，table lock + - 锁机制：读取时使用共享锁（S Lock），更新时使用排他锁（X Lock），并使用间隙锁（Gap Lock）防止幻读。 -```mysql -SELECT * FROM products WHERE id<>'3' FOR UPDATE; -``` + - 并发性：较低。 -- 主键不明确，table lock + - 数据一致性：避免脏读和不可重复读，但在默认设置下防止幻读。 -```mysql -SELECT * FROM products WHERE id LIKE '3' FOR UPDATE; -``` -**注1**: FOR UPDATE仅适用于InnoDB，且必须在交易区块(BEGIN/COMMIT)中才能生效。 -**注2**: 要测试锁定的状况，可以利用MySQL的Command Mode ，开二个视窗来做测试。 +4. **串行化（Serializable）** + - 锁机制：事务间完全隔离，所有读取和写入都使用表级锁或行级锁，确保完全隔离。 + - 并发性：最低。 -> MySQL 遇到过死锁问题吗，你是如何解决的？ + - 数据一致性：最高，避免所有并发问题。 -### 死锁 -**死锁产生**： +选择适当的隔离级别和锁机制，可以在数据一致性和系统并发性之间找到最佳平衡。 -- 死锁是指两个或多个事务在同一资源上相互占用，并请求锁定对方占用的资源，从而导致恶性循环 -- 当事务试图以不同的顺序锁定资源时，就可能产生死锁。多个事务同时锁定同一个资源时也可能会产生死锁 + + +### 🎯 两个事务 update 同一条数据会发生什么？ + +在数据库中，如果两个事务同时尝试更新同一条数据，会发生以下几种情况之一，具体取决于数据库的隔离级别和锁机制： + +1. **脏读（Dirty Read）**：在读未提交（Read Uncommitted）隔离级别下，一个事务可能会读取到另一个事务未提交的更新数据。如果第二个事务回滚了更改，那么第一个事务就会读到一些最终不会被提交的“脏数据”。 +2. **不可重复读（Non-Repeatable Read）**：在读已提交（Read Committed）隔离级别下，一个事务在读取某条数据后，另一个事务可能会修改或更新这条数据。这导致第一个事务无法再次读取到之前的数据状态，即不可重复读。 +3. **幻读（Phantom Read）**：在可重复读（Repeatable Read）隔离级别下，一个事务在读取某个范围内的数据后，另一个事务可能会插入新数据，导致第一个事务再次读取该范围时，会发现一些“幻读”的数据行。 +4. **更新丢失（Lost Update）**：当两个事务都读取了同一数据项，并且基于读取的值进行更新时，如果它们几乎同时提交，那么第二个提交的事务的更新可能会覆盖第一个事务的更新，导致第一个事务的更新丢失。 +5. **第一提交者胜出**：在大多数情况下，数据库会使用悲观锁或乐观锁来管理并发更新。如果使用悲观锁，那么通常只有一个事务能够锁定数据并进行更新，其他事务必须等待。如果使用乐观锁，事务会检查在读取数据后是否有其他事务对其进行了修改，如果没有修改，才会提交更新。 +6. **写入冲突**：在某些数据库系统中，如果两个事务尝试同时更新同一数据，可能会引发写入冲突。数据库会根据其冲突解决策略来决定如何处理这种情况，可能会回滚其中一个事务，或者提供一种机制让应用层解决冲突。 + +为了避免这些问题，数据库通常提供不同的隔离级别，允许开发者根据业务需求选择合适的隔离级别。此外，通过使用事务锁和一致性检查，可以在应用层提供额外的控制来处理并发更新。 + +在设计系统时，应该仔细考虑并发控制策略，以确保数据的完整性和一致性。在实现时，可以使用数据库提供的锁机制，如行级锁、表级锁等，或者在应用层实现乐观锁或悲观锁逻辑。 + + + +### 🎯 在高并发情况下，如何做到安全的修改同一行数据？ + +在高并发情况下安全地修改同一行数据需要确保数据的一致性和完整性，避免并发导致的问题，如脏读、不可重复读和幻读。以下是一些常见的策略： + +1. **乐观锁**：乐观锁通过在数据表中添加一个版本号或时间戳字段来实现。每次更新数据时，检查版本号或时间戳是否一致，如果不一致，说明数据在读取后被其他事务修改过，当前事务应该放弃更新。 +2. **悲观锁**：悲观锁在事务开始时就对数据行进行锁定，直到事务结束才释放锁。这可以防止其他事务同时修改同一数据行。 +3. **原子操作**：使用数据库提供的原子操作，如 `UPDATE ... WHERE` 语句，确保修改操作的原子性。 +4. **事务隔离级别**：适当设置事务的隔离级别，如可重复读（REPEATABLE READ）或串行化（SERIALIZABLE），以减少并发事务间的干扰。 +5. **数据库的锁机制**：利用数据库的锁机制，如行锁、表锁等，来控制对数据的并发访问。 +6. **应用层控制**：在应用层实现重试逻辑，当检测到更新冲突时，可以重试事务。 +7. **分布式锁**：使用分布式锁来确保在分布式系统中，同一时间只有一个操作能够修改数据。 +8. **消息队列**：将更新操作封装在消息中，通过消息队列顺序处理更新，以避免并发冲突。 +9. **唯一索引**：在可能发生冲突的列上设置唯一索引，确保数据库层面上避免重复数据的插入。 +10. **条件更新**：使用条件更新，如 `UPDATE ... IF EXISTS`，来避免在数据已被其他事务修改的情况下进行更新。 + + + +### 🎯 数据库的乐观锁和悲观锁？ + +**乐观锁与悲观锁是两种并发控制的思想，可用于解决丢失更新问题** + +乐观锁会“乐观地”假定大概率不会发生并发更新冲突，访问、处理数据过程中不加锁，只在更新数据时再根据版本号或时间戳判断是否有冲突，有则处理，无则提交事务。用数据版本（Version）记录机制实现，这是乐观锁最常用的一种实现方式 + +悲观锁会“悲观地”假定大概率会发生并发更新冲突，访问、处理数据前就加排他锁，在整个数据处理过程中锁定数据，事务提交或回滚后才释放锁。另外与乐观锁相对应的，**悲观锁是由数据库自己实现了的，要用的时候，我们直接调用数据库的相关语句就可以了。** + + + +### 🎯 怎么在数据库里面使用乐观锁？ + +乐观锁（Optimistic Locking）是一种用于解决并发控制问题的技术，特别适用于读多写少的场景。它假设冲突很少发生，因此不在操作开始时加锁，而是在提交更新时检查是否有冲突。通常通过以下几种方式实现： + +- 版本号：在表中添加一个版本号字段，每次更新记录时，版本号加1。更新操作时，检查版本号是否匹配，匹配则更新，不匹配则说明有冲突，需要重新读取数据再进行操作。 +- 时间戳：用时间戳字段记录最后修改时间，更新操作时检查时间戳是否匹配。 + +> 可以用来解决电商中的“超卖”问题。 + + + +### 🎯 select for update有什么含义，会锁表还是锁行还是其他？ + +for update 仅适用于 InnoDB，且必须在事务块(BEGIN/COMMIT)中才能生效。在进行事务操作时，通过“for update”语句，MySQL 会对查询结果集中每行数据都添加排他锁，其他线程对该记录的更新与删除操作都会阻塞。排他锁包含行锁、表锁。 + +InnoDB 这种行锁实现特点意味着：只有通过索引条件检索数据，InnoDB 才使用行级锁，否则，InnoDB 将使用表锁！ + +假设有个表单 products ，里面有 id 跟 name 二个栏位，id 是主键。 + +- 明确指定主键，并且有此笔资料，row lock + +```mysql +SELECT * FROM products WHERE id='3' FOR UPDATE; +SELECT * FROM products WHERE id='3' and type=1 FOR UPDATE; +``` + +- 明确指定主键，若查无此笔资料，无lock + +```mysql +SELECT * FROM products WHERE id='-1' FOR UPDATE; +``` + +- 无主键，table lock + +```mysql +SELECT * FROM products WHERE name='Mouse' FOR UPDATE; +``` + +- 主键不明确，table lock + +```mysql +SELECT * FROM products WHERE id <>'3' FOR UPDATE; +``` + +- 主键不明确，table lock + +```mysql +SELECT * FROM products WHERE id LIKE '3' FOR UPDATE; +``` + +**注1**: FOR UPDATE 仅适用于 InnoDB，且必须在交易区块(BEGIN/COMMIT)中才能生效。 +**注2**: 要测试锁定的状况，可以利用 MySQL 的 Command Mode ，开二个视窗来做测试。 + + + +### 🎯 死锁有没有了解，写一段会造成死锁的 sql 语句? + +死锁是指两个或多个事务在等待彼此持有的资源，从而导致所有事务都无法继续执行的情况。在 MySQL 中，死锁通常发生在并发更新相同的资源时。 + +```mysql +START TRANSACTION; +UPDATE table_name SET column_a = 1 WHERE id = 1; +-- 这里事务 1 锁定了 id = 1 的记录 +UPDATE table_name SET column_b = 2 WHERE id = 2; +-- 等待事务 2 释放 id = 2 的锁 +``` + +```mysql +START TRANSACTION; +UPDATE table_name SET column_b = 2 WHERE id = 2; +-- 这里事务 2 锁定了 id = 2 的记录 +UPDATE table_name SET column_a = 1 WHERE id = 1; +-- 等待事务 1 释放 id = 1 的锁 +``` + + + +### 🎯 死锁发生了如何解决，MySQL 有没有提供什么机制去解决死锁？ + +> MySQL 遇到过死锁问题吗，你是如何解决的？ + +**死锁产生**： + +- 死锁是指两个或多个事务在同一资源上相互占用，并请求锁定对方占用的资源，从而导致恶性循环 +- 当事务试图以不同的顺序锁定资源时，就可能产生死锁。多个事务同时锁定同一个资源时也可能会产生死锁 - 锁的行为和顺序和存储引擎相关。以同样的顺序执行语句，有些存储引擎会产生死锁有些不会——死锁有双重原因：真正的数据冲突；存储引擎的实现方式。 -**检测死锁**：数据库系统实现了各种死锁检测和死锁超时的机制。InnoDB存储引擎能检测到死锁的循环依赖并立即返回一个错误。 +> 死锁产生的四个必要条件 +> +> 1. 互斥： 多个线程不能同时使用一个资源。比如线程 A 已经持有的资源，不能再同时被线程 B 持有。 +> 2. 持有并等待： 当线程 A 已经持有了资源 1，又提出申请资源 2，但是资源 2 已经被线程 C 占用，所以线程 A 就会处于等待状态，但它在等待资源 2 的同时并不会释放自己已经获取的资源 1。 +> 3. 不可剥夺： 线程 A 获取到资源 1 之后，在自己使用完之前不能被其他线程（比如线程 B）抢占使用。如果线程 B 也想使用资源 1，只能在线程 A 使用完后，主动释放后再获取 +> 4. 循环等待： 发生死锁时，必然会存在一个线程，也就是资源的环形链。比如线程 A 已经获取了资源 1，但同时又请求获取资源 2。线程 B 已经获取了资源 2，但同时又请求获取资源 1，这就会形成一个线程和资源请求等待的环形图。 + +**检测死锁**：数据库系统实现了各种死锁检测和死锁超时的机制。InnoDB 存储引擎能检测到死锁的循环依赖并立即返回一个错误。 + +**死锁恢复**：死锁发生以后，只有部分或完全回滚其中一个事务，才能打破死锁，**InnoDB 目前处理死锁的方法是，将持有最少行级排他锁的事务进行回滚**。所以事务型应用程序在设计时必须考虑如何处理死锁，多数情况下只需要重新执行因死锁回滚的事务即可。 + +**外部锁的死锁检测**：发生死锁后，InnoDB 一般都能自动检测到，并使一个事务释放锁并回退，另一个事务获得锁，继续完成事务。但在涉及外部锁，或涉及表锁的情况下，InnoDB 并不能完全自动检测到死锁， 这需要通过设置锁等待超时参数 innodb_lock_wait_timeout 来解决 + +**死锁影响性能**：死锁会影响性能而不是会产生严重错误，因为InnoDB会自动检测死锁状况并回滚其中一个受影响的事务。在高并发系统上，当许多线程等待同一个锁时，死锁检测可能导致速度变慢。 有时当发生死锁时，禁用死锁检测（使用innodb_deadlock_detect配置选项）可能会更有效，这时可以依赖`innodb_lock_wait_timeout`设置进行事务回滚。 + + + +### 🎯 如何尽可能避免死锁？ + +1. 合理的设计索引，区分度高的列放到组合索引前面，使业务 SQL 尽可能通过索引`定位更少的行，减少锁竞争`。 +2. 调整业务逻辑 SQL 执行顺序， 避免 update/delete 长时间持有锁的 SQL 在事务前面。 +3. 避免`大事务`，尽量将大事务拆成多个小事务来处理，小事务发生锁冲突的几率也更小。 +4. 以`固定的顺序`访问表和行。比如两个更新数据的事务，事务 A 更新数据的顺序为 1，2;事务 B 更新数据的顺序为 2，1。这样更可能会造成死锁。 +5. 在并发比较高的系统中，不要显式加锁，特别是是在事务里显式加锁。如 select … for update 语句，如果是在事务里`（运行了 start transaction 或设置了autocommit 等于0）`,那么就会锁定所查找到的记录。 +6. 尽量按`主键/索引`去查找记录，范围查找增加了锁冲突的可能性，也不要利用数据库做一些额外额度计算工作。比如有的程序会用到 “select … where … order by rand();”这样的语句，由于类似这样的语句用不到索引，因此将导致整个表的数据都被锁住。 +7. 优化 SQL 和表设计，减少同时占用太多资源的情况。比如说，`减少连接的表`，将复杂 SQL `分解`为多个简单的 SQL。 + + + +### 🎯 说个自己遇到的死锁案例？ + +早期我优化过一个死锁问题，是**临键锁引起**的。业务逻辑很简单，先用 SELECT FOR UPDATE 查询数据。如果查询到了数据，那么就执行一段业务逻辑，然后更新结果；如果没有查询到，那么就执行另外一段业务逻辑，然后插入计算结果。 + +那么如果 SELECT FOR UPDATE 查找的数据不存在，那么数据库会使用一个临键锁。此时，如果有两个线程加了临键锁，然后又希望插入计算结果，那么就会造成死锁。 + +我这个优化也很简单，就是上来先不管三七二十一，直接插入数据。如果插入成功，那么就执行没有数据的逻辑，此时不会再持有临键锁，而是持有了行锁。如果插入不成功，那么就执行有数据的业务逻辑。 + +此外，还有两个思路。一个是修改数据库的隔离级别为 RC，那么自然不存在临键锁了，但是这个修改影响太大，被 DBA 否决了。 + +另外一个思路就是使用乐观锁，不过代码改起来要更加复杂，所以就没有使用。 + +> 假设表 t 中最大 id 是 10，那么如果两个业务进来，同时执行这个逻辑。一个准备插入 id=11 的数据，一个准备插入 id = 12 的数据。如果它们的执行时序如下图，那么你就会得到一个死锁错误![](https://img.starfish.ink/mysql/MySQL-dead-lock-demo-1.png) +> +> 造成死锁的原因也很简单。在线程 A 执行 SELECT FOR UPDATE 的时候，因为 id=11 的数据不存在，所以实际上数据库会产生一个(11，supremum] 的临键锁。类似地，线程 B 也会产生一个(10，supremum] 临键锁。 + + + +------ + + + + + +## 五、数据类型与查询优化📊 + +**核心理念**：合理的数据类型选择和高效的查询编写是数据库性能的基础，掌握查询优化技巧能显著提升系统响应速度。 + +> 主要包括以下五大类： +> +> - 整数类型：BIT、BOOL、TINY INT、SMALL INT、MEDIUM INT、 INT、 BIG INT +> - 浮点数类型：FLOAT、DOUBLE、DECIMAL +> - 字符串类型：CHAR、VARCHAR、TINY TEXT、TEXT、MEDIUM TEXT、LONGTEXT、TINY BLOB、BLOB、MEDIUM BLOB、LONG BLOB +> - 日期类型：Date、DateTime、TimeStamp、Time、Year +> - 其他数据类型：BINARY、VARBINARY、ENUM、SET、Geometry、Point、MultiPoint、LineString、MultiLineString、Polygon、GeometryCollection等 + + + +### 🎯 CHAR 和 VARCHAR 的区别？ + +`CHAR` 和 `VARCHAR` 都是用于存储字符串的字段类型，但它们在存储方式、性能和使用场景上存在一些关键区别 + +- **存储方式**：char 是固定长度，varchar 长度可变： + - char(n) 和 varchar(n) 中括号中 n 代表字符的个数，并不代表字节个数，比如 CHAR(30) 就可以存储 30 个字符。 + + - 存储时，char 不管实际存储数据的长度，直接按 char 规定的长度分配存储空间；而 varchar 会根据实际存储的数据分配最终的存储空间，加上 1 或 2 个额外字节用于存储字符串的长度信息 + +- **性能不同**： + + - `CHAR` 因为其固定长度的特性，在某些情况下可能会提供更好的性能，尤其是当处理大量具有相同长度的数据时。 + - `VARCHAR` 由于需要存储长度信息，并且可能需要额外的空间来存储不同长度的字符串，因此在性能上可能略逊于 `CHAR` + +- 使用场景：char 适用于固定长度字符串：如存储状态码、国家代码、MD5 哈希值等。varchar 适用于可变长度字符串：如用户名、电子邮件地址、描述等。 + + + +### 🎯 MySQL 里记录货币用什么字段类型比较好？ + +> 阿里规范：【强制】任何货币金额，均以最小货币单位且整型类型来进行存储。 + +在 MySQL 中记录货币数据时，通常推荐使用 `DECIMAL` 类型。`DECIMAL` 类型提供高精度的存储和计算，非常适合用于存储货币值。以下是使用 `DECIMAL` 类型的原因以及其他可能选择的字段类型和其适用场景： + +**使用 `DECIMAL` 类型的原因** + +1. **高精度**： + - `DECIMAL` 类型可以精确存储货币值，没有浮点运算误差。例如，定义为 `DECIMAL(10, 2)` 表示最多 10 位数字，其中 2 位小数，适合存储最多到亿位的金额，精确到小数点后两位。 +2. **存储效率**： + - 由于货币值通常需要精确到小数点后两位，`DECIMAL` 能够确保存储的每一个值都是精确的，避免了浮点数可能带来的舍入误差。 +3. **计算正确性**： + - 在涉及到财务计算时，使用浮点数类型（如 `FLOAT` 或 `DOUBLE`）可能会因为舍入误差导致计算不准确。`DECIMAL` 类型避免了这些问题，确保计算结果的准确性。 + +当然还**有些业务**，用 `INT` 或者 `BIGINT` 类型存储货币的最小单位也可以（如美分、分），适合对性能有更高要求的场景（整数运算比浮点运算更快），但需要处理转换逻辑。 + + + +### 🎯 datetime 与 timestamp 的区别？ + +> MySQL 的 DATETIME 和 TIMESTAMP 都用来存储日期时间，但有几个关键区别： +> +> 1. DATETIME 与时区无关，存储的是什么取出来就是什么，范围更大（1000 ~ 9999 年），占 8 字节。 +> 2. TIMESTAMP 与时区相关，存储时会转为 UTC，取出时按会话时区转换，范围是 1970 ~ 2038 年，占 4~7 字节。 +> 3. TIMESTAMP 支持 `CURRENT_TIMESTAMP` 和自动更新，常用来存记录的创建时间、修改时间；而 DATETIME 更适合业务逻辑相关的时间点。 + +**1. 存储方式** + +- **DATETIME** + - 以字符串形式存储日期和时间，格式为 `'YYYY-MM-DD HH:MM:SS'`。 + - 与 **时区无关**，存进去什么值，取出来就是什么值。 +- **TIMESTAMP** + - 以 **Unix 时间戳**（从 1970-01-01 00:00:01 UTC 到某一时刻的秒数）存储。 + - 与 **时区相关**，存储时会转成 UTC，取出时会根据会话时区转换。 + +**2. 取值范围** + +- **DATETIME**：`1000-01-01 00:00:00` ~ `9999-12-31 23:59:59` +- **TIMESTAMP**：`1970-01-01 00:00:01 UTC` ~ `2038-01-19 03:14:07 UTC`（32 位整型限制，存在 2038 年问题） + +**3. 存储大小** + +- **DATETIME**：8 字节 +- **TIMESTAMP**：4 字节（MySQL 5.6 之后扩展为 4 ~ 7 字节，取决于小数秒精度） + +**4. 时区影响** + +- **DATETIME**：不受时区影响，适合存储固定的“绝对时间”。 +- **TIMESTAMP**：受时区影响，存储时转 UTC，取出时根据当前时区转换，适合存储事件发生的“真实时间点”。 + +**5. 默认值与自动更新** + +- **DATETIME**：默认值需要显式指定，一般不会自动更新。 +- **TIMESTAMP**：可以默认 `CURRENT_TIMESTAMP`，并支持 `ON UPDATE CURRENT_TIMESTAMP`，自动存储当前时间。 + +**6. 应用场景** + +- **DATETIME**：适合存储“业务层面的日期时间”，如生日、会议时间、账单周期等，保证跨时区不变。 +- **TIMESTAMP**：适合存储“记录生成或修改的时间”，如 `create_time`、`update_time`，自动随时区变化而调整。 + + + +### 🎯 BLOB 和 TEXT 有什么区别？ + +> 这个问题其实很不重要，因为大部分公司不让让使用这两种类型。 +> +> **禁止使用TEXT、BLOB类型**：会浪费更多的磁盘和内存空间，非必要的大量的大字段查询会淘汰掉热数据，导致内存命中率急剧降低，影响数据库性能 + +`BLOB`（Binary Large Object）和 `TEXT` 是 MySQL 中用于存储大型二进制数据和大型文本数据的两种不同的数据类型。它们之间的主要区别包括： + +1. **存储内容**：`BLOB` 用于存储二进制数据，如图片、音频、视频等。`TEXT` 用于存储大型文本数据，如文章、评论等。 +2. **最大长度**： + - `BLOB` 和 `TEXT` 类型的最大长度可以达到 65,535 字节，即 64KB（在 MySQL 5.0.3 之前的版本中，最大长度为 255 字节）。 + - 从 MySQL 5.0.3 版本开始，`BLOB` 和 `TEXT` 类型的列可以存储更大的数据，最大可达到 4GB（使用 `LONGBLOB` 和 `LONGTEXT`）。 +3. **字符编码**：`BLOB` 存储的是二进制数据，与字符编码无关。`TEXT` 存储的是字符数据，受字符编码的影响，如 `utf8`、`latin1` 等。 +4. **存储效率**：`BLOB` 由于存储的是二进制数据，不涉及字符编码转换，通常存储效率更高。`TEXT` 类型在存储时会进行字符编码转换，可能会占用更多的存储空间。 +5. **排序和比较**：`BLOB` 类型的列不能进行排序和比较，因为它们是二进制数据。`TEXT` 类型的列可以进行排序和比较，因为它们是字符数据。 + +在选择 `BLOB` 还是 `TEXT` 时，需要根据数据的特性和应用场景来决定。如果需要存储非文本的二进制数据，应选择 `BLOB`；如果需要存储大量的文本数据，则应选择 `TEXT`。 + + + +### 🎯 count(*) 和 count(1)和count(列名)区别 + +- **COUNT(\*)**：统计表中所有行的数量，包括 `NULL` 值，性能最佳，适用于需要统计总行数的情况 +- **COUNT(1)**：与 `COUNT(*)` 类似，统计表中所有行的数量，包括 `NULL` 值，性能与 `COUNT(*)` 基本相同 +- **COUNT(列名)**：统计指定列中非 `NULL` 值的数量，适用于需要统计特定列中实际值数量的情况 + + + +### 🎯 UNION和UNION ALL的区别? + +UNION和UNION ALL都是将两个结果集合并为一个，**两个要联合的SQL语句 字段个数必须一样，而且字段类型要“相容”（一致）；** + +- UNION 在进行表连接后会筛选掉重复的数据记录（效率较低），而 UNION ALL 则不会去掉重复的数据记录； + +- UNION 会按照字段的顺序进行排序，而 UNION ALL 只是简单的将两个结果合并就返回； + + + +### 🎯 SQL执行顺序 + +![The Essential Guide to SQL’s Execution Order](https://img.starfish.ink/mysql/ferrer_essential_guide_sql_execution_order_6.png) + +- 手写 + + ```mysql + SELECT DISTINCT + FROM + JOIN ON + WHERE + GROUP BY + HAVING + ORDER BY + LIMIT + ``` + +- 机读 + + ```mysql + FROM + ON + JOIN + WHERE + GROUP BY + HAVING + SELECT + DISTINCT + ORDER BY + LIMIT + ``` + + + +> mysql 的内连接、左连接、右连接有什么区别？ +> +> 什么是内连接、外连接、交叉连接、笛卡尔积呢？ + +**Join图** + +![sql-joins](https://img.starfish.ink/mysql/sql-joins.jpg) + +------ + +## 六、日志系统 📝 + +MySQL 日志其实是各种其他知识模块的基础 + +### 🎯 MySQL 都有哪些日志，分别介绍下作用，执行顺序是怎么样的？ + +> - **错误日志**：记录出错信息，也记录一些警告信息或者正确的信息。 +> +> - **通用查询日志**：记录所有对数据库请求的信息，不论这些请求是否得到了正确的执行。 +> +> - **慢查询日志**：设置一个阈值，将运行时间超过该值的所有SQL语句都记录到慢查询的日志文件中。 +> +> - **二进制日志**：记录对数据库执行更改的所有操作。 +> +> - **中继日志**：中继日志也是二进制日志，用来给slave 库恢复 +> +> - **事务日志**：重做日志 redo 和回滚日志 undo + +时序上先 undo log，redo log 先 prepare， 再写 binlog，最后再把 redo log commit + +![](https://img.starfish.ink/mysql/log-seq.png) + +### 🎯 说下 一条 MySQL 更新语句的执行流程是怎样的吧？ + +1. **解析和优化** + + - **语法解析**：MySQL解析器会对`UPDATE`语句进行语法解析，生成语法树，并检查语句的合法性。 + + - **查询优化器**：优化器会根据表的索引、统计信息等来选择最优的执行计划，以最快的方式查找需要更新的记录。 + +2. **定位记录**（这个过程也被称作**加锁读**） + + - **通过索引定位**：如果更新语句涉及索引，MySQL会利用索引快速定位需要更新的记录。如果没有索引，则会进行全表扫描。 + + - **在Buffer Pool中查找页**：MySQL首先会在Buffer Pool中查找包含目标记录的页（数据页或索引页），如果该页已经存在于Buffer Pool中，则直接使用；否则会将其从磁盘加载到 Buffer Pool 中。 + +3. **生成Undo日志** + + - **生成Undo日志**：为了支持事务的回滚，MySQL 会在更新操作前生成一条 Undo 日志，该日志记录了被更新行的**旧值**。Undo日志存储在回滚段（Rollback Segment）中。 + + - **保存Undo日志**：在发生事务回滚时，MySQL会使用Undo日志将数据恢复到更新前的状态。 + +4. **更新数据** + + - **更新Buffer Pool中的数据页**：MySQL在Buffer Pool中对定位到的记录进行更新操作，修改记录的值。这是一个内存中的操作，数据页此时还没有写回到磁盘。 + + - **标记页为脏页**：被修改的页会被标记为“脏页”，表示其内容与磁盘上的数据不同，需要在适当的时候写回磁盘。 + +5. **生成Redo日志** + + - **生成Redo日志**：Redo日志记录了数据页的物理变化，用于在数据库崩溃时恢复数据一致性。Redo日志包括这次更新操作的具体细节，如页号、偏移量和新值。 + + - **写入Redo日志缓存**：生成的Redo日志首先被写入Redo日志缓存（Log Buffer），而不是直接写入磁盘。 + + > 此时 redo log 处于 prepare 状态。然后告知执行器执行完成了，随时可以提交事务 + +6. **写入Binlog** + + - **生成Binlog**：MySQL会生成一条对应的Binlog（针对事务性表的DML语句）。Binlog记录的是这次更新的逻辑变化（类似于SQL语句或行级变化），主要用于主从复制和数据恢复。 + + - **刷写Binlog**：Binlog首先被写入到Binlog缓存，当事务提交时会将缓存中的Binlog刷写到磁盘上的Binlog文件。 + +7. **事务提交** + + - **刷写Redo日志**：在事务提交前，MySQL会将Redo日志从日志缓存刷写到磁盘的Redo日志文件（ib_logfile0, ib_logfile1）。此操作通常是顺序写入，因此效率较高。 + + - **刷写Binlog**：在事务提交时，Binlog也会被刷写到磁盘。根据MySQL的设置，可能会先刷写Binlog再刷写Redo日志，或以不同顺序执行。 + + - **两阶段提交**：为了确保Binlog和Redo日志的原子性，MySQL使用两阶段提交的机制。具体步骤是：首先写入Redo日志（处于Prepare状态），然后写入Binlog，最后更新Redo日志为Commit状态。 + +8. **数据页刷盘** + + - **异步刷盘**：脏页的刷盘（即将修改后的数据页从Buffer Pool写回到磁盘）通常是异步进行的。InnoDB存储引擎会根据脏页数量、系统空闲时间等条件定期执行刷盘操作。 + + - **检查点**：MySQL通过检查点机制来控制Redo日志的循环使用和脏页的刷盘。当达到检查点时，会将所有脏页刷写到磁盘，同时更新Redo日志的检查点信息。 + +9. **完成更新操作** + + - **事务结束**：当Redo日志和Binlog都刷写到磁盘后，事务就完成了。对于非事务性表，操作可能不会涉及Undo日志和Redo日志。 + + - **释放锁**：在事务完成后，MySQL会释放占用的锁资源。 + + + +### 🎯 两阶段提交？ + +在 MySQL 中，`UPDATE` 操作涉及对数据的修改，为了保证数据的一致性和可靠性，特别是在使用 **InnoDB** 存储引擎时，`UPDATE` 操作需要使用**两阶段提交（Two-Phase Commit）**。两阶段提交的设计主要是为了确保数据的**原子性**和**一致性**，特别是在支持事务和涉及到数据库的**Binlog**（二进制日志）记录的情况下。 + +1. **两阶段提交的原因** + + MySQL 的两阶段提交机制主要用于协调 **InnoDB 引擎的事务日志（Redo Log）** 和 **MySQL 的 Binlog**。当数据库发生崩溃时，保证事务的持久性和一致性是非常重要的。事务的执行不能一部分成功，另一部分失败，这会导致数据不一致。 + + **关键问题：** + + - **Redo Log** 是 InnoDB 的内部事务日志，确保事务的**持久性**（即便崩溃后数据也能恢复）。 + + - **Binlog** 是 MySQL 的二进制日志，主要用于数据的**复制**和**备份**。 + + 如果没有两阶段提交，在执行 `UPDATE` 操作时可能会出现以下问题： + + - 写入 Binlog 成功，但事务日志（Redo Log）失败**：**事务被记录在 Binlog 中，但实际修改的数据没有持久化，导致数据不一致。 + - 写入 Redo Log 成功，但 Binlog 失败：数据修改在本地持久化了，但 Binlog 失败，导致主从复制出现问题，主库和从库数据不一致。 + +​ 因此，MySQL 使用两阶段提交来确保在事务修改数据的过程中，**Binlog 和 Redo Log 同时成功**，从而保证事务的一致性。 + +2. **两阶段提交的流程** + + MySQL 在执行 `UPDATE` 操作时，InnoDB 和 Binlog 通过两阶段提交进行协调。具体步骤如下： + + a. 第一阶段：预提交阶段（Prepare Phase） + + - **开始事务**：InnoDB 首先会执行 `UPDATE` 操作，并将修改记录到内存中的 Undo Log，以便在事务失败时可以回滚。 + + - **记录 Redo Log（prepare）**：修改数据的操作被记录在**Redo Log**中，并标记为 "Prepare" 状态，此时数据尚未真正提交，只是标记为“准备提交”。 + + - **等待事务进入准备状态**：InnoDB 引擎等着上层的 MySQL Server 层确认 Binlog 日志写入成功。 + + 此时，事务还没有被真正提交，但 Redo Log 已经有了准备提交的记录。 + + b. 第二阶段：提交阶段（Commit Phase） + + - **写入 Binlog**：MySQL Server 层会将这次 `UPDATE` 操作记录到 Binlog 中，确保日志记录成功。 + - **提交事务**：当 Binlog 成功写入后，MySQL Server 会通知 InnoDB 提交事务，InnoDB 会将 Redo Log 的状态从 "Prepare" 变为 "Commit"。 + - **持久化 Redo Log**：最后，InnoDB 将 Redo Log 的状态变为已提交，保证数据的修改真正生效，并确保即使系统崩溃，事务也可以在恢复过程中继续完成。 + +​ 通过这样的两阶段提交，MySQL 确保了 Binlog 和 Redo Log 都正确记录，避免了系统崩溃时数据不一致的问题。 + +3. 为什么需要两阶段提交？ + + 两阶段提交的必要性源自于以下几点： + + - **数据的一致性**：事务的修改操作必须同时写入 InnoDB 的事务日志和 MySQL 的 Binlog，确保数据和日志保持一致，尤其是在崩溃恢复或主从复制的场景下。 + - **事务的原子性**：保证要么事务的所有操作都成功，要么所有操作都回滚，不会出现事务部分提交的情况。 + - **崩溃恢复**：在崩溃恢复时，InnoDB 可以依靠已提交的 Redo Log 来恢复数据，而 Binlog 用于恢复操作步骤，并保持主从数据库同步。 + +4. 解决的问题 + + - **防止数据不一致**：没有两阶段提交机制，可能会导致数据和日志之间的不一致，尤其在分布式环境下，数据库的主从复制和日志备份可能会出现问题。 + - **确保事务完整性**：确保所有日志（Binlog 和 Redo Log）和数据一致，保证事务在任何情况下都能正确提交或回滚。 + + + +### 🎯 说说 redo log 、undo log 和 bin log ？ + +在 MySQL 中，特别是在使用 InnoDB 存储引擎时，`redo log`（重做日志）、`undo log`（回滚日志）和 `binlog`（二进制日志）各自承担着不同的角色： + +**1. Redo Log（重做日志）：** + +- **目的**：确保事务的持久性。在系统崩溃后，`redo log` 允许恢复未提交的事务更改，保证数据的完整性和一致性。 +- **内容**：记录了事务对数据页所做的物理更改【**物理日志**】，以便在崩溃恢复时重新应用这些更改。 +- **写入时机**：在事务提交时，将更改刷新到磁盘上的 `redo log` 文件中。 +- **大小和循环**：`redo log` 通常配置为固定大小的日志文件，并且可以循环使用。 +- **用途**：**崩溃恢复**，在数据库崩溃后，通过 redo log 恢复到崩溃前的状态，保证数据一致性。 + +> redo log 不需要写磁盘吗？如果 redo log 也要写磁盘，干嘛不直接修改数据呢？redo log 是需要写磁盘的，但是 redo log 是**顺序写**的，所以也是 WAL（writeahead-log） 的一种。 +> +> redo log 本身也是先写进 redo log buffer，后面再刷新到操作系统的 page cache，或者一步到位刷新到磁盘 +> +> InnoDB 引擎本身提供了参数 `innodb_flush_log_at_trx_commit` 来控制写到磁盘的时机，里面有三个不同值。 +> +> - 0：每秒刷新到磁盘，是从 redo log buffer 到磁盘。 +> - 1：每次提交的时候刷新到磁盘上，也就是最安全的选项，InnoDB 的**默认值**。 +> - 2：每次提交的时候刷新到 page cache 里，依赖于操作系统后续刷新到磁盘。 + +**2. Undo Log（回滚日志）：** + +- **目的**：提供事务的原子性和一致性。它允许撤销事务的更改，以保持数据的一致状态。 +- **内容**：记录了事务对数据页所做的更改的逆操作【**逻辑日志**】，使得在事务失败或需要回滚时可以恢复原始数据。 +- **写入时机**：当事务进行修改操作时，`undo log` 会记录这些更改的逆操作，通常在事务提交前就已经写入。 +- **用途**：主要用于 MVCC（多版本并发控制）和事务回滚。 + +**3. Binlog（二进制日志）：** + +- **目的**：记录数据库的所有修改操作，用于数据恢复、主从复制和数据审计。 +- **内容**：记录了所有修改数据的 SQL 语句，如 `INSERT`、`UPDATE` 和 `DELETE`，但不记录 `SELECT` 和 `SHOW` 这类的语句。 +- **写入时机**：在 SQL 语句执行后，根据配置，`binlog` 可以立即或事务提交时写入磁盘。 +- **大小和存储**：`binlog` 文件通常不循环使用，它们会随着时间持续增长，直到通过配置的策略进行清理。 + +| 特性 | Redo Log | Undo Log | Bin Log | +| -------- | ---------------------- | -------------------------------- | ---------------------- | +| 主要用途 | 崩溃恢复 | 事务回滚、多版本并发控制（MVCC） | 主从复制和数据恢复 | +| 日志类型 | 物理日志 | 逻辑日志 | 逻辑日志 | +| 存储内容 | 页级物理更改 | 数据快照 | SQL 语句或行级数据变化 | +| 写入方式 | 循环写入 | 按需写入 | 追加写入 | +| 写入时机 | 事务提交时 | 事务操作时 | 事务提交时 | +| 大小 | 固定大小 | 可变大小 | 可变大小 | +| 作用 | 保证数据一致性和持久性 | 提供事务回滚和一致性读支持 | 实现数据复制和备份 | + + + +### 🎯 MySQL 的 binlog 有几种录入格式？分别有什么区别？ + +`binlog`日志有三种格式，分别为`STATMENT`、`ROW`和`MIXED`。 + +> 在 `MySQL 5.7.7`之前，默认的格式是`STATEMENT`，`MySQL 5.7.7`之后，默认值是 `ROW`。日志格式通过 `binlog-format` 指定。 + +- `STATMENT` ：基于 SQL 语句的复制(`statement-based replication, SBR`)，每一条会修改数据的 sql 语句会记录到 binlog 中**。** + - 优点：不需要记录每一行的变化，减少了 binlog 日志量，节约了 IO，从而提高了性能； + - 缺点：在某些情况下会导致主从数据不一致，比如执行`sysdate()`、`slepp()`等。 + +- `ROW` ：基于行的复制(`row-based replication, RBR`)，不记录每条sql语句的上下文信息，仅需记录哪条数据被修改了**。 ** + - 优点：不会出现某些特定情况下的存储过程、或function、或trigger的调用和触发无法被正确复制的问题**；** + - 缺点：会产生大量的日志，尤其是 `alter table` 的时候会让日志暴涨 + +- `MIXED` ：基于 STATMENT 和 ROW 两种模式的混合复制(`mixed-based replication, MBR`)，mixed 格式的意思是，MySQL 自己会判断这条 SQL 语句是否可能引起主备不一致，如果有可能，就用 row 格式，否则就用 statement 格式 + +------ + + + +## 七、性能调优 ⚡ + +### 🎯 影响 MySQL 的性能因素有哪些? + +- 业务需求对MySQL的影响(合适合度) + +- 存储定位对MySQL的影响 + - 不适合放进MySQL的数据 + - 二进制多媒体数据 + - 流水队列数据 + - 超大文本数据 + - 需要放进缓存的数据 + - 系统各种配置及规则数据 + - 活跃用户的基本信息数据 + - 活跃用户的个性化定制信息数据 + - 准实时的统计信息数据 + - 其他一些访问频繁但变更较少的数据 + +- Schema设计对系统的性能影响 + - 尽量减少对数据库访问的请求 + - 尽量减少无用数据的查询请求 + +- 硬件环境对系统性能的影响 + + + +### 🎯 日常工作中你是怎么优化SQL的？ + +优化SQL的日常工作可以从以下几个方面进行： + +1. **索引优化** + + - **创建索引**：为频繁查询的列创建合适的索引，特别是主键和外键列。 + + - **使用覆盖索引**：通过索引来满足查询需求，避免回表查询。 + + - **删除冗余索引**：清理不常用或重复的索引，以减少维护开销。 + + +2. **查询优化** + + - **避免选择所有列**：只选择需要的列，避免使用 `SELECT *`。 + + - **使用合适的SQL语法**：如使用 `JOIN` 替代子查询，避免N+1查询问题。 + + - **优化WHERE子句**：使用索引列进行过滤，避免在过滤条件中进行函数运算或转换。 + + +3. **数据库结构优化** + + - **规范化与反规范化**：根据实际需求选择合适的数据库设计，平衡数据冗余和查询性能。 + + - **分区表**：对于大表，使用分区技术来提升查询和维护性能。 + + +4. **缓存机制** + + - **应用层缓存**：如Memcached或Redis，缓存频繁访问的数据，减少数据库负载。 + + - **数据库缓存**：合理设置数据库缓存参数，优化数据库内存使用。 + + +5. **SQL分析与监控** + + - **执行计划分析**：使用 `EXPLAIN` 分析SQL的执行计划，了解查询的执行步骤和时间。 + + - **慢查询日志**：启用慢查询日志，找出并优化执行时间较长的SQL语句。 + + - **性能监控工具**：如使用New Relic、APM等工具，持续监控数据库性能。 + + +6. **事务控制** + + - **减少事务范围**：尽量缩小事务的范围和持续时间，避免长时间锁定资源。 + + - **合理设置隔离级别**：根据业务需求选择合适的事务隔离级别，平衡并发性和数据一致性。 + + +7. **数据库参数调整** + + - **调整连接池**：合理设置数据库连接池大小，避免过多连接导致资源争用。 + + - **优化数据库配置**：根据硬件资源和业务需求，调整数据库内存、缓存、IO等参数。 + + +通过以上方法，可以有效地优化SQL性能，提高数据库的响应速度和稳定性。持续关注数据库的运行情况，并根据实际需求进行调整，是保持数据库高效运行的关键。 + + + +### 🎯 什么是最左前缀原则？什么是最左匹配原则？ + +**最左前缀原则** 和 **最左匹配原则** 是在使用索引时的两个相关概念，它们通常与复合索引（即在多个列上创建的索引）的使用相关： + +1. **最左前缀原则**： + - 当使用复合索引时，最左前缀原则指的是查询优化器只会使用复合索引中最左边的列（或前几列）来查找数据。 + - 这意味着，如果要利用复合索引，查询条件中必须包含最左边的列。如果查询条件不包含最左边的列，那么索引将不会被使用，或者不会完全被使用。 +2. **最左匹配原则**： + - 最左匹配原则是指在复合索引中，只有当前面的列匹配后，才能继续向后匹配。换句话说，只有当前一个列的值已经确定，才能利用下一个列的索引。 + - 例如，如果你有一个 (`col1`, `col2`, `col3`) 的复合索引，那么只有当 `col1` 的值确定后，`col2` 的索引才会被使用；同样，只有当 `col1` 和 `col2` 的值都确定后，`col3` 的索引才会被使用。 + +> 在 MySQL 中，**联合索引 (a, b, c)** 的最左匹配原则使用情况如下： +> +> | **查询条件** | **是否使用索引** | **使用索引的字段** | **底层原因** | +> | --------------------------- | -------------------- | ------------------ | -------------------------------------------- | +> | `WHERE a=1 AND b=2 AND c=3` | ✅ 完全使用 | `a, b, c` | 严格遵循最左顺序，B+ 树逐层定位 | +> | `WHERE c=3 AND b=2 AND a=1` | ✅ 完全使用 | `a, b, c` | 优化器调整顺序后等价于 `a=1 AND b=2 AND c=3` | +> | `WHERE a=1 AND c=3` | ⚠️ 部分使用（仅 `a`） | `a` | 跳过 `b`，`c` 无法直接通过索引定位 | +> | `WHERE b=2 AND c=3` | ❌ 索引失效 | 无 | 未包含最左列 `a`，无法触发索引路径 | + + + +### 🎯 MySQL常见性能分析手段？ + +**MySQL Query Optimizer** + +1. MySQL 中有专门负责优化 SELECT 语句的优化器模块，主要功能：通过计算分析系统中收集到的统计信息，为客户端请求的 Query 提供他认为最优的执行计划（他认为最优的数据检索方式，但不见得是 DBA 认为是最优的，这部分最耗费时间） + +2. 当客户端向 MySQL 请求一条 Query，命令解析器模块完成请求分类，区别出是 SELECT 并转发给 MySQL Query Optimizer 时，MySQL Query Optimizer 首先会对整条 Query 进行优化，处理掉一些常量表达式的预算，直接换算成常量值。并对 Query 中的查询条件进行简化和转换，如去掉一些无用或显而易见的条件、结构调整等。然后分析 Query 中的 Hint 信息（如果有），看显示 Hint 信息是否可以完全确定该 Query 的执行计划。如果没有 Hint 或 Hint 信息还不足以完全确定执行计划，则会读取所涉及对象的统计信息，根据 Query 进行写相应的计算分析，然后再得出最后的执行计划。 + +**MySQL常见瓶颈** + +- CPU：CPU 在饱和的时候一般发生在数据装入内存或从磁盘上读取数据时候 + +- IO：磁盘 I/O 瓶颈发生在装入数据远大于内存容量的时候 + +- 服务器硬件的性能瓶颈：top，free，iostat 和 vmstat 来查看系统的性能状态 + +**性能下降SQL慢 执行时间长 等待时间长 原因分析** + +- 查询语句写的烂 +- 索引失效（单值、复合） +- 关联查询太多 join（设计缺陷或不得已的需求） +- 服务器调优及各个参数设置（缓冲、线程数等） + +在优化 MySQL 时，通常需要对数据库进行分析，常见的分析手段有**慢查询日志**，**EXPLAIN 分析查询**，**profiling分析**以及**show命令查询系统状态及系统变量**，通过定位分析性能的瓶颈，才能更好的优化数据库系统的性能。 + + + +### 🎯 性能瓶颈定位 + +我们可以通过 show 命令查看 MySQL 状态及变量，找到系统的瓶颈： + +```mysql +Mysql> show status ——显示状态信息（扩展show status like ‘XXX’） + +Mysql> show variables ——显示系统变量（扩展show variables like ‘XXX’） + +Mysql> show innodb status ——显示InnoDB存储引擎的状态 + +Mysql> show processlist ——查看当前SQL执行，包括执行状态、是否锁表等 + +Shell> mysqladmin variables -u username -p password——显示系统变量 + +Shell> mysqladmin extended-status -u username -p password——显示状态信息 +``` + + + +### 🎯 怎么看执行计划（explain），如何理解其中各个字段的含义？ + +使用 **Explain** 关键字可以模拟优化器执行 SQL 查询语句，从而知道 MySQL 是如何处理你的 SQL 语句的。分析你的查询语句或是表结构的性能瓶颈 + +> ```sql +> create table t1 +> ( +> id int auto_increment primary key, +> col1 varchar(100) null, +> col2 int null, +> col3 varchar(100) null, +> part1 varchar(100) null, +> part2 varchar(100) null, +> part3 varchar(100) null, +> common_field varchar(100) null, +> constraint idx_key2 unique (col2) +> )charset = utf8mb3; +> +> create index idx_key1 +> on t1 (col1); +> +> create index idx_key3 +> on t1 (col3); +> +> create index idx_key_part +> on t1 (part1, part2, part3); +> ``` + +各字段解释 + +- **id**（select 查询的序列号，包含一组数字，表示查询中执行 select 子句或操作表的顺序） + + - id 相同，执行顺序从上往下 + - id 不同，数字越大，越优先执行（子查询会有更大的 id） + - id 相同不同，同时存在，相同的属于一组，从上往下执行 + +- **select_type**（查询的类型，用于区别普通查询、联合查询、子查询等复杂查询） + + - **SIMPLE** ：简单的 select 查询，查询中不包含子查询或 UNION + - **PRIMARY**：查询中若包含任何复杂的子部分，最外层查询被标记为 PRIMARY + - **SUBQUERY**：在 select 或 where 列表中包含了子查询 + - **DERIVED**：在 from 列表中包含的子查询被标记为 DERIVED，mysql 会递归执行这些子查询，把结果放在临时表里 + - **UNION**：若第二个 select 出现在 UNION 之后，则被标记为 UNION，若 UNION 包含在 from 子句的子查询中，外层 select 将被标记为 DERIVED + - **UNION RESULT**：从 UNION 表获取结果的 select + +- **table**（显示这一行的数据是关于哪张表的） + +- **partitions**（匹配的分区信息，高版本才有的） + +- **type**（显示查询使用了那种类型，从最好到最差依次排列 **system > const > eq_ref > ref > fulltext > ref_or_null > index_merge > unique_subquery > index_subquery > range > index > ALL** ） + + - system：表只有一行记录（等于系统表），是 const 类型的特例，平时不会出现 + - const：通过主键或唯一索引查找时，表中最多返回一条数据 + - eq_ref：唯一性索引扫描，对于每个索引键，表中只有一条记录与之匹配，常见于主键或唯一索引扫描 + - ref：使用非唯一性索引查询，返回匹配某个索引值的所有行 + - ref_or_null：当对普通二级索引进行等值匹配查询，该索引列的值也可以是`NULL`值时，那么对该表的访问方法就可能是ref_or_null + - index_merge: 在某些场景下可以使用`Intersection`、`Union`、`Sort-Union`这三种索引合并的方式来执行查询 + - range：只检索给定范围的行，使用一个索引来选择行。key 列显示使用了哪个索引，一般就是在你的 where 语句中出现了between、<、>、in等的查询，这种范围扫描索引比全表扫描要好，因为它只需开始于索引的某一点，而结束于另一点，不用扫描全部索引 + - index：全表扫描,但仅扫描索引树（**也就是说虽然 all 和 index 都是读全表，但 index 是从索引中读取的，而 all 是从硬盘中读的**） + - all：全表扫描，将遍历全表找到匹配的行 + + > 一般来说，得保证查询至少达到 range 级别，最好到达 ref + +- **possible_keys**（显示可能应用在这张表中的索引，一个或多个，查询涉及到的字段若存在索引，则该索引将被列出，但不一定被查询实际使用） + +- **key** + + - 实际使用的索引，如果为NULL，则没有使用索引 + + - **查询中若指定了使用了覆盖索引，则该索引和查询的 select 字段重叠，仅出现在 key 列表中**![](https://img.starfish.ink/mysql/explain-key.png) + +- **key_len** + + - 表示索引中使用的字节数，可通过该列计算查询中使用的索引的长度。在不损失精确性的情况下，长度越短越好 + - key_len 显示的值为索引字段的最大可能长度，并非实际使用长度，即 key_len 是根据表定义计算而得，不是通过表内检索出的 + +- **ref** （显示索引的哪一列被使用了，如果可能的话，是一个常数。哪些列或常量被用于查找索引列上的值）![](https://img.starfish.ink/mysql/explain-ref.png) + +- **rows** （根据表统计信息及索引选用情况，大致估算找到所需的记录所需要读取的行数） + +- **filtered**（某个表经过搜索条件过滤后剩余记录条数的百分比） + +- **Extra**（包含不适合在其他列中显示但十分重要的额外信息） + + 额外信息有好几十个，我们看几个常见的 + + 1. `using filesort`：说明 MySQL 会对数据使用一个外部的索引排序，不是按照表内的索引顺序进行读取。MySQL 中无法利用索引完成的排序操作称为“文件排序”![](https://img.starfish.ink/mysql/explain-extra-using-filesort.png) + + 2. `Using temporary`：使用了临时表保存中间结果，比如去重、排序之类的，比如我们在执行许多包含`DISTINCT`、`GROUP BY`、`UNION`等子句的查询过程中，如果不能有效利用索引来完成查询，`MySQL`很有可能寻求通过建立内部的临时表来执行查询。![](https://img.starfish.ink/mysql/explain-extra-using-tmp.png) + + 3. `using index`：表示相应的 select 操作中使用了覆盖索引，避免访问了表的数据行，效率不错，如果同时出现 `using where`，表明索引被用来执行索引键值的查找；否则索引被用来读取数据而非执行查找操作![](https://img.starfish.ink/mysql/explain-extra-using-index.png) + + 4. `using where`：当某个搜索条件需要在`server层`进行判断时![](https://img.starfish.ink/mysql/explain-extra-using-where.png) + + 5. `using join buffer`：使用了连接缓存![](https://img.starfish.ink/mysql/explain-extra-using-join-buffer.png) + + 6. `impossible where`：where 子句的值总是 false，不能用来获取任何元祖![](https://img.starfish.ink/mysql/explain-extra-impossible-where.png) + + 7. `Using index condition` : 查询使用了索引，但是查询条件不能完全由索引本身来满足![](https://img.starfish.ink/mysql/explain-extra-using-index-condition.png) + + `Using index condition `通常出现在以下几种情况： + + - **索引条件下推（Index Condition Pushdown, ICP）**：这是 MySQL 的一个优化策略，它将查询条件的过滤逻辑“下推”到存储引擎层，而不是在服务器层处理。这样可以减少从存储引擎检索的数据量，从而提高查询效率。 + - **部分索引**：当查询条件只涉及索引的一部分列时，MySQL 可以使用索引来快速定位到满足条件的行，但是可能需要回表（即访问表的实际数据行）来检查剩余的条件。 + - **复合索引**：在使用复合索引（即索引包含多个列）的情况下，如果查询条件只匹配索引的前几列，那么剩余的列可能需要通过 `Using index condition` 来进一步过滤。 + + 8. `select tables optimized away`：在没有 group by 子句的情况下，基于索引优化操作或对于 MyISAM 存储引擎优化COUNT(*) 操作，不必等到执行阶段再进行计算，查询执行计划生成的阶段即完成优化 + + 9. `distinct`：优化 distinct 操作，在找到第一匹配的元祖后即停止找同样值的动作 + +> 在 MySQL 的 EXPLAIN 输出中，`possible_keys` 和 `key` 这两个字段看似相似，但实际上提供了不同的信息： +> +> - `possible_keys`：显示 MySQL 可能使用哪些索引来查找表中的行。 +> - `key`：显示 MySQL 实际决定使用的索引。 +> +> 为什么两者都需要？ +> +> 1. **优化潜力**： `possible_keys` 显示了所有可能的选项，而 `key` 显示了优化器最终的选择。这种对比可以帮助数据库管理员（DBA）了解是否有更好的索引选择。 +> 2. **查询分析**： 通过比较 `possible_keys` 和 `key`，我们可以了解优化器的决策过程，这对于复杂查询的优化非常有用。 +> 3. **索引使用情况**： `possible_keys` 可能列出多个索引，而 `key` 只会显示实际使用的一个。这有助于识别冗余索引或缺失的索引。 +> 4. **优化器行为**： 有时，`possible_keys` 可能列出多个索引，但 `key` 可能是 NULL，这表明 MySQL 认为不使用索引更有效。 + + + +### 🎯 一般你们怎么建 MySQL 索引，基于什么原则，遇到过索引失效的情况么，怎么优化的? + +#### 1. MySQL索引建立的原则 + +> 1. 最左前缀匹配原则，非常重要的原则，MySQL 会一直向右匹配直到遇到范围查询(>、<、between、like)就停止匹配，比如 `a = 1 and b = 2 and c > 3 and d = 4` 如果建立(a,b,c,d)顺序的索引，d是用不到索引的，如果建立(a,b,d,c)的索引则都可以用到，a,b,d的顺序可以任意调整。 +> 2. =和in可以乱序，比如 `a = 1 and b = 2 and c = 3` 建立(a,b,c)索引可以任意顺序，mysql 的查询优化器会帮你优化成索引可以识别的形式。 +> 3. 尽量选择**区分度高**的列作为索引，区分度的公式是 `区分度 = distinct(col)/count(*)`，表示字段不重复的比例，比例越大我们扫描的记录数越少，唯一键的区分度是1，而一些状态、性别字段可能在大数据面前区分度就是0，那可能有人会问，这个比例有什么经验值吗？使用场景不同，这个值也很难确定，一般需要join的字段我们都要求是0.1以上，即平均1条扫描10条记录。 +> 4. 索引列不能参与计算，保持列“干净”，比如 `from_unixtime(create_time) = ’2014-05-29’` 就不能使用到索引，原因很简单，b+树中存的都是数据表中的字段值，但进行检索时，需要把所有元素都应用函数才能比较，显然成本太大。所以语句应该写成 `create_time = unix_timestamp(’2014-05-29’)`。 +> 5. 尽量的扩展索引，不要新建索引。比如表中已经有 a 的索引，现在要加 (a,b) 的索引，那么只需要修改原来的索引即可。 + +- **选择合适的列**：优先考虑在查询条件、排序（`ORDER BY`）、分组（`GROUP BY`）、连接（`JOIN`）中频繁使用的列上建立索引。这些列通常对查询性能有显著影响。 +- **唯一性**：在唯一性要求较高的列上建立唯一索引（`UNIQUE`），如ID号、邮箱等。唯一索引不仅能加快查询速度，还能保证数据的唯一性。 +- **覆盖索引**：尽量选择创建覆盖索引（即索引包含了查询所需的所有列），避免回表操作。这样可以显著提升查询性能，尤其是涉及多列的查询。 +- **前缀索引**：对于文本类型（如`VARCHAR`、`TEXT`）的列，如果列值较长且前缀具有较高区分度，可以使用前缀索引来节省空间和提高查询效率。 +- **复合索引**：在多个列上建立复合索引（即组合索引），以优化涉及多列的查询。但要注意列的顺序，通常将选择性更高的列放在最前面。 +- **考虑查询频率**：根据查询的频率和响应时间要求，对高频查询的列进行索引优化。在写操作频繁的表上，要平衡索引的数量，以避免过多索引导致的插入和更新性能下降。 +- **避免过多索引**：虽然索引可以加快查询速度，但过多的索引会增加插入、更新、删除操作的成本。因此，需要在查询性能和写性能之间取得平衡。 + +#### 2. 常见的索引失效情况 + +- **查询条件中使用了函数或表达式**：当在查询条件中对索引列使用了函数或表达式（如`UPPER(column_name)`），索引可能失效。MySQL需要扫描所有行来计算函数结果，导致全表扫描。 +- **隐式类型转换**：如果查询条件中的列和参数类型不匹配（如将字符串列与数字比较），MySQL会进行隐式类型转换，导致索引失效。 +- **模糊查询以通配符开头**：使用`LIKE '%value'`形式的模糊查询时，由于通配符位于开头，索引无法使用，MySQL需要进行全表扫描。 +- **索引列不在最左侧**：对于复合索引，如果查询条件中未使用复合索引的最左侧列，索引将无法使用（“最左前缀”原则）。 +- **`OR`条件未全部使用索引**：如果查询条件中有`OR`，且每个条件都未使用索引，则MySQL无法利用索引，需要进行全表扫描。 +- **查询条件中有NULL值**：在某些情况下，索引列的查询条件中如果包含`IS NULL`或`IS NOT NULL`，可能会导致索引失效。 +- **查询条件中使用不等号**：使用`<>`或`!=`查询条件时，MySQL可能会选择不使用索引，因为这种条件通常需要扫描大量行。 + +#### 3. 索引优化方法 + +- **重构查询**：避免在索引列上使用函数、表达式、隐式类型转换等操作。尽可能让查询条件直接作用于索引列，确保索引生效。 +- **使用合适的类型**：确保查询条件中的类型与列的类型匹配，避免隐式类型转换。 +- **合理使用通配符**：对于模糊查询，尽量避免通配符开头。如果业务允许，可以考虑在应用层进行拆分查询或引入全文索引（`FULLTEXT`）来处理文本搜索。 +- **优化复合索引的顺序**：根据查询条件的使用情况，调整复合索引的列顺序，确保最左前缀列经常在查询条件中使用。 +- **拆分复杂查询**：对于使用`OR`的复杂查询，可以尝试将查询拆分为多个子查询，并使用`UNION`合并结果，确保每个子查询都能利用索引。 +- **使用覆盖索引**：如果可能，创建覆盖索引，使查询能够直接从索引中获取所需数据，避免回表，提高查询效率。 +- **分析查询性能**：使用`EXPLAIN`命令分析查询的执行计划，检查索引是否被使用。根据执行计划的结果，调整索引设计和查询语句。 +- **定期维护索引**：对于频繁更新的表，定期进行索引重建或优化，以保持索引结构的高效性。 + +通过合理的索引设计和优化，可以显著提高MySQL的查询性能。但在实际应用中，需要根据具体的业务需求、数据量和查询频率等因素来灵活调整索引策略，避免索引失效带来的性能问题。 -**死锁恢复**：死锁发生以后，只有部分或完全回滚其中一个事务，才能打破死锁，InnoDB目前处理死锁的方法是，将持有最少行级排他锁的事务进行回滚。所以事务型应用程序在设计时必须考虑如何处理死锁，多数情况下只需要重新执行因死锁回滚的事务即可。 -**外部锁的死锁检测**：发生死锁后，InnoDB 一般都能自动检测到，并使一个事务释放锁并回退，另一个事务获得锁，继续完成事务。但在涉及外部锁，或涉及表锁的情况下，InnoDB 并不能完全自动检测到死锁， 这需要通过设置锁等待超时参数 innodb_lock_wait_timeout 来解决 -**死锁影响性能**：死锁会影响性能而不是会产生严重错误，因为InnoDB会自动检测死锁状况并回滚其中一个受影响的事务。在高并发系统上，当许多线程等待同一个锁时，死锁检测可能导致速度变慢。 有时当发生死锁时，禁用死锁检测（使用innodb_deadlock_detect配置选项）可能会更有效，这时可以依赖`innodb_lock_wait_timeout`设置进行事务回滚。 +> ##### 如何写sql能够有效的使用到复合索引？ +> +> 1. 全值匹配我最爱 +> +> 2. **最佳左前缀法则**，比如建立了一个联合索引(a,b,c)，那么其实我们可利用的索引就有(a), (a,b), (a,b,c) +> +> 3. 不在索引列上做任何操作（计算、函数、(自动or手动)类型转换），会导致索引失效而转向全表扫描 +> +> 4. 存储引擎不能使用索引中范围条件右边的列 +> +> 5. 尽量使用**覆盖索引**(只访问索引的查询(索引列和查询列一致))，减少select +> +> 6. is null ,is not null 也无法使用索引 +> +> 7. like "xxxx%" 是可以用到索引的，like "%xxxx" 则不行(like "%xxx%" 同理)。like以通配符开头('%abc...')索引失效会变成全表扫描的操作， +> +> 8. 字符串不加单引号索引失效 +> +> 9. 少用or，用它来连接时会索引失效 +> +> 10. <，<=，=，>，>=，BETWEEN，IN 可用到索引，<>，not in ，!= 则不行，会导致全表扫描 +> +> 11. 前缀索引：前缀索引就是用某个字段中，字符串的前几个字符建立索引，比如我们可以在订单表上对商品名称字段的前 5 个字符建立索引。使用前缀索引是为了减小索引字段大小，可以增加一个索引页中存储的索引值，有效提高索引的查询速度。在一些大字符串的字段作为索引时，使用前缀索引可以帮助我们减小索引项的大小。 +> +> 但是，前缀索引有一定的局限性，例如 order by 就无法使用前缀索引，无法把前缀索引用作覆盖索引。 -**MyISAM避免死锁**： +### 🎯 一条sql执行过长的时间，你如何优化，从哪些方面？ -- 在自动加锁的情况下，MyISAM 总是一次获得 SQL 语句所需要的全部锁，所以 MyISAM 表不会出现死锁。 +1. 查看sql是否涉及多表的联表或者子查询，如果有，看是否能进行业务拆分，相关字段冗余或者合并成临时表（业务和算法的优化） +2. 涉及连表的查询，是否能进行分表查询，单表查询之后的结果进行字段整合 +3. 如果以上两种都不能操作，非要连表查询，那么考虑对相对应的查询条件做索引。加快查询速度 +4. 针对数量大的表进行历史表分离（如交易流水表） +5. 数据库主从分离，读写分离，降低读写针对同一表同时的压力，至于主从同步，mysql有自带的binlog实现 主从同步 +6. explain分析sql语句，查看执行计划，分析索引是否用上，分析扫描行数等等 +7. 查看mysql执行日志，看看是否有其他方面的问题 -**InnoDB避免死锁**： -- 为了在单个InnoDB表上执行多个并发写入操作时避免死锁，可以在事务开始时通过为预期要修改的每个元祖（行）使用`SELECT ... FOR UPDATE`语句来获取必要的锁，即使这些行的更改语句是在之后才执行的。 -- 在事务中，如果要更新记录，应该直接申请足够级别的锁，即排他锁，而不应先申请共享锁、更新时再申请排他锁，因为这时候当用户再申请排他锁时，其他事务可能又已经获得了相同记录的共享锁，从而造成锁冲突，甚至死锁 -- 如果事务需要修改或锁定多个表，则应在每个事务中以相同的顺序使用加锁语句。 在应用中，如果不同的程序会并发存取多个表，应尽量约定以相同的顺序来访问表，这样可以大大降低产生死锁的机会 -- 通过`SELECT ... LOCK IN SHARE MODE`获取行的读锁后，如果当前事务再需要对该记录进行更新操作，则很有可能造成死锁。 -- 改变事务隔离级别 -如果出现死锁，可以用 `show engine innodb status; `命令来确定最后一个死锁产生的原因。返回结果中包括死锁相关事务的详细信息，如引发死锁的 SQL 语句，事务已经获得的锁，正在等待什么锁，以及被回滚的事务等。据此可以分析死锁产生的原因和改进措施。 +### 🎯 大表优化思路？ ------- +大表优化我一般从几个层面考虑： +1. **存储层面**：分库分表、冷热数据分离； +2. **索引层面**：建合适的联合索引、覆盖索引，必要时用分区表； +3. **SQL 层面**：避免全表扫描、避免函数导致索引失效、批量处理代替逐行； +4. **维护层面**：归档历史数据、定期优化表结构； +5. **架构层面**：读写分离、引入缓存、甚至用 Elasticsearch/ClickHouse 处理分析类查询。 + 通过这些手段，把大表的单次查询和维护压力控制在可接受范围内。 -## 八、MySQL调优 -> 日常工作中你是怎么优化SQL的？ -> -> SQL优化的一般步骤是什么，怎么看执行计划（explain），如何理解其中各个字段的含义？ -> -> 如何写sql能够有效的使用到复合索引？ +### 🎯 数据库是如何调优的？ + +> 数据库调优我一般从五个层次来考虑： > -> 一条sql执行过长的时间，你如何优化，从哪些方面入手？ +> 1. **架构层**：比如读写分离、分库分表、加缓存，把整体压力降下来。 +> 2. **索引优化**：结合最左前缀原则，避免索引失效，尽量用覆盖索引减少回表。 +> 3. **SQL 优化**：用 explain 看执行计划，避免 select *、大事务，分页优化。 +> 4. **参数调优**：调整 buffer_pool、连接池、慢查询日志。 +> 5. **硬件和运维**：SSD、加内存、监控慢 SQL。 > -> 什么是最左前缀原则？什么是最左匹配原则？ +> 在项目里，我做过分页优化，把 `limit offset` 深分页改成基于主键范围查询，单次查询延迟从几秒降到几十毫秒；也做过索引重构，把一个多表 join 改成冗余字段 + 单表查询，性能提升明显。 -### 影响mysql的性能因素 +这个问题很常见，面试官想听你是否系统性理解过数据库调优（不仅仅是写 `explain` 看执行计划，而是从 **架构、SQL、索引、参数、硬件** 全链路考虑） -- 业务需求对MySQL的影响(合适合度) +**数据库调优思路（五个层次）** -- 存储定位对MySQL的影响 - - 不适合放进MySQL的数据 - - 二进制多媒体数据 - - 流水队列数据 - - 超大文本数据 - - 需要放进缓存的数据 - - 系统各种配置及规则数据 - - 活跃用户的基本信息数据 - - 活跃用户的个性化定制信息数据 - - 准实时的统计信息数据 - - 其他一些访问频繁但变更较少的数据 +**1. 架构层面** -- Schema设计对系统的性能影响 - - 尽量减少对数据库访问的请求 - - 尽量减少无用数据的查询请求 +- **读写分离**：主库写，从库读，缓解单库压力。 +- **分库分表**：数据量过大时按业务或范围拆分（如订单库按用户 ID hash 分表）。 +- **引入缓存**：热点数据放在 Redis，减少 DB 压力。 +- **异步化**：非核心操作走消息队列（MQ），削峰填谷。 -- 硬件环境对系统性能的影响 +**2. 索引优化** +- 合理建立 **联合索引**，遵循最左前缀原则。 +- 避免索引失效（如函数操作、隐式类型转换、`!=`、`like '%xx'`）。 +- 使用 **覆盖索引**，减少回表操作。 +- 定期分析表和索引碎片，必要时 `analyze table` 或重建索引。 +**3. SQL 优化** -### 性能分析 +- 使用 `EXPLAIN` 查看执行计划，关注 `type`、`rows`、`Extra`。 +- 避免 `select *`，只查需要的列。 +- 控制分页深度：大 offset 的分页用 **覆盖索引 + id 范围** 或 **search_after**。 +- 大事务拆小事务，避免长时间锁表。 +- 批量写入用 `batch insert`，减少单条 SQL。 -#### MySQL Query Optimizer +**4. 参数调优** -1. MySQL 中有专门负责优化 SELECT 语句的优化器模块，主要功能：通过计算分析系统中收集到的统计信息，为客户端请求的 Query 提供他认为最优的执行计划（他认为最优的数据检索方式，但不见得是 DBA 认为是最优的，这部分最耗费时间） +- **连接池**：合理配置 `max_connections`，避免过大导致上下文切换。 +- **缓冲池**：InnoDB 的 `innodb_buffer_pool_size` 一般配置为物理内存的 60%~70%。 +- **日志参数**：调优 `innodb_log_file_size`，提升写性能。 +- **慢查询日志**：开启 `slow_query_log`，对慢 SQL 定位和优化。 -2. 当客户端向 MySQL 请求一条 Query，命令解析器模块完成请求分类，区别出是 SELECT 并转发给 MySQL Query Optimize r时，MySQL Query Optimizer 首先会对整条 Query 进行优化，处理掉一些常量表达式的预算，直接换算成常量值。并对 Query 中的查询条件进行简化和转换，如去掉一些无用或显而易见的条件、结构调整等。然后分析 Query 中的 Hint 信息（如果有），看显示 Hint 信息是否可以完全确定该 Query 的执行计划。如果没有 Hint 或 Hint 信息还不足以完全确定执行计划，则会读取所涉及对象的统计信息，根据 Query 进行写相应的计算分析，然后再得出最后的执行计划。 +**5. 硬件与运维** -#### MySQL常见瓶颈 +- SSD 替代机械硬盘，IOPS 提升明显。 +- 提升内存，增大缓存命中率。 +- 分区表、冷热数据分离。 +- 监控（Prometheus + Grafana），实时发现慢 SQL。 -- CPU：CPU在饱和的时候一般发生在数据装入内存或从磁盘上读取数据时候 -- IO：磁盘I/O瓶颈发生在装入数据远大于内存容量的时候 -- 服务器硬件的性能瓶颈：top，free，iostat 和 vmstat来查看系统的性能状态 +### 🎯 MySQL 8.0 升级点有哪些 -#### 性能下降SQL慢 执行时间长 等待时间长 原因分析 +MySQL 8.0 相比 5.7 有很多重要升级，最典型的有： -- 查询语句写的烂 -- 索引失效（单值、复合） -- 关联查询太多join（设计缺陷或不得已的需求） -- 服务器调优及各个参数设置（缓冲、线程数等） +1. **数据字典统一化**：以前的元数据分散在 `.frm`、`.par` 文件里，现在集中在系统表里，更一致也更安全。 +2. 8.0 支持 **函数索引**（如 `INDEX idx_name (UPPER(name))`），5.7 中对索引列使用函数（如 `UPPER(name)`）会导致索引失效，8.0 可直接命中函数索引，查询效率提升显著。 +3. **默认字符集改为 utf8mb4**，彻底支持 Emoji 等四字节字符。 +4. **事务性 DDL**：大多数 DDL（比如 `ALTER TABLE`）可以回滚，不像以前执行失败会半途残留。 +5. **窗口函数和公用表表达式（CTE）**，SQL 能力更强。 + - 新增 `ROW_NUMBER()`、`RANK()`、`LEAD()`、`LAG()` 等函数，支持复杂排名和分组统计。 + - 以前需要子查询 + 变量才能实现，现在 SQL 更简洁。 +6. **更强的 JSON 支持**，比如 JSON 表函数、索引优化。 +7. **性能优化**：更高效的并行查询、持久化执行计划、历史表统计信息。 +8. **安全性增强**：默认使用 `caching_sha2_password` 认证插件，加密更强。 +| 维度 | MySQL 5.7 现状 | MySQL 8.0 优势 | 适用场景收益 | +| -------- | ------------------------------- | ------------------------------------- | ------------------------------ | +| 性能 | 大事务、大表关联性能瓶颈 | 事务 / 索引 / 查询优化，性能提升 30%+ | 电商订单、金融交易等高并发场景 | +| 安全性 | 默认不加密，权限管理繁琐 | TLS 1.2 + 角色权限，防破解 / 篡改 | 金融、医疗等敏感数据场景 | +| 主从同步 | 延迟高，切换复杂 | 增强半同步 + GTID 自动定位 | 高可用架构（如 MGR、主从切换） | +| 功能扩展 | JSON / 时间精度有限，无函数索引 | JSON 表函数 + 微秒级时间 + 函数索引 | 混合数据存储、高频交易场景 | +| 可运维性 | 部分参数需重启，监控粒度粗 | 在线参数修改 + 线程级监控 | 大规模集群运维，减少停机时间 | +------ -#### MySQL常见性能分析手段 -在优化MySQL时，通常需要对数据库进行分析，常见的分析手段有**慢查询日志**，**EXPLAIN 分析查询**，**profiling分析**以及**show命令查询系统状态及系统变量**，通过定位分析性能的瓶颈，才能更好的优化数据库系统的性能。 -##### 性能瓶颈定位 +## 八、分库分表与集群 🚀 -我们可以通过 show 命令查看 MySQL 状态及变量，找到系统的瓶颈： +### 🎯 MySQL分区？ -```mysql -Mysql> show status ——显示状态信息（扩展show status like ‘XXX’） +一般情况下我们创建的表对应一组存储文件 -Mysql> show variables ——显示系统变量（扩展show variables like ‘XXX’） +1. **未分区的表文件结构** + - **MyISAM引擎**： `.frm`（表结构） + `.MYD`（数据文件） + `.MYI`（索引文件） *例如：`user.frm`、`user.MYD`、`user.MYI`* + - **InnoDB引擎**： `.frm`（表结构） + `.ibd`（数据+索引文件） *例如：`user.frm`、`user.ibd`* -Mysql> show innodb status ——显示InnoDB存储引擎的状态 +当数据量较大时（一般千万条记录级别以上），MySQL的性能就会开始下降，这时我们就需要将数据分散到多组存储文件，保证其单个文件的执行效率 -Mysql> show processlist ——查看当前SQL执行，包括执行状态、是否锁表等 +2. **分区后的表文件结构** 每个分区对应独立的物理文件，文件命名规则为： `表名#分区名.ibd` *例如：`user#p0.ibd`、`user#p1.ibd`* -Shell> mysqladmin variables -u username -p password——显示系统变量 +MySQL分区是一种数据库优化技术，通过将表的数据划分为更小、更易管理的部分，来提高查询性能和管理效率。下面是关于MySQL分区的一些关键点，适合在面试中讨论： -Shell> mysqladmin extended-status -u username -p password——显示状态信息 -``` +**能干嘛** +- 逻辑数据分割 +- 提高单一的写和读应用速度 +- 提高分区范围读查询的速度 +- 分割数据能够有多个不同的物理文件路径 +- 高效的保存历史数据 +**分区类型及操作** -##### Explain(执行计划) +| 类型 | 说明 | 适用场景 | 性能风险点 | 示例场景 | +| --------- | ------------------------------------------------------------ | ------------------------ | -------------------------------- | ---------------------- | +| **RANGE** | 基于属于一个给定连续区间的列值，把多行分配给分区 | 时间序列、连续数值 | 数据倾斜导致热点分区（如最新月） | 订单表按创建年份分区 | +| **LIST** | 按列表划分，类似于RANGE分区，但使用的是明确的值列表 | 离散枚举值（地区、状态） | 分区键值变更需重构分区 | 用户表按国家代码分区 | +| **HASH** | 按哈希算法划分，将数据根据某个列的哈希值均匀分布到不同的分区中 | 均匀分布请求压力 | 扩容需重新计算哈希分布 | 评论表按用户ID哈希分区 | +| **KEY** | 类似于HASH分区，但使用MySQL内部的哈希函数 | 非整型字段的均匀分布 | 依赖MySQL内置哈希算法 | 日志表按UUID前缀分区 | -是什么：使用 **Explain** 关键字可以模拟优化器执行SQL查询语句，从而知道 MySQL 是如何处理你的 SQL 语句的。分析你的查询语句或是表结构的性能瓶颈 +**看上去分区表很帅气，为什么大部分互联网还是更多的选择自己分库分表来水平扩展咧？** -能干吗： -- 表的读取顺序 -- 数据读取操作的操作类型 -- 哪些索引可以使用 -- 哪些索引被实际使用 -- 表之间的引用 -- 每张表有多少行被优化器查询 +- 分区表，分区键设计不太灵活，如果不走分区键，很容易出现全表锁 +- 一旦数据并发量上来，如果在分区表实施关联，就是一个灾难 +- 自己分库分表，自己掌控业务场景与访问模式，可控。分区表，研发写了一个sql，都不确定mysql是怎么玩的，不太可控 -怎么玩： -- Explain + SQL语句 -- 执行计划包含的信息（如果有分区表的话还会有**partitions**） -![expalin](https://tva1.sinaimg.cn/large/007S8ZIlly1gf2hsjk9zcj30kq01adfn.jpg) +### 🎯 如何确定分库还是分表？ -各字段解释 +> 针对“如何确定分库还是分表？”的问题，你要结合具体的场景。 -- **id**（select 查询的序列号，包含一组数字，表示查询中执行select子句或操作表的顺序） +**何时分表** - - id相同，执行顺序从上往下 - - id全不同，如果是子查询，id的序号会递增，id值越大优先级越高，越先被执行 - - id部分相同，执行顺序是先按照数字大的先执行，然后数字相同的按照从上往下的顺序执行 +当数据量过大造成事务执行缓慢时，就要考虑分表，因为减少每次查询数据总量是解决数据查询缓慢的主要原因。你可能会问：“查询可以通过主从分离或缓存来解决，为什么还要分表？”但这里的查询是指事务中的查询和更新操作。 -- **select_type**（查询类型，用于区别普通查询、联合查询、子查询等复杂查询） +**何时分库** - - **SIMPLE** ：简单的select查询，查询中不包含子查询或UNION - - **PRIMARY**：查询中若包含任何复杂的子部分，最外层查询被标记为PRIMARY - - **SUBQUERY**：在select或where列表中包含了子查询 - - **DERIVED**：在from列表中包含的子查询被标记为DERIVED，MySQL会递归执行这些子查询，把结果放在临时表里 - - **UNION**：若第二个select出现在UNION之后，则被标记为UNION，若UNION包含在from子句的子查询中，外层select将被标记为DERIVED - - **UNION RESULT**：从UNION表获取结果的select +为了应对高并发，一个数据库实例撑不住，即单库的性能无法满足高并发的要求，就把并发请求分散到多个实例中去 -- **table**（显示这一行的数据是关于哪张表的） +总的来说，分库分表使用的场景不一样： -- **type**（显示查询使用了那种类型，从最好到最差依次排列 **system > const > eq_ref > ref > fulltext > ref_or_null > index_merge > unique_subquery > index_subquery > range > index > ALL** ） +- **分表是因为数据量比较大，导致事务执行缓慢；** - - system：表只有一行记录（等于系统表），是 const 类型的特例，平时不会出现 - - const：表示通过索引一次就找到了，const 用于比较 primary key 或 unique 索引，因为只要匹配一行数据，所以很快，如将主键置于 where 列表中，mysql 就能将该查询转换为一个常量 - - eq_ref：唯一性索引扫描，对于每个索引键，表中只有一条记录与之匹配，常见于主键或唯一索引扫描 - - ref：非唯一性索引扫描，范围匹配某个单独值得所有行。本质上也是一种索引访问，他返回所有匹配某个单独值的行，然而，它可能也会找到多个符合条件的行，多以他应该属于查找和扫描的混合体 - - range：只检索给定范围的行，使用一个索引来选择行。key列显示使用了哪个索引，一般就是在你的where语句中出现了between、<、>、in等的查询，这种范围扫描索引比全表扫描要好，因为它只需开始于索引的某一点，而结束于另一点，不用扫描全部索引 - - index：Full Index Scan，index于ALL区别为index类型只遍历索引树。通常比ALL快，因为索引文件通常比数据文件小。（**也就是说虽然all和index都是读全表，但index是从索引中读取的，而all是从硬盘中读的**） - - ALL：Full Table Scan，将遍历全表找到匹配的行 +- **分库是因为单库的性能无法满足要求。** - tip: 一般来说，得保证查询至少达到range级别，最好到达ref -- **possible_keys**（显示可能应用在这张表中的索引，一个或多个，查询涉及到的字段若存在索引，则该索引将被列出，但不一定被查询实际使用） -- **key** +### 🎯 MySQL分库? - - 实际使用的索引，如果为NULL，则没有使用索引 +**为什么要分库?** - - **查询中若使用了覆盖索引，则该索引和查询的 select 字段重叠，仅出现在key列表中** +数据库集群环境后都是多台 slave，基本满足了读取操作; 但是写入或者说大数据、频繁的写入操作对 master 性能影响就比较大，这个时候，单库并不能解决大规模并发写入的问题，所以就会考虑分库。 -![explain-key](https://tva1.sinaimg.cn/large/007S8ZIlly1gf2hsty7iaj30nt0373yb.jpg) +**分库是什么？** -- **key_len** +一个库里表太多了，导致了海量数据，系统性能下降，把原本存储于一个库的表拆分存储到多个库上， 通常是将表按照功能模块、关系密切程度划分出来，部署到不同库上。 - - 表示索引中使用的字节数，可通过该列计算查询中使用的索引的长度。在不损失精确性的情况下，长度越短越好 - - key_len显示的值为索引字段的最大可能长度，并非实际使用长度，即key_len是根据表定义计算而得，不是通过表内检索出的 +**分库的优点：** -- **ref** （显示索引的哪一列被使用了，如果可能的话，是一个常数。哪些列或常量被用于查找索引列上的值） +- 减少增量数据写入时的锁对查询的影响 -- **rows** （根据表统计信息及索引选用情况，大致估算找到所需的记录所需要读取的行数） +- 由于单表数量下降，常见的查询操作由于减少了需要扫描的记录，使得单表单次查询所需的检索行数变少，减少了磁盘IO，时延变短 -- **Extra**（包含不适合在其他列中显示但十分重要的额外信息） +但是它无法解决单表数据量太大的问题 - 1. using filesort: 说明mysql会对数据使用一个外部的索引排序，不是按照表内的索引顺序进行读取。mysql中无法利用索引完成的排序操作称为“文件排序”。常见于order by和group by语句中 - 2. Using temporary：使用了临时表保存中间结果，mysql在对查询结果排序时使用临时表。常见于排序order by和分组查询group by。 - 3. using index：表示相应的select操作中使用了覆盖索引，避免访问了表的数据行，效率不错，如果同时出现using where，表明索引被用来执行索引键值的查找；否则索引被用来读取数据而非执行查找操作 +### 🎯 MySQL分表? - 4. using where：使用了where过滤 +分表有两种分割方式，一种垂直拆分，另一种水平拆分。 - 5. using join buffer：使用了连接缓存 +- **垂直拆分** - 6. impossible where：where子句的值总是false，不能用来获取任何元祖 + 垂直分表，通常是按照业务功能的使用频次，把主要的、热门的字段放在一起做为主要表。然后把不常用的，按照各自的业务属性进行聚集，拆分到不同的次要表中；主要表和次要表的关系一般都是一对一的。 - 7. select tables optimized away：在没有group by子句的情况下，基于索引优化操作或对于MyISAM存储引擎优化COUNT(*)操作，不必等到执行阶段再进行计算，查询执行计划生成的阶段即完成优化 +- **水平拆分(数据分片)** - 8. distinct：优化distinct操作，在找到第一匹配的元祖后即停止找同样值的动作 + 单表的容量不超过500W，否则建议水平拆分。是把一个表复制成同样表结构的不同表，然后把数据按照一定的规则划分，分别存储到这些表中，从而保证单表的容量不会太大，提升性能；当然这些结构一样的表，可以放在一个或多个数据库中。 - + 水平分割的几种方法： -**case**: + - 使用MD5哈希，做法是对UID进行md5加密，然后取前几位（我们这里取前两位），然后就可以将不同的UID哈希到不同的用户表（user_xx）中了。 + - 还可根据时间放入不同的表，比如：article_201601，article_201602。 + - 按热度拆分，高点击率的词条生成各自的一张表，低热度的词条都放在一张大表里，待低热度的词条达到一定的贴数后，再把低热度的表单独拆分成一张表。 + - 根据ID的值放入对应的表，第一个表user_0000，第二个100万的用户数据放在第二 个表user_0001中，随用户增加，直接添加用户表就行了。 -![explain-demo](https://tva1.sinaimg.cn/large/007S8ZIlly1gf2hszmc0lj30lc05w75c.jpg) -1. 第一行（执行顺序4）：id列为1，表示是union里的第一个select，select_type列的primary表示该查询为外层查询，table列被标记为，表示查询结果来自一个衍生表，其中derived3中3代表该查询衍生自第三个select查询，即id为3的select。【select d1.name......】 -2. 第二行（执行顺序2）：id为3，是整个查询中第三个select的一部分。因查询包含在from中，所以为derived。【select id,name from t1 where other_column=''】 -3. 第三行（执行顺序3）：select列表中的子查询select_type为subquery，为整个查询中的第二个select。【select id from t3】 -4. 第四行（执行顺序1）：select_type为union，说明第四个select是union里的第二个select，最先执行【select name,id from t2】 -5. 第五行（执行顺序5）：代表从union的临时表中读取行的阶段，table列的表示用第一个和第四个select的结果进行union操作。【两个结果union操作】 +### 🎯 做过分库分表么，为什么要分库分表，会有什么问题，多少数据适合分库分表，跨库，聚合操作怎么做？ +在电商和金融类项目中，我负责过订单、交易记录等核心表的分库分表设计，主要基于 Sharding-JDBC 实现。分库分表本质是 “突破单库单表的性能和容量瓶颈”，但也会引入新的复杂度，具体可以从 **“为什么做→怎么做→问题与解决”** 展开说明： +**一、为什么要分库分表？（核心痛点）** -##### 慢查询日志 +单库单表在数据量增长到一定规模后，会遇到 **性能瓶颈** 和 **容量瓶颈**，具体表现为： -MySQL 的慢查询日志是 MySQL 提供的一种日志记录，它用来记录在 MySQL 中响应时间超过阈值的语句，具体指运行时间超过 `long_query_time` 值的 SQL，则会被记录到慢查询日志中。 +1. 查询性能急剧下降：单表数据量超过 1000 万行后，即使加了索引，SQL 执行也会变慢（B + 树索引层级增加，磁盘 IO 次数增多）。比如电商订单表，3 年数据可能达 5000 万行，查询 “近 3 个月订单” 需要扫描大量数据，响应时间从 100ms 飙升到 1s+。 +2. 写入性能受限：单库的并发写入能力有限（MySQL 单库 TPS 一般在 1-2 万），若秒杀场景下每秒产生 5 万订单，单库会出现 “锁等待”“连接耗尽”，导致写入超时。 +3. 运维风险高：单表数据量过大，备份 / 恢复时间极长（比如 100GB 的表，备份需几小时），一旦数据库故障，恢复周期长，影响业务可用性。 -- `long_query_time` 的默认值为10，意思是运行10秒以上的语句 -- 默认情况下，MySQL数据库没有开启慢查询日志，需要手动设置参数开启 +分库分表通过 “将大表拆成小表、大库拆成小库”，把压力分散到多个数据库节点，解决上述问题。 -**查看开启状态** +**二、多少数据适合分库分表？（无绝对标准，看业务）** -```mysql -SHOW VARIABLES LIKE '%slow_query_log%' -``` +没有固定阈值，核心看 **“当前数据量是否影响业务性能”**，结合数据库类型和硬件配置，行业有通用参考： -**开启慢查询日志** +- **MySQL**：单表数据量建议控制在 **500 万 - 1000 万行**，单库表数量控制在 200-300 张以内（超过后，元数据管理、锁竞争会变慢）。 +- **业务驱动优先**：即使数据量没到阈值，但未来 6-12 个月会快速增长（如预期从 300 万涨到 1500 万），建议提前分库分表（避免后期数据迁移的复杂度）。 +- **反例**：若表是 “配置表”“字典表”，数据量长期稳定在 10 万以内，无需分库分表（过度设计反而增加复杂度）。 -- 临时配置： +**三、分库分表会有什么问题？（核心挑战）** -```mysql -mysql> set global slow_query_log='ON'; -mysql> set global slow_query_log_file='/var/lib/mysql/hostname-slow.log'; -mysql> set global long_query_time=2; -``` +分库分表打破了 “单库单表的完整性”，会引入新的技术难题： -​ 也可set文件位置，系统会默认给一个缺省文件host_name-slow.log +1. 跨库跨表查询 / 聚合难：比如 “查询用户近 1 年的所有订单 + 关联商品信息”，若订单表按用户 ID 分库，商品表按商品 ID 分库，跨库关联查询无法直接用 SQL 实现，需业务层处理。 +2. 分布式事务一致性：比如 “创建订单” 需要同时写订单表（分库 A）和库存表（分库 B），若其中一个库写入失败，如何保证 “要么全成功，要么全失败”？单库事务无法覆盖。 +3. 全局 ID 生成难：单库可用自增 ID，但分库分表后，多个表不能用自增（会重复），需生成全局唯一的 ID（如雪花算法、UUID）。 +4. 数据迁移与扩容复杂：后期需要增加分库分表节点（如从 8 分表扩到 16 分表），需迁移历史数据，且迁移过程中要保证业务不中断。 +5. 运维成本高：多个数据库节点需要监控、备份、故障转移，运维复杂度比单库高一个量级。 -​ 使用set操作开启慢查询日志只对当前数据库生效，如果MySQL重启则会失效。 +**四、跨库 / 聚合操作怎么做？（解决方案）** -- 永久配置 +针对分库分表后的跨库、聚合问题，行业有成熟的解决思路，核心是 “业务层适配 + 工具支持”： - 修改配置文件my.cnf或my.ini，在[mysqld]一行下面加入两个配置参数 +**1. 跨库查询 / 关联：优先 “避免跨库”，其次 “业务层拆分”** -```mysql -[mysqld] -slow_query_log = ON -slow_query_log_file = /var/lib/mysql/hostname-slow.log -long_query_time = 3 -``` +- **设计阶段规避**：尽量让关联表按同一字段分片（如订单表和订单明细表都按 “订单 ID” 分库，用户表和用户地址表都按 “用户 ID” 分库），确保关联查询在同一库内。 -注：log-slow-queries 参数为慢查询日志存放的位置，一般这个目录要有 MySQL 的运行帐号的可写权限，一般都将这个目录设置为 MySQL 的数据存放目录；long_query_time=2 中的 2 表示查询超过两秒才记录；在my.cnf或者 my.ini 中添加 log-queries-not-using-indexes 参数，表示记录下没有使用索引的查询。 +- 业务层拆分查询：若必须跨库，分两步处理： -可以用 `select sleep(4)` 验证是否成功开启。 + 例：查询 “用户 ID=123 的订单及关联商品”（订单表按用户 ID 分库，商品表按商品 ID 分库）： -在生产环境中，如果手工分析日志，查找、分析SQL，还是比较费劲的，所以MySQL提供了日志分析工具**mysqldumpslow**。 + ① 先查订单表（找到用户 123 的所有订单，获取商品 ID 列表）； -通过 mysqldumpslow --help 查看操作帮助信息 + ② 再查商品表（根据商品 ID 列表，批量获取商品信息）； -- 得到返回记录集最多的10个SQL + ③ 业务层将订单和商品数据拼接，返回结果。 - `mysqldumpslow -s r -t 10 /var/lib/mysql/hostname-slow.log` +- **工具支持**：用 Sharding-JDBC 等中间件的 “跨库关联” 能力（本质是中间件帮你做了拆分查询 + 结果合并），但性能比单库关联差，需谨慎使用。 -- 得到访问次数最多的10个SQL +**2. 聚合操作（如 count、sum、group by）：分 “预计算” 和 “实时计算”** - `mysqldumpslow -s c -t 10 /var/lib/mysql/hostname-slow.log` +- 实时聚合（小数据量）：中间件（Sharding-JDBC、MyCat）会先在每个分表上执行聚合 SQL（如 `count(*)`），再将结果汇总（如 8 个分表各返回 1000，总 count=8000）。 -- 得到按照时间排序的前10条里面含有左连接的查询语句 + 适合数据量小的场景（如 “查询用户近 7 天的订单数”），若数据量大（如 “查询全量订单的 sum (金额)”），实时聚合会很慢。 - `mysqldumpslow -s t -t 10 -g "left join" /var/lib/mysql/hostname-slow.log` +- 预计算（大数据量）：用 “离线计算 + 实时同步” 的方式，提前计算聚合结果： -- 也可以和管道配合使用 + ① 用 Flink/Spark 等计算引擎，每天离线计算 “全量订单的 sum/avg/count”，结果存到 “聚合结果表”（单库）； - `mysqldumpslow -s r -t 10 /var/lib/mysql/hostname-slow.log | more` + ② 实时新增的数据，通过 Binlog 同步到计算引擎，更新聚合结果； -**也可使用 pt-query-digest 分析 RDS MySQL 慢查询日志** + ③ 业务查询时，直接查 “聚合结果表”，无需跨库计算（如报表统计、大屏展示常用此方案）。 +**3. 分布式事务：用 “柔性事务” 平衡一致性和性能** +- 非核心业务：用 “最终一致性” 方案（如可靠消息队列）： -##### Show Profile 分析查询 + 例：订单创建→扣减库存： -通过慢日志查询可以知道哪些 SQL 语句执行效率低下，通过 explain 我们可以得知 SQL 语句的具体执行情况，索引使用等，还可以结合`Show Profile`命令查看执行状态。 + ① 订单表写入成功后，发送 “扣库存” 消息到 MQ； -- Show Profile 是 MySQL 提供可以用来分析当前会话中语句执行的资源消耗情况。可以用于SQL的调优的测量 + ② 库存服务消费消息，扣减库存； -- 默认情况下，参数处于关闭状态，并保存最近15次的运行结果 + ③ 若库存扣减失败，MQ 重试，直到成功（需保证库存扣减接口幂等）。 -- 分析步骤 +- 核心业务：用 “TCC 事务” 或 “Seata 等中间件”： - 1. 是否支持，看看当前的mysql版本是否支持 + TCC 分三步：Try（预留资源，如冻结库存）→ Confirm（确认执行，如扣减冻结的库存）→ Cancel（回滚，如释放冻结的库存），通过业务层代码保证跨库一致性； - ```mysql - mysql>Show variables like 'profiling'; --默认是关闭，使用前需要开启 - ``` + Seata 等中间件封装了 TCC、SAGA 等模式，降低开发成本（如电商订单支付场景常用 Seata）。 - 2. 开启功能，默认是关闭，使用前需要开启 +**五、总结** - ```mysql - mysql>set profiling=1; - ``` +分库分表是 “业务发展到一定阶段的必然选择”，但不是银弹 ——**能不分就不分，要分就早分**。设计时需优先规避跨库、分布式事务等复杂问题，选择成熟的中间件（如 Sharding-JDBC）降低开发和运维成本，同时做好数据扩容、监控的预案。我们项目中，订单表按 “用户 ID 哈希” 分 8 库 16 表，核心解决了 “千万级订单的查询和写入性能问题”，跨库操作通过 “业务层拆分 + 预计算” 处理，整体性能和稳定性满足了业务需求。 - 3. 运行SQL - 4. 查看结果 - ```mysql - mysql> show profiles; - +----------+------------+---------------------------------+ - | Query_ID | Duration | Query | - +----------+------------+---------------------------------+ - | 1 | 0.00385450 | show variables like "profiling" | - | 2 | 0.00170050 | show variables like "profiling" | - | 3 | 0.00038025 | select * from t_base_user | - +----------+------------+---------------------------------+ - ``` - ``` - - 5. 诊断SQL，show profile cpu,block io for query id(上一步前面的问题SQL数字号码) +### 🎯 分布式ID生成方案？ + +> 分库分表之后，id主键如何处理？ +> +> 推荐：https://zhuanlan.zhihu.com/p/107939861 + +- UUID：`UUID`的生成简单到只有一行代码，输出结果 `c2b8c2b9e46c47e3b30dca3b0d447718`，但UUID却并不适用于实际的业务需求。像用作订单号`UUID`这样的字符串没有丝毫的意义，看不出和订单相关的有用信息；而对于数据库来说用作业务`主键ID`，它不仅是太长还是字符串，而且不是自增的，存储性能差查询也很耗时，所以不推荐用作`分布式ID`。 + + > UUID 最大的缺陷是它产生的 ID 不是递增的。一般来说，我们倾向于在数据库中使用自增主键，因为这样可以迫使数据库的树朝着一个方向增长，而不会造成中间叶节点分裂，这样插入性能最好。而整体上 UUID 生成的 ID 可以看作是随机，那么就会导致数据往页中间插入，引起更加频繁地页分裂，在糟糕的情况下，这种分裂可能引起连锁反应，整棵树的树形结构都会受到影响。所以我们普遍倾向于采用递增的主键。 + +- 数据库自增ID：需要一个单独的MySQL实例用来生成ID（DB单点存在宕机风险，无法扛住高并发场景） + +- 数据库多主模式 + +- 号段模式 + +- Redis：利用`redis`的 `incr`命令实现ID的原子性自增。 + +- 雪花算法（SnowFlake）：`Snowflake`生成的是Long类型的ID，一个Long类型占8个字节，每个字节占8比特，也就是说一个Long类型占64个比特 + + Snowflake ID 组成结构：`正数位`（占1比特）+ `时间戳`（占41比特）+ `机器ID`（占5比特）+ `数据中心`（占5比特）+ `自增值`（占12比特），总共64比特组成的一个Long类型。 + + - **缺点**：时钟回拨需特殊处理 + - 改进： + - 百度UidGenerator：自定义时间位、引入RingBuffer + - 美团Leaf：混用号段模式应对时钟问题 + +- 滴滴出品（TinyID）：基于ZooKeeper的号段服务化 + +- 百度 （Uidgenerator） + +- 美团（Leaf）：号段模式 + 双Buffer预加载 + + + +### 🎯 分库分表容量是怎么确定的？ + +**容量确定的原则** + +1. **单表数据量控制：** + - **建议单表数据量控制在**500万到2000万 条记录以内。 + + > **单表数据量超过阈值** + > + > - 行数阈值：单表行数超过500万（一般建议）或2000万（B+树存储结构限制，3层树高对应约2000万行数据，查询效率最优）。 + > - **表容量阈值**：单表存储超过2GB，尤其是包含大字段（如BLOB、TEXT）时，读写性能显著下降 + + - **原因**：单表数据量过大，可能导致查询和索引效率下降，备份和恢复时间也会变长。 + +2. **单库数据量控制：** + - **建议单库的数据量不超过**100GB。 + - **原因**：单库数据量过大，可能导致磁盘I/O成为瓶颈，影响数据库性能。 + +3. **分片数量规划：** + - **预留空间**：根据未来的数据增长预期，预留足够的分片数量，避免频繁扩容。 + - **均衡分布**：确保数据在各个分片之间均匀分布，避免出现热点分片。 + + + +### 🎯 id哈希映射分库的话会产生什么问题？如何解决？ + +使用 **ID 哈希映射**进行分库时，会将数据分散到不同的数据库实例或节点上，基于用户 ID 或其他字段的哈希值来决定数据的存储位置。这种方法看似能解决大规模数据的存储和查询性能问题，但在实际操作中会带来一些挑战和问题。 + +1. **数据倾斜 (Data Skew)** + + 问题**： 哈希映射的目的是均匀分散数据到多个数据库中，但如果哈希函数或分库策略不合理，可能导致某些数据库或分片存储的数据量远大于其他数据库，产生**数据倾斜**，也就是数据分布不均匀。这样，某些数据库会成为性能瓶颈。** + + 解决方案： + + - **改进哈希函数**：选择更加均匀的哈希算法，避免出现某些特定范围的值过于集中。 + + - **范围分片**：有时可以结合哈希分片与范围分片（例如，按地域、注册时间等划分）来确保更均匀的数据分布。 + + - **分片重新分配**：在检测到数据倾斜后，可以定期或动态地调整数据的分布，通过数据迁移来平衡负载。 + +2. **跨库查询复杂度增加 (Cross-shard Querying)** + + 问题： 使用哈希分库后，查询可能会涉及多个数据库，尤其是当查询需要合并不同分片的数据时。传统的 JOIN 或聚合查询跨越多个分库时会变得非常复杂和低效，尤其是如果分库策略没有设计好，查询性能会显著下降。 + + 解决方案： + + - **避免跨库 JOIN**：尽量避免需要跨多个分片或数据库的复杂 JOIN 操作。通过数据 denormalization 或者将数据聚合到单个数据库中来减少跨库查询的复杂度。 + + - **分库查询优化**：当跨库查询不可避免时，可以使用 **分布式查询引擎** 或中间件（例如 **Hadoop**, **Presto**, **Apache Drill**）来优化跨库查询。 + + - **聚合和计算预处理**：如果查询频繁，考虑将结果预计算并存储（例如，通过缓存或者周期性计算）。减少实时计算的需求，提升查询性能。 + +3. **数据迁移与扩展困难 (Data Migration and Scaling)** + + 问题： 当数据量增加时，可能需要重新划分分片或迁移数据。在哈希映射的分库策略下，若重新划分分库或添加新的分片，现有的数据必须重新哈希并迁移到新的数据库实例，这个过程会非常复杂，且可能需要停机或者长时间的迁移操作。 + + 解决方案： + + - **分片重新平衡**：设计支持**动态扩展**和分片重新平衡的机制。使用可以实时调整的 **虚拟节点** 或 **哈希槽** 来减少数据迁移的复杂性。 + + - **使用分布式数据库系统**：一些分布式数据库系统（如 **Cassandra**, **CockroachDB**）提供自动扩展和数据迁移的能力，可以在不中断服务的情况下平衡分片和迁移数据。 + + - **预留扩展性**：在设计分库方案时，可以预留扩展的空间，并考虑将来可能需要添加更多分库或分片的情况。 + +4. **跨库事务管理 (Distributed Transactions)** + + 问题： 哈希分库导致数据分布在多个数据库中，而在一些操作中可能需要跨多个数据库的事务操作。例如，用户在多个数据库中有数据需要更新或修改时，如何保证事务的一致性（即 ACID 特性）就变得非常复杂。 + + 解决方案： + + - **使用 Saga 模式**：Saga 是一种长事务模式，将一个大的分布式事务分解成多个小的子事务，并在子事务失败时通过补偿操作进行回滚。适用于大多数分布式事务场景，尤其是对于微服务架构中的分布式数据更新。 + + - **采用最终一致性**：在一些业务场景中，避免使用强一致性，而是采用**最终一致性**来允许系统在短时间内不一致，但最终会恢复一致性。可以使用消息队列或事件驱动的方式来保证数据的最终一致性。 + + - **分布式事务管理器**：使用分布式事务管理器（如 **Atomikos** 或 **Narayana**）来处理跨多个数据库的事务。 + +5. **查找和聚合性能差 (Lookup and Aggregation Performance)** + + 问题： 哈希映射分库后，对于某些查询（如获取某个用户的所有数据或跨多个分片进行聚合查询），如果不采取合适的优化策略，查找和聚合性能会大大下降。 + + **解决方案**： + + - **数据冗余**：可以使用 **数据冗余** 或 **复制** 来减少跨库查询。例如，某些字段的冗余存储可以提高查询效率，避免每次都跨多个分片查询。 + + - **聚合操作分片**：对于聚合类操作，采用 **分布式计算框架**（如 **Apache Spark**、**Flink**）来进行分片内聚合，然后再合并结果。 + +6. **数据一致性和延迟问题 (Consistency and Latency)** + + 问题： 在分库哈希策略中，可能会因为多个数据库或分片的网络延迟而引入一定的延迟问题，尤其是在高并发环境下，多个数据库的访问可能会导致较高的延迟，影响用户体验。 + + 解决方案： + + - **本地缓存与副本**：通过使用本地缓存（如 **Redis**）来减少对远程数据库的访问，提升响应速度。同时，可以在多个分片之间保持副本，提高数据访问速度。 + + - **数据同步机制**：采用实时或批量数据同步机制，将热点数据或常用数据同步到访问频繁的分片。 + + + +### 集群 + +> 配主从，正经公司的话，也不会让 Javaer 去搞的，但还是要知道 + +### 🎯 说下 MySQL 主从？ + +1. **什么是 MySQL 主从复制？** + - 主从复制用于建立一个或多个与主库相同的数据库，称为从库，实现读写分离，提高并发处理能力。 +2. **主从复制的作用是什么？** + - 主从复制用于数据热备份、架构扩展、提高并发处理能力，实现读写分离。 +3. **MySQL 主从复制是如何实现的？** + - 主从复制通过 I/O 线程和 SQL 线程实现，I/O 线程负责从主库请求 binlog，SQL 线程负责将 binlog 应用到从库。 +4. **主从复制可能会遇到哪些问题？** + - 可能遇到的问题包括复制延迟、数据不一致、锁表导致的阻塞、宕机导致的数据丢失。 +5. **如何解决主从复制的问题？** + - 可以通过半同步复制策略减少数据丢失风险，采用并行复制减少复制延迟。 +6. **什么是 MySQL 集群？** + - MySQL 集群是一组 MySQL 服务器的集合，它们协同工作以提供高可用性、负载均衡和读写分离。 +7. **MySQL 集群有哪些类型？** + - 包括主从集群、互主集群、Galera 集群等，每种集群有其特定的应用场景和特点。 +8. **如何搭建 MySQL 集群？** + - 搭建 MySQL 集群通常涉及配置多个 MySQL 服务实例，实现主从复制，配置读写分离，以及设置故障转移机制。 +9. **什么是 GTID 同步集群？** + - GTID（全局事务 ID）同步集群是一种基于全局唯一 ID 标识事务的复制方式，引入于 MySQL 5.6 版本，用于确保事务在复制集群中的一致性。 +10. **如何实现 MySQL 的读写分离？** + - 读写分离通常由业务层实现，可以通过智能路由、负载均衡器或中间件如 ShardingSphere、MyCat 来实现。 +11. **什么是半同步复制？** + - 半同步复制是一种提高数据安全性的机制，主库在事务提交后等待至少一个从库接收并写入 relay log 后才返回客户端响应。 +12. **MySQL 集群扩容和数据迁移怎么做？** + - 扩容集群可能需要数据迁移，可以通过 mysqldump 工具备份数据，然后在新节点上恢复数据，再配置数据同步。 +13. **如何解决 MySQL 主从复制的延迟问题？** + - 可以通过优化网络条件、升级硬件、使用并行复制等方法减少延迟。 +14. **MySQL 集群的高可用性是如何实现的？** + - 高可用性可以通过故障检测、自动故障转移、多副本等机制实现。 +15. **MySQL 集群中的分库分表是如何考虑的？** + - 分库分表需要考虑数据量、查询模式、业务逻辑等因素，以优化性能和扩展性 + + + +### 🎯 复制的基本原理？ + +主从复制用于建立一个或多个与主库相同的数据库，称为从库，实现读写分离，提高并发处理能力 + +- slave 会从 master 读取 binlog 来进行数据同步 + +- 三个步骤 + + **1. 主库写入 binlog（Binary Log）** - 6. 日常开发需要注意的结论 + - 主库上的所有增删改操作（DDL/DML），都会被记录到 **binlog** 中，形成一条条事件（event）。 + - binlog 是逻辑日志，记录了 SQL 或行级别的更改。 - - converting HEAP to MyISAM 查询结果太大，内存都不够用了往磁盘上搬了。 + **2. 从库读取 relay log（中继日志）** - - create tmp table 创建临时表，这个要注意 + - 从库会启动一个 **I/O 线程**，连接主库，请求新的 binlog。 + - 主库的 **binlog dump 线程** 会把 binlog 内容发送给从库。 + - 从库的 I/O 线程把接收到的内容写入 **relay log**（中继日志）。 - - Copying to tmp table on disk 把内存临时表复制到磁盘 + **3. 从库重放 relay log** - - locked - ``` + - 从库启动 **SQL 线程**，从 relay log 里取出日志，执行里面的 SQL/行事件，最终更新从库数据。 -> 查询中哪些情况不会使用索引？ -### 性能优化 +### 🎯 MySQL 一主多从？ -#### 索引优化 +一旦你提及“一主多从”，面试官很容易设陷阱问你：那大促流量大时，是不是只要多增加几台从库，就可以抗住大促的并发读请求了？ -1. 全值匹配我最爱 -2. 最佳左前缀法则，比如建立了一个联合索引(a,b,c)，那么其实我们可利用的索引就有(a), (a,b), (a,b,c) -3. 不在索引列上做任何操作（计算、函数、(自动or手动)类型转换），会导致索引失效而转向全表扫描 -4. 存储引擎不能使用索引中范围条件右边的列 -5. 尽量使用覆盖索引(只访问索引的查询(索引列和查询列一致))，减少select -7. is null ,is not null 也无法使用索引 -8. like "xxxx%" 是可以用到索引的，like "%xxxx" 则不行(like "%xxx%" 同理)。like以通配符开头('%abc...')索引失效会变成全表扫描的操作， -9. 字符串不加单引号索引失效 -10. 少用or，用它来连接时会索引失效 -10. <，<=，=，>，>=，BETWEEN，IN 可用到索引，<>，not in ，!= 则不行，会导致全表扫描 +当然不是。 +因为从库数量增加，从库连接上来的 I/O 线程也比较多，主库也要创建同样多的 log dump 线程来处理复制的请求，对主库资源消耗比较高，同时还受限于主库的网络带宽。所以在实际使用中，一个主库一般跟 2～3 个从库（1 套数据库，1 主 2 从 1 备主），这就是一主多从的 MySQL 集群结构。 +其实，你从 MySQL 主从复制过程也能发现，MySQL 默认是异步模式：MySQL 主库提交事务的线程并不会等待 binlog 同步到各从库，就返回客户端结果。这种模式一旦主库宕机，数据就会发生丢失。 -**一般性建议** +而这时，面试官一般会追问你“**MySQL 主从复制还有哪些模型？”**主要有三种。 -- 对于单键索引，尽量选择针对当前query过滤性更好的索引 +- 同步复制：事务线程要等待所有从库的复制成功响应。 -- 在选择组合索引的时候，当前Query中过滤性最好的字段在索引字段顺序中，位置越靠前越好。 +- 异步复制：事务线程完全不等待从库的复制成功响应。 -- 在选择组合索引的时候，尽量选择可以能够包含当前query中的where字句中更多字段的索引 +- 半同步复制：MySQL 5.7 版本之后增加的一种复制方式，介于两者之间，事务线程不用等待所有的从库复制成功响应，只要一部分复制成功响应回来就行，比如一主二从的集群，只要数据成功复制到任意一个从库上，主库的事务线程就可以返回给客户端。 -- 尽可能通过分析统计信息和调整query的写法来达到选择合适索引的目的 +这种半同步复制的方式，兼顾了异步复制和同步复制的优点，即使出现主库宕机，至少还有一个从库有最新的数据，不存在数据丢失的风险。 -- 少用Hint强制索引 - -#### 查询优化 +### 🎯 主从复制的延迟与一致性问题 -**永远小标驱动大表（小的数据集驱动大的数据集）** +> MySQL 主从复制是基于 binlog 的异步过程，主库写入 binlog，从库通过 relay log 重放。复制延迟的原因主要有网络问题、主库压力大、从库执行慢、单线程复制瓶颈、大事务等。 +> 复制延迟会导致读写分离场景下的数据不一致。 +> 优化方法有：使用 MySQL 5.7+ 的多线程复制，避免大事务，优化从库 SQL 和索引，提升硬件和网络。 +> 一致性上，可以通过 **强制读主库、半同步复制、组复制** 或 **Proxy 层路由策略** 来解决。 -```mysql -slect * from A where id in (select id from B)`等价于 -#等价于 -select id from B -select * from A where A.id=B.id -``` +**1. 主从复制原理（简要）** -当 B 表的数据集必须小于 A 表的数据集时，用 in 优于 exists +MySQL 主从复制通常基于 **binlog**： -```mysql -select * from A where exists (select 1 from B where B.id=A.id) -#等价于 -select * from A -select * from B where B.id = A.id` -``` +1. **主库（Master）**：把事务操作写入 **binlog**。 +2. **从库 I/O 线程**：从主库拉取 binlog，写入 **relay log**。 +3. **从库 SQL 线程**：读取 relay log，并在从库重放执行。 + +因此，从库的数据落后于主库，会存在一定延迟。 + +**2. 延迟的原因** + +1. **网络延迟**：主从之间的网络传输慢。 +2. **主库压力大**：binlog 写入速度快，从库拉取不过来。 +3. **从库 SQL 执行性能差**：从库执行 binlog SQL 比主库慢（比如没有合适的索引、单机性能差）。 +4. **单线程复制瓶颈（传统复制）**：MySQL 5.6 之前，SQL 线程是单线程，执行大事务时延迟严重。 +5. **大事务 / 批量更新**：一次性更新、删除上百万行，导致 binlog 很大，从库应用时间很长。 + +**3. 一致性问题** + +- **读写分离下的数据不一致**：如果应用在从库读数据，可能读到的是**旧数据**，因为主库刚写入，从库还没同步。 +- **事务一致性问题**：主库写入成功，但从库延迟，导致业务逻辑判断错误（比如库存、余额）。 + +**4. 解决思路** -当 A 表的数据集小于B表的数据集时，用 exists优于用 in +（1）减少延迟 -注意：A表与B表的ID字段应建立索引。 +- **多线程复制**：MySQL 5.7 开始支持 **并行复制**（基于库级并行、基于组提交并行），大幅降低延迟。 +- **优化 SQL** + - 避免大事务，改成小批量操作。 + - 保证主从索引一致，防止从库回放慢。 +- **硬件优化**：提升从库磁盘/CPU 性能，减少执行耗时。 +- **网络优化**：使用更低延迟的链路。 +（2）一致性保证 +- **强制读主库**：对于关键数据（如支付、库存），读写都走主库。 +- **semi-sync 半同步复制** + - 主库写事务时，至少要等一个从库确认收到 binlog 才返回客户端，保证数据至少写到一个从库。 + - 缺点：降低写性能。 +- **组复制 / MGR (MySQL Group Replication)** + - 多主复制，保证强一致性，但性能有损耗。 +- **Proxy 层策略** + - 在中间件（如 MyCat、ShardingSphere、ProxySQL）里，对延迟敏感的查询强制走主库，其它查询走从库。 +- **延迟监控 + 降级** + - 通过 `SHOW SLAVE STATUS` 监控 `Seconds_Behind_Master`，大于阈值时从库不提供读服务。 -**order by关键字优化** -- order by子句，尽量使用 Index 方式排序，避免使用 FileSort 方式排序 -- MySQL 支持两种方式的排序，FileSort 和 Index，Index效率高，它指 MySQL 扫描索引本身完成排序，FileSort 效率较低； -- ORDER BY 满足两种情况，会使用Index方式排序；①ORDER BY语句使用索引最左前列 ②使用where子句与ORDER BY子句条件列组合满足索引最左前列 +### 🎯 复制的最大问题 -- 尽可能在索引列上完成排序操作，遵照索引建的最佳最前缀 -- 如果不在索引列上，filesort 有两种算法，mysql就要启动双路排序和单路排序 - - 双路排序：MySQL 4.1之前是使用双路排序,字面意思就是两次扫描磁盘，最终得到数据 - - 单路排序：从磁盘读取查询需要的所有列，按照order by 列在 buffer对它们进行排序，然后扫描排序后的列表进行输出，效率高于双路排序 +- 延时 + + + +------ + + + +## 九、SQL实战编程 💻 -- 优化策略 +### 🎯 MySQL 设计需要注意什么? - - 增大sort_buffer_size参数的设置 - - 增大max_lencth_for_sort_data参数的设置 +在设计MySQL数据库时，有许多方面需要注意，以确保数据库的性能、可扩展性、安全性和可维护性 +- 表结构设计：**规范化**、合适的数据类型 +- 索引：覆盖索引 +- 查询优化：合理使用JOIN +- 分区与分库分表 -**GROUP BY关键字优化** +### 🎯 如何在不停机的情况下保证迁移数据的一致性？ -- group by实质是先排序后进行分组，遵照索引建的最佳左前缀 -- 当无法使用索引列，增大 `max_length_for_sort_data` 参数的设置，增大`sort_buffer_size`参数的设置 -- where高于having，能写在where限定的条件就不要去having限定了 +迁移数据时最核心的挑战： +1. **源库与目标库要保持一致**（全量 + 增量）。 +2. **迁移过程不中断业务**（不停机，业务可读写）。 +3. **最终一致性**（迁移完成后保证数据不丢、不重、不错）。 +**1. 常见方案步骤** -#### 数据类型优化 +步骤 1：全量迁移 -MySQL 支持的数据类型非常多，选择正确的数据类型对于获取高性能至关重要。不管存储哪种类型的数据，下面几个简单的原则都有助于做出更好的选择。 +- 使用工具（如 **mysqldump、mydumper、pt-archiver**，或大数据同步工具如 **Canal、Debezium、DataX、DTS**）将源库的历史数据 **批量导入**到目标库。 +- 这一步通常需要 **只读快照** 或 **事务隔离** 保证导出一致性。 -- 更小的通常更好：一般情况下，应该尽量使用可以正确存储数据的最小数据类型。 +步骤 2：增量同步 - 简单就好：简单的数据类型通常需要更少的CPU周期。例如，整数比字符操作代价更低，因为字符集和校对规则（排序规则）使字符比较比整型比较复杂。 +- 全量迁移的过程中，源库业务仍在写入，这部分数据必须捕捉并同步。 +- 常见做法是基于 **binlog**（MySQL）或者 **WAL**（Postgres），利用 **CDC（Change Data Capture）机制** 实时同步增量数据到目标库。 +- 代表工具： + - MySQL → **Canal、Debezium、Maxwell** + - Kafka + sink → 目标库 -- 尽量避免NULL：通常情况下最好指定列为NOT NULL +步骤 3：双写验证（可选） + +- 在迁移过程中，业务层可以同时写 **源库和目标库**，然后通过 **校验服务** 或 **对账机制**（hash 校验、抽样比对）确认一致性。 + +步骤 4：流量切换（灰度迁移） + +- 在验证数据无误后，将业务读写请求逐步切换到新库。 +- 可以采用 **读流量先切**，再切写流量，避免一次性切换导致大故障。 +- 这里推荐 **蓝绿发布** 或 **双活切换**。 + +步骤 5：确认一致性并下线旧库 + +- 切换后短时间保留双写或增量同步，待验证无误后下线旧库。 ------ +**2. 保证一致性的关键点** +- **全量 + 增量结合**：先拷贝静态数据，再捕获实时变化。 +- **校验机制**：对关键表做 **row count + checksum/hash 校验**，确保无丢失。 +- **幂等性**：迁移程序必须保证重复消费 binlog 时不会产生脏数据。 +- **最终一致性**：接受迁移过程中存在短暂延迟，但最终必须对齐。 +- **流量切换要平滑**：避免一次性大规模切换引发不可控风险。 -## 九、分区、分表、分库 +------ -### MySQL分区 +**3. 常见工具链** -一般情况下我们创建的表对应一组存储文件，使用`MyISAM`存储引擎时是一个`.MYI`和`.MYD`文件，使用`Innodb`存储引擎时是一个`.ibd`和`.frm`（表结构）文件。 +- **MySQL**：pt-online-schema-change、gh-ost（用于表结构变更不停机） +- **数据迁移/同步**：Canal、Debezium、Maxwell、DataX、DTS（阿里云） +- **消息队列中转**：Kafka（承接 binlog 流，再同步到目标库） -当数据量较大时（一般千万条记录级别以上），MySQL的性能就会开始下降，这时我们就需要将数据分散到多组存储文件，保证其单个文件的执行效率 -**能干嘛** -- 逻辑数据分割 -- 提高单一的写和读应用速度 -- 提高分区范围读查询的速度 -- 分割数据能够有多个不同的物理文件路径 -- 高效的保存历史数据 +### 🎯 说一说三个范式？ -**怎么玩** +数据库设计中的三个范式（3NF）是用于规范数据库的结构，以减少数据冗余和提高数据的一致性。 -首先查看当前数据库是否支持分区 +- 第一范式（1NF）：数据库表中的字段都是单一属性的，不可再分。这个单一属性由基本类型构成，包括整型、实数、字符型、逻辑型、日期型等。 +- 第二范式（2NF）：第二范式在第一范式的基础上进一步规范化，要求表中的每一列都与主键直接相关，而不是间接相关 +- 第三范式（3NF）：第三范式要求列之间没有传递依赖，即非主键列之间不能相互依赖。所谓传递函数依赖，指的是如果存在"A → B → C"的决定关系，则C传递函数依赖于A。因此，满足第三范式的数据库表应该不存在如下依赖关系： 关键字段 → 非关键字段 x → 非关键字段y -- MySQL5.6以及之前版本： - ```mysql - SHOW VARIABLES LIKE '%partition%'; - ``` -- MySQL5.6： +### 🎯 limit 100000 加载很慢的话，你是怎么解决的呢？ - ```mysql - show plugins; - ``` +在 mysql 中 limit 可以实现快速分页，但是如果数据到了几百万时我们的 limit 必须优化才能有效的合理的实现分页了，否则可能卡死你的服务器 -**分区类型及操作** +**当一个表数据有几百万的数据的时候成了问题！** -- **RANGE分区**：基于属于一个给定连续区间的列值，把多行分配给分区。mysql将会根据指定的拆分策略，,把数据放在不同的表文件上。相当于在文件上,被拆成了小块.但是,对外给客户的感觉还是一张表，透明的。 +日常分页SQL语句 - 按照 range 来分，就是每个库一段连续的数据，这个一般是按比如**时间范围**来的，比如交易表啊，销售表啊等，可以根据年月来存放数据。可能会产生热点问题，大量的流量都打在最新的数据上了。 +```mysql +select id,name,content from users order by id asc limit 100000,20 +``` - range 来分，好处在于说，扩容的时候很简单。 +扫描100020行 -- **LIST分区**：类似于按RANGE分区，每个分区必须明确定义。它们的主要区别在于，LIST分区中每个分区的定义和选择是基于某列的值从属于一个值列表集中的一个值，而RANGE分区是从属于一个连续区间值的集合。 +如果记录了上次的最大ID -- **HASH分区**：基于用户定义的表达式的返回值来进行选择的分区，该表达式使用将要插入到表中的这些行的列值进行计算。这个函数可以包含MySQL 中有效的、产生非负整数值的任何表达式。 +```mysql + select id,name,content from users where id>100073 order by id asc limit 20 +``` - hash 分发，好处在于说，可以平均分配每个库的数据量和请求压力；坏处在于说扩容起来比较麻烦，会有一个数据迁移的过程，之前的数据需要重新计算 hash 值重新分配到不同的库或表 +扫描 20 行。 -- **KEY分区**：类似于按HASH分区，区别在于KEY分区只支持计算一列或多列，且MySQL服务器提供其自身的哈希函数。必须有一列或多列包含整数值。 +总数据有500万左右，以下例子 -**看上去分区表很帅气，为什么大部分互联网还是更多的选择自己分库分表来水平扩展咧？** +```mysql +select * from wl_tagindex where byname='f' order by id limit 300000,10 +``` -- 分区表，分区键设计不太灵活，如果不走分区键，很容易出现全表锁 -- 一旦数据并发量上来，如果在分区表实施关联，就是一个灾难 -- 自己分库分表，自己掌控业务场景与访问模式，可控。分区表，研发写了一个sql，都不确定mysql是怎么玩的，不太可控 +执行时间是 3.21s +优化后： +```mysql +select * from ( + select id from wl_tagindex +where byname='f' order by id limit 300000,10 +) a +left join wl_tagindex b on a.id=b.id +``` -> 随着业务的发展，业务越来越复杂，应用的模块越来越多，总的数据量很大，高并发读写操作均超过单个数据库服务器的处理能力怎么办？ + 执行时间为 0.11s 速度明显提升 -这个时候就出现了**数据分片**，数据分片指按照某个维度将存放在单一数据库中的数据分散地存放至多个数据库或表中。数据分片的有效手段就是对关系型数据库进行分库和分表。 +- 原查询需要扫描并丢弃前 300,000 行数据。优化后的子查询只选择 id 列，大大减少了需要处理的数据量 -区别于分区的是，分区一般都是放在单机里的，用的比较多的是时间范围分区，方便归档。只不过分库分表需要代码实现，分区则是mysql内部实现。分库分表和分区并不冲突，可以结合使用。 + 这里需要说明的是 我这里用到的字段是 byname ,id 需要把这两个字段做复合索引，否则的话效果提升不明显 ->说说分库与分表的设计 +### 🎯 在高并发情况下，如何做到安全的修改同一行数据？ -### MySQL分表 +**1、使用悲观锁** -分表有两种分割方式，一种垂直拆分，另一种水平拆分。 +悲观锁本质是当前只有一个线程执行操作，排斥外部请求的修改。遇到加锁的状态，就必须等待。结束了唤醒其他线程进行处理。虽然此方案的确解决了数据安全的问题，但是，我们的场景是“高并发”。也就是说，会很多这样的修改请求，每个请求都需要等待“锁”，某些线程可能永远都没有机会抢到这个“锁”，这种请求就会死在那里。同时，这种请求会很多，瞬间增大系统的平均响应时间，结果是可用连接数被耗尽，系统陷入异常。 -- **垂直拆分** +**2、FIFO（First Input First Output，先进先出）缓存队列思路** - 垂直分表，通常是按照业务功能的使用频次，把主要的、热门的字段放在一起做为主要表。然后把不常用的，按照各自的业务属性进行聚集，拆分到不同的次要表中；主要表和次要表的关系一般都是一对一的。 +直接将请求放入队列中，就不会导致某些请求永远获取不到锁。看到这里，是不是有点强行将多线程变成单线程的感觉哈。 -- **水平拆分(数据分片)** +![](https://img-blog.csdnimg.cn/20190508231654761.) - 单表的容量不超过500W，否则建议水平拆分。是把一个表复制成同样表结构的不同表，然后把数据按照一定的规则划分，分别存储到这些表中，从而保证单表的容量不会太大，提升性能；当然这些结构一样的表，可以放在一个或多个数据库中。 +然后，我们现在解决了锁的问题，全部请求采用“先进先出”的队列方式来处理。那么新的问题来了，高并发的场景下，因为请求很多，很可能一瞬间将队列内存“撑爆”，然后系统又陷入到了异常状态。或者设计一个极大的内存队列，也是一种方案，但是，系统处理完一个队列内请求的速度根本无法和疯狂涌入队列中的数目相比。也就是说，队列内的请求会越积累越多，最终Web系统平均响应时间还是会大幅下降，系统还是陷入异常。 - 水平分割的几种方法： +**3、使用乐观锁** - - 使用MD5哈希，做法是对UID进行md5加密，然后取前几位（我们这里取前两位），然后就可以将不同的UID哈希到不同的用户表（user_xx）中了。 - - 还可根据时间放入不同的表，比如：article_201601，article_201602。 - - 按热度拆分，高点击率的词条生成各自的一张表，低热度的词条都放在一张大表里，待低热度的词条达到一定的贴数后，再把低热度的表单独拆分成一张表。 - - 根据ID的值放入对应的表，第一个表user_0000，第二个100万的用户数据放在第二 个表user_0001中，随用户增加，直接添加用户表就行了。 +这个时候，我们就可以讨论一下“乐观锁”的思路了。乐观锁，是相对于“悲观锁”采用更为宽松的加锁机制，大都是采用带版本号（Version）更新。实现就是，这个数据所有请求都有资格去修改，但会获得一个该数据的版本号，只有版本号符合的才能更新成功，其他的返回抢购失败。这样的话，我们就不需要考虑队列的问题，不过，它会增大CPU的计算开销。但是，综合来说，这是一个比较好的解决方案。 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1geuibkd9mjj31ns0u0aj1.jpg) +### 🎯 表中有大字段 **X**(例如:**text** 类型)，且字段 **X** 不会经常更新，以读为 为主，将该字段拆成子表好处是什么? -### MySQL分库 +如果字段里面有大字段(text,blob)类型的，而且这些字段的访问并不多，这 时候放在一起就变成缺点了。 MYSQL 数据库的记录存储是按行存储的，数据 块大小又是固定的(16K)，每条记录越小，相同的块存储的记录就越多。此 时应该把大字段拆走，这样应付大部分小字段的查询时，就能提高效率。当需 要查询大字段时，此时的关联查询是不可避免的，但也是值得的。拆分开后， 对字段的 UPDAE 就要 UPDATE 多个表了 -> 为什么要分库? -数据库集群环境后都是多台 slave，基本满足了读取操作; 但是写入或者说大数据、频繁的写入操作对master性能影响就比较大，这个时候，单库并不能解决大规模并发写入的问题，所以就会考虑分库。 -> 分库是什么？ +### 🎯 MySQL 数据达到多少会产生瓶颈？ -一个库里表太多了，导致了海量数据，系统性能下降，把原本存储于一个库的表拆分存储到多个库上， 通常是将表按照功能模块、关系密切程度划分出来，部署到不同库上。 +MySQL在处理大型数据集时，性能瓶颈的出现并非仅取决于数据量的大小，还与硬件配置、表结构设计、索引情况、查询复杂度和并发访问量等多种因素密切相关。因此，很难给出一个精确的数据量阈值来确定何时会出现瓶颈。 -优点： +**一般情况下，以下情况可能会导致MySQL产生性能瓶颈：** -- 减少增量数据写入时的锁对查询的影响 +1. **单表数据量过大：** + - **数据量级别**：当单表记录数达到**数百万到数千万**时，查询性能可能会明显下降。 + - **影响因素**：如果缺乏合理的索引和优化，查询速度会受到显著影响。 +2. **索引设计不合理：** + - **缺少必要索引**：没有为常用查询添加索引，导致全表扫描。 + - **过多索引**：索引过多会增加写入和更新的开销。 + - **索引碎片**：频繁的插入和删除操作会导致索引碎片化。 +3. **硬件资源限制：** + - **内存不足**：无法将常用数据缓存到内存中，导致频繁的磁盘I/O。 + - **磁盘性能**：传统HDD的读写速度较慢，可能成为瓶颈。 + - **CPU性能**：复杂查询和高并发需要更高的CPU处理能力。 +4. **高并发访问：** + - **连接数过多**：大量的并发连接会消耗系统资源，导致性能下降。 + - **锁竞争**：高并发写操作会导致锁竞争，影响事务的执行效率。 +5. **查询复杂度高：** + - **复杂的JOIN操作**：多表关联查询会增加数据库的计算负担。 + - **未优化的SQL语句**：如使用`SELECT *`或缺少条件过滤。 +6. **配置参数不当：** + - **默认配置不适合大数据量**：需要根据业务场景调整MySQL的配置参数，如`innodb_buffer_pool_size`。 + - **连接池设置不合理**：可能导致资源浪费或不足。 +7. **事务和锁机制的影响：** + - **长事务**：长时间占用锁资源，阻塞其他事务。 + - **死锁问题**：不合理的事务管理可能导致死锁。 -- 由于单表数量下降，常见的查询操作由于减少了需要扫描的记录，使得单表单次查询所需的检索行数变少，减少了磁盘IO，时延变短 -但是它无法解决单表数据量太大的问题 +### 🎯 SQL 注入？ +SQL注入是一种常见且危险的安全漏洞，但有几种有效的方法可以防止它。以下是解决SQL注入问题的主要方法： -**分库分表后的难题** +1. 使用参数化查询（预处理语句）： -分布式事务的问题，数据的完整性和一致性问题。 + 这是防止SQL注入最有效和推荐的方法。参数化查询将SQL语句和数据分开处理，从而防止恶意输入被解释为SQL命令。 -数据操作维度问题：用户、交易、订单各个不同的维度，用户查询维度、产品数据分析维度的不同对比分析角度。 跨库联合查询的问题，可能需要两次查询 跨节点的count、order by、group by以及聚合函数问题，可能需要分别在各个节点上得到结果后在应用程序端进行合并 额外的数据管理负担，如：访问数据表的导航定位 额外的数据运算压力，如：需要在多个节点执行，然后再合并计算程序编码开发难度提升，没有太好的框架解决，更多依赖业务看如何分，如何合，是个难题。 + **不安全的 SQL 查询：** + ```sql + String query = "SELECT * FROM users WHERE username = '" + username + "' AND password = '" + password + "'"; + ``` + 攻击者可以通过输入 `" OR "1" = "1` 这样类似的内容绕过验证，导致 SQL 注入。 -> 配主从，正经公司的话，也不会让 Javaer 去搞的，但还是要知道 + **使用预编译语句的安全查询：** -## 十、主从复制 + ```java + String query = "SELECT * FROM users WHERE username = ? AND password = ?"; + PreparedStatement stmt = connection.prepareStatement(query); + stmt.setString(1, username); + stmt.setString(2, password); + ResultSet rs = stmt.executeQuery(); + ``` -### 复制的基本原理 +2. 使用ORM（对象关系映射）：ORM工具通常会自动使用参数化查询，提供额外的安全层。 +3. 输入验证和清洗： + - 使用白名单策略（允许合法字符），拒绝非预期的输入 + - 限制输入的长度，避免输入过多字符 + - 对输入的数据类型进行严格验证，如预期是数字类型的字段必须验证输入为数字 + - 对特殊字符进行转义或过滤（如 `'`, `"`, `;` 等） + +4. **最小权限原则**：确保应用程序连接数据库的账号仅具备完成任务所需的最小权限，避免攻击者一旦突破防线就能全面操控数据库。 +5. 当然，还有些 统一编码、定期安全审计、保持软件更新 等措施 + + + +## 十、 手撕 SQL 🖊️ + +### 🎯 十岁为一组，统计每个年龄段的用户数量 + +要以每 10 岁为一组统计用户数量，可以使用 `FLOOR` 函数（向下取整）对用户的年龄进行分组，然后使用 `GROUP BY` 和 `COUNT(*)` 来统计每个年龄段的用户数量 + +```sql +SELECT + CASE + WHEN age BETWEEN 0 AND 9 THEN '0-9' + WHEN age BETWEEN 10 AND 19 THEN '10-19' + WHEN age BETWEEN 20 AND 29 THEN '20-29' + WHEN age BETWEEN 30 AND 39 THEN '30-39' + WHEN age BETWEEN 40 AND 49 THEN '40-49' + WHEN age BETWEEN 50 AND 59 THEN '50-59' + WHEN age BETWEEN 60 AND 69 THEN '60-69' + WHEN age BETWEEN 70 AND 79 THEN '70-79' + WHEN age BETWEEN 80 AND 89 THEN '80-89' + WHEN age >= 90 THEN '90+' + ELSE 'Unknown' + END AS age_group, + COUNT(*) AS user_count +FROM users +GROUP BY age_group +ORDER BY age_group; + +--------- +SELECT + CONCAT(FLOOR(age / 10) * 10, '-', FLOOR(age / 10) * 10 + 9) AS age_range, + COUNT(*) AS user_count +FROM users +GROUP BY FLOOR(age / 10) +ORDER BY FLOOR(age / 10); +``` -- slave 会从 master 读取 binlog 来进行数据同步 -- 三个步骤 - 1. master将改变记录到二进制日志（binary log）。这些记录过程叫做二进制日志事件，binary log events； - 2. salve 将 master 的 binary log events 拷贝到它的中继日志（relay log）; - 3. slave 重做中继日志中的事件，将改变应用到自己的数据库中。MySQL 复制是异步且是串行化的。 +### 🎯 给学生表、课程成绩表，求不存在01课程但存在02课程的学生的成绩 - ![img](http://img.wandouip.com/crawler/article/201942/94aec4abf353527cbbe2bef5a484471d) +这种方法比较多，我用最简单的， 使用 `LEFT JOIN` 和 `IS NULL` -### 复制的基本原则 +```sql +SELECT + cg.student_id, + s.name, + cg.course_id, + cg.grade +FROM + course_grades cg + JOIN students s ON cg.student_id = s.student_id + LEFT JOIN course_grades cg2 ON cg.student_id = cg2.student_id AND cg2.course_id = '01' +WHERE + cg.course_id = '02' + AND cg2.student_id IS NULL; +``` -- 每个 slave只有一个 master -- 每个 salve只能有一个唯一的服务器 ID -- 每个master可以有多个salve +还可以使用 `NOT IN` 子查询 -### 复制的最大问题 -- 延时 ------- +### 🎯 查询第二大的数值 +1. 使用 ORDER BY 和 LIMIT + ```sql + SELECT DISTINCT column_name FROM table_name + ORDER BY column_name DESC + LIMIT 1 OFFSET 1 + ``` -## 十一、其他问题 + 这个查询先按降序排列,然后跳过第一个结果(OFFSET 1),取下一个结果(LIMIT 1) -### 说一说三个范式 +2. 使用子查询 -- 第一范式（1NF）：数据库表中的字段都是单一属性的，不可再分。这个单一属性由基本类型构成，包括整型、实数、字符型、逻辑型、日期型等。 -- 第二范式（2NF）：数据库表中不存在非关键字段对任一候选关键字段的部分函数依赖（部分函数依赖指的是存在组合关键字中的某些字段决定非关键字段的情况），也即所有非关键字段都完全依赖于任意一组候选关键字。 -- 第三范式（3NF）：在第二范式的基础上，数据表中如果不存在非关键字段对任一候选关键字段的传递函数依赖则符合第三范式。所谓传递函数依赖，指的是如 果存在"A → B → C"的决定关系，则C传递函数依赖于A。因此，满足第三范式的数据库表应该不存在如下依赖关系： 关键字段 → 非关键字段 x → 非关键字段y + ```sql + SELECT MAX(column_name) + FROM table_name + WHERE column_name < (SELECT MAX(column_name) FROM table_name) + ``` + +3. 使用 `DENSE_RANK()` 窗口函数【MySQL 8.0+ 支持】 + + ```sql + SELECT column_name + FROM ( + SELECT column_name, DENSE_RANK() OVER (ORDER BY column_name DESC) AS rank + FROM table_name + ) AS ranked + WHERE rank = 2; + ``` + + + +### 🎯 经典排名问题 + +排名问题是SQL面试的经典题型，考察对窗口函数和聚合查询的掌握： + +**💻 排名问题SQL实现**： + +```sql +-- ===== 基础排名问题 ===== + +-- 题目1：查询各科目成绩前3名的学生 +CREATE TABLE student_scores ( + student_id INT, + student_name VARCHAR(50), + subject VARCHAR(50), + score INT, + exam_date DATE +); + +-- 插入测试数据 +INSERT INTO student_scores VALUES +(1, '张三', '数学', 95, '2024-01-15'), +(2, '李四', '数学', 88, '2024-01-15'), +(3, '王五', '数学', 92, '2024-01-15'), +(4, '赵六', '数学', 85, '2024-01-15'), +(5, '张三', '英语', 90, '2024-01-15'), +(6, '李四', '英语', 95, '2024-01-15'), +(7, '王五', '英语', 88, '2024-01-15'); + +-- 解法1：使用窗口函数 +WITH ranked_scores AS ( + SELECT + student_id, + student_name, + subject, + score, + ROW_NUMBER() OVER (PARTITION BY subject ORDER BY score DESC) as rn, + RANK() OVER (PARTITION BY subject ORDER BY score DESC) as rnk, + DENSE_RANK() OVER (PARTITION BY subject ORDER BY score DESC) as dense_rnk + FROM student_scores +) +SELECT + student_id, + student_name, + subject, + score, + rn as row_number_rank, + rnk as rank_with_gaps, + dense_rnk as dense_rank +FROM ranked_scores +WHERE rn <= 3 +ORDER BY subject, score DESC; + +-- 题目2：查询每个部门薪资前20%的员工 +WITH salary_percentiles AS ( + SELECT + employee_id, + employee_name, + department, + salary, + PERCENT_RANK() OVER (PARTITION BY department ORDER BY salary DESC) as percentile_rank + FROM employees +) +SELECT + employee_id, + employee_name, + department, + salary, + ROUND(percentile_rank * 100, 2) as top_percentage +FROM salary_percentiles +WHERE percentile_rank <= 0.2 -- 前20% +ORDER BY department, salary DESC; +``` + +### 🎯 连续问题求解 + +连续问题是考察逻辑思维的重要题型： + +**💻 连续问题SQL实现**： + +```sql +-- ===== 连续登录问题 ===== + +-- 题目：找出连续登录3天以上的用户 +CREATE TABLE user_login_log ( + user_id INT, + login_date DATE, + login_time TIMESTAMP +); + +-- 解法：使用ROW_NUMBER()找连续区间 +WITH date_with_row AS ( + SELECT DISTINCT + user_id, + login_date, + ROW_NUMBER() OVER (PARTITION BY user_id ORDER BY login_date) as rn + FROM user_login_log +), +consecutive_groups AS ( + SELECT + user_id, + login_date, + DATE_SUB(login_date, INTERVAL rn DAY) as group_date + FROM date_with_row +), +consecutive_counts AS ( + SELECT + user_id, + group_date, + COUNT(*) as consecutive_days, + MIN(login_date) as start_date, + MAX(login_date) as end_date + FROM consecutive_groups + GROUP BY user_id, group_date +) +SELECT + user_id, + consecutive_days, + start_date, + end_date +FROM consecutive_counts +WHERE consecutive_days >= 3 +ORDER BY user_id, start_date; + +-- 题目：股票连续上涨问题 +WITH price_changes AS ( + SELECT + stock_code, + trade_date, + closing_price, + LAG(closing_price) OVER (PARTITION BY stock_code ORDER BY trade_date) as prev_price, + CASE + WHEN closing_price > LAG(closing_price) OVER (PARTITION BY stock_code ORDER BY trade_date) + THEN 1 ELSE 0 + END as is_up + FROM stock_prices +), +consecutive_ups AS ( + SELECT + stock_code, + COUNT(*) as up_days, + MIN(trade_date) as start_date, + MAX(trade_date) as end_date + FROM ( + SELECT + stock_code, + trade_date, + SUM(CASE WHEN is_up = 0 THEN 1 ELSE 0 END) + OVER (PARTITION BY stock_code ORDER BY trade_date) as group_id + FROM price_changes + WHERE is_up = 1 + ) grouped + GROUP BY stock_code, group_id +) +SELECT * FROM consecutive_ups +WHERE up_days >= 3 +ORDER BY stock_code, start_date; +``` +### 🎯 复杂业务场景 + +实际业务中的复杂查询场景： + +**💻 业务场景SQL实现**： + +```sql +-- ===== 电商用户行为分析 ===== + +-- 题目：找出"流失预警"用户（最后购买>30天，但历史活跃） +WITH user_purchase_stats AS ( + SELECT + user_id, + COUNT(*) as total_orders, + SUM(order_amount) as total_spent, + AVG(order_amount) as avg_order_value, + MAX(order_date) as last_purchase_date, + DATEDIFF(NOW(), MAX(order_date)) as days_since_last_purchase + FROM orders + WHERE order_status = 'completed' + GROUP BY user_id +), +user_segments AS ( + SELECT + *, + CASE + WHEN total_orders >= 10 AND avg_order_value >= 200 THEN 'VIP客户' + WHEN total_orders >= 5 AND total_spent >= 500 THEN '优质客户' + WHEN total_orders >= 2 THEN '普通客户' + ELSE '新客户' + END as customer_segment, + CASE + WHEN days_since_last_purchase > 60 THEN '高风险流失' + WHEN days_since_last_purchase > 30 AND total_orders >= 5 THEN '流失预警' + WHEN days_since_last_purchase <= 7 THEN '活跃用户' + ELSE '一般用户' + END as churn_risk_level + FROM user_purchase_stats +) +SELECT + customer_segment, + churn_risk_level, + COUNT(*) as user_count, + ROUND(AVG(total_orders), 2) as avg_orders, + ROUND(AVG(total_spent), 2) as avg_total_spent +FROM user_segments +GROUP BY customer_segment, churn_risk_level +ORDER BY + FIELD(customer_segment, 'VIP客户', '优质客户', '普通客户', '新客户'), + FIELD(churn_risk_level, '高风险流失', '流失预警', '一般用户', '活跃用户'); + +-- ===== 树形递归查询 ===== + +-- 题目：查询员工的所有下属 +WITH RECURSIVE subordinates AS ( + -- 锚点：直接下属 + SELECT + emp_id, + emp_name, + manager_id, + 1 as level + FROM employees + WHERE manager_id = 1001 -- 查询员工1001的下属 + + UNION ALL + + -- 递归：下属的下属 + SELECT + e.emp_id, + e.emp_name, + e.manager_id, + s.level + 1 + FROM employees e + INNER JOIN subordinates s ON e.manager_id = s.emp_id + WHERE s.level < 5 -- 限制层级 +) +SELECT + emp_id, + emp_name, + level, + CONCAT(REPEAT(' ', level-1), emp_name) as hierarchy_display +FROM subordinates +ORDER BY level, emp_id; +``` + +--- + +## + +## 十一、运维与监控 🔧 +**核心理念**：数据库运维与监控是保障系统稳定运行的关键，及时发现和解决问题能够避免业务中断和数据丢失。 -### 百万级别或以上的数据如何删除 +### 🎯 MySQL备份和恢复策略？ -关于索引：由于索引需要额外的维护成本，因为索引文件是单独存在的文件,所以当我们对数据的增加,修改,删除,都会产生额外的对索引文件的操作,这些操作需要消耗额外的IO,会降低增/改/删的执行效率。所以，在我们删除数据库百万级别数据的时候，查询MySQL官方手册得知删除数据的速度和创建的索引数量是成正比的。 +MySQL备份恢复是数据安全的最后一道防线，需要制定完善的备份策略： -1. 所以我们想要删除百万数据的时候可以先删除索引（此时大概耗时三分多钟） -2. 然后删除其中无用数据（此过程需要不到两分钟） -3. 删除完成后重新创建索引(此时数据较少了)创建索引也非常快，约十分钟左右。 -4. 与之前的直接删除绝对是要快速很多，更别说万一删除中断,一切删除会回滚。那更是坑了。 +**💻 备份恢复实战**： +```sql +-- ===== 逻辑备份（mysqldump）===== +/* +# 完整数据库备份 +mysqldump -u root -p --single-transaction --routines --triggers \ + --all-databases > full_backup_$(date +%Y%m%d_%H%M%S).sql +# 单个数据库备份 +mysqldump -u root -p --single-transaction --routines --triggers \ + ecommerce > ecommerce_backup_$(date +%Y%m%d_%H%M%S).sql +# 只备份结构不备份数据 +mysqldump -u root -p --no-data ecommerce > ecommerce_schema.sql +# 只备份数据不备份结构 +mysqldump -u root -p --no-create-info ecommerce > ecommerce_data.sql + +# 备份特定表 +mysqldump -u root -p ecommerce users orders > tables_backup.sql + +# 压缩备份 +mysqldump -u root -p --single-transaction ecommerce | gzip > backup.sql.gz +*/ + +-- ===== 物理备份（MySQL Enterprise Backup / Percona XtraBackup）===== + +/* +# 使用Percona XtraBackup进行热备份 +# 完整备份 +innobackupex --user=backup_user --password=backup_pass /backup/full/ + +# 增量备份（基于完整备份） +innobackupex --user=backup_user --password=backup_pass \ + --incremental /backup/inc1/ \ + --incremental-basedir=/backup/full/ + +# 准备备份（应用redo log） +innobackupex --apply-log /backup/full/ + +# 恢复数据 +innobackupex --copy-back /backup/full/ +*/ + +-- ===== 基于binlog的时点恢复 ===== + +-- 查看binlog位置信息 +SHOW MASTER STATUS; +SHOW BINARY LOGS; + +-- 查看特定时间点的binlog位置 +SHOW BINLOG EVENTS IN 'mysql-bin.000001' +WHERE Start_time >= '2024-01-01 10:00:00' +LIMIT 1; + +/* +# 时点恢复步骤 +# 1. 恢复最新全备份 +mysql -u root -p ecommerce < ecommerce_backup_20240101_020000.sql + +# 2. 应用binlog到指定时间点 +mysqlbinlog --start-datetime="2024-01-01 02:00:00" \ + --stop-datetime="2024-01-01 09:30:00" \ + mysql-bin.000001 mysql-bin.000002 | mysql -u root -p ecommerce + +# 3. 跳过错误事务，继续恢复 +mysqlbinlog --start-position=1000 --stop-position=2000 \ + mysql-bin.000001 | mysql -u root -p ecommerce +*/ + +-- ===== 自动化备份脚本示例 ===== + +/* +#!/bin/bash +# MySQL自动备份脚本 + +DATE=$(date +%Y%m%d_%H%M%S) +BACKUP_DIR="/backup/mysql" +LOG_FILE="$BACKUP_DIR/backup.log" +RETENTION_DAYS=7 + +# 创建备份目录 +mkdir -p $BACKUP_DIR + +# 执行备份 +echo "$(date): Starting backup..." >> $LOG_FILE +mysqldump -u backup_user -p$BACKUP_PASSWORD \ + --single-transaction --routines --triggers \ + --all-databases | gzip > $BACKUP_DIR/full_backup_$DATE.sql.gz + +if [ $? -eq 0 ]; then + echo "$(date): Backup completed successfully" >> $LOG_FILE + + # 清理过期备份 + find $BACKUP_DIR -name "full_backup_*.sql.gz" -mtime +$RETENTION_DAYS -delete + echo "$(date): Old backups cleaned" >> $LOG_FILE +else + echo "$(date): Backup failed!" >> $LOG_FILE + # 发送告警邮件 + echo "MySQL backup failed at $(date)" | mail -s "Backup Alert" admin@company.com +fi +*/ + +-- ===== 备份验证 ===== + +-- 验证备份完整性的存储过程 +DELIMITER // +CREATE PROCEDURE ValidateBackup(IN backup_file VARCHAR(255)) +BEGIN + DECLARE backup_size BIGINT; + DECLARE table_count INT; + DECLARE record_count BIGINT; + + -- 这里应该包含备份文件大小检查 + -- 恢复到临时数据库进行验证 + -- 检查表数量和记录数量 + + SELECT + COUNT(*) as tables, + SUM(TABLE_ROWS) as total_rows + INTO table_count, record_count + FROM information_schema.TABLES + WHERE TABLE_SCHEMA = 'backup_validation_db'; + + SELECT + backup_file, + table_count, + record_count, + CASE + WHEN table_count > 0 AND record_count > 0 THEN 'VALID' + ELSE 'INVALID' + END as validation_result; +END // +DELIMITER ; +``` + +### 🎯 MySQL监控指标和故障排查？ + +MySQL监控需要关注多个维度的指标，及时发现性能瓶颈和潜在问题： + +**💻 监控和故障排查**： + +```sql +-- ===== 关键性能指标监控 ===== + +-- 连接状态监控 +SELECT + VARIABLE_NAME, + VARIABLE_VALUE, + CASE VARIABLE_NAME + WHEN 'Max_used_connections' THEN + CONCAT(ROUND(VARIABLE_VALUE / @@max_connections * 100, 2), '%') + ELSE '' + END as usage_percentage +FROM performance_schema.global_status +WHERE VARIABLE_NAME IN ( + 'Connections', 'Max_used_connections', 'Threads_connected', 'Threads_running' +); + +-- 查询性能指标 +SELECT + VARIABLE_NAME, + VARIABLE_VALUE, + CASE VARIABLE_NAME + WHEN 'Questions' THEN CONCAT(ROUND(VARIABLE_VALUE / @@uptime, 2), ' QPS') + WHEN 'Slow_queries' THEN + CONCAT(ROUND(VARIABLE_VALUE / + (SELECT VARIABLE_VALUE FROM performance_schema.global_status WHERE VARIABLE_NAME = 'Questions') * 100, 4), '%') + ELSE '' + END as rate_info +FROM performance_schema.global_status +WHERE VARIABLE_NAME IN ( + 'Questions', 'Queries', 'Slow_queries', 'Select_scan', 'Sort_merge_passes' +); + +-- InnoDB缓冲池监控 +SELECT + 'Buffer Pool Hit Rate' as metric, + ROUND(( + 1 - (reads.VARIABLE_VALUE / requests.VARIABLE_VALUE) + ) * 100, 2) as hit_rate_percent, + CASE + WHEN (1 - (reads.VARIABLE_VALUE / requests.VARIABLE_VALUE)) * 100 >= 99 THEN '优秀' + WHEN (1 - (reads.VARIABLE_VALUE / requests.VARIABLE_VALUE)) * 100 >= 95 THEN '良好' + ELSE '需要优化' + END as status +FROM + (SELECT VARIABLE_VALUE FROM performance_schema.global_status WHERE VARIABLE_NAME = 'Innodb_buffer_pool_reads') reads, + (SELECT VARIABLE_VALUE FROM performance_schema.global_status WHERE VARIABLE_NAME = 'Innodb_buffer_pool_read_requests') requests; + +-- ===== 实时性能分析 ===== + +-- 当前运行的查询 +SELECT + ID, + USER, + HOST, + DB, + COMMAND, + TIME, + STATE, + LEFT(INFO, 100) as QUERY_SAMPLE, + CASE + WHEN TIME > 60 THEN '长时间运行' + WHEN TIME > 10 THEN '需要关注' + ELSE '正常' + END as status +FROM information_schema.PROCESSLIST +WHERE COMMAND != 'Sleep' +ORDER BY TIME DESC; + +-- 锁等待分析 +SELECT + r.trx_id AS waiting_trx, + r.trx_mysql_thread_id AS waiting_thread, + r.trx_query AS waiting_query, + b.trx_id AS blocking_trx, + b.trx_mysql_thread_id AS blocking_thread, + b.trx_query AS blocking_query, + TIMESTAMPDIFF(SECOND, r.trx_started, NOW()) as wait_time_seconds +FROM information_schema.INNODB_LOCK_WAITS w +JOIN information_schema.INNODB_TRX b ON b.trx_id = w.blocking_trx_id +JOIN information_schema.INNODB_TRX r ON r.trx_id = w.requesting_trx_id; + +-- ===== 故障排查查询 ===== + +-- 查找占用空间最大的表 +SELECT + TABLE_SCHEMA, + TABLE_NAME, + ROUND(((DATA_LENGTH + INDEX_LENGTH) / 1024 / 1024), 2) AS size_mb, + TABLE_ROWS, + ROUND((INDEX_LENGTH / 1024 / 1024), 2) AS index_size_mb, + ROUND((DATA_LENGTH / 1024 / 1024), 2) AS data_size_mb +FROM information_schema.TABLES +WHERE TABLE_SCHEMA NOT IN ('information_schema', 'mysql', 'performance_schema', 'sys') +ORDER BY (DATA_LENGTH + INDEX_LENGTH) DESC +LIMIT 20; + +-- 分析表碎片化情况 +SELECT + TABLE_SCHEMA, + TABLE_NAME, + TABLE_ROWS, + AVG_ROW_LENGTH, + ROUND((DATA_LENGTH / 1024 / 1024), 2) as data_mb, + ROUND((DATA_FREE / 1024 / 1024), 2) as free_mb, + ROUND((DATA_FREE / (DATA_LENGTH + DATA_FREE)) * 100, 2) as fragmentation_percent +FROM information_schema.TABLES +WHERE TABLE_SCHEMA NOT IN ('information_schema', 'mysql', 'performance_schema', 'sys') + AND DATA_FREE > 0 +ORDER BY fragmentation_percent DESC; +``` -## 参考与感谢： +--- -https://zhuanlan.zhihu.com/p/29150809 +## 🎯 面试重点总结 -https://juejin.im/post/5e3eb616f265da570d734dcb#heading-105 +### 高频考点速览 -https://blog.csdn.net/yin767833376/article/details/81511377 +- **🏗️ 基础与架构**：MySQL架构组件、DDL/DML/DCL区别、SQL语法和最佳实践 +- **🗄️ 存储引擎**：InnoDB vs MyISAM特性、内存结构、磁盘结构、缓冲池机制 +- **🔍 索引机制**：B+树原理、索引类型、联合索引最左前缀、覆盖索引、索引失效场景 +- **🔒 事务与锁**：ACID特性实现、隔离级别对比、锁机制、MVCC原理、死锁处理 +- **📊 查询优化**：JOIN类型优化、窗口函数、执行计划分析、数据类型选择 +- **📝 日志系统**：redo log、undo log、binlog机制、WAL原理、恢复策略 +- **⚡ 性能调优**：慢查询分析、参数调优、缓存策略、硬件优化 +- **🚀 分库分表**：主从复制、读写分离、分片策略、数据迁移、集群部署 +- **💻 SQL实战**：复杂查询编写、存储过程、函数、CTE递归查询 +- **🔧 运维监控**：备份恢复、监控指标、故障排查、容量规划、日常维护 +### 面试答题策略 +1. **基础概念题**：先说定义和原理，再举具体应用例子，最后分析优缺点和适用场景 +2. **性能优化题**：分析性能瓶颈，提出具体优化方案，说明效果评估方法 +3. **架构设计题**：从业务需求出发，考虑数据量和并发量，选择合适的架构方案 +4. **故障排查题**：描述排查思路和工具，定位根本原因，提供解决方案和预防措施 +### 核心设计原则 +1. **性能优先**：合理设计索引，优化查询语句，配置合适的参数，选择适合的存储引擎 +2. **数据安全**：事务保证一致性，完善的备份恢复策略，严格的权限控制机制 +3. **高可用设计**：消除单点故障，实现故障自动切换，多层次的监控告警体系 +4. **可扩展性**：分库分表应对数据增长，读写分离提升并发能力，集群化部署 +5. **运维规范**：标准化的部署流程，完善的监控体系，及时的故障响应机制 +--- +## 📚 扩展学习 +- **MySQL官方文档**：深入学习存储引擎原理、性能调优最佳实践 +- **经典书籍**：《高性能MySQL》、《MySQL技术内幕》、《数据库系统概念》 +- **开源工具**：Percona Toolkit、pt-query-digest、MySQL Workbench使用 +- **监控方案**：Prometheus+Grafana、Zabbix、云监控平台实践 +- **新版本特性**：MySQL 8.0新特性、JSON支持、窗口函数、CTE等 +**记住：数据库是应用系统的核心，扎实的MySQL基础和丰富的优化经验是后端工程师的核心竞争力！** 🚀 diff --git a/docs/interview/Netty-FAQ.md b/docs/interview/Netty-FAQ.md new file mode 100755 index 0000000000..e0f837ed82 --- /dev/null +++ b/docs/interview/Netty-FAQ.md @@ -0,0 +1,40 @@ +1. Netty 的高性能表现在哪些方面？对你平时的项目开发有何启发？ +2. Netty 中有哪些重要组件，它们之间有什么联系？ +3. Netty 的内存池、对象池是如何设计的？ +4. 针对 Netty 你有哪些印象比较深刻的系统调优案例？ + +![image ](https://learn.lianglianglee.com/%E4%B8%93%E6%A0%8F/Netty%20%E6%A0%B8%E5%BF%83%E5%8E%9F%E7%90%86%E5%89%96%E6%9E%90%E4%B8%8E%20RPC%20%E5%AE%9E%E8%B7%B5-%E5%AE%8C/assets/CgqCHl-NAQaABGcDAAZa0pmBs40719.png) + +**通过 Netty 的学习，还可以锻炼你的编程思维，对 Java 其他的知识体系起到融会贯通的作用** + + + + + +### Netty 整体结构 + +Netty 是一个设计非常用心的**网络基础组件**，Netty 官网给出了有关 Netty 的整体功能模块结构，却没有其他更多的解释。从图中，我们可以清晰地看出 Netty 结构一共分为三个模块： + +![Drawing 0.png](https://learn.lianglianglee.com/%e4%b8%93%e6%a0%8f/Netty%20%e6%a0%b8%e5%bf%83%e5%8e%9f%e7%90%86%e5%89%96%e6%9e%90%e4%b8%8e%20RPC%20%e5%ae%9e%e8%b7%b5-%e5%ae%8c/assets/CgqCHl-NO7eATPMMAAH8t8KvehQ985.png) + +#### 1. Core 核心层 + +Core 核心层是 Netty 最精华的内容，它提供了底层网络通信的通用抽象和实现，包括可扩展的事件模型、通用的通信 API、支持零拷贝的 ByteBuf 等。 + +#### 2. Protocol Support 协议支持层 + +协议支持层基本上覆盖了主流协议的编解码实现，如 HTTP、SSL、Protobuf、压缩、大文件传输、WebSocket、文本、二进制等主流协议，此外 Netty 还支持自定义应用层协议。Netty 丰富的协议支持降低了用户的开发成本，基于 Netty 我们可以快速开发 HTTP、WebSocket 等服务。 + +#### 3. Transport Service 传输服务层 + +传输服务层提供了网络传输能力的定义和实现方法。它支持 Socket、HTTP 隧道、虚拟机管道等传输方式。Netty 对 TCP、UDP 等数据传输做了抽象和封装，用户可以更聚焦在业务逻辑实现上，而不必关系底层数据传输的细节。 + +Netty 的模块设计具备较高的**通用性和可扩展性**，它不仅是一个优秀的网络框架，还可以作为网络编程的工具箱。Netty 的设计理念非常优雅，值得我们学习借鉴。 + + + +### Netty 逻辑架构 + +下图是 Netty 的逻辑处理架构。Netty 的逻辑处理架构为典型网络分层架构设计，共分为网络通信层、事件调度层、服务编排层，每一层各司其职。图中包含了 Netty 每一层所用到的核心组件。我将为你介绍 Netty 的每个逻辑分层中的各个核心组件以及组件之间是如何协调运作的。 + +![Drawing 1.png](https://learn.lianglianglee.com/%e4%b8%93%e6%a0%8f/Netty%20%e6%a0%b8%e5%bf%83%e5%8e%9f%e7%90%86%e5%89%96%e6%9e%90%e4%b8%8e%20RPC%20%e5%ae%9e%e8%b7%b5-%e5%ae%8c/assets/Ciqc1F-NO9KAUOtaAAE1S5uRlDE275.png) \ No newline at end of file diff --git a/docs/interview/Network-FAQ.md b/docs/interview/Network-FAQ.md index 870e72ae9a..174134bb66 100644 --- a/docs/interview/Network-FAQ.md +++ b/docs/interview/Network-FAQ.md @@ -1,817 +1,2388 @@ -> 你好，我是 π大新，目前在一家名字等于周角的公司就职，精通Java，熟悉计算机网络，， +--- +title: 网络编程面试题大全 +date: 2024-12-15 +tags: + - Network + - HTTP + - TCP + - UDP + - Interview +categories: Interview +--- -然后就~~~~ +![](https://img.starfish.ink/common/faq-banner.png) -> 在浏览器中输入一个 URL 至页面呈现，网络上都发生了什么事？ -> -> 能说说 ISO 七层模型和 TCP/IP 四层模型吗？ -> -> TCP/IP 与 HTTP 有什么关系吗？ -> -> TCP协议与UDP协议的区别？ -> -> 请详细介绍一下 TCP 的三次握手机制，为什么要三次握手？挥手却又是四次呢？ -> -> 详细讲一下TCP的滑动窗口？知道流量控制和拥塞控制吗？ -> -> 说一下对称加密与非对称加密？ -> -> 状态码 206 是什么意思？ -> -> 你们用的 https 是吧，https 工作原理是什么？ -> -> ...... +网络编程作为后端开发的**核心基础技能**，是Java后端面试的**必考重点**。从OSI七层模型到TCP/UDP协议，从HTTP协议到Socket编程，从网络IO模型到性能优化，每个知识点都可能成为面试的关键。本文档将**网络编程核心技术**整理成**系统化知识体系**，涵盖协议原理、网络编程、性能调优等关键领域，助你在面试中游刃有余！ -> 文章收录在 GitHub [JavaKeeper](https://github.com/Jstarfish/JavaKeeper) ，N线互联网开发必备技能兵器谱，有你想要的。 +网络编程面试，围绕着这么几个核心方向准备： -## 一、计算机网络 +- **网络基础**（OSI模型、TCP/IP协议栈、网络分层、数据封装） +- **传输层协议**（TCP可靠性保证、UDP特性、三次握手、四次挥手、拥塞控制） +- **应用层协议**（HTTP协议详解、HTTPS加密、HTTP/1.1 vs HTTP/2、WebSocket） +- **网络编程**（Socket编程、BIO/NIO/AIO、Netty框架、Reactor模式） +- **性能优化**（网络调优、连接池、缓存策略、负载均衡） +- **高级话题**（网络安全、CORS、CDN、DNS解析、分布式网络） -### 通信协议 +## 🗺️ 知识导航 -通信协议（communications protocol）是指双方实体完成通信或服务所必须遵循的规则和约定。通过通信信道和设备互连起来的多个不同地理位置的数据通信系统，要使其能协同工作实现信息交换和资源共享，它们之间必须具有共同的语言。交流什么、怎样交流及何时交流，都必须遵循某种互相都能接受的规则。这个规则就是通信协议。 +### 📊 按面试频率和重要性分类 +#### 🔥 高频必考（面试必备核心） +1. **传输层协议**：TCP/UDP原理、三次握手四次挥手、滑动窗口、拥塞控制 +2. **应用层协议**：HTTP/HTTPS详解、状态码、缓存机制、Keep-Alive +3. **网络编程实战**：Socket编程、BIO/NIO/AIO模型、Netty框架、Reactor模式 -### 网络模型 +#### 📈 中频重点（提升竞争力） -随着技术的发展，计算机的应用越来越广泛，计算机之间的通信开始了百花齐放的状态，每个具有独立计算服务体系的信息技术公司都会建立自己的计算机通信规则，而这种情况会导致异构计算机之间无法通信，极大的阻碍了网络通信的发展，至此为了解决这个问题，国际标准化组织（ISO）制定了OSI模型，该模型定义了不同计算机互联的标准，OSI模型把网络通信的工作分为7层，分别是**物理层、数据链路层、网络层、传输层、会话层、表示层和应用层**。 +4. **网络层协议**：IP协议、路由转发、ARP解析、ICMP应用 +5. **网络安全**：常见攻击防护、加密算法、DDoS防范 -这七层模型是设计层面的概念，每一层都有固定要完成的职责和功能，分层的好处在于清晰和功能独立性，但分层过多会使层次变的更加复杂，虽然不需要实现本层的功能，但是也需要构造本层的上下文，空耗系统资源，所以在落地实施网络通信模型的时候将这七层模型简化合并为四层模型分别是**应用层、传输层、网络层、网络接口层**（各层之间的模型、协议统称为：**TCP/IP协议簇**）。 +#### 📚 低频基础（知识体系完整性） -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gds66rxwnaj30ku0dr0tn.jpg) +6. **网络理论基础**：OSI七层模型、TCP/IP分层、协议设计原理 +7. **物理层基础**：信号传输、编码调制、传输介质 +8. **数据链路层**：帧结构、错误检测、MAC地址、以太网协议 +9. **性能优化与监控**：网络调优、故障排查、监控体系、性能分析 -从上图可以看到，TCP/IP模型合并了OSI模型的应用层、表示层和会话层，将OSI模型的数据链路层和物理层合并为网络访问层。 +--- -上图还列出了各层模型对应TCP/IP协议栈中的协议以及各层协议之间的关系。比如DNS协议是建立在TCP和UDP协议的基础上，FTP、HTTP、TELNET协议建立在TCP协议的基础上，NTP、TFTP、SNMP建立在UDP协议的基础上，而TCP、UDP协议又建立在IP协议的基础上，以此类推…. +## 一、传输层协议 -| OSI中的层 | 功能 | TCP/IP协议族 | -| :------------- | ------------------------------------------------------ | :----------------------------------------------------------- | -| **应用层** | 文件传输，电子邮件，文件服务，虚拟终端 | TFTP，HTTP，SNMP，FTP，SMTP，DNS，RIP，Telnet | -| **表示层** | 数据格式化，代码转换，数据加密 | 无 | -| **会话层** | 控制应用程序之间会话能力；如不同软件数据分发给不同软件 | ASAP、TLS、SSH、ISO 8327 / CCITT X.225、RPC、NetBIOS、ASP、Winsock、BSD sockets | -| **传输层** | 端到端传输数据的基本功能 | TCP、UDP | -| **网络层** | 定义IP编址，定义路由功能；如不同设备的数据转发 | IP，ICMP，RIP，OSPF，BGP，IGMP | -| **数据链路层** | 定义数据的基本格式，如何传输，如何标识 | SLIP，CSLIP，PPP，ARP，RARP，MTU | -| **物理层** | 以**二进制**数据形式在物理媒体上传输数据 | ISO2110，IEEE802 | +### 🎯 TCP和UDP协议的区别？ -当我们某一个网站上不去的时候。通常会ping一下这个网站 +都属于传输层协议。 -`ping` 可以说是ICMP的最著名的应用，是TCP/IP协议的一部分。利用`ping`命令可以检查网络是否连通，可以很好地帮助我们分析和判定网络故障。 +- TCP（Transmission Control Protocol，传输控制协议）是面向连接的协议，也就是说，在收发数据前，必须和对方建立可靠的连接。一个 TCP 连接必须有三次握手、四次挥手。 +- UDP（User Data Protocol，用户数据报协议）是一个非连接的协议，传输数据之前源端和终端不建立连接， 当它想传送时就简单地去抓取来自应用程序的数据，并尽可能快地把它扔到网络上 +| | TCP | UDP | +| :--------- | :--------------------------------------------- | :--------------------------- | +| 连接性 | 面向连接 | 面向非连接 | +| 传输可靠性 | 可靠 | 不可靠 | +| 报文 | 面向字节流 | 面向报文 | +| 效率 | 传输效率低 | 传输效率高 | +| 流量控制 | 滑动窗口 | 无 | +| 拥塞控制 | 慢开始、拥塞避免、快重传、快恢复 | 无 | +| 传输速度 | 慢 | 快 | +| 应用场合 | 对效率要求低，对准确性要求高或要求有连接的场景 | 对效率要求高，对准确性要求低 | -## 二、TCP/IP +TCP 和 UDP 协议的一些应用 -数据在网络中传输最终一定是通过物理介质传输。物理介质就是把电脑连接起来的物理手段，常见的有光纤、双绞线，以及无线电波，它决定了电信号(0和1)的传输方式，物理介质的不同决定了电信号的传输带宽、速率、传输距离以及抗干扰性等等。网络数据传输就像快递邮寄，数据就是快件。只有路打通了，你的”快递”才能送到，因此物理介质是网络通信的基石。 +![img](https://img.starfish.ink/network/nr4vfd9rjq.jpeg) -寄快递首先得称重、确认体积(确认数据大小)，贵重物品还得层层包裹填充物确保安全，封装，然后填写发件地址(源主机地址)和收件地址(目标主机地址)，确认快递方式。对于偏远地区，快递不能直达，还需要中途转发。网络通信也是一样的道理，只不过把这些步骤都规定成了各种协议。 -TCP/IP的模型的每一层都需要下一层所提供的协议来完成自己的目的。我们来看下数据是怎么通过TCP/IP协议模型从一台主机发送到另一台主机的。 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gds66lxjnqj30i00f6t9b.jpg) +### 🎯 TCP 连接的建立与终止？ -当用户通过HTTP协议发起一个请求，应用层、传输层、网络互联层和网络访问层的相关协议依次对该请求进行包装并携带对应的首部，最终在网络访问层生成以太网数据包，以太网数据包通过物理介质传输给对方主机，对方接收到数据包以后，然后再一层一层采用对应的协议进行拆包，最后把应用层数据交给应用程序处理。 +> TCP 和 UDP 的报文结构了解么 +TCP 虽然是面向字节流的，但TCP传送的数据单元却是报文段。一个 TCP 报文段分为首部和数据两部分，而 TCP 的全部功能体现在它首部中的各字段的作用。 +TCP 报文段首部的前 20 个字节是固定的（下图），后面有 4n 字节是根据需要而增加的选项（n是整数）。因此 TCP 首部的最小长度是20 字节。 -### TCP/IP 与 HTTP +``` + 源端口号（16位） | 目的端口号（16位） +----------------------------------------------- +序列号（32位） +----------------------------------------------- +确认号（32位） +----------------------------------------------- +数据偏移（4位）|保留位（6位）|控制位（6位）|窗口大小（16位） +----------------------------------------------- +校验和（16位）|紧急指针（16位） +----------------------------------------------- +选项（可选，最长40字节） +----------------------------------------------- +数据部分（长度可变） +``` -TCP/IP（Transmission Control Protocol/Internet Protocol，传输控制协议/网际协议）是指能够在多个不同网络间实现信息传输的协议簇。TCP/IP 协议不仅仅指的是 TCP 和 IP 两个协议，而是指一个由FTP、SMTP、TCP、UDP、IP等协议构成的协议簇， 只是因为在TCP/IP协议中TCP协议和IP协议最具代表性，所以被称为TCP/IP协议。 +![](https://img.starfish.ink/network/cs0sawogj4.jpeg) -**而HTTP是应用层协议，主要解决如何包装数据。** +**TCP报文首部** -“IP”代表网际协议，TCP 和 UDP 使用该协议从一个网络传送数据包到另一个网络。把**IP想像成一种高速公路**，它允许其它协议在上面行驶并找到到其它电脑的出口。**TCP和UDP是高速公路上的“卡车”，它们携带的货物就是像HTTP**，文件传输协议FTP这样的协议等。 +- 源端口和目的端口，各占 2 个字节，都是 16 位，分别写入源端口和目的端口； +- **序列号**（Sequence number），占4字节，32位。序号范围是【0，2^32 - 1】，共2^32个序号。序号增加到 2^32-1后，下一个序号就又回到 0。TCP是面向字节流的。在一个TCP连接中传送的字节流中的每一个字节都按顺序编号。整个要传送的字节流的起始序号必须在连接建立时设置。首部中的序号字段值则是指的是本报文段所发送的数据的第一个字节的序号。例如，一报文段的序号是301，而接待的数据共有100字节。这就表明：本报文段的数据的第一个字节的序号是301，最后一个字节的序号是400。显然，下一个报文段（如果还有的话）的数据序号应当从401开始，即下一个报文段的序号字段值应为401。这个字段的序号也叫“报文段序号”; +- **确认号**（Acknowledge number），占4个字节，是期望收到对方下一个报文的第一个数据字节的序号。例如，B收到了A发送过来的报文，其序列号字段是501，而数据长度是200字节，这表明B正确的收到了A发送的到序号700为止的数据。因此，B期望收到A的下一个数据序号是701，于是B在发送给A的确认报文段中把确认号置为701； +- **数据偏移（Data Offset）**，占 4 位，它指出TCP报文段的数据起始处距离TCP报文段的起始处有多远。 +- **保留（Reserved）**，占 6 位，保留为今后使用，但目前应置为0； +- **控制位（Control Flags）**： + - 6 位，包括 URG（紧急指针有效）、ACK（确认号有效）、PSH（推送功能）、RST（重置连接）、SYN（同步序列编号）、FIN（结束连接）。 +- **窗口大小（Window Size）**： + - 16 位，用于流量控制，指示接收方的接收窗口大小。 +- **校验和（Checksum）**： + - 16 位，用于错误检测，对 TCP 头部和数据部分进行校验。 +- **紧急指针（Urgent Pointer）**： + - 16 位，仅当 URG 标志位被设置时使用，指向数据中紧急数据的末尾。 +- **选项（Options）**： + - 可变长度，用于设置 TCP 参数，如最大报文段长度（MSS）、窗口缩放因子、选择性确认（SACK）等。 +- **填充（Padding）**： + - 确保 TCP 头部是 32 位字的整数倍。 +- **数据（Data）**： + - TCP 报文的实际数据部分，长度可变。 -### TCP 与 UDP +**UDP 报文头（8 字节，简单高效）** -都属于传输层协议。 +UDP 报文头非常精简，仅 4 个字段： -TCP（Transmission Control Protocol，传输控制协议）是面向连接的协议，也就是说，在收发数据前，必须和对方建立可靠的连接。一个TCP连接必须有三次握手、四次挥手。 +| 字段 | 长度 | 作用 | +| ---------------------------- | ------ | ------------------- | +| 源端口（Source Port） | 16 bit | 标识发送方端口 | +| 目标端口（Destination Port） | 16 bit | 标识接收方端口 | +| 长度（Length） | 16 bit | 报头 + 数据的总长度 | +| 校验和（Checksum） | 16 bit | 检测报文是否出错 | -UDP（User Data Protocol，用户数据报协议）是一个非连接的协议，传输数据之前源端和终端不建立连接， 当它想传送时就简单地去抓取来自应用程序的数据，并尽可能快地把它扔到网络上 + 特点： -| | TCP | UDP | -| :--------- | :--------------------------------------------- | :--------------------------- | -| 连接性 | 面向连接 | 面向非连接 | -| 传输可靠性 | 可靠 | 不可靠 | -| 报文 | 面向字节流 | 面向报文 | -| 效率 | 传输效率低 | 传输效率高 | -| 流量控制 | 滑动窗口 | 无 | -| 拥塞控制 | 慢开始、拥塞避免、快重传、快恢复 | 无 | -| 传输速度 | 慢 | 快 | -| 应用场合 | 对效率要求低，对准确性要求高或要求有连接的场景 | 对效率要求高，对准确性要求低 | +- 没有连接管理、序列号、确认机制。 +- 只提供最基本的端口定位和数据完整性校验。 +- **优势**：开销小，实时性好，常用于视频、语音、DNS。 -TCP和UDP协议的一些应用 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gds67b566dj30yw0r6gza.jpg) +### 🎯 介绍一下 TCP 的三次握手机制，为什么要三次握手？挥手却又是四次呢？ +TCP是一种面向连接的**单播协议**，在发送数据前，通信双方必须在彼此间建立一条连接。所谓的“连接”，其实是客户端和服务器的内存里保存的一份关于对方的信息，如ip地址、端口号等。 -### TCP连接的建立与终止 +**TCP 三次握手** -TCP虽然是面向字节流的，但TCP传送的数据单元却是报文段。一个TCP报文段分为首部和数据两部分，而TCP的全部功能体现在它首部中的各字段的作用。 +所谓三次握手(Three-way Handshake)，是指建立一个 TCP 连接时，需要客户端和服务器总共发送 3 个包。 -TCP报文段首部的前20个字节是固定的（下图），后面有4n字节是根据需要而增加的选项（n是整数）。因此TCP首部的最小长度是20字节。 +三次握手的目的是连接服务器指定端口，建立 TCP 连接，并同步连接双方的序列号和确认号，交换 TCP 窗口大小信息。 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gdr6e2kp1lj30u60jsdhn.jpg) +![img](https://miro.medium.com/v2/resize:fit:1102/0*8j0qdKAShOds5Cof.png) -#### TCP报文首部 +- **第一次握手**（SYN）：客户端向服务器发送一个SYN（Synchronize）报文段，用于请求建立连接。这个报文段包含客户端的初始序列号（ISN），用于同步序列号。 -- 源端口和目的端口，各占2个字节，分别写入源端口和目的端口； -- **序列号**（Sequence number），占4字节。序号范围是【0，2^32 - 1】，共2^32个序号。序号增加到 2^32-1后，下一个序号就又回到 0。TCP是面向字节流的。在一个TCP连接中传送的字节流中的每一个字节都按顺序编号。整个要传送的字节流的起始序号必须在连接建立时设置。首部中的序号字段值则是指的是本报文段所发送的数据的第一个字节的序号。例如，一报文段的序号是301，而接待的数据共有100字节。这就表明：本报文段的数据的第一个字节的序号是301，最后一个字节的序号是400。显然，下一个报文段（如果还有的话）的数据序号应当从401开始，即下一个报文段的序号字段值应为401。这个字段的序号也叫“报文段序号”; -- **确认号**（Acknowledge number），占4个字节，是期望收到对方下一个报文的第一个数据字节的序号。例如，B收到了A发送过来的报文，其序列号字段是501，而数据长度是200字节，这表明B正确的收到了A发送的到序号700为止的数据。因此，B期望收到A的下一个数据序号是701，于是B在发送给A的确认报文段中把确认号置为701； -- 数据偏移，占4位，它指出TCP报文段的数据起始处距离TCP报文段的起始处有多远。 -- 保留，占6位，保留为今后使用，但目前应置为0； -- 紧急URG（URGent），当URG=1，表明紧急指针字段有效。告诉系统此报文段中有紧急数据； -- 确认ACK（ACKnowledgment），仅当ACK=1时，确认号字段才有效。**TCP规定，在连接建立后所有报文的传输都必须把ACK置1**； -- 推送PSH（PuSH） ，当两个应用进程进行交互式通信时，有时在一端的应用进程希望在键入一个命令后立即就能收到对方的响应，这时候就将PSH=1； -- 复位RST（ReSeT），当RST=1，表明TCP连接中出现严重差错，必须释放连接，然后再重新建立连接； -- 同步SYN（SYNchronization），在连接建立时用来同步序号。**当SYN=1，ACK=0，表明是连接请求报文，若同意连接，则响应报文中应该使SYN=1，ACK=1**； -- 终止FIN（FINis），用来释放连接。**当FIN=1，表明此报文的发送方的数据已经发送完毕，并且要求释放**； - - 窗口，占2字节，指的是通知接收方，发送本报文你需要有多大的空间来接受； -- 检验和，占2字节，校验首部和数据这两部分； -- 紧急指针，占2字节，指出本报文段中的紧急数据的字节数； -- 选项，长度可变，定义一些其他的可选的参数 +- **第二次握手**（SYN-ACK）：服务器收到SYN报文后，确认收到请求，并向客户端发送一个SYN-ACK（Synchronize-Acknowledgment）报文。这个报文段包含服务器的初始序列号，并确认接收到的客户端的序列号。 +- **第三次握手**（ACK）：客户端收到服务器的SYN-ACK报文后，再次向服务器发送一个ACK（Acknowledgment）报文，确认收到服务器的序列号。至此，连接建立完毕，客户端和服务器可以开始数据传输。 -TCP是一种面向连接的单播协议，在发送数据前，通信双方必须在彼此间建立一条连接。所谓的“连接”，其实是客户端和服务器的内存里保存的一份关于对方的信息，如ip地址、端口号等。 -#### TCP 三次握手 -所谓三次握手(Three-way Handshake)，是指建立一个 TCP 连接时，需要客户端和服务器总共发送3个包。 +#### 为什么需要三次握手呢？两次不行吗？ -三次握手的目的是连接服务器指定端口，建立 TCP 连接，并同步连接双方的序列号和确认号，交换 TCP 窗口大小信息。 +- **确保双向通信**：三次握手可以确保通信的双向性。通过三次握手，客户端和服务器都确认了对方的存在，并且双方都同步了初始序列号，为接下来的数据传输做准备。 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gds67rwlvsj30ra0e9n07.jpg) +- **防止旧连接的混淆**：如果只有两次握手，可能会存在旧的、失效的 SYN 报文被误认为是新的连接请求，导致混淆。三次握手能够有效避免这种情况。 -- **第一次握手**(SYN=1, seq=x) +> 具体例子：“已失效的连接请求报文段”的产生在这样一种情况下：client 发出的第一个连接请求报文段并没有丢失，而是在某个网络结点长时间的滞留了，以致延误到连接释放以后的某个时间才到达 server。本来这是一个早已失效的报文段。但 server 收到此失效的连接请求报文段后，就误认为是 client 再次发出的一个新的连接请求。于是就向client发出确认报文段，同意建立连接。假设不采用“三次握手”，那么只要server发出确认，新的连接就建立了。由于现在client并没有发出建立连接的请求，因此不会理睬server的确认，也不会向server发送数据。但server却以为新的运输连接已经建立，并一直等待client发来数据。这样，server的很多资源就白白浪费掉了。采用“三次握手”的办法可以防止上述现象发生。例如刚才那种情况，client不会向server的确认发出确认。server由于收不到确认，就知道client并没有要求建立连接。” - 建立连接。客户端发送连接请求报文段，这是报文首部中的同步位SYN=1，同时选择一个初始序列号 seq=x ，此时，客户端进程进入了 SYN-SENT（同步已发送状态）状态。TCP规定，SYN报文段（SYN=1的报文段）不能携带数据，但需要消耗掉一个序号； -- **第二次握手**(SYN=1, ACK=1, seq=y, ACKnum=x+1) - 服务器收到客户端的SYN报文段，如果同意连接，则发出确认报文。确认报文中应该 ACK=1，SYN=1，确认号ACKnum=x+1，同时，自己还要发送SYN请求信息，SYN=1，为自己初始化一个序列号 seq=y，服务器端将上述所有信息放到一个报文段（即SYN+ACK报文段）中，一并发送给客户端，此时，TCP服务器进程进入了SYN-RCVD（同步收到）状态。这个报文也不能携带数据，但是同样要消耗一个序号 +#### TCP 四次挥手 -- **第三次握手**(ACK=1，ACKnum=y+1) +TCP 的连接释放过程需要 **四次挥手（Four-way Handshake）**，因为 TCP 是全双工的，关闭时两端都要单独发 FIN 包确认关闭。 - 客户端收到服务器的SYN+ACK报文段，再次发送确认包(ACK)，**SYN 标志位为0**，ACK 标志位为1，确认号 ACKnum = y+1，这个报文段发送完毕以后，客户端和服务器端都进入ESTABLISHED（已建立连接）状态，完成TCP三次握手。 +**过程拆解：** +1. **第一次挥手**： + - 主机 A（主动关闭方）发送 `FIN=1`，告诉主机 B：我这边没有数据要发了。 + - A 进入 `FIN_WAIT_1` 状态。 +2. **第二次挥手**： + - 主机 B 收到 FIN 后，返回 `ACK=1`，表示“我知道了”。 + - A 收到 ACK 后进入 `FIN_WAIT_2` 状态。 + - 注意：此时 B 可能还有数据要发给 A，所以连接还没完全关闭。 +3. **第三次挥手**： + - B 处理完数据后，发送 `FIN=1`，告诉 A：我也要关闭了。 + - B 进入 `LAST_ACK` 状态，等待 A 的确认。 +4. **第四次挥手**： + - A 收到 B 的 FIN 后，返回 `ACK=1`，表示确认。 + - A 进入 `TIME_WAIT` 状态，等待 **2MSL** 时间（两个最大报文生存时间），确保 B 收到自己的 ACK。 + - B 收到 ACK 后直接进入 `CLOSED` 状态，连接关闭。 + - A 在 `TIME_WAIT` 结束后也进入 `CLOSED`。 +![img](https://img.starfish.ink/network/tv0i5jc3xp.jpeg) -> 为什么需要三次握手呢？两次不行吗？ +TCP 的连接的拆除需要发送四个包，因此称为四次挥手(Four-way handshake)，也叫做改进的三次握手。**客户端或服务器均可主动发起挥手动作**。 -为了防止已失效的连接请求报文段突然又传送到了服务端，因而产生错误。 -具体例子：“已失效的连接请求报文段”的产生在这样一种情况下：client发出的第一个连接请求报文段并没有丢失，而是在某个网络结点长时间的滞留了，以致延误到连接释放以后的某个时间才到达server。本来这是一个早已失效的报文段。但server收到此失效的连接请求报文段后，就误认为是client再次发出的一个新的连接请求。于是就向client发出确认报文段，同意建立连接。假设不采用“三次握手”，那么只要server发出确认，新的连接就建立了。由于现在client并没有发出建立连接的请求，因此不会理睬server的确认，也不会向server发送数据。但server却以为新的运输连接已经建立，并一直等待client发来数据。这样，server的很多资源就白白浪费掉了。采用“三次握手”的办法可以防止上述现象发生。例如刚才那种情况，client不会向server的确认发出确认。server由于收不到确认，就知道client并没有要求建立连接。” +#### 为什么需要 四次？ +- **全双工通信的特性**：TCP是全双工通信协议，双方的发送和接收通道是独立的。在关闭连接时，双方都需要独立地关闭各自的发送和接收通道，因此需要四次挥手。（A 发 FIN 只是表示自己不再发送数据，但还能接收数据；所以 B 需要分开确认（ACK）和关闭（FIN）） -#### TCP 四次挥手 +- **确保数据完整传输**：四次挥手允许双方有机会处理完所有未发送的数据。即使主动关闭一方不再发送数据，被动关闭一方仍然可以继续发送尚未传输完毕的数据，直到确认所有数据都已接收。 -TCP 的连接的拆除需要发送四个包，因此称为四次挥手(Four-way handshake)，也叫做改进的三次握手。**客户端或服务器均可主动发起挥手动作**。 +> **由于 TCP 协议是全双工的，也就是说客户端和服务端都可以发起断开连接。两边各发起一次断开连接的申请，加上各自的两次确认，看起来就像执行了四次挥手**。 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gds6835mzjj30qw0g4dit.jpg) +#### 为什么需要 **TIME_WAIT（2MSL）**？ -- 第一次挥手(FIN=1，seq=x) +MSL 是Maximum Segment Lifetime英文的缩写，中文可以译为“报文最大生存时间”，他是任何报文在网络上存在的最长时间，超过这个时间报文将被丢弃。 - 主机1（可以使客户端，也可以是服务器端），设置seq=x，向主机2发送一个FIN报文段；此时，主机1进入`FIN_WAIT_1`状态；这表示主机1没有数据要发送给主机2了； +2MSL 是两倍的这个时间。 -- 第二次挥手(ACK=1，ACKnum=x+1) +虽然按道理，四个报文都发送完毕，我们可以直接进入 CLOSE 状态了，但是我们必须假想网络是不可靠的，有可能最后一个ACK丢失。所以 TIME_WAIT 状态就是用来重发可能丢失的 ACK 报文。 - 主机2收到了主机1发送的FIN报文段，向主机1回一个ACK报文段，Acknnum=x+1，主机1进入`FIN_WAIT_2`状态；主机2告诉主机1，我“同意”你的关闭请求； +还有一个原因，防止类似与“三次握手”中提到了的“已经失效的连接请求报文段”出现在本连接中。客户端发送完最后一个确认报文后，在这个 2MSL 时间中，就可以使本连接持续的时间内所产生的所有报文段都从网络中消失。这样新的连接中不会出现旧连接的请求报文。 -- 第三次挥手(FIN=1，seq=y) +主机 1 等待了某个固定时间（两个最大段生命周期，2MSL，2 Maximum Segment Lifetime）之后，没有收到服务器端的 ACK ，认为服务器端已经正常关闭连接，于是自己也关闭连接，进入 `CLOSED` 状态。 - 主机2向主机1发送FIN报文段，请求关闭连接，同时主机2进入`LAST_ACK` 状态 +> **Linux 系统**：默认 MSL 通常为 **60 秒**（可通过 `/proc/sys/net/ipv4/tcp_fin_timeout` 查看），因此 2MSL 对应 **120 秒（120,000 毫秒）** +> +> 1. 确保最后一个 ACK 能到达 B，避免 B 认为没有收到 ACK 而重发 FIN。 +> 2. 保证旧连接的报文不会影响新连接（等待足够长，旧报文会自然消失）。 -- 第四次挥手(ACK=1，ACKnum=y+1) - 主机1收到主机2发送的FIN报文段，向主机2发送ACK报文段，然后主机1进入`TIME_WAIT`状态；主机2收到主机1的ACK报文段以后，就关闭连接；此时，**主机1等待2MSL后依然没有收到回复**，则证明Server端已正常关闭，那好，主机1也可以关闭连接了，进入 `CLOSED` 状态。 - +### 🎯 UDP 为什么是不可靠的？bind 和 connect 对于 UDP 的作用是什么 - 主机 1 等待了某个固定时间（两个最大段生命周期，2MSL，2 Maximum Segment Lifetime）之后，没有收到服务器端的 ACK ，认为服务器端已经正常关闭连接，于是自己也关闭连接，进入 `CLOSED` 状态。 +UDP 只有一个 socket 接收缓冲区，没有 socket 发送缓冲区，即只要有数据就发，不管对方是否可以正确接收。而在对方的 socket 接收缓冲区满了之后，新来的数据报无法进入到 socket 接受缓冲区，此数据报就会被丢弃，因此 UDP 不能保证数据能够到达目的地，此外，UDP 也没有流量控制和重传机制，故UDP的数据传输是不可靠的。 +和 TCP 建立连接时采用三次握手不同，UDP 中调用 connect 只是把对端的 IP 和 端口号记录下来，并且 UDP 可多多次调用 connect 来指定一个新的 IP 和端口号，或者断开旧的 IP 和端口号（通过设置 connect 函数的第二个参数）。和普通的 UDP 相比，调用 connect 的 UDP 会提升效率，并且在高并发服务中会增加系统稳定性。 +当 UDP 的发送端调用 bind 函数时，就会将这个套接字指定一个端口，若不调用 bind 函数，系统内核会随机分配一个端口给该套接字。当手动绑定时，能够避免内核来执行这一操作，从而在一定程度上提高性能。 -> 为什么连接的时候是三次握手，关闭的时候却是四次握手？ -因为当Server端收到Client端的SYN连接请求报文后，可以直接发送SYN+ACK报文。其中ACK报文是用来应答的，SYN报文是用来同步的。但是关闭连接时，当Server端收到FIN报文时，很可能并不会立即关闭SOCKET，所以只能先回复一个ACK报文，告诉Client端，"你发的FIN报文我收到了"。只有等到我Server端所有的报文都发送完了，我才能发送FIN报文，因此不能一起发送。故需要四步握手。 -由于 TCP 协议是全双工的，也就是说客户端和服务端都可以发起断开连接。两边各发起一次断开连接的申请，加上各自的两次确认，看起来就像执行了四次挥手。 +### 🎯 TCP 协议如何来保证传输的可靠性？ +> 如何设计一个稳定的 UDP 协议， 也可以参考这个回答 +1. **数据分段与重组** + + - **分段**：TCP将应用层数据分割成多个数据段（segment），每个数据段都附有一个序列号（Sequence Number），用于标识该段在整个数据流中的位置。 + + - **重组**：接收方根据序列号将接收到的数据段重新组装成原始数据，确保数据按正确的顺序传递给应用层。 + +2. **确认应答机制（Acknowledgment）** + - **确认（ACK）**：接收方在收到数据段后，会向发送方发送一个确认（ACK）报文，告知发送方已成功接收到数据段，并且表明下一个期望接收的数据段的序列号。 + + - **超时重传**：发送方在发送数据段后，会启动一个定时器，如果在一定时间内没有收到接收方的ACK报文，发送方会认为数据段丢失，并重新发送该数据段。 + +3. **滑动窗口机制（Sliding Window）** + - **窗口大小**：TCP使用滑动窗口来控制发送方可以连续发送而不必等待确认的最大数据量。窗口大小是动态调整的，取决于接收方的处理能力和网络的实际状况。 + + - **流量控制**：滑动窗口机制还实现了流量控制，防止发送方发送过多的数据段，导致接收方无法及时处理，从而引发网络拥塞或数据丢失。 + +4. **重传机制** + - **超时重传**：如果发送方在预定的时间内没有收到ACK，TCP会触发超时重传，将未被确认的数据段重新发送。 + + - **快速重传**：当接收方发现数据段丢失时，会发送多个重复的ACK报文，发送方在收到三个相同的ACK报文后，不等待定时器超时，直接重传丢失的数据段。 + +5. **拥塞控制** + - **拥塞避免（Congestion Avoidance）**：一旦到达拥塞阈值，窗口大小增长速度会放缓，采取线性增长方式，以进一步防止拥塞。 + + - **快速恢复（Fast Recovery）**：在快速重传后，发送方会将窗口大小减半，并继续数据传输，而不进入慢启动阶段，尽量保持网络效率。 + +6. **数据校验** + - **校验和（Checksum）**：每个TCP报文段都包含一个校验和字段，用于验证数据在传输过程中是否被修改。接收方会计算收到的数据段的校验和，并与报文段中的校验和进行比较，如果不匹配，则认为数据有误，丢弃该段，并请求重传。 -> **为什么TIME_WAIT状态需要经过2MSL(最大报文段生存时间)才能返回到CLOSE状态？** +7. **序列号与确认号** + - **序列号（Sequence Number）**：用于标识发送的数据段在数据流中的位置。 + + - **确认号（Acknowledgment Number）**：用于通知发送方下一个期望接收的数据段的序列号。序列号与确认号结合使用，确保所有数据段都能被正确接收和重组。 + +8. **连接管理（Connection Management）** -虽然按道理，四个报文都发送完毕，我们可以直接进入CLOSE状态了，但是我们必须假象网络是不可靠的，有可以最后一个ACK丢失。所以TIME_WAIT状态就是用来重发可能丢失的ACK报文。 + - **三次握手**：在建立连接时，通过三次握手机制确认通信双方都已准备好，确保连接的可靠性。 + + + - **四次挥手**：在断开连接时，通过四次挥手机制确保双方都已经完成数据传输，安全地关闭连接。 + -还有一个原因，防止类似与“三次握手”中提到了的“已经失效的连接请求报文段”出现在本连接中。客户端发送完最后一个确认报文后，在这个2MSL时间中，就可以使本连接持续的时间内所产生的所有报文段都从网络中消失。这样新的连接中不会出现旧连接的请求报文。 +通过以上这些机制，TCP能够在一个不可靠的网络环境中，提供端到端的可靠数据传输服务。这些机制确保数据完整、有序地传输，并能够适应网络中的动态变化。 -### TCP协议如何来保证传输的可靠性 -对于可靠性，TCP通过以下方式进行保证： -- 数据包校验：目的是检测数据在传输过程中的任何变化，若校验出包有错，则丢弃报文段并且不给出响应，这时TCP发送数据端超时后会重发数据； +### 🎯 详细讲一下TCP的滑动窗口？知道流量控制和拥塞控制吗？ -- 对失序数据包重排序：既然TCP报文段作为IP数据报来传输，而IP数据报的到达可能会失序，因此TCP报文段的到达也可能会失序。TCP将对失序数据进行重新排序，然后才交给应用层； +**1. 滑动窗口（Sliding Window）——流量控制的核心机制** -- 丢弃重复数据：对于重复数据，能够丢弃重复数据； +- **本质**：窗口是缓冲区的可视化抽象，用于限制「未被确认的数据」数量。 -- 应答机制：当TCP收到发自TCP连接另一端的数据，它将发送一个确认。这个确认不是立即发送，通常将推迟几分之一秒； +- **两类窗口**： -- 超时重发：当TCP发出一个段后，它启动一个定时器，等待目的端确认收到这个报文段。如果不能及时收到一个确认，将重发这个报文段； + - **发送窗口**：分为已发送已确认、已发送未确认、可发送、不可发送四个区域。 + - **接收窗口**：表示接收方还能接收多少数据，通过 ACK 包的 `Window` 字段告诉发送方。 -- 流量控制：TCP连接的每一方都有固定大小的缓冲空间。TCP的接收端只允许另一端发送接收端缓冲区所能接纳的数据，这可以防止较快主机致使较慢主机的缓冲区溢出，这就是流量控制。TCP使用的流量控制协议是可变大小的滑动窗口协议。 +- **原理**： + 发送方在窗口范围内可以连续发送多个报文段，不必逐个等待 ACK；接收方确认后，窗口右移，新的数据进入窗口。 + > 如果发送方把数据发送得过快，接收方可能会来不及接收，这就会造成数据的丢失。`所谓流量控制就是让发送方的发送速率不要太快`，要让接收方来得及接收。 + > + > 利用**滑动窗口机制**可以很方便地在 TCP 连接上实现对发送方的流量控制。 + > + > ![](https://img.starfish.ink/network/sliding-window-sender-side-cumulative-acknowledgments-n.jpg) + > + > 从上面的图可以看到滑动窗口左边的是已发送并且被确认的分组，滑动窗口右边是还没有轮到的分组。 + > + > 滑动窗口里面也分为两块，一块是已经发送但是未被确认的分组，另一块是窗口内等待发送的分组。随着已发送的分组不断被确认，窗口内等待发送的分组也会不断被发送。整个窗口就会往右移动，让还没轮到的分组进入窗口内。 + > + > **滑动窗口的工作原理** + > + > 1. **窗口大小（Window Size）**：滑动窗口的大小表示发送方在等待接收方确认之前，可以发送的未确认数据段的总量。窗口大小是一个动态的值，由接收方通过TCP报文中的窗口字段（window field）告知发送方。 + > 2. **发送窗口（Send Window）**：发送方维护一个发送窗口，表示可以发送的数据段范围。窗口左边界是已确认的数据段，右边界是窗口的大小。发送方可以在发送窗口范围内连续发送数据段，而无需等待前一个数据段的确认。 + > 3. **接收窗口（Receive Window）**：接收方也维护一个接收窗口，表示可以接收的数据段范围。接收方会根据自己的缓冲区大小，动态调整接收窗口的大小，并通过ACK报文通知发送方。 + > 4. **窗口滑动**：每当发送方收到接收方的ACK确认报文时，窗口会向前滑动，允许发送方继续发送更多的数据段。窗口的滑动意味着已确认的数据段从发送窗口中移出，新的未发送数据段可以加入发送窗口。 +**2. 流量控制（Flow Control）** -> 详细讲一下TCP的滑动窗口 +- **目的**：避免**接收方被撑爆**。 +- **实现**：基于滑动窗口，接收方根据自己的缓冲区大小，动态调整 `rwnd`（receiver window），在 ACK 中返回给发送方。 +- **典型场景**： + 如果接收方应用层消费太慢，缓冲区快满，就会把窗口缩小甚至设为 0，发送方必须暂停发送，直到窗口更新。 -### 滑动窗口机制 +**3. 拥塞控制（Congestion Control）** -如果发送方把数据发送得过快，接收方可能会来不及接收，这就会造成数据的丢失。所谓**流量控制**就是让发送方的发送速率不要太快，要让接收方来得及接收。 +- **目的**：避免**网络被撑爆**（链路丢包、路由器队列溢出）。 +- **方法**：TCP 在发送方额外维护一个 **拥塞窗口 cwnd**，实际能发的窗口 = `min(rwnd, cwnd)`。 +- **四个核心算法**： + 1. **慢启动**：cwnd 从 1 MSS 开始，每次 ACK 指数级增长（1→2→4→8…），直到阈值。 + 2. **拥塞避免**：达到阈值后改为线性增长。 + 3. **快速重传**：收到 3 个重复 ACK，立即重传丢失报文，而不用等超时。 + 4. **快速恢复**：出现丢包后 cwnd 减半而不是归 1，避免网络利用率大幅下降。 -利用**滑动窗口机制**可以很方便地在TCP连接上实现对发送方的流量控制。 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gds6d75md5j30si0ermyk.jpg) -从上面的图可以看到滑动窗口左边的是已发送并且被确认的分组，滑动窗口右边是还没有轮到的分组。滑动窗口里面也分为两块，一块是已经发送但是未被确认的分组，另一块是窗口内等待发送的分组。随着已发送的分组不断被确认，窗口内等待发送的分组也会不断被发送。整个窗口就会往右移动，让还没轮到的分组进入窗口内。 +### 🎯 如果接收方滑动窗口满了，发送方会怎么做？ -可以看到滑动窗口起到了一个限流的作用，也就是说当前滑动窗口的大小决定了当前 TCP 发送包的速率，而滑动窗口的大小取决于拥塞控制窗口和流量控制窗口的两者间的最小值。 +> 如果接收方的滑动窗口满了，它会在 ACK 中通告 `rwnd=0`。发送方收到后会暂停发送，并启动 **持续计时器**定期发送探测报文，避免零窗口死锁。一旦接收方窗口恢复（rwnd>0），发送方就会继续正常发送数据。 +**接收方滑动窗口满了** +- 接收方的缓冲区写满了，无法再接收新的数据。 +- 此时，接收方在返回的 **ACK 包里，把 `rwnd`（接收窗口大小）置为 0**，告诉发送方“暂停发送”。 -#### 流量控制 +**发送方的行为** -TCP 是全双工的，客户端和服务器均可作为发送方或接收方，我们现在假设一个发送方向接收方发送数据的场景来讲解流量控制。首先我们的接收方有一块接收缓存，当数据来到时会先把数据放到缓存中，上层应用等缓存中有数据时就会到缓存中取数据。假如发送方没有限制地不断地向接收方发送数据，接收方的应用程序又没有及时把接收缓存中的数据读走，就会出现缓存溢出，数据丢失的现象，为了解决这个问题，我们引入流量控制窗口。 +1. **停止发送新的数据**：因为窗口大小是 0，不能再发，避免接收方溢出。 +2. **保留已发送但未确认的数据**：发送方的发送窗口会冻结在某个位置，等待 ACK。 +3. **启动持续计时器（Persist Timer）**： + - 防止“零窗口死锁”——如果接收方窗口恢复了，但 ACK 包丢了，发送方可能永远收不到更新的窗口信息。 + - 所以发送方会定时发送 **探测报文（Window Probe）**，确认接收方窗口是否恢复。 +4. **窗口恢复后继续发送**：接收方应用层消费掉数据，缓冲区有空间，会在 ACK 中汇报一个大于 0 的 `rwnd`，发送方就恢复发送。 -假设应用程序最后读走的数据序号是 lastByteRead，接收缓存中接收到的最后一个数据序号是 lastByteRcv，接收缓存的大小为 RcvSize，那么必须要满足 lastByteRcv - lastByteRead <= RcvSize 才能保证接收缓存不会溢出，所以我们定义流量窗口为接收缓存剩余的空间，也就是Rcv = RcvSize - (lastByteRcv - lastByteRead)。只要接收方在响应 ACK 的时候把这个窗口的值带给发送方，发送方就能知道接收方的接收缓存还有多大的空间，进而设置滑动窗口的大小。 +### 🎯 滑动窗口、流量控制与拥塞控制的关系 -#### 拥塞控制 +- **滑动窗口**：是TCP实现高效数据传输的基本机制，它在不等待每个数据段确认的情况下，允许发送多个数据段。这一机制与流量控制和拥塞控制密切相关。 +- **流量控制**：通过接收窗口的调整，控制发送方的发送速度，确保接收方能够处理接收到的数据。 +- **拥塞控制**：通过动态调整拥塞窗口（cwnd），管理发送方的发送速率，以防止网络拥塞，确保网络的稳定性和数据传输的可靠性。 -拥塞控制是指发送方先设置一个小的窗口值作为发送速率，当成功发包并接收到ACK时，便以指数速率增大发送窗口的大小，直到遇到丢包（超时/三个冗余ACK），才停止并调整窗口的大小。这么做能最大限度地利用带宽，又不至于让网络环境变得太过拥挤。 -最终滑动窗口的值将设置为流量控制窗口和拥塞控制窗口中的较小值。 +### 🎯 TCP的拥塞处理 ？ +计算机网络中的带宽、交换结点中的缓存及处理机等都是网络的资源。在某段时间，若对网络中某一资源的需求超过了该资源所能提供的可用部分，网络的性能就会变坏，这种情况就叫做拥塞。 -### TCP的拥塞处理 +拥塞控制就是防止过多的数据注入网络中，这样可以使网络中的路由器或链路不致过载。注意，**拥塞控制和流量控制不同，前者是一个全局性的过程，而后者指点对点通信量的控制**。拥塞控制的方法主要有以下四种： -计算机网络中的带宽、交换结点中的缓存及处理机等都是网络的资源。在某段时间，若对网络中某一资源的需求超过了该资源所能提供的可用部分，网络的性能就会变坏，这种情况就叫做拥塞。拥塞控制就是防止过多的数据注入网络中，这样可以使网络中的路由器或链路不致过载。注意，拥塞控制和流量控制不同，前者是一个全局性的过程，而后者指点对点通信量的控制。拥塞控制的方法主要有以下四种： +##### 1. **慢启动（Slow Start）** -1. 慢启动：不要一开始就发送大量的数据，先探测一下网络的拥塞程度，也就是说由小到大逐渐增加拥塞窗口的大小; -2. 拥塞避免：拥塞避免算法让拥塞窗口缓慢增长，即每经过一个往返时间RTT就把发送方的拥塞窗口cwnd加1，而不是加倍，这样拥塞窗口按线性规律缓慢增长。　　　　　　　　　　 -3. 快重传：快重传要求接收方在收到一个 失序的报文段 后就立即发出 重复确认（为的是使发送方及早知道有报文段没有到达对方）而不要等到自己发送数据时捎带确认。快重传算法规定，发送方只要一连收到三个重复确认就应当立即重传对方尚未收到的报文段，而不必继续等待设置的重传计时器时间到期。　　　　　　　　　 -4. 快恢复：快重传配合使用的还有快恢复算法，当发送方连续收到三个重复确认时，就执行“乘法减小”算法，把ssthresh门限减半，但是接下去并不执行慢开始算法：因为如果网络出现拥塞的话就不会收到好几个重复的确认，所以发送方现在认为网络可能没有出现拥塞。所以此时不执行慢开始算法，而是将cwnd设置为ssthresh的大小，然后执行拥塞避免算法。 +- **目标**：初始阶段探测网络容量，避免突发流量导致拥塞。 +- 机制： + - 发送方初始拥塞窗口（`cwnd`）设为 1 个 MSS（最大段大小），每次收到确认后，`cwnd` 按指数增长（翻倍）。 + - 当 `cwnd` 超过拥塞阈值（`ssthresh`）时，切换至「拥塞避免」阶段。 +- **示例**：若初始 `cwnd=1`，则第 1 轮发 1 个包，第 2 轮发 2 个包，第 3 轮发 4 个包，依此类推，直到达到 `ssthresh`。 +##### 2. **拥塞避免（Congestion Avoidance）** +- **目标**：在探测到网络拥塞前，缓慢增加发送速率，避免过载。 +- 机制： + - 当 `cwnd ≥ ssthresh` 时，进入线性增长阶段：每经过一个 RTT（往返时间），`cwnd` 仅增加 1 个 MSS。 + - 此阶段 `cwnd` 按线性规律增长，降低拥塞风险。 +- **示例**：若 `ssthresh=8`，`cwnd` 从 8 开始，每轮 RTT 增加 1，即 8→9→10→… -### 服务器出现了大量CLOSE_WAIT状态如何解决 +##### 3. **快重传（Fast Retransmit）** -大量 CLOSE_WAIT 表示程序出现了问题，对方的 socket 已经关闭连接，而我方忙于读或写没有及时关闭连接，需要检查代码，特别是释放资源的代码，或者是处理请求的线程配置。 +- **目标**：快速检测并重传丢失的数据包，减少丢包导致的延迟。 +- 机制： + - 接收方收到失序报文段时，立即发送重复确认（Duplicate ACK），不等待确认。 + - 发送方若收到 **3 个重复 ACK**，判定数据包丢失，立即重传该包（无需等待超时）。 +- **优势**：相比超时重传（需等待 RTO 时间），快重传可减少约一半的延迟。 +##### 4. **快恢复（Fast Recovery）** +- **目标**：在快重传后，快速恢复发送速率，避免过度降低带宽利用率。 +- 机制： + 1. 收到 3 个重复 ACK 时，执行「乘法减小」：`ssthresh = cwnd / 2`（门限减半）。 + 2. 但此时不进入慢启动，而是将 `cwnd` 设置为 `ssthresh`，直接进入拥塞避免阶段（线性增长）。 +- **逻辑**：发送方认为丢包可能由短暂拥塞引起，而非网络严重过载，因此无需从 `cwnd=1` 重新开始。 -### 讲一讲SYN超时，洪泛攻击，以及解决策略 -什么 SYN 是洪泛攻击？ 在 TCP 的三次握手机制的第一步中，客户端会向服务器发送 SYN 报文段。服务器接收到 SYN 报文段后会为该TCP分配缓存和变量，如果攻击分子大量地往服务器发送 SYN 报文段，服务器的连接资源终将被耗尽，导致内存溢出无法继续服务。 -解决策略： 当服务器接受到 SYN 报文段时，不直接为该 TCP 分配资源，而只是打开一个半开的套接字。接着会使用 SYN 报文段的源Id，目的Id，端口号以及只有服务器自己知道的一个秘密函数生成一个 cookie，并把 cookie 作为序列号响应给客户端。 +### 🎯 拥塞控制和流量控制的本质区别是什么？ -如果客户端是正常建立连接，将会返回一个确认字段为 cookie + 1 的报文段。接下来服务器会根据确认报文的源Id，目的Id，端口号以及秘密函数计算出一个结果，如果结果的值 + 1等于确认字段的值，则证明是刚刚请求连接的客户端，这时候才为该 TCP 分配资源 +流量控制是端到端的控制（接收方→发送方），目标是保护接收方不被过量数据淹没； -这样一来就不会为恶意攻击的 SYN 报文段分配资源空间，避免了攻击。 +而拥塞控制是全局控制（发送方根据网络状态自我调整），目标是防止网络中所有节点因过载而丢包。前者关注接收方处理能力，后者关注网络整体容量。 +| **维度** | **流量控制** | **拥塞控制** | +| ------------ | ------------------------------------ | --------------------------------- | +| **核心目标** | 避免接收方缓冲区溢出 | 避免网络拥塞（路由器队列溢出） | +| **控制方** | 接收方（通过 Window 字段通知发送方） | 发送方（根据网络状态自适应调整） | +| **影响因素** | 接收方缓冲区剩余空间 | 网络链路带宽、路由器队列容量 | +| **典型机制** | 滑动窗口、糊涂窗口避免 | 慢启动、拥塞避免、快速重传 / 恢复 | +| **触发条件** | 接收方处理速度慢于数据到达速度 | 网络中数据量超过链路承载能力 | -## 三、HTTP -> HTTP1.0、HTTP1.1、HTTP2.0 的区别 -> -> post 和 get 的区别 +### 🎯 TCP 超时重传的原理 -HTTP全称是 HyperText Transfer Protocal，即：超文本传输协议。是互联网上应用最为广泛的一种**网络通信协议**，它允许将超文本标记语言（HTML）文档从Web服务器传送到客户端的浏览器。目前我们使用的是**HTTP/1.1 版本**。所有的WWW文件都必须遵守这个标准。设计HTTP最初的目的是为了提供一种发布和接收HTML页面的方法。1960年美国人 Ted Nelson 构思了一种通过计算机处理文本信息的方法，并称之为超文本（hypertext），这成为了HTTP超文本传输协议标准架构的发展根基。 +发送方在发送一次数据后就开启一个定时器，在一定时间内如果没有得到发送数据包的 ACK 报文，那么就重新发送数据，在达到一定次数还没有成功的话就放弃重传并发送一个复位信号。其中超时时间的计算是超时的核心，而定时时间的确定往往需要进行适当的权衡，因为当定时时间过长会造成网络利用率不高，定时太短会造成多次重传，使得网络阻塞。在 TCP 连接过程中，会参考当前的网络状况从而找到一个合适的超时时间。 -### URI 和 URL -每个Web 服务器资源都有一个名字，这样客户端就可以说明他们感兴趣的资源是什么了，服务器资源名被称为统一资源标识符（Uniform Resource Identifier,URI）。URI 就像因特网上的邮政地址一样，在世界范围内唯一标识并定位信息资源。 -统一资源定位符（URL）是资源标识符最常见的形式。 URL 描述了一台特定服务器上某资源的特定位置。 +### 🎯 TCP 的停止等待协议是什么 -现在几乎所有的 URI 都是 URL。 +停止等待协议是为了实现 TCP 可靠传输而提出的一种相对简单的协议，该协议指的是发送方每发完一组数据后，直到收到接收方的确认信号才继续发送下一组数据。我们通过四种情形来帮助理解停等协议是如何实现可靠传输的： -URI 的第二种形式就是统一资源名（URN）。URN 是作为特定内容的唯一名称使用的，与目前的资源所在地无关。　 +![img](https://p9-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/eb64698e5881443d9455c76bf01597e0~tplv-k3u1fbpfcp-watermark.awebp) -### HTTP消息的结构 +① 无差错传输 -**事务和报文** +如上述左图所示，A 发送分组 Msg 1，发完就暂停发送，直到收到接收方确认收到 Msg 1 的报文后，继续发送 Msg 2，以此类推，该情形是通信中的一种理想状态。 -客户端是怎样通过HTTP与Web服务器及其资源进行事务处理的呢？一个**HTTP事务**由一条请求命令（从客户端发往服务器）和一个响应（从服务器发回客户端）结果组成。这种通信是通过名为**HTTP报文**（HTTP Message）的格式化数据块进行的。 +② 出现差错 -#### HTTP事务： +如上述右图所示，发送方发送的报文出现差错导致接收方不能正确接收数据，出现差错的情况主要分为两种： -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gds68n7iduj30jx08caag.jpg) +- 发送方发送的 Msg 1 在中途丢失了，接收方完全没收到数据。 +- 接收方收到 Msg 1 后检测出现了差错，直接丢弃 Msg 1。 -#### 报文： +上面两种情形，接收方都不会回任何消息给发送方，此时就会触发超时传输机制，即发送方在等待一段时间后仍然没有收到接收方的确认，就认为刚才发送的数据丢失了，因此重传前面发送过的数据。 -HTTP 报文是纯文本，不是二进制代码。从 Web 客户端发往 Web 服务器的 HTTP 报文称为请求报文（request message）。从服务器发往客户端的报文称为响应报文。 +![img](https://p9-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/99b8534c772647f3b0fc7c41272e72f3~tplv-k3u1fbpfcp-watermark.awebp) -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gdpnh0qepnj30p006eacq.jpg) +③ 确认丢失 -HTTP 报文包括三部分： +当接收方回应的 Msg 1 确认报文在传输过程中丢失，发送方无法接收到确认报文。于是发送方等待一段时间后重传 Msg 1，接收方将收到重复的 Msg1 数据包，此时接收方会丢弃掉这个重复报文并向发送方再次发送 Msg1 的确认报文。 -- 起始行 -- 首部字段 -- 主体 +④ 确认迟到 -#### 常见HTTP 首部字段： +当接收方回应的 Msg 1 确认报文由于网络各种原因导致发送方没有及时收到，此时发送方在超时重传机制的作用下再次发送了 Msg 数据包，接收方此时进行和确认丢失情形下相同的动作（丢弃重复的数据包并再次发送 Msg 1 确认报文）。发送方此时收到了接收方的确认数据包，于是继续进行数据发送。过了一段时间后，发送方收到了迟到的 Msg 1 确认包会直接丢弃。 -**a、通用首部字段**（请求报文与响应报文都会使用的首部字段） +上述四种情形即停止等待协议中所出现的所有可能情况。 -- Date：创建报文时间 -- Connection：连接的管理 -- Cache-Control：缓存的控制 -- Transfer-Encoding：报文主体的传输编码方式 -**b、请求首部字段**（请求报文会使用的首部字段） -- Host：请求资源所在服务器 -- Accept：可处理的媒体类型 -- Accept-Charset：可接收的字符集 -- Accept-Encoding：可接受的内容编码 -- Accept-Language：可接受的自然语言 +### 🎯 TCP 最大连接数限制 -**c、响应首部字段（**响应报文会使用的首部字段） +- **Client 最大 TCP 连接数** -- Accept-Ranges：可接受的字节范围 -- Location：令客户端重新定向到的URI -- Server：HTTP服务器的安装信息 + client 在每次发起 TCP 连接请求时，如果自己并不指定端口的话，系统会随机选择一个本地端口（local port），该端口是独占的，不能和其他 TCP 连接共享。TCP 端口的数据类型是 unsigned short，因此本地端口个数最大只有 65536，除了端口 0不能使用外，其他端口在空闲时都可以正常使用，这样可用端口最多有 65535 个。 -**d、实体首部字段**（请求报文与响应报文的的实体部分使用的首部字段） +- **Server最大 TCP 连接数** -- Allow：资源可支持的HTTP方法 -- Content-Type：实体主类的类型 -- Content-Encoding：实体主体适用的编码方式 -- Content-Language：实体主体的自然语言 -- Content-Length：实体主体的的字节数 -- Content-Range：实体主体的位置范围，一般用于发出部分请求时使用 + server 通常固定在某个本地端口上监听，等待 client 的连接请求。不考虑地址重用（Unix 的 SO_REUSEADDR 选项）的情况下，即使 server 端有多个 IP，本地监听端口也是独占的，因此 server 端 TCP 连接 4 元组中只有客户端的 IP 地址和端口号是可变的，因此最大 TCP 连接为客户端 IP 数 × 客户端 port 数，对 IPV4，在不考虑 IP 地址分类的情况下，最大 TCP 连接数约为 2 的 32 次方（IP 数）× 2 的 16 次方（port 数），也就是 server 端单机最大 TCP 连接数约为 2 的 48 次方。 + 然而上面给出的是只是理论上的单机最大连接数，在实际环境中，受到明文规定（一些 IP 地址和端口具有特殊含义，没有对外开放）、机器资源、操作系统等的限制，特别是 sever 端，其最大并发 TCP 连接数远不能达到理论上限。对 server 端，通过增加内存、修改最大文件描述符个数等参数，单机最大并发 TCP 连接数超过 10 万 是没问题的。 -### 方法 -Http协议定义了很多与服务器交互的方法，最基本的有4种，分别是**GET,POST,PUT,DELETE**. 一个URL地址用于描述一个网络上的资源，而HTTP中的GET, POST, PUT, DELETE就对应着对这个资源的查，改，增，删4个操作。 我们最常见的就是GET和POST了。GET一般用于获取/查询资源信息，而POST一般用于更新资源信息。 +### 🎯 TCP 连接client和server有哪些状态? -- GET -- HEAD -- PUT -- POST -- TRACE -- OPTIONS -- DELETE +在 TCP 连接建立和断开过程中，客户端和服务器会经历一系列状态转换，这些状态定义了 TCP 连接的不同阶段。了解这些状态有助于理解 TCP 如何建立、维护和终止连接。 +**TCP连接的主要状态** +1. **CLOSED**：初始状态。这个状态表示连接未建立或者已关闭。 + - **适用对象**：客户端或服务器在启动前或连接结束后。 +2. **LISTEN**：服务器处于监听状态，等待客户端发起连接请求。 + - **适用对象**：服务器。 +3. **SYN-SENT**：客户端发送了SYN报文并等待服务器的SYN-ACK响应。这是三次握手的第一步。 + - **适用对象**：客户端。 +4. **SYN-RECEIVED**：服务器收到客户端的SYN报文后，发送SYN-ACK报文，并等待客户端的ACK。这是三次握手的第二步。 + - **适用对象**：服务器。 +5. **ESTABLISHED**：连接已经建立，客户端和服务器可以进行数据传输。 + - **适用对象**：客户端和服务器。 +6. **FIN-WAIT-1**：客户端或服务器主动关闭连接，发送FIN报文，并等待对方的ACK。 + - **适用对象**：主动关闭连接的一方。 +7. **FIN-WAIT-2**：主动关闭连接的一方收到对方的ACK后，进入FIN-WAIT-2状态，等待对方发送FIN报文。 + - **适用对象**：主动关闭连接的一方。 +8. **CLOSE-WAIT**：被动关闭连接的一方收到FIN报文后，发送ACK并进入CLOSE-WAIT状态。此时，这一方准备关闭连接，但可能还有未处理的数据。 + - **适用对象**：被动关闭连接的一方。 +9. **CLOSING**：双方几乎同时发送FIN报文。此时双方都在等待对方的ACK。 + - **适用对象**：客户端或服务器。 +10. **LAST-ACK**：被动关闭连接的一方在发送了自己的FIN报文并收到ACK后，进入LAST-ACK状态，等待最后一个ACK的到来。 + - **适用对象**：被动关闭连接的一方。 +11. **TIME-WAIT**：主动关闭连接的一方在发送最后的ACK报文后进入TIME-WAIT状态，等待一段时间，以确保对方收到ACK。如果在此期间未收到对方的任何数据包，连接最终关闭。 + - **适用对象**：主动关闭连接的一方。 +12. **CLOSED**：连接最终关闭，所有资源释放。 + - **适用对象**：客户端和服务器。 -### Get与POST的区别 -GET与POST是我们常用的两种HTTP Method，二者之间的区别主要包括如下五个方面： -1. 从功能上讲，GET一般用来从服务器上获取资源，POST一般用来更新服务器上的资源； -2. 从REST服务角度上说，GET是幂等的，即读取同一个资源，总是得到相同的数据，而POST不是幂等的，因为每次请求对资源的改变并不是相同的；进一步地，GET不会改变服务器上的资源，而POST会对服务器资源进行改变； -3. 从请求参数形式上看，GET请求的数据会附在URL之后，即将请求数据放置在HTTP报文的 请求头 中，以?分割URL和传输数据，参数之间以&相连。特别地，如果数据是英文字母/数字，原样发送；否则，会将其编码为 application/x-www-form-urlencoded MIME 字符串(如果是空格，转换为+，如果是中文/其他字符，则直接把字符串用BASE64加密，得出如：%E4%BD%A0%E5%A5%BD，其中％XX中的XX为该符号以16进制表示的ASCII)；而POST请求会把提交的数据则放置在是HTTP请求报文的 请求体 中。 -4. 就安全性而言，POST的安全性要比GET的安全性高，因为GET请求提交的数据将明文出现在URL上，而且POST请求参数则被包装到请求体中，相对更安全。 -5. 从请求的大小看，GET请求的长度受限于浏览器或服务器对URL长度的限制，允许发送的数据量比较小，而POST请求则是没有大小限制的。 +### 🎯 服务器出现了大量 CLOSE_WAIT 状态如何解决？ +大量 CLOSE_WAIT 表示程序出现了问题，对方的 socket 已经关闭连接，而我方忙于读或写没有及时关闭连接，需要检查代码，特别是释放资源的代码，或者是处理请求的线程配置。 -HTTP请求结构： 请求方式 + 请求URI + 协议及其版本 -HTTP响应结构： 状态码 + 原因短语 + 协议及其版本 +### 🎯 讲一讲SYN超时，洪泛攻击，以及解决策略 -### 状态码 +SYN 超时是指在 TCP 三次握手的过程中，客户端发送了 SYN 包，但由于网络问题或服务器未响应，客户端在等待 SYN-ACK 包的过程中发生了超时。 -每条HTTP响应报文返回时都会携带一个状态码。状态码是一个三位数字的代码，告知客户端请求是否成功，或者是都需要采取其他动作。 +**解决策略** -- 1xx：表明服务端接收了客户端请求，客户端继续发送请求； -- 2xx：客户端发送的请求被服务端成功接收并成功进行了处理； -- 3xx：服务端给客户端返回用于重定向的信息； -- 4xx：客户端的请求有非法内容； -- 5xx：服务端未能正常处理客户端的请求而出现意外错误。 +1. **重试机制**：客户端在发生 SYN 超时时，会重试发送 SYN 包。一般情况下，TCP 协议会尝试重发 SYN 包若干次，直到达到最大重试次数。如果超过最大重试次数仍未收到 SYN-ACK 包，TCP 会放弃连接建立。 +2. **调整超时时间**：根据网络环境调整 SYN 包的超时时间和重试次数，以适应网络的实际情况，确保在合理的时间内重试足够多次。 +3. **网络诊断**：使用网络诊断工具（如 `ping`、`traceroute`、`tcpdump`）检查网络路径是否存在问题，排查网络设备和路由的故障。 +##### 什么是 SYN 洪泛攻击？ +**SYN 洪泛攻击（SYN Flood Attack）** 是一种经典的 **拒绝服务攻击（DoS, Denial of Service）** 或分布式拒绝服务攻击（DDoS）方式。攻击者通过向目标服务器发送大量伪造的 TCP **SYN 请求包**，试图耗尽服务器的资源（如内存、CPU 或连接表），从而导致服务器无法正常处理合法用户的请求。 -- **200 OK**：表示从客户端发送给服务器的请求被正常处理并返回； +**解决策略** -- **204 No Content**：表示客户端发送给客户端的请求得到了成功处理，但在返回的响应报文中不含实体的主体部分（没有资源可以返回） +1. **SYN Cookies**：SYN Cookies 是一种防御 SYN 洪泛攻击的技术。服务器在收到 SYN 包时，不立即分配资源，而是生成一个特殊的序列号（Cookie）并发送给客户端。如果客户端返回正确的 Cookie，服务器才分配资源并建立连接。 +2. **限制半开连接数量**：配置服务器限制半开连接（即已发送 SYN-ACK 但未收到 ACK）的数量。当半开连接数量达到限制时，新的 SYN 请求将被丢弃或延迟处理。 +3. **缩短超时时间**：减小半开连接的超时时间，使得服务器更快地释放未完成的连接资源。 +4. **网络层防护**：使用防火墙和入侵检测系统（IDS）来过滤和阻止可疑的 SYN 包。可以配置防火墙规则来限制每秒钟的 SYN 包数量，或者启用流量分析来检测和防御洪泛攻击。 +5. **负载均衡**：通过部署负载均衡器，将流量分散到多个服务器，从而缓解单个服务器的负载压力，提高抗攻击能力。 +6. **云服务防护**：利用云服务提供商的安全防护措施（如 DDoS 防护服务），通过全球分布的节点和强大的处理能力来抵御大规模的洪泛攻击。 -- **206 Patial Content**：表示客户端进行了范围请求，并且服务器成功执行了这部分的GET请求，响应报文中包含由Content-Range指定范围的实体内容。 -- **301 Moved Permanently**：永久性重定向，表示请求的资源被分配了新的URL，之后应使用更改的URL； -- **302 Found**：临时性重定向，表示请求的资源被分配了新的URL，希望本次访问使用新的URL； +### 🎯 linux 最多可以建立多少个tcp连接，client端，server端，超过了怎么办? -- **303 See Other**：表示请求的资源被分配了新的URL，应使用GET方法定向获取请求的资源 +Linux系统上TCP连接的数量限制主要受以下几个因素影响： -- 304 Not Modified：表示客户端发送附带条件（是指采用GET方法的请求报文中包含if-Match、If-Modified-Since、If-None-Match、If-Range、If-Unmodified-Since中任一首部）的请求时，服务器端允许访问资源，但是请求为满足条件的情况下返回改状态码； +1. **文件描述符限制**：Linux系统中每个进程可以打开的文件描述符数量是有限的。可以通过修改`/etc/security/limits.conf`文件来调整这一限制，例如设置`* soft nofile 65535`和`* hard nofile 65535`来分别设置软限制和硬限制 。 +2. **系统级文件打开限制**：Linux系统级也有一个最大文件打开数量的硬限制，可以通过`/proc/sys/fs/file-max`查看和设置。如果需要，可以通过修改`/etc/rc.local`脚本来调整这个限制 。 +3. **端口范围限制**：TCP连接的端口范围通常是0到65535，其中1024以下的端口通常保留给系统或特殊用途。因此，对于用户应用来说，可以使用的端口数大约为65535 - 1024 = 64511个。 客户端通常在临时端口范围内分配端口，默认范围在32768到60999之间（根据系统不同而不同），因此单个客户端最多能建立约28000个连接。可以通过调整`/proc/sys/net/ipv4/ip_local_port_range`来扩展这个范围。 +4. **TCP连接跟踪限制**：Linux内核的网络防火墙模块会对TCP连接状态进行跟踪，这需要内存资源。可以通过修改`/etc/sysctl.conf`文件中的`net.ipv4.ip_conntrack_max`来调整TCP连接跟踪的最大数量 。 +5. **内核参数**：Linux内核的多个参数影响TCP连接数，例如`net.core.netdev_max_backlog`（网络设备接收队列的最大包数）、`net.ipv4.somaxconn`（监听套接字的 backlog 限制）、`net.ipv4.tcp_max_orphans`（没有文件句柄的TCP套接字的最大数量）等。这些参数可以通过`sysctl`命令或`/etc/sysctl.conf`文件进行调整 。 +6. **编程技术**：使用支持高并发网络I/O的编程技术，如epoll或AIO，可以提高程序对高并发TCP连接的支持 。 +7. **防火墙规则**：防火墙规则可能限制了最大连接数，例如使用iptables的`-A INPUT -p tcp --syn --dport`命令可以限制每个IP地址的连接数 。 +8. **Socket缓冲区大小**：TCP连接的性能也受到Socket缓冲区大小的限制，可以通过`net.ipv4.tcp_rmem`和`net.ipv4.tcp_wmem`进行调整 。 +9. **TIME_WAIT套接字重用**：在高并发场景下，允许重用处于TIME_WAIT状态的套接字可以节省资源，可以通过设置`net.ipv4.tcp_tw_reuse`来实现 。 +10. **系统资源**：最终，系统资源（如内存和CPU）也会影响TCP连接的最大数量。 -- **400 Bad Request**:表示请求报文中存在语法错误； +**超出限制时的处理方法** -- **401 Unauthorized**：经许可，需要通过HTTP认证； +- **客户端**： + - **端口耗尽**：当客户端的端口耗尽时，它将无法再建立新的连接。可以通过扩展临时端口范围或使用多台客户端来分担连接负载。 + - **负载均衡**：使用负载均衡器将连接分发到多个服务器，减少单一服务器的压力。 +- **服务器**： + - **文件描述符限制**：如果服务器的文件描述符限制达到了上限，新的连接请求将被拒绝。可以增加文件描述符限制或分布式处理连接。 + - **内存不足**：内存不足时，可能会导致系统变慢或崩溃。可以增加物理内存，或优化应用程序的内存使用。 + - **连接重用**：通过启用连接重用和TIME_WAIT状态的优化，可以减少对可用端口和连接数量的需求。 + - **缩短TIME_WAIT持续时间**：通过调整`tcp_fin_timeout`和`tcp_tw_reuse`，可以减少TIME_WAIT状态的持续时间和对资源的占用。 -- **403 Forbidden**：服务器拒绝该次访问（访问权限出现问题） -- **404 Not Found**：表示服务器上无法找到请求的资源，除此之外，也可以在服务器拒绝请求但不想给拒绝原因时使用； -- **500 Inter Server Error**：表示服务器在执行请求时发生了错误，也有可能是web应用存在的bug或某些临时的错误时； +### 🎯 TCP 粘包问题 -- **503 Server Unavailable**：表示服务器暂时处于超负载或正在进行停机维护，无法处理请求； +> TCP 是面向字节流的协议，不保证消息边界，所以会出现粘包和拆包。拆包通常发生在数据超过缓冲区或 MSS 时；粘包则是多个小包被合并发送或接收方缓存合并。常见解决办法有三类：**消息长度字段、固定长度、分隔符协议**。实际项目中，像 RPC、消息队列、IM 系统都需要处理粘包，而文件传输、流媒体则不需要。 - +**TCP 粘包/拆包问题** + +**1. 为什么会发生粘包和拆包？** + +TCP 是**面向字节流**的协议，本身不保留消息边界。发送和接收的数据以字节流形式传输，底层可能发生以下情况： + +- **拆包**： + ① 发送方写入的数据 **大于套接字缓冲区**或 **大于 MSS**（最大报文长度），必须拆成多个包发送。 + ② 一个完整的应用层消息被拆分成多个 TCP 报文。 +- **粘包**： + ① 发送方写入的数据 **小于套接字缓冲区大小**，TCP 可能合并多个小包一起发送（Nagle 算法作用）。 + ② **发送太快，接收方来不及处理**，多个小包在接收缓冲区中合并成一个大包。 + +简单理解：**拆包是“消息被切开”，粘包是“消息黏一起”。** + +**2. 常见解决方法** + +1. **消息头 + 长度字段**（最常见） + - 在每条消息前加上消息体长度字段，接收方先读长度，再按长度读完整消息。 + - ✅ 高效、通用，广泛用于 RPC、HTTP/2 等协议。 +2. **固定长度消息** + - 每条消息固定长度，不够补齐，超过则拆分。 + - ❌ 简单但浪费带宽。 +3. **分隔符协议** + - 在消息间加分隔符（如 `\n`、`$`），接收方按分隔符切分。 + - 典型应用：FTP、Redis 协议。 + +**3. 什么时候需要处理？** + +- **需要处理**：应用层消息是离散的，接收端必须严格区分（如 RPC 请求/响应、IM 聊天消息）。 +- **不需要处理**：如果本身消息就是一个连续流（如文件下载、视频流），不存在粘包的语义问题。 +--- + +## 二、应用层协议 + +### 🎯 HTTP/HTTPS详解 + +HTTP 全称是 HyperText Transfer Protocal，即：超文本传输协议。是互联网上应用最为广泛的一种**网络通信协议**，它允许将超文本标记语言（HTML）文档从 Web 服务器传送到客户端的浏览器。目前我们使用的是**HTTP/1.1 版本**。所有的 WWW 文件都必须遵守这个标准。设计 HTTP 最初的目的是为了提供一种发布和接收 HTML 页面的方法。1960 年美国人 Ted Nelson 构思了一种通过计算机处理文本信息的方法，并称之为超文本（hypertext），这成为了HTTP超文本传输协议标准架构的发展根基。 + HTTP 是个应用层协议。HTTP 无需操心网络通信的具体细节，而是把这些细节都交给了通用可靠的因特网传输协议 TCP/IP。 -在 HTTP 客户端向服务器发送报文之前，需要用网络协议（Internet Protocol，IP）地址和端口号在客户端和服务器之间建立一条 TCP/IP 协议。而 IP 地址就是通过 URL 提供的，像 `http://207.200.21.11:80/index.html`，还有使用域名服务（Domain Name Services，DNS）的 `http://www.lazyegg.net`。 +在 HTTP 客户端向服务器发送报文之前，需要用网络协议（Internet Protocol，IP）地址和端口号在客户端和服务器之间建立一条 TCP/IP 协议。而 IP 地址就是通过 URL 提供的，像 `http://207.200.21.11:80/index.html`，还有使用域名服务（Domain Name Services，DNS）的 `http://www.starfish.ink`。 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gdpo1kf1lhj30rq0p8k4t.jpg) +![img](https://ask.qcloudimg.com/http-save/yehe-5359587/vtnvzdv7gt.jpeg) -### 协议版本 +### 🎯 协议版本之间的区别？ - **HTTP/0.9** - HTTP协议的最初版本，功能简陋，仅支持 GET 方法，并且仅能请求访问 HTML 格式的资源 + HTTP协议的最初版本，功能简陋 + + - **特点**： + + - 仅支持 GET 方法。 + + - 仅支持简单的 HTML 文本传输，不支持图片、CSS、JS 等内容。 + + - 无状态，无请求头和响应头。 + + - **用途**：非常基础，主要用于早期的超文本传输。 - **HTTP/1.0** - - 增加了请求方式 POST 和 HEAD - - 不再局限于0.9版本的HTML格式，根据Content-Type可以支持多种数据格式，即MIME多用途互联网邮件扩展，例如text/html、image/jpeg等 - - 同时也开始支持 cache，就是当客户端在规定时间内访问统一网站，直接访问cache即可 - - HTTP请求和回应的格式也变了。除了数据部分，每次通信都必须包括头信息（HTTP header），用来描述一些元数据。其他的新增功能还包括状态码（status code）、多字符集支持、多部分发送（multi-part type）、权限（authorization）、缓存（cache）、内容编码（content encoding）等 - - 但是1.0版本的工作方式是每次TCP连接只能发送一个请求，当服务器响应后就会关闭这次连接，下一个请求需要再次建立TCP连接，就是不支持keepalive + - **新增功能**： + - 支持更多请求方法：GET、POST、HEAD。 + - 支持请求头和响应头，允许传递元信息（如编码、内容类型等）。 + - 支持非 HTML 类型文件传输（如图片、视频）。 + - 每次请求建立一个新的 TCP 连接（无连接复用）。 + + - **缺点**： + - 1.0版本的工作方式是每次TCP连接只能发送一个请求，当服务器响应后就会关闭这次连接，下一个请求需要再次建立TCP连接，就是不支持keepalive。每次请求都需要新建和关闭 TCP 连接，导致性能低下。 - **HTTP/1.0+** - 在20世纪90年代中叶，为满足飞快发展的万维网，很多流行的 Web 客户端和服务器飞快的向 HTTP 中添加各种特性，包括持久的 keep-alive 连接、虚拟主机支持，以及代理连接支持都被假如到 HTTP 中，并称为非官方的事实标准。这种非正式的 HTTP 扩展版本通常称为 HTTP/1.0+ + 在 20 世纪 90 年代中叶，为满足飞快发展的万维网，很多流行的 Web 客户端和服务器飞快的向 HTTP 中添加各种特性，包括持久的 keep-alive 连接、虚拟主机支持，以及代理连接支持都被加入到 HTTP 中，并称为非官方的事实标准。这种非正式的 HTTP 扩展版本通常称为 HTTP/1.0+ - **HTTP/1.1** - - http1.1是目前最为主流的http协议版本，从1997年发布至今，仍是主流的http协议版本。 - - 引入了持久连接，或叫长连接（ persistent connection），即TCP连接默认不关闭，可以被多个请求复用，不用声明Connection: keep-alive。 - - 引入了管道机制（ pipelining），即在同一个TCP连接里，客户端可以同时发送多个请求，进一步改进了HTTP协议的效率。 - - 新增方法：PUT、 PATCH、 OPTIONS、 DELETE。 - - http协议不带有状态，每次请求都必须附上所有信息。请求的很多字段都是重复的，浪费带宽，影响速度。 + http1.1是目前最为主流的http协议版本，从1997年发布至今，仍是主流的http协议版本。 + + - **改进点**： + 1. 引入了持久连接，或叫长连接（ persistent connection），即TCP连接默认不关闭，可以被多个请求复用，不用声明Connection: keep-alive + 2. 分块传输编码（Chunked Encoding）：支持动态内容的传输，服务器可以分块发送数据，不需要知道整体数据长度。 + 3. 增加新方法：增加 PUT、DELETE、OPTIONS 等方法，增强了 RESTful 风格接口支持。 + 4. 缓存机制增强：引入了更复杂的缓存控制（如 `Cache-Control` 和 `ETag`）。 + 5. 引入了管道机制（ pipelining），即在同一个TCP连接里，客户端可以同时发送多个请求，进一步改进了HTTP协议的效率 + + - **缺点**： + - 受限于 TCP 的阻塞问题（即“队头阻塞”）。 - **HTTP/2.0（又名 HTTP-NG）** - - http/2发布于2015年，目前应用还比较少。 - - http/2是一个彻底的二进制协议，头信息和数据体都是二进制，并且统称为"帧"（frame）：头信息帧和数据帧。 - - 复用TCP连接，在一个连接里，客户端和浏览器都可以同时发送多个请求或回应，且不用按顺序一一对应，避免了队头堵塞的问题,此双向的实时通信称为多工（ Multiplexing）。 - - HTTP/2 允许服务器未经请求，主动向客户端发送资源，即服务器推送。 - - 引入头信息压缩机制（ header compression） ,头信息使用gzip或compress压缩后再发送。 + http/2 发布于 2015 年，目前应用还比较少。 + - **改进点**： + 1. 二进制帧传输：http/2是一个彻底的二进制协议，头信息和数据体都是二进制，并且统称为"帧"（frame）：头信息帧和数据帧，提升传输效率和解析速度。 + 2. 多路复用（Multiplexing）：同一个 TCP 连接中可以同时处理多个请求，不会相互阻塞。 + 3. Header 压缩：通过 HPACK 算法对请求头和响应头进行压缩，减少传输大小。 + 4. 服务器推送（Server Push）：服务器可以主动推送资源（如 CSS、JS 文件），减少客户端请求等待时间。 + - **缺点**： + - 仍然依赖 TCP 协议，队头阻塞的问题未完全解决。 +- **HTTP/3** -## 四、HTTPS + - 改进点： + 1. 基于 QUIC 协议：使用 UDP 代替 TCP，解决了队头阻塞问题。 + 2. 连接迁移：支持连接迁移功能（如在网络切换时，无需重新建立连接）。 + 3. 更低的延迟：减少了连接建立时的握手延迟。 -HTTP缺点： + - 优势：更高效的传输性能，适合现代化的应用需求。 -1. 通信使用明文不对数据进行加密（内容容易被窃听） -2. 不验证通信方身份（容易伪装） -3. 无法确定报文完整性（内容易被篡改） -因此，HTTP协议不适合传输一些敏感信息，比如：信用卡号、密码等支付信息。 -为了解决HTTP协议的这一缺陷，需要使用另一种协议：安全套接字层超文本传输协议 HTTPS，为了数据传输的安全，HTTPS在HTTP的基础上加入了SSL（安全套接层）协议，SSL依靠证书来验证服务器的身份，并为浏览器和服务器之间的通信加密。 +### 🎯 HTTP/3 了解吗？ -**与 SSL（安全套接层）组合使用的 HTTP 就是 HTTPS** +**HTTP/2 存在的问题** -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gdrb5imj5uj30ro0aqq6q.jpg) +我们知道，传统 Web 平台的数据传输都基于 TCP 协议，而 TCP 协议在创建连接之前不可避免的需要三次握手，如果需要提高数据交互的安全性，即增加传输层安全协议（TLS），还会增加更多的握手次数。 HTTP 从 1.0 到 2.0，其传输层都是基于 TCP 协议的。即使是带来巨大性能提升的 HTTP/2，也无法完全解决 TCP 协议存在的固有问题（慢启动，拥塞窗口尺寸的设置等）。此外，HTTP/2 多路复用只是减少了连接数，其队头的拥塞问题并没有完全解决，倘若 TCP 丢包率过大，则 HTTP/2 的表现将不如 HTTP/1.1。 +**QUIC 协议** +QUIC（Quick UDP Internet Connections），直译为快速 UDP 网络连接，是谷歌制定的一种基于 UDP 的低延迟传输协议。其主要目的是解决采用传输层 TCP 协议存在的问题，同时满足传输层和应用层对多连接、低延迟等的需求。该协议融合了 TCP, TLS, HTTP/2 等协议的特性，并基于 UDP传输。该协议带来的主要提升有： -![img](https://tva1.sinaimg.cn/large/007S8ZIlly1gds4ejm6cuj30r60cm40n.jpg) +低延迟连接。当客户端第一次连接服务器时，QUIC 只需要 1 RTT（Round-Trid Time）延迟就可以建立安全可靠的连接（采用 TLS 1.3 版本），相比于 TCP + TLS 的 3 次 RTT 要更加快捷。之后，客户端可以在本地缓存加密的认证信息，当再次与服务器建立连接时可以实现 0 RTT 的连接建立延迟。 -### HTTP和HTTPS对比 +QUIC 复用了 HTTP/2 协议的多路复用功能，由于 QUIC 基于 UDP，所以也避免了 HTTP/2存在的队头阻塞问题。 -HTTP协议传输的数据都是未加密的，也就是明文的，因此使用HTTP协议传输隐私信息非常不安全，为了保证这些隐私数据能加密传输，于是网景公司设计了SSL（Secure Sockets Layer）协议用于对HTTP协议传输的数据进行加密，从而就诞生了HTTPS。简单来说，HTTPS协议是由SSL+HTTP协议构建的可进行加密传输、身份认证的网络协议，要比http协议安全。 +基于 UDP 协议的 QUIC 运行在用户域而不是系统内核，这使得 QUIC 协议可以快速的更新和部署，从而很好地解决了 TPC 协议部署及更新的困难。 -HTTPS和HTTP的区别主要如下： +QUIC 的报文是经过加密和认证的，除了少量的报文，其它所有的 QUIC 报文头部都经过了认证，报文主体经过了加密。只要有攻击者篡改 QUIC 报文，接收端都能及时发现。 -1. https协议需要到ca申请证书，一般免费证书较少，因而需要一定费用。 -2. http是超文本传输协议，信息是明文传输，https则是具有安全性的ssl加密传输协议。 -3. http和https使用的是完全不同的连接方式，用的端口也不一样，前者是80，后者是443。 -4. http的连接很简单，是无状态的；HTTPS协议是由SSL+HTTP协议构建的可进行加密传输、身份认证的网络协议，比http协议安全。 +具有向前纠错机制，每个数据包携带了除了本身内容外的部分其他数据包的内容，使得在出现少量丢包的情况下，尽量地减少其它包的重传次数，其通过牺牲单个包所携带的有效数据大小换来更少的重传次数，这在丢包数量较小的场景下能够带来一定程度的性能提升。 +**HTTP/3** +HTTP/3 是在 QUIC 基础上发展起来的，其底层使用 UDP 进行数据传输，上层仍然使用 HTTP/2。在 UDP 与 HTTP/2 之间存在一个 QUIC 层，其中 TLS 加密过程在该层进行处理。HTTP/3 主要有以下几个特点： -### 对称加密与非对称加密 +1. 使用 UDP 作为传输层进行通信； +2. 在 UDP 之上的 QUIC 协议保证了 HTTP/3 的安全性。QUIC 在建立连接的过程中就完成了 TLS 加密握手； +3. 建立连接快，正常只需要 1 RTT 即可建立连接。如果有缓存之前的会话信息，则直接验证和建立连接，此过程 0 RTT。建立连接时，也可以带有少量业务数据； +4. 不和具体底层连接绑定，QUIC 为每个连接的两端分别分配了一个唯一 ID，上层连接只认这对逻辑 ID。网络切换或者断连时，只需要继续发送数据包即可完成连接的建立； +5. 使用 QPACK 进行头部压缩，因为 在 HTTP/2 中的 HPACK 要求传输过程有序，这会导致队头阻塞，而 QPACK 不存在这个问题。 -主要的加密方法分为两种：一种是共享密钥加密（对称密钥加密），一种是公开密钥加密（非对称密钥加密） +最后我们使用一张图来清晰的表示出 HTTP 协议的发展变化： ![img](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/1ef086415a1b455d98f8b4a74daff170~tplv-k3u1fbpfcp-watermark.awebp) -#### 共享密钥加密(对称秘钥加密) -加密与解密使用同一个密钥，常见的对称加密算法：DES，AES，3DES等。 -![img](https://tva1.sinaimg.cn/large/007S8ZIlly1gds4ee4n2nj30pd0fhgoc.jpg) +### 🎯 URI 和 URL 区别？ -也就是说在加密的同时，也会把密钥发送给对方。在发送密钥过程中可能会造成密钥被窃取，那么如何解决这一问题呢？ +1. **URI（统一资源标识符）** -#### 公开密钥（非对称密钥） + - **定义**：URI 是一个更宽泛的概念，用于唯一标识网络中的资源，不强调资源的位置信息。 -公开密钥使用一对非对称密钥。一把叫私有密钥，另一把叫公开密钥。私有密钥不让任何人知道，公有密钥随意发送。公钥加密的信息，只有私钥才能解密。常见的非对称加密算法：RSA，ECC等。 + - **本质**：URI 的核心是 “标识” 资源，就像给资源一个 “身份证号”，只要能唯一区分不同资源即可。 -也就是说，发送密文方使用对方的公开密钥进行加密，对方接受到信息后，使用私有密钥进行解密。 + - 示例： + - `mailto:user@example.com`（标识邮件地址） + - `isbn:1234567890`（标识书籍 ISBN 编号） + - `urn:uuid:123e4567-e89b-12d3-a456-426614174000`（用 URN 格式标识资源） -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gds4ewaqefj30ff0bpjtq.jpg) +2. **URL（统一资源定位符）** + - **定义**：URL 是 URI 的子集，不仅标识资源，还明确指出资源在网络中的具体位置（如通过网络协议、服务器地址、路径等）。 + - **本质**：URL 的核心是 “定位” 资源，相当于给资源一个 “家庭地址”，可直接通过地址访问。 -对称加密加密与解密使用的是同样的密钥，所以速度快，但由于需要将密钥在网络传输，所以安全性不高。 + - 示例： + - `https://www.example.com/index.html`（通过 HTTP 协议访问网页） + - `file:///C:/data/document.pdf`（定位本地文件路径） + - `ftp://ftp.example.com/pub/files`（通过 FTP 协议访问文件） -非对称加密使用了一对密钥，公钥与私钥，所以安全性高，但加密与解密速度慢。 +　 -为了解决这一问题，https采用对称加密与非对称加密的混合加密方式。 +### 🎯 HTTP消息的结构 +**事务和报文** +客户端是怎样通过 HTTP 与 Web 服务器及其资源进行事务处理的呢？一个 **HTTP 事务**由一条请求命令（从客户端发往服务器）和一个响应（从服务器发回客户端）结果组成。这种通信是通过名为**HTTP报文**（HTTP Message）的格式化数据块进行的。 -### SSL/TSL +![img](https://ask.qcloudimg.com/http-save/yehe-5359587/66w65aallf.jpeg) -SSL（Secure Sockets Layer），中文叫做“安全套接层”。它是在上世纪90年代中期，由网景公司设计的。 +HTTP 报文是纯文本，不是二进制代码。从 Web 客户端发往 Web 服务器的 HTTP 报文称为请求报文（request message）。从服务器发往客户端的报文称为响应报文。 -SSL 协议就是用来解决 HTTP 传输过程的不安全问题，到了1999年，SSL 因为应用广泛，已经成为互联网上的事实标准。IETF 就在那年把 SSL 标准化。标准化之后的名称改为 TLS（是“Transport Layer Security”的缩写），中文叫做“传输层安全协议”。 +![img](https://ask.qcloudimg.com/http-save/yehe-5359587/mhguwb92lc.jpeg) -很多相关的文章都把这两者并列称呼（SSL/TLS），因为这两者可以视作同一个东西的不同阶段。 +HTTP 报文包括三部分： -SSL/TLS协议的基本思路是采用[公钥加密法](http://en.wikipedia.org/wiki/Public-key_cryptography)，也就是说，客户端先向服务器端索要公钥，然后用公钥加密信息，服务器收到密文后，用自己的私钥解密。 +- 起始行 +- 首部字段 +- 主体 -但是，这里有两个问题。 +**常见HTTP 首部字段：** -- **如何保证公钥不被篡改？** +**a、通用首部字段**（请求报文与响应报文都会使用的首部字段） -​ 解决方法：将公钥放在数字证书中。只要证书是可信的，公钥就是可信的。 +![img](https://p9-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/f8b770232b8947b6aa7f6cfd90529281~tplv-k3u1fbpfcp-watermark.awebp) -- **公钥加密计算量太大，如何减少耗用的时间？** +**b、请求首部字段**（请求报文会使用的首部字段） - 每一次对话（session），客户端和服务器端都生成一个"对话密钥"（session key），用它来加密信息。由于"对话密钥"是对称加密，所以运算速度非常快，而服务器公钥只用于加密"对话密钥"本身，这样就减少了加密运算的消耗时间。 +![img](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/dae9893ab7ba4b14be5e26c2e90498b7~tplv-k3u1fbpfcp-watermark.awebp) -因此，SSL/TLS协议的基本过程是这样的： +**c、响应首部字段（**响应报文会使用的首部字段） -1. 服务端将非对称加密的公钥发送给客户端； +![img](https://p9-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/3ffb1fa2a46544d0a5da0658d4dddf07~tplv-k3u1fbpfcp-watermark.awebp) -2. 客户端拿着服务端发来的公钥，对对称加密的key做加密并发给服务端； +**d、实体首部字段**（请求报文与响应报文的的实体部分使用的首部字段） -3. 服务端拿着自己的私钥对发来的密文解密，从来获取到对称加密的key； +![img](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/717e4a10362c4efb8dae4bd81a953c27~tplv-k3u1fbpfcp-watermark.awebp) -4. 二者利用对称加密的key对需要传输的消息做加解密传输。 +> HTTP请求结构： 请求方式 + 请求URI + 协议及其版本 +> +> HTTP响应结构： 状态码 + 原因短语 + 协议及其版本 -HTTPS相比HTTP，在请求前多了一个「握手」的环节。 -握手过程中确定了数据加密的密码。在握手过程中，网站会向浏览器发送 SSL 证书，SSL 证书和我们日常用的身份证类似，是一个支持 HTTPS 网站的身份证明，SSL 证书里面包含了网站的域名，证书有效期，证书的颁发机构以及用于加密传输密码的公钥等信息，由于公钥加密的密码只能被在申请证书时生成的私钥解密，因此浏览器在生成密码之前需要先核对当前访问的域名与证书上绑定的域名是否一致，同时还要对证书的颁发机构进行验证，如果验证失败浏览器会给出证书错误的提示。 -### 证书 +### 🎯 Keep-Alive 和非 Keep-Alive 区别，对服务器性能有影响吗 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gds4f2pq0cj30bc0d2dg2.jpg) +在早期的 HTTP/1.0 中，浏览器每次 发起 HTTP 请求都要与服务器创建一个新的 TCP 连接，服务器完成请求处理后立即断开 TCP 连接，服务器不跟踪每个客户也不记录过去的请求。然而创建和关闭连接的过程需要消耗资源和时间，为了减少资源消耗，缩短响应时间，就需要重用连接。在 HTTP/1.1 版本中默认使用持久连接，在此之前的 HTTP 版本的默认连接都是使用非持久连接，如果想要在旧版本的 HTTP 协议上维持持久连接，则需要指定 connection 的首部字段的值为 Keep-Alive 来告诉对方这个请求响应完成后不要关闭，下一次咱们还用这个请求继续交流，我们用一个示意图来更加生动的表示两者的区别： -实际上，我们使用的证书分很多种类型，SSL证书只是其中的一种。证书的格式是由 X.509 标准定义。SSL 证书负责传输公钥，是一种PKI（Public Key Infrastructure，公钥基础结构）证书。 +![img](https://p9-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/10194b7d0e304fd786bebda28fb69718~tplv-k3u1fbpfcp-watermark.awebp) -我们常见的证书根据用途不同大致有以下几种： +对于非 Keep=Alive 来说，必须为每一个请求的对象建立和维护一个全新的连接。对于每一个这样的连接，客户机和服务器都要分配 TCP 的缓冲区和变量，这给服务器带来的严重的负担，因为一台 Web 服务器可能同时服务于数以百计的客户机请求。在 Keep-Alive 方式下，服务器在响应后保持该 TCP 连接打开，在同一个客户机与服务器之间的后续请求和响应报文可通过相同的连接进行传送。甚至位于同一台服务器的多个 Web 页面在从该服务器发送给同一个客户机时，可以在单个持久 TCP 连接上进行。 -1. SSL证书，用于加密HTTP协议，也就是HTTPS。 -2. 代码签名证书，用于签名二进制文件，比如Windows内核驱动，Firefox插件，Java代码签名等等。 -3. 客户端证书，用于加密邮件。 -4. 双因素证书，网银专业版使用的USB Key里面用的就是这种类型的证书。 +然而，Keep-Alive 并不是没有缺点的，当长时间的保持 TCP 连接时容易导致系统资源被无效占用，若对 Keep-Alive 模式配置不当，将有可能比非 Keep-Alive 模式带来的损失更大。因此，我们需要正确地设置 keep-alive timeout 参数，当 TCP 连接在传送完最后一个 HTTP 响应，该连接会保持 keepalive_timeout 秒，之后就开始关闭这个链接。 -这些证书都是由受认证的证书颁发机构——我们称之为CA（Certificate Authority）机构来颁发，针对企业与个人的不同，可申请的证书的类型也不同，价格也不同。CA机构颁发的证书都是受信任的证书，对于 SSL 证书来说，如果访问的网站与证书绑定的网站一致就可以通过浏览器的验证而不会提示错误。 +### 🎯 GET与POST的区别? -**为什么服务端要发送证书给客户端** +> HTTP/1.0 定义了三种请求方法：GET, POST 和 HEAD 方法。 +> +> HTTP/1.1 增加了六种请求方法：OPTIONS, PUT, PATCH, DELETE, TRACE 和 CONNECT 方法。 -互联网有太多的服务需要使用证书来验证身份，以至于客户端(操作系统或浏览器等)无法内置所有证书，需要通过服务端将证书发送给客户端。 +GET与POST是我们常用的两种HTTP Method，二者之间的区别主要包括如下五个方面： -**客户端为什么要验证接收到的证书** +1. 从功能上讲，GET一般用来从服务器上获取资源，POST一般用来更新服务器上的资源； +2. 从REST服务角度上说，GET是幂等的，即读取同一个资源，总是得到相同的数据，而POST不是幂等的，因为每次请求对资源的改变并不是相同的；进一步地，GET不会改变服务器上的资源，而POST会对服务器资源进行改变； +3. 从请求参数形式上看，GET请求的数据会附在URL之后，即将请求数据放置在HTTP报文的请求头中，以 ? 分割URL和传输数据，参数之间以 & 相连。特别地，如果数据是英文字母/数字，原样发送；否则，会将其编码为 application/x-www-form-urlencoded MIME 字符串(如果是空格，转换为+，如果是中文/其他字符，则直接把字符串用BASE64加密，得出如：%E4%BD%A0%E5%A5%BD，其中％XX中的XX为该符号以16进制表示的ASCII)；而POST请求会把提交的数据则放置在是HTTP请求报文的 请求体 中。 +4. 就安全性而言，POST的安全性要比GET的安全性高，因为GET请求提交的数据将明文出现在URL上，而且POST请求参数则被包装到请求体中，相对更安全。 +5. 从请求的大小看，GET请求的长度受限于浏览器或服务器对URL长度的限制，允许发送的数据量比较小，而POST请求则是没有大小限制的。 -中间人攻击 -``` -客户端<------------攻击者<------------服务端 - 伪造证书 拦截请求 -``` -**客户端如何验证接收到的证书** +### 🎯 GET 的长度限制是多少? -为了回答这个问题，需要引入数字签名(Digital Signature)。 +HTTP 中的 GET 方法是通过 URL 传递数据的，而 URL 本身并没有对数据的长度进行限制。 -``` -+---------------------+ -| A digital signature | -|(not to be confused | -|with a digital | -|certificate) | +---------+ +--------+ -| is a mathematical |----哈希--->| 消息摘要 |---私钥加密--->| 数字签名 | -|technique used | +---------+ +--------+ -|to validate the | -|authenticity and | -|integrity of a | -|message, software | -|or digital document. | -+---------------------+ -``` +GET 请求的长度限制主要来自浏览器和服务器的双重约束。主流浏览器限制 URL 在 8KB 左右（IE 仅 2KB），而 Nginx/Apache 等服务器默认限制 8KB。实际开发中建议优先使用 POST 传输大数据，或通过参数拆分解决。当触发 414 错误时需要服务端调优配置。 -将一段文本通过哈希（hash）和私钥加密处理后生成数字签名。 -假设消息传递在Bob，Susan和Pat三人之间发生。Susan将消息连同数字签名一起发送给Bob，Bob接收到消息后，可以这样验证接收到的消息就是Susan发送的 -``` -+---------------------+ -| A digital signature | -|(not to be confused | -|with a digital | -|certificate) | +---------+ -| is a mathematical |----哈希--->| 消息摘要 | -|technique used | +---------+ -|to validate the | | -|authenticity and | | -|integrity of a | | -|message, software | 对 -|or digital document. | 比 -+---------------------+ | - | - | - +--------+ +---------+ - | 数字签名 |---公钥解密--->| 消息摘要 | - +--------+ +---------+ -``` +### 🎯 常见的状态码? -当然，这个前提是Bob知道Susan的公钥。更重要的是，和消息本身一样，公钥不能在不安全的网络中直接发送给Bob。此时就引入了[证书颁发机构](https://en.wikipedia.org/wiki/Certificate_authority)（Certificate Authority，简称CA），CA数量并不多，Bob客户端内置了所有受信任CA的证书。CA对Susan的公钥（和其他信息）数字签名后生成证书。 +> 之前面试被问到过，206 是什么意思、状态码 301 和 302 的区别？ -Susan将证书发送给Bob后，Bob通过CA证书的公钥验证证书签名。 +每条HTTP响应报文返回时都会携带一个状态码。状态码是一个三位数字的代码，告知客户端请求是否成功，或者是都需要采取其他动作。 -Bob信任CA，CA信任Susan 使得 Bob信任Susan，[信任链](https://en.wikipedia.org/wiki/Chain_of_trust)（Chain Of Trust）就是这样形成的。 +> - 1xx：表明服务端接收了客户端请求，客户端继续发送请求； +> - 2xx：客户端发送的请求被服务端成功接收并成功进行了处理； +> - 3xx：服务端给客户端返回用于重定向的信息； +> - 4xx：客户端的请求有非法内容； +> - 5xx：服务端未能正常处理客户端的请求而出现意外错误。 -事实上，Bob客户端内置的是CA的根证书(Root Certificate)，HTTPS协议中服务器会发送证书链（Certificate Chain）给客户端。 +- **200 OK**：表示从客户端发送给服务器的请求被正常处理并返回； +- **204 No Content**：表示客户端发送给客户端的请求得到了成功处理，但在返回的响应报文中不含实体的主体部分（没有资源可以返回） +- **206 Patial Content**：表示客户端进行了范围请求，并且服务器成功执行了这部分的GET请求，响应报文中包含由Content-Range指定范围的实体内容。 -### HTTPS的工作原理 +- **301 Moved Permanently**：永久性重定向，表示请求的资源被分配了新的URL，之后应使用更改的URL； -1. Client 使用https的URL访问 Server，要求与 Server 建立 SSL 连接 -2. Server 把事先配置好的公钥证书返回给客户端。 -3. Client验证公钥证书：比如是否在有效期内，证书的用途是不是匹配Client请求的站点，是不是在CRL吊销列表里面，它的上一级证书是否有效，这是一个递归的过程，直到验证到根证书（操作系统内置的Root证书或者Client内置的Root证书）。如果验证通过则继续，不通过则显示警告信息。 -4. Client使用伪随机数生成器生成加密所使用的对称密钥，然后用证书的公钥加密这个对称密钥，发给Server。 -5. Server使用自己的私钥（private key）解密这个消息，得到对称密钥。至此，Client和Server双方都持有了相同的对称密钥。 -6. Server使用对称密钥加密“明文内容A”，发送给Client。 -7. Client使用对称密钥解密响应的密文，得到“明文内容A”。 -8. Client再次发起HTTPS的请求，使用对称密钥加密请求的“明文内容B”，然后Server使用对称密钥解密密文，得到“明文内容B”。 +- **302 Found**：临时性重定向，表示请求的资源被分配了新的URL，希望本次访问使用新的URL； + +- **303 See Other**：表示请求的资源被分配了新的URL，应使用GET方法定向获取请求的资源 + +- 304 Not Modified：表示客户端发送附带条件（是指采用GET方法的请求报文中包含if-Match、If-Modified-Since、If-None-Match、If-Range、If-Unmodified-Since中任一首部）的请求时，服务器端允许访问资源，但是请求为满足条件的情况下返回改状态码； -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gdqyp5t210j31ey0u0n0y.jpg) +- **400 Bad Request**：表示请求报文中存在语法错误； +- **401 Unauthorized**：未经许可，需要通过 HTTP 认证； +- **403 Forbidden**：服务器拒绝该次访问（访问权限出现问题） -### HTTPS的优点 + > - **身份验证 vs. 授权**：401 表示身份验证问题，即用户未提供凭证或凭证无效。403 表示授权问题，即用户已经提供了凭证，但凭证不足以访问特定资源。 + > - **原因**：401 通常是因为缺少认证信息，而 403 是因为服务器拒绝了用户的请求，即使认证信息是有效的。 -尽管HTTPS并非绝对安全，掌握根证书的机构、掌握加密算法的组织同样可以进行中间人形式的攻击，但HTTPS仍是现行架构下最安全的解决方案，主要有以下几个好处： +- **404 Not Found**：表示服务器上无法找到请求的资源，除此之外，也可以在服务器拒绝请求但不想给拒绝原因时使用； -1. 使用HTTPS协议可认证用户和服务器，确保数据发送到正确的客户机和服务器； -2. HTTPS协议是由SSL+HTTP协议构建的可进行加密传输、身份认证的网络协议，要比http协议安全，可防止数据在传输过程中不被窃取、改变，确保数据的完整性。 -3. HTTPS是现行架构下最安全的解决方案，虽然不是绝对安全，但它大幅增加了中间人攻击的成本。 -4. 谷歌曾在2014年8月份调整搜索引擎算法，并称“比起同等HTTP网站，采用HTTPS加密的网站在搜索结果中的排名将会更高”。 +- **500 Inter Server Error**：表示服务器在执行请求时发生了错误，也有可能是web应用存在的bug或某些临时的错误时； -### HTTPS的缺点 +- **502 Bad Gateway**：服务器作为网关或代理，从上游服务器收到无效响应。 -虽然说HTTPS有很大的优势，但其相对来说，还是存在不足之处的： +- **503 Server Unavailable**：表示服务器暂时处于超负载或正在进行停机维护，无法处理请求； -1. HTTPS协议握手阶段比较费时，会使页面的加载时间延长近50%，增加10%到20%的耗电； -2. HTTPS连接缓存不如HTTP高效，会增加数据开销和功耗，甚至已有的安全措施也会因此而受到影响； -3. SSL证书需要钱，功能越强大的证书费用越高，个人网站、小网站没有必要一般不会用。 -4. SSL证书通常需要绑定IP，不能在同一IP上绑定多个域名，IPv4资源不可能支撑这个消耗。 -5. HTTPS协议的加密范围也比较有限，在黑客攻击、拒绝服务攻击、服务器劫持等方面几乎起不到什么作用。最关键的，SSL证书的信用链体系并不安全，特别是在某些国家可以控制CA根证书的情况下，中间人攻击一样可行。 + +### 🎯 HTTPS 聊一聊 +HTTP 缺点： -### HTTP 切换到 HTTPS +1. 通信使用明文不对数据进行加密（内容容易被窃听） +2. 不验证通信方身份（容易伪装） +3. 无法确定报文完整性（内容易被篡改） -如果需要将网站从http切换到https到底该如何实现呢？ +因此，HTTP 协议不适合传输一些敏感信息，比如：信用卡号、密码等支付信息。 -这里需要将页面中所有的链接，例如js，css，图片等等链接都由http改为https。例如：http://www.baidu.com改为https://www.baidu.com +为了解决 HTTP 协议的这一缺陷，需要使用另一种协议：安全套接字层超文本传输协议 HTTPS，为了数据传输的安全，HTTPS 在 HTTP 的基础上加入了 SSL（安全套接层）协议，SSL 依靠证书来验证服务器的身份，并为浏览器和服务器之间的通信加密。 -BTW，这里虽然将http切换为了https，还是建议保留http。所以我们在切换的时候可以做http和https的兼容，具体实现方式是，去掉页面链接中的http头部，这样可以自动匹配http头和https头。例如：将http://www.baidu.com改为//www.baidu.com。然后当用户从http的入口进入访问页面时，页面就是http，如果用户是从https的入口进入访问页面，页面即使https的。 +**与 SSL（安全套接层）组合使用的 HTTP 就是 HTTPS** +![img](https://ask.qcloudimg.com/http-save/yehe-5359587/1qyntrrxsm.jpeg) +![img](https://ask.qcloudimg.com/http-save/yehe-5359587/0tmozbz23f.jpeg) -### 什么是Cookie，Cookie的使用过程是怎么样的？ +### 🎯 HTTP 与 HTTPs 的工作方式【建立连接的过程】 -由于 http 协议是无状态协议，如果客户通过浏览器访问 web 应用时没有一个保存用户访问状态的机制，那么将不能持续跟踪应用的操作。比如当用户往购物车中添加了商品，web 应用必须在用户浏览别的商品的时候仍保存购物车的状态，以便用户继续往购物车中添加商品。 +**HTTP** -cookie 是浏览器的一种缓存机制，它可用于维持客户端与服务器端之间的会话。由于下面一题会讲到session，所以这里要强调cookie会将会话保存在客户端（session则是把会话保存在服务端） +HTTP（Hyper Text Transfer Protocol: 超文本传输协议） 是一种简单的请求 - 响应协议，被用于在 Web 浏览器和网站服务器之间传递消息。HTTP 使用 TCP（而不是 UDP）作为它的支撑运输层协议。其默认工作在 TCP 协议 80 端口，HTTP 客户机发起一个与服务器的 TCP 连接，一旦连接建立，浏览器和服务器进程就可以通过套接字接口访问 TCP。客户机从套接字接口发送 HTTP 请求报文和接收 HTTP 响应报文。类似地，服务器也是从套接字接口接收 HTTP 请求报文和发送 HTTP 响应报文。其通信内容以明文的方式发送，不通过任何方式的数据加密。当通信结束时，客户端与服务器关闭连接。 -这里以最常见的登陆案例讲解cookie的使用过程： +**HTTPS** -1. 首先用户在客户端浏览器向服务器发起登陆请求 -2. 登陆成功后，服务端会把登陆的用户信息设置 cookie 中，返回给客户端浏览器 -3. 客户端浏览器接收到 cookie 请求后，会把 cookie 保存到本地（可能是内存，也可能是磁盘，看具体使用情况而定） -4. 以后再次访问该 web 应用时，客户端浏览器就会把本地的 cookie 带上，这样服务端就能根据 cookie 获得用户信息了 +HTTPS（Hyper Text Transfer Protocol over Secure Socket Layer）是以安全为目标的 HTTP 协议，在 HTTP 的基础上通过传输加密和身份认证的方式保证了传输过程的安全性。其工作流程如下： +1. Client 使用 https 的 URL 访问 Server，要求与 Server 建立 SSL 连接 +2. Server 把事先配置好的公钥证书返回给客户端。 +3. Client 验证公钥证书：比如是否在有效期内，证书的用途是不是匹配 Client 请求的站点，是不是在 CRL 吊销列表里面，它的上一级证书是否有效，这是一个递归的过程，直到验证到根证书（操作系统内置的Root证书或者Client内置的Root证书）。如果验证通过则继续，不通过则显示警告信息。 +4. Client 使用伪随机数生成器生成加密所使用的对称密钥，然后用证书的公钥加密这个对称密钥，发给Server。 +5. Server使用自己的私钥（private key）解密这个消息，得到对称密钥。至此，Client和Server双方都持有了相同的对称密钥。 +6. Server使用对称密钥加密“明文内容A”，发送给Client。 +7. Client使用对称密钥解密响应的密文，得到“明文内容A”。 +8. Client再次发起HTTPS的请求，使用对称密钥加密请求的“明文内容B”，然后Server使用对称密钥解密密文，得到“明文内容B”。 +![img](https://ask.qcloudimg.com/http-save/yehe-5359587/qvh9qg13jv.jpeg) -### 什么是session，有哪些实现session的机制？ +### 🎯 HTTP 和 HTTPS 的区别？ -session 是一种维持客户端与服务器端会话的机制。但是与 **cookie 把会话信息保存在客户端本地不一样，session 把会话保留在浏览器端。** +HTTP 协议传输的数据都是未加密的，也就是明文的，因此使用 HTTP 协议传输隐私信息非常不安全，为了保证这些隐私数据能加密传输，于是网景公司设计了 SSL（Secure Sockets Layer）协议用于对 HTTP 协议传输的数据进行加密，从而就诞生了 HTTPS。简单来说，HTTPS 协议是由 SSL+HTTP 协议构建的可进行加密传输、身份认证的网络协议，要比 HTTP 协议安全。 -我们同样以登陆案例为例子讲解 session 的使用过程： +1. HTTP 协议以明文方式发送内容，数据都是未加密的，安全性较差。HTTPS 数据传输过程是加密的，安全性较好。 +2. HTTP 和 HTTPS 使用的是完全不同的连接方式，用的端口也不一样，前者是 80 端口，后者是 443 端口。 +3. HTTPS 协议需要到数字认证机构（Certificate Authority, CA）申请证书，一般需要一定的费用。 +4. HTTP 页面响应比 HTTPS 快，主要因为 HTTP 使用 3 次握手建立连接，客户端和服务器需要握手 3 次，而 HTTPS 除了 TCP 的 3 次握手，还需要经历一个 SSL 协商过程。 -1. 首先用户在客户端浏览器发起登陆请求 -2. 登陆成功后，服务端会把用户信息保存在服务端，并返回一个唯一的 session 标识给客户端浏览器。 -3. 客户端浏览器会把这个唯一的 session 标识保存在起来 -4. 以后再次访问 web 应用时，客户端浏览器会把这个唯一的 session 标识带上，这样服务端就能根据这个唯一标识找到用户信息。 -看到这里可能会引起疑问：把唯一的 session 标识返回给客户端浏览器，然后保存起来，以后访问时带上，这难道不是 cookie 吗？ -没错，s**ession 只是一种会话机制，在许多 web 应用中，session 机制就是通过 cookie 来实现的**。也就是说它只是使用了 cookie 的功能，并不是使用 cookie 完成会话保存。与 cookie 在保存客户端保存会话的机制相反，session 通过 cookie 的功能把会话信息保存到了服务端。 +### 🎯 说一下对称加密与非对称加密？ -进一步地说，session 是一种维持服务端与客户端之间会话的机制，它可以有不同的实现。以现在比较流行的小程序为例，阐述一个 session 的实现方案： +主要的加密方法分为两种：一种是共享密钥加密（对称密钥加密），一种是公开密钥加密（非对称密钥加密） -1. 首先用户登陆后，需要把用户登陆信息保存在服务端，这里我们可以采用 redis。比如说给用户生成一个 userToken，然后以 userId 作为键，以 userToken 作为值保存到 redis 中，并在返回时把 userToken 带回给小程序端。 -2. 小程序端接收到 userToken 后把它缓存起来，以后每当访问后端服务时就把 userToken 带上。 -3. 在后续的服务中服务端只要拿着小程序端带来的 userToken 和 redis 中的 userToken 进行比对，就能确定用户的登陆状态了。 +**共享密钥加密(对称秘钥加密)** +加密与解密使用同一个密钥，常见的对称加密算法：DES，AES，3DES等。 +![img](https://ask.qcloudimg.com/http-save/yehe-5359587/31ra254n5d.jpeg) -### session和cookie有什么区别 +也就是说在加密的同时，也会把密钥发送给对方。在发送密钥过程中可能会造成密钥被窃取，那么如何解决这一问题呢？ -经过上面两道题的阐述，这道题就很清晰了 +**公开密钥（非对称密钥）** -1. cookie 是浏览器提供的一种缓存机制，它可以用于维持客户端与服务端之间的会话 -2. session 指的是维持客户端与服务端会话的一种机制，它可以通过 cookie 实现，也可以通过别的手段实现。 -3. 如果用 cookie 实现会话，那么会话会保存在客户端浏览器中 -4. 而 session 机制提供的会话是保存在服务端的。 +公开密钥使用一对非对称密钥。一把叫私有密钥，另一把叫公开密钥。私有密钥不让任何人知道，公有密钥随意发送。公钥加密的信息，只有私钥才能解密。常见的非对称加密算法：RSA，ECC等。 +也就是说，发送密文方使用对方的公开密钥进行加密，对方接收到信息后，使用私有密钥进行解密。 +![img](https://ask.qcloudimg.com/http-save/yehe-5359587/ngbjeh0t39.jpeg) -## Other FAQ　　　　　 -### 从输入网址到获得页面的过程 -1. 浏览器查询 DNS，获取域名对应的IP地址:具体过程包括浏览器搜索自身的DNS缓存、搜索操作系统的DNS缓存、读取本地的Host文件和向本地DNS服务器进行查询等。对于向本地DNS服务器进行查询，如果要查询的域名包含在本地配置区域资源中，则返回解析结果给客户机，完成域名解析(此解析具有权威性)；如果要查询的域名不由本地DNS服务器区域解析，但该服务器已缓存了此网址映射关系，则调用这个IP地址映射，完成域名解析（此解析不具有权威性）。如果本地域名服务器并未缓存该网址映射关系，那么将根据其设置发起递归查询或者迭代查询； -2. 浏览器获得域名对应的IP地址以后，浏览器向服务器请求建立链接，发起三次握手； -3. TCP/IP链接建立起来后，浏览器向服务器发送HTTP请求； -4. 服务器接收到这个请求，并根据路径参数映射到特定的请求处理器进行处理，并将处理结果及相应的视图返回给浏览器； -5. 浏览器解析并渲染视图，若遇到对js文件、css文件及图片等静态资源的引用，则重复上述步骤并向服务器请求这些资源； -6. 浏览器根据其请求到的资源、数据渲染页面，最终向用户呈现一个完整的页面。 +对称加密加密与解密使用的是同样的密钥，所以速度快，但由于需要将密钥在网络传输，所以安全性不高。 -**简单版** +非对称加密使用了一对密钥，公钥与私钥，所以安全性高，但加密与解密速度慢。 -1. DNS解析 -2. TCP连接 -3. 发送HTTP请求 -4. 服务器处理请求并返回HTTP报文 -5. 浏览器解析渲染页面 -6. 连接结束 +为了解决这一问题，https 采用对称加密与非对称加密的混合加密方式。 -### XSS 攻击 -XSS 是一种经常出现在web应用中的计算机安全漏洞，与SQL注入一起成为web中最主流的攻击方式。XSS是指恶意攻击者利用网站没有对用户提交数据进行转义处理或者过滤不足的缺点，进而添加一些脚本代码嵌入到web页面中去，使别的用户访问都会执行相应的嵌入代码，从而盗取用户资料、利用用户身份进行某种动作或者对访问者进行病毒侵害的一种攻击方式。 -　　　　　　　　　　 +**非对称加密有哪些缺点？** -### IP地址的分类 +非对称加密在某些方面存在一些缺点，主要包括： -IP地址是指互联网协议地址，是IP协议提供的一种统一的地址格式，它为互联网上的每一个网络和每一台主机分配一个逻辑地址，以此来屏蔽物理地址的差异。IP地址编址方案将IP地址空间划分为A、B、C、D、E五类，其中A、B、C是基本类，D、E类作为多播和保留使用，为特殊地址。 +1. **性能开销**：非对称加密通常比对称加密慢，因为它涉及更复杂的数学运算，这使得它在处理大量数据时效率较低。 +2. **加密速度**：非对称加密的加密和解密速度通常较慢，这限制了它在需要快速处理的场景中的应用。 +3. **密钥管理**：虽然非对称加密的公钥可以公开，但私钥必须严格保密，这增加了密钥管理的复杂性。 +4. **密钥长度**：为了确保安全性，非对称加密通常需要较长的密钥长度，这可能导致存储和传输的开销增加。 +5. **资源消耗**：非对称加密算法在执行时可能消耗更多的计算资源，这在资源受限的环境中可能是一个问题。 +6. **数字签名**：非对称加密常用于数字签名，但如果私钥被泄露，可能会对系统的安全性造成威胁。 +7. **算法限制**：某些非对称加密算法可能受到特定的算法限制，例如RSA算法受到素数生成技术的限制。 +8. **适用场景限制**：非对称加密通常不适用于需要快速加解密的场景，如实时通信或大量数据的加密存储。 -每个IP地址包括两个标识码（ID），即网络ID和主机ID。同一个物理网络上的所有主机都使用同一个网络ID，网络上的一个主机（包括网络上工作站，服务器和路由器等）有一个主机ID与其对应。A~E类地址的特点如下： -A类地址：以0开头，第一个字节范围：0~127； -B类地址：以10开头，第一个字节范围：128~191； +### 🎯 HTTPS加密过程概述？ -C类地址：以110开头，第一个字节范围：192~223； +最开始还是TCP三次握手，之后，HTTPS需要进行TLS握手以建立安全连接 -D类地址：以1110开头，第一个字节范围为224~239； +1. **客户端发起HTTPS请求**： + - 用户在浏览器中输入网址（如`https://www.example.com`），浏览器会向服务器发起HTTPS请求。 +2. **服务器响应并发送证书**： + - 服务器接收到客户端请求后，向客户端发送服务器的数字证书。这个证书由受信任的证书颁发机构（CA）签发，包含服务器的公钥和其他信息。 +3. **客户端验证证书**： + - 客户端（浏览器）接收到服务器的数字证书后，会验证证书的有效性，检查证书是否由受信任的CA签发，证书是否在有效期内，证书的域名是否与访问的域名匹配。如果证书验证通过，客户端将继续加密过程；否则，客户端会显示警告，提示用户证书无效。 +4. **生成会话密钥**： + - 客户端在验证证书通过后，会生成一个随机的会话密钥（对称密钥），用于加密后续的通信数据。因为对称加密速度快且效率高，所以HTTPS在数据传输阶段使用对称加密。 +5. **加密会话密钥并传输**： + - 客户端使用服务器的公钥（从证书中获取）加密生成的会话密钥，然后将加密后的会话密钥发送给服务器。由于非对称加密的特性，只有服务器能够使用其私钥解密这个会话密钥。 +6. **服务器解密会话密钥**： + - 服务器使用自己的私钥解密客户端传来的加密会话密钥，得到会话密钥。至此，客户端和服务器都拥有了相同的会话密钥，可以用它来加密和解密后续的通信数据。 +7. **使用对称加密进行数据传输**： + - 在整个会话期间，客户端和服务器都使用这个对称会话密钥对数据进行加密和解密。数据在传输过程中即使被拦截，由于使用了对称加密，攻击者无法解密数据。 +8. **会话结束**： + - 当通信结束时，客户端和服务器都会丢弃会话密钥。如果需要再进行通信，会重新启动一个新的加密过程。 -E类地址：以1111开头，保留地址 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gds4m1xeinj30ee0apdkh.jpg) - +### 🎯 SSL/TSL + +SSL（Secure Sockets Layer），中文叫做“安全套接层”。它是在上世纪90年代中期，由网景公司设计的。 + +SSL 协议就是用来解决 HTTP 传输过程的不安全问题，到了1999年，SSL 因为应用广泛，已经成为互联网上的事实标准。IETF 就在那年把 SSL 标准化。标准化之后的名称改为 TLS（是“Transport Layer Security”的缩写），中文叫做“传输层安全协议”。 + +很多相关的文章都把这两者并列称呼（SSL/TLS），因为这两者可以视作同一个东西的不同阶段。 + +SSL/TLS协议的基本思路是采用[公钥加密法](http://en.wikipedia.org/wiki/Public-key_cryptography)，也就是说，客户端先向服务器端索要公钥，然后用公钥加密信息，服务器收到密文后，用自己的私钥解密。 + +但是，这里有两个问题。 + +- **如何保证公钥不被篡改？** + +​ 解决方法：将公钥放在数字证书中。只要证书是可信的，公钥就是可信的。 + +- **公钥加密计算量太大，如何减少耗用的时间？** + + 每一次对话（session），客户端和服务器端都生成一个"对话密钥"（session key），用它来加密信息。由于"对话密钥"是对称加密，所以运算速度非常快，而服务器公钥只用于加密"对话密钥"本身，这样就减少了加密运算的消耗时间。 + +因此，SSL/TLS协议的基本过程是这样的： + +1. 服务端将非对称加密的公钥发送给客户端； + +2. 客户端拿着服务端发来的公钥，对对称加密的key做加密并发给服务端； + +3. 服务端拿着自己的私钥对发来的密文解密，从而获取到对称加密的key； + +4. 二者利用对称加密的key对需要传输的消息做加解密传输。 + + + +HTTPS 相比 HTTP，在请求前多了一个「握手」的环节。 + +握手过程中确定了数据加密的密码。在握手过程中，网站会向浏览器发送 SSL 证书，SSL 证书和我们日常用的身份证类似，是一个支持 HTTPS 网站的身份证明，SSL 证书里面包含了网站的域名，证书有效期，证书的颁发机构以及用于加密传输密码的公钥等信息，由于公钥加密的密码只能被在申请证书时生成的私钥解密，因此浏览器在生成密码之前需要先核对当前访问的域名与证书上绑定的域名是否一致，同时还要对证书的颁发机构进行验证，如果验证失败浏览器会给出证书错误的提示。 + +### 🎯 证书 + +![img](https://ask.qcloudimg.com/http-save/yehe-5359587/ndvlxlzgic.jpeg) + +实际上，我们使用的证书分很多种类型，SSL证书只是其中的一种。证书的格式是由 X.509 标准定义。SSL 证书负责传输公钥，是一种PKI（Public Key Infrastructure，公钥基础结构）证书。 + +我们常见的证书根据用途不同大致有以下几种： + +1. SSL证书，用于加密HTTP协议，也就是HTTPS。 +2. 代码签名证书，用于签名二进制文件，比如Windows内核驱动，Firefox插件，Java代码签名等等。 +3. 客户端证书，用于加密邮件。 +4. 双因素证书，网银专业版使用的USB Key里面用的就是这种类型的证书。 + +这些证书都是由受认证的证书颁发机构——我们称之为CA（Certificate Authority）机构来颁发，针对企业与个人的不同，可申请的证书的类型也不同，价格也不同。CA机构颁发的证书都是受信任的证书，对于 SSL 证书来说，如果访问的网站与证书绑定的网站一致就可以通过浏览器的验证而不会提示错误。 + + + +### 🎯 客户端为什么信任第三方证书 + +假设中间人篡改了证书原文，由于他没有 CA 机构的私钥，所以无法得到此时加密后的签名，因此无法篡改签名。客户端浏览器收到该证书后会发现原文和签名解密后的值不一致，则说明证书被中间人篡改，证书不可信，从而终止向服务器传输信息。 + +上述过程说明证书无法被篡改，我们考虑更严重的情况，例如中间人拿到了 CA 机构认证的证书，它想窃取网站 A 发送给客户端的信息，于是它成为中间人拦截到了 A 传给客户端的证书，然后将其替换为自己的证书。此时客户端浏览器收到的是被中间人掉包后的证书，但由于证书里包含了客户端请求的网站信息，因此客户端浏览器只需要把证书里的域名与自己请求的域名比对一下就知道证书有没有被掉包了。 + + + +### 🎯 HTTPS的优缺点 + +尽管 HTTPS 并非绝对安全，掌握根证书的机构、掌握加密算法的组织同样可以进行中间人形式的攻击，但 HTTPS仍是现行架构下最安全的解决方案，主要有以下几个好处： + +1. 使用HTTPS协议可认证用户和服务器，确保数据发送到正确的客户机和服务器； +2. HTTPS协议是由SSL+HTTP协议构建的可进行加密传输、身份认证的网络协议，要比http协议安全，可防止数据在传输过程中不被窃取、改变，确保数据的完整性。 +3. HTTPS是现行架构下最安全的解决方案，虽然不是绝对安全，但它大幅增加了中间人攻击的成本。 +4. 谷歌曾在2014年8月份调整搜索引擎算法，并称“比起同等HTTP网站，采用HTTPS加密的网站在搜索结果中的排名将会更高”。 + +**HTTPS的缺点** + +虽然说HTTPS有很大的优势，但其相对来说，还是存在不足之处的： + +1. HTTPS协议握手阶段比较费时，会使页面的加载时间延长近50%，增加10%到20%的耗电； +2. HTTPS连接缓存不如HTTP高效，会增加数据开销和功耗，甚至已有的安全措施也会因此而受到影响； +3. SSL证书需要钱，功能越强大的证书费用越高，个人网站、小网站没有必要一般不会用。 +4. SSL证书通常需要绑定IP，不能在同一IP上绑定多个域名，IPv4资源不可能支撑这个消耗。 +5. HTTPS协议的加密范围也比较有限，在黑客攻击、拒绝服务攻击、服务器劫持等方面几乎起不到什么作用。最关键的，SSL证书的信用链体系并不安全，特别是在某些国家可以控制CA根证书的情况下，中间人攻击一样可行。 + +> **标准回答**： “HTTPS 的核心优势是提供加密传输、身份认证和数据完整性，防止中间人攻击；主要缺点是性能开销和部署复杂度。现代优化手段如 TLS 1.3 和 CDN 加速已大幅降低损耗，对于涉及用户数据的场景，HTTPS 是必备的安全实践。” + +> **进阶回答**： “HTTPS 本质是通过 TLS 在 TCP 与应用层之间插入安全层。性能损耗主要来自握手阶段的非对称加密（RSA/ECC）和证书链验证。我们通过 OSCP Stapling 减少 300ms 验证延迟，配合 QUIC 协议实现 0-RTT 握手，实践中性能差距已控制在 5% 以内。安全收益远超成本，尤其对抗公共 WiFi 嗅探等威胁。” + + + +### 🎯 HTTP 切换到 HTTPS + +1. 通过获取 SSL/TLS 证书 +2. 配置 Web 服务器：不同的 Web 服务器（如 Nginx、Apache）配置方式不同 +3. 重定向 HTTP 到 HTTPS、 +4. 更新应用配置和链接 + - 更新应用配置：如果你的应用中有硬编码的 HTTP 链接，确保将它们更新为 HTTPS + - 更新静态资源链接：确保所有静态资源（如图片、CSS、JS 文件）的链接使用 HTTPS +5. 配置 HSTS：HSTS 强制客户端（如浏览器）使用 HTTPS 访问你的站点，进一步提高安全性 +6. 进行测试和验证 + +--- + +## 三、网络编程实战 + +### 🎯 从输入网址到获得页面的过程 + +1. 浏览器查询 DNS，获取域名对应的 IP 地址：具体过程包括浏览器搜索自身的DNS缓存、搜索操作系统的DNS缓存、读取本地的Host文件和向本地DNS服务器进行查询等。对于向本地DNS服务器进行查询，如果要查询的域名包含在本地配置区域资源中，则返回解析结果给客户机，完成域名解析(此解析具有权威性)；如果要查询的域名不由本地DNS服务器区域解析，但该服务器已缓存了此网址映射关系，则调用这个IP地址映射，完成域名解析（此解析不具有权威性）。如果本地域名服务器并未缓存该网址映射关系，那么将根据其设置发起递归查询或者迭代查询； +2. 浏览器获得域名对应的 IP 地址以后，浏览器向服务器请求建立链接，发起三次握手； +3. TCP/IP 链接建立起来后，浏览器向服务器发送 HTTP 请求； +4. 服务器接收到这个请求，并根据路径参数映射到特定的请求处理器进行处理，并将处理结果及相应的视图返回给浏览器； +5. 浏览器解析并渲染视图，若遇到对js文件、css文件及图片等静态资源的引用，则重复上述步骤并向服务器请求这些资源； +6. 浏览器根据其请求到的资源、数据渲染页面，最终向用户呈现一个完整的页面。 + +**简单版** + +1. DNS解析 +2. TCP连接 +3. 发送HTTP请求 +4. 服务器处理请求并返回HTTP报文 +5. 浏览器解析渲染页面 +6. 连接结束 + +![img](https://p6-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/dadeadd36a41416bb7f89026d68b6a77~tplv-k3u1fbpfcp-watermark.awebp) + +### 🎯 socket() 套接字有哪些 + +套接字（Socket）是对网络中不同主机上的应用进程之间进行双向通信的端点的抽象，网络进程通信的一端就是一个套接字，不同主机上的进程便是通过套接字发送报文来进行通信。例如 TCP 用主机的 IP 地址 + 端口号作为 TCP 连接的端点，这个端点就叫做套接字。 + +套接字主要有以下三种类型： + +1. 流套接字（SOCK_STREAM）：流套接字基于 TCP 传输协议，主要用于提供面向连接、可靠的数据传输服务。由于 TCP 协议的特点，使用流套接字进行通信时能够保证数据无差错、无重复传送，并按顺序接收，通信双方不需要在程序中进行相应的处理。 +2. 数据报套接字（SOCK_DGRAM）：和流套接字不同，数据报套接字基于 UDP 传输协议，对应于无连接的 UDP 服务应用。该服务并不能保证数据传输的可靠性，也无法保证对端能够顺序接收到数据。此外，通信两端不需建立长时间的连接关系，当 UDP 客户端发送一个数据给服务器后，其可以通过同一个套接字给另一个服务器发送数据。当用 UDP 套接字时，丢包等问题需要在程序中进行处理。 +3. 原始套接字（SOCK_RAW）：由于流套接字和数据报套接字只能读取 TCP 和 UDP 协议的数据，当需要传送非传输层数据包（例如 Ping 命令时用的 ICMP 协议数据包）或者遇到操作系统无法处理的数据包时，此时就需要建立原始套接字来发送。 + +--- + + + +## 四、网络层协议 + +### 🎯 IP 协议的定义和作用 + +IP 协议（Internet Protocol）又称互联网协议，是支持网间互联的数据包协议。该协议工作在网络层，主要目的就是为了提高网络的可扩展性，和传输层 TCP 相比，IP 协议提供一种无连接/不可靠、尽力而为的数据包传输服务，其与TCP协议（传输控制协议）一起构成了TCP/IP 协议族的核心。IP 协议主要有以下几个作用： + +- 寻址和路由：在IP 数据包中会携带源 IP 地址和目的 IP 地址来标识该数据包的源主机和目的主机。IP 数据报在传输过程中，每个中间节点（IP 网关、路由器）只根据网络地址进行转发，如果中间节点是路由器，则路由器会根据路由表选择合适的路径。IP 协议根据路由选择协议提供的路由信息对 IP 数据报进行转发，直至抵达目的主机。 +- 分段与重组：IP 数据包在传输过程中可能会经过不同的网络，在不同的网络中数据包的最大长度限制是不同的，IP 协议通过给每个 IP 数据包分配一个标识符以及分段与组装的相关信息，使得数据包在不同的网络中能够传输，被分段后的 IP 数据报可以独立地在网络中进行转发，在到达目的主机后由目的主机完成重组工作，恢复出原来的 IP 数据包。 + +### 🎯 域名和 IP 的关系，一个 IP 可以对应多个域名吗 + +IP 在同一个网络中是唯一的，用来标识每一个网络上的设备，其相当于一个人的身份证号；域名在同一个网络中也是唯一的，就像一个人的名字，绰号。假如你有多个不同的绰号，你的朋友可以用其中任何一个绰号叫你，但你的身份证号码却是唯一的。由此我们可以看出一个域名只能对应一个 IP 地址，是一对一的关系；而一个 IP 却可以对应多个域名，是一对多的关系。 + +> 一个域名可以对应多个IP地址，这种能力主要得益于DNS（域名系统）的工作原理 +> +> 在DNS记录中，A记录（Address Record）用于将域名映射到一个IPv4地址 +> +> 一个域名可以有多个A记录或AAAA记录，这意味着它可以对应多个IPv4或IPv6地址 + +### 🎯 IPV4 地址不够如何解决 + +- DHCP：动态主机配置协议。动态分配 IP 地址，只给接入网络的设备分配IP地址，因此同一个 MAC 地址的设备，每次接入互联网时，得到的IP地址不一定是相同的，该协议使得空闲的 IP 地址可以得到充分利用。 +- CIDR：无类别域间路由。CIDR 消除了传统的 A 类、B 类、C 类地址以及划分子网的概念，因而更加有效的分配 IPv4 的地址空间，但无法从根本上解决地址耗尽问题。 +- NAT：网络地址转换协议。我们知道属于不同局域网的主机可以使用相同的 IP 地址，从而一定程度上缓解了 IP 资源枯竭的问题。然而主机在局域网中使用的 IP 地址是不能在公网中使用的，当局域网主机想要与公网进行通信时， NAT 方法可以将该主机 IP 地址转换成全球 IP 地址。该协议能够有效解决 IP 地址不足的问题。 +- IPv6 ：作为接替 IPv4 的下一代互联网协议，其可以实现 2 的 128 次方个地址，而这个数量级，即使是给地球上每一颗沙子都分配一个IP地址，该协议能够从根本上解决 IPv4 地址不够用的问题。 + +> ##### IPv4 和 IPv6 的区别？ +> +> **地址长度** +> +> - IPv4：地址长度为 32 位（二进制位），通常用点分十进制表示，如：`192.168.1.1`。 +> - 地址数量：大约 **42 亿个**（2322^{32}232）。 +> - IPv6：地址长度为 128 位，通常用冒分十六进制表示，如：`2001:0db8:85a3:0000:0000:8a2e:0370:7334`。 +> - 地址数量：大约 21282^{128}2128，理论上几乎是无限的（约 3.4×10383.4 \times 10^{38}3.4×1038 个地址）。 +> +> **地址表示** +> +> - **IPv4**：点分十进制表示，分为 4 个八位字节（4 个字节），例如：`192.168.1.1`。 +> - **IPv6**：冒分十六进制表示，由 8 组 16 位（每组用冒号分隔），例如：`2001:0db8:85a3:0000:0000:8a2e:0370:7334`。 + +### 🎯 路由器的分组转发流程 + +① 从 IP 数据包中提取出目的主机的 IP 地址，找到其所在的网络； + +② 判断目的 IP 地址所在的网络是否与本路由器直接相连，如果是，则不需要经过其它路由器直接交付，否则执行 ③； + +③ 检查路由表中是否有目的 IP 地址的特定主机路由。如果有，则按照路由表传送到下一跳路由器中，否则执行 ④； + +④ 逐条检查路由表，若找到匹配路由，则按照路由表转发到下一跳路由器中，否则执行步骤 ⑤； + +⑤ 若路由表中设置有默认路由，则按照默认路由转发到默认路由器中，否则执行步骤 ⑥； + +⑥ 无法找到合适路由，向源主机报错。 + +### 🎯 路由器和交换机的区别 + +- 交换机：交换机用于局域网，利用主机的物理地址（MAC 地址）确定数据转发的目的地址，它工作与数据链路层。 +- 路由器：路由器通过数据包中的目的 IP 地址识别不同的网络从而确定数据转发的目的地址，网络号是唯一的。路由器根据路由选择协议和路由表信息从而确定数据的转发路径，直到到达目的网络，它工作于网络层。 + +### 🎯 ICMP 协议概念/作用 + +ICMP（Internet Control Message Protocol）是因特网控制报文协议，主要是实现 IP 协议中未实现的部分功能，是一种网络层协议。该协议并不传输数据，只传输控制信息来辅助网络层通信。其主要的功能是验证网络是否畅通（确认接收方是否成功接收到 IP 数据包）以及辅助 IP 协议实现可靠传输（若发生 IP 丢包，ICMP 会通知发送方 IP 数据包被丢弃的原因，之后发送方会进行相应的处理）。 + +### 🎯 ICMP 的应用 + +- Ping + +Ping（Packet Internet Groper），即因特网包探测器，是一种工作在网络层的服务命令，主要用于测试网络连接量。本地主机通过向目的主机发送 ICMP Echo 请求报文，目的主机收到之后会发送 Echo 响应报文，Ping 会根据时间和成功响应的次数估算出数据包往返时间以及丢包率从而推断网络是否通常、运行是否正常等。 + +- TraceRoute + +TraceRoute 是 ICMP 的另一个应用，其主要用来跟踪一个分组从源点耗费最少 TTL 到达目的地的路径。TraceRoute 通过逐渐增大 TTL 值并重复发送数据报来实现其功能，首先，TraceRoute 会发送一个 TTL 为 1 的 IP 数据报到目的地，当路径上的第一个路由器收到这个数据报时，它将 TTL 的值减 1，此时 TTL = 0，所以路由器会将这个数据报丢掉，并返回一个差错报告报文，之后源主机会接着发送一个 TTL 为 2 的数据报，并重复此过程，直到数据报能够刚好到达目的主机。此时 TTL = 0，因此目的主机要向源主机发送 ICMP 终点不可达差错报告报文，之后源主机便知道了到达目的主机所经过的路由器 IP 地址以及到达每个路由器的往返时间。 + +### 🎯 两台电脑连起来后 ping 不通，你觉得可能存在哪些问题？ + +1. 首先看网络是否连接正常，检查网卡驱动是否正确安装。 +2. 局域网设置问题，检查 IP 地址是否设置正确。 +3. 看是否被防火墙阻拦（有些设置中防火墙会对 ICMP 报文进行过滤），如果是的话，尝试关闭防火墙 。 +4. 看是否被第三方软件拦截。 +5. 两台设备间的网络延迟是否过大（例如路由设置不合理），导致 ICMP 报文无法在规定的时间内收到。 + +### 🎯 ARP 地址解析协议的原理和地址解析过程 + +ARP（Address Resolution Protocol）是地址解析协议的缩写，该协议提供根据 IP 地址获取物理地址的功能，它工作在第二层，是一个数据链路层协议，其在本层和物理层进行联系，同时向上层提供服务。当通过以太网发送 IP 数据包时，需要先封装 32 位的 IP 地址和 48位 MAC 地址。在局域网中两台主机进行通信时需要依靠各自的物理地址进行标识，但由于发送方只知道目标 IP 地址，不知道其 MAC 地址，因此需要使用地址解析协议。 ARP 协议的解析过程如下： + +① 首先，每个主机都会在自己的 ARP 缓冲区中建立一个 ARP 列表，以表示 IP 地址和 MAC 地址之间的对应关系； + +② 当源主机要发送数据时，首先检查 ARP 列表中是否有 IP 地址对应的目的主机 MAC 地址，如果存在，则可以直接发送数据，否则就向同一子网的所有主机发送 ARP 数据包。该数据包包括的内容有源主机的 IP 地址和 MAC 地址，以及目的主机的 IP 地址。 + +③ 当本网络中的所有主机收到该 ARP 数据包时，首先检查数据包中的 目的 主机IP 地址是否是自己的 IP 地址，如果不是，则忽略该数据包，如果是，则首先从数据包中取出源主机的 IP 和 MAC 地址写入到 ARP 列表中，如果已经存在，则覆盖，然后将自己的 MAC 地址写入 ARP 响应包中，告诉源主机自己是它想要找的 MAC 地址。 + +④ 源主机收到 ARP 响应包后。将目的主机的 IP 和 MAC 地址写入 ARP 列表，并利用此信息发送数据。如果源主机一直没有收到 ARP 响应数据包，表示 ARP 查询失败。 + +### 🎯 网络地址转换 NAT + +NAT（Network Address Translation），即网络地址转换，它是一种把内部私有网络地址翻译成公有网络 IP 地址的技术。该技术不仅能解决 IP 地址不足的问题，而且还能隐藏和保护网络内部主机，从而避免来自外部网络的攻击。 + +NAT 的实现方式主要有三种： + +- 静态转换：内部私有 IP 地址和公有 IP 地址是一对一的关系，并且不会发生改变。通过静态转换，可以实现外部网络对内部网络特定设备的访问，这种方式原理简单，但当某一共有 IP 地址被占用时，跟这个 IP 绑定的内部主机将无法访问 Internet。 +- 动态转换：采用动态转换的方式时，私有 IP 地址每次转化成的公有 IP 地址是不唯一的。当私有 IP 地址被授权访问 Internet 时会被随机转换成一个合法的公有 IP 地址。当 ISP 通过的合法 IP 地址数量略少于网络内部计算机数量时，可以采用这种方式。 +- 端口多路复用：该方式将外出数据包的源端口进行端口转换，通过端口多路复用的方式，实现内部网络所有主机共享一个合法的外部 IP 地址进行 Internet 访问，从而最大限度地节约 IP 地址资源。同时，该方案可以隐藏内部网络中的主机，从而有效避免来自 Internet 的攻击。 + +### 🎯 TTL 是什么？有什么作用 + +TTL 是指生存时间，简单来说，它表示了数据包在网络中的时间。每经过一个路由器后 TTL 就减一，这样 TTL 最终会减为 0 ，当 TTL 为 0 时，则将数据包丢弃。通过设置 TTL 可以避免这两个路由器之间形成环导致数据包在环路上死转的情况，由于有了 TTL ，当 TTL 为 0 时，数据包就会被抛弃。 + +--- + + + +## 五、网络安全 + +### 🎯 安全攻击有哪些 + +网络安全攻击主要分为被动攻击和主动攻击两类： + +- 被动攻击：攻击者窃听和监听数据传输，从而获取到传输的数据信息，被动攻击主要有两种形式：消息内容泄露攻击和流量分析攻击。由于攻击者并没有修改数据，使得这种攻击类型是很难被检测到的。 +- 主动攻击：攻击者修改传输的数据流或者故意添加错误的数据流，例如假冒用户身份从而得到一些权限，进行权限攻击，除此之外，还有重放、改写和拒绝服务等主动攻击的方式。 + +### 🎯 ARP 攻击 + +在 ARP 的解析过程中，局域网上的任何一台主机如果接收到一个 ARP 应答报文，并不会去检测这个报文的真实性，而是直接记入自己的 ARP 缓存表中。并且这个 ARP 表是可以被更改的，当表中的某一列长时间不适使用，就会被删除。ARP 攻击就是利用了这一点，攻击者疯狂发送 ARP 报文，其源 MAC 地址为攻击者的 MAC 地址，而源 IP 地址为被攻击者的 IP 地址。通过不断发送这些伪造的 ARP 报文，让网络内部的所有主机和网关的 ARP 表中被攻击者的 IP 地址所对应的 MAC 地址为攻击者的 MAC 地址。这样所有发送给被攻击者的信息都会发送到攻击者的主机上，从而产生 ARP 欺骗。通常可以把 ARP 欺骗分为以下几种： + +- 洪泛攻击 + + 攻击者恶意向局域网中的网关、路由器和交换机等发送大量 ARP 报文，设备的 CPU 忙于处理 ARP 协议，而导致难以响应正常的服务请求。其表现通常为：网络中断或者网速很慢。 + +- 欺骗主机 + + 这种攻击方式也叫仿冒网关攻击。攻击者通过 ARP 欺骗使得网络内部被攻击主机发送给网关的信息实际上都发送给了攻击者，主机更新的 ARP 表中对应的 MAC 地址为攻击者的 MAC。当用户主机向网关发送重要信息使，该攻击方式使得用户的数据存在被窃取的风险。 + +- 欺骗网关 + + 该攻击方式和欺骗主机的攻击方式类似，不过这种攻击的欺骗对象是局域网的网关，当局域网中的主机向网关发送数据时，网关会把数据发送给攻击者，这样攻击者就会源源不断地获得局域网中用户的信息。该攻击方式同样会造成用户数据外泄。 + +- 中间人攻击：攻击者同时欺骗网关和主机，局域网的网关和主机发送的数据最后都会到达攻击者这边。这样，网关和用户的数据就会泄露。 + +- IP 地址冲突：攻击者对局域网中的主机进行扫描，然后根据物理主机的 MAC 地址进行攻击，导致局域网内的主机产生 IP 冲突，使得用户的网络无法正常使用。 + + + +### 🎯 AES 的过程 + +AES（Advanced Encryption Standard）即密码学的高级加密标准，也叫做 Rijndeal 加密法，是最为常见的一种对称加密算法，和传统的对称加密算法大致的流程类似，在发送端需要采用加密算法对明文进行加密，在接收端需要采用与加密算法相同的算法进行解密，不同的是， AES 采用分组加密的方式，将明文分成一组一组的，每组长度相等，每次加密一组数据，直到加密完整个明文。在 AES 标准中，分组长度固定为 128 位，即每个分组为 16 个字节（每个字节有 8 位）。而密钥的长度可以是 128 位，192 位或者 256 位。并且密钥的长度不同，推荐加密的轮数也不同。 + +我们以 128 位密钥为例（加密轮次为 10），已知明文首先需要分组，每一组大小为16个字节并形成 4 × 4 的状态矩阵（矩阵中的每一个元素代表一个字节）。类似地，128 位密钥同样用 4 × 4 的字节矩阵表示，矩阵中的每一列称为 1 个 32 位的比特字。通过密钥编排函数该密钥矩阵被扩展成一个由 44 个字组成的序列，该序列的前四个字是原始密钥，用于 AES 的初始密钥加过程，后面 40 个字分为 10 组，每组 4 个字分别用于 10 轮加密运算中的轮密钥加。在每轮加密过程中主要包括四个步骤： + +① **字节代换**：AES 的字节代换其实是一个简易的查表操作，在 AES 中定义了一个 S-box 和一个逆 S-box，我们可以将其简单地理解为两个映射表，在做字节代换时，状态矩阵中的每一个元素（字节）的高四位作为行值，低四位作为列值，取出 S-box 或者逆 S-box 中对应的行或者列作为输出。 + +② **行位移**：顾名思义，就是对状态矩阵的每一行进行位移操作，其中状态矩阵的第 0 行左移 0 位，第 1 行左移 1 位，以此类推。 + +③ **列混合**：列混合变换是通过矩阵相乘来实现的，经唯一后的状态矩阵与固定的矩阵相乘，从而得到混淆后的状态矩阵。其中矩阵相乘中涉及到的加法等价于两个字节的异或运算，而乘法相对复杂一些，对于状态矩阵中的每一个 8 位二进制数来说，首先将其与 00000010 相乘，其等效为将 8 位二进制数左移一位，若原二进制数的最高位是 1 的话再将左移后的数与 00011011 进行异或运算。 + +④ **轮密相加**：在开始时我们提到，128 位密钥通过密钥编排函数被扩展成 44 个字组成的序列，其中前 4 个字用于加密过程开始时对原始明文矩阵进行异或运算，而后 40 个字中每四个一组在每一轮中与状态矩阵进行异或运算（共计 10 轮）。 + +上述过程即为 AES 加密算法的主要流程，在我们的例子中，上述过程需要经过 10 轮迭代。而 AES 的解密过程的各个步骤和加密过程是一样的，只是用逆变换取代原来的变换。 + +### 🎯 RSA 和 AES 算法有什么区别 + +- RSA：采用非对称加密的方式，采用公钥进行加密，私钥解密的形式。其私钥长度一般较长，除此之外，由于需要大数的乘幂求模等运算，其运算速度较慢，不适合大量数据文件加密。 + +- AES：采用对称加密的方式，其密钥长度最长只有 256 个比特，加密和解密速度较快，易于硬件实现。由于是对称加密，通信双方在进行数据传输前需要获知加密密钥。 + +基于上述两种算法的特点，一般使用 RSA 传输密钥给对方，之后使用 AES 进行加密通信。 + +### 🎯 DDoS 有哪些，如何防范 + +DDoS 为分布式拒绝服务攻击，是指处于不同位置的多个攻击者同时向一个或数个目标发动攻击，或者一个攻击者控制了不同位置上的多台机器并利用这些机器对受害者同时实施攻击。和单一的 DoS 攻击相比，DDoS 是借助数百台或者数千台已被入侵并添加了攻击进程的主机一起发起网络攻击。 + +DDoS 攻击主要有两种形式：流量攻击和资源耗尽攻击。前者主要针对网络带宽，攻击者和已受害主机同时发起大量攻击导致网络带宽被阻塞，从而淹没合法的网络数据包；后者主要针对服务器进行攻击，大量的攻击包会使得服务器资源耗尽或者 CPU 被内核应用程序占满从而无法提供网络服务。 + +常见的 DDos 攻击主要有：TCP 洪水攻击（SYN Flood）、放射性攻击（DrDos）、CC 攻击（HTTP Flood）等。 + +针对 DDoS 中的流量攻击，最直接的方法是增加带宽，理论上只要带宽大于攻击流量就可以了，但是这种方法成本非常高。在有充足网络带宽的前提下，我们应尽量提升路由器、网卡、交换机等硬件设施的配置。 + +针对资源耗尽攻击，我们可以升级主机服务器硬件，在网络带宽得到保证的前提下，使得服务器能有效对抗海量的 SYN 攻击包。我们也可以安装专业的抗 DDoS 防火墙，从而对抗 SYN Flood等流量型攻击。此外，负载均衡，CDN 等技术都能够有效对抗 DDoS 攻击 + + + +### 🎯 XSS 攻击 + +**XSS 攻击**（Cross-Site Scripting，跨站脚本攻击）是一种常见的网络安全攻击，攻击者通过在网站的输入字段中注入恶意脚本，当其他用户访问该网站时，恶意脚本会在他们的浏览器中执行，从而窃取敏感信息、篡改页面内容或进行其他恶意操作。 + +XSS 攻击可以分为三种类型： + +1. **存储型 XSS**： + - 攻击者的脚本被存储在目标服务器上，通常是在数据库中。 + - 当其他用户访问受感染的页面时，恶意脚本作为正常内容的一部分被发送到用户的浏览器中执行。 +2. **反射型 XSS**： + - 攻击者的脚本不是存储在服务器上，而是在用户访问特定页面或请求时，通过 URL 参数或其他方式传递给服务器。 + - 服务器将恶意脚本作为响应的一部分发送回用户的浏览器，如果浏览器解析并执行了这些脚本，就构成了攻击。 +3. **DOM 型 XSS**： + - 这种类型的 XSS 攻击与服务器无关，恶意脚本完全在客户端执行。 + - 攻击者利用浏览器的 DOM 解析特性，通过修改页面的 DOM 来注入恶意脚本。 + +**防御 XSS 攻击的策略：** + +1. **输入验证**：对所有用户输入进行严格的验证，确保不接受潜在的恶意代码。 +2. **输出编码**：在将数据发送到浏览器之前，对所有输出进行适当的编码，以确保潜在的脚本被安全地呈现。 +3. **内容安全策略（CSP）**：使用内容安全策略来限制可以执行的脚本的来源，减少 XSS 攻击的风险。 +4. **HTTP 头**：设置适当的 HTTP 头，如 `X-XSS-Protection`，以启用浏览器的 XSS 过滤功能。 + +--- + +## 六、网络理论基础 + +### 🎯 OSI七层模型和TCP/IP四层模型 + +**OSI七层模型**： + +1. **物理层**：负责比特流的传输，定义电压、接口等物理特性 +2. **数据链路层**：负责帧的传输，错误检测和纠正 +3. **网络层**：负责数据包的路由和转发，如IP协议 +4. **传输层**：负责端到端的数据传输，如TCP、UDP协议 +5. **会话层**：负责建立、管理和终止会话 +6. **表示层**：负责数据的编码、加密和压缩 +7. **应用层**：负责为用户提供网络服务，如HTTP、FTP等 + +**TCP/IP四层模型**： + +1. **网络接口层**（对应OSI的物理层和数据链路层） +2. **网络层**（对应OSI的网络层）：IP协议 +3. **传输层**（对应OSI的传输层）：TCP、UDP协议 +4. **应用层**（对应OSI的会话层、表示层、应用层）：HTTP、FTP、DNS等 + + +### 🎯 IP地址的分类 + +IP 地址是指互联网协议地址，是 IP 协议提供的一种统一的地址格式，它为互联网上的每一个网络和每一台主机分配一个逻辑地址，以此来屏蔽物理地址的差异。IP地址编址方案将IP地址空间划分为A、B、C、D、E五类，其中A、B、C是基本类，D、E类作为多播和保留使用，为特殊地址。 + +每个IP地址包括两个标识码（ID），即网络ID和主机ID。同一个物理网络上的所有主机都使用同一个网络ID，网络上的一个主机（包括网络上工作站，服务器和路由器等）有一个主机ID与其对应。A~E类地址的特点如下： + +A类地址：以0开头，第一个字节范围：0~127； + +B类地址：以10开头，第一个字节范围：128~191； + +C类地址：以110开头，第一个字节范围：192~223； + +D类地址：以1110开头，第一个字节范围为224~239； + +E类地址：以1111开头，保留地址 + +![img](https://ask.qcloudimg.com/http-save/yehe-5359587/3jrhedfg36.jpeg) + + + +### 🎯 HTTP 是不保存状态的协议,如何保存用户状态 + +我们知道，假如某个特定的客户机在短时间内两次请求同一个对象，服务器并不会因为刚刚为该用户提供了该对象就不再做出反应，而是重新发送该对象，就像该服务器已经完全忘记不久之前所做过的事一样。因为一个 HTTP 服务器并不保存关于客户机的任何信息，所以我们说 HTTP 是一个无状态协议。 + +通常有两种解决方案： + +① 基于 Session 实现的会话保持 + +在客户端第一次向服务器发送 HTTP 请求后，服务器会创建一个 Session 对象并将客户端的身份信息以键值对的形式存储下来，然后分配一个会话标识（SessionId）给客户端，这个会话标识一般保存在客户端 Cookie 中，之后每次该浏览器发送 HTTP 请求都会带上 Cookie 中的 SessionId 到服务器，服务器根据会话标识就可以将之前的状态信息与会话联系起来，从而实现会话保持。 + +优点：安全性高，因为状态信息保存在服务器端。 + +缺点：由于大型网站往往采用的是分布式服务器，浏览器发送的 HTTP 请求一般要先通过负载均衡器才能到达具体的后台服务器，倘若同一个浏览器两次 HTTP 请求分别落在不同的服务器上时，基于 Session 的方法就不能实现会话保持了。 + +【解决方法：采用中间件，例如 Redis，我们通过将 Session 的信息存储在 Redis 中，使得每个服务器都可以访问到之前的状态信息】 + +② 基于 Cookie 实现的会话保持 + +当服务器发送响应消息时，在 HTTP 响应头中设置 Set-Cookie 字段，用来存储客户端的状态信息。客户端解析出 HTTP 响应头中的字段信息，并根据其生命周期创建不同的 Cookie，这样一来每次浏览器发送 HTTP 请求的时候都会带上 Cookie 字段，从而实现状态保持。基于 Cookie 的会话保持与基于 Session 实现的会话保持最主要的区别是前者完全将会话状态信息存储在浏览器 Cookie 中。 + +优点：服务器不用保存状态信息， 减轻服务器存储压力，同时便于服务端做水平拓展。 + +缺点：该方式不够安全，因为状态信息存储在客户端，这意味着不能在会话中保存机密数据。除此之外，浏览器每次发起 HTTP 请求时都需要发送额外的 Cookie 到服务器端，会占用更多带宽。 + +拓展：Cookie被禁用了怎么办？ + +若遇到 Cookie 被禁用的情况，则可以通过重写 URL 的方式将会话标识放在 URL 的参数里，也可以实现会话保持。 + + + +### 🎯 什么是Cookie，Cookie的使用过程是怎么样的？ + +由于 http 协议是无状态协议，如果客户通过浏览器访问 web 应用时没有一个保存用户访问状态的机制，那么将不能持续跟踪应用的操作。比如当用户往购物车中添加了商品，web 应用必须在用户浏览别的商品的时候仍保存购物车的状态，以便用户继续往购物车中添加商品。 + +cookie 是浏览器的一种缓存机制，它可用于维持客户端与服务器端之间的会话。由于下面一题会讲到 session，所以这里要强调cookie会将会话保存在客户端（session则是把会话保存在服务端） + +这里以最常见的登陆案例讲解cookie的使用过程： + +1. 首先用户在客户端浏览器向服务器发起登陆请求 +2. 登陆成功后，服务端会把登陆的用户信息设置 cookie 中，返回给客户端浏览器 +3. 客户端浏览器接收到 cookie 请求后，会把 cookie 保存到本地（可能是内存，也可能是磁盘，看具体使用情况而定） +4. 以后再次访问该 web 应用时，客户端浏览器就会把本地的 cookie 带上，这样服务端就能根据 cookie 获得用户信息了 + + + +### 🎯 什么是session，有哪些实现session的机制？ + +session 是一种维持客户端与服务器端会话的机制。但是与 **cookie 把会话信息保存在客户端本地不一样，session 把会话保留在浏览器端。** + +我们同样以登陆案例为例子讲解 session 的使用过程： + +1. 首先用户在客户端浏览器发起登陆请求 +2. 登陆成功后，服务端会把用户信息保存在服务端，并返回一个唯一的 session 标识给客户端浏览器。 +3. 客户端浏览器会把这个唯一的 session 标识保存在起来 +4. 以后再次访问 web 应用时，客户端浏览器会把这个唯一的 session 标识带上，这样服务端就能根据这个唯一标识找到用户信息。 + +看到这里可能会引起疑问：把唯一的 session 标识返回给客户端浏览器，然后保存起来，以后访问时带上，这难道不是 cookie 吗？ + +没错，s**ession 只是一种会话机制，在许多 web 应用中，session 机制就是通过 cookie 来实现的**。也就是说它只是使用了 cookie 的功能，并不是使用 cookie 完成会话保存。与 cookie 在保存客户端保存会话的机制相反，session 通过 cookie 的功能把会话信息保存到了服务端。 + +进一步地说，session 是一种维持服务端与客户端之间会话的机制，它可以有不同的实现。以现在比较流行的小程序为例，阐述一个 session 的实现方案： + +1. 首先用户登陆后，需要把用户登陆信息保存在服务端，这里我们可以采用 redis。比如说给用户生成一个 userToken，然后以 userId 作为键，以 userToken 作为值保存到 redis 中，并在返回时把 userToken 带回给小程序端。 +2. 小程序端接收到 userToken 后把它缓存起来，以后每当访问后端服务时就把 userToken 带上。 +3. 在后续的服务中服务端只要拿着小程序端带来的 userToken 和 redis 中的 userToken 进行比对，就能确定用户的登陆状态了。 + + + +### 🎯 session和cookie有什么区别 + +经过上面两道题的阐述，这道题就很清晰了 + +1. cookie 是浏览器提供的一种缓存机制，它可以用于维持客户端与服务端之间的会话 +2. session 指的是维持客户端与服务端会话的一种机制，它可以通过 cookie 实现，也可以通过别的手段实现。 +3. 如果用 cookie 实现会话，那么会话会保存在客户端浏览器中 +4. 而 session 机制提供的会话是保存在服务端的。 + + +## 七、应用层深入 + +### 🎯 如果你访问一个网站很慢，怎么排查和解决 + +网页打开速度慢的原因有很多，这里列举出一些较常出现的问题： + +① 首先最直接的方法是查看本地网络是否正常，可以通过网络测速软件例如电脑管家等对电脑进行测速，若网速正常，我们查看网络带宽是否被占用，例如当你正在下载电影时并且没有限速，是会影响你打开网页的速度的，这种情况往往是处理器内存小导致的； + +② 当网速测试正常时，我们对网站服务器速度进行排查，通过 ping 命令查看链接到服务器的时间和丢包等情况，一个速度好的机房，首先丢包率不能超过 1%，其次 ping 值要小，最后是 ping 值要稳定，如最大和最小差值过大说明路由不稳定。或者我们也可以查看同台服务器上其他网站的打开速度，看是否其他网站打开也慢。 + +③ 如果网页打开的速度时快时慢，甚至有时候打不开，有可能是空间不稳定的原因。当确定是该问题时，就要找你的空间商解决或换空间商了，如果购买空间的话，可选择购买购买双线空间或多线空间；如果是在有的地方打开速度快，有的地方打开速度慢，那应该是网络线路的问题。电信线路用户访问放在联通服务器的网站，联通线路用户访问放在电信服务器上的网站，相对来说打开速度肯定是比较慢。 + +④ 从网站本身找原因。网站的问题主要包括网站程序设计、网页设计结构和网页内容三个部分。 + +- 网站程序设计：当访问网页中有拖慢网站打开速度的代码，会影响网页的打开速度，例如网页中的统计代码，我们最好将其放在网站的末尾。因此我们需要查看网页程序的设计结构是否合理； +- 网页设计结构：如果是 table 布局的网站，查看是否嵌套次数太多，或是一个大表格分成多个表格这样的网页布局，此时我们可以采用 div 布局并配合 css 进行优化。 +- 网页内容：查看网页中是否有许多尺寸大的图片或者尺寸大的 flash 存在，我们可以通过降低图片质量，减小图片尺寸，少用大型 flash 加以解决。此外，有的网页可能过多地引用了其他网站的内容，若某些被引用的网站访问速度慢，或者一些页面已经不存在了，打开的速度也会变慢。一种直接的解决方法是去除不必要的加载项。 + + + +> 这部分内容对常规后端开发，，，了解点就可以　　　 + +### 🎯 DNS 的作用和原理？ + +DNS（Domain Name System）是域名系统的英文缩写，是一种组织成域层次结构的计算机和网络服务命名系统，用于 TCP/IP 网络。 + +通常我们有两种方式识别主机：通过主机名或者 IP 地址。人们喜欢便于记忆的主机名表示，而路由器则喜欢定长的、有着层次结构的 IP 地址。为了满足这些不同的偏好，我们就需要一种能够进行主机名到 IP 地址转换的目录服务，域名系统作为将域名和 IP 地址相互映射的一个分布式数据库，能够使人更方便地访问互联网。 + +**DNS 域名解析原理** + +DNS 采用了分布式的设计方案，其域名空间采用一种树形的层次结构： ![img](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/86c84543fb6c40149ec721b5f015247d~tplv-k3u1fbpfcp-watermark.awebp) + +上图展示了 DNS 服务器的部分层次结构，从上到下依次为根域名服务器、顶级域名服务器和权威域名服务器。其实根域名服务器在因特网上有13个，大部分位于北美洲。第二层为顶级域服务器，这些服务器负责顶级域名（如 com、org、net、edu）和所有国家的顶级域名（如uk、fr、ca 和 jp）。在第三层为权威 DNS 服务器，因特网上具有公共可访问主机（例如 Web 服务器和邮件服务器）的每个组织机构必须提供公共可访问的 DNS 记录，这些记录由组织机构的权威 DNS 服务器负责保存，这些记录将这些主机的名称映射为 IP 地址。 + +除此之外，还有一类重要的 DNS 服务器，叫做本地 DNS 服务器。本地 DNS 服务器严格来说不在 DNS 服务器的层次结构中，但它对 DNS 层次结构是很重要的。一般来说，每个网络服务提供商（ISP） 都有一台本地 DNS 服务器。当主机与某个 ISP 相连时，该 ISP 提供一台主机的 IP 地址，该主机具有一台或多台其本地 DNS 服务器的 IP 地址。主机的本地 DNS 服务器通常和主机距离较近，当主机发起 DNS 请求时，该请求被发送到本地 DNS 服务器，它起着代理的作用，并将该请求转发到 DNS 服务器层次结构中。 + +我们以一个例子来了解 DNS 的工作原理，假设主机 A（IP 地址为 abc.xyz.edu） 想知道主机 B 的 IP 地址 （def.mn.edu），如下图所示，主机 A 首先向它的本地 DNS 服务器发送一个 DNS 查询报文。该查询报文含有被转换的主机名 def.mn.edu。本地 DNS 服务器将该报文转发到根 DNS 服务器，根 DNS 服务器注意到查询的 IP 地址前缀为 edu 后向本地 DNS 服务器返回负责 edu 的顶级域名服务器的 IP 地址列表。该本地 DNS 服务器则再次向这些 顶级域名服务器发送查询报文。该顶级域名服务器注意到 mn.edu 的前缀，并用权威域名服务器的 IP 地址进行响应。通常情况下，顶级域名服务器并不总是知道每台主机的权威 DNS 服务器的 IP 地址，而只知道中间的某个服务器，该中间 DNS 服务器依次能找到用于相应主机的 IP 地址，我们假设中间经历了权威服务器 ① 和 ②，最后找到了负责 def.mn.edu 的权威 DNS 服务器 ③，之后，本地 DNS 服务器直接向该服务器发送查询报文从而获得主机 B 的IP 地址。 + +![img](https://p3-juejin.byteimg.com/tos-cn-i-k3u1fbpfcp/f6dba7c39fd1418aa3f8b007771b0953~tplv-k3u1fbpfcp-watermark.awebp) + +在上图中，IP 地址的查询其实经历了两种查询方式，分别是递归查询和迭代查询。 + +**拓展：域名解析查询的两种方式** + +**递归查询**：如果主机所询问的本地域名服务器不知道被查询域名的 IP 地址，那么本地域名服务器就以 DNS 客户端的身份，向其他根域名服务器继续发出查询请求报文，即替主机继续查询，而不是让主机自己进行下一步查询，如上图步骤（1）和（10）。 + +**迭代查询**：当根域名服务器收到本地域名服务器发出的迭代查询请求报文时，要么给出所要查询的 IP 地址，要么告诉本地服务器下一步应该找哪个域名服务器进行查询，然后让本地服务器进行后续的查询，如上图步骤（2）~（9）。 + +### 🎯 DNS 为什么用 UDP + +DNS 既使用 TCP 又使用 UDP。 + +当进行区域传送（主域名服务器向辅助域名服务器传送变化的那部分数据）时会使用 TCP，因为数据同步传送的数据量比一个请求和应答的数据量要多，而 TCP 允许的报文长度更长，因此为了保证数据的正确性，会使用基于可靠连接的 TCP。 + +当客户端向 DNS 服务器查询域名 ( 域名解析) 的时候，一般返回的内容不会超过 UDP 报文的最大长度，即 512 字节。用 UDP 传输时，不需要经过 TCP 三次握手的过程，从而大大提高了响应速度，但这要求域名解析器和域名服务器都必须自己处理超时和重传从而保证可靠性。 + +### 🎯 怎么实现 DNS 劫持 + +DNS 劫持即域名劫持，是通过将原域名对应的 IP 地址进行替换从而使得用户访问到错误的网站或者使得用户无法正常访问网站的一种攻击方式。域名劫持往往只能在特定的网络范围内进行，范围外的 DNS 服务器能够返回正常的 IP 地址。攻击者可以冒充原域名所属机构，通过电子邮件的方式修改组织机构的域名注册信息，或者将域名转让给其它组织，并将新的域名信息保存在所指定的 DNS 服务器中，从而使得用户无法通过对原域名进行解析来访问目的网址。 + +具体实施步骤如下： + +1. 获取要劫持的域名信息：攻击者首先会访问域名查询站点查询要劫持的域名信息。 +2. 控制域名相应的 E-MAIL 账号：在获取到域名信息后，攻击者通过暴力破解或者专门的方法破解公司注册域名时使用的 E-mail 账号所对应的密码。更高级的攻击者甚至能够直接对 E-mail 进行信息窃取。 +3. 修改注册信息：当攻击者破解了 E-MAIL 后，会利用相关的更改功能修改该域名的注册信息，包括域名拥有者信息，DNS 服务器信息等。 +4. 使用 E-MAIL 收发确认函：在修改完注册信息后，攻击者在 E-mail 真正拥有者之前收到修改域名注册信息的相关确认信息，并回复确认修改文件，待网络公司恢复已成功修改信件后，攻击者便成功完成 DNS 劫持。 + +**用户端的一些预防手段：** + +直接通过 IP 地址访问网站，避开 DNS 劫持。 由于域名劫持往往只能在特定的网络范围内进行，因此一些高级用户可以通过网络设置让 DNS 指向正常的域名服务器以实现对目的网址的正常访问，例如将计算机首选 DNS 服务器的地址固定为 8.8.8.8。 + +### 🎯 socket() 套接字有哪些 + +套接字（Socket）是对网络中不同主机上的应用进程之间进行双向通信的端点的抽象，网络进程通信的一端就是一个套接字，不同主机上的进程便是通过套接字发送报文来进行通信。例如 TCP 用主机的 IP 地址 + 端口号作为 TCP 连接的端点，这个端点就叫做套接字。 + +套接字主要有以下三种类型： + +1. 流套接字（SOCK_STREAM）：流套接字基于 TCP 传输协议，主要用于提供面向连接、可靠的数据传输服务。由于 TCP 协议的特点，使用流套接字进行通信时能够保证数据无差错、无重复传送，并按顺序接收，通信双方不需要在程序中进行相应的处理。 +2. 数据报套接字（SOCK_DGRAM）：和流套接字不同，数据报套接字基于 UDP 传输协议，对应于无连接的 UDP 服务应用。该服务并不能保证数据传输的可靠性，也无法保证对端能够顺序接收到数据。此外，通信两端不需建立长时间的连接关系，当 UDP 客户端发送一个数据给服务器后，其可以通过同一个套接字给另一个服务器发送数据。当用 UDP 套接字时，丢包等问题需要在程序中进行处理。 +3. 原始套接字（SOCK_RAW）：由于流套接字和数据报套接字只能读取 TCP 和 UDP 协议的数据，当需要传送非传输层数据包（例如 Ping 命令时用的 ICMP 协议数据包）或者遇到操作系统无法处理的数据包时，此时就需要建立原始套接字来发送。 + + + +### 🎯 REST 和 WebSocket 的区别？ + +REST（Representational State Transfer）和WebSocket是两种不同的网络通信协议，它们在用途、工作原理和使用场景上有所区别： + +1. **协议类型**： + - **REST**：是一种基于HTTP协议的架构风格，用于网络资源的状态转移和管理系统。 + - **WebSocket**：是一种网络通信协议，提供了在单个TCP连接上进行全双工通信的能力。 +2. **通信模式**： + - **REST**：通常是基于请求/响应模式，客户端发送请求，服务器响应请求。 + - **WebSocket**：允许服务器主动向客户端发送消息，实现服务器推送。 +3. **连接持久性**： + - **REST**：每次请求/响应完成后，连接通常会关闭，除非使用了HTTP持久连接（HTTP/1.1的keep-alive或HTTP/2）。 + - **WebSocket**：建立了一个持久的连接，可以保持开放状态，用于实时双向通信。 +4. **使用场景**： + - **REST**：适用于需要获取、创建、修改或删除网络资源的场景，如API调用、Web服务等。 + - **WebSocket**：适用于需要实时数据传输的场景，如在线游戏、实时聊天应用、股票行情更新等。 +5. **数据格式**： + - **REST**：通常使用JSON或XML作为数据交换格式。 + - **WebSocket**：可以传输任何格式的数据，包括文本和二进制数据。 +6. **性能**： + - **REST**：可能涉及较多的HTTP请求和响应，对于需要频繁通信的场景可能性能较低。 + - **WebSocket**：由于连接持久，减少了建立和关闭连接的开销，适用于需要频繁通信的场景。 +7. **安全性**： + - **REST**：可以通过HTTPS提供安全通信。 + - **WebSocket**：也可以通过WSS（WebSocket Secure）提供安全通信。 +8. **兼容性**： + - **REST**：几乎所有的Web服务器和客户端都支持HTTP协议。 + - **WebSocket**：需要服务器和客户端支持WebSocket协议。 +9. **API设计**： + - **REST**：强调资源的统一接口和无状态操作，易于理解和使用。 + - **WebSocket**：提供了一种更为灵活的通信方式，但可能需要更复杂的逻辑来管理连接和数据流。 + +总的来说，REST和WebSocket各有优势，选择哪种技术取决于具体的应用需求和场景。 + + + +### 🎯 HTTP缓存怎么实现的 + +**HTTP 缓存** 是一种通过在客户端（如浏览器）或中间服务器（如代理服务器、CDN）存储资源副本，减少服务器负载并提高网页加载速度的技术。它通过缓存经常请求的资源，避免每次都向服务器发送相同的请求，从而提升性能。 + +**HTTP 缓存的基本概念** + +1. **强缓存（强制缓存 / 本地缓存）**： + - 浏览器直接从缓存中读取资源，而不与服务器通信。 + - 如果缓存资源在缓存期间有效，浏览器不会向服务器发送请求。 +2. **协商缓存（对比缓存）**： + - 浏览器会向服务器验证缓存是否仍然有效。 + - 如果资源没有更改，服务器会返回 `304 Not Modified`，浏览器继续使用缓存中的副本。 + +**缓存相关的 HTTP 响应头** + +服务器通过响应头告诉客户端如何缓存资源。以下是常用的缓存控制头： + +1. `Cache-Control`：`Cache-Control` 是控制缓存行为的最重要的 HTTP 头，可以定义资源的缓存策略。常见的 `Cache-Control` 指令包括： + + - **`max-age`**：定义资源可以缓存的最大时间（以秒为单位）。例如，`max-age=3600` 表示缓存 1 小时。 + + - **`no-cache`**：每次请求都会去服务器进行验证，协商缓存是否有效。 + + - **`no-store`**：禁止任何缓存，无论是强缓存还是协商缓存。 + + - **`public`**：资源可以被任何缓存（包括客户端和代理服务器）缓存。 + + - **`private`**：资源只能被客户端缓存，不能被代理服务器缓存。 + + - **`must-revalidate`**：在缓存过期后，必须向服务器验证资源是否还有效。 + +2. `Expires` + + `Expires` 头部用于指定资源的过期时间，表示在此时间之前可以直接使用缓存。它是 `HTTP/1.0` 版本中的缓存机制，通常会被 `Cache-Control` 的 `max-age` 覆盖。 + +3. `ETag` + + `ETag` 是资源的唯一标识符，用于协商缓存。当客户端请求资源时，会带上 `ETag`，服务器通过对比 `ETag` 判断资源是否发生变化。 + +4. `Last-Modified` + + `Last-Modified` 记录资源最后一次修改的时间。客户端可以通过 `If-Modified-Since` 请求头询问服务器该资源是否自该时间以来发生了变化。 + +**强缓存与协商缓存的工作流程** + +1. **强缓存**：强缓存是浏览器在缓存资源有效期内直接从本地缓存中获取资源，不与服务器通信。常用的头部有 `Cache-Control` 和 `Expires`。 + + **工作流程：** + + - 浏览器发起请求并获取资源及其缓存策略（例如 `Cache-Control: max-age=3600`）。 + + - 在 3600 秒内，浏览器再次请求该资源时，直接从本地缓存中读取资源，而不会发送请求到服务器。 + +2. **协商缓存**：当强缓存失效时，浏览器会与服务器进行通信，询问资源是否有更新。协商缓存通过 `ETag` 或 `Last-Modified` 进行判断。 + + 工作流程： + + - 浏览器发送请求，附带 `If-None-Match` 或 `If-Modified-Since` 头部。 + + - 服务器检查资源是否发生变化： + - 如果资源未变化，服务器返回 `304 Not Modified`，浏览器使用本地缓存的副本。 + - 如果资源已变化，服务器返回最新的资源及状态码 `200`。 + +**缓存的优先级** + +1. **`Cache-Control` vs `Expires`**： 如果 `Cache-Control` 和 `Expires` 同时存在，`Cache-Control` 的 `max-age` 优先级更高。 +2. **`ETag` vs `Last-Modified`**： `ETag` 的优先级高于 `Last-Modified`。如果两者都存在，服务器通常会首先比较 `ETag`，然后再检查 `Last-Modified`。 + + + +### 🎯 什么是幂等性？如何设计一个幂等的接口？ + +幂等性（Idempotence）是指一个操作多次执行和一次执行的效果相同，不会因为多次请求而产生额外的副作用。 + +在数学中，如果函数 f(x) 满足 f(f(x)) = f(x)，则称该函数是幂等的。在计算机科学和网络通信中，幂等性通常指以下两种情况： + +1. **重复请求**：对于同一个操作的多次请求，系统应该能够处理重复的请求而不产生额外的副作用。 +2. **并发请求**：对于同时发起的多个相同请求，系统应该能够确保操作只被执行一次。 + +设计幂等接口时，需要考虑以下几个关键点： + +1. **根据 HTTP 方法的语义设计** + + - **GET**：天然幂等，因为它只是查询资源，不会修改服务器状态。 + + - **DELETE**：确保重复删除同一个资源不会出错，比如返回 "Resource Not Found" 表示已经删除。 + + - **PUT**：确保更新逻辑只会修改指定资源，比如通过资源 ID 定位并更新。 + +2. **唯一标识符**：为每个请求分配一个唯一的标识符（如订单号、交易ID等），确保可以通过这个标识符检查操作是否已经执行过。 +3. **状态检查**：在执行操作前，检查当前状态是否允许执行该操作。如果操作已经完成或正在进行中，则拒绝重复的请求。 +4. **数据库设计**：在数据库中设计适当的约束（如唯一索引）和事务控制，确保数据的一致性和完整性。 + +------ + + + +## 八、数据链路层 + +### 🎯 数据链路层概述 + +数据链路层（Data Link Layer）是OSI七层模型的第二层，位于物理层之上、网络层之下。它的主要功能是在相邻的网络节点间提供可靠的数据传输服务，并处理物理层可能产生的错误。 + +#### 数据链路层的主要功能 + +1. **成帧（Framing）**：将网络层传下来的数据包封装成帧，添加帧头和帧尾标识 +2. **错误检测和纠正**：通过校验和、CRC等方式检测和纠正传输错误 +3. **流量控制**：调节数据发送速率，防止接收方缓冲区溢出 +4. **访问控制**：在共享媒体中控制对传输媒体的访问 + +#### 帧结构详解 + +**基本帧格式**： + +``` ++----------+----------+----------+----------+ +| 帧起始符 | 帧头部 | 数据部分 | 帧尾部 | ++----------+----------+----------+----------+ +``` + +**以太网帧结构**： + +``` ++----------+----------+------+----------+--------+----------+ +| 前导码 | 目的地址 | 源地址| 类型/长度| 数据 | 帧校验序列| +| 8字节 | 6字节 | 6字节 | 2字节 |46-1500 | 4字节 | ++----------+----------+------+----------+--------+----------+ +``` + +- **前导码**：用于同步，包含7个字节的前导码和1个字节的帧起始定界符 +- **目的地址/源地址**：48位MAC地址，标识帧的接收方和发送方 +- **类型/长度**：表示上层协议类型或帧中数据的长度 +- **数据**：实际传输的数据，最小46字节，最大1500字节 +- **帧校验序列**：32位CRC校验码，用于错误检测 + +### 🎯 MAC地址深入解析 + +#### MAC地址的结构和特性 + +MAC（Media Access Control）地址是数据链路层和物理层使用的地址，长度为48位（6字节），通常用十六进制表示。 + +**MAC地址结构**： + +``` +XX:XX:XX:XX:XX:XX +|----| |---------| + OUI NIC +组织唯一 网络接口 +标识符 控制器 +``` + +- **前24位（3字节）**：OUI（Organizationally Unique Identifier），由IEEE分配给厂商 +- **后24位（3字节）**：NIC（Network Interface Controller），由厂商自行分配 + +#### MAC地址的类型 + +1. **单播地址**：第一个字节的最低位为0，用于点对点通信 +2. **多播地址**：第一个字节的最低位为1，用于一对多通信 +3. **广播地址**：FF:FF:FF:FF:FF:FF，用于向网段内所有设备发送数据 + +#### MAC 地址和 IP 地址分别有什么作用 + +- **MAC 地址**：数据链路层和物理层使用的地址，是写在网卡上的物理地址。MAC 地址用来定义网络设备的位置，在局域网内唯一标识设备。 +- **IP 地址**：网络层和以上各层使用的地址，是一种逻辑地址。IP 地址用来区别网络上的计算机，实现跨网络的路由寻址。 + +### 🎯 错误检测机制 + +#### CRC循环冗余校验 + +CRC（Cyclic Redundancy Check）是数据链路层最常用的错误检测方法： + +**工作原理**： +1. 发送方根据数据和预定的生成多项式计算CRC校验码 +2. 将校验码附加在数据后面发送 +3. 接收方用相同的生成多项式对收到的数据进行校验 +4. 如果校验结果为0，说明传输正确；否则存在错误 + +**常用CRC标准**： +- CRC-8：8位校验码 +- CRC-16：16位校验码 +- CRC-32：32位校验码（以太网使用） + +#### 奇偶校验 + +**简单奇偶校验**： +- 奇校验：数据位中1的个数加上校验位为奇数 +- 偶校验：数据位中1的个数加上校验位为偶数 + +**海明码**： +- 能够检测双比特错误，纠正单比特错误 +- 使用多个校验位对数据的不同位进行保护 + +### 🎯 以太网协议详解 + +#### 以太网的发展历史 + +1. **标准以太网**（10 Mbps）：10BASE-5, 10BASE-2, 10BASE-T +2. **快速以太网**（100 Mbps）：100BASE-TX, 100BASE-FX +3. **千兆以太网**（1 Gbps）：1000BASE-T, 1000BASE-SX +4. **万兆以太网**（10 Gbps）：10GBASE-T, 10GBASE-SR + +#### 以太网的特点 + +- **无连接**：发送数据前不需要建立连接 +- **不可靠**：不保证数据一定能够到达目的地 +- **使用CSMA/CD协议**：载波侦听多路访问/冲突检测 + +### 🎯 以太网中的 CSMA/CD 协议详解 + +CSMA/CD（Carrier Sense Multiple Access with Collision Detection）为载波侦听多路访问/冲突检测，是以太网采用的一种重要机制。 + +#### CSMA/CD工作原理 + +1. **载波侦听（Carrier Sense）**： + - 发送前先侦听信道是否空闲 + - 如果信道忙，则等待直到信道变空闲 + - 如果信道空闲，立即开始发送 + +2. **多路访问（Multiple Access）**： + - 多个站点共享同一个传输媒体 + - 允许多个节点同时监听信道状态 + +3. **冲突检测（Collision Detection）**： + - 边发送边监听，检测是否发生冲突 + - 一旦检测到冲突，立即停止发送 + - 发送干扰信号通知其他站点 + +#### 二进制指数退避算法 + +当发生冲突时，CSMA/CD使用二进制指数退避算法： + +1. 确定参数k = min(重传次数, 10) +2. 在{0, 1, 2, ..., 2^k-1}中随机选择一个数r +3. 等待r×512位时间后重传 +4. 重传16次后放弃，向上层报告错误 + +### 🎯 交换技术 + +#### 网桥和交换机 + +**网桥的工作原理**： +- 存储转发：先接收完整帧，检查无误后再转发 +- 学习功能：通过学习建立MAC地址表 +- 过滤功能：根据MAC地址表决定是否转发 + +**交换机的优势**： +- 多端口网桥，每个端口独立的冲突域 +- 全双工通信，发送和接收同时进行 +- 多种帧转发方式：存储转发、直通交换、片段释放 + +#### VLAN技术 + +**VLAN（Virtual Local Area Network）虚拟局域网**： + +**VLAN的优点**： +- 减少广播域的大小 +- 增强网络安全性 +- 灵活的网络管理 +- 节约网络设备投资 + +**VLAN标记**： +- IEEE 802.1Q标准 +- 在以太网帧中插入4字节VLAN标记 +- 包含12位VLAN ID（支持4094个VLAN） + +### 🎯 数据链路层的三个基本问题 + +#### 封装成帧 + +**成帧的目的**： +- 将网络层的IP数据报封装成帧 +- 在帧的前面和后面分别添加帧头和帧尾 +- 帧头和帧尾包含重要的控制信息 + +**帧的边界标识方法**： +1. **字符计数法**：帧头包含帧的长度信息 +2. **字符填充法**：使用特殊字符作为帧的开始和结束标志 +3. **零比特填充法**：在数据中遇到特定模式时插入0比特 +4. **物理层编码违例**：利用物理层编码规则的违例来标识帧边界 + +#### 透明传输 + +**透明传输的含义**： +- 不管数据是什么样的比特组合，都能在数据链路上传输 +- 对上层来说，数据链路层是透明的 + +**字符填充法实现透明传输**： +- 发送端：在数据中出现控制字符前插入转义字符 +- 接收端：删除转义字符，恢复原始数据 + +**零比特填充法实现透明传输**： +- 发送端：在5个连续的1后面插入一个0 +- 接收端：在5个连续的1后面删除一个0 + +#### 差错检测 + +**差错产生的原因**： +- 噪声干扰 +- 信号衰减 +- 多径传播 +- 设备故障 + +**检错编码和纠错编码**： +- **检错编码**：只能检测错误，如奇偶校验码、CRC +- **纠错编码**：既能检测又能纠正错误，如海明码 + +### 🎯 停止等待协议 + +停止等待协议是最简单的流量控制和差错控制协议： + +#### 工作原理 + +1. 发送方发送一个数据帧后停止发送 +2. 等待接收方的确认帧 +3. 收到确认后才发送下一帧 +4. 如果在规定时间内没有收到确认，重传该帧 + +#### 四种情况处理 + +1. **无差错情况**：正常发送和确认 +2. **数据帧丢失**：发送方超时重传 +3. **确认帧丢失**：发送方超时重传，接收方丢弃重复帧 +4. **确认帧迟到**：发送方已重传并继续，丢弃迟到的确认 + +#### 为什么有了 MAC 地址还需要 IP 地址 + +如果我们只使用 MAC 地址进行寻址的话，我们需要路由器记住每个 MAC 地址属于哪一个子网，不然每一次路由器收到数据包时都要满世界寻找目的 MAC 地址。而我们知道 MAC 地址的长度为 48 位，也就是说最多总共有 2 的 48 次方个 MAC 地址，这就意味着每个路由器需要 256 T 的内存，这显然是不现实的。 + +和 MAC 地址不同，IP 地址是和地域相关的，在一个子网中的设备，我们给其分配的 IP 地址前缀都是一样的，这样路由器就能根据 IP 地址的前缀知道这个设备属于哪个子网，剩下的寻址就交给子网内部实现，从而大大减少了路由器所需要的内存。 + +#### 为什么有了 IP 地址还需要 MAC 地址 + +只有当设备连入网络时，才能根据他进入了哪个子网来为其分配 IP 地址，在设备还没有 IP 地址的时候或者在分配 IP 地址的过程中，我们需要 MAC 地址来区分不同的设备。 + +#### 私网地址和公网地址之间进行转换：同一个局域网内的两个私网地址，经过转换之后外面看到的一样吗 + +当采用静态或者动态转换时，由于一个私网 IP 地址对应一个公网地址，因此经过转换之后的公网 IP 地址是不同的；而采用端口复用方式的话，在一个子网中的所有地址都采用一个公网地址，但是使用的端口是不同的。 + +#### 以太网中的 CSMA/CD 协议 + +CSMA/CD 为载波侦听多路访问/冲突检测，是像以太网这种广播网络采用的一种机制，我们知道在以太网中多台主机在同一个信道中进行数据传输，CSMA/CD 很好的解决了共享信道通信中出现的问题，它的工作原理主要包括两个部分： + +- 载波监听：当使用 CSMA/CD 协议时，总线上的各个节点都在监听信道上是否有信号在传输，如果有的话，表明信道处于忙碌状态，继续保持监听，直到信道空闲为止。如果发现信道是空闲的，就立即发送数据。 +- 冲突检测：当两个或两个以上节点同时监听到信道空闲，便开始发送数据，此时就会发生碰撞（数据的传输延迟也可能引发碰撞）。当两个帧发生冲突时，数据帧就会破坏而失去了继续传输的意义。在数据的发送过程中，以太网是一直在监听信道的，当检测到当前信道冲突，就立即停止这次传输，避免造成网络资源浪费，同时向信道发送一个「冲突」信号，确保其它节点也发现该冲突。之后采用一种二进制退避策略让待发送数据的节点随机退避一段时间之后重新。 + +#### 数据链路层上的三个基本问题 + +**封装成帧**：将网络层传下来的分组前后分别添加首部和尾部，这样就构成了帧。首部和尾部的一个重要作用是帧定界，也携带了一些必要的控制信息，对于每种数据链路层协议都规定了帧的数据部分的最大长度。 + +**透明传输**：帧使用首部和尾部进行定界，如果帧的数据部分含有和首部和尾部相同的内容， 那么帧的开始和结束的位置就会判断错，因此需要在数据部分中出现有歧义的内容前边插入转义字符，如果数据部分出现转义字符，则在该转义字符前再加一个转义字符。在接收端进行处理之后可以还原出原始数据。这个过程透明传输的内容是转义字符，用户察觉不到转义字符的存在。 + +**差错检测**：目前数据链路层广泛使用循环冗余检验（CRC）来检查数据传输过程中是否产生比特差错。 + +#### PPP 协议 + +互联网用户通常需要连接到某个 ISP 之后才能接入到互联网，PPP（点对点）协议是用户计算机和 ISP 进行通信时所使用的数据链路层协议。点对点协议为点对点连接上传输多协议数据包提供了一个标准方法。该协议设计的目的主要是用来通过拨号或专线方式建立点对点连接发送数据，使其成为各种主机、网桥和路由器之间简单连接的一种解决方案。 + +PPP 协议具有以下特点： + +- PPP 协议具有动态分配 IP 地址的能力，其允许在连接时刻协商 IP 地址。 +- PPP 支持多种网络协议，例如 TCP/IP、NETBEUI 等。 +- PPP 具有差错检测能力，但不具备纠错能力，所以 PPP 是不可靠传输协议。 +- 无重传的机制，网络开销小，速度快。 +- PPP 具有身份验证的功能。 + +#### 为什么 PPP 协议不使用序号和确认机制 + +- IETF 在设计因特网体系结构时把齐总最复杂的部分放在 TCP 协议中，而网际协议 IP 则相对比较简单，它提供的是不可靠的数据包服务，在这种情况下，数据链路层没有必要提供比 IP 协议更多的功能。若使用能够实现可靠传输的数据链路层协议，则开销就要增大，这在数据链路层出现差错概率不大时是得不偿失的。 +- 即使数据链路层实现了可靠传输，但其也不能保证网络层的传输也是可靠的，当数据帧在路由器中从数据链路层上升到网络层后，仍有可能因为网络层拥塞而被丢弃。 +- PPP 协议在帧格式中有帧检验序列，对每一个收到的帧，PPP 都会进行差错检测，若发现差错，则丢弃该帧。 + + + +### 🎯 物理层 + +#### 物理层主要做什么事情 + +作为 OSI 参考模型最低的一层，物理层是整个开放系统的基础，该层利用传输介质为通信的两端建立、管理和释放物理连接，实现比特流的透明传输。物理层考虑的是怎样才能在连接各种计算机的传输媒体上传输数据比特流，其尽可能地屏蔽掉不同种类传输媒体和通信手段的差异，使物理层上面的数据链路层感觉不到这些差异，这样就可以使数据链路层只考虑完成本层的协议和服务，而不必考虑网络的具体传输媒体和通信手段是什么。 + +#### 主机之间的通信方式 + +**单工通信**：也叫单向通信，发送方和接收方是固定的，消息只能单向传输。例如采集气象数据、家庭电费，网费等数据收集系统，或者打印机等应用主要采用单工通信。 **半双工通信**：也叫双向交替通信，通信双方都可以发送消息，但同一时刻同一信道只允许单方向发送数据。例如传统的对讲机使用的就是半双工通信。 **全双工通信**：也叫双向同时通信，全双工通信允许通信双方同时在两个方向是传输，其要求通信双方都具有独立的发送和接收数据的能力。例如平时我们打电话，自己说话的同时也能听到对面的声音。 + +#### 通道复用技术 + +**频分复用（FDM，Frequency Division Multiplexing）** 频分复用将传输信道的总带宽按频率划分为若干个子频带或子信道，每个子信道传输一路信号。用户分到一定的频带后，在数据传输的过程中自始至终地占用这个频带。由于每个用户所分到的频带不同，使得传输信道在同一时刻能够支持不同用户进行数据传输，从而实现复用。除了传统意义上的 FDM 外，目前正交频分复用（OFDM）已在高速通信系统中得到广泛应用。 + +**时分复用（TDM，Time Division Multiplexing）** 顾名思义，时分复用将信道传输信息的时间划分为若干个时间片，每一个时分复用的用户在每一个 TDM 帧中占用固定时隙进行数据传输。用户所分配到的时隙是固定的，所以时分复用有时也叫做同步时分复用。这种分配方式能够便于调节控制，但是也存在缺点，当某个信道空闲时，其他繁忙的信道无法占用该空闲信道，因此会降低信道利用率。 + +**波分复用（WDM，Wavelength Division Multiplexing）** 在光通信领域通常按照波长而不是频率来命名，因为光的频率和波长具有单一对应关系，因此 WDM 本质上也是 FDM，光通信系统中，通常由光来运载信号进行传输，WDM 是在一条光纤上传输多个波长光信号，其将 1 根光纤看做多条「虚拟」光纤，每条「虚拟」光纤工作在不同的波长上，从而极大地提高了光纤的传输容量。 + +**码分复用（CDM，Code Division Multiplexing）** 码分复用是靠不同的编码来区分各路原始信号的一种复用方式，不同的用户使用相互正交的码字携带信息。由于码组相互正交，因此接收方能够有效区分不同的用户数据，从而实现每一个用户可以在同样的时间在同样的频带进行数据传输，频谱资源利用率高。其主要和各种多址接入技术相结合从而产生各种接入技术，包括无线和优先接入。 + +#### 几种常用的宽带接入技术 + +我们一般将速率超过 1 Mbps 的接入称为宽带接入，目前常用的宽带接入技术主要包括：ADSL 和 FTTx + LAN。 + +- ADSL + +ADSL 全称为非对称用户数字环路，是铜线宽带接入技术的一种。其非对称体现在用户上行和下行的传输速率不相等，一般上行速率较低，下行速率高。这种接入技术适用于有宽带业务需求的家庭用户或者中小型商务用户等。 + +- FTTx + LAN + +其中 FTTx 英文翻译为 Fiber To The X，这里的 X 指任何地方，我们可以理解为光纤可以接入到任何地方，而 LAN 指的是局域网。FTTx + LAN 是一种在接入网全部或部分采用光纤传输介质，构成光纤用户线路，从而实现用户高速上网的接入技术，其中用户速率可达 20 Mbps。这种接入技术投资规模小，网络拓展性强，网络可靠稳定，使得其应用广泛，目前是城市汇总较为普及的一种宽带接入技术。 + +其它还有 光纤同轴混合网（HFC）、光接入技术（有源和无源光纤系统）和无线接入技术等等。 + +--- + +## 📚 九、性能优化与监控 + +### 🎯 网络性能指标 + +#### 关键性能指标（KPI） + +**带宽（Bandwidth）**： +- 定义：网络在单位时间内能传输的最大数据量 +- 单位：bps（bits per second） +- 影响因素：物理链路容量、网络设备处理能力 + +**延迟（Latency）**： +- 定义：数据从源到目的地所需的时间 +- 组成：传播延迟、传输延迟、处理延迟、排队延迟 +- 单位：毫秒（ms） + +**吞吐量（Throughput）**： +- 定义：实际传输的数据速率 +- 与带宽关系：吞吐量 ≤ 带宽 +- 影响因素：网络拥塞、协议开销、设备性能 + +**丢包率（Packet Loss Rate）**： +- 定义：丢失数据包占总数据包的比例 +- 计算：丢包率 = 丢失包数 / 总发送包数 × 100% +- 影响：数据重传、应用性能下降 + +**抖动（Jitter）**： +- 定义：延迟变化的程度 +- 单位：毫秒（ms） +- 影响：实时应用（语音、视频）质量 + +#### 网络利用率计算 + +``` +网络利用率 = (实际传输流量 / 链路容量) × 100% + +示例： +链路容量：100 Mbps +实际流量：75 Mbps +网络利用率 = (75 / 100) × 100% = 75% +``` + +### 🎯 网络调优策略 + +#### TCP调优参数 + +**1. TCP窗口相关参数**： +```bash +# 调整TCP接收窗口大小 +net.core.rmem_default = 262144 +net.core.rmem_max = 16777216 + +# 调整TCP发送窗口大小 +net.core.wmem_default = 262144 +net.core.wmem_max = 16777216 + +# TCP自动窗口调优 +net.ipv4.tcp_window_scaling = 1 +``` + +**2. TCP连接相关参数**： +```bash +# 最大TCP连接数 +net.core.somaxconn = 65535 + +# TCP连接回收 +net.ipv4.tcp_tw_reuse = 1 +net.ipv4.tcp_fin_timeout = 30 + +# TCP Keepalive设置 +net.ipv4.tcp_keepalive_time = 600 +net.ipv4.tcp_keepalive_intvl = 30 +net.ipv4.tcp_keepalive_probes = 3 +``` + +**3. TCP拥塞控制算法**： +```bash +# 查看可用拥塞控制算法 +cat /proc/sys/net/ipv4/tcp_available_congestion_control + +# 设置拥塞控制算法 +net.ipv4.tcp_congestion_control = bbr +``` + +#### 网络设备优化 + +**交换机优化**： +- 启用流量控制（Flow Control） +- 配置QoS（Quality of Service）策略 +- 使用VLAN分割广播域 +- 启用端口聚合（Link Aggregation） + +**路由器优化**： +- 优化路由表结构 +- 配置负载均衡 +- 启用硬件加速 +- 调整缓冲区大小 + +**防火墙优化**： +- 规则优化和排序 +- 连接跟踪表调优 +- 会话超时配置 +- 硬件卸载功能 + +### 🎯 故障排查方法 + +#### 分层排查法 + +**1. 物理层排查**： +```bash +# 检查网卡状态 +ethtool eth0 + +# 查看接口统计信息 +cat /proc/net/dev + +# 检查链路状态 +ip link show +``` + +**2. 数据链路层排查**： +```bash +# 检查ARP表 +arp -a + +# 查看MAC地址表 +bridge fdb show + +# 检查网桥状态 +brctl show +``` + +**3. 网络层排查**： +```bash +# 路由表检查 +route -n +ip route show + +# 连通性测试 +ping -c 4 destination +traceroute destination + +# ICMP错误检查 +ping -M do -s 1472 destination +``` + +**4. 传输层排查**： +```bash +# 端口连接性测试 +telnet host port +nc -zv host port + +# TCP连接状态查看 +netstat -ant +ss -ant + +# UDP端口测试 +nc -u host port +``` + +#### 常用排查命令 + +**网络连接查看**： +```bash +# 查看所有连接 +netstat -tuln + +# 查看指定端口连接 +lsof -i :80 + +# 查看进程网络连接 +lsof -p PID + +# 查看网络连接统计 +ss -s +``` + +**流量分析**： +```bash +# 实时流量监控 +iftop -i eth0 + +# 网络流量统计 +vnstat -i eth0 + +# 详细流量分析 +tcpdump -i eth0 -n host 192.168.1.1 +``` + +**性能测试**： +```bash +# 网络带宽测试 +iperf3 -s # 服务端 +iperf3 -c server_ip # 客户端 + +# HTTP性能测试 +ab -n 1000 -c 10 http://example.com/ + +# 网络延迟测试 +mtr destination +``` + +### 🎯 网络监控体系 + +#### 监控指标分类 + +**1. 基础设施监控**： +- 设备状态（CPU、内存、温度） +- 接口状态（up/down、速率） +- 电源状态 +- 风扇状态 + +**2. 流量监控**： +- 接口流量（输入/输出字节数） +- 包数统计 +- 错误包统计 +- 广播包统计 + +**3. 性能监控**： +- 延迟监控 +- 丢包率监控 +- 抖动监控 +- 可用性监控 + +**4. 安全监控**： +- 异常流量监控 +- 攻击检测 +- 访问控制监控 +- 安全事件记录 + +#### 监控工具和协议 + +**SNMP（Simple Network Management Protocol）**： +```bash +# SNMP查询示例 +snmpwalk -v2c -c public 192.168.1.1 1.3.6.1.2.1.1 + +# 获取接口流量 +snmpget -v2c -c public 192.168.1.1 1.3.6.1.2.1.2.2.1.10.1 +``` + +**Syslog**： +- 集中化日志收集 +- 实时日志监控 +- 日志分析和告警 +- 日志归档和查询 + +**NetFlow/sFlow**： +- 流量分析 +- 行为分析 +- 容量规划 +- 安全分析 + +#### 监控系统架构 + +**分布式监控架构**： +``` +[网络设备] → [采集器] → [数据存储] → [分析展示] + ↓ ↓ ↓ ↓ + SNMP/ Collector Time Series Dashboard + Syslog Database & Alert +``` + +**关键组件**： +- **数据采集**：SNMP、Syslog、NetFlow采集器 +- **数据存储**：时序数据库（InfluxDB、Prometheus） +- **数据处理**：实时计算、数据聚合、异常检测 +- **可视化**：Dashboard、图表、拓扑图 +- **告警系统**：阈值告警、趋势告警、智能告警 + +### 🎯 性能分析方法 + +#### 瓶颈识别 + +**1. CPU瓶颈识别**： +```bash +# 查看网络相关CPU使用 +top -p $(pidof -x networking-process) + +# 查看软中断使用情况 +cat /proc/softirqs + +# 网络接口队列查看 +cat /proc/interrupts | grep eth +``` + +**2. 内存瓶颈识别**： +```bash +# 网络缓冲区使用情况 +cat /proc/net/sockstat + +# TCP内存使用 +cat /proc/net/tcp_mem + +# 网络设备缓冲区 +cat /proc/net/dev_mcast +``` + +**3. 磁盘I/O影响**： +```bash +# 查看I/O等待 +iostat -x 1 + +# 网络相关I/O +iotop -a -o -d 1 +``` + +#### 容量规划 + +**流量增长预测**： +- 历史流量数据分析 +- 业务增长趋势预测 +- 季节性变化考虑 +- 突发流量预估 + +**设备容量评估**： +- 当前设备利用率 +- 性能余量评估 +- 扩容阈值设定 +- 设备生命周期考虑 + +**网络架构优化**： +- 负载均衡策略 +- 冗余链路规划 +- QoS策略调整 +- 缓存策略优化 + +### 🎯 实用优化技巧 + +#### 应用层优化 + +**HTTP优化**： +- 启用HTTP/2 +- 开启GZIP压缩 +- 使用CDN加速 +- 合理设置缓存策略 +- 减少HTTP请求数量 + +**DNS优化**： +- 使用本地DNS缓存 +- 配置多个DNS服务器 +- DNS预加载 +- 减少DNS查询次数 + +#### 系统层优化 + +**网络栈调优**： +```bash +# 增加网络缓冲区 +echo 'net.core.netdev_max_backlog = 5000' >> /etc/sysctl.conf + +# 增加TCP连接队列长度 +echo 'net.core.somaxconn = 65535' >> /etc/sysctl.conf + +# 启用TCP快速回收 +echo 'net.ipv4.tcp_tw_reuse = 1' >> /etc/sysctl.conf + +# 应用配置 +sysctl -p +``` + +**中断优化**： +```bash +# 查看网卡中断分布 +cat /proc/interrupts | grep eth0 + +# 手动分配网卡中断到不同CPU +echo 2 > /proc/irq/24/smp_affinity + +# 使用irqbalance自动平衡 +service irqbalance start +``` + +### 🎯 故障预防和维护 + +#### 预防性维护 + +**定期检查项目**： +- 设备健康状态检查 +- 链路质量检测 +- 性能基线更新 +- 安全漏洞扫描 -## 参考与感谢 +**备份和恢复**： +- 配置文件备份 +- 拓扑图更新 +- 应急预案制定 +- 故障恢复演练 -- 《HTTP 权威指南》 -- https://arch-long.cn/articles/network/OSI模型TCPIP协议栈.html -- https://blog.csdn.net/qq_32998153/article/details/79680704 +#### 变更管理 +**网络变更流程**： +1. 变更申请和评估 +2. 变更计划制定 +3. 变更实施和监控 +4. 变更结果验证 +5. 变更记录归档 +**回滚策略**： +- 变更前状态记录 +- 快速回滚方案 +- 回滚验证步骤 +- 紧急联系机制 -![end.png](https://i.loli.net/2020/04/16/UWdFN1aJlmOMsfS.png) \ No newline at end of file diff --git a/docs/interview/README.md b/docs/interview/README.md new file mode 100644 index 0000000000..e69de29bb2 diff --git a/docs/interview/RPC-FAQ.md b/docs/interview/RPC-FAQ.md new file mode 100644 index 0000000000..084e710699 --- /dev/null +++ b/docs/interview/RPC-FAQ.md @@ -0,0 +1,1241 @@ +--- +title: RPC框架/分布式通信面试题大全 +date: 2024-12-15 +tags: + - RPC + - Dubbo + - gRPC + - Interview +categories: Interview +--- + +![](https://img.starfish.ink/common/faq-banner.png) + + RPC作为分布式系统的**核心通信机制**，是Java后端面试的**必考重点**。从基础原理到框架实现，从性能优化到服务治理，每个知识点都可能成为面试的关键。本文档将**RPC核心技术**整理成**系统化知识体系**，涵盖协议设计、序列化机制、负载均衡等关键领域，助你在面试中游刃有余！ + + + RPC 面试，围绕着这么几个核心方向准备： + + - **RPC基础原理**（调用流程、协议设计、网络通信、代理机制） + - **序列化与协议**（Protobuf、JSON、Hessian、自定义协议设计） + - **服务发现与治理**（注册中心、负载均衡、熔断降级、限流机制） + - **性能优化**（连接复用、异步调用、批量处理、网络调优） + - **主流框架对比**（Dubbo、gRPC、Thrift、Spring Cloud对比分析） + - **高级特性**（泛化调用、多版本支持、灰度发布、监控埋点） + - **实战应用**（架构设计、问题排查、性能调优、最佳实践） + +## 🗺️ 知识导航 + +### 🏷️ 核心知识分类 + +1. **基础与原理**：RPC定义、调用流程、网络通信、代理实现 +2. **协议与序列化**：传输协议、序列化方案、编解码优化 +3. **服务治理**：注册发现、负载均衡、容错机制、配置管理 +4. **性能与调优**：连接管理、异步处理、批量优化、监控指标 +5. **框架对比**：Dubbo、gRPC、Thrift等主流框架特性分析 +6. **工程实践**：架构设计、问题排查、运维监控、最佳实践 + +--- + +## 🧠 一、RPC基础与原理 + + **核心理念**：RPC让分布式调用像本地调用一样简单，通过网络通信、序列化、代理机制实现远程服务透明调用。 + +### 🎯 什么是 RPC？它和 HTTP/REST 有什么区别？ + +> RPC 是远程过程调用，核心目标是让调用远程服务像本地函数调用一样简单。它屏蔽了网络通信、序列化、反序列化等细节。和 HTTP/REST 相比，RPC 更偏向于内部服务间通信，通常使用二进制序列化（比如 Protobuf），性能更高，支持流式通信；而 REST 基于 HTTP + JSON，跨语言、可读性和调试友好，更适合对外接口。 +> 在实际项目里，我们常常 **对内用 RPC（gRPC）提升性能，对外用 REST 保证通用性**。 + +**什么是 RPC？** + +- **RPC（Remote Procedure Call，远程过程调用）** 是一种 **像调用本地函数一样调用远程服务的方法**。 +- 本质：屏蔽网络通信细节（序列化、传输、反序列化），让开发者像调用本地函数一样调用远程函数。 +- 核心步骤： + 1. **客户端 Stub**：将调用方法和参数序列化 + 2. **网络传输**：通过 TCP/HTTP/HTTP2 传输到远程服务 + 3. **服务端 Stub**：反序列化参数，执行实际逻辑 + 4. **结果返回**：序列化结果，通过网络返回给客户端 +- **关键点**： + - **通信协议**：如HTTP/2（gRPC）、TCP（Dubbo）。 + - **序列化方式**：如JSON、Protobuf、Thrift。 + - **服务治理**：负载均衡、熔断、限流等 + +👉 面试一句话总结：**RPC 是一种通信协议和调用方式，目标是“透明调用远程服务”。** + +**RPC 和 HTTP/REST 的区别** + +| 维度 | RPC | HTTP/REST | +| -------------- | ----------------------------------------------- | --------------------------------------------------- | +| **定义方式** | 通过接口（IDL，如gRPC 的 proto）定义服务 | 通过 URL + HTTP 方法（GET/POST/PUT/DELETE）定义 API | +| **传输协议** | TCP、HTTP/2（如 gRPC） | HTTP/1.1 或 HTTP/2 | +| **数据格式** | 高效序列化协议（Protobuf、Thrift、Avro） | 一般是 JSON（文本格式，易读但性能差） | +| **性能** | 高性能，二进制序列化，占用带宽小，延迟低 | 较低性能，JSON 解析慢，报文体积大 | +| **通信模式** | 多样（同步、异步、单向、流式） | 主要是请求-响应 | +| **可读性** | 抽象层高，调试工具少 | URL+JSON，可读性强，调试方便 | +| **跨语言支持** | IDL 定义，多语言 Stub 自动生成（gRPC 跨语言强） | HTTP/JSON 天然跨语言 | +| **适用场景** | 内部微服务调用，高性能、低延迟场景 | 对外 API，跨语言、跨团队、跨系统的服务调用 | + +> 话术： +> +> 1. RPC是解决分布式系统中**远程服务调用复杂性**的工具，核心目标是让开发者像调用本地方法一样调用远程服务，隐藏网络通信细节。（**一句话定义**） +> +> 2. RPC框架包含三个核心模块：代理层（生成接口代理）、序列化层（Protobuf/JSON转换）、网络传输层（Netty实现）。 +> +> 调用流程是：客户端代理封装请求→序列化为二进制→经TCP/HTTP2传输→服务端反序列化→执行真实方法→结果原路返回。关键技术是动态代理屏蔽远程调用细节，配合长连接复用提升性能。（**核心原理拆解（展示技术深度）**） +> +> 3. 相比直接使用HTTP（**对比延伸（突出思考广度）**）： +> +> - RPC优势是性能更高（二进制协议省带宽）、开发更高效（IDL生成代码）、内置服务治理（熔断/负载均衡） +> - HTTP优势是通用性强（浏览器直接支持）、调试更方便 +> - 在微服务内部通信选RPC（如Dubbo），开放API用HTTP（如SpringCloud OpenFeign）。 +> - 腾讯的tRPC通过插件化架构解决协议兼容问题，而gRPC强在跨语言支持。 +> +> 4. 在电商订单系统中，我用Dubbo实现库存服务调用（**实战结合（证明落地能力）**）： +> +> - 问题：HTTP调用库存接口QPS仅2000，超时率15% +> - 方案：改用Dubbo+Protobuf，Nacos服务发现，随机负载均衡 +> - 难点：解决序列化兼容性（添加@Adaptive注解） +> - 结果：QPS提升到12000，超时率降至0.2%，GC次数减少60% + + + +### 🎯 RPC的基本原理是什么？ + +**基本原理流程** + +一次完整的 RPC 调用，大致分为以下几个步骤： + +1. **客户端调用** + - 客户端调用本地代理对象（Stub/Proxy），就像调用本地方法。 + - 代理负责将方法名、参数等信息打包。 +2. **序列化** + - 把方法调用的信息（类名、方法名、参数等）序列化成字节流。 +3. **网络传输** + - 通过底层传输协议（TCP/HTTP/HTTP2 等）发送给服务端。 +4. **服务端接收** + - 服务端接收到请求后，反序列化数据，解析出方法和参数。 +5. **方法调用** + - 调用本地对应的服务实现，并获得结果。 +6. **序列化返回值** + - 把执行结果序列化，再通过网络传回客户端。 +7. **客户端接收结果** + - 反序列化后，代理对象将结果返回给调用者。 + +👉 总结：**本地调用 → 序列化 → 网络传输 → 远程执行 → 返回结果**。 + +**核心要素** + +- **通信协议**：决定客户端和服务端如何交互（TCP、HTTP/2、gRPC 使用 HTTP/2）。 +- **序列化机制**：把对象转成字节流，常见 JSON、Protobuf、Avro 等。 +- **服务发现**：客户端如何找到服务端地址（Zookeeper、Nacos 等）。 +- **代理机制**：屏蔽远程调用细节，让调用像本地方法一样。 + +**💻 代码示例**： + +```java +// RPC调用示例 +public class RPCExample { + + // 定义服务接口 + public interface UserService { + User getUserById(Long id); + List getUsers(UserQuery query); + } + + // 客户端调用 + public class UserController { + + @Reference // RPC注解 + private UserService userService; + + public User getUser(Long id) { + // 像本地调用一样使用远程服务 + return userService.getUserById(id); + } + } + + // 服务端实现 + @Service // 暴露RPC服务 + public class UserServiceImpl implements UserService { + + @Override + public User getUserById(Long id) { + return userRepository.findById(id); + } + } +} +``` + + + +### 🎯 为什么我们要用RPC? + +在分布式系统架构中，**RPC是一种核心通信机制，用于解决跨进程、跨机器的函数 / 方法调用问题。其存在的核心价值在于将复杂的分布式系统拆解为可协作的服务单元**，同时尽可能让开发者像调用本地函数一样使用远程服务。 + +以下是使用 RPC 的核心原因及典型场景： + +**一、分布式架构的必然选择** + +1. **服务拆分与微服务化** + + - **单体应用的瓶颈**：传统单体架构中，所有功能模块耦合在一个进程内，难以扩展、维护和迭代。 + + - **分布式拆分的需求**：将系统拆分为独立部署的服务（如用户服务、订单服务、支付服务），每个服务负责单一业务领域，通过 RPC 实现跨服务协作。 + + - **示例**：电商系统中，前端请求用户服务查询用户信息，用户服务通过 RPC 调用订单服务获取历史订单数据。 + +2. **资源隔离与弹性扩展** + + - **按需扩展特定服务**：不同服务的负载可能差异显著（如促销期间订单服务压力远高于用户服务），通过 RPC 解耦后，可独立对高负载服务扩容。 + + - **故障隔离**：某个服务故障不会导致整个系统崩溃，仅影响依赖该服务的功能模块（需配合熔断、重试等机制）。 + +**二、跨技术栈协作的桥梁** + +1. **多语言混合开发** + + - 不同服务可采用最适合的语言实现（如 Java 用于业务逻辑、Go 用于高并发场景、Python 用于数据分析），通过 RPC 屏蔽语言差异。 + + - **示例**：Java 编写的网关服务通过 RPC 调用 Go 编写的库存服务，获取商品库存信息。 + +2. **遗留系统集成** + + - 新老系统并存时，通过 RPC 为遗留系统提供统一接口，避免重构成本。 + + - **示例**：用 Node.js 开发新前端系统，通过 RPC 调用 COBOL 编写的核心账务系统接口。 + +**三、高性能与透明化的远程调用** + +1. **接近本地调用的开发体验** + + - RPC 框架通过动态代理、代码生成等技术，将远程调用封装为本地函数调用形式，开发者无需关注网络细节（如 Socket 编程、数据序列化）。 + + - 伪代码示例： + + ```java + // 本地调用风格的RPC + User user = userService.getUser(123); // 实际通过网络调用远程服务 + ``` + +2. **比 HTTP 更高效的通信协议** + + - 多数 RPC 框架采用二进制协议（如 Protobuf、Thrift），相比 JSON/XML 格式的 HTTP 请求，**传输体积更小、解析更快**，适合高频、大数据量场景。 + + - 性能对比： + + | **协议** | 传输体积 | 解析耗时 | 典型场景 | + | ---------------- | -------- | -------- | ---------------- | + | REST（JSON） | 100KB | 10ms | 通用 Web 服务 | + | gRPC（Protobuf） | 30KB | 2ms | 微服务间高频调用 | + +**四、典型应用场景** + +1. **微服务架构中的服务间通信** + + - 微服务架构中，每个服务通过 RPC 调用上下游服务，形成复杂的调用链路。 + + - **案例**：Netflix 的微服务体系通过 Eureka（服务注册）+ Ribbon（负载均衡）+ Feign（RPC 客户端）实现跨服务通信。 + +2. **云服务与分布式计算** + + - 云计算平台（如 AWS、阿里云）通过 RPC 提供 API 接口（如 EC2 实例管理、S3 存储操作）。 + + - 分布式计算框架（如 Hadoop、Spark）通过 RPC 协调节点间任务调度与数据传输。 + +3. **实时数据处理与流计算** + + - 实时系统中，不同组件（如消息队列、计算引擎、存储系统）通过 RPC 传递实时数据。 + + - **案例**：Kafka Streams 通过 RPC 将实时数据流分发到不同计算节点进行处理。 + +4. **跨数据中心 / 跨地域调用** + + - 全球化业务中，服务部署在多个数据中心，通过 RPC 实现异地容灾或就近访问。 + + - **挑战**：需解决跨地域网络延迟（如通过边缘节点缓存热点数据）。 + +**五、与其他通信方式的对比** + +| **维度** | **RPC** | **REST/HTTP** | **消息队列（如 Kafka）** | +| ------------ | ----------------------- | ------------------------ | ------------------------ | +| **通信模型** | 同步调用（请求 - 响应） | 同步调用（RESTful 风格） | 异步消息传递 | +| **实时性** | 高（适合即时响应场景） | 中（受限于 HTTP 协议） | 低（适合异步处理） | +| **数据格式** | 二进制（高效） | 文本（JSON/XML） | 自定义（二进制 / 文本） | +| **适用场景** | 微服务间强依赖调用 | 开放 API、跨团队协作 | 异步任务、流量削峰 | + +**六、RPC 的核心价值** + +1. **架构层面**：支撑分布式系统的服务拆分与协作，提升可扩展性和可维护性。 +2. **开发层面**：屏蔽网络复杂性，降低分布式开发门槛，允许混合技术栈。 +3. **性能层面**：提供比传统 HTTP 更高效的通信方式，满足高并发、低延迟需求。 + +**选择建议**：若需要**强一致性、实时响应的服务间调用**，优先选择 RPC；若需要**开放 API、跨团队 / 跨平台协作**，则更适合 REST/HTTP；若业务场景以**异步解耦**为主，可结合消息队列使用。 + + + +### 🎯 RPC与HTTP、消息队列的区别？ + +不同通信方式的特点对比： + +**详细对比分析**： + +| 特性 | RPC | HTTP | 消息队列 | +|------|-----|------|----------| +| **通信方式** | 同步调用 | 同步请求 | 异步消息 | +| **调用方式** | 方法调用 | 资源访问 | 消息发送 | +| **性能** | 高性能 | 中等 | 高吞吐 | +| **实时性** | 实时 | 实时 | 准实时 | +| **耦合度** | 较高 | 中等 | 低 | +| **复杂度** | 中等 | 简单 | 较高 | +| **适用场景** | 微服务调用 | Web API | 解耦异步 | + +**使用场景选择**： +- **RPC**：微服务间同步调用、高性能要求 +- **HTTP**：对外API、跨语言调用、简单场景 +- **消息队列**：异步处理、系统解耦、削峰填谷 + + + +### 🎯 动态代理在RPC中的作用？ + +> 在 RPC 框架中，动态代理的作用是 **屏蔽远程调用的复杂性**。 +> 调用方调用接口方法时，代理会拦截调用，把方法名、参数封装成 RPC 请求，通过网络发送给服务端，执行完再把结果返回。 +> 这样开发者感觉就是调用本地方法，实际底层完成了 **序列化、传输、服务发现、负载均衡** 等工作。 +> gRPC、Dubbo 等框架都会用动态代理来实现客户端 Stub。 + +动态代理是RPC框架的核心技术： + +**作用机制**： +- 屏蔽网络通信细节 +- 提供透明的远程调用体验 +- 统一处理序列化和协议 +- 实现AOP功能（监控、重试等） + +**实现方式**： +1. **JDK动态代理**：基于接口的代理 +2. **CGLIB代理**：基于类的代理 +3. **字节码生成**：编译时生成代理类 + +**💻 代码示例**： + +```java +// RPC动态代理实现 +public class RPCProxyFactory { + + public static T createProxy(Class serviceInterface, + String serverAddress) { + + return (T) Proxy.newProxyInstance( + serviceInterface.getClassLoader(), + new Class[]{serviceInterface}, + new RPCInvocationHandler(serviceInterface, serverAddress) + ); + } +} + +// 代理调用处理器 +public class RPCInvocationHandler implements InvocationHandler { + + private final Class serviceInterface; + private final String serverAddress; + private final RPCClient rpcClient; + + @Override + public Object invoke(Object proxy, Method method, Object[] args) { + // 构造RPC请求 + RPCRequest request = RPCRequest.builder() + .serviceName(serviceInterface.getName()) + .methodName(method.getName()) + .parameterTypes(method.getParameterTypes()) + .parameters(args) + .build(); + + // 发送请求并获取响应 + RPCResponse response = rpcClient.call(serverAddress, request); + + if (response.hasException()) { + throw new RuntimeException(response.getException()); + } + + return response.getResult(); + } +} +``` + + + +### 🎯 RPC需要解决的三个问题？ + +**一、Call ID 映射（函数标识与路由）** + +**问题本质** + +本地调用通过函数指针直接寻址，而远程调用中客户端与服务端处于不同地址空间，需建立**函数到唯一标识的映射关系**，确保服务端准确识别目标函数。 + +**解决方案** + +1. **唯一标识符（Call ID）** + + - 为每个函数分配全局唯一 ID（如整数、字符串或 UUID），例如： + + ```python + # 服务端映射表示例 + { + "userService.queryUser": 1001, # 字符串标识 + "orderService.createOrder": 2002 + } + ``` + + - 客户端通过该 ID 指定调用目标，服务端通过 ID 查找对应函数实现。 + +2. **动态注册与发现** + + - 服务启动时向注册中心（如 Consul、Nacos）注册函数 ID 与地址的映射关系。 + - 客户端通过注册中心获取服务列表及函数 ID 路由规则，实现动态寻址。 + +**技术挑战** + +- **版本兼容**：函数升级时需保留旧 Call ID 或提供兼容映射，避免客户端请求失效。 +- **跨语言映射**：不同语言开发的客户端与服务端需统一 ID 规范（如 Thrift 通过 IDL 文件生成一致的 ID）。 + +**二、序列化与反序列化（数据格式转换）** + +**问题本质** + +跨进程通信无法直接传递内存对象，且可能存在语言差异（如 Java 对象与 Go 结构体），需将数据结构转换为**通用字节流格式**，确保跨语言、跨平台解析。 + +**解决方案** + +1. **序列化协议选择** + + | **协议** | **特点** | **适用场景** | + | -------- | -------------------------------------- | ---------------------- | + | JSON | 可读性强，解析效率低 | 轻量级服务、浏览器交互 | + | Protobuf | 二进制格式，高效压缩，支持自动生成代码 | 高性能、大数据量场景 | + | Thrift | 多语言支持，通过 IDL 定义数据结构 | 跨语言微服务架构 | + | Avro | 模式动态演变，适合数据格式频繁变更场景 | 日志系统、实时数据管道 | + +2. **对象与字节流转换** + + - **客户端**：将参数对象序列化为字节流（如 Java 的`ObjectOutputStream`、Go 的`json.Marshal`）。 + - **服务端**：将字节流反序列化为本地对象（如 Python 的`json.loads`、C++ 的 Protobuf 解析器）。 + +**技术挑战** + +- **性能瓶颈**：高频调用场景下，序列化 / 反序列化可能成为性能短板（如 JSON 解析耗时高于 Protobuf）。 +- **数据兼容性**：字段增减或类型变更时，需确保新旧协议兼容（如 Protobuf 的可选字段、JSON 的默认值处理）。 + +**三、网络传输（数据通信与可靠性）** + +**问题本质** + +需建立客户端与服务端的**可靠数据传输通道**，解决网络延迟、丢包、连接管理等问题，确保 Call ID 与序列化数据准确传输。 + +**解决方案** + +1. **传输协议选择** + - **TCP**：面向连接，提供可靠传输（如 gRPC 基于 HTTP/2，Netty 自定义协议）。 + - **UDP**：无连接，适合实时性要求高但允许少量丢包的场景（如游戏状态同步）。 + - **HTTP/2**：多路复用、头部压缩，适合 RESTful 风格的 RPC（如 Spring Cloud Feign）。 +2. **网络层核心组件** + - **客户端负载均衡**：通过轮询、随机或加权最小连接等策略选择目标服务节点（如 Ribbon、Spring Cloud LoadBalancer）。 + - **连接池管理**：复用网络连接，减少 TCP 三次握手开销（如 Hystrix 的连接池配置）。 + - **超时与重试**：设置请求超时时间（如 gRPC 默认 1 秒），失败后按策略重试（如指数退避）。 + +**技术挑战** + +- **网络拥塞**：高并发场景下可能导致传输延迟陡增，需通过流量控制（如 TCP 滑动窗口）或服务降级缓解。 +- **防火墙与 NAT 穿透**：跨网络环境调用时，需解决端口限制或使用反向代理（如 Ngrok）。 + +**总结：RPC 核心技术栈** + +| **问题维度** | **关键技术** | **典型工具 / 框架** | +| -------------- | --------------------------------- | --------------------------------- | +| Call ID 映射 | 唯一 ID 生成、注册中心、动态路由 | Consul、Nacos、Apache ZooKeeper | +| 序列化反序列化 | Protobuf、JSON、Thrift、Avro | Google Protobuf、Apache Thrift | +| 网络传输 | TCP/UDP、HTTP/2、负载均衡、连接池 | gRPC、Netty、Spring Cloud Netflix | + +**设计目标**：通过上述技术的有机组合，实现**透明化远程调用**（调用者无需感知网络细节）、**高性能通信**（低延迟、高吞吐）和**强兼容性**（跨语言、跨平台）。实际应用中需根据业务场景（如实时性、数据量、语言栈）选择合适的技术方案，平衡开发成本与系统性能。 + + + +### 🎯 实现高可用RPC框架需要考虑到的问题 + +- 既然系统采用分布式架构，那一个服务势必会有多个实例，要解决**如何获取实例的问题**。所以需要一个服务注册中心，比如在Dubbo中，就可以使用Zookeeper作为注册中心，在调用时，从Zookeeper获取服务的实例列表，再从中选择一个进行调用； +- 如何选择实例呢？就要考虑负载均衡，例如dubbo提供了4种负载均衡策略； +- 如果每次都去注册中心查询列表，效率很低，那么就要加缓存； +- 客户端总不能每次调用完都等着服务端返回数据，所以就要支持异步调用； +- 服务端的接口修改了，老的接口还有人在用，这就需要版本控制； +- 服务端总不能每次接到请求都马上启动一个线程去处理，于是就需要线程池； + + + +### 🎯 一次完整的 RPC 流程？ + +1. **代理拦截**：客户端代理拦截本地调用，解析方法和参数。 +2. **序列化**：将对象转为字节流（如 Protobuf）。 +3. **网络传输**：通过 TCP/HTTP2 发送，处理粘包、负载均衡。 +4. **服务端解析**：拆包、反序列化，路由到目标方法。 +5. **结果返回**：序列化响应，逆向流程返回客户端。 +6. **关键技术**：协议设计、超时重试、流控等。 + +![](https://learn.lianglianglee.com/%E4%B8%93%E6%A0%8F/%E6%9E%B6%E6%9E%84%E8%AE%BE%E8%AE%A1%E9%9D%A2%E8%AF%95%E7%B2%BE%E8%AE%B2/assets/Ciqc1GABbyeAWysgAAGQtM8Kx4Q574.png) + +--- + + + +## 📦 二、序列化与协议 + + **核心理念**：高效的序列化机制和协议设计是RPC性能的关键，需要平衡速度、大小和兼容性。 + +### 🎯 常见的序列化方案有哪些？各自的特点？ + +主流序列化技术对比： + +**1. JDK序列化**： +- **优点**：Java原生支持，使用简单 +- **缺点**：性能差，序列化数据大，版本兼容性问题 +- **适用场景**：简单场景，内部系统 + +**2. JSON序列化**： +- **优点**：可读性好，跨语言支持 +- **缺点**：性能一般，数据冗余 +- **适用场景**：Web API，调试友好场景 + +**3. Protobuf**： +- **优点**：性能优秀，数据紧凑，向后兼容 +- **缺点**：需要定义.proto文件，学习成本 +- **适用场景**：高性能场景，跨语言调用 + +**4. Hessian**： +- **优点**：性能好，支持多语言 +- **缺点**：复杂类型支持有限 +- **适用场景**：Dubbo等RPC框架 + +**5. Kryo**： +- **优点**：性能极佳，序列化数据小 +- **缺点**：只支持Java，版本兼容性问题 +- **适用场景**：Java内部系统，性能要求极高 + +**💻 性能对比数据**： + +| 序列化方案 | 序列化速度 | 反序列化速度 | 数据大小 | 跨语言 | +|-----------|------------|--------------|----------|--------| +| JDK | 慢 | 慢 | 大 | ❌ | +| JSON | 中等 | 中等 | 中等 | ✅ | +| Protobuf | 快 | 快 | 小 | ✅ | +| Hessian | 快 | 快 | 小 | ✅ | +| Kryo | 极快 | 极快 | 极小 | ❌ | + +### 🎯 如何设计一个高效的RPC协议？ + +RPC协议设计的核心要素： + +**协议结构设计**： +```java +// RPC协议格式 ++-------+-------+-------+-------+ +| Magic | Ver | Type | Length| // 协议头 ++-------+-------+-------+-------+ +| RequestId | // 请求ID（8字节） ++-----------------------+ +| Header Length | // 头部长度 ++-----------------------+ +| Body Length | // 消息体长度 ++-----------------------+ +| Headers... | // 扩展头部 ++-----------------------+ +| Body... | // 消息体 ++-----------------------+ +``` + +**设计原则**： +1. **固定头部**：便于快速解析 +2. **版本控制**：支持协议升级 +3. **类型标识**：区分请求、响应、心跳等 +4. **长度字段**：支持变长消息 +5. **扩展性**：预留扩展字段 + +**💻 协议实现示例**： + +```java +// RPC协议定义 +public class RPCProtocol { + + public static final int MAGIC_NUMBER = 0xCAFEBABE; + public static final byte VERSION = 1; + + // 消息类型 + public static final byte TYPE_REQUEST = 1; + public static final byte TYPE_RESPONSE = 2; + public static final byte TYPE_HEARTBEAT = 3; + + // 协议编码 + public static ByteBuf encode(RPCMessage message) { + ByteBuf buffer = Unpooled.buffer(); + + // 协议头 + buffer.writeInt(MAGIC_NUMBER); + buffer.writeByte(VERSION); + buffer.writeByte(message.getType()); + buffer.writeShort(0); // flags预留 + + // 请求ID + buffer.writeLong(message.getRequestId()); + + // 序列化消息体 + byte[] body = serialize(message.getBody()); + buffer.writeInt(body.length); + buffer.writeBytes(body); + + return buffer; + } + + // 协议解码 + public static RPCMessage decode(ByteBuf buffer) { + // 检查魔数 + int magic = buffer.readInt(); + if (magic != MAGIC_NUMBER) { + throw new IllegalArgumentException("Invalid magic number"); + } + + // 读取协议头 + byte version = buffer.readByte(); + byte type = buffer.readByte(); + short flags = buffer.readShort(); + long requestId = buffer.readLong(); + + // 读取消息体 + int bodyLength = buffer.readInt(); + byte[] body = new byte[bodyLength]; + buffer.readBytes(body); + + return RPCMessage.builder() + .type(type) + .requestId(requestId) + .body(deserialize(body)) + .build(); + } +} +``` + +--- + + + +## 🏗️ 三、服务治理 + + **核心理念**：服务治理是RPC框架的高级特性，包括服务发现、负载均衡、容错处理等企业级功能。 + +### 🎯 服务注册与发现机制是什么？ + +服务注册发现是微服务架构的基础： + +**核心功能**： +- **服务注册**：服务启动时向注册中心注册 +- **服务发现**：客户端从注册中心获取服务列表 +- **健康检查**：监控服务实例健康状态 +- **负载均衡**：在多个实例间分配请求 + +**主流注册中心**： +1. **Zookeeper**：强一致性，复杂度高 +2. **Eureka**：AP模式，简单易用 +3. **Consul**：功能丰富，支持多数据中心 +4. **Nacos**：阿里开源，功能全面 + +**💻 代码示例**： + +```java +// 服务注册实现 +@Component +public class ServiceRegistry { + + private final ZooKeeperClient zkClient; + + // 注册服务 + public void registerService(ServiceInfo serviceInfo) { + String servicePath = "/rpc/services/" + serviceInfo.getServiceName(); + String instancePath = servicePath + "/" + serviceInfo.getInstanceId(); + + // 创建临时顺序节点 + zkClient.createEphemeralSequential(instancePath, + JSON.toJSONString(serviceInfo).getBytes()); + + // 注册shutdown hook + Runtime.getRuntime().addShutdownHook(new Thread(() -> { + zkClient.delete(instancePath); + })); + } + + // 发现服务 + public List discoverServices(String serviceName) { + String servicePath = "/rpc/services/" + serviceName; + + List children = zkClient.getChildren(servicePath); + return children.stream() + .map(child -> { + byte[] data = zkClient.getData(servicePath + "/" + child); + return JSON.parseObject(new String(data), ServiceInfo.class); + }) + .collect(Collectors.toList()); + } +} + +// 服务信息定义 +@Data +public class ServiceInfo { + private String serviceName; + private String instanceId; + private String host; + private int port; + private Map metadata; + private long timestamp; +} +``` + + + +### 🎯 负载均衡算法有哪些？如何实现？ + +常见负载均衡算法及其实现： + +**1. 轮询（Round Robin）**： +- 按顺序依次分配请求 +- 简单公平，但不考虑服务器性能差异 + +**2. 加权轮询（Weighted Round Robin）**： + +- 根据权重分配请求 +- 考虑服务器性能差异 + +**3. 随机（Random）**： +- 随机选择服务器 +- 实现简单，长期来看分布均匀 + +**4. 最少活跃数（Least Active）**： +- 选择当前活跃请求数最少的服务器 +- 适应不同服务器的处理能力 + +**5. 一致性哈希（Consistent Hash）**： + +- 相同参数的请求路由到同一服务器 +- 适用于有状态的服务 + +**💻 代码实现**： + +```java +// 负载均衡接口 +public interface LoadBalancer { + ServiceInfo select(List services, RPCRequest request); +} + +// 轮询负载均衡 +public class RoundRobinLoadBalancer implements LoadBalancer { + + private final AtomicInteger index = new AtomicInteger(0); + + @Override + public ServiceInfo select(List services, RPCRequest request) { + if (services.isEmpty()) { + return null; + } + + int currentIndex = Math.abs(index.getAndIncrement()); + return services.get(currentIndex % services.size()); + } +} + +// 加权轮询负载均衡 +public class WeightedRoundRobinLoadBalancer implements LoadBalancer { + + private final ConcurrentHashMap servers = new ConcurrentHashMap<>(); + + @Override + public ServiceInfo select(List services, RPCRequest request) { + if (services.isEmpty()) { + return null; + } + + // 更新权重信息 + updateWeights(services); + + // 选择权重最高的服务器 + WeightedServer selected = null; + int totalWeight = 0; + + for (ServiceInfo service : services) { + WeightedServer server = servers.get(service.getInstanceId()); + server.currentWeight += server.effectiveWeight; + totalWeight += server.effectiveWeight; + + if (selected == null || server.currentWeight > selected.currentWeight) { + selected = server; + } + } + + if (selected != null) { + selected.currentWeight -= totalWeight; + return selected.serviceInfo; + } + + return services.get(0); + } + + private static class WeightedServer { + ServiceInfo serviceInfo; + int effectiveWeight; // 有效权重 + int currentWeight; // 当前权重 + } +} + +// 一致性哈希负载均衡 +public class ConsistentHashLoadBalancer implements LoadBalancer { + + private final TreeMap virtualNodes = new TreeMap<>(); + private final int virtualNodeCount = 160; // 虚拟节点数量 + + @Override + public ServiceInfo select(List services, RPCRequest request) { + if (services.isEmpty()) { + return null; + } + + // 构建哈希环 + buildHashRing(services); + + // 计算请求的哈希值 + String key = buildKey(request); + long hash = hash(key); + + // 找到第一个大于等于该哈希值的虚拟节点 + Map.Entry entry = virtualNodes.ceilingEntry(hash); + if (entry == null) { + entry = virtualNodes.firstEntry(); + } + + return entry.getValue(); + } + + private void buildHashRing(List services) { + virtualNodes.clear(); + + for (ServiceInfo service : services) { + for (int i = 0; i < virtualNodeCount; i++) { + String virtualNodeKey = service.getInstanceId() + "#" + i; + long hash = hash(virtualNodeKey); + virtualNodes.put(hash, service); + } + } + } + + private String buildKey(RPCRequest request) { + // 根据请求参数构建key + return request.getServiceName() + "#" + + Arrays.hashCode(request.getParameters()); + } + + private long hash(String key) { + // 使用MurmurHash或其他哈希算法 + return key.hashCode(); + } +} +``` + + + +### 🎯 如何实现熔断降级机制？ + +熔断降级是保障系统稳定性的重要机制： + +**熔断器状态**： +1. **CLOSED**：正常状态，请求正常通过 +2. **OPEN**：熔断状态，快速失败 +3. **HALF_OPEN**：半开状态，允许少量请求测试 + +**降级策略**： +- **快速失败**：直接返回错误 +- **默认值**：返回预设的默认值 +- **缓存数据**：返回缓存的历史数据 +- **调用备用服务**：调用备用的服务实现 + +--- + + + +## ⚡ 四、性能优化 + + **核心理念**：RPC性能优化需要从网络、序列化、连接管理、异步处理等多个维度进行系统性优化。 + +### 🎯 RPC性能优化有哪些手段？ + +RPC性能优化的系统性方法： + +**1. 网络层优化**： +- 使用高性能的NIO框架（Netty） +- 启用TCP_NODELAY减少延迟 +- 调整TCP发送/接收缓冲区大小 +- 使用连接池复用连接 + +**2. 序列化优化**： +- 选择高性能序列化框架 +- 减少序列化对象的复杂度 +- 使用对象池减少GC压力 +- 压缩传输数据 + +**3. 异步化处理**： +- 客户端异步调用 +- 服务端异步处理 +- 批量调用减少网络开销 +- 流水线处理提高吞吐量 + + + +### 🎯 如何提升网络通信性能？ + +如何提升 RPC 的网络通信性能，这句话翻译一下就是：一个 RPC 框架如何选择高性能的网络编程 I/O 模型？这样一来，和 I/O 模型相关的知识点就是你需要掌握的了。 + +对于 RPC 网络通信问题，你首先要掌握网络编程中的五个 I/O 模型： + +- 同步阻塞 I/O（BIO） + +- 同步非阻塞 I/O + +- I/O 多路复用（NIO） + +- 信号驱动 + +- 以及异步 I/O（AIO） + +但在实际开发工作，最为常用的是 BIO 和 NIO（这两个 I/O 模型也是面试中面试官最常考察候选人的）。 + +NIO 比 BIO 提高了服务端工作线程的利用率，并增加了一个调度者，来实现 Socket 连接与 Socket 数据读写之间的分离。 + +在目前主流的 RPC 框架中，广泛使用的也是 I/O 多路复用模型，Linux 系统中的 select、poll、epoll等系统调用都是 I/O 多路复用的机制。 + +在面试中，对于高级研发工程师的考察，还会有两个技术扩展考核点。 + +Reactor 模型（即反应堆模式），以及 Reactor 的 3 种线程模型，分别是单线程 Reactor 线程模型、多线程 Reactor 线程模型，以及主从 Reactor 线程模型。 + +Java 中的高性能网络编程框架 Netty。 + +可以这么说，在高性能网络编程中，大多数都是基于 Reactor 模式，其中最为典型的是 Java 的 Netty 框架，而 Reactor 模式是基于 I/O 多路复用的，所以，对于 Reactor 和 Netty 的考察也是避免不了的。 + +--- + + + +## 🔍 五、主流框架对比 + + **核心理念**：了解不同RPC框架的特点和适用场景，根据业务需求选择合适的技术方案。 + +### 🎯 Dubbo、gRPC、Thrift有什么区别？ + +主流RPC框架特性对比： + +| 特性 | Dubbo | gRPC | Thrift | +|------|-------|------|--------| +| **开发语言** | Java | 多语言 | 多语言 | +| **传输协议** | TCP | HTTP/2 | TCP | +| **序列化** | 多种选择 | Protobuf | Thrift | +| **服务治理** | 丰富 | 基础 | 基础 | +| **性能** | 优秀 | 优秀 | 优秀 | +| **社区活跃度** | 高 | 高 | 中等 | +| **学习成本** | 中等 | 低 | 中等 | + +**选择建议**： +- **Dubbo**：Java生态，服务治理功能丰富 +- **gRPC**：跨语言场景，Google支持 +- **Thrift**：Facebook出品，性能优秀 + + + +### 🎯 Spring Cloud与Dubbo的对比？ + +微服务框架的不同理念： + +**Spring Cloud特点**： +- 基于HTTP协议，简单易用 +- 与Spring生态深度集成 +- 组件丰富，生态完整 +- 适合快速开发 + +**Dubbo特点**： +- 基于TCP协议，性能更优 +- 专注RPC通信 +- 服务治理功能强大 +- 更适合高性能场景 + +**💻 使用示例**： + +```java +// Dubbo使用示例 +@Service +public class UserServiceImpl implements UserService { + + @Override + public User getUserById(Long id) { + return userRepository.findById(id); + } +} + +@Component +public class UserController { + + @Reference + private UserService userService; + + public User getUser(Long id) { + return userService.getUserById(id); + } +} + +// Spring Cloud使用示例 +@RestController +public class UserController { + + @Autowired + private UserServiceClient userServiceClient; + + @GetMapping("/users/{id}") + public User getUser(@PathVariable Long id) { + return userServiceClient.getUserById(id); + } +} + +@FeignClient("user-service") +public interface UserServiceClient { + + @GetMapping("/users/{id}") + User getUserById(@PathVariable Long id); +} +``` + + + +### 🎯 gRPC与HTTP的区别是什么？ + +HTTP是应用层协议，主要用于传输超文本，而RPC是一种远程过程调用框架，用于分布式系统中的服务间通信。HTTP基于文本传输，而RPC通常使用二进制序列化协议，减少数据传输体积。 + +在现代分布式系统中，选择 **RPC**（Remote Procedure Call）而非单纯的 **HTTP** 协议，主要出于 **性能、服务治理能力、通信模型灵活性** 以及 **开发效率** 等方面的考量。 + +| **对比维度** | **HTTP** | **gRPC** | +| -------------- | ----------------------------------- | ------------------------------------------------ | +| **协议设计** | 基于文本的请求-响应模型（GET/POST） | 基于HTTP/2的二进制分帧协议，支持双向流式通信 | +| **性能** | 文本解析开销大，性能较低 | 二进制传输，头部压缩，多路复用，延迟更低 | +| **使用场景** | Web服务、RESTful API | 微服务间高性能通信、实时数据流（如聊天、视频流） | +| **接口定义** | 无强制规范 | 使用Protocol Buffers定义接口，强类型约束 | +| **跨语言支持** | 天然支持 | 通过Protobuf生成多语言客户端/服务端代码 | + + + +### 🎯 **gRPC的四种通信模式？** + +1. 简单RPC（Unary RPC）：客户端发送单个请求，服务端返回单个响应（如函数调用）。 +2. 服务端流式RPC：客户端发送请求，服务端返回数据流（如股票实时行情推送）。 +3. 客户端流式RPC：客户端持续发送数据流，服务端最终返回响应（如文件分片上传）。 +4. 双向流式RPC：双方独立发送和接收数据流，适用于实时交互（如聊天机器人） + + + +### 🎯 Dubbo的核心组件及工作流程？ + +**核心组件**： + +- **服务提供者（Provider）**：暴露服务接口的实现。 +- **服务消费者（Consumer）**：调用远程服务。 +- **注册中心（Registry）**：服务注册与发现（如Zookeeper、Nacos）。 +- **配置中心**：管理服务配置。 + +**工作流程**： + +1. **服务注册**：Provider启动时向注册中心注册自身信息（IP、端口等）。 +2. **服务发现**：Consumer从注册中心获取Provider列表。 +3. **负载均衡**：Consumer通过负载均衡策略（如随机、轮询）选择Provider。 +4. **远程调用**：通过Netty等通信框架发起RPC调用。 + + + +### 🎯 **如何选择RPC框架？** + +| **框架** | **特点** | **适用场景** | +| -------------------------- | ----------------------------------------------------------- | ------------------------------ | +| **Dubbo** | Java生态成熟，支持服务治理（负载均衡、熔断），依赖Zookeeper | 微服务架构，需复杂服务治理 | +| **gRPC** | 高性能、跨语言、支持流式通信，依赖Protobuf | 跨语言服务间通信、实时数据传输 | +| **Thrift** | 支持多种语言，接口定义语言灵活，性能较高 | 多语言混合架构、高吞吐量场景 | +| **Spring Cloud OpenFeign** | 基于HTTP，集成Ribbon、Hystrix，易用性强 | 快速构建微服务，对性能要求不高 | + + + +## 🎯 Feign 是什么？ + +- **Feign** 是 Spring Cloud 提供的一种 **声明式 HTTP 客户端**。 +- 底层是基于 **HTTP/REST** 协议，通过注解（`@FeignClient`）定义接口，Spring 自动生成代理类去发起 HTTP 请求。 +- 特点： + - 简单易用，和 Spring Cloud 无缝集成（Ribbon、Eureka、Nacos、Sentinel）。 + - 天然支持负载均衡、熔断、降级。 + +- **Feign 适合：** + - Java 生态项目，尤其是 **Spring Cloud 微服务架构**。 + - 场景中以 REST 接口为主，团队希望开发简单、调试方便。 + - 业务接口不是特别高频或性能敏感（比如电商商品、订单接口调用）。 +- **gRPC 适合：** + - **跨语言系统**（如 Java 服务和 Go、Python 服务交互）。 + - **高性能场景**（如推荐系统、广告系统、金融交易系统）。 + - **实时通信**（如 IM 聊天、流式日志处理、视频推送）。 + - **云原生环境**（K8s、Istio 微服务治理）。 + +--- + + + +## 🚀 六、高级特性与实践 + + **核心理念**：掌握RPC框架的高级特性，如泛化调用、多版本支持等企业级应用场景。 + +### 🎯 如何实现RPC的多版本支持？ + +多版本支持是企业级RPC的重要特性： + +**版本管理策略**： + +1. **接口版本化**：在接口中添加版本信息 +2. **服务分组**：不同版本部署到不同分组 +3. **灰度发布**：新旧版本并行运行 +4. **兼容性设计**：向后兼容的API设计 + +**💻 代码实现**： + +```java +// 版本化接口定义 +@Service(version = "1.0") +public class UserServiceV1 implements UserService { + + @Override + public User getUserById(Long id) { + return userRepository.findById(id); + } +} + +@Service(version = "2.0") +public class UserServiceV2 implements UserService { + + @Override + public User getUserById(Long id) { + User user = userRepository.findById(id); + // V2版本增加了额外的处理逻辑 + if (user != null) { + user.setLastAccessTime(new Date()); + userRepository.save(user); + } + return user; + } +} + +// 客户端版本选择 +@Component +public class UserServiceClient { + + @Reference(version = "2.0", check = false) + private UserService userServiceV2; + + @Reference(version = "1.0", check = false) + private UserService userServiceV1; + + public User getUser(Long id, String version) { + if ("2.0".equals(version)) { + return userServiceV2.getUserById(id); + } else { + return userServiceV1.getUserById(id); + } + } +} +``` + + + +### 🎯 如何实现RPC调用监控？ + +> 在项目里做 RPC 调用监控，核心思路是 **埋点采集 + 指标统计 + 链路追踪 + 可视化告警**。 +> +> 具体来说，先在 RPC 框架的调用入口和出口做埋点，采集 QPS、RT、成功率等指标，再把 TraceId 注入请求头，实现分布式链路追踪。最后配合 Prometheus + Grafana 或 SkyWalking 做可视化和告警。这样可以快速定位调用慢、失败率高的问题，保障服务可用性。 + +**RPC 调用监控的实现思路** + +**1. 埋点采集** + +- 在 RPC 框架的 **调用链路关键位置**打点，收集调用信息： + - **请求前**：服务名、方法名、参数、调用方、开始时间 + - **请求中**：序列化/网络耗时 + - **请求后**：返回结果、耗时、异常情况 +- 常见方式：AOP 拦截（Spring）、Filter/Interceptor（Dubbo/Feign）、字节码增强（ByteBuddy/Agent）。 + +**2. 指标采集** + +- 每次调用埋点后，把数据写入监控系统： + - **QPS**（调用量/秒） + - **RT**（响应时间分布，P95/P99） + - **成功率**（成功数 / 总数） + - **异常统计**（超时、网络错误、业务异常） +- 可以通过 **Metrics 库**（Micrometer、Dropwizard Metrics）直接暴露。 + +**3. 链路追踪** + +- 问题：调用链可能跨多个微服务，需要知道整个链路在哪卡住。 +- 解决：引入 **分布式追踪系统**（如 Zipkin、Jaeger、SkyWalking）： + - 给每个 RPC 请求加 **TraceId/SpanId** + - 在调用链中透传 TraceId + - 最终可以在可视化界面看到调用树，快速定位慢点/故障点。 + +**4. 可视化与告警** + +- 数据存储到 **Prometheus + Grafana** 或 **ELK** +- 配置告警策略： + - **超时率 > 2%** 发告警 + - **调用失败率 > 5%** 触发熔断 + - **RT > 200ms** 报警 + + + +### 🎯 CAP理论在RPC中的应用？ + +CAP理论指出分布式系统无法同时满足一致性（Consistency）、可用性（Availability）、分区容错性（Partition Tolerance），需根据业务权衡： + +- **Zookeeper**：CP系统，保证数据一致性，网络分区时拒绝写请求。 +- Eureka：AP系统，优先保证可用性，容忍网络分区，但可能返回旧数据。 + +**RPC选型建议**： + +- 对一致性要求高（如金融交易）：选择Zookeeper作为注册中心。 +- 对可用性要求高（如高并发Web服务）：选择Eureka或Nacos + +--- + + + +## 🎯 面试重点总结 + +### 高频考点速览 + +- **RPC基本原理**：调用流程、网络通信、动态代理的实现机制 +- **序列化方案**：各种序列化技术的特点、性能对比、选择策略 +- **服务治理**：注册发现、负载均衡、熔断降级的设计与实现 +- **性能优化**：网络优化、异步处理、连接复用的系统性方法 +- **框架对比**：Dubbo、gRPC、Spring Cloud的特点和适用场景 +- **高级特性**：多版本支持、监控埋点、故障处理的企业级实践 + +### 面试答题策略 + +1. **原理阐述**：先说概念定义，再讲技术原理，最后谈应用场景 +2. **对比分析**：不同技术方案的优缺点对比，选择依据 +3. **实战经验**：结合具体项目经验，展示解决问题的思路 +4. **代码展示**：关键技术点用简洁代码示例说明 + +--- + +## 📚 扩展学习 + +- **官方文档**：Dubbo、gRPC、Thrift等框架的官方文档 +- **源码分析**：深入理解框架的实现原理和设计思想 +- **性能测试**：对比不同框架和配置的性能表现 +- **最佳实践**：学习企业级RPC应用的架构设计和运维经验 \ No newline at end of file diff --git a/docs/interview/Redis-FAQ.md b/docs/interview/Redis-FAQ.md index 3b78e8e122..08c9f2782f 100644 --- a/docs/interview/Redis-FAQ.md +++ b/docs/interview/Redis-FAQ.md @@ -1,1007 +1,3561 @@ -> 你能说说 Redis 是什么吗，有什么特点 +--- +title: Redis 面试专场 +date: 2024-05-31 +tags: + - Redis + - Interview +categories: Interview +--- + +![](https://img.starfish.ink/redis/redis-faq-banner.png) + +> **导读：**不管哪个模板的面试题，其实都是分原理和实践两部分。所以两方面都要准备。 +> +> 比如你们项目是怎么用缓存的，服务是怎么部署的，不要像有些同学自己项目中的 Redis 是集群部署还是哨兵都不清楚。 +> +> 面试是需要准备的~ + +## 🗺️ 知识导航 + +### 🏷️ 核心知识分类 +1. **🔑 Redis基础架构**：基本概念、线程模型、I/O多路复用、Reactor模式、与Memcached对比 +2. **🗂️ 数据结构与底层实现**：五种基本类型、SDS、Hash实现、跳表、扩展数据类型、使用场景 +3. **💾 持久化与内存管理**：RDB/AOF、混合持久化、过期策略、淘汰策略、内存模型、Fork/COW +4. **🔗 事务与脚本**：事务特性、WATCH机制、Lua脚本、原子性保障 +5. **🌐 高可用与分布式**：主从复制、哨兵机制、Cluster集群、分片策略、一致性保障、CAP权衡 +6. **🚀 性能优化与异常处理**：缓存策略、大Key/热Key、缓存三大问题、性能调优、监控指标 +7. **🔐 分布式锁与并发控制**：Redis分布式锁、Redlock算法、续期机制、与ZK对比、乐观锁应用 +8. **📨 消息队列与异步处理**：List/Stream队列、Pub/Sub、延时队列、异步消息机制 +9. **🛠️ 实战应用与最佳实践**：管道技术、UV统计、实践案例、使用误区、优化技巧 +### 🔑 面试话术模板 -## 一、Redis 基础问题 +| 问题类型 | 回答框架 | 关键要点 | 深入扩展 | +| --- | --- | --- | --- | +| 机制原理 | 背景→实现→特点→适用 | 底层数据结构/流程图 | 版本差异、源码细节 | +| 架构能力 | 架构→优缺点→权衡 | CAP、延迟、可用性 | 参数与部署建议 | +| 性能优化 | 痛点→策略→数据 | 命中率、RT/QPS | 压测与监控指标 | +| 故障治理 | 现象→定位→修复 | 工具与指标 | 预防与演练 | -### Redis是什么 +## 一、Redis基础架构 + +### 🎯 Redis是什么？ Redis：**REmote DIctionary Server**(远程字典服务器)。 Redis 是一个全开源免费（BSD许可）的，内存中的数据结构存储系统，它可以用作**数据库、缓存和消息中间件**， -和 Memcached 类似，它支持存储的 value 类型相对更多，包括**string(字符串)、list(链表)、set(集合)、zset(sorted set --有序集合)和hash（哈希类型）、bitmap、hyperloglog、GeoHash、streams**。这些数据类型都支持push/pop、add/remove及取交集并集和差集及更丰富的操作，而且这些操作都是原子性的。后来的版本还提供了HyperLogLog、GeoHash等更高级的数据类型。 +和 Memcached 类似，它支持存储的 value 类型相对更多，包括**string(字符串)、list(链表)、set(集合)、zset(sorted set --有序集合)和hash（哈希类型）、bitmap、hyperloglog、GeoHash、streams**。这些数据类型都支持push/pop、add/remove及取交集并集和差集及更丰富的操作，而且这些操作都是原子性的。 + +Redis 内置了复制（Replication），LUA脚本（Lua scripting）， LRU驱动事件（LRU eviction），事务（Transactions） 和不同级别的磁盘持久化（Persistence），并通过 Redis 哨兵（Sentinel）和自动分区（Cluster）提供高可用性（High Availability）。 + +- 性能优秀，数据在内存中，读写速度非常快，支持并发10W QPS +- 单进程单线程，是线程安全的，采用IO多路复用机制 +- Redis 数据库完全在内存中，使用磁盘仅用于持久性 +- 相比许多键值数据存储，Redis 拥有一套较为丰富的数据类型 +- 操作都是**原子性**：所有 Redis 操作是原子的，这保证了如果两个客户端同时访问的Redis服务器将获得更新后的值 +- Redis 可以将数据复制到任意数量的从服务器（主从复制，哨兵，高可用） + + + +### 🎯 为什么要用缓存？为什么使用 Redis？ + +**提一下现在 Web 应用的现状** + +在日常的 Web 应用对数据库的访问中，**读操作的次数远超写操作**，比例大概在 **1:9** 到 **3:7**，所以需要读的可能性是比写的可能大得多的。当我们使用 SQL 语句去数据库进行读写操作时，数据库就会 **去磁盘把对应的数据索引取回来**，这是一个相对较慢的过程。 + +**使用 Redis or 使用缓存带来的优势** + +如果我们把数据放在 Redis 中，也就是直接放在内存之中，让服务端直接去读取内存中的数据，那么这样 **速度** 明显就会快上不少 *(高性能)*，并且会 **极大减小数据库的压力** *(特别是在高并发情况下)*。 + +**也要提一下使用缓存的考虑** + +但是使用内存进行数据存储开销也是比较大的，**限于成本** 的原因，一般我们只是使用 Redis 存储一些 **常用和主要的数据**，比如用户登录的信息等。 + +一般而言在使用 Redis 进行存储的时候，我们需要从以下几个方面来考虑： + +- **业务数据常用吗？命中率如何？** 如果命中率很低，就没有必要写入缓存； +- **该业务数据是读操作多，还是写操作多？** 如果写操作多，频繁需要写入数据库，也没有必要使用缓存； +- **业务数据大小如何？** 如果要存储几百兆字节的文件，会给缓存带来很大的压力，这样也没有必要； + +在考虑了这些问题之后，如果觉得有必要使用缓存，那么就使用它！ + + + +### 🎯 用缓存，肯定是因为他快，那 Redis 为什么这么快？ + +Redis快在“三件套+两层优化”： + +- 三件套：全内存操作（避免磁盘IO）、高效数据结构（哈希表/跳表/quicklist/listpack）、单线程事件驱动（无锁、无切换）。 + +- 两层优化：I/O多路复用（epoll/kqueue）配合流水线/批处理降低系统调用；工程级细节（RESP协议简单、jemalloc、渐进式rehash、异步持久化/删除、6.0起I/O线程分流收发）。 + +> - **纯内存操作**：读取不需要进行磁盘 I/O，所以比传统数据库要快上不少；*(但不要有误区说磁盘就一定慢，例如 Kafka 就是使用磁盘顺序读取但仍然较快)* +> +> - 用 hash table 作为键空间，查找任意的 key 只需 $O(1)$ +> +> - **单线程，无锁竞争**：天生的队列模式，避免了因多线程竞争而导致的上下文切换和抢锁的开销 +> +> - 事件机制，Redis服务器将所有处理的任务分为两类事件，一类是采用 I/O 多路复用处理客户端请求的网络事件；一类是处理定时任务的时间事件，包括更新统计信息、清理过期键、持久化、主从同步等； +> +> - **多路 I/O 复用模型，非阻塞 I/O**：采用多路 I/O 复用技术可以让单个线程高效的处理多个网络连接请求（尽量减少网络 IO 的时间消耗）； +> +> - Redis 基于 Reactor 模式开发了自己的网络事件处理器，这个处理器被称为文件事件处理器 file event handler。由于这个文件事件处理器是单线程的，所以 Redis 才叫做单线程的模型，但是它采用 IO 多路复用机制同时监听多个Socket，并根据Socket上的事件来选择对应的事件处理器进行处理。 +> +> 多个 Socket 可能会产生不同的操作，每个操作对应不同的文件事件，但是IO多路复用程序会监听多个Socket，将Socket产生的事件放入队列中排队，事件分派器每次从队列中取出一个事件，把该事件交给对应的事件处理器进行处理。 +> +> Redis客户端对服务端的每次调用都经历了发送命令，执行命令，返回结果三个过程。其中执行命令阶段，由于Redis是单线程来处理命令的，所有每一条到达服务端的命令不会立刻执行，所有的命令都会进入一个队列中，然后逐个被执行。并且多个客户端发送的命令的执行顺序是不确定的。但是可以确定的是不会有两条命令被同时执行，不会产生并发问题，这就是Redis的单线程基本模型。 +> +> - Redis直接自己构建了VM 机制 ，避免调用系统函数的时候，浪费时间去移动和请求 +> +> - **高效的数据结构，加上底层做了大量优化**：Redis 对于底层的数据结构和内存占用做了大量的优化，例如不同长度的字符串使用不同的结构体表示，HyperLogLog 的密集型存储结构等等.. + +但是因为 Redis 不同版本的特殊性，所以对于 Redis 的线程模型要分版本来看。 + +Redis 4.0 版本之前，使用单线程速度快的原因就是内存、数据结构、单线程、IO 多路复用； + +Redis 4.0 版本之后，Redis 添加了多线程的支持，但这时的多线程主要体现在大数据的异步删除功能上，例如 unlink key、flushdb async、flushall async 等。 + +Redis 6.0 版本之后，为了更好地提高 Redis 的性能，新增了多线程 I/O 的读写并发能力，但是在面试中，能把 Redis 6.0 中的多线程模型回答上来的人很少，如果你能在面试中补充 Redis 6.0 多线程的原理，势必会增加面试官对你的认可。 + +你可以在面试中这样补充： + +虽然 Redis 一直是单线程模型，但是在 Redis 6.0 版本之后，也采用了多个 I/O 线程来处理网络请求，这是因为随着网络硬件的性能提升，Redis 的性能瓶颈有时会出现在网络 I/O 的处理上，所以为了提高网络请求处理的并行度，Redis 6.0 对于网络请求采用多线程来处理。但是对于读写命令，Redis 仍然使用单线程来处理。 + + + +### 🎯 Redis 属于单线程还是多线程？ + +> Redis 的 **核心网络模型与命令执行** 采用 **单线程**（基于 I/O 多路复用 `epoll/kqueue`），所以不会出现多线程并发修改数据的问题。 +> +> **但**：从 Redis 4.0 开始引入了一些 **多线程辅助功能**（比如异步删除、后台持久化、RDB/AOF 重写），到 **Redis 6.0 才正式引入多线程 I/O**，用于处理网络数据读写，减少单线程瓶颈。 + +- 线程模型分版本 + + - ≤4.0：主线程处理网络+命令，后台有少量BIO线程（fsync、close/unlink），bgsave/aof重写通过 fork 子进程。 + + - 4.x：引入异步删除（UNLINK、flush async），但命令与I/O仍由主线程。 + + - ≥6.0：I/O 多线程（io-threads）分摊“读取请求/写回响应”的开销；主线程负责命令解析与执行，保证一致的内存访问模型。 + +- Reactor 模型 + - 单个事件循环（file events + time events），用 epoll/kqueue 等 I/O 多路复用等待就绪 FD，回调驱动处理；这让“单执行线程”支撑高并发。 + +- 为什么快（面试官高频点） + + - 内存操作+紧凑结构（listpack/skiplist） + + - 单线程避免锁竞争与上下文切换 + + - I/O 多路复用减少阻塞 + + - 简单协议（RESP）+批量/流水线 + +- 6.0 I/O 线程原理（答出就加分） + + - 读取路径：I/O 线程把 socket 数据读入 client buffer，主线程解析+执行命令。 + + - 写回路径：主线程生成响应后，I/O 线程负责把大响应写回客户端，降低主线程被大包 write 阻塞。 + + - 配置要点：io-threads N，默认只用于写；开启读用 io-threads-do-reads yes（读多且网络瓶颈明显时再开）。线程不是越多越好，常见 4～8。 + +- “不是全单线程”的补充 + + - 后台线程/子进程：AOF fsync、RDB/AOF 重写（fork+COW）、UNLINK/lazyfree、active defrag、复制/集群通信线程等。 + + - 这些与主执行线程解耦，降低主线程停顿，但 fork 时要关注 COW 内存与IO抖动。 + + + **为什么这样设计？** + + 单线程优势： + + 1. 避免锁竞争：无需复杂的并发控制 + 2. 简化实现：代码逻辑清晰，易于维护 + 3. 原子性保证：所有操作天然原子性 + 4. 高性能：配合I/O多路复用，单线程也能处理高并发 + + 多线程补充： + + 1. I/O瓶颈优化：网络I/O处理能力提升 + 2. 后台任务分离：避免阻塞主线程 + 3. 充分利用多核：在不影响核心逻辑的前提下提升性能 + +**误区澄清（面试官常挖的坑）** + +- “Redis 多线程了能并行执行命令吗？”不能，命令仍是主线程串行执行；I/O 才并发。 + +- “开很多 I/O 线程就一定更快？”不一定，CPU 竞争、调度与缓存失效可能适得其反；先定位瓶颈再开。 + +- “单线程吃不满多核？”用集群/分片横向扩展；或多实例多核亲和部署。 + +> **Redis 6.0 之前：全单线程** +> +> 在 Redis 6.0 之前，整个请求流程是这样的： +> +> ``` +> 客户端 -> [主线程] +> 1️⃣ 读取 socket 数据 +> 2️⃣ 解析命令（字符串解析、参数拆分） +> 3️⃣ 执行命令（访问数据结构、修改内存） +> 4️⃣ 写回响应数据 +> ``` +> +> ➡️ 所有步骤都在同一个线程完成，**I/O + 命令执行全串行**。 +> +> ------ +> +> **Redis 6.0 之后：I/O 多线程化（但执行仍单线程）** +> +> Redis 6.0 为了提升 **网络吞吐量**（尤其是高并发下的收发性能），把请求流程拆成了两层： +> +> ``` +> 客户端 -> [I/O 线程池] -> [主线程] +> ``` +> +> 具体分工如下： +> +> | 阶段 | 执行线程 | 职责说明 | +> | --------------- | ---------------------- | --------------------------------------------------------- | +> | 1️⃣ 网络 I/O 读写 | **I/O 线程（多线程）** | 从 socket 中读取数据包 / 写回响应包，负责纯粹的“收”和“发” | +> | 2️⃣ 命令解析 | **主线程** | 把字节流解析成 Redis 命令，比如解析 `"SET key value"` | +> | 3️⃣ 命令执行 | **主线程** | 执行命令逻辑，如操作字典、跳表等内存结构 | +> | 4️⃣ 返回结果 | **I/O 线程（多线程）** | 将命令结果写入 socket，发给客户端 | + +### 🎯 Redis多路复用？ + +Redis的“多路复用”指用一个事件循环线程，借助内核的 I/O 复用接口（Linux 上主要是 epoll，BSD 上是 kqueue），同时监听成千上万个套接字的可读/可写事件。就绪才处理，非阻塞读写，配合 Reactor 模型把网络 I/O 的并发性和命令执行的串行性解耦。6.0 起增加了 I/O 线程加速“收发数据”，但“命令解析与执行”仍在主线程，避免并发共享数据的复杂度。 + + 简单来说： + + - 多路：指多个TCP连接（或多个Channel） + - 复用：指复用一个或少量线程 + - 核心：用一个线程监视多个文件描述符，一旦某个描述符就绪就能进行相应的I/O操作 + +以下是 Redis 多路复用的工作原理： + +**1. 多路复用简介** + +I/O多路复用，I/O就是指的我们网络I/O，多路指多个TCP连接(或多个Channel)，复用指复用一个或少量线程。串起来理解就是很多个网络I/O复用一个或少量的线程来处理这些连接。 + +> 多路复用是一种 I/O 复用技术，可以让一个线程监视多个文件描述符（FD），一旦某个描述符准备好进行 I/O 操作（如读或写），程序就可以对其进行相应的处理。这样可以有效地提高系统资源利用率和并发性能。 +> +> IO 多路复用是 Redis 快速响应众多客户端请求的核心技术之一。IO 多路复用允许 Redis 同时监听多个文件描述符（通常是网络套接字），当其中任何一个描述符就绪（可读或可写）时，程序可以进行相应的处理。这样可以在单线程中高效处理大量并发连接。 + +**2. Redis 使用的多路复用机制** + +Redis 根据不同操作系统，选择最合适的多路复用机制： + +- 在 Linux 系统上，使用 `epoll` +- 在 BSD 系统上，使用 `kqueue` +- 在 Solaris 上，使用 `evport` +- 在其他一些系统上，使用 `select`：最古老的 IO 多路复用技术，支持的文件描述符数量有限。 + +**3. 工作流程** + + Redis基于Reactor模式实现了文件事件处理器(File Event Handler)，这个处理器包含4个组成部分： + + 1. 多个套接字(Socket)：客户端连接 + 2. I/O多路复用程序：监听多个套接字的事件 + 3. 文件事件分派器：将就绪的事件分配给对应的处理器 + 4. 事件处理器：处理具体的业务逻辑 + +以下是 Redis 使用多路复用的工作流程： + +``` + [客户端1] ──┐ + [客户端2] ──┼─→ [I/O多路复用程序] ──→ [事件分派器] ──→ [事件处理器] + [客户端3] ──┘ (epoll/select) (单线程队列) (命令执行) +``` + +1. **初始化事件循环**： Redis 在启动时会初始化一个事件循环，其中包含一个事件表，用于记录所有需要监视的文件描述符及其相关的事件（如可读、可写）。 +2. **注册事件**： 当 Redis 需要监视某个文件描述符（如新客户端连接或现有客户端的读写操作）时，会将该描述符及其事件类型（如读、写）注册到事件表中。 +3. **等待事件触发**： Redis 使用多路复用函数（如 `epoll_wait`、`kqueue` 或 `select`）等待注册的文件描述符上有事件发生。这个函数会阻塞，直到至少有一个文件描述符准备好进行 I/O 操作。 +4. **处理事件**： 一旦多路复用函数返回，Redis 会遍历触发事件的文件描述符，并对其进行相应的处理。这可能包括读取客户端请求、向客户端发送响应、接受新的连接等。 +5. **事件处理完毕后继续等待**： 处理完所有触发的事件后，Redis 会再次调用多路复用函数，继续等待新的事件发生。 + +**4. 优点** + +Redis 使用多路复用的优点包括： + +- **高效性**：可以高效地监视大量的文件描述符，并且只在有事件发生时才进行处理，避免了轮询方式的高开销。 +- **单线程模型**：通过多路复用，Redis 能够在单线程中高效地处理大量的并发连接，简化了编程模型和线程间同步问题。 +- **跨平台性**：Redis 封装了多种操作系统的多路复用机制，使其可以在不同平台上高效运行。 + + + +### 🎯 为什么早期版本的 Redis 选择单线程？ + +官方FAQ表示，因为Redis是基于内存的操作，CPU不是Redis的瓶颈，Redis 的瓶颈最有可能是机器内存的大小或者网络带宽。既然单线程容易实现，而且 CPU 不会成为瓶颈，那就顺理成章地采用单线程的方案了（毕竟采用多线程会有很多麻烦！）。 + +看到这里，你可能会气哭！本以为会有什么重大的技术要点才使得Redis使用单线程就可以这么快，没想到就是一句官方看似糊弄我们的回答！但是，我们已经可以很清楚的解释了为什么 Redis 这么快，并且正是由于在单线程模式的情况下已经很快了，就没有必要在使用多线程了！ + +**简单总结一下** + +1. 使用单线程模型能带来更好的可维护性，方便开发和调试； +2. 使用单线程模型也能并发的处理客户端的请求； +3. Redis 服务中运行的绝大多数操作的性能瓶颈都不是 CPU； + +这里我们一直在强调的单线程，只是在处理我们的网络请求的时候只有一个线程来处理，一个正式的 Redis Server 运行的时候肯定是不止一个线程的，这里需要大家明确的注意一下！例如 Redis 进行持久化的时候会以子进程或者子线程的方式执行； + +Redis 选择使用单线程模型处理客户端的请求主要还是因为 CPU 不是 Redis 服务器的瓶颈，所以使用多线程模型带来的性能提升并不能抵消它带来的开发成本和维护成本，系统的性能瓶颈也主要在网络 I/O 操作上； + +而 Redis 引入多线程操作也是出于性能上的考虑，对于一些大键值对的删除操作，通过多线程非阻塞地释放内存空间也能减少对 Redis 主线程阻塞的时间，提高执行的效率。 + +在高并发场景下可以利用多个线程 并发处理 IO 任务、命令解析和数据回写。这些线程也被叫做 IO 线程。默认情况下，多线程模式是被禁用了的，需要显式地开启。 + +> 推荐阅读：https://draveness.me/whys-the-design-redis-single-thread/ + + + +### 🎯 Redis 和 Memcached 的区别？ + +1. 存储方式上：memcache 会把数据全部存在内存之中，断电后会挂掉，数据不能超过内存大小。redis 有部分数据存在硬盘上，这样能保证数据的持久性。 +2. 数据支持类型上：memcache 对数据类型的支持简单，只支持简单的 key-value，而 redis 支持五种数据类型。 +3. 使用底层模型不同：它们之间底层实现方式以及与客户端之间通信的应用协议不一样。redis 直接自己构建了 VM 机制，因为一般的系统调用系统函数的话，会浪费一定的时间去移动和请求。 +4. value 的大小：redis 可以达到 512M，而 memcache 只有 1M。 + + + +### 🎯 key 最大是多少 ，单个实例最多支持多少个key？ + +Redis 对键（key）的最大长度和单个实例最多支持的键数量有明确的限制。下面详细解释这些限制： + +1. 键（Key）的最大长度 + +​ Redis 对每个键的最大长度有严格的限制。根据 Redis 的官方文档：键的最大长度为 512 MB（512 * 1024 * 1024 bytes）。 + +​ 虽然 Redis 允许非常长的键，但在实际应用中建议避免使用过长的键，以提高内存利用效率和操作性能。 + +2. 单个实例最多支持的键数量 + + Redis 是一个内存数据库，理论上它可以存储的键数量没有严格的上限，取决于可用内存和系统的限制。然而，实际应用中单个实例的键数量会受到以下因素的限制： + + - 内存限制 + + Redis 存储的数据都在内存中，因此可用内存是决定单个实例能存储多少键的主要因素。Redis 可以使用的最大内存量取决于机器的物理内存和 Redis 的配置。 + + **配置最大内存**：通过配置 `maxmemory` 参数，可以限制 Redis 使用的最大内存量。例如，在 redis.conf 文件中设置：`maxmemory 4GB` + + 这将限制 Redis 使用的最大内存为 4 GB。当达到这个限制时，可以通过 `maxmemory-policy` 参数配置内存淘汰策略，如 LRU（Least Recently Used）、LFU（Least Frequently Used）、ALLKEYS-RANDOM 等。 + + - 数据结构的内存开销 + + 不同的数据结构（如字符串、哈希、列表、集合、有序集合）在 Redis 中的内存开销不同。每个键值对的存储不仅包括键和值的实际内容，还包括 Redis 内部维护的数据结构（如哈希表节点、指针等）的开销。 + +Redis 内部实现限制 + +虽然 Redis 没有明确限制单个实例的最大键数量，但 Redis 的哈希表实现有默认的初始大小和扩展策略： + +- **初始哈希表大小**：Redis 默认初始化的哈希表大小较小，但会根据键数量自动扩展。 + +- **扩展策略**：Redis 使用渐进式 rehashing 来扩展哈希表，以减少扩展过程中对性能的影响。 + + + +### 🎯 为什么 Redis 不建议 key 太长，原理？ + +Redis 的键（key）设计不建议太长，这主要是出于以下几个方面的考虑： + +1. **内存使用效率**： + - Redis 的键和值在内存中是成对存储的。键过长意味着每个键值对占用的内存空间会增加，这会降低内存的使用效率。 + - 较长的键名会增加内存占用，因为 Redis 需要为每个键分配额外的内存空间来存储键名。 +2. **性能影响**： + - 键的查找、存储和删除操作都需要对键名进行处理。键名较长会增加这些操作的执行时间，从而影响性能。 + - 特别是当使用具有前缀或模式匹配的键进行操作时，如 `keys` 命令或 `SCAN` 命令，长键名会增加处理时间，影响性能。 +3. **网络传输效率**： + - 在客户端与 Redis 服务器之间传输数据时，键名也是需要传输的一部分。键名较长会增加网络传输的数据量，降低传输效率。 +4. **可读性和维护性**： + - 较短的键名通常更易于理解和维护。长键名可能会使得代码更难阅读，也更容易出错。 + - 使用有意义的短键名可以提高代码的可读性和可维护性。 +5. **散列算法的影响**： + - Redis 使用哈希表来存储键值对，长键名在哈希算法中可能会产生更多的冲突，这会增加处理哈希冲突的复杂性。 +6. **限制和配置**： + - Redis 并没有严格的键名长度限制，但是过长的键名可能会受到特定 Redis 配置或客户端库的限制。 +7. **命令行和脚本处理**： + - 在使用命令行工具或编写自动化脚本时，长键名可能会使得命令行变得复杂，增加出错的风险。 + +因此，为了优化内存使用、提高性能、简化开发和维护，通常建议设计键名时尽量简短且具有描述性。在实际应用中，可以根据业务逻辑和需求来设计合适的键名长度，以平衡可读性和性能。 + + + +### 🎯 最后总结下 Redis 优缺点 + +优点 + +- **读写性能优异**， Redis能读的速度是 `110000` 次/s，写的速度是 `81000` 次/s。 +- **支持数据持久化**，支持 AOF 和 RDB 两种持久化方式。 +- **支持事务**，Redis 的所有操作都是原子性的，同时 Redis 还支持对几个操作合并后的原子性执行。 +- **数据结构丰富**，除了支持 string 类型的 value 外还支持 hash、set、zset、list 等数据结构。 +- **支持主从复制**，主机会自动将数据同步到从机，可以进行读写分离。 + +缺点 + +- 数据库 **容量受到物理内存的限制**，不能用作海量数据的高性能读写，因此 Redis 适合的场景主要局限在较小数据量的高性能操作和运算上。 +- Redis **不具备自动容错和恢复功能**，主机从机的宕机都会导致前端部分读写请求失败，需要等待机器重启或者手动切换前端的 IP 才能恢复。 +- 主机宕机，宕机前有部分数据未能及时同步到从机，切换 IP 后还会引入数据不一致的问题，降低了 **系统的可用性**。 +- **Redis 较难支持在线扩容**，在集群容量达到上限时在线扩容会变得很复杂。为避免这一问题，运维人员在系统上线时必须确保有足够的空间，这对资源造成了很大的浪费。 + +------ + + + +## 二、数据结构与底层实现 + +### 🎯 Redis 都支持哪些数据类型 + +首先在 Redis 内部会使用一个 **RedisObject** 对象来表示所有的 `key` 和 `value`： + +Redis 提供了五种基本数据类型，String、Hash、List、Set、Zset(sorted set：有序集合) + +> Redis 不是简单的键值存储，它实际上是一个数据结构服务器，支持不同类型的值。 +> +> 1. String（字符串） +> - 特点：二进制安全，最大512MB +> - 底层实现：SDS（Simple Dynamic String） +> - 应用场景：缓存、计数器、分布式锁、序列化对象存储 +> +> 2. Hash（哈希） +> - 特点：键值对集合，类似于Map +> - 底层实现：ziplist（小数据量）+ hashtable（大数据量） +> - 应用场景：用户信息存储、对象缓存 +> +> 3. List（列表） +> - 特点：有序字符串列表，支持双端操作 +> - 底层实现：quicklist（ziplist + linkedlist 的混合结构） +> - 应用场景：消息队列、时间线、最新列表 +> +> 4. Set（集合） +> - 特点：无序、唯一元素集合 +> - 底层实现：intset（整数）+ hashtable +> - 应用场景：标签系统、共同关注、去重 +> +> 5. Zset（有序集合） +> - 特点：有序且唯一，每个成员关联一个score +> - 底层实现：ziplist（小数据量）+ skiplist + hashtable +> - 应用场景：排行榜、范围查询、优先级队列 +> +> 除了支持最 **基础的五种数据类型** 外，还支持一些 **高级数据类型**： +> +> 1. Bitmap（位图） +> +> - 本质：基于String的位操作 +> +> - 应用：用户签到、在线状态、权限系统、布隆过滤器 +> +> +> 2. HyperLogLog（基数统计） +> +> - 特点：概率算法，固定12KB内存，0.81%标准误差 +> +> - 应用：UV统计、大数据去重计数 +> +> 3. Geospatial（地理空间） +> +> - 底层：基于Zset，使用Geohash编码 +> +> - 应用：LBS服务、附近的人、地理围栏 +> +> 4. Streams（流） +> +> - 特点：持久化日志结构，支持消费者组 +> +> - 应用：消息队列、事件溯源、审计日志 +> +> 5. Bloom Filter（布隆过滤器） +> +> - 特点：概率型数据结构，判断存在性 +> +> - 应用：缓存穿透防护、URL去重、推荐系统 + +由于 Redis 是基于标准 C 写的，只有最基础的数据类型，因此 Redis 为了满足对外使用的 5 种数据类型，开发了属于自己**独有的一套基础数据结构**，使用这些数据结构来实现5种数据类型。 + +Redis底层的数据结构包括：**简单动态数组SDS、链表、字典、跳跃链表、整数集合、压缩列表、对象。** + +Redis 为了平衡空间和时间效率，针对 value 的具体类型在底层会采用不同的数据结构来实现，其中哈希表和压缩列表是复用比较多的数据结构，如下图展示了对外数据类型和底层数据结构之间的映射关系： + +![](https://img.starfish.ink/redis/redis-data-types.png) + +> 源码中，`redisObject`（或 `robj`）是 Redis 用于表示数据对象的核心结构。每一个 Redis 数据对象，无论是字符串、列表、集合、哈希还是有序集合，都会被封装在一个 `redisObject` 结构体中 +> +> ```c +> //简化版 +> typedef struct redisObject { +> unsigned type:4; // 数据类型 +> unsigned encoding:4; // 对象的编码方式 +> unsigned lru:LRU_BITS; // LRU 时间，用于内存淘汰策略 +> int refcount; // 引用计数 +> void *ptr; // 指向实际存储数据的指针 +> } robj; +> ``` +> +> `redisObject` **各字段解析** +> +> - **type**：用来标识对象的数据类型（如字符串、列表、集合等）。Redis 支持的几种核心数据类型在源码中定义为常量，例如： +> +> ```c +> #define OBJ_STRING 0 // 字符串 +> #define OBJ_LIST 1 // 列表 +> #define OBJ_SET 2 // 集合 +> #define OBJ_ZSET 3 // 有序集合 +> #define OBJ_HASH 4 // 哈希表 +> ``` +> +> - **encoding**：用来标识对象的具体编码方式，也就是该对象的底层数据结构（如 `intset`, `ziplist`, `hashtable` 等）。常见的编码方式如下： +> +> ```c +> #define OBJ_ENCODING_RAW 0 // 普通字符串编码 +> #define OBJ_ENCODING_INT 1 // 整数编码 +> #define OBJ_ENCODING_HT 2 // 哈希表编码 +> #define OBJ_ENCODING_ZIPMAP 3 // 压缩地图编码（老版本 Redis） +> #define OBJ_ENCODING_LINKEDLIST 4 // 链表编码（旧的列表编码） +> #define OBJ_ENCODING_ZIPLIST 5 // 压缩列表编码 +> #define OBJ_ENCODING_INTSET 6 // 整数集合编码 +> #define OBJ_ENCODING_SKIPLIST 7 // 跳表编码（用于有序集合） +> #define OBJ_ENCODING_QUICKLIST 8 // 快速列表编码（新的列表编码） +> ``` +> +> - **lru**：用于记录该对象的最后访问时间，Redis 使用这个字段来实现内存淘汰策略（LRU，最近最少使用算法）。在新的 Redis 版本中，它可能改为 LRU 或 LFU（频率淘汰）。 +> +> - **refcount**：对象的引用计数。Redis 使用引用计数机制来管理对象的内存。如果一个对象的引用计数为 0，则该对象可以被释放。 +> +> - **ptr**：这是一个通用指针，指向该对象实际存储的数据。根据 `type` 和 `encoding` 的不同，`ptr` 指向的结构也不同。例如： +> +> - 对于字符串对象，`ptr` 可能指向的是一个 `sds`（简单动态字符串）。 +> - 对于列表对象，`ptr` 可能指向的是 `quicklist`。 +> - 对于有序集合对象，`ptr` 可能指向的是 `skiplist` 或 `ziplist`。 + +### 🎯 那你能说说这些数据类型的使用指令吗？ + +String：就是基本的 SET、GET、MSET、MGET、INCR、DECR + +List：LPUSH、RPUSH、LRANGE、LINDEX + +Hash：HSET、HMSET、HSETNX、HKEYS、HVALS + +Set：SADD、SCARD、SDIFF、SREM + +SortSet：ZADD、ZCARD、ZCOUNT、ZRANGE + + + +### 🎯 Redis的数据结构， 字符串用什么实现？ + +Redis 中的字符串数据类型并不是直接通过 C 字符串（以 NULL 结尾的字符数组）来实现的，而是使用了一种叫做 **SDS (Simple Dynamic String)** 的数据结构来存储字符串。SDS 的主要优势在于支持快速的字符串操作和高效的内存管理。 + +SDS 结构一般包含以下几个部分： + +- **len**：当前字符串的长度（不包括末尾的 null 字符）。这使得 Redis 不需要在每次操作时都遍历整个字符串来获取其长度。 +- **alloc**：当前为字符串分配的总空间（包括字符串数据和额外的内存空间）。由于 Redis 使用的是动态分配内存，因此可以避免频繁的内存分配和释放。 +- **buf**：实际的字符串数据部分，存储字符串的字符数组。Redis 通过这个区域存储字符串的内容。 + + + +### 🎯 SDS 如何保证二进制安全？ + +Redis 中的 `String` 是 **二进制安全的**，这意味着 Redis 的 `String` 可以存储任何形式的数据，不仅仅是文本（如 JSON、二进制文件、图像数据、序列化数据等）。这一点非常重要，因为 Redis 的 `String` 类型没有对数据内容的限制，可以安全地存储二进制数据。 + +二进制安全是一种主要用于字符串操作函数相关的计算机编程术语。其描述的是：**将输入作为原始的、无任何特殊格式意义的数据流。对于每个字符都公平对待，不特殊处理某一个字符**。 + +- **不依赖于 null 字符**：传统的 C 字符串依赖于 null 字符（'\0'）来表示字符串的结束，但 Redis 的 SDS 不依赖于 null 字符来确定字符串的结束位置。SDS 存储了字符串的实际长度（`len` 字段），因此可以正确处理包含 null 字符的二进制数据。 +- **动态扩展**：SDS 会根据需要动态地扩展其内部缓冲区。Redis 会使用 `alloc` 字段来记录已分配的内存大小。当你向 SDS 中追加数据时，Redis 会确保分配足够的内存，而不需要担心数据的终止符。 +- **不需要二次编码**：二进制数据直接存储在 SDS 的 `buf` 区域内，不需要进行任何编码或转换。因此，Redis 可以原样存储任意二进制数据。 + +> 在早期版本的 Redis（特别是在 2.x 版本中），`SDS` 数据结构包含了一个 `free` 字段，它用于表示当前字符串缓冲区中未使用的内存量。 +> +> ```C +> struct sdshdr { +> int len; // 当前字符串的长度 +> int free; // buf数组中未使用的字节的数量（即额外的分配空间） +> char buf[]; // 字符串内容 +> }; +> ``` +> +> 在现代 Redis（即 Redis 3.x 及之后的版本）中，`SDS` 的实现发生了一些变化，尤其是去除了 `free` 字段。 +> +> ```C +> struct sdshdr { +> int len; // 当前字符串的长度 +> int alloc; // 分配的内存空间大小 +> unsigned char buf[]; // 字符串数据 +> }; +> ``` +> +> 现在 Redis 使用 `alloc` 字段来表示已分配的内存空间，而不再使用 `free`。`alloc` 存储的是当前为字符串分配的内存的总大小，而 `len` 表示已用的部分。 +> +> 为什么移除 `free` 字段？ +> +> - **更简洁的内存管理**：`alloc` 字段可以更直接地表示当前分配的内存大小，简化了内存管理。 +> - **惰性空间回收**：Redis 采用了懒加载和空间回收机制，通过这种方式避免了频繁的内存管理操作。 + + + +### 🎯 Redis 的 SDS 和 C 中字符串相比有什么优势？ + +C 语言使用了一个长度为 `N+1` 的字符数组来表示长度为 `N` 的字符串，并且字符数组最后一个元素总是 `\0`，这种简单的字符串表示方式 **不符合 Redis 对字符串在安全性、效率以及功能方面的要求**。 + +再来说 C 语言字符串的问题 + +这样简单的数据结构可能会造成以下一些问题： + +- **获取字符串长度为 O(N) 级别的操作** → 因为 C 不保存数组的长度，每次都需要遍历一遍整个数组； +- 不能很好的杜绝 **缓冲区溢出/内存泄漏** 的问题 → 跟上述问题原因一样，如果执行拼接 or 缩短字符串的操作，如果操作不当就很容易造成上述问题； +- C 字符串 **只能保存文本数据** → 因为 C 语言中的字符串必须符合某种编码（比如 ASCII），例如中间出现的 `'\0'` 可能会被判定为提前结束的字符串而识别不了； + +**Redis 如何解决的 | SDS 的优势** + +如果去看 Redis 的源码 `sds.h/sdshdr` 文件，你会看到 SDS 完整的实现细节，这里简单来说一下 Redis 如何解决的： + +1. **多增加 len 表示当前字符串的长度**：这样就可以直接获取长度了，复杂度 $O(1)$； +2. **自动扩展空间**：当 SDS 需要对字符串进行修改时，首先借助于 `len` 和 `alloc` 检查空间是否满足修改所需的要求，如果空间不够的话，SDS 会自动扩展空间，避免了像 C 字符串操作中的覆盖情况； +3. **有效降低内存分配次数**：C 字符串在涉及增加或者清除操作时会改变底层数组的大小造成重新分配，SDS 使用了 **空间预分配** 和 **惰性空间释放** 机制，简单理解就是每次在扩展时是成倍的多分配的，在缩容是也是先留着并不正式归还给 OS； +4. **二进制安全**：C 语言字符串只能保存 `ascii` 码，对于图片、音频等信息无法保存，SDS 是二进制安全的，写入什么读取就是什么，不做任何过滤和限制； + + + +### 🎯 SDS，数字类型的自增自减如何实现? + +在 Redis 中，SDS（Simple Dynamic String）是一种用于存储字符串的动态数据结构，它被设计为二进制安全且能够高效地执行字符串操作。然而，SDS 主要用于存储字符串数据，并不直接用于存储整数值。Redis 使用一个专门的数据结构来存储整数类型的数据，这通常是一个长整型（`long`）变量。 + +对于数字类型的自增（INCR）和自减（DECR）操作，Redis 并不是通过 SDS 实现的，而是通过以下方式： + +1. **内存中的整数值**：Redis 的字符串对象内部可能包含一个长整型（`long`）的值，如果该字符串对象被用作计数器（即通过 INCR 或 DECR 命令操作）。 +2. **自增操作（INCR）**：当执行 INCR 命令时，Redis 会获取当前键对应的整数值，将其加一，然后将新的整数值更新到内存中的字符串对象。 +3. **自减操作（DECR）**：类似地，DECR 命令会获取当前键对应的整数值，将其减一，并更新。 +4. **范围检查**：在执行自增或自减操作时，如果整数值超出了 `long long` 类型的范围，Redis 会将值回绕到该类型的最小值或最大值。 +5. **持久化**：如果启用了持久化，Redis 还会将自增或自减操作的结果同步到磁盘上的 RDB 文件或 AOF 日志中。 +6. **事务和原子性**：自增和自减操作是原子性的，即使在多客户端并发访问的情况下，每个操作都能保证正确地执行。 +7. **内存优化**：当字符串对象仅用作计数器时，Redis 会使用内存效率更高的内部表示来存储整数值。 +8. **编码转换**：在某些情况下，如果字符串对象同时包含文本和数字，或者执行了某些操作导致字符串对象不再适合作为整数存储，Redis 可能会在 SDS 和整数之间转换编码。 +9. **使用场景**：自增和自减操作通常用于实现计数器、限制速率、生成唯一序列号等场景。 + +在 Redis 中，数字类型的自增自减操作是直接针对内存中的整数值进行的，而不是通过 SDS 来实现。Redis 的设计确保了这些操作的效率和原子性，使其成为执行这类操作的理想选择。 + + + + +### 🎯 说说 List? + +Redis 列表是简单的字符串列表，按照插入顺序排序。你可以添加一个元素到列表的头部（左边）或者尾部（右边）。 + +- **有序集合**：按插入顺序排序的字符串集合，支持双向操作 + +- **数据结构**：基于链表实现（类似 Java LinkedList），具备以下特性： + + - **高效插入删除**：头尾操作 O(1) 时间复杂度 + + - **低效随机访问**：索引定位 O(n) 时间复杂度 + + - **天然队列特性**：常用于异步任务队列、消息流处理等场景 + +- **底层实现演进** + + 阶段一：双编码模式（Redis 3.2 前） + + 创建新列表时 redis 默认使用 redis_encoding_ziplist 编码， 当以下任意一个条件被满足时， 列表会被转换成 redis_encoding_linkedlist 编码 + + 1. **压缩列表（ziplist）** + - **内存优化结构**：连续内存块存储，每个节点包含 ▶️ 前驱节点长度（prevlen） ▶️ 当前节点编码（encoding） ▶️ 数据内容（content） + - 优势： + - 消除指针内存开销（比传统链表节约 50%+ 内存） + - CPU缓存友好（局部性原理） + - **触发条件**（默认值）： `list-max-ziplist-entries 512`（元素数量）或 `list-max-ziplist-value 64`（单个元素字节数） + + 2. **双向链表（linkedlist）** + + - **传统结构**：包含 prev/next 指针的标准双向链表 + + - 优势： + - 超大元素操作性能稳定 + - 支持快速遍历 + + 阶段二：混合结构（Redis 3.2+ 默认） + + 1. **快速列表（quicklist）** 可以认为quickList，是 ziplist 和 linkedlist 二者的结合；quickList 将二者的优点结合起来。 + + - 实现方式： + - 双向链表节点存储 ziplist 片段，每一个节点是一个quicklistNode，包含prev和next指针，每一个quicklistNode 包含 一个ziplist，*zp 压缩链表里存储键值 + - 单个 ziplist 片段大小通过 `list-max-ziplist-size` 配置（默认 8KB） + + - 设计优势： + - 内存利用率接近 ziplist + - 避免超大ziplist的完整重分配问题 + - 通过控制片段大小平衡内存碎片与操作效率 + +> ##### 压缩列表ziplist +> +> 压缩列表 ziplist 是为 Redis 节约内存而开发的。 +> +>Redis官方对于ziplist的定义是（出自ziplist.c的文件头部注释）： +> +>```text +> /* The ziplist is a specially encoded dually linked list that is designed +> * to be very memory efficient. It stores both strings and integer values, +> * where integers are encoded as actual integers instead of a series of +> * characters. It allows push and pop operations on either side of the list +> * in O(1) time. However, because every operation requires a reallocation of +> * the memory used by the ziplist, the actual complexity is related to the +> * amount of memory used by the ziplist. +> * +> ``` +> +> ziplist 是由一系列特殊编码的内存块构成的列表(像内存连续的数组，但每个元素长度不同)， 一个 ziplist 可以包含多个节点（entry）。 +>ziplist 将表中每一项存放在前后连续的地址空间内，每一项因占用的空间不同，而采用变长编码。 +> +>**ziplist 是一个特殊的双向链表** +> 特殊之处在于：没有维护双向指针:prev next；而是存储上一个 entry的长度和当前entry的长度，通过长度推算下一个元素在什么地方。 +>牺牲读取的性能，获得高效的存储空间，因为(简短字符串的情况)存储指针比存储entry长度 更费内存。这是典型的“时间换空间”。 + + + +### 🎯 字典Hash是如何实现的？Rehash 了解吗？ + +**Redis** 中的字典相当于 Java 中的 **HashMap**，内部实现也差不多类似，都是通过 **“数组 + 链表”** 的 **链地址法** 来解决部分哈希冲突，同时这样的结构也吸收了两种不同数据结构的优点。 + +Redis 字典由 **嵌套的三层结构** 构成，采用链地址法处理哈希冲突： + +```c +// 哈希表结构 +typedef struct dictht { + dictEntry **table; // 二维数组（哈希桶） + unsigned long size; // 总槽位数（2^n 对齐） + unsigned long sizemask; // 槽位掩码（size-1） + unsigned long used; // 已用槽位数量 +} dictht; + +// 字典结构 +typedef struct dict { + dictType *type; // 类型特定函数（实现多态） + void *privdata; // 私有数据 + dictht ht[2]; // 双哈希表（用于渐进式rehash） + long rehashidx; // rehash进度标记（-1表示未进行） +} dict; + +// 哈希节点结构 +typedef struct dictEntry { + void *key; // 键（SDS字符串） + union { + void *val; // 值（Redis对象） + uint64_t u64; + int64_t s64; + } v; + struct dictEntry *next; // 链地址法指针 +} dictEntry; +``` + +字典结构内部包含 **两个 hashtable**，通常情况下只有一个 `hashtable` 有值，但是在字典扩容缩容时，需要分配新的 `hashtable`，然后进行 **渐进式搬迁** *(rehash)*，这时候两个 `hashtable` 分别存储旧的和新的 `hashtable`，待搬迁结束后，旧的将被删除，新的 `hashtable` 取而代之。 + +**扩缩容的条件** + +正常情况下，当 hash 表中 **元素的个数等于第一维数组（第一个hashtable）的长度时**，就会开始扩容，扩容的新数组是 **原数组大小的 2 倍**。不过如果 Redis 正在做 `bgsave(持久化命令)`，为了减少内存过多分离，Redis 尽量不去扩容，但是如果 hash 表非常满了，**达到了第一维数组长度的 5 倍了**，这个时候就会 **强制扩容**。 + +当 hash 表因为元素逐渐被删除变得越来越稀疏时，Redis 会对 hash 表进行缩容来减少 hash 表的第一维数组空间占用。所用的条件是 **元素个数低于数组长度的 10%**，缩容不会考虑 Redis 是否在做 `bgsave` + + + +### 🎯 说说 Zset 吧 + +**它类似于 Java 的 SortedSet 和 HashMap 的结合体**，一方面它是一个 set，保证了内部 value 的唯一性，另一方面它可以给每个 value 赋予一个 score，代表这个 value 的排序权重。它的内部实现用的是一种叫做「跳跃列表」的数据结构。 + +Redis 正是通过 score 来为集合中的成员进行从小到大的排序。Zset 的成员是唯一的，但 score 却可以重复。 + +Zset采用双重数据结构设计，同时使用两种数据结构来优化不同操作： + +1. 跳跃表（skiplist）- 核心排序结构 + +2. 哈希表（dict）- 快速查找结构 + + - 键：成员对象（SDS字符串） + + - 值：对应的score值 + + - 作用：实现O(1)时间复杂度的成员查找和score获取 + +**核心操作实现** + +查找操作 - O(log N) + + 1. 从最高层开始，水平向右查找 + 2. 如果下一节点score > 目标score，下降一层 + 3. 重复直到找到目标或到达最底层 + +插入操作 - O(log N) + + 1. 确定插入位置（类似查找过程） + 2. 随机生成新节点层数 + 3. 更新各层的前向指针和跨度值 + 4. 同时插入哈希表，建立成员→score映射 + +范围查询 - O(log N + M) + + ZRANGE/ZRANGEBYSCORE利用跳跃表的有序性： + + 1. 定位起始位置 - O(log N) + 2. 顺序遍历M个结果 - O(M) + +**编码优化策略** + + Redis根据数据量大小采用不同编码： + + ziplist编码（小数据集）： + + - 触发条件：zset-max-ziplist-entries ≤ 128 且 zset-max-ziplist-value ≤ 64字节 + - 优势：内存紧凑，缓存友好 + - 劣势：插入/删除为O(N) + + skiplist+hashtable编码（大数据集）： + + - 查找成员：O(1) - 通过hashtable + - 范围操作：O(log N) - 通过skiplist + - 内存开销：双重存储成员信息 + + + +### 🎯 跳跃表是如何实现的？原理？ + +![](https://redisbook.readthedocs.io/en/latest/_images/skiplist.png) + +Redis 跳跃表采用 **概率平衡** 的多层链表结构，由三个关键组件构成： + +1. 跳跃表节点结构 (zskiplistNode) + + ```c + typedef struct zskiplistNode { + sds ele; // 成员对象（SDS字符串） + double score; // 排序分值 + struct zskiplistNode *backward; // 后向指针（L0层逆向遍历） + struct zskiplistLevel { + struct zskiplistNode *forward; // 前向指针 + unsigned long span; // 到下一个节点的跨度（用于排名计算） + } level[]; // 柔性数组，层级高度随机生成（1~32） + } zskiplistNode; + ``` + +2. 跳跃表结构 (zskiplist) + + ```c + typedef struct zskiplist { + struct zskiplistNode *header, *tail; // 头尾节点（头节点有32层） + unsigned long length; // 节点总数 + int level; // 最大层数 + } zskiplist; + ``` + +3. 有序集合结构 (zset) + + ```c + typedef struct zset { + dict *dict; // 哈希表（实现 O(1) 查找） + zskiplist *zsl; // 跳跃表（实现范围操作） + } zset; + ``` + + **随机层数生成算法** + +```c +int zslRandomLevel(void) { + int level = 1; + // Redis 5.0 使用位运算优化随机算法 + while ((random()&0xFFFF) < (ZSKIPLIST_P * 0xFFFF)) // ZSKIPLIST_P=0.25 + level += 1; + return (level < ZSKIPLIST_MAXLEVEL) ? level : ZSKIPLIST_MAXLEVEL; // MAXLEVEL=32 +} +``` + +- 概率分布： + - 第1层：100% + - 第2层：25% + - 第3层：6.25% (0.25²) + - ...以此类推 +- **数学期望**：平均层数为 1/(1-p) = 1.33 层（p=0.25） + + + +### 🎯 除了5种基本数据类型，还知道其他数据结构不？ + +**Bitmaps（位图）** + +位图不是实际的数据类型，而是在 String 类型上定义的一组面向位的操作。可以看作是 byte 数组。我们可以使用普通的 get/set 直接获取和设置整个位图的内容，也可以使用位图操作 getbit/setbit 等将 byte 数组看成「位数组」来处理。 + +- **本质**：基于 String 类型的扩展位操作，最大支持 2^32 位 + +- **内存模型**：每个 bit 对应一个偏移量（offset），二进制存储效率极高 + +- 应用场景 + 1. **实时统计**：DAU/MAU 统计（每个用户对应1bit） + 2. **特征标记**：用户权限系统（如 admin:1=权限A，admin:2=权限B） + 3. **布隆过滤器**：通过多个位组合实现概率型存在判断 + +**HyperLogLog（基数统计）** + +- **原理**：基于概率算法估算集合基数，标准误差 0.81% +- **数据结构**：16384 个 6bit 寄存器（总占用 12KB 固定内存） + +```redis +PFADD uv:page:123 user1 user2 # 添加访问用户 +PFCOUNT uv:page:123 # 获取UV估算值 +PFMERGE total_uv uv:page:* # 合并多日UV数据 +``` + +- 应用场景 + 1. **流量统计**：网站 UV/广告曝光量 + 2. **去重分析**：大规模数据集近似去重 + 3. **实时大屏**：实时显示千万级用户量 + +**Geospatial（地理空间）** + +- **底层结构**：使用 ZSET 存储，score 为 Geohash 编码值 +- **编码原理**：将经纬度转换为 52 位整型（精度可达厘米级） + +```Redis +GEOADD cities 116.405285 39.904269 "北京" # 添加坐标 +GEODIST cities 北京 上海 km # 计算两地距离 +GEORADIUS cities 116 39 100 km WITHCOORD # 半径100km内城市 +``` + +- **应用场景** + 1. **LBS服务**：附近的人/商铺搜索 + 2. **轨迹分析**：移动目标位置追踪 + 3. **区域监控**：电子围栏报警系统 + +**Streams（流）** + +- **数据结构**：持久化消息链表 + 消费者组管理 + +- **核心组件**： + + - `Message ID`：时间戳-序列号（如 `1526569415636-0`） + + - `Consumer Group`：支持消息的负载均衡和故障转移 + +```Redis +XADD orders * product_id 1001 # 添加订单事件 +XREAD COUNT 10 STREAMS orders $ # 读取最新消息 +XGROUP CREATE orders group1 $ # 创建消费者组 +``` + +**应用场景** + +1. **消息队列**：订单处理/日志收集 +2. **事件溯源**：用户行为追踪 +3. **审计日志**：操作记录永久保存 + +**Bloom Filter（布隆过滤器）** + +- **实现方式**：通过多个 Bitmaps + 哈希函数实现 + +- **特性**： + + - 判断存在时可能有误报（false positive） + + - 判断不存在时绝对准确 + +```Redis +BF.RESERVE user_filter 0.01 1000000 # 初始化过滤器（误差率1%） +BF.ADD user_filter 1001 # 添加用户ID +BF.EXISTS user_filter 1001 # 检查是否存在 +``` + +- **应用场景** + 1. **缓存穿透防护**：拦截非法请求 + 2. **推荐去重**：避免重复推荐内容 + 3. **爬虫URL判重**：过滤已抓取链接 + + + +### 🎯 怎么统计一亿用户的日活，hyperloglog有什么缺点，bitmap不行么? + +统计一亿用户的日活（Daily Active Users, DAU）时，需要一个能够高效处理和存储大量数据的方案。以下是几种适用于此类场景的数据结构及其优缺点： + +HyperLogLog 是一种用于基数统计的数据结构，它提供了一个近似的、不精确的解决方案来估算集合中唯一元素的数量。 + +**优点**： + +- **内存效率**：HyperLogLog 消耗的内存固定，与集合中元素的数量无关，通常每个 HyperLogLog 实例只需要 12.4KB 左右，无论集合中有多少元素。 +- **性能**：HyperLogLog 可以快速处理数据，因为它只存储元素的哈希值的一些位信息。 + +**缺点**： + +- **近似值**：HyperLogLog 提供的是近似值，标准误差大约为 0.81%，这意味着实际基数可能与估算值有所偏差。 +- **更新频率**：如果数据更新非常频繁，HyperLogLog 可能需要频繁地调整其内部数据结构，这可能会影响性能。 + +Bitmap 是另一种数据结构，它使用位数组来表示数据，每个位对应一个元素的状态（例如，用户是否活跃）。 + +**优点**： + +- **精确计数**：Bitmap 提供精确的计数，没有 HyperLogLog 的近似误差。 +- **简单直观**：Bitmap 的概念简单，易于理解和实现。 + +**缺点**： + +- **内存消耗**：对于一亿用户，Bitmap 需要大约 100MB（1亿位 / 8位/字节 ÷ 1024KB/MB）的内存，这比 HyperLogLog 高得多。 +- **扩展性问题**：随着用户数量的增加，Bitmap 所需的内存也会线性增长，这可能在大规模数据集上造成问题。 + +**综合考虑** + +- 如果对日活用户数的精确度要求不高，并且希望最小化内存使用，HyperLogLog 是一个很好的选择。 +- 如果需要精确的日活用户数，并且可以承受较高的内存消耗，可以使用 Bitmap。 +- 在实际应用中，可能还会考虑其他因素，如数据的更新频率、系统的扩展性、维护成本等。 + +对于一亿用户的日活统计，如果对精度要求不是特别高，HyperLogLog 是一个更合适的选择，因为它在内存使用和性能方面具有优势。Bitmap 虽然可以提供精确的统计，但其内存消耗较高，可能不适合大规模的用户统计。 + + + +### 🎯 这些都会，那你能说说 Redis 使用场景不，你们项目中是怎么用的？ + +在 Redis 中，常用的 5 种数据结构和应用场景如下： + +- **String**：缓存、计数器、分布式锁等。 + - 什么是计数器，如电商网站商品的浏览量、视频网站视频的播放数等。为了保证数据实时效，每次浏览都得+1，并发量高时如果每次都请求数据库操作无疑会对数据库提出挑战。Redis提供的incr命令来实现计数器功能，内存操作，性能非常好，非常适用于这些计数场景。 + - 在很多互联网公司中都使用了分布式技术，分布式技术带来的技术挑战是对同一个资源的并发访问，如全局ID、减库存、秒杀等场景，并发量不大的场景可以使用数据库的悲观锁、乐观锁来实现，但在并发量高的场合中，利用数据库锁来控制资源的并发访问是不太理想的，大大影响了数据库的性能。可以利用Redis的setnx功能来编写分布式的锁，如果设置返回1说明获取锁成功，否则获取锁失败，实际应用中要考虑的细节要更多。 + +- **List**：链表、队列、微博关注人时间轴列表等。 + - Redis列表结构，LPUSH可以在列表头部插入一个内容ID作为关键字，LTRIM可用来限制列表的数量，这样列表永远为N个ID，无需查询最新的列表，直接根据ID去到对应的内容页即可。 +- **Hash**：用户信息、Hash 表等。 +- **Set**：**社交网络** + - 点赞、踩、关注/被关注、共同好友等是社交网站的基本功能，社交网站的访问量通常来说比较大，而且传统的关系数据库类型不适合存储这种类型的数据，Redis提供的哈希、集合等数据结构能很方便的的实现这些功能。 +- **Zset**：访问量排行榜、点击量排行榜等 + +还有一些，比如： + +- 取最新N个数据的操作 + +- 需要精确设定过期时间的应用 + +- Uniq操作,获取某段时间所有数据排重值 + +- 实时系统,反垃圾系统 + +- Pub/Sub构建实时消息系统 + +- 构建队列系统 + +- **分布式会话** + + 集群模式下，在应用不多的情况下一般使用容器自带的session复制功能就能满足，当应用增多相对复杂的系统中，一般都会搭建以Redis等内存数据库为中心的session服务，session不再由容器管理，而是由session服务及内存数据库管理。 + +------ + + + +## 三、持久化与内存管理 + +### 🎯 你对 Redis 的持久化机制了解吗？能讲一下吗？ + +> 或者不会这么直白的问，而是问 Redis 是如何实现数据不丢失的? + +Redis 的数据全部在内存里，如果突然宕机，数据就会全部丢失，因此必须有一种机制来保证 Redis 的数据不会因为故障而丢失，这种机制就是 Redis 的持久化机制，它会将内存中的数据库状态 **保存到磁盘** 中。 + +> 回答思路：先说明 Redis 有几种持久化的方式，然后分析 AOF 和 RDB 的原理以及存在的问题，最后分析一下 Redis 4.0 版本之后的持久化机制。 + + + +### 🎯 Redis 持久化的方式有哪写？ + +Redis的持久化机制主要有两种：RDB和AOF，以及两者的混合模式。 + +1. **RDB（Redis Database）** + +- **原理**：通过生成内存数据的**快照**（Snapshot）持久化到磁盘。触发方式包括手动（`SAVE`/`BGSAVE`）和自动（配置`save`规则，如`save 900 1`表示900秒内至少1次修改则触发）。 + + rdb 默认保存的是 **dump.rdb** 文件 + + **在指定的时间间隔内将内存中的数据集快照写入磁盘**，也就是行话讲的 Snapshot 快照，它恢复时是将快照文件直接读到内存里。 + + Redis会单独创建（fork）一个子进程来进行持久化，会先将数据写入到一个临时文件中，待持久化过程都结束了，再用这个临时文件替换上次持久化好的文件。整个过程中，主进程是不进行任何 IO 操作的，这就确保了极高的性能，如果需要进行大规模数据的恢复，且对于数据恢复的完整性不是非常敏感，那 RDB 方式要比 AOF 方式更加的高效。RDB的缺点是最后一次持久化后的数据可能丢失。 + + > What ? Redis 不是单进程的吗? + + Redis 使用操作系统的多进程 COW(Copy On Write) 机制来实现快照持久化， fork 是类Unix操作系统上**创建进程**的主要方法。COW(Copy On Write)是计算机编程中使用的一种优化策略。 + + fork 的作用是复制一个与当前进程一样的进程。新进程的所有数据（变量、环境变量、程序计数器等）数值都和原进程一致，但是是一个全新的进程，并作为原进程的子进程。 子进程读取数据，然后序列化写到磁盘中。 + +- 优点： + + - **性能高**：通过`BGSAVE`后台生成快照，不影响主进程。 + - **文件紧凑**：二进制格式，适合备份与灾难恢复。 + - **快速重启**：加载RDB文件恢复数据比AOF更快。 + +- 缺点： + + - **数据丢失风险**：最后一次快照后的修改可能丢失（依赖配置的触发频率）。 + - **大内存Fork开销**：频繁快照时，写时复制（Copy-On-Write）可能导致内存占用陡增。 + +2. **AOF（Append-Only File）** + +- 原理：记录每个写操作命令（如 `SET`/`DEL`），追加到文件末尾。支持三种同步策略： + - **always**：每个命令同步写入磁盘（安全，但性能最低）。 + - **everysec**：每秒批量同步（默认，最多丢失1秒数据）。 + - **no**：依赖操作系统刷盘（性能高，但丢失数据风险最大）。 +- 优点： + - **数据更安全**：可配置为秒级/命令级持久化。 + - **可读性强**：AOF文件为文本格式，便于人工分析。 +- 缺点： + - **文件体积大**：长期运行后文件膨胀，需定期执行`BGREWRITEAOF`重写压缩。 + - **恢复速度慢**：重放所有命令恢复数据，比RDB耗时。 + +3. **混合持久化（Redis 4.0+）** + +- **原理**：结合RDB和AOF，AOF文件由**RDB头部（全量数据） + AOF尾部（增量命令）**组成，通过配置`aof-use-rdb-preamble yes`启用。 +- 优点： + - **快速恢复**：先加载RDB快照，再重放增量命令，兼顾速度与数据完整性。 + - **文件更小**：相比纯AOF，混合模式文件体积更小。 + +4. **选型与配置建议** + +- 场景选择： + - **RDB**：允许分钟级数据丢失，追求高性能备份（如缓存场景）。 + - **AOF**：对数据安全性要求高（如金融场景）。 + - **混合模式**：综合两者优势，推荐生产环境使用。 +- 注意事项： + - 同时启用RDB和AOF时，Redis重启**优先加载AOF**文件。 + - 监控`fork`耗时及内存压力，避免频繁快照导致性能抖动。 + +**总结**：理解RDB和AOF的机制与权衡，结合业务需求选择持久化策略，混合模式通常是平衡性能与安全性的最佳实践。 + + + +### 🎯 RDB 做快照时会阻塞线程吗？ + +因为 Redis 的单线程模型决定了它所有操作都要尽量避免阻塞主线程，所以对于 RDB 快照也不例外，这关系到是否会降低 Redis 的性能。 + +为了解决这个问题，Redis 提供了两个命令来生成 RDB 快照文件，分别是 save 和 bgsave。save 命令在主线程中执行，会导致阻塞。而 bgsave 命令则会创建一个子进程，用于写入 RDB 文件的操作，避免了对主线程的阻塞，这也是 Redis RDB 的默认配置。 + + + +### 🎯 RDB 做快照的时候数据能修改吗？ + +Redis利用操作系统的写时复制机制，使得在RDB快照生成过程中，主线程仍然可以正常处理写操作： + +> 它利用了 bgsave 的子进程，具体操作如下： +> +> 如果主线程执行读操作，则主线程和 bgsave 子进程互相不影响； +> +> 如果主线程执行写操作，则被修改的数据会复制一份副本，然后 bgsave 子进程会把该副本数据写入 RDB 文件，在这个过程中，主线程仍然可以直接修改原来的数据。 + +``` + 主进程 (Redis主线程) 子进程 (BGSAVE) + | | + 处理写请求 读取内存数据 + | | + 修改数据页 生成RDB文件 + | | + 触发COW复制 | +``` + + 具体工作流程 + + 1. BGSAVE执行时 + + - Redis通过fork()创建子进程 + + - 子进程继承父进程的内存空间（共享物理内存） + + - 子进程开始遍历内存数据生成RDB快照 + + 2. 读操作处理 + + - 主线程执行读操作：不影响快照生成 + + - 子进程读取数据：直接访问共享内存 + + - 两者互不干扰，性能良好 + + 3. 写操作处理（COW机制核心） + + 写操作发生时： + + - 主线程要修改某个数据页 + + - 操作系统检测到写入操作 + + - 复制该数据页到新的物理地址 + + - 主线程在新页面上执行写操作 + + - 子进程继续访问原始页面数据 + + + +### 🎯 AOF 日志是如何实现的？ + +通常情况下，关系型数据库（如 MySQL）的日志都是“写前日志”（Write Ahead Log, WAL），也就是说，在实际写数据之前，先把修改的数据记到日志文件中，以便当出现故障时进行恢复，比如 MySQL 的 redo log（重做日志），记录的就是修改后的数据。 + +而 AOF 里记录的是 Redis 收到的每一条命令，这些命令是以文本形式保存的，不同的是，Redis 的 AOF 日志的记录顺序与传统关系型数据库正好相反，它是写后日志，“写后”是指 Redis 要先执行命令，把数据写入内存，然后再记录日志到文件。 + +那么面试的考察点来了：Reids 为什么先执行命令，在把数据写入日志呢？为了方便你理解，我整理了关键的记忆点： + +- 因为 ，Redis 在写入日志之前，不对命令进行语法检查； + +- 所以，只记录执行成功的命令，避免了出现记录错误命令的情况； + +- 并且，在命令执行完之后再记录，不会阻塞当前的写操作。 + +当然，这样做也会带来风险（这一点你也要在面试中给出解释）。 + +- 数据可能会丢失： 如果 Redis 刚执行完命令，此时发生故障宕机，会导致这条命令存在丢失的风险。 + +- 可能阻塞其他操作： 虽然 AOF 是写后日志，避免阻塞当前命令的执行，但因为 AOF 日志也是在主线程中执行，所以当 Redis 把日志文件写入磁盘的时候，还是会阻塞后续的操作无法执行。 + + + +### 🎯 AOF 如果文件越来愈大 怎么办？ + +**rewrite（AOF 重写）** + +- 是什么：AOF采用文件追加方式，文件会越来越大为避免出现此种情况，新增了重写机制，当 AOF 文件的大小超过所设定的阈值时，Redis就会启动 AOF 文件的内容压缩，只保留可以恢复数据的最小指令集，可以使用命令 `bgrewriteaof`，这个操作相当于对 AOF文件“瘦身”。 +- 重写原理：AOF 文件持续增长而过大时，会 fork 出一条新进程来将文件重写(也是先写临时文件最后再rename)，遍历新进程的内存中数据，每条记录有一条的 Set 语句。重写 aof 文件的操作，并没有读取旧的aof文件，而是将整个内存中的数据库内容用命令的方式重写了一个新的 aof 文件，这点和快照有点类似 +- 触发机制：Redis 会记录上次重写时的 AOF 大小，默认配置是当 AOF 文件大小是上次 rewrite 后大小的一倍且文件大于64M 时触发 + + + +### 🎯 fork 耗时问题定位 + +**Fork操作** + +当Redis做RDB或AOF重写时，一个必不可少的操作就是执行fork操作创建子进程，对于大多数操作系统来说fork是个重量级操作 + +虽然fork创建的子进程不需要拷贝父进程的物理内存空间，但是会复制父进程的空间内存页表。例如对于10GB的Redis进程，需要复制大约20MB的内存页表，因此fork 操作耗时跟进程总内存量息息相关，如果使用虚拟化技术，特别是Xen虚拟 机，fork操作会更耗时 + +- 在做 RDB 或 AOF 重写时, fork 是必不可少的 +- 对于大多数操作系统来说, fork 是个重量级错误 +- fork 会复制符进程的空间内存页表 +- 如果使用虚拟化技术, 特别是 Xen 虚拟机, fork 操作会更耗时 + +**fork 耗时问题定位**: + +- 高流量的 Redis 实例 ops 可达5万以上 +- 正常情况 fork 耗时应该是每 GB 消耗 20ms 左右 +- 可以用 info stats 命令查看 latest_fork_usec 指标, 获取最近一次 fork 操作耗时, 单位微秒 + +**如何改善 fork 操作的耗时**: + +- 优先使用物理机或者高效支持 fork 操作的虚拟化技术, 避免使用 Xen +- 控制 Redis 实例最大可用内存, fork 耗时跟内存量成正比, 线上建议每个 Redis 实例内存控制在 10GB 以内 +- 合理配置 Linux 内存分配策略, 避免物理内存不足导致 fork 失败 +- 降低 fork 操作的频率, 如适度放宽 AOF 自动触发时机, 避免不必要的全量复制等 + + + +### 🎯 两种持久化方式如何选择？ + +- RDB 持久化方式能够在指定的时间间隔能对你的数据进行快照存储，启动快、文件小，但会丢“快照周期内”的数据。 +- AOF 持久化方式记录每次对服务器写的操作,当服务器重启的时候会重新执行这些命令来恢复原始的数据，AOF命令以 redis 协议追加保存每次写的操作到文件末尾。Redis还能对AOF文件进行后台重写（**bgrewriteaof**）,使得 AOF 文件的体积不至于过大 +- 只做缓存：如果你只希望你的数据在服务器运行的时候存在,你也可以不使用任何持久化方式。 +- 同时开启两种持久化方式 + - 在这种情况下,当 redis 重启的时候会优先载入 AOF 文件来恢复原始的数据,因为在通常情况下 AOF 文件保存的数据集要比 RDB 文件保存的数据集要完整。 + - RDB 的数据不实时，同时使用两者时服务器重启也只会找 AOF 文件。那要不要只使用AOF 呢？建议不要，因为 RDB 更适合用于备份数据库(AOF 在不断变化不好备份)，快速重启，而且不会有 AOF 可能潜在的bug，留着作为一个万一的手段。 + +Redis4.0 之后有了混合持久化的功能，将 bgsave 的全量 和 aof 的增量做了融合处理，这样既保证了恢复的效率又兼顾了数据的安全性。bgsave 的 原理，fork 和 cow, fork 是指 redis 通过创建子进程来进行 bgsave 操作，cow 指的是 copy on write，子进程创建后，父子进程共享数据段，父进程继续提供读写服务，写脏的页面数据 会逐渐和子进程分离开来。 + + + +### 🎯 Redis 过期键的删除策略？ + +先抛开 Redis 想一下几种可能的删除策略： + +1. **定时删除**：在设置键的过期时间的同时，创建一个定时器 timer. 让定时器在键的过期时间来临时，立即执行对键的删除操作。 +2. **惰性删除**：放任键过期不管，但是每次从键空间中获取键时，都检查取得的键是否过期，如果过期的话，就删除该键;如果没有过期，就返回该键。 +3. **定期删除**：每隔一段时间程序就对数据库进行一次检查，删除里面的过期键。至于要删除多少过期键，以及要检查多少个数据库，则由算法决定。 +4. **延迟队列**：也就是把对象放到一个延迟队列里面。当从队列里取出这个对象的时候，就说明它已经过期了，这时候就可以删除。 + +在上述的几种策略中定时删除和定期删除属于不同时间粒度的 **主动删除**，惰性删除属于 **被动删除**。 + +**四种策略都有各自的优缺点** + +1. 定时删除对内存使用率有优势，但是对 CPU 不友好； +2. 惰性删除对内存不友好，如果某些键值对一直不被使用，那么会造成一定量的内存浪费； +3. 延迟删除，队列本省就有开销 +4. 定期删除，时间不准确，性能损耗不可控，如果触发删除的时候，很多已经过期了，那么当期定时删除就很耗时； + +**Redis 中的实现** + +Redis 过期键的删除策略有三种主要方法：**定期删除**（Scheduled Deletion）、**惰性删除**（Lazy Deletion）和**主动删除**（Active Deletion）。这些策略结合使用，以确保在性能和内存之间取得平衡。 + +**1. 定期删除（Scheduled Deletion）** + +Redis 定期检查键的过期时间并删除过期键。这个过程是通过 Redis 的后台任务进行的，每隔一段时间会随机检查一部分键，删除其中已过期的键。 + +- **实现方式**：默认情况下，Redis 每隔 100 毫秒运行一次过期扫描任务，检查设置了过期时间的键，并删除过期的键。 +- **优点**：分摊了删除操作的开销，避免了集中删除大量键导致的性能问题。 +- **缺点**：不能保证过期键在过期后立即被删除，可能会存在一段时间的滞留。 + +**2. 惰性删除（Lazy Deletion）** + +当客户端访问某个键时，Redis 会检查该键是否过期，如果过期则立即删除。 + +`expireIfNeeded` 的作用是， 如果输入键已经过期的话， 那么将键、键的值、键保存在 `expires` 字典中的过期时间都删除掉。 + +- **实现方式**：每次访问键时，都会进行过期时间检查，若键已过期则删除该键并返回空结果。 +- **优点**：确保访问时一定不会返回已过期的键，删除操作与键的访问相结合，不额外消耗 CPU 资源。 +- **缺点**：如果某些过期键长时间不被访问，它们将继续占用内存，直到被定期删除任务或其他方式删除。 + +**3. 主动删除（Active Deletion）** + +当 Redis 内存不足时，会主动扫描并删除过期键，以释放内存。 + +- **实现方式**：Redis 配置了内存淘汰策略（如 `volatile-lru`、`allkeys-lru` 等），当内存达到限制时，Redis 会通过删除过期键来释放内存。 +- **优点**：确保在内存不足时能够及时释放内存，避免系统因内存不足崩溃。 +- **缺点**：这种方式通常作为内存淘汰策略的一部分，不单独使用。 + +> 在 Redis 的 3.2 版本之前，如果读从库的话，是有可能读取到已经过期的key。后来在 3.2 版本之后这个 Bug 就被修复了。不过从库上的懒惰删除特性和主库不一样。主库上的懒惰删除是在发现 key 已经过期之后，就直接删除了。但是在从库上，即便 key 已经过期了，它也不会删除，只是会给你返回一个 NULL 值。 + + + +### 🎯 Redis 的淘汰策略有哪些？ + +**Redis 有八种淘汰策略** + +为了保证 Redis 的安全稳定运行，设置了一个 `max-memory` 的阈值，那么当内存用量到达阈值，新写入的键值对无法写入，此时就需要内存淘汰机制，在 Redis 的配置中有几种淘汰策略可以选择，详细如下： + +| 策略 | 描述 | +| --------------- | ------------------------------------------------------------ | +| volatile-lru | 从已设置过期时间的 KV 集中优先对最近最少使用(less recently used)的数据淘汰 | +| volitile-ttl | 从已设置过期时间的 KV 集中优先对剩余时间短(time to live)的数据淘汰 | +| volitile-random | 从已设置过期时间的 KV 集中随机选择数据淘汰 | +| allkeys-lru | 从所有 KV 集中优先对最近最少使用(less recently used)的数据淘汰 | +| allKeys-random | 从所有 KV 集中随机选择数据淘汰 | +| noeviction | 不淘汰策略，若超过最大内存，返回错误信息 | + +**4.0 版本后增加以下两种** + +- volatile-lfu：从已设置过期时间的数据集(server.db[i].expires)中挑选最不经常使用的数据淘汰 +- allkeys-lfu：当内存不足以容纳新写入数据时，在键空间中，移除最不经常使用的 key + + + +### 🎯 一组曾经是热点数据，后面不是了，对于lru和lfu处理时会有什么区别？ + +> **LRU（Least Recently Used）**：基于**时间维度**，淘汰最久未被访问的数据。 +> +> **LFU（Least Frequently Used）**：基于**频率维度**，淘汰一定时期内访问次数最少的数据。 + +**曾经是热点数据，后来不再是热点，对于 LRU 和 LFU 的处理区别** + +- **对于 LRU 策略**： + - **行为**：当一组数据曾经是热点，被频繁访问，其最近访问时间会被更新为最新值。然而，当它们不再被访问时，随着时间推移，其时间戳会逐渐变旧。 + - **淘汰机制**：当需要淘汰数据时，LRU 会选择那些最近最少被访问的键，即时间戳最早的键。因此，这些曾经的热点数据会因为长时间未被访问而被优先淘汰。 + - **效果**：LRU 能够较快地清除不再被使用的旧热点数据，为新的数据腾出空间。 +- **对于 LFU 策略**： + - **行为**：曾经是热点的数据，其访问计数器较高，即使之后不再被访问，其计数值仍然保持在高位。 + - **淘汰机制**：LFU 会选择访问频率最低的键进行淘汰。由于旧热点数据的计数器较高，它们不会被立即淘汰。 + - **衰减机制**：Redis 为了避免旧热点数据长期占用内存，引入了计数器衰减机制。计数器会随着时间逐渐降低，如果键长时间未被访问，计数器会减小，最终可能被淘汰。 + - **效果**：LFU 会更长时间地保留曾经的热点数据，即使它们近期未被访问。这在一定程度上保护了历史上重要的数据，但也可能导致旧数据占用内存空间。 + + + +### 🎯 Redis 内存满了怎么办 + +1. 增加内存； +2. 使用内存淘汰策略（redis设置配置文件的***maxmemory**参数，可以控制其最大可用内存大小，可以通过配置 **maxmemory-policy** 设置淘汰策略） +3. 压缩数据 +4. 集群 + + + +### 🎯 Redis 线程模型 + +Redis 的线程模型是单线程事件驱动模型，这意味着它在处理客户端请求时使用单个线程。然而，Redis 使用 I/O 多路复用技术来高效地管理多个客户端连接。以下是 Redis 线程模型的主要特点和工作原理： + +**单线程模型** + +1. **单线程架构**： + - Redis 主要使用单个线程来处理所有客户端请求，包括读写操作和命令执行。这使得 Redis 的实现相对简单且高效，因为不需要处理多线程并发问题，如锁竞争和线程同步。 +2. **I/O 多路复用**： + - Redis 使用 I/O 多路复用（I/O multiplexing）技术，通过 `epoll`（Linux）、`kqueue`（BSD）或 `select` 等系统调用来同时处理多个客户端连接。I/O 多路复用允许 Redis 在一个线程内同时处理大量连接而不会阻塞。 + - I/O 多路复用机制使得 Redis 可以在一个事件循环中处理多个套接字的 I/O 事件，从而提高并发处理能力。 + +**事件驱动模型** + +Redis 使用事件驱动模型（Event-Driven Model），在主线程的事件循环中执行各种事件，包括网络事件和定时事件。事件驱动模型确保 Redis 可以高效地处理大量并发请求。 + +1. **事件循环**： + - Redis 的事件循环不断地检查网络事件（如客户端连接和数据传输）和定时事件（如键过期检查）。当事件发生时，调用相应的事件处理函数进行处理。 +2. **事件处理器**： + - Redis 将网络 I/O 操作和命令执行分成不同的事件处理器。网络 I/O 处理器负责接受客户端连接、读取请求和发送响应，命令处理器负责执行具体的 Redis 命令。 + +**多线程 I/O 模式** + +虽然 Redis 本质上是单线程模型，但在 6.0 版本开始引入了有限的多线程支持，用于处理 I/O 操作。通过启用 I/O 线程，可以在处理网络 I/O 时利用多核 CPU，从而提高性能。 + +1. I/O 线程 + + - Redis 6.0 及更高版本可以配置多个 I/O 线程用于读取和处理客户端请求，但命令执行仍然在主线程中进行。这样可以减少 I/O 操作对主线程的阻塞，提高整体吞吐量。 + +2. 配置示例： 在 `redis.conf`中启用和配置 I/O 线程： + + ```shel + io-threads-do-reads yes + io-threads 4 # 配置 I/O 线程数 + ``` + +> Redis 基于 Reactor 模式开发了网络事件处理器，这个处理器被称为文件事件处理器（file event handler）。它的组成结构为4部分：多个套接字、IO多路复用程序、文件事件分派器、事件处理器。因为文件事件分派器队列的消费是单线程的，所以 Redis 才叫单线程模型。 +> +> 文件事件处理器使用 I/O 多路复用（multiplexing）程序来同时监听多个套接字，并根据套接字目前执行的任务来为套接字关联不同的事件处理器。 +> +> 当被监听的套接字准备好执行连接应答（accept）、读取（read）、写入（write）、关闭（close）等操作时， 与操作相对应的文件事件就会产生，这时文件事件处理器就会调用套接字之前关联好的事件处理器来处理这些事件。 +> +> 虽然文件事件处理器以单线程方式运行， 但通过使用 I/O 多路复用程序来监听多个套接字，文件事件处理器既实现了高性能的网络通信模型， 又可以很好地与 redis 服务器中其他同样以单线程方式运行的模块进行对接， 这保持了 Redis 内部单线程设计的简单性。 + + + +### 🎯 Redis事件模型 + +Redis 事件模型基于 **Reactor 模式 + I/O 多路复用**，单线程事件循环负责监听 socket 的可读/可写事件。 + +当事件就绪时，主线程回调处理： + +- 可读事件 → 读取客户端请求 +- 可写事件 → 发送响应 + +6.0 之后增加 I/O 线程专门处理网络读写，但命令解析和执行仍在主线程完成。 + Redis 还使用定时器事件处理周期性任务，如过期 key 删除、AOF 重写、慢日志检查等。 + + + +### 🎯 什么是 Reactor 模式？ + +**Reactor 模式**是一种基于事件驱动的设计模式，广泛应用于高性能网络服务开发中，例如 Redis、Netty 和 Java 的 NIO 编程中。它通过非阻塞 IO 和事件循环的方式高效地处理并发请求，避免了线程阻塞和资源浪费问题。Redis 的高性能在很大程度上依赖于这种模式。 + +**Reactor 模式的基本原理** + +Reactor 模式将应用程序的请求处理分为以下几个关键角色： + +1. **Reactor (事件分发器)**： + - 负责监听事件并将其分发给相应的事件处理器。 + - Reactor 运行在单线程或多线程上，集中管理 IO 操作。 +2. **事件源 (资源提供者)**：事件源是产生 IO 事件的实体，例如网络连接、文件描述符等。 +3. **事件处理器 (Handlers)**：负责具体的事件处理逻辑，例如读取数据、处理业务逻辑、写回数据等。 +4. **Demultiplexer (IO 多路复用器)**：使用操作系统提供的 IO 多路复用机制（如 `select`、`poll` 或 `epoll`）监听多个 IO 通道，将准备好的事件传递给 Reactor。 + +**Reactor 模式的工作流程** + +1. **事件监听**：Reactor 使用多路复用器不断监听 IO 事件，如读事件、写事件或连接事件。 +2. **事件分发**：当有事件发生时，Reactor 从多路复用器获取事件，并将其分发给对应的事件处理器。 +3. **事件处理**：事件处理器根据事件类型执行相应的操作（如读取数据、业务处理、写回数据等）。 + +**Redis 中的 Reactor 模式** + +Redis 是一个基于事件驱动模型的高性能内存数据库，单线程模型的高效运转很大程度上归功于 Reactor 模式。 + +**Redis 的工作流程：** + +1. **单线程事件循环**：Redis 使用单线程来处理所有客户端的请求，通过 IO 多路复用机制（如 `epoll`）监听多个连接上的 IO 事件。 +2. **事件分发**：当某个连接上有事件发生（如可读、可写），事件被分发给对应的处理器。 +3. **事件处理**：Redis 的事件处理器包括： + - **文件事件处理器**：负责处理客户端请求、网络 IO 操作。 + - **时间事件处理器**：处理定时任务，例如持久化、清理过期键等。 + +**Redis 的事件模型细节：** + +- Redis 的事件模型实现基于 `ae.c` 文件（Async Event）。 +- 它通过 **文件事件处理器**（处理网络 IO）和 **时间事件处理器**（处理定时任务）协同工作。 +- **文件事件处理器**利用操作系统的 IO 多路复用机制监听客户端的请求，单线程模型避免了多线程带来的复杂性（如线程同步问题）。 + + + +### 🎯 Redis 内存模型 + +Redis 内存主要可以分为：数据部分、Redis进程本身、缓冲区内存、内存碎片这四个部分。Redis 默认通过jemalloc 来分配内存。 + +- **数据内存**：数据内存用来存储 Redis 的键值对、慢查询日志等，是主要占用内存的部分，这部分内存会统计在used_memory中 + +- **Redis进程内存**：Redis进程本身也会占用一部分内存，这部分内存不是jemalloc分配，不会统计在used_memory中。执行RDB和AOF时创建的子进程也会占用内存，但也不会统计在used_memory中。 + +- **缓冲内存**（动态管理）： + + - 客户端缓冲区：存储客户端连接的输入/输出数据，通过 `client-output-buffer-limit` 限制大小，防止溢出。 + - 复制积压缓冲区：用于主从复制的增量同步（`PSYNC`），大小由 `repl-backlog-size` 配置，默认 1MB。 + - AOF缓冲区：暂存最近写入的命令，持久化时同步到磁盘。 + + **分配方式**：由 `jemalloc` 分配，计入 `used_memory` + +- **内存碎片**： + + - 来源：频繁的键值增删及 jemalloc 内存块分配策略导致未完全利用的内存空间。 + - 监控指标： + - 碎片率：`mem_fragmentation_ratio = used_memory_rss / used_memory`，健康值约 1.03（jemalloc）。 + - 若碎片率 >1.5，需考虑优化；>2 可能触发性能问题。 + - 优化手段： + - 重启重排：安全重启后通过 RDB/AOF 恢复数据，内存重新分配以减少碎片。 + - **自动碎片整理（Redis 4.0+）**：开启 `activedefrag`，根据阈值动态整理碎片 + +------ + + + +## 四、事务与脚本 + +### 🎯 Redis事务？ + +**Redis事务是通过`MULTI`、`EXEC`、`WATCH`等命令实现的一种批量操作机制**，它的核心特点是**将多个命令打包成一个原子化操作序列**，保证以下特性： + +1. **事务的三大核心命令** + + - **`MULTI`**：标记事务开始，后续所有命令会被放入队列缓存，返回`QUEUED`表示命令已入队，它总是返回 OK。 + + - **`EXEC`**：触发事务执行，按顺序执行队列中的所有命令，返回所有命令的结果。 + + - **`DISCARD`**：取消事务，清空命令队列，放弃事务执行。 + +2. **事务的四大特性** + + - 原子性（Partial）： + - 若事务中某条命令**语法错误**（如命令不存在），整个事务在`EXEC`时会被拒绝执行。 + - 若命令**运行时错误**（如对字符串执行`INCR`），错误命令会抛出异常，**但其他命令仍会执行**（不支持回滚）。 + + - **隔离性**：事务执行期间不会被其他客户端命令打断（串行化执行）。 + + - **一致性**：事务完成后，数据符合Redis的校验规则（如数据类型约束）。 + + - **持久性**：取决于Redis的持久化配置（与RDB/AOF相关）。 + +3. **`WATCH`命令与乐观锁** + + - **作用**：监控一个或多个Key，若在事务提交前（`EXEC`执行前）这些Key的值被其他客户端修改，则当前事务会被**主动放弃**（返回`nil`）。 + + - **本质**：基于`CAS`（Compare And Swap）的乐观锁机制，解决并发冲突问题。 + + - 示例： + + ```Shell + WATCH balance # 监听balance键 + MULTI + DECRBY balance 100 + INCRBY debt 100 + EXEC # 若balance被其他客户端修改，此处返回nil + ``` + +4. **与关系型数据库事务的差异** + + - **不支持回滚（Rollback）**：Redis 设计哲学强调简单高效，开发者需自行处理错误逻辑。 + + - **无隔离级别概念**：所有命令串行执行，天然隔离。 + + - **批量执行而非原子性保证**：更适用于批量操作场景，而非强一致事务。 + +5. **适用场景** + + - **批量操作**：例如批量更新用户状态。 + + - **简单一致性控制**：配合`WATCH`实现余额扣减、库存抢购等场景。 + + - **非强事务需求**：容忍部分失败（如日志记录）。 + + + +### 🎯 Redis事务的三个阶段、三特性 + +**三阶段** + +1. 开启：以MULTI开始一个事务 + +2. 入队：将多个命令入队到事务中，接到这些命令并不会立即执行，而是放到等待执行的事务队列里面 + +3. 执行：由EXEC命令触发事务 + +**三特性** + +1. 单独的隔离操作：事务中的所有命令都会序列化、按顺序地执行。事务在执行的过程中，不会被其他客户端发送来的命令请求所打断。 + +2. **没有隔离级别的概念**：队列中的命令没有提交之前都不会实际的被执行，因为事务提交前任何指令都不会被实际执行，也就不存在”事务内的查询要看到事务里的更新，在事务外查询不能看到”这个让人万分头痛的问题 + +3. 不保证原子性：redis同一个事务中如果有一条命令执行失败，其后的命令仍然会被执行，没有回滚 + + + +### 🎯 Redis事务支持隔离性吗？ + +Redis 是单进程程序，并且它保证在执行事务时，不会对事务进行中断，事务可以运行直到执行完所有事务队列中的命令为止。因此，**Redis 的事务是总是带有隔离性的**。 + + + +### 🎯 Redis事务保证原子性吗，支持回滚吗？ + +Redis中，单条命令是原子性执行的，但**事务不保证原子性，且没有回滚**。事务中任意命令执行失败，其余的命令仍会被执行。 + +1. **如果在一个事务中的命令出现错误，那么所有的命令都不会执行**； +2. **如果在一个事务中出现运行错误，那么正确的命令会被执行**。 + +------ + + + +## 五、高可用与分布式 + +> Redis单节点存在单点故障问题，为了解决单点问题，一般都需要对Redis配置从节点，然后使用哨兵来监听主节点的存活状态，如果主节点挂掉，从节点能继续提供缓存功能 + +### 🎯 主从同步了解吗？ + +**主从复制**，是指将一台 Redis 服务器的数据，复制到其他的 Redis 服务器。前者称为 **主节点(master)**，后者称为 **从节点(slave)**。且数据的复制是 **单向** 的，只能由主节点到从节点。Redis 主从复制支持 **主从同步** 和 **从从同步** 两种，后者是 Redis 后续版本新增的功能，以减轻主节点的同步负担。 + +**主从复制主要的作用** + +- **数据冗余：** 主从复制实现了数据的热备份，是持久化之外的一种数据冗余方式。 +- **故障恢复：** 当主节点出现问题时，可以由从节点提供服务，实现快速的故障恢复 *(实际上是一种服务的冗余)*。 +- **负载均衡：** 在主从复制的基础上，配合读写分离，可以由主节点提供写服务，由从节点提供读服务 *（即写 Redis 数据时应用连接主节点，读 Redis 数据时应用连接从节点）*，分担服务器负载。尤其是在写少读多的场景下，通过多个从节点分担读负载，可以大大提高 Redis 服务器的并发量。 +- **高可用基石：** 除了上述作用以外，主从复制还是哨兵和集群能够实施的 **基础**，因此说主从复制是 Redis 高可用的基础。 + +**实现原理** + +![redis-replicaof](https://img.starfish.ink/redis/redis-replicaof.png) + +为了节省篇幅，主要的步骤都 **浓缩** 在了上图中，其实也可以 **简化成三个阶段：准备阶段-数据同步阶段-命令传播阶段**。 + +> redis2.8 之前使用`sync[runId][offset]`同步命令，redis2.8 之后使用`psync[runId][offset]`命令。两者不同在于，sync 命令仅支持全量复制过程，psync 支持全量和部分复制 +> +> **同步原理与流程** +> +> 主从复制分为三阶段，以Redis 2.8为分界点，优化了同步效率： +> +> **阶段一：建立连接（准备阶段）** +> +> 1. 从节点执行`REPLICAOF `，向主节点发起连接请求。 +> 2. 主节点验证后建立Socket连接，准备数据同步。 +> +> ##### **阶段二：数据同步（全量/增量）** +> +> - 全量同步（首次或落后过多时触发）： +> 1. 主节点执行`BGSAVE`生成**RDB快照**，同时缓存期间的写命令至**复制缓冲区**。 +> 2. 传输RDB文件给从节点，从节点清空旧数据并加载快照。 +> 3. 主节点发送缓冲区中的增量命令，从节点执行以保持数据最新。 +> - 增量同步（断线重连且条件满足时触发）： +> 1. 从节点通过`psync`命令上报自身的`runID`（主节点唯一标识）和`offset`（复制偏移量）。 +> 2. 主节点检查 `offset`是否在复制缓冲区内： +> - 若存在，发送缓冲区中从`offset`之后的增量命令（部分同步）。 +> - 若不存在，触发全量同步。 +> +> ##### **阶段三：命令传播（持续同步）** +> +> - 主节点每执行一个写命令，会异步发送给所有从节点，从节点重放命令以保持数据一致性。 + +**关键机制与优化** + +- 复制缓冲区（Repl Backlog Buffer）：环形缓冲区记录主节点最近的写命令，默认大小1MB（可配置）。若从节点落后数据超出缓冲区范围，需触发全量同步。 +- 心跳检测：主从节点定期互发心跳（`REPLCONF ACK`），检测连接状态与延迟，超时则触发重同步。 + +**注意事项** + +- **数据延迟**：主从同步是异步的，从节点数据可能存在短暂延迟，强一致性场景需谨慎。 +- 全量同步风险： + - 主节点生成RDB时会Fork子进程，大内存实例可能导致短暂阻塞。 + - 网络带宽占用高，需避免频繁全量同步（如合理配置`repl-backlog-size`）。 +- 读写分离陷阱： + - 从节点默认可能返回过期数据（需开启`replica-serve-stale-data no`配置）。 + - 读负载需均衡到多个从节点，避免单点过载。 + + + +### 🎯 主从复制会存在哪些问题呢？ + +Redis主从复制虽然提供了数据冗余和读写分离的能力，但在实际使用中仍存在以下几个典型问题，需要特别关注： + +1. **数据不一致性（最终一致性）** + - 异步复制延迟：主从同步是异步的，主节点写入成功后立即返回客户端，数据同步到从节点存在延迟（通常毫秒级，但网络波动时可能秒级）。 + - **问题场景**：若客户端从从节点读取刚写入主节点的数据，可能读到旧值（如电商库存扣减后立刻查询）。 + - 缓解方案： + - 对一致性要求高的读操作强制走主节点（通过代码控制）。 + - 使用`WAIT`命令（Redis 3.0+）阻塞直到数据同步到指定数量的从节点（牺牲性能换取强一致性）。 + +2. **主节点性能压力** + - 全量同步资源消耗： + - **Fork阻塞**：主节点执行`BGSAVE`生成RDB快照时，若数据量大，Fork子进程可能短暂阻塞主线程（尤其内存超过10GB时）。 + - **网络带宽占用**：全量同步传输RDB文件会占用大量带宽，尤其从节点数量多或跨机房同步时。 + - 优化手段： + - 避免频繁全量同步（合理配置`repl-backlog-size`缓冲区大小）。 + - 使用**从节点级联复制**（从节点作为其他从节点的主节点），分散主节点压力。 + +3. **复制中断与数据丢失风险** + - 复制缓冲区溢出： + - 主节点的**复制缓冲区**（Repl Backlog Buffer）是固定大小的环形队列，若从节点断开时间过长，缓冲区被新数据覆盖，重连后需触发全量同步。 + - **风险**：全量同步期间主节点的新写操作可能丢失（缓冲区溢出后未同步的数据）。 + - **配置建议**：增大`repl-backlog-size`（如512MB）并监控`master_repl_offset`与从节点偏移量差距。 + +4. **从节点数据过期问题** + - 过期键同步缺陷： + - Redis的过期键删除依赖主节点定时扫描（惰性+定期），从节点不会主动删除过期键，而是等待主节点同步`DEL`命令。 + - **问题表现**：从节点可能返回已逻辑过期但未收到`DEL`命令的数据。 + - **解决方案**：Redis 3.2+后开启`replica-serve-stale-data no`配置，从节点在未完成同步时不响应读请求。 + +5. **故障切换复杂度** + - 手动故障转移：原生主从复制不提供自动故障转移能力，主节点宕机后需人工介入切换从节点为主节点，导致服务中断。 + - **依赖组件**：需配合**Redis Sentinel**（哨兵）或**Cluster**实现自动故障转移。 + +6. **脑裂问题（Split-Brain）** + - 场景：网络分区导致主节点与部分从节点失联，哨兵可能选举出新主节点，形成两个主节点同时写入。 + - **数据冲突**：网络恢复后，旧主节点作为从节点同步新主节点数据，其间的写操作会丢失。 + - 规避措施： + - 配置`min-replicas-to-write`：主节点需至少存在N个从节点才允许写入。 + - 设置哨兵的`quorum`参数和`down-after-milliseconds`，减少误判。 + +> 主从复制是Redis高可用的基础，但使用时需警惕异步复制导致的数据不一致、全量同步性能开销、缓冲区溢出风险等问题。生产环境中建议： +> +> 1. **监控同步延迟**（`master_repl_offset`与从节点偏移量差值）。 +> 2. **合理配置缓冲区大小与心跳参数**。 +> 3. **结合哨兵或集群**实现自动故障转移。 +> 4. **避免单主节点挂载过多从节点**（建议级联复制分散压力）。 + + + +### 🎯 Redis读写分离的场景下,怎么保证从数据库读到最新的数据? + +在 Redis 读写分离场景下，保证从库读到最新数据是一个典型的 **数据一致性** 问题。以下是结合 Redis 特性、配置优化和业务逻辑设计的实践方案： + +1. **强制读主库（最直接方案）** + + - **原理**：将需要强一致性的读请求直接发给主库，绕过从库。 + + - 实现方式： + - **代码标记**：在业务代码中区分关键请求（如支付状态、库存查询），显式指定连接主库。 + - **中间件路由**：通过代理（如 ProxySQL）或客户端分片库（如 Lettuce）自动路由强一致性读请求到主库。 + + - **优点**：零延迟，强一致性。 + + - **缺点**：主库压力增加，违背读写分离初衷，适合低频关键操作。 + +2. **利用 `WAIT` 命令（同步复制）** + + - 原理：写操作后通过 `WAIT` 命令阻塞客户端，直到数据同步到指定数量的从库。 + + ```Bash + SET key value + WAIT # 示例：WAIT 1 1000（等待1个从库确认，超时1秒） + ``` + + - **适用场景**：写后需立即读取最新数据的场景（如用户注册后立刻查询信息）。 + + - 注意事项： + - 增加写操作的延迟（依赖网络和从库处理速度）。 + - 不保证所有从库同步成功（仅等待指定数量）。 + +3. **监控主从偏移量（动态切换读库）** + + - 原理：通过监控主从节点的 `repl_offset`（复制偏移量），仅在从库偏移量≥主库时允许读请求。 + + ```Bash + # 主库复制偏移量 + INFO replication | grep master_repl_offset + # 从库复制偏移量 + INFO replication | grep slave_repl_offset + ``` + + - 实现方式： + 1. 客户端或中间件定期获取主从偏移量。 + 2. 若从库延迟超过阈值（如 10ms），临时将读请求切换至主库。 + + - **优点**：平衡一致性与性能。 + + - **缺点**：实现复杂度高，需维护偏移量监控逻辑。 + +4. **无磁盘化复制（Diskless Replication）** + + - 原理：Redis 4.0+ 支持 `repl-diskless-sync yes` 配置，主节点直接通过 Socket 传输 RDB 快照到从节点内存，跳过磁盘 IO。 + + ```Bash + # 主节点配置 + repl-diskless-sync yes + repl-diskless-sync-delay 5 # 延迟5秒等待更多从库连接 + ``` + + - **优点**：减少全量同步时的磁盘 IO 延迟，加速数据同步。 + + - 缺点： + - 仅优化全量同步，增量同步仍依赖网络传输。 + - 主节点内存压力增大（需同时处理传输和写操作）。 + +5. **业务逻辑补偿（最终一致性兜底）** + + - 原理：通过业务代码补偿从库延迟，例如： + - **版本号校验**：写入时记录数据版本号，读取时校验从库版本是否匹配。 + - **二次确认**：先读从库，若数据不一致则重试读主库。 + + - 示例： + + ```Python + def read_with_retry(key): + value = read_from_slave(key) + if value != expected_version: + value = read_from_master(key) + return value + ``` + + - **适用场景**：对一致性要求高但允许少量重试的业务（如库存扣减）。 + +| **方案** | **一致性** | **性能影响** | **实现复杂度** | **适用场景** | +| ------------------ | ---------- | ------------ | -------------- | -------------------------- | +| 强制读主库 | 强一致性 | 高 | 低 | 低频关键操作（如支付状态） | +| `WAIT` 命令 | 强一致性 | 中 | 中 | 写后立即读（如注册信息） | +| 监控偏移量动态切换 | 最终一致性 | 低 | 高 | 高吞吐量业务（如商品列表） | +| 无磁盘化复制 | 优化同步 | 低 | 中 | 全量同步频繁场景 | +| 业务逻辑补偿 | 最终一致性 | 中 | 高 | 允许重试的业务（如库存） | + +**注意事项**： + +- Redis 主从复制是 AP 系统（CAP 定理），无法做到完全强一致，需结合业务容忍度设计。 +- 对一致性要求极高的场景（如金融交易），建议引入外部数据库（如 MySQL）兜底。 + + + +### 🎯 什么是哨兵？ + +![](https://img.starfish.ink/redis/redis-sentinel.png) + +*上图* 展示了一个典型的哨兵架构图，它由两部分组成，哨兵节点和数据节点： + +- **哨兵节点：** 哨兵系统由一个或多个哨兵节点组成，**哨兵节点是特殊的 Redis 节点，不存储数据**； +- **数据节点：** 主节点和从节点都是数据节点； + +**哨兵的介绍** + +sentinel，中文名是哨兵。哨兵是 redis 集群机构中非常重要的一个组件，主要有以下功能： + +1. 集群监控：负责监控 redis master 和 slave 进程是否正常工作。 +2. 消息通知：如果某个 redis 实例有故障，那么哨兵负责发送消息作为报警通知给管理员。 +3. 故障转移：如果 master node 挂掉了，会自动转移到 slave node 上。 +4. 配置中心：如果故障转移发生了，通知 client 客户端新的 master 地址。 + +哨兵用于实现 redis 集群的高可用，本身也是分布式的，作为一个哨兵集群去运行，互相协同工作。 + +**哨兵的核心知识** + +1. 哨兵至少需要 3 个实例，来保证自己的健壮性。 +2. 哨兵 + redis 主从的部署架构，是不保证数据零丢失的，只能保证 redis 集群的高可用性。 +3. 对于哨兵 + redis 主从这种复杂的部署架构，尽量在测试环境和生产环境，都进行充足的测试和演练。 + + + +### 🎯 说下哨兵的工作原理？ + +1. **监控机制：健康检查与状态判定** + - **周期性心跳检测**： + - 每个哨兵以**每秒1次**的频率向所有主节点、从节点、其他哨兵节点发送`PING`命令。 + - **响应超时判定**：若某节点在`down-after-milliseconds`（默认30秒）内未响应，哨兵将其标记为**主观下线（SDOWN）**。 + + - **客观下线（ODOWN）投票**： + - 当哨兵判定主节点主观下线后，向其他哨兵发送`SENTINEL is-master-down-by-addr`命令发起投票。 + - 若**超过半数哨兵**同意主节点不可用，则主节点被标记为**客观下线（ODOWN）**，触发故障转移流程。 + +2. **领导者选举：Raft共识算法适配** + - 主节点被标记为客观下线后，所有哨兵节点通过**Raft-like算法**选举领导者（Leader）。 + - 选举规则： + - 每个哨兵在**epoch（任期）**内只能投一票。 + - 获得**半数以上投票**的哨兵成为Leader，负责执行故障转移。 + - 若选举失败，等待随机时间后重新发起选举（避免活锁）。 + +3. **故障转移：新主节点选择与切换** + - **筛选候选从节点**： + - 排除已下线或延迟过高的从节点。 + - 按优先级选择（`slave-priority`配置，值越小优先级越高）。 + - 若优先级相同，选择**复制偏移量（repl_offset）最大**的从节点（数据最新）。 + - 若仍相同，选择**运行ID（run ID）较小**的从节点。 + + - **提升新主节点**： + + - 向目标从节点发送`SLAVEOF NO ONE`命令，使其成为主节点。 + + - 等待新主节点晋升完成（确认其可写且数据完整）。 + + - **重配置从节点**： + - 向所有其他从节点发送`SLAVEOF `，指向新主节点。 + - 异步更新旧主节点（若恢复）为从节点，指向新主节点。 + +4. **通知与客户端重定向** + + - **发布订阅通知**： + - 哨兵通过`__sentinel__:hello`频道发布主节点变更事件。 + - 订阅该频道的客户端可实时感知新主节点地址。 + + - **客户端服务发现**： + - 客户端通过哨兵API（如`SENTINEL get-master-addr-by-name `）获取当前主节点地址。 + - 支持自动重连的客户端库（如Jedis、Lettuce）内置故障转移处理逻辑。 + +**核心配置参数** + +| 参数 | 默认值 | 作用 | +| ------------------------- | -------- | ------------------------------------ | +| `down-after-milliseconds` | 30000ms | 主观下线判定时间 | +| `quorum` | 2 | 触发客观下线所需最少哨兵投票数 | +| `parallel-syncs` | 1 | 故障转移后同时向新主同步的从节点数量 | +| `failover-timeout` | 180000ms | 故障转移超时时间 | + +![](https://mmbiz.qpic.cn/mmbiz_png/iaIdQfEric9TzlTcnXg26t1Dia266foajMicV4uRLib3FmS9KibcSMycB36MwicA3GTygLnQTl3VkAGb8mPE47pLzcz0g/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + + + +### 🎯 哨兵模式下新的主服务器是怎样被挑选出来的？ + +**故障转移操作的第一步** 要做的就是在已下线主服务器属下的所有从服务器中，挑选出一个状态良好、数据完整的从服务器，然后向这个从服务器发送 `slaveof no one` 命令，将这个从服务器转换为主服务器。但是这个从服务器是怎么样被挑选出来的呢？ + +简单来说 Sentinel 使用以下规则来选择新的主服务器： + +1. 在失效主服务器属下的从服务器当中， 那些被标记为主观下线、已断线、或者最后一次回复 PING 命令的时间大于五秒钟的从服务器都会被 **淘汰**。 +2. 在失效主服务器属下的从服务器当中， 那些与失效主服务器连接断开的时长超过 down-after 选项指定的时长十倍的从服务器都会被 **淘汰**。 +3. 在 **经历了以上两轮淘汰之后** 剩下来的从服务器中， 我们选出 **复制偏移量（replication offset）最大** 的那个 **从服务器** 作为新的主服务器；如果复制偏移量不可用，或者从服务器的复制偏移量相同，那么 **带有最小运行 ID** 的那个从服务器成为新的主服务器。 + + + +Redis Sentinel 着眼于高可用，在 master 宕机时会自动将 slave 提升为 master，继续提供服务。 + +Redis Cluster 着眼于扩展性，在单个 redis 内存不足时，使用 Cluster 进行分片存储。 + +### 🎯 Redis 集群使用过吗？原理？ + +![Redis Cluster Architecture](https://img.starfish.ink/redis/redis-cluster-architecture.webp) + +*上图* 展示了 **Redis Cluster** 典型的架构图，集群中的每一个 Redis 节点都 **互相两两相连**，客户端任意 **直连** 到集群中的 **任意一台**，就可以对其他 Redis 节点进行 **读写** 的操作。 + + + +### 🎯 Redis集群（Cluster）工作原理与核心机制 + +> Redis Cluster 通过 **槽位分片机制（16384 槽）** 来分布存储数据，每个 key 根据 CRC16 哈希落到对应的槽位。 +> 集群由多个 Master 节点组成，每个 Master 负责一部分槽位，并通过 Replica 节点做高可用备份。 +> 节点之间通过 **Gossip 协议**维护集群状态，客户端访问时通过 **重定向机制（MOVED/ASK）** 找到目标节点。 +> 当某个 Master 故障时，集群会通过投票机制把其 Replica 提升为新的 Master，实现自动故障转移。 + +1. **数据分片：虚拟哈希槽（Virtual Hash Slot）** + + - **分片原理**： + - 整个集群划分为 **16384个哈希槽**（固定值），每个键通过`CRC16(key) % 16384`计算所属槽位。 + - 槽位均匀分配到所有主节点（例如：3主节点时，每个主节点负责约5461个槽）。 + - **槽位分配可见性**：每个节点维护槽位映射表，客户端首次连接时获取并缓存。 + + - **动态扩缩容**： + - **添加节点**：通过`CLUSTER ADDSLOTS`分配空槽，或迁移其他节点的槽至新节点。 + - **移除节点**：迁移槽至其他节点后下线。 + - **数据迁移**：使用`CLUSTER SETSLOT MIGRATING `和`CLUSTER SETSLOT IMPORTING `命令实现槽位迁移。 + +2. **高可用：主从复制与故障转移** + + - **主从架构**： + - 每个主节点对应1个或多个从节点（推荐至少1从）。 + - 从节点复制主节点数据，并在主节点故障时通过选举晋升为新主。 + + - **故障检测与恢复**： + - **节点间心跳**：节点间通过Gossip协议定期交换PING/PONG消息。 + - **故障判定**：若某主节点被多数主节点判定为不可达（超过`cluster-node-timeout`），触发故障转移。 + - **自动选举**：从节点基于`slave-priority`、复制偏移量等条件自动选举新主。 + +3. **请求路由与重定向** + + - **客户端直连**： + + - 客户端可连接任意节点，节点根据键的槽位返回正确数据或重定向指令。 + + - MOVED重定向：当键不属于当前节点时，返回 `MOVED ` 告知正确节点。 + + ```Bash + # 示例：操作不属于当前节点的键 + 127.0.0.1:6379> GET user:1001 + -MOVED 1234 192.168.1.2:6380 + ``` + + - **ASK重定向**：在槽迁移过程中，若数据未完全迁移，返回`ASK `临时重定向。 + + - 成熟客户端（如Lettuce、Jedis）缓存槽位映射表，直接路由请求到正确节点，减少重定向。 + +4. **节点通信：Gossip协议** + + - **信息交换**： + - 节点间周期性（默认每秒10次）随机选择节点发送PING消息，携带其他节点状态信息。 + - 接收节点回应PONG消息，更新本地集群状态视图。 + + - **集群状态同步**： + - 每个节点维护集群元数据（节点列表、槽位分配、主从关系等）。 + - 新节点加入时，通过种子节点（配置的初始节点）同步全量集群状态。 + + + +### 🎯 为什么选择16384个槽？ + + - 2^14 = 16384，便于位运算优化 + - 心跳包大小控制：槽位信息用bitmap传输，16384位 = 2KB + - 集群规模考量：支持最多1000个节点的合理上限 + + + +### 🎯 Redis Cluster核心优势 | 集群的主要作用 + +1. **线性扩展性**：数据分区 *(或称数据分片)* 是集群最核心的功能。集群将数据分散到多个节点，**一方面** 突破了 Redis 单机内存大小的限制，**存储容量大大增加**；**另一方面** 每个主节点都可以对外提供读服务和写服务，**大大提高了集群的响应能力** +2. **高可用性**：内置主从复制与自动故障转移，无需依赖哨兵。 +3. **去中心化架构**：无单点故障，节点平等（除主从角色差异）。 +4. **客户端透明路由**：智能客户端自动处理重定向，业务无感知。 + + + +### 🎯 集群中数据如何分区？ + +Redis集群采用**带有虚拟节点的哈希槽（Hash Slot）分区方案**，在数据分布、扩展性和容错性之间取得了高效平衡。以下是核心原理与实现细节： + +1. **虚拟哈希槽的核心设计** + + - **固定槽位数量**：预分配16384个虚拟槽（Slot），编号0-16383，**槽位总数固定不变**。 + + - 动态槽位映射：每个主节点负责管理一部分槽位，槽位与节点的映射关系可动态调整。 + + ```Bash + # 手动分配槽位给节点（示例：将槽0-5000分配给节点A） + CLUSTER ADDSLOTS {0..5000} + ``` + +- 数据路由规则： + - 键通过`CRC16(key) % 16384`计算所属槽位。 + - 客户端根据槽位映射表直接访问目标节点，或通过`MOVED`/`ASK`重定向获取最新路由。 + +2. **对比三种分区方案的优势** + +| **方案** | **优点** | **缺点** | Redis集群的选择原因 | +| -------------------- | -------------------- | ---------------------------- | ---------------------------- | +| **哈希取余（%）** | 实现简单 | 扩缩容时数据全量迁移，成本高 | 不采用 | +| **一致性哈希** | 扩缩容仅影响相邻节点 | 节点少时数据倾斜严重 | 改进后采用（引入虚拟槽） | +| **虚拟槽一致性哈希** | 数据均匀、扩缩容敏捷 | 需维护槽位映射关系 | 采用，平衡扩展性与管理复杂度 | + +3. **扩缩容与数据迁移流程** + + - **扩容**： + + - 新节点加入集群，初始不负责任何槽。 + - 从现有节点迁移部分槽至新节点（如均匀分配）。 + - 更新集群元数据，广播新槽位分配。 + + ```Bash + # 将槽1000从节点A迁移到新节点B + CLUSTER SETSLOT 1000 IMPORTING B-node-id # 目标节点执行 + CLUSTER SETSLOT 1000 MIGRATING A-node-id # 源节点执行 + # 迁移数据后，最终设置槽归属 + CLUSTER SETSLOT 1000 NODE B-node-id + ``` + + - **缩容**： + + - 将待移除节点的槽迁移至其他节点。 + + - 节点无槽后，执行`CLUSTER FORGET`从集群移除。 + +4. **数据均衡与故障转移** + + - **自动均衡**：通过`redis-cli --cluster rebalance`命令触发槽位均衡分配，确保各节点负载均匀。 + + - 故障恢复： + - 主节点宕机时，其从节点通过选举晋升为新主，接管槽位。 + - 集群自动更新槽位映射，客户端通过重定向无缝切换。 + +5. **客户端路由优化** + + - **智能缓存**：客户端首次连接集群时获取槽位映射表，本地缓存以减少重定向次数。 + + - MOVED重定向： + + ```Bash + # 客户端访问错误节点时的响应 + 127.0.0.1:6379> GET user:101 + -MOVED 9252 192.168.1.2:6380 # 槽9252当前由192.168.1.2:6380管理 + ``` + + - **ASK重定向**：在槽迁移过程中，若数据尚未完全迁移，临时重定向客户端到目标节点。 + +**总结**：Redis集群通过虚拟槽机制将数据解耦于物理节点，实现高效的动态扩缩容与负载均衡。相比传统哈希取余和一致性哈希方案，虚拟槽避免了大规模数据迁移和数据倾斜问题，成为支撑高可用、高性能分布式Redis服务的核心基础。 + +> #### 一致性哈希分区 +> +> 一致性哈希算法将 **整个哈希值空间** 组织成一个虚拟的圆环，范围是 *[0 - 2^32 - 1]*，对于每一个数据，根据 `key` 计算 hash 值，确数据在环上的位置，然后从此位置沿顺时针行走，找到的第一台服务器就是其应该映射到的服务器： +> +> 与哈希取余分区相比，一致性哈希分区将 **增减节点的影响限制在相邻节点**。以上图为例，如果在 `node1` 和 `node2` 之间增加 `node5`，则只有 `node2` 中的一部分数据会迁移到 `node5`；如果去掉 `node2`，则原 `node2` 中的数据只会迁移到 `node4` 中，只有 `node4` 会受影响。 +> +> 一致性哈希分区的主要问题在于，当 **节点数量较少** 时，增加或删减节点，**对单个节点的影响可能很大**，造成数据的严重不平衡。还是以上图为例，如果去掉 `node2`，`node4` 中的数据由总数据的 `1/4` 左右变为 `1/2` 左右，与其他节点相比负载过高。 +> +> #### 带有虚拟节点的一致性哈希分区 +> +> 该方案在 **一致性哈希分区的基础上**，引入了 **虚拟节点** 的概念。Redis 集群使用的便是该方案，其中的虚拟节点称为 **槽（slot）**。槽是介于数据和实际节点之间的虚拟概念，每个实际节点包含一定数量的槽，每个槽包含哈希值在一定范围内的数据。 +> +> 在使用了槽的一致性哈希分区中，**槽是数据管理和迁移的基本单位**。槽 **解耦** 了 **数据和实际节点** 之间的关系，增加或删除节点对系统的影响很小。仍以上图为例，系统中有 `4` 个实际节点，假设为其分配 `16` 个槽(0-15)； +> +> - 槽 0-3 位于 node1；4-7 位于 node2；以此类推…. +> +> 如果此时删除 `node2`，只需要将槽 4-7 重新分配即可，例如槽 4-5 分配给 `node1`，槽 6 分配给 `node3`，槽 7 分配给 `node4`；可以看出删除 `node2` 后，数据在其他节点的分布仍然较为均衡。 +> + + + +### 🎯 节点之间的通信机制了解吗？ + +集群的建立离不开节点之间的通信，假如我们启动六个集群节点之后通过 `redis-cli` 命令帮助我们搭建起来了集群，实际上背后每个集群之间的两两连接是通过了 `CLUSTER MEET` 命令发送 `MEET` 消息完成的，下面我们展开详细说说。 + +**两个端口** + +在 **哨兵系统** 中，节点分为 **数据节点** 和 **哨兵节点**：前者存储数据，后者实现额外的控制功能。在 **集群** 中，没有数据节点与非数据节点之分：**所有的节点都存储数据，也都参与集群状态的维护**。为此，集群中的每个节点，都提供了两个 TCP 端口： + +- **普通端口：** 即我们在前面指定的端口 *(7000等)*。普通端口主要用于为客户端提供服务 *（与单机节点类似）*；但在节点间数据迁移时也会使用。 +- **集群端口：** 端口号是普通端口 + 10000 *（10000是固定值，无法改变）*，如 `7000` 节点的集群端口为 `17000`。**集群端口只用于节点之间的通信**，如搭建集群、增减节点、故障转移等操作时节点间的通信；不要使用客户端连接集群接口。为了保证集群可以正常工作，在配置防火墙时，要同时开启普通端口和集群端口。 + +**Gossip 协议** + +节点间通信，按照通信协议可以分为几种类型：单对单、广播、Gossip 协议等。重点是广播和 Gossip 的对比。 + +- 广播是指向集群内所有节点发送消息。**优点** 是集群的收敛速度快(集群收敛是指集群内所有节点获得的集群信息是一致的)，**缺点** 是每条消息都要发送给所有节点，CPU、带宽等消耗较大。 +- Gossip 协议的特点是：在节点数量有限的网络中，**每个节点都 “随机” 的与部分节点通信** （并不是真正的随机，而是根据特定的规则选择通信的节点），经过一番杂乱无章的通信，每个节点的状态很快会达到一致。Gossip 协议的 **优点**有负载 (比广播) 低、去中心化、容错性高 *(因为通信有冗余)* 等；**缺点** 主要是集群的收敛速度慢。 + +**消息类型** + +集群中的节点采用 **固定频率（每秒10次）** 的 **定时任务** 进行通信相关的工作：判断是否需要发送消息及消息类型、确定接收节点、发送消息等。如果集群状态发生了变化，如增减节点、槽状态变更，通过节点间的通信，所有节点会很快得知整个集群的状态，使集群收敛。 + +节点间发送的消息主要分为 `5` 种：`meet 消息`、`ping 消息`、`pong 消息`、`fail 消息`、`publish 消息`。不同的消息类型，通信协议、发送的频率和时机、接收节点的选择等是不同的： + +- **MEET 消息：** 在节点握手阶段，当节点收到客户端的 `CLUSTER MEET` 命令时，会向新加入的节点发送 `MEET` 消息，请求新节点加入到当前集群；新节点收到 MEET 消息后会回复一个 `PONG` 消息。 +- **PING 消息：** 集群里每个节点每秒钟会选择部分节点发送 `PING` 消息，接收者收到消息后会回复一个 `PONG` 消息。**PING 消息的内容是自身节点和部分其他节点的状态信息**，作用是彼此交换信息，以及检测节点是否在线。`PING` 消息使用 Gossip 协议发送，接收节点的选择兼顾了收敛速度和带宽成本，**具体规则如下**：(1)随机找 5 个节点，在其中选择最久没有通信的 1 个节点；(2)扫描节点列表，选择最近一次收到 `PONG` 消息时间大于 `cluster_node_timeout / 2` 的所有节点，防止这些节点长时间未更新。 +- **PONG消息：** `PONG` 消息封装了自身状态数据。可以分为两种：**第一种** 是在接到 `MEET/PING` 消息后回复的 `PONG` 消息；**第二种** 是指节点向集群广播 `PONG` 消息，这样其他节点可以获知该节点的最新信息，例如故障恢复后新的主节点会广播 `PONG` 消息。 +- **FAIL 消息：** 当一个主节点判断另一个主节点进入 `FAIL` 状态时，会向集群广播这一 `FAIL` 消息；接收节点会将这一 `FAIL` 消息保存起来，便于后续的判断。 +- **PUBLISH 消息：** 节点收到 `PUBLISH` 命令后，会先执行该命令，然后向集群广播这一消息，接收节点也会执行该 `PUBLISH` 命令。 + -Redis 内置了复制（Replication），LUA脚本（Lua scripting）， LRU驱动事件（LRU eviction），事务（Transactions） 和不同级别的磁盘持久化（Persistence），并通过 Redis哨兵（Sentinel）和自动分区（Cluster）提供高可用性（High Availability）。 -- 性能优秀，数据在内存中，读写速度非常快，支持并发10W QPS -- 单进程单线程，是线程安全的，采用IO多路复用机制 -- Redis 数据库完全在内存中，使用磁盘仅用于持久性 -- 相比许多键值数据存储，Redis 拥有一套较为丰富的数据类型 -- 操作都是**原子性**：所有 Redis 操作是原子的，这保证了如果两个客户端同时访问的Redis服务器将获得更新后的值 -- Redis 可以将数据复制到任意数量的从服务器（主从复制，哨兵，高可用） +### 🎯 集群数据如何存储的有了解吗？ +节点需要专门的数据结构来存储集群的状态。所谓集群的状态，是一个比较大的概念，包括：集群是否处于上线状态、集群中有哪些节点、节点是否可达、节点的主从状态、槽的分布…… +节点为了存储集群状态而提供的数据结构中，最关键的是 `clusterNode` 和 `clusterState` 结构：前者记录了一个节点的状态，后者记录了集群作为一个整体的状态。 -> Redis 都支持哪些数据类型 +**clusterNode 结构** -### Redis数据类型 +`clusterNode` 结构保存了 **一个节点的当前状态**，包括创建时间、节点 id、ip 和端口号等。每个节点都会用一个 `clusterNode` 结构记录自己的状态，并为集群内所有其他节点都创建一个 `clusterNode` 结构来记录节点状态。 -Redis 不是简单的键值存储，它实际上是一个数据结构服务器，支持不同类型的值。 +下面列举了 `clusterNode` 的部分字段，并说明了字段的含义和作用： -- String（字符串）：二进制安全字符串 -- List（列表）：根据插入顺序排序的字符串元素的集合。它们基本上是链表 -- Set（集合）：唯一，未排序的字符串元素的集合 -- zset(sorted set：有序集合)：相当于有序的 Set集合，每个字符串元素都与一个称为 *score* 的浮点值相关联。元素总是按它们的分数排序（eg，找出前10名或后10名） -- Hash（字典）：是一个键值对集合。KV模式不变，但V是一个键值对 -- Bit arrays （位数组，简称位图）： -- HyperLogLog（）：这是一个概率数据结构，用于估计集合的基数 -- Streams： +```c +typedef struct clusterNode { + //节点创建时间 + mstime_t ctime; + //节点id + char name[REDIS_CLUSTER_NAMELEN]; + //节点的ip和端口号 + char ip[REDIS_IP_STR_LEN]; + int port; + //节点标识：整型，每个bit都代表了不同状态，如节点的主从状态、是否在线、是否在握手等 + int flags; + //配置纪元：故障转移时起作用，类似于哨兵的配置纪元 + uint64_t configEpoch; + //槽在该节点中的分布：占用16384/8个字节，16384个比特；每个比特对应一个槽：比特值为1，则该比特对应的槽在节点中；比特值为0，则该比特对应的槽不在节点中 + unsigned char slots[16384/8]; + //节点中槽的数量 + int numslots; + ………… +} clusterNode; +``` -![img](https://imgkr.cn-bj.ufileos.com/240fbd7d-a6de-4320-8e41-0e0a7fde7ea6.png) +除了上述字段，`clusterNode` 还包含节点连接、主从复制、故障发现和转移需要的信息等。 +**clusterState 结构** +`clusterState` 结构保存了在当前节点视角下，集群所处的状态。主要字段包括： -> 那你能说说这些数据类型的使用指令吗？ +```c +typedef struct clusterState { + //自身节点 + clusterNode *myself; + //配置纪元 + uint64_t currentEpoch; + //集群状态：在线还是下线 + int state; + //集群中至少包含一个槽的节点数量 + int size; + //哈希表，节点名称->clusterNode节点指针 + dict *nodes; + //槽分布信息：数组的每个元素都是一个指向clusterNode结构的指针；如果槽还没有分配给任何节点，则为NULL + clusterNode *slots[16384]; + ………… +} clusterState; +``` -### Redis 常用命令 +除此之外，`clusterState` 还包括故障转移、槽迁移等需要的信息。 +### 🎯 Redis集群最大节点个数是多少？ +16384 -> 这些基本知识都会了，那你知道 Redis 一般用在哪些场景吗，你们项目中是怎么用 Redis 的 -### redis使用场景 -在 Redis 中，常用的 5 种数据结构和应用场景如下： +### 🎯 Redis集群会有写操作丢失吗？为什么？ -- **String：**缓存、计数器、分布式锁等。 -- **List：**链表、队列、微博关注人时间轴列表等。 -- **Hash：**用户信息、Hash 表等。 -- **Set：**去重、赞、踩、共同好友等。 -- **Zset：**访问量排行榜、点击量排行榜等 +Redis并不能保证数据的强一致性，这意味这在实际中集群在特定的条件下可能会丢失写操作。 -- 取最新N个数据的操作 -- 排行榜应用,取TOP N 操作 -- 需要精确设定过期时间的应用 -- 定时器、计数器应用 -- Uniq操作,获取某段时间所有数据排重值 -- 实时系统,反垃圾系统 -- Pub/Sub构建实时消息系统 -- 构建队列系统 -- 缓存 +### 🎯 Redis集群之间是如何复制的？ - +Redis集群使用异步复制机制在主从节点之间进行数据复制。以下是Redis集群复制的关键点和工作原理： -> 用缓存，肯定是因为他快，那为什么快呢 -> -> redis这么快，它的“多线程模型”你了解吗？（露出邪魅一笑） +**主从复制** -### Redis为什么这么快 +1. **主节点（Master）和从节点（Slave）**： + - 每个主节点负责处理特定的槽（slots）范围，并可以有多个从节点。 + - 从节点通过复制主节点的数据来保持同步，并在主节点不可用时自动提升为新的主节点。 +2. **异步复制**： + - 主节点会将写操作命令异步发送给从节点，从节点异步接收并执行这些命令。 + - 由于是异步复制，主节点不会等待从节点确认写操作已经完成，这提高了性能，但也可能导致数据在短时间内不一致。 -- 完全基于内存，绝大部分请求是纯粹的内存操作，非常快速。数据存在内存中，类似于HashMap，HashMap的优势就是查找和操作的时间复杂度都是O(1) -- 数据结构简单，对数据操作也简单，Redis中的数据结构是专门进行设计的 -- 采用单线程，避免了不必要的上下文切换和竞争条件，也不存在多进程或者多线程导致的切换而消耗 CPU，不用去考虑各种锁的问题，不存在加锁释放锁操作，没有因为可能出现死锁而导致的性能消耗 -- 使用多路I/O复用模型，非阻塞IO。“多路”指的是多个网络连接，“复用”指的是复用同一个线程。采用多路 I/O 复用技术可以让单个线程高效的处理多个连接请求（尽量减少网络IO的时间消耗）。可以直接理解为：单线程的原子操作，避免上下文切换的时间和性能消耗；加上对内存中数据的处理速度，很自然的提高 Redis 的吞吐量。 -- 使用底层模型不同，它们之间底层实现方式以及与客户端之间通信的应用协议不一样，Redis直接自己构建了VM 机制 ，因为一般的系统调用系统函数的话，会浪费一定的时间去移动和请求； +**复制过程** +1. **初次同步（Initial Synchronization）**： + - 当一个从节点第一次连接到主节点时，会进行全量复制。 + - 主节点会生成一个 RDB 快照文件，并将其发送给从节点。 + - 在 RDB 文件传输过程中，主节点会将新的写操作命令存储在缓冲区中。 + - RDB 文件传输完成后，主节点会将缓冲区中的写操作命令发送给从节点，从节点执行这些命令以完成数据同步。 +2. **增量同步（Incremental Synchronization）**： + - 在初次同步之后，主节点只会将新的写操作命令发送给从节点，从节点接收并执行这些命令。 +**故障转移** -### Redis 为什么早期版本选择单线程？ +1. **故障检测**： + - 当主节点不可用时，从节点可以通过选举算法选举一个新的主节点。 + - 哨兵（Sentinel）或 Redis Cluster 本身的机制可以实现自动故障转移。 +2. **数据一致性**： + - Redis 使用“最终一致性”模型，虽然复制是异步的，但最终所有节点的数据会一致。 -我们首先要明白，上边的种种分析，都是为了营造一个Redis很快的氛围！官方FAQ表示，因为Redis是基于内存的操作，CPU不是Redis的瓶颈，Redis的瓶颈最有可能是机器内存的大小或者网络带宽。既然单线程容易实现，而且CPU不会成为瓶颈，那就顺理成章地采用单线程的方案了（毕竟采用多线程会有很多麻烦！）。 +**集群通信协议** -看到这里，你可能会气哭！本以为会有什么重大的技术要点才使得Redis使用单线程就可以这么快，没想到就是一句官方看似糊弄我们的回答！但是，我们已经可以很清楚的解释了为什么Redis这么快，并且正是由于在单线程模式的情况下已经很快了，就没有必要在使用多线程了！ +1. **Gossip 协议**： + - Redis Cluster 节点之间使用 Gossip 协议进行通信，以传播节点状态和槽分配信息。 + - 每个节点会定期向其他节点发送消息，以分享自身的状态和接收到的其他节点的状态信息。 +2. **复制偏移量和 ACK**： + - 主节点会维护一个全局复制偏移量，并将其发送给从节点。 + - 从节点会定期向主节点发送 ACK 消息，告知主节点它们已经接收到的数据偏移量。 -但是，我们使用单线程的方式是无法发挥多核CPU 性能，不过我们可以通过在单机开多个Redis 实例来完善！ -警告1：这里我们一直在强调的单线程，只是在处理我们的网络请求的时候只有一个线程来处理，一个正式的Redis Server运行的时候肯定是不止一个线程的，这里需要大家明确的注意一下！例如Redis进行持久化的时候会以子进程或者子线程的方式执行（具体是子线程还是子进程待读者深入研究）；例如我在测试服务器上查看Redis进程，然后找到该进程下的线程： -![img](https://user-gold-cdn.xitu.io/2018/8/22/1656042975a7dff7?imageView2/0/w/1280/h/960/format/webp/ignore-error/1) +### 🎯 Redis是单线程的，如何提高多核CPU的利用率？ -ps命令的“-T”参数表示显示线程（Show threads, possibly with SPID column.）“SID”栏表示线程ID，而“CMD”栏则显示了线程名称。 +Redis 是单线程的，意味着其核心功能（如处理命令请求、数据存储和检索等）主要在单个线程中执行。尽管如此，Redis 还是有一些方法可以提高在多核 CPU 系统上的利用率： -警告2：在上图中FAQ中的最后一段，表述了从Redis 4.0版本开始会支持多线程的方式，但是，只是在某一些操作上进行多线程的操作！所以该篇文章在以后的版本中是否还是单线程的方式需要读者考证！ +1. **使用多个 Redis 实例**： + - 可以在同一个服务器上运行多个 Redis 实例，每个实例绑定到不同的 CPU 核心上。这种方法可以使得每个核心运行一个 Redis 实例，从而提高 CPU 的利用率。 +2. **分片（Sharding）**： + - 通过分片将数据分布到多个 Redis 实例，每个实例可以运行在不同的服务器或同一服务器的不同核心上。 +3. **Redis 集群**： + - 使用 Redis 集群可以将数据自动分区到多个节点上。每个节点可以独立运行在不同的 CPU 核心上。 +4. **后台任务**： + - Redis 的一些任务，如持久化（RDB 快照和 AOF 日志写入）、过期键的清理等，可以在后台线程中执行，不阻塞主线程。 +5. **使用多线程 I/O**： + - 从 Redis 6.0 开始，引入了多线程 I/O 来提高网络通信的效率。虽然核心命令处理仍然是单线程的，但多线程 I/O 可以显著提高网络数据的读写速度。 -> 你有提到过memcached，那你为什么选择Redis的缓存方案而不用memcached呢 +### 🎯 为什么要做Redis分区？ -### Redis和Memcached的区别 +分区可以让Redis管理更大的内存，Redis将可以使用所有机器的内存。如果没有分区，你最多只能使用一台机器的内存。分区使Redis的计算能力通过简单地增加计算机得到成倍提升，Redis的网络带宽也会随着计算机和网卡的增加而成倍增长。 -1. 存储方式上：memcache会把数据全部存在内存之中，断电后会挂掉，数据不能超过内存大小。redis有部分数据存在硬盘上，这样能保证数据的持久性。 -2. 数据支持类型上：memcache对数据类型的支持简单，只支持简单的key-value，而redis支持五种数据类型。 -3. 使用底层模型不同：它们之间底层实现方式以及与客户端之间通信的应用协议不一样。redis直接自己构建了VM机制，因为一般的系统调用系统函数的话，会浪费一定的时间去移动和请求。 -4. value的大小：redis可以达到1GB，而memcache只有1MB。 +### 🎯 有哪些Redis分区实现方案？ -### Redis 优缺点 +1. 客户端分区：客户端通过哈希算法（如哈希取模、一致性哈希）计算数据的存储节点，直接与目标节点通信，无需中间代理。 -优点 +2. **服务端分区**： -- **读写性能优异**， Redis能读的速度是 `110000` 次/s，写的速度是 `81000` 次/s。 -- **支持数据持久化**，支持 AOF 和 RDB 两种持久化方式。 -- **支持事务**，Redis 的所有操作都是原子性的，同时 Redis 还支持对几个操作合并后的原子性执行。 -- **数据结构丰富**，除了支持 string 类型的 value 外还支持 hash、set、zset、list 等数据结构。 -- **支持主从复制**，主机会自动将数据同步到从机，可以进行读写分离。 + - Redis 自身提供分区能力，节点通过集群协议（如 Redis Cluster）自动管理数据分布，客户端只需连接任意节点，由集群路由请求到目标节点。 + - **基于代理的服务端分区（如 Codis）**：通过中间件（如 Codis Proxy）接收客户端请求，代理层维护槽位与节点的映射关系，并路由请求。 -缺点 +3. **代理层分区**：通过独立的代理服务（如 Twemproxy、Codis、Redis Cluster Proxy）实现分区逻辑，客户端只需连接代理，由代理转发请求到后端 Redis 节点。 -- 数据库 **容量受到物理内存的限制**，不能用作海量数据的高性能读写，因此 Redis 适合的场景主要局限在较小数据量的高性能操作和运算上。 -- Redis **不具备自动容错和恢复功能**，主机从机的宕机都会导致前端部分读写请求失败，需要等待机器重启或者手动切换前端的 IP 才能恢复。 -- 主机宕机，宕机前有部分数据未能及时同步到从机，切换 IP 后还会引入数据不一致的问题，降低了 **系统的可用性**。 -- **Redis 较难支持在线扩容**，在集群容量达到上限时在线扩容会变得很复杂。为避免这一问题，运维人员在系统上线时必须确保有足够的空间，这对资源造成了很大的浪费。 + ------- +### 🎯 Redis分区有什么缺点？ +1. 涉及多个key的操作通常不会被支持。例如你不能对两个集合求交集，因为他们可能被存储到不同的Redis实例（实际上这种情况也有办法，但是不能直接使用交集指令）。 +2. 同时操作多个key,则不能使用Redis事务. -## 二、Redis 数据结构问题 +3. 分区使用的粒度是key，不能使用一个非常长的排序key存储一个数据集 -首先在 Redis 内部会使用一个 **RedisObject** 对象来表示所有的 `key` 和 `value`： +4. 当使用分区的时候，数据处理会非常复杂，例如为了备份你必须从不同的Redis实例和主机同时收集RDB / AOF文件。 -![img](https://imgkr.cn-bj.ufileos.com/3c186fd2-7683-40e8-a9ca-bbc3cd2d61a3.png) +5. 分区时动态扩容或缩容可能非常复杂。Redis集群在运行时增加或者删除Redis节点，能做到最大程度对用户透明地数据再平衡，但其他一些客户端分区或者代理分区方法则不支持这种特性。然而，有一种预分片的技术也可以较好的解决这个问题。 -Redis 提供了五种基本数据类型，String、Hash、List、Set、Zset(sorted set：有序集合) -由于Redis是基于标准 C 写的，只有最基础的数据类型，因此 Redis 为了满足对外使用的 5 种数据类型，开发了属于自己**独有的一套基础数据结构**，使用这些数据结构来实现5种数据类型。 -Redis底层的数据结构包括：**简单动态数组SDS、链表、字典、跳跃链表、整数集合、压缩列表、对象。** +### 🎯 Redis 的高可用 -Redis为了平衡空间和时间效率，针对value的具体类型在底层会采用不同的数据结构来实现，其中哈希表和压缩列表是复用比较多的数据结构，如下图展示了对外数据类型和底层数据结构之间的映射关系： +Redis 的高可用主要依靠 **主从复制 + Sentinel** 实现。 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1ge0k8do3s3j30im0el7aa.jpg) +**主从复制**：主服务器把数据同步给从服务器，从 Redis 2.8 开始支持 **部分重同步**（PSYNC），避免全量同步消耗大量资源。 -#### String（字符串） +**Sentinel 哨兵系统**：监控集群节点，发现主服务器故障后，通过投票和 Raft 选举选出 Leader 来执行 **Failover**，将合适的从服务器提升为主服务器，保证集群可用。 -Redis 是用 C 语言开发完成的，但在 Redis 字符串中，并没有使用 C 语言中的字符串，而是用一种称为 **SDS**（Simple Dynamic String）的结构体来保存字符串。 +**复制拓扑**： -![img](https://pic2.zhimg.com/80/v2-c04de91ae2ec8948c8961e1884acbe6d_1440w.jpg) +- 单层主从：所有从直接挂在主节点，延迟低但主节点压力大 +- 级联主从：从节点也可以做主的上游节点，降低主节点压力，但越下游延迟越大 -String 是 Redis 最基本的类型，你可以理解成与 Memcached一模一样的类型，一个 key 对应一个 value。 +注意：Redis 的主从复制保证 **最终一致性**，无法保证强一致性。 -String类型是二进制安全的。意思是 Redis 的 String 可以包含任何数据。比如jpg图片或者序列化的对象 。 -Redis 的字符串是动态字符串，是可以修改的字符串，**内部结构实现上类似于 Java 的 ArrayList**，采用预分配冗余空间的方式来减少内存的频繁分配，内部为当前字符串实际分配的空间 capacity 一般要高于实际字符串长度 len。当字符串长度小于 1M 时，扩容都是加倍现有的空间，如果超过 1M，扩容时一次只会多扩 1M 的空间。需要注意的是**字符串最大长度为 512M** -![](http://ww1.sinaimg.cn/large/9b9f09a9ly1g9ypoobef5j20fw04pq2p.jpg) +### 🎯 CAP 理论对 Redis 选型的影响？ -##### Redis 的 SDS 和 C 中字符串相比有什么优势？ +> **CAP 理论**指出：在分布式系统中，不可能同时完全满足以下三点，只能取两点： +> +> 1. **Consistency（强一致性）**：所有节点的数据在同一时间是一致的 +> 2. **Availability（高可用）**：每次请求都能得到响应（可能不是最新的数据） +> 3. **Partition tolerance（分区容错）**：系统能容忍节点间网络分区 -C 语言使用了一个长度为 `N+1` 的字符数组来表示长度为 `N` 的字符串，并且字符数组最后一个元素总是 `\0`，这种简单的字符串表示方式 **不符合 Redis 对字符串在安全性、效率以及功能方面的要求**。 +- CAP 落到 Redis 的选型：单机不谈 CAP；一旦主从/哨兵/Cluster，就是“在网络分区时要在 C 和 A 之间取舍”。Redis 默认偏 AP（高可用+分区容错，牺牲强一致），通过参数可往 C 侧拉，但会掉可用性和时延。 -再来说 C 语言字符串的问题 +- 面试关键词：异步复制（最终一致/可能丢写）、WAIT 半同步、min-replicas-to-write、cluster-require-full-coverage、只读从/强读主、幂等与补偿。 -这样简单的数据结构可能会造成以下一些问题： +**分场景怎么选（直给可落地结论）** -- **获取字符串长度为 O(N) 级别的操作** → 因为 C 不保存数组的长度，每次都需要遍历一遍整个数组； -- 不能很好的杜绝 **缓冲区溢出/内存泄漏** 的问题 → 跟上述问题原因一样，如果执行拼接 or 缩短字符串的操作，如果操作不当就很容易造成上述问题； -- C 字符串 **只能保存文本数据** → 因为 C 语言中的字符串必须符合某种编码（比如 ASCII），例如中间出现的 `'\0'` 可能会被判定为提前结束的字符串而识别不了； +- 单机（非分布式）：不涉及 CAP。强一致=强可用不可用？单点故障A差。适合缓存/非关键写。 -##### Redis 如何解决的 | SDS 的优势 +- 主从 + 哨兵： -![img](https://imgkr.cn-bj.ufileos.com/f1cb9bd1-53e3-4bfb-b88d-bcb1a274d419.png) + - 默认 AP：复制异步，主挂+选主可能丢最近写；读从为最终一致。 -如果去看 Redis 的源码 `sds.h/sdshdr` 文件，你会看到 SDS 完整的实现细节，这里简单来说一下 Redis 如何解决的： + - 往 C 拉的手段： -1. **多增加 len 表示当前字符串的长度**：这样就可以直接获取长度了，复杂度 O(1)； -2. **自动扩展空间**：当 SDS 需要对字符串进行修改时，首先借助于 `len` 和 `alloc` 检查空间是否满足修改所需的要求，如果空间不够的话，SDS 会自动扩展空间，避免了像 C 字符串操作中的覆盖情况； -3. **有效降低内存分配次数**：C 字符串在涉及增加或者清除操作时会改变底层数组的大小造成重新分配，SDS 使用了 **空间预分配** 和 **惰性空间释放** 机制，简单理解就是每次在扩展时是成倍的多分配的，在缩容是也是先留着并不正式归还给 OS； -4. **二进制安全**：C 语言字符串只能保存 `ascii` 码，对于图片、音频等信息无法保存，SDS 是二进制安全的，写入什么读取就是什么，不做任何过滤和限制； + - 只读主（强读），从只做读缓存/非关键读； + - 写后 WAIT N T 要求 N 个副本 ack（半同步），提升 C，牺牲 A 和时延； + - min-replicas-to-write + min-replicas-max-lag，副本不足或落后大时拒写，保 C 降 A。 +- Redis Cluster： -#### List（列表） + - CAP 可调： -Redis 列表是简单的字符串列表，按照插入顺序排序。你可以添加一个元素到列表的头部（左边）或者尾部（右边）。 + - cluster-require-full-coverage yes（偏 C）：有槽不可用时返回 CLUSTERDOWN，牺牲 A； -**Redis 的列表相当于 Java 语言里面的 LinkedList，注意它是链表而不是数组**。这意味着 list 的插入和删除操作非常快，时间复杂度为 O(1)，但是索引定位很慢，时间复杂度为 O(n)。 + - 设 no（偏 A）：可服务剩余槽，整体可用但一致性/完整覆盖降低； -**Redis 的列表结构常用来做异步队列使用**。将需要延后处理的任务结构体序列化成字符串塞进 Redis 的列表，另一个线程从这个列表中轮询数据进行处理。 + - cluster-allow-reads-when-down yes 进一步偏 A（读也放开）。 -#### Hash（字典） + - 复制仍异步，failover 仍可能丢最近写；跨槽多键事务与强一致不保证。 -Redis Hash 是一个键值对集合。KV模式不变，但V是一个键值对。 +- 跨机房/多活： -##### 字典是如何实现的？Rehash 了解吗？ + - 典型选 AP（最终一致）：业务侧幂等/补偿/对账；或用 Redis Enterprise Active-Active（CRDT）做冲突收敛。 -**Redis** 中的字典相当于 Java 中的 **HashMap**，内部实现也差不多类似，都是通过 **“数组 + 链表”** 的 **链地址法** 来解决部分 哈希冲突，同时这样的结构也吸收了两种不同数据结构的优点。 + - 要强一致建议用 Raft/etcd/DB，而不是把 Redis 硬拗成 CP。 -字典结构内部包含 **两个 hashtable**，通常情况下只有一个 `hashtable` 有值，但是在字典扩容缩容时，需要分配新的 `hashtable`，然后进行 **渐进式搬迁** *(rehash)*，这时候两个 `hashtable` 分别存储旧的和新的 `hashtable`，待搬迁结束后，旧的将被删除，新的 `hashtable` 取而代之。 +**核心取舍与参数** -##### 扩缩容的条件 +- 提升一致性（向 C 靠拢）： -正常情况下，当 hash 表中 **元素的个数等于第一维数组的长度时**，就会开始扩容，扩容的新数组是 **原数组大小的 2 倍**。不过如果 Redis 正在做 `bgsave(持久化命令)`，为了减少内存也得过多分离，Redis 尽量不去扩容，但是如果 hash 表非常满了，**达到了第一维数组长度的 5 倍了**，这个时候就会 **强制扩容**。 + - 写后 WAIT N T；强读主；禁读从；min-replicas-to-write、min-replicas-max-lag； -当 hash 表因为元素逐渐被删除变得越来越稀疏时，Redis 会对 hash 表进行缩容来减少 hash 表的第一维数组空间占用。所用的条件是 **元素个数低于数组长度的 10%**，缩容不会考虑 Redis 是否在做 `bgsave` + - Cluster 开 cluster-require-full-coverage yes； + - 代价：更高 RT、更低可用性（分区/副本不足时拒写/报错）。 +- 提升可用（向 A 靠拢）： + - 允许读从；Cluster 设 full-coverage no、allow-reads-when-down yes； -#### Set（集合） + - 代价：读到旧值/写丢失窗口扩大，靠幂等与补偿兜底。 -Set 是 String 类型的无序集合， **相当于 Java 语言里面的 HashSet**，它内部的键值对是无序的唯一的。它的内部实现相当于一个特殊的字典，字典中所有的 value 都是 NULL。 +**典型应用建议** -#### Zset(sorted set：有序集合) +- 缓存/会话/排行榜：选 AP，接受最终一致；TTL+逻辑过期+幂等刷新。 -**它类似于 Java 的 SortedSet 和 HashMap 的结合体**，一方面它是一个 set，保证了内部 value 的唯一性，另一方面它可以给每个 value 赋予一个 score，代表这个 value 的排序权重。它的内部实现用的是一种叫做「跳跃列表」的数据结构。 +- 购物车/库存展示：读从可接受，写走主，必要时 WAIT 1 容错。 -Redis 正是通过 score 来为集合中的成员进行从小到大的排序。Zset 的成员是唯一的，但 score 却可以重复。 +- 订单/扣款/账务：Redis 不做真源，落库为准；若必须用，强读主+WAIT+幂等与对账，或直接用 CP 系统。 -#### Bitmaps（位图） +**故障与防御** -位图不是实际的数据类型，而是在 String 类型上定义的一组面向位的操作。可以看作是 byte 数组。我们可以使用普通的 get/set 直接获取和设置整个位图的内容，也可以使用位图操作 getbit/setbit 等将 byte 数组看成「位数组」来处理。 +- 主从异步复制导致“主挂+回切丢写”：降低 down-after-milliseconds、设置 min-replicas-to-write、关键写用 WAIT。 -一般用于：各种实时分析；存储与对象 ID 相关的布尔信息 +- Cluster 槽缺失报错与可用性：按业务决定 full-coverage，强一致优先就宁可报错不脏写。 -#### HyperLogLog +一句话总结：Redis 分布式形态下天生更偏 AP；能通过 WAIT、副本滞后阈值、Cluster 覆盖策略把“C/A旋钮”往任一侧拧，但代价是时延或可用性。强一致交易场景慎用 Redis 做主存。 -HyperLogLog是一种概率数据结构，用于对唯一事物进行计数（从技术上讲，这是指估计集合的基数） +### 🎯 Redis的扩展性 -### 布隆过滤器 +读扩展，基于主从架构，可以很好的平行扩展读的能力。写扩展，主要受限于主服务器的硬件资源的限制，一是单个实例内存容量受限，二是一个实例只使用到CPU一个核。下面讨论基于多套主从架构Redis实例的集群实现，目前主要有以下几种方案： +1. 客户端分片 实现方案，业务进程通过对key进行hash来分片，用Sentinel做failover。优点：运维简单，每个实例独立部署；可使用lua脚本，业务进程执行的key均hash到同一个分片即可；缺点：一旦重新分片，由于数据无法自动迁移，部分数据需要回源； +2. Redis集群 是官方提供的分布式数据库方案，通过分片实现数据共享，并提供复制和failover。按照16384个槽位进行分片，且实例之间共享分片视图。优点：当发生重新分片时，数据可以自动迁移；缺点：客户端需要升级到支持集群协议的版本；客户端需要感知分片实例，最坏的情况，每个key需要一次重定向；不支持lua脚本；不支持pipeline； +3. Codis 是由豌豆荚团队开源的一款分布式组件，它将分布式的逻辑从Redis集群剥离出来，交由几个组件来完成，与数据的读写解耦。Codis proxy负责分片和聚合，dashboard作为管理后台，zookeeper做配置管理，Sentinel做failover。优点：底层透明，客户端兼容性好；重新分片时，数据可自动迁移；支持pipeline；支持lua脚本，业务进程保证执行的key均hash到同一个分片即可；缺点：运维较为复杂；引入了中间层； -### 压缩列表了解吗？ -这是 Redis **为了节约内存** 而使用的一种数据结构，**zset** 和 **hash** 容器对象会在元素个数较少的时候，采用压缩列表（ziplist）进行存储。压缩列表是 **一块连续的内存空间**，元素之间紧挨着存储，没有任何冗余空隙。 +------ -> 因为之前自己也没有学习过，所以找了一篇比较好比较容易理解的文章： -> -> - 图解Redis之数据结构篇——压缩列表 - https://mp.weixin.qq.com/s/nba0FUEAVRs0vi24KUoyQg -> - 这一篇稍微底层稍微硬核一点：http://www.web-lovers.com/redis-source-ziplist.html +## 六、性能优化与异常处理 + +### 🎯 Redis常见性能问题和解决方案？ + +Redis 的常见性能问题和解决方案包括但不限于以下几点： + +1. **内存问题**： + - **问题**：内存不足或内存碎片导致性能下降。 + - **解决方案**：使用 `MEMORY USAGE` 命令监控内存使用情况，优化数据结构，合理配置 `maxmemory` 并选择合适的内存淘汰策略。 +2. **高并发下的响应延迟**： + - **问题**：在高并发访问时，响应时间变长。 + - **解决方案**：优化命令使用，避免使用耗时的命令，使用 Pipelining 技术批量执行命令，考虑使用 Redis 集群进行负载均衡。 +3. **慢查询**： + - **问题**：执行慢查询导致阻塞。 + - **解决方案**：使用 `slowlog` 命令找出并优化慢查询，确保单次操作尽可能快速。 +4. **主从复制延迟**： + - **问题**：主从复制延迟影响数据的实时性。 + - **解决方案**：优化网络条件，升级硬件，使用更高效的复制协议如 PSYNC，考虑使用无磁盘化复制减少延迟。 +5. **持久化性能问题**： + - **问题**：RDB 和 AOF 持久化影响性能。 + - **解决方案**：合理配置持久化策略，如 AOF 刷盘策略 `everysec`，或使用 RDB-AOF 混合持久化。 +6. **热 Key 问题**： + - **问题**：某些键被频繁访问，成为热点，可能导致单个实例负载过高。 + - **解决方案**：使用本地缓存减轻 Redis 负担，或在多个实例间分散热点数据。 +7. **数据结构选择**： + - **问题**：使用不合适的数据结构导致性能问题。 + - **解决方案**：根据数据类型和操作需求选择合适的数据结构，例如使用 intset 代替普通列表存储整数列表。 +8. **连接数过多**： + - **问题**：大量客户端连接可能会导致资源耗尽。 + - **解决方案**：使用连接池、限制最大客户端连接数、优化客户端连接管理。 +9. **单线程阻塞**： + - **问题**：由于单线程模型，阻塞操作会影响性能。 + - **解决方案**：避免使用耗时的单个命令，如 `KEYS`、`FLUSHALL`、`FLUSHDB` 等，使用 `SCAN` 替代。 +10. **版本问题**： + - **问题**：使用过时的 Redis 版本可能会导致性能问题。 + - **解决方案**：升级到最新稳定版本的 Redis，以利用性能改进和新特性。 +11. **监控和告警**： + - **问题**：缺乏监控和告警可能导致性能问题被忽视。 + - **解决方案**：实施 Redis 监控策略，使用工具如 Redis `INFO` 命令、`redis-cli` 工具等进行性能监控和调优。 + +针对 Redis 的性能问题，解决方案通常需要根据具体的业务场景和需求来定制。在设计和构建应用时，应考虑到 Redis 的特点，合理使用其提供的各种功能和命令，以确保系统的高性能和稳定性 + + + +### 🎯 如何保证缓存与数据库双写时的数据一致性？ + +> 在缓存与数据库双写时，一致性问题主要出在写操作的并发顺序上。 +> 实际上，最常见的方案是 **Cache Aside 模式**：更新数据库后删除缓存，而不是更新缓存，这样能保证缓存一定会从数据库回源到最新值。 +> 为了进一步提升一致性，可以用 **延迟双删策略**，或者用 **消息队列、订阅 binlog** 来保证最终一致性。 +> 不同业务场景会有不同选择：对一致性要求特别高的，可以考虑引入 MQ 或 Canal；而大部分互联网场景下，删除缓存 + 延迟双删就足够了。 + +**📊 缓存一致性方案对比** + +| 模式 | 核心流程 | 一致性 | 优点 | 缺点 | 适用场景 | +| ----------------------------------------- | ----------------------------------------------------------- | -------------------------- | --------------------------------------- | ------------------------------------------------ | ------------------------------------------ | +| **Cache Aside**（旁路缓存，最常用 ✅） | 读：缓存 miss → 查 DB → 回填缓存；写：先更新 DB，再删除缓存 | 最终一致（可能短暂不一致） | 简单、灵活、业界广泛使用 | 写后可能有短暂脏读；高并发下可能缓存回填旧值 | 电商详情页、商品库存、用户信息（读多写少） | +| **Read/Write Through**（读写穿透） | 读：缓存 miss 由缓存加载 DB；写：写缓存，由缓存写 DB | 强一致（由缓存代理 DB） | 应用只依赖缓存，不直接访问 DB，逻辑清晰 | 实现复杂，缓存需支持代理数据库写（Redis 不常用） | 一些分布式 KV 系统（Memcached+proxy） | +| **Write Behind / Write Back**（写后缓存） | 写：只写缓存，异步刷 DB；读：只读缓存 | 弱一致（DB 延迟更新） | 写性能极高，支持批量异步落库 | 数据库可能落后，宕机可能丢数据 | 计数器、日志、非强一致性业务 | +| **延迟双删策略**（Cache Aside 优化） | 更新 DB → 删除缓存 → 延迟一段时间后再删缓存 | 更强的最终一致性 | 解决缓存回填脏数据问题 | 需要设置合理延迟时间，增加复杂度 | 高并发场景下的 Cache Aside 加强版 | +| **MQ 异步删除缓存** | 更新 DB → 发 MQ 消息 → 消费者删除缓存 | 高一致性（依赖 MQ） | 削峰填谷，保证一致性 | 依赖 MQ，系统复杂度高 | 高并发核心业务（订单、支付） | +| **订阅 Binlog（如 Canal）** | DB 更新 → Binlog → 消费并删除/更新缓存 | 准实时一致性 | 无需侵入应用，解耦好 | 延迟依赖 Binlog 消费速度 | 电商订单、金融交易等强一致场景 | + +> 1. **没有银弹方案**：一致性方案需平衡业务需求（如延迟容忍度、吞吐量）与技术成本（如引入分布式锁可能增加系统复杂度）。 +> 2. **监控与补偿**： +> - 增加缓存与数据库的对账监控（如定时任务对比关键数据，或通过 Canal 的增量订阅做实时校验）。 +> - 对于不一致数据，通过人工介入或自动任务（如 Redis Lua 脚本批量修复）进行补偿。 + + + +### 🎯 使用缓存会出现什么问题？ + +#### 缓存雪崩 + +缓存雪崩是指缓存同一时间大面积的失效，所以，后面的请求都会落到数据库上，造成数据库短时间内承受大量请求而崩掉。 +**解决方案** +1. 缓存数据的过期时间设置随机，防止同一时间大量数据过期现象发生。 -### 快速列表 quicklist 了解吗？ + > 这个随机时间也是有讲究的，我们假设过期时间是 10 分钟，那要在这个基础上加一个 0-210左右秒的“偏移量”都可以的，这个偏移量要跟过期时间成正比，不能过低或者过高 -Redis 早期版本存储 list 列表数据结构使用的是压缩列表 ziplist 和普通的双向链表 linkedlist，也就是说当元素少时使用 ziplist，当元素多时用 linkedlist。但考虑到链表的附加空间相对较高，`prev` 和 `next` 指针就要占去 `16` 个字节（64 位操作系统占用 `8` 个字节），另外每个节点的内存都是单独分配，会家具内存的碎片化，影响内存管理效率。 +2. 一般并发量不是特别多的时候，使用最多的解决方案是加锁排队(key上锁，其他线程不能访问，假设在高并发下，缓存重建期间key是锁着的，这是过来1000个请求999个都在阻塞的。同样会导致用户等待超时，这是个治标不治本的方法！)。 -后来 Redis 新版本（3.2）对列表数据结构进行了改造，使用 `quicklist` 代替了 `ziplist` 和 `linkedlist`。 +3. 给每一个缓存数据增加相应的缓存标记，记录缓存的是否失效，如果缓存标记失效，则更新数据缓存。 -> 同上..建议阅读一下以下的文章： -> -> - Redis列表list 底层原理 - https://zhuanlan.zhihu.com/p/102422311 +4. 设置热点数据静态化，把访问量较大的数据做静态化处理，减少数据库的访问。 +#### 缓存穿透 +缓存穿透是指缓存和数据库中都没有的数据，导致所有的请求都落到数据库上，造成数据库短时间内承受大量请求而崩掉。 ------- +**解决方案** +1. 接口层增加校验，如用户鉴权校验，id做基础校验，id<=0 的直接拦截； +2. **回写特殊值**：从缓存取不到的数据，在数据库中也没有取到，这时也可以将 key-value 对写为 key-null，缓存有效时间可以设置短点，如 30 秒（设置太长会导致正常情况也没法使用）。这样可以防止攻击用户反复用同一个id暴力攻击 -## 三、Redis持久化问题 + > 如果攻击者每次都用不同的且都不存在的 key 来请求数据，那么这种措施毫无 效果。并且，因为要回写特殊值，那么这些不存在的 key 都会有特殊值，浪费 了 Redis 的内存。这可能会进一步引起另外一个问题，就是 Redis 在内存不 足，执行淘汰的时候，把其他有用的数据淘汰掉。 -> 你对redis的持久化机制了解吗？能讲一下吗？ +3. 采用布隆过滤器，将所有可能存在的数据哈希到一个足够大的 bitmap 中，一个一定不存在的数据会被这个 bitmap 拦截掉，从而避免了对底层存储系统的查询压力。 -Redis 的数据全部在内存里，如果突然宕机，数据就会全部丢失，因此必须有一种机制来保证 Redis 的数据不会因为故障而丢失，这种机制就是 Redis 的持久化机制。 + > 但是布隆过滤器本身存在假阳性的问题，所以当攻击者请求一个不存在的 key 的时候，布隆过滤器可能会返回数据存在的假阳性响应。在这种情况下，业务 代码依旧会去查询缓存和数据库。不过这个不需要担心，因为假阳性的概率是 很低的。假如说假阳性概率是万分之一，那么就算攻击的并发有百万，也只有 100 个查询请求会落到数据库上，这一点查询请求就是毛毛雨了。 -**Redis有两种持久化的方式：快照（RDB文件）和追加式文件（AOF文件)** +#### 缓存击穿 -### RDB（Redis DataBase） +> 某明星直播时，粉丝反复刷新其主页导致缓存击穿，数据库压力激增，如何设计多级防护策略？ -**在指定的时间间隔内将内存中的数据集快照写入磁盘**，也就是行话讲的 Snapshot 快照，它恢复时是将快照文件直接读到内存里。 +缓存击穿是指缓存中没有但数据库中有的数据（一般是缓存时间到期），这时由于并发用户特别多，同时读缓存没读到数据，又同时去数据库去取数据，引起数据库压力瞬间增大，造成过大压力。 -Redis会单独创建（fork）一个子进程来进行持久化，会先将数据写入到一个临时文件中，待持久化过程都结束了，再用这个临时文件替换上次持久化好的文件。整个过程中，主进程是不进行任何 IO 操作的，这就确保了极高的性能，如果需要进行大规模数据的恢复，且对于数据恢复的完整性不是非常敏感，那 RDB 方式要比 AOF 方式更加的高效。RDB的缺点是最后一次持久化后的数据可能丢失。 +缓存击穿是指一个 Key 非常热点，在不停地扛着大量的请求，大并发集中对这一个点进行访问，当这个 Key 在失效的瞬间，持续的大并发直接落到了数据库上，就在这个 Key 的点上击穿了缓存 -> What ? Redis 不是单进程的吗? +> **和缓存雪崩不同的是，缓存击穿指并发查同一条数据，缓存雪崩是不同数据都过期了，很多数据都查不到从而查数据库。** -Redis 使用操作系统的多进程 COW(Copy On Write) 机制来实现快照持久化， fork 是类Unix操作系统上**创建进程**的主要方法。COW(Copy On Write)是计算机编程中使用的一种优化策略。 +**解决方案** -fork 的作用是复制一个与当前进程一样的进程。新进程的所有数据（变量、环境变量、程序计数器等）数值都和原进程一致，但是是一个全新的进程，并作为原进程的子进程。 子进程读取数据，然后序列化写到磁盘中。 +1. 热点数据永远不过期：物理不过期，但逻辑过期(后台异步线程去刷新) -rdb 默认保存的是 **dump.rdb** 文件 +2. 使用互斥锁：当缓存失效时，不立即去 load db，先使用如 Redis 的 setnx 去设置一个互斥锁，当操作成功返回时再进行 load db 的操作并回设缓存，否则重试 get 缓存的方法 -你可以对 Redis 进行设置， 让它在“ N 秒内数据集至少有 M 个改动”这一条件被满足时， 自动保存一次数据集。 +#### 缓存预热 -你也可以通过调用 [SAVE](http://redisdoc.com/server/save.html#save) 或者 [BGSAVE](http://redisdoc.com/server/bgsave.html#bgsave) ， 手动让 Redis 进行数据集保存操作。 +缓存预热就是系统上线后，将相关的缓存数据直接加载到缓存系统。这样就可以避免在用户请求的时候，先查询数据库，然后再将数据缓存的问题！用户直接查询事先被预热的缓存数据！ -比如说， 以下设置会让 Redis 在满足“ 60 秒内有至少有 1000 个键被改动”这一条件时， 自动保存一次数据集： +**解决方案** -``` -save 60 1000 -``` +1. 直接写个缓存刷新页面，上线时手工操作一下； +2. 数据量不大，可以在项目启动的时候自动进行加载； +3. 定时刷新缓存； -### AOF（Append Only File） +#### 缓存降级 -以日志的形式来记录每个写操作，将 Redis 执行过的所有写指令记录下来(读操作不记录)，只许追加文件但不可以改写文件，redis 启动之初会读取该文件重新构建数据，也就是「重放」。换言之，redis 重启的话就根据日志文件的内容将写指令从前到后执行一次以完成数据的恢复工作。 +当访问量剧增、服务出现问题（如响应时间慢或不响应）或非核心服务影响到核心流程的性能时，仍然需要保证服务还是可用的，即使是有损服务。系统可以根据一些关键数据进行自动降级，也可以配置开关实现人工降级。 -AOF 默认保存的是 **appendonly.aof ** 文件 +**缓存降级的最终目的是保证核心服务可用，即使是有损的。而且有些服务是无法降级的（如加入购物车、结算）。** +在进行降级之前要对系统进行梳理，看看系统是不是可以丢卒保帅；从而梳理出哪些必须誓死保护，哪些可降级；比如可以参考日志级别设置预案： +1. 一般：比如有些服务偶尔因为网络抖动或者服务正在上线而超时，可以自动降级； -#### Redis 4.0 的混合持久化 +2. 警告：有些服务在一段时间内成功率有波动（如在95~100%之间），可以自动降级或人工降级，并发送告警； -重启 Redis 时，我们很少使用 `rdb` 来恢复内存状态，因为会丢失大量数据。我们通常使用 AOF 日志重放，但是重放 AOF 日志性能相对 `rdb` 来说要慢很多，这样在 Redis 实例很大的情况下，启动需要花费很长的时间。 +3. 错误：比如可用率低于90%，或者数据库连接池被打爆了，或者访问量突然猛增到系统能承受的最大阀值，此时可以根据情况自动降级或者人工降级； -**Redis 4.0** 为了解决这个问题，带来了一个新的持久化选项——**混合持久化**。将 `rdb` 文件的内容和增量的 AOF 日志文件存在一起。这里的 AOF 日志不再是全量的日志，而是 **自持久化开始到持久化结束** 的这段时间发生的增量 AOF 日志，通常这部分 AOF 日志很小： +4. 严重错误：比如因为特殊原因数据错误了，此时需要紧急人工降级。 -![img](https://imgkr.cn-bj.ufileos.com/242f636b-8943-45bb-ab04-1a739866462a.png) +服务降级的目的，是为了防止 Redis 服务故障，导致数据库跟着一起发生雪崩问题。因此，对于不重要的缓存数据，可以采取服务降级策略，例如一个比较常见的做法就是，Redis出现问题，不去数据库查询，而是直接返回默认值给用户。 -于是在 Redis 重启的时候，可以先加载 `rdb` 的内容，然后再重放增量 AOF 日志就可以完全替代之前的 AOF 全量文件重放，重启效率因此大幅得到提升。 +#### 缓存热点 key -> 关于两种持久化方式的更多细节 *(原理)* 可以参考： https://www.wmyskxz.com/2020/03/13/redis-7-chi-jiu-hua-yi-wen-liao-jie/ +缓存中的一个 Key(比如一个促销商品)，在某个时间点过期的时候，恰好在这个时间点对这个 Key 有大量的并发请求过来，这些请求发现缓存过期一般都会从后端 DB 加载数据并回设到缓存，这个时候大并发的请求可能会瞬间把后端 DB 压垮。 +**解决方案** +1. 对缓存查询加锁，如果 KEY 不存在，就加锁，然后查 DB 入缓存，然后解锁； +2. 其他进程如果发现有锁就等待，然后等解锁后返回数据或者进入 DB 查询 -### Which one -- RDB 持久化方式能够在指定的时间间隔能对你的数据进行快照存储 -- AOF 持久化方式记录每次对服务器写的操作,当服务器重启的时候会重新执行这些命令来恢复原始的数据,AOF命令以 redis 协议追加保存每次写的操作到文件末尾。Redis还能对AOF文件进行后台重写（**bgrewriteaof**）,使得 AOF 文件的体积不至于过大 -- 只做缓存：如果你只希望你的数据在服务器运行的时候存在,你也可以不使用任何持久化方式。 -- 同时开启两种持久化方式 - - 在这种情况下,当 redis 重启的时候会优先载入 AOF 文件来恢复原始的数据,因为在通常情况下 AOF 文件保存的数据集要比 RDB 文件保存的数据集要完整。 - - RDB 的数据不实时，同时使用两者时服务器重启也只会找 AOF 文件。那要不要只使用AOF 呢？建议不要，因为 RDB 更适合用于备份数据库(AOF 在不断变化不好备份)，快速重启，而且不会有 AOF 可能潜在的bug，留着作为一个万一的手段。 +### 🎯 Redis 大 key 和 热 Key 问题 +> - 大 Key：单个 Key 的值或集合特别大，导致“删除/过期/迁移/备份/访问”都阻塞，常见是一个 Hash/List/ZSet 里塞了几万到几百万元素。 +> +> - 热 Key：少数 Key 被高频读写，造成单节点/单线程瓶颈，集群出现槽位倾斜。 +> +> - 我的做法：先“识别”再“治理”。大 Key按“拆/删/搬/压缩”处理；热 Key按“多级缓存/请求合并/读写分散/逻辑过期/副本读”组合拳。治理目标是“降低单 Key 的大小与热点度”，把负载摊平。 -### 性能建议 +Redis 的过程中，如果未能及时发现并处理 Big keys（下文称为“大Key”）与 Hotkeys（下文称为“热Key”），可能会导致服务性能下降、用户体验变差，甚至引发大面积故障 -- 因为 RDB 文件只用作后备用途，建议只在 Slave上持久化 RDB 文件，而且只要15分钟备份一次就够了，只保留save 900 1这条规则。 -- 如果Enalbe AOF，好处是在最恶劣情况下也只会丢失不超过两秒数据，启动脚本较简单只load自己的 AOF 文件就可以了。代价一是带来了持续的 IO，二是 AOF rewrite 的最后将 rewrite 过程中产生的新数据写到新文件造成的阻塞几乎是不可避免的。只要硬盘许可，应该尽量减少 AOF rewrite 的频率，AOF 重写的基础大小默认值64M太小了，可以设到5G以上。默认超过原大小100%大小时重写可以改到适当的数值。 -- 如果不 Enable AOF ，仅靠 Master-Slave Replication 实现高可用性也可以。能省掉一大笔 IO ，也减少了rewrite 时带来的系统波动。代价是如果 Master/Slav e同时宕掉，会丢失十几分钟的数据，启动脚本也要比较两个Master/Slave中的RDB文件，载入较新的那个。 +#### 一、大Key ------- +通常以Key的大小和Key中成员的数量来综合判定，例如： +1. **数据量过大**：单个 Key 的 Value 过大（如 String 类型 Key 值超过 5MB）。 +2. **成员数过多**：集合类型 Key 的成员数量过多（如 Hash/ZSet 成员数超过 1 万）。 +3. **成员总大小过大**：集合类型 Key 的总大小过大（如 Hash 成员总大小超过 100MB）。 +##### 引发的问题 -## 四、Redis事务问题 +- **读写性能劣化**：操作大 Key 时耗时增加，阻塞其他请求。 +- **内存压力**：触发内存淘汰策略（LRU/LFU），导致重要数据被逐出，甚至 OOM。 +- **集群分片不均**：单个分片内存或带宽使用率远高于其他节点，破坏负载均衡。 +- **删除阻塞**：删除大 Key 可能导致主线程阻塞（如删除百万成员的 Hash）。 -**Redis事务的概念？** +##### 原因 -Redis 事务的本质是通过MULTI、EXEC、WATCH等一组命令的集合。事务支持一次执行多个命令，一个事务中所有命令都会被序列化。在事务执行过程，会按照顺序串行化执行队列中的命令，其他客户端提交的命令请求不会插入到事务执行命令序列中。 +- 在不适用的场景下使用Redis，易造成Key的value过大，如使用String类型的Key存放大体积二进制文件型数据； +- 业务上线前规划设计不足，没有对Key中的成员进行合理的拆分，造成个别Key中的成员数量过多； +- 未定期清理无效数据，造成如HASH类型Key中的成员持续不断地增加； +- 使用LIST类型Key的业务消费侧发生代码故障，造成对应Key的成员只增不减。 -总结说：redis事务就是一次性、顺序性、排他性的执行一个队列中的一系列命令。 +##### **如何识别** -[MULTI](http://redisdoc.com/transaction/multi.html#multi) 命令用于开启一个事务，它总是返回 OK 。 +- 线上安全方式：redis-cli --bigkeys（抽样），或用 SCAN + MEMORY USAGE key、TYPE/HLEN/LLEN/ZCARD统计。 -[MULTI](http://redisdoc.com/transaction/multi.html#multi) 执行之后， 客户端可以继续向服务器发送任意多条命令， 这些命令不会立即被执行， 而是被放到一个队列中， 当 [EXEC](http://redisdoc.com/transaction/exec.html#exec) 命令被调用时， 所有队列中的命令才会被执行。 +- “删除慢/过期卡顿/迁移阻塞/CPU飙高”通常是信号。注意 DEL 大 Key 会阻塞主线程。 -另一方面， 通过调用 [DISCARD](http://redisdoc.com/transaction/discard.html#discard) ， 客户端可以清空事务队列， 并放弃执行事务。 +##### 大 Key怎么治理（思路与落地） -[WATCH](http://redisdoc.com/transaction/watch.html#watch) 使得 [EXEC](http://redisdoc.com/transaction/exec.html#exec) 命令需要有条件地执行： 事务只能在所有被监视键都没有被修改的前提下执行， 如果这个前提不能满足的话，事务就不会被执行。 +- 拆分（首选）： -**2、Redis事务的三个阶段** + - 大 JSON 切分为多个子 Key；大集合按分片/时间窗口/业务维度拆，控制每片元素数（建议<1万）。 -事务开始 MULTI、命令入队、事务执行 EXEC + - 集群下用“哈希标签”把分片均匀落到多个 slot，例如 user:{123}:feed:0..N。 -**3、Redis事务支持隔离性吗**？ +- 删除与过期： -Redis 是单进程程序，并且它保证在执行事务时，不会对事务进行中断，事务可以运行直到执行完所有事务队列中的命令为止。因此，**Redis 的事务是总是带有隔离性的**。 + - 用 UNLINK（异步删除）代替 DEL，避免主线程阻塞；或对集合用 HSCAN/SSCAN/ZSCAN 分批删。 -**4、Redis事务保证原子性吗，支持回滚吗？** + ```bash + UNLINK big_key # 异步释放内存 + HSCAN h 0 COUNT 1000 ... # 批量HDEL + ``` -Redis中，单条命令是原子性执行的，但**事务不保证原子性，且没有回滚**。事务中任意命令执行失败，其余的命令仍会被执行。 +- 压缩与结构选择： -1. **如果在一个事务中的命令出现错误，那么所有的命令都不会执行**； -2. **如果在一个事务中出现运行错误，那么正确的命令会被执行**。 + - 文本用 Snappy/LZ4 压缩（权衡 CPU）；大 Hash 适度分裂；合理设置 listpack/quicklist 阈值，开启 activedefrag。 ------- +- 运维配置建议： + - lazyfree-lazy-eviction yes、lazyfree-lazy-expire yes、lazyfree-lazy-server-del yes、activedefrag yes。 -## 五、Redis集群问题 -> redis单节点存在单点故障问题，为了解决单点问题，一般都需要对redis配置从节点，然后使用哨兵来监听主节点的存活状态，如果主节点挂掉，从节点能继续提供缓存功能，你能说说redis主从复制的过程和原理吗？ +#### 二、热Key -#### 什么是哨兵 +热 Key 的特征是**访问频率显著高于其他 Key**，表现为： -**哨兵的介绍** +1. **QPS 倾斜**：单个 Key 的 QPS 占总 QPS 的 70% 以上（如总 QPS 1 万，某 Key 占 7 千）。 +2. **带宽集中**：对大集合 Key 高频读取（如频繁调用 `HGETALL`）。 +3. **CPU 占用高**：对复杂结构 Key 高频操作（如大量 `ZRANGE` 操作消耗 CPU）。 -sentinel，中文名是哨兵。哨兵是 redis 集群机构中非常重要的一个组件，主要有以下功能： +##### 引发的问题 -1. 集群监控：负责监控 redis master 和 slave 进程是否正常工作。 -2. 消息通知：如果某个 redis 实例有故障，那么哨兵负责发送消息作为报警通知给管理员。 -3. 故障转移：如果 master node 挂掉了，会自动转移到 slave node 上。 -4. 配置中心：如果故障转移发生了，通知 client 客户端新的 master 地址。 +- **资源争抢**：CPU/带宽被热 Key 独占，其他请求排队甚至超时。 +- **缓存击穿**：热 Key 失效瞬间，大量请求直接穿透到数据库，引发雪崩。 +- **分片热点**：集群模式下单个分片负载过高，导致整体服务不可用。 -哨兵用于实现 redis 集群的高可用，本身也是分布式的，作为一个哨兵集群去运行，互相协同工作。 +##### 典型原因 -**哨兵的核心知识** +- **突发流量**：热点新闻、秒杀活动、直播间刷屏等场景。 +- **设计不合理**：全局配置 Key（如系统开关）被高频读取。 +- **缓存策略缺失**：未对热 Key 进行多级缓存或负载分散。 -1. 哨兵至少需要 3 个实例，来保证自己的健壮性。 -2. 哨兵 + redis 主从的部署架构，是不保证数据零丢失的，只能保证 redis 集群的高可用性。 -3. 对于哨兵 + redis 主从这种复杂的部署架构，尽量在测试环境和生产环境，都进行充足的测试和演练。 +**如何识别** -#### redis 集群模式的工作原理能说一下么？ +- redis-cli --hotkeys（基于 LFU 频次，需要配置 LFU 策略）； -Redis Cluster是一种服务端Sharding技术，3.0版本开始正式提供。Redis Cluster并没有使用一致性hash，而是采用slot(槽)的概念，一共分成16384个槽。将请求发送到任意节点，接收到请求的节点会将查询请求发送到正确的节点上执行。 +- 业务侧埋点统计命中分布（Top N Key）； +- Redis Exporter 观测“命中率/回源/单节点 QPS/slot 倾斜”。 +##### 热 Key怎么治理（思路与落地） +- 读优化（优先）： + - 本地 L1 缓存（Caffeine）+ Redis L2，多级缓存；热点 Key 逻辑过期，返回旧值异步刷新，避免击穿。 -**5、Redis Sharding如何实现的？** + - 单飞（请求合并）：对同一 Key 的 miss 回源只允许一个线程执行，其它等待。 -Redis Sharding是Redis Cluster出来之前，业界普遍使用的多Redis实例集群方法。其主要思想是采用哈希算法将Redis数据的key进行散列，通过hash函数，特定的key会映射到特定的Redis节点上。Java redis客户端驱动jedis，支持Redis Sharding功能，即ShardedJedis以及结合缓存池的ShardedJedisPool +- 分散热点： -**6、Redis 主从架构原理** + - 读扩散（复制 N份）：将同一值写入k#0..k#N-1多个 Key，读侧随机挑一份；一致性靠短 TTL或版本号。 -单机的 redis，能够承载的 QPS 大概就在上万到几万不等。对于缓存来说，一般都是用来支撑读高并发的。因此架构做成主从(master-slave)架构，一主多从，主负责写，并且将数据复制到其它的 slave 节点，从节点负责读。所有的读请求全部走从节点。这样也可以很轻松实现水平扩容，支撑读高并发。 + ```java + // 写：N 份复制 + for (int i=0;i *SET* 命令的行为可以通过一系列参数来修改 +> +> - `EX second` ：设置键的过期时间为 `second` 秒。 `SET key value EX second` 效果等同于 `SETEX key second value` 。 +> - `PX millisecond` ：设置键的过期时间为 `millisecond` 毫秒。 `SET key value PX millisecond` 效果等同于 `PSETEX key millisecond value` 。 +> - `NX` ：只在键不存在时，才对键进行设置操作。 `SET key value NX` 效果等同于 `SETNX key value` 。 +> - `XX` ：只在键已经存在时，才对键进行设置操作。 -（3）查询路由(Query routing) 的意思是客户端随机地请求任意一个redis实例，然后由Redis将请求转发给正确的Redis节点。Redis Cluster实现了一种混合形式的查询路由，但并不是直接将请求从一个redis节点转发到另一个redis节点，而是在客户端的帮助下直接redirected到正确的redis节点。 +```sh +SET resource_name my_random_value NX PX 30000 +``` -**13、Redis分区有什么缺点？** +这条指令的意思：当 key——resource_name 不存在时创建这样的 key，设值为 my_random_value，并设置过期时间 30000 毫秒。 -（1）涉及多个key的操作通常不会被支持。例如你不能对两个集合求交集，因为他们可能被存储到不同的Redis实例（实际上这种情况也有办法，但是不能直接使用交集指令）。 +别看这干了两件事，因为 Redis 是单线程的，这一条指令不会被打断，所以是原子性的操作。 -（2）同时操作多个key,则不能使用Redis事务. +Redis 实现分布式锁的主要步骤： -（3）分区使用的粒度是key，不能使用一个非常长的排序key存储一个数据集 +1. 指定一个 key 作为锁标记，存入 Redis 中，指定一个 **唯一的标识** 作为 value。 +2. 当 key 不存在时才能设置值，确保同一时间只有一个客户端进程获得锁，满足 **互斥性** 特性。 +3. 设置一个过期时间，防止因系统异常导致没能删除这个 key，满足 **防死锁** 特性。 +4. 当处理完业务之后需要清除这个 key 来释放锁，清除 key 时需要校验 value 值，需要满足 **解铃还须系铃人** 。 -（4）当使用分区的时候，数据处理会非常复杂，例如为了备份你必须从不同的Redis实例和主机同时收集RDB / AOF文件。 +设置一个随机值的意思是在解锁时候判断 key 的值和我们存储的随机数是不是一样，一样的话，才是自己的锁，直接 `del` 解锁就行。 -（5）分区时动态扩容或缩容可能非常复杂。Redis集群在运行时增加或者删除Redis节点，能做到最大程度对用户透明地数据再平衡，但其他一些客户端分区或者代理分区方法则不支持这种特性。然而，有一种预分片的技术也可以较好的解决这个问题。 +当然这个两个操作要保证原子性，所以 Redis 给出了一段 lua 脚本（Redis 服务器会单线程原子性执行 lua 脚本，保证 lua 脚本在处理的过程中不会被任意其它请求打断。）： -**14、Redis如何实现分布式锁？** +```lua +if redis.call("get",KEYS[1]) == ARGV[1] then + return redis.call("del",KEYS[1]) +else + return 0 +end +``` -使用SETNX完成同步锁的流程及事项如下： -使用SETNX命令获取锁，若返回0（key已存在，锁已存在）则获取失败，反之获取成功 -为了防止获取锁后程序出现异常，导致其他线程/进程调用SETNX命令总是返回0而进入死锁状态，需要为该key设置一个“合理”的过期时间 +### 🎯 上述 Redis 分布式锁的缺点 -释放锁，使用DEL命令将锁数据删除 +1. **单点故障**：如果 Redis 服务器出现故障，整个分布式锁服务将不可用。尽管可以通过 Redis 集群或哨兵模式提高可用性，但这些方法会增加系统的复杂性。 -**15、如何解决 Redis 的并发竞争 Key 问题** +2. **锁失效问题**：由于网络延迟或 Redis 服务器负载高等原因，设置的锁可能会在预期的时间之前失效。如果锁失效时间过短，业务逻辑可能还未完成就会失去锁，导致数据不一致。 -所谓 Redis 的并发竞争 Key 的问题也就是多个系统同时对一个 key 进行操作，但是最后执行的顺序和我们期望的顺序不同，这样也就导致了结果的不同！ +3. **时钟漂移**：Redis 分布式锁依赖于系统时间。如果多个节点的系统时间不同步，可能会导致锁的时间计算错误，进而引发锁的竞争和数据一致性问题。 -推荐一种方案：分布式锁（zookeeper 和 redis 都可以实现分布式锁）。（如果不存在 Redis 的并发竞争 Key 问题，不要使用分布式锁，这样会影响性能） +4. **不可重入性**：Redis 分布式锁通常是不可重入的，这意味着一个持有锁的线程不能再次获得同一个锁。如果业务逻辑中存在重入需求，需要额外的设计来处理。 【可以考虑使用 Redisson 等工具，它们提供了可重入锁的封装】 -zookeeper分布式锁准备在今后的面试文章中提到。 +5. **原子性和一致性** -**16、分布式Redis是前期做还是后期规模上来了再做好？为什么？** + 尽管使用 `SET NX PX` 命令可以实现锁的基本原子性，但在处理锁的释放、续租等复杂场景时，需要小心处理原子性和一致性。例如，在释放锁时，如果释放锁的客户端在删除锁之前崩溃，可能会导致锁无法正确释放。 -既然Redis是如此的轻量（单实例只使用1M内存），为防止以后的扩容，最好的办法就是一开始就启动较多实例。即便你只有一台服务器，你也可以一开始就让Redis以分布式的方式运行，使用分区，在同一台服务器上启动多个实例。 +6. **锁超时和业务时间不匹配** -一开始就多设置几个Redis实例，例如32或者64个实例，对大多数用户来说这操作起来可能比较麻烦，但是从长久来看做这点牺牲是值得的。 + 设置的锁超时时间可能和业务实际执行时间不匹配，特别是在业务执行时间不可预期的情况下。过短的锁超时时间可能导致锁在业务未完成时被其他节点获取，过长的锁超时时间则可能降低系统并发性。 -这样的话，当你的数据不断增长，需要更多的Redis服务器时，你需要做的就是仅仅将Redis实例从一台服务迁移到另外一台服务器而已（而不用考虑重新分区的问题）。一旦你添加了另一台服务器，你需要将你一半的Redis实例从第一台机器迁移到第二台机器。 +7. **主从复制问题**：在 Redis 主从复制模式下，如果主节点宕机，从节点被提升为新的主节点，可能会导致锁丢失，从而产生并发问题 +8. 客户端实现复杂：实现一个健壮的分布式锁需要处理很多细节问题，如锁的续租、锁的过期等，这些会增加客户端的实现复杂度。尽管有一些成熟的库（如 Redisson）可以帮助简化这些操作，但依然需要谨慎使用和配置。 -**17、什么是 RedLock** +8. 性能开销：虽然 Redis 的性能很高，但频繁的锁操作（获取、续租、释放等）会对 Redis 服务器造成一定的压力，特别是在高并发环境下。 -Redis 官方站提出了一种权威的基于 Redis 实现分布式锁的方式名叫 Redlock，此种方式比原先的单节点的方法更安全。它可以保证以下特性： -（1）安全特性：互斥访问，即永远只有一个 client 能拿到锁 -（2）避免死锁：最终 client 都可能拿到锁，不会出现死锁的情况，即使原本锁住某资源的 client crash 了或者出现了网络分区 +### 🎯 Redis实现分布式锁，还有其他方式么，zookeeper怎么实现，各有什么有缺点，你们为什么用redis实现 -（3）容错性：只要大部分 Redis 节点存活就可以正常提供服务 +> 分布式锁常见有三种实现：基于 **Redis、Zookeeper 和数据库**。 +> Redis 基于 `SETNX` 实现，性能高，适合高并发，但需要考虑过期续期和主从同步带来的锁丢失问题； +> Zookeeper 基于临时顺序节点和 Watch 机制，可靠性和公平性好，但 QPS 不高，适合强一致性场景； +> 数据库基于唯一索引或行锁，简单但性能差，容易造成死锁。 +> +> 我们项目里用的是 **Redis 实现分布式锁**，主要原因是系统对性能要求极高（QPS 十万级），Redis 锁的延迟更低，而且 Redisson 已经帮我们封装了续期和可重入机制，开发和维护成本低。 +| 实现方式 | 原理 | 优点 | 缺点 | 适用场景 | +| ------------- | --------------------------- | -------------------- | ------------------------ | ---------------------------- | +| **Redis** | SETNX/RedLock，过期时间控制 | 高性能，简单，生态好 | 锁丢失风险，需要续期机制 | 高并发、对性能敏感 | +| **Zookeeper** | 临时顺序节点 + Watch 机制 | 强一致，公平性好 | QPS 较低，依赖 zk 集群 | 金融、电商下单等一致性要求高 | +| **数据库** | 唯一索引 / 行锁 | 简单易实现 | 性能差，容易死锁 | 小规模系统，临时使用 | +**1. Redis 实现** +- **原理**：利用 `SETNX key value EX expire`（或 Redisson 的 `lock`），保证只有一个客户端能成功写入，拿到锁。解锁时 `DEL key`，结合 Lua 脚本保证原子性。 +- **优化**：RedLock 算法（多节点加锁），避免单点 Redis 故障。 +✅ 优点： -### 主从同步了解吗？ +- 性能高，适合高并发场景 +- 实现简单，生态成熟（Redisson） -![img](https://imgkr.cn-bj.ufileos.com/77eb458c-df60-4f23-a1bd-d691358cdd5b.png) +❌ 缺点： -**主从复制**，是指将一台 Redis 服务器的数据，复制到其他的 Redis 服务器。前者称为 **主节点(master)**，后者称为 **从节点(slave)**。且数据的复制是 **单向** 的，只能由主节点到从节点。Redis 主从复制支持 **主从同步** 和 **从从同步** 两种，后者是 Redis 后续版本新增的功能，以减轻主节点的同步负担。 +- 需要考虑锁过期、自动续期（看门狗） +- 单节点 Redis 挂掉会有风险，主从异步可能导致锁丢失 -#### 主从复制主要的作用 +**2. Zookeeper 实现** -- **数据冗余：** 主从复制实现了数据的热备份，是持久化之外的一种数据冗余方式。 -- **故障恢复：** 当主节点出现问题时，可以由从节点提供服务，实现快速的故障恢复 *(实际上是一种服务的冗余)*。 -- **负载均衡：** 在主从复制的基础上，配合读写分离，可以由主节点提供写服务，由从节点提供读服务 *（即写 Redis 数据时应用连接主节点，读 Redis 数据时应用连接从节点）*，分担服务器负载。尤其是在写少读多的场景下，通过多个从节点分担读负载，可以大大提高 Redis 服务器的并发量。 -- **高可用基石：** 除了上述作用以外，主从复制还是哨兵和集群能够实施的 **基础**，因此说主从复制是 Redis 高可用的基础。 +- **原理**：基于 **临时顺序节点 + Watch 机制** + - 客户端在 `/lock` 下创建 **临时顺序节点** + - 序号最小的客户端获得锁 + - 其他客户端监听前一个节点，前一个节点释放时被唤醒 +- **公平锁**：严格 FIFO 顺序 -#### 实现原理 +✅ 优点： -![img](https://imgkr.cn-bj.ufileos.com/441d3f2d-a4b8-484d-99f2-d0dcc683447d.png) +- 天然保证锁的可靠性（会话断开，临时节点自动删除） +- 公平性好（按顺序排队） -为了节省篇幅，我把主要的步骤都 **浓缩** 在了上图中，其实也可以 **简化成三个阶段：准备阶段-数据同步阶段-命令传播阶段**。 +❌ 缺点： -> 更多细节 **推荐阅读** 之前的系列文章，不仅有原理讲解，还有实战环节： -> -> - Redis(9)——史上最强【集群】入门实践教程 - https://www.wmyskxz.com/2020/03/17/redis-9-shi-shang-zui-qiang-ji-qun-ru-men-shi-jian-jiao-cheng/ +- Zookeeper 是 CP 模型，性能不如 Redis（QPS 万级 vs Redis 十万级以上） +- 依赖 ZooKeeper 集群，运维成本高 +**3. 数据库实现** +- **方式 1**：基于唯一索引，比如 `insert into lock_table (lock_key) values ('xxx')`，失败说明已被占用 +- **方式 2**：基于 `select ... for update` 行锁 -### 哨兵模式了解吗？ +✅ 优点： -![img](https://imgkr.cn-bj.ufileos.com/2b916e2b-5ee2-4287-b981-b6d900bc6df5.png) +- 易于理解，直接利用数据库 +- 无需额外组件 -*上图* 展示了一个典型的哨兵架构图，它由两部分组成，哨兵节点和数据节点： +❌ 缺点： -- **哨兵节点：** 哨兵系统由一个或多个哨兵节点组成，哨兵节点是特殊的 Redis 节点，不存储数据； -- **数据节点：** 主节点和从节点都是数据节点； +- 性能差，不适合高并发 +- 锁超时、死锁处理复杂 -在复制的基础上，哨兵实现了 **自动化的故障恢复** 功能，下方是官方对于哨兵功能的描述： -- **监控（Monitoring）：** 哨兵会不断地检查主节点和从节点是否运作正常。 -- **自动故障转移（Automatic failover）：** 当 **主节点** 不能正常工作时，哨兵会开始 **自动故障转移操作**，它会将失效主节点的其中一个 **从节点升级为新的主节点**，并让其他从节点改为复制新的主节点。 -- **配置提供者（Configuration provider）：** 客户端在初始化时，通过连接哨兵来获得当前 Redis 服务的主节点地址。 -- **通知（Notification）：** 哨兵可以将故障转移的结果发送给客户端。 -其中，监控和自动故障转移功能，使得哨兵可以及时发现主节点故障并完成转移。而配置提供者和通知功能，则需要在与客户端的交互中才能体现。 +### 🎯 分布式Redis是前期做还是后期规模上来了再做好？为什么？ -#### 新的主服务器是怎样被挑选出来的？ +既然Redis是如此的轻量（单实例只使用1M内存），为防止以后的扩容，最好的办法就是一开始就启动较多实例。即便你只有一台服务器，你也可以一开始就让Redis以分布式的方式运行，使用分区，在同一台服务器上启动多个实例。 -**故障转移操作的第一步** 要做的就是在已下线主服务器属下的所有从服务器中，挑选出一个状态良好、数据完整的从服务器，然后向这个从服务器发送 `slaveof no one` 命令，将这个从服务器转换为主服务器。但是这个从服务器是怎么样被挑选出来的呢？ +一开始就多设置几个Redis实例，例如32或者64个实例，对大多数用户来说这操作起来可能比较麻烦，但是从长久来看做这点牺牲是值得的。 +这样的话，当你的数据不断增长，需要更多的Redis服务器时，你需要做的就是仅仅将Redis实例从一台服务迁移到另外一台服务器而已（而不用考虑重新分区的问题）。一旦你添加了另一台服务器，你需要将你一半的Redis实例从第一台机器迁移到第二台机器。 -![img](https://imgkr.cn-bj.ufileos.com/54e5497b-952b-4559-a503-4c08c78a46a5.png) +### 🎯 Redis分布式锁有什么问题 怎么解决？ +> Redis 分布式锁虽然简单高效，但有几个问题： +> +> 1. 锁可能因为 TTL 到期而被误释放； +> 2. 锁不可重入； +> 3. 主从延迟可能导致锁丢失； +> 4. 锁过期时间不好控制。 +> +> 常见的优化方法有： +> +> - 使用唯一标识 + Lua 脚本保证安全释放； +> - 看门狗机制自动续期，解决超时问题； +> - 支持可重入锁； +> - 多节点加锁（RedLock）提升容错性； +> - 如果对一致性要求极高，可以用 ZooKeeper 或 etcd。 +> +> 综上，Redis 分布式锁适合高性能但对一致性容忍度较高的业务，而金融级别的强一致性分布式锁更建议用 ZooKeeper。 + +| **问题** | **说明** | **解决思路** | +| ---------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | +| 锁过期导致误释放 | A 线程锁还没执行完，锁 TTL 到期释放，B 获取锁 → A 执行完释放时误删了 B 的锁 | 使用 **唯一标识（UUID/线程ID）+ Lua 脚本**，保证只删除自己的锁 | +| 不可重入 | 同一线程重复获取锁时会死锁 | 使用 **可重入锁**，在 value 中存储线程 ID 和重入计数 | +| 锁超时不可控 | 业务执行时间可能比 TTL 长，导致锁过期 | **看门狗机制**（定时续期），如 Redisson 默认 30s，10s 自动续期 | +| 主从复制延迟 | 主节点加锁后宕机，数据还没同步到从节点 → 锁丢失 | **RedLock 算法**：在多个独立 Redis 节点加锁，超过半数成功算成功 | +| 单点问题 | 单 Redis 节点宕机 → 锁全丢失 | Redis **集群部署** 或 使用 **ZooKeeper/etcd** 这类 CP 系统保证一致性 | + + + +### 🎯 Redis分布式锁，过期时间怎么定的? + +**锁过期时间的设定** + +1. **设定原则** + - **经验值法**：基于业务逻辑的平均耗时设定，推荐 `TTL = 平均耗时 × 2~3`（如平均耗时 5s → TTL=15s）。 + - **动态调整**：结合历史数据监控动态调整（如 Prometheus 统计 P99 耗时）。 + - **兜底策略**：必须设置过期时间，避免死锁（即使业务崩溃，锁也能自动释放）。 + +2. **极端场景优化** + + - **自动续期（Watchdog）**： 客户端启动后台线程，定期（如 `TTL/3`）重置锁过期时间。 **示例**：Redisson 的 `lockWatchdogTimeout` 默认 30s，每 10s 续期。 + + > **续期机制实现** + > + > 1. **开启一个定时任务**：在获取锁后，开启一个定时任务，每隔一定时间（如过期时间的一半）延长锁的过期时间。 + > + > 2. **判断锁的持有者**：在续期时，确保当前续期操作仍然是由锁的持有者执行，以防止锁误续期。 + > + > ```java + > public class RedisLockWithRenewal { + > + > private static final String LOCK_KEY = "my_lock"; + > private static final String LOCK_VALUE = "unique_value"; + > private static final int EXPIRE_TIME = 7; // 过期时间为7秒 + > + > public boolean acquireLock(Jedis jedis) { + > String result = jedis.set(LOCK_KEY, LOCK_VALUE, "NX", "EX", EXPIRE_TIME); + > return "OK".equals(result); + > } + > + > public void releaseLock(Jedis jedis) { + > if (LOCK_VALUE.equals(jedis.get(LOCK_KEY))) { + > jedis.del(LOCK_KEY); + > } + > } + > + > public void renewLock(Jedis jedis) { + > Timer timer = new Timer(); + > timer.schedule(new TimerTask() { + > @Override + > public void run() { + > if (LOCK_VALUE.equals(jedis.get(LOCK_KEY))) { + > jedis.expire(LOCK_KEY, EXPIRE_TIME); + > System.out.println("Lock renewed."); + > } else { + > timer.cancel(); + > } + > } + > }, EXPIRE_TIME * 500, EXPIRE_TIME * 500); // 每 3.5 秒续期一次 + > } + > + > public static void main(String[] args) { + > Jedis jedis = new Jedis("localhost", 6379); + > RedisLockWithRenewal lock = new RedisLockWithRenewal(); + > + > if (lock.acquireLock(jedis)) { + > lock.renewLock(jedis); + > try { + > // 业务逻辑 + > System.out.println("Lock acquired, performing business logic..."); + > Thread.sleep(15000); // 模拟长时间业务执行 + > } catch (InterruptedException e) { + > e.printStackTrace(); + > } finally { + > lock.releaseLock(jedis); + > System.out.println("Lock released."); + > } + > } else { + > System.out.println("Failed to acquire lock."); + > } + > } + > } + > + > ``` + > + > **注意事项** + > + > - **锁的唯一性**：确保锁的值是唯一的，可以使用 UUID 或业务唯一标识符。 + > - **原子性操作**：使用 Redis 的 Lua 脚本保证锁的获取和续期操作的原子性，以防止竞态条件。 + + - **Fencing Token**： 锁服务返回单调递增 Token，业务操作时校验 Token 的时效性（如 ZooKeeper 的 zxid）。 + + + +### 🎯 如果一个业务执行时间比较长，锁过期了怎么办? + +1. **锁自动续期机制** + + ```java + // Redisson 自动续期示例 + RLock lock = redisson.getLock("order_lock"); + lock.lock(); // 默认启动看门狗线程自动续期 + try { + // 长耗时业务（如 60s 的订单处理） + } finally { + lock.unlock(); + } + ``` + + - **优势**：无需手动管理 TTL，锁在业务完成前持续有效。 + + - **限制**：需确保客户端进程存活（若客户端宕机，看门狗线程终止，锁仍会超时释放）。 + +2. **分段锁与锁降级** + + - 分段锁：将大任务拆分为多个子任务，每个子任务单独加锁。 + + ```Java + for (int i = 0; i < segments; i++) { + RLock segmentLock = redisson.getLock("order_lock_" + i); + segmentLock.lock(); + try { + // 处理子任务 + } finally { + segmentLock.unlock(); + } + } + ``` + + - 锁降级：写锁释放前获取读锁，避免锁完全过期后数据不一致。 + + ```Java + RReadWriteLock rwLock = redisson.getReadWriteLock("data_lock"); + rwLock.writeLock().lock(); + try { + // 写操作 + rwLock.readLock().lock(); // 降级为读锁 + } finally { + rwLock.writeLock().unlock(); + } + // 后续继续持有读锁 + ``` + +3. **业务超时熔断**：监控业务耗时：若业务执行超过阈值（如 TTL 的 80%），触发熔断并回滚。 + + ```Java + try { + Future future = executor.submit(() -> processOrder()); + future.get(ttl * 0.8, TimeUnit.MILLISECONDS); // 设置超时等待 + } catch (TimeoutException e) { + future.cancel(true); // 中断业务线程 + rollback(); // 事务回滚 + } + ``` + + + +### 🎯 自动续期怎么做? + +“分布式锁的自动续期，核心是通过‘**短租约 + 心跳续期**’解决‘锁提前过期’和‘锁无限占用’的问题，具体实现分‘手写方案’和‘Redisson 开箱即用方案’： + +**1. 手写方案的核心逻辑** + +- **初始加锁**：用 `SET key token NX PX=leaseMs` 原子加锁（`token` 是 UUID，防锁漂移；`leaseMs` 是短租约，如 30s）； + +- **心跳续期**：加锁成功后，启动一个定时线程（用 `ScheduledExecutorService`），每 `leaseMs/3` 时间（如 10s）执行一次 `Lua` 脚本 —— 先校验锁的 `token` 是否为当前线程所有，若是则调用 `PEXPIRE` 刷新租约到 `leaseMs`； -简单来说 Sentinel 使用以下规则来选择新的主服务器： + > 检测 Token 所有权是 ** 防止 “锁漂移”** 的关键。它确保了只有获取锁的线程才能续期或释放锁。 + > + > **核心原理：** + > + > 1. **存储 Token**：当线程获取锁时，生成一个唯一的 Token（通常是 UUID），并将这个 Token 作为值存入 Redis 的 Key 中（`SET key token ...`）。 + > 2. **传递 Token**：这个 Token 必须与获取锁的线程**绑定**。在 Java 中，最佳实践是使用 `ThreadLocal` 来存储，这样可以确保每个线程只能看到自己持有的 Token。 + > 3. **原子性校验与操作**：在执行续期或释放锁操作时，必须使用 Lua 脚本，在 Redis 服务器端原子地完成 “**获取值 -> 比较 Token -> 执行操作（续期 / 释放）**” 这一系列步骤。 -1. 在失效主服务器属下的从服务器当中， 那些被标记为主观下线、已断线、或者最后一次回复 PING 命令的时间大于五秒钟的从服务器都会被 **淘汰**。 -2. 在失效主服务器属下的从服务器当中， 那些与失效主服务器连接断开的时长超过 down-after 选项指定的时长十倍的从服务器都会被 **淘汰**。 -3. 在 **经历了以上两轮淘汰之后** 剩下来的从服务器中， 我们选出 **复制偏移量（replication offset）最大** 的那个 **从服务器** 作为新的主服务器；如果复制偏移量不可用，或者从服务器的复制偏移量相同，那么 **带有最小运行 ID** 的那个从服务器成为新的主服务器。 +- **安全兜底**： -> 更多细节 **推荐阅读** 之前的系列文章，不仅有原理讲解，还有实战环节： -> -> - Redis(9)——史上最强【集群】入门实践教程 - https://www.wmyskxz.com/2020/03/17/redis-9-shi-shang-zui-qiang-ji-qun-ru-men-shi-jian-jiao-cheng/ + 1. 加 “最大租期”（如 5 分钟），超过后强制停止续期，避免无限占用； + 2. 续期失败（`Lua` 返回 0）或业务结束时，立即停止续期线程，再用 `Lua` 脚本（先校验 `token` 再 `DEL`）释放锁； +- **性能优化**：续期间隔加随机抖动（如 ±500ms），避免 Redis 瞬时高负载。 +**2. Redisson 的看门狗方案（生产首选）** -### Redis 集群使用过吗？原理？ +Redisson 自带的看门狗本质和手写逻辑一致，但封装更完善： -Redis Sentinal着眼于高可用，在master宕机时会自动将slave提升为master，继续提供服务。 +- 默认 30s 租约，每 10s 自动续期，只需调用 `lock.lock()` 即可开启； +- 若业务是短任务，显式设置 `leaseTime`（如 `lock.lock(5, TimeUnit.SECONDS)`），可关闭续期，锁到期自动释放； +- 释放锁时无需手动停止续期，`unlock()` 会自动终止看门狗，且自带 “当前线程持有锁校验”，避免误删。 -Redis Cluster着眼于扩展性，在单个redis内存不足时，使用Cluster进行分片存储。 +**3. 核心安全点** -(1)所有的redis节点彼此互联(PING-PONG机制),内部使用二进制协议优化传输速度和带宽. +无论哪种方案，都要保证两点：一是续期和释放锁必须用 `Lua` 脚本保证原子性；二是必须校验 “只有锁持有者才能续期 / 释放”，通过 `token` 避免锁被其他线程误操作。” -(2)节点的fail是通过集群中超过半数的节点检测失效时才生效. +自动续期就是“短租约 + 心跳续期”。加锁用短 TTL（如10–30s），后台起一个心跳线程每 TTL/3 刷新一次过期时间，直到任务完成或超过最大租期就停止。续期必须校验“只有锁持有者才能续”，用 token + Lua 原子校验。再配合最大租期与栅栏令牌，防“锁过期后旧持有者继续写”。 -(3)客户端与redis节点直连,不需要中间proxy层.客户端不需要连接集群所有节点,连接集群中任何一个可用节点即可 +**关键实现（简化）** -(4)redis-cluster把所有的物理节点映射到[0-16383]slot上,cluster 负责维护node<->slot<->value +- 获取锁 -Redis 集群中内置了 16384 个哈希槽，当需要在 Redis 集群中放置一个 key-value 时，redis 先对 key 使用 crc16 算法算出一个结果，然后把结果对 16384 求余数，这样每个 key 都会对应一个编号在 0-16383 之间的哈希槽，redis 会根据节点数量大致均等的将哈希槽映射到不同的节点 + ```java + String token = UUID.randomUUID().toString(); + boolean ok = redis.set(key, token, NX, PX, leaseMs); + if (!ok) return false; + startRenewal(key, token, leaseMs, maxLeaseMs); + ``` -![img](https://imgkr.cn-bj.ufileos.com/171b8bfb-78df-4f9e-945b-acc09e102bd4.png) +- 续期线程（建议用 ScheduledExecutorService，间隔 = leaseMs/3，加入随机抖动） -*上图* 展示了 **Redis Cluster** 典型的架构图，集群中的每一个 Redis 节点都 **互相两两相连**，客户端任意 **直连** 到集群中的 **任意一台**，就可以对其他 Redis 节点进行 **读写** 的操作。 + ```java + void startRenewal(String key, String token, long leaseMs, long maxLeaseMs) { + long begin = System.currentTimeMillis(); + ScheduledFuture future = scheduler.scheduleAtFixedRate(() -> { + if (System.currentTimeMillis() - begin > maxLeaseMs) { cancel(); return; } + try { + Long updated = (Long) jedis.eval( + "if redis.call('GET', KEYS[1]) == ARGV[1] then " + + " return redis.call('PEXPIRE', KEYS[1], ARGV[2]) " + + "else return 0 end", + Collections.singletonList(key), + Arrays.asList(token, String.valueOf(leaseMs)) + ); + if (updated == 0L) { cancel(); } // 不再持有锁 + } catch (Exception e) { /* 记录并重试 */ } + }, leaseMs/3, leaseMs/3, TimeUnit.MILLISECONDS); + // 保存 future, 在业务完成或异常时 cancel + } + ``` -#### 基本原理 +- 释放锁（只允许持有者释放） -![img](https://imgkr.cn-bj.ufileos.com/4d1c2ab5-8357-48e6-a948-7dd998c61621.png) + ```lua + -- KEYS[1]=lockKey, ARGV[1]=token + if redis.call('GET', KEYS[1]) == ARGV[1] then + return redis.call('DEL', KEYS[1]) + else + return 0 + end + ``` -Redis 集群中内置了 `16384` 个哈希槽。当客户端连接到 Redis 集群之后，会同时得到一份关于这个 **集群的配置信息**，当客户端具体对某一个 `key` 值进行操作时，会计算出它的一个 Hash 值，然后把结果对 `16384` **求余数**，这样每个 `key` 都会对应一个编号在 `0-16383` 之间的哈希槽，Redis 会根据节点数量 **大致均等** 的将哈希槽映射到不同的节点。 +**Redisson怎么配（现成的看门狗）** -再结合集群的配置信息就能够知道这个 `key` 值应该存储在哪一个具体的 Redis 节点中，如果不属于自己管，那么就会使用一个特殊的 `MOVED` 命令来进行一个跳转，告诉客户端去连接这个节点以获取数据： +- Redisson自带自动续期（watchdog，默认30s，续期每10s） -```bash -GET x --MOVED 3999 127.0.0.1:6381 -``` +- 配置要点：lockWatchdogTimeout=30_000，短任务可直接设置 leaseTime 关闭看门狗；长任务走看门狗更稳。 -`MOVED` 指令第一个参数 `3999` 是 `key` 对应的槽位编号，后面是目标节点地址，`MOVED` 命令前面有一个减号，表示这是一个错误的消息。客户端在收到 `MOVED` 指令后，就立即纠正本地的 **槽位映射表**，那么下一次再访问 `key` 时就能够到正确的地方去获取了。 +- 思路与上面一致：拿到锁→看门狗续期→业务完成后 unlock。 -#### 集群的主要作用 -1. **数据分区：** 数据分区 *(或称数据分片)* 是集群最核心的功能。集群将数据分散到多个节点，**一方面** 突破了 Redis 单机内存大小的限制，**存储容量大大增加**；**另一方面** 每个主节点都可以对外提供读服务和写服务，**大大提高了集群的响应能力**。Redis 单机内存大小受限问题，在介绍持久化和主从复制时都有提及，例如，如果单机内存太大，`bgsave` 和 `bgrewriteaof` 的 `fork` 操作可能导致主进程阻塞，主从环境下主机切换时可能导致从节点长时间无法提供服务，全量复制阶段主节点的复制缓冲区可能溢出…… -2. **高可用：** 集群支持主从复制和主节点的 **自动故障转移** *（与哨兵类似）*，当任一节点发生故障时，集群仍然可以对外提供服务。 -### 集群中数据如何分区？| 分布式寻址都有哪些算法？ +### 🎯 怎么保证释放锁的一个原子性？ -- hash 算法（大量缓存重建） +1. **Lua 脚本实现原子操作** -- 一致性 hash 算法（自动缓存迁移）+ 虚拟节点（自动负载均衡） + ```lua + -- 解锁脚本：校验 Value 匹配后删除 Key + if redis.call("GET", KEYS[1]) == ARGV[1] then + return redis.call("DEL", KEYS[1]) + else + return 0 + end + ``` -- redis cluster 的 hash slot 算法 + - 优势： + - 避免非原子操作（先 `GET` 后 `DEL`）导致误删其他客户端的锁。 + - Redis 单线程执行 Lua 脚本，天然原子性。 -Redis 采用方案三。 +2. **误删锁的防御** -#### 方案一：哈希值 % 节点数 + - 唯一 Value 设计：使用 UUID + 线程ID 作为 Value，确保锁归属可验证。 -哈希取余分区思路非常简单：计算 `key` 的 hash 值，然后对节点数量进行取余，从而决定数据映射到哪个节点上。 + ```Java + String lockValue = UUID.randomUUID() + ":" + Thread.currentThread().getId(); + ``` -不过该方案最大的问题是，**当新增或删减节点时**，节点数量发生变化，系统中所有的数据都需要 **重新计算映射关系**，引发大规模数据迁移。 -#### 方案二：一致性哈希分区 -一致性哈希算法将 **整个哈希值空间** 组织成一个虚拟的圆环，范围是 *[0 - 232 - 1]*，对于每一个数据，根据 `key` 计算 hash 值，确数据在环上的位置，然后从此位置沿顺时针行走，找到的第一台服务器就是其应该映射到的服务器： +### 🎯 用Redis实现分布式锁，主从切换导致锁失效，如何解决？ -![img](https://imgkr.cn-bj.ufileos.com/042c2199-ed4e-4dd6-b4ef-60b55d104c0a.png) +**核心原因**： Redis 主从复制是**异步**的，主节点宕机时可能未将锁信息同步到从节点，导致新主节点丢失锁，引发并发风险。 -与哈希取余分区相比，一致性哈希分区将 **增减节点的影响限制在相邻节点**。以上图为例，如果在 `node1` 和 `node2` 之间增加 `node5`，则只有 `node2` 中的一部分数据会迁移到 `node5`；如果去掉 `node2`，则原 `node2` 中的数据只会迁移到 `node4` 中，只有 `node4` 会受影响。 +**解决方案** -一致性哈希分区的主要问题在于，当 **节点数量较少** 时，增加或删减节点，**对单个节点的影响可能很大**，造成数据的严重不平衡。还是以上图为例，如果去掉 `node2`，`node4` 中的数据由总数据的 `1/4` 左右变为 `1/2` 左右，与其他节点相比负载过高。 +1. **Redlock 算法（多节点多数派）** + - **原理**：向多个独立 Redis 节点同时加锁，只有 **半数以上节点成功** 且总耗时 < TTL 才算加锁成功。 + + - **优点**：容错性强，避免单点故障。 + + - **缺点**：部署复杂，性能开销大。 + + - **适用场景**：对一致性要求高的金融级场景。 + +2. **Redisson 看门狗机制（自动续期）** -#### 方案三：带有虚拟节点的一致性哈希分区 + - **原理**：加锁后启动后台线程定期刷新锁过期时间，避免业务执行期间锁失效。 -该方案在 **一致性哈希分区的基础上**，引入了 **虚拟节点** 的概念。Redis 集群使用的便是该方案，其中的虚拟节点称为 **槽（slot）**。槽是介于数据和实际节点之间的虚拟概念，每个实际节点包含一定数量的槽，每个槽包含哈希值在一定范围内的数据。 + - **优点**：简化开发，支持自动续期和可重入。 -在使用了槽的一致性哈希分区中，**槽是数据管理和迁移的基本单位**。槽 **解耦** 了 **数据和实际节点** 之间的关系，增加或删除节点对系统的影响很小。仍以上图为例，系统中有 `4` 个实际节点，假设为其分配 `16` 个槽(0-15)； + - **缺点**：无法完全避免主从切换问题。 -- 槽 0-3 位于 node1；4-7 位于 node2；以此类推…. + - **适用场景**：高并发、低延迟场景（如秒杀、订单系统）。 -如果此时删除 `node2`，只需要将槽 4-7 重新分配即可，例如槽 4-5 分配给 `node1`，槽 6 分配给 `node3`，槽 7 分配给 `node4`；可以看出删除 `node2` 后，数据在其他节点的分布仍然较为均衡。 +3. **配置优化（主从同步保障）** -### 节点之间的通信机制了解吗？ + - **原理**：设置 `min-slaves-to-write 1` 和 `min-slaves-max-lag 10`，确保主节点至少有一个从节点同步数据。 -集群元数据的维护有两种方式：集中式、Gossip 协议。redis cluster 节点间采用 gossip 协议进行通信。 + - **优点**：减少锁丢失风险。 -集群的建立离不开节点之间的通信，例如我们在 *快速体验* 中刚启动六个集群节点之后通过 `redis-cli` 命令帮助我们搭建起来了集群，实际上背后每个集群之间的两两连接是通过了 `CLUSTER MEET ` 命令发送 `MEET` 消息完成的，下面我们展开详细说说。 + - **缺点**：性能下降，需合理配置参数。 -#### 两个端口 + - **适用场景**：单节点部署，对一致性要求中等的场景。 -在 **哨兵系统** 中，节点分为 **数据节点** 和 **哨兵节点**：前者存储数据，后者实现额外的控制功能。在 **集群** 中，没有数据节点与非数据节点之分：**所有的节点都存储数据，也都参与集群状态的维护**。为此，集群中的每个节点，都提供了两个 TCP 端口： +4. **降级 Zookeeper 分布式锁（强一致）** -- **普通端口：** 即我们在前面指定的端口 *(7000等)*。普通端口主要用于为客户端提供服务 *（与单机节点类似）*；但在节点间数据迁移时也会使用。 -- **集群端口：** 端口号是普通端口 + 10000 *（10000是固定值，无法改变）*，如 `7000` 节点的集群端口为 `17000`。**集群端口只用于节点之间的通信**，如搭建集群、增减节点、故障转移等操作时节点间的通信；不要使用客户端连接集群接口。为了保证集群可以正常工作，在配置防火墙时，要同时开启普通端口和集群端口。 + - **原理**：基于 Zookeeper 的临时顺序节点和 Watcher 机制实现分布式锁。 -#### Gossip 协议 + - **优点**：强一致性，锁自动释放。 -节点间通信，按照通信协议可以分为几种类型：单对单、广播、Gossip 协议等。重点是广播和 Gossip 的对比。 + - **缺点**：性能低，运维复杂。 -- 广播是指向集群内所有节点发送消息。**优点** 是集群的收敛速度快(集群收敛是指集群内所有节点获得的集群信息是一致的)，**缺点** 是每条消息都要发送给所有节点，CPU、带宽等消耗较大。 -- Gossip 协议的特点是：在节点数量有限的网络中，**每个节点都 “随机” 的与部分节点通信** *（并不是真正的随机，而是根据特定的规则选择通信的节点）\*，经过一番杂乱无章的通信，每个节点的状态很快会达到一致。Gossip 协议的 **优点** 有负载 \*(比广播)* 低、去中心化、容错性高 *(因为通信有冗余)* 等；**缺点** 主要是集群的收敛速度慢。 + - **适用场景**：金融交易、支付等强一致要求的场景。 -#### 消息类型 + “根据业务一致性要求和性能需求，选择 Redlock 保障容错，Redisson 简化实现，或降级 Zookeeper 强一致方案。” -集群中的节点采用 **固定频率（每秒10次）** 的 **定时任务** 进行通信相关的工作：判断是否需要发送消息及消息类型、确定接收节点、发送消息等。如果集群状态发生了变化，如增减节点、槽状态变更，通过节点间的通信，所有节点会很快得知整个集群的状态，使集群收敛。 -节点间发送的消息主要分为 `5` 种：`meet 消息`、`ping 消息`、`pong 消息`、`fail 消息`、`publish 消息`。不同的消息类型，通信协议、发送的频率和时机、接收节点的选择等是不同的： -- **MEET 消息：** 在节点握手阶段，当节点收到客户端的 `CLUSTER MEET` 命令时，会向新加入的节点发送 `MEET` 消息，请求新节点加入到当前集群；新节点收到 MEET 消息后会回复一个 `PONG` 消息。 -- **PING 消息：** 集群里每个节点每秒钟会选择部分节点发送 `PING` 消息，接收者收到消息后会回复一个 `PONG` 消息。**PING 消息的内容是自身节点和部分其他节点的状态信息**，作用是彼此交换信息，以及检测节点是否在线。`PING` 消息使用 Gossip 协议发送，接收节点的选择兼顾了收敛速度和带宽成本，**具体规则如下**：(1)随机找 5 个节点，在其中选择最久没有通信的 1 个节点；(2)扫描节点列表，选择最近一次收到 `PONG` 消息时间大于 `cluster_node_timeout / 2` 的所有节点，防止这些节点长时间未更新。 -- **PONG消息：** `PONG` 消息封装了自身状态数据。可以分为两种：**第一种** 是在接到 `MEET/PING` 消息后回复的 `PONG` 消息；**第二种** 是指节点向集群广播 `PONG` 消息，这样其他节点可以获知该节点的最新信息，例如故障恢复后新的主节点会广播 `PONG` 消息。 -- **FAIL 消息：** 当一个主节点判断另一个主节点进入 `FAIL` 状态时，会向集群广播这一 `FAIL` 消息；接收节点会将这一 `FAIL` 消息保存起来，便于后续的判断。 -- **PUBLISH 消息：** 节点收到 `PUBLISH` 命令后，会先执行该命令，然后向集群广播这一消息，接收节点也会执行该 `PUBLISH` 命令。 +### 🎯 讲讲RedLock算法 ? -### 集群数据如何存储的有了解吗？ +> RedLock 是 Redis 作者提出的一种分布式锁算法，它通过在多个 Redis 节点上并行加锁，并且只要超过半数节点成功，就认为获取到锁。这样可以在部分节点宕机的情况下仍然保证锁的可用性。 +> +> 但 RedLock 的争议也很大，因为它依赖时钟和网络假设，在极端情况下仍可能出现锁的安全性问题。所以在生产中，如果对一致性要求非常高，大家一般会选用 ZooKeeper 或 etcd 的分布式锁，而 Redis 锁更适合对性能敏感、允许弱一致性的场景。 + +- **概念**：RedLock 是 Redis 的作者 Salvatore Sanfilippo（antirez）提出的一种分布式锁算法，用于在分布式环境中实现安全可靠的锁定机制，解决了单个 Redis 实例锁在分布式系统中存在的可靠性问题。 +- 算法原理： + - **多节点独立**：假设有 N 个完全独立的 Redis master 节点，这些节点之间不存在主从复制或其他集群协调机制，确保在最苛刻的环境下也能正常工作。 + - **获取时间戳**：客户端首先获取当前时间戳，单位为毫秒，作为整个操作的起始时间。 + - **逐个请求锁**：客户端轮流使用相同的 key 和具有唯一性的 value（如 UUID）在 N 个 Redis 节点上请求锁，每个请求都设置一个远小于锁释放时间的超时时间，避免在某个宕掉的节点上阻塞过长时间。 + - **判断锁获取情况**：客户端获取所有节点锁后，计算获取锁的总时间，即当前时间减去起始时间。当且仅当从大多数（N/2 + 1 个）Redis 节点都取到锁，并且总时间小于锁的失效时间时，才认为获取锁成功。 + - **计算有效时间**：如果获取锁成功，锁的真正有效时间是设置的锁总超时时间（TTL）减去获取锁的总时间，再减去时钟漂移时间（通常可忽略不计）。 + - **释放锁**：如果锁获取失败，无论原因是获取成功的锁数量不足还是总消耗时间超过锁释放时间，客户端都会到每个 master 节点上释放锁，包括那些没有获取锁成功的节点。 +- 优点： + - **高可靠性**：通过多个独立节点来获取锁，即使部分节点出现故障，只要大多数节点正常工作，就能保证锁的安全性和可用性，有效防止单点故障。 + - **满足分布式锁特性**：能满足互斥性、防死锁、持锁人解锁等分布式锁的基本特性，适用于各种需要分布式锁的场景。 +- 缺点： + - **对时钟同步要求高**：所有 Redis 服务器的时钟必须同步，否则可能由于时钟漂移导致锁的有效时间计算错误，从而引发问题。 + - **性能开销**：需要与多个 Redis 节点进行通信来获取和释放锁，相比单节点分布式锁，性能上有一定的开销。 + - **网络分区问题**：在网络分区的情况下，可能会出现不同分区的节点对锁的状态判断不一致，导致锁的安全性无法保证。 + +Redlock 算法的注意事项： + +- **多数节点**：客户端必须至少在大多数 Redis 实例上成功获取锁，才能认为获得了分布式锁。 +- **时钟同步**：所有 Redis 服务器的时钟必须同步，以避免由于时钟漂移导致的问题。 +- **网络分区**：在网络分区的情况下，Redlock 算法可能无法保证锁的安全。 +- **锁超时**：客户端必须设置合理的锁超时时间，以避免死锁。 +- **重试机制**：客户端需要实现重试机制，并在重试时等待随机时间，以避免多个客户端同时重试导致的竞争条件。 -节点需要专门的数据结构来存储集群的状态。所谓集群的状态，是一个比较大的概念，包括：集群是否处于上线状态、集群中有哪些节点、节点是否可达、节点的主从状态、槽的分布…… +------ -节点为了存储集群状态而提供的数据结构中，最关键的是 `clusterNode` 和 `clusterState` 结构：前者记录了一个节点的状态，后者记录了集群作为一个整体的状态。 -#### clusterNode 结构 -`clusterNode` 结构保存了 **一个节点的当前状态**，包括创建时间、节点 id、ip 和端口号等。每个节点都会用一个 `clusterNode` 结构记录自己的状态，并为集群内所有其他节点都创建一个 `clusterNode` 结构来记录节点状态。 +## 八、消息队列与异步处理 -下面列举了 `clusterNode` 的部分字段，并说明了字段的含义和作用： +### 🎯 什么是 Redis 消息队列？有哪些实现方式？ -```c -typedef struct clusterNode { - //节点创建时间 - mstime_t ctime; - //节点id - char name[REDIS_CLUSTER_NAMELEN]; - //节点的ip和端口号 - char ip[REDIS_IP_STR_LEN]; - int port; - //节点标识：整型，每个bit都代表了不同状态，如节点的主从状态、是否在线、是否在握手等 - int flags; - //配置纪元：故障转移时起作用，类似于哨兵的配置纪元 - uint64_t configEpoch; - //槽在该节点中的分布：占用16384/8个字节，16384个比特；每个比特对应一个槽：比特值为1，则该比特对应的槽在节点中；比特值为0，则该比特对应的槽不在节点中 - unsigned char slots[16384/8]; - //节点中槽的数量 - int numslots; - ………… -} clusterNode; -``` +Redis 消息队列利用 Redis 的数据结构（如 list、stream）实现生产者-消费者模型。实现方式包括： -除了上述字段，`clusterNode` 还包含节点连接、主从复制、故障发现和转移需要的信息等。 +- 基于 `list` 的 `LPUSH` 和 `RPOP`（简单队列）。 +- 基于 `pub/sub` 的发布订阅模式（实时通知）。 +- 基于 `stream` 的消息流功能（高效、可靠的队列）。 -#### clusterState 结构 +### 🎯 Redis 的 `pub/sub` 机制的原理是什么？优缺点是什么？ -`clusterState` 结构保存了在当前节点视角下，集群所处的状态。主要字段包括： +原理：`pub/sub` 是一种广播机制，发布者将消息发送到指定的频道，订阅者接收频道中的消息。 -```c -typedef struct clusterState { - //自身节点 - clusterNode *myself; - //配置纪元 - uint64_t currentEpoch; - //集群状态：在线还是下线 - int state; - //集群中至少包含一个槽的节点数量 - int size; - //哈希表，节点名称->clusterNode节点指针 - dict *nodes; - //槽分布信息：数组的每个元素都是一个指向clusterNode结构的指针；如果槽还没有分配给任何节点，则为NULL - clusterNode *slots[16384]; - ………… -} clusterState; -``` +优点：实时性强，轻量级实现简单。 -除此之外，`clusterState` 还包括故障转移、槽迁移等需要的信息。 +缺点： ------- +- 不保证消息持久化。 +- 无法保证订阅者一定能收到消息（离线订阅无效）。 +- 不能实现复杂的消费分组需求。 +### 🎯 Redis Stream 是什么？与传统队列有什么区别？ +Redis Stream 是 Redis 5.0 引入的日志型数据结构，支持消费分组和持久化。 +优势： -## 六、Redis缓存异常问题 +- 消息持久化，保证消息可靠性。 +- 支持消费分组（类似 Kafka 的消费模型）。 +- 可记录消费偏移量，适用于复杂的消息队列场景。 +### 🎯 如何用 Redis 实现一个延时队列？ -> 我看你们有把 Redis 用作缓存，缓存和数据库数据一致性问题 +- 使用 zset（有序集合）： + - 将任务的执行时间作为分值 `score`，任务内容作为成员 `member`。 + - 定期扫描 `zset`，取出分值小于当前时间的任务执行。 -### 缓存和数据库数据一致性问题 +**示例伪代码：** -分布式环境下非常容易出现缓存和数据库间数据一致性问题，针对这一点，如果项目对缓存的要求是强一致性的，那么就不要使用缓存。我们只能采取合适的策略来降低缓存和数据库间数据不一致的概率，而无法保证两者间的强一致性。合适的策略包括合适的缓存更新策略，更新数据库后及时更新缓存、缓存失败时增加重试机制。 +```plaintext +ZADD delay_queue +ZREMRANGEBYSCORE delay_queue -inf -> 执行并删除到期任务 +``` -**18、如何保证缓存与数据库双写时的数据一致性？** +### 🎯 Redis 消息队列的瓶颈在哪？如何优化？ -你只要用缓存，就可能会涉及到缓存与数据库双存储双写，你只要是双写，就一定会有数据一致性的问题，那么你如何解决一致性问题？ +- 瓶颈： + - 单线程处理模型下，队列写入和读取的高并发可能导致性能瓶颈。 + - 数据量大时，内存消耗过高。 +- 优化： + - 使用 Redis Cluster 分片存储队列。 + - 调整内存策略或淘汰策略（如 `noeviction`）。 -（1）一般来说，就是如果你的系统不是严格要求缓存+数据库必须一致性的话，缓存可以稍微的跟数据库偶尔有不一致的情况，最好不要做这个方案，读请求和写请求串行化，串到一个内存队列里去，这样就可以保证一定不会出现不一致的情况 +### 🎯 使用Redis做过异步队列吗，是如何实现的？ -串行化之后，就会导致系统的吞吐量会大幅度的降低，用比正常情况下多几倍的机器去支撑线上的一个请求。 +使用 list 类型保存数据信息，rpush 生产消息，lpop 消费消息，当 lpop 没有消息时，可以 sleep 一段时间，然后再检查有没有信息，如果不想 sleep 的话，可以使用 blpop，在没有信息的时候，会一直阻塞，直到信息的到来。redis 可以通过 pub/sub 主题订阅模式实现一个生产者，多个消费者，当然也存在一定的缺点，当消费者下线时，生产的消息会丢失。 -（2）还有一种方式就是可能会暂时产生不一致的情况，但是发生的几率特别小，就是先更新数据库，然后再删除缓存。 +### 🎯 Redis如何实现延时队列 -### Redis雪崩 +使用 Redis 作为延时队列的实现方法有很多，其中一种常见的方式是使用 Redis 的有序集合（Sorted Set）。有序集合通过成员的分数进行排序，非常适合实现延时队列功能。 -> 你们 怎么解决？ +**实现步骤** -![img](https://imgkr.cn-bj.ufileos.com/dd5424c4-edf6-4597-9db8-9488151068d4.png) +1. **添加任务到延时队列**： 将任务添加到 Redis 有序集合中，使用任务的执行时间作为分数。执行时间可以使用 Unix 时间戳表示。 +2. **轮询检查和执行任务**： 使用一个定时任务（如每秒运行一次）来轮询检查有序集合中是否有需要执行的任务。当任务的执行时间小于等于当前时间时，执行该任务并将其从集合中移除。 -**1、缓存雪崩** +> 使用 sortedset，使用时间戳做 score，消息内容作为 key，调用 zadd 来生产消息，消费者使用 zrangbyscore 获取 n 秒之前的数据做轮询处理。 -缓存雪崩是指缓存同一时间大面积的失效，所以，后面的请求都会落到数据库上，造成数据库短时间内承受大量请求而崩掉。 +------ -**解决方案** +## 九、实战应用与最佳实践 -（1）缓存数据的过期时间设置随机，防止同一时间大量数据过期现象发生。 +### 🎯 Redis乐观锁的应用场景，举例说明? -（2）一般并发量不是特别多的时候，使用最多的解决方案是加锁排队。 +Redis 乐观锁是一种用于并发控制的机制，主要用于在高并发场景下确保数据一致性。与悲观锁不同，乐观锁假设大部分情况下数据竞争不会发生，因此不会像悲观锁那样阻塞其他操作。乐观锁通过检查数据的版本号或其他标识符，在更新数据时确保数据没有被其他事务修改。 -（3）给每一个缓存数据增加相应的缓存标记，记录缓存的是否失效，如果缓存标记失效，则更新数据缓存。 +Redis 的乐观锁可以通过事务中的 **`WATCH`** 命令实现、也可以基于类似 CAS（Compare and Swap） 的机制，把 GET、SET 放入 lua 脚本中执行。 +```lua +-- KEYS[1] = key 名称 +-- ARGV[1] = 期望的旧值 +-- ARGV[2] = 要设置的新值 +local current = redis.call('GET', KEYS[1]) +if current == ARGV[1] then + return redis.call('SET', KEYS[1], ARGV[2]) +else + return nil -- 表示失败 +end +``` -**2、缓存穿透** +比如**库存扣减（电商秒杀场景）**、**计数器更新（网站访问统计）**、**配置热更新（分布式服务配置同步）**等 -![img](https://imgkr.cn-bj.ufileos.com/9b0bcb3d-916d-4903-bac6-e32c5279a741.png) +**乐观锁应用场景** -缓存穿透是指缓存和数据库中都没有的数据，导致所有的请求都落到数据库上，造成数据库短时间内承受大量请求而崩掉。 +乐观锁在需要高并发、高性能和数据一致性的应用场景中非常有用。以下是几个典型的应用场景： -**解决方案** +1. **库存管理**： + - 在电商平台中，商品库存数量的更新就是一个典型的乐观锁应用场景。当用户下单时，系统首先读取库存数量，然后尝试减去相应的数量。这个过程可以通过`WATCH`命令和事务中的`MULTI`/`EXEC`命令来实现。如果库存数量在读取和更新之间被其他事务修改了，事务将失败，用户会被提示库存不足。 +2. **订单号生成**： + - 在需要生成唯一订单号的系统中，可以使用Redis的原子自增操作`INCR`或`INCRBY`。乐观锁假设在生成订单号的过程中不太可能出现冲突，即使出现，也可以通过重试机制解决。 +3. **秒杀活动**： + - 秒杀活动通常在极短的时间内有大量用户尝试购买同一商品。使用乐观锁，系统可以先检查库存数量，然后尝试更新。如果库存不足，可以快速返回失败信息，而不需要锁住库存资源。 +4. **分布式序列号生成**： + - 在分布式系统中生成全局唯一的序列号时，可以使用Redis的乐观锁特性。通过`INCR`命令，不同的服务实例可以并发地生成唯一的序列号，而不需要复杂的协调机制。 +5. **投票或点赞功能**： + - 在社交媒体应用中，用户的点赞操作可以通过Redis的乐观锁来实现。系统首先读取当前的点赞数，然后将其加一。如果在这个过程中点赞数被其他用户更新了，当前操作可以重试或忽略，因为点赞数的最终一致性通常比实时一致性更重要。 +6. **缓存数据的并发更新**： + - 当多个服务实例需要更新同一个缓存数据时，可以使用Redis乐观锁来避免数据不一致的问题。每个实例在更新前先读取当前版本号，然后尝试更新数据和版本号。如果版本号在更新过程中发生变化，说明有其他实例已经更新了数据，当前操作可以放弃或重试。 +7. **分布式锁**： + - 虽然Redis也常用于实现分布式锁，但在某些情况下，可以使用乐观锁的方式来实现一种更轻量级的分布式锁。例如，使用`SETNX`命令设置一个键，如果操作成功，则获得锁；如果失败，则表示锁被其他进程持有。 -（1）接口层增加校验，如用户鉴权校验，id做基础校验，id<=0的直接拦截； -（2）从缓存取不到的数据，在数据库中也没有取到，这时也可以将key-value对写为key-null，缓存有效时间可以设置短点，如30秒（设置太长会导致正常情况也没法使用）。这样可以防止攻击用户反复用同一个id暴力攻击 -（3）采用布隆过滤器，将所有可能存在的数据哈希到一个足够大的 bitmap 中，一个一定不存在的数据会被这个 bitmap 拦截掉，从而避免了对底层存储系统的查询压力。 +### 🎯 Redis 过期时间优化？如何确定过期时间？ +Redis 的过期时间优化主要考虑三个方面： +1. **避免雪崩**：过期时间要加随机因子，避免大批量 key 同时失效； +2. **分场景设定 TTL**：实时性要求高的数据过期时间短，稳定性高的数据过期时间长，热点数据甚至可以用逻辑过期来控制； +3. **更新策略**：部分核心数据可以不依赖 TTL，而是由业务逻辑来主动更新，保证数据可用性。 -**3、缓存击穿** +过期时间的确定没有固定标准，一般取决于**业务能容忍的数据延迟**和**数据库能承受的回源压力**。 -缓存击穿是指缓存中没有但数据库中有的数据（一般是缓存时间到期），这时由于并发用户特别多，同时读缓存没读到数据，又同时去数据库去取数据，引起数据库压力瞬间增大，造成过大压力。 -> **和缓存雪崩不同的是，缓存击穿指并发查同一条数据，缓存雪崩是不同数据都过期了，很多数据都查不到从而查数据库。** -**解决方案** +### 🎯 Redis怎么确认命中率? -（1）设置热点数据永远不过期。 +要确认 Redis 的缓存命中率，可以使用 Redis 提供的统计信息来进行分析。Redis 提供了一个命令 `INFO`，该命令会返回 Redis 服务器的各种统计和状态信息，其中包括与缓存命中率相关的两个关键指标：`keyspace_hits` 和 `keyspace_misses`。 -（2）加互斥锁，互斥锁 +使用 `INFO stats` 命令获取统计信息，可以看到这两个指标 -**4、缓存预热** +```sh +> info stats +# Stats +total_connections_received:6119693 +total_commands_processed:346700954 +instantaneous_ops_per_sec:84 +total_net_input_bytes:95242250343 +total_net_output_bytes:74348467113 +... +keyspace_hits:8337999 +keyspace_misses:910002 +``` -缓存预热就是系统上线后，将相关的缓存数据直接加载到缓存系统。这样就可以避免在用户请求的时候，先查询数据库，然后再将数据缓存的问题！用户直接查询事先被预热的缓存数据！ +$ \text{命中率} = \frac{\text{keyspace\_hits}}{\text{keyspace\_hits} + \text{keyspace\_misses}} $ -**解决方案** +其中： -（1）直接写个缓存刷新页面，上线时手工操作一下； +- `keyspace_hits`：缓存命中的次数 +- `keyspace_misses`：缓存未命中的次数 -（2）数据量不大，可以在项目启动的时候自动进行加载； -（3）定时刷新缓存； -**5、缓存降级** +### 🎯 使用Redis做过异步队列吗，是如何实现的？ -当访问量剧增、服务出现问题（如响应时间慢或不响应）或非核心服务影响到核心流程的性能时，仍然需要保证服务还是可用的，即使是有损服务。系统可以根据一些关键数据进行自动降级，也可以配置开关实现人工降级。 +使用 list 类型保存数据信息，rpush 生产消息，lpop 消费消息，当 lpop 没有消息时，可以 sleep 一段时间，然后再检查有没有信息，如果不想 sleep 的话，可以使用 blpop，在没有信息的时候，会一直阻塞，直到信息的到来。redis 可以通过 pub/sub 主题订阅模式实现一个生产者，多个消费者，当然也存在一定的缺点，当消费者下线时，生产的消息会丢失。 -**缓存降级的最终目的是保证核心服务可用，即使是有损的。而且有些服务是无法降级的（如加入购物车、结算）。** -在进行降级之前要对系统进行梳理，看看系统是不是可以丢卒保帅；从而梳理出哪些必须誓死保护，哪些可降级；比如可以参考日志级别设置预案： -（1）一般：比如有些服务偶尔因为网络抖动或者服务正在上线而超时，可以自动降级； +### 🎯 Redis如何实现延时队列 -（2）警告：有些服务在一段时间内成功率有波动（如在95~100%之间），可以自动降级或人工降级，并发送告警； +使用 Redis 作为延时队列的实现方法有很多，其中一种常见的方式是使用 Redis 的有序集合（Sorted Set）。有序集合通过成员的分数进行排序，非常适合实现延时队列功能。 -（3）错误：比如可用率低于90%，或者数据库连接池被打爆了，或者访问量突然猛增到系统能承受的最大阀值，此时可以根据情况自动降级或者人工降级； +**实现步骤** -（4）严重错误：比如因为特殊原因数据错误了，此时需要紧急人工降级。 +1. **添加任务到延时队列**： 将任务添加到 Redis 有序集合中，使用任务的执行时间作为分数。执行时间可以使用 Unix 时间戳表示。 +2. **轮询检查和执行任务**： 使用一个定时任务（如每秒运行一次）来轮询检查有序集合中是否有需要执行的任务。当任务的执行时间小于等于当前时间时，执行该任务并将其从集合中移除。 -服务降级的目的，是为了防止Redis服务故障，导致数据库跟着一起发生雪崩问题。因此，对于不重要的缓存数据，可以采取服务降级策略，例如一个比较常见的做法就是，Redis出现问题，不去数据库查询，而是直接返回默认值给用户。 +> 使用 sortedset，使用时间戳做 score，消息内容作为 key，调用 zadd 来生产消息，消费者使用 zrangbyscore 获取 n 秒之前的数据做轮询处理。 -**6、缓存热点key** -缓存中的一个Key(比如一个促销商品)，在某个时间点过期的时候，恰好在这个时间点对这个Key有大量的并发请求过来，这些请求发现缓存过期一般都会从后端DB加载数据并回设到缓存，这个时候大并发的请求可能会瞬间把后端DB压垮。 -**解决方案** +### 🎯 Redis如何做内存优化？ -（1）对缓存查询加锁，如果KEY不存在，就加锁，然后查DB入缓存，然后解锁； +尽可能使用散列表（hashes），散列表（是说散列表里面存储的数少）使用的内存非常小，所以你应该尽可能的将你的数据模型抽象到一个散列表里面。比如你的web系统中有一个用户对象，不要为这个用户的名称，姓氏，邮箱，密码设置单独的key，而是应该把这个用户的所有信息存储到一张散列表里面。 -（2）其他进程如果发现有锁就等待，然后等解锁后返回数据或者进入DB查询 +### 🎯 Redis 使用误区？ -缓存与数据库双写一致问题 +Redis最大的问题不是“不会用”，而是“用得像内存版MySQL”。线上常见坑集中在：大Key/热Key、阻塞命令、过期风暴、持久化/集群误解、分布式锁误用、内存淘汰与删除方式、以及缺乏可观测性。做法上遵循：小Key+分片、旁路缓存+限流、TTL抖动+逻辑过期、UNLINK/SCAN、AOF everysec+RDB、只允许持有者解锁+续期、Cluster key-tag、以及指标告警。 ------- +**键过大** +Redis的key是string类型，最大可以是512MB，那么实际中是不是也可以这样用呢？答案是否定的，redis将key保存在一个全局的hashtable，如果key过大，一是占用过多的内存，二是计算hash和字符串比较都会更耗时；一般建议key的大小不超过2kB。 +**Big key** -## 七、分布式相关问题 +或者说是big value，这会导致删除key的操作比较耗时，会阻塞主线程。比如有些同学喜欢用集合类的对象，动辄上百万的元素。对于这类超大集合，一般有两种优化方案，一是采取分片的方式，将每个集合分片控制在较小的范围内，比如小于1000个元素；二是起一个异步任务，对集合中的元素分批进行老化。 -### Redis实现分布式锁 +**全集合扫描** -Redis为单进程单线程模式，采用队列模式将并发访问变成串行访问，且多客户端对Redis的连接并不存在竞争关系Redis中可以使用SETNX命令实现分布式锁。 +比如在业务代码使用了keys*，hgetall，zrange(0, -1)等返回集合中所有元素，这些都属于阻塞操作，一般考虑用scan，hscan等迭代操作代替。 -当且仅当 key 不存在，将 key 的值设为 value。 若给定的 key 已经存在，则 SETNX 不做任何动作 +**单个实例内存过大** -SETNX 是『SET if Not eXists』(如果不存在，则 SET)的简写。 +内存过大有什么问题呢？上文中在讲到持久化的时候其实有说到，无论是生成RDB文件，还是AOF重写，都是要对整个实例的内存数据进行扫描，非常消耗CPU和磁盘资源；当使用Backgroud方式创建子进程时也会涉及到内存空间的拷贝，即便使用了COW机制，也会占用相当的内存开销。另外，在主从复制的第一阶段，save、传输和加载RDB文件的开销，也会随着RDB文件的变大而变大。当单个实例达到瓶颈时，更好的解决方案应该是采用集群方案。 -返回值：设置成功，返回 1 。设置失败，返回 0 。 +**大量key同时过期** -![img](https://img-blog.csdnimg.cn/20191213103148681.png) +redis删除过期键采用了惰性删除和定期删除相结合的策略，惰性删除则是在每次GET/SET操作时去删，定期删除，则是在时间事件中，从整个key空间随机取样，直到过期键比率小于25%，如果同时有大量key过期的话，极可能导致主线程阻塞。一般可以通过做散列来优化处理。 -使用SETNX完成同步锁的流程及事项如下： -使用SETNX命令获取锁，若返回0（key已存在，锁已存在）则获取失败，反之获取成功 -为了防止获取锁后程序出现异常，导致其他线程/进程调用SETNX命令总是返回0而进入死锁状态，需要为该key设置一个“合理”的过期时间 +### 🎯 Redis 中的管道有什么用? -释放锁，使用DEL命令将锁数据删除 +Redis 中的管道（Pipelining）是一种优化技术，允许客户端在一次网络往返中发送多个命令，而不是每个命令发送一次。这种方法减少了网络延迟，提高了吞吐量和性能。 +**管道的工作原理** +在使用管道时，客户端会将一系列命令打包，然后一次性发送给 Redis 服务器。服务器执行这些命令后，将结果一次性返回给客户端。这样做的好处是减少了客户端和服务器之间的网络往返次数，从而提高了性能。 -### 如何解决 Redis 的并发竞争 Key 问题 +```python +import redis -所谓 Redis 的并发竞争 Key 的问题也就是多个系统同时对一个 key 进行操作，但是最后执行的顺序和我们期望的顺序不同，这样也就导致了结果的不同！ +# 创建 Redis 连接 +r = redis.Redis(host='localhost', port=6379, db=0) -推荐一种方案：分布式锁（zookeeper 和 redis 都可以实现分布式锁）。（如果不存在 Redis 的并发竞争 Key 问题，不要使用分布式锁，这样会影响性能） +# 创建管道对象 +pipe = r.pipeline() -基于zookeeper临时有序节点可以实现的分布式锁。大致思想为：每个客户端对某个方法加锁时，在zookeeper上的与该方法对应的指定节点的目录下，生成一个唯一的瞬时有序节点。 判断是否获取锁的方式很简单，只需要判断有序节点中序号最小的一个。 当释放锁的时候，只需将这个瞬时节点删除即可。同时，其可以避免服务宕机导致的锁无法释放，而产生的死锁问题。完成业务流程后，删除对应的子节点释放锁。 +# 批量添加命令到管道 +pipe.set('key1', 'value1') +pipe.set('key2', 'value2') +pipe.get('key1') +pipe.get('key2') -在实践中，当然是从以可靠性为主。所以首推Zookeeper。 +# 执行管道中的所有命令 +results = pipe.execute() -参考：https://www.jianshu.com/p/8bddd381de06 +# 打印结果 +for result in results: + print(result) +``` +**管道的优点** +1. **减少网络延迟**：通过一次性发送多个命令，减少了客户端和服务器之间的网络往返次数，从而减少了网络延迟。 +2. **提高吞吐量**：由于减少了每个命令的网络开销，服务器可以更快地处理更多的命令，从而提高了系统的吞吐量。 +3. **原子性操作**：管道中的所有命令是按顺序执行的，但它们之间不是原子操作。如果需要原子性，可以使用事务（MULTI/EXEC）。 -### 分布式Redis是前期做还是后期规模上来了再做好？为什么？ +**管道的限制** -既然Redis是如此的轻量（单实例只使用1M内存），为防止以后的扩容，最好的办法就是一开始就启动较多实例。即便你只有一台服务器，你也可以一开始就让Redis以分布式的方式运行，使用分区，在同一台服务器上启动多个实例。 +1. **非原子性**：管道中的命令不是原子操作，如果需要原子性，需要使用事务。 +2. **错误处理**：管道执行中，如果某个命令出错，Redis 服务器不会立即返回错误，而是继续执行剩余的命令。客户端在接收到结果时，需要检查每个命令的执行结果。 -一开始就多设置几个Redis实例，例如32或者64个实例，对大多数用户来说这操作起来可能比较麻烦，但是从长久来看做这点牺牲是值得的。 +**高级用法** -这样的话，当你的数据不断增长，需要更多的Redis服务器时，你需要做的就是仅仅将Redis实例从一台服务迁移到另外一台服务器而已（而不用考虑重新分区的问题）。一旦你添加了另一台服务器，你需要将你一半的Redis实例从第一台机器迁移到第二台机器。 +1. **事务中的管道**： 管道可以与事务一起使用，确保一组命令在执行过程中不被其他命令打断。 -### 什么是 RedLock + ```python + pipe = r.pipeline(transaction=True) + ``` -Redis 官方站提出了一种权威的基于 Redis 实现分布式锁的方式名叫 Redlock，此种方式比原先的单节点的方法更安全。它可以保证以下特性： +2. **批量操作**： 对于需要批量操作的大量数据，管道非常适用。例如，批量插入数据或批量获取数据。 -安全特性：互斥访问，即永远只有一个 client 能拿到锁 -避免死锁：最终 client 都可能拿到锁，不会出现死锁的情况，即使原本锁住某资源的 client crash 了或者出现了网络分区 -容错性：只要大部分 Redis 节点存活就可以正常提供服务 ------- +### 🎯 使用Redis统计网站的UV，应该怎么做？ +使用 Redis 统计网站的 UV（Unique Visitors，独立访客）可以通过 HyperLogLog 或 Set 数据结构来实现。以下是两种方法的具体实现方式： -## 八、其他问题 +**方法一：使用 HyperLogLog 统计 UV** -- mySQL里有2000w数据，redis中只存20w的数据，如何保证redis中的数据都是热点数据 +HyperLogLog 是一种基于概率的数据结构，适用于大规模去重计数。它使用少量内存就能提供高准确率的去重计数。 -- 请用Redis和任意语言实现一段恶意登录保护的代码，限制1小时内每用户Id最多只能登录5次。具体登录函数或功能用空函数即可，不用详细写出 +**实现步骤** -- **3. redis常见性能问题和解决方案：** +1. **记录访问**： 每当有用户访问网站时，将用户的唯一标识（例如 IP 地址或用户 ID）添加到 HyperLogLog 中。 +2. **获取 UV**： 使用 `PFCOUNT` 命令获取 HyperLogLog 的基数（即独立访客数）。 - (1) Master最好不要做任何持久化工作，如RDB内存快照和AOF日志文件 +**示例代码** - (2) 如果数据比较重要，某个Slave开启AOF备份数据，策略设置为每秒同步一次 +```python +import redis - (3) 为了主从复制的速度和连接的稳定性，Master和Slave最好在同一个局域网内 +# 创建 Redis 连接 +r = redis.Redis(host='localhost', port=6379, db=0) - (4) 尽量避免在压力很大的主库上增加从库 +def record_visit(user_id): + r.pfadd('site_uv', user_id) - (5) 主从复制不要用图状结构，用单向链表结构更为稳定，即：Master <- Slave1 <- Slave2 <- Slave3... +def get_uv(): + return r.pfcount('site_uv') - 这样的结构方便解决单点故障问题，实现Slave对Master的替换。如果Master挂了，可以立刻启用Slave1做Master，其他不变。 +# 示例：记录用户访问 +record_visit('user_123') +record_visit('user_456') +# 获取 UV +print(f"Unique Visitors: {get_uv()}") +``` +**方法二：使用 Set 统计 UV** -### 过期删除策略 +Set 是一种集合数据结构，适用于精确去重计数。虽然 Set 的内存占用比 HyperLogLog 大，但它能精确统计独立访客数。 -**Redis的过期键的删除策略** +**实现步骤** -过期策略通常有以下三种： +1. **记录访问**： 每当有用户访问网站时，将用户的唯一标识添加到 Set 中。 +2. **获取 UV**： 使用 `SCARD` 命令获取 Set 的基数。 -1. 定时过期：每个设置过期时间的key都需要创建一个定时器，到过期时间就会立即清除。该策略可以立即清除过期的数据，对内存很友好；但是会占用大量的CPU资源去处理过期的数据，从而影响缓存的响应时间和吞吐量。 -2. 惰性过期：只有当访问一个key时，才会判断该key是否已过期，过期则清除。该策略可以最大化地节省CPU资源，却对内存非常不友好。极端情况可能出现大量的过期key没有再次被访问，从而不会被清除，占用大量内存。 -3. 定期过期：每隔一定的时间，会扫描一定数量的数据库的expires字典中一定数量的key，并清除其中已过期的key。该策略是前两者的一个折中方案。通过调整定时扫描的时间间隔和每次扫描的限定耗时，可以在不同情况下使得CPU和内存资源达到最优的平衡效果。 +**示例代码** -Redis中同时使用了**惰性过期**和**定期过期**两种过期策略。 +```python +import redis -**设置过期时间和永久有效的命令是什么？** +# 创建 Redis 连接 +r = redis.Redis(host='localhost', port=6379, db=0) -EXPIRE和PERSIST命令 +def record_visit(user_id): + r.sadd('site_uv_set', user_id) -**Redis的内存淘汰策略有哪些？** +def get_uv(): + return r.scard('site_uv_set') -全局的键空间选择性移除 +# 示例：记录用户访问 +record_visit('user_123') +record_visit('user_456') -1. noeviction：当内存不足以容纳新写入数据时，新写入操作会报错。 -2. allkeys-lru：当内存不足以容纳新写入数据时，在键空间中，移除最近最少使用的key。（这个是最常用的） -3. allkeys-random：当内存不足以容纳新写入数据时，在键空间中，随机移除某个key。 +# 获取 UV +print(f"Unique Visitors: {get_uv()}") +``` -设置过期时间的键空间选择性移除 +**方法选择** -1. volatile-lru：当内存不足以容纳新写入数据时，在设置了过期时间的键空间中，移除最近最少使用的key。 -2. volatile-random：当内存不足以容纳新写入数据时，在设置了过期时间的键空间中，随机移除某个key。 -3. volatile-ttl：当内存不足以容纳新写入数据时，在设置了过期时间的键空间中，有更早过期时间的key优先移除。 +- **HyperLogLog**：适用于大量数据且对内存使用敏感的场景。它使用固定大小的内存（约 12 KB），但统计结果有一定误差（误差率约 0.81%）。 +- **Set**：适用于需要精确统计结果的场景。它能精确去重计数，但内存使用随着数据量增加而增加。 -在2.8.13的版本里，默认是noeviction，在3.2.3版本里默认是volatile-lru。 -| 策略 | 描述 | -| --------------- | ------------------------------------------------------------ | -| volatile-lru | 从已设置过期时间的KV集中优先对最近最少使用(less recently used)的数据淘汰 | -| volitile-ttl | 从已设置过期时间的KV集中优先对剩余时间短(time to live)的数据淘汰 | -| volitile-random | 从已设置过期时间的KV集中随机选择数据淘汰 | -| allkeys-lru | 从所有KV集中优先对最近最少使用(less recently used)的数据淘汰 | -| allKeys-random | 从所有KV集中随机选择数据淘汰 | -| noeviction | 不淘汰策略，若超过最大内存，返回错误信息 | +### 🎯 假如 Redis 里面有 **1** 亿个 key，其中有 10w 个 key 是以某个固定的已知的前缀开头的，如何将它们全部找出来? +使用 keys 指令可以扫出指定模式的 key 列表。 -**Redis如何做内存优化？** +对方接着追问:如果这个 redis 正在给线上的业务提供服务，那使用 keys 指令会有什么问题? -尽可能使用散列表（hashes），散列表（是说散列表里面存储的数少）使用的内存非常小，所以你应该尽可能的将你的数据模型抽象到一个散列表里面。比如你的web系统中有一个用户对象，不要为这个用户的名称，姓氏，邮箱，密码设置单独的key，而是应该把这个用户的所有信息存储到一张散列表里面。 +这个时候你要回答 redis 关键的一个特性：redis 的单线程的。keys 指令会导致线程阻塞一段时间，线上服务会停顿，直到指令执行完毕，服务才能恢复。这个时候可以使用 scan 指 令，scan 指令可以无阻塞的提取出指定模式的 key 列表，但是会有一定的重复概率，在客户端做一次去重就可以了，但是整体所花费的时间会比直接用 keys 指令长。 +## References +- https://juejin.im/post/6844904017387077640 -## 参考与感谢 +- https://www.wmyskxz.com/2020/03/25/dong-yi-dian-python-xi-lie-kuai-su-ru-men-1/#toc-heading-22 -https://www.wmyskxz.com/2020/03/25/dong-yi-dian-python-xi-lie-kuai-su-ru-men-1/#toc-heading-22 +- https://mp.weixin.qq.com/s/f9N13fnyTtnu2D5sKZiu9w -https://mp.weixin.qq.com/s/f9N13fnyTtnu2D5sKZiu9w +- https://blog.csdn.net/ThinkWon/article/details/103522351/ -https://www.wmyskxz.com/2020/03/25/dong-yi-dian-python-xi-lie-kuai-su-ru-men-1/#toc-heading-58 \ No newline at end of file diff --git a/docs/interview/Spring-FAQ.md b/docs/interview/Spring-FAQ.md index d25393270c..88ec1f31d0 100644 --- a/docs/interview/Spring-FAQ.md +++ b/docs/interview/Spring-FAQ.md @@ -1,130 +1,358 @@ +--- +title: Spring框架面试题大全 +date: 2024-12-15 +tags: + - Spring + - Interview +categories: Interview +--- + +![](https://img.starfish.ink/common/faq-banner.png) + +> Spring框架作为Java生态系统的**核心基础设施**，是每个Java开发者必须掌握的技术栈。从简单的IOC容器到复杂的微服务架构，从传统的SSM到现代的响应式编程，Spring技术栈的深度和广度决定了开发者的职业高度。 +> +> +> Spring 面试，围绕着这么几个核心方向准备： +> +> - **Spring Framework核心**（IOC容器、依赖注入、Bean生命周期、AOP编程） +> - **Spring Boot自动配置**（Starter机制、条件装配、外部化配置、Actuator监控） +> - **Spring MVC请求处理**（DispatcherServlet、HandlerMapping、视图解析、异常处理） +> - **Spring事务管理**（声明式事务、传播行为、隔离级别、事务失效问题） +> - **Spring Data数据访问**（Repository模式、分页查询、事务管理、多数据源） +> - **Spring Security安全框架**（认证授权、JWT集成、CSRF防护、方法级安全） +> - **Spring Cloud微服务**（服务注册发现、配置中心、熔断降级、网关路由） -> 基于Spring Framework 4.x 总结的常见面试题，系统学习建议还是官方文档走起：https://spring.io/projects/spring-framework#learn -## 一、一般问题 +## 🗺️ 知识导航 -### 开发中主要使用 Spring 的什么技术 ? +### 🏷️ 核心知识分类 -1. IOC 容器管理各层的组件 -2. 使用 AOP 配置声明式事务 -3. 整合其他框架 +1. **🏗️ Spring Framework核心**：IOC容器、依赖注入、Bean生命周期、作用域、循环依赖、BeanFactory vs ApplicationContext +2. **🎯 AOP面向切面编程**：AOP概念、代理模式、通知类型、切点表达式、AspectJ集成、动态代理 vs CGLIB +3. **🌐 Spring MVC架构**：请求处理流程、DispatcherServlet、HandlerMapping、视图解析、参数绑定、异常处理 +4. **🚀 Spring Boot核心特性**：自动配置原理、Starter机制、条件装配、外部化配置、Actuator监控、打包部署 +5. **💾 数据访问与事务**：Spring Data、事务管理、传播行为、隔离级别、事务失效、多数据源、分页查询 +6. **🔒 Spring Security安全**：认证授权流程、SecurityContext、JWT集成、CSRF防护、方法级安全、OAuth2集成 +7. **☁️ Spring Cloud微服务**：服务注册发现、配置中心、熔断降级、API网关、链路追踪、分布式事务 +8. **📝 注解与配置**：常用注解、JavaConfig、Profile管理、属性注入、条件装配、自定义注解 +9. **🔧 高级特性与实践**：事件机制、国际化、缓存抽象、任务调度、WebSocket、响应式编程 +10. **🎯 面试重点与实践**：设计模式、性能优化、最佳实践、常见陷阱、Spring vs SpringBoot、整体架构设计 +### 🔑 面试话术模板 +| **问题类型** | **回答框架** | **关键要点** | **深入扩展** | +| ------------ | ------------------------------------- | ---------------------------- | -------------------------------- | +| **概念解释** | 定义→核心特性→工作原理→应用场景 | 准确定义，突出核心价值 | 源码分析，设计思想，最佳实践 | +| **原理分析** | 背景问题→解决方案→实现机制→优势劣势 | 图解流程，关键步骤 | 底层实现，性能考量，扩展机制 | +| **技术对比** | 相同点→差异点→适用场景→选择建议 | 多维度对比，实际场景 | 性能差异，实现复杂度，维护成本 | +| **实践应用** | 业务场景→技术选型→具体实现→踩坑经验 | 实际项目案例，代码示例 | 性能优化，监控运维，故障处理 | +| **架构设计** | 需求分析→技术选型→架构设计→实施方案 | 系统思考，全局视角 | 可扩展性，高可用，一致性保证 | -### Spring有哪些优点？ +--- -- **轻量级：**Spring在大小和透明性方面绝对属于轻量级的，基础版本的Spring框架大约只有2MB。 -- **控制反转(IOC)：**Spring使用控制反转技术实现了松耦合。依赖被注入到对象，而不是创建或寻找依赖对象。 -- **面向切面编程(AOP)：** Spring支持面向切面编程，同时把应用的业务逻辑与系统的服务分离开来。 -- **容器：**Spring包含并管理应用程序对象的配置及生命周期。 -- **MVC框架：**Spring的web框架是一个设计优良的web MVC框架，很好的取代了一些web框架。 -- **事务管理：**Spring对下至本地业务上至全局业务(JAT)提供了统一的事务管理接口。 -- **异常处理：**Spring提供一个方便的API将特定技术的异常(由JDBC, Hibernate, 或JDO抛出)转化为一致的、Unchecked异常。 +## 🏗️ 一、Spring Framework核心 +**核心理念**：通过IOC控制反转和AOP面向切面编程，实现松耦合、高内聚的企业级应用开发框架。 +### 🎯 使用 **Spring** 框架能带来哪些好处? -### Spring模块 +1. **简化开发**：Spring 框架通过高度的 **抽象** 和 **自动化配置**，大大简化了 Java 开发，减少了开发者手动编写大量配置代码的需要。 + - **依赖注入（DI）**：Spring 提供了 **依赖注入**（DI）机制，通过 `@Autowired` 或构造器注入，自动管理组件之间的依赖关系，减少了代码的耦合性，提高了可维护性。 -![spring overview](https://docs.spring.io/spring/docs/4.3.27.RELEASE/spring-framework-reference/htmlsingle/images/spring-overview.png) + - **面向切面编程（AOP）**：Spring 提供了 **AOP** 功能，使得日志记录、安全控制、事务管理等横切关注点的代码能够与业务逻辑分离，降低了系统的复杂度。 -### 简述 AOP 和 IOC 概念 +2. **松耦合架构**：Spring 提供了 **松耦合的架构**，通过 **依赖注入** 和 **接口/抽象类**，让组件之间的依赖关系松散，方便了模块的解耦和扩展。 -AOP：Aspect Oriented Program, 面向(方面)切面的编程;Filter(过滤器)也是一种 AOP. AOP 是一种新的 方法论, 是对传统 OOP(Object-OrientedProgramming, 面向对象编程) 的补充. AOP 的主要编程对象是切面(aspect),而切面模块化横切关注点.可以举例通过事务说明. + - 通过 **依赖注入（DI）**，组件之间的依赖不再通过硬编码连接，而是由 Spring 容器管理，依赖关系在运行时通过配置注入。 -IOC：Invert Of Control, 控制反转. 也称为 DI(依赖注入)其思想是反转资源获取的方向. 传统的资源查找方式要求组件向容器发起请求查找资源.作为回应, 容器适时的返回资源. 而应用了 IOC 之后, 则是容器主动地将资源推送给它所管理的组件,组件所要做的仅是选择一种合适的方式来接受资源. 这种行为也被称为查找的被动形式 + - Spring 提供了丰富的 **接口** 和 **抽象**，让开发者可以轻松替换和扩展功能。 +3. **更好的可维护性和可测试性** + - **依赖注入**：使得组件之间的依赖关系通过容器进行管理，从而更容易进行单元测试。可以通过 **Mocking** 或 **Stubbing** 来模拟依赖项，方便进行单元测试。 -## 二、依赖注入 + - **分层架构支持**：Spring 提供的服务和 DAO 层可以更加清晰地进行分层，使得系统结构更加清晰，便于维护。 -IoC（Inverse of Control:控制反转）是一种**设计思想**，就是 **将原本在程序中手动创建对象的控制权，交由Spring框架来管理。** IoC 在其他语言中也有应用，并非 Spring 特有。 **IoC 容器是 Spring 用来实现 IoC 的载体， IoC 容器实际上就是个Map（key，value）,Map 中存放的是各种对象。** + - **事务管理**：Spring 提供了声明式事务管理，方便进行数据库操作的事务控制，降低了编码复杂度，同时提高了事务管理的灵活性。 -将对象之间的相互依赖关系交给 IoC 容器来管理，并由 IoC 容器完成对象的注入。这样可以很大程度上简化应用的开发，把应用从复杂的依赖关系中解放出来。 **IoC 容器就像是一个工厂一样，当我们需要创建一个对象的时候，只需要配置好配置文件/注解即可，完全不用考虑对象是如何被创建出来的。** 在实际项目中一个 Service 类可能有几百甚至上千个类作为它的底层，假如我们需要实例化这个 Service，你可能要每次都要搞清这个 Service 所有底层类的构造函数，这可能会把人逼疯。如果利用 IoC 的话，你只需要配置好，然后在需要的地方引用就行了，这大大增加了项目的可维护性且降低了开发难度。 +4. **集成性强**：Spring 的另一个关键优势是其 **良好的集成能力**。Spring 提供了许多与常见技术框架的集成，包括： -### 什么是 Spring IOC 容器？ + - **JPA、Hibernate、MyBatis** 等持久化框架的集成。 -Spring 框架的核心是 Spring 容器。容器创建对象，将它们装配在一起，配置它们并管理它们的完整生命周期。Spring 容器使用依赖注入来管理组成应用程序的组件。容器通过读取提供的配置元数据来接收对象进行实例化，配置和组装的指令。该元数据可以通过 XML，Java 注解或 Java 代码提供。 + - **Spring Security**：提供强大的安全框架，支持身份验证、授权控制等功能。 -![image.png](https://upload-images.jianshu.io/upload_images/3101171-33099411d16ca051.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240) + - **Spring MVC**：可以与不同的 Web 框架（如 JSP、Freemarker、Thymeleaf）以及 RESTful 风格的接口进行集成。 + - **Spring Boot**：让构建、部署 Spring 应用更加便捷，简化了 Spring 项目的配置和启动。 +5. **事务管理**：Spring 提供了 **声明式事务管理**，可以通过 `@Transactional` 注解来实现事务的管理，避免了手动管理事务的麻烦，并且支持多种事务传播机制和隔离级别。 -### 什么是依赖注入？ +6. **Spring Boot 提升开发效率**：Spring Boot 是 Spring 的子项目，旨在简化 Spring 应用的配置和部署。它提供了以下优点： -**依赖注入（DI）是在编译阶段尚未知所需的功能是来自哪个的类的情况下，将其他对象所依赖的功能对象实例化的模式**。这就需要一种机制用来激活相应的组件以提供特定的功能，所以**依赖注入是控制反转的基础**。否则如果在组件不受框架控制的情况下，框架又怎么知道要创建哪个组件？ + - **自动配置**：Spring Boot 自动配置了很多常见的功能，如数据库连接、Web 配置、消息队列等，减少了手动配置的工作量。 -依赖注入有以下三种实现方式： + - **内嵌 Web 容器**：Spring Boot 提供了内嵌的 Tomcat、Jetty 等 Web 容器，可以将应用打包成独立的 JAR 或 WAR 文件，便于部署和运行。 -1. 构造器注入 -2. Setter方法注入（属性注入） -3. 接口注入 + - **快速开发**：Spring Boot 提供了大量的默认配置和开箱即用的功能，可以快速启动项目，并减少了配置和开发的时间。 -### Spring 中有多少种 IOC 容器？ +### 🎯 Spring有哪些优点？ -在 Spring IOC 容器读取 Bean 配置创建 Bean 实例之前，必须对它进行实例化。只有在容器实例化后， 才可以从 IOC 容器里获取 Bean 实例并使用 +- **轻量级**：Spring在大小和透明性方面绝对属于轻量级的，基础版本的Spring框架大约只有2MB。 +- **控制反转(IOC)**：Spring使用控制反转技术实现了松耦合。依赖被注入到对象，而不是创建或寻找依赖对象。 +- **面向切面编程(AOP)**：Spring支持面向切面编程，同时把应用的业务逻辑与系统的服务分离开来。 +- **容器**：Spring包含并管理应用程序对象的配置及生命周期。 +- **MVC框架**：Spring的web框架是一个设计优良的web MVC框架，很好的取代了一些web框架。 +- **事务管理**：Spring对下至本地业务上至全局业务(JAT)提供了统一的事务管理接口。 +- **异常处理**：Spring提供一个方便的API将特定技术的异常(由JDBC, Hibernate, 或JDO抛出)转化为一致的、Unchecked异常。 -Spring 提供了两种类型的 IOC 容器实现 -- BeanFactory：IOC 容器的基本实现 -- ApplicationContext：提供了更多的高级特性，是 BeanFactory 的子接口 +### 🎯 什么是Spring框架？核心特性有哪些？ -BeanFactory 是 Spring 框架的基础设施，面向 Spring 本身；ApplicationContext 面向使用 Spring 框架的开发者，几乎所有的应用场合都直接使用 ApplicationContext 而非底层的 BeanFactory； +Spring 是一个开源的企业级Java应用开发框架，由Rod Johnson创建，目标是简化企业级应用开发。 -无论使用何种方式, 配置文件是相同的。 +**核心特性包括**： + +**1. IOC控制反转**： +- 对象创建和依赖关系管理交给Spring容器 +- 通过依赖注入实现松耦合架构 +- 提高了代码的可测试性和可维护性 + +**2. AOP面向切面编程**： +- 将横切关注点从业务逻辑中分离 +- 支持声明式事务、日志、安全等 +- 基于动态代理和CGLIB实现 + +**3. 轻量级容器**： +- 非侵入式设计，POJO即可 +- 容器启动快，内存占用少 +- 可以选择性使用框架功能 + +**4. 一站式解决方案**： +- 提供完整的企业级技术栈 +- 良好的框架集成能力 +- Spring Boot、Spring Cloud等生态丰富 + +**💻 代码示例**： +```java +// IOC容器基本使用 +@Component +public class UserService { + @Autowired + private UserRepository userRepository; + + public User findById(Long id) { + return userRepository.findById(id); + } +} + +// AOP切面示例 +@Aspect +@Component +public class LoggingAspect { + @Around("@annotation(Log)") + public Object logExecutionTime(ProceedingJoinPoint joinPoint) throws Throwable { + long start = System.currentTimeMillis(); + Object result = joinPoint.proceed(); + long executionTime = System.currentTimeMillis() - start; + System.out.println("Method executed in: " + executionTime + "ms"); + return result; + } +} +``` + +### 🎯 什么是IOC？依赖注入的实现原理是什么？ +IoC（Inverse of Control:控制反转）是一种**设计思想**，就是 **将原本在程序中手动创建对象的控制权，交由Spring框架来管理。** IoC 在其他语言中也有应用，并非 Spring 特有。 **IoC 容器是 Spring 用来实现 IoC 的载体， IoC 容器实际上就是个Map（key，value）,Map 中存放的是各种对象。** + +**IoC 的核心思想** +在传统的编程中，程序中会有显式的代码来创建对象并管理它们的生命周期。IoC 则通过将这些控制权交给容器，让容器在适当的时机创建对象，并根据需要注入对象的依赖项。 -### BeanFactory 和 ApplicationContext 区别 +IoC 的核心思想是“控制权反转”——对象不再主动创建依赖，而是由容器在运行时自动注入依赖，实现代码解耦与统一管理。 -| BeanFactory | ApplicationContext | -| -------------------------- | ------------------------ | -| 懒加载 | 即时加载 | -| 它使用语法显式提供资源对象 | 它自己创建和管理资源对象 | -| 不支持国际化 | 支持国际化 | -| 不支持基于依赖的注解 | 支持基于依赖的注解 | +将对象之间的相互依赖关系交给 IoC 容器来管理，并由 IoC 容器完成对象的注入。这样可以很大程度上简化应用的开发，把应用从复杂的依赖关系中解放出来。 **IoC 容器就像是一个工厂一样，当我们需要创建一个对象的时候，只需要配置好配置文件/注解即可，完全不用考虑对象是如何被创建出来的。** +> "IOC全称Inversion of Control，即控制反转，是Spring框架的核心思想： +> +> **传统模式 vs IOC模式**： +> +> - **传统模式**：对象主动创建和管理依赖对象（我找你） +> - **IOC模式**：对象被动接受外部注入的依赖（你找我） +> - **控制权反转**：从对象内部转移到外部容器 +**IoC 的实现原理** -**ApplicationContext** +IoC（控制反转）是一种思想，它将对象的控制权从代码中“反转”给外部容器。 +目前 IoC 的主流实现方式是 **依赖注入（Dependency Injection, DI）**。 -ApplicationContext 的主要实现类： +**依赖注入（DI）** 是通过容器在创建对象时，将所需依赖自动注入到对象中的一种机制。常见注入方式包括： -- ClassPathXmlApplicationContext：从类路径下加载配置文件 -- FileSystemXmlApplicationContext: 从文件系统中加载配置文件 -- ConfigurableApplicationContext 扩展于 ApplicationContext，新增加两个主要方法：refresh() 和 close()， 让 ApplicationContext具有启动、刷新和关闭上下文的能力 -- WebApplicationContext 是专门为 WEB 应用而准备的，它允许从相对于 WEB 根目录的路径中完成初始化工作 -- ApplicationContext 在初始化上下文时就实例化所有单例的 Bean +- **构造器注入** +- **Setter 注入** +- **字段注入** -![javadoop.com](https://www.javadoop.com/blogimages/spring-context/1.png) +在 Spring 中，**IoC 容器**（如 `ApplicationContext`）负责管理 Bean 的生命周期、依赖注入与装配，它是 IoC 思想的具体落地载体。 -**从 IOC 容器中获取 Bean** +此外，早期 IoC 还有“依赖查找（Dependency Lookup）”的实现方式，但由于耦合性较高，目前几乎被依赖注入完全取代。 -- 调用 ApplicationContext 的 getBean() 方法 +**IoC 的好处**「列举 IoC 的一些好处」 -![](https://imgkr.cn-bj.ufileos.com/3aa6c769-3b9a-4882-b2e1-622c127437a3.png) +- **解耦**：对象之间不需要直接依赖，可以让系统更容易扩展和维护。 +- **易于测试**：通过容器，可以方便地替换依赖项，进行单元测试时可以使用模拟对象（Mock Object）来替代真实对象。 +- **灵活性**：IoC 容器提供了配置和管理对象依赖关系的能力，可以灵活配置依赖项，而不需要改变代码。 +**💻 代码示例**： ```java -ApplicationContext ctx = new ClassPathXmlApplicationContext("beans.xml"); -HelloWorld helloWorld = (HelloWorld) ctx.getBean("helloWorld"); -helloWorld.hello(); +// 构造器注入（推荐） +@Service +public class OrderService { + private final PaymentService paymentService; + private final InventoryService inventoryService; + + public OrderService(PaymentService paymentService, + InventoryService inventoryService) { + this.paymentService = paymentService; + this.inventoryService = inventoryService; + } +} + +// Setter注入 +@Service +public class UserService { + private EmailService emailService; + + @Autowired + public void setEmailService(EmailService emailService) { + this.emailService = emailService; + } +} ``` -### 列举 IoC 的一些好处 +### 🎯 什么是 Spring IOC 容器？ + +Spring IoC（Inverse of Control，控制反转）容器是Spring框架的核心组件，它负责管理应用程序中的对象（称为Bean）。IoC容器通过控制反转的方式，将组件的创建、配置和依赖关系管理从应用程序代码中分离出来，由容器来处理。 + +IoC容器的主要功能包括： + +1. **依赖注入（DI）**：IoC容器能够自动注入Bean的依赖项，而不需要手动创建或查找这些依赖项。 +2. **配置管理**：IoC容器使用配置元数据（如XML、注解或Java配置类）来创建和管理Bean。 +3. **Bean生命周期管理**：IoC容器管理Bean的整个生命周期，包括实例化、初始化、使用和销毁。 +4. **作用域控制**：IoC容器支持不同的Bean作用域，如单例（Singleton）、原型（Prototype）、请求（Request）、会话（Session）等。 +5. **依赖关系解析**：IoC容器能够解析Bean之间的依赖关系，并根据配置自动组装这些Bean。 +6. **事件发布和监听**：IoC容器支持事件驱动模型，允许Bean发布事件和监听其他Bean的事件。 +7. **类型转换和数据绑定**：IoC容器提供类型转换服务，能够将配置数据绑定到Bean的属性上。 +8. **自动装配**：IoC容器能够根据Bean的类型和名称自动装配依赖关系，减少显式配置的需要。 +9. **扩展点**：IoC容器提供了多个扩展点，如BeanFactoryPostProcessor和BeanPostProcessor，允许开发者自定义容器的行为。 + +通过使用Spring IoC容器，开发者可以专注于业务逻辑的实现，而不必关心对象的创建和依赖管理，从而提高代码的可维护性、可测试性和灵活性。 + +> “Spring IoC 容器的主要功能包括对象的创建与生命周期管理、依赖注入（DI）、自动装配、bean 的作用域管理、AOP 支持、事件机制、资源管理与外部配置集成等。通过这些功能，Spring IoC 容器使得开发者能够更加灵活、高效地管理应用中的对象和它们之间的依赖关系，从而实现解耦、提高代码的可维护性和扩展性。” + + + +### 🎯 Spring 默认是单例的，为什么选择用容器这种方式实现呢，还有别的单例实现方式，为什么不用呢 + +> Spring 采用的是 **容器式单例（Container-managed Singleton）**， +> 也就是在 IoC 容器中一个 Bean 只会有一个实例存在。 +> 这种方式比传统单例模式更灵活，因为对象的创建、依赖注入、生命周期管理都交给了容器，而不是写死在类中。 +> +> 传统的懒汉、饿汉、双检锁、静态内部类虽然能实现单例，但缺乏可扩展性和可配置性，也无法支持 AOP、依赖注入、Mock 等框架特性。 +> +> 所以 Spring 选择用容器来管理单例，这是对“单例模式”的一种**框架级实现和增强**。 + + + +### 🎯 Spring 中有多少种 IOC 容器？ + +Spring 中的 org.springframework.beans 包和 org.springframework.context 包构成了 Spring 框架 IoC 容器的基础。 + +在 Spring IOC 容器读取 Bean 配置创建 Bean 实例之前，必须对它进行实例化。只有在容器实例化后， 才可以从 IOC 容器里获取 Bean 实例并使用 + +Spring 提供了两种类型的 IOC 容器实现 + +- BeanFactory：IOC 容器的基本实现 + +- ApplicationContext：提供了更多的高级特性，是 BeanFactory 的子接口 + +BeanFactory 是 Spring 框架的基础设施，面向 Spring 本身；ApplicationContext 面向使用 Spring 框架的开发者，几乎所有的应用场合都直接使用 ApplicationContext 而非底层的 BeanFactory； + +无论使用何种方式, 配置文件是相同的。 + + + +### 🎯 BeanFactory 和 ApplicationContext 区别？ -- 它将最小化应用程序中的代码量； -- 它将使您的应用程序易于测试，因为它不需要单元测试用例中的任何单例或 JNDI 查找机制； -- 它以最小的影响和最少的侵入机制促进松耦合； -- 它支持即时的实例化和延迟加载服务 +在 Spring 框架中，`BeanFactory` 和 `ApplicationContext` 都是 IoC 容器的核心接口，它们都负责管理和创建 Bean，但它们之间有一些关键的区别。了解这些区别可以帮助你选择合适的容器类型，并正确使用它们。 +- **BeanFactory**：是 Spring 最基础的容器，负责管理 Bean 的生命周期以及依赖注入。它提供了最基础的功能，通常用于内存或资源较为有限的环境中。 +- **ApplicationContext**：是 `BeanFactory` 的一个子接口，扩展了 `BeanFactory` 的功能，提供了更多的企业级特性，如事件发布、国际化支持、AOP 支持等。`ApplicationContext` 是一个功能更为丰富的容器，通常在大多数 Spring 应用中使用。 +| 功能/特性 | **BeanFactory** | **ApplicationContext** | +| -------------------------- | -------------------------------------- | ------------------------------------------------------------ | +| **继承关系** | `BeanFactory` 是最基本的容器接口。 | `ApplicationContext` 继承自 `BeanFactory`，并扩展了更多功能。 | +| **懒加载（Lazy Loading）** | 默认懒加载 | 支持懒加载，容器启动时不会立即初始化所有的 Bean，直到真正需要它们时才初始化。 | +| **国际化支持** | 不支持国际化。 | 提供国际化支持，可以使用 `MessageSource` 进行消息的本地化。 | +| **事件机制** | 不支持事件发布与监听。 | 提供事件发布与监听机制，可以使用 `ApplicationEventPublisher` 发布和监听事件。 | +| **AOP 支持** | 不支持 AOP | 支持 AOP（如事务管理、日志记录等）。 | +| **注解驱动配置** | 不支持自动扫描和注解驱动配置 | 支持注解驱动配置，支持 `@ComponentScan` 自动扫描组件。 | +| **Bean 定义的配置** | 仅支持通过 XML 或 Java 配置来定义 Bean | 支持通过 XML、JavaConfig 或注解配置来定义 Bean。 | +| **刷新容器功能** | 没有刷新容器的功能 | 提供 `refresh()` 方法，可以刷新容器，重新加载配置文件和 Bean。 | -### Spring IoC 的实现机制 +**常用的实现类** + +- **BeanFactory 的实现类**： + - `XmlBeanFactory`（已废弃） + - `SimpleBeanFactory` + - `DefaultListableBeanFactory`（最常用） +- **ApplicationContext 的实现类**： + - `ClassPathXmlApplicationContext`：基于 XML 配置文件的上下文。 + - `AnnotationConfigApplicationContext`：基于注解配置的上下文。 + - `GenericWebApplicationContext`：适用于 Web 应用的上下文。 + +> “`BeanFactory` 是 Spring 最基本的容器接口，负责创建和管理 Bean，但它功能较为简洁，通常用于资源有限的环境。而 `ApplicationContext` 继承了 `BeanFactory`，并提供了更多企业级功能，如国际化支持、事件机制和 AOP 等。`ApplicationContext` 是大多数 Spring 应用中使用的容器，它具有更强的功能和灵活性。通常推荐使用 `ApplicationContext`，除非你有特定的性能或资源要求。” + +```java +// BeanFactory基础用法 +DefaultListableBeanFactory beanFactory = new DefaultListableBeanFactory(); +XmlBeanDefinitionReader reader = new XmlBeanDefinitionReader(beanFactory); +reader.loadBeanDefinitions("applicationContext.xml"); +UserService userService = beanFactory.getBean(UserService.class); + +// ApplicationContext用法（推荐） +ApplicationContext context = new AnnotationConfigApplicationContext(AppConfig.class); +UserService userService = context.getBean(UserService.class); + +// 利用ApplicationContext的高级特性 +@Component +public class EventPublisher implements ApplicationContextAware { + private ApplicationContext applicationContext; + + @Override + public void setApplicationContext(ApplicationContext applicationContext) { + this.applicationContext = applicationContext; + } + + public void publishEvent(String message) { + applicationContext.publishEvent(new CustomEvent(this, message)); + } +} +``` + + + +### 🎯 Spring IoC 的实现机制？ + +> Spring IoC 本质上是通过 **容器 + 工厂模式** 实现的，它把对象的创建和依赖注入交给容器完成。 +> 容器启动时会先解析配置文件或注解，把类信息解析成 `BeanDefinition` 并注册到容器中。 +> 当需要创建 Bean 时，Spring 通过反射实例化对象，并根据依赖关系进行注入，然后执行初始化方法，最后把 Bean 放入单例池中管理。 +> 在这个过程中，还可以通过 `BeanPostProcessor`、AOP、事务等扩展点，对 Bean 做增强。 +> 简单理解就是：**Spring IoC 就是一个 Bean 工厂，负责统一管理 Bean 的生命周期和依赖注入，核心机制依赖反射、BeanDefinition 和容器缓存**。 Spring 中的 IoC 的实现原理就是工厂模式加反射机制，示例： @@ -144,7 +372,7 @@ class Orange implements Fruit { } class Factory { public static Fruit getInstance(String ClassName) { - Fruit f=null; + Fruit f = null; try { f=(Fruit)Class.forName(ClassName).newInstance(); } catch (Exception e) { @@ -165,20 +393,136 @@ class Client { -## 三、Beans +### 🎯 什么是 BeanDefinition？ + +> Spring IoC 的核心是 **BeanDefinition + BeanFactory**。 +> +> IoC 容器启动时会先解析配置文件或注解，把每个 Bean 转换成一个 `BeanDefinition`，里面记录了 Bean 的类名、作用域、依赖、初始化方法等信息，并存到一个 Map 里。 +> +> 当需要实例化时，Spring 会根据 BeanDefinition 创建对象，并执行依赖注入和初始化，把最终的 Bean 存到单例池里。 +> +> 也就是说，**BeanDefinition 是 Spring IoC 内部的蓝图，IoC 容器就是靠它来知道“应该创建什么对象、怎么装配、生命周期怎么管理”的**。 +> +> 举个例子，如果我在类上写了 `@Component`，Spring 会扫描到这个类，把它解析成一个 BeanDefinition（scope=singleton, className=xx），再交给容器去实例化和管理。 + +**什么是 BeanDefinition？** + +- `BeanDefinition` 就是 Spring IoC 容器内部对 **Bean 元数据** 的抽象，它是容器能正确创建、配置、管理 Bean 的“蓝图”。 +- **来源**：XML 配置、注解（@Component、@Bean）、JavaConfig、第三方 FactoryBean。 +- **内容**（核心属性）： + - `beanClassName`：Bean 的全限定类名 + - `scope`：作用域（singleton / prototype / request / session） + - `propertyValues`：依赖注入的属性值（构造器参数 / Setter 注入） + - `autowireMode`：自动装配方式（byName / byType） + - `initMethodName`、`destroyMethodName`：生命周期回调 + - `lazyInit`：是否懒加载 + - `dependsOn`：依赖的其他 Bean + +可以理解为 **BeanDefinition 是 Spring IoC 容器里所有 Bean 的配置清单**。 + +**IoC 容器加载与 BeanDefinition 流程** + +1. **配置解析** + + - XML 配置 → `XmlBeanDefinitionReader` 解析 XML 元素，转成 BeanDefinition。 + - 注解配置 → `ClassPathBeanDefinitionScanner` 扫描包路径，解析 @Component/@Bean，转成 BeanDefinition。 + +2. **注册 BeanDefinition** + + - 把 BeanDefinition 存到 `DefaultListableBeanFactory` 的一个 **Map**： + + ``` + beanDefinitionMap: ConcurrentHashMap + ``` + + - key 是 beanName，value 是 BeanDefinition 对象。 + +3. **实例化 Bean** + + - 容器启动时，会先实例化 **非懒加载的单例 Bean**。 + - 通过反射或 CGLIB 调用构造器创建对象。 + +4. **依赖注入** + + - 遍历 BeanDefinition 中记录的依赖信息（构造器参数、Setter 属性），注入对应 Bean 或值。 + - 支持自动装配（byName / byType / @Autowired）。 + +5. **初始化 & 生命周期** -### 什么是 Spring Beans？ + - 调用初始化方法（`@PostConstruct`、`InitializingBean`、自定义 init 方法）。 + - 应用 `BeanPostProcessor`（AOP 就是在这里织入）。 + + + +### 🎯 什么是 JavaConfig？ + +“JavaConfig 是 Spring 提供的一种基于 Java 类的配置方式，它通过 `@Configuration` 注解标识配置类，通过 `@Bean` 注解声明 Bean。相较于传统的 XML 配置，JavaConfig 提供了类型安全、灵活的配置方式，支持注解和 Java 语言的优势，使得配置更加简洁、可维护，并且与其他框架集成更加方便。” + + + +### 🎯 请举例说明如何在 **Spring** 中注入一个 **Java Collection**? + +Spring 提供了以下四种集合类的配置元素: + +- `` : 该标签用来装配可重复的 list 值。 +- `` : 该标签用来装配没有重复的 set 值。 +- ``: 该标签可用来注入键和值可以为任何类型的键值对。 +- `` : 该标签支持注入键和值都是字符串类型的键值对。 + +```xml + + + + + + + INDIA + Pakistan + USA + UK + + + + + + INDIA + Pakistan + USA + UK + + + + + + + + + + + + + + + admin@nospam.com + support@nospam.com + + + + +``` + + + +### 🎯 什么是 Spring Beans？ - 它们是构成用户应用程序主干的对象 - Bean 由 Spring IoC 容器管理 - 它们由 Spring IoC 容器实例化，配置，装配和管理 - Bean 是基于用户提供给容器的配置元数据创建 -![Bean generation - Spring Interview Questions - Edureka!](https://d1jnx9ba8s6j9r.cloudfront.net/blog/wp-content/uploads/2017/05/bean.png) - -### Spring 提供了哪些配置方式？ +### 🎯 Spring 提供了哪些配置方式？ - 基于 xml 配置 @@ -205,10 +549,8 @@ class Client { Spring 的 Java 配置是通过使用 @Bean 和 @Configuration 来实现。 - 1. @Bean 注解扮演与 ` ` 元素相同的角色。 - 2. @Configuration 类允许通过简单地调用同一个类中的其他 @Bean 方法来定义 bean 间依赖关系。 - - 例如： + 1. @Bean 注解扮演与 `` 元素相同的角色。 + 2. @Configuration 类允许通过简单地调用同一个类中的其他 @Bean 方法来定义 bean 间依赖关系。 ```java @Configuration @@ -220,467 +562,1029 @@ class Client { } ``` - - -### Spring Bean的作用域？ -- 在 Spring 中, 可以在 \ 元素的 scope属性里设置 Bean 的作用域。 -- 默认情况下，Spring 只为每个在 IOC 容器里声明的 Bean 创建唯一一个实例, 整个 IOC 容器范围内都能共享该实例：所有后续的 `getBean()` 调用和 Bean 引用都将返回这个唯一的 Bean 实例。该作用域被称为 **singleton**，它是所有 Bean 的默认作用域。 -Spring容器中的bean可以分为5个范围。所有范围的名称都是自说明的，但是为了避免混淆，还是让我们来解释一下： +### 🎯 通过注解的方式配置bean | 什么是基于注解的容器配置 -1. **singleton**：这种bean范围是默认的，这种范围确保不管接受到多少个请求，每个容器中只有一个bean的实例，单例的模式由bean factory自身来维护。 -2. **prototype**：原型范围与单例范围相反，为每一个bean请求提供一个实例。 -3. **request**：每次HTTP请求都会创建一个新的bean，该作用于仅适用于WebApplicationContext环境，在请求完成以后，bean会失效并被垃圾回收器回收。 -4. **Session**：同一个HTTP Session 共享一个bean，不同的 HTTP Session使用不同的bean。该作用于仅适用于WebApplicationContext环境，在session过期后，bean会随之失效。 -5. **global-session**：全局session作用域，仅仅在基于portlet的web应用中才有意义，Spring5已经没有了。Portlet是能够生成语义代码(例如：HTML)片段的小型Java Web插件。它们基于portlet容器，可以像servlet一样处理HTTP请求。但是，与 servlet 不同，每个 portlet 都有不同的会话 +**组件扫描**(component scanning): Spring 能够从 classpath下自动扫描, 侦测和实例化具有特定注解的组件。 -全局作用域与Servlet中的session作用域效果相同。 +特定组件包括: +- **@Component**：基本注解，标识了一个受 Spring 管理的组件 +- **@Respository**：标识持久层组件 +- **@Service**：标识服务层(业务层)组件 +- **@Controller**： 标识表现层组件 +对于扫描到的组件，Spring 有默认的命名策略：使用非限定类名，第一个字母小写。也可以在注解中通过 value 属性值标识组件的名称。 -### Spring bean 容器的生命周期是什么样的？ +当在组件类上使用了特定的注解之后，还需要在 Spring 的配置文件中声明 ``： -Spring IOC 容器可以管理 Bean 的生命周期, Spring 允许在 Bean 生命周期的特定点执行定制的任务. +- `base-package` 属性指定一个需要扫描的基类包，Spring 容器将会扫描这个基类包里及其子包中的所有类 -Spring bean 容器的生命周期流程如下： +- 当需要扫描多个包时, 可以使用逗号分隔 -1. Spring 容器根据配置中的 bean 定义实例化 bean； -2. Spring 使用依赖注入填充所有属性，如 bean 中所定义的配置； -3. 如果 bean 实现 BeanNameAware 接口，则工厂通过传递 bean 的 ID 来调用 setBeanName()； -4. 如果 bean 实现 BeanFactoryAware 接口，工厂通过传递自身的实例来调用 setBeanFactory()； -5. 与上面的类似，如果实现了其他 `*.Aware`接口，就调用相应的方法； -6. 如果存在与 bean 关联的任何 BeanPostProcessors，则调用 preProcessBeforeInitialization() 方法； -7. 如果为 bean 指定了 init 方法（`` 的 init-method 属性），那么将调用它； -8. 最后，如果存在与 bean 关联的任何 BeanPostProcessors，则将调用 postProcessAfterInitialization() 方法； -9. 如果 bean 实现 DisposableBean 接口，当 spring 容器关闭时，会调用 destory()； -10. 如果为 bean 指定了 destroy 方法（`` 的 destroy-method 属性），那么将调用它 +- 如果仅希望扫描特定的类而非基包下的所有类，可使用 `resource-pattern` 属性过滤特定的类，示例： -![](https://imgkr.cn-bj.ufileos.com/6125ce48-cdfe-4779-9c25-c98088b4cf39.png) + ```xml + + ``` + +### 🎯 Spring Bean的生命周期是怎样的？ -在 bean 初始化时会经历几个阶段，要与容器对 bean 生命周期的管理交互，可以实现 `InitializingBean` 和 `DisposableBean` 接口。容器对前者调用 `afterPropertiesSet()`，对后者调用 `destroy()`，以允许 bean 在初始化和销毁 bean 时执行某些操作。 +“Spring Bean 的生命周期，是指一个 Bean 从被 Spring 容器创建，经过初始化，到最终被销毁的整个过程。以最常见的单例 Bean 为例，其完整生命周期可以概括为以下几个关键阶段： -官方不建议使用这两个接口，而是建议使用 ``@PostConstruct` 和 `@PreDestroy`，或者 XML 配置中使用 `init-method`和`destroy-method` 属性 +1. **实例化（Instantiation）**：Spring 容器通过反射调用 Bean 的构造函数，创建一个 Bean 的实例。 +2. **属性赋值（Populate Properties）**：Spring 容器根据 Bean 定义，为 Bean 的属性注入值，这包括处理`@Autowired`、`@Resource`等依赖注入注解。 +3. **`Aware`接口回调**：如果 Bean 实现了`Aware`系列接口（如`BeanNameAware`, `ApplicationContextAware`），Spring 容器会调用相应的`setXxx()`方法，将容器的相关资源注入给 Bean，让 Bean 能够 “感知” 到容器环境。 +4. **`BeanPostProcessor`前置处理**：容器中所有`BeanPostProcessor`的`postProcessBeforeInitialization`方法被调用。 +5. **初始化（Initialization）**：这是 Bean 准备就绪的关键阶段，执行顺序为： + - 执行`@PostConstruct`注解的方法。 + - 执行`InitializingBean`接口的`afterPropertiesSet`方法。 + - 执行自定义的`init-method`方法。 +6. **`BeanPostProcessor`后置处理**：容器中所有`BeanPostProcessor`的`postProcessAfterInitialization`方法被调用。**AOP 的代理对象就是在这个步骤生成的**。 +7. **Bean 就绪并使用**：此时，Bean 已经完全创建好，并被放入容器中，随时可以被应用程序调用。对于单例 Bean，它会一直存在于容器中，直到容器关闭。 +8. **销毁（Destruction）**：当 Spring 容器关闭时，会销毁所有单例 Bean，执行顺序为： + - 执行`@PreDestroy`注解的方法。 + - 执行`DisposableBean`接口的`destroy`方法。 + - 执行自定义的`destroy-method`方法。 -```xml - -``` +这个生命周期为我们提供了丰富的扩展点，允许我们在 Bean 的不同阶段执行自定义逻辑，例如使用`@PostConstruct`进行资源加载，或利用`BeanPostProcessor`实现 AOP 等高级功能。” ```java -public class ExampleBean { - - public void init() { - // do some initialization work +@Component +public class LifecycleDemo implements BeanNameAware, InitializingBean, DisposableBean { + + private String beanName; + + public LifecycleDemo() { + System.out.println("1. 构造器执行"); } -} -``` - -等价于 - -```java -public class AnotherExampleBean implements InitializingBean { - + + @Autowired + public void setDependency(SomeDependency dependency) { + System.out.println("2. 属性注入"); + } + + @Override + public void setBeanName(String name) { + this.beanName = name; + System.out.println("3. BeanNameAware回调：" + name); + } + + @PostConstruct + public void postConstruct() { + System.out.println("4. @PostConstruct执行"); + } + + @Override public void afterPropertiesSet() { - // do some initialization work + System.out.println("5. InitializingBean.afterPropertiesSet()"); + } + + @PreDestroy + public void preDestroy() { + System.out.println("6. @PreDestroy执行"); + } + + @Override + public void destroy() { + System.out.println("7. DisposableBean.destroy()"); } } ``` -> Spring Bean生命周期回调——初始化回调和销毁回调方法 - -实现 Bean 初始化回调和销毁回调各有三种方法，一是实现接口方法，二是在XML配置，三是使用注解 -- 使用注解 ``@PostConstruct` 和 `@PreDestroy` -- 实现 `InitializingBean` 和 `DisposableBean` 接口 -- XML 中配置 ``init-method` 和 `destroy-method` -在一个 bean 中，如果配置了多种生命周期回调机制，会按照上边从上到下的次序调用 +### 🎯 什么是 Spring 装配? +在 Spring 框架中，**装配（Wiring）**是指将一个对象的依赖关系注入到另一个对象中的过程。通过装配，Spring 容器能够自动管理对象之间的依赖关系，从而减少了应用程序中显式地创建和管理对象的代码。装配是 Spring IoC（控制反转）容器的核心概念之一，它使得 Spring 应用能够轻松地将不同的组件连接在一起，形成完整的应用程序。 +**Spring 装配的类型** -### 在 Spring 中如何配置 Bean? +Spring 提供了几种不同的方式来装配 Bean，主要包括以下几种： -Bean 的配置方式: 通过全类名 （反射）、 通过工厂方法 （静态工厂方法 & 实例工厂方法）、FactoryBean +1. **构造器注入（Constructor Injection）** +2. **Setter 注入（Setter Injection）** +3. **字段注入（Field Injection）** +4. **自动装配（Autowiring）** +5. **基于 XML 配置的装配** +> 依赖注入的本质就是装配，装配是依赖注入的具体行为。 -### 什么是 Spring 装配 -当 bean 在 Spring 容器中组合在一起时，它被称为装配或 bean 装配，装配是创建应用对象之间协作关系的行为。 Spring 容器需要知道需要什么 bean 以及容器应该如何使用依赖注入来将 bean 绑定在一起，同时装配 bean。 +### 🎯 什么是bean自动装配？ -依赖注入的本质就是装配，装配是依赖注入的具体行为。 +**Bean 自动装配（Bean Autowiring）** 是 Spring 框架中的一项重要功能，用于自动满足一个对象对其他对象的依赖。通过自动装配，Spring 容器能够根据配置的规则，将所需的依赖对象自动注入到目标 Bean 中，而无需手动显式定义依赖关系。这种机制极大地简化了依赖注入的过程，使代码更加简洁和易于维护。 -注入是实例化的过程，将创建的bean放在Spring容器中，分为属性注入（setter方式）、构造器注入 +在Spring框架有多种自动装配，让我们逐一分析 +1. **no**：这是Spring框架的默认设置，在该设置下自动装配是关闭的，开发者需要自行在beanautowire属性里指定自动装配的模式 +2. **byName**：**按名称自动装配（结合 `@Qualifier` 注解）**如果容器中有多个相同类型的 Bean，可以使用 `@Qualifier` 注解结合 `@Autowired` 来按名称指定具体的 Bean。 -### 什么是bean自动装配？ +3. **byType**：按类型自动装配 (`@Autowired` 默认方式) -Spring 容器可以自动配置相互协作 beans 之间的关联关系。这意味着 Spring 可以自动配置一个 bean 和其他协作bean 之间的关系，通过检查 BeanFactory 的内容里有没有使用< property>元素。 +4. **constructor**：通过在构造器上添加 `@Autowired` 注解，Spring 会根据构造器参数的类型自动注入对应的 Bean。这种方式可以确保在对象创建时，所有依赖项都已完全注入。 -在Spring框架中共有5种自动装配，让我们逐一分析 +5. **autodetect**：Spring首先尝试通过 *constructor* 使用自动装配来连接，如果它不执行，Spring 尝试通过 *byType* 来自动装配【Spring 4.x 中已经被废弃】 -1. **no：**这是Spring框架的默认设置，在该设置下自动装配是关闭的，开发者需要自行在beanautowire属性里指定自动装配的模式 +在自动装配时，Spring 会检查容器中的所有 Bean，并根据规则选择一个合适的 Bean 来满足依赖。如果找不到匹配的 Bean 或找到多个候选 Bean，可能会抛出异常。 -2. **byName：**该选项可以根据bean名称设置依赖关系。当向一个bean中自动装配一个属性时，容器将根据bean的名称自动在在配置文件中查询一个匹配的bean。如果找到的话，就装配这个属性，如果没找到的话就报错。 -3. **byType：**该选项可以根据bean类型设置依赖关系。当向一个bean中自动装配一个属性时，容器将根据bean的类型自动在在配置文件中查询一个匹配的bean。如果找到的话，就装配这个属性，如果没找到的话就报错。 -4. **constructor：**构造器的自动装配和byType模式类似，但是仅仅适用于与有构造器相同参数的bean，如果在容器中没有找到与构造器参数类型一致的bean，那么将会抛出异常。 - -5. **autodetect：**Spring首先尝试通过 *constructor* 使用自动装配来连接，如果它不执行，Spring 尝试通过 *byType* 来自动装配 - - - -### 自动装配有什么局限？ +### 🎯 自动装配有什么局限？ - 基本数据类型的值、字符串字面量、类字面量无法使用自动装配来注入。 - 装配依赖中若是出现匹配到多个bean（出现歧义性），装配将会失败 -### 通过注解的方式配置bean | 什么是基于注解的容器配置 - -**组件扫描**(component scanning): Spring 能够从 classpath下自动扫描, 侦测和实例化具有特定注解的组件。 - -特定组件包括: +### 🎯 Spring Bean的作用域有哪些？如何选择？ -- **@Component**：基本注解, 标识了一个受 Spring 管理的组件 -- **@Respository**：标识持久层组件 -- **@Service**：标识服务层(业务层)组件 -- **@Controller**： 标识表现层组件 +"Spring支持多种Bean作用域，用于控制Bean的创建策略和生命周期： -![annotations - Spring Framework Tutorial - Edureka!](https://d1jnx9ba8s6j9r.cloudfront.net/blog/wp-content/uploads/2017/05/annotations.png) +**核心作用域**： -对于扫描到的组件,，Spring 有默认的命名策略：使用非限定类名,，第一个字母小写。也可以在注解中通过 value 属性值标识组件的名称。 +**1. singleton（单例，默认）**： +- 整个Spring容器中只有一个Bean实例 +- 线程不安全，需注意并发访问 +- 适用于无状态的服务层组件 -当在组件类上使用了特定的注解之后,，还需要在 Spring 的配置文件中声明 ``： +**2. prototype（原型）**： +- 每次getBean()都创建新实例 +- Spring不管理prototype Bean的完整生命周期 +- 适用于有状态的Bean -- `base-package` 属性指定一个需要扫描的基类包，Spring 容器将会扫描这个基类包里及其子包中的所有类 +**Web环境作用域**： -- 当需要扫描多个包时, 可以使用逗号分隔 +**3. request（请求作用域）**： +- 每个HTTP请求创建一个Bean实例 +- 请求结束后实例被销毁 +- 用于存储请求相关数据 -- 如果仅希望扫描特定的类而非基包下的所有类，可使用 `resource-pattern` 属性过滤特定的类，示例： +**4. session（会话作用域）**： +- 每个HTTP Session创建一个实例 +- Session失效后实例被销毁 +- 用于存储用户会话数据 - ```xml - - ``` +**5. application（应用作用域）**： +- 整个Web应用只有一个实例 +- 绑定到ServletContext生命周期 - +**选择原则**： +- 无状态Bean → singleton +- 有状态Bean → prototype +- Web数据 → request/session +- 全局共享 → application" -### 如何在 spring 中启动注解装配？ +**💻 代码示例**： +```java +// 单例Bean（默认） +@Component +public class SingletonService { + private int counter = 0; // 线程不安全！ + + public void increment() { + counter++; + } +} -默认情况下，Spring 容器中未打开注解装配。因此，要使用基于注解装配，我们必须通过配置`` 元素在 Spring 配置文件中启用它。 +// 原型Bean +@Component +@Scope("prototype") +public class PrototypeBean { + private int counter = 0; // 每个实例独立 +} +// 请求作用域 +@Component +@RequestScope +public class RequestBean { + private String requestId = UUID.randomUUID().toString(); +} +// 会话作用域 +@Component +@SessionScope +public class UserSession { + private String userId; + private Map attributes = new HashMap<>(); +} +``` -## 四、AOP +### 🎯 Spring 框架中的单例 **Beans** 是线程安全的么? ->👴：描述一下Spring AOP 呗？ -> ->​ 你有没有⽤过Spring的AOP? 是⽤来⼲嘛的? ⼤概会怎么使⽤？ -> +> Spring 容器中的Bean是否线程安全，容器本身并没有提供Bean的线程安全策略，因此可以说Spring容器中的Bean本身不具备线程安全的特性，但是具体还是要结合具体scope的Bean去研究。 > +> 线程安全这个问题，要从单例与原型Bean分别进行说明。 > +> **「原型Bean」**对于原型Bean,每次创建一个新对象，也就是线程之间并不存在Bean共享，自然是不会有线程安全的问题。 > +> **「单例Bean」**对于单例Bean,所有线程都共享一个单例实例Bean,因此是存在资源的竞争。 -### 什么是 AOP？ +在 Spring 框架中，单例（**Singleton**）Beans 默认是**线程不安全的**，这取决于 Bean 的内部状态以及是否对其进行了适当的同步和管理。具体来说，Spring 的 **单例作用域** 表示容器只会创建该 Bean 的单一实例并共享，但它并没有自动保证 Bean 实例本身的线程安全性。 -AOP(Aspect-Oriented Programming，面向切面编程)：是一种新的方法论，是对传统 OOP(Object-Oriented Programming，面向对象编程) 的补充。在 OOP 中, 我们以类(class)作为我们的基本单元，而 AOP 中的基本单元是 **Aspect(切面)** +1. 单例 Bean 线程安全问题的原因 -AOP 的主要编程对象是切面(aspect) + - **单例模式**意味着 Spring 容器会在应用启动时创建该 Bean 的唯一实例，并且整个应用程序生命周期内都会使用这个实例。因此，如果该 Bean 被多个线程共享并且内部状态是可变的（即 Bean 的属性值发生改变），则必须小心处理，以避免线程安全问题。 -在应用 AOP 编程时, 仍然需要定义公共功能，但可以明确的定义这个功能在哪里,，以什么方式应用,，并且不必修改受影响的类。这样一来横切关注点就被模块化到特殊的对象(切面)里。 + - **线程不安全的实例**：如果单例 Bean 中的字段是可变的且没有正确同步，那么多个线程访问该 Bean 时，可能会出现竞态条件、脏读、写冲突等问题。 -AOP 的好处: +2. 单例 Bean 线程安全的几种情况 -- 每个事物逻辑位于一个位置，代码不分散，便于维护和升级 -- 业务模块更简洁, 只包含核心业务代码 +- 无状态的单例 Bean（线程安全） -![](https://d1jnx9ba8s6j9r.cloudfront.net/blog/wp-content/uploads/2017/05/unnamed.png) + 如果单例 Bean 没有任何可变的成员变量，或者所有成员变量都是不可变的（例如 `final` 类型或 `@Value` 注入的常量），则它是线程安全的，因为不同线程在访问该 Bean 时不会修改其状态。 + ```java + @Component + public class MyService { + public String greet(String name) { + return "Hello, " + name; + } + } + ``` + 在这个例子中，`MyService` 是无状态的，方法内部没有任何成员变量，因此多个线程可以同时调用该方法，而不会出现线程安全问题。 -### **AOP 术语** +- 有状态的单例 Bean（非线程安全） -- 切面（Aspect）：横切关注点（跨越应用程序多个模块的功能），被模块化的特殊对象 -- 连接点（Joinpoint）：程序执行的某个特定位置，如类某个方法调用前、调用后、方法抛出异常后等。在这个位置我们可以插入一个 AOP 切面，它实际上是应用程序执行 Spring AOP 的位置 + 如果单例 Bean 的某些字段是可变的，或者它们会随着方法调用而变化（例如，实例变量依赖于请求参数或其他外部因素），那么它可能会变得线程不安全。 -![](https://d1jnx9ba8s6j9r.cloudfront.net/blog/wp-content/uploads/2017/05/JoinPoint-1.png) + ```java + @Component + public class CounterService { + private int counter = 0; + + public void increment() { + counter++; // 非线程安全操作 + } + + public int getCounter() { + return counter; + } + } + ``` -- 通知（Advice）： 通知是个在方法执行前或执行后要做的动作，实际上是程序执行时要通过 SpringAOP 框架触发的代码段。Spring 切面可以应用五种类型的通知： - - before： 前置通知 ， 在一个方法执行前被调用 - - after：在方法执行之后调用的通知，无论方式执行是否成功 - - after-returning：仅当方法成功完成后执行的通知 - - after-throwing：在方法抛出异常退出时执行的通知 - - around：在方法执行之前和之后调用的通知 + 在这个例子中，`CounterService` 是有状态的，因为 `counter` 字段的值会根据 `increment()` 方法的调用而变化。如果多个线程同时调用 `increment()` 方法，可能会导致竞态条件（race condition），从而导致线程安全问题。 -![advice - Spring Framework Interview Questions - Edureka!](https://d1jnx9ba8s6j9r.cloudfront.net/blog/wp-content/uploads/2017/05/advice-2.png) +- 有状态的单例 Bean 的线程安全处理 -- 目标（Target）：被通知的对象，通常是一个代理对象，也指被通知（advice）对象 -- 代理（Proxy）：向目标对象应用通知之后创建的对象 -- 切点（pointcut）：每个类都拥有多个连接点，程序运行中的一些时间点，例如一个方法的执行，或者是一个异常的处理。AOP 通过切点定位到特定的连接点。类比：连接点相当于数据库中的记录，切点相当于查询条件。切点和连接点不是一对一的关系，一个切点匹配多个连接点，切点通过 `org.springframework.aop.Pointcut` 接口进行描述，它使用类和方法作为连接点的查询条件 -- 引入（Introduction）：引入允许我们向现有的类添加新方法或属性 -- 织入（Weaving）：织入是把切面应用到目标对象并创建新的代理对象的过程 + 如果你的单例 Bean 是有状态的，且你需要在多线程环境中使用，可以自己来确保线程安全：比如同步方法、原子类等 +> - **无状态的单例 Bean**：如果单例 Bean 没有可变状态（即没有实例字段或者所有字段都是 `final` 的），那么它是线程安全的。 +> - **有状态的单例 Bean**：如果单例 Bean 的字段是可变的，且在多线程环境中可能会被同时访问，默认情况下它是**线程不安全的**。这种情况下，必须通过同步、原子类、或者 `ThreadLocal` 等技术来确保线程安全。 +> - **Spring 不会自动为单例 Bean 提供线程安全机制**，开发者需要根据实际情况来保证线程安全性。 +> “Spring 框架中的单例 Bean 默认情况下并不保证线程安全性。单例模式意味着同一个 Bean 实例会被多个线程共享，因此如果 Bean 有可变的状态（例如成员变量会在方法调用中发生变化），就可能导致线程安全问题。为了确保线程安全，我们可以使用同步机制、原子类或者 `ThreadLocal` 来保证在多线程环境中的安全访问。如果 Bean 是无状态的，那么它就是线程安全的。” -**Spring AOP** -- **AspectJ：**Java 社区里最完整最流行的 AOP 框架 -- 在 Spring2.0 以上版本中, 可以使用基于 AspectJ 注解或基于 XML 配置的 AOP -**在 Spring 中启用 AspectJ 注解支持** +### 🎯 什么是循环依赖？Spring如何解决？ -- 要在 Spring 应用中使用 AspectJ 注解, 必须在 classpath 下包含 AspectJ 类库：`aopalliance.jar`、`aspectj.weaver.jar` 和 `spring-aspects.jar` -- 将 aop Schema 添加到 `` 根元素中. -- 要在 Spring IOC 容器中启用 AspectJ 注解支持, 只要在 Bean 配置文件中定义一个空的 XML 元素 `` -- 当 Spring IOC 容器侦测到 Bean 配置文件中的` ` 元素时, 会自动为与 AspectJ切面匹配的 Bean 创建代理. +> 循环依赖是指 Bean 之间的相互依赖导致的创建死循环。Spring 通过 **三级缓存** 提前暴露半成品 Bean 的引用来解决单例 Bean 的循环依赖，支持 setter/field 注入。但构造器注入和 prototype Bean 的循环依赖无法解决，会报错。 +**循环依赖**： 两个或多个 Bean 之间相互依赖，形成一个环。 +- 例如：`A` 依赖 `B`, `B` 又依赖 `A` -### 有哪写类型的通知（Advice） | 用 AspectJ 注解声明切面 +如果 Spring 不处理，就会在 Bean 创建过程中死循环，导致启动失败。 -- 要在 Spring 中声明 AspectJ切面, 只需要在 IOC 容器中将切面声明为 Bean 实例. 当在 Spring IOC 容器中初始化 AspectJ切面之后, Spring IOC 容器就会为那些与 AspectJ切面相匹配的 Bean 创建代理. -- 在 AspectJ注解中, 切面只是一个带有 @Aspect 注解的 Java 类. -- 通知是标注有某种注解的简单的 Java 方法. -- AspectJ支持 5 种类型的通知注解: +------ -- - @Before: 前置通知, 在方法执行之前执行 - - @After: 后置通知, 在方法执行之后执行 - - @AfterRunning: 返回通知, 在方法返回结果之后执行 - - @AfterThrowing: 异常通知, 在方法抛出异常之后 - - @Around: 环绕通知, 围绕着方法执行 +**Spring 解决循环依赖的机制** +Spring 默认支持 **单例 Bean 的循环依赖**（构造器注入除外）。核心依赖 **三级缓存机制**： +1. **singletonObjects**（一级缓存）：完成初始化的单例对象的 cache，这里的 bean 经历过 `实例化->属性填充->初始化` 以及各种后置处理 +2. **earlySingletonObjects**（二级缓存）：存放提前曝光的“半成品” Bean（**完成实例化但是尚未填充属性和初始化**），仅仅能作为指针提前曝光，被其他 bean 所引用，用于解决循环依赖的 +3. **singletonFactories**（三级缓存）：存放对象工厂（主要用于 AOP 代理提前暴露） -### AOP 有哪些实现方式？ +**解决流程：** -实现 AOP 的技术，主要分为两大类： +1. Spring 创建 Bean `A` → 实例化 `A`，但属性未注入 + - 把 `A` 的 **工厂对象（ObjectFactory）** 放入 **三级缓存** +2. `A` 需要注入 `B` → 创建 `B` +3. `B` 又需要 `A` → 从三级缓存找到 `A` 的工厂，拿到 `A` 的引用（可能是代理对象），放入二级缓存 +4. `B` 完成创建 → 注入到 `A` +5. `A` 完成属性注入 → 移到一级缓存，删除二三级缓存中的引用 -- 静态代理 - 指使用 AOP 框架提供的命令进行编译，从而在编译阶段就可生成 AOP 代理类，因此也称为编译时增强； - - 编译时编织（特殊编译器实现） - - 类加载时编织（特殊的类加载器实现）。 -- 动态代理 - 在运行时在内存中“临时”生成 AOP 动态代理类，因此也被称为运行时增强。 - - JDK 动态代理 - - CGLIB +------ +**注意点** +- **支持的情况**： + - 单例 Bean + setter 注入 / field 注入（可以先实例化再填充属性） +- **不支持的情况**： + - **构造器注入**循环依赖（因为对象还没实例化就要彼此依赖，没法提前暴露） + - **prototype Bean**（因为原型 Bean 不走单例缓存机制） -### 有哪些不同的AOP实现 +```java +protected Object getSingleton(String beanName, boolean allowEarlyReference) { + // 从 singletonObjects 获取实例，singletonObjects 中的实例都是准备好的 bean 实例，可以直接使用 + Object singletonObject = this.singletonObjects.get(beanName); + //isSingletonCurrentlyInCreation() 判断当前单例bean是否正在创建中 + if (singletonObject == null && isSingletonCurrentlyInCreation(beanName)) { + synchronized (this.singletonObjects) { + // 一级缓存没有，就去二级缓存找 + singletonObject = this.earlySingletonObjects.get(beanName); + if (singletonObject == null && allowEarlyReference) { + // 二级缓存也没有，就去三级缓存找 + ObjectFactory singletonFactory = this.singletonFactories.get(beanName); + if (singletonFactory != null) { + // 三级缓存有的话，就把他移动到二级缓存,.getObject() 后续会讲到 + singletonObject = singletonFactory.getObject(); + this.earlySingletonObjects.put(beanName, singletonObject); + this.singletonFactories.remove(beanName); + } + } + } + } + return singletonObject; +} +``` -![AOP Implementations - Spring Framework Interview Questions - Edureka!](https://d1jnx9ba8s6j9r.cloudfront.net/blog/wp-content/uploads/2017/05/AOP-Implementations.png) +### 🎯 为什么 Spring 要用三级缓存？二级缓存是不是就够了？ -### Spring AOP and AspectJ AOP 有什么区别？ +> 如果只考虑循环依赖，**二级缓存就能解决**； +> +> Spring 使用 **三级缓存** 是为了兼顾 **AOP 代理场景**，确保即使 Bean 被代理，依赖注入的也是同一个最终对象。 -- Spring AOP 基于动态代理方式实现，AspectJ 基于静态代理方式实现。 -- Spring AOP 仅支持方法级别的 PointCut；提供了完全的 AOP 支持，它还支持属性级别的 PointCut。 +**二级缓存能解决吗？** +从“解决循环依赖”的角度看，其实 **二级缓存就够了**。 +- 当 `A` 需要 `B`，`B` 又需要 `A`，Spring 可以把 `A` 的“半成品对象”直接放到二级缓存里（`earlySingletonObjects`），这样 `B` 在创建时就能拿到 `A` 的引用，从而解决循环依赖。 -## 五、数据访问 +------ -### Spring对JDBC的支持 +**为什么需要三级缓存？** -JdbcTemplate简介 +三级缓存的作用是为了 **支持 AOP 代理等场景**。 -- 为了使 JDBC 更加易于使用, Spring 在 JDBC API 上定义了一个抽象层, 以此建立一个 JDBC 存取框架 -- 作为 Spring JDBC 框架的核心， JDBCTemplate 的设计目的是为不同类型的 JDBC 操作提供模板方法。每个模板方法都能控制整个过程，并允许覆盖过程中的特定任务。通过这种方式，可以在尽可能保留灵活性的情况下，将数据库存取的工作量降到最低。 +- 假如 `A` 是一个需要被代理的 Bean（比如加了 `@Transactional`），如果只用二级缓存： + - `B` 注入的是原始的 `A` 对象（还没生成代理） + - 之后 `A` 在 BeanPostProcessor 里生成了代理对象，但 `B` 已经持有了原始对象，最终导致依赖的不是同一个对象（代理失效） +- 所以 Spring 在三级缓存中放的不是对象本身，而是一个 **ObjectFactory**，可以在需要的时候返回真正的对象（原始的或者代理过的）。 + - `getEarlyBeanReference()` 会在 BeanPostProcessor 中执行，把原始对象包装成代理对象。 + - 这样保证了 `A` 和 `B` 拿到的都是最终的代理对象，而不是半成品。 -### Spring 支持哪些 ORM 框架 +--- -Hibernate、iBatis、JPA、JDO、OJB +## 🎯 二、AOP面向切面编程 -## 六、事务 +**核心理念**：将横切关注点从业务逻辑中分离，实现关注点分离，提高代码的模块化程度。 -### Spring 中的事务管理 +### 🎯 什么是AOP？核心概念有哪些？ -作为企业级应用程序框架,，Spring 在不同的事务管理 API 之上定义了一个抽象层，而应用程序开发人员不必了解底层的事务管理 API，就可以使用 Spring 的事务管理机制 +"AOP全称Aspect Oriented Programming，即面向切面编程，是对OOP的补充和扩展： -Spring 既支持**编程式事务管理**，也支持**声明式的事务管理** +**AOP核心思想**： +- 将横切关注点（如日志、事务、安全）从业务逻辑中分离 +- 通过动态代理技术实现方法增强 +- 提高代码的模块化程度和可维护性 -- 编程式事务管理：将事务管理代码嵌入到业务方法中来控制事务的提交和回滚，在编程式管理事务时，必须在每个事务操作中包含额外的事务管理代码，属于硬编码 -- 声明式事务管理：大多数情况下比编程式事务管理更好用。它将事务管理代码从业务方法中分离出来，以声明的方式来实现事务管理。事务管理作为一种横切关注点，可以通过 AOP 方法模块化。Spring 通过 Spring AOP 框架支持声明式事务管理，**声明式事务又分为两种：** - - 基于XML的声明式事务 - - 基于注解的声明式事务 +**核心概念**： +**1. 切面（Aspect）**： +- 横切关注点的模块化封装 +- 包含切点和通知的组合 +- 使用@Aspect注解定义 +**2. 连接点（JoinPoint）**： +- 程序执行过程中能插入切面的点 +- Spring AOP中特指方法执行点 +- 包含方法信息、参数、目标对象等 -### 事务管理器 +**3. 切点（Pointcut）**： -Spring 并不直接管理事务，而是提供了多种事务管理器，他们将事务管理的职责委托给 Hibernate 或者 JTA 等持久化机制所提供的相关平台框架的事务来实现。 +- 匹配连接点的表达式 +- 定义在哪些方法上应用通知 +- 使用AspectJ表达式语言 -Spring 事务管理器的接口是 `org.springframework.transaction.PlatformTransactionManager`，通过这个接口，Spring为各个平台如 JDBC、Hibernate 等都提供了对应的事务管理器，但是具体的实现就是各个平台自己的事情了。 +**4. 通知（Advice）**： +- 在特定连接点执行的代码 +- 包含前置、后置、环绕、异常、最终通知 -#### Spring 中的事务管理器的不同实现 +**5. 目标对象（Target）**： +- 被通知的对象，通常是业务逻辑对象 -**事务管理器以普通的 Bean 形式声明在 Spring IOC 容器中** +**6. 织入（Weaving）**： +- 将切面应用到目标对象创建代理的过程 +- Spring采用运行时织入" -- 在应用程序中只需要处理一个数据源, 而且通过 JDBC 存取 +**💻 代码示例**： +```java +@Aspect +@Component +public class LoggingAspect { + + // 定义切点 + @Pointcut("execution(* com.example.service.*.*(..))") + public void serviceLayer() {} + + @Pointcut("@annotation(com.example.annotation.Log)") + public void logAnnotation() {} + + // 前置通知 + @Before("serviceLayer()") + public void beforeAdvice(JoinPoint joinPoint) { + String methodName = joinPoint.getSignature().getName(); + Object[] args = joinPoint.getArgs(); + System.out.println("方法执行前: " + methodName + ", 参数: " + Arrays.toString(args)); + } + + // 环绕通知 + @Around("logAnnotation()") + public Object aroundAdvice(ProceedingJoinPoint joinPoint) throws Throwable { + long startTime = System.currentTimeMillis(); + Object result = joinPoint.proceed(); + long endTime = System.currentTimeMillis(); + System.out.println("方法执行耗时: " + (endTime - startTime) + "ms"); + return result; + } + + // 异常通知 + @AfterThrowing(pointcut = "serviceLayer()", throwing = "ex") + public void afterThrowingAdvice(JoinPoint joinPoint, Exception ex) { + System.out.println("方法执行异常: " + ex.getMessage()); + } +} +``` - ```java - org.springframework.jdbc.datasource.DataSourceTransactionManager - ``` -- 在 JavaEE 应用服务器上用 JTA(Java Transaction API) 进行事务管理 - ``` - org.springframework.transaction.jta.JtaTransactionManager - ``` +### 🎯 AOP 有哪些实现方式？ -- 用 Hibernate 框架存取数据库 +实现 AOP 的技术，主要分为两大类： - ``` - org.springframework.orm.hibernate3.HibernateTransactionManager - ``` +- 静态代理 - 指使用 AOP 框架提供的命令进行编译，从而在编译阶段就可生成 AOP 代理类，因此也称为编译时增强； + - 编译时编织（特殊编译器实现） + - 类加载时编织（特殊的类加载器实现）。 +- 动态代理 - 在运行时在内存中“临时”生成 AOP 动态代理类，因此也被称为运行时增强。 + - JDK 动态代理 + - CGLIB -**事务管理器以普通的 Bean 形式声明在 Spring IOC 容器中** +### 🎯 Spring AOP 实现原理? -### 用事务通知声明式地管理事务 +Spring AOP 的实现原理基于**动态代理**和**字节码增强**，其核心是通过在运行时生成代理对象，将横切逻辑（如日志、事务）织入目标方法中。以下是其实现原理的详细解析： -- 事务管理是一种横切关注点 -- 为了在 Spring 2.x 中启用声明式事务管理，可以通过 tx Schema 中定义的 元素声明事务通知，为此必须事先将这个 Schema 定义添加到 根元素中去 -- 声明了事务通知后，就需要将它与切入点关联起来。由于事务通知是在 元素外部声明的, 所以它无法直接与切入点产生关联，所以必须在 元素中声明一个增强器通知与切入点关联起来. -- 由于 Spring AOP 是基于代理的方法，所以只能增强公共方法。因此, 只有公有方法才能通过 Spring AOP 进行事务管理。 +> `Spring`的`AOP`实现原理其实很简单，就是通过**动态代理**实现的。如果我们为`Spring`的某个`bean`配置了切面，那么`Spring`在创建这个`bean`的时候，实际上创建的是这个`bean`的一个代理对象，我们后续对`bean`中方法的调用，实际上调用的是代理类重写的代理方法。而`Spring`的`AOP`使用了两种动态代理，分别是**JDK的动态代理**，以及**CGLib的动态代理**。 -![](https://imgkr.cn-bj.ufileos.com/8342e671-cd06-4ccc-b206-51a355780cea.png) +### 🎯 Spring AOP的实现原理是什么？ +"Spring AOP基于动态代理技术实现，根据目标对象的不同采用不同的代理策略： +一、**核心实现机制：动态代理** -### 用 @Transactional 注解声明式地管理事务 +Spring AOP 通过两种动态代理技术实现切面逻辑的织入： -- 除了在带有切入点，通知和增强器的 Bean 配置文件中声明事务外，Spring 还允许简单地用 @Transactional 注解来标注事务方法 -- 为了将方法定义为支持事务处理的，可以为方法添加 @Transactional 注解，根据 Spring AOP 基于代理机制，**只能标注公有方法.** -- 可以在方法或者类级别上添加 @Transactional 注解。当把这个注解应用到类上时， 这个类中的所有公共方法都会被定义成支持事务处理的 -- 在 Bean 配置文件中只需要启用 ``元素, 并为之指定事务管理器就可以了 -- 如果事务处理器的名称是 transactionManager, 就可以在 `` 元素中省略 `transaction-manager` 属性，这个元素会自动检测该名称的事务处理器 +1. **JDK 动态代理** -![](https://imgkr.cn-bj.ufileos.com/800f0b12-550d-49b9-b5a3-82fd645c51e9.png) + - **适用条件**：目标对象实现了至少一个接口。 + - **适用条件**：目标对象实现了接口 + - **实现原理**：基于 `java.lang.reflect.Proxy` 类生成代理对象，代理类实现目标接口并重写方法。 -### 事务传播属性 + - 关键源码： -- 当事务方法被另一个事务方法调用时， 必须指定事务应该如何传播。例如：方法可能继续在现有事务中运行，也可能开启一个新事务，并在自己的事务中运行 -- 事务的传播行为可以由传播属性指定，Spring 定义了 7 种类传播行为： + ```java + Proxy.newProxyInstance(ClassLoader, interfaces, InvocationHandler); + ``` -![](https://imgkr.cn-bj.ufileos.com/910901d5-de7e-4d06-8825-39e8ae051060.png) + 在 `InvocationHandler#invoke()` 方法中拦截目标方法，执行切面逻辑（如前置通知、后置通知）。 + - **代理方式**：生成接口的实现类作为代理 + - **优点**：JDK内置，无需额外依赖 -### Spring 支持的事务隔离级别 + - **缺点**：只能代理接口方法 -![](https://imgkr.cn-bj.ufileos.com/a4fb6d55-d32b-41d8-9a98-e6873970196d.png) +2. **CGLIB 动态代理** -事务的隔离级别要得到底层数据库引擎的支持，而不是应用程序或者框架的支持； + - **适用条件**：目标对象未实现接口。 -Oracle 支持的 2 种事务隔离级别，Mysql支持 4 种事务隔离级别。 + - **实现原理**：基于ASM字节码操作，通过继承目标类生成子类代理，覆盖父类方法并插入切面逻辑。 + - 关键源码： + ```java + Enhancer enhancer = new Enhancer(); + enhancer.setSuperclass(targetClass); + enhancer.setCallback(MethodInterceptor); + ``` -### 设置隔离事务属性 + 在 `MethodInterceptor#intercept()` 方法中实现方法拦截 -用 @Transactional 注解声明式地管理事务时可以在 @Transactional 的 isolation 属性中设置隔离级别 + - **代理方式**：生成目标类的子类作为代理 -在 Spring 事务通知中, 可以在 `` 元素中指定隔离级别 + - **优点**：可以代理普通类 -### 设置回滚事务属性 + - **缺点**：无法代理final类和方法 -- 默认情况下只有未检查异常(RuntimeException和Error类型的异常)会导致事务回滚，而受检查异常不会。 -- 事务的回滚规则可以通过 @Transactional 注解的 rollbackFor和 noRollbackFor属性来定义，这两个属性被声明为 Class[] 类型的，因此可以为这两个属性指定多个异常类。 +代理创建流程： -- - rollbackFor：遇到时必须进行回滚 - - noRollbackFor： 一组异常类，遇到时必须不回滚 +1. Spring检查目标对象是否实现接口 +2. 有接口→JDK动态代理，无接口→CGLIB代理 +3. 创建代理对象，织入切面逻辑 +4. 返回代理对象供客户端使用 -### 超时和只读属性 +**方法调用流程**： -- 由于事务可以在行和表上获得锁， 因此长事务会占用资源, 并对整体性能产生影响 -- 如果一个事物只读取数据但不做修改，数据库引擎可以对这个事务进行优化 -- 超时事务属性：事务在强制回滚之前可以保持多久，这样可以防止长期运行的事务占用资源 -- 只读事务属性：表示这个事务只读取数据但不更新数据，这样可以帮助数据库引擎优化事务 +1. 客户端调用代理对象方法 +2. 代理拦截方法调用 +3. 执行前置通知 +4. 调用目标对象方法 +5. 执行后置通知 +6. 返回结果给客户端 -**设置超时和只读事务属性** +**强制使用CGLIB**： -- 超时和只读属性可以在 @Transactional 注解中定义，超时属性以秒为单位来计算 +- @EnableAspectJAutoProxy(proxyTargetClass=true) +- 或配置spring.aop.proxy-target-class=true" -列出两种方式的示例： +**💻 代码示例**： ```java -@Transactional(propagation = Propagation.NESTED, timeout = 1000, isolation = Isolation.READ_COMMITTED, rollbackFor = Exception.class) -``` +// 目标接口和实现（会使用JDK动态代理） +public interface UserService { + void saveUser(String username); +} -```xml - - - - - - - - - +@Service +public class UserServiceImpl implements UserService { + @Override + public void saveUser(String username) { + System.out.println("保存用户: " + username); + } +} - - - - - +// 无接口的类（会使用CGLIB代理） +@Service +public class OrderService { + public void createOrder(String orderId) { + System.out.println("创建订单: " + orderId); + } +} + +// 强制使用CGLIB +@Configuration +@EnableAspectJAutoProxy(proxyTargetClass = true) +public class AopConfig { +} + +// 代理工厂使用示例 +public class ProxyFactoryDemo { + public void testProxy() { + ProxyFactory factory = new ProxyFactory(); + factory.setTarget(new UserServiceImpl()); + factory.addAdvice(new MethodInterceptor() { + @Override + public Object invoke(MethodInvocation invocation) throws Throwable { + System.out.println("方法调用前"); + Object result = invocation.proceed(); + System.out.println("方法调用后"); + return result; + } + }); + + UserService proxy = (UserService) factory.getProxy(); + proxy.saveUser("张三"); + } +} +``` + +### 🎯 JDK 动态代理和 CGLIB 的区别？ + +| **特性** | **JDK 动态代理** | **CGLIB 动态代理** | +| ------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | +| **底层技术** | 基于 Java 反射机制，通过 `Proxy` 类和 `InvocationHandler` 接口生成代理类。 | 基于 **ASM 字节码框架**，通过生成目标类的子类实现代理。 | +| **代理方式** | 只能代理实现了接口的类，生成接口的代理类。 | 可代理普通类（无需接口），生成目标类的子类覆盖方法。 | +| **性能** | 方法调用通过反射实现，性能较低。 | 通过 **FastClass 机制** 直接调用方法，性能更高（以空间换时间）。 | +| **代码生成** | 运行时动态生成代理类的字节码。 | 生成目标类的子类字节码，修改原始类结构。 | +| **适用场景** | 代理接口实现类，适用于轻量级应用。 | 代理无接口的类，适用于高性能需求场景（如 Spring AOP）。 | +| **优点** | 1. 无需第三方库依赖；2. 代码简单易用。 | 1. 支持代理普通类和 final 方法；2. 性能更高（FastClass 机制）。 | +| **缺点** | 1. 只能代理接口；2. 反射调用性能较低。 | 1. 生成代理类耗时较长；2. 可能破坏类封装性（如代理 final 方法需特殊处理）。 | + +### 🎯 代理是在什么时候生成的？ + +> Spring 的代理对象在 **Bean 初始化完成之后（即初始化阶段的后置处理环节）** 由 `BeanPostProcessor` 生成。 +> +> Spring 的代理对象不是在 Bean 实例化时创建的，而是在 **初始化完成后**，由 AOP 的后置处理器（`AnnotationAwareAspectJAutoProxyCreator`）在 `postProcessAfterInitialization()` 阶段通过 JDK 动态代理或 CGLIB 生成的。 +> +> 它会判断 Bean 是否匹配切面规则，如果匹配，就创建代理对象并放入容器，从此容器中拿到的都是这个代理对象。 + +### 🎯 为什么同类方法调用时 AOP 不生效？ + +简单来说，当一个 Bean 的方法 A 内部直接调用了同一个 Bean 的另一个方法 B 时，如果方法 B 上有 AOP 切面（比如`@Transactional`、`@Log`），那么这个切面在这次内部调用中**不会生效**。 + +```java +@Service +public class UserService { + + // 方法A + public void createUser(User user) { + // ... 一些逻辑 ... + // 内部调用方法B + this.updateUserStatus(user.getId(), "ACTIVE"); + // 注意：这里的this指的是原始的目标对象，而非代理对象 + } + + // 方法B上有AOP切面（比如事务） + @Transactional + public void updateUserStatus(Long userId, String status) { + // ... 更新数据库 ... + } +} +``` + +“同类方法调用 AOP 失效，根本原因在于 Spring AOP 的**代理机制**。 + +简单来说，当一个 Bean 的方法 A 内部调用方法 B 时，它使用的是`this`指针，这个`this`指向的是**原始的目标对象**，而不是 Spring 为它创建的**代理对象**。 + +由于 AOP 的切面逻辑（比如`@Transactional`）是织入在**代理对象**中的，所以绕过了代理，切面自然就不会生效。 + +| 解决方案 | 核心思路 | 优点 | 缺点 | +| ----------------------- | ------------------------------------------------------------ | --------------------------------------------- | ------------------------------------------------- | +| **1. 重构代码（推荐）** | 将需要切面的逻辑拆分到另一个独立的 Bean 中，通过依赖注入调用。 | **代码清晰，解耦，符合 OOP 原则，无副作用。** | 可能需要较大的代码调整。 | +| **2. 自注入代理** | 在同一个 Bean 中，注入自身的代理对象（通过`ApplicationContext`或`@Autowired`），然后用注入的对象调用方法。 | 无需修改方法签名，能快速解决问题。 | 与 Spring 框架耦合度高，可读性稍差。 | +| **3. 使用`AopContext`** | 开启`exposeProxy = true`，然后在方法内通过`AopContext.currentProxy()`获取代理对象进行调用。 | Spring 官方解决方案，代码侵入性中等。 | 依赖`ThreadLocal`，在异步场景下有风险，可读性差。 | + + + +### 🎯 为什么@Transactional有时不生效？ + +`@Transactional`注解的生效，依赖于 Spring 的**AOP 代理机制**和**事务管理器（TransactionManager）**。 + +1. **AOP 代理**：当一个类被`@Transactional`注解修饰时，Spring 容器会为它创建一个**代理对象**。这个代理对象会拦截所有被`@Transactional`标记的方法。 +2. **拦截与事务管理**：当外部调用该方法时，调用请求会先到达代理对象。代理对象会在方法执行**前**，通过`TransactionManager`开启一个事务；在方法执行**后**，根据是否发生异常来决定提交（commit）或回滚（rollback）事务。 + +我们可以推导出所有失效场景的根源：**调用流程没有经过 Spring 的代理对象，或者代理对象无法正确地管理事务。** + +**`@Transactional`失效场景与解决方案全解析** + +| 失效场景 | 根本原因 | 代码示例 | 解决方案 | +| --------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | +| **1. 同类方法内部调用** | `this`指向原始目标对象，调用绕过了 Spring 代理，导致切面无法生效。 | `java @Service public class UserService { public void createUser(User user) { // 内部调用，@Transactional失效 this.updateStatus(user.getId(), "NEW"); } @Transactional public void updateStatus(Long userId, String status) { /* ... */ } }` | **最佳实践**：将`updateStatus`方法拆分到另一个`@Service`中（如`StatusService`），在`UserService`中注入`StatusService`并调用。**备选方案**：通过`ApplicationContext`或`AopContext`获取自身的代理对象再调用。 | +| **2. 方法非`public`** | Spring AOP 代理（JDK/CGLIB）默认只拦截`public`方法。非`public`方法无法被代理，切面无法织入。 | `java @Service public class UserService { // private方法，@Transactional失效 @Transactional private void updateStatus(Long userId, String status) { /* ... */ } }` | 将方法的访问权限修改为`public`。 | +| **3. 类未被 Spring 管理** | Spring 事务依赖 IOC 容器。未被容器管理的对象，Spring 无法为其创建代理。 | `java // 错误：忘记加@Service注解 public class UserService { @Transactional public void updateStatus(...) { /* ... */ } } // 或在代码中手动new UserService()` | 确保类上添加了`@Service`、`@Component`等注解，并通过 Spring 容器（如`@Autowired`）获取 Bean，而非手动`new`。 | +| **4. 异常被 “吃了”** | Spring 默认仅对未捕获的`RuntimeException`回滚。若异常被`try-catch`捕获且未重新抛出，事务会被提交。 | `java @Service public class UserService { @Transactional public void updateUser(User user) { try { // ... 更新数据库 ... int i = 1 / 0; // 发生异常 } catch (Exception e) { // 异常被捕获，事务将提交 log.error("更新失败", e); } } }` | 在`catch`块中，将异常包装成`RuntimeException`或其子类重新抛出，或直接抛出。`java catch (Exception e) { throw new RuntimeException(e); }` | +| **5. 抛出不被回滚的异常** | 默认不回滚受检异常（`Checked Exception`）。若方法抛出`Exception`或其子类，事务不会回滚。 | `java @Service public class UserService { // 抛出IOException，默认不回滚 @Transactional public void readFile() throws IOException { throw new IOException("文件读取失败"); } }` | 在`@Transactional`注解中，使用`rollbackFor`属性明确指定需要回滚的异常类型。`java @Transactional(rollbackFor = Exception.class)` | +| **6. 配置了错误的回滚规则** | 显式配置了`noRollbackFor`，指定了某些异常不回滚，或`rollbackFor`配置错误。 | `java @Service public class UserService { // 即使发生RuntimeException也不回滚 @Transactional(noRollbackFor = RuntimeException.class) public void riskyOperation() { throw new RuntimeException("业务失败"); } }` | 检查并修正`rollbackFor`和`noRollbackFor`的配置，确保其符合预期的业务逻辑。 | +| **7. 数据库引擎不支持事务** | 底层数据库的存储引擎不支持事务，如 MySQL 的`MyISAM`。 | `sql -- 表使用了不支持事务的MyISAM引擎 CREATE TABLE `user` ( `id` int NOT NULL AUTO_INCREMENT, PRIMARY KEY (`id`) ) ENGINE=MyISAM DEFAULT CHARSET=utf8mb4;` | 将数据库表的存储引擎修改为支持事务的引擎，如 MySQL 的`InnoDB`。`sql ALTER TABLE `user` ENGINE=InnoDB;` | + + + +### 🎯 什么是 Introduction？ + +Introduction 是 Spring AOP 的一种特殊功能，核心是**不修改目标类代码，通过代理动态为目标类添加新接口的实现**。它需要两个核心要素：一是通过切入点指定‘为哪些类添加’，二是通过 IntroductionInterceptor 提供‘新接口的默认实现’。与普通 AOP 通知不同，它不增强已有方法，而是扩展类的能力范围 —— 比如给 UserService 动态添加 Loggable 接口，最终代理对象既能调用原有方法，又能调用日志方法，适用于第三方类扩展、批量能力注入等无侵入场景。 + + + +### 🎯 Spring AOP的通知类型有哪些？ + +"Spring AOP提供了五种通知类型，分别在方法执行的不同时机生效： + +**通知类型详解**： + +**1. @Before 前置通知**： +- 在目标方法执行前执行 +- 不能阻止目标方法执行 +- 适用于参数校验、权限检查 + +**2. @AfterReturning 返回后通知**： + +- 在目标方法正常返回后执行 +- 可以访问方法返回值 +- 适用于结果处理、缓存更新 + +**3. @AfterThrowing 异常通知**： +- 在目标方法抛出异常后执行 +- 可以访问异常信息 +- 适用于异常处理、错误记录 + +**4. @After 最终通知**： +- 无论方法正常还是异常都会执行 +- 类似finally块的作用 +- 适用于资源清理 + +**5. @Around 环绕通知**： +- 包围目标方法执行 +- 功能最强大，可控制是否执行目标方法 +- 适用于性能监控、事务控制 + +**执行顺序**： +- 正常流程：@Around前 → @Before → 目标方法 → @AfterReturning → @After → @Around后 +- 异常流程：@Around前 → @Before → 目标方法 → @AfterThrowing → @After → @Around后" + +**💻 代码示例**： +```java +@Aspect +@Component +public class AdviceTypeDemo { + + @Pointcut("execution(* com.example.service.*.*(..))") + public void serviceLayer() {} + + @Before("serviceLayer()") + public void beforeAdvice(JoinPoint joinPoint) { + System.out.println("1. 前置通知：方法执行前"); + } + + @AfterReturning(pointcut = "serviceLayer()", returning = "result") + public void afterReturningAdvice(JoinPoint joinPoint, Object result) { + System.out.println("2. 返回后通知：方法正常返回，结果=" + result); + } + + @AfterThrowing(pointcut = "serviceLayer()", throwing = "ex") + public void afterThrowingAdvice(JoinPoint joinPoint, Exception ex) { + System.out.println("3. 异常通知：方法抛出异常，异常=" + ex.getMessage()); + } + + @After("serviceLayer()") + public void afterAdvice(JoinPoint joinPoint) { + System.out.println("4. 最终通知：无论正常还是异常都执行"); + } + + @Around("serviceLayer()") + public Object aroundAdvice(ProceedingJoinPoint joinPoint) throws Throwable { + System.out.println("5. 环绕通知：方法执行前"); + try { + Object result = joinPoint.proceed(); // 调用目标方法 + System.out.println("6. 环绕通知：方法执行后"); + return result; + } catch (Exception e) { + System.out.println("7. 环绕通知：捕获异常"); + throw e; + } + } +} +``` + + + +### 🎯 切点表达式怎么写？ + +"切点表达式使用AspectJ语法，用于精确匹配连接点，Spring AOP主要支持execution表达式： + +**execution表达式语法**： +``` +execution([修饰符] 返回值类型 [类名].方法名(参数列表) [throws 异常]) +``` + +**通配符说明**： +- `*`：匹配任意字符（除了包分隔符.） +- `..`：匹配任意数量的参数或包层级 +- `+`：匹配子类型 + +**常用表达式模式**： + +**1. 方法级别匹配**： +- `execution(* com.example.service.UserService.findById(..))` +- 匹配UserService类的findById方法 + +**2. 类级别匹配**： +- `execution(* com.example.service.UserService.*(..))` +- 匹配UserService类的所有方法 + +**3. 包级别匹配**： +- `execution(* com.example.service.*.*(..))` +- 匹配service包下所有类的所有方法 + +**4. 递归包匹配**： +- `execution(* com.example.service..*.*(..))` +- 匹配service包及子包下所有类的所有方法 + +**其他切点指示符**： +- `@annotation`：匹配标注了指定注解的方法 +- `@within`：匹配标注了指定注解的类 +- `args`：匹配参数类型 +- `target`：匹配目标对象类型 +- `this`：匹配代理对象类型" + +**💻 代码示例**： +```java +@Aspect +@Component +public class PointcutExpressionDemo { + + // 匹配所有public方法 + @Pointcut("execution(public * *(..))") + public void publicMethods() {} + + // 匹配Service层所有方法 + @Pointcut("execution(* com.example.service.*.*(..))") + public void serviceLayer() {} + + // 匹配返回值为String的方法 + @Pointcut("execution(String com.example..*.*(..))") + public void stringMethods() {} + + // 匹配单个参数为String的方法 + @Pointcut("execution(* com.example..*.*(String))") + public void stringParameterMethods() {} + + // 匹配第一个参数为String的方法 + @Pointcut("execution(* com.example..*.*(String,..))") + public void firstStringParameterMethods() {} + + // 匹配标注了@Transactional的方法 + @Pointcut("@annotation(org.springframework.transaction.annotation.Transactional)") + public void transactionalMethods() {} + + // 匹配Controller类中的所有方法 + @Pointcut("@within(org.springframework.stereotype.Controller)") + public void controllerMethods() {} + + // 组合切点表达式 + @Pointcut("serviceLayer() && !stringMethods()") + public void serviceLayerExceptString() {} + + @Before("publicMethods() && args(username)") + public void beforePublicMethodWithUsername(String username) { + System.out.println("调用public方法，用户名参数: " + username); + } +} + +// 自定义注解示例 +@Target(ElementType.METHOD) +@Retention(RetentionPolicy.RUNTIME) +public @interface Log { + String value() default ""; +} + +@Service +public class UserService { + @Log("查询用户") + public User findById(String id) { + return new User(id); + } +} ``` -## 七、MVC +### 🎯 Spring AOP and AspectJ AOP 有什么区别？ + +- Spring AOP 基于动态代理方式实现，AspectJ 基于静态代理方式实现。 +- Spring AOP 仅支持方法级别的 PointCut；提供了完全的 AOP 支持，它还支持属性级别的 PointCut。 + + + +### 🎯 你有没有⽤过Spring的AOP? 是⽤来⼲嘛的? ⼤概会怎么使⽤？ + +Spring AOP 主要用于以下几个方面： + +1. **日志记录**：在方法执行前后自动记录日志。 +2. **性能监控**：在方法执行前后记录时间，监控性能。 +3. **事务管理**：在方法执行前后自动管理事务。 +4. **安全检查**：在方法执行前检查用户权限。 +5. **缓存**：在方法执行后缓存结果以提高性能。 +6. **异常处理**：在方法执行时统一处理异常。 + + + +### 🎯 过滤器和拦截器的区别？ + +> Filter 是在 **Servlet 容器层** 工作的组件，属于 **JavaEE 规范**，可以过滤所有请求（包括静态资源），适合做通用的请求预处理，比如日志、编码、XSS 防御等。 +> 而 Interceptor 是在 **Spring MVC 框架层** 实现的，基于反射调用 Controller 方法前后执行，主要用于业务层面的拦截，比如登录验证、权限校验、接口耗时监控。 +> +> 它们的执行顺序是：**Filter → Interceptor → Controller → Interceptor → Filter**。 +> 一般建议：**通用的跨项目逻辑放在 Filter，跟业务强相关的逻辑放在 Interceptor。** + +| 阶段 | Filter | Interceptor | +| --------------- | ------------------------------------------------------------ | ---------------------------------------------- | +| 作用层级 | Servlet 容器层（Tomcat、Jetty） | SpringMVC 框架层 | +| 拦截范围 | 所有资源（包括静态资源、JSP、Controller） | 仅拦截 Controller（Handler） 请求 | +| 执行顺序 | 请求进入 → Filter → Servlet → Controller → 响应返回 → Filter | 请求进入 → Interceptor → Controller → 返回视图 | +| 是否依赖 Spring | ❌ 不依赖（属于 javax.servlet） | ✅ 依赖 Spring 容器（属于 Spring MVC） | +| 配置方式 | web.xml 或 @WebFilter | 实现 `HandlerInterceptor` 并注册为 Spring Bean | +| 使用场景 | 编码过滤、日志、跨域、XSS、防止SQL注入等 | 权限校验、登录验证、接口耗时统计、业务日志 | + +### 🎯 有 Interceptor 还需要 filter 吗 + +> **需要！** +> 因为 `Filter` 和 `Interceptor` 作用层级不同、职责不同，很多场景下它们是**互补关系**，而不是替代关系。 + + **场景 1：跨域（CORS）处理** + +- **必须用 Filter** +- 因为浏览器的预检请求（OPTIONS）不会进入 Spring Controller，也不会被拦截器拦到。 +- 所以要在 Filter 层统一处理跨域。 + +**场景 2：字符编码（UTF-8）统一处理** + +- 过滤器层配置 `CharacterEncodingFilter`，确保所有请求响应编码一致。 +- Spring 拦截器此时还没执行，没办法控制编码。 + +**场景 3：请求日志与耗时统计** + +- 如果你要统计**整个请求链路耗时**（包括静态资源、文件下载），要在 Filter 层； +- 如果只关注 **Controller 层逻辑耗时**，可以放在 Interceptor。 + +**场景 4：登录鉴权、权限校验** -### Spring MVC 框架有什么用？ +- 适合放在 Interceptor，因为它能访问 Spring 容器 Bean，比如 Redis 用户信息、JWT 校验等。 + +--- + + + +## 🌐 三、Spring MVC架构 + +**核心理念**：基于MVC设计模式的Web框架，提供灵活的请求处理和视图渲染机制。 + +### 🎯 Spring MVC 框架有什么用？ Spring Web MVC 框架提供 **模型-视图-控制器** 架构和随时可用的组件，用于开发灵活且松散耦合的 Web 应用程序。 MVC 模式有助于分离应用程序的不同方面，如输入逻辑，业务逻辑和 UI 逻辑，同时在所有这些元素之间提供松散耦合。 -### Spring MVC的优点 +### 🎯 Spring MVC的优点？ -- 可以支持各种视图技术,而不仅仅局限于JSP +- 可以支持各种视图技术，而不仅仅局限于JSP - 与Spring框架集成（如IoC容器、AOP等） - 清晰的角色分配：前端控制器(dispatcherServlet) ，请求到处理器映射（handlerMapping)，处理器适配器（HandlerAdapter)， 视图解析器（ViewResolver） - 支持各种请求资源的映射策略 -### Spring MVC 的运行流程 | DispatcherServlet描述 +### 🎯 Spring MVC 的整体架构和核心组件？ -在整个 Spring MVC 框架中， DispatcherServlet 处于核心位置，负责协调和组织不同组件以完成请求处理并返回响应的工作 +> Spring MVC 的架构以 `DispatcherServlet` 为核心，负责请求的调度和分发，通过 `HandlerMapping` 找到具体的控制器方法，控制器方法执行后返回 `ModelAndView`，并通过 `ViewResolver` 渲染视图。这样的架构使得 Web 应用中的请求处理过程更加清晰和模块化。 -SpringMVC 处理请求过程： +**Spring MVC** 是一个基于 Servlet 的 Web 框架，它遵循了 **MVC（Model-View-Controller）设计模式**，将应用程序的不同功能分离，增强了应用的可维护性、可扩展性和解耦性。Spring MVC 是 Spring Framework 中的一部分，提供了一个灵活的请求处理流程和 Web 层的解决方案。 -1. 若一个请求匹配 DispatcherServlet 的请求映射路径(在 web.xml中指定)，WEB 容器将该请求转交给 DispatcherServlet 处理 -2. DispatcherServlet 接收到请求后, 将根据请求信息(包括 URL、HTTP方法、请求头、请求参数、Cookie 等)及 HandlerMapping 的配置找到处理请求的处理器(Handler)。可将 HandlerMapping 看成路由控制器， 将 Handler 看成目标主机 -3. 当 DispatcherServlet 根据 HandlerMapping 得到对应当前请求的 Handler 后，通过 HandlerAdapter 对 Handler 进行封装，再以统一的适配器接口调用 Handler -4. 处理器完成业务逻辑的处理后将返回一个 ModelAndView 给 DispatcherServlet，ModelAndView 包含了视图逻辑名和模型数据信息 -5. DispatcherServlet 借助 ViewResoler 完成逻辑视图名到真实视图对象的解析 -6. 得到真实视图对象 View 后, DispatcherServlet 使用这个 View 对ModelAndView 中的模型数据进行视图渲染 +**Spring MVC 的整体架构** -![](https://d1jnx9ba8s6j9r.cloudfront.net/blog/wp-content/uploads/2017/05/dispatcherservlet.png) +Spring MVC 架构是基于请求驱动的模式，处理请求的过程分为以下几个关键步骤： +1. **DispatcherServlet（前端控制器）**： + - Spring MVC 的核心组件之一，所有的 HTTP 请求都首先会进入 `DispatcherServlet`。它作为前端控制器（Front Controller），接收客户端请求并将请求分发给合适的处理器。 + - `DispatcherServlet` 从 web.xml 中进行配置，通常是 Web 应用的入口。 +2. **HandlerMapping（处理器映射器）**： + - `HandlerMapping` 的作用是根据请求的 URL 找到相应的处理器（Controller）。 + - 它会将 URL 映射到相应的处理方法上，`HandlerMapping` 通过查找配置文件中定义的映射关系来决定哪个控制器方法应该处理当前请求。 +3. **Controller（控制器）**： + - 控制器是业务逻辑的核心，负责处理具体的请求并返回一个视图。 + - `@Controller` 注解定义了一个控制器类，而方法上通常使用 `@RequestMapping` 或者更细粒度的注解（如 `@GetMapping`、`@PostMapping`）来映射请求。 +4. **HandlerAdapter（处理器适配器）**： + - `HandlerAdapter` 的作用是根据 `HandlerMapping` 返回的控制器和方法，选择合适的适配器来调用相应的业务逻辑。 + - 它是为了支持不同类型的控制器而设计的，通常会选择具体的适配器，如 `HttpRequestHandlerAdapter` 或 `AnnotationMethodHandlerAdapter`。 +5. **ViewResolver（视图解析器）**： + - `ViewResolver` 负责根据控制器返回的视图名（例如 JSP 文件名）解析出具体的视图对象（如 `InternalResourceViewResolver`）。 + - 它根据视图名称将其解析为一个实际的视图，比如 JSP 或 Thymeleaf 模板。 +6. **ModelAndView（模型与视图）**： + - `ModelAndView` 是控制器方法返回的对象，它包含了模型数据和视图名称。 + - `Model` 存储控制器执行后的数据，`View` 则指向具体的视图模板。 +7. **View（视图）**： + - 视图组件负责将模型数据渲染成用户可以查看的页面，常见的视图技术包括 JSP、Thymeleaf、FreeMarker 等。 -![](https://imgkr.cn-bj.ufileos.com/986949cb-e4dc-42cf-acb9-8cd07bbb4d05.png) +### 🎯 Spring MVC 的运行流程? +在整个 Spring MVC 框架中， DispatcherServlet 处于核心位置，负责协调和组织不同组件以完成请求处理并返回响应的工作 + +SpringMVC 处理请求过程： + +> 1. **DispatcherServlet**接收请求，委托给 HandlerMapping； +> 2. HandlerMapping 匹配处理器（Controller），返回 HandlerExecutionChain； +> 3. 调用 HandlerAdapter 执行 Controller 方法，返回 ModelAndView； +> 4. ViewResolver 解析视图，渲染响应结果。 -### Spring的Controller是单例的吗？多线程情况下Controller是线程安全吗？ +1. **请求接收与分发** + - 入口：用户通过浏览器发送 HTTP 请求，所有请求首先到达 `DispatcherServlet`（前端控制器），它是整个流程的统一入口。 + - 核心作用：`DispatcherServlet` 负责接收请求并协调后续处理流程，类似“调度中心”。 +2. **处理器映射（HandlerMapping）** + - 查找处理器：`DispatcherServlet` 调用 `HandlerMapping` 组件，根据请求的 URL 路径匹配对应的处理器（Controller 或其方法）。例如，带有 `@RequestMapping`注解的方法会被识别为 Handler。 + - 返回执行链：`HandlerMapping` 返回 `HandlerExecutionChain`，包含目标处理器及关联的拦截器（Interceptor）。 +3. **处理器适配（HandlerAdapter）** + - 适配调用：`DispatcherServlet` 通过 `HandlerAdapter` 调用处理器方法。HandlerAdapter 负责将 Servlet 的请求参数转换为处理器方法的输入，并执行具体业务逻辑。 + - 返回值处理：处理器返回 `ModelAndView` 对象，包含模型数据（Model）和视图名称（View）。 +4. **视图解析与渲染** + - 视图解析器：`DispatcherServlet` 将逻辑视图名传递给 `ViewResolver`，解析为具体的视图对象（如 JSP、Thymeleaf 模板）。 + - 数据渲染：视图对象将模型数据填充到请求域，生成响应内容（如 HTML），最终由 DispatcherServlet 返回客户端。 +5. **异常处理** + - **异常捕获**：若处理过程中发生异常，DispatcherServlet 调用 **HandlerExceptionResolver** 组件处理，生成错误视图或 JSON 响应 + + + +### 🎯 Spring 的 Controller 是单例的吗？多线程情况下 Controller 是线程安全吗？ controller默认是单例的，不要使用非静态的成员变量，否则会发生数据逻辑混乱。正因为单例所以不是线程安全的 @@ -714,7 +1618,7 @@ public class ScopeTestController { **单例是不安全的，会导致属性重复使用**。 -#### 解决方案 +**解决方案** 1. 不要在controller中定义成员变量 2. 万一必须要定义一个非静态成员变量时候，则通过注解@Scope(“prototype”)，将其设置为多例模式。 @@ -722,194 +1626,3269 @@ public class ScopeTestController { -## 八、注解 +### 🎯 ResponseBody 是怎么生效的？ + +`@ResponseBody` 的作用是： + **将 Controller 方法的返回值序列化为 HTTP 响应体（Response Body）**，而不是作为视图名称进行解析。 + +Spring 是通过 `RequestResponseBodyMethodProcessor` 这个类完成的，它利用 `HttpMessageConverter`（如 Jackson）将对象转换为 JSON、XML 等格式，最终写入响应流。 + +**@ResponseBody 的作用机制（通俗说明）** + +在没有 `@ResponseBody` 时： + +- Spring MVC 默认认为方法返回的是**视图名（View Name）**； +- DispatcherServlet 会去找对应的 JSP / HTML 模板来渲染。 + +加上 `@ResponseBody` 后： + +- Spring 告诉框架：“这个方法返回的不是视图名，而是数据本身”； +- 它会跳过视图解析流程，进入 **消息转换（Message Conversion）流程**； +- 最终将返回对象序列化后写入 `HttpServletResponse`。 + + + +### 🎯 Spring MVC 全局异常处理机制？ + +Spring MVC 全局异常处理的核心是**集中化管理异常，解耦业务与异常处理**，避免重复 `try-catch` 并保证响应格式统一。 + +主流实现方式是 `@ControllerAdvice + @ExceptionHandler` 组合： + +- `@ControllerAdvice` 定义一个全局通知类，作用于所有或指定包下的 Controller； +- `@ExceptionHandler` 标记在方法上，指定处理的异常类型（如自定义业务异常、参数校验异常）。 + +当 Controller 抛出未捕获的异常时，异常会传递到 `DispatcherServlet`，Spring 会找到 `@ControllerAdvice` 中匹配的 `@ExceptionHandler` 方法，执行后返回统一格式的响应（如包含错误码、提示的 JSON）。 + +此外还有两种补充方案：`@ResponseStatus` 适合异常与固定 HTTP 状态码绑定的场景（如 404 用户不存在）；`HandlerExceptionResolver` 是底层接口，适用于深度定制处理流程，但现在很少用。 + +实际项目中，我们会用 `@ControllerAdvice` 分场景处理异常：自定义业务异常返回具体提示，参数校验异常返回字段错误信息，兜底的 `Exception` 处理器返回友好提示并记录日志，确保用户体验和问题可排查。 + + + +### 🎯 @RequestMapping 注解有哪些属性？ + +"@RequestMapping是Spring MVC最核心的注解，用于映射HTTP请求到处理方法： + +| 属性 | 类型 | 作用 | +| ---------------- | ----------------- | ------------------------------------------------------------ | +| **value / path** | `String[]` | 指定请求的 URL 路径（别名关系，常用 `value`）。 | +| **method** | `RequestMethod[]` | 限定请求方式，如 `GET`、`POST`、`PUT`、`DELETE` 等。 | +| **params** | `String[]` | 请求必须包含的参数条件（或不能包含），如 `params="id=1"`。 | +| **headers** | `String[]` | 请求必须包含的 Header 条件，如 `headers="Content-Type=application/json"`。 | +| **consumes** | `String[]` | 限定请求体（Content-Type），如 `consumes="application/json"`。 | +| **produces** | `String[]` | 限定响应体类型（Accept），如 `produces="application/json"`。 | +| **name** | `String` | 为映射起个名字，方便在工具或日志中区分。 | + +**简化注解**： +- @GetMapping = @RequestMapping(method = GET) +- @PostMapping = @RequestMapping(method = POST) +- @PutMapping = @RequestMapping(method = PUT) +- @DeleteMapping = @RequestMapping(method = DELETE) +- @PatchMapping = @RequestMapping(method = PATCH)" + +**💻 代码示例**： +```java +@Controller +@RequestMapping("/api/users") +public class UserController { + + // 基本映射 + @RequestMapping("/list") + public String listUsers() { + return "users/list"; + } + + // 指定请求方法 + @RequestMapping(value = "/save", method = RequestMethod.POST) + public String saveUser() { + return "redirect:/api/users/list"; + } + + // 路径变量 + @RequestMapping("/detail/{id}") + public String getUserDetail(@PathVariable Long id, Model model) { + model.addAttribute("userId", id); + return "users/detail"; + } + + // 参数限定 + @RequestMapping(value = "/search", params = {"name", "age"}) + public String searchUsers(@RequestParam String name, + @RequestParam int age) { + return "users/search-result"; + } + + // 请求头限定 + @RequestMapping(value = "/api/data", + headers = {"Accept=application/json", "X-API-Version=1.0"}) + @ResponseBody + public List getApiData() { + return userService.findAll(); + } + + // Content-Type限定 + @RequestMapping(value = "/api/users", + method = RequestMethod.POST, + consumes = "application/json", + produces = "application/json") + @ResponseBody + public User createUser(@RequestBody User user) { + return userService.save(user); + } + + // 使用简化注解 + @GetMapping("/api/users/{id}") + @ResponseBody + public User getUser(@PathVariable Long id) { + return userService.findById(id); + } + + @PostMapping(value = "/api/users", + consumes = MediaType.APPLICATION_JSON_VALUE, + produces = MediaType.APPLICATION_JSON_VALUE) + @ResponseBody + public ResponseEntity createUserRest(@RequestBody @Valid User user) { + User savedUser = userService.save(user); + return ResponseEntity.status(HttpStatus.CREATED).body(savedUser); + } + + @PutMapping("/api/users/{id}") + @ResponseBody + public ResponseEntity updateUser(@PathVariable Long id, + @RequestBody @Valid User user) { + user.setId(id); + User updatedUser = userService.update(user); + return ResponseEntity.ok(updatedUser); + } + + @DeleteMapping("/api/users/{id}") + @ResponseBody + public ResponseEntity deleteUser(@PathVariable Long id) { + userService.deleteById(id); + return ResponseEntity.noContent().build(); + } + + // 复杂路径匹配 + @GetMapping("/files/**") + public ResponseEntity getFile(HttpServletRequest request) { + String filePath = request.getRequestURI().substring("/files/".length()); + Resource resource = resourceLoader.getResource("classpath:static/" + filePath); + return ResponseEntity.ok().body(resource); + } + + // 正则表达式路径 + @GetMapping("/users/{id:[0-9]+}") + @ResponseBody + public User getUserByNumericId(@PathVariable Long id) { + return userService.findById(id); + } + + // 矩阵变量 + @GetMapping("/users/{id}/books/{isbn}") + @ResponseBody + public Book getBook(@PathVariable Long id, + @PathVariable String isbn, + @MatrixVariable(name = "edition", pathVar = "isbn") String edition) { + return bookService.findByIsbnAndEdition(isbn, edition); + } +} +``` + + + +### 🎯 CORS 的原理与实现方式？ + +“CORS 是浏览器为解决同源策略限制而制定的跨域规范，核心是**服务器通过 HTTP 响应头告知浏览器允许跨域请求**，分简单请求和预检请求两种交互方式。 + +**原理部分：** + +1. 简单请求（GET/POST/HEAD + 简单头）：前端直接发请求，携带 `Origin` 头；服务器返回 `Access-Control-Allow-Origin`，浏览器验证通过则放行； +2. 预检请求（如 PUT 方法、带自定义头）：浏览器先发 OPTIONS 请求，携带 `Origin`、`Access-Control-Request-Method` 等；服务器返回 `Allow-Origin`、`Allow-Methods` 等许可头；预检通过后，再发真实业务请求。 + +**实现方式：** + +核心在服务器端，以 Spring Boot 为例： + +- 局部用 `@CrossOrigin` 注解，对单个接口生效； +- 全局推荐实现 `WebMvcConfigurer`，统一配置允许的源、方法、头，支持携带 Cookie； +- 也可自定义 Filter 手动设置响应头。 + +前端只需配合配置 `withCredentials: true`（如需携带 Cookie），其他由浏览器自动处理。 + +关键注意点：允许携带 Cookie 时，`Allow-Origin` 不能为 `*`；预检请求需正确处理 OPTIONS 方法，避免 403 错误。 + + + +### 🎯 Spring MVC 如何处理跨域请求（CORS）？ + +- 配置 `@CrossOrigin` + + ```java + @RestController + @CrossOrigin(origins = "http://localhost:8080/", maxAge = 3600) + public class ApiController { ... } + ``` + +- 或通过 `WebMvcConfigurer`全局配置： + + ```java + @Configuration + public class WebConfig implements WebMvcConfigurer { + @Override + public void addCorsMappings(CorsRegistry registry) { + registry.addMapping("/api/**").allowedOrigins("*"); + } + } + ``` + + +--- + + -### 什么是基于Java的Spring注解配置? 给一些注解的例子 +## 🚀 四、Spring Boot核心特性 -基于Java的配置，允许你在少量的Java注解的帮助下，进行你的大部分Spring配置而非通过XML文件。 +**核心理念**：约定大于配置，提供开箱即用的快速开发体验，简化Spring应用的搭建和部署。 -以@Configuration 注解为例，它用来标记类可以当做一个bean的定义，被Spring IOC容器使用。 +### 🎯 什么是Spring Boot？解决了什么问题？ -另一个例子是@Bean注解，它表示此方法将要返回一个对象，作为一个bean注册进Spring应用上下文。 +"Spring Boot是Spring团队提供的快速开发框架，旨在简化Spring应用的初始搭建和开发过程： + +**Spring Boot 的特点（相比 Spring）** + +1. **开箱即用**：几乎零配置，能快速启动应用。 +2. **内嵌服务器**：内置 Tomcat/Jetty/Undertow，不需要单独部署 WAR 包。 +3. **自动装配**：通过条件注解自动配置常用的 Bean，减少 XML 配置。 +4. **约定大于配置**：合理的默认值，配置量大幅减少。 +5. **外部化配置**：支持多环境（application-dev.yml、application-prod.yml）。 +6. **与微服务天然结合**：与 Spring Cloud 无缝对接。 +7. **依赖管理问题**：Spring Boot提供starter依赖，一键解决场景。内置版本仲裁，避免依赖冲突 + +**💻 代码示例**： ```java +// 传统Spring配置（复杂） @Configuration -public class StudentConfig { +@EnableWebMvc +@EnableTransactionManagement +@ComponentScan("com.example") +public class WebConfig implements WebMvcConfigurer { + + @Bean + public DataSource dataSource() { + HikariDataSource dataSource = new HikariDataSource(); + dataSource.setJdbcUrl("jdbc:mysql://localhost:3306/test"); + dataSource.setUsername("root"); + dataSource.setPassword("password"); + return dataSource; + } + @Bean - public StudentBean myStudent() { - return new StudentBean(); + public JdbcTemplate jdbcTemplate(DataSource dataSource) { + return new JdbcTemplate(dataSource); + } + + // 还需要配置视图解析器、事务管理器等... +} + +// Spring Boot方式（简洁） +@SpringBootApplication +public class Application { + public static void main(String[] args) { + SpringApplication.run(Application.class, args); } } + +// application.yml配置即可 +/* +spring: + datasource: + url: jdbc:mysql://localhost:3306/test + username: root + password: password +*/ ``` -### 怎样开启注解装配？ -注解装配在默认情况下是不开启的，为了使用注解装配，我们必须在Spring配置文件中配置 `` 元素。 -### Spring MVC 常用注解: +### 🎯说说 Spring Boot 和 Spring 的关系？ + +> Spring Boot 我理解就是把 spring spring mvc spring data jpa 等等的一些常用的常用的基础框架组合起来，提供默认的配置，然后提供可插拔的设计，就是各种 starter ，来方便开发者使用这一系列的技术，套用官方的一句话， spring 家族发展到今天，已经很庞大了，作为一个开发者，如果想要使用 spring 家族一系列的技术，需要一个一个的搞配置，然后还有个版本兼容性问题，其实挺麻烦的，偶尔也会有小坑出现，其实挺影响开发进度， spring boot 就是来解决这个问题，提供了一个解决方案吧，可以先不关心如何配置，可以快速的启动开发，进行业务逻辑编写，各种需要的技术，加入 starter 就配置好了，直接使用，可以说追求开箱即用的效果吧. + +> 如果说 Spring 是一个家族，其实就是；它包含 spring core, spring mvc，spring boot与spring Cloud 等等； +> +> 那 spring boot 就像是这个家族中的大管家 -##### @Controller +Spring Boot 是 Spring 开源组织下的子项目，是 Spring 组件一站式解决方案，主要是简化了使用 Spring 的难度，简省了繁重的配置，提供了各种启动器，开发者能快速上手。 -在SpringMVC 中，控制器Controller 负责处理由DispatcherServlet 分发的请求，它把用户请求的数据经过业务处理层处理之后封装成一个Model ，然后再把该Model 返回给对应的View 进行展示。在SpringMVC 中只需使用@Controller 标记一个类是Controller ，然后使用@RequestMapping 和@RequestParam 等一些注解用以定义URL 请求和Controller 方法之间的映射，这样的Controller 就能被外界访问到。 +Spring Boot 主要有如下优点： -##### @RequestMapping +1. 容易上手，提升开发效率，为 Spring 开发提供一个更快、更广泛的入门体验。 +2. 开箱即用，远离繁琐的配置。 +3. 提供了一系列大型项目通用的非业务性功能，例如：内嵌服务器、安全管理、运行数据监控、运行状况检查和外部化配置等。 +4. 没有代码生成，也不需要XML配置。 +5. 避免大量的 Maven 导入和各种版本冲突。 -RequestMapping是一个用来处理请求地址映射的注解，可用于类或方法上 -##### @RequestMapping -Spring Framework 4.3 之后引入的基于HTTP方法的变体 +### 🎯 Spring Boot 启动流程？ -- `@GetMapping` -- `@PostMapping` -- `@PutMapping` -- `@DeleteMapping` -- `@PatchMapping` +Spring Boot 的启动流程围绕 `SpringApplication.run()` 方法展开，分为 **初始化阶段、环境准备、上下文创建与刷新、自动配置** 等步骤。整体流程可概括为： -##### @PathVariable +```wiki +启动类 main() → SpringApplication.run() +├─ 初始化阶段：推断应用类型、加载初始化器/监听器 +├─ 环境准备：加载配置、创建监听器集合 +├─ 上下文创建：实例化 ApplicationContext +├─ 上下文刷新：加载 Bean、自动配置、启动容器 +└─ 后置处理：执行 Runner、发布完成事件 +``` -用于将请求URL中的模板变量映射到功能处理方法的参数上，即取出uri模板中的变量作为参数 + **1. 初始化阶段（SpringApplication 构造）** -##### @RequestParam +- 推断应用类型：通过类路径判断是 Web（Servlet/Reactive）或非 Web 应用，决定后续创建哪种 `ApplicationContext` -使用@RequestParam绑定请求参数值，在处理方法入参处使用@RequestParam可以把请求参数传递给请求方法 + （如web应用 `AnnotationConfigServletWebServerApplicationContext`、普通应用`AnnotationConfigApplicationContext`）。 -- value：参数名 -- required：是否必须。默认为true, 表示请求参数中必须包含对应的参数，若不存在，将抛出异常 +- 加载初始化器与监听器：从 `META-INF/spring.factories` 加载 `ApplicationContextInitializer`（用于自定义上下文初始化逻辑）和 `ApplicationListener`（监听启动事件，如 `ConfigFileApplicationListener` 加载配置文件）。 -##### @RequestBody +- 确定主类：通过堆栈信息解析包含 `main()`的启动类，用于组件扫描。 -@RequestBody 表明方法参数应该绑定到HTTP请求体的值 +**2. 环境准备（`run()` 方法前半段）** -##### @ResponseBody +> 不是指 `run()` 之外，而是指 `run()` 里 **在 ApplicationContext 创建之前**的部分 -@Responsebody 表示该方法的返回结果直接写入HTTP response body中 +- 创建监听器集合：初始化 `SpringApplicationRunListeners`（如 `EventPublishingRunListener`），发布 `ApplicationStartingEvent`事件。 +- 加载配置环境： + - 构建 `ConfigurableEnvironment`，解析命令行参数和 `application.properties/yml`文件。 + - 通过 `EnvironmentPostProcessor` 扩展环境变量（如 `RandomValuePropertySource` 支持随机值）。 +- 打印 Banner：加载并显示启动 Banner，支持自定义文本或图片。 -一般在异步获取数据时使用，在使用@RequestMapping后，返回值通常解析为跳转路径，加上@Responsebody后返回结果不会被解析为跳转路径，而是直接写入HTTP response body中。比如异步获取 json 数据，加上@Responsebody后，会直接返回 json 数据。 +**3. 上下文创建与刷新（核心阶段）** -##### @Resource和@Autowired +**调用 `SpringApplication.run()` 方法**： -@Resource和@Autowired都是做bean的注入时使用，其实@Resource并不是Spring的注解，它的包是javax.annotation.Resource，需要导入，但是Spring支持该注解的注入。 +- 创建应用上下文：根据应用类型实例化 `ApplicationContext`（如 Web 应用使用 `AnnotationConfigServletWebServerApplicationContext`）。 +- 准备上下文：关联环境变量、注册初始 Bean（如 `BeanDefinitionLoader` 加载启动类）。 +- 刷新上下文（`ApplicationContext.refresh()` 方法）： + 1. 加载 Bean 定义：扫描 `@Component`、`@Configuration` 等注解，注册 Bean。 + 2. 执行自动配置：通过 `@EnableAutoConfiguration` 加载 `spring.factories` 中的自动配置类（如 `DataSourceAutoConfiguration`），结合条件注解（`@ConditionalOnClass`）按需加载。 + 3. 启动内嵌容器：Web 应用初始化 Tomcat/Jetty 服务器，监听端口 + 4. 完成单例 Bean 初始化：调用 `finishBeanFactoryInitialization()` 实例化所有非懒加载的单例 Bean -- 共同点：两者都可以写在字段和 setter 方法上。两者如果都写在字段上，那么就不需要再写 setter 方法。 -- 不同点 - - @Autowired 为 Spring 提供的注解，@Autowired 注解是按照类型（byType）装配依赖对象，默认情况下它要求依赖对象必须存在，如果允许 null 值，可以设置它的 required 属性为 false。如果我们想使用按照名称（byName）来装配，可以结合 @Qualifier 注解一起使用 - - @Resource 默认按照 ByName 自动注入，由 J2EE 提供，需要导入包 `javax.annotation.Resource`。@Resource 有两个重要的属性：name 和 type，而 Spring 将@ Resource 注解的 name 属性解析为bean 的名字，而 type 属性则解析为 bean 的类型。所以，如果使用 name 属性，则使用 byName 的自动注入策略，而使用 type 属性时则使用 byType 自动注入策略。如果既不制定 name 也不制定 type 属性，这时将通过反射机制使用 byName 自动注入策略。 +**4. 后置处理与启动完成** -##### @ModelAttribute +- 执行 Runner 接口：调用所有 `CommandLineRunner` 和 `ApplicationListener` ，执行启动后自定义逻辑（如初始化数据）。 +- **发布完成事件**：触发 `ApplicationReadyEvent`，通知应用已就绪 -方法入参标注该注解后, 入参的对象就会放到数据模型中 -##### @SessionAttribute -将模型中的某个属性暂存到**HttpSession**中，以便多个请求之间可以共享这个属性 +### 🎯 `CommandLineRunner` 或 `ApplicationRunner`接口区别？ -##### @CookieValue +> “两个接口都是在 Spring Boot 启动完成后执行的，区别在于参数处理：`CommandLineRunner` 直接拿原始字符串数组，而 `ApplicationRunner` 对参数做了解析和封装，更适合处理复杂的启动参数。” -@CookieValue可让处理方法入参绑定某个Cookie 值 +Spring Boot 提供了两个接口，用来在 **应用启动完成之后** 执行一些自定义逻辑： -##### @RequestHeader +**① 相同点** -请求头包含了若干个属性，服务器可据此获知客户端的信息，通过@RequestHeader即可将请求头中的属性值绑定到处理方法的入参中 +- 都会在 `SpringApplication.run()` **上下文刷新完成**，Spring 容器准备就绪后执行。 +- 都可以用 `@Order` 注解或者实现 `Ordered` 接口控制执行顺序。 +- 常见应用场景：数据初始化、缓存预热、检查配置等。 -### @Component, @Controller, @Repository, @Service 有何区别？ +**② 区别点** -- @Component：将 java 类标记为 bean。它是任何 Spring 管理组件的通用构造型。Spring 的组件扫描机制可以将其拾取并将其拉入应用程序环境中 -- @Controller：将一个类标记为 Spring Web MVC 控制器。标有它的 Bean 会自动导入到 IoC 容器中 -- @Service：此注解是组件注解的特化。它不会对 @Component 注解提供任何其他行为。你可以在服务层类中使用 @Service 而不是 @Component，因为它以更好的方式指定了意图 -- @Repository：这个注解是具有类似用途和功能的 @Component 注解的特化。它为 DAO 提供了额外的好处。它将 DAO 导入 IoC 容器，并使未经检查的异常有资格转换为 Spring DataAccessException。 +- **CommandLineRunner**：`run(String... args)`，直接拿到命令行参数数组。 +- **ApplicationRunner**：`run(ApplicationArguments args)`，对命令行参数进行了封装，支持更方便的解析（比如区分 option 参数和非 option 参数）。 -### @Required 注解有什么作用 +举个例子： -这个注解表明bean的属性必须在配置的时候设置，通过一个bean定义的显式的属性值或通过自动装配，若@Required注解的bean属性未被设置，容器将抛出 BeanInitializationException。示例： +- `--name=starfish foo bar` + - 在 **CommandLineRunner** 里拿到的是 `["--name=starfish", "foo", "bar"]` + - 在 **ApplicationRunner** 里，可以通过 `args.getOptionNames()` 拿到 `["name"]`，`args.getOptionValues("name")` 拿到 `["starfish"]`，而非 option 参数就是 `[foo, bar]`。 ```java -public class Employee { - private String name; - @Required - public void setName(String name){ - this.name=name; - } - public string getName(){ - return name; +@Component +public class MyCommandLineRunner implements CommandLineRunner { + @Override + public void run(String... args) throws Exception { + // 可以访问命令行参数args } } ``` -### @Autowired 注解有什么作用 - -@Autowired默认是按照类型装配注入的，默认情况下它要求依赖对象必须存在（可以设置它required属性为false）。@Autowired 注解提供了更细粒度的控制，包括在何处以及如何完成自动装配。它的用法和@Required一样，修饰setter方法、构造器、属性或者具有任意名称和/或多个参数的PN方法。 - ```java -public class Employee { - private String name; - @Autowired - public void setName(String name) { - this.name=name; - } - public string getName(){ - return name; +@Component +public class MyApplicationRunner implements ApplicationRunner { + @Override + public void run(ApplicationArguments args) throws Exception { + // 使用ApplicationArguments来处理命令行参数 } } ``` -### @Autowired和@Resource之间的区别 +两者都是在Spring应用的`ApplicationContext`完全初始化之后，且所有的`@PostConstruct`注解的方法执行完毕后调用的。选择使用哪一个接口取决于你的具体需求，如果你只需要简单的命令行参数，`CommandLineRunner`可能就足够了。如果你需要更复杂的参数解析功能，那么`ApplicationRunner`会是更好的选择。在Spring Boot应用中，你可以同时使用这两个接口，它们并不互相排斥。 + + -用途：做bean的注入时使用 +### 🎯 Spring Boot自动配置原理是什么？ -- @Autowired，属于Spring的注解，`org.springframework.beans.factory.annotation.Autowired`　　　　 +> Spring Boot 相比 Spring 的最大特点是开箱即用、内嵌服务器和自动装配。 +> 自动装配的原理是：启动类上的 `@SpringBootApplication` 启动了 `@EnableAutoConfiguration`，它通过 `AutoConfigurationImportSelector` 扫描 `spring.factories` 或 `spring-autoconfigure-metadata` 中的配置类，并结合条件注解（如 @ConditionalOnClass、@ConditionalOnMissingBean）来决定哪些 Bean 注入到容器中，从而实现按需自动配置。 +> +> 核心机制： +> +> 1. `@SpringBootApplication`组合`@EnableAutoConfiguration`，扫描`META-INF/spring.factories`文件； +> 2. 根据类路径中的依赖（如`spring-boot-starter-web`）自动配置 Bean（如 Tomcat、MVC 组件）； +> 3. 可通过`application.properties/yaml`或`@Conditional`注解覆盖默认配置。 -- @Resource，不属于Spring的注解，JDK1.6支持的注解，`javax.annotation.Resource` +Spring Boot 自动配置（Auto-Configuration）是 Spring Boot 的核心特性之一，旨在根据项目中的依赖自动配置 Spring 应用。通过自动配置，开发者无需手动编写大量的配置代码，可以专注于业务逻辑的开发。其实现原理主要基于以下几个方面： -共同点：都用来装配bean。写在字段上，或写在setter方法 +1. **启动类注解的复合结构** -不同点：@Autowired 默认按类型装配。依赖对象必须存在，如果要允许null值，可以设置它的required属性为false @Autowired(required=false)，也可以使用名称装配，配合@Qualifier注解 + Spring Boot 应用通常使用 `@SpringBootApplication` 注解来启动，该注解本质上是以下三个注解的组合： -@Resource默认是按照名称来装配注入的，只有当找不到与名称匹配的bean才会按照类型来装配注入 + - @SpringBootConfiguration：标识当前类为配置类，继承自 `@Configuration`，支持 Java Config 配置方式。 -### @Qualifier 注解有什么作用 + - @ComponentScan：自动扫描当前包及其子包下的组件（如 `@Controller`、`@Service`等），将其注册为 Bean。 -当创建多个相同类型的 bean 并希望仅使用属性装配其中一个 bean 时，可以使用 @Qualifier 注解和 @Autowired 通过指定应该装配哪个确切的 bean 来消除歧义。 + - **@EnableAutoConfiguration**：**自动配置的核心入口**，通过 `@Import` 导入 `AutoConfigurationImportSelector` 类，触发自动配置流程 -### @RequestMapping 注解有什么用？ + ```java + @SpringBootApplication + public class MyApplication { + public static void main(String[] args) { + SpringApplication.run(MyApplication.class, args); + } + } + ------------- + @Target(ElementType.TYPE) + @Retention(RetentionPolicy.RUNTIME) + @Documented + @Inherited + @SpringBootConfiguration + @EnableAutoConfiguration + @ComponentScan + public @interface SpringBootApplication { + } + ``` -@RequestMapping 注解用于将特定 HTTP 请求方法映射到将处理相应请求的控制器中的特定类/方法。此注释可应用于两个级别： +2. **自动配置的触发机制** -- 类级别：映射请求的 URL -- 方法级别：映射 URL 以及 HTTP 请求方法 + `@EnableAutoConfiguration` 通过 `AutoConfigurationImportSelector` 类加载配置： ------- + - 读取 `spring.factories` 文件：从所有依赖的 `META-INF/spring.factories` 文件中，查找 `org.springframework.boot.autoconfigure.EnableAutoConfiguration` 键值对应的全类名列表。 + ``` + # Auto Configure + org.springframework.boot.autoconfigure.EnableAutoConfiguration=\ + org.springframework.boot.autoconfigure.admin.SpringApplicationAdminJmxAutoConfiguration,\ + org.springframework.boot.autoconfigure.aop.AopAutoConfiguration,\ + org.springframework.boot.autoconfigure.amqp.RabbitAutoConfiguration,\ + ... + ``` + - 条件化筛选配置类：通过条件注解（如 `@ConditionalOnClass`、`@ConditionalOnMissingBean`）过滤掉不满足当前环境的配置类，例如： -## 九、其他问题 + - 类路径中缺少某个类时禁用相关配置（`@ConditionalOnClass`）。 + - 容器中已存在某个 Bean 时跳过重复注册（`@ConditionalOnMissingBean`）。 -### Spring 框架中用到了哪些设计模式？ + - **加载有效配置类**：筛选后的配置类通过反射实例化，并注册到 Spring 容器中 -- **工厂设计模式** : Spring使用工厂模式通过 `BeanFactory`、`ApplicationContext` 创建 bean 对象。 -- **代理设计模式** : Spring AOP 功能的实现。 -- **单例设计模式** : Spring 中的 Bean 默认都是单例的。 -- **模板方法模式** : Spring 中 `jdbcTemplate`、`hibernateTemplate` 等以 Template 结尾的对数据库操作的类，它们就使用到了模板模式。 -- **包装器设计模式** : 我们的项目需要连接多个数据库，而且不同的客户在每次访问中根据需要会去访问不同的数据库。这种模式让我们可以根据客户的需求能够动态切换不同的数据源。 -- **观察者模式:** Spring 事件驱动模型就是观察者模式很经典的一个应用。 -- **适配器模式** :Spring AOP 的增强或通知(Advice)使用到了适配器模式、spring MVC 中也是用到了适配器模式适配`Controller`。 +3. **自动配置类的实现逻辑** + + 自动配置类通常包含以下内容： + + - 条件注解控制：例如，`DataSourceAutoConfiguration` 仅在类路径存在 `javax.sql.DataSource` 时生效。 + + - 默认 Bean 定义：通过 `@Bean` 方法定义默认组件（如 `JdbcTemplate`），开发者可通过配置文件覆盖默认值。 + + - **外部化配置支持**：结合 `@ConfigurationProperties` 将 `application.properties` 中的属性注入到 Bean 中 + +4. **条件注解的作用** + + Spring Boot 提供丰富的条件注解，用于动态控制配置类的加载和 Bean 的注册： + + - @ConditionalOnClass：类路径存在指定类时生效。 + + - @ConditionalOnMissingBean：容器中不存在指定 Bean 时生效。 + + - @ConditionalOnProperty：根据配置文件属性值决定是否加载。 + - **@ConditionalOnWebApplication**：仅在 Web 应用环境下生效 + +5. **自动配置的优化与扩展** + +- 按需加载：通过条件筛选避免加载未使用的组件，减少内存占用。 +- 自定义 Starter：开发者可封装自定义 Starter，遵循相同机制（`spring.factories` \+ 条件注解）实现模块化自动配置。 +- **配置文件优先级**：通过 `spring.autoconfigure.exclude` 显式排除不需要的自动配置类 + +Spring Boot 的自动配置本质上是 **基于约定和条件判断的 Bean 注册机制**，通过以下流程实现“开箱即用”： + +``` +启动类注解 → 加载 spring.factories → 条件筛选 → 注册有效 Bean +``` + +**💻 代码示例**： + +```java +// 自动配置类示例 +@Configuration +@ConditionalOnClass(DataSource.class) +@ConditionalOnProperty(prefix = "spring.datasource", name = "url") +@EnableConfigurationProperties(DataSourceProperties.class) +public class DataSourceAutoConfiguration { + + @Bean + @ConditionalOnMissingBean(DataSource.class) + public DataSource dataSource(DataSourceProperties properties) { + HikariDataSource dataSource = new HikariDataSource(); + dataSource.setJdbcUrl(properties.getUrl()); + dataSource.setUsername(properties.getUsername()); + dataSource.setPassword(properties.getPassword()); + return dataSource; + } +} +// 配置属性类 +@ConfigurationProperties(prefix = "spring.datasource") +public class DataSourceProperties { + private String url; + private String username; + private String password; + // getter/setter... +} + +// spring.factories文件内容 +/* +# Auto Configure +org.springframework.boot.autoconfigure.EnableAutoConfiguration=\ +com.example.autoconfigure.DataSourceAutoConfiguration,\ +com.example.autoconfigure.WebMvcAutoConfiguration +*/ + +// 自定义条件注解 +@Target({ ElementType.TYPE, ElementType.METHOD }) +@Retention(RetentionPolicy.RUNTIME) +@Conditional(OnRedisCondition.class) +public @interface ConditionalOnRedis { +} + +public class OnRedisCondition implements Condition { + @Override + public boolean matches(ConditionContext context, AnnotatedTypeMetadata metadata) { + try { + context.getClassLoader().loadClass("redis.clients.jedis.Jedis"); + return context.getEnvironment().getProperty("spring.redis.host") != null; + } catch (ClassNotFoundException e) { + return false; + } + } +} +``` + + + +### 🎯Spring Boot 的核心注解是哪个？它主要由哪几个注解组成的？ + +启动类上面的注解是@SpringBootApplication，它也是 Spring Boot 的核心注解，主要组合包含了以下 3 个注解： + +- @SpringBootConfiguration：组合了 @Configuration 注解，实现配置文件的功能。 +- @EnableAutoConfiguration：打开自动配置的功能，也可以关闭某个自动配置的选项，如关闭数据源自动配置功能： @SpringBootApplication(exclude = { DataSourceAutoConfiguration.class })。 +- @ComponentScan：Spring组件扫描。 + + + +### 🎯 springboot是怎么加载类的，通过什么方式 + +Spring Boot 加载类的过程是一个结合了 Spring 框架核心机制与自身自动配置特性的复杂流程，主要通过**类路径扫描**、**依赖注入**和**自动配置**三大核心方式实现，具体如下： + +**一、核心入口：主类的 `@SpringBootApplication` 注解** + +Spring Boot 项目的类加载入口是标注了 `@SpringBootApplication` 的主类，该注解是一个复合注解，包含三个关键注解，共同触发类加载机制： + +1. **`@SpringBootConfiguration`**：标识当前类为配置类，允许通过 `@Bean` 注解定义 Bean。 + +2. `@ComponentScan`：核心注解，指定 Spring 扫描类路径下的组件（如`@Component`、`@Service`、`@Controller` + + 等注解的类）并加载到容器中。 + + - 默认扫描范围：主类所在包及其子包（避免全类路径扫描的性能损耗）。 + - 可通过 `basePackages` 或 `basePackageClasses` 属性自定义扫描范围。 + +3. **`@EnableAutoConfiguration`**：触发 Spring Boot 自动配置机制，加载默认或第三方依赖的配置类。 + +**二、类加载的核心方式** + +**1. 基于 `@ComponentScan` 的组件扫描加载** + +- **作用**：扫描类路径下所有标注了 `@Component` 及其派生注解（`@Service`、`@Repository`、`@Controller` 等）的类，将其注册为 Spring 容器中的 Bean。 + +- **流程**： + + - 启动时，`@ComponentScan` 会触发 `ClassPathBeanDefinitionScanner` 类，遍历指定包路径下的 `.class` 文件。 + + - 对每个类进行注解解析，若存在组件注解，则生成 `BeanDefinition`（Bean 的元数据描述）并注册到 `BeanFactory` 中。 + + - 后续通过依赖注入（如 `@Autowired`）将这些 Bean 组装起来。 + + ```java + @Service // 被 @ComponentScan 扫描并加载 + public class UserService { + // ... + } + ``` + +**2. 基于 `@EnableAutoConfiguration` 的自动配置加载** + +- **作用**：自动加载第三方依赖或内置的配置类（如数据库、Web 服务器等），无需手动编写 XML 或 Java 配置。 +- **核心机制**： + - **SPI（Service Provider Interface）**：Spring Boot 会扫描所有依赖的 `META-INF/spring.factories` 文件（或 Spring Boot 2.7+ 的 `META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports` 文件），这些文件中声明了需要自动配置的类全路径。 + - **条件注解过滤**：自动配置类通常包含 `@Conditional` 系列注解（如 `@ConditionalOnClass`、`@ConditionalOnMissingBean` 等），只有满足条件时才会被加载（例如，只有引入了 `spring-boot-starter-web` 依赖，才会加载 Tomcat 相关配置类）。 +- **示例**： + `spring-boot-autoconfigure` 依赖的 `spring.factories` 中声明了 `DataSourceAutoConfiguration`，当项目引入数据库依赖时，该类会被加载，自动配置数据源。 + +**3. 基于 `@Bean` 注解的手动注册加载** + +- **作用**：通过配置类中的 `@Bean` 注解，手动定义需要加载的类（通常用于第三方库中的类，无法直接添加 `@Component` 注解）。 +- **流程**： + - 标注 `@Configuration` 的类会被 `@ComponentScan` 扫描到。 + - 容器解析 `@Bean` 注解的方法，执行方法并将返回的对象注册为 Bean。 + +**4. 基于 `@Import` 的强制加载** + +- **作用**：直接导入指定类，强制将其加载到容器中，无需依赖注解扫描。 + +- **适用场景**：动态加载类、导入非组件类（如普通 POJO）。 + + ```java + @Configuration + @Import(MyUtil.class) // 强制加载 MyUtil 类 + public class AppConfig { + // ... + } + ``` + +**三、类加载的整体流程总结** + +1. **启动触发**：执行主类的 `main` 方法，通过 `SpringApplication.run()` 启动容器。 +2. **扫描准备**：解析 `@SpringBootApplication`，激活 `@ComponentScan` 和 `@EnableAutoConfiguration`。 +3. **组件扫描**：`@ComponentScan` 扫描指定包路径，加载带组件注解的类。 +4. **自动配置**：`@EnableAutoConfiguration` 解析 `spring.factories`，加载满足条件的自动配置类。 +5. **Bean 注册**：所有加载的类通过 `BeanDefinition` 注册到 `BeanFactory`，最终实例化为 Bean 并放入容器。 +6. **依赖注入**：容器通过 `@Autowired` 等注解，将加载的 Bean 自动组装，完成应用初始化。 + +**核心特点** + +- **约定优于配置**：默认扫描主类所在包，减少手动配置。 +- **按需加载**：通过条件注解和依赖检测，只加载必要的类，避免资源浪费。 +- **扩展性**：支持多种加载方式（扫描、手动注册、导入），灵活应对不同场景。 + +这种加载机制使得 Spring Boot 既能简化开发（自动配置），又能保持 Spring 框架的灵活性（手动控制）。 + + + +### 🎯 什么是Starter？ + +> Starter 就是 Spring Boot 的“功能插件”，本质上是一个依赖模块，里面封装了某种功能所需的依赖和自动配置类。Spring Boot 通过 `@EnableAutoConfiguration` 机制扫描 Starter 中的自动配置类，并结合条件注解，最终把所需的 Bean 注入到容器中。这样开发者只需引入 Starter 依赖，就能快速使用对应功能，极大减少了配置工作。 + +**Starter** 就是 **Spring Boot 提供的依赖模块**，它把某个功能所需的依赖、配置、自动装配类都打包好，开发者只需要引入 Starter，就能“开箱即用”。 + +换句话说：**Starter = 依赖管理 + 自动配置 + 约定默认值** + +**🛠 Starter 的组成** + +一个 Starter 一般包含三部分： + +1. **依赖**：把常用的第三方库统一打包（比如 spring-boot-starter-web 就打包了 Spring MVC、Jackson、Tomcat 等）。 +2. **自动配置类**：在 `META-INF/spring.factories` 或 `AutoConfiguration.imports` 中声明。 +3. **条件注解控制**：如 `@ConditionalOnClass`、`@ConditionalOnMissingBean`，保证灵活性。 + +**常见官方 Starter** + +- `spring-boot-starter-web`：Web 开发（Spring MVC、Tomcat） +- `spring-boot-starter-data-jpa`：JPA 和 Hibernate +- `spring-boot-starter-data-redis`：Redis 集成 +- `spring-boot-starter-security`：Spring Security +- `spring-boot-starter-test`：测试依赖 + + + +### 🎯 让你设计一个spring-boot-starter你会怎么设计？ + +设计一个 Spring Boot Starter 需要考虑其目的、功能、易用性和扩展性。以下是设计一个 Spring Boot Starter 的一般步骤： + +1. **定义 Starter 的目的和功能**： + - 确定 Starter 要解决的问题或要提供的功能。 + - 例如，可能是为了简化数据库操作、提供缓存解决方案、实现特定的业务功能等。 +2. **创建 Maven 项目**： + - 使用 Spring Initializr 或者手动创建一个 Maven 项目。 + - 包含必要的依赖，如 Spring Boot 和其他相关库。 +3. **设计自动配置**： + - 创建 `@Configuration` 类来定义自动配置。 + - 使用 `@EnableAutoConfiguration` 和 `@ComponentScan` 注解来启用自动配置和组件扫描。 + - 通过 `@ConditionalOnClass`、`@ConditionalOnMissingBean` 等条件注解来控制配置的生效条件。 +4. **定义默认配置属性**： + - 在 `application.properties` 或 `application.yml` 中定义默认配置属性。 + - 创建一个配置类，使用 `@ConfigurationProperties` 注解来绑定这些属性。 +5. **创建自定义注解**：如果需要，可以创建自定义注解来进一步简化配置。 +6. **实现依赖管理**：在 `pom.xml` 中定义 `` 标签，管理 Starter 依赖的版本。 +7. **编写文档**：提供清晰的文档说明如何使用 Starter，包括配置属性的详细说明。 +8. **打包和命名**： + - 按照 `spring-boot-starter-xxx` 的格式命名你的 Starter。 + - 在 `pom.xml` 中配置 ``、`` 和 ``。 +9. **测试**：编写单元测试和集成测试来验证自动配置的正确性。 +10. **发布**：将 Starter 发布到 Maven 中央仓库或私有仓库，以便其他项目可以依赖。 +11. **提供示例**：提供一个示例项目，展示如何使用你的 Starter。 + +**💻 代码示例**： + +```java +// 1. 自动配置类 +@Configuration +@ConditionalOnClass(SmsService.class) +@ConditionalOnProperty(prefix = "sms", name = "enabled", havingValue = "true") +@EnableConfigurationProperties(SmsProperties.class) +public class SmsAutoConfiguration { + + @Bean + @ConditionalOnMissingBean + public SmsService smsService(SmsProperties properties) { + return new SmsService(properties.getApiKey(), properties.getSecret()); + } +} + +// 2. 配置属性类 +@ConfigurationProperties(prefix = "sms") +@Data +public class SmsProperties { + + private boolean enabled = false; + private String apiKey; + private String secret; + private String templateId; + private int timeout = 5000; +} + +// 3. 核心服务类 +public class SmsService { + + private final String apiKey; + private final String secret; + + public SmsService(String apiKey, String secret) { + this.apiKey = apiKey; + this.secret = secret; + } + + public boolean sendSms(String phone, String message) { + // 发送短信的具体实现 + System.out.println("发送短信到: " + phone + ", 内容: " + message); + return true; + } +} + +// 4. META-INF/spring.factories文件 +/* +org.springframework.boot.autoconfigure.EnableAutoConfiguration=\ +com.example.starter.SmsAutoConfiguration +*/ + +// 5. pom.xml依赖管理（Starter模块） +/* + + com.example + sms-spring-boot-autoconfigure + +*/ + +// 6. 使用自定义Starter +@RestController +public class SmsController { + + @Autowired + private SmsService smsService; + + @PostMapping("/send") + public String sendSms(@RequestParam String phone, @RequestParam String message) { + boolean success = smsService.sendSms(phone, message); + return success ? "发送成功" : "发送失败"; + } +} + +// 7. application.yml配置 +/* +sms: + enabled: true + api-key: your-api-key + secret: your-secret + template-id: SMS_001 + timeout: 10000 +*/ +``` + + + +### 🎯 Spring Boot的配置优先级是怎样的？ + +"Spring Boot遵循约定大于配置原则，支持多种配置方式，具有明确的优先级顺序： + +**配置优先级（从高到低）**： + +**1. 命令行参数**： + +- java -jar app.jar --server.port=8081 +- 优先级最高，可覆盖任何配置 + +**2. 系统环境变量**： + +- export SERVER_PORT=8081 +- 通过操作系统设置的环境变量 + +**3. Java系统属性**： + +- -Dserver.port=8081 +- JVM启动参数设置的系统属性 + +**4. JNDI属性**： + +- java:comp/env/server.port +- 从JNDI获取的属性 + +**5. 随机值属性**： + +- RandomValuePropertySource支持的属性 + +**6. jar包外的配置文件**： + +- application-{profile}.properties/yml +- 放在jar包同级目录 + +**7. jar包内的配置文件**： + +- classpath中的application-{profile}.properties/yml + +**8. @PropertySource注解**： + +- 通过@PropertySource加载的属性文件 + +**9. 默认属性**： + +- SpringApplication.setDefaultProperties() + +**配置文件搜索顺序**： + +1. file:./config/ (当前目录config子目录) +2. file:./ (当前目录) +3. classpath:/config/ (classpath config目录) +4. classpath:/ (classpath根目录) + +**Profile配置**： + +- application.yml (通用配置) +- application-dev.yml (开发环境) +- application-prod.yml (生产环境)" + + + +### 🎯 spring-boot-starter-parent 有什么用 ? + +`spring-boot-starter-parent` 是 Spring Boot 提供的一个父 POM（项目对象模型），用于简化 Spring Boot 项目的构建和依赖管理。它作为父级 POM，主要提供了许多方便的配置，包括版本管理、插件配置、依赖管理等。使用 `spring-boot-starter-parent` 可以减少很多手动配置的工作，让开发者专注于应用程序的开发而非构建过程： + +1. **依赖管理**：`spring-boot-starter-parent` 预定义了 Spring Boot 相关依赖的版本，这意味着你不需要在每个模块的 `pom.xml` 中显式指定版本号，可以避免版本冲突和兼容性问题。 +2. **插件管理**：它提供了一些预先配置的 Maven 插件，例如 `maven-compiler-plugin`（用于Java编译）、`maven-surefire-plugin`（用于单元测试）等，这些插件都已经配置好了适合大多数Spring Boot应用的参数。 +3. **资源过滤和属性替换**：通过在父 POM 中定义资源过滤和属性替换规则，可以确保应用程序的配置文件（如 `application.properties`）在不同环境间正确切换。 +4. **Spring Boot 应用的打包优化**：它配置了 `maven-war-plugin` 插件，使得打包的 WAR 文件中不包含 `META-INF` 目录下的 `pom.xml` 和 `pom.properties` 文件，这对于部署到 Servlet 容器是有益的。 +5. **自动配置的依赖**：可以方便地引入 Spring Boot 的自动配置依赖，例如 `spring-boot-starter`，这可以自动配置应用程序的大部分设置。 +6. **版本一致性**：确保所有 Spring Boot 相关依赖的版本一致，避免不同库之间的版本冲突。 +7. **快速开始**：对于新项目，使用 `spring-boot-starter-parent` 可以快速开始，无需手动配置大量的 Maven 设置。 +8. **继承和自定义**：如果需要，你可以继承 `spring-boot-starter-parent` 并根据项目需求进行自定义配置。 + + + +### 🎯 Spring Boot 打成的 jar 和普通的 jar 有什么区别 ? + +Spring Boot 项目最终打包成的 jar 是可执行 jar ，这种 jar 可以直接通过 `java -jar xxx.jar` 命令来运行，这种 jar 不可以作为普通的 jar 被其他项目依赖，即使依赖了也无法使用其中的类。 + +Spring Boot 的 jar 无法被其他项目依赖，主要还是他和普通 jar 的结构不同。普通的 jar 包，解压后直接就是包名，包里就是我们的代码，而 Spring Boot 打包成的可执行 jar 解压后，在 `\BOOT-INF\classes` 目录下才是我们的代码，因此无法被直接引用。如果非要引用，可以在 pom.xml 文件中增加配置，将 Spring Boot 项目打包成两个 jar ，一个可执行，一个可引用。 + + + +### 🎯 如何使用 Spring Boot 实现异常处理？ + +Spring Boot 提供了异常处理的多种方式，主要通过以下几种手段实现： + +- 使用全局异常处理类（`@ControllerAdvice` 和 `@ExceptionHandler`）。 +- 配置自定义错误页面或返回统一的 JSON 响应。 +- 利用 Spring Boot 自带的 `ErrorController` 接口扩展默认异常处理逻辑。 + + + +### 🎯 Spring Boot Actuator有什么用？ + +"Spring Boot Actuator提供了生产级别的监控和管理功能，是运维必备组件： + +**核心功能**： + +**1. 健康检查**： + +- /health端点显示应用健康状态 +- 支持自定义健康指示器 +- 可集成数据库、Redis等组件检查 + +**2. 指标监控**： + +- /metrics端点提供JVM、HTTP等指标 +- 集成Micrometer指标库 +- 支持Prometheus、InfluxDB等监控系统 + +**3. 应用信息**： + +- /info端点显示应用版本、Git信息 +- /env端点显示环境变量和配置 +- /configprops显示配置属性 + +**4. 运行时管理**： + +- /shutdown端点优雅关闭应用 +- /loggers端点动态修改日志级别 +- /threaddump获取线程转储 + +**常用端点**： + +- /health - 健康检查 +- /metrics - 应用指标 +- /info - 应用信息 +- /env - 环境变量 +- /loggers - 日志配置 +- /heapdump - 堆转储 +- /threaddump - 线程转储 +- /shutdown - 关闭应用 + +**安全配置**： + +- 默认情况下大部分端点需要认证 +- 可通过management.endpoints.web.exposure.include配置暴露端点 +- 建议生产环境配置安全访问控制" + + + +### 🎯 Spring boot 核心配置文件是什么？bootstrap.properties 和 application.properties 有何区别 ? + +单纯做 Spring Boot 开发，可能不太容易遇到 bootstrap.properties 配置文件，但是在结合 Spring Cloud 时，这个配置就会经常遇到了，特别是在需要加载一些远程配置文件的时侯。 + +spring boot 核心的两个配置文件： + +- bootstrap (. yml 或者 . properties)：boostrap 由父 ApplicationContext 加载的，比 applicaton 优先加载，配置在应用程序上下文的引导阶段生效。一般来说我们在 Spring Cloud Config 或者 Nacos 中会用到它。且 boostrap 里面的属性不能被覆盖； +- application (. yml 或者 . properties)： 由 ApplicatonContext 加载，用于 spring boot 项目的自动化配置。 + + + +### 🎯 shutdown hook 原理？ + +`Shutdown Hook` 是 Java 提供的一种机制，允许程序在 JVM**正常关闭**时执行一段自定义的清理代码。它的核心原理是： + +1. **注册**：开发者通过 `Runtime.getRuntime().addShutdownHook(new Thread(...))` 方法，将一个包含清理逻辑的线程注册到 JVM 中。 +2. **触发**：当 JVM 接收到正常关闭信号时（如`System.exit()`、`Ctrl+C`或`SIGTERM`信号），它会启动一个关闭序列。 +3. **执行**：在这个序列中，JVM 会**并发地**启动所有已注册的钩子线程，并**等待所有线程执行完毕**后，才会彻底退出。 + +需要注意的是，钩子是并发执行的，执行顺序不确定，且它只在正常关闭时生效。在强制关闭（如`kill -9`）的情况下，钩子不会被执行。 + +它的主要用途包括：在应用关闭前优雅地关闭数据库连接池、线程池，保存应用状态，或发送关闭通知等。 + +当我们配置 `server.shutdown=graceful` 时，Spring Boot 会在启动时向 JVM 注册一个特殊的 Shutdown Hook。当应用收到 `SIGTERM` 信号时，JVM 会触发这个钩子，进而启动 Spring Boot 自身的关闭流程。这个流程非常强大，它会： + +1. 通知 Web 服务器停止接收新请求。 +2. 等待正在处理的请求完成。 +3. 按顺序销毁所有 Bean，并调用 `@PreDestroy` 方法。 + +这远比我们手动写一个简单的 Shutdown Hook 要健壮和全面得多，它是 Spring Boot 为我们提供的一个开箱即用的企业级特性。 + +------ + + + +## 💾 五、数据访问与事务（Transaction核心） + +**核心理念**：Spring提供统一的事务管理抽象，支持声明式事务，简化数据访问层开发。 + +### 🎯 Spring事务管理机制是什么？ + +> Spring 的事务管理是基于 AOP 的声明式事务机制。 +> 当一个方法加上 `@Transactional` 注解后，Spring 会通过 AOP 代理拦截这个方法调用，进入事务切面逻辑：在方法执行前开启事务，执行过程中如果抛出异常则回滚，否则正常提交。 +> 事务的实现是通过 `PlatformTransactionManager`，它屏蔽了底层 JDBC、JPA 等差异，开发者只需要声明事务即可。 +> 此外，Spring 事务支持传播行为和隔离级别，可以灵活控制事务边界，适应不同的业务场景。 + +**1. 核心思想** + +- Spring 提供了 **统一的事务抽象层**，通过 `PlatformTransactionManager` 接口来管理事务。 +- 常见实现： + - **JDBC**：`DataSourceTransactionManager` + - **JPA**：`JpaTransactionManager` + - **Hibernate**：`HibernateTransactionManager` +- 上层通过 `@Transactional` 或 XML 配置声明事务，Spring 自动完成事务的开启、提交、回滚。 + +**2. 工作原理** + +1. **AOP 代理拦截** + - 带有 `@Transactional` 的方法会被 Spring AOP 拦截（默认 JDK 动态代理或 CGLIB）。 +2. **事务切面处理** + - 拦截后交给 `TransactionInterceptor`，由它负责事务边界： + - 方法执行前：获取 `TransactionManager`，开启事务。 + - 方法执行中：正常执行目标方法。 + - 方法抛异常：根据 `rollbackFor` 等规则回滚事务。 + - 方法执行成功：提交事务。 +3. **事务传播 & 隔离级别** + - 事务属性由 `TransactionAttributeSource` 解析，支持传播行为（Propagation）和隔离级别（Isolation）。 + +**3. 关键机制** + +- **传播行为（Propagation）**：定义事务方法嵌套调用时的事务边界，比如 `REQUIRED`、`REQUIRES_NEW`、`NESTED` 等。 +- **隔离级别（Isolation）**：避免脏读、不可重复读、幻读，比如 `READ_COMMITTED`、`REPEATABLE_READ`。 +- **回滚规则**：默认遇到 `RuntimeException` 回滚，`CheckedException` 不回滚，可以通过 `rollbackFor` 修改。 + +```java +@Service +@Transactional +public class UserService { + + // 声明式事务 - 默认配置 + public void saveUser(User user) { + userRepository.save(user); + // 运行时异常自动回滚 + } + + // 指定传播行为和隔离级别 + @Transactional( + propagation = Propagation.REQUIRES_NEW, + isolation = Isolation.READ_COMMITTED, + timeout = 30, + rollbackFor = Exception.class + ) + public void processOrder(Order order) { + orderRepository.save(order); + updateInventory(order); + } + + // 只读事务优化 + @Transactional(readOnly = true) + public List findActiveUsers() { + return userRepository.findByStatus("ACTIVE"); + } +} +``` + + + +### 🎯 事务管理器? + +Spring 并不直接管理事务，而是提供了多种事务管理器，他们将事务管理的职责委托给 Hibernate 或者 JTA 等持久化机制所提供的相关平台框架的事务来实现。 + +Spring 事务管理器的接口是 `org.springframework.transaction.PlatformTransactionManager`，通过这个接口，Spring 为各个平台如 JDBC、Hibernate 等都提供了对应的事务管理器，但是具体的实现就是各个平台自己的事情了。 + +#### Spring 中的事务管理器的不同实现 + +**事务管理器以普通的 Bean 形式声明在 Spring IOC 容器中** + +- 在应用程序中只需要处理一个数据源, 而且通过 JDBC 存取 + + ```java + org.springframework.jdbc.datasource.DataSourceTransactionManager + ``` + +- 在 JavaEE 应用服务器上用 JTA(Java Transaction API) 进行事务管理 + + ``` + org.springframework.transaction.jta.JtaTransactionManager + ``` + +- 用 Hibernate 框架存取数据库 + + ``` + org.springframework.orm.hibernate3.HibernateTransactionManager + ``` + +**事务管理器以普通的 Bean 形式声明在 Spring IOC 容器中** + + + +### 🎯 Spring Boot 的事务是怎么实现的？ + +>Spring Boot 的事务实现依赖 Spring 的声明式事务机制，本质上还是基于 `PlatformTransactionManager` 和 AOP 拦截器来完成的。 +> 不同的是，Spring Boot 提供了自动配置：比如你用 JDBC，它会自动创建 `DataSourceTransactionManager`；用 JPA，它会自动创建 `JpaTransactionManager`，而且自动启用了 `@EnableTransactionManagement`，所以我们只需要在业务方法上加 `@Transactional` 就能用。 +> 整个流程是：方法调用被代理拦截 → 进入事务拦截器 → 由事务管理器控制事务的开启、提交或回滚。这样开发者只需关注业务逻辑，不需要关心底层事务细节。 + +Spring Boot 的事务管理是基于 **Spring Framework** 的事务抽象实现的，利用了 Spring 的 **声明式事务**（基于 AOP）和 **编程式事务**（通过 `PlatformTransactionManager`）来进行事务控制。在 Spring Boot 中，事务的实现本质上是 Spring 提供的事务管理功能的自动化配置，通过自动配置机制来简化事务管理的配置。 + +下面我们详细探讨 Spring Boot 如何实现事务管理。 + +1. **Spring 事务管理的核心组件** + + Spring 的事务管理主要依赖于以下几个关键组件： + + - **`PlatformTransactionManager`**：这是 Spring 提供的事务管理接口，支持各种类型的事务管理。常用的实现类包括： + - `DataSourceTransactionManager`：用于 JDBC 数据源的事务管理。 + - `JpaTransactionManager`：用于 JPA（Java Persistence API）持久化框架的事务管理。 + - `HibernateTransactionManager`：用于 Hibernate 的事务管理。 + + - **`TransactionDefinition`**：定义了事务的传播行为、隔离级别、回滚规则等事务属性。 + + - **`TransactionStatus`**：代表事务的状态，记录事务的执行情况，可以提交或回滚事务。 + +2. **Spring Boot 如何自动配置事务管理** + + Spring Boot 会自动根据项目中包含的库来配置合适的事务管理器。例如，如果项目中使用的是 **JPA**（通过 `spring-boot-starter-data-jpa`），Spring Boot 会自动配置 `JpaTransactionManager`；如果使用的是 **JDBC**，则会自动配置 `DataSourceTransactionManager`。 + + Spring Boot 的自动配置基于 **Spring Framework** 的 `@EnableTransactionManagement` 和 **AOP**（面向切面编程）机制。 + +- **自动配置事务管理器** + + Spring Boot 会根据 `application.properties` 或 `application.yml` 中的配置自动选择合适的 `PlatformTransactionManager`，并且为你配置一个 `@Transactional` 注解所需的代理。 + + 例如，如果你使用了 JPA 和 Spring Data JPA，Spring Boot 会自动配置 `JpaTransactionManager`。 + + ```java + @Configuration + @EnableTransactionManagement + public class DataSourceConfig { + // 配置事务管理器 + @Bean + public PlatformTransactionManager transactionManager(EntityManagerFactory entityManagerFactory) { + return new JpaTransactionManager(entityManagerFactory); + } + } + ``` + + Spring Boot 会根据自动配置为我们生成这个 `transactionManager`，从而简化了配置过程。 + +- **事务代理和 AOP** + + Spring Boot 的事务管理基于 **AOP（面向切面编程）** 实现的。当你在方法上使用 `@Transactional` 注解时，Spring 会在底层创建一个代理，这个代理会拦截方法的调用，并在方法执行前后执行事务相关的操作（如开启、提交、回滚事务）。 + + 具体来说，Spring 通过 `@Transactional` 注解创建的代理类会通过 **动态代理（JDK 动态代理或 CGLIB 代理）** 来拦截目标方法的调用，执行事务管理逻辑： + + - 在方法执行之前，代理会先开启事务。 + + - 在方法执行后，代理会根据方法执行的结果来决定是提交事务还是回滚事务（如果发生异常）。 + +3. **Spring Boot 事务管理的工作原理** + +- **声明式事务（基于 AOP）** + + Spring Boot 默认使用声明式事务管理。这是通过 **`@Transactional`** 注解实现的。Spring Boot 的事务管理是通过 AOP（面向切面编程）技术来动态处理事务的。 + + 具体流程如下： + + 1. **方法执行之前**：代理对象会在方法执行前调用 `PlatformTransactionManager` 的 `begin()` 方法，开启事务。 + + 2. **方法执行之后**：方法执行完成后，代理对象会根据方法的返回值和抛出的异常来判断是提交事务还是回滚事务： + + - 如果方法正常执行完成，事务会被提交（`commit()`）。 + + - 如果方法抛出 `RuntimeException` 或其他配置的异常类型，事务会被回滚（`rollback()`）。 + +- **事务的传播行为和隔离级别** + + Spring 事务还可以通过配置 `@Transactional` 注解的属性来指定事务的 **传播行为** 和 **隔离级别**，这也是 Spring 事务管理的一大特点。 + + - **传播行为（Propagation）**：指定一个方法如何参与当前事务的上下文。例如： + + - `REQUIRED`（默认）：如果当前方法有事务，加入当前事务；如果没有事务，创建一个新的事务。 + - `REQUIRES_NEW`：总是创建一个新的事务，挂起当前事务。 + - `NESTED`：在当前事务中执行，但支持事务嵌套。 + + - **隔离级别（Isolation）**：控制事务之间的隔离程度。例如： + + - `READ_COMMITTED`：确保读取的数据是已提交的数据。 + + - `READ_UNCOMMITTED`：允许读取未提交的数据。 + + - `SERIALIZABLE`：最严格的隔离级别，保证事务完全隔离，避免并发问题。 + + ```java + @Transactional(propagation = Propagation.REQUIRES_NEW, isolation = Isolation.READ_COMMITTED) + public void someMethod() { + // 该方法会开启新的事务并设置隔离级别 + } + ``` + +- **回滚规则** + + 默认情况下，Spring 的事务管理会在发生 **运行时异常**（`RuntimeException`）或 **Error** 时进行回滚。如果抛出的是 **检查型异常**（`CheckedException`），则默认不会回滚，但可以通过 `@Transactional` 注解的 `rollbackFor` 属性来修改这一行为。 + + ```java + @Transactional(rollbackFor = Exception.class) + public void someMethod() throws Exception { + // 这里如果抛出 Exception 类型的异常，事务会回滚 + } + ``` + +4. **编程式事务管理** + + 除了声明式事务（通过 `@Transactional` 注解），Spring 还支持编程式事务管理。编程式事务管理通常用于对事务进行更加细粒度的控制。Spring 提供了 `PlatformTransactionManager` 接口来进行编程式事务管理，最常用的实现是 `DataSourceTransactionManager` 和 `JpaTransactionManager`。 + + ```java + @Autowired + private PlatformTransactionManager transactionManager; + + public void someMethod() { + TransactionDefinition definition = new DefaultTransactionDefinition(); + TransactionStatus status = transactionManager.getTransaction(definition); + + try { + // 执行业务逻辑 + transactionManager.commit(status); + } catch (Exception e) { + transactionManager.rollback(status); + } + } + ``` + +5. **事务管理的优点和特点** + + - **简化了配置**：Spring Boot 自动根据使用的持久化框架配置事务管理器，减少了手动配置的工作量。 + + - **基于 AOP**：声明式事务管理使用 AOP 技术动态代理，拦截方法调用，实现事务的自动控制。 + + - **灵活的传播行为和隔离级别**：Spring 提供了多种传播行为和隔离级别，开发者可以根据需求灵活配置。 + + - **事务回滚**：通过 `@Transactional` 注解，Spring 可以自动根据异常类型决定是否回滚事务，支持自定义回滚规则。 + +> **Spring Boot 的事务管理是基于 Spring 框架的事务抽象实现的**。Spring Boot 自动配置事务管理器，具体事务管理器（如 `JpaTransactionManager` 或 `DataSourceTransactionManager`）会根据项目中使用的持久化框架自动选择。事务管理主要通过 **声明式事务** 和 **编程式事务** 来实现。 +> +> - **声明式事务**：通过 `@Transactional` 注解来管理事务，Spring 会通过 AOP（面向切面编程）技术，拦截标注了 `@Transactional` 注解的方法，自动处理事务的开始、提交和回滚。 +> - **编程式事务**：通过 `PlatformTransactionManager` 和 `TransactionStatus` 来手动控制事务的提交和回滚，适用于更加灵活的场景。 +> +> Spring Boot 默认支持常见的传播行为（如 `REQUIRED`、`REQUIRES_NEW`）和隔离级别（如 `READ_COMMITTED`、`SERIALIZABLE`），并允许自定义事务回滚规则（默认回滚 `RuntimeException` 和 `Error`）。这些特性使得 Spring Boot 的事务管理既灵活又高效。 + + + +### 🎯 Spring事务传播行为有哪些？ + +"Spring定义了7种事务传播行为，控制事务方法调用时的事务边界： + +**支持当前事务**： + +**REQUIRED（默认）**： +- 有事务就加入，没有就新建 +- 最常用的传播行为 + +**SUPPORTS**： +- 有事务就加入，没有就以非事务执行 +- 适用于查询方法 + +**MANDATORY**： +- 必须在事务中执行，否则抛异常 +- 强制要求调用方提供事务 + +**不支持当前事务**： + +**REQUIRES_NEW**： +- 总是创建新事务，挂起当前事务 +- 适用于独立的子事务 + +**NOT_SUPPORTED**： +- 以非事务方式执行，挂起当前事务 +- 适用于不需要事务的操作 + +**NEVER**： +- 以非事务方式执行，有事务就抛异常 +- 强制非事务执行 + +**嵌套事务**： + +**NESTED**： +- 嵌套事务，基于保存点实现 +- 内层事务回滚不影响外层事务" + + + +### 🎯 Spring事务失效的常见原因有哪些？ + +"Spring事务失效是常见问题，主要原因包括： + +**AOP代理失效场景**： + +**1. 方法不是public**： + +- @Transactional只能作用于public方法 +- private/protected/默认方法事务不生效 + +**2. 内部方法调用**： +- 同一个类内部方法调用，不经过代理 +- 解决方案：通过ApplicationContext获取代理对象 + +**3. 方法被final修饰**： +- final方法无法被代理重写 +- static方法同样无法被代理 + +**异常处理问题**： + +**4. 异常被捕获**： +- try-catch捕获异常但不重新抛出 +- 事务管理器感知不到异常 + +**5. 异常类型不匹配**： +- 默认只回滚RuntimeException +- 检查异常需要显式配置rollbackFor + +**配置和环境问题**： + +**6. 事务管理器未正确配置**： +- 数据源与事务管理器不匹配 +- 多数据源配置错误 + +**7. 传播行为配置错误**： +- NOT_SUPPORTED、NEVER等传播行为 +- 导致方法以非事务方式执行" + +**💻 代码示例**： +```java +@Service +public class UserService { + + // ❌ 错误：非public方法 + @Transactional + private void updateUserInternal(User user) { + userRepository.save(user); + } + + // ❌ 错误：内部调用不经过代理 + @Transactional + public void updateUser(User user) { + this.updateUserInternal(user); // 事务失效 + } + + // ❌ 错误：异常被捕获 + @Transactional + public void saveUserWithCatch(User user) { + try { + userRepository.save(user); + throw new RuntimeException("测试异常"); + } catch (Exception e) { + // 异常被捕获，事务不回滚 + log.error("保存失败", e); + } + } + + // ✅ 正确：重新抛出异常 + @Transactional + public void saveUserCorrect(User user) { + try { + userRepository.save(user); + } catch (Exception e) { + log.error("保存失败", e); + throw e; // 重新抛出异常 + } + } + + // ✅ 正确：指定回滚异常类型 + @Transactional(rollbackFor = Exception.class) + public void saveUserWithCheckedException(User user) throws Exception { + userRepository.save(user); + if (user.getAge() < 0) { + throw new Exception("年龄不能为负数"); + } + } +} +``` + + + +### 🎯 Spring Data JPA和MyBatis的区别？ + +"Spring Data JPA和MyBatis是两种不同的数据访问技术： + +**Spring Data JPA**： +- 基于JPA标准的ORM框架 +- 约定大于配置，自动生成SQL +- Repository接口自动实现CRUD +- 适合领域驱动设计，对象关系映射 +- 学习成本低，开发效率高 + +**MyBatis**： +- 半自动ORM框架，SQL与Java分离 +- 需要手写SQL，控制精确 +- 支持动态SQL，适合复杂查询 +- 适合数据库优先设计 +- 性能可控，但开发工作量大 + +**选择建议**： +- 简单CRUD，快速开发：Spring Data JPA +- 复杂查询，性能要求高：MyBatis +- 团队熟悉SQL：MyBatis +- 团队偏向对象模型：Spring Data JPA" + +**💻 代码示例**： +```java +// Spring Data JPA方式 +@Repository +public interface UserRepository extends JpaRepository { + // 方法名自动生成SQL + List findByNameAndAge(String name, Integer age); + + // 自定义查询 + @Query("SELECT u FROM User u WHERE u.email = ?1") + User findByEmail(String email); + + // 分页查询 + Page findByNameContaining(String name, Pageable pageable); +} + +// MyBatis方式 +@Mapper +public interface UserMapper { + @Select("SELECT * FROM user WHERE name = #{name} AND age = #{age}") + List findByNameAndAge(@Param("name") String name, @Param("age") Integer age); + + @Insert("INSERT INTO user(name, email, age) VALUES(#{name}, #{email}, #{age})") + @Options(useGeneratedKeys = true, keyProperty = "id") + int insertUser(User user); + + // XML配置复杂查询 + List findUsersWithConditions(UserQueryCondition condition); +} +``` + + + +### 🎯 @Transactional 是如何实现的？ + +“`@Transactional`是通过**Spring AOP**和**事务管理器**共同实现的。 + +首先，在 Spring 容器启动时，它会扫描到所有带有`@Transactional`注解的类，并为它们创建**代理对象**。 + +当客户端调用代理对象的方法时，调用会被 AOP 拦截，并交给`TransactionInterceptor`（事务拦截器）处理。 + +`TransactionInterceptor`会执行以下操作： + +1. 读取`@Transactional`注解中的属性，如传播行为、隔离级别等。 +2. 查找对应的`PlatformTransactionManager`。 +3. 根据传播行为，决定是创建新事务、加入现有事务还是以非事务方式执行。 +4. 调用目标方法，执行真正的业务逻辑。 +5. 根据方法执行结果（正常返回或抛出异常），决定是调用`commit()`提交事务，还是调用`rollback()`回滚事务。 + +简单来说，它就是一个环绕在业务方法周围的、自动完成事务管理的 AOP 切面。” + + + +### 🎯 Spring 如何实现嵌套事务？ + +"Spring 实现嵌套事务主要通过**传播行为**和**数据库保存点**。当内层方法使用`Propagation.NESTED`时，Spring 不会创建新事务，而是在当前事务中创建一个保存点。内层回滚时，只会回滚到这个保存点，不影响外层操作；而外层回滚则会导致整个事务全部回滚。 + +这与`REQUIRES_NEW`不同，后者会创建一个完全独立的新事务，内层回滚和提交都不会影响外层事务。使用嵌套事务时，需要确保数据库支持保存点，并且内层异常被外层捕获，否则整个事务会被回滚。” + + + +### 🎯 @Transactional 为什么不能用在私有方法上？ + +`@Transactional` 注解是 Spring 提供的一种声明式事务管理方式，它用于指定某个方法在执行时需要进行事务管理。通常，这个注解被用在公有（public）方法上，原因包括： + +1. **代理机制**：Spring 的声明式事务管理是基于 AOP（面向切面编程）实现的。当使用 `@Transactional` 注解时，Spring 会为被注解的方法创建一个代理对象。对于非公有方法，Spring 无法在运行时动态地创建代理，因为这些方法不能被代理对象所调用。 +2. **事务传播**：事务的传播依赖于方法调用链。`@Transactional` 注解通常用于服务层或对外的 API 方法上，这样可以确保在业务逻辑的开始处就开启事务，并在业务逻辑结束时提交或回滚事务。如果将 `@Transactional` 注解用在私有方法上，那么事务的传播行为（如传播级别、事务的保存点等）可能不会按预期工作。 +3. **代码设计**：从设计的角度来看，将 `@Transactional` 注解用在公有方法上更符合业务逻辑的封装和分层。私有方法通常被设计为辅助方法，不应该独立承担事务管理的责任。 +4. **事务的可见性**：将 `@Transactional` 注解用在公有方法上可以清晰地表明事务的边界，这对于理解和维护代码都是有益的。如果用在私有方法上，可能会隐藏事务的边界，使得其他开发者难以理解事务的范围。 +5. **事务的粒度**：事务应该在业务逻辑的适当粒度上进行管理。通常，一个业务操作会跨越多个方法调用，将 `@Transactional` 注解用在私有方法上可能会导致过细的事务粒度，这不利于事务管理。 +6. **Spring 版本限制**：在 Spring 5 之前的版本中，`@Transactional` 注解确实不能用于非公有方法。从 Spring 5 开始，引入了对非公有方法的支持，但是仍然推荐将 `@Transactional` 注解用在公有方法上。 + + + +### 🎯 事务传播属性 + +- 当事务方法被另一个事务方法调用时， 必须指定事务应该如何传播。例如：方法可能继续在现有事务中运行，也可能开启一个新事务，并在自己的事务中运行 +- 事务的传播行为可以由传播属性指定，Spring 定义了 7 种类传播行为： + +| 传播行为 | 意义 | +| ------------------------- | ------------------------------------------------------------ | +| PROPAGATION_MANDATORY | 表示该方法必须运行在一个事务中。如果当前没有事务正在发生，将抛出一个异常 | +| PROPAGATION_NESTED | 表示如果当前正有一个事务在进行中，则该方法应当运行在一个嵌套式事务中。被嵌套的事务可以独立于封装事务进行提交或回滚。如果封装事务不存在，行为就像PROPAGATION_REQUIRES一样。 | +| PROPAGATION_NEVER | 表示当前的方法不应该在一个事务中运行。如果一个事务正在进行，则会抛出一个异常。 | +| PROPAGATION_NOT_SUPPORTED | 表示该方法不应该在一个事务中运行。如果一个现有事务正在进行中，它将在该方法的运行期间被挂起。 | +| PROPAGATION_SUPPORTS | 表示当前方法不需要事务性上下文，但是如果有一个事务已经在运行的话，它也可以在这个事务里运行。 | +| PROPAGATION_REQUIRES_NEW | 表示当前方法必须在它自己的事务里运行。一个新的事务将被启动，而且如果有一个现有事务在运行的话，则将在这个方法运行期间被挂起。 | +| PROPAGATION_REQUIRES | 表示当前方法必须在一个事务中运行。如果一个现有事务正在进行中，该方法将在那个事务中运行，否则就要开始一个新事务。 | + + + +### 🎯 Spring 支持的事务隔离级别 + +| 隔离级别 | 含义 | +| -------------------------- | ------------------------------------------------------------ | +| ISOLATION_DEFAULT | 使用后端数据库默认的隔离级别。 | +| ISOLATION_READ_UNCOMMITTED | 允许读取尚未提交的更改。可能导致脏读、幻影读或不可重复读。 | +| ISOLATION_READ_COMMITTED | 允许从已经提交的并发事务读取。可防止脏读，但幻影读和不可重复读仍可能会发生。 | +| ISOLATION_REPEATABLE_READ | 对相同字段的多次读取的结果是一致的，除非数据被当前事务本身改变。可防止脏读和不可重复读，但幻影读仍可能发生。 | +| ISOLATION_SERIALIZABLE | 完全服从ACID的隔离级别，确保不发生脏读、不可重复读和幻影读。这在所有隔离级别中也是最慢的，因为它通常是通过完全锁定当前事务所涉及的数据表来完成的。 | + +事务的隔离级别要得到底层数据库引擎的支持，而不是应用程序或者框架的支持； + +Oracle 支持的 2 种事务隔离级别，Mysql支持 4 种事务隔离级别。 + + + +### 🎯 设置隔离事务属性 + +用 @Transactional 注解声明式地管理事务时可以在 @Transactional 的 isolation 属性中设置隔离级别 + +在 Spring 事务通知中, 可以在 `` 元素中指定隔离级别 + +### 🎯 设置回滚事务属性 + +- 默认情况下只有未检查异常(RuntimeException和Error类型的异常)会导致事务回滚，而受检查异常不会。 +- 事务的回滚规则可以通过 @Transactional 注解的 rollbackFor和 noRollbackFor属性来定义，这两个属性被声明为 Class[] 类型的，因此可以为这两个属性指定多个异常类。 + + - rollbackFor：遇到时必须进行回滚 + - noRollbackFor： 一组异常类，遇到时必须不回滚 + +### 🎯 超时和只读属性 + +- 由于事务可以在行和表上获得锁， 因此长事务会占用资源，并对整体性能产生影响 +- 如果一个事物只读取数据但不做修改，数据库引擎可以对这个事务进行优化 +- 超时事务属性：事务在强制回滚之前可以保持多久，这样可以防止长期运行的事务占用资源 +- 只读事务属性：表示这个事务只读取数据但不更新数据，这样可以帮助数据库引擎优化事务 + +**设置超时和只读事务属性** + +- 超时和只读属性可以在 @Transactional 注解中定义，超时属性以秒为单位来计算 + +列出两种方式的示例： + +```java +@Transactional(propagation = Propagation.NESTED, timeout = 1000, isolation = Isolation.READ_COMMITTED, rollbackFor = Exception.class) +``` + +```xml + + + + + + + + + + + + + + + +``` + +--- + +## 🔒 六、Spring Security安全框架 + +**核心理念**：Spring Security提供全面的安全解决方案，包括认证、授权、攻击防护等企业级安全功能。 + +### 🎯 Spring Security的核心概念？ + +"Spring Security是基于过滤器链的安全框架： + +**核心概念**： + +**Authentication（认证）**： +- 验证用户身份的过程 +- 包含用户凭证和权限信息 +- 存储在SecurityContext中 + +**Authorization（授权）**： +- 控制用户对资源的访问权限 +- 基于角色或权限进行访问控制 +- 支持方法级和URL级授权 + +**SecurityContext**： +- 安全上下文，存储当前用户的安全信息 +- 通过ThreadLocal实现线程隔离 +- SecurityContextHolder提供访问接口 + +**核心组件**： + +**AuthenticationManager**： +- 认证管理器，处理认证请求 +- 通常使用ProviderManager实现 + +**UserDetailsService**： +- 用户详情服务，加载用户信息 +- 自定义用户数据源的关键接口 + +**PasswordEncoder**： +- 密码编码器，处理密码加密 +- 推荐使用BCryptPasswordEncoder + +**过滤器链**： +- SecurityFilterChain处理HTTP请求 +- 包含认证、授权、CSRF防护等过滤器" + +**💻 代码示例**： +```java +@Configuration +@EnableWebSecurity +public class SecurityConfig { + + @Bean + public SecurityFilterChain filterChain(HttpSecurity http) throws Exception { + http.authorizeHttpRequests(auth -> auth + .requestMatchers("/public/**").permitAll() + .requestMatchers("/admin/**").hasRole("ADMIN") + .anyRequest().authenticated() + ) + .formLogin(form -> form + .loginPage("/login") + .defaultSuccessUrl("/dashboard") + ) + .logout(logout -> logout + .logoutSuccessUrl("/login?logout") + ); + return http.build(); + } + + @Bean + public PasswordEncoder passwordEncoder() { + return new BCryptPasswordEncoder(); + } + + @Bean + public UserDetailsService userDetailsService(PasswordEncoder encoder) { + UserDetails admin = User.builder() + .username("admin") + .password(encoder.encode("admin123")) + .roles("ADMIN") + .build(); + return new InMemoryUserDetailsManager(admin); + } +} +``` + +### 🎯 Spring Security如何实现JWT认证？ + +"JWT认证是无状态的认证方式，Spring Security可以很好地集成： + +**JWT认证流程**： +1. 用户登录，验证用户名密码 +2. 生成JWT Token返回给客户端 +3. 客户端在请求头中携带Token +4. 服务器验证Token并提取用户信息 +5. 基于用户信息进行授权决策 + +**实现要点**： + +**JWT工具类**： +- 生成Token：包含用户信息和过期时间 +- 验证Token：检查签名和过期状态 +- 解析Token：提取用户身份信息 + +**JWT过滤器**： +- 继承OncePerRequestFilter +- 从请求头提取Token +- 验证Token并设置SecurityContext + +**认证端点**： +- 处理登录请求 +- 验证用户凭证 +- 生成并返回JWT Token + +**优势**：无状态、跨域友好、易于扩展 +**注意**：Token安全存储、合理设置过期时间" + +**💻 代码示例**： +```java +@RestController +public class AuthController { + + @Autowired + private AuthenticationManager authManager; + @Autowired + private JwtUtil jwtUtil; + + @PostMapping("/login") + public ResponseEntity login(@RequestBody LoginRequest request) { + Authentication auth = authManager.authenticate( + new UsernamePasswordAuthenticationToken( + request.getUsername(), request.getPassword()) + ); + + UserDetails user = (UserDetails) auth.getPrincipal(); + String token = jwtUtil.generateToken(user.getUsername()); + + return ResponseEntity.ok(new JwtResponse(token)); + } +} + +@Component +public class JwtAuthenticationFilter extends OncePerRequestFilter { + + @Override + protected void doFilterInternal(HttpServletRequest request, + HttpServletResponse response, + FilterChain chain) throws ServletException, IOException { + String token = extractToken(request); + + if (token != null && jwtUtil.validateToken(token)) { + String username = jwtUtil.getUsernameFromToken(token); + UserDetails user = userDetailsService.loadUserByUsername(username); + + Authentication auth = new UsernamePasswordAuthenticationToken( + user, null, user.getAuthorities()); + SecurityContextHolder.getContext().setAuthentication(auth); + } + + chain.doFilter(request, response); + } +} +``` + +### 🎯 Spring Security方法级安全怎么使用？ + +"Spring Security提供了方法级的安全控制，可以在方法上直接配置权限： + +**启用方法安全**： +- @EnableGlobalMethodSecurity注解 +- @EnableMethodSecurity（Spring Security 6+） +- 支持多种注解方式 + +**安全注解类型**： + +**@PreAuthorize**： +- 方法执行前进行权限检查 +- 支持SpEL表达式 +- 最常用的方法级安全注解 + +**@PostAuthorize**： +- 方法执行后进行权限检查 +- 可以基于返回值进行权限控制 + +**@Secured**： +- 简单的角色检查 +- 只支持角色名称，不支持复杂表达式 + +**@RolesAllowed**： +- JSR-250标准注解 +- 功能类似@Secured + +**SpEL表达式**： +- hasRole('ROLE_USER')：检查角色 +- hasAuthority('READ_USER')：检查权限 +- authentication.name == #username：复杂权限逻辑" + +**💻 代码示例**： +```java +@Configuration +@EnableMethodSecurity(prePostEnabled = true) +public class MethodSecurityConfig { +} + +@Service +public class UserService { + + // 只有ADMIN角色可以访问 + @PreAuthorize("hasRole('ADMIN')") + public void deleteUser(Long userId) { + userRepository.deleteById(userId); + } + + // 用户只能访问自己的信息 + @PreAuthorize("hasRole('ADMIN') or authentication.name == #username") + public User getUserInfo(String username) { + return userRepository.findByUsername(username); + } + + // 基于返回值的权限检查 + @PostAuthorize("hasRole('ADMIN') or returnObject.username == authentication.name") + public User findUserById(Long id) { + return userRepository.findById(id).orElse(null); + } + + // 多个权限检查 + @PreAuthorize("hasAnyRole('ADMIN', 'MANAGER') and hasAuthority('USER_READ')") + public List getAllUsers() { + return userRepository.findAll(); + } +} +``` + + + +### 🎯 比较一下 Spring Security 和 Shiro 各自的优缺点 ? + +由于 Spring Boot 官方提供了大量的非常方便的开箱即用的 Starter ，包括 Spring Security 的 Starter ，使得在 Spring Boot 中使用 Spring Security 变得更加容易，甚至只需要添加一个依赖就可以保护所有的接口，所以，如果是 Spring Boot 项目，一般选择 Spring Security 。当然这只是一个建议的组合，单纯从技术上来说，无论怎么组合，都是没有问题的。Shiro 和 Spring Security 相比，主要有如下一些特点： + +1. Spring Security 是一个重量级的安全管理框架；Shiro 则是一个轻量级的安全管理框架 +2. Spring Security 概念复杂，配置繁琐；Shiro 概念简单、配置简单 +3. Spring Security 功能强大；Shiro 功能简单 + +--- + + + +## ☁️ 七、Spring Cloud微服务 + +**核心理念**：Spring Cloud为分布式系统开发提供工具集，简化微服务架构的实现和运维。 + +### 🎯 Spring Cloud 核心组件有哪些？各自作用？ + +- **服务注册与发现**：Eureka/Nacos（服务实例自动注册与发现）； +- **服务调用**：Feign（基于接口的声明式 REST 调用）； +- **负载均衡**：Ribbon（客户端负载均衡算法）； +- **熔断降级**：Hystrix/Sentinel（防止级联故障）； +- **网关**：Gateway（统一入口，路由、限流、认证）。 + +💻 代码示例 +```java +// 服务提供者 +@RestController +@EnableEurekaClient +public class UserController { + + @GetMapping("/users/{id}") + public User getUser(@PathVariable Long id) { + return userService.findById(id); + } +} + +// 服务消费者 - Feign客户端 +@FeignClient(name = "user-service", fallback = UserServiceFallback.class) +public interface UserServiceClient { + @GetMapping("/users/{id}") + User getUser(@PathVariable Long id); +} + +// 断路器降级 +@Component +public class UserServiceFallback implements UserServiceClient { + @Override + public User getUser(Long id) { + return new User(id, "默认用户", "服务降级"); + } +} + +// Gateway网关配置 +@Configuration +public class GatewayConfig { + + @Bean + public RouteLocator customRoutes(RouteLocatorBuilder builder) { + return builder.routes() + .route("user-service", r -> r.path("/users/**") + .uri("lb://user-service")) + .route("order-service", r -> r.path("/orders/**") + .uri("lb://order-service")) + .build(); + } +} +``` + +### 🎯 微服务之间如何进行通信？ + +"微服务间通信有多种方式，各有适用场景： + +**同步通信**： + +**HTTP/REST**： + +- 最常用的通信方式 +- 基于HTTP协议，简单易理解 +- 使用JSON格式传输数据 +- 工具：RestTemplate、OpenFeign + +**RPC调用**： +- 远程过程调用，性能较好 +- 支持多种协议：gRPC、Dubbo +- 强类型接口，开发效率高 + +**异步通信**： + +**消息队列**： +- 基于消息中间件：RabbitMQ、Kafka +- 解耦服务间依赖关系 +- 支持削峰填谷、事件驱动 + +**事件驱动**： +- 基于事件的异步通信 +- 服务发布和订阅事件 +- 实现最终一致性 + +**选择原则**： +- 实时性要求高：同步通信 +- 解耦要求高：异步通信 +- 性能要求高：RPC调用 +- 简单场景：HTTP/REST" + +**💻 代码示例**： +```java +// HTTP通信 - RestTemplate +@Service +public class OrderService { + + @Autowired + private RestTemplate restTemplate; + + public User getUserInfo(Long userId) { + String url = "http://user-service/users/" + userId; + return restTemplate.getForObject(url, User.class); + } +} + +// HTTP通信 - OpenFeign +@FeignClient("user-service") +public interface UserClient { + @GetMapping("/users/{id}") + User getUser(@PathVariable Long id); +} + +// 消息队列通信 +@Component +public class OrderEventPublisher { + + @Autowired + private RabbitTemplate rabbitTemplate; + + public void publishOrderCreated(Order order) { + OrderCreatedEvent event = new OrderCreatedEvent(order); + rabbitTemplate.convertAndSend("order.exchange", + "order.created", event); + } +} + +@RabbitListener(queues = "inventory.queue") +public void handleOrderCreated(OrderCreatedEvent event) { + // 处理订单创建事件，更新库存 + inventoryService.updateInventory(event.getOrder()); +} +``` + +### 🎯 微服务的分布式事务怎么处理？ + +"分布式事务是微服务架构的核心挑战，有多种解决方案： + +**两阶段提交（2PC）**： +- 传统的强一致性解决方案 +- 协调者和参与者模式 +- 存在性能和可用性问题，不适合微服务 + +**TCC模式**： +- Try-Confirm-Cancel三个阶段 +- 业务层面的补偿机制 +- 实现复杂但控制精确 + +**Saga模式**： +- 长事务拆分为多个短事务 +- 每个事务有对应的补偿操作 +- 适合业务流程复杂的场景 + +**最终一致性**： +- 基于消息队列的异步处理 +- 通过重试和补偿达到最终一致 +- 性能好，但需要处理中间状态 + +**分布式事务框架**： +- Seata：阿里开源的分布式事务解决方案 +- 支持AT、TCC、Saga等模式 +- 对业务代码侵入性小 + +**最佳实践**： +- 避免分布式事务，通过设计规避 +- 优先考虑最终一致性 +- 重要业务场景使用TCC或Saga" + +**💻 代码示例**： +```java +// Saga模式示例 +@SagaOrchestrationStart +@Transactional +public class OrderSagaService { + + public void processOrder(Order order) { + // 1. 创建订单 + orderService.createOrder(order); + + // 2. 扣减库存 + inventoryService.decreaseStock(order.getItems()); + + // 3. 扣减积分 + pointService.decreasePoints(order.getUserId(), order.getPoints()); + + // 4. 发送通知 + notificationService.sendOrderNotification(order); + } + + // 补偿方法 + @SagaOrchestrationCancel + public void cancelOrder(Order order) { + notificationService.cancelNotification(order); + pointService.compensatePoints(order.getUserId(), order.getPoints()); + inventoryService.compensateStock(order.getItems()); + orderService.cancelOrder(order); + } +} + +// 基于消息的最终一致性 +@Service +public class OrderService { + + @Transactional + public void createOrder(Order order) { + // 1. 保存订单 + orderRepository.save(order); + + // 2. 发送事件消息 + OrderCreatedEvent event = new OrderCreatedEvent(order); + messageProducer.send("order.created", event); + } +} + +@EventListener +public class InventoryEventHandler { + + @RabbitListener(queues = "inventory.order.queue") + public void handleOrderCreated(OrderCreatedEvent event) { + try { + inventoryService.decreaseStock(event.getOrder().getItems()); + // 发送库存扣减成功事件 + messageProducer.send("inventory.decreased", + new InventoryDecreasedEvent(event.getOrder())); + } catch (Exception e) { + // 发送库存扣减失败事件 + messageProducer.send("inventory.decrease.failed", + new InventoryDecreaseFailedEvent(event.getOrder())); + } + } +} +``` + +--- + + + +## 📝 八、注解与配置 + +**核心理念**：Spring注解简化配置，提供声明式编程模型，实现约定大于配置的设计理念。 + +### 🎯 Spring常用注解有哪些？ + +"Spring提供了丰富的注解简化开发配置： + +**核心注解**： + +**@Component及其派生**： +- @Component：通用组件注解 +- @Service：服务层组件 +- @Repository：数据访问层组件 +- @Controller：控制器组件 +- @RestController：REST控制器（@Controller + @ResponseBody） + +**依赖注入注解**： +- @Autowired：自动装配，按类型注入 +- @Qualifier：指定具体实现类 +- @Resource：按名称注入（JSR-250） +- @Value：注入配置值 + +**配置相关注解**： +- @Configuration：配置类 +- @Bean：定义Bean +- @Import：导入其他配置 +- @ComponentScan：组件扫描 +- @Profile：环境配置 + +**Web相关注解**： +- @RequestMapping：请求映射 +- @GetMapping/@PostMapping：HTTP方法映射 +- @RequestParam：请求参数 +- @PathVariable：路径变量 +- @RequestBody/@ResponseBody：请求/响应体 + +**AOP注解**： +- @Aspect：切面 +- @Pointcut：切点 +- @Before/@After/@Around：通知类型 + +**事务注解**： +- @Transactional：声明式事务 +- @EnableTransactionManagement：启用事务管理" + +**💻 代码示例**： +```java +// 组件注解 +@Service +@Transactional +public class UserService { + + @Autowired + private UserRepository userRepository; + + @Value("${app.default.page-size:10}") + private int defaultPageSize; + + public Page findUsers(Pageable pageable) { + return userRepository.findAll(pageable); + } +} + +// Web注解 +@RestController +@RequestMapping("/api/users") +public class UserController { + + @Autowired + private UserService userService; + + @GetMapping("/{id}") + public ResponseEntity getUser(@PathVariable Long id) { + User user = userService.findById(id); + return ResponseEntity.ok(user); + } + + @PostMapping + public ResponseEntity createUser(@RequestBody @Valid User user) { + User saved = userService.save(user); + return ResponseEntity.status(HttpStatus.CREATED).body(saved); + } +} + +// 配置注解 +@Configuration +@EnableWebMvc +@ComponentScan("com.example") +public class WebConfig implements WebMvcConfigurer { + + @Bean + @Profile("dev") + public DataSource devDataSource() { + return new H2DataSource(); + } + + @Bean + @Profile("prod") + public DataSource prodDataSource() { + return new MySQLDataSource(); + } +} +``` + +### 🎯 @Autowired和@Resource的区别？ + +`@Autowired`和`@Resource`都是 Spring 中用于依赖注入的注解，但它们有几个关键区别。 + +首先，**来源不同**。`@Autowired`是 Spring 框架自带的注解，而`@Resource`是 Java EE 规范定义的标准注解，因此`@Resource`的耦合度更低，可移植性更好。 + +其次，也是最核心的区别，**默认的注入方式不同**。`@Autowired`默认 按类型（byType）**查找 Bean，而`@Resource`默认**按名称（byName） 查找 Bean。如果`@Resource`找不到名称匹配的 Bean，它会降级为按类型查找。 + +第三，**解决歧义性的方式不同**。当容器中有多个同类型的 Bean 时，`@Autowired`需要与`@Qualifier`注解结合来指定 Bean 的名称；而`@Resource`可以直接通过其`name`属性来指定，使用上更直接。 + +在实际开发中，如果是纯 Spring 项目，两者都可以使用。但为了遵循 Java 标准和降低框架耦合，推荐使用`@Resource`。不过，`@Autowired`在 Spring 生态中更为常见，尤其是在构造器注入和字段注入中。 + + + +### 🎯 @component注解是单例模式吗？ + +在 Spring 框架中，使用 `@Component` 注解（包括其派生注解如 `@Service`、`@Controller`、`@Repository`）标注的组件，**默认是单例模式**。 + +具体来说： + +- Spring 容器在初始化时，会为 `@Component` 注解的类创建**唯一实例**，并在整个容器生命周期内复用该实例。 +- 所有依赖注入（如 `@Autowired`）获取到的都是同一个实例，除非手动修改作用域。 + +如果需要改变默认的单例行为，可以通过 `@Scope` 注解指定作用域 + + + +### 🎯 Spring中如何自定义注解？ + +"Spring支持自定义注解来简化重复配置： + +**自定义注解步骤**： + +**1. 定义注解**： +- 使用@interface关键字 +- 添加@Target指定作用范围 +- 添加@Retention指定生命周期 +- 定义注解属性 + +**2. 注解处理**： +- AOP方式：通过切面处理 +- BeanPostProcessor：Bean后处理器 +- 反射机制：运行时处理 + +**3. 与Spring集成**： +- 结合Spring AOP实现横切关注点 +- 利用Spring的依赖注入能力 +- 配合事件机制实现解耦 + +**常见应用场景**： +- 日志记录：@Log +- 缓存控制：@Cache +- 权限检查：@RequireRole +- 重试机制：@Retry +- 限流控制：@RateLimit + +**最佳实践**： +- 注解语义清晰，避免过度设计 +- 提供合理的默认值 +- 考虑注解的组合使用 +- 完善的文档和示例" + +**💻 代码示例**： +```java +// 自定义日志注解 +@Target(ElementType.METHOD) +@Retention(RetentionPolicy.RUNTIME) +@Documented +public @interface Log { + String value() default ""; + LogLevel level() default LogLevel.INFO; + boolean logParams() default true; + boolean logResult() default true; +} + +// 日志处理切面 +@Aspect +@Component +public class LogAspect { + + private static final Logger logger = LoggerFactory.getLogger(LogAspect.class); + + @Around("@annotation(log)") + public Object doLog(ProceedingJoinPoint point, Log log) throws Throwable { + String methodName = point.getSignature().getName(); + Object[] args = point.getArgs(); + + // 记录方法调用 + if (log.logParams()) { + logger.info("调用方法: {}, 参数: {}", methodName, args); + } + + long startTime = System.currentTimeMillis(); + Object result = point.proceed(); + long endTime = System.currentTimeMillis(); + + // 记录方法结果 + if (log.logResult()) { + logger.info("方法: {} 执行完成, 耗时: {}ms, 结果: {}", + methodName, endTime - startTime, result); + } + + return result; + } +} + +// 自定义缓存注解 +@Target(ElementType.METHOD) +@Retention(RetentionPolicy.RUNTIME) +public @interface Cache { + String key() default ""; + int expire() default 300; // 5分钟 + TimeUnit timeUnit() default TimeUnit.SECONDS; +} + +// 缓存处理器 +@Component +public class CacheAnnotationProcessor implements BeanPostProcessor { + + @Override + public Object postProcessAfterInitialization(Object bean, String beanName) { + Class clazz = bean.getClass(); + for (Method method : clazz.getDeclaredMethods()) { + if (method.isAnnotationPresent(Cache.class)) { + // 创建缓存代理 + return createCacheProxy(bean, method); + } + } + return bean; + } + + private Object createCacheProxy(Object bean, Method method) { + return Proxy.newProxyInstance( + bean.getClass().getClassLoader(), + bean.getClass().getInterfaces(), + new CacheInvocationHandler(bean, method) + ); + } +} + +// 使用自定义注解 +@Service +public class UserService { + + @Log("查询用户信息") + @Cache(key = "user:#{args[0]}", expire = 600) + public User findById(Long id) { + return userRepository.findById(id).orElse(null); + } + + @Log(value = "创建用户", logResult = false) + public User createUser(User user) { + return userRepository.save(user); + } +} +``` + + + +### 🎯 Spring Boot 注解 原理? + +**Spring Boot** 是基于 Spring 框架的一个开源框架，旨在简化 Spring 应用程序的配置和部署。Spring Boot 利用了大量的自动配置和开箱即用的功能，其中许多功能都通过注解来实现。理解 Spring Boot 的注解原理，能够帮助我们更好地理解其自动化配置的过程，以及如何扩展和自定义 Spring Boot 应用。 + +**常见的 Spring Boot 注解及其原理** + +1. `@SpringBootApplication：复合注解，通常用在主类上，启动 Spring Boot 应用程序。它实际上包含了三个注解： + + - `@SpringBootConfiguration`：标识当前类为配置类，类似于 `@Configuration`，表示该类是 Spring 配置类，会被 Spring 容器加载。 + + - `@EnableAutoConfiguration`：启用 Spring Boot 的自动配置功能，让 Spring Boot 根据项目的依赖和配置自动做出配置选择。 + + - `@ComponentScan`：启用组件扫描，默认会扫描当前包及其子包下的所有 Spring 组件（包括 `@Component`、`@Service`、`@Repository`、`@Controller` 等注解）。 + + **原理**：`@SpringBootApplication` 通过组合这些注解，实现了自动配置、组件扫描和 Spring 配置的功能，简化了开发人员的配置工作。 + +2. `@EnableAutoConfiguration`：开启了自动配置功能。它告诉 Spring Boot 自动为应用程序配置所需要的组件，避免了手动配置大量的 Bean。 + + **原理**：`@EnableAutoConfiguration` 是通过 **条件化注解** (`@Conditional` 系列注解) 来实现的，Spring Boot 会根据项目的依赖（如类路径中的 jar 包）来判断是否启用相应的自动配置类。例如，如果项目中有 `spring-boot-starter-web` 依赖，Spring Boot 会自动配置 Tomcat、DispatcherServlet 等 Web 相关组件。 + +3. `@Configuration`：标记该类是一个配置类，类似于传统的 XML 配置文件，用于声明和定义 Spring Bean。 + + **原理**：Spring 在类上使用 `@Configuration` 时，会将该类作为一个配置类处理。通过解析该类中的 `@Bean` 注解来生成 Spring Bean。`@Configuration` 实际上是 `@Component` 注解的扩展，意味着配置类也会被 Spring 容器管理。 + +4. `@ComponentScan`：用于告诉 Spring 框架去扫描指定包（及其子包）中的类，并将其作为 Spring Bean 进行注册。通常该注解与 `@Configuration` 一起使用。 + + **原理**：Spring Boot 启动类使用 `@ComponentScan` 默认扫描当前包及其子包，查找 `@Component`、`@Service`、`@Repository` 等注解标注的类，将它们加入到 Spring 容器中。如果需要更改扫描路径，可以在注解中指定 `basePackages` 属性。 + +5. `@Conditional` ：是一种根据某些条件决定是否加载 Bean 的机制。在 Spring Boot 中，自动配置大量使用了 `@Conditional` 注解，以实现基于环境或配置的条件加载。 + + **原理**：Spring Boot 使用 `@Conditional` 注解和多个具体的条件类（如 `@ConditionalOnProperty`、`@ConditionalOnClass`、`@ConditionalOnMissingBean` 等）来判断是否需要创建某些 Bean。例如，`@ConditionalOnClass` 会检查类路径中是否存在某个类，从而决定是否启用某个配置。 + +**Spring Boot 注解的工作原理总结** + +1. **自动配置（`@EnableAutoConfiguration`）**：通过条件化注解，Spring Boot 根据项目的依赖和配置，自动选择并加载合适的配置类。每个自动配置类通常都有一个 `@Conditional` 注解，确保仅在满足特定条件时才会加载。 +2. **组件扫描（`@ComponentScan`）**：Spring Boot 默认扫描主应用类所在包及其子包，自动将符合条件的类（如 `@Component` 注解的类）注册为 Spring Bean。 +3. **配置绑定（`@Value` 和 `@ConfigurationProperties`）**：Spring Boot 提供的注解允许将配置文件中的值自动注入到 Java 类中，使得配置管理更加方便。 +4. **Web 控制（`@RestController`）**：`@RestController` 注解结合了 `@Controller` 和 `@ResponseBody`，使得开发 RESTful API 更加简洁。 + +> **Spring Boot 注解** 在 Spring Boot 中发挥了重要作用，简化了开发流程。最核心的注解包括： +> +> - **@SpringBootApplication**：复合注解，包含了 `@Configuration`、`@EnableAutoConfiguration` 和 `@ComponentScan`，负责配置类的定义、自动配置的启用和组件扫描的开启。 +> - **@EnableAutoConfiguration**：启用 Spring Boot 自动配置，根据类路径、环境等信息自动配置应用所需的组件。 +> - **@ConfigurationProperties** 和 **@Value**：用于将外部配置（如 `application.properties`）注入到 Java 类中，`@ConfigurationProperties` 适合批量注入复杂配置，而 `@Value` 更适用于简单的单个值注入。 +> - **@RestController**：简化 REST API 开发，结合了 `@Controller` 和 `@ResponseBody` 注解，使得方法返回的对象直接作为 HTTP 响应的内容。 +> +> 这些注解共同作用，通过自动配置和条件化加载，为开发者提供了高效、简便的配置和开发体验。 + +--- + + + +## 🔧 九、高级特性与实践 + +**核心理念**：Spring提供了丰富的高级特性，包括事件机制、缓存抽象、国际化支持等，助力企业级应用开发。 + +| 模块 | 面试问题 | 考察点 | +| ------------------------ | -------------------------------------------------- | -------------------------------------------------------- | +| BeanDefinition | Spring 是如何解析 @Bean、@Component 的？ | BeanDefinitionReader、AnnotationConfigApplicationContext | +| refresh() 方法 | ApplicationContext 刷新流程？ | prepare → load → instantiate → inject → finishRefresh | +| BeanFactoryPostProcessor | 这个接口的作用？ | 修改 BeanDefinition 元信息 | +| 环境与属性加载 | Environment、PropertySource 的作用？ | 外部化配置解析 | +| Aware 接口体系 | ApplicationContextAware / BeanNameAware 有什么用？ | 让 Bean 获取容器资源 | +| Spring 事件机制 | ApplicationEvent 与 Listener 的原理？ | 观察者模式 | +| 延迟加载 | @Lazy 的底层原理？ | CGLIB 代理 + 延迟初始化 | + +| 类型 | 题目 | 考察方向 | +| -------- | ----------------------------------------------- | ------------------------------------------------------- | +| 架构题 | 如何在 Spring 项目中实现插件化？ | ClassLoader + ApplicationContext 分层 | +| 架构题 | Spring 容器启动慢怎么优化？ | 懒加载、Bean 预编译、减少反射 | +| 框架集成 | Spring 与 MyBatis 是如何整合的？ | SqlSessionTemplate、MapperFactoryBean | +| 多线程 | Spring Bean 在多线程下安全吗？ | 无状态单例安全、有状态需 ThreadLocal | +| 热加载 | Spring Boot DevTools 实现原理？ | ClassLoader 重新加载 | +| 性能调优 | 如何优化 Spring Bean 初始化性能？ | 缓存反射信息、减少代理层级 | +| 源码理解 | ApplicationContext.refresh() 的主要阶段有哪些？ | prepare、invokeBeanFactoryPostProcessors、finishRefresh | + +### 🎯 Spring的事件机制怎么使用？ + +"Spring事件机制基于观察者模式，实现组件间的解耦通信： + +**核心组件**： + +**ApplicationEvent**： +- 事件基类，所有自定义事件需继承 +- 包含事件源和时间戳信息 +- 可以携带自定义数据 + +**ApplicationEventPublisher**： +- 事件发布器接口 +- ApplicationContext实现了此接口 +- 通过publishEvent方法发布事件 + +**ApplicationListener**： +- 事件监听器接口 +- 泛型指定监听的事件类型 +- 可以使用@EventListener注解简化 + +**事件处理特点**： +- 默认同步执行，可配置异步 +- 支持事件过滤和条件监听 +- 自动类型匹配，支持继承关系 +- 可以设置监听器优先级 + +**应用场景**： +- 业务解耦：订单创建后通知库存、积分等 +- 系统监控：性能指标收集 +- 缓存更新：数据变更后缓存失效 +- 日志审计：操作记录和审计日志" + +**💻 代码示例**： +```java +// 自定义事件 +public class OrderCreatedEvent extends ApplicationEvent { + private final Order order; + + public OrderCreatedEvent(Object source, Order order) { + super(source); + this.order = order; + } + + public Order getOrder() { + return order; + } +} + +// 事件发布 +@Service +public class OrderService { + + @Autowired + private ApplicationEventPublisher eventPublisher; + + @Transactional + public Order createOrder(Order order) { + Order savedOrder = orderRepository.save(order); + + // 发布事件 + OrderCreatedEvent event = new OrderCreatedEvent(this, savedOrder); + eventPublisher.publishEvent(event); + + return savedOrder; + } +} + +// 事件监听 +@Component +public class OrderEventListener { + + // 库存处理 + @EventListener + @Async + public void handleOrderCreated(OrderCreatedEvent event) { + Order order = event.getOrder(); + inventoryService.reserveStock(order.getItems()); + } + + // 积分处理 + @EventListener + @Order(1) // 设置优先级 + public void handlePointsCalculation(OrderCreatedEvent event) { + Order order = event.getOrder(); + pointService.calculatePoints(order); + } + + // 条件监听 + @EventListener(condition = "#event.order.amount > 1000") + public void handleVipOrder(OrderCreatedEvent event) { + // 处理VIP订单逻辑 + vipService.processVipOrder(event.getOrder()); + } +} + +// 异步事件配置 +@Configuration +@EnableAsync +public class AsyncEventConfig { + + @Bean + public TaskExecutor taskExecutor() { + ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor(); + executor.setCorePoolSize(5); + executor.setMaxPoolSize(10); + executor.setQueueCapacity(25); + executor.setThreadNamePrefix("Event-"); + executor.initialize(); + return executor; + } +} +``` + +### 🎯 Spring Cache缓存抽象怎么使用？ + +"Spring Cache提供了声明式缓存抽象，支持多种缓存实现： + +**核心注解**： + +**@Cacheable**： +- 方法结果缓存，如果缓存存在直接返回 +- 适用于查询方法 +- 支持条件缓存和SpEL表达式 + +**@CacheEvict**： +- 缓存清除，删除指定缓存 +- 适用于更新、删除操作 +- 支持批量清除和条件清除 + +**@CachePut**： +- 缓存更新，总是执行方法并更新缓存 +- 适用于更新操作 + +**@Caching**： +- 组合缓存操作 +- 支持同时使用多个缓存注解 + +**缓存实现**： +- ConcurrentHashMap：简单内存缓存 +- Redis：分布式缓存 +- Ehcache：本地缓存 +- Caffeine：高性能本地缓存 + +**高级特性**： +- 缓存穿透保护 +- 缓存雪崩保护 +- 自定义缓存键生成策略 +- 缓存统计和监控" + +**💻 代码示例**： +```java +// 启用缓存 +@Configuration +@EnableCaching +public class CacheConfig { + + @Bean + public CacheManager cacheManager() { + RedisCacheManager.Builder builder = RedisCacheManager + .RedisCacheManagerBuilder + .fromConnectionFactory(jedisConnectionFactory()) + .cacheDefaults(cacheConfiguration()); + return builder.build(); + } + + private RedisCacheConfiguration cacheConfiguration() { + return RedisCacheConfiguration.defaultCacheConfig() + .entryTtl(Duration.ofMinutes(30)) + .disableCachingNullValues(); + } +} + +@Service +public class UserService { + + // 缓存查询结果 + @Cacheable(value = "users", key = "#id") + public User findById(Long id) { + return userRepository.findById(id).orElse(null); + } + + // 条件缓存 + @Cacheable(value = "users", key = "#username", condition = "#username.length() > 3") + public User findByUsername(String username) { + return userRepository.findByUsername(username); + } + + // 缓存更新 + @CachePut(value = "users", key = "#user.id") + public User updateUser(User user) { + return userRepository.save(user); + } + + // 缓存清除 + @CacheEvict(value = "users", key = "#id") + public void deleteUser(Long id) { + userRepository.deleteById(id); + } + + // 清除所有缓存 + @CacheEvict(value = "users", allEntries = true) + public void clearAllUserCache() { + // 清空所有用户缓存 + } + + // 组合缓存操作 + @Caching( + put = @CachePut(value = "users", key = "#user.id"), + evict = @CacheEvict(value = "userStats", allEntries = true) + ) + public User saveUser(User user) { + return userRepository.save(user); + } +} +``` + +### 🎯 Spring如何实现国际化？ + +"Spring提供了完整的国际化（i18n）支持： + +**核心组件**： + +**MessageSource**： +- 消息资源接口，管理国际化消息 +- ResourceBundleMessageSource：基于properties文件 +- ReloadableResourceBundleMessageSource：支持热加载 + +**LocaleResolver**： + +- 语言环境解析器，确定用户的Locale +- AcceptHeaderLocaleResolver：基于HTTP头 +- SessionLocaleResolver：基于Session +- CookieLocaleResolver：基于Cookie + +**LocaleChangeInterceptor**： +- 语言切换拦截器 +- 监听语言切换请求参数 + +**实现步骤**： +1. 配置MessageSource和LocaleResolver +2. 创建不同语言的资源文件 +3. 添加语言切换拦截器 +4. 在代码中使用MessageSource获取消息 +5. 在JSP/Thymeleaf模板中使用国际化标签 + +**最佳实践**： +- 统一的资源文件命名规范 +- 使用键值对管理消息 +- 支持参数化消息 +- 提供友好的语言切换界面" + +**💻 代码示例**： +```java +// 国际化配置 +@Configuration +public class I18nConfig implements WebMvcConfigurer { + + @Bean + public MessageSource messageSource() { + ResourceBundleMessageSource messageSource = new ResourceBundleMessageSource(); + messageSource.setBasename("messages"); + messageSource.setDefaultEncoding("UTF-8"); + return messageSource; + } + + @Bean + public LocaleResolver localeResolver() { + SessionLocaleResolver resolver = new SessionLocaleResolver(); + resolver.setDefaultLocale(Locale.CHINA); + return resolver; + } + + @Bean + public LocaleChangeInterceptor localeChangeInterceptor() { + LocaleChangeInterceptor interceptor = new LocaleChangeInterceptor(); + interceptor.setParamName("lang"); + return interceptor; + } + + @Override + public void addInterceptors(InterceptorRegistry registry) { + registry.addInterceptor(localeChangeInterceptor()); + } +} + +// 资源文件 +// messages_zh_CN.properties +user.name=用户名 +user.age=年龄 +welcome.message=欢迎 {0}! + +// messages_en_US.properties +user.name=Username +user.age=Age +welcome.message=Welcome {0}! + +// 控制器使用 +@Controller +public class HomeController { + + @Autowired + private MessageSource messageSource; + + @GetMapping("/welcome") + public String welcome(@RequestParam String username, + Model model, HttpServletRequest request) { + Locale locale = RequestContextUtils.getLocale(request); + String message = messageSource.getMessage("welcome.message", + new Object[]{username}, locale); + model.addAttribute("message", message); + return "welcome"; + } +} + +// 工具类 +@Component +public class I18nUtil { + + @Autowired + private MessageSource messageSource; + + public String getMessage(String key, Object... args) { + Locale locale = LocaleContextHolder.getLocale(); + return messageSource.getMessage(key, args, locale); + } +} +``` + +### 🎯 Spring WebFlux响应式编程怎么使用？ + +> 完全异步非阻塞的web框架，通过Reactor项目实现了Reactive Streams 规范，内置了netty容器。我们使用一般基于 Mono、Flux 数据流，像 Stream API 那样可以用 map、zip、filter 等，也有 onErrorReturn、onErrorMap 等。 +> +> 基于事件驱动：依赖少量线程处理 + +"Spring WebFlux是Spring 5引入的响应式Web框架，基于Reactor实现： + +**核心概念**： + +**响应式编程**： +- 基于异步数据流的编程模式 +- 采用观察者模式，数据流驱动 +- 非阻塞I/O，提高系统吞吐量 +- 支持背压（Backpressure）处理 + +**Reactor核心类型**： +- Mono：表示0到1个元素的异步序列 +- Flux：表示0到N个元素的异步序列 +- 支持链式操作和函数式编程 + +**WebFlux vs Spring MVC**： + +**Spring MVC**： +- 基于Servlet API，同步阻塞模型 +- 每个请求对应一个线程 +- 适合传统的CRUD应用 +- 成熟稳定，生态完善 + +**Spring WebFlux**： +- 基于Reactive Streams，异步非阻塞 +- 少量线程处理大量请求 +- 适合高并发、I/O密集型应用 +- 支持函数式和注解式编程 + +**适用场景**： +- 高并发的微服务应用 +- 实时数据流处理 +- 服务间异步通信 +- SSE（Server-Sent Events）和WebSocket + +**技术栈支持**： +- Netty、Undertow等非阻塞服务器 +- R2DBC响应式数据库访问 +- Spring Cloud Gateway响应式网关 +- Redis Reactive、MongoDB Reactive等" + +**💻 代码示例**： + +```java +// WebFlux控制器 +@RestController +public class ReactiveUserController { + + @Autowired + private ReactiveUserService userService; + + // 返回单个用户 + @GetMapping("/users/{id}") + public Mono getUser(@PathVariable String id) { + return userService.findById(id) + .switchIfEmpty(Mono.error(new UserNotFoundException("用户不存在"))); + } + + // 返回用户列表 + @GetMapping("/users") + public Flux getUsers() { + return userService.findAll() + .delayElements(Duration.ofMillis(100)); // 模拟延迟 + } + + // 创建用户 + @PostMapping("/users") + public Mono createUser(@RequestBody Mono userMono) { + return userMono + .flatMap(userService::save) + .doOnSuccess(user -> log.info("用户创建成功: {}", user.getId())); + } + + // 流式处理 + @GetMapping(value = "/users/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE) + public Flux streamUsers() { + return userService.findAll() + .delayElements(Duration.ofSeconds(1)) + .repeat(); + } +} + +// 响应式Service +@Service +public class ReactiveUserService { + + @Autowired + private ReactiveUserRepository userRepository; + + public Mono findById(String id) { + return userRepository.findById(id) + .doOnNext(user -> log.debug("找到用户: {}", user)) + .doOnError(error -> log.error("查询用户失败: {}", error.getMessage())); + } + + public Flux findAll() { + return userRepository.findAll() + .filter(user -> user.isActive()) + .sort((u1, u2) -> u1.getName().compareTo(u2.getName())); + } + + public Mono save(User user) { + return Mono.just(user) + .filter(u -> StringUtils.hasText(u.getName())) + .switchIfEmpty(Mono.error(new IllegalArgumentException("用户名不能为空"))) + .flatMap(userRepository::save); + } + + // 批量处理 + public Flux saveAll(Flux users) { + return users + .filter(user -> StringUtils.hasText(user.getName())) + .flatMap(this::save) + .onErrorContinue((error, user) -> + log.error("保存用户失败: {}, 错误: {}", user, error.getMessage())); + } +} + +// 函数式编程风格 +@Configuration +public class RouterConfig { + + @Bean + public RouterFunction userRoutes(UserHandler userHandler) { + return RouterFunctions.route() + .GET("/api/users", userHandler::getAllUsers) + .GET("/api/users/{id}", userHandler::getUser) + .POST("/api/users", userHandler::createUser) + .PUT("/api/users/{id}", userHandler::updateUser) + .DELETE("/api/users/{id}", userHandler::deleteUser) + .build(); + } +} + +@Component +public class UserHandler { + + @Autowired + private ReactiveUserService userService; + + public Mono getAllUsers(ServerRequest request) { + return ServerResponse.ok() + .contentType(MediaType.APPLICATION_JSON) + .body(userService.findAll(), User.class); + } + + public Mono getUser(ServerRequest request) { + String id = request.pathVariable("id"); + return userService.findById(id) + .flatMap(user -> ServerResponse.ok().bodyValue(user)) + .switchIfEmpty(ServerResponse.notFound().build()); + } + + public Mono createUser(ServerRequest request) { + return request.bodyToMono(User.class) + .flatMap(userService::save) + .flatMap(user -> ServerResponse.status(HttpStatus.CREATED).bodyValue(user)) + .onErrorResume(error -> ServerResponse.badRequest().bodyValue(error.getMessage())); + } +} + +// WebClient响应式HTTP客户端 +@Service +public class ExternalApiService { + + private final WebClient webClient; + + public ExternalApiService() { + this.webClient = WebClient.builder() + .baseUrl("https://api.example.com/") + .defaultHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE) + .codecs(configurer -> configurer.defaultCodecs().maxInMemorySize(1024 * 1024)) + .build(); + } + + public Mono getUserInfo(String userId) { + return webClient.get() + .uri("/users/{id}", userId) + .retrieve() + .onStatus(HttpStatus::is4xxClientError, + response -> Mono.error(new RuntimeException("客户端错误"))) + .onStatus(HttpStatus::is5xxServerError, + response -> Mono.error(new RuntimeException("服务器错误"))) + .bodyToMono(UserInfo.class) + .timeout(Duration.ofSeconds(5)) + .retry(3); + } + + public Flux getAllUsers() { + return webClient.get() + .uri("/users") + .retrieve() + .bodyToFlux(UserInfo.class) + .onErrorReturn(Collections.emptyList()) + .flatMapIterable(list -> list); + } +} +``` + + + +### 🎯 springMVC 和 spring WebFlux 区别？ + +`Spring MVC` 和 `Spring WebFlux` 都是 Spring 框架中用于构建 Web 应用程序的模块，但它们有一些根本性的区别，特别是在处理请求的方式和支持的编程模型上。以下是两者的主要区别和工作原理的详细说明： + +1. **编程模型和架构** + +- Spring MVC（同步阻塞式） + + Spring MVC 是一个基于 Servlet 的 Web 框架，遵循经典的请求-响应模型。它是一个 **同步阻塞** 的框架，即每个请求都由一个独立的线程处理，直到请求完成，线程才会被释放。其工作流程如下： + + 1. **请求到达**：客户端发送请求到服务器，Tomcat 接收请求并交给 Spring MVC 处理。 + 2. **请求分发**：DispatcherServlet 将请求传递给合适的控制器（Controller）。 + 3. **处理请求**：控制器处理业务逻辑，调用服务层、数据访问层等。 + 4. **视图解析**：返回模型和视图（ModelAndView），然后交由视图解析器（如 JSP）渲染成最终的 HTML。 + 5. **响应返回**：最终的响应通过 Tomcat 返回到客户端。 + +​ Spring MVC 适用于传统的基于 Servlet 的 Web 应用，通常用于同步场景，像表单提交、处理重定向等。 + +2. **Spring WebFlux（异步非阻塞式）** + + Spring WebFlux 是一个 **异步非阻塞** 的 Web 框架，可以更好地处理高并发场景和 I/O 密集型操作。Spring WebFlux 适用于响应式编程模型，通过使用 **Reactor**（一个响应式编程库）提供的支持，能够以异步和非阻塞的方式处理请求。 + + Spring WebFlux 可以运行在多种容器上，除了传统的 Servlet 容器（如 Tomcat），它还支持基于 Netty 等非 Servlet 容器的运行模式。它的工作原理如下： + + 1. **请求到达**：客户端发送请求到服务器，Tomcat 或 Netty 接收请求。 + 2. **请求分发**：DispatcherHandler 将请求传递给合适的控制器。 + 3. **异步处理**：控制器在处理请求时，通常会返回一个 `Mono` 或 `Flux`（Reactor 库的异步数据流类型），这些类型代表了一个单一元素（Mono）或者多个元素（Flux）的异步响应。 + 4. **响应返回**：通过事件驱动的方式，最终响应异步返回给客户端。 + +​ Spring WebFlux 适合高并发、I/O 密集型和实时数据流场景，像聊天系统、实时通知、数据流处理等。 + +| 特性 | Spring MVC | Spring WebFlux | +| ------------------- | ----------------------------------------------- | ------------------------------------------------------ | +| **编程模型** | 同步阻塞模型（传统的请求-响应模型） | 异步非阻塞模型（响应式编程） | +| **线程模型** | 每个请求分配一个线程，阻塞等待响应 | 请求通过事件循环处理，异步返回结果 | +| **请求处理方式** | 阻塞，处理请求时，线程会被占用 | 非阻塞，线程可以用于其他任务，直到结果返回 | +| **I/O 模型** | 阻塞式 I/O | 非阻塞式 I/O | +| **适用场景** | 对同步请求较为适合，例如表单提交，传统 Web 应用 | 高并发、实时数据流处理，I/O 密集型场景，微服务 | +| **支持的容器** | Servlet 容器（如 Tomcat） | Servlet 容器（如 Tomcat）及非 Servlet 容器（如 Netty） | +| **响应体类型** | `ModelAndView`（同步返回） | `Mono` 和 `Flux`（异步返回） | +| **扩展性与性能** | 高并发时性能较差，容易出现线程饱和 | 高并发时性能更优，线程利用率更高 | +| **学习曲线** | 更易上手，熟悉的编程模型 | 学习曲线较陡，需要理解异步编程和响应式流 | +| **支持的 API 类型** | REST API、Web 应用 | 支持 REST API，且更适合长连接和实时通信 | + + + +### 🎯 Spring 框架中用到了哪些设计模式？ + +- **工厂设计模式** : Spring使用工厂模式通过 `BeanFactory`、`ApplicationContext` 创建 bean 对象。 +- **代理设计模式** : Spring AOP 功能的实现。 +- **单例设计模式** : Spring 中的 Bean 默认都是单例的。 +- **模板方法模式** : Spring 中 `jdbcTemplate`、`hibernateTemplate` 等以 Template 结尾的对数据库操作的类，它们就使用到了模板模式。 +- **包装器设计模式** : 我们的项目需要连接多个数据库，而且不同的客户在每次访问中根据需要会去访问不同的数据库。这种模式让我们可以根据客户的需求能够动态切换不同的数据源。 +- **观察者模式:** Spring 事件驱动模型就是观察者模式很经典的一个应用。 +- **适配器模式** :Spring AOP 的增强或通知(Advice)使用到了适配器模式、spring MVC 中也是用到了适配器模式适配`Controller`。 + +--- + + + +## 🎯 十、面试重点与实践 + +**核心理念**：掌握Spring面试的关键技巧和常见陷阱，结合实际项目经验展现技术深度。 + +### 🎯 Spring面试常见陷阱和注意事项？ + +"Spring面试中有一些常见的陷阱和需要注意的点： + +**概念混淆陷阱**： +- BeanFactory vs ApplicationContext：功能和使用场景区别 +- @Component vs @Bean：注解驱动 vs Java配置 +- AOP vs 过滤器：实现机制和适用场景不同 + +**生命周期理解陷阱**： +- Bean生命周期的完整流程要准确 +- 循环依赖的解决机制要清楚 +- 事务失效的常见场景要熟悉 + +**性能相关陷阱**： +- 单例Bean的线程安全问题 +- @Transactional的性能影响 +- AOP代理的创建开销 + +**配置相关陷阱**： +- 注解扫描范围要合理 +- 配置文件的优先级顺序 +- Profile环境配置的使用 + +**实践经验陷阱**： +- 不要只停留在理论层面 +- 要能结合具体项目场景 +- 要了解Spring在微服务中的应用 + +**回答技巧**： +- 先回答核心概念，再展开细节 +- 结合代码示例说明 +- 提及性能和最佳实践 +- 展现解决实际问题的能力" + +### 🎯 Spring Boot vs Spring Framework的区别？ + +"Spring Boot和Spring Framework是包含关系，不是替代关系： + +**Spring Framework**： +- Spring生态的核心基础框架 +- 提供IoC、AOP、事务管理等核心功能 +- 需要大量XML或Java配置 +- 学习曲线较陡峭，配置复杂 + +**Spring Boot**： +- 基于Spring Framework的快速开发框架 +- 提供自动配置和起步依赖 +- 约定大于配置，零配置启动 +- 内嵌Web服务器，简化部署 + +**核心区别**： + +**配置方式**： +- Spring Framework：需要显式配置Bean +- Spring Boot：自动配置，按需覆盖 + +**依赖管理**： +- Spring Framework：手动管理版本兼容 +- Spring Boot：Starter依赖，版本仲裁 + +**部署方式**： +- Spring Framework：需要外部Web容器 +- Spring Boot：内嵌容器，jar包直接运行 + +**开发效率**： +- Spring Framework：配置繁琐，开发慢 +- Spring Boot：快速启动，开发效率高 + +**适用场景**： +- Spring Framework：需要精确控制配置的场景 +- Spring Boot：快速开发，微服务架构" + +### 🎯 如何设计一个Spring应用的整体架构？ + +"设计Spring应用架构需要考虑多个方面： + +**分层架构设计**： + +**表现层（Presentation Layer）**： +- 使用Spring MVC处理HTTP请求 +- RestController提供RESTful API +- 参数校验和异常统一处理 + +**业务层（Business Layer）**： +- Service层封装业务逻辑 +- 使用@Transactional管理事务 +- 业务规则验证和流程控制 + +**数据访问层（Data Access Layer）**： +- Repository模式封装数据访问 +- Spring Data JPA简化CRUD操作 +- 多数据源配置和读写分离 + +**技术架构选型**： + +**配置管理**： +- Spring Boot的外部化配置 +- Config Server集中配置管理 +- Profile环境区分 + +**安全框架**： +- Spring Security统一安全管理 +- JWT无状态认证 +- 方法级权限控制 + +**缓存策略**： +- Spring Cache抽象层 +- Redis分布式缓存 +- 多级缓存设计 + +**监控运维**： +- Spring Boot Actuator健康检查 +- Micrometer指标收集 +- 分布式链路追踪 + +**微服务架构**： +- Spring Cloud服务治理 +- API网关统一入口 +- 服务间通信和容错 + +**最佳实践**： +- 遵循SOLID设计原则 +- 合理的包结构组织 +- 统一的异常处理机制 +- 完善的单元测试覆盖" + +**💻 代码示例**： +```java +// 分层架构示例 +@RestController +@RequestMapping("/api/users") +public class UserController { + + @Autowired + private UserService userService; + + @GetMapping("/{id}") + public ResponseEntity getUser(@PathVariable Long id) { + User user = userService.findById(id); + UserDTO dto = UserMapper.toDTO(user); + return ResponseEntity.ok(dto); + } + + @PostMapping + public ResponseEntity createUser(@RequestBody @Valid CreateUserRequest request) { + User user = userService.createUser(request); + UserDTO dto = UserMapper.toDTO(user); + return ResponseEntity.status(HttpStatus.CREATED).body(dto); + } +} + +@Service +@Transactional +public class UserService { + + @Autowired + private UserRepository userRepository; + @Autowired + private EmailService emailService; + @Autowired + private ApplicationEventPublisher eventPublisher; + + public User createUser(CreateUserRequest request) { + // 业务逻辑验证 + validateUserRequest(request); + + // 创建用户 + User user = User.builder() + .username(request.getUsername()) + .email(request.getEmail()) + .build(); + + User savedUser = userRepository.save(user); + + // 发送欢迎邮件 + emailService.sendWelcomeEmail(savedUser); + + // 发布用户创建事件 + eventPublisher.publishEvent(new UserCreatedEvent(savedUser)); + + return savedUser; + } + + private void validateUserRequest(CreateUserRequest request) { + if (userRepository.existsByUsername(request.getUsername())) { + throw new BusinessException("用户名已存在"); + } + if (userRepository.existsByEmail(request.getEmail())) { + throw new BusinessException("邮箱已存在"); + } + } +} + +@Repository +public interface UserRepository extends JpaRepository { + boolean existsByUsername(String username); + boolean existsByEmail(String email); + + @Query("SELECT u FROM User u WHERE u.status = 'ACTIVE'") + List findActiveUsers(); +} + +// 全局异常处理 +@ControllerAdvice +public class GlobalExceptionHandler { + + @ExceptionHandler(BusinessException.class) + @ResponseStatus(HttpStatus.BAD_REQUEST) + public ErrorResponse handleBusinessException(BusinessException e) { + return ErrorResponse.builder() + .code("BUSINESS_ERROR") + .message(e.getMessage()) + .build(); + } + + @ExceptionHandler(MethodArgumentNotValidException.class) + @ResponseStatus(HttpStatus.BAD_REQUEST) + public ErrorResponse handleValidationException(MethodArgumentNotValidException e) { + Map errors = new HashMap<>(); + e.getBindingResult().getFieldErrors().forEach(error -> + errors.put(error.getField(), error.getDefaultMessage())); + + return ErrorResponse.builder() + .code("VALIDATION_ERROR") + .message("参数校验失败") + .errors(errors) + .build(); + } +} +``` +--- +## 📚 总结与进阶 +### 🎯 Spring学习路径建议 +**基础阶段**： +1. **Spring Framework核心**：IoC、DI、AOP基础概念 +2. **Spring MVC**：Web开发基础，请求处理流程 +3. **Spring Data**：数据访问和事务管理 +4. **Spring Security**：安全框架基础 +**进阶阶段**： +1. **Spring Boot**：自动配置、Starter机制、监控 +2. **Spring Cloud**：微服务架构、服务治理 +3. **响应式编程**：WebFlux、Reactive Streams +4. **Native编译**：GraalVM Native Image +**实战阶段**： +1. **企业级项目**：完整的业务系统开发 +2. **性能优化**：JVM调优、缓存策略、数据库优化 +3. **架构设计**：微服务拆分、分布式系统设计 +4. **DevOps实践**：CI/CD、容器化、云原生部署 +### ⚡ 面试准备清单 +- [ ] Spring Framework核心原理（IoC、AOP、Bean生命周期） +- [ ] Spring Boot自动配置和Starter机制 +- [ ] Spring MVC请求处理流程和核心组件 +- [ ] Spring事务管理和传播机制 +- [ ] Spring Security认证授权流程 +- [ ] Spring Cloud微服务组件 +- [ ] 常用注解的作用和原理 +- [ ] 实际项目中Spring的应用经验 +- [ ] Spring相关的性能优化实践 +- [ ] Spring生态的发展趋势 +### 🚀 技术发展趋势 -## 参考与来源 +**云原生时代**： +- Spring Boot 3.x和Spring Framework 6.x +- 基于GraalVM的Native编译 +- Kubernetes原生支持 +- Serverless架构适配 -https://www.edureka.co/blog/interview-questions/spring-interview-questions/ +**响应式编程**： +- Spring WebFlux异步非阻塞 +- R2DBC响应式数据库访问 +- 事件驱动架构设计 +**人工智能集成**： +- Spring AI项目 +- 向量数据库集成 +- 机器学习模型服务化 +--- +**🎯 面试成功秘诀**：不仅要掌握Spring的理论知识，更要结合实际项目经验，展现解决实际问题的能力。准备好详细的项目案例，能够深入讲解Spring在项目中的应用和遇到的挑战及解决方案。记住：**技术深度 + 实战经验 + 清晰表达 = 面试成功**！ -https://crossoverjie.top/2018/07/29/java-senior/ThreadPool/ \ No newline at end of file +**🔥 最后提醒**：Spring生态庞大且发展迅速，保持持续学习的心态，关注官方文档和社区动态，在实践中不断提升技术深度和广度。祝你面试顺利，offer满满！ 🎉 \ No newline at end of file diff --git a/docs/interview/ZooKeeper-FAQ.md b/docs/interview/ZooKeeper-FAQ.md new file mode 100644 index 0000000000..81a9323db5 --- /dev/null +++ b/docs/interview/ZooKeeper-FAQ.md @@ -0,0 +1,856 @@ +--- +title: ZooKeeper 核心面试八股文 +date: 2023-06-31 +tags: + - ZooKeeper + - Interview +categories: Interview +--- + +![](https://img.starfish.ink/common/faq-banner.png) + +> ZooKeeper是Apache的分布式协调服务框架，也是面试官考察**分布式系统理解**的核心知识点。从基础概念到一致性协议，从集群管理到典型应用，每一个知识点都体现着对分布式架构的深度理解。本文档将**最常考的ZK知识点**整理成**标准话术**，助你在面试中展现分布式技术功底！ + +### 🔥 为什么ZK如此重要？ + +- **📈 分布式必备**：90%的分布式系统都会涉及ZK相关技术 +- **🧠 架构体现**：体现你对分布式一致性、选举、协调的深度理解 +- **💼 工作基础**：配置中心、服务发现、分布式锁等场景无处不在 +- **🎓 技术进阶**：理解ZK是掌握分布式系统设计的关键一步 + +--- + +## 🗺️ 知识导航 + +### 🏷️ 核心知识分类 + +1. **🔥 基础概念类**：ZK定义、核心功能、数据模型、节点类型 +2. **📊 一致性协议**：2PC/3PC、Paxos算法、ZAB协议原理 +3. **🌐 集群与选举**：选举机制、节点角色、故障处理、数据同步 +4. **⚡ 核心特性**：Watcher机制、ACL权限、会话管理、数据一致性 +5. **🔧 典型应用**：分布式锁、配置管理、服务发现、负载均衡 +6. **🚨 运维实践**：集群部署、性能监控、故障排查、最佳实践 +7. **💼 对比分析**：ZK vs 其他中间件、应用场景选择 + +### 🔑 面试话术模板 + +| **问题类型** | **回答框架** | **关键要点** | **深入扩展** | +| ------------ | ----------------------------------- | ------------------ | ------------------ | +| **概念解释** | 定义→特点→应用场景→示例 | 准确定义，突出特点 | 底层原理，协议分析 | +| **对比分析** | 相同点→不同点→使用场景→选择建议 | 多维度对比 | 性能差异，实际应用 | +| **原理解析** | 背景→实现机制→执行流程→注意事项 | 图解流程 | 协议层面，算法细节 | +| **实践应用** | 问题现象→分析思路→解决方案→监控验证 | 实际案例 | 最佳实践，踩坑经验 | + +--- + +## 🔥 一、基础概念类（ZK核心） + +> **核心思想**：ZooKeeper是分布式协调服务的经典实现，提供统一的命名空间、数据发布/订阅、分布式同步等核心功能。 + +- **ZK基础定义**：[ZooKeeper是什么](#🎯-谈下你对-zookeeper-的认识) | [ZK核心功能](#🎯-zookeeper-都有哪些功能) +- **数据模型**：[文件系统](#🎯-zookeeper-文件系统) | [节点类型](#🎯-说下四种类型的数据节点-znode) +- **核心特性**：[数据一致性](#🎯-zookeeper-如何保证数据一致性) | [会话管理](#🎯-zookeeper-会话管理) + +### 🎯 谈下你对 Zookeeper 的认识？ + +ZooKeeper 是一个**分布式协调服务框架**，为分布式应用提供一致性服务。它的核心作用是： + +**定义**：ZooKeeper 是 Apache 的开源分布式协调服务，基于观察者模式设计的分布式服务管理框架。 + +**核心特点**： +1. **顺序一致性** - 按照客户端发送请求的顺序执行 +2. **原子性** - 事务要么成功要么失败，不存在中间状态 +3. **单一视图** - 客户端看到的服务端数据模型都是一致的 +4. **可靠性** - 一旦事务被确认，将被持久化保存直到客户端覆盖更新 +5. **实时性** - 保证客户端将在一个时间间隔范围内获得服务器的更新信息 + +**应用场景**：配置管理、服务发现、分布式锁、集群管理、负载均衡等。 + +### 🎯 Zookeeper 都有哪些功能？ + +1. **集群管理**：监控节点存活状态、运行请求等 +2. **主节点选举**：协助完成分布式系统中的Leader选举过程 +3. **分布式锁**：提供独占锁和共享锁，实现分布式环境下的资源控制 +4. **命名服务**：提供统一的命名空间，通过名称获取资源或服务地址 +5. **统一配置管理**：集群中所有节点的配置信息保持一致，支持动态配置更新 +6. **负载均衡**：配合客户端实现服务的负载均衡 +7. **数据发布/订阅**：支持数据的发布和订阅模式 + +### 🎯 Zookeeper 文件系统 + +ZooKeeper 提供一个**多层级的节点命名空间**（节点称为 znode），类似于文件系统的树状结构。 + +**关键特点**： +- **内存存储**：为保证高吞吐和低延迟，ZK在内存中维护树状目录结构 +- **数据限制**：每个节点最多存储 **1MB** 数据，适合存储配置信息而非大数据 +- **路径唯一**：每个节点都有唯一的路径标识 +- **数据+子节点**：与传统文件系统不同，ZK的每个节点都可以存储数据和拥有子节点 + +### 🎯 说下四种类型的数据节点 Znode？ + +1. **PERSISTENT（持久节点）**： + - 除非手动删除，否则节点一直存在于 Zookeeper 上 + - 适合存储需要持久化的配置信息 + +2. **EPHEMERAL（临时节点）**： + - 生命周期与客户端会话绑定 + - 客户端会话失效时，临时节点自动被删除 + - 适合实现服务注册、心跳检测等场景 + +3. **PERSISTENT_SEQUENTIAL（持久顺序节点）**： + - 基本特性同持久节点，增加顺序属性 + - 节点名后追加一个由父节点维护的自增整型数字 + - 适合实现分布式队列等需要顺序的场景 + +4. **EPHEMERAL_SEQUENTIAL（临时顺序节点）**： + - 基本特性同临时节点，增加顺序属性 + - 适合实现分布式锁、选举等场景 + +--- + + + +## 📊 二、一致性协议（核心原理） + +> **核心思想**：理解分布式一致性协议是掌握ZooKeeper的关键，包括2PC/3PC的演进、Paxos算法的精髓，以及ZAB协议的实现细节。 + +- **经典协议**：[2PC协议](#🎯-一致性协议2pc3pc) | [3PC协议](#🎯-一致性协议2pc3pc) +- **Paxos算法**：[Paxos原理](#🎯-讲一讲-paxos-算法) | [死循环问题](#🎯-paxos-死循环问题) +- **ZAB协议**：[ZAB协议原理](#🎯-谈下你对-zab-协议的了解) | [ZAB vs Paxos对比](#🎯-zab-和-paxos-算法的联系与区别) + +### 🎯 一致性协议2PC、3PC？ + +#### 2PC（Two-Phase Commit） + +**阶段一：提交事务请求（”投票阶段“）** + +当要执行一个分布式事务的时候，事务发起者首先向协调者发起事务请求，然后协调者会给所有参与者发送 `prepare` 请求（其中包括事务内容）告诉参与者你们需要执行事务了，如果能执行我发的事务内容那么就先执行但不提交，执行后请给我回复。然后参与者收到 `prepare` 消息后，他们会开始执行事务（但不提交），并将 `Undo` 和 `Redo` 信息记入事务日志中，之后参与者就向协调者反馈是否准备好了 + +**阶段二：执行事务提交** + +协调者根据各参与者的反馈情况决定最终是否可以提交事务，如果反馈都是Yes，发送提交`commit`请求，参与者提交成功后返回 `Ack` 消息，协调者接收后就完成了。如果反馈是No 或者超时未反馈，发送 `Rollback` 请求，利用阶段一记录表的 `Undo` 信息执行回滚，并反馈给协调者`Ack` ，中断消息 + +![](https://tva1.sinaimg.cn/large/00831rSTly1gclosfvncqj30hs09j0td.jpg) + +优点：原理简单、实现方便。 + +缺点： + +- **单点故障问题**，如果协调者挂了那么整个系统都处于不可用的状态了 +- **阻塞问题**，即当协调者发送 `prepare` 请求，参与者收到之后如果能处理那么它将会进行事务的处理但并不提交，这个时候会一直占用着资源不释放，如果此时协调者挂了，那么这些资源都不会再释放了，这会极大影响性能 +- **数据不一致问题**，比如当第二阶段，协调者只发送了一部分的 `commit` 请求就挂了，那么也就意味着，收到消息的参与者会进行事务的提交，而后面没收到的则不会进行事务提交，那么这时候就会产生数据不一致性问题 + + + +#### 3PC（Three-Phase Commit） + +3PC，是 Three-Phase-Comimit 的缩写，即「**三阶段提交**」，是二阶段的改进版，将二阶段提交协议的“提交事务请求”过程一分为二。 + +**阶段一：CanCommit** + +协调者向所有参与者发送 `CanCommit` 请求，参与者收到请求后会根据自身情况查看是否能执行事务，如果可以则返回 YES 响应并进入预备状态，否则返回 NO + +**阶段二：PreCommit** + +协调者根据参与者返回的响应来决定是否可以进行下面的 `PreCommit` 操作。如果上面参与者返回的都是 YES，那么协调者将向所有参与者发送 `PreCommit` 预提交请求，**参与者收到预提交请求后，会进行事务的执行操作，并将 Undo 和 Redo 信息写入事务日志中** ，最后如果参与者顺利执行了事务则给协调者返回成功的 `Ack` 响应。如果在第一阶段协调者收到了 **任何一个 NO** 的信息，或者 **在一定时间内** 并没有收到全部的参与者的响应，那么就会中断事务，它会向所有参与者发送中断请求 `abort`，参与者收到中断请求之后会立即中断事务，或者在一定时间内没有收到协调者的请求，它也会中断事务 + +**阶段三：DoCommit** + +这个阶段其实和 `2PC` 的第二阶段差不多，如果协调者收到了所有参与者在 `PreCommit` 阶段的 YES 响应，那么协调者将会给所有参与者发送 `DoCommit` 请求，**参与者收到 DoCommit 请求后则会进行事务的提交工作**，完成后则会给协调者返回响应，协调者收到所有参与者返回的事务提交成功的响应之后则完成事务。若协调者在 `PreCommit` 阶段 **收到了任何一个 NO 或者在一定时间内没有收到所有参与者的响应** ，那么就会进行中断请求的发送，参与者收到中断请求后则会 **通过上面记录的回滚日志** 来进行事务的回滚操作，并向协调者反馈回滚状况，协调者收到参与者返回的消息后，中断事务。 + +![](https://tva1.sinaimg.cn/large/00831rSTly1gclot2rul3j30j60cpgmo.jpg) + +降低了参与者的阻塞范围，且能在单点故障后继续达成一致。 + +但是最重要的一致性并没有得到根本的解决，比如在 `PreCommit` 阶段，当一个参与者收到了请求之后其他参与者和协调者挂了或者出现了网络分区，这个时候收到消息的参与者都会进行事务提交，这就会出现数据不一致性问题。 + +------ + + + +### 🎯 讲一讲 Paxos 算法？ + +`Paxos` 算法是基于**消息传递且具有高度容错特性的一致性算法**，是目前公认的解决分布式一致性问题最有效的算法之一，**其解决的问题就是在分布式系统中如何就某个值（决议）达成一致** 。 + +在 `Paxos` 中主要有三个角色，分别为 `Proposer提案者`、`Acceptor表决者`、`Learner学习者`。`Paxos` 算法和 `2PC` 一样，也有两个阶段，分别为 `Prepare` 和 `accept` 阶段。 + +在具体的实现中，一个进程可能同时充当多种角色。比如一个进程可能既是 Proposer 又是 Acceptor 又是Learner。Proposer 负责提出提案，Acceptor 负责对提案作出裁决（accept与否），learner 负责学习提案结果。 + +还有一个很重要的概念叫「**提案**」（Proposal）。最终要达成一致的 value 就在提案里。只要 Proposer 发的提案被半数以上的 Acceptor 接受，Proposer 就认为该提案里的 value 被选定了。Acceptor 告诉 Learner 哪个 value 被选定，Learner 就认为那个 value 被选定。 + +**阶段一：prepare 阶段** + +1. `Proposer` 负责提出 `proposal`，每个提案者在提出提案时都会首先获取到一个 **具有全局唯一性的、递增的提案编号N**，即在整个集群中是唯一的编号 N，然后将该编号赋予其要提出的提案，在**第一阶段是只将提案编号发送给所有的表决者**。 + +2. 如果一个 Acceptor 收到一个编号为 N 的 Prepare 请求，如果小于它已经响应过的请求，则拒绝，不回应或回复error。若 N 大于该 Acceptor 已经响应过的所有 Prepare 请求的编号（maxN），那么它就会将它**已经批准过的编号最大的提案**（如果有的话，如果还没有的accept提案的话返回{pok，null，null}）作为响应反馈给 Proposer，同时该 Acceptor 承诺不再接受任何编号小于 N 的提案 + + eg：假定一个 Acceptor 已经响应过的所有 Prepare 请求对应的提案编号分别是1、2、...5和7，那么该 Acceptor 在接收到一个编号为8的 Prepare 请求后，就会将 7 的提案作为响应反馈给 Proposer。 + +**阶段二：accept 阶段** + +1. 如果一个 Proposer 收到半数以上 Acceptor 对其发出的编号为 N 的 Prepare 请求的响应，那么它就会发送一个针对 [N,V] 提案的 Accept 请求半数以上的 Acceptor。注意：V 就是收到的响应中编号最大的提案的 value，如果响应中不包含任何提案，那么 V 就由 Proposer 自己决定 +2. 如果 Acceptor 收到一个针对编号为N的提案的Accept请求，只要该 Acceptor 没有对编号大于 N 的 Prepare 请求做出过响应，它就通过该提案。如果N小于 Acceptor 以及响应的 prepare 请求，则拒绝，不回应或回复error（当proposer没有收到过半的回应，那么他会重新进入第一阶段，递增提案号，重新提出prepare请求） +3. 最后是 Learner 获取通过的提案（有多种方式） + +![](https://tva1.sinaimg.cn/large/00831rSTly1gcloyv70qsj30sg0lc0ve.jpg) + +**`paxos` 算法的死循环问题** + +其实就有点类似于两个人吵架，小明说我是对的，小红说我才是对的，两个人据理力争的谁也不让谁🤬🤬。 + +比如说，此时提案者 P1 提出一个方案 M1，完成了 `Prepare` 阶段的工作，这个时候 `acceptor` 则批准了 M1，但是此时提案者 P2 同时也提出了一个方案 M2，它也完成了 `Prepare` 阶段的工作。然后 P1 的方案已经不能在第二阶段被批准了（因为 `acceptor` 已经批准了比 M1 更大的 M2），所以 P1 自增方案变为 M3 重新进入 `Prepare` 阶段，然后 `acceptor` ，又批准了新的 M3 方案，它又不能批准 M2 了，这个时候 M2 又自增进入 `Prepare` 阶段。。。 + +就这样无休无止的永远提案下去，这就是 `paxos` 算法的死循环问题。 + + + +### 🎯 谈下你对 ZAB 协议的了解？ + +ZAB（Zookeeper Atomic Broadcast） 协议是为分布式协调服务 Zookeeper 专门设计的一种支持**崩溃恢复的原子广播协议**。 + +在 Zookeeper 中，主要依赖 ZAB 协议来实现分布式数据一致性，基于该协议，ZooKeeper 实现了一种主备模式的系统架构来保持集群中各副本之间数据的一致性。 + +尽管 ZAB 不是 Paxos 的实现，但是 ZAB 也参考了一些 Paxos 的一些设计思想，比如： + +- leader 向 follows 提出提案(proposal) +- leader 需要在达到法定数量(半数以上)的 follows 确认之后才会进行 commit +- 每一个 proposal 都有一个纪元(epoch)号，类似于 Paxos 中的选票(ballot) + + `ZAB` 中有三个主要的角色，`Leader 领导者`、`Follower跟随者`、`Observer观察者` 。 + +- `Leader` ：集群中 **唯一的写请求处理者** ，能够发起投票（投票也是为了进行写请求）。 +- `Follower`：能够接收客户端的请求，如果是读请求则可以自己处理，**如果是写请求则要转发给 Leader 。在选举过程中会参与投票，有选举权和被选举权 。** +- **Observer ：就是没有选举权和被选举权的 Follower 。** + +在 ZAB 协议中对 zkServer(即上面我们说的三个角色的总称) 还有两种模式的定义，分别是消息广播和崩溃恢复 + +**消息广播模式** + +![ZAB广播](http://file.sunwaiting.com/zab_broadcast.png) + +1. Leader从客户端收到一个事务请求（如果是集群中其他机器接收到客户端的事务请求，会直接转发给 Leader 服务器） +2. Leader 服务器生成一个对应的事务 Proposal，并为这个事务生成一个全局递增的唯一的ZXID（通过其 ZXID 来进行排序保证顺序性） +3. Leader 将这个事务发送给所有的 Follows 节点 +4. Follower 节点将收到的事务请求加入到历史队列(Leader 会为每个 Follower 分配一个单独的队列先进先出，顺序保证消息的因果关系)中，并发送 ack 给 Leader +5. 当 Leader 收到超过半数 Follower 的 ack 消息，Leader会广播一个 commit 消息 +6. 当 Follower 收到 commit 请求时，会判断该事务的 ZXID 是不是比历史队列中的任何事务的 ZXID 都小，如果是则提交，如果不是则等待比它更小的事务的 commit + +![zab commit流程](http://file.sunwaiting.com/zab_commit_1.png) + +**崩溃恢复模式** + +ZAB 的原子广播协议在正常情况下运行良好，但天有不测风云，一旦 Leader 服务器挂掉或者由于网络原因导致与半数的 Follower 的服务器失去联系，那么就会进入崩溃恢复模式。整个恢复过程结束后需要选举出一个新的 Leader 服务器。 + +恢复模式大致可以分为四个阶段：**选举、发现、同步、广播** + +1. 当 leader 崩溃后，集群进入选举阶段，开始选举出潜在的新 leader(一般为集群中拥有最大 ZXID 的节点) +2. 进入发现阶段，follower 与潜在的新 leader 进行沟通，如果发现超过法定人数的 follower 同意，则潜在的新leader 将 epoc h加1，进入新的纪元。新的 leader 产生 +3. 集群间进行数据同步，保证集群中各个节点的事务一致 +4. 集群恢复到广播模式，开始接受客户端的写请求 + +--- + +## 🌐 三、集群与选举（高可用） + +> **核心思想**：ZK通过选举机制、节点角色分工、数据同步等保证集群的高可用性和数据一致性，理解这些机制对于掌握分布式系统至关重要。 + +- **选举机制**：[选举机制](#🎯-zookeeper选举机制) | [集群选主原理](#🎯-集群选主的原理是什么) +- **节点角色**：[节点角色分工](#🎯-服务器角色) | [节点状态](#🎯-zookeeper-下-server-工作状态) +- **故障处理**：[节点宕机处理](#🎯-zookeeper-宕机如何处理) | [数据同步](#🎯-数据同步机制) +- **状态同步**：[主从同步](#🎯-zookeeper-怎么保证主从节点的状态同步) | [事务顺序](#🎯-zookeeper-是如何保证事务的顺序一致性的) + +### 🎯 Zookeeper 怎么保证主从节点的状态同步？或者说同步流程是什么样的 + +Zookeeper 的核心是原子广播机制，这个机制保证了各个 server 之间的同步。实现这个机制的协议叫做 Zab 协议。Zab 协议有两种模式，它们分别是恢复模式和广播模式。同上 + +------ + + + +### 🎯 集群中为什么要有主节点？ + +在分布式环境中，有些业务逻辑只需要集群中的某一台机器进行执行，其他的机器可以共享这个结果，这样可以大大减少重复计算，提高性能，于是就需要进行 leader 选举。 + +------ + + + +### 🎯 集群中有 3 台服务器，其中一个节点宕机，这个时候 Zookeeper 还可以使用吗？ + +可以继续使用，单数服务器只要没超过一半的服务器宕机就可以继续使用。 + +集群规则为 2N+1 台，N >0，即最少需要 3 台。 + + + +### 🎯 Zookeeper 宕机如何处理？ + +Zookeeper 本身也是集群，推荐配置不少于 3 个服务器。Zookeeper 自身也要保证当一个节点宕机时，其他节点会继续提供服务。如果是一个 Follower 宕机，还有 2 台服务器提供访问，因为 Zookeeper 上的数据是有多个副本的，数据并不会丢失；如果是一个 Leader 宕机，Zookeeper 会选举出新的 Leader。 + +Zookeeper 集群的机制是只要超过半数的节点正常，集群就能正常提供服务。只有在 Zookeeper 节点挂得太多，只剩一半或不到一半节点能工作，集群才失效。所以： + +3 个节点的 cluster 可以挂掉 1 个节点(leader 可以得到 2 票 > 1.5) + +2 个节点的 cluster 就不能挂掉任何1个节点了(leader 可以得到 1 票 <= 1) + +------ + + + +### 🎯 说下四种类型的数据节点 Znode？ + +1. PERSISTENT：持久节点，除非手动删除，否则节点一直存在于 Zookeeper 上。 + +2. EPHEMERAL：临时节点，临时节点的生命周期与客户端会话绑定，一旦客户端会话失效（客户端与 Zookeeper连接断开不一定会话失效），那么这个客户端创建的所有临时节点都会被移除。 + +3. PERSISTENT_SEQUENTIAL：持久顺序节点，基本特性同持久节点，只是增加了顺序属性，节点名后边会追加一个由父节点维护的自增整型数字。 + +4. EPHEMERAL_SEQUENTIAL：临时顺序节点，基本特性同临时节点，增加了顺序属性，节点名后边会追加一个由父节点维护的自增整型数字。 + +------ + + + +### 🎯 Zookeeper选举机制 + +1. 首先对比zxid。zxid大的服务器优先作为Leader +2. 若zxid相同，比如初始化的时候，每个Server的zxid都为0，就会比较myid，myid大的选出来做Leader。 + + **服务器初始化时选举** + +> 目前有3台服务器，每台服务器均没有数据，它们的编号分别是1,2,3按编号依次启动，它们的选择举过程如下： + +1. Server1启动，给自己投票（1,0），然后发投票信息，由于其它机器还没有启动所以它收不到反馈信息，Server1的状态一直属于Looking。 +2. Server2启动，给自己投票（2,0），同时与之前启动的Server1交换结果，由于Server2的编号大所以Server2胜出，**但此时投票数正好大于半数**，所以Server2成为领导者，Server1成为小弟。 +3. Server3启动，给自己投票（3,0），同时与之前启动的Server1,Server2换信息，尽管Server3的编号大，但之前Server2已经胜出，所以Server3只能成为小弟。 +4. 当确定了Leader之后，每个Server更新自己的状态，Leader将状态更新为Leading，Follower将状态更新为Following。 + +**服务器运行期间的选举** + +> zookeeper运行期间，如果有新的Server加入，或者非Leader的Server宕机，那么Leader将会同步数据到新Server或者寻找其他备用Server替代宕机的Server。若Leader宕机，此时集群暂停对外服务，开始在内部选举新的Leader。假设当前集群中有Server1、Server2、Server3三台服务器，Server2为当前集群的Leader，由于意外情况，Server2宕机了，便开始进入选举状态。过程如下 + +1. 变更状态。其他的非Observer服务器将自己的状态改变为Looking，开始进入Leader选举。 +2. 每个Server发出一个投票（myid，zxid），由于此集群已经运行过，所以每个Server上的zxid可能不同。假设Server1的zxid为100，Server3的为99，第一轮投票中，Server1和Server3都投自己，票分别为（1，100）,（3,99）,将自己的票发送给集群中所有机器。 +3. 每个Server接收接收来自其他Server的投票，接下来的步骤与启动时步骤相同。 + + +--- + +## ⚡ 四、核心特性（Watcher机制与权限控制） + +> **核心思想**：ZooKeeper的核心特性包括Watcher事件通知机制、ACL权限控制、会话管理等，这些特性是ZK实现分布式协调的关键技术。 + +- **事件通知**：[Watcher机制](#🎯-zookeeper-watcher-机制--数据变更通知) | [客户端注册](#🎯-客户端注册-watcher-实现) | [服务端处理](#🎯-服务端处理-watcher-实现) | [客户端回调](#🎯-客户端回调-watcher) +- **权限控制**：[ACL机制](#🎯-acl-权限控制机制) | [Chroot特性](#🎯-chroot-特性) +- **会话管理**：[会话机制](#🎯-会话管理) | [分桶策略](#分桶策略) + +### 🎯 Zookeeper Watcher 机制 – 数据变更通知 + +Zookeeper 允许客户端向服务端的某个 Znode 注册一个 Watcher 监听，当服务端的一些指定事件触发了这个 Watcher，服务端会向指定客户端发送一个事件通知来实现分布式的通知功能，然后客户端根据 Watcher 通知状态和事件类型做出业务上的改变。 + +工作机制： + +（1）客户端注册 watcher + +（2）服务端处理 watcher + +（3）客户端回调 watcher + +Watcher 特性总结： + +（1）一次性 + +无论是服务端还是客户端，一旦一个 Watcher 被 触 发 ，Zookeeper 都会将其从相应的存储中移除。这样的设计有效的减轻了服务端的压力，不然对于更新非常频繁的节点，服务端会不断的向客户端发送事件通知，无论对于网络还是服务端的压力都非常大。 + +（2）客户端串行执行 + +客户端 Watcher 回调的过程是一个串行同步的过程。 + +（3）轻量 + +3.1、Watcher 通知非常简单，只会告诉客户端发生了事件，而不会说明事件的具体内容。 + +3.2、客户端向服务端注册 Watcher 的时候，并不会把客户端真实的 Watcher 对象实体传递到服务端，仅仅是在客户端请求中使用 boolean 类型属性进行了标记。 + +（4）watcher event 异步发送 watcher 的通知事件从 server 发送到 client 是异步的，这就存在一个问题，不同的客户端和服务器之间通过 socket 进行通信，由于网络延迟或其他因素导致客户端在不通的时刻监听到事件，由于 Zookeeper 本身提供了 ordering guarantee，即客户端监听事件后，才会感知它所监视 znode发生了变化。所以我们使用 Zookeeper 不能期望能够监控到节点每次的变化。Zookeeper 只能保证最终的一致性，而无法保证强一致性。 + +（5）注册 watcher getData、exists、getChildren + +（6）触发 watcher create、delete、setData + +（7）当一个客户端连接到一个新的服务器上时，watch 将会被以任意会话事件触发。当与一个服务器失去连接的时候，是无法接收到 watch 的。而当 client 重新连接时，如果需要的话，所有先前注册过的 watch，都会被重新注册。通常这是完全透明的。只有在一个特殊情况下，watch 可能会丢失：对于一个未创建的 znode的 exist watch，如果在客户端断开连接期间被创建了，并且随后在客户端连接上之前又删除了，这种情况下，这个 watch 事件可能会被丢失。 + +### 🎯 客户端注册 Watcher 实现 + +（1）调用 getData()/getChildren()/exist()三个 API，传入 Watcher 对象 + +（2）标记请求 request，封装 Watcher 到 WatchRegistration + +（3）封装成 Packet 对象，发服务端发送 request + +（4）收到服务端响应后，将 Watcher 注册到 ZKWatcherManager 中进行管理 + +（5）请求返回，完成注册。 + +### 🎯 服务端处理 Watcher 实现 + +（1）服务端接收 Watcher 并存储 + +接收到客户端请求，处理请求判断是否需要注册 Watcher，需要的话将数据节点的节点路径和 ServerCnxn（ServerCnxn 代表一个客户端和服务端的连接，实现了 Watcher 的 process 接口，此时可以看成一个 Watcher 对象）存储在WatcherManager 的 WatchTable 和 watch2Paths 中去。 + +（2）Watcher 触发 + +以服务端接收到 setData() 事务请求触发 NodeDataChanged 事件为例： + +2.1 封装 WatchedEvent + +将通知状态（SyncConnected）、事件类型（NodeDataChanged）以及节点路径封装成一个 WatchedEvent 对象 + +2.2 查询 Watcher + +从 WatchTable 中根据节点路径查找 Watcher + +2.3 没找到；说明没有客户端在该数据节点上注册过 Watcher + +2.4 找到；提取并从 WatchTable 和 Watch2Paths 中删除对应 Watcher（从这里可以看出 Watcher 在服务端是一次性的，触发一次就失效了） + +（3）调用 process 方法来触发 Watcher + +这里 process 主要就是通过 ServerCnxn 对应的 TCP 连接发送 Watcher 事件通知。 + +### 🎯 客户端回调 Watcher + +客户端 SendThread 线程接收事件通知，交由 EventThread 线程回调 Watcher。 + +客户端的 Watcher 机制同样是一次性的，一旦被触发后，该 Watcher 就失效了。 + +### 🎯 ACL 权限控制机制 + +UGO（User/Group/Others） + +目前在 Linux/Unix 文件系统中使用，也是使用最广泛的权限控制方式。是一种粗粒度的文件系统权限控制模式。 + +ACL（Access Control List）访问控制列表 + +包括三个方面： + +权限模式（Scheme） + +（1）IP：从 IP 地址粒度进行权限控制 + +（2）Digest：最常用，用类似于 username:password 的权限标识来进行权限配置，便于区分不同应用来进行权限控制 + +（3）World：最开放的权限控制方式，是一种特殊的 digest 模式，只有一个权限标识“world:anyone” + +（4）Super：超级用户 + +授权对象 + +授权对象指的是权限赋予的用户或一个指定实体，例如 IP 地址或是机器灯。 + +权限 Permission + +（1）CREATE：数据节点创建权限，允许授权对象在该 Znode 下创建子节点 + +（2）DELETE：子节点删除权限，允许授权对象删除该数据节点的子节点 + +（3）READ：数据节点的读取权限，允许授权对象访问该数据节点并读取其数据内容或子节点列表等 + +（4）WRITE：数据节点更新权限，允许授权对象对该数据节点进行更新操作 + +（5）ADMIN：数据节点管理权限，允许授权对象对该数据节点进行 ACL 相关设置操作 + +### 🎯 Chroot 特性 + +3.2.0 版本后，添加了 Chroot 特性，该特性允许每个客户端为自己设置一个命名空间。如果一个客户端设置了 Chroot，那么该客户端对服务器的任何操作，都将会被限制在其自己的命名空间下。 + +通过设置 Chroot，能够将一个客户端应用于 Zookeeper 服务端的一颗子树相对应，在那些多个应用公用一个 Zookeeper 进群的场景下，对实现不同应用间的相互隔离非常有帮助。 + +### 🎯 会话管理 + +分桶策略：将类似的会话放在同一区块中进行管理，以便于 Zookeeper 对会话进行不同区块的隔离处理以及同一区块的统一处理。 + +分配原则：每个会话的“下次超时时间点”（ExpirationTime） + +计算公式： + +ExpirationTime_ = currentTime + sessionTimeout + +ExpirationTime = (ExpirationTime_ / ExpirationInrerval + 1) * + +ExpirationInterval , ExpirationInterval 是指 Zookeeper 会话超时检查时间间隔，默认 tickTime + +### 🎯 服务器角色 + +Leader + +（1）事务请求的唯一调度和处理者，保证集群事务处理的顺序性 + +（2）集群内部各服务的调度者 + +Follower + +（1）处理客户端的非事务请求，转发事务请求给 Leader 服务器 + +（2）参与事务请求 Proposal 的投票 + +（3）参与 Leader 选举投票 + +Observer + +（1）3.0 版本以后引入的一个服务器角色，在不影响集群事务处理能力的基础上提升集群的非事务处理能力 + +（2）处理客户端的非事务请求，转发事务请求给 Leader 服务器 + +（3）不参与任何形式的投票 + +### 🎯 Zookeeper 下 Server 工作状态 + +服务器具有四种状态，分别是 LOOKING、FOLLOWING、LEADING、OBSERVING。 + +（1）LOOKING：寻 找 Leader 状态。当服务器处于该状态时，它会认为当前集群中没有 Leader，因此需要进入 Leader 选举状态。 + +（2）FOLLOWING：跟随者状态。表明当前服务器角色是 Follower。 + +（3）LEADING：领导者状态。表明当前服务器角色是 Leader。 + +（4）OBSERVING：观察者状态。表明当前服务器角色是 Observer。 + +--- + +## 🔧 五、典型应用场景（分布式协调实践） + +> **核心思想**：ZooKeeper的典型应用场景包括分布式锁、配置管理、服务发现、负载均衡等，这些场景体现了ZK在分布式系统中的实际价值。 + +- **配置管理**：[数据发布/订阅](#数据发布订阅) | [负载均衡](#负载均衡) | [命名服务](#zk-的命名服务文件系统) +- **分布式协调**：[分布式通知](#分布式通知和协调) | [集群管理](#zookeeper-集群管理) | [Master选举](#master-选举) +- **同步原语**：[分布式锁](#zookeeper-分布式锁) | [队列管理](#zookeeper-队列管理) + +### 🎯 Zookeeper 实现分布式锁的原理 + +> Zookeeper 分布式锁基于 **临时顺序节点和 Watch 机制**实现： +> 客户端在 `/lock` 下创建临时顺序节点，序号最小的客户端获得锁，其他客户端监听前一个节点的删除事件，等到前一个节点释放锁后再依次获得。 +> +> 这种方式可以保证锁的 **公平性**，同时利用临时节点避免了死锁。 +> +> 不过 Zookeeper 的锁性能一般在万级 QPS，适合金融、广告投放这种需要强一致性的场景，而高并发写场景更推荐 Redis 分布式锁。 + +Zookeeper 本身是一个分布式协调服务，它提供了 **强一致性、顺序性和事件通知**，用来做分布式锁非常合适。核心思想是利用 **临时顺序节点 + Watch 机制**。 + +**1. 基本实现步骤（公平锁）** + +假设我们约定锁的路径为 `/lock`： + +1. **所有客户端到 `/lock` 下创建临时顺序节点** + - 比如 `clientA` 创建 `/lock/lock_0001`，`clientB` 创建 `/lock/lock_0002`。 + - 节点是 **临时的**，客户端宕机或会话断开会自动删除，避免死锁。 +2. **获取子节点列表，判断自己是否是最小节点** + - 谁的节点序号最小，谁就获得锁。 + - 比如 `/lock/lock_0001` 最小，则 `clientA` 获得锁。 +3. **没拿到锁的客户端监听前一个节点** + - 比如 `clientB` 创建了 `lock_0002`，就监听 `lock_0001`。 + - 而不是监听所有节点，避免“羊群效应”。 +4. **前一个节点被删除时，触发 Watch 事件** + - 如果 `clientA` 释放锁（删除 `lock_0001`），Zookeeper 会通知 `clientB`。 + - `clientB` 再检查是否轮到自己（是否是最小节点），如果是就获得锁。 +5. **释放锁** + - 拿到锁的客户端执行完任务后，主动删除自己创建的节点，触发通知，锁转交下一个客户端。 + +**2. 关键点** + +- **临时节点**：保证客户端异常宕机时，锁能自动释放。 +- **顺序节点**：保证锁的公平性（FIFO）。 +- **Watch 机制**：实现事件驱动，避免轮询浪费资源。 + +**3. 示例流程图** + +``` +/lock + ├── lock_0001 (clientA) + ├── lock_0002 (clientB) + ├── lock_0003 (clientC) +``` + +- clientA → 最小节点 → 获得锁 +- clientB → 监听 lock_0001 +- clientC → 监听 lock_0002 +- 当 clientA 删除 lock_0001 → 通知 clientB +- clientB 获得锁，以此类推 + +**优缺点** + +✅ 优点 + +- 公平锁，顺序执行，避免“饥饿”。 +- 临时节点避免死锁。 +- Watch 机制高效。 + +❌ 缺点 + +- Zookeeper 本身是 CP 系统，性能有限，**QPS 一般在万级**，不适合高频抢锁场景（比 Redis 慢）。 +- 如果客户端大量竞争，Zookeeper 节点会有较大压力。 + + + +### 🎯 数据同步 + +整个集群完成 Leader 选举之后，Learner（Follower 和 Observer 的统称）回向Leader 服务器进行注册。当 Learner 服务器想 Leader 服务器完成注册后，进入数据同步环节。 + +数据同步流程：（均以消息传递的方式进行） + +Learner 向 Leader 注册 + +数据同步 + +同步确认 + +Zookeeper 的数据同步通常分为四类： + +（1）直接差异化同步（DIFF 同步） + +（2）先回滚再差异化同步（TRUNC+DIFF 同步） + +（3）仅回滚同步（TRUNC 同步） + +（4）全量同步（SNAP 同步） + +在进行数据同步前，Leader 服务器会完成数据同步初始化： + +peerLastZxid： + +· 从 learner 服务器注册时发送的 ACKEPOCH 消息中提取 lastZxid（该Learner 服务器最后处理的 ZXID） + +minCommittedLog： + +· Leader 服务器 Proposal 缓存队列 committedLog 中最小 ZXIDmaxCommittedLog： + +· Leader 服务器 Proposal 缓存队列 committedLog 中最大 ZXID直接差异化同步（DIFF 同步） + +· 场景：peerLastZxid 介于 minCommittedLog 和 maxCommittedLog之间先回滚再差异化同步（TRUNC+DIFF 同步） + +· 场景：当新的 Leader 服务器发现某个 Learner 服务器包含了一条自己没有的事务记录，那么就需要让该 Learner 服务器进行事务回滚–回滚到 Leader服务器上存在的，同时也是最接近于 peerLastZxid 的 ZXID仅回滚同步（TRUNC 同步） + +· 场景：peerLastZxid 大于 maxCommittedLog + +全量同步（SNAP 同步） + +· 场景一：peerLastZxid 小于 minCommittedLog + +· 场景二：Leader 服务器上没有 Proposal 缓存队列且 peerLastZxid 不等于 lastProcessZxid + +### 🎯 zookeeper 是如何保证事务的顺序一致性的？ + +zookeeper 采用了全局递增的事务 Id 来标识，所有的 proposal（提议）都在被提出的时候加上了 zxid，zxid 实际上是一个 64 位的数字，高 32 位是 epoch（ 时期; 纪元; 世; 新时代）用来标识 leader 周期，如果有新的 leader 产生出来，epoch会自增，低 32 位用来递增计数。当新产生 proposal 的时候，会依据数据库的两阶段过程，首先会向其他的 server 发出事务执行请求，如果超过半数的机器都能执行并且能够成功，那么就会开始执行。 + +### 🎯 分布式集群中为什么会有 Master主节点？ + +在分布式环境中，有些业务逻辑只需要集群中的某一台机器进行执行，其他的机器可以共享这个结果，这样可以大大减少重复计算，提高性能，于是就需要进行 leader 选举。 + +### 🎯 zk 节点宕机如何处理？ + +Zookeeper 本身也是集群，推荐配置不少于 3 个服务器。Zookeeper 自身也要保证当一个节点宕机时，其他节点会继续提供服务。 + +如果是一个 Follower 宕机，还有 2 台服务器提供访问，因为 Zookeeper 上的数据是有多个副本的，数据并不会丢失； + +如果是一个 Leader 宕机，Zookeeper 会选举出新的 Leader。 + +ZK 集群的机制是只要超过半数的节点正常，集群就能正常提供服务。只有在 ZK节点挂得太多，只剩一半或不到一半节点能工作，集群才失效。 + +所以 + +3 个节点的 cluster 可以挂掉 1 个节点(leader 可以得到 2 票>1.5) + +2 个节点的 cluster 就不能挂掉任何 1 个节点了(leader 可以得到 1 票<=1) + +### 🎯 zookeeper 负载均衡和 nginx 负载均衡区别 + +zk 的负载均衡是可以调控，nginx 只是能调权重，其他需要可控的都需要自己写插件；但是 nginx 的吞吐量比 zk 大很多，应该说按业务选择用哪种方式。 + +### 🎯 Zookeeper 有哪几种几种部署模式？ + +Zookeeper 有三种部署模式： + +1. 单机部署：一台集群上运行； +2. 集群部署：多台集群运行； +3. 伪集群部署：一台集群启动多个 Zookeeper 实例运行。 + +### 🎯 集群最少要几台机器，集群规则是怎样的？集群中有 3 台服务器，其中一个节点宕机，这个时候 Zookeeper 还可以使用吗？ + +集群规则为 2N+1 台，N>0，即 3 台。可以继续使用，单数服务器只要没超过一半的服务器宕机就可以继续使用。 + +### 🎯 集群支持动态添加机器吗？ + +其实就是水平扩容了，Zookeeper 在这方面不太好。两种方式： + +全部重启：关闭所有 Zookeeper 服务，修改配置之后启动。不影响之前客户端的会话。 + +逐个重启：在过半存活即可用的原则下，一台机器重启不影响整个集群对外提供服务。这是比较常用的方式。 + +3.5 版本开始支持动态扩容。 + +### 🎯 Zookeeper 对节点的 watch 监听通知是永久的吗？为什么不是永久的? + +不是。官方声明：一个 Watch 事件是一个一次性的触发器，当被设置了 Watch的数据发生了改变的时候，则服务器将这个改变发送给设置了 Watch 的客户端，以便通知它们。 + +为什么不是永久的，举个例子，如果服务端变动频繁，而监听的客户端很多情况下，每次变动都要通知到所有的客户端，给网络和服务器造成很大压力。 + +一般是客户端执行 getData(“/节点 A”,true)，如果节点 A 发生了变更或删除，客户端会得到它的 watch 事件，但是在之后节点 A 又发生了变更，而客户端又没有设置 watch 事件，就不再给客户端发送。 + +在实际应用中，很多情况下，我们的客户端不需要知道服务端的每一次变动，我只要最新的数据即可。 + +### 🎯 Zookeeper 的 java 客户端都有哪些？ + +java 客户端：zk 自带的 zkclient 及 Apache 开源的 Curator。 + +### 🎯 chubby 是什么，和 zookeeper 比你怎么看？ + +chubby 是 google 的，完全实现 paxos 算法，不开源。zookeeper 是 chubby的开源实现，使用 zab 协议，paxos 算法的变种。 + +### 🎯 说几个 zookeeper 常用的命令。 + +常用命令：ls get set create delete 等。 + +### 🎯 ZAB 和 Paxos 算法的联系与区别？ + +相同点： + +（1）两者都存在一个类似于 Leader 进程的角色，由其负责协调多个 Follower 进程的运行 + +（2）Leader 进程都会等待超过半数的 Follower 做出正确的反馈后，才会将一个提案进行提交 + +（3）ZAB 协议中，每个 Proposal 中都包含一个 epoch 值来代表当前的 Leader周期，Paxos 中名字为 Ballot + +不同点： + +ZAB 用来构建高可用的分布式数据主备系统（Zookeeper），Paxos 是用来构建分布式一致性状态机系统。 + + + +### 🎯 Zookeeper 都有哪些功能？ + +1. 集群管理：监控节点存活状态、运行请求等； +2. 主节点选举：主节点挂掉了之后可以从备用的节点开始新一轮选主，主节点选举说的就是这个选举的过程，使用 Zookeeper 可以协助完成这个过程； +3. 分布式锁：Zookeeper 提供两种锁：独占锁、共享锁。独占锁即一次只能有一个线程使用资源，共享锁是读锁共享，读写互斥，即可以有多线线程同时读同一个资源，如果要使用写锁也只能有一个线程使用。Zookeeper 可以对分布式锁进行控制。 +4. 命名服务：在分布式系统中，通过使用命名服务，客户端应用能够根据指定名字来获取资源或服务的地址，提供者等信息。 + +### 🎯 说一下 Zookeeper 的通知机制？ + +client 端会对某个 znode 建立一个 watcher 事件，当该 znode 发生变化时，这些 client 会收到 zk 的通知，然后 client 可以根据 znode 变化来做出业务上的改变等。 + +### 🎯 Zookeeper 和 Dubbo 的关系？ + +Zookeeper的作用： + +zookeeper用来注册服务和进行负载均衡，哪一个服务由哪一个机器来提供必需让调用者知道，简单来说就是ip地址和服务名称的对应关系。当然也可以通过硬编码的方式把这种对应关系在调用方业务代码中实现，但是如果提供服务的机器挂掉调用者无法知晓，如果不更改代码会继续请求挂掉的机器提供服务。zookeeper通过心跳机制可以检测挂掉的机器并将挂掉机器的ip和服务对应关系从列表中删除。至于支持高并发，简单来说就是横向扩展，在不更改代码的情况通过添加机器来提高运算能力。通过添加新的机器向zookeeper注册服务，服务的提供者多了能服务的客户就多了。 + +dubbo： + +是管理中间层的工具，在业务层到数据仓库间有非常多服务的接入和服务提供者需要调度，dubbo提供一个框架解决这个问题。 +注意这里的dubbo只是一个框架，至于你架子上放什么是完全取决于你的，就像一个汽车骨架，你需要配你的轮子引擎。这个框架中要完成调度必须要有一个分布式的注册中心，储存所有服务的元数据，你可以用zk，也可以用别的，只是大家都用zk。 + +zookeeper和dubbo的关系： + +Dubbo 的将注册中心进行抽象，它可以外接不同的存储媒介给注册中心提供服务，有 ZooKeeper，Memcached，Redis 等。 + +引入了 ZooKeeper 作为存储媒介，也就把 ZooKeeper 的特性引进来。首先是负载均衡，单注册中心的承载能力是有限的，在流量达到一定程度的时 候就需要分流，负载均衡就是为了分流而存在的，一个 ZooKeeper 群配合相应的 Web 应用就可以很容易达到负载均衡；资源同步，单单有负载均衡还不 够，节点之间的数据和资源需要同步，ZooKeeper 集群就天然具备有这样的功能；命名服务，将树状结构用于维护全局的服务地址列表，服务提供者在启动 的时候，向 ZooKeeper 上的指定节点 /dubbo/${serviceName}/providers 目录下写入自己的 URL 地址，这个操作就完成了服务的发布。 其他特性还有 Mast 选举，分布式锁等。 + +![img](https://imgconvert.csdnimg.cn/aHR0cHM6Ly91c2VyLWdvbGQtY2RuLnhpdHUuaW8vMjAxOS8xMC8zMS8xNmUyMTliYzc3MDA5OGRm?x-oss-process=image/format,png) + + + +--- + +## 🚨 六、运维实践与故障处理（生产经验） + +> **核心思想**：ZooKeeper在生产环境中的运维实践，包括部署配置、性能监控、故障排查等，这些经验是高级工程师必备的技能。 + +- **部署配置**：[部署模式](#🎯-zookeeper-有哪几种几种部署模式) | [集群规划](#🎯-集群最少要几台机器集群规则是怎样的集群中有-3-台服务器其中一个节点宕机这个时候-zookeeper-还可以使用吗) | [动态扩容](#🎯-集群支持动态添加机器吗) +- **监控运维**：[客户端工具](#🎯-zookeeper-的-java-客户端都有哪些) | [常用命令](#🎯-说几个-zookeeper-常用的命令) | [通知机制](#🎯-说一下-zookeeper-的通知机制) | [Watch监听](#🎯-zookeeper-对节点的-watch-监听通知是永久的吗为什么不是永久的) + +--- + +## 💼 七、对比分析与技术选型（架构决策） + +> **核心思想**：理解ZooKeeper与其他分布式协调系统的差异，掌握在不同场景下的技术选型原则。 + +### 📋 本章知识点 + +- **技术对比**：[ZK vs 其他系统](#🎯-chubby-是什么和-zookeeper-比你怎么看) | [ZAB vs Paxos](#🎯-zab-和-paxos-算法的联系与区别) | [CAP选择](#🎯-zk-是-cp-还是-ap) +- **应用集成**：[与Dubbo关系](#🎯-zookeeper-和-dubbo-的关系) | [负载均衡对比](#🎯-zookeeper-负载均衡和-nginx-负载均衡区别) + +### 🎯 zk 是 CP 还是 AP + +zk的ap和cp是从不同的角度分析的。 + +从一个读写请求分析，保证了可用性（不用阻塞等待全部follwer同步完成），保证不了数据的一致性，所以是ap。 + +但是从zk架构分析，zk在leader选举期间，会暂停对外提供服务（为啥会暂停，因为zk依赖leader来保证数据一致性)，所以丢失了可用性，保证了一致性，即cp。 + +再细点话，这个c不是强一致性，而是最终一致性。即上面的写案例，数据最终会同步到一致，只是时间问题。 + +综上，zk广义上来说是cp，狭义上是ap。 + +--- + +## 🎯 ZooKeeper面试备战指南 + +### 💡 高频考点Top10 + +1. **🔥 ZAB协议原理** - ZK一致性算法的核心，必考概念 +2. **⚡ Leader选举机制** - 分布式一致性的关键技术 +3. **📊 数据一致性保证** - 顺序一致性、原子性等特性 +4. **🚨 脑裂问题** - 分布式系统经典问题，解决思路要清晰 +5. **🔍 Watcher机制** - 事件通知的实现原理 +6. **💾 分片与副本** - 数据分布和高可用保证 +7. **⚙️ 节点类型与特性** - 持久、临时、顺序节点的应用 +8. **🔧 分布式锁实现** - ZK的典型应用场景 +9. **📈 集群架构设计** - 生产环境的部署和运维 +10. **💼 实际项目经验** - 能结合具体场景谈技术应用 + +### 🎭 面试答题技巧 + +**📝 标准回答结构** +1. **概念定义**（30秒） - 用一句话说清楚是什么 +2. **工作原理**（1分钟） - 阐述核心机制和流程 +3. **应用场景**（30秒） - 什么时候用，解决什么问题 +4. **具体示例**（1分钟） - 最好是自己项目的真实案例 +5. **注意事项**（30秒） - 体现深度思考和实战经验 + +**🗣️ 表达话术模板** +- "从我的项目经验来看..." +- "在生产环境中，我们通常会..." +- "这里有个需要注意的点是..." +- "相比于其他分布式协调系统，ZK的优势在于..." +- "在大规模集群场景下，推荐的做法是..." + +### 🚀 进阶加分点 + +- **底层原理**：能从ZAB协议层面解释ZK的一致性保证 +- **性能调优**：有具体的集群优化经验和数据对比 +- **架构设计**：能设计适合业务场景的ZK集群方案 +- **故障处理**：有排查和解决ZK集群问题的经验 +- **技术选型**：能准确分析ZK与其他技术的适用场景 + +### 📚 延伸学习建议 + +- **官方文档**：ZooKeeper官方文档是最权威的学习资料 +- **源码研读**：深入理解ZAB协议、Watcher机制的实现 +- **实战练习**：搭建ZK集群，动手验证各种特性 +- **案例分析**：研究大厂的ZK应用案例和最佳实践 +- **社区交流**：关注ZK相关的技术博客和开源项目 + +--- + +## 🎉 总结 + +**ZooKeeper作为分布式协调服务的经典实现**，是构建大规模分布式系统的基石。从配置管理到服务发现，从分布式锁到Leader选举，ZK在分布式系统中发挥着不可替代的作用。 + +**掌握ZK的核心在于理解其一致性保证机制**：通过ZAB协议实现分布式一致性，通过Leader选举保证集群稳定，通过Watcher机制实现事件通知，通过节点类型支持各种分布式应用场景。 + +**记住：面试官考察的不是你背了多少概念，而是你能否在实际项目中灵活运用ZK解决分布式协调问题。** + +**最后一句话**：*"分布式系统的复杂性在于一致性，而ZooKeeper正是解决这一复杂性的优雅方案！"* + +--- + +> 💌 **坚持学习，持续成长！** diff --git a/docs/java/.DS_Store b/docs/java/.DS_Store new file mode 100644 index 0000000000..d82ae5d606 Binary files /dev/null and b/docs/java/.DS_Store differ diff --git a/docs/java/Assert.md b/docs/java/Assert.md new file mode 100644 index 0000000000..d258a4f789 --- /dev/null +++ b/docs/java/Assert.md @@ -0,0 +1,24 @@ +# Assert + +## 概述 + +业务代码中我们是不会使用断言的，但是看各种源码或单元测试的时候，肯定会遇到 Assert，你有了解过吗？这玩意到底是干嘛的？ + +> 编写代码时，我们总是会做出一些假设，断言就是用于在代码中捕捉这些假设，可以将断言看作是**异常处理**的一种高级形式。断言表示为一些布尔表达式，程序员相信在程序中的某个特定点该表达式值为真。可以在任何时候启用和禁用断言验证，因此可以在测试时启用断言，而在部署时禁用断言。同样，程序投入运行后，最终用户在遇到问题时可以重新启用断言。 + +Assert 其实就是用来调试程序的 + +java 断言 assert 是 jdk1.4 引入的。 + +　　jvm断言默认是关闭的。 + +断言可以局部开启的，如：父类禁止断言，而子类开启断言，所以一般说“**断言不具有继承性**”。 + +**断言只适用复杂的调式过程。** + +**断言一般用于程序执行结构的判断，千万不要让断言处理业务流程。** + + + +## 语法 + diff --git a/docs/java/Collections/Collections-FAQ.md b/docs/java/Collections/Collections-FAQ.md deleted file mode 100644 index dae5b35573..0000000000 --- a/docs/java/Collections/Collections-FAQ.md +++ /dev/null @@ -1,1760 +0,0 @@ -「我是大厂面试官」—— Java 集合，你肯定会被问到这些 - -> 文章收录在 GitHub [JavaKeeper](https://github.com/Jstarfish/JavaKeeper) ，N线互联网开发必备技能兵器谱 - -作为一位小菜 ”一面面试官“，面试过程中，我肯定会问 Java 集合的内容，同时作为求职者，也肯定会被问到集合，所以整理下 Java 集合面试题 - -![](https://i02piccdn.sogoucdn.com/fe487f455e5b1eb6) - -> 说说常见的集合有哪些吧？ -> -> HashMap说一下，其中的Key需要重写hashCode()和equals()吗？ -> -> HashMap中key和value可以为null吗？允许几个为null呀？ -> -> HashMap线程安全吗?ConcurrentHashMap和hashTable有什么区别？ -> -> List和Set说一下，现在有一个ArrayList，对其中的所有元素按照某一属性大小排序，应该怎么做？ -> -> ArrayList 和 Vector 的区别 -> -> list 可以删除吗，遍历的时候可以删除吗，为什么 - -面向对象语言对事物的体现都是以对象的形式，所以为了方便对多个对象的操作，需要将对象进行存储，集合就是存储对象最常用的一种方式，也叫容器。 - -![img](https://tva1.sinaimg.cn/large/00831rSTly1gdp03vldkkg30hv0gzwet.gif) - - - -从上面的集合框架图可以看到，Java 集合框架主要包括两种类型的容器 - -- 一种是集合（Collection），存储一个元素集合 -- 另一种是图（Map），存储键/值对映射。 - -Collection 接口又有 3 种子类型，List、Set 和 Queue，再下面是一些抽象类，最后是具体实现类，常用的有 ArrayList、LinkedList、HashSet、LinkedHashSet、HashMap、LinkedHashMap 等等。 - -集合框架是一个用来代表和操纵集合的统一架构。所有的集合框架都包含如下内容： - -- **接口**：是代表集合的抽象数据类型。例如 Collection、List、Set、Map 等。之所以定义多个接口，是为了以不同的方式操作集合对象 - -- **实现（类）**：是集合接口的具体实现。从本质上讲，它们是可重复使用的数据结构，例如：ArrayList、LinkedList、HashSet、HashMap。 - -- **算法**：是实现集合接口的对象里的方法执行的一些有用的计算，例如：搜索和排序。这些算法被称为多态，那是因为相同的方法可以在相似的接口上有着不同的实现。 - ------- - - - -## 说说常用的集合有哪些吧？ - -Map 接口和 Collection 接口是所有集合框架的父接口： - -1. Collection接口的子接口包括：Set、List、Queue -2. List是有序的允许有重复元素的Collection，实现类主要有：ArrayList、LinkedList、Stack以及Vector等 -3. Set是一种不包含重复元素且无序的Collection，实现类主要有：HashSet、TreeSet、LinkedHashSet等 -4. Map没有继承Collection接口，Map提供key到value的映射。实现类主要有：HashMap、TreeMap、Hashtable、ConcurrentHashMap 以及 Properties 等 - - - -## ArrayList 和 Vector 的区别 - -相同点： - -- ArrayList 和 Vector 都是继承了相同的父类和实现了相同的接口（都实现了List，有序、允许重复和null） - - ```java - extends AbstractList - implements List, RandomAccess, Cloneable, java.io.Serializable - ``` - -- 底层都是数组（Object[]）实现的 - -- 初始默认长度都为**10** - -不同点： - -- 同步性：Vector 中的 public 方法多数添加了 synchronized 关键字、以确保方法同步、也即是 Vector 线程安全、ArrayList 线程不安全 - -- 性能：Vector 存在 synchronized 的锁等待情况、需要等待释放锁这个过程、所以性能相对较差 - -- 扩容大小：ArrayList在底层数组不够用时在原来的基础上扩展 0.5 倍，Vector默认是扩展 1 倍 - - 扩容机制，扩容方法其实就是新创建一个数组，然后将旧数组的元素都复制到新数组里面。其底层的扩容方法都在 **grow()** 中（基于JDK8） - - - ArrayList 的 grow()，在满足扩容条件时、ArrayList以**1.5** 倍的方式在扩容（oldCapacity >> **1** ，右移运算，相当于除以 2，结果为二分之一的 oldCapacity） - - ```java - private void grow(int minCapacity) { - // overflow-conscious code - int oldCapacity = elementData.length; - //newCapacity = oldCapacity + O.5*oldCapacity,此处扩容0.5倍 - int newCapacity = oldCapacity + (oldCapacity >> 1); - if (newCapacity - minCapacity < 0) - newCapacity = minCapacity; - if (newCapacity - MAX_ARRAY_SIZE > 0) - newCapacity = hugeCapacity(minCapacity); - // minCapacity is usually close to size, so this is a win: - elementData = Arrays.copyOf(elementData, newCapacity); - } - ``` - - - Vector 的 grow()，Vector 比 ArrayList多一个属性，扩展因子capacityIncrement，可以扩容大小。当扩容容量增量大于**0**时、新数组长度为原数组长度**+**扩容容量增量、否则新数组长度为原数组长度的**2**倍 - - ```java - private void grow(int minCapacity) { - // overflow-conscious code - int oldCapacity = elementData.length; - // - int newCapacity = oldCapacity + ((capacityIncrement > 0) ? - capacityIncrement : oldCapacity); - if (newCapacity - minCapacity < 0) - newCapacity = minCapacity; - if (newCapacity - MAX_ARRAY_SIZE > 0) - newCapacity = hugeCapacity(minCapacity); - elementData = Arrays.copyOf(elementData, newCapacity); - } - ``` - - - -## ArrayList 与 LinkedList 区别 - -- 是否保证线程安全： ArrayList 和 LinkedList 都是不同步的，也就是不保证线程安全； -- **底层数据结构**： Arraylist 底层使用的是 Object 数组；LinkedList 底层使用的是**双向循环链表**数据结构； -- **插入和删除是否受元素位置的影响：** - - **ArrayList 采用数组存储，所以插入和删除元素的时间复杂度受元素位置的影响。** 比如：执行 `add(E e)`方法的时候， ArrayList 会默认在将指定的元素追加到此列表的末尾，这种情况时间复杂度就是O(1)。但是如果要在指定位置 i 插入和删除元素的话（ `add(intindex,E element)`）时间复杂度就为 O(n-i)。因为在进行上述操作的时候集合中第 i 和第 i 个元素之后的(n-i)个元素都要执行向后位/向前移一位的操作。 - - **LinkedList 采用链表存储，所以插入，删除元素时间复杂度不受元素位置的影响，都是近似 $O(1)$，而数组为近似 $O(n)$。** - - ArrayList 一般应用于查询较多但插入以及删除较少情况，如果插入以及删除较多则建议使用 LinkedList -- **是否支持快速随机访问**： LinkedList 不支持高效的随机元素访问，而 ArrayList 实现了 RandomAccess 接口，所以有随机访问功能。快速随机访问就是通过元素的序号快速获取元素对象(对应于 `get(intindex)`方法)。 -- **内存空间占用**： ArrayList 的空间浪费主要体现在在 list 列表的结尾会预留一定的容量空间，而 LinkedList 的空间花费则体现在它的每一个元素都需要消耗比 ArrayList 更多的空间（因为要存放直接后继和直接前驱以及数据）。 - - - -高级工程师的我，可不得看看源码，具体分析下： - -- ArrayList工作原理其实很简单，底层是动态数组，每次创建一个 ArrayList 实例时会分配一个初始容量（没有指定初始容量的话，默认是 10），以add方法为例，如果没有指定初始容量，当执行add方法，先判断当前数组是否为空，如果为空则给保存对象的数组分配一个最小容量，默认为10。当添加大容量元素时，会先增加数组的大小，以提高添加的效率； - -- LinkedList 是有序并且支持元素重复的集合，底层是基于双向链表的，即每个节点既包含指向其后继的引用也包括指向其前驱的引用。链表无容量限制，但双向链表本身使用了更多空间，也需要额外的链表指针操作。按下标访问元素 `get(i)/set(i,e)` 要悲剧的遍历链表将指针移动到位(如果i>数组大小的一半，会从末尾移起)。插入、删除元素时修改前后节点的指针即可，但还是要遍历部分链表的指针才能移动到下标所指的位置，只有在链表两头的操作`add()`， `addFirst()`，`removeLast()`或用 `iterator()` 上的 `remove()` 能省掉指针的移动。此外 LinkedList 还实现了 Deque（继承自Queue接口）接口，可以当做队列使用。 - -不会囊括所有方法，只是为了学习，记录思想。 - -ArrayList 和 LinkedList 两者都实现了 List 接口 - -```java -public class ArrayList extends AbstractList - implements List, RandomAccess, Cloneable, java.io.Serializable{ -``` - -```java -public class LinkedList - extends AbstractSequentialList - implements List, Deque, Cloneable, java.io.Serializable -``` - -### 构造器 - -ArrayList 提供了 3 个构造器，①无参构造器 ②带初始容量构造器 ③参数为集合构造器 - -```java -public class ArrayList extends AbstractList - implements List, RandomAccess, Cloneable, java.io.Serializable{ - -public ArrayList(int initialCapacity) { - if (initialCapacity > 0) { - // 创建初始容量的数组 - this.elementData = new Object[initialCapacity]; - } else if (initialCapacity == 0) { - this.elementData = EMPTY_ELEMENTDATA; - } else { - throw new IllegalArgumentException("Illegal Capacity: "+initialCapacity); - } -} -public ArrayList() { - // 默认为空数组 - this.elementData = DEFAULTCAPACITY_EMPTY_ELEMENTDATA; -} - -public ArrayList(Collection c) { //...} -} -``` - -LinkedList 提供了 2 个构造器，因为基于链表，所以也就没有初始化大小，也没有扩容的机制，就是一直在前面或者后面插插插~~ - -```java -public LinkedList() { -} - -public LinkedList(Collection c) { - this(); - addAll(c); -} -// LinkedList 既然作为链表，那么肯定会有节点 -private static class Node { - E item; - Node next; - Node prev; - - Node(Node prev, E element, Node next) { - this.item = element; - this.next = next; - this.prev = prev; - } -} -``` - -### 插入 - -**ArrayList**: - -```java -public boolean add(E e) { - // 确保数组的容量，保证可以添加该元素 - ensureCapacityInternal(size + 1); // Increments modCount!! - // 将该元素放入数组中 - elementData[size++] = e; - return true; -} -private void ensureCapacityInternal(int minCapacity) { - // 如果数组是空的，那么会初始化该数组 - if (elementData == DEFAULTCAPACITY_EMPTY_ELEMENTDATA) { - // DEFAULT_CAPACITY 为 10,所以调用无参默认 ArrayList 构造方法初始化的话，默认的数组容量为 10 - minCapacity = Math.max(DEFAULT_CAPACITY, minCapacity); - } - - ensureExplicitCapacity(minCapacity); -} - -private void ensureExplicitCapacity(int minCapacity) { - modCount++; - - // 确保数组的容量，如果不够的话，调用 grow 方法扩容 - if (minCapacity - elementData.length > 0) - grow(minCapacity); -} -//扩容具体的方法 -private void grow(int minCapacity) { - // 当前数组的容量 - int oldCapacity = elementData.length; - // 新数组扩容为原来容量的 1.5 倍 - int newCapacity = oldCapacity + (oldCapacity >> 1); - // 如果新数组扩容容量还是比最少需要的容量还要小的话，就设置扩充容量为最小需要的容量 - if (newCapacity - minCapacity < 0) - newCapacity = minCapacity; - //判断新数组容量是否已经超出最大数组范围，MAX_ARRAY_SIZE = Integer.MAX_VALUE - 8 - if (newCapacity - MAX_ARRAY_SIZE > 0) - newCapacity = hugeCapacity(minCapacity); - // minCapacity is usually close to size, so this is a win: - // 复制元素到新的数组中 - elementData = Arrays.copyOf(elementData, newCapacity); -} -``` - -当然也可以插入指定位置，还有一个重载的方法 `add(int index, E element)` - -```java -public void add(int index, E element) { - // 判断 index 有没有超出索引的范围 - rangeCheckForAdd(index); - // 和之前的操作是一样的，都是保证数组的容量足够 - ensureCapacityInternal(size + 1); // Increments modCount!! - // 将指定位置及其后面数据向后移动一位 - System.arraycopy(elementData, index, elementData, index + 1, - size - index); - // 将该元素添加到指定的数组位置 - elementData[index] = element; - // ArrayList 的大小改变 - size++; -} -``` - -可以看到每次插入指定位置都要移动元素，效率较低。 - -再来看 **LinkedList** 的插入，也有插入末尾，插入指定位置两种，由于基于链表，肯定得先有个 Node - -```java -private static class Node { - E item; - Node next; - Node prev; - - Node(Node prev, E element, Node next) { - this.item = element; - this.next = next; - this.prev = prev; - } -} -``` - -```java -public boolean add(E e) { - // 直接往队尾加元素 - linkLast(e); - return true; -} - -void linkLast(E e) { - // 保存原来链表尾部节点，last 是全局变量，用来表示队尾元素 - final Node l = last; - // 为该元素 e 新建一个节点 - final Node newNode = new Node<>(l, e, null); - // 将新节点设为队尾 - last = newNode; - // 如果原来的队尾元素为空，那么说明原来的整个列表是空的，就把新节点赋值给头结点 - if (l == null) - first = newNode; - else - // 原来尾结点的后面为新生成的结点 - l.next = newNode; - // 节点数 +1 - size++; - modCount++; -} - -public void add(int index, E element) { - // 检查 index 有没有超出索引范围 - checkPositionIndex(index); - // 如果追加到尾部，那么就跟 add(E e) 一样了 - if (index == size) - linkLast(element); - else - // 否则就是插在其他位置 - linkBefore(element, node(index)); -} - -//linkBefore方法中调用了这个node方法，类似二分查找的优化 -Node node(int index) { - // assert isElementIndex(index); - // 如果 index 在前半段，从前往后遍历获取 node - if (index < (size >> 1)) { - Node x = first; - for (int i = 0; i < index; i++) - x = x.next; - return x; - } else { - // 如果 index 在后半段，从后往前遍历获取 node - Node x = last; - for (int i = size - 1; i > index; i--) - x = x.prev; - return x; - } -} - -void linkBefore(E e, Node succ) { - // assert succ != null; - // 保存 index 节点的前节点 - final Node pred = succ.prev; - // 新建一个目标节点 - final Node newNode = new Node<>(pred, e, succ); - succ.prev = newNode; - // 如果是在开头处插入的话 - if (pred == null) - first = newNode; - else - pred.next = newNode; - size++; - modCount++; -} -``` - -### 获取 - -**ArrayList** 的 get() 方法很简单，就是在数组中返回指定位置的元素即可，所以效率很高 - -```java -public E get(int index) { - // 检查 index 有没有超出索引的范围 - rangeCheck(index); - // 返回指定位置的元素 - return elementData(index); -} -``` - -**LinkedList** 的 get() 方法，就是在内部调用了上边看到的 node() 方法，判断在前半段还是在后半段，然后遍历得到即可。 - -```java -public E get(int index) { - checkElementIndex(index); - return node(index).item; -} -``` - ------- - - - -## HashMap的底层实现 - -> 什么时候会使用HashMap？他有什么特点？ -> -> 你知道HashMap的工作原理吗？ -> -> 你知道get和put的原理吗？equals()和hashCode()的都有什么作用？ -> -> 你知道hash的实现吗？为什么要这样实现？ -> -> 如果HashMap的大小超过了负载因子(load factor)定义的容量，怎么办？ - -HashMap 在 JDK 7 和 JDK8 中的实现方式略有不同。分开记录。 - -深入 HahsMap 之前，先要了解的概念 - -1. initialCapacity：初始容量。指的是 HashMap 集合初始化的时候自身的容量。可以在构造方法中指定；如果不指定的话，总容量默认值是 16 。需要注意的是初始容量必须是 2 的幂次方。（**1.7中，已知HashMap中将要存放的KV个数的时候，设置一个合理的初始化容量可以有效的提高性能**） - - ```java - static final int DEFAULT_INITIAL_CAPACITY = 1 << 4; // aka 16 - ``` - -2. size：当前 HashMap 中已经存储着的键值对数量，即 `HashMap.size()` 。 - -3. loadFactor：加载因子。所谓的加载因子就是 HashMap (当前的容量/总容量) 到达一定值的时候，HashMap 会实施扩容。加载因子也可以通过构造方法中指定，默认的值是 0.75 。举个例子，假设有一个 HashMap 的初始容量为 16 ，那么扩容的阀值就是 0.75 * 16 = 12 。也就是说，在你打算存入第 13 个值的时候，HashMap 会先执行扩容。 - -4. threshold：扩容阀值。即 扩容阀值 = HashMap 总容量 * 加载因子。当前 HashMap 的容量大于或等于扩容阀值的时候就会去执行扩容。扩容的容量为当前 HashMap 总容量的两倍。比如，当前 HashMap 的总容量为 16 ，那么扩容之后为 32 。 - -5. table：Entry 数组。我们都知道 HashMap 内部存储 key/value 是通过 Entry 这个介质来实现的。而 table 就是 Entry 数组。 - -### JDK1.7 实现 - -JDK1.7 中 HashMap 由 **数组+链表** 组成（**“链表散列”** 即数组和链表的结合体），数组是 HashMap 的主体，链表则是主要为了解决哈希冲突而存在的（HashMap 采用 **“拉链法也就是链地址法”** 解决冲突），如果定位到的数组位置不含链表（当前 entry 的 next 指向 null ）,那么对于查找，添加等操作很快，仅需一次寻址即可；如果定位到的数组包含链表，对于添加操作，其时间复杂度依然为 O(1)，因为最新的 Entry 会插入链表头部，即需要简单改变引用链即可，而对于查找操作来讲，此时就需要遍历链表，然后通过 key 对象的 equals 方法逐一比对查找。 - -> 所谓 **“拉链法”** 就是将链表和数组相结合。也就是说创建一个链表数组，数组中每一格就是一个链表。若遇到哈希冲突，则将冲突的值加到链表中即可。 - -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gdx45a2fbbj31dz0oaacx.jpg) - -#### 源码解析 - -##### 构造方法 - -《阿里巴巴 Java 开发手册》推荐集合初始化时，指定集合初始值大小。（说明：HashMap 使用HashMap(int initialCapacity) 初始化）建议原因： https://www.zhihu.com/question/314006228/answer/611170521 - -```java -// 默认的构造方法使用的都是默认的初始容量和加载因子 -// DEFAULT_INITIAL_CAPACITY = 16，DEFAULT_LOAD_FACTOR = 0.75f -public HashMap() { - this(DEFAULT_INITIAL_CAPACITY, DEFAULT_LOAD_FACTOR); -} - -// 可以指定初始容量，并且使用默认的加载因子 -public HashMap(int initialCapacity) { - this(initialCapacity, DEFAULT_LOAD_FACTOR); -} - -public HashMap(int initialCapacity, float loadFactor) { - // 对初始容量的值判断 - if (initialCapacity < 0) - throw new IllegalArgumentException("Illegal initial capacity: " + - initialCapacity); - if (initialCapacity > MAXIMUM_CAPACITY) - initialCapacity = MAXIMUM_CAPACITY; - if (loadFactor <= 0 || Float.isNaN(loadFactor)) - throw new IllegalArgumentException("Illegal load factor: " + - loadFactor); - // 设置加载因子 - this.loadFactor = loadFactor; - threshold = initialCapacity; - // 空方法 - init(); -} - -public HashMap(Map m) { - this(Math.max((int) (m.size() / DEFAULT_LOAD_FACTOR) + 1, - DEFAULT_INITIAL_CAPACITY), DEFAULT_LOAD_FACTOR); - inflateTable(threshold); - putAllForCreate(m); -} -``` - -HashMap 的前 3 个构造方法最后都会去调用 `HashMap(int initialCapacity, float loadFactor)` 。在其内部去设置初始容量和加载因子。而最后的 `init()` 是空方法，主要给子类实现，比如LinkedHashMap。 - -##### put() 方法 - -```java -public V put(K key, V value) { - // 如果 table 数组为空时先创建数组，并且设置扩容阀值 - if (table == EMPTY_TABLE) { - inflateTable(threshold); - } - // 如果 key 为空时，调用 putForNullKey 方法特殊处理 - if (key == null) - return putForNullKey(value); - // 计算 key 的哈希值 - int hash = hash(key); - // 根据计算出来的哈希值和当前数组的长度计算在数组中的索引 - int i = indexFor(hash, table.length); - // 先遍历该数组索引下的整条链表 - // 如果该 key 之前已经在 HashMap 中存储了的话，直接替换对应的 value 值即可 - for (Entry e = table[i]; e != null; e = e.next) { - Object k; - //先判断hash值是否一样，如果一样，再判断key是否一样，不同对象的hash值可能一样 - if (e.hash == hash && ((k = e.key) == key || key.equals(k))) { - V oldValue = e.value; - e.value = value; - e.recordAccess(this); - return oldValue; - } - } - - modCount++; - // 如果该 key 之前没有被存储过，那么就进入 addEntry 方法 - addEntry(hash, key, value, i); - return null; -} - -void addEntry(int hash, K key, V value, int bucketIndex) { - // 当前容量大于或等于扩容阀值的时候，会执行扩容 - if ((size >= threshold) && (null != table[bucketIndex])) { - // 扩容为原来容量的两倍 - resize(2 * table.length); - // 重新计算哈希值 - hash = (null != key) ? hash(key) : 0; - // 重新得到在新数组中的索引 - bucketIndex = indexFor(hash, table.length); - } - // 创建节点 - createEntry(hash, key, value, bucketIndex); -} - -//扩容，创建了一个新的数组，然后把数据全部复制过去，再把新数组的引用赋给 table -void resize(int newCapacity) { - Entry[] oldTable = table; //引用扩容前的Entry数组 - int oldCapacity = oldTable.length; - if (oldCapacity == MAXIMUM_CAPACITY) { //扩容前的数组大小如果已经达到最大(2^30)了 - threshold = Integer.MAX_VALUE; //修改阈值为int的最大值(2^31-1)，这样以后就不会扩容了 - return; - } - // 创建新的 entry 数组 - Entry[] newTable = new Entry[newCapacity]; - // 将旧 entry 数组中的数据复制到新 entry 数组中 - transfer(newTable, initHashSeedAsNeeded(newCapacity)); - // 将新数组的引用赋给 table - table = newTable; - // 计算新的扩容阀值 - threshold = (int)Math.min(newCapacity * loadFactor, MAXIMUM_CAPACITY + 1); -} - -void transfer(Entry[] newTable) { - Entry[] src = table; //src引用了旧的Entry数组 - int newCapacity = newTable.length; - for (int j = 0; j < src.length; j++) { //遍历旧的Entry数组 - Entry e = src[j]; //取得旧Entry数组的每个元素 - if (e != null) { - src[j] = null;//释放旧Entry数组的对象引用（for循环后，旧的Entry数组不再引用任何对象） - do { - Entry next = e.next; - int i = indexFor(e.hash, newCapacity); //！！重新计算每个元素在数组中的位置 - e.next = newTable[i]; //标记[1] - newTable[i] = e; //将元素放在数组上 - e = next; //访问下一个Entry链上的元素 - } while (e != null); - } - } -} - -void createEntry(int hash, K key, V value, int bucketIndex) { - // 取出table中下标为bucketIndex的Entry - Entry e = table[bucketIndex]; - // 利用key、value来构建新的Entry - // 并且之前存放在table[bucketIndex]处的Entry作为新Entry的next - // 把新创建的Entry放到table[bucketIndex]位置 - table[bucketIndex] = new Entry<>(hash, key, value, e); - // 当前 HashMap 的容量加 1 - size++; -} -``` - -最后的 createEntry() 方法就说明了当hash冲突时，采用的拉链法来解决hash冲突的，并且是把新元素是插入到单边表的表头。 - -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gdx45xr0x5j31hl0u0ncd.jpg) - -##### get() 方法 - -```java -public V get(Object key) { - // 如果 key 是空的，就调用 getForNullKey 方法特殊处理 - if (key == null) - return getForNullKey(); - // 获取 key 相对应的 entry - Entry entry = getEntry(key); - - return null == entry ? null : entry.getValue(); -} - -//找到对应 key 的数组索引，然后遍历链表查找即可 -final Entry getEntry(Object key) { - if (size == 0) { - return null; - } - // 计算 key 的哈希值 - int hash = (key == null) ? 0 : hash(key); - // 得到数组的索引，然后遍历链表，查看是否有相同 key 的 Entry - for (Entry e = table[indexFor(hash, table.length)]; - e != null; - e = e.next) { - Object k; - if (e.hash == hash && - ((k = e.key) == key || (key != null && key.equals(k)))) - return e; - } - // 没有的话，返回 null - return null; -} -``` - -### JDK1.8 实现 - -JDK 1.7 中，如果哈希碰撞过多，拉链过长，极端情况下，所有值都落入了同一个桶内，这就退化成了一个链表。通过 key 值查找要遍历链表，效率较低。 JDK1.8 在解决哈希冲突时有了较大的变化，**当链表长度大于阈值（默认为8）时，将链表转化为红黑树**，以减少搜索时间。 - -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gdx468uknnj31650ogjth.jpg) - -TreeMap、TreeSet 以及 JDK1.8 之后的 HashMap 底层都用到了红黑树。红黑树就是为了解决二叉查找树的缺陷，因为二叉查找树在某些情况下会退化成一个线性结构。 - -#### 源码解析 - -##### 构造方法 - -JDK8 构造方法改动不是很大 - -```java -public HashMap() { - this.loadFactor = DEFAULT_LOAD_FACTOR; // all other fields defaulted -} - -public HashMap(int initialCapacity) { - this(initialCapacity, DEFAULT_LOAD_FACTOR); -} - -public HashMap(int initialCapacity, float loadFactor) { - if (initialCapacity < 0) - throw new IllegalArgumentException("Illegal initial capacity: " + - initialCapacity); - if (initialCapacity > MAXIMUM_CAPACITY) - initialCapacity = MAXIMUM_CAPACITY; - if (loadFactor <= 0 || Float.isNaN(loadFactor)) - throw new IllegalArgumentException("Illegal load factor: " + - loadFactor); - this.loadFactor = loadFactor; - this.threshold = tableSizeFor(initialCapacity); -} - -public HashMap(Map m) { - this.loadFactor = DEFAULT_LOAD_FACTOR; - putMapEntries(m, false); -} -``` - -##### 确定哈希桶数组索引位置（hash 函数的实现） - -```java -//方法一： -static final int hash(Object key) { //jdk1.8 & jdk1.7 - int h; - // h = key.hashCode() 为第一步 取hashCode值 - // h ^ (h >>> 16) 为第二步 高位参与运算 - return (key == null) ? 0 : (h = key.hashCode()) ^ (h >>> 16); -} -//方法二： -static int indexFor(int h, int length) { //jdk1.7的源码，jdk1.8没有提取这个方法，而是放在了其他方法中，比如 put 的p = tab[i = (n - 1) & hash] - return h & (length-1); //第三步 取模运算 -} -``` - -HashMap 定位数组索引位置，直接决定了 hash 方法的离散性能。Hash 算法本质上就是三步：**取key的hashCode值、高位运算、取模运算**。 - -![hash](https://tva1.sinaimg.cn/large/007S8ZIlly1gdwv5m4g4sj30ga09cdfy.jpg) - -> 为什么要这样呢？ -> -> HashMap 的长度为什么是2的幂次方? - -目的当然是为了减少哈希碰撞，使 table 里的数据分布的更均匀。 - -1. HashMap 中桶数组的大小 length 总是2的幂，此时，`h & (table.length -1)` 等价于对 length 取模 `h%length`。但取模的计算效率没有位运算高，所以这是是一个优化。假设 `h = 185`，`table.length-1 = 15(0x1111)`，其实散列真正生效的只是低 4bit 的有效位，所以很容易碰撞。 - - ![img](https://tva1.sinaimg.cn/large/007S8ZIlly1gdx46gac50j30m8037mxf.jpg) - -2. 图中的 hash 是由键的 hashCode 产生。计算余数时，由于 n 比较小，hash 只有低4位参与了计算，高位的计算可以认为是无效的。这样导致了计算结果只与低位信息有关，高位数据没发挥作用。为了处理这个缺陷，我们可以上图中的 hash 高4位数据与低4位数据进行异或运算，即 `hash ^ (hash >>> 4)`。通过这种方式，让高位数据与低位数据进行异或，以此加大低位信息的随机性，变相的让高位数据参与到计算中。此时的计算过程如下： - - ![img](https://tva1.sinaimg.cn/large/007S8ZIlly1gdx46jwezjj30m804cjrw.jpg) - - 在 Java 中，hashCode 方法产生的 hash 是 int 类型，32 位宽。前16位为高位，后16位为低位，所以要右移16位，即 `hash ^ (hash >>> 16)` 。这样还增加了hash 的复杂度，进而影响 hash 的分布性。 - -**HashMap 的长度为什么是2的幂次方？** - -为了能让HashMap存取高效，尽量减少碰撞，也就是要尽量把数据分配均匀，Hash值的范围是-2147483648到2147483647，前后加起来有40亿的映射空间，只要哈希函数映射的比较均匀松散，一般应用是很难出现碰撞的，但一个问题是40亿的数组内存是放不下的。所以这个散列值是不能直接拿来用的。用之前需要先对数组长度取模运算，得到余数才能用来存放位置也就是对应的数组小标。这个数组下标的计算方法是 `(n-1)&hash`，n代表数组长度。 - -这个算法应该如何设计呢？ - -我们首先可能会想到采用%取余的操作来实现。但是，重点来了。 - -取余操作中如果除数是2的幂次则等价于其除数减一的与操作，也就是说 `hash%length=hash&(length-1)`，但前提是 length 是 2 的 n 次方，并且采用 &运算比 %运算效率高，这也就解释了 HashMap 的长度为什么是2的幂次方。 - -##### get() 方法 - -```java -public V get(Object key) { - Node e; - return (e = getNode(hash(key), key)) == null ? null : e.value; -} - -final Node getNode(int hash, Object key) { - Node[] tab; Node first, e; int n; K k; - //定位键值对所在桶的位置 - if ((tab = table) != null && (n = tab.length) > 0 && - (first = tab[(n - 1) & hash]) != null) { - // 直接命中 - if (first.hash == hash && // always check first node - ((k = first.key) == key || (key != null && key.equals(k)))) - return first; - // 未命中 - if ((e = first.next) != null) { - // 如果 first 是 TreeNode 类型，则调用黑红树查找方法 - if (first instanceof TreeNode) - return ((TreeNode)first).getTreeNode(hash, key); - do { - // 在链表中查找 - if (e.hash == hash && - ((k = e.key) == key || (key != null && key.equals(k)))) - return e; - } while ((e = e.next) != null); - } - } - return null; -} -``` - -##### put() 方法 - -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gdx470vyzej311c0u0120.jpg) - -```java -public V put(K key, V value) { - // 对key的hashCode()做hash - return putVal(hash(key), key, value, false, true); -} - -final V putVal(int hash, K key, V value, boolean onlyIfAbsent, - boolean evict) { - Node[] tab; Node p; int n, i; - // tab为空则创建 - if ((tab = table) == null || (n = tab.length) == 0) - n = (tab = resize()).length; - // 计算index，并对null做处理 - if ((p = tab[i = (n - 1) & hash]) == null) - tab[i] = newNode(hash, key, value, null); - else { - Node e; K k; - // 节点key存在，直接覆盖value - if (p.hash == hash && - ((k = p.key) == key || (key != null && key.equals(k)))) - e = p; - // 判断该链为红黑树 - else if (p instanceof TreeNode) - e = ((TreeNode)p).putTreeVal(this, tab, hash, key, value); - else { - //该链为链表 - for (int binCount = 0; ; ++binCount) { - if ((e = p.next) == null) { - p.next = newNode(hash, key, value, null); - //链表长度大于8转换为红黑树进行处理 - if (binCount >= TREEIFY_THRESHOLD - 1) // -1 for 1st - treeifyBin(tab, hash); - break; - } - //key已经存在直接覆盖value - if (e.hash == hash && - ((k = e.key) == key || (key != null && key.equals(k)))) - break; - p = e; - } - } - if (e != null) { // existing mapping for key - V oldValue = e.value; - if (!onlyIfAbsent || oldValue == null) - e.value = value; - afterNodeAccess(e); - return oldValue; - } - } - ++modCount; - // 超过最大容量 就扩容 - if (++size > threshold) - resize(); - afterNodeInsertion(evict); - return null; -} -``` - -> HashMap的put方法的具体流程？ - -①.判断键值对数组table[i]是否为空或为null，否则执行resize()进行扩容； - -②.根据键值key计算hash值得到插入的数组索引i，如果table[i]==null，直接新建节点添加，转向⑥，如果table[i]不为空，转向③； - -③.判断table[i]的首个元素是否和key一样，如果相同直接覆盖value，否则转向④，这里的相同指的是hashCode以及equals； - -④.判断table[i] 是否为treeNode，即table[i] 是否是红黑树，如果是红黑树，则直接在树中插入键值对，否则转向⑤； - -⑤.遍历table[i]，判断链表长度是否大于8，大于8的话把链表转换为红黑树，在红黑树中执行插入操作，否则进行链表的插入操作；遍历过程中若发现key已经存在直接覆盖value即可； - -⑥.插入成功后，判断实际存在的键值对数量size是否超多了最大容量threshold，如果超过，进行扩容。 - -##### resize() 扩容 - -```java -final Node[] resize() { - Node[] oldTab = table; - int oldCap = (oldTab == null) ? 0 : oldTab.length; - int oldThr = threshold; - int newCap, newThr = 0; - if (oldCap > 0) { - // 超过最大值就不再扩充了，就只好随你碰撞了 - if (oldCap >= MAXIMUM_CAPACITY) { - //修改阈值为int的最大值(2^31-1)，这样以后就不会扩容了 - threshold = Integer.MAX_VALUE; - return oldTab; - } - // 没超过最大值，就扩充为原来的2倍 - else if ((newCap = oldCap << 1) < MAXIMUM_CAPACITY && - oldCap >= DEFAULT_INITIAL_CAPACITY) - newThr = oldThr << 1; // double threshold - } - else if (oldThr > 0) // initial capacity was placed in threshold - newCap = oldThr; - else { // zero initial threshold signifies using defaults - newCap = DEFAULT_INITIAL_CAPACITY; - newThr = (int)(DEFAULT_LOAD_FACTOR * DEFAULT_INITIAL_CAPACITY); - } - // 计算新的resize上限 - if (newThr == 0) { - float ft = (float)newCap * loadFactor; - newThr = (newCap < MAXIMUM_CAPACITY && ft < (float)MAXIMUM_CAPACITY ? - (int)ft : Integer.MAX_VALUE); - } - threshold = newThr; - // 开始复制到新的数组 - @SuppressWarnings({"rawtypes","unchecked"}) - Node[] newTab = (Node[])new Node[newCap]; - table = newTab; - // 把每个bucket都移动到新的buckets中 - if (oldTab != null) { - // 循环遍历旧的 table 数组 - for (int j = 0; j < oldCap; ++j) { - Node e; - if ((e = oldTab[j]) != null) { - oldTab[j] = null; - // 如果该桶只有一个元素，那么直接复制 - if (e.next == null) - newTab[e.hash & (newCap - 1)] = e; - // 如果是红黑树，那么对红黑树进行拆分 - else if (e instanceof TreeNode) - ((TreeNode)e).split(this, newTab, j, oldCap); - // 遍历链表，将原链表节点分成lo和hi两个链表 - // 其中 lo 表示在原来的桶位置，hi 表示在新的桶位置 - else { // preserve order 链表优化重hash的代码块 - Node loHead = null, loTail = null; - Node hiHead = null, hiTail = null; - Node next; - do { - // 原索引 - next = e.next; - if ((e.hash & oldCap) == 0) { - if (loTail == null) - loHead = e; - else - loTail.next = e; - loTail = e; - } - // 原索引+oldCap - else { - if (hiTail == null) - hiHead = e; - else - hiTail.next = e; - hiTail = e; - } - } while ((e = next) != null); - // 原索引放到bucket里 - if (loTail != null) { - loTail.next = null; - newTab[j] = loHead; - } - // 原索引+oldCap放到bucket里 - if (hiTail != null) { - hiTail.next = null; - newTab[j + oldCap] = hiHead; - } - } - } - } - } - return newTab; -} -``` - -> HashMap的扩容操作是怎么实现的？ - -1. 在jdk1.8中，resize方法是在HashMap中的键值对大于阀值时或者初始化时，就调用resize方法进行扩容； -2. 每次扩展的时候，都是扩展2倍； -3. 扩展后Node对象的位置要么在原位置，要么移动到原偏移量两倍的位置。 - -在 `putVal()` 中，我们看到在这个函数里面使用到了2次 `resize()` 方法，`resize()` 方法表示在进行第一次初始化时会对其进行扩容，或者当该数组的实际大小大于其扩容阈值（第一次为0.75 * 16 = 12），这个时候在扩容的同时也会伴随的桶上面的元素进行重新分发，这也是JDK1.8版本的一个优化的地方，在1.7中，扩容之后需要重新去计算其Hash值，根据Hash值对其进行分发，但在1.8版本中，则是根据在同一个桶的位置中进行判断(e.hash & oldCap)是否为0，重新进行hash分配后，该元素的位置要么停留在原始位置，要么移动到原始位置+增加的数组大小这个位置上 - - - -##### 链表树化 - -当桶中链表长度超过 TREEIFY_THRESHOLD（默认为8）时，就会调用 treeifyBin 方法进行树化操作。但此时并不一定会树化，因为在 treeifyBin 方法中还会判断 HashMap 的容量是否大于等于 64。如果容量大于等于 64，那么进行树化，否则优先进行扩容。 - -为什么树化还要判断整体容量是否大于 64 呢？ - -当桶数组容量比较小时，键值对节点 hash 的碰撞率可能会比较高，进而导致链表长度较长，从而导致查询效率低下。这时候我们有两种选择，一种是扩容，让哈希碰撞率低一些。另一种是树化，提高查询效率。 - -如果我们采用扩容，那么我们需要做的就是做一次链表数据的复制。而如果我们采用树化，那么我们需要将链表转化成红黑树。到这里，貌似没有什么太大的区别，但我们让场景继续延伸下去。当插入的数据越来越多，我们会发现需要转化成树的链表越来越多。 - -到了一定容量，我们就需要进行扩容了。这个时候我们有许多树化的红黑树，在扩容之时，我们需要将许多的红黑树拆分成链表，这是一个挺大的成本。而如果我们在容量小的时候就进行扩容，那么需要树化的链表就越少，我们扩容的成本也就越低。 - -接下来我们看看链表树化是怎么做的： - -```java - final void treeifyBin(Node[] tab, int hash) { - int n, index; Node e; - // 1. 容量小于 MIN_TREEIFY_CAPACITY，优先扩容 - if (tab == null || (n = tab.length) < MIN_TREEIFY_CAPACITY) - resize(); - // 2. 桶不为空，那么进行树化操作 - else if ((e = tab[index = (n - 1) & hash]) != null) { - TreeNode hd = null, tl = null; - // 2.1 先将链表转成 TreeNode 表示的双向链表 - do { - TreeNode p = replacementTreeNode(e, null); - if (tl == null) - hd = p; - else { - p.prev = tl; - tl.next = p; - } - tl = p; - } while ((e = e.next) != null); - // 2.2 将 TreeNode 表示的双向链表树化 - if ((tab[index] = hd) != null) - hd.treeify(tab); - } - } -``` - -我们可以看到链表树化的整体思路也比较清晰。首先将链表转成 TreeNode 表示的双向链表，之后再调用 `treeify()` 方法进行树化操作。那么我们继续看看 `treeify()` 方法的实现。 - -```java -final void treeify(Node[] tab) { - TreeNode root = null; - // 1. 遍历双向 TreeNode 链表，将 TreeNode 节点一个个插入 - for (TreeNode x = this, next; x != null; x = next) { - next = (TreeNode)x.next; - x.left = x.right = null; - // 2. 如果 root 节点为空，那么直接将 x 节点设置为根节点 - if (root == null) { - x.parent = null; - x.red = false; - root = x; - } - else { - K k = x.key; - int h = x.hash; - Class kc = null; - // 3. 如果 root 节点不为空，那么进行比较并在合适的地方插入 - for (TreeNode p = root;;) { - int dir, ph; - K pk = p.key; - // 4. 计算 dir 值，-1 表示要从左子树查找插入点，1表示从右子树 - if ((ph = p.hash) > h) - dir = -1; - else if (ph < h) - dir = 1; - else if ((kc == null && - (kc = comparableClassFor(k)) == null) || - (dir = compareComparables(kc, k, pk)) == 0) - dir = tieBreakOrder(k, pk); - - TreeNode xp = p; - // 5. 如果查找到一个 p 点，这个点是叶子节点 - // 那么这个就是插入位置 - if ((p = (dir <= 0) ? p.left : p.right) == null) { - x.parent = xp; - if (dir <= 0) - xp.left = x; - else - xp.right = x; - // 做插入后的动平衡 - root = balanceInsertion(root, x); - break; - } - } - } - } - // 6.将 root 节点移动到链表头 - moveRootToFront(tab, root); -} -``` - -从上面代码可以看到，treeify() 方法其实就是将双向 TreeNode 链表进一步树化成红黑树。其中大致的步骤为： - -- 遍历 TreeNode 双向链表，将 TreeNode 节点一个个插入 -- 如果 root 节点为空，那么表示红黑树现在为空，直接将该节点作为根节点。否则需要查找到合适的位置去插入 TreeNode 节点。 -- 通过比较与 root 节点的位置，不断寻找最合适的点。如果最终该节点的叶子节点为空，那么该节点 p 就是插入节点的父节点。接着，将 x 节点的 parent 引用指向 xp 节点，xp 节点的左子节点或右子节点指向 x 节点。 -- 接着，调用 balanceInsertion 做一次动态平衡。 -- 最后，调用 moveRootToFront 方法将 root 节点移动到链表头。 - -关于 balanceInsertion() 动平衡可以参考红黑树的插入动平衡，这里暂不深入讲解。最后我们继续看看 moveRootToFront 方法。 - -```java -static void moveRootToFront(Node[] tab, TreeNode root) { - int n; - if (root != null && tab != null && (n = tab.length) > 0) { - int index = (n - 1) & root.hash; - TreeNode first = (TreeNode)tab[index]; - // 如果插入红黑树的 root 节点不是链表的第一个元素 - // 那么将 root 节点取出来，插在 first 节点前面 - if (root != first) { - Node rn; - tab[index] = root; - TreeNode rp = root.prev; - // 下面的两个 if 语句，做的事情是将 root 节点取出来 - // 让 root 节点的前继指向其后继节点 - // 让 root 节点的后继指向其前继节点 - if ((rn = root.next) != null) - ((TreeNode)rn).prev = rp; - if (rp != null) - rp.next = rn; - // 这里直接让 root 节点插入到 first 节点前方 - if (first != null) - first.prev = root; - root.next = first; - root.prev = null; - } - assert checkInvariants(root); - } -} -``` - -##### 红黑树拆分 - -扩容后，普通节点需要重新映射，红黑树节点也不例外。按照一般的思路，我们可以先把红黑树转成链表，之后再重新映射链表即可。但因为红黑树插入的时候，TreeNode 保存了元素插入的顺序，所以直接可以按照插入顺序还原成链表。这样就避免了将红黑树转成链表后再进行哈希映射，无形中提高了效率。 - -```java -final void split(HashMap map, Node[] tab, int index, int bit) { - TreeNode b = this; - // Relink into lo and hi lists, preserving order - TreeNode loHead = null, loTail = null; - TreeNode hiHead = null, hiTail = null; - int lc = 0, hc = 0; - // 1. 将红黑树当成是一个 TreeNode 组成的双向链表 - // 按照链表扩容一样，分别放入 loHead 和 hiHead 开头的链表中 - for (TreeNode e = b, next; e != null; e = next) { - next = (TreeNode)e.next; - e.next = null; - // 1.1. 扩容后的位置不变，还是原来的位置，该节点放入 loHead 链表 - if ((e.hash & bit) == 0) { - if ((e.prev = loTail) == null) - loHead = e; - else - loTail.next = e; - loTail = e; - ++lc; - } - // 1.2 扩容后的位置改变了，放入 hiHead 链表 - else { - if ((e.prev = hiTail) == null) - hiHead = e; - else - hiTail.next = e; - hiTail = e; - ++hc; - } - } - // 2. 对 loHead 和 hiHead 进行树化或链表化 - if (loHead != null) { - // 2.1 如果链表长度小于阈值，那就链表化，否则树化 - if (lc <= UNTREEIFY_THRESHOLD) - tab[index] = loHead.untreeify(map); - else { - tab[index] = loHead; - if (hiHead != null) // (else is already treeified) - loHead.treeify(tab); - } - } - if (hiHead != null) { - if (hc <= UNTREEIFY_THRESHOLD) - tab[index + bit] = hiHead.untreeify(map); - else { - tab[index + bit] = hiHead; - if (loHead != null) - hiHead.treeify(tab); - } - } -} -``` - -从上面的代码我们知道红黑树的扩容也和链表的转移是一样的，不同的是其转化成 hiHead 和 loHead 之后，会根据链表长度选择拆分成链表还是继承维持红黑树的结构。 - -##### 红黑树链化 - -我们在说到红黑树拆分的时候说到，红黑树结构在扩容的时候如果长度低于阈值，那么就会将其转化成链表。其实现代码如下： - -```java -final Node untreeify(HashMap map) { - Node hd = null, tl = null; - for (Node q = this; q != null; q = q.next) { - Node p = map.replacementNode(q, null); - if (tl == null) - hd = p; - else - tl.next = p; - tl = p; - } - return hd; -} -``` - -因为红黑树中包含了插入元素的顺序，所以当我们将红黑树拆分成两个链表 hiHead 和 loHead 时，其还是保持着原来的顺序的。所以此时我们只需要循环遍历一遍，然后将 TreeNode 节点换成 Node 节点即可。 - - - -#### HashMap：JDK1.7 VS JDK1.8 - -JDK1.8主要解决或优化了一下问题： - -- resize 扩容优化 -- 引入了红黑树，目的是避免单条链表过长而影响查询效率 -- 解决了多线程死循环问题，但仍是非线程安全的，多线程时可能会造成数据丢失问题 - -| 不同 | JDK 1.7 | JDK 1.8 | -| ------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | -| 存储结构 | 数组 + 链表 | 数组 + 链表 + 红黑树 | -| 初始化方式 | 单独函数：inflateTable() | 直接集成到了扩容函数resize()中 | -| hash值计算方式 | 扰动处理 = 9次扰动 = 4次位运算 + 5次异或运算 | 扰动处理 = 2次扰动 = 1次位运算 + 1次异或运算 | -| 存放数据的规则 | 无冲突时，存放数组；冲突时，存放链表 | 无冲突时，存放数组；冲突 & 链表长度 < 8：存放单链表；冲突 & 链表长度 > 8：树化并存放红黑树 | -| 插入数据方式 | 头插法（先讲原位置的数据移到后1位，再插入数据到该位置） | 尾插法（直接插入到链表尾部/红黑树） | -| 扩容后存储位置的计算方式 | 全部按照原来方法进行计算（即hashCode ->> 扰动函数 ->> (h&length-1)） | 按照扩容后的规律计算（即扩容后的位置=原位置 or 原位置 + 旧容量 | - ------- - - - -## Hashtable - -Hashtable 和 HashMap 都是散列表，也是用”拉链法“实现的哈希表。保存数据和 JDK7 中的 HashMap 一样，是 Entity 对象，只是 Hashtable 中的几乎所有的 public 方法都是 synchronized 的，而有些方法也是在内部通过 synchronized 代码块来实现，效率肯定会降低。且 put() 方法不允许空值。 - - - -### HashMap 和 Hashtable 的区别 - -1. **线程是否安全：** HashMap 是非线程安全的，HashTable 是线程安全的；HashTable 内部的方法基本都经过 `synchronized` 修饰。（如果你要保证线程安全的话就使用 ConcurrentHashMap 吧！）； - -2. **效率：** 因为线程安全的问题，HashMap 要比 HashTable 效率高一点。另外，HashTable 基本被淘汰，不要在代码中使用它； - -3. **对Null key 和Null value的支持：** HashMap 中，null 可以作为键，这样的键只有一个，可以有一个或多个键所对应的值为 null。。但是在 HashTable 中 put 进的键值只要有一个 null，直接抛出 NullPointerException。 - -4. **初始容量大小和每次扩充容量大小的不同 ：** - - ① 创建时如果不指定容量初始值，Hashtable 默认的初始大小为11，之后每次扩充，容量变为原来的2n+1。HashMap 默认的初始化大小为16。之后每次扩充，容量变为原来的2倍。 - - ② 创建时如果给定了容量初始值，那么 Hashtable 会直接使用你给定的大小，而 HashMap 会将其扩充为2的幂次方大小。也就是说 HashMap 总是使用2的幂次方作为哈希表的大小,后面会介绍到为什么是2的幂次方。 - -5. **底层数据结构：** JDK1.8 以后的 HashMap 在解决哈希冲突时有了较大的变化，当链表长度大于阈值（默认为8）时，将链表转化为红黑树，以减少搜索时间。Hashtable 没有这样的机制。 - -6. HashMap的迭代器（`Iterator`）是fail-fast迭代器，但是 Hashtable的迭代器（`enumerator`）不是 fail-fast的。如果有其它线程对HashMap进行的添加/删除元素，将会抛出`ConcurrentModificationException`，但迭代器本身的`remove`方法移除元素则不会抛出异常。这条同样也是 Enumeration 和 Iterator 的区别。 - - - -## ConcurrentHashMap - -HashMap 在多线程情况下，在 put 的时候，插入的元素超过了容量（由负载因子决定）的范围就会触发扩容操作，就是rehash，这个会重新将原数组的内容重新hash到新的扩容数组中，在多线程的环境下，存在同时其他的元素也在进行put操作，如果hash值相同，可能出现同时在同一数组下用链表表示，造成**闭环**，导致在get时会出现死循环，所以 HashMap 是线程不安全的。(可参考：https://www.jianshu.com/p/e2f75c8cce01） - -Hashtable，是线程安全的，它在所有涉及到多线程操作的都加上了synchronized关键字来锁住整个table，这就意味着所有的线程都在竞争一把锁，在多线程的环境下，它是安全的，但是无疑是效率低下的。 - -### JDK1.7 实现 - -Hashtable 容器在竞争激烈的并发环境下表现出效率低下的原因，是因为所有访问 Hashtable 的线程都必须竞争同一把锁，那假如容器里有多把锁，每一把锁用于锁容器其中一部分数据，那么当多线程访问容器里不同数据段的数据时，线程间就不会存在锁竞争，这就是 ConcurrentHashMap 所使用的锁分段技术。 - -在 JDK1.7 版本中，ConcurrentHashMap 的数据结构是由一个 Segment 数组和多个 HashEntry 组成。Segment 数组的意义就是将一个大的 table 分割成多个小的 table 来进行加锁。每一个 Segment 元素存储的是 HashEntry数组+链表，这个和 HashMap 的数据存储结构一样。 - -![](https://yfzhou.oss-cn-beijing.aliyuncs.com/blog/img/JDK1.7%20ConcurrentHashMap.jpg) - -ConcurrentHashMap 类中包含两个静态内部类 HashEntry 和 Segment。 -HashEntry 用来封装映射表的键值对，Segment 用来充当锁的角色，每个 Segment 对象守护整个散列映射表的若干个桶。每个桶是由若干个 HashEntry 对象链接起来的链表。一个 ConcurrentHashMap 实例中包含由若干个 Segment 对象组成的数组。每个 Segment 守护着一个 HashEntry 数组里的元素，当对 HashEntry 数组的数据进行修改时，必须首先获得它对应的 Segment 锁。 - -#### Segment 类 - -Segment 类继承于 ReentrantLock 类，从而使得 Segment 对象能充当可重入锁的角色。一个 Segment 就是一个子哈希表，Segment 里维护了一个 HashEntry 数组，并发环境下，对于不同 Segment 的数据进行操作是不用考虑锁竞争的。 - -从源码可以看到，Segment 内部类和我们上边看到的 HashMap 很相似。也有负载因子，阈值等各种属性。 - -```java -static final class Segment extends ReentrantLock implements Serializable { - - static final int MAX_SCAN_RETRIES = - Runtime.getRuntime().availableProcessors() > 1 ? 64 : 1; - transient volatile HashEntry[] table; - transient int count; - transient int modCount; //记录修改次数 - transient int threshold; - final float loadFactor; - - Segment(float lf, int threshold, HashEntry[] tab) { - this.loadFactor = lf; - this.threshold = threshold; - this.table = tab; - } - - //put 方法会有加锁操作， - final V put(K key, int hash, V value, boolean onlyIfAbsent) { - HashEntry node = tryLock() ? null : - scanAndLockForPut(key, hash, value); - // ... - } - - @SuppressWarnings("unchecked") - private void rehash(HashEntry node) { - // ... - } - - private HashEntry scanAndLockForPut(K key, int hash, V value) { - //... - } - - private void scanAndLock(Object key, int hash) { - //... - } - - final V remove(Object key, int hash, Object value) { - //... - } - - final boolean replace(K key, int hash, V oldValue, V newValue) { - //... - } - - final V replace(K key, int hash, V value) { - //... - } - - final void clear() { - //... - } -} -``` - -#### HashEntry 类 - -HashEntry 是目前最小的逻辑处理单元。一个 ConcurrentHashMap 维护一个 Segment 数组，一个 Segment 维护一个 HashEntry 数组。 - -```java -static final class HashEntry { - final int hash; - final K key; - volatile V value; // value 为 volatie 类型，保证可见 - volatile HashEntry next; - //... -} -``` - -#### ConcurrentHashMap 类 - -默认的情况下，每个 ConcurrentHashMap 类会创建16个并发的 segment，每个 segment 里面包含多个 Hash表，每个 Hash 链都是由 HashEntry 节点组成的。 - -```java -public class ConcurrentHashMap extends AbstractMap - implements ConcurrentMap, Serializable { - //默认初始容量为 16，即初始默认为 16 个桶 - static final int DEFAULT_INITIAL_CAPACITY = 16; - static final float DEFAULT_LOAD_FACTOR = 0.75f; - //默认并发级别为 16。该值表示当前更新线程的估计数 - static final int DEFAULT_CONCURRENCY_LEVEL = 16; - - static final int MAXIMUM_CAPACITY = 1 << 30; - static final int MIN_SEGMENT_TABLE_CAPACITY = 2; - static final int MAX_SEGMENTS = 1 << 16; // slightly conservative - static final int RETRIES_BEFORE_LOCK = 2; - final int segmentMask; //段掩码,主要为了定位Segment - final int segmentShift; - final Segment[] segments; //主干就是这个分段锁数组 - - //构造器 - public ConcurrentHashMap(int initialCapacity, - float loadFactor, int concurrencyLevel) { - if (!(loadFactor > 0) || initialCapacity < 0 || concurrencyLevel <= 0) - throw new IllegalArgumentException(); - //MAX_SEGMENTS 为1<<16=65536，也就是最大并发数为65536 - if (concurrencyLevel > MAX_SEGMENTS) - concurrencyLevel = MAX_SEGMENTS; - // 2的sshif次方等于ssize，例:ssize=16,sshift=4;ssize=32,sshif=5 - int sshift = 0; - // ssize 为segments数组长度，根据concurrentLevel计算得出 - int ssize = 1; - while (ssize < concurrencyLevel) { - ++sshift; - ssize <<= 1; - } - this.segmentShift = 32 - sshift; - this.segmentMask = ssize - 1; - if (initialCapacity > MAXIMUM_CAPACITY) - initialCapacity = MAXIMUM_CAPACITY; - int c = initialCapacity / ssize; - if (c * ssize < initialCapacity) - ++c; - int cap = MIN_SEGMENT_TABLE_CAPACITY; - while (cap < c) - cap <<= 1; - // 创建segments数组并初始化第一个Segment，其余的Segment延迟初始化 - Segment s0 = - new Segment(loadFactor, (int)(cap * loadFactor), - (HashEntry[])new HashEntry[cap]); - Segment[] ss = (Segment[])new Segment[ssize]; - UNSAFE.putOrderedObject(ss, SBASE, s0); // ordered write of segments[0] - this.segments = ss; - } -} -``` - -#### put() 方法 - -1. **定位segment并确保定位的Segment已初始化 ** -2. **调用 Segment的 put 方法。** - -```java -public V put(K key, V value) { - Segment s; - //concurrentHashMap不允许key/value为空 - if (value == null) - throw new NullPointerException(); - //hash函数对key的hashCode重新散列，避免差劲的不合理的hashcode，保证散列均匀 - int hash = hash(key); - //返回的hash值无符号右移segmentShift位与段掩码进行位运算，定位segment - int j = (hash >>> segmentShift) & segmentMask; - if ((s = (Segment)UNSAFE.getObject // nonvolatile; recheck - (segments, (j << SSHIFT) + SBASE)) == null) // in ensureSegment - s = ensureSegment(j); - return s.put(key, hash, value, false); -} -``` - -#### get() 方法 - -**get方法无需加锁，由于其中涉及到的共享变量都使用volatile修饰，volatile可以保证内存可见性，所以不会读取到过期数据** - -```java -public V get(Object key) { - Segment s; // manually integrate access methods to reduce overhead - HashEntry[] tab; - int h = hash(key); - long u = (((h >>> segmentShift) & segmentMask) << SSHIFT) + SBASE; - if ((s = (Segment)UNSAFE.getObjectVolatile(segments, u)) != null && - (tab = s.table) != null) { - for (HashEntry e = (HashEntry) UNSAFE.getObjectVolatile - (tab, ((long)(((tab.length - 1) & h)) << TSHIFT) + TBASE); - e != null; e = e.next) { - K k; - if ((k = e.key) == key || (e.hash == h && key.equals(k))) - return e.value; - } - } - return null; -} -``` - -### JDK1.8 实现 - -![img](https://yfzhou.oss-cn-beijing.aliyuncs.com/blog/img/JDK1.8%20ConcurrentHashMap.jpg) - -ConcurrentHashMap 在 JDK8 中进行了巨大改动，光是代码量就从1000多行增加到6000行！1.8摒弃了`Segment`(锁段)的概念，采用了 `CAS + synchronized` 来保证并发的安全性。 - -可以看到，和 HashMap 1.8 的数据结构很像。底层数据结构改变为采用**数组+链表+红黑树**的数据形式。 - -#### 和HashMap1.8相同的一些地方 - -- 底层数据结构一致 -- HashMap初始化是在第一次put元素的时候进行的，而不是init -- HashMap的底层数组长度总是为2的整次幂 -- 默认树化的阈值为 8，而链表化的阈值为 6（当低于这个阈值时，红黑树转成链表） -- hash算法也很类似，但多了一步`& HASH_BITS`，该步是为了消除最高位上的负符号，hash的负在ConcurrentHashMap中有特殊意义表示在**扩容或者是树节点** - -```java -static final int HASH_BITS = 0x7fffffff; // usable bits of normal node hash - -static final int spread(int h) { - return (h ^ (h >>> 16)) & HASH_BITS; -} -``` - -#### 一些关键属性 - -```java -private static final int MAXIMUM_CAPACITY = 1 << 30; //数组最大大小 同HashMap - -private static final int DEFAULT_CAPACITY = 16;//数组默认大小 - -static final int MAX_ARRAY_SIZE = Integer.MAX_VALUE - 8; //数组可能最大值，需要与toArray（）相关方法关联 - -private static final int DEFAULT_CONCURRENCY_LEVEL = 16; //兼容旧版保留的值，默认线程并发度，类似信号量 - -private static final float LOAD_FACTOR = 0.75f;//默认map扩容比例，实际用(n << 1) - (n >>> 1)代替了更高效 - -static final int TREEIFY_THRESHOLD = 8; // 链表转树阀值，大于8时 - -static final int UNTREEIFY_THRESHOLD = 6; //树转链表阀值，小于等于6（tranfer时，lc、hc=0两个计数器分别++记录原bin、新binTreeNode数量，<=UNTREEIFY_THRESHOLD 则untreeify(lo)）。【仅在扩容tranfer时才可能树转链表】 - -static final int MIN_TREEIFY_CAPACITY = 64; - -private static final int MIN_TRANSFER_STRIDE = 16;//扩容转移时的最小数组分组大小 - -private static int RESIZE_STAMP_BITS = 16;//本类中没提供修改的方法 用来根据n生成位置一个类似时间戳的功能 - -private static final int MAX_RESIZERS = (1 << (32 - RESIZE_STAMP_BITS)) - 1; // 2^15-1，help resize的最大线程数 - -private static final int RESIZE_STAMP_SHIFT = 32 - RESIZE_STAMP_BITS; // 32-16=16，sizeCtl中记录size大小的偏移量 - -static final int MOVED = -1; // hash for forwarding nodes（forwarding nodes的hash值）、标示位 - -static final int TREEBIN = -2; // hash for roots of trees（树根节点的hash值） - -static final int RESERVED = -3; // ReservationNode的hash值 - -static final int HASH_BITS = 0x7fffffff; // 用在计算hash时进行安位与计算消除负hash - -static final int NCPU = Runtime.getRuntime().availableProcessors(); // 可用处理器数量 - -/* ---------------- Fields -------------- */ - -transient volatile Node[] table; //装载Node的数组，作为ConcurrentHashMap的数据容器，采用懒加载的方式，直到第一次插入数据的时候才会进行初始化操作，数组的大小总是为2的幂次方。 - -private transient volatile Node[] nextTable; //扩容时使用，平时为null，只有在扩容的时候才为非null - -/** -* 实际上保存的是hashmap中的元素个数 利用CAS锁进行更新但它并不用返回当前hashmap的元素个数 -*/ -private transient volatile long baseCount; - -/** -*该属性用来控制table数组的大小，根据是否初始化和是否正在扩容有几种情况： -*当值为负数时：如果为-1表示正在初始化，如果为-N则表示当前正有N-1个线程进行扩容操作； -*当值为正数时：如果当前数组为null的话表示table在初始化过程中，sizeCtl表示为需要新建数组的长度；若已经初始化了，表示当前数据容器（table数组）可用容量也可以理解成临界值（插入节点数超过了该临界值就需要扩容），具体指为数组的长度n 乘以 加载因子loadFactor；当值为0时，即数组长度为默认初始值。 -*/ -private transient volatile int sizeCtl; -``` - -#### put() 方法 - -1. 首先会判断 key、value是否为空，如果为空就抛异常！ -2. `spread()`方法获取hash，减小hash冲突 -3. 判断是否初始化table数组，没有的话调用`initTable()`方法进行初始化 -4. 判断是否能直接将新值插入到table数组中 -5. 判断当前是否在扩容，`MOVED`为-1说明当前ConcurrentHashMap正在进行扩容操作，正在扩容的话就进行协助扩容 -6. 当table[i]为链表的头结点，在链表中插入新值，通过synchronized (f)的方式进行加锁以实现线程安全性。 - 1. 在链表中如果找到了与待插入的键值对的key相同的节点，就直接覆盖 - 2. 如果没有找到的话，就直接将待插入的键值对追加到链表的末尾 -7. 当table[i]为红黑树的根节点，在红黑树中插入新值/覆盖旧值 -8. 根据当前节点个数进行调整，否需要转换成红黑树(个数大于等于8，就会调用`treeifyBin`方法将tabel[i]`第i个散列桶`拉链转换成红黑树) -9. 对当前容量大小进行检查，如果超过了临界值（实际大小*加载因子）就进行扩容 - -```java -final V putVal(K key, V value, boolean onlyIfAbsent) { - // key 和 value 均不允许为 null - if (key == null || value == null) throw new NullPointerException(); - // 根据 key 计算出 hash 值 - int hash = spread(key.hashCode()); - int binCount = 0; - for (Node[] tab = table;;) { - Node f; int n, i, fh; - // 判断是否需要进行初始化 - if (tab == null || (n = tab.length) == 0) - tab = initTable(); - // f 即为当前 key 定位出的 Node，如果为空表示当前位置可以写入数据，利用 CAS 尝试写入，失败则自旋保证成功 - else if ((f = tabAt(tab, i = (n - 1) & hash)) == null) { - if (casTabAt(tab, i, null, - new Node(hash, key, value, null))) - break; // no lock when adding to empty bin - } - // 如果当前位置的 hashcode == MOVED == -1,则需要进行扩容 - else if ((fh = f.hash) == MOVED) - tab = helpTransfer(tab, f); - // 如果都不满足，则利用 synchronized 锁写入数据 - else { - // 剩下情况又分两种，插入链表、插入红黑树 - V oldVal = null; - //采用同步内置锁实现并发控制 - synchronized (f) { - if (tabAt(tab, i) == f) { - // 如果 fh=f.hash >=0，当前为链表，在链表中插入新的键值对 - if (fh >= 0) { - binCount = 1; - //遍历链表，如果找到对应的 node 节点，修改 value,否则直接在链表尾部加入节点 - for (Node e = f;; ++binCount) { - K ek; - if (e.hash == hash && - ((ek = e.key) == key || - (ek != null && key.equals(ek)))) { - oldVal = e.val; - if (!onlyIfAbsent) - e.val = value; - break; - } - Node pred = e; - if ((e = e.next) == null) { - pred.next = new Node(hash, key, - value, null); - break; - } - } - } - // 当前为红黑树，将新的键值对插入到红黑树中 - else if (f instanceof TreeBin) { - Node p; - binCount = 2; - if ((p = ((TreeBin)f).putTreeVal(hash, key, - value)) != null) { - oldVal = p.val; - if (!onlyIfAbsent) - p.val = value; - } - } - } - } - // 插入完键值对后再根据实际大小看是否需要转换成红黑树 - if (binCount != 0) { - if (binCount >= TREEIFY_THRESHOLD) - treeifyBin(tab, i); - if (oldVal != null) - return oldVal; - break; - } - } - } - // 对当前容量大小进行检查，如果超过了临界值（实际大小*加载因子）就需要扩容 - addCount(1L, binCount); - return null; -} -``` - -我们可以发现 JDK8 中的实现也是**锁分离**的思想，只是锁住的是一个Node，而不是JDK7中的Segment，而锁住Node之前的操作是无锁的并且也是线程安全的，建立在之前提到的原子操作上。 - -#### get() 方法 - -get方法无需加锁，由于其中涉及到的共享变量都使用volatile修饰，volatile可以保证内存可见性，所以不会读取到过期数据 - -```java -public V get(Object key) { - Node[] tab; Node e, p; int n, eh; K ek; - int h = spread(key.hashCode()); - // 判断数组是否为空 - if ((tab = table) != null && (n = tab.length) > 0 && - (e = tabAt(tab, (n - 1) & h)) != null) { - // 判断node 节点第一个元素是不是要找的，如果是直接返回 - if ((eh = e.hash) == h) { - if ((ek = e.key) == key || (ek != null && key.equals(ek))) - return e.val; - } - // // hash小于0,说明是特殊节点（TreeBin或ForwardingNode）调用find - else if (eh < 0) - return (p = e.find(h, key)) != null ? p.val : null; - // 不是上面的情况,那就是链表了,遍历链表 - while ((e = e.next) != null) { - if (e.hash == h && - ((ek = e.key) == key || (ek != null && key.equals(ek)))) - return e.val; - } - } - return null; -} -``` - - - -### Hashtable 和 ConcurrentHashMap 的区别 - -ConcurrentHashMap 和 Hashtable 的区别主要体现在实现线程安全的方式上不同。 - -- **底层数据结构：** JDK1.7的 ConcurrentHashMap 底层采用 **分段的数组+链表** 实现，JDK1.8 采用的数据结构和 HashMap1.8 的结构类似，**数组+链表/红黑二叉树**。Hashtable 和 JDK1.8 之前的 HashMap 的底层数据结构类似都是采用 **数组+链表** 的形式，数组是 HashMap 的主体，链表则是主要为了解决哈希冲突而存在的； -- **实现线程安全的方式（重要）：** - - **在JDK1.7的时候，ConcurrentHashMap（分段锁）** 对整个桶数组进行了分割分段(Segment)，每一把锁只锁容器其中一部分数据，多线程访问容器里不同数据段的数据，就不会存在锁竞争，提高并发访问率。（默认分配16个Segment，比Hashtable效率提高16倍。） **到了 JDK1.8 的时候已经摒弃了Segment的概念，而是直接用 Node 数组+链表/红黑树的数据结构来实现，并发控制使用 synchronized 和 CAS 来操作。（JDK1.6以后 对 synchronized锁做了很多优化）** 整个看起来就像是优化过且线程安全的 HashMap，虽然在 JDK1.8 中还能看到 Segment 的数据结构，但是已经简化了属性，只是为了兼容旧版本； - - Hashtable(同一把锁) ：使用 synchronized 来保证线程安全，效率非常低下。当一个线程访问同步方法时，其他线程也访问同步方法，可能会进入阻塞或轮询状态，如使用 put 添加元素，另一个线程不能使用 put 添加元素，也不能使用 get，竞争越激烈效率越低。 - - - -## Java快速失败（fail-fast）和安全失败（fail-safe）区别 - -### 快速失败（fail—fast） - -在用迭代器遍历一个集合对象时，如果遍历过程中对集合对象的内容进行了修改（增加、删除、修改），则会抛出 `ConcurrentModificationException`。 - -原理：迭代器在遍历时直接访问集合中的内容，并且在遍历过程中使用一个 modCount 变量。集合在被遍历期间如果内容发生变化，就会改变 modCount 的值。每当迭代器使用 hashNext()/next() 遍历下一个元素之前，都会检测 modCount 变量是否为 expectedmodCount 值，是的话就返回遍历；否则抛出异常，终止遍历。 - -注意：这里异常的抛出条件是检测到 modCount！=expectedmodCount 这个条件。如果集合发生变化时修改modCount 值刚好又设置为了 expectedmodCount 值，则异常不会抛出。因此，不能依赖于这个异常是否抛出而进行并发操作的编程，这个异常只建议用于检测并发修改的bug。 - -场景：`java.util` 包下的集合类都是快速失败的，不能在多线程下发生并发修改（迭代过程中被修改）。 - -### 安全失败（fail—safe） - -采用安全失败机制的集合容器，在遍历时不是直接在集合内容上访问的，而是先复制原有集合内容，在拷贝的集合上进行遍历。 - -原理：由于迭代时是对原集合的拷贝进行遍历，所以在遍历过程中对原集合所作的修改并不能被迭代器检测到，所以不会触发 `Concurrent Modification Exception`。 - -缺点：基于拷贝内容的优点是避免了 `Concurrent Modification Exception`，但同样地，迭代器并不能访问到修改后的内容，即：迭代器遍历的是开始遍历那一刻拿到的集合拷贝，在遍历期间原集合发生的修改迭代器是不知道的。 - -场景：`java.util.concurrent` 包下的容器都是安全失败，可以在多线程下并发使用，并发修改。 - -快速失败和安全失败是对迭代器而言的。 - -快速失败：当在迭代一个集合的时候，如果有另外一个线程在修改这个集合，就会抛出`ConcurrentModification`异常，`java.util` 下都是快速失败。 - -安全失败：在迭代时候会在集合二层做一个拷贝，所以在修改集合上层元素不会影响下层。在 `java.util.concurrent` 下都是安全失败 - - - -### 如何避免**fail-fast** ？ - -- 在单线程的遍历过程中，如果要进行 remove 操作，可以调用迭代器 ListIterator 的 remove 方法而不是集合类的 remove方法。看看 ArrayList 中迭代器的 remove 方法的源码，该方法不能指定元素删除，只能 remove 当前遍历元素。 - -```java -public void remove() { - if (lastRet < 0) - throw new IllegalStateException(); - checkForComodification(); - - try { - SubList.this.remove(lastRet); - cursor = lastRet; - lastRet = -1; - expectedModCount = ArrayList.this.modCount; // - } catch (IndexOutOfBoundsException ex) { - throw new ConcurrentModificationException(); - } -} -``` - -- 使用并发包(`java.util.concurrent`)中的类来代替 ArrayList 和 hashMap - - CopyOnWriterArrayList 代替 ArrayList - - ConcurrentHashMap 代替 HashMap - - - -## Iterator 和 Enumeration 区别 - -在 Java 集合中，我们通常都通过 “Iterator(迭代器)” 或 “Enumeration(枚举类)” 去遍历集合。 - -```java -public interface Enumeration { - boolean hasMoreElements(); - E nextElement(); -} -``` - -```java -public interface Iterator { - boolean hasNext(); - E next(); - void remove(); -} -``` - -- **函数接口不同**，Enumeration**只有2个函数接口。**通过Enumeration，我们只能读取集合的数据，而不能对数据进行修改。Iterator**只有3个函数接口。**Iterator除了能读取集合的数据之外，也能数据进行删除操作。 -- **Iterator支持 fail-fast机制，而Enumeration不支持**。Enumeration 是 JDK 1.0 添加的接口。使用到它的函数包括 Vector、Hashtable 等类，这些类都是 JDK 1.0中加入的，Enumeration 存在的目的就是为它们提供遍历接口。Enumeration 本身并没有支持同步，而在 Vector、Hashtable 实现 Enumeration 时，添加了同步。 - 而 Iterator 是 JDK 1.2 才添加的接口，它也是为了 HashMap、ArrayList 等集合提供遍历接口。Iterator 是支持 fail-fast 机制的：当多个线程对同一个集合的内容进行操作时，就可能会产生 fail-fast 事件 - - - -## Comparable 和 Comparator 接口有何区别？ - -Java中对集合对象或者数组对象排序，有两种实现方式： - -- 对象实现Comparable 接口 - - - Comparable 在 `java.lang` 包下，是一个接口，内部只有一个方法 `compareTo()` - - ```java - public interface Comparable { - public int compareTo(T o); - } - ``` - - - Comparable 可以让实现它的类的对象进行比较，具体的比较规则是按照 compareTo 方法中的规则进行。这种顺序称为 **自然顺序**。 - - 实现了 Comparable 接口的 List 或则数组可以使用 `Collections.sort()` 或者 `Arrays.sort()` 方法进行排序 - -- 定义比较器，实现 Comparator接口 - - - Comparator 在 `java.util` 包下，也是一个接口，JDK 1.8 以前只有两个方法： - - ```java - public interface Comparator { - public int compare(T lhs, T rhs); - public boolean equals(Object object); - } - ``` - -**comparable相当于内部比较器。comparator相当于外部比较器** - -区别： - -- Comparator 位于 `java.util` 包下，而 Comparable 位于 `java.lang` 包下 - -- Comparable 接口的实现是在类的内部（如 String、Integer已经实现了 Comparable 接口，自己就可以完成比较大小操作），Comparator 接口的实现是在类的外部（可以理解为一个是自已完成比较，一个是外部程序实现比较） - -- 实现 Comparable 接口要重写 compareTo 方法, 在 compareTo 方法里面实现比较。一个已经实现Comparable 的类的对象或数据，可以通过 **Collections.sort(list) 或者 Arrays.sort(arr)**实现排序。通过 **Collections.sort(list,Collections.reverseOrder())** 对list进行倒序排列。 - - ![](https://tva1.sinaimg.cn/large/007S8ZIlly1gdzefrcjolj30ui0u0dm0.jpg) - -- 实现Comparator需要重写 compare 方法 - - ![](https://tva1.sinaimg.cn/large/007S8ZIlly1gdzefws30uj311s0t643v.jpg) - - - -## HashSet - -HashSet 是用来存储没有重复元素的集合类，并且它是无序的。HashSet 内部实现是基于 HashMap ，实现了 Set 接口。 - -从 HahSet 提供的构造器可以看出，除了最后一个 HashSet 的构造方法外，其他所有内部就是去创建一个 HashMap 。没有其他的操作。而最后一个构造方法不是 public 的，所以不对外公开。 - -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gdzeqxhvj3j311s0ka787.jpg) - - - -### HashSet如何检查重复 - -HashSet 的底层其实就是 HashMap，只不过我们 **HashSet 是实现了 Set 接口并且把数据作为 K 值，而 V 值一直使用一个相同的虚值来保存**，HashMap的 K 值本身就不允许重复，并且在 HashMap 中如果 K/V 相同时，会用新的 V 覆盖掉旧的 V，然后返回旧的 V。 - -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gdzeyvg5h0j311s0jen0n.jpg) - - - -### Iterater 和 ListIterator 之间有什么区别？ - -- 我们可以使用Iterator来遍历Set和List集合，而ListIterator只能遍历List -- ListIterator有add方法，可以向List中添加对象，而Iterator不能 -- ListIterator和Iterator都有hasNext()和next()方法，可以实现顺序向后遍历，但是ListIterator有hasPrevious()和previous()方法，可以实现逆向（顺序向前）遍历。Iterator不可以 -- ListIterator可以定位当前索引的位置，nextIndex()和previousIndex()可以实现。Iterator没有此功能 -- 都可实现删除操作，但是 ListIterator可以实现对象的修改，set()方法可以实现。Iterator仅能遍历，不能修改 - - - - - - - -## 参考与感谢 - -所有内容都是基于源码阅读和各种大佬之前总结的知识整理而来，输入并输出，奥利给。 - -https://www.javatpoint.com/java-arraylist - -https://www.runoob.com/java/java-collections.html - -https://www.javazhiyin.com/21717.html - -https://yuqirong.me/2018/01/31/LinkedList内部原理解析/ - -https://youzhixueyuan.com/the-underlying-structure-and-principle-of-hashmap.html - -《HashMap源码详细分析》http://www.tianxiaobo.com/2018/01/18/HashMap-源码详细分析-JDK1-8/ - -《ConcurrentHashMap1.7源码分析》https://www.cnblogs.com/chengxiao/p/6842045.html - -http://www.justdojava.com/2019/12/18/java-collection-15.1/ \ No newline at end of file diff --git a/docs/java/IO.md b/docs/java/IO.md new file mode 100755 index 0000000000..7ee387638e --- /dev/null +++ b/docs/java/IO.md @@ -0,0 +1,137 @@ +# 从 BIO、NIO、AIO 到 Netty 再到 Spring WebFlux：演进与应用 + +> IO模型说起，貌似工作中很少使用IO,更别提NIO，但实际上我们工作中每天都在和IO打交道。我们所用到的中间件redis，rocketMq，nacos，mse，dubbo等等存在文件操作，存在网络通信的地方就存在IO。所以深入了解IO模型重要性可想而知，IO操作作为计算机系统中一个基本操作，几乎所有应用程序都需要,了解不同IO模型可助我们理解应用程序中IO操作底层原理，从而更好地优化应用程序的性能和可靠性。 + + + +### IO 分类 + +1.BIO（Blocking I/O）：同步阻塞I/O，传统的I/O模型。在进行I/O操作时，必须等待数据读取或写入完成后才能进行下一步操作。 + +2.NIO（Non-blocking I/O）：同步非阻塞I/O，是一种事件驱动的I/O模型。在进行I/O操作时，不需要等待操作完成，可以进行其他操作。 + +3.AIO（Asynchronous I/O）：异步非阻塞I/O，是一种更高级别的I/O模型。在进行I/O操作时，不需要等待操作完成，就可继续进行其他操作，当操作完成后会自动回调通知 + + + +### 从 BIO、NIO、AIO 多路复用 到 Netty 再到 Spring WebFlux 的演进与应用 + +#### 引言 + +在现代高并发应用中，服务器的并发能力决定了系统的响应速度和处理性能。传统的阻塞式I/O (BIO) 模型已经无法满足高并发场景下的需求，随着技术的进步，NIO（非阻塞I/O）、AIO（异步I/O）等技术逐步发展并广泛应用。近年来，基于多路复用机制的高性能框架如 Netty 以及响应式编程框架 Spring WebFlux 的兴起，为现代 Web 开发提供了新的思路和技术手段。本文将全面梳理从 BIO、NIO 到 AIO，再到 Netty、Spring WebFlux 的技术演进，深入探讨其工作原理、应用场景和优缺点。 + +### 一、阻塞式I/O (BIO) + +#### 1.1 BIO 模型介绍 + +BIO (Blocking I/O) 是最基础的 I/O 模型，在 BIO 模型中，服务器与每个客户端之间的连接是通过阻塞的 socket 方式来进行的。每一个连接都会占用一个线程，服务器在接收到客户端请求时，会创建一个新的线程来处理该请求。如果有大量的客户端连接请求，服务器需要创建大量线程来处理，从而增加了线程上下文切换的开销。 + +#### 1.2 BIO 工作流程 + +1. 服务器端启动监听某个端口，等待客户端连接。 +2. 每个客户端连接后，服务器会为其分配一个线程专门处理这个连接上的 I/O 操作。 +3. 客户端发送请求，服务器处理并返回结果。 +4. 如果没有连接进来，线程会一直阻塞在 `accept()` 函数处。 + +#### 1.3 BIO 的缺点 + +BIO 模型虽然简单易用，但由于其阻塞的特性，使得它在高并发场景下效率低下，面临着以下几个问题： + +- **线程资源消耗大**：每个客户端连接都会占用一个线程，线程过多会导致上下文切换的开销增加，性能下降。 +- **并发能力有限**：当连接数剧增时，线程资源会被耗尽，从而导致服务器响应速度下降，甚至宕机。 +- **可扩展性差**：由于每个连接都需要占用一个独立的线程，因此无法轻易扩展到大规模并发的应用场景中。 + +### 二、非阻塞 I/O (NIO) + +#### 2.1 NIO 概念与模型 + +为了克服 BIO 模型在高并发场景下的不足，JDK 在 1.4 版本中引入了 NIO（Non-blocking I/O）模型。NIO 是一种非阻塞的 I/O 模型，它通过通道（Channel）和缓冲区（Buffer）来进行数据的读写，并结合多路复用器（Selector）来实现单线程处理多连接的机制，从而提升了系统的并发能力。 + +#### 2.2 NIO 的核心组件 + +1. **Channel**：通道类似于传统的流（Stream），但与流不同的是，通道是双向的，可以同时进行读和写操作。常见的通道包括 `FileChannel`、`SocketChannel`、`ServerSocketChannel` 等。 +2. **Buffer**：缓冲区是 NIO 中用于数据存储的容器。所有数据都必须通过缓冲区来进行读写。常见的缓冲区有 `ByteBuffer`、`CharBuffer` 等。 +3. **Selector**：多路复用器是 NIO 中实现单线程处理多个连接的关键组件。通过 Selector，应用程序可以同时监听多个通道上的事件，如连接就绪、读写就绪等，而不必为每个连接创建独立的线程。 + +#### 2.3 NIO 工作流程 + +1. 创建 `ServerSocketChannel` 并设置为非阻塞模式。 +2. 创建 `Selector`，并将 `ServerSocketChannel` 注册到 Selector 上。 +3. 通过 `select()` 方法轮询所有注册的通道，监听就绪的事件。 +4. 当有 I/O 事件就绪时，取出对应的通道进行处理。 + +#### 2.4 NIO 的优点 + +- **资源利用率高**：通过多路复用技术，一个线程可以处理多个连接，避免了 BIO 中每个连接占用一个线程的情况。 +- **非阻塞操作**：I/O 操作不会阻塞线程，线程可以处理其他任务，从而提高了吞吐量。 +- **高并发场景性能优越**：由于线程数量不再与连接数成正比，NIO 能够在高并发场景下保持较高的性能。 + +#### 2.5 NIO 的缺点 + +- **编程复杂度高**：NIO 需要开发者自己处理通道的读写、缓冲区的管理和事件的分发，代码相较于 BIO 复杂很多。 +- **性能瓶颈**：虽然 NIO 模型相比 BIO 提升了性能，但在某些情况下，轮询操作的效率可能不够高。 + +### 三、异步 I/O (AIO) + +#### 3.1 AIO 模型介绍 + +AIO (Asynchronous I/O) 模型，也称为 NIO.2，是在 JDK 7 中引入的一种新的 I/O 模型。与 NIO 的非阻塞操作不同，AIO 采用了真正的异步 I/O 操作。开发者无需手动处理 I/O 事件的轮询和回调，而是将 I/O 操作委托给操作系统，操作系统在 I/O 操作完成后通知应用程序。 + +#### 3.2 AIO 的工作机制 + +1. 应用程序发起一个异步 I/O 请求，并立即返回，继续处理其他任务。 +2. 操作系统后台处理 I/O 操作。 +3. I/O 操作完成后，操作系统通过回调函数通知应用程序。 + +#### 3.3 AIO 的优点 + +- **真正的异步操作**：开发者无需手动处理轮询，操作系统完成 I/O 操作后会自动通知应用程序。 +- **高效的资源利用**：AIO 能够充分利用系统资源，特别是在高并发、大量 I/O 操作的场景下。 + +#### 3.4 AIO 的缺点 + +- **平台依赖性**：AIO 的实现依赖于操作系统的底层异步 I/O 支持，某些操作系统可能对 AIO 的支持并不完善。 + +### 四、Netty + +#### 4.1 Netty 简介 + +Netty 是一个基于 NIO 的高性能网络框架，封装了 Java NIO 的复杂性，提供了更友好的 API 来处理网络编程，特别适用于高并发和大规模应用场景。Netty 支持多种协议（如 HTTP、WebSocket、TCP），并能够很好地处理复杂的网络通信需求。 + +#### 4.2 Netty 的核心组件 + +1. **Channel**：Netty 中的 `Channel` 表示一个网络连接，类似于 NIO 中的 `SocketChannel`。 +2. **EventLoop**：Netty 中的事件循环，负责处理通道中的 I/O 操作。每个 `EventLoop` 可以处理多个通道。 +3. **ChannelPipeline 和 ChannelHandler**：`ChannelPipeline` 是一个处理网络事件的责任链，而 `ChannelHandler` 则是负责处理实际的 I/O 事件（如读、写、连接等）。 + +#### 4.3 Netty 的优点 + +- **高性能**：Netty 底层使用 NIO，并对其进行了进一步的优化，在高并发场景下表现优越。 +- **API 简单**：Netty 屏蔽了底层复杂的 NIO 细节，提供了更高层次的 API，使得开发者能够专注于业务逻辑。 +- **多协议支持**：Netty 支持多种网络协议，具有良好的扩展性。 + +#### 4.4 Netty 的应用场景 + +- 分布式系统中的 RPC 通信。 +- 高并发 WebSocket 服务。 +- 游戏服务器、大数据传输等。 + +### 五、Spring WebFlux + +#### 5.1 Spring WebFlux 简介 + +Spring WebFlux 是 Spring 5 引入的一个响应式非阻塞 Web 框架，旨在解决传统 Spring MVC 在高并发场景下性能瓶颈问题。WebFlux 基于 Reactor 库实现，采用了响应式流（Reactive Streams）标准，并支持异步非阻塞的编程模型。 + +#### 5.2 Spring WebFlux 的特性 + +1. **非阻塞 I/O**：WebFlux 采用 Reactor 异步非阻塞框架，能够高效处理大量并发请求。 +2. **响应式编程**：通过 `Mono` 和 `Flux` 两种响应式类型，WebFlux 支持对流式数据进行处理。 +3. **容器支持**：WebFlux 支持多种异步容器，如 Netty、Undertow 和 Servlet 3.1+ 容器。 + + + + + + + +- [从Java BIO到NIO再到多路复用，看这篇就够了](https://mp.weixin.qq.com/s/VdyXDBevE48Wtr95ug_aKw) \ No newline at end of file diff --git a/docs/java/JUC/.DS_Store b/docs/java/JUC/.DS_Store new file mode 100644 index 0000000000..1af07b544d Binary files /dev/null and b/docs/java/JUC/.DS_Store differ diff --git a/docs/java/JUC/AQS.md b/docs/java/JUC/AQS.md index 5f6a76c3c7..f62a561598 100644 --- a/docs/java/JUC/AQS.md +++ b/docs/java/JUC/AQS.md @@ -1,375 +1,100 @@ -Java中的大部分同步类（Lock、Semaphore、ReentrantLock等）都是基于AbstractQueuedSynchronizer（简称为AQS）实现的。AQS是一种提供了原子式管理同步状态、阻塞和唤醒线程功能以及队列模型的简单框架。 +# 队列同步器 AQS 以及 Reentrantlock 应用 +Java 中的大部分同步类都是基于AbstractQueuedSynchronizer（简称为AQS）实现的。 +`ReentrantLock`、`ReentrantReadWriteLock`、`Semaphore(信号量)`、`CountDownLatch`、`公平锁`、`非公平锁`、 -> 文章来源 https://dayarch.top/p/java-aqs-and-reentrantlock.html +`ThreadPoolExecutor` 都和 AQS 有直接关系，所以了解 AQS 的抽象实现，并在此基础上结合上述各类的实现细节，很快就可以把 JUC 一网打尽，不至于查看源码时一头雾水，丢失主线。 - - -## 队列同步器 AQS - -队列同步器 （AbstractQueuedSynchronizer），简称同步器或AQS，就是我们今天的主人公 - -> **问：**为什么你分析 JUC 源码，要从 AQS 说起呢？ -> -> **答：**看下图 - -[![img](https://rgyb.sunluomeng.top/20200517201817.png)](https://rgyb.sunluomeng.top/20200517201817.png) - -相信看到这个截图你就明白一二了，你听过的，面试常被问起的，工作中常用的 - -- `ReentrantLock` -- `ReentrantReadWriteLock` -- `Semaphore(信号量)` -- `CountDownLatch` -- `公平锁` -- `非公平锁` -- `ThreadPoolExecutor` (关于线程池的理解，可以查看 [为什么要使用线程池?](https://dayarch.top/p/why-we-need-to-use-threadpool.html) ) - -都和 AQS 有直接关系，所以了解 AQS 的抽象实现，在此基础上再稍稍查看上述各类的实现细节，很快就可以全部搞定，不至于查看源码时一头雾水，丢失主线 - - +### 是什么 AQS 是 `AbstractQueuedSynchronizer` 的简称，翻译成中文就是 `抽象队列同步器` ，这三个单词分开来看： - Abstract （抽象）：也就是说， AQS 是一个抽象类，只实现一些主要的逻辑，有些方法推迟到子类实现 - Queued （队列）：队列有啥特征呢？先进先出（ FIFO ）对吧？也就是说， AQS 是用先进先出队列来存储数据的 -- Synchronizer （同步）：即 AQS 实现同步功能 +- Synchronizer （同步器）：即 AQS是 实现同步功能的 以上概括一下， AQS 是一个用来构建锁和同步器的框架，使用 AQS 能简单而又高效地构造出同步器。 -AQS 队列在内部维护了一个 FIFO 的双向链表，如果对数据结构比较熟的话，应该很容易就能想到，在双向链表中，每个节点都有两个指针，分别指向直接前驱节点和直接后继节点。使用双向链表的优点之一，就是从任意一个节点开始都很容易访问它的前驱节点和后继节点。 - -在 AQS 中，每个 Node 其实就是一个线程封装，当线程在竞争锁失败之后，会封装成 Node 加入到 AQS 队列中；获取锁的线程释放锁之后，会从队列中唤醒一个阻塞的 Node （也就是线程） - -AQS 使用 volatile 的变量 state 来作为资源的标识: - -``` -private volatile int state; -``` - -关于 state 状态的读取与修改，子类可以通过覆盖 getState() 和 setState() 方法来实现自己的逻辑，其中比较重要的是: - -``` -// 传入期望值 expect ,想要修改的值 update ,然后通过 Unsafe 的 compareAndSwapInt() 即 CAS 操作来实现 -protected final boolean compareAndSetState(int expect, int update) { - // See below for intrinsics setup to support this - return unsafe.compareAndSwapInt(this, stateOffset, expect, update); -} -``` - -下面是 AQS 中两个重要的成员变量: - -``` -private transient volatile Node head; // 头结点 -private transient volatile Node tail; // 尾节点 -``` - -关于 AQS 维护的双向链表，在源码中是这样解释的: - -``` -The wait queue is a variant of a "CLH" (Craig, Landin, and Hagersten) lock queue. -CLH locks are normally used for spinlocks. We instead use them for blocking synchronizers, -but use the same basic tactic of holding some of the control information -about a thread in the predecessor of its node. -``` - -也就是 AQS 的等待队列是 “CLH” 锁定队列的变体 - -直接来一张图会更形象一些： - -![img](https://mmbiz.qpic.cn/mmbiz_jpg/laEmibHFxFw5V4PsAed1hMNib2ich2E5tvJNaR0VJnMgq0Nq54lbAG0a3W5ctu8I0eWfgwtq0VHQmqt2qsdyv4Evg/640?wx_fmt=jpeg&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) - -Node 节点维护的是线程，控制线程的一些操作，具体来看看是 Node 是怎么做的： - -``` -static final class Node { - /** Marker to indicate a node is waiting in shared mode */ - // 标记一个节点,在 共享模式 下等待 - static final Node SHARED = new Node(); - - /** Marker to indicate a node is waiting in exclusive mode */ - // 标记一个节点,在 独占模式 下等待 - static final Node EXCLUSIVE = null; - - /** waitStatus value to indicate thread has cancelled */ - // waitStatus 的值,表示该节点从队列中取消 - static final int CANCELLED = 1; - - /** waitStatus value to indicate successor's thread needs unparking */ - // waitStatus 的值,表示后继节点在等待唤醒 - // 只有处于 signal 状态的节点,才能被唤醒 - static final int SIGNAL = -1; - - /** waitStatus value to indicate thread is waiting on condition */ - // waitStatus 的值,表示该节点在等待一些条件 - static final int CONDITION = -2; - - /** - * waitStatus value to indicate the next acquireShared should - * unconditionally propagate - */ - // waitStatus 的值,表示有资源可以使用,新 head 节点需要唤醒后继节点 - // 如果是在共享模式下,同步状态应该无条件传播下去 - static final int PROPAGATE = -3; - - // 节点状态,取值为 -3,-2,-1,0,1 - volatile int waitStatus; - - // 前驱节点 - volatile Node prev; - - // 后继节点 - volatile Node next; - - // 节点所对应的线程 - volatile Thread thread; - - // condition 队列中的后继节点 - Node nextWaiter; - - // 判断是否是共享模式 - final boolean isShared() { - return nextWaiter == SHARED; - } - - /** - * 返回前驱节点 - */ - final Node predecessor() throws NullPointerException { - Node p = prev; - if (p == null) - throw new NullPointerException(); - else - return p; - } - - Node() { // Used to establish initial head or SHARED marker - } - - /** - * 将线程构造成一个 Node 节点,然后添加到 condition 队列中 - */ - Node(Thread thread, Node mode) { // Used by addWaiter - this.nextWaiter = mode; - this.thread = thread; - } - - /** - * 等待队列用到的方法 - */ - Node(Thread thread, int waitStatus) { // Used by Condition - this.waitStatus = waitStatus; - this.thread = thread; - } -} -``` - -## AQS 如何获取资源 - -在 AQS 中，获取资源的入口是 acquire(int arg) 方法，其中 arg 是获取资源的个数，来看下代码： - -``` -public final void acquire(int arg) { - if (!tryAcquire(arg) && - acquireQueued(addWaiter(Node.EXCLUSIVE), arg)) - selfInterrupt(); -} -``` - -在获取资源时，会首先调用 tryAcquire 方法，这个方法是在子类中具体实现的 - -如果通过 tryAcquire 获取资源失败，接下来会通过 addWaiter(Node.EXCLUSIVE) 方法，将这个线程插入到等待队列中，具体代码: - -``` -private Node addWaiter(Node mode) { - // 生成该线程所对应的 Node 节点 - Node node = new Node(Thread.currentThread(), mode); - // 将 Node 插入到队列中 - Node pred = tail; - if (pred != null) { - node.prev = pred; - // 使用 CAS 操作,如果成功就返回 - if (compareAndSetTail(pred, node)) { - pred.next = node; - return node; - } - } - // 如果 pred == null 或者 CAS 操作失败,则调用 enq 方法再次自旋插入 - enq(node); - return node; -} - -// 自旋 CAS 插入等待队列 -private Node enq(final Node node) { - for (;;) { - Node t = tail; - if (t == null) { // Must initialize - if (compareAndSetHead(new Node())) - tail = head; - } else { - node.prev = t; - if (compareAndSetTail(t, node)) { - t.next = node; - return t; - } - } - } -} -``` - -在上面能够看到使用的是 CAS 自旋插入，这是因为在 AQS 中会存在多个线程同时竞争资源的情况，进而一定会出现多个线程同时插入节点的操作，这里使用 CAS 自旋插入是为了保证操作的线程安全性 - -现在呢，申请 acquire(int arg) 方法，然后通过调用 addWaiter 方法，将一个 Node 插入到了队列尾部。处于等待队列节点是从头结点开始一个一个的去获取资源，获取资源方式如下： - -``` -final boolean acquireQueued(final Node node, int arg) { - boolean failed = true; - try { - boolean interrupted = false; - for (;;) { - final Node p = node.predecessor(); - // 如果 Node 的前驱节点 p 是 head,说明 Node 是第二个节点,那么它就可以尝试获取资源 - if (p == head && tryAcquire(arg)) { - // 如果资源获取成功,则将 head 指向自己 - setHead(node); - p.next = null; // help GC - failed = false; - return interrupted; - } - // 节点进入等待队列后,调用 shouldParkAfterFailedAcquire 或者 parkAndCheckInterrupt 方法 - // 进入阻塞状态,即只有头结点的线程处于活跃状态 - if (shouldParkAfterFailedAcquire(p, node) && - parkAndCheckInterrupt()) - interrupted = true; - } - } finally { - if (failed) - cancelAcquire(node); - } -} -``` - -在获取资源时，除了 acquire 之外，还有三个方法： +程序员么，看原理类内容，不看源码“下饭”，就不叫深入学习，所以，适当打开下源码结合着看，很容易理解了。 -- acquireInterruptibly ：申请可中断的资源（独占模式） -- acquireShared ：申请共享模式的资源 -- acquireSharedInterruptibly ：申请可中断的资源（共享模式） -到这里，关于 AQS 如何获取资源就说的差不多了，接下来看看 AQS 是如何释放资源的 -## AQS 如何释放资源 +## 一、框架结构 -释放资源相对于获取资源来说，简单了很多。源码如下： +首先，我们通过下面的架构图来整体了解一下 AQS 框架 -``` -public final boolean release(int arg) { - // 如果释放锁成功 - if (tryRelease(arg)) { - // 获取 AQS 队列中的头结点 - Node h = head; - // 如果头结点不为空,且状态 != 0 - if (h != null && h.waitStatus != 0) - // 调用 unparkSuccessor(h) 方法,唤醒后续节点 - unparkSuccessor(h); - return true; - } - return false; -} +![美团技术团队](https://p1.meituan.net/travelcube/82077ccf14127a87b77cefd1ccf562d3253591.png) -private void unparkSuccessor(Node node) { - int ws = node.waitStatus; - // 如果状态是负数,尝试将它改为 0 - if (ws < 0) - compareAndSetWaitStatus(node, ws, 0); - // 得到头结点的后继节点 - Node s = node.next; - // 如果 waitStatus 大于 0 ,说明这个节点被取消 - if (s == null || s.waitStatus > 0) { - s = null; - // 那就从尾节点开始,找到距离 head 最近的一个 waitStatus<=0 的节点进行唤醒 - for (Node t = tail; t != null && t != node; t = t.prev) - if (t.waitStatus <= 0) - s = t; - } - // 如果后继节点不为空,则将其从阻塞状态变为非阻塞状态 - if (s != null) - LockSupport.unpark(s.thread); -} -``` +- 上图中有颜色的为 Method，无颜色的为 Attribution。 +- 总的来说，AQS 框架共分为五层，自上而下由浅入深，从 AQS 对外暴露的 API 到底层基础数据。 +- 当有自定义同步器接入时，只需重写第一层所需要的部分方法即可，不需要关注底层具体的实现流程。当自定义同步器进行加锁或者解锁操作时，先经过第一层的 API 进入 AQS 内部方法，然后经过第二层进行锁的获取，接着对于获取锁失败的流程，进入第三层和第四层的等待队列处理，而这些处理方式均依赖于第五层的基础数据提供层。 -## AQS 两种资源共享模式 -资源有两种共享模式: -- 独占模式( Exclusive )：资源是独占的，也就是一次只能被一个线程占有，比如 ReentrantLock -- 共享模式( Share )：同时可以被多个线程获取，具体的资源个数可以通过参数来确定，比如 Semaphore/CountDownLatch +### 第一层 +如果你现在打开 IDE， 你会发现我们经常使用的 `ReentrantLock` 、`ReentrantReadWriteLock`、 `Semaphore`、 `CountDownLatch` ，都是【聚合】了一个【队列同步器】的子类完成线程访问控制的，也就是我们说的第一层，API 层。 +为什么要聚合一个同步器的子类呢，这其实就是一个典型的模板方法模式的优点： +- 我们使用的锁是面向使用者的，它定义了使用者与锁交互的接口，隐藏了实现的细节，我们就像范式那样直接使用就可以了，很简单 +- 而同步器面向的是锁的实现，比如 Doug Lea 大神，或者我们业务自定义的同步器，它简化了锁的实现方式，屏蔽了同步状态管理，线程排队，等待/唤醒等底层操作 +这可以让我们使用起来更加方便，因为我们绝大多数都是在使用锁，实现锁之后，其核心就是要使用方便。 -上面提到，在锁的实现类中会聚合同步器，然后利同步器实现锁的语义，那么问题来了： -> 为什么要用聚合模式，怎么进一步理解锁和同步器的关系呢？ -[![img](https://rgyb.sunluomeng.top/20200530125122.png)](https://rgyb.sunluomeng.top/20200530125122.png) +同步器的设计是就基于模板方法模式的，如果需要自定义同步器一般的方式是这样（模板方法模式很经典的一个应用）： -我们绝大多数都是在使用锁，实现锁之后，其核心就是要使用方便 +1. 使用者继承 AbstractQueuedSynchronizer 并重写指定的方法。（这些重写方法很简单，无非是对于共享资源 state 的获取和释放） +2. 将 AQS 组合在自定义同步组件的实现中，并调用其模板方法，而这些模板方法又会调用使用者重写的方法。 -[![img](https://rgyb.sunluomeng.top/20200530130025.png)](https://rgyb.sunluomeng.top/20200530130025.png) +这和我们以往通过实现接口的方式有很大区别，这是模板方法模式很经典的一个运用，下面简单的给大家介绍一下模板方法模式，模板方法模式是一个很容易理解的设计模式之一。 -从 AQS 的类名称和修饰上来看，这是一个抽象类，所以从设计模式的角度来看同步器一定是基于【模版模式】来设计的，使用者需要继承同步器，实现自定义同步器，并重写指定方法，随后将同步器组合在自定义的同步组件中，并调用同步器的模版方法，而这些模版方法又回调用使用者重写的方法 +> 模板方法模式是基于”继承“的，主要是为了在不改变模板结构的前提下在子类中重新定义模板中的内容以实现复用代码。 -我不想将上面的解释说的这么抽象，其实想理解上面这句话，我们只需要知道下面两个问题就好了 +模板方法模式，都有两类方法，子类可重写的方法和模板类提供的模板方法，那 AQS 中肯定也有这两类方法，其实就是我们说的第一层 API 层中的所有方法，我们来看看 1. 哪些是自定义同步器可重写的方法？ 2. 哪些是抽象同步器提供的模版方法？ - - -### 同步器可重写的方法 +#### 同步器可重写的方法 同步器提供的可重写方法只有5个，这大大方便了锁的使用者： -![img](https://rgyb.sunluomeng.top/20200523160830.png) - -![image-20200710100815089](https://imgkr.cn-bj.ufileos.com/a1afaf46-5661-465b-a7b3-ab28e704c3d4.png) +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200929094818.png) 按理说，需要重写的方法也应该有 abstract 来修饰的，为什么这里没有？原因其实很简单，上面的方法我已经用颜色区分成了两类： -- `独占式` -- `共享式` +- `独占式`：一个时间点只能执行一个线程 +- `共享式`：一个时间点可多个线程同时执行 -自定义的同步组件或者锁不可能既是独占式又是共享式，为了避免强制重写不相干方法，所以就没有 abstract 来修饰了，但要抛出异常告知不能直接使用该方法： +表格方法描述中所说的`同步状态`就是上文提到的有 volatile 修饰的 state，所以我们在`重写`上面几个方法时，还可以通过同步器提供的下面三个方法（AQS 提供的）来获取或修改同步状态： -```java -protected boolean tryAcquire(int arg) { - throw new UnsupportedOperationException(); -} -``` +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200929094916.png) -暖暖的很贴心（如果你有类似的需求也可以仿照这样的设计） +而独占式和共享式操作 state 变量的区别也就很简单了，我们可以通过修改 State 字段表示的同步状态来实现多线程的独占模式和共享模式（加锁过程） -表格方法描述中所说的`同步状态`就是上文提到的有 volatile 修饰的 state，所以我们在`重写`上面几个方法时，还要通过同步器提供的下面三个方法（AQS 提供的）来获取或修改同步状态： +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200929094956.png) -[![img](https://rgyb.sunluomeng.top/20200523160906.png)](https://rgyb.sunluomeng.top/20200523160906.png) +稍微详细点步骤如下： -而独占式和共享式操作 state 变量的区别也就很简单了 +![](https://p0.meituan.net/travelcube/27605d483e8935da683a93be015713f331378.png) -[![img](https://rgyb.sunluomeng.top/20200523160705.png)](https://rgyb.sunluomeng.top/20200523160705.png) +![](https://p0.meituan.net/travelcube/3f1e1a44f5b7d77000ba4f9476189b2e32806.png) -所以你看到的 `ReentrantLock` `ReentrantReadWriteLock` `Semaphore(信号量)` `CountDownLatch` 这几个类其实仅仅是在实现以上几个方法上略有差别，其他的实现都是通过同步器的模版方法来实现的，到这里是不是心情放松了许多呢？我们来看一看模版方法： -### 同步器提供的模版方法 + +#### 同步器提供的模版方法 上面我们将同步器的实现方法分为独占式和共享式两类，模版方法其实除了提供以上两类模版方法之外，只是多了`响应中断`和`超时限制` 的模版方法供 Lock 使用，来看一下 -[![img](https://rgyb.sunluomeng.top/20200523195957.png)](https://rgyb.sunluomeng.top/20200523195957.png) +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200929095331.png) 先不用记上述方法的功能，目前你只需要了解个大概功能就好。另外，相信你也注意到了： @@ -377,21 +102,13 @@ protected boolean tryAcquire(int arg) { 看到这你也许有点乱了，我们稍微归纳一下： -[![img](https://rgyb.sunluomeng.top/20200523213113.png)](https://rgyb.sunluomeng.top/20200523213113.png) +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200929095609.png) 程序员还是看代码心里踏实一点，我们再来用代码说明一下上面的关系（注意代码中的注释，以下的代码并不是很严谨，只是为了简单说明上图的代码实现）： ```java -import java.util.concurrent.TimeUnit; -import java.util.concurrent.locks.AbstractQueuedSynchronizer; -import java.util.concurrent.locks.Condition; -import java.util.concurrent.locks.Lock; - /** * 自定义互斥锁 - * - * @author tanrgyb - * @date 2020/5/23 9:33 PM */ public class MyMutex implements Lock { @@ -436,7 +153,7 @@ public class MyMutex implements Lock { } } - // 聚合自定义同步器 + // 聚合自定义同步器 private final MySync sync = new MySync(); @@ -478,21 +195,33 @@ public class MyMutex implements Lock { } ``` -如果你现在打开 IDE， 你会发现上文提到的 `ReentrantLock` `ReentrantReadWriteLock` `Semaphore(信号量)` `CountDownLatch` 都是按照这个结构实现，所以我们就来看一看 AQS 的模版方法到底是怎么实现锁 +再打开 IDE， 看看 `ReentrantLock` `ReentrantReadWriteLock` `Semaphore(信号量)` `CountDownLatch` 的实现，会发现，他们都是按照这个结构实现的，是不感觉会了一个，剩下的几个常见的也差不多了。 + +接着我们就来看一看 AQS 的模版方法到底是怎么实现锁的 -## AQS实现分析 -从上面的代码中，你应该理解了`lock.tryLock()` 非阻塞式获取锁就是调用自定义同步器重写的 `tryAcquire()` 方法，通过 CAS 设置state 状态，不管成功与否都会马上返回；那么 lock.lock() 这种阻塞式的锁是如何实现的呢？ -有阻塞就需要排队，实现排队必然需要队列 +## 二、AQS实现分析 | 原理 -> CLH：Craig、Landin and Hagersten 队列，是一个单向链表，AQS中的队列是CLH变体的虚拟双向队列（FIFO）——概念了解就好，不要记 +AQS 核心思想是，如果被请求的共享资源空闲，那么就将当前请求资源的线程设置为有效的工作线程，将共享资源设置为锁定状态；如果共享资源被占用，就需要一定的阻塞等待唤醒机制来保证锁分配。这个机制主要用的是CLH 队列的变体实现的，将暂时获取不到锁的线程加入到队列中。 + +> CLH：Craig、Landin and Hagersten队列，是单向链表，AQS 中的队列是 CLH 变体的虚拟双向队列（FIFO），AQS 是通过将每条请求共享资源的线程封装成一个节点来实现锁的分配。 + +主要原理图如下： + +![](https://tva1.sinaimg.cn/large/007S8ZIlly1gj8k40jvfej30ix051aa8.jpg) + +AQS 使用一个 volatile 的 int 类型的成员变量来表示同步状态，通过内置的 FIFO 队列来完成资源获取的排队工作，通过 CAS 完成对 state 值的修改。 队列中每个排队的个体就是一个 Node，所以我们来看一下 Node 的结构 -### Node 节点 + + +### AQS数据结构 + +#### Node 节点 AQS 内部维护了一个同步队列，用于管理同步状态。 @@ -501,21 +230,51 @@ AQS 内部维护了一个同步队列，用于管理同步状态。 为了将上述步骤弄清楚，我们需要来看一看 Node 结构 （如果你能打开 IDE 一起看那是极好的） -[![img](https://rgyb.sunluomeng.top/20200524183916.png)](https://rgyb.sunluomeng.top/20200524183916.png) +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200929100137.png) + +解释一下几个方法和属性值的含义： + +| 方法和属性值 | 含义 | +| :----------- | :----------------------------------------------------------- | +| waitStatus | 当前节点在队列中的状态 | +| thread | 表示处于该节点的线程 | +| prev | 前驱指针 | +| predecessor | 返回前驱节点，没有的话抛出 NullPointerException | +| nextWaiter | 指向下一个处于 CONDITION 状态的节点（由于本篇文章不讲述Condition Queue队列，这个指针不多介绍） | +| next | 后继指针 | + +线程两种锁的模式： + +| 模式 | 含义 | +| :-------- | :----------------------------- | +| SHARED | 表示线程以共享的模式等待锁 | +| EXCLUSIVE | 表示线程正在以独占的方式等待锁 | + +waitStatus 有下面几个枚举值： + +| 枚举 | 含义 | +| :-------- | :------------------------------------------------ | +| 0 | 当一个 Node 被初始化的时候的默认值 | +| CANCELLED | 为 1，表示线程获取锁的请求已经取消了 | +| CONDITION | 为 -2，表示节点在等待队列中，节点线程等待唤醒 | +| PROPAGATE | 为 -3，当前线程处在 SHARED 情况下，该字段才会使用 | +| SIGNAL | 为 -1，表示线程已经准备好了，就等资源释放了 | 乍一看有点杂乱，我们还是将其归类说明一下： -[![img](https://rgyb.sunluomeng.top/20200524184014.png)](https://rgyb.sunluomeng.top/20200524184014.png) +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200929100553.png) 上面这几个状态说明有个印象就好，有了Node 的结构说明铺垫，你也就能想象同步队列的基本结构了： -![img](https://rgyb.sunluomeng.top/20200525072245.png) +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200929100626.png) + +一般来说，自定义同步器要么是独占方式，要么是共享方式，它们也只需实现 `tryAcquire-tryRelease`、`tryAcquireShared-tryReleaseShared` 中的一种即可。AQS 也支持自定义同步器同时实现独占和共享两种方式，如 ReentrantReadWriteLock。ReentrantLock 是独占锁，所以实现了 tryAcquire-tryRelease。 前置知识基本铺垫完毕，我们来看一看独占式获取同步状态的整个过程 ### 独占式获取同步状态 -故事要从范式 `lock.lock()` 开始 +故事要从范式 `lock.lock()` 开始，，或者可以结合着 ReentrantLock 来看，也可以（先不要在意公平锁和非公平锁，他们在底层是相同的） ```java public void lock() { @@ -535,7 +294,7 @@ public final void acquire(int arg) { } ``` -首先，也会尝试非阻塞的获取同步状态，如果获取失败（tryAcquire返回false），则会调用 `addWaiter` 方法构造 Node 节点（Node.EXCLUSIVE 独占式）并安全的（CAS）加入到同步队列【尾部】 +首先，会通过 API 层实现的 tryAcquire() 方法，尝试非阻塞的获取同步状态，如果该方法返回了 True，则说明当前线程获取锁成功，如果获取失败（tryAcquire 返回 false），则会调用 `addWaiter` 方法构造 Node 节点（Node.EXCLUSIVE 独占式）并安全的（CAS）加入到同步队列【尾部】 ```java private Node addWaiter(Node mode) { @@ -563,6 +322,7 @@ private Node addWaiter(Node mode) { return node; } +//如果Pred指针是Null（说明等待队列中没有元素），或者当前Pred指针和Tail指向的位置不同（说明被别的线程已经修改），就需要看一下Enq的方法 private Node enq(final Node node) { // 通过“死循环”确保节点被正确添加，最终将其设置为尾节点之后才会返回，这里使用 CAS 的理由和上面一样 for (;;) { @@ -587,27 +347,37 @@ private Node enq(final Node node) { } ``` -你可能比较迷惑 enq() 的处理方式，进入该方法就是一个“死循环”，我们就用图来描述它是怎样跳出循环的 +如果没有被初始化，需要进行初始化一个头结点出来（注释中的哨兵结点）。但请注意，初始化的头结点并不是当前线程节点，而是调用了无参构造函数的节点。如果经历了初始化或者并发导致队列中有元素，则与之前的方法相同。其实，addWaiter 就是一个在双端链表添加尾节点的操作，需要注意的是，双端链表的头结点是一个无参构造函数的头结点。 + +总结一下，线程获取锁的时候，过程大体如下： + +1. 当没有线程获取到锁时，线程 1 获取锁成功。 +2. 线程 2 申请锁，但是锁被线程 1 占有。 +3. 如果再有线程要获取锁，依次在队列中往后排队即可。 -![img](https://rgyb.sunluomeng.top/20200530135150.png) +![img](https://p0.meituan.net/travelcube/e9e385c3c68f62c67c8d62ab0adb613921117.png) -有些同学可能会有疑问，为什么会有哨兵节点？ +上边解释了 addWaiter 方法，这个方法其实就是把对应的线程以 Node 的数据结构形式加入到双端队列里，返回的是一个包含该线程的 Node。而这个 Node 会作为参数，进入到 acquireQueued 方法中。acquireQueued 方法可以对排队中的线程进行“获锁”操作。 -> 哨兵，顾名思义，是用来解决国家之间边界问题的，不直接参与生产活动。同样，计算机科学中提到的哨兵，也用来解决边界问题，如果没有边界，指定环节，按照同样算法可能会在边界处发生异常，比如要继续向下分析的 `acquireQueued()` 方法 +总的来说，一个线程获取锁失败了，被放入等待队列，acquireQueued 会把放入队列中的线程不断去获取锁，直到获取成功或者不再需要获取（中断）。 + +下面我们从“何时出队列？”和“如何出队列？”两个方向来分析一下 acquireQueued 源码： ```java final boolean acquireQueued(final Node node, int arg) { + // 标记是否成功拿到资源 boolean failed = true; try { + // 标记等待过程中是否中断过 boolean interrupted = false; - // "死循环"，尝试获取锁，或者挂起 + // "死循环"，自旋，要么获取锁，要么中断 for (;;) { // 获取当前节点的前驱节点 final Node p = node.predecessor(); - // 只有当前节点的前驱节点是头节点，才会尝试获取锁 - // 看到这你应该理解添加哨兵节点的含义了吧 + // 如果p是头结点，说明当前节点在真实数据队列的首部，就尝试获取锁（别忘了头结点是虚节点） + // 这就是为什么有个空的头结点 if (p == head && tryAcquire(arg)) { - // 获取同步状态成功，将自己设置为头 + // 获取锁成功，头指针移动到当前node setHead(node); // 将哨兵节点的后继节点置为空，方便GC p.next = null; // help GC @@ -617,6 +387,7 @@ final boolean acquireQueued(final Node node, int arg) { } // 当前节点的前驱节点不是头节点 //【或者】当前节点的前驱节点是头节点但获取同步状态失败 + // 说明p为头节点且当前没有获取到锁（可能是非公平锁被抢占了）或者是p不为头结点，这个时候就要判断当前node是否要被阻塞（被阻塞条件：前驱节点的waitStatus为-1），防止无限循环浪费资源。具体两个方法下面细细分析 if (shouldParkAfterFailedAcquire(p, node) && parkAndCheckInterrupt()) interrupted = true; @@ -628,26 +399,42 @@ final boolean acquireQueued(final Node node, int arg) { } ``` +注：setHead 方法是把当前节点置为虚节点，但并没有修改 waitStatus，因为它是一直需要用的数据。 + +```java +private void setHead(Node node) { + head = node; + node.thread = null; + node.prev = null; +} +``` + 获取同步状态成功会返回可以理解了，但是如果失败就会一直陷入到“死循环”中浪费资源吗？很显然不是，`shouldParkAfterFailedAcquire(p, node)` 和 `parkAndCheckInterrupt()` 就会将线程获取同步状态失败的线程挂起，我们继续向下看 ```java +// 靠前驱节点判断当前线程是否应该被阻塞 private static boolean shouldParkAfterFailedAcquire(Node pred, Node node) { // 获取前驱节点的状态 int ws = pred.waitStatus; // 如果是 SIGNAL 状态，即等待被占用的资源释放，直接返回 true // 准备继续调用 parkAndCheckInterrupt 方法 + // 说明头结点处于唤醒状态 if (ws == Node.SIGNAL) return true; - // ws 大于0说明是CANCELLED状态， + // ws 大于0说明是CANCELLED状态，取消状态 + /* + * 如果前驱放弃了，那就一直往前找，直到找到最近一个正常等待的状态，并排在它的后边。 + * 注意：那些放弃的结点，由于被自己“加塞”到它们前边，它们相当于形成一个无引用链，稍后就会被保安大叔赶走了(GC回收)！ + */ if (ws > 0) { // 循环判断前驱节点的前驱节点是否也为CANCELLED状态，忽略该状态的节点，重新连接队列 do { + // 循环向前查找取消节点，把取消节点从队列中剔除 node.prev = pred = pred.prev; } while (pred.waitStatus > 0); pred.next = node; } else { // 将当前节点的前驱节点设置为设置为 SIGNAL 状态，用于后续唤醒操作 - // 程序第一次执行到这返回为false，还会进行外层第二次循环，最终从代码第7行返回 compareAndSetWaitStatus(pred, ws, Node.SIGNAL); } return false; @@ -660,11 +447,11 @@ private static boolean shouldParkAfterFailedAcquire(Node pred, Node node) { 保留这个问题，我们陆续揭晓 -如果前驱节点的 waitStatus 是 SIGNAL状态，即 shouldParkAfterFailedAcquire 方法会返回 true ，程序会继续向下执行 `parkAndCheckInterrupt` 方法，用于将当前线程挂起 +如果前驱节点的 waitStatus 是 SIGNAL 状态，即 `shouldParkAfterFailedAcquire` 方法会返回 true ，程序会继续向下执行 `parkAndCheckInterrupt` 方法，parkAndCheckInterrupt 主要用于挂起当前线程，阻塞调用栈，返回当前线程的中断状态 ```java private final boolean parkAndCheckInterrupt() { - // 线程挂起，程序不会继续向下执行 + // 线程挂起，程序不会继续向下执行 调用park()使线程进入waiting状态 LockSupport.park(this); // 根据 park 方法 API描述，程序在下述三种情况会继续向下执行 // 1. 被 unpark @@ -677,6 +464,16 @@ private final boolean parkAndCheckInterrupt() { } ``` +上述方法的流程图如下： + +![](https://p0.meituan.net/travelcube/c124b76dcbefb9bdc778458064703d1135485.png) + +从上图可以看出，跳出当前循环的条件是当“前置节点是头结点，且当前线程获取锁成功”。为了防止因死循环导致CPU 资源被浪费，我们会判断前置节点的状态来决定是否要将当前线程挂起，具体挂起流程用流程图表示如下（shouldParkAfterFailedAcquire 流程）： + +![](https://p0.meituan.net/travelcube/9af16e2481ad85f38ca322a225ae737535740.png) + + + 被唤醒的程序会继续执行 `acquireQueued` 方法里的循环，如果获取同步状态成功，则会返回 `interrupted = true` 的结果 程序继续向调用栈上层返回，最终回到 AQS 的模版方法 `acquire` @@ -699,9 +496,7 @@ static void selfInterrupt() { } ``` -[![img](https://rgyb.sunluomeng.top/20200530171736.png)](https://rgyb.sunluomeng.top/20200530171736.png) - -如果你不能理解中断，强烈建议你回看 [Java多线程中断机制](https://dayarch.top/p/java-concurrency-interrupt-mechnism.html) +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200929102755.png) 到这里关于获取同步状态我们还遗漏了一条线，acquireQueued 的 finally 代码块如果你仔细看你也许马上就会有疑惑: @@ -712,7 +507,7 @@ if (failed) cancelAcquire(node); ``` -这段代码被执行的条件是 failed 为 true，正常情况下，如果跳出循环，failed 的值为false，如果不能跳出循环貌似怎么也不能执行到这里，所以只有不正常的情况才会执行到这里，也就是会发生异常，才会执行到此处 +这段代码被执行的条件是 failed 为 true，正常情况下，如果跳出循环，failed 的值为 false，如果不能跳出循环貌似怎么也不能执行到这里，所以只有不正常的情况才会执行到这里，也就是会发生异常，才会执行到此处 查看 try 代码块，只有两个方法会抛出异常： @@ -721,16 +516,41 @@ if (failed) 先看前者： -[![img](https://rgyb.sunluomeng.top/20200525201815.png)](https://rgyb.sunluomeng.top/20200525201815.png) +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200929102933.png) 很显然，这里抛出的异常不是重点，那就以 ReentrantLock 重写的 tryAcquire() 方法为例 -[![img](https://rgyb.sunluomeng.top/20200525202215.png)](https://rgyb.sunluomeng.top/20200525202215.png) +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200929103016.png) 另外，上面分析 `shouldParkAfterFailedAcquire` 方法还对 CANCELLED 的状态进行了判断，那么 > 什么时候会生成取消状态的节点呢？ +acquireQueued 方法中的 finally 代码： + +```java +// java.util.concurrent.locks.AbstractQueuedSynchronizer + +final boolean acquireQueued(final Node node, int arg) { + boolean failed = true; //标记是否成功拿到资源 + try { + ... + for (;;) { + final Node p = node.predecessor(); //拿到前驱 + //如果前驱是head，即该结点已成老二，那么便有资格去尝试获取资源（可能是老大释放完资源唤醒自己的，当然也可能被interrupt了）。 + if (p == head && tryAcquire(arg)) { + ... + failed = false; + ... + } + ... + } finally { + if (failed) + cancelAcquire(node); + } +} +``` + 答案就在 `cancelAcquire` 方法中， 我们来看看 cancelAcquire到底怎么设置/处理 CANNELLED 的 ```java @@ -738,7 +558,7 @@ private void cancelAcquire(Node node) { // 忽略无效节点 if (node == null) return; - // 将关联的线程信息清空 + // 将关联的线程信息清空 node.thread = null; // 跳过同样是取消状态的前驱节点 @@ -753,6 +573,7 @@ private void cancelAcquire(Node node) { node.waitStatus = Node.CANCELLED; // 如果当前节点处在尾节点，直接从队列中删除自己就好 + // 更新失败的话，则进入else，如果更新成功，将tail的后继节点设置为null if (node == tail && compareAndSetTail(node, pred)) { compareAndSetNext(pred, predNext, null); } else { @@ -779,21 +600,46 @@ private void cancelAcquire(Node node) { } ``` -看到这个注释你可能有些乱了，其核心目的就是从等待队列中移除 CANCELLED 的节点，并重新拼接整个队列，总结来看，其实设置 CANCELLED 状态节点只是有三种情况，我们通过画图来分析一下： +看到这个注释你可能有些乱了，其核心目的就是从等待队列中移除 CANCELLED 的节点，并重新拼接整个队列， + +当前的流程： + +- 获取当前节点的前驱节点，如果前驱节点的状态是CANCELLED，那就一直往前遍历，找到第一个waitStatus <= 0的节点，将找到的Pred节点和当前Node关联，将当前Node设置为CANCELLED。 +- 根据当前节点的位置，考虑以下三种情况： + + (1) 当前节点是尾节点。 + + (2) 当前节点是 Head 的后继节点。 + + (3) 当前节点不是 Head 的后继节点，也不是尾节点。 + +根据上述第二条，我们来分析每一种情况的流程。 + +当前节点是尾节点。 + +![img](https://p1.meituan.net/travelcube/b845211ced57561c24f79d56194949e822049.png) -[![img](https://rgyb.sunluomeng.top/20200527104935.png)](https://rgyb.sunluomeng.top/20200527104935.png) +当前节点是 Head 的后继节点。 ------- +![](https://p1.meituan.net/travelcube/ab89bfec875846e5028a4f8fead32b7117975.png) -[![img](https://rgyb.sunluomeng.top/20200527105017.png)](https://rgyb.sunluomeng.top/20200527105017.png) +当前节点不是 Head 的后继节点，也不是尾节点。 ------- +![](https://p0.meituan.net/travelcube/45d0d9e4a6897eddadc4397cf53d6cd522452.png) -[![img](https://rgyb.sunluomeng.top/20200527105040.png)](https://rgyb.sunluomeng.top/20200527105040.png) +通过上面的流程，我们对于 CANCELLED 节点状态的产生和变化已经有了大致的了解，但是为什么所有的变化都是对 Next 指针进行了操作，而没有对 Prev 指针进行操作呢？什么情况下会对 Prev 指针进行操作？ + +> 执行cancelAcquire的时候，当前节点的前置节点可能已经从队列中出去了（已经执行过Try代码块中的shouldParkAfterFailedAcquire方法了），如果此时修改Prev指针，有可能会导致Prev指向另一个已经移除队列的Node，因此这块变化Prev指针不安全。 shouldParkAfterFailedAcquire方法中，会执行下面的代码，其实就是在处理Prev指针。shouldParkAfterFailedAcquire是获取锁失败的情况下才会执行，进入该方法后，说明共享资源已被获取，当前节点之前的节点都不会出现变化，因此这个时候变更Prev指针比较安全。 +> +> ``` +> do { +> node.prev = pred = pred.prev; +> } while (pred.waitStatus > 0); +> ``` 至此，获取同步状态的过程就结束了，我们简单的用流程图说明一下整个过程 -[![img](https://rgyb.sunluomeng.top/20200527112235.png)](https://rgyb.sunluomeng.top/20200527112235.png) +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200929103201.png) 获取锁的过程就这样的结束了，先暂停几分钟整理一下自己的思路。我们上面还没有说明 SIGNAL 的作用， SIGNAL 状态信号到底是干什么用的？这就涉及到锁的释放了，我们来继续了解，整体思路和锁的获取是一样的， 但是释放过程就相对简单很多了 @@ -886,11 +732,11 @@ node.prev = pred; compareAndSetTail(pred, node) ``` -这两个地方可以看作是尾节点入队的原子操作，如果此时代码还没执行到 pred.next = node; 这时又恰巧执行了unparkSuccessor方法，就没办法从前往后找了，因为后继指针还没有连接起来，所以需要从后往前找 +这两个地方可以看作是尾节点入队的原子操作，如果此时代码还没执行到 pred.next = node; 这时又恰巧执行了 unparkSuccessor 方法，就没办法从前往后找了，因为后继指针还没有连接起来，所以需要从后往前找 -第二点原因，在上面图解产生 CANCELLED 状态节点的时候，先断开的是 Next 指针，Prev指针并未断开，因此这也是必须要从后往前遍历才能够遍历完全部的Node +第二点原因，在上面图解产生 CANCELLED 状态节点的时候，先断开的是 Next 指针，Prev 指针并未断开，因此这也是必须要从后往前遍历才能够遍历完全部的 Node -同步状态至此就已经成功释放了，之前获取同步状态被挂起的线程就会被唤醒，继续从下面代码第 3 行返回执行： +同步状态至此就已经成功释放了，之前获取同步状态被挂起的线程就会被唤醒，继续从下面代码第 3 行返回执行： ```java private final boolean parkAndCheckInterrupt() { @@ -899,7 +745,7 @@ private final boolean parkAndCheckInterrupt() { } ``` -继续返回上层调用栈, 从下面代码15行开始执行，重新执行循环，再次尝试获取同步状态 +继续返回上层调用栈，从下面代码 15 行开始执行，重新执行循环，再次尝试获取同步状态 ```java final boolean acquireQueued(final Node node, int arg) { @@ -930,11 +776,9 @@ final boolean acquireQueued(final Node node, int arg) { - `响应中断` - `超时限制` -[![img](https://rgyb.sunluomeng.top/20200530195432.png)](https://rgyb.sunluomeng.top/20200530195432.png) - ### 独占式响应中断获取同步状态 -故事要从lock.lockInterruptibly() 方法说起 +故事要从 `lock.lockInterruptibly()` 方法说起 ```java public void lockInterruptibly() throws InterruptedException { @@ -1076,19 +920,21 @@ static final long spinForTimeoutThreshold = 1000L; 到这里，我们自定义的 MyMutex 只差 Condition 没有说明了，不知道你累了吗？我还在坚持 -[![img](https://rgyb.sunluomeng.top/20200530195521.png)](https://rgyb.sunluomeng.top/20200530195521.png) + ### Condition -如果你看过之前写的 [并发编程之等待通知机制](https://dayarch.top/p/waiting-notification-mechanism.html) ，你应该对下面这个图是有印象的： +上面已经介绍了`AQS`所提供的核心功能，当然它还有很多其他的特性，这里我们来继续说下`Condition`这个组件。Condition 是在 Java 1.5 中才出现的，它用来替代传统的 `Object` 的 `wait()`、`notify()` 实现线程间的协作，相比使用 `Object` 的 `wait()`、`notify()`，使用 `Condition` 中的 `await()`、`signal() `这种方式实现线程间协作更加安全和高效。因此通常来说比较推荐使用 Condition -[![img](https://cdn.jsdelivr.net/gh/FraserYu/img-host/blog-img20200315110223.png)](https://cdn.jsdelivr.net/gh/FraserYu/img-host/blog-img20200315110223.png) +其中 `AbstractQueueSynchronizer` 中实现了 `Condition` 中的方法，主要对外提供 `awaite(Object.wait())` 和 `signal(Object.notify()) `调用。 + +![](https://cdn.jsdelivr.net/gh/FraserYu/img-host/blog-img20200315110223.png) 如果当时你理解了这个模型，再看 Condition 的实现，根本就不是问题了，首先 Condition 还是一个接口，肯定也是需要有实现类的 -[![img](https://rgyb.sunluomeng.top/20200530200503.png)](https://rgyb.sunluomeng.top/20200530200503.png) +![](https://tva1.sinaimg.cn/large/007S8ZIlly1gj8k4jiqe3j30g80d6wfm.jpg) -那故事就从 `lock.newnewCondition` 说起吧 +那故事就从 `lock.newCondition` 说起吧 ```java public Condition newCondition() { @@ -1164,13 +1010,13 @@ private Node addConditionWaiter() { 这里有朋友可能会有疑问： -> 为什么这里是单向队列，也没有使用CAS 来保证加入队列的安全性呢？ +> 为什么这里是单向队列，也没有使用 CAS 来保证加入队列的安全性呢？ 因为 await 是 Lock 范式 try 中使用的，说明已经获取到锁了，所以就没必要使用 CAS 了，至于是单向，因为这里还不涉及到竞争锁，只是做一个条件等待队列 在 Lock 中可以定义多个条件，每个条件都会对应一个 条件等待队列，所以将上图丰富说明一下就变成了这个样子： -[![img](https://rgyb.sunluomeng.top/20200530205315.png)](https://rgyb.sunluomeng.top/20200530205315.png) +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200929114003.png) 线程已经按相应的条件加入到了条件等待队列中，那如何再尝试获取锁呢？signal / signalAll 方法就已经排上用场了 @@ -1217,7 +1063,7 @@ final boolean transferForSignal(Node node) { 所以我们再用图解一下唤醒的整个过程 -[![img](https://rgyb.sunluomeng.top/20200530210706.png)](https://rgyb.sunluomeng.top/20200530210706.png) +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200929114041.png) 到这里，理解 signalAll 就非常简单了，只不过循环判断是否还有 nextWaiter，如果有就像 signal 操作一样，将其从条件等待队列中移到同步队列中 @@ -1239,23 +1085,71 @@ private void doSignalAll(Node first) { 什么时候可以用 signal 方法也在其中做了说明，请大家自行查看吧 -[![img](https://rgyb.sunluomeng.top/20200530211202.png)](https://rgyb.sunluomeng.top/20200530211202.png) + + +### LockSupport + +从上面我可以看到，当需要阻塞或者唤醒一个线程的时候，AQS都是使用LockSupport这个工具类来完成的。 + +> LockSupport是用来创建锁和其他同步类的基本线程阻塞原语 + +每个使用LockSupport的线程都会与一个许可关联，如果该许可可用，并且可在进程中使用，则调用park()将会立即返回，否则可能阻塞。如果许可尚不可用，则可以调用 unpark 使其可用。但是注意许可不可重入，也就是说只能调用一次park()方法，否则会一直阻塞。 + +LockSupport定义了一系列以park开头的方法来阻塞当前线程，unpark(Thread thread)方法来唤醒一个被阻塞的线程。如下： + +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200929140316.jpg) + +park(Object blocker) 方法的 blocker 参数，主要是用来标识当前线程在等待的对象，该对象主要用于问题排查和系统监控。 + +park 方法和 unpark(Thread thread)都是成对出现的，同时 unpark 必须要在 park 执行之后执行，当然并不是说没有不调用 unpark 线程就会一直阻塞，park 有一个方法，它带了时间戳（parkNanos(long nanos)：为了线程调度禁用当前线程，最多等待指定的等待时间，除非许可可用）。 + +park() 方法的源码如下： + +```java +public static void park() { + UNSAFE.park(false, 0L); +} +``` + +unpark(Thread thread)方法源码如下： + +```java +public static void unpark(Thread thread) { + if (thread != null) + UNSAFE.unpark(thread); +} +``` + +从上面可以看出，其内部的实现都是通过UNSAFE（sun.misc.Unsafe UNSAFE）来实现的，其定义如下： + +```java +public native void park(boolean var1, long var2); +public native void unpark(Object var1); +``` + +两个都是native本地方法。Unsafe 是一个比较危险的类，主要是用于执行低级别、不安全的方法集合。尽管这个类和所有的方法都是公开的（public），但是这个类的使用仍然受限，你无法在自己的java程序中直接使用该类，因为只有授信的代码才能获得该类的实例。 + + + + + + 这里我还要多说一个细节，从条件等待队列移到同步队列是有时间差的，所以使用 await() 方法也是范式的， 同样在该文章中做了解释 -[![img](https://cdn.jsdelivr.net/gh/FraserYu/img-host/blog-img20200312154011.png)](https://cdn.jsdelivr.net/gh/FraserYu/img-host/blog-img20200312154011.png) +![](https://cdn.jsdelivr.net/gh/FraserYu/img-host/blog-img20200312154011.png) -有时间差，就会有公平和不公平的问题，想要全面了解这个问题，我们就要走近 ReentrantLock 中来看了，除了了解公平/不公平问题，查看 ReentrantLock 的应用还是要反过来验证它使用的AQS的，我们继续吧 +有时间差，就会有公平和不公平的问题，想要全面了解这个问题，我们就要走近 ReentrantLock 中来看了，除了了解公平/不公平问题，查看 ReentrantLock 的应用还是要反过来验证它使用的 AQS 的，我们继续吧 -## ReentrantLock 是如何应用的AQS +## 三、ReentrantLock 是如何应用的AQS 独占式的典型应用就是 ReentrantLock 了，我们来看看它是如何重写这个方法的 -[![img](https://rgyb.sunluomeng.top/20200530113417.png)](https://rgyb.sunluomeng.top/20200530113417.png) +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200929111301.png) 乍一看挺奇怪的，怎么里面自定义了三个同步器：其实 NonfairSync，FairSync 只是对 Sync 做了进一步划分： -[![img](https://rgyb.sunluomeng.top/20200531100921.png)](https://rgyb.sunluomeng.top/20200531100921.png) +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200929111333.png) 从名称上你应该也知道了，这就是你听到过的 `公平锁/非公平锁`了 @@ -1265,7 +1159,7 @@ private void doSignalAll(Node first) { 我们来对比一下 ReentrantLock 是如何实现公平锁和非公平锁的 -[![img](https://rgyb.sunluomeng.top/20200531102752.png)](https://rgyb.sunluomeng.top/20200531102752.png) +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200929111424.png) 其实没什么大不了，公平锁就是判断同步队列是否还有先驱节点的存在，只有没有先驱节点才能获取锁；而非公平锁是不管这个事的，能获取到同步状态就可以，就这么简单，那问题来了： @@ -1273,19 +1167,19 @@ private void doSignalAll(Node first) { 考虑这个问题，我们需重新回忆上面的锁获取实现图了，其实上面我已经透露了一点 -[![img](https://rgyb.sunluomeng.top/20200530210706.png)](https://rgyb.sunluomeng.top/20200530210706.png) +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200929111526.png) 主要有两点原因： #### 原因一： -恢复挂起的线程到真正锁的获取还是有时间差的，从人类的角度来看这个时间微乎其微，但是从CPU的角度来看，这个时间差存在的还是很明显的。所以非公平锁能更充分的利用 CPU 的时间片，尽量减少 CPU 空闲状态时间 +恢复挂起的线程到真正锁的获取还是有时间差的，从人类的角度来看这个时间微乎其微，但是从 CPU 的角度来看，这个时间差存在的还是很明显的。所以非公平锁能更充分的利用 CPU 的时间片，尽量减少 CPU 空闲状态时间 -\####原因二： +#### 原因二： -不知你是否还记得我在 [面试问，创建多少个线程合适？](https://dayarch.top/p/how-many-threads-should-be-created.html) 文章中反复提到过，使用多线程很重要的考量点是线程切换的开销，想象一下，如果采用非公平锁，当一个线程请求锁获取同步状态，然后释放同步状态，因为不需要考虑是否还有前驱节点，所以刚释放锁的线程在此刻再次获取同步状态的几率就变得非常大，所以就减少了线程的开销 +不知你是否还记得我在 [面试问，创建多少个线程合适？](https://my.oschina.net/u/4149877/blog/3224325) 文章中反复提到过，使用多线程很重要的考量点是线程切换的开销，想象一下，如果采用非公平锁，当一个线程请求锁获取同步状态，然后释放同步状态，因为不需要考虑是否还有前驱节点，所以刚释放锁的线程在此刻再次获取同步状态的几率就变得非常大，所以就减少了线程的开销 -[![img](https://rgyb.sunluomeng.top/20200531104701.png)](https://rgyb.sunluomeng.top/20200531104701.png) +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200929111849.png) 相信到这里，你也就明白了，为什么 ReentrantLock 默认构造器用的是非公平锁同步器 @@ -1297,7 +1191,7 @@ public ReentrantLock() { 看到这里，感觉非公平锁 perfect，非也，有得必有失 -> 使用公平锁会有什么问题？ +> 使用非公平锁会有什么问题？ 公平锁保证了排队的公平性，非公平锁霸气的忽视这个规则，所以就有可能导致排队的长时间在排队，也没有机会获取到锁，这就是传说中的 **“饥饿”** @@ -1324,9 +1218,9 @@ else if (current == getExclusiveOwnerThread()) 仔细看代码， 你也许发现，我前面的一个说明是错误的，我要重新解释一下 -[![img](https://rgyb.sunluomeng.top/20200531110129.png)](https://rgyb.sunluomeng.top/20200531110129.png) +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200929112202.png) -重入的线程会一直将 state + 1， 释放锁会 state - 1直至等于0，上面这样写也是想帮助大家快速的区分 +重入的线程会一直将 state + 1， 释放锁会 state - 1 直至等于0，上面这样写也是想帮助大家快速的区分 ## 总结 @@ -1336,7 +1230,7 @@ else if (current == getExclusiveOwnerThread()) 另外也欢迎大家的留言，如有错误之处还请指出，我的手酸了，眼睛干了，我去准备撸下一篇….. -[![img](https://rgyb.sunluomeng.top/20200531112135.png)](https://rgyb.sunluomeng.top/20200531112135.png) +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200929112228.png) ## 灵魂追问 @@ -1371,32 +1265,28 @@ else if (current == getExclusiveOwnerThread()) } ``` -## 参考 - -1. Java 并发实战 -2. Java 并发编程的艺术 -3. https://tech.meituan.com/2019/12/05/aqs-theory-and-apply.html - - - - +**AQS 核心思想是，如果被请求的共享资源空闲，则将当前请求资源的线程设置为有效的工作线程，并且将共享资源设置为锁定状态。如果被请求的共享资源被占用，那么就需要一套线程阻塞等待以及被唤醒时锁分配的机制，这个机制 AQS 是用 CLH 队列锁实现的，即将暂时获取不到锁的线程加入到队列中。** +## 参考与来源 +1. Java 并发实战 +2. Java 并发编程的艺术 +3. https://tech.meituan.com/2019/12/05/aqs-theory-and-apply.html +4. https://github.com/Snailclimb/JavaGuide/blob/master/docs/java/Multithread/AQS.md +5. https://www.javadoop.com/post/AbstractQueuedSynchronizer-2 +6. https://www.cnblogs.com/waterystone/p/4920797.html +7. https://www.cnblogs.com/chengxiao/archive/2017/07/24/7141160.html +8. https://dayarch.top/p/java-aqs-and-reentrantlock.html +9. https://ifeve.com/java%E5%B9%B6%E5%8F%91%E4%B9%8Baqs%E8%AF%A6%E8%A7%A3/ +10. https://www.cnblogs.com/liqiangchn/p/11960944.html -参考与感谢 -Java极客技术 -https://tech.meituan.com/2019/12/05/aqs-theory-and-apply.html -https://github.com/Snailclimb/JavaGuide/blob/master/docs/java/Multithread/AQS.md -https://www.javadoop.com/post/AbstractQueuedSynchronizer-2 -https://www.cnblogs.com/waterystone/p/4920797.html -https://www.cnblogs.com/chengxiao/archive/2017/07/24/7141160.html \ No newline at end of file diff --git a/docs/java/JUC/BlockingQueue.md b/docs/java/JUC/BlockingQueue.md index a6d3aa0c19..f87350060c 100644 --- a/docs/java/JUC/BlockingQueue.md +++ b/docs/java/JUC/BlockingQueue.md @@ -1,12 +1,10 @@ # 阻塞队列——手写生产者消费者模式、线程池原理面试题真正的答案 -> 文章收录在 GitHub [JavaKeeper](https://github.com/Jstarfish/JavaKeeper) ，N线互联网开发必备技能兵器谱 - ## 队列和阻塞队列 ### 队列 -队列（`Queue`）是一种经常使用的集合。`Queue`实际上是实现了一个先进先出（FIFO：First In First Out）的有序表。和 List、Set一样都继承自 Collection。它和`List`的区别在于，`List`可以在任意位置添加和删除元素，而`Queue` 只有两个操作： +队列（`Queue`）是一种经常使用的集合。`Queue` 实际上是实现了一个先进先出（FIFO：First In First Out）的有序表。和 List、Set 一样都继承自 Collection。它和 `List` 的区别在于，`List`可以在任意位置添加和删除元素，而`Queue` 只有两个操作： - 把元素添加到队列末尾； - 从队列头部取出元素。 @@ -17,7 +15,7 @@ -我们常用的 LinkedList 就可以当队列使用，实现了Dequeue接口，还有 ConcurrentLinkedQueue，他们都属于非阻塞队列。 +我们常用的 LinkedList 就可以当队列使用，实现了 Dequeue 接口，还有 ConcurrentLinkedQueue，他们都属于非阻塞队列。 ### 阻塞队列 @@ -64,7 +62,7 @@ JDK 提供了 7 个阻塞队列。分别是 - PriorityBlockingQueue ：一个支持优先级排序的无界阻塞队列 - DelayQueue：一个使用优先级队列实现的无界阻塞队列 - SynchronousQueue：一个不存储元素的阻塞队列 -- LinkedTransferQueue：一个由链表结构组成的无界阻塞队列（实现了继承于 BlockingQueue的 TransferQueue） +- LinkedTransferQueue：一个由链表结构组成的无界阻塞队列（实现了继承于 BlockingQueue 的 TransferQueue） - LinkedBlockingDeque：一个由链表结构组成的双向阻塞队列 @@ -79,7 +77,7 @@ JDK 提供了 7 个阻塞队列。分别是 | 移除（取出） | remove() | poll() | take() | poll(time,unit) | | 检查 | element() | peek() | 不可用 | 不可用 | -以 ArrayBlockingQueue 来看下 Java 阻塞队列提供的常用方法 +以 ArrayBlockingQueue 为例来看下 Java 阻塞队列提供的常用方法 - 抛出异常： @@ -87,7 +85,7 @@ JDK 提供了 7 个阻塞队列。分别是 - 当队列为空时，从队列里 remove 移除元素时会抛出 `NoSuchElementException` 异常 。 - element()，返回队列头部的元素，如果队列为空，则抛出一个 `NoSuchElementException` 异常 - ![](https://imgkr.cn-bj.ufileos.com/fb1a5a4c-e438-4308-92ec-9297f61136da.png) + ![](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWdrci5jbi1iai51ZmlsZW9zLmNvbS9mYjFhNWE0Yy1lNDM4LTQzMDgtOTJlYy05Mjk3ZjYxMTM2ZGEucG5n?x-oss-process=image/format.png) - 返回特殊值： @@ -95,21 +93,21 @@ JDK 提供了 7 个阻塞队列。分别是 - poll()，移除方法，成功返回出队列的元素，队列里没有则返回 null - peek() ，返回队列头部的元素，如果队列为空，则返回 null - ![](https://imgkr.cn-bj.ufileos.com/00d85478-0871-40da-900d-4d6cd9047b8c.png) + ![](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWdrci5jbi1iai51ZmlsZW9zLmNvbS8wMGQ4NTQ3OC0wODcxLTQwZGEtOTAwZC00ZDZjZDkwNDdiOGMucG5n?x-oss-process=image/format.png) - 一直阻塞： - 当阻塞队列满时，如果生产线程继续往队列里 put 元素，队列会一直阻塞生产线程，直到拿到数据，或者响应中断退出； - 当阻塞队列空时，消费线程试图从队列里 take 元素，队列也会一直阻塞消费线程，直到队列可用。 - ![](https://imgkr.cn-bj.ufileos.com/003143ab-68bb-4f2b-943b-bc870ac96900.png) + ![](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWdrci5jbi1iai51ZmlsZW9zLmNvbS8wMDMxNDNhYi02OGJiLTRmMmItOTQzYi1iYzg3MGFjOTY5MDAucG5n?x-oss-process=image/format.png) - 超时退出： - 当阻塞队列满时，队列会阻塞生产线程一定时间，如果超过一定的时间，生产线程就会退出，返回 false -- 当阻塞队列空时，队列会阻塞消费线程一定时间，如果超过一定的时间，消费线程会退出，返回 null + - 当阻塞队列空时，队列会阻塞消费线程一定时间，如果超过一定的时间，消费线程会退出，返回 null - ![](https://imgkr.cn-bj.ufileos.com/b82734a0-8dbe-4ae9-8e4e-ea44445f46ff.png) + ![](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9pbWdrci5jbi1iai51ZmlsZW9zLmNvbS9iODI3MzRhMC04ZGJlLTRhZTktOGU0ZS1lYTQ0NDQ1ZjQ2ZmYucG5n?x-oss-process=image/format.png) @@ -648,9 +646,9 @@ public class LinkedBlockingQueue extends AbstractQueue #### LinkedBlockingQueue 与 ArrayBlockingQueue 对比 - ArrayBlockingQueue 入队出队采用一把锁，导致入队出队相互阻塞，效率低下； -- LinkedBlockingQueue入队出队采用两把锁，入队出队互不干扰，效率较高； +- LinkedBlockingQueue 入队出队采用两把锁，入队出队互不干扰，效率较高； - 二者都是有界队列，如果长度相等且出队速度跟不上入队速度，都会导致大量线程阻塞； -- LinkedBlockingQueue 如果初始化不传入初始容量，则使用最大int值，如果出队速度跟不上入队速度，会导致队列特别长，占用大量内存； +- LinkedBlockingQueue 如果初始化不传入初始容量，则使用最大 int 值，如果出队速度跟不上入队速度，会导致队列特别长，占用大量内存； @@ -658,7 +656,7 @@ public class LinkedBlockingQueue extends AbstractQueue PriorityBlockingQueue 是一个支持优先级的无界阻塞队列。(虽说是无界队列，但是由于资源耗尽的话，也会OutOfMemoryError，无法添加元素) -默认情况下元素采用自然顺序升序排列。也可以自定义类实现 compareTo() 方法来指定元素排序规则，或者初始化 PriorityBlockingQueue 时，指定构造参数 Comparator 来对元素进行排序。但需要注意的是不能保证同优先级元素的顺序。PriorityBlockingQueue 是基于**最小二叉堆**实现，使用基于 CAS 实现的自旋锁来控制队列的动态扩容，保证了扩容操作不会阻塞 take 操作的执行。 +默认情况下元素采用自然顺序升序排列。也可以自定义类实现 `compareTo()` 方法来指定元素排序规则，或者初始化 PriorityBlockingQueue 时，指定构造参数 Comparator 来对元素进行排序。但需要注意的是不能保证同优先级元素的顺序。PriorityBlockingQueue 是基于**最小二叉堆**实现，使用基于 CAS 实现的自旋锁来控制队列的动态扩容，保证了扩容操作不会阻塞 take 操作的执行。 @@ -826,7 +824,7 @@ static final class QNode { } ``` -从 put() 方法和 take() 方法可以看出最终调用的都是 TransferQueue 的 transfer() 方法。 +从 `put()` 方法和 `take()` 方法可以看出最终调用的都是 TransferQueue 的 `transfer()` 方法。 ```java public void put(E e) throws InterruptedException { @@ -1050,7 +1048,11 @@ public ThreadPoolExecutor(int corePoolSize, 不同的线程池实现用的是不同的阻塞队列，newFixedThreadPool 和 newSingleThreadExecutor 用的是LinkedBlockingQueue，newCachedThreadPool 用的是 SynchronousQueue。 -![](https://user-gold-cdn.xitu.io/2020/3/20/170f5beacffbc730?w=750&h=390&f=jpeg&s=29031) + + +> 文章持续更新，可以微信搜「 **JavaKeeper** 」第一时间阅读，无套路领取 500+ 本电子书和 30+ 视频教学和源码，本文 **GitHub** [github.com/JavaKeeper](https://github.com/Jstarfish/JavaKeeper) 已经收录，Javaer 开发、面试必备技能兵器谱，有你想要的。 + + ## 参考与感谢 diff --git a/docs/java/JUC/CAS.md b/docs/java/JUC/CAS.md index e557a744d9..283e435449 100644 --- a/docs/java/JUC/CAS.md +++ b/docs/java/JUC/CAS.md @@ -1,7 +1,7 @@ # 从 Atomic 到 CAS ，竟然衍生出这么多 20k+ 面试题 > CAS 知道吗，如何实现？ -> 讲一讲AtomicInteger，为什么要用 CAS 而不是 synchronized？ +> 讲一讲 AtomicInteger，为什么要用 CAS 而不是 synchronized？ > CAS 底层原理，谈谈你对 UnSafe 的理解？ > AtomicInteger 的ABA问题，能说一下吗，原子更新引用知道吗？ > CAS 有什么缺点吗？ 如何规避 ABA 问题？ @@ -31,10 +31,10 @@ Atomic 原子类可以保证多线程环境下，当某个线程在执行 atomic Atomic 包中的类可以分成 4 组： 1. 基本类型：AtomicBoolean，AtomicInteger，AtomicLong -2. 数组类型：tomicIntegerArray，AtomicLongArray，AtomicReferenceArray +2. 数组类型：AtomicIntegerArray，AtomicLongArray，AtomicReferenceArray 3. 引用类型：AtomicReference，AtomicMarkableReference，AtomicStampedReference -4. 对象的属性修改类型 ：AtomicIntegerFieldUpdater，AtomicLongFieldUpdater，AtomicReferenceFieldUpdater -5. JDK1.8 新增：DoubleAccumulator、LongAccumulator、DoubleAdder、LongAdder、Striped64 +4. 对象的属性修改类型（原子化对象属性更新器） ：AtomicIntegerFieldUpdater，AtomicLongFieldUpdater，AtomicReferenceFieldUpdater +5. JDK1.8 新增（原子化的累加器）：DoubleAccumulator、LongAccumulator、DoubleAdder、LongAdder、Striped64 @@ -83,7 +83,7 @@ false current num:7 ### 2.1 CAS 算法 - CAS：全称 `Compare and swap`，即**比较并交换**，它是一条 **CPU 同步原语**。 是一种硬件对并发的支持，针对多处理器操作而设计的一种特殊指令，用于管理对共享数据的并发访问。 -- CAS 是一种无锁的非阻塞算法的实现。 +- CAS 是一种**无锁的非阻塞算法**的实现。 - CAS 包含了 3 个操作数： - 需要读写的内存值 V - 旧的预期值 A diff --git a/docs/java/JUC/CompletableFuture.md b/docs/java/JUC/CompletableFuture.md new file mode 100644 index 0000000000..7f5338770f --- /dev/null +++ b/docs/java/JUC/CompletableFuture.md @@ -0,0 +1,248 @@ +前面我们不止一次提到，用多线程优化性能，其实不过就是将串行操作变成并行操作。如果仔细观察，你还会发现在串行转换成并行的过程中，一定会涉及到异步化，例如下面的示例代码，现在是串行的，为了提升性能，我们得把它们并行化，那具体实施起来该怎么做呢？ + +```java + +//以下两个方法都是耗时操作 +doBizA(); +doBizB(); +``` + +还是挺简单的，就像下面代码中这样，创建两个子线程去执行就可以了。你会发现下面的并行方案，主线程无需等待 doBizA() 和 doBizB() 的执行结果，也就是说 doBizA() 和 doBizB() 两个操作已经被异步化了。 + +```java + +new Thread(()->doBizA()) + .start(); +new Thread(()->doBizB()) + .start(); +``` + +异步化，是并行方案得以实施的基础，更深入地讲其实就是：利用多线程优化性能这个核心方案得以实施的基础。看到这里，相信你应该就能理解异步编程最近几年为什么会大火了，因为优化性能是互联网大厂的一个核心需求啊。Java 在 1.8 版本提供了 CompletableFuture 来支持异步编程，CompletableFuture 有可能是你见过的最复杂的工具类了，不过功能也着实让人感到震撼。 + +### CompletableFuture 的核心优势 + +为了领略 CompletableFuture 异步编程的优势，这里我们用 CompletableFuture 重新实现前面曾提及的烧水泡茶程序。首先还是需要先完成分工方案，在下面的程序中，我们分了 3 个任务：任务 1 负责洗水壶、烧开水，任务 2 负责洗茶壶、洗茶杯和拿茶叶，任务 3 负责泡茶。其中任务 3 要等待任务 1 和任务 2 都完成后才能开始。这个分工如下图所示 + +![img](https://static001.geekbang.org/resource/image/b3/78/b33f823a4124c1220d8bd6d91b877e78.png) + +下面是代码实现，你先略过 runAsync()、supplyAsync()、thenCombine() 这些不太熟悉的方法，从大局上看，你会发现 + +1. 无需手工维护线程，没有繁琐的手工维护线程的工作，给任务分配线程的工作也不需要我们关注； +2. 语义更清晰，例如 f3 = f1.thenCombine(f2, ()->{}) 能够清晰地表述“任务 3 要等待任务 1 和任务 2 都完成后才能开始”； +3. 代码更简练并且专注于业务逻辑，几乎所有代码都是业务逻辑相关的。 + +```java + +//任务1：洗水壶->烧开水 +CompletableFuture f1 = + CompletableFuture.runAsync(()->{ + System.out.println("T1:洗水壶..."); + sleep(1, TimeUnit.SECONDS); + + System.out.println("T1:烧开水..."); + sleep(15, TimeUnit.SECONDS); +}); +//任务2：洗茶壶->洗茶杯->拿茶叶 +CompletableFuture f2 = + CompletableFuture.supplyAsync(()->{ + System.out.println("T2:洗茶壶..."); + sleep(1, TimeUnit.SECONDS); + + System.out.println("T2:洗茶杯..."); + sleep(2, TimeUnit.SECONDS); + + System.out.println("T2:拿茶叶..."); + sleep(1, TimeUnit.SECONDS); + return "龙井"; +}); +//任务3：任务1和任务2完成后执行：泡茶 +CompletableFuture f3 = + f1.thenCombine(f2, (__, tf)->{ + System.out.println("T1:拿到茶叶:" + tf); + System.out.println("T1:泡茶..."); + return "上茶:" + tf; + }); +//等待任务3执行结果 +System.out.println(f3.join()); + +void sleep(int t, TimeUnit u) { + try { + u.sleep(t); + }catch(InterruptedException e){} +} +// 一次执行结果： +T1:洗水壶... +T2:洗茶壶... +T1:烧开水... +T2:洗茶杯... +T2:拿茶叶... +T1:拿到茶叶:龙井 +T1:泡茶... +上茶:龙井 +``` + +领略 CompletableFuture 异步编程的优势之后，下面我们详细介绍 CompletableFuture 的使用，首先是如何创建 CompletableFuture 对象。 + +### 创建 CompletableFuture 对象 + +创建 CompletableFuture 对象主要靠下面代码中展示的这 4 个静态方法，我们先看前两个。在烧水泡茶的例子中，我们已经使用了runAsync(Runnable runnable)和supplyAsync(Supplier supplier)，它们之间的区别是：Runnable 接口的 run() 方法没有返回值，而 Supplier 接口的 get() 方法是有返回值的。 + +前两个方法和后两个方法的区别在于：后两个方法可以指定线程池参数。 + +默认情况下 CompletableFuture 会使用公共的 ForkJoinPool 线程池，这个线程池默认创建的线程数是 CPU 的核数（也可以通过 JVM option:-Djava.util.concurrent.ForkJoinPool.common.parallelism 来设置 ForkJoinPool 线程池的线程数）。如果所有 CompletableFuture 共享一个线程池，那么一旦有任务执行一些很慢的 I/O 操作，就会导致线程池中所有线程都阻塞在 I/O 操作上，从而造成线程饥饿，进而影响整个系统的性能。所以，强烈建议你要根据不同的业务类型创建不同的线程池，以避免互相干扰。 + +```java + +//使用默认线程池 +static CompletableFuture + runAsync(Runnable runnable) +static CompletableFuture + supplyAsync(Supplier supplier) +//可以指定线程池 +static CompletableFuture + runAsync(Runnable runnable, Executor executor) +static CompletableFuture + supplyAsync(Supplier supplier, Executor executor) +``` + +创建完 CompletableFuture 对象之后，会自动地异步执行 runnable.run() 方法或者 supplier.get() 方法，对于一个异步操作，你需要关注两个问题：一个是异步操作什么时候结束，另一个是如何获取异步操作的执行结果。因为 CompletableFuture 类实现了 Future 接口，所以这两个问题你都可以通过 Future 接口来解决。另外，CompletableFuture 类还实现了 CompletionStage 接口，这个接口内容实在是太丰富了，在 1.8 版本里有 40 个方法，这些方法我们该如何理解呢？ + +### 如何理解 CompletionStage 接口 + +我觉得，你可以站在分工的角度类比一下工作流。任务是有时序关系的，比如有串行关系、并行关系、汇聚关系等。这样说可能有点抽象，这里还举前面烧水泡茶的例子，其中洗水壶和烧开水就是串行关系，洗水壶、烧开水和洗茶壶、洗茶杯这两组任务之间就是并行关系，而烧开水、拿茶叶和泡茶就是汇聚关系。 + +![串行关系](https://static001.geekbang.org/resource/image/e1/9f/e18181998b82718da811ce5807f0ad9f.png) + +![并行关系](https://static001.geekbang.org/resource/image/ea/d2/ea8e1a41a02b0104b421c58b25343bd2.png) + +![汇聚关系](https://static001.geekbang.org/resource/image/3f/3b/3f1a5421333dd6d5c278ffd5299dc33b.png) + + + +CompletionStage 接口可以清晰地描述任务之间的这种时序关系，例如前面提到的 f3 = f1.thenCombine(f2, ()->{}) 描述的就是一种汇聚关系。烧水泡茶程序中的汇聚关系是一种 AND 聚合关系，这里的 AND 指的是所有依赖的任务（烧开水和拿茶叶）都完成后才开始执行当前任务（泡茶）。既然有 AND 聚合关系，那就一定还有 OR 聚合关系，所谓 OR 指的是依赖的任务只要有一个完成就可以执行当前任务。 + + + +在编程领域，还有一个绕不过去的山头，那就是异常处理，CompletionStage 接口也可以方便地描述异常处理。 + +下面我们就来一一介绍，CompletionStage 接口如何描述串行关系、AND 聚合关系、OR 聚合关系以及异常处理。 + +### 1、描述串行关系 + +CompletionStage 接口里面描述串行关系，主要是 thenApply、thenAccept、thenRun 和 thenCompose 这四个系列的接口。thenApply 系列函数里参数 fn 的类型是接口 Function，这个接口里与 CompletionStage 相关的方法是 R apply(T t)，这个方法既能接收参数也支持返回值，所以 thenApply 系列方法返回的是CompletionStage。而 thenAccept 系列方法里参数 consumer 的类型是接口Consumer，这个接口里与 CompletionStage 相关的方法是 void accept(T t)，这个方法虽然支持参数，但却不支持回值，所以 thenAccept 系列方法返回的是CompletionStage。thenRun 系列方法里 action 的参数是 Runnable，所以 action 既不能接收参数也不支持返回值，所以 thenRun 系列方法返回的也是CompletionStage。这些方法里面 Async 代表的是异步执行 fn、consumer 或者 action。其中，需要你注意的是 thenCompose 系列方法，这个系列的方法会新创建出一个子流程，最终结果和 thenApply 系列是相同的。 + +```java + +CompletionStage thenApply(fn); +CompletionStage thenApplyAsync(fn); +CompletionStage thenAccept(consumer); +CompletionStage thenAcceptAsync(consumer); +CompletionStage thenRun(action); +CompletionStage thenRunAsync(action); +CompletionStage thenCompose(fn); +CompletionStage thenComposeAsync(fn); +``` + +通过下面的示例代码，你可以看一下 thenApply() 方法是如何使用的。首先通过 supplyAsync() 启动一个异步流程，之后是两个串行操作，整体看起来还是挺简单的。不过，虽然这是一个异步流程，但任务①②③却是串行执行的，②依赖①的执行结果，③依赖②的执行结果。 + +```java + +CompletableFuture f0 = + CompletableFuture.supplyAsync( + () -> "Hello World") //① + .thenApply(s -> s + " QQ") //② + .thenApply(String::toUpperCase);//③ + +System.out.println(f0.join()); +//输出结果 +HELLO WORLD QQ +``` + +### 2、描述 AND 汇聚关系 + +CompletionStage 接口里面描述 AND 汇聚关系，主要是 thenCombine、thenAcceptBoth 和 runAfterBoth 系列的接口，这些接口的区别也是源自 fn、consumer、action 这三个核心参数不同。它们的使用你可以参考上面烧水泡茶的实现程序，这里就不赘述了。 + +```java + +CompletionStage thenCombine(other, fn); +CompletionStage thenCombineAsync(other, fn); +CompletionStage thenAcceptBoth(other, consumer); +CompletionStage thenAcceptBothAsync(other, consumer); +CompletionStage runAfterBoth(other, action); +CompletionStage runAfterBothAsync(other, action); +``` + +### 3、描述 OR 汇聚关系 + +CompletionStage 接口里面描述 OR 汇聚关系，主要是 applyToEither、acceptEither 和 runAfterEither 系列的接口，这些接口的区别也是源自 fn、consumer、action 这三个核心参数不同。 + +```java + +CompletionStage applyToEither(other, fn); +CompletionStage applyToEitherAsync(other, fn); +CompletionStage acceptEither(other, consumer); +CompletionStage acceptEitherAsync(other, consumer); +CompletionStage runAfterEither(other, action); +CompletionStage runAfterEitherAsync(other, action); +``` + +下面的示例代码展示了如何使用 applyToEither() 方法来描述一个 OR 汇聚关系。 + +```java + +CompletableFuture f1 = + CompletableFuture.supplyAsync(()->{ + int t = getRandom(5, 10); + sleep(t, TimeUnit.SECONDS); + return String.valueOf(t); +}); + +CompletableFuture f2 = + CompletableFuture.supplyAsync(()->{ + int t = getRandom(5, 10); + sleep(t, TimeUnit.SECONDS); + return String.valueOf(t); +}); + +CompletableFuture f3 = + f1.applyToEither(f2,s -> s); + +System.out.println(f3.join()); +``` + +### 4、异常处理 + +虽然上面我们提到的 fn、consumer、action 它们的核心方法都不允许抛出可检查异常，但是却无法限制它们抛出运行时异常，例如下面的代码，执行 7/0 就会出现除零错误这个运行时异常。非异步编程里面，我们可以使用 try{}catch{}来捕获并处理异常，那在异步编程里面，异常该如何处理呢？ + +```java + +CompletableFuture + f0 = CompletableFuture. + .supplyAsync(()->(7/0)) + .thenApply(r->r*10); +System.out.println(f0.join()); +``` + +CompletionStage 接口给我们提供的方案非常简单，比 try{}catch{}还要简单，下面是相关的方法，使用这些方法进行异常处理和串行操作是一样的，都支持链式编程方式。 + +```java + +CompletionStage exceptionally(fn); +CompletionStage whenComplete(consumer); +CompletionStage whenCompleteAsync(consumer); +CompletionStage handle(fn); +CompletionStage handleAsync(fn); +``` + +下面的示例代码展示了如何使用 exceptionally() 方法来处理异常，exceptionally() 的使用非常类似于 try{}catch{}中的 catch{}，但是由于支持链式编程方式，所以相对更简单。既然有 try{}catch{}，那就一定还有 try{}finally{}，whenComplete() 和 handle() 系列方法就类似于 try{}finally{}中的 finally{}，无论是否发生异常都会执行 whenComplete() 中的回调函数 consumer 和 handle() 中的回调函数 fn。whenComplete() 和 handle() 的区别在于 whenComplete() 不支持返回结果，而 handle() 是支持返回结果的。 + +```java + +CompletableFuture + f0 = CompletableFuture + .supplyAsync(()->(7/0)) + .thenApply(r->r*10) + .exceptionally(e->0); +System.out.println(f0.join()); +``` + diff --git a/docs/java/JUC/Concurrent-Container.md b/docs/java/JUC/Concurrent-Container.md new file mode 100644 index 0000000000..5fa06bd73a --- /dev/null +++ b/docs/java/JUC/Concurrent-Container.md @@ -0,0 +1,327 @@ +# Javaer 对集合类要有的“大局观”——同步容器和并发容器总结版 + +容器这部分，工作或是面试遇到的太多了，因为它牵扯的东西也比较多，要放数据，肯定会有数据结构，数据结构又会牵扯到算法，再或者牵扯到高并发，又会有各种安全策略，所以写这一篇，不是为了巩固各种容器的实现细节，而是在心里有个“大局观”，全方位掌握容器。 + +不扯了，开始唠~ + +Java 的集合容器框架中，主要有四大类别：List、Set、Queue、Map。 + +为了在心里有个“大局观”，我们贴张图： + +![图源：pierrchen.blogspot.com](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200910141411.png) + +- 在语言架构上，集合类分为了 Map 和 Collection 两个大的类别。List、Set、Queue 都继承于 Collection。 +- 左上角的那一块灰色里面的四个类（Dictionary、HashTable、Vector、Stack）都是 JDK 遗留下的类，太老了，已经没人使用，而且都有了对应的取代类 +- 图片分上下两部分，最上边粉红色部分是集合类所有接口关系图，绿色部分是他们的主要实现类，也就是我们真正使用的常用集合类 +- 下半部分中都是 `java.util.concurrent` 包下内容，也就是我们常用的并发集合包。同样粉色部分是接口，绿色是其实现类。 + + + +总体回顾一番后，我们知道了上半部分的 ArrayList、LinkedList、HashMap 这些容器都是非线程安全的，但是 Java 也是并发编程的一把利器呀，如果有多个线程并发地访问这些容器时，就会出现问题。因此，在编写程序时，在多线程环境下必须要求程序员手动地在任何访问到这些容器的地方进行同步处理，这样导致在使用这些容器的时候非常地不方便。 + +所以，Java 先提供了同步容器供用户使用。 + +```java + +``` + +其实以上三个容器都是Collections通过代理模式对原本的操作加上了synchronized同步。而synchronized的同步粒度太大，导致在多线程处理的效率很低。所以在 JDK1.5 的时候推出了并发包下的并发容器，来应对多线程下容器处理效率低的问题。 + + + +**同步容器可以简单地理解为通过synchronized来实现同步的容器**，比如 Vector、Hashtable 以及SynchronizedList、SynchronizedMap 等容器。 + + + +## 同步容器 + +- Vector +- Stack +- HashTable +- Collections.synchronizedXXX 方法生成 + +#### 1. Vector + +Vector 和 ArrayList 一样都实现了 List 接口，其对于数组的各种操作和 ArrayList 一样，只是 Vertor 在可能出现线程不安全的所有方法都用 synchronized 进行了修饰。 + +#### 2. Stack + +Stack 是 Vertor 的子类，Stack 实现的是先进后出的栈。在出栈入栈等操作都进行了 synchronized 修饰。 + +#### 3. HashTable + +HashTable 实现了 Map 接口，它实现的功能和 HashMap 基本一致（HashTable 不可存 null，而 HashMap 的键和值都可以存 null）。区别也是 HashTable 使用了 synchronized 修饰了对外方法。 + +#### 4. Collections提供的同步集合类 + +```java +List list = Collections.synchronizedList(new ArrayList()); + +Set set = Collections.synchronizedSet(new HashSet()); + +Map map = Collections.synchronizedMap(new HashMap()); +``` + +可以通过查看 Vector，Hashtable 等这些同步容器的实现代码，可以看到这些容器实现线程安全的方式就是将它们的状态封装起来，**并在需要同步的方法上加上关键字synchronized**。 + +这样做的代价是削弱了并发性，当多个线程共同竞争容器级的锁时，吞吐量就会降低。 + +因此为了解决同步容器的性能问题，Java 5.0 提供了多种并发容器来改进同步容器的性能。 + + + +> 终于进入主题，下面内容才是这篇文章的主角 + +## 并发容器 + +常见并发容器 + +- ConcurrentHashMap：并发版 HashMap +- CopyOnWriteArrayList：并发版 ArrayList +- CopyOnWriteArraySet：并发 Set +- ConcurrentLinkedQueue：并发队列(基于链表) +- ConcurrentLinkedDeque：并发队列(基于双向链表) +- ConcurrentSkipListMap：基于跳表的并发 Map +- ConcurrentSkipListSet：基于跳表的并发 Set +- SynchronousQueue：读写成对的队列 +- DelayQueue：延时队列 + +7个阻塞队列（这个之前的文章详细介绍过 [「阻塞队列」手写生产者消费者、线程池原理面试题真正的答案](https://mp.weixin.qq.com/s/SqFmOV7lCmtJpF1dcYiF-w)）： + +- ArrayBlockingQueue ：一个由数组结构组成的有界阻塞队列 +- LinkedBlockingQueue ：一个由链表结构组成的有界阻塞队列 +- PriorityBlockingQueue ：一个支持优先级排序的无界阻塞队列 +- DelayQueue：一个使用优先级队列实现的无界阻塞队列 +- SynchronousQueue：一个不存储元素的阻塞队列 +- LinkedTransferQueue：一个由链表结构组成的无界阻塞队列（实现了继承于 BlockingQueue的 TransferQueue） +- LinkedBlockingDeque：一个由链表结构组成的双向阻塞队列 + + + +### ConcurrentHashMap + +并发版的 HashMap，不管是写代码还是面试，这都是最常见的并发容器之一。 JDK 7 和 JDK 8 在实现方式上有一些不同，在 JDK 7 版本中，ConcurrentHashMap 的数据结构是由一个 Segment 数组和多个 HashEntry 组成。Segment 数组的意义就是将一个大的 table 分割成多个小的 table 来进行加锁。每一个 Segment 元素存储的是 HashEntry 数组+链表，这个和 HashMap 的数据存储结构一样。一句话就是 JDK7 采用分段锁来降低锁的竞争。 + +![img](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200910104046.jpg) + +JDK8 弃用了段锁，改为采用**数组+链表+红黑树**的数据形式，虽然也是采用了**锁分离**的思想，只是锁住的是一个Node，**并发控制使用 synchronized 和 CAS 来操作**，进一步优化 + +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200910104629.jpg) + + + +> `Copy-On-Write`，顾名思义，在计算机中就是当你想要对一块内存进行修改时，我们不在原有内存块中进行`写`操作，而是将内存拷贝一份，在新的内存中进行`写`操作，`写`完之后呢，就将指向原来内存指针指向新的内存，原来的内存就可以被回收掉嘛！ + +> 从 JDK1.5 开始 Java 并发包里提供了两个使用 `CopyOnWrite` 机制实现的并发容器,它们是`CopyOnWriteArrayList` 和 `CopyOnWriteArraySet`。 + +### CopyOnWriteArrayList + +并发版 ArrayList，底层结构也是数组，其实现机制是在对容器有写入操作时，copy 出一份副本数组，完成操作后将副本数组引用赋值给容器。底层是通过 ReentrantLock 来保证同步的。 + +适用场景： + +- 集合中数据不是特别多，因为涉及到写时复制 +- 读多写少的场景，因为读操作不加锁，写（增、删、改）操作加锁 + +局限性：通过牺牲容器的一致性来换取容器的高并发效率（在 copy 期间读到的是旧数据）。所以不能在需要强一致性的场景下使用。 + +看看源码感受下： + +```java +public class CopyOnWriteArrayList + implements List, RandomAccess, Cloneable, java.io.Serializable { + + final transient ReentrantLock lock = new ReentrantLock(); + private transient volatile Object[] array; + + // 添加元素，有锁 + public boolean add(E e) { + final ReentrantLock lock = this.lock; + //1. 使用Lock,保证写线程在同一时刻只有一个 + lock.lock(); + try { + //2. 获取旧数组引用 + Object[] elements = getArray(); + int len = elements.length; + //3. 创建新的数组，并将旧数组的数据复制到新数组中，比老的大一个空间 + Object[] newElements = Arrays.copyOf(elements, len + 1); + //4. 要添加的元素放进新数组 + newElements[len] = e; + //5. 将旧数组引用指向新的数组（用新数组替换原来的数组） + setArray(newElements); + return true; + } finally { + lock.unlock(); // 解锁 + } + } + + // 读元素，不加锁，因此可能读取到旧数据 + private E get(Object[] a, int index) { + return (E) a[index]; + } +} +``` + + + +### CopyOnWriteArraySet + +并发版本的 Set，看到这个名字的时候我就有个疑惑，我们用 Set 的实现类一般是 HashSet 或 TreeSet，没见过ArraySet，为什么不叫 CopyOnWriteHashSet 呢？ + +> 我敢肯定，这玩意和 CopyOnWriteArrayList 有关系，上部分源码 + +```java +public class CopyOnWriteArraySet extends AbstractSet + implements java.io.Serializable { + + private final CopyOnWriteArrayList al; + + public boolean add(E e) { + return al.addIfAbsent(e); + } + + public void forEach(Consumer action) { + al.forEach(action); + } +``` + +可以看到，类内部以后一个成员变量 CopyOnWriteArrayList，所有相关操作都是围绕这个成员变量操作的，所以它是基于 CopyOnWriteArrayList 实现的，也就是说底层是一个动态数组，而不是散列表（HashMap） + +和 CopyOnWriteArrayList 类似，它的适用场景： + +- Set 大小通常保持很小 +- 只读操作远多于可变操作（可变操作（`add()`、`set()`和 `remove()`等等的开销很大） + + + +> 下面再看两个基于跳表的并发容器，SkipList 即跳表，跳表是一种空间换时间的数据结构，通过冗余数据，将链表一层一层索引，达到类似二分查找的效果，其数据元素默认按照key值升序，天然有序。运用的场景特别多，Redis 中的 Zset 就是跳表实现的 +> +> ![](https://upload.wikimedia.org/wikipedia/commons/8/86/Skip_list.svg) + +### ConcurrentSkipListMap + +ConcurrentSkipListMap 是一个并发安全, 基于 skip list 实现有序存储的 Map。 + +它与 TreeMap 一样，实现了对有序队列的快速查找，但同时，它还是多线程安全的。在多线程环境下，它要比加锁的TreeMap效率高。 + + + +### ConcurrentSkipListSet + +有木有发现我们使用的 Set 一般都是基于 Map 的某种实现类而实现的，ConcurrentSkipListSet 也不例外，他内部维护了一个 ConcurrentNavigableMap 成员变量，可以理解为并发版的 TreeSet，只是有序且线程安全的。 + +```java +public class ConcurrentSkipListSet + extends AbstractSet + implements NavigableSet, Cloneable, java.io.Serializable { + + private final ConcurrentNavigableMap m; + + public ConcurrentSkipListSet() { + m = new ConcurrentSkipListMap(); + } +} +``` + + + + + + + +> 剩下的就是各种队列了，队列又分为两种，阻塞队列和非阻塞队列。 +> +> 阻塞队列是如果你试图向一个 已经满了的队列中添加一个元素或者是从一个空的阻塞队列中移除一个元素，将导致消费者线程或者生产者线程阻塞，非阻塞队列是能即时返回结果（消费者），但必须自行编码解决返回为空的情况处理（以及消费重试等问题）。这两种都是线程安全的。 +> +> 阻塞队列可以用一个锁（入队和出队共享一把锁）或者两个锁（入队使用一把锁，出队使用一把锁）来实现线程安全，JDK中典型的实现是`BlockingQueue`； +> +> 非阻塞队列可以用循环CAS的方式来保证数据的一致性，来达到线程安全的目的。 +> +> ![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200927140845.png) + +先来看下非阻塞队列 + +### ConcurrentLinkedQueue + +ConcurrentLinkedQueue 是一个基于链表实现的无界线程安全队列，遵循队列的FIFO原则，队尾入队，队首出队。使用乐观锁（CAS）保证线程安全。 + + + +> **Deque**（double-ended queue）是一种双端队列，也就是说可以在任意一端进行“入列”，也可以在任意一端进行“出列”。 +> +> 所以 Deque 其实既可以当做队列使用，也可以当做栈来使用。 + +### ConcurrentLinkedDeque + +基于双向链表实现的并发队列，可以分别对头尾进行操作，因此除了先进先出（FIFO），也可以先进后出（FILO）。 + +在 JDK1.7 之前，除了 Stack 类外，并没有其它适合并发环境的“栈”数据结构。ConcurrentLinkedDeque 作为双端队列，可以当作“栈”来使用，并且高效地支持并发环境。 + +ConcurrentLinkedDeque 和 ConcurrentLinkedQueue 一样，采用了无锁算法，底层也是基于**自旋+CAS **的方式实现 + + + +### DelayQueue + +DelayQueue 是一个无界的 BlockingQueue，我们叫做“延时队列”，用于放置实现了 Delayed 接口的对象，其中的对象只能在其到期时才能从队列中取走。这种队列是有序的，即队头对象的延迟到期时间最长。 + +比如我们的自动取消订单业务、下单成功多久发送短信、或者一些任务超时处理，都可能会用到 DelayQueue。 + + + +> 下边说阻塞队列，7 种阻塞队列的详细介绍，之前详细介绍了，他们都是 BlockingQueue 的实现类，所以这里简单回顾下。 + +### ArrayBlockingQueue + +ArrayBlockingQueue，一个由**数组**实现的**有界**阻塞队列。该队列采用先进先出（FIFO）的原则对元素进行排序添加的。 + +ArrayBlockingQueue 为**有界且固定**，其大小在构造时由构造函数来决定，确认之后就不能再改变了。 + + + +### LinkedBlockingQueue + +LinkedBlockingQueue 是一个用单向链表实现的有界阻塞队列。此队列的默认和最大长度为 `Integer.MAX_VALUE`。此队列按照先进先出的原则对元素进行排序。 + +如果不是特殊业务，LinkedBlockingQueue 使用时，切记要定义容量 `new LinkedBlockingQueue(capacity)` + +，防止过度膨胀。 + + + +### LinkedBlockingDeque + +LinkedBlockingDeque 是一个由链表结构组成的双向阻塞队列。 + +所谓双向队列指的你可以从队列的两端插入和移出元素。双端队列因为多了一个操作队列的入口，在多线程同时入队时，也就减少了一半的竞争。 + + + +### PriorityBlockingQueue + +PriorityBlockingQueue 是一个支持优先级的无界阻塞队列。(虽说是无界队列，但是由于资源耗尽的话，也会OutOfMemoryError，无法添加元素) + +默认情况下元素采用自然顺序升序排列。也可以自定义类实现 compareTo() 方法来指定元素排序规则，或者初始化 PriorityBlockingQueue 时，指定构造参数 Comparator 来对元素进行排序。但需要注意的是不能保证同优先级元素的顺序。PriorityBlockingQueue 是基于**最小二叉堆**实现，使用基于 CAS 实现的自旋锁来控制队列的动态扩容，保证了扩容操作不会阻塞 take 操作的执行。 + + + +### SynchronousQueue + +SynchronousQueue 是一个不存储元素的阻塞队列，也即是单个元素的队列。 + +每一个 put 操作必须等待一个 take 操作，否则不能继续添加元素。SynchronousQueue 可以看成是一个传球手，负责把生产者线程处理的数据直接传递给消费者线程。队列本身并不存储任何元素，非常适合于传递性场景, 比如在一个线程中使用的数据，传递给另外一个线程使用，SynchronousQueue 的吞吐量高于 LinkedBlockingQueue 和 ArrayBlockingQueue。 + + + +### LinkedTransferQueue + +LinkedTransferQueue 是一个由链表结构组成的无界阻塞 TransferQueue 队列。 + +LinkedTransferQueue采用一种预占模式。意思就是消费者线程取元素时，如果队列不为空，则直接取走数据，若队列为空，那就生成一个节点（节点元素为null）入队，然后消费者线程被等待在这个节点上，后面生产者线程入队时发现有一个元素为null的节点，生产者线程就不入队了，直接就将元素填充到该节点，并唤醒该节点等待的线程，被唤醒的消费者线程取走元素，从调用的方法返回。我们称这种节点操作为“匹配”方式。 + + + +## 总结 + +Java 并发容器的所有，都列出来了，现在应该对 容器 有个“大局观”了，业务中遇到适合的场景，我们就可以“对症下药”了。 + diff --git "a/docs/java/JUC/CountDownLatch\343\200\201CyclicBarrier\343\200\201Semaphore.md" "b/docs/java/JUC/CountDownLatch\343\200\201CyclicBarrier\343\200\201Semaphore.md" index 6604ba1d08..a94699b969 100644 --- "a/docs/java/JUC/CountDownLatch\343\200\201CyclicBarrier\343\200\201Semaphore.md" +++ "b/docs/java/JUC/CountDownLatch\343\200\201CyclicBarrier\343\200\201Semaphore.md" @@ -1,6 +1,8 @@ +# Java并发编程：CountDownLatch、CyclicBarrier 和 Semaphore + ## CountDownLatch -在多线程协作完成业务功能时，有时候需要等待其他多个线程完成任务之后，主线程才能继续往下执行业务功能，在这种的业务场景下，通常可以使用 Thread 类的 `join()` 方法，让主线程等待被 join 的线程执行完之后，主线程才能继续往下执行。当然，使用线程间消息通信机制也可以完成。其实，Java 并发工具类中为我们提供了类似“倒计时”这样的工具类，可以十分方便的完成所说的这种业务场景。 +在多线程协作完成业务功能时，有时候需要等待其他多个线程完成任务之后，主线程才能继续往下执行业务功能，在这种的业务场景下，通常可以使用 Thread 类的 `join()` 方法，让主线程等待被 join 的线程执行完之后，主线程才能继续往下执行。当然，使用线程间消息通信机制也可以完成。其实，Java 并发工具类中为我们提供了类似“**倒计时**”这样的工具类，可以十分方便的完成所说的这种业务场景。 简单概括他的作用就是：让一些线程阻塞直到另一些线程完成一系列操作后才被唤醒。 @@ -35,7 +37,7 @@ public class CountDownLatchDemo { 上边的代码，如果有人通宵上自习，死心眼的班长会一直等，一直等~ -为了不让班长随机应变，CountDownLatch 还提供了等待限制时间的方法，常用方法一览： +为了让班长随机应变，CountDownLatch 还提供了等待限制时间的方法，常用方法一览： - `await()`：调用该方法的线程等到构造方法传入的 N 减到 0 的时候，才能继续往下执行； - `await(long timeout, TimeUnit unit)`：与上面的 await 方法功能一致，只不过这里有了时间限制，调用该方法的线程等到指定的 timeout 时间后，不管 N 是否减至为 0，都会继续往下执行； @@ -48,9 +50,9 @@ public class CountDownLatchDemo { ## CyclicBarrier -CyclicBarrier 的字面意思是可循环（Cyclic）使用的屏障（Barrier），它要做的事情是，让一组线程到达一个屏障（也可以叫同步点）时被阻塞，直到最后一个线程到达屏障时，屏障才会开门，所有被屏障拦截的线程才会继续干活，线程进入屏障通过 CyclicBarrier 的 await() 方法。 +CyclicBarrier 的字面意思是可循环（Cyclic）使用的屏障（Barrier），它要做的事情是，让一组线程到达一个屏障（也可以叫同步点）时被阻塞，直到最后一个线程到达屏障时，屏障才会开门，所有被屏障拦截的线程才会继续干活，线程进入屏障通过 CyclicBarrier 的 `await()` 方法。 -![dayarch.top](https://rgyb.sunluomeng.top/20200627150846.png) +![](https://tva1.sinaimg.cn/large/007S8ZIlly1gicop7b8boj31hu0s0doe.jpg) 可以理解为：**集齐七颗龙珠，才能召唤神龙**（主力开发都到齐了，才能需求评审，也有点报数的感觉） @@ -84,11 +86,177 @@ public class CyclieBarrierDemo { } ``` -> countDownLatch 相当于做减法，CyclicBarrier 相当于做加法 +> 可以这么理解：**countDownLatch 相当于做减法，CyclicBarrier 相当于做加法** + + + + + +> 看个真实业务的例子：目前对账系统的处理逻辑是首先查询订单，然后查询派送单，之后对比订单和派送单，将差异写入差异库。 +> +> ![img](https://static001.geekbang.org/resource/image/06/fe/068418bdc371b8a1b4b740428a3b3ffe.png) +> +> ```java +> while(存在未对账订单){ +> // 查询未对账订单 +> pos = getPOrders(); +> // 查询派送单 +> dos = getDOrders(); +> // 执行对账操作 +> diff = check(pos, dos); +> // 差异写入差异库 +> save(diff); +> } +> ``` +> +> 我们创建了两个线程 T1 和 T2，并行执行查询未对账订单 getPOrders() 和查询派送单 getDOrders() 这两个操作。在主线程中执行对账操作 check() 和差异写入 save() 两个操作。不过需要注意的是：主线程需要等待线程 T1 和 T2 执行完才能执行 check() 和 save() 这两个操作，为此我们通过调用 T1.join() 和 T2.join() 来实现等待，当 T1 和 T2 线程退出时，调用 T1.join() 和 T2.join() 的主线程就会从阻塞态被唤醒，从而执行之后的 check() 和 save()。 +> +> ```java +> while(存在未对账订单){ +> // 查询未对账订单 +> Thread T1 = new Thread(()->{ +> pos = getPOrders(); +> }); +> T1.start(); +> // 查询派送单 +> Thread T2 = new Thread(()->{ +> dos = getDOrders(); +> }); +> T2.start(); +> // 等待T1、T2结束 +> T1.join(); +> T2.join(); +> // 执行对账操作 +> diff = check(pos, dos); +> // 差异写入差异库 +> save(diff); +> } +> ``` +> +> while 循环里面每次都会创建新的线程，而创建线程可是个耗时的操作。所以最好是创建出来的线程能够循环利用，估计这时你已经想到线程池了，是的，线程池就能解决这个问题。 +> +> 而下面的代码就是用线程池优化后的：我们首先创建了一个固定大小为 2 的线程池，之后在 while 循环里重复利用。一切看上去都很顺利，但是有个问题好像无解了，那就是主线程如何知道 getPOrders() 和 getDOrders() 这两个操作什么时候执行完。前面主线程通过调用线程 T1 和 T2 的 join() 方法来等待线程 T1 和 T2 退出，但是在线程池的方案里，线程根本就不会退出，所以 join() 方法已经失效了。 +> +> ```java +> // 创建2个线程的线程池 +> Executor executor = +> Executors.newFixedThreadPool(2); +> while(存在未对账订单){ +> // 查询未对账订单 +> executor.execute(()-> { +> pos = getPOrders(); +> }); +> // 查询派送单 +> executor.execute(()-> { +> dos = getDOrders(); +> }); +> +> /* ？？如何实现等待？？*/ +> +> // 执行对账操作 +> diff = check(pos, dos); +> // 差异写入差异库 +> save(diff); +> } +> ``` +> +> 直接用 CountDownlatch，计数器的初始值等于 2，之后在pos = getPOrders();和dos = getDOrders();两条语句的后面对计数器执行减 1 操作，这个对计数器减 1 的操作是通过调用 latch.countDown(); 来实现的。在主线程中，我们通过调用 latch.await() 来实现对计数器等于 0 的等待。 +> +> ```java +> // 创建2个线程的线程池 +> Executor executor = Executors.newFixedThreadPool(2); +> while(存在未对账订单){ +> // 计数器初始化为2 +> CountDownLatch latch = new CountDownLatch(2); +> // 查询未对账订单 +> executor.execute(()-> { +> pos = getPOrders(); +> latch.countDown(); +> }); +> // 查询派送单 +> executor.execute(()-> { +> dos = getDOrders(); +> latch.countDown(); +> }); +> +> // 等待两个查询操作结束 +> latch.await(); +> +> // 执行对账操作 +> diff = check(pos, dos); +> // 差异写入差异库 +> save(diff); +> } +> ``` +> +> 经过上面的重重优化之后，长出一口气，终于可以交付了。不过在交付之前还需要再次审视一番，看看还有没有优化的余地，仔细看还是有的。前面我们将 getPOrders() 和 getDOrders() 这两个查询操作并行了，但这两个查询操作和对账操作 check()、save() 之间还是串行的。很显然，这两个查询操作和对账操作也是可以并行的，也就是说，在执行对账操作的时候，可以同时去执行下一轮的查询操作，这个过程可以形象化地表述为下面这幅示意图。 +> +> ![img](https://static001.geekbang.org/resource/image/e6/8b/e663d90f49d9666e618ac1370ccca58b.png) +> +> 那接下来我们再来思考一下如何实现这步优化，两次查询操作能够和对账操作并行，对账操作还依赖查询操作的结果，这明显有点生产者 - 消费者的意思，两次查询操作是生产者，对账操作是消费者。既然是生产者 - 消费者模型，那就需要有个队列，来保存生产者生产的数据，而消费者则从这个队列消费数据。 +> +> 这时候可以用 CyclicBarrier 实现了，一个线程 T1 执行订单的查询工作，一个线程 T2 执行派送单的查询工作，当线程 T1 和 T2 都各自生产完 1 条数据的时候，通知线程 T3 执行对账操作。 +> +> ![img](https://static001.geekbang.org/resource/image/65/ad/6593a10a393d9310a8f864730f7426ad.png) +> +> ```java +> // 订单队列 +> Vector

pos; +> // 派送单队列 +> Vector dos; +> // 执行回调的线程池 +> Executor executor = Executors.newFixedThreadPool(1); +> final CyclicBarrier barrier = +> new CyclicBarrier(2, ()->{ +> executor.execute(()->check()); +> }); +> +> void check(){ +> P p = pos.remove(0); +> D d = dos.remove(0); +> // 执行对账操作 +> diff = check(p, d); +> // 差异写入差异库 +> save(diff); +> } +> +> void checkAll(){ +> // 循环查询订单库 +> Thread T1 = new Thread(()->{ +> while(存在未对账订单){ +> // 查询订单库 +> pos.add(getPOrders()); +> // 等待 +> barrier.await(); +> } +> }); +> T1.start(); +> // 循环查询运单库 +> Thread T2 = new Thread(()->{ +> while(存在未对账订单){ +> // 查询运单库 +> dos.add(getDOrders()); +> // 等待 +> barrier.await(); +> } +> }); +> T2.start(); +> } +> ``` + +### 总结 + +CountDownLatch 和 CyclicBarrier 是 Java 并发包提供的两个非常易用的线程同步工具类，这两个工具类用法的区别在这里还是有必要再强调一下：**CountDownLatch 主要用来解决一个线程等待多个线程的场景**，可以类比旅游团团长要等待所有的游客到齐才能去下一个景点；**而 CyclicBarrier 是一组线程之间互相等待**，更像是几个驴友之间不离不弃。除此之外 CountDownLatch 的计数器是不能循环利用的，也就是说一旦计数器减到 0，再有线程调用 await()，该线程会直接通过。但 CyclicBarrier 的计数器是可以循环利用的，而且具备自动重置的功能，一旦计数器减到 0 会自动重置到你设置的初始值。除此之外，CyclicBarrier 还可以设置回调函数，可以说是功能丰富。 + + + + + + ## Semaphore -Semaphore 翻译过来是信号量的意思，其实我们叫它令牌或者许可更好理解一些。 +Semaphore 翻译过来是**信号量**的意思，其实我们叫它令牌或者许可更好理解一些。 官方是这样解释的： @@ -96,9 +264,11 @@ Semaphore 翻译过来是信号量的意思，其实我们叫它令牌或者许 这个解释太官方，我们用例子来理解： -如果我 `Semaphore s = new Semaphore(1)` 写的是1，我取一下，acquire 一下他就变成0，当变成0之后别人是acquire 不到的，然后继续执行，线程结束之后注意要 `s.release()`,，执行完该执行的就把他 release 掉，release 又把0变回去1， 还原化。 +如果我 `Semaphore s = new Semaphore(1)` 写的是1，我取一下（acquire），他就变成 0，当变成 0 之后别人是 acquire 不到的，然后继续执行，线程结束之后注意要 `s.release()`，执行完该执行的就把他 release 掉，release 又把 0 变回去 1， 还原化。 + +Semaphore 的含义就是限流，比如说你在买票，Semaphore 写 5 就是只能有5个人可以同时买票。acquire 的意思叫获得这把锁，线程如果想继续往下执行，必须得从 Semaphore 里面获得一 个许可， 他一共有 5 个许可，用到 0 了剩下的就得等着。 -Semaphore的含义就是限流，比如说你在买票，Semaphore 写5就是只能有5个人可以同时买票。acquire的意思叫获得这把锁，线程如果想继续往下执行，必须得从 Semaphore 里面获得一 个许可， 他一共有5个许可，用到0了剩下的就得等着。 +> 这是 Lock 不容易实现的一个功能：Semaphore 可以允许多个线程访问一个临界区。 > 我们去海底捞吃火锅，假设海底捞有10 张桌子，同一时间最多有10 桌客人进餐，第 11 以后来的客人必须在候餐区等着，有客人出来后，你就可以进去了 @@ -110,7 +280,7 @@ public class SemaphoreDemo { //模拟 5 张桌子, 这里用公平锁，前5波先进去吃 Semaphore semaphore = new Semaphore(5,true); - //15 波吃饭的客人 + //7 波吃饭的客人 for (int i = 1; i <= 7 ; i++) { new Thread(()->{ try { @@ -147,7 +317,57 @@ public class SemaphoreDemo { 6 吃完离开 ``` -Semaphore 信号量主要用于两个目的，一个是用于多个共享资源的互斥使用，另一个用于并发线程数的控制。 +**Semaphore 信号量主要用于两个目的，一个是用于多个共享资源的互斥使用，另一个用于并发线程数的控制**。 + + + +#### 我们用 Semaphore 实现一个限流器 + +Semaphore 可以允许多个线程访问一个临界区。 + +比较常见的需求就是我们工作中遇到的各种池化资源，例如连接池、对象池、线程池等等。其中，你可能最熟悉数据库连接池，在同一时刻，一定是允许多个线程同时使用连接池的，当然，每个连接在被释放前，是不允许其他线程使用的。 + +假设有一个对象池的需求，所谓对象池呢，指的是一次性创建出 N 个对象，之后所有的线程重复利用这 N 个对象，当然对象在被释放前，也是不允许其他线程使用的。对象池，可以用 List 保存实例对象 + +```java +class ObjPool { + final List pool; + // 用信号量实现限流器 + final Semaphore sem; + // 构造函数 + ObjPool(int size, T t){ + pool = new Vector(){}; + for(int i=0; i func) { + T t = null; + sem.acquire(); + try { + t = pool.remove(0); + return func.apply(t); + } finally { + pool.add(t); + sem.release(); + } + } +} +// 创建对象池 +ObjPool pool = + new ObjPool(10, 2); +// 通过对象池获取t，之后执行 +pool.exec(t -> { + System.out.println(t); + return t.toString(); +}); +``` + +我们用一个 List来保存对象实例，用 Semaphore 实现限流器。 + +关键的代码是 ObjPool 里面的 exec() 方法，这个方法里面实现了限流的功能。在这个方法里面，我们首先调用 acquire() 方法（与之匹配的是在 finally 里面调用 release() 方法），假设对象池的大小是 10，信号量的计数器初始化为 10，那么前 10 个线程调用 acquire() 方法，都能继续执行，相当于通过了信号灯，而其他线程则会阻塞在 acquire() 方法上。对于通过信号灯的线程，我们为每个线程分配了一个对象 t（这个分配工作是通过 pool.remove(0) 实现的），分配完之后会执行一个回调函数 func，而函数的参数正是前面分配的对象 t ；执行完回调函数之后，它们就会释放对象（这个释放工作是通过 pool.add(t) 实现的），同时调用 release() 方法来更新信号量的计数器。如果此时信号量里计数器的值小于等于 0，那么说明有线程在等待，此时会自动唤醒等待的线程。 @@ -193,3 +413,10 @@ Semaphore 信号量主要用于两个目的，一个是用于多个共享资源 > 他们底层逻辑是基于 AbstractQueuedSynchronizer 实现的。 + + +## 小总结 + +1. CountDownLatch 可以实现计数等待，主要用于某个线程等待其他几个线程 +2. CyclicBarrier 实现循环栅栏，主要用于多个线程同时等待其他线程 +3. Semaphore 信号量，主要强调只有某些个数量的线程能拿到资源执行 \ No newline at end of file diff --git a/docs/java/JUC/ForkJoin.md b/docs/java/JUC/ForkJoin.md new file mode 100644 index 0000000000..88990a2aa0 --- /dev/null +++ b/docs/java/JUC/ForkJoin.md @@ -0,0 +1,160 @@ +前面几篇文章我们介绍了线程池、Future、CompletableFuture 和 CompletionService，仔细观察你会发现这些工具类都是在帮助我们站在任务的视角来解决并发问题，而不是让我们纠缠在线程之间如何协作的细节上（比如线程之间如何实现等待、通知等）。对于简单的并行任务，你可以通过“线程池 +Future”的方案来解决；如果任务之间有聚合关系，无论是 AND 聚合还是 OR 聚合，都可以通过 CompletableFuture 来解决；而批量的并行任务，则可以通过 CompletionService 来解决。我们一直讲，并发编程可以分为三个层面的问题，分别是分工、协作和互斥，当你关注于任务的时候，你会发现你的视角已经从并发编程的细节中跳出来了，你应用的更多的是现实世界的思维模式，类比的往往是现实世界里的分工，所以我把线程池、Future、CompletableFuture 和 CompletionService 都列到了分工里面。下面我用现实世界里的工作流程图描述了并发编程领域的简单并行任务、聚合任务和批量并行任务，辅以这些流程图，相信你一定能将你的思维模式转换到现实世界里来。 + +![img](https://static001.geekbang.org/resource/image/47/2d/47f3e1e8834c99d9a1933fb496ffde2d.png) + +上面提到的简单并行、聚合、批量并行这三种任务模型，基本上能够覆盖日常工作中的并发场景了，但还是不够全面，因为还有一种“分治”的任务模型没有覆盖到。分治，顾名思义，即分而治之，是一种解决复杂问题的思维方法和模式；具体来讲，指的是把一个复杂的问题分解成多个相似的子问题，然后再把子问题分解成更小的子问题，直到子问题简单到可以直接求解。理论上来讲，解决每一个问题都对应着一个任务，所以对于问题的分治，实际上就是对于任务的分治。分治思想在很多领域都有广泛的应用，例如算法领域有分治算法（归并排序、快速排序都属于分治算法，二分法查找也是一种分治算法）；大数据领域知名的计算框架 MapReduce 背后的思想也是分治。既然分治这种任务模型如此普遍，那 Java 显然也需要支持，Java 并发包里提供了一种叫做 Fork/Join 的并行计算框架，就是用来支持分治这种任务模型的。 + + + +### 分治任务模型 + +这里你需要先深入了解一下分治任务模型，分治任务模型可分为两个阶段：一个阶段是任务分解，也就是将任务迭代地分解为子任务，直至子任务可以直接计算出结果；另一个阶段是结果合并，即逐层合并子任务的执行结果，直至获得最终结果。下图是一个简化的分治任务模型图，你可以对照着理解。 + +![img](https://static001.geekbang.org/resource/image/d2/6a/d2649d8db8e5642703aa5563d76eb86a.png) + +在这个分治任务模型里，任务和分解后的子任务具有相似性，这种相似性往往体现在任务和子任务的算法是相同的，但是计算的数据规模是不同的。具备这种相似性的问题，我们往往都采用递归算法。 + + + +### Fork/Join 的使用 + +Fork/Join 是一个并行计算的框架，主要就是用来支持分治任务模型的，这个计算框架里的 Fork 对应的是分治任务模型里的任务分解，Join 对应的是结果合并。Fork/Join 计算框架主要包含两部分，一部分是分治任务的线程池 ForkJoinPool，另一部分是分治任务 ForkJoinTask。这两部分的关系类似于 ThreadPoolExecutor 和 Runnable 的关系，都可以理解为提交任务到线程池，只不过分治任务有自己独特类型 ForkJoinTask。ForkJoinTask 是一个抽象类，它的方法有很多，最核心的是 fork() 方法和 join() 方法，其中 fork() 方法会异步地执行一个子任务，而 join() 方法则会阻塞当前线程来等待子任务的执行结果。ForkJoinTask 有两个子类——RecursiveAction 和 RecursiveTask，通过名字你就应该能知道，它们都是用递归的方式来处理分治任务的。这两个子类都定义了抽象方法 compute()，不过区别是 RecursiveAction 定义的 compute() 没有返回值，而 RecursiveTask 定义的 compute() 方法是有返回值的。这两个子类也是抽象类，在使用的时候，需要你定义子类去扩展。接下来我们就来实现一下，看看如何用 Fork/Join 这个并行计算框架计算斐波那契数列（下面的代码源自 Java 官方示例）。首先我们需要创建一个分治任务线程池以及计算斐波那契数列的分治任务，之后通过调用分治任务线程池的 invoke() 方法来启动分治任务。由于计算斐波那契数列需要有返回值，所以 Fibonacci 继承自 RecursiveTask。分治任务 Fibonacci 需要实现 compute() 方法，这个方法里面的逻辑和普通计算斐波那契数列非常类似，区别之处在于计算 Fibonacci(n - 1) 使用了异步子任务，这是通过 f1.fork() 这条语句实现的。 + +```java + +static void main(String[] args){ + //创建分治任务线程池 + ForkJoinPool fjp = + new ForkJoinPool(4); + //创建分治任务 + Fibonacci fib = + new Fibonacci(30); + //启动分治任务 + Integer result = + fjp.invoke(fib); + //输出结果 + System.out.println(result); +} +//递归任务 +static class Fibonacci extends + RecursiveTask{ + final int n; + Fibonacci(int n){this.n = n;} + protected Integer compute(){ + if (n <= 1) + return n; + Fibonacci f1 = + new Fibonacci(n - 1); + //创建子任务 + f1.fork(); + Fibonacci f2 = + new Fibonacci(n - 2); + //等待子任务结果，并合并结果 + return f2.compute() + f1.join(); + } +} +``` + +### ForkJoinPool 工作原理 + +Fork/Join 并行计算的核心组件是 ForkJoinPool，所以下面我们就来简单介绍一下 ForkJoinPool 的工作原理。通过专栏前面文章的学习，你应该已经知道 ThreadPoolExecutor 本质上是一个生产者 - 消费者模式的实现，内部有一个任务队列，这个任务队列是生产者和消费者通信的媒介；ThreadPoolExecutor 可以有多个工作线程，但是这些工作线程都共享一个任务队列。ForkJoinPool 本质上也是一个生产者 - 消费者的实现，但是更加智能，你可以参考下面的 ForkJoinPool 工作原理图来理解其原理。ThreadPoolExecutor 内部只有一个任务队列，而 ForkJoinPool 内部有多个任务队列，当我们通过 ForkJoinPool 的 invoke() 或者 submit() 方法提交任务时，ForkJoinPool 根据一定的路由规则把任务提交到一个任务队列中，如果任务在执行过程中会创建出子任务，那么子任务会提交到工作线程对应的任务队列中。如果工作线程对应的任务队列空了，是不是就没活儿干了呢？不是的，ForkJoinPool 支持一种叫做“任务窃取”的机制，如果工作线程空闲了，那它可以“窃取”其他工作任务队列里的任务，例如下图中，线程 T2 对应的任务队列已经空了，它可以“窃取”线程 T1 对应的任务队列的任务。如此一来，所有的工作线程都不会闲下来了。ForkJoinPool 中的任务队列采用的是双端队列，工作线程正常获取任务和“窃取任务”分别是从任务队列不同的端消费，这样能避免很多不必要的数据竞争。我们这里介绍的仅仅是简化后的原理，ForkJoinPool 的实现远比我们这里介绍的复杂，如果你感兴趣，建议去看它的源码。 + +![img](https://static001.geekbang.org/resource/image/e7/31/e75988bd5a79652d8325ca63fcd55131.png) + + + +### 模拟 MapReduce 统计单词数量 + +学习 MapReduce 有一个入门程序，统计一个文件里面每个单词的数量，下面我们来看看如何用 Fork/Join 并行计算框架来实现。我们可以先用二分法递归地将一个文件拆分成更小的文件，直到文件里只有一行数据，然后统计这一行数据里单词的数量，最后再逐级汇总结果，你可以对照前面的简版分治任务模型图来理解这个过程。思路有了，我们马上来实现。下面的示例程序用一个字符串数组 String[] fc 来模拟文件内容，fc 里面的元素与文件里面的行数据一一对应。关键的代码在 compute() 这个方法里面，这是一个递归方法，前半部分数据 fork 一个递归任务去处理（关键代码 mr1.fork()），后半部分数据则在当前任务中递归处理（mr2.compute()）。 + +```java + +static void main(String[] args){ + String[] fc = {"hello world", + "hello me", + "hello fork", + "hello join", + "fork join in world"}; + //创建ForkJoin线程池 + ForkJoinPool fjp = + new ForkJoinPool(3); + //创建任务 + MR mr = new MR( + fc, 0, fc.length); + //启动任务 + Map result = + fjp.invoke(mr); + //输出结果 + result.forEach((k, v)-> + System.out.println(k+":"+v)); +} +//MR模拟类 +static class MR extends + RecursiveTask> { + private String[] fc; + private int start, end; + //构造函数 + MR(String[] fc, int fr, int to){ + this.fc = fc; + this.start = fr; + this.end = to; + } + @Override protected + Map compute(){ + if (end - start == 1) { + return calc(fc[start]); + } else { + int mid = (start+end)/2; + MR mr1 = new MR( + fc, start, mid); + mr1.fork(); + MR mr2 = new MR( + fc, mid, end); + //计算子任务，并返回合并的结果 + return merge(mr2.compute(), + mr1.join()); + } + } + //合并结果 + private Map merge( + Map r1, + Map r2) { + Map result = + new HashMap<>(); + result.putAll(r1); + //合并结果 + r2.forEach((k, v) -> { + Long c = result.get(k); + if (c != null) + result.put(k, c+v); + else + result.put(k, v); + }); + return result; + } + //统计单词数量 + private Map + calc(String line) { + Map result = + new HashMap<>(); + //分割单词 + String [] words = + line.split("\\s+"); + //统计单词数量 + for (String w : words) { + Long v = result.get(w); + if (v != null) + result.put(w, v+1); + else + result.put(w, 1L); + } + return result; + } +} +``` + + + +### 总结 + +Fork/Join 并行计算框架主要解决的是分治任务。分治的核心思想是“分而治之”：将一个大的任务拆分成小的子任务去解决，然后再把子任务的结果聚合起来从而得到最终结果。这个过程非常类似于大数据处理中的 MapReduce，所以你可以把 Fork/Join 看作单机版的 MapReduce。Fork/Join 并行计算框架的核心组件是 ForkJoinPool。ForkJoinPool 支持任务窃取机制，能够让所有线程的工作量基本均衡，不会出现有的线程很忙，而有的线程很闲的状况，所以性能很好。Java 1.8 提供的 Stream API 里面并行流也是以 ForkJoinPool 为基础的。不过需要你注意的是，默认情况下所有的并行流计算都共享一个 ForkJoinPool，这个共享的 ForkJoinPool 默认的线程数是 CPU 的核数；如果所有的并行流计算都是 CPU 密集型计算的话，完全没有问题，但是如果存在 I/O 密集型的并行流计算，那么很可能会因为一个很慢的 I/O 计算而拖慢整个系统的性能。所以建议用不同的 ForkJoinPool 执行不同类型的计算任务。如果你对 ForkJoinPool 详细的实现细节感兴趣，也可以参考Doug Lea 的论文。 \ No newline at end of file diff --git a/docs/java/JUC/JUC.md b/docs/java/JUC/JUC.md index eb560a3a01..c1c224bb78 100644 --- a/docs/java/JUC/JUC.md +++ b/docs/java/JUC/JUC.md @@ -1,6 +1,6 @@ -### Java JUC 简介 +### 说说 JUC -在 Java 5.0 提供了 `java.util.concurrent` （简称 JUC ）包，在此包中增加了在并发编程中很常用 的实用工具类，用于定义类似于线程的自定义子 系统，包括线程池、异步 IO 和轻量级任务框架。 提供可调的、灵活的线程池。还提供了设计用于 多线程上下文中的 Collection 实现等。 +在 Java 5.0 提供了 `java.util.concurrent` （简称 JUC ）包，在此包中增加了在并发编程中很常用的实用工具类，用于定义类似于线程的自定义子 系统，包括线程池、异步 IO 和轻量级任务框架。 提供可调的、灵活的线程池。还提供了设计用于 多线程上下文中的 Collection 实现等。 diff --git a/docs/java/JUC/Java-Memory-Model.md b/docs/java/JUC/Java-Memory-Model.md index 9f42d80415..d93197d235 100644 --- a/docs/java/JUC/Java-Memory-Model.md +++ b/docs/java/JUC/Java-Memory-Model.md @@ -17,7 +17,7 @@ -## 硬件内存架构 +## 一、硬件内存架构 计算机在执行程序的时候，每条指令都是在 CPU 中执行的，而执行的时候，又免不了要和数据打交道。而计算机上面的数据，是存放在主存当中的，也就是计算机的物理内存。 @@ -25,19 +25,27 @@ ![](https://tva1.sinaimg.cn/large/00831rSTly1gcw2g16h50j31ho0tqq5w.jpg) -我们以多核 CPU 为例，每个CPU 核都包含**一组 「CPU 寄存器」**，这些寄存器本质上是在 CPU 内存中。CPU 在这些寄存器上执行操作的速度要比在主内存(RAM)中执行的速度快得多。 +> 这些年，我们的 CPU、内存、I/O 设备都在不断迭代，不断朝着更快的方向努力。但是，在这个快速发展的过程中，有一个**核心矛盾一直存在，就是这三者的速度差异**。CPU 和内存的速度差异可以形象地描述为：CPU 是天上一天，内存是地上一年（假设 CPU 执行一条普通指令需要一天，那么 CPU 读写内存得等待一年的时间）。内存和 I/O 设备的速度差异就更大了，内存是天上一天，I/O 设备是地上十年。 -因为**CPU速率高， 内存速率慢，为了让存储体系可以跟上CPU的速度，所以中间又加上 Cache 层，就是我们说的 「CPU 高速缓存」**。 +我们以多核 CPU 为例，每个 CPU 核都包含**一组 「CPU 寄存器」**，这些寄存器本质上是在 CPU 内存中。CPU 在这些寄存器上执行操作的速度要比在主内存(RAM)中执行的速度快得多。 + +因为**CPU速率高， 内存速率慢，为了让存储体系可以跟上 CPU 的速度，所以中间又加上 Cache 层，就是我们说的 『CPU 高速缓存』**。 + +> 为了合理利用 CPU 的高性能，平衡 CPU 、内存、I/O 设备的速度差异，计算机体系结构、操作系统、编译程序都做出了贡献，主要体现为： +> +> 1. CPU 增加了缓存，以均衡与内存的速度差异； +> 2. 操作系统增加了进程、线程，以分时复用 CPU，进而均衡 CPU 与 I/O 设备的速度差异； +> 3. 编译程序优化指令执行次序，使得缓存能够得到更加合理地利用。 ### CPU多级缓存 -由于CPU的运算速度远远超越了1级缓存的数据I\O能力，CPU厂商又引入了多级的缓存结构。通常L1、L2 是每个CPU 核有一个，L3 是多个核共用一个。 +由于 CPU 的运算速度远远超越了 1 级缓存的数据 I\O 能力，CPU 厂商又引入了多级的缓存结构。通常 L1、L2 是每个 CPU 核有一个，L3 是多个核共用一个。 ### Cache Line -Cache又是由很多个**「缓存行」**(Cache line) 组成的。Cache line 是 Cache 和 RAM 交换数据的最小单位。 +Cache 又是由很多个「**缓存行**」(Cache line) 组成的。Cache line 是 Cache 和 RAM 交换数据的最小单位。 -Cache 存储数据是固定大小为单位的，称为一个**Cache entry**，这个单位称为**Cache line**或**Cache block**。给定Cache 容量大小和 Cache line size 的情况下，它能存储的条目个数(number of cache entries)就是固定的。因为Cache 是固定大小的，所以它从主内存获取数据也是固定大小。对于X86来讲，是 64Bytes。对于ARM来讲，较旧的架构的Cache line是32Bytes，但一次内存访存只访问一半的数据也不太合适，所以它经常是一次填两个 Cache line，叫做 double fill。 +Cache 存储数据是固定大小为单位的，称为一个**Cache entry**，这个单位称为 **Cache line** 或 **Cache block**。给定 Cache 容量大小和 Cache line size 的情况下，它能存储的条目个数(number of cache entries)就是固定的。因为Cache 是固定大小的，所以它从主内存获取数据也是固定大小。对于 X86 来讲，是 64Bytes。对于 ARM 来讲，较旧的架构的 Cache line 是 32Bytes，但一次内存访存只访问一半的数据也不太合适，所以它经常是一次填两个 Cache line，叫做 double fill。 @@ -63,15 +71,15 @@ Cache 存储数据是固定大小为单位的，称为一个**Cache entry**， 为了解决这个问题，先后有过两种方法：**总线锁机制**和**缓存锁机制**。 -总线锁就是使用 CPU 提供的一个`LOCK#`信号，当一个处理器在总线上输出此信号，其他处理器的请求将被阻塞，那么该处理器就可以独占共享锁。这样就保证了数据一致性。 +总线锁就是使用 CPU 提供的一个 `LOCK#` 信号，当一个处理器在总线上输出此信号，其他处理器的请求将被阻塞，那么该处理器就可以独占共享锁。这样就保证了数据一致性。 -但是总线锁开销太大，我们需要控制锁的粒度，所以又有了缓存锁，核心就是“**缓存一致性协议**”，不同的 CPU 硬件厂商实现方式稍有不同，有MSI、MESI、MOSI等。 +但是总线锁开销太大，我们需要控制锁的粒度，所以又有了缓存锁，核心就是“**缓存一致性协议**”，不同的 CPU 硬件厂商实现方式稍有不同，有 MSI、MESI、MOSI 等。 ### 代码乱序执行优化 -为了使得处理器内部的运算单元尽量被充分利用，提高运算效率，处理器可能会对输入的代码进行「乱序执行」**(Out-Of-Order Execution)，处理器会在计算之后将乱序执行的结果重组**，乱序优化可以保证在单线程下该执行结果与顺序执行的结果是一致的，但不保证程序中各个语句计算的先后顺序与输入代码中的顺序一致。 +为了使得处理器内部的运算单元尽量被充分利用，提高运算效率，处理器可能会对输入的代码进行「**乱序执行**」**(Out-Of-Order Execution)，处理器会在计算之后将乱序执行的结果重组**，乱序优化可以保证在单线程下该执行结果与顺序执行的结果是一致的，但不保证程序中各个语句计算的先后顺序与输入代码中的顺序一致。 **乱序执行技术是处理器为提高运算速度而做出违背代码原有顺序的优化**。在单核时代，处理器保证做出的优化不会导致执行结果远离预期目标，但在多核环境下却并非如此。 @@ -84,15 +92,13 @@ Cache 存储数据是固定大小为单位的，称为一个**Cache entry**， ### 内存屏障 -又称为内存栅栏，是一个 CPU 指令。尽管我们看到乱序执行初始目的是为了提高效率，但是它看来其好像在这多核时代不尽人意，其中的某些”自作聪明”的优化导致多线程程序产生各种各样的意外。因此有必要存在一种机制来消除乱序执行带来的坏影响，也就是说应该允许程序员显式的告诉处理器对某些地方禁止乱序执行。这种机制就是所谓内存屏障。不同架构的处理器在其指令集中提供了不同的指令来发起内存屏障，对应在编程语言当中就是提供特殊的关键字来调用处理器相关的指令，JMM里我们再探讨。 - - +又称为内存栅栏，是一个 CPU 指令。尽管我们看到乱序执行初始目的是为了提高效率，但是在这多核时代效果好像不尽人意，其中的某些”自作聪明”的优化导致多线程程序产生各种各样的意外。因此有必要存在一种机制来消除乱序执行带来的坏影响，也就是说应该允许程序员显式的告诉处理器对某些地方禁止乱序执行。这种机制就是所谓内存屏障。不同架构的处理器在其指令集中提供了不同的指令来发起内存屏障，对应在编程语言当中就是提供特殊的关键字来调用处理器相关的指令，JMM 里我们再探讨。 ------ -## Java内存模型 +## 二、Java 内存模型 Java 内存模型即 `Java Memory Model`，简称 **JMM**。 @@ -100,19 +106,19 @@ Java 内存模型即 `Java Memory Model`，简称 **JMM**。 「内存模型」可以理解为**在特定操作协议下，对特定的内存或高速缓存进行读写访问的过程抽象**。 -不同架构的物理计算机可以有不一样的内存模型，Java虚拟机也有自己的内存模型。 +不同架构的物理计算机可以有不一样的内存模型，Java 虚拟机也有自己的内存模型。 -Java虚拟机规范中试图定义一种「 **Java 内存模型**」来**屏蔽掉各种硬件和操作系统的内存访问差异**，以实现**让 Java 程序在各种平台下都能达到一致的内存访问效果**，不必因为不同平台上的物理机的内存模型的差异，对各平台定制化开发程序。 +Java 虚拟机规范中试图定义一种「 **Java 内存模型**」来**屏蔽掉各种硬件和操作系统的内存访问差异**，以实现**让 Java 程序在各种平台下都能达到一致的内存访问效果**，不必因为不同平台上的物理机的内存模型的差异，对各平台定制化开发程序。 Java 内存模型的主要目标是定义程序中各个变量的访问规则，即在虚拟机中将变量存储到内存和从内存中取出变量这样的底层细节。这里的变量与我们写 Java 代码中的变量不同，它包括了实例字段、静态字段和构成数组对象的元素，但不包括局部变量和方法参数，因为他们是线程私有的，不会被共享。 - +> Java 内存模型是个很复杂的规范，可以从不同的视角来解读，站在程序员的视角，本质上可以理解为，Java 内存模型规范了 JVM 如何提供**按需禁用缓存**（解决可见性问题）和编译优化（解决有序性问题）的方法。具体来说，这些方法包括 volatile、synchronized 和 final 三个关键字，以及六项 Happens-Before 规则，这也正是本期的重点内容。 ### JMM 组成 - **主内存**：Java 内存模型规定了所有变量都存储在主内存(Main Memory)中（此处的主内存与物理硬件的主内存 RAM 名字一样，两者可以互相类比，但此处仅是虚拟机内存的一部分）。 -- **工作内存**：每条线程都有自己的工作内存(Working Memory，又称本地内存，可与CPU高速缓存类比)，线程的工作内存中保存了该线程使用到的主内存中的共享变量的副本拷贝。**线程对变量的所有操作都必须在工作内存进行，而不能直接读写主内存中的变量**。**工作内存是 JMM 的一个抽象概念，并不真实存在**。 +- **工作内存**：每条线程都有自己的工作内存(Working Memory，又称本地内存，可与 CPU 高速缓存类比)，线程的工作内存中保存了该线程使用到的主内存中的共享变量的副本拷贝。**线程对变量的所有操作都必须在工作内存进行，而不能直接读写主内存中的变量**。**工作内存是 JMM 的一个抽象概念，并不真实存在**。 ![](https://tva1.sinaimg.cn/large/00831rSTly1gcw2gtxpaej314o0lw0uc.jpg) @@ -124,7 +130,7 @@ JMM 与 Java 内存区域中的堆、栈、方法区等并不是同一个层次 ### JMM 与计算机内存结构 - Java 内存模型和硬件内存体系结构也没有什么关系。硬件内存体系结构不区分栈和堆。在硬件上，线程栈和堆都位于主内存中。线程栈和堆的一部分有时可能出现在高速缓存和CPU寄存器中。如下图所示: + Java 内存模型和硬件内存体系结构也没有什么关系。硬件内存体系结构不区分栈和堆。在硬件上，线程栈和堆都位于主内存中。线程栈和堆的一部分有时可能出现在高速缓存和 CPU 寄存器中。如下图所示: ![img](https://tva1.sinaimg.cn/large/00831rSTly1gcw2heypd6j31ee0kc76r.jpg) @@ -136,7 +142,8 @@ JMM 与 Java 内存区域中的堆、栈、方法区等并不是同一个层次 #### 可见性问题（Visibility of Shared Objects） 如果两个或多个线程共享一个对象，则一个线程对共享对象的更新可能对其他线程不可见（当然可以用 Java 提供的关键字 volatile）。 -假设共享对象最初存储在主内存中。在 CPU 1上运行的线程将共享对象读入它的CPU缓存后修改，但是还没来得及即刷新回主内存，这时其他 CPU 上运行的线程就不会看到共享对象的更改。这样，每个线程都可能以自己的线程结束，就出现了可见性问题，如下 + +假设共享对象最初存储在主内存中。在 CPU 1上运行的线程将共享对象读入它的 CPU 缓存后修改，但是还没来得及刷新回主内存，这时其他 CPU 上运行的线程就不会看到共享对象的更改。这样，每个线程都可能以自己的线程结束，就出现了可见性问题，如下 ![](https://tva1.sinaimg.cn/large/00831rSTly1gcw2hs7adij30pu0hq0tr.jpg) @@ -144,16 +151,83 @@ JMM 与 Java 内存区域中的堆、栈、方法区等并不是同一个层次 #### 竞争条件（Race Conditions） -这个其实就是我们常说的原子问题。 +这个其实就是我们常说的「原子性问题」。 如果两个或多个线程共享一个对象，并且多个线程更新该共享对象中的变量，则可能出现竞争条件。 -想象一下，如果线程A将一个共享对象的变量读入到它的CPU缓存中。此时，线程B执行相同的操作，但是进入不同的CPU缓存。现在线程A执行 +1 操作，线程B也这样做。现在该变量增加了两次，在每个CPU缓存中一次。 - -如果这些增量是按顺序执行的，则变量结果会是3，并将原始值+ 2写回主内存。但是，这两个增量是同时执行的，没有适当的同步。不管将哪个线程的结构写回主内存，更新后的值只比原始值高1，显然是有问题的。如下（当然可以用 Java 提供的关键字 Synchronized） +> 由于 IO 太慢，早期的操作系统就发明了多进程，即便在单核的 CPU 上我们也可以一边听着歌，一边写 Bug，这个就是多进程的功劳。 +> +> 操作系统允许某个进程执行一小段时间，例如 50 毫秒，过了 50 毫秒操作系统就会重新选择一个进程来执行（我们称为“任务切换”），这个 50 毫秒称为“时间片”。 +> +> ![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/juc/254b129b145d80e9bb74123d6e620efb.png) +> +> 这里的进程在等待 IO 时之所以会释放 CPU 使用权，是为了让 CPU 在这段等待时间里可以做别的事情，这样一来 CPU 的使用率就上来了；此外，如果这时有另外一个进程也读文件，读文件的操作就会排队，磁盘驱动在完成一个进程的读操作后，发现有排队的任务，就会立即启动下一个读操作，这样 IO 的使用率也上来了。 +> +> 是不是很简单的逻辑？但是，虽然看似简单，支持多进程分时复用在操作系统的发展史上却具有里程碑意义，Unix 就是因为解决了这个问题而名噪天下的。 +> +> 早期的操作系统基于进程来调度 CPU，不同进程间是不共享内存空间的，所以进程要做任务切换就要切换内存映射地址，而一个进程创建的所有线程，都是共享一个内存空间的，所以线程做任务切换成本就很低了。现代的操作系统都基于更轻量的线程来调度，现在我们提到的“任务切换”都是指“线程切换”。 +> +> Java 并发程序都是基于多线程的，自然也会涉及到任务切换，也许你想不到，任务切换竟然也是并发编程里诡异 Bug 的源头之一。任务切换的时机大多数是在时间片结束的时候，我们现在基本都使用高级语言编程，高级语言里一条语句往往需要多条 CPU 指令完成，例如 count += 1，至少需要三条 CPU 指令。 +> +> - 指令 1：首先，需要把变量 count 从内存加载到 CPU 的寄存器； +> - 指令 2：之后，在寄存器中执行 +1 操作； +> - 指令 3：最后，将结果写入内存（缓存机制导致可能写入的是 CPU 缓存而不是内存）。 +> +> 操作系统做任务切换，可以发生在任何一条 CPU 指令执行完，是的，是 CPU 指令，而不是高级语言里的一条语句。这样不同步的操作，就会出现 bug。 + +想象一下，如果线程 A 将一个共享对象的变量读入到它的 CPU 缓存中。此时，线程 B 执行相同的操作，但是进入不同的 CPU 缓存。现在线程 A 执行 +1 操作，线程 B 也这样做。现在该变量增加了两次，在每个 CPU 缓存中一次。 + +如果这些增量是按顺序执行的，则变量结果会是 3，并将原始值 +2 写回主内存。但是，这两个增量是同时执行的，没有适当的同步。不管将哪个线程的结果写回主内存，更新后的值只比原始值高 1，显然是有问题的。如下（当然可以用 Java 提供的关键字 Synchronized） ![](https://tva1.sinaimg.cn/large/00831rSTly1gcw2i23173j30pu0hqgml.jpg) +#### 有序性问题 + +顾名思义，有序性指的是程序按照代码的先后顺序执行。编译器为了优化性能，有时候会改变程序中语句的先后顺序，例如程序中：“a=6；b=7；”编译器优化后可能变成“b=7；a=6；”，在这个例子中，编译器调整了语句的顺序，但是不影响程序的最终结果。 + +这个就是我们上文说到的代码乱序执行优化。 + +不过有时候编译器及解释器的优化可能导致意想不到的 Bug。 + +> 在 Java 领域一个经典的案例就是利用双重检查创建单例对象，例如下面的代码：在获取实例 getInstance() 的方法中，我们首先判断 instance 是否为空，如果为空，则锁定 Singleton.class 并再次检查 instance 是否为空，如果还为空则创建 Singleton 的一个实例。 +> +> ```java +> public class Singleton { +> static Singleton instance; +> static Singleton getInstance(){ +> if (instance == null) { +> synchronized(Singleton.class) { +> if (instance == null) +> instance = new Singleton(); +> } +> } +> return instance; +> } +> } +> ``` +> +> 假设有两个线程 A、B 同时调用 getInstance() 方法，他们会同时发现 instance == null ，于是同时对 Singleton.class 加锁，此时 JVM 保证只有一个线程能够加锁成功（假设是线程 A），另外一个线程则会处于等待状态（假设是线程 B）；线程 A 会创建一个 Singleton 实例，之后释放锁，锁释放后，线程 B 被唤醒，线程 B 再次尝试加锁，此时是可以加锁成功的，加锁成功后，线程 B 检查 instance == null 时会发现，已经创建过 Singleton 实例了，所以线程 B 不会再创建一个 Singleton 实例。 +> +> 这看上去一切都很完美，无懈可击，但实际上这个 getInstance() 方法并不完美。问题出在哪里呢？ +> +> 出在 new 操作上，我们以为的 new 操作应该是： +> +> 1. 分配一块内存 M； +> 2. 在内存 M 上初始化 Singleton 对象； +> 3. 然后 M 的地址赋值给 instance 变量。 +> +> 但是实际上优化后的执行路径可能是这样的： +> +> 1. 分配一块内存 M； +> 2. 将 M 的地址赋值给 instance 变量； +> 3. 最后在内存 M 上初始化 Singleton 对象。 +> +> 优化后会导致什么问题呢？我们假设线程 A 先执行 getInstance() 方法，当执行完指令 2 时恰好发生了线程切换，切换到了线程 B 上；如果此时线程 B 也执行 getInstance() 方法，那么线程 B 在执行第一个判断时会发现 instance != null ，所以直接返回 instance，而此时的 instance 是没有初始化过的，如果我们这个时候访问 instance 的成员变量就可能触发空指针异常。 + + + + + ### JMM 特性 JMM 就是用来解决如上问题的。 **JMM是围绕着并发过程中如何处理可见性、原子性和有序性这 3 个 特征建立起来的** @@ -178,11 +252,11 @@ JMM 就是用来解决如上问题的。 **JMM是围绕着并发过程中如何 ### 内存之间的交互操作 -关于主内存和工作内存之间具体的交互协议，即一个变量如何从主内存拷贝到工作内存、如何从工作内存同步回主内存之类的实现细节，Java 内存模型中定义了 8 种 操作来完成，虚拟机实现必须保证每一种操作都是原子的、不可再拆分的（double和long类型例外） +关于主内存和工作内存之间具体的交互协议，即一个变量如何从主内存拷贝到工作内存、如何从工作内存同步回主内存之类的实现细节，Java 内存模型中定义了 8 种 操作来完成，虚拟机实现必须保证每一种操作都是原子的、不可再拆分的（double 和 long 类型例外） - **lock**（锁定）：作用于主内存的变量，它把一个变量标识为一条线程独占的状态。 - **unlock**（解锁）：作用于主内存的变量，它把一个处于锁定状态的变量释放出来，释放后的变量才可以被其他线程锁定。 -- **read**（读取）：作用于主内存的变量，它把一个变量的值从主内存传输到线程的工作内存中，以便随后的load动作使用。 +- **read**（读取）：作用于主内存的变量，它把一个变量的值从主内存传输到线程的工作内存中，以便随后的load 动作使用。 - **load**（载入）：作用于工作内存的变量，它把 read 操作从主内存中得到的变量值放入工作内存的变量副本中。 - **use**（使用）：作用于工作内存的变量，它把工作内存中一个变量的值传递给执行引擎，每当虚拟机遇到一个需要使用到变量的值的字节码指令时将会执行这个操作。 - **assign**（赋值）：作用于工作内存的变量，它把一个从执行引擎接收到的值赋给工作内存的变量，每当虚拟机遇到一个给变量赋值的字节码指令时执行这个操作。 @@ -208,7 +282,11 @@ JMM 就是用来解决如上问题的。 **JMM是围绕着并发过程中如何 Java 内存模型要求 lock，unlock，read，load，assign，use，store，write 这 8 个操作都具有原子性，但对于64 位的数据类型（ long 或 double），在模型中定义了一条相对宽松的规定，允许虚拟机将没有被 volatile 修饰的 64 位数据的读写操作划分为两次 32 位的操作来进行，即允许虚拟机实现选择可以不保证 64 位数据类型的load，store，read，write 这 4 个操作的原子性，即 **long 和 double 的非原子性协定**。 -如果多线程的情况下double 或 long 类型并未声明为 volatile，可能会出现“半个变量”的数值，也就是既非原值，也非修改后的值。 +> 以 32 位 CPU 上执行 long 型变量的写操作为例来说明这个问题，long 型变量是 64 位，在 32 位 CPU 上执行写操作会被拆分成两次写操作（写高 32 位和写低 32 位，如下图所示）。 +> +> ![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/juc/381b657801c48b3399f19d946bad9e28.png) + +如果多线程的情况下 double 或 long 类型并未声明为 volatile，可能会出现“半个变量”的数值，也就是既非原值，也非修改后的值。 虽然 Java 规范允许上面的实现，但商用虚拟机中基本都采用了原子性的操作，因此在日常使用中几乎不会出现读取到“半个变量”的情况，so，这个了解下就行。 @@ -216,17 +294,21 @@ Java 内存模型要求 lock，unlock，read，load，assign，use，store，wri ### 先行发生原则 -先行发生（happens-before）是 Java 内存模型中定义的两项操作之间的偏序关系，**如果操作A 先行发生于操作B，那么A的结果对B可见**。happens-before关系的分析需要分为**单线程和多线程**的情况： +先行发生（happens-before）是 Java 内存模型中定义的两项操作之间的偏序关系，**如果操作 A 先行发生于操作 B，那么 A 的结果对 B 可见**。 + +> Happens-Before 约束了编译器的优化行为，虽允许编译器优化，但是要求编译器优化后一定遵守 Happens-Before 规则。 -- **单线程下的 happens-before** 字节码的先后顺序天然包含happens-before关系：因为单线程内共享一份工作内存，不存在数据一致性的问题。 在程序控制流路径中靠前的字节码 happens-before 靠后的字节码，即靠前的字节码执行完之后操作结果对靠后的字节码可见。然而，这并不意味着前者一定在后者之前执行。实际上，如果后者不依赖前者的运行结果，那么它们可能会被重排序。 -- **多线程下的 happens-before** 多线程由于每个线程有共享变量的副本，如果没有对共享变量做同步处理，线程1更新执行操作A共享变量的值之后，线程2开始执行操作B，此时操作A产生的结果对操作B不一定可见。 +happens-before 关系的分析需要分为**单线程和多线程**的情况： -为了方便程序开发，Java 内存模型实现了下述的先行发生关系： +- **单线程下的 happens-before** 字节码的先后顺序天然包含 happens-before 关系：因为单线程内共享一份工作内存，不存在数据一致性的问题。 在程序控制流路径中靠前的字节码 happens-before 靠后的字节码，即靠前的字节码执行完之后操作结果对靠后的字节码可见。然而，这并不意味着前者一定在后者之前执行。实际上，如果后者不依赖前者的运行结果，那么它们可能会被重排序。 +- **多线程下的 happens-before** 多线程由于每个线程有共享变量的副本，如果没有对共享变量做同步处理，线程 1 更新执行操作 A 共享变量的值之后，线程 2 开始执行操作 B，此时操作 A 产生的结果对操作 B 不一定可见。 + +为了方便程序开发，Java 内存模型实现了下述的先行发生关系（“天然的”先行发生关系，无需任何同步器协助就存在）： - **程序次序规则：** 一个线程内，按照代码顺序，书写在前面的操作先行发生于书写在后面的操作。 -- **管程锁定规则：** 一个unLock操作先行发生于后面对同一个锁的lock操作。 -- **volatile变量规则：** 对一个变量的写操作 happens-before 后面对这个变量的读操作。 -- **传递规则：** 如果操作A 先行发生于操作B，而操作B又先行发生于操作C，则可以得出操作A 先行发生于操作C。 +- **管程锁定规则：** 一个 unLock 操作先行发生于后面对同一个锁的 lock 操作。 +- **volatile变量规则：** 对一个变量的写操作先行发生于后面对这个变量的读操作。 +- **传递规则：** 如果操作 A 先行发生于操作 B，而操作 B 又先行发生于操作 C，则可以得出操作 A 先行发生于操作 C。 - **线程启动规则：** Thread对象的 `start()` 方法先行发生于此线程的每一个动作。 - **线程中断规则：** 对线程 `interrupt()` 方法的调用先行发生于被中断线程的代码检测到中断事件的发生。 - **线程终结规则：** 线程中所有的操作都先行发生于线程的终止检测，我们可以通过`Thread.join()`方法结束、`Thread.isAlive()`的返回值手段检测到线程已经终止执行。 @@ -252,20 +334,20 @@ Load2; Load3; ``` -对于上面的一组 CPU 指令（Store表示写入指令，Load表示读取指令），StoreLoad 屏障之前的 Store 指令无法与StoreLoad 屏障之后的 Load 指令进行交换位置，即**重排序**。但是 StoreLoad 屏障之前和之后的指令是可以互换位置的，即 Store1 可以和 Store2 互换，Load2 可以和 Load3 互换。 +对于上面的一组 CPU 指令（Store表示写入指令，Load表示读取指令），StoreLoad 屏障之前的 Store 指令无法与 StoreLoad 屏障之后的 Load 指令进行交换位置，即**重排序**。但是 StoreLoad 屏障之前和之后的指令是可以互换位置的，即 Store1 可以和 Store2 互换，Load2 可以和 Load3 互换。 常见的 4 种屏障 -- **LoadLoad** 屏障： 对于这样的语句 Load1; LoadLoad; Load2，在Load2及后续读取操作要读取的数据被访问前，保证Load1要读取的数据被读取完毕。 -- **StoreStore** 屏障： 对于这样的语句 Store1; StoreStore; Store2，在Store2及后续写入操作执行前，保证Store1的写入操作对其它处理器可见。 -- **LoadStore** 屏障： 对于这样的语句Load1; LoadStore; Store2，在Store2及后续写入操作被执行前，保证Load1要读取的数据被读取完毕。 -- **StoreLoad** 屏障： 对于这样的语句Store1; StoreLoad; Load2，在Load2及后续所有读取操作执行前，保证Store1的写入对所有处理器可见。它的开销是四种屏障中最大的（冲刷写缓冲器，清空无效化队列）。在大多数处理器的实现中，这个屏障也被称为**全能屏障**，兼具其它三种内存屏障的功能。 +- **LoadLoad** 屏障： 对于这样的语句 Load1; LoadLoad; Load2，在 Load2 及后续读取操作要读取的数据被访问前，保证 Load1 要读取的数据被读取完毕。 +- **StoreStore** 屏障： 对于这样的语句 Store1; StoreStore; Store2，在 Store2 及后续写入操作执行前，保证Store1 的写入操作对其它处理器可见。 +- **LoadStore** 屏障： 对于这样的语句 Load1; LoadStore; Store2，在 Store2 及后续写入操作被执行前，保证Load1 要读取的数据被读取完毕。 +- **StoreLoad** 屏障： 对于这样的语句 Store1; StoreLoad; Load2，在 Load2 及后续所有读取操作执行前，保证Store1 的写入对所有处理器可见。它的开销是四种屏障中最大的（冲刷写缓冲器，清空无效化队列）。在大多数处理器的实现中，这个屏障也被称为**全能屏障**，兼具其它三种内存屏障的功能。 Java 中对内存屏障的使用在一般的代码中不太容易见到，常见的有 volatile 和 synchronized 关键字修饰的代码块，还可以通过 Unsafe 这个类来使用内存屏障。（下一章扯扯这些） -Java 内存模型就是通过定义的这些来解决可见性、原子性和有序性的。 +噢啦，Java 内存模型就是通过以上定义的这些来解决可见性、原子性和有序性问题的。 @@ -273,6 +355,8 @@ Java 内存模型就是通过定义的这些来解决可见性、原子性和有 《深入理解 Java 虚拟机》第二版 +《Java 并发编程实战》 + http://tutorials.jenkov.com/java-concurrency/java-memory-model.html https://juejin.im/post/5bf2977751882505d840321d#heading-5 http://rsim.cs.uiuc.edu/Pubs/popl05.pdf diff --git a/docs/java/JUC/Java-Thread.md b/docs/java/JUC/Java-Thread.md new file mode 100644 index 0000000000..bb6d122178 --- /dev/null +++ b/docs/java/JUC/Java-Thread.md @@ -0,0 +1,98 @@ +# Java 线程 + +在 Java 领域，实现并发程序的主要手段就是多线程。线程是操作系统里的一个概念，虽然各种不同的开发语言如 Java、C# 等都对其进行了封装，但是万变不离操作系统。Java 语言里的线程本质上就是操作系统的线程，它们是一一对应的。 + +## 线程生命周期 + +在操作系统层面，线程也有“生老病死”，专业的说法叫有生命周期。对于有生命周期的事物，要学好它，思路非常简单，只要能搞懂生命周期中各个节点的状态转换机制就可以了。 + +Java 语言中线程共有六种状态，分别是： + +- NEW（初始化状态） + +- RUNNABLE（可运行 / 运行状态） + +- BLOCKED（阻塞状态） + +- WAITING（无时限等待） +- TIMED_WAITING（有时限等待） +- TERMINATED（终止状态） + + + + + +## 创建多少线程 + +### 为什么要使用多线程？ + +使用多线程，本质上就是提升程序性能。不过此刻谈到的性能，可能在你脑海里还是比较笼统的，基本上就是快、快、快，这种无法度量的感性认识很不科学，所以在提升性能之前，首要问题是：如何度量性能。 + +度量性能的指标有很多，但是有两个指标是最核心的，它们就是**延迟**和**吞吐量**。 + +延迟指的是发出请求到收到响应这个过程的时间；延迟越短，意味着程序执行得越快，性能也就越好。 + +吞吐量指的是在单位时间内能处理请求的数量；吞吐量越大，意味着程序能处理的请求越多，性能也就越好。 + +这两个指标内部有一定的联系（同等条件下，延迟越短，吞吐量越大），但是由于它们隶属不同的维度（一个是时间维度，一个是空间维度），并不能互相转换。 + +我们所谓提升性能，从度量的角度，主要是降低延迟，提高吞吐量。这也是我们使用多线程的主要目的。那我们该怎么降低延迟，提高吞吐量呢？这个就要从多线程的应用场景说起了。 + +### 多线程的应用场景 + +要想“降低延迟，提高吞吐量”，对应的方法呢，基本上有两个方向，**一个方向是优化算法，另一个方向是将硬件的性能发挥到极致**。前者属于算法范畴，后者则是和并发编程息息相关了。那计算机主要有哪些硬件呢？主要是两类：一个是 I/O，一个是 CPU。简言之，在并发编程领域，提升性能本质上就是提升硬件的利用率，再具体点来说，就是提升 I/O 的利用率和 CPU 的利用率。 + +估计这个时候你会有个疑问，操作系统不是已经解决了硬件的利用率问题了吗？的确是这样，例如操作系统已经解决了磁盘和网卡的利用率问题，利用中断机制还能避免 CPU 轮询 I/O 状态，也提升了 CPU 的利用率。但是操作系统解决硬件利用率问题的对象往往是单一的硬件设备，而我们的并发程序，往往需要 CPU 和 I/O 设备相互配合工作，也就是说，我们需要解决 CPU 和 I/O 设备综合利用率的问题。关于这个综合利用率的问题，操作系统虽然没有办法完美解决，但是却给我们提供了方案，那就是：多线程。 + +下面我们用一个简单的示例来说明：如何利用多线程来提升 CPU 和 I/O 设备的利用率？假设程序按照 CPU 计算和 I/O 操作交叉执行的方式运行，而且 CPU 计算和 I/O 操作的耗时是 1:1。 + +如下图所示，如果只有一个线程，执行 CPU 计算的时候，I/O 设备空闲；执行 I/O 操作的时候，CPU 空闲，所以 CPU 的利用率和 I/O 设备的利用率都是 50%。 + +![img](https://static001.geekbang.org/resource/image/d1/22/d1d7dfa1d574356cc5cb1019a4b7ca22.png) + +如果有两个线程，如下图所示，当线程 A 执行 CPU 计算的时候，线程 B 执行 I/O 操作；当线程 A 执行 I/O 操作的时候，线程 B 执行 CPU 计算，这样 CPU 的利用率和 I/O 设备的利用率就都达到了 100%。 + +![img](https://static001.geekbang.org/resource/image/68/2c/68a415b31b72844eb81889e9f0eb3f2c.png) + +我们将 CPU 的利用率和 I/O 设备的利用率都提升到了 100%，会对性能产生了哪些影响呢？通过上面的图示，很容易看出：单位时间处理的请求数量翻了一番，也就是说吞吐量提高了 1 倍。此时可以逆向思维一下，**如果 CPU 和 I/O 设备的利用率都很低，那么可以尝试通过增加线程来提高吞吐量**。 + +在单核时代，多线程主要就是用来平衡 CPU 和 I/O 设备的。如果程序只有 CPU 计算，而没有 I/O 操作的话，多线程不但不会提升性能，还会使性能变得更差，原因是增加了线程切换的成本。但是在多核时代，这种纯计算型的程序也可以利用多线程来提升性能。为什么呢？因为利用多核可以降低响应时间。 + +为便于你理解，这里我举个简单的例子说明一下：计算 1+2+… … +100 亿的值，如果在 4 核的 CPU 上利用 4 个线程执行，线程 A 计算[1，25 亿)，线程 B 计算[25 亿，50 亿)，线程 C 计算[50，75 亿)，线程 D 计算[75 亿，100 亿]，之后汇总，那么理论上应该比一个线程计算[1，100 亿]快将近 4 倍，响应时间能够降到 25%。一个线程，对于 4 核的 CPU，CPU 的利用率只有 25%，而 4 个线程，则能够将 CPU 的利用率提高到 100%。 + +![img](https://static001.geekbang.org/resource/image/95/8c/95367d49f55e0dfd099f2749c532098c.png) + +### 创建多少线程合适？ + +创建多少线程合适，要看多线程具体的应用场景。我们的程序一般都是 CPU 计算和 I/O 操作交叉执行的，由于 I/O 设备的速度相对于 CPU 来说都很慢，所以大部分情况下，I/O 操作执行的时间相对于 CPU 计算来说都非常长，这种场景我们一般都称为 I/O 密集型计算；和 I/O 密集型计算相对的就是 CPU 密集型计算了，CPU 密集型计算大部分场景下都是纯 CPU 计算。I/O 密集型程序和 CPU 密集型程序，计算最佳线程数的方法是不同的。 + +下面我们对这两个场景分别说明。 + +对于 CPU 密集型计算，多线程本质上是提升多核 CPU 的利用率，所以对于一个 4 核的 CPU，每个核一个线程，理论上创建 4 个线程就可以了，再多创建线程也只是增加线程切换的成本。 + +所以，**对于 CPU 密集型的计算场景，理论上“线程的数量 =CPU 核数”就是最合适的。不过在工程上，线程的数量一般会设置为“CPU 核数 +1”**，这样的话，当线程因为偶尔的内存页失效或其他原因导致阻塞时，这个额外的线程可以顶上，从而保证 CPU 的利用率。 + +对于 I/O 密集型的计算场景，比如前面我们的例子中，如果 CPU 计算和 I/O 操作的耗时是 1:1，那么 2 个线程是最合适的。如果 CPU 计算和 I/O 操作的耗时是 1:2，那多少个线程合适呢？是 3 个线程，如下图所示：CPU 在 A、B、C 三个线程之间切换，对于线程 A，当 CPU 从 B、C 切换回来时，线程 A 正好执行完 I/O 操作。这样 CPU 和 I/O 设备的利用率都达到了 100%。 + +![img](https://static001.geekbang.org/resource/image/98/cb/98b71b72f01baf5f0968c7c3a2102fcb.png) + +通过上面这个例子，我们会发现，对于 I/O 密集型计算场景，最佳的线程数是与程序中 CPU 计算和 I/O 操作的耗时比相关的，我们可以总结出这样一个公式： + +最佳线程数 =1 +（I/O 耗时 / CPU 耗时） + +我们令 R=I/O 耗时 / CPU 耗时，综合上图，可以这样理解：当线程 A 执行 IO 操作时，另外 R 个线程正好执行完各自的 CPU 计算。这样 CPU 的利用率就达到了 100%。 + +不过上面这个公式是针对单核 CPU 的，至于多核 CPU，也很简单，只需要等比扩大就可以了，计算公式如下： + +最佳线程数 =CPU 核数 * [ 1 +（I/O 耗时 / CPU 耗时）] + +### 总结 + +很多人都知道线程数不是越多越好，但是设置多少是合适的，却又拿不定主意。其实只要把握住一条原则就可以了，这条原则就是将硬件的性能发挥到极致。上面我们针对 CPU 密集型和 I/O 密集型计算场景都给出了理论上的最佳公式，这些公式背后的目标其实就是将硬件的性能发挥到极致。 + +对于 I/O 密集型计算场景，I/O 耗时和 CPU 耗时的比值是一个关键参数，不幸的是这个参数是未知的，而且是动态变化的，所以工程上，我们要估算这个参数，然后做各种不同场景下的压测来验证我们的估计。不过工程上，原则还是将硬件的性能发挥到极致，所以压测时，我们需要重点关注 CPU、I/O 设备的利用率和性能指标（响应时间、吞吐量）之间的关系。 + + + +## 为什么局部变量是线程安全的 + diff --git a/docs/java/JUC/Lock.md b/docs/java/JUC/Lock.md new file mode 100644 index 0000000000..d00f2e8cd2 --- /dev/null +++ b/docs/java/JUC/Lock.md @@ -0,0 +1,40 @@ +在并发编程领域，有两大核心问题：一个是互斥，即同一时刻只允许一个线程访问共享资源；另一个是同步，即线程之间如何通信、协作。 + +Java SDK 并发包通过 Lock 和 Condition 两个接口来实现管程，其中 Lock 用于解决互斥问题，Condition 用于解决同步问题。 + + + +## 可重入锁 + +如果你细心观察，会发现我们创建的锁的具体类名是 ReentrantLock，这个翻译过来叫可重入锁，这个概念前面我们一直没有介绍过。所谓可重入锁，顾名思义，指的是线程可以重复获取同一把锁。例如下面代码中，当线程 T1 执行到 ① 处时，已经获取到了锁 rtl ，当在 ① 处调用 get() 方法时，会在 ② 再次对锁 rtl 执行加锁操作。此时，如果锁 rtl 是可重入的，那么线程 T1 可以再次加锁成功；如果锁 rtl 是不可重入的，那么线程 T1 此时会被阻塞。 + +```java +class X { + private final Lock rtl = + new ReentrantLock(); + int value; + public int get() { + // 获取锁 + rtl.lock(); ② + try { + return value; + } finally { + // 保证锁能释放 + rtl.unlock(); + } + } + public void addOne() { + // 获取锁 + rtl.lock(); + try { + value = 1 + get(); ① + } finally { + // 保证锁能释放 + rtl.unlock(); + } + } +} +``` + +除了可重入锁，可能你还听说过可重入函数，可重入函数怎么理解呢？指的是线程可以重复调用？显然不是，所谓可重入函数，指的是多个线程可以同时调用该函数，每个线程都能得到正确结果；同时在一个线程内支持线程切换，无论被切换多少次，结果都是正确的。多线程可以同时执行，还支持线程切换，这意味着什么呢？线程安全啊。所以，可重入函数是线程安全的。 + diff --git "a/docs/java/JUC/\345\220\204\347\247\215\351\224\201.md" b/docs/java/JUC/Locks.md similarity index 96% rename from "docs/java/JUC/\345\220\204\347\247\215\351\224\201.md" rename to docs/java/JUC/Locks.md index 549d019cae..d25bbb062b 100644 --- "a/docs/java/JUC/\345\220\204\347\247\215\351\224\201.md" +++ b/docs/java/JUC/Locks.md @@ -4,7 +4,7 @@ Java提供了种类丰富的锁，每种锁因其特性的不同，在适当的 Java中往往是按照是否含有某一特性来定义锁，我们通过特性将锁进行分组归类 -![img](https://awps-assets.meituan.net/mit-x/blog-images-bundle-2018b/7f749fc8.png) +![](https://awps-assets.meituan.net/mit-x/blog-images-bundle-2018b/7f749fc8.png) - 公平锁、非公平锁 - 可重入锁（又名递归锁）、非可重入锁 @@ -71,7 +71,7 @@ public class ReentrantLock implements Lock, java.io.Serializable { } ``` -![](../../_images/java/juc/ReentrantLockCode.jpg) +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200929143721.jpg) 两个构造方法对比，可以看出公平锁和非公平锁的区别 @@ -142,17 +142,9 @@ public class Widget { } ``` -在上面的代码中，类中的两个方法都是被内置锁synchronized修饰的，doSomething()方法中调用doOthers()方法。因为内置锁是可重入的，所以同一个线程在调用doOthers()时可以直接获得当前对象的锁，进入doOthers()进行操作。 +在上面的代码中，类中的两个方法都是被内置锁 synchronized 修饰的，doSomething() 方法中调用 doOthers() 方法。因为内置锁是可重入的，所以同一个线程在调用 doOthers() 时可以直接获得当前对象的锁，进入doOthers() 进行操作。 -如果是一个不可重入锁，那么当前线程在调用doOthers()之前需要将执行doSomething()时获取当前对象的锁释放掉，实际上该对象锁已被当前线程所持有，且无法释放。所以此时会出现死锁。 - - - - - - - ------- +如果是一个不可重入锁，那么当前线程在调用 doOthers() 之前需要将执行 doSomething() 时获取当前对象的锁释放掉，实际上该对象锁已被当前线程所持有，且无法释放。所以此时会出现死锁。 @@ -160,7 +152,7 @@ public class Widget { 自旋锁是指尝试获取锁的线程不会立即阻塞，而是采用循环的方式去尝试获取锁，这样的好处是减少线程上下文切换的消耗，缺点是循环会消耗 CPU -![img](https://awps-assets.meituan.net/mit-x/blog-images-bundle-2018b/452a3363.png) +![](https://awps-assets.meituan.net/mit-x/blog-images-bundle-2018b/452a3363.png) ```java /** @@ -462,7 +454,7 @@ public class MCSLock { ## 4. 独占锁（互斥锁/写锁）、共享锁（读锁） -独占锁：指该锁一次只能被一个线程所持有，对 ReentrantLock和 Synchronized 而言都是独占锁 +独占锁：指该锁一次只能被一个线程所持有，对 ReentrantLock 和 Synchronized 而言都是独占锁 共享锁：指该锁可被多个线程所持有 @@ -541,19 +533,13 @@ class MyCache { -## 对各种锁的理解？请手写一个自旋锁 - - - - - ### 无锁 VS 偏向锁 VS 轻量级锁 VS 重量级锁 -这四种锁是指锁的状态，专门针对synchronized的。在介绍这四种锁状态之前还需要介绍一些额外的知识。 +这四种锁是指锁的状态，专门针对 synchronized 的。在介绍这四种锁状态之前还需要介绍一些额外的知识。 -首先为什么Synchronized能实现线程同步？ +首先为什么 Synchronized 能实现线程同步？ 在回答这个问题之前我们需要了解两个重要的概念：“Java对象头”、“Monitor”。 @@ -626,7 +612,7 @@ Monitor是线程私有的数据结构，每一个线程都有一个可用monitor 整体的锁状态升级流程如下： -![img](https://awps-assets.meituan.net/mit-x/blog-images-bundle-2018b/8afdf6f2.png) +![](https://awps-assets.meituan.net/mit-x/blog-images-bundle-2018b/8afdf6f2.png) 综上，偏向锁通过对比Mark Word解决加锁问题，避免执行CAS操作。而轻量级锁是通过用CAS操作和自旋来解决加锁问题，避免线程阻塞和唤醒而影响性能。重量级锁是将除了拥有锁的线程以外的线程都阻塞。 diff --git a/docs/java/JUC/ReadWriteLock.md b/docs/java/JUC/ReadWriteLock.md new file mode 100644 index 0000000000..94e40d8323 --- /dev/null +++ b/docs/java/JUC/ReadWriteLock.md @@ -0,0 +1,179 @@ +今天我们就介绍一种非常普遍的并发场景：读多写少场景。实际工作中，为了优化性能，我们经常会使用缓存，例如缓存元数据、缓存基础数据等，这就是一种典型的读多写少应用场景。缓存之所以能提升性能，一个重要的条件就是缓存的数据一定是读多写少的，例如元数据和基础数据基本上不会发生变化（写少），但是使用它们的地方却很多（读多）。 + +针对读多写少这种并发场景，Java SDK 并发包提供了读写锁——ReadWriteLock，非常容易使用，并且性能很好。 + +### 那什么是读写锁呢？ + +读写锁，并不是 Java 语言特有的，而是一个广为使用的通用技术，所有的读写锁都遵守以下三条基本原则： + +1. 允许多个线程同时读共享变量； +2. 只允许一个线程写共享变量； +3. 如果一个写线程正在执行写操作，此时禁止读线程读共享变量。 + +读写锁与互斥锁的一个重要区别就是读写锁允许多个线程同时读共享变量，而互斥锁是不允许的，这是读写锁在读多写少场景下性能优于互斥锁的关键。但读写锁的写操作是互斥的，当一个线程在写共享变量的时候，是不允许其他线程执行写操作和读操作。 + + + +### 快速实现一个缓存 + +下面我们就实践起来，用 ReadWriteLock 快速实现一个通用的缓存工具类。 + +在下面的代码中，我们声明了一个 Cache 类，其中类型参数 K 代表缓存里 key 的类型，V 代表缓存里 value 的类型。缓存的数据保存在 Cache 类内部的 HashMap 里面，HashMap 不是线程安全的，这里我们使用读写锁 ReadWriteLock 来保证其线程安全。ReadWriteLock 是一个接口，它的实现类是 ReentrantReadWriteLock，通过名字你应该就能判断出来，它是支持可重入的。 + +下面我们通过 rwl 创建了一把读锁和一把写锁。Cache 这个工具类，我们提供了两个方法，一个是读缓存方法 get()，另一个是写缓存方法 put()。读缓存需要用到读锁，读锁的使用和前面我们介绍的 Lock 的使用是相同的，都是 try{}finally{}这个编程范式。写缓存则需要用到写锁，写锁的使用和读锁是类似的。这样看来，读写锁的使用还是非常简单的。 + +```java + +class Cache { + final Map m = + new HashMap<>(); + final ReadWriteLock rwl = + new ReentrantReadWriteLock(); + // 读锁 + final Lock r = rwl.readLock(); + // 写锁 + final Lock w = rwl.writeLock(); + // 读缓存 + V get(K key) { + r.lock(); + try { return m.get(key); } + finally { r.unlock(); } + } + // 写缓存 + V put(K key, V value) { + w.lock(); + try { return m.put(key, v); } + finally { w.unlock(); } + } +} +``` + +如果你曾经使用过缓存的话，你应该知道使用缓存首先要解决缓存数据的初始化问题。缓存数据的初始化，可以采用一次性加载的方式，也可以使用按需加载的方式。 + +如果源头数据的数据量不大，就可以采用一次性加载的方式，这种方式最简单（可参考下图），只需在应用启动的时候把源头数据查询出来，依次调用类似上面示例代码中的 put() 方法就可以了。 + +![img](https://static001.geekbang.org/resource/image/62/1e/627be6e80f96719234007d0a6426771e.png) + +如果源头数据量非常大，那么就需要按需加载了，按需加载也叫懒加载，指的是只有当应用查询缓存，并且数据不在缓存里的时候，才触发加载源头相关数据进缓存的操作。下面你可以结合文中示意图看看如何利用 ReadWriteLock 来实现缓存的按需加载。 + +![img](https://static001.geekbang.org/resource/image/4e/73/4e036a6b38244accfb74a0d18300f073.png) + +### 实现缓存的按需加载 + +文中下面的这段代码实现了按需加载的功能，这里我们假设缓存的源头是数据库。需要注意的是，如果缓存中没有缓存目标对象，那么就需要从数据库中加载，然后写入缓存，写缓存需要用到写锁，所以在代码中的⑤处，我们调用了 w.lock() 来获取写锁。 + +另外，还需要注意的是，在获取写锁之后，我们并没有直接去查询数据库，而是在代码⑥⑦处，重新验证了一次缓存中是否存在，再次验证如果还是不存在，我们才去查询数据库并更新本地缓存。为什么我们要再次验证呢？ + +```java + +class Cache { + final Map m = + new HashMap<>(); + final ReadWriteLock rwl = + new ReentrantReadWriteLock(); + final Lock r = rwl.readLock(); + final Lock w = rwl.writeLock(); + + V get(K key) { + V v = null; + //读缓存 + r.lock(); ① + try { + v = m.get(key); ② + } finally{ + r.unlock(); ③ + } + //缓存中存在，返回 + if(v != null) { ④ + return v; + } + //缓存中不存在，查询数据库 + w.lock(); ⑤ + try { + //再次验证 + //其他线程可能已经查询过数据库 + v = m.get(key); ⑥ + if(v == null){ ⑦ + //查询数据库 + v=省略代码无数 + m.put(key, v); + } + } finally{ + w.unlock(); + } + return v; + } +} +``` + +原因是在高并发的场景下，有可能会有多线程竞争写锁。假设缓存是空的，没有缓存任何东西，如果此时有三个线程 T1、T2 和 T3 同时调用 get() 方法，并且参数 key 也是相同的。那么它们会同时执行到代码⑤处，但此时只有一个线程能够获得写锁，假设是线程 T1，线程 T1 获取写锁之后查询数据库并更新缓存，最终释放写锁。此时线程 T2 和 T3 会再有一个线程能够获取写锁，假设是 T2，如果不采用再次验证的方式，此时 T2 会再次查询数据库。T2 释放写锁之后，T3 也会再次查询一次数据库。而实际上线程 T1 已经把缓存的值设置好了，T2、T3 完全没有必要再次查询数据库。所以，再次验证的方式，能够避免高并发场景下重复查询数据的问题。 + +### 读写锁的升级与降级 + +上面按需加载的示例代码中，在①处获取读锁，在③处释放读锁，那是否可以在②处的下面增加验证缓存并更新缓存的逻辑呢？详细的代码如下。 + +```java + +//读缓存 +r.lock(); ① +try { + v = m.get(key); ② + if (v == null) { + w.lock(); + try { + //再次验证并更新缓存 + //省略详细代码 + } finally{ + w.unlock(); + } + } +} finally{ + r.unlock(); ③ +} +``` + +这样看上去好像是没有问题的，先是获取读锁，然后再升级为写锁，对此还有个专业的名字，叫锁的升级。可惜 ReadWriteLock 并不支持这种升级。在上面的代码示例中，读锁还没有释放，此时获取写锁，会导致写锁永久等待，最终导致相关线程都被阻塞，永远也没有机会被唤醒。锁的升级是不允许的，这个你一定要注意。 + +不过，虽然锁的升级是不允许的，但是锁的降级却是允许的。以下代码来源自 ReentrantReadWriteLock 的官方示例，略做了改动。你会发现在代码①处，获取读锁的时候线程还是持有写锁的，这种锁的降级是支持的。 + +```java + +class CachedData { + Object data; + volatile boolean cacheValid; + final ReadWriteLock rwl = + new ReentrantReadWriteLock(); + // 读锁 + final Lock r = rwl.readLock(); + //写锁 + final Lock w = rwl.writeLock(); + + void processCachedData() { + // 获取读锁 + r.lock(); + if (!cacheValid) { + // 释放读锁，因为不允许读锁的升级 + r.unlock(); + // 获取写锁 + w.lock(); + try { + // 再次检查状态 + if (!cacheValid) { + data = ... + cacheValid = true; + } + // 释放写锁前，降级为读锁 + // 降级是可以的 + r.lock(); ① + } finally { + // 释放写锁 + w.unlock(); + } + } + // 此处仍然持有读锁 + try {use(data);} + finally {r.unlock();} + } +} +``` + diff --git a/docs/java/JUC/StampedLock.md b/docs/java/JUC/StampedLock.md new file mode 100644 index 0000000000..3a434866ee --- /dev/null +++ b/docs/java/JUC/StampedLock.md @@ -0,0 +1,187 @@ +“读写锁允许多个线程同时读共享变量，适用于读多写少的场景”。那在读多写少的场景中，还有没有更快的技术方案呢？还真有，Java 在 1.8 这个版本里，提供了一种叫 StampedLock 的锁，它的性能就比读写锁还要好。 + +### StampedLock 支持的三种锁模式 + +我们先来看看在使用上 StampedLock 和上一篇文章讲的 ReadWriteLock 有哪些区别。 + +ReadWriteLock 支持两种模式：一种是读锁，一种是写锁。 + +而 StampedLock 支持三种模式，分别是：**写锁、悲观读锁和乐观读**。其中，写锁、悲观读锁的语义和 ReadWriteLock 的写锁、读锁的语义非常类似，允许多个线程同时获取悲观读锁，但是只允许一个线程获取写锁，写锁和悲观读锁是互斥的。 + +不同的是：StampedLock 里的写锁和悲观读锁加锁成功之后，都会返回一个 stamp；然后解锁的时候，需要传入这个 stamp。相关的示例代码如下。 + +```java +final StampedLock sl = new StampedLock(); + +// 获取/释放悲观读锁示意代码 +long stamp = sl.readLock(); +try { + //省略业务相关代码 +} finally { + sl.unlockRead(stamp); +} + +// 获取/释放写锁示意代码 +long stamp = sl.writeLock(); +try { + //省略业务相关代码 +} finally { + sl.unlockWrite(stamp); +} +``` + +StampedLock 的性能之所以比 ReadWriteLock 还要好，其关键是 StampedLock 支持乐观读的方式。 + +ReadWriteLock 支持多个线程同时读，但是当多个线程同时读的时候，所有的写操作会被阻塞；而 StampedLock 提供的乐观读，是允许一个线程获取写锁的，也就是说不是所有的写操作都被阻塞。 + +注意这里，我们用的是“乐观读”这个词，而不是“乐观读锁”，是要提醒你，乐观读这个操作是无锁的，所以相比较 ReadWriteLock 的读锁，乐观读的性能更好一些。 + +文中下面这段代码是出自 Java SDK 官方示例，并略做了修改。在 distanceFromOrigin() 这个方法中，首先通过调用 tryOptimisticRead() 获取了一个 stamp，这里的 tryOptimisticRead() 就是我们前面提到的乐观读。之后将共享变量 x 和 y 读入方法的局部变量中，不过需要注意的是，由于 tryOptimisticRead() 是无锁的，所以共享变量 x 和 y 读入方法局部变量时，x 和 y 有可能被其他线程修改了。因此最后读完之后，还需要再次验证一下是否存在写操作，这个验证操作是通过调用 validate(stamp) 来实现的。 + +```java + +class Point { + private int x, y; + final StampedLock sl = + new StampedLock(); + //计算到原点的距离 + int distanceFromOrigin() { + // 乐观读 + long stamp = + sl.tryOptimisticRead(); + // 读入局部变量， + // 读的过程数据可能被修改 + int curX = x, curY = y; + //判断执行读操作期间， + //是否存在写操作，如果存在， + //则sl.validate返回false + if (!sl.validate(stamp)){ + // 升级为悲观读锁 + stamp = sl.readLock(); + try { + curX = x; + curY = y; + } finally { + //释放悲观读锁 + sl.unlockRead(stamp); + } + } + return Math.sqrt( + curX * curX + curY * curY); + } +} +``` + +在上面这个代码示例中，如果执行乐观读操作的期间，存在写操作，会把乐观读升级为悲观读锁。这个做法挺合理的，否则你就需要在一个循环里反复执行乐观读，直到执行乐观读操作的期间没有写操作（只有这样才能保证 x 和 y 的正确性和一致性），而循环读会浪费大量的 CPU。升级为悲观读锁，代码简练且不易出错，建议你在具体实践时也采用这样的方法。 + + + +### 进一步理解乐观读 + +如果你曾经用过数据库的乐观锁，可能会发现 StampedLock 的乐观读和数据库的乐观锁有异曲同工之妙。的确是这样的，就拿我个人来说，我是先接触的数据库里的乐观锁，然后才接触的 StampedLock，我就觉得我前期数据库里乐观锁的学习对于后面理解 StampedLock 的乐观读有很大帮助，所以这里有必要再介绍一下数据库里的乐观锁。还记得我第一次使用数据库乐观锁的场景是这样的：在 ERP 的生产模块里，会有多个人通过 ERP 系统提供的 UI 同时修改同一条生产订单，那如何保证生产订单数据是并发安全的呢？我采用的方案就是乐观锁。乐观锁的实现很简单，在生产订单的表 product_doc 里增加了一个数值型版本号字段 version，每次更新 product_doc 这个表的时候，都将 version 字段加 1。生产订单的 UI 在展示的时候，需要查询数据库，此时将这个 version 字段和其他业务字段一起返回给生产订单 UI。假设用户查询的生产订单的 id=777，那么 SQL 语句类似下面这样： + +```sql +select id，... ，version +from product_doc +where id=777 +``` + +用户在生产订单 UI 执行保存操作的时候，后台利用下面的 SQL 语句更新生产订单，此处我们假设该条生产订单的 version=9。 + +```sql +update product_doc +set version=version+1，... +where id=777 and version=9 +``` + +如果这条 SQL 语句执行成功并且返回的条数等于 1，那么说明从生产订单 UI 执行查询操作到执行保存操作期间，没有其他人修改过这条数据。因为如果这期间其他人修改过这条数据，那么版本号字段一定会大于 9。 + +你会发现数据库里的乐观锁，查询的时候需要把 version 字段查出来，更新的时候要利用 version 字段做验证。这个 version 字段就类似于 StampedLock 里面的 stamp。这样对比着看，相信你会更容易理解 StampedLock 里乐观读的用法。 + + + +### StampedLock 使用注意事项 + +对于读多写少的场景 StampedLock 性能很好，简单的应用场景基本上可以替代 ReadWriteLock，但是 StampedLock 的功能仅仅是 ReadWriteLock 的子集，在使用的时候，还是有几个地方需要注意一下。 + +StampedLock 在命名上并没有增加 Reentrant，想必你已经猜测到 StampedLock 应该是不可重入的。事实上，的确是这样的，StampedLock 不支持重入。这个是在使用中必须要特别注意的。 + +另外，StampedLock 的悲观读锁、写锁都不支持条件变量，这个也需要你注意。 + +还有一点需要特别注意，那就是：如果线程阻塞在 StampedLock 的 readLock() 或者 writeLock() 上时，此时调用该阻塞线程的 interrupt() 方法，会导致 CPU 飙升。例如下面的代码中，线程 T1 获取写锁之后将自己阻塞，线程 T2 尝试获取悲观读锁，也会阻塞；如果此时调用线程 T2 的 interrupt() 方法来中断线程 T2 的话，你会发现线程 T2 所在 CPU 会飙升到 100%。 + +```java + +final StampedLock lock + = new StampedLock(); +Thread T1 = new Thread(()->{ + // 获取写锁 + lock.writeLock(); + // 永远阻塞在此处，不释放写锁 + LockSupport.park(); +}); +T1.start(); +// 保证T1获取写锁 +Thread.sleep(100); +Thread T2 = new Thread(()-> + //阻塞在悲观读锁 + lock.readLock() +); +T2.start(); +// 保证T2阻塞在读锁 +Thread.sleep(100); +//中断线程T2 +//会导致线程T2所在CPU飙升 +T2.interrupt(); +T2.join(); +``` + +所以，使用 StampedLock 一定不要调用中断操作，如果需要支持中断功能，一定使用可中断的悲观读锁 readLockInterruptibly() 和写锁 writeLockInterruptibly()。这个规则一定要记清楚。 + + + +### 总结 + +StampedLock 的使用看上去有点复杂，但是如果你能理解乐观锁背后的原理，使用起来还是比较流畅的。建议你认真揣摩 Java 的官方示例，这个示例基本上就是一个最佳实践。我们把 Java 官方示例精简后，形成下面的代码模板，建议你在实际工作中尽量按照这个模板来使用 StampedLock。 + +StampedLock 读模板： + +```java + +final StampedLock sl = + new StampedLock(); + +// 乐观读 +long stamp = + sl.tryOptimisticRead(); +// 读入方法局部变量 +...... +// 校验stamp +if (!sl.validate(stamp)){ + // 升级为悲观读锁 + stamp = sl.readLock(); + try { + // 读入方法局部变量 + ..... + } finally { + //释放悲观读锁 + sl.unlockRead(stamp); + } +} +//使用方法局部变量执行业务操作 +...... +``` + +StampedLock 写模板 + +```java + +long stamp = sl.writeLock(); +try { + // 写共享变量 + ...... +} finally { + sl.unlockWrite(stamp); +} +``` + diff --git a/docs/java/JUC/Thread-Pool.md b/docs/java/JUC/Thread-Pool.md index dc67e22bc2..b85d35542b 100644 --- a/docs/java/JUC/Thread-Pool.md +++ b/docs/java/JUC/Thread-Pool.md @@ -1,15 +1,18 @@ +# 线程池 + > 线程池的工作原理，几个重要参数，给了具体几个参数分析线程池会怎么做，阻塞队列的作用是什么？ > > 说说几种常见的线程池及使用场景? > -> 线程池的构造类的方法的 5 个参数的具体意义是什么 +> 线程池的构造类的方法的 5 个参数的具体意义是什么？ > -> 按线程池内部机制，当提交新任务时，有哪些异常要考虑 +> 按线程池内部机制，当提交新任务时，有哪些异常要考虑？ > > 单机上一个线程池正在处理服务，如果忽然断电怎么办（正在处理和阻塞队列里的请求怎么处理）？ > > 生产上如何合理设置参数？ > +> 说说线程池的拒绝策略？ @@ -65,19 +68,78 @@ ThreadPoolExecutor 实现的顶层接口是 Executor，顶层接口 Executor 提 1. 扩充执行任务的能力，补充可以为一个或一批异步任务生成 Future 的方法； 2. 提供了管控线程池的方法，比如停止线程池的运行。 -AbstractExecutorService 则是上层的抽象类，将执行任务的流程串联了起来，保证下层的实现只需关注一个执行任务的方法即可。最下层的实现类 ThreadPoolExecutor 实现最复杂的运行部分，ThreadPoolExecutor 将会一方面维护自身的生命周期，另一方面同时管理线程和任务，使两者良好的结合从而执行并行任务。 +`AbstractExecutorService` 则是上层的抽象类，将执行任务的流程串联了起来，保证下层的实现只需关注一个执行任务的方法即可。最下层的实现类 `ThreadPoolExecutor` 实现最复杂的运行部分，`ThreadPoolExecutor` 将会一方面维护自身的生命周期，另一方面同时管理线程和任务，使两者良好的结合从而执行并行任务。 +我们来了解下 **ThreadPoolExecutor** 的构造函数。 +### 线程池的几个重要参数 -### Hello ThreadPool +从使用中我们可以看到，常用的构造线程池方法其实最后都是通过 **ThreadPoolExecutor** 实例来创建的，且该构造器有 7 大参数。 -常见的线程池的使用方式 +```java +public ThreadPoolExecutor(int corePoolSize, + int maximumPoolSize, + long keepAliveTime, + TimeUnit unit, + BlockingQueue workQueue, + ThreadFactory threadFactory, + RejectedExecutionHandler handler) { + if (corePoolSize < 0 || + maximumPoolSize <= 0 || + maximumPoolSize < corePoolSize || + keepAliveTime < 0) + throw new IllegalArgumentException(); + if (workQueue == null || threadFactory == null || handler == null) + throw new NullPointerException(); + this.acc = System.getSecurityManager() == null ? + null : + AccessController.getContext(); + this.corePoolSize = corePoolSize; + this.maximumPoolSize = maximumPoolSize; + this.workQueue = workQueue; + this.keepAliveTime = unit.toNanos(keepAliveTime); + this.threadFactory = threadFactory; + this.handler = handler; + } +``` - `Executors`，提供了一系列静态工厂方法用于创建各种线程池，工具类，类似于我们常用的 `Arrays`、`Collections` +- **corePoolSize：** 线程池中的常驻核心线程数（线程池保有的最小线程数） + + - 创建线程池后，当有请求任务进来之后，就会安排池中的线程去执行请求任务，近似理解为近日当值线程 + - 当线程池中的线程数目达到 corePoolSize 后，就会把到达的任务放到缓存队列中 + +- **maximumPoolSize：** 线程池最大线程数大小，该值必须大于等于 1 + +- **keepAliveTime：** 线程池中非核心线程空闲后的存活时间（表示线程没有任务执行时最多保持多久时间会终止） + + 当前线程池数量超过 corePoolSize 时，当空闲时间达到 keepAliveTime 值时，非核心线程会被销毁直到只剩下 corePoolSize 个线程为止 + +- **unit：** keepAliveTime 的时间单位 + +- **workQueue：** 存放任务的阻塞队列，被提交但尚未被执行的任务 + +- **threadFactory：** 用于设置创建线程的工厂，可以给创建的线程设置有意义的名字，可方便排查问题 + +- **handler：** 拒绝策略，表示当队列满了且工作线程大于等于线程池的最大线程数（maximumPoolSize）时如何来拒绝请求执行的线程的策略，主要有四种类型。 + + 等待队列也已经满了，再也塞不下新任务。同时，线程池中的 max 线程也达到了，无法继续为新任务服务，这时候我们就需要拒绝策略合理的处理这个问题了 + + - AbortPolicy ——直接抛出 RegectedExcutionException 异常阻止系统正常进行，**默认策略** + - DiscardPolicy ——直接丢弃任务，不予任何处理也不抛出异常，如果允许任务丢失，这是最好的一种方案 + - DiscardOldestPolicy ——抛弃队列中等待最久的任务，然后把当前任务加入队列中尝试再次提交当前任务 + - CallerRunsPolicy ——交给线程池调用所在的线程进行处理，“调用者运行”的一种调节机制，该策略既不会抛弃任务，也不会抛出异常，而是将某些任务回退到调用者，从而降低新任务的流量 + + 以上内置拒绝策略均实现了 RejectExcutionHandler 接口 + + + +### Executors + +常见的线程池的使用方式 `Executors`，它提供了一系列静态工厂方法用于创建各种线程池，工具类，类似于我们常用的 `Arrays`、`Collections` #### newFixedThreadPool -``` +```java public static ExecutorService newFixedThreadPool(int nThreads) { return new ThreadPoolExecutor(nThreads, nThreads, 0L, TimeUnit.MILLISECONDS, @@ -86,7 +148,7 @@ public static ExecutorService newFixedThreadPool(int nThreads) { ``` - 创建一个指定工作线程数量的线程池。每当提交一个任务就创建一个工作线程，如果工作线程数量达到线程池初始的最大数，则将提交的任务存入到池队列中，可控制线程的最大并发数。 -- newFixedThreadPool 创建的线程池 corePoolSize 和 MaximumPoolSize 值是相等的，它使用的 **LinkedBolckingQueue** +- `newFixedThreadPool` 创建的线程池 corePoolSize 和 MaximumPoolSize 值是相等的，它使用的 **LinkedBolckingQueue** - 这种方式即使线程池中没有可运行任务时，它也不会释放工作线程，还会占用一定的系统资源。 @@ -102,10 +164,10 @@ public static ExecutorService newSingleThreadExecutor() { } ``` -- 创建一个单线程化的Executor，它只会用唯一的工作线程来执行任务，保证所有任务按照指定顺序执行。如果这个线程异常结束，会有另一个取代它，保证顺序执行。 +- 创建一个单线程化的 Executor，它只会用唯一的工作线程来执行任务，保证所有任务按照指定顺序执行。如果这个线程异常结束，会有另一个取代它，保证顺序执行。 - newSingleThreadExecutor 将 corePoolSize 和 maximumPoolSize 都设置为 1，它使用的 **LinkedBlockingQueue** -- + #### newCachedThreadPool @@ -134,7 +196,7 @@ public ScheduledThreadPoolExecutor(int corePoolSize, } ``` -- 创建一个定长的线程池，而且支持定时的以及周期性的任务执行，支持定时及周期性任务执行。 +- 创建一个定长的线程池，而且支持定时的以及周期性的任务执行。 @@ -152,113 +214,82 @@ public static ExecutorService newWorkStealingPool() { - Java8 新特性，使用目前机器上可用的处理器作为它的并行级别 - 可以通过参数 parallelism 指定并行数量 - - -## 四、线程池的几个重要参数 - -从使用中我们可以看到，常用的构造线程池方法其实最后都是通过 **ThreadPoolExecutor** 实例来创建的，且该构造器有 7 大参数。 - -```java -public ThreadPoolExecutor(int corePoolSize, - int maximumPoolSize, - long keepAliveTime, - TimeUnit unit, - BlockingQueue workQueue, - ThreadFactory threadFactory, - RejectedExecutionHandler handler) { - if (corePoolSize < 0 || - maximumPoolSize <= 0 || - maximumPoolSize < corePoolSize || - keepAliveTime < 0) - throw new IllegalArgumentException(); - if (workQueue == null || threadFactory == null || handler == null) - throw new NullPointerException(); - this.acc = System.getSecurityManager() == null ? - null : - AccessController.getContext(); - this.corePoolSize = corePoolSize; - this.maximumPoolSize = maximumPoolSize; - this.workQueue = workQueue; - this.keepAliveTime = unit.toNanos(keepAliveTime); - this.threadFactory = threadFactory; - this.handler = handler; - } -``` - -- **corePoolSize：** 线程池中的常驻核心线程数 - - 创建线程池后，当有请求任务进来之后，就会安排池中的线程去执行请求任务，近似理解为近日当值线程 - - 当线程池中的线程数目达到corePoolSize后，就会把到达的任务放到缓存队列中 - -- **maximumPoolSize：** 线程池最大线程数大小，该值必须大于等于 1 - -- **keepAliveTime：** 线程池中非核心线程空闲的存活时间 - -- 当前线程池数量超过 corePoolSize 时，当空闲时间达到 keepAliveTime 值时，非核心线程会被销毁直到只剩下 corePoolSize 个线程为止 - -- **unit：** keepAliveTime 的时间单位 - -- **workQueue：** 存放任务的阻塞队列，被提交但尚未被执行的任务 - -- **threadFactory：** 用于设置创建线程的工厂，可以给创建的线程设置有意义的名字，可方便排查问题 - -- **handler：** 拒绝策略，表示当队列满了且工作线程大于等于线程池的最大线程数（maximumPoolSize）时如何来拒绝请求执行的线程的策略，主要有四种类型。 - - 等待队列也已经满了，再也塞不下新任务。同时，线程池中的 max 线程也达到了，无法继续为新任务服务，这时候我们就需要拒绝策略合理的处理这个问题了。 - - - AbortPolicy 直接抛出RegectedExcutionException 异常阻止系统正常进行，**默认策略** - - DiscardPolicy 直接丢弃任务，不予任何处理也不抛出异常，如果允许任务丢失，这是最好的一种方案 - - DiscardOldestPolicy 抛弃队列中等待最久的任务，然后把当前任务加入队列中尝试再次提交当前任务 - - CallerRunsPolicy 交给线程池调用所在的线程进行处理，“调用者运行”的一种调节机制，该策略既不会抛弃任务，也不会抛出异常，而是将某些任务回退到调用者，从而降低新任务的流量 - - 以上内置拒绝策略均实现了 RejectExcutionHandler 接口 - - +> 目前大厂的编码规范中基本上都不建议使用 Executors，不建议使用 Executors 的最重要的原因是：Executors 提供的很多方法默认使用的都是无界的 LinkedBlockingQueue，高负载情境下，无界队列很容易导致 OOM，而 OOM 会导致所有请求都无法处理，这是致命问题。所以强烈建议使用有界队列。 ## 五、线程池的底层工作原理 ThreadPoolExecutor 是如何运行，如何同时维护线程和执行任务的呢？其运行机制如下图所示： -![img](https://tva1.sinaimg.cn/large/00831rSTly1gdl1arsqkbj30u50d8dgz.jpg) +![](https://tva1.sinaimg.cn/large/00831rSTly1gdl1arsqkbj30u50d8dgz.jpg) + +线程池在内部实际上构建了一个生产者消费者模型，将线程和任务两者解耦，并不直接关联，从而良好的缓冲任务，复用线程。线程池的运行主要分成两部分：**任务管理、线程管理**。 -线程池在内部实际上构建了一个生产者消费者模型，将线程和任务两者解耦，并不直接关联，从而良好的缓冲任务，复用线程。线程池的运行主要分成两部分：**任务管理、线程管理**。任务管理部分充当生产者的角色，当任务提交后，线程池会判断该任务后续的流转： +**任务管理部分充当生产者的角色**，当任务提交后，线程池会判断该任务后续的流转： - 直接申请线程执行该任务； - 缓冲到队列中等待线程执行； - 拒绝该任务。 -线程管理部分是消费者，它们被统一维护在线程池内，根据任务请求进行线程的分配，当线程执行完任务后则会继续获取新的任务去执行，最终当线程获取不到任务的时候，线程就会被回收。 - - - -说说线程池的工作原理？ +**线程管理部分是消费者**，它们被统一维护在线程池内，根据任务请求进行线程的分配，当线程执行完任务后则会继续获取新的任务去执行，最终当线程获取不到任务的时候，线程就会被回收。 -1. 在创建线程池后，等待提交过来的任务请求 - -2. 当调用 execute() 方法添加一个请求任务时，线程池会做如下判断： - - - 如果正在运行的线程数量小于 corePoolSize，那么马上创建线程运行这个任务 - - 如果正在运行的线程数量大于或等于 corePoolSize，那么将这个任务**放入队列** - - 如果这个时候队列满了且正在运行的线程数量还小于 maximumPoolSize，那么创建非核心线程立刻运行这个任务 - - 如果队列满了且正在运行的线程数量大于或等于 maximumPoolSize，那么线程池**会启动饱和拒绝策略来执行** - -3. 当一个线程完成任务时，它会从队列中取下一个任务来执行 - -4. 当一个线程无事可做超过一定的时间（keepAliveTime）时，线程池会判断： - - - 如果当前运行的线程数大于 corePoolSize，那么这个线程就被停掉 - - 所以线程池的所有任务完成后它**最终会收缩到 corePoolSize 的大小** - - - -用生活中的例子类比这一过程 +> 我们自己实现一个简易的线程池 +> +> ```java +> +> //简化的线程池，仅用来说明工作原理 +> class MyThreadPool{ +> //利用阻塞队列实现生产者-消费者模式 +> BlockingQueue workQueue; +> //保存内部工作线程 +> List threads +> = new ArrayList<>(); +> // 构造方法 +> MyThreadPool(int poolSize, +> BlockingQueue workQueue){ +> this.workQueue = workQueue; +> // 创建工作线程 +> for(int idx=0; idx WorkerThread work = new WorkerThread(); +> work.start(); +> threads.add(work); +> } +> } +> // 提交任务 +> void execute(Runnable command){ +> workQueue.put(command); +> } +> // 工作线程负责消费任务，并执行任务 +> class WorkerThread extends Thread{ +> public void run() { +> //循环取任务并执行 +> while(true){ ① +> Runnable task = workQueue.take(); +> task.run(); +> } +> } +> } +> } +> +> /** 下面是使用示例 **/ +> // 创建有界阻塞队列 +> BlockingQueue workQueue = +> new LinkedBlockingQueue<>(2); +> // 创建线程池 +> MyThreadPool pool = new MyThreadPool( +> 10, workQueue); +> // 提交任务 +> pool.execute(()->{ +> System.out.println("hello"); +> }); +> ``` +> +> -- 核心线程比作公司正式员工 -- 非核心线程比作外包员工 -- 阻塞队列比作需求池 -- 提交任务比作提需求 -- 公司最多可聘请或可容纳的员工数量比作最大线程数量 +接下来，我们会按照以下三个部分去详细讲解线程池运行机制： -当产品提了新需求，正式员工（核心线程）先接需求（执行任务）。如果正式员工都有需求在做（核心线程数已满），产品就把需求先放需求池（阻塞队列）。如果需求池(阻塞队列)也满了，但是这时候产品继续提需求，这个时候就请外包（非核心线程）来做。如果所有员工（最大线程数也满了）都有需求在做了，那就执行拒绝策略，不接新需求了。如果外包员工把需求做完了，且过了一定时间（keepAliveTime），公司就不会再与外包员工续约了，外包员工这时就会离开公司。 +1. 线程池如何维护自身状态。 +2. 线程池如何管理任务。 +3. 线程池如何管理线程。 @@ -300,7 +331,7 @@ private static final int TERMINATED = 3 << COUNT_BITS; 转化关系图： -![](https://tva1.sinaimg.cn/large/00831rSTly1gdlc42wt0dj30ok0r4tb9.jpg) +![线程池生命周期](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200928113108.png) ### 5.2 任务执行机制 @@ -311,10 +342,10 @@ private static final int TERMINATED = 3 << COUNT_BITS; 首先，所有任务的调度都是由 execute 方法完成的，这部分完成的工作是：检查现在线程池的运行状态、运行线程数、运行策略，决定接下来执行的流程，是直接申请线程执行，或是缓冲到队列中执行，亦或是直接拒绝该任务。其执行过程如下： 1. 首先检测线程池运行状态，如果不是 RUNNING，则直接拒绝，线程池要保证在 RUNNING 的状态下执行任务。 -2. 如果 workerCount < corePoolSize，则创建并启动一个线程来执行新提交的任务。 -3. 如果 workerCount >= corePoolSize，且线程池内的阻塞队列未满，则将任务添加到该阻塞队列中。 -4. 如果 workerCount >= corePoolSize && workerCount < maximumPoolSize，且线程池内的阻塞队列已满，则创建并启动一个线程来执行新提交的任务。 -5. 如果 workerCount >= maximumPoolSize，并且线程池内的阻塞队列已满， 则根据拒绝策略来处理该任务, 默认的处理方式是直接抛异常。 +2. 如果 `workerCount < corePoolSize`，则创建并启动一个线程来执行新提交的任务。 +3. 如果 `workerCount >= corePoolSize`，且线程池内的阻塞队列未满，则将任务添加到该阻塞队列中。 +4. 如果` workerCount >= corePoolSize && workerCount < maximumPoolSize`，且线程池内的阻塞队列已满，则创建并启动一个线程来执行新提交的任务。 +5. 如果 `workerCount >= maximumPoolSize`，并且线程池内的阻塞队列已满， 则根据拒绝策略来处理该任务，默认的处理方式是直接抛异常。 ```java public void execute(Runnable command) { @@ -342,15 +373,53 @@ public void execute(Runnable command) { ![](https://tva1.sinaimg.cn/large/00831rSTly1gdlddysj52j30g80eljs8.jpg) +> 面试官：**说说线程池的工作原理？** +> +> 1. 在创建线程池后，等待提交过来的任务请求 +> +> 2. 当调用 execute() 方法添加一个请求任务时，线程池会做如下判断： +> +> - 如果正在运行的线程数量小于 corePoolSize，那么马上创建线程运行这个任务 +> - 如果正在运行的线程数量大于或等于 corePoolSize，那么将这个任务**放入队列** +> - 如果这个时候队列满了且正在运行的线程数量还小于 maximumPoolSize，那么创建非核心线程立刻运行这个任务 +> - 如果队列满了且正在运行的线程数量大于或等于 maximumPoolSize，那么线程池**会启动饱和拒绝策略来执行** +> +> 3. 当一个线程完成任务时，它会从队列中取下一个任务来执行 +> +> 4. 当一个线程无事可做超过一定的时间（keepAliveTime）时，线程池会判断： +> +> - 如果当前运行的线程数大于 corePoolSize，那么这个线程就被停掉 +> - 所以线程池的所有任务完成后它**最终会收缩到 corePoolSize 的大小** +> +> +> +> 用生活中的例子类比这一过程 +> +> - 核心线程比作公司正式员工 +> - 非核心线程比作外包员工 +> - 阻塞队列比作需求池 +> - 提交任务比作提需求 +> - 公司最多可聘请或可容纳的员工数量比作最大线程数量 +> +> 当产品提了新需求，正式员工（核心线程）先接需求（执行任务）。 +> +> 如果正式员工都有需求在做（核心线程数已满），产品就把需求先放需求池（阻塞队列）。 +> +> 如果需求池（阻塞队列）也满了，但是这时候产品继续提需求，这个时候就请外包（非核心线程）来做。 +> +> 如果所有员工（最大线程数也满了）都有需求在做了，那就执行拒绝策略，不接新需求了。 +> +> 如果外包员工把需求做完了，且过了一定时间（keepAliveTime），公司就不会再与外包员工续约了，外包员工这时就会离开公司。 + #### 任务缓冲 -任务缓冲模块是线程池能够管理任务的核心部分。线程池的本质是对任务和线程的管理，而做到这一点最关键的思想就是将任务和线程两者解耦，不让两者直接关联，才可以做后续的分配工作。线程池中是以生产者消费者模式，通过一个阻塞队列来实现的。阻塞队列缓存任务，工作线程从阻塞队列中获取任务。 +任务缓冲模块是线程池能够管理任务的核心部分。线程池的本质是对任务和线程的管理，而做到这一点最关键的思想就是**将任务和线程两者解耦**，不让两者直接关联，才可以做后续的分配工作。线程池中是以生产者消费者模式，通过一个阻塞队列来实现的。阻塞队列缓存任务，工作线程从阻塞队列中获取任务。 -阻塞队列(BlockingQueue)是一个支持两个附加操作的队列。这两个附加的操作是：在队列为空时，获取元素的线程会等待队列变为非空。当队列满时，存储元素的线程会等待队列可用。阻塞队列常用于生产者和消费者的场景，生产者是往队列里添加元素的线程，消费者是从队列里拿元素的线程。阻塞队列就是生产者存放元素的容器，而消费者也只从容器里拿元素。 +阻塞队列（BlockingQueue）是一个支持两个附加操作的队列。这两个附加的操作是：在队列为空时，获取元素的线程会等待队列变为非空。当队列满时，存储元素的线程会等待队列可用。阻塞队列常用于生产者和消费者的场景，生产者是往队列里添加元素的线程，消费者是从队列里拿元素的线程。阻塞队列就是生产者存放元素的容器，而消费者也只从容器里拿元素。 -下图中展示了线程1往阻塞队列中添加元素，而线程2从阻塞队列中移除元素： +下图中展示了线程 1 往阻塞队列中添加元素，而线程 2 从阻塞队列中移除元素： ![](https://tva1.sinaimg.cn/large/00831rSTly1gdldf026r1j309o033wef.jpg) @@ -358,7 +427,7 @@ public void execute(Runnable command) { 使用不同的队列可以实现不一样的任务存取策略。在这里，我们可以再介绍下阻塞队列的成员： -![](https://tva1.sinaimg.cn/large/00831rSTly1gdldg06a43j31b20l848n.jpg) +![img](/Users/apple/picBed/others/blocking-queue.png) #### 任务申请 @@ -412,7 +481,7 @@ getTask 这部分进行了多次判断，为的是控制线程的数量，使其 #### 任务拒绝 -任务拒绝模块是线程池的保护部分，线程池有一个最大的容量，当线程池的任务缓存队列已满，并且线程池中的线程数目达到 maximumPoolSize 时，就需要拒绝掉该任务，采取任务拒绝策略，保护线程池。 +任务拒绝模块是线程池的保护部分，线程池有一个最大的容量，当线程池的任务缓存队列已满，并且线程池中的线程数目达到 `maximumPoolSize` 时，就需要拒绝掉该任务，采取任务拒绝策略，保护线程池。 拒绝策略是一个接口，其设计如下： @@ -424,13 +493,13 @@ public interface RejectedExecutionHandler { 用户可以通过实现这个接口去定制拒绝策略，也可以选择 JDK 提供的四种已有拒绝策略，其特点如下： -![](https://tva1.sinaimg.cn/large/00831rSTly1gdldgt1vd5j30vm0r6jsz.jpg) +![](../../_images/java/juc/thread-pool-reject.png) ### 5.3 工作线程管理 #### Worker线程 -线程池为了掌握线程的状态并维护线程的生命周期，设计了线程池内的工作线程Worker。我们来看一下它的部分代码： +线程池为了掌握线程的状态并维护线程的生命周期，设计了线程池内的工作线程 Worker。我们来看一下它的部分代码： ```Java private final class Worker extends AbstractQueuedSynchronizer implements Runnable{ @@ -440,18 +509,18 @@ private final class Worker extends AbstractQueuedSynchronizer implements Runnabl } ``` -Worker 这个工作线程，实现了 Runnable 接口，并持有一个线程 thread，一个初始化的任务 firstTask。thread是在调用构造方法时通过 ThreadFactory 来创建的线程，可以用来执行任务；firstTask 用它来保存传入的第一个任务，这个任务可以有也可以为 null。如果这个值是非空的，那么线程就会在启动初期立即执行这个任务，也就对应核心线程创建时的情况；如果这个值是null，那么就需要创建一个线程去执行任务列表（workQueue）中的任务，也就是非核心线程的创建。 +Worker 这个工作线程，实现了 Runnable 接口，并持有一个线程 thread，一个初始化的任务 firstTask。thread是在调用构造方法时通过 ThreadFactory 来创建的线程，可以用来执行任务；firstTask 用它来保存传入的第一个任务，这个任务可以有也可以为 null。如果这个值是非空的，那么线程就会在启动初期立即执行这个任务，也就对应核心线程创建时的情况；如果这个值是 null，那么就需要创建一个线程去执行任务列表（workQueue）中的任务，也就是非核心线程的创建。 ![](https://tva1.sinaimg.cn/large/00831rSTly1gdldh6s3ahj30yc0d675z.jpg) -线程池需要管理线程的生命周期，需要在线程长时间不运行的时候进行回收。线程池使用一张 Hash表去持有线程的引用，这样可以通过添加引用、移除引用这样的操作来控制线程的生命周期。这个时候重要的就是如何判断线程是否在运行。 +线程池需要管理线程的生命周期，需要在线程长时间不运行的时候进行回收。线程池使用一张 Hash 表去持有线程的引用，这样可以通过添加引用、移除引用这样的操作来控制线程的生命周期。这个时候重要的就是如何判断线程是否在运行。 -Worker是通过继承 `AbstractQueuedSynchronizer`，使用AQS来实现独占锁这个功能。没有使用可重入锁ReentrantLock，而是使用AQS，为的就是实现不可重入的特性去反应线程现在的执行状态。 +Worker是通过继承 `AbstractQueuedSynchronizer`，使用 AQS 来实现独占锁这个功能。没有使用可重入锁ReentrantLock，而是使用 AQS，为的就是实现不可重入的特性去反应线程现在的执行状态。 -1. lock方法一旦获取了独占锁，表示当前线程正在执行任务中。 +1. lock 方法一旦获取了独占锁，表示当前线程正在执行任务中。 2. 如果正在执行任务，则不应该中断线程。 3. 如果该线程现在不是独占锁的状态，也就是空闲的状态，说明它没有在处理任务，这时可以对该线程进行中断。 -4. 线程池在执行 shutdown 方法或 tryTerminate 方法时会调用 interruptIdleWorkers 方法来中断空闲的线程，interruptIdleWorkers 方法会使用 tryLock 方法来判断线程池中的线程是否是空闲状态；如果线程是空闲状态则可以安全回收。 +4. 线程池在执行 `shutdown` 方法或 `tryTerminate` 方法时会调用 `interruptIdleWorkers` 方法来中断空闲的线程，`interruptIdleWorkers` 方法会使用 `tryLock` 方法来判断线程池中的线程是否是空闲状态；如果线程是空闲状态则可以安全回收。 在线程回收过程中就使用到了这种特性，回收过程如下图所示： @@ -461,7 +530,11 @@ Worker是通过继承 `AbstractQueuedSynchronizer`，使用AQS来实现独占锁 #### Worker线程增加 -增加线程是通过线程池中的 addWorker 方法，该方法的功能就是增加一个线程，该方法不考虑线程池是在哪个阶段增加的该线程，这个分配线程的策略是在上个步骤完成的，该步骤仅仅完成增加线程，并使它运行，最后返回是否成功这个结果。addWorker方法有两个参数：firstTask、core。firstTask 参数用于指定新增的线程执行的第一个任务，该参数可以为空；core 参数为 true 表示在新增线程时会判断当前活动线程数是否少于 corePoolSize，false 表示新增线程前需要判断当前活动线程数是否少于 maximumPoolSize +增加线程是通过线程池中的 addWorker 方法，该方法的功能就是增加一个线程，该方法不考虑线程池是在哪个阶段增加的该线程，这个分配线程的策略是在上个步骤完成的，该步骤仅仅完成增加线程，并使它运行，最后返回是否成功这个结果。addWorker 方法有两个参数：firstTask、core。 + +firstTask 参数用于指定新增的线程执行的第一个任务，该参数可以为空； + +core 参数为 true 表示在新增线程时会判断当前活动线程数是否少于 corePoolSize，false 表示新增线程前需要判断当前活动线程数是否少于 maximumPoolSize ```java private boolean addWorker(Runnable firstTask, boolean core) { @@ -540,11 +613,11 @@ private boolean addWorker(Runnable firstTask, boolean core) { 在 Worker类中的 run 方法调用了 runWorker 方法来执行任务，runWorker 方法的执行过程如下： -1. while循环不断地通过getTask()方法获取任务。 -2. getTask()方法从阻塞队列中取任务。 +1. while循环不断地通过 `getTask()` 方法获取任务。 +2. `getTask()` 方法从阻塞队列中取任务。 3. 如果线程池正在停止，那么要保证当前线程是中断状态，否则要保证当前线程不是中断状态。 4. 执行任务。 -5. 如果getTask结果为null则跳出循环，执行processWorkerExit()方法，销毁线程。 +5. 如果 getTask 结果为 null 则跳出循环，执行 `processWorkerExit()` 方法，销毁线程。 ```java final void runWorker(Worker w) { @@ -596,7 +669,7 @@ final void runWorker(Worker w) { #### Worker线程回收 -线程池中线程的销毁依赖 JVM 自动的回收，线程池做的工作是根据当前线程池的状态维护一定数量的线程引用，防止这部分线程被 JVM 回收，当线程池决定哪些线程需要回收时，只需要将其引用消除即可。Worker被创建出来后，就会不断地进行轮询，然后获取任务去执行，核心线程可以无限等待获取任务，非核心线程要限时获取任务。当 Worker 无法获取到任务，也就是获取的任务为空时，循环会结束，Worker会主动消除自身在线程池内的引用。 +线程池中线程的销毁依赖 JVM 自动的回收，线程池做的工作是根据当前线程池的状态维护一定数量的线程引用，防止这部分线程被 JVM 回收，当线程池决定哪些线程需要回收时，只需要将其引用消除即可。Worker 被创建出来后，就会不断地进行轮询，然后获取任务去执行，核心线程可以无限等待获取任务，非核心线程要限时获取任务。当 Worker 无法获取到任务，也就是获取的任务为空时，循环会结束，Worker 会主动消除自身在线程池内的引用。 ```java try { @@ -616,56 +689,60 @@ try { +## 六、合理配置线程池 + +合理配置线程池你是如何考虑的？ + +首先要考虑到 CPU 核心数，那么在 Java 中如何获取核心线程数？ +可以使用 `Runtime.getRuntime().availableProcessor()` 方法来获取（可能不准确，作为参考） -在工作中，单一的、固定数的、可变的三种创建线程池的方法，那个用的最多？ +在确认了核心数后，再去判断是 CPU 密集型任务还是 IO 密集型任务： -你在工作中是如何使用线程池的，是否自定义过线程池使用 +- **CPU 密集型任务**：CPU密集型也叫计算密集型，这种类型大部分状况下，CPU使用时间远高于I/O耗时。有许多计算要处理、许多逻辑判断，几乎没有I/O操作的任务就属于 CPU 密集型。 -合理配置线程池你是如何考虑的 + CPU 密集任务只有在真正的多核 CPU 上才可能得到加速（通过多线程） + 而在单核 CPU 上，无论开几个模拟的多线程该任务都不可能得到加速，因为 CPU 总的运算能力就那些。 + 如果是 CPU 密集型任务，频繁切换上下线程是不明智的，此时应该设置一个较小的线程数 + 一般公式：**CPU 核数 + 1 个线程的线程池** + 为什么 +1 呢？ -到底用哪个线程池，阿里规范规定不允许使用 Excutiors 创建。 + 《Java并发编程实战》一书中给出的原因是：**即使当计算（CPU）密集型的线程偶尔由于页缺失故障或者其他原因而暂停时，这个“额外”的线程也能确保 CPU 的时钟周期不会被浪费。** -``` +- **IO 密集型任务**：与之相反，IO 密集型则是系统运行时，大部分时间都在进行 I/O 操作，CPU 占用率不高。比如像 MySQL 数据库、文件的读写、网络通信等任务，这类任务**不会特别消耗 CPU 资源，但是 IO 操作比较耗时，会占用比较多时间**。 -``` + 在单线程上运行 IO 密集型的任务会导致浪费大量的 CPU 运算能力浪费在等待。 + 所以在 IO 密集型任务中使用多线程可以大大的加速程序运行，即使在单核 CPU 上，这种加速主要就是利用了被浪费调的阻塞时间。所以在 IO 密集型任务中使用多线程可以大大的加速程序运行，即使在单核 CPU 上，这种加速主要就是利用了被浪费掉的阻塞时间。 + IO 密集型时，大部分线程都阻塞，故需要多配置线程数： + + 参考公式： CPU 核数/（1- 阻塞系数） 阻塞系数在 0.8~0.9 之间 + + 比如 8 核 CPU：8/（1 -0.9）= 80个线程数 + + + 这个其实没有一个特别适用的公式，肯定适合自己的业务，美团给出了个**动态更新**的逻辑，可以看看 + + + + + +**参考与来源：** + +https://tech.meituan.com/2020/04/02/java-pooling-pratice-in-meituan.html + +https://xie.infoq.cn/article/ea8c896d893243e49fe867b0c -合理配置线程池你是如何考虑的？ -- CPU 密集型 - CPU 密集的意思是该任务需要大量的运算，而没有阻塞，CPU 一直全速运行 - CPU 密集任务只有在真正的多核 CPU 上才可能得到加速（通过多线程） - 而在单核 CPU 上，无论开几个模拟的多线程该任务都不可能得到加速，因为 CPU 总的运算能力就那些。 - CPU 密集型任务配置尽可能少的线程数量： - - 一般公式：CPU 合数 + 1 个线程的线程池 - -- IO 密集型 - - IO密集型任务线程并不是一直在执行任务，则应配置尽可能多的线程，如 CPU 核心数*2 - - - IO 密集型，即该任务需要大量的 IO，即大量的阻塞、 - - 在单线程上运行 IO 密集型的任务会导致浪费大量的 CPU 运算能力浪费在等待。 - - 所以在 IO 密集型任务中使用多线程可以大大的加速程序运行，即使在单核 CPU 上，这种加速主要就是利用了被浪费调的阻塞时间。所以在 IO 密集型任务中使用多线程可以大大的加速程序运行，即使在单核 CPU 上，这种加速主要就是利用了被浪费掉的阻塞时间。 - - - - ​ IO 密集型时，大部分线程都阻塞，故需要多配置线程数： - - ​ 参考公式： CPU 核数/（1- 阻塞系数） 阻塞系数在 0.8~0.9 之间 - - ​ 比如 8 核 CPU：8/（1 -0.9）= 80个线程数 diff --git a/docs/java/JUC/ThreadLocal.md b/docs/java/JUC/ThreadLocal.md index 5c64d7b889..bd28c8ad88 100644 --- a/docs/java/JUC/ThreadLocal.md +++ b/docs/java/JUC/ThreadLocal.md @@ -1,6 +1,8 @@ +# ThreadLocal + > 什么是ThreadLocal？ThreadLocal出现的背景是什么？解决了什么问题？ -> ThreadLocal的使用方法是什么？使用的效果如何？ -> ThreadLocal是如何实现它的功能的，即ThreadLocal的原理是什么？ +> ThreadLocal 的使用方法是什么？使用的效果如何？ +> ThreadLocal 是如何实现它的功能的，即 ThreadLocal 的原理是什么？ ## ThreadLocal @@ -8,6 +10,8 @@ ThreadLocal是一个关于创建线程局部变量的类。是`java.lang` 包下 通常情况下，我们创建的变量是可以被任何一个线程访问并修改的。而使用ThreadLocal创建的变量只能被当前线程访问，其他线程则无法访问和修改。 +https://www.cnblogs.com/dolphin0520/p/3920407.html + ## 使用 @@ -205,7 +209,7 @@ public class QUsercenterUtils { ThreadLoal 变量，它的基本原理是，同一个 ThreadLocal 所包含的对象（对ThreadLocal< String >而言即为 String 类型变量），在不同的 Thread 中有不同的副本（实际是不同的实例，后文会详细阐述）。这里有几点需要注意 -- 因为每个 Thread 内有自己的实例副本，且该副本只能由当前 Thread 使用。这是也是 ThreadLocal 命名的由来 +- 因为每个 Thread 内有自己的实例副本，且该副本只能由当前 Thread 使用。这也是 ThreadLocal 命名的由来 - 既然每个 Thread 有自己的实例副本，且其它 Thread 不可访问，那就不存在多线程间共享的问题 - 既无共享，何来同步问题，又何来解决同步问题一说？ diff --git a/docs/java/JUC/readJUC.md b/docs/java/JUC/readJUC.md index c0ab3dc3e9..29295d9b7e 100644 --- a/docs/java/JUC/readJUC.md +++ b/docs/java/JUC/readJUC.md @@ -1 +1,14 @@ -readJUC \ No newline at end of file +在 Java 5.0 提供了 java.util.concurrent（简称 JUC ）包，在此包中增加了在并发编程中很常用的实用工具类，用于定义类似于线程的自定义子系统，包括线程池、异步 IO 和轻量级任务框架。 + + + +其实并发编程可以总结为三个核心问题：分工、同步、互斥。 + +所谓分工指的是如何高效地拆解任务并分配给线程，而同步指的是线程之间如何协作，互斥则是保证同一时刻只允许一个线程访问共享资源。 + +Java SDK 并发包很大部分内容都是按照这三个维度组织的，例如 Fork/Join 框架就是一种分工模式，CountDownLatch 就是一种典型的同步方式，而可重入锁则是一种互斥手段。 + +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/juc/SouthEast.jpeg) + +![](https://tva1.sinaimg.cn/large/0081Kckwly1gkc498bgkkj310r0u0dhr.jpg) + diff --git a/docs/java/JUC/sidebar.md b/docs/java/JUC/sidebar.md deleted file mode 100644 index 9acea159eb..0000000000 --- a/docs/java/JUC/sidebar.md +++ /dev/null @@ -1,61 +0,0 @@ -- **Java基础** -- [![](https://icongr.am/simple/oracle.svg?size=25&color=231c82&colored=false)JVM](java/JVM/readJVM.md) -- [![img](https://icongr.am/fontawesome/expeditedssl.svg?size=25&color=f23131)JUC](java/JUC/readJUC.md) - - [JMM](java/JUC/Java-Memory-Model.md) - - [阻塞队列](java/JUC/BlockingQueue.md) - - [volatile](java/JUC/volatile.md) - - [阻塞队列](java/JUC/BlockingQueue.md) - - [ThreadLocal](java/JUC/ThreadLocal.md) - - [线程池](java/JUC/Thread-Pool.md) - - [CAS](java/JUC/CAS.md) - - [synchronized](java/JUC/synchronized.md) - - [各种锁](java/JUC/各种锁.md) - - [CountDownLatch、CyclicBarrier、Semaphore](java/JUC/CountDownLatch、CyclicBarrier、Semaphore.md) - - [AQS](java/JUC/AQS.md) -- [![](https://icongr.am/devicon/java-original.svg?size=25&color=f23131)Java 8](java/Java8.md) -- [![img](https://icongr.am/entypo/address.svg?size=25&color=074ca6)设计模式](design-pattern/readDisignPattern.md) -- **数据存储和缓存** -- [![MySQL](https://icongr.am/devicon/mysql-original.svg?&size=25)MySQL](data-store/MySQL/readMySQL.md) -- [![Redis](https://icongr.am/devicon/redis-original.svg?size=25)Redis](data-store/Redis/2.readRedis.md) -- [![mongoDB](https://icongr.am/devicon/mongodb-original.svg?&size=25)mongoDB]( https://redis.io/ ) -- [![ **Elasticsearch** ](https://icongr.am/simple/elasticsearch.svg?&size=20) Elasticsearch]( https://redis.io/ ) -- [![S3](https://icongr.am/devicon/amazonwebservices-original.svg?&size=25)S3]( https://aws.amazon.com/cn/s3/ ) -- **直击面试** -- [![](https://icongr.am/material/basket.svg?size=25)Java集合面试](interview/Collections-FAQ.md) -- [![](https://icongr.am/devicon/java-plain-wordmark.svg?size=25)JVM面试](interview/JVM-FAQ.md) -- [![](https://icongr.am/devicon/mysql-original-wordmark.svg?size=25)MySQL面试](interview/MySQL-FAQ.md) -- [![](https://icongr.am/devicon/redis-original-wordmark.svg?size=25)Redis面试](interview/Redis-FAQ.md) -- [![](https://icongr.am/jam/leaf.svg?size=25&color=00FF00)Spring面试](interview/Spring-FAQ.md) -- [![](https://icongr.am/simple/bower.svg?size=25)MyBatis面试](interview/MyBatis-FAQ.md) -- [![img](https://icongr.am/entypo/network.svg?size=25&color=6495ED)计算机网络](interview/Network-FAQ.md) -- **单体架构** -- **RPC** -- [Hello Protocol Buffers](rpc/Hello-Protocol-Buffers.md) -- **面向服务架构** -- [![](https://icongr.am/fontawesome/group.svg?size=25&color=182d10)Zookeeper](soa/ZooKeeper/readZK.md) -- [![message](https://icongr.am/clarity/email.svg?&size=25) 消息中间件](message-queue/readMQ.md) -- [![](https://icongr.am/devicon/nginx-original.svg?size=25&color=182d10)Nginx](nginx/nginx.md) -- **微服务架构** -- [![](https://icongr.am/simple/leaflet.svg?size=25&color=11b041&colored=false)Spring Boot](framework/SpringBoot/Hello-SpringBoot.md) -- [![](https://icongr.am/clarity/alarm-clock.svg?size=25&color=2d2b50)定时任务@Scheduled](framework/SpringBoot/@Scheduled.md) -- **大数据** -- [Hello 大数据](big-data/Hello-BigData.md) -- [![](https://icongr.am/simple/apachekafka.svg?size=25&color=121417&colored=false)Kafka](message-queue/Kafka/readKafka.md) -- **性能优化** -- [![](https://icongr.am/octicons/cpu.svg?size=25&color=780ebe)CPU 飙升问题](optimization/CPU飙升.md) -- \> JVM优化 -- \> web调优 -- \> DB调优 -- **数据结构与算法** -- [![](https://icongr.am/entypo/tree.svg?size=25&color=44c016)树](data-structure/tree.md) -- **工程化与工具** -- [![Maven](https://icongr.am/simple/apachemaven.svg?size=25&color=c93ddb&colored=false)Maven](tools/Maven.md) -- [![Git](https://icongr.am/devicon/git-original.svg?&size=16)Git](tools/Git-Specification.md) -- [![](https://icongr.am/devicon/github-original.svg?size=25&color=currentColor)Git](tools/GitHub.md) -- **其他** -- [![Linux](https://icongr.am/devicon/linux-original.svg?&size=16)Linux](linux/linux.md) -- [缕清各种Java Logging](logging/Java-Logging.md) -- [hello logback](logging/logback简单使用.md) -- **Links** -- [![Github](https://icongram.jgog.in/simple/github.svg?color=808080&size=16)Github](https://github.com/jhildenbiddle/docsify-tabs) -- [![Blog](https://icongr.am/simple/aboutme.svg?colored&size=16)My Blog](https://www.lazyegg.net) \ No newline at end of file diff --git a/docs/java/JUC/synchronized.md b/docs/java/JUC/synchronized.md index 558a6c2453..52f7fc7644 100644 --- a/docs/java/JUC/synchronized.md +++ b/docs/java/JUC/synchronized.md @@ -1,85 +1,231 @@ -```java -public class SynchronizedDemo implements Runnable{ +# synchronized 关键字 - private static int count = 0; +> synchronoized 是如何保证原子性、可见性、有序性的？ - public static void main(String[] args) { - for (int i = 0; i < 10; i++) { - Thread thread = new Thread(new SynchronizedDemo()); - thread.start(); - } - try { - Thread.sleep(500); - } catch (InterruptedException e) { - e.printStackTrace(); - } - System.out.println("result: " + count); - } +## 一、前言 - @Override - public void run() { - for (int i = 0; i < 1000000; i++) { - count++; - } - } +记得开始学习 Java 的时候，一遇到多线程情况就使用 synchronized，相对于当时的我们来说 synchronized 是这么的神奇而又强大，那个时候我们赋予它一个名字“同步”，也成为了我们解决多线程情况的百试不爽的良药。但是，随着学习的进行我们知道在 JDK1.5 之前 synchronized 是一个重量级锁，相对于 j.u.c.Lock，它会显得那么笨重，以至于我们认为它不是那么的高效而慢慢摒弃它。 -} +不过，随着 Javs SE 1.6 对 synchronized 进行的各种优化后，synchronized 并不会显得那么重了。下面来一起探索 synchronized 的基本使用、实现机制、Java是如何对它进行了优化、锁优化机制、锁的存储结构等升级过程。 + + + +## 二、作用 + +Synchronized 是 Java 中解决并发问题的一种最常用的方法，也是最简单的一种方法。Synchronized 的作用主要有三个： + +1. 原子性：确保线程互斥的访问同步代码； +2. 可见性：保证共享变量的修改能够及时可见，其实是通过 Java 内存模型中的 “**对一个变量unlock操作之前，必须要同步到主内存中；如果对一个变量进行lock操作，则将会清空工作内存中此变量的值，在执行引擎使用此变量前，需要重新从主内存中load操作或assign操作初始化变量值**” 来保证的； +3. 有序性：有效解决重排序问题，即 “一个 unlock 操作先行发生(happen-before)于后面对同一个锁的 lock 操作”； + + + +## 三、使用 + +在 Java 代码中使用 synchronized 可以使用在代码块和方法中，根据 synchronized 用的位置可以有这些使用场景： + +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200925184110.png) + +如图，synchronized 可以用在**方法**上也可以使用在**代码块**中，其中方法是实例方法和静态方法分别锁的是该类的实例对象和该类的对象。而使用在代码块中也可以分为三种，具体的可以看上面的表格。这里需要注意的是：**如果锁的是类对象的话，尽管 new 多个实例对象，但他们仍然是属于同一个类依然会被锁住，即线程之间保证同步关系**。 + + + +synchronized 锁的是对象而不是代码，锁方法锁的是 this，锁 static 方法锁的是 class。 + +## 四、原理 + +从语法上讲，synchronized 可以把任何一个非 null 对象作为"锁"，在 HotSpot JVM 实现中，**锁有个专门的名字：对象监视器（Object Monitor）**。 + +synchronized 概括来说其实总共有三种用法： + +1. 当 synchronized 作用在实例方法时，监视器锁（monitor）便是对象实例（this）； +2. 当 synchronized 作用在静态方法时，监视器锁（monitor）便是对象的 Class 实例，因为 Class 数据存在于永久代，因此静态方法锁相当于该类的一个全局锁； +3. 当 synchronized 作用在某一个对象实例时（即代码块的形式），监视器锁（monitor）便是括号括起来的对象实例； + + + +了解原理之前，我们先要知道两个预备知识：对象头和监视器。 + +### Java对象头 + +在 JVM 中，对象在内存中的布局分为三块区域：对象头、实例数据和对齐填充。 + +**实例数据**：存放类的属性数据信息，包括父类的属性信息 + +**对齐填充**：填充数据不是必须存在的，它仅仅起着占位符的作用。由于虚拟机要求对象起始地址必须是 8 字节的整数倍，因此，当对象实例数据部分没有对齐时，就需要通过对齐填充来补全 + +**对象头**：包含两部分信息： + +- 第一部分用于存储对象自身的运行时数据（**标记字段，Mard Word**），如哈希码(HashCode)、GC分代年龄、锁状态标志、线程持有的锁、偏向线程ID、偏向时间戳等。 +- 对象的另一部分**类型指针（Class Pointer）**，即对象指向它的类元数据的指针，虚拟机通过这个指针来确定这个对象是哪个类的实例（并不是所有的虚拟机实现都必须在对象数据上保留类型指针，也就是说，查找对象的元数据信息并不一定要经过对象本身）。 + +- 如果对象是一个 Java 数组，那在对象头中还必须有一块用于记录数组长度的数据。 + +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200925104122.png) + + + +Java 对象头一般占有 2 个机器码（在 32 位虚拟机中，1 个机器码等于 4 字节，也就是 32bit，在 64 位虚拟机中，1 个机器码是 8 个字节，也就是 64bit），但是如果对象是数组类型，则需要 3 个机器码，因为 JVM 虚拟机可以通过 Java 对象的元数据信息确定 Java 对象的大小，但是无法从数组的元数据来确认数组的大小，所以用一块来记录数组长度。 + + + +可以看到对象头中的 Mark Word 记录了对象和锁的有关信息，嗯，你没猜错，我们的 synchronized 和对象头息息相关。 + + + +### 监视器（Monitor） + +> 这个类似操作系统中的管程。 +> +> 管程 (英语：Monitors，也称为监视器) 是一种程序结构，结构内的多个子程序（对象或模块）形成的多个工作线程互斥访问共享资源。这些共享资源一般是硬件设备或一群变量。管程实现了在一个时间点，最多只有一个线程在执行管程的某个子程序。与那些通过修改数据结构实现互斥访问的并发程序设计相比，管程实现很大程度上简化了程序设计。 管程提供了一种机制，线程可以临时放弃互斥访问，等待某些条件得到满足后，重新获得执行权恢复它的互斥访问。 + +#### Java 线程同步相关的 Moniter + +在多线程访问共享资源的时候，经常会带来可见性和原子性的安全问题。为了解决这类线程安全的问题，Java 提供了同步机制、互斥锁机制，这个机制保证了在同一时刻只有一个线程能访问共享资源。这个机制的保障来源于监视锁 Monitor。 + +我们也用某国外大佬的例子来说明 [Moniter 是什么](https://www.programcreek.com/2011/12/monitors-java-synchronization-mechanism/) + +> 监视器可以看做是经过特殊布置的建筑，这个建筑有一个特殊的房间，该房间通常包含一些数据和代码，但是一次只能一个消费者(thread)使用此房间， +> +> ![Java-Monitor](http://ifeve.com/wp-content/uploads/2014/11/Java-Monitor.jpg) +> +> 当一个消费者(线程)使用了这个房间，首先他必须到一个大厅(Entry Set)等待，调度程序将基于某些标准(e.g. FIFO)将从大厅中选择一个消费者(线程)，进入特殊房间，如果这个线程因为某些原因被“挂起”，它将被调度程序安排到“等待房间”，并且一段时间之后会被重新分配到特殊房间，按照上面的线路，这个建筑物包含三个房间，分别是“特殊房间”、“大厅”以及“等待房间”。 +> +> ![java-monitor-associate-with-object](http://ifeve.com/wp-content/uploads/2014/11/java-monitor-associate-with-object.jpg) +> +> 简单来说，监视器用来监视线程进入这个特别房间，他确保同一时间只能有一个线程可以访问特殊房间中的数据和代码。 + +任何一个对象都有一个 Monitor 与之关联，当且一个 Monitor 被持有后，它将处于锁定状态。synchronized 在 JVM 里的实现都是基于进入和退出 Monitor 对象来实现方法同步和代码块同步，虽然具体实现细节不一样，但是都可以通过成对的 MonitorEnter 和 MonitorExit 指令来实现。 + +1. **MonitorEnter 指令：插入在同步代码块的开始位置，当代码执行到该指令时，将会尝试获取该对象 Monitor 的所有权，即尝试获得该对象的锁；** +2. **MonitorExit 指令：插入在方法结束处和异常处，JVM 保证每个 MonitorEnter 必须有对应的MonitorExit；** + +那什么是 Monitor？可以把它理解为 一个同步工具，也可以描述为一种同步机制，它通常被描述为一个对象。 + +与万物皆对象一样，所有的 Java 对象是天生的 Monitor，每一个 Java 对象都有成为 Monitor 的潜质，因为在 Java 的设计中 ，**每一个 Java 对象自打娘胎里出来就带了一把看不见的锁，它叫做内部锁或者 Monitor 锁**。 + +#### 监视器的实现 + +在 Java虚拟机(HotSpot)中，Monitor 是基于 C++ 实现的，由 [ObjectMonitor ](https://github.com/openjdk-mirror/jdk7u-hotspot/blob/50bdefc3afe944ca74c3093e7448d6b889cd20d1/src/share/vm/runtime/objectMonitor.cpp)实现的，其主要数据结构如下： + +```c + ObjectMonitor() { + _header = NULL; + _count = 0; // 记录个数 + _waiters = 0, + _recursions = 0; + _object = NULL; + _owner = NULL; + _WaitSet = NULL; // 处于wait状态的线程，会被加入到_WaitSet + _WaitSetLock = 0 ; + _Responsible = NULL ; + _succ = NULL ; + _cxq = NULL ; + FreeNext = NULL ; + _EntryList = NULL ; // 处于等待锁block状态的线程，会被加入到该列表 + _SpinFreq = 0 ; + _SpinClock = 0 ; + OwnerIsThread = 0 ; + } ``` -开启了10个线程，每个线程都累加了1000000次，如果结果正确的话自然而然总数就应该是10 * 1000000 = 10000000。可就运行多次结果都不是这个数，而且每次运行结果都不一样。这是为什么了？有什么解决方案了？这就是我们今天要聊的事情。 +源码地址：[objectMonitor.hpp](https://github.com/openjdk-mirror/jdk7u-hotspot/blob/50bdefc3afe944ca74c3093e7448d6b889cd20d1/src/share/vm/runtime/objectMonitor.hpp#L193) +ObjectMonitor 中有几个关键属性（每个等待锁的线程都会被封装成 ObjectWaiter 对象）： +> _owner：指向持有 ObjectMonitor 对象的线程 +> +> _WaitSet：存放处于 wait 状态的线程队列 +> +> _EntryList：存放处于等待锁 block 状态的线程队列 +> +> _recursions：锁的重入次数 +> +> _count：用来记录该线程获取锁的次数 +当多个线程同时访问一段同步代码时，首先会进入 `_EntryList` 队列中，当某个线程获取到对象的 monitor 后进入 `_Owner` 区域并把 monitor 中的 `_owner` 变量设置为当前线程，同时 monitor 中的计数器 `_count` 加 1。即获得对象锁。 +若持有 monitor 的线程调用 `wait()` 方法，将释放当前持有的 monitor，`_owner ` 变量恢复为 `null`，`_count `自减 1，同时该线程进入 `_WaitSet` 集合中等待被唤醒。 -synchronized 是 Java 中的关键字，是利用锁的机制来实现同步的。 +若当前线程执行完毕也将释放 monitor(锁) 并复位变量的值，以便其他线程进入获取 monitor(锁)。 -锁机制有如下两种特性： +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200925184419.png) -- 互斥性：即在同一时间只允许一个线程持有某个对象锁，通过这种特性来实现多线程中的协调机制，这样在同一时间只有一个线程对需同步的代码块(复合操作)进行访问。互斥性我们也往往称为操作的原子性。 -- 可见性：必须确保在锁被释放之前，对共享变量所做的修改，对于随后获得该锁的另一个线程是可见的（即在获得锁时应获得最新共享变量的值），否则另一个线程可能是在本地缓存的某个副本上继续操作从而引起不一致。 +如上图所示，一个线程通过 1 号门进入Entry Set(入口区)，如果在入口区没有线程等待，那么这个线程就会获取监视器成为监视器的 Owner，然后执行监视区域的代码。 +如果在入口区中有其它线程在等待，那么新来的线程也会和这些线程一起等待。线程在持有监视器的过程中，有两个选择，一个是正常执行监视器区域的代码，释放监视器，通过 5 号门退出监视器； +还有可能等待某个条件的出现，于是它会通过 3 号门到 Wait Set（等待区）休息，直到相应的条件满足后再通过 4号门进入重新获取监视器再执行。 -# 对象锁和类锁 +注意： -## 1. 对象锁 +> 当一个线程释放监视器时，在入口区和等待区的等待线程都会去竞争监视器，如果入口区的线程赢了，会从 2号门进入；如果等待区的线程赢了会从 4 号门进入。只有通过 3 号门才能进入等待区，在等待区中的线程只有通过 4 号门才能退出等待区，也就是说一个线程只有在持有监视器时才能执行 wait 操作，处于等待的线程只有再次获得监视器才能退出等待状态。 -在 Java 中，每个对象都会有一个 monitor 对象，这个对象其实就是 Java 对象的锁，通常会被称为“内置锁”或“对象锁”。类的对象可以有多个，所以每个对象有其独立的对象锁，互不干扰。 -## 2. 类锁 -在 Java 中，针对每个类也有一个锁，可以称为“类锁”，类锁实际上是通过对象锁实现的，即类的 Class 对象锁。每个类只有一个 Class 对象，所以每个类只有一个类锁。 +### synchronized与原子性 + +原子性是指一个操作是不可中断的，要全部执行完成，要不就都不执行。 + +线程是 CPU 调度的基本单位。CPU 有时间片的概念，会根据不同的调度算法进行线程调度。当一个线程获得时间片之后开始执行，在时间片耗尽之后，就会失去 CPU 使用权。所以在多线程场景下，由于时间片在线程间轮换，就会发生原子性问题。 + +在 Java 中，为了保证原子性，提供了两个高级的字节码指令 `monitorenter` 和 `monitorexit`。前面中，介绍过，这两个字节码指令，在 Java 中对应的关键字就是 `synchronized`。 + +通过 `monitorenter` 和 `monitorexit` 指令，可以保证被 `synchronized` 修饰的代码在同一时间只能被一个线程访问，在锁未释放之前，无法被其他线程访问到。因此，在 Java 中可以使用 `synchronized` 来保证方法和代码块内的操作是原子性的。 +> 线程 1 在执行 `monitorenter` 指令的时候，会对 Monitor 进行加锁，加锁后其他线程无法获得锁，除非线程1 主动解锁。即使在执行过程中，由于某种原因，比如 CPU 时间片用完，线程 1 放弃了 CPU，但是，他并没有进行解锁。而由于 `synchronized` 的锁是可重入的，下一个时间片还是只能被他自己获取到，还是会继续执行代码。直到所有代码执行完。这就保证了原子性。 +### synchronized与可见性 -# synchronized 的用法分类 +可见性是指当多个线程访问同一个变量时，一个线程修改了这个变量的值，其他线程能够立即看得到修改的值。 -synchronized 的用法可以从两个维度上面分类： +Java 内存模型规定了所有的变量都存储在主内存中，每条线程还有自己的工作内存，线程的工作内存中保存了该线程中是用到的变量的主内存副本拷贝，线程对变量的所有操作都必须在工作内存中进行，而不能直接读写主内存。不同的线程之间也无法直接访问对方工作内存中的变量，线程间变量的传递均需要自己的工作内存和主存之间进行数据同步进行。所以，就可能出现线程 1 改了某个变量的值，但是线程 2 不可见的情况。 -## 1. 根据修饰对象分类 +前面我们介绍过，被 `synchronized` 修饰的代码，在开始执行时会加锁，执行完成后会进行解锁。而为了保证可见性，有一条规则是这样的：对一个变量解锁之前，必须先把此变量同步回主存中。这样解锁后，后续线程就可以访问到被修改后的值。 -synchronized 可以修饰方法和代码块 +所以，synchronized 关键字锁住的对象，其值是具有可见性的。 -- 修饰代码块 - - synchronized(this|object) {} - - synchronized(类.class) {} -- 修饰方法 - - 修饰非静态方法 - - 修饰静态方法 +### synchronized与有序性 -## 2. 根据获取的锁分类 +有序性即程序执行的顺序按照代码的先后顺序执行。 -- 获取对象锁 - - synchronized(this|object) {} - - 修饰非静态方法 -- 获取类锁 - - synchronized(类.class) {} - - 修饰静态方法 +除了引入了时间片以外，由于处理器优化和指令重排等，CPU 还可能对输入代码进行乱序执行，比如 `load->add->save` 有可能被优化成 `load->save->add`。这就是可能存在有序性问题。 +这里需要注意的是，`synchronized` 是无法禁止指令重排和处理器优化的。也就是说，`synchronized `无法避免上述提到的问题。 +那么，为什么还说 `synchronized` 也提供了有序性保证呢？ +这就要再把有序性的概念扩展一下了。Java 程序中天然的有序性可以总结为一句话：如果在本线程内观察，所有操作都是天然有序的。如果在一个线程中观察另一个线程，所有操作都是无序的。 +以上这句话也是《深入理解Java虚拟机》中的原句，但是怎么理解呢？周志明并没有详细的解释。这里我简单扩展一下，这其实和 `as-if-serial语义` 有关。 + +`as-if-serial` 语义的意思指：不管怎么重排序（编译器和处理器为了提高并行度），单线程程序的执行结果都不能被改变。编译器和处理器无论如何优化，都必须遵守`as-if-serial`语义。 + +这里不对 `as-if-serial语义` 详细展开了，简单说就是，`as-if-serial语义` 保证了单线程中，指令重排是有一定的限制的，而只要编译器和处理器都遵守了这个语义，那么就可以认为单线程程序是按照顺序执行的。当然，实际上还是有重排的，只不过我们无须关心这种重排的干扰。 + +所以呢，由于 `synchronized` 修饰的代码，同一时间只能被同一线程访问。那么也就是单线程执行的。所以，可以保证其有序性。 + + + +预备知识结束，其实 synchronized 的原理也就差不多了，我们再细看 + + + +### synchronized 原理 + +从上文 synchronized 的使用中，我们可以看到从加锁位置的不同，它其实可以只有两种：锁住的是类，或者锁住的是对象。我们把它称为类锁和对象锁。 + +#### 对象锁 + +在 Java 中，每个对象都会有一个 monitor 对象，这个对象其实就是 Java 对象的锁，通常会被称为“**内置锁**”或“**对象锁**”。类的对象可以有多个，所以每个对象有其独立的对象锁，互不干扰。 + +#### 类锁 + +在 Java 中，针对每个类也有一个锁，可以称为“类锁”，类锁实际上是通过对象锁实现的，即类的 Class 对象锁。每个类只有一个 Class 对象，所以每个类只有一个类锁。 + + + +接下来从代码，看下 synchronized 的实现 ```java public class SynchronizedDemo { @@ -94,14 +240,376 @@ public class SynchronizedDemo { } ``` -上面的代码中有一个同步代码块，锁住的是类对象，并且还有一个同步静态方法，锁住的依然是该类的类对象。编译之后，切换到SynchronizedDemo.class的同级目录之后，然后用**javap -v SynchronizedDemo.class**查看字节码文件： +上面的代码中有一个同步代码块，锁住的是类对象，并且还有一个静态方法，锁住的依然是该类的类对象。编译之后，切换到 `SynchronizedDemo.class` 的同级目录之后，然后用 **javap -v SynchronizedDemo.class** 查看字节码文件： + +![SynchronizedDemo.class](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200925184342.png) + +如图，上面用黄色高亮的部分就是需要注意的部分了，这也是添加 synchronized 关键字之后独有的。执行同步代码块后首先要先执行 **monitorenter** 指令，退出的时候执行 **monitorexit** 指令。通过分析之后可以看出，使用synchronized 进行同步，其关键就是必须要对对象的监视器 monitor 进行获取，当线程获取 monitor 后才能继续往下执行，否则就只能等待。而这个获取的过程是**互斥**的，即同一时刻只有一个线程能够获取到 monitor。上面的 demo 中在执行完同步代码块之后紧接着再会去执行一个静态同步方法，而这个方法锁的对象依然就这个类对象，那么这个正在执行的线程还需要获取该锁吗？答案是不必的，从上图中就可以看出来，执行静态同步方法的时候就只有一条 monitorexit 指令，并没有 monitorenter 获取锁的指令。这就是锁的重入性，即在同一锁程中，线程不需要再次获取同一把锁。synchronized 先天具有重入性。**每个对象拥有一个计数器，当线程获取该对象锁后，计数器就会加一，释放锁后就会将计数器减一**。 + +1. **monitorenter**：每个对象都是一个监视器锁（monitor）。当 monitor 被占用时就会处于锁定状态，线程执行 monitorenter 指令时尝试获取 monitor 的所有权，过程如下： + + > 1. 如果 monitor 的进入数为 0，则该线程进入 monitor，然后将进入数设置为 1，该线程即为 monitor的所有者； + > 2. 如果线程已经占有该 monitor，只是重新进入，则进入 monitor 的进入数加 1； + > 3. 如果其他线程已经占用了 monitor，则该线程进入阻塞状态，直到 monitor 的进入数为 0，再重新尝试获取 monitor 的所有权； + +2. monitorexit：执行 monitorexit 指令的线程必须是对象实例所对应的监视器的所有者。指令执行时，monitor的进入数减 1，如果减 1 后进入数为 0，那线程退出 monitor，不再是这个 monitor 的所有者。其他被这个monitor 阻塞的线程可以尝试去获取这个 monitor 的所有权。 + + +通过上面两段描述，我们应该能很清楚的看出 synchronized 的实现原理，**synchronized 的语义底层是通过一个monitor 的对象来完成，其实 wait/notify 等方法也依赖于monitor对象，这就是为什么只有在同步的块或者方法中才能调用 wait/notify 等方法，否则会抛出 java.lang.IllegalMonitorStateException 的异常的原因。** + + + +> JVM需要保证每一个 monitorenter 都有一个 monitorexit 与之相对应，但每个 monitorexit 不一定都有一个monitorenter，比如 [JVM 规范中的代码](https://docs.oracle.com/javase/specs/jvms/se12/html/jvms-3.html#jvms-3.14) 中的示例，我们也会看到有两次 monitorexit，它的注释是 Be sure to exit the monitor! +> +> 看到有个这样的解释： +> +> 该代码没有两次“调用” `monitorexit`指令。它在两个不同的代码路径上执行一次。 +> +> - 第一个代码路径用于`synchronized`块中的代码正常退出时。 +> - 对于块异常终止的情况，第二个代码路径位于*隐式*异常处理路径中。 +> +> 您可以在示例中将字节码写为 *pseudo-code* ，如下所示： +> +> ``` +> void onlyMe(Foo f) { +> monitorEntry(f); +> +> try { +> doSomething(); +> monitorExit(); +> } catch (Throwable any) { +> monitorExit(); +> throw any; +> } +> } +> ``` + + + +再来看一下同步方法： + +```java +public class SynchronizedMethod { + public synchronized void method() { + System.out.println("Hello World!"); + } +} +``` + +查看反编译后结果： + +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200925185206.png) + +从编译的结果来看，方法的同步并没有通过指令 `monitorenter` 和 `monitorexit` 来完成（理论上其实也可以通过这两条指令来实现），不过相对于普通方法，其常量池中多了 `ACC_SYNCHRONIZED` 标示符。JVM 就是根据该标示符来实现方法的同步的 + +[The Java® Virtual Machine Specification](https://docs.oracle.com/javase/specs/jvms/se8/html/jvms-2.html#jvms-2.11.10)中有关于方法级同步的介绍： + +> Method-level synchronization is performed implicitly, as part of method invocation and return. A synchronized method is distinguished in the run-time constant pool’s method_info structure by the ACC_SYNCHRONIZED flag, which is checked by the method invocation instructions. When invoking a method for which ACC_SYNCHRONIZED is set, the executing thread enters a monitor, invokes the method itself, and exits the monitor whether the method invocation completes normally or abruptly. During the time the executing thread owns the monitor, no other thread may enter it. If an exception is thrown during invocation of the synchronized method and the synchronized method does not handle the exception, the monitor for the method is automatically exited before the exception is rethrown out of the synchronized method. + +主要说的是： 方法级的同步是隐式的。同步方法的常量池中会有一个 `ACC_SYNCHRONIZED` 标志。当某个线程要访问某个方法的时候，会检查是否有 `ACC_SYNCHRONIZED`，如果有设置，执行线程将先获取 monitor，获取成功之后才能执行方法体，方法执行完后再释放 monitor。这时如果其他线程来请求执行方法，会因为无法获得监视器锁而被阻断住。值得注意的是，如果在方法执行过程中，发生了异常，并且方法内部并没有处理该异常，那么在异常被抛到方法外面之前监视器锁会被自动释放。 + + + +注意，synchronized 内置锁是一种对象锁（锁的是对象而非引用变量），**作用粒度是对象 ，可以用来实现对临界资源的同步互斥访问 ，是可重入的。其可重入最大的作用是避免死锁**，如： + +子类同步方法调用了父类同步方法，如没有可重入的特性，则会发生死锁； + +> 面试官：父类中有一个加锁的方法A，而子类中也有一个加锁的方法B，B在执行过程中，会调用A方法，问此时会不会产生死锁？ +> +> 不会，创建子类对象时，不会创建父类对象，其实创建子类对象的时候，JVM会为子类对象分配内存空间，并调用父类的构造函数。我们可以这样理解：创建了一个子类对象的时候，在子类对象内存中，有两份数据，一份继承自父类，一份来自子类，但是他们属于同一个对象（子类对象）。 + + + + + +两种同步方式本质上没有区别，只是方法的同步是一种隐式的方式来实现，无需通过字节码来完成。两个指令的执行是 JVM 通过调用操作系统的互斥原语 mutex 来实现，被阻塞的线程会被挂起、等待重新调度，会导致“用户态和内核态”两个态之间来回切换，对性能有较大影响。 + +所以频繁的通过Synchronized实现同步会严重影响到程序效率，这种锁机制也被称为重量级锁，为了减少重量级锁带来的性能开销，JDK对Synchronized进行了种种优化。 + + + +## 五、锁优化 + +从 JDK5 引入了现代操作系统新增加的 CAS 原子操作（ JDK5 中并没有对 synchronized 关键字做优化，而是体现在 J.U.C 中，所以在该版本 concurrent 包有更好的性能 ），从 JDK6 开始，就对 synchronized 的实现机制进行了较大调整，包括使用 JDK5 引进的 CAS 自旋之外，还增加了自适应的 CAS 自旋、锁消除、锁粗化、偏向锁、轻量级锁这些优化策略。由于此关键字的优化使得性能极大提高，同时语义清晰、操作简单、无需手动关闭，所以推荐在允许的情况下尽量使用此关键字，同时在性能上此关键字还有优化的空间。 + +锁主要存在四种状态，依次是：**无锁状态、偏向锁状态、轻量级锁状态、重量级锁状态**，锁可以从偏向锁升级到轻量级锁，再升级的重量级锁。但是锁的升级是单向的，也就是说只能从低到高升级，不会出现锁的降级。 + +在 JDK 1.6 中默认是开启偏向锁和轻量级锁的，可以通过 `-XX:-UseBiasedLocking` 来禁用偏向锁。 + +### 自旋锁 + +线程的阻塞和唤醒需要 CPU 从用户态转为核心态，频繁的阻塞和唤醒对 CPU 来说是一件负担很重的工作，势必会给系统的并发性能带来很大的压力。同时我们发现在许多应用上面，对象锁的锁状态只会持续很短一段时间，为了这一段很短的时间频繁地阻塞和唤醒线程是非常不值得的。 + +所以引入了自旋锁，何谓自旋锁？ -![SynchronizedDemo.class](https://user-gold-cdn.xitu.io/2018/4/30/16315cce259af0d2?imageView2/0/w/1280/h/960/format/webp/ignore-error/1) +所谓自旋锁，就是指当一个线程尝试获取某个锁时，如果该锁已被其他线程占用，就一直循环检测锁是否被释放，而不是进入线程挂起或睡眠状态。 +**自旋锁适用于锁保护的临界区很小的情况**，临界区很小的话，锁占用的时间就很短。自旋等待不能替代阻塞，虽然它可以避免线程切换带来的开销，但是它占用了 CPU 处理器的时间。如果持有锁的线程很快就释放了锁，那么自旋的效率就非常好，反之，自旋的线程就会白白消耗掉处理的资源，它不会做任何有意义的工作，典型的占着茅坑不拉屎，这样反而会带来性能上的浪费。所以说，自旋等待的时间（自旋的次数）必须要有一个限度，如果自旋超过了定义的时间仍然没有获取到锁，则应该被挂起。 +自旋锁在 JDK 1.4.2 中引入，默认关闭的，但是可以使用 `-XX:+UseSpinning` 开启，在 JDK1.6 中默认开启。同时自旋的默认次数为 10 次，可以通过参数 `-XX:PreBlockSpin` 来调整。 + +如果通过参数 `-XX:PreBlockSpin` 来调整自旋锁的自旋次数，会带来诸多不便。假如将参数调整为 10，但是系统很多线程都是等你刚刚退出的时候就释放了锁（假如多自旋一两次就可以获取锁），是不是很尴尬。于是 JDK1.6引入了自适应的自旋锁，让虚拟机变得越来越聪明。 + +### 适应性自旋锁 + +JDK 1.6 引入了更加聪明的自旋锁，即自适应自旋锁。所谓自适应就意味着自旋的次数不再是固定的了，它是由前一次在同一个锁上的自旋时间及锁的拥有者的状态来决定的。那它如何进行适应性自旋呢？ + +**线程如果自旋成功了，那么下次自旋的次数会更加多，因为虚拟机认为既然上次成功了，那么此次自旋也很有可能会再次成功，那么它就会允许自旋等待持续的次数更多。反之，如果对于某个锁，很少有自旋能够成功，那么在以后要获取这个锁的时候自旋的次数就会减少甚至省略掉自旋过程，以免浪费处理器资源。** + +有了自适应自旋锁，随着程序运行和性能监控信息的不断完善，虚拟机对程序锁的状况预测会越来越准确，虚拟机会变得越来越聪明。 + +### 锁消除 + +为了保证数据的完整性，在进行操作时需要对这部分操作进行同步控制，但是在有些情况下，JVM 检测到不可能存在共享数据竞争，这时 JVM 就会对这些同步锁进行锁消除。 + +> 锁消除的依据是逃逸分析的数据支持 + +如果不存在竞争，为什么还需要加锁呢？所以锁消除可以节省毫无意义的请求锁的时间。变量是否逃逸，对于虚拟机来说需要使用数据流分析来确定，但是对于程序员来说这还不清楚么？在明明知道不存在数据竞争的代码块前加上同步吗？但是有时候程序并不是我们所想的那样？虽然没有显示使用锁，但是在使用一些 JDK 的内置 API 时，如 StringBuffer、Vector、HashTable 等，这个时候会存在隐形的加锁操作。比如 StringBuffer 的 `append()` 方法，Vector的 `add()` 方法： + +```java +public void vectorTest(){ + Vector vector = new Vector(); + for(int i = 0 ; i < 10 ; i++){ + vector.add(i + ""); + } + + System.out.println(vector); +} +``` + +在运行这段代码时，JVM 可以明显检测到变量 vector 没有逃逸出方法 vectorTest() 之外，所以 JVM可以大胆地将 vector 内部的加锁操作消除。 + +### 锁粗化 + +很多时候，我们提倡尽量减小锁的粒度，可以避免不必要的阻塞。 让同步块的作用范围尽可能小，仅在共享数据的实际作用域中才进行同步，如果存在锁竞争，那么等待锁的线程也能尽快拿到锁。 + +但是如果在一段代码中连续的用同一个监视器锁反复的加锁解锁，甚至加锁操作出现在循环体中的时候，就会导致不必要的性能损耗，这种情况就需要锁粗化。 + +锁粗话概念比较好理解，就是将多个连续的加锁、解锁操作连接在一起，扩展成一个范围更大的锁 + +举例： + +```java +for(int i=0;i<100000;i++){ + synchronized(this){ + do(); + } +} +``` + +会被粗化成： + +```java +synchronized(this){ + for(int i=0;i<100000;i++){ + do(); + } +} +``` + +这里我们再回过头看下对象头中 Mark Word 的存储结构（以 32 位虚拟机为例，《Java并发编程艺术》） + +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200925170526.jpg) + + + +从 Java 对象头的 Mark word 中可以看到，synchronized 锁一共具有四种状态：无锁、偏向锁、轻量级锁、重量级锁。 + +偏向锁、轻量级锁、重量级锁三种形式，分别对应了锁只被一个线程持有、不同线程交替持有锁、多线程竞争锁三种情况。 + + + +### 偏向锁 + +偏向锁是 JDK6 中的重要引进，因为 HotSpot 作者经过研究实践发现，在大多数情况下，锁不仅不存在多线程竞争，而且总是由同一线程多次获得，为了让线程获得锁的代价更低，引进了偏向锁。 + +1. 偏向锁主要用来优化**同一线程多次申请同一个锁**的竞争，在某些情况下，大部分时间都是同一个线程竞争锁资源 + +2. 偏向锁的作用 + + - 当一个线程再次访问同一个同步代码时，该线程只需对该对象头的**Mark Word**中去判断是否有偏向锁指向它 + - **无需再进入Monitor去竞争对象**（避免用户态和内核态的**切换**） + +3. 当对象被当做同步锁，并有一个线程抢到锁时 + + - 锁标志位还是**01**，是否偏向锁标志位设置为**1**，并且记录抢到锁的**线程ID**，进入**偏向锁状态** + +4. 偏向锁 + + **不会主动释放锁** + + - 当线程1再次获取锁时，会比较**当前线程的ID**与**锁对象头部的线程ID**是否一致，如果一致，无需CAS来抢占锁 + + - 如果不一致，需要查看锁对象头部记录的线程是否存活 + + - 如果**没有存活**，那么锁对象被重置为**无锁**状态（也是一种撤销），然后重新偏向线程2 + + - 如果存活，查找线程1的栈帧信息 + + - 如果线程1还是需要继续持有该锁对象，那么暂停线程1（**STW**），**撤销偏向锁**，**升级为轻量级锁** +- 如果线程1不再使用该锁对象，那么将该锁对象设为**无锁**状态（也是一种撤销），然后重新偏向线程2 + +5. 一旦出现其他线程竞争锁资源时，偏向锁就会被撤销 + + - 偏向锁的撤销**可能需要**等待**全局安全点**，暂停持有该锁的线程，同时检查该线程**是否还在执行该方法** +- 如果还没有执行完，说明此刻有**多个线程**竞争，升级为**轻量级锁**；如果已经执行完毕，唤醒其他线程继续**CAS**抢占 + +6. 在高并发场景下，当大量线程同时竞争同一个锁资源时，偏向锁会被撤销，发生 STW ，加大了性能开销 + + - 默认配置 + + - `-XX:+UseBiasedLocking -XX:BiasedLockingStartupDelay=4000` + - 默认开启偏向锁，并且**延迟生效**，因为JVM刚启动时竞争非常激烈 + + - 关闭偏向锁 + + - `-XX:-UseBiasedLocking` + + - 直接设置为重量级锁 + + - `-XX:+UseHeavyMonitors` + +红线流程部分：偏向锁的**获取**和**撤销** +![](https://java-performance-1253868755.cos.ap-guangzhou.myqcloud.com/java-performance-synchronized-lock-upgrade-1.png) + + + +### 轻量级锁 + +目的：在大多数情况下同步块并不会出现竞争情况，大部分情况是不同线程交替持有锁，所以引入轻量级锁可以减少重量级锁对线程的阻塞带来的开销。 + +轻量级锁认为环境中线程几乎没有对锁对象的竞争，即使有竞争也只需要稍微等待（自旋）下就可以获取锁，但是自旋次数有限制，如果超过该次数，则会升级为重量级锁。 + +1. 当有另外一个线程竞争锁时，由于该锁处于**偏向锁**状态 + +2. 发现对象头Mark Word中的线程ID不是自己的线程ID，该线程就会执行 CAS 操作获取锁 + + - 如果获取**成功**，直接替换Mark Word中的线程ID为自己的线程ID，该锁会**保持偏向锁状态** +- 如果获取**失败**，说明当前锁有一定的竞争，将偏向锁**升级**为轻量级锁 + +3. 线程获取轻量级锁时会有两步 + + - 先把**锁对象的Mark Word**复制一份到线程的**栈帧**中（**DisplacedMarkWord**），主要为了**保留现场**!! + - 然后使用**CAS**，把对象头中的内容替换为**线程栈帧中DisplacedMarkWord的地址** + +4. 场景 + + - 在线程1复制对象头Mark Word的同时（CAS之前），线程2也准备获取锁，也复制了对象头Mark Word + - 在线程2进行CAS时，发现线程1已经把对象头换了，线程2的CAS失败，线程2会尝试使用**自旋锁**来等待线程1释放锁 + +5. 轻量级锁的适用场景：线程**交替执行**同步块，***绝大部分的锁在整个同步周期内都不存在长时间的竞争*** + +红线流程部分：升级轻量级锁 +[![img](https://java-performance-1253868755.cos.ap-guangzhou.myqcloud.com/java-performance-synchronized-lock-upgrade-2.png)](https://java-performance-1253868755.cos.ap-guangzhou.myqcloud.com/java-performance-synchronized-lock-upgrade-2.png) + + + +### 重量级锁 + +Synchronized 是通过对象内部的一个叫做 监视器锁（Monitor）来实现的。但是监视器锁本质又是依赖于底层的操作系统的 Mutex Lock 来实现的。而操作系统实现线程之间的切换这就需要从用户态转换到核心态，这个成本非常高，状态之间的转换需要相对比较长的时间，这就是为什么 JDK5 之前 synchronized 效率低的原因。因此，这种依赖于操作系统 Mutex Lock 所实现的锁我们称之为 “重量级锁”。 + +1. 轻量级锁 CAS 抢占失败，线程将会被挂起进入阻塞状态 + + - 如果正在持有锁的线程在**很短的时间**内释放锁资源，那么进入**阻塞**状态的线程被**唤醒**后又要**重新抢占**锁资源 + +2. JVM 提供了**自旋锁**，可以通过**自旋**的方式**不断尝试获取锁**，从而**避免线程被挂起阻塞** + +3. 从 JDK 1.6 开始，自旋锁默认启用，自旋次数不建议设置过大（意味着长时间占用CPU） + + - `-XX:+UseSpinning -XX:PreBlockSpin=10` + +4. 自旋锁重试之后如果依然抢锁失败，同步锁会升级至重量级锁，锁标志位为 10 在这个状态下，未抢到锁的线程都会**进入Monitor**，之后会被阻塞在**WaitSet**中 + +5. 在锁竞争不激烈且锁占用时间非常短的场景下，自旋锁可以提高系统性能 + + - 一旦锁竞争激烈或者锁占用的时间过长，自旋锁将会导致大量的线程一直处于**CAS重试状态**，**占用CPU资源** + +6. 在高并发的场景下，可以通过关闭自旋锁来优化系统性能 + + - ``` + -XX:-UseSpinning + ``` + + - 关闭自旋锁优化 + + - ``` + -XX:PreBlockSpin + ``` + + - 默认的自旋次数，在**JDK 1.7**后，**由JVM控制** + +![](https://java-performance-1253868755.cos.ap-guangzhou.myqcloud.com/java-performance-synchronized-lock-upgrade-3.png) + + + +### 重量级锁、轻量级锁和偏向锁之间转换 + +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200925191059.png) + +重量级锁、轻量级锁和偏向锁之间转换 + +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20200925184704.png) + +### 锁的膨胀过程 + + + + + +### 锁的优劣 + +各种锁并不是相互代替的，而是在不同场景下的不同选择，绝对不是说重量级锁就是不合适的。每种锁是只能升级，不能降级，即由偏向锁->轻量级锁->重量级锁，而这个过程就是开销逐渐加大的过程。 + +1. 如果是单线程使用，那偏向锁毫无疑问代价最小，并且它就能解决问题，连 CAS 都不用做，仅仅在内存中比较下对象头就可以了； +2. 如果出现了其他线程竞争，则偏向锁就会升级为轻量级锁； +3. 如果其他线程通过一定次数的 CAS 尝试没有成功，则进入重量级锁； + +在第 3 种情况下进入同步代码块就要做偏向锁建立、偏向锁撤销、轻量级锁建立、升级到重量级锁，最终还是得靠重量级锁来解决问题，那这样的代价就比直接用重量级锁要大不少了。所以使用哪种技术，一定要看其所处的环境及场景，在绝大多数的情况下，偏向锁是有效的，这是基于 HotSpot 作者发现的“大多数锁只会由同一线程并发申请”的经验规律。 + +| 锁 | 优点 | 缺点 | 应用场景 | +| :------- | :----------------------------------- | :------------------------------------------- | :----------------------------------------------------------- | +| 偏向锁 | 加锁与解锁基本不消耗资源 | 如果存在线程竞争则撤销锁需要额外的消耗 | 只有一个线程访问同步块的情景 | +| 轻量级锁 | 竞争锁不需要线程切换，提供了执行效率 | 如果存在大量线程竞争锁，自旋会消耗CPU资源 | 追求响应时间。适用于少量线程访问同步块，追求访问同步块的速度 | +| 重量级锁 | 线程不需要自旋，不会消耗过多cpu资源 | 线程切换需要消耗大量资源，线程阻塞，执行缓慢 | 追求吞吐量。同步块执行时间较长的情况。 | + + + +### 常见锁优化方案 + +1. 减少锁持有时间：尽可能减少同步代码块，加快同步代码块执行速度。 + +2. 减少锁的粒度：分段锁概念 + +3. 锁粗化 + +4. 锁分离（读写锁） + +5. 使用CAS + 自旋的形式 + +6. 消除缓存行的伪共享： + 每个CPU都有自己独占的一级缓存，二级缓存，为了提供性能，CPU读写数据是以缓存行尾最小单元读写的；32位的cpu缓存行为32字节，64位cpu的缓存行为64字节，这就导致了一些问题，例如，多个不需要同步的变量因为存储在连续的32字节或64字节里面，当需要其中的一个变量时，就将它们作为一个缓存行一起加载到某个cup-1私有的缓存中（虽然只需要一个变量，但是cpu读取会以缓存行为最小单位，将其相邻的变量一起读入），被读入cpu缓存的变量相当于是对主内存变量的一个拷贝，也相当于变相的将在同一个缓存行中的几个变量加了一把锁，这个缓存行中任何一个变量发生了变化，当cup-2需要读取这个缓存行时，就需要先将cup-1中被改变了的整个缓存行更新回主存（即使其它变量没有更改），然后cup-2才能够读取，而cup-2可能需要更改这个缓存行的变量与cpu-1已经更改的缓存行中的变量是不一样的，所以这相当于给几个毫不相关的变量加了一把同步锁； + + 为了防止伪共享，不同jdk版本实现方式是不一样的： + + 1. 在jdk1.7之前会 将需要独占缓存行的变量前后添加一组long类型的变量，依靠这些无意义的数组的填充做到一个变量自己独占一个缓存行； + 2. 在jdk1.7因为jvm会将这些没有用到的变量优化掉，所以采用继承一个声明了好多long变量的类的方式来实现； + 3. 在jdk1.8中通过添加sun.misc.Contended注解来解决这个问题，若要使该注解有效必须在jvm中添加以下参数： + `-XX:-RestrictContended` + + `sun.misc.Contended`注解会在变量前面添加128字节的 padding 将当前变量与其他变量进行隔离； ## 参考 -https://juejin.im/post/5ae6dc04f265da0ba351d3ff \ No newline at end of file +https://juejin.im/post/5ae6dc04f265da0ba351d3ff + +https://blog.csdn.net/zhengwangzw/article/details/105141484 + +https://www.cnblogs.com/aspirant/p/11470858.html + +https://www.hollischuang.com/archives/2030 + +http://zhongmingmao.me/2019/08/15/java-performance-synchronized-opt/ + +https://www.hollischuang.com/archives/2637 \ No newline at end of file diff --git a/docs/java/JUC/volatile.md b/docs/java/JUC/volatile.md index 05979a5c24..a0c1cbca99 100644 --- a/docs/java/JUC/volatile.md +++ b/docs/java/JUC/volatile.md @@ -1,3 +1,5 @@ +# volatile 关键字 + > 谈谈你对 volatile 的理解？ > > 你知道 volatile 底层的实现机制吗？ @@ -6,21 +8,17 @@ > > volatile 的使用场景，你能举两个例子吗？ -> 文章收录在 GitHub [JavaKeeper](https://github.com/Jstarfish/JavaKeeper) ，包含 N 线互联网开发必备技能兵器谱 +之前算是比较详细的介绍了 [Java 内存模型](https://mp.weixin.qq.com/s/FUHFppzcISLDMx4vc8tz4A)——JMM， **JMM 是围绕着并发过程中如何处理可见性、原子性和有序性这 3 个 特征建立起来的**，而 volatile 可以保证其中的两个特性，下面具体探讨下这个工作常用、面试必问的关键字。 -之前算是比较详细的介绍了 [Java 内存模型](https://mp.weixin.qq.com/s?__biz=MzIwOTIxNTg0OQ==&mid=2247483928&idx=1&sn=87401493c2378ae5a291fe9c07f6bd09&chksm=97760a1ea0018308ad9b5608ffa74937abf37e5ec54e7d773fd4c4199daace67a98e390ea4ce&token=954870021&lang=zh_CN#rd)——JMM， **JMM是围绕着并发过程中如何处理可见性、原子性和有序性这 3 个 特征建立起来的**，而 volatile 可以保证其中的两个特性，下面具体探讨下这个面试必问的关键字。 - ![img](https://i02piccdn.sogoucdn.com/b1d5cd0fd6acf03a) -## 1. 概念 +## 一、概念 volatile 是 Java 中的关键字，是一个变量修饰符，用来修饰会被不同线程访问和修改的变量。 ------- - -## 2. Java 内存模型 3 个特性 +## 二、 Java 内存模型 3 个特性 ### 2.1 可见性 @@ -30,9 +28,13 @@ volatile 是 Java 中的关键字，是一个变量修饰符，用来修饰会 在 Java 中 volatile、synchronized 和 final 都可以实现可见性。 +> synchronized 同步块的可见性是由“对一个变量执行 unlock 操作之前，必须先把此变量同步回主内存中（执行 store、write 操作）”这条规则获得的。 +> +> final 关键字的可见性是指：被 final 修饰的字段在构造器中一旦被初始化完成，并且构造器没有把 “this” 的引用传递出去，那么其他线程中就能看见 final 字段的值。 + ### 2.2 原子性 -原子性指的是某个线程正在执行某个操作时，中间不可以被加塞或分割，要么整体成功，要么整体失败。比如 a=0；（a非long和double类型） 这个操作是不可分割的，那么我们说这个操作是原子操作。再比如：a++； 这个操作实际是a = a + 1；是可分割的，所以他不是一个原子操作。非原子操作都会存在线程安全问题，需要我们使用同步技术（sychronized）来让它变成一个原子操作。一个操作是原子操作，那么我们称它具有原子性。Java 的 concurrent 包下提供了一些原子类，AtomicInteger、AtomicLong、AtomicReference等。 +原子性指的是某个线程正在执行某个操作时，中间不可以被加塞或分割，要么整体成功，要么整体失败。比如 a=0；（a非long和double类型） 这个操作是不可分割的，那么我们说这个操作是原子操作。再比如：a++； 这个操作实际是 `a = a + 1` 是可分割的，所以他不是一个原子操作。非原子操作都会存在线程安全问题，需要我们使用同步技术（sychronized）来让它变成一个原子操作。一个操作是原子操作，那么我们称它具有原子性。Java 的 concurrent 包下提供了一些原子类，AtomicInteger、AtomicLong、AtomicReference 等。 在 Java 中 synchronized 和在 lock、unlock 中操作保证原子性。 @@ -40,11 +42,9 @@ volatile 是 Java 中的关键字，是一个变量修饰符，用来修饰会 Java 语言提供了 volatile 和 synchronized 两个关键字来保证线程之间操作的有序性，volatile 是因为其本身包含“禁止指令重排序”的语义，synchronized 是由“一个变量在同一个时刻只允许一条线程对其进行 lock 操作“这条规则获得的，此规则决定了持有同一个对象锁的两个同步块只能串行执行。 ------- - -## 3. volatile 是 Java 虚拟机提供的轻量级的同步机制 +## 三、volatile 是 Java 虚拟机提供的轻量级的同步机制 - 保证可见性 - **不保证原子性** @@ -109,8 +109,8 @@ main execute over，main get number is:1 ```java class MyData { volatile int number = 0; - public void add() { - this.number = number + 1; + public void addPlusPlus(){ + number ++; } } @@ -120,7 +120,7 @@ private static void testAtomic() throws InterruptedException { for (int i = 0; i < 10; i++) { new Thread(() ->{ for (int j = 0; j < 1000; j++) { - myData.addPlusPlus(); + myData.add(); } },"addPlusThread:"+ i).start(); } @@ -141,7 +141,7 @@ private static void testAtomic() throws InterruptedException { final value：9856 ``` -为什么会这样呢，因为 `i++` 在转化为字节码指令的时候是4条指令 +为什么会这样呢，因为 `i++` 在转化为字节码指令的时候是 4 条指令 - `getfield` 获取原始值 - `iconst_1` 将值入栈 @@ -162,7 +162,7 @@ final value：9856 计算机在执行程序时，为了提高性能，编译器和处理器常常会对指令做重排，一般分为以下 3 种 -![img](https://tva1.sinaimg.cn/large/00831rSTly1gcrgrycnj0j31bs04k74y.jpg) +![](https://tva1.sinaimg.cn/large/00831rSTly1gcrgrycnj0j31bs04k74y.jpg) 处理器在进行重排序时必须要考虑指令之间的**数据依赖性**，我们叫做 `as-if-serial` 语义 @@ -220,39 +220,53 @@ public class Singleton { why ? -Because: `instance = new Singleton();` 初始化对象的过程其实并不是一个原子的操作，它会分为三部分执行， +Because: `instance = new Singleton();` 初始化对象的过程其实并不是一个原子的操作，它会分为三部分执行： 1. 给 instance 分配内存 2. 调用 instance 的构造函数来初始化对象 3. 将 instance 对象指向分配的内存空间（执行完这步 instance 就为非 null 了） -步骤 2 和 3 不存在数据依赖关系，如果虚拟机存在指令重排序优化，则步骤 2和 3 的顺序是无法确定的。如果A线程率先进入同步代码块并先执行了 3 而没有执行 2，此时因为 instance 已经非 null。这时候线程 B 在第一次检查的时候，会发现 instance 已经是 非null 了，就将其返回使用，但是此时 instance 实际上还未初始化，自然就会出错。所以我们要限制实例对象的指令重排，用 volatile 修饰（JDK 5 之前使用了 volatile 的双检锁是有问题的）。 +步骤 2 和 3 不存在数据依赖关系，如果虚拟机存在指令重排序优化，则步骤 2 和 3 的顺序是无法确定的。如果 A 线程率先进入同步代码块并先执行了 3 而没有执行 2，此时因为 instance 已经非 null。这时候线程 B 在第一次检查的时候，会发现 instance 已经是 非null 了，就将其返回使用，但是此时 instance 实际上还未初始化，自然就会出错。所以我们要限制实例对象的指令重排，用 volatile 修饰（JDK 5 之前使用了 volatile 的双检锁是有问题的）。 ------ -## 4. 原理 +## 四、原理 -volatile 可以保证线程可见性且提供了一定的有序性，但是无法保证原子性。在 JVM 底层是基于内存屏障实现的。 +volatile 可以保证线程可见性且提供了一定的有序性，但是无法保证原子性。在 JVM 底层是基于**内存屏障**实现的。 -- 当对非 volatile 变量进行读写的时候，每个线程先从内存拷贝变量到 CPU 缓存中。如果计算机有多个CPU，每个线程可能在不同的 CPU 上被处理，这意味着每个线程可以拷贝到不同的 CPU cache 中 +- 当对非 volatile 变量进行读写的时候，每个线程先从内存拷贝变量到 CPU 缓存中。如果计算机有多个 CPU，每个线程可能在不同的 CPU 上被处理，这意味着每个线程可以拷贝到不同的 CPU cache 中 - 而声明变量是 volatile 的，JVM 保证了每次读变量都从内存中读，跳过 CPU cache 这一步，所以就不会有可见性问题 - 对 volatile 变量进行写操作时，会在写操作后加一条 store 屏障指令，将工作内存中的共享变量刷新回主内存； - 对 volatile 变量进行读操作时，会在写操作后加一条 load 屏障指令，从主内存中读取共享变量； +> 可见性和“线程如何对变量进行操作(取值、赋值等)”有关系： +> +> 我们要先明确一个**定律**：线程对变量的所有操作(取值、赋值等)都必须在工作内存（各线程独立拥有）中进行，而不能直接读写内存中的变量，各工作内存间也不能相互访问。对于volatile变量来说，由于它特殊的操作顺序性规定，看起来如同操作主内存一般，但实际上 volatile变量也是遵循这一定律的。 +> +> 关于主存与工作内存之间具体的交互协议（即一个变量如何从主存拷贝到工作内存、如何从工作内存同步到主存等实现细节），Java内存模型中定义了以下八种操作来完成： +> +> 　　　　lock:(锁定)，unlock(解锁)，read(读取)，load(载入)，use(试用)， assign(赋值)，store(存储)，write(写入)。 +> +> volatile 对这八种操作有着两个特殊的限定，正因为有这些限定才让volatile修饰的变量有可见性以及可以禁止指令重排序 ： +> +> 1. use动作之前必须要有read和load动作， 这三个动作必须是连续出现的。【表示：每次工作内存要使用volatile变量之前必须去主存中拿取最新的volatile变量】 +> +> 2. assign动作之后必须跟着store和write动作，这三个动作必须是连续出现的。【表示: 每次工作内存改变了volatile变量的值，就必须把该值写回到主存中】 +> +> 有以上两条规则就能保证每个线程每次去拿volatile变量的时候，那个变量肯定是最新的， 其实也就相当于好多个线程用的是同一个内存，无工作内存和主存之分。而操作没有用volatile修饰的变量则不能保证每次都能获取到最新的变量值。 - -通过 hsdis 工具获取 JIT 编译器生成的汇编指令来看看对 volatile 进行写操作CPU会做什么事情，还是用上边的单例模式，可以看到 +通过 hsdis 工具获取 JIT 编译器生成的汇编指令来看看对 volatile 进行写操作 CPU 会做什么事情，还是用上边的单例模式，可以看到 ![](https://tva1.sinaimg.cn/large/00831rSTly1gdhgvi4pz9j30xf0tgjuy.jpg) -（PS：具体的汇编指令对我这个 Javaer 太南了，但是 JVM 字节码我们可以认识，`putstatic` 的含义是给一个静态变量设置值，那这里的 `putstatic instance` ,而且是第 17 行代码，更加确定是给 instance 赋值了。果然像各种资料里说的，找到了 `lock add1` 据说还得翻阅。这里可以看下这两篇 https://www.jianshu.com/p/6ab7c3db13c3 、 https://www.cnblogs.com/xrq730/p/7048693.html ） +（PS：具体的汇编指令对我这个 Javaer 太南了，但是 JVM 字节码我们可以认识，`putstatic` 的含义是给一个静态变量设置值，那这里的 `putstatic instance` ，而且是第 17 行代码，更加确定是给 instance 赋值了。果然像各种资料里说的，找到了 `lock add1` 据说还得翻阅。这里可以看下这两篇 https://www.jianshu.com/p/6ab7c3db13c3 、 https://www.cnblogs.com/xrq730/p/7048693.html ） -有 volatile 修饰的共享变量进行写操作时会多出第二行汇编代码，该句代码的意思是**对原值加零**，其中相加指令addl前有 **lock** 修饰。通过查IA-32架构软件开发者手册可知，lock前缀的指令在多核处理器下会引发两件事情： +有 volatile 修饰的共享变量进行写操作时会多出第二行汇编代码，该句代码的意思是**对原值加零**，其中相加指令addl 前有 **lock** 修饰。通过查 IA-32 架构软件开发者手册可知，lock 前缀的指令在多核处理器下会引发两件事情： - 将当前处理器缓存行的数据写回到系统内存 -- 这个写回内存的操作会引起在其他CPU里缓存了该内存地址的数据无效 +- 这个写回内存的操作会引起在其他 CPU 里缓存了该内存地址的数据无效 **正是 lock 实现了 volatile 的「防止指令重排」「内存可见」的特性** @@ -260,9 +274,9 @@ volatile 可以保证线程可见性且提供了一定的有序性，但是无 -## 5. 使用场景 +## 五、使用场景 -您只能在有限的一些情形下使用 volatile 变量替代锁。要使 volatile 变量提供理想的线程安全，必须同时满足下面两个条件： +你只能在有限的一些情形下使用 volatile 变量替代锁。要使 volatile 变量提供理想的线程安全，必须同时满足下面两个条件： - 对变量的写操作不依赖于当前值 - 该变量没有包含在具有其他变量的不变式中 @@ -273,13 +287,13 @@ volatile 可以保证线程可见性且提供了一定的有序性，但是无 -## 6. volatile 性能 +## 六、volatile 性能 volatile 的读性能消耗与普通变量几乎相同，但是写操作稍慢，因为它需要在本地代码中插入许多内存屏障指令来保证处理器不发生乱序执行。 引用《正确使用 volaitle 变量》一文中的话： -很难做出准确、全面的评价，例如 “X 总是比 Y 快”，尤其是对 JVM 内在的操作而言。（例如，某些情况下 JVM 也许能够完全删除锁机制，这使得我们难以抽象地比较 `volatile` 和 `synchronized` 的开销。）就是说，在目前大多数的处理器架构上，volatile 读操作开销非常低 —— 几乎和非 volatile 读操作一样。而 volatile 写操作的开销要比非 volatile 写操作多很多，因为要保证可见性需要实现内存界定（Memory Fence），即便如此，volatile 的总开销仍然要比锁获取低。 +> 很难做出准确、全面的评价，例如 “X 总是比 Y 快”，尤其是对 JVM 内在的操作而言。（例如，某些情况下 JVM 也许能够完全删除锁机制，这使得我们难以抽象地比较 `volatile` 和 `synchronized` 的开销。）就是说，在目前大多数的处理器架构上，volatile 读操作开销非常低 —— 几乎和非 volatile 读操作一样。而 volatile 写操作的开销要比非 volatile 写操作多很多，因为要保证可见性需要实现内存界定（Memory Fence），即便如此，volatile 的总开销仍然要比锁获取低。 volatile 操作不会像锁一样造成阻塞，因此，在能够安全使用 volatile 的情况下，volatile 可以提供一些优于锁的可伸缩特性。如果读操作的次数要远远超过写操作，与锁相比，volatile 变量通常能够减少同步的性能开销。 @@ -287,11 +301,7 @@ volatile 操作不会像锁一样造成阻塞，因此，在能够安全使用 v ## 参考 -《深入理解Java虚拟机》 - http://tutorials.jenkov.com/java-concurrency/java-memory-model.html - https://juejin.im/post/5dbfa0aa51882538ce1a4ebc -《正确使用 Volatile 变量》https://www.ibm.com/developerworks/cn/java/j-jtp06197.html - - - -![图怪兽_b2195efd95f95e83c90c74142a4b2001_47863.png](https://i.loli.net/2020/03/25/PaVfB1psAndLTOv.png) \ No newline at end of file +- 《深入理解Java虚拟机》 +- http://tutorials.jenkov.com/java-concurrency/java-memory-model.html +- https://juejin.im/post/5dbfa0aa51882538ce1a4ebc +- 《正确使用 Volatile 变量》https://www.ibm.com/developerworks/cn/java/j-jtp06197.html \ No newline at end of file diff --git "a/docs/java/JUC/\345\244\232\344\270\252\347\272\277\347\250\213\351\241\272\345\272\217\346\211\247\350\241\214\351\227\256\351\242\230.md" "b/docs/java/JUC/\345\244\232\344\270\252\347\272\277\347\250\213\351\241\272\345\272\217\346\211\247\350\241\214\351\227\256\351\242\230.md" new file mode 100644 index 0000000000..a61a89e83e --- /dev/null +++ "b/docs/java/JUC/\345\244\232\344\270\252\347\272\277\347\250\213\351\241\272\345\272\217\346\211\247\350\241\214\351\227\256\351\242\230.md" @@ -0,0 +1,472 @@ +# 手撕面试题：多个线程顺序执行问题 + +大家在换工作面试中，除了一些常规算法题，还会遇到各种需要手写的题目，所以打算总结出来，给大家个参考。 + +第一篇打算总结下阿里最喜欢问的多个线程顺序打印问题，我遇到的是机试，直接写出运行。同类型的题目有很多，比如 + +1. 三个线程分别打印 A，B，C，要求这三个线程一起运行，打印 n 次，输出形如“ABCABCABC....”的字符串 +2. 两个线程交替打印 0~100 的奇偶数 +3. 通过 N 个线程顺序循环打印从 0 至 100 +4. 多线程按顺序调用，A->B->C，AA 打印 5 次，BB 打印10 次，CC 打印 15 次，重复 10 次 +5. 用两个线程，一个输出字母，一个输出数字，交替输出 1A2B3C4D...26Z + +其实这类题目考察的都是**线程间的通信问题**，基于这类题目，做一个整理，方便日后手撕面试官，文明的打工人，手撕面试题。 + +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20201029114231.jpg) + +## 使用 Lock + +我们以第一题为例：三个线程分别打印 A，B，C，要求这三个线程一起运行，打印 n 次，输出形如“ABCABCABC....”的字符串。 + +思路：使用一个取模的判断逻辑 **C%M ==N**，题为 3 个线程，所以可以按取模结果编号：0、1、2，他们与 3 取模结果仍为本身，则执行打印逻辑。 + +```java +public class PrintABCUsingLock { + + private int times; // 控制打印次数 + private int state; // 当前状态值：保证三个线程之间交替打印 + private Lock lock = new ReentrantLock(); + + public PrintABCUsingLock(int times) { + this.times = times; + } + + private void printLetter(String name, int targetNum) { + for (int i = 0; i < times; ) { + lock.lock(); + if (state % 3 == targetNum) { + state++; + i++; + System.out.print(name); + } + lock.unlock(); + } + } + + public static void main(String[] args) { + PrintABCUsingLock loopThread = new PrintABCUsingLock(1); + + new Thread(() -> { + loopThread.printLetter("B", 1); + }, "B").start(); + + new Thread(() -> { + loopThread.printLetter("A", 0); + }, "A").start(); + + new Thread(() -> { + loopThread.printLetter("C", 2); + }, "C").start(); + } +} +``` + +main 方法启动后，3 个线程会抢锁，但是 state 的初始值为 0，所以第一次执行 if 语句的内容只能是 **线程 A**，然后还在 for 循环之内，此时 `state = 1`，只有 **线程 B** 才满足 `1% 3 == 1`，所以第二个执行的是 B，同理只有 **线程 C** 才满足 `2% 3 == 2`，所以第三个执行的是 C，执行完 ABC 之后，才去执行第二次 for 循环，所以要把 i++ 写在 for 循环里边，不能写成 `for (int i = 0; i < times;i++)` 这样。 + + + +## 使用 wait/notify + +其实遇到这类型题目，好多同学可能会先想到的就是 join()，或者 wati/notify 这样的思路。算是比较传统且万能的解决方案。也有些面试官会要求不能使用这种方式。 + +思路：还是以第一题为例，我们用对象监视器来实现，通过 `wait` 和 `notify()` 方法来实现等待、通知的逻辑，A 执行后，唤醒 B，B 执行后唤醒 C，C 执行后再唤醒 A，这样循环的等待、唤醒来达到目的。 + +```java +public class PrintABCUsingWaitNotify { + + private int state; + private int times; + private static final Object LOCK = new Object(); + + public PrintABCUsingWaitNotify(int times) { + this.times = times; + } + + public static void main(String[] args) { + PrintABCUsingWaitNotify printABC = new PrintABCUsingWaitNotify(10); + new Thread(() -> { + printABC.printLetter("A", 0); + }, "A").start(); + new Thread(() -> { + printABC.printLetter("B", 1); + }, "B").start(); + new Thread(() -> { + printABC.printLetter("C", 2); + }, "C").start(); + } + + private void printLetter(String name, int targetState) { + for (int i = 0; i < times; i++) { + synchronized (LOCK) { + while (state % 3 != targetState) { + try { + LOCK.wait(); + } catch (InterruptedException e) { + e.printStackTrace(); + } + } + state++; + System.out.print(name); + LOCK.notifyAll(); + } + } + } +} +``` + + + +同样的思路，来解决下第 2 题：两个线程交替打印奇数和偶数 + +使用对象监视器实现，两个线程 A、B 竞争同一把锁，只要其中一个线程获取锁成功，就打印 ++i，并通知另一线程从等待集合中释放，然后自身线程加入等待集合并释放锁即可。 + +![图：throwable-blog](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20201029160341.png) + +```java +public class OddEvenPrinter { + + private Object monitor = new Object(); + private final int limit; + private volatile int count; + + OddEvenPrinter(int initCount, int times) { + this.count = initCount; + this.limit = times; + } + + public static void main(String[] args) { + + OddEvenPrinter printer = new OddEvenPrinter(0, 10); + new Thread(printer::print, "odd").start(); + new Thread(printer::print, "even").start(); + } + + private void print() { + synchronized (monitor) { + while (count < limit) { + try { + System.out.println(String.format("线程[%s]打印数字:%d", Thread.currentThread().getName(), ++count)); + monitor.notifyAll(); + monitor.wait(); + } catch (InterruptedException e) { + e.printStackTrace(); + } + } + //防止有子线程被阻塞未被唤醒，导致主线程不退出 + monitor.notifyAll(); + } + } +} +``` + + + +同样的思路，来解决下第 5 题：用两个线程，一个输出字母，一个输出数字，交替输出 1A2B3C4D...26Z + +```java +public class NumAndLetterPrinter { + private static char c = 'A'; + private static int i = 0; + static final Object lock = new Object(); + + public static void main(String[] args) { + new Thread(() -> printer(), "numThread").start(); + new Thread(() -> printer(), "letterThread").start(); + } + + private static void printer() { + synchronized (lock) { + for (int i = 0; i < 26; i++) { + if (Thread.currentThread().getName() == "numThread") { + //打印数字1-26 + System.out.print((i + 1)); + // 唤醒其他在等待的线程 + lock.notifyAll(); + try { + // 让当前线程释放锁资源，进入wait状态 + lock.wait(); + } catch (InterruptedException e) { + e.printStackTrace(); + } + } else if (Thread.currentThread().getName() == "letterThread") { + // 打印字母A-Z + System.out.print((char) ('A' + i)); + // 唤醒其他在等待的线程 + lock.notifyAll(); + try { + // 让当前线程释放锁资源，进入wait状态 + lock.wait(); + } catch (InterruptedException e) { + e.printStackTrace(); + } + } + } + lock.notifyAll(); + } + } +} +``` + + + +## 使用 Lock/Condition + +还是以第一题为例，使用 Condition 来实现，其实和 wait/notify 的思路一样。 + +> Condition 中的 `await()` 方法相当于 Object 的 `wait()` 方法，Condition 中的 `signal()` 方法相当于Object 的 `notify()` 方法，Condition 中的 `signalAll()` 相当于 Object 的 `notifyAll()` 方法。 +> +> 不同的是，Object 中的 `wait(),notify(),notifyAll()`方法是和`"同步锁"`(synchronized关键字)捆绑使用的；而 Condition 是需要与`"互斥锁"/"共享锁"`捆绑使用的。 + +```java +public class PrintABCUsingLockCondition { + + private int times; + private int state; + private static Lock lock = new ReentrantLock(); + private static Condition c1 = lock.newCondition(); + private static Condition c2 = lock.newCondition(); + private static Condition c3 = lock.newCondition(); + + public PrintABCUsingLockCondition(int times) { + this.times = times; + } + + public static void main(String[] args) { + PrintABCUsingLockCondition print = new PrintABCUsingLockCondition(10); + new Thread(() -> { + print.printLetter("A", 0, c1, c2); + }, "A").start(); + new Thread(() -> { + print.printLetter("B", 1, c2, c3); + }, "B").start(); + new Thread(() -> { + print.printLetter("C", 2, c3, c1); + }, "C").start(); + } + + private void printLetter(String name, int targetState, Condition current, Condition next) { + for (int i = 0; i < times; ) { + lock.lock(); + try { + while (state % 3 != targetState) { + current.await(); + } + state++; + i++; + System.out.print(name); + next.signal(); + } catch (Exception e) { + e.printStackTrace(); + } finally { + lock.unlock(); + } + } + } +} +``` + + + +使用 Lock 锁的多个 Condition 可以实现精准唤醒，所以碰到那种多个线程交替打印不同次数的题就比较容易想到，比如解决第四题：多线程按顺序调用，A->B->C，AA 打印 5 次，BB 打印10 次，CC 打印 15 次，重复 10 次 + +代码就不贴了，思路相同。 + + + +> 以上几种方式，其实都会存在一个锁的抢夺过程，如果抢锁的的线程数量足够大，就会出现很多线程抢到了锁但不该自己执行，然后就又解锁或 wait() 这种操作，这样其实是有些浪费资源的。 + + + +## 使用 Semaphore + +> 在信号量上我们定义两种操作： 信号量主要用于两个目的，一个是用于多个共享资源的互斥使用，另一个用于并发线程数的控制。 +> +> 1. acquire（获取） 当一个线程调用 acquire 操作时，它要么通过成功获取信号量（信号量减1），要么一直等下去，直到有线程释放信号量，或超时。 +> 2. release（释放）实际上会将信号量的值加1，然后唤醒等待的线程。 + +先看下如何解决第一题：三个线程循环打印 A，B，C + +```java +public class PrintABCUsingSemaphore { + private int times; + private static Semaphore semaphoreA = new Semaphore(1); // 只有A 初始信号量为1,第一次获取到的只能是A + private static Semaphore semaphoreB = new Semaphore(0); + private static Semaphore semaphoreC = new Semaphore(0); + + public PrintABCUsingSemaphore(int times) { + this.times = times; + } + + public static void main(String[] args) { + PrintABCUsingSemaphore printer = new PrintABCUsingSemaphore(1); + new Thread(() -> { + printer.print("A", semaphoreA, semaphoreB); + }, "A").start(); + + new Thread(() -> { + printer.print("B", semaphoreB, semaphoreC); + }, "B").start(); + + new Thread(() -> { + printer.print("C", semaphoreC, semaphoreA); + }, "C").start(); + } + + private void print(String name, Semaphore current, Semaphore next) { + for (int i = 0; i < times; i++) { + try { + System.out.println("111" + Thread.currentThread().getName()); + current.acquire(); // A获取信号执行,A信号量减1,当A为0时将无法继续获得该信号量 + System.out.print(name); + next.release(); // B释放信号，B信号量加1（初始为0），此时可以获取B信号量 + System.out.println("222" + Thread.currentThread().getName()); + } catch (InterruptedException e) { + e.printStackTrace(); + } + } + } +} +``` + + + +如果题目中是多个线程循环打印的话，一般使用信号量解决是效率较高的方案，上一个线程持有下一个线程的信号量，通过一个信号量数组将全部关联起来，这种方式不会存在浪费资源的情况。 + +接着用信号量的方式解决下第三题：通过 N 个线程顺序循环打印从 0 至 100 + +```java +public class LoopPrinter { + + private final static int THREAD_COUNT = 3; + static int result = 0; + static int maxNum = 10; + + public static void main(String[] args) throws InterruptedException { + final Semaphore[] semaphores = new Semaphore[THREAD_COUNT]; + for (int i = 0; i < THREAD_COUNT; i++) { + //非公平信号量，每个信号量初始计数都为1 + semaphores[i] = new Semaphore(1); + if (i != THREAD_COUNT - 1) { + System.out.println(i+"==="+semaphores[i].getQueueLength()); + //获取一个许可前线程将一直阻塞, for 循环之后只有 syncObjects[2] 没有被阻塞 + semaphores[i].acquire(); + } + } + for (int i = 0; i < THREAD_COUNT; i++) { + // 初次执行，上一个信号量是 syncObjects[2] + final Semaphore lastSemphore = i == 0 ? semaphores[THREAD_COUNT - 1] : semaphores[i - 1]; + final Semaphore currentSemphore = semaphores[i]; + final int index = i; + new Thread(() -> { + try { + while (true) { + // 初次执行，让第一个 for 循环没有阻塞的 syncObjects[2] 先获得令牌阻塞了 + lastSemphore.acquire(); + System.out.println("thread" + index + ": " + result++); + if (result > maxNum) { + System.exit(0); + } + // 释放当前的信号量，syncObjects[0] 信号量此时为 1，下次 for 循环中上一个信号量即为syncObjects[0] + currentSemphore.release(); + } + } catch (Exception e) { + e.printStackTrace(); + } + }).start(); + } + } +} +``` + + + +## 使用 LockSupport + +LockSupport 是 JDK 底层的基于 `sun.misc.Unsafe` 来实现的类，用来创建锁和其他同步工具类的基本线程阻塞原语。它的静态方法`unpark()`和`park()`可以分别实现阻塞当前线程和唤醒指定线程的效果，所以用它解决这样的问题会更容易一些。 + +（在 AQS 中，就是通过调用 `LockSupport.park( )`和 `LockSupport.unpark()` 来实现线程的阻塞和唤醒的。） + +```java +public class PrintABCUsingLockSupport { + + private static Thread threadA, threadB, threadC; + + public static void main(String[] args) { + threadA = new Thread(() -> { + for (int i = 0; i < 10; i++) { + // 打印当前线程名称 + System.out.print(Thread.currentThread().getName()); + // 唤醒下一个线程 + LockSupport.unpark(threadB); + // 当前线程阻塞 + LockSupport.park(); + } + }, "A"); + threadB = new Thread(() -> { + for (int i = 0; i < 10; i++) { + // 先阻塞等待被唤醒 + LockSupport.park(); + System.out.print(Thread.currentThread().getName()); + // 唤醒下一个线程 + LockSupport.unpark(threadC); + } + }, "B"); + threadC = new Thread(() -> { + for (int i = 0; i < 10; i++) { + // 先阻塞等待被唤醒 + LockSupport.park(); + System.out.print(Thread.currentThread().getName()); + // 唤醒下一个线程 + LockSupport.unpark(threadA); + } + }, "C"); + threadA.start(); + threadB.start(); + threadC.start(); + } +} +``` + +理解了思路，解决其他问题就容易太多了。 + +比如，我们再解决下第五题：用两个线程，一个输出字母，一个输出数字，交替输出 1A2B3C4D...26Z + +```java +public class NumAndLetterPrinter { + + private static Thread numThread, letterThread; + + public static void main(String[] args) { + letterThread = new Thread(() -> { + for (int i = 0; i < 26; i++) { + System.out.print((char) ('A' + i)); + LockSupport.unpark(numThread); + LockSupport.park(); + } + }, "letterThread"); + + numThread = new Thread(() -> { + for (int i = 1; i <= 26; i++) { + System.out.print(i); + LockSupport.park(); + LockSupport.unpark(letterThread); + } + }, "numThread"); + numThread.start(); + letterThread.start(); + } +} +``` + + + +## 写在最后 + +好了，以上就是常用的五种实现方案，多练习几次，手撕没问题。 + +当然，这类问题，解决方式不止是我列出的这些，还会有 join、CountDownLatch、也有放在队列里解决的，思路有很多，面试官想考察的其实只是对多线程的编程功底，其实自己练习的时候，是个很好的巩固理解 JUC 的过程。 + +> 以梦为马，越骑越傻。诗和远方，越走越慌。不忘初心是对的，但切记要出发，加油吧，程序员。 + +> 在路上的你，可以微信搜「 **JavaKeeper** 」一起前行，无套路领取 500+ 本电子书和 30+ 视频教学和源码，本文 **GitHub** [github.com/JavaKeeper](https://github.com/Jstarfish/JavaKeeper) 已经收录，服务端开发、面试必备技能兵器谱，有你想要的。 + diff --git "a/docs/java/JUC/\345\244\232\347\272\277\347\250\213\344\270\216\351\253\230\345\271\266\345\217\221\345\274\200\347\257\207.md" "b/docs/java/JUC/\345\244\232\347\272\277\347\250\213\344\270\216\351\253\230\345\271\266\345\217\221\345\274\200\347\257\207.md" new file mode 100644 index 0000000000..f1103e0cc5 --- /dev/null +++ "b/docs/java/JUC/\345\244\232\347\272\277\347\250\213\344\270\216\351\253\230\345\271\266\345\217\221\345\274\200\347\257\207.md" @@ -0,0 +1,27 @@ +多线程与高并发是 Java 程序员绕不开的坎 + + + +主要包括： + +- 一些基本概念，什么线程、进程、同步、阻塞 +- JUC 同步工具，就是各种同步锁 +- 同步容器 +- 线程池 + + + + + +作为一名 Javaer，想要进一个差不多点的互联网公司有个好的薪水，面试时呈现出了两个方向的现象： + +一个是上天 + +- 项目经验 +- 高并发、缓存、大数据量的架构设计 + +一个是入地 + +- 各种基础，数据结构，算法 +- JVM OS 线程 IO 等 + diff --git "a/docs/java/JUC/\345\246\202\344\275\225\350\256\251HashMap\347\272\277\347\250\213\345\256\211\345\205\250.md" "b/docs/java/JUC/\345\246\202\344\275\225\350\256\251HashMap\347\272\277\347\250\213\345\256\211\345\205\250.md" new file mode 100644 index 0000000000..6be1ae6b92 --- /dev/null +++ "b/docs/java/JUC/\345\246\202\344\275\225\350\256\251HashMap\347\272\277\347\250\213\345\256\211\345\205\250.md" @@ -0,0 +1,39 @@ +> HashMap 如何保证线程安全 + + + +无论哪个级别的 Javaer，都会经常用到 HashMap，我们也知道她不是线程安全的，那为什么不是线程安全的呢？ + + + + + +> 什么时候会使用HashMap？他有什么特点？ +> +> 你知道HashMap的工作原理吗？ +> +> 你知道get和put的原理吗？equals()和hashCode()的都有什么作用？ +> +> 你知道hash的实现吗？为什么要这样实现？ +> +> 如果HashMap的大小超过了负载因子(load factor)定义的容量，怎么办？ +> +> * HashMap +> * TreeMap +> * Hashtable +> * SortedMap +> +> 用过没，有什么区别 + + + + + +HashMap 在多线程环境下存在线程安全问题，那你一般都是怎么处理这种情况的？ + +一般在多线程的场景，可以通过以下三种方法来实现： + +- 替换成 Hashtable，Hashtable 通过对整个表上锁实现线程安全，因此效率比较低 +- 使用 Collections 类的 synchronizedMap 方法包装一下 +- 使用 ConcurrentHashMap，它使用分段锁来保证线程安全 + diff --git "a/docs/java/JUC/\346\255\273\351\224\201-\346\264\273\351\224\201.md" "b/docs/java/JUC/\346\255\273\351\224\201-\346\264\273\351\224\201.md" new file mode 100644 index 0000000000..e69de29bb2 diff --git "a/docs/java/JUC/\350\201\212\350\201\212\347\272\277\347\250\213\345\256\211\345\205\250.md" "b/docs/java/JUC/\350\201\212\350\201\212\347\272\277\347\250\213\345\256\211\345\205\250.md" new file mode 100644 index 0000000000..24874d845f --- /dev/null +++ "b/docs/java/JUC/\350\201\212\350\201\212\347\272\277\347\250\213\345\256\211\345\205\250.md" @@ -0,0 +1,17 @@ +# 聊聊线程安全 + +> 《深入理解 Java 虚拟机》在“高效并发”部分中有这么一段话： +> +> 并发处理的广泛应用是 Amdahl 定律代替摩尔定律称为计算机性能发展源动力的根本原因，也是人类压榨计算机运算能力的有力武器。 + +## 什么是线程安全 + + + +## 线程安全的实现方法 + +互斥同步 + +非阻塞同步 + +无同步方案 \ No newline at end of file diff --git a/docs/java/JVM/.DS_Store b/docs/java/JVM/.DS_Store new file mode 100644 index 0000000000..b40385dc7f Binary files /dev/null and b/docs/java/JVM/.DS_Store differ diff --git "a/docs/java/JVM/\347\261\273\345\212\240\350\275\275\345\255\220\347\263\273\347\273\237.md" b/docs/java/JVM/Class-Loading.md similarity index 50% rename from "docs/java/JVM/\347\261\273\345\212\240\350\275\275\345\255\220\347\263\273\347\273\237.md" rename to docs/java/JVM/Class-Loading.md index 63b8a4d3b0..d23921dc36 100644 --- "a/docs/java/JVM/\347\261\273\345\212\240\350\275\275\345\255\220\347\263\273\347\273\237.md" +++ b/docs/java/JVM/Class-Loading.md @@ -1,17 +1,17 @@ +# 类加载子系统 + > 带着问题，尤其是面试问题的学习才是最高效的。加油，奥利给！ > > 点赞+收藏 就学会系列，文章收录在 GitHub [JavaKeeper](https://github.com/Jstarfish/JavaKeeper) ，N线互联网开发必备技能兵器谱 ## 直击面试 -1. 看你简历写得熟悉JVM，那你说说类的加载过程吧？ -2. 我们可以自定义一个String类来使用吗？ +1. 看你简历写得熟悉 JVM，那你说说类的加载过程吧？ +2. 我们可以自定义一个 String 类来使用吗？ 3. 什么是类加载器，类加载器有哪些？这些类加载器都加载哪些文件？ 4. 多线程的情况下，类的加载为什么不会出现重复加载的情况？ 5. 什么是双亲委派机制？它有啥优势？可以打破这种机制吗？ -![](https://tva1.sinaimg.cn/large/0082zybply1gc0vxdcutjj30dw10hju9.jpg) - ------ ## 类加载子系统 @@ -34,11 +34,11 @@ -## 类加载器ClassLoader角色 +## 类加载器 ClassLoader 角色 -1. class file 存在于本地硬盘上，可以理解为设计师画在纸上的模板，而最终这个模板在执行的时候是要加载到JVM 当中来根据这个文件实例化出 n 个一模一样的实例 +1. class file 存在于本地硬盘上，可以理解为设计师画在纸上的模板，而最终这个模板在执行的时候是要加载到 JVM 当中来根据这个文件实例化出 n 个一模一样的实例 2. class file 加载到 JVM 中，被称为 DNA 元数据模板，放在方法区 -3. 在.calss文件 -> JVM -> 最终成为元数据模板，此过程就要一个运输工具（类装载器），扮演一个快递员的角色 +3. 在 .calss 文件 -> JVM -> 最终成为元数据模板，此过程就要一个运输工具（类装载器），扮演一个快递员的角色 ------ @@ -46,24 +46,24 @@ ## 类加载过程 -类从被加载到虚拟机内存中开始，到卸载出内存为止，它的整个生命周期包括：**加载、验证、准备、解析、初始化、使用和卸载**七个阶段。(验证、准备和解析又统称为连接，为了支持Java语言的**运行时绑定**，所以**解析阶段也可以是在初始化之后进行的**。以上顺序都只是说开始的顺序，实际过程中是交叉的混合式进行的，加载过程中可能就已经开始验证了) +类从被加载到虚拟机内存中开始，到卸载出内存为止，它的整个生命周期包括：**加载、验证、准备、解析、初始化、使用和卸载**七个阶段。(验证、准备和解析又统称为连接，为了支持 Java 语言的**运行时绑定**，所以**解析阶段也可以是在初始化之后进行的**。以上顺序都只是说开始的顺序，实际过程中是交叉的混合式进行的，加载过程中可能就已经开始验证了) -![jvm-class-load](https://tva1.sinaimg.cn/large/0082zybply1gbnxhplvkrj30yi0d60ty.jpg) +![jvm-class-load](https://img.starfish.ink/jvm/jvm-class-load.png) ### 1. 加载（Loading）： -1. 通过一个类的全限定名获取定义此类的二进制字节流 +1. 通过一个类的全限定名获取定义此类的二进制字节流（简单点说就是找到文件系统中/jar 包中/或存在于任何地方的“`class 文件`”。 如果找不到二进制表示形式，则会抛出 `NoClassDefFound` 错误。） 2. 将这个字节流所代表的的静态存储结构转化为方法区的运行时数据结构 3. **在内存中生成一个代表这个类的 `java.lang.Class` 对象**，作为方法区这个类的各种数据的访问入口 -加载 `.calss` 文件的方式 +加载 `.class` 文件的方式 - 从本地系统中直接加载 - 通过网络获取，典型场景：Web Applet -- 从zip压缩文件中读取，成为日后jar、war格式的基础 +- 从 zip 压缩文件中读取，成为日后 jar、war 格式的基础 - 运行时计算生成，使用最多的是：动态代理技术 - 由其他文件生成，比如 JSP 应用 -- 从专有数据库提取.class 文件，比较少见 +- 从专有数据库提取 .class 文件，比较少见 - 从加密文件中获取，典型的防 Class 文件被反编译的保护措施 ### 2. 连接（Linking） @@ -71,12 +71,19 @@ #### 验证（Verify） - 目的在于确保 Class 文件的字节流中包含信息符合当前虚拟机要求，保证被加载类的正确性，不会危害虚拟机自身安全 - - 主要包括四种验证，**文件格式验证，元数据验证，字节码验证，符号引用验证** +> 校验过程检查 classfile 的语义，判断常量池中的符号，并执行类型检查， 主要目的是判断字节码的合法性，比如 magic number, 对版本号进行验证。 这些检查过程中可能会抛出 `VerifyError`， `ClassFormatError` 或 `UnsupportedClassVersionError`。 +> +> 因为 classfile 的验证属是链接阶段的一部分，所以这个过程中可能需要加载其他类，在某个类的加载过程中，JVM 必须加载其所有的超类和接口。 +> +> 如果类层次结构有问题（例如，该类是自己的超类或接口,死循环了），则 JVM 将抛出 `ClassCircularityError`。 而如果实现的接口并不是一个 interface，或者声明的超类是一个 interface，也会抛出 `IncompatibleClassChangeError`。 + #### 准备（Prepare） -- 为类变量分配内存并且设置该类变量的默认初始值，即**零值** +然后进入准备阶段，这个阶段将会创建静态字段, 并将其初始化为标准默认值(比如`null`或者`0 值`)，并分配方法表，即在方法区中分配这些变量所使用的内存空间。 + +- 为**类变量**分配内存并且设置该类变量的默认初始值，即**零值** | 数据类型 | 零值 | | --------- | -------- | | int | 0 | @@ -89,7 +96,7 @@ | double | 0.0d | | reference | null | - 这里不包含用 final 修饰的 static，因为 final 在编译的时候就会分配了，准备阶段会显示初始化 -- 这里**不会为实例变量分配初始化**，类变量会分配在**方法区**中，而实例变量是会随着对象一起分配到 Java 堆中 +- 这里**不会为实例变量分配初始化**，类变量会分配在**方法区**中，而实例变量是会在对象实例化时随着对象一起分配到 Java 堆中 ```java private static int i = 1; //变量i在准备阶只会被赋值为0，初始化时才会被赋值为1 @@ -98,19 +105,33 @@ #### 解析（Resolve） +然后进入可选的解析符号引用阶段。 也就是解析常量池，主要有以下四种：类或接口的解析、字段解析、类方法解析、接口方法解析。 + - 将常量池内的符号引用转换为直接引用的过程 - 事实上，解析操作往往会伴随着 JVM 在执行完初始化之后再执行 -- 符号引用就是一组符号来描述所引用的目标。符号引用的字面量形式明确定义在《Java虚拟机规范》的Class文件格式中。直接引用就是直接指向目标的指针、相对偏移量或一个间接定位到目标的句柄 +- 符号引用就是一组符号来描述所引用的目标。符号引用的字面量形式明确定义在《Java虚拟机规范》的 Class文件格式中。直接引用就是直接指向目标的指针、相对偏移量或一个间接定位到目标的句柄（如果有了直接引用，那引用的目标必定在堆中存在） - 解析动作主要针对类或接口、字段、类方法、接口方法、方法类型等。对应常量池中的`CONSTANT_Class_info`、`CONSTANT_Fieldref_info`、`CONSTANT_Methodref_info`等 +> [《JVM里的符号引用如何存储？》](https://www.zhihu.com/question/30300585) + ### 3. 初始化（Initialization） -- 初始化阶段就是执行**类构造器方法**\()的过程 -- 此方法不需要定义，是 javac 编译器自动收集类中的所有类变量的赋值动作和静态代码块中的语句合并而来 -- 构造器方法中指令按语句在源文件中出现的顺序执行 -- \()不同于类的构造器（构造器是虚拟机视角下的\()） -- 若该类具有父类，JVM会保证子类的\()执行前，父类的\()已经执行完毕 -- 虚拟机必须保证一个类的\()方法在多线程下被同步加锁 +JVM 规范明确规定, 必须在类的首次“主动使用”时才能执行类初始化。 + +初始化的过程包括执行： + +- 类构造器方法 +- static 静态变量赋值语句 +- static 静态代码块 + +如果是一个子类进行初始化会先对其父类进行初始化，保证其父类在子类之前进行初始化。所以其实在 java 中初始化一个类，那么必然先初始化过 `java.lang.Object` 类，因为所有的 java 类都继承自 java.lang.Object。 + +> - 初始化阶段就是执行**类构造器方法** `()` 的过程 +> - 此方法不需要定义，是 javac 编译器自动收集类中的所有类变量的赋值动作和静态代码块中的语句合并而来 +> - 构造器方法中指令按语句在源文件中出现的顺序执行 +> - `()` 不同于类的构造器（构造器是虚拟机视角下的 `()`） +> - 若该类具有父类，JVM 会保证子类的 `()` 执行前，父类的 `()` 已经执行完毕 +> - 虚拟机必须保证一个类的 `()` 方法在多线程下被同步加锁 ```java public class ClassInitTest{ @@ -130,7 +151,9 @@ public class ClassInitTest{ -## 类的主动使用和被动使用 +## 类加载时机 + +#### Java类何时会被加载 Java 程序对类的使用方式分为：主动使用和被动使用。虚拟机规范规定**有且只有 5 种情况必须立即对类进行“初始化”**，即类的主动使用。 @@ -138,10 +161,32 @@ Java 程序对类的使用方式分为：主动使用和被动使用。虚拟机 - 反射 - 初始化一个类的子类 - Java 虚拟机启动时被标明为启动类的类 -- JDK7 开始提供的动态语言支持：`java.lang.invoke.MethodHandle`实例的解析结果，`REF_getStatic`、`REF_putStatic`、`REF_invokeStatic`句柄对应的类没有初始化，则初始化 +- JDK7 开始提供的动态语言支持：`java.lang.invoke.MethodHandle` 实例的解析结果，`REF_getStatic`、`REF_putStatic`、`REF_invokeStatic` 句柄对应的类没有初始化，则初始化 除以上五种情况，其他使用 Java 类的方式被看作是对**类的被动使用**，都不**会导致类的初始化**。 +> JVM 规范枚举了下述多种触发情况： +> +> - 当虚拟机启动时，初始化用户指定的主类，就是启动执行的 main 方法所在的类； +> - 当遇到用以新建目标类实例的 new 指令时，初始化 new 指令的目标类，就是 new 一个类的时候要初始化； +> - 当遇到调用静态方法的指令时，初始化该静态方法所在的类； +> - 当遇到访问静态字段的指令时，初始化该静态字段所在的类； +> - 子类的初始化会触发父类的初始化； +> - 如果一个接口定义了 default 方法，那么直接实现或者间接实现该接口的类的初始化，会触发该接口的初始化； +> - 使用反射 API 对某个类进行反射调用时，初始化这个类，其实跟前面一样，反射调用要么是已经有实例了，要么是静态方法，都需要初始化； +> - 当初次调用 MethodHandle 实例时，初始化该 MethodHandle 指向的方法所在的类。 +> +> 同时以下几种情况不会执行类初始化： +> +> - 通过子类引用父类的静态字段，只会触发父类的初始化，而不会触发子类的初始化。 +> - 定义对象数组，不会触发该类的初始化。 +> - 常量在编译期间会存入调用类的常量池中，本质上并没有直接引用定义常量的类，不会触发定义常量所在的类。 +> - 通过类名获取 Class 对象，不会触发类的初始化，Hello.class 不会让 Hello 类初始化。 +> - 通过 Class.forName 加载指定类时，如果指定参数 initialize 为 false 时，也不会触发类初始化，其实这个参数是告诉虚拟机，是否要对类进行初始化。Class.forName(“jvm.Hello”)默认会加载 Hello 类。 +> - 通过 ClassLoader 默认的 loadClass 方法，也不会触发初始化动作（加载了，但是不初始化）。 +> +> 示例: 诸如 Class.forName(), classLoader.loadClass() 等 Java API, 反射API, 以及 JNI_FindClass 都可以启动类加载。 JVM 本身也会进行类加载。 比如在 JVM 启动时加载核心类，java.lang.Object, java.lang.Thread 等等。 + #### eg: ```java @@ -177,6 +222,18 @@ class SubClass extends SuperClass { - 从概念上来讲，自定义类加载器一般指的是程序中由开发人员自定义的一类类加载器，但是 Java 虚拟机规范却没有这么定义，而是将所有派生于抽象类 ClassLoader 的类加载器都划分为自定义类加载器 +系统自带的类加载器分为三种： + +- 启动类加载器（BootstrapClassLoader） +- 扩展类加载器（ExtClassLoader） +- 应用类加载器（AppClassLoader） + +一般启动类加载器是由 JVM 内部实现的，在 Java 的 API 里无法拿到，但是我们可以侧面看到和影响它。后 2 种类加载器在 Oracle Hotspot JVM 里，都是在中`sun.misc.Launcher`定义的，扩展类加载器和应用类加载器一般都继承自`URLClassLoader`类，这个类也默认实现了从各种不同来源加载 class 字节码转换成 Class 的方法。 + +![classloader](https://tva1.sinaimg.cn/large/e6c9d24ely1h35g2zl2pdj223y0u042i.jpg) + +> 不同类加载器看似是继承(Inheritance)关系，实际是采用组合关系来复用父类加载器的相关代码 + #### 启动类加载器（引导类加载器，Bootstrap ClassLoader） @@ -185,21 +242,21 @@ class SubClass extends SuperClass { - 它用来加载 Java 的核心库（`JAVA_HOME/jre/lib/rt.jar`、`resource.jar`或`sun.boot.class.path`路径下的内容），用于提供 JVM 自身需要的类 - 并不继承自 `java.lang.ClassLoader`，没有父加载器 - 加载扩展类和应用程序类加载器，并指定为他们的父类加载器 -- 出于安全考虑，Bootstrap 启动类加载器只加载名为java、Javax、sun等开头的类 +- 出于安全考虑，Bootstrap 启动类加载器只加载名为 java、Javax、sun 等开头的类 #### 扩展类加载器（Extension ClassLoader） -- Java 语言编写，由`sun.misc.Launcher$ExtClassLoader`实现 +- Java 语言编写，由 `sun.misc.Launcher$ExtClassLoader` 实现 - 派生于 ClassLoader - 父类加载器为启动类加载器 -- 从 `java.ext.dirs` 系统属性所指定的目录中加载类库，或从 JDK 的安装目录的`jre/lib/ext` 子目录（扩展目录）下加载类库。如果用户创建的 JAR 放在此目录下，也会自动由扩展类加载器加载 +- 从 `java.ext.dirs` 系统属性所指定的目录中加载类库，或从 JDK 的安装目录的 `jre/lib/ext` 子目录（扩展目录）下加载类库。如果用户创建的 JAR 放在此目录下，也会自动由扩展类加载器加载 #### 应用程序类加载器（也叫系统类加载器，AppClassLoader） - Java 语言编写，由 `sun.misc.Lanucher$AppClassLoader` 实现 - 派生于 ClassLoader - 父类加载器为扩展类加载器 -- 它负责加载环境变量`classpath`或系统属性` java.class.path` 指定路径下的类库 +- 它负责加载环境变量 `classpath` 或系统属性 ` java.class.path` 指定路径下的类库 - 该类加载是**程序中默认的类加载器**，一般来说，Java 应用的类都是由它来完成加载的 - 通过 `ClassLoader#getSystemClassLoader()` 方法可以获取到该类加载器 @@ -241,6 +298,8 @@ public class ClassLoaderTest { 在 Java 的日常应用程序开发中，类的加载几乎是由 3 种类加载器相互配合执行的，在必要时，我们还可以自定义类加载器，来定制类的加载方式 +> 如果用户自定义了类加载器，则自定义类加载器都以应用类加载器作为父加载器。应用类加载器的父类加载器为扩展类加载器。这些类加载器是有层次关系的，启动加载器又叫根加载器，是扩展加载器的父加载器，但是直接从 ExClassLoader 里拿不到它的引用，同样会返回 null。 + ##### 为什么要自定义类加载器？ - 隔离加载类 @@ -254,7 +313,59 @@ public class ClassLoaderTest { 2. 在 JDK1.2 之前，在自定义类加载器时，总会去继承 ClassLoader 类并重写 loadClass() 方法，从而实现自定义的类加载类，但是 JDK1.2 之后已经不建议用户去覆盖 `loadClass()` 方式，而是建议把自定义的类加载逻辑写在 `findClass()` 方法中 3. 编写自定义类加载器时，如果没有太过于复杂的需求，可以直接继承 URLClassLoader 类，这样就可以避免自己去编写 findClass() 方法及其获取字节码流的方式，使自定义类加载器编写更加简洁 -### ClassLoader常用方法 +**eg**: + +> 比如我们试着实现一个可以用来处理简单加密的字节码的类加载器，用来保护我们的 class 字节码文件不被使用者直接拿来破解 +> +> ```java +> public class Hello { +> static { +> System.out.println("Hello Class Initialized!"); +> } +> } +> ``` +> +> 这个 Hello 类非常简单，就是在自己被初始化的时候，打印出来一句“Hello Class Initialized!”。假设这个类的内容非常重要，我们不想把编译到得到的 Hello.class 给别人，但是我们还是想别人可以调用或执行这个类，应该怎么办呢？一个简单的思路是，我们把这个类的 class 文件二进制作为字节流先加密一下，然后尝试通过自定义的类加载器来加载加密后的数据。为了演示简单，我们使用 jdk 自带的 Base64 算法，把字节码加密成一个文本。在下面这个例子里，我们实现一个 HelloClassLoader，它继承自 ClassLoader 类，但是我们希望它通过我们提供的一段 Base64 字符串，来还原出来，并执行我们的 Hello 类里的打印一串字符串的逻辑。 +> +> ```java +> public class HelloClassLoader extends ClassLoader { +> +> public static void main(String[] args) { +> try { +> new HelloClassLoader().findClass("jvm.Hello").newInstance(); // 加载并初始化Hello类 +> } catch (ClassNotFoundException e) { +> e.printStackTrace(); +> } catch (IllegalAccessException e) { +> e.printStackTrace(); +> } catch (InstantiationException e) { +> e.printStackTrace(); +> } +> } +> +> @Override +> protected Class findClass(String name) throws ClassNotFoundException { +> +> String helloBase64 = "yv66vgAAADQAHwoABgARCQASABMIABQKABUAFgcAFwcAGAEABjxpbml0PgEAAygpVgEABENvZGUBAA9MaW5lTnVtYmVyVGFibGUBABJMb2NhbFZhcmlhYmxlVGFibGUBAAR0aGlzAQALTGp2bS9IZWxsbzsBAAg8Y2xpbml0PgEAClNvdXJjZUZpbGUBAApIZWxsby5qYXZhDAAHAAgHABkMABoAGwEAGEhlbGxvIENsYXNzIEluaXRpYWxpemVkIQcAHAwAHQAeAQAJanZtL0hlbGxvAQAQamF2YS9sYW5nL09iamVjdAEAEGphdmEvbGFuZy9TeXN0ZW0BAANvdXQBABVMamF2YS9pby9QcmludFN0cmVhbTsBABNqYXZhL2lvL1ByaW50U3RyZWFtAQAHcHJpbnRsbgEAFShMamF2YS9sYW5nL1N0cmluZzspVgAhAAUABgAAAAAAAgABAAcACAABAAkAAAAvAAEAAQAAAAUqtwABsQAAAAIACgAAAAYAAQAAAAMACwAAAAwAAQAAAAUADAANAAAACAAOAAgAAQAJAAAAJQACAAAAAAAJsgACEgO2AASxAAAAAQAKAAAACgACAAAABgAIAAcAAQAPAAAAAgAQ"; +> +> byte[] bytes = decode(helloBase64); +> return defineClass(name,bytes,0,bytes.length); +> } +> +> public byte[] decode(String base64){ +> return Base64.getDecoder().decode(base64); +> } +> } +> ``` +> +> 直接执行这个类： +> +> ```shell +> $ java jvm.HelloClassLoader Hello Class Initialized! +> ``` + + + +### ClassLoader 常用方法 ClassLoader 类，是一个抽象类，其后所有的类加载器都继承自 ClassLoader（不包括启动类加载器） @@ -277,7 +388,7 @@ JVM 必须知道一个类型是由启动加载器加载的还是由用户类加 ## 双亲委派机制 -Java 虚拟机对 class 文件采用的是**按需加载**的方式，也就是说当需要使用该类的时候才会将它的 class 文件加载到内存生成 class 对象。而且加载某个类的 class 文件时，Java 虚拟机采用的是双亲委派模式，即把请求交给父类处理，它是一种任务委派模式。 +Java 虚拟机对 class 文件采用的是**按需加载**的方式，也就是说当需要使用该类的时候才会将它的 class 文件加载到内存生成 class 对象。而且加载某个类的 class 文件时，Java 虚拟机采用的是双亲委派模式，即把请求交给父类处理，它是一种任务委派模式。 ### 工作过程 @@ -289,25 +400,46 @@ Java 虚拟机对 class 文件采用的是**按需加载**的方式，也就是 ### 优势 -- 避免类的重复加载，JVM 中区分不同类，不仅仅是根据类名，相同的 class 文件被不同的 ClassLoader 加载就属于两个不同的类（比如，Java中的Object类，无论哪一个类加载器要加载这个类，最终都是委派给处于模型最顶端的启动类加载器进行加载，如果不采用双亲委派模型，由各个类加载器自己去加载的话，系统中会存在多种不同的 Object 类） -- 保护程序安全，防止核心 API 被随意篡改，避免用户自己编写的类动态替换 Java 的一些核心类，比如我们自定义类：`java.lang.String` +- 避免类的重复加载，JVM 中区分不同类，不仅仅是根据类名，相同的 class 文件被不同的 ClassLoader 加载就属于两个不同的类（比如，Java中的Object类，无论哪一个类加载器要加载这个类，最终都是委派给处于模型最顶端的启动类加载器进行加载，如果不采用双亲委派模型，由各个类加载器自己去加载的话，系统中会存在多种不同的 Object 类） +- 保护程序安全，防止核心 API 被随意篡改，避免用户自己编写的类动态替换 Java 的一些核心类，比如我们自定义类：`java.lang.String` 在 JVM 中表示两个 class 对象是否为同一个类存在两个必要条件： -- 类的完成类名必须一致，包括包名 +- 类的完整类名必须一致，包括包名 - 加载这个类的 ClassLoader（指ClassLoader实例对象）必须相同 ### 沙箱安全机制 -如果我们自定义 String 类，但是在加载自定义 String 类的时候会率先使用引导类加载器加载，而引导类加载器在加载的过程中会先加载 jdk 自带的文件（rt.jar包中 `java\lang\String.class`），报错信息说没有 main 方法就是因为加载的是`rt.jar`包中的String类。这样就可以保证对 java 核心源代码的保护，这就是简单的沙箱安全机制。 +如果我们自定义 String 类，但是在加载自定义 String 类的时候会率先使用引导类加载器加载，而引导类加载器在加载的过程中会先加载 jdk 自带的文件（rt.jar包中 `java\lang\String.class`），报错信息说没有 main 方法就是因为加载的是`rt.jar`包中的 String 类。这样就可以保证对 java 核心源代码的保护，这就是简单的沙箱安全机制。 ### 破坏双亲委派模型 -- 双亲委派模型并不是一个强制性的约束模型，而是 Java 设计者推荐给开发者的类加载器实现方式，可以“被破坏”，只要我们自定义类加载器，**重写loadClass()方法**，指定新的加载逻辑就破坏了，重写findClass()方法不会破坏双亲委派。 -- 双亲委派模型有一个问题：顶层 ClassLoader，无法加载底层 ClassLoader 的类。典型例子JNDI、JDBC，所以加入了线程上下文类加载器（Thread Context ClassLoader），可以通过`Thread.setContextClassLoaser()`设置该类加载器，然后顶层ClassLoader再使用`Thread.getContextClassLoader()`获得底层的ClassLoader进行加载。 -- Tomcat 中使用了自定 ClassLoader，并且也破坏了双亲委托机制。每个应用使用 WebAppClassloader 进行单独加载，他首先使用 WebAppClassloader 进行类加载，如果加载不了再委托父加载器去加载，这样可以保证每个应用中的类不冲突。每个tomcat中可以部署多个项目，每个项目中存在很多相同的class文件（很多相同的jar包），他们加载到 jvm 中可以做到互不干扰。 -- 利用破坏双亲委派来实现**代码热替换**（每次修改类文件，不需要重启服务）。因为一个Class只能被一个ClassLoader加载一次，否则会报`java.lang.LinkageError`。当我们想要实现代码热部署时，可以每次都new一个自定义的 ClassLoader 来加载新的 Class文件。JSP的实现动态修改就是使用此特性实现。 +- 双亲委派模型并不是一个强制性的约束模型，而是 Java 设计者推荐给开发者的类加载器实现方式，可以“被破坏”，只要我们自定义类加载器，**重写 `loadClass()` 方法**，指定新的加载逻辑就破坏了，重写 `findClass()` 方法不会破坏双亲委派。 + +- 双亲委派模型有一个问题：顶层 ClassLoader，无法加载底层 ClassLoader 的类。典型例子 JNDI、JDBC，所以加入了线程上下文类加载器（Thread Context ClassLoader），可以通过 `Thread.setContextClassLoaser()`设置该类加载器，然后顶层 ClassLoader 再使用 `Thread.getContextClassLoader()` 获得底层的 ClassLoader 进行加载。 + +- Tomcat 中使用了自定 ClassLoader，并且也破坏了双亲委托机制。每个应用使用 WebAppClassloader 进行单独加载，他首先使用 WebAppClassloader 进行类加载，如果加载不了再委托父加载器去加载，这样可以保证每个应用中的类不冲突。每个 tomcat 中可以部署多个项目，每个项目中存在很多相同的 class 文件（很多相同的jar包），他们加载到 jvm 中可以做到互不干扰。 + + ![](https://tva1.sinaimg.cn/large/e6c9d24ely1h35ghbt9uej20ib0g4abi.jpg) + +- 利用破坏双亲委派来实现**代码热替换**（每次修改类文件，不需要重启服务）。因为一个 Class 只能被一个 ClassLoader 加载一次，否则会报 `java.lang.LinkageError`。当我们想要实现代码热部署时，可以每次都 new 一个自定义的 ClassLoader 来加载新的 Class文件。JSP 的实现动态修改就是使用此特性实现。 + + + +## 如何替换 JDK 的类 + +如何替换 JDK 中的类？比如，我们现在就拿 HashMap为例。 + +当 Java 的原生 API 不能满足需求时，比如我们要修改 HashMap 类，就必须要使用到 Java 的 endorsed 技术。我们需要将自己的 HashMap 类，打包成一个 jar 包，然后放到 -Djava.endorsed.dirs 指定的目录中。注意类名和包名，应该和 JDK 自带的是一样的。但是，java.lang 包下面的类除外，因为这些都是特殊保护的。 + +因为我们上面提到的双亲委派机制，是无法直接在应用中替换 JDK 的原生类的。但是，有时候又不得不进行一下增强、替换，比如你想要调试一段代码，或者比 Java 团队早发现了一个 Bug。所以，Java 提供了 endorsed 技术，用于替换这些类。这个目录下的 jar 包，会比 rt.jar 中的文件，优先级更高，可以被最先加载到。 + + + +### References + +- http://blog.itpub.net/31561269/viewspace-2222522/ diff --git "a/docs/java/JVM/GC-\345\256\236\346\210\230.md" "b/docs/java/JVM/GC-\345\256\236\346\210\230.md" new file mode 100644 index 0000000000..f6f5162994 --- /dev/null +++ "b/docs/java/JVM/GC-\345\256\236\346\210\230.md" @@ -0,0 +1,435 @@ +# 垃圾回收-实战篇 + +上文（[看完这篇垃圾回收，和面试官扯皮没问题了](https://mp.csdn.net/console/editor/html/104602987)）GC 理论颇受大家好评，学习了之后，相信大家对 GC 的工作原理有了比较深刻的认识,这一篇我们继续趁热打铁，来学习下 GC 的实战内容，主要包括以下几点 + +- JVM 参数简介 +- 发生 OOM 的主要几种场景及相应解决方案 +- OOM 问题排查的一些常用工具 +- GC 日志格式怎么看 +- jstat 与可视化 APM 工具构建 +- 再谈 JVM 参数设置 + +## JVM 参数简介 + +在开始实践之前我们有必要先简单了解一下 JVM 参数配置，因为本文之后的实验中提到的 JVM 中的栈，堆大小，使用的垃圾收集器等都需要通过 JVM 参数来设置 + +先来看下如何运行一个 Java 程序 + +``` +public class Test { + public static void main(String[] args) { + System.out.println("test"); + } +} +``` + +1. 首先我们通过 **javac Test.java** 将其转成字节码 + +2. 其次我们往往会输入 **java Test** 这样的命令来启动 JVM 进程来执行此程序,其实我们在启动 JVM 进程的时候，可以指定相应的 JVM 的参数,如下蓝色部分 + + ![img](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9PeXdleXNDU2VMVklvWE5xaWN5V3hpYmViQXZUdUp4azQ0aWI0SndSanpCQWRpYUk3b1k0ZG1YZTFvTklRZlJsdVV5OXhQdmpYWDVaRjE1WE5aRkttRG54VkEvNjQw?x-oss-process=image/format,png) + +指定这些 JVM 参数我们就可以指定启动 JVM 进程以哪种模式（server 或 client），运行时分配的堆大小，栈大小，用什么垃圾收集器等等，JVM 参数主要分以下三类 + +1、 标准参数（-），所有的 JVM 实现都必须实现这些参数的功能，而且向后兼容；例如 **-verbose:gc**（输出每次GC的相关情况) + +2、 非标准参数（-X），默认 JVM 实现这些参数的功能，但是并不保证所有 JVM 实现都满足，且不保证向后兼容，栈，堆大小的设置都是通过这个参数来配置的，用得最多的如下 + +| 参数示例 | 表示意义 | +| :------- | :-------------------------------- | +| -Xms512m | JVM 启动时设置的初始堆大小为 512M | +| -Xmx512m | JVM 可分配的最大堆大小为 512M | +| -Xmn200m | 设置的年轻代大小为 200M | +| -Xss128k | 设置每个线程的栈大小为 128k | + +3、非Stable参数（-XX），此类参数各个 jvm 实现会有所不同，将来可能会随时取消，需要慎重使用， `-XX:-option` 代表关闭 option 参数，`-XX:+option` 代表要启用 option 参数，例如要启用串行 GC，对应的 JVM 参数即为 `-XX:+UseSerialGC`。非 Stable 参数主要有三大类 + +- 行为参数（Behavioral Options）：用于改变 JVM 的一些基础行为，如启用串行/并行 GC + +| 参数示例 | 表示意义 | +| :---------------------- | :-------------------------------------------------------- | +| -XX:-DisableExplicitGC | 禁止调用System.gc()；但jvm的gc仍然有效 | +| -XX:-UseConcMarkSweepGC | 对老生代采用并发标记交换算法进行GC | +| -XX:-UseParallelGC | 启用并行GC | +| -XX:-UseParallelOldGC | 对Full GC启用并行，当-XX:-UseParallelGC启用时该项自动启用 | +| -XX:-UseSerialGC | 启用串行GC | + +- 性能调优（Performance Tuning）：用于 jvm 的性能调优，如设置新老生代内存容量比例 + +| 参数示例 | 表示意义 | +| :---------------------------- | :------------------------------------ | +| -XX:MaxHeapFreeRatio=70 | GC后java堆中空闲量占的最大比例 | +| -XX:NewRatio=2 | 新生代内存容量与老生代内存容量的比例 | +| -XX:NewSize=2.125m | 新生代对象生成时占用内存的默认值 | +| -XX:ReservedCodeCacheSize=32m | 保留代码占用的内存容量 | +| -XX:ThreadStackSize=512 | 设置线程栈大小，若为0则使用系统默认值 | + +- 调试参数（Debugging Options）：一般用于打开跟踪、打印、输出等 JVM 参数，用于显示 JVM 更加详细的信息 + +| 参数示例 | 表示意义 | +| :-------------------------------- | :---------------------------------- | +| -XX:HeapDumpPath=./java_pid.hprof | 指定导出堆信息时的路径或文件名 | +| -XX:-HeapDumpOnOutOfMemoryError | 当首次遭遇OOM时导出此时堆中相关信息 | +| -XX:-PrintGC | 每次GC时打印相关信息 | +| -XX:-PrintGC Details | 每次GC时打印详细信息 | + +*画外音：以上只是列出了比较常用的 JVM 参数，更多的 JVM 参数介绍请查看文末的参考资料* + +明白了 JVM 参数是干啥用的，接下来我们进入实战演练，下文中所有程序运行时对应的 JVM 参数都以 VM Args 的形式写在开头的注释里，读者如果在执行程序时记得要把这些 JVM 参数给带上哦 + +## 发生 OOM 的主要几种场景及相应解决方案 + +有些人可能会觉得奇怪， GC 不是会自动帮我们清理垃圾以腾出使用空间吗，怎么还会发生 OOM, 我们先来看下有哪些场景会发生 OOM + +**1、Java 虚拟机规范中描述在栈上主要会发生以下两种异常** + +- StackOverflowError 异常 + + 这种情况主要是因为**单个线程**请求栈深度大于虚拟机所允许的最大深度（如常用的递归调用层级过深等），再比如单个线程定义了大量的本地变量，导致方法帧中本地变量表长度过大等也会导致 StackOverflowError 异常， 一句话：**在单线程下**，当栈桢太大或虚拟机容量太小导致内存无法分配时，都会发生 StackOverflowError 异常。 + +- 虚拟机在扩展栈时无法申请到足够的内存空间，会抛出 OOM 异常 + 在[刨根问底---一次 OOM 试验造成的电脑雪崩引发的思考](https://mp.weixin.qq.com/s?__biz=MzI5MTU1MzM3MQ==&mid=2247483922&idx=1&sn=8bafd48fb09e1badf8f513e5a4cd5916&scene=21#wechat_redirect) 一文中我们已经详细地剖析了此例子，再来看看 + +```java +/** + * VM Args:-Xss160k + */ + +public class Test { + private void dontStop() { + while(true) { + } + } + + public void stackLeakByThread() { + while (true) { + Thread thread = new Thread(new Runnable() { + @Override + public void run() { + dontStop(); + } + }); + thread.start(); + } + } + + public static void main(String[] args) { + Test oom = new Test(); + oom.stackLeakByThread(); + } +} +``` + +运行以上代码会抛出「**java.lang.OutOfMemoryError: unable to create new native thread**」的异常，原因不难理解，操作系统给每个进程分配的内存是有限制的，比如 32 位的 Windows 限制为 2G，虚拟机提供了参数来控制 Java 堆和方法的这两部内存的最大值，剩余的内存为 「2G - Xmx（最大堆容量）= 线程数 * 每个线程分配的虚拟机栈（-Xss)+本地方法栈 」（程序计数器消耗内存很少，可忽略）,每个线程都会被分配对应的虚拟机栈大小，所以总可创建的线程数肯定是固定的， 像以上代码这样不断地创建线程当然会造成最终无法分配，不过这也给我们提供了一个新思路，如果是因为建立过多的线程导致的内存溢出，而我们又想多创建线程，可以通过减少最大堆（-Xms）和减少虚拟机栈大小（-Xss）来实现。 + +**2、堆溢出 （java.lang.OutOfMemoryError:Java heap space**） + +主要原因有两点 + +- 1.大对象的分配，最有可能的是大数组分配 + +示例如下： + +```java +/** +* VM Args:-Xmx12m + */ + +class OOM { + static final int SIZE=2*1024*1024; + public static void main(String[] a) { + int[] i = new int[SIZE]; + } +} +``` + +我们指定了堆大小为 12M,执行 「java -Xmx12m OOM」命令就发生了 OOM 异常，如果指定 13M 则以上程序就能正常执行，所以对于由于大对象分配导致的堆溢出这种 OOM，我们一般采用增大堆内存的方式来解决 + +*画外音：**有人可能会说分配的数组大小不是只有 2 \* *1024 ** 1024 \* 4（一个 int 元素占 4 个字节）= 8M, 怎么分配 12 M 还不够，因为 默认新老代1:2即4,8M都装不下数组，JVM 进程除了分配数组大小，还有指向类(数组中元素对应的类)信息的指针、锁信息等，实际需要的堆空间是可能超过 12M 的， 12M 也只是尝试出来的值，不同的机器可能不一样* + +- 2.内存泄漏 + 我们知道在 Java 中，开发者创建和销毁对象是不需要自己开辟空间的，JVM 会自动帮我们完成，在应用程序整个生命周期中，JVM 会定时检查哪些对象可用，哪些不再使用，如果对象不再使用的话理论上这块内存会被回收再利用（即GC），如果无法回收就会发生内存泄漏 + +```java +/** +* VM Args:-Xmx4m + */ + +public class KeylessEntry { + + static class Key { + Integer id; + Key(Integer id) { + this.id = id; + } + + @Override + + public int hashCode() { + return id.hashCode(); + } + } + + public static void main(String[] args) { + Map m = new HashMap(); + while(true) { + for(int i=0;i<10000;i++) { + if(!m.containsKey(new Key(i))) { + m.put(new Key(i), "Number:" + i); + } + } + } + } +} +``` + +执行以上代码就会发生内存泄漏，第一次循环，map 里存有 10000 个 key value,但之后的每次循环都会**新增** 10000 个元素，因为 Key 这个 class 漏写了 equals 方法，导致对于每一个新创建的 new Key(i) 对象，即使 i 相同也会被认定为属于两个不同的对象，这样 **m.containsKey(new Key(i))** 结果均为 false,结果就是 HashMap 中的元素将一直增加，解决方式也很简单，为 Key 添加 equals 方法即可，如下 + +```java +@Override +public boolean equals(Object o) { + boolean response = false; + if (o instanceof Key) { + response = (((Key)o).id).equals(this.id); + } + return response; + +} +``` + +对于这种内存泄漏导致的 OOM， 单纯地增大堆大小是无法解决根本问题的，只不过是延缓了 OOM 的发生，最根本的解决方式还是要通过 **heap dump analyzer** 等方式来找出内存泄漏的代码来修复解决，后文会给出一个例子来分析 + +**3、java.lang.OutOfMemoryError:GC overhead limit exceeded** + +Sun 官方对此的定义：超过98%的时间用来做 GC 并且回收了不到 2% 的堆内存时会抛出此异常 + +![img](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9PeXdleXNDU2VMVklvWE5xaWN5V3hpYmViQXZUdUp4azQ0MWs5UG1VYUcyZkwyY0kxVVFZaWNMVkg2SjZpYjJpY0VDdmM0bFM3ZGw3MEZpYTNTZWlhU3FBVlUxUFEvNjQw?x-oss-process=image/format,png) + +导致的后果就是由于经过几个 GC 后只回收了不到 2% 的内存，堆很快又会被填满，然后又频繁发生 GC，导致 CPU 负载很快就达到 100%，另外我们知道 GC 会引起 「Stop The World 」的问题，阻塞工作线程，所以会导致严重的性能问题，产生这种 OOM 的原因与「**java.lang.OutOfMemoryError:Java heap space**」类似，主要是由于分配大内存数组或内存泄漏导致的， 解决方案如下： + +- 检查项目中是否有大量的死循环或有使用大内存的代码，优化代码。 +- dump 内存（后文会讲述如何 dump 出内存），检查是否存在内存泄露，如果没有，可考虑通过 -Xmx 参数设置加大内存。 + +**4、java.lang.OutOfMemoryError:Permgen space** + +在 Java 8 以前有永久代（其实是用永久代实现了方法区的功能）的概念，存放了被虚拟机加载的类，常量，静态变量，JIT 编译后的代码等信息，所以如果错误地频繁地使用 String.intern() 方法或运行期间生成了大量的代理类都有可能导致永久代溢出，解决方案如下 + +- 是否永久代设置的过小，如果可以，适应调大一点 +- 检查代码是否有大量的反射操作 +- dump 之后通过 mat 检查是否存在大量由于反射生成的代码类 + +**5、java.lang.OutOfMemoryError:Requested array size exceeds VM limit** + +该错误由 JVM 中的 native code 抛出。JVM 在为数组分配内存之前，会执行基于所在平台的检查：分配的数据结构是否在此平台中是可寻址的，平台一般允许分配的数据大小在 1 到 21 亿之间，如果超过了这个数就会抛出这种异常 + +![img](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9PeXdleXNDU2VMVklvWE5xaWN5V3hpYmViQXZUdUp4azQ0dDR6TjVaWHVoMUNrUDhqa3hwWk5aOHNqQktGVGxJejlBQTVQdUF4MVZiUnZpYTNXUGljcXNPaWNBLzY0MA?x-oss-process=image/format,png) + +碰到这种异常一般我们只要检查代码中是否有创建超大数组的地方即可。 + +**6、java.lang.OutOfMemoryError: Out of swap space** + +Java 应用启动的时候分被分配一定的内存空间(通过 -Xmx 及其他参数来指定)， 如果 JVM 要求的总内存空间大小大于可用的本机内存，则操作系统会将内存中的部分数据交换到硬盘上 + +![img](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9PeXdleXNDU2VMVklvWE5xaWN5V3hpYmViQXZUdUp4azQ0ek5vQjdpYklIZ0p0N09LMlNpY0VmaWFTdFYzcEJwYTRKemljN2hzbjk4dXlMMVp5endGbHJTdEFuQS82NDA?x-oss-process=image/format,png) + + +如果此时 swap 分区大小不足或者其他进程耗尽了本机的内存，则会发生 OOM, 可以通过增大 swap 空间大小来解决,但如果在交换空间进行 GC 造成的 「Stop The World」增加大个数量级，所以增大 swap 空间一定要慎重，所以一般是通过增大本机内存或优化程序减少内存占用来解决。 + +**7、Out of memory:Kill process or sacrifice child** + +为了理解这个异常，我们需要知识一些操作系统的知识，我们知道，在操作系统中执行的程序，都是以进程的方式运行的，而进程是由内核调度的，在内核的调度任务中，有一个「Out of memory killer」的调度器，它会在系统可用内存不足时被激活，然后选择一个进程把它干掉，哪个进程会被干掉呢，简单地说会优先干掉占用内存大的应用型进程 + +![img](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9PeXdleXNDU2VMVklvWE5xaWN5V3hpYmViQXZUdUp4azQ0aFpQNDR5VHl0MXhXcXVBS2ZqSGljWUdwQ3Q3SEFpYTI4RmxlbkQweTlPQWlhdnUybW55OGNEMVJBLzY0MA?x-oss-process=image/format,png) + + +*如图示，进程 4 占用内存最大，最有可能被干掉* + +解决这种 OOM 最直接简单的方法就是升级内存，或者调整 OOM Killer 的优先级，减少应用的不必要的内存使用等等 + +看了以上的各种 OOM 产生的情况，可以看出：**GC 和是否发生 OOM 没有必然的因果关系!** GC 主要发生在堆上，而 从以上列出的几种发生 OOM 的场景可以看出，空间不足无法再创建线程，或者存在死循环一直在分配对象导致 GC 无法回收对象或者一次分配大内存数组（超过堆的大小）等都可能导致 OOM, 所以 **OOM 与 GC 并没有必然的因果关系** + +## OOM 问题排查的一些常用工具 + +接下来我们来看下如何排查造成 OOM 的原因，内存泄漏是最常见的造成 OOM 的一种原因，所以接下来我们以来看看怎么使用工具来排查这种问题，使用到的工具主要有两大类 + +**1、使用 mat（Eclipse Memory Analyzer） 来分析 dump（堆转储快照） 文件** + +主要步骤如下 + +- 运行 Java 时添加 「-XX:+HeapDumpOnOutOfMemoryError」 参数来导出内存溢出时的堆信息，生成 hrof 文件, 添加 「-XX:HeapDumpPath」可以指定 hrof 文件的生成路径,如果不指定则 hrof 文件生成在与字节码文件相同的目录下 +- 使用 MAT（Eclipse Memory Analyzer）来分析 hrof 文件，查出内存泄漏的原因 + +接下来我们就来看看如何用以上的工具查看如下内存泄漏案例 + +```java +/** +* VM Args:-Xmx10m + */ + +import java.util.ArrayList; +import java.util.List; + +public class Main { + + public static void main(String[] args) { + List list = new ArrayList(); + while (true) { + list.add("OutOfMemoryError soon"); + } + } +} +``` + +为了让以上程序快速产生 OOM， 我把堆大小设置成了 10M，这样执行 「java -Xmx10m -XX:+HeapDumpOnOutOfMemoryError Main」后很快就发生了 OOM，此时我们就拿到了 hrof 文件，下载 MAT 工具，打开 hrof，进行分析，打开之后选择 「Leak Suspects Report」进行分析，可以看到发生 OOM 的线程的堆栈信息，明确定位到是哪一行造成的 + +![img](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9PeXdleXNDU2VMVklvWE5xaWN5V3hpYmViQXZUdUp4azQ0UVdlcjhUTEFobnkxdE9aY1MwZ1lWSThDRWxua3dvSHhReHF4ZFZNaWM0aFFCWW9tcFRLTzZhZy82NDA?x-oss-process=image/format,png) + +*如图示，可以看到 Main.java 文件的第 12 行导致了这次的 OOM* + +**2、使用 jvisualvm 来分析** + +用第一种方式必须等 OOM 后才能 dump 出 hprof 文件，但如果我们想在运行中观察堆的使用情况以便查出可能的内存泄漏代码就无能为力了，这时我们可以借助 **jvisualvm** 这款工具，jvisualvm 的功能强大，除了可以实时监控堆内存的使用情况，还可以跟踪垃圾回收，运行中 dump 中堆内存使用情况、cpu分析，线程分析等，是查找分析问题的利器，更骚的是它不光能分析本地的 Java 程序，还可以分析线上的 Java 程序运行情况，本身这款工具也是随 JDK 发布的，是官方力推的一款运行监视，故障处理的神器。我们来看看如何用 jvisualvm 来分析上文所述的存在内存泄漏的如下代码 + +```java +import java.util.Map; +import java.util.HashMap; + +public class KeylessEntry { + static class Key { + Integer id; + Key(Integer id) { + this.id = id; + } + + @Override + public int hashCode() { + return id.hashCode(); + } + } + + public static void main(String[] args) { + Map m = new HashMap(); + while(true) { + for(int i=0;i<10000;i++) { + if(!m.containsKey(new Key(i))) { + m.put(new Key(i), "Number:" + i); + } + } + } + } +} +``` + +打开 jvisualvm （终端输入 jvisualvm 执行即可），打开后，将堆大小设置为 500M，执行命令 **java Xms500m -Xmx500m KeylessEntry**，此时可以观察到左边出现了对应的应用 KeylessEntry,双击点击 open + +![img](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9PeXdleXNDU2VMVklvWE5xaWN5V3hpYmViQXZUdUp4azQ0NUZPbDdlM1NGRk1sbUJpYkt2bHVHck1sempJaWNJb0x0ZlgycDVpYXFQWXBCYkJqWU9MekJBbVp3LzY0MA?x-oss-process=image/format,png) + +打开之后可以看到展示了 CPU，堆内存使用，加载类及线程的情况 + +![img](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9PeXdleXNDU2VMVklvWE5xaWN5V3hpYmViQXZUdUp4azQ0ODlyYXRvYnp0V3hzSUlqeUpiV0ZvV0lnOGljNlpVRVJpYkM4MXBjMzV3dXFXNTVrZ2FyM3VVZncvNjQw?x-oss-process=image/format,png) + +注意看堆（Heap）的使用情况，一直在上涨 + +![img](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9PeXdleXNDU2VMVklvWE5xaWN5V3hpYmViQXZUdUp4azQ0aElVOGwxWDcwaHFYYmhmSFVQTktGMkVSc2pTd1Q1NGljaWExdlNJQXFBV0JpYklRRzNkQ0JYOVdnLzY0MA?x-oss-process=image/format,png) + +此时我们再点击 「Heap Dump」 + +![img](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9PeXdleXNDU2VMVklvWE5xaWN5V3hpYmViQXZUdUp4azQ0QlZ4VndvTjk5akhoaEs5N0FBUW5pY1h0N3RsT0VSV0g3VjZSSnFDRGpXWlhKMlRsZmNrbEgyQS82NDA?x-oss-process=image/format,png) + +过一会儿即可看到内存中对象的使用情况 + +![img](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9PeXdleXNDU2VMVklvWE5xaWN5V3hpYmViQXZUdUp4azQ0OHozejZidTc2TmtyRGVoUjNQeEtkMnRQTlJDZmhaMk9iTUZSWDdqREphN0F5bEdjMFoyTFdnLzY0MA?x-oss-process=image/format,png) + +可以看到相关的 TreeNode 有 291w 个，远超正常情况下的 10000 个！说明 HashMap 一直在增长，自此我们可以定位出问题代码所在！ + +**3、使用 jps + jmap 来获取 dump 文件** + +jps 可以列出正在运行的虚拟机进程，并显示执行虚拟机主类及这些进程的本地虚拟机唯一 ID，如图示 + +![img](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9PeXdleXNDU2VMVklvWE5xaWN5V3hpYmViQXZUdUp4azQ0OWljN1pCQ1dXY2E4OG56UXJrdldUZlV3N2t1aERYdEtMam5sbFNEWEhyVEZQZ0FGVUVLUU11Zy82NDA?x-oss-process=image/format,png) + +拿到进程的 pid 后，我们就可以用 jmap 来 dump 出堆转储文件了，执行命令如下 + +``` +jmap -dump:format=b,file=heapdump.phrof pid +``` + +拿到 dump 文件后我们就可以用 MAT 工具来分析了。 +但这个命令在生产上一定要慎用！因为JVM 会将整个 heap 的信息 dump 写入到一个文件，heap 比较大的话会导致这个过程比较耗时，并且执行过程中为了保证 dump 的信息是可靠的，会暂停应用！ + +## GC 日志格式怎么看 + +接下来我们看看 GC 日志怎么看，日志可以有效地帮助我们定位问题，所以搞清楚 GC 日志的格式非常重要，来看下如下例子 + +```java +/** + * VM Args:-verbose:gc -Xms20M -Xmx20M -Xmn10M -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseSerialGC -XX:SurvivorRatio=8 + */ +public class TestGC { + private static final int _1MB = 1024 * 1024; + + public static void main(String[] args) { + byte[] allocation1, allocation2, allocation3, allocation4; + allocation1 = new byte[2 * _1MB]; + allocation2 = new byte[2 * _1MB]; + allocation3 = new byte[2 * _1MB]; + allocation4 = new byte[4 * _1MB]; // 这里会出现一次 Minor GC + + } +} +``` + +执行以上代码，会输出如下 GC 日志信息 + +10.080: 2[GC 3(Allocation Failure) 0.080: 4[DefNew: 56815K->280K(9216K),6 0.0043690 secs] 76815K->6424K(19456K), 80.0044111 secs]9 [Times: user=0.00 sys=0.01, real=0.01 secs] + +以上是发生 Minor GC 的 GC 是日志，如果发生 Full GC 呢，格式如下 + +10.088: 2[Full GC 3(Allocation Failure) 0.088: 4[Tenured: 50K->210K(10240K), 60.0009420 secs] 74603K->210K(19456K), [Metaspace: 2630K->2630K(1056768K)], 80.0009700 secs]9[Times: user=0.01 sys=0.00, real=0.02 secs] + +两者格式其实差不多，一起来看看，主要以本例触发的 Minor GC 来讲解, 以上日志中标的每一个数字与以下序号一一对应 + +1. 开头的 0.080，0.088 代表了 GC 发生的时间，这个数字的含义是从 Java 虚拟机启动以来经过的秒数 +2. **[GC** 或者 **[Full GC** 说明了这次垃圾收集的停顿类型，注意不是用来区分新生代 GC 还是老年化 GC 的，如果有 **Full**，说明这次 GC 是发生了 **Stop The World** 的，如果是调用 System.gc() 所触发的收集，这里会显示 **[Full GC（System）** +3. 之后的 **Allocation Failure** 代表了触发 GC 的原因，在这个程序中我们设置了新生代的大小为 10M（-Xmn10M），Eden:S0:S1 = 8：1：1（-XX:SurvivorRatio=8），也就是说 Eden 区占了 8M, 当分配 allocation4 时，由于将要分配的总大小为 10M,超过了 Eden 区，所以此时会发生 GC +4. 接下来的 **[DefNew**，**[Tenured**，**[Metaspace** 表示 GC 发生的区域，这里显示的区域名与使用的 GC 收集器是密切相关的，在此例中由于新生代我们使用了 Serial 收集器，此收集器新生代名为「Default New Generation」，所以显示的是 **[DefNew**，如果是 ParNew 收集器，新生代名称就会变为 **[ParNew**`，意为 「Parallel New Generation」,如果采用 「Parallel Scavenge」收集器，则配套的新生代名称为「PSYoungGen」,老年代与新生代一样，名称也是由收集器决定的 +5. 再往后 **6815K->280K(9216K)** 表示 「GC 前该内存区域已使用容量 -> GC 后该内存区域已使用容量（该内存区域总容量）」 +6. 0.0043690 secs 表示该块内存区域 GC 所占用的时间，单位是秒 +7. 6815K->6424K(19456K) 表示「GC 前 Java 堆已使用容量 -> GC 后 Java 堆已使用容量（java 堆总容量）」。 +8. **0.0044111 secs** 表示整个 GC 执行时间，注意和 6 中 **0.0043690 secs** 的区别，后者专指**相关区域**所花的 GC 时间，而前者指的 GC 的整体堆内存变化所花时间（新生代与老生代的的内存整理），所以前者是肯定大于后者的！ +9. 最后一个 [Times: user=0.01 sys=0.00, real=0.02 secs] 这里的 user, sys 和 real 与Linux 的 time 命令所输出的时间一致，分别代表用户态消耗的 CPU 时间，内核态消耗的 CPU 时间，和操作从开始到结束所经过的墙钟时间，墙钟时间包括各种非运算的等待耗时，例如等待磁盘 I/O，等待线程阻塞，而 CPU 时间不包括这些耗时，但当系统有多 CPU 或者多核的话，多线程操作会叠加这些 CPU 时间，所以 user 或 sys 时间是可能超过 real 时间的。 + +知道了 GC 日志怎么看，我们就可以根据 GC 日志有效定位问题了，如我们发现 Full GC 发生时间过长，则结合我们上文应用中打印的 OOM 日志可能可以快速定位到问题 + +## jstat 与可视化 APM 工具构建 + +jstat 是用于监视虚拟机各种运行状态信息的命令行工具，可以显示本地或者远程虚拟机进程中的类加载，内存，垃圾收集，JIT 编译等运行数据，jstat 支持定时查询相应的指标，如下 + +``` +jstat -gc 2764 250 22 +``` + +定时针对 2764 进程输出堆的垃圾收集情况的统计，可以显示 gc 的信息，查看gc的次数及时间,利用这些指标，把它们可视化，对分析问题会有很大的帮助,如图示，下图就是我司根据 jstat 做的一部分 gc 的可视化报表，能快速定位发生问题的问题点，如果大家需要做 APM 可视化工具，建议配合使用 jstat 来完成。 + +![img](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9PeXdleXNDU2VMVklvWE5xaWN5V3hpYmViQXZUdUp4azQ0TzRsdGpTR2ZpYkVzYUd2NE9TRE52OXNnY3FpY1N3SnlkU0hJSEtWZXR5Wms4Sm9QUnpmb085OGcvNjQw?x-oss-process=image/format,png) + +## 再谈 JVM 参数设置 + +经过前面对 JVM 参数的介绍及相关例子的实验，相信大家对 JVM 的参数有了比较深刻的理解，接下来我们再谈谈如何设置 JVM 参数 + +1、首先 Oracle 官方推荐堆的初始化大小与堆可设置的最大值一般是相等的，即 Xms = Xmx，因为起始堆内存太小（Xms），会导致启动初期频繁 GC，起始堆内存较大（Xmx）有助于减少 GC 次数 + +2、调试的时候设置一些打印参数，如-XX:+PrintClassHistogram -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -Xloggc:log/gc.log，这样可以从gc.log里看出一些端倪出来 + +3、系统停顿时间过长可能是 GC 的问题也可能是程序的问题，多用 jmap 和 jstack 查看，或者killall -3 Java，然后查看 Java 控制台日志，能看出很多问题 + +4、 采用并发回收时，年轻代小一点，年老代要大，因为年老大用的是并发回收，即使时间长点也不会影响其他程序继续运行，网站不会停顿 + +5、仔细了解自己的应用，如果用了缓存，那么年老代应该大一些，缓存的 HashMap 不应该无限制长，建议采用 LRU 算法的 Map 做缓存，LRUMap 的最大长度也要根据实际情况设定 + +要设置好各种 JVM 参数，还可以对 server 进行压测， 预估自己的业务量，设定好一些 JVM 参数进行压测看下这些设置好的 JVM 参数是否能满足要求 + +## 总结 + +本文通过详细介绍了 JVM 参数及 GC 日志， OOM 发生的原因及相应的调试工具，相信读者应该掌握了基本的 MAT，jvisualvm 这些工具排查问题的技巧，不过这些工具的介绍本文只是提到了一些皮毛，大家可以在再深入了解相应工具的一些进阶技能，这能对自己排查问题等大有裨益！文中的例子大家可以去试验一下，修改一下参数看下会发生哪些神奇的现象，亲自动手做一遍能对排查问题的思路更加清晰哦 + diff --git a/docs/java/JVM/GC.md b/docs/java/JVM/GC.md index 397c1c9c2a..d011b6ad4f 100644 --- a/docs/java/JVM/GC.md +++ b/docs/java/JVM/GC.md @@ -1,4 +1,8 @@ -## 前言 +![article-image](https://cdn.packtpub.com/article-hub/articles/23c5e054335d39909b5020b1f241ff4d.jpg) + +# 垃圾回收机制 + +## 一、前言 Java 与 C++ 之间有一堵由内存动态分配和垃圾收集技术所围成的“高墙”，墙外面的人想进去，墙里面的人却想出来。 @@ -8,33 +12,17 @@ Java 相比 C/C++ 最显著的特点便是引入了自动垃圾回收 ，它解 -## JVM 内存区域 - -要搞懂垃圾回收的机制，我们首先要知道垃圾回收主要回收的是哪些数据，这些数据主要在哪一块区域，所以我们一起来看下 JVM 的内存区域 - -![img](https://tva1.sinaimg.cn/large/007S8ZIlly1gfwrx0a1jqj30ol0ck754.jpg) - -- 虚拟机栈：描述的是方法执行时的内存模型,是线程私有的，生命周期与线程相同,每个方法被执行的同时会创建**栈桢**，主要保存执行方法时的局部变量表、操作数栈、动态连接和方法返回地址等信息,方法执行时入栈，方法执行完出栈，出栈就相当于清空了数据，入栈出栈的时机很明确，所以这块区域**不需要进行 GC**。 - -- 本地方法栈：与虚拟机栈功能非常类似，主要区别在于虚拟机栈为虚拟机执行 Java 方法时服务，而本地方法栈为虚拟机执行本地方法时服务的。这块区域也**不需要进行 GC** +### 回顾下 JVM 内存区域 -- 程序计数器：线程独有的， 可以把它看作是当前线程执行的字节码的行号指示器，比如如下字节码内容，在每个字节码`前面都有一个数字（行号），我们可以认为它就是程序计数器存储的内容![img](https://tva1.sinaimg.cn/large/007S8ZIlly1gfwrx4g1s8j30iz07rgll.jpg) +![](https://dl-harmonyos.51cto.com/images/202212/d37207e632cd193510c4713b1db551924c2d36.png) - 记录这些数字（指令地址）有啥用呢，我们知道 Java 虚拟机的多线程是通过线程轮流切换并分配处理器的时间来完成的，在任何一个时刻，一个处理器只会执行一个线程，如果这个线程被分配的时间片执行完了（线程被挂起），处理器会切换到另外一个线程执行，当下次轮到执行被挂起的线程（唤醒线程）时，怎么知道上次执行到哪了呢，通过记录在程序计数器中的行号指示器即可知道，所以程序计数器的主要作用是记录线程运行时的状态，方便线程被唤醒时能从上一次被挂起时的状态继续执行，需要注意的是，程序计数器是**唯一一个**在 Java 虚拟机规范中没有规定任何 OOM 情况的区域，所以这块区域也**不需要进行 GC** -- 本地内存：线程共享区域，Java 8 中，本地内存，也是我们通常说的**堆外内存**，包含元空间和直接内存,注意到上图中 Java 8 和 Java 8 之前的 JVM 内存区域的区别了吗，在 Java 8 之前有个**永久代**的概念，实际上指的是 HotSpot 虚拟机上的永久代，它用永久代实现了 JVM 规范定义的方法区功能，主要存储类的信息，常量，静态变量，即时编译器编译后代码等，这部分由于是在堆中实现的，受 GC 的管理，不过由于永久代有` -XX:MaxPermSize` 的上限，所以如果动态生成类（将类信息放入永久代）或大量地执行 **String.intern** （将字段串放入永久代中的常量区），很容易造成 OOM，有人说可以把永久代设置得足够大，但很难确定一个合适的大小，受类数量，常量数量的多少影响很大。所以在 Java 8 中就把方法区的实现移到了本地内存中的元空间中，这样方法区就不受 JVM 的控制了,也就不会进行 GC，也因此提升了性能（发生 GC 会发生 Stop The Word,造成性能受到一定影响，后文会提到），也就不存在由于永久代限制大小而导致的 OOM 异常了（假设总内存1G，JVM 被分配内存 100M， 理论上元空间可以分配 2G-100M = 1.9G，空间大小足够），也方便在元空间中统一管理。综上所述，在 Java 8 以后这一区域也**不需要进行 GC** -​ *画外音：* *思考一个问题，堆外内存不受 GC控制，无法通过 GC 释放内存，那该以什么样的形式释放呢，总不能只创建不释放吧，这样的话内存可能很快就满了，这里不做详细阐述，请看文末的参考文章* +## 二、如何识别垃圾 -- 堆：前面几块数据区域都不进行 GC，那只剩下堆了，是的，这里是 GC 发生的区域！对象实例和数组都是在堆上分配的，GC 也主要对这两类数据进行回收，这块也是我们之后重点需要分析的区域 +我们都知道 GC 主要发生在堆，那么 GC 该怎么判断堆中的对象实例或数据是不是垃圾呢，或者说判断某些数据是否是垃圾的方法有哪些。 - - -## 如何识别垃圾 - -上一节我们详细讲述了 JVM 的内存区域，知道了 GC 主要发生在堆，那么 GC 该怎么判断堆中的对象实例或数据是不是垃圾呢，或者说判断某些数据是否是垃圾的方法有哪些。 - -### 引用计数法 +### 2.1 引用计数法 最容易想到的一种方式是引用计数法，啥叫引用计数法，简单地说，就是对象被引用一次，在它的对象头上加一次引用次数，如果没有被引用（引用次数为 0），则此对象可回收 @@ -44,13 +32,13 @@ String ref = new String("Java"); 以上代码 ref1 引用了右侧定义的对象，所以引用次数是 1 -![img](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9PeXdleXNDU2VMVXJZcVBpY2pWd2p1TUNoUHJQaWNOSGRYamljaWM0aERkdkRmT3lUUWFobHQzQkt2VXdJU3p3T3NHcUZqZmFXdXZWajk5Yk9rUzdBb1JaNGcvNjQw?x-oss-process=image/format,png) +![](https://dl-harmonyos.51cto.com/images/202212/36620eb50bc2337a78a525cc3e139c029255f0.png) 如果在上述代码后面添加一个 ref = null，则由于对象没被引用，引用次数置为 0，由于不被任何变量引用，此时即被回收，动图如下 -![img](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X2dpZi9PeXdleXNDU2VMVXJZcVBpY2pWd2p1TUNoUHJQaWNOSGRYWXpkcGwzNmZtRUh3RDFYZ0Z2ZTM5WXlqSnNXMmliU09NZDZPTU81SExJbENIYVlpYnZFUFNpYmliQS82NDA?x-oss-process=image/format,png) +![](https://dl-harmonyos.51cto.com/images/202212/18eb87f042f7160d0d39383a1e82528e775cb1.gif) -看起来用引用计数确实没啥问题了，不过它无法解决一个主要的问题：循环引用！啥叫循环引用 +看起来用引用计数确实没啥问题了，不过它无法解决一个主要的问题：**循环引用**！啥叫循环引用 ```java public class TestRC { @@ -77,26 +65,40 @@ public class TestRC { 按步骤一步步画图 -![img](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9PeXdleXNDU2VMVXJZcVBpY2pWd2p1TUNoUHJQaWNOSGRYaWJkYjN3RERoQXJuRmliaFdoV2NIa2NNUjVRcndyRTJwTm9nSjZySFdpY1l6N2JqRTJOaWJXMW9jZy82NDA?x-oss-process=image/format,png) +![](https://dl-harmonyos.51cto.com/images/202212/6610b0168edd882e5ed921e4eef6daf9c5d378.png) 到了第三步，虽然 a，b 都被置为 null 了，但是由于之前它们指向的对象互相指向了对方（引用计数都为 1），所以无法回收，也正是由于无法解决循环引用的问题，所以主流的 Java 虚拟机都没有选用引用计数法来管理内存。 -### 可达性算法 -现代虚拟机基本都是采用这种算法来判断对象是否存活，可达性算法的原理是以一系列叫做 **GC Root** 的对象为起点出发，引出它们指向的下一个节点，再以下个节点为起点，引出此节点指向的下一个结点。。。（这样通过 GC Root 串成的一条线就叫引用链），直到所有的结点都遍历完毕,如果相关对象不在任意一个以 **GC Root** 为起点的引用链中，则这些对象会被判断为「垃圾」,会被 GC 回收。 -![img](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9PeXdleXNDU2VMVXJZcVBpY2pWd2p1TUNoUHJQaWNOSGRYdlpMbXg5SGRlTmliUlFqbkZad1VCRlhUUTZtSnk3OWFqeDRCZHFGTWljSWlhVzRCTlFhR1RaV2liZy82NDA?x-oss-process=image/format,png) +#### 优点和缺点 + +引用计数法可以在对象不活跃时（引用计数为0）立刻回收其内存。因此可以保证堆上时时刻刻都没有垃圾对象的存在（先不考虑循环引用导致无法回收的情况）。 -如图示，如果用可达性算法即可解决上述循环引用的问题，因为从**GC Root** 出发没有到达 a,b,所以 a，b 可回收 +引用计数法的最大暂停时间短。由于没有了独立的GC过程，而且不需要遍历整个堆来标记和清除对象，取而代之的是在对象引用计数为0时立即回收对象，这相当于将GC过程“分摊”到了每个对象上，不会有最大暂停时间特别长的情况发生。 + +引用计数法也有一些问题，引用计数的增减开销在一些情况下会比较大，比如一些根引用的指针更新非常频繁，此时这种开销是不能忽视的。另外对象引用计数器本身是需要空间的，而计数器要占用多少位也是一个问题，理论上系统内存可寻址的范围越大，对象计数器占用的空间就要越大，这样在一些小对象上就会出现计数器空间比对象本身的域还要大的情况，内存空间利用率就会降低。还有一个问题是循环引用的问题，假设两个对象A和B，A引用B，B也引用A，除此之外它们都没有其他引用关系了，这个时候A和B就形成了循环引用，变成一个“孤岛”，且它们的引用计数都是1，按照引用计数法的要求，它们将无法被回收，造成内存泄漏。 + +> https://nullcc.github.io/2017/11/11/%E5%9E%83%E5%9C%BE%E5%9B%9E%E6%94%B6%28GC%29%E7%AE%97%E6%B3%95%E4%BB%8B%E7%BB%8D%282%29%E2%80%94%E2%80%94GC%E5%BC%95%E7%94%A8%E8%AE%A1%E6%95%B0%E7%AE%97%E6%B3%95/ + + + +### 2.2 可达性算法 + +现代虚拟机基本都是采用这种算法来判断对象是否存活，可达性算法的原理是以一系列叫做 **GC Root** 的对象为起点出发，引出它们指向的下一个节点，再以下个节点为起点，引出此节点指向的下一个结点。。。（这样通过 GC Root 串成的一条线就叫引用链），直到所有的结点都遍历完毕，如果相关对象不在任意一个以 **GC Root** 为起点的引用链中，则这些对象会被判断为「垃圾」,会被 GC 回收。 + +![](https://dl-harmonyos.51cto.com/images/202212/c89f360980bd1793f756862748b88614ca44a7.png) + +如图示，如果用可达性算法即可解决上述循环引用的问题，因为从**GC Root** 出发没有到达 a,b，所以 a，b 可回收 > a, b 对象可回收，就一定会被回收吗? 即使在可达性分析算法中不可达的对象，也并非是“非死不可”的，这时候他们暂时处于“缓刑”阶段，要真正宣告一个对象死亡，至少要两次标记过程： -1. 对象不可达（可回收），会进行第一次标记并进行一次筛选，筛选条件是此对象是否有必要执行 finalize() 方法 -2. 如果有必要执行finaize() 方法，这个对象会被放在一个叫做 F-Queue 的队列中，稍后会由JVM 自动建立的、低优先级的 Finalizer 线程去执行（触发，并不会等其运行结束），这时进行第二次标记，仍然不可达，则会被真的回收。 +1. 对象不可达（可回收），会进行第一次标记并进行一次筛选，筛选条件是此对象是否有必要执行 `finalize()` 方法 +2. 如果有必要执行 `finaize()` 方法，这个对象会被放在一个叫做 F-Queue 的队列中，稍后会由 JVM 自动建立的、低优先级的 Finalizer 线程去执行（触发，并不会等其运行结束），这时进行第二次标记，仍然不可达，则会被真的回收。 -**注意：**任何一个对象的 finalize() 方法只会被系统自动调用一次，如果第一次执行 finalize 方法此对象变成了可达确实不会回收，但如果对象再次被 GC，则会忽略 finalize 方法，对象会被回收！这一点切记! +**注意**：任何一个对象的 `finalize()` 方法只会被系统自动调用一次，如果第一次执行 finalize 方法此对象变成了可达确实不会回收，但如果对象再次被 GC，则会忽略 finalize 方法，对象会被回收！这一点切记! 那么这些 **GC Roots** 到底是什么东西呢，哪些对象可以作为 GC Root 呢，有以下几类 @@ -104,8 +106,11 @@ public class TestRC { - 方法区中类静态属性引用的对象 - 方法区中常量引用的对象 - 本地方法栈中 JNI（即一般说的 Native 方法）引用的对象 +- Java 虚拟机内部的引用，如基本数据类型对应的 Class 对象，一些常驻的异常对象 +- 所有被同步锁（synchronized 关键字）持有的对象 +- 反映 Java 虚拟机内部情况的 JMXBean、JVMTI 中注册的回调、本地代码缓存 -#### 虚拟机栈中引用的对象 +#### 2.2.1 虚拟机栈中引用的对象 如下代码所示，a 是栈帧中的本地变量，当 a = null 时，由于此时 a 充当了 **GC Root** 的作用，a 与原来指向的实例 **new Test()** 断开了连接，所以对象会被回收。 @@ -118,7 +123,7 @@ public class Test { } ``` -#### 方法区中类静态属性引用的对象 +#### 2.2.2 方法区中类静态属性引用的对象 如下代码所示，当栈帧中的本地变量 a = null 时，由于 a 原来指向的对象与 GC Root (变量 a) 断开了连接，所以 a 原来指向的对象会被回收，而由于我们给 s 赋值了变量的引用，s 在此时是类静态属性引用，充当了 GC Root 的作用，它指向的对象依然存活! @@ -133,7 +138,7 @@ public class Test { } ``` -#### 方法区中常量引用的对象 +#### 2.2.3 方法区中常量引用的对象 如下代码所示，常量 s 指向的对象并不会因为 a 指向的对象被回收而回收 @@ -147,13 +152,13 @@ public class Test { } ``` -#### 本地方法栈中 JNI 引用的对象 +#### 2.2.4 本地方法栈中 JNI 引用的对象 所谓本地方法就是一个 Java 调用非 Java 代码的接口，该方法并非 Java 实现的，可能由 C 或 Python等其他语言实现的， Java 通过 JNI 来调用本地方法， 而本地方法是以库文件的形式存放的（在 WINDOWS 平台上是 DLL 文件形式，在 UNIX 机器上是 SO 文件形式）。通过调用本地的库文件的内部方法，使 JAVA 可以实现和本地机器的紧密联系，调用系统级的各接口方法。 当调用 Java 方法时，虚拟机会创建一个栈桢并压入 Java 栈，而当它调用的是本地方法时，虚拟机会保持 Java 栈不变，不会在 Java 栈祯中压入新的祯，虚拟机只是简单地动态连接并直接调用指定的本地方法。 -![img](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9PeXdleXNDU2VMVXJZcVBpY2pWd2p1TUNoUHJQaWNOSGRYdWhiZDlIQ0xCSVlSQmVrUXFpY3M3WEJyajB2VlVWU1BUSFRpYkpidXRyZEdSV2liMU1sRGQxNkpRLzY0MA?x-oss-process=image/format,png) +![](https://dl-harmonyos.51cto.com/images/202212/23a7586494ad8a59df60500df2b64936a33572.png) ```java JNIEXPORT void JNICALL Java_com_pecuyu_jnirefdemo_MainActivity_newStringNative(JNIEnv *env, jobject instance，jstring jmsg) { @@ -166,13 +171,13 @@ JNIEXPORT void JNICALL Java_com_pecuyu_jnirefdemo_MainActivity_newStringNative(J } ``` -如上代码所示，当 Java 调用以上本地方法时，jc 会被本地方法栈压入栈中, jc 就是我们说的本地方法栈中 JNI 的对象引用，因此只会在此本地方法执行完成后才会被释放。 +如上代码所示，当 Java 调用以上本地方法时，jc 会被本地方法栈压入栈中， jc 就是我们说的本地方法栈中 JNI 的对象引用，因此只会在此本地方法执行完成后才会被释放。 ### JDK8 之前的方法区回收 -永久代的垃圾收集主要回收两部分内容：废弃常量和无用的类。 +永久代的垃圾收集主要回收两部分内容：**废弃常量和无用的类**。 废弃常量：与堆中对象回收类似，以常量池中字面量的回收为例，例如资格字符串“abc”已经进入常量池，但没有任何一个 String 对象引用常量池中的"abc"常量，也没有其他地方引用了这个字面量，如果发生内存回收，且有必要的话，这个常量就会被系统清理出常量池。 @@ -182,13 +187,13 @@ JNIEXPORT void JNICALL Java_com_pecuyu_jnirefdemo_MainActivity_newStringNative(J - 加载该类的 ClassLoader 已经被回收 - 该类对应的 java.lang.Class 对象没有在任何地方被引用，无法在任何地方通过反射访问该类的方法。 -虚拟机**可以**对满足以上 3 个条件的无用类进行回收，并不是一定会被回收。是否对类回收，HotSpot虚拟机提供了`-Xnoclassgc` 参数进行控制 +虚拟机**可以**对满足以上 3 个条件的无用类进行回收，并不是一定会被回收。是否对类回收，HotSpot 虚拟机提供了 `-Xnoclassgc` 参数进行控制 ------ -## 垃圾收集算法 +## 三、垃圾收集算法 可以通过可达性算法来识别哪些数据是垃圾，那该怎么对这些垃圾进行回收呢。主要有以下几种方式 @@ -197,45 +202,43 @@ JNIEXPORT void JNICALL Java_com_pecuyu_jnirefdemo_MainActivity_newStringNative(J - 标记整理法 - 分代收集算法 -### 标记清除算法 +### 3.1 标记清除算法 步骤很简单，和名字一样，分为“标记”和“清除”两个阶段 1. 先根据可达性算法**标记**出相应的可回收对象（图中黄色部分） - - -![img](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9PeXdleXNDU2VMVXJZcVBpY2pWd2p1TUNoUHJQaWNOSGRYVWFrUFc2STJoMXhvUXBYTWV2VXpNMmVHd0cwaWE5WXJQTDFRQ0R2WGFXMW13YTNDcnZIZVJpYkEvNjQw?x-oss-process=image/format,png) +![](https://dl-harmonyos.51cto.com/images/202212/f7a858709c100d56d5122606c0a18d51fa3b8e.png) -​ 2.对可回收的对象进行回收操作起来确实很简单，也不用做移动数据的操作，那有啥问题呢？仔细看上图，没错，内存碎片！假如我们想在上图中的堆中分配一块需要**连续内存**占用 4M 或 5M 的区域，显然是会失败，怎么解决呢，如果能把上面未使用的 2M， 2M，1M 内存能连起来就能连成一片可用空间为 5M 的区域即可，怎么做呢? +2. 对可回收的对象进行回收操作起来确实很简单，也不用做移动数据的操作，那有啥问题呢？仔细看上图，没错，内存碎片！假如我们想在上图中的堆中分配一块需要**连续内存**占用 4M 或 5M 的区域，显然是会失败，怎么解决呢，如果能把上面未使用的 2M， 2M，1M 内存能连起来就能连成一片可用空间为 5M 的区域即可，怎么做呢? -### 复制算法 +### 3.2 复制算法 -把堆等分成两块区域, A 和 B，区域 A 负责分配对象，区域 B 不分配, 对区域 A 使用以上所说的标记法把存活的对象标记出来，然后把区域 A 中存活的对象都复制到区域 B（存活对象都依次**紧邻排列**）最后把 A 区对象全部清理掉释放出空间，这样就解决了内存碎片的问题了。 +把堆等分成两块区域, A 和 B，区域 A 负责分配对象，区域 B 不分配，对区域 A 使用以上所说的标记法把存活的对象标记出来，然后把区域 A 中存活的对象都复制到区域 B（存活对象都依次**紧邻排列**）最后把 A 区对象全部清理掉释放出空间，这样就解决了内存碎片的问题了。 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gfws4ow2i1j31gn0q075y.jpg) +![](https://dl-harmonyos.51cto.com/images/202212/c731dc529a180c970ab3849232c06fae11d56d.png) 不过复制算法的缺点很明显，比如给堆分配了 500M 内存，结果只有 250M 可用，空间平白无故减少了一半！这肯定是不能接受的！另外每次回收也要把存活对象移动到另一半，效率低下（我们可以想想删除数组元素再把非删除的元素往一端移，效率显然堪忧） -### 标记整理法 +### 3.3 标记整理法 -前面两步和标记清除法一样，不同的是它在标记清除法的基础上添加了一个整理的过程 ，即将所有的存活对象都往一端移动,紧邻排列（如图示），再清理掉另一端的所有区域，这样的话就解决了内存碎片的问题。 +前面两步和标记清除法一样，不同的是它在标记清除法的基础上添加了一个整理的过程 ，即将所有的存活对象都往一端移动，紧邻排列（如图示），再清理掉另一端的所有区域，这样的话就解决了内存碎片的问题。 -![img](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9PeXdleXNDU2VMVXJZcVBpY2pWd2p1TUNoUHJQaWNOSGRYUzM1aWN4MTltaWIzZFpDRW0zZk9kbHcyY2xZYldUSnRoVTVpYm45WnRjOGtRdWFvc3hPYWNoUVd3LzY0MA?x-oss-process=image/format,png) +![](https://dl-harmonyos.51cto.com/images/202212/c5a4a0054791c5339646160ea1f1ccc03a4818.png) 但是缺点也很明显：每进一次垃圾清除都要频繁地移动存活的对象，效率十分低下。 -### 分代收集算法 +### 3.4 分代收集算法 -分代收集算法整合了以上算法，综合了这些算法的优点，最大程度避免了它们的缺点，所以是现代虚拟机采用的首选算法,与其说它是算法，倒不是说它是一种策略，因为它是把上述几种算法整合在了一起，为啥需要分代收集呢，来看一下对象的分配有啥规律 +分代收集算法整合了以上算法，综合了这些算法的优点，最大程度避免了它们的缺点，所以是现代虚拟机采用的首选算法，与其说它是算法，倒不是说它是一种策略，因为它是把上述几种算法整合在了一起，为啥需要分代收集呢，来看一下对象的分配有啥规律 -![img](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X2pwZy9PeXdleXNDU2VMVXJZcVBpY2pWd2p1TUNoUHJQaWNOSGRYRjFmRXJWTmlhTHJ3ZW5NdVJGcHhOTE03a0pVSkdocVlnMWljUnpiaEVXT3lwWmZpY3F5eUdiaWFIdy82NDA?x-oss-process=image/format,png) +![](https://dl-harmonyos.51cto.com/images/202212/74acec408804279b3d42517a39878d08efa77f.jpg) *如图示：纵轴代表已分配的字节，而横轴代表程序运行时间* -由图可知，大部分的对象都很短命，都在很短的时间内都被回收了（IBM 专业研究表明，一般来说，98% 的对象都是朝生夕死的，经过一次 Minor GC 后就会被回收），所以分代收集算法根据**对象存活周期的不同**将堆分成新生代和老生代（Java8以前还有个永久代）,默认比例为 1 : 2，新生代又分为 Eden 区， from Survivor 区（简称S0），to Survivor 区(简称 S1),三者的比例为 8: 1 : 1，这样就可以根据新老生代的特点选择最合适的垃圾回收算法，我们把新生代发生的 GC 称为 Young GC（也叫 Minor GC），老年代发生的 GC 称为 Old GC（也称为 Full GC）。 +由图可知，大部分的对象都很短命，都在很短的时间内都被回收了（IBM 专业研究表明，一般来说，98% 的对象都是朝生夕死的，经过一次 Minor GC 后就会被回收），所以分代收集算法根据**对象存活周期的不同**将堆分成新生代和老生代（Java8 以前还有个永久代），默认比例为 1 : 2，新生代又分为 Eden 区， from Survivor 区（简称S0），to Survivor 区(简称 S1)，三者的比例为 `8: 1 : 1`，这样就可以根据新老生代的特点选择最合适的垃圾回收算法，我们把新生代发生的 GC 称为 Young GC（也叫 Minor GC），老年代发生的 GC 称为 Old GC。 -![img](https://tva1.sinaimg.cn/large/00831rSTly1gdc2yeeoz8j30aj04mglm.jpg) +![看完这篇垃圾回收，和面试官扯皮没问题了-鸿蒙开发者社区](https://dl-harmonyos.51cto.com/images/202212/74ab84b02f3754a4a10605675df0c59a6ce4ab.png) *画外音：思考一下，新生代为啥要分这么多区？* @@ -247,82 +250,110 @@ JNIEXPORT void JNICALL Java_com_pecuyu_jnirefdemo_MainActivity_newStringNative(J 由以上的分析可知，大部分对象在很短的时间内都会被回收，对象一般分配在 Eden 区 -![img](https://tva1.sinaimg.cn/large/00831rSTly1gdc2y9r7vsj30qk07kgls.jpg) +![看完这篇垃圾回收，和面试官扯皮没问题了-鸿蒙开发者社区](https://dl-harmonyos.51cto.com/images/202212/5116f2039552d5d4ed23789b7cc86ffa62247a.png) + +当 Eden 区将满时，触发 Minor GC!![看完这篇垃圾回收，和面试官扯皮没问题了-鸿蒙开发者社区](https://dl-harmonyos.51cto.com/images/202212/535f4cf00d99ad7495a3963baebe4d8481c205.png) + +我们之前怎么说来着，大部分对象在短时间内都会被回收，所以经过 Minor GC 后只有少部分对象会存活，它们会被移到 S0 区（这就是为啥空间大小 `Eden: S0: S1 = 8:1:1`，Eden 区远大于 S0，S1 的原因，因为在 Eden 区触发的 Minor GC 把大部对象（接近98%）都回收了，只留下少量存活的对象，此时把它们移到 S0 或 S1 绰绰有余）同时对象年龄加一（对象的年龄即发生 Minor GC 的次数），最后把 Eden 区对象全部清理以释放出空间，如下 -当 Eden 区将满时，触发 Minor GC![img](https://tva1.sinaimg.cn/large/00831rSTly1gdc2ynwz4oj30q007wmxf.jpg) +![看完这篇垃圾回收，和面试官扯皮没问题了](https://dl-harmonyos.51cto.com/images/202212/a2e29f7447f1c3aac22916ea7c2aabd1dea1e9.gif) -我们之前怎么说来着，大部分对象在短时间内都会被回收，所以经过 Minor GC 后只有少部分对象会存活，它们会被移到 S0 区（这就是为啥空间大小 Eden: S0: S1 = 8:1:1, Eden 区远大于 S0,S1 的原因，因为在 Eden 区触发的 Minor GC 把大部对象（接近98%）都回收了,只留下少量存活的对象，此时把它们移到 S0 或 S1 绰绰有余）同时对象年龄加一（对象的年龄即发生 Minor GC 的次数），最后把 Eden 区对象全部清理以释放出空间，如下 +当触发下一次 Minor GC 时，会把 Eden 区的存活对象和 S0（或S1） 中的存活对象（S0 或 S1 中的存活对象经过每次 Minor GC 都可能被回收）一起移到 S1（Eden 和 S0 的存活对象年龄+1）, 同时清空 Eden 和 S0 的空间。![看完这篇垃圾回收，和面试官扯皮没问题了-鸿蒙开发者社区](https://dl-harmonyos.51cto.com/images/202212/97280d7105a02be986e3552ff90ef295b13529.gif) -![img](https://tva1.sinaimg.cn/large/00831rSTly1gdc2z7ahusg30hr09l0v2.gif) +若再触发下一次 Minor GC，则重复上一步，只不过此时变成了从 Eden，S1 区将存活对象复制到 S0 区，每次垃圾回收，S0，S1 角色互换，都是从 Eden ，S0(或S1) 将存活对象移动到 S1(或S0)。也就是说在 Eden 区的垃圾回收我们采用的是**复制算法**，因为在 Eden 区分配的对象大部分在 Minor GC 后都消亡了，只剩下极少部分存活对象，S0，S1 区域也比较小，所以最大限度地降低了复制算法造成的对象频繁拷贝带来的开销。 -当触发下一次 Minor GC 时，会把 Eden 区的存活对象和 S0（或S1） 中的存活对象（S0 或 S1 中的存活对象经过每次 Minor GC 都可能被回收）一起移到 S1（Eden 和 S0 的存活对象年龄+1）, 同时清空 Eden 和 S0 的空间。![img](https://tva1.sinaimg.cn/large/00831rSTly1gdc2zeh8bvg30hq09h76r.gif) -若再触发下一次 Minor GC，则重复上一步，只不过此时变成了 从 Eden，S1 区将存活对象复制到 S0 区，每次垃圾回收,，S0，S1 角色互换，都是从 Eden ，S0(或S1) 将存活对象移动到 S1(或S0)。也就是说在 Eden 区的垃圾回收我们采用的是**复制算法**，因为在 Eden 区分配的对象大部分在 Minor GC 后都消亡了，只剩下极少部分存活对象（这也是为啥 Eden:S0:S1 默认为 8:1:1 的原因），S0，S1 区域也比较小，所以最大限度地降低了复制算法造成的对象频繁拷贝带来的开销。 **2、对象何时晋升老年代** -- 当对象的年龄达到了我们设定的阈值，则会从S0（或S1）晋升到老年代![img](https://tva1.sinaimg.cn/large/00831rSTly1gdc2zt5evgg30hs0axgnj.gif) +- 当对象的年龄达到了我们设定的阈值，则会从S0（或S1）晋升到老年代![看完这篇垃圾回收，和面试官扯皮没问题了-鸿蒙开发者社区](https://dl-harmonyos.51cto.com/images/202212/c1b3566218e225dae99945839dc3fa01bea3ba.gif) 如图示：年龄阈值设置为 15（默认为15岁）， 当发生下一次 Minor GC 时，S0 中有个对象年龄达到 15，达到我们的设定阈值，晋升到老年代！ -- 大对象，当某个对象分配需要大量的连续内存时，此时对象的创建不会分配在 Eden 区，会直接分配在老年代，因为如果把大对象分配在 Eden 区，Minor GC 后再移动到 S0,S1 会有很大的开销（对象比较大，复制会比较慢，也占空间），也很快会占满 S0,S1 区，所以干脆就直接移到老年代。最典型的大对象就是那种很长的字符串以及数组 +- 大对象，当某个对象分配需要大量的连续内存时，此时对象的创建不会分配在 Eden 区，会直接分配在老年代，因为如果把大对象分配在 Eden 区，Minor GC 后再移动到 S0，S1 会有很大的开销（对象比较大，复制会比较慢，也占空间），也很快会占满 S0，S1 区，所以干脆就直接移到老年代。最典型的大对象就是那种很长的字符串以及数组 - 还有一种情况也会让对象晋升到老年代，即在 S0（或S1） 区相同年龄的对象大小之和大于 S0（或S1）空间一半以上时，则年龄大于等于该年龄的对象也会晋升到老年代。 **3、空间分配担保** -在发生 MinorGC 之前，虚拟机会先检查老年代最大可用的连续空间是否大于新生代所有对象的总空间，如果大于，那么Minor GC 可以确保是安全的，如果不大于，那么虚拟机会查看 `HandlePromotionFailure` 设置值是否允许担保失败。如果允许，那么会继续检查老年代最大可用连续空间是否大于历次晋升到老年代对象的平均大小，如果大于则进行 Minor GC，否则可能进行一次 Full GC。 +在发生 MinorGC 之前，虚拟机会先检查老年代最大可用的连续空间是否大于新生代所有对象的总空间，如果大于，那么 Minor GC 可以确保是安全的，如果不大于，那么虚拟机会查看 `HandlePromotionFailure` 设置值是否允许担保失败。如果允许，那么会继续检查老年代最大可用连续空间是否大于历次晋升到老年代对象的平均大小，如果大于则进行 Minor GC，否则可能进行一次 Full GC。 **4、Stop The World** 如果老年代满了，会触发 Full GC， Full GC 会同时回收新生代和老年代（即对整个堆进行GC），它会导致 Stop The World（简称 STW），造成挺大的性能开销。 -什么是 STW ？所谓的 STW, 即在 GC（minor GC 或 Full GC）期间，只有垃圾回收器线程在工作，其他工作线程则被挂起。 +什么是 STW ？所谓的 STW，即在 GC（minor GC 或 Full GC）期间，只有垃圾回收器线程在工作，其他工作线程则被挂起。 -![img](https://tva1.sinaimg.cn/large/00831rSTly1gdc2zzpdzfj30tk0go3zo.jpg) +![看完这篇垃圾回收，和面试官扯皮没问题了-鸿蒙开发者社区](https://dl-harmonyos.51cto.com/images/202212/d39094014a05d6db78370782eca38f2f38b6db.png) *画外音：为啥在垃圾收集期间其他工作线程会被挂起？想象一下，你一边在收垃圾，另外一群人一边丢垃圾，垃圾能收拾干净吗。* 一般 Full GC 会导致工作线程停顿时间过长（因为Full GC 会清理**整个堆**中的不可用对象，一般要花较长的时间），如果在此 server 收到了很多请求，则会被拒绝服务！所以我们要尽量减少 Full GC（Minor GC 也会造成 STW，但只会触发轻微的 STW，因为 Eden 区的对象大部分都被回收了，只有极少数存活对象会通过复制算法转移到 S0 或 S1 区，所以相对还好）。 -现在我们应该明白把新生代设置成 Eden, S0，S1区或者给对象设置年龄阈值或者默认把新生代与老年代的空间大小设置成 1:2 都是为了**尽可能地避免对象过早地进入老年代，尽可能晚地触发 Full GC**。想想新生代如果只设置 Eden 会发生什么，后果就是每经过一次 Minor GC，存活对象会过早地进入老年代，那么老年代很快就会装满，很快会触发 Full GC，而对象其实在经过两三次的 Minor GC 后大部分都会消亡，所以有了 S0,S1的缓冲，只有少数的对象会进入老年代，老年代大小也就不会这么快地增长，也就避免了过早地触发 Full GC。 +现在我们应该明白把新生代设置成 Eden， S0，S1区或者给对象设置年龄阈值或者默认把新生代与老年代的空间大小设置成 1:2 都是为了**尽可能地避免对象过早地进入老年代，尽可能晚地触发 Full GC**。想想新生代如果只设置 Eden 会发生什么，后果就是每经过一次 Minor GC，存活对象会过早地进入老年代，那么老年代很快就会装满，很快会触发 Full GC，而对象其实在经过两三次的 Minor GC 后大部分都会消亡，所以有了 S0，S1的缓冲，只有少数的对象会进入老年代，老年代大小也就不会这么快地增长，也就避免了过早地触发 Full GC。 由于 Full GC（或Minor GC） 会影响性能，所以我们要在一个合适的时间点发起 GC，这个时间点被称为 **Safe Point（安全点）**，这个时间点的选定既不能太少以让 GC 时间太长导致程序过长时间卡顿，也不能过于频繁以至于过分增大运行时的负荷。一般当线程在这个时间点上状态是可以确定的，如确定 GC Root 的信息等，可以使 JVM 开始安全地 GC。Safe Point 主要指的是以下特定位置： - 循环的末尾 - 方法返回前 - 调用方法的 call 之后 -- 抛出异常的位置 另外需要注意的是由于新生代的特点（大部分对象经过 Minor GC后会消亡）， Minor GC 用的是复制算法，而在老生代由于对象比较多，占用的空间较大，使用复制算法会有较大开销（复制算法在对象存活率较高时要进行多次复制操作，同时浪费一半空间）所以根据老生代特点，在老年代进行的 GC 一般采用的是标记整理法来进行回收。 +- 抛出异常的位置 另外需要注意的是由于新生代的特点（大部分对象经过 Minor GC后会消亡）， Minor GC 用的是复制算法，而在老生代由于对象比较多，占用的空间较大，使用复制算法会有较大开销（复制算法在对象存活率较高时要进行多次复制操作，同时浪费一半空间）所以根据老年代的特点，在老年代进行的 GC 一般采用的是标记整理法来进行回收。 + + + +> 《深入理解 Java 虚拟机》中有个注意事项，整理了下各种 GC，避免混乱 +> +> - 部分收集（Partial GC）：指目标不是完整收集整个 Java 堆的垃圾收集，其中又分为： +> - 新生代收集（Minor GC/Young GC）：指目标只是新生代的垃圾收集。 +> - 老年代收集（Major GC/Old GC）：指目标只是老年代的垃圾收集。目前只有 CMS 收集器会有单独收集老年代的行为。另外请注意“Major GC”这个说法现在有点混淆，在不同资料上常有不同所指，读者需按上下文区分到底是指老年代的收集还是整堆收集。 +> - 混合收集（Mixed GC）：指目标是收集整个新生代以及部分老年代的垃圾收集。目前只有 G1 收集器会有这种行为。 +> +> - 整堆收集（Full GC）：收集整个 Java 堆和方法区的垃圾收集。 + + + +**什么时候会触发 Full GC ?** + +1. `System.gc()` 方法的调用 + + 此方法的调用是建议 JVM 进行 Full GC，虽然只是建议而非一定，但很多情况下它会触发 Full GC，从而增加Full GC 的频率，也即增加了间歇性停顿的次数。强烈建议能不使用此方法就别使用，让虚拟机自己去管理它的内存，可通过通过 -XX:+ DisableExplicitGC 来禁止 RMI 调用 System.gc。 + +2. 老年代空间不足 + + 老年代空间只有在新生代对象转入及创建大对象、大数组时才会出现不足的现象，当执行 Full GC 后空间仍然不足，则抛出如下错误：java.lang.OutOfMemoryError: Java heap space + 为避免以上两种状况引起的 Full GC，调优时应尽量做到让对象在 Minor GC 阶段被回收、让对象在新生代多存活一段时间及不要创建过大的对象及数组。 + +3. 老年代的内存使用率达到了一定阈值（可通过参数调整），直接触发 FGC +4. 空间分配担保：在 YGC 之前，会先检查老年代最大可用的连续空间是否大于新生代所有对象的总空间。如果小于，说明 YGC 是不安全的，则会查看参数 HandlePromotionFailure 是否被设置成了允许担保失败，如果不允许则直接触发 Full GC；如果允许，那么会进一步检查老年代最大可用的连续空间是否大于历次晋升到老年代对象的平均大小，如果小于也会触发 Full GC +5. Metaspace（元空间）在空间不足时会进行扩容，当扩容到了-XX:MetaspaceSize 参数的指定值时，也会触发FGC ------ -## 垃圾收集器种类 +## 四、垃圾收集器种类 如果说收集算法是内存回收的方法论，那么垃圾收集器就是内存回收的具体实现。Java 虚拟机规范并没有规定垃圾收集器应该如何实现，因此一般来说不同厂商，不同版本的虚拟机提供的垃圾收集器实现可能会有差别，一般会给出参数来让用户根据应用的特点来组合各个年代使用的收集器，主要有以下垃圾收集器 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gehop8c2u8j30uu0kgk46.jpg) +![看完这篇垃圾回收，和面试官扯皮没问题了-鸿蒙开发者社区](https://dl-harmonyos.51cto.com/images/202212/f8c72c72318c032f5e802774787b4cfa9b940f.png) -![img](https://tva1.sinaimg.cn/large/00831rSTly1gdc303ew8zj30py0mojsm.jpg) - -- 在新生代工作的垃圾回收器：Serial, ParNew, ParallelScavenge -- 在老年代工作的垃圾回收器：CMS，Serial Old(MSC), Parallel Old +- 在新生代工作的垃圾回收器：Serial、ParNew、ParallelScavenge +- 在老年代工作的垃圾回收器：CMS、Serial Old(MSC)、Parallel Old - 同时在新老生代工作的垃圾回收器：G1 图片中的垃圾收集器如果存在连线，则代表它们之间可以配合使用，接下来我们来看看各个垃圾收集器的具体功能。 -### 新生代收集器 +### 4.1 新生代收集器 #### Serial 收集器 -Serial 收集器是工作在新生代的，单线程的垃圾收集器，单线程意味着它只会使用一个 CPU 或一个收集线程来完成垃圾回收，不仅如此，还记得我们上文提到的 Stop The World 吗，它在进行垃圾收集时，其他用户线程会暂停，直到垃圾收集结束，也就是说在 GC 期间，此时的应用不可用。 +Serial 收集器是工作在新生代的，**单线程的垃圾收集器**，单线程意味着它只会使用一个 CPU 或一个收集线程来完成垃圾回收，不仅如此，还记得我们上文提到的 Stop The World 吗，它在进行垃圾收集时，其他用户线程会暂停，直到垃圾收集结束，也就是说在 GC 期间，此时的应用不可用。 看起来单线程垃圾收集器不太实用，不过我们需要知道的任何技术的使用都不能脱离场景，在 **Client 模式**下，它简单有效（与其他收集器的单线程比），对于限定单个 CPU 的环境来说，Serial 单线程模式无需与其他线程交互，减少了开销，专心做 GC 能将其单线程的优势发挥到极致，另外在用户的桌面应用场景，分配给虚拟机的内存一般不会很大，收集几十甚至一两百兆（仅是新生代的内存，桌面应用基本不会再大了），STW 时间可以控制在一百多毫秒内，只要不是频繁发生，这点停顿是可以接受的，所以对于运行在 Client 模式下的虚拟机，Serial 收集器是新生代的默认收集器 #### ParNew 收集器 -ParNew 收集器是 Serial 收集器的多线程版本，除了使用多线程，其他像收集算法、STW、对象分配规则、回收策略与 Serial 收集器完成一样，在底层上，这两种收集器也共用了相当多的代码，它的垃圾收集过程如下![img](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9PeXdleXNDU2VMVXJZcVBpY2pWd2p1TUNoUHJQaWNOSGRYYUNIUWxRNTRtR2ptSjVhYzRENzAwWDhuRkJjTXFwYXlqR1dkcmRoTmNNUFk4OU5acFB2TFNBLzY0MA?x-oss-process=image/format,png) +ParNew 收集器是 Serial 收集器的多线程版本，除了使用多线程，其他像收集算法、STW、对象分配规则、回收策略与 Serial 收集器完全一样，在底层上，这两种收集器也共用了相当多的代码，它的垃圾收集过程如下![看完这篇垃圾回收，和面试官扯皮没问题了-鸿蒙开发者社区](https://dl-harmonyos.51cto.com/images/202212/d80fe3083c3264766921433926400276041c00.png) ParNew 主要工作在 Server 模式，我们知道服务端如果接收的请求多了，响应时间就很重要了，多线程可以让垃圾回收得更快，也就是减少了 STW 时间，能提升响应时间，所以是许多运行在 Server 模式下的虚拟机的首选新生代收集器，另一个与性能无关的原因是因为除了 Serial 收集器，**只有它能与 CMS 收集器配合工作**，CMS 是一个划时代的垃圾收集器，是真正意义上的**并发收集器**，它第一次实现了垃圾收集线程与用户线程（基本上）同时工作，它采用的是传统的 GC 收集器代码框架，与 Serial，ParNew 共用一套代码框架，所以能与这两者一起配合工作，而后文提到的 Parallel Scavenge 与 G1 收集器没有使用传统的 GC 收集器代码框架，而是另起炉灶独立实现的，另外一些收集器则只是共用了部分的框架代码，所以无法与 CMS 收集器一起配合工作。 @@ -338,11 +369,11 @@ Parallel Scavenge 收集器提供了两个参数来精确控制吞吐量，分 除了以上两个参数，还可以用 Parallel Scavenge 收集器提供的第三个参数 `-XX:UseAdaptiveSizePolicy`，开启这个参数后，就不需要手工指定新生代大小，Eden 与 Survivor 比例（SurvivorRatio）等细节，只需要设置好基本的堆大小（-Xmx 设置最大堆），以及最大垃圾收集时间与吞吐量大小，虚拟机就会根据当前系统运行情况收集监控信息，动态调整这些参数以尽可能地达到我们设定的最大垃圾收集时间或吞吐量大小这两个指标。**自适应策略**也是 Parallel Scavenge 与 ParNew 的重要区别！ -### 老年代收集器 +### 4.2 老年代收集器 #### Serial Old 收集器 -Serial 收集器是工作于新生代的单线程收集器，与之相对地，Serial Old 是工作于老年代的单线程收集器，此收集器的主要意义在于给 Client 模式下的虚拟机使用，如果在 Server 模式下，则它还有两大用途：一种是在 JDK 1.5 及之前的版本中与 Parallel Scavenge 配合使用，另一种是作为 CMS 收集器的后备预案,在并发收集发生 Concurrent Mode Failure 时使用，它与 Serial 收集器配合使用示意图如下 +Serial 收集器是工作于新生代的单线程收集器，与之相对地，Serial Old 是工作于老年代的单线程收集器，此收集器的主要意义在于给 Client 模式下的虚拟机使用，如果在 Server 模式下，则它还有两大用途：一种是在 JDK 1.5 及之前的版本中与 Parallel Scavenge 配合使用，另一种是作为 CMS 收集器的后备预案，在并发收集发生 Concurrent Mode Failure 时使用，它与 Serial 收集器配合使用示意图如下 ![img](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9PeXdleXNDU2VMVXJZcVBpY2pWd2p1TUNoUHJQaWNOSGRYWHNFaEpMN29haFo5SmpLdzdFS0oxcnIxaWM2ZlBUckV6TGlhOEVkZTRUMnVxWmRPVWVxcmYwbncvNjQw?x-oss-process=image/format,png) @@ -356,7 +387,7 @@ Parallel Old 是相对于 Parallel Scavenge 收集器的老年代版本，使用 CMS（Concurrent Mark Sweep） 收集器是以实现最短 STW 时间为目标的收集器，如果应用很重视服务的响应速度，希望给用户最好的体验，则 CMS 收集器是个很不错的选择！ -我们之前说老年代主要用标记整理法，而 CMS 虽然工作于老年代，但采用的是**标记清除算法**，主要有以下四个步骤 +我们之前说老年代主要用标记整理法，而 CMS 虽然工作于老年代，但采用的是**标记清除算法**，主要有以下四个步骤 1. 初始标记 2. 并发标记 @@ -365,15 +396,15 @@ CMS（Concurrent Mark Sweep） 收集器是以实现最短 STW 时间为目标 ![img](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9PeXdleXNDU2VMVXJZcVBpY2pWd2p1TUNoUHJQaWNOSGRYQmswSGRhQTR4MUZxa29iS1J4ZEVyaWJSTVFuODZ6RHQ5RnBjZU8xaWNtTHdkNm9pY2hTams0aWJaUS82NDA?x-oss-process=image/format,png) -从图中可以的看到初始标记和重新标记两个阶段会发生 STW，造成用户线程挂起，不过初始标记仅标记 GC Roots 能关联的对象，速度很快，并发标记是进行 GC Roots Tracing 的过程，重新标记是为了修正并发标记期间因用户线程继续运行而导致标记产生变动的那一部分对象的标记记录，这一阶段停顿时间一般比初始标记阶段稍长，但**远比并发标记时间短**。 +从图中可以的看到**初始标记和重新标记两个阶段会发生 STW**，造成用户线程挂起，不过初始标记仅标记 GC Roots 能关联的对象，速度很快，并发标记就是从 GC Roots 的直接关联对象开始遍历整个对象图的过程，这个过程耗时较长但是不需要停顿用户线程，重新标记是为了修正并发标记期间因用户线程继续运行而导致标记产生变动的那一部分对象的标记记录，这一阶段停顿时间一般比初始标记阶段稍长，但**远比并发标记时间短**。 整个过程中耗时最长的是并发标记和标记清理，不过这两个阶段用户线程都可工作，所以不影响应用的正常使用，所以总体上看，可以认为 CMS 收集器的内存回收过程是与用户线程一起并发执行的。 但是 CMS 收集器远达不到完美的程度，主要有以下三个缺点 - CMS 收集器对 CPU 资源非常敏感 。原因也可以理解，比如本来我本来可以有 10 个用户线程处理请求，现在却要分出 3 个作为回收线程，吞吐量下降了30%，CMS 默认启动的回收线程数是 （CPU数量+3）/ 4，如果 CPU 数量只有一两个，那吞吐量就直接下降 50%，显然是不可接受的 -- CMS 无法处理浮动垃圾（Floating Garbage）,可能出现 「Concurrent Mode Failure」而导致另一次 Full GC 的产生，由于在并发清理阶段用户线程还在运行，所以清理的同时新的垃圾也在不断出现，这部分垃圾只能在下一次 GC 时再清理掉（即浮云垃圾），同时在垃圾收集阶段用户线程也要继续运行，就需要预留足够多的空间要确保用户线程正常执行，这就意味着 CMS 收集器不能像其他收集器一样等老年代满了再使用，JDK 1.5 默认当老年代使用了68%空间后就会被激活，当然这个比例可以通过 -XX:CMSInitiatingOccupancyFraction 来设置，但是如果设置地太高很容易导致在 CMS 运行期间预留的内存无法满足程序要求，会导致 **Concurrent Mode Failure** 失败，这时会启用 Serial Old 收集器来重新进行老年代的收集，而我们知道 Serial Old 收集器是单线程收集器，这样就会导致 STW 更长了。 -- CMS 采用的是标记清除法，上文我们已经提到这种方法会产生大量的内存碎片，这样会给大内存分配带来很大的麻烦，如果无法找到足够大的连续空间来分配对象，将会触发 Full GC，这会影响应用的性能。当然我们可以开启 -XX:+UseCMSCompactAtFullCollection（默认是开启的），用于在 CMS 收集器顶不住要进行 FullGC 时开启内存碎片的合并整理过程，内存整理会导致 STW，停顿时间会变长，还可以用另一个参数 -XX:CMSFullGCsBeforeCompation 用来设置执行多少次不压缩的 Full GC 后跟着带来一次带压缩的。 +- **CMS 无法处理浮动垃圾**（Floating Garbage），可能出现 「Concurrent Mode Failure」而导致另一次 Full GC 的产生，由于在并发清理阶段用户线程还在运行，所以清理的同时新的垃圾也在不断出现，这部分垃圾只能在下一次 GC 时再清理掉（即浮云垃圾），同时在垃圾收集阶段用户线程也要继续运行，就需要预留足够多的空间要确保用户线程正常执行，这就意味着 CMS 收集器不能像其他收集器一样等老年代满了再使用，JDK 1.5 默认当老年代使用了 68% 空间后就会被激活，当然这个比例可以通过 `-XX:CMSInitiatingOccupancyFraction` 来设置，但是如果设置的太高很容易导致在 CMS 运行期间预留的内存无法满足程序要求，会导致 **Concurrent Mode Failure** 失败，这时会启用 Serial Old 收集器来重新进行老年代的收集，而我们知道 Serial Old 收集器是单线程收集器，这样就会导致 STW 更长了。 +- CMS 采用的是标记清除法，上文我们已经提到这种方法**会产生大量的内存碎片**，这样会给大内存分配带来很大的麻烦，如果无法找到足够大的连续空间来分配对象，将会触发 Full GC，这会影响应用的性能。当然我们可以开启 `-XX:+UseCMSCompactAtFullCollection`（默认是开启的），用于在 CMS 收集器顶不住要进行 FullGC 时开启内存碎片的合并整理过程，内存整理会导致 STW，停顿时间会变长，还可以用另一个参数 `-XX:CMSFullGCsBeforeCompation` 用来设置执行多少次不压缩的 Full GC 后跟着带来一次带压缩的。 #### G1（Garbage First） 收集器 @@ -389,23 +420,23 @@ G1 收集器是面向服务端的垃圾收集器，被称为驾驭一切的垃 - 不需要更大的 Java Heap - +> Java8 默认使用 Parallel Scavenge + Parallel Old 组合，Java9 开始 G1 取代了它们。 与 CMS 相比，它在以下两个方面表现更出色 -1. 运作期间不会产生内存碎片，G1 从整体上看采用的是标记-整理法，局部（两个 Region）上看是基于复制算法实现的，两个算法都不会产生内存碎片，收集后提供规整的可用内存，这样有利于程序的长时间运行。 +1. 运作期间不会产生内存碎片，G1 从整体上看采用的是标记-整理法，局部（两个 Region）上看是基于复制算法实现的，两个算法都不会产生内存碎片，收集后提供规整的可用内存，这样有利于程序的长时间运行。 2. 在 STW 上建立了**可预测**的停顿时间模型，用户可以指定期望停顿时间，G1 会将停顿时间控制在用户设定的停顿时间以内。 -为什么G1能建立可预测的停顿模型呢，主要原因在于 G1 对堆空间的分配与传统的垃圾收集器不一器，传统的内存分配就像我们前文所述，是连续的，分成新生代，老年代，新生代又分 Eden,S0,S1,如下 +为什么 G1 能建立可预测的停顿模型呢，主要原因在于 G1 对堆空间的分配与传统的垃圾收集器不一样，传统的内存分配就像我们前文所述，是连续的，分成新生代，老年代，新生代又分 Eden，S0，S1,如下 -![img](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9PeXdleXNDU2VMVXJZcVBpY2pWd2p1TUNoUHJQaWNOSGRYdW5xUk11aHE2RlpiY0tpY05pYmN2S21lVHdpYzA1b3Jkb1oyd2dXNGhINUt0TWxLdThEdlFpYkU1dy82NDA?x-oss-process=image/format,png) +![看完这篇垃圾回收，和面试官扯皮没问题了-鸿蒙开发者社区](https://dl-harmonyos.51cto.com/images/202212/e3876dc9892d4d6b6c90814065700c28618baf.png) -而 G1 各代的存储地址不是连续的，每一代都使用了 n 个不连续的大小相同的 Region，每个Region占有一块连续的虚拟内存地址，如图示 +而 G1 各代的存储地址不是连续的，每一代都使用了 n 个不连续的大小相同的 Region，每个 Region 占有一块连续的虚拟内存地址，如图示 ![img](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9PeXdleXNDU2VMVXJZcVBpY2pWd2p1TUNoUHJQaWNOSGRYN2VZbmliVUZIdW83QWliWnlMc2FIcTZ0V3BXbndMTzNqajFyazNtVGhyQVBTc2RtVk1yZ3psTFEvNjQw?x-oss-process=image/format,png) -除了和传统的新老生代，幸存区的空间区别，Region还多了一个H，它代表Humongous，这表示这些Region存储的是巨大对象（humongous object，H-obj），即大小大于等于region一半的对象，这样超大对象就直接分配到了老年代，防止了反复拷贝移动。那么 G1 分配成这样有啥好处呢？ +除了和传统的新老生代，幸存区的空间区别，Region 还多了一个H，它代表 Humongous，这表示这些 Region 存储的是巨大对象（humongous object，H-obj），即大小大于等于 region 一半的对象，这样超大对象就直接分配到了老年代，防止了反复拷贝移动。那么 G1 分配成这样有啥好处呢？ 传统的收集器如果发生 Full GC 是对整个堆进行全区域的垃圾收集，而分配成各个 Region 的话，方便 G1 跟踪各个 Region 里垃圾堆积的价值大小（回收所获得的空间大小及回收所需经验值），这样根据价值大小维护一个优先列表，根据允许的收集时间，优先收集回收价值最大的 Region，也就避免了整个老年代的回收，也就减少了 STW 造成的停顿时间。同时由于只收集部分 Region，也就做到了 STW 时间的可控。 @@ -422,35 +453,66 @@ G1 收集器的工作步骤如下 -## GC 日志 +## 五、GC 日志 +接下来我们看看 GC 日志怎么看，日志可以有效地帮助我们定位问题，所以搞清楚 GC 日志的格式非常重要，来看下如下例子 +```java +/** + * VM Args:-verbose:gc -Xms20M -Xmx20M -Xmn10M -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseSerialGC -XX:SurvivorRatio=8 + */ +public class TestGC { + private static final int _1MB = 1024 * 1024; + public static void main(String[] args) { + byte[] allocation1, allocation2, allocation3, allocation4; + allocation1 = new byte[2 * _1MB]; + allocation2 = new byte[2 * _1MB]; + allocation3 = new byte[2 * _1MB]; + allocation4 = new byte[4 * _1MB]; // 这里会出现一次 Minor GC + } +} +``` -## 内存分配和回收策略 +执行以上代码，会输出如下 GC 日志信息 -Java 技术体系所提倡的自动内存管理最终可以归结为自动化的解决两个问题 +``` +10.080: 2[GC 3(Allocation Failure) 0.080: 4[DefNew: 56815K->280K(9216K),6 0.0043690 secs] 76815K->6424K(19456K), 80.0044111 secs]9 [Times: user=0.00 sys=0.01, real=0.01 secs] +``` -- 给对象分配内存 -- 回收分配给对象的内存 +以上是发生 Minor GC 的 GC 是日志，如果发生 Full GC 呢，格式如下 -回收内存上边已经介绍了很多了，而对象的内存分配，往大方向上讲，就是在堆上分配，对象主要分配在新生代的 Eden 区上，如果启动了本地线程分配缓冲，将按线程优先在 TLAB 上分配。少数情况可能直接分配在老年代，分配的规则取决于使用哪种垃圾收集器。 +``` +10.088: 2[Full GC 3(Allocation Failure) 0.088: 4[Tenured: 50K->210K(10240K), 60.0009420 secs] 74603K->210K(19456K), [Metaspace: 2630K->2630K(1056768K)], 80.0009700 secs]9 [Times: user=0.01 sys=0.00, real=0.02 secs] +``` -## 总结 +两者格式其实差不多，一起来看看，主要以本例触发的 Minor GC 来讲解，以上日志中标的每一个数字与以下序号一一对应 -本文简述了垃圾回收的原理与垃圾收集器的种类，相信大家对开头提的一些问题应该有了更深刻的认识，在生产环境中我们要根据**不同的场景**来选择垃圾收集器组合，如果是运行在桌面环境处于 Client 模式的，则用 Serial + Serial Old 收集器绰绰有余，如果需要响应时间快，用户体验好的，则用 ParNew + CMS 的搭配模式，即使是号称是「驾驭一切」的 G1，也需要根据吞吐量等要求适当调整相应的 JVM 参数，没有最牛的技术，只有最合适的使用场景，切记！ +1. 开头的 0.080，0.088 代表了 GC 发生的时间，这个数字的含义是从 Java 虚拟机启动以来经过的秒数 +2. **[GC** 或者 **[Full GC** 说明了这次垃圾收集的停顿类型，注意不是用来区分新生代 GC 还是老年化 GC 的，如果有 **Full**，说明这次 GC 是发生了**Stop The World** 的，如果是调用 System.gc() 所触发的收集，这里会显示 **[Full GC（System）** +3. 之后的 **Allocation Failure** 代表了触发 GC 的原因，在这个程序中我们设置了新生代的大小为 10M（-Xmn10M），Eden:S0:S1 = 8：1：1（-XX:SurvivorRatio=8），也就是说 Eden 区占了 8M, 当分配 allocation4 时，由于将要分配的总大小为 10M,超过了 Eden 区，所以此时会发生 GC +4. 接下来的 **[DefNew**，**[Tenured**，**[Metaspace** 表示 GC 发生的区域，这里显示的区域名与使用的 GC 收集器是密切相关的，在此例中由于新生代我们使用了 Serial 收集器，此收集器新生代名为「Default New Generation」，所以显示的是 **[DefNew**，如果是 ParNew 收集器，新生代名称就会变为 **[ParNew**`，意为 「Parallel New Generation」,如果采用 「Parallel Scavenge」收集器，则配套的新生代名称为「PSYoungGen」,老年代与新生代一样，名称也是由收集器决定的 +5. 再往后 **6815K->280K(9216K)** 表示 「GC 前该内存区域已使用容量 -> GC 后该内存区域已使用容量（该内存区域总容量）」 +6. 0.0043690 secs 表示该块内存区域 GC 所占用的时间，单位是秒 +7. 6815K->6424K(19456K) 表示「GC 前 Java 堆已使用容量 -> GC 后 Java 堆已使用容量（java 堆总容量）」。 +8. **0.0044111 secs** 表示整个 GC 执行时间，注意和 6 中 **0.0043690 secs**的区别，后者专指**相关区域**所花的 GC 时间，而前者指的 GC 的整体堆内存变化所花时间（新生代与老生代的的内存整理），所以前者是肯定大于后者的！ +9. 最后一个 [Times: user=0.01 sys=0.00, real=0.02 secs] 这里的 user, sys 和 real 与Linux 的 time 命令所输出的时间一致，分别代表用户态消耗的 CPU 时间，内核态消耗的 CPU 时间，和操作从开始到结束所经过的墙钟时间，墙钟时间包括各种非运算的等待耗时，例如等待磁盘 I/O，等待线程阻塞，而 CPU 时间不包括这些耗时，但当系统有多 CPU 或者多核的话，多线程操作会叠加这些 CPU 时间，所以 user 或 sys 时间是可能超过 real 时间的。 +知道了 GC 日志怎么看，我们就可以根据 GC 日志有效定位问题了，如我们发现 Full GC 发生时间过长，则结合打印的 OOM 日志可能可以快速定位到问题 - -## 参考 +## 六、总结 -堆外内存的回收机制分析 https://www.jianshu.com/p/35cf0f348275 +本文简述了垃圾回收的原理与垃圾收集器的种类，相信大家对开头提的一些问题应该有了更深刻的认识，在生产环境中我们要根据**不同的场景**来选择垃圾收集器组合，如果是运行在桌面环境处于 Client 模式的，则用 Serial + Serial Old 收集器绰绰有余，如果需要响应时间快，用户体验好的，则用 ParNew + CMS 的搭配模式，即使是号称是「驾驭一切」的 G1，也需要根据吞吐量等要求适当调整相应的 JVM 参数，没有最牛的技术，只有最合适的使用场景，切记！ -java调用本地方法--jni简介 https://blog.csdn.net/w1992wishes/article/details/80283403 -咱们从头到尾说一次 Java 垃圾回收 https://mp.weixin.qq.com/s/pR7U1OTwsNSg5fRyWafucA -深入理解 Java 虚拟机 +**参考与来源** -Java Hotspot G1 GC的一些关键技术 https://tech.meituan.com/2016/09/23/g1.html \ No newline at end of file +- 主要来源“码海”公号 +- 《深入理解 Java 虚拟机》 +- 堆外内存的回收机制分析 https://www.jianshu.com/p/35cf0f348275 +- java调用本地方法--jni简介 https://blog.csdn.net/w1992wishes/article/details/80283403 +- 咱们从头到尾说一次 Java 垃圾回收 https://mp.weixin.qq.com/s/pR7U1OTwsNSg5fRyWafucA +- Java Hotspot G1 GC的一些关键技术 https://tech.meituan.com/2016/09/23/g1.html +- [Java中9种常见的CMS GC问题分析与解决](https://tech.meituan.com/2020/11/12/java-9-cms-gc.html) \ No newline at end of file diff --git a/docs/java/JVM/JVM-Issue.md b/docs/java/JVM/JVM-Issue.md deleted file mode 100644 index 098f1c6100..0000000000 --- a/docs/java/JVM/JVM-Issue.md +++ /dev/null @@ -1,85 +0,0 @@ -### JMM内存模型 - -- 为什么要有内存模型 -- 简单描述 JMM 和 JVM 两个概念 -- 你知道 Java 内存模型 JMM 吗？那你知道它的三大特性吗？ -- Java 是如何解决指令重排问题的？ -- 为什么要重排序，重排序在什么时候排 -- 如何约束重排序规则 -- happens-before -- 什么是顺序一致性 -- 原子性，可见性，有序性如何保证 -- volatile -- volatile 内存语义，什么时候用，用的时候需要考虑什么问题 -- 既然CPU有缓存一致性协议（MESI），为什么 JMM 还需要 volatile 关键字？ -- CAS 实现的原理，是阻塞还是非阻塞方式？什么时候用，使用时需要考虑的问题 -- 处理器和 Java 分别怎么保证原子操作 -- 保证了原子性就能保证可见性吗？ -- final 内存语义？什么时候用，使用时需要考虑的问题 -- synchronized 内存语义，什么时候用，和锁比较一下优缺点 -- synchronized 中涉及的锁升级流程 -- 锁的内存语义，举例说明，加锁失败时候的处理流程 -- 比较下 CAS 、volatile 、synchronized、Lock 区别 -- 原子操作类底层实现机制？自增操作是怎么保证原子性的？ - - - -### 类加载子系统 - -- 类加载机制？类加载过程 -- 什么是类加载器，类加载器有哪些？这些类加载器都加载哪些文件？ -- 多线程的情况下，类的加载为什么不会出现重复加载的情况？ -- 什么是双亲委派机制？它有啥优势？可以打破这种机制吗？ -- 自定义了一个String，那么会加载哪个String？ - - - -### 内存管理 - -- Java内存分配 -- 内存泄露和内存溢出 -- 永久代、元空间、方法区的关系 -- 如何覆盖 JDK 提供的组件，比如覆盖 ArrayList 的实现 -- new 一个对象的过程发生了什么（类加载、变量初始化、内存分配） -- 对象的死亡过程 -- JVM 可能会抛出哪些 OOM -- String内容存放在哪儿 - - - -### GC - -- 为什么要有GC -- 简述垃圾回收机制 -- 深拷贝和浅拷贝 -- System.gc() 和 Runtime.gc() 会做什么事情 -- 垃圾回收算法有哪些？优缺点比较 -- 熟知的垃圾回收器有哪些，简单描述每个应用场景 -- CMS 和 G1 的垃圾回收步骤是？ -- G1 相对于 CMS 的优缺点 -- 可达性分析算法中根节点有哪些 - - - -### 性能监控与调优 - -- 如何监控 GC -- 常见 OutOfMemoryError 有哪些 -- 常见的 JDK 诊断命令有哪些，应用场景？ -- CPU 较高，如何定位问题 -- 内存占用较高，如何定位大对象 -- 内存泄漏时，如何实时跟踪内存变化情况 -- 内存泄漏时，如何定位问题代码 -- 大型项目如何进行性能瓶颈调优？ - - - -### 虚拟机子系统 - -- 字节码是如何在 JVM 中进行流转的（栈帧） -- 方法调用的底层实现 -- 方法重写和重载的实现过程 -- invokedynamic 指令实现 -- 如何修改字节码 -- JIT 参数配置如何影响程序运行？ -- 虚拟机有哪些性能优化策略 \ No newline at end of file diff --git a/docs/java/JVM/JVM-Java.md b/docs/java/JVM/JVM-Java.md index 6d88ebb2d6..cf41bc2401 100644 --- a/docs/java/JVM/JVM-Java.md +++ b/docs/java/JVM/JVM-Java.md @@ -1,3 +1,5 @@ +# JVM 与 Java 体系结构 + 你是否也遇到过这些问题？ - 运行线上系统突然卡死，系统无法访问，甚至直接OOM @@ -18,8 +20,8 @@ Java开发都知道JVM是Java虚拟机，上学时还用过的VM也叫虚拟机 所谓虚拟机（Virtual Machine），就是一台虚拟的计算机。它是一款软件，用来执行一系列虚拟计算机指令。大体上，虚拟机可以分为**系统虚拟机**和**程序虚拟机**。 -- Visaual Box，VMware就属于系统虚拟机，它们完全是对物理计算机的仿真，提供了一个可运行完整操作系统的软件平台 -- 程序虚拟机的典型代表就是Java虚拟机，它专门为执行单个计算机程序而设计，在Java虚拟机中执行的指令我们称为Java字节码指令 +- Visaual Box，VMware 就属于系统虚拟机，它们完全是对物理计算机的仿真，提供了一个可运行完整操作系统的软件平台 +- 程序虚拟机的典型代表就是 Java 虚拟机，它专门为执行单个计算机程序而设计，在 Java 虚拟机中执行的指令我们称为 Java 字节码指令 @@ -27,7 +29,7 @@ Java开发都知道JVM是Java虚拟机，上学时还用过的VM也叫虚拟机 `JVM` 是 `Java Virtual Machine`（**Java虚拟机**)的缩写，`JVM`是一种用于计算设备的**规范**，它是一个**虚构**的计算机，是通过在实际的计算机上仿真模拟各种计算机功能来实现的。 -Java虚拟机是二进制字节码的运行环境，负责装载**字节码**到其内部，解释/编译为对应平台的机器指令执行。每一条Java指令，Java虚拟机规范中都有详细定义，如怎么取操作数，怎么处理操作数，处理结果放在哪里。 +Java 虚拟机是二进制字节码的运行环境，负责装载**字节码**到其内部，解释/编译为对应平台的机器指令执行。每一条 Java 指令，Java 虚拟机规范中都有详细定义，如怎么取操作数，怎么处理操作数，处理结果放在哪里。 ### 特点 @@ -37,25 +39,19 @@ Java虚拟机是二进制字节码的运行环境，负责装载**字节码**到 ### 字节码 -我们平时所说的java字节码，指的是用java语言编写的字节码，准确的说任何能在jvm平台上执行的字节码格式都是一样的，所以应该统称为**jvm字节码**。 +我们平时所说的 java 字节码，指的是用 java 语言编写的字节码，准确的说任何能在 jvm 平台上执行的字节码格式都是一样的，所以应该统称为 **jvm字节码**。 -不同的编译器可以编译出相同的字节码文件，字节码文件也可以在不同的jvm上运行。 +不同的编译器可以编译出相同的字节码文件，字节码文件也可以在不同的 jvm 上运行。 -Java虚拟机与Java语言没有必然的联系，它只与特定的二进制文件格式——Class文件格式关联，Class文件中包含了Java虚拟机指令集（或者称为字节码、Bytecodes）和符号集，还有一些其他辅助信息。 +Java 虚拟机与 Java 语言没有必然的联系，它只与特定的二进制文件格式——Class 文件格式关联，Class 文件中包含了 Java 虚拟机指令集（或者称为字节码、Bytecodes）和符号集，还有一些其他辅助信息。 -### Java代码执行过程 +### Java 代码执行过程 ![](https://tva1.sinaimg.cn/large/0082zybply1gbnkxsppg8j30jg0pk0x9.jpg) +## JVM 的位置 - - - - - -## JVM的位置 - -JVM是运行在操作系统之上的，它与硬件没有直接的交互。 +JVM 是运行在操作系统之上的，它与硬件没有直接的交互。 `JDK`(Java Development Kit) 是 `Java` 语言的软件开发工具包（`SDK`）。`JDK` 物理存在，是 ` Java Language`、`Tools`、`JRE` 和 `JVM` 的一个集合。 @@ -65,15 +61,15 @@ JVM是运行在操作系统之上的，它与硬件没有直接的交互。 -## JVM整体结构 +## JVM 整体结构 ![jvm-framework](https://tva1.sinaimg.cn/large/0082zybply1gbnqgrxfz4j30u00wp12d.jpg) -## JVM的架构模型 +## JVM 的架构模型 -Java编译器输入的指令流基本上是一种基于**栈的指令集架构**，另外一种指令集架构则是基于**寄存器的指令集架构**。 +Java 编译器输入的指令流基本上是一种基于**栈的指令集架构**，另外一种指令集架构则是基于**寄存器的指令集架构**。 两种架构之间的区别： @@ -83,17 +79,17 @@ Java编译器输入的指令流基本上是一种基于**栈的指令集架构** - 指令流中的指令大部分是零地址指令，其执行过程依赖于操作栈。指令集更小，编译器容易实现； - 不需要硬件支持，可移植性更好，更好实现跨平台 - 基于寄存器架构的特点 - - 典型的应用是X86的二进制指令集：比如传统的PC以及Android的Davlik虚拟机； + - 典型的应用是X86的二进制指令集：比如传统的 PC 以及 Android 的 Davlik 虚拟机； - 指令集架构则完全依赖硬件，可移植性差； - 性能优秀和执行更高效； - 花费更少的指令去完成一项操作； - 大部分情况下，基于寄存器架构的指令集往往都以一地址指令、二地址指令和三地址指令为主，而基于栈式架构的指令集却是以零地址指令为主 -由于跨平台性的设计，Java的指令都是根据栈来设计的。不同平台CPU架构不同，所以不能设计为基于寄存器的，优点是跨平台，指令集小，编译器容易实现，缺点是性能下降，实现同样的功能需要更多的指令。 +由于跨平台性的设计，Java 的指令都是根据栈来设计的。不同平台 CPU 架构不同，所以不能设计为基于寄存器的，优点是跨平台，指令集小，编译器容易实现，缺点是性能下降，实现同样的功能需要更多的指令。 ##### 分析基于栈式架构的JVM代码执行过程 -进入class文件所在目录，执行`javap -v xx.class`反解析（或者通过IDEA插件`Jclasslib`直接查看），可以看到当前类对应的code区（汇编指令）、本地变量表、异常表和代码行偏移量映射表、常量池等信息。 +进入 class 文件所在目录，执行`javap -v xx.class`反解析（或者通过IDEA插件`Jclasslib`直接查看），可以看到当前类对应的 code 区（汇编指令）、本地变量表、异常表和代码行偏移量映射表、常量池等信息。 ![jvm-javap](https://tva1.sinaimg.cn/large/0082zybply1gbnnern41cj31cd0u0qbc.jpg) @@ -179,18 +175,18 @@ Constant pool: -## JVM生命周期 +## JVM 生命周期 #### 虚拟机的启动 -Java虚拟机的启动是通过引导类加载器（Bootstrap Class Loader）创建一个初始类（initial class）来完成的，这个类是由虚拟机的具体实现指定的。 +Java 虚拟机的启动是通过引导类加载器（Bootstrap Class Loader）创建一个初始类（initial class）来完成的，这个类是由虚拟机的具体实现指定的。 #### 虚拟机的执行 -- 一个运行中的Java虚拟机有着一个清晰的任务：执行Java程序 +- 一个运行中的 Java 虚拟机有着一个清晰的任务：执行 Java 程序 - 程序开始执行时它才运行，程序结束时它就停止 -- 执行一个所谓的Java程序的时候，真正执行的是一个叫做Java虚拟机的进程 -- 你在同一台机器上运行三个程序，就会有三个运行中的Java虚拟机。 Java虚拟机总是开始于一个**main()**方法，这个方法必须是公有、返回void、只接受一个字符串数组。在程序执行时，你必须给Java虚拟机指明这个包含main()方法的类名。 +- 执行一个所谓的 Java 程序的时候，真正执行的是一个叫做 Java 虚拟机的进程 +- 你在同一台机器上运行三个程序，就会有三个运行中的 Java 虚拟机。 Java 虚拟机总是开始于一个 **main()** 方法，这个方法必须是公有、返回 void、只接受一个字符串数组。在程序执行时，你必须给 Java 虚拟机指明这个包含 main() 方法的类名。 #### 虚拟机的退出 @@ -198,37 +194,37 @@ Java虚拟机的启动是通过引导类加载器（Bootstrap Class Loader）创 - 程序正常执行结束 - 程序在执行过程中遇到了异常或错误而异常终止 -- 由于操作系统出现错误而导致Java虚拟机进程终止 -- 某线程调用Runtime类或System类的exit方法，或Runtime类的halt方法，并且Java安全管理器也允许这次exit或halt操作 -- 除此之外，JNI(Java Native Interface)规范描述了用`JNI Invocation API`来加载或卸载Java虚拟机时，Java虚拟机的退出情况 +- 由于操作系统出现错误而导致 Java 虚拟机进程终止 +- 某线程调用 Runtime 类或 System 类的 exit 方法，或 Runtime 类的 halt 方法，并且 Java 安全管理器也允许这次 exit 或 halt 操作 +- 除此之外，JNI(Java Native Interface)规范描述了用`JNI Invocation API`来加载或卸载 Java 虚拟机时，Java 虚拟机的退出情况 -## Java和JVM规范 +## Java 和 JVM 规范 [Java Language and Virtual Machine Specifications](https://docs.oracle.com/javase/specs/index.html) -## JVM发展历程 +## JVM 发展历程 JDK 版本升级不仅仅体现在语言和功能特性上，还包括了其编译和执行的 Java 虚拟机的升级。 -- 1990年，在Sun计算机公司中，由Patrick Naughton、MikeSheridan及James Gosling领导的小组Green Team，开发出的新的程序语言，命名为Oak，后期命名为Java -- 1995年，Sun正式发布Java和HotJava产品，Java首次公开亮相 +- 1990 年，在 Sun 计算机公司中，由 Patrick Naughton、MikeSheridan 及 James Gosling 领导的小组 Green Team，开发出的新的程序语言，命名为 Oak，后期命名为 Java +- 1995 年，Sun 正式发布 Java 和 HotJava 产品，Java 首次公开亮相 - 1996 年，JDK 1.0 发布时，提供了纯解释执行的 Java 虚拟机实现：Sun Classic VM。 - 1997 年，JDK 1.1 发布时，虚拟机没有做变更，依然使用 Sun Classic VM 作为默认的虚拟机 - 1998 年，JDK 1.2 发布时，提供了运行在 Solaris 平台的 Exact VM 虚拟机，但此时还是用 Sun Classic VM 作为默认的 Java 虚拟机，同时发布了JSP/Servlet、EJB规范，以及将Java分成J2EE、J2SE、J2ME - 2000 年，JDK1.3 发布，默认的 Java 虚拟机由 Sun Classic VM 改为 Sun HotSopt VM，而 Sun Classic VM 则作为备用虚拟机 - 2002 年，JDK 1.4 发布，Sun Classic VM 退出商用虚拟机舞台，直接使用 Sun HotSpot VM 作为默认虚拟机一直到现在 -- 2003年，Java平台的Scala正式发布，同年Groovy也加入了Java阵营 -- 2004年，JDK1.5发布，同时JDK1.5改名为JDK5.0 -- 2006年，JDK6发布，同年，Java开源并建立了OpenJDK。顺理成章，Hotspot虚拟机也成为了OpenJDK默认虚拟机 -- 2008年，Oracle收购BEA，得到了JRockit虚拟机 -- 2010年，Oracle收购了Sun，获得Java商标和HotSpot虚拟机 -- 2011年，JDK7发布，在JDK1.7u4中，正式启用了新的垃圾回收器G1 -- 2014年，JDK8发布，用元空间MetaSpace取代了PermGen -- 2017年，JDK9发布，将G1设置为默认GC，替代CMS +- 2003 年，Java 平台的 Scala 正式发布，同年 Groovy 也加入了 Java 阵营 +- 2004 年，JDK1.5 发布，同时 JDK1.5 改名为 JDK5.0 +- 2006 年，JDK6 发布，同年，Java 开源并建立了 OpenJDK。顺理成章，Hotspot 虚拟机也成为了 OpenJDK 默认虚拟机 +- 2008 年，Oracle 收购 BEA，得到了 JRockit 虚拟机 +- 2010 年，Oracle 收购了 Sun，获得 Java 商标和 HotSpot 虚拟机 +- 2011 年，JDK7 发布，在 JDK1.7u4 中，正式启用了新的垃圾回收器 G1 +- 2014 年，JDK8 发布，用元空间 MetaSpace 取代了 PermGen +- 2017 年，JDK9 发布，将 G1 设置为默认 GC，替代 CMS @@ -236,8 +232,8 @@ JDK 版本升级不仅仅体现在语言和功能特性上，还包括了其编 - 世界上第一款商用 Java 虚拟机。1996年随着Java1.0的发布而发布，JDK1.4时完全被淘汰； - 这款虚拟机内部只提供解释器； -- 如果使用JIT编译器，就需要进行外挂。但是一旦使用了JIT编译器，JIT就会接管虚拟机的执行系统，解释器就不再工作，解释器和编译器不能配合工作； -- 现在hotspot内置了此虚拟机 +- 如果使用 JIT 编译器，就需要进行外挂。但是一旦使用了 JIT 编译器，JIT 就会接管虚拟机的执行系统，解释器就不再工作，解释器和编译器不能配合工作； +- 现在 hotspot 内置了此虚拟机 ### Exact VM diff --git "a/docs/java/JVM/JVM\345\217\202\346\225\260\351\205\215\347\275\256.md" "b/docs/java/JVM/JVM\345\217\202\346\225\260\351\205\215\347\275\256.md" index 8656391a23..129fb854f4 100644 --- "a/docs/java/JVM/JVM\345\217\202\346\225\260\351\205\215\347\275\256.md" +++ "b/docs/java/JVM/JVM\345\217\202\346\225\260\351\205\215\347\275\256.md" @@ -1,12 +1,14 @@ -# 你说你做过 JVM 调优和参数配置，那你平时工作用过的配置参数有哪些？ +# JVM 参数配置 -![](https://images.pexels.com/photos/207924/pexels-photo-207924.jpeg?cs=srgb&dl=pexels-207924.jpg&fm=jpg) +> 面试官：你说你做过 JVM 调优和参数配置，那你平时工作用过的配置参数有哪些？ -## JVM参数类型 +![](https://tva1.sinaimg.cn/large/007S8ZIlly1gjlfuwkbr3j31hi0u0qv7.jpg) + +## JVM 参数类型 JVM 参数类型大致分为以下几类： -- **标准参数**（-），即在 JVM 的各个版本中基本不变的，相对比较稳定的参数，向后兼容 +- **标准参数**（-），即在 JVM 的各个版本中基本不变的，相对比较稳定的参数，向后兼容; - **非标准参数**（-X），变化比较小的参数，默认 JVM 实现这些参数的功能，但是并不保证所有 JVM 实现都满足，且不保证向后兼容； - **非Stable参数**（-XX），此类参数各个 JVM 实现会有所不同，将来可能会随时取消，需要慎重使用； @@ -14,7 +16,9 @@ JVM 参数类型大致分为以下几类： ### 标准参数 -![](https://imgkr.cn-bj.ufileos.com/798c9e1c-5aae-4798-a2eb-7bb7e454c9e1.png) +通过命令 `java` 即可查看 + +![](https://tva1.sinaimg.cn/large/0081Kckwly1gkamd9p5slj30u011mjzk.jpg) - `-version`：输出 java 的版本信息，比如 jdk 版本、vendor、model - `-help`：输出 java 标准参数列表及其描述 @@ -33,9 +37,9 @@ JVM 参数类型大致分为以下几类： ### X 参数 -非标准参数又称为扩展参数，其列表如下 +非标准参数又称为扩展参数，通过命令 `java -X` 查看，其列表如下 -![](https://imgkr.cn-bj.ufileos.com/52aa1112-79d8-495b-9dae-ff284aefc204.png) +![](https://tva1.sinaimg.cn/large/0081Kckwly1gkamht10qaj30u00yaqbu.jpg) - `-Xint`：设置 jvm 以解释模式运行，所有的字节码将被直接执行，而不会编译成本地码 - -Xmixed：混合模式，JVM自己来决定是否编译成本地代码，默认使用的就是混合模式 @@ -52,8 +56,41 @@ JVM 参数类型大致分为以下几类： +### xx 参数 + +- -XX:+PrintFlagsInitial -### xx参数 + - 主要查看初始默认值 + + - java -XX:+PrintFlagsInitial + + - java -XX:+PrintFlagsInitial -version + + - ![](https://tva1.sinaimg.cn/large/0081Kckwly1gkan7em5moj318o0peq9p.jpg) + + **等号前有冒号** := 说明 jvm 参数有人为修改过或者 JVM加载修改 + + false 说明是Boolean 类型 参数，数字说明是 KV 类型参数 + +- -XX:+PrintFlagsFinal + + ![](https://tva1.sinaimg.cn/large/0081Kckwly1gkanbdiozxj31700q2dmr.jpg) + + - 主要查看修改更新 + - java -XX:+PrintFlagsFinal + - java -XX:+PrintFlagsFinal -version + - 运行java命令的同时打印出参数 java -XX:+PrintFlagsFinal -XX:MetaspaceSize=512m Hello.java + +- -XX:+PrintCommondLineFlags + + - 打印命令行参数 + - java -XX:+PrintCommandLineFlags -version + - 可以方便的看到垃圾回收器 + - ![](https://tva1.sinaimg.cn/large/0081Kckwly1gkangitwxaj31py06ijtn.jpg) + + + +xx 参数主要分为 Boolean 类型参数和 KV 类型参数，我们一一介绍下 #### Boolean 类型 @@ -66,7 +103,7 @@ JVM 参数类型大致分为以下几类： - `-XX:+PrintGCDetails ` - `-XX:-PrintGCDetails ` - ![img](https://tva1.sinaimg.cn/large/00831rSTly1gdebpozfgwj315o0sgtcy.jpg) + ![](https://tva1.sinaimg.cn/large/00831rSTly1gdebpozfgwj315o0sgtcy.jpg) 添加如下参数后，重新查看，发现是 + 号了 @@ -186,13 +223,31 @@ System.out.println("max_memory(-xmx)="+maxMemory+"字节，" +(maxMemory/(double +## 再谈 JVM 参数设置 + +经过前面对 JVM 参数的介绍及相关例子的实验，相信大家对 JVM 的参数有了比较深刻的理解，接下来我们再谈谈如何设置 JVM 参数 +1. 首先 Oracle 官方推荐堆的初始化大小与堆可设置的最大值一般是相等的，即 Xms = Xmx，因为起始堆内存太小（Xms），会导致启动初期频繁 GC，起始堆内存较大（Xmx）有助于减少 GC 次数 +2. 调试的时候设置一些打印参数，如 `-XX:+PrintClassHistogram -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -Xloggc:log/gc.log`，这样可以从 gc.log 里看出一些端倪出来 +3. 系统停顿时间过长可能是 GC 的问题也可能是程序的问题，多用 jmap 和 jstack 查看，或者 `killall -3 Java`，然后查看 Java 控制台日志，能看出很多问题 + +4. 采用并发回收时，年轻代小一点，年老代要大，因为年老代用的是并发回收，即使时间长点也不会影响其他程序继续运行，网站不会停顿 + +5. 仔细了解自己的应用，如果用了缓存，那么年老代应该大一些，缓存的 HashMap 不应该无限制长，建议采用 LRU 算法的 Map 做缓存，LRUMap 的最大长度也要根据实际情况设定 + +要设置好各种 JVM 参数，还可以对 server 进行压测， 预估自己的业务量，设定好一些 JVM 参数进行压测看下这些设置好的 JVM 参数是否能满足要求 + + + + + +## 工作中常用配置 https://docs.oracle.com/javacomponents/jrockit-hotspot/migration-guide/cloptions.htm#JRHMG127 -参数不懂，推荐直接去看官网， +参数不懂，推荐直接去看官网 - -Xms @@ -230,7 +285,7 @@ https://docs.oracle.com/javacomponents/jrockit-hotspot/migration-guide/cloptions 定义一个大对象，撑爆堆内存， - ``` + ```java public static void main(String[] args) throws InterruptedException { System.out.println("==hello gc==="); @@ -240,15 +295,9 @@ https://docs.oracle.com/javacomponents/jrockit-hotspot/migration-guide/cloptions byte[] bytes = new byte[11 * 1024 * 1024]; - }![](https://tva1.sinaimg.cn/large/007S8ZIlly1gehkvas3vzj31a90u0n7t.jpg) + } ``` - - Full GC![img](https://tva1.sinaimg.cn/large/00831rSTly1gdefrc3lmbj31hy0gk7of.jpg) - - ![img](https://tva1.sinaimg.cn/large/00831rSTly1gdefr8tvx0j31h60m41eq.jpg) - - - GC![img](https://tva1.sinaimg.cn/large/00831rSTly1gdefrf0dfqj31fs0honjk.jpg) - - -XX:SurvivorRatio - 设置新生代中 eden 和S0/S1空间的比例 @@ -264,63 +313,24 @@ https://docs.oracle.com/javacomponents/jrockit-hotspot/migration-guide/cloptions - -XX:MaxTenuringThreshold - 设置垃圾的最大年龄（java8 固定设置最大 15） - - ![img](https://tva1.sinaimg.cn/large/00831rSTly1gdefr4xeq1j31g80lek6e.jpg) -![img](https://tva1.sinaimg.cn/large/00831rSTly1gdee0iss88j31eu0n6aqi.jpg) - -## 3. 你平时工作用过的 JVM 常用基本配置参数有哪些？ - -- -XX:+PrintFlagsInitial - - - 主要查看初始默认值 - - - java -XX:+PrintFlagsInitial - - - java -XX:+PrintFlagsInitial -version - - - ![img](https://tva1.sinaimg.cn/large/00831rSTly1gdee0ndg33j31ci0m6k5w.jpg) - - **等号前有冒号** := 说明 jvm 参数有人为修改过或者 JVM加载修改 - - false 说明是Boolean 类型 参数，数字说明是 KV 类型参数 - -- -XX:+PrintFlagsFinal - - - 主要查看修改更新 - - java -XX:+PrintFlagsFinal - - java -XX:+PrintFlagsFinal -version - - 运行java命令的同时打印出参数 java -XX:+PrintFlagsFinal -XX:MetaspaceSize=512m Hello.java - -- -XX:+PrintCommondLineFlags - - - 打印命令行参数 - - java -XX:+PrintCommondLineFlags -version - - 可以方便的看到垃圾回收器 - - ![img](https://tva1.sinaimg.cn/large/007S8ZIlly1gehf1e54soj31e006qjz6.jpg) - -### 盘点家底查看JVM默认值 - - +## 最后 参数不懂，推荐直接去看官网， -https://docs.oracle.com/javacomponents/jrockit-hotspot/migration-guide/cloptions.htm#JRHMG127 - - - https://docs.oracle.com/javase/8/docs/technotes/tools/unix/java.html#BGBCIEFC +https://docs.oracle.com/javacomponents/jrockit-hotspot/migration-guide/cloptions.htm#JRHMG127 + https://docs.oracle.com/javase/8/ Java SE Tools Reference for UNIX](https://docs.oracle.com/javase/8/docs/technotes/tools/unix/index.html) - - - +参考： https://www.cnblogs.com/duanxz/p/3482366.html diff --git "a/docs/java/JVM/JVM\346\200\247\350\203\275\347\233\221\346\216\247\345\222\214\346\225\205\351\232\234\345\244\204\347\220\206\345\267\245\345\205\267.md" "b/docs/java/JVM/JVM\346\200\247\350\203\275\347\233\221\346\216\247\345\222\214\346\225\205\351\232\234\345\244\204\347\220\206\345\267\245\345\205\267.md" index 8672b3cb64..12c41f6bd3 100644 --- "a/docs/java/JVM/JVM\346\200\247\350\203\275\347\233\221\346\216\247\345\222\214\346\225\205\351\232\234\345\244\204\347\220\206\345\267\245\345\205\267.md" +++ "b/docs/java/JVM/JVM\346\200\247\350\203\275\347\233\221\346\216\247\345\222\214\346\225\205\351\232\234\345\244\204\347\220\206\345\267\245\345\205\267.md" @@ -1,3 +1,5 @@ +# JVM 性能监控和故障处理工具 + 在线上处理问题的时候，知识，经验是关键基础，数据是依据，工具是知识处理数据的手段，这里说的数据包括但不限于运行日志、异常堆栈、GC日志、线程快照（threaddump/javacore 文件）、堆转存快照（heapdump/hprof 文件）等。 在本文中，工具主要是指 JDK 自带的工具，都位于 JDK 的 bin 目录下 @@ -87,7 +89,7 @@ jinfo（Configuration Info for Java）的作用是实时地查看和调整虚拟 我们可以通过 jinfo 实时的修改虚拟机的参数，但是不是任何命令都可以修改，可以修改的参数我们先来执行这个命令：`java -XX:+PrintFlagsFinal -version`，会列出当前机器支持的所有参数，那么用 jinfo 可以修改的参数是什么呢？只有最后一列显示 `manageable` 的这一列才能进行修改。 -仔细查看发现可修改的参数其实并不多，jvm 的运行内存一旦在运行时确定下来，那么就无法修改。但是无法一些错误信息没有记录，或者是处于关闭状态，还是可以修改的。 +仔细查看发现可修改的参数其实并不多，jvm 的运行内存一旦在运行时确定下来，那么就无法修改。但是一些错误信息没有记录，或者是处于关闭状态，还是可以修改的。 jinfo 命令格式： @@ -205,4 +207,129 @@ VisualVM 是一款免费的，集成了多个 JDK 命令行工具的可视化工 -恰当的使用虚拟机故障处理、分析工具可以提升我们分析数据、定位并解决问题的效率，但我们也要知道工具永远都是知识技能的一层包装，没有什么工具是"秘密武器"。 \ No newline at end of file +恰当的使用虚拟机故障处理、分析工具可以提升我们分析数据、定位并解决问题的效率，但我们也要知道工具永远都是知识技能的一层包装，没有什么工具是"秘密武器"。 + + + + + +## OOM 问题排查的一些常用工具 + +接下来我们来看下如何排查造成 OOM 的原因，内存泄漏是最常见的造成 OOM 的一种原因，所以接下来我们以来看看怎么使用工具来排查这种问题,使用到的工具主要有两大类 + +**1、使用 mat（Eclipse Memory Analyzer） 来分析 dump（堆转储快照） 文件** + +主要步骤如下 + +- 运行 Java 时添加 「-XX:+HeapDumpOnOutOfMemoryError」 参数来导出内存溢出时的堆信息，生成 hrof 文件, 添加 「-XX:HeapDumpPath」可以指定 hrof 文件的生成路径,如果不指定则 hrof 文件生成在与字节码文件相同的目录下 +- 使用 MAT（Eclipse Memory Analyzer）来分析 hrof 文件，查出内存泄漏的原因 + +接下来我们就来看看如何用以上的工具查看如下内存泄漏案例 + +``` +/** +* VM Args:-Xmx10m + */ +import java.util.ArrayList; +import java.util.List; +public class Main { + public static void main(String[] args) { + List list = new ArrayList(); + while (true) { + list.add("OutOfMemoryError soon"); + } + } +} +``` + +为了让以上程序快速产生 OOM, 我把堆大小设置成了 10M, 这样执行 「java -Xmx10m -XX:+HeapDumpOnOutOfMemoryError Main」后很快就发生了 OOM，此时我们就拿到了 hrof 文件，下载 MAT 工具，打开 hrof,进行分析，打开之后选择 「Leak Suspects Report」进行分析，可以看到发生 OOM 的线程的堆栈信息，明确定位到是哪一行造成的 + +![img](https://mmbiz.qpic.cn/mmbiz_png/OyweysCSeLVIoXNqicyWxibebAvTuJxk44QWer8TLAhny1tOZcS0gYVI8CElnkwoHxQxqxdVMic4hQBYompTKO6ag/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + + + +*如图示，可以看到 Main.java 文件的第 12 行导致了这次的 OOM* + +**2、使用 jvisualvm 来分析** + +用第一种方式必须等 OOM 后才能 dump 出 hprof 文件，但如果我们想在运行中观察堆的使用情况以便查出可能的内存泄漏代码就无能为力了，这时我们可以借助 **jvisualvm** 这款工具, jvisualvm 的功能强大，除了可以实时监控堆内存的使用情况，还可以跟踪垃圾回收，运行中 dump 中堆内存使用情况、cpu分析，线程分析等，是查找分析问题的利器，更骚的是它不光能分析本地的 Java 程序，还可以分析线上的 Java 程序运行情况, 本身这款工具也是随 JDK 发布的，是官方力推的一款运行监视，故障处理的神器。我们来看看如何用 jvisualvm 来分析上文所述的存在内存泄漏的如下代码 + +``` +import java.util.Map; +import java.util.HashMap; + +public class KeylessEntry { + static class Key { + Integer id; + Key(Integer id) { + this.id = id; + } + @Override + public int hashCode() { + return id.hashCode(); + } + } + + public static void main(String[] args) { + Map m = new HashMap(); + while(true) { + for(int i=0;i<10000;i++) { + if(!m.containsKey(new Key(i))) { + m.put(new Key(i), "Number:" + i); + } + } + } + } +} +``` + +打开 jvisualvm （终端输入 jvisualvm 执行即可），打开后，将堆大小设置为 500M，执行命令 **java Xms500m -Xmx500m KeylessEntry**，此时可以观察到左边出现了对应的应用 KeylessEntry,双击点击 open + +![img](https://tva1.sinaimg.cn/large/007S8ZIlly1gjlg8p0960j30ci09775m.jpg) + +打开之后可以看到展示了 CPU，堆内存使用，加载类及线程的情况 + +![img](https://mmbiz.qpic.cn/mmbiz_png/OyweysCSeLVIoXNqicyWxibebAvTuJxk4489ratobztWxsIIjyJbWFoWIg8ic6ZUERibC81pc35wuqW55kgar3uUfw/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +注意看堆（Heap）的使用情况，一直在上涨 + +![img](https://mmbiz.qpic.cn/mmbiz_png/OyweysCSeLVIoXNqicyWxibebAvTuJxk44hIU8l1X70hqXbhfHUPNKF2ERsjSwT54icia1vSIAqAWBibIQG3dCBX9Wg/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +此时我们再点击 「Heap Dump」 + +![img](https://mmbiz.qpic.cn/mmbiz_png/OyweysCSeLVIoXNqicyWxibebAvTuJxk44BVxVwoN99jHhhK97AAQnicXt7tlOERWH7V6RJqCDjWZXJ2TlfcklH2A/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +过一会儿即可看到内存中对象的使用情况 + +![img](https://mmbiz.qpic.cn/mmbiz_png/OyweysCSeLVIoXNqicyWxibebAvTuJxk448z3z6bu76NkrDehR3PxKd2tPNRCfhZ2ObMFRX7jDJa7AylGc0Z2LWg/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +可以看到相关的 TreeNode 有 291w 个，远超正常情况下的 10000 个！说明 HashMap 一直在增长，自此我们可以定位出问题代码所在！ + +**3、使用 jps + jmap 来获取 dump 文件** + +jps 可以列出正在运行的虚拟机进程，并显示执行虚拟机主类及这些进程的本地虚拟机唯一 ID，如图示 + +![img](https://mmbiz.qpic.cn/mmbiz_png/OyweysCSeLVIoXNqicyWxibebAvTuJxk449ic7ZBCWWca88nzQrkvWTfUw7kuhDXtKLjnllSDXHrTFPgAFUEKQMug/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +拿到进程的 pid 后，我们就可以用 jmap 来 dump 出堆转储文件了，执行命令如下 + +``` +jmap -dump:format=b,file=heapdump.phrof pid +``` + +拿到 dump 文件后我们就可以用 MAT 工具来分析了。 +但这个命令在生产上一定要慎用！因为JVM 会将整个 heap 的信息 dump 写入到一个文件，heap 比较大的话会导致这个过程比较耗时，并且执行过程中为了保证 dump 的信息是可靠的，会暂停应用！ + + + +## jstat 与可视化 APM 工具构建 + +jstat 是用于监视虚拟机各种运行状态信息的命令行工具，可以显示本地或者远程虚拟机进程中的类加载，内存，垃圾收集，JIT 编译等运行数据，jstat 支持定时查询相应的指标，如下 + +``` +jstat -gc 2764 250 22 +``` + +定时针对 2764 进程输出堆的垃圾收集情况的统计，可以显示 gc 的信息，查看gc的次数及时间,利用这些指标，把它们可视化，对分析问题会有很大的帮助,如图示，下图就是我司根据 jstat 做的一部分 gc 的可视化报表，能快速定位发生问题的问题点，如果大家需要做 APM 可视化工具，建议配合使用 jstat 来完成。 + +![img](https://mmbiz.qpic.cn/mmbiz_png/OyweysCSeLVIoXNqicyWxibebAvTuJxk44O4ltjSGfibEsaGv4OSDNv9sgcqicSwJydSHIHKVetyZk8JoPRzfoO98g/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) \ No newline at end of file diff --git a/docs/java/JVM/Java-Object.md b/docs/java/JVM/Java-Object.md index d7935f29b7..68366c0047 100644 --- a/docs/java/JVM/Java-Object.md +++ b/docs/java/JVM/Java-Object.md @@ -1,12 +1,23 @@ +# 你有认真了解过自己的 “Java 对象”吗 + > 对象在 JVM 中是怎么存储的 > > 对象头里有什么？ +> +> 文章收录在 GitHub [JavaKeeper](https://github.com/Jstarfish/JavaKeeper) ，N线互联网开发必备技能兵器谱，有你想要的。 作为一名 Javaer，生活中的我们可能暂时没有对象，但是工作中每天都会创建大量的 Java 对象，你有试着去了解下自己的“对象”吗？ -## 一、对象的实例化 +我们从四个方面重新认识下自己的“对象” + +1. 创建对象的 6 种方式 +2. 创建一个对象在 JVM 中都发生了什么 +3. 对象在 JVM 中的内存布局 +4. 对象的访问定位 -### 创建对象的方式 + + +## 一、创建对象的方式 - 使用 new 关键字 @@ -16,7 +27,7 @@ Person p = new Person(); ``` -- 使用 Class 类的 newInstance() +- 使用 Class 类的 newInstance()，只能调用空参的构造器，权限必须为 public ```java //获取类对象 @@ -24,7 +35,7 @@ Person p1 = (Person) aClass.newInstance(); ``` -- Constructor 的 newInstance(xxx) +- Constructor 的 newInstance(xxx)，对构造器没有要求 ```java Class aClass = Class.forName("priv.starfish.Person"); @@ -35,7 +46,7 @@ - clone() - java8 去掉了该方法 + 深拷贝，需要实现 Cloneable 接口并实现 clone()，不调用任何的构造器 ```java Person p3 = (Person) p.clone(); @@ -43,38 +54,69 @@ - 反序列化 - 每当我们序列化和反序列化对象时，JVM会为我们创建了一个独立的对象。在 deserialization 中，JVM 不使用任何构造函数来创建对象。 + 通过序列化和反序列化技术从文件或者网络中获取对象的二进制流。 + + 每当我们序列化和反序列化对象时，JVM 会为我们创建了一个独立的对象。在 deserialization 中，JVM 不使用任何构造函数来创建对象。（序列化的对象需要实现 Serializable） + + ```java + //准备一个文件用于存储该对象的信息 + File f = new File("person.obj"); + FileOutputStream fos = new FileOutputStream(f); + ObjectOutputStream oos = new ObjectOutputStream(fos); + //序列化对象，写入到磁盘中 + oos.writeObject(p); + //反序列化 + FileInputStream fis = new FileInputStream(f); + ObjectInputStream ois = new ObjectInputStream(fis); + //反序列化对象 + Person p4 = (Person) ois.readObject(); + ``` + +- 第三方库 Objenesls + + Java 已经支持通过 `Class.newInstance()` 动态实例化 Java 类，但是这需要 Java 类有个适当的构造器。很多时候一个 Java 类无法通过这种途径创建，例如：构造器需要参数、构造器有副作用、构造器会抛出异常。Objenesis 可以绕过上述限制 -- 第三方库Objenesls -### 创建对象的步骤 -这里讨论的仅仅是普通Java对象，不包含数组和Class对象 +## 二、创建对象的步骤 -#### 1. new指令 +这里讨论的仅仅是普通 Java 对象，不包含数组和 Class 对象（普通对象和数组对象的创建指令是不同的。创建类实例的指令：new，创建数组的指令：newarray，anewarray，multianewarray） + +#### 1. new 指令 虚拟机遇到一条 new 指令时，首先去检查这个指令的参数是否能在 Metaspace 的常量池中定位到一个类的符号引用，并且检查这个符号引用代表的类是否已被加载、解析和初始化过（即判断类元信息是否存在）。如果没有，那么须在双亲委派模式下，先执行相应的类加载过程。 #### 2. 分配内存 -接下来虚拟机将为新生代对象分配内存。对象所需的内存的大小在类加载完成后便可完全确定。如果实例成员变量是引用变量，仅分配引用变量空间即可，即 4 个字节大小。分配方式有“指针碰撞（Bump the Pointer）”和“空闲列表（Free List）”两种方式，具体由所采用的垃圾收集器是否带有压缩整理功能决定。 +接下来虚拟机将为新生代对象分配内存。**对象所需的内存的大小在类加载完成后便可完全确定**。如果实例成员变量是引用变量，仅分配引用变量空间即可，即 4 个字节大小。分配方式有“**指针碰撞**（Bump the Pointer）”和“**空闲列表**（Free List）”两种方式，具体由所采用的垃圾收集器是否带有压缩整理功能决定。 - 如果内存是规整的，就采用“指针碰撞”来为对象分配内存。意思是所有用过的内存在一边，空闲的内存在另一边，中间放着一个指针作为分界点的指示器，分配内存就仅仅是把指针指向空闲那边挪动一段与对象大小相等的距离罢了。如果垃圾收集器采用的是 Serial、ParNew 这种基于压缩算法的，就采用这种方法。（一般使用带整理功能的垃圾收集器，都采用指针碰撞） -- 如果内存是不规整的，虚拟机需要维护一个列表，这个列表会记录哪些内存是可用的，在为对象分配内存的时候从列表中找到一块足够大的空间划分给该对象实例，并更新列表内容，这种分配方式就是“空闲列表” + + ![](https://img-blog.csdnimg.cn/2020060220241513.png) + +- 如果内存是不规整的，虚拟机需要维护一个列表，这个列表会记录哪些内存是可用的，在为对象分配内存的时候从列表中找到一块足够大的空间划分给该对象实例，并更新列表内容，这种分配方式就是“空闲列表”。使用 CMS 这种基于 Mark-Sweep 算法的收集器时，通常采用空闲列表。 + + ![](https://img-blog.csdnimg.cn/20200602202424136.png) > 我们都知道堆内存是线程共享的，那在分配内存的时候就会存在并发安全问题，JVM 是如何解决的呢？ -采用 CAS 失败重试，区域加锁保证更新的原子性 +一般有两种解决方案： + +1. 对分配内存空间的动作做同步处理，采用 CAS 机制，配合失败重试的方式保证更新操作的原子性 -每个线程预先分配一快 TLAB，通过`-XX:/-UserTLAB`来设定（JDK8 默认就有了） +2. 每个线程在 Java 堆中预先分配一小块内存，然后再给对象分配内存的时候，直接在自己这块"私有"内存中分配，当这部分区域用完之后，再分配新的"私有"内存。这种方案称为 **TLAB**（Thread Local Allocation Buffer），这部分 Buffer 是从堆中划分出来的，但是是本地线程独享的。 + + **这里值得注意的是，我们说 TLAB 是线程独享的，只是在“分配”这个动作上是线程独占的，至于在读取、垃圾回收等动作上都是线程共享的。而且在使用上也没有什么区别**。另外，TLAB 仅作用于新生代的 Eden Space，对象被创建的时候首先放到这个区域，但是新生代分配不了内存的大对象会直接进入老年代。**因此在编写 Java 程序时，通常多个小的对象比大的对象分配起来更加高效**。 + + 虚拟机是否使用 TLAB 是可以选择的，可以通过设置 `-XX:+/-UseTLAB` 参数来指定，JDK8 默认开启。 #### 3. 初始化 -内存分配完成后，虚拟机需要将分配到的内存空间都初始化为零值（不包括对象头），这一步操作保证了对象的实例字段在 Java 代码中可以不赋初始值就直接使用，程序能访问到这些字段的数据类型所对应的零值。 +内存分配完成后，虚拟机需要将分配到的内存空间都初始化为零值（不包括对象头），这一步操作保证了对象的实例字段在 Java 代码中可以不赋初始值就直接使用，程序能访问到这些字段的数据类型所对应的零值。如：byte、short、long 转化为对象后初始值为 0，Boolean 初始值为 false。 #### 4. 对象的初始设置（设置对象的对象头） -接下来虚拟机要对对象进行必要的设置，例如这个对象是哪个类的实例、如何才能找到类的元数据信息、对象的哈希码、对象的GC分代年龄等信息。这些信息存放在对象的对象头（Object Header）之中。根据虚拟机当前的运行状态的不同，如对否启用偏向锁等，对象头会有不同的设置方式。 +接下来虚拟机要对对象进行必要的设置，例如这个对象是哪个类的实例、如何才能找到类的元数据信息、对象的哈希码、对象的 GC 分代年龄等信息。这些信息存放在对象的对象头（Object Header）之中。根据虚拟机当前的运行状态的不同，如是否启用偏向锁等，对象头会有不同的设置方式。 #### 5. \方法初始化 @@ -84,7 +126,7 @@ -## 对象的内存布局 +## 三、对象的内存布局 在 HotSpot 虚拟机中，对象在内存中存储的布局可以分为 3 块区域：对象头（Header）、实例数据（Instance Data）、对其填充（Padding）。 @@ -92,7 +134,7 @@ HotSpot 虚拟机的对象头包含两部分信息。 -- 第一部分用于存储对象自身的运行时数据，如哈希码(HashCode)、GC分代年龄、锁状态标志、线程持有的锁、偏向线程ID、偏向时间戳等。 +- 第一部分用于存储对象自身的运行时数据，如哈希码(HashCode)、GC 分代年龄、锁状态标志、线程持有的锁、偏向线程 ID、偏向时间戳等。 - 对象的另一部分类型指针，即对象指向它的类元数据的指针，虚拟机通过这个指针来确定这个对象是哪个类的实例（并不是所有的虚拟机实现都必须在对象数据上保留类型指针，也就是说，查找对象的元数据信息并不一定要经过对象本身）。 如果对象是一个 Java 数组，那在对象头中还必须有一块用于记录数组长度的数据。 @@ -113,13 +155,10 @@ HotSpot 虚拟机的对象头包含两部分信息。 对齐填充部分并不是必然存在的，也没有特别的含义，它仅仅起着占位符的作用。由于 HotSpot VM 的自动内存管理系统要求对象的起始地址必须是 8 字节的整数倍，也就是说，对象的大小必须是 8 字节的整数倍。而对象头部分正好是 8 字节的倍数（1倍或者2倍），因此，当对象实例数据部分没有对齐时，就需要通过对齐填充来补全。 - - 我们通过一个简单的例子加深下理解 ```java public class PersonObject { - public static void main(String[] args) { Person person = new Person(); } @@ -150,7 +189,7 @@ public class Department { -## 对象的访问定位 +## 四、对象的访问定位 我们创建对象的目的，肯定是为了使用它，那 JVM 是如何通过栈帧中的对象引用访问到其内存的对象实例呢？ @@ -160,13 +199,13 @@ public class Department { 如果使用句柄访问方式，Java堆中会划分出一块内存来作为句柄池，reference中存储的就是对象的句柄地址，而句柄中包含了对象实例数据和类型数据各自的具体地址信息。使用句柄方式最大的好处就是reference中存储的是稳定的句柄地址，在对象被移动（垃圾收集时移动对象是非常普遍的行为）时只会改变句柄中的实例数据指针，而reference本身不需要被修改。 - ![img](https://tva1.sinaimg.cn/large/007S8ZIlly1ggojeoey4yj30oz0hadgu.jpg) + ![](https://tva1.sinaimg.cn/large/007S8ZIlly1ggojeoey4yj30oz0hadgu.jpg) - 直接指针（Hotspot 使用该方式） 如果使用该方式，Java堆对象的布局就必须考虑如何放置访问类型数据的相关信息，reference中直接存储的就是对象地址。使用直接指针方式最大的好处就是**速度更快**，他**节省了一次指针定位的时间开销**。 - ![img](https://tva1.sinaimg.cn/large/007S8ZIlly1ggojeyqqagj30p10hit9l.jpg) + ![](https://tva1.sinaimg.cn/large/007S8ZIlly1ggojeyqqagj30p10hit9l.jpg) @@ -175,4 +214,6 @@ public class Department { **参考**: - https://zhuanlan.zhihu.com/p/44948944 -- https://blog.csdn.net/boy1397081650/article/details/89930710 \ No newline at end of file +- https://blog.csdn.net/boy1397081650/article/details/89930710 +- https://www.cnblogs.com/lusaisai/p/12748869.html +- https://juejin.im/post/5d4250def265da03ab422c79 diff --git a/docs/java/JVM/OOM.md b/docs/java/JVM/OOM.md index b788d473f1..1ecb25febd 100644 --- a/docs/java/JVM/OOM.md +++ b/docs/java/JVM/OOM.md @@ -1,6 +1,14 @@ -# 谈谈你对 OOM 的认识 +--- +title: 谈谈你对 OOM 的认识 +date: 2022-3-1 +tags: + - JVM +categories: JVM Java +--- -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gggcqh5gsdj324d0ol0y7.jpg) +> 点赞+收藏 就学会系列，文章收录在 GitHub [JavaKeeper](https://github.com/Jstarfish/JavaKeeper) ，N线互联网开发必备技能兵器谱，笔记自取 + +![oom](https://img.starfish.ink/jvm/oom.png) 在《Java虚拟机规范》的规定里，除了程序计数器外，虚拟机内存的其他几个运行时区域都有发生 OutOfMemoryError 异常的可能。 @@ -19,7 +27,7 @@ > 我们常说的 OOM 异常，其实是 Error -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gggbu55wwgj30sy0ku3z0.jpg) +![error-oom](https://img.starfish.ink/jvm/error-oom.png) @@ -108,17 +116,15 @@ Exception in thread "main" java.lang.OutOfMemoryError: Java heap space - 如果是业务峰值压力，可以考虑添加机器资源，或者做限流降级。 - 如果是内存泄漏，需要找到持有的对象，修改代码设计，比如关闭没有释放的连接 - - ![img](https://i03piccdn.sogoucdn.com/1b2bed506484c61d) > 面试官：说说内存泄露和内存溢出 加送个知识点，三连的终将成为大神~~ -## 内存泄露和内存溢出 +### 内存泄露和内存溢出 -内存溢出(out of memory)，是指程序在申请内存时，没有足够的内存空间供其使用，出现out of memory；比如申请了一个 Integer，但给它存了 Long 才能存下的数，那就是内存溢出。 +内存溢出(out of memory)，是指程序在申请内存时，没有足够的内存空间供其使用，出现 out of memory；比如申请了一个 Integer，但给它存了 Long 才能存下的数，那就是内存溢出。 内存泄露( memory leak)，是指程序在申请内存后，无法释放已申请的内存空间，一次内存泄露危害可以忽略，但内存泄露堆积后果很严重，无论多少内存，迟早会被占光。 @@ -128,7 +134,7 @@ Exception in thread "main" java.lang.OutOfMemoryError: Java heap space ## 三、GC overhead limit exceeded -JVM 内置了垃圾回收机制GC，所以作为 Javaer 的我们不需要手工编写代码来进行内存分配和释放，但是当 Java 进程花费 98% 以上的时间执行 GC，但只恢复了不到 2% 的内存，且该动作连续重复了 5 次，就会抛出 `java.lang.OutOfMemoryError:GC overhead limit exceeded` 错误（**俗称：垃圾回收上头**）。简单地说，就是应用程序已经基本耗尽了所有可用内存， GC 也无法回收。 +JVM 内置了垃圾回收机制 GC，所以作为 Javaer 的我们不需要手工编写代码来进行内存分配和释放，但是当 Java 进程花费 98% 以上的时间执行 GC，但只恢复了不到 2% 的内存，且该动作连续重复了 5 次，就会抛出 `java.lang.OutOfMemoryError:GC overhead limit exceeded` 错误（**俗称：垃圾回收上头**）。简单地说，就是应用程序已经基本耗尽了所有可用内存， GC 也无法回收。 假如不抛出 `GC overhead limit exceeded` 错误，那 GC 清理的那么一丢丢内存很快就会被再次填满，迫使 GC 再次执行，这样恶性循环，CPU 使用率 100%，而 GC 没什么效果。 @@ -179,13 +185,13 @@ Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceede 从输出结果可以看到，我们的限制 1000 条数据没有起作用，map 容量远超过了 1000，而且最后也出现了我们想要的错误，这是因为类 Key 只重写了 `hashCode()` 方法，却没有重写 `equals()` 方法，我们在使用 `containsKey()` 方法其实就出现了问题，于是就会一直往 HashMap 中添加 Key，直至 GC 都清理不掉。 -> 🧑🏻‍💻 面试官又来了：说一下HashMap原理以及为什么需要同时实现equals和hashcode +> 🧑🏻‍💻 面试官又来了：说一下 HashMap 原理以及为什么需要同时实现 equals 和 hashcode > 执行这个程序的最终错误，和 JVM 配置也会有关系，如果设置的堆内存特别小，会直接报 `Java heap space`。算是被这个错误截胡了，所以有时，在资源受限的情况下，无法准确预测程序会死于哪种具体的原因。 ### 3.2 解决方案 -- 添加 JVM 参数`-XX:-UseGCOverheadLimit` 不推荐这么干，没有真正解决问题，只是将异常推迟 +- 添加 JVM 参数 `-XX:-UseGCOverheadLimit` 不推荐这么干，没有真正解决问题，只是将异常推迟 - 检查项目中是否有大量的死循环或有使用大内存的代码，优化代码 - dump内存分析，检查是否存在内存泄露，如果没有，加大内存 @@ -199,9 +205,9 @@ Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceede ### 4.1 写个 bug -- ByteBuffer.allocate(capability) 是分配 JVM 堆内存，属于 GC 管辖范围，需要内存拷贝所以速度相对较慢； +- `ByteBuffer.allocate(capability)` 是分配 JVM 堆内存，属于 GC 管辖范围，需要内存拷贝所以速度相对较慢； -- ByteBuffer.allocateDirect(capability) 是分配 OS 本地内存，不属于 GC 管辖范围，由于不需要内存拷贝所以速度相对较快； +- `ByteBuffer.allocateDirect(capability)`是分配 OS 本地内存，不属于 GC 管辖范围，由于不需要内存拷贝所以速度相对较快； 如果不断分配本地内存，堆内存很少使用，那么 JVM 就不需要执行 GC，DirectByteBuffer 对象就不会被回收，这时虽然堆内存充足，但本地内存可能已经不够用了，就会出现 OOM，**本地直接内存溢出**。 @@ -264,7 +270,7 @@ java.lang.OutOfMemoryError: unable to create new native thread ### 5.2 原因分析 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1ggg8qlm7f0j30jg066gly.jpg) +![](https://img.starfish.ink/jvm/toomanythread.png) JVM 向 OS 请求创建 native 线程失败，就会抛出 `Unableto createnewnativethread`，常见的原因包括以下几类： @@ -358,11 +364,11 @@ JVM 在为数组分配内存前，会检查要分配的数据结构在系统中 ## 八、Out of swap space -启动 Java 应用程序会分配有限的内存。此限制是通过-Xmx和其他类似的启动参数指定的。 +启动 Java 应用程序会分配有限的内存。此限制是通过 `-Xmx` 和其他类似的启动参数指定的。 在 JVM 请求的总内存大于可用物理内存的情况下，操作系统开始将内容从内存换出到硬盘驱动器。 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gggbbrbgj2j30jg066t8k.jpg) +![](https://img.starfish.ink/jvm/swapspace.png) @@ -395,14 +401,14 @@ JVM 在为数组分配内存前，会检查要分配的数据结构在系统中 最后附上一张“涯海”大神的图 -![涯海](https://tva1.sinaimg.cn/large/007S8ZIlly1gggc8i8yk4j31qo0te49o.jpg) +![涯海](https://img.starfish.ink/jvm/oom-end.png) ## 参考与感谢 -《深入理解 Java 虚拟机 第 3 版》 +- 《深入理解 Java 虚拟机 第 3 版》 -https://plumbr.io/outofmemoryerror +- https://plumbr.io/outofmemoryerror -https://yq.aliyun.com/articles/711191 +- https://yq.aliyun.com/articles/711191 -https://github.com/StabilityMan/StabilityGuide/blob/master/docs/diagnosis/jvm/exception \ No newline at end of file +- https://github.com/StabilityMan/StabilityGuide/blob/master/docs/diagnosis/jvm/exception \ No newline at end of file diff --git a/docs/java/JVM/Online-Error-Check.md b/docs/java/JVM/Online-Error-Check.md new file mode 100644 index 0000000000..8d8a23b069 --- /dev/null +++ b/docs/java/JVM/Online-Error-Check.md @@ -0,0 +1,294 @@ +# JAVA线上故障排查全套路 + +> 来源：https://fredal.xin/java-error-check + +线上故障主要会包括 cpu、磁盘、内存以及网络问题，而大多数故障可能会包含不止一个层面的问题，所以进行排查时候尽量四个方面依次排查一遍。同时例如 jstack、jmap 等工具也是不囿于一个方面的问题的，基本上出问题就是 df、free、top 三连，然后依次 jstack、jmap 伺候，具体问题具体分析即可。 + +## CPU + +一般来讲我们首先会排查 cpu 方面的问题。cpu 异常往往还是比较好定位的。原因包括业务逻辑问题(死循环)、频繁 gc 以及上下文切换过多。而最常见的往往是业务逻辑(或者框架逻辑)导致的，可以使用 jstack 来分析对应的堆栈情况。 + +### 使用 jstack 分析 cpu 问题 + +我们先用 ps 命令找到对应进程的 pid(如果你有好几个目标进程，可以先用 top 看一下哪个占用比较高)。 +接着用`top -H -p pid`来找到 cpu 使用率比较高的一些线程 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083804.png) + +然后将占用最高的 pid 转换为16进制`printf '%x\n' pid`得到 nid + +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083806.png) + +接着直接在 jstack 中找到相应的堆栈信息`jstack pid |grep 'nid' -C5 –color` + +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-83807.png) + +可以看到我们已经找到了nid为0x42的堆栈信息，接着只要仔细分析一番即可。 + +当然更常见的是我们对整个 jstack 文件进行分析，通常我们会比较关注 **WAITING** 和 **TIMED_WAITING** 的部分，BLOCKED 就不用说了。我们可以使用命令 `cat jstack.log | grep "java.lang.Thread.State" | sort -nr | uniq -c`来对 jstack 的状态有一个整体的把握，如果 WAITING 之类的特别多，那么多半是有问题啦。 + +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083807.png) + + + +### 频繁 gc + +当然我们还是会使用 jstack 来分析问题，但有时候我们可以先确定下 gc 是不是太频繁，使用 `jstat -gc pid 1000` 命令来对 gc 分代变化情况进行观察，1000 表示采样间隔(ms)，S0C/S1C、S0U/S1U、EC/EU、OC/OU、MC/MU 分别代表两个 Survivor 区、Eden 区、老年代、元数据区的容量和使用量。YGC/YGT、FGC/FGCT、GCT则代表 YoungGc、FullGc 的耗时和次数以及总耗时。如果看到 gc 比较频繁，再针对 gc 方面做进一步分析。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083808.png) + + + +### 上下文切换 + +针对频繁上下文问题，我们可以使用 `vmstat` 命令来进行查看 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083809.png) +cs(context switch)一列则代表了上下文切换的次数。 +如果我们希望对特定的pid进行监控那么可以使用 `pidstat -w pid`命令，cswch和nvcswch表示自愿及非自愿切换。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-83810.png) + + + +## 磁盘 + +磁盘问题和 cpu 一样是属于比较基础的。首先是磁盘空间方面，我们直接使用 `df -hl` 来查看文件系统状态 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083810.png) + +更多时候，磁盘问题还是性能上的问题。我们可以通过 iostat： `iostat -d -k -x` 来进行分析 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083811.png) +最后一列`%util`可以看到每块磁盘写入的程度，而`rrqpm/s`以及`wrqm/s`分别表示读写速度，一般就能帮助定位到具体哪块磁盘出现问题了。 + +另外我们还需要知道是哪个进程在进行读写，一般来说开发自己心里有数，或者用iotop命令来进行定位文件读写的来源。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083812.png) +不过这边拿到的是tid，我们要转换成pid，可以通过readlink来找到pid`readlink -f /proc/*/task/tid/../..`。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-83813.png) +找到pid之后就可以看这个进程具体的读写情况`cat /proc/pid/io` +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083813.png) +我们还可以通过lsof命令来确定具体的文件读写情况`lsof -p pid` +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083814.png) + + + +## 内存 + +内存问题排查起来相对比 CPU 麻烦一些，场景也比较多。主要包括 OOM、GC 问题和堆外内存。一般来讲，我们会先用`free`命令先来检查一发内存的各种情况。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083815.png) + +### 堆内内存 + +内存问题大多还都是堆内内存问题。表象上主要分为 OOM 和 StackOverflow。 + +#### OOM + +JMV 中的内存不足，OOM 大致可以分为以下几种： + +**Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread** +这个意思是没有足够的内存空间给线程分配java栈，基本上还是线程池代码写的有问题，比如说忘记shutdown，所以说应该首先从代码层面来寻找问题，使用jstack或者jmap。如果一切都正常，JVM方面可以通过指定`Xss`来减少单个 thread stack 的大小。另外也可以在系统层面，可以通过修改`/etc/security/limits.conf`nofile和nproc来增大os对线程的限制 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-83816.png) + +**Exception in thread "main" java.lang.OutOfMemoryError: Java heap space** +这个意思是堆的内存占用已经达到-Xmx设置的最大值，应该是最常见的OOM错误了。解决思路仍然是先应该在代码中找，怀疑存在内存泄漏，通过jstack和jmap去定位问题。如果说一切都正常，才需要通过调整`Xmx`的值来扩大内存。 + +**Caused by: java.lang.OutOfMemoryError: Meta space** +这个意思是元数据区的内存占用已经达到`XX:MaxMetaspaceSize`设置的最大值，排查思路和上面的一致，参数方面可以通过`XX:MaxPermSize`来进行调整(这里就不说1.8以前的永久代了)。 + +#### Stack Overflow + +栈内存溢出，这个大家见到也比较多。 +**Exception in thread "main" java.lang.StackOverflowError** +表示线程栈需要的内存大于Xss值，同样也是先进行排查，参数方面通过`Xss`来调整，但调整的太大可能又会引起OOM。 + + + +#### 使用JMAP定位代码内存泄漏 + +上述关于OOM和StackOverflow的代码排查方面，我们一般使用JMAP`jmap -dump:format=b,file=filename pid`来导出dump文件 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083817.png) +通过mat(Eclipse Memory Analysis Tools)导入dump文件进行分析，内存泄漏问题一般我们直接选Leak Suspects即可，mat给出了内存泄漏的建议。另外也可以选择Top Consumers来查看最大对象报告。和线程相关的问题可以选择thread overview进行分析。除此之外就是选择Histogram类概览来自己慢慢分析，大家可以搜搜mat的相关教程。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083818.png) + +日常开发中，代码产生内存泄漏是比较常见的事，并且比较隐蔽，需要开发者更加关注细节。比如说每次请求都new对象，导致大量重复创建对象；进行文件流操作但未正确关闭；手动不当触发gc；ByteBuffer缓存分配不合理等都会造成代码OOM。 + +另一方面，我们可以在启动参数中指定`-XX:+HeapDumpOnOutOfMemoryError`来保存OOM时的dump文件。 + +#### gc问题和线程 + +gc问题除了影响cpu也会影响内存，排查思路也是一致的。一般先使用jstat来查看分代变化情况，比如youngGC或者fullGC次数是不是太多呀；EU、OU等指标增长是不是异常呀等。 +线程的话太多而且不被及时gc也会引发oom，大部分就是之前说的`unable to create new native thread`。除了jstack细细分析dump文件外，我们一般先会看下总体线程，通过`pstreee -p pid |wc -l`。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083819.png) +或者直接通过查看`/proc/pid/task`的数量即为线程数量。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083820.png) + +### 堆外内存 + +如果碰到堆外内存溢出，那可真是太不幸了。首先堆外内存溢出表现就是物理常驻内存增长快，报错的话视使用方式都不确定，如果由于使用Netty导致的，那错误日志里可能会出现`OutOfDirectMemoryError`错误，如果直接是DirectByteBuffer，那会报`OutOfMemoryError: Direct buffer memory`。 + +堆外内存溢出往往是和NIO的使用相关，一般我们先通过pmap来查看下进程占用的内存情况`pmap -x pid | sort -rn -k3 | head -30`，这段意思是查看对应pid倒序前30大的内存段。这边可以再一段时间后再跑一次命令看看内存增长情况，或者和正常机器比较可疑的内存段在哪里。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-83821.png) +我们如果确定有可疑的内存端，需要通过gdb来分析`gdb --batch --pid {pid} -ex "dump memory filename.dump {内存起始地址} {内存起始地址+内存块大小}"` +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083821.png) +获取dump文件后可用heaxdump进行查看`hexdump -C filename | less`，不过大多数看到的都是二进制乱码。 + +NMT是Java7U40引入的HotSpot新特性，配合jcmd命令我们就可以看到具体内存组成了。需要在启动参数中加入 `-XX:NativeMemoryTracking=summary` 或者 `-XX:NativeMemoryTracking=detail`，会有略微性能损耗。 + +一般对于堆外内存缓慢增长直到爆炸的情况来说，可以先设一个基线`jcmd pid VM.native_memory baseline`。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083822.png) +然后等放一段时间后再去看看内存增长的情况，通过`jcmd pid VM.native_memory detail.diff(summary.diff)`做一下summary或者detail级别的diff。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083823.png) +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-83824.png) +可以看到jcmd分析出来的内存十分详细，包括堆内、线程以及gc(所以上述其他内存异常其实都可以用nmt来分析)，这边堆外内存我们重点关注Internal的内存增长，如果增长十分明显的话那就是有问题了。 +detail级别的话还会有具体内存段的增长情况，如下图。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083824.png) + +此外在系统层面，我们还可以使用strace命令来监控内存分配 `strace -f -e "brk,mmap,munmap" -p pid` +这边内存分配信息主要包括了pid和内存地址。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083825.jpg) + +不过其实上面那些操作也很难定位到具体的问题点，关键还是要看错误日志栈，找到可疑的对象，搞清楚它的回收机制，然后去分析对应的对象。比如DirectByteBuffer分配内存的话，是需要full GC或者手动system.gc来进行回收的(所以最好不要使用`-XX:+DisableExplicitGC`)。那么其实我们可以跟踪一下DirectByteBuffer对象的内存情况，通过`jmap -histo:live pid`手动触发fullGC来看看堆外内存有没有被回收。如果被回收了，那么大概率是堆外内存本身分配的太小了，通过`-XX:MaxDirectMemorySize`进行调整。如果没有什么变化，那就要使用jmap去分析那些不能被gc的对象，以及和DirectByteBuffer之间的引用关系了。 + + + +## GC问题 + +堆内内存泄漏总是和GC异常相伴。不过GC问题不只是和内存问题相关，还有可能引起CPU负载、网络问题等系列并发症，只是相对来说和内存联系紧密些，所以我们在此单独总结一下GC相关问题。 + +我们在cpu章介绍了使用jstat来获取当前GC分代变化信息。而更多时候，我们是通过GC日志来排查问题的，在启动参数中加上`-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps`来开启GC日志。 +常见的Young GC、Full GC日志含义在此就不做赘述了。 + +针对gc日志，我们就能大致推断出youngGC与fullGC是否过于频繁或者耗时过长，从而对症下药。我们下面将对G1垃圾收集器来做分析，这边也建议大家使用G1`-XX:+UseG1GC`。 + +**youngGC过频繁** +youngGC频繁一般是短周期小对象较多，先考虑是不是Eden区/新生代设置的太小了，看能否通过调整-Xmn、-XX:SurvivorRatio等参数设置来解决问题。如果参数正常，但是young gc频率还是太高，就需要使用Jmap和MAT对dump文件进行进一步排查了。 + +**youngGC耗时过长** +耗时过长问题就要看GC日志里耗时耗在哪一块了。以G1日志为例，可以关注Root Scanning、Object Copy、Ref Proc等阶段。Ref Proc耗时长，就要注意引用相关的对象。Root Scanning耗时长，就要注意线程数、跨代引用。Object Copy则需要关注对象生存周期。而且耗时分析它需要横向比较，就是和其他项目或者正常时间段的耗时比较。比如说图中的Root Scanning和正常时间段比增长较多，那就是起的线程太多了。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083826.png) + +**触发fullGC** +G1中更多的还是mixedGC，但mixedGC可以和youngGC思路一样去排查。触发fullGC了一般都会有问题，G1会退化使用Serial收集器来完成垃圾的清理工作，暂停时长达到秒级别，可以说是半跪了。 +fullGC的原因可能包括以下这些，以及参数调整方面的一些思路： + +- 并发阶段失败：在并发标记阶段，MixGC之前老年代就被填满了，那么这时候G1就会放弃标记周期。这种情况，可能就需要增加堆大小，或者调整并发标记线程数`-XX:ConcGCThreads`。 +- 晋升失败：在GC的时候没有足够的内存供存活/晋升对象使用，所以触发了Full GC。这时候可以通过`-XX:G1ReservePercent`来增加预留内存百分比，减少`-XX:InitiatingHeapOccupancyPercent`来提前启动标记，`-XX:ConcGCThreads`来增加标记线程数也是可以的。 +- 大对象分配失败：大对象找不到合适的region空间进行分配，就会进行fullGC，这种情况下可以增大内存或者增大`-XX:G1HeapRegionSize`。 +- 程序主动执行System.gc()：不要随便写就对了。 + +另外，我们可以在启动参数中配置`-XX:HeapDumpPath=/xxx/dump.hprof`来dump fullGC相关的文件，并通过jinfo来进行gc前后的dump + +```java +jinfo -flag +HeapDumpBeforeFullGC pid +jinfo -flag +HeapDumpAfterFullGC pid +``` + +这样得到2份dump文件，对比后主要关注被gc掉的问题对象来定位问题。 + + + +## 网络 + +涉及到网络层面的问题一般都比较复杂，场景多，定位难，成为了大多数开发的噩梦，应该是最复杂的了。这里会举一些例子，并从tcp层、应用层以及工具的使用等方面进行阐述。 + +### 超时 + +超时错误大部分处在应用层面，所以这块着重理解概念。超时大体可以分为连接超时和读写超时，某些使用连接池的客户端框架还会存在获取连接超时和空闲连接清理超时。 + +- 读写超时。readTimeout/writeTimeout，有些框架叫做so_timeout或者socketTimeout，均指的是数据读写超时。注意这边的超时大部分是指逻辑上的超时。soa的超时指的也是读超时。读写超时一般都只针对客户端设置。 +- 连接超时。connectionTimeout，客户端通常指与服务端建立连接的最大时间。服务端这边connectionTimeout就有些五花八门了，jetty中表示空闲连接清理时间，tomcat则表示连接维持的最大时间。 +- 其他。包括连接获取超时connectionAcquireTimeout和空闲连接清理超时idleConnectionTimeout。多用于使用连接池或队列的客户端或服务端框架。 + +我们在设置各种超时时间中，需要确认的是尽量保持客户端的超时小于服务端的超时，以保证连接正常结束。 + +在实际开发中，我们关心最多的应该是接口的读写超时了。 + +如何设置合理的接口超时是一个问题。如果接口超时设置的过长，那么有可能会过多地占用服务端的tcp连接。而如果接口设置的过短，那么接口超时就会非常频繁。 + +服务端接口明明rt降低，但客户端仍然一直超时又是另一个问题。这个问题其实很简单，客户端到服务端的链路包括网络传输、排队以及服务处理等，每一个环节都可能是耗时的原因。 + +### TCP队列溢出 + +tcp队列溢出是个相对底层的错误，它可能会造成超时、rst等更表层的错误。因此错误也更隐蔽，所以我们单独说一说。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083827.jpg) + +如上图所示，这里有两个队列：syns queue(半连接队列）、accept queue（全连接队列）。三次握手，在server收到client的syn后，把消息放到syns queue，回复syn+ack给client，server收到client的ack，如果这时accept queue没满，那就从syns queue拿出暂存的信息放入accept queue中，否则按tcp_abort_on_overflow指示的执行。 + +tcp_abort_on_overflow 0表示如果三次握手第三步的时候accept queue满了那么server扔掉client发过来的ack。tcp_abort_on_overflow 1则表示第三步的时候如果全连接队列满了，server发送一个rst包给client，表示废掉这个握手过程和这个连接，意味着日志里可能会有很多`connection reset / connection reset by peer`。 + +那么在实际开发中，我们怎么能快速定位到tcp队列溢出呢？ + +**netstat命令，执行netstat -s | egrep "listen|LISTEN"** +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-83828.jpg) +如上图所示，overflowed表示全连接队列溢出的次数，sockets dropped表示半连接队列溢出的次数。 + +**ss命令，执行ss -lnt** +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083828.jpg) +上面看到Send-Q 表示第三列的listen端口上的全连接队列最大为5，第一列Recv-Q为全连接队列当前使用了多少。 + +接着我们看看怎么设置全连接、半连接队列大小吧： + +全连接队列的大小取决于min(backlog, somaxconn)。backlog是在socket创建的时候传入的，somaxconn是一个os级别的系统参数。而半连接队列的大小取决于max(64, /proc/sys/net/ipv4/tcp_max_syn_backlog)。 + +在日常开发中，我们往往使用servlet容器作为服务端，所以我们有时候也需要关注容器的连接队列大小。在tomcat中backlog叫做`acceptCount`，在jetty里面则是`acceptQueueSize`。 + +### RST异常 + +RST包表示连接重置，用于关闭一些无用的连接，通常表示异常关闭，区别于四次挥手。 + +在实际开发中，我们往往会看到`connection reset / connection reset by peer`错误，这种情况就是RST包导致的。 + +**端口不存在** + +如果像不存在的端口发出建立连接SYN请求，那么服务端发现自己并没有这个端口则会直接返回一个RST报文，用于中断连接。 + +**主动代替FIN终止连接** + +一般来说，正常的连接关闭都是需要通过FIN报文实现，然而我们也可以用RST报文来代替FIN，表示直接终止连接。实际开发中，可设置SO_LINGER数值来控制，这种往往是故意的，来跳过TIMED_WAIT，提供交互效率，不闲就慎用。 + +**客户端或服务端有一边发生了异常，该方向对端发送RST以告知关闭连接** + +我们上面讲的tcp队列溢出发送RST包其实也是属于这一种。这种往往是由于某些原因，一方无法再能正常处理请求连接了(比如程序崩了，队列满了)，从而告知另一方关闭连接。 + +**接收到的TCP报文不在已知的TCP连接内** + +比如，一方机器由于网络实在太差TCP报文失踪了，另一方关闭了该连接，然后过了许久收到了之前失踪的TCP报文，但由于对应的TCP连接已不存在，那么会直接发一个RST包以便开启新的连接。 + +**一方长期未收到另一方的确认报文，在一定时间或重传次数后发出RST报文** + +这种大多也和网络环境相关了，网络环境差可能会导致更多的RST报文。 + +之前说过RST报文多会导致程序报错，在一个已关闭的连接上读操作会报`connection reset`，而在一个已关闭的连接上写操作则会报`connection reset by peer`。通常我们可能还会看到`broken pipe`错误，这是管道层面的错误，表示对已关闭的管道进行读写，往往是在收到RST，报出`connection reset`错后继续读写数据报的错，这个在glibc源码注释中也有介绍。 + +我们在排查故障时候怎么确定有RST包的存在呢？当然是使用tcpdump命令进行抓包，并使用wireshark进行简单分析了。`tcpdump -i en0 tcp -w xxx.cap`，en0表示监听的网卡。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083829.jpg) + +接下来我们通过wireshark打开抓到的包，可能就能看到如下图所示，红色的就表示RST包了。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083830.jpg) + +### TIME_WAIT和CLOSE_WAIT + +TIME_WAIT和CLOSE_WAIT是啥意思相信大家都知道。 +在线上时，我们可以直接用命令`netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'`来查看time-wait和close_wait的数量 + +用ss命令会更快`ss -ant | awk '{++S[$1]} END {for(a in S) print a, S[a]}'` + +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083830.png) + +#### TIME_WAIT + +time_wait的存在一是为了丢失的数据包被后面连接复用，二是为了在2MSL的时间范围内正常关闭连接。它的存在其实会大大减少RST包的出现。 + +过多的time_wait在短连接频繁的场景比较容易出现。这种情况可以在服务端做一些内核参数调优: + +```java +#表示开启重用。允许将TIME-WAIT sockets重新用于新的TCP连接，默认为0，表示关闭 +net.ipv4.tcp_tw_reuse = 1 +#表示开启TCP连接中TIME-WAIT sockets的快速回收，默认为0，表示关闭 +net.ipv4.tcp_tw_recycle = 1 +``` + +当然我们不要忘记在NAT环境下因为时间戳错乱导致数据包被拒绝的坑了，另外的办法就是改小`tcp_max_tw_buckets`，超过这个数的time_wait都会被干掉，不过这也会导致报`time wait bucket table overflow`的错。 + +#### CLOSE_WAIT + +close_wait 往往都是因为应用程序写的有问题，没有在 ACK 后再次发起 FIN 报文。close_wait 出现的概率甚至比time_wait要更高，后果也更严重。往往是由于某个地方阻塞住了，没有正常关闭连接，从而渐渐地消耗完所有的线程。 + +想要定位这类问题，最好是通过jstack来分析线程堆栈来排查问题，具体可参考上述章节。这里仅举一个例子。 + +开发同学说应用上线后CLOSE_WAIT就一直增多，直到挂掉为止，jstack后找到比较可疑的堆栈是大部分线程都卡在了`countdownlatch.await`方法，找开发同学了解后得知使用了多线程但是确没有catch异常，修改后发现异常仅仅是最简单的升级sdk后常出现的`class not found`。 \ No newline at end of file diff --git a/docs/java/JVM/Reference.md b/docs/java/JVM/Reference.md index 8e623b6120..1ad3bc4fb1 100644 --- a/docs/java/JVM/Reference.md +++ b/docs/java/JVM/Reference.md @@ -6,18 +6,18 @@ ### 引用 -先说说引用，Java中的引用，类似 C 语言中的指针。初学 Java时，我们就知道 Java 数据类型分两大类，基本类型和引用类型。 +先说说引用，Java中的引用，类似 C 语言中的指针。初学 Java 时，我们就知道 Java 数据类型分两大类，基本类型和引用类型。 >基本类型：编程语言中内置的最小粒度的数据类型。它包括四大类八种类型： > ->- 4种整数类型：byte、short、int、long ->- 2种浮点数类型：float、double ->- 1种字符类型：char ->- 1种布尔类型：boolean +>- 4 种整数类型：byte、short、int、long +>- 2 种浮点数类型：float、double +>- 1 种字符类型：char +>- 1 种布尔类型：boolean > >引用类型：引用类型指向一个对象，不是原始值，指向对象的变量是引用变量。在 Java 里，除了基本类型，其他类型都属于引用类型，它主要包括：类、接口、数组、枚举、注解 -有了数据类型，JVM对程序数据的管理就规范化了，不同的数据类型，它的存储形式和位置是不一样的 +有了数据类型，JVM 对程序数据的管理就规范化了，不同的数据类型，它的存储形式和位置是不一样的 怎么跑偏了，回归正题，通过引用，可以对堆中的对象进行操作。引用《Java编程思想》中的一段话， @@ -25,11 +25,11 @@ 比如： -``` +```java Person person = new Person("张三"); ``` -这里的 person 就是指向Person 实例“张三”的引用，我们一般都是通过 person 来操作“张三”实例。 +这里的 person 就是指向 Person 实例“张三”的引用，我们一般都是通过 person 来操作“张三”实例。 @@ -52,7 +52,7 @@ Java 中引入四种引用的目的是让程序自己决定对象的生命周期 -### JDK 8中的 UML关系图 +### JDK8 中的 UML关系图 ![](https://tva1.sinaimg.cn/large/007S8ZIlly1genmesm4hej30lw08lmx1.jpg) @@ -66,7 +66,7 @@ FinalReference 类是包内可见，其他三种引用类型均为 public，可 当一个对象被强引用变量引用时，它处于可达状态，是不可能被垃圾回收器回收的，即使该对象永远不会被用到也不会被回收。 -当内存不足，JVM 开始垃圾回收，对于强引用的对象，**就算是出现了 OOM 也不会对该对象进行回收，打死都不收。**因此强引用有时也是造成 Java 内存泄露的原因之一。 +当内存不足，JVM 开始垃圾回收，对于强引用的对象，**就算是出现了 OOM 也不会对该对象进行回收，打死都不收**。因此强引用有时也是造成 Java 内存泄露的原因之一。 对于一个普通的对象，如果没有其他的引用关系，只要超过了引用的作用域或者显示地将相应（强）引用赋值为 null，一般认为就是可以被垃圾收集器回收。（具体回收时机还要要看垃圾收集策略）。 @@ -86,13 +86,13 @@ public class StrongRefenenceDemo { } ``` -demo 中尽管 o1已经被回收，但是 o2 强引用 o1，一直存在，所以不会被GC回收 +demo 中尽管 o1 已经被回收，但是 o2 强引用 o1，一直存在，所以不会被 GC 回收。 ### 软引用 -软引用是一种相对强引用弱化了一些的引用，需要用`java.lang.ref.SoftReference` 类来实现，可以让对象豁免一些垃圾收集。 +软引用是一种相对强引用弱化了一些的引用，需要用 `java.lang.ref.SoftReference` 类来实现，可以让对象豁免一些垃圾收集。 软引用用来描述一些还有用，但并非必需的对象。对于软引用关联着的对象，在系统将要发生内存溢出异常之前，将会把这些对象列进回收范围之中并进行第二次回收。如果这次回收还是没有足够的内存，才会抛出内存溢出异常。 @@ -289,7 +289,7 @@ static class ThreadLocalMap { 虚引用，顾名思义，就是形同虚设，与其他几种引用都不太一样，一个对象是否有虚引用的存在，完全不会对其生存时间构成影响，也无法通过虚引用来取得一个对象实例。 -虚引用需要`java.lang.ref.PhantomReference` 来实现。 +虚引用需要 `java.lang.ref.PhantomReference` 来实现。 如果一个对象仅持有虚引用，那么它就和没有任何引用一样，在任何时候都可能被垃圾回收器回收，它不能单独使用也不能通过它访问对象，虚引用必须和引用队列（RefenenceQueue）联合使用。 @@ -338,7 +338,7 @@ null ### 引用队列 -ReferenceQueue 是用来配合引用工作的，没有ReferenceQueue 一样可以运行。 +ReferenceQueue 是用来配合引用工作的，没有 ReferenceQueue 一样可以运行。 SoftReference、WeakReference、PhantomReference 都有一个可以传递 ReferenceQueue 的构造器。 @@ -360,16 +360,16 @@ SoftReference、WeakReference、PhantomReference 都有一个可以传递 Refere > Abstract base class for reference objects. This class defines the operations common to all reference objects. Because reference objects are implemented in close cooperation with the garbage collector, this class may not be subclassed directly. -JDK 官方文档是这么说的，`Reference`是所有引用对象的基类。这个类定义了所有引用对象的通用操作。因为引用对象是与垃圾收集器紧密协作而实现的，所以这个类可能不能直接子类化。 +JDK 官方文档是这么说的，`Reference` 是所有引用对象的基类。这个类定义了所有引用对象的通用操作。因为引用对象是与垃圾收集器紧密协作而实现的，所以这个类可能不能直接子类化。 -#### Reference 的4种状态 +#### Reference 的 4 种状态 -- Active：新创建的引用实例处于Active状态，但当GC检测到该实例引用的实际对象的可达性发生某些改变(实际对象处于 GC roots 不可达)后，它的状态将变化为`Pending`或者`Inactive`。如果 Reference 注册了ReferenceQueue，则会切换为`Pending`，并且Reference会加入`pending-Reference`链表中，如果没有注册ReferenceQueue，会切换为`Inactive`。 +- Active：新创建的引用实例处于 Active 状态，但当 GC 检测到该实例引用的实际对象的可达性发生某些改变(实际对象处于 GC roots 不可达)后，它的状态将变化为`Pending`或者`Inactive`。如果 Reference 注册了ReferenceQueue，则会切换为`Pending`，并且Reference会加入`pending-Reference`链表中，如果没有注册ReferenceQueue，会切换为`Inactive`。 - Pending：当引用实例被放置在pending-Reference 链表中时，它处于Pending状态。此时，该实例在等待一个叫Reference-handler的线程将此实例进行enqueue操作。如果某个引用实例没有注册在一个引用队列中，该实例将永远不会进入Pending状态 - Enqueued：在ReferenceQueue队列中的Reference的状态，如果Reference从队列中移除，会进入`Inactive`状态 - Inactive：一旦某个引用实例处于Inactive状态，它的状态将不再会发生改变，同时说明该引用实例所指向的实际对象一定会被GC所回收 -![img](http://imushan.com/img/image-20180820230137796.png) +![img](https://p1-jj.byteimg.com/tos-cn-i-t2oaga2asx/gold-user-assets/2020/5/22/1723a44a630c8f3c~tplv-t2oaga2asx-watermark.awebp) @@ -432,11 +432,11 @@ public boolean enqueue() { } ``` -#### ReferenceHandler线程 +#### ReferenceHandler 线程 -通过上文的讨论，我们知道一个Reference实例化后状态为Active，其引用的对象被回收后，垃圾回收器将其加入到`pending-Reference`链表，等待加入ReferenceQueue。 +通过上文的讨论，我们知道一个 Reference 实例化后状态为 Active，其引用的对象被回收后，垃圾回收器将其加入到`pending-Reference`链表，等待加入 ReferenceQueue。 -ReferenceHandler线程是由`Reference`静态代码块中建立并且运行的线程，它的运行方法中依赖了比较多的本地(native)方法，ReferenceHandler线程的主要功能就pending list中的引用实例添加到引用队列中，并将pending指向下一个引用实例。 +ReferenceHandler 线程是由 `Reference` 静态代码块中建立并且运行的线程，它的运行方法中依赖了比较多的本地(native)方法，ReferenceHandler 线程的主要功能就 pending list 中的引用实例添加到引用队列中，并将pending 指向下一个引用实例。 ```java // 控制垃圾回收器操作与Pending状态的Reference入队操作不冲突执行的全局锁 @@ -535,7 +535,7 @@ static { } ``` -由于ReferenceHandler线程是`Reference`的静态代码创建的，所以只要`Reference`这个父类被初始化，该线程就会创建和运行，由于它是守护线程，除非 JVM 进程终结，否则它会一直在后台运行(注意它的`run()`方法里面使用了死循环)。 +由于 ReferenceHandler 线程是 `Reference` 的静态代码创建的，所以只要 `Reference` 这个父类被初始化，该线程就会创建和运行，由于它是守护线程，除非 JVM 进程终结，否则它会一直在后台运行(注意它的`run()`方法里面使用了死循环)。 @@ -688,7 +688,3 @@ https://blog.csdn.net/Jesministrator/article/details/78786162 http://throwable.club/2019/02/16/java-reference/ 《深入理解java虚拟机》 - - - -![](H:\Technical-Learning\images\end.png) \ No newline at end of file diff --git a/docs/java/JVM/Runtime-Data-Areas.md b/docs/java/JVM/Runtime-Data-Areas.md index 469faf4b4c..22c93695df 100644 --- a/docs/java/JVM/Runtime-Data-Areas.md +++ b/docs/java/JVM/Runtime-Data-Areas.md @@ -1,10 +1,10 @@ -2万字长文包教包会 JVM 内存结构 保姆级学习笔记 +# 2 万字长文包教包会 JVM 内存结构 保姆级学习笔记 > JVM ≠ Japanese Video's Man > -> 写这篇的主要原因呢，就是为了能在简历上写个“熟悉JVM底层结构”，另一个原因就是能让读我文章的大家也写上这句话，真是个助人为乐的帅小伙。。。。嗯，不单单只是面向面试学习哈，更重要的是构建自己的JVM 知识体系，Javaer 们技术栈要有广度，但是 JVM 的掌握必须有深度 +> 写这篇的主要原因呢，就是为了能在简历上写个“熟悉JVM底层结构”，另一个原因就是能让读我文章的大家也写上这句话，真是个助人为乐的帅小伙。。。。嗯，不单单只是面向面试学习哈，更重要的是构建自己的 JVM 知识体系，Javaer 们技术栈要有广度，但是 JVM 的掌握必须有深度 > -> 点赞+收藏 就学会系列，文章收录在 GitHub [JavaKeeper](https://github.com/Jstarfish/JavaKeeper) ，N线互联网开发必备技能兵器谱 +> 点赞+收藏 就学会系列，文章收录在 GitHub [JavaKeeper](https://github.com/Jstarfish/JavaKeeper) ，N线互联网开发必备技能兵器谱，笔记自取 @@ -20,9 +20,7 @@ - 垃圾回收是否会涉及到虚拟机栈？ - 方法中定义的局部变量是否线程安全？ ------- -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gg9kuge8ovj32150tt7cd.jpg) # 运行时数据区 @@ -30,7 +28,7 @@ 下图是 JVM 整体架构，中间部分就是 Java 虚拟机定义的各种运行时数据区域。 -![jvm-framework](https://tva1.sinaimg.cn/large/0082zybply1gc6fz21n8kj30u00wpn5v.jpg) +![图片](https://img.starfish.ink/jvm/jvm-framework.png) Java 虚拟机定义了若干种程序运行期间会使用到的运行时数据区，其中有一些会随着虚拟机启动而创建，随着虚拟机退出而销毁。另外一些则是与线程一一对应的，这些与线程一一对应的数据区域会随着线程开始和结束而创建和销毁。 @@ -55,9 +53,9 @@ Java 虚拟机定义了若干种程序运行期间会使用到的运行时数据 PC 寄存器用来存储指向下一条指令的地址，即将要执行的指令代码。由执行引擎读取下一条指令。 -![jvm-pc-counter](https://tva1.sinaimg.cn/large/0082zybply1gc5kmznm1sj31m50u0wph.jpg) +![图片](https://mmbiz.qpic.cn/mmbiz_jpg/Z0fxkgAKKLMAVEVNlW7gDqZIFzSvVq29SIHYJWZ8MdtTBic8VyW9UFMkjDGzibzr6MRxUxHemmXKQibWAO5eVq93A/640?wx_fmt=jpeg&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) -（分析：进入class文件所在目录，执行`javap -v xx.class`反解析（或者通过IDEA插件`Jclasslib`直接查看，上图），可以看到当前类对应的Code区（汇编指令）、本地变量表、异常表和代码行偏移量映射表、常量池等信息。） +（分析：进入class文件所在目录，执行 `javap -v xx.class` 反解析（或者通过 IDEA 插件 `Jclasslib` 直接查看，上图），可以看到当前类对应的Code区（汇编指令）、本地变量表、异常表和代码行偏移量映射表、常量池等信息。） @@ -137,7 +135,7 @@ Java 虚拟机规范允许 **Java虚拟机栈的大小是动态的或者是固 IDEA 在 debug 时候，可以在 debug 窗口看到 Frames 中各种方法的压栈和出栈情况 -![](https://tva1.sinaimg.cn/large/0082zybply1gc9lezaxrbj319v0u0k4w.jpg) +![图片](https://mmbiz.qpic.cn/mmbiz_jpg/Z0fxkgAKKLMAVEVNlW7gDqZIFzSvVq29a7Bfd0qAUR98Brib5UNvia4yDCsvETZeO8lZoW3NljxkTic1ibDvj6PcJw/640?wx_fmt=jpeg&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) @@ -151,7 +149,7 @@ IDEA 在 debug 时候，可以在 debug 窗口看到 Frames 中各种方法的 - 方法返回地址（Return Address）：方法正常退出或异常退出的地址 - 一些附加信息 -![jvm-stack-frame](https://tva1.sinaimg.cn/large/0082zybply1gc8tjehg8bj318m0lbtbu.jpg) +![图片](https://mmbiz.qpic.cn/mmbiz_jpg/Z0fxkgAKKLMAVEVNlW7gDqZIFzSvVq29fMr12sicgr3HIgRFtFRY8IAcDvwP6orNRRIojrn3edcS3h2ibblgAgQg/640?wx_fmt=jpeg&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) 继续深抛栈帧中的五部分~~ @@ -160,28 +158,28 @@ IDEA 在 debug 时候，可以在 debug 窗口看到 Frames 中各种方法的 - 局部变量表也被称为局部变量数组或者本地变量表 - 是一组变量值存储空间，**主要用于存储方法参数和定义在方法体内的局部变量**，包括编译器可知的各种 Java 虚拟机**基本数据类型**（boolean、byte、char、short、int、float、long、double）、**对象引用**（reference类型，它并不等同于对象本身，可能是一个指向对象起始地址的引用指针，也可能是指向一个代表对象的句柄或其他与此相关的位置）和 **returnAddress** 类型（指向了一条字节码指令的地址，已被异常表取代） - 由于局部变量表是建立在线程的栈上，是线程的私有数据，因此**不存在数据安全问题** -- **局部变量表所需要的容量大小是编译期确定下来的**，并保存在方法的 Code 属性的`maximum local variables` 数据项中。在方法运行期间是不会改变局部变量表的大小的 +- **局部变量表所需要的容量大小是编译期确定下来的**，并保存在方法的 Code 属性的 `maximum local variables` 数据项中。在方法运行期间是不会改变局部变量表的大小的 - 方法嵌套调用的次数由栈的大小决定。一般来说，**栈越大，方法嵌套调用次数越多**。对一个函数而言，它的参数和局部变量越多，使得局部变量表膨胀，它的栈帧就越大，以满足方法调用所需传递的信息增大的需求。进而函数调用就会占用更多的栈空间，导致其嵌套调用次数就会减少。 - **局部变量表中的变量只在当前方法调用中有效**。在方法执行时，虚拟机通过使用局部变量表完成参数值到参数变量列表的传递过程。当方法调用结束后，随着方法栈帧的销毁，局部变量表也会随之销毁。 - 参数值的存放总是在局部变量数组的 index0 开始，到数组长度 -1 的索引结束 ##### 槽 Slot -- 局部变量表最基本的存储单元是Slot（变量槽） -- 在局部变量表中，32位以内的类型只占用一个Slot(包括returnAddress类型)，64位的类型（long和double）占用两个连续的 Slot - - byte、short、char 在存储前被转换为int，boolean也被转换为int，0 表示 false，非 0 表示 true +- 局部变量表最基本的存储单元是 Slot（变量槽） +- 在局部变量表中，32 位以内的类型只占用一个 Slot(包括returnAddress类型)，64 位的类型（long和double）占用两个连续的 Slot + - byte、short、char 在存储前被转换为 int，boolean 也被转换为 int，0 表示 false，非 0 表示 true - long 和 double 则占据两个 Slot - JVM 会为局部变量表中的每一个 Slot 都分配一个访问索引，通过这个索引即可成功访问到局部变量表中指定的局部变量值，索引值的范围从 0 开始到局部变量表最大的 Slot 数量 - 当一个实例方法被调用的时候，它的方法参数和方法体内部定义的局部变量将会**按照顺序被复制**到局部变量表中的每一个 Slot 上 -- **如果需要访问局部变量表中一个64bit的局部变量值时，只需要使用前一个索引即可**。（比如：访问 long 或double 类型变量，不允许采用任何方式单独访问其中的某一个 Slot） -- 如果当前帧是由构造方法或实例方法创建的，那么该对象引用 this 将会存放在 index 为 0 的 Slot 处，其余的参数按照参数表顺序继续排列（这里就引出一个问题：静态方法中为什么不可以引用 this，就是因为this 变量不存在于当前方法的局部变量表中） +- **如果需要访问局部变量表中一个 64bit 的局部变量值时，只需要使用前一个索引即可**。（比如：访问 long 或 double 类型变量，不允许采用任何方式单独访问其中的某一个 Slot） +- 如果当前帧是由构造方法或实例方法创建的，那么该对象引用 this 将会存放在 index 为 0 的 Slot 处，其余的参数按照参数表顺序继续排列（这里就引出一个问题：静态方法中为什么不可以引用 this，就是因为 this 变量不存在于当前方法的局部变量表中） - **栈帧中的局部变量表中的槽位是可以重用的**，如果一个局部变量过了其作用域，那么在其作用域之后申明的新的局部变量就很有可能会复用过期局部变量的槽位，从而**达到节省资源的目的**。（下图中，this、a、b、c 理论上应该有 4 个变量，c 复用了 b 的槽） -![](https://tva1.sinaimg.cn/large/0082zybply1gc9s12g5wlj31li0owdm9.jpg) +![图片](https://mmbiz.qpic.cn/mmbiz_jpg/Z0fxkgAKKLMAVEVNlW7gDqZIFzSvVq29Jqp1Sop3BNUXFa3FiagNgVYjC9maNFtyltyrQVMQYrRe5wo7CFRTMUg/640?wx_fmt=jpeg&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) - 在栈帧中，与性能调优关系最为密切的就是局部变量表。在方法执行时，虚拟机使用局部变量表完成方法的传递 -- 局部变量表中的变量也是重要的垃圾回收根节点，只要被局部变量表中直接或间接引用的对象都不会被回收 +- **局部变量表中的变量也是重要的垃圾回收根节点，只要被局部变量表中直接或间接引用的对象都不会被回收** @@ -202,17 +200,17 @@ IDEA 在 debug 时候，可以在 debug 窗口看到 Frames 中各种方法的 - 32bit 的类型占用一个栈单位深度 - 64bit 的类型占用两个栈单位深度 - 操作数栈并非采用访问索引的方式来进行数据访问的，而是只能通过标准的入栈和出栈操作来完成一次数据访问 -- **如果被调用的方法带有返回值的话，其返回值将会被压入当前栈帧的操作数栈中**，并更新PC寄存器中下一条需要执行的字节码指令 +- **如果被调用的方法带有返回值的话，其返回值将会被压入当前栈帧的操作数栈中**，并更新 PC 寄存器中下一条需要执行的字节码指令 - 操作数栈中元素的数据类型必须与字节码指令的序列严格匹配，这由编译器在编译期间进行验证，同时在类加载过程中的类检验阶段的数据流分析阶段要再次验证 -- 另外，我们说**Java虚拟机的解释引擎是基于栈的执行引擎**，其中的栈指的就是操作数栈 +- 另外，我们说 **Java虚拟机的解释引擎是基于栈的执行引擎**，其中的栈指的就是操作数栈 ##### 栈顶缓存（Top-of-stack-Cashing） -HotSpot 的执行引擎采用的并非是基于寄存器的架构，但这并不代表 HotSpot VM 的实现并没有间接利用到寄存器资源。寄存器是物理 CPU 中的组成部分之一，它同时也是 CPU 中非常重要的高速存储资源。一般来说，寄存器的读/写速度非常迅速，甚至可以比内存的读/写速度快上几十倍不止，不过寄存器资源却非常有限，不同平台下的CPU 寄存器数量是不同和不规律的。寄存器主要用于缓存本地机器指令、数值和下一条需要被执行的指令地址等数据。 +HotSpot 的执行引擎采用的并非是基于寄存器的架构，但这并不代表 HotSpot VM 的实现并没有间接利用到寄存器资源。寄存器是物理 CPU 中的组成部分之一，它同时也是 CPU 中非常重要的高速存储资源。一般来说，寄存器的读/写速度非常迅速，甚至可以比内存的读/写速度快上几十倍不止，不过寄存器资源却非常有限，不同平台下的 CPU 寄存器数量是不同和不规律的。寄存器主要用于缓存本地机器指令、数值和下一条需要被执行的指令地址等数据。 -基于栈式架构的虚拟机所使用的零地址指令更加紧凑，但完成一项操作的时候必然需要使用更多的入栈和出栈指令，这同时也就意味着将需要更多的指令分派（instruction dispatch）次数和内存读/写次数。由于操作数是存储在内存中的，因此频繁的执行内存读/写操作必然会影响执行速度。为了解决这个问题，HotSpot JVM设计者们提出了栈顶缓存技术，**将栈顶元素全部缓存在物理 CPU 的寄存器中，以此降低对内存的读/写次数，提升执行引擎的执行效率** +基于栈式架构的虚拟机所使用的零地址指令更加紧凑，但完成一项操作的时候必然需要使用更多的入栈和出栈指令，这同时也就意味着将需要更多的指令分派（instruction dispatch）次数和内存读/写次数。由于操作数是存储在内存中的，因此频繁的执行内存读/写操作必然会影响执行速度。为了解决这个问题，HotSpot JVM 设计者们提出了栈顶缓存技术，**将栈顶元素全部缓存在物理 CPU 的寄存器中，以此降低对内存的读/写次数，提升执行引擎的执行效率** @@ -221,7 +219,7 @@ HotSpot 的执行引擎采用的并非是基于寄存器的架构，但这并不 - **每一个栈帧内部都包含一个指向运行时常量池中该栈帧所属方法的引用**。包含这个引用的目的就是为了支持当前方法的代码能够实现动态链接(Dynamic Linking)。 - 在 Java 源文件被编译到字节码文件中时，所有的变量和方法引用都作为**符号引用**（Symbolic Reference）保存在 Class 文件的常量池中。比如：描述一个方法调用了另外的其他方法时，就是通过常量池中指向方法的符号引用来表示的，那么**动态链接的作用就是为了将这些符号引用转换为调用方法的直接引用** -![jvm-dynamic-linking](https://tva1.sinaimg.cn/large/0082zybply1gca4k4gndgj31d20o2td0.jpg) +![图片](https://mmbiz.qpic.cn/mmbiz_jpg/Z0fxkgAKKLMAVEVNlW7gDqZIFzSvVq29g9IVw1nI30nh6BeSgA7ldbONVZicqAyetWkndD4cDwrWeRlCUwDMcKQ/640?wx_fmt=jpeg&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) ##### JVM 是如何执行方法调用的 @@ -231,8 +229,8 @@ HotSpot 的执行引擎采用的并非是基于寄存器的架构，但这并不 在 JVM 中，将符号引用转换为调用方法的直接引用与方法的绑定机制有关 -- 静态链接：当一个字节码文件被装载进 JVM 内部时，如果被调用的**目标方法在编译期可知**，且运行期保持不变时。这种情况下将调用方法的符号引用转换为直接引用的过程称之为静态链接 -- 动态链接：如果被调用的方法在编译期无法被确定下来，也就是说，只能在程序运行期将调用方法的符号引用转换为直接引用，由于这种引用转换过程具备动态性，因此也就被称之为动态链接 +- **静态链接**：当一个字节码文件被装载进 JVM 内部时，如果被调用的**目标方法在编译期可知**，且运行期保持不变时。这种情况下将调用方法的符号引用转换为直接引用的过程称之为静态链接 +- **动态链接**：如果被调用的方法在编译期无法被确定下来，也就是说，只能在程序运行期将调用方法的符号引用转换为直接引用，由于这种引用转换过程具备动态性，因此也就被称之为动态链接 对应的方法的绑定机制为：早期绑定（Early Binding）和晚期绑定（Late Binding）。**绑定是一个字段、方法或者类在符号引用被替换为直接引用的过程，这仅仅发生一次**。 @@ -241,7 +239,7 @@ HotSpot 的执行引擎采用的并非是基于寄存器的架构，但这并不 ##### 虚方法和非虚方法 -- 如果方法在编译器就确定了具体的调用版本，这个版本在运行时是不可变的。这样的方法称为非虚方法，比如静态方法、私有方法、final方法、实例构造器、父类方法都是非虚方法 +- 如果方法在编译器就确定了具体的调用版本，这个版本在运行时是不可变的。这样的方法称为非虚方法，比如静态方法、私有方法、final 方法、实例构造器、父类方法都是非虚方法 - 其他方法称为虚方法 ##### 虚方法表 @@ -270,7 +268,7 @@ HotSpot 的执行引擎采用的并非是基于寄存器的架构，但这并不 一个方法的正常调用完成之后究竟需要使用哪一个返回指令还需要根据方法返回值的实际数据类型而定 - 在字节码指令中，返回指令包含 ireturn(当返回值是boolean、byte、char、short和int类型时使用)、lreturn、freturn、dreturn以及areturn，另外还有一个 return 指令供声明为 void 的方法、实例初始化方法、类和接口的初始化方法使用。 + 在字节码指令中，返回指令包含 ireturn(当返回值是 boolean、byte、char、short 和 int 类型时使用)、lreturn、freturn、dreturn 以及 areturn，另外还有一个 return 指令供声明为 void 的方法、实例初始化方法、类和接口的初始化方法使用。 2. 在方法执行的过程中遇到了异常，并且这个异常没有在方法内进行处理，也就是只要在本方法的异常表中没有搜索到匹配的异常处理器，就会导致方法退出。简称**异常完成出口** @@ -311,13 +309,13 @@ Java 使用起来非常方便，然而有些层次的任务用 Java 实现起来 - 允许线程固定或者可动态扩展的内存大小 - 如果线程请求分配的栈容量超过本地方法栈允许的最大容量，Java 虚拟机将会抛出一个 `StackOverflowError` 异常 - 如果本地方法栈可以动态扩展，并且在尝试扩展的时候无法申请到足够的内存，或者在创建新的线程时没有足够的内存去创建对应的本地方法栈，那么 Java虚拟机将会抛出一个`OutofMemoryError`异常 -- 本地方法是使用C语言实现的 -- 它的具体做法是 `Mative Method Stack` 中登记native方法，在 `Execution Engine` 执行时加载本地方法库当某个线程调用一个本地方法时，它就进入了一个全新的并且不再受虚拟机限制的世界。它和虚拟机拥有同样的权限。 +- 本地方法是使用 C 语言实现的 +- 它的具体做法是 `Mative Method Stack` 中登记 native 方法，在 `Execution Engine` 执行时加载本地方法库当某个线程调用一个本地方法时，它就进入了一个全新的并且不再受虚拟机限制的世界。它和虚拟机拥有同样的权限。 - 本地方法可以通过本地方法接口来访问虚拟机内部的运行时数据区，它甚至可以直接使用本地处理器中的寄存器，直接从本地内存的堆中分配任意数量的内存 - 并不是所有 JVM 都支持本地方法。因为 Java 虚拟机规范并没有明确要求本地方法栈的使用语言、具体实现方式、数据结构等。如果 JVM 产品不打算支持 native 方法，也可以无需实现本地方法栈 -- 在 Hotspot JVM 中，直接将本地方栈和虚拟机栈合二为一 +- 在 Hotspot JVM 中，直接将本地方栈和虚拟机栈合二为一 ------ @@ -333,13 +331,13 @@ Java 使用起来非常方便，然而有些层次的任务用 Java 实现起来 对于大多数应用，Java 堆是 Java 虚拟机管理的内存中最大的一块，被所有线程共享。此内存区域的唯一目的就是存放对象实例，几乎所有的对象实例以及数据都在这里分配内存。 -为了进行高效的垃圾回收，虚拟机把堆内存**逻辑上**划分成三块区域（分代的唯一理由就是优化 GC 性能）： +为了进行高效的垃圾回收，虚拟机把堆内存**逻辑上**划分成三块区域（分代的唯一理由就是优化 GC 性能）： - 新生带（年轻代）：新对象和没达到一定年龄的对象都在新生代 - 老年代（养老区）：被长时间使用的对象，老年代的内存空间应该要比年轻代更大 -- 元空间（JDK1.8之前叫永久代）：像一些方法中的操作临时对象等，JDK1.8之前是占用JVM内存，JDK1.8之后直接使用物理内存 +- 元空间（JDK1.8 之前叫永久代）：像一些方法中的操作临时对象等，JDK1.8 之前是占用 JVM 内存，JDK1.8 之后直接使用物理内存 -![JDK7](https://tva1.sinaimg.cn/large/00831rSTly1gdbr7ek6pfj30ci0560t4.jpg) +![图片](https://mmbiz.qpic.cn/mmbiz_jpg/Z0fxkgAKKLMAVEVNlW7gDqZIFzSvVq29bhZk1leXwNkO3Bo2iavTichwqE0W7ItpmlX5Ebr6BSputueQXutDhGOA/640?wx_fmt=jpeg&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) Java 虚拟机规范规定，Java 堆可以是处于物理上不连续的内存空间中，只要逻辑上是连续的即可，像磁盘空间一样。实现时，既可以是固定大小，也可以是可扩展的，主流虚拟机都是可扩展的（通过 `-Xmx` 和 `-Xms` 控制），如果堆中没有完成实例分配，并且堆无法再扩展时，就会抛出 `OutOfMemoryError` 异常。 @@ -347,7 +345,7 @@ Java 虚拟机规范规定，Java 堆可以是处于物理上不连续的内存 #### 年轻代 (Young Generation) -年轻代是所有新对象创建的地方。当填充年轻代时，执行垃圾收集。这种垃圾收集称为**Minor GC**。年轻一代被分为三个部分——伊甸园（**Eden Memory**）和两个幸存区（**Survivor Memory**，被称为from/to或s0/s1），默认比例是`8:1:1` +年轻代是所有新对象创建的地方。当填充年轻代时，执行垃圾收集。这种垃圾收集称为 **Minor GC**。年轻一代被分为三个部分——伊甸园（**Eden Memory**）和两个幸存区（**Survivor Memory**，被称为from/to或s0/s1），默认比例是`8:1:1` - 大多数新创建的对象都位于 Eden 内存空间中 - 当 Eden 空间被对象填充时，执行**Minor GC**，并将所有幸存者对象移动到一个幸存者空间中 @@ -356,11 +354,11 @@ Java 虚拟机规范规定，Java 堆可以是处于物理上不连续的内存 #### 老年代(Old Generation) -旧的一代内存包含那些经过许多轮小型 GC 后仍然存活的对象。通常，垃圾收集是在老年代内存满时执行的。老年代垃圾收集称为主GC，通常需要更长的时间。 +旧的一代内存包含那些经过许多轮小型 GC 后仍然存活的对象。通常，垃圾收集是在老年代内存满时执行的。老年代垃圾收集称为 主GC（Major GC），通常需要更长的时间。 -大对象直接进入老年代（大对象是指需要大量连续内存空间的对象）。这样做的目的是避免在Eden区和两个Survivor区之间发生大量的内存拷贝 +大对象直接进入老年代（大对象是指需要大量连续内存空间的对象）。这样做的目的是避免在 Eden 区和两个Survivor 区之间发生大量的内存拷贝 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gg06065oa9j31kw0u0q69.jpg) +![图片](https://mmbiz.qpic.cn/mmbiz_jpg/Z0fxkgAKKLMAVEVNlW7gDqZIFzSvVq29MhFR2A88icyjajrvRZewIe0IXiaHsSzQ7mfzm5libWG37dFyY0eoq72hg/640?wx_fmt=jpeg&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) #### 元空间 @@ -411,9 +409,9 @@ public static void main(String[] args) { 2. 默认情况下新生代和老年代的比例是 1:2，可以通过 `–XX:NewRatio` 来配置 - - 新生代中的 **Eden**:**From Survivor**:**To Survivor** 的比例是 **8:1:1**，可以通过`-XX:SurvivorRatio`来配置 + - 新生代中的 **Eden**:**From Survivor**:**To Survivor** 的比例是 **8:1:1**，可以通过 `-XX:SurvivorRatio` 来配置 -3. 若在JDK 7中开启了 `-XX:+UseAdaptiveSizePolicy`，JVM 会动态调整 JVM 堆中各个区域的大小以及进入老年代的年龄 +3. 若在 JDK 7 中开启了 `-XX:+UseAdaptiveSizePolicy`，JVM 会动态调整 JVM 堆中各个区域的大小以及进入老年代的年龄 此时 `–XX:NewRatio` 和 `-XX:SurvivorRatio` 将会失效，而 JDK 8 是默认开启`-XX:+UseAdaptiveSizePolicy` @@ -445,8 +443,8 @@ $ jmap -heap 进程号 ### 4.3 对象在堆中的生命周期 1. 在 JVM 内存模型的堆中，堆被划分为新生代和老年代 - - 新生代又被进一步划分为**Eden区**和**Survivor区**，Survivor区由**From Survivor**和**To Survivor**组成 -2. 当创建一个对象时，对象会被优先分配到新生代的Eden区 + - 新生代又被进一步划分为 **Eden区** 和 **Survivor区**，Survivor 区由 **From Survivor** 和 **To Survivor** 组成 +2. 当创建一个对象时，对象会被优先分配到新生代的 Eden 区 - 此时 JVM 会给对象定义一个**对象年轻计数器**（`-XX:MaxTenuringThreshold`） 3. 当 Eden 空间不足时，JVM 将执行新生代的垃圾回收（Minor GC） - JVM 会把存活的对象转移到 Survivor 中，并且对象年龄 +1 @@ -491,6 +489,8 @@ JVM 在进行 GC 时，并非每次都对堆内存（新生代、老年代；方 ### 4.6 TLAB +Thread Local Allocation Buffer 的简写，基于 CAS 的独享线程（Mutator Threads）可以优先将对象分配在 Eden 中的一块内存，因为是 Java 线程独享的内存区没有锁竞争，所以分配速度更快，每个 TLAB 都是一个线程独享的。 + #### 什么是 TLAB （Thread Local Allocation Buffer）? - 从内存模型而不是垃圾回收的角度，对 Eden 区域继续进行划分，JVM 为每个线程分配了一个私有缓存区域，它包含在 Eden 空间内 @@ -517,7 +517,7 @@ JVM 在进行 GC 时，并非每次都对堆内存（新生代、老年代；方 ### 4.7 堆是分配对象存储的唯一选择吗 -随着 JIT 编译期的发展和逃逸分析技术的逐渐成熟，栈上分配、标量替换优化技术将会导致一些微妙的变化，所有的对象都分配到堆上也渐渐变得不那么“绝对”了。 ——《深入理解 Java 虚拟机》 +> 随着 JIT 编译期的发展和逃逸分析技术的逐渐成熟，栈上分配、标量替换优化技术将会导致一些微妙的变化，所有的对象都分配到堆上也渐渐变得不那么“绝对”了。 ——《深入理解 Java 虚拟机》 #### 逃逸分析 @@ -558,7 +558,7 @@ public static String createStringBuffer(String s1, String s2) { **参数设置：** -- 在 JDK 6u23版本之后，HotSpot 中默认就已经开启了逃逸分析 +- 在 JDK 6u23 版本之后，HotSpot 中默认就已经开启了逃逸分析 - 如果使用较早版本，可以通过`-XX"+DoEscapeAnalysis`显式开启 开发中使用局部变量，就不要在方法外定义。 @@ -580,7 +580,7 @@ JIT 编译器在编译期间根据逃逸分析的结果，发现如果一个对 ##### 代码优化之同步省略（消除） - 线程同步的代价是相当高的，同步的后果是降低并发性和性能 -- 在动态编译同步块的时候，JIT 编译器可以借助逃逸分析来判断同步块所使用的锁对象是否能够被一个线程访问而没有被发布到其他线程。如果没有，那么 JIT 编译器在编译这个同步块的时候就会取消对这个代码的同步。这样就能大大提高并发性和性能。这个取消同步的过程就叫做同步省略，也叫锁消除。 +- 在动态编译同步块的时候，JIT 编译器可以借助逃逸分析来判断同步块所使用的锁对象是否能够被一个线程访问而没有被发布到其他线程。如果没有，那么 JIT 编译器在编译这个同步块的时候就会取消对这个代码的同步。这样就能大大提高并发性和性能。这个取消同步的过程就叫做**同步省略，也叫锁消除**。 ```java public void keep() { @@ -608,7 +608,7 @@ public void keep() { 相对的，那些的还可以分解的数据叫做**聚合量**（Aggregate），Java 中的对象就是聚合量，因为其还可以分解成其他聚合量和标量。 -在 JIT 阶段，通过逃逸分析确定该对象不会被外部访问，并且对象可以被进一步分解时，JVM不会创建该对象，而会将该对象成员变量分解若干个被这个方法使用的成员变量所代替。这些代替的成员变量在栈帧或寄存器上分配空间。这个过程就是**标量替换**。 +在 JIT 阶段，通过逃逸分析确定该对象不会被外部访问，并且对象可以被进一步分解时，JVM 不会创建该对象，而会将该对象成员变量分解若干个被这个方法使用的成员变量所代替。这些代替的成员变量在栈帧或寄存器上分配空间。这个过程就是**标量替换**。 通过 `-XX:+EliminateAllocations` 可以开启标量替换，`-XX:+PrintEliminateAllocations` 查看标量替换情况。 @@ -627,7 +627,7 @@ class Point{ } ``` -以上代码中，point 对象并没有逃逸出`alloc()`方法，并且 point 对象是可以拆解成标量的。那么，JIT 就不会直接创建 Point 对象，而是直接使用两个标量 int x ，int y 来替代 Point 对象。 +以上代码中，point 对象并没有逃逸出 `alloc()` 方法，并且 point 对象是可以拆解成标量的。那么，JIT 就不会直接创建 Point 对象，而是直接使用两个标量 int x ，int y 来替代 Point 对象。 ```java private static void alloc() { @@ -663,7 +663,7 @@ private static void alloc() { - 方法区（Method Area）与 Java 堆一样，是所有线程共享的内存区域。 - 虽然 Java 虚拟机规范把方法区描述为堆的一个逻辑部分，但是它却有一个别名叫 Non-Heap（非堆），目的应该是与 Java 堆区分开。 -- 运行时常量池（Runtime Constant Pool）是方法区的一部分。Class 文件中除了有类的版本/字段/方法/接口等描述信息外，还有一项信息是常量池（Constant Pool Table），用于存放编译期生成的各种字面量和符号引用，这部分内容将类在加载后进入方法区的运行时常量池中存放。运行期间也可能将新的常量放入池中，这种特性被开发人员利用得比较多的是 `String.intern()`方法。受方法区内存的限制，当常量池无法再申请到内存时会抛出 `OutOfMemoryErro`r 异常。 +- 运行时常量池（Runtime Constant Pool）是方法区的一部分。Class 文件中除了有类的版本/字段/方法/接口等描述信息外，还有一项信息是常量池（Constant Pool Table），用于存放编译期生成的各种字面量和符号引用，这部分内容将类在加载后进入方法区的运行时常量池中存放。运行期间也可能将新的常量放入池中，这种特性被开发人员利用得比较多的是 `String.intern()`方法。受方法区内存的限制，当常量池无法再申请到内存时会抛出 `OutOfMemoryError` 异常。 - 方法区的大小和堆空间一样，可以选择固定大小也可选择可扩展，方法区的大小决定了系统可以放多少个类，如果系统类太多，导致方法区溢出，虚拟机同样会抛出内存溢出错误 - JVM 关闭后方法区即被释放 @@ -673,7 +673,7 @@ private static void alloc() { 你是否也有看不同的参考资料，有的内存结构图有方法区，有的又是永久代，元数据区，一脸懵逼的时候？ -- **方法区（method area）**只是**JVM规范**中定义的一个**概念**，用于存储类信息、常量池、静态变量、JIT编译后的代码等数据，并没有规定如何去实现它，不同的厂商有不同的实现。而**永久代（PermGen）**是 **Hotspot** 虚拟机特有的概念， Java8 的时候又被**元空间**取代了，永久代和元空间都可以理解为方法区的落地实现。 +- **方法区（method area）**只是 **JVM 规范**中定义的一个**概念**，用于存储类信息、常量池、静态变量、JIT编译后的代码等数据，并没有规定如何去实现它，不同的厂商有不同的实现。而**永久代（PermGen）**是 **Hotspot** 虚拟机特有的概念， Java8 的时候又被**元空间**取代了，永久代和元空间都可以理解为方法区的落地实现。 - 永久代物理是堆的一部分，和新生代，老年代地址是连续的（受垃圾回收器管理），而元空间存在于本地内存（我们常说的堆外内存，不受垃圾回收器管理），这样就不受 JVM 限制了，也比较难发生OOM（都会有溢出异常） - Java7 中我们通过`-XX:PermSize` 和 `-xx:MaxPermSize` 来设置永久代参数，Java8 之后，随着永久代的取消，这些参数也就随之失效了，改为通过`-XX:MetaspaceSize` 和 `-XX:MaxMetaspaceSize` 用来设置元空间参数 - 存储内容不同，元空间存储类的元信息，静态变量和常量池等并入堆中。相当于永久代的数据被分到了堆和元空间中 @@ -693,7 +693,7 @@ private static void alloc() { ### 5.2 设置方法区内存的大小 -jdk8及以后： +JDK8 及以后： - 元数据区大小可以使用参数 `-XX:MetaspaceSize` 和 `-XX:MaxMetaspaceSize` 指定，替代上述原有的两个参数 - 默认值依赖于平台。Windows 下，`-XX:MetaspaceSize` 是 21M，`-XX:MaxMetaspacaSize` 的值是 -1，即没有限制 @@ -705,7 +705,7 @@ jdk8及以后： ### 5.3 方法区内部结构 -方法区用于存储已被虚拟机加载的类型信息、常量、静态变量、即时编译器编译后的代码缓存等。 +方法区用于存储已被虚拟机加载的类型信息、常量、静态变量、即时编译器编译后的代码缓存等。 #### 类型信息 @@ -737,7 +737,7 @@ JVM 必须保存所有方法的 **栈、堆、方法区的交互关系** -![](https://static01.imgkr.com/temp/db050d0052a44605a13043a0bec204f0.png) +![图片](https://mmbiz.qpic.cn/mmbiz_png/Z0fxkgAKKLMAVEVNlW7gDqZIFzSvVq29TRzlibbP9xYfmZna5XO8A4ePIf0Jibs0BIBkq1VTouIaMMl1icRRp8Ygg/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) ### 5.4 运行时常量池 @@ -749,11 +749,11 @@ JVM 必须保存所有方法的 ##### 为什么需要常量池？ -一个 java 源文件中的类、接口，编译后产生一个字节码文件。而 Java 中的字节码需要数据支持，通常这种数据会很大以至于不能直接存到字节码里，换另一种方式，可以存到常量池，这个字节码包含了指向常量池的引用。在动态链接的时候用到的就是运行时常量池。 +一个 Java 源文件中的类、接口，编译后产生一个字节码文件。而 Java 中的字节码需要数据支持，通常这种数据会很大以至于不能直接存到字节码里，换另一种方式，可以存到常量池，这个字节码包含了指向常量池的引用。在动态链接的时候用到的就是运行时常量池。 -如下，我们通过jclasslib 查看一个只有 Main 方法的简单类，字节码中的 #2 指向的就是 Constant Pool +如下，我们通过 jclasslib 查看一个只有 Main 方法的简单类，字节码中的 #2 指向的就是 Constant Pool -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gg9i91ze2gj320i0riahe.jpg) +![图片](https://mmbiz.qpic.cn/mmbiz_jpg/Z0fxkgAKKLMAVEVNlW7gDqZIFzSvVq29PYTCbU8LShia2TQcibe0xefQibWmCVtKicttXza4gLfdicibp0un6yhqUX2w/640?wx_fmt=jpeg&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) 常量池可以看作是一张表，虚拟机指令根据这张常量表找到要执行的类名、方法名、参数类型、字面量等类型。 @@ -783,7 +783,7 @@ JVM 必须保存所有方法的 http://openjdk.java.net/jeps/122 -![](https://tva1.sinaimg.cn/large/007S8ZIlly1gg04ve34c2j30z00u0dp7.jpg) +![图片](https://mmbiz.qpic.cn/mmbiz_jpg/Z0fxkgAKKLMAVEVNlW7gDqZIFzSvVq29TOfUbUic8p6tVfwicc2Xn3hIEKaaq2wBoPOzplTrrobZUcx1V9DKUiaUw/640?wx_fmt=jpeg&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) - 为永久代设置空间大小是很难确定的。 @@ -797,7 +797,7 @@ http://openjdk.java.net/jeps/122 方法区的垃圾收集主要回收两部分内容：**常量池中废弃的常量和不再使用的类型**。 -先来说说方法区内常量池之中主要存放的两大类常量：字面量和符号引用。字面量比较接近 java 语言层次的常量概念，如文本字符串、被声明为 final 的常量值等。而符号引用则属于编译原理方面的概念，包括下面三类常量： +先来说说方法区内常量池之中主要存放的两大类常量：字面量和符号引用。字面量比较接近 Java 语言层次的常量概念，如文本字符串、被声明为 final 的常量值等。而符号引用则属于编译原理方面的概念，包括下面三类常量： - 类和接口的全限定名 - 字段的名称和描述符 @@ -821,12 +821,12 @@ Java 虚拟机被允许堆满足上述三个条件的无用类进行回收，这 算是一篇学习笔记，共勉，主要来源： -宋红康 JVM 教程 - 《深入理解 Java 虚拟机 第三版》 +宋红康老师的 JVM 教程 + https://docs.oracle.com/javase/specs/index.html https://www.cnblogs.com/wicfhwffg/p/9382677.html -https://www.cnblogs.com/hollischuang/p/12501950.html \ No newline at end of file +https://www.cnblogs.com/hollischuang/p/12501950.html diff --git a/docs/java/JVM/readJVM.md b/docs/java/JVM/readJVM.md index 7840dcd174..258eb4931d 100644 --- a/docs/java/JVM/readJVM.md +++ b/docs/java/JVM/readJVM.md @@ -1,2 +1,7 @@ JVM是Java Virtual Machine（Java虚拟机）的缩写，JVM是一种用于计算设备的规范，它是一个虚构出来的计算机，是通过在实际的计算机上仿真模拟各种计算机功能来实现的。 + + + + +JVM 相当于Java 程序员的内功心法 \ No newline at end of file diff --git a/docs/java/JVM/sidebar.md b/docs/java/JVM/sidebar.md index 1ad0fc7d48..0d077c88fe 100644 --- a/docs/java/JVM/sidebar.md +++ b/docs/java/JVM/sidebar.md @@ -1,12 +1,14 @@ - **Java基础** - [![](https://icongr.am/simple/oracle.svg?size=25&color=231c82&colored=false)JVM](java/JVM/readJVM.md) - [JVM 与 Java 体系结构](java/JVM/JVM-Java.md) - - [类加载子系统](java/JVM/类加载子系统.md) + - [类加载子系统](java/JVM/Class-Loading.md) - [运行时数据区](java/JVM/Runtime-Data-Areas.md) - [垃圾收集器](java/JVM/GC.md) + - [强引用、软引用、弱引用、虚引用](java/JVM/Reference.md) - [OOM 异常](java/JVM/OOM.md) - [JVM 参数配置](java/JVM/JVM参数配置.md) - [JVM 性能监控和故障处理工具](java/JVM/JVM性能监控和故障处理工具.md) + - [你了解自己的 ”Java对象“吗](java/JVM/Java-Object.md) - [![img](https://icongr.am/fontawesome/expeditedssl.svg?size=25&color=f23131)JUC](java/JUC/readJUC.md) - [![](https://icongr.am/devicon/java-original.svg?size=25&color=f23131)Java 8](java/Java8.md) - [![img](https://icongr.am/entypo/address.svg?size=25&color=074ca6)设计模式](design-pattern/readDisignPattern.md) diff --git a/docs/java/java8.md b/docs/java/Java-8.md similarity index 97% rename from docs/java/java8.md rename to docs/java/Java-8.md index d984ed6bd5..fbf13907ef 100644 --- a/docs/java/java8.md +++ b/docs/java/Java-8.md @@ -1,3 +1,5 @@ +# Java 8 通关攻略 + > 点赞+收藏 就学会系列，文章收录在 GitHub [JavaEgg](https://github.com/Jstarfish/JavaEgg) ，N线互联网开发必备技能兵器谱 > Java8早在2014年3月就发布了，还不得全面了解下 @@ -10,12 +12,12 @@ - **Lambda表达式**：一个新的语言特性， 它们使您能够将函数视为方法参数，或将代码视为数据 - **方法引用**： 方法引用为已经有名称的方法提供易于阅读的lambda表达式 - **默认方法**：使用 default 关键字为接口定义默认方法（有实现的方法） - - **重复注解**提供了将同一注解多次应用于同一声明或类型使用的能力 - - **类型注解**提供了在使用类型的任何地方应用注解的能力，而不仅仅是在声明上 - - Java8 增强了**类型推断** - - 方法参数反射 - - `java.util.function`: 一个新的包，它包含为lambda表达式和方法引用提供目标类型的通用功能接口 -- 集合（Collections） + - **重复注解**提供了将同一注解多次应用于同一声明或类型使用的能力 + - **类型注解**提供了在使用类型的任何地方应用注解的能力，而不仅仅是在声明上 + - Java8 增强了**类型推断** + - 方法参数反射 + - `java.util.function`: 一个新的包，它包含为lambda表达式和方法引用提供目标类型的通用功能接口 +- 集合（Collections） - `java.util.stream`包中新增了 **Stream API** ，用来支持对元素流的函数式操作 - 改进了有键冲突问题的 **HashMap** - 精简运行时（Compact Profiles） @@ -78,9 +80,9 @@ Lambda表达式使您能够封装单个行为单元并将其传递给其他代 ### 1. 为什么要使用Lambda表达式 -Lambda 是一个匿名函数，我们可以把 Lambda表达式理解为是一段可以传递的代码(**将代码像数据一样进行传递**——**行为参数化**)。可以写出更简洁、更灵活的代码。作为一种更紧凑的代码风格，使Java的语言表达能力得到了提升。 +Lambda 是一个匿名函数，我们可以把 Lambda表达式理解为是一段可以传递的代码(**将代码像数据一样进行传递**——**行为参数化**)。可以写出更简洁、更灵活的代码。作为一种更紧凑的代码风格，使 Java 的语言表达能力得到了提升。 -匿名类的一个问题是，如果您的匿名类的实现非常简单，例如一个接口只包含一个方法，那么匿名类的语法可能看起来很笨拙和不清楚。在这些情况下，您通常试图将功能作为参数传递给另一个方法，例如当有人单击按钮时应该采取什么操作。Lambda表达式允许您这样做，将功能视为方法参数，或将代码视为数据。 +匿名类的一个问题是，如果您的匿名类的实现非常简单，例如一个接口只包含一个方法，那么匿名类的语法可能看起来很笨拙和不清楚。在这些情况下，您通常试图将功能作为参数传递给另一个方法，例如当有人单击按钮时应该采取什么操作。Lambda 表达式允许您这样做，将功能视为方法参数，或将代码视为数据。 ![hello-lambda](https://i.loli.net/2019/12/26/WugthVbdwUEm5J2.png) @@ -142,7 +144,7 @@ Lambda 表达式在 Java 语言中引入了一个新的语法元素和操作符 **类型推断** -上述 Lambda 表达式中的参数类型都是由编译器推断得出的。Lambda 表达式中无需指定类型，程序依然可以编译，这是因为 javac 根据程序的上下文，在后台推断出了参数的类型。Lambda 表达式的类型依赖于上下文环境，是由编译器推断出来的。这就是所谓的“类型推断”。Java7中引入的**菱形运算符**（**<>**）,就是利用泛型从上下文推断类型。 +上述 Lambda 表达式中的参数类型都是由编译器推断得出的。Lambda 表达式中无需指定类型，程序依然可以编译，这是因为 javac 根据程序的上下文，在后台推断出了参数的类型。Lambda 表达式的类型依赖于上下文环境，是由编译器推断出来的。这就是所谓的“类型推断”。Java7 中引入的**菱形运算符**（**<>**）,就是利用泛型从上下文推断类型。 ```java List list = new ArrayList<>(); @@ -158,7 +160,7 @@ List list = new ArrayList<>(); **行为参数化**就是可以帮助你处理频繁变更的需求的一种软件开发模式。 -官方提供的demo，一步步告诉你使用Java8的好处（从值参数化到行为参数化）。[代码](https://github.com/Jstarfish/starfish-learning/tree/master/starfish-learn-java8/src/lambda) +官方提供的demo，一步步告诉你使用 Java8 的好处（从值参数化到行为参数化）。[代码](https://github.com/Jstarfish/starfish-learning/tree/master/starfish-learn-java8/src/lambda) ```java import java.util.List; @@ -238,7 +240,7 @@ public class Person { ``` -![](https://i04piccdn.sogoucdn.com/f55480c7c84bb963) + ```java import java.util.ArrayList; @@ -857,6 +859,8 @@ public void test7(){ ## 四、Stream——函数式数据处理 +![Redis Streams demonstration graphic](https://redislabs.com/wp-content/uploads/2018/07/Streams-1.png?_t=1532973073) + Stream 是 Java8 中处理集合的关键抽象概念，它可以指定你希望对集合进行的操作，可以执行非常复杂的查找、过滤和映射数据等操作。 使用Stream API 对集合数据进行操作，就类似于使用 SQL 执行的数据库查询。也可以使用 Stream API 来并行执行操作。简而言之， **Stream API 提供了一种高效且易于使用的处理数据的方式**。 ### 1. Stream是个啥 @@ -1241,9 +1245,9 @@ Collector接口中方法的实现决定了如何对流执行收集操作(如收 | 方法 | 返回类型 | 作用 | 示例 | | ----------------- | --------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | -| toList | List | 把流中元素收集到List | List list= list.stream().collect(Collectors.toList()); | -| toSet | Set | 把流中元素收集到Set | Set set= list.stream().collect(Collectors.toSet()); | -| toCollection | Collection | 把流中元素收集到创建的集合 | Collectione mps=list.stream().collect(Collectors.toCollection(ArrayList::new)); | +| toList | List\ | 把流中元素收集到List | List list= list.stream().collect(Collectors.toList()); | +| toSet | Set\ | 把流中元素收集到Set | Set set= list.stream().collect(Collectors.toSet()); | +| toCollection | Collection\ | 把流中元素收集到创建的集合 | Collectione mps=list.stream().collect(Collectors.toCollection(ArrayList::new)); | | counting | Long | 计算流中元素的个数 | long count = list.stream().collect(Collectors.counting()); | | summingInt | Integer | 对流中元素的整数属性求和 | Integer sum = persons.stream() .collect(Collectors.summingInt(Person::getAge)); | | averagingInt | Double | 计算流中元素Integer属性的平均值 | double avg= list.stream().collect(Collectors.averagingInt(Person::getAge)); | @@ -2049,6 +2053,26 @@ String类也新增􏱗了一个静态方法，名叫join。它可以用一个分 Reflection API的变化就是为了支持Java 8中注解机制的改变。 除此之外，Relection接口的另一个变化是**新增了可以查询方法参数信息的API**，比如，你现在可以使用新的`java.lang.reflect.Parameter`类查询方法参数的名称和修饰符。 + + +### 级联表达式和珂里化 + +```java +/** + * 珂里化：把多个参数的函数转换为只有一个参数的函数 + * 珂里化的目的：函数标准化 + * 高阶函数：返回函数的函数 + **/ +public static void main(String[] args) { + //实现了 x+y 的级联表达式 + Function> fun = x ->y -> x + y; + + System.out.println(fun.apply(2).apply(3)); +} +``` + + + ------ diff --git a/docs/java/Java-Throwable.md b/docs/java/Java-Throwable.md new file mode 100644 index 0000000000..d06a735c66 --- /dev/null +++ b/docs/java/Java-Throwable.md @@ -0,0 +1,294 @@ +# Java 中的异常机制 + +> 先问大家个面试题：在 Java 中，为什么有些异常，我们必须抛出或者 `try/catch`，像 `Thread.sleep(1000);` 这种必须处理 InterruptedException，否则编译都通过不了，而 NullPointerException 这类异常却“无所谓”？ + +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20201030111326.png) + + + +## 概述 + +异常，就是我们 **写代码** 或者 **编译、运行代码** 时候遇到的各种状况，比如：参数不合法、找不到该类或该方法、数组下标越界、空指针异常等等。 + +作为一名合格的程序员，我们肯定要预见异常、处理异常。 + +Java 通过 Throwable 类的众多子类描述各种不同的异常。所以，**Java 异常都是对象，是 Throwable 子类的实例，描述了出现在一段编码中的错误条件**。当条件生成时，错误将引发异常。 + +Throwable 类是所以异常的“祖先”，他有两个“女儿”：Exception（异常）和 Error（错误），她两又有好多“子孙”。 + +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20201030153338.png) + +## Error + +先说大女儿 Error（错误），她比较厉害，从名字就能看出，她都不算“异常”，是“错误”。是程序无法处理的错误，表示运行应用程序中较严重问题。 + +她代表的大多数错误和我们这些 Coder 执行的操作关系不大，而是指代码运行时 JVM 出现的问题。比如上图中的虚拟机错误、内存溢出、线程死锁这些，这些错误是不可查的，我们写代码时候可不好把控。 + +当遇到这类错误的时候，JVM 一般会选择终止线程。可以说这个对象很是棘手。 + +> 对于设计合理的应用程序来说，即使确实发生了错误，本质上也不应该试图去处理它所引起的异常状况。 + + + +## Exception + +接着说二女儿 Exception，Exception（异常）的“子女”又分两大阵营：**运行时异常（RuntimeException，又叫非检查异常)**和**非运行时异常**(编译异常，也叫检查异常)。 + +程序中应当尽可能去处理这些异常。 + +### 运行时异常 + +运行时异常都是 RuntimeException 类及其子类，比如 NullPointerException、IndexOutOfBoundsException 等 + +这些异常是不检查异常，程序中可以选择捕获处理，也可以不处理。比如遇到这些错误，我们再 IDEA 中也不会报错，这类错误一般都是由于我们程序逻辑错误造成的，比如数组长度明明只有 10，你却要获取索引是 11 的元素，这类错误应该在 Coding 的时候就避免掉。 + +运行时异常的特点是 Java 编译器不会检查它，也就是说，当程序中可能出现这类异常，即使没有用 try-catch 语句捕获它，也没有用 throws 子句声明抛出它，也会编译通过。 + +### 非运行时异常 （编译异常、检查异常） + +RuuntimeException 以外的异常，都属于检查异常。也就是必须处理的异常，否则程序无法正常编译。这个我们代码中就随处可见了，比如 IOException、ClassNotFoundException 这类必须抛出或处理的类异常。 + +当然，我们写代码一般通过 IDEA，这类检查异常，都会在编译器中报错，所以也很好识别。 + + + +最后附一张完整的图： + +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/20201030153846.png) + +## 异常处理最佳实践 + +#### 不要忽略捕捉的异常 + +```java +catch (NoSuchMethodException e) { + return null; +} +``` + +虽然捕捉了异常但是却没有做任何处理，除非你确信这个异常可以忽略，不然不应该这样做。这样会导致外面无法知晓该方法发生了错误，无法确定定位错误原因。 + +#### 在你的方法里抛出定义具体的检查性异常 + +```java +public void foo() throws Exception { //错误方式 +} +``` + +一定要避免出现上面的代码示例，它破坏了检查性异常的目的。 声明你的方法可能抛出的具体检查性异常，如果只有太多这样的检查性异常，你应该把它们包装在你自己的异常中，并在异常消息中添加信息。 如果可能的话，你也可以考虑代码重构。 + +```java +public void foo() throws SpecificException1, SpecificException2 { //正确方式 +} +``` + +#### 捕获具体的子类而不是捕获 Exception 类 + +```java +try { + someMethod(); +} catch (Exception e) { //错误方式 + LOGGER.error("method has failed", e); +} +``` + +捕获异常的问题是，如果稍后调用的方法为其方法声明添加了新的检查性异常，则开发人员的意图是应该处理具体的新异常。如果你的代码只是捕获异常（或 Throwable），永远不会知道这个变化，以及你的代码现在是错误的，并且可能会在运行时的任何时候中断。 + +#### 永远不要捕获 Throwable 类 + +这是一个更严重的麻烦，因为 Java Error 也是 Throwable 的子类，Error 是 JVM 本身无法处理的不可逆转的条件，对于某些 JVM 的实现，JVM 可能实际上甚至不会在 Error 上调用 catch 子句。 + +#### 始终正确包装自定义异常中的异常，以便堆栈跟踪不会丢失 + +```java +catch (NoSuchMethodException e) { + throw new MyServiceException("Some information: " + e.getMessage()); //错误方式 +} +``` + +这破坏了原始异常的堆栈跟踪，并且始终是错误的，正确的做法是： + +```java +catch (NoSuchMethodException e) { + throw new MyServiceException("Some information: " , e); //正确方式 +}复 +``` + +#### 要么记录异常要么抛出异常，但不要一起执行 + +```java +catch (NoSuchMethodException e) { +//错误方式 + LOGGER.error("Some information", e); + throw e; +} +``` + +正如上面的代码中，记录和抛出异常会在日志文件中产生多条日志消息，代码中存在单个问题，并且对尝试分析日志的同事很不友好。 + +#### finally 块中永远不要抛出任何异常 + +```java +try { + someMethod(); //Throws exceptionOne +} finally { + cleanUp(); //如果finally还抛出异常，那么exceptionOne将永远丢失 +} +``` + +只要 cleanUp() 永远不会抛出任何异常，上面的代码没有问题，但是如果 someMethod() 抛出一个异常，并且在 finally 块中，cleanUp() 也抛出另一个异常，那么程序只会把第二个异常抛出来，原来的第一个异常（正确的原因）将永远丢失。如果在 finally 块中调用的代码可能会引发异常，请确保要么处理它，要么将其记录下来。永远不要让它从 finally 块中抛出来。 + +#### 始终只捕获实际可处理的异常 + +```java +catch (NoSuchMethodException e) { + throw e; //避免这种情况，因为它没有任何帮助 +} +``` + +这是最重要的概念，不要为了捕捉异常而捕捉，只有在想要处理异常时才捕捉异常，或者希望在该异常中提供其他上下文信息。如果你不能在 catch 块中处理它，那么最好的建议就是不要只为了重新抛出它而捕获它。 + +#### 不要使用 printStackTrace() 语句或类似的方法 + +完成代码后，切勿忽略 printStackTrace()，最终别人可能会得到这些堆栈，并且对于如何处理它完全没有任何方法，因为它不会附加任何上下文信息。 + +#### 对于不打算处理的异常，直接使用 finally + +```java +try { + someMethod(); //Method 2 +} finally { + cleanUp(); //do cleanup here +} +``` + +这是一个很好的做法，如果在你的方法中你正在访问 Method 2，而 Method 2 抛出一些你不想在 Method 1 中处理的异常，但是仍然希望在发生异常时进行一些清理，然后在 finally 块中进行清理，不要使用 catch 块。 + +#### 记住早 throw 晚 catch 原则 + +这可能是关于异常处理最著名的原则，简单说，应该尽快抛出(throw)异常，并尽可能晚地捕获(catch)它。应该等到有足够的信息来妥善处理它。 + +这个原则隐含地说，你将更有可能把它放在低级方法中，在那里你将检查单个值是否为空或不适合。而且你会让异常堆栈跟踪上升好几个级别，直到达到足够的抽象级别才能处理问题。 + +#### 在异常处理后清理资源 + +如果你正在使用数据库连接或网络连接等资源，请确保清除它们。如果你正在调用的 API 仅使用非检查性异常，则仍应使用 try-finally 块来清理资源。 在 try 模块里面访问资源，在 finally 里面最后关闭资源。即使在访问资源时发生任何异常，资源也会优雅地关闭。 + +#### 只抛出和方法相关的异常 + +相关性对于保持应用程序清洁非常重要。一种尝试读取文件的方法，如果抛出 NullPointerException，那么它不会给用户任何相关的信息。相反，如果这种异常被包裹在自定义异常中，则会更好。NoSuchFileFoundException 则对该方法的用户更有用。 + +#### 切勿在程序中使用异常来进行流程控制 + +不要在项目中出现使用异常来处理应用程序逻辑。永远不要这样做，它会使代码很难阅读和理解。 + +#### 尽早验证用户输入以在请求处理的早期捕获异常 + +始终要在非常早的阶段验证用户输入，甚至在达到 controller 之前，它将帮助你把核心应用程序逻辑中的异常处理代码量降到最低。如果用户输入出现错误，还可以保证与应用程序一致。 + +例如：如果在用户注册应用程序中，遵循以下逻辑： + +1. 验证用户 +2. 插入用户 +3. 验证地址 +4. 插入地址 +5. 如果出问题回滚一切 + +这是不正确的做法，它会使数据库在各种情况下处于不一致的状态，应该首先验证所有内容，然后将用户数据置于 dao 层并进行数据库更新。正确的做法是： + +1. 验证用户 +2. 验证地址 +3. 插入用户 +4. 插入地址 +5. 如果问题回滚一切 + +#### 一个异常只能包含在一个日志中 + +```java +LOGGER.debug("Using cache sector A"); +LOGGER.debug("Using retry sector B");复制代码 +``` + +不要像上面这样做，对多个 `LOGGER.debug()` 调用使用多行日志消息可能在你的测试用例中看起来不错，但是当它在具有 100 个并行运行的线程的应用程序服务器的日志文件中显示时，所有信息都输出到相同的日志文件，即使它们在实际代码中为前后行，但是在日志文件中这两个日志消息可能会间隔 100 多行。应该这样做： + +```java +LOGGER.debug("Using cache sector A, using retry sector B");复制代码 +``` + +#### 将所有相关信息尽可能地传递给异常 + +有用的异常消息和堆栈跟踪非常重要，如果你的日志不能定位异常位置，那要日志有什么用呢？ + +#### 终止掉被中断线程 + +```java +while (true) { + try { + Thread.sleep(100000); + } catch (InterruptedException e) {} //别这样做 + doSomethingCool(); +} +``` + +InterruptedException 异常提示应该停止程序正在做的事情，比如事务超时或线程池被关闭等。 + +应该尽最大努力完成正在做的事情，并完成当前执行的线程，而不是忽略 InterruptedException。修改后的程序如下： + +```java +while (true) { + try { + Thread.sleep(100000); + } catch (InterruptedException e) { + break; + } +} +doSomethingCool(); +``` + +#### 对于重复的 try-catch，使用模板方法 + +在代码中有许多类似的 catch 块是无用的，只会增加代码的重复性，针对这样的问题可以使用模板方法。 + +例如，在尝试关闭数据库连接时的异常处理。 + +```java +class DBUtil{ + public static void closeConnection(Connection conn){ + try{ + conn.close(); + } catch(Exception ex){ + //Log Exception - Cannot close connection + } + } +} +``` + +这类的方法将在应用程序很多地方使用。不要把这块代码放的到处都是，而是定义上面的方法，然后像下面这样使用它： + +```java +public void dataAccessCode() { + Connection conn = null; + try{ + conn = getConnection(); + .... + } finally{ + DBUtil.closeConnection(conn); + } +} +``` + +#### 使用 JavaDoc 中记录应用程序中的所有异常 + +把用 JavaDoc 记录运行时可能抛出的所有异常作为一种习惯，其中也尽量包括用户应该遵循的操作，以防这些异常发生。 + + + + +### 参考与来源： + +https://blog.csdn.net/fuzhongmin05/article/details/77600410 + +https://juejin.im/post/6844903981299433479 + + + diff --git a/docs/java/Online-Error-Check.md b/docs/java/Online-Error-Check.md new file mode 100644 index 0000000000..a848d1ef1f --- /dev/null +++ b/docs/java/Online-Error-Check.md @@ -0,0 +1,355 @@ +# JAVA线上故障排查全套路 + +> 来源：https://fredal.xin/java-error-check + +线上故障主要会包括cpu、磁盘、内存以及网络问题，而大多数故障可能会包含不止一个层面的问题，所以进行排查时候尽量四个方面依次排查一遍。同时例如jstack、jmap等工具也是不囿于一个方面的问题的，基本上出问题就是df、free、top 三连，然后依次jstack、jmap伺候，具体问题具体分析即可。 + +## CPU + +一般来讲我们首先会排查cpu方面的问题。cpu异常往往还是比较好定位的。原因包括业务逻辑问题(死循环)、频繁gc以及上下文切换过多。而最常见的往往是业务逻辑(或者框架逻辑)导致的，可以使用jstack来分析对应的堆栈情况。 + +### 使用jstack分析cpu问题 + +我们先用ps命令找到对应进程的pid(如果你有好几个目标进程，可以先用top看一下哪个占用比较高)。 +接着用`top -H -p pid`来找到cpu使用率比较高的一些线程 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083804.png) +然后将占用最高的pid转换为16进制`printf '%x\n' pid`得到nid +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083806.png) +接着直接在jstack中找到相应的堆栈信息`jstack pid |grep 'nid' -C5 –color` +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-83807.png) +可以看到我们已经找到了nid为0x42的堆栈信息，接着只要仔细分析一番即可。 + +当然更常见的是我们对整个jstack文件进行分析，通常我们会比较关注WAITING和TIMED_WAITING的部分，BLOCKED就不用说了。我们可以使用命令`cat jstack.log | grep "java.lang.Thread.State" | sort -nr | uniq -c`来对jstack的状态有一个整体的把握，如果WAITING之类的特别多，那么多半是有问题啦。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083807.png) + +### 频繁gc + +当然我们还是会使用jstack来分析问题，但有时候我们可以先确定下gc是不是太频繁，使用`jstat -gc pid 1000`命令来对gc分代变化情况进行观察，1000表示采样间隔(ms)，S0C/S1C、S0U/S1U、EC/EU、OC/OU、MC/MU分别代表两个Survivor区、Eden区、老年代、元数据区的容量和使用量。YGC/YGT、FGC/FGCT、GCT则代表YoungGc、FullGc的耗时和次数以及总耗时。如果看到gc比较频繁，再针对gc方面做进一步分析，具体可以参考一下gc章节的描述。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083808.png) + +### 上下文切换 + +针对频繁上下文问题，我们可以使用`vmstat`命令来进行查看 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083809.png) +cs(context switch)一列则代表了上下文切换的次数。 +如果我们希望对特定的pid进行监控那么可以使用 `pidstat -w pid`命令，cswch和nvcswch表示自愿及非自愿切换。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-83810.png) + +## 磁盘 + +磁盘问题和cpu一样是属于比较基础的。首先是磁盘空间方面，我们直接使用`df -hl`来查看文件系统状态 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083810.png) + +更多时候，磁盘问题还是性能上的问题。我们可以通过iostat`iostat -d -k -x`来进行分析 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083811.png) +最后一列`%util`可以看到每块磁盘写入的程度，而`rrqpm/s`以及`wrqm/s`分别表示读写速度，一般就能帮助定位到具体哪块磁盘出现问题了。 + +另外我们还需要知道是哪个进程在进行读写，一般来说开发自己心里有数，或者用iotop命令来进行定位文件读写的来源。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083812.png) +不过这边拿到的是tid，我们要转换成pid，可以通过readlink来找到pid`readlink -f /proc/*/task/tid/../..`。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-83813.png) +找到pid之后就可以看这个进程具体的读写情况`cat /proc/pid/io` +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083813.png) +我们还可以通过lsof命令来确定具体的文件读写情况`lsof -p pid` +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083814.png) + +## 内存 + +内存问题排查起来相对比CPU麻烦一些，场景也比较多。主要包括OOM、GC问题和堆外内存。一般来讲，我们会先用`free`命令先来检查一发内存的各种情况。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083815.png) + +### 堆内内存 + +内存问题大多还都是堆内内存问题。表象上主要分为OOM和StackOverflow。 + +#### OOM + +JMV中的内存不足，OOM大致可以分为以下几种： + +**Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread** +这个意思是没有足够的内存空间给线程分配java栈，基本上还是线程池代码写的有问题，比如说忘记shutdown，所以说应该首先从代码层面来寻找问题，使用jstack或者jmap。如果一切都正常，JVM方面可以通过指定`Xss`来减少单个thread stack的大小。另外也可以在系统层面，可以通过修改`/etc/security/limits.conf`nofile和nproc来增大os对线程的限制 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-83816.png) + +**Exception in thread "main" java.lang.OutOfMemoryError: Java heap space** +这个意思是堆的内存占用已经达到-Xmx设置的最大值，应该是最常见的OOM错误了。解决思路仍然是先应该在代码中找，怀疑存在内存泄漏，通过jstack和jmap去定位问题。如果说一切都正常，才需要通过调整`Xmx`的值来扩大内存。 + +**Caused by: java.lang.OutOfMemoryError: Meta space** +这个意思是元数据区的内存占用已经达到`XX:MaxMetaspaceSize`设置的最大值，排查思路和上面的一致，参数方面可以通过`XX:MaxPermSize`来进行调整(这里就不说1.8以前的永久代了)。 + +#### Stack Overflow + +栈内存溢出，这个大家见到也比较多。 +**Exception in thread "main" java.lang.StackOverflowError** +表示线程栈需要的内存大于Xss值，同样也是先进行排查，参数方面通过`Xss`来调整，但调整的太大可能又会引起OOM。 + +#### 使用JMAP定位代码内存泄漏 + +上述关于OOM和StackOverflow的代码排查方面，我们一般使用JMAP`jmap -dump:format=b,file=filename pid`来导出dump文件 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083817.png) +通过mat(Eclipse Memory Analysis Tools)导入dump文件进行分析，内存泄漏问题一般我们直接选Leak Suspects即可，mat给出了内存泄漏的建议。另外也可以选择Top Consumers来查看最大对象报告。和线程相关的问题可以选择thread overview进行分析。除此之外就是选择Histogram类概览来自己慢慢分析，大家可以搜搜mat的相关教程。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083818.png) + +日常开发中，代码产生内存泄漏是比较常见的事，并且比较隐蔽，需要开发者更加关注细节。比如说每次请求都new对象，导致大量重复创建对象；进行文件流操作但未正确关闭；手动不当触发gc；ByteBuffer缓存分配不合理等都会造成代码OOM。 + +另一方面，我们可以在启动参数中指定`-XX:+HeapDumpOnOutOfMemoryError`来保存OOM时的dump文件。 + +#### gc问题和线程 + +gc问题除了影响cpu也会影响内存，排查思路也是一致的。一般先使用jstat来查看分代变化情况，比如youngGC或者fullGC次数是不是太多呀；EU、OU等指标增长是不是异常呀等。 +线程的话太多而且不被及时gc也会引发oom，大部分就是之前说的`unable to create new native thread`。除了jstack细细分析dump文件外，我们一般先会看下总体线程，通过`pstreee -p pid |wc -l`。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083819.png) +或者直接通过查看`/proc/pid/task`的数量即为线程数量。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083820.png) + +### 堆外内存 + +如果碰到堆外内存溢出，那可真是太不幸了。首先堆外内存溢出表现就是物理常驻内存增长快，报错的话视使用方式都不确定，如果由于使用Netty导致的，那错误日志里可能会出现`OutOfDirectMemoryError`错误，如果直接是DirectByteBuffer，那会报`OutOfMemoryError: Direct buffer memory`。 + +堆外内存溢出往往是和NIO的使用相关，一般我们先通过pmap来查看下进程占用的内存情况`pmap -x pid | sort -rn -k3 | head -30`，这段意思是查看对应pid倒序前30大的内存段。这边可以再一段时间后再跑一次命令看看内存增长情况，或者和正常机器比较可疑的内存段在哪里。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-83821.png) +我们如果确定有可疑的内存端，需要通过gdb来分析`gdb --batch --pid {pid} -ex "dump memory filename.dump {内存起始地址} {内存起始地址+内存块大小}"` +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083821.png) +获取dump文件后可用heaxdump进行查看`hexdump -C filename | less`，不过大多数看到的都是二进制乱码。 + +NMT是Java7U40引入的HotSpot新特性，配合jcmd命令我们就可以看到具体内存组成了。需要在启动参数中加入 `-XX:NativeMemoryTracking=summary` 或者 `-XX:NativeMemoryTracking=detail`，会有略微性能损耗。 + +一般对于堆外内存缓慢增长直到爆炸的情况来说，可以先设一个基线`jcmd pid VM.native_memory baseline`。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083822.png) +然后等放一段时间后再去看看内存增长的情况，通过`jcmd pid VM.native_memory detail.diff(summary.diff)`做一下summary或者detail级别的diff。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083823.png) +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-83824.png) +可以看到jcmd分析出来的内存十分详细，包括堆内、线程以及gc(所以上述其他内存异常其实都可以用nmt来分析)，这边堆外内存我们重点关注Internal的内存增长，如果增长十分明显的话那就是有问题了。 +detail级别的话还会有具体内存段的增长情况，如下图。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083824.png) + +此外在系统层面，我们还可以使用strace命令来监控内存分配 `strace -f -e "brk,mmap,munmap" -p pid` +这边内存分配信息主要包括了pid和内存地址。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083825.jpg) + +不过其实上面那些操作也很难定位到具体的问题点，关键还是要看错误日志栈，找到可疑的对象，搞清楚它的回收机制，然后去分析对应的对象。比如DirectByteBuffer分配内存的话，是需要full GC或者手动system.gc来进行回收的(所以最好不要使用`-XX:+DisableExplicitGC`)。那么其实我们可以跟踪一下DirectByteBuffer对象的内存情况，通过`jmap -histo:live pid`手动触发fullGC来看看堆外内存有没有被回收。如果被回收了，那么大概率是堆外内存本身分配的太小了，通过`-XX:MaxDirectMemorySize`进行调整。如果没有什么变化，那就要使用jmap去分析那些不能被gc的对象，以及和DirectByteBuffer之间的引用关系了。 + + + +## GC问题 + +堆内内存泄漏总是和 GC 异常相伴。不过 GC 问题不只是和内存问题相关，还有可能引起 CPU 负载、网络问题等系列并发症，只是相对来说和内存联系紧密些，所以我们在此单独总结一下 GC 相关问题。 + +我们在 cpu 章介绍了使用jstat来获取当前GC分代变化信息。而更多时候，我们是通过GC日志来排查问题的，在启动参数中加上`-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps`来开启GC日志。 +常见的Young GC、Full GC日志含义在此就不做赘述了。 + +针对gc日志，我们就能大致推断出youngGC与fullGC是否过于频繁或者耗时过长，从而对症下药。我们下面将对G1垃圾收集器来做分析，这边也建议大家使用G1`-XX:+UseG1GC`。 + +**youngGC过频繁** +youngGC 频繁一般是短周期小对象较多，先考虑是不是 Eden区/新生代设置的太小了，看能否通过调整 -Xmn、-XX:SurvivorRatio 等参数设置来解决问题。如果参数正常，但是 young gc 频率还是太高，就需要使用 Jmap 和MAT 对 dump 文件进行进一步排查了。 + +**youngGC耗时过长** +耗时过长问题就要看GC日志里耗时耗在哪一块了。以G1日志为例，可以关注Root Scanning、Object Copy、Ref Proc等阶段。Ref Proc耗时长，就要注意引用相关的对象。Root Scanning耗时长，就要注意线程数、跨代引用。Object Copy 则需要关注对象生存周期。而且耗时分析它需要横向比较，就是和其他项目或者正常时间段的耗时比较。比如说图中的Root Scanning和正常时间段比增长较多，那就是起的线程太多了。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083826.png) + +**触发fullGC** +G1中更多的还是mixedGC，但 mixedGC 可以和 youngGC 思路一样去排查。触发 fullGC 了一般都会有问题，G1会退化使用 Serial 收集器来完成垃圾的清理工作，暂停时长达到秒级别，可以说是半跪了。 +fullGC的原因可能包括以下这些，以及参数调整方面的一些思路： + +- 并发阶段失败：在并发标记阶段，MixGC之前老年代就被填满了，那么这时候G1就会放弃标记周期。这种情况，可能就需要增加堆大小，或者调整并发标记线程数`-XX:ConcGCThreads`。 +- 晋升失败：在GC的时候没有足够的内存供存活/晋升对象使用，所以触发了Full GC。这时候可以通过`-XX:G1ReservePercent`来增加预留内存百分比，减少`-XX:InitiatingHeapOccupancyPercent`来提前启动标记，`-XX:ConcGCThreads`来增加标记线程数也是可以的。 +- 大对象分配失败：大对象找不到合适的region空间进行分配，就会进行fullGC，这种情况下可以增大内存或者增大`-XX:G1HeapRegionSize`。 +- 程序主动执行System.gc()：不要随便写就对了。 + +另外，我们可以在启动参数中配置`-XX:HeapDumpPath=/xxx/dump.hprof`来dump fullGC相关的文件，并通过jinfo来进行gc前后的dump + +```java +jinfo -flag +HeapDumpBeforeFullGC pid +jinfo -flag +HeapDumpAfterFullGC pid +``` + +这样得到2份dump文件，对比后主要关注被gc掉的问题对象来定位问题。 + + + +### Full GC 问题 + +下面4种情况，对象会进入到老年代中： + +- YGC时，To Survivor区不足以存放存活的对象，对象会直接进入到老年代。 +- 经过多次YGC后，如果存活对象的年龄达到了设定阈值，则会晋升到老年代中。 +- 动态年龄判定规则，To Survivor区中相同年龄的对象，如果其大小之和占到了 To Survivor区一半以上的空间，那么大于此年龄的对象会直接进入老年代，而不需要达到默认的分代年龄。 +- 大对象：由-XX:PretenureSizeThreshold启动参数控制，若对象大小大于此值，就会绕过新生代, 直接在老年代中分配。 + +当晋升到老年代的对象大于了老年代的剩余空间时，就会触发FGC（Major GC），FGC处理的区域同时包括新生代和老年代。除此之外，还有以下4种情况也会触发FGC： + +- 老年代的内存使用率达到了一定阈值（可通过参数调整），直接触发FGC。 +- 空间分配担保：在YGC之前，会先检查老年代最大可用的连续空间是否大于新生代所有对象的总空间。如果小于，说明YGC是不安全的，则会查看参数 HandlePromotionFailure 是否被设置成了允许担保失败，如果不允许则直接触发Full GC；如果允许，那么会进一步检查老年代最大可用的连续空间是否大于历次晋升到老年代对象的平均大小，如果小于也会触发 Full GC。 +- Metaspace（元空间）在空间不足时会进行扩容，当扩容到了-XX:MetaspaceSize 参数的指定值时，也会触发FGC。 +- System.gc() 或者Runtime.gc() 被显式调用时，触发FGC。 + + + +##### 在什么情况下，GC会对程序产生影响？ + +不管YGC还是FGC，都会造成一定程度的程序卡顿（即Stop The World问题：GC线程开始工作，其他工作线程被挂起），即使采用ParNew、CMS或者G1这些更先进的垃圾回收算法，也只是在减少卡顿时间，而并不能完全消除卡顿。 + +那到底什么情况下，GC会对程序产生影响呢？根据严重程度从高到底，我认为包括以下4种情况： + +- FGC过于频繁：FGC通常是比较慢的，少则几百毫秒，多则几秒，正常情况FGC每隔几个小时甚至几天才执行一次，对系统的影响还能接受。但是，一旦出现FGC频繁（比如几十分钟就会执行一次），这种肯定是存在问题的，它会导致工作线程频繁被停止，让系统看起来一直有卡顿现象，也会使得程序的整体性能变差。 +- YGC耗时过长：一般来说，YGC的总耗时在几十或者上百毫秒是比较正常的，虽然会引起系统卡顿几毫秒或者几十毫秒，这种情况几乎对用户无感知，对程序的影响可以忽略不计。但是如果YGC耗时达到了1秒甚至几秒（都快赶上FGC的耗时了），那卡顿时间就会增大，加上YGC本身比较频繁，就会导致比较多的服务超时问题。 +- FGC耗时过长：FGC耗时增加，卡顿时间也会随之增加，尤其对于高并发服务，可能导致FGC期间比较多的超时问题，可用性降低，这种也需要关注。 +- YGC过于频繁：即使YGC不会引起服务超时，但是YGC过于频繁也会降低服务的整体性能，对于高并发服务也是需要关注的。 + +其中，「FGC过于频繁」和「YGC耗时过长」，这两种情况属于比较典型的GC问题，大概率会对程序的服务质量产生影响。剩余两种情况的严重程度低一些，但是对于高并发或者高可用的程序也需要关注。 + + + +### 排查FGC问题的实践指南 + +通过上面的案例分析以及理论介绍，再总结下FGC问题的排查思路，作为一份实践指南供大家参考。 + +#### 1. 清楚从程序角度，有哪些原因导致FGC？ + +- 大对象：系统一次性加载了过多数据到内存中（比如SQL查询未做分页），导致大对象进入了老年代。 +- 内存泄漏：频繁创建了大量对象，但是无法被回收（比如IO对象使用完后未调用close方法释放资源），先引发FGC，最后导致OOM. +- 程序频繁生成一些长生命周期的对象，当这些对象的存活年龄超过分代年龄时便会进入老年代，最后引发FGC. +- 程序BUG导致动态生成了很多新类，使得 Metaspace 不断被占用，先引发FGC，最后导致OOM. +- 代码中显式调用了gc方法，包括自己的代码甚至框架中的代码。 +- JVM参数设置问题：包括总内存大小、新生代和老年代的大小、Eden区和S区的大小、元空间大小、垃圾回收算法等等。 + +#### 2. 清楚排查问题时能使用哪些工具 + +- 公司的监控系统：大部分公司都会有，可全方位监控JVM的各项指标。 + +- JDK的自带工具，包括jmap、jstat等常用命令： + + \# 查看堆内存各区域的使用率以及GC情况 + + jstat -gcutil -h20 pid 1000 + + \# 查看堆内存中的存活对象，并按空间排序 + + jmap -histo pid | head -n20 + + \# dump堆内存文件 + + jmap -dump:format=b,file=heap pid + +- 可视化的堆内存分析工具：JVisualVM、MAT等 + +#### 3. 排查指南 + +- 查看监控，以了解出现问题的时间点以及当前FGC的频率（可对比正常情况看频率是否正常） +- 了解该时间点之前有没有程序上线、基础组件升级等情况。 +- 了解JVM的参数设置，包括：堆空间各个区域的大小设置，新生代和老年代分别采用了哪些垃圾收集器，然后分析JVM参数设置是否合理。 +- 再对步骤1中列出的可能原因做排除法，其中元空间被打满、内存泄漏、代码显式调用gc方法比较容易排查。 +- 针对大对象或者长生命周期对象导致的FGC，可通过 [jmap ](http://mp.weixin.qq.com/s?__biz=MzI3ODcxMzQzMw==&mid=2247484637&idx=1&sn=9fc5de9be941b6b6eb6f36da09237d0c&chksm=eb5381ebdc2408fdaa5ff6224a2bad0c6f5fe60ae57718c808028fe1f537dd8855fea0d23757&scene=21#wechat_redirect)-histo 命令并结合dump堆内存文件作进一步分析，需要先定位到可疑对象。 +- 通过可疑对象定位到具体代码再次分析，这时候要结合GC原理和JVM参数设置，弄清楚可疑对象是否满足了进入到老年代的条件才能下结论。 + + + +# 网络 + +涉及到网络层面的问题一般都比较复杂，场景多，定位难，成为了大多数开发的噩梦，应该是最复杂的了。这里会举一些例子，并从tcp层、应用层以及工具的使用等方面进行阐述。 + +### 超时 + +超时错误大部分处在应用层面，所以这块着重理解概念。超时大体可以分为连接超时和读写超时，某些使用连接池的客户端框架还会存在获取连接超时和空闲连接清理超时。 + +- 读写超时。readTimeout/writeTimeout，有些框架叫做so_timeout或者socketTimeout，均指的是数据读写超时。注意这边的超时大部分是指逻辑上的超时。soa的超时指的也是读超时。读写超时一般都只针对客户端设置。 +- 连接超时。connectionTimeout，客户端通常指与服务端建立连接的最大时间。服务端这边connectionTimeout就有些五花八门了，jetty中表示空闲连接清理时间，tomcat则表示连接维持的最大时间。 +- 其他。包括连接获取超时connectionAcquireTimeout和空闲连接清理超时idleConnectionTimeout。多用于使用连接池或队列的客户端或服务端框架。 + +我们在设置各种超时时间中，需要确认的是尽量保持客户端的超时小于服务端的超时，以保证连接正常结束。 + +在实际开发中，我们关心最多的应该是接口的读写超时了。 + +如何设置合理的接口超时是一个问题。如果接口超时设置的过长，那么有可能会过多地占用服务端的tcp连接。而如果接口设置的过短，那么接口超时就会非常频繁。 + +服务端接口明明rt降低，但客户端仍然一直超时又是另一个问题。这个问题其实很简单，客户端到服务端的链路包括网络传输、排队以及服务处理等，每一个环节都可能是耗时的原因。 + +### TCP队列溢出 + +tcp队列溢出是个相对底层的错误，它可能会造成超时、rst等更表层的错误。因此错误也更隐蔽，所以我们单独说一说。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083827.jpg) + +如上图所示，这里有两个队列：syns queue(半连接队列）、accept queue（全连接队列）。三次握手，在server收到client的syn后，把消息放到syns queue，回复syn+ack给client，server收到client的ack，如果这时accept queue没满，那就从syns queue拿出暂存的信息放入accept queue中，否则按tcp_abort_on_overflow指示的执行。 + +tcp_abort_on_overflow 0表示如果三次握手第三步的时候accept queue满了那么server扔掉client发过来的ack。tcp_abort_on_overflow 1则表示第三步的时候如果全连接队列满了，server发送一个rst包给client，表示废掉这个握手过程和这个连接，意味着日志里可能会有很多`connection reset / connection reset by peer`。 + +那么在实际开发中，我们怎么能快速定位到tcp队列溢出呢？ + +**netstat命令，执行netstat -s | egrep "listen|LISTEN"** +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-83828.jpg) +如上图所示，overflowed表示全连接队列溢出的次数，sockets dropped表示半连接队列溢出的次数。 + +**ss命令，执行ss -lnt** +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083828.jpg) +上面看到Send-Q 表示第三列的listen端口上的全连接队列最大为5，第一列Recv-Q为全连接队列当前使用了多少。 + +接着我们看看怎么设置全连接、半连接队列大小吧： + +全连接队列的大小取决于min(backlog, somaxconn)。backlog是在socket创建的时候传入的，somaxconn是一个os级别的系统参数。而半连接队列的大小取决于max(64, /proc/sys/net/ipv4/tcp_max_syn_backlog)。 + +在日常开发中，我们往往使用servlet容器作为服务端，所以我们有时候也需要关注容器的连接队列大小。在tomcat中backlog叫做`acceptCount`，在jetty里面则是`acceptQueueSize`。 + +### RST异常 + +RST包表示连接重置，用于关闭一些无用的连接，通常表示异常关闭，区别于四次挥手。 + +在实际开发中，我们往往会看到`connection reset / connection reset by peer`错误，这种情况就是RST包导致的。 + +**端口不存在** + +如果像不存在的端口发出建立连接SYN请求，那么服务端发现自己并没有这个端口则会直接返回一个RST报文，用于中断连接。 + +**主动代替FIN终止连接** + +一般来说，正常的连接关闭都是需要通过FIN报文实现，然而我们也可以用RST报文来代替FIN，表示直接终止连接。实际开发中，可设置SO_LINGER数值来控制，这种往往是故意的，来跳过TIMED_WAIT，提供交互效率，不闲就慎用。 + +**客户端或服务端有一边发生了异常，该方向对端发送RST以告知关闭连接** + +我们上面讲的tcp队列溢出发送RST包其实也是属于这一种。这种往往是由于某些原因，一方无法再能正常处理请求连接了(比如程序崩了，队列满了)，从而告知另一方关闭连接。 + +**接收到的TCP报文不在已知的TCP连接内** + +比如，一方机器由于网络实在太差TCP报文失踪了，另一方关闭了该连接，然后过了许久收到了之前失踪的TCP报文，但由于对应的TCP连接已不存在，那么会直接发一个RST包以便开启新的连接。 + +**一方长期未收到另一方的确认报文，在一定时间或重传次数后发出RST报文** + +这种大多也和网络环境相关了，网络环境差可能会导致更多的RST报文。 + +之前说过RST报文多会导致程序报错，在一个已关闭的连接上读操作会报`connection reset`，而在一个已关闭的连接上写操作则会报`connection reset by peer`。通常我们可能还会看到`broken pipe`错误，这是管道层面的错误，表示对已关闭的管道进行读写，往往是在收到RST，报出`connection reset`错后继续读写数据报的错，这个在glibc源码注释中也有介绍。 + +我们在排查故障时候怎么确定有RST包的存在呢？当然是使用tcpdump命令进行抓包，并使用wireshark进行简单分析了。`tcpdump -i en0 tcp -w xxx.cap`，en0表示监听的网卡。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083829.jpg) + +接下来我们通过wireshark打开抓到的包，可能就能看到如下图所示，红色的就表示RST包了。 +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083830.jpg) + +### TIME_WAIT和CLOSE_WAIT + +TIME_WAIT和CLOSE_WAIT是啥意思相信大家都知道。 +在线上时，我们可以直接用命令`netstat -n | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'`来查看time-wait和close_wait的数量 + +用ss命令会更快`ss -ant | awk '{++S[$1]} END {for(a in S) print a, S[a]}'` + +![img](https://fredal-blog.oss-cn-hangzhou.aliyuncs.com/2019-11-04-083830.png) + +#### TIME_WAIT + +time_wait的存在一是为了丢失的数据包被后面连接复用，二是为了在2MSL的时间范围内正常关闭连接。它的存在其实会大大减少RST包的出现。 + +过多的time_wait在短连接频繁的场景比较容易出现。这种情况可以在服务端做一些内核参数调优: + +```java +#表示开启重用。允许将TIME-WAIT sockets重新用于新的TCP连接，默认为0，表示关闭 +net.ipv4.tcp_tw_reuse = 1 +#表示开启TCP连接中TIME-WAIT sockets的快速回收，默认为0，表示关闭 +net.ipv4.tcp_tw_recycle = 1 +``` + +当然我们不要忘记在NAT环境下因为时间戳错乱导致数据包被拒绝的坑了，另外的办法就是改小`tcp_max_tw_buckets`，超过这个数的time_wait都会被干掉，不过这也会导致报`time wait bucket table overflow`的错。 + +#### CLOSE_WAIT + +close_wait往往都是因为应用程序写的有问题，没有在ACK后再次发起FIN报文。close_wait出现的概率甚至比time_wait要更高，后果也更严重。往往是由于某个地方阻塞住了，没有正常关闭连接，从而渐渐地消耗完所有的线程。 + +想要定位这类问题，最好是通过jstack来分析线程堆栈来排查问题，具体可参考上述章节。这里仅举一个例子。 + +开发同学说应用上线后CLOSE_WAIT就一直增多，直到挂掉为止，jstack后找到比较可疑的堆栈是大部分线程都卡在了`countdownlatch.await`方法，找开发同学了解后得知使用了多线程但是确没有catch异常，修改后发现异常仅仅是最简单的升级sdk后常出现的`class not found`。 \ No newline at end of file diff --git a/docs/java/PriorityQueue.md b/docs/java/PriorityQueue.md new file mode 100644 index 0000000000..e69de29bb2 diff --git a/docs/java/README.md b/docs/java/README.md new file mode 100644 index 0000000000..14eba9632d --- /dev/null +++ b/docs/java/README.md @@ -0,0 +1,29 @@ +

+
+ + + + + +

+ + +## 欢迎加入我们 + +- 搜索公众号 **`JavaKeeper`** 或者 **`扫描右侧二维码`** 进行关注 +- 持续输出一个深耕在互联网的 Javaer 的所学和所见 + + + +> 以梦为马，越骑越傻。诗和远方，越走越慌。不忘初心是对的，但切记要出发，加油吧，程序员。 + +> 在路上的你，可以微信搜「 **JavaKeeper** 」一起前行，无套路领取 500+ 本电子书和 30+ 视频教学和源码，本文 **GitHub** [github.com/JavaKeeper](https://github.com/Jstarfish/JavaKeeper) 已经收录，服务端开发、面试必备技能兵器谱，有你想要的！ + + + +**Quick Learner, Earnest Curiosity, Self-driven, Get Things Done** + + + +![](https://img.starfish.ink/oceanus/end.jpg) + diff --git a/docs/java/Supplier.md b/docs/java/Supplier.md new file mode 100755 index 0000000000..b0000f177d --- /dev/null +++ b/docs/java/Supplier.md @@ -0,0 +1,223 @@ +## 函数式接口 + +### 1. 什么是函数式接口 + +- 只包含一个抽象方法的接口，称为函数式接口，该抽象方法也被称为函数方法。 我们熟知的Comparator和Runnable、Callable就属于函数式接口。 +- 这样的接口这么简单，都不值得在程序中定义，所以，JDK8在 `java.util.function` 中定义了几个标准的函数式接口，供我们使用。[Package java.util.function](https://docs.oracle.com/javase/8/docs/api/java/util/function/package-summary.html) +- 可以通过 Lambda 表达式来创建该接口的对象。（若 Lambda 表达式抛出一个受检异常，那么该异常需要在目标接口的抽象方法上进行声明）。 +- 我们可以在任意函数式接口上使用 **@FunctionalInterface** 注解， 这样做可以检查它是否是一个函数式接口，同时 javadoc 也会包含一条声明，说明这个接口是一个函数式接口。 + + + +### 2. 自定义函数式接口 + +```java +@FunctionalInterface //@FunctionalInterface标注该接口会被设计成一个函数式接口，否则会编译错误 +public interface MyFunc { + T getValue(T t); +} +public static String toUpperString(MyFunc myFunc, String str) { + return myFunc.getValue(str); +} + +public static void main(String[] args) { + String newStr = toUpperString((str) -> str.toUpperCase(), "abc"); + System.out.println(newStr); +} +``` + +作为参数传递 Lambda 表达式：为了将 Lambda 表达式作为参数传递，**接收Lambda 表达式的参数类型必须是与该 Lambda 表达式兼容的函数式接口的类型**。 + +函数接口为lambda表达式和方法引用提供目标类型 + +### 3. Java 内置四大核心函数式接口 + +| 函数式接口 | 参数类型 | 返回类型 | 用途 | +| ------------- | -------- | -------- | ------------------------------------------------------------ | +| Consumer | T | void | 对类型为T的对象应用操作，包含方法：void accept(T t) | +| Supplier | 无 | T | 返回类型为T的对象，包 含方法：T get(); | +| Function | T | R | 对类型为T的对象应用操作，并返回结果。结果是R类型的对象。包含方法：R apply(T t); | +| Predicate | T | boolean | 确定类型为T的对象是否满足某约束，并返回 boolean 值。包含方法 boolean test(T t); | + +```java +import org.junit.Test; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.function.Consumer; +import java.util.function.Function; +import java.util.function.Predicate; +import java.util.function.Supplier; + +/* + * Java8 内置的四大核心函数式接口 + * Consumer : 消费型接口 void accept(T t); + * Supplier : 供给型接口 T get(); + * Function : 函数型接口 R apply(T t); + * Predicate : 断言型接口 boolean test(T t); + */ +public class FunctionalInterfaceTest { + + //Predicate 断言型接口：将满足条件的字符串放入集合 + public List filterStr(List list, Predicate predicate) { + List newList = new ArrayList<>(); + for (String s : list) { + if (predicate.test(s)) { + newList.add(s); + } + } + return newList; + } + + @Test + public void testPredicate() { + List list = Arrays.asList("hello", "java8", "function", "predicate"); + List newList = filterStr(list, s -> s.length() > 5); + for (String s : newList) { + System.out.println(s); + } + } + + // Function 函数型接口：处理字符串 + public String strHandler(String str, Function function) { + return function.apply(str); + } + + @Test + public void testFunction() { + String str1 = strHandler("测试内置函数式接口", s -> s.substring(2)); + System.out.println(str1); + + String str2 = strHandler("abcdefg", s -> s.toUpperCase()); + System.out.println(str2); + } + + //Supplier 供给型接口 :产生指定个数的整数，并放入集合 + public List getNumList(int num, Supplier supplier) { + List list = new ArrayList<>(); + for (int i = 0; i < num; i++) { + Integer n = supplier.get(); + list.add(n); + } + return list; + } + + @Test + public void testSupplier() { + List numList = getNumList(10, () -> (int) (Math.random() * 100)); + + for (Integer num : numList) { + System.out.println(num); + } + } + + //Consumer 消费型接口 :修改参数 + public void modifyValue(Integer value, Consumer consumer) { + consumer.accept(value); + } + + @Test + public void testConsumer() { + modifyValue(3, s -> System.out.println(s * 3)); + } +} +``` + +**Package java.util.function** 包下还提供了很多其他的演变方法。 + +![java8-function.png](https://i.loli.net/2019/12/31/tzNWejl7gdnvSrK.png) + +**?> Tip** + +Java类型要么是引用类型（Byte、Integer、Objuct、List），要么是原始类型（int、double、byte、char）。但是泛型只能绑定到引用类型。将原始类型转换为对应的引用类型，叫**装箱**，相反，将引用类型转换为对应的原始类型，叫**拆箱**。当然Java提供了自动装箱机制帮我们执行了这一操作。 + +```java +List list = new ArrayList(); + for (int i = 0; i < 10; i++) { + list.add(i); //int被装箱为Integer +} +``` + +但这在性能方面是要付出代价的。装箱后的值本质上就是把原始类型包裹起来，并保存在堆里。因此，装箱后的值需要更多的内存，并需要额外的内存搜索来获取被包裹的原始值。 + +以上funciton包中的**IntPredicate、DoubleConsumer、LongBinaryOperator、ToDoubleFuncation等就是避免自动装箱的操作**。一般，针对专门的输入参数类型的函数式接口的名称都要加上对应的原始类型前缀。 + + + +## 函数式接口有必要吗 + +方法也可以当做参数传递， + +有必要存在，这样就可以方便传函数对象了 + +函数式接口 可以更方便的达到设计模式中模板方法模式一样的效果，即让模板方法在不改变算法整体结构的情况下，重新定义算法中的某些步骤 + +lambda和函数式接口是用来配合简化写 Java **异步回调**的。包括 Java 内置的最常见的函数式接口，Runable、Callable 也是用来写异步/多线程的，不写多线程，就没必要写成函数式。 + + + +Java8 以前，guava 其实就已经提供了函数式接口(guava-18) + +```java +package com.google.common.base; + +import com.google.common.annotations.GwtCompatible; +import javax.annotation.Nullable; + +@GwtCompatible +public interface Function { + @Nullable + T apply(@Nullable F var1); + + boolean equals(@Nullable Object var1); +} +``` + +Java 8 出现后，guava 的函数式接口开始就继承 Java 了 + +```java +package com.google.common.base; + +import com.google.common.annotations.GwtCompatible; +import com.google.errorprone.annotations.CanIgnoreReturnValue; +import javax.annotation.Nullable; + +@FunctionalInterface +@GwtCompatible +public interface Function extends java.util.function.Function { + @Nullable + @CanIgnoreReturnValue + T apply(@Nullable F var1); + + boolean equals(@Nullable Object var1); +} +``` + + + +最常用的可能有 Supplier + +Guava 还提供了 Suppliers + +1. Suppliers.ofInstance(T instance) 方法 + + 该种调用方式为最简单的实现方式，在调用`get`方法时会直接将传递的instance返回。 + +2. Suppliers.memoize(Supplier delegate)方法 + + 该方法返回的`Supplier`会通过传递的`delegate`参数来获取值并缓存，之后再次调用`get`方法时会直接将缓存中的值返回。 + +3. Suppliers.memoizeWithExpiration(Supplier delegate, long duration, TimeUnit unit)方法 + + 该方法会根据指定的duration 和 unit每隔指定时间调用delegate的get方法一次，相当于是定时刷新结果值。 + +4. Suppliers.synchronizedSupplier(Supplier delegate)方法 + + 该方法会以`synchronized`的方式实现线程安全的调用，并且每次都会执行delegate的get方法来获取值。 + +5. Suppliers.compose(Function function, Supplier delegate)方法 + + 每次获取值都会调用`delegate`的`get`方法，并且使用传递的`function`对返回值进行处理。 + + + diff --git "a/docs/java/other/DBeaver\344\270\200\346\254\276\345\205\215\350\264\271\345\274\200\346\272\220\347\232\204\351\200\232\347\224\250\346\225\260\346\215\256\345\272\223\345\267\245\345\205\267.md" "b/docs/java/other/DBeaver\344\270\200\346\254\276\345\205\215\350\264\271\345\274\200\346\272\220\347\232\204\351\200\232\347\224\250\346\225\260\346\215\256\345\272\223\345\267\245\345\205\267.md" new file mode 100644 index 0000000000..f7f07392b7 --- /dev/null +++ "b/docs/java/other/DBeaver\344\270\200\346\254\276\345\205\215\350\264\271\345\274\200\346\272\220\347\232\204\351\200\232\347\224\250\346\225\260\346\215\256\345\272\223\345\267\245\345\205\267.md" @@ -0,0 +1,68 @@ +DBeaver 是一个基于 Java 开发，免费开源的通用数据库管理和开发工具，使用非常友好的 [ASL](https://dbeaver.io/files/dbeaver_license.txt) 协议。可以通过[官方网站](https://dbeaver.io/)或者 [Github](https://github.com/dbeaver/dbeaver/) 进行下载。 + +由于 DBeaver 基于 Java 开发，可以运行在各种操作系统上，包括：Windows、Linux、macOS 等。DBeaver 采用 Eclipse 框架开发，支持插件扩展，并且提供了许多数据库管理工具：ER 图、数据导入/导出、数据库比较、模拟数据生成等。 + +DBeaver 通过 JDBC 连接到数据库，可以支持几乎所有的数据库产品，包括：MySQL、PostgreSQL、MariaDB、SQLite、Oracle、Db2、SQL Server、Sybase、MS Access、Teradata、Firebird、Derby 等等。[商业版本](https://dbeaver.com/)更是可以支持各种 NoSQL 和大数据平台：MongoDB、InfluxDB、Apache Cassandra、Redis、Apache Hive 等。 + +### 下载与安装 + +DBeaver 社区版可以通过[官方网站](https://dbeaver.io/download/)或者 [Github](https://github.com/dbeaver/dbeaver/releases) 进行下载。两者都为不同的操作系统提供了安装包或者解压版，可以选择是否需要同时安装 JRE。另外，官方网站还提供了 DBeaver 的 Eclipse 插件，可以在 Eclipse 中进行集成。 + +DBeaver 支持中文，安装过程非常简单，不多说，唯一需要注意的是 DBeaver 的运行依赖于 JRE。不出意外，安装完成后运行安装目录下的 dbeaver.exe 可以看到以下界面（Windows 10）： + +![new_connection](https://img-blog.csdnimg.cn/2019043017554476.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hvcnNlcw==,size_16,color_FFFFFF,t_70#pic_center) +这个界面其实是新建数据库连接，我们可以看到它支持的各种数据平台；先点击“**取消**”按钮，进入主窗口界面。 + +此时，它会提示我们是否建立一个示例数据库。 + +![create_database](https://img-blog.csdnimg.cn/20190430180321861.JPG#pic_center) +如果点击“是(Y)”，它会创建一个默认的 SQLite 示例数据库。下图是它的主窗口界面。 + +![DBeaver](https://img-blog.csdnimg.cn/20190430202853429.JPG?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hvcnNlcw==,size_16,color_FFFFFF,t_70) +DBeaver 和我们常用的软件类似，最上面是菜单项和快捷工具，左侧是已经建立的数据库连接和项目信息，右侧是主要的工作区域。 + +### 连接数据库 + +打开 DBeaver 之后，首先要做的就是创建数据库连接。可以通过菜单“**数据库**” -> “**新建连接**”打开新建连接向导窗口，也就是我们初次运行 DBeaver 时弹出的窗口。 + +![在这里插入图片描述](https://img-blog.csdnimg.cn/20190430204330601.JPG?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hvcnNlcw==,size_16,color_FFFFFF,t_70#pic_center) +我们以 PostgreSQL 为例，新建一个数据库连接。选择 PostgreSQL 图标，点击“**下一步(N)**”。 + +![connection](https://img-blog.csdnimg.cn/20190430205252763.JPG?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hvcnNlcw==,size_16,color_FFFFFF,t_70#pic_center) +然后是设置数据库的连接信息：主机、端口、数据库、用户、密码。“**Advanced settings**”高级设置选项可以配置 SSH、SSL 以及代理等，也可以为连接指定自己的名称和连接类型（开发、测试、生产）。 + +点击最下面的“**测试链接(T)**”可以测试连接配置的正确性。初次创建某种数据库的连接时，会提示下载相应的 JDBC 驱动。 + +![driver](https://img-blog.csdnimg.cn/20190430210119221.JPG?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hvcnNlcw==,size_16,color_FFFFFF,t_70#pic_center) +它已经为我们查找到了相应的驱动，只需要点击“**下载**”即可，非常方便。下载完成后，如果连接信息正确，可以看到连接成功的提示。 + +![success](https://img-blog.csdnimg.cn/20190430210536891.JPG#pic_center) +确认后完成连接配置即可。左侧的数据库导航中会增加一个新的数据库连接。 + +由于某些数据库（例如 Oracle、Db2）的 JDBC 驱动需要登录后才能下载，因此可以使用手动的方式进行配置。选择菜单“**数据库**” -> “**驱动管理器**”。 + +![在这里插入图片描述](https://img-blog.csdnimg.cn/20190430211557737.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hvcnNlcw==,size_16,color_FFFFFF,t_70#pic_center) +选择 Oracle ，点击“**编辑(E)…**”按钮。 + +![在这里插入图片描述](https://img-blog.csdnimg.cn/20190430211734232.JPG?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hvcnNlcw==,size_16,color_FFFFFF,t_70#pic_center) +通过界面提示的网址，手动下载 Oracle 数据库的 JDBC 驱动文件，例如 ojdbc8.jar。然后点击“**添加文件(F)**”按钮，选择并添加该文件。 + +![在这里插入图片描述](https://img-blog.csdnimg.cn/20190430212156376.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hvcnNlcw==,size_16,color_FFFFFF,t_70#pic_center) +下次建立 Oracle 数据库连接时即可使用该驱动。 + +新建连接之后，就可以通过这些连接访问相应的数据库，查看和编辑数据库中的对象，执行 SQL 语句，完成各种管理和开发工作。 +![在这里插入图片描述](https://img-blog.csdnimg.cn/20190430212552322.JPG?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hvcnNlcw==,size_16,color_FFFFFF,t_70) + +### 生成 ER 图 + +最后介绍一下如何生成数据库对象的 ER 图。点击窗口左侧“**数据库导航**”旁边的“**项目**”视图。 + +![在这里插入图片描述](https://img-blog.csdnimg.cn/20190430214154882.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hvcnNlcw==,size_16,color_FFFFFF,t_70#pic_center) +其中有个“**ER Diagrams**”，就是实体关系图。右击该选项，点击“**创建新的 ER 图**”。 + +![在这里插入图片描述](https://img-blog.csdnimg.cn/20190430214817578.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hvcnNlcw==,size_16,color_FFFFFF,t_70#pic_center) +输入一个名称并选择数据库连接和需要展示的对象，然后点击“**完成**”，即可生成相应的 ER 图。 +![在这里插入图片描述](https://img-blog.csdnimg.cn/20190430215113385.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L2hvcnNlcw==,size_16,color_FFFFFF,t_70) +ER 图可以进行排版和显示设置，也支持打印为图片。DBeaver 目前还不支持自己创建 ER 图，只能从现有的数据库中生成。 + +对于图形工具，很多功能我们都可以自己去使用体会；当然，DBeaver 也提供了[用户指南](https://github.com/dbeaver/dbeaver/wiki)，自行参考。 \ No newline at end of file diff --git a/docs/tools/Git-Specification.md b/docs/java/other/Git-Specification.md similarity index 100% rename from docs/tools/Git-Specification.md rename to docs/java/other/Git-Specification.md diff --git a/docs/tools/IDEA.md b/docs/java/other/IDEA.md similarity index 100% rename from docs/tools/IDEA.md rename to docs/java/other/IDEA.md diff --git a/docs/java/other/Java-XML.md b/docs/java/other/Java-XML.md new file mode 100755 index 0000000000..ea087d406a --- /dev/null +++ b/docs/java/other/Java-XML.md @@ -0,0 +1,67 @@ +# XML 解析方式对比 + +https://blog.csdn.net/wuwenxiang91322/article/details/10230363 + + + + + +## DOM + +DOM的全称是Document ObjectModel，也即文档对象模型。在应用程序中，基于DOM的XML分析器将一个XML文档转换成一个对象模型的集合（通常称DOM树），应用程序正是通过对这个对象模型的操作，来实现对XML文档数据的操作。通过DOM接口，应用程序可以在任何时候访问XML文档中的任何一部分数据，因此，这种利用DOM接口的机制也被称作随机访问机制。 + +DOM接口提供了一种通过分层对象模型来访问XML文档信息的方式，这些分层对象模型依据XML的文档结构形成了一棵节点树。无论XML文档中所描述的是什么类型的信息，即便是制表数据、项目列表或一个文档，利用DOM所生成的模型都是节点树的形式。也就是说，DOM强制使用树模型来访问XML文档中的信息。由于XML本质上就是一种分层结构，所以这种描述方法是相当有效的。 + +DOM树所提供的随机访问方式给应用程序的开发带来了很大的灵活性，它可以任意地控制整个XML文档中的内容。然而，由于DOM分析器把整个XML文档转化成DOM树放在了内存中，因此，当文档比较大或者结构比较复杂时，对内存的需求就比较高。而且，对于结构复杂的树的遍历也是一项耗时的操作。所以，DOM分析器对机器性能的要求比较高，实现效率不十分理想。不过，由于DOM分析器所采用的树结构的思想与XML文档的结构相吻合，同时鉴于随机访问所带来的方便，因此，DOM分析器还是有很广泛的使用价值的。 + + + +## SAX + +SAX的全称是Simple APIs for XML，也即XML简单应用程序接口。与DOM不同，SAX提供的访问模式是一种顺序模式，这是一种快速读写XML数据的方式。当使用SAX分析器对XML文档进行分析时，会触发一系列事件，并激活相应的事件处理函数，应用程序通过这些事件处理函数实现对XML文档的访问，因而SAX接口也被称作事件驱动接口。 + +​ SAX 解析器采用了基于事件的模型，它在解析 XML 文档的时候可以触发一系列的事件，当发现给定的tag的时候，它可以激活一个回调方法，告诉该方法制定的标签已经找到。SAX 对内存的要求通常会比较低，因为它让开发人员自己来决定所要处理的tag。特别是当开发人员只需要处理文档中所包含的部分数据时，SAX 这种扩展能力得到了更好的体现。但用 SAX 解析器的时候编码工作会比较困难，而且很难同时访问同一个文档中的多处不同数据 + + + +## JDOM + +JDOM是一个开源项目，它基于树型结构，利用纯JAVA的技术对XML文档实现解析、生成、序列化以及多种操作。（http://jdom.org） + +•JDOM 直接为JAVA编程服务。它利用更为强有力的JAVA语言的诸多特性（方法重载、集合概念等），把SAX和DOM的功能有效地结合起来。 + +•JDOM是用Java语言读、写、操作XML的新API函数。在直接、简单和高效的前提下，这些API函数被最大限度的优化。 + +　JDOM 与 DOM 主要有两方面不同。首先，JDOM 仅使用具体类而不使用接口。这在某些方面简化了 API，但是也限制了灵活性。第二，API 大量使用了 Collections 类，简化了那些已经熟悉这些类的 Java 开发者的使用。 +　　JDOM 文档声明其目的是“使用 20%（或更少）的精力解决 80%（或更多）Java/XML 问题”（根据学习曲线假定为 20%）。JDOM 对于大多数 Java/XML 应用程序来说当然是有用的，并且大多数开发者发现 API 比 DOM 容易理解得多。JDOM 还包括对程序行为的相当广泛检查以防止用户做任何在 XML 中无意义的事。然而，它仍需要您充分理解 XML 以便做一些超出基本的工作（或者甚至理解某些情况下的错误）。这也许是比学习 DOM 或 JDOM 接口都更有意义的工作。 +　　JDOM 自身不包含解析器。它通常使用 SAX2 解析器来解析和验证输入 XML 文档（尽管它还可以将以前构造的 DOM 表示作为输入）。它包含一些转换器以将 JDOM 表示输出成 SAX2 事件流、DOM 模型或 XML 文本文档。JDOM 是在 Apache 许可证变体下发布的开放源码。 + + + +## Dom4j + +官网：DOM4Jhttp://dom4j.sourceforge.net/ + + 虽然 DOM4J 代表了完全独立的开发结果，但最初，它是 JDOM 的一种智能分支。它合并了许多超出基本 XML 文档表示的功能，包括集成的 XPath 支持、XML Schema 支持以及用于大文档或流化文档的基于事件的处理。它还提供了构建文档表示的选项，它通过 DOM4J API 和标准 DOM 接口具有并行访问功能。从 2000 下半年开始，它就一直处于开发之中。 + +　　为支持所有这些功能，DOM4J 使用接口和抽象基本类方法。DOM4J 大量使用了 API 中的 Collections 类，但是在许多情况下，它还提供一些替代方法以允许更好的性能或更直接的编码方法。直接好处是，虽然 DOM4J 付出了更复杂的 API 的代价，但是它提供了比 JDOM 大得多的灵活性。 + +　　在添加灵活性、XPath 集成和对大文档处理的目标时，DOM4J 的目标与 JDOM 是一样的：针对 Java 开发者的易用性和直观操作。它还致力于成为比 JDOM 更完整的解决方案，实现在本质上处理所有 Java/XML 问题的目标。在完成该目标时，它比 JDOM 更少强调防止不正确的应用程序行为。 + +　　DOM4J 是一个非常非常优秀的Java XML API，具有性能优异、功能强大和极端易用使用的特点，同时它也是一个开放源代码的软件。如今你可以看到越来越多的 Java 软件都在使用 DOM4J 来读写 XML，特别值得一提的是连 Sun 的 JAXM 也在用 DOM4J。 + +**Dom解析：** + +​ 在内存中创建一个DOM树，该结构通常需要加载整个文档然后才能做工作。由于它是基于信息层次的，因而DOM被认为是基于树或基于对象的，树在内存中是持久的，因此可以修改它以便应用程序能对数据和结构作出更改能随机访问文件内容，也可以修改原文件内容. + +**SAX解析 ：** + +​ SAX处理的优点非常类似于流媒体的优点。分析能够立即开始，而不是等待所有的数据被处理。SAX解析器采用了基于事件的模型，它在解析XML文档的时候可以触发一系列的事件，当发现给定的tag的时候，它可以激活一个回调方法，告诉该方法制定的标签已经找到。而且，由于应用程序只是在读取数据时检查数据，因此不需要将数据存储在内存中。这对于大型文档来说是个巨大的优点线性解析，不能随机访问，也无法修改原文件 + +**JDOM解析：** + +​ JDOM的目的是成为Java特定文档模型,它简化与XML的交互并且比使用DOM实现更快.JDOM仅使用具体类而不使用接口。这在某些方面简化了API，但是也限制了灵活性。第二，API大量使用了Collections类，简化了那些已经熟悉这些类的Java开发者的使用。 + + **DOM4j：** + +​ 解析 DOM4J使用接口和抽象基本类方法。DOM4J大量使用了API中的Collections类，但是在许多情况下，它还提供一些替代方法以允许更好的性能或更直接的编码方法。直接好处是，虽然DOM4J付出了更复杂的API的代价，但是它提供了比JDOM大得多的灵活性。 \ No newline at end of file diff --git a/docs/tools/Maven.md b/docs/java/other/Maven.md similarity index 100% rename from docs/tools/Maven.md rename to docs/java/other/Maven.md diff --git "a/docs/java/other/Redis\351\233\206\347\276\244\345\214\226\346\226\271\346\241\210\345\257\271\346\257\224\357\274\232Codis\343\200\201Twemproxy\343\200\201Redis Cluster.md" "b/docs/java/other/Redis\351\233\206\347\276\244\345\214\226\346\226\271\346\241\210\345\257\271\346\257\224\357\274\232Codis\343\200\201Twemproxy\343\200\201Redis Cluster.md" new file mode 100644 index 0000000000..0ad4ab9d68 --- /dev/null +++ "b/docs/java/other/Redis\351\233\206\347\276\244\345\214\226\346\226\271\346\241\210\345\257\271\346\257\224\357\274\232Codis\343\200\201Twemproxy\343\200\201Redis Cluster.md" @@ -0,0 +1,179 @@ +之前我们提到，为了保证Redis的高可用，主要需要以下几个方面： + +- 数据持久化 +- 主从复制 +- 自动故障恢复 +- 集群化 + +我们简单理一下这几个方案的特点，以及它们之间的联系。 + +数据持久化本质上是为了做数据备份，有了数据持久化，当Redis宕机时，我们可以把数据从磁盘上恢复回来，但在数据恢复之前，服务是不可用的，而且数据恢复的时间取决于实例的大小，数据量越大，恢复起来越慢。Redis的持久化过程可以参考[Redis持久化是如何做的？RDB和AOF对比分析](http://kaito-kidd.com/2020/06/29/redis-persistence-rdb-aof/)。 + +而主从复制则是部署多个副本节点，多个副本节点实时复制主节点的数据，当主节点宕机时，我们有完整的副本节点可以使用。另一方面，如果我们业务的读请求量很大，主节点无法承受所有的读请求，多个副本节点可以分担读请求，实现读写分离，这样可以提高Redis的访问性能。Redis主从复制的原理可以参考[Redis的主从复制是如何做的？复制过程中也会产生各种问题？](http://kaito-kidd.com/2020/06/30/redis-replication/)。 + +但有个问题是，当主节点宕机时，我们虽然有完整的副本节点，但需要手动操作把从节点提升为主节点继续提供服务，如果每次主节点故障，都需要人工操作，这个过程既耗时耗力，也无法保证及时性，高可用的程度将大打折扣。如何优化呢？ + + + +此时我们就需要有自动故障恢复机制，当主节点故障时，可以自动把从节点提上来，这个过程是完全自动化的，无需人工干预，这样才能最大程度保证服务的可用性，降低不可用时间。Redis的故障自动恢复是通过哨兵实现的，具体的故障恢复原理，可以参考[Redis如何实现故障自动恢复？浅析哨兵的工作原理](http://kaito-kidd.com/2020/07/02/redis-sentinel/)。 + +有了**数据持久化、主从复制、故障自动恢复**这些功能，我们在使用Redis时是不是就可以高枕无忧了？ + +答案是否定的，如果我们的业务大部分都是读请求，可以使用读写分离提升性能。但如果**写请求量**也很大呢？现在是大数据时代，像阿里、腾讯这些大体量的公司，每时每刻都拥有非常大的写入量，此时如果只有一个主节点是无法承受的，那如何处理呢？ + +这就需要**集群化**！简单来说实现方式就是，**多个主从节点构成一个集群，每个节点存储一部分数据，这样写请求也可以分散到多个主节点上，解决写压力大的问题。同时，集群化可以在节点容量不足和性能不够时，动态增加新的节点，对进群进行扩容，提升性能。** + +从这篇文章开始，我们就开始介绍Redis的集群化方案。当然，集群化也意味着Redis部署架构更复杂，管理和维护起来成本也更高。而且在使用过程中，也会遇到很多问题，这也衍生出了不同的集群化解决方案，它们的侧重点各不相同。 + +这篇文章我们先来整体介绍一下Redis集群化比较流行的几个解决方案，先对它们有整体的认识，后面我会专门针对我比较熟悉的集群方案进行详细的分析。 + +# 集群化方案 + +要想实现集群化，就必须部署多个主节点，每个主节点还有可能有多个从节点，以这样的部署结构组成的集群，才能更好地承担更大的流量请求和存储更多的数据。 + +可以承担更大的流量是集群最基础的功能，一般集群化方案还包括了上面提到了数据持久化、数据复制、故障自动恢复功能，利用这些技术，来保证集群的高性能和高可用。 + +另外，优秀的集群化方案还实现了**在线水平扩容**功能，当节点数量不够时，可以动态增加新的节点来提升整个集群的性能，而且这个过程是在线完成的，业务无感知。 + +业界主流的Redis集群化方案主要包括以下几个： + +- 客户端分片 +- Codis +- Twemproxy +- Redis Cluster + +它们还可以用**是否中心化**来划分，其中**客户端分片、Redis Cluster属于无中心化的集群方案，Codis、Tweproxy属于中心化的集群方案。** + +是否中心化是指客户端访问多个Redis节点时，是**直接访问**还是通过一个**中间层Proxy**来进行操作，直接访问的就属于无中心化的方案，通过中间层Proxy访问的就属于中心化的方案，它们有各自的优劣，下面分别来介绍。 + +# 客户端分片 + +客户端分片主要是说，我们只需要部署多个Redis节点，具体如何使用这些节点，主要工作在客户端。 + +客户端通过固定的Hash算法，针对不同的key计算对应的Hash值，然后对不同的Redis节点进行读写。 + +[![客户端分片集群模式](https://kaito-blog-1253469779.cos.ap-beijing.myqcloud.com/2020/07/15941324303742.jpg)](https://kaito-blog-1253469779.cos.ap-beijing.myqcloud.com/2020/07/15941324303742.jpg) + +[客户端分片集群模式](https://kaito-blog-1253469779.cos.ap-beijing.myqcloud.com/2020/07/15941324303742.jpg) + + + +客户端分片需要业务开发人员事先评估业务的**请求量和数据量**，然后让DBA部署足够的节点交给开发人员使用即可。 + +这个方案的优点是部署非常方便，业务需要多少个节点DBA直接部署交付即可，剩下的事情就需要业务开发人员根据节点数量来编写key的**请求路由逻辑**，制定一个规则，一般采用固定的Hash算法，把不同的key写入到不同的节点上，然后再根据这个规则进行数据读取。 + +可见，它的缺点是业务开发人员**使用Redis的成本较高**，需要编写路由规则的代码来使用多个节点，而且如果事先对业务的数据量评估不准确，后期的**扩容和迁移成本非常高**，因为节点数量发生变更后，Hash算法对应的节点也就不再是之前的节点了。 + +所以后来又衍生出了**一致性哈希算法**，就是为了解决当节点数量变更时，尽量减少数据的迁移和性能问题。 + +这种客户端分片的方案一般用于业务数据量比较稳定，后期不会有大幅度增长的业务场景下使用，只需要前期评估好业务数据量即可。 + +# Codis + +随着业务和技术的发展，人们越发觉得，当我需要使用Redis时，我们**不想关心集群后面有多少个节点**，我们希望我们使用的Redis是一个大集群，当我们的业务量增加时，这个大集群可以**增加新的节点来解决容量不够用和性能问题**。 + +这种方式就是**服务端分片方案**，客户端不需要关心集群后面有多少个Redis节点，只需要像使用一个Redis的方式去操作这个集群，这种方案将大大降低开发人员的使用成本，开发人员可以只需要关注业务逻辑即可，不需要关心Redis的资源问题。 + +多个节点组成的集群，如何让开发人员像操作一个Redis时那样来使用呢？这就涉及到多个节点是如何组织起来提供服务的，一般我们会在客户端和服务端中间增加一个**代理层**，客户端只需要操作这个代理层，代理层实现了具体的请求转发规则，然后转发请求到后面的多个节点上，因此这种方式也叫做**中心化**方式的集群方案，[Codis](https://github.com/CodisLabs/codis)就是以这种方式实现的集群化方案。 + +[![Proxy集群模式](https://kaito-blog-1253469779.cos.ap-beijing.myqcloud.com/2020/07/15941324303751.jpg)](https://kaito-blog-1253469779.cos.ap-beijing.myqcloud.com/2020/07/15941324303751.jpg) + +[Proxy集群模式](https://kaito-blog-1253469779.cos.ap-beijing.myqcloud.com/2020/07/15941324303751.jpg) + + + +[![Codis架构图](https://kaito-blog-1253469779.cos.ap-beijing.myqcloud.com/2020/07/15941324303756.png)](https://kaito-blog-1253469779.cos.ap-beijing.myqcloud.com/2020/07/15941324303756.png) + +[Codis架构图](https://kaito-blog-1253469779.cos.ap-beijing.myqcloud.com/2020/07/15941324303756.png) + + + +Codis是由国人前豌豆荚大神开发的，采用中心化方式的集群方案。因为需要代理层Proxy来进行所有请求的转发，所以对Proxy的性能要求很高，Codis采用Go语言开发，兼容了开发效率和性能。 + +Codis包含了多个组件： + +- codis-proxy：主要负责对请求的读写进行转发 +- codis-dashbaord：统一的控制中心，整合了数据转发规则、故障自动恢复、数据在线迁移、节点扩容缩容、自动化运维API等功能 +- codis-group：基于Redis 3.2.8版本二次开发的Redis Server，增加了异步数据迁移功能 +- codis-fe：管理多个集群的UI界面 + +可见Codis的组件还是挺多的，它的功能非常全，**除了请求转发功能之外，还实现了在线数据迁移、节点扩容缩容、故障自动恢复等功能**。 + +Codis的Proxy就是负责请求转发的组件，它内部维护了请求转发的具体规则，Codis把整个集群划分为1024个槽位，在处理读写请求时，采用`crc32`Hash算法计算key的Hash值，然后再根据Hash值对1024个槽位取模，最终找到具体的Redis节点。 + +Codis最大的特点就是可以在线扩容，在扩容期间不影响客户端的访问，也就是不需要停机。这对业务使用方是极大的便利，当集群性能不够时，就可以动态增加节点来提升集群的性能。 + +为了实现在线扩容，保证数据在迁移过程中还有可靠的性能，Codis针对Redis进行了修改，增加了针对**异步迁移数据**相关命令，它基于Redis 3.2.8进行开发，上层配合Dashboard和Proxy组件，完成对业务无损的数据迁移和扩容功能。 + +因此，要想使用Codis，必须使用它内置的Redis，这也就意味着Codis中的Redis是否能跟上官方最新版的功能特性，可能无法得到保障，这取决于Codis的维护方，目前Codis已经不再维护，所以使用Codis时只能使用3.2.8版的Redis，这是一个痛点。 + +另外，由于集群化都需要部署多个节点，因此操作集群并不能完全像操作单个Redis一样实现所有功能，主要是对于**操作多个节点可能产生问题的命令进行了禁用或限制**，具体可参考[Codis不支持的命令列表](https://github.com/CodisLabs/codis/blob/release3.2/doc/unsupported_cmds.md)。 + +但这不影响它是一个优秀的集群化方案，由于我司使用Redis集群方案较早，那时Redis Cluster还不够成熟，所以我司使用的Redis集群方案就是Codis。目前我的工作主要是围绕Codis展开的，我们公司对Codis进行了定制开发，还对Redis进行了一些改造，让Codis支持了跨多个数据中心的数据同步，因此我对Codis的代码比较熟悉，后面会专门写一些文章来剖析Codis的实现原理，学习它的原理，这对我们理解分布式存储有很大的帮助！ + +# Twemproxy + +[Twemproxy](https://github.com/twitter/twemproxy)是由Twitter开源的集群化方案，它既可以做Redis Proxy，还可以做Memcached Proxy。 + +它的功能比较单一，只实现了请求路由转发，没有像Codis那么全面有在线扩容的功能，它解决的重点就是把客户端分片的逻辑统一放到了Proxy层而已，其他功能没有做任何处理。 + +[![Twemproxy架构图](https://kaito-blog-1253469779.cos.ap-beijing.myqcloud.com/2020/07/15941324303761.jpg)](https://kaito-blog-1253469779.cos.ap-beijing.myqcloud.com/2020/07/15941324303761.jpg) + +[Twemproxy架构图](https://kaito-blog-1253469779.cos.ap-beijing.myqcloud.com/2020/07/15941324303761.jpg) + + + +Tweproxy推出的时间最久，在早期没有好的服务端分片集群方案时，应用范围很广，而且性能也极其稳定。 + +但它的痛点就是**无法在线扩容、缩容**，这就导致运维非常不方便，而且也没有友好的运维UI可以使用。Codis就是因为在这种背景下才衍生出来的。 + +# Redis Cluster + +采用中间加一层Proxy的中心化模式时，这就对Proxy的要求很高，因为它一旦出现故障，那么操作这个Proxy的所有客户端都无法处理，要想实现Proxy的高可用，还需要另外的机制来实现，例如Keepalive。 + +而且增加一层Proxy进行转发，必然会有一定的**性能损耗**，那么除了客户端分片和上面提到的中心化的方案之外，还有比较好的解决方案么？ + +Redis官方推出的Redis Cluster另辟蹊径，它没有采用中心化模式的Proxy方案，而是把请求转发逻辑一部分放在客户端，一部分放在了服务端，它们之间互相配合完成请求的处理。 + +Redis Cluster是在Redis 3.0推出的，早起的Redis Cluster由于没有经过严格的测试和生产验证，所以并没有广泛推广开来。也正是在这样的背景下，业界衍生了出了上面所说的中心化集群方案：Codis和Tweproxy。 + +但随着Redis的版本迭代，Redis官方的Cluster也越来越稳定，更多人开始采用官方的集群化方案。也正是因为它是官方推出的，所以它的持续维护性可以得到保障，这就比那些第三方的开源方案更有优势。 + +Redis Cluster没有了中间的Proxy代理层，那么是如何进行请求的转发呢？ + +Redis把请求转发的逻辑放在了Smart Client中，要想使用Redis Cluster，必须升级Client SDK，这个SDK中内置了请求转发的逻辑，所以业务开发人员同样不需要自己编写转发规则，Redis Cluster采用16384个槽位进行路由规则的转发。 + +[![Redis Cluster](https://kaito-blog-1253469779.cos.ap-beijing.myqcloud.com/2020/07/15941324303764.jpg)](https://kaito-blog-1253469779.cos.ap-beijing.myqcloud.com/2020/07/15941324303764.jpg) + +[Redis Cluster](https://kaito-blog-1253469779.cos.ap-beijing.myqcloud.com/2020/07/15941324303764.jpg) + + + +没有了Proxy层进行转发，客户端可以直接操作对应的Redis节点，这样就少了Proxy层转发的性能损耗。 + +Redis Cluster也提供了**在线数据迁移、节点扩容缩容**等功能，内部还**内置了哨兵完成故障自动恢复功能**，可见它是一个集成所有功能于一体的Cluster。因此它在部署时非常简单，不需要部署过多的组件，对于运维极其友好。 + +Redis Cluster在节点数据迁移、扩容缩容时，对于客户端的请求处理也做了相应的处理。当客户端访问的数据正好在迁移过程中时，服务端与客户端制定了一些协议，来告知客户端去正确的节点上访问，帮助客户端订正自己的路由规则。 + +虽然Redis Cluster提供了在线数据迁移的功能，但它的迁移性能并不高，迁移过程中遇到大key时还有可能长时间阻塞迁移的两个节点，这个功能相较于Codis来说，Codis数据迁移性能更好。这里先了解一个大概就好，后面我会专门针对Codis和Redis Cluster在线迁移功能的性能对比写一些文章。 + +现在越来越多的公司开始采用Redis Cluster，有能力的公司还在它的基础上进行了二次开发和定制，来解决Redis Cluster存在的一些问题，我们期待Redis Cluster未来有更好的发展。 + +# 总结 + +比较完了这些集群化方案，下面我们来总结一下。 + +| # | 客户端分片 | Codis | Tweproxy | Redis Cluster | +| ------------------ | --------------------------------- | --------------------- | ----------------------- | --------------------------------------------------- | +| 集群模式 | 无中心化 | 中心化 | 中心化 | 无中心化 | +| 使用方式 | 客户端编写路由规则代码，直连Redis | 通过Proxy访问 | 通过Proxy访问 | 使用Smart Client直连Redis，Smart Client内置路由规则 | +| 性能 | 高 | 有性能损耗 | 有性能损耗 | 高 | +| 支持的数据库数量 | 多个 | 多个 | 多个 | 一个 | +| Pipeline | 支持 | 支持 | 支持 | 仅支持单个节点Pipeline，不支持跨节点 | +| 需升级客户端SDK？ | 否 | 否 | 否 | 是 | +| 支持在线水平扩容？ | 不支持 | 支持 | 不支持 | 支持 | +| Redis版本 | 支持最新版 | 仅支持3.2.8，升级困难 | 支持最新版 | 支持最新版 | +| 可维护性 | 运维简单，开发人员使用成本高 | 组件较多，部署复杂 | 只有Proxy组件，部署简单 | 运维简单，官方持续维护 | +| 故障自动恢复 | 需部署哨兵 | 需部署哨兵 | 需部署哨兵 | 内置哨兵逻辑，无需额外部署 | + +业界主流的集群化方案就是以上这些，并对它们的特点和区别做了简单的介绍，我们在开发过程中选择自己合适的集群方案即可，但最好是理解它们的实现原理，在使用过程中遇到问题才可以更从容地去解决。 \ No newline at end of file diff --git a/docs/tools/Tools.md b/docs/java/other/Tools.md similarity index 100% rename from docs/tools/Tools.md rename to docs/java/other/Tools.md diff --git a/docs/tools/github.md b/docs/java/other/github.md similarity index 100% rename from docs/tools/github.md rename to docs/java/other/github.md diff --git a/docs/tools/summary.md b/docs/java/other/summary.md similarity index 100% rename from docs/tools/summary.md rename to docs/java/other/summary.md diff --git "a/docs/tools/\344\270\264\346\227\266.md" "b/docs/java/other/\344\270\264\346\227\266.md" similarity index 100% rename from "docs/tools/\344\270\264\346\227\266.md" rename to "docs/java/other/\344\270\264\346\227\266.md" diff --git "a/docs/java/other/\345\215\225\345\205\203\346\265\213\350\257\225\346\241\206\346\236\266.md" "b/docs/java/other/\345\215\225\345\205\203\346\265\213\350\257\225\346\241\206\346\236\266.md" new file mode 100644 index 0000000000..e2bf8a9669 --- /dev/null +++ "b/docs/java/other/\345\215\225\345\205\203\346\265\213\350\257\225\346\241\206\346\236\266.md" @@ -0,0 +1,533 @@ +## 测试 + +测试是开发的一个非常重要的方面，可以在很大程度上决定一个应用程序的命运。良好的测试可以在早期捕获导致应用程序崩溃的问题，但较差的测试往往总是导致故障和停机。 + +![img](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X2dpZi9mQ3BkMWNmOGlhY2FMcVVseWZTQ1JWTHdKQ2dhNUNCc1drR2FmVGlhd1k4TThqejFCRW9TeTEzaWJ1OUVVcktCV2ljRWdkYVZZaWI1dUpiQWE2aWIzcklmSEtnUS82NDA?x-oss-process=image/format,png) + +三种主要类型的软件测试：单元测试，功能测试和集成测试 + +- 单元测试用于测试各个代码组件，并确保代码按照预期的方式工作。单元测试由开发人员编写和执行。大多数情况下，使用JUnit或TestNG之类的测试框架。测试用例通常是在方法级别写入并通过自动化执行。 + +- 集成测试检查系统是否作为一个整体而工作。集成测试也由开发人员完成，但不是测试单个组件，而是旨在跨组件测试。系统由许多单独的组件组成，如代码，数据库，Web服务器等。集成测试能够发现如组件布线，网络访问，数据库问题等问题。 + +- 功能测试通过将给定输入的结果与规范进行比较来检查每个功能是否正确实现。通常，这不是在开发人员级别的。功能测试由单独的测试团队执行。测试用例基于规范编写，并且实际结果与预期结果进行比较。有若干工具可用于自动化的功能测试，如Selenium和QTP。 + +![img](https://bbs-img.huaweicloud.com/blogs/img/1614221444691083141.png) + +从服务端的角度把这三层稍微改一下： + +- 契约测试：测试服务与服务之间的契约，接口保证。代价最高，测试速度最慢。 + +- 集成测试（Integration）：集成当前spring容器、中间件等，对服务内的接口，或者其他依赖于环境的方法的测试。 + + ```java + // 加载spring环境 + @RunWith(SpringBootRunner.class) + @DelegateTo(SpringJUnit4ClassRunner.class) + @SpringBootTest(classes = {Application.class}) + public class ApiServiceTest { + + @Autowired + ApiService apiService; + //do some test + } + ``` + +- 单元测试（Unit Test）：纯函数，方法的测试，不依赖于spring容器，也不依赖于其他的环境。 + + + +### 单元测试与集成测试的区别 + +在实际工作中，不少同学用集成测试代替了单元测试，或者认为集成测试就是单元测试。这里，总结为了单元测试与集成测试的区别： + +**测试对象不同** + +单元测试对象是实现了具体功能的程序单元，集成测试对象是概要设计规划中的模块及模块间的组合。 + +**测试方法不同** + +单元测试中的主要方法是基于代码的白盒测试，集成测试中主要使用基于功能的黑盒测试。 + +**测试时间不同** + +集成测试要晚于单元测试。 + +**测试内容不同** + +单元测试主要是模块内程序的逻辑、功能、参数传递、变量引用、出错处理及需求和设计中具体要求方面的测试；而集成测试主要验证各个接口、接口之间的数据传递关系，及模块组合后能否达到预期效果。 + + + +## 一、单元测试 + +您可以在编写代码后使用『单元测试』来检查代码的质量，同时也可以使用该功能来改进开发过程。建议您一边开发一边编写测试，而不是在完成应用的开发之后才编写测试。这样做有助于您设计出可维护、可重复使用的小型代码单元，也方便您迅速而彻底地测试您的代码。 + +#### 为什么要做单测 + +- 对产品质量非常重要 +- 是唯一一次保证代码覆盖率达到100%的测试 +- 修正一个软件错误所需的费用将随着软件生命期的进展而上升 +- 代码规范、优化，可测试性的代码 +- 放心重构 +- 自动化执行，多次执行 + + + +## 二、单测框架(Java) + +### 1、JUnit + +JUnit是一个开放源代码的Java测试框架，用于编写和运行可重复的测试 + +现在主流的 Junit 有使用 4 和 5（需要JDK8及以上） + +| 特征 | JUNIT 4 | JUNIT 5 | +| :------------------------------- | :------------- | :------------- | +| 声明一种测试方法 | `@Test` | `@Test` | +| 在当前类中的所有测试方法之前执行 | `@BeforeClass` | `@BeforeAll` | +| 在当前类中的所有测试方法之后执行 | `@AfterClass` | `@AfterAll` | +| 在每个测试方法之前执行 | `@Before` | `@BeforeEach` | +| 每种测试方法后执行 | `@After` | `@AfterEach` | +| 禁用测试方法/类 | `@Ignore` | `@Disabled` | +| 测试工厂进行动态测试 | NA | `@TestFactory` | +| 嵌套测试 | NA | `@Nested` | +| 标记和过滤 | `@Category` | `@Tag` | +| 注册自定义扩展 | NA | `@ExtendWith` | + +- JUnit 5利用了Java 8或更高版本的特性，例如lambda函数，使测试更强大，更容易维护。 +- JUnit 5为描述、组织和执行测试添加了一些非常有用的新功能。例如，测试得到了更好的显示名称，并且可以分层组织。 +- JUnit 5被组织成多个库，所以只将你需要的功能导入到你的项目中。通过Maven和Gradle等构建系统，包含合适的库很容易。 +- JUnit 5可以同时使用多个扩展，这是JUnit 4无法做到的（一次只能使用一个runner）。这意味着你可以轻松地将Spring扩展与其他扩展（如你自己的自定义扩展）结合起来。 + +### 2、TestNG + +TestNG类似于JUnit，但是它配置了特殊的注释和高级功能(JUnit不支持)。 + +TestNG中的NG表示“下一代”。TestNG可以覆盖几乎所有类型的软件测试，包括**端到端、单元、集成和功能**测试。TestNG和JUnit都是基于java的框架，允许你编写测试和检查最终结果。如果测试成功，你将看到绿色条，否则将看到红色条。 + +**TestNG的优点和缺点** + +**1. 优点：** + +- 该框架使您能够在多个代码片段上运行并行测试。 +- 在测试用例执行期间，您可以生成HTML报告。 +- 可以根据优先级对测试用例进行分组和排列 +- 可以参数化数据并使用注释来轻松设置优先级。 + +**2. 缺点**：取决于您的要求，此外，设置TestNG需要一点时间。 + + + +[《关于testNG和JUnit的对比》](https://developer.aliyun.com/article/572271) + + + +### 3、Spock + +Spock 由 Gradleware首席工程师 于 2008 年创建。虽然灵感来自于 JUnit，Spock 的特性不仅仅是 JUnit 的扩展： + +- 测试代码使用 Groovy 语言编写，而被测代码可以由 Java 编写。 +- **内置 mock 框架以减少引入第三方框架**。 +- 可支持自定义测试件名称。 +- 为创建测试代码预定义了行为驱动块（given:、when:、then:、expect: 等）。 +- 使用数据表格以减少数据结构的使用需求。 + + + +### 4、JBehave + +Behave是一种令人难以置信的、支持BDD(行为驱动开发)的最佳Java测试框架之一。BDD是TDD(测试驱动开发)和ATDD([验收测试](https://link.zhihu.com/?target=https%3A//www.zmtests.com/%3Fpk_campaign%3Dzhihu-w)驱动开发)的演变。 + +Java提供了若干用于单元测试的框架。TestNG和JUnit是最流行的测试框架。JUnit和TestNG的一些重要功能： + +- 易于设置和运行。 +- 支持注释。 +- 允许忽略或分组并一起执行某些测试。 +- 支持参数化测试，即通过在运行时指定不同的值来运行单元测试。 +- 通过与构建工具，如Ant，Maven和Gradle集成来支持自动化的测试执行。 + +缺点： + +- 需要具备基本的 Groovy 语言知识 + + + +### 5、Spring Test + + + +### 6、Selenide + +Selenide是一个流行的开源Java测试框架，它是由Selenium WebDriver提供支持。它是为Java应用程序编写精确的、交流的、稳定的UI测试用例的工具。它扩展了WebDriver和JUnit功能。 + +WebDriver是一个非常受欢迎的用户界面测试工具，但是它缺乏处理超时的特性。例如，对Ajax等web技术的测试。Selenide框架管理所有这些问题在一个简单的方式。此外，它更容易安装和学习。你只需要把注意力集中在逻辑上，Selenide就会完成剩下的工作。 + +**特性** + +（1）开箱即用，并设置使用框架； + +（2）编写更少的自动化代码； + +（3）节省大量的时间； + +（4）配置理想的CI工具，如Jenkins。 + +**安装Selenide的先决条件** + +由于这个开放源码框架是基于Java的，所以你将需要Java或其他面向对象编程语言方面的经验。在你的工作站，你还将需要 + +（1）JDK 5或更新版本； + +（2）安装了Maven 3.0或其他版本； + +（3）集成开发环境(IDE)工具——大多数情况下，Eclipse是所有开发人员的首选，因为Eclipse和Maven的组合使用起来更简单。 + + + +### 7、JWebUnit + +JWebUnit是一个基于java的测试框架，是用于集成、回归和功能测试的首选JUnit扩展之一。它用一个简单的测试界面包装了当前的活动框架，如HTMLUnit和Selenium。因此，你可以立即测试web应用程序的准确性。 + +JWebUnit可用于执行屏幕导航测试。该框架还提供了一个高级Java应用程序编程接口，用于使用一组断言导航web应用程序，以检查应用程序的准确性。它计算通过链接的导航、表单入口和提交、表内容的调整以及其他常见的业务web应用程序特征。 + + + + + +## 三、Mock 框架 + +单元测试不应该依赖数据，依赖外部服务或组件等，会对其他数据产生影响的情况。启动Spring容器，一般比较慢，可能会启动消息监听消费消息，定时任务的执行等，对数据产生影响。 + +Mock测试就是在测试过程中，对那些当前测试不关心的，不容易构建的对象，用一个虚拟对象来代替测试的情形。 + +说白了：就是解耦(虚拟化)要测试的目标方法中调用的其它方法，例如：Service的方法调用Mapper类的方法，这时候就要把Mapper类Mock掉（产生一个虚拟对象），这样我们可以自由的控制这个Mapper类中的方法，让它们返回想要的结果、抛出指定异常、验证方法的调用次数等等。 + +- **Mock可以用来解除外部服务依赖，从而保证了测试用例的独立性** +- **Mock可以减少全链路测试数据准备，从而提高了编写测试用例的速度** +- **Mock可以模拟一些非正常的流程，从而保证了测试用例的代码覆盖率** +- **Mock可以不用加载项目环境配置，从而保证了测试用例的执行速度** + + + + + +### 1、EasyMock + +EasyMock 是早期比较流行的MocK测试框架。它提供对接口的模拟，能够通过录制、回放、检查三步来完成大体的测试过程，可以验证方法的调用种类、次数、顺序，可以令 Mock 对象返回指定的值或抛出指定异常。通过 EasyMock，我们可以方便的构造 Mock 对象从而使单元测试顺利进行。 + + + +### 2、Mockito + +EasyMock之后流行的mock工具。相对EasyMock学习成本低，而且具有非常简洁的API，测试代码的可读性很高。 + +Sprinng-boot-starter-test 内置 + + + +### 3、PowerMock + +这个工具是在EasyMock和Mockito上扩展出来的，目的是为了解决EasyMock和Mockito不能解决的问题，比如对static, final, private方法均不能mock。其实测试架构设计良好的代码，一般并不需要这些功能，但如果是在已有项目上增加单元测试，老代码有问题且不能改时，就不得不使用这些功能了。 + + + +### 4、Jmockit + +JMockit 是一个轻量级的mock框架是用以帮助开发人员编写测试程序的一组工具和API，该项目完全基于 Java 5 SE 的 java.lang.instrument 包开发，内部使用 ASM 库来修改Java的Bytecode。 + + + +### 5、TestableMock + +阿里开源的一款 mock 工具，让Mock的定义和置换干净利落 + +https://github.com/alibaba/testable-mock + + + +#### 常见 Mock 场景 + +1. Mock远程服务调用 +2. 从数据库或文件读取数据 +3. 跳过检查操作 +4. 跳过AOP处理 +5. 检查特定方法是否执行/调用参数 +6. 禁止特定方法执行 + + + + + +## 四、现状 + +没有单测 + +单元测试起不来 + +单元测试的“单元”太大 + + + +## 五、初步方案 + +都有一定的学习成本 + + + +#### 数据库测试 + +数据库测试多用在DAO中，DAO对数据库的操作依赖于mybatis的sql mapper 文件，这些sql mapper多是手工写的，在单测中验证所有sql mapper的正确性非常重要，在DAO层有足够的覆盖度和强度后，Service层的单测才能仅仅关注自身的业务逻辑 + +为了验证sql mapper，我们需要一个能实际运行的数据库。为了提高速度和减少依赖，可以使用内存数据库。内存数据库和目标数据库(MySQL,TDDL)在具体函数上有微小差别，不过只要使用标准的SQL 92，两者都是兼容的。 + +使用 [H2](http://www.h2database.com/) 作为单测数据库 + + + +#### 其它外部依赖 + +对于除数据库以外的依赖，包括各种中间件以及外部的HSF/HTTP服务，在单测中全部采用Mock进行解耦。 + +junit + mockito + +与spring结合性感觉更强，可以充分兼容spring的各种注解，能较完美接近项目启动后实际运行情况。可以用来做一定的业务自动化回归测试，但可能超出了ut的范围 + + + + + + + +- 一个类里面测试太多怎么办？ +- 不知道别人mock了哪些数据怎么办？ +- 测试结构太复杂？ +- 测试莫名奇妙起不来？ + +FSC（Fixture-Scenario-Case）是一种组织测试代码的方法，目标是尽量将一些MOCK信息在不同的测试中共享。其结构如下： + +![图片](https://mmbiz.qpic.cn/mmbiz_png/Z6bicxIx5naJkibdPicqyKZw3IFlzLsRSickr20z2q3xU5FuicZJotzc422ffkPOPkGeZnQvjeKRWZykXXNz24ricgkQ/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + + + +## 六、单测规约 + +#### 使用断言而不是Print语句 + +许多开发人员习惯于在每行代码之后编写System.out.println语句来验证代码是否正确执行。这种做法常常扩展到单元测试，从而导致测试代码变得杂乱。除了混乱，这需要开发人员手动干预去验证控制台上打印的输出，以检查测试是否成功运行。更好的方法是使用自动指示测试结果的断言。 + +#### 构建具有确定性结果的测试 + +#### 除了正面情景外，还要测试负面情景和边缘情况 + +#### 单测要有覆盖度 + +#### 单测要稳定 + + + +### 阿里 java 开发规范 —— 单测部分： + +> 1. 【强制】好的单元测试必须遵守AIR原则。 +> +> 说明：单元测试在线上运行时，感觉像空气（AIR）一样并不存在，但在测试质量的保障上，却是非常关键的。好的单元测试宏观上来说，具有自动化、独立性、可重复执行的特点。 +> +> - A：Automatic（自动化） +> - I：Independent（独立性） +> - R：Repeatable（可重复） +> +> 2. 【强制】单元测试应该是全自动执行的，并且非交互式的。测试框架通常是定期执行的，执行过程必须完全自动化才有意义。输出结果需要人工检查的测试不是一个好的单元测试。单元测试中不准使用System.out来进行人肉验证，必须使用assert来验证。 +> +> 3. 【强制】保持单元测试的独立性。为了保证单元测试稳定可靠且便于维护，单元测试用例之间决不能互相调用，也不能依赖执行的先后次序。 +> +> 反例：method2需要依赖method1的执行，将执行结果做为method2的输入。 +> +> 4. 【强制】单元测试是可以重复执行的，不能受到外界环境的影响。 +> +> 说明：单元测试通常会被放到持续集成中，每次有代码check in时单元测试都会被执行。如果单测对外部环境（网络、服务、中间件等）有依赖，容易导致持续集成机制的不可用。 +> +> 正例：为了不受外界环境影响，要求设计代码时就把SUT的依赖改成注入，在测试时用spring 这样的DI框架注入一个本地（内存）实现或者Mock实现。 +> +> 5. 【强制】对于单元测试，要保证测试粒度足够小，有助于精确定位问题。单测粒度至多是类级别，一般是方法级别。 +> +> 说明：只有测试粒度小才能在出错时尽快定位到出错位置。单测不负责检查跨类或者跨系统的交互逻辑，那是集成测试的领域。 +> +> 6. 【强制】核心业务、核心应用、核心模块的增量代码确保单元测试通过。 +> +> 说明：新增代码及时补充单元测试，如果新增代码影响了原有单元测试，请及时修正。 +> +> 7. 【强制】单元测试代码必须写在如下工程目录：src/test/java，不允许写在业务代码目录下。 +> +> 说明：源码构建时会跳过此目录，而单元测试框架默认是扫描此目录。 +> +> 8. 【推荐】单元测试的基本目标：语句覆盖率达到70%；核心模块的语句覆盖率和分支覆盖率都要达到100% +> +> 说明：**在工程规约的应用分层中提到的DAO层，Manager层，可重用度高的Service，都应该进行单元测试**。 +> +> 9. 【推荐】编写单元测试代码遵守BCDE原则，以保证被测试模块的交付质量。 +> +> - B：Border，边界值测试，包括循环边界、特殊取值、特殊时间点、数据顺序等。 +> +> - C：Correct，正确的输入，并得到预期的结果。 +> +> - D：Design，与设计文档相结合，来编写单元测试。 +> +> - E：Error，强制错误信息输入（如：非法数据、异常流程、非业务允许输入等），并得到预期的结果。 +> +> 10. 【推荐】对于数据库相关的查询，更新，删除等操作，不能假设数据库里的数据是存在的，或者直接操作数据库把数据插入进去，请使用程序插入或者导入数据的方式来准备数据。 +> +> 反例：删除某一行数据的单元测试，在数据库中，先直接手动增加一行作为删除目标，但是这一行新增数据并不符合业务插入规则，导致测试结果异常。 +> +> 11. 【推荐】和数据库相关的单元测试，可以设定自动回滚机制，不给数据库造成脏数据。或者对单元测试产生的数据有明确的前后缀标识。 +> +> 正例：在RDC内部单元测试中，使用RDC_UNIT_TEST_的前缀标识数据。 +> +> 12. 【推荐】对于不可测的代码建议做必要的重构，使代码变得可测，避免为了达到测试要求而书写不规范测试代码。 +> +> 13. 【推荐】在设计评审阶段，开发人员需要和测试人员一起确定单元测试范围，单元测试最好覆盖所有测试用例（UC）。 +> +> 14. 【推荐】单元测试作为一种质量保障手段，不建议项目发布后补充单元测试用例，建议在项目提测前完成单元测试。 +> +> 15. 【参考】为了更方便地进行单元测试，业务代码应避免以下情况： +> +> - 构造方法中做的事情过多。 +> +> - 存在过多的全局变量和静态方法。 +> +> - 存在过多的外部依赖。 +> +> - 存在过多的条件语句。 +> +> 说明：多层条件语句建议使用卫语句、策略模式、状态模式等方式重构。 +> +> 16. 【参考】不要对单元测试存在如下误解： +> +> - 那是测试同学干的事情。本文是开发手册，凡是本文内容都是与开发同学强相关的。  单元测试代码是多余的。汽车的整体功能与各单元部件的测试正常与否是强相关的。 +> - 单元测试代码不需要维护。一年半载后，那么单元测试几乎处于废弃状态。 +> - 单元测试与线上故障没有辩证关系。好的单元测试能够最大限度地规避线上故障 + + + + + +action 测试还是通过 海星系统 + +- 《SpringBoot - 單元測試工具 Mockito》https://kucw.github.io/blog/2020/2/spring-unit-test-mockito/ + +- https://cloud.tencent.com/developer/article/1338791 +- 《写有价值的单元测试》https://developer.aliyun.com/article/54478 +- 《Mockito.mock() vs @Mock vs @MockBean》https://www.baeldung.com/java-spring-mockito-mock-mockbean +- 《Mock服务插件在接口测试中的设计与应用》 https://tech.youzan.com/mock/ + + + + + +``` +package com.sogou.bizdev.kuaitou.api.test; + +import com.sogou.bizdev.kuaitou.api.dao.IndustryFieldDao; +import com.sogou.bizdev.kuaitou.api.po.IndustryField; +import com.sogou.bizdev.kuaitou.api.service.IndustryFieldService; +import com.sogou.bizdev.kuaitou.dto.IndustryFieldDto; +import org.junit.Assert; +import org.junit.Test; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.extension.ExtendWith; +import org.junit.runner.RunWith; +import org.mockito.InjectMocks; +import org.mockito.Mock; +import org.mockito.Mockito; +import org.mockito.MockitoAnnotations; +import org.mockito.junit.MockitoJUnitRunner; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.test.context.junit4.SpringRunner; +import org.springframework.test.web.servlet.MockMvc; +import org.springframework.test.web.servlet.setup.MockMvcBuilders; +import org.springframework.web.context.WebApplicationContext; + +import java.util.ArrayList; +import java.util.Date; +import java.util.List; + +/** + * @description: 行业字段单测 + * @author: starfish + * @date: 2021/5/11 20:01 + */ +//@RunWith(MockitoJUnitRunner.class) +//@RunWith(SpringRunner.class) +//@ExtendWith(MockitoExtension.class) +public class IndustryFieldTest { + + @Autowired + private WebApplicationContext webApplicationContext; + private MockMvc mockMvc; + + //在每个测试方法执行之前都初始化MockMvc对象 +// @BeforeEach +// public void setupMockMvc() { +// mockMvc = MockMvcBuilders.webAppContextSetup(webApplicationContext).build(); +// } + + @BeforeEach + public void beforeEach() { + MockitoAnnotations.initMocks(this); + } +// +// @Autowired +// //@InjectMocks +// private IndustryFieldService industryFieldService; + + IndustryFieldService industryFieldService = Mockito.mock(IndustryFieldService.class); + +// @MockBean +// private IndustryFieldDao fieldDao; + +// @Mock +// private IndustryFieldDao fieldDao; + + IndustryFieldDao fieldDao = Mockito.mock(IndustryFieldDao.class); + + + @Test + public void listIndustryFields() { + + //industryFieldService 调用 dao 层去查 DB + List fields = new ArrayList<>(); + + IndustryField locField = new IndustryField(); + locField.setId(1L); + locField.setFieldKey("loc"); + locField.setRequired(1); + locField.setFieldKeyCn("商品移动端url"); + + IndustryField nameField = new IndustryField(); + nameField.setId(2L); + nameField.setFieldKey("name"); + nameField.setRequired(1); + nameField.setFieldKeyCn("商品名称"); + + fields.add(locField); + fields.add(nameField); + + Mockito.when(fieldDao.listIndustryFields(0)).thenReturn(fields); + + List fieldDtos = industryFieldService.listIndustryFields(0); + + Assert.assertNotNull(fieldDtos); + Assert.assertEquals(fieldDtos.get(0).getFieldKey(),"loc"); + + } + + +// @MockBean +// private IndustryFieldDao industryFieldDao; + +} +``` + + + +- 背景 +- 单测实践 \ No newline at end of file diff --git "a/docs/java/other/\346\225\260\346\215\256\345\272\223\346\226\207\346\241\243\345\257\274\345\207\272.md" "b/docs/java/other/\346\225\260\346\215\256\345\272\223\346\226\207\346\241\243\345\257\274\345\207\272.md" new file mode 100644 index 0000000000..15610e9c64 --- /dev/null +++ "b/docs/java/other/\346\225\260\346\215\256\345\272\223\346\226\207\346\241\243\345\257\274\345\207\272.md" @@ -0,0 +1,131 @@ +> 来源：https://segmentfault.com/a/1190000023485195 + +# 超给力，一键生成数据库文档-数据库表结构逆向工程 +![](https://tva1.sinaimg.cn/large/007S8ZIlly1gicozj2o86j30lp07adgr.jpg) + +## 一、解决什么问题 + +数据库文档是我们在企业项目开发中需要交付的文档，通常需要开发人员去手工编写。编写完成后，数据库发生变更又需要手动的进行修改,从而浪费了大量的人力。并且这种文档并没有什么技术含量，被安排做这个工作的程序员往往自己心里会有抵触情绪，悲观的预期自己在团队的位置，造成离职也是可能的。如下面的这种文档的内容： + +![](https://tva1.sinaimg.cn/large/007S8ZIlly1gicoznbqsfj30m809ymzo.jpg) + +笔者最近在github上面发现一个数据库文档生成工具：**screw**（螺丝钉）。该工具能够通过简单地配置，快速的根据数据库表结构进行逆向工程，将数据库表结构及字段逆向生成为文档。 + +## 二、特点 + +- 简洁、轻量、设计良好 +- 多数据库支持：MySQL、MariaDB、TIDB、Oracle、 SqlServer、PostgreSQL、Cache DB +- 多种格式文档: html、word、 markdwon +- 灵活扩展：支持用户自定义模板和展示样式修改（freemarker模板） + +## 三、依赖库探究 + +[mvn中央仓库查看最新版本](https://mvnrepository.com/artifact/cn.smallbun.screw/screw-core)，将如下的maven坐标引入到Spring Boot项目中去： + +```xml + + cn.smallbun.screw + screw-core + 1.0.3 + +``` + +从maven仓库的编译依赖中可以看到，screw-core其实现依赖了如下的内容。重点关注freemarker，因为该项目是使用freemarker作为模板生成文档。 + +![](https://tva1.sinaimg.cn/large/007S8ZIlly1gicozsgjsfj30ew0aijrx.jpg) + +除此之外，screw使用了HikariCP作为数据库连接池，所以： + +- 你的Spring Boot项目需要引入HikariCP数据库连接池。 +- 根据你的数据库类型及版本，引入正确的JDBC驱动 + +## 四、开始造作吧 + +以上的工作都做好之后，我们就可以来配置文档生成参数了。实现文档生成有两种方式，一种是写代码，一种是使用maven 插件。 + +- 我个人还是比较喜欢使用代码的当时，写一个单元测试用例就可以了，相对独立，使用方式也灵活。 +- 如果放在pom.xml的插件配置里面，让本就很冗长的pom.xml变的更加的冗长，不喜欢。 + +所以maven插件的这种方式我就不给大家演示了，直接把下面的代码Ctrl + C/V到你的src/test/java目录下。简单的修改配置，运行就可以了 + +```java +public class ScrewTest { + + @Test + void testScrew() { + //数据源 + HikariConfig hikariConfig = new HikariConfig(); + hikariConfig.setDriverClassName("com.mysql.cj.jdbc.Driver"); + hikariConfig.setJdbcUrl("jdbc:mysql://127.0.0.1:3306/database"); + hikariConfig.setUsername("db-username"); + hikariConfig.setPassword("db-password"); + //设置可以获取tables remarks信息 + hikariConfig.addDataSourceProperty("useInformationSchema", "true"); + hikariConfig.setMinimumIdle(2); + hikariConfig.setMaximumPoolSize(5); + DataSource dataSource = new HikariDataSource(hikariConfig); + + //生成配置 + EngineConfig engineConfig = EngineConfig.builder() + //生成文件路径 + .fileOutputDir("d://") + //打开目录 + .openOutputDir(true) + //生成文件类型：HTML + .fileType(EngineFileType.HTML) + //生成模板实现 + .produceType(EngineTemplateType.freemarker) + .build(); + + //忽略表 + ArrayList ignoreTableName = new ArrayList<>(); + ignoreTableName.add("test_user"); + ignoreTableName.add("test_group"); + //忽略表前缀 + ArrayList ignorePrefix = new ArrayList<>(); + ignorePrefix.add("test_"); + //忽略表后缀 + ArrayList ignoreSuffix = new ArrayList<>(); + ignoreSuffix.add("_test"); + ProcessConfig processConfig = ProcessConfig.builder() + //指定生成逻辑、当存在指定表、指定表前缀、指定表后缀时，将生成指定表，其余表不生成、并跳过忽略表配置 + //根据名称指定表生成 + .designatedTableName(new ArrayList<>()) + //根据表前缀生成 + .designatedTablePrefix(new ArrayList<>()) + //根据表后缀生成 + .designatedTableSuffix(new ArrayList<>()) + //忽略表名 + .ignoreTableName(ignoreTableName) + //忽略表前缀 + .ignoreTablePrefix(ignorePrefix) + //忽略表后缀 + .ignoreTableSuffix(ignoreSuffix).build(); + //配置 + Configuration config = Configuration.builder() + //版本 + .version("1.0.0") + //描述,文档名称 + .description("数据库设计文档生成") + //数据源 + .dataSource(dataSource) + //生成配置 + .engineConfig(engineConfig) + //生成配置 + .produceConfig(processConfig) + .build(); + //执行生成 + new DocumentationExecute(config).execute(); + + } +} +``` + +在测试用例里面运行上面的代码，就会自动生成数据库文档到fileOutputDir配置目录下。 + +![](https://tva1.sinaimg.cn/large/007S8ZIlly1gicozy2jk0j30gc07o0tn.jpg) + +## 五、效果 + +![](https://tva1.sinaimg.cn/large/007S8ZIlly1gicp01wz8fj30m80li41c.jpg) +![](https://tva1.sinaimg.cn/large/007S8ZIlly1gicp07iiu5j30m80agmzh.jpg) \ No newline at end of file diff --git "a/docs/java/\350\256\260\344\270\200\346\254\241 Java \347\272\277\344\270\212\345\206\205\345\255\230\346\263\204\351\234\262\345\256\232\344\275\215.md" "b/docs/java/\350\256\260\344\270\200\346\254\241 Java \347\272\277\344\270\212\345\206\205\345\255\230\346\263\204\351\234\262\345\256\232\344\275\215.md" new file mode 100644 index 0000000000..ffed3ec456 --- /dev/null +++ "b/docs/java/\350\256\260\344\270\200\346\254\241 Java \347\272\277\344\270\212\345\206\205\345\255\230\346\263\204\351\234\262\345\256\232\344\275\215.md" @@ -0,0 +1,213 @@ +# 记一次 Java 应用内存泄漏的定位过程 + +## 问题现象 + +最近，笔者负责测试的某个算法模块机器出现大量报警，报警表现为机器CPU持续高占用。该算法模块是一个优化算法，本身就是CPU密集型应用，一开始怀疑可能是算法在正常运算，但很快这种猜测就被推翻：同算法同学确认后，该算法应用只使用了一个核心，而报警时，一个算法进程占用了服务机器的全部8个核心，这显然不是正常计算造成的。 + +## 定位步骤 + +首先按照CPU问题的定位思路进行定位，对 Java 调用堆栈进行分析： + +1. 使用`top -c` 查看 CPU 占用高的进程： + + ![](https://testerhome.com/uploads/photo/2020/edad345c-b58a-4026-8fe8-7366e22468c7.png!large) + + ，从 top 命令的结果看，19272 号进程 CPU 占用率最高，基本确定问题是该进程引起，可以从 Command + + 栏看到这正是算法模块程序，注意图是线下4C机器上复现时的截图。 + +2. 使用`ps -mp pid -o THREAD,tid,time`命令定位问题线程。 + + ``` + ps -mp 19272 -o THREAD,tid,time + USER %CPU PRI SCNT WCHAN USER SYSTEM TID TIME + USER 191 - - - - - - 00:36:54 + USER 0.0 19 - futex_ - - 19272 00:00:00 + USER 68.8 19 - futex_ - - 19273 00:13:18 + USER 30.2 19 - - - - 19274 00:05:50 + USER 30.2 19 - - - - 19275 00:05:50 + USER 30.2 19 - - - - 19276 00:05:50 + USER 30.1 19 - - - - 19277 00:05:49 + USER 0.4 19 - futex_ - - 19278 00:00:05 + USER 0.0 19 - futex_ - - 19279 00:00:00 + USER 0.0 19 - futex_ - - 19280 00:00:00 + USER 0.0 19 - futex_ - - 19281 00:00:00 + USER 0.4 19 - futex_ - - 19282 00:00:04 + USER 0.3 19 - futex_ - - 19283 00:00:03 + USER 0.0 19 - futex_ - - 19284 00:00:00 + USER 0.0 19 - futex_ - - 19285 00:00:00 + USER 0.0 19 - futex_ - - 19286 00:00:00 + USER 0.0 19 - skb_wa - - 19362 00:00:00 + ``` + + 从结果可以看到，出现问题的线程主要是 19273-19277。 + +3. 使用`jstack`查看出现问题的线程堆栈信息。 + +由于 `jstack` 使用的线程号是十六进制，因此需要先把线程号从十进制转换为十六进制。 + +``` +$ printf "%x\n" 19273 +4b49 +$ jstack 12262 |grep -A 15 4b49 +"main" #1 prio=5 os_prio=0 tid=0x00007f98c404c000 nid=0x4b49 runnable [0x00007f98cbc58000] +java.lang.Thread.State: RUNNABLE + at java.util.ArrayList.iterator(ArrayList.java:840) + at optional.score.MultiSkuDcAssignmentEasyScoreCalculator.updateSolution(MultiSkuDcAssignmentEasyScoreCalculator.java:794) + at optional.score.MultiSkuDcAssignmentEasyScoreCalculator.calculateScore(MultiSkuDcAssignmentEasyScoreCalculator.java:80) + at optional.score.MultiSkuDcAssignmentEasyScoreCalculator.calculateScore(MultiSkuDcAssignmentEasyScoreCalculator.java:17) + at org.optaplanner.core.impl.score.director.easy.EasyScoreDirector.calculateScore(EasyScoreDirector.java:60) + at org.optaplanner.core.impl.score.director.AbstractScoreDirector.doAndProcessMove(AbstractScoreDirector.java:188) + at org.optaplanner.core.impl.localsearch.decider.LocalSearchDecider.doMove(LocalSearchDecider.java:132) + at org.optaplanner.core.impl.localsearch.decider.LocalSearchDecider.decideNextStep(LocalSearchDecider.java:116) + at org.optaplanner.core.impl.localsearch.DefaultLocalSearchPhase.solve(DefaultLocalSearchPhase.java:70) + at org.optaplanner.core.impl.solver.AbstractSolver.runPhases(AbstractSolver.java:88) + at org.optaplanner.core.impl.solver.DefaultSolver.solve(DefaultSolver.java:191) + at app.DistributionCenterAssignmentApp.main(DistributionCenterAssignmentApp.java:61) + +"VM Thread" os_prio=0 tid=0x00007f98c419d000 nid=0x4b4e runnable + +"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f98c405e800 nid=0x4b4a runnable + +"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f98c4060800 nid=0x4b4b runnable + +"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f98c4062800 nid=0x4b4c runnable + +"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f98c4064000 nid=0x4b4d runnable + +"VM Periodic Task Thread" os_prio=0 tid=0x00007f98c4240800 nid=0x4b56 waiting on condition +``` + +可以看到，除了 0x4b49 线程是正常工作线程，其它都是 gc 线程。 + +此时怀疑：**是频繁 GC 导致的 CPU 被占满。** + +我们可以使用 `jstat` 命令查看 GC 统计： + +``` +$ jstat -gcutil 19272 2000 10 +S0 S1 E O M CCS YGC YGCT FGC FGCT GCT +0.00 0.00 22.71 100.00 97.16 91.53 2122 19.406 282 809.282 828.688 +0.00 0.00 100.00 100.00 97.16 91.53 2122 19.406 283 809.282 828.688 +0.00 0.00 92.46 100.00 97.16 91.53 2122 19.406 283 812.730 832.135 +0.00 0.00 100.00 100.00 97.16 91.53 2122 19.406 284 812.730 832.135 +0.00 0.00 100.00 100.00 97.16 91.53 2122 19.406 285 815.965 835.371 +0.00 0.00 100.00 100.00 97.16 91.53 2122 19.406 285 815.965 835.371 +0.00 0.00 100.00 100.00 97.16 91.53 2122 19.406 286 819.492 838.898 +0.00 0.00 100.00 100.00 97.16 91.53 2122 19.406 286 819.492 838.898 +0.00 0.00 100.00 100.00 97.16 91.53 2122 19.406 287 822.751 842.157 +0.00 0.00 30.78 100.00 97.16 91.53 2122 19.406 287 825.835 845.240 +``` + +重点关注一下几列： +**YGC**：年轻代垃圾回收次数 +**YGCT**：年轻代垃圾回收消耗时间 +**FGC**：老年代垃圾回收次数 +**FGCT**：老年代垃圾回收消耗时间 +**GCT**：垃圾回收消耗总时间 +可以看到，20s 的时间中进行了 5 次 full GC，仅仅耗费在 GC 的时间已经到了 17s。 + +1. 增加启动参数，展示详细 GC 过程。 通过增加 jvm 参数，更快暴露 GC 问题，并展示 GC 详细过程`java -Xmx1024m -verbose:gc`。 + +``` +[Full GC (Ergonomics) 1046527K->705881K(1047552K), 1.8974837 secs] +[Full GC (Ergonomics) 1046527K->706191K(1047552K), 2.5837756 secs] +[Full GC (Ergonomics) 1046527K->706506K(1047552K), 2.6142270 secs] +[Full GC (Ergonomics) 1046527K->706821K(1047552K), 1.9044987 secs] +[Full GC (Ergonomics) 1046527K->707130K(1047552K), 2.0856625 secs] +[Full GC (Ergonomics) 1046527K->707440K(1047552K), 2.6273944 secs] +[Full GC (Ergonomics) 1046527K->707755K(1047552K), 2.5668877 secs] +[Full GC (Ergonomics) 1046527K->708068K(1047552K), 2.6924427 secs] +[Full GC (Ergonomics) 1046527K->708384K(1047552K), 3.1084132 secs] +[Full GC (Ergonomics) 1046527K->708693K(1047552K), 1.9424100 secs] +[Full GC (Ergonomics) 1046527K->709007K(1047552K), 1.9996261 secs] +[Full GC (Ergonomics) 1046527K->709314K(1047552K), 2.4190958 secs] +[Full GC (Ergonomics) 1046527K->709628K(1047552K), 2.8139132 secs] +[Full GC (Ergonomics) 1046527K->709945K(1047552K), 3.0484079 secs] +[Full GC (Ergonomics) 1046527K->710258K(1047552K), 2.6983539 secs] +[Full GC (Ergonomics) 1046527K->710571K(1047552K), 2.1663274 secs] +``` + +至此基本可以确定，CPU 高负载的根本原因是内存不足导致频繁 GC。 + +## 根本原因 + +虽然我们经过上面的分析可以知道，是频繁 GC 导致的 CPU 占满，但是并没有找到问题的根本原因，因此也无从谈起如何解决。GC 的直接原因是内存不足，怀疑算法程序存在内存泄漏。 + +### 为什么会内存泄漏 + +虽然 Java 语言天生就有垃圾回收机制，但是这并不意味着 Java 就没有内存泄漏问题。 + +正常情况下，在 Java 语言中如果一个对象不再被使用，那么 Java 的垃圾回收机制会及时把这些对象所占用的内存清理掉。但是有些情况下，有些对象虽然不再被程序使用，但是仍然有引用指向这些对象，所以垃圾回收机制无法处理。随着这些对象占用内存数量的增长，最终会导致内存溢出。 + +Java 的内存泄漏问题比较难以定位，下面针对一些常见的内存泄漏场景做介绍： + +1. 持续在堆上创建对象而不释放。例如，持续不断的往一个列表中添加对象，而不对列表清空。这种问题，通常可以给程序运行时添加 JVM 参数`-Xmx` 指定一个较小的运行堆大小，这样可以比较容易的发现这类问题。 +2. 不正确的使用静态对象。因为 static 关键字修饰的对象的生命周期与 Java 程序的运行周期是一致的，所以垃圾回收机制无法回收静态变量引用的对象。所以，发生内存泄漏问题时，我们要着重分析所有的静态变量。 +3. 对大 String 对象调用 String.intern()方法，该方法会从字符串常量池中查询当前字符串是否存在，若不存在就会将当前字符串放入常量池中。而在 jdk6 之前，字符串常量存储在 `PermGen` 区的，但是默认情况下 `PermGen` 区比较小，所以较大的字符串调用此方法，很容易会触发内存溢出问题。 +4. 打开的输入流、连接没有争取关闭。由于这些资源需要对应的内存维护状态，因此不关闭会导致这些内存无法释放。 + +### 如何进行定位 + +以上介绍了一些常见的内存泄漏场景，在实际的问题中还需要针对具体的代码进行确定排查。下面结合之前的频繁 GC 问题，讲解一下定位的思路，以及相关工具的使用方法。 + +#### 线上定位 + +对于线上服务，如果不能开启 Debug 模式，那么可用的工具较少。推荐方式： +使用 `top -c` 命令查询 Java 高内存占用程序的进程 pid。然后使用 `jcmd` 命令获取进程中对象的计数、内存占用信息。 + +``` +$ jcmd 24600 GC.class_histogram |head -n 10 +24600: + + num #instances #bytes class name +---------------------------------------------- + 1: 2865351 103154208 [J + 2: 1432655 45844960 org.optaplanner.core.impl.localsearch.scope.LocalSearchMoveScope + 3: 1432658 34383792 org.optaplanner.core.api.score.buildin.bendablelong.BendableLongScore + 4: 1193860 28652640 org.optaplanner.core.impl.heuristic.selector.move.generic.ChangeMove + 5: 241961 11986056 [Ljava.lang.Object; + 6: 239984 5759616 java.util.ArrayList +``` + +结果中，`#instances` 为对象数量，`#bytes` 为占用内存大小，单位是 byte，`class name` 为对应的类名。 +排名第一的是 Java 原生类型，实际上是 long 类型。 + +另外，要注意的是结果中的类可能存在包含关系，例如一个类中含有多个 long 类型数据，那 long 对应的计数也会增加，所以我们要排除一些基本类型，它们可能是我们程序中使用导致的计数增加，重点关注我们程序中的类。 + +如果仅仅有 jcmd 的结果，其实很难直接找到问题的根本原因。如果问题不能在线下复现，我们基本上只能针对计数较多的类名跟踪变量的数据流，重点关注 new 对象附近的代码逻辑。观察代码逻辑时，重点考虑上述几种常见内存泄漏场景。 + +#### 线下定位 + +如果内存泄漏问题可以在线下复现，那么问题定位的工具就比较丰富了。下面主要推荐的两种工具，VisualVM & IDEA。 + +这里主要讲一下IDEA调试定位思路： + +##### 使用 IDEA 调试器定位内存泄漏问题 + +如果以上过程依然不能有效的分析出问题的根本原因，还可以使用 IDEA 的调试功能进行定位。 +配置好程序的运行参数，正常复现问题之后，对程序打断点并逐步追踪。 + +重点关注的是程序需要大量运行时间的代码部分，我们可以使用调试暂停功能获得一个内存快照。 +然后在此运行并暂停，这时候在调试的 Memory 视图中可以看到哪些类在快速增加。基本上可以断定问题的原因是两次运行中 new 该对象的语句。 + +[![img](https://testerhome.com/uploads/photo/2020/a561b603-54d2-4c2d-b079-01525827506d.png!large)](https://testerhome.com/uploads/photo/2020/a561b603-54d2-4c2d-b079-01525827506d.png!large) + + + +### 定位结果 + +经过上述定位步骤，最终发现问题的根本原因，在求解器的 LocalSearch 阶段，如果使用了禁忌搜索（Tabu Search）策略，并且长时间找不到更好的解，会不断把当前经历过的解加到禁忌表中。对应的代码部分，finalListScore 是一个 list，在 55 行代码不断的添加 moveScope 对象，导致了内存泄漏： + +[![img](https://testerhome.com/uploads/photo/2020/15151282-fcb9-4f66-bce9-e007ee2f9546.png!large)](https://testerhome.com/uploads/photo/2020/15151282-fcb9-4f66-bce9-e007ee2f9546.png!large) + + + +## 解决方案 + +在求解器该处代码对 `finalListScore` 进行长度限制，添加对象前发现达到了上限就清空，彻底避免内存泄漏的发生。由于出问题的是一个开源求解器框架：optaplanner，为了方便以后维护，按照开源项目贡献流程，把改fix提PR给项目即可，如何给开源项目提PR可以参考社区文章：https://testerhome.com/topics/2114。 + +细节参考PR链接：https://github.com/kiegroup/optaplanner/pull/726。 +项目维护者从代码维护的角度没有接受这个PR，但是使用了类似的fix思路最终修复了这个存在了几年bug：https://github.com/kiegroup/optaplanner/pull/763。 + +最后，算法模块升级到最新版本的optaplanner依赖即可修复该问题。 \ No newline at end of file diff --git a/docs/leetcode/complexity.md b/docs/leetcode/complexity.md deleted file mode 100644 index 433abed7cd..0000000000 --- a/docs/leetcode/complexity.md +++ /dev/null @@ -1,201 +0,0 @@ -> 高级工程师title的我，最近琢磨着好好刷刷算法题更高级一些，然鹅，当我准备回忆大学和面试时候学的数据结构之时，我发现自己对这个算法复杂度的记忆只有OOOOOooo -> -> 文章收录在 GitHub [JavaKeeper](https://github.com/Jstarfish/JavaKeeper) ，N线互联网开发必备技能兵器谱 - -算法（Algorithm）是指用来操作数据、解决程序问题的一组方法。对于同一个问题，使用不同的算法，也许最终得到的结果是一样的，但在过程中消耗的资源和时间却会有很大的区别。 - -那么我们应该如何去衡量不同算法之间的优劣呢？ - -主要还是从算法所占用的「时间」和「空间」两个维度去考量。 - -- 时间维度：是指执行当前算法所消耗的时间，我们通常用「时间复杂度」来描述。 -- 空间维度：是指执行当前算法需要占用多少内存空间，我们通常用「空间复杂度」来描述。 - -因此，评价一个算法的效率主要是看它的时间复杂度和空间复杂度情况。然而，有的时候时间和空间却又是「鱼和熊掌」，不可兼得的，那么我们就需要从中去取一个平衡点。 - -## **时间复杂度** - -一个算法执行所耗费的时间，从理论上是不能算出来的，必须上机运行测试才能知道。但我们不可能也没有必要对每个算法都上机测试，只需知道哪个算法花费的时间多，哪个算法花费的时间少就可以了。并且一个算法花费的时间与算法中语句的执行次数成正比例，哪个算法中语句执行次数多，它花费时间就多。一个算法中的语句执行次数称为语句频度或「**时间频度**」。记为T(n)。 - -时间频度T(n)中，n称为问题的规模，当n不断变化时，时间频度T(n)也会不断变化。但有时我们想知道它变化时呈现什么规律，为此我们引入时间复杂度的概念。算法的时间复杂度也就是算法的时间度量，记作：T(n) = O(f(n))。它表示随问题规模n的增大，算法执行时间的增长率和f(n)的增长率相同，称作算法的渐进时间复杂度，简称「**时间复杂度**」。 - -这种表示方法我们称为「 **大O符号表示法** 」，又称为**渐进符号**，是用于描述函数渐进行为的数学符号 - -常见的时间复杂度量级有： - -- 常数阶$O(1)$ -- 线性阶$O(n)$ -- 平方阶$O(n^2)$ -- 立方阶$O(n^3)$ -- 对数阶$O(logn)$ -- 线性对数阶$O(nlogn)$ -- 指数阶$O(2^n)$ - -#### 常数阶$O(1)$ - -$O(1)$，表示该算法的执行时间（或执行时占用空间）总是为一个常量，不论输入的数据集是大是小，只要是没有循环等复杂结构，那这个代码的时间复杂度就都是O(1)，如： - -```java -int i = 1; -int j = 2; -int k = i + j; -``` - -上述代码在执行的时候，它消耗的时候并不随着某个变量的增长而增长，那么无论这类代码有多长，即使有几万几十万行，都可以用$O(1)$来表示它的时间复杂度。 - -#### 线性阶$O(n)$ - -$O(n)$，表示一个算法的性能会随着输入数据的大小变化而线性变化，如 - -```java -for (int i = 0; i < n; i++) { - j = i; - j++; -} -``` - -这段代码，for循环里面的代码会执行n遍，因此它消耗的时间是随着n的变化而变化的，因此这类代码都可以用$O(n)$来表示它的时间复杂度。 - -#### 平方阶$O(n^2)$ - -$O(n²)$ 表示一个算法的性能将会随着输入数据的增长而呈现出二次增长。最常见的就是对输入数据进行嵌套循环。如果嵌套层级不断深入的话，算法的性能将会变为立方阶$O(n^3)$，$O(n^4)$，$O(n^k)$以此类推 - -```java -for(x=1; i<=n; x++){ - for(i=1; i<=n; i++){ - j = i; - j++; - } -} -``` - -#### 指数阶$O(2^n)$ - -$O(2^n)$，表示一个算法的性能会随着输入数据的每次增加而增大两倍，典型的方法就是裴波那契数列的递归计算实现 - -```java -int Fibonacci(int number) -{ - if (number <= 1) return number; - - return Fibonacci(number - 2) + Fibonacci(number - 1); -} -``` - -#### 对数阶$O(logn)$ - -```java -int i = 1; -while(i arrs[j + 1]) { - int tmp = arrs[j]; - arrs[j] = arrs[j + 1]; - arrs[j + 1] = tmp; - } - } - } - - for (int arr : arrs) { - System.out.print(arr + " "); - } - } -} -``` - -嵌套循环，应该立马就可以得出这个算法的时间复杂度为 O(n²)。 - - - -## 选择排序 - -选择排序的思路是这样的：首先，找到数组中最小的元素，拎出来，将它和数组的第一个元素交换位置，第二步，在剩下的元素中继续寻找最小的元素，拎出来，和数组的第二个元素交换位置，如此循环，直到整个数组排序完成。 - -选择排序是一种简单直观的排序算法，无论什么数据进去都是 O(n²) 的时间复杂度。所以用到它的时候，数据规模越小越好。唯一的好处可能就是不占用额外的内存空间了吧。 - -### 1. 算法步骤 - -1. 首先在未排序序列中找到最小（大）元素，存放到排序序列的起始位置 -2. 再从剩余未排序元素中继续寻找最小（大）元素，然后放到已排序序列的末尾。 -3. 重复第二步，直到所有元素均排序完毕。 - -### 2. 动图演示 - -![img](https://miro.medium.com/max/551/1*OA7a3OGWmGMRJQmwkGIwAw.gif) - -[![动图演示](https://github.com/hustcc/JS-Sorting-Algorithm/raw/master/res/selectionSort.gif)](https://github.com/hustcc/JS-Sorting-Algorithm/blob/master/res/selectionSort.gif) - -```java -public class SelectionSort { - - public static void main(String[] args) { - int[] arrs = {5, 2, 4, 6, 1, 3}; - - for (int i = 0; i < arrs.length; i++) { - //最小元素下标 - int min = i; - for (int j = i +1; j < arrs.length; j++) { - if (arrs[j] < arrs[min]) { - min = j; - } - } - //交换位置 - int temp = arrs[i]; - arrs[i] = arrs[min]; - arrs[min] = temp; - } - for (int arr : arrs) { - System.out.println(arr); - } - } -} -``` - - - -## 插入排序 - -插入排序的代码实现虽然没有冒泡排序和选择排序那么简单粗暴，但它的原理应该是最容易理解的了，因为只要打过扑克牌的人都应该能够秒懂。插入排序是一种最简单直观的排序算法，它的工作原理是通过构建有序序列，对于未排序数据，在已排序序列中从后向前扫描，找到相应位置并插入。 - -插入排序和冒泡排序一样，也有一种优化算法，叫做拆半插入。 - -### 1. 算法步骤 - -1. 从第一个元素开始，该元素可以认为已经被排序； -2. 取出下一个元素，在已经排序的元素序列中从后向前扫描； -3. 如果该元素（已排序）大于新元素，将该元素移到下一位置； -4. 重复步骤3，直到找到已排序的元素小于或者等于新元素的位置； -5. 将新元素插入到该位置后； -6. 重复步骤2~5。 - -### 2. 动图演示 - -![img](https://miro.medium.com/max/500/1*onU9OmVftR5WeoLWh14iZw.gif) - -[![动图演示](https://github.com/hustcc/JS-Sorting-Algorithm/raw/master/res/insertionSort.gif)](https://github.com/hustcc/JS-Sorting-Algorithm/blob/master/res/insertionSort.gif) - -```java -public static void main(String[] args) { - int[] arr = {5, 2, 4, 6, 1, 3}; - // 从下标为1的元素开始选择合适的位置插入，因为下标为0的只有一个元素，默认是有序的 - for (int i = 1; i < arr.length; i++) { - - // 记录要插入的数据 - int tmp = arr[i]; - - // 从已经排序的序列最右边的开始比较，找到比其小的数 - int j = i; - while (j > 0 && tmp < arr[j - 1]) { - arr[j] = arr[j - 1]; - j--; - } - - // 存在比其小的数，插入 - if (j != i) { - arr[j] = tmp; - } - } - - for (int i : arr) { - System.out.println(i); - } -} -} -``` - - - -## 快速排序 - -快速排序的核心思想也是分治法，分而治之。它的实现方式是每次从序列中选出一个基准值，其他数依次和基准值做比较，比基准值大的放右边，比基准值小的放左边，然后再对左边和右边的两组数分别选出一个基准值，进行同样的比较移动，重复步骤，直到最后都变成单个元素，整个数组就成了有序的序列。 - -> 快速排序的最坏运行情况是 O(n²)，比如说顺序数列的快排。但它的平摊期望时间是 O(nlogn)，且 O(nlogn) 记号中隐含的常数因子很小，比复杂度稳定等于 O(nlogn) 的归并排序要小很多。所以，对绝大多数顺序性较弱的随机数列而言，快速排序总是优于归并排序。 - -### 1. 算法步骤 - -1. 从数列中挑出一个元素，称为 “基准”（pivot）; -2. 重新排序数列，所有元素比基准值小的摆放在基准前面，所有元素比基准值大的摆在基准的后面（相同的数可以到任一边）。在这个分区退出之后，该基准就处于数列的中间位置。这个称为分区（partition）操作； -3. 递归地（recursive）把小于基准值元素的子数列和大于基准值元素的子数列排序； - -递归的最底部情形，是数列的大小是零或一，也就是永远都已经被排序好了。虽然一直递归下去，但是这个算法总会退出，因为在每次的迭代（iteration）中，它至少会把一个元素摆到它最后的位置去。 - -### 2. 动图演示 - -![img](https://miro.medium.com/max/300/1*hk2TL8m8Kn1TVvewAbAclQ.gif) - -[![动图演示](https://github.com/hustcc/JS-Sorting-Algorithm/raw/master/res/quickSort.gif)](https://github.com/hustcc/JS-Sorting-Algorithm/blob/master/res/quickSort.gif) - -### 单边扫描 - -快速排序的关键之处在于切分，切分的同时要进行比较和移动，这里介绍一种叫做单边扫描的做法。 - -我们随意抽取一个数作为基准值，同时设定一个标记 mark 代表左边序列最右侧的下标位置，当然初始为 0 ，接下来遍历数组，如果元素大于基准值，无操作，继续遍历，如果元素小于基准值，则把 mark + 1 ，再将 mark 所在位置的元素和遍历到的元素交换位置，mark 这个位置存储的是比基准值小的数据，当遍历结束后，将基准值与 mark 所在元素交换位置即可。 - -```java -public static void sort(int[] arr) { - sort(arr， 0， arr.length - 1); -} - -private static void sort(int[] arr， int startIndex， int endIndex) { - if (endIndex <= startIndex) { - return; - } - //切分 - int pivotIndex = partitionV2(arr， startIndex， endIndex); - sort(arr， startIndex， pivotIndex-1); - sort(arr， pivotIndex+1， endIndex); -} - -private static int partition(int[] arr， int startIndex， int endIndex) { - int pivot = arr[startIndex];//取基准值 - int mark = startIndex;//Mark初始化为起始下标 - - for(int i=startIndex+1; i<=endIndex; i++){ - if(arr[i]= right) { - break; - } - - //交换左右数据 - int temp = arr[left]; - arr[left] = arr[right]; - arr[right] = temp; - } - - //将基准值插入序列 - int temp = arr[startIndex]; - arr[startIndex] = arr[right]; - arr[right] = temp; - return right; -} -``` - - - - - -## 希尔排序 - -希尔排序这个名字，来源于它的发明者希尔，也称作“缩小增量排序”，是插入排序的一种更高效的改进版本。但希尔排序是非稳定排序算法。 - -希尔排序是基于插入排序的以下两点性质而提出改进方法的： - -- 插入排序在对几乎已经排好序的数据操作时，效率高，即可以达到线性排序的效率； -- 但插入排序一般来说是低效的，因为插入排序每次只能将数据移动一位； - -希尔排序的基本思想是：先将整个待排序的记录序列分割成为若干子序列分别进行直接插入排序，待整个序列中的记录“基本有序”时，再对全体记录进行依次直接插入排序。 - -### 1. 算法步骤 - -1. 选择一个增量序列 t1，t2，……，tk，其中 ti > tj, tk = 1； -2. 按增量序列个数 k，对序列进行 k 趟排序； -3. 每趟排序，根据对应的增量 ti，将待排序列分割成若干长度为 m 的子序列，分别对各子表进行直接插入排序。仅增量因子为 1 时，整个序列作为一个表来处理，表长度即为整个序列的长度。 - -### 2. 动图演示 - -![img](https://mmbiz.qpic.cn/mmbiz_gif/951TjTgiabkzow2ORRzgpfHIGAKIAWlXm6GpRDRhiczgOdibbGBtpibtIhX4YRzibicUyEOSVh3JZBHtiaZPN30X1WOhA/640?wx_fmt=gif&tp=webp&wxfrom=5&wx_lazy=1) - - - -## 归并排序 - -归并排序（Merge sort）是建立在归并操作上的一种有效的排序算法。该算法是采用分治法（Divide and Conquer）的一个非常典型的应用。 - -作为一种典型的分而治之思想的算法应用，归并排序的实现由两种方法： - -- 自上而下的递归（所有递归的方法都可以用迭代重写，所以就有了第 2 种方法）； -- 自下而上的迭代； - -在《数据结构与算法 JavaScript 描述》中，作者给出了自下而上的迭代方法。但是对于递归法，作者却认为： - -> However, it is not possible to do so in JavaScript, as the recursion goes too deep for the language to handle. -> -> 然而，在 JavaScript 中这种方式不太可行，因为这个算法的递归深度对它来讲太深了。 - -说实话，我不太理解这句话。意思是 JavaScript 编译器内存太小，递归太深容易造成内存溢出吗？还望有大神能够指教。 - -和选择排序一样，归并排序的性能不受输入数据的影响，但表现比选择排序好的多，因为始终都是 O(nlogn) 的时间复杂度。代价是需要额外的内存空间。 - -### 2. 算法步骤 - -1. 申请空间，使其大小为两个已经排序序列之和，该空间用来存放合并后的序列； -2. 设定两个指针，最初位置分别为两个已经排序序列的起始位置； -3. 比较两个指针所指向的元素，选择相对小的元素放入到合并空间，并移动指针到下一位置； -4. 重复步骤 3 直到某一指针达到序列尾； -5. 将另一序列剩下的所有元素直接复制到合并序列尾。 - -### 3. 动图演示 - -![img](https://miro.medium.com/max/300/1*fE7yGW2WPaltJWo6OnZ8LQ.gif) - -[![动图演示](https://github.com/hustcc/JS-Sorting-Algorithm/raw/master/res/mergeSort.gif)](https://github.com/hustcc/JS-Sorting-Algorithm/blob/master/res/mergeSort.gif) - - - - - - - -## 堆排序 - -堆排序（Heapsort）是指利用堆这种数据结构所设计的一种排序算法。堆积是一个近似完全二叉树的结构，并同时满足堆积的性质：即子结点的键值或索引总是小于（或者大于）它的父节点。堆排序可以说是一种利用堆的概念来排序的选择排序。分为两种方法： - -1. 大顶堆：每个节点的值都大于或等于其子节点的值，在堆排序算法中用于升序排列； -2. 小顶堆：每个节点的值都小于或等于其子节点的值，在堆排序算法中用于降序排列； - -堆排序的平均时间复杂度为 Ο(nlogn)。 - -### 1. 算法步骤 - -1. 将待排序序列构建成一个堆 H[0……n-1]，根据（升序降序需求）选择大顶堆或小顶堆； -2. 把堆首（最大值）和堆尾互换； -3. 把堆的尺寸缩小 1，并调用 shift_down(0)，目的是把新的数组顶端数据调整到相应位置； -4. 重复步骤 2，直到堆的尺寸为 1。 - -### 2. 动图演示 - -[![动图演示](https://github.com/hustcc/JS-Sorting-Algorithm/raw/master/res/heapSort.gif)](https://github.com/hustcc/JS-Sorting-Algorithm/blob/master/res/heapSort.gif) - - - - - -## 计数排序 - -计数排序的核心在于将输入的数据值转化为键存储在额外开辟的数组空间中。作为一种线性时间复杂度的排序，计数排序要求输入的数据必须是有确定范围的整数。 - -### 1. 动图演示 - -[![动图演示](https://github.com/hustcc/JS-Sorting-Algorithm/raw/master/res/countingSort.gif)](https://github.com/hustcc/JS-Sorting-Algorithm/blob/master/res/countingSort.gif) - - - - - -## 桶排序 - -桶排序是计数排序的升级版。它利用了函数的映射关系，高效与否的关键就在于这个映射函数的确定。为了使桶排序更加高效，我们需要做到这两点： - -1. 在额外空间充足的情况下，尽量增大桶的数量 -2. 使用的映射函数能够将输入的 N 个数据均匀的分配到 K 个桶中 - -同时，对于桶中元素的排序，选择何种比较排序算法对于性能的影响至关重要。 - -### 1. 什么时候最快 - -当输入的数据可以均匀的分配到每一个桶中。 - -### 2. 什么时候最慢 - -当输入的数据被分配到了同一个桶中。 - - - -## 基数排序 - -基数排序是一种非比较型整数排序算法，其原理是将整数按位数切割成不同的数字，然后按每个位数分别比较。由于整数也可以表达字符串（比如名字或日期）和特定格式的浮点数，所以基数排序也不是只能使用于整数。 - -### 1. 基数排序 vs 计数排序 vs 桶排序 - -基数排序有两种方法： - -这三种排序算法都利用了桶的概念，但对桶的使用方法上有明显差异案例看大家发的： - -- 基数排序：根据键值的每位数字来分配桶； -- 计数排序：每个桶只存储单一键值； -- 桶排序：每个桶存储一定范围的数值； - -### 2. LSD 基数排序动图演示 - -[![动图演示](https://github.com/hustcc/JS-Sorting-Algorithm/raw/master/res/radixSort.gif)](https://github.com/hustcc/JS-Sorting-Algorithm/blob/master/res/radixSort.gif) - diff --git "a/docs/leetcode/\344\270\244\346\225\260\344\271\213\345\222\214.md" "b/docs/leetcode/\344\270\244\346\225\260\344\271\213\345\222\214.md" deleted file mode 100644 index 0b8454c746..0000000000 --- "a/docs/leetcode/\344\270\244\346\225\260\344\271\213\345\222\214.md" +++ /dev/null @@ -1,107 +0,0 @@ -![Multicolored Abacus Photography](https://images.pexels.com/photos/1019470/abacus-mathematics-addition-subtraction-1019470.jpeg?auto=compress&cs=tinysrgb&h=750&w=1260) - -# 1. 两数之和 - -题目来源于 LeetCode 上第 1 号问题：两数之和。题目难度为 Easy 。 - -### 题目描述 - -给定一个整数数组 `nums` 和一个目标值 `target`，请你在该数组中找出和为目标值的那两个整数，并返回他们的数组下标。 - -你可以假设每种输入只会对应一个答案。但是，你不能重复利用这个数组中同样的元素。 - -**示例:** - -``` -给定 nums = [2, 7, 11, 15], target = 9 - -因为 nums[0] + nums[1] = 2 + 7 = 9 -所以返回 [0, 1] -``` - -### 代码实现 - -#### 方法一：暴力法 - -暴力法很简单，遍历每个元素 x，并查找是否存在一个值与`xtarget−x` 相等的目标元素。 - -```java -class Solution { - public int[] twoSum(int[] nums, int target) { - for (int i = 0; i < nums.length; i++) { - for (int j = i + 1; j < nums.length; j++) { - if (nums[j] == target - nums[i]) { - return new int[] { i, j }; - } - } - } - throw new IllegalArgumentException("No two sum solution"); - } -} -``` - -复杂度分析： - -- 时间复杂度 $O(n^2)$： - 对于每个元素，我们试图通过遍历数组的其余部分来寻找它所对应的目标元素，这将耗费 $O(n)$ 的时间。因此时间复杂度为 $O(n^2)$ -- 空间复杂度：$O(1)$ - -#### 方法二：两遍哈希表 - -为了对运行时间复杂度进行优化，我们需要一种更有效的方法来检查数组中是否存在目标元素。如果存在，我们需要找出它的索引。保持数组中的每个元素与其索引相互对应的最好方法是什么？哈希表。 - -通过以空间换取速度的方式，我们可以将查找时间从 $O(n)$降低到 $O(1)$。哈希表正是为此目的而构建的，它支持以近似恒定的时间进行快速查找。我用“近似”来描述，是因为一旦出现冲突，查找用时可能会退化到 $O(n)$。 - -一个简单的实现使用了两次迭代。在第一次迭代中，我们将每个元素的值和它的索引添加到表中。然后，在第二次迭代中，我们将检查每个元素所对应的目标元素$（target - nums[i]）$是否存在于表中。注意，该目标元素不能是 $nums[i]$ 本身！ - -```java -class Solution { - public int[] twoSum(int[] nums, int target) { - Map map = new HashMap<>(); - for (int i = 0; i < nums.length; i++) { - map.put(nums[i], i); - } - for (int i = 0; i < nums.length; i++) { - int complement = target - nums[i]; - if (map.containsKey(complement) && map.get(complement) != i) { - return new int[] { i, map.get(complement) }; - } - } - throw new IllegalArgumentException("No two sum solution"); - } -} -``` - -复杂度分析： - -- 时间复杂度：$O(n)$， - 我们把包含有 nn 个元素的列表遍历两次。由于哈希表将查找时间缩短到 $O(1)$ ，所以时间复杂度为 $O(n)$。 -- 空间复杂度：$O(n)$， - 所需的额外空间取决于哈希表中存储的元素数量，该表中存储了 nn 个元素。 - -#### 方法三：一遍哈希表 - -事实证明，我们可以一次完成。在进行迭代并将元素插入到表中的同时，我们还会回过头来检查表中是否已经存在当前元素所对应的目标元素。如果它存在，那我们已经找到了对应解，并立即将其返回。 - -```java -class Solution { - public int[] twoSum(int[] nums, int target) { - Map map = new HashMap<>(); - for (int i = 0; i < nums.length; i++) { - int complement = target - nums[i]; - if (map.containsKey(complement)) { - return new int[] { map.get(complement), i }; - } - map.put(nums[i], i); - } - throw new IllegalArgumentException("No two sum solution"); - } -} -``` - -复杂度分析： - -- 时间复杂度：$O(n)$ - 我们只遍历了包含有 nn 个元素的列表一次。在表中进行的每次查找只花费 $O(1)$的时间。 -- 空间复杂度：$O(n)$ - 所需的额外空间取决于哈希表中存储的元素数量，该表最多需要存储 n 个元素。 \ No newline at end of file diff --git a/docs/libs/prism-markdown.min.js b/docs/libs/prism-markdown.min.js deleted file mode 100644 index 012565b4a6..0000000000 --- a/docs/libs/prism-markdown.min.js +++ /dev/null @@ -1 +0,0 @@ -Prism.languages.markdown=Prism.languages.extend("markup",{}),Prism.languages.insertBefore("markdown","prolog",{blockquote:{pattern:/^>(?:[\t ]*>)*/m,alias:"punctuation"},code:[{pattern:/^(?: {4}|\t).+/m,alias:"keyword"},{pattern:/``.+?``|`[^`\n]+`/,alias:"keyword"}],title:[{pattern:/\w+.*(?:\r?\n|\r)(?:==+|--+)/,alias:"important",inside:{punctuation:/==+$|--+$/}},{pattern:/(^\s*)#+.+/m,lookbehind:!0,alias:"important",inside:{punctuation:/^#+|#+$/}}],hr:{pattern:/(^\s*)([*-])(?:[\t ]*\2){2,}(?=\s*$)/m,lookbehind:!0,alias:"punctuation"},list:{pattern:/(^\s*)(?:[*+-]|\d+\.)(?=[\t ].)/m,lookbehind:!0,alias:"punctuation"},"url-reference":{pattern:/!?\[[^\]]+\]:[\t ]+(?:\S+|<(?:\\.|[^>\\])+>)(?:[\t ]+(?:"(?:\\.|[^"\\])*"|'(?:\\.|[^'\\])*'|\((?:\\.|[^)\\])*\)))?/,inside:{variable:{pattern:/^(!?\[)[^\]]+/,lookbehind:!0},string:/(?:"(?:\\.|[^"\\])*"|'(?:\\.|[^'\\])*'|\((?:\\.|[^)\\])*\))$/,punctuation:/^[\[\]!:]|[<>]/},alias:"url"},bold:{pattern:/(^|[^\\])(\*\*|__)(?:(?:\r?\n|\r)(?!\r?\n|\r)|.)+?\2/,lookbehind:!0,inside:{punctuation:/^\*\*|^__|\*\*$|__$/}},italic:{pattern:/(^|[^\\])([*_])(?:(?:\r?\n|\r)(?!\r?\n|\r)|.)+?\2/,lookbehind:!0,inside:{punctuation:/^[*_]|[*_]$/}},url:{pattern:/!?\[[^\]]+\](?:\([^\s)]+(?:[\t ]+"(?:\\.|[^"\\])*")?\)| ?\[[^\]\n]*\])/,inside:{variable:{pattern:/(!?\[)[^\]]+(?=\]$)/,lookbehind:!0},string:{pattern:/"(?:\\.|[^"\\])*"(?=\)$)/}}}}),Prism.languages.markdown.bold.inside.url=Prism.languages.markdown.url,Prism.languages.markdown.italic.inside.url=Prism.languages.markdown.url,Prism.languages.markdown.bold.inside.italic=Prism.languages.markdown.italic,Prism.languages.markdown.italic.inside.bold=Prism.languages.markdown.bold; \ No newline at end of file diff --git a/docs/libs/prism-scss.min.js b/docs/libs/prism-scss.min.js deleted file mode 100644 index 245db662e1..0000000000 --- a/docs/libs/prism-scss.min.js +++ /dev/null @@ -1 +0,0 @@ -Prism.languages.scss=Prism.languages.extend("css",{comment:{pattern:/(^|[^\\])(?:\/\*[\s\S]*?\*\/|\/\/.*)/,lookbehind:!0},atrule:{pattern:/@[\w-]+(?:\([^()]+\)|[^(])*?(?=\s+[{;])/,inside:{rule:/@[\w-]+/}},url:/(?:[-a-z]+-)*url(http://github.com/?=\()/i,selector:{pattern:/(?=\S)[^@;{}()]?(?:[^@;{}()]|&|#\{\$[-\w]+\})+(?=\s*\{(?:\}|\s|[^}]+[:{][^}]+))/m,inside:{parent:{pattern:/&/,alias:"important"},placeholder:/%[-\w]+/,variable:/\$[-\w]+|#\{\$[-\w]+\}/}}}),Prism.languages.insertBefore("scss","atrule",{keyword:[/@(?:if|else(?: if)?|for|each|while|import|extend|debug|warn|mixin|include|function|return|content)/i,{pattern:/( +)(?:from|through)(?= )/,lookbehind:!0}]}),Prism.languages.scss.property={pattern:/(?:[\w-]|\$[-\w]+|#\{\$[-\w]+\})+(?=\s*:)/i,inside:{variable:/\$[-\w]+|#\{\$[-\w]+\}/}},Prism.languages.insertBefore("scss","important",{variable:/\$[-\w]+|#\{\$[-\w]+\}/}),Prism.languages.insertBefore("scss","function",{placeholder:{pattern:/%[-\w]+/,alias:"selector"},statement:{pattern:/\B!(?:default|optional)\b/i,alias:"keyword"},"boolean":/\b(?:true|false)\b/,"null":/\bnull\b/,operator:{pattern:/(\s)(?:[-+*\/%]|[=!]=|<=?|>=?|and|or|not)(?=\s)/,lookbehind:!0}}),Prism.languages.scss.atrule.inside.rest=Prism.languages.scss; \ No newline at end of file diff --git a/docs/logging/Java-Logging.md b/docs/logging/Java-Logging.md deleted file mode 100644 index f877b0a31f..0000000000 --- a/docs/logging/Java-Logging.md +++ /dev/null @@ -1,194 +0,0 @@ -## Java日志 - -日志就是记录程序的运行轨迹，方便查找关键信息，也方便快速定位解决问题。 - - - -## Java常用框架 - -- **Jul** (Java Util Logging) : 自Java1.4以来，Java在Java.util中提供的一个内置框架，也常称为JDKLog、jdk-logging。 -- **Log4j** : Apache Log4j是一个基于Java的日志记录工具。它是由Ceki Gülcü首创的，现在则是Apache软件基金会的一个项目。 -- **Log4j 2** : Apache Log4j 2是apache开发的一款Log4j的升级产品，Log4j 2与Log4j 1发生了很大的变化，Log4j 2不兼容Log4j 1。 -- **Logback** : 一套日志组件的实现(Slf4j阵营)。 -- **tinylog** : 一个轻量级的日志框架 -- **Apache Commons Logging** : Apache基金会所属的项目，是一套Java日志接口，之前叫 Jakarta Commons Logging，后更名为Commons Logging。 -- **Slf4j** : Simple Logging Facade for Java，类似于Commons Logging，是一套简易Java日志门面，本身并无日志的实现。（Simple Logging Facade for Java，缩写Slf4j）。 - - - -JUL、LOG4J1、LOG4J2、LOGBACK是**日志实现框架**，而Commons Logging和SLF4J是**日志实现门面**，可以理解为一个适配器，**可以将你的应用程序从日志框架中解耦**。 - -【强制】应用中不可直接使用日志系统（Log4j、Logback）中的 API，而应依赖使用日志框架 SLF4J 中的 API，使用门面模式的日志框架，有利于维护和各个类的日志处理方式统一。 import org.slf4j.Logger; import org.slf4j.LoggerFactory; private static final Logger logger = LoggerFactory.getLogger(Abc.class); - - - -## Java常用日志框架历史 - -- 1996年早期，欧洲安全电子市场项目组决定编写它自己的程序跟踪API(Tracing API)。经过不断的完善，这个API终于成为一个十分受欢迎的Java日志软件包，即Log4j。后来Log4j成为Apache基金会项目中的一员。 - -- 期间Log4j近乎成了Java社区的日志标准。据说Apache基金会还曾经建议Sun引入Log4j到java的标准库中，但Sun拒绝了。 - -- 2002年Java1.4发布，Sun推出了自己的日志库JUL(Java Util Logging),其实现基本模仿了Log4j的实现。在JUL出来以前，Log4j就已经成为一项成熟的技术，使得Log4j在选择上占据了一定的优势。 - -- 接着，Apache推出了Jakarta Commons Logging，JCL只是定义了一套日志接口(其内部也提供一个Simple Log的简单实现)，支持运行时动态加载日志组件的实现，也就是说，在你应用代码里，只需调用Commons Logging的接口，底层实现可以是Log4j，也可以是Java Util Logging。 - -- 后来(2006年)，Ceki Gülcü不适应Apache的工作方式，离开了Apache。然后先后创建了Slf4j(日志门面接口，类似于Commons Logging)和Logback(Slf4j的实现)两个项目，并回瑞典创建了QOS公司，QOS官网上是这样描述Logback的：The Generic，Reliable Fast&Flexible Logging Framework(一个通用，可靠，快速且灵活的日志框架)。 - -- 现今，Java日志领域被划分为两大阵营：Commons Logging阵营和Slf4j阵营。 - Commons Logging在Apache大树的笼罩下，有很大的用户基数。但有证据表明，形式正在发生变化。2013年底有人分析了GitHub上30000个项目，统计出了最流行的100个Libraries，可以看出Slf4j的发展趋势更好： - -- Apache眼看有被Logback反超的势头，于2012-07重写了Log4j 1.x，成立了新的项目Log4j 2, Log4j 2具有Logback的所有特性。 - - - - -![](http://cnblogpic.oss-cn-qingdao.aliyuncs.com/blogpic/java_log/java_populor_jar.png) - -## java常用日志框架关系 - -- Log4j 2与Log4j 1发生了很大的变化，Log4j 2不兼容Log4j 1。 -- Commons Logging和Slf4j是日志门面(门面模式是软件工程中常用的一种软件设计模式，也被称为正面模式、外观模式。它为子系统中的一组接口提供一个统一的高层接口，使得子系统更容易使用)。Log4j和Logback则是具体的日志实现方案。可以简单的理解为接口与接口的实现，调用者只需要关注接口而无需关注具体的实现，做到解耦。 -- 比较常用的组合使用方式是Slf4j与Logback组合使用，Commons Logging与Log4j组合使用。 -- Logback必须配合Slf4j使用。由于Logback和Slf4j是同一个作者，其兼容性不言而喻。 - - - -## Commons Logging与Slf4j实现机制对比 - -#### Commons Logging实现机制 - -> Commons Logging是通过动态查找机制，在程序运行时，使用自己的ClassLoader寻找和载入本地具体的实现。详细策略可以查看commons-logging-*.jar包中的org.apache.commons.logging.impl.LogFactoryImpl.java文件。由于Osgi不同的插件使用独立的ClassLoader，Osgi的这种机制保证了插件互相独立, 其机制限制了Commons Logging在Osgi中的正常使用。 - -#### Slf4j实现机制 - -> Slf4j在编译期间，静态绑定本地的Log库，因此可以在Osgi中正常使用。它是通过查找类路径下org.slf4j.impl.StaticLoggerBinder，然后在StaticLoggerBinder中进行绑定。 - - - -## 项目中选择日志框架选择 - -如果是在一个新的项目中建议使用Slf4j与Logback组合，这样有如下的几个优点。 - -- Slf4j实现机制决定Slf4j限制较少，使用范围更广。由于Slf4j在编译期间，静态绑定本地的LOG库使得通用性要比Commons Logging要好。 -- Logback拥有更好的性能。Logback声称：某些关键操作，比如判定是否记录一条日志语句的操作，其性能得到了显著的提高。这个操作在Logback中需要3纳秒，而在Log4J中则需要30纳秒。LogBack创建记录器（logger）的速度也更快：13毫秒，而在Log4J中需要23毫秒。更重要的是，它获取已存在的记录器只需94纳秒，而Log4J需要2234纳秒，时间减少到了1/23。跟JUL相比的性能提高也是显著的。 -- Commons Logging开销更高 - -``` -# 在使Commons Logging时为了减少构建日志信息的开销，通常的做法是 -if(log.isDebugEnabled()){ - log.debug("User name： " + - user.getName() + " buy goods id ：" + good.getId()); -} - -# 在Slf4j阵营，你只需这么做： -log.debug("User name：{} ,buy goods id ：{}", user.getName(),good.getId()); - -# 也就是说，Slf4j把构建日志的开销放在了它确认需要显示这条日志之后，减少内存和Cup的开销，使用占位符号，代码也更为简洁 -``` - -- Logback文档免费。Logback的所有文档是全面免费提供的，不象Log4J那样只提供部分免费文档而需要用户去购买付费文档。 - - - -## Java日志组件 - -[**Loggers**](http://www.loggly.com/ultimate-guide/logging/java-logging-basics/#loggers)**：**记录器，Logger 负责捕捉事件并将其发送给合适的 Appender。 - -[**Loggers**](http://www.loggly.com/ultimate-guide/logging/java-logging-basics/#loggers)**：**记录器，Logger 负责捕捉事件并将其发送给合适的 Appender。 - -[**Appenders**](http://www.loggly.com/ultimate-guide/logging/java-logging-basics/#appenders)**：**也被称为 Handlers，处理器，负责将日志事件记录到目标位置。在将日志事件输出之前， Appenders 使用Layouts来对事件进行格式化处理。 - -[**Layouts**](http://www.loggly.com/ultimate-guide/logging/java-logging-basics/#layouts)**：**也被称为 Formatters，格式化器，它负责对日志事件中的数据进行转换和格式化。Layouts 决定了数据在一条日志记录中的最终形式。 - -当 Logger 记录一个事件时，它将事件转发给适当的 Appender。然后 Appender 使用 Layout 来对日志记录进行格式化，并将其发送给控制台、文件或者其它目标位置。另外，Filters 可以让你进一步指定一个 Appender 是否可以应用在一条特定的日志记录上。在日志配置中，Filters 并不是必需的，但可以让你更灵活地控制日志消息的流动。 - -![]( https://logglyultimate.wpengine.com/wp-content/uploads/2015/09/Picture1-2.png ) - - - -## Java日志级别 - -不同的日志框架，级别也会有些差异 - -**log4j** —— **OFF、FATAL、ERROR、WARN、INFO、DEBUG、TRACE、 ALL** - -**logback** —— **OFF、ERROR、WARN、INFO、DEBUG、TRACE、 ALL** - -| 日志级别 | 描述 | -| -------- | -------------------------------------------------- | -| OFF | 关闭：最高级别，不输出日志。 | -| FATAL | 致命：输出非常严重的可能会导致应用程序终止的错误。 | -| ERROR | 错误：输出错误，但应用还能继续运行。 | -| WARN | 警告：输出可能潜在的危险状况。 | -| INFO | 信息：输出应用运行过程的详细信息。 | -| DEBUG | 调试：输出更细致的对调试应用有用的信息。 | -| TRACE | 跟踪：输出更细致的程序运行轨迹。 | -| ALL | 所有：输出所有级别信息。 | - - - -## SLF4J绑定日志框架 - -![](http://cnblogpic.oss-cn-qingdao.aliyuncs.com/blogpic/java_log/slf4j-bind.png) - -![](http://www.slf4j.org/images/concrete-bindings.png) - - - -## 阿里Java开发手册——日志规约 - -1. 【强制】应用中不可直接使用日志系统（Log4j、Logback）中的 API，而应依赖使用日志框架 SLF4J 中的 API，使用门面模式的日志框架，有利于维护和各个类的日志处理方式统一。 - - ```java - import org.slf4j.Logger; - import org.slf4j.LoggerFactory; - - private static final Logger logger = LoggerFactory.getLogger(Abc.class); - ``` - -2. 【强制】日志文件至少保存 15 天，因为有些异常具备以“周”为频次发生的特点。 - -3. 【强制】应用中的扩展日志（如打点、临时监控、访问日志等）命名方式： appName_logType_logName.log。 logType:日志类型，如 stats/monitor/access 等；logName:日志描述。这种命名的好处： 通过文件名就可知道日志文件属于什么应用，什么类型，什么目的，也有利于归类查找。 正例：mppserver 应用中单独监控时区转换异常，如： mppserver_monitor_timeZoneConvert.log 说明：推荐对日志进行分类，如将错误日志和业务日志分开存放，便于开发人员查看，也便于 通过日志对系统进行及时监控。 - -4. 【强制】对 trace/debug/info 级别的日志输出，必须使用条件输出形式或者使用占位符的方 式。 说明：logger.debug("Processing trade with id: " + id + " and symbol: " + symbol); 如果日志级别是 warn，上述日志不会打印，但是会执行字符串拼接操作，如果 symbol 是对象， 会执行 toString()方法，浪费了系统资源，执行了上述操作，最终日志却没有打印。 - - 正例：（条件）建设采用如下方式 - - ```java - if (logger.isDebugEnabled()) { - logger.debug("Processing trade with id: " + id + " and symbol: " + symbol); - } - ``` - - 正例：（占位符） - - ```java - logger.debug("Processing trade with id: {} and symbol : {} ", id, symbol); - ``` - -5. 【强制】避免重复打印日志，浪费磁盘空间，务必在 log4j.xml 中设置 additivity=false。 正例： - -6. 【强制】异常信息应该包括两类信息：案发现场信息和异常堆栈信息。如果不处理，那么通过 关键字 throws 往上抛出。 正例：logger.error(各类参数或者对象 toString() + "_" + e.getMessage(), e); - -7. 【推荐】谨慎地记录日志。生产环境禁止输出 debug 日志；有选择地输出 info 日志；如果使 用 warn 来记录刚上线时的业务行为信息，一定要注意日志输出量的问题，避免把服务器磁盘 撑爆，并记得及时删除这些观察日志。 说明：大量地输出无效日志，不利于系统性能提升，也不利于快速定位错误点。记录日志时请 思考：这些日志真的有人看吗？看到这条日志你能做什么？能不能给问题排查带来好处？ - -8. 【推荐】可以使用 warn 日志级别来记录用户输入参数错误的情况，避免用户投诉时，无所适 从。如非必要，请不要在此场景打出 error 级别，避免频繁报警。 说明：注意日志输出的级别，error 级别只记录系统逻辑出错、异常或者重要的错误信息。 - -9. 【推荐】尽量用英文来描述日志错误信息，如果日志中的错误信息用英文描述不清楚的话使用 中文描述即可，否则容易产生歧义。国际化团队或海外部署的服务器由于字符集问题，【强制】 使用全英文来注释和描述日志错误信息。 - - - -> [Ultimate Guide to Logging](https://www.loggly.com/ultimate-guide/java-logging-basics/#layouts) -> -> [《Java常用日志框架介绍》](https://www.cnblogs.com/chenhongliang/p/5312517.html) - - - - - - - - - - - diff --git a/docs/message-queue/Kafka/Hello-Kafka.md b/docs/message-queue/Kafka/Hello-Kafka.md deleted file mode 100644 index 519af290c8..0000000000 --- a/docs/message-queue/Kafka/Hello-Kafka.md +++ /dev/null @@ -1,569 +0,0 @@ -## 1. Kafka概述 - -### 1.1 定义 - -Kafka 是一个**分布式**的基于**发布/订阅模式的消息队列**（Message Queue），主要应用于大数据实时处理领域。 - - - -### 1.2 消息队列 - -#### 1.2.1 传统消息队列的应用场景 - -![](https://imgkr.cn-bj.ufileos.com/f6434adf-be2e-48d3-a2fa-95c6c229f846.png) - -#### 1.2.2 为什么需要消息队列 - -1. **解耦**： 允许你独立的扩展或修改两边的处理过程，只要确保它们遵守同样的接口约束。 -2. **冗余**：消息队列把数据进行持久化直到它们已经被完全处理，通过这一方式规避了数据丢失风险。许多消息队列所采用的"插入-获取-删除"范式中，在把一个消息从队列中删除之前，需要你的处理系统明确的指出该消息已经被处理完毕，从而确保你的数据被安全的保存直到你使用完毕。 -3. **扩展性**： 因为消息队列解耦了你的处理过程，所以增大消息入队和处理的频率是很容易的，只要另外增加处理过程即可。 -4. **灵活性 & 峰值处理能力**： 在访问量剧增的情况下，应用仍然需要继续发挥作用，但是这样的突发流量并不常见。 如果为以能处理这类峰值访问为标准来投入资源随时待命无疑是巨大的浪费。使用消息队列能够使关键组件顶住突发的访问压力，而不会因为突发的超负荷的请求而完全崩溃。 -5. **可恢复性**： 系统的一部分组件失效时，不会影响到整个系统。消息队列降低了进程间的耦合度，所以即使一个处理消息的进程挂掉，加入队列中的消息仍然可以在系统恢复后被处理。 -6. **顺序保证**： 在大多使用场景下，数据处理的顺序都很重要。大部分消息队列本来就是排序的，并且能保证数据会按照特定的顺序来处理。（Kafka 保证一个 Partition 内的消息的有序性） -7. **缓冲**： 有助于控制和优化数据流经过系统的速度， 解决生产消息和消费消息的处理速度不一致的情况。 -8. **异步通信**： 很多时候，用户不想也不需要立即处理消息。消息队列提供了异步处理机制，允许用户把一个消息放入队列，但并不立即处理它。想向队列中放入多少消息就放多少，然后在需要的时候再去处理它们。 - - - -#### 1.2.3 消息队列的两种模式 - -- **点对点模式**（一对一，消费者主动拉取数据，消息收到后消息清除） - - 消息生产者生产消息发送到 Queue 中，然后消息消费者从 Queue 中取出并且消费消息。 消息被消费以后，queue 中不再有存储，所以消息消费者不可能消费到已经被消费的消息。 Queue 支持存在多个消费者，但是对一个消息而言，只会有一个消费者可以消费。 - - ![图片：mrbird.cc](https://mrbird.cc/img/QQ20200324-202328@2x.png) - -- **发布/订阅模式**（一对多，数据生产后，推送给所有订阅者） - - 消息生产者（发布）将消息发布到 topic 中，同时有多个消息消费者（订阅）消费该消 息。和点对点方式不同，发布到 topic 的消息会被所有订阅者消费。 - - ![图片：mrbird.cc](https://mrbird.cc/img/QQ20200324-203201@2x.png) - - - -### 1.3 Kafka 基础架构图 - -![图片：mrbird.cc](https://mrbird.cc/img/QQ20200324-210522@2x.png) - -- Producer ：消息生产者，就是向 kafka broker 发消息的客户端； -- Consumer ：消息消费者，向 kafka broker 取消息的客户端； -- Consumer Group （CG）：消费者组，由多个 consumer 组成。**消费者组内每个消费者负责消费不同分区的数据，一个分区只能由一个组内消费者消费；消费者组之间互不影响**。所有的消费者都属于某个消费者组，即消费者组是逻辑上的一个订阅者。 -- Broker ：一台 kafka 服务器就是一个 broker。一个集群由多个 broker 组成。一个 broker 可以容纳多个 topic。 -- Topic ：可以理解为一个队列，Kafka 的消息通过 Topics(主题) 进行分类，生产者和消费者面向的都是一个 topic； -- Partition：为了实现扩展性，一个非常大的 topic 可以分布到多个 broker（即服务器）上， 一个 topic 可以分为多个 partition，每个 partition 是一个有序的队列； partition 中的每条消息 都会被分配一个有序的 id（ offset）。 kafka 只保证按一个 partition 中的顺序将消息发给 consumer，不保证一个 topic 的整体（多个 partition 间）的顺序； -- Replica：副本，为保证集群中的某个节点发生故障时，该节点上的 partition 数据不丢失，且 kafka 仍然能够继续工作，kafka 提供了副本机制，一个 topic 的每个分区都有若干个副本， 一个 leader 和若干个 follower。 -- leader：每个分区多个副本的“主”，生产者发送数据的对象，以及消费者消费数据的对象都是 leader。 -- follower：每个分区多个副本中的“从”，实时从 leader 中同步数据，保持和 leader 数据的同步。leader 发生故障时，某个 follower 会成为新的 follower。 -- Offset： kafka 的存储文件都是按照 offset.kafka 来命名，用 offset 做名字的好处是方便查找。例如你想找位于 2049 的位置，只要找到 2048.kafka 的文件即可。当然 the first offset 就是 00000000000.kafka。 - ------- - - - -## 2. Hello Kafka - -### 2.1 动起手来 - -[Quickstart]( ) - -[中文版入门指南]( ) - -### 2.2 基本概念(官方介绍翻译) - -Kafka是一个分布式的流处理平台。是支持分区的（partition）、多副本的（replica），基于 ZooKeeper 协调的分布式消息系统，它的最大的特性就是可以实时的处理大量数据以满足各种需求场景：比如基于 hadoop 的批处理系统、低延迟的实时系统、storm/Spark 流式处理引擎，web/nginx 日志、访问日志，消息服务等等 - -#### 有三个关键能力 - -- 它可以让你发布和订阅记录流。在这方面，它类似于一个消息队列或企业消息系统 -- 它可以让你持久化收到的记录流，从而具有容错能力。 -- 它可以让你处理收到的记录流。 - -#### 应用于两大类应用 - -- 构建实时的流数据管道，可靠地获取系统和应用程序之间的数据。 -- 构建实时流的应用程序，对数据流进行转换或反应。 - -想要了解 Kafka 如何具有这些能力，首先，明确几个概念： - -- Kafka 作为一个集群运行在一个或多个服务器上 -- Kafka 集群存储的消息是以主题（topics）为类别记录的 -- 每个消息记录包含一个键，一个值和时间戳 - -#### Kafka有五个核心API： - -- **Producer API** 允许应用程序发布记录流至一个或多个Kafka的话题(Topics) - -- **Consumer API** 允许应用程序订阅一个或多个主题，并处理这些主题接收到的记录流 - -- **Streams API** 允许应用程序充当流处理器（stream processor），从一个或多个主题获取输入流，并生产一个输出流至一个或多个的主题，能够有效地变换输入流为输出流 - -- **Connector API** 允许构建和运行可重用的生产者或消费者，能够把 Kafka主题连接到现有的应用程序或数据系统。例如，一个连接到关系数据库的连接器(connector)可能会获取每个表的变化 - -- **Admin API** 允许管理和检查主题、brokes 和其他 Kafka 对象。（这个是新版本才有的） - - ![img](../../_images/message-queue/Kafka/kafka-apis.png) - -Kafka 的客户端和服务器之间的通信是靠一个简单的，高性能的，与语言无关的 TCP 协议完成的。这个协议有不同的版本，并保持向后兼容旧版本。Kafka 不光提供了一个 Java 客户端，还有许多语言版本的客户端。 - -#### 主题和日志 - -主题是同一类别的消息记录（record）的集合。Kafka 的主题支持多用户订阅，也就是说，一个主题可以有零个，一个或多个消费者订阅写入的数据。对于每个主题，Kafka 集群都会维护一个分区日志，如下所示： - -![图片来源：官方文档](../../_images/message-queue/Kafka/log_anatomy.png) - -**每个分区是一个有序的，不可变的消息序列**，新的消息不断追加到 partition 的末尾。在每个 partition 中，每条消息都会被分配一个顺序的唯一标识，这个标识被称为 **offset**，即偏移量。**kafka 不能保证全局有序，只能保证分区内有序** 。 - -Kafka 集群保留所有发布的记录，不管这个记录有没有被消费过，**Kafka 提供可配置的保留策略去删除旧数据**(还有一种策略根据分区大小删除数据)。例如，如果将保留策略设置为两天，在数据发布后两天，它可用于消费，之后它将被丢弃以腾出空间。Kafka 的性能跟存储的数据量的大小无关（会持久化到硬盘）， 所以将数据存储很长一段时间是没有问题的。 - -![图片来源：官方文档](../../_images/message-queue/Kafka/log_consumer.png) - -事实上，在单个消费者层面上，每个消费者保存的唯一的元数据就是它所消费的数据日志文件的偏移量。偏移量是由消费者来控制的，通常情况下，消费者会在读取记录时线性的提高其偏移量。不过由于偏移量是由消费者控制，所以消费者可以将偏移量设置到任何位置，比如设置到以前的位置对数据进行重复消费，或者设置到最新位置来跳过一些数据。 - -#### 分布式 - -日志的分区会跨服务器的分布在 Kafka 集群中，每个服务器会共享分区进行数据请求的处理。**每个分区可以配置一定数量的副本分区提供容错能力。** - -**每个分区都有一个服务器充当“leader”和零个或多个服务器充当“followers”**。 leader 处理所有的读取和写入分区的请求，而 followers 被动的从领导者拷贝数据。如果 leader 失败了，followers 之一将自动成为新的领导者。每个服务器可能充当一些分区的 leader 和其他分区的 follower，所以 Kafka 集群内的负载会比较均衡。 - -#### 生产者 - -生产者发布数据到他们所选择的主题。生产者负责选择把记录分配到主题中的哪个分区。这可以使用轮询算法( round-robin)进行简单地平衡负载，也可以根据一些更复杂的语义分区算法（比如基于记录一些键值）来完成。 - -#### 消费者 - -消费者以消费群（**consumer group** ）的名称来标识自己，每个发布到主题的消息都会发送给订阅了这个主题的消费群里面的一个消费者的一个实例。消费者的实例可以在单独的进程或单独的机器上。 - -如果所有的消费者实例都属于相同的消费群，那么记录将有效地被均衡到每个消费者实例。 - -如果所有的消费者实例有不同的消费群，那么每个消息将被广播到所有的消费者进程。 - -这是 kafka 用来实现一个 topic 消息的广播（发给所有的consumer） 和单播（发给任意一个 consumer）的手段。一个 topic 可以有多个 CG。 topic 的消息会复制 （不是真的复制，是概念上的）到所有的 CG，但每个 partion 只会把消息发给该 CG 中的一 个 consumer。如果需要实现广播，只要每个 consumer 有一个独立的 CG 就可以了。要实现单播只要所有的 consumer 在同一个 CG。用 CG 还可以将 consumer 进行自由的分组而不需 要多次发送消息到不同的 topic； - -![img](../../_images/message-queue/Kafka/sumer-groups.png) - -举个栗子： - -如上图所示，一个两个节点的 Kafka 集群上拥有一个四个 partition（P0-P3）的 topic。有两个消费者组都在消费这个 topic 中的数据，消费者组 A 有两个消费者实例，消费者组 B 有四个消费者实例。 - -从图中我们可以看到，在同一个消费者组中，每个消费者实例可以消费多个分区，但是每个分区最多只能被消费者组中的一个实例消费。也就是说，如果有一个4个分区的主题，那么消费者组中最多只能有4个消费者实例去消费，多出来的都不会被分配到分区。其实这也很好理解，如果允许两个消费者实例同时消费同一个分区，那么就无法记录这个分区被这个消费者组消费的 offset 了。如果在消费者组中动态的上线或下线消费者，那么 Kafka 集群会自动调整分区与消费者实例间的对应关系。 - -**Kafka消费群的实现方式是通过分割日志的分区，分给每个 Consumer 实例，使每个实例在任何时间点的都可以“公平分享”独占的分区**。维持消费群中的成员关系的这个过程是通过Kafka动态协议处理。如果新的实例加入该组，他将接管该组的其他成员的一些分区; 如果一个实例死亡，其分区将被分配到剩余的实例。 - -Kafka 只保证一个分区内的消息有序，不能保证一个主题的不同分区之间的消息有序。分区的消息有序与依靠主键进行数据分区的能力相结合足以满足大多数应用的要求。但是，如果你想要保证所有的消息都绝对有序可以只为一个主题分配一个分区，虽然这将意味着每个消费群同时只能有一个消费进程在消费。 - -#### 保证 - -Kafka 提供了以下一些高级别的保证： - -- 由生产者发送到一个特定的主题分区的消息将被以他们被发送的顺序来追加。也就是说，如果一个消息 M1 和消息 M2 都来自同一个生产者，M1 先发，那么 M1 将有一个低于 M2 的偏移，会更早在日志中出现。 -- 消费者看到的记录排序就是记录被存储在日志中的顺序。 -- 对于副本因子 N 的主题，我们将承受最多 N-1 次服务器故障切换而不会损失任何的已经保存的记录。 - - - -### 3.2 Kafka的使用场景 - -#### 消息 - -Kafka 被当作传统消息中间件的替代品。消息中间件的使用原因有多种（从数据生产者解耦处理，缓存未处理的消息等）。与大多数消息系统相比，Kafka 具有更好的吞吐量，内置的分区，多副本和容错功能，这使其成为大规模消息处理应用程序的良好解决方案。 - -#### 网站行为跟踪 - -Kafka 的初衷就是能够将用户行为跟踪管道重构为一组实时发布-订阅数据源。这意味着网站活动（页面浏览量，搜索或其他用户行为）将被发布到中心主题，这些中心主题是每个用户行为类型对应一个主题的。这些数据源可被订阅者获取并用于一系列的场景，包括实时处理，实时监控和加载到 Hadoop 或离线数据仓库系统中进行离线处理和报告。用户行为跟踪通常会产生巨大的数据量，因为用户每个页面的浏览都会生成许多行为活动消息。 - -#### 测量 - -Kafka 通常用于监测数据的处理。这涉及从分布式应用程序聚集统计数据，生产出集中的运行数据源 feeds（以便订阅）。 - -#### 日志聚合 - -许多人用 Kafka 作为日志聚合解决方案的替代品。日志聚合通常从服务器收集物理日志文件，并将它们集中放置（可能是文件服务器或HDFS），以便后续处理。kafka 抽象出文件的细节，并将日志或事件数据作为消息流清晰地抽象出来。这为低时延的处理提供支持，而且更容易支持多个数据源和分布式的数据消费。相比集中式的日志处理系统（如 Scribe 或 Flume），Kafka 性能同样出色，而且因为副本备份提供了更强的可靠性保证和更低的端到端延迟。 - -#### 流处理 - -Kafka 的流数据管道在处理数据的时候包含多个阶段，其中原始输入数据从 Kafka 主题被消费然后汇总，加工，或转化成新主题用于进一步的消费或后续处理。例如，用于推荐新闻文章的数据流处理管道可能从 RSS 源抓取文章内容，并将其发布到“文章”主题; 进一步的处理可能是标准化或删除重复数据，然后发布处理过的文章内容到一个新的主题， 最后的处理阶段可能会尝试推荐这个内容给用户。这种处理管道根据各个主题创建实时数据流图。从版本 0.10.0.0 开始，Apache Kafka 加入了轻量级的但功能强大的流处理库 **Kafka Streams**，Kafka Streams 支持如上所述的数据处理。除了Kafka Streams，可以选择的开源流处理工具包括 `Apache Storm and Apache Samza`。 - -#### 事件源 - -事件源是一种应用程序设计风格，是按照时间顺序记录的状态变化的序列。Kafka 的非常强大的存储日志数据的能力使它成为构建这种应用程序的极好的后端选择。 - -#### 提交日志 - -Kafka 可以为分布式系统提供一种外部提交日志(commit-log)服务。日志有助于节点之间复制数据，并作为一种数据重新同步机制用来恢复故障节点的数据。Kafka 的 log compaction 功能有助于支持这种用法。Kafka 在这种用法中类似于Apache BookKeeper 项目。 - ------- - - - -## 3. Kafka架构深入 - -### 3.1 Kafka 工作流程和文件存储机制 - -![img](../../_images/message-queue/Kafka/kafka-workflow.jpg) - -#### topic构成 - -**Kafka 中消息是以 topic 进行分类的**，生产者生产消息，消费者消费消息，都是面向 topic 的。 - -**topic 是逻辑上的概念，而 patition 是物理上的概念**，每个 patition 对应一个 log 文件，而 log 文件中存储的就是 producer 生产的数据，patition 生产的数据会被不断的添加到 log 文件的末端，且每条数据都有自己的 offset。消费组中的每个消费者，都是实时记录自己消费到哪个 offset，以便出错恢复，从上次的位置继续消费。 - -![img](../../_images/message-queue/Kafka/kafka-partition.jpg) - - - -#### 消息存储原理 - -由于生产者生产的消息会不断追加到 log 文件末尾，为防止 log 文件过大导致数据定位效率低下，Kafka 采取了**分片**和**索引**机制，将每个 partition 分为多个 segment。每个 segment 对应两个文件——`.index文件`和 `.log文件`。这些文件位于一个文件夹下，该文件夹的命名规则为：topic名称+分区序号。例如，first 这个 topic 有三个分区，则其对应的文件夹为 first-0，first-1，first-2。 - -![QQ20200330-183839@2x](https://mrbird.cc/img/QQ20200330-183839@2x.png) - -这些文件的含义如下： - -| 类别 | 作用 | -| :---------------------- | :----------------------------------------------------------- | -| .index | 偏移量索引文件，存储数据对应的偏移量 | -| .timestamp | 时间戳索引文件 | -| .log | 日志文件，存储生产者生产的数据 | -| .snaphot | 快照文件 | -| Leader-epoch-checkpoint | 保存了每一任leader开始写入消息时的offset，会定时更新。 follower被选为leader时会根据这个确定哪些消息可用 | - -index 和 log 文件以当前 segment 的第一条消息的 offset 命名。偏移量 offset 是一个64位的长整形数，固定是20位数字，长度未达到，用0进行填补，索引文件和日志文件都由此作为文件名命名规则。所以从上图可以看出，我们的偏移量是从0开始的，`.index`和`.log`文件名称都为 `00000000000000000000`。下图为 index 文件和 log 文件的结构示意图。 - -![img](../../_images/message-queue/Kafka/kafka-segement.jpg) - -`.index文件` 存储大量的索引信息，`.log文件` 存储大量的数据，索引文件中的元数据指向对应数据文件中 message 的物理偏移地址。 - - - -上节中，我们通过生产者发送了hello和world两个数据，所以我们可以查看下.log文件下是否有这两条数据： - -![QQ20200331-151427@2x](https://mrbird.cc/img/QQ20200331-151427@2x.png) - -内容存在一些”乱码“，因为数据是经过序列化压缩的。 - -那么数据文件.log大小有限制吗，能保存多久时间？这些我们都可以通过Kafka目录下conf/server.properties配置文件修改： - -``` -# log文件存储时间，单位为小时，这里设置为1周 -log.retention.hours=168 - -# log文件大小的最大值，这里为1g，超过这个值，则会创建新的segment（也就是新的.index和.log文件） -log.segment.bytes=1073741824 -``` - - - -比如，当生产者生产数据量较多，一个 segment 存储不下触发分片时，在日志 topic 目录下你会看到类似如下所示的文件： - -``` -00000000000000000000.index -00000000000000000000.log -00000000000000170410.index -00000000000000170410.log -00000000000000239430.index -00000000000000239430.log -``` - - - -下图展示了 Kafka 查找数据的过程： - -![QQ20200331-155820@2x](https://mrbird.cc/img/QQ20200331-155820@2x.png) - -比如现在要查找偏移量offset为3的消息，根据.index文件命名我们可以知道，offset为3的索引应该从00000000000000000000.index 里查找。根据上图所示，其对应的索引地址为 756~911，所以 Kafka 将读取00000000000000000000.log 756~911区间的数据。 - -### 3.2 Kafka 生产过程 - -#### 3.2.1 写入流程 - -producer 写入消息流程如下： - -![img](../../_images/message-queue/Kafka/kafka-write-flow.png) - - - -1. producer 先从 zookeeper 的 "/brokers/.../state"节点找到该 partition 的 leader -2. producer 将消息发送给该 leader -3. leader 将消息写入本地 log -4. followers 从 leader pull 消息，写入本地 log 后向 leader 发送 ACK -5. leader 收到所有 ISR 中的 replication 的 ACK 后，增加 HW（high watermark，最后 commit 的 offset）并向 producer 发送 ACK - -#### 3.2.2 写入方式 - -producer 采用推（push） 模式将消息发布到 broker，每条消息都被追加（append） 到分区（patition） 中，属于顺序写磁盘（顺序写磁盘效率比随机写内存要高，保障 kafka 吞吐率）。 - -#### 3.2.3 分区（Partition） - -消息发送时都被发送到一个 topic，其本质就是一个目录，而 topic 是由一些 Partition Logs(分区日志)组成 - -**分区的原因：** - -1. **方便在集群中扩展**，每个 Partition 可以通过调整以适应它所在的机器，而一个 topic 又可以有多个 Partition 组成，因此整个集群就可以适应任意大小的数据了； - -2. **可以提高并发**，因为可以以 Partition 为单位读写了。 - -**分区的原则：** - -我们需要将 producer 发送的数据封装成一个 ProducerRecord 对象。 - -```java -public ProducerRecord (String topic, Integer partition, Long timestamp, K key, V value, Iterable
headers) -public ProducerRecord (String topic, Integer partition, Long timestamp, K key, V value) -public ProducerRecord (String topic, Integer partition, K key, V value, Iterable
headers) -public ProducerRecord (String topic, Integer partition, K key, V value) -public ProducerRecord (String topic, K key, V value) -public ProducerRecord (String topic, V value) -``` - -1. 指明 partition 的情况下，直接将指明的值直接作为 partiton 值； -2. 没有指明 partition 值但有 key 的情况下，将 key 的 hash 值与 topic 的 partition 数进行取余得到 partition 值； -3. 既没有 partition 值又没有 key 值的情况下，第一次调用时随机生成一个整数（后面每次调用在这个整数上自增），将这个值与 topic 可用的 partition 总数取余得到 partition 值，也就是常说的 round-robin 算法。 - - -#### 3.2.4 副本（Replication） - -同一个 partition 可能会有多个 replication（ 对应 server.properties 配置中的 default.replication.factor=N）。没有 replication 的情况下，一旦 broker 宕机，其上所有 patition 的数据都不可被消费，同时 producer 也不能再将数据存于其上的 patition。引入 replication 之后，同一个 partition 可能会有多个 replication，而这时需要在这些 replication 之间选出一 个 leader， producer 和 consumer 只与这个 leader 交互，其它 replication 作为 follower 从 leader 中复制数据。 - -#### 3.2.5 数据可靠性保证 - -为保证 producer 发送的数据，能可靠的发送到指定的 topic，topic 的每个 partition 收到 producer 数据后，都需要向 producer 发送 ack（acknowledgement确认收到），如果 producer 收到 ack，就会进行下一轮的发送，否则重新发送数据。 - -![img](../../_images/message-queue/Kafka/kafka-ack-slg.png) - -##### a) 副本数据同步策略 - -| 方案 | 优点 | 缺点 | -| --------------------------- | ------------------------------------------------------ | ----------------------------------------------------- | -| 半数以上完成同步，就发送ack | 延迟低 | 选举新的 leader 时，容忍n台节点的故障，需要2n+1个副本 | -| 全部完成同步，才发送ack | 选举新的 leader 时，容忍n台节点的故障，需要 n+1 个副本 | 延迟高 | - -Kafka 选择了第二种方案，原因如下： - -- 同样为了容忍 n 台节点的故障，第一种方案需要的副本数相对较多，而 Kafka 的每个分区都有大量的数据，第一种方案会造成大量的数据冗余； -- 虽然第二种方案的网络延迟会比较高，但网络延迟对 Kafka 的影响较小。 - -##### b) ISR - -采用第二种方案之后，设想一下情景：leader 收到数据，所有 follower 都开始同步数据，但有一个 follower 挂了，迟迟不能与 leader 保持同步，那 leader 就要一直等下去，直到它完成同步，才能发送 ack，这个问题怎么解决呢？ - -leader 维护了一个动态的 **in-sync replica set**(ISR)，意为和 leader 保持同步的 follower 集合。当 ISR 中的follower 完成数据的同步之后，leader 就会给 follower 发送 ack。如果 follower 长时间未向 leader 同步数据，则该 follower 将会被踢出 ISR，该时间阈值由 `replica.lag.time.max.ms` 参数设定。leader 发生故障之后，就会从 ISR 中选举新的 leader。（之前还有另一个参数，0.9 版本之后 `replica.lag.max.messages` 参数被移除了） - -##### c) ack应答机制 - -对于某些不太重要的数据，对数据的可靠性要求不是很高，能够容忍数据的少量丢失，所以没必要等 ISR 中的follower全部接收成功。 - -所以Kafka为用户提供了**三种可靠性级别**，用户根据对可靠性和延迟的要求进行权衡，选择以下的配置。 - -**acks参数配置：** - -- **acks:** - - 0：producer 不等待 broker 的 ack，这一操作提供了一个最低的延迟，broker 一接收到还没有写入磁盘就已经返回，当 broker 故障时有可能**丢失数据**； - - 1：producer 等待 broker 的 ack，partition 的 leader 落盘成功后返回 ack，如果在 follower 同步成功之前 leader 故障，那么将会**丢失数据**（下图为acks=1数据丢失案例）； - -![img](../../_images/message-queue/Kafka/kafka-ack=1.png) - --1（all）：producer 等待 broker 的 ack，partition 的 leader 和 follower 全部落盘成功后才返回 ack。但是 如果在 follower 同步完成后，broker 发送 ack 之前，leader 发生故障，那么就会造成**数据重复**。（下图为acks=-1数据重复案例） - -![img](../../_images/message-queue/Kafka/kafka-ack=-1.png) - -##### d) 故障处理 - -由于我们并不能保证 Kafka 集群中每时每刻 follower 的长度都和 leader 一致（即数据同步是有时延的），那么当leader 挂掉选举某个 follower 为新的 leader 的时候（原先挂掉的 leader 恢复了成为了 follower），可能会出现leader 的数据比 follower 还少的情况。为了解决这种数据量不一致带来的混乱情况，Kafka 提出了以下概念： - -![QQ20200401-093957@2x](https://mrbird.cc/img/QQ20200401-093957@2x.png) - -- LEO（Log End Offset）：指的是每个副本最后一个offset； -- HW（High Wather）：指的是消费者能见到的最大的 offset，ISR 队列中最小的 LEO。 - -消费者和leader通信时，只能消费 HW 之前的数据，HW 之后的数据对消费者不可见。 - -针对这个规则： - -- **当follower发生故障时**：follower 发生故障后会被临时踢出 ISR，待该 follower 恢复后，follower 会读取本地磁盘记录的上次的 HW，并将 log 文件高于 HW 的部分截取掉，从 HW 开始向 leader 进行同步。等该 follower 的 LEO 大于等于该 Partition 的 HW，即 follower 追上 leader 之后，就可以重新加入 ISR 了。 -- **当leader发生故障时**：leader 发生故障之后，会从 ISR 中选出一个新的 leader，之后，为保证多个副本之间的数据一致性，其余的 follower 会先将各自的 log 文件高于 HW 的部分截掉，然后从新的 leader 同步数据。 - -所以数据一致性并不能保证数据不丢失或者不重复，这是由 ack 控制的。HW 规则只能保证副本之间的数据一致性！ - -#### 3.2.6 Exactly Once语义 - -将服务器的 ACK 级别设置为 -1，可以保证 Producer 到 Server 之间不会丢失数据，即 At Least Once 语义。相对的，将服务器 ACK 级别设置为 0，可以保证生产者每条消息只会被发送一次，即 At Most Once语义。 - -**At Least Once 可以保证数据不丢失，但是不能保证数据不重复。相对的，At Most Once 可以保证数据不重复，但是不能保证数据不丢失**。但是，对于一些非常重要的信息，比如说交易数据，下游数据消费者要求数据既不重复也不丢失，即 Exactly Once 语义。在 0.11 版本以前的 Kafka，对此是无能为力的，只能保证数据不丢失，再在下游消费者对数据做全局去重。对于多个下游应用的情况，每个都需要单独做全局去重，这就对性能造成了很大的影响。 - -0.11 版本的 Kafka，引入了一项重大特性：**幂等性**。所谓的幂等性就是指 Producer 不论向 Server 发送多少次重复数据。Server 端都会只持久化一条，幂等性结合 At Least Once 语义，就构成了 Kafka 的 Exactily Once 语义，即： **At Least Once + 幂等性 = Exactly Once** - -要启用幂等性，只需要将 Producer 的参数中 `enable.idompotence` 设置为 `true` 即可。Kafka 的幂等性实现其实就是将原来下游需要做的去重放在了数据上游。开启幂等性的 Producer 在初始化的时候会被分配一个 PID，发往同一 Partition 的消息会附带 Sequence Number。而 Broker 端会对 做缓存，当具有相同主键的消息提交时，Broker 只会持久化一条。 - -但是 PID 重启就会变化，同时不同的 Partition 也具有不同主键，所以幂等性无法保证跨分区会话的 Exactly Once。 - - - - -### 3.3 Broker 保存消息 - -#### 3.3.1 存储方式 - -物理上把 topic 分成一个或多个 patition（对应 server.properties 中的 num.partitions=3 配 置），每个 patition 物理上对应一个文件夹（该文件夹存储该 patition 的所有消息和索引文件）。 - -#### 3.3.2 存储策略 - -无论消息是否被消费， kafka 都会保留所有消息。有两种策略可以删除旧数据： - -1. 基于时间： log.retention.hours=168 - -2. 基于大小： log.retention.bytes=1073741824 需要注意的是，因为 Kafka 读取特定消息的时间复杂度为 O(1)，即与文件大小无关， 所以这里删除过期文件与提高 Kafka 性能无关。 - - - -### 3.4 Kafka 消费过程 - -#### 3.4.1 消费者组 - -![img](../../_images/message-queue/Kafka/kafka-consume-group.png) - -消费者是以 consumer group 消费者组的方式工作，由一个或者多个消费者组成一个组， 共同消费一个 topic。每个分区在同一时间只能由 group 中的一个消费者读取，但是多个 group 可以同时消费这个 partition。在图中，有一个由三个消费者组成的 group，有一个消费者读取主题中的两个分区，另外两个分别读取一个分区。某个消费者读取某个分区，也可以叫做某个消费者是某个分区的拥有者。 - -在这种情况下，消费者可以通过水平扩展的方式同时读取大量的消息。另外，如果一个消费者失败了，那么其他的 group 成员会自动负载均衡读取之前失败的消费者读取的分区。 - -消费者组最为重要的一个功能是实现广播与单播的功能。一个消费者组可以确保其所订阅的Topic的每个分区只能被从属于该消费者组中的唯一一个消费者所消费；如果不同的消费者组订阅了同一个Topic，那么这些消费者组之间是彼此独立的，不会受到相互的干扰。 - -> 如果我们希望一条消息可以被多个消费者所消费，那么可以将这些消费者放到不同的消费者组中，这实际上就是广播的效果；如果希望一条消息只能被一个消费者所消费，那么可以将这些消费者放到同一个消费者组中，这实际上就是单播的效果。 - -#### 3.4.2 消费方式 - -**consumer 采用 pull（拉） 模式从 broker 中读取数据。** - -push（推）模式很难适应消费速率不同的消费者，因为消息发送速率是由 broker 决定的。 它的目标是尽可能以最快速度传递消息，但是这样很容易造成 consumer 来不及处理消息， 典型的表现就是拒绝服务以及网络拥塞。而 pull 模式则可以根据 consumer 的消费能力以适当的速率消费消息。 - -对于 Kafka 而言， pull 模式更合适，它可简化 broker 的设计， consumer 可自主控制消费消息的速率，同时 consumer 可以自己控制消费方式——即可批量消费也可逐条消费，同时 还能选择不同的提交方式从而实现不同的传输语义。 - -pull 模式不足之处是，如果 kafka 没有数据，消费者可能会陷入循环中，一直等待数据到达，一直返回空数据。为了避免这种情况，我们在我们的拉请求中有参数，允许消费者请求在等待数据到达的“长轮询”中进行阻塞（并且可选地等待到给定的字节数，以确保大的传输大小）。 - -#### 3.4.3 分区分配策略 - -一个 consumer group 中有多个 consumer，一个 topic 有多个 partition，所以必然会涉及到 partition 的分配问题，即确定哪个 partition 由哪个 consumer 来消费。 - -Kafka有两种分配策略，一是 RoundRobin，一是 Range。 - -##### RoundRobin - -RoundRobin即轮询的意思，比如现在有一个三个消费者ConsumerA、ConsumerB和ConsumerC组成的消费者组，同时消费TopicA主题消息，TopicA分为7个分区，如果采用RoundRobin分配策略，过程如下所示： - -![QQ20200401-145222@2x](https://mrbird.cc/img/QQ20200401-145222@2x.png) - -这种轮询的方式应该很好理解。但如果消费者组消费多个主题的多个分区，会发生什么情况呢？比如现在有一个两个消费者ConsumerA和ConsumerB组成的消费者组，同时消费TopicA和TopicB主题消息，如果采用RoundRobin分配策略，过程如下所示： - -![QQ20200401-150317@2x](https://mrbird.cc/img/QQ20200401-150317@2x.png) - -> 注：TAP0表示TopicA Partition0分区数据，以此类推。 - -这种情况下，采用RoundRobin算法分配，多个主题会被当做一个整体来看，这个整体包含了各自的Partition，比如在 Kafka-clients 依赖中，与之对应的对象为`TopicPartition`。接着将这些`TopicPartition`根据其哈希值进行排序，排序后采用轮询的方式分配给消费者。 - -但这会带来一个问题：假如上图中的消费者组中，ConsumerA只订阅了TopicA主题，ConsumerB只订阅了TopicB主题，采用RoundRobin轮询算法后，可能会出现ConsumerA消费了TopicB主题分区里的消息，ConsumerB消费了TopicA主题分区里的消息。 - -综上所述，RoundRobin算法只适用于消费者组中消费者订阅的主题相同的情况。同时会发现，采用RoundRobin算法，消费者组里的消费者之间消费的消息个数最多相差1个。 - -##### Range - -Kafka 默认采用 Range 分配策略，Range 顾名思义就是按范围划分的意思。 - -比如现在有一个三个消费者 ConsumerA、ConsumerB和ConsumerC组成的消费者组，同时消费TopicA主题消息，TopicA分为7个分区，如果采用 Range 分配策略，过程如下所示： - -![QQ20200401-152904@2x](https://mrbird.cc/img/QQ20200401-152904@2x.png) - -假如现在有一个两个消费者ConsumerA和ConsumerB组成的消费者组，同时消费TopicA和TopicB主题消息，如果采用Range分配策略，过程如下所示： - -![QQ20200401-153300@2x](https://mrbird.cc/img/QQ20200401-153300@2x.png) - -Range算法并不会把多个主题分区当成一个整体。 - -从上面的例子我们可以总结出Range算法的一个弊端：那就是同一个消费者组内的消费者消费的消息数量相差可能较大。 - -#### 3.4.4 offset的维护 - -由于consumer在消费过程中可能会出现断电宕机等故障，consumer 恢复后，需要从故障前的位置继续消费，所以 consumer 需要实时记录自己消费到了哪个 offset，以便故障恢复后继续消费。 - -Kafka 0.9 版本之前，consumer 默认将 offset 保存在 Zookeeper 中，从 0.9 版本开始，consumer 默认将offset保存在 Kafka 一个内置的 topic 中，该 topic 为 **_consumer_offsets**。 - -- 修改配置文件 `consumer.properties` - -```shell -exclude.internal.topics=false -``` - -- **查询__consumer_offsets topic所有内容** - -**注意：运行下面命令前先要在consumer.properties中设置exclude.internal.topics=false** - -0.11.0.0之前版本 - -```shell -bin/kafka-console-consumer.sh --topic __consumer_offsets --zookeeper localhost:2181 --formatter "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter" --consumer.config config/consumer.properties --from-beginning -``` - -**0.11.0.0之后版本(含)** - -```shell -bin/kafka-console-consumer.sh --topic __consumer_offsets --zookeeper localhost:2181 --formatter "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter" --consumer.config config/consumer.properties --from-beginning -``` - -默认情况下__consumer_offsets 有 50 个分区，如果你的系统中 consumer group 也很多的话，那么这个命令的输出结果会很多 - - - -### 3.5 Kafka高效读写数据的原因 - -#### 3.5.1 顺序写磁盘 - -Kafka 的 producer 生产数据，要写入到 log 文件中，写的过程是一直追加到文件末端，为顺序写。官网有数据表明，同样的磁盘，顺序写能到到 600M/s，而随机写只有 100k/s。这与磁盘的机械机构有关，顺序写之所以快，是因为其省去了大量磁头寻址的时间 。 - -#### 3.5.2 零拷贝技术（todo...） - -![img](../../_images/message-queue/Kafka/zero-copy.png) - - - ------- - - - -### 3.6 Zookeeper在Kafka中的作用 - -- **存储结构** - -![img](../../_images/message-queue/Kafka/zookeeper-store.png) - - - -注意： **producer 不在 zk 中注册， 消费者在 zk 中注册。** - -Kafka集群中有一个broker会被选举为Controller，**负责管理集群broker的上线下，所有topic的分区副本分配和leader选举等工作**。 - -Controller的管理工作都是依赖于Zookeeper的。 - -下图为 partition 的 leader 选举过程： - -![img](../../_images/message-queue/Kafka/controller-leader.png) - - - -### 3.7 Kafka事务 - -Kafka 从 0.11 版本开始引入了事务支持。事务可以保证 Kafka 在 Exactly Once 语义的基础上，生产和消费可以跨分区和会话，要么全部成功，要么全部失败。 - -#### 3.7.1 Producer事务 - -为了了实现跨分区跨会话的事务，需要引入一个全局唯一的 TransactionID，并将 Producer 获得的 PID 和Transaction ID 绑定。这样当 Producer 重启后就可以通过正在进行的 TransactionID 获得原来的 PID。 - -为了管理 Transaction，Kafka 引入了一个新的组件 Transaction Coordinator。Producer 就是通过和 Transaction Coordinator 交互获得 Transaction ID 对应的任务状态。Transaction Coordinator 还负责将事务所有写入 Kafka 的一个内部 Topic，这样即使整个服务重启，由于事务状态得到保存，进行中的事务状态可以得到恢复，从而继续进行。 - -#### 3.7.2 Consumer事务 - -对 Consumer 而言，事务的保证就会相对较弱，尤其是无法保证 Commit 的消息被准确消费。这是由于Consumer 可以通过 offset 访问任意信息，而且不同的 SegmentFile 生命周期不同，同一事务的消息可能会出现重启后被删除的情况。 - ------- - - - diff --git a/docs/message-queue/Kafka/Kafka-install.md b/docs/message-queue/Kafka/Kafka-install.md deleted file mode 100644 index d5f716aa95..0000000000 --- a/docs/message-queue/Kafka/Kafka-install.md +++ /dev/null @@ -1,9 +0,0 @@ - - -Kafka监控： Kafka Engle - - - -系列文章 - - \ No newline at end of file diff --git "a/docs/message-queue/Kafka/Kafka\350\243\205\346\207\202\346\214\207\345\215\227.md" "b/docs/message-queue/Kafka/Kafka\350\243\205\346\207\202\346\214\207\345\215\227.md" deleted file mode 100644 index 925bb05d2e..0000000000 --- "a/docs/message-queue/Kafka/Kafka\350\243\205\346\207\202\346\214\207\345\215\227.md" +++ /dev/null @@ -1,49 +0,0 @@ -![1570607503512](C:\Users\JIAHAI~1\AppData\Local\Temp\1570607503512.png) - -![1570613549970](C:\Users\JIAHAI~1\AppData\Local\Temp\1570613549970.png) - - topic分区可增 不可减 - - - -![1570613613517](C:\Users\JIAHAI~1\AppData\Local\Temp\1570613613517.png) - - - - - -kafka架构 - -![1570613957695](C:\Users\JIAHAI~1\AppData\Local\Temp\1570613957695.png) - - - -![1570613975018](C:\Users\JIAHAI~1\AppData\Local\Temp\1570613975018.png) - -![1570614118956](C:\Users\JIAHAI~1\AppData\Local\Temp\1570614118956.png) - - - -![1570614141530](C:\Users\JIAHAI~1\AppData\Local\Temp\1570614141530.png) - - - - - -https://www.iteblog.com/archives/2605.html - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/docs/message-queue/Kafka/readKafka.md b/docs/message-queue/Kafka/readKafka.md deleted file mode 100644 index ffcbc99549..0000000000 --- a/docs/message-queue/Kafka/readKafka.md +++ /dev/null @@ -1 +0,0 @@ -Kafka \ No newline at end of file diff --git a/docs/message-queue/Kafka/sidebar.md b/docs/message-queue/Kafka/sidebar.md deleted file mode 100644 index 03908f5b22..0000000000 --- a/docs/message-queue/Kafka/sidebar.md +++ /dev/null @@ -1,54 +0,0 @@ -- **Java基础** -- [![](https://icongr.am/simple/oracle.svg?size=25&color=231c82&colored=false)JVM](java/JVM/readJVM.md) -- [![img](https://icongr.am/fontawesome/expeditedssl.svg?size=25&color=f23131)JUC](java/JUC/readJUC.md) -- [![](https://icongr.am/devicon/java-original.svg?size=25&color=f23131)Java 8](java/Java8.md) -- [![img](https://icongr.am/entypo/address.svg?size=25&color=074ca6)设计模式](design-pattern/readDisignPattern.md) -- **数据存储和缓存** -- [![MySQL](https://icongr.am/devicon/mysql-original.svg?&size=25)MySQL](data-store/MySQL/readMySQL.md) -- [![Redis](https://icongr.am/devicon/redis-original.svg?size=25)Redis](data-store/Redis/2.readRedis.md) -- [![mongoDB](https://icongr.am/devicon/mongodb-original.svg?&size=25)mongoDB]( https://redis.io/ ) -- [![ **Elasticsearch** ](https://icongr.am/simple/elasticsearch.svg?&size=20) Elasticsearch]( https://redis.io/ ) -- [![S3](https://icongr.am/devicon/amazonwebservices-original.svg?&size=25)S3]( https://aws.amazon.com/cn/s3/ ) -- **直击面试** -- [![](https://icongr.am/material/basket.svg?size=25)Java集合面试](interview/Collections-FAQ.md) -- [![](https://icongr.am/devicon/java-plain-wordmark.svg?size=25)JVM面试](interview/JVM-FAQ.md) -- [![](https://icongr.am/devicon/mysql-original-wordmark.svg?size=25)MySQL面试](interview/MySQL-FAQ.md) -- [![](https://icongr.am/devicon/redis-original-wordmark.svg?size=25)Redis面试](interview/Redis-FAQ.md) -- [![](https://icongr.am/jam/leaf.svg?size=25&color=00FF00)Spring面试](interview/Spring-FAQ.md) -- [![](https://icongr.am/simple/bower.svg?size=25)MyBatis面试](interview/MyBatis-FAQ.md) -- [![img](https://icongr.am/entypo/network.svg?size=25&color=6495ED)计算机网络](interview/Network-FAQ.md) -- **单体架构** -- **RPC** -- [Hello Protocol Buffers](rpc/Hello-Protocol-Buffers.md) -- **面向服务架构** -- [![message](https://icongr.am/clarity/email.svg?&size=16) 消息中间件](message-queue/readMQ.md) -- [![Nginx](https://icongr.am/devicon/nginx-original.svg?&size=16)Nginx](nginx/nginx.md) -- **微服务架构** -- [🍃 Spring Boot](springboot/Hello-SpringBoot.md) -- [🍃 定时任务@Scheduled](springboot/Spingboot定时任务@Scheduled.md) -- [🍃 Spring Cloud](https://spring.io/projects/spring-cloud) -- **大数据** -- [Hello 大数据](big-data/Hello-BigData.md) -- [![](https://icongr.am/simple/apachekafka.svg?size=25&color=121417&colored=false)Kafka](message-queue/Kafka/readKafka.md) - - [Kafka 入门](message-queue/Kafka/Hello-Kafka.md) - - \> Kakfa 存储机制 - - \> Kafka 生产者 - - \> Kafka 消费者 - - \> Kafka API -- **性能优化** -- JVM优化 -- web调优 -- DB调优 -- **工程化与工具** -- [![Maven](https://icongr.am/devicon//fontawesome/maxcdn.svg?&size=16)Maven](logging/logback简单使用.md) -- [![Git](https://icongr.am/devicon/git-original.svg?&size=16)Git](logging/logback简单使用.md) -- [Sonar](https://www.sonarqube.org/) -- **其他** -- [![Linux](https://icongr.am/devicon/linux-original.svg?&size=16)Linux](linux/linux.md) -- [缕清各种Java Logging](logging/Java-Logging.md) -- [hello logback](logging/logback简单使用.md) -- SHELL -- TCP与HTTP -- **Links** -- [![Github](https://icongram.jgog.in/simple/github.svg?color=808080&size=16)Github](https://github.com/jhildenbiddle/docsify-tabs) -- [![Blog](https://icongr.am/simple/aboutme.svg?colored&size=16)My Blog](https://www.lazyegg.net) \ No newline at end of file diff --git a/docs/message-queue/MQ-FAQ.md b/docs/message-queue/MQ-FAQ.md deleted file mode 100644 index 16deb8a2d7..0000000000 --- a/docs/message-queue/MQ-FAQ.md +++ /dev/null @@ -1,31 +0,0 @@ -### 为什么使用MQ？MQ的优点 - -- 异步处理 - 相比于传统的串行、并行方式，提高了系统吞吐量。 -- 应用解耦 - 系统间通过消息通信，不用关心其他系统的处理。 -- 流量削锋 - 可以通过消息队列长度控制请求量；可以缓解短时间内的高并发请求。 -- 日志处理 - 解决大量日志传输。 -- 消息通讯 - 消息队列一般都内置了高效的通信机制，因此也可以用在纯的消息通讯。比如实现点对点消息队列，或者聊天室等 - - - -### Kafka、ActiveMQ、RabbitMQ、RocketMQ 有什么优缺点？ - -| | RabbitMQ | ActiveMQ | RocketMQ | Kafka | -| ------------- | ----------------------------------------------- | ------------------------------------------- | ----------------------------------------------------- | --------------------------------------------------------- | -| 所属社区/公司 | Rabbit | Apache | Ali | Apache | -| 开发语言 | Erlang | Java | Java | Scala&Java | -| 多语言支持 | 语言无关 | 支持，Java优先 | Java | 支持，Java优先 | -| 消息推拉模式 | 多协议，Pull/Push均支持 | 多协议，Pull/Push均支持 | 多协议，Pull/Push均支持 | Pull | -| HA | master/slave模式，master提供服务，slave仅作备份 | 基于zookeeper+levelDB的master-slave实现方式 | 支持多master模式、多master多slave模式、异步复制模式、 | 支持replica机制。leader宕掉后，备份自动顶替，并重选leader | -| 事务 | 不支持 | 支持 | 支持 | 不支持，可通过Low Level API保证仅消费一次 | -| 集群 | 支持 | 支持 | 支持 | 支持 | -| 负载均衡 | 支持 | 支持 | 支持 | 支持 | - - - -### MQ 有哪些常见问题？如何解决这些问题？ - -MQ 的常见问题有： - -1. 消息的顺序问题 -2. 消息的重复问题 \ No newline at end of file diff --git a/docs/mind-maps/JUC.xmind b/docs/mind-maps/JUC.xmind new file mode 100644 index 0000000000..de8f6c21d9 Binary files /dev/null and b/docs/mind-maps/JUC.xmind differ diff --git a/docs/mind-maps/MySQL.xmind b/docs/mind-maps/MySQL.xmind new file mode 100644 index 0000000000..56d5ca06ff Binary files /dev/null and b/docs/mind-maps/MySQL.xmind differ diff --git a/docs/mind-maps/Tree.xmind b/docs/mind-maps/Tree.xmind new file mode 100644 index 0000000000..2f8adb3bbb Binary files /dev/null and b/docs/mind-maps/Tree.xmind differ diff --git a/docs/network/.DS_Store b/docs/network/.DS_Store new file mode 100644 index 0000000000..2e3f13200f Binary files /dev/null and b/docs/network/.DS_Store differ diff --git a/docs/network/README.md b/docs/network/README.md new file mode 100644 index 0000000000..6e5df8b4e6 --- /dev/null +++ b/docs/network/README.md @@ -0,0 +1,2 @@ +![](https://tva1.sinaimg.cn/large/0081Kckwly1gm72m1y8k4j31900u0hdt.jpg) + diff --git a/docs/network/RMI.md b/docs/network/RMI.md new file mode 100644 index 0000000000..ff9e100b49 --- /dev/null +++ b/docs/network/RMI.md @@ -0,0 +1,120 @@ +# RMI 远程调用 + +> 来源：[《廖雪峰RMI远程调用》](https://www.liaoxuefeng.com/wiki/1252599548343744/1323711850348577) + +## RMI 概述 + +Java 的 RMI 远程调用是指，一个 JVM 中的代码可以通过网络实现远程调用另一个 JVM 的某个方法。RMI 是 Remote Method Invocation的缩写。 + +提供服务的一方我们称之为服务器，而实现远程调用的一方我们称之为客户端。 + + + +## 服务提供者 + +我们先来实现一个最简单的RMI：服务器会提供一个 `WorldClock` 服务，允许客户端获取指定时区的时间，即允许客户端调用下面的方法： + +```java +LocalDateTime getLocalDateTime(String zoneId); +``` + +要实现RMI，服务器和客户端必须共享同一个接口。我们定义一个 `WorldClock` 接口，代码如下： + +```java +public interface WorldClock extends Remote { + + /** + * 获取指定区域时间 + * @param zoneId + * @return + * @throws RemoteException + */ + LocalDateTime getLocalDateTime(String zoneId) throws RemoteException; +} +``` + +Java 的 RMI 规定此接口必须派生自 `java.rmi.Remote`，并在每个方法声明抛出 `RemoteException`。 + +下一步是编写服务器的实现类，因为客户端请求的调用方法 `getLocalDateTime()` 最终会通过这个实现类返回结果。实现类 `WorldClockService` 代码如下： + +```java +public class WorldClockService implements WorldClock { + @Override + public LocalDateTime getLocalDateTime(String zoneId) throws RemoteException { + return LocalDateTime.now(ZoneId.of(zoneId)).withNano(0); + } +} +``` + +现在，服务器端的服务相关代码就编写完毕。我们需要通过 Java RMI 提供的一系列底层支持接口，把上面编写的服务以 RMI 的形式暴露在网络上，客户端才能调用： + +```java +public class Server { + public static void main(String[] args) throws RemoteException { + System.out.println("create World clock remote service..."); + // 实例化一个WorldClock: + WorldClock worldClock = new WorldClockService(); + // 将此服务转换为远程服务接口: + WorldClock skeleton = (WorldClock) UnicastRemoteObject.exportObject(worldClock, 0); + // 将RMI服务注册到1099端口: + Registry registry = LocateRegistry.createRegistry(1099); + // 注册此服务，服务名为"WorldClock": + registry.rebind("WorldClock", skeleton); + } +} +``` + +上述代码主要目的是通过 RMI 提供的相关类，将我们自己的 `WorldClock` 实例注册到 RMI 服务上。RMI 的默认端口是 `1099`，最后一步注册服务时通过 `rebind()` 指定服务名称为 `"WorldClock"`。 + +## 客户端调用 + +下一步我们就可以编写客户端代码。RMI 要求服务器和客户端共享同一个接口，因此我们要把 `WorldClock.java` 这个接口文件复制到客户端，然后在客户端实现RMI调用： + +```java +public class Client { + public static void main(String[] args) throws RemoteException, NotBoundException { + // 连接到服务器localhost，端口1099: + Registry registry = LocateRegistry.getRegistry("localhost", 1099); + // 查找名称为"WorldClock"的服务并强制转型为WorldClock接口: + WorldClock worldClock = (WorldClock) registry.lookup("WorldClock"); + // 正常调用接口方法: + LocalDateTime now = worldClock.getLocalDateTime("Asia/Shanghai"); + // 打印调用结果: + System.out.println(now); + } +} +``` + + + +先运行服务器，再运行客户端。 + +![rmi-dmeo](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/img/rmi-dmeo.png) + +从运行结果可知，因为客户端只有接口，并没有实现类，因此，客户端获得的接口方法返回值实际上是通过网络从服务器端获取的。整个过程实际上非常简单，对客户端来说，客户端持有的 `WorldClock` 接口实际上对应了一个“实现类”，它是由 `Registry` 内部动态生成的，并负责把方法调用通过网络传递到服务器端。而服务器端接收网络调用的服务并不是我们自己编写的 `WorldClockService`，而是 `Registry` 自动生成的代码。我们把客户端的“实现类”称为 `stub`，而服务器端的网络服务类称为 `skeleton`，它会真正调用服务器端的 `WorldClockService`，获取结果，然后把结果通过网络传递给客户端。整个过程由RMI底层负责实现序列化和反序列化： + +```ascii +┌ ─ ─ ─ ─ ─ ─ ─ ─ ┐ ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐ + ┌─────────────┐ ┌─────────────┐ +│ │ Service │ │ │ │ Service │ │ + └─────────────┘ └─────────────┘ +│ ▲ │ │ ▲ │ + │ │ +│ │ │ │ │ │ + ┌─────────────┐ Network ┌───────────────┐ ┌─────────────┐ +│ │ Client Stub ├─┼─────────┼>│Server Skeleton│──>│Service Impl │ │ + └─────────────┘ └───────────────┘ └─────────────┘ +└ ─ ─ ─ ─ ─ ─ ─ ─ ┘ └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘ +``` + +## 弊端 + +Java 的 RMI 严重依赖序列化和反序列化，而这种情况下可能会造成严重的安全漏洞，因为 Java 的序列化和反序列化不但涉及到数据，还涉及到二进制的字节码，即使使用白名单机制也很难保证 100% 排除恶意构造的字节码。因此，使用 RMI 时，双方必须是内网互相信任的机器，不要把 1099 端口暴露在公网上作为对外服务。 + +此外，Java 的 RMI 调用机制决定了双方必须是 Java 程序，其他语言很难调用 Java 的 RMI。如果要使用不同语言进行 RPC 调用，可以选择更通用的协议，例如 [gRPC](https://grpc.io/)。 + + + + + +https://www.jianshu.com/p/2c78554a3f36 \ No newline at end of file diff --git a/docs/nginx/nginx.md b/docs/nginx/nginx.md index 18f7f3ebed..5008692ed4 100644 --- a/docs/nginx/nginx.md +++ b/docs/nginx/nginx.md @@ -2,23 +2,23 @@ # Nginx 学习一路向西 -> Java大猿帅成长手册，**GitHub** [JavaEgg](https://github.com/Jstarfish/JavaEgg) ，N线互联网开发必备技能兵器谱 +> Java大猿帅成长手册，**GitHub** [JavaKeeper](https://github.com/Jstarfish/JavaKeeper) ，N线互联网开发必备技能兵器谱 ## 1. Nginx简介 ### 1.1 Nginx 概述 -- NGINX是一个免费、开源、高性能、轻量级的HTTP和反向代理服务器，也是一个电子邮件（IMAP/POP3）代理服务器，其特点是占有内存少，并发能力强。 Nginx 因为它的稳定性、丰富的模块库、灵活的配置和较低的资源消耗而闻名 。目前应该是几乎所有项目建设必备。 +- NGINX是一个免费、开源、高性能、轻量级的 HTTP 和反向代理服务器，也是一个电子邮件（IMAP/POP3）代理服务器，其特点是占有内存少，并发能力强。 Nginx 因为它的稳定性、丰富的模块库、灵活的配置和较低的资源消耗而闻名 。目前应该是几乎所有项目建设必备。 -- Nginx由内核和一系列模块组成，内核提供web服务的基本功能，如启用网络协议，创建运行环境，接收和分配客户端请求，处理模块之间的交互。Nginx的各种功能和操作都由模块来实现。Nginx的模块从结构上分为核心模块、基础模块和第三方模块。 +- Nginx 由内核和一系列模块组成，内核提供 web 服务的基本功能，如启用网络协议，创建运行环境，接收和分配客户端请求，处理模块之间的交互。Nginx 的各种功能和操作都由模块来实现。Nginx 的模块从结构上分为核心模块、基础模块和第三方模块。 - 核心模块： HTTP模块、EVENT模块和MAIL模块 + 核心模块： HTTP 模块、EVENT 模块和 MAIL 模块 - 基础模块： HTTP Access模块、HTTP FastCGI模块、HTTP Proxy模块和HTTP Rewrite模块 + 基础模块： HTTP Access 模块、HTTP FastCGI 模块、HTTP Proxy 模块和 HTTP Rewrite 模块 - 第三方模块： HTTP Upstream Request Hash模块、Notice模块和HTTP Access Key模块及用户自己开发的模块 + 第三方模块： HTTP Upstream Request Hash 模块、Notice 模块和 HTTP Access Key 模块及用户自己开发的模块 - 这样的设计使Nginx方便开发和扩展，也正因此才使得Nginx功能如此强大。Nginx的模块默认编译进nginx中，如果需要增加或删除模块，需要重新编译Nginx,这一点不如Apache的动态加载模块方便。如果有需要动态加载模块，可以使用由淘宝网发起的web服务器Tengine，在nginx的基础上增加了很多高级特性，完全兼容Nginx，已被国内很多网站采用。 + 这样的设计使 Nginx 方便开发和扩展，也正因此才使得 Nginx 功能如此强大。Nginx 的模块默认编译进 Nginx 中，如果需要增加或删除模块，需要重新编译 Nginx，这一点不如 Apache 的动态加载模块方便。如果有需要动态加载模块，可以使用由淘宝网发起的web 服务器 Tengine，在 Nginx 的基础上增加了很多高级特性，完全兼容 Nginx，已被国内很多网站采用。 - Nginx有很多扩展版本 @@ -31,10 +31,10 @@ ### 1.2 Nginx 作为 web 服务器 -Web服务器也称为WWW(WORLD WIDE WEB)服务器，主要功能是提供网上信息浏览服务，常常以B/S（Browser/Server）方式提供服务。 +Web服务器也称为 WWW(WORLD WIDE WEB)服务器，主要功能是提供网上信息浏览服务，常常以B/S（Browser/Server）方式提供服务。 -- 应用层使用HTTP协议。 -- HTML文档格式。 +- 应用层使用 HTTP 协议。 +- HTML 文档格式。 - 浏览器统一资源定位器(URL)。 Nginx 可以作为静态页面的 web 服务器，同时还支持 CGI 协议的动态语言，比如 perl、php 等，但是不支持 java。Java 程序一般都通过与 Tomcat 配合完成。 @@ -73,11 +73,11 @@ Apache HTTP Server和Nginx本身不支持生成动态页面，但它们可以通 ### 1.4 反向代理 与 负载均衡 - 反向代理正好与正向代理相反，反向代理是指以代理服务器来接收Internet上的连接请求，然后将请求转发到内部网络上的服务器，并将服务器上得到的结果返回给客户端，此时代理服务器对外表现就是一个服务器，客户端对代理是无感知的。 +反向代理正好与正向代理相反，反向代理是指以代理服务器来接收Internet上的连接请求，然后将请求转发到内部网络上的服务器，并将服务器上得到的结果返回给客户端，此时代理服务器对外表现就是一个服务器，客户端对代理是无感知的。 **反向代理“代理”的是服务端**。 -再比如，你想本本分分的在“优酷”上看个“爱情片”，youku.com会把你的请求分发到存放片片的那台机器上，这个就是所谓的”反向代理“。 +再比如，你想本本分分的在“优酷”上看个“爱情片”，youku.com 会把你的请求分发到存放片片的那台机器上，这个就是所谓的”反向代理“。 ![reverse-proxy](../_images/nginx/reverse-proxy.png) @@ -98,7 +98,7 @@ TODO: 留一个负载均衡详细介绍传送门 #### 地址重定向 -Nginx 的Rewrite主要的功能就是实现URL重写 +Nginx 的 Rewrite 主要的功能就是实现 URL 重写 比如输入360.com 跳转到了360.cn，baidu.cn跳转到了baidu.com @@ -140,7 +140,7 @@ Nginx 的Rewrite主要的功能就是实现URL重写 3. 解压、配置（Nginx支持各种配置选项,文末一一列出 [Nginx配置选项](#Nginx配置选项) ）、编译、安装nginx ```sh - tar -zxvf nginx-1.15.tar.gz cd nginx-1.16.1 + tar -zxvf nginx-1.15.tar.gz cd nginx-1.16.1 ./configure make && sudo make install diff --git a/docs/notes/short-video.md b/docs/notes/short-video.md new file mode 100755 index 0000000000..2038ea31c8 --- /dev/null +++ b/docs/notes/short-video.md @@ -0,0 +1,93 @@ +![img](https://static001.geekbang.org/resource/image/2b/2c/2be44cf21a81b42f9637c049d025fd2c.jpg) + +## 定位 + +## 创作 + +### 选题 + +一个好选题是创作爆款短视频的第一步。 + +选题决定着创作方向，也代表着创作者对外输出的价值观，而这种价值观会直接影响创作者的个人定位和 IP 塑造。我见到过很多陷入瓶颈期的短视频创作者，并不是他们不知道怎么拍，而是不知道要拍什么，觉得自己能拍的东西都拍完了，从而陷入到创作焦虑中。 + +当然，现在的短视频平台一般都会推出一些官方活动和话题，我们可以结合自身经验、日常观察、职业背景或用户需求等角度来进行选择和创作。不过这只能算是一种权宜之策，我们还是需要真正掌握确定合适选题的方法，这样不管在哪个创作阶段，我们的创作方向才都是清晰、不迷茫的。 + +技巧一：和用户关心的话题相结合 + +不管你是采用哪种方法来寻找选题，都不要忘记热点是最重要的维度之一。 + +技巧二：和个人视角相匹配 + +不能转化成自我视角的选题，没有任何意义。 + +这句话是什么意思呢？就是指不管我们找的选题话题度有多高，如果没有办法以自我视角去剖析、解读，没有办法带给用户更多价值的话，那么这个选题本身也就没有任何意义。 + +### 标题 + +《溥仪重游故宫，看到墙上挂错了照片》 + +《溥仪重游故宫，指出一处错误，专家反倒骂其无知》 + +试想一下，如果你正在浏览短视频，你更愿意点开哪条视频来看呢？很显然，是第 2 条，对不对？因为它具备足够的悬念性，会吸引用户迫切地想了解发生了什么事，以及最后的答案是什么，视频的打开率自然就会很高。 + +数字标题 + +所谓的数字标题，就是将自己创作的短视频中最核心、最重要、最吸引人的内容以数字的方式呈现出来，作为标题的主打卖点。 + +看完这本书明白了 5 个道理，让我受益匪浅！ +明白了这本书中的 5 个道理，我卸载了游戏，还清了欠款！ + +### 封面 + +“**看书先看皮，看报先看题**” + +**悬念封面**：利用吸引人的画面、场景、人物等方式留住用户的注意力。一般来说，封面上往往是事件的起因，而视频内容是答案。一定要记得，不要在封面上留足噱头，但是内容中没有解释，这样的行为会非常影响用户的观感。 +**效果封面**：通过多种修饰的方法将最好的视觉效果呈现在封面上，同时还可以添加一些内容中的关键词，方便后台机器准确识别自己的内容，然后匹配给合适的用户群体。 +**借力封面**：借助外界有影响力的热点、人物、事件等元素，来吸引用户观看内容。这里一定要注意，采用借力的封面上不能只体现了热点，而没有凸显内容价值，否则会留不住用户。 +**猎奇封面**：以满足用户的好奇心为出发点，通过新鲜、新奇的事物来吸引用户观看自己的内容。注意，采用猎奇封面时，需要能在内容中提供具体的价值点。另外，我建议最好不要经常使用猎奇封面，不然一旦内容中提供的信息不能满足用户的需求，就会被平台判定为标题党。 + + +**所有短视频的本质其实就是在为用户提供价值。不管你创作的是搞笑类视频还是知识科普类视频，你都在对外传递价值，不过就是传递的价值种类不一样罢了。** + + + +**故事封面**：就是采用“图片 + 文字”的搭配方式将用户代入具体的情感场景下，让用户产生情感上的共鸣。请注意，故事封面最好选择的是带有情绪力量的图片。 +**精彩封面**：就是将内容中最惹人注意、或者让人眼前一亮的画面作为主打卖点，以此吸引用户的观看兴趣。请注意，千万不要从审丑或者哗众取宠的方向入手。 +**人设封面**：就是通过真人出镜的方式作为主打，封面上要提供有热度的关键词，内容中要提供信息的完整性。 +**重点信息封面**：就是将内容中最有价值或有亮点的信息以醒目的文字方式呈现出来，它可以作为标题的补充，以此快速让用户停留下来点开内容，寻找答案。 + + + +### 剪辑 + +使用具体案例学习剪辑时，一定要从易到难、循序渐进，一点点地拓展自己的剪辑水平。同时，你也可以利用碎片化时间进行反复练习，这样更能提升你的剪辑水平。 +在使用数据反馈法学习剪辑时，你要注意，各个短视频平台推出的剪辑 App 上都有热门的案例、模版、道具、教程等，上面也都有具体数字的标注，比如 10 万人在使用等字样。如果数据越大，就证明这个剪辑功能、方法越受用户的喜爱，你可以优先放到自己的作品中进行学习。 +在使用解决问题的方法来学习剪辑技巧时，注意不要将剪辑的目标定得太高，而是要一个个地来解决具体的剪辑问题。因为一口吃不成个胖子，只有不断地给予自己正向反馈，你才能更有信心，更能坚持下去。 +在使用参与平台活动来学习剪辑技巧时，你可以多关注下目前热门平台上的官方信息，试验着将最新的剪辑手法一步步应用到自己的创作上。 + + +日本的认知科学家今井睦美在《深度学习》中提到：**实现更好的学习，首先需要明确自己的学习目的。** 也就是说，我们必须搞清楚自己学习剪辑的目的是什么，这样才能真正学到有用的剪辑技巧。 + + + +## 运营 + +内容运营和数据运营的角度 + +我们先来学习下如何用电影创作的方式，即利用“拉片法”来掌握爆款短视频的内容运营技巧。 + +### 点赞率 + + + +## 变现 + +## 小结 + +重视时间的复利 + +“复利”这个词一般是出现在金融领域里的，通俗点来说就是“利滚利”。而所谓“时间的复利”，就是指在单向的时间维度里，通过点滴地学习，可以获得多倍的增长回报，爱因斯坦也把这种时间 + 复利的效应比作世界第八大奇迹。 + +看未来你想成为什么样的人，你就清楚现在的选择是否重要；看未来你想实现什么样的梦想，你就知道现在要选择什么样的路 + +另外，我的经验还告诉我，在砥砺前行时，一定记得要多跟自己比较，少拿别人的标准来影响自己，因为自己前行的一小步，才是成为人生赢家的一大步。 \ No newline at end of file diff --git a/docs/others/.DS_Store b/docs/others/.DS_Store new file mode 100644 index 0000000000..12eaa0d36d Binary files /dev/null and b/docs/others/.DS_Store differ diff --git "a/docs/others/HashMap14\351\227\256.md" "b/docs/others/HashMap14\351\227\256.md" new file mode 100644 index 0000000000..ce4696d0ea --- /dev/null +++ "b/docs/others/HashMap14\351\227\256.md" @@ -0,0 +1,286 @@ +### 1.HashMap的底层数据结构是什么？ + +在JDK1.7中和JDK1.8中有所区别： + +在JDK1.7中，由”数组+链表“组成，数组是HashMap的主体，链表则是主要为了解决哈希冲突而存在的。 + +在JDK1.8中，有“数组+链表+红黑树”组成。当链表过长，则会严重影响HashMap的性能，红黑树搜索时间复杂度是O(logn)，而链表是O(n)。因此，JDK1.8对数据结构做了进一步的优化，引入了红黑树，链表和红黑树在达到一定条件会进行转换： + +当链表超过8且数组长度(数据总量)超过64才会转为红黑树 +将链表转换成红黑树前会判断，如果当前数组的长度小于64，那么会选择先进行数组扩容，而不是转换为红黑树，以减少搜索时间。 + +### 2.说一下HashMap的特点 + +hashmap存取是无序的 +键和值位置都可以是null，但是键位置只能是一个null +键位置是唯一的，底层的数据结构是控制键的 +jdk1.8前数据结构是：链表+数组jdk1.8之后是：数组+链表+红黑树 +阈值（边界值）>8并且数组长度大于64，才将链表转换成红黑树，变成红黑树的目的是提高搜索速度，高效查询 + +### 3.解决hash冲突的办法有哪些?HashMap用的哪种？ + +解决Hash冲突方法有：开放定址法、再哈希法、链地址法（HashMap中常见的拉链法）、简历公共溢出区。HashMap中采用的是链地址法。 + +开放定址法也称为再散列法，基本思想就是，如果p=H(key)出现冲突时，则以p为基础，再次hash，p1=H(p)，如果p1再次出现冲突，则以p1为基础，以此类推，直到找到一个不冲突的哈希地址pi。因此开放定址法所需要的hash表的长度要大于等于所需要存放的元素，而且因为存在再次hash，所以只能在删除的节点上做标记，而不能真正删除节点 +再哈希法（双重散列，多重散列），提供多个不同的hash函数，R1=H1(key1)发生冲突时，再计算R2=H2（key1），直到没有冲突为止。这样做虽然不易产生堆集，但增加了计算的时间。 +链地址法（拉链法），将哈希值相同的元素构成一个同义词的单链表，并将单链表的头指针存放在哈希表的第i个单元中，查找、插入和删除主要在同义词链表中进行，链表法适用于经常进行插入和删除的情况。 +建立公共溢出区，将哈希表分为公共表和溢出表，当溢出发生时，将所有溢出数据统一放到溢出区 +注意开放定址法和再哈希法的区别是 + +开放定址法只能使用同一种hash函数进行再次hash，再哈希法可以调用多种不同的hash函数进行再次hash + +### 4.为什么要在数组长度大于64之后，链表才会进化为红黑树 + +在数组比较小时如果出现红黑树结构，反而会降低效率，而红黑树需要进行左旋右旋，变色，这些操作来保持平衡，同时数组长度小于64时，搜索时间相对要快些，总之是为了加快搜索速度，提高性能 + +JDK1.8以前HashMap的实现是数组+链表，即使哈希函数取得再好，也很难达到元素百分百均匀分布。当HashMap中有大量的元素都存放在同一个桶中时，这个桶下有一条长长的链表，此时HashMap就相当于单链表，假如单链表有n个元素，遍历的时间复杂度就从O（1）退化成O（n），完全失去了它的优势，为了解决此种情况，JDK1.8中引入了红黑树（查找的时间复杂度为O（logn））来优化这种问题 + +### 5.为什么加载因子设置为0.75，初始化临界值是12？ + +HashMap中的threshold是HashMap所能容纳键值对的最大值。计算公式为length*LoadFactory。也就是说，在数组定义好长度之后，负载因子越大，所能容纳的键值对个数也越大 + +loadFactory越趋近于1，那么数组中存放的数据（entry也就越来越多），数据也就越密集，也就会有更多的链表长度处于更长的数值，我们的查询效率就会越低，当我们添加数据，产生hash冲突的概率也会更高 + +默认的loadFactory是0.75，loadFactory越小，越趋近于0，数组中个存放的数据(entry)也就越少，表现得更加稀疏 + + +0.75是对空间和时间效率的一种平衡选择 + +如果负载因子小一些比如是0.4，那么初始长度16*0.4=6，数组占满6个空间就进行扩容，很多空间可能元素很少甚至没有元素，会造成大量的空间被浪费 + +如果负载因子大一些比如是0.9，这样会导致扩容之前查找元素的效率非常低 + +loadfactory设置为0.75是经过多重计算检验得到的可靠值，可以最大程度的减少rehash的次数，避免过多的性能消耗 + +### 6.哈希表底层采用何种算法计算hash值？还有哪些算法可以计算出hash值？ + +hashCode方法是Object中的方法，所有的类都可以对其进行使用，首先底层通过调用hashCode方法生成初始hash值h1，然后将h1无符号右移16位得到h2，之后将h1与h2进行按位异或（^）运算得到最终hash值h3，之后将h3与(length-1)进行按位与（&）运算得到hash表索引 + +其他可以计算出hash值的算法有 + +平方取中法 +取余数 +伪随机数法 + +### 7.当两个对象的hashCode相等时会怎样 + +hashCode相等产生hash碰撞，hashCode相等会调用equals方法比较内容是否相等，内容如果相等则会进行覆盖，内容如果不等则会连接到链表后方，链表长度超过8且数组长度超过64，会转变成红黑树节点 + +### 8.何时发生哈希碰撞和什么是哈希碰撞，如何解决哈希碰撞？ + +只要两个元素的key计算的hash码值相同就会发生hash碰撞，jdk8之前使用链表解决哈希碰撞，jdk8之后使用链表+红黑树解决哈希碰撞 + +### 9.HashMap的put方法流程 + +以jdk8为例，简要流程如下： + +首先根据key的值计算hash值，找到该元素在数组中存储的下标 +如果数组是空的，则调用resize进行初始化； +如果没有哈希冲突直接放在对应的数组下标里 +如果冲突了，且key已经存在，就覆盖掉value +如果冲突后是链表结构，就判断该链表是否大于8，如果大于8并且数组容量小于64，就进行扩容；如果链表节点数量大于8并且数组的容量大于64，则将这个结构转换成红黑树；否则，链表插入键值对，若key存在，就覆盖掉value +如果冲突后，发现该节点是红黑树，就将这个节点挂在树上 + +### 10.HashMap的扩容方式 + +HashMap在容量超过负载因子所定义的容量之后，就会扩容。java里的数组是无法自己扩容的，将HashMap的大小扩大为原来数组的两倍 + +我们来看jdk1.8扩容的源码 + + final Node[] resize() { + //oldTab：引用扩容前的哈希表 + Node[] oldTab = table; + //oldCap：表示扩容前的table数组的长度 + int oldCap = (oldTab == null) ? 0 : oldTab.length; + //获得旧哈希表的扩容阈值 + int oldThr = threshold; + //newCap:扩容之后table数组大小 + //newThr:扩容之后下次触发扩容的条件 + int newCap, newThr = 0; + //条件成立说明hashMap中的散列表已经初始化过了，是一次正常扩容 + if (oldCap > 0) { + //判断旧的容量是否大于等于最大容量，如果是，则无法扩容，并且设置扩容条件为int最大值， + //这种情况属于非常少数的情况 + if (oldCap >= MAXIMUM_CAPACITY) { + threshold = Integer.MAX_VALUE; + return oldTab; + }//设置newCap新容量为oldCap旧容量的二倍（<<1）,并且<最大容量，而且>=16，则新阈值等于旧阈值的两倍 + else if ((newCap = oldCap << 1) < MAXIMUM_CAPACITY && + oldCap >= DEFAULT_INITIAL_CAPACITY) + newThr = oldThr << 1; // double threshold + } + //如果oldCap=0并且边界值大于0，说明散列表是null，但此时oldThr>0 + //说明此时hashMap的创建是通过指定的构造方法创建的,新容量直接等于阈值 + //1.new HashMap(intitCap,loadFactor) + //2.new HashMap(initCap) + //3.new HashMap(map) + else if (oldThr > 0) // initial capacity was placed in threshold + newCap = oldThr; + //这种情况下oldThr=0;oldCap=0，说明没经过初始化，创建hashMap + //的时候是通过new HashMap()的方式创建的 + else { // zero initial threshold signifies using defaults + newCap = DEFAULT_INITIAL_CAPACITY; + newThr = (int)(DEFAULT_LOAD_FACTOR * DEFAULT_INITIAL_CAPACITY); + } + //newThr为0时，通过newCap和loadFactor计算出一个newThr + if (newThr == 0) { + //容量*0.75 + float ft = (float)newCap * loadFactor; + newThr = (newCap < MAXIMUM_CAPACITY && ft < (float)MAXIMUM_CAPACITY ? + (int)ft : Integer.MAX_VALUE); + } + threshold = newThr; + @SuppressWarnings({"rawtypes","unchecked"}) + //根据上面计算出的结果创建一个更长更大的数组 + Node[] newTab = (Node[])new Node[newCap]; + //将table指向新创建的数组 + table = newTab; + //本次扩容之前table不为null + if (oldTab != null) { + //对数组中的元素进行遍历 + for (int j = 0; j < oldCap; ++j) { + //设置e为当前node节点 + Node e; + //当前桶位数据不为空，但不能知道里面是单个元素，还是链表或红黑树， + //e = oldTab[j]，先用e记录下当前元素 + if ((e = oldTab[j]) != null) { + //将老数组j桶位置为空，方便回收 + oldTab[j] = null; + //如果e节点不存在下一个节点，说明e是单个元素，则直接放置在新数组的桶位 + if (e.next == null) + newTab[e.hash & (newCap - 1)] = e; + //如果e是树节点，证明该节点处于红黑树中 + else if (e instanceof TreeNode) + ((TreeNode)e).split(this, newTab, j, oldCap); + //e为链表节点，则对链表进行遍历 + else { // preserve order + //低位链表：存放在扩容之后的数组的下标位置，与当前数组下标位置一致 + //loHead：低位链表头节点 + //loTail低位链表尾节点 + Node loHead = null, loTail = null; + //高位链表，存放扩容之后的数组的下标位置，=原索引+扩容之前数组容量 + //hiHead:高位链表头节点 + //hiTail:高位链表尾节点 + Node hiHead = null, hiTail = null; + Node next; + do { + next = e.next; + //oldCap为16:10000，与e.hsah做&运算可以得到高位为1还是0 + //高位为0，放在低位链表 + if ((e.hash & oldCap) == 0) { + if (loTail == null) + //loHead指向e + loHead = e; + else + loTail.next = e; + loTail = e; + } + //高位为1，放在高位链表 + else { + if (hiTail == null) + hiHead = e; + else + hiTail.next = e; + hiTail = e; + } + } while ((e = next) != null); + //低位链表已成，将头节点loHead指向在原位 + if (loTail != null) { + loTail.next = null; + newTab[j] = loHead; + } + //高位链表已成，将头节点指向新索引 + if (hiTail != null) { + hiTail.next = null; + newTab[j + oldCap] = hiHead; + } + } + } + } + } + return newTab; + } + +扩容之后原位置的节点只有两种调整 + +保持原位置不动（新bit位为0时） +散列原索引+扩容大小的位置去（新bit位为1时） +扩容之后元素的散列设置的非常巧妙，节省了计算hash值的时间，我们来看一下具体的实现 + + +当数组长度从16到32，其实只是多了一个bit位的运算，我们只需要在意那个多出来的bit为是0还是1，是0的话索引不变，是1的话索引变为当前索引值+扩容的长度，比如5变成5+16=21 + +这样的扩容方式不仅节省了重新计算hash的时间，而且保证了当前桶中的元素总数一定小于等于原来桶中的元素数量，避免了更严重的hash冲突，均匀的把之前冲突的节点分散到新的桶中去 + +### 11.一般用什么作为HashMap的key？ + +一般用Integer、String这种不可变类当HashMap当key + +因为String是不可变的，当创建字符串时，它的hashcode被缓存下来，不需要再次计算，相对于其他对象更快 +因为获取对象的时候要用到equals()和hashCode()方法，那么键对象正确的重写这两个方法是非常重要的，这些类很规范的重写了hashCode()以及equals()方法 + +### 12.为什么Map桶中节点个数超过8才转为红黑树？ + +8作为阈值作为HashMap的成员变量，在源码的注释中并没有说明阈值为什么是8 + +在HashMap中有这样一段注释说明，我们继续看 + + * Because TreeNodes are about twice the size of regular nodes, we + * use them only when bins contain enough nodes to warrant use + * (see TREEIFY_THRESHOLD). And when they become too small (due to + * removal or resizing) they are converted back to plain bins. In + * usages with well-distributed user hashCodes, tree bins are + * rarely used. Ideally, under random hashCodes, the frequency of + * nodes in bins follows a Poisson distribution + * (http://en.wikipedia.org/wiki/Poisson_distribution) with a + * parameter of about 0.5 on average for the default resizing + * threshold of 0.75, although with a large variance because of + * resizing granularity. Ignoring variance, the expected + * occurrences of list size k are (exp(-0.5) * pow(0.5, k) / + * factorial(k)). + + +因为树节点的大小大约是普通节点的两倍，所以我们只在箱子包含足够的节点时才使用树节点（参见TREEIFY_THRESHOLD）。 +当他们边的太小（由于删除或调整大小）时，就会被转换回普通的桶，在使用分布良好的hashcode时，很少使用树箱。 +理想情况下，在随机哈希码下，箱子中节点的频率服从泊松分布 +第一个值是： + + * 0: 0.60653066 + * 1: 0.30326533 + * 2: 0.07581633 + * 3: 0.01263606 + * 4: 0.00157952 + * 5: 0.00015795 + * 6: 0.00001316 + * 7: 0.00000094 + * 8: 0.00000006 + +more: less than 1 in ten million + +树节点占用空间是普通Node的两倍，如果链表节点不够多却转换成红黑树，无疑会耗费大量的空间资源，并且在随机hash算法下的所有bin节点分布频率遵从泊松分布，链表长度达到8的概率只有0.00000006，几乎是不可能事件，所以8的计算是经过重重科学考量的 + +从平均查找长度来看，红黑树的平均查找长度是logn，如果长度为8，则logn=3，而链表的平均查找长度为n/4，长度为8时，n/2=4，所以阈值8能大大提高搜索速度 +当长度为6时红黑树退化为链表是因为logn=log6约等于2.6，而n/2=6/2=3，两者相差不大，而红黑树节点占用更多的内存空间，所以此时转换最为友好 + +### 13.HashMap为什么线程不安全？ + +多线程下扩容死循环。JDK1.7中的HashMap使用头插法插入元素，在多线程的环境下，扩容的时候有可能导致环形链表的出现，形成死循环。因此JDK1.8使用尾插法插入元素，在扩容时会保持链表元素原本的顺序，不会出现环形链表的问题 +多线程的put可能导致元素的丢失。多线程同时执行put操作，如果计算出来的索引位置是相同的，那会造成前一个key被后一个key覆盖，从而导致元素的丢失。此问题在JDK1.7和JDK1.8中都存在 +put和get并发时，可能导致get为null。线程1执行put时，因为元素个数超出threshold而导致rehash，线程2此时执行get，有可能导致这个问题，此问题在JDK1.7和JDK1.8中都存在 + +### 14.计算hash值时为什么要让低16bit和高16bit进行异或处理 + +我们计算索引需要将hashCode值与length-1进行按位与运算，如果数组长度很小，比如16，这样的值和hashCode做异或实际上只有hashCode值的后4位在进行运算，hash值是一个随机值，而如果产生的hashCode值高位变化很大，而低位变化很小，那么有很大概率造成哈希冲突，所以我们为了使元素更好的散列，将hash值的高位也利用起来\ +举个例子 + +如果我们不对hashCode进行按位异或，直接将hash和length-1进行按位与运算就有可能出现以下的情况 + + +如果下一次生成的hashCode值高位起伏很大，而低位几乎没有变化时，高位无法参与运算 + +可以看到，两次计算出的hash相等，产生了hash冲突 + +所以无符号右移16位的目的是使高混乱度地区与地混乱度地区做一个中和，提高低位的随机性，减少哈希冲突 +———————————————— +版权声明：本文为CSDN博主「温文艾尔」的原创文章，遵循CC 4.0 BY-SA版权协议，转载请附上原文出处链接及本声明。 +原文链接：https://blog.csdn.net/wenwenaier/article/details/123335563 \ No newline at end of file diff --git "a/docs/others/Hash\345\206\262\347\252\201\350\247\243\345\206\263\346\226\271\346\263\225.md" "b/docs/others/Hash\345\206\262\347\252\201\350\247\243\345\206\263\346\226\271\346\263\225.md" new file mode 100755 index 0000000000..e69de29bb2 diff --git a/docs/others/Unit-Testing.md b/docs/others/Unit-Testing.md new file mode 100755 index 0000000000..831068cd3a --- /dev/null +++ b/docs/others/Unit-Testing.md @@ -0,0 +1,29 @@ +```java + +@Test +public void should_add_todo_item() { + // 准备 + TodoItemRepository repository = mock(TodoItemRepository.class); + when(repository.save(any())).then(returnsFirstArg()); + TodoItemService service = new TodoItemService(repository); + + // 执行 + TodoItem item = service.addTodoItem(new TodoParameter("foo")); + + // 断言 + assertThat(item.getContent()).isEqualTo("foo"); + + // 清理(可选) + +} +``` + +我把这个测试分成了四段，分别是准备、执行、断言和清理，这也是一般测试都会具备的四个阶段，我们分别来看一下。 + +准备。这个阶段是为了测试所做的一些准备，比如启动外部依赖的服务，存储一些预置的数据。在我们这个例子里面就是设置所需组件的行为，然后将这些组件组装了起来。 + +执行。这个阶段是整个测试中最核心的部分，触发被测目标的行为。通常来说，它就是一个测试点，在大多数情况下，执行应该就是一个函数调用。如果是测试外部系统，就是发出一个请求。在我们这段代码里，它就是调用了一个函数。 + +断言。断言是我们的预期，它负责验证执行的结果是否正确。比如，被测系统是否返回了正确的应答。在这个例子，我们验证的是 Todo 项的内容是否是我们添加进去的内容。 + +清理。清理是一个可能会有的部分。如果在测试中使到了外部资源，在这个部分要及时地释放掉，保证测试环境被还原到一个最初的状态，就像什么都没发生过一样。比如，我们在测试过程中向数据库插入了数据，执行之后，要删除测试过程中插入的数据。一些测试框架对一些通用的情况已经提供支持，比如之前我们用到的临时文件。 \ No newline at end of file diff --git "a/docs/others/\344\273\243\347\240\201\344\271\213\344\270\221.md" "b/docs/others/\344\273\243\347\240\201\344\271\213\344\270\221.md" new file mode 100755 index 0000000000..9de91e9a81 --- /dev/null +++ "b/docs/others/\344\273\243\347\240\201\344\271\213\344\270\221.md" @@ -0,0 +1,191 @@ +# 你的代码 ——— 味儿不对 + +> 看过《重构》的程序员，都知道 “bad smell ——坏味道”，书里列出了 22 项代码坏味道 +> +> 其实最早时候，我很 “普信”，我写的代码，不管是命名和结构都很好，没有什么坏味道，后来项目组有 CR 之后，才发现好多问题 + +![](https://tva1.sinaimg.cn/large/e6c9d24ely1h0hifxh3igj20fg0igabu.jpg) + +“写代码”有两个维度：正确性和可维护性，不要只关注正确性。能把代码写对，是每个程序员的必备技能，但能够把代码写得更具可维护性，这是一个程序员从业余迈向职业的第一步。 + + + +![http://www.osnews.com/images/comics/wtfm.jpg](http://www.osnews.com/images/comics/wtfm.jpg) + + + +## 起名有意义 + +先说下命名吧，其实我也知道命名要有意义，不管是项目名、模块名、包名、类名、方法名、变量名或者参数名， + +虽然现在很少有人用 int a ,b ,c 这样的东西，但是起一个有业务含义的好名字，对英语水平要求还是挺高的 + +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/others/%E6%88%AA%E5%B1%8F2022-03-18%20%E4%B8%8B%E5%8D%888.03.46.png) + + + +```java +public void processData(){ + +} +``` + +命名最好有业务含义，符合英语语法，我第一份工作做的都是海外产品，有对应的“词库”，很顺利成章的写代码也直接查就行了 + +![]() + + + +那没有词库呢，我们写代码还是得查词，除了查词典，我们可以把我们想到的词去 github 搜一搜，看下别人怎么用的，https://unbug.github.io/codelf/ 可以试试这个 + +> 类名是一个名词，表示一个对象，而方法名则是一个动词，或者是一个动宾短语，表示一个动作 + +这个说个我遇到的错误，Service 和 Server 用错之后被无情嘲笑 + + + +当然，取好名字前提是问题分解的干净合理，如果你的代码整个处理流程就 1 个方法，叫个 processData 没问题吧？ 这又属于其他范畴了（下文的代码太长） + + + + + +有些命名，可能一个词会对应好多单词，如果是不同的人负责不同模块，有些同学会不管3721，各自用各自的命名，然后又会出现各种 + +convertUtils 进去发现这玩意就是个转换的 + + + +## 减少 Ctrl C + Ctrl V + +还有就是重复代码 + +开发中，其实会有各种问题，现在的互联网，迭代周期快，当接到迭代需求的时候，最简单粗暴的办法就是找个类似的逻辑，直接 Ctrl C + Ctrl V（idea 都不需要 Ctrl S），改叭改叭，能写个单测就不错了，哪有时间去改写之前的烂代码， + +> 这个我想说个题外话，互联网中我感觉感触最深的就是搜狗同学，刚被腾讯收购并入pcg epc 规范，每天就是改各种规范 + +这个最大的问题，其实就是重复代码，我们会复制很多结构或功能相似的代码到各个业务逻辑中，而不是去想办法提炼函数 + + + +## 代码别太长 + +写代码其实不止是为了让程序读代码，更重要的应该是让程序员读代码， + +所以代码别太长，其实就是想让我们看起来更方便一些，这里也说下几种 + +- 类行数太长 +- 方法行数太长 +- 参数列表太长 +- 注释别啰里啰嗦 +- 无用代码舍得删 + +### 类太大、方法太大、参数太长 + +![](https://cdn.jsdelivr.net/gh/Jstarfish/picBed/others/%E6%88%AA%E5%B1%8F2022-03-18%20%E4%B8%8B%E5%8D%888.37.44.png) + + + +多少行算长，有人说 80，有人说 100 ，还有人说要一屏幕放的下 + +写代码，如果“平铺直叙”的话，很容易写出“面条型代码”，好多逻辑都在一个大方法 + + + +参数太长，如果都是需要的，我们一般会合并成一个对象，如果传入的某个参数可以通过另一个参数获取到，那我们可以在方法内查询，而不是全部作为入参 + + + +### 注释内容 + +- 要明白，注释不能美化糟糕的代码 +- 不要不舍得删不需要的代码，不删除的意义不就是为了下次可以更方便 Ctrl C V 吗，有版本控制会帮我们记录的，要那么一坨坨的注释代码干啥 + + + +### 发散式变化、霰弹式修改、依恋情结 + +- 如果一个类不是单一职责的，则不同的变化可能都需要修改这个类，说明存在发散式变化，应考虑将不同的变化分离开。 +- 如果某个变化需要修改多个类的内容，则说明存在霰弹式修改，应考虑用「搬移函数」和「搬移字段」将这些需要修改的代码放进同一个模块，通过组合或者变换的策略将分散的代码拽到一起 +- 如果函数对于某个类的兴趣高于了自己所处的类，说明存在依恋情结，应考虑将函数转移到他应有的类中。 + + + +### 数据泥团 + +有时候会发现三四个相同的字段，在多个类和函数中均出现，这时候说明有必要给这一组字段建立一个类，将其封装起来。 + + + +### 过多的 if...else 或者使用 switch + +过多的 if...else 或者 switch ，都应该考虑用多态来替换掉。甚至有些人认为除个别情况外，代码中就不应该存在 if...else + + + + + +## 圈复杂度 + +> 在软件开发中，有一个衡量代码复杂度常用的标准，叫做「圈复杂度」（Cyclomatic complexity，简称 CC），圈复杂度越高，代码越复杂，理解和维护成本就越高。 + +## 语句 + +使用嵌套 for 循环，更扯淡的是在 for 循环里加了个 update sql + +是否else + +是否有重复的 switch + +变量申明后是否立即赋值 + +返回值是否可以用 Optional + + + + + + + +知道自己的代码味道不对，那接着我再说说改味道的方法 + + + +> +> +> 可读性 +> +> 适当的注释，方法内适当的拆分以保证方法内主流程简洁，减少多层嵌套判断，规避复杂判断条件，采用公司统一编码风格等等 + + + +## 重构的手法 + +Idea 中可以直接全局改名，提取方法（Extrat Method），我们 Javaer 现在大部分都用 Idea，这玩意很多强大的功能我们却没用到，其实好多重构可以通过 Idea 自动化 + + + +## 巧用 IDEA + + + + + + + +介绍codeCC + +https://cloud.tencent.com/developer/article/1371896 + + + +- 《重构》 + +- https://refactoring.guru/refactoring + +- 《代码整洁之道》 + +- https://mp.weixin.qq.com/s/AjubL4vVhFa_FIlaopLVCA + + + diff --git "a/docs/others/\345\215\225\345\205\203\346\265\213\350\257\225\350\260\203\347\240\224.md" "b/docs/others/\345\215\225\345\205\203\346\265\213\350\257\225\350\260\203\347\240\224.md" new file mode 100644 index 0000000000..52183ce765 --- /dev/null +++ "b/docs/others/\345\215\225\345\205\203\346\265\213\350\257\225\350\260\203\347\240\224.md" @@ -0,0 +1,717 @@ +# 单元测试 + +[TOC] + +# 一、什么是单元测试 + +## 1.1 定义 + +维基百科的解释： + +>单元测试（英语：Unit Testing）又称为模块测试，是针对程序模块)（软件设计的最小单位）来进行正确性检验的测试工作 +> +>指由开发人员对软件中的最小可测试单元进行检查和验证。对于单元测试中单元的含义，一般来说，要根据实际情况去判定其具体含义，如C语言中单元指一个函数，Java里单元指一个方法，图形化的软件中可以指一个窗口或一个菜单等。总的来说，单元就是人为规定的最小的被测功能模块。单元测试是在软件开发过程中要进行的最低级别的测试活动，软件的独立单元将在与程序的其他部分相隔离的情况下进行测试。 + +1. 单元测试（unit testing），是指对软件中的最小可测试单元进行检查和验证。在Java中单元测试的最小单元是方法。 +2. 单元测试是开发者编写的一小段代码，用于检验被测代码的一个很小的、很明确的功能是否正确。执行单元测试，就是为了证明这段代码的行为和我们期望是否一致。 + +## 1.2 好处 + +测试是开发的一个非常重要的方面，可以在很大程度上决定一个应用程序的命运。良好的测试可以在早期捕获导致应用程序崩溃的问题，但较差的测试往往总是导致故障和停机。 + +![img](https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X2dpZi9mQ3BkMWNmOGlhY2FMcVVseWZTQ1JWTHdKQ2dhNUNCc1drR2FmVGlhd1k4TThqejFCRW9TeTEzaWJ1OUVVcktCV2ljRWdkYVZZaWI1dUpiQWE2aWIzcklmSEtnUS82NDA?x-oss-process=image/format,png) + +单元测试的好处包括但不限于： + +- **提升软件质量** + 优质的单元测试可以保障开发质量和程序的鲁棒性。越早发现的缺陷，其修复的成本越低。 + +- **促进代码优化** + 单元测试的编写者和维护者都是开发工程师，在这个过程当中开发人员会不断去审视自己的代码，从而（潜意识）去优化自己的代码。 + +- **提升研发效率** + 编写单元测试，表面上是占用了项目研发时间，但是在后续的联调、集成、回归测试阶段，单测覆盖率高的代码缺陷少、问题已修复，有助于提升整体的研发效率。 + +- **增加重构自信** + 代码的重构一般会涉及较为底层的改动，比如修改底层的数据结构等，上层服务经常会受到影响；在有单元测试的保障下，我们对重构出来的代码会多一份底气。 + +## 1.3 目的 + +#### 异（che）常（huo）场（xian）景（chang） + +相信大家肯定遇到过以下几种情况： + +- 测试环境没问题，线上怎么就不行。 + +- 所有异常捕获，一切尽在掌控（你以为你以为的是你以为的）。 + +- 祖传代码，改个小功能（只有上帝知道）。 + +- ..... + +要想故障出的少，还得单测好好搞。 + +#### **写单元测试的两个动机**： + +- 保证或验证实现功能。 +- 保护已经实现的功能不被破坏。 + +#### 为什么要写单测： + +**提高代码正确性** + +- 流程判读符合预期，按照步骤运行，逻辑正确。 + +- 执行结果符合预期，代码执行后，结果正确。 + +- 异常输出符合预期，执行异常或者错误，超越程序边界，保护自身。 + +- 代码质量符合预期，效率，响应时间，资源消耗等。 + +**发现设计问题** + +- 代码可测性差 +- 方法封装不合理 +- 流程不合理 +- 设计漏洞等 + +**提升代码可读性** + +- 易写单测的方法一定是简单好理解的，可读性是高的，反之难写的单测代码是复杂的，可读性差的。 + +**顺便微重构** + +- 如设计不合理可微重构，保证代码的可读性以及健壮性。 + +**提升开发人员自信心** + +- 经过单元测试，能让程序员对自己的代码质量更有信心，对实现方式记忆更深。 + +**启动速度，提升效率** + +- 不用重复启动容器，浪费大量时间在容器启动上，方便逻辑验证。 + +**场景保存（多场景）** + +- 在HSF控制台中只能保存一套参数，而单测可保存多套参数，覆盖各个场景，多条分支，就是一个个测试用例。 + +**CodeReview时作为重点CR的地方** + +**好的单测可作为指导文档，方便使用者使用及阅读** + +- 写起来，相信你会发现更多单测带来的价值。 + + + +## 1.4 基本原则 + +宏观上，单元测试要符合 AIR 原则： + +* A: Automatic（自动化） +* I: Independent（独立性） +* R: Repeatable（可重复） + +微观上，单元测试代码层面要符合 BCDE 原则： + +* B: Border，边界性测试，包括循环边界、特殊取值、特殊时间点、数据顺序等 +* C: Correct，正确的输入，并且得到预期的结果 +* D: Design，与设计文档相符合，来编写单元测试 +* E: Error，单元测试的目的是为了证明程序有错，而不是证明程序无错。 + +>为了发现代码中潜藏的错误，我们需要在编写测试用例时有一些强制的错误输入（如非法数据、异常流程、非业务允许输入等）来得到预期的错误结果。 + +## 1.5 单元测试的常见场景 + +1. 开发前写单元测试，通过测试描述需求，由测试驱动开发。 +2. 在开发过程中及时得到反馈，提前发现问题。 +3. 应用于自动化构建或持续集成流程，对每次代码修改做回归测试。（CI/CD 质量保障） +4. 作为重构的基础，验证重构是否可靠。 + + +## 1.6 容易混淆的概念 + +单元测试和集成测试的界线对于大部分开发来说界线不是那么不清晰。 + +单测说是最小单元？但这个太抽象，具体的可以对应，一个类的方法，最大也可以作为一个类。 + +一般认为，单元测试不应该包含外部依赖，反之就是集成测试了。 + + - 本地其他的service + - dao调用（查数据库） + - 外部缓存 + - rpc调用、消息中间件 + - 微服务调用 + - http-Restful、网络服务 + - 文件系统等等外部基础设施 + +> 如果有这些依赖，还想当做单元测试，就应该用测试替身（Mockito这样的库可以替你生成测试替身）替换掉所有的外部依赖。 + + +除了容易混淆的单测和集成测试，三种主要的软件测试还包括功能测试， 下面简单比较下这三种测试。 + +### 1.6.1 单测 VS 集成测试 VS 功能测试 + + +- **单元测试**：用于测试各个代码组件，并确保代码按照预期的方式工作。 + - 单元测试由**开发人员编写和执行**。 + - 大多数情况下，使用**JUnit 或 TestNG**之类的测试框架。 + - 测试用例**通常是在方法级别**写入并通过自动化执行。 + +- **集成测试**：检查系统是否作为一个整体而工作。 + - 集成测试也由**开发人员完成**，但不是测试单个组件，而是旨在跨组件测试。 + - 系统由许多单独的组件组成，如代码，数据库，Web服务器等。 + - 集成测试能够发现如组件布线，网络访问，数据库问题等问题。 + +- **功能测试**：通过将给定输入的结果与规范进行比较来检查每个功能是否正确实现。 + - 通常，这不是在开发人员级别的。 + - 功能测试由单独的测试团队执行。 + - 测试用例基于规范编写，并且实际结果与预期结果进行比较。 + - 有若干**工具可用于自动化**的**功能**测试，如**Selenium和QTP**。 + +--- + +从**服务端的角度**把这三层稍微改一下： + +- **契约测试**：测试服务与服务之间的契约，接口保证。代价最高，测试速度最慢。 + +- **集成测试**（Integration）：集成当前spring容器、中间件等，对服务内的接口，或者其他依赖于环境的方法的测试。 + + ```java + // 加载spring环境 + @RunWith(SpringBootRunner.class) + @DelegateTo(SpringJUnit4ClassRunner.class) + @SpringBootTest(classes = {Application.class}) + public class ApiServiceTest { + + @Autowired + ApiService apiService; + //do some test + } + ``` + +- **单元测试**（Unit Test）：**纯函数，方法的测试**，不依赖于spring容器，也不依赖于其他的环境。 + +### 1.6.2 单元测试与集成测试的区别 + +在实际工作中，不少同学用集成测试代替了单元测试，或者认为集成测试就是单元测试。这里，总结为了单元测试与集成测试的区别： + +- **测试对象不同**：单元测试对象是实现了具体功能的程序单元，集成测试对象是概要设计规划中的模块及模块间的组合。 + +- **测试方法不同**：单元测试中的主要方法是基于代码的白盒测试，集成测试中主要使用基于功能的黑盒测试。 + +- **测试时间不同**：集成测试要晚于单元测试。 + +- **测试内容不同**：单元测试主要是模块内程序的逻辑、功能、参数传递、变量引用、出错处理及需求和设计中具体要求方面的测试；而集成测试主要验证各个接口、接口之间的数据传递关系，及模块组合后能否达到预期效果。 + + + +# 二、 单元测试的常见痛点 + +## 2.1 **为什么大部分开发不喜欢写单测？** + +1. 产品经理天天催进度，没有有时间写UT。 单测的成本很高，特别是系统化、自动化的单测，而排期一般不会考虑进去，导致没有时间写。 + +2. 自测能测出bug嘛？代码和测试代码都是基于自身思维，就像考试做完第一遍，第二遍检查一样，基本检查不出什么东西 +3. UT维护成本太高,投入产出比太低。 对于单测的作用理解和认识不足。 +4. 集成测试由很多单元组成，直接集成测试不就可以了？ 而集成测试是服务于功能的，现在一键部署这么方便，直接部署后页面点点点，进行功能测试，没问题就万事大吉，有问题，远程debug就好了。 +5. 不会写UT。 或者说些的UT都很简单，体现不出UT的作用，十分鸡肋。 + +总之有无数种理由不想写UT，其实很多“UT”都只是最基本的集成测试，应该称之为“验证”更合适，**验证不等于测试**。 + +>**验证**往往只写主逻辑是否通过，且就一个`Case`，且没有`Assert`，有的是`System.out`。 +>比如RPC的服务端，因为RPC的服务端没有页面可以功能测试，部署到测试环境测试太麻烦，只能写UT了。 + + + + +有个**测试模型图**, 图的意思就是：**越底层**做的**测试效果越好**，越往上则越差。也就是说大部分公司现在做的**功能测试**（即UI测试）其实是效果最差的一种测试方式。 + +![img](https://bbs-img.huaweicloud.com/blogs/img/1614221444691083141.png) + + + + +# 三、怎么写好单测 + +即便我们知道单测的好处，且确定下决心要好好写单测了，但单测并不是那么好些的， + +1. 首先，我们需要知道些单测会遇到的坎、痛点？ +2. 然后，学习一些写单测的技巧，常用的框架？ +3. 最后，比较业内一些最佳实践，结合咱们的实际情况，选择一种进行尝试验证。 + +## 3.1 写单测的痛点和难点 + +下列痛点是日常开发中可能会遇到的， + +1. 测试上下文依赖外部服务（如数据库服务） +2. 测试上下文存在代码依赖（如框架等） +3. 单元测试难以维护和理解（语义不清） +4. 对于多场景不同输入输出的函数，单元测试代码量会很多 +5. 需要连接一些微服务 + ... + + + + +## 3.2 写好单测的技巧 + +### 3.2.1 一些基本的建议 + +- 您可以在编写代码后使用『单元测试』来检查代码的质量，同时也可以使用该功能来改进开发过程。 + + >建议您一边开发一边编写测试，而不是在完成应用的开发之后才编写测试。这样做有助于您设计出可维护、可重复使用的小型代码单元，也方便您迅速而彻底地测试您的代码。 + +1. 使用断言而不是Print语句 + +>许多开发人员习惯于在每行代码之后编写`System.out.println`语句来验证代码是否正确执行。 这种做法常常扩展到单元测试，从而导致**测试代码变得杂乱**。除了混乱，这需要开发人员**手动干预去验证控制台上打印的输出**，以检查测试是否成功运行。更好的方法是**使用自动指示测试结果的断言**。 + +2. 构建具有确定性结果的测试 + +3. 除了正面情景外，还要测试负面情景和边缘情况 + +4. 单测要有覆盖度 + +5. 单测要稳定 + +### 3.2.2 写可测性好的代码 + +>单测难写很多时候是因为——代码结构有问题。 + +1. 保持测试代码的紧凑和可读性 +2. 避免编写重复累赘的断言 +3. 覆盖尽可能多的范围，包括正面情况，以及（甚至更重要的）出错的代码路径。 +4. 不要Mock你不拥有的类型！ + - 这不是一个硬界限，但越过这条线很可能会产生反作用力！ + - TDD是关于设计的，也是关于测试的，两者一样重要，在模拟外部API时，测试不能用于驱动设计，API属于第三方；这个第三方可以，并且实际上也经常会更改API的签名和行为。 + - 另一个问题是第三方库可能很复杂，需要大量的Mock才能正常工作。这导致过度指定的测试和复杂的测试辅助装置，这本身就损害了紧凑和可读的目标。或者由于模拟外部系统过于复杂，从而导致测试代码对生产代码的覆盖不足。 + +5、不要Mock一切，这是一种反模式 + + +### 3.2.3 单测规约 + +《**阿里java 开发规范** —— **单测部分**》： + +>1.【强制】好的单元测试必须遵守AIR原则。 +> +>说明：单元测试在线上运行时，感觉像空气（AIR）一样并不存在，但在测试质量的保障上，却是非常关键的。好的单元测试宏观上来说，具有自动化、独立性、可重复执行的特点。 +> +>- A：Automatic（自动化） +>- I：Independent（独立性） +>- R：Repeatable（可重复） +> +>2.【强制】单元测试应该是全自动执行的，并且非交互式的。测试用例通常是被定期执行的，执行过程必须完全自动化才有意义。输出结果需要人工检查的测试不是一个好的单元测试。单元测试中不准使用System.out来进行人肉验证，必须使用assert来验证。 +> +>3.【强制】保持单元测试的独立性。为了保证单元测试稳定可靠且便于维护，单元测试用例之间决不能互相调用，也不能依赖执行的先后次序。 反例：method2需要依赖method1的执行，将执行结果做为method2的参数输入。 +> +>4.【强制】单元测试是可以重复执行的，不能受到外界环境的影响。 说明：单元测试通常会被放到持续集成中，每次有代码check in时单元测试都会被执行。如果单测对外部环境(网络、服务、中间件等)有依赖，容易导致持续集成机制的不可用。 正例：为了不受外界环境影响，要求设计代码时就把SUT的依赖改成注入，在测试时用spring 这样的DI框架注入一个本地（内存）实现或者Mock实现。 +> +>5.【强制】对于单元测试，要保证测试粒度足够小，有助于精确定位问题。单测粒度至多是类级别，一般是方法级别。 说明：只有测试粒度小才能在出错时尽快定位到出错位置。单测不负责检查跨类或者跨系统的交互逻辑，那是集成测试的领域。 +> +>6.【强制】核心业务、核心应用、核心模块的增量代码确保单元测试通过。 说明：新增代码及时补充单元测试，如果新增代码影响了原有单元测试，请及时修正。 +> +>7.【强制】单元测试代码必须写在如下工程目录：src/test/java，不允许写在业务代码目录下。 说明：源码构建时会跳过此目录，而单元测试框架默认是扫描此目录。 +> +>8.【推荐】单元测试的基本目标：语句覆盖率达到70%；核心模块的语句覆盖率和分支覆盖率都要达到100% 说明：在工程规约>应用分层中提到的DAO层，Manager层，可重用度高的Service，都应该进行单元测试。 +> +>9.【推荐】编写单元测试代码遵守BCDE原则，以保证被测试模块的交付质量。 +> +>- B：Border，边界值测试，包括循环边界、特殊取值、特殊时间点、数据顺序等。 +>- C：Correct，正确的输入，并得到预期的结果。 +>- D：Design，与设计文档相结合，来编写单元测试。 +>- E：Error，强制错误信息输入（如：非法数据、异常流程、非业务允许输入等），并得到预期的结果。 +> +>10.【推荐】对于数据库相关的查询，更新，删除等操作，不能假设数据库里的数据是存在的，或者直接操作数据库把数据插入进去，请使用程序插入或者导入数据的方式来准备数据。 反例：删除某一行数据的单元测试，在数据库中，先直接手动增加一行作为删除目标，但是这一行新增数据并不符合业务插入规则，导致测试结果异常。 +> +>11.【推荐】和数据库相关的单元测试，可以设定自动回滚机制，不给数据库造成脏数据。或者对单元测试产生的数据有明确的前后缀标识。 正例：在Aone内部单元测试中，使用AONE_UNIT_TEST_的前缀标识数据。 +> +>12.【推荐】对于不可测的代码在适当时机做必要的重构，使代码变得可测，避免为了达到测试要求而书写不规范测试代码。 +> +>13.【推荐】在设计评审阶段，开发人员需要和测试人员一起确定单元测试范围，单元测试最好覆盖所有测试用例（UC）。 +> +>14.【推荐】单元测试作为一种质量保障手段，不建议项目发布后补充单元测试用例，在项目提测前完成单元测试。 +> +>15.【参考】为了更方便地进行单元测试，业务代码应避免以下情况： +> +>- 构造方法中做的事情过多。 +>- 存在过多的全局变量和静态方法。 +>- 存在过多的外部依赖。 +>- 存在过多的条件语句。 +> +>说明：多层条件语句建议使用卫语句、策略模式、状态模式等方式重构。 +> +>16.【参考】不要对单元测试存在如下误解： +> +>- 那是测试同学干的事情。本文是开发规约，凡是本文出现的内容都是与开发同学强相关的。 +>- 单元测试代码是多余的。软件系统的整体功能是否正常，与各单元部件的测试正常与否是强相关的。 +>- 单元测试代码不需要维护。一年半载后，那么单元测试几乎处于废弃状态。 +>- 单元测试与线上故障没有辩证关系。好的单元测试能够最大限度地规避线上故障。 + + + +## 3.3 单测框架(Java) + +### 1、JUnit + +JUnit是一个开放源代码的Java测试框架，用于编写和运行可重复的测试，也是我们 Javer 最常见的单测框架 + +现在主流的 Junit 有使用 4 和 5（需要JDK8及以上） + +| 特征 | JUNIT 4 | JUNIT 5 | +| :------------------------------- | :------------- | :------------- | +| 声明一种测试方法 | `@Test` | `@Test` | +| 在当前类中的所有测试方法之前执行 | `@BeforeClass` | `@BeforeAll` | +| 在当前类中的所有测试方法之后执行 | `@AfterClass` | `@AfterAll` | +| 在每个测试方法之前执行 | `@Before` | `@BeforeEach` | +| 每种测试方法后执行 | `@After` | `@AfterEach` | +| 禁用测试方法/类 | `@Ignore` | `@Disabled` | +| 测试工厂进行动态测试 | NA | `@TestFactory` | +| 嵌套测试 | NA | `@Nested` | +| 标记和过滤 | `@Category` | `@Tag` | +| 注册自定义扩展 | NA | `@ExtendWith` | + +- JUnit 5利用了Java 8或更高版本的特性，例如lambda函数，使测试更强大，更容易维护。 +- JUnit 5为描述、组织和执行测试添加了一些非常有用的新功能。例如，测试得到了更好的显示名称，并且可以分层组织。 +- JUnit 5被组织成多个库，所以只将你需要的功能导入到你的项目中。通过Maven和Gradle等构建系统，包含合适的库很容易。 +- JUnit 5可以同时使用多个扩展，这是JUnit 4无法做到的（一次只能使用一个runner）。这意味着你可以轻松地将Spring扩展与其他扩展（如你自己的自定义扩展）结合起来。 + +### 2、TestNG + +TestNG类似于JUnit，但是它配置了特殊的注释和高级功能(JUnit不支持)。 + +TestNG中的NG表示“下一代”。TestNG可以覆盖几乎所有类型的软件测试，包括**端到端、单元、集成和功能**测试。TestNG和JUnit都是基于java的框架，允许你编写测试和检查最终结果。如果测试成功，你将看到绿色条，否则将看到红色条。 + +**TestNG的优点和缺点** + +**1. 优点：** + +- 该框架使您能够在多个代码片段上运行并行测试。 +- 在测试用例执行期间，您可以生成HTML报告。 +- 可以根据优先级对测试用例进行分组和排列 +- 可以参数化数据并使用注释来轻松设置优先级。 + +**2. 缺点**：取决于您的要求，此外，设置TestNG需要一点时间。 + + + +[《关于testNG和JUnit的对比》](https://developer.aliyun.com/article/572271) + + + +### 3、Spock + +Spock 由 Gradleware首席工程师 于 2008 年创建。虽然灵感来自于 JUnit，Spock 的特性不仅仅是 JUnit 的扩展： + +- 测试代码使用 Groovy 语言编写，而被测代码可以由 Java 编写。 +- **内置 mock 框架以减少引入第三方框架**。 +- 可支持自定义测试件名称。 +- 为创建测试代码预定义了行为驱动块（given:、when:、then:、expect: 等）。 +- 使用数据表格以减少数据结构的使用需求。 + + + +### 4、JBehave + +Behave是一种令人难以置信的、支持BDD(行为驱动开发)的最佳Java测试框架之一。BDD是TDD(测试驱动开发)和ATDD([验收测试](https://link.zhihu.com/?target=https%3A//www.zmtests.com/%3Fpk_campaign%3Dzhihu-w)驱动开发)的演变。 + +Java提供了若干用于单元测试的框架。TestNG和JUnit是最流行的测试框架。JUnit和TestNG的一些重要功能： + +- 易于设置和运行。 +- 支持注释。 +- 允许忽略或分组并一起执行某些测试。 +- 支持参数化测试，即通过在运行时指定不同的值来运行单元测试。 +- 通过与构建工具，如Ant，Maven和Gradle集成来支持自动化的测试执行。 + +缺点： + +- 需要具备基本的 Groovy 语言知识 + + + +### 5、Spring Test + +Spring Test 是 Spring MVC 自带了一个测试框架 + + + +### 6、Selenide + +Selenide是一个流行的开源Java测试框架，它是由Selenium WebDriver提供支持。它是为Java应用程序编写精确的、交流的、稳定的UI测试用例的工具。它扩展了WebDriver和JUnit功能。 + +WebDriver是一个非常受欢迎的用户界面测试工具，但是它缺乏处理超时的特性。例如，对Ajax等web技术的测试。Selenide框架管理所有这些问题在一个简单的方式。此外，它更容易安装和学习。你只需要把注意力集中在逻辑上，Selenide就会完成剩下的工作。 + +**特性** + +（1）开箱即用，并设置使用框架； + +（2）编写更少的自动化代码； + +（3）节省大量的时间； + +（4）配置理想的CI工具，如Jenkins。 + + +### 7、JWebUnit + +JWebUnit是一个基于java的测试框架，是用于集成、回归和功能测试的首选JUnit扩展之一。它用一个简单的测试界面包装了当前的活动框架，如HTMLUnit和Selenium。因此，你可以立即测试web应用程序的准确性。 + +JWebUnit可用于执行屏幕导航测试。该框架还提供了一个高级Java应用程序编程接口，用于使用一组断言导航web应用程序，以检查应用程序的准确性。它计算通过链接的导航、表单入口和提交、表内容的调整以及其他常见的业务web应用程序特征。 + + + +## 3.4 Mock 框架 + + **什么是Mock?** + +>在面向对象的程序设计中，模拟对象（英语：mock object）是以可控的方式模拟真实对象行为的假对象。在编程过程中，通常通过模拟一些输入数据，来验证程序是否达到预期结果。 + +**为什么使用Mock对象？** + +>使用模拟对象，可以模拟复杂的、真实的对象行为。如果在单元测试中无法使用真实对象，可采用模拟对象进行替代。 + + Mock测试就是在测试过程中，对那些当前测试不关心的，不容易构建的对象，用一个虚拟对象来代替测试的情形。 + +说白了：就是解耦(虚拟化)要测试的目标方法中调用的其它方法，例如：Service的方法调用Mapper类的方法，这时候就要把Mapper类Mock掉（产生一个虚拟对象），这样我们可以自由的控制这个Mapper类中的方法，让它们返回想要的结果、抛出指定异常、验证方法的调用次数等等。 + +- **Mock可以用来解除外部服务依赖，从而保证了测试用例的独立性** +- **Mock可以减少全链路测试数据准备，从而提高了编写测试用例的速度** +- **Mock可以模拟一些非正常的流程，从而保证了测试用例的代码覆盖率** +- **Mock可以不用加载项目环境配置，从而保证了测试用例的执行速度** + + +单元测试不应该依赖数据，依赖外部服务或组件等，会对其他数据产生影响的情况。启动Spring容器，一般比较慢，可能会启动消息监听消费消息，定时任务的执行等，对数据产生影响。 + + +### EasyMock + +EasyMock 是早期比较流行的MocK测试框架。它提供对接口的模拟，能够通过录制、回放、检查三步来完成大体的测试过程，可以验证方法的调用种类、次数、顺序，可以令 Mock 对象返回指定的值或抛出指定异常。通过 EasyMock，我们可以方便的构造 Mock 对象从而使单元测试顺利进行。 + + + +### Mockito + +EasyMock之后流行的mock工具。相对EasyMock学习成本低，而且具有非常简洁的API，测试代码的可读性很高。 + +Spring-boot-starter-test 内置框架 + +- MockMvc是由spring-test包提供，实现了对Http请求的模拟，能够直接使用网络的形式，转换到Controller的调用，使得测试速度快、不依赖网络环境。同时提供了一套验证的工具，结果的验证十分方便。 +- 接口MockMvcBuilder，提供一个唯一的build方法，用来构造MockMvc。主要有两个实现：StandaloneMockMvcBuilder和DefaultMockMvcBuilder。 + + + +### PowerMock + +这个工具是在EasyMock和Mockito上扩展出来的，目的是为了解决EasyMock和Mockito不能解决的问题，比如对static, final, private方法均不能mock。其实测试架构设计良好的代码，一般并不需要这些功能，但如果是在已有项目上增加单元测试，老代码有问题且不能改时，就不得不使用这些功能了。 + + + +### Jmockit + +JMockit 是一个轻量级的mock框架是用以帮助开发人员编写测试程序的一组工具和API，该项目完全基于 Java 5 SE 的 java.lang.instrument 包开发，内部使用 ASM 库来修改Java的Bytecode。 + + + +### TestableMock + +阿里开源的一款 mock 工具，让Mock的定义和置换干净利落 + +https://github.com/alibaba/testable-mock + + + + +## 五、 测试的实践&初步方案 + +PS：都有一点学习成本 + +### 本地集成测试 + +测试主要包含三块内容： + +1. 数据准备 +2. 执行逻辑 +3. 输出验证。 + +**第一步：数据准备** +在本地集成测试里，数据来源基本上来自于dao，dao来自于sql。也就是在执行一个case之前，执行一些sql脚本，数据库则使用h2这类memory database， 切记不要依赖公司测试环境的db。可以在case执行之前准备我们所需要的各种数据， 另外在执行完case之后，执行clean.sql脚本来清理脏数据。这里也说明一个case的执行环境是完全独立的，case之间互不干扰，这很重要。 + +**第二步：执行逻辑** + +最简单，就是调用一下我们测试的方法即可 + +**第三步：验证** + + + +### 数据库访问测试 + +数据库测试多用在DAO中，DAO对数据库的操作依赖于mybatis的sql mapper 文件，这些sql mapper多是手工写的，在单测中验证所有sql mapper的正确性非常重要，在DAO层有足够的覆盖度和强度后，Service层的单测才能仅仅关注自身的业务逻辑 + +为了验证sql mapper，我们需要一个能实际运行的数据库。为了提高速度和减少依赖，可以使用内存数据库。内存数据库和目标数据库(MySQL,TDDL)在具体函数上有微小差别，不过只要使用标准的SQL 92，两者都是兼容的。 + +使用 [H2](http://www.h2database.com/) 作为单测数据库 + + + +1. 不要使用共享数据库，要使用本地数据库，最好是内存数据库 + + - 使用共享数据库的根本问题是：别的开发者也会操作这个数据库，从而导致你的测试结果是不确定的。 + + >实际情况共同开发者不多时，可以用共享数据库。 + + - 通过网络访问远程数据库的速度肯定不如访问本机数据库快。如果远程数据库由于维护需要而停机了，或者由于各种原因网络中断了，都会导致测试无法进行。 + + - 如果有可能，请安装和使用内存数据库。如果你的产品是针对某一种特定类型的数据库的（例如MySQL），就寻找这种数据库的内存版本。如果你的产品是数据库中立的，最好是选择不同种类的内存数据库（例如H2）。 内存数据库一方面是更快，另一方面是不需要在测试后清理数据。 + + - 如果换用本地数据库，是不是需要修改数据库配置？答案是：通常不需要。一般而言，你会把数据库访问的url等信息抽取出来，放在类似jdbc.properties这样的文本文件中，由Java代码读取这个文件来获取数据库连接信息。你可以分别为产品和测试准备一个不同的jdbc.properties。产品连接到正式的数据库，测试连接到本地/内存数据库。 + + >这里我们部门都有开发、测试、线上数据库，所以问题不大。 + +2. 增删改查单元测试顺序问题 + + - 好的测试必须是独立的，不依赖于其他测试的执行 + - 我们应该在query测试中通过其他手段插入样例数据到数据库中，然后才执行query。可以使用DBUnit或者类似的工具，将XML或JSON格式的样例数据插入数据库。Spring也提供了这样的手段。注意：要保证这些工具的数据处理逻辑是替换而不是添加。就是不管原来的数据表中有没有内容，都整体清空，用数据文件中的数据代替。这样才能保证测试条件是确定的。 + - 无论如何，要将准备测试数据这样的工作作为测试的一部分自动执行，不应该进行手工插入。测试应该是全自动的，既不需要手工准备测试数据，也不需要手工验证测试结果。 + + + +### 其它外部依赖 + +对于除数据库以外的依赖，包括各种中间件以及外部的HSF/HTTP服务，在单测中全部采用Mock进行解耦。 + +junit + mockito + +与spring结合性感觉更强，可以充分兼容spring的各种注解，能较完美接近项目启动后实际运行情况。可以用来做一定的业务自动化回归测试，但可能超出了ut的范围 + + + +### Mock模板化 + + +- 一个类里面测试太多怎么办？ +- 不知道别人mock了哪些数据怎么办？ +- 测试结构太复杂？ +- 测试莫名奇妙起不来？ + +FSC（Fixture-Scenario-Case）是一种组织测试代码的方法，目标是尽量将一些MOCK信息在不同的测试中共享。其结构如下： + +![图片](https://mmbiz.qpic.cn/mmbiz_png/Z6bicxIx5naJkibdPicqyKZw3IFlzLsRSickr20z2q3xU5FuicZJotzc422ffkPOPkGeZnQvjeKRWZykXXNz24ricgkQ/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +- 通过组合Fixture（固定设施），来构造一个Scenario（场景）。 + +- 通过组合Scenario（场景）+ Fixture（固定设施），构造一个case（用例）。 + + + +![图片](https://mmbiz.qpic.cn/mmbiz_png/Z6bicxIx5naJkibdPicqyKZw3IFlzLsRSickTibss6iaBqKojTpJicFBuOia22Jl45xB33SlSbgOl6MQicMWO5sxtjMClrw/640?wx_fmt=png&tp=webp&wxfrom=5&wx_lazy=1&wx_co=1) + +- Case：当用户正常登录后，获取当前登录信息时，应该返回正确的用户信息。这是一个简单的用户登录的case，这个case里面总共有两个动作、场景，一个是用户正常登录，一个是获取用户信息，演化为两个scenario。 + +- Scenario：用户正常登录，肯定需要登录参数，如：手机号、验证码等，另外隐含着数据库中应该有一个对应的用户，如果登录时需要与第三方系统进行交互，还需要对第三方系统进行mock或者stub。获取用户信息时，肯定需要上一阶段颁发的凭证信息，另外该凭证可能是存储于一些缓存系统的，所以还需要对中间件进行mock或者stub。 + +- Fixture + - 利用Builder模式构造请求参数。 + - 利用DataFile来存储构造用户的信息，例如DB transaction进行数据的存储和隔离。 + - 利用Mockito进行三方系统、中间件的Mock。 + + + +当这样组织测试时，如果另外一个Case中需要用户登录，则可以直接复用用户登录的Scenario。也可以通过复用Fixture来减少数据的Mock。下面我们来详细解释看一下每一层如何实现，show the code。 + + + +## 六、单测示例&尝试 + +```xml + + + org.mockito + mockito-core + 3.3.3 + test + + + + + org.junit.jupiter + junit-jupiter + 5.7.1 + test + +``` + + +```java +import org.mockito.runners.MockitoJUnitRunner; + + +@RunWith(MockitoJUnitRunner.class) +public class IndustryFieldTest { + + //自动注入Mock类（fieldDao）到被测试类（IndustryFieldServiceImpl），作为一个属性 + @InjectMocks + private IndustryFieldServiceImpl industryFieldService; + + @Mock + private IndustryFieldDao fieldDao; + + @Test + public void testListIndustryFields() { + + List fields = new ArrayList<>(); + + IndustryField locField = new IndustryField(); + locField.setId(1L); + locField.setFieldKey("loc"); + locField.setRequired(1); + locField.setFieldKeyCn("商品移动端url"); + + IndustryField nameField = new IndustryField(); + nameField.setId(2L); + nameField.setFieldKey("name"); + nameField.setRequired(1); + nameField.setFieldKeyCn("商品名称"); + + fields.add(locField); + fields.add(nameField); + + Mockito.when(fieldDao.listIndustryFields(0)).thenReturn(fields); + + //测试service 方法 + List fieldDtos = industryFieldService.listIndustryFields(0); + + Assert.assertNotNull(fieldDtos); + Assert.assertEquals(fieldDtos.get(0).getFieldKey(),"loc1"); + } + +} +``` + + + + + +### 工具篇 + +**Squaretest** + +自动生成单测框架 + +**Fast-tester** + +阿里提供的 fast_tester，只需要启动应用一次（tip: 添加注解及测试方法需要重新启动应用），支持测试代码热更新，后续可随意编写测试方法 + +**JUnitGenerator** + +idea 插件，自动生成单测方法体 + + + +## 参考 + +- 《SpringBoot - 單元測試工具 Mockito》https://kucw.github.io/blog/2020/2/spring-unit-test-mockito/ + +- https://cloud.tencent.com/developer/article/1338791 +- 《写有价值的单元测试》https://developer.aliyun.com/article/54478 +- 《Mockito.mock() vs @Mock vs @MockBean》https://www.baeldung.com/java-spring-mockito-mock-mockbean +- 《Mock服务插件在接口测试中的设计与应用》 https://tech.youzan.com/mock/ \ No newline at end of file diff --git a/docs/rpc/Hello-RPC.md b/docs/rpc/Hello-RPC.md deleted file mode 100644 index 324705b071..0000000000 --- a/docs/rpc/Hello-RPC.md +++ /dev/null @@ -1,230 +0,0 @@ - - -## RPC是个啥玩意 - -**远程过程调用**（英语：Remote Procedure Call，缩写为 RPC）是一个计算机通信[协议](https://zh.wikipedia.org/wiki/%E7%B6%B2%E7%B5%A1%E5%82%B3%E8%BC%B8%E5%8D%94%E8%AD%B0)。该协议允许运行于一台计算机的[程序](https://zh.wikipedia.org/wiki/%E7%A8%8B%E5%BA%8F)调用另一台计算机的[子程序](https://zh.wikipedia.org/wiki/%E5%AD%90%E7%A8%8B%E5%BA%8F)，而程序员无需额外地为这个交互作用编程,区别于本地过程调用。 - - - -**RPC要解决的两个问题：** - -1. **解决分布式系统中，服务之间的调用问题。** -2. **远程调用时，要能够像本地调用一样方便，让调用者感知不到远程调用的逻辑。** - - - - - -更通俗的解释：[如何给老婆解释什么是RPC]( ) - - - - - - - - - - - -## 如何调用他人的远程服务？ - -　　由于各个服务部署在不同机器，服务间的调用涉及到网络通信过程，如果服务消费方每调用一个服务都要写一坨网络通信相关的代码，不仅使用复杂而且极易出错。 - -如果有一种方式能让我们像调用本地服务一样调用远程服务，而让调用者对网络通信这些细节透明，那么将大大提高生产力。这种方式其实就是RPC（Remote Procedure Call Protocol），在各大互联网公司中被广泛使用，如阿里巴巴的hsf、dubbo（开源）、Facebook的thrift（开源）、Google grpc（开源）、Twitter的finagle等。 - -　　我们首先看下一个RPC调用的具体流程： - -![img](https://user-gold-cdn.xitu.io/2016/11/29/f99457a58f231dbeb3ca0b2f29c7f113?imageView2/0/w/1280/h/960/format/webp/ignore-error/1) - -1）服务消费方（client）调用以本地调用方式调用服务； - -2）client stub接收到调用后负责将方法、参数等组装成能够进行网络传输的消息体； - -3）client stub找到服务地址，并将消息发送到服务端； - -4）server stub收到消息后进行解码； - -5）server stub根据解码结果调用本地的服务； - -6）本地服务执行并将结果返回给server stub； - -7）server stub将返回结果打包成消息并发送至消费方； - -8）client stub接收到消息，并进行解码； - -9）服务消费方得到最终结果。 - -**RPC的目标就是要2~8这些步骤都封装起来，让用户对这些细节透明。** - - - -RPC仅仅是一种技术，为什么会与微服务框架搞混呢？ - -因为随着RPC的大量使用，必然伴随着服务的发现、服务的治理、服务的监控这些，这就组成了微服务框架。 - -RPC仅仅是微服务中的一部分。 - - - - - -维度 RPC REST -耦合性 强耦合 松散耦合 -消息协议 二进制thrift、protobuf、avro 文本型XML、JSON -通讯协议 TCP为主，也可以是HTTP HTTP/HTTP2 -性能 高 一般低于RPC -接口契约IDL Thrift、protobuf idl Swagger -客户端 强类型客户端、一般自动生成，可支持多语言客户端 一般http client可访问，也可支持多语言 -案例 dubbo、motan、tars、grpc、thrift spring boot/mvc、Jax-rs -开发者友好 客户端比较方便，但是二进制消息不可读 文本消息开发者可读、浏览器可直接访问查看结果 -对外开放 需要转换成REST/文本协议 直接对外开放 - - -### RPC VS HTTP - -RPC=Remote Produce Call 是一种技术的概念名词. HTTP是一种协议,**RPC可以通过HTTP来实现**,也可以通过Socket自己实现一套协议来实现. - -HTTP严格来说跟RPC不是一个层级的概念。HTTP本身也可以作为RPC的传输层协议。 - -首先 http 和 rpc 并不是一个并行概念。 - -rpc是远端过程调用，其调用协议通常包含传输协议和序列化协议。 - -传输协议包含: 如著名的 [gRPC]([grpc / grpc.io](https://link.zhihu.com/?target=http%3A//www.grpc.io/)) 使用的 http2 协议，也有如dubbo一类的自定义报文的tcp协议。 - -序列化协议包含: 如基于文本编码的 xml json，也有二进制编码的 protobuf hessian等。 - - - -# RPC vs Restful - -其实这两者并不是一个维度的概念，总得来说RPC涉及的维度更广。 - -如果硬要比较，那么可以从RPC风格的url和Restful风格的url上进行比较。 - -比如你提供一个查询订单的接口，用RPC风格，你可能会这样写： - -``` -/queryOrder?orderId=123 -``` - -用Restful风格呢？ - -``` -Get -/order?orderId=123 -``` - -**RPC是面向过程，Restful是面向资源**，并且使用了Http动词。从这个维度上看，Restful风格的url在表述的精简性、可读性上都要更好。 - -REST是一种架构风格，指的是一组架构约束条件和原则。满足这些约束条件和原则的应用程序或设计就是 RESTful。REST规范把所有内容都视为资源，网络上一切皆资源。 - -# RPC vs RMI - -严格来说这两者也不是一个维度的。 - -RMI是Java提供的一种访问远程对象的协议，是已经实现好了的，可以直接用了。 - -而RPC呢？人家只是一种编程模型，并没有规定你具体要怎样实现，**你甚至都可以在你的RPC框架里面使用RMI来实现数据的传输**，比如Dubbo：[Dubbo - rmi协议](https://link.jianshu.com?t=http%3A%2F%2Fdubbo.apache.org%2Fbooks%2Fdubbo-user-book%2Freferences%2Fprotocol%2Frmi.html) - - - - - -## **一、RPC架构简介** - -RPC的全称是Remote Procedure Call，它是一种进程间通信方式。允许像调用本地服务一样调用远程服务，它的具体实现方式可以不同，例如Spring的 HTTP Invoker，Facebook的 Thrift二进制私有协议通信。 - -RPC概念术语在上世纪80年代由 Bruce Jay Nelson提出，在他的论文中对RPC进行了如下总结。 - -1)简单:RPC概念的语义十分清晰和简单，这样建立分布式计算就更容易。 - -2)高效:过程调用看起来十分简单而且高效。 - -3)通用:在单机计算中过程往往是不同算法和APl，跨进程调用最重要的是通用的通信机制。 - -2006年之后，随着移动互联网的发展，各种智能终端的普及，远程分布式调用已经成为主流，RPC框架也如雨后春笋般诞生，开源和自研的RPC框架的普及标志着传统垂直应用架构时代的终结。 - - - -## **二、RPC框架原理** - -RPC框架的目标就是让远程过程(服务)调用更加简单、透明，RPC框架负责屏蔽底层的传输方式(TCP或者UDP)、序列化方式( XML/JSON/二进制)和通信细节。框架使用者只需要了解谁在什么位置提供了什么样的远程服务接口即可，开发者不需要关心底层通信细节和调用过程。 - - - -RPC框架的调用原理图如下： - - - - ![img](http://jiangew.me/assets/images/post/20181013/grpc-01-01.png) - - - - - - - -## **三：RPC框架核心技术点** - -RPC框架实现的几个核心技术点总结如下： - - - -1)远程服务提供者需要以某种形式提供服务调用相关的信息，包括但不限于服务接口定义、数据结构，或者中间态的服务定义文件，例如 Thrift的IDL文件， WS-RPC的WSDL文件定义，甚至也可以是服务端的接口说明文档;服务调用者需要通过一定的途径获取远程服务调用相关信息，例如服务端接口定义Jar包导入，获取服务端1DL文件等。 - -2)远程代理对象:服务调用者调用的服务实际是远程服务的本地代理，对于Java语言，它的实现就是JDK的动态代理，通过动态代理的拦截机制，将本地调用封装成远程服务调用. - -3)通信:RPC框架与具体的协议无关，例如Spring的远程调用支持 HTTP Invoke、RMI Invoke， MessagePack使用的是私有的二进制压缩协议。 - -4)序列化:远程通信，需要将对象转换成二进制码流进行网络传输，不同的序列化框架，支持的数据类型、数据包大小、异常类型及性能等都不同。不同的RPC框架应用场景不同，因此技术选择也会存在很大差异。一些做得比较好的RPC框架可以支持多种序列化方式，有的甚至支持用户自定义序列化框架( Hadoop Avro)。 - - - - - -## **四、业界主流的RPC框架** - -业界主流的RPC框架很多，比较出名的RPC主要有以下4种： - -1）、由Facebook开发的原创服务调用框架Apache Thrift; - -2）、Hadoop的子项目Avro-RPC; - -3）、caucho提供的基于binary-RPC实现的远程通信框架Hessian; - -4）、Google开源的基于HTTP/2和ProtoBuf的通用RPC框架gRPC - - - -| 功能 | Hessian | Montan | rpcx | gRPC | Thrift | Dubbo | Dubbox | Tars | Spring Cloud | | -| ---------------- | ------- | ---------------------------- | ------ | ---------------- | ------------- | ------- | -------- | ---- | ------------ | ---- | -| 开发语言 | 跨语言 | Java | Go | 跨语言 | 跨语言 | Java | Java | | Java | | -| 分布式(服务治理) | × | √ | √ | × | × | √ | √ | | √ | | -| 多序列化框架支持 | hessian | √(支持Hessian2、Json,可扩展) | √ | × 只支持protobuf | ×(thrift格式) | √ | √ | | √ | | -| 多种注册中心 | × | √ | √ | × | × | √ | √ | | √ | | -| 管理中心 | × | √ | √ | × | × | √ | √ | | √ | | -| 跨编程语言 | √ | × | × | √ | √ | × | × | | × | | -| 支持REST | × | × | × | × | × | × | √ | | √ | | -| 开源机构 | Caucho | Weibo | Apache | Google | Apache | Alibaba | Dangdang | | Apache | | -| | | | | | | | | | | | -| | | | | | | | | | | | -| | | | | | | | | | | | - - - -https://blog.csdn.net/u013452337/article/details/86593291 - - - - - - - - - - - - - diff --git a/docs/sidebar.md b/docs/sidebar.md index e488fd8184..4c27cd8a8c 100644 --- a/docs/sidebar.md +++ b/docs/sidebar.md @@ -1,11 +1,11 @@ - **Java基础** - [![](https://icongr.am/simple/oracle.svg?size=25&color=231c82&colored=false)JVM](java/JVM/readJVM.md) - [![img](https://icongr.am/fontawesome/expeditedssl.svg?size=25&color=f23131)JUC](java/JUC/readJUC.md) -- [![](https://icongr.am/devicon/java-original.svg?size=25&color=f23131)Java 8](java/Java8.md) +- [![](https://icongr.am/devicon/java-original.svg?size=25&color=f23131)Java 8](java/Java-8.md) - [![img](https://icongr.am/entypo/address.svg?size=25&color=074ca6)设计模式](design-pattern/readDisignPattern.md) - **数据存储和缓存** - [![MySQL](https://icongr.am/devicon/mysql-original.svg?&size=25)MySQL](data-store/MySQL/readMySQL.md) -- [![Redis](https://icongr.am/devicon/redis-original.svg?size=25)Redis](data-store/Redis/2.readRedis.md) +- [![Redis](https://icongr.am/devicon/redis-original.svg?size=25)Redis](data-store/Redis/readRedis.md) - [![mongoDB](https://icongr.am/devicon/mongodb-original.svg?&size=25)mongoDB]( https://redis.io/ ) - [![ **Elasticsearch** ](https://icongr.am/simple/elasticsearch.svg?&size=20) Elasticsearch]( https://redis.io/ ) - [![S3](https://icongr.am/devicon/amazonwebservices-original.svg?&size=25)S3]( https://aws.amazon.com/cn/s3/ ) @@ -18,7 +18,10 @@ - [![](https://icongr.am/jam/leaf.svg?size=25&color=00FF00)Spring面试](interview/Spring-FAQ.md) - [![](https://icongr.am/simple/bower.svg?size=25)MyBatis面试](interview/MyBatis-FAQ.md) - [![img](https://icongr.am/entypo/network.svg?size=25&color=6495ED)计算机网络](interview/Network-FAQ.md) -- **单体架构** +- [![](https://icongr.am/simple/apachekafka.svg?size=25&color=121417&colored=false)Kafka 面试](interview/Kafka-FAQ.md) +- [![](https://icongr.am/entypo/user.svg?size=25&color=864646)ZooKeeper 面试](interview/ZooKeeper-FAQ.md) +- **开发架构** +- [![](https://icongr.am/simple/leaflet.svg?size=25&color=11b041&colored=false)Spring](framework/Spring/readSpring.md) - **RPC** - [Hello Protocol Buffers](rpc/Hello-Protocol-Buffers.md) - **面向服务架构** @@ -30,7 +33,10 @@ - [![](https://icongr.am/feather/cloud.svg?size=25&color=36b305)Spring Cloud](framework/SpringCloud/readSpringCloud.md) - [![](https://icongr.am/clarity/alarm-clock.svg?size=25&color=2d2b50)定时任务@Scheduled](framework/SpringBoot/@Scheduled.md) - **大数据** -- [Hello 大数据](big-data/Hello-BigData.md) +- [![](https://icongr.am/fontawesome/ellipsis-h.svg?size=25&color=currentColor)Hello 大数据](big-data/Hello-BigData.md) +- [![](https://icongr.am/devicon/apache-original.svg?size=25&color=currentColor)HDFS](big-data/HDFS.md) +- [![](https://icongr.am/devicon/apache-original.svg?size=25&color=currentColor)Map-Reduce](big-data/Hadoop-MapReduce.md) +- [![](https://icongr.am/simple/hive.svg?size=25&color=currentColor&colored=false)Hive](big-data/Hive.md) - [![](https://icongr.am/simple/apachekafka.svg?size=25&color=121417&colored=false)Kafka](message-queue/Kafka/readKafka.md) - **性能优化** - [![](https://icongr.am/octicons/cpu.svg?size=25&color=780ebe)CPU 飙升问题](optimization/CPU飙升.md) @@ -38,7 +44,13 @@ - \> web调优 - \> DB调优 - **数据结构与算法** -- [![](https://icongr.am/entypo/tree.svg?size=25&color=44c016)树](data-structure/tree.md) +- [![](https://icongr.am/octicons/home.svg?size=25&color=currentColor)数据结构](data-structure/hello-dataStructure.md) +- [![](https://icongr.am/entypo/dots-two-vertical.svg?size=25&color=e24040)数组](data-structure/Array.md) +- [![](https://icongr.am/clarity/ellipsis-horizontal.svg?size=25&color=47579a)链表](data-structure/Linked-List.md) +- [![](https://icongr.am/octicons/arrow-left.svg?size=25&color=currentColor)栈](data-structure/Stack.md) +- [![](https://icongr.am/octicons/arrow-right.svg?size=25&color=currentColor)队列](data-structure/Queue.md) +- [![](https://icongr.am/entypo/tree.svg?size=25&color=44c016)树](data-structure/Tree.md) +- [![](https://icongr.am/octicons/skip.svg?size=25&color=currentColor)跳表](data-structure/Skip-List.md) - **工程化与工具** - [![Maven](https://icongr.am/simple/apachemaven.svg?size=25&color=c93ddb&colored=false)Maven](tools/Maven.md) - [![Git](https://icongr.am/devicon/git-original.svg?&size=16)Git](tools/Git-Specification.md) @@ -48,5 +60,4 @@ - [缕清各种Java Logging](logging/Java-Logging.md) - [hello logback](logging/logback简单使用.md) - **Links** -- [![Github](https://icongram.jgog.in/simple/github.svg?color=808080&size=16)Github](https://github.com/jhildenbiddle/docsify-tabs) -- [![Blog](https://icongr.am/simple/aboutme.svg?colored&size=16)My Blog](https://www.lazyegg.net) \ No newline at end of file +- [![Github](https://icongram.jgog.in/simple/github.svg?color=808080&size=16)Github](https://github.com/jhildenbiddle/docsify-tabs) \ No newline at end of file diff --git a/docs/soa/zookeeper/Zookeeper-FAQ.md b/docs/soa/zookeeper/Zookeeper-FAQ.md deleted file mode 100644 index 869b37ba57..0000000000 --- a/docs/soa/zookeeper/Zookeeper-FAQ.md +++ /dev/null @@ -1,252 +0,0 @@ -## 谈下你对 Zookeeper 的认识？ - -ZooKeeper 是一个分布式的，开放源码的分布式应用程序协调服务。它是一个为分布式应用提供一致性服务的软件，提供的功能包括：配置维护、域名服务、分布式同步、组服务等。 - -ZooKeeper 的目标就是封装好复杂易出错的关键服务，将简单易用的接口和性能高效、功能稳定的系统提供给用户。 - ------- - - - -## Zookeeper 都有哪些功能？ - -1. 集群管理：监控节点存活状态、运行请求等； - -2. 主节点选举：主节点挂掉了之后可以从备用的节点开始新一轮选主，主节点选举说的就是这个选举的过程，使用 Zookeeper 可以协助完成这个过程； - -3. 分布式锁：Zookeeper 提供两种锁：独占锁、共享锁。独占锁即一次只能有一个线程使用资源，共享锁是读锁共享，读写互斥，即可以有多线线程同时读同一个资源，如果要使用写锁也只能有一个线程使用。Zookeeper 可以对分布式锁进行控制。 - -4. 命名服务：在分布式系统中，通过使用命名服务，客户端应用能够根据指定名字来获取资源或服务的地址，提供者等信息。 -5. 统一配置管理：分布式环境下，配置文件管理和同步是一个常见问题，一个集群中，所有节点的配置信息是一致的，比如 Hadoop 集群、集群中的数据库配置信息等全局配置 - ------- - - - -## zookeeper负载均衡和nginx负载均衡区别 - -Nginx是著名的反向代理服务器，zk是分布式协调服务框架，都可以做负载均衡 - -zk的负载均衡是可以调控，nginx只是能调权重，其他需要可控的都需要自己写插件； - -但是nginx的吞吐量比zk大很多，应该说按业务选择用哪种方式 - ------- - - - -## 一致性协议2PC、3PC？ - -### 2PC - -**阶段一：提交事务请求（”投票阶段“）** - -当要执行一个分布式事务的时候，事务发起者首先向协调者发起事务请求，然后协调者会给所有参与者发送 `prepare` 请求（其中包括事务内容）告诉参与者你们需要执行事务了，如果能执行我发的事务内容那么就先执行但不提交，执行后请给我回复。然后参与者收到 `prepare` 消息后，他们会开始执行事务（但不提交），并将 `Undo` 和 `Redo` 信息记入事务日志中，之后参与者就向协调者反馈是否准备好了 - -**阶段二：执行事务提交** - -协调者根据各参与者的反馈情况决定最终是否可以提交事务，如果反馈都是Yes，发送提交`commit`请求，参与者提交成功后返回 `Ack` 消息，协调者接收后就完成了。如果反馈是No 或者超时未反馈，发送 `Rollback` 请求，利用阶段一记录表的 `Undo` 信息执行回滚，并反馈给协调者`Ack` ，中断消息 - -![](https://tva1.sinaimg.cn/large/00831rSTly1gclosfvncqj30hs09j0td.jpg) - -优点：原理简单、实现方便。 - -缺点： - -- **单点故障问题**，如果协调者挂了那么整个系统都处于不可用的状态了 -- **阻塞问题**，即当协调者发送 `prepare` 请求，参与者收到之后如果能处理那么它将会进行事务的处理但并不提交，这个时候会一直占用着资源不释放，如果此时协调者挂了，那么这些资源都不会再释放了，这会极大影响性能 -- **数据不一致问题**，比如当第二阶段，协调者只发送了一部分的 `commit` 请求就挂了，那么也就意味着，收到消息的参与者会进行事务的提交，而后面没收到的则不会进行事务提交，那么这时候就会产生数据不一致性问题 - - - -### 3PC - -3PC，是 Three-Phase-Comimit 的缩写，即「**三阶段提交**」，是二阶段的改进版，将二阶段提交协议的“提交事务请求”过程一分为二。 - -**阶段一：CanCommit** - -协调者向所有参与者发送 `CanCommit` 请求，参与者收到请求后会根据自身情况查看是否能执行事务，如果可以则返回 YES 响应并进入预备状态，否则返回 NO - -**阶段二：PreCommit** - -协调者根据参与者返回的响应来决定是否可以进行下面的 `PreCommit` 操作。如果上面参与者返回的都是 YES，那么协调者将向所有参与者发送 `PreCommit` 预提交请求，**参与者收到预提交请求后，会进行事务的执行操作，并将 Undo 和 Redo 信息写入事务日志中** ，最后如果参与者顺利执行了事务则给协调者返回成功的 `Ack` 响应。如果在第一阶段协调者收到了 **任何一个 NO** 的信息，或者 **在一定时间内** 并没有收到全部的参与者的响应，那么就会中断事务，它会向所有参与者发送中断请求 `abort`，参与者收到中断请求之后会立即中断事务，或者在一定时间内没有收到协调者的请求，它也会中断事务 - -**阶段三：DoCommit** - -这个阶段其实和 `2PC` 的第二阶段差不多，如果协调者收到了所有参与者在 `PreCommit` 阶段的 YES 响应，那么协调者将会给所有参与者发送 `DoCommit` 请求，**参与者收到 DoCommit 请求后则会进行事务的提交工作**，完成后则会给协调者返回响应，协调者收到所有参与者返回的事务提交成功的响应之后则完成事务。若协调者在 `PreCommit` 阶段 **收到了任何一个 NO 或者在一定时间内没有收到所有参与者的响应** ，那么就会进行中断请求的发送，参与者收到中断请求后则会 **通过上面记录的回滚日志** 来进行事务的回滚操作，并向协调者反馈回滚状况，协调者收到参与者返回的消息后，中断事务。 - -![](https://tva1.sinaimg.cn/large/00831rSTly1gclot2rul3j30j60cpgmo.jpg) - -降低了参与者的阻塞范围，且能在单点故障后继续达成一致。 - -但是最重要的一致性并没有得到根本的解决，比如在 `PreCommit` 阶段，当一个参与者收到了请求之后其他参与者和协调者挂了或者出现了网络分区，这个时候收到消息的参与者都会进行事务提交，这就会出现数据不一致性问题。 - ------- - - - -## 讲一讲 Paxos 算法？ - -`Paxos` 算法是基于**消息传递且具有高度容错特性的一致性算法**，是目前公认的解决分布式一致性问题最有效的算法之一，**其解决的问题就是在分布式系统中如何就某个值（决议）达成一致** 。 - -在 `Paxos` 中主要有三个角色，分别为 `Proposer提案者`、`Acceptor表决者`、`Learner学习者`。`Paxos` 算法和 `2PC` 一样，也有两个阶段，分别为 `Prepare` 和 `accept` 阶段。 - -在具体的实现中，一个进程可能同时充当多种角色。比如一个进程可能既是 Proposer 又是 Acceptor 又是Learner。Proposer 负责提出提案，Acceptor 负责对提案作出裁决（accept与否），learner 负责学习提案结果。 - -还有一个很重要的概念叫「**提案**」（Proposal）。最终要达成一致的 value 就在提案里。只要 Proposer 发的提案被半数以上的 Acceptor 接受，Proposer 就认为该提案里的 value 被选定了。Acceptor 告诉 Learner 哪个 value 被选定，Learner 就认为那个 value 被选定。 - -**阶段一：prepare 阶段** - -1. `Proposer` 负责提出 `proposal`，每个提案者在提出提案时都会首先获取到一个 **具有全局唯一性的、递增的提案编号N**，即在整个集群中是唯一的编号 N，然后将该编号赋予其要提出的提案，在**第一阶段是只将提案编号发送给所有的表决者**。 - -2. 如果一个 Acceptor 收到一个编号为 N 的 Prepare 请求，如果小于它已经响应过的请求，则拒绝，不回应或回复error。若 N 大于该 Acceptor 已经响应过的所有 Prepare 请求的编号（maxN），那么它就会将它**已经批准过的编号最大的提案**（如果有的话，如果还没有的accept提案的话返回{pok，null，null}）作为响应反馈给 Proposer，同时该 Acceptor 承诺不再接受任何编号小于 N 的提案 - - eg：假定一个 Acceptor 已经响应过的所有 Prepare 请求对应的提案编号分别是1、2、...5和7，那么该 Acceptor 在接收到一个编号为8的 Prepare 请求后，就会将 7 的提案作为响应反馈给 Proposer。 - -**阶段二：accept 阶段** - -1. 如果一个 Proposer 收到半数以上 Acceptor 对其发出的编号为 N 的 Prepare 请求的响应，那么它就会发送一个针对 [N,V] 提案的 Accept 请求半数以上的 Acceptor。注意：V 就是收到的响应中编号最大的提案的 value，如果响应中不包含任何提案，那么 V 就由 Proposer 自己决定 -2. 如果 Acceptor 收到一个针对编号为N的提案的Accept请求，只要该 Acceptor 没有对编号大于 N 的 Prepare 请求做出过响应，它就通过该提案。如果N小于 Acceptor 以及响应的 prepare 请求，则拒绝，不回应或回复error（当proposer没有收到过半的回应，那么他会重新进入第一阶段，递增提案号，重新提出prepare请求） -3. 最后是 Learner 获取通过的提案（有多种方式） - -![](https://tva1.sinaimg.cn/large/00831rSTly1gcloyv70qsj30sg0lc0ve.jpg) - -**`paxos` 算法的死循环问题** - -其实就有点类似于两个人吵架，小明说我是对的，小红说我才是对的，两个人据理力争的谁也不让谁🤬🤬。 - -比如说，此时提案者 P1 提出一个方案 M1，完成了 `Prepare` 阶段的工作，这个时候 `acceptor` 则批准了 M1，但是此时提案者 P2 同时也提出了一个方案 M2，它也完成了 `Prepare` 阶段的工作。然后 P1 的方案已经不能在第二阶段被批准了（因为 `acceptor` 已经批准了比 M1 更大的 M2），所以 P1 自增方案变为 M3 重新进入 `Prepare` 阶段，然后 `acceptor` ，又批准了新的 M3 方案，它又不能批准 M2 了，这个时候 M2 又自增进入 `Prepare` 阶段。。。 - -就这样无休无止的永远提案下去，这就是 `paxos` 算法的死循环问题。 - - - -## 谈下你对 ZAB 协议的了解？ - -ZAB（Zookeeper Atomic Broadcast） 协议是为分布式协调服务 Zookeeper 专门设计的一种支持**崩溃恢复的原子广播协议**。 - -在 Zookeeper 中，主要依赖 ZAB 协议来实现分布式数据一致性，基于该协议，ZooKeeper 实现了一种主备模式的系统架构来保持集群中各副本之间数据的一致性。 - -尽管 ZAB 不是 Paxos 的实现，但是 ZAB 也参考了一些 Paxos 的一些设计思想，比如： - -- leader 向 follows 提出提案(proposal) -- leader 需要在达到法定数量(半数以上)的 follows 确认之后才会进行 commit -- 每一个 proposal 都有一个纪元(epoch)号，类似于 Paxos 中的选票(ballot) - - `ZAB` 中有三个主要的角色，`Leader 领导者`、`Follower跟随者`、`Observer观察者` 。 - -- `Leader` ：集群中 **唯一的写请求处理者** ，能够发起投票（投票也是为了进行写请求）。 -- `Follower`：能够接收客户端的请求，如果是读请求则可以自己处理，**如果是写请求则要转发给 Leader 。在选举过程中会参与投票，有选举权和被选举权 。** -- **Observer ：就是没有选举权和被选举权的 Follower 。** - -在 ZAB 协议中对 zkServer(即上面我们说的三个角色的总称) 还有两种模式的定义，分别是消息广播和崩溃恢复 - -**消息广播模式** - -![ZAB广播](http://file.sunwaiting.com/zab_broadcast.png) - -1. Leader从客户端收到一个事务请求（如果是集群中其他机器接收到客户端的事务请求，会直接转发给 Leader 服务器） -2. Leader 服务器生成一个对应的事务 Proposal，并为这个事务生成一个全局递增的唯一的ZXID（通过其 ZXID 来进行排序保证顺序性） -3. Leader 将这个事务发送给所有的 Follows 节点 -4. Follower 节点将收到的事务请求加入到历史队列(Leader 会为每个 Follower 分配一个单独的队列先进先出，顺序保证消息的因果关系)中，并发送 ack 给 Leader -5. 当 Leader 收到超过半数 Follower 的 ack 消息，Leader会广播一个 commit 消息 -6. 当 Follower 收到 commit 请求时，会判断该事务的 ZXID 是不是比历史队列中的任何事务的 ZXID 都小，如果是则提交，如果不是则等待比它更小的事务的 commit - -![zab commit流程](http://file.sunwaiting.com/zab_commit_1.png) - -**崩溃恢复模式** - -ZAB 的原子广播协议在正常情况下运行良好，但天有不测风云，一旦 Leader 服务器挂掉或者由于网络原因导致与半数的 Follower 的服务器失去联系，那么就会进入崩溃恢复模式。整个恢复过程结束后需要选举出一个新的 Leader 服务器。 - -恢复模式大致可以分为四个阶段：**选举、发现、同步、广播** - -1. 当 leader 崩溃后，集群进入选举阶段，开始选举出潜在的新 leader(一般为集群中拥有最大 ZXID 的节点) -2. 进入发现阶段，follower 与潜在的新 leader 进行沟通，如果发现超过法定人数的 follower 同意，则潜在的新leader 将 epoc h加1，进入新的纪元。新的 leader 产生 -3. 集群间进行数据同步，保证集群中各个节点的事务一致 -4. 集群恢复到广播模式，开始接受客户端的写请求 - ------- - - - -## Zookeeper 怎么保证主从节点的状态同步？或者说同步流程是什么样的 - -Zookeeper 的核心是原子广播机制，这个机制保证了各个 server 之间的同步。实现这个机制的协议叫做 Zab 协议。Zab 协议有两种模式，它们分别是恢复模式和广播模式。同上 - ------- - - - -## 集群中为什么要有主节点？ - -在分布式环境中，有些业务逻辑只需要集群中的某一台机器进行执行，其他的机器可以共享这个结果，这样可以大大减少重复计算，提高性能，于是就需要进行 leader 选举。 - ------- - - - -## 集群中有 3 台服务器，其中一个节点宕机，这个时候 Zookeeper 还可以使用吗？ - -可以继续使用，单数服务器只要没超过一半的服务器宕机就可以继续使用。 - -集群规则为 2N+1 台，N >0，即最少需要 3 台。 - - - -## Zookeeper 宕机如何处理？ - -Zookeeper 本身也是集群，推荐配置不少于 3 个服务器。Zookeeper 自身也要保证当一个节点宕机时，其他节点会继续提供服务。如果是一个 Follower 宕机，还有 2 台服务器提供访问，因为 Zookeeper 上的数据是有多个副本的，数据并不会丢失；如果是一个 Leader 宕机，Zookeeper 会选举出新的 Leader。 - -Zookeeper 集群的机制是只要超过半数的节点正常，集群就能正常提供服务。只有在 Zookeeper 节点挂得太多，只剩一半或不到一半节点能工作，集群才失效。所以： - -3 个节点的 cluster 可以挂掉 1 个节点(leader 可以得到 2 票 > 1.5) - -2 个节点的 cluster 就不能挂掉任何1个节点了(leader 可以得到 1 票 <= 1) - ------- - - - -## 说下四种类型的数据节点 Znode？ - -1. PERSISTENT：持久节点，除非手动删除，否则节点一直存在于 Zookeeper 上。 - -2. EPHEMERAL：临时节点，临时节点的生命周期与客户端会话绑定，一旦客户端会话失效（客户端与 Zookeeper连接断开不一定会话失效），那么这个客户端创建的所有临时节点都会被移除。 - -3. PERSISTENT_SEQUENTIAL：持久顺序节点，基本特性同持久节点，只是增加了顺序属性，节点名后边会追加一个由父节点维护的自增整型数字。 - -4. EPHEMERAL_SEQUENTIAL：临时顺序节点，基本特性同临时节点，增加了顺序属性，节点名后边会追加一个由父节点维护的自增整型数字。 - ------- - - - -## Zookeeper选举机制 - -1. 首先对比zxid。zxid大的服务器优先作为Leader -2. 若zxid相同，比如初始化的时候，每个Server的zxid都为0，就会比较myid，myid大的选出来做Leader。 - - **服务器初始化时选举** - -> 目前有3台服务器，每台服务器均没有数据，它们的编号分别是1,2,3按编号依次启动，它们的选择举过程如下： - -1. Server1启动，给自己投票（1,0），然后发投票信息，由于其它机器还没有启动所以它收不到反馈信息，Server1的状态一直属于Looking。 -2. Server2启动，给自己投票（2,0），同时与之前启动的Server1交换结果，由于Server2的编号大所以Server2胜出，**但此时投票数正好大于半数**，所以Server2成为领导者，Server1成为小弟。 -3. Server3启动，给自己投票（3,0），同时与之前启动的Server1,Server2换信息，尽管Server3的编号大，但之前Server2已经胜出，所以Server3只能成为小弟。 -4. 当确定了Leader之后，每个Server更新自己的状态，Leader将状态更新为Leading，Follower将状态更新为Following。 - -**服务器运行期间的选举** - -> zookeeper运行期间，如果有新的Server加入，或者非Leader的Server宕机，那么Leader将会同步数据到新Server或者寻找其他备用Server替代宕机的Server。若Leader宕机，此时集群暂停对外服务，开始在内部选举新的Leader。假设当前集群中有Server1、Server2、Server3三台服务器，Server2为当前集群的Leader，由于意外情况，Server2宕机了，便开始进入选举状态。过程如下 - -1. 变更状态。其他的非Observer服务器将自己的状态改变为Looking，开始进入Leader选举。 -2. 每个Server发出一个投票（myid，zxid），由于此集群已经运行过，所以每个Server上的zxid可能不同。假设Server1的zxid为100，Server3的为99，第一轮投票中，Server1和Server3都投自己，票分别为（1，100）,（3,99）,将自己的票发送给集群中所有机器。 -3. 每个Server接收接收来自其他Server的投票，接下来的步骤与启动时步骤相同。 - - - - - - - - diff --git a/docs/soa/zookeeper/readZK.md b/docs/soa/zookeeper/readZK.md deleted file mode 100644 index c0d1c3066d..0000000000 --- a/docs/soa/zookeeper/readZK.md +++ /dev/null @@ -1 +0,0 @@ -zookeeper \ No newline at end of file diff --git a/docs/soa/zookeeper/sidebar.md b/docs/soa/zookeeper/sidebar.md deleted file mode 100644 index 5e83eb1909..0000000000 --- a/docs/soa/zookeeper/sidebar.md +++ /dev/null @@ -1,53 +0,0 @@ -- **Java基础** -- [![](https://icongr.am/simple/oracle.svg?size=25&color=231c82&colored=false)JVM](java/JVM/readJVM.md) -- [![img](https://icongr.am/fontawesome/expeditedssl.svg?size=25&color=f23131)JUC](java/JUC/readJUC.md) -- [![](https://icongr.am/devicon/java-original.svg?size=25&color=f23131)Java 8](java/Java8.md) -- [![img](https://icongr.am/entypo/address.svg?size=25&color=074ca6)设计模式](design-pattern/readDisignPattern.md) -- **数据存储和缓存** -- [![MySQL](https://icongr.am/devicon/mysql-original.svg?&size=25)MySQL](data-store/MySQL/readMySQL.md) -- [![Redis](https://icongr.am/devicon/redis-original.svg?size=25)Redis](data-store/Redis/2.readRedis.md) -- [![mongoDB](https://icongr.am/devicon/mongodb-original.svg?&size=25)mongoDB]( https://redis.io/ ) -- [![ **Elasticsearch** ](https://icongr.am/simple/elasticsearch.svg?&size=20) Elasticsearch]( https://redis.io/ ) -- [![S3](https://icongr.am/devicon/amazonwebservices-original.svg?&size=25)S3]( https://aws.amazon.com/cn/s3/ ) -- **直击面试** -- [![](https://icongr.am/material/basket.svg?size=25)Java集合面试](interview/Collections-FAQ.md) -- [![](https://icongr.am/devicon/java-plain-wordmark.svg?size=25)JVM面试](interview/JVM-FAQ.md) -- [![](https://icongr.am/devicon/mysql-original-wordmark.svg?size=25)MySQL面试](interview/MySQL-FAQ.md) -- [![](https://icongr.am/devicon/redis-original-wordmark.svg?size=25)Redis面试](interview/Redis-FAQ.md) -- [![](https://icongr.am/jam/leaf.svg?size=25&color=00FF00)Spring面试](interview/Spring-FAQ.md) -- [![](https://icongr.am/simple/bower.svg?size=25)MyBatis面试](interview/MyBatis-FAQ.md) -- [![img](https://icongr.am/entypo/network.svg?size=25&color=6495ED)计算机网络](interview/Network-FAQ.md) -- **单体架构** -- **RPC** -- [Hello Protocol Buffers](rpc/Hello-Protocol-Buffers.md) -- **面向服务架构** -- [![](https://icongr.am/fontawesome/group.svg?size=25&color=182d10)Zookeeper](soa/ZooKeeper/readZK.md) - - [分布式一致性协议](soa/ZooKeeper/Consistency-Protocol.md) - - [Hello Zookeeper](soa/ZooKeeper/Hello-Zookeeper.md) - - [Zookeeper 实战](soa/ZooKeeper/Zookeeper实战.md) -- [![message](https://icongr.am/clarity/email.svg?&size=25) 消息中间件](message-queue/readMQ.md) -- [![](https://icongr.am/devicon/nginx-original.svg?size=25&color=182d10)Nginx](nginx/nginx.md) -- **微服务架构** -- [![](https://icongr.am/simple/leaflet.svg?size=25&color=11b041&colored=false)Spring Boot](framework/SpringBoot/Hello-SpringBoot.md) -- [![](https://icongr.am/clarity/alarm-clock.svg?size=25&color=2d2b50)定时任务@Scheduled](framework/SpringBoot/@Scheduled.md) -- **大数据** -- [Hello 大数据](big-data/Hello-BigData.md) -- [![](https://icongr.am/simple/apachekafka.svg?size=25&color=121417&colored=false)Kafka](message-queue/Kafka/readKafka.md) -- **性能优化** -- [![](https://icongr.am/octicons/cpu.svg?size=25&color=780ebe)CPU 飙升问题](optimization/CPU飙升.md) -- \> JVM优化 -- \> web调优 -- \> DB调优 -- **数据结构与算法** -- [![](https://icongr.am/entypo/tree.svg?size=25&color=44c016)树](data-structure/tree.md) -- **工程化与工具** -- [![Maven](https://icongr.am/simple/apachemaven.svg?size=25&color=c93ddb&colored=false)Maven](tools/Maven.md) -- [![Git](https://icongr.am/devicon/git-original.svg?&size=16)Git](tools/Git-Specification.md) -- [![](https://icongr.am/devicon/github-original.svg?size=25&color=currentColor)Git](tools/GitHub.md) -- **其他** -- [![Linux](https://icongr.am/devicon/linux-original.svg?&size=16)Linux](linux/linux.md) -- [缕清各种Java Logging](logging/Java-Logging.md) -- [hello logback](logging/logback简单使用.md) -- **Links** -- [![Github](https://icongram.jgog.in/simple/github.svg?color=808080&size=16)Github](https://github.com/jhildenbiddle/docsify-tabs) -- [![Blog](https://icongr.am/simple/aboutme.svg?colored&size=16)My Blog](https://www.lazyegg.net) \ No newline at end of file diff --git a/docs/soa/zookeeper/sidle.md b/docs/soa/zookeeper/sidle.md deleted file mode 100644 index 40167ade99..0000000000 --- a/docs/soa/zookeeper/sidle.md +++ /dev/null @@ -1,50 +0,0 @@ -- **Java基础** -- [![](https://icongr.am/simple/oracle.svg?size=25&color=231c82&colored=false)JVM](java/JVM/readJVM.md) -- [![img](https://icongr.am/fontawesome/expeditedssl.svg?size=25&color=f23131)JUC](java/JUC/readJUC.md) -- [![](https://icongr.am/devicon/java-original.svg?size=25&color=f23131)Java 8](java/Java8.md) -- [![img](https://icongr.am/entypo/address.svg?size=25&color=074ca6)设计模式](design-pattern/readDisignPattern.md) -- **数据存储和缓存** -- [![MySQL](https://icongr.am/devicon/mysql-original.svg?&size=25)MySQL](data-store/MySQL/readMySQL.md) -- [![Redis](https://icongr.am/devicon/redis-original.svg?size=25)Redis](data-store/Redis/2.readRedis.md) -- [![mongoDB](https://icongr.am/devicon/mongodb-original.svg?&size=25)mongoDB]( https://redis.io/ ) -- [![ **Elasticsearch** ](https://icongr.am/simple/elasticsearch.svg?&size=20) Elasticsearch]( https://redis.io/ ) -- [![S3](https://icongr.am/devicon/amazonwebservices-original.svg?&size=25)S3]( https://aws.amazon.com/cn/s3/ ) -- **直击面试** -- [![](https://icongr.am/material/basket.svg?size=25)Java集合面试](interview/Collections-FAQ.md) -- [![](https://icongr.am/devicon/java-plain-wordmark.svg?size=25)JVM面试](interview/JVM-FAQ.md) -- [![](https://icongr.am/devicon/mysql-original-wordmark.svg?size=25)MySQL面试](interview/MySQL-FAQ.md) -- [![](https://icongr.am/devicon/redis-original-wordmark.svg?size=25)Redis面试](interview/Redis-FAQ.md) -- [![](https://icongr.am/jam/leaf.svg?size=25&color=00FF00)Spring面试](interview/Spring-FAQ.md) -- [![](https://icongr.am/simple/bower.svg?size=25)MyBatis面试](interview/MyBatis-FAQ.md) -- [![img](https://icongr.am/entypo/network.svg?size=25&color=6495ED)计算机网络](interview/Network-FAQ.md) -- **单体架构** -- **RPC** -- [Hello Protocol Buffers](rpc/Hello-Protocol-Buffers.md) -- **面向服务架构** -- [![](https://icongr.am/fontawesome/group.svg?size=25&color=182d10)Zookeeper](soa/ZooKeeper/readZK.md) -- [![message](https://icongr.am/clarity/email.svg?&size=25) 消息中间件](message-queue/readMQ.md) -- [![](https://icongr.am/devicon/nginx-original.svg?size=25&color=182d10)Nginx](nginx/nginx.md) -- **微服务架构** -- [![](https://icongr.am/simple/leaflet.svg?size=25&color=11b041&colored=false)Spring Boot](framework/SpringBoot/Hello-SpringBoot.md) -- [![](https://icongr.am/clarity/alarm-clock.svg?size=25&color=2d2b50)定时任务@Scheduled](framework/SpringBoot/@Scheduled.md) -- **大数据** -- [Hello 大数据](big-data/Hello-BigData.md) -- [![](https://icongr.am/simple/apachekafka.svg?size=25&color=121417&colored=false)Kafka](message-queue/Kafka/readKafka.md) -- **性能优化** -- [![](https://icongr.am/octicons/cpu.svg?size=25&color=780ebe)CPU 飙升问题](optimization/CPU飙升.md) -- \> JVM优化 -- \> web调优 -- \> DB调优 -- **数据结构与算法** -- [![](https://icongr.am/entypo/tree.svg?size=25&color=44c016)树](data-structure/tree.md) -- **工程化与工具** -- [![Maven](https://icongr.am/simple/apachemaven.svg?size=25&color=c93ddb&colored=false)Maven](tools/Maven.md) -- [![Git](https://icongr.am/devicon/git-original.svg?&size=16)Git](tools/Git-Specification.md) -- [![](https://icongr.am/devicon/github-original.svg?size=25&color=currentColor)Git](tools/GitHub.md) -- **其他** -- [![Linux](https://icongr.am/devicon/linux-original.svg?&size=16)Linux](linux/linux.md) -- [缕清各种Java Logging](logging/Java-Logging.md) -- [hello logback](logging/logback简单使用.md) -- **Links** -- [![Github](https://icongram.jgog.in/simple/github.svg?color=808080&size=16)Github](https://github.com/jhildenbiddle/docsify-tabs) -- [![Blog](https://icongr.am/simple/aboutme.svg?colored&size=16)My Blog](https://www.lazyegg.net) \ No newline at end of file diff --git a/docs/test/.DS_Store b/docs/test/.DS_Store new file mode 100644 index 0000000000..3825b2e526 Binary files /dev/null and b/docs/test/.DS_Store differ diff --git a/docs/test/index.html b/docs/test/index.html deleted file mode 100644 index 9f3737cdc4..0000000000 --- a/docs/test/index.html +++ /dev/null @@ -1,77 +0,0 @@ - - - - - - - - docsify-themeable (Test) - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - diff --git a/images/1.png b/images/1.png deleted file mode 100644 index 4bd7a42012..0000000000 Binary files a/images/1.png and /dev/null differ diff --git a/images/996.jpg b/images/996.jpg deleted file mode 100644 index fec421333b..0000000000 Binary files a/images/996.jpg and /dev/null differ diff --git a/images/JavaKeeper.jpg b/images/JavaKeeper.jpg deleted file mode 100644 index 0a85a351ad..0000000000 Binary files a/images/JavaKeeper.jpg and /dev/null differ diff --git "a/images/Java\346\236\266\346\236\204\344\275\223\347\263\273.jpg" "b/images/Java\346\236\266\346\236\204\344\275\223\347\263\273.jpg" deleted file mode 100644 index a8981c8545..0000000000 Binary files "a/images/Java\346\236\266\346\236\204\344\275\223\347\263\273.jpg" and /dev/null differ diff --git a/images/article_end.png b/images/article_end.png deleted file mode 100644 index 3c7196ec78..0000000000 Binary files a/images/article_end.png and /dev/null differ diff --git a/images/background.png b/images/background.png deleted file mode 100644 index 5d653706ac..0000000000 Binary files a/images/background.png and /dev/null differ diff --git a/images/blog_end.png b/images/blog_end.png deleted file mode 100644 index 2181ab3e51..0000000000 Binary files a/images/blog_end.png and /dev/null differ diff --git a/images/end.png b/images/end.png deleted file mode 100644 index 18ea4d83cb..0000000000 Binary files a/images/end.png and /dev/null differ diff --git a/images/keeper.jpg b/images/keeper.jpg deleted file mode 100644 index 044df8a7ba..0000000000 Binary files a/images/keeper.jpg and /dev/null differ diff --git a/images/mine.jpg b/images/mine.jpg deleted file mode 100644 index 05e5264fe0..0000000000 Binary files a/images/mine.jpg and /dev/null differ diff --git a/images/public (2).png b/images/public (2).png deleted file mode 100644 index 794c3e08f3..0000000000 Binary files a/images/public (2).png and /dev/null differ diff --git a/images/qrcode_for_gh_bb8aa4c0a688_344.jpg b/images/qrcode_for_gh_bb8aa4c0a688_344.jpg deleted file mode 100644 index 0517245993..0000000000 Binary files a/images/qrcode_for_gh_bb8aa4c0a688_344.jpg and /dev/null differ diff --git a/images/wechat-Official.png b/images/wechat-Official.png deleted file mode 100644 index f89b2ff093..0000000000 Binary files a/images/wechat-Official.png and /dev/null differ diff --git a/images/wechatsou.png b/images/wechatsou.png deleted file mode 100644 index ef42206199..0000000000 Binary files a/images/wechatsou.png and /dev/null differ