pacemaker+drbd+neo4j高可用

博客展示了防火墙端口添加和时间同步的命令。通过firewall - cmd命令为public区域永久添加7474、7473、7687的TCP端口,还给出了每5分钟同步一次时间到cn.pool.ntp.org的命令。

主节点
192.168.154.101(s101)
CentOS Linux release 7.2.1511 (Core)
root/123
从节点
192.168.154.102(s102)
CentOS Linux release 7.2.1511 (Core)
root/123
neo4j
7473/7474/7687/tcp
corosync
5405/udp
 pcsd
2224/tcp
drbd
7789/tcp




一、磁盘格式化(drbd的前置条件)
=========================================================================
[root@s101 ~]# fdisk -l
Disk /dev/sda: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000a87cb
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     1026047      512000   83  Linux
/dev/sda2         1026048    41943039    20458496   8e  Linux LVM
Disk /dev/sdb: 5368 MB, 5368709120 bytes, 10485760 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/centos-root: 18.8 GB, 18756927488 bytes, 36634624 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/centos-swap: 2147 MB, 2147483648 bytes, 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
#查看到/dev/sdb没有挂载的,配置分区
[root@s101 ~]# fdisk /dev/sdb     
Welcome to fdisk (util-linux 2.23.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Command (m for help): p
Disk /dev/sdb: 5368 MB, 5368709120 bytes, 10485760 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xce34fc98
   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048    10242047     5120000   83  Linux
Command (m for help): d
Selected partition 1
Partition 1 is deleted
Command (m for help): p
Disk /dev/sdb: 5368 MB, 5368709120 bytes, 10485760 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xce34fc98
   Device Boot      Start         End      Blocks   Id  System
Command (m for help): n
Partition type:
   p   primary (0 primary, 0 extended, 4 free)
   e   extended
Select (default p): e
Partition number (1-4, default 1): 1
First sector (2048-10485759, default 2048):
Using default value 2048
Last sector, +sectors or +size{K,M,G} (2048-10485759, default 10485759): +4G
Partition 1 of type Extended and of size 4 GiB is set
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
[root@s101 ~]# fdisk -l
Disk /dev/sdb: 5368 MB, 5368709120 bytes, 10485760 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0xce34fc98
   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1            2048     8390655     4194304    5  Extended
Disk /dev/sda: 21.5 GB, 21474836480 bytes, 41943040 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x000a87cb
   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     1026047      512000   83  Linux
/dev/sda2         1026048    41943039    20458496   8e  Linux LVM
Disk /dev/mapper/centos-root: 18.8 GB, 18756927488 bytes, 36634624 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/mapper/centos-swap: 2147 MB, 2147483648 bytes, 4194304 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
[root@s101 ~]#partprobe
二、配置两台机子的通信互信(Neo4j+DRBD+pacemaker的前置条件)
=========================================================================
[root@s102 ~]# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
cb:45:82:1f:4a:6a:e9:bd:19:1a:37:66:da:28:f5:7b root@s102
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|       .         |
|      o o .      |
|     + o +       |
|    + . S .      |
|   o.. . o       |
|   .o.B o        |
|  .  X.=E        |
|   .+ =o         |
[root@s102 .ssh]#  ssh-copy-id -i /root/.ssh/id_rsa.pub  192.168.154.101
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@192.168.154.101's password:
Number of key(s) added: 1
Now try logging into the machine, with:   "ssh '192.168.154.101'"
and check to make sure that only the key(s) you wanted were added.
[root@s102 .ssh]# ssh s101     #同样在s102上也进行操作
三、关闭防火墙 seliux  同步时间
--======================================================================
关闭s102的防火墙及seliux
[root@s102 .ssh]#sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config   #关闭seliux
[root@s102 .ssh]#sudo systemctl stop firewalld  #关闭防火墙
[root@s102 .ssh]#sudo systemctl disable firewalld #禁止自系统启动防火墙
关闭s101的防火墙及seliux
[root@s101 .ssh]#sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config   #关闭seliux
[root@s101 .ssh]#sudo systemctl stop firewalld  #关闭防火墙
[root@s101 .ssh]#sudo systemctl disable firewalld #禁止自系统启动防火墙
=========================================================================
#注意:真实环境需开放端口,而不是关闭防火墙

[root@s101 .ssh]#firewall-cmd --zone=public --add-port=7474/tcp --permanent

[root@s101 .ssh]#firewall-cmd --zone=public --add-port=7473/tcp --permanent

[root@s101 .ssh]#firewall-cmd --zone=public --add-port=7687/tcp --permanent

[root@s101 .ssh]#firewall-cmd --zone=public --add-port=5405/udp --permanent
[root@s101 .ssh]#firewall-cmd --zone=public --add-port=2224/tcp --permanent
[root@s101 .ssh]#firewall-cmd --zone=public --add-port=7789/tcp --permanent
[root@s101 .ssh]#firewall-cmd --reload 
=========================================================================
在s101和s102节点上设置时钟同步
[root@node-001 ~]#crontab -e
*/5 * * * * ntpdate cn.pool.ntp.org   ###添加任务
[root@node-002 ~]#crontab -e
*/5 * * * * ntpdate   cn.pool.ntp.org  ###添加任务
在s101和s102上可以看到已经添加时间任务:
[root@node-001 ~]#crontab -l
*/5 * * * * ntpdate cn.pool.ntp.org
[root@node-002 ~]#crontab -l

*/5 * * * * ntpdate   cn.pool.ntp.org


四、安装DRBD主从模式配置
=========================================================================
在s101和s102安装
[root@node ~]# yum install gcc gcc-c++ make glibc flex kernel kernel-devel kernel-headers -y
[root@node ~]# yum install -y kmod-drbd84 drbd84-utils
查看s101和s102中/etc/drbd.conf
[root@s101 ~]# cat /etc/drbd.conf  #查看主配置文件包含了全局配置文件及资源的路径

# You can find an example in  /usr/share/doc/drbd.../drbd.conf.example

include "drbd.d/global_common.conf";
include "drbd.d/*.res";
查看s101和s102中修改global_common.conf文件
[root@s101 ~]# cat /etc/drbd.d/global_common.conf
# DRBD is the result of over a decade of development by LINBIT.
# In case you need professional services for DRBD or have
# feature requests visit http://www.linbit.com

global {
        usage-count no; #是否参加DRBD使用统计,默认为yes。官方统计drbd的装机量,改为no

        # Decide what kind of udev symlinks you want for "implicit" volumes
        # (those without explicit volume <vnr> {} block, implied vnr=0):
        # /dev/drbd/by-resource/<resource>/<vnr>   (explicit volumes)
        # /dev/drbd/by-resource/<resource>         (default for implict)
        udev-always-use-vnr; # treat implicit the same as explicit volumes

        # minor-count dialog-refresh disable-ip-verification
        # cmd-timeout-short 5; cmd-timeout-medium 121; cmd-timeout-long 600;
}

common {

        handlers {
                # These are EXAMPLE handlers only.
                # They may have severe implications,
                # like hard resetting the node under certain circumstances.
                # Be careful when choosing your poison.

                 #pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
                # pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
                 #local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
                # fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
                # split-brain "/usr/lib/drbd/notify-split-brain.sh root";
                # out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
                # before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
                # after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
                # quorum-lost "/usr/lib/drbd/notify-quorum-lost.sh root";
        }

        startup {
                # wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb
        }

        options {
                # cpu-mask on-no-data-accessible

                # RECOMMENDED for three or more storage nodes with DRBD 9:
                # quorum majority;
                # on-no-quorum suspend-io | io-error;
        }

        disk {
                 on-io-error detach ; #配置I/O错误处理策略为分离
                #fencing resource-only;
                # size on-io-error fencing disk-barrier disk-flushes
                # disk-drain md-flushes resync-rate resync-after al-extents
                # c-plan-ahead c-delay-target c-fill-target c-max-rate
                # c-min-rate disk-timeout
        }

        net {
                protocol C;#使用DRBD的同步协议,C为实时同步
                #配置集群中出现的脑裂问题,根据具体情况选定策略
                after-sb-0pri discard-zero-changes; #表示集群中0个primary节点,将按照没有改变数据的节点被抛弃的规则,作为从节点,也就是说如果某个节点没有改变数据,说明这个节点发生了故障,要作为从节点
                after-sb-1pri discard-secondary;  #表示集群中存在1个primary节点, 将按照以主节点数据为主进行同步
                after-sb-2pri disconnect;         #表示集群中存在2个primary节点,退出集群
                # allow-two-primaries yes;
                # protocol timeout max-epoch-size max-buffers
                # connect-int ping-int sndbuf-size rcvbuf-size ko-count
                # allow-two-primaries cram-hmac-alg shared-secret after-sb-0pri
                # after-sb-1pri after-sb-2pri always-asbp rr-conflict
                # ping-timeout data-integrity-alg tcp-cork on-congestion
                # congestion-fill congestion-extents csums-alg verify-alg
                # use-rle
        }
       #配置主备节点同步的网络带宽
       syncer{
                rate 100M;
             }
}
查看s101和s102中创建git.res文件
[root@s101 ~]# vi /etc/drbd.d/git.res
 #资源名称
resource  git {
              protocol C; #使用协议
              # 该参数有2个选项:internal和externally,其中internal表示将元数据和数据存储在同一个磁盘上;而externally表示将元数据和数据分开存储,元数据被放在另一个磁盘上。
              meta-disk internal; 
              device /dev/drbd1; #DRBD设备名称
              syncer {
                  verify-alg sha1;# 加密算法
                     }
              on s101 {
                  disk /dev/sdb;
                  address 192.168.154.101:7789; #设置DRBD监听地址与端口
                       }
              on s102 {
                  disk /dev/sdb;
                  address 192.168.154.102:7789;
                      }
                }

启用并初始化DRBD
[root@s101 ~]#modprobe drbd
[root@s101 ~]#drbdadm create-md git
[root@s101 ~]#drbdadm up git
[root@s101 ~]#drbdadm primary git
[root@s101 ~]#ssh s102 "drbdadm up git"
创建文件系统
#文件系统只能挂载在主(Primary)节点上,因此在设置好主节点后才可以对DRBD设备进行格式化操作格式化文件系统Primary:当前节点为主;在前面为当前节点
[root@s101 ~]# drbdadm role git  #目前资源主节点切换在是101上
Primary/Secondary
[root@s101 ~]#mkfs.ext4  /dev/drbd1  #
[root@s101 ~]#mount /dev/drbd1 /data
#查看DRBD的数据情况
[root@s101 ~]# cat /proc/drbd
version: 8.4.11-1 (api:1/proto:86-101)
GIT-hash: 66145a308421e9c124ec391a7848ac20203bb03c build by mockbuild@, 2018-11-03 01:26:55

1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
    ns:15952 nr:1104 dw:17104 dr:2237 al:8 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

#查看常用命令

drbdadm disconnect git                           #断开当前资源
#数据不同步或者脑裂的步骤
drbdadm connect --discard-my-data git    #同步主节点数据
drbdadm connect git                              #主节点重新连接才生效

drbdadm adjust   all                               #重新加载全局文件
drbdadm status git                                 #查看当前节点状态
drbdsetup events2 --now git                    #跟踪事件
drbdadm up git                                      #启动资源
drbdadm role git                                    #查看节点角色
drbdadm cstate git                                 #查看连接状态
drbdadm dstate git                                 #查看硬盘状态
drbdadm  primary git                              #设置当前节点为主节点
drbdadm secondary git                                  #降级secondary




相关参考的文章
DRBD+pacemaker+数据库的高可用问题总结: http://itindex.net/ Neo4j+DRBD+pacemaker高可用架构 detail/50477-drbd-pacemaker-%E6%95%B0%E6%8D%AE%E5%BA%93
五、pacemaker+corosync安装
=========================================================================
[root@s102 conf]#yum -y install corosync pacemaker
[root@s102 conf]#yum install -y pacemaker pcs psmisc policycoreutils-python
#默认安装hacluster,创建密码
[root@s102 conf]# passwd hacluster
#启动并设置开机启动PCSD服务,两台电脑都要执行
[root@s102 conf]# systemctl start pcsd
[root@s102 conf]#systemctl enable pcsd
[root@s101 conf]# systemctl start pcsd
[root@s101 conf]#systemctl enable pcsd
#配置节点间认证,以下命令只在s102节点上执行
[root@s101 conf]# pcs cluster auth s101 s102
#创建名为mycluster的新的集群 
[root@s101 conf]#pcs cluster setup --name mycluster s101 s102
#启动集群
[root@s101 conf]#pcs cluster start --all
#设置自启动集群
[root@s101 conf]#pcs cluster enable  --all
#查看集群状态
[root@s101 conf]#pcs status

#常用命令
查看集群状态:    #pcs status
查看集群当前配置: #pcs config
开机后集群自启动: #pcs cluster enable –all
启动集群:        #pcs cluster start –all
查看集群资源状态: #pcs resource show
验证集群配置情况: #crm_verify -L -V
测试资源配置:     #pcs resource debug-start resource
设置节点为备用状态:#pcs cluster standby node1
#参考安装的文章
pacemaker+corosync安装: https://blog.51cto.com/xiaozhagn/1976185
六、安装neo4j 及注册服务
#网上下载neo4j安装包,解压即可,安装忽略
#配置neo4j的数据文件路径到drbd的/data下
[root@s102 yum.repos.d]# vi /home/yx/neo4j-enterprise-3.4.0/conf/neo4j.conf
dbms.directories.data=/data/neo4j_data       #修改neo4j的路径
dbms.connectors.default_listen_address=0.0.0.0  #启用所有网卡上ip,供外网访问
dbms.connector.bolt.listen_address=:7687     #启用端口
dbms.connector.http.listen_address=:7474    #启用网页端口
dbms.connector.https.listen_address=:7473   #启用网页端口
#两个配置注册服务在centos 7以上注册/etc/systemd/system
[root@s102 yum.repos.d]# vi  /etc/systemd/system/neo4j.service
[Unit]
Description=Neo4j Graph Database
After=network.target
[Service]
Type=forking
User=root
Group=root
ExecStart=/home/neo4j-enterprise-3.4.0/bin/neo4j start
ExecStop=/home/neo4j-enterprise-3.4.0/bin/neo4j stop
ExecReload=/home/neo4j-enterprise-3.4.0/bin/neo4j restart
RemainAfterExit=no
Restart=on-failure
PIDFile = /home/neo4j-enterprise-3.4.0/run/neo4j.pid
Environment="JAVA_HOME=/opt/jdk" "NEO4J_CONF=/home/neo4j-enterprise-3.4.0/conf" "NEO4J_HOME=/home/neo4j-enterprise-3.4.0"
LimitNOFILE=60000
TimeoutSec=600
[Install]
WantedBy=multi-user.target
#两个节点都加载单元
[root@s101 ~]#systemctl  daemon-reload
#测试是否成功
[root@s102]#systemctl start neo4j.service


参考资料
七、配置crmsh
#仅需一个节点安装即可
[root@s101 ~]#cd /etc/yum.repos.d
[root@s101 yum.repos.d]# yum install crmsh
[root@s101 ~]# crm configure
---===========================================================================
#添加drbd到pacemaker上去管理,名为neo4jdrbd
crm(live)configure#primitive neo4jdrbd ocf:linbit:drbd params drbd_resource=git op start timeout=240 op stop timeout=100 op monitor role=Master interval=20 timeout=30 op monitor role=Slave interval=30 timeout=30   meta migration-threshold=10  #故障次数超过10,则切换到其他节点
#配置主从节点,限制drbd服务最多只有一个MASTER节点,名为ms_neo4jdrbd
crm(live)configure#ms ms_neo4jdrbd neo4jdrbd meta master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true 
#配置每个节点的权重,默认每个节点是100,主要是为了防止drbd中master节点离线修复之后,重新成为主节点。这样可以避免切换频繁,同时还能避免因脑裂导致数据一致性问题
crm(live)configure#rsc_defaults resource-stickiness=100  
#配置挂载的磁盘名为neo4jstore
crm(live)configure#primitive neo4jstore ocf:heartbeat:Filesystem params device=/dev/drbd1 directory=/data fstype=ext4 op start timeout=60 op stop timeout=60   
#配置绑定磁盘的加载与drbd主节点服务在同一节点
crm(live)configure# colocation  neo4jstore_with_ms_neo4jdrbd inf: neo4jstore ms_neo4jdrbd:Master  
#配置启动顺序:drbd-->磁盘
crm(live)configure#order neo4jstore_after_ms_neo4jdrbd mandatory: ms_neo4jdrbd:promote neo4jstore:start  
#配置neo4j服务
crm(live)configure#primitive neo4j  systemd:neo4j op monitor interval=20  timeout=100 op start  timeout=180 op stop  timeout=240
#绑定磁盘服务与neo4j服务在同一节点
crm(live)configure#colocation  neo4j_with_neo4jstore inf: neo4j neo4jstore
#启动顺序:磁盘--> neo4j
crm(live)configure#order neo4j_after_neo4jstore mandatory: neo4jstore neo4j
#配置集群的vip
crm(live)configure#primitive nvip ocf:heartbeat:IPaddr params ip=192.168.154.200 nic=eno16777736  cidr_netmask=255.255.255.0    
#配置vip与主从节点的主节点上
crm(live)configure#colocation vip_with_ms_neo4jdrbd inf: ms_neo4jdrbd:Master nvip  
#验证
crm(live)configure#verfiy
#查看配置
crm(live)configure#show
#提交
crm(live)configure#commit
#常用命令
crm  status  #查看集群状态
crm  node  standy  s101  #离线s101
crm  node  online   s101  #在线s101
crm resource  start  **      #启动某资源

其他补充内容

# 在双节点集群中,由于票数是偶数,当心跳出现问题(脑裂)时,两个节点都将达不到法定票数,默认quorum策略会关闭集群服务,为了避免这种情况,可以增加票数为奇数(如前文的增加ping节点),或者调整默认quorum策略为【ignore】
crm(live)configure# property no-quorum-policy=ignore

参考文档:


#drbd的优化
调整优化drbd参数:
vim /etc/drbd.d/global_common.conf
disk {
                on-io-error detach;
                disk-flushes no;
        }
        net {
                max-buffers 8000; #增大缓存区为8000
                max-epoch-size 8000;
                sndbuf-size 0; #是sendbuffer自动调整
        }
        syncer {
        rate 10M; #我的是百兆带宽,所以调整10M就行了
        al-extents 257; #增大活动日志区为257个
        }
common {
        protocol C; #协议仍采用C,即主要把数据发送到从的tcp缓存区才算完成,这是最安全严格的方式了。
}
#补充资料,centos 7之前的版本注册服务
#将下面文件放到/etc/init.d下

#修改neo4j配置文件
[root@s101 bin]# vi /home/neo4j-enterprise-3.4.0/bin/neo4j
#添加java路径
JAVA_HOME=/opt/jdk
#pacemaker 添加节点
1、先安装corosync+pacemaker软件
2、跟各节点添加互信机制
3、开放2224/tcp端口 、5405/udp
4、同步时间操作
[root@node-001 ~]# crontab -e
*/5 * * * * ntpdate cn.pool.ntp.org ###添加任务
[root@node-002 ~]# crontab -e
*/5 * * * * ntpdate cn.pool.ntp.org ###添加任务
5、默认安装hacluster,创建密码
[root@s103 conf]# passwd hacluster
#启动并设置开机启动PCSD服务,两台电脑都要执行
[root@s103 conf]# systemctl start pcsd
[root@s103 conf]#systemctl enable pcsd
6、测试是否集群验证成功
[root@s103 conf]#pcs cluster auth s101 s102  s103 -u hacluster -p 1 --debug
[root@s103 conf]#sed -i "s/SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config   #关闭seliux
7、在主节点上,将s103添加到集群
[root@s102 conf]#pcs cluster add node s103
[root@s102 conf]# pcs cluster enable --all
[root@s102 conf]# pcs cluster restart --all
8、将drbd在新增s103节点也能同时运行起来
[root@s102 conf]#crm configure edit ms_neo4jdrbd
#修改 clone-max=3即可

来自 “ ITPUB博客 ” ,链接:http://blog.itpub.net/29741328/viewspace-2648222/,如需转载,请注明出处,否则将追究法律责任。

转载于:http://blog.itpub.net/29741328/viewspace-2648222/

评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值