Linux 虚拟机：大数据集群基础环境搭建（Hadoop、Spark、Flink、Hive、Zookeeper、Kafka、Nginx）

最新推荐文章于 2023-05-16 10:23:14 发布

原创

最新推荐文章于 2023-05-16 10:23:14 发布 · 6.1k 阅读

基本信息：Centos-7.9、Java-1.8、Python-3.9、Scala-2.12、Hadoop-3.2.1、Spark-3.1.2、Flink-1.13.1、Hive-3.1.3、Zookeeper-3.8.0、Kafka-3.2.0、Nginx-1.23.1

Centos-7.9
- http://mirrors.aliyun.com/centos/7.9.2009/isos/x86_64
Java-1.8
- https://www.oracle.com/java/technologies/downloads/#java8
Python-3.9
- https://www.python.org/ftp/python/3.9.6/Python-3.9.6.tgz
Scala-2.12
- https://www.scala-lang.org/download/2.12.12.html
Hadoop-3.2.1
- http://archive.apache.org/dist/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz
Spark-3.1.2
- http://archive.apache.org/dist/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz
Flink-1.13.1
- http://archive.apache.org/dist/flink/flink-1.13.1/flink-1.13.1-bin-scala_2.12.tgz
Hive-3.1.3
- http://archive.apache.org/dist/hive/hive-3.1.3/apache-hive-3.1.3-bin.tar.gz
Zookeeper-3.8.0
- http://archive.apache.org/dist/zookeeper/zookeeper-3.8.0/apache-zookeeper-3.8.0-bin.tar.gz
Kafka-3.2.0
- http://archive.apache.org/dist/kafka/3.2.0/kafka_2.12-3.2.0.tgz
Nginx-1.23.1
- https://nginx.org/download/nginx-1.23.1.tar.gz

修改静态 IP
- vi /etc/sysconfig/network-scripts/ifcfg-eth0
- 修改完后重启网络
  - systemctl restart network
- 相关配置根据自己机器修改

BOOTPROTO="static"
ONBOOT="yes"
GATEWAY="10.211.55.1"
IPADDR="10.211.55.101"
NETMASK="255.255.255.0"
DNS1="114.114.114.114"
DNS2="8.8.8.8"

创建用户
- 创建
  - useradd -m ac_cluster
- 密码
  - passwd ac_cluster
- sudo 权限
  - vi /etc/sudoers
  - 在 root 配置下面添加对应用户数据
修改 yum 源
- 配置位置
  - /etc/yum.repos.d
- 下载 wget
  - sudo yum -y install wget
- 获取 repo 文件
  - wget http://mirrors.aliyun.com/repo/Centos-7.repo
- 备份原 repo 文件
  - mv CentOS-Base.repo CentOS-Base.repo.bak
- 改名
  - mv Centos-7.repo CentOS-Base.repo
- 刷新
  - yum clean all
  - yum makecache
下载 vim
- yum -y install vim
修改 hostname
- vim /etc/hostname
- reboot
关闭防火墙
- systemctl stop firewalld
- systemctl disable firewalld
修改域名映射
- vim /etc/hosts
配置 ssh 免密
- ssh-keygen-t rsa
  - 三次回车
- ssh-copy-id hybrid01
  - 根据子节点配置修改，有几个子节点执行几次
配置时间同步
- yum -y install ntpdate
- ntpdate ntp1.aliyun.com
- 可以配置自动执行时间同步
  - crontab -e */1 * * * * sudo /usr/sbin/ntpdate ntp1.aliyun.com

下载好安装包后解压到指定目录
- tar -zxvf xxx -C /xx/xx
wget 一键安装
- wget --no-check-certificate --no-cookies --header “Cookie: oraclelicense=accept-securebackup-cookie” http://download.oracle.com/otn-pub/java/jdk/8u131-b11/d54c1d3a095b4ff2b6607d096fa80163/jdk-8u131-linux-x64.tar.gz
环境配置
- /etc/profile 或者用户目录下的 ~/.bash_profile
- 改完记得 source

export JAVA_HOME=/xx/xx
export PATH=$JAVA_HOME/bin:$PATH

下载源码包或者 wget 下载
- wget https://www.python.org/ftp/python/3.9.6/Python-3.9.6.tgz
解压到指定目录
- tar -zxvf xxx -C /xx/xx
依赖环境安装
- sudo yum -y install vim unzip net-tools && sudo yum -y install wget && sudo yum -y install bzip2 && sudo yum -y install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel && sudo yum -y install libglvnd-glx && sudo yum -y install gcc gcc-c++
预配置
- ./configure --prefix=/xxx/program/python3
编译安装
- make && make install
配置环境变量或者将 python3 软链接放到 /usr/bin 中

export SCALA_HOME=/xx/xx
export PATH=$SCALA_HOME/bin:$PATH

标签

#linux