|
|
|
|
|
### 一、DataX
|
|
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
|
# 实验机器：10.10.14.210
|
|
|
|
|
|
|
|
|
|
|
|
# 官网
|
|
|
|
|
|
https://github.com/alibaba/DataX
|
|
|
|
|
|
|
|
|
|
|
|
# 下载最新稳定版本
|
|
|
|
|
|
|
|
|
|
|
|
# 因网络原因，不建议在Linux中使用下面的命令下载，最好是在有魔法的Winodws机器上下载完成后，上传到Linux主机中
|
|
|
|
|
|
cd /usr/local
|
|
|
|
|
|
wget https://github.com/alibaba/DataX/archive/refs/tags/datax_v202309.tar.gz
|
|
|
|
|
|
|
|
|
|
|
|
# 解压
|
|
|
|
|
|
tar -xvzf DataX-datax_v202309.tar.gz
|
|
|
|
|
|
# 阅后即焚
|
|
|
|
|
|
rm -rf DataX-datax_v202309.tar.gz
|
|
|
|
|
|
# 修改目录
|
|
|
|
|
|
mv DataX-datax_v202309 datax
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
**环境要求**
|
|
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
|
# 我使用的是10.10.14.210 RockyLinux 9.5,满足要求
|
|
|
|
|
|
Linux
|
|
|
|
|
|
|
|
|
|
|
|
# 我使用的是JDK 17，应该可以
|
|
|
|
|
|
JDK(1.8以上，推荐1.8)
|
|
|
|
|
|
|
|
|
|
|
|
# RockyLinx 9.5自带了Python3
|
|
|
|
|
|
Python(2或3都可以)
|
|
|
|
|
|
|
|
|
|
|
|
# 这个应该不用，我不想在Linux中编译DataX
|
|
|
|
|
|
Apache Maven 3.x (Compile DataX)
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
**检查运行环境**
|
|
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
|
[root@RockyLinux datax]# java --version
|
|
|
|
|
|
openjdk 17.0.15 2025-04-15 LTS
|
|
|
|
|
|
OpenJDK Runtime Environment (Red_Hat-17.0.15.0.6-1) (build 17.0.15+6-LTS)
|
|
|
|
|
|
OpenJDK 64-Bit Server VM (Red_Hat-17.0.15.0.6-1) (build 17.0.15+6-LTS, mixed mode, sharing)
|
|
|
|
|
|
|
|
|
|
|
|
[root@RockyLinux datax]# python --version
|
|
|
|
|
|
Python 3.9.18
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
**替换Python3的脚本**
|
|
|
|
|
|
|
|
|
|
|
|
因为默认使用的是python2脚本，如果需要使用python3脚本，需要替换一下：
|
|
|
|
|
|
|
|
|
|
|
|
将`C:\Users\Administrator\Desktop\DataX-Web\datax-web\doc\datax-web\datax-python3` 目录下的三个文件
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
上传到Linux的`/usr/local/datax/core/src/main/bin`目录下，替换原来的文件即可。
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### 二、DataX Web后端代码
|
|
|
|
|
|
|
|
|
|
|
|
```html
|
|
|
|
|
|
https://github.com/WeiYe-Jing/datax-web
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
Git下载
|
|
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
https://github.com/WeiYe-Jing/datax-web.git
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
安装数据库
|
|
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
|
# 创建数据库
|
|
|
|
|
|
执行bin/db下面的datax_web.sql文件(注意老版本更新语句有指定库名)
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
开发环境配置
|
|
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
https://github.com/WeiYe-Jing/datax-web/blob/master/userGuid.md
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
<font color='red' size='4px'>**注意：因为项目依赖的SpringBoot版本较老，阿里云镜像中已经找不到这个版本，还用阿里云镜像加速下载会出现找不到JAR的问题，一定要保证 $Maven$的设置中去掉阿里云的镜像服务!**</font>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
下载Hadoop插件
|
|
|
|
|
|
|
|
|
|
|
|
> **注**：这里只完成了Windows开发环境的配置过程，Linux部署Hadoop的方法在实际部署时需要再测试记录。
|
|
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
|
https://github.com/steveloughran/winutils
|
|
|
|
|
|
|
|
|
|
|
|
# 配置Hadoop目录
|
|
|
|
|
|
D:\dsWork\dsProject\dsData\hadoop
|
|
|
|
|
|
|
|
|
|
|
|
# 添加系统变量
|
|
|
|
|
|
HADOOP_HOME = D:\dsWork\dsProject\dsData\hadoop
|
|
|
|
|
|
编辑Path变量 → 新增条目：%HADOOP_HOME%\bin
|
|
|
|
|
|
|
|
|
|
|
|
# 验证
|
|
|
|
|
|
winutils version # 应显示版本信息
|
|
|
|
|
|
hadoop fs -ls / # 测试HDFS命令是否报错
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
访问地址：
|
|
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
|
http://10.10.21.20:8080/index.html
|
|
|
|
|
|
#（默认管理员用户名：admin 密码：123456）
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### 三、DataX前端代码
|
|
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
https://github.com/WeiYe-Jing/datax-web-ui
|
|
|
|
|
|
```
|
|
|
|
|
|
|
