iot/readme.md at 008af04431680c137f75b0afd9343eb382f73aa1 - iot

5.2 KiB

Raw Blame History Unescape Escape

This file contains ambiguous Unicode characters that may be confused with others in your current locale. If your use case is intentional and legitimate, you can safely ignore this warning. Use the Escape button to highlight these characters.

#配置

doris 参考资料

http://doris.apache.org/master/zh-CN/
https://cloud.baidu.com/doc/PALO/s/Ikivhcwb5
http://doc.dorisdb.com/2146003 doris 不支持 superset 和 metabase，以后通过二次开发解决

doris 硬件配置

http://doc.dorisdb.com/2228586 BE推荐16核64GB以上，FE推荐8核16GB以上。磁盘可以使用HDD或者SSD。 CPU需要支持AVX2指令集，cat /proc/cpuinfo |grep avx2 确认有输出即可，如果没有支持，建议更换机器，DorisDB的向量化技术需要CPU指令集支持才能发挥更好的效果。网络需要万兆网卡和万兆交换机。

kafka connect

cockroachdb cdc

cockroachdb to kafka :https://www.cockroachlabs.com/docs/v21.1/stream-data-out-of-cockroachdb-using-changefeeds.html#create-a-core-changefeed

mysql

mysql配置了主从，创建了数据库example，新建了User表

kafka

通过 kafkacat 查看实时信息

kafaka-connect

查看全部插件：http://localhost:8083/connector-plugins 查看运行插件：http://localhost:8083/connectors 查看插件

mysql 导入 kafka

debezium 默认每个表一个 topic

设置时间格式： time.precision.mode

多个表导入一个 topic： "transforms":"Reroute", "transforms.Reroute.type":"io.debezium.transforms.ByLogicalTableRouter", "transforms.Reroute.topic.regex":"(.*)", "transforms.Reroute.topic.replacement":"mysql.example"

elastic 导入 doris：

使用 topics.regex 动态处理多个 topic 到 index

kibana

查看 elasticsearch 状态: http://localhost:5601

进入 mysql 容器内，连接 doris fe：

mysql -h doris-fe -P 9030 -u root

设置root密码：

SET PASSWORD FOR 'root' = PASSWORD('aA123456!');

添加 doris be：

进入 doris be 容器内，运行 cat /etc/hosts 获取当前的ip，使用该 ip 替换 hostname 或固定容器的ip ALTER SYSTEM ADD BACKEND "172.172.0.41:9050";

移除 doris be：

ALTER SYSTEM DROPP BACKEND "172.172.0.41:9050";

查看be状态：

SHOW PROC '/backends';

创建数据库：

CREATE DATABASE example;

切换数据库：

USE example;

创建表：

//curl -i -v --location-trusted -u root:aA123456! -H "format: json" -H "strip_outer_array: true" -T example.json http://doris-fe:8030/api/example/User/_stream_load CREATE TABLE User ( Id char(36) NOT NULL COMMENT 'Id', UserName varchar(255) NOT NULL COMMENT 'UserName', SecurityStamp varchar(255) REPLACE NULL COMMENT 'SecurityStamp', PasswordHash varchar(255) REPLACE NULL COMMENT 'PasswordHash', PasswordConfirmed tinyint(1) REPLACE NOT NULL COMMENT 'PasswordConfirmed', Email varchar(255) REPLACE NULL DEFAULT NULL COMMENT 'Email', EmailConfirmed tinyint(1) REPLACE NOT NULL COMMENT 'EmailConfirmed', PhoneNumber varchar(255) REPLACE NULL DEFAULT NULL COMMENT 'PhoneNumber', PhoneNumberConfirmed tinyint(1) REPLACE NOT NULL COMMENT 'PhoneNumberConfirmed', RealName varchar(255) REPLACE NULL COMMENT 'RealName', IdentityNumber varchar(255) REPLACE NULL COMMENT 'IdentityNumber', IdentityConfirmed tinyint(1) REPLACE NOT NULL COMMENT 'IdentityConfirmed', NickName varchar(255) REPLACE NULL COMMENT 'NickName', Avatar varchar(255) REPLACE NULL COMMENT 'Avatar', Sex int(0) REPLACE NULL DEFAULT NULL COMMENT 'Sex', Birthday BIGINT REPLACE NULL COMMENT 'Birthday', LockoutEnabled tinyint(1) REPLACE NOT NULL COMMENT 'LockoutEnabled', AccessFailedCount int(0) REPLACE NOT NULL COMMENT 'AccessFailedCount', LockoutEnd BIGINT REPLACE NULL DEFAULT NULL COMMENT 'LockoutEnd', RowVersion varchar(255) REPLACE NULL COMMENT 'RowVersion', Created BIGINT REPLACE NOT NULL COMMENT 'Created', Modified BIGINT REPLACE NULL DEFAULT NULL COMMENT 'Modified', Deleted BIGINT REPLACE NULL DEFAULT NULL COMMENT 'Deleted' ) AGGREGATE KEY(Id,UserName) DISTRIBUTED BY HASH(Id) BUCKETS 10 PROPERTIES("replication_num" = "1");

doris 創建 Elasticsearch 外部表：

參考：http://doc.dorisdb.com/2146013#6531__85

CREATE EXTERNAL TABLE ES_User01 ( Id CHAR(36), UserName VARCHAR(2048) ) ENGINE=ELASTICSEARCH PROPERTIES ( "hosts" = "http://172.172.0.30:9200", "index" = "mysql.example.user", "type" = "_doc" );

kafka 导入 doris（未完成）

查看导入任务： SHOW ALL ROUTINE LOAD;

创建导入任务：

CREATE ROUTINE LOAD example.job3 on User --ALTER ROUTINE LOAD FOR example.job1 columns ( Id, UserName, SecurityStamp, PasswordHash, PasswordConfirmed, Email, EmailConfirmed, PhoneNumber, PhoneNumberConfirmed, RealName, IdentityNumber, IdentityConfirmed, NickName, Avatar, Sex, Birthday, LockoutEnabled, AccessFailedCount, LockoutEnd, RowVersion, Created, Modified, Deleted ) PROPERTIES ( "format"="json",
"json_root"="$.payload.after", "desired_concurrent_number"="1",
"max_error_number"="1000", "timezone" = "Asia/Shanghai" ) FROM KAFKA ( "kafka_broker_list"= "kafka:9092", "kafka_topic" = "mysql.example.User" );

5.2 KiB Raw Blame History Unescape Escape