1.Crate简介
最近工作中使用Crate来做搜索.总结工作中Crate的知识.
官方网站:https://crate.io/
2.Crate安装要求
Crate最低版本要求Java 7,并且Crate的所有节点和客户端需要使用相同的JVM版本.
并且Java 7要求update 55或者之后的版本,Java 8则要求update 20或者之后的版本.
警告:JAVA 7之前的版本会造成脏数据和数据丢失.
3.Crate安装
这里选择最基本的通过下载tar.gz安装包的形式安装,还可以通过Docker等形式安装
详见:https://crate.io/docs/en/latest/installation.html
3.1.下载
从下面的url下载最新的稳定版本:
https://crate.io/download/
所有可下载版本见:
https://cdn.crate.io/downloads/releases/
这里下载的是0.47.7版本:
1
wget https://cdn.crate.io/downloads/releases/crate-0.47.7.tar.gz --no-check-certificate
3.2.解压
自动生成crate-0.47.7目录
1
2
cd /home/work
sudo tar zxf ~/crate-0.47.7.tar.gz
4.Crate配置
Crate配置参考:
https://crate.io/docs/en/latest/configuration.html
Crate的有一套默认配置,通常情况下不需要额外的配置.
Crate的配置主要通过config/crate.yml文件,下载中的config/crate.yml包含了所有的配置项,包括其默认值和注释.
启动Crate的时候可以指定自己的配置文件
1
./bin/crate -Des.config=/path/to/config.yml
任何配置都可以在配置文件中指定或者作为启动时候的系统属性.比如据群名称的设置可以在启动时设置:
1
./bin/crate -Des.cluster.name=cluster
也可以在配置文件中配置:
配置的优先级如下,越往后优先级越大(即后面的配置会覆盖前面的配置):
1.internal defaults
2.system properties
3.options from config file
4.command-line properties
4.1 基本配置
一些基本的配置,工作目录和机器列表等:
1
2
3
4
5
6
7
8
9
10
11
12
13
path.data: /home/work/crate/data
path.work: /home/work/crate/work
path.logs: /home/work/crate/logs
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.timeout: 6s
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["192.168.1.100:4300", "192.168.1.101:4300", "192.168.1.102:4300"]
discovery.zen.fd.ping_interval: 10s
cluster.name: crate1
transport.tcp.port: 4300
http.port: 4200
5.Crate运行
5.1.启动
在前台运行(可以通过Control-C中断其运行):
1
2
cd ./crate-0.47.7
./bin/crate
加上-d参数让crate在后台运行.
5.2.Crash
bin下面还包含Crash(The Crate Shell),需要python环境运行,可以通过这个Crash输入诸如sql等命令
1
2
3
4
5
6
7
8
9
10
[/workspace/crate/bin]$ ./crash
...
CONNECT OK
cr> select count(1) from test;
+----------+
| count(1) |
+----------+
| 522835 |
+----------+
SELECT 1 row in set (0.032 sec)
5.3.Admin
Crate同时也提供了一个基于web的administration接口.在端口4200上提供服务,可以在浏览器中通过机器名访问:
1
http://host1.example.com:4200/_plugin/crate-admin/#/console
这个Admin在Crate的每个节点上都可以运行.
6.Crate安装简单示例
这里使用3台机器搭建Crate集群,假设三台机器的IP如下:
host1.example.com 192.168.1.100
host2.example.com 192.168.1.101
host3.example.com 192.168.1.102
每台机器上做相同的工作:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
1.将Crate下载到自己的目录
wget https://cdn.crate.io/downloads/releases/crate-0.47.7.tar.gz --no-check-certificate
2.解压
cd /home/work
sudo tar zxf ~/crate-0.47.7.tar.gz
3.创建Crate工作目录
cd /home/work
sudo mkdir crate
cd crate
sudo mkdir data
sudo mkdir logs
sudo mkdir work
4.修改基本配置(/config/crate.yml文件)
path.data: /home/work/crate/data
path.work: /home/work/crate/work
path.logs: /home/work/crate/logs
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.timeout: 6s
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["192.168.1.100:4300", "192.168.1.101:4300", "192.168.1.102:4300"]
discovery.zen.fd.ping_interval: 10s
5.配置内存大小(/bin/crate文件)
CRATE_HEAP_SIZE=4g
6.启动Crate
sudo /home/work/crate-0.47.7/bin/crate -d
7.查看启动日志
tail -f /home/work/crate/logs/crate.log
三台机器都启动了,它们之间会相互感应,形成一个集群.
7.Crate SQL
crate的sql操作和mysql基本一致,详细可以参考:
https://crate.io/docs/en/latest/sql/index.html
8.Crate运维脚本
因为dev环境需求,搭建了6套dev的crate环境,发现每次启动停止crate比较麻烦,因此自己写了点基本的维护脚本,只适用于我当前的环境,即在/home/q/crate_workspace下存在6套环境,其他环境可以根据情况自行修改使用:
1
2
3
4
5
6
7
8
9
drwxr-xr-x 4 root root 4096 Apr 28 17:37 crate1
drwxr-xr-x 4 root root 4096 Apr 28 17:37 crate2
drwxr-xr-x 4 root root 4096 Apr 28 17:37 crate3
drwxr-xr-x 4 root root 4096 Apr 30 11:25 crate4
drwxr-xr-x 4 root root 4096 Apr 28 17:37 crate5
drwxr-xr-x 4 root root 4096 Apr 28 17:37 crate6
-rwxr-xr-x 1 root root 53 Apr 28 15:41 restart_crate.sh
-rwxr-xr-x 1 root root 724 Apr 28 16:31 start_crate.sh
-rwxr-xr-x 1 root root 1041 Apr 28 17:15 stop_crate.sh
每套环境之下有自己的配置(crate-source)和工作目录(crate)和一个存放crate线程id的文件:
1
2
3
drwxr-xr-x 6 root root 4096 Apr 28 15:05 crate
drwxr-xr-x 6 root root 4096 Apr 28 15:54 crate-source
-rw-r--r-- 1 root root 5 Apr 28 17:37 pid
脚本如下,启动crate的脚本start_crate.sh:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
#!/bin/bash
if echo $1 | grep -q "/home/q/crate_workspace"
then
export CRATE_BASE=${1%/}
else
export CRATE_BASE=/home/q/crate_workspace/${1%/}
fi
echo -e $CRATE_BASE
if ! [ -e $CRATE_BASE/crate-source/bin/crate ]
then
echo -e " usage: $0 home/q/crate_workspace/CRATE_DIR\n"
exit 1;
fi
if [ -e $CRATE_BASE/pid ]
then
CRATE_ID=`cat $CRATE_BASE/pid`
echo "crate(${CRATE_ID}) still running now , please shutdown it firest";
exit 2;
fi
#CRATE_START_LOG=`$CRATE_BASE/crate-source/bin/crate -p $CRATE_BASE/pid -d`
$CRATE_BASE/crate-source/bin/crate -p $CRATE_BASE/pid -d
if [ "$?" = "0" ]; then
echo "$0 ${1%/} start succeed"
else
echo "$0 ${1%/} start failed"
echo $CRATE_START_LOG
fi
停止crate的脚本stop_crate.sh:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
#!/bin/bash
if echo $1 | grep -q "/home/q/crate_workspace"
then
export CRATE_BASE=${1%/}
else
export CRATE_BASE=/home/q/crate_workspace/${1%/}
fi
echo -e $CRATE_BASE
if ! [ -e $CRATE_BASE/crate-source/bin/crate ]
then
echo -e " usage: $0 home/q/crate_workspace/CRATE_DIR\n"
exit 1;
fi
if ! [ -e $CRATE_BASE/pid ]
then
echo "crate instance not found : ${1%/}";
exit;
fi
CRATE_ID=`cat $CRATE_BASE/pid`
for i in {1..10}; do
if [ -e $CRATE_BASE/pid ]; then
CRATE_ID=`cat $CRATE_BASE/pid`
if [ "$i" = "1" ]; then
echo -n "trying stop ($CRATE_ID): $i"
else
echo -n -e "\b$i"
fi
if [ $i -ge 5 ]; then
kill "$CRATE_ID"
rm -f "$CRATE_BASE/pid"
fi
sleep 1
else
if [ $i -gt 5 ]; then
echo -e "\n$CRATE_BASE was killed($i)"
else
echo -e "\n$CRATE_BASE was stoped"
fi
exit;
fi
done;
kill -9 "$CRATE_ID"
echo "$CRATE_BASE was force killed"
重启crate的脚本restart_crate.sh:
1
2
3
4
5
#!/bin/bash
./stop_crate.sh $1
./start_crate.sh $1