xiaobaoqiu Blog

Think More, Code Less

Crate服务load飙高

前段时间搜索处理个P3的故障,原因是用户输入超级长的搜索字段,导致crate服务器load飙高,进而对外服务失败.

持续时间约10分钟.但是对我而言,最大的问题是这10分钟内,对crate服务器我什么都做不了.经过这次问题,准备好好看看crate以及背后ES的源码.

虽然解决方案是限制用户输入(这一点和google以及百度做法是一样的,百度的查询限制在38个汉字以内).

1.like查询

有问题的字段是name字段,使用crate的简单的like查询.根据官方文档,crate的like查询支持两种通配符:

% : 0个或者多个字符
_ : 单个字符

需要注意的注意,使用like查询可能导致慢查询.特别是当使用前置通配符开头的like查询.因为这种情况下crate需要去迭代所有行,而不能使用索引. 如果向获取更好的性能,可以考虑使用全文索引.

like参考:https://crate.io/docs/en/latest/sql/queries.html#like

全文索引参考: https://crate.io/docs/en/latest/sql/fulltext.html https://crate.io/docs/en/latest/sql/ddl.html#fulltext-indices

2.crate符取异常日志

这里贴一下当时crate服务器的异常:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

org.elasticsearch.index.query.QueryParsingException: [merchant_sea] Failed to parse
        at org.elasticsearch.index.query.IndexQueryParserService.parseQuery(IndexQueryParserService.java:370)
        at org.elasticsearch.action.count.TransportCountAction.shardOperation(TransportCountAction.java:187)
        at org.elasticsearch.action.count.CrateTransportCountAction.shardOperation(CrateTransportCountAction.java:119)
        at org.elasticsearch.action.count.CrateTransportCountAction.shardOperation(CrateTransportCountAction.java:49)
        at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:171)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.lucene.util.automaton.TooComplexToDeterminizeException: Determinizing automaton would result in more than 10000 states.
        at org.apache.lucene.util.automaton.Operations.determinize(Operations.java:743)
        at org.apache.lucene.util.automaton.RunAutomaton.<init>(RunAutomaton.java:138)
        at org.apache.lucene.util.automaton.ByteRunAutomaton.<init>(ByteRunAutomaton.java:32)
        at org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:203)
        at org.apache.lucene.search.AutomatonQuery.<init>(AutomatonQuery.java:84)
        at org.apache.lucene.search.AutomatonQuery.<init>(AutomatonQuery.java:65)
        at org.apache.lucene.search.WildcardQuery.<init>(WildcardQuery.java:57)
        at org.elasticsearch.index.query.WildcardQueryParser.parse(WildcardQueryParser.java:106)
        at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:281)
        at org.elasticsearch.index.query.BoolQueryParser.parse(BoolQueryParser.java:93)
        at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:281)
        at org.elasticsearch.index.query.BoolQueryParser.parse(BoolQueryParser.java:93)
        at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:281)
        at org.elasticsearch.index.query.BoolQueryParser.parse(BoolQueryParser.java:93)
        at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:281)
        at org.elasticsearch.index.query.BoolQueryParser.parse(BoolQueryParser.java:93)
        at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:281)
        at org.elasticsearch.index.query.BoolQueryParser.parse(BoolQueryParser.java:93)
        at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:281)
        at org.elasticsearch.index.query.IndexQueryParserService.innerParse(IndexQueryParserService.java:382)
        at org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:281)
        at org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:276)
        at org.elasticsearch.index.query.IndexQueryParserService.parseQuery(IndexQueryParserService.java:354)
        ... 7 more

3.背后的原理

查询的第一步是query词的解析，这里使用的like查询，查询类型是通配符的查询(WildcardQuery)。WildcardQuery主要处理三种情况：

1
2
3
4
5
6
7
8

  /** 匹配字符串 */
  public static final char WILDCARD_STRING = '*';

  /** 匹配单个字符 */
  public static final char WILDCARD_CHAR = '?';

  /** 转义符 */
  public static final char WILDCARD_ESCAPE = '\\';

WildcardQuery继承自AutomatonQuery，AutomatonQuery实现了有限状态机匹配的搜索(记得算法导论上有介绍，和KMP算法同时出现的)。

构造AutomatonQuery的时候有一个参数maxDeterminizedStates，意义是构造出来的状态机的最大状态个数。默认值是Operations.DEFAULT_MAX_DETERMINIZED_STATES:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

public AutomatonQuery(final Term term, Automaton automaton) {
    this(term, automaton, Operations.DEFAULT_MAX_DETERMINIZED_STATES);
  }

/**
   * Create a new AutomatonQuery from an {@link Automaton}.
   * 
   * @param term Term containing field and possibly some pattern structure. The
   *        term text is ignored.
   * @param automaton Automaton to run, terms that are accepted are considered a
   *        match.
   * @param maxDeterminizedStates maximum number of states in the resulting
   *   automata.  If the automata would need more than this many states
   *   TooComplextToDeterminizeException is thrown.  Higher number require more
   *   space but can process more complex automata.
   */
  public AutomatonQuery(final Term term, Automaton automaton, int maxDeterminizedStates) {
    super(term.field());
    this.term = term;
    this.automaton = automaton;
    this.compiled = new CompiledAutomaton(automaton, null, true, maxDeterminizedStates);
  }

Operations.DEFAULT_MAX_DETERMINIZED_STATES的真实值为10000

1
2
3
4
5
6
7

final public class Operations {
  /**
   * Default maximum number of states that {@link Operations#determinize} should create.
   */
  public static final int DEFAULT_MAX_DETERMINIZED_STATES = 10000;
  ...
}