xiaobaoqiu Blog

Think More, Code Less

反射的效率

1.toString实现

测试不同的toString实现方式的效率,目前包括的case:

1.String的+连接生成;
2.StringBuilder生成;
3.StringBuffer生成;
4.Guava的ToStringHelper生成;
5.Apache.commons.lang3的lang3的ToStringBuilder生成;
6.Apache.commons.lang3的ReflectionToStringBuilder生成;

toString的对象包含各种类型的属性:

1
2
3
4
5
6
7
8
9
10
11
private int age;

private String name;

private String address;

private List<String> phone;

private BigDecimal salary;

private Map<String, Object> attrs;

各个toString的实现版本,其中ReflectionToStringBuilder是我们项目中使用最多的,因为它最简单:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
/**
 * 使用字符串+
 * @return
 */
public String concatToString() {
    return "Person{" +
            "age=" + age +
            ", name='" + name + '\'' +
            ", address='" + address + '\'' +
            ", phone=" + phone +
            ", salary=" + salary +
            ", attrs=" + attrs +
            '}';
}

/**
 * 使用StringBuilder
 * @return
 */
public String stringBuilderToString() {
    StringBuilder builder = new StringBuilder("Person{");
    builder.append("age=").append(age)
            .append(", name='").append(name).append("'")
            .append(", address='").append(address).append("'")
            .append(", phone=").append(phone)
            .append(", salary=").append(salary)
            .append(", attrs=").append(attrs)
            .append("}");
    return builder.toString();
}

/**
 * 使用StringBuffer
 * @return
 */
public String stringBufferToString() {
    StringBuffer buffer = new StringBuffer("Person{");
    buffer.append("age=").append(age)
            .append(", name='").append(name).append("'")
            .append(", address='").append(address).append("'")
            .append(", phone=").append(phone)
            .append(", salary=").append(salary)
            .append(", attrs=").append(attrs)
            .append("}");
    return buffer.toString();
}

/**
 * Guava的Objects,即ToStringHelper
 * @return
 */
public String toStringHelperToString() {
    return Objects.toStringHelper(this)
            .add("age", age)
            .add("name", name)
            .add("address", address)
            .add("phone", phone)
            .add("salary", salary)
            .add("attrs", attrs)
            .toString();
}

/**
 * apache.commons.lang3的ToStringBuilder
 * @return
 */
public String toStringBuilderToString() {
    return new ToStringBuilder(this)
            .append("age", age)
            .append("name", name)
            .append("address", address)
            .append("phone", phone)
            .append("salary", salary)
            .append("attrs", attrs)
            .toString();
}

/**
 * apache.commons.lang3的ReflectionToStringBuilder
 * @return
 */
public String reflectionToStringBuilderToString() {
    return ReflectionToStringBuilder.reflectionToString(this, ToStringStyle.SIMPLE_STYLE);
}

OS是Ubuntu 14.04 LTS版本,Java版本是1.7.0_80。跑1000次,记录最大最小平均时间:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
Concat Min(微秒)=25.385
Concat Max(微秒)=2218.851
Concat Avg(微秒)=56.728432999999995

StringBuilder Min(微秒)=22.501
StringBuilder Max(微秒)=6934.066
StringBuilder Avg(微秒)=50.724183

StringBuffer Min(微秒)=29.553
StringBuffer Max(微秒)=3159.808
StringBuffer Avg(微秒)=64.113464

ToStringHelper Min(微秒)=44.893
ToStringHelper Max(微秒)=9503.836
ToStringHelper Avg(微秒)=112.351293

ToStringBuilder Min(微秒)=62.806
ToStringBuilder Max(微秒)=20228.186
ToStringBuilder Avg(微秒)=194.728656

ReflectionToStringBuilder Min(微秒)=37.057
ReflectionToStringBuilder Max(微秒)=11402.172
ReflectionToStringBuilder Avg(微秒)=219.219879

整理成表格,数据如下:

toString策略 平均(微秒) 最大(微秒) 最小(微秒)
Concat 56.728433 2218.851 25.385
StringBuilder 50.724183 6934.066 22.501
StringBuffer 64.113464 3159.808 29.553
ToStringHelper 112.351293 9503.836 44.893
ToStringBuilder 194.728656 20228.186 62.806
ReflectionToStringBuilder 219.219879 11402.172 37.057

通过上面的数据,concat和StringBuilder及StringBuffer的速度差不多。其他集中语法糖普遍慢一些,我们最常用的ReflectionToStringBuilder实际上是最慢的一个版本。

1.1 实现原理

简单分析一下各个toString方式的实现原理。

  • 1.Concat

没什么说的,使用的是字符串的拼接。

  • 2.StringBuilder

内部使用char数组,每次append都会先检查数组空间释放足够,不够的话先申请一块临时数组,大小是原来的两倍大小,再用Arrays.copyOf将数据拷贝。

  • 3.StringBuffer

和StringBuilder一样,也是继承自AbstractStringBuilder类,唯一区别是为了保证线程安全,变更数据的方法都加上了synchronized关键字。

  • 4.ToStringHelper

Guava的ToStringHelper内部实现是一个简单链表,每个链表节点包含name,value以及下一个节点的指针。

1
2
3
4
5
6
7
8
private static final class ValueHolder {
        String name;
        Object value;
        Objects.ToStringHelper.ValueHolder next;

        private ValueHolder() {
        }
}

ToStringHelper类持有链表头节点和尾节点。每次调用add都是在链表尾部加一个节点,toString() 的时候遍历整个链表讲内容加到一个StringBuilder中。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
public static final class ToStringHelper {
    private final String className;
    private Objects.ToStringHelper.ValueHolder holderHead;
    private Objects.ToStringHelper.ValueHolder holderTail;

    public Objects.ToStringHelper add(String name, long value) {
        return this.addHolder(name, String.valueOf(value));
    }

    private Objects.ToStringHelper addHolder(String name, @Nullable Object value) {
        Objects.ToStringHelper.ValueHolder valueHolder = this.addHolder();
        valueHolder.value = value;
        valueHolder.name = (String)Preconditions.checkNotNull(name);
        return this;
    }

    private Objects.ToStringHelper.ValueHolder addHolder() {
        Objects.ToStringHelper.ValueHolder valueHolder = new Objects.ToStringHelper.ValueHolder();
        this.holderTail = this.holderTail.next = valueHolder;
        return valueHolder;
    }
  • 5.ToStringBuilder

ToStringBuilder内部使用一个StringBuffer,默认大小为512.另外一个style设定来toString的样式,每次执行append都跳转到对应的ToStringStyle的append函数实现上,并将当前的StringBuffer带过去(StringBuffer来存储数据,ToStringStyle来保证格式):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
public class ToStringBuilder implements Builder<String> {
    private static volatile ToStringStyle defaultStyle;
    private final StringBuffer buffer;
    private final Object object;
    private final ToStringStyle style;

    static {
    defaultStyle = ToStringStyle.DEFAULT_STYLE;
    }

    public ToStringBuilder append(String fieldName, int value) {
        this.style.append(this.buffer, fieldName, value);
        return this;
    }

ToStringStyle的append实现,每个append都很简单,就是往StringBuffer中append数据:

1
2
3
4
5
6
7
8
9
public void append(StringBuffer buffer, String fieldName, int value) {
    this.appendFieldStart(buffer, fieldName);
    this.appendDetail(buffer, fieldName, value);
    this.appendFieldEnd(buffer, fieldName);
}

protected void appendDetail(StringBuffer buffer, String fieldName, int value) {
    buffer.append(value);
}
  • 6.ReflectionToStringBuilder

ReflectionToStringBuilder实际上是继承自ToStringBuilder,每次都构建一个ReflectionToStringBuilder对象,toString的实现是使用反射完成,首先反射当前类的所以属性,再逐级往上获取父类并反射获取其属性,每拿到一个属性,调用一次ToStringBuilder的append方法:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
public static <T> String toString(
        final T object, final ToStringStyle style, final boolean outputTransients,
        final boolean outputStatics, final Class<? super T> reflectUpToClass) {
    return new ReflectionToStringBuilder(object, style, null, reflectUpToClass, outputTransients, outputStatics)
            .toString();
    }

    public String toString() {
        if (this.getObject() == null) {
            return this.getStyle().getNullText();
        }
        Class<?> clazz = this.getObject().getClass();
        this.appendFieldsIn(clazz);
        while (clazz.getSuperclass() != null && clazz != this.getUpToClass()) {
            clazz = clazz.getSuperclass();
            this.appendFieldsIn(clazz);
        }
        return super.toString();
    }

    protected void appendFieldsIn(final Class<?> clazz) {
        if (clazz.isArray()) {
            this.reflectionAppendArray(this.getObject());
            return;
        }
        final Field[] fields = clazz.getDeclaredFields();
        AccessibleObject.setAccessible(fields, true);
        for (final Field field : fields) {
            final String fieldName = field.getName();
            if (this.accept(field)) {
                try {
                    // Warning: Field.get(Object) creates wrappers objects
                    // for primitive types.
                    final Object fieldValue = this.getValue(field);
                    this.append(fieldName, fieldValue);
                } catch (final IllegalAccessException ex) {
                    //this can't happen. Would get a Security exception
                    // instead
                    //throw a runtime exception in case the impossible
                    // happens.
                    throw new InternalError("Unexpected IllegalAccessException: " + ex.getMessage());
                }
            }
        }
    }

2.beanCopy

beanCopy由很多种方式,Spring和Apache都有实现版本,这里测试各个版本的性能.包括:

1.原始的get/set
2.Spring的BeanUtils
3.cglib的BeanCopier
4.apache的BeanUtils
5.apache的PropertyUtils

原因是前段时间项目上很简单的接口时间很慢,最后同事跟到的问题是使用Spring的beanCopy方式,因为接口涉及上百个bean的属性拷贝,因此比较明显的反应了各种beancopy实现方式的差异;我们最常用的应该是Spring版本的BeanUtils。apache的BeanUtils的设计有点反人类(第一个参数是dest,第二个参数是source)。

各个包的版本:

1.Spring : 3.2.1
2.cglib : 3.1
3.commons-beanutils : 1.9.2

各种方式的实现很简单:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
public class GetSetCopy {
    public static HotelVo copy(HotelEntity entity) {
        HotelVo vo = new HotelVo();
        vo.setId(entity.getId());
        vo.setName(entity.getName());
        vo.setAddress(entity.getAddress());
        vo.setPhone(entity.getPhone());
        vo.setCreateTime(entity.getCreateTime());
        vo.setUpdateTime(entity.getUpdateTime());
        vo.setPrice(entity.getPrice());
        vo.setKp(entity.getKp());

        return vo;
    }
}

public class ApacheBeanUtilsCopy {
    public static HotelVo copy(HotelEntity entity) throws Exception {
        HotelVo vo = new HotelVo();
        BeanUtils.copyProperties(vo, entity);
        return vo;
    }
}

public class ApachePropertyUtilsCopy {
    public static HotelVo copy(HotelEntity entity) throws Exception {
        HotelVo vo = new HotelVo();
        PropertyUtils.copyProperties(vo, entity);
        return vo;
    }
}

public class CglibBeanCopy {
    public static HotelVo copy(HotelEntity entity) {
        HotelVo vo = new HotelVo();
        BeanCopier copier = BeanCopier.create(entity.getClass(), HotelVo.class, false);
        copier.copy(entity, vo, null);
        return vo;
    }
}

public class SpringBeanUtilsCopy {
    public static HotelVo copy(HotelEntity entity) {
        HotelVo vo = new HotelVo();
        BeanUtils.copyProperties(entity, vo);
        return vo;
    }
}

测试的一个结果,每个copy运行10000次(各个Copy的第一次copy都很慢,跑10000次基本可以抹平这些差异):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
GetSetCopy Min(微秒)=1.466
GetSetCopy Max(微秒)=804.236
GetSetCopy Avg(微秒)=1.7923529

Spring-BeanUtils Min(微秒)=3.339
Spring-BeanUtils Max(微秒)=99795.719
Spring-BeanUtils Avg(微秒)=28.2915534

Apache-BeanUtils Min(微秒)=9.812
Apache-BeanUtils Max(微秒)=92416.696
Apache-BeanUtils Avg(微秒)=79.0868131

Apache-PropertyUtils Min(微秒)=7.083
Apache-PropertyUtils Max(微秒)=6405.457
Apache-PropertyUtils Avg(微秒)=22.114272

Cglib-BeanCopy Min(微秒)=7.707
Cglib-BeanCopy Max(微秒)=156932.173
Cglib-BeanCopy Avg(微秒)=33.135989
copy策略 平均(微秒) 最大(微秒) 最小(微秒)
GetSet 1.7923529 804.236 1.466
Spring-BeanUtils 28.2915534 99795.719 3.339
Apache-BeanUtils 79.0868131 6405.457 9.812
Apache-PropertyUtils 22.114272 9503.836 7.083
Cglib-BeanCopy 33.135989 156932.173 7.707

原始的get/set是最快的,而且是快一个数量级。Apache-BeanUtils是最慢的。Spring-BeanUtils的性能还是比其他几个强一些(这是个错觉,其实Cglib-BeanCopy比Spring-BeanUtils快,见下面的分析)。

2.1 实现原理

简单分析一下各个bean copy方式的实现原理。

  • 1.原始的get/set

没什么好说的,最原始。

  • 2.Spring的BeanUtils

Spring的BeanUtils可以使用String [] ignoreProperties 指定忽略某些属性的复制,当我们需要手动处理一些特定属性的时候挺有用。

copyProperties的原理很我们想想的一样:

1.获取target类的属性列表;
2.循环遍历target属性,对每个属性,如果由write方法并且不在ignoreProperties里面,调用soure对应属性的read方法获取属性值,再调用target的write方法;

细节:

1.缓存
获取target类的属性列表的时候有缓存,见CachedIntrospectionResults类。这也再一定成都上解释了为什么第一次调用会慢一些。
2.Accessible
注意,source的read和target的write方法可以不是Accessible的,即可以是private等修饰的方法;
3.浅拷贝
注意其拷贝是浅拷贝。

大致的代码(精简了一些):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
private static void copyProperties(Object source, Object target, Class<?> editable, String[] ignoreProperties) throws BeansException {

        PropertyDescriptor[] targetPds = getPropertyDescriptors(actualEditable);    //获取target的属性描述符,有缓存
        List ignoreList = ignoreProperties != null?Arrays.asList(ignoreProperties):null;
        PropertyDescriptor[] arr$ = targetPds;
        int len$ = targetPds.length;

        for(int i$ = 0; i$ < len$; ++i$) {
            PropertyDescriptor targetPd = arr$[i$];
            if(targetPd.getWriteMethod() != null && (ignoreProperties == null || !ignoreList.contains(targetPd.getName()))) {   //有write方法并且不在ignoreProperties中
                PropertyDescriptor sourcePd = getPropertyDescriptor(source.getClass(), targetPd.getName());
                if(sourcePd != null && sourcePd.getReadMethod() != null) {
                    try {
                        Method ex = sourcePd.getReadMethod();   //获取source的read方法
                        if(!Modifier.isPublic(ex.getDeclaringClass().getModifiers())) {
                            ex.setAccessible(true);
                        }

                        Object value = ex.invoke(source, new Object[0]);    //调用source的read方法获取属性对应的value
                        Method writeMethod = targetPd.getWriteMethod();
                        if(!Modifier.isPublic(writeMethod.getDeclaringClass().getModifiers())) {
                            writeMethod.setAccessible(true);
                        }

                        writeMethod.invoke(target, new Object[]{value});    //调用target的write方法
                    } catch (Throwable var15) {
                        throw new FatalBeanException("Could not copy properties from source to target", var15);
                    }
                }
            }
        }
    }
  • 3.cglib的BeanCopier

BeanCopier是CGLIB包中的一个工具类,使用基于动态代理机制的方式进行Bean Copy。因为使用生成的子类中的方法进行属性复制,所以会比使用反射机制的复制在效率上会高出很多。前面一直想写一片代理方面的Blog,一直没完成,尽量他会补上,到时候会分析CGLIB的原理。

我们这里的测试结果显示BeanCopier还不如Spring的BeanUtils,原因说BeanCopier.create的创建效率会比较低,所以在使用过程中建议将 BeanCopier.create 创建的对象声明为 static,避免使用中每次都去创建新的 BeanCopier 对象。我测试中发现,如果我们只显示的调用一次create方法,发现起速度明显快于Spring的BeanUtils(起最小值甚至小于get/set方式的最小值):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
GetSetCopy Min(微秒)=1.46
GetSetCopy Max(微秒)=2149.855
GetSetCopy Avg(微秒)=2.0715596

Spring-BeanUtils Min(微秒)=3.669
Spring-BeanUtils Max(微秒)=101262.127
Spring-BeanUtils Avg(微秒)=29.613658

Apache-BeanUtils Min(微秒)=10.174
Apache-BeanUtils Max(微秒)=80342.749
Apache-BeanUtils Avg(微秒)=82.383995

Apache-PropertyUtils Min(微秒)=5.468
Apache-PropertyUtils Max(微秒)=17185.561
Apache-PropertyUtils Avg(微秒)=26.5104518

Cglib-BeanCopy Min(微秒)=0.745
Cglib-BeanCopy Max(微秒)=130832.581
Cglib-BeanCopy Avg(微秒)=14.6252958
  • 4.apache的BeanUtils

主要代码大致如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
public static void copyProperties(Object dest, Object orig) throws IllegalAccessException, InvocationTargetException {
    BeanUtilsBean.getInstance().copyProperties(dest, orig);
}

public void copyProperties(Object dest, Object orig) throws IllegalAccessException, InvocationTargetException {
        int i;
        String name;
        Object e;
        if(orig instanceof DynaBean) {  //DynaBean ??
            DynaProperty[] origDescriptors = ((DynaBean)orig).getDynaClass().getDynaProperties(); 

            for(i = 0; i < origDescriptors.length; ++i) {
                name = origDescriptors[i].getName();
                if(this.getPropertyUtils().isReadable(orig, name) && this.getPropertyUtils().isWriteable(dest, name)) {
                    e = ((DynaBean)orig).get(name);
                    this.copyProperty(dest, name, e);
                }
            }
        } else if(orig instanceof Map) {    //原始数据是一个Map,即支持从一个Map到target属性的拷贝
            Map var8 = (Map)orig;
            Iterator var10 = var8.entrySet().iterator();

            while(var10.hasNext()) {
                Entry var11 = (Entry)var10.next();
                String var12 = (String)var11.getKey();
                if(this.getPropertyUtils().isWriteable(dest, var12)) {
                    this.copyProperty(dest, var12, var11.getValue());
                }
            }
        } else { 
            PropertyDescriptor[] var9 = this.getPropertyUtils().getPropertyDescriptors(orig);   //获取target的属性描述符,有缓存

            for(i = 0; i < var9.length; ++i) {
                name = var9[i].getName();
                //要求source的read方法是Accessible,target的write方法是Accessible
                if(!"class".equals(name) && this.getPropertyUtils().isReadable(orig, name) && this.getPropertyUtils().isWriteable(dest, name)) {
                    try {
                        e = this.getPropertyUtils().getSimpleProperty(orig, name);  //获取source对应属性的value
                        this.copyProperty(dest, name, e);   //设置target对应属性
                    } catch (NoSuchMethodException var7) {
                        ;
                    }
                }
            }
        }
}

细节:

1.BeanUtilsBean实例
BeanUtilsBean需要第一次new实例,因此第一次调用会慢一些。
2.缓存
获取target类的属性列表的时候有缓存,见BeanIntrospectionData类。这也再一定成都上解释了为什么第一次调用会慢一些。
3.Accessible
注意,source的read和target的write方法要求都是Accessible的;
4.浅拷贝
其拷贝是浅拷贝。
  • 5.apache的PropertyUtils PropertyUtils的实现和BeanUtils的实现基本一致。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
public static void copyProperties(Object dest, Object orig) throws IllegalAccessException, InvocationTargetException, NoSuchMethodException {
    PropertyUtilsBean.getInstance().copyProperties(dest, orig);
}

public void copyProperties(Object dest, Object orig) throws IllegalAccessException, InvocationTargetException, NoSuchMethodException {
        int i;
        String name;
        Object e;
        if(orig instanceof DynaBean) {
            DynaProperty[] origDescriptors = ((DynaBean)orig).getDynaClass().getDynaProperties();

            for(i = 0; i < origDescriptors.length; ++i) {
                name = origDescriptors[i].getName();
                if(this.isReadable(orig, name) && this.isWriteable(dest, name)) {
                    try {
                        e = ((DynaBean)orig).get(name);
                        if(dest instanceof DynaBean) {
                            ((DynaBean)dest).set(name, e);
                        } else {
                            this.setSimpleProperty(dest, name, e);
                        }
                    } catch (NoSuchMethodException var9) {
                        if(this.log.isDebugEnabled()) {
                            this.log.debug("Error writing to \'" + name + "\' on class \'" + dest.getClass() + "\'", var9);
                        }
                    }
                }
            }
        } else if(orig instanceof Map) {    //sorcce是一个Map
            Iterator var10 = ((Map)orig).entrySet().iterator();

            while(true) {
                Entry var12;
                do {
                    if(!var10.hasNext()) {
                        return;
                    }

                    var12 = (Entry)var10.next();
                    name = (String)var12.getKey();
                } while(!this.isWriteable(dest, name)); //要求target对应属性是Accessible的

                try {
                    if(dest instanceof DynaBean) {
                        ((DynaBean)dest).set(name, var12.getValue());
                    } else {
                        this.setSimpleProperty(dest, name, var12.getValue());
                    }
                } catch (NoSuchMethodException var8) {
                    if(this.log.isDebugEnabled()) {
                        this.log.debug("Error writing to \'" + name + "\' on class \'" + dest.getClass() + "\'", var8);
                    }
                }
            }
        } else {
            PropertyDescriptor[] var11 = this.getPropertyDescriptors(orig); //获取target对象属性的描述符,有缓存

            for(i = 0; i < var11.length; ++i) {
                name = var11[i].getName();
                if(this.isReadable(orig, name) && this.isWriteable(dest, name)) {   //要求source的read方法是Accessible,target的write方法是Accessible
                    try {
                        e = this.getSimpleProperty(orig, name); //获取source对应属性的值
                        if(dest instanceof DynaBean) {
                            ((DynaBean)dest).set(name, e);
                        } else {
                            this.setSimpleProperty(dest, name, e);  //设置target对应属性的值
                        }
                    } catch (NoSuchMethodException var7) {
                        if(this.log.isDebugEnabled()) {
                            this.log.debug("Error writing to \'" + name + "\' on class \'" + dest.getClass() + "\'", var7);
                        }
                    }
                }
            }
        }

    }

3.问题

碰到一个小问题,因为涉及到同一个方法的多次调用,发现第一次调用的时间远大于后面的调用:

1
2
3
4
5
6
7
8
9
10
GetSetCopy-1(微秒) : 1114.744
GetSetCopy-2(微秒) : 2.087
GetSetCopy-3(微秒) : 1.684
GetSetCopy-4(微秒) : 1.694
GetSetCopy-5(微秒) : 1.828
GetSetCopy-6(微秒) : 1.742
GetSetCopy-7(微秒) : 1.705
GetSetCopy-8(微秒) : 1.68
GetSetCopy-9(微秒) : 1.669
GetSetCopy-10(微秒) : 1.693

怀疑:

1.GetSetCopy类的初始化;
2.拷贝的目标对象类(这里即HotelVo)初始化;
3.方法调用的缓存;

尝试发现如果调用前先显示的new一个GetSetCopy对象和一个HotelVo对象,可以将第一次的时间下降到50微妙左右。如果再外面显示的调用一次copy方法,则循环内的第一次copy会降到和其他次的拷贝时间一样。

3.1 原理

待补充,会涉及到JVM一些JIT的优化等问题,自己也很期待到底什么原因…

Mycli

相信很多同学都喜欢使用命令行和Mysql交互而不喜欢各种图形化的客户端,我也是其中之一。

使用命令行交互的一个缺点就是没有自动补全等功能,很多时候忘记字段叫什么,不得不去看一下见表语句。

现在Mycli帮我们搞定这个问题了。Mycli是一个自带自动补全和语法高亮的Mysql客户端,也适用于MariaDB和Percona。赶紧使用Mycli替换你所以的alias吧。

Mycli官网:http://mycli.net/

1.安装

Linux下可以使用Python的包管理器pip安装,首先你得有pip,我使用的是Ubuntu:

1
sudo apt-get install python-pip

之后就可以安装Mycli了,可能需要sudo权限,因为它会往python的lib目录下加一些包

1
pip install mycli

高端的Mac同学使用brew就可以了。

感觉Windows被抛弃,更详细的安装见:http://mycli.net/install

源代码github地址:https://github.com/dbcli/mycli

使用pg数据库的同学可以使用类似的工具:pgcli

2.使用

使用上没啥特别,看看Mycli的手册就可以了,这里将英文简单翻译一下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
xiaobaoqiu@xiaobaoqiu:~/Sublime/Sublime2$ mycli --help
Usage: mycli [OPTIONS] [DATABASE]

Options:
  -h, --host TEXT             数据库host
  -P, --port TEXT             数据库端口. Honors $MYSQL_TCP_PORT
  -u, --user TEXT             用户名
  -S, --socket TEXT       链接数据库使用的socket文件
  -p, --password          密码
  --pass TEXT                 密码
  -v, --version               版本
  -D, --database TEXT         数据库名字
  -R, --prompt TEXT           提示格式(默认: "\t \u@\h:\d> ")
  -l, --logfile FILENAME      将每个查询和查询的结构记录到文件
  --help                      帮助文档

一个本机的截图如下:

3.配置

配置文件默认为:~/.myclirc,我们可以简单的看一下配置,都比较简单:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
# vi: ft=dosini
[main]

# Enables context sensitive auto-completion. If this is disabled the all
# possible completions will be listed.
smart_completion = True

# Multi-line mode allows breaking up the sql statements into multiple lines. If
# this is set to True, then the end of the statements must have a semi-colon.
# If this is set to False then sql statements can't be split into multiple
# lines. End of line (return) is considered as the end of the statement.
multi_line = False

# log_file location.
log_file = ~/.mycli.log

# Default log level. Possible values: "CRITICAL", "ERROR", "WARNING", "INFO"
# and "DEBUG".
log_level = INFO

# Timing of sql statments and table rendering.
timing = True

# Table format. Possible values: psql, plain, simple, grid, fancy_grid, pipe,
# orgtbl, rst, mediawiki, html, latex, latex_booktabs.
# Recommended: psql, fancy_grid and grid.
table_format = psql

# Syntax Style. Possible values: manni, igor, xcode, vim, autumn, vs, rrt,
# native, perldoc, borland, tango, emacs, friendly, monokai, paraiso-dark,
# colorful, murphy, bw, pastie, paraiso-light, trac, default, fruity
syntax_style = default

# Keybindings: Possible values: emacs, vi
key_bindings = emacs

# MySQL prompt
# \t - Product type (Percona, MySQL, Mariadb)
# \u - Username
# \h - Hostname of the server
# \d - Database name
prompt = '\t \u@\h:\d> '

4.源码

Mycli是python开发的,原因是起基于的prompt_toolkit正是python开发的。

prompt_toolkit的git地址:https://github.com/jonathanslenders/python-prompt-toolkit.git

prompt_toolkit使用著名的Pygments来做语法高亮。相信很多博客系统(包括我使用的octopress)也是使用这个来做语法高亮,突然发现世界这么小…

分词

最近有需求需要使用crate搜索时候的分词功能,正好研究一下搜索中分词相关的基础知识 .

1.什么是分词

2.为什么要分词

3.怎么做分词

参考:http://my.oschina.net/apdplat/blog/412921

Crate服务load飙高

前段时间搜索处理个P3的故障,原因是用户输入超级长的搜索字段,导致crate服务器load飙高,进而对外服务失败.

持续时间约10分钟.但是对我而言,最大的问题是这10分钟内,对crate服务器我什么都做不了.经过这次问题,准备好好看看crate以及背后ES的源码.

虽然解决方案是限制用户输入(这一点和google以及百度做法是一样的,百度的查询限制在38个汉字以内).

1.like查询

有问题的字段是name字段,使用crate的简单的like查询.根据官方文档,crate的like查询支持两种通配符:

% : 0个或者多个字符
_ : 单个字符

需要注意的注意,使用like查询可能导致慢查询.特别是当使用前置通配符开头的like查询.因为这种情况下crate需要去迭代所有行,而不能使用索引. 如果向获取更好的性能,可以考虑使用全文索引.

like参考:https://crate.io/docs/en/latest/sql/queries.html#like

全文索引参考: https://crate.io/docs/en/latest/sql/fulltext.html https://crate.io/docs/en/latest/sql/ddl.html#fulltext-indices

2.crate符取异常日志

这里贴一下当时crate服务器的异常:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
org.elasticsearch.index.query.QueryParsingException: [merchant_sea] Failed to parse
        at org.elasticsearch.index.query.IndexQueryParserService.parseQuery(IndexQueryParserService.java:370)
        at org.elasticsearch.action.count.TransportCountAction.shardOperation(TransportCountAction.java:187)
        at org.elasticsearch.action.count.CrateTransportCountAction.shardOperation(CrateTransportCountAction.java:119)
        at org.elasticsearch.action.count.CrateTransportCountAction.shardOperation(CrateTransportCountAction.java:49)
        at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:171)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.lucene.util.automaton.TooComplexToDeterminizeException: Determinizing automaton would result in more than 10000 states.
        at org.apache.lucene.util.automaton.Operations.determinize(Operations.java:743)
        at org.apache.lucene.util.automaton.RunAutomaton.<init>(RunAutomaton.java:138)
        at org.apache.lucene.util.automaton.ByteRunAutomaton.<init>(ByteRunAutomaton.java:32)
        at org.apache.lucene.util.automaton.CompiledAutomaton.<init>(CompiledAutomaton.java:203)
        at org.apache.lucene.search.AutomatonQuery.<init>(AutomatonQuery.java:84)
        at org.apache.lucene.search.AutomatonQuery.<init>(AutomatonQuery.java:65)
        at org.apache.lucene.search.WildcardQuery.<init>(WildcardQuery.java:57)
        at org.elasticsearch.index.query.WildcardQueryParser.parse(WildcardQueryParser.java:106)
        at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:281)
        at org.elasticsearch.index.query.BoolQueryParser.parse(BoolQueryParser.java:93)
        at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:281)
        at org.elasticsearch.index.query.BoolQueryParser.parse(BoolQueryParser.java:93)
        at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:281)
        at org.elasticsearch.index.query.BoolQueryParser.parse(BoolQueryParser.java:93)
        at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:281)
        at org.elasticsearch.index.query.BoolQueryParser.parse(BoolQueryParser.java:93)
        at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:281)
        at org.elasticsearch.index.query.BoolQueryParser.parse(BoolQueryParser.java:93)
        at org.elasticsearch.index.query.QueryParseContext.parseInnerQuery(QueryParseContext.java:281)
        at org.elasticsearch.index.query.IndexQueryParserService.innerParse(IndexQueryParserService.java:382)
        at org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:281)
        at org.elasticsearch.index.query.IndexQueryParserService.parse(IndexQueryParserService.java:276)
        at org.elasticsearch.index.query.IndexQueryParserService.parseQuery(IndexQueryParserService.java:354)
        ... 7 more

3.背后的原理

查询的第一步是query词的解析,这里使用的like查询,查询类型是通配符的查询(WildcardQuery)。WildcardQuery主要处理三种情况:

1
2
3
4
5
6
7
8
  /** 匹配字符串 */
  public static final char WILDCARD_STRING = '*';

  /** 匹配单个字符 */
  public static final char WILDCARD_CHAR = '?';

  /** 转义符 */
  public static final char WILDCARD_ESCAPE = '\\';

WildcardQuery继承自AutomatonQuery,AutomatonQuery实现了有限状态机匹配的搜索(记得算法导论上有介绍,和KMP算法同时出现的)。

构造AutomatonQuery的时候有一个参数maxDeterminizedStates,意义是构造出来的状态机的最大状态个数。默认值是Operations.DEFAULT_MAX_DETERMINIZED_STATES:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
public AutomatonQuery(final Term term, Automaton automaton) {
    this(term, automaton, Operations.DEFAULT_MAX_DETERMINIZED_STATES);
  }

/**
   * Create a new AutomatonQuery from an {@link Automaton}.
   * 
   * @param term Term containing field and possibly some pattern structure. The
   *        term text is ignored.
   * @param automaton Automaton to run, terms that are accepted are considered a
   *        match.
   * @param maxDeterminizedStates maximum number of states in the resulting
   *   automata.  If the automata would need more than this many states
   *   TooComplextToDeterminizeException is thrown.  Higher number require more
   *   space but can process more complex automata.
   */
  public AutomatonQuery(final Term term, Automaton automaton, int maxDeterminizedStates) {
    super(term.field());
    this.term = term;
    this.automaton = automaton;
    this.compiled = new CompiledAutomaton(automaton, null, true, maxDeterminizedStates);
  }

Operations.DEFAULT_MAX_DETERMINIZED_STATES的真实值为10000

1
2
3
4
5
6
7
final public class Operations {
  /**
   * Default maximum number of states that {@link Operations#determinize} should create.
   */
  public static final int DEFAULT_MAX_DETERMINIZED_STATES = 10000;
  ...
}

系统当前用户

经常需要查看当前服务器哪些用户在登录,以及他们都在干什么.

下面总结一些命令,其中last应该是最常用的命令.

1.w

w - Show who is logged on and what they are doing.

主要字段:

LOGIN@: 什么时间登录的
PCPU: 当前进程所用时间
WHAT: 用户当前正在使用的命令
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
xiaobaoqiu@xiaobaoqiu:~/octopress$ w
 20:32:42 up  9:59, 15 users,  load average: 1.80, 1.26, 1.17
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
xiaobaoq tty7     :0               10:34    9:58m 22:26   0.48s init --user
xiaobaoq pts/1    :0               10:35    2:22m  0.12s  0.12s /bin/bash
xiaobaoq pts/2    :0               11:12   44:10   0.26s  0.19s ssh cn6
xiaobaoq pts/7    :0               11:12    9:02m  0.15s  0.09s ssh cn6
xiaobaoq pts/8    :0               11:28    5:56m  0.09s  0.03s ssh cn5
xiaobaoq pts/12   :0               13:09    3:03m  0.30s  0.24s ssh cn0
xiaobaoq pts/13   :0               13:09    3:14m  0.13s  0.07s mysql -h...
xiaobaoq pts/14   :0               13:09    3:00m  1.23s  1.17s ssh cn0
xiaobaoq pts/18   :0               13:09    3:03m  3.42s  3.35s ssh cn0
xiaobaoq pts/20   :0               13:14    3:01m  0.45s  0.39s mysql -h...
xiaobaoq pts/22   :0               13:26    7:01m  0.13s  0.07s ssh cn0
xiaobaoq pts/23   :0               14:02   40:34   0.17s  0.17s /bin/bash
xiaobaoq pts/24   :0               14:03    2.00s  0.23s  0.00s w
xiaobaoq pts/25   :0               14:36    2:02   0.24s  0.18s ssh cn5
xiaobaoq pts/26   :0               17:41    2:47m  0.09s  0.09s /bin/bash

2.who

who - show who is logged on

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
xiaobaoqiu@xiaobaoqiu:~/octopress$ who
xiaobaoqiu tty7         2015-07-15 10:34 (:0)
xiaobaoqiu pts/1        2015-07-15 10:35 (:0)
xiaobaoqiu pts/2        2015-07-15 11:12 (:0)
xiaobaoqiu pts/7        2015-07-15 11:12 (:0)
xiaobaoqiu pts/8        2015-07-15 11:28 (:0)
xiaobaoqiu pts/12       2015-07-15 13:09 (:0)
xiaobaoqiu pts/13       2015-07-15 13:09 (:0)
xiaobaoqiu pts/14       2015-07-15 13:09 (:0)
xiaobaoqiu pts/18       2015-07-15 13:09 (:0)
xiaobaoqiu pts/20       2015-07-15 13:14 (:0)
xiaobaoqiu pts/22       2015-07-15 13:26 (:0)
xiaobaoqiu pts/23       2015-07-15 14:02 (:0)
xiaobaoqiu pts/24       2015-07-15 14:03 (:0)
xiaobaoqiu pts/25       2015-07-15 14:36 (:0)
xiaobaoqiu pts/26       2015-07-15 17:41 (:0)

3.users

users - print the user names of users currently logged in to the current host

1
2
xiaobaoqiu@xiaobaoqiu:~/octopress$ users
xiaobaoqiu xiaobaoqiu xiaobaoqiu xiaobaoqiu xiaobaoqiu xiaobaoqiu xiaobaoqiu xiaobaoqiu xiaobaoqiu xiaobaoqiu xiaobaoqiu xiaobaoqiu xiaobaoqiu xiaobaoqiu xiaobaoqiu

4.whoami

whoami - print effective userid

1
2
3
4
5
6
7
8
xiaobaoqiu@xiaobaoqiu:~/octopress$ whoami
xiaobaoqiu

xiaobaoqiu@xiaobaoqiu:~/octopress$ who am i
xiaobaoqiu pts/24       2015-07-15 14:03 (:0)

xiaobaoqiu@xiaobaoqiu:~/octopress$ who mom likes
xiaobaoqiu pts/24       2015-07-15 14:03 (:0)

5.id

id - print real and effective user and group IDs

1
2
xiaobaoqiu@xiaobaoqiu:~/octopress$ id
uid=1000(xiaobaoqiu) gid=1000(xiaobaoqiu) 组=1000(xiaobaoqiu),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),108(lpadmin),124(sambashare)

6.last

last, lastb - show listing of last logged in users

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
xiaobaoqiu@xiaobaoqiu:~/octopress$ last
xiaobaoq pts/26       :0               Wed Jul 15 17:41   still logged in   
xiaobaoq pts/25       :0               Wed Jul 15 14:36   still logged in   
xiaobaoq pts/24       :0               Wed Jul 15 14:03   still logged in   
xiaobaoq pts/23       :0               Wed Jul 15 14:02   still logged in   
xiaobaoq pts/22       :0               Wed Jul 15 13:26   still logged in   
xiaobaoq pts/20       :0               Wed Jul 15 13:14   still logged in   
xiaobaoq pts/20       :0               Wed Jul 15 13:14 - 13:14  (00:00)    
xiaobaoq pts/18       :0               Wed Jul 15 13:09   still logged in   
xiaobaoq pts/14       :0               Wed Jul 15 13:09   still logged in   
xiaobaoq pts/13       :0               Wed Jul 15 13:09   still logged in   
xiaobaoq pts/12       :0               Wed Jul 15 13:09   still logged in   
xiaobaoq pts/8        :0               Wed Jul 15 11:28   still logged in   
xiaobaoq pts/7        :0               Wed Jul 15 11:12   still logged in   
xiaobaoq pts/2        :0               Wed Jul 15 11:12   still logged in   
xiaobaoq pts/1        :0               Wed Jul 15 10:35   still logged in