问题描述

健康插件报错，Jira Cluster Cache replication

查看Cluster状态， 看不到另外一个节点的 负载和运行情况

日志报错

2021-06-04 18:26:25,830+0000 localq-reader-12 ERROR      [c.a.j.c.distribution.localq.LocalQCacheOpReader] [LOCALQ] [VIA-COPY] Abandoning sending: LocalQCacheOp{cacheName='com.atlassian.jira.plugins.healthcheck.service.HeartBeatService.heartbeat', action=PUT, key=node2, value == null ? false, replicatePutsViaCopy=true, creationTimeInMillis=1622831185825} from cache replication queue: [queueId=queue_node1_2_164546f60261c7e4be0c5f5f9aaeec86_put, queuePath=/var/atlassian/application-data/jira-home/localq/queue_node1_2_164546f60261c7e4be0c5f5f9aaeec86_put], failuresCount: 1/1. Removing from queue. Error: java.rmi.MarshalException: error marshalling arguments; nested exception is: 
    	java.net.SocketException: Broken pipe (Write failed)
com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpSender$UnrecoverableFailure: java.rmi.MarshalException: error marshalling arguments; nested exception is: 
	java.net.SocketException: Broken pipe (Write failed)
	at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.send(LocalQCacheOpRMISender.java:90)
	at com.atlassian.jira.cluster.distribution.localq.LocalQCacheOpReader.run(LocalQCacheOpReader.java:96)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.rmi.MarshalException: error marshalling arguments; nested exception is: 
	java.net.SocketException: Broken pipe (Write failed)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:158)
	at net.sf.ehcache.distribution.RMICachePeer_Stub.put(Unknown Source)
	at com.atlassian.jira.cluster.distribution.localq.rmi.LocalQCacheOpRMISender.lambda$send$2(LocalQCacheOpRMISender.java:69)
	at com.atlassian.jira.cluster.distribution.localq.rmi.CachingRMICachePeerManager.withCachePeer(CachingRMICachePeerManager.java:94)
	... 6 more
Caused by: java.net.SocketException: Broken pipe (Write failed)
	at java.net.SocketOutputStream.socketWrite0(Native Method)
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
	at java.io.BufferedOutputStream.write(BufferedOutputStream.java:126)
	at java.io.ObjectOutputStream$BlockDataOutputStream.writeByte(ObjectOutputStream.java:1915)
	at java.io.ObjectOutputStream.writeFatalException(ObjectOutputStream.java:1576)
	at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:351)
	at sun.rmi.server.UnicastRef.marshalValue(UnicastRef.java:294)
	at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:153)
CODE


原因分析

这个错误在一些云服器上挺常见， 按照官方的解释  ， 这是由于地址loop 造成的

Jira 节点间缓存通信的时候默认使用了 主机的hostname, 比如node1的hostname 是 devpod-jira-1,   node2 是  devpod-jira-2 ， node1 访问node2 时 需要解析到 devpod-jira-2 的IP地址

但只有这样还不够，一些云服务把本机的hostname 解析到了 looback 得地址，如

127.0.1.1 devpod-jira-1
127.0.0.1 devpod-jira-1
CODE

这样就会造成问题，具体为什么会造成问题，我认为是程序处理的不合理

那么，我们如果解决这个问题呢

解决

在每个节点上，把hostname 指向自己的loopback 去掉， 加成服务期间通信的IP，如

/etc/hosts

#把这个去掉
#127.0.1.1 devpod-jira-1
#127.0.0.1 devpod-jira-1

#保证都加上主机的IP地址， 如
172.16.10.11 devpod-jira-1
172.16.10.12 devpod-jira-2
CODE

每个节点上都要修改

然后依次重启各个节点就可以了