文章地址: CoSENT(一):比Sentence-BERT更有效的句向量方案
loss公式:
\(log\bigg (1+ \displaystyle \sum_{sim(i,j)>sim(k,l)}\exp^{\lambda(cos(u_k,u_l)-cos(u_i,u_j))}\bigg)\)
源码和解析:
def cosent_loss(y_true, y_pred):
"""排序交叉熵
y_true:标签/打分,y_pred:句向量
"""
#取出真实的标签(batch_size, )
y_true = y_true[::2, 0]
#标记出需要计算的 neg-pos对
y_true = K.cast(y_true[:, None] < y_true[None, :], K.floatx())
# 下面两行代码计算两句的cos值
y_pred = K.l2_normalize(y_pred, axis=1)
y_pred = K.sum(y_pred[::2] * y_pred[1::2], axis=1) * 20
# 计算出所有两两相减的cos值
y_pred = y_pred[:, None] - y_pred[None, :]
# 取出所有neg-pos对的值
y_pred = K.reshape(y_pred - (1 - y_true) * 1e12, [-1])
# 前面加0,表示log中的1
y_pred = K.concatenate([[0], y_pred], axis=0)
return K.logsumexp(y_pred)