除了关于基于类别的模型和多重语法（单词的一部分）的内容之外，我没有找到关于这个主题的任何好的解释或论文。所以，我自己想出了一个。我正在使用 Java，但这里是长度规范化转换为 Python 的代码。此外，这是假设您将每个单词的对数概率相加，就像 KenLM 对总句子（或短语）得分所做的那样。

# Get likelihood normalized by length and number of words.
# Since the likelihoods of each word are added together, phrases with more words tend to 
# have a lower probability.
def getNormLikelihood(phrase):
    score = getLikelihood(phrase) # The normal score
    numWords = phrase.count(' ')+1 # The number of words in the phrase.
    # Get the absolute difference between the 
    #   average word length of all words and the average word length of the phrase.
    # Then, weight that weight by the reciprocal of the normal score, such that the 
    #   lenWeight is less if the overall score is less.
    # Intuitively, this means that we care less about normalizing by word length the more 
    #   common the phrase is.
    # This effectively solves the problem of giving a greater likelihood for long words 
    #   that aren't even in the language model (and are given the default "<unk>" score).
    lenWeight = abs(len(phrase) / numWords - avgLen) * (-1/score)
    return score / (1/lenWeight)

您希望如何使用这个标准化分数以及它的有效性取决于您的应用程序。在我的情况下（拆分词），我使用这个归一化分数对每个拆分选项进行评分，但随后还添加（对数）带有上下文的选项的正常分数，您可以考虑该选项的先验（在朴素贝叶斯术语中） . 然后，我在原始的未拆分选项中添加了一个整体先验（手动选择，但可以调整），以获得需要拆分单词的概率。归一化分数有很大帮助，结合这些概率，我能够为我的测试用例获得 100% 的准确率。