数据挖掘 - 使用 word2vec 的语义网络 - 吾爱随笔录

我有成千上万的头条新闻，我想使用 word2vec 构建一个语义网络，特别是谷歌新闻文件。我的句子看起来像

Titles
Dogs are humans’ best friends
A dog died because of an accident
You can clean dogs’ paws using natural products.
A cat was found in the kitchen

等等。

我想做的是使用语义网络在这些数据中找到一些特定的模式，例如关于狗和猫的主题的相似性。你能给我一些建议吗？

代码：

import pandas as pd
import gensim
import seaborn as sns
import matplotlib.pyplot as plt
import numpy as np
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from sklearn.manifold import TSNE

main_data.Titles = np.where(main_data.Titles.isnull(),'NA', main_data.Titles)

article_titles = main_data['Titles']

titles_list = [title for title in article_titles]

big_title_string = ' '.join(titles_list)

tokens = word_tokenize(big_title_string)

words = [word.lower() for word in tokens if word.isalpha()]

stop_words = set(stopwords.words('english'))

words = [word for word in words if not a word in stop_words]

model = gensim.models.KeyedVectors.load_word2vec_format('path/GoogleNews-vectors-negative300.bin', binary = True) 

model.vector_size

vector_list = [model[word] for word in words if word in model.vocab]

words_filtered = [word for word in words if the word in `model.vocab`]

word_vec_zip = zip(words_filtered, vector_list)

word_vec_dict = dict(word_vec_zip)
df = pd.DataFrame.from_dict(word_vec_dict, orient='index')

tsne = TSNE(n_components = 2, init = 'random', random_state = 10, perplexity = 100)

tsne_df = tsne.fit_transform(df[:400])

sns.set()
fig, ax = plt.subplots(figsize = (11.7, 8.27))
sns.scatterplot(tsne_df[:, 0], tsne_df[:, 1], alpha = 0.5)

from adjustText import adjust_text
texts = []
words_to_plot = list(np.arange(0, 400, 10))

for word in words_to_plot:
    texts.append(plt.text(tsne_df[word, 0], tsne_df[word, 1], df.index[word], fontsize = 14))
    
adjust_text(texts, force_points = 0.4, force_text = 0.4, 
            expand_points = (2,1), expand_text = (1,2),
            arrowprops = dict(arrowstyle = "-", color = 'black', lw = 0.5))

plt.show()

但是，我无法理解如何解释结果。我认为他们错了，这可能不是构建语义网络的好方法。也许我错过了一些东西......例如，这段代码在部分之后仍然保留停用词

words = [word for word in words if not a word in stop_words]

这是一个难以阅读和解释的输出示例（至少对我而言）：

如果您能给我一些关于如何执行可以在标题中显示语义相似性的语义网络的提示和建议，我将不胜感激。