基于协同过滤的音乐推荐系统实践

废话：近年来，「人工智能/大数据/机器学习/区块链」等等词汇越发「甚嚣尘上」(误)。作为一个仅仅对web开发有些许了解并且不想成为码畜的普通计算机本科生，虽然顶着「计算机科学与技术」这样一个看似响亮的名号，但说实在的，我对上述词汇的认知可以说是「聊胜于无」(误)，直到本学期学校开了人工智能的相关课程，才逐渐对相关信息有所了解，也直到完成本次课程实践作业，我或许才算得上真正的入门「机器学习」。

项目背景

据报道 [1]，中国有超过 9.77 亿人每周都听音乐，而 66%的人通过流媒体来听音乐。为了给用户提供更好的体验，如何为用户推荐喜爱的音乐就变得非常重要。本项目使用的数据集来自 Last.fm 音乐网站 [2]，数据集在 2011 推荐系统会议(ACM RecSys)中发布。——人工智能实践课程项目一音乐推荐系统

Recommenders: systems, sites and software that mine data to find stuff you’ll like——reddit

据报道，推荐系统给亚马逊带来了35%的销售收入，给Netflix带来了高达75%的消费，并且Youtube主页上60%的浏览来自推荐服务。——msra

完成课程目标(误，其实还是很感兴趣，不然也不会专门写博客) ——citizen5

通过本次项目实践，我们学会了如何为推荐系统预处理数据，通过相对简单的方法和第三方库实现了「协同过滤」和「随机梯度下降」，最终达到了普通人也能实现的「人工智能」。——总结

在经过大量的资料查阅和长期的小组讨论后，我们最终决定采用「Collaborative Filtering」即「协同过滤」的推荐方法。

协同过滤，简单来说是利用某兴趣相投、拥有共同经验之群体的喜好来推荐用户感兴趣的信息，个人透过合作的机制给予信息相当程度的回应（如评分）并记录下来以达到过滤的目的进而帮助别人筛选信息，回应不一定局限于特别感兴趣的，特别不感兴趣信息的纪录也相当重要。协同过滤又可分为评比（rating）或者群体过滤（social filtering）。其后成为电子商务当中很重要的一环，即根据某顾客以往的购买行为以及从具有相似购买行为的顾客群的购买行为去推荐这个顾客其“可能喜欢的品项”，也就是借由社群的喜好提供个人化的信息、商品等的推荐服务。——Wikipedia

我认为，通俗的来说，你可以这样理解协同过滤：在一个无聊的漫漫长夜里，你难以入睡并打开了某听歌软件，面对如此庞大的曲库你不知道要听些什么歌来衬托黑夜。这时，或许具备传说中「artificial intelligence」能力的听歌软件会获取和你具有相似行为的用户信息，又或许它获取了你对哪些歌曲点过红心，从而根据上述两种方式找到你或许会喜欢的歌曲。「伟大的协同过滤诞生了」。

A lot of research has been done on collaborative filtering (CF), and most popular approaches are based on low-dimensional factor models (model based matrix factorization. I will discuss these in detail). The CF techniques are broadly divided into 2-types:——Prince Grover

在查阅众多资料后，我认为协同过滤可以分为两个大类：基于记忆「memory」的协同过滤（不知道怎么翻译更准确）和基于模型的协同过滤。而「Memory-Based Collaborative Filtering」，正是我在前文中所提到的「user-item filtering」和「 item-item filtering」，在此就不再引例。

而「基于模型的协同过滤」又基于「 matrix factorization」即「矩阵分解」，矩阵分解被用作一种在潜在变量分解和降维时的无人监管的学习方法，他比「Memory-Based Collaborative Filtering」更具有可扩展性和稀疏性。在查了矩阵分解的定义后我逐渐明白了到底什么是「基于模型的协同过滤」，但又被众多的定义和术语搞得云里雾里。那么有了矩阵分解或者说是「基于模型的协同过滤」后我们到底能做些什么呢？简而言之，我们可以通过「预测」的方式来模拟原始矩阵中的缺失条目，也就是下文中即将提到的「ratings」。

Collaborative Filtering (CF)
Memory-Based Collaborative Filtering
Memory-Based CF methods can be divided into two sections: user-item filtering and item-item filtering. Here is the difference:

Item-Item Collaborative Filtering: “Users who liked this item also liked …”

User-Item Collaborative Filtering: “Users who are similar to you (kinda like the twin you never knew you had) also liked …”

Both methods require user-item matrix that contain the ratings for user uu for item ii. From that, you can calculate the similarity matrix.

The similarity values in Item-Item Collaborative Filtering are calculated by taking into account all users who have rated a pair of items.

For User-Item Collaborative Filtering, the similarity values are calculated by observing all items that are rated by a pair of users.

Model-Based Collaborative Filtering
Model-based CF methods are based on matrix factorization (MF). MF methods are used as an unsupervised learning method for latent variable decomposition and dimensionality reduction. They can handle scalability and sparsity problems better than Memory-based CF.

The goal of MF is to learn latent user preferences and item attributes from known ratings. Then use those variable to predict unknown ratings through the dot product of the latent features of users and items.

Matrix factorization restructures the user-item matrix into a low-rank matrix. You can represent it by the multiplication of two low-rank matrices, where the rows contain a vector of latent variables. You want this matrix to approximate the original matrix, as closely as possible, by multiplying the low-rank matrices together. That way, you predict the missing entries in the original matrix.

引用第三方库

import graphlab as gl
import pandas as pd
import numpy as np
import math
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
from math import sqrt
from sklearn.metrics import mean_squared_error

导入last.fm数据集并作预处理

使用pd.read_csv()导入user_artists.dat和artists.dat，并将两个dframe合并为一个新的dframe:「ap」，并将weight重命名为playcount。

plays = pd.read_csv('C:\Users\Eric000\dataSets\dat\user_artists.dat', sep='\t')
artists = pd.read_csv('artists.dat',sep='\t')

ap = pd.merge(
  artists, plays,
  how="inner",
  left_on="id",
  right_on="artistID"
)
ap = ap.rename(columns={"weight": "playCount"})

排序

通过pandas的groupby()对播放量的相关信息进行处理，并按照playCount降序排列。

artist_rank = ap.groupby(['name']) \
  .agg({'userID' : 'count', 'playCount' : 'sum'}) \
  .rename(columns={"userID" : 'totalUniqueUsers', "playCount" : "totalArtistPlays"}) \
  .sort_values(['totalArtistPlays'], ascending=False)
artist_rank['avgUserPlays'] = artist_rank['totalArtistPlays'] / artist_rank['totalUniqueUsers']

ap = ap.join(artist_rank, on="name", how="inner") \
  .sort_values(['playCount'], ascending=False)

graphlab入场

使用

ap.to_csv('artistsANDplays.csv')

sf = gl.SFrame('artistsANDplays.csv')
sf.remove_columns(['X1','id','url', 'pictureURL'])

将dfrmae输出为「artistsANDplays.csv」并实例化一个sFrme对象sf，通过remove_columns()删除暂时用不到的数据。

数据可视化

通过graphlab的canvas，我们可以非常容易地对数据进行可视化处理，这将及其简化我们的工作，不得不再次感叹python强大的工具库。

我们只需要将处理后的数据实例化为一个SFrame对象，即可通过鼠标的交互得到我们想要的可视化结果，免于手动使用代码绘制图像。

sf.show()

总览

总播放量对应的歌手

有多少独立用户听过对应歌手的歌

通过独立用户和总播放量的对应关系基本可以看出歌手的流行程度

测量误差

我们采用「RMSE」即「均方根误差」来测量误差，通过这一算法，我们对真值和预测值的偏差有了一个大概的认识。

训练推荐系统

使用「SGD」即「随机梯度下降」训练，赋予未知对象预测值

def fit(self, X_train, X_val):
  m, n = X_train.shape
  self.P = 3 * np.random.rand(self.n_latent_features, m)
  self.Q = 3 * np.random.rand(self.n_latent_features, n)
  self.train_error = []
  self.val_error = []
  users, items = X_train.nonzero()
  for epoch in range(self.n_epochs):
      for u, i in zip(users, items):
          error = X_train[u, i] - self.predictions(self.P[:,u], self.Q[:,i])
          self.P[:, u] += self.learning_rate * \
           (error * self.Q[:, i] - self.lmbda * self.P[:, u])
          self.Q[:, i] += self.learning_rate * \
           (error * self.P[:, u] - self.lmbda * self.Q[:, i])
      train_rmse = rmse(self.predictions(self.P, self.Q), X_train)
      val_rmse = rmse(self.predictions(self.P, self.Q), X_val)
      self.train_error.append(train_rmse)
      self.val_error.append(val_rmse)

实验结果

评估推荐结果

在使用RMSE比较训练组和校验组，并且通过SGD训练模型之后，我们很直观的发现两个对照组之间的差距越来越小，训练时间与预测误差呈反比例关系，即训练时间越长，误差就越小。这也是我们推荐系统的评价标准，简而言之，当「训练组」和「校验组」的差距越来越小（即二者的重合程度越来越大时），我们的推荐系统也就越精确，即项目要求文档中的「推荐的精确度」。

做出最后的推荐

还记得上文中我们将矩阵中的一些对象置零吗，接下来，我们将用推荐系统获得的预测值将这些稀疏矩阵填充。

def predict(self, X_train, user_index):
  y_hat = self.predictions(self.P, self.Q)
  predictions_index = np.where(X_train[user_index, :] == 0)[0]
  return y_hat[user_index, predictions_index].flatten()

这是预测前的结果

user_id = 999
user_index = ratings_df.index.get_loc(user_id)
predictions_index = np.where(train[user_index, :] == 0)[0]
rating_predictions = recommender.predict(train, user_index)
existing_ratings_index = np.where(train[user_index, :] > 0)[0]
existing_ratings = train[user_index, existing_ratings_index]
create_artist_ratings(
  artists,
  existing_ratings_index,
  existing_ratings
)

这是预测后的结果

create_artist_ratings(
  artists,
  predictions_index,
  rating_predictions
)

可以看出，id为999的用户，说不定不只是ColdPlay的忠实听众，Minnie Riperton或许也能让他在某个暗流涌动的寂寞之夜产生感官的愉悦。

「伟大的推荐系统诞生了」。

参考文献

[1] pandas-docs https://pandas.pydata.org/pandas-docs

[2] graphlab-docs https://turi.com/products/create/docs/index.html

[3] Wikipedia

https://en.wikipedia.org/wiki/Root-mean-square_deviation

https://en.wikipedia.org/wiki/Stochastic_gradient_descent

https://en.wikipedia.org/wiki/SGD_(disambiguation)

https://en.wikipedia.org/wiki/Recommender_system

and so on

[4] smartweed https://www.cnblogs.com/smartweed/p/7210689.html

[5] lastfm https://www.last.fm/.

[6] googlecolab https://colab.research.google.com/

[7] google Scholar https://scholar.google.com

[8] StackOverflow

https://stackoverflow.com/questions/33000061/graphlab-create-canopy-runtime-exception-unable-to-evaluate-lambdas

https://stackoverflow.com/questions/38983295/issues-downloading-graphlab-dependencies-get-dependencies?r=SearchResults

https://stackoverflow.com/search?q=Downloading+xz.

https://stackoverflow.com/questions/44994717/import-graphlab-does-not-work-properly/45596896?r=SearchResults&s=6

https://stackoverflow.com/questions/25960070/graphlab-create-importerror-no-module-named-graphlab

https://stackoverflow.com/questions/38938236/attributeerror-module-object-has-no-attribute-sframe

and so on

[9] Music artist Recommender System using Stochastic Gradient Descent | Machine Learning from Scratch https://www.curiousily.com/posts/music-artist-recommender-system-using-stochastic-gradient-descent/

[10] Various Implementations of Collaborative Filtering https://towardsdatascience.com/various-implementations-of-collaborative-filtering-100385c6dfe0

[11] v2ex https://v2ex.com

[12] github https://github.com

[13] Music Recommendation https://www.kaggle.com/myonin/music-recommendation-random-forest-xgboost

[14] surprise-docs https://surprise.readthedocs.io/en/stable/

[15] Building the optimal Book Recommender and measuring the role of Book Covers in predicting user ratings

基于协同过滤的音乐推荐系统实践

项目背景

相关工作

什么是推荐系统

推荐系统常用算法

实验方法

推荐方法选型

Collaborative Filtering (CF)

Memory-Based Collaborative Filtering

Model-Based Collaborative Filtering