Models

In the following sections, we will more systematically introduce the following models:

📄️ A3C

A3C stands for Asynchronous Advantage Actor-Critic. The A3C algorithm builds upon the Actor-Critic class of algorithms by using a neural network to approximate the actor (and critic). The actor learns the policy function using a deep neural network, while the critic estimates the value function. The asynchronous nature of the algorithm allows the agent to learn from different parts of the state space, allowing parallel learning and faster convergence. Unlike DQN agents, which use an experience replay memory, the A3C agent uses multiple workers to gather more samples for learning.

📄️ AFM

AFM stands for Attentional Factorization Machines. It Improves FM by discriminating the importance of different feature interactions, and learns the importance of each feature interaction from data via a neural attention network. Empirically, it is shown on regression task that AFM performs betters than FM with a 8.6% relative improvement, and consistently outperforms the state-of-the-art deep learning methods Wide&Deep and DeepCross with a much simpler structure and fewer model parameters.

📄️ AFN

AFN stands for Adaptive Factorization Network.

📄️ AR

Simple Association Rules (AR) are a simplified version of the association rule mining technique [Agrawal et al. 1993] with a maximum rule size of two. The method is designed to capture the frequency of two co-occurring events, e.g., “Customers who bought . . . also bought”.

📄️ ASMG

ASMG stands for Adaptive Sequential Model Generation.

📄️ AttRec

AttRec stands for Self-Attentive Sequential Recommendation.

📄️ AUSH

Attacking Recommender Systems with Augmented User Profiles

📄️ AutoInt

Song et. al., “Automatic Feature Interaction Learning via Self-Attentive Neural Networks”. CIKM, 2018.

📄️ BASR

Black-Box Attacks on Sequential Recommenders via Data-Free Model Extraction

📄️ BCQ

Current off-policy deep reinforcement learning algorithms fail to address extrapolation error by selecting actions with respect to a learned value estimate, without consideration of the accuracy of the estimate. As a result, certain outof-distribution actions can be erroneously extrapolated to higher values. However, the value of an off-policy agent can be accurately evaluated in regions where data is available.

📄️ Behavior Propensity Modeling

behavior-propensity-modeling

📄️ BiasOnly

BiasOnly is a simple baseline that assumes no interactions between users and items. Formally, it learns: (1) a global bias 𝛼; (2) scalar biases $\betau$ for each user 𝑢 ∈ U; and (3) scalar biases $\betai$ for each item 𝑖 ∈ I. Ultimately, the rating/relevance for user 𝑢 and item 𝑖 is modeled as $\hati^u = \alpha + \betau + \beta_i$.

📄️ BigGraph

PyTorch-BigGraph: A Large-scale Graph Embedding System

📄️ BPR

BPR stands for Bayesian Personalized Ranking. In matrix factorization (MF), to compute the prediction we have to multiply the user factors to the item factors:

📄️ BST

It stands for Behavior Sequence Transformer.

📄️ CASER

CASER stands for Convolutional Sequence Embedding Recommendation. Top-N sequential recommendation models each user as a sequence of items interacted in the past and aims to predict top-N ranked items that a user will likely interact in a 'near future'. The order of interaction implies that sequential patterns play an important role where more recent items in a sequence have a larger impact on the next item. Convolutional Sequence Embedding Recommendation Model (Caser) address this requirement by embedding a sequence of recent items into an image' in the time and latent spaces and learn sequential patterns as local features of the image using convolutional filters. This approach provides a unified and flexible network structure for capturing both general preferences and sequential patterns. In other words, Caser adopts convolutional neural networks capture the dynamic pattern influences of users’ recent activities.

📄️ CIGC

To pursue high efficiency, we set the target as using only new data for model updating, meanwhile not sacrificing the recommendation accuracy compared with full model retraining. This is non-trivial to achieve, since the interaction data participates in both the graph structure for model construction and the loss function for model learning, whereas the old graph structure is not allowed to use in model updating. Causal Incremental Graph Convolution (CIGC) estimates the output of full graph convolution. Incremental Graph Convolution (IGC) ingeniously combine the old representations and the incremental graph and effectively fuse the long-term and short-term preference signals. Colliding Effect Distillation (CED) aims to avoid the out-of-date issue of inactive nodes that are not in the incremental graph, which connects the new data with inactive nodes through causal inference. In particular, CED estimates the causal effect of new data on the representation of inactive nodes through the control of their collider.

📄️ CoKE

Contextualized Knowledge Graph Embedding

📄️ DCN

DCN stands for Deep and Cross Network. Manual explicit feature crossing process is very laborious and inefficient. On the other hand, automatic implicit feature crossing methods like MLPs cannot efficiently approximate even 2nd or 3rd-order feature crosses. Deep-cross networks provides a solution to this problem. DCN was designed to learn explicit and bounded-degree cross features more effectively. It starts with an input layer (typically an embedding layer), followed by a cross network containing multiple cross layers that models explicit feature interactions, and then combines with a deep network that models implicit feature interactions.

📄️ DDPG

Deterministic Policy Gradient (DPG) is a type of Actor-Critic RL algorithm that uses two neural networks: one for estimating the action value function, and the other for estimating the optimal target policy. The Deep Deterministic Policy Gradient (DDPG) agent builds upon the idea of DPG and is quite efficient compared to vanilla Actor-Critic agents due to the use of deterministic action policies.

📄️ DeepCross

Shan, Y., Hoens, T., Jiao, J., Wang, H., Yu, D. and Mao, J., 2016. Deep Crossing: Web-Scale Modeling without Manually Crafted Combinatorial Features. [online] Kdd.org.

📄️ DeepFM

DeepFM stands for Deep Factorization Machines. It consists of an FM component and a deep component which are integrated in a parallel structure. The FM component is the same as the 2-way factorization machines which is used to model the low-order feature interactions. The deep component is a multi-layered perceptron that is used to capture high-order feature interactions and nonlinearities. These two components share the same inputs/embeddings and their outputs are summed up as the final prediction.

📄️ DeepWalk

DeepWalk learns representations of online social networks graphs. By performing random walks to generate sequences, the paper demonstrated that it was able to learn vector representations of nodes (e.g., profiles, content) in the graph.

📄️ DGTN

DGTN stands for Dual-channel Graph Transition Network.

📄️ DM

The direct method (DM) involves training a model on the historical data to predict the reward for each context-action instance. In this model, the reward is the target variable, while the context and action are the input features. Using this model, we can predict the rewards for each context-action instance associated with the target policy. This is illustrated below.

📄️ DMT

Deep Multifaceted Transformers for Multi-objective Ranking in Large-Scale E-commerce Recommender Systems.

📄️ DPADL

Data Poisoning Attacks to Deep Learning Based Recommender Systems

📄️ DQN

The Q-learning component of DQN was invented in 1989 by Christopher Watkins in his PhD thesis titled “Learning from Delayed Rewards”. Experience replay quickly followed, invented by Long-Ji Lin in 1992. This played a major role in improving the efficiency of Q-learning. In the years that followed, however, there were no major success stories involving deep Q-learning. This is perhaps not surprising given the combination of limited computational power in the 1990s and early 2000s, data-hungry deep learning architectures, and the sparse, noisy, and delayed feedback signals experienced in RL. Progress had to wait for the emergence of general-purpose GPU programming, for example with the launch of CUDA in 2006, and the reignition of interest in deep learning within the machine learning community that began in the mid-2000s and rapidly accelerated after 2012.

📄️ DRQN

DRQN stands for Deep Recurrent Q-Learning. It is a combination of a recurrent neural network (RNN) and a deep Q-network (DQN). The idea being that the RNN will be able to retain information from states further back in time and incorporate that into predicting better Q values and thus performing better on games that require long term planning.

📄️ DRR

DRR Framework

📄️ Dueling DQN

A Dueling DQN agent explicitly estimates two quantities through a modified network architecture:

📄️ FFM

FFM stands for Field-aware Factorization Machines. In the official FFM paper, it is empirically proven that for large, sparse datasets with many categorical features, FFM performs better. Conversely, for small and dense datasets or numerical datasets, FFM may not be as effective as FM. FFM is also prone to overfitting on the training dataset, hence one should use a standalone validation set and use early stopping when the loss increases.

📄️ FGNN

Ruihong Qiu, Jingjing Li, Zi Huang and Hongzhi Yin, “Rethinking the Item Order in Session-based Recommendation with Graph Neural Networks”. CIKM, 2019.

📄️ FM

Factorization Machines (FMs) are a supervised learning approach that enhances the linear regression model by incorporating the second-order feature interactions. Factorization Machine type algorithms are a combination of linear regression and matrix factorization, the cool idea behind this type of algorithm is it aims model interactions between features (a.k.a attributes, explanatory variables) using factorized parameters. By doing so it has the ability to estimate all interactions between features even with extremely sparse data.

📄️ GAT

GAT stands for Graph Attention Networks. This is a special GNN model that addresses several key challenges of spectral models, such as poor ability of generalization from a specific graph structure to another and sophisticated computation of matrix inverse. GAT utilizes attention mechanisms to aggregate neighborhood features (embeddings) by specifying different weights to different nodes.

📄️ GC-SAN

GC-SAN stands for Graph contextualized self-attention.

📄️ GCE-GNN

GCE-GNN stands for Global Context Enhanced Graph Neural Networks. It exploit item transitions over all sessions in a more subtle manner for better inferring the user preference of the current session.

📄️ GLMix

Generalized Linear Mixed Effects model (GLMix) decomposes a personalized recommender system into 2 submodels we first train the fixed effects model, and then train random effects models on the residuals after scoring the fixed effects model, and go back to fixed effects model training again until convergence (Zhang et al., 2016).

📄️ GRU4Rec

It uses session-parallel mini-batch approach where we first create an order for the sessions and then, we use the first event of the first X sessions to form the input of the first mini-batch (the desired output is the second events of our active sessions). The second mini-batch is formed from the second events and so on. If any of the sessions end, the next available session is put in its place. Sessions are assumed to be independent, thus we reset the appropriate hidden state when this switch occurs.

📄️ HMLET

HMLET stands for Hybrid Method of Linear and nonlinEar collaborative filTering (HMLET, pronounced as Hamlet). It is a GCN-based CF method.

📄️ IncCTR

Recently, various deep CTR models are proposed, such as DeepFM, Wide & Deep, PIN, DIN, and DIEN. Generally, deep CTR models include three parts: embedding layer, interaction layer, and prediction layer.

📄️ IPW

Inverse Propensity Weighting

📄️ ItemPop

Itempop is a naïve baseline that simply ranks items according to overall train-set popularity. Note that this method is unaffected by the user for which items are being recommended, and has the same global ranking of all items

📄️ KHGT

Knowledge-Enhanced Hierarchical Graph Transformer Network for Multi-Behavior Recommendation

📄️ LESSR

Tianwen Chen and Raymond Chi-Wing Wong, “LESSR: Handling Information Loss of Graph Neural Networks for Session-based Recommendation”. KDD, 2020.

📄️ LightFM WARP

LightFM is probably the only recommender package implementing the WARP (Weighted Approximate-Rank Pairwise) loss for implicit feedback learning-to-rank. Generally, it performs better than the more popular BPR (Bayesian Personalised Ranking) loss --- often by a large margin.

📄️ LightGCN

GCN is a representative model of graph neural networks that applies message passing to aggregate neighborhood information. The message passing layer with self-loops is defined as follows:

📄️ LINE

Large-scale Information Network Embedding

📄️ LIRD

Existing reinforcement learning recommender methods also could recommend a list of items. E.g. DQN can calculate Q-values of all recalled items separately, and recommend a list of items with highest Q-values. But these recommendations are similar in Euclidean space and we want to find similarity in associative space. For instance, for a bread 🍞, I want egg 🥚, milk 🥛 in my recommendation list instead of white bread 🍞, brown bread 🥪, bun 🫓 etc.

📄️ Markov Chains

Markov chains, named after Andrey Markov, are mathematical systems that hop from one "state" (a situation or set of values) to another. For example, if you made a Markov chain model of a baby's behavior, you might include "playing," "eating", "sleeping," and "crying" as states, which together with other behaviors could form a 'state space': a list of all possible states. In addition, on top of the state space, a Markov chain tells you the probability of hopping, or "transitioning," from one state to any other state---e.g., the chance that a baby currently playing will fall asleep in the next five minutes without crying first.

📄️ MATN

Multiplex Behavioral Relation Learning for Recommendation via Memory Augmented Transformer Network.

📄️ MB-GMN

MB-GMN stands for Multi-behavior pattern modeling with meta-knowledge learner.

📄️ MDP

The Markov decision process (MDP), a reinforcement learning (RL) algorithm, perfectly illustrates how machines have become intelligent in their own unique way. Humans build their decision process on experience. MDPs are memoryless. Humans use logic and reasoning to think problems through. MDPs apply random decisions 100% of the time. Humans think in words, labeling everything they perceive. MDPs have an unsupervised approach that uses no labels or training data. MDPs boost the machine thought process of self-driving cars (SDCs), translation tools, scheduling software, and more. This memoryless, random, and unlabeled machine thought process marks a historical change in the way a former human problem was solved.

📄️ MF

Matrix Factorization is an iterative approach of SVD called Regularized SVD. It uses the gradient-descent method to estimate the resulting matrices. The obtained model will not be a true SVD of the rating-matrix, as the component matrices are no longer orthogonal, but tends to be more accurate at predicting unseen preferences than the standard SVD [Ekstrand et al. 2011].

📄️ MIAN

MIAN stands for Multi-Interactive Attention Network. It aggregate multiple information, and gain latent representations through interactions between candidate items and other fine-grained features.

📄️ MMoE

MMoE stands for Multi-gate Mixture-of-Experts.

📄️ MPNN

Message Passing Neural Networks

📄️ NeuMF

NMF Leverages the representation power of deep neural-networks to capture nonlinear correlations between user and item embeddings. Formally, the rating/relevance for user 𝑢 and item 𝑖 is modeled as $\hati^u = \alpha + \betau + \betai + f(\gammau || \gammai || \gammau \cdot \gammai)$ where $\gammau , \gamma_i \in \mathbb{R}^d$, ‘||’ represents the concatenation operation, and $f: \mathbb{R}^{3d} \rightarrow \mathbb{R}$ represents an arbitrarily complex neural network.

📄️ NGCF

NGCF stands for Neural Graph Collaborative Filtering. This GNN-based approach follows basic operations inherited from the standard GCN to explore the high-order connectivity information. More specifically, NGCF stacks embedding layers and concatenates embeddings obtained in all layers to constitute the final embeddings.

📄️ NMRN

NMRN is a streaming recommender model based on neural memory networks with external memories to capture and store both long-term stable interests and short-term dynamic interests in a unified way. An adaptive negative sampling framework based on Generative Adversarial Nets (GAN) is developed to optimize our proposed streaming recommender model, which effectively overcomes the limitations of classical negative sampling approaches and improves the effectiveness of the model parameter inference.

📄️ Node2vec

Nodes in networks could be organized based on communities they belong to (i.e., homophily); in other cases, the organization could be based on the structural roles of nodes in the network (i.e., structural equivalence). For instance, in the below figure, we observe nodes $u$ and $s1$ belonging to the same tightly knit community of nodes, while the nodes $u$ and $s6$ in the two distinct communities share the same structural role of a hub node. Real-world networks commonly exhibit a mixture of such equivalences.

📄️ PNN

PNN stands for Product-based Neural Network.

📄️ PPO

The PPO (Proximal Policy Optimization) algorithm was introduced by the OpenAI team in 2017 and quickly became one of the most popular Reinforcement Learning method that pushed all other RL methods at that moment aside. PPO involves collecting a small batch of experiences interacting with the environment and using that batch to update its decision-making policy. Once the policy is updated with that batch, the experiences are thrown away and a newer batch is collected with the newly updated policy. This is the reason why it is an “on-policy learning” approach where the experience samples collected are only useful for updating the current policy.

📄️ Q-learning

Q-learning can be applied to model-free RL problems. It supports off-policy learning and therefore provides a practical solution to problems where available experiences were/are collected using some other policy or by some other agent (even humans).

📄️ Random Walk

The term "random walk" was first mentioned by Karl Pearson in 1905 in a letter to Nature magazine titled The Problem of the Random Walk. Study of random walks date back even further to the Gambler’s ruin problem, where it could be used to show that a gambler would eventually go bankrupt against an opponent with infinite wealth. It’s only in the last couple of decades, however, that researchers have studied them with respect to networks.

📄️ RKMF

A kernel function allows to transform the product of the factor matrices. Kernels like the s-shaped logistic function allow to impose bounds on the prediction (e.g. one to five stars) while still being differentiable.

📄️ SAC

SAS stands for Soft Actor-Critic. It not only boasts of being more sample efficient than traditional RL algorithms but also promises to be robust to brittleness in convergence.

📄️ SARSA

The SARSA algorithm can be applied to model-free control problems and allows us to optimize the value function of an unknown MDP. SARSA is an on-policy temporal difference learning-based control algorithm. The SARSA algorithm can be summarized as follows:

📄️ SASRec

SASRec stands for Self-Attentive Sequential Recommendation. It relies on the sequence modeling capabilities of self-attentive neural networks to predict the occurence of the next item in a user’s consumption sequence. To be precise, given a user 𝑢 and their time-ordered consumption history $S^𝑢 = (S1^u, S2^u, \dots, S_{|S^u|}^𝑢),$ SASRec first applies self-attention on $S^𝑢$ followed by a series of non-linear feed-forward layers to finally obtain the next item likelihood.

📄️ SDNE

Structural Deep Network Embedding

📄️ SGL

SGL is the latest baseline for top-k recommendations. It introduces self-supervised learning into the recommendation system based on the contrastive learning framework. It is implemented on LightGCN and uses a multitask approach that unites the contrastive loss and the BPR loss function. SGL mainly benefits from graph contrastive learning to reinforce user and item representations.

📄️ Shared Bottom

The shared-bottom model is the simplest and most common multi-task learning architecture. The model has a single base (the shared bottom) from which all of the task-specific subnetworks begin from. This means that this single representation is used for all tasks, and there is no way for individual tasks to adjust what information they get out of the shared bottom compared to other tasks.

📄️ SiReN

Existing literature often ignores the negative feedback e.g. dislikes on YouTube videos, and only capture the homophily (or assortativity) patterns by positive feedback. This is a missed opportunity situation. Performance of GNN-based Recommender Systems can be improved by including negative feedbacks. Disassortivity patterns can be learned by negative feedback. LightGCN can capture the assortativity patterns. and the MLP network can capture the disassortivity patterns.

📄️ SLIST

SLIST stands for Session-aware Linear Similarity/Transition. It is built by unifying two linear models - SLIS and SLIT.

📄️ SML

Problem formulation

📄️ SPop

SPop stands for Session-based Popularity. It is a session popularity predictor that gives higher scores to items with higher number of occurrences in the session. Ties are broken up by adding the popularity score of the item.

📄️ SR-GNN

SR-GNN stands for Session-based Recommendation with Graph Neural Networks.

📄️ SR-SAN

SR-SAN stands for Session-based Recommendation with Self-Attention Networks.

📄️ SSE-PT

Temporal information is crucial for recommendation problems because user preferences are naturally dynamic in the real world. Recent advances in deep learning, especially the discovery of various attention mechanisms and newer architectures in addition to widely used RNN and CNN in natural language processing, have allowed for better use of the temporal ordering of items that each user has engaged with. In particular, the SASRec model, inspired by the popular Transformer model in natural languages processing, has achieved state-of-the-art results. However, SASRec, just like the original Transformer model, is inherently an un-personalized model and does not include personalized user embeddings. SSE-PT overcomes this limitation by employing a Personalized Transformer.

📄️ STAMP

/img/content-models-raw-mp2-stamp-untitled.png

📄️ Struc2Vec

Learning Node Representations from Structural Identity

📄️ SVAE

SVAE stands for Sequential Variational Autoencoder.

📄️ TAaMR

Targeted Adversarial Attack against Multimedia Recommender Systems

📄️ TAGNN-PP

TAGNN-PP models item interactions with GNN, and both local and global user interactions with a Transformer.

📄️ TAGNN

TAGNN stands for Target Attentive Graph Neural Network. Session-based recommendations are challenging due to limited user-item interactions. Typical sequential models are not able to capture complex patterns from all previous interactions. SessionGraph (a graph representation of sessions) can capture the complex patterns from all previous interactions.

📄️ TGIN

Jiang et. al., “Triangle Graph Interest Network for Click-through Rate Prediction”. WSDM, 2022.

📄️ VNCF

VNCF stands for Variational Neural Collaborative Filtering.

📄️ VSKNN

VSKNN stands for Vector Multiplication Session-Based kNN. The idea of this variant is to put more emphasis on the more recent events of a session when computing the similarities. Instead of encoding a session as a binary vector, we use real-valued vectors to encode the current session. Only the very last element of the session obtains a value of “1”; the weights of the other elements are determined using a linear decay function that depends on the position of the element within the session, where elements appearing earlier in the session obtain a lower weight. As a result, when using the dot product as a similarity function between the current weight-encoded session and a binary-encoded past session, more emphasis is given to elements that appear later in the sessions.

📄️ Wide and Deep

Wide and Deep Learning Model, proposed by Google, 2016, is a DNN-Linear mixed model, which combines the strength of memorization and generalization. It's useful for generic large-scale regression and classification problems with sparse input features (e.g., categorical features with a large number of possible feature values). It has been used for Google App Store for their app recommendation.

📄️ Word2vec

/img/content-models-raw-mp2-word2vec-untitled.png

📄️ xDeepFM

xDeepFM stands for Extreme Deep Factorization Machines.

CTR Prediction Models

In the early stage of recommendation systems, people spent much time on tedious and onerous feature engineering. At that time, the dimensions of the raw features are relatively small, which makes it possible to implement different combinations of raw features. The newly created features are then fed into a shallow model, such as Logistic Regression (LR) and Gradient Boosting Decision Trees (GBDT) are widely used in CTR prediction. Then, Factorization Machine (FM) transforms the learning of users and items features into a small, shared vector shape. Based on the above method, Field-aware and Field-weighted FM (FFM) further consider the different impact of the fields that a feature belongs to in order to improve the performance of CTR prediction. Along this line, Attentional Factorization Machines (AFM) are proposed to automatically learn weights of deep cross-features and Neural Factorization Machines (NFM) enhances FMs by modelling higher-order and non-linear feature interactions.

Recently, the success of deep neural networks (DNN) in natural language processing and computer vision brings a new direction to the recommendation system. Among them, the Wide & Deep learning introduces deep neural networks to the CTR prediction. It jointly trains a deep neural network along with the traditional wide linear model. The deep neural networks liberated people from feature engineering while generalizing better combinations of the features. Lots of variants of the Wide & Deep learning have been proposed since it revolutionizes the development of the CTR prediction. Deep & Cross network (DCN) replaces the wide linear part with cross-network, which generates explicit feature crossing among low and high level layers. DeepFM combines the power of DNN and factorization machines for feature representation in the recommendation system. Furthermore, xDeepFM extends the work of DNN by proposing a Compressed Interaction Network to enumerate and compress all feature interactions. Overall, the deep models mentioned above all construct a similar model structure by combing the low-order and high-order features, which greatly reduce the effort of feature engineering and improve the performance of CTR prediction.

However, these aforementioned shallow or deep models take statistical and categorical features as input, while discarding the sequential behavior information of users. For example, users may search items at an e-commerce system, then view some items of interest, and these items are likely to be clicked or purchased next time. Since the historical behaviors explicitly indicate the preference of the users, it has gained much more attention in the recommendation systems. Among them, DIN proposes a local activation unit that learns the dynamic user interests from the sequential behavior features. DIEN designs an interest evolving layer an attentional update gate to model the dependency between sequential behaviors. The researches above realized the importance of user’s historical behaviors. Unfortunately, they just project other information (i.e., user-specific and context) into one vector and did not pay equal attention to the interactions between the candidate item and fine-grained information, while modeling this interaction has shown extensively progress in many tasks, such as search recommendations and knowledge distillation.

Different from all previous methods, MIAN can explore the sequential behavior and other fine-grained information simultaneously. Specifically, compared to shallow and deep models, MIAN has a remarkable ability to encode user’s preference through sequential behavior. Compared to sequential models, MIAN can model fine-grained feature interactions better when historical behavior is not enough or representative.

Model	Paper	Publication	Type	Description
LR	Predicting Clicks: Estimating the Click-Through Rate for New Ads	WWW'07	Shallow	Logistic regression (LR) is a simple baseline model for CTR prediction. With the online learning algorithm, FTRL, proposed by Google, LR has been widely adopted in industry. It’s a widely used baseline and applies linear transformation to model the relationship of all the features.
FM	Factorization Machines	ICDM'10	Shallow	While LR fails to capture non-linear feature interactions, Rendle et al. propose factorization machine (FM) that embeds features into dense vectors and models pairwise feature interactions as inner products of the corresponding embedding vectors. Notably, FM also has a linear time complexity in terms of the number of features.
CCPM	A Convolutional Click Prediction Model	CIKM'15	Deep	CCPM reports the first attempt to use convolution for CTR prediction, where feature embeddings are aggregated hierarchically through convolution networks.
FFM	Field-aware Factorization Machines for CTR Prediction	RecSys'16	Shallow	Field-aware factorization machine (FFM) is an extension of FM that considers field information for feature interactions. It was a winner model in several Kaggle contests on CTR prediction.
YoutubeDNN	Deep Neural Networks for YouTube Recommendations	RecSys'16	Deep	DNN is a straightforward deep model, which applies a fully-connected network (termed DNN) after the concatenation of feature embeddings for CTR prediction.
Wide&Deep	Wide & Deep Learning for Recommender Systems	DLRS'16	Deep	Wide&Deep is a general learning framework proposed by Google that combines a wide (or shallow) network and deep network to achieve the advantages of both. It jointly trains a linear model and a deep MLP model to the CTR prediction.
IPNN	Product-based Neural Networks for User Response Prediction	ICDM'16	Deep	PNN is a product-based network that feeds the inner (or outer) products of features embeddings as the input of DNN. Due to the huge memory requirement of pairwise outer products, we use the inner product version, IPNN.
DeepCross	Deep Crossing: Web-Scale Modeling without Manually Crafted Combinatorial Features	KDD'16	Deep	Inspired by residual networks, deep crossing to add residual connections between layers of DNNs. DCN is proposed to handle a set of sparse and dense features, and learn cross high-order features jointly with traditional deep MLP.
HOFM	Higher-Order Factorization Machines	NIPS'16	Shallow	Since FM only captures second-order feature interactions, HOFM aims to extend FM to higher-order factorization machines. However, it results in exponential feature combinations that consume huge memory and take a long running time.
DeepFM	DeepFM: A Factorization-Machine based Neural Network for CTR Prediction	IJCAI'17	Deep	DeepFM is an extension of Wide&Deep that substitutes LR with FM to explicitly model second-order feature interactions. It combines the explicit high-order interaction module with deep MLP module and traditional FM module, and requires no manual feature engineering.
NFM	Neural Factorization Machines for Sparse Predictive Analytics	SIGIR'17	Deep	Similar to PNN, NFM proposes a Bi-interaction layer that pools the pairwise feature interactions to a vector and then feed it to a DNN for CTR prediction.
AFM	Attentional Factorization Machines: Learning the Weight of Feature Interactions via Attention Networks	IJCAI'17	Deep	Instead of treating all feature interactions equally as in FM, AFM learns the weights of feature interactions via attentional networks. Different from FwFM, AFM adjusts the weights dynamically according to the input data sample.
DCN	Deep & Cross Network for Ad Click Predictions	ADKDD'17	Deep	In DCN, a cross network is proposed to perform high-order feature interactions in an explicit way. In addition, it also integrates a DNN network following the Wide&Deep framework.
FwFM	Field-weighted Factorization Machines for Click-Through Rate Prediction in Display Advertising	WWW'18	Shallow	It considers field-wise weights of features interactions. Compared with FFM, it reports comparable performance but uses much fewer model parameters.
xDeepFM	xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems	KDD'18	xDeepFM	While high-order feature interactions modeled by DCN are bit-wise, xDeepFM proposes to capture high-order feature interactions in a vector-wise way via a compressed interaction network (CIN). It uses Compressed Interaction Network to enumerate and compress all feature interactions, for modeling an explicit order of interactions.
DIN	Deep Interest Network for Click-Through Rate Prediction	KDD'18	Deep	It’s an early work exploits users’ historical behaviors and uses the attention mechanism to activate user behaviors in which the user be interested in different items.
FiGNN	FiGNN: Modeling Feature Interactions via Graph Neural Networks for CTR Prediction	CIKM'19	Deep	FiGNN leverages the message passing mechanism of graph neural networks to learn high-order features interactions.
AutoInt/AutoInt+	AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks	CIKM'19	Deep	AutoInt leverages self-attention networks to learn high-order features interactions. AutoInt+ integrates AutoInt with a DNN network.
FiBiNET	FiBiNET: Combining Feature Importance and Bilinear feature Interaction for Click-Through Rate Prediction	RecSys'19	Deep	FiBiNET leverages squeeze-excitation network to capture important features, and proposes bilinear interactions to enhance feature interactions.
FGCNN	Feature Generation by Convolutional Neural Network for Click-Through Rate Prediction	WWW'19	Deep	FGCNN applies convolution networks and recombination layers to generate additional combinatorial features to enrich existing feature representations.
HFM/HFM+	Holographic Factorization Machines for Recommendation	AAAI'19	Deep	HFM proposes holographic representation and computes compressed outer products via circular convolution to model pairwise feature interactions. HFM+ further integrates a DNN network with HFM.
ONN	Operation-aware Neural Networks for User Response Prediction	Neural Networks'20	Deep	ONN (a.k.a., NFFM) is a model built on FFM. It feeds the interaction outputs from FFM to a DNN network for CTR prediction.
AFN/AFN+	Adaptive Factorization Network: Learning Adaptive-Order Feature Interactions	AAAI'20	Deep	AFN applies logarithmic transformation layers to learn adaptive-order feature interactions. AFN+ further integrates AFN with a DNN network.
LorentzFM	Learning Feature Interactions with Lorentzian Factorization	AAAI'20	Shallow	LorentzFM embed features into a hyperbolic space and model feature interactions via triangle inequality of Lorentz distance.
InterHAt	Interpretable Click-through Rate Prediction through Hierarchical Attention	WSDM'20	Deep	InterHAt employs hierarchical attention networks to model high-order feature interactions in an efficient manner.
FLEN	FLEN: Leveraging Field for Scalable CTR Prediction	DLP-KDD'20	Deep	FLEN: Leveraging Field for Scalable CTR Prediction
FmFM	FM^2: Field-matrixed Factorization Machines for Recommender Systems	WWW'21	Deep	FM^2: Field-matrixed Factorization Machines for Recommender Systems