mertgurkan0 / inttower

Source code of CIKM 2022 and DLP-KDD workshop 2022 Best Paper: IntTower-“ IntTower: the Next Generation of Two-Tower Model for Pre-ranking System”

Description

Official Pytorch Version code of CIKM 2022 Paper "IntTower: the Next Generation of Two-Tower Model for Pre-ranking System"

Other Mindspore Version

Accept by CIKM 2022 and DLP-KDD 2022 workshop best paper

Easy to use

pip install -r requirements.txt
python train_movielens_IntTower.py 

Source code of Paper "IntTower: the Next Generation of Two-Tower Model for Pre-ranking System"

Contents

• Contents
• IntTower Description
• Dataset
• Environment Requirements
• Quick Start
• Performance

IntTower Description

The proposed model, IntTower (short for Interaction enhanced Two-Tower), consists of Light-SE, FE-Block and CIR modules. Specifically, lightweight Light-SE module is used to identify the importance of different features and obtain refined feature representations in each tower. FE-Block module performs fine-grained and early feature interactions to capture the interactive signals between user and item towers explicitly and CIR module leverages a contrastive interaction regularization to further enhance the interactions implicitly.

Paper: Xiangyang Li*, Bo Chen*, Huifeng Guo, Jingjie Li, Chenxu Zhu, Xiang Long, Yichao Wang, Wei Guo, Longxia Mao, Jinxing Liu, Zhenhua Dong, Ruiming Tang. IntTower: the Next Generation of Two-Tower Model for Pre-Ranking System

• Light-SE Module

  class LightSE(nn.Module):
    """LightSELayer used in IntTower.
      Input shape
        - A list of 3D tensor with shape: ``(batch_size,filed_size,embedding_size)``.
      Output shape
        - A list of 3D tensor with shape: ``(batch_size,filed_size,embedding_size)``.
      Arguments
        - **filed_size** : Positive integer, number of feature groups.
      """
  
    def __init__(self, field_size, embedding_size=32):
        super(LightSE, self).__init__()
        self.softmax = nn.Softmax(dim=1)
        self.field_size = field_size
        self.embedding_size = embedding_size
        self.excitation = nn.Sequential(
            nn.Linear(self.field_size, self.field_size)
        )
  
    def forward(self, inputs):
        if len(inputs.shape) != 3:
            raise ValueError(
                "Unexpected inputs dimensions %d, expect to be 3 dimensions" % (len(inputs.shape)))
        Z = torch.mean(inputs, dim=-1, out=None)
        A = self.excitation(Z) #(batch,reduction_size)
        A = self.softmax(A) #(batch,reduction_size)
        out = inputs * torch.unsqueeze(A, dim=2)
        return inputs + out

  • FE Score

    def fe_score(user_rep, item_rep, user_fea_col, item_fea_col, user_embedding_dim, item_embedding_dim):
      score = []
      for i in range(len(user_embedding_dim)):
        user_temp = torch.reshape(user_rep[i], (-1, user_fea_col, user_embedding_dim[i]))
        item_temp = torch.reshape(item_rep[-1], (-1, item_fea_col, item_embedding_dim[i]))
        score.append((user_temp @ item_temp.permute(0, 2, 1)).max(2).values.sum(1))
      score = torch.stack(score).transpose(1, 0)
      return torch.sum(score, 1)

Dataset

• Movie-Lens-1M
• Amazon(electronic)
• Alibaba ads

Environment Requirements

• Hardware（CPU/GPU）
  • Prepare hardware environment with CPU or GPU processor.
• Framework
• Requirements
  • deepctr==0.9.0
  • deepctr_torch==0.2.7
  • deepmatch==0.2.0
  • keras==2.8.0
  • matplotlib==3.5.2
  • numpy==1.21.4
  • pandas==1.4.2
  • pytorch_lightning==1.6.3
  • scikit_learn==1.1.1
  • tensorflow==2.8.0
  • torch==1.10.0
  • torchkeras==3.0.2
  • torchsummary==1.5.1
  • torchvision==0.12.0
  • tqdm==4.51.0
  • xgboost==1.6.1

Quick Start

• running on CPU or GPU

  # run training and evaluation example
  pip install -r requirements.txt
  python train_movielens_IntTower.py

• Compare with other model

  We supply 8 models to compare inttower, you can change your model such as:

  from model.dcn import DCN
  model = DCN(linear_feature_columns, dnn_feature_columns, task='binary', dnn_dropout=dropout,
                    device=device)

• Accelerate

  For online inference , you can only use the last layer FE-Module to compute MaxSim Score.

  Just change preprocessing/utils.py fe_score function

  def fe_score(user_rep, item_rep, user_fea_col, item_fea_col, user_embedding_dim, item_embedding_dim):
    user_embedding, item_embedding  = user_rep[0],item_rep[0]
    user_rep = torch.reshape(user_embedding, (-1, user_fea_col, user_embedding_dim[0]))
    item_rep = torch.reshape(item_embedding, (-1, item_fea_col, item_embedding_dim[0]))
    
    return (user_rep @ item_rep.permute(0, 2, 1)).max(2).values.sum(1)

Performance

cite our work

@inproceedings{10.1145/3511808.3557072,
author = {Li, Xiangyang and Chen, Bo and Guo, Huifeng and Li, Jingjie and Zhu, Chenxu and Long, Xiang and Li, Sujian and Wang, Yichao and Guo, Wei and Mao, Longxia and Liu, Jinxing and Dong, Zhenhua and Tang, Ruiming},
title = {IntTower: The Next Generation of Two-Tower Model for Pre-Ranking System},
year = {2022},
isbn = {9781450392365},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3511808.3557072},
doi = {10.1145/3511808.3557072},
abstract = {Scoring a large number of candidates precisely in several milliseconds is vital for industrial pre-ranking systems. Existing pre-ranking systems primarily adopt the two-tower model since the "user-item decoupling architecture" paradigm is able to balance the efficiency and effectiveness. However, the cost of high efficiency is the neglect of the potential information interaction between user and item towers, hindering the prediction accuracy critically. In this paper, we show it is possible to design a two-tower model that emphasizes both information interactions and inference efficiency. The proposed model, IntTower (short for Interaction enhanced Two-Tower), consists of Light-SE, FE-Block and CIR modules. Specifically, lightweight Light-SE module is used to identify the importance of different features and obtain refined feature representations in each tower. FE-Block module performs fine-grained and early feature interactions to capture the interactive signals between user and item towers explicitly and CIR module leverages a contrastive interaction regularization to further enhance the interactions implicitly. Experimental results on three public datasets show that IntTower outperforms the SOTA pre-ranking models significantly and even achieves comparable performance in comparison with the ranking models. Moreover, we further verify the effectiveness of IntTower on a large-scale advertisement pre-ranking system. The code of IntTower is publicly available https://gitee.com/mindspore/models/tree/master/research/recommend/IntTower.},
booktitle = {Proceedings of the 31st ACM International Conference on Information & Knowledge Management},
pages = {3292–3301},
numpages = {10},
keywords = {pre-ranking system, recommender systems, neural networks},
location = {Atlanta, GA, USA},
series = {CIKM '22}
}