cyberzhg / keras-pos-embd

Position embedding layers in Keras

Keras Position Embedding

[中文|English]

Position embedding layers in Keras.

Install

pip install keras-pos-embd

Usage

Trainable Embedding

from tensorflow import keras
from keras_pos_embd import PositionEmbedding

model = keras.models.Sequential()
model.add(PositionEmbedding(
    input_shape=(None,),
    input_dim=10,     # The maximum absolute value of positions.
    output_dim=2,     # The dimension of embeddings.
    mask_zero=10000,  # The index that presents padding (because `0` will be used in relative positioning).
    mode=PositionEmbedding.MODE_EXPAND,
))
model.compile('adam', 'mse')
model.summary()

Note that you don't need to enable mask_zero if you want to add/concatenate other layers like word embeddings with masks:

from tensorflow import keras
from keras_pos_embd import PositionEmbedding

model = keras.models.Sequential()
model.add(keras.layers.Embedding(
    input_shape=(None,),
    input_dim=10,
    output_dim=5,
    mask_zero=True,
))
model.add(PositionEmbedding(
    input_dim=100,
    output_dim=5,
    mode=PositionEmbedding.MODE_ADD,
))
model.compile('adam', 'mse')
model.summary()

Sin & Cos Embedding

The sine and cosine embedding has no trainable weights. The layer has three modes, it works just like PositionEmbedding in expand mode:

from tensorflow import keras
from keras_pos_embd import TrigPosEmbedding

model = keras.models.Sequential()
model.add(TrigPosEmbedding(
    input_shape=(None,),
    output_dim=30,                      # The dimension of embeddings.
    mode=TrigPosEmbedding.MODE_EXPAND,  # Use `expand` mode
))
model.compile('adam', 'mse')
model.summary()

If you want to add this embedding to existed embedding, then there is no need to add a position input in add mode:

from tensorflow import keras
from keras_pos_embd import TrigPosEmbedding

model = keras.models.Sequential()
model.add(keras.layers.Embedding(
    input_shape=(None,),
    input_dim=10,
    output_dim=5,
    mask_zero=True,
))
model.add(TrigPosEmbedding(
    output_dim=5,
    mode=TrigPosEmbedding.MODE_ADD,
))
model.compile('adam', 'mse')
model.summary()

keras-pos-embd's Issues

Input to the position embedding model

Hi CyberZHG,

I want to use this position embedding in an existing model. Can I add this as a layer to the existing model? Another doubt is how to give the relative postions vector to this embedding model.

Confusion regarding using other embeddings

Note that you don't need to enable mask_zero if you would concatenate other layers like word/Char embeddings with masks
Can you please explain this or give a snippet that illustrates what you mean.... for example, what if I want to user vanilla Keras embedding layer
@CyberZHG

当我使用tensorflow2.1和keras2.3.1时，代码不能正常运行

维度不一致，相加报错

add模式下，多个不同维度的输入，共用一个PositionEmbedding时，将会报错。因为相加时，PositionEmbedding的维度不可能既满足第一个输入的加法，又满足第二个输入的加法。

或许133行可以改为
inputs + pos_embeddings[:,:,:output_dim]
或者
把129行第二个self去掉
但改129行应该会影响其他模式

小白飘过，看大神怎么改法

ValueError: Unknown layer: TrigPosEmbedding

Any idea how to fix it? there was no any problem during the training process but this error came out when i tried to load the model, thx!