Skip to main content

Mbedder: A pytorch powered framework for seemlessly adding contextual text embeddings from pretrained models

Project description


Mbedder is a language framework for adding contextual embeddings of pretrained language models to deep learning models.Mbedder is powered by PyTorch and HuggingFace and requires as less as 1 line of code to add embeddings and works similar to how the Embedding Layer works in PyTorch.

List of supported architectures

  • Bert
  • XLNet
  • Albert
  • TransfoXL
  • DistilBert
  • Roberta
  • XLM
  • XLMRoberta
  • GPT
  • GPT2
  • Flaubert

The pretrained models for the mentioned architecures can be found here.

Features

  • Addition of embeddings with 1 line of code
  • Embeddings can output Sentence as well as Token level embeddings
  • Task specific combination strategies can be applied to hidden states and token embeddings
  • Custom pre-trained hugging face transformer models can be used with Mbedder.

Requirements and Installation

  • PyTorch version >= 1.6.0
  • Python version >= 3.6
  • Transformer >= 3.0.2

Mbedder can be using Pip as follows

pip install Mbedder

Getting Started

A basic example of using a Mbedder Bert embedding is shown below:

import torch
from Mbedder import BertEmbedding

class BertClassifier(torch.nn.Module):
    def __init__(self, num_classes):
        super(BertClassifier, self).__init__()
        self.embedding = BertEmbedding.from_pretrained('bert-base-uncased')
        self.fc = torch.nn.Linear(self.embedding.embedding_size, num_classes)
    
    def forward(self, input_ids, attention_mask):
        x = self.embedding(input_ids, attention_mask, output_token_embeddings=False)
        logits = self.fc(x[0])
        return logits

More advanced examples can be found in the examples folder.

License

Mbedder is MIT-licensed.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

Mbedder-0.0.4.tar.gz (5.9 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page