text2array 0.0.1

Sample

Sample is just a Mapping[FieldName, FieldValue], where FieldName = str and FieldValue = Union[float, int, str, Sequence['FieldValue']. It is easiest to use a dict to represent a sample, but you can essentially use any object you like as long as it implements Mapping[FieldName, FieldValue] (which can be ensured by subclassing from this type).

Dataset

There are actually 2 classes for a dataset. Dataset is what you would normally use. It implements Sequence[Sample] so it requires all the samples to fit in the memory. To create a Dataset object, pass an object of type Sequence[Sample] as an argument.

>>> from text2array import Dataset
>>> samples = [
...   {'ws': ['john', 'talks']},
...   {'ws': ['john', 'loves', 'mary']},
...   {'ws': ['mary']}
... ]
>>>
>>> # Create a Dataset
>>> dataset = Dataset(samples)
>>> len(dataset)
3
>>> dataset[1]
{'ws': ['john', 'loves', 'mary']}

If the samples can’t fit in the memory, use StreamDataset instead. It implements Iterable[Sample] and streams the samples one by one, only when iterated over. To instantiate, pass an Iterable[Sample] object.

>>> from text2array import StreamDataset
>>> samples = [
...   {'ws': ['john', 'talks']},
...   {'ws': ['john', 'loves', 'mary']},
...   {'ws': ['mary']}
... ]
>>> class Stream:
...   def __init__(self, seq):
...     self.seq = seq
...   def __iter__(self):
...     return iter(self.seq)
...
>>> dataset = StreamDataset(Stream(samples))  # simulate a stream of samples
>>> list(dataset)
[{'ws': ['john', 'talks']}, {'ws': ['john', 'loves', 'mary']}, {'ws': ['mary']}]

Because StreamDataset is an iterable, you can’t ask for its length nor access by index, but it can be iterated over.

>>> from text2array import Dataset
>>> samples = [
...   {'ws': ['john', 'talks']},
...   {'ws': ['john', 'loves', 'mary']},
...   {'ws': ['mary']}
... ]
>>> dataset = Dataset(samples)
>>> dataset.shuffle_by(lambda s: len(s['ws']))

>>> from text2array import Dataset
>>> samples = [
...   {'ws': ['john', 'talks']},
...   {'ws': ['john', 'loves', 'mary']},
...   {'ws': ['mary']}
... ]
>>> dataset = Dataset(samples)
>>> for batch in dataset.batch(2):
...   print('batch:', list(batch))
...
batch: [{'ws': ['john', 'talks']}, {'ws': ['john', 'loves', 'mary']}]
batch: [{'ws': ['mary']}]

>>> from pprint import pprint
>>> from text2array import Dataset
>>> samples = [
...   {'ws': ['john', 'talks'], 'i': 10, 'label': 'pos'},
...   {'ws': ['john', 'loves', 'mary'], 'i': 20, 'label': 'pos'},
...   {'ws': ['mary'], 'i': 30, 'label': 'neg'}
... ]
>>> dataset = Dataset(samples)
>>> vocab = {
...   'ws': {'john': 0, 'talks': 1, 'loves': 2, 'mary': 3},
...   'i': {10: 5, 20: 10, 30: 15}
... }
>>> dataset.apply_vocab(vocab)
>>> pprint(list(dataset))
[{'i': 5, 'label': 'pos', 'ws': [0, 1]},
 {'i': 10, 'label': 'pos', 'ws': [0, 2, 3]},
 {'i': 15, 'label': 'neg', 'ws': [3]}]

Vocabulary

Creating a vocabulary object from scratch is tedious. So, it’s common to learn the vocabulary from a dataset. The Vocab class can be used for this purpose.

>>> from text2array import Vocab
>>> samples = [
...   {'ws': ['john', 'talks'], 'i': 10, 'label': 'pos'},
...   {'ws': ['john', 'loves', 'mary'], 'i': 20, 'label': 'pos'},
...   {'ws': ['mary'], 'i': 30, 'label': 'neg'}
... ]
>>> vocab = Vocab.from_samples(samples)
>>> vocab.keys()
dict_keys(['ws', 'label'])
>>> dict(vocab['ws'])
{'<pad>': 0, '<unk>': 1, 'john': 2, 'mary': 3}
>>> dict(vocab['label'])
{'<unk>': 0, 'pos': 1}
>>> 'john' in vocab['ws'], 'talks' in vocab['ws']
(True, False)
>>> vocab['ws']['john'], vocab['ws']['talks']
(2, 1)

There are several things to note:

1. Vocabularies are only created for fields which contain str values.

2. Words that occur only once are not included in the vocabulary.

3. Non-sequence fields do not have a padding token in the vocabulary.

4. Out-of-vocabulary words are assigned a single ID for unknown words.

Vocab.from_samples actually accepts an Iterable[Sample], which means a Dataset or a StreamDataset can be passed as well. See the docstring to see other arguments that it accepts to customize vocabulary creation.

Batch

Both batch and batch_exactly methods return Iterator[Batch] where Batch implements Sequence[Sample]. This is true even for StreamDataset. So, although samples may not all fit in the memory, a batch of them should. Given a Batch object, it can be converted into Numpy’s ndarray by to_array method. Normally, you’d want to apply the vocabulary beforehand to ensure all values contain only ints or floats.

>>> from text2array import Dataset, Vocab
>>> samples = [
...   {'ws': ['john', 'talks'], 'i': 10, 'label': 'pos'},
...   {'ws': ['john', 'loves', 'mary'], 'i': 20, 'label': 'pos'},
...   {'ws': ['mary'], 'i': 30, 'label': 'neg'}
... ]
>>> dataset = Dataset(samples)
>>> vocab = Vocab.from_samples(dataset)
>>> dict(vocab['ws'])
{'<pad>': 0, '<unk>': 1, 'john': 2, 'mary': 3}
>>> dict(vocab['label'])
{'<unk>': 0, 'pos': 1}
>>> dataset.apply_vocab(vocab)
>>> batches = dataset.batch(2)
>>> batch = next(batches)
>>> arr = batch.to_array()
>>> arr.keys()
dict_keys(['ws', 'i', 'label'])
>>> arr['ws']
array([[2, 1, 0],
       [2, 1, 3]])
>>> arr['i']
array([10, 20])
>>> arr['label']
array([1, 1])

Note that to_array returns a Mapping[FieldName, np.ndarray] object, and sequential fields are automatically padded. One of the nice things is that the field can be deeply nested and the padding just works!

>>> from pprint import pprint
>>> from text2array import Dataset, Vocab
>>> samples = [
...   {'ws': ['john', 'talks'], 'cs': [list('john'), list('talks')]},
...   {'ws': ['john', 'loves', 'mary'], 'cs': [list('john'), list('loves'), list('mary')]},
...   {'ws': ['mary'], 'cs': [list('mary')]}
... ]
>>> dataset = Dataset(samples)
>>> vocab = Vocab.from_samples(dataset)
>>> dataset.apply_vocab(vocab)
>>> dict(vocab['ws'])
{'<pad>': 0, '<unk>': 1, 'john': 2, 'mary': 3}
>>> pprint(dict(vocab['cs']))
{'<pad>': 0,
 '<unk>': 1,
 'a': 3,
 'h': 5,
 'j': 4,
 'l': 7,
 'm': 9,
 'n': 6,
 'o': 2,
 'r': 10,
 's': 8,
 'y': 11}
>>> batches = dat.batch(2)
>>> batch = next(batches)
>>> arr = batch.to_array()
>>> arr['ws']
array([[2, 1, 0],
       [2, 1, 3]])
>>> arr['cs']
array([[[ 4,  2,  5,  6,  0],
        [ 1,  3,  7,  1,  8],
        [ 0,  0,  0,  0,  0]],

       [[ 4,  2,  5,  6,  0],
        [ 7,  2,  1,  1,  8],
        [ 9,  3, 10, 11,  0]]])

So, you can go crazy and have a field representing a document hierarchically as paragraphs, sentences, words, and characters, and it will be padded correctly.