flexparser 0.1

flexparser

Why writing another parser? I have asked myself the same question while working in this project. It is clear that there are excellent parsers out there but I wanted to experiment with another way of writing them.

The idea is quite simple. You write a class for every type of content (called here ParsedStatement) you need to parse. Each class should have a from_string constructor. We used extensively the typing module to make the output structure easy to use and less error prone.

For example:

from dataclasses import dataclass

import flexparser as fp

@dataclass(frozen=True)
class Assigment(fp.ParsedStatement):
    """Parses the following `this <- other`
    """

    lhs: str
    rhs: str

    @classmethod
    def from_string(cls, s):
        lhs, rhs = s.split("<-")
        return cls(lhs.strip(), rhs.strip())

(using a frozen dataclass is not necessary but I found it very useful)

In certain cases you might want to signal the parser that his class is not appropriate to parse the statement.

@dataclass(frozen=True)
class Assigment(fp.ParsedStatement):
    """Parses the following `this <- other`
    """

    lhs: str
    rhs: str

    @classmethod
    def from_string(cls, s):
        if "<-" not in s:
            # This means: I do not know how to parse it
            # try with another ParsedStatement class.
            return None
        lhs, rhs = s.split("<-")
        return cls(lhs.strip(), rhs.strip())

You might also want to indicate that this is the right ParsedStatement but something is not right:

@dataclass(frozen=True)
class InvalidIdentifier(fp.ParsingError):
    value: str


@dataclass(frozen=True)
class Assigment(fp.ParsedStatement):
    """Parses the following `this <- other`
    """

    lhs: str
    rhs: str

    @classmethod
    def from_string(cls, s):
        if "<-" not in s:
            # This means: I do not know how to parse it
            # try with another ParsedStatement class.
            return None
        lhs, rhs = (p.strip() for p in s.split("<-"))

        if not str.isidentifier(lhs):
            return InvalidIdentifier(lhs)

        return cls(lhs, rhs)

Put this into source.txt

one <- other
2two <- new
three <- newvalue
one == three

and then run the following code:

parsed = fp.parse("source.txt", Assigment)
for el in parsed.iter_statements():
    print(repr(el))

will produce the following output:

BOS(lineno=0, colno=0)
Assigment(lineno=1, colno=0, lhs='one', rhs='other')
InvalidIdentifier(lineno=2, colno=0, origin='', value='2two')
Assigment(lineno=3, colno=0, lhs='three', rhs='newvalue')
UnknownStatement(lineno=4, colno=0, origin='', statement='one == three')
EOS(lineno=-1, colno=-1)

The result is a collection of ParsedStatement or ParsingError (flanked by BOS and EOS indicating beginning and ending of stream respectively). Notice that there are two correctly parsed statements (Assigment), one error found (InvalidIdentifier) and one unknown (UnknownStatement).

Cool, right? Just writing a from_string method that outputs a datastructure produces a usable structure of parsed objects.

Now what? Let’s say we want to support equality comparison. Simply do:

@dataclass(frozen=True)
class EqualityComparison(fp.ParsedStatement):
    """Parses the following `this == other`
    """

    lhs: str
    rhs: str

    @classmethod
    def from_string(cls, s):
        if "==" not in s:
            return None
        lhs, rhs = (p.strip() for p in s.split("=="))

        return cls(lhs, rhs)

parsed = fp.parse("source.txt", (Assigment, Equality))
for el in parsed.iter_statements():
    print(repr(el))

and run it again:

BOS(lineno=0, colno=0)
Assigment(lineno=1, colno=0, lhs='one', rhs='other')
InvalidIdentifier(lineno=2, colno=0, origin='', value='2two')
Assigment(lineno=3, colno=0, lhs='three', rhs='newvalue')
EqualityComparison(lineno=4, colno=0,  lhs='one', rhs='three')
EOS(lineno=-1, colno=-1)

You need to group certain statements together: welcome to Block This construct allows you to group

class Begin(fp.ParsedStatement):

    @classmethod
    def from_string(cls, s):
        if s == "begin":
            return cls()

        return None

class End(fp.ParsedStatement):

    @classmethod
    def from_string(cls, s):
        if s == "end":
            return cls()

        return None

AssigmentBlock = fp.Block.build(
    Begin,
    (Assigment, ),
    End,
)

parsed = fp.parse("source.txt", (AssigmentBlock, Equality))

Run the code:

BOS(lineno=0, colno=0)
UnknownStatement(lineno=1, colno=0, origin='', statement='one <- other')
UnknownStatement(lineno=2, colno=0, origin='', statement='2two <- new')
UnknownStatement(lineno=3, colno=0, origin='', statement='three <- newvalue')
Equality(lineno=4, colno=0, lhs='one', rhs='three')
EOS(lineno=-1, colno=-1)

Notice that there are a lot of UnknownStatement now, because we instructed the parser to only look for assignment within a block. So change your text file to:

begin
one <- other
2two <- new
three <- newvalue
end
one == three

and try again:

BOS(lineno=0, colno=0)
Begin(lineno=1, colno=0)
Assigment(lineno=2, colno=0, lhs='one', rhs='other')
InvalidIdentifier(lineno=3, colno=0, origin='', value='2two')
Assigment(lineno=4, colno=0, lhs='three', rhs='newvalue')
End(lineno=5, colno=0)
Equality(lineno=6, colno=0, lhs='one', rhs='three')
EOS(lineno=-1, colno=-1)

Until now we have used parsed.iter_statements to iterate over all parsed statements. But let’s look inside parsed, an object of ParsedProject type. It is a thin wrapper over a dictionary mapping files to parsed content. Because we have provided a single file and this does not contain a link another, our parsed object contains a single element. The key is something like (None, 'source.txt') indicating that the file ‘source.txt’ was loaded from the root location (None). The content is a ParsedSourceFile object with the following attributes:

• filename: full path of the source file

• mtime: modification file of the source file

• content_hash: sha1 hash of the pickled content (this is currently not the same as hashing the file)

• config: extra parameters that can be given to the parser (see below).

parse.<locals>.CustomRootBlock(
    opening=BOS(lineno=0, colno=0),
    body=(
        Block.subclass_with.<locals>.CustomBlock(
            opening=Begin(lineno=1, colno=0),
            body=(
                Assigment(lineno=2, colno=0, lhs='one', rhs='other'),
                InvalidIdentifier(lineno=3, colno=0, origin='', value='2two'),
                Assigment(lineno=4, colno=0, lhs='three', rhs='newvalue')
            ),
            closing=End(lineno=5, colno=0)
          ),
        Equality(lineno=6, colno=0, lhs='one', rhs='three')
    ),
    closing=EOS(lineno=-1, colno=-1)
)

A few things to notice:

1. We were using a block before without knowing. The RootBlock is a special type of Block that starts and ends automatically with the file.

2. opening, body, closing are automatically annotated with the possible ParsedStatement (plus ParsingError), therefore autocompletes works in most IDEs.

3. The same is true for the defined ParsedStatement (we have use dataclass for a reason). This makes using the actual result of the parsing a charm!.

4. That annoying subclass_with.<locals> is because we have built a class on the fly when we used Block.subclass_with. You can get rid of it (which is actually useful for pickling) by explicit subclassing Block in your code (see below).

parsed = fp.parse(["source.txt", "other_source.txt"], , (AssigmentBlock, Equality))

@dataclass(frozen=True)
class Include(fp.IncludeStatement):
    """A naive implementation of #include "file"
    """

    value: str

    @classmethod
    def from_string(cls, s):
        if s.startwith("#include "):
            return None

        value = s[len("#include "):].strip().strip('"')

        return cls(value)

    @propery
    def target(self):
        return self.value

## This will work as a single statement
# This will work as a single statement #
# This will work as # a single statement #
a = 3 # this will produce two statements (a=3, and the rest)

class AssigmentBlock:

    opening: fp.Single[Begin]
    body: fp.Multi[Assigment]
    closing: fp.Single[End]

class EntryBlock(fp.RootBlock):

    body: fp.Multi[typing.Union[AssigmentBlock, Equality]]

parsed = fp.parse("source.txt", EntryBlock)