Eldritch-Trace DSL

Eldritch-Trace is a Domain Specific Language (DSL) and semantic model, which is used to filter sequences of events whose tag contains a u32 encoded type (primarily used with aetherus-events)

Eldritch-Trace DSL defines the clearly what the patterns and sequences of interest should be matched to, then the semantic model runs something akin to a parses combinator in order to find all sequences of events that satisfy the conditions.

API Docs

In addition to this book, you may also wish to read [the API documentation](TOOD: Add link to docs generated in github pages or crates.io)

License

Eldritch-Trace DSL is pre-emptively licensed as MIT, but might decide to change that.

Sample

Here is an example script used to filter the events for the scene described in the diagram below.

Getting Started

CLI

Prebuilt

TODO:

Build release in CI/CD for Linux/Mac/Windows and make it available to download for users to make use of right away

Build from source

Clone the repository
Build with cargo build --release

Usage

Run with ./target/release/eldritch --spec spec/encoding_spec.md <my_script>.et

This will generate for each rule:

graphviz .dot diagram of each one of the rules described, showing all possible histories. The files can be found in the same directory as the ledger path or in the explicit output directory.
serialised filtered signals in the same directory as the input signals stream

Library

This can also be used as a library, which exposes a FFI compatible with C-ABI, such that it can be called from most languages. The input will still be a path to the script or a string input with the script content.

TODO:

Expose the bindgen version and use it with Python/Julia

User Guide

Here is an example script used to filter the events for the scene described in the diagram below.

Imaging of target through a translucent plate

# Define the serialised Ledger and Signals collected
# ============================================================
ledger  = "../../aetherus-scene/underwater-paper/out/simulation_ledger.json"
signals = "../../aetherus-scene/underwater-paper/out/multispectral/photon_collector_spad_sensor.csv"

# Define material and surface identifiers we want to match for
# ============================================================

src water_id = any[Mat("seawater"), MatSurf("Water:Water_material")]
src glass_id = any[Mat("glass"), any[MatSurf("Tank:Tank_material"), MatSurf("Tank-Water:Tank_material")]]
src toy_id   = Surf("TargetToy")
src tube_id  = Surf("TargetTube")
src air_id   = Mat(0)

# Define encoding patterns to match for
# =====================================

pattern water_scatter     = MCRT | Material  | Elastic | X            | water_id
pattern water_backscatter = MCRT | Material  | Elastic | X | Backward | water_id
pattern glass_interf      = MCRT | Interface | X                      | glass_id

pattern toy_surf          = MCRT | Reflector | X                      | toy_id
pattern tube_surf         = MCRT | Reflector | X                      | tube_id


# Define sequences of events in the photon history to search for
# ==============================================================

sequence toy_detect = seq[
    Emission | X | Light(0),
    * X,
    toy_surf,
    * X,
    Detection | X | Detector(0),
]

sequence tube_detect = seq[
    Emission | X | Light(0),
    * X,
    tube_surf,
    * X,
    Detection | X | Detector(0),
]

sequence seq_water_backscatter = seq[
    Emission | X | Light(0),
    * X,
    water_scatter,
    * X,
    Detection | X | Detector(0),
]

# Note how rule tries to validate all conditions enumerated
rule backscatter = {
    ! any[tube_surf, toy_surf],
    seq[
        Emission | X | Light(0),
        * X,
        + water_scatter,
        * X,
        Detection | X | Detector(0),
    ],
}

rule toy_or_tube_detect = {
    any[toy_surf, tube_surf],
    seq[
        Emission | X | Light(0),
        * X,
        + water_scatter,
        * X,
        Detection | X | Detector(0),
    ],
}

The ledger serialised to a JSON file is as follows:

{
  "grps": {},
  "src_map": {
    "Mat(0)": [ { "Mat": "air" } ],
    "Mat(2)": [ { "Mat": "translucent_pla" } ],
    "Mat(1)": [ { "Mat": "mist" } ],
    "Surf(0)": [ { "Surf": "Background" } ],
    "Surf(1)": [ { "Surf": "Target" } ],
    "MatSurf(65535)": [ { "MatSurf": "TranslucentPlate:TranslucentPLA" } ]
  },
  "start_events": [ { "seq_id": 0, "event": "0x1010000" } ],
  "next_mat_id": 3,
  "next_surf_id": 2,
  "next_matsurf_id": 65534,
  "next_light_id": 0,
  "next": {
    "0": { "0x01010000": 1 },
    "1": { "0x0300FFFF": 9, "0x0301FFFF": 2, "0x03A00001": 70268 },
    "2": { "0x03A0FFFF": 3 },
    "3": { "0x0300FFFF": 391, "0x03010001": 4, "0x03A0FFFF": 5 },
    "4": { "0x05000000": 50 },
    // ...more entries...
  },
  "prev": {
    "1": "0, 0x01010000",
    "2": "1, 0x0301FFFF",
    "3": "2, 0x03A0FFFF",
    "4": "3, 0x03010001",
    "5": "3, 0x03A0FFFF",
    // ...more entries...
  },
  "next_seq_id": 3747279,
}

Language Reference Manual

Bachus-Naur Form compatible with ANTLR parser generator:

Declarations = { Declaration };

Declaration = SrcDecl
            | SeqDecl
            | PatternDecl
            | RuleDecl
            ;
Comment     = "#" { !"#" anycharacter } ;

SrcDecl     = "src" Ident "=" SrcIdValue ;
PatternDecl = "pattern" Ident "=" Pattern ;
SeqDecl     = "sequence" Ident "=" Seq ;
RuleDecl    = "rule" Ident "=" "{" ConditionItems "}" ;

SrcIdValue = SrcId | SrcIdAny | Ident | "X" ;

SrcId      = SrcIdName | SrcIdVal ;
SrcIdName  = SrcIdType "(" String ")" ;
SrcIdVal   = SrcIdType "(" Number ")" ;
SrcIdType  = "Mat" | "Surf" | "MatSurf" | "Light" | "Detector" ;

SrcIdAny   = "any" "[" SrcIdItems "]" ;
SrcIdItems = SrcIdItem { "," SrcIdItem } ;
SrcIdItem  = SrcId | Ident ;

Pattern    = Fields "|" SrcIdValue ;
Fields     = FieldExpr { "|" FieldExpr } ;
FieldExpr  = FieldId | "X" ;

InlinePattern       = Pattern | Ident ;
InlinePatternItems  = InlinePattern { "," InlinePattern } ;
PatternAny          = "any" "[" InlinePatternItems "]" ;
PatternSet          = InlinePattern | PatternAny | "X" ;
PredicatedPattern   = "!" PatternSet | PatternSet ;

Repetition = "*"
           | "+"
           | "?"
           | "{" [ Number ] [ "," [ Number ] ] "}"
           ;

RepetitionPattern = Repetition PredicatedPattern
                   | PredicatedPattern ;

PatternItems = RepetitionPattern { "," RepetitionPattern } ;

Seq = "seq" "[" PatternItems "]" ;

Condition      = RepetitionPattern | Seq ;
ConditionItems = Condition { "," Condition } ;

FieldId   = Ident ;
Ident     = letter { letter | digit | "_" | "." } ;
Decimal   = digit { digit } ;
Hex       = "0x" hex_digit { hex_digit } ;
Number    = Decimal | Hex ;
String    = "\"" { Character } "\"" ;
Character = character
            | ["\\"] anycharacter ;

Example:

# Define the serialised Ledger and Signals collected
# ============================================================
ledger  = "../../aetherus-scene/underwater-paper/out/simulation_ledger.json"
signals = "../../aetherus-scene/underwater-paper/out/multispectral/photon_collector_spad_sensor.csv"

# Define material and surface identifiers we want to match for
# ============================================================

src water_id = any[Mat("seawater"), MatSurf("Water:Water_material")]
src glass_id = any[Mat("glass"), any[MatSurf("Tank:Tank_material"), MatSurf("Tank-Water:Tank_material")]]
src toy_id   = Surf("TargetToy")
src tube_id  = Surf("TargetTube")
src air_id   = Mat(0)

# Define encoding patterns to match for
# =====================================

pattern water_scatter     = MCRT | Material  | Elastic | X            | water_id
pattern water_backscatter = MCRT | Material  | Elastic | X | Backward | water_id
pattern glass_interf      = MCRT | Interface | X                      | glass_id

pattern toy_surf          = MCRT | Reflector | X                      | toy_id
pattern tube_surf         = MCRT | Reflector | X                      | tube_id


# Define sequences of events in the photon history to search for
# ==============================================================
sequence seq_water_backscatter = seq[
    Emission | X | Light(0),
    * X,
    water_scatter,
    * X,
    Detection | X | Detector(0),
]

# Note how rule tries to validate all conditions enumerated
rule backscatter = {
    ! any[tube_surf, toy_surf],
    seq[
        Emission | X | Light(0),
        * X,
        + water_scatter,
        * X,
        Detection | X | Detector(0),
    ],
}

rule toy_or_tube_detect = {
    any[toy_surf, tube_surf],
    seq[
        Emission | X | Light(0),
        * X,
        + water_scatter,
        * X,
        Detection | X | Detector(0),
    ],
}

Predicates

The predicates are identical to a parser generator or BNF description, since our sequence matching has very much in common to the logic behind a parser combinator. The predicates are inspired from pest.rs with the change that the operator shows as a prefix in our grammar.

Negation: `!`

The ! unnary/monadic operator signifies that the pattern described should not be matched, not progress the event chains and move on to check next pattern.

Repetition

Repetition syntax	Meaning
`? <pattern>`	Optionally match pattern
`\* <pattern>`	Match pattern zero or more times
`+ <pattern>`	Match pattern one or more times
`{n} <pattern>`	Match pattern exactly `n` times
`{m,n} <pattern>`	Match pattern between `m` and `n` times
`{,n} <pattern>`	Match pattern at most `n` times
`{n,} <pattern>`	Match pattern at least `n` times

Sets

any
seq
perm

Any: `any`

any is effectively an OR of a list of src fields or patterns listed:

any[Mat(0), Mat("water"), MatSurf("water_interface")]
any[pattern_id_0, MCRT | Material | X | SrcId(0)]

Sequence: `seq`

seq is the set constructor of a sequence, describing a series of pattern that must match in order. Each of the patterns might be predicated by a negative or repetition operator.

Example:

seq[
    Emission | X | Light(0),
    * X,
    target_pattern,
    * X,
    Detection | X | Detector(0),
]

Permutation: `perm`

![WARN] This functionality is not covered yet, but it should provide the possibility to be consumed only within a sequence, such that some events can be described out-of-order.

Source Declaration: `src`

src water_src = any[Mat(0), Mat("water"), MatSurf("water_interf")]

Pattern Declaration

Patterns are described as concatenation of field specifiers, that are described in the encoding specification used, don’t care (X) fields when they are required to be masked out and a source field that can be explicitly constructed, or used from a previous src declaration.

pattern water_scatter = MCRT | Material | Elastic | X | Mat("water")
pattern water_scatter = MCRT | Material | Elastic | X | any[Mat(0), Mat("water")]
pattern water_scatter = MCRT | Material | Elastic | X | water_src

Sequence

A sequence describes the order of event patterns to be matched.

sequence seq_water_backscatter = seq[
    Emission | X | Light(0),
    * X,
    water_scatter,
    * X,
    Detection | X | Detector(0),
]

Rule: `rule`

A rule describes the set of conditions to be met by a terminal UID in the chain of events.

Conditions can be:

Patterns, predicated or not
Set of patterns i.e. any[pattern_1, pattern_2, ...]
Sequence to be matched

Bot patterns and sequences can be written inline the rule declarations or used as identifier of previously declared values.

rule toy_or_tube_detect = {
    any[toy_surf, tube_surf], # Set match
    ! boundary_surf,          # Predicated indentifier pattern
    seq[
        Emission | X | Light(0),
        * X,
        + water_scatter,      # Repetition of identifier inside sequence
        * X,
        Detection | X | Detector(0),
    ],
    my_other_sequence,
}

Analysis Process

TODO:

Here shown an example of how the output can be used.

FAQ

Placeholder for future questions, used to aid other users and provide a track of what’s unclear inside the docs/book.

Keyboard shortcuts

Eldritch-Trace Documentation