Pipelines

Pipelines are defined by TypeScript functions that determine how text flows through processing commands.

Basic Pipeline

import { StringEntry, Command } from './.divvun-rt/mod.ts';
import * as hfst from './.divvun-rt/hfst.ts';
import * as cg3 from './.divvun-rt/cg3.ts';

export default function myPipeline(entry: StringEntry): Command {
    let x = hfst.tokenize(entry, { model_path: "tokeniser.pmhfst" });
    x = cg3.vislcg3(x, { model_path: "grammar.bin" });
    return x;
}

This pipeline: 1. Takes string input 2. Tokenizes with HFST 3. Applies CG3 rules 4. Returns result

Multiple Pipelines

Export multiple functions for different use cases:

// Default pipeline
export default function full(entry: StringEntry): Command {
    let x = hfst.tokenize(entry, { model_path: "tokeniser.pmhfst" });
    x = cg3.vislcg3(x, { model_path: "grammar.bin" });
    return x;
}

// Alternative pipeline
export function quick(entry: StringEntry): Command {
    return hfst.tokenize(entry, { model_path: "tokeniser.pmhfst" });
}

Run specific pipeline:

divvun-runtime run --pipeline quick ./pipeline.ts "text"

Function names convert to kebab-case: myPipeline → my-pipeline

Dev Pipelines

Pipelines ending in _dev are development-only:

// Production - must reference files in assets/
export default function prod(entry: StringEntry): Command {
    return hfst.tokenize(entry, { model_path: "tokeniser.pmhfst" });
}

// Dev - can reference files anywhere using @ prefix
export function test_dev(entry: StringEntry): Command {
    return hfst.tokenize(entry, {
        model_path: "@../test-models/tokeniser.pmhfst"
    });
}

@-prefixed paths can be relative to the pipeline.ts file, or be absolute paths.

Path Resolution

Standard paths (both pipeline types):

model_path: "tokeniser.pmhfst"  // → assets/tokeniser.pmhfst

@ prefix paths (dev pipelines only):

model_path: "@../shared/model.hfst"     // Relative to pipeline.ts
model_path: "@/opt/models/model.hfst"   // Absolute path

Project Structure

my-pipeline/
├── pipeline.ts       # Your pipeline definitions
├── assets/          # Production files (included in bundles)
│   └── *.hfst
└── .divvun-rt/     # Generated types

TypeScript Types

Import modules and use type-safe commands:

import { StringEntry, Command } from './.divvun-rt/mod.ts';
import * as hfst from './.divvun-rt/hfst.ts';
import * as cg3 from './.divvun-rt/cg3.ts';
import * as divvun from './.divvun-rt/divvun.ts';
import * as speech from './.divvun-rt/speech.ts';

Types are generated by divvun-runtime sync.

Entry Types

import { StringEntry, BytesEntry } from './.divvun-rt/mod.ts';

// String input (most common)
export function textPipeline(entry: StringEntry): Command { ... }