Enable lookup enrichment

Optional path after Install Padas Motion and a running Motion Engine. Lookup is not required for the sample TCP pipeline.

PDL lookup reads a local replica on the engine. It does not call lookup-server once per event. Keep [lookup].enabled off until lookup-server is enrolled, started, and this engine is cache-bound.

Use Install Padas Motion for UI + engine (same rpm/deb/tgz pattern). Enroll/start pitfalls (TTL, code reuse, --ca-file): padas-motion operator runbook Phases 3–6.

1. Install the nested lookup package

From $PADAS_HOME (usually /opt/padas), install packages/padas-lookup-* with the same format as the engine. Do not use tar --strip-components=1 (root must stay lookup/).

test -x "$PADAS_HOME/lookup/bin/padas-lookup-server"

2. Enroll lookup

Same flags as the engine: --ui-url and --code. There is no --service flag.

In the console, open Settings → Services (/services?service=lookup) and choose the Lookup tab. Create an inventory row (name + lookup-server HTTPS base URL, default port 8998). The console registers a matching S2S client (client_id = inventory id). Use Enroll code (or Rotate & enroll code), then on the lookup host:

"$PADAS_HOME/lookup/bin/padas-lookup-server" enroll \
  --ui-url https://<host>:9000 \
  --code <enrollment-uuid>

Automation without the SPA (admin session cookie) remains available via the embedded AS admin API; prefer Settings → Services (Lookup tab) for day-to-day operators.

Optional --ca-file PATH when the console TLS cert is not trusted by the default store (same enroll pitfalls as the engine: padas-motion operator runbook).

Enroll does not bind REST (8998) or gRPC (50099).

3. Start lookup-server

Same systemd vs foreground pattern as the engine; unit name is padas-lookup.service.

sudo -u padas "$PADAS_HOME/lookup/bin/padas-lookup-server"
curl --insecure https://127.0.0.1:8998/ready

Expect HTTP 200 when the store is open and gRPC is bound.

Optional: create a Lookup reader client

For Prometheus or read-only automation, use Settings → Services (Lookup tab) → Add reader client or Settings → Services (Service clients tab) → Add client → Lookup. The console creates a grant-ready client limited to aud=padas-lookup and lookup:read; copy its secret once. Copy example produces Prometheus configuration for the Lookup inventory targets. Do not reuse the Lookup enroll client, which also carries write scope.

4. Enable the engine consumer and restart

Lookup stays disabled in shipped defaults. Add connectivity only to $PADAS_HOME/core/etc/padas.toml. Do not list table names in TOML.

[lookup]
enabled = true
http_url = "https://127.0.0.1:8998"
grpc_addr = "https://127.0.0.1:50099"
cache_path = "./data/lookup-cache"
binding_poll_interval_secs = 60

[lookup.tls]
verify_cert = false
# ca_file = "etc/certs/install-ca.crt"   # when verify_cert = true

Same-host HTTPS defaults match the commented block in core/etc/padas.default.toml. If lookup-server is on another host, point http_url / grpc_addr at that host.

Shipped [[core.auth.outbound]] already includes audience padas-lookup and scope lookup:read. Do not duplicate it.

Restart the engine after this overlay. Enabling [lookup] without a restart leaves PDL lookup a silent no-op.

sudo systemctl restart padas-core.service

Tarball / foreground: stop and run "$PADAS_HOME/core/bin/padas" start again.

Lab overlays that use http://127.0.0.1:8998 are for TLS-off test harnesses. Packaged lookup-server listens HTTPS by default — do not copy lab http:// URLs into this install.

Keys: Motion Engine TOML → Lookup.

5. Configure population and tables

Motion 0.2.0 populates tables from the console with Upload CSV only (Lookup connector authoring and deploy are a later slice).

Open Management → Lookup, choose the enrolled service, and use Upload CSV. Create or upload geo_ip with ip in Index fields (and optionally Row id field). Optional Max matches, Match order, Delimiter, and First row is header map to the upload API; leave them empty to keep defaults (1, stable, comma, header on). Then use View / Edit / Download / Delete on the table row. Console View and Download are the first 1000 rows; a larger table is truncated. Full CSV is lookup-server GET /api/v1/lookup/tables/{table}/export (lookup:read), not those buttons. To change index columns, max matches, or match order on an existing table, PATCH that table; do not re-upload. Query matches one indexed field and one value.

The browser calls the authenticated UI BFF; it never calls port 8998 directly. This is admin ingest, not the PDL hot path.

Automation can still call lookup-server connector REST directly; the console does not author or deploy those connectors in 0.2.0.

Automation alternative: call lookup-server REST directly

Mint a token from the console AS using the enrolled Lookup client (lookup:write). client_id matches the inventory row id from Settings → Services (Lookup tab). The secret is on the Lookup host:

export PADAS_HOME=/opt/padas
CLIENT_ID=<lookup-client-id>
SECRET=$(sudo cat "$PADAS_HOME/lookup/data/security/${CLIENT_ID}.client.secret")

TOKEN=$(curl -sk -X POST "https://127.0.0.1:9000/api/v1/auth/token" \
  -H "Content-Type: application/x-www-form-urlencoded" \
  --data-urlencode "grant_type=client_credentials" \
  --data-urlencode "client_id=${CLIENT_ID}" \
  --data-urlencode "client_secret=${SECRET}" \
  --data-urlencode "scope=lookup:write" \
  --data-urlencode "aud=padas-lookup" \
  | python3 -c "import sys,json; print(json.load(sys.stdin).get('access_token',''))")

Create geo_ip indexed on ip (JSON body). Repeat with /upload and -F file=@geo_ip.csv for a CSV file (index_fields / row_id_field as query params).

curl -sk -X POST "https://127.0.0.1:8998/api/v1/lookup/tables/geo_ip?index_fields=ip&row_id_field=ip" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '[{"key":"10.0.0.1","ip":"10.0.0.1","city":"Amsterdam","country":"NL"}]'

Confirm metadata (row_count, index_fields) with GET /api/v1/lookup/tables/geo_ip. Change indexes without replacing rows (lookup:write):

curl -sk -X PATCH "https://127.0.0.1:8998/api/v1/lookup/tables/geo_ip" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"index_fields":["ip","city"]}'

Full CSV is GET /api/v1/lookup/tables/geo_ip/export with lookup:read. Do not drive the pipeline by fetching rows over REST on every event.

6. Cache-bind the engine instance_uuid

Table lists live in cache bindings, not in Core TOML. After enroll, the first successful core status probe writes service_instance_uuid on the Core inventory row. Open Management → Cores, choose Pick tables on the Core row, select geo_ip, and save. The UI resolves the Core's Lookup endpoint and uses that UUID as the wire consumer_id; operators do not paste UUIDs.

An empty table list still yields a silent PDL no-op. Binding poll defaults to 60 seconds—wait one interval (or restart the engine) before expecting replica rows.

Advanced/debug: bind by UUID with direct REST

The consumer id is [service].instance_uuid if set, otherwise [core].instance_uuid in $PADAS_HOME/core/etc/padas.toml:

grep instance_uuid "$PADAS_HOME/core/etc/padas.toml"
CID=<that-uuid>

If both [service].instance_uuid and [core].instance_uuid are present, bind [service].instance_uuid. The engine uses that value when it is set; otherwise it falls back to [core].instance_uuid.

Register then assign (PUT without a prior GET can 404):

curl -sk "https://127.0.0.1:8998/api/v1/lookup/cache-bindings/${CID}" \
  -H "Authorization: Bearer $TOKEN"

curl -sk -X PUT "https://127.0.0.1:8998/api/v1/lookup/cache-bindings/${CID}" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"tables":["geo_ip"]}'

7. Pipeline PDL

If the event has src_ip and the table indexes ip, name the index column first and the event field after AS:

lookup geo_ip ip AS src_ip OUTPUT country, city

Add that as the PDL Query on a processing task (Tasks), put the task in a pipeline, then assign and deploy (Management → Pipelines).

OUTPUT copies those columns onto the event. lookup() inside eval is for a single value; it returns null when the store is missing — same no-op class as the command.

Troubleshooting

Symptom	Likely cause	Fix
Pipeline runs; extra fields never appear	[lookup].enabled still false or overlay not loaded	Set enabled = true in core/etc/padas.toml; restart the engine
Same, after enable	Empty cache binding (tables: []) or wrong consumer_id	Bind the UUID from padas.toml; PUT {"tables":["geo_ip"]}
Fields appear then freeze / stay empty	lookup-server down; replica stale or never synced	curl --insecure https://127.0.0.1:8998/ready; start padas-lookup-server; wait for poll
Index miss	Event field name differs from the index column and the command has no AS	lookup geo_ip ip AS src_ip OUTPUT country, city (§7)
Silent no-op	No store injected (enabled false, empty URLs, or manager not started)	Command leaves the event unchanged; lookup() is null — not a PDL parse error
LOOKUP_TABLE_NOT_FOUND on PUT binding	Table not created yet	Create/upload the table first (§5)
Token 401	Used engine lookup:read client for write	Mint with the lookup enroll client and scope=lookup:write

Enrichment missing after a good PUT: wait binding_poll_interval_secs, then re-check task output. Do not add per-event GET …/rows to the pipeline.

Where next

• Motion Engine TOML — Lookup
• Tasks · Pipelines
• Troubleshooting & Logs
• Next steps