Enable lookup enrichment
Optional path after Install Padas Motion and a running Motion Engine. Lookup is not required for the sample TCP pipeline.
PDL lookup reads a local replica on the engine. It does not call lookup-server once per event. Keep [lookup].enabled off until lookup-server is enrolled, started, and this engine is cache-bound.
Use Install Padas Motion for UI + engine (same rpm/deb/tgz pattern). Enroll/start pitfalls (TTL, code reuse, --ca-file): padas-motion operator runbook Phases 3–6.
1. Install the nested lookup package
From $PADAS_HOME (usually /opt/padas), install packages/padas-lookup-* with the same format as the engine. Do not use tar --strip-components=1 (root must stay lookup/).
test -x "$PADAS_HOME/lookup/bin/padas-lookup-server"
2. Enroll lookup
Same flags as the engine: --ui-url and --code. There is no --service flag.
Console SPA for lookup enroll is not shipped yet (Service clients retired; Services → Lookup is a follow-on). Until then, create a lookup auth client and enroll code with the admin HTTP API (admin session cookie), then run enroll on the lookup host:
# Example: after logging into the console and capturing the session cookie
curl -sk -b admin.cookies -X POST 'https://<host>:9000/api/v1/admin/auth/clients' \
-H 'Content-Type: application/json' \
-d '{"client_id":"lab_lookup","service":"lookup"}'
curl -sk -b admin.cookies -X POST \
'https://<host>:9000/api/v1/admin/auth/clients/lab_lookup/enroll-code'
"$PADAS_HOME/lookup/bin/padas-lookup-server" enroll \
--ui-url https://<host>:9000 \
--code <enrollment-uuid>
Optional --ca-file PATH when the console TLS cert is not trusted by the default store (same enroll pitfalls as the engine: padas-motion operator runbook).
Enroll does not bind REST (8998) or gRPC (50099).
3. Start lookup-server
Same systemd vs foreground pattern as the engine; unit name is padas-lookup.service.
sudo -u padas "$PADAS_HOME/lookup/bin/padas-lookup-server"
curl --insecure https://127.0.0.1:8998/ready
Expect HTTP 200 when the store is open and gRPC is bound.
4. Enable the engine consumer and restart
Lookup stays disabled in shipped defaults. Add connectivity only to $PADAS_HOME/core/etc/padas.toml. Do not list table names in TOML.
[lookup]
enabled = true
http_url = "https://127.0.0.1:8998"
grpc_addr = "https://127.0.0.1:50099"
cache_path = "./data/lookup-cache"
binding_poll_interval_secs = 60
[lookup.tls]
verify_cert = false
# ca_file = "etc/certs/install-ca.crt" # when verify_cert = true
Same-host HTTPS defaults match the commented block in core/etc/padas.default.toml. If lookup-server is on another host, point http_url / grpc_addr at that host.
Shipped [[core.auth.outbound]] already includes audience padas-lookup and scope lookup:read. Do not duplicate it.
Restart the engine after this overlay. Enabling [lookup] without a restart leaves PDL lookup a silent no-op.
sudo systemctl restart padas-core.service
Tarball / foreground: stop and run "$PADAS_HOME/core/bin/padas" start again.
Lab overlays that use http://127.0.0.1:8998 are for TLS-off test harnesses. Packaged lookup-server listens HTTPS by default — do not copy lab http:// URLs into this install.
Keys: Motion Engine TOML → Lookup.
5. Create a table
The console Tables tab (when Managed services is on your build) is metadata only. Seed rows with CSV or JSON against lookup-server. That is admin ingest, not the PDL hot path.
Mint a token from the console AS using the lookup enroll client (lookup:write). client_id is the id you created via admin API (or later Services → Lookup). The secret is on the lookup host:
export PADAS_HOME=/opt/padas
CLIENT_ID=<lookup-client-id>
SECRET=$(sudo cat "$PADAS_HOME/lookup/data/security/${CLIENT_ID}.client.secret")
TOKEN=$(curl -sk -X POST "https://127.0.0.1:9000/api/v1/auth/token" \
-H "Content-Type: application/x-www-form-urlencoded" \
--data-urlencode "grant_type=client_credentials" \
--data-urlencode "client_id=${CLIENT_ID}" \
--data-urlencode "client_secret=${SECRET}" \
--data-urlencode "scope=lookup:write" \
--data-urlencode "aud=padas-lookup" \
| python3 -c "import sys,json; print(json.load(sys.stdin).get('access_token',''))")
Create geo_ip indexed on ip (JSON body). Repeat with /upload and -F file=@geo_ip.csv for a CSV file (index_fields / row_id_field as query params).
curl -sk -X POST "https://127.0.0.1:8998/api/v1/lookup/tables/geo_ip?index_fields=ip&row_id_field=ip" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '[{"key":"10.0.0.1","ip":"10.0.0.1","city":"Amsterdam","country":"NL"}]'
Confirm metadata (row_count, index_fields) with GET /api/v1/lookup/tables/geo_ip. Do not drive the pipeline by fetching rows over REST on every event.
6. Cache-bind the engine instance_uuid
Table lists live in cache bindings, not in Core TOML. consumer_id is the engine UUID: [service].instance_uuid if set, otherwise [core].instance_uuid in $PADAS_HOME/core/etc/padas.toml (written on first engine start).
grep instance_uuid "$PADAS_HOME/core/etc/padas.toml"
CID=<that-uuid>
If both [service].instance_uuid and [core].instance_uuid are present, bind [service].instance_uuid. The engine uses that value when it is set; otherwise it falls back to [core].instance_uuid.
Register then assign (PUT without a prior GET can 404):
curl -sk "https://127.0.0.1:8998/api/v1/lookup/cache-bindings/${CID}" \
-H "Authorization: Bearer $TOKEN"
curl -sk -X PUT "https://127.0.0.1:8998/api/v1/lookup/cache-bindings/${CID}" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"tables":["geo_ip"]}'
An empty tables list still yields a silent PDL no-op. Binding poll defaults to 60 seconds — wait one interval (or restart the engine) before expecting replica rows.
When the console Managed services page is enabled: open the lookup inventory row → Cache bindings → paste the same UUID → assign geo_ip. That tab talks to the same REST API.
7. Pipeline PDL
The event field name must match the table index column. If the event has src_ip and the table indexes ip:
eval ip = src_ip | lookup geo_ip ip OUTPUT city, country
Add that as the PDL Query on a processing task (Tasks), put the task in a pipeline, then assign and deploy (Management → Pipelines).
OUTPUT copies those columns onto the event. lookup() inside eval is for a single value; it returns null when the store is missing — same no-op class as the command.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| Pipeline runs; extra fields never appear | [lookup].enabled still false or overlay not loaded | Set enabled = true in core/etc/padas.toml; restart the engine |
| Same, after enable | Empty cache binding (tables: []) or wrong consumer_id | Bind the UUID from padas.toml; PUT {"tables":["geo_ip"]} |
| Fields appear then freeze / stay empty | lookup-server down; replica stale or never synced | curl --insecure https://127.0.0.1:8998/ready; start padas-lookup-server; wait for poll |
| Index miss | Event field ≠ table index name | eval or rename before lookup (example in §7) |
| Silent no-op | No store injected (enabled false, empty URLs, or manager not started) | Command leaves the event unchanged; lookup() is null — not a PDL parse error |
LOOKUP_TABLE_NOT_FOUND on PUT binding | Table not created yet | Create/upload the table first (§5) |
| Token 401 | Used engine lookup:read client for write | Mint with the lookup enroll client and scope=lookup:write |
Enrichment missing after a good PUT: wait binding_poll_interval_secs, then re-check task output. Do not add per-event GET …/rows to the pipeline.