Repository navigation
Tags: feldera/feldera
Tags
[web-console] Mock navigating in DeletedPipelineState and check test … …files load in CI The DeletedPipelineState integration test reaches sveltekit-superforms 3 through AppHeader > ProfileButton > the API key and OIDC trust forms. superforms 3 imports `navigating` from `$app/state`, so the import fails if the mock has no `navigating`. CI found this only in the Web Console Integration Tests job, which waits for the Docker build. The new `test-list` script loads the integration and e2e test files without running them and without a Feldera instance (`vitest list`, `playwright test --list`). The Web Console Unit Tests job runs it, so an import error fails CI before the Docker build. To let the files load without a Feldera instance: - The integration setup file warms the compilation cache in a `beforeAll` hook, because `vitest list` runs no hooks. - Under CI with no origin configured, `configureTestClient()` installs a fetch that fails every request with the "No Feldera origin configured" error, instead of throwing when the file loads. - `playwright.config.ts` calls `configureTestClient()` once. Playwright loads the config in the runner and in each worker, so the e2e specs and `globalSetup` no longer call it.
[python] Encode table and view names in connector request paths The connector methods of FelderaClient (input/output_connector_stats, pause/start_input_connector, pause/start_output_connector and generate_completion_token) built their URL from the raw table or view name. A quoted SQL identifier can contain any character, so a name like "numbers #1?" turned into a path that ended at the `#`, and the manager answered UnknownEndpoint. The ingress and egress paths already quote every component; do the same here. Pipeline and connector names are restricted to [a-zA-Z0-9_-] by the manager, so quoting them changes nothing, but it keeps these paths in the same form as the others. The new platform test drives the same connector calls, plus an input_json insert, through the SDK against a table whose name contains a space, `#` and `?`, which also covers the check asked for in #1446. Fixes #4743 Fixes #1446 Signed-off-by: 555aaditya <107977627+555aaditya@users.noreply.github.com>
[tests] Expect the reduced duration spelling in the enterprise edit test Since the duration rename the manager stores each duration in its reduced spelling, so a `checkpoint_interval` written as "60s" reads back as "1m". The enterprise-only restricted-edit test still expected "60s". The OSS CI never runs it, and the cloud merge queue failed on it (feldera/cloud#2080).
Recognize the GitHub OIDC auth path in the deletion-vector test's env… … gate The module-level MISSING_ENV check only accepted DELTA_TABLE_TEST_UNITY_TOKEN or a CLIENT_ID/CLIENT_SECRET pair, but unity_api.token_from_env also accepts UNITY_CI_SERVICE_PRINCIPAL_APPLICATION_ID plus the GitHub Actions OIDC token, which is the path CI uses. In CI none of the other vars are set, so the IN_CI gate failed collection even though OIDC auth would have worked.
[dbsp] Batches that project away a trailing column on the fly. This commit adds a new mechanism required by the upcoming lazy upsert operator. Lazy upsert will stamp updates with their arrival order. The stamp column must be removed before adding updates to the state of the collection. To do this efficiently, we will simply hide this column instead of rewriting the entire batch. The column physically disappears only when the batch gets merged with other batches. A stamped collection has the shape OrdIndexedZSet<K, Tup2<V, H>>. The new wrapper makes it look like OrdIndexedZSet<K, V>. The tricky part is that after removing the H column we must consolidate updates with the same K and V. Much of the complexity in this commit deals with doing this consolidation on the fly while scanning the batch. One other complication is that we now have multiple possible file-based representatons of an OrdIndexedZSet: with and without the stamp, so we add a new metadata field to distinguish the two. Related to this, a file-based batch may require one of two factory objects, and we don't want to pay to compile both of them for every type in the program. We therefore create the stamped batch factory on-demand only in those operators that may need it. In summary, the commit has four parts: * `FallbackIndexedWSetFactories` becomes a struct with an optional second factory set describing the wider value type, filled by `with_projection`. * `BatchMetadata` gains the trailing `value_stamp`, written by the file builders and checked on open. `read_metadata` reads a file's metadata from its trailer without factories, so a batch type whose value layout varies can learn which file it is looking at before it picks the factories to decode it. * `ProjectedValCursor` does the hiding, and the projected variants of `FallbackIndexedWSet` put it to work. Such a batch spills to an ordinary file batch, since a reader of it sees `V` alone. * `merge_cursor_over` comes out of `BatchReader::merge_cursor`, so a batch that builds its own cursor can share the filter dispatch. Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
PreviousNext