spark_application engine reports successful single-feature-view materializations as failure #6673

Expected Behavior

When a SparkApplication reaches COMPLETED, Feast should report the materialization as successful.

Current Behavior

Every successful single-feature-view materialization using the spark_application batch engine is reported as a failure:

Exception: SparkApplication feast-sa-<job-id> not found

This is deterministic: Feast deletes the SparkApplication CR before performing the status check that fails.

SparkApplicationComputeEngine.materialize() waits for the job to reach a terminal status and then unconditionally deletes the CR:

try:
    self._wait_for_completion(job)
    return self._build_per_fv_jobs(registry, tasks, job_id, job)
finally:
    self._cleanup(job_id)

After materialize() returns, FeatureStore._submit_and_process_materialization_jobs() calls job.status() again. For a single feature view, _build_per_fv_jobs() returns the live polling job:

if len(tasks) <= 1:
    return [job for _ in tasks]

The second status check therefore queries the CR that _cleanup() just deleted. The resulting 404 sets _error, changes the status to ERROR, and is subsequently raised by FeatureStore.

By this point, the Spark driver has already materialized the features and updated the registry. Automation calling feast materialize or the feature server's /materialize endpoint therefore sees a failure and may retry work that already succeeded.

This is not caused by the Spark Operator's timeToLiveSeconds; Feast explicitly deletes the CR immediately after _wait_for_completion() returns.

The relevant flow is still present on master:

• Cleanup after completion
• Single-FV path returns the polling job
• Status re-queries the Kubernetes API
• 404 changes the job to an error
• FeatureStore performs the second status check

Steps to reproduce

1. Configure a Feast project with batch_engine.type: spark_application.

2. Apply the feature definitions.

3. Materialize exactly one feature view.

4. Wait for the SparkApplication to reach COMPLETED.

5. Observe Feast delete the CR and report:

   SparkApplication feast-sa-<job-id> not found
   

The feature-view count matters. For multiple feature views, _build_per_fv_jobs() returns CompletedMaterializationJob instances for successful views. Their status() method returns SUCCEEDED without querying Kubernetes, so the successful multi-FV path does not exhibit this bug.

The failure can also be reproduced without a cluster:

from kubernetes.client.exceptions import ApiException

from feast.infra.compute_engines.spark_application.job import (
    SparkApplicationMaterializationJob,
)


class Api:
    """Return COMPLETED once, then simulate the CR having been deleted."""

    def __init__(self):
        self.calls = 0

    def get_namespaced_custom_object(self, **kwargs):
        self.calls += 1
        if self.calls == 1:
            return {
                "status": {
                    "applicationState": {
                        "state": "COMPLETED",
                    }
                }
            }
        raise ApiException(status=404, reason="Not Found")


job = SparkApplicationMaterializationJob("abc123", "default", Api())

print(job.status())  # SUCCEEDED: observed by _wait_for_completion
print(job.status())  # ERROR: observed by FeatureStore after cleanup
print(job.error())   # SparkApplication feast-sa-abc123 not found

Specifications

• Version: Feast v0.65.0; also present on master at ca355cb
• Platform: Kubernetes/EKS with the Kubeflow Spark Operator (sparkoperator.k8s.io/v1beta2)
• Subsystem: spark_application batch compute engine and materialization

Possible Solution

Cache the first terminal status (SUCCEEDED or ERROR) in SparkApplicationMaterializationJob and return it on subsequent calls instead of polling Kubernetes again. Non-terminal states should continue polling normally.

An alternative is to return a CompletedMaterializationJob from the successful single-FV path, matching the existing multi-FV behavior. However, caching terminal state is more robust because it prevents any future caller from losing a known result after cleanup.

Regression tests should verify that:

• COMPLETED followed by a 404 remains SUCCEEDED, with error() returning None.
• FAILED remains ERROR and preserves its errorMessage.
• Non-terminal states continue polling.

I have the terminal-status caching fix and regression tests working locally and can open a PR