BigQuery data quality monitoring fails: PRIMARY KEY placed outside the column list in monitoring table DDL #7020

Expected Behavior

With a BigQuery offline store, data quality monitoring (feast monitor run, the /monitoring REST endpoints, and the auto_baseline jobs submitted by feast apply) creates its four monitoring tables (feast_monitoring_feature_metrics, feast_monitoring_feature_view_metrics, feast_monitoring_feature_service_metrics, feast_monitoring_jobs) if they don't already exist, then computes and stores metrics.

Current Behavior

Every monitoring call on a BigQuery deployment fails before any metric is computed, because _bq_ensure_monitoring_tables sends DDL that BigQuery can't parse:

google.api_core.exceptions.BadRequest: 400 PRIMARY KEY must be defined in the table element list as column attribute or constraint. at [27:1]

The failure happens when BigQuery parses the statement, so CREATE TABLE IF NOT EXISTS doesn't help. Creating the tables by hand beforehand doesn't help either: the same statements are still sent and rejected on every run.

Steps to reproduce

1. Configure a feature repo with offline_store: type: bigquery.
2. Run feast monitor run --project <project> (or call a /monitoring endpoint, or feast apply with auto_baseline enabled).
3. The command fails with the 400 PRIMARY KEY must be defined in the table element list error above.

You can reproduce the parser behavior on its own with a BigQuery dry run. A dry run creates nothing, and the result is the same in every location I tried (US, EU, us-central1, us-south1):

# Constraint after the closing parenthesis (what Feast generates today): rejected
bq query --dry_run --use_legacy_sql=false \
  'CREATE TABLE IF NOT EXISTS `my-project.my_dataset.t` (job_id STRING NOT NULL, status STRING) PRIMARY KEY (job_id) NOT ENFORCED'
# Error: PRIMARY KEY must be defined in the table element list as column attribute or constraint.

# Constraint inside the element list: accepted
bq query --dry_run --use_legacy_sql=false \
  'CREATE TABLE IF NOT EXISTS `my-project.my_dataset.t` (job_id STRING NOT NULL, status STRING, PRIMARY KEY (job_id) NOT ENFORCED)'
# Query successfully validated.

Specifications

• Version: 0.67.0 (Feast Operator v0.67.0, feature server image 0.67.0). Affects v0.64.0 through v0.67.0 and current master.
• Platform: Kubernetes (Feast Operator); google-cloud-bigquery 3.46.1
• Subsystem: BigQuery offline store, data quality monitoring

Possible Solution

Root cause. In _bq_ensure_monitoring_tables (bigquery.py#L873), each of the four CREATE TABLE IF NOT EXISTS statements puts PRIMARY KEY (...) NOT ENFORCED after the closing ) of the column list:

• L909 (feast_monitoring_feature_metrics)
• L927 (feast_monitoring_feature_view_metrics)
• L944 (feast_monitoring_feature_service_metrics)
• L960 (feast_monitoring_jobs)

GoogleSQL only accepts table constraints inside the table element list: CREATE TABLE ... ( column | constraint_definition [, ...] ) (docs). This code arrived in #6202 (first released in v0.64.0).

Fix. Move each PRIMARY KEY (...) NOT ENFORCED inside its column list, after a trailing comma on the last column. The key columns stay the same. The other offline stores that create these tables (Redshift, Snowflake, Postgres, Oracle) already put the constraint inside the list, so only BigQuery is affected.

Test gap. No unit test covers _bq_ensure_monitoring_tables, and sdk/python/tests/integration/monitoring/test_monitoring_integration.py mocks ensure_monitoring_tables, so no test ever checks the generated DDL. I'll open a PR with the fix and a unit test that asserts the constraint sits inside the element list for all four tables.