Pewarta Proxy

📋 Server Request & Proxy Logs

Loading active proxy settings...
Developed by Khairul Adha (adhexa.id)
Mengambil log server...

⚙️ Pengaturan Proxy Target

Skip to content

Slow materialization #5707

Closed
Closed
Slow materialization#5707
Labels
kind/bugpriority/p2

Description

@miroslawas
miroslawas
opened on Oct 31, 2025

Expected Behavior

Feature materialization used to call "get latest feature values" function and return only the latest values - this was much faster

Current Behavior

Materializatio now pulls all features now and deduplicates in pandas - this makes materialization much slower (almost unusable for some cases)

Steps to reproduce

Try materialize

Specifications

  • Version: 0.56
  • Platform:
  • Subsystem:

offline_store: Clickhouse, online_store: Redis

Possible Solution

Give ability to pull only latest values by introducing flag/env variable

Activity

  1. miroslawas
    added
    kind/bug
    priority/p2
    on Oct 31, 2025

  2. miroslawas commented on Oct 31, 2025

    @miroslawas
    miroslawas
    on Oct 31, 2025
    ContributorAuthor

    @franciscojavierarceo @HaoXuAI

  3. miroslawas commented on Oct 31, 2025

    @miroslawas
    miroslawas
    on Oct 31, 2025
    ContributorAuthor

    FYI, @astronautas

  4. astronautas commented on Oct 31, 2025

    @astronautas
    astronautas
    on Oct 31, 2025
    Contributor

    We could contribute this param ourselves, no issue at all, just wanted to see what maintainers think.

  5. franciscojavierarceo commented on Oct 31, 2025

    @franciscojavierarceo
    franciscojavierarceo
    on Oct 31, 2025
    Member

    was this always the case or recently changed? It'd be helpful to know what exposed this. If some users like this behavior than probably the exercise is for us to make this configurable. As a general point, I do agree with you that having pandas execute deduplication is wildly expensive.

  6. astronautas commented on Oct 31, 2025

    @astronautas
    astronautas
    on Oct 31, 2025
    Contributor

    was this always the case or recently changed? It'd be helpful to know what exposed this. If some users like this behavior than probably the exercise is for us to make this configurable. As a general point, I do agree with you that having pandas execute deduplication is wildly expensive.

    Seems from 0.50.0, the version which broke Clickhouse offline store , since the core had started calling get_all_... instead of get_latest_... from offline store (which eventually requires deduplication taken care by online store e.g. Redis does this in pure Python / Pandas).

  7. haoxu0 commented on Nov 1, 2025

    @haoxu0
    haoxu0
    on Nov 1, 2025
    Collaborator

    It's probably a bug, even it changed to pull_all, default will still dedup and get the latest features only

  8. astronautas commented on Nov 3, 2025

    @astronautas
    astronautas
    on Nov 3, 2025
    Contributor

    It's probably a bug, even it changed to pull_all, default will still dedup and get the latest features only

    @HaoXuAI I don't think it's a bug. It's a feature. Functionally, it does the job - online store receives only the latest values. The issue is not about this.

    Non-functionally, it's much slower. That is the issue. Why? It dedups just before pushing to the online store. Pure Python is used for that. For large periods between start_ts and end_ts, that pulls in a significant amount of data from offline store to be deduped in later downstream. Before it would dedup using SQL via get_latest, so it was multiple times faster and used many times less memory. Basically, it did predicate pushdown, in some sense.

    See here. Pretty sure before 0.50.0 it used to call pull_latest..., which instead delegated deduplication (via window functions) to offline store, which would pull-in significantly lower amount of data into materialization.

    Does this make sense?

  9. haoxu0 commented on Nov 3, 2025

    @haoxu0
    haoxu0
    on Nov 3, 2025
    Collaborator

    It's probably a bug, even it changed to pull_all, default will still dedup and get the latest features only

    @HaoXuAI I don't think it's a bug. It's a feature. Functionally, it does the job - online store receives only the latest values. The issue is not about this.

    Non-functionally, it's much slower. That is the issue. Why? It dedups just before pushing to the online store. Pure Python is used for that. For large periods between start_ts and end_ts, that pulls in a significant amount of data from offline store to be deduped in later downstream. Before it would dedup using SQL via get_latest, so it was multiple times faster and used many times less memory. Basically, it did predicate pushdown, in some sense.

    Does this make sense?

    If using pandas that's right. For engines like polars or spark lazy execution it should run the transformation in an optimal way.
    I think we could arrange the dedup at early order if not aggregation or transformation provided for compute without optimization such as pandas

  10. astronautas commented on Nov 3, 2025

    @astronautas
    astronautas
    on Nov 3, 2025
    Contributor

    It's probably a bug, even it changed to pull_all, default will still dedup and get the latest features only

    @HaoXuAI I don't think it's a bug. It's a feature. Functionally, it does the job - online store receives only the latest values. The issue is not about this.
    Non-functionally, it's much slower. That is the issue. Why? It dedups just before pushing to the online store. Pure Python is used for that. For large periods between start_ts and end_ts, that pulls in a significant amount of data from offline store to be deduped in later downstream. Before it would dedup using SQL via get_latest, so it was multiple times faster and used many times less memory. Basically, it did predicate pushdown, in some sense.
    Does this make sense?

    If using pandas that's right. For engines like polars or spark lazy execution it should run the transformation in an optimal way. I think we could arrange the dedup at early order if not aggregation or transformation provided for compute without optimization such as pandas

    Any way to switch to Polars? Could you give a YAML reference? I would give it a shot at least. Anything similar - local compute engine without an external cluster - would be also fine to test out :).

    Local compute engine - 404.

  11. astronautas commented on Nov 3, 2025

    @astronautas
    astronautas
    on Nov 3, 2025
    Contributor

    @HaoXuAI

  12. astronautas commented on Nov 3, 2025

    @astronautas
    astronautas
    on Nov 3, 2025
    Contributor

    All right, I just tried this in feature_store.yaml:

    ...
batch_engine:
  type: local
  backend: polars

    Unfortunately, the total materialization duration got reduced insignificantly :/ I can observe increased CPU usage (good, it parallelizes something), but sadly e2e it's not as fast as it used before when it delegated deduplication to offline store (Clickhouse, in our case). So no, it does not run the materialization in optimal way (optimal meaning compared to < 0.50.0).

    Deduplication delegated to offlne store in our case works fast because we set partitioning on the timestamp column. Likely the case for every other analytical store like Snowflake. This significantly increases the performance of materialization.

    I propose we introduce some sort of a flag - pushdown_deduplication_to_offline_store - for folks who know what they're doing to be pushed down to offline store to deduplicate early. Maybe part of RepoConfig, as an experimental flag? Or offline_config: Any = Field(None, alias="offline_store")?

    We're more than OK to raise a PR, if that would make sense :).

  13. astronautas commented on Nov 11, 2025

    @astronautas
    astronautas
    on Nov 11, 2025
    Contributor

    @franciscojavierarceo Let's close, solved

  14. franciscojavierarceo
    closed this as completedon Nov 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugpriority/p2

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions