Partition Filter Mapping

Tables on Hadoop-family engines are often partitioned on a technical column — an epoch integer, a lowercased region key — that no analyst would ever filter on. Unless a query carries a predicate on that column, the engine scans every partition.

Partition filter mapping makes this a dataset setting instead of a per-chart chore. A dataset owner names the partition column, the business column whose filters should be mirrored onto it, and a value transform. Superset then appends an equivalent predicate on the partition column to every query. Chart authors change nothing; queries prune.

Experimental

This feature is behind the PARTITION_FILTER_MAPPING feature flag and is off by default.

Enabling it

FEATURE_FLAGS = {
    "PARTITION_FILTER_MAPPING": True,
}

Configure it as a static boolean. A callable placed in FEATURE_FLAGS is not evaluated: FeatureFlagManager stores the configured values unchanged, so "PARTITION_FILTER_MAPPING": lambda: False resolves to a truthy function object and turns the feature on. Per-request resolution comes only from GET_FEATURE_FLAGS_FUNC and IS_FEATURE_ENABLED_FUNC, and for this flag both must return the same answer for every user: a flag that resolves differently per user or tenant would let a user with the feature off read a cached chart result that was produced from pruned SQL by a user with it on.

Configuring a mapping

In the dataset editor's Columns tab, under Default Column Settings, pick a Partition column. By default the mapping follows the dataset's default datetime column, so re-pointing that column moves the mapping with it; set an explicit override if you want it pinned to a different column.

Expand the mapped column's row in Column Settings and set the value transform: a SQL expression containing a :value placeholder, which stands for the filter value being mirrored. The Transform preserves ordering checkbox sits directly beneath it.

No engine default can infer how your pipeline buckets a numeric partition key. Enter the expression that actually populates it, including units and timezone.

Mapped column	Stored partition key	Transform
Epoch seconds (BIGINT, datetime format epoch_s)	Hour-start epoch seconds	CAST(floor(:value / 3600) * 3600 AS BIGINT)
Epoch seconds (BIGINT, datetime format epoch_s)	UTC day-start epoch seconds	CAST(floor(:value / 86400) * 86400 AS BIGINT)
UTC TIMESTAMP, with UTC epoch conversion in the database session	Hour-start epoch seconds	CAST(floor(unix_timestamp(:value) / 3600) * 3600 AS BIGINT)
UTC TIMESTAMP, with UTC epoch conversion in the database session	UTC day-start epoch seconds	CAST(floor(unix_timestamp(:value) / 86400) * 86400 AS BIGINT)
country (VARCHAR)	Lowercase region_key	lower(:value)

The timestamp examples use Hive/Impala syntax. Confirm conversion settings in both query and transform-evaluation sessions; unix_timestamp can depend on session timezone. The numeric epoch examples avoid that conversion. Use mathematical floor, including for negative epochs. Millisecond inputs require conversion to seconds before these expressions; local-calendar-day keys are not interchangeable with fixed UTC 86,400-second buckets. A session in a non-UTC zone shifts these boundaries: under IST (UTC+05:30) the hourly timestamp recipe lands on the wrong hour key, five or six hours earlier, so mirrored filters miss the matching rows. The daily recipe puts early-morning rows on the previous day. Set the evaluating session to UTC, or use the numeric epoch recipes.

Bare unix_timestamp(:value) is not suggested or mirrored for temporal-to-numeric mappings on Hive/Impala and their engine-spec subclasses. It produces an instant, not an hour/day boundary, and can silently exclude matching rows. Existing expressions remain editable; replace them with the pipeline's actual partition expression. On the same engines, a transform from a timestamp or text column onto a numeric key is also evaluated at two instants 2,843 seconds apart within one hour. A text column is asked in each common layout (2001-01-01 10:00:00, 2001-01-01T10:00:00, 2001-01-01T10:00:00Z, and its own datetime format, if set), whether or not it is flagged as a datetime. A datetime format whose output contains a quote or backslash the engine would read differently is skipped rather than sent. If any answers are exactly that far apart in seconds, milliseconds, microseconds or nanoseconds, the transform returns raw instants, however it is spelled: CAST(unix_timestamp(:value) AS BIGINT), unix_timestamp(:value) + 0, floor(unix_timestamp(:value)) and utc_to_unix_micros(:value) are all declined. A key that really stores raw epoch instants is declined too; queries then scan every partition but return correct rows. This check cannot catch a wrong bucket size or timezone, so still check every transform with Validate against data.

Dataset time-range bounds already pass through Superset's timezone/hour-offset adjustment before mirroring. Do not convert those bounds from the dataset timezone a second time. The added partition predicate remains visible in View query.

Validate against stored data

The automatic preview reports Expression evaluated — data not checked. This means Superset could evaluate the expression and build a predicate, not that it matches your data.

For a physical dataset with a numeric epoch key, enter one to three existing partition keys (whole numbers within the 64-bit range), then choose Validate against data. The section is not shown for a text partition key, and a disabled button says what is missing. Superset reads at most 50 rows from each selected partition and compares each transformed business value with that same row's stored key. It applies your data-access permissions and row-level security. It does not run a total-count query or infer sample partitions from the candidate expression.

• Matched sampled rows reports the checked rows and covered partitions. This is evidence about that sample, not proof for all rows, backfills, or future ingestion.
• Partition mismatch found shows the business value, stored key, and transform output. An output found in another partition still fails the same-row comparison.
• Validation inconclusive names each selected key that had no checkable rows or whose evaluation/query failed. NULL business values cannot establish a match.
• Filters will not be mirrored means the mapping itself refuses the transform, for example one that answers raw epoch instants. No rows are sampled: rows on the hour boundary would match it exactly and report a transform the mirror declines as matching.

Choose partitions containing off-boundary business timestamps, such as 10:47:23, and repeat across relevant dates and timezone transitions. The read is restricted to the selected keys; a row limit does not guarantee a cheap scan within a large partition. Existing warehouse query limits still apply. Changing the candidate clears the displayed validation. Validation does not save a global verdict, change chart behavior, or override the bare-epoch safety refusal. Before enabling a production mapping, compare chart row IDs and counts with and without its partition predicates, including equality, IN, and partial-bucket ranges.

Transform preserves ordering

Range filters — including the Explore time range, the most important case — are only mirrored when you check Transform preserves ordering.

Monotonicity is a property of the transform, not of the column's data type. CAST(floor(unix_timestamp(:value) / 3600) * 3600 AS BIGINT) preserves ordering. hour(:value), date_format(:value, 'dd') and dayofweek(:value) are all perfectly reasonable partition transforms on a TIMESTAMP column and none of them do: hour('2026-01-01 23:00') is greater than hour('2026-01-02 01:00') even though the first instant is earlier. Mirroring a range through one of those would silently return wrong numbers, so Superset asks you to declare it rather than guessing.

A bucketing transform preserves ordering and is the common case rather than an exception: to_char(:value, 'YYYYMMDD'), date(:value) and CAST(floor(unix_timestamp(:value) / 86400) * 86400 AS BIGINT) each map a whole day onto one partition key, which is exactly what a daily-partitioned table wants. Check the box for these.

One caveat if your partition column is temporal and the transform answers with text: a mirror such as part_date = '20260115' compares a date column against a string, which is an implicit coercion the database has to actually have. PostgreSQL and Redshift do, in both the ISO and unseparated spellings, so a text day key mirrors there. Trino has no varchar-to-date comparison at all, BigQuery coerces only a canonical YYYY-MM-DD literal, and Snowflake reads an all-digit string as epoch seconds — on those the predicate would fail the whole query rather than merely lose its pruning, so Superset declines it and the preview says why. Any other engine declines too, until it is confirmed and declared.

If your partition column is temporal and your transform returns text, the robust answer is to make the transform's return type match the column: date(:value) or cast(... as date) rather than to_char(...). A numeric partition column — the epoch shape, such as CAST(floor(unix_timestamp(:value) / 3600) * 3600 AS BIGINT) — is unaffected by any of this.

When the box is unchecked, = and IN filters still mirror; ranges do not.

What is and isn't mirrored

Filter	Mirrored
=, IN	Always, except on a text column whose engine does not compare text exactly, or when the engine compares less of the value than the filter carries
>, >=, <, <=, time ranges	Only when the transform preserves ordering — and a strict bound mirrors non-strictly
!=, NOT IN, LIKE, ILIKE, IS NULL, IS TRUE	Never

Strict bounds. Preserving ordering means non-decreasing, not strictly increasing, so event_time < '2026-07-06 12:00' mirrors to day_key <= '20260706' rather than day_key < '20260706'. Under a day key both sides of a mid-day boundary share a key, and the strict form would prune away the partition the boundary rows live in. The mirror reads one extra partition and keeps every row the filter keeps.

Negations are never safe. A transform need not be injective: lower(:value) with country != 'US' would mirror to region_key != 'us', which excludes rows whose country is already lowercase 'us' — rows the original filter keeps.

Text comparison. Mirroring col = v onto partition_col = T(v) assumes the engine compares col the way the transform was written against. Under a case-insensitive collation it does not: a stored country of 'us' satisfies a filter for 'US', while the mirror is derived from 'US' and excludes the row. So = and IN on a text mapped column only mirror on engines whose default text comparison is exact — Hive, Impala, Trino, Presto, Spark, Databricks, PostgreSQL, Snowflake, BigQuery and SQLite. On MySQL, SQL Server and Redshift, whose default comparisons ignore case or trailing blanks, a text mapping is stored and shown but no = or IN filter mirrors through it; map a numeric or temporal column instead. Numeric and temporal comparison is exact everywhere, and ranges are unaffected either way — checking Transform preserves ordering is already a statement about the column's own ordering.

A fixed-width character type is its own case, on every engine: CHAR(2), CHARACTER(2) and PostgreSQL's bpchar compare without their trailing blanks, so a stored 'US' matches a filter for 'US ' while lower('US ') does not match the partition key. Mapping such a column stores and shows, and = and IN do not mirror through it. VARCHAR and CHARACTER VARYING are unaffected.

One case Superset cannot check for you: a column that declares its own collation, such as country COLLATE NOCASE, on an engine that is otherwise exact. Nothing in the metadata exposes that, so it falls under the same pipeline assumption as the partition key itself.

Value resolution. Mirroring also assumes the engine compares the value your filter carries. It does not always. On a DATE mapped column, event_date = '2026-07-06 10:08:11' is compared on the date part alone — Postgres casts the string, Presto and Trino render DATE '2026-07-06' — so the filter keeps the whole of July 6 while a mirror derived from 10:08:11 keeps one instant of it. Superset detects what your engine drops, from the column's type rather than from a list of engines, and responds by operator: a range widens to the day, so event_date >= '2026-07-05 10:00' mirrors as if the bound were '2026-07-05 00:00' — one extra partition read, every row the filter keeps — while = and IN have nowhere to widen to and do not mirror at all. A = or IN whose value carries no time of day, such as '2026-07-06' or a date picked in the filter popover, mirrors normally. The same applies one grain finer on engines that compare timestamps only to the second.

Reading the value is what makes all of that possible, and Superset reads ISO 8601 only — 2026-07-06, 2026-07-06 10:08:11, 2026-07-06T10:08:11. It is the one format whose meaning is not a guess: 06/07/2026 is July 6 to a US reader and June 7 to most of the world, and a mirror built from the wrong one drops rows silently. So on a column the engine compares coarsely, a datetime typed in any other format — 07/06/2026 10:08:11, 2026/07/06 10:08:11, July 6 2026 10:08:11 — does not mirror, for any operator, and no pruning indicator appears. The filter itself is unaffected; the query is correct and simply reads the whole table. Write the bound in ISO 8601, or pick it from the filter popover, and the mirror comes back. Most of this stops applying where the engine compares the whole value: on a TIMESTAMP column any format your database accepts mirrors, because there is nothing left for Superset to resolve.

One exception on every column: a UTC offset. A bound written as 2026-01-01T01:00:00+01:00 does not mirror, on a TIMESTAMP column as much as on a DATE one, and no pruning indicator appears. Your transform is evaluated by the database in its own time frame — CAST(:value AS TIMESTAMP) discards the offset rather than converting through it — so a mirror built from the offset-bearing value could ask for a partition the matching rows are not in. Write the bound without an offset, in the frame the column is stored in, and it mirrors. The preview says so in those words when a sample carries one.

One exception on some databases: a quote or a backslash. A value is written into the probe as a literal, and on some databases the driver that writes it and the engine that reads it disagree about what a quote or a backslash inside it means -- Hive and Spark honour a backslash escape and not a doubled apostrophe, which is the other way round from the driver. Superset checks each value rather than assuming either spelling, so where the two disagree a filter value carrying one of those characters does not mirror, for any operator, and no pruning indicator appears. The query is unaffected and simply reads more partitions. Values carrying neither still mirror everywhere, and which of the two is affected depends on the database: Hive and Spark decline both, Postgres declines only a backslash, and MySQL, SQLite and Trino decline neither. The preview says so in those words when a sample carries one.

If = and IN on a date column matter to you, map a TIMESTAMP column instead: the engine then compares the whole value and every operator mirrors.

Time grains. A filter that carries a time grain — drill-to-detail, mostly — compares the truncated column, so the raw bounds it carries do not describe the rows it keeps: a row in the final partial bucket satisfies DATE_TRUNC(...) < until while col < until excludes it. Grained ranges still mirror, with both bounds widened by one bucket so the mirror stays no narrower than the real filter. A P1D drill therefore reads three days of partitions rather than one, instead of scanning the table. Grained = and IN filters do not mirror.

Widening is only applied to grains whose bucket width Superset knows, which means the built-in ones. A grain you added through TIME_GRAIN_ADDONS, or a built-in grain whose SQL you replaced through TIME_GRAIN_ADDON_EXPRESSIONS, has no width Superset can rely on and simply does not mirror.

Known gaps, all of which are out of scope rather than bugs:

• Filter-value dropdowns do not prune. Populating a filter's value list runs its own SELECT DISTINCT, which never goes through the chart query path. There is no filter to mirror from.
• Row-level security predicates do not mirror. They are stored as raw SQL and appended downstream of the structured filters.
• Custom SQL WHERE clauses do not mirror, for the same reason.
• Columns with an active advanced data type do not mirror. Those build their own predicate shape from translated values, so there is no operator/value pair to mirror.
• An array partition column does not mirror. A transform answers with one scalar, and there is no engine that compares an array against one — region_keys = 5 on a ClickHouse Array(Int32) key, or region_keys = 'us' on a PostgreSQL text[] one, is not an operator that exists. The mapping stores and shows, and the preview says why. Map a scalar partition column instead.
• Dashboard native filters and cross-filters do mirror — they arrive as ordinary filters — they just carry no visual indicator in the filter bar.
• The Explore glyph is advisory. It is computed from the dataset's mapping, which knows nothing about any one chart's SQL. A virtual dataset whose SQL consumes a filter through get_filters(..., remove_filter=True) gets neither that filter nor its mirror — the query path skips both — yet the glyph still shows. View query is the authority.

The assumption this rests on

Superset emits a predicate on the partition column that stands in for one on the mapped column. That substitution is only valid if, for every row in the table:

partition_column = <transform>(mapped_column)

Sample validation cannot prove this for the entire dataset. It is a property of whatever ETL populates the partition column. If that job lags, backfills with different logic, or writes the partition key in a different timezone than the transform resolves, mirrored predicates silently drop real rows and charts show quietly wrong numbers. Confirm the invariant with whoever owns the pipeline before enabling a mapping on a production dataset.

Rows in a NULL partition are the one case Superset does defend against. A predicate like dt_epoch >= X is NULL — and so drops the row — wherever dt_epoch is NULL, even when the original filter matches it. That is reachable: a dynamic-partition insert on Hive or Impala parks rows whose partition key was NULL in the default partition (__HIVE_DEFAULT_PARTITION__), and the column reads back as NULL when you query it. A transform that returns NULL for an input it cannot convert produces the same thing on any engine.

The mirrored predicate is therefore emitted as:

(dt_epoch >= X AND dt_epoch <= Y) OR dt_epoch IS NULL

The mirror only has to be no narrower than the filter it stands in for, so admitting the NULL partition costs one extra partition read and keeps those rows. Engines still prune everything else.

How the transform is evaluated

The transform is evaluated against the engine — pinned to the dataset's database, catalog and schema — and the result is emitted as a literal constant. Results are cached (PARTITION_TRANSFORM_PROBE_CACHE_TIMEOUT, 24 hours by default), which matters because this adds a round trip to the chart query path. Day-aligned ranges like "Last month" hit the cache constantly; second-granularity relative ranges like "Last 24 hours" essentially never do.

If the evaluation fails for any reason, no predicate is added: the query still runs and is still correct, it just scans more partitions.

The same goes for a transform that evaluates but answers with something the partition column cannot hold — a number against a text key, or text against a numeric one. Not every engine accepts a comparison across those types: Postgres, Trino and BigQuery refuse it outright, and SQLite compares it as never equal rather than erroring, which would drop every row the filter keeps. So Superset declines the mirror instead. The preview panel names the column and the value it got, and the fix is to cast the result to the key's own type — date_format(:value, 'yyyyMMddHH') for a text partition key holding hours.

Because the evaluation happens in a different session from the chart query, transforms that call non-deterministic functions are rejected when you save. That includes now(), current_date, current_timestamp, rand() and the zero-argument unix_timestamp(), which means "now" on Hive and Impala. The one-argument function is deterministic only under consistent session settings; the bare expression is declined for temporal-to-numeric Hive/Impala-family mappings because it does not specify a bucket.

Session-dependent behaviour that Superset cannot detect is still your responsibility: unix_timestamp(:value) is timezone-dependent on Hive and Impala, so if the evaluating session and the query session resolve different timezones the emitted bounds will disagree with the timestamp bounds they mirror. Prefer explicitly-anchored transforms.

Jinja templating is not supported in a transform. The template would render in a different context and at a different time from the chart query.

What a transform may be

A transform has to be one scalar SQL expression around :value and nothing more. These are refused when you save:

• A clause of its own, or a sub-query — password || :value FROM users, (SELECT secret FROM vault LIMIT 1) || :value. Superset splices the transform into the probe as SQL text and the engine runs it, so a transform carrying its own FROM reads a table the mapping has no business reading.
• A :value the engine would never evaluate, because it sits inside a string literal (':value') or inside a comment (1 -- :value). Neither is a transform of the filter value: the first is the constant text :value, and the second comments out the rest of the probe's own query.
• More than 1,024 characters. A transform is one hand-written expression, and the stored value is parsed on every Explore load.
• Anything that changes data — nextval(:value), a call to a stored procedure, a PostgreSQL large-object writer. The transform is evaluated against the engine to check that it mirrors, and that evaluation is cached and repeated, so it has to be a read.

An ordinary comment that does not swallow :value, such as floor(unix_timestamp(:value) / 3600) * 3600 -- hourly, is allowed. Superset removes it before evaluating the transform.

Only the mapped column may carry a transform. Setting one on any other column through the API, or in an import bundle, has no effect: Superset drops it on write, because a value parked on a column the mapping does not mirror is invisible in the editor and would go live the moment the mapped column resolved back to it.

Related configuration

Setting	Default	Purpose
PARTITION_TRANSFORM_PROBE_CACHE_TIMEOUT	24 hours	How long an evaluated transform stays cached
PARTITION_TRANSFORM_PREVIEW_RATE_LIMIT	30	Per-user, per-dataset preview requests per minute; the editor's preview panel runs a real query

Three things need a working cache backend

None of them does anything on its own. The probe cache, the preview budget and the pruning indicator all live in CACHE_CONFIG, whose default is {"CACHE_TYPE": "NullCache"} — so out of the box:

• every chart load on a mapped dataset re-probes the warehouse, because nothing is cached;
• the preview budget is never enforced, because a null cache reports every request as the first one in its window; and
• the pruning indicator stays optimistic. A transform the database rejects parses fine, so Superset cannot tell it apart from a working one until something probes — and with nothing to record that answer in, the editor's banner and the glyph on a filter chip go on promising a speed-up the query gives up. Charts stay correct either way; what is lost is the UI's ability to stop claiming otherwise.

Configuring DATA_CACHE_CONFIG does not help; all three read CACHE_CONFIG.

The backend also has to be shared across processes. A per-process SimpleCache or a FileSystemCache on a local disk gives each gunicorn worker its own counter and its own probe cache, so the effective preview budget is the configured one multiplied by the number of workers. Use Redis or Memcached if you run more than one.

Throttling fails open by design: if the cache backend errors, or cannot count, the preview is allowed rather than refused, so a cache outage does not take the dataset editor down with it. That is also why an unconfigured cache leaves previews unthrottled rather than blocked.

Mappings travel with the dataset in import/export.