Skip to content

Releases: pola-rs/polars

Python Polars 1.44.2

Choose a tag to compare

@github-actions github-actions released this 09 Sep 07:42

🛠️ Other improvements

Thank you to all our contributors for making this release possible!
@EndPositive, @dsprenkels, @kdn36, @lun3x, @nameexhaustion and @orlp

Python Polars 2.0.0-rc.1

Pre-release

Choose a tag to compare

@github-actions github-actions released this 02 Sep 11:49
4296b83

💥 Breaking changes

  • Set the default engine for SQL to the streaming engine (#28973)

🚀 Performance improvements

  • Reuse native metadata for Iceberg sinks (#29062)
  • SQL exists predicate pushdown before subqueries (#29078)
  • Collect primitive group-by aggregations into single chunk (#28825)
  • Use cost information to determine whether to keep cached nodes (#29042)
  • Use scan cardinality in build side if sampling is exhausted (#29040)

✨ Enhancements

  • Defer SQL resolving until we convert to IR (#29077)
  • Expose Iceberg sink writer options (#29065)
  • Support partitioned Iceberg sinks (#28823)
  • Expose fixed-array inner product in SQL (#29033)
  • Add strip_prefix and strip_suffix to expression names (#29041)
  • Use scan information in join reordering (#29038)
  • Add cardinality information (#29024)
  • Allow customizing explain output for debugging custom io sources (#23978)
  • Add more expression properties to meta namespace (#28091)
  • Improve SQL coverage (#29010)
  • Improve SQL coverage (#29006)
  • Add Join reordering. (#28985)
  • Merge 2.0 into main (#28982)
  • Set the default engine for SQL to the streaming engine (#28973)

🐞 Bug fixes

  • Don't keep incrementing morsel id in MergeSortedNode pass-through (#29055)
  • Ensure show_graph shows physical plan for auto engine (#29011)
  • Assign SQL JOIN literal comparisons to the correct input (#28701)
  • Make distributor linearizer capacity consistent (#29057)
  • Correct peak visitors, dictionary bounds, and Date feature gating (#29031)
  • Keep extension type schema consistent (#29039)
  • Fix Series.sample order when with_replacement=True and shuffle=False (#28990)
  • Reduce memory in multi_scan pipeline PostApplyExtraOps (#29049)
  • Fix SO in rayon caused by join_where (#29051)
  • Clarify GPU installation guidance (#29037)
  • Remove POLARS_AUTO_STRUCTIFY env var backdoor (#29032)
  • Ensure metrics are flushed between phases and before logging (#29014)
  • SQL parse a typed DATE literal instead of casting it (#29007)
  • Fix parquet decode of plain pages in dictionary column (#29001)
  • Reserve fixed-size-list child capacity (#28980)
  • Fix SQL casts (#28986)
  • Fix incorrect condition in combine_validities_and_many and add test (#28909)
  • Give sink_batches lazy: Literal[False] overload a default (#28951)

📖 Documentation

  • Update 2.0.0-rc1 migration guide with latest changes (#28983)
  • Fix invalid output in DataFrame.sample example (#28998)

🛠️ Other improvements

  • Bump rust toolchain (#29080)
  • Increase rustc stack size (#29076)
  • Log pytest worker crashes and disable restarting (#29058)
  • Replace FileInfo row_estimation tuple with typed ScanStats (#29023)
  • Use an if for cfg(debug_assertions) to fix the benchmark job (#29002)
  • Remove 2.0 branch from ci-tested branches (#28981)
  • Skip tests that are currently failing on windows (#28962)

Thank you to all our contributors for making this release possible!
@0guban0v, @0xC61, @Aniket-a14, @LarryHu0217, @MatteoPossamai, @NeejWeej, @TNieuwdorp, @WaterWhisperer, @adamreeve, @dsprenkels, @jonasdedden, @kdn36, @math-hiyoko, @matthewbayer, @nameexhaustion, @orlp, @ritchie46, @severinh and @sivakumar-mahalingam

Python Polars 1.44.1

Choose a tag to compare

@github-actions github-actions released this 26 Aug 07:09
e4ae5ea

Thank you to all our contributors for making this release possible!
@Kevin-Patyk, @henrytsanford, @jonasdedden, @orlp and @ritchie46

Python Polars 1.44.0

Choose a tag to compare

@github-actions github-actions released this 24 Aug 09:38
6f46eb2

⚠️ Deprecations

  • Deprecate rechunk parameter for all read/scan functions (#28063)
  • Deprecate Expr.rechunk() (#28692)
  • Deprecate struct.rename_fields() with an incorrect number of fields (#28672)

🚀 Performance improvements

  • Add private env var toggle for HTTP rate limit (#28882)
  • Relax default CloudRetryConfig for rate limit stability (#28885)
  • Elide generic cross join on subquery decorrelation in equality predicates (#28876)
  • cache CTE's in SQL layer (#28864)
  • Allow CSPE if manual caches are set (#28859)
  • Lower unqualified join predicates to inner joins in SQL (#28854)
  • Rechunk mask for DataFrame filter (#28762)
  • Improve performance of when/then/otherwise by masking out unevaluated elements (#28498)
  • Drop unused projections in filter on streaming engine (#28713)
  • Retain Parquet Partial metadata on filter (#28737)

✨ Enhancements

  • Support integer fixed-array dot products (#28829)
  • SQL correlated in (#28927)
  • Support more join_types in join_where (#28880)
  • Support Iceberg schema evolution (#28794)
  • SQL conformance (#28494)
  • Introduce RemoteEngine and a common base class for all engines (#28800)
  • Support Iceberg table with V3 deletion vectors in native scan_iceberg (#28772)
  • Support Iceberg snapshot properties (#28793)
  • Expose sinked_paths_callback on sink_parquet as unstable parameter (#28814)
  • Add infer_schema_files to CSV inference hint (#28809)
  • Add native fixed-array dot (#28504)
  • Export query metrics to polars_cloud (#28757)
  • Support Iceberg object storage paths (#28634)
  • More CSEE simplification (#28731)
  • Improve error in Series constructor with name in context (#28743)
  • Add adaptive HTTP rate-limiter for cloud IO (#28591)
  • Add struct.drop() (#28666)
  • Improve error message when CSV name de-duplication fails (#28658)
  • Always keep first metadata per source for Parquet (#28661)
  • Improve plan-time row estimates for multi-file parquet scans (#28380)
  • Expose array and plugin function views in Python visitor (#28635)

🐞 Bug fixes

  • SQL handle quantified comparisons against a subquery (#28929)
  • Fix SQL subquery qualified lowering and cache row-index (#28926)
  • Clippy-stable failures (#28919)
  • Use uuidv7 for cloud observed queries (#28910)
  • Unsoundness in AnyValue::into_static for structs (#28889)
  • Invalidate suborders on projection (#28866)
  • Ensure SQL function registry is available in isolated subqueries (#28881)
  • Handle null in Expr.is_not_nan (#28883)
  • Ensure eager unpivot column selection aligns with the lazy engine (#28846)
  • Fix assertion error / incorrect data load from parquet written by fastparquet (#28656)
  • Fix projection pushdown incorrect select() height (#28751)
  • Restore monitoring as an engine-level parameter (#28821)
  • Bincode serialization of IR hints (#28837)
  • Retain datetime inference across CSV fallback (#28663)
  • Avoid chunk fragmentation on bool agg min and max (#28789)
  • Merge join with coalesce and empty suffix (#28799)
  • Wrong function name call in test (#28801)
  • Preserve row mapping in nested over expressions (#28767)
  • Restore CI compatibility with NumPy 2.5.2 (#28782)
  • Do not deduplicate inherently nondeterministic plan nodes (#28768)
  • Fix OOC spilling of categoricals (#28761)
  • Correct partitionable filter handling for pivot (#28749)
  • Small typo in docstring (#28763)
  • Avoid chunk fragmentation on bool agg any and all (#28754)
  • Ignore nans in (min|max)_horizontal (#28710)
  • Correct function display for str.replace_all (#28742)
  • Exclude debug tests from nodebug streaming run (#28747)
  • Convert predicate series dtype during prefiltered parquet reading (#28711)
  • Do not remove nested caches when performing predicate pushdown past a cache node (#28740)
  • Correct handling of nulls at the start of business day functions (#28703)
  • Do not panic when exceeding Parquet metadata scan cache (#28721)
  • Remove cryptographic hashing from CSPE (#28716)
  • Unsoundness in rayon block_on (#28709)
  • Check for (shallow) IR node equality on top of hashing in CSPE (#28506)
  • Don't hand IO plugins a predicate they cannot evaluate (#28643)
  • Raise on overflow in Decimal sum instead of silently wrapping (#28688)
  • Preserve nulls when importing Arrow maps (#28680)
  • Propagate null by column values in rolling_*_by (#27367)
  • Fix self-referencing field in struct.with_fields with over (#28678)
  • Fix Arrow buffer offset for Utf8 and Binary (#28662)
  • Clamp group-by slice offset (#28579)
  • Release the GIL while resolving the schema in __arrow_c_stream__ (#28642)
  • Fix data corruption on Series from nested Arrow LargeList (#28632)

📖 Documentation

  • Interlink join docstrings (#28907)
  • Migration guide for 2.0 release (#28788)
  • Clarify read_database_uri requirements for SQLAlchemy (#28366)
  • Update engine tags (#28852)
  • Clarify "that that side" comment in asof_join (#28847)
  • Update mage.ai documentation link (#28813)
  • Add engine tags (#28708)
  • Document that arg_extremum functions return any index on ties (#28691)
  • Fix Polars Cloud API reference link (#28693)

🛠️ Other improvements

  • Bump runtime toolchains (#28924)
  • Add min-publish-age for cargo (#28906)
  • Tighten join invariants DSL (#28913)
  • Bump Rust toolchain (#28900)
  • Ensure that predicates are pushed down to the scans (#28857)
  • Clarify "that that side" comment in asof_join (#28847)
  • Restore monitoring as an engine-level parameter (#28821)
  • Remove workaround for inconsistent CSE (#28834)
  • Introduce RemoteEngine and a common base class for all engines (#28800)
  • Add row count and file byte size to sinked paths callback (#28522)
  • Use canonical expression IDs in CSEE (#28786)
  • Enforce 100% type coverage in CI with Pyrefly (#28795)
  • Attribute annotations for CredentialProviderAWS and CredentialProviderAzure (#28790)
  • Attribute annotations for ScanCastOptions (#27906)
  • Disable credentials in upload chunk size tests (#28745)
  • Improve backtrace path formatting (#28732)
  • Exclude debug tests from nodebug streaming run (#28747)
  • Mark test_grouped_agg_parametric as slow (and add param ids) (#28715)
  • Fix first expect in test_group_by_arg_max_boolean_26978 (#28690)
  • Fix the name of struct.drop() in rename_fields() deprecation message (#28687)
  • Add dist=loadgroup to the pytest config (#28676)
  • Enable a lot of may_fail_auto_streaming tests (#28675)
  • Re-enable test_extension() for streaming engine (#28611)
  • Update test_hive_join_rewrite_semi_join test to work with streaming engine (#28610)

Thank you to all our contributors for making this release possible!
@0guban0v, @JakubValtar, @Joosboy, @Kevin-Patyk, @MatteoPossamai, @NicoOhR, @TNieuwdorp, @VedantMadane, @aarushkandukoori, @alexander-beedie, @azimafroozeh, @borchero, @carnarez, @dancsi, @dependabot[bot], @dsprenkels, @gautamvarmadatla, @jonasdedden, @jorenham, @kdn36, @lzcmian, @malhotrashivam, @matthewbayer, @mikhail5555, @nameexhaustion, @orlp, @ritchie46, @subotac and dependabot[bot]

Rust Polars 0.55.2

Choose a tag to compare

@github-actions github-actions released this 06 Aug 09:48

✨ Enhancements

  • Add adaptive HTTP rate-limiter for cloud IO (#28591)

🐞 Bug fixes

  • Unsoundness in rayon block_on (#28709)
  • Check for (shallow) IR node equality on top of hashing in CSPE (#28506)

📖 Documentation

  • Document that arg_extremum functions return any index on ties (#28691)

Thank you to all our contributors for making this release possible!
@dancsi, @dsprenkels, @kdn36, @orlp and @ritchie46

Rust Polars 0.55.1

Choose a tag to compare

@github-actions github-actions released this 05 Aug 07:39

This Releases' DSL matches Python 1.43.2.

🚀 Performance improvements

  • Pushdown len() to concat/union inputs (#28570)
  • Optimize not(bool_f) to not_bool_f (#28474)
  • Split multiplexers that directly scan from in-memory DataFrame (#28376)
  • Make DNS cache global (#28352)
  • Do not remove cache if predicates not pushed to all inputs (#28341)
  • Environment variable for logging slow DNS lookup (#28211)
  • Pre-partition on left, right and semi joins on hive partitioned data (#28374)
  • Rewrite inner joins on hive partitioned data to a union of partitions filtered joins (#28327)
  • Tighten redundant comparisons and propagate filter constraints (#27999)
  • Canonicalize predicates (#28212)
  • Remove excess memory copy in streaming IPC sink (#28141)
  • Parallelize scan_iceberg resolve (#28168)
  • Chunk-aware morsel splitting on single-file IO sink (#27950)
  • Don't allocate for full_null() with small lengths (#28181)
  • Use monotonic deque for rolling min_by/max_by (O(n)) (#27436)
  • Add a sampled resolve mode for multi-file parquet metadata (#28111)
  • Speed up small dtype series sums with a upcast sum kernel (#27958)
  • Don't block on path expansion (#28073)
  • Add bytes-based concurrency control for cloud IO (#27924)
  • Stop flushing multiplexer on phase change (#28013)
  • Prevent empty chunks from triggering copy in rechunk (#27953)
  • Improve performance of struct unnest inside list eval on streaming (#27905)

✨ Enhancements

  • Add struct.drop() (#28666)
  • Improve error message when CSV name de-duplication fails (#28658)
  • Always keep first metadata per source for Parquet (#28661)
  • Improve plan-time row estimates for multi-file parquet scans (#28380)
  • Serialize and deserialize SinkTypeIR::Partitioned (#28112)
  • Add infer_schema_files parameter to scan_csv (#28440)
  • Support reading IEEE 754 total order Parquet column order (#27896)
  • Expose more Expr nodes for cudf_polars (pt 2) (#28404)
  • Allow callback sinks on cloud (#28458)
  • Pre-partition group-by on hive keys (#28444)
  • Add ewm_sum and ewm_sum_by #28151 (#28215)
  • Serve stale records from object_store DNS cache (#28256)
  • Expose more expr nodes for cudf-polars (#28117)
  • Make partition_hive an opt flag (#28345)
  • When displaying IRExprs, only add parentheses when needed for disambiguation (#28319)
  • Show only differing dtypes in assert_frame_equal error (#27816)
  • Add Clone to SpillFrame/Token (#28302)
  • Add Series.degrees/radians and clarify trig functions use radians (#22961)
  • Merge sorted with multikey (#28067)
  • Add list expression which consistently packs elements together into new List type (#27990)
  • Construct pyarrow predicates via PyO3 instead of eval str (#27271)
  • Optimize joins with redundant equi-join keys (#28017)
  • Implement missing datetime options for CastColumnsPolicy (#28084)
  • Add Expr.cat.to and Expr.cat.physical (#28096)
  • Add POLARS_OOC_DISK_BUDGET_MB (#28045)
  • Close ResourceWarning leaks in database tests (#28066)
  • Expose IRFunctionExpr::Fused in the python node visitor (#27889)
  • Deprecate casts from string to temporal dtypes (#28056)
  • Add naive out-of-core spilling to Polars (#27998)
  • Add experimental strict mode (#28023)
  • Optimize non-negative len and null_count comparisons (#28002)
  • Optimize len and null_count comparisons (#27727)
  • Support SQL's implicit JOIN syntax (#27890)
  • Sample from bivariate distribution for spill context Thompson sampling (#27973)
  • Add Expr.is_sorted (#26708)
  • Add suggestions in wrong api usage or mismatched column names (#27954)
  • Add external object_store support for non-native schemes (#27871)
  • Skip Delta files and parquet row groups using per-field struct statistics (#27887)
  • Add re-registration of OOC SpillTokens (#27909)
  • Improve IR sortedness detection for with_row_index and explode (#27875)
  • Add DataFrame.is_sorted() (#27870)

🐞 Bug fixes

  • Don't hand IO plugins a predicate they cannot evaluate (#28643)
  • Don't hang or crash on sqrt/cbrt/pct_change/ewm_* with a Struct column (#28696)
  • Raise on overflow in Decimal sum instead of silently wrapping (#28688)
  • Preserve nulls when importing Arrow maps (#28680)
  • Propagate null by column values in rolling_*_by (#27367)
  • Fix self-referencing field in struct.with_fields with over (#28678)
  • Fix Arrow buffer offset for Utf8 and Binary (#28662)
  • Clamp group-by slice offset (#28579)
  • Ensure categorical fill_null(strategy="min"/"max") uses lexical min/max (#28636)
  • Release the GIL while resolving the schema in __arrow_c_stream__ (#28642)
  • Fix data corruption on Series from nested Arrow LargeList (#28632)
  • Properly propagate enum metadata for iceberg column mapping (#28580)
  • Correctly write parquet field IDs for enums and categoricals (#28581)
  • Fix arrow C interop binview offset (#28623)
  • Fix duplicate hive values in hive pre-partition rewrite (#28621)
  • Double-counted offset for Arrow export of sliced Series(Array) (#28602)
  • Struct/List/Array equality handling in sort and join (#28601)
  • Use try_new in StructArray construction in polars-json (#27489)
  • Raise ComputeError instead of panicking in repeat_by when output exceeds IdxSize::MAX (#27892)
  • Run type coercion pass on pivot's internally generated group_by (#27897)
  • More careful slice pushdown into joins (#28578)
  • Preserve ordering in sliced unions (#28576)
  • Drop input sortedness when casting to a string (#28574)
  • Clear sortedness flags in StringChunked substring kernels (#28573)
  • Bad mask handling when reading optional parquet column (#28547)
  • Any operation on Unknown(Int) and Unknown(Float) should result in Unknown(Float) (#28545)
  • Flip nulls_last after Expr.reverse() (#28572)
  • Fix high blocking thread use in sink_parquet with async local path (#28543)
  • Serialize LazyFrames backed by bytes (#28568)
  • Release GIL in SQLContext.execute() (#28549)
  • Propagate nulls_last in function_expr_sortedness (#28544)
  • Incorrect slicing when a join requires sorting (#28541)
  • Fix panic on self join of scan_delta / scan_iceberg (#28468)
  • Decrement ParkGroup worker count when worker exits (#28492)
  • Add missing Send bounds in Connector (#28497)
  • Fix UB on first/last_non_null with empty chunk (#28495)
  • Do not CSE non-column height expr on streaming engine (#28480)
  • Propagate adjusted sortedness through Gather (#28493)
  • Deduplicate the expression equality checks (#28483)
  • Address 3VL issue with SQL NOT IN interaction with NULL values and joins (#28484)
  • Use total comparison in Expr literal values (#28482)
  • Ensure SQL SUM and CORR aggregates return NULL for all-null inputs, add TOTAL (#28475)
  • Share null_count_dtype helper between Delta and Iceberg, fixing SchemaError (#28479)
  • Remove non-output columns from the equi-join and semi/anti-join operators (#28446)
  • Fix dropped slice on multiple unions (#28477)
  • Fix in-memory engine incorrect slice on maintain order join (#28478)
  • Check join schema by position (#28455)
  • Block predicate pushdown past overwritten window keys (#28429)
  • Avoid panic when union slice skips all rows (#28420)
  • Solve panic in dt.replace when there were multiple chunks (#28437)
  • Propagate is_scalar from the input to the output of .sort() and .sort_by() (#28438)
  • Insert missing coercions from Unknown(_) in list/array arithmetic (#28411)
  • Resolve CSV column names overwrite in DSL->IR conversion (#28383)
  • Avoid IEJoin rewrite for Categorical comparisons (#28427)
  • Do not rewrite sort().reverse() to sort(descending=True) when maintain_order=True (#28403)
  • Propagate bigidx to polars-plan from polars (#28396)
  • Ensure BinaryView offset+len does not exceed i32::MAX where possible (#28048)
  • Invalid offset in strptime (#28388)
  • Incorrect schema type for decimal <-> primitive division (#28373)
  • Panic in in-memory CSEE handling (#28371)
  • Apply same type coercion to IsBetween as binary comparisons (#28300)
  • Fix offset in arrow ffi export of sliced struct arrays (#28369)
  • Fix cross filter not applied with sink and CSE (#28297)
  • Fix panic on projection pushdown with caches (#28280)
  • Fix write_json() null values in Array columns being written incorrectly as null (#28330)
  • Fix EntityTooSmall on sink_ipc to S3 (#28255)
  • Raise error instead of silent wrapping for select(len()) (#28355)
  • Float16 groupby aggregates (#28361)
  • Respect lexical ordering of Categorical in top_k/bottom_k (#28359)
  • Support dtype-list schema overrides in scan_csv (#28226)
  • Resolve unknown type in fused multiply add (#28321)
  • Honor aws_checksum_algorithm when provided (#28328)
  • Fix regression in sourcing HF_TOKEN (#28317)
  • Raise on addition/subtraction between temporal and non-temporal Series (#28266)
  • Rewrite with_columns on empty df to select during DSL -> IR (#28298)
  • Fix sink deadlock regression (#28296)
  • Avoid panic when consumer of vertical concat is done (#28272)
  • Ensure predicate simplification is deterministically ordered (#28269)
  • Don't attribute untimed execution to optimization in profile (#28273)
  • Harden IPC mmap API (#28258)
  • Classify S3 region request failures as IO errors (#28225)
  • Incorrect inner dtype in ArrayChunked::from_aligned_values (#28249)
  • Unsoundness in arg_sort_ascending (#28245)
  • Add column length check to row encoding (#28248)
  • Raise on implode of Object dtype instead of creating invalid List(Object) (#28246)
  • Don't panic in qcut on degenerate float input (all-NaN, infinite, include_breaks) (#28220)
  • Fix streaming Strptime feature flags (#28242)
  • Return null not 0 for the first unbiased ewm_var/ewm_std value (#28235)
  • Use after free in file cache (#28236)
  • Incorrect length calculation in parallel hashmap (#28227)
  • Avoid slice pushdown panic on shared cache inputs (#28157)
  • Incorrect dtype for Unknown rig...
Read more

Python Polars 1.43.2

Choose a tag to compare

@github-actions github-actions released this 01 Aug 06:28
ae588a9

⚠️ Deprecations

  • Deprecate casts from Categorical to integer dtypes (#28525)
  • Deprecate not setting the plan_stage argument in show_graph() (#28391)

🚀 Performance improvements

  • Pushdown len() to concat/union inputs (#28570)

✨ Enhancements

  • Issue FutureWarning on using from_arrow of ArrowStreamExportable (#28442)
  • Add infer_schema_files parameter to scan_csv (#28440)

🐞 Bug fixes

  • Properly propagate enum metadata for iceberg column mapping (#28580)
  • Correctly write parquet field IDs for enums and categoricals (#28581)
  • Fix arrow C interop binview offset (#28623)
  • Fix duplicate hive values in hive pre-partition rewrite (#28621)
  • Double-counted offset for Arrow export of sliced Series(Array) (#28602)
  • Struct/List/Array equality handling in sort and join (#28601)
  • More careful slice pushdown into joins (#28578)
  • Preserve ordering in sliced unions (#28576)
  • Drop input sortedness when casting to a string (#28574)
  • Clear sortedness flags in StringChunked substring kernels (#28573)
  • Bad mask handling when reading optional parquet column (#28547)
  • Any operation on Unknown(Int) and Unknown(Float) should result in Unknown(Float) (#28545)
  • Flip nulls_last after Expr.reverse() (#28572)
  • Fix high blocking thread use in sink_parquet with async local path (#28543)
  • Serialize LazyFrames backed by bytes (#28568)
  • Release GIL in SQLContext.execute() (#28549)
  • Propagate nulls_last in function_expr_sortedness (#28544)
  • Incorrect slicing when a join requires sorting (#28541)

🛠️ Other improvements

  • Fix duckdb delta extension install collision (#28607)

Thank you to all our contributors for making this release possible!
@BitWeaverDev, @Matt711, @Samoilov2004, @borchero, @dancsi, @dsprenkels, @kdn36, @masumi-ryugo, @nameexhaustion and @ritchie46

Python Polars 1.43.1

Choose a tag to compare

@github-actions github-actions released this 27 Jul 12:08
8d438be

🚀 Performance improvements

  • Optimize not(bool_f) to not_bool_f (#28474)

✨ Enhancements

  • Allow callback sinks on cloud (#28458)

🐞 Bug fixes

  • Fix panic on self join of scan_delta / scan_iceberg (#28468)
  • Fix UB on first/last_non_null with empty chunk (#28495)
  • Do not CSE non-column height expr on streaming engine (#28480)
  • Propagate adjusted sortedness through Gather (#28493)
  • Deduplicate the expression equality checks (#28483)
  • Address 3VL issue with SQL NOT IN interaction with NULL values and joins (#28484)
  • Use total comparison in Expr literal values (#28482)
  • Ensure SQL SUM and CORR aggregates return NULL for all-null inputs, add TOTAL (#28475)
  • Share null_count_dtype helper between Delta and Iceberg, fixing SchemaError (#28479)
  • Remove non-output columns from the equi-join and semi/anti-join operators (#28446)
  • Fix dropped slice on multiple unions (#28477)
  • Fix in-memory engine incorrect slice on maintain order join (#28478)
  • Check join schema by position (#28455)
  • Block predicate pushdown past overwritten window keys (#28429)

📖 Documentation

  • Update and restructure README (#28490)
  • Minor fixes to Spark migration guide (#27730)
  • Relocate Polars Cloud & On-Prem User Guide (#28462)
  • Add notes on k8s operator (#28445)

🛠️ Other improvements

  • Make hive_part extraction a function and public (#28507)
  • Deduplicate the expression equality checks (#28483)
  • Bump ruff and mypy package versions (#28456)
  • Fix flaky test (#28454)

Thank you to all our contributors for making this release possible!
@AnirudhRahul, @EndPositive, @Jesse-Bakker, @alexander-beedie, @carnarez, @dancsi, @mdroogh, @mroeschke, @nameexhaustion, @nchammas, @orlp, @r-brink and @ritchie46

Python Polars 1.43.0

Choose a tag to compare

@github-actions github-actions released this 21 Jul 04:30
adb194d

⚠️ Deprecations

  • Deprecate casting numeric types to categoricals (#28349)
  • Deprecate cat.get_categories() and cat.to_local() (#28299)
  • Deprecate LazyFrame.profile() (#28275)
  • Deprecate casts from non-nested dtypes into Lists (#28193)
  • Deprecate bitwise ops between integers and booleans (#28186)
  • Deprecate list/arr.to_struct() calls that don't pass field names (#28243)
  • Rename missing_utf8_is_empty_string to empty_string_is_null (#28173)

🚀 Performance improvements

  • Split multiplexers that directly scan from in-memory DataFrame (#28376)
  • Make DNS cache global (#28352)
  • Do not remove cache if predicates not pushed to all inputs (#28341)
  • Environment variable for logging slow DNS lookup (#28211)
  • Pre-partition on left, right and semi joins on hive partitioned data (#28374)
  • Rewrite inner joins on hive partitioned data to a union of partitions filtered joins (#28327)
  • Tighten redundant comparisons and propagate filter constraints (#27999)
  • Canonicalize predicates (#28212)
  • Remove excess memory copy in streaming IPC sink (#28141)
  • Parallelize scan_iceberg resolve (#28168)
  • Chunk-aware morsel splitting on single-file IO sink (#27950)
  • Don't allocate for full_null() with small lengths (#28181)
  • Use monotonic deque for rolling min_by/max_by (O(n)) (#27436)

✨ Enhancements

  • Pre-partition group-by on hive keys (#28444)
  • Add ewm_sum and ewm_sum_by #28151 (#28215)
  • Serve stale records from object_store DNS cache (#28256)
  • Make partition_hive an opt flag (#28345)
  • Add scan_arrow_c_stream (#28340)
  • When displaying IRExprs, only add parentheses when needed for disambiguation (#28319)
  • Support Iceberg field initial-default in native scan_iceberg (#28324)
  • Show only differing dtypes in assert_frame_equal error (#27816)
  • Expose ewm and rolling *_by expressions in Python visitor (#28286)
  • More accurate rolling moments (#28309)
  • Remove hard block on native scan for Iceberg V3 tables (#28277)
  • Add Series.degrees/radians and clarify trig functions use radians (#22961)
  • Merge sorted with multikey (#28067)
  • Add list expression which consistently packs elements together into new List type (#27990)
  • Add read_database Arrow fast-path for "python-oracledb" (#28230)
  • Optimize joins with redundant equi-join keys (#28017)
  • Implement missing datetime options for CastColumnsPolicy (#28084)
  • Add Expr.cat.to and Expr.cat.physical (#28096)
  • Add POLARS_OOC_DISK_BUDGET_MB (#28045)
  • Add build side preference parameter for join (#28154)

🐞 Bug fixes

  • Avoid panic when union slice skips all rows (#28420)
  • Solve panic in dt.replace when there were multiple chunks (#28437)
  • Propagate is_scalar from the input to the output of .sort() and .sort_by() (#28438)
  • Resolve CSV column names overwrite in DSL->IR conversion (#28383)
  • Avoid IEJoin rewrite for Categorical comparisons (#28427)
  • Do not rewrite sort().reverse() to sort(descending=True) when maintain_order=True (#28403)
  • Ensure BinaryView offset+len does not exceed i32::MAX where possible (#28048)
  • Invalid offset in strptime (#28388)
  • Incorrect schema type for decimal <-> primitive division (#28373)
  • Panic in in-memory CSEE handling (#28371)
  • Fix offset in arrow ffi export of sliced struct arrays (#28369)
  • Fix cross filter not applied with sink and CSE (#28297)
  • Fix panic on projection pushdown with caches (#28280)
  • Fix write_json() null values in Array columns being written incorrectly as null (#28330)
  • Fix EntityTooSmall on sink_ipc to S3 (#28255)
  • Raise error instead of silent wrapping for select(len()) (#28355)
  • Float16 groupby aggregates (#28361)
  • Respect lexical ordering of Categorical in top_k/bottom_k (#28359)
  • Support dtype-list schema overrides in scan_csv (#28226)
  • Resolve unknown type in fused multiply add (#28321)
  • Honor aws_checksum_algorithm when provided (#28328)
  • Remove usage of deprecated pyarrow.feather.read_table (#28323)
  • Fix regression in sourcing HF_TOKEN (#28317)
  • Raise on addition/subtraction between temporal and non-temporal Series (#28266)
  • Avoid panic constructing Series from unaligned numpy arrays (#28120)
  • Rewrite with_columns on empty df to select during DSL -> IR (#28298)
  • Fix sink deadlock regression (#28296)
  • Avoid panic when consumer of vertical concat is done (#28272)
  • Harden IPC mmap API (#28258)
  • Classify S3 region request failures as IO errors (#28225)
  • Incorrect inner dtype in ArrayChunked::from_aligned_values (#28249)
  • Raise on implode of Object dtype instead of creating invalid List(Object) (#28246)
  • Return null not 0 for the first unbiased ewm_var/ewm_std value (#28235)
  • Database appends with ADBC shouldn't require "CREATE" privileges unless the table is confirmed not to exist (#28228)
  • Avoid slice pushdown panic on shared cache inputs (#28157)
  • Incorrect dtype for Unknown right literal in arithmetic (#28184)
  • Remove unnecessary unsafe Python object downcast (#28189)
  • Normalize non-native byte order in numpy array construction (#28197)
  • Use-after-free in numpy ufunc FFI (#28195)
  • Unpickling for pl.when().then() (#28165)
  • Reduce unknown casts (#28145)
  • Consider a rolling_rank position invalid if ost.len() < min_samples (#28142)
  • Incorrect broadcast with empty len in pl.concat_list (#28134)

📖 Documentation

  • Update comparison page (#28418)
  • Update GPU support documentation with the cudf-polars 26.06 release (#27830)
  • Fix dev docs being canonical for SEO (#28306)
  • Document experimental HDFS support on-premises (#28115)
  • Add Series.degrees/radians and clarify trig functions use radians (#22961)
  • Add polars-bio to community plugins (#28103)
  • Add AWS Marketplace walkthrough to Cloud User Guide (#28214)
  • Correct merge_sorted() docs also for DataFrame (#28183)
  • Clarify DataFrame.remove combines predicates with AND (#28171)
  • Update config reference (#28150)
  • Correct that merge_sorted() null inputs should be nulls first (#28177)
  • Clarify parts of the Polars Cloud and On-prem user guide (#28170)
  • Add On-Premise release 0.6.3 to releases page (#28175)
  • Update PolarsCloud snippets for 0.9.0 release (#28166)
  • Fix to_titlecase incorrectly claiming alphanumeric separation (#28136)

📦 Build system

  • Set the default dev profile to line-tables-only (#28358)
  • Reduce binary size by optimizing for size in the SQL transpilation (#28336)

🛠️ Other improvements

  • Update AI policy for comments (#28436)
  • Fix flaky ordering expectation in top_k_by (#28435)
  • Ignore all deprecation warnings in Python doctest (#28398)
  • Re-work shuffle parameter for sample() (#27460)
  • Ignore deprecation warnings on to_struct() in doctest (#28370)
  • Update spin dependency to v0.10.1 (#28360)
  • Bloat dispatch reduction (#28337)
  • Show only differing dtypes in assert_frame_equal error (#27816)
  • Accept IR in optimize (#28318)
  • Annotate untyped module constants (#27907)
  • Remove unencountered deny entries (#28260)
  • Bump crossbeam-epoch dependency (#28259)
  • Change object_store custom DNS resolver patch to rev-based (#28241)
  • Mark shape tests as passing (#28133)
  • Attribute annotations for *GroupBy (#27903)
  • Clarify AI policy, explicitly forbid agents from interacting with our repository (#28206)
  • Tighten first-time-contributor screenshot requirements (#28207)
  • Covariant classinstmethod and attribute annotation (#27904)
  • Ignore quickxml DoS rustsec (#28185)
  • Update typos to v1.48.0 (#28162)

Thank you to all our contributors for making this release possible!
@0guban0v, @AnirudhRahul, @Bharath-970, @CodingSelim, @EndPositive, @Jesse-Bakker, @Kevin-Patyk, @LarryHu0217, @MarcoGorelli, @MatteoPossamai, @TNieuwdorp, @alex-h-sun, @alexander-beedie, @azimafroozeh, @borchero, @dancsi, @dependabot[bot], @dsprenkels, @harrywhalen, @jorenham, @kdn36, @makinzm, @malhotrashivam, @mdavis-xyz, @mikhail5555, @mkzung, @mroeschke, @nameexhaustion, @orlp, @r-brink, @raphaelroshan, @ritchie46, @tylerriccio33, @useredsa, @wence-, @yannbolliger, @zichen0116 and dependabot[bot]

Python Polars 1.42.1

Choose a tag to compare

@github-actions github-actions released this 30 Jun 04:58
0df0c25

⚠️ Deprecations

  • Deprecate strict parameter of pl.concat, replace with new how='horizontal_extend' (#27965)

🚀 Performance improvements

  • Add a sampled resolve mode for multi-file parquet metadata (#28111)
  • Speed up small dtype series sums with a upcast sum kernel (#27958)
  • Don't block on path expansion (#28073)

✨ Enhancements

  • Close ResourceWarning leaks in database tests (#28066)

🐞 Bug fixes

  • Incorrect sorted fast-path returning nan for grouped max (#28129)
  • Fix panic on scan_parquet filter of fixed-size binary column (#28122)
  • Fix decimal dynamic float cast (#28126)
  • Fix pyiceberg key dot HDFS prefix filter (#28109)
  • Fix projection pushdown panic on select(len()) after groupby (#28108)
  • Fully resolve remaining ResourceWarning leaks in database/iceberg tests (#28107)
  • Expiry time loading for AWS DeferredRefreshableCredentials (#28099)
  • Close ResourceWarning leaks in database tests (#28066)
  • Avoid panic on temporal extraction for datetime columns with nulls (#28054)
  • Prevent panic in replace when old/new contain Expr or object dtype values (#27433)

📖 Documentation

  • Fix dataset regression in the examples (#28131)
  • Emphasize to generally not ever write Rust tests (#28082)
  • Mention Polars skill in User Guide's LLM section (#28113)

🛠️ Other improvements

  • Disallow usage of tokio::spawn in clippy (#28123)
  • Improve Makefile with respect to venv robustness (#28110)
  • Remove legacy prelude glob imports from leaf files (#28114)
  • Skip pandas 3.0.4 due to pd.TimeDelta segfault (#28125)
  • Update mypy to the new 2.x release (#28116)
  • Ensure pyrefly checks run as part of make pre-commit (#28095)
  • Fix test_select_explode_height_filter_order_by failure when POLARS_MAX_THREADS=2 (#28090)
  • Add Morsel::height (#28060)

Thank you to all our contributors for making this release possible!
@0guban0v, @Kevin-Patyk, @TNieuwdorp, @alexander-beedie, @azimafroozeh, @dsprenkels, @kdn36, @nameexhaustion, @orlp, @raphaelroshan, @ritchie46 and @sar-cheng