Releases: pola-rs/polars
Release list
Python Polars 1.44.2
🛠️ Other improvements
- Backport 1.44.2 (#29216)
Thank you to all our contributors for making this release possible!
@EndPositive, @dsprenkels, @kdn36, @lun3x, @nameexhaustion and @orlp
Python Polars 2.0.0-rc.1
💥 Breaking changes
- Set the default engine for SQL to the streaming engine (#28973)
🚀 Performance improvements
- Reuse native metadata for Iceberg sinks (#29062)
- SQL exists predicate pushdown before subqueries (#29078)
- Collect primitive group-by aggregations into single chunk (#28825)
- Use cost information to determine whether to keep cached nodes (#29042)
- Use scan cardinality in build side if sampling is exhausted (#29040)
✨ Enhancements
- Defer SQL resolving until we convert to IR (#29077)
- Expose Iceberg sink writer options (#29065)
- Support partitioned Iceberg sinks (#28823)
- Expose fixed-array inner product in SQL (#29033)
- Add strip_prefix and strip_suffix to expression names (#29041)
- Use scan information in join reordering (#29038)
- Add cardinality information (#29024)
- Allow customizing explain output for debugging custom io sources (#23978)
- Add more expression properties to
metanamespace (#28091) - Improve SQL coverage (#29010)
- Improve SQL coverage (#29006)
- Add Join reordering. (#28985)
- Merge 2.0 into main (#28982)
- Set the default engine for SQL to the streaming engine (#28973)
🐞 Bug fixes
- Don't keep incrementing morsel id in MergeSortedNode pass-through (#29055)
- Ensure show_graph shows physical plan for auto engine (#29011)
- Assign SQL JOIN literal comparisons to the correct input (#28701)
- Make distributor linearizer capacity consistent (#29057)
- Correct peak visitors, dictionary bounds, and
Datefeature gating (#29031) - Keep extension type schema consistent (#29039)
- Fix
Series.sampleorder when with_replacement=True and shuffle=False (#28990) - Reduce memory in multi_scan pipeline
PostApplyExtraOps(#29049) - Fix SO in rayon caused by join_where (#29051)
- Clarify GPU installation guidance (#29037)
- Remove POLARS_AUTO_STRUCTIFY env var backdoor (#29032)
- Ensure metrics are flushed between phases and before logging (#29014)
- SQL parse a typed DATE literal instead of casting it (#29007)
- Fix parquet decode of plain pages in dictionary column (#29001)
- Reserve fixed-size-list child capacity (#28980)
- Fix SQL casts (#28986)
- Fix incorrect condition in
combine_validities_and_manyand add test (#28909) - Give
sink_batcheslazy: Literal[False]overload a default (#28951)
📖 Documentation
- Update 2.0.0-rc1 migration guide with latest changes (#28983)
- Fix invalid output in DataFrame.sample example (#28998)
🛠️ Other improvements
- Bump rust toolchain (#29080)
- Increase rustc stack size (#29076)
- Log pytest worker crashes and disable restarting (#29058)
- Replace FileInfo row_estimation tuple with typed ScanStats (#29023)
- Use an
ifforcfg(debug_assertions)to fix the benchmark job (#29002) - Remove 2.0 branch from ci-tested branches (#28981)
- Skip tests that are currently failing on windows (#28962)
Thank you to all our contributors for making this release possible!
@0guban0v, @0xC61, @Aniket-a14, @LarryHu0217, @MatteoPossamai, @NeejWeej, @TNieuwdorp, @WaterWhisperer, @adamreeve, @dsprenkels, @jonasdedden, @kdn36, @math-hiyoko, @matthewbayer, @nameexhaustion, @orlp, @ritchie46, @severinh and @sivakumar-mahalingam
Python Polars 1.44.1
Thank you to all our contributors for making this release possible!
@Kevin-Patyk, @henrytsanford, @jonasdedden, @orlp and @ritchie46
Python Polars 1.44.0
⚠️ Deprecations
- Deprecate
rechunkparameter for all read/scan functions (#28063) - Deprecate
Expr.rechunk()(#28692) - Deprecate
struct.rename_fields()with an incorrect number of fields (#28672)
🚀 Performance improvements
- Add private env var toggle for HTTP rate limit (#28882)
- Relax default
CloudRetryConfigfor rate limit stability (#28885) - Elide generic cross join on subquery decorrelation in equality predicates (#28876)
- cache CTE's in SQL layer (#28864)
- Allow CSPE if manual caches are set (#28859)
- Lower unqualified join predicates to inner joins in SQL (#28854)
- Rechunk
maskfor DataFramefilter(#28762) - Improve performance of when/then/otherwise by masking out unevaluated elements (#28498)
- Drop unused projections in filter on streaming engine (#28713)
- Retain Parquet
Partialmetadata on filter (#28737)
✨ Enhancements
- Support integer fixed-array dot products (#28829)
- SQL correlated in (#28927)
- Support more join_types in
join_where(#28880) - Support Iceberg schema evolution (#28794)
- SQL conformance (#28494)
- Introduce
RemoteEngineand a common base class for all engines (#28800) - Support Iceberg table with V3 deletion vectors in native
scan_iceberg(#28772) - Support Iceberg snapshot properties (#28793)
- Expose
sinked_paths_callbackonsink_parquetas unstable parameter (#28814) - Add
infer_schema_filesto CSV inference hint (#28809) - Add native fixed-array dot (#28504)
- Export query metrics to polars_cloud (#28757)
- Support Iceberg object storage paths (#28634)
- More CSEE simplification (#28731)
- Improve error in Series constructor with name in context (#28743)
- Add adaptive HTTP rate-limiter for cloud IO (#28591)
- Add
struct.drop()(#28666) - Improve error message when CSV name de-duplication fails (#28658)
- Always keep first metadata per source for Parquet (#28661)
- Improve plan-time row estimates for multi-file parquet scans (#28380)
- Expose array and plugin function views in Python visitor (#28635)
🐞 Bug fixes
- SQL handle quantified comparisons against a subquery (#28929)
- Fix SQL subquery qualified lowering and cache row-index (#28926)
- Clippy-stable failures (#28919)
- Use uuidv7 for cloud observed queries (#28910)
- Unsoundness in AnyValue::into_static for structs (#28889)
- Invalidate suborders on projection (#28866)
- Ensure SQL function registry is available in isolated subqueries (#28881)
- Handle null in Expr.is_not_nan (#28883)
- Ensure eager
unpivotcolumn selection aligns with the lazy engine (#28846) - Fix assertion error / incorrect data load from parquet written by fastparquet (#28656)
- Fix projection pushdown incorrect
select()height (#28751) - Restore
monitoringas an engine-level parameter (#28821) - Bincode serialization of IR hints (#28837)
- Retain datetime inference across CSV fallback (#28663)
- Avoid chunk fragmentation on bool agg
minandmax(#28789) - Merge join with coalesce and empty suffix (#28799)
- Wrong function name call in test (#28801)
- Preserve row mapping in nested over expressions (#28767)
- Restore CI compatibility with NumPy 2.5.2 (#28782)
- Do not deduplicate inherently nondeterministic plan nodes (#28768)
- Fix OOC spilling of categoricals (#28761)
- Correct partitionable filter handling for pivot (#28749)
- Small typo in docstring (#28763)
- Avoid chunk fragmentation on bool agg
anyandall(#28754) - Ignore nans in (min|max)_horizontal (#28710)
- Correct function display for str.replace_all (#28742)
- Exclude debug tests from nodebug streaming run (#28747)
- Convert predicate series dtype during prefiltered parquet reading (#28711)
- Do not remove nested caches when performing predicate pushdown past a cache node (#28740)
- Correct handling of nulls at the start of business day functions (#28703)
- Do not panic when exceeding Parquet metadata scan cache (#28721)
- Remove cryptographic hashing from CSPE (#28716)
- Unsoundness in rayon block_on (#28709)
- Check for (shallow) IR node equality on top of hashing in CSPE (#28506)
- Don't hand IO plugins a predicate they cannot evaluate (#28643)
- Raise on overflow in Decimal sum instead of silently wrapping (#28688)
- Preserve nulls when importing Arrow maps (#28680)
- Propagate null
bycolumn values inrolling_*_by(#27367) - Fix self-referencing
fieldinstruct.with_fieldswithover(#28678) - Fix Arrow buffer offset for
Utf8andBinary(#28662) - Clamp group-by slice offset (#28579)
- Release the GIL while resolving the schema in
__arrow_c_stream__(#28642) - Fix data corruption on Series from nested Arrow
LargeList(#28632)
📖 Documentation
- Interlink join docstrings (#28907)
- Migration guide for 2.0 release (#28788)
- Clarify
read_database_urirequirements for SQLAlchemy (#28366) - Update engine tags (#28852)
- Clarify "that that side" comment in asof_join (#28847)
- Update mage.ai documentation link (#28813)
- Add engine tags (#28708)
- Document that arg_extremum functions return any index on ties (#28691)
- Fix Polars Cloud API reference link (#28693)
🛠️ Other improvements
- Bump runtime toolchains (#28924)
- Add min-publish-age for cargo (#28906)
- Tighten join invariants DSL (#28913)
- Bump Rust toolchain (#28900)
- Ensure that predicates are pushed down to the scans (#28857)
- Clarify "that that side" comment in asof_join (#28847)
- Restore
monitoringas an engine-level parameter (#28821) - Remove workaround for inconsistent CSE (#28834)
- Introduce
RemoteEngineand a common base class for all engines (#28800) - Add row count and file byte size to sinked paths callback (#28522)
- Use canonical expression IDs in CSEE (#28786)
- Enforce 100% type coverage in CI with Pyrefly (#28795)
- Attribute annotations for
CredentialProviderAWSandCredentialProviderAzure(#28790) - Attribute annotations for
ScanCastOptions(#27906) - Disable credentials in upload chunk size tests (#28745)
- Improve backtrace path formatting (#28732)
- Exclude debug tests from nodebug streaming run (#28747)
- Mark
test_grouped_agg_parametricas slow (and add param ids) (#28715) - Fix first
expectintest_group_by_arg_max_boolean_26978(#28690) - Fix the name of
struct.drop()inrename_fields()deprecation message (#28687) - Add
dist=loadgroupto the pytest config (#28676) - Enable a lot of
may_fail_auto_streamingtests (#28675) - Re-enable
test_extension()for streaming engine (#28611) - Update
test_hive_join_rewrite_semi_jointest to work with streaming engine (#28610)
Thank you to all our contributors for making this release possible!
@0guban0v, @JakubValtar, @Joosboy, @Kevin-Patyk, @MatteoPossamai, @NicoOhR, @TNieuwdorp, @VedantMadane, @aarushkandukoori, @alexander-beedie, @azimafroozeh, @borchero, @carnarez, @dancsi, @dependabot[bot], @dsprenkels, @gautamvarmadatla, @jonasdedden, @jorenham, @kdn36, @lzcmian, @malhotrashivam, @matthewbayer, @mikhail5555, @nameexhaustion, @orlp, @ritchie46, @subotac and dependabot[bot]
Rust Polars 0.55.2
✨ Enhancements
- Add adaptive HTTP rate-limiter for cloud IO (#28591)
🐞 Bug fixes
- Unsoundness in rayon block_on (#28709)
- Check for (shallow) IR node equality on top of hashing in CSPE (#28506)
📖 Documentation
- Document that arg_extremum functions return any index on ties (#28691)
Thank you to all our contributors for making this release possible!
@dancsi, @dsprenkels, @kdn36, @orlp and @ritchie46
Rust Polars 0.55.1
This Releases' DSL matches Python 1.43.2.
🚀 Performance improvements
- Pushdown
len()to concat/union inputs (#28570) - Optimize not(bool_f) to not_bool_f (#28474)
- Split multiplexers that directly scan from in-memory DataFrame (#28376)
- Make DNS cache global (#28352)
- Do not remove cache if predicates not pushed to all inputs (#28341)
- Environment variable for logging slow DNS lookup (#28211)
- Pre-partition on left, right and semi joins on hive partitioned data (#28374)
- Rewrite inner joins on hive partitioned data to a union of partitions filtered joins (#28327)
- Tighten redundant comparisons and propagate filter constraints (#27999)
- Canonicalize predicates (#28212)
- Remove excess memory copy in streaming IPC sink (#28141)
- Parallelize scan_iceberg resolve (#28168)
- Chunk-aware morsel splitting on single-file IO sink (#27950)
- Don't allocate for
full_null()with small lengths (#28181) - Use monotonic deque for rolling min_by/max_by (O(n)) (#27436)
- Add a sampled resolve mode for multi-file parquet metadata (#28111)
- Speed up small dtype series sums with a upcast sum kernel (#27958)
- Don't block on path expansion (#28073)
- Add bytes-based concurrency control for cloud IO (#27924)
- Stop flushing multiplexer on phase change (#28013)
- Prevent empty chunks from triggering copy in rechunk (#27953)
- Improve performance of struct unnest inside list eval on streaming (#27905)
✨ Enhancements
- Add
struct.drop()(#28666) - Improve error message when CSV name de-duplication fails (#28658)
- Always keep first metadata per source for Parquet (#28661)
- Improve plan-time row estimates for multi-file parquet scans (#28380)
- Serialize and deserialize
SinkTypeIR::Partitioned(#28112) - Add
infer_schema_filesparameter toscan_csv(#28440) - Support reading IEEE 754 total order Parquet column order (#27896)
- Expose more Expr nodes for cudf_polars (pt 2) (#28404)
- Allow callback sinks on cloud (#28458)
- Pre-partition group-by on hive keys (#28444)
- Add ewm_sum and ewm_sum_by #28151 (#28215)
- Serve stale records from object_store DNS cache (#28256)
- Expose more expr nodes for cudf-polars (#28117)
- Make partition_hive an opt flag (#28345)
- When displaying IRExprs, only add parentheses when needed for disambiguation (#28319)
- Show only differing dtypes in
assert_frame_equalerror (#27816) - Add Clone to SpillFrame/Token (#28302)
- Add Series.degrees/radians and clarify trig functions use radians (#22961)
- Merge sorted with multikey (#28067)
- Add
listexpression which consistently packs elements together into new List type (#27990) - Construct pyarrow predicates via PyO3 instead of eval str (#27271)
- Optimize joins with redundant equi-join keys (#28017)
- Implement missing datetime options for
CastColumnsPolicy(#28084) - Add Expr.cat.to and Expr.cat.physical (#28096)
- Add POLARS_OOC_DISK_BUDGET_MB (#28045)
- Close ResourceWarning leaks in database tests (#28066)
- Expose
IRFunctionExpr::Fusedin the python node visitor (#27889) - Deprecate casts from string to temporal dtypes (#28056)
- Add naive out-of-core spilling to Polars (#27998)
- Add experimental strict mode (#28023)
- Optimize non-negative len and null_count comparisons (#28002)
- Optimize len and null_count comparisons (#27727)
- Support SQL's implicit
JOINsyntax (#27890) - Sample from bivariate distribution for spill context Thompson sampling (#27973)
- Add
Expr.is_sorted(#26708) - Add suggestions in wrong api usage or mismatched column names (#27954)
- Add external object_store support for non-native schemes (#27871)
- Skip Delta files and parquet row groups using per-field struct statistics (#27887)
- Add re-registration of OOC SpillTokens (#27909)
- Improve IR sortedness detection for with_row_index and explode (#27875)
- Add
DataFrame.is_sorted()(#27870)
🐞 Bug fixes
- Don't hand IO plugins a predicate they cannot evaluate (#28643)
- Don't hang or crash on
sqrt/cbrt/pct_change/ewm_*with a Struct column (#28696) - Raise on overflow in Decimal sum instead of silently wrapping (#28688)
- Preserve nulls when importing Arrow maps (#28680)
- Propagate null
bycolumn values inrolling_*_by(#27367) - Fix self-referencing
fieldinstruct.with_fieldswithover(#28678) - Fix Arrow buffer offset for
Utf8andBinary(#28662) - Clamp group-by slice offset (#28579)
- Ensure categorical
fill_null(strategy="min"/"max")uses lexical min/max (#28636) - Release the GIL while resolving the schema in
__arrow_c_stream__(#28642) - Fix data corruption on Series from nested Arrow
LargeList(#28632) - Properly propagate enum metadata for iceberg column mapping (#28580)
- Correctly write parquet field IDs for enums and categoricals (#28581)
- Fix arrow C interop binview offset (#28623)
- Fix duplicate hive values in hive pre-partition rewrite (#28621)
- Double-counted offset for Arrow export of sliced
Series(Array)(#28602) - Struct/List/Array equality handling in sort and join (#28601)
- Use try_new in StructArray construction in polars-json (#27489)
- Raise ComputeError instead of panicking in repeat_by when output exceeds IdxSize::MAX (#27892)
- Run type coercion pass on pivot's internally generated group_by (#27897)
- More careful slice pushdown into joins (#28578)
- Preserve ordering in sliced unions (#28576)
- Drop input sortedness when casting to a string (#28574)
- Clear sortedness flags in
StringChunkedsubstring kernels (#28573) - Bad mask handling when reading optional parquet column (#28547)
- Any operation on
Unknown(Int)andUnknown(Float)should result inUnknown(Float)(#28545) - Flip
nulls_lastafterExpr.reverse()(#28572) - Fix high blocking thread use in sink_parquet with async local path (#28543)
- Serialize LazyFrames backed by bytes (#28568)
- Release GIL in
SQLContext.execute()(#28549) - Propagate
nulls_lastinfunction_expr_sortedness(#28544) - Incorrect slicing when a join requires sorting (#28541)
- Fix panic on self join of scan_delta / scan_iceberg (#28468)
- Decrement ParkGroup worker count when worker exits (#28492)
- Add missing Send bounds in Connector (#28497)
- Fix UB on first/last_non_null with empty chunk (#28495)
- Do not CSE non-column height expr on streaming engine (#28480)
- Propagate adjusted sortedness through Gather (#28493)
- Deduplicate the expression equality checks (#28483)
- Address 3VL issue with SQL
NOT INinteraction withNULLvalues and joins (#28484) - Use total comparison in Expr literal values (#28482)
- Ensure SQL
SUMandCORRaggregates return NULL for all-null inputs, addTOTAL(#28475) - Share
null_count_dtypehelper between Delta and Iceberg, fixingSchemaError(#28479) - Remove non-output columns from the equi-join and semi/anti-join operators (#28446)
- Fix dropped slice on multiple unions (#28477)
- Fix in-memory engine incorrect slice on maintain order join (#28478)
- Check join schema by position (#28455)
- Block predicate pushdown past overwritten window keys (#28429)
- Avoid panic when union slice skips all rows (#28420)
- Solve panic in
dt.replacewhen there were multiple chunks (#28437) - Propagate
is_scalarfrom the input to the output of.sort()and.sort_by()(#28438) - Insert missing coercions from
Unknown(_)in list/array arithmetic (#28411) - Resolve CSV column names overwrite in DSL->IR conversion (#28383)
- Avoid IEJoin rewrite for Categorical comparisons (#28427)
- Do not rewrite
sort().reverse()tosort(descending=True)whenmaintain_order=True(#28403) - Propagate bigidx to
polars-planfrompolars(#28396) - Ensure BinaryView offset+len does not exceed i32::MAX where possible (#28048)
- Invalid offset in strptime (#28388)
- Incorrect schema type for decimal <-> primitive division (#28373)
- Panic in in-memory CSEE handling (#28371)
- Apply same type coercion to IsBetween as binary comparisons (#28300)
- Fix offset in arrow ffi export of sliced struct arrays (#28369)
- Fix cross filter not applied with sink and CSE (#28297)
- Fix panic on projection pushdown with caches (#28280)
- Fix
write_json()null values inArraycolumns being written incorrectly asnull(#28330) - Fix EntityTooSmall on sink_ipc to S3 (#28255)
- Raise error instead of silent wrapping for
select(len())(#28355) - Float16 groupby aggregates (#28361)
- Respect lexical ordering of Categorical in
top_k/bottom_k(#28359) - Support dtype-list schema overrides in scan_csv (#28226)
- Resolve unknown type in fused multiply add (#28321)
- Honor
aws_checksum_algorithmwhen provided (#28328) - Fix regression in sourcing HF_TOKEN (#28317)
- Raise on addition/subtraction between temporal and non-temporal Series (#28266)
- Rewrite with_columns on empty df to select during DSL -> IR (#28298)
- Fix sink deadlock regression (#28296)
- Avoid panic when consumer of vertical concat is done (#28272)
- Ensure predicate simplification is deterministically ordered (#28269)
- Don't attribute untimed execution to optimization in profile (#28273)
- Harden IPC mmap API (#28258)
- Classify S3 region request failures as IO errors (#28225)
- Incorrect inner dtype in ArrayChunked::from_aligned_values (#28249)
- Unsoundness in arg_sort_ascending (#28245)
- Add column length check to row encoding (#28248)
- Raise on implode of
Objectdtype instead of creating invalidList(Object)(#28246) - Don't panic in qcut on degenerate float input (all-NaN, infinite, include_breaks) (#28220)
- Fix streaming Strptime feature flags (#28242)
- Return null not 0 for the first unbiased
ewm_var/ewm_stdvalue (#28235) - Use after free in file cache (#28236)
- Incorrect length calculation in parallel hashmap (#28227)
- Avoid slice pushdown panic on shared cache inputs (#28157)
- Incorrect dtype for Unknown rig...
Python Polars 1.43.2
⚠️ Deprecations
- Deprecate casts from
Categoricalto integer dtypes (#28525) - Deprecate not setting the
plan_stageargument inshow_graph()(#28391)
🚀 Performance improvements
- Pushdown
len()to concat/union inputs (#28570)
✨ Enhancements
- Issue FutureWarning on using
from_arrowof ArrowStreamExportable (#28442) - Add
infer_schema_filesparameter toscan_csv(#28440)
🐞 Bug fixes
- Properly propagate enum metadata for iceberg column mapping (#28580)
- Correctly write parquet field IDs for enums and categoricals (#28581)
- Fix arrow C interop binview offset (#28623)
- Fix duplicate hive values in hive pre-partition rewrite (#28621)
- Double-counted offset for Arrow export of sliced
Series(Array)(#28602) - Struct/List/Array equality handling in sort and join (#28601)
- More careful slice pushdown into joins (#28578)
- Preserve ordering in sliced unions (#28576)
- Drop input sortedness when casting to a string (#28574)
- Clear sortedness flags in
StringChunkedsubstring kernels (#28573) - Bad mask handling when reading optional parquet column (#28547)
- Any operation on
Unknown(Int)andUnknown(Float)should result inUnknown(Float)(#28545) - Flip
nulls_lastafterExpr.reverse()(#28572) - Fix high blocking thread use in sink_parquet with async local path (#28543)
- Serialize LazyFrames backed by bytes (#28568)
- Release GIL in
SQLContext.execute()(#28549) - Propagate
nulls_lastinfunction_expr_sortedness(#28544) - Incorrect slicing when a join requires sorting (#28541)
🛠️ Other improvements
- Fix duckdb delta extension install collision (#28607)
Thank you to all our contributors for making this release possible!
@BitWeaverDev, @Matt711, @Samoilov2004, @borchero, @dancsi, @dsprenkels, @kdn36, @masumi-ryugo, @nameexhaustion and @ritchie46
Python Polars 1.43.1
🚀 Performance improvements
- Optimize not(bool_f) to not_bool_f (#28474)
✨ Enhancements
- Allow callback sinks on cloud (#28458)
🐞 Bug fixes
- Fix panic on self join of scan_delta / scan_iceberg (#28468)
- Fix UB on first/last_non_null with empty chunk (#28495)
- Do not CSE non-column height expr on streaming engine (#28480)
- Propagate adjusted sortedness through Gather (#28493)
- Deduplicate the expression equality checks (#28483)
- Address 3VL issue with SQL
NOT INinteraction withNULLvalues and joins (#28484) - Use total comparison in Expr literal values (#28482)
- Ensure SQL
SUMandCORRaggregates return NULL for all-null inputs, addTOTAL(#28475) - Share
null_count_dtypehelper between Delta and Iceberg, fixingSchemaError(#28479) - Remove non-output columns from the equi-join and semi/anti-join operators (#28446)
- Fix dropped slice on multiple unions (#28477)
- Fix in-memory engine incorrect slice on maintain order join (#28478)
- Check join schema by position (#28455)
- Block predicate pushdown past overwritten window keys (#28429)
📖 Documentation
- Update and restructure README (#28490)
- Minor fixes to Spark migration guide (#27730)
- Relocate Polars Cloud & On-Prem User Guide (#28462)
- Add notes on k8s operator (#28445)
🛠️ Other improvements
- Make hive_part extraction a function and public (#28507)
- Deduplicate the expression equality checks (#28483)
- Bump
ruffandmypypackage versions (#28456) - Fix flaky test (#28454)
Thank you to all our contributors for making this release possible!
@AnirudhRahul, @EndPositive, @Jesse-Bakker, @alexander-beedie, @carnarez, @dancsi, @mdroogh, @mroeschke, @nameexhaustion, @nchammas, @orlp, @r-brink and @ritchie46
Python Polars 1.43.0
⚠️ Deprecations
- Deprecate casting numeric types to categoricals (#28349)
- Deprecate
cat.get_categories()andcat.to_local()(#28299) - Deprecate
LazyFrame.profile()(#28275) - Deprecate casts from non-nested dtypes into Lists (#28193)
- Deprecate bitwise ops between integers and booleans (#28186)
- Deprecate
list/arr.to_struct()calls that don't pass field names (#28243) - Rename
missing_utf8_is_empty_stringtoempty_string_is_null(#28173)
🚀 Performance improvements
- Split multiplexers that directly scan from in-memory DataFrame (#28376)
- Make DNS cache global (#28352)
- Do not remove cache if predicates not pushed to all inputs (#28341)
- Environment variable for logging slow DNS lookup (#28211)
- Pre-partition on left, right and semi joins on hive partitioned data (#28374)
- Rewrite inner joins on hive partitioned data to a union of partitions filtered joins (#28327)
- Tighten redundant comparisons and propagate filter constraints (#27999)
- Canonicalize predicates (#28212)
- Remove excess memory copy in streaming IPC sink (#28141)
- Parallelize scan_iceberg resolve (#28168)
- Chunk-aware morsel splitting on single-file IO sink (#27950)
- Don't allocate for
full_null()with small lengths (#28181) - Use monotonic deque for rolling min_by/max_by (O(n)) (#27436)
✨ Enhancements
- Pre-partition group-by on hive keys (#28444)
- Add ewm_sum and ewm_sum_by #28151 (#28215)
- Serve stale records from object_store DNS cache (#28256)
- Make partition_hive an opt flag (#28345)
- Add
scan_arrow_c_stream(#28340) - When displaying IRExprs, only add parentheses when needed for disambiguation (#28319)
- Support Iceberg field
initial-defaultin nativescan_iceberg(#28324) - Show only differing dtypes in
assert_frame_equalerror (#27816) - Expose ewm and rolling *_by expressions in Python visitor (#28286)
- More accurate rolling moments (#28309)
- Remove hard block on native scan for Iceberg V3 tables (#28277)
- Add Series.degrees/radians and clarify trig functions use radians (#22961)
- Merge sorted with multikey (#28067)
- Add
listexpression which consistently packs elements together into new List type (#27990) - Add
read_databaseArrow fast-path for "python-oracledb" (#28230) - Optimize joins with redundant equi-join keys (#28017)
- Implement missing datetime options for
CastColumnsPolicy(#28084) - Add Expr.cat.to and Expr.cat.physical (#28096)
- Add POLARS_OOC_DISK_BUDGET_MB (#28045)
- Add build side preference parameter for join (#28154)
🐞 Bug fixes
- Avoid panic when union slice skips all rows (#28420)
- Solve panic in
dt.replacewhen there were multiple chunks (#28437) - Propagate
is_scalarfrom the input to the output of.sort()and.sort_by()(#28438) - Resolve CSV column names overwrite in DSL->IR conversion (#28383)
- Avoid IEJoin rewrite for Categorical comparisons (#28427)
- Do not rewrite
sort().reverse()tosort(descending=True)whenmaintain_order=True(#28403) - Ensure BinaryView offset+len does not exceed i32::MAX where possible (#28048)
- Invalid offset in strptime (#28388)
- Incorrect schema type for decimal <-> primitive division (#28373)
- Panic in in-memory CSEE handling (#28371)
- Fix offset in arrow ffi export of sliced struct arrays (#28369)
- Fix cross filter not applied with sink and CSE (#28297)
- Fix panic on projection pushdown with caches (#28280)
- Fix
write_json()null values inArraycolumns being written incorrectly asnull(#28330) - Fix EntityTooSmall on sink_ipc to S3 (#28255)
- Raise error instead of silent wrapping for
select(len())(#28355) - Float16 groupby aggregates (#28361)
- Respect lexical ordering of Categorical in
top_k/bottom_k(#28359) - Support dtype-list schema overrides in scan_csv (#28226)
- Resolve unknown type in fused multiply add (#28321)
- Honor
aws_checksum_algorithmwhen provided (#28328) - Remove usage of deprecated
pyarrow.feather.read_table(#28323) - Fix regression in sourcing HF_TOKEN (#28317)
- Raise on addition/subtraction between temporal and non-temporal Series (#28266)
- Avoid panic constructing Series from unaligned numpy arrays (#28120)
- Rewrite with_columns on empty df to select during DSL -> IR (#28298)
- Fix sink deadlock regression (#28296)
- Avoid panic when consumer of vertical concat is done (#28272)
- Harden IPC mmap API (#28258)
- Classify S3 region request failures as IO errors (#28225)
- Incorrect inner dtype in ArrayChunked::from_aligned_values (#28249)
- Raise on implode of
Objectdtype instead of creating invalidList(Object)(#28246) - Return null not 0 for the first unbiased
ewm_var/ewm_stdvalue (#28235) - Database appends with
ADBCshouldn't require "CREATE" privileges unless the table is confirmed not to exist (#28228) - Avoid slice pushdown panic on shared cache inputs (#28157)
- Incorrect dtype for Unknown right literal in arithmetic (#28184)
- Remove unnecessary unsafe Python object downcast (#28189)
- Normalize non-native byte order in numpy array construction (#28197)
- Use-after-free in numpy ufunc FFI (#28195)
- Unpickling for pl.when().then() (#28165)
- Reduce unknown casts (#28145)
- Consider a
rolling_rankposition invalid ifost.len() < min_samples(#28142) - Incorrect broadcast with empty len in pl.concat_list (#28134)
📖 Documentation
- Update comparison page (#28418)
- Update GPU support documentation with the cudf-polars 26.06 release (#27830)
- Fix dev docs being canonical for SEO (#28306)
- Document experimental HDFS support on-premises (#28115)
- Add Series.degrees/radians and clarify trig functions use radians (#22961)
- Add polars-bio to community plugins (#28103)
- Add AWS Marketplace walkthrough to Cloud User Guide (#28214)
- Correct
merge_sorted()docs also forDataFrame(#28183) - Clarify DataFrame.remove combines predicates with AND (#28171)
- Update config reference (#28150)
- Correct that
merge_sorted()null inputs should be nulls first (#28177) - Clarify parts of the Polars Cloud and On-prem user guide (#28170)
- Add On-Premise release 0.6.3 to releases page (#28175)
- Update PolarsCloud snippets for 0.9.0 release (#28166)
- Fix to_titlecase incorrectly claiming alphanumeric separation (#28136)
📦 Build system
- Set the default
devprofile toline-tables-only(#28358) - Reduce binary size by optimizing for size in the SQL transpilation (#28336)
🛠️ Other improvements
- Update AI policy for comments (#28436)
- Fix flaky ordering expectation in top_k_by (#28435)
- Ignore all deprecation warnings in Python doctest (#28398)
- Re-work
shuffleparameter forsample()(#27460) - Ignore deprecation warnings on
to_struct()in doctest (#28370) - Update
spindependency tov0.10.1(#28360) - Bloat dispatch reduction (#28337)
- Show only differing dtypes in
assert_frame_equalerror (#27816) - Accept IR in
optimize(#28318) - Annotate untyped module constants (#27907)
- Remove unencountered deny entries (#28260)
- Bump crossbeam-epoch dependency (#28259)
- Change
object_storecustom DNS resolver patch to rev-based (#28241) - Mark shape tests as passing (#28133)
- Attribute annotations for
*GroupBy(#27903) - Clarify AI policy, explicitly forbid agents from interacting with our repository (#28206)
- Tighten first-time-contributor screenshot requirements (#28207)
- Covariant
classinstmethodand attribute annotation (#27904) - Ignore quickxml DoS rustsec (#28185)
- Update
typostov1.48.0(#28162)
Thank you to all our contributors for making this release possible!
@0guban0v, @AnirudhRahul, @Bharath-970, @CodingSelim, @EndPositive, @Jesse-Bakker, @Kevin-Patyk, @LarryHu0217, @MarcoGorelli, @MatteoPossamai, @TNieuwdorp, @alex-h-sun, @alexander-beedie, @azimafroozeh, @borchero, @dancsi, @dependabot[bot], @dsprenkels, @harrywhalen, @jorenham, @kdn36, @makinzm, @malhotrashivam, @mdavis-xyz, @mikhail5555, @mkzung, @mroeschke, @nameexhaustion, @orlp, @r-brink, @raphaelroshan, @ritchie46, @tylerriccio33, @useredsa, @wence-, @yannbolliger, @zichen0116 and dependabot[bot]
Python Polars 1.42.1
⚠️ Deprecations
- Deprecate
strictparameter ofpl.concat, replace with newhow='horizontal_extend'(#27965)
🚀 Performance improvements
- Add a sampled resolve mode for multi-file parquet metadata (#28111)
- Speed up small dtype series sums with a upcast sum kernel (#27958)
- Don't block on path expansion (#28073)
✨ Enhancements
- Close ResourceWarning leaks in database tests (#28066)
🐞 Bug fixes
- Incorrect sorted fast-path returning nan for grouped max (#28129)
- Fix panic on scan_parquet filter of fixed-size binary column (#28122)
- Fix decimal dynamic float cast (#28126)
- Fix pyiceberg key dot HDFS prefix filter (#28109)
- Fix projection pushdown panic on
select(len())aftergroupby(#28108) - Fully resolve remaining
ResourceWarningleaks in database/iceberg tests (#28107) - Expiry time loading for AWS
DeferredRefreshableCredentials(#28099) - Close ResourceWarning leaks in database tests (#28066)
- Avoid panic on temporal extraction for datetime columns with nulls (#28054)
- Prevent panic in
replacewhenold/newcontain Expr or object dtype values (#27433)
📖 Documentation
- Fix dataset regression in the examples (#28131)
- Emphasize to generally not ever write Rust tests (#28082)
- Mention Polars skill in User Guide's LLM section (#28113)
🛠️ Other improvements
- Disallow usage of
tokio::spawnin clippy (#28123) - Improve
Makefilewith respect tovenvrobustness (#28110) - Remove legacy prelude glob imports from leaf files (#28114)
- Skip pandas 3.0.4 due to
pd.TimeDeltasegfault (#28125) - Update
mypyto the new2.xrelease (#28116) - Ensure
pyreflychecks run as part ofmake pre-commit(#28095) - Fix
test_select_explode_height_filter_order_byfailure whenPOLARS_MAX_THREADS=2(#28090) - Add Morsel::height (#28060)
Thank you to all our contributors for making this release possible!
@0guban0v, @Kevin-Patyk, @TNieuwdorp, @alexander-beedie, @azimafroozeh, @dsprenkels, @kdn36, @nameexhaustion, @orlp, @raphaelroshan, @ritchie46 and @sar-cheng