v0.9.0

[qinxuye]

This is the release notes of v0.9.0. See here for the complete list of solved issues and merged PRs.

This release note only covers the difference from v0.9.0rc3; for all highlights and changes, please refer to the release notes of the pre-releases:

alpha1
alpha2
beta1
beta2
rc1
rc2
rc3

Highlights

• Performance is fully optimized in this version, welcome to give your feedback.

New Features

• Oscar
  • Stop importing main module when starting Mars local cluster ([BACKPORT] Stop importing main module when starting Mars local cluster (#3110) #3113)
• Tensor
  • Integrate special error functions ([BACKPORT] Integrate special error functions (#3060) #3062)
  • Integrate part of scipy elliptic functions and integrals ([BACKPORT] Integrate part of scipy elliptic functions and integrals (#3111) #3112)
• DataFrame
  • Support sort=True for Groupby ([BACKPORT] Support sort=True for Groupby (#2959) #3063, thanks @sak2002!)

Enhancements

• Dump remote tracebacks to make local ones more friendly ([BACKPORT] Dump remote tracebacks to make local ones more friendly (#3028) #3030)
• Optimize import speed for Mars package ([BACKPORT] Optimize import speed for Mars package (#3022) #3035)
• [Ray] Implement ray task executor progress ([BACKPORT][Ray] Implement ray task executor progress (#3008) #3065)
• Shuffle both sides at the same time for md.merge ([BACKPORT] Shuffle both sides at the same time for md.merge (#3041) #3066)
• Refine ThreadedServiceContext.get_chunks_meta usage ([BACKPORT] Refine ThreadedServiceContext.get_chunks_meta usage (#3037) #3067)
• Do not aggressively choose tree method in tile of groupby for distributed setting ([BACKPORT] Do not aggressively choose tree method in tile of groupby for distributed setting (#3032) #3070)
• Disable bloom filter in merge for now ([BACKPORT] Disable bloom filter in merge for now (#2967) #3071)
• [Ray] Implements get_chunks_result for Ray execution context ([BACKPORT][Ray] Implements get_chunks_result for Ray execution context (#3023) #3072)
• Use tell when remove mapper data after execution ([BACKPORT] Use tell when remove mapper data after execution (#3027) #3073)
• Assign reducer ops in task assigner to make them more balanced across cluster ([BACKPORT] Assign reducer ops in task assigner to make them more balanced across cluster (#3048) #3075)
• [Ray] Destroy Ray executor when the task finish ([BACKPORT][Ray] Destroy Ray executor when the task finish (#3049) #3074)
• Combine tree and shuffle methods in DataFrameGroupBy.agg tile ([BACKPORT] Combine tree and shuffle methods in DataFrameGroupBy.agg tile (#3051) #3077)
• [Ray] Implements get_chunks_meta for Ray execution context ([BACKPORT][Ray] Implements get_chunks_meta for Ray execution context (#3052) #3076)
• Use OS-designated ports instead of random ports to create sub pools ([BACKPORT] Use OS-designated ports instead of random ports to create sub pools (#3053) #3087)
• Call immutable web API only once when previous call blocks ([BACKPORT] Call immutable web API only once when previous call blocks (#3085) #3088)
• Unify DataFrameGroupByAgg's tile logic for auto method ([BACKPORT] Unify DataFrameGroupByAgg's tile logic for auto method (#3084) #3094)
• [Ray] Support basic subtask retry and lineage reconstruction ([BACKPORT] [Ray] Support basic subtask retry and lineage reconstruction (#2969) #3097)
• Simplify argument passing in actor batch calls ([BACKPORT] Simplify argument passing in actor batch calls (#3098) #3100)
• [Ray] Implements get_total_n_cpu for Ray execution context ([BACKPORT] [Ray] Implements get_total_n_cpu for Ray execution context (#3059) #3104)
• Optimize performance of transfer ([BACKPORT] Optimize performance of transfer (#3091) #3105)
• Add n_reducers and reducer_ordinal to shuffle operands ([BACKPORT] Add n_reducers and reducer_ordinal to shuffle operands (#3055) #3107)
• [Ray] Implement cancel method on Ray task executor ([BACKPORT][Ray] Implement cancel method on Ray task executor (#3044) #3093)
• [Ray] Create RayTaskState actor as needed by default ([BACKPORT] [Ray] Create RayTaskState actor as needed by default (#3081) #3114)
• [Ray] Implement gc for ray task executor context ([BACKPORT][Ray] Implement gc for ray task executor context (#3061) #3116)
• Optimize serializable memory ([BACKPORT] Optimize serializable memory (#3120) #3126)

Bug fixes

• Patch pandas to make pickle compatible between 1.2 and 1.3 ([BACKPORT] Patch pandas to make pickle compatible between 1.2 and 1.3 (#3047) #3050)
• Fix errors when deleting mapper data ([BACKPORT] Fix errors when deleting mapper data (#3018) #3064)
• Fix chunk index error in auto_merge_chunks ([BACKPORT] Fix chunk index error in auto_merge_chunks (#3057) #3068)
• Fix recursive_tile that it may cause duplicated tile for one tileable ([BACKPORT] Fix recursive_tile that it may cause duplicated tile for one tileable (#3021) #3069)
• [Ray] Fix ray worker failover ([BACKPORT] [Ray] Fix ray worker failover (#3080) #3115)
• [Ray] Fix pandas schema parsing when reading Ray dataset ([BACKPORT][Ray] Fix pandas schema parsing when reading Ray dataset (#2946) #3117)
• [Ray] fix auto scale-in hang ([BACKPORT] [Ray] fix auto scale-in hang (#3043) #3125)
• [Metric] Fix prometheus metric backend ([BACKPORT] [Metric] Fix prometheus metric backend (#3124) #3127)
• Fix mt.{cumsum, cumprod} when the first chunk is empty ([BACKPORT] Fix mt.{cumsum, cumprod} when the first chunk is empty (#3134) #3136)

Tests

• Check initialization of serializables on CI ([BACKPORT] Check initialization of serializables on CI (#3007) #3013)
• [Ray] Optimize Ray CI execution time and stability ([BACKPORT][Ray] Optimize Ray CI execution time and stability (#3102) #3121)
• Update pytest imports for test_special.py ([BACKPORT] Update pytest imports for test_special.py (#3129) #3131)
• [Ray] Fix flaky test test_optional_supervisor_node ([BACKPORT][Ray] Fix flaky test test_optional_supervisor_node (#3133) #3135)

Others

• Build web code before CIBW when deploying to PyPI ([BACKPORT] Build web code before CIBW when deploying to PyPI (#3014) #3016)

---

This discussion was created from the release v0.9.0.