Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Polars assumes microseconds instead of reading numpy timedelta units #16029

Open
2 tasks done
hixan opened this issue May 3, 2024 · 1 comment · May be fixed by #16332
Open
2 tasks done

Polars assumes microseconds instead of reading numpy timedelta units #16029

hixan opened this issue May 3, 2024 · 1 comment · May be fixed by #16332
Labels
A-interop Area: interoperability with other libraries A-interop-numpy Area: interoperability with NumPy A-timeseries Area: date/time functionality bug Something isn't working P-medium Priority: medium python Related to Python Polars

Comments

@hixan
Copy link

hixan commented May 3, 2024

Checks

  • I have checked that this issue has not already been reported.
  • I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import numpy as np, numpy as np
print(pl.DataFrame([[np.timedelta64(1, 'ns')]]))
>>> shape: (1, 1)
┌──────────────┐
│ column_0     │
│ ---          │
│ duration[μs] │
╞══════════════╡
│ 1µs          │
└──────────────┘

Log output

No response

Issue description

Polars does not seem to read the time unit associated with the numpy timedelta64, and instead assumes microsecond.

Expected behavior

The returned column should have the same time unit as the numpy object (in this case ns)

Installed versions

--------Version info---------
Polars:               0.20.23
Index type:           UInt32
Platform:             Linux-5.14.0-284.18.1.el9_2.x86_64-x86_64-with-glibc2.28
Python:               3.11.8 (main, Feb 16 2024, 19:42:16) [GCC 8.5.0 20210514 (Red Hat 8.5.0-20)]

----Optional dependencies----
adbc_driver_manager:  <not installed>
cloudpickle:          3.0.0
connectorx:           <not installed>
deltalake:            <not installed>
fastexcel:            <not installed>
fsspec:               2024.3.1
gevent:               <not installed>
hvplot:               <not installed>
matplotlib:           3.8.4
nest_asyncio:         1.6.0
numpy:                1.26.4
openpyxl:             3.1.1
pandas:               1.5.3
pyarrow:              11.0.0
pydantic:             1.10.15
pyiceberg:            <not installed>
pyxlsb:               <not installed>
sqlalchemy:           1.4.52
xlsx2csv:             <not installed>
xlsxwriter:           <not installed>
@hixan hixan added bug Something isn't working needs triage Awaiting prioritization by a maintainer python Related to Python Polars labels May 3, 2024
@MarcoGorelli MarcoGorelli added A-interop Area: interoperability with other libraries P-medium Priority: medium A-interop-numpy Area: interoperability with NumPy A-timeseries Area: date/time functionality and removed needs triage Awaiting prioritization by a maintainer labels May 3, 2024
@datenzauberai
Copy link
Contributor

datenzauberai commented May 17, 2024 •
edited

Handling of time units and proper casting is missing when the numpy values are inside a python list instead of a numpy arrray:

pl.DataFrame({
    # ok
    "np_array_ns": np.array([np.timedelta64(1000000000, "ns"), np.timedelta64(1000000000, "ns")]),
    # ok
    "np_array_us": np.array([np.timedelta64(1000000, "us"),    np.timedelta64(1000000, "us")]),
    # ok
    "np_array_ms": np.array([np.timedelta64(1000, "ms"),    np.timedelta64(1000, "ms")]),
    # Ignores time unit="ns"
    "py_list_ns": [np.timedelta64(1000000000, "ns"), np.timedelta64(1000000000, "ns")],
    # ComputeError: cannot cast 'Object' type
    "py_list_us": [np.timedelta64(1000000, "us"),    np.timedelta64(1000000, "us")],
    # ComputeError: cannot cast 'Object' type
    "py_list_ms": [np.timedelta64(1000, "ms"),    np.timedelta64(1000, "ms")],
})

@datenzauberai datenzauberai linked a pull request May 20, 2024 that will close this issue
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-interop Area: interoperability with other libraries A-interop-numpy Area: interoperability with NumPy A-timeseries Area: date/time functionality bug Something isn't working P-medium Priority: medium python Related to Python Polars
Projects
Backlog
Status: Ready
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(python): Casting and time unit handling for lists of temporal numpy types datenzauberai/polars
3 participants
@datenzauberai @hixan @MarcoGorelli