Add Python 3.11 to GHA #1090

* add downloads tile (capitalone#1085) * Replace snappy with cramjam * Delete test_no_snappy --------- Co-authored-by: Taylor Turner <[email protected]>

taylorfturner · 2024-03-14T13:14:53Z

@gliptak #1091 merged ... rebase this and we'll take a look. Thanks for the contribution! 🎉

gliptak · 2024-03-14T14:48:32Z

dask packaging changed? https://pypi.org/project/dask/#history

https://github.com/capitalone/DataProfiler/actions/runs/8282560184/job/22663633408?pr=1090

____ ERROR collecting dataprofiler/tests/validators/test_base_validators.py ____
/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/dask/dataframe/__init__.py:22: in _dask_expr_enabled
    import dask_expr  # noqa: F401
E   ModuleNotFoundError: No module named 'dask_expr'

During handling of the above exception, another exception occurred:
dataprofiler/tests/validators/test_base_validators.py:4: in <module>
    from dask import dataframe as dd
/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/dask/dataframe/__init__.py:87: in <module>
    if _dask_expr_enabled():
/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/dask/dataframe/__init__.py:24: in _dask_expr_enabled
    raise ValueError("Must install dask-expr to activate query planning.")
E   ValueError: Must install dask-expr to activate query planning.
=============================== warnings summary ===============================
dataprofiler/tests/profilers/test_histogram_utils.py:35
  /home/runner/work/DataProfiler/DataProfiler/dataprofiler/tests/profilers/test_histogram_utils.py:35: PytestCollectionWarning: cannot collect test class 'TestColumn' because it has a __init__ constructor (from: dataprofiler/tests/profilers/test_histogram_utils.py)
    class TestColumn(NumericStatsMixin):

dataprofiler/tests/profilers/test_numeric_stats_mixin_profile.py:[21](https://github.com/capitalone/DataProfiler/actions/runs/8282560184/job/22663633408?pr=1090#step:6:22)
  /home/runner/work/DataProfiler/DataProfiler/dataprofiler/tests/profilers/test_numeric_stats_mixin_profile.py:21: PytestCollectionWarning: cannot collect test class 'TestColumn' because it has a __init__ constructor (from: dataprofiler/tests/profilers/test_numeric_stats_mixin_profile.py)
    class TestColumn(NumericStatsMixin):

dataprofiler/tests/profilers/test_numeric_stats_mixin_profile.py:[34](https://github.com/capitalone/DataProfiler/actions/runs/8282560184/job/22663633408?pr=1090#step:6:35)
  /home/runner/work/DataProfiler/DataProfiler/dataprofiler/tests/profilers/test_numeric_stats_mixin_profile.py:34: PytestCollectionWarning: cannot collect test class 'TestColumnWProps' because it has a __init__ constructor (from: dataprofiler/tests/profilers/test_numeric_stats_mixin_profile.py)
    class TestColumnWProps(TestColumn):

taylorfturner · 2024-03-14T14:58:47Z

@gliptak yeah I just started seeing this yesterday due to the package change by dask on the 12th. Haven't had the bandwidth to research why. I'd imagine a simple tag to not allow for this version would be a temporary fix to unblock

taylorfturner · 2024-03-14T15:02:57Z

seeing on #1115 too from @carlsonp

gliptak · 2024-03-14T15:13:26Z

https://github.com/capitalone/DataProfiler/actions/runs/8282934358/job/22664948501

2024-03-14T15:03:11.8884631Z WARNING: dask 2024.3.0 does not provide the extra 'dask-expr'

dask/dask-expr#968
dask/dask#10917
https://docs.dask.org/en/stable/changelog.html#v2024-3-0

gliptak · 2024-03-14T15:36:41Z

@taylorfturner corrected dask modules install

now there is a Keras(?) error https://github.com/capitalone/DataProfiler/actions/runs/8283138450/job/22665615645?pr=1090

taylorfturner

rebase onto dev

gliptak · 2024-06-07T20:11:01Z

dask/dask#11038

The advised solution is to upgrade to Dask version 2024.4.1.

@taylorfturner am I to proceed with Dask bump as per above?

https://github.com/capitalone/DataProfiler/actions/runs/9421219740/job/25954837467?pr=1090

=========================== short test summary info ============================
ERROR dataprofiler/tests/validators/test_base_validators.py - TypeError: descriptor '__call__' for 'type' objects doesn't apply to a 'property' object
!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!

gliptak · 2024-06-09T19:13:36Z

https://github.com/capitalone/DataProfiler/actions/runs/9438503984/job/25995558872?pr=1090

___________________ TestEvaluateAccuracy.test_save_conf_mat ____________________
self = <dataprofiler.tests.labelers.test_labeler_utils.TestEvaluateAccuracy testMethod=test_save_conf_mat>
mock_dataframe = <MagicMock name='DataFrame' id='140514062390544'>
mock_report = <MagicMock name='classification_report' id='[140](https://github.com/capitalone/DataProfiler/actions/runs/9438503984/job/25995558872?pr=1090#step:6:141)514060067088'>
    @mock.patch("dataprofiler.labelers.labeler_utils.classification_report")
    @mock.patch("pandas.DataFrame")
    def test_save_conf_mat(self, mock_dataframe, mock_report):
    
        # ideally mock out the actual contents written to file, but
        # would be difficult to get this completely worked out.
        expected_conf_mat = np.array(
            [
                [1, 0, 1],
                [1, 0, 0],
                [0, 1, 2],
            ]
        )
        expected_row_col_names = dict(
            columns=["pred:PAD", "pred:UNKNOWN", "pred:OTHER"],
            index=["true:PAD", "true:UNKNOWN", "true:OTHER"],
        )
>       mock_instance_df = mock.Mock(spec=pd.DataFrame)()
dataprofiler/tests/labelers/test_labeler_utils.py:255: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/unittest/mock.py:1106: in __init__
    _safe_super(CallableMixin, self).__init__(
/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/unittest/mock.py:457: in __init__
    self._mock_add_spec(spec, spec_set, _spec_as_instance, _eat_self)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <[AttributeError('_mock_methods') raised in repr()] Mock object at 0x7fcc6aa22410>
spec = <MagicMock name='DataFrame' id='140514062390544'>, spec_set = None
_spec_as_instance = False, _eat_self = False
    def _mock_add_spec(self, spec, spec_set, _spec_as_instance=False,
                       _eat_self=False):
        if _is_instance_mock(spec):
>           raise InvalidSpecError(f'Cannot spec a Mock object. [object={spec!r}]')
E           unittest.mock.InvalidSpecError: Cannot spec a Mock object. [object=<MagicMock name='DataFrame' id='140514062390544'>]
/opt/hostedtoolcache/Python/3.11.9/x64/lib/python3.11/unittest/mock.py:508: InvalidSpecError

taylorfturner · 2024-06-10T14:33:34Z

.pre-commit-config.yaml

- python-snappy>=0.5.4,
+ cramjam>=2.7.0,


I think this should be reverted 🤔

should match requirements.txt

python-snappy>=0.7.1 supports Python 3.11 by switching over to cramjam and removing libsnappy-dev dependency (I contributed intake/python-snappy#130)

I will work dependencies in a minute

we reverted these cramjam changes due to failing tests locally though in #1122.

do we have to include cramjam + 3.11 in same pr?

I'm game for the 3.11 add

is the test suite running locally for you using DATAPROFILER_SEED=0 python3 -m unittest discover -p "test*.py" and tox -e py311?

dataprofiler/__init__.py

gliptak · 2024-06-10T16:42:59Z

this is the above outstanding test fail python/cpython#87644

DataProfiler/dataprofiler/tests/labelers/test_labeler_utils.py

Line 255 in a448694

mock_instance_df = mock.Mock(spec=pd.DataFrame)()

taylorfturner · 2024-06-10T20:15:15Z

this is the above outstanding test fail python/cpython#87644

DataProfiler/dataprofiler/tests/labelers/test_labeler_utils.py

Line 255 in a448694

mock_instance_df = mock.Mock(spec=pd.DataFrame)()

I see -- you are welcome to propose a fix for this as part of this PR (instead of a separate PR). If you get something operational, we can include this in the 0.12.0 release; otherwise, I will need to deploy without. Thanks, @gliptak!

gliptak · 2024-06-10T21:08:06Z

@taylorfturner I rewrote the test and it ran green locally. please review

also let me know if separate bump PRs would work better (and guide on how you would like to split)

taylorfturner · 2024-06-11T12:36:39Z

.pre-commit-config.yaml

- python-snappy>=0.5.4,
+ cramjam>=2.7.0,


we reverted these cramjam changes due to failing tests locally though in #1122.

do we have to include cramjam + 3.11 in same pr?

I'm game for the 3.11 add

is the test suite running locally for you using DATAPROFILER_SEED=0 python3 -m unittest discover -p "test*.py" and tox -e py311?

taylorfturner · 2024-06-11T12:40:18Z

.pre-commit-config.yaml

@@ -55,7 +55,7 @@ repos:
 pyarrow>=1.0.1,
 chardet>=3.0.4,
 fastavro>=1.0.0.post1,
- python-snappy>=0.5.4,


why remove here? it is kept in requirements.txt file

synced with requirements

taylorfturner · 2024-06-11T12:42:05Z

@taylorfturner I rewrote the test and it ran green locally. please review

also let me know if separate bump PRs would work better (and guide on how you would like to split)

Thanks, @gliptak !

keep this PR but only do 3.11 add and the test update
remove the cramjam / snappy(?)

gliptak · 2024-06-11T13:43:33Z

python-snappy>=0.7.1 bump is required for Python 3.11

taylorfturner · 2024-06-11T14:50:09Z

.pre-commit-config.yaml

@@ -55,7 +55,7 @@ repos:
 pyarrow>=1.0.1,
 chardet>=3.0.4,
 fastavro>=1.0.0.post1,
- python-snappy>=0.5.4,


dataprofiler/__init__.py

dataprofiler/tests/test_data_profiler.py

taylorfturner and others added 2 commits January 16, 2024 16:19

add downloads tile (capitalone#1085)

3d9b5ff

Add Python 3.11 to GHA

516c6f5

gliptak requested a review from a team as a code owner February 2, 2024 18:11

taylorfturner changed the base branch from main to dev February 2, 2024 19:15

Merge branch 'dev' into python1

276861c

taylorfturner assigned gliptak Feb 2, 2024

taylorfturner requested review from taylorfturner, micdavis and ksneab7 February 2, 2024 19:19

taylorfturner changed the base branch from old-old-dev to dev March 7, 2024 12:29

Replace snappy with cramjam (capitalone#1091)

f411110

* add downloads tile (capitalone#1085) * Replace snappy with cramjam * Delete test_no_snappy --------- Co-authored-by: Taylor Turner <[email protected]>

Merge branch 'dev' into python1

5b234fb

Update dask modules

f814ab7

Install dask dataframe

eb9d89e

gliptak added 3 commits March 14, 2024 11:39

Update dask modules in precommit

d23c485

Correct copy/paste error

63acf45

Try again to clear Unicode

bd1874e

taylorfturner requested changes Jun 7, 2024

View reviewed changes

Add py311 to tox

95bba82

auto-merge was automatically disabled June 9, 2024 18:04
Head branch was pushed to by a user without write access

Bump dask to 2024.4.1

c507ade

gliptak mentioned this pull request Jun 9, 2024

Dask Max Version Tag #1121

Open

taylorfturner reviewed Jun 10, 2024

View reviewed changes

Bump python-snappy 0.7.1

56b8b2c

taylorfturner enabled auto-merge (squash) June 10, 2024 14:59

Rewrite labeler test

e118ee8

auto-merge was automatically disabled June 10, 2024 20:51
Head branch was pushed to by a user without write access

Correct isort

9021a7e

gliptak added 2 commits June 10, 2024 17:09

Satisfy black

c594034

And flake8

94236da

taylorfturner reviewed Jun 11, 2024

View reviewed changes

taylorfturner enabled auto-merge (squash) June 11, 2024 12:40

Synced with requirements

6ca2c87

auto-merge was automatically disabled June 11, 2024 13:39
Head branch was pushed to by a user without write access

taylorfturner requested changes Jun 11, 2024

View reviewed changes

taylorfturner enabled auto-merge (squash) June 11, 2024 17:54

taylorfturner approved these changes Jun 11, 2024

View reviewed changes

micdavis approved these changes Jun 12, 2024

View reviewed changes

taylorfturner merged commit 4e4450a into capitalone:dev Jun 12, 2024
5 checks passed

gliptak mentioned this pull request Jun 12, 2024

Add Python 3.11 to GHA capitalone/synthetic-data#346

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Python 3.11 to GHA #1090

Add Python 3.11 to GHA #1090

gliptak commented Feb 2, 2024

taylorfturner commented Feb 2, 2024

gliptak commented Feb 2, 2024

gliptak commented Feb 2, 2024

gliptak commented Feb 2, 2024

gliptak commented Feb 26, 2024

taylorfturner commented Mar 7, 2024

gliptak commented Mar 7, 2024

taylorfturner commented Mar 14, 2024

gliptak commented Mar 14, 2024

taylorfturner commented Mar 14, 2024

taylorfturner commented Mar 14, 2024

gliptak commented Mar 14, 2024

gliptak commented Mar 14, 2024

taylorfturner left a comment

gliptak commented Jun 7, 2024

gliptak commented Jun 9, 2024

taylorfturner Jun 10, 2024

taylorfturner Jun 10, 2024

gliptak Jun 10, 2024

taylorfturner Jun 11, 2024

gliptak commented Jun 10, 2024 •

edited

taylorfturner commented Jun 10, 2024

gliptak commented Jun 10, 2024

taylorfturner Jun 11, 2024

taylorfturner Jun 11, 2024

gliptak Jun 11, 2024

taylorfturner Jun 11, 2024

taylorfturner commented Jun 11, 2024

gliptak commented Jun 11, 2024

taylorfturner Jun 11, 2024

Add Python 3.11 to GHA #1090

Add Python 3.11 to GHA #1090

Conversation

gliptak commented Feb 2, 2024

taylorfturner commented Feb 2, 2024

gliptak commented Feb 2, 2024

gliptak commented Feb 2, 2024

gliptak commented Feb 2, 2024

gliptak commented Feb 26, 2024

taylorfturner commented Mar 7, 2024

gliptak commented Mar 7, 2024

taylorfturner commented Mar 14, 2024

gliptak commented Mar 14, 2024

taylorfturner commented Mar 14, 2024

taylorfturner commented Mar 14, 2024

gliptak commented Mar 14, 2024

gliptak commented Mar 14, 2024

taylorfturner left a comment

Choose a reason for hiding this comment

gliptak commented Jun 7, 2024

gliptak commented Jun 9, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gliptak commented Jun 10, 2024 • edited

taylorfturner commented Jun 10, 2024

gliptak commented Jun 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

taylorfturner commented Jun 11, 2024

gliptak commented Jun 11, 2024

Choose a reason for hiding this comment

gliptak commented Jun 10, 2024 •

edited