Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add storages and memory storage clients #82

Merged
merged 47 commits into from
Apr 4, 2024
Merged

Add storages and memory storage clients #82

merged 47 commits into from
Apr 4, 2024

Conversation

vdusek
Copy link
Collaborator

@vdusek vdusek commented Mar 28, 2024 •
edited
Loading

Description

  • Copy & align of storages & memory storage clients (and all related utilities) from SDK.
  • Docstring improvements.
  • Test coverage improvements.
  • Tried to replace all the occurrences of Apify with Crawlee.
  • Added (dataclass) typing - get rid of general dicts.
  • Rm unnecessary class attrs.
  • Rm wrap_internal hacking.
  • Fixed all the linter & type checker issues.

Issue

@github-actions github-actions bot added this to the 86th sprint - Tooling team milestone Mar 28, 2024
@github-actions github-actions bot added t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics. labels Mar 28, 2024
github-actions[bot]
github-actions bot reviewed Mar 28, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Pull Request Tookit has failed!

Pull request is neither linked to an issue or epic nor labeled as adhoc!

github-actions[bot]
github-actions bot reviewed Mar 28, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Pull Request Tookit has failed!

Pull request is neither linked to an issue or epic nor labeled as adhoc!

vdusek added 23 commits March 29, 2024 17:37
@vdusek
Copy from SDK: storages & clients
a40e533
@vdusek
Copy from SDK: Dataset, Request Queue
cdd0409
@vdusek
Utils are copied from SDK and tests are updated
e2981c1
@vdusek
only 3 tests are failing
89b1993
@vdusek
rm memory_storage (without _)
f5529c5
@vdusek
tests are almost ok
0235fa7
@vdusek
importing from crawlee only
67096d8
@vdusek
First update of Dataset & RQ
8435895
@vdusek
Memory storage & tests work
d63e96f
@vdusek
Memory storage client update
ce9f3d7
@vdusek
First update of storages
1e7c08a
@vdusek
Storages seem to be working
03cd40c
@vdusek
Tests working
b211ba6
@vdusek
Add tests for LRUCache
2bf7a30
@vdusek
Add tests for crypto module
f0c87a2
@vdusek
Add tests for request helpers
9f35c9a
@vdusek
filter_out_none_values_recursively is tested
ee0d830
@vdusek
add tests for data processing utils
5a03905
@vdusek
Add some tests for file utilities
f7cc60b
@vdusek
File utilities update and refactor
f425211
@vdusek
just some reordering
48a2f02
@vdusek
Refactor and fix linter issues for resource clients
c2a4c5c
@vdusek
Refactor and fix linter issues for storages
e7d5dfe
@vdusek vdusek marked this pull request as ready for review April 2, 2024 12:34
@vdusek vdusek changed the title [WIP] Add storages and memory storage clients Add storages and memory storage clients Apr 2, 2024
@vdusek vdusek requested a review from janbuchar April 2, 2024 13:40
janbuchar
janbuchar reviewed Apr 2, 2024
View reviewed changes
Copy link
Collaborator

@janbuchar janbuchar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First batch of comments, please forgive me if it sounds harsh - I was trying to be quick

src/crawlee/_utils/data_processing.py Outdated Show resolved Hide resolved
src/crawlee/_utils/env_vars.py Show resolved Hide resolved
src/crawlee/_utils/file.py Outdated Show resolved Hide resolved
src/crawlee/_utils/file.py Outdated Show resolved Hide resolved
src/crawlee/_utils/file.py Show resolved Hide resolved
src/crawlee/config.py Outdated Show resolved Hide resolved
src/crawlee/consts.py Outdated Show resolved Hide resolved
src/crawlee/memory_storage/base_resource_client.py Outdated Show resolved Hide resolved
src/crawlee/memory_storage/base_resource_client.py Outdated Show resolved Hide resolved
src/crawlee/memory_storage/base_resource_client.py Show resolved Hide resolved
@vdusek
Copy link
Collaborator Author

vdusek commented Apr 3, 2024

Issue for refactoring the init process of storages - #85.

@vdusek vdusek mentioned this pull request Apr 3, 2024
Merged
@vdusek vdusek mentioned this pull request Apr 3, 2024
Open
@vdusek vdusek requested a review from janbuchar April 3, 2024 16:34
janbuchar
janbuchar approved these changes Apr 4, 2024
View reviewed changes
Copy link
Collaborator

@janbuchar janbuchar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't find much that should be fixed in this PR, except one nit. Let's merge, I guess.

src/crawlee/storages/base_storage.py Outdated Show resolved Hide resolved
@vdusek vdusek merged commit 2891d12 into master Apr 4, 2024
19 checks passed
@vdusek vdusek deleted the add-storages branch April 4, 2024 15:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@janbuchar janbuchar janbuchar approved these changes

Assignees

@vdusek vdusek

Labels
t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.
Projects
None yet
Milestone
86th sprint - Tooling team
Development

Successfully merging this pull request may close these issues.

Implement local storages & storage clients
2 participants
@vdusek @janbuchar