Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Respect ACTOR_MAX_PAID_DATASET_ITEMS when pushing to dataset #304

Open
B4nan opened this issue Jun 5, 2024 · 0 comments
Open

Respect ACTOR_MAX_PAID_DATASET_ITEMS when pushing to dataset #304

B4nan opened this issue Jun 5, 2024 · 0 comments
Assignees
Labels
t-tooling Issues with this label are in the ownership of the tooling team.

Comments

@B4nan
Copy link
Member

B4nan commented Jun 5, 2024

Feature

For PPR actors we need a way to limit the results.

PoC: https://github.com/metalwarrior665/max-paid-items-example/blob/master/src/push-data.ts

Motivation

..

Ideal solution or implementation, and any additional constraints

Respect the ACTOR_MAX_PAID_DATASET_ITEMS env var when calling pushData. Have a custom Dataset implementation for the SDK similar to what we do with KVS, it will override the core method on require time, that way it will work with the crawlee method too (including the one from context to allow adaptive crawling).

Alternative solutions or implementations

No response

Other context

https://apify.slack.com/archives/CD0SF6KD4/p1717532797512629

@B4nan B4nan self-assigned this Jun 5, 2024
@B4nan B4nan added the t-tooling Issues with this label are in the ownership of the tooling team. label Jun 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
t-tooling Issues with this label are in the ownership of the tooling team.
Projects
None yet
Development

No branches or pull requests

1 participant