Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

favorites --stop_after=N stops after min(N, 200) #50

Open
mikepqr opened this issue Sep 11, 2020 · 2 comments
Open

favorites --stop_after=N stops after min(N, 200) #50

mikepqr opened this issue Sep 11, 2020 · 2 comments

Comments

@mikepqr
Copy link
Contributor

mikepqr commented Sep 11, 2020

For any number greater than 200, favorites --stop_after stops after getting 200 tweets, e.g.

$ twitter-to-sqlite favorites tweets.db --stop_after=300
Importing favorites  [####################################]  199
$

I don't think this is a limitation of the API (if you omit --stop_after you get some very large number, possibly all of them), so I think this is a bug.

@mikepqr
Copy link
Contributor Author

mikepqr commented Sep 11, 2020

There's probably a nicer way of doing (hence this is a comment rather than a PR), but this appears to fix it:

--- a/twitter_to_sqlite/utils.py
+++ b/twitter_to_sqlite/utils.py
@@ -181,6 +181,7 @@ def fetch_timeline(
     args["tweet_mode"] = "extended"
     min_seen_id = None
     num_rate_limit_errors = 0
+    seen_count = 0
     while True:
         if min_seen_id is not None:
             args["max_id"] = min_seen_id - 1
@@ -208,6 +209,7 @@ def fetch_timeline(
             yield tweet
         min_seen_id = min(t["id"] for t in tweets)
         max_seen_id = max(t["id"] for t in tweets)
+        seen_count += len(tweets)
         if last_since_id is not None:
             max_seen_id = max((last_since_id, max_seen_id))
             last_since_id = max_seen_id
@@ -217,7 +219,9 @@ def fetch_timeline(
                 replace=True,
             )
         if stop_after is not None:
-            break
+            if seen_count >= stop_after:
+                break
+            args["count"] = min(args["count"], stop_after - seen_count)
         time.sleep(sleep)

@bcongdon
Copy link

This seems to be an issue even with larger values of --stop_after:

$ twitter-to-sqlite favorites twitter.db --stop_after=2000
Importing favorites  [####################################]  198
$

@mikepqr mikepqr changed the title favorites --stop_after=300 stops after 200 favorites --stop_after=N stops after min(N, 200) Sep 13, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants