Skip to content

Commit

Permalink
Fix set logging level in Python Scrapy template
Browse files Browse the repository at this point in the history
  • Loading branch information
vdusek committed Oct 2, 2023
1 parent ec76f9b commit 464188b
Show file tree
Hide file tree
Showing 2 changed files with 23 additions and 12 deletions.
12 changes: 12 additions & 0 deletions templates/python-scrapy/pr.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
**Description**

- I formatted the `__main__.py` using Isort & Autopep8.
- Now the `setLevel` works.

**Ticket**

- Closes #185

**Test plan**

- @fnesveda Franto could you tell me what were your concerns about the specific import order (especially `scrapy.utils.log`)? So I could test if this is still a problem. Because currently, I don't see any. Thanks.
23 changes: 11 additions & 12 deletions templates/python-scrapy/src/__main__.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,17 @@
# This file is executed only when the project is run as an Apify Actor,
# not when you run it locally using `scrapy crawl`.

# We need to configure the logging first before we import anything else,
# so that nothing else imports `scrapy.utils.log` before we patch it.
import asyncio
import logging
from apify.log import ActorLogFormatter
import os

import nest_asyncio
import scrapy.utils.log
from scrapy.utils.reactor import install_reactor

from apify.log import ActorLogFormatter

from .main import main

handler = logging.StreamHandler()
handler.setFormatter(ActorLogFormatter())
Expand All @@ -18,7 +24,6 @@
apify_client_logger.setLevel(logging.DEBUG)
apify_client_logger.addHandler(handler)


# We can't attach our log handler to the loggers normally,
# because Scrapy would remove them in the `configure_logging` call here:
# https://github.com/scrapy/scrapy/blob/a5c1ef82762c6c0910abea00c0a6249c40005e44/scrapy/utils/log.py#L95
Expand All @@ -28,6 +33,7 @@
# because otherwise we would lose some log messages.
old_configure_logging = scrapy.utils.log.configure_logging


def new_configure_logging(*args, **kwargs):
old_configure_logging(*args, **kwargs)

Expand All @@ -37,17 +43,10 @@ def new_configure_logging(*args, **kwargs):
logging.getLogger('filelock').addHandler(handler)
logging.getLogger('hpack').addHandler(handler)

scrapy.utils.log.configure_logging = new_configure_logging

scrapy.utils.log.configure_logging = new_configure_logging

# Now we can do the rest of the setup
import asyncio
import os

import nest_asyncio

from scrapy.utils.reactor import install_reactor
from .apify.main import main

# This is necessary so that twisted and asyncio work well together
install_reactor('twisted.internet.asyncioreactor.AsyncioSelectorReactor')
Expand Down

0 comments on commit 464188b

Please sign in to comment.