Update to Airflow 2.0.0 #618

neylsoncrepalde · 2020-12-19T16:26:06Z

This is a pull request to update your repo to work with Airflow 2.0.0 and Python 3.8. Everything was tested locally and is apparently working fine.

The main changes were:

update commands worker and flower to celery in scripts/entrypoint.sh
update image name in docker-compose files
update Dockerfile to use python:3.8-slim-buster
changed airflow.cfg for the 2.0.0 version default one
set up a default username and password to log in airflow (needed for 2.0.0 version).

Many thanks for your hard work putting this together and I hope this pull request can be handy.

rikr2sanchez · 2020-12-28T23:45:45Z

After login the app crash:

Something bad has happened.
Please consider letting us know by creating a bug report using GitHub.

Python version: 3.8.7
Airflow version: 2.0.0
Node: 7de345de891a
-------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/local/lib/python3.8/site-packages/airflow/www/auth.py", line 34, in decorated
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/www/views.py", line 491, in index
    filter_dag_ids = current_app.appbuilder.sm.get_accessible_dag_ids(g.user)
  File "/usr/local/lib/python3.8/site-packages/airflow/www/security.py", line 250, in get_accessible_dag_ids
    accessible_dags = self.get_accessible_dags(
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/session.py", line 65, in wrapper
    return func(*args, session=session, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/www/security.py", line 264, in get_accessible_dags
    joinedload(User.roles)
AttributeError: '_UnboundLoad' object has no attribute 'options'

shihabcsedu09 · 2020-12-29T11:23:09Z

Hi @neylsoncrepalde

I am getting the following error while running the container after building it.

Unable to load the config, contains a configuration error.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/logging/config.py", line 563, in configure
    handler = self.configure_handler(handlers[name])
  File "/usr/local/lib/python3.8/logging/config.py", line 744, in configure_handler
    result = factory(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 43, in __init__
    self.filename_template, self.filename_jinja_template = parse_template_string(filename_template)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/helpers.py", line 134, in parse_template_string
    return None, Template(template_string)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 1031, in __new__
    return env.from_string(source, template_class=cls)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 941, in from_string
    return cls.from_code(self, self.compile(source), globals, None)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 638, in compile
    self.handle_exception(source=source_hint)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/usr/local/lib/python3.8/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "<unknown>", line 1, in template
jinja2.exceptions.TemplateSyntaxError: expected token ':', got '}'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 5, in <module>
    from airflow.__main__ import main
  File "/usr/local/lib/python3.8/site-packages/airflow/__init__.py", line 46, in <module>
    settings.initialize()
  File "/usr/local/lib/python3.8/site-packages/airflow/settings.py", line 432, in initialize
    LOGGING_CLASS_PATH = configure_logging()
  File "/usr/local/lib/python3.8/site-packages/airflow/logging_config.py", line 62, in configure_logging
    raise e
  File "/usr/local/lib/python3.8/site-packages/airflow/logging_config.py", line 57, in configure_logging
    dictConfig(logging_config)
  File "/usr/local/lib/python3.8/logging/config.py", line 808, in dictConfig
    dictConfigClass(config).configure()
  File "/usr/local/lib/python3.8/logging/config.py", line 570, in configure
    raise ValueError('Unable to configure handler '
ValueError: Unable to configure handler 'processor'
Unable to load the config, contains a configuration error.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/logging/config.py", line 563, in configure
    handler = self.configure_handler(handlers[name])
  File "/usr/local/lib/python3.8/logging/config.py", line 744, in configure_handler
    result = factory(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 43, in __init__
    self.filename_template, self.filename_jinja_template = parse_template_string(filename_template)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/helpers.py", line 134, in parse_template_string
    return None, Template(template_string)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 1031, in __new__
    return env.from_string(source, template_class=cls)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 941, in from_string
    Unable to load the config, contains a configuration error.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/logging/config.py", line 563, in configure
return cls.from_code(self, self.compile(source), globals, None)    handler = self.configure_handler(handlers[name])
  File "/usr/local/lib/python3.8/logging/config.py", line 744, in configure_handler

  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 638, in compile
    result = factory(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 43, in __init__
    self.filename_template, self.filename_jinja_template = parse_template_string(filename_template)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/helpers.py", line 134, in parse_template_string
    return None, Template(template_string)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 1031, in __new__
    return env.from_string(source, template_class=cls)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 941, in from_string
        return cls.from_code(self, self.compile(source), globals, None)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 638, in compile
self.handle_exception(source=source_hint)    self.handle_exception(source=source_hint)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception

  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/usr/local/lib/python3.8/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "<unknown>", line 1, in template
jinja2.exceptions.TemplateSyntaxError: expected token ':', got '}'

neylsoncrepalde · 2020-12-29T14:35:15Z

Hello @shihabcsedu09 and @rikr2sanchez.

Could you please provide information on the commands you ran to get the error you described?

I tried locally and everything is working fine. Maybe, the problem is that up to this point, there is no public docker image puckel/docker-airflow:2.0.0 so I had to build it. In order to do that, I ran

git clone https://github.com/neylsoncrepalde/docker-airflow.git
cd docker-airflow
docker build -t puckel/docker-airflow:2.0.0 .
docker-compose -f docker-compose-CeleryExecutor.yml up

Many thanks! Best!

shihabcsedu09 · 2020-12-30T08:28:53Z

@neylsoncrepalde Ignore it. I can now run this. I didn't use the updated cfg file. Thanks for this pull request

rikr2sanchez · 2020-12-30T15:37:42Z

Hello @shihabcsedu09 and @rikr2sanchez.

Could you please provide information on the commands you ran to get the error you described?

I tried locally and everything is working fine. Maybe, the problem is that up to this point, there is no public docker image puckel/docker-airflow:2.0.0 so I had to build it. In order to do that, I ran
git clone https://github.com/neylsoncrepalde/docker-airflow.git
cd docker-airflow
docker build -t puckel/docker-airflow:2.0.0 .
docker-compose -f docker-compose-CeleryExecutor.yml up
Many thanks! Best!

Up and running ! Thanks !

BenoitHugonnard

VERSION should be changed on top of the Dockerfile
REDIS_HOST and REDIS_PORT in the entrypoint.sh should be changed to use REDIS_ENDPOINT

brow86 · 2021-02-22T19:30:15Z

Hi @neylsoncrepalde

I am getting the following error while running the container after building it.

Unable to load the config, contains a configuration error.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/logging/config.py", line 563, in configure
    handler = self.configure_handler(handlers[name])
  File "/usr/local/lib/python3.8/logging/config.py", line 744, in configure_handler
    result = factory(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 43, in __init__
    self.filename_template, self.filename_jinja_template = parse_template_string(filename_template)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/helpers.py", line 134, in parse_template_string
    return None, Template(template_string)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 1031, in __new__
    return env.from_string(source, template_class=cls)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 941, in from_string
    return cls.from_code(self, self.compile(source), globals, None)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 638, in compile
    self.handle_exception(source=source_hint)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/usr/local/lib/python3.8/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "<unknown>", line 1, in template
jinja2.exceptions.TemplateSyntaxError: expected token ':', got '}'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 5, in <module>
    from airflow.__main__ import main
  File "/usr/local/lib/python3.8/site-packages/airflow/__init__.py", line 46, in <module>
    settings.initialize()
  File "/usr/local/lib/python3.8/site-packages/airflow/settings.py", line 432, in initialize
    LOGGING_CLASS_PATH = configure_logging()
  File "/usr/local/lib/python3.8/site-packages/airflow/logging_config.py", line 62, in configure_logging
    raise e
  File "/usr/local/lib/python3.8/site-packages/airflow/logging_config.py", line 57, in configure_logging
    dictConfig(logging_config)
  File "/usr/local/lib/python3.8/logging/config.py", line 808, in dictConfig
    dictConfigClass(config).configure()
  File "/usr/local/lib/python3.8/logging/config.py", line 570, in configure
    raise ValueError('Unable to configure handler '
ValueError: Unable to configure handler 'processor'
Unable to load the config, contains a configuration error.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/logging/config.py", line 563, in configure
    handler = self.configure_handler(handlers[name])
  File "/usr/local/lib/python3.8/logging/config.py", line 744, in configure_handler
    result = factory(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 43, in __init__
    self.filename_template, self.filename_jinja_template = parse_template_string(filename_template)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/helpers.py", line 134, in parse_template_string
    return None, Template(template_string)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 1031, in __new__
    return env.from_string(source, template_class=cls)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 941, in from_string
    Unable to load the config, contains a configuration error.
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/logging/config.py", line 563, in configure
return cls.from_code(self, self.compile(source), globals, None)    handler = self.configure_handler(handlers[name])
  File "/usr/local/lib/python3.8/logging/config.py", line 744, in configure_handler

  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 638, in compile
    result = factory(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/log/file_processor_handler.py", line 43, in __init__
    self.filename_template, self.filename_jinja_template = parse_template_string(filename_template)
  File "/usr/local/lib/python3.8/site-packages/airflow/utils/helpers.py", line 134, in parse_template_string
    return None, Template(template_string)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 1031, in __new__
    return env.from_string(source, template_class=cls)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 941, in from_string
        return cls.from_code(self, self.compile(source), globals, None)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 638, in compile
self.handle_exception(source=source_hint)    self.handle_exception(source=source_hint)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception

  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/usr/local/lib/python3.8/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "<unknown>", line 1, in template
jinja2.exceptions.TemplateSyntaxError: expected token ':', got '}'

I hit this today too, got it to work by using:

log_filename_template = {{ ti.dag_id }}/{{ ti.task_id }}/{{ ts }}/{{ try_number }}.log
log_processor_filename_template = {{ filename }}.log

with the default 1.10.14 from github, in case anyone else runs into this.

levyitay · 2021-03-31T04:15:26Z

@neylsoncrepalde
Using docker-compose-LocalExecuter.yml does not work when building locally, but does work with your image: neylsoncrepalde/airflow-docker
This is the Error I'm getting:

webserver_1  | INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
webserver_1  | INFO  [alembic.runtime.migration] Will assume transactional DDL.
webserver_1  | Traceback (most recent call last):
webserver_1  |   File "/usr/local/bin/airflow", line 8, in <module>
webserver_1  |     sys.exit(main())
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
webserver_1  |     args.func(args)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
webserver_1  |     return func(*args, **kwargs)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/cli/commands/db_command.py", line 31, in initdb
webserver_1  |     db.initdb()
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/utils/db.py", line 549, in initdb
webserver_1  |     upgradedb()
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/utils/db.py", line 688, in upgradedb
webserver_1  |     command.upgrade(config, 'heads')
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/command.py", line 294, in upgrade
webserver_1  |     script.run_env()
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/script/base.py", line 490, in run_env
webserver_1  |     util.load_python_file(self.dir, "env.py")
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 97, in load_python_file
webserver_1  |     module = load_module_py(module_id, path)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/util/compat.py", line 182, in load_module_py
webserver_1  |     spec.loader.exec_module(module)
webserver_1  |   File "<frozen importlib._bootstrap_external>", line 783, in exec_module
webserver_1  |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/migrations/env.py", line 108, in <module>
webserver_1  |     run_migrations_online()
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/migrations/env.py", line 102, in run_migrations_online
webserver_1  |     context.run_migrations()
webserver_1  |   File "<string>", line 8, in run_migrations
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/runtime/environment.py", line 813, in run_migrations
webserver_1  |     self.get_context().run_migrations(**kw)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/runtime/migration.py", line 548, in run_migrations
webserver_1  |     for step in self._migrations_fn(heads, self):
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/command.py", line 283, in upgrade
webserver_1  |     return script._upgrade_revs(revision, rev)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/script/base.py", line 365, in _upgrade_revs
webserver_1  |     revs = list(revs)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/script/revision.py", line 916, in _iterate_revisions
webserver_1  |     uppers = util.dedupe_tuple(self.get_revisions(upper))
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/script/revision.py", line 457, in get_revisions
webserver_1  |     resolved_id, branch_label = self._resolve_revision_number(id_)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/script/revision.py", line 640, in _resolve_revision_number
webserver_1  |     self._revision_map
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/util/langhelpers.py", line 234, in __get__
webserver_1  |     obj.__dict__[self.__name__] = result = self.fget(obj)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/script/revision.py", line 156, in _revision_map
webserver_1  |     for revision in self._generator():
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/script/base.py", line 115, in _load_revisions
webserver_1  |     script = Script._from_filename(self, vers, file_)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/script/base.py", line 904, in _from_filename
webserver_1  |     module = util.load_python_file(dir_, filename)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/util/pyfiles.py", line 97, in load_python_file
webserver_1  |     module = load_module_py(module_id, path)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/alembic/util/compat.py", line 182, in load_module_py
webserver_1  |     spec.loader.exec_module(module)
webserver_1  |   File "<frozen importlib._bootstrap_external>", line 783, in exec_module
webserver_1  |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/migrations/versions/2c6edca13270_resource_based_permissions.py", line 27, in <module>
webserver_1  |     from airflow.www.app import create_app
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/www/app.py", line 38, in <module>
webserver_1  |     from airflow.www.extensions.init_views import (
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/www/extensions/init_views.py", line 29, in <module>
webserver_1  |     from airflow.www.views import lazy_add_provider_discovered_options_to_connection_form
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/www/views.py", line 95, in <module>
webserver_1  |     from airflow.www import auth, utils as wwwutils
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/www/utils.py", line 27, in <module>
webserver_1  |     from flask_appbuilder.models.sqla.interface import SQLAInterface
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/models/sqla/interface.py", line 16, in <module>
webserver_1  |     from sqlalchemy_utils.types.uuid import UUIDType
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy_utils/__init__.py", line 1, in <module>
webserver_1  |     from .aggregates import aggregated  # noqa
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy_utils/aggregates.py", line 372, in <module>
webserver_1  |     from .functions.orm import get_column_key
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy_utils/functions/__init__.py", line 1, in <module>
webserver_1  |     from .database import (  # noqa
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy_utils/functions/database.py", line 11, in <module>
webserver_1  |     from .orm import quote
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy_utils/functions/orm.py", line 14, in <module>
webserver_1  |     from sqlalchemy.orm.query import _ColumnEntity
webserver_1  | ImportError: cannot import name '_ColumnEntity' from 'sqlalchemy.orm.query' (/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py)
webserver_1  | Traceback (most recent call last):
webserver_1  |   File "/usr/local/bin/airflow", line 8, in <module>
webserver_1  |     sys.exit(main())
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
webserver_1  |     args.func(args)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 47, in command
webserver_1  |     func = import_string(import_path)
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/utils/module_loading.py", line 32, in import_string
webserver_1  |     module = import_module(module_path)
webserver_1  |   File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
webserver_1  |     return _bootstrap._gcd_import(name[level:], package, level)
webserver_1  |   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
webserver_1  |   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
webserver_1  |   File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
webserver_1  |   File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
webserver_1  |   File "<frozen importlib._bootstrap_external>", line 783, in exec_module
webserver_1  |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/cli/commands/user_command.py", line 29, in <module>
webserver_1  |     from airflow.www.app import cached_app
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/www/app.py", line 38, in <module>
webserver_1  |     from airflow.www.extensions.init_views import (
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/www/extensions/init_views.py", line 29, in <module>
webserver_1  |     from airflow.www.views import lazy_add_provider_discovered_options_to_connection_form
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/www/views.py", line 95, in <module>
webserver_1  |     from airflow.www import auth, utils as wwwutils
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/airflow/www/utils.py", line 27, in <module>
webserver_1  |     from flask_appbuilder.models.sqla.interface import SQLAInterface
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/models/sqla/interface.py", line 16, in <module>
webserver_1  |     from sqlalchemy_utils.types.uuid import UUIDType
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy_utils/__init__.py", line 1, in <module>
webserver_1  |     from .aggregates import aggregated  # noqa
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy_utils/aggregates.py", line 372, in <module>
webserver_1  |     from .functions.orm import get_column_key
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy_utils/functions/__init__.py", line 1, in <module>
webserver_1  |     from .database import (  # noqa
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy_utils/functions/database.py", line 11, in <module>
webserver_1  |     from .orm import quote
webserver_1  |   File "/usr/local/lib/python3.8/site-packages/sqlalchemy_utils/functions/orm.py", line 14, in <module>
webserver_1  |     from sqlalchemy.orm.query import _ColumnEntity
webserver_1  | ImportError: cannot import name '_ColumnEntity' from 'sqlalchemy.orm.query' (/usr/local/lib/python3.8/site-packages/sqlalchemy/orm/query.py)

levyitay · 2021-03-31T04:25:16Z

@neylsoncrepalde The only thing that fixed that for me is adding
pip install SQLAlchemy==1.3.23 \
in the Dockerfile

K-7 · 2021-04-15T12:42:30Z

I dont run celery , I use executor = LocalExecutor. After doing the migration to 2.0 when I run my DAG i get the following error

[INFO] Handling signal: ttin
[INFO] Booting worker with pid: 32308
INFO - DAG test already has 18 active runs, not queuing any tasks for run 2021-04-15
[INFO] Worker exiting (pid: 458)
[INFO] Handling signal: ttou

Not sure what exactly is wrong

cjphaha · 2021-05-13T09:14:09Z

This is a pull request to update your repo to work with Airflow 2.0.0 and Python 3.8. Everything was tested locally and is apparently working fine.

The main changes were:

update commands worker and flower to celery in scripts/entrypoint.sh

update image name in docker-compose files

update Dockerfile to use python:3.8-slim-buster

changed airflow.cfg for the 2.0.0 version default one

set up a default username and password to log in airflow (needed for 2.0.0 version).

Many thanks for your hard work putting this together and I hope this pull request can be handy.

thank u bro, u have solved my problem!👍

elliot-jsq · 2021-06-07T23:02:12Z

Are there any remaining issues for this PR or is this ready to go?

agung2001 · 2021-08-08T07:13:20Z

Hello @shihabcsedu09 and @rikr2sanchez.

Could you please provide information on the commands you ran to get the error you described?

I tried locally and everything is working fine. Maybe, the problem is that up to this point, there is no public docker image puckel/docker-airflow:2.0.0 so I had to build it. In order to do that, I ran
git clone https://github.com/neylsoncrepalde/docker-airflow.git
cd docker-airflow
docker build -t puckel/docker-airflow:2.0.0 .
docker-compose -f docker-compose-CeleryExecutor.yml up
Many thanks! Best!

thx mate, you save my day, I can run smoothly upgrading from version 1.10.9 to 2.0.0

neylsoncrepalde added 7 commits December 19, 2020 12:19

changing to airflow 2.0.0

0549ea7

fix worker connection to redis

be381af

rename docker-compose-CeleryExecutor.yml

06ea878

fix image names

70a06d0

update to airflow 2.0.0 and Python 3.8

b7a3897

default webserver credentials

22c2941

leave the same volumes as puckels

6473088

change all image tags in readme to 2.0.0

2fe7ff5

shihabcsedu09 approved these changes Dec 30, 2020

View reviewed changes

BenoitHugonnard reviewed Feb 17, 2021

View reviewed changes

abachu approved these changes Mar 29, 2021

View reviewed changes

ednarb29 mentioned this pull request Jan 26, 2022

Bumped up Airflow version to 2.2.3 #651

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to Airflow 2.0.0 #618

Update to Airflow 2.0.0 #618

neylsoncrepalde commented Dec 19, 2020

rikr2sanchez commented Dec 28, 2020 •

edited

Loading

shihabcsedu09 commented Dec 29, 2020 •

edited

Loading

neylsoncrepalde commented Dec 29, 2020

shihabcsedu09 commented Dec 30, 2020

rikr2sanchez commented Dec 30, 2020

BenoitHugonnard left a comment •

edited

Loading

brow86 commented Feb 22, 2021 •

edited

Loading

levyitay commented Mar 31, 2021

levyitay commented Mar 31, 2021

K-7 commented Apr 15, 2021

cjphaha commented May 13, 2021

elliot-jsq commented Jun 7, 2021

agung2001 commented Aug 8, 2021

Update to Airflow 2.0.0 #618

Are you sure you want to change the base?

Update to Airflow 2.0.0 #618

Conversation

neylsoncrepalde commented Dec 19, 2020

rikr2sanchez commented Dec 28, 2020 • edited Loading

shihabcsedu09 commented Dec 29, 2020 • edited Loading

neylsoncrepalde commented Dec 29, 2020

shihabcsedu09 commented Dec 30, 2020

rikr2sanchez commented Dec 30, 2020

BenoitHugonnard left a comment • edited Loading

Choose a reason for hiding this comment

brow86 commented Feb 22, 2021 • edited Loading

levyitay commented Mar 31, 2021

levyitay commented Mar 31, 2021

K-7 commented Apr 15, 2021

cjphaha commented May 13, 2021

elliot-jsq commented Jun 7, 2021

agung2001 commented Aug 8, 2021

rikr2sanchez commented Dec 28, 2020 •

edited

Loading

shihabcsedu09 commented Dec 29, 2020 •

edited

Loading

BenoitHugonnard left a comment •

edited

Loading

brow86 commented Feb 22, 2021 •

edited

Loading