Skip to content
Snippets Groups Projects
  1. Dec 22, 2021
  2. Dec 21, 2021
  3. Dec 09, 2021
  4. Dec 07, 2021
    • Ash Berlin-Taylor's avatar
      Improve handling edge-cases in airlfow.models by applying mypy (#20000) · 7d8e3b82
      Ash Berlin-Taylor authored
      
      * Fix many of the mypy typing issues in airflow.models.dag
      
      And to fix these, I needed to fix a few other mistakes that are
      used/called by DAG's methods
      
      * Fix timetable-related typing errors in dag.py
      
      Also moved the sentinel value implementation to a utils module. This
      should be useful when fixing typing issues in other modules.
      
      * Add note about assert allowed inside a TYPE_CHECKING conditional
      
      * Fix docs build of airflow.models.dagrun
      
      * Apply NEW_SESSION to dag, dagrun, ti and operator.subdag
      
      Co-authored-by: default avatarTzu-ping Chung <tp@astronomer.io>
  5. Nov 12, 2021
  6. Oct 28, 2021
  7. Oct 10, 2021
  8. Sep 28, 2021
    • Kanthi's avatar
      Influxdb Hook (#17068) · c747bce1
      Kanthi authored
      
      * Added transfer operator for bigquery to mssql and tests
      
      * Update airflow/providers/google/cloud/transfers/bigquery_to_mssql.py
      
      Co-authored-by: default avatarTomek Urbaszek <turbaszek@gmail.com>
      
      * Update airflow/providers/google/cloud/transfers/bigquery_to_mssql.py
      
      Code change based on review comments.
      
      Co-authored-by: default avatarTomek Urbaszek <turbaszek@gmail.com>
      
      * Added comment in bigquery to mssql transfer
      
      * Added influx operator files
      
      * Delete bigquery_to_mssql.py
      
      * Update test_bigquery_to_bigquery.py
      
      * Delete test_bigquery_to_mssql.py
      
      * [14168] Added functions to create organization, bucket and run query to influxdb.
      
      * [14168] Added file that was accidentally deleted.
      
      * [14168] Added support to write Point.
      
      * [14168] Added Influx provider support
      
      * [14168] Added Influx provider support
      
      * [14168] Added test case for InfluxDBHook
      
      * [14168] Added test case for InfluxDBHook
      
      * [14168] Fixed test case for InfluxDBHook
      
      * [14168] Fixed test case for InfluxDBHook
      
      * [14168] Added example DAG for influxDBHook
      
      * [14168] Changed org_id to org_name for clarity.
      
      * [14168] Removed README, fixed influxDBHook class name in provider.yml
      
      * [14168] Static code check fixes.
      
      * [14168] fixed license header influxdb.rst for connections
      
      * [14168] fixed flak tests and order of requirements - influxdb
      
      * [14168] Renamed provider.yaml from provider.yml
      
      * [14168] fixed imports in influxdb.py
      
      * [14168] Renamed function for passing flak8 tests.
      
      * [14168] Fixed influxDB connection rst
      
      * [14168] Fixed order in CONTRIBUTING.rst and spelling_wordlist.txt
      
      * [14168] ADded operators/influxdb.rst to satisfy pre commit check
      
      * [14168] fixed docs error influxdb
      
      * [14168] fixed example DAG path for influxDB
      
      * Update airflow/providers/influxdb/hooks/influxdb.py
      
      Co-authored-by: default avatarTzu-ping Chung <uranusjr@gmail.com>
      
      * Rebase from master
      
      * Fixed PR review comments
      
      * Fixed influxdb documentation
      
      * Fixed documentation
      
      * Added empty doc for InfluxDB operator to workaround build-docs error
      
      * Fixed table for influxdb dependency
      
      * Fixed connection-type in influxdb provider
      
      * Added init to example_dags
      
      * Added influxdb to airflow_providers_bug_report
      
      * Added missing init file in influxdb hook
      
      * Added missing init in tests/influxdb
      
      * Removed link in toc tree for influxdb
      
      * Moved influxdb tests from operators to hooks, added tests
      
      * Added more tests for influxdb hook
      
      * Added commits.rst and installing-providers-from-sources.rst for influxdb
      
      * Added commits to toctree of index
      
      * Added pandas dependency to influxdb
      
      Co-authored-by: default avatarKanthi <kanthi@Kanthis-MacBook-Air.local>
      Co-authored-by: default avatarTomek Urbaszek <turbaszek@gmail.com>
      Co-authored-by: default avatarKanthi Subramanian <subkanthi@gmail.ccom>
      Co-authored-by: default avatarTzu-ping Chung <uranusjr@gmail.com>
  9. Aug 25, 2021
  10. Aug 23, 2021
    • Jarek Potiuk's avatar
      Move instriuctions of constraint/image refreshing to dev · 36c5fd3d
      Jarek Potiuk authored
      When we have a prolonged issue with flaky tests or Github runners
      instabilities, our automated constraint and image refresh might
      not work, so we might need to manually refresh the constraints
      and images. Documentation about that was in CONTRIBUTING.rst
      but it is more appriate to keep it in ``dev`` as it only applies
      to committers.
      
      Also during testing the parallell refresh without delays an error
      was discovered  which prevented parallell check of random image
      hash during the build. This has been fixed and parallell
      image cache building should work flawlessly now.
      36c5fd3d
  11. Aug 19, 2021
    • Jarek Potiuk's avatar
      Improve description of Python compatibility approach (#17721) · b5e66615
      Jarek Potiuk authored
      The entry was terribly out-dated. We did not have many
      problems with different python versions backwards compatibilty for
      a looong time. Seems that the new SenVer approach and release
      control and cadence work pretty well comparing to 3.5/3.6 times
      (BTW. good job Python developers!) so I updated the entry to
      reflect the `recommended` approach we should take.
    • D. Ferruzzi's avatar
      Implemented Basic EKS Integration (#16571) · bee48f31
      D. Ferruzzi authored
      * Implemented Basic EKS Integration
      
      * Remove explicit region defaulting
      
      cr https://code.amazon.com/reviews/CR-52973030
      
      * Refactor the token generation and remove the AWS CLI dependency.
      
      * move kubeconfig generator into `hooks/eks.py`
      
      * EKS List hooks return all results
      
      * Use a tempfile to store kubeconfig data
      
      * Move kube config into Hook class as a contextmanager
      
      * Removed random traits in tests
      
      * Rework the eks.rst doc file
      
      * Implemented Jinja templates for operators
      
      - Added jinja template fields
      - Refactored fields to snake_case since they are now exposed
      - Removed a couple of straggling pylint instructions; pylint is no longer used
      
      * conn_id refactor
      
      - Refactored using IDE magic:
       - any field named `self.conn_id` is now `self.aws_conn_id`
       - any param named `conn_id` is now `aws_conn_id`
       - any constant named `CONN_ID` is now `DEFAULT_CONN_ID`
      - Sensors were missing template fields, added those.
      
      * Remove try/log blocks from hooks
      
      * Implemented EKS system tests
      
      * Remove List and Describe Operators and supporting code.
      
      * Fixed `nextToken` final result bug
      
      * Remove some nesting and some unnecessary logging before raising an exception
      
      * Use the force
      
      * Improved docs and samples
      
      * Additional jinja templating
      
      * Corrected some misused Optionals.
      
      * Doc formatting fix
      
      * Corrected logo
      
      * Corrected logo - background and size
  12. Aug 18, 2021
  13. Aug 12, 2021
    • Kaxil Naik's avatar
      Make `pandas` an optional core dependency (#17575) · 2c26b15a
      Kaxil Naik authored
      We only use `pandas` in `DbApiHook.get_pandas_df`. Not all users use it, plus
      while `pandas` now supports many pre-compiled packages it still can take forever where
      it needs to be compiled.
      
      So for first-time users this can be a turn off. If pandas is already installed this
      will work fine, but if not users have an option to run `pip install apache-airflow[pandas]`
      
      closes #12500
  14. Aug 08, 2021
  15. Aug 07, 2021
  16. Jul 21, 2021
  17. Jul 17, 2021
    • Nijanthan Vijayakumar's avatar
      Update CONTRIBUTING.rst and CONTRIBUTORS_QUICK_START.rst files with jq... · 9bc13539
      Nijanthan Vijayakumar authored
      Update CONTRIBUTING.rst and CONTRIBUTORS_QUICK_START.rst files with jq installation instructions (#17060)
      
      * Add jq installation info as a prerequisite for setting up breeze
      
      * Add macOS example for jq installation using Homebrew
      
      The "Configure Your Environment" section was missing the examples on
      installing jq on macOS. Earlier version of the document only had the
      example for the Ubuntu OS.
  18. Jul 14, 2021
  19. Jul 12, 2021
    • dzamo's avatar
    • Jarek Potiuk's avatar
      Switch Breeze/CI to ghcr.io excusively (#16780) · af0598f5
      Jarek Potiuk authored
      Breeze used traditionally DockerHub to pull images, because
      they were public and GitHub Packages were not. With GitHub Container
      Regisry however, we can switch fully to using GitHub Container
      Registry also for Breeze.
      
      Thanks to moving to Github Container Registry we can remove
      a lot of code responsible for maintaining different naming
      and different versions of the images in DockerHub and
      GitHub Container Registry. Also it streamlines and simplifies
      the process of refreshing the images when new python versions
      are released - the CI push builds will check if the new Python
      image is released in DockerHub and it will rebuild the base
      image automatically if needed (and push it as cache)
      
      The CI documentation (including sequence diagrams) has been
      refreshed to reflect those changes (and other changes done in
      the meantime). The flows are now simplified as DockerHub is
      largely moved out of the picture.
      
      The only remaining DockerHub Images now are:
      
      * images used during CI for integrations (airflow-ci)
      * officially released Production Airflow images (airflow)
      
      The integration images will be moved to GitHub Container Registry
      in a subsequent PR and the only images remaining in DockerHub
      will be the officially released Production Airflow images.
      
      Part of #16555
  20. Jul 06, 2021
  21. Jun 28, 2021
  22. Jun 25, 2021
    • Jarek Potiuk's avatar
      Add Python 3.9 support (#15515) · ce44b628
      Jarek Potiuk authored
      This includes several things:
      
      * added per-provider support for python version. Each provider
        can now declare python versions it does not support
      * excluded ldap core extra from Python 3.9.
      * skip relevant tests in Python 3.9
  23. Jun 03, 2021
  24. Jun 01, 2021
  25. May 27, 2021
  26. May 19, 2021
    • Jarek Potiuk's avatar
      Updates branches and branch documentation after 2.1.0rc1 (#15528) · b7096190
      Jarek Potiuk authored
      * Updates branches and branch documentation after 2.1.0rc1
      
      This PR updates branches and corresponding documentation and
      tools after 2.1.0rc1 release.
      
      It describes what needs to be done when new release branch is created,
      and provides tools that allow to do most of the work
      semi-automatically.
      
      Wherever possible and easy, the 2-0 references were replaced with 2-*
      and where it was more difficult, TODOS were left.
      
      The `dev/retag_docker_images.py` script will also be useful
      when we get to renaming the `master` tag to `main` tag.
    • Daniel Standish's avatar
      Move plyvel to google provider extra (#15812) · 76a80bb1
      Daniel Standish authored
      Plyvel does not build on macOS without levelDB installed in system.  Its better to make it an optional install.
  27. May 17, 2021
    • Jennifer Melot's avatar
      Add Asana Provider (#14521) · 162e3204
      Jennifer Melot authored
    • Jarek Potiuk's avatar
      Get rid of requests as core dependency (#15781) · bb115da0
      Jarek Potiuk authored
      This change gets rid of requests as core dependency. We have to
      change requests to become an optional dependency because it
      (so far) pulls in chardet as dependency and chardet is
      LGPL, which is not allowed to be mandatory dependency by
      ASF policies.
      
      More info here:
      
      https://issues.apache.org/jira/browse/LEGAL-572
      
      The changes:
      
      * connexion is vendored-in (and requests usage is replaced with httpx)
      * Http Provider is turned into optional provider (not preinstalled)
      * Few places where requests were used in core and in cloud_sql provider
        which did not cause compatibility problem, it was replaced by httpx.
      * new extra added for deprecated experimental API (which is disabled
        by default and optional)
      * tests are fixed (using pytest-httpx fixture package)
      * The providers: http, airbyte, apache.livy, opsgenie, slack (all depend
        on http) now explicitely depend on `requirements`.
  28. Apr 29, 2021
  29. Apr 27, 2021
    • Jarek Potiuk's avatar
      Use Pip 21.* to install airflow officially (#15513) · e229f354
      Jarek Potiuk authored
      
      * Use Pip 21.* to install airflow officially
      
      The PIP 20.2.4 was so far the only officially supported installation
      mechanism for Airflow as there were some problems with conflicting
      dependencies (which were ignored by previous versio of PIP).
      
      This change attempts to solve this by removing a [gcp] extra
      from `apache-beam` which turns out to be the major source of
      the problem - as it contains requirements to the old version of
      google client libraries (but apparently only used for tests).
      
      The "apache-beam" provider migh however need the [gcp] extra
      for other components so in order to not break the backwards
      compatibility, another approach is used.
      
      Instead of adding [gcp] as extra in the apache-beam extra,
      the apache.beam provider's [google] extra is extended with
      'apache-beam[gcp]' additional requirement so that whenever the
      provider is installed, the apache-beam with [gcp] extra is installed
      as well.
      
      * Update airflow/providers/apache/beam/CHANGELOG.rst
      
      Co-authored-by: default avatarTzu-ping Chung <uranusjr@gmail.com>
      
      * Update airflow/providers/apache/beam/CHANGELOG.rst
      
      Co-authored-by: default avatarTzu-ping Chung <uranusjr@gmail.com>
      
      * Update airflow/providers/google/CHANGELOG.rst
      
      Co-authored-by: default avatarTzu-ping Chung <uranusjr@gmail.com>
      
      * Update airflow/providers/google/CHANGELOG.rst
      
      Co-authored-by: default avatarTzu-ping Chung <uranusjr@gmail.com>
      
      Co-authored-by: default avatarTzu-ping Chung <uranusjr@gmail.com>
  30. Apr 24, 2021
  31. Apr 19, 2021
  32. Apr 06, 2021
    • Jarek Potiuk's avatar
      Adds 'Trino' provider (with lower memory footprint for tests) (#15187) · eae22cec
      Jarek Potiuk authored
      While checking the test status of various CI tests we came to
      conclusion that Presto integration took a lot of memory (~1GB)
      and was the main source of failures during integration tests,
      especially with MySQL8. The attempt to fine-tune the memory
      used turned out in the discovery, that Presto DB stopped
      publishing their Docker image (prestosql/presto) - apparently
      after the aftermath of splitting-off Trino from Presto.
      
      Th split-off was already discussed in #14281 and it was planned
      to add support for Trino (which is the more community-driven
      fork of the Presto - Presto remained at Facebook Governance,
      where Trino is an effort continued by the original creators.
      
      You can read more about it in the announcement:
      https://trino.io/blog/2020/12/27/announcing-trino.html. While
      Presto continues their way under The Linux Foundation, Trino
      lives its own live and keeps on maintaining all artifacts and
      libraries (including the image). That allowed us to update
      our tests and decrease the memory footprint by around 400MB.
      
      This commit:
      
      * adds the new Trino provider
      * removes `presto` integration and replaces it with `trino`
      * the `trino` integartion image is built with 400MB less memory
        requirementes and published as `apache/airflow:trino-*`
      * moves the integration tests from Presto to Trino
      
      Fixes: #14281
    • Jarek Potiuk's avatar
      Constraints are now parallelized and merged in single job (#15211) · aebacd74
      Jarek Potiuk authored
      Originally, the constraints were generated in separate jobs and uploaded as
      artifacts and then joined be a separate push job. Thanks to parallel
      processing, we can now do that all in a single job, with both cost and
      time savings.
  33. Apr 04, 2021
  34. Apr 02, 2021
Loading