- Dec 22, 2021
-
-
Guilherme Da Silva Gonçalves authored
* Azure: New sftp to wasb operator Co-authored-by: Guilherme da Silva Goncalves <guilherme.goncalves@bancointer.com.br> Co-authored-by: Josh Fell <48934154+josh-fell@users.noreply.github.com>
-
- Dec 21, 2021
-
-
pierrejbrun authored
* rewrite opsgenie alert hook with official python sdk
-
- Dec 09, 2021
-
-
Tzu-ping Chung authored
-
- Dec 07, 2021
-
-
Ash Berlin-Taylor authored
* Fix many of the mypy typing issues in airflow.models.dag And to fix these, I needed to fix a few other mistakes that are used/called by DAG's methods * Fix timetable-related typing errors in dag.py Also moved the sentinel value implementation to a utils module. This should be useful when fixing typing issues in other modules. * Add note about assert allowed inside a TYPE_CHECKING conditional * Fix docs build of airflow.models.dagrun * Apply NEW_SESSION to dag, dagrun, ti and operator.subdag Co-authored-by: Tzu-ping Chung <tp@astronomer.io>
-
- Nov 12, 2021
-
-
Brent Bovenzi authored
Add react to existing ui and build a new tree view that can properly handle task groups. Co-authored-by: Ephraim Anierobi <splendidzigy24@gmail.com> Co-authored-by: Sam Wheating <samwheating@gmail.com>
-
- Oct 28, 2021
-
-
Edith Puclla authored
-
- Oct 10, 2021
-
-
Mario Taddeucci authored
* Remove extra prostgres dependency * Removed postgres cross dependency on aws provider
-
- Sep 28, 2021
-
-
Kanthi authored
* Added transfer operator for bigquery to mssql and tests * Update airflow/providers/google/cloud/transfers/bigquery_to_mssql.py Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com> * Update airflow/providers/google/cloud/transfers/bigquery_to_mssql.py Code change based on review comments. Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com> * Added comment in bigquery to mssql transfer * Added influx operator files * Delete bigquery_to_mssql.py * Update test_bigquery_to_bigquery.py * Delete test_bigquery_to_mssql.py * [14168] Added functions to create organization, bucket and run query to influxdb. * [14168] Added file that was accidentally deleted. * [14168] Added support to write Point. * [14168] Added Influx provider support * [14168] Added Influx provider support * [14168] Added test case for InfluxDBHook * [14168] Added test case for InfluxDBHook * [14168] Fixed test case for InfluxDBHook * [14168] Fixed test case for InfluxDBHook * [14168] Added example DAG for influxDBHook * [14168] Changed org_id to org_name for clarity. * [14168] Removed README, fixed influxDBHook class name in provider.yml * [14168] Static code check fixes. * [14168] fixed license header influxdb.rst for connections * [14168] fixed flak tests and order of requirements - influxdb * [14168] Renamed provider.yaml from provider.yml * [14168] fixed imports in influxdb.py * [14168] Renamed function for passing flak8 tests. * [14168] Fixed influxDB connection rst * [14168] Fixed order in CONTRIBUTING.rst and spelling_wordlist.txt * [14168] ADded operators/influxdb.rst to satisfy pre commit check * [14168] fixed docs error influxdb * [14168] fixed example DAG path for influxDB * Update airflow/providers/influxdb/hooks/influxdb.py Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> * Rebase from master * Fixed PR review comments * Fixed influxdb documentation * Fixed documentation * Added empty doc for InfluxDB operator to workaround build-docs error * Fixed table for influxdb dependency * Fixed connection-type in influxdb provider * Added init to example_dags * Added influxdb to airflow_providers_bug_report * Added missing init file in influxdb hook * Added missing init in tests/influxdb * Removed link in toc tree for influxdb * Moved influxdb tests from operators to hooks, added tests * Added more tests for influxdb hook * Added commits.rst and installing-providers-from-sources.rst for influxdb * Added commits to toctree of index * Added pandas dependency to influxdb Co-authored-by: Kanthi <kanthi@Kanthis-MacBook-Air.local> Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com> Co-authored-by: Kanthi Subramanian <subkanthi@gmail.ccom> Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
-
- Aug 25, 2021
-
-
Mateusz Nojek authored
-
- Aug 23, 2021
-
-
Jarek Potiuk authored
When we have a prolonged issue with flaky tests or Github runners instabilities, our automated constraint and image refresh might not work, so we might need to manually refresh the constraints and images. Documentation about that was in CONTRIBUTING.rst but it is more appriate to keep it in ``dev`` as it only applies to committers. Also during testing the parallell refresh without delays an error was discovered which prevented parallell check of random image hash during the build. This has been fixed and parallell image cache building should work flawlessly now.
-
- Aug 19, 2021
-
-
Jarek Potiuk authored
The entry was terribly out-dated. We did not have many problems with different python versions backwards compatibilty for a looong time. Seems that the new SenVer approach and release control and cadence work pretty well comparing to 3.5/3.6 times (BTW. good job Python developers!) so I updated the entry to reflect the `recommended` approach we should take.
-
D. Ferruzzi authored
* Implemented Basic EKS Integration * Remove explicit region defaulting cr https://code.amazon.com/reviews/CR-52973030 * Refactor the token generation and remove the AWS CLI dependency. * move kubeconfig generator into `hooks/eks.py` * EKS List hooks return all results * Use a tempfile to store kubeconfig data * Move kube config into Hook class as a contextmanager * Removed random traits in tests * Rework the eks.rst doc file * Implemented Jinja templates for operators - Added jinja template fields - Refactored fields to snake_case since they are now exposed - Removed a couple of straggling pylint instructions; pylint is no longer used * conn_id refactor - Refactored using IDE magic: - any field named `self.conn_id` is now `self.aws_conn_id` - any param named `conn_id` is now `aws_conn_id` - any constant named `CONN_ID` is now `DEFAULT_CONN_ID` - Sensors were missing template fields, added those. * Remove try/log blocks from hooks * Implemented EKS system tests * Remove List and Describe Operators and supporting code. * Fixed `nextToken` final result bug * Remove some nesting and some unnecessary logging before raising an exception * Use the force * Improved docs and samples * Additional jinja templating * Corrected some misused Optionals. * Doc formatting fix * Corrected logo * Corrected logo - background and size
-
- Aug 18, 2021
-
-
Kaxil Naik authored
* Update docs on syncing forks closes https://github.com/apache/airflow/issues/17665
-
- Aug 12, 2021
-
-
Kaxil Naik authored
We only use `pandas` in `DbApiHook.get_pandas_df`. Not all users use it, plus while `pandas` now supports many pre-compiled packages it still can take forever where it needs to be compiled. So for first-time users this can be a turn off. If pandas is already installed this will work fine, but if not users have an option to run `pip install apache-airflow[pandas]` closes #12500
-
- Aug 08, 2021
-
-
Malthe Borch authored
-
- Aug 07, 2021
-
-
Gabriel authored
-
- Jul 21, 2021
-
-
josh-fell authored
-
- Jul 17, 2021
-
-
Nijanthan Vijayakumar authored
Update CONTRIBUTING.rst and CONTRIBUTORS_QUICK_START.rst files with jq installation instructions (#17060) * Add jq installation info as a prerequisite for setting up breeze * Add macOS example for jq installation using Homebrew The "Configure Your Environment" section was missing the examples on installing jq on macOS. Earlier version of the document only had the example for the Ubuntu OS.
-
- Jul 14, 2021
-
-
Jarek Potiuk authored
-
- Jul 12, 2021
-
-
dzamo authored
-
Jarek Potiuk authored
Breeze used traditionally DockerHub to pull images, because they were public and GitHub Packages were not. With GitHub Container Regisry however, we can switch fully to using GitHub Container Registry also for Breeze. Thanks to moving to Github Container Registry we can remove a lot of code responsible for maintaining different naming and different versions of the images in DockerHub and GitHub Container Registry. Also it streamlines and simplifies the process of refreshing the images when new python versions are released - the CI push builds will check if the new Python image is released in DockerHub and it will rebuild the base image automatically if needed (and push it as cache) The CI documentation (including sequence diagrams) has been refreshed to reflect those changes (and other changes done in the meantime). The flows are now simplified as DockerHub is largely moved out of the picture. The only remaining DockerHub Images now are: * images used during CI for integrations (airflow-ci) * officially released Production Airflow images (airflow) The integration images will be moved to GitHub Container Registry in a subsequent PR and the only images remaining in DockerHub will be the officially released Production Airflow images. Part of #16555
-
- Jul 06, 2021
-
-
Kamil Breguła authored
-
Kaxil Naik authored
Since Airflow is a Top-level project and graduated from incubation, instead of PPMC the url we should use is different.
-
- Jun 28, 2021
-
-
Jarek Potiuk authored
We've agreed during the voting process that Pylint support should be disabled: https://lists.apache.org/thread.html/r9e2cc385db8737ec0874ad09872081bd083593ee29e8303e58d21efb%40%3Cdev.airflow.apache.org%3E This PR: * removes all # pylint comments * removes pylint pre-commits and related scripts/files * removes CI jobs running pylint checks * removes documentation about pylint * removes unnecessary #noga (adds pre-commit for that) * fixes some remaining pydocstyle errors after removing #noqa's
-
- Jun 25, 2021
-
-
Jarek Potiuk authored
This includes several things: * added per-provider support for python version. Each provider can now declare python versions it does not support * excluded ldap core extra from Python 3.9. * skip relevant tests in Python 3.9
-
- Jun 03, 2021
-
-
Kamil Breguła authored
-
- Jun 01, 2021
-
-
Jarek Potiuk authored
-
- May 27, 2021
-
-
Kamil Breguła authored
* Check syntactic correctness for code-snippets * fixup! Check syntactic correctness for code-snippets * fixup! fixup! Check syntactic correctness for code-snippets * fixup! fixup! fixup! Check syntactic correctness for code-snippets
-
- May 19, 2021
-
-
Jarek Potiuk authored
* Updates branches and branch documentation after 2.1.0rc1 This PR updates branches and corresponding documentation and tools after 2.1.0rc1 release. It describes what needs to be done when new release branch is created, and provides tools that allow to do most of the work semi-automatically. Wherever possible and easy, the 2-0 references were replaced with 2-* and where it was more difficult, TODOS were left. The `dev/retag_docker_images.py` script will also be useful when we get to renaming the `master` tag to `main` tag.
-
Daniel Standish authored
Plyvel does not build on macOS without levelDB installed in system. Its better to make it an optional install.
-
- May 17, 2021
-
-
Jennifer Melot authored
-
Jarek Potiuk authored
This change gets rid of requests as core dependency. We have to change requests to become an optional dependency because it (so far) pulls in chardet as dependency and chardet is LGPL, which is not allowed to be mandatory dependency by ASF policies. More info here: https://issues.apache.org/jira/browse/LEGAL-572 The changes: * connexion is vendored-in (and requests usage is replaced with httpx) * Http Provider is turned into optional provider (not preinstalled) * Few places where requests were used in core and in cloud_sql provider which did not cause compatibility problem, it was replaced by httpx. * new extra added for deprecated experimental API (which is disabled by default and optional) * tests are fixed (using pytest-httpx fixture package) * The providers: http, airbyte, apache.livy, opsgenie, slack (all depend on http) now explicitely depend on `requirements`.
-
- Apr 29, 2021
-
-
Daniel Standish authored
-
- Apr 27, 2021
-
-
Jarek Potiuk authored
* Use Pip 21.* to install airflow officially The PIP 20.2.4 was so far the only officially supported installation mechanism for Airflow as there were some problems with conflicting dependencies (which were ignored by previous versio of PIP). This change attempts to solve this by removing a [gcp] extra from `apache-beam` which turns out to be the major source of the problem - as it contains requirements to the old version of google client libraries (but apparently only used for tests). The "apache-beam" provider migh however need the [gcp] extra for other components so in order to not break the backwards compatibility, another approach is used. Instead of adding [gcp] as extra in the apache-beam extra, the apache.beam provider's [google] extra is extended with 'apache-beam[gcp]' additional requirement so that whenever the provider is installed, the apache-beam with [gcp] extra is installed as well. * Update airflow/providers/apache/beam/CHANGELOG.rst Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> * Update airflow/providers/apache/beam/CHANGELOG.rst Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> * Update airflow/providers/google/CHANGELOG.rst Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> * Update airflow/providers/google/CHANGELOG.rst Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com> Co-authored-by: Tzu-ping Chung <uranusjr@gmail.com>
-
- Apr 24, 2021
-
-
Tomek Urbaszek authored
Some time ago we agreed to no use plain assert outside tests. This was voted but is missing from our 'coding style' guide.
-
- Apr 19, 2021
-
-
yangrong688 authored
-
- Apr 06, 2021
-
-
Jarek Potiuk authored
While checking the test status of various CI tests we came to conclusion that Presto integration took a lot of memory (~1GB) and was the main source of failures during integration tests, especially with MySQL8. The attempt to fine-tune the memory used turned out in the discovery, that Presto DB stopped publishing their Docker image (prestosql/presto) - apparently after the aftermath of splitting-off Trino from Presto. Th split-off was already discussed in #14281 and it was planned to add support for Trino (which is the more community-driven fork of the Presto - Presto remained at Facebook Governance, where Trino is an effort continued by the original creators. You can read more about it in the announcement: https://trino.io/blog/2020/12/27/announcing-trino.html. While Presto continues their way under The Linux Foundation, Trino lives its own live and keeps on maintaining all artifacts and libraries (including the image). That allowed us to update our tests and decrease the memory footprint by around 400MB. This commit: * adds the new Trino provider * removes `presto` integration and replaces it with `trino` * the `trino` integartion image is built with 400MB less memory requirementes and published as `apache/airflow:trino-*` * moves the integration tests from Presto to Trino Fixes: #14281
-
Jarek Potiuk authored
Originally, the constraints were generated in separate jobs and uploaded as artifacts and then joined be a separate push job. Thanks to parallel processing, we can now do that all in a single job, with both cost and time savings.
-
- Apr 04, 2021
-
- Apr 02, 2021
-
-
eladkal authored
-