Airflow log monitoring
This change adds a correlation Id field in the log file path for a particular dag run. If correlation Id is not passed in the dag_run config, then 'None' is used as a correlation Id. This change also includes changes in the log processor which appends correlation id in every log line so that the logs can be filtered in the azure log monitoring portal.
Infrastructure Submissions:
- [YES] Have you added an explanation of what your changes do and why you'd like us to include them?
- [YES] I have updated the documentation accordingly.
- [NA] I have added tests to cover my changes.
- [NA] All new and existing tests passed.
- [NA] I have formatted the terraform code. (
terraform fmt -recursive && go fmt ./...)
Current Behavior or Linked Issues
Does this introduce a breaking change?
- [NO]
Other information
NA