Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • osdu/platform/domain-data-mgmt-services/wellbore/wellbore-domain-services
  • Vkamani/vkamani-wellbore-domain-services
  • Yan_Sushchynski/wellbore-domain-services-comm-impl
3 results
Show changes
Commits on Source (18)
...@@ -23,6 +23,7 @@ The following software have components provided under the terms of this license: ...@@ -23,6 +23,7 @@ The following software have components provided under the terms of this license:
- google-cloud-trace (from https://github.com/googleapis/googleapis) - google-cloud-trace (from https://github.com/googleapis/googleapis)
- googleapis-common-protos (from https://github.com/googleapis/googleapis) - googleapis-common-protos (from https://github.com/googleapis/googleapis)
- grpcio (from https://grpc.io) - grpcio (from https://grpc.io)
- importlib-metadata
- jsonpath-ng (from https://github.com/h2non/jsonpath-ng) - jsonpath-ng (from https://github.com/h2non/jsonpath-ng)
- msgpack (from http://msgpack.org/) - msgpack (from http://msgpack.org/)
- multidict (from https://github.com/aio-libs/multidict/) - multidict (from https://github.com/aio-libs/multidict/)
...@@ -49,7 +50,6 @@ The following software have components provided under the terms of this license: ...@@ -49,7 +50,6 @@ The following software have components provided under the terms of this license:
- s3transfer (from https://github.com/boto/s3transfer) - s3transfer (from https://github.com/boto/s3transfer)
- sniffio (from https://github.com/python-trio/sniffio) - sniffio (from https://github.com/python-trio/sniffio)
- sortedcontainers (from http://www.grantjenks.com/docs/sortedcontainers/) - sortedcontainers (from http://www.grantjenks.com/docs/sortedcontainers/)
- structlog (from http://www.structlog.org/)
- tblib (from https://github.com/ionelmc/python-tblib) - tblib (from https://github.com/ionelmc/python-tblib)
- toposort (from https://bitbucket.org/ericvsmith/toposort) - toposort (from https://bitbucket.org/ericvsmith/toposort)
- tornado (from http://www.tornadoweb.org/) - tornado (from http://www.tornadoweb.org/)
...@@ -82,15 +82,12 @@ The following software have components provided under the terms of this license: ...@@ -82,15 +82,12 @@ The following software have components provided under the terms of this license:
- Jinja2 (from http://jinja.pocoo.org/, https://palletsprojects.com/p/jinja/) - Jinja2 (from http://jinja.pocoo.org/, https://palletsprojects.com/p/jinja/)
- MarkupSafe (from https://palletsprojects.com/p/markupsafe/) - MarkupSafe (from https://palletsprojects.com/p/markupsafe/)
- adlfs (from https://github.com/hayesgb/adlfs/) - adlfs (from https://github.com/hayesgb/adlfs/)
- asgiref (from http://github.com/django/asgiref/)
- click (from http://github.com/mitsuhiko/click) - click (from http://github.com/mitsuhiko/click)
- cloudpickle (from https://github.com/cloudpipe/cloudpickle) - cloudpickle (from https://github.com/cloudpipe/cloudpickle)
- colorama (from https://github.com/tartley/colorama)
- cryptography (from https://github.com/pyca/cryptography) - cryptography (from https://github.com/pyca/cryptography)
- dask (from http://github.com/dask/dask/) - dask (from http://github.com/dask/dask/)
- decorator (from https://github.com/micheles/decorator) - decorator (from https://github.com/micheles/decorator)
- distributed (from https://distributed.readthedocs.io/en/latest/)
- fsspec (from http://github.com/intake/filesystem_spec)
- gcsfs (from https://github.com/dask/gcsfs)
- grpcio (from https://grpc.io) - grpcio (from https://grpc.io)
- hiredis (from https://github.com/redis/hiredis-py) - hiredis (from https://github.com/redis/hiredis-py)
- httpcore (from https://github.com/encode/httpcore) - httpcore (from https://github.com/encode/httpcore)
...@@ -142,13 +139,6 @@ The following software have components provided under the terms of this license: ...@@ -142,13 +139,6 @@ The following software have components provided under the terms of this license:
- s3fs (from http://github.com/dask/s3fs/) - s3fs (from http://github.com/dask/s3fs/)
- toolz (from http://github.com/pytoolz/toolz/) - toolz (from http://github.com/pytoolz/toolz/)
========================================================================
CC-BY-SA-3.0
========================================================================
The following software have components provided under the terms of this license:
- numpy
======================================================================== ========================================================================
CC0-1.0 CC0-1.0
======================================================================== ========================================================================
...@@ -178,7 +168,6 @@ The following software have components provided under the terms of this license: ...@@ -178,7 +168,6 @@ The following software have components provided under the terms of this license:
- coverage (from https://github.com/nedbat/coveragepy) - coverage (from https://github.com/nedbat/coveragepy)
- grpcio (from https://grpc.io) - grpcio (from https://grpc.io)
- pyparsing (from http://pyparsing.wikispaces.com/)
- rfc3986 (from https://rfc3986.readthedocs.org) - rfc3986 (from https://rfc3986.readthedocs.org)
======================================================================== ========================================================================
...@@ -187,7 +176,6 @@ ISC ...@@ -187,7 +176,6 @@ ISC
The following software have components provided under the terms of this license: The following software have components provided under the terms of this license:
- grpcio (from https://grpc.io) - grpcio (from https://grpc.io)
- requests-oauthlib (from https://github.com/requests/requests-oauthlib)
======================================================================== ========================================================================
Info-ZIP Info-ZIP
...@@ -207,7 +195,6 @@ LGPL-2.1-only ...@@ -207,7 +195,6 @@ LGPL-2.1-only
The following software have components provided under the terms of this license: The following software have components provided under the terms of this license:
- chardet (from https://github.com/chardet/chardet) - chardet (from https://github.com/chardet/chardet)
- charset-normalizer (from https://github.com/ousret/charset_normalizer)
======================================================================== ========================================================================
LGPL-2.1-or-later LGPL-2.1-or-later
...@@ -241,11 +228,6 @@ The following software have components provided under the terms of this license: ...@@ -241,11 +228,6 @@ The following software have components provided under the terms of this license:
- azure-common (from https://github.com/Azure/azure-sdk-for-python) - azure-common (from https://github.com/Azure/azure-sdk-for-python)
- azure-core (from https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/core/azure-core) - azure-core (from https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/core/azure-core)
- azure-datalake-store (from https://github.com/Azure/azure-data-lake-store-python) - azure-datalake-store (from https://github.com/Azure/azure-data-lake-store-python)
- azure-identity (from https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/identity/azure-identity)
- azure-keyvault (from https://github.com/Azure/azure-sdk-for-python)
- azure-keyvault-certificates (from https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/keyvault/azure-keyvault-certificates)
- azure-keyvault-keys (from https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/keyvault/azure-keyvault-keys)
- azure-keyvault-secrets (from https://github.com/Azure/azure-sdk-for-python/tree/master/sdk/keyvault/azure-keyvault-secrets)
- azure-storage-blob (from https://github.com/Azure/azure-storage-python) - azure-storage-blob (from https://github.com/Azure/azure-storage-python)
- backoff - backoff
- botocore (from https://github.com/boto/botocore) - botocore (from https://github.com/boto/botocore)
...@@ -262,9 +244,7 @@ The following software have components provided under the terms of this license: ...@@ -262,9 +244,7 @@ The following software have components provided under the terms of this license:
- jsonschema - jsonschema
- mockito (from https://github.com/kaste/mockito-python) - mockito (from https://github.com/kaste/mockito-python)
- msal (from https://github.com/AzureAD/microsoft-authentication-library-for-python) - msal (from https://github.com/AzureAD/microsoft-authentication-library-for-python)
- msal-extensions (from https://pypi.org/project/msal-extensions/0.1.3/)
- msrest (from https://github.com/Azure/msrest-for-python) - msrest (from https://github.com/Azure/msrest-for-python)
- munch (from http://github.com/Infinidat/munch)
- natsort (from https://github.com/SethMMorton/natsort) - natsort (from https://github.com/SethMMorton/natsort)
- numpy - numpy
- ordered-set (from http://github.com/LuminosoInsight/ordered-set) - ordered-set (from http://github.com/LuminosoInsight/ordered-set)
...@@ -273,20 +253,14 @@ The following software have components provided under the terms of this license: ...@@ -273,20 +253,14 @@ The following software have components provided under the terms of this license:
- pluggy - pluggy
- py (from http://pylib.readthedocs.org/) - py (from http://pylib.readthedocs.org/)
- pyarrow (from https://arrow.apache.org/) - pyarrow (from https://arrow.apache.org/)
- pydantic (from https://github.com/samuelcolvin/pydantic)
- pyparsing (from http://pyparsing.wikispaces.com/) - pyparsing (from http://pyparsing.wikispaces.com/)
- pyrsistent (from http://github.com/tobgu/pyrsistent/) - pyrsistent (from http://github.com/tobgu/pyrsistent/)
- pytest (from http://pytest.org, https://docs.pytest.org/en/latest/) - pytest (from http://pytest.org, https://docs.pytest.org/en/latest/)
- pytest-cov (from https://github.com/pytest-dev/pytest-cov) - pytest-cov (from https://github.com/pytest-dev/pytest-cov)
- pytest-httpx (from https://colin-b.github.io/pytest_httpx/)
- pytest-mock (from https://github.com/pytest-dev/pytest-mock/) - pytest-mock (from https://github.com/pytest-dev/pytest-mock/)
- python-rapidjson (from https://github.com/python-rapidjson/python-rapidjson)
- python-ulid (from https://github.com/mdomke/python-ulid)
- pytz (from http://pythonhosted.org/pytz) - pytz (from http://pythonhosted.org/pytz)
- requests-oauthlib (from https://github.com/requests/requests-oauthlib)
- six (from http://pypi.python.org/pypi/six/) - six (from http://pypi.python.org/pypi/six/)
- sniffio (from https://github.com/python-trio/sniffio) - sniffio (from https://github.com/python-trio/sniffio)
- structlog (from http://www.structlog.org/)
- toml (from https://github.com/uiri/toml) - toml (from https://github.com/uiri/toml)
- tomli (from https://pypi.org/project/tomli/1.2.2/, https://pypi.org/project/tomli/2.0.0/) - tomli (from https://pypi.org/project/tomli/1.2.2/, https://pypi.org/project/tomli/2.0.0/)
- urllib3 (from https://urllib3.readthedocs.io/) - urllib3 (from https://urllib3.readthedocs.io/)
...@@ -401,7 +375,6 @@ public-domain ...@@ -401,7 +375,6 @@ public-domain
======================================================================== ========================================================================
The following software have components provided under the terms of this license: The following software have components provided under the terms of this license:
- botocore (from https://github.com/boto/botocore)
- grpcio (from https://grpc.io) - grpcio (from https://grpc.io)
- hypothesis (from https://hypothesis.works) - hypothesis (from https://hypothesis.works)
- numpy - numpy
......
...@@ -321,7 +321,7 @@ Then access app on `http://127.0.0.1:<LOCAL_PORT>/api/os-wellbore-ddms/docs` ...@@ -321,7 +321,7 @@ Then access app on `http://127.0.0.1:<LOCAL_PORT>/api/os-wellbore-ddms/docs`
IMAGE_TAG="os-wellbore-ddms:dev" IMAGE_TAG="os-wellbore-ddms:dev"
# Build Image # Build Image
docker build -t=$IMAGE_TAG --rm . -f ./build/dockerfile --build-arg PIP_WHEEL_DIR=python-packages docker build -t=$IMAGE_TAG --rm . -f ./build/Dockerfile --build-arg PIP_WHEEL_DIR=python-packages
``` ```
#### Run Image #### Run Image
......
...@@ -305,7 +305,7 @@ def cloud_provider_additional_environment(config: ConfigurationContainer): ...@@ -305,7 +305,7 @@ def cloud_provider_additional_environment(config: ConfigurationContainer):
env_var_key='AZ_AI_INSTRUMENTATION_KEY', env_var_key='AZ_AI_INSTRUMENTATION_KEY',
description='azure app insights instrumentation key', description='azure app insights instrumentation key',
secret=True, secret=True,
is_mandatory=True, is_mandatory=False,
override=True) override=True)
config.add_from_env(attribute_name='az_logger_level', config.add_from_env(attribute_name='az_logger_level',
......
...@@ -121,20 +121,18 @@ def create_azure_logger(*, service_name, az_ai_instrumentation_key, az_logger_le ...@@ -121,20 +121,18 @@ def create_azure_logger(*, service_name, az_ai_instrumentation_key, az_logger_le
# AzurelogHandler for logging to azure appinsight # AzurelogHandler for logging to azure appinsight
key = az_ai_instrumentation_key key = az_ai_instrumentation_key
logger_level = az_logger_level logger_level = az_logger_level
az_handler = AzureLogHandler(connection_string=f'InstrumentationKey={key}') if key:
az_handler.setLevel(logging.getLevelName(logger_level)) az_handler = AzureLogHandler(connection_string=f'InstrumentationKey={key}')
az_handler.add_telemetry_processor(rename_cloud_role_func(service_name)) az_handler.setLevel(logging.getLevelName(logger_level))
az_handler.add_telemetry_processor(rename_cloud_role_func(service_name))
else:
az_handler = None
# Acquire the logger for azure library # Acquire the logger for azure library
az_logger = logging.getLogger('azure') az_logger = __getLogger(logger_name='azure', log_level=logging.DEBUG, stdout_handler=stdout_handler, azure_handler=az_handler)
az_logger.setLevel(logging.INFO)
az_logger.addHandler(stdout_handler)
# Acquire the logger for osdu-core-lib-python-azure # Acquire the logger for osdu-core-lib-python-azure
osdu_core_lib_logger = logging.getLogger('osdu_az') osdu_core_lib_logger = __getLogger(logger_name='osdu_az', log_level=logging.DEBUG, stdout_handler=stdout_handler, azure_handler=az_handler)
osdu_core_lib_logger.setLevel(logging.INFO)
osdu_core_lib_logger.addHandler(stdout_handler)
osdu_core_lib_logger.addHandler(az_handler)
# Ensure logging messages from Dask (killing, restart worker) are exported # Ensure logging messages from Dask (killing, restart worker) are exported
dask_nanny_logger = logging.getLogger('distributed.nanny') dask_nanny_logger = logging.getLogger('distributed.nanny')
...@@ -142,10 +140,7 @@ def create_azure_logger(*, service_name, az_ai_instrumentation_key, az_logger_le ...@@ -142,10 +140,7 @@ def create_azure_logger(*, service_name, az_ai_instrumentation_key, az_logger_le
dask_nanny_logger.addHandler(az_handler) dask_nanny_logger.addHandler(az_handler)
# Acquire the logger for wdms # Acquire the logger for wdms
logger = logging.getLogger(__name__) logger = __getLogger(logger_name=__name__, log_level=logging.DEBUG, stdout_handler=stdout_handler, azure_handler=az_handler)
logger.setLevel(logging.INFO)
logger.addHandler(stdout_handler)
logger.addHandler(az_handler)
# return wdms logger with Context adapter # return wdms logger with Context adapter
return AzureContextLoggerAdapter(logger, extra=dict()) return AzureContextLoggerAdapter(logger, extra=dict())
...@@ -190,3 +185,13 @@ def create_gcp_logger(service_name): ...@@ -190,3 +185,13 @@ def create_gcp_logger(service_name):
std_ddms_app.propagate = False std_ddms_app.propagate = False
return my_logger return my_logger
def __getLogger(logger_name, log_level, stdout_handler, azure_handler=None):
logger = logging.getLogger(logger_name)
logger.setLevel(log_level)
logger.addHandler(stdout_handler)
if azure_handler:
logger.addHandler(azure_handler)
return logger
\ No newline at end of file
from app.conf import Config
from prometheus_fastapi_instrumentator import Instrumentator
def init_metric(wdms_app):
if Config.cloud_provider.value == 'az':
Instrumentator().instrument(wdms_app).expose(wdms_app)
\ No newline at end of file
...@@ -91,10 +91,11 @@ def create_exporter(*, service_name, config): ...@@ -91,10 +91,11 @@ def create_exporter(*, service_name, config):
key = config.get('az_ai_instrumentation_key') key = config.get('az_ai_instrumentation_key')
try: try:
az_exporter = _create_azure_exporter(key) if type(key) is not None:
az_exporter.add_telemetry_processor(rename_cloud_role_func(service_name)) az_exporter = _create_azure_exporter(key)
az_exporter.add_telemetry_processor(azure_traces_processing) az_exporter.add_telemetry_processor(rename_cloud_role_func(service_name))
combined_exporter.add_exporter(az_exporter) az_exporter.add_telemetry_processor(azure_traces_processing)
combined_exporter.add_exporter(az_exporter)
except ValueError as e: except ValueError as e:
print('Unable to create AzureExporter:', str(e)) print('Unable to create AzureExporter:', str(e))
else: else:
......
...@@ -27,7 +27,7 @@ from app.helper.traces import TracingRoute ...@@ -27,7 +27,7 @@ from app.helper.traces import TracingRoute
from app.model.entity_utils import Entity from app.model.entity_utils import Entity
from app.modules import discoverer from app.modules import discoverer
from app.helper import traces, logger from app.helper import traces, logger, metric
from app.injector.app_injector import AppInjector from app.injector.app_injector import AppInjector
from app.injector.main_injector import MainInjector from app.injector.main_injector import MainInjector
from app.middleware import CreateBasicContextMiddleware, TracingMiddleware from app.middleware import CreateBasicContextMiddleware, TracingMiddleware
...@@ -151,6 +151,7 @@ async def startup_event(): ...@@ -151,6 +151,7 @@ async def startup_event():
asyncio.create_task(run_in_pool_executor(executor_startup_task)) asyncio.create_task(run_in_pool_executor(executor_startup_task))
add_modules_routers() add_modules_routers()
metric.init_metric(wdms_app)
@base_app.on_event('shutdown') @base_app.on_event('shutdown')
......
...@@ -12,6 +12,7 @@ python-rapidjson ...@@ -12,6 +12,7 @@ python-rapidjson
python-multipart python-multipart
jsonpath-ng # TODO: remove when bulk_path no longer supported jsonpath-ng # TODO: remove when bulk_path no longer supported
natsort natsort
prometheus-fastapi-instrumentator
opencensus opencensus
opencensus-ext-stackdriver opencensus-ext-stackdriver
......
...@@ -135,6 +135,7 @@ fastapi==0.68.1 ...@@ -135,6 +135,7 @@ fastapi==0.68.1
# -r requirements.in # -r requirements.in
# osdu-data-ecosystem-search # osdu-data-ecosystem-search
# osdu-data-ecosystem-storage # osdu-data-ecosystem-storage
# prometheus-fastapi-instrumentator
fsspec==2021.7.0 fsspec==2021.7.0
# via # via
# -r requirements.in # -r requirements.in
...@@ -285,6 +286,10 @@ ply==3.11 ...@@ -285,6 +286,10 @@ ply==3.11
# via jsonpath-ng # via jsonpath-ng
portalocker==1.7.1 portalocker==1.7.1
# via msal-extensions # via msal-extensions
prometheus-client==0.13.1
# via prometheus-fastapi-instrumentator
prometheus-fastapi-instrumentator==5.7.1
# via -r requirements.in
protobuf==3.17.3 protobuf==3.17.3
# via # via
# google-api-core # google-api-core
......