Merge branch 'feature/update-azure-credentials-service' into 'master'

feat(azure): update azure credentials service

See merge request !56

osdu_api/providers/README.md

-Providers folder structure.
+## Providers folder structure
 ```
 providers
 │   README.md
@@ -11,19 +11,39 @@ providers
 └───aws
     │   ...
 ```
-In the base folder there are the following base classes:
-`types.py`: Stores all the interfaces that could be implemented by cloud providers.
-`factory.py` Provides a mechanism to register and retrieve cloud specific implementations using a class decorator @ProviderFactory.register. A registry per interface is required, if a new interface is implemented a new registry should be added.
-`constants.py` Reusable constants.
-`exceptions.py` Provider specific exceptions should be thrown here and bubble up all the way up.
+In the base folder, there are the following base classes:
+* `types.py`: Stores all the interfaces that could be implemented by cloud providers.
+* `factory.py` Provides a mechanism to register and retrieve cloud-specific implementations using a class decorator @ProviderFactory.register. A registry per interface is required; if a new interface is implemented, a new registry should be added.
+* `constants.py`: Reusable constants.
+* `exceptions.py`: Provider-specific exceptions should be thrown here and bubble up all the way up.
+
 All interfaces will require a **wrapper module** that registers specific implementations and provides a factory method. Examples of wrapper modules are: `blob_storage.py` and `credentials.py`. Please pay attention to the import section as it's required that all modules that implement an interface to be imported here so they can be registered.
 
-Adding a new implementation.
-1. Create a module (e.g., gcp_blob_storage_client.py) under the cloud provider subfolder. Within the module, create a class that inherits from given base interface and decorate it with @ProviderFactory.register(`CLOUD_PROVIDER_NAME_CONSTANT`)
+## Adding a new implementation
+1. Create a module (e.g., gcp_blob_storage_client.py) under the cloud provider subfolder. Within the module, create a class that inherits from given base interface and decorates it with @ProviderFactory.register(`CLOUD_PROVIDER_NAME_CONSTANT`)
 2. Add an import in the **wrapper module** of the interface (e.g., blob_storage.py -> from providerds.gcp.gcp_blob_storage_client.py) to register the new implementation. You may optionally modify the factory method if the initialization of the new implemented class needs some customatization. Just don't bubble up customatization as factory method should be transparent for all clients.
 
-Adding a new interface.
+## Adding a new interface
 1. Add interface to types.py.
-2. Add a new registry to ProvidersFactory.py and modify register logic to take into account new type. Add a new get method for the new type.
+2. Add a new registry to ProvidersFactory.py and modify the register logic to take the new type into account. Add a new get method for the new type.
 3. Add the new implementation.
 
+## Provider-specific environment variables
+
+### Azure:
+
+#### Getting token through MSI (Managed Service Identity)
+
+* AZURE_ENABLE_MSI (AIRFLOW_VAR_AZURE_ENABLE_MSI)
+
+* IDENTITY_ENDPOINT
+* IDENTITY_HEADER
+
+#### Getting token through Azure Active Directory credentials
+
+* KEYVAULT_URI (AZURE_KEYVAULT_URI, AIRFLOW_VAR_KEYVAULT_URI)
+
+* AZURE_TENANT_ID (AIRFLOW_VAR_AZURE_TENANT_ID, AZURE_AD_TENANT_ID)
+* AZURE_CLIENT_ID (AIRFLOW_VAR_AZURE_CLIENT_ID, AZURE_PRINCIPAL_ID)
+* AZURE_CLIENT_SECRET (AIRFLOW_VAR_AZURE_CLIENT_SECRET, AZURE_PRINCIPAL_SECRET)
+* AZURE_RESOURCE_ID (AIRFLOW_VAR_AAD_CLIENT_ID, AZURE_APP_ID, AZURE_AD_APP_RESOURCE_ID)

osdu_api/providers/azure/azure_credentials.py

 #  Copyright © Microsoft Corporation
+#  Copyright 2021 EPAM Systems
 #
 #  Licensed under the Apache License, Version 2.0 (the "License");
 #  you may not use this file except in compliance with the License.
@@ -13,18 +14,20 @@
 #  limitations under the License.
 """Azure Credential Module."""
 
+import json
 import logging
-from osdu_api.providers.constants import AZURE_CLOUD_PROVIDER
-from osdu_api.providers.factory import ProvidersFactory
-from osdu_api.providers.types import BaseCredentials
-from tenacity import retry, stop_after_attempt
-import msal
 import os
-from azure.keyvault import secrets
-from azure import identity
+
+import msal
 import requests
-import json
+from azure import identity
+from azure.keyvault import secrets
+from tenacity import retry, stop_after_attempt
 
+from osdu_api.providers.constants import AZURE_CLOUD_PROVIDER
+from osdu_api.providers.factory import ProvidersFactory
+from osdu_api.providers.types import BaseCredentials
+from osdu_api.utils.env import getenv_by_names
 
 logger = logging.getLogger(__name__)
 RETRIES = 3
@@ -40,28 +43,54 @@ class AzureCredentials(BaseCredentials):
         self._client_secret = None
         self._tenant_id = None
         self._resource_id = None
-        self._azure_paas_podidentity_isEnabled= os.getenv("AIRFLOW_VAR_AZURE_ENABLE_MSI")
+        self._msi = getenv_by_names([
+            "AZURE_ENABLE_MSI",
+            "AIRFLOW_VAR_AZURE_ENABLE_MSI",
+            ])
 
     def _populate_ad_credentials(self) -> None:
-        uri = os.getenv("AIRFLOW_VAR_KEYVAULT_URI")
-        credential = identity.DefaultAzureCredential()
-        client = secrets.SecretClient(vault_url=uri, credential=credential)
-        self._client_id = client.get_secret("app-dev-sp-username").value
-        self._client_secret = client.get_secret('app-dev-sp-password').value
-        self._tenant_id = client.get_secret('app-dev-sp-tenant-id').value
-        self._resource_id = client.get_secret("aad-client-id").value
+        """Prepare credentials for azure active directory."""
+        uri = getenv_by_names([
+            "KEYVAULT_URI",
+            "AZURE_KEYVAULT_URI",
+            "AIRFLOW_VAR_KEYVAULT_URI",
+            ])
+
+        if uri:
+            # getting azure AD credentials through azure keyvault
+            credential = identity.DefaultAzureCredential()
+            client = secrets.SecretClient(vault_url=uri, credential=credential)
+            self._client_id = client.get_secret("app-dev-sp-username").value
+            self._client_secret = client.get_secret('app-dev-sp-password').value
+            self._tenant_id = client.get_secret('app-dev-sp-tenant-id').value
+            self._resource_id = client.get_secret("aad-client-id").value
+        else:
+            # getting azure AD credentials through env variables
+            self._tenant_id = getenv_by_names([
+                "AIRFLOW_VAR_AZURE_TENANT_ID", "AZURE_TENANT_ID", "AZURE_AD_TENANT_ID"])
+            self._client_id = getenv_by_names([
+                "AIRFLOW_VAR_AZURE_CLIENT_ID", "AZURE_CLIENT_ID", "AZURE_PRINCIPAL_ID"])
+            self._client_secret = getenv_by_names([
+                "AIRFLOW_VAR_AZURE_CLIENT_SECRET", "AZURE_CLIENT_SECRET", "AZURE_PRINCIPAL_SECRET"])
+            self._resource_id = getenv_by_names([
+                "AIRFLOW_VAR_AAD_CLIENT_ID", "AZURE_RESOURCE_ID", "AZURE_APP_ID", "AZURE_AD_APP_RESOURCE_ID"])
 
     def _generate_token(self) -> str:
-        if self._azure_paas_podidentity_isEnabled == "true":
+        if self._msi == "true":
             try:
-                print("MSI Token generation")
-                headers = {
-                    'Metadata': 'true'
-                }
-                url = 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.azure.com%2F'
+                logger.info("MSI Token generation")
+
+                endpoint = os.getenv("IDENTITY_ENDPOINT", "http://169.254.169.254/metadata/identity/oauth2/token")
+                url = f"{endpoint}?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.azure.com%2F"
+
+                headers = {'Metadata': 'true'}
+                if os.getenv("IDENTITY_HEADER"):
+                    headers['X-IDENTITY-HEADER'] = os.getenv("IDENTITY_HEADER")
+
                 response = requests.request("GET", url, headers=headers)
                 data_msi = json.loads(response.text)
                 token = data_msi["access_token"]
+
                 return token
             except Exception as e:
                 logger.error(e)
@@ -83,6 +112,8 @@ class AzureCredentials(BaseCredentials):
                 raise ValueError("Please pass client secret to generate token")
 
             try:
+                logger.info("Generating token by AD credentials")
+
                 authority_host_uri = 'https://login.microsoftonline.com'
                 authority_uri = authority_host_uri + '/' + self._tenant_id
                 scopes = [self._resource_id + '/.default']
@@ -114,4 +145,3 @@ class AzureCredentials(BaseCredentials):
         :rtype: str
         """
         return self._access_token
-

osdu_api/utils/__init__.py

+#  Copyright © Microsoft Corporation
+#  Copyright 2021 EPAM Systems
+#
+#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#  You may obtain a copy of the License at
+#
+#       http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.

osdu_api/utils/env.py

+#  Copyright © Microsoft Corporation
+#  Copyright 2021 EPAM Systems
+#
+#  Licensed under the Apache License, Version 2.0 (the "License");
+#  you may not use this file except in compliance with the License.
+#  You may obtain a copy of the License at
+#
+#       http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+"""Utils for env management."""
+
+import os
+from typing import List, Optional
+
+
+def getenv_by_names(names: List[str]) -> Optional[str]:
+    """
+    Checking several env variables and return result value by priority.
+
+    :param names: Env variables names for checking
+    :type names: List[str]
+    :return: Env variable result
+    :rtype: Optional[str]
+    """
+    for env_name in names:
+        env_value = os.getenv(env_name)
+        if env_value:
+            return env_value
+
+    return None