Commit a50708bf authored by harshit aggarwal's avatar harshit aggarwal
Browse files

Merge branch 'feature/update-azure-credentials-service' into 'master'

feat(azure): update azure credentials service

See merge request !56
parents ba888696 ad6461ee
Pipeline #85562 passed with stages
in 25 seconds
Providers folder structure.
## Providers folder structure
```
providers
│ README.md
......@@ -11,19 +11,39 @@ providers
└───aws
│ ...
```
In the base folder there are the following base classes:
`types.py`: Stores all the interfaces that could be implemented by cloud providers.
`factory.py` Provides a mechanism to register and retrieve cloud specific implementations using a class decorator @ProviderFactory.register. A registry per interface is required, if a new interface is implemented a new registry should be added.
`constants.py` Reusable constants.
`exceptions.py` Provider specific exceptions should be thrown here and bubble up all the way up.
In the base folder, there are the following base classes:
* `types.py`: Stores all the interfaces that could be implemented by cloud providers.
* `factory.py` Provides a mechanism to register and retrieve cloud-specific implementations using a class decorator @ProviderFactory.register. A registry per interface is required; if a new interface is implemented, a new registry should be added.
* `constants.py`: Reusable constants.
* `exceptions.py`: Provider-specific exceptions should be thrown here and bubble up all the way up.
All interfaces will require a **wrapper module** that registers specific implementations and provides a factory method. Examples of wrapper modules are: `blob_storage.py` and `credentials.py`. Please pay attention to the import section as it's required that all modules that implement an interface to be imported here so they can be registered.
Adding a new implementation.
1. Create a module (e.g., gcp_blob_storage_client.py) under the cloud provider subfolder. Within the module, create a class that inherits from given base interface and decorate it with @ProviderFactory.register(`CLOUD_PROVIDER_NAME_CONSTANT`)
## Adding a new implementation
1. Create a module (e.g., gcp_blob_storage_client.py) under the cloud provider subfolder. Within the module, create a class that inherits from given base interface and decorates it with @ProviderFactory.register(`CLOUD_PROVIDER_NAME_CONSTANT`)
2. Add an import in the **wrapper module** of the interface (e.g., blob_storage.py -> from providerds.gcp.gcp_blob_storage_client.py) to register the new implementation. You may optionally modify the factory method if the initialization of the new implemented class needs some customatization. Just don't bubble up customatization as factory method should be transparent for all clients.
Adding a new interface.
## Adding a new interface
1. Add interface to types.py.
2. Add a new registry to ProvidersFactory.py and modify register logic to take into account new type. Add a new get method for the new type.
2. Add a new registry to ProvidersFactory.py and modify the register logic to take the new type into account. Add a new get method for the new type.
3. Add the new implementation.
## Provider-specific environment variables
### Azure:
#### Getting token through MSI (Managed Service Identity)
* AZURE_ENABLE_MSI (AIRFLOW_VAR_AZURE_ENABLE_MSI)
* IDENTITY_ENDPOINT
* IDENTITY_HEADER
#### Getting token through Azure Active Directory credentials
* KEYVAULT_URI (AZURE_KEYVAULT_URI, AIRFLOW_VAR_KEYVAULT_URI)
* AZURE_TENANT_ID (AIRFLOW_VAR_AZURE_TENANT_ID, AZURE_AD_TENANT_ID)
* AZURE_CLIENT_ID (AIRFLOW_VAR_AZURE_CLIENT_ID, AZURE_PRINCIPAL_ID)
* AZURE_CLIENT_SECRET (AIRFLOW_VAR_AZURE_CLIENT_SECRET, AZURE_PRINCIPAL_SECRET)
* AZURE_RESOURCE_ID (AIRFLOW_VAR_AAD_CLIENT_ID, AZURE_APP_ID, AZURE_AD_APP_RESOURCE_ID)
# Copyright © Microsoft Corporation
# Copyright 2021 EPAM Systems
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
......@@ -13,18 +14,20 @@
# limitations under the License.
"""Azure Credential Module."""
import json
import logging
from osdu_api.providers.constants import AZURE_CLOUD_PROVIDER
from osdu_api.providers.factory import ProvidersFactory
from osdu_api.providers.types import BaseCredentials
from tenacity import retry, stop_after_attempt
import msal
import os
from azure.keyvault import secrets
from azure import identity
import msal
import requests
import json
from azure import identity
from azure.keyvault import secrets
from tenacity import retry, stop_after_attempt
from osdu_api.providers.constants import AZURE_CLOUD_PROVIDER
from osdu_api.providers.factory import ProvidersFactory
from osdu_api.providers.types import BaseCredentials
from osdu_api.utils.env import getenv_by_names
logger = logging.getLogger(__name__)
RETRIES = 3
......@@ -40,28 +43,54 @@ class AzureCredentials(BaseCredentials):
self._client_secret = None
self._tenant_id = None
self._resource_id = None
self._azure_paas_podidentity_isEnabled= os.getenv("AIRFLOW_VAR_AZURE_ENABLE_MSI")
self._msi = getenv_by_names([
"AZURE_ENABLE_MSI",
"AIRFLOW_VAR_AZURE_ENABLE_MSI",
])
def _populate_ad_credentials(self) -> None:
uri = os.getenv("AIRFLOW_VAR_KEYVAULT_URI")
credential = identity.DefaultAzureCredential()
client = secrets.SecretClient(vault_url=uri, credential=credential)
self._client_id = client.get_secret("app-dev-sp-username").value
self._client_secret = client.get_secret('app-dev-sp-password').value
self._tenant_id = client.get_secret('app-dev-sp-tenant-id').value
self._resource_id = client.get_secret("aad-client-id").value
"""Prepare credentials for azure active directory."""
uri = getenv_by_names([
"KEYVAULT_URI",
"AZURE_KEYVAULT_URI",
"AIRFLOW_VAR_KEYVAULT_URI",
])
if uri:
# getting azure AD credentials through azure keyvault
credential = identity.DefaultAzureCredential()
client = secrets.SecretClient(vault_url=uri, credential=credential)
self._client_id = client.get_secret("app-dev-sp-username").value
self._client_secret = client.get_secret('app-dev-sp-password').value
self._tenant_id = client.get_secret('app-dev-sp-tenant-id').value
self._resource_id = client.get_secret("aad-client-id").value
else:
# getting azure AD credentials through env variables
self._tenant_id = getenv_by_names([
"AIRFLOW_VAR_AZURE_TENANT_ID", "AZURE_TENANT_ID", "AZURE_AD_TENANT_ID"])
self._client_id = getenv_by_names([
"AIRFLOW_VAR_AZURE_CLIENT_ID", "AZURE_CLIENT_ID", "AZURE_PRINCIPAL_ID"])
self._client_secret = getenv_by_names([
"AIRFLOW_VAR_AZURE_CLIENT_SECRET", "AZURE_CLIENT_SECRET", "AZURE_PRINCIPAL_SECRET"])
self._resource_id = getenv_by_names([
"AIRFLOW_VAR_AAD_CLIENT_ID", "AZURE_RESOURCE_ID", "AZURE_APP_ID", "AZURE_AD_APP_RESOURCE_ID"])
def _generate_token(self) -> str:
if self._azure_paas_podidentity_isEnabled == "true":
if self._msi == "true":
try:
print("MSI Token generation")
headers = {
'Metadata': 'true'
}
url = 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.azure.com%2F'
logger.info("MSI Token generation")
endpoint = os.getenv("IDENTITY_ENDPOINT", "http://169.254.169.254/metadata/identity/oauth2/token")
url = f"{endpoint}?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.azure.com%2F"
headers = {'Metadata': 'true'}
if os.getenv("IDENTITY_HEADER"):
headers['X-IDENTITY-HEADER'] = os.getenv("IDENTITY_HEADER")
response = requests.request("GET", url, headers=headers)
data_msi = json.loads(response.text)
token = data_msi["access_token"]
return token
except Exception as e:
logger.error(e)
......@@ -83,6 +112,8 @@ class AzureCredentials(BaseCredentials):
raise ValueError("Please pass client secret to generate token")
try:
logger.info("Generating token by AD credentials")
authority_host_uri = 'https://login.microsoftonline.com'
authority_uri = authority_host_uri + '/' + self._tenant_id
scopes = [self._resource_id + '/.default']
......@@ -114,4 +145,3 @@ class AzureCredentials(BaseCredentials):
:rtype: str
"""
return self._access_token
# Copyright © Microsoft Corporation
# Copyright 2021 EPAM Systems
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Copyright © Microsoft Corporation
# Copyright 2021 EPAM Systems
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
"""Utils for env management."""
import os
from typing import List, Optional
def getenv_by_names(names: List[str]) -> Optional[str]:
"""
Checking several env variables and return result value by priority.
:param names: Env variables names for checking
:type names: List[str]
:return: Env variable result
:rtype: Optional[str]
"""
for env_name in names:
env_value = os.getenv(env_name)
if env_value:
return env_value
return None
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment