Merge branch 'gcz-qa-artifacts-m25-helper-lr' into 'main'

Add supporting docs and scripts for GCZ/AGO/Portal testing and sync

See merge request !905

R3-M25/QA_Artifacts_M25/GCZ/GCZ_AGO_Testing.md

+# GCZ - ArcGIS Online Testing
+
+Aside from querying GCZ Layers directly in Postman, you may also wish to test them in a map client. You have a few options:
+
+* Try out a local test using the ArcGIS Maps SDK for JavaScript. We have a sample [here](https://community.opengroup.org/osdu/platform/consumption/geospatial/-/blob/master/docs/test-assets/webapps/esri_jsapi_test.html?ref_type=heads).
+* Try connecting GCZ layers to ArcGIS Online or ArcGIS Enterprise Portal (instructions below)
+* Use alternative mapping client that is compatible with the Esri Feature Service or GeoJSON REC 7946 spec
+
+## ArcGIS Online or Enterprise Connection Tutorial
+
+### Prerequisite
+These instructions assume the following:
+* GCZ is up and running and queriable
+    * i.e. this kind of query should return records: {{PROVIDER_URL}}/rest/services/gcz/FeatureServer/1/query
+* GCZ may be secured behind something like ISTIO service mesh which requires custom headers (access-token, data-partition-id) to be included in every request
+    * GCZ security background [here](https://community.opengroup.org/osdu/platform/consumption/geospatial/-/blob/master/docs/security/gcz-security-considerations.md?ref_type=heads)
+* Service Mesh configuration has been updated to optionally allow for the use of custom URL parameters instead of relying on custom Headers alone
+
+Why do custom URL parameters need to be supported? Because ArcGIS Online and ArcGIS Enterprise Portal do not allow custom Headers to be baked into any externally linked item or request, whereas they do include limited support for custom URL parameters.
+
+### Instructions for immediate access
+1. Open Map Viewer
+2. Select “Add layer from URL”
+3. Copy/Paste the entire GCZ Feature Layer URL – Please include the access-token and data-partition-id URL parameters
+    * e.g. {{PROVIDER_URL}}/rest/services/gcz/FeatureServer/1?data-partition-id=osdu&access-token=123
+    * Note that this is not the “/query” endpoint. That endpoint will not work here.
+4. The type should default to “ArcGIS Server web service”
+5. Next, add the two custom parameters again using the “Add custom parameter” option
+    * e.g. data-partition-id=osdu
+    * e.g. access-token=123
+6. Finally, remove the custom parameters from the URL
+    * e.g. {{PROVIDER_URL}}/rest/services/gcz/FeatureServer/1
+    * If you have issues with these step, you can try leaving the Parameters in the URL, though they should not be needed once custom parameters are supplied.
+8. Click “Add to map”
+9. Observe the layer in the map
+10. If the GCZ Cache is up and running and data is contained within, then features should be visible on the map
+11. Click “Save as” to save the layer as a portal item
+12. See portal item which can now be opened directly in map viewer
+13. **Note that this portal item will only be valid for the current life of the access token**
+    * Once the token expires, the item is essentially broken until it has been updated with a valid token either through the Python API (i.e. our script) or through something like ArcGIS Assistant.
+
+### Instructions for long-term access
+
+#### Prerequisite
+* You have successfully completed instructions for immediate access above
+* You wish to sustain access by updating the access-token automatically, through a script
+* You have access to ArcGIS Notebook or some other Python script hosting environment
+    * If non-Notebook environment, then you will need credentials for a built-in AGOL account with editing priveleges on at least the GCZ Items in question
+
+#### Steps
+1. Retrieve the script hosted alongside this document. It is labeled GCZ_Custom_Parameters_sync.py
+2. Follow the instructions / comments in the script to make sure it (and the associated config.ini, if applicable) are updated for your intended usage and environment.
+3. Use ArcGIS Notebook (or some script hosting equal) to run the script ad-hoc or on a schedule that is equal to or lesser than the lifespan of an access-token.
+4. This should ensure that all GCZ Portal Items are kept up-to-date with the latest access token from OSDU
\ No newline at end of file

R3-M25/QA_Artifacts_M25/GCZ/Portal_Sync_Script/GCZ_Custom_Parameters_sync.py

+# %% [markdown]
+# ## GCZ - Custom Parameter Sync
+#
+
+# %%
+
+"""
+Script for AGOL GCZ integration - will update portal items matching a given tag so that their customParameters contain an updated token key. This is to enable layers to fetch data from GCZ through an ISTIO security layer
+author: Austin Zang, Levi Remington
+updated: 4/03/2025
+
+Prerequisites for testing:
+- GCZ up and running and feature layer is accessible
+- GCZ Portal Item (pointing to one of the layers) created using the steps outlined in the ArcGIS Online Compatibility section of the GCZ Security docs: 
+https://community.opengroup.org/osdu/platform/consumption/geospatial/-/blob/master/docs/security/gcz-security-considerations.md#arcgis-online-compatibility
+   - Please ensure that you've defined a custom parameter with a key equivalent to the TARGET_KEY variable specified below
+- GCZ Portal Item has been tagged with a unique value matching the TARGET_TAG variable specified below
+
+Notes:
+- It is best to run this script on a schedule a few minutes short of the token lifespan, this way the associated portal items can remain up-to-date without fear of expiring
+- In the absense of a scheduler, it is ok to run this script ad-hoc for testing
+- Please review all Env variables defined below, and make sure they are up-to-date and accurate for your environment
+"""
+
+from arcgis.gis import GIS
+import ssl
+import json
+import requests
+import configparser
+import arcgis
+
+
+# If outside of Notebook, you may wish to use an external config file for retrieving sensitive credentials
+configFile = "config.ini"
+env = "MY_ENV"
+
+config = configparser.ConfigParser(strict=False, interpolation=None)
+config.read(configFile)
+
+# If inside Notebook, you could leverage Script parameters to extract the values from an isolated source.
+# Otherwise, please explicitly define each value below in the GLOBAL Env variables section:
+
+# %%
+# GLOBAL Env variables for token retrieval
+URL = config[env]["URL"]
+CLIENT_ID = config[env]["CLIENT_ID"]
+CLIENT_SECRET = config[env]["CLIENT_SECRET"]
+SCOPE = config[env]["SCOPE"]
+
+PAYLOAD = f"grant_type=client_credentials&client_id={CLIENT_ID}&client_secret={CLIENT_SECRET}&scope={SCOPE}"
+HEADERS = {"Content-Type": "application/x-www-form-urlencoded"}
+
+# Env variables for portal
+PORTAL_URL = config[env]["PORTAL_URL"]
+PORTAL_USERNAME = config[env]["PORTAL_USERNAME"]
+PORTAL_PASSWORD = config[env]["PORTAL_PASSWORD"]
+
+# Env variables to specify details about the GCZ Item
+# What tag are you using to distinguish GCZ items within Portal? (e.g. my-gcz-items)
+TARGET_TAG = config[env]["TARGET_TAG"]
+# What URL Parameter needs to be updated? (e.g. access-token)
+TARGET_KEY = config[env]["TARGET_KEY"]
+
+# Construct an enterprise login in the style of the following:
+# CHOOSE ONE OF THE FOLLOWING TWO OPTIONS:
+# OPTION 1: If within Notebook, you can use this:
+# GIS('home')
+gis = GIS('home')
+
+# OPTION 2: If outside of notebook, you must include credentials of a built-in account for access
+# Built-in account should have editing access to the GCZ items. You can use Groups to make this easier.
+# GIS connection API docs: https://developers.arcgis.com/python/latest/api-reference/arcgis.gis.toc.html
+# Working with different auth schemes: https://developers.arcgis.com/python/latest/guide/working-with-different-authentication-schemes/
+# GIS("https://portalname.domain.com/webadapter_name", "sharinguser", "password")
+# gis = GIS(PORTAL_URL, PORTAL_USERNAME, PORTAL_PASSWORD, use_gen_token=True)
+
+
+# %%
+# TODO docstrings
+def get_token(url, headers, payload):
+    """Intended to fetch an access token from OSDU endpoint using the provided URL, headers, and payload.
+
+    Args:
+        url (str): api endpoint to request from
+        headers (str): header information for the request
+        payload (str): payload information for the request
+
+    Returns:
+        str: token if successful, None otherwise
+    """
+
+    response = requests.request("POST", url, headers=headers, data=payload)
+
+    if response.ok:
+        print("Successful token fetch request")
+        return response.json()["access_token"]
+    else:
+        print(f"Request error: {response.text}")
+        return None
+
+
+# %%
+def search_by_tag(gis, tag):
+    """Search gis portal for items with a specific tag
+
+    Args:
+        gis (GIS): Esri Python API GIS object
+        tag (str): Tag that items should be searched for
+
+    Returns:
+        results list from gis.content.search() if successful, None otherwise
+    """
+
+    try:
+        # results = gis.content.search(query=f'tags:{tag}')
+        results = gis.content.search(query="*", max_items=-1, filter=f'tags:"{tag}"')
+    except Exception as e:
+        print(f"GIS content search failed with error {e}")
+        return None
+
+    if results:
+        return results
+    else:
+        print(f"No items found with tag {tag}")
+
+        unfiltered_results = gis.content.search(query="*", max_items=-1)
+
+        print(f"If we search without a tag-filter, we see {len(unfiltered_results)} items.")
+
+        return None
+
+
+# %%
+def update_custom_parameters(portal_item, property_key, new_token):
+    """Update the custom parameters within the JSON of a portal item - only updates layers with existing customParameters
+
+    Args:
+        portal_item (ArcGIS Python API Portal Item):
+        property_key (str): The JSON Key of the custom parameter to update. Ex: 'token', 'api_token', 'access_token'
+        new_token (str): API token to insert into the custom parameter, with the key defined in property_key
+
+    Returns:
+        dict: The updated JSON of the portal item, or None if the JSON was unchanged or unsuccessful
+    """
+
+    json_changed = False
+
+    print(f"Updating json of {portal_item.title}")
+
+    try:
+        data = portal_item.get_data()
+        
+    except Exception as e:
+        print(
+            f"Failed get JSON data of {portal_item.title} with full error below: \n {e}"
+        )
+        return None
+
+    layer_object_name = ""
+    if "operationalLayers" in data:
+        layer_object_name = "operationalLayers"
+        print(
+            f"{len(data[layer_object_name])} layers under key '{layer_object_name}' found to process"
+        )
+
+    elif "layers" in data:
+        layer_object_name = "layers"
+        print(
+            f"{len(data[layer_object_name])} layers under key '{layer_object_name}' found to process"
+        )
+
+    else:
+        print("Unable to identify layers object within item JSON data")
+        return
+
+    # Add custom parameters
+    for i, layer in enumerate(data[layer_object_name]):
+        if "customParameters" in layer:
+            layer["customParameters"][property_key] = new_token
+            print(f"Layer {i+1} will be updated with new token")
+            json_changed = True
+        else:
+            print(
+                f"Layer {i+1} skipped due to not containing customParameters to update"
+            )
+
+    # Update JSON
+    item_properties = {"text": json.dumps(data)}
+
+    if json_changed:
+        try:
+            portal_item.update(item_properties=item_properties)
+        except Exception as e:
+            print(
+                f"Failed to commit JSON update of {portal_item.title}. Full error below: \n {e}"
+            )
+            return None
+
+        print(f"Successfully commited updates of {portal_item.title} JSON")
+        return portal_item.get_data()
+
+    print("Portal item unchanged, no changes to commit")
+    return
+
+
+# %%
+def main():
+    """
+    Run script using global env variables defined at top block
+    """
+
+    current_token = get_token(URL, HEADERS, PAYLOAD)
+
+    if not current_token:
+        print("Token fetch unsuccessful, ending")
+        return
+    
+    search_results = search_by_tag(gis, TARGET_TAG)
+
+    if not search_results:
+        print("No valid portal items found, ending")
+        return
+
+    print(f"Found {len(search_results)} portal items under tag {TARGET_TAG}")
+    for p_item in search_results:
+
+        update_result = update_custom_parameters(p_item, TARGET_KEY, current_token)
+       
+
+# %%
+main()

R3-M25/QA_Artifacts_M25/GCZ/Portal_Sync_Script/config.ini

+; config.ini
+[MY_ENV]
+URL = some-token-host/oauth2/v2.0/token
+CLIENT_ID = 
+CLIENT_SECRET = 
+SCOPE = 
+PORTAL_URL = https://portalname.domain.com/webadapter_name
+PORTAL_USERNAME = non_iwa_gczitemowner
+PORTAL_PASSWORD = non_iwa_gczitemownerpassword
+TARGET_TAG = my-gcz-items
+TARGET_KEY = access-token