eds_naturalization DAG doesn't use correct schema authority

This issue has been identified as part of the M24 eds_naturalization DAG basic workflow validation. The problem arises from the incorrect selection of schema authority, specifically when fetching the schema for the given kind. In M23, the schema authority was sourced from an environment variable. However, in M24, the system now sends a request to the schema service, providing the source, entity type, and version information (major and minor). When multiple schemas meet these criteria, the eds_naturalization DAG selects the first schema in the returned list. This leads to the selection of an incorrect schema authority.

Could you please clarify why the eds_naturalization DAG defaults to choosing the first element in the response when multiple schemas are returned?

i.e. request: curl --request GET
--url 'https://host/api/schema-service/v1/schema?source=wks&entityType=work-product-component--WellLog&schemaVersionMajor=1&schemaVersionMinor=0&latestVersion=True'
--header 'Authorization: Bearer '
--header 'data-partition-id: '

response:

{
 "schemaInfos": [
 {
 "schemaIdentity": {
 "authority": "authority3",
 "source": "wks",
 "entityType": "work-product-component--WellLog",
 "schemaVersionMajor": 1,
 "schemaVersionMinor": 0,
 "schemaVersionPatch": 0,
 "id": "authority3:wks:work-product-component--WellLog:1.0.0"
 },
 "createdBy": "<user3>",
 "dateCreated": "<date3>",
 "status": "DEVELOPMENT",
 "scope": "INTERNAL"
 },
 {
 "schemaIdentity": {
 "authority": "authority2",
 "source": "wks",
 "entityType": "work-product-component--WellLog",
 "schemaVersionMajor": 1,
 "schemaVersionMinor": 0,
 "schemaVersionPatch": 0,
 "id": "authority2:wks:work-product-component--WellLog:1.0.0"
 },
 "createdBy": "<user2>",
 "dateCreated": "<date2>",
 "status": "DEVELOPMENT",
 "scope": "INTERNAL"
 },
 {
 "schemaIdentity": {
 "authority": "authority1",
 "source": "wks",
 "entityType": "work-product-component--WellLog",
 "schemaVersionMajor": 1,
 "schemaVersionMinor": 0,
 "schemaVersionPatch": 0,
 "id": "authority1:wks:work-product-component--WellLog:1.0.0"
 },
 "createdBy": "<user1>",
 "dateCreated": "<date>",
 "status": "PUBLISHED",
 "scope": "SHARED"
 }
 ],
 "offset": 0,
 "count": 3,
 "totalCount": 3
 }

in the scenario described, the eds_naturalization DAG sends a request to the search service with the authority for the kind set to "authority3," even though the welllog record’s kind has the authority set to "authority1." This mismatch results in an empty response from the search service, which is causing the following error.

Upon further investigation, I have confirmed that the error below is directly attributable to the authority mismatch I described earlier:

  File "/home/airflow/.local/lib/python3.11/site-packages/osdu_airflow/eds/eds_naturalization/naturalization_of_dataset.py", line 134, in naturalization
    for dataset_id in wpc_record["data"]["Datasets"]:
                      ~~~~~~~~~~^^^^^^^^
TypeError: string indices must be integers, not 'str'