Project 'osdu/platform/data-flow/data-loading/open-test-data' was moved to 'osdu/data/open-test-data'. Please update any links and bookmarks that may still have the old path.
WorkProduct manifests generation fails while validating Manifest.1.0.0.json
WorkProduct manifests generation fails with the following error each time Manifest.1.0.0.json is validated:
2024-01-10 09:46:53 ERROR Unable to process document file: 69_D_CH_11
Traceback (most recent call last):
File "/home/zhubin/.local/lib/python3.10/site-packages/jsonschema/validators.py", line 1098, in resolve_from_url
document = self.store[url]
File "/home/zhubin/.local/lib/python3.10/site-packages/jsonschema/_utils.py", line 20, in __getitem__
return self.store[self.normalize(uri)]
KeyError: 'https://schema.osdu.opengroup.org/json/abstract/GenericWorkProductComponent.1.0.0.json'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/zhubin/.local/lib/python3.10/site-packages/urllib3/connection.py", line 159, in _new_conn
conn = connection.create_connection(
File "/home/zhubin/.local/lib/python3.10/site-packages/urllib3/util/connection.py", line 61, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/usr/lib/python3.10/socket.py", line 955, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/zhubin/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 670, in urlopen
httplib_response = self._make_request(
File "/home/zhubin/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 381, in _make_request
self._validate_conn(conn)
File "/home/zhubin/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 978, in _validate_conn
conn.connect()
File "/home/zhubin/.local/lib/python3.10/site-packages/urllib3/connection.py", line 309, in connect
conn = self._new_conn()
File "/home/zhubin/.local/lib/python3.10/site-packages/urllib3/connection.py", line 171, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f67bb1e4e80>: Failed to establish a new connection: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/zhubin/.local/lib/python3.10/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/home/zhubin/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 726, in urlopen
retries = retries.increment(
File "/home/zhubin/.local/lib/python3.10/site-packages/urllib3/util/retry.py", line 446, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='schema.osdu.opengroup.org', port=443): Max retries exceeded with url: /json/abstract/GenericWorkProductComponent.1.0.0.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f67bb1e4e80>: Failed to establish a new connection: [Errno -2] Name or service not known'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/zhubin/.local/lib/python3.10/site-packages/jsonschema/validators.py", line 1101, in resolve_from_url
document = self.resolve_remote(url)
File "/home/zhubin/.local/lib/python3.10/site-packages/jsonschema/validators.py", line 1202, in resolve_remote
result = requests.get(uri).json()
File "/home/zhubin/.local/lib/python3.10/site-packages/requests/api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "/home/zhubin/.local/lib/python3.10/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/home/zhubin/.local/lib/python3.10/site-packages/requests/sessions.py", line 530, in request
resp = self.send(prep, **send_kwargs)
File "/home/zhubin/.local/lib/python3.10/site-packages/requests/sessions.py", line 643, in send
r = adapter.send(request, **kwargs)
File "/home/zhubin/.local/lib/python3.10/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='schema.osdu.opengroup.org', port=443): Max retries exceeded with url: /json/abstract/GenericWorkProductComponent.1.0.0.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f67bb1e4e80>: Failed to establish a new connection: [Errno -2] Name or service not known'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/zhubin/open-test-data/rc--3.0.0/2-scripts/load_manifest_scripts/src/loading_manifest/osdu_document_manifest.py", line 155, in create_document_manifest_from_path
obj=create_document_manifest(
File "/home/zhubin/open-test-data/rc--3.0.0/2-scripts/load_manifest_scripts/src/loading_manifest/osdu_document_manifest.py", line 75, in create_document_manifest
document = cm.create_loading_manifest(
File "/home/zhubin/open-test-data/rc--3.0.0/2-scripts/load_manifest_scripts/src/loading_manifest/common_manifest.py", line 272, in create_loading_manifest
validate_schema(lm_group, schema_id_lm, dict_schemas)
File "/home/zhubin/open-test-data/rc--3.0.0/2-scripts/load_manifest_scripts/src/loading_manifest/common_manifest.py", line 661, in validate_schema
jsonschema.validate(lm, schema, resolver=resolver)
File "/home/zhubin/.local/lib/python3.10/site-packages/jsonschema/validators.py", line 1306, in validate
error = exceptions.best_match(validator.iter_errors(instance))
File "/home/zhubin/.local/lib/python3.10/site-packages/jsonschema/exceptions.py", line 456, in best_match
best = next(errors, None)
File "/home/zhubin/.local/lib/python3.10/site-packages/jsonschema/validators.py", line 368, in iter_errors
for error in errors:
File "/home/zhubin/.local/lib/python3.10/site-packages/jsonschema/_keywords.py", line 295, in properties
yield from validator.descend(
File "/home/zhubin/.local/lib/python3.10/site-packages/jsonschema/validators.py", line 416, in descend
for error in errors:
File "/home/zhubin/.local/lib/python3.10/site-packages/jsonschema/_keywords.py", line 295, in properties
yield from validator.descend(
File "/home/zhubin/.local/lib/python3.10/site-packages/jsonschema/validators.py", line 416, in descend
for error in errors:
File "/home/zhubin/.local/lib/python3.10/site-packages/jsonschema/_legacy_keywords.py", line 135, in items_draft6_draft7_draft201909
yield from validator.descend(item, items, path=index)
File "/home/zhubin/.local/lib/python3.10/site-packages/jsonschema/validators.py", line 416, in descend
for error in errors:
File "/home/zhubin/.local/lib/python3.10/site-packages/jsonschema/_keywords.py", line 274, in ref
yield from validator._validate_reference(ref=ref, instance=instance)
File "/home/zhubin/.local/lib/python3.10/site-packages/jsonschema/validators.py", line 461, in _validate_reference
scope, resolved = resolve(ref)
File "/home/zhubin/.local/lib/python3.10/site-packages/jsonschema/validators.py", line 1087, in resolve
return url, self._remote_cache(url)
File "/home/zhubin/.local/lib/python3.10/site-packages/jsonschema/validators.py", line 1103, in resolve_from_url
raise exceptions._RefResolutionError(exc)
jsonschema.exceptions._RefResolutionError: HTTPSConnectionPool(host='schema.osdu.opengroup.org', port=443): Max retries exceeded with url: /json/abstract/GenericWorkProductComponent.1.0.0.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f67bb1e4e80>: Failed to establish a new connection: [Errno -2] Name or service not known'))
After discussion with @deepapathak she found out that changing the version of jsonschema
to 3.2.0
in open-test-data/rc--3.0.0/2-scripts/load_manifest_scripts/requirements.txt
fixes the issue: