Repository navigation
Invoke ML Pipeline step in Trigger ML Pipeline is failing. #411
Description
Invoke ML Pipeline step in Trigger ML Pipeline is failing. But i do not see any error .Appreciate for any help.
Task : Invoke ML pipeline
Description : Invoke Azure Machine Learning published pipelines.
Version : 0.1.20
Author : ms-air-aiagility
Help URL : https://marketplace.visualstudio.com/items?itemName=ms-air-aiagility.vss-services-azureml&ssr=false#overview
2022-10-27T20:28:35.0008554Z POST https://centralus.aether.ms/api/v1.0/subscriptions/71fc788e-df0b-4d38-9d88-67702c37e3b7/resourceGroups/mlops-RG/providers/Microsoft.MachineLearningServices/workspaces/mlops-AML-WS/PipelineRuns/PipelineSubmit/7616c9f1-6800-4fb8-a1ce-a939671633a4
Request body: {
"AzDevOpsCallBackInfo": {
"JobId": "77aee9c5-85bc-59ae-fa90-c3f20f9c111e",
"HubName": "build",
"TaskInstanceId": "7ffef4a3-944d-544a-2764-6fcc510d84c9",
"PlanId": "62957fff-3f5b-4233-af61-4c199f045e84",
"ProjectId": "18154d0f-4be3-420a-9ef3-1eb573c98452",
"PlanUri": "https://dev.azure.com/MLOpsPython123/",
"AuthToken": ***
},
"ExperimentName": "mlopspython",
"RunSource": "PipelineRun",
"ParameterAssignments": {
"model_name": "diabetes_regression_model.pkl"
},
"tags": {
"BuildId": "43",
"BuildUri": "https://dev.azure.com/MLOpsPython123/ML/_build/results?buildId=43"
},
"StepTags": {
"BuildId": "43",
"BuildUri": "https://dev.azure.com/MLOpsPython123/ML/_build/results?buildId=43"
}
}
Response Code: OK
Response: {
"Description": null,
"Status": {
"StatusCode": 0,
"StatusDetail": null,
"CreationTime": "2022-10-27T20:28:34.520663Z",
"EndTime": null
},
"GraphId": "ba7b15ff-8c7b-4445-b451-7d65e2a7ae5f",
"IsSubmitted": false,
"HasErrors": false,
"HasWarnings": false,
"UploadState": 0,
"ParameterAssignments": {
"model_name": "diabetes_regression_model.pkl",
"dataset_version": "latest",
"data_file_path": "none",
"caller_run_id": "none"
},
"DataPathAssignments": {},
"DataSetDefinitionValueAssignments": {},
"AssetOutputSettingsAssignments": {},
"RunHistoryExperimentName": "mlopspython",
"DisplayName": null,
"PipelineRunId": "bc80c6dd-6c94-486a-af73-18ea1e387633",
"PipelineId": "7616c9f1-6800-4fb8-a1ce-a939671633a4",
"PipelineEndpointId": null,
"RunSource": "PipelineRun",
"RunType": 0,
"TotalRunSteps": 3,
"ScheduleId": null,
"RunUrl": "https://ml.azure.com/experiments/mlopspython/runs/bc80c6dd-6c94-486a-af73-18ea1e387633?tid=a6fc8bb4-56cc-4ac5-a77e-3257928f9ffe&wsid=/subscriptions/71fc788e-df0b-4d38-9d88-67702c37e3b7/resourcegroups/mlops-RG/workspaces/mlops-AML-WS",
"tags": {
"BuildId": "43",
"BuildUri": "https://dev.azure.com/MLOpsPython123/ML/_build/results?buildId=43"
},
"StepTags": {
"BuildId": "43",
"BuildUri": "https://dev.azure.com/MLOpsPython123/ML/_build/results?buildId=43"
},
"Properties": {},
"StepProperties": {},
"CreatedBy": {
"UserObjectId": "831de4ec-e437-4ade-9da6-1ca41782937f",
"UserTenantId": "a6fc8bb4-56cc-4ac5-a77e-3257928f9ffe",
"UserName": "49103e81-f656-4c39-a843-93fab82bc857"
},
"PreserveSubGraphs": true,
"RootPipelineRunId": "bc80c6dd-6c94-486a-af73-18ea1e387633",
"EnforceRerun": null,
"ContinueRunOnFailedOptionalInput": true,
"UserAlias": null,
"DefaultCloudPriority": null,
"DefaultCompute": null,
"DefaultDatastore": null,
"IdentityConfig": null,
"PipelineTimeoutSeconds": null,
"ContinueRunOnStepFailure": false,
"EntityStatus": 0,
"Id": "bc80c6dd-6c94-486a-af73-18ea1e387633",
"Etag": ""8c0053de-0000-0300-0000-635ae9f20000"",
"CreatedDate": "2022-10-27T20:28:34.520914Z",
"LastModifiedDate": "2022-10-27T20:28:34.520914Z"
}
Activity
ruqianq commentedon Oct 28, 2022
You might need to log into your Azure ML studio and find that experiment. it would have the detail error log.
cotus997 commentedon Nov 16, 2022
Are you running on a free subscription? if yes, you have to change the low priority clusters into dedicated one in the diabete-regression-variable-template.yml file.
- name: AML_CLUSTER_PRIORITY_SCORINGvalue: dedicatedcodemohsmohammad789 commentedon Jul 12, 2023
I am getting the same error please help
havfront commentedon Aug 11, 2023
Also getting this error. Although on a paid subscription, tried to change from 'lowpriority' to 'dedicated', but again same problem.
In my case, it seems like the ML-model is not being registered, as reported by the next pipeline step (Determine if evaluation succeeded and new model is registered (CLI)):
.../bin/bash /__w/_temp/azureclitaskscript1691755595349.shModel was not registered for this run.##[error]Script failed with error: Error: The process '/bin/bash' failed with exit code 1at ExecState._setResult (/__w/_tasks/AzureCLI_46e4be58-730b-4389-8a2f-ea10b3e5e815/1.225.0/node_modules/azure-pipelines-task-lib/toolrunner.js:943:25)at ExecState.CheckComplete (/__w/_tasks/AzureCLI_46e4be58-730b-4389-8a2f-ea10b3e5e815/1.225.0/node_modules/azure-pipelines-task-lib/toolrunner.js:926:18)at ChildProcess.<anonymous> (/__w/_tasks/AzureCLI_46e4be58-730b-4389-8a2f-ea10b3e5e815/1.225.0/node_modules/azure-pipelines-task-lib/toolrunner.js:839:19)at ChildProcess.emit (node:events:513:28)at maybeClose (node:internal/child_process:1100:16)at Socket.<anonymous> (node:internal/child_process:458:11)at Socket.emit (node:events:513:28)at Pipe.<anonymous> (node:net:301:12)...I've not yet figured out where to find a log of the model registration process, or some other useful info about it. Seems like I would need some help for this.
codervux commentedon Dec 10, 2023
hello havfront, I also get this error. Did you fix it? please give me a hand...