PKS force delete

Playing in the lab the today I came across a scenario in which a pks create-cluster failed;

pks cluster team

Name:                     team
Plan Name:                small
UUID:                     3ef60125-5f93-4b98-b50c-7050b6877fdc
Last Action:              CREATE
Last Action State:        failed
Last Action Description:  Instance provisioning failed: There was a problem completing your request. Please contact your operations team providing the following information: service: p.pks, service-instance-guid: 3ef60125-5f93-4b98-b50c-7050b6877fdc, broker-request-id: dc11b0b4-2a76-4b01-ad42-fe40507b70e6, task-id: 561, operation: create
Kubernetes Master Host:   team.pks.lab01.pcf.pw
Kubernetes Master Port:   8443
Worker Instances:         3
Kubernetes Master IP(s):  In Progress

First of all I looked at why the cluster create task failed (task 561 from the error message) using bosh task 561. That highlighted a typo I had with regards to access keys. Now that I know what the issue is I want to delete the cluster;

pks delete-cluster team

Deletion of team in progress
pks cluster team

Name:                     team
Plan Name:                small
UUID:                     3ef60125-5f93-4b98-b50c-7050b6877fdc
Last Action:              DELETE
Last Action State:        failed
Last Action Description:  Instance deletion failed: There was a problem completing your request. Please contact your operations team providing the following information: service: p.pks, service-instance-guid: 3ef60125-5f93-4b98-b50c-7050b6877fdc, broker-request-id: cb5f0ad3-9aa1-4b88-a857-d14fe0a6860f, task-id: 610, operation: delete
Kubernetes Master Host:   team.pks.lab01.pcf.pw
Kubernetes Master Port:   8443
Worker Instances:         3
Kubernetes Master IP(s):  In Progress

However this failed! Looking at the bosh logs (task id in the error message) highlighted that the deployment was missing (not worked/completed etc.).

bosh -e gcp task 610
Using environment '192.168.101.10' as user 'director' (bosh.*.read, openid, bosh.*.admin, bosh.read, bosh.admin)

Task 610

Task 610 | 15:58:40 | Preparing deployment: Preparing deployment (00:00:00)
                    L Error: - Deployment '' doesn't exist
Task 610 | 15:58:40 | Error: - Deployment '' doesn't exist

Task 610 Started  Tue Aug 14 15:58:40 UTC 2018
Task 610 Finished Tue Aug 14 15:58:40 UTC 2018
Task 610 Duration 00:00:00
Task 610 error

Capturing task '610' output:
  Expected task '610' to succeed but state is 'error'

Exit code 1

As a result I decided to look at what Bosh thought it had deployed via a different command. As you can see deployment 3ef60125-5f93-4b98-b50c-7050b6877fdc has no VMs (and the UUID is the one related to the pks commands.

bosh -e gcp vms
Using environment '192.168.101.10' as user 'director' (bosh.*.read, openid, bosh.*.admin, bosh.read, bosh.admin)

Deployment 'pivotal-container-service-20a20b27578b472d13ce'

Instance                                                        Process State  AZ              IPs             VM CID                                   VM Type  Active
pivotal-container-service/5cd61225-c054-49c7-8173-99bb4178b493  running        europe-west2-a  192.168.101.11  vm-f37ec602-73e0-4205-5f17-94d609a24182  large    true

1 vms

Deployment 'service-instance_3ef60125-5f93-4b98-b50c-7050b6877fdc'

Instance  Process State  AZ  IPs  VM CID  VM Type  Active

0 vms

Succeeded

So rather than typing to get the pks CLI to force delete the deployment I used bosh CLI to delete the deployment in question.

bosh -e gcp -d service-instance_3ef60125-5f93-4b98-b50c-7050b6877fdc delete-deployment
Using environment '192.168.101.10' as user 'director' (bosh.*.read, openid, bosh.*.admin, bosh.read, bosh.admin)

Using deployment 'service-instance_3ef60125-5f93-4b98-b50c-7050b6877fdc'

Continue? [yN]: y

Task 926

Task 926 | 08:01:28 | Deleting instances: apply-addons/41df5316-c626-4c24-837d-2261d7cd4bf7 (0)
Task 926 | 08:01:28 | Deleting instances: master/10b23cee-b223-45e0-964c-d3ffdf18abcc (0)
Task 926 | 08:01:28 | Deleting instances: worker/4ce63992-1886-4f99-a844-aabe675849e5 (1)
Task 926 | 08:01:28 | Deleting instances: worker/1fcfc532-7a9f-4799-b65d-e6bcfd94750e (0)
Task 926 | 08:01:28 | Deleting instances: worker/0ac0f25a-8738-4e55-9a62-789ffec98572 (2)
Task 926 | 08:01:29 | Deleting instances: master/10b23cee-b223-45e0-964c-d3ffdf18abcc (0) (00:00:01)
Task 926 | 08:01:29 | Deleting instances: worker/1fcfc532-7a9f-4799-b65d-e6bcfd94750e (0) (00:00:01)
Task 926 | 08:01:29 | Deleting instances: apply-addons/41df5316-c626-4c24-837d-2261d7cd4bf7 (0) (00:00:01)
Task 926 | 08:01:29 | Deleting instances: worker/4ce63992-1886-4f99-a844-aabe675849e5 (1) (00:00:01)
Task 926 | 08:01:29 | Deleting instances: worker/0ac0f25a-8738-4e55-9a62-789ffec98572 (2) (00:00:01)
Task 926 | 08:01:29 | Removing deployment artifacts: Detaching stemcells (00:00:00)
Task 926 | 08:01:29 | Removing deployment artifacts: Detaching releases (00:00:00)
Task 926 | 08:01:29 | Deleting properties: Destroying deployment (00:00:00)

Task 926 Started  Wed Aug 15 08:01:28 UTC 2018
Task 926 Finished Wed Aug 15 08:01:29 UTC 2018
Task 926 Duration 00:00:01
Task 926 done

Succeeded

Now that is complete lets make sure that the pks CLI reflects that the deployment (K8s cluster) has gone.

pks clusters

Name     Plan Name  UUID                                  Status  Action
team     small      3ef60125-5f93-4b98-b50c-7050b6877fdc  failed  DELETE

Initially it still showed up in the list so to try and kick the CLI into querying the lasted bosh state I reissued the delete-cluster.

pks delete-cluster team

Error: Cluster has already been deleted

Which then cleared up the error'ed cluster and lets me start again once I have fixed the original typo with service accounts.

pks clusters

Name  Plan Name  UUID  Status  Action

Show Comments

Get the latest posts delivered right to your inbox.