Pas Apicella

It taken some time but now I officially was able to test PKS with NSX-T rather then using Flannel.

While there is a bit of initial setup to install NSX-T and PKS and then ensure PKS networking is NSX-T, the ease of rolling out multiple Kubernetes clusters with unique networking is greatly simplified by NSX-T. Here I am going to show what happens after pushing a workload to my PKS K8s cluster

First Before we can do anything we need the following...

Pre Steps

1. Ensure you have NSX-T setup and a dashboard UI as follows


2. Ensure you have PKS installed in this example I have it installed on vSphere which at the time of this blog is the only supported / applicable version we can use for NSX-T



PKS tile would need to ensure it's setup to use NSX-T which is done on this page of the tile configuration



3. You can see from the NSX-T manager UI we have a Load Balancers setup as shown below. Navigate to "Load Balancing -> Load Balancers"



And this Load Balancer is backed by few "Virtual Servers", one for http (port 80) and the other for https (port 443), which can be seen when you select the Virtual Servers link


From here we have logical switches created for each of the Kubernetes namespaces. We see two for our load balancer, and the other 3 are for the 3 K8s namespaces which are (default, kube-public, kube-system)


Here is how we verify the namespaces we have in our K8s cluster

pasapicella@pas-macbook:~/pivotal $ kubectl get ns
NAME          STATUS    AGE
default       Active    5h
kube-public   Active    5h
kube-system   Active    5h

All of the logical switches are connected to the T0 Logical Switch by a set of T1 Logical Routers


For these to be accessible, they are linked to the T0 Logical Router via a set of router ports



Now lets push a basic K8s workload and see what NSX-T and PKS give us out of the box...

Steps

Lets create our K8s cluster using the PKS CLI. You will need a PKS CLI user which can be created following this doc

https://docs.pivotal.io/runtimes/pks/1-1/manage-users.html

1. Login using the PKS CLI as follows

$ pks login -k -a api.pks.haas-148.pez.pivotal.io -u pas -p ****

2. Create a cluster as shown below

$ pks create-cluster apples --external-hostname apples.haas-148.pez.pivotal.io --plan small

Name:                     apples
Plan Name:                small
UUID:                     d9f258e3-247c-4b4c-9055-629871be896c
Last Action:              CREATE
Last Action State:        in progress
Last Action Description:  Creating cluster
Kubernetes Master Host:   apples.haas-148.pez.pivotal.io
Kubernetes Master Port:   8443
Worker Instances:         3
Kubernetes Master IP(s):  In Progress

3. Wait for the cluster to have created as follows

$ pks cluster apples

Name:                     apples
Plan Name:                small
UUID:                     d9f258e3-247c-4b4c-9055-629871be896c
Last Action:              CREATE
Last Action State:        succeeded
Last Action Description:  Instance provisioning completed
Kubernetes Master Host:   apples.haas-148.pez.pivotal.io
Kubernetes Master Port:   8443
Worker Instances:         3
Kubernetes Master IP(s):  10.1.1.10

The PKS CLI is basically telling BOSH to go ahead an based on the small plan create me a fully functional/working K8's cluster from VM's to all the processes that go along with it and when it's up keep it up and running for me in the event of failure.

His an example of the one of the WORKER VM's of the cluster shown in vSphere Web Client



4. Using the following YAML file as follows lets push that workload to our K8s cluster

apiVersion: v1
kind: Service
metadata:
  labels:
    app: fortune-service
    deployment: pks-workshop
  name: fortune-service
spec:
  ports:
  - port: 80
    name: ui
  - port: 9080
    name: backend
  - port: 6379
    name: redis
  type: LoadBalancer
  selector:
    app: fortune
---
apiVersion: v1
kind: Pod
metadata:
  labels:
    app: fortune
    deployment: pks-workshop
  name: fortune
spec:
  containers:
  - image: azwickey/fortune-ui:latest
    name: fortune-ui
    ports:
    - containerPort: 80
      protocol: TCP
  - image: azwickey/fortune-backend-jee:latest
    name: fortune-backend
    ports:
    - containerPort: 9080
      protocol: TCP
  - image: redis
    name: redis
    ports:
    - containerPort: 6379
      protocol: TCP

5. Push the workload as follows once the above YAML is saved to a file

$ kubectl create -f fortune-teller.yml
service "fortune-service" created
pod "fortune" created

6. Verify the PODS are running as follows

$ kubectl get all
NAME         READY     STATUS    RESTARTS   AGE
po/fortune   3/3       Running   0          35s

NAME                  TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)                                      AGE
svc/fortune-service   LoadBalancer   10.100.200.232   10.195.3.134   80:30591/TCP,9080:32487/TCP,6379:32360/TCP   36s
svc/kubernetes        ClusterIP      10.100.200.1              443/TCP                                      5h

Great so now lets head back to our NSX-T manager UI and see what has been created. From the above output you can see a LB service is created and external IP address assigned

7. First thing you will notice is in "Virtual Servers" we have some new entries for each of our containers as shown below


and ...


Finally the LB we previously had in place shows our "Virtual Servers" added to it's config and routable



More Information

Pivotal Container Service
https://docs.pivotal.io/runtimes/pks/1-1/

VMware NSX-T
https://docs.vmware.com/en/VMware-NSX-T/index.html

Previously I blogged about using "Bubble" or bosh-bootloader as per the post below.

http://theblasfrompas.blogspot.com/2018/08/bosh-bootloader-or-bubble-as-pronounced.html

... and from there setting up Concourse

http://theblasfrompas.blogspot.com/2018/08/deploying-concourse-using-my-bubble.html

.. of course this was created so I can now use the PCF Pipelines to deploy Pivotal Cloud Foundry's Pivotal Application Service (PAS). At a high level this is how to achieve this with some screen shots on the end result

Steps

1. To get started you would use this link as follows. In my example I was deploying PCF to AWS

https://github.com/pivotal-cf/pcf-pipelines/tree/master/install-pcf

AWS Install Pipeline

https://github.com/pivotal-cf/pcf-pipelines/tree/master/install-pcf/aws

2. Create a versioned bucket for holding terraform state. on AWS that will look as follows


3. Unless you ensure AWS pre-reqs are meet you won't be able to install PCF so this link highlights all that you will need for installing PCF on AWS such as key pairs, limits, etc

https://docs.pivotal.io/pivotalcf/2-1/customizing/aws.html

4. Create a public DNS zone, get its zone ID we will need that when we setup the pipeline shortly. I also created a self signed public certificate used for my DNS as part of the setup which is required as well.





5. At this point we can download the PCF Pipelines from network.pivotal.io or you can use the link as follows

https://network.pivotal.io/products/pcf-automation/



6. Once you have unzipped the file you would then change to the directory for the write IaaS in my case "aws"

$ cd pcf-pipelines/install-pcf/aws


7. Change all of the CHANGEME values in params.yml with real values for your AWS env. This file is documented so you are clear with what you need to add and where. Most of the values are defaults of course.

8. Login to concourse using the "fly" command line

$ fly --target pcfconcourse login  --concourse-url https://bosh-director-aws-concourse-lb-f827ef220d02270c.elb.ap-southeast-2.amazonaws.com -k

9. Add pipeline

$ fly -t pcfconcourse set-pipeline -p deploy-pcf -c pipeline.yml -l params.yml

10. Unpause pipeline

$ fly -t pcfconcourse unpause-pipeline -p deploy-pcf

pasapicella@pas-macbook:~/pivotal/aws/pcf-pipelines/pcf-pipelines/install-pcf/aws$ fly -t pcfconcourse pipelines
name        paused  public
deploy-pcf  no      no

11. The pipeline on concourse will look as follows



12. Now to execute the pipeline you have to manually run 2 tasks


At this point the pipeline will kick of automatically. If you need to run-run due to an issue you can manually kick off the task after you fix what you need to fix. The “wipe-env” task will take everything for PAS down and terraform removes all IaaS config as well.

While running each task current state is shown as per the image below


If successful your AWS account will the PCF VM's created for example


Verify that PCF installed is best done using Pivotal Operations Manager as shown below



More Information

https://network.pivotal.io/products/pcf-automation/

Previously I blogged about using "Bubble" or bosh-bootloader as per the post below.

http://theblasfrompas.blogspot.com/2018/08/bosh-bootloader-or-bubble-as-pronounced.html

Now with bosh director deployed it's time to deploy concourse itself. The process is very straight forward as per the steps below

1. First let's clone the bosh concourse deployment using the GitHub project as follows

I decided to try out installing bosh using the bosh-bootloader CLI today. bbl currently supports AWS, GCP, Microsoft Azure, Openstack and vSphere. In this example I started with AWS but it won't be long until try this on GCP

It's worth noting that this can all be done remotely from your laptop once you give BBL the access it needs for the cloud environment.

Steps

1. First your going to need the bosh v2 CLI which you can install here

  https://bosh.io/docs/cli-v2/

Verify:

pasapicella@pas-macbook:~$ bosh -version
version 5.0.1-2432e5e9-2018-07-18T21:41:03Z

Succeeded

2. Second you will need Terrform having a Mac I use brew

$ brew install terrafrom

Verify:

pasapicella@pas-macbook:~$ terraform version
Terraform v0.11.7
3. Now we need to install BBL which is done as follows on a Mac. I also show how to install bosh CLI as well if you missed step 1

$ brew tap cloudfoundry/tap
$ brew install bosh-cli
$ brew install bbl

Further instructions on this link

https://github.com/cloudfoundry/bosh-bootloader

4. At this point your ready to deploy BOSH the instructions for AWS are here

https://github.com/cloudfoundry/bosh-bootloader/blob/master/docs/getting-started-aws.md

Pretty straight forward but here is what I did at this point

5. In order for bbl to interact with AWS, an IAM user must be created. This user will be issuing API requests to create the infrastructure such as EC2 instances, load balancers, subnets, etc.

The user must have the following policy which I just copy into my clipboard to use later:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "logs:*",
                "elasticloadbalancing:*",
                "cloudformation:*",
                "iam:*",
                "kms:*",
                "route53:*",
                "ec2:*"
            ],
            "Resource": "*"
        }
    ]
}


$ aws iam create-user --user-name "bbl-user”

This next command requires you to copy the policy JSON above

$ aws iam put-user-policy --user-name "bbl-user" --policy-name "bbl-policy" --policy-document "$(pbpaste)"

$ aws iam create-access-key --user-name "bbl-user"

You will get a JSON response at this point as follows. Save file created here as it’s used next few steps

{
    "AccessKey": {
        "UserName": "bbl-user",
        "Status": "Active",
        "CreateDate": "2018-08-07T03:30:39.993Z",
        "SecretAccessKey": ".....",
        "AccessKeyId": "........"
    }
}

In the next step BBL will use these commands to create infrastructure on AWS.

6. Now we can pave the infrastructure, Create a Jumpbox, and Create a BOSH Director as well as a LB which I need as I plan to deploy concourse using BOSH.

$ bbl up --aws-access-key-id ..... --aws-secret-access-key ... --aws-region ap-southeast-2 --lb-type concourse --name bosh-director -d -s state --iaas aws

The process takes around 5-8 minutes.

The bbl state directory contains all of the files that were used to create your bosh director. This should be checked in to version control, so that you have all the information necessary to later destroy or update this environment at a later date.

7.  Finally we target the the bosh director as follows. Keep in mind everything we need is stored in the "state" directory as per above

$ eval "$(bbl print-env -s state)"

8. This will set various ENV variables which the bosh CLI will then use to target the bosh director.  Now we need to just prepare ourselves to actually log in. I use a script as follows

target-bosh.sh

bbl director-ca-cert -s state > bosh.crt
export BOSH_CA_CERT=bosh.crt

export BOSH_ENVIRONMENT=$(bbl director-address -s state)

echo ""
echo "Username: $(bbl director-username -s state)"
echo "Password: $(bbl director-password -s state)"
echo ""
echo "Log in using -> bosh log-in"
echo ""

bosh alias-env aws-env

echo "ENV set to -> aws-env"
echo ""

Output When run with password omitted ->

pasapicella@pas-macbook:~/pivotal/aws/pcf-pipelines$ ./target-bosh.sh

Username: admin
Password: ......

Log in using -> bosh log-in

Using environment 'https://10.0.0.6:25555' as client 'admin'

Name      bosh-bosh-director-aws
UUID      3ade0d28-77e6-4b5b-9be7-323a813ac87c
Version   266.4.0 (00000000)
CPI       aws_cpi
Features  compiled_package_cache: disabled
          config_server: enabled
          dns: disabled
          snapshots: disabled
User      admin

Succeeded
ENV set to -> aws-env

9. Finally lets log-in as follows

$ bosh -e aws-env log-in

Output ->

pasapicella@pas-macbook:~/pivotal/aws/pcf-pipelines$ bosh -e aws-env log-in
Successfully authenticated with UAA

Succeeded

10. Last but not least lets see what VM's bosh has under management. These VM's are for my concourse I installed. If you would like to install concourse use this link - https://github.com/cloudfoundry/bosh-bootloader/blob/master/docs/concourse.md

pasapicella@pas-macbook:~/pivotal/aws/pcf-pipelines$ bosh -e aws-env vms
Using environment 'https://10.0.0.6:25555' as client 'admin'

Task 20. Done

Deployment 'concourse'

Instance                                     Process State  AZ  IPs        VM CID               VM Type  Active
db/ec8aa978-1ec5-4402-9835-9a1cbce9c1e5      running        z1  10.0.16.5  i-0d33949ece572beeb  default  true
web/686546be-09d1-43ec-bbb7-d96bb5edc3df     running        z1  10.0.16.4  i-03af52f574399af28  default  true
worker/679be815-6250-477c-899c-b962076f26f5  running        z1  10.0.16.6  i-0efac99165e12f2e6  default  true

3 vms

Succeeded

More Information

https://github.com/cloudfoundry/bosh-bootloader/blob/master/docs/getting-started-aws.md

https://github.com/cloudfoundry/bosh-bootloader/blob/master/docs/howto-target-bosh-director.md

I decided to use CFDOT (CF Diego Operator Toolkit) on my PCF 2.1 vSphere ENV today. Setting it up isn't required as it's installed out of the box on Bosh Managed Diego Cell as shown below. It gives nice detailed information around Cell Capacity and other useful metrics.

1. SSH into Ops Manager VM

pasapicella@pas-macbook:~/pivotal/PCF/APJ/PEZ-HaaS/haas-165$ ssh ubuntu@opsmgr.haas-165.mydns.com
Unauthorized use is strictly prohibited. All access and activity
is subject to logging and monitoring.
ubuntu@opsmgr.haas-165.mydns.com's password:
Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 4.4.0-124-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

...

ubuntu@bosh-stemcell:~$

5. Verify CFDOT CLI is installed using "cfdot"

diego_cell/7ca12f7d-737f-47fb-a8bc-91d73e4791cf:~# cfdot
A command-line tool to interact with a Cloud Foundry Diego deployment

Usage:
  cfdot [command]

Available Commands:
  actual-lrp-groups            List actual LRP groups
  actual-lrp-groups-for-guid   List actual LRP groups for a process guid
  cancel-task                  Cancel task
  cell                         Show the specified cell presence
  cell-state                   Show the specified cell state
  cell-states                  Show cell states for all cells
  cells                        List registered cell presences
  claim-lock                   Claim Locket lock
  claim-presence               Claim Locket presence
  create-desired-lrp           Create a desired LRP
  create-task                  Create a Task
  delete-desired-lrp           Delete a desired LRP
  delete-task                  Delete a Task
  desired-lrp                  Show the specified desired LRP
  desired-lrp-scheduling-infos List desired LRP scheduling infos
  desired-lrps                 List desired LRPs
  domains                      List domains
  help                         Get help on [command]
  locks                        List Locket locks
  lrp-events                   Subscribe to BBS LRP events
  presences                    List Locket presences
  release-lock                 Release Locket lock
  retire-actual-lrp            Retire actual LRP by index and process guid
  set-domain                   Set domain
  task                         Display task
  task-events                  Subscribe to BBS Task events
  tasks                        List tasks in BBS
  update-desired-lrp           Update a desired LRP

Flags:
  -h, --help   help for cfdot

Use "cfdot [command] --help" for more information about a command.

6. Lets see what each Diego CELL has for Capacity as a whole

diego_cell/7ca12f7d-737f-47fb-a8bc-91d73e4791cf:~# cfdot cells | jq -r
{
  "cell_id": "7ca12f7d-737f-47fb-a8bc-91d73e4791cf",
  "rep_address": "http://10.193.229.62:1800",
  "zone": "RP01",
  "capacity": {
    "memory_mb": 16047,
    "disk_mb": 103549,
    "containers": 249
  },
  "rootfs_provider_list": [
    {
      "name": "preloaded",
      "properties": [
        "cflinuxfs2"
      ]
    },
    {
      "name": "preloaded+layer",
      "properties": [
        "cflinuxfs2"
      ]
    },
    {
      "name": "docker"
    }
  ],
  "rep_url": "https://7ca12f7d-737f-47fb-a8bc-91d73e4791cf.cell.service.cf.internal:1801"
}
{
  "cell_id": "9452a3b4-d40c-49f1-9dbf-8d74202f7dff",
  "rep_address": "http://10.193.229.61:1800",
  "zone": "RP01",
  "capacity": {
    "memory_mb": 16047,
    "disk_mb": 103549,
    "containers": 249
  },
  "rootfs_provider_list": [
    {
      "name": "preloaded",
      "properties": [
        "cflinuxfs2"
      ]
    },
    {
      "name": "preloaded+layer",
      "properties": [
        "cflinuxfs2"
      ]
    },
    {
      "name": "docker"
    }
  ],
  "rep_url": "https://9452a3b4-d40c-49f1-9dbf-8d74202f7dff.cell.service.cf.internal:1801"
}
{
  "cell_id": "dfc8e214-2e59-4050-9312-1113662ce79f",
  "rep_address": "http://10.193.229.63:1800",
  "zone": "RP01",
  "capacity": {
    "memory_mb": 16047,
    "disk_mb": 103549,
    "containers": 249
  },
  "rootfs_provider_list": [
    {
      "name": "preloaded",
      "properties": [
        "cflinuxfs2"
      ]
    },
    {
      "name": "preloaded+layer",
      "properties": [
        "cflinuxfs2"
      ]
    },
    {
      "name": "docker"
    }
  ],
  "rep_url": "https://dfc8e214-2e59-4050-9312-1113662ce79f.cell.service.cf.internal:1801"
}

More Information

https://github.com/cloudfoundry/cfdot

I have been "cf pushing" for as long as I can remember so with Pivotal Container Service (PKS) let's walk through the process of deploying a basic Spring Boot Application with a PKS cluster running on GCP.

Few assumptions:

1. PKS is already installed as shown by my Operations Manager UI below



2. A PKS Cluster already exists as shown by the command below

pasapicella@pas-macbook:~$ pks list-clusters

Name        Plan Name  UUID                                  Status     Action
my-cluster  small      1230fafb-b5a5-4f9f-9327-55f0b8254906  succeeded  CREATE

Example:

We will be using this Spring Boot application at the following GitHub URL

  https://github.com/papicella/springboot-actuator-2-demo


1. In this example my Spring Boot application has what is required within my maven build.xml file to allow me to create a Docker image as shown below

  
<!-- tag::plugin[] -->
   <plugin>
    <groupId>com.spotify</groupId>
    <artifactId>dockerfile-maven-plugin</artifactId>
    <version>1.3.6</version>
    <configuration>
     <repository>${docker.image.prefix}/${project.artifactId}</repository>
     <buildArgs>
      <JAR_FILE>target/${project.build.finalName}.jar</JAR_FILE>
     </buildArgs>
    </configuration>
   </plugin>
   <!-- end::plugin[] -->

   <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-dependency-plugin</artifactId>
    <executions>
     <execution>
      <id>unpack</id>
      <phase>package</phase>
      <goals>
       <goal>unpack</goal>
      </goals>
      <configuration>
       <artifactItems>
        <artifactItem>
         <groupId>${project.groupId}</groupId>
         <artifactId>${project.artifactId}</artifactId>
         <version>${project.version}</version>
        </artifactItem>
       </artifactItems>
      </configuration>
     </execution>
    </executions>
</plugin>

2. Once a docker image was built I then pushed that to Docker Hub as shown below



3. Now we will need a PKS cluster as shown below before we can continue

pasapicella@pas-macbook:~$ pks cluster my-cluster

Name:                     my-cluster
Plan Name:                small
UUID:                     1230fafb-b5a5-4f9f-9327-55f0b8254906
Last Action:              CREATE
Last Action State:        succeeded
Last Action Description:  Instance provisioning completed
Kubernetes Master Host:   cluster1.pks.pas-apples.online
Kubernetes Master Port:   8443
Worker Instances:         3
Kubernetes Master IP(s):  192.168.20.10

4. Now we want to wire "kubectl" using a command as follows

pasapicella@pas-macbook:~$ pks get-credentials my-cluster

Fetching credentials for cluster my-cluster.
Context set for cluster my-cluster.

You can now switch between clusters by using:
$kubectl config use-context

pasapicella@pas-macbook:~$ kubectl cluster-info
Kubernetes master is running at https://cluster1.pks.pas-apples.online:8443
Heapster is running at https://cluster1.pks.pas-apples.online:8443/api/v1/namespaces/kube-system/services/heapster/proxy
KubeDNS is running at https://cluster1.pks.pas-apples.online:8443/api/v1/namespaces/kube-system/services/kube-dns/proxy
monitoring-influxdb is running at https://cluster1.pks.pas-apples.online:8443/api/v1/namespaces/kube-system/services/monitoring-influxdb/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

5. Now we are ready to deploy a Spring Boot workload to our cluster. To do that lets download the YAML file below

https://github.com/papicella/springboot-actuator-2-demo/blob/master/lb-withspringboot.yml

Once downloaded create a deployment as follows

$ kubectl create -f lb-withspringboot.yml

pasapicella@pas-macbook:~$ kubectl create -f lb-withspringboot.yml
service "spring-boot-service" created
deployment "spring-boot-deployment" created

6. Now let’s verify our deployment using some kubectl commands as follows

$ kubectl get deployment spring-boot-deployment
$ kubectl get pods
$ kubectl get svc



RESTful End Point

pasapicella@pas-macbook:~$ http http://35.197.187.43:8080/employees/1
HTTP/1.1 200
Content-Type: application/hal+json;charset=UTF-8
Date: Wed, 09 May 2018 05:26:19 GMT
Transfer-Encoding: chunked

{
    "_links": {
        "employee": {
            "href": "http://35.197.187.43:8080/employees/1"
        },
        "self": {
            "href": "http://35.197.187.43:8080/employees/1"
        }
    },
    "name": "pas"
}

More Information

Using PKS
https://docs.pivotal.io/runtimes/pks/1-0/using.html

Recently at a customer site I was asked to show how the Autoscale service shipped by default with Pivotal Cloud Foundry would work. Here is how we demoed that in less then 5 minutes.

1. Select an application to Autoscale and click on the "Autoscaling" radio option.


2. Select "Manage Autoscaling" link as shown below.


3. Set the maximum instance limit to "4" and click Save as shown below. You can also set minimum to 1 instance if you want to which will make it easier to verify the scaling of instances as one instance can easily be put under pressure.


4. Now lets set a "Scaling Rule" by clicking on the "Edit" link as shown below.


5. Now lets add a CPU rule by clicking on the "Add" link as shown below.


6. Now define a CPU rule as shown below and click on Save. Don't forget to make it active using the radio option. In this example we use very low threshold BUT it would be better to increase this to something more realistic like 30% and 60% respectively.



Now at this point we are ready to test the Autoscale service BUT to do that we are going to have to create some load. Many different ways to do that but "ab" on my Mac was the fastest way.

8. Create some load on an endpoint for your application to force CPU utilization to increase as shown below

pasapicella@pas-macbook:~$ ab -n 10000 -c 25 http://springboot-actuator-appsmanager-delightful-jaguar.cfapps.io/employees
This is ApacheBench, Version 2.3 <$Revision: 1807734 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking springboot-actuator-appsmanager-delightful-jaguar.cfapps.io (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests

....

9. If you return to Apps Manager UI soon enough you will see that the Autoscale service has fired events to add more instances as per the screen shots below.




It's worth noting that the CF CLI Plugin for Autoscale can also show us what we have defined as as shown below. More information on this plugin is as follows

https://docs.run.pivotal.io/appsman-services/autoscaler/using-autoscaler-cli.html#install

View which applications are using the Autoscaler service:

pasapicella@pas-macbook:~$ cf autoscaling-apps
Presenting autoscaler apps in org apples-pivotal-org / space development as papicella@pivotal.io
OK
Name                              Guid                                   Enabled   Min Instances   Max Instances
springboot-actuator-appsmanager   6c137fea-6a99-4069-8031-a2aa3978804c   true      2               4

View events for an application that has Autoscaler service bound to it:

pasapicella@pas-macbook:~$ cf autoscaling-events springboot-actuator-appsmanager
Presenting autoscaler events for app springboot-actuator-appsmanager for org apples-pivotal-org / space development as papicella@pivotal.io
OK
Time                   Description
2018-04-20T09:56:30Z   Scaled down from 3 to 2 instances. All metrics are currently below minimum thresholds.
2018-04-20T09:55:56Z   Scaled down from 4 to 3 instances. All metrics are currently below minimum thresholds.
2018-04-20T09:54:46Z   Can not scale up. At max limit of 4 instances. Current CPU of 20.75% is above upper threshold of 8.00%.
2018-04-20T09:54:11Z   Can not scale up. At max limit of 4 instances. Current CPU of 30.53% is above upper threshold of 8.00%.
2018-04-20T09:53:36Z   Can not scale up. At max limit of 4 instances. Current CPU of 32.14% is above upper threshold of 8.00%.
2018-04-20T09:53:02Z   Can not scale up. At max limit of 4 instances. Current CPU of 31.51% is above upper threshold of 8.00%.
2018-04-20T09:52:27Z   Scaled up from 3 to 4 instances. Current CPU of 19.59% is above upper threshold of 8.00%.
2018-04-20T09:51:51Z   Scaled up from 2 to 3 instances. Current CPU of 8.99% is above upper threshold of 8.00%.
2018-04-20T09:13:24Z   Scaling from 1 to 2 instances: app below minimum instance limit
2018-04-20T09:13:23Z   Enabled autoscaling.

More Information

https://docs.run.pivotal.io/appsman-services/autoscaler/using-autoscaler-cli.html#install

https://docs.run.pivotal.io/appsman-services/autoscaler/using-autoscaler.html

The Spring Cloud Services plugin for the Cloud Foundry Command Line Interface tool (cf CLI) adds commands for interacting with Spring Cloud Services service instances. It provides easy access to functionality relating to the Config Server and Service Registry; for example, it can be used to send values to a Config Server service instance for encryption or to list all applications registered with a Service Registry service instance.

Here is a simple example of how we can view various bound apps for a Service Registry

1. Install the CF CLI Plugin for Spring Cloud Services using the link below

$ cf add-plugin-repo CF-Community https://plugins.cloudfoundry.org

More Information

http://docs.pivotal.io/spring-cloud-services/1-5/common/cf-cli-plugin.html

If you followed along on the previous blogs you would of installed PKS 1.0 on GCP (Google Cloud Platform) and created your first PKS cluster and wired it into kubectl as well as provided an external load balancer as per the previous two posts.

Previous posts:

Install Pivotal Container Service (PKS) on GCP and getting started
http://theblasfrompas.blogspot.com.au/2018/04/install-pivotal-container-service-pks.html

Wiring kubectl / Setup external LB on GCP into Pivotal Container Service (PKS) clusters to get started
http://theblasfrompas.blogspot.com.au/2018/04/wiring-kubectl-setup-external-lb-on-gcp.html

So lets now create our first workload as shown below

1. Download YML demo from here

https://github.com/cloudfoundry-incubator/kubo-ci/blob/master/specs/nginx-lb.yml

2. Deploy as shown below

pasapicella@pas-macbook:~/pivotal/GCP/install/21/PKS/demo-workload$ kubectl create -f nginx-lb.yml
service "nginx" created
deployment "nginx" created

3. Check current status

pasapicella@pas-macbook:~/pivotal/GCP/install/21/PKS/demo-workload$ kubectl get pods
NAME                     READY     STATUS    RESTARTS   AGE
nginx-679dc9c764-8cwzq   1/1       Running   0          22s
nginx-679dc9c764-p8tf2   1/1       Running   0          22s
nginx-679dc9c764-s79mp   1/1       Running   0          22s

4. Wait for External IP address of the nginx service to be assigned

pasapicella@pas-macbook:~/pivotal/GCP/install/21/PKS/demo-workload$ kubectl get svc
NAME         TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)        AGE
kubernetes   ClusterIP      10.100.200.1               443/TCP        17h
nginx        LoadBalancer   10.100.200.143   35.189.23.119   80:30481/TCP   1m

5. In a browser access the K8's workload as follows, using the external IP

http://35.189.23.119



More Info

https://docs.pivotal.io/runtimes/pks/1-0/index.html

Now that I have PCF 2.1 running with PKS 1.0 installed and a cluster up and running how would I get started accessing that cluster? Here are the steps for GCP (Google Cloud Platform) install of PCF 2.1 with PKS 1.0. It goes through the requirements around an External LB for the cluster as well as wiring kubectl into the cluster to get started creating deployments.

Previous blog as follows:

http://theblasfrompas.blogspot.com.au/2018/04/install-pivotal-container-service-pks.html

1. First we will want an external Load Balancer for our K8's clusters which will need to exist and it would be a TCP Load balancer using Port 8443 which is the port the master node would run on. The external IP address is what you will need to use in the next step



2. Create a Firewall Rule for the LB with details as follows.

Note: the LB name is "pks-cluster-api-1". Make sure to include the network tag and select the network you installed PKS on.

• Network: Make sure to select the right network. Choose the value that matches with the VPC Network name you installed PKS on
• Ingress - Allow
• Target: pks-cluster-api-1
• Source: 0.0.0.0/0
• Ports: tcp:8443

3. Now you could easily just create a cluster using the external IP address from above or use a DNS entry which is mapped to the external IP address which is what I have done so I have use a FQDN instead

pasapicella@pas-macbook:~$ pks create-cluster my-cluster --external-hostname cluster1.pks.pas-apples.online --plan small

Name:                     my-cluster
Plan Name:                small
UUID:                     64a086ce-c94f-4c51-95f8-5a5edb3d1476
Last Action:              CREATE
Last Action State:        in progress
Last Action Description:  Creating cluster
Kubernetes Master Host:   cluster1.pks.pas-apples.online
Kubernetes Master Port:   8443
Worker Instances:         3
Kubernetes Master IP(s):  In Progress


4. Now just wait a while while it creates a VM's and runs some tests , it's roughly around 10 minutes. Once done you will see the cluster as created as follows

pasapicella@pas-macbook:~/pivotal/GCP/install/21/PKS$ pks list-clusters

Name        Plan Name  UUID                                  Status     Action
my-cluster  small      64a086ce-c94f-4c51-95f8-5a5edb3d1476  succeeded  CREATE

5. Now one of the VM's created would be the master Vm for the cluster , their a few ways to determine the master VM as shown below.

5.1. Use GCP Console VM instances page and filter by "master"



5.2. Run a bosh command to view the VM's of your deployments. We are interested in the VM's for our cluster service. The master instance is named as "master/ID" as shown below.

$ bosh -e gcp vms --column=Instance --column "Process State" --column "VM CID"

Task 187. Done

Deployment 'service-instance_64a086ce-c94f-4c51-95f8-5a5edb3d1476'

Instance                                     Process State  VM CID
master/13b42afb-bd7c-4141-95e4-68e8579b015e  running        vm-4cfe9d2e-b26c-495c-4a62-77753ce792ca
worker/490a184e-575b-43ab-b8d0-169de6d708ad  running        vm-70cd3928-317c-400f-45ab-caf6fa8bd3a4
worker/79a51a29-2cef-47f1-a6e1-25580fcc58e5  running        vm-e3aa47d8-bb64-4feb-4823-067d7a4d4f2c
worker/f1f093e2-88bd-48ae-8ffe-b06944ea0a9b  running        vm-e14dde3f-b6fa-4dca-7f82-561da9c03d33

4 vms

6. Attach the VM to the load balancer backend configuration as shown below.



7. Now we can get the credentials from PKS CLI and pass them to kubectl as shown below

pasapicella@pas-macbook:~/pivotal/GCP/install/21/PKS$ pks get-credentials my-cluster

Fetching credentials for cluster my-cluster.
Context set for cluster my-cluster.

You can now switch between clusters by using:
$kubectl config use-context

pasapicella@pas-macbook:~/pivotal/GCP/install/21/PKS$ kubectl cluster-info
Kubernetes master is running at https://cluster1.pks.domain-name:8443
Heapster is running at https://cluster1.pks.domain-name:8443/api/v1/namespaces/kube-system/services/heapster/proxy
KubeDNS is running at https://cluster1.pks.domain-name:8443/api/v1/namespaces/kube-system/services/kube-dns/proxy
monitoring-influxdb is running at https://cluster1.pks.domain-name:8443/api/v1/namespaces/kube-system/services/monitoring-influxdb/proxy
8. To verify it worked for you here are some commands you would run. The "kubectl cluster-info" is one of those.

pasapicella@pas-macbook:~/pivotal/GCP/install/21/PKS$ kubectl get componentstatus
NAME                 STATUS    MESSAGE              ERROR
controller-manager   Healthy   ok
scheduler            Healthy   ok
etcd-0               Healthy   {"health": "true"}

pasapicella@pas-macbook:~/pivotal/GCP/install/21/PKS$ kubectl get pods
No resources found.

pasapicella@pas-macbook:~/pivotal/GCP/install/21/PKS$ kubectl get deployments
No resources found.

9. Finally lets start the Kubernetes UI to monitor this cluster. We do that as easily as this.

pasapicella@pas-macbook:~/pivotal/GCP/install/21/PKS$ kubectl proxy
Starting to serve on 127.0.0.1:8001  

The UI URL requires you to append /ui to the url above

Eg: http://127.0.0.1:8001/ui

Note: It will prompt you for the kubectl config file which would be in the $HOME/.kube/config file. Failure to present this means the UI won't show you much and give lost of warnings




More Info

https://docs.pivotal.io/runtimes/pks/1-0/index.html

With the release of Pivotal Cloud Foundry 2.1 (PCF) I decided this time to install Pivotal Application Service (PAS) as well as Pivotal Container Service (PKS) using the one Bosh Director which isn't recommended for production installs BUT ok for dev installs. Once installed you will have both the PAS tile and PKS tile as shown below.

https://content.pivotal.io/blog/pivotal-cloud-foundry-2-1-adds-cloud-native-net-envoy-native-service-discovery-to-boost-your-transformation


So here is how to get started with PKS once it's installed

1. Create a user for the PKS client to login with.

1.1. ssh into the ops manager VM

1.2. Target the UAA endpoint for PKS this was setup in the PKS tile

ubuntu@opsman-pcf:~$ uaac target https://PKS-ENDPOINT:8443 --skip-ssl-validation
Unknown key: Max-Age = 86400

Target: https://PKS-ENDPOINT:8443
More Info

https://docs.pivotal.io/runtimes/pks/1-0/index.html

I decided to eventually install PCF Healthwatch on my Google Cloud Platform PCF 2.0 instance. Installing it is straight forward using Ops Manager UI and once installed it will look like this.

Note: This is PCF 2.0 on GCP


Once installed the application for the Web UI end point would be as follows. The login username and password is the UAA admin user . By default the property "healthwatch.read" credential is given to this user only. You can always create a new user that has this credential role if you like.

https://healthwatch.SYSTEM-DOMAIN

The main page has various useful information and more then enough to show you what's happening in your PCF instance as shown below.



Clicking on any of the headings for each tile you can get more detailed information. The two screen shots below show some CF CLI command history tests like a "cf push" , "cf logs" and also what is happening within the Diego cells in terms of "Memory, Disk and the Containers" themselves.




More Information

https://docs.pivotal.io/pcf-healthwatch/1-1/index.html

CF Dev is a new distribution of Cloud Foundry designed to run on a developer’s laptop or workstation using native hypervisors and a fully functional BOSH Director.

I decided to give it a test run today and it's fast and easy full CF experience deployed through the CF CLI plugin as described in the GitHub project

  https://github.com/pivotal-cf/cfdev

Here we run some bosh commands once it's up and running. You can't run BOSH commands without first setting your ENV to use the correct bosh director which you do as follows

  $ eval "$(cf dev bosh env)"

pasapicella@pas-macbook:~/apps/ENV/cfdev$ bosh deployments
Using environment '10.245.0.2' as client 'admin'

Name  Release(s)                    Stemcell(s)                                          Team(s)  Cloud Config
cf    binary-buildpack/1.0.15       bosh-warden-boshlite-ubuntu-trusty-go_agent/3468.17  -        latest
      bosh-dns/0.2.0
      capi/1.46.0
      cf-mysql/36.10.0
      cf-networking/1.9.0
      cf-smoke-tests/40
      cf-syslog-drain/5
      cflinuxfs2/1.179.0
      consul/191
      diego/1.32.1
      dotnet-core-buildpack/1.0.32
      garden-runc/1.10.0
      go-buildpack/1.8.15
      grootfs/0.30.0
      java-buildpack/4.7.1
      loggregator/99
      nats/22
      nodejs-buildpack/1.6.13
      php-buildpack/4.3.46
      python-buildpack/1.6.4
      routing/0.169.0
      ruby-buildpack/1.7.8
      staticfile-buildpack/1.4.20
      statsd-injector/1.0.30
      uaa/53.3

1 deployments

Succeeded

pasapicella@pas-macbook:~/apps/ENV/cfdev$ bosh stemcells
Using environment '10.245.0.2' as client 'admin'

Name                                         Version   OS             CPI  CID
bosh-warden-boshlite-ubuntu-trusty-go_agent  3468.17*  ubuntu-trusty  -    54a8d4c1-5a02-4d89-5648-1132914a0cb8

(*) Currently deployed

1 stemcells

Succeeded

You can simply use the CF CLI as follows once you target the correct API endpoint and login as follows

pasapicella@pas-macbook:~/apps/ENV/cfdev$ cf api https://api.v3.pcfdev.io --skip-ssl-validation
Setting api endpoint to https://api.v3.pcfdev.io...
OK

api endpoint:   https://api.v3.pcfdev.io
api version:    2.100.0
Not logged in. Use 'cf login' to log in.

and to log in ...

pasapicella@pas-macbook:~/apps/ENV/cfdev$ cf login -o cfdev-org -u admin -p admin
API endpoint: https://api.v3.pcfdev.io
Authenticating...
OK

Targeted org cfdev-org

Targeted space cfdev-space

API endpoint:   https://api.v3.pcfdev.io (API version: 2.100.0)
User:           admin
Org:            cfdev-org
Space:          cfdev-space

With Spring Boot Actuator you get production-ready features to your application. The main benefit of this library is that we can get production grade tools without having to actually implement these features ourselves.

Developers who want to obtain debug data for a specific instance of an app can use the HTTP header X-CF-APP-INSTANCE to make a request to an app instance. To demonstrate how we can write a Spring Boot application which simply outputs the current CF app index so we are sure we are hitting the right application container.

Simplest way to do that is to define a RestController using Spring Boot as follows which then enables us to get the current application index and verify we are hitting the right container instance.

  
package com.example.pas;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class DemoRest
{
    private final String ip;
    private final String index;

    @Autowired
    public DemoRest
            (@Value("${CF_INSTANCE_IP:127.0.0.1}") String ip, 
             @Value("${CF_INSTANCE_INDEX:0}") String index) {
        this.ip = ip;
        this.index = index;
    }

    @RequestMapping("/")
    public InstanceDetail getAppDetails()
    {
        InstanceDetail instanceDetail = new InstanceDetail(ip, index);

        return instanceDetail;
    }
}

So with the application deployed as see we have 3 instances as follows

pasapicella@pas-macbook:~$ cf app pas-pcf-routingdemo
Showing health and status for app pas-pcf-routingdemo in org pivot-papicella / space dotnet as papicella@pivotal.io...

name:                pas-pcf-routingdemo
requested state:     started
instances:           3/3
isolation segment:   main
usage:               756M x 3 instances
routes:              pas-pcf-routingdemo-incidental-curia.pcfbeta.io
last uploaded:       Thu 18 Jan 20:41:26 AEDT 2018
stack:               cflinuxfs2
buildpack:           client-certificate-mapper=1.4.0_RELEASE container-security-provider=1.11.0_RELEASE java-buildpack=v4.7.1-offline-https://github.com/cloudfoundry/java-buildpack.git#6a3361a
                     java-main java-opts java-security jvmkill-agent=1... (no decorators apply)

     state     since                  cpu    memory           disk           details
#0   running   2018-01-18T09:44:07Z   0.4%   224.8M of 756M   137.5M of 1G
#1   running   2018-01-18T09:44:13Z   0.8%   205M of 756M     137.5M of 1G
#2   running   2018-01-18T09:44:06Z   0.7%   221.1M of 756M   137.5M of 1G

Now lets simply access our application a few times using the "/" end point and verify we are accessing different application containers via round robin routing as per GoRouter

pasapicella@pas-macbook:~$ http https://pas-pcf-routingdemo-incidental-curia.pcfbeta.io/
HTTP/1.1 200 OK
Content-Length: 34
Content-Type: application/json;charset=UTF-8
Date: Thu, 18 Jan 2018 09:58:10 GMT
Set-Cookie: dtCookie=6$B570EBB532CD9D8DAA2BCAE14C4277FC|RUM+Default+Application|1; Domain=pcfbeta.io; Path=/
X-Vcap-Request-Id: 336ba633-685b-4235-467d-b9833a9e6435

{
    "index": "2",
    "ip": "192.168.16.34"
}

pasapicella@pas-macbook:~$ http https://pas-pcf-routingdemo-incidental-curia.pcfbeta.io/
HTTP/1.1 200 OK
Content-Length: 34
Content-Type: application/json;charset=UTF-8
Date: Thu, 18 Jan 2018 09:58:15 GMT
Set-Cookie: dtCookie=5$3389B3DFBAD936D68CBAF30657653465|RUM+Default+Application|1; Domain=pcfbeta.io; Path=/
X-Vcap-Request-Id: aa74e093-9031-4df5-73a5-bc9f1741a942

{
    "index": "1",
    "ip": "192.168.16.32"
}

Now we can request access to just the container with application index "1" as follows

1. First get the Application GUID as shown below

pasapicella@pas-macbook:~$ cf app pas-pcf-routingdemo --guid
5bdf2f08-34a5-402f-b7cb-f29c81d171e0

2. Now lets invoke a call to the application and set the HEADER required to instruct GoRouter to target a specific application index

eg: curl app.example.com -H "X-CF-APP-INSTANCE":"YOUR-APP-GUID:YOUR-INSTANCE-INDEX"

Example below is using HTTPie 

Accessing Instance 1

pasapicella@pas-macbook:~$ http https://pas-pcf-routingdemo-incidental-curia.pcfbeta.io/ "X-CF-APP-INSTANCE":"5bdf2f08-34a5-402f-b7cb-f29c81d171e0:1"
HTTP/1.1 200 OK
Content-Length: 34
Content-Type: application/json;charset=UTF-8
Date: Thu, 18 Jan 2018 10:20:31 GMT
Set-Cookie: dtCookie=5$FD08A5C88469AF379C8AD3F36FA7984B|RUM+Default+Application|1; Domain=pcfbeta.io; Path=/
X-Vcap-Request-Id: cb19b960-713a-49d0-4529-a0766a8880a7

{
    "index": "1",
    "ip": "192.168.16.32"
}

Accessing Instance 2 

pasapicella@pas-macbook:~$ http https://pas-pcf-routingdemo-incidental-curia.pcfbeta.io/ "X-CF-APP-INSTANCE":"5bdf2f08-34a5-402f-b7cb-f29c81d171e0:2"
HTTP/1.1 200 OK
Content-Length: 34
Content-Type: application/json;charset=UTF-8
Date: Thu, 18 Jan 2018 10:21:09 GMT
Set-Cookie: dtCookie=7$53957A744D473BB024EB1FF4F0CD60A9|RUM+Default+Application|1; Domain=pcfbeta.io; Path=/
X-Vcap-Request-Id: 33cc7922-9b43-4182-5c36-13eee42a9919

{
    "index": "2",
    "ip": "192.168.16.34"
}

More Information

https://docs.cloudfoundry.org/concepts/http-routing.html#http-headers

After installing PCF 2.0 here is how you can verify your installation using the new bosh2 CLI. In this example I use "bosh2" BUT with PCF 2.0 you can actually use "bosh". bosh2 v2 existed for a while in PCF 1.12 and some previous versions while we left bosh v1

1. SSH into your ops manager VM as shown below, in this example we using GCP

https://docs.pivotal.io/pivotalcf/2-0/customizing/trouble-advanced.html#ssh

2. Create an alias for your ENV as shown below

Note: You will need the bosh director IP address which you can obtain using

  https://docs.pivotal.io/pivotalcf/2-0/customizing/trouble-advanced.html#gather

ubuntu@opsman-pcf:~$ bosh2 alias-env gcp -e y.y.y.y --ca-cert /var/tempest/workspaces/default/root_ca_certificate
Using environment 'y.y.y.y' as anonymous user

Name      p-bosh
UUID      3c886290-144f-4ec7-86dd-b7586b98dc3b
Version   264.4.0 (00000000)
CPI       google_cpi
Features  compiled_package_cache: disabled
          config_server: enabled
          dns: disabled
          snapshots: disabled
User      (not logged in)

Succeeded

3. Log in to the BOSH Director with UAA

Note: You will need the username / password for the bosh director which you can obtain as follows

  https://docs.pivotal.io/pivotalcf/2-0/customizing/trouble-advanced.html#gather

ubuntu@opsman-pcf:~$ bosh2 -e gcp log-in
Email (): director
Password ():

Successfully authenticated with UAA

Succeeded

4. View all the VM's managed by BOSH as follows

ubuntu@opsman-pcf:~/scripts$ bosh2 -e gcp vms --column=Instance --column="Process State" --column=AZ --column="VM Type"
Using environment 'y.y.y.y' as user 'director' (bosh.*.read, openid, bosh.*.admin, bosh.read, bosh.admin)

Task 65. Done

Deployment 'cf-adee3657c74c7b9a8e35'

Instance                                             Process State  AZ                      VM Type
backup-prepare/996340c7-4114-472e-b660-a5e353493fa4  running        australia-southeast1-a  micro
blobstore/cdd6fc8d-25c9-4cfb-9908-89eb0164fb80       running        australia-southeast1-a  medium.mem
compute/2dfcc046-c16a-4a36-9170-ef70d1881818         running        australia-southeast1-a  xlarge.disk
control/2f3d0bc6-9a2d-4c08-9ccc-a88bad6382a3         running        australia-southeast1-a  xlarge
database/da60f0e7-b8e3-4f8d-945d-306b267ac161        running        australia-southeast1-a  large.disk
mysql_monitor/a88331c4-1659-4fe4-b8e9-89ce4bf092fd   running        australia-southeast1-a  micro
router/276e308e-a476-4c8d-9555-21623dada492          running        australia-southeast1-a  micro

7 vms

Succeeded

** Few other examples **

- View all the deployments, in this example we just have the PAS small footprint tile installed so it only exists and no other bosh managed tiles xist

ubuntu@opsman-pcf:~/scripts$ bosh2 -e gcp deployments --column=name
Using environment 'y.y.y.y' as user 'director' (bosh.*.read, openid, bosh.*.admin, bosh.read, bosh.admin)

Name
cf-adee3657c74c7b9a8e35

1 deployments

Succeeded

- Run cloud check to check for issues

ubuntu@opsman-pcf:~/scripts$ bosh2 -e gcp -d cf-adee3657c74c7b9a8e35 cloud-check
Using environment 'y.y.y.y' as user 'director' (bosh.*.read, openid, bosh.*.admin, bosh.read, bosh.admin)

Using deployment 'cf-adee3657c74c7b9a8e35'

Task 66

Task 66 | 04:20:52 | Scanning 7 VMs: Checking VM states (00:00:06)
Task 66 | 04:20:58 | Scanning 7 VMs: 7 OK, 0 unresponsive, 0 missing, 0 unbound (00:00:00)
Task 66 | 04:20:58 | Scanning 3 persistent disks: Looking for inactive disks (00:00:01)
Task 66 | 04:20:59 | Scanning 3 persistent disks: 3 OK, 0 missing, 0 inactive, 0 mount-info mismatch (00:00:00)

Task 66 Started  Fri Dec 29 04:20:52 UTC 2017
Task 66 Finished Fri Dec 29 04:20:59 UTC 2017
Task 66 Duration 00:00:07
Task 66 done

#  Type  Description

0 problems

Succeeded

More Information

https://docs.pivotal.io/pivotalcf/2-0/customizing/trouble-advanced.html

I was recently asked how to terminate a specific application instance rather then terminate all instances using "cf delete".

We can easily using the CF REST API or even easier the CF CLI "cf curl" command which makes it straight forward to make REST based calls into cloud foundry as shown below.

CF REST API Docs

https://apidocs.cloudfoundry.org/280/

Below assumes you already logged into PCF using the CF CLI

1. First find an application that has multiple instances

pasapicella@pas-macbook:~$ cf app pas-cf-manifest
Showing health and status for app pas-cf-manifest in org apples-pivotal-org / space development as papicella@pivotal.io...

name:              pas-cf-manifest
requested state:   started
instances:         2/2
usage:             756M x 2 instances
routes:            pas-cf-manifest.cfapps.io
last uploaded:     Sun 19 Nov 21:26:26 AEDT 2017
stack:             cflinuxfs2
buildpack:         client-certificate-mapper=1.2.0_RELEASE container-security-provider=1.8.0_RELEASE java-buildpack=v4.5-offline-https://github.com/cloudfoundry/java-buildpack.git#ffeefb9 java-main
                   java-opts jvmkill-agent=1.10.0_RELEASE open-jdk-like-jre=1.8.0_1...

     state     since                  cpu    memory           disk           details
#0   running   2017-12-16T00:11:27Z   0.0%   241.5M of 756M   139.9M of 1G
#1   running   2017-12-17T10:39:09Z   0.3%   221.3M of 756M   139.9M of 1G

2. Use a "cf curl" curl which uses the application GUID to determine which application to check all application instances and their current state

pasapicella@pas-macbook:~$ cf curl /v2/apps/`cf app pas-cf-manifest --guid`/instances
{
   "0": {
      "state": "RUNNING",
      "uptime": 293653,
      "since": 1513383087
   },
   "1": {
      "state": "RUNNING",
      "uptime": 169591,
      "since": 1513507149
   }
}

3. Now let's delete instance with index "1". Don't forget that PCF will determine the current desired state of the application is not the current state and will re-start the application instance very quickly

pasapicella@pas-macbook:~$ cf curl /v2/apps/`cf app pas-cf-manifest --guid`/instances/1 -X DELETE

Pivotal Cloud Foundry (PCF) now has a small footprint edition. It features a deployment configuration with as few as 6 VMs. Review the documentation for download and installation instructions as follows

http://docs.pivotal.io/pivotalcf/1-12/customizing/small-footprint.html

There was also a Pivotal blog post on this as follows:

https://content.pivotal.io/blog/big-things-come-in-small-packages-getting-started-with-pivotal-cloud-foundry-small-footprint

As you can see from this image it's considerably smaller control plane that's obvious.



It is important to understand what the limitations of such an install are as per the docs link below.

http://docs.pivotal.io/pivotalcf/1-12/customizing/small-footprint.html#limits

Installing the small footprint looks identical from the Operations Manager UI in fact it's still labelled ERT and from the home page of Operations Manager UI your wouldn't even know you had the small footprint



If you dig a bit further and click on the "ERT tile" and then select "Resource Config" left hand link you will then clearly know it's the Small Footprint PCF install.


I choose to use internal MySQL database and if I didn't then it could be scaled back even more then the default 7 VM's I ended up with.

Lastly I was very curious to find out what jobs are placed on which service VM's. Here is what it looked like for me when I logged into bosh director and run some bosh CLI commands

  
ubuntu@ip-10-0-0-241:~$ bosh2 -e aws vms --column=Instance --column="Process State" --column=AZ --column="VM Type"
Using environment '10.0.16.5' as user 'director' (bosh.*.read, openid, bosh.*.admin, bosh.read, bosh.admin)

Task 73. Done

Deployment 'cf-a96683b17697c86b8c90'

Instance                                             Process State  AZ               VM Type
backup-prepare/16356c40-1f20-42f0-8f2e-de45549be797  running        ap-southeast-2a  t2.micro
blobstore/b6e22107-018b-425d-8fe4-ab47eeaf2c75       running        ap-southeast-2a  m4.large
compute/5439f18f-c842-40a2-b6f3-faf6b6848716         running        ap-southeast-2a  r4.xlarge
control/68979d93-d12b-4d87-b110-d3d41a48b261         running        ap-southeast-2a  r4.xlarge
database/a3efedaa-4df6-48f5-9f20-61cf3d9f3c1b        running        ap-southeast-2a  r4.large
mysql_monitor/d40ef638-d2d0-488e-b937-99a7f5b5b334   running        ap-southeast-2a  t2.micro
router/f9573547-c5a1-43d4-be02-38a1d9e9c73e          running        ap-southeast-2a  t2.micro

7 vms

Succeeded

  
ubuntu@ip-10-0-0-241:~$ bosh2 -e aws instances --ps --column=Instance --column=Process
Using environment '10.0.16.5' as user 'director' (bosh.*.read, openid, bosh.*.admin, bosh.read, bosh.admin)

Task 68. Done

Deployment 'cf-a96683b17697c86b8c90'

Instance                                                          Process
autoscaling-register-broker/9feaef45-994e-472c-8ca3-f0c39467dd6b  -
autoscaling/184df31d-a64c-49e0-8b6b-27eafdb31ca0                  -
backup-prepare/16356c40-1f20-42f0-8f2e-de45549be797               -
~                                                                 service-backup
blobstore/b6e22107-018b-425d-8fe4-ab47eeaf2c75                    -
~                                                                 blobstore_nginx
~                                                                 blobstore_url_signer
~                                                                 consul_agent
~                                                                 metron_agent
~                                                                 route_registrar
bootstrap/0dc22a1f-a1ee-4a03-85c6-fed08f37c44a                    -
compute/5439f18f-c842-40a2-b6f3-faf6b6848716                      -
~                                                                 consul_agent
~                                                                 garden
~                                                                 iptables-logger
~                                                                 metron_agent
~                                                                 netmon
~                                                                 nfsv3driver
~                                                                 rep
~                                                                 route_emitter
~                                                                 silk-daemon
~                                                                 vxlan-policy-agent
control/68979d93-d12b-4d87-b110-d3d41a48b261                      -
~                                                                 adapter
~                                                                 auctioneer
~                                                                 bbs
~                                                                 cc_uploader
~                                                                 cloud_controller_clock
~                                                                 cloud_controller_ng
~                                                                 cloud_controller_worker_1
~                                                                 cloud_controller_worker_local_1
~                                                                 cloud_controller_worker_local_2
~                                                                 consul_agent
~                                                                 doppler
~                                                                 file_server
~                                                                 locket
~                                                                 loggregator_trafficcontroller
~                                                                 metron_agent
~                                                                 nginx_cc
~                                                                 policy-server
~                                                                 reverse_log_proxy
~                                                                 route_registrar
~                                                                 routing-api
~                                                                 scheduler
~                                                                 silk-controller
~                                                                 ssh_proxy
~                                                                 statsd_injector
~                                                                 syslog_drain_binder
~                                                                 tps_watcher
~                                                                 uaa
database/a3efedaa-4df6-48f5-9f20-61cf3d9f3c1b                     -
~                                                                 cluster_health_logger
~                                                                 consul_agent
~                                                                 galera-healthcheck
~                                                                 gra-log-purger-executable
~                                                                 mariadb_ctrl
~                                                                 metron_agent
~                                                                 mysql-diag-agent
~                                                                 mysql-metrics
~                                                                 nats
~                                                                 route_registrar
~                                                                 streaming-mysql-backup-tool
~                                                                 switchboard
mysql-rejoin-unsafe/01a0aec3-b103-4c09-bc69-cabc61c513cc          -
mysql_monitor/d40ef638-d2d0-488e-b937-99a7f5b5b334                -
~                                                                 replication-canary
nfsbrokerpush/829f0292-59ab-4824-8b0b-c4af4bddbce0                -
notifications-ui/50972440-36cd-499d-ad7c-eef4df7e604b             -
notifications/968d625e-af63-4e2f-a59c-f6b789ef1cff                -
push-apps-manager/3d9760d7-6f09-453c-a052-32604b6a3235            -
push-pivotal-account/5a8782ad-82eb-4469-8242-f0873bc4a587         -
push-usage-service/9b781db1-171b-4abe-92f4-7445fd3d487f           -
router/f9573547-c5a1-43d4-be02-38a1d9e9c73e                       -
~                                                                 consul_agent
~                                                                 gorouter
~                                                                 metron_agent
smoke-tests/a8e2ff97-bae1-4594-90fc-ec8c430fd620                  -

77 instances

Succeeded

Now, with Small Footprint, you have yet another way to bring PCF to your organization!

An application health check is a monitoring process that continually checks the status of a running Cloud Foundry application. When deploying an app, a developer can configure the health check type (port, process, or HTTP), a timeout for starting the application, and an endpoint (for HTTP only) for the application health check.

To use the HTTP option your manifest.yml would look like this

---
applications:
- name: pas-cf-manifest
  memory: 756M
  instances: 1
  hostname: pas-cf-manifest
  path: ./target/demo-0.0.1-SNAPSHOT.jar
  health-check-type: http
  health-check-http-endpoint: /health
  stack: cflinuxfs2
  timeout: 80
  env:
    JAVA_OPTS: -Djava.security.egd=file:///dev/urandom
    NAME: Apples

Using a HTTP endpoint such as "/health" is possible once you add the Spring Boot Actuator maven dependency as follows

  
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

More Information

https://docs.run.pivotal.io/devguide/deploy-apps/healthchecks.html

This app below simply tests whether a host:port is accessible from a CF app instance. For example can my application instance access my Oracle Database Instance running outside of PCF given application instances need network access to the database database for example.

You can use bosh2 ssh to get to the Diego Cells if you have access to the environment or even "cf ssh" if that has been enabled.

GitHub URL:

https://github.com/papicella/cloudfoundry-socket-test

Success

pasapicella@pas-macbook:~$ http http://pas-cf-sockettest.cfapps.io/www.google.com.au/80
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 81
Content-Type: application/json;charset=UTF-8
Date: Wed, 25 Oct 2017 08:38:33 GMT
X-Vcap-Request-Id: 8fd05c77-f680-4966-558b-c45e71825fa0

{
    "errorMessage": "N/A",
    "hostname": "www.google.com.au",
    "port": "80",
    "res": "SUCCESS"
}

Failure

pasapicella@pas-macbook:~$ http http://pas-cf-sockettest.cfapps.io/10.0.0.10/8080
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 110
Content-Type: application/json;charset=UTF-8
Date: Wed, 25 Oct 2017 11:52:18 GMT
X-Vcap-Request-Id: 91ad2359-7d95-49c6-4538-548e480b7820

{
    "errorMessage": "Connection refused (Connection refused)",
    "hostname": "10.0.0.10",
    "port": "8080",
    "res": "FAILED"
}