Pas Apicella

Subscribe to Pas Apicella feed
Information on Snyk : Develop Fast. Stay Secure.Pas Apicellahttp://www.blogger.com/profile/09389663166398991762noreply@blogger.comBlogger460125
Updated: 7 hours 14 min ago

Snyk provides native integration for Atlassian Bitbucket Cloud security - Here is how!!!

Thu, 2021-07-22 23:56

The Snyk security integration is free and easy to set up with just a few clicks inside the Bitbucket Cloud product. For the first time, developers can consume information that was previously only available inside Snyk now within Bitbucket Cloud. Snyk enables developers to see new vulnerabilities as they emerge and implement fixes early and quickly in the process. 

In this post we show how easily you can integrate Snyk into Bitbucket Cloud

Steps
Note: For the following to work you have to integrate Bitbucket Cloud with the Snyk App as per this link 


1. Once logged into Bitbucket Cloud navigate to your "Workplace Settings" and authenticate with Snyk as shown below.





2. Next select the repository you wish to use as shown below.





3. You should see a "Snyk" tab on the left hand side click on this and then click on "Import this repository" as shown below.




4. In a couple of minutes or less you should see a "Snyk" overview report as per below. 




5. Click on "pom.xml" to get more information as shown below. Here we get a list of all Vulnerabilities as per a scan of the package manifest file "pom.xml" in this example.





6. On the top of this page you can directly go to the project page on "Snyk App" by clicking on "Visit Snyk" as shown below.




It's as simple as that!
 
More Information
Demo Video

Categories: Fusion Middleware

Snyk Import Project API using Azure Repos

Thu, 2021-07-01 20:43

In this example below we show how you could import a Azure Repo directly from the Snyk API. The Snyk API is available to customers on paid plans and allows you to programatically integrate with Snyk.

API url

The base URL for all API endpoints is https://snyk.io/api/v1/

Authorization

To use this API, you must get your token from Snyk. It can be seen on https://snyk.io/account/ after you register with Snyk and login.

The token should be supplied in an Authorization header with the token, preceded by token:

Authorization: token API_KEY

Steps

Note: You would have configured an Azure Repos Integration as shown below prior to running these steps




1. Authenticate with the CLI using your Snyk Token as follows

$ snyk auth TOKEN

Your account has been authenticated. Snyk is now ready to be used.

2. Log into your Azure DevOps account and verify which Project Repo you wish to import as per the image below


3. With your project selected you will need the following details to perform the Snyk API import request

owner: Name of your project


name: Name of your Repository


branch: Branch name you wish to import



4. Finally we will need our Organisation ID and Azure Repos ID which we retrieve from the Snyk UI as follows

organization ID

Setting > General -> Organization ID

Azure Repos ID

Setting > Integrations -> Azure Repos -> Edit Settings -> Scroll to bottom of page



Take of note of both ID's we will need them in the steps below

5. Now we are ready to import our Azure Repo using a command as follows. We are using a "curl" command to issue a POST request and this command is for a Mac OSX or linux system so it would need tweaking if running on Windows for example

Command

curl --include \
     --request POST \
     --header "Content-Type: application/json; charset=utf-8" \
     --header "Authorization: token `snyk config get api`" \
     --data-binary "{
  \"target\": {
    \"owner\": \"spring-book-service\",
    \"name\": \"spring-book-service\",
    \"branch\": \"master\"
  }
}" \
'https://snyk.io/api/v1/org/$ORG_ID/integrations/$AZURE_REPO_ID/import'

6. Set the ENV variables for ORG_ID and AZURE_REPO_ID as follows

export ORG_ID=FROM_STEP-4
export AZURE_REPO_ID=FROM_STEP-4

7. Run command

$ curl --include \
>      --request POST \
>      --header "Content-Type: application/json; charset=utf-8" \
>      --header "Authorization: token `snyk config get api`" \
>      --data-binary "{
>   \"target\": {
>     \"owner\": \"spring-book-service\",
>     \"name\": \"spring-book-service\",
>     \"branch\": \"master\"
>   }
> }" \
> "https://snyk.io/api/v1/org/${ORG_ID}/integrations/${AZURE_REPO_ID}/import"
HTTP/2 201
content-security-policy: base-uri 'none'; script-src 'self' https: 'nonce-ENlk6rSQsdLgbcWNcCaA7A==' 'unsafe-inline' 'unsafe-eval' 'strict-dynamic' 'report-sample'; img-src https: data:; object-src 'none'; report-to csp-report-group; report-uri https://web-reports.snyk.io/csp?version=39e8721a74a6dffcb97fc790cfbf1fca91cefc03;
report-to: {"group":"csp-report-group","max_age":1800,"endpoints":[{"url":"https://web-reports.snyk.io/csp?version=39e8721a74a6dffcb97fc790cfbf1fca91cefc03"}],"include_subdomains":true}
x-snyk-version: undefined
snyk-request-id: 1669e85e-abe5-401b-80bb-dae41829d6e1
location: https://snyk.io/api/v1/org/.....
content-type: application/json; charset=utf-8
content-length: 2
etag: W/"2-vyGp6PvFo4RvsFtPoIWeCReyIC8"
date: Fri, 02 Jul 2021 01:39:45 GMT
x-frame-options: deny
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
strict-transport-security: max-age=31536000; preload

{}

Finally switch back to the Snyk UI and verify you have imported the Azure repo as shown below


More Information
Import Projects API


Categories: Fusion Middleware

Basic Pipeline using Snyk Container, OCI Images, Azure DevOps all part of Cloud Native Application Security

Mon, 2021-06-14 00:20

Snyk Container will find vulnerabilities in containers and Kubernetes workloads throughout the SDLC by scanning any compliant OCI image which includes those created by Cloud Native Buildpacks or other build tools that create OCI images.

So what could an Azure DevOps Pipeline look like that incorporates the following using Snyk?

Running a Snyk Scan against the project repository
Here we run a "snyk test" from the root folder of the repository itself and that report is then



Building your Artifact
Here we use a Maven task which packages the application Artifact as a JAR file ready to run




Creating an OCI compliant container image from the Artifact itself
There are various ways to create a OCI compliant image but by the far the simplest is using Cloud Native Buildpacks and for this we use the pack CLI which in turns using the Java Buildpack from our JAR file directly avoid a compilation step from the source code given we already did that on the step above.



Running a Snyk Scan against the container image directly on the Container Registry
With our container image now in our Container Registry we can use "snyk container" to check for issues directly from the registry and also check for application security issues from the open source dependancies as well.





The finished Pipeline ...







azure-pipeline.yml Pipeline used in Azure DevOps

# Starter pipeline
# Start with a minimal pipeline that you can customize to build and deploy your code.
# Add steps that build, run tests, deploy, and more:
# https://aka.ms/yaml

trigger:
- master

pool:
vmImage: ubuntu-latest

steps:

- task: Maven@3
inputs:
mavenPomFile: 'pom.xml'
mavenOptions: '-Xmx3072m'
javaHomeOption: 'JDKVersion'
jdkVersionOption: '1.11'
jdkArchitectureOption: 'x64'
publishJUnitResults: true
options: "-Dskiptests -Dsnyk.skip"
goals: 'package'
displayName: "Build artifact JAR"
- task: SnykSecurityScan@0
inputs:
serviceConnectionEndpoint: 'snyk-token'
testType: 'app'
monitorOnBuild: false
failOnIssues: false
displayName: "snyk test from source"
- task: Docker@2
inputs:
containerRegistry: 'docker-pasapples-connection'
command: 'login'
displayName: "Login to DockerHub"

- script: |
curl -sSL "https://github.com/buildpacks/pack/releases/download/v0.19.0/pack-v0.19.0-linux.tgz" | tar -C ./ --no-same-owner -xzv pack
./pack build pasapples/springbootemployee:cnb-paketo-base --builder paketobuildpacks/builder:base --publish --path ./target/springbootemployee-0.0.1-SNAPSHOT.jar
displayName: 'Build Container with Pack'

- task: SnykSecurityScan@0
inputs:
serviceConnectionEndpoint: 'snyk-token'
testType: 'container'
dockerImageName: 'pasapples/springbootemployee:cnb-paketo-base'
severityThreshold: 'low'
monitorOnBuild: false
failOnIssues: false
additionalArguments: "--app-vulns"
displayName: "snyk container scan from image"


More Information

So, for Container and Kubernetes security, designed to help developers find and fix vulnerabilities in cloud native applications, click the links below to learn more and get started today.

Snyk Container

Snyk Platform

Categories: Fusion Middleware

Installing Snyk Controller into a k3d kubernetes cluster to enable runtime container scanning with the Snyk Platform

Thu, 2021-06-03 00:16

Snyk integrates with Kubernetes, enabling you to import and test your running workloads and identify vulnerabilities in their associated images and configurations that might make those workloads less secure. Once imported, Snyk continues to monitor those workloads, identifying additional security issues as new images are deployed and the workload configuration changes

In the example below we show you how easy it is to integrate the Snyk Platform with any K8s distribution in this case k3d running on my laptop.

Steps 

1. Install k3d using the instructions from the link below.

https://k3d.io/

2. Create a cluster as shown below.

pasapicella@192-168-1-113:~/snyk/demos/kubernetes/k3d$ k3d cluster create snyk-k3d --servers 1 --agents 2
INFO[0000] Prep: Network
INFO[0003] Created network 'k3d-snyk-k3d'
INFO[0003] Created volume 'k3d-snyk-k3d-images'
INFO[0004] Creating node 'k3d-snyk-k3d-server-0'
INFO[0005] Creating node 'k3d-snyk-k3d-agent-0'
INFO[0005] Creating node 'k3d-snyk-k3d-agent-1'
INFO[0005] Creating LoadBalancer 'k3d-snyk-k3d-serverlb'
INFO[0005] Starting cluster 'snyk-k3d'
INFO[0005] Starting servers...
INFO[0005] Starting Node 'k3d-snyk-k3d-server-0'
INFO[0012] Starting agents...
INFO[0012] Starting Node 'k3d-snyk-k3d-agent-0'
INFO[0023] Starting Node 'k3d-snyk-k3d-agent-1'
INFO[0031] Starting helpers...
INFO[0031] Starting Node 'k3d-snyk-k3d-serverlb'
INFO[0033] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access
INFO[0036] Successfully added host record to /etc/hosts in 4/4 nodes and to the CoreDNS ConfigMap
INFO[0036] Cluster 'snyk-k3d' created successfully!
INFO[0036] --kubeconfig-update-default=false --> sets --kubeconfig-switch-context=false
INFO[0036] You can now use it like this:
kubectl config use-context k3d-snyk-k3d
kubectl cluster-info

3. View the Kubernetes nodes.

$ kubectl get nodes
NAME                    STATUS   ROLES                  AGE   VERSION
k3d-snyk-k3d-server-0   Ready    control-plane,master   21h   v1.20.5+k3s1
k3d-snyk-k3d-agent-0    Ready    <none>                 21h   v1.20.5+k3s1
k3d-snyk-k3d-agent-1    Ready    <none>                 21h   v1.20.5+k3s1

4.  Run the following command in order to add the Snyk Charts repository to Helm.

$ helm repo add snyk-charts https://snyk.github.io/kubernetes-monitor/
"snyk-charts" already exists with the same configuration, skipping

5. Once the repository is added, create a unique namespace for the Snyk controller:

$ kubectl create namespace snyk-monitor

6. Now, log in to your Snyk account and navigate to Integrations. Search for and click Kubernetes. Click Connect from the page that loads, copy the Integration ID. The Snyk Integration ID is a UUID, similar to this format: abcd1234-abcd-1234-abcd-1234abcd1234. Save it for use from your Kubernetes environment in the next step

Instructions link : https://support.snyk.io/hc/en-us/articles/360006368657-Viewing-your-Kubernetes-integration-settings

7. Snyk monitor runs by using your Snyk Integration ID, and using a dockercfg file. If you are not using any private registries which we are not in this demo, create a Kubernetes secret called snyk-monitor containing the Snyk Integration ID from the previous step and run the following command:

$ kubectl create secret generic snyk-monitor -n snyk-monitor \
         --from-literal=dockercfg.json={} \
         --from-literal=integrationId=INTEGRATION_TOKEN_FROM_STEP_6
secret/snyk-monitor created

8. Install the Snyk Helm chart as follows:

$ helm upgrade --install snyk-monitor snyk-charts/snyk-monitor \
                          --namespace snyk-monitor \
                          --set clusterName="k3d Dev cluster"
Release "snyk-monitor" does not exist. Installing it now.
NAME: snyk-monitor
LAST DEPLOYED: Wed Jun  2 17:47:13 2021
NAMESPACE: snyk-monitor
STATUS: deployed
REVISION: 1
TEST SUITE: None

9. Verify the Snyk Controller is running using either 

$ kubectl get pods -n snyk-monitor
NAME                           READY   STATUS    RESTARTS   AGE
snyk-monitor-64c94685b-fwpvx   1/1     Running   3          21h

10. At this point we can create some workloads as follows let's just add a single POD to the cluster for a basic Spring Boot application.

$ kubectl run springboot-app --image=pasapples/spring-boot-jib --port=8080
pod/springboot-app created

11. Head back to the Snyk Dashboard and click on your Kubernetes Integration Tile and you should see a list of applicable workloads to monitor in our case  we just have the single app called "springboot-app".



12. Add the selected workload and your done!!!



More Information

Install the Snyk controller with Helm

Categories: Fusion Middleware

Cloud Native Buildpacks meets Snyk Container

Thu, 2021-05-27 07:35

Cloud Native Buildpacks transform your application source code into images that can run on any cloud and avoid ever having a write Dockerfile again after all why would you even care about a Dockerfile?

So in today's post we are going to take a look at how Snyk App with Snyk Container can scan those Cloud Native Buildpack OCI images for you.  

Snyk Container equips developers to quickly fix container issues. Use the following to find out more about Snyk Container - https://snyk.io/product/container-vulnerability-management/

You may not always have access to the original source code that runs in your containers, but vulnerabilities in your code dependencies are still important. Snyk can detect and monitor open source dependencies for popular languages as part of the container scan which is important because Cloud Native Buildpacks are building container images from source code using popular programming languages.

Demo

1. First let's clone spring petclinic app and create our artifact for deployment.

$ git clone https://github.com/spring-projects/spring-petclinic
$ cd spring-petclinic
$ /mvnw package 

2. Install pack so we can build some OCI images directly into our registries from a simple CLI. Instructions as follows


3. Test kpack is installed by listing the suggested builders

$ pack --version
0.18.1+git-b5c1a96.build-2373

$ pack builder suggest
Suggested builders:
Google:                       gcr.io/buildpacks/builder:v1      
Heroku:                       heroku/buildpacks:18              
Heroku:                       heroku/buildpacks:20      
Paketo Buildpacks:     paketobuildpacks/builder:base    
Paketo Buildpacks:     paketobuildpacks/builder:full    
Paketo Buildpacks:     paketobuildpacks/builder:tiny

At this point we are ready to go so let's create 3 OCI images and push them to 3 separate regitries as follows

  • Dockerhub
  • Amazon Elastic Container Registry (ECR)
  • Google Cloud Registry (GCR)

Note: Your local docker desktop needs to be able to establish connections to the 3 registries above

4. Build an OCI image of petclinic to Dockerhub.

$ pack build pasapples/petclinic:latest --builder paketobuildpacks/builder:base --publish --path ./spring-petclinic/target/spring-petclinic-2.4.5.jar
base: Pulling from paketobuildpacks/builder
Digest: sha256:a6f81cb029d4d3272981c12dad7212a5063ec0076e2438b7b5bb702f2e1fd11a
Status: Image is up to date for paketobuildpacks/builder:base
===> DETECTING
5 of 18 buildpacks participating
paketo-buildpacks/ca-certificates   2.2.0
paketo-buildpacks/bellsoft-liberica 8.0.0

...

Adding label 'org.opencontainers.image.version'
Adding label 'org.springframework.boot.version'
Setting default process type 'web'
Saving pasapples/petclinic:latest...
*** Images (sha256:d730612833826cd9e39a7241c1fba411eacd9b5f771915b00af4b3b499838bd5):
      pasapples/petclinic:latest
Successfully built image pasapples/petclinic:latest


5. Build an OCI image of petclinic to ECR.

$ pack build 300326902600.dkr.ecr.us-east-1.amazonaws.com/petclinic:latest --builder paketobuildpacks/builder:base --publish --path ./spring-petclinic/target/spring-petclinic-2.4.5.jar
base: Pulling from paketobuildpacks/builder
Digest: sha256:a6f81cb029d4d3272981c12dad7212a5063ec0076e2438b7b5bb702f2e1fd11a
Status: Image is up to date for paketobuildpacks/builder:base
===> DETECTING
5 of 18 buildpacks participating
paketo-buildpacks/ca-certificates   2.2.0
paketo-buildpacks/bellsoft-liberica 8.0.0

...

Adding label 'org.opencontainers.image.version'
Adding label 'org.springframework.boot.version'
Setting default process type 'web'
Saving 300326902600.dkr.ecr.us-east-1.amazonaws.com/petclinic:latest...
*** Images (sha256:d730612833826cd9e39a7241c1fba411eacd9b5f771915b00af4b3b499838bd5):
      300326902600.dkr.ecr.us-east-1.amazonaws.com/petclinic:latest
Successfully built image 300326902600.dkr.ecr.us-east-1.amazonaws.com/petclinic:latest


6. Build an OCI image of petclinic to GCR.

$ pack build us.gcr.io/snyk-cx-se-demo/petclinic-google:latest --builder paketobuildpacks/builder:base --publish --path ./spring-petclinic/target/spring-petclinic-2.4.5.jar
base: Pulling from paketobuildpacks/builder
Digest: sha256:a6f81cb029d4d3272981c12dad7212a5063ec0076e2438b7b5bb702f2e1fd11a
Status: Image is up to date for paketobuildpacks/builder:base
===> DETECTING
5 of 18 buildpacks participating
paketo-buildpacks/ca-certificates   2.2.0
paketo-buildpacks/bellsoft-liberica 8.0.0

...

Adding label 'org.opencontainers.image.version'
Adding label 'org.springframework.boot.version'
Setting default process type 'web'
Saving us.gcr.io/snyk-cx-se-demo/petclinic-google:latest...
*** Images (sha256:d730612833826cd9e39a7241c1fba411eacd9b5f771915b00af4b3b499838bd5):
      us.gcr.io/snyk-cx-se-demo/petclinic-google:latest
Successfully built image us.gcr.io/snyk-cx-se-demo/petclinic-google:latest


Three OCI compliant images of petclinic in our 3 registries done!!!

So now it's over to Snyk.

7. Login Snyk App - https://app.snyk.io 

8. Lets do a scan from the CLI by first authenticating with Snyk App as follows which will direct you to a browser to authenticate once done return to the prompt.

$ snyk auth 

9. Now you can run a snyk container test to any of the 3 registries we used as follows.

$ snyk container test us.gcr.io/snyk-cx-se-demo/petclinic-google:latest

$ snyk container test 300326902600.dkr.ecr.us-east-1.amazonaws.com/petclinic:latest

$ snyk container test pasapples/petclinic:latest

Demo Output

Testing us.gcr.io/snyk-cx-se-demo/petclinic-google:latest...

✗ Low severity vulnerability found in shadow/passwd
  Description: Time-of-check Time-of-use (TOCTOU)
  Info: https://snyk.io/vuln/SNYK-UBUNTU1804-SHADOW-306209
  Introduced through: shadow/passwd@1:4.5-1ubuntu2, apt@1.6.13, shadow/login@1:4.5-1ubuntu2
  From: shadow/passwd@1:4.5-1ubuntu2
  From: apt@1.6.13 > adduser@3.116ubuntu1 > shadow/passwd@1:4.5-1ubuntu2
  From: shadow/login@1:4.5-1ubuntu2

...

✗ Medium severity vulnerability found in gcc-8/libstdc++6
  Description: Information Exposure
  Info: https://snyk.io/vuln/SNYK-UBUNTU1804-GCC8-572149
  Introduced through: gcc-8/libstdc++6@8.4.0-1ubuntu1~18.04, apt/libapt-pkg5.0@1.6.13, apt@1.6.13, meta-common-packages@meta
  From: gcc-8/libstdc++6@8.4.0-1ubuntu1~18.04
  From: apt/libapt-pkg5.0@1.6.13 > gcc-8/libstdc++6@8.4.0-1ubuntu1~18.04
  From: apt@1.6.13 > gcc-8/libstdc++6@8.4.0-1ubuntu1~18.04
  and 2 more...

Organization:      pas.apicella-41p
Package manager:   deb
Project name:      docker-image|us.gcr.io/snyk-cx-se-demo/petclinic-google
Docker image:      us.gcr.io/snyk-cx-se-demo/petclinic-google:latest
Platform:          linux/amd64
Licenses:          enabled

Tested 97 dependencies for known issues, found 25 issues.

10. Back to Snyk App we can import all 3 OCI images from all 3 registries once we configure each integration for the registries as shown below





11. And take a look at where the vulnerabilities exist within those open source dependencies used in our petclinic source code as well as base image layer vulnerabilities







Give Snyk App a go yourself and start scanning those OCI container images built using Cloud Native Buildpacks now!!!

More Information
Main Snyk Web Page

Snyk Container
Categories: Fusion Middleware

Elastic Cloud with Observability 7.11 Using Rancher k3d for K8s

Thu, 2021-02-25 03:02

Started working locally with rancher's k3d to spin up K8s clusters so I thought why not use Elastic Cloud with Elastic Observability to monitor this local K8s cluster and even run container workloads that use Elastic APM to monitor the application through tracing.

A successful Kubernetes monitoring solution has a few requirements:
  • Monitors all layers of your technology stack, including:
    • The host systems where Kubernetes is running.
    • Kubernetes core components, nodes, pods, and containers running within the cluster.
    • All of the applications and services running in Kubernetes containers.
  • Automatically detects and monitors services as they appear dynamically.
  • Provides a way to correlate related data so that you can group and explore related metrics, logs, and other observability data.

Some basic steps to get this running as per below. This was tested with Elastic Stack 7.11 using the Elastic Cloud Service

Pre Steps

1. I have my Elastic stack running using Elastic Cloud as per the screen shot below

2. I have k3d installed on my Mac as follows

$ k3d --version
k3d version v4.0.0
k3s version latest (default)

3. You will need the kubectl CLI as well

Steps

1. First let's start a K8s cluster as follows. I have created a 5 node K8s cluster with 1 master node but you can reduce the number of worker nodes if you don't have the memory to support this

$ k3d cluster create elastic-k3d --servers 1 --agents 5

2. Start your k3d cluster as follows if not already started

$ k3d cluster start elastic-k3d
INFO[0000] Starting cluster 'elastic-k3d'
INFO[0000] Starting Node 'k3d-elastic-k3d-agent-4'
INFO[0000] Starting Node 'k3d-elastic-k3d-agent-3'
INFO[0000] Starting Node 'k3d-elastic-k3d-agent-2'
INFO[0001] Starting Node 'k3d-elastic-k3d-agent-1'
INFO[0001] Starting Node 'k3d-elastic-k3d-agent-0'
INFO[0001] Starting Node 'k3d-elastic-k3d-server-0'
INFO[0002] Starting Node 'k3d-elastic-k3d-serverlb'

3. Let's list out our nodes just to verify what you created

$ k get nodes
NAME                       STATUS   ROLES                  AGE   VERSION
k3d-elastic-k3d-server-0   Ready    control-plane,master   34h   v1.20.2+k3s1
k3d-elastic-k3d-agent-4    Ready    <none>                 34h   v1.20.2+k3s1
k3d-elastic-k3d-agent-1    Ready    <none>                 34h   v1.20.2+k3s1
k3d-elastic-k3d-agent-3    Ready    <none>                 34h   v1.20.2+k3s1
k3d-elastic-k3d-agent-2    Ready    <none>                 34h   v1.20.2+k3s1
k3d-elastic-k3d-agent-0    Ready    <none>                 34h   v1.20.2+k3s1

Now it's time to "Monitor Kubernetes: Observe the health and performance of your Kubernetes deployments" To do that we could just follow this tutorial as shown below

https://www.elastic.co/guide/en/observability/7.11/monitor-kubernetes.html

This tutorial will walk you through how to do the following

Deploy filebeat into your k3d cluster
Deploy metricbeat into your k3d cluster
Deploy an application which is instrumented using the Elastic APM agent 

4. When installing filebeat and metricbeat make sure you add your Elastic Cloud credentials which can be obtained from the Elastic Cloud deployments page and would have also been provided as a XLS file to download once you create your deployment

filebeat-kubernetes.yaml

    processors:
      - add_cloud_metadata:
      - add_host_metadata:

    cloud.id: pas-K8s:{PASSWORD}
    cloud.auth: elastic:{PASSWORD}

metricbeat-kubernetes.yaml (Notice how I have added a kubernetes_metadata processor)

    processors:
      - add_host_metadata:
      - add_kubernetes_metadata:

    cloud.id: pas-K8s:{PASSWORD}
    cloud.auth: elastic:{PASSWORD}

5. Once installed filebeat and metricbeat PODS should be running as per the output below

Filebeat

$ kubectl get pods -n kube-system -l k8s-app=filebeat
NAME             READY   STATUS    RESTARTS   AGE
filebeat-m8s6s   1/1     Running   2          32h
filebeat-g86vf   1/1     Running   2          32h
filebeat-bj548   1/1     Running   2          32h
filebeat-5cpcn   1/1     Running   2          32h
filebeat-nwj2h   1/1     Running   2          32h
filebeat-4hs8j   1/1     Running   2          32h

Metricbeat

$ kubectl get pods -n kube-system -l k8s-app=metricbeat
NAME               READY   STATUS    RESTARTS   AGE
metricbeat-pbfvs   1/1     Running   2          32h
metricbeat-v5n8l   1/1     Running   2          32h
metricbeat-cdfz9   1/1     Running   2          32h
metricbeat-z85g8   1/1     Running   2          32h
metricbeat-4fxhh   1/1     Running   2          32h
metricbeat-g25lh   1/1     Running   2          32h

6. At this point it's worth heading to Kibana Obserability page and from there you will see we have logs and metrics from our local k3d K8s cluster as shown below


7. Now click on Metrics view and you can clearly see each of your k3d K8s nodes and by clicking on any node you get a very convenient view of metrics, process, metadata and more all from a single page. This was one of the new Elastic 7.11 features for Observability



8. Finally to deploy a application to this K8s cluster and have Elastic APM instrument it's activity you can either use the example in the tutorial link provided above OR just follow this very simple example to get this done

https://github.com/papicella/elastic-customer-api-rest



More Information
k3d Home Page
k3d 

Elastic Cloud Service
Elastic Cloud 

Categories: Fusion Middleware

Elastic APM with Java Applications on Kubernetes

Tue, 2021-02-02 19:11

This GitHub demo is a Spring Boot application which accesses relational database with Spring Data JPA through a hypermedia-based RESTful front end. You can use the repo instructions to deploy to K8s and inject automatically an Elastic Agent for APM Monitoring with Elastic Observability.

There is no code changes required simply use an Init Container on K8s to instrument your application on Elastic APM server as shown by this GitHub repo below.

https://github.com/papicella/elastic-customer-api-rest



Try it out on your Kubernetes cluster using Elastic Cloud using our free 14 day trial











Categories: Fusion Middleware

Loading Australian Football League (AFL) Data into the Elastic Stack with some cool visulaizations

Mon, 2020-12-28 18:41

I decided to load some AFL data into the Elastic Stack and do some basic visualisations. I loaded data for all home and away plus finals games since 2017 so four seasons in total. Follow below if you want to do the same. 

Steps

Note: We already have Elasticsearch cluster running for this demo

$ curl -u "elastic:welcome1" localhost:9200
{
"name" : "node1",
"cluster_name" : "apples-cluster",
"cluster_uuid" : "hJrp2eJaRGCfBt7Zg_-EJQ",
"version" : {
"number" : "7.10.0",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "51e9d6f22758d0374a0f3f5c6e8f3a7997850f96",
"build_date" : "2020-11-09T21:30:33.964949Z",
"build_snapshot" : false,
"lucene_version" : "8.7.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}

First I need the data loaded into the Elastic Stack I did that using Squiggle API which you would do as follows

1. I use HTTPie rather then curl. 

http "https://api.squiggle.com.au/?q=games;complete=100" > games-2017-2020.json

2. Now this data itself needs to be altered slightly so I can BULK load it into Elasticsearch cluster and I do that as follows. I use JQ to do this.

cat games-2017-2020.json | jq -c '.games[] | {"index": {"_id": .id}}, .' > converted-games-2017-2020.json

Snippet I what the JSON file now looks like

{"index":{"_id":1}}

{"round":1,"hgoals":14,"roundname":"Round 1","hteamid":3,"hscore":89,"winner":"Richmond","ateam":"Richmond","hbehinds":5,"venue":"M.C.G.","year":2017,"complete":100,"id":1,"localtime":"2017-03-23 19:20:00","agoals":20,"date":"2017-03-23 19:20:00","hteam":"Carlton","updated":"2017-04-15 15:59:16","tz":"+11:00","ascore":132,"ateamid":14,"winnerteamid":14,"is_grand_final":0,"abehinds":12,"is_final":0}

{"index":{"_id":2}}

{"date":"2017-03-24 19:50:00","agoals":15,"ateamid":18,"winnerteamid":18,"hteam":"Collingwood","updated":"2017-04-15 15:59:16","tz":"+11:00","ascore":100,"is_grand_final":0,"abehinds":10,"is_final":0,"round":1,"hgoals":12,"hscore":86,"winner":"Western Bulldogs","ateam":"Western Bulldogs","roundname":"Round 1","hteamid":4,"hbehinds":14,"venue":"M.C.G.","year":2017,"complete":100,"id":2,"localtime":"2017-03-24 19:50:00"}

{"index":{"_id":3}}

{"hscore":82,"ateam":"Port Adelaide","winner":"Port Adelaide","roundname":"Round 1","hteamid":16,"round":1,"hgoals":12,"complete":100,"id":3,"localtime":"2017-03-25 16:35:00","venue":"S.C.G.","hbehinds":10,"year":2017,"ateamid":13,"winnerteamid":13,"updated":"2017-04-15 15:59:16","hteam":"Sydney","tz":"+11:00","ascore":110,"date":"2017-03-25 16:35:00","agoals":17,"is_final":0,"is_grand_final":0,"abehinds":8}

3. Using DevTools with Kibana we can run a query as follows

Question: Get each teams winning games for the season 2020 before finals - Final Ladder

Query:

GET afl_games/_search
{
"size": 0,
"query": {
"bool": {
"must": [
{
"match": {
"year": 2020
}
},
{
"match": {
"is_final": 0
}
}
]
}
},
"aggs": {
"group_by_winner": {
"terms": {
"field": "winner.keyword",
"size": 20
}
}
}
}

Results:

Results  
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 153,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
},
"aggregations" : {
"group_by_winner" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "Brisbane Lions",
"doc_count" : 14
},
{
"key" : "Port Adelaide",
"doc_count" : 14
},
{
"key" : "Geelong",
"doc_count" : 12
},
{
"key" : "Richmond",
"doc_count" : 12
},
{
"key" : "West Coast",
"doc_count" : 12
},
{
"key" : "St Kilda",
"doc_count" : 10
},
{
"key" : "Western Bulldogs",
"doc_count" : 10
},
{
"key" : "Collingwood",
"doc_count" : 9
},
{
"key" : "Melbourne",
"doc_count" : 9
},
{
"key" : "Greater Western Sydney",
"doc_count" : 8
},
{
"key" : "Carlton",
"doc_count" : 7
},
{
"key" : "Fremantle",
"doc_count" : 7
},
{
"key" : "Essendon",
"doc_count" : 6
},
{
"key" : "Gold Coast",
"doc_count" : 5
},
{
"key" : "Hawthorn",
"doc_count" : 5
},
{
"key" : "Sydney",
"doc_count" : 5
},
{
"key" : "Adelaide",
"doc_count" : 3
},
{
"key" : "North Melbourne",
"doc_count" : 3
}
]
}
}
}

4. Finally using Kibana Lens to easily visualize this data using a Kibana Dasboard


Of course you could do much more plus load more data from Squiggle and with the power of Kibana feel free to create your own visualizations.

More Information

Squiggle API

https://api.squiggle.com.au/

Getting Started with the Elastic Stack

https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html

Categories: Fusion Middleware

VMware Solutions Hub - Elastic Cloud on Kubernetes - the official Elasticsearch Operator from the creators

Tue, 2020-12-22 03:08

Proud to have worked on this with the VMware Tanzu team and Elastic team to add this to VMware Solution Hub page clearly highlighting what the Elastic Stack on Kubernetes really means.

Do you need to run your Elastic Stack on a certified Kubernetes distribution, bolstered by the global Kubernetes community allowing you to focus on delivering innovative applications powered by Elastic?

If so click below to get started:

https://tanzu.vmware.com/solutions-hub/data-management/elastic

More Information

https://tanzu.vmware.com/solutions-hub/data-management/elastic

Categories: Fusion Middleware

How to Become a Kubernetes Admin from the Comfort of Your vSphere

Tue, 2020-10-27 17:18

 My Talk at VMworld 2020 with Olive power can be found here.

Talk Details

In this session, we will walk through the integration of VMware vSphere and Kubernetes, and how this union of technologies can fundamentally change how virtual infrastructure and operational engineers view the management of Kubernetes platforms. We will demonstrate the capability of vSphere to host Kubernetes clusters internally, allocate capacity to those clusters, and monitor them side by side with virtual machines (VMs). We will talk about how extended vSphere functionality eases the transition of enterprises to running yet another platform (Kubernetes) by treating all managed endpoints—be they VMs, Kubernetes clusters or pods—as one platform. We want to demonstrate that platforms for running modern applications can be facilitated through the intuitive interface of vSphere and its ecosystem of automation tooling

https://www.vmworld.com/en/video-library/search.html#text=%22KUB2038%22&year=2020

Categories: Fusion Middleware

java-cfenv : A library for accessing Cloud Foundry Services on the new Tanzu Application Service for Kubernetes

Wed, 2020-09-02 19:19

The Spring Cloud Connectors library has been with us since the launch event of Cloud Foundry itself back in 2011. This library would create the required Spring Beans from bound VCAP_SERVICE ENV variable from a pushed Cloud Foundry Application such as connecting to databases for example. The java buildpack then replaces these bean definitions you had in your application with those created by the connector library through a feature called ‘auto-reconfiguration’

Auto-reconfiguration is great for getting started. However, it is not so great when you want more control, for example changing the size of the connection pool associated with a DataSource.

With the up coming Tanzu Application Service for Kubernetes the original Cloud Foundry buildpacks are now replaced with the new Tanzu Buildpacks which are based on the Cloud Native Buildpacks CNCF Sandbox project. As a result of this auto-reconfiguration is no longer included in java cloud native buildpacks which means auto-configuration for the backing services is no longer available.

So is their another option for this? The answer is "Java CFEnv". This provide a simple API for retrieving credentials from the JSON strings contained inside the VCAP_SERVICES environment variable.

https://github.com/pivotal-cf/java-cfenv



So if you after exactly how it worked previously all you need to do is add this maven dependancy to your project as shown below.

  
<dependency>
<groupId>io.pivotal.cfenv</groupId>
<artifactId>java-cfenv-boot</artifactId>
</dependency>

Of course this new library is much more flexible then this and by using the class CfEnv as the entry point to the API for accessing Cloud Foundry environment variables your free to use the Spring Expression Language to invoke methods on the bean of type CfEnv to set properties for example plus more.

For more information read the full blog post as per below

https://spring.io/blog/2019/02/15/introducing-java-cfenv-a-new-library-for-accessing-cloud-foundry-services

Finally this Spring Boot application is an example of using this new library with an application deployed to the new Tanzu Application Service for Kubernetes.

https://github.com/papicella/spring-book-service


More Information

1. Introducing java-cfenv: A new library for accessing Cloud Foundry Services

https://spring.io/blog/2019/02/15/introducing-java-cfenv-a-new-library-for-accessing-cloud-foundry-services

2. Java CFEnv GitHub Repo

https://github.com/pivotal-cf/java-cfenv#pushing-your-application-to-cloud-foundry

Categories: Fusion Middleware

Configure a MySQL Marketplace service for the new Tanzu Application Service on Kubernetes using Container Services Manager for VMware Tanzu

Thu, 2020-08-06 00:35
The following post shows how to configure a MySQL service into the new Tanzu Application Service BETA version 0.3.0. For instructions on how to install the Container Services Manager for VMware Tanzu (KSM) see post below.

http://www.clue2solve.io/tanzu/2020/07/14/install-ksm-and-configure-the-cf-marketplace.html
Steps
It's assumed you have already installed KSM into your Kubernetes Cluster as shown below. If not please refer to the documentation to get this done first


$ kubectl get all -n ksm
NAME READY STATUS RESTARTS AGE
pod/ksm-chartmuseum-78d5d5bfb-2ggdg 1/1 Running 0 15d
pod/ksm-ksm-broker-6db696894c-blvpp 1/1 Running 0 15d
pod/ksm-ksm-broker-6db696894c-mnshg 1/1 Running 0 15d
pod/ksm-ksm-daemon-587b6fd549-cc7sv 1/1 Running 1 15d
pod/ksm-ksm-daemon-587b6fd549-fgqx5 1/1 Running 1 15d
pod/ksm-postgresql-0 1/1 Running 0 15d

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ksm-chartmuseum ClusterIP 10.100.200.107 <none> 8080/TCP 15d
service/ksm-ksm-broker LoadBalancer 10.100.200.229 10.195.93.188 80:30086/TCP 15d
service/ksm-ksm-daemon LoadBalancer 10.100.200.222 10.195.93.179 80:31410/TCP 15d
service/ksm-postgresql ClusterIP 10.100.200.213 <none> 5432/TCP 15d
service/ksm-postgresql-headless ClusterIP None <none> 5432/TCP 15d

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ksm-chartmuseum 1/1 1 1 15d
deployment.apps/ksm-ksm-broker 2/2 2 2 15d
deployment.apps/ksm-ksm-daemon 2/2 2 2 15d

NAME DESIRED CURRENT READY AGE
replicaset.apps/ksm-chartmuseum-78d5d5bfb 1 1 1 15d
replicaset.apps/ksm-ksm-broker-6db696894c 2 2 2 15d
replicaset.apps/ksm-ksm-broker-8645dfcf98 0 0 0 15d
replicaset.apps/ksm-ksm-daemon-587b6fd549 2 2 2 15d

NAME READY AGE
statefulset.apps/ksm-postgresql 1/1 15d

1. let's start by getting the Broker IP address which when installed using LoadBalancer type can be retrieved as shown below.

$ kubectl get service ksm-ksm-broker -n ksm -o=jsonpath='{@.status.loadBalancer.ingress[0].ip}'
10.195.93.188

2. Upgrade your Helm release by running the following using the IP address from above

$ export BROKER_IP=$(kubectl get service ksm-ksm-broker -n ksm -o=jsonpath='{@.status.loadBalancer.ingress[0].ip}')
$ helm upgrade ksm ./ksm -n ksm --reuse-values \
            --set cf.brokerUrl="http://$BROKER_IP" \
            --set cf.brokerName=KSM \
            --set cf.apiAddress="https://api.system.run.haas-210.pez.pivotal.io" \
            --set cf.username="admin" \
            --set cf.password="admin-password"

3. Next we configure the ksm CLI. You can download the CLI from here

configure-ksm-cli.sh

export KSM_IP=$(kubectl get service ksm-ksm-daemon -n ksm -o=jsonpath='{@.status.loadBalancer.ingress[0].ip}')
export KSM_TARGET=http://$KSM_IP:$(kubectl get svc ksm-ksm-daemon -n ksm -o=jsonpath='{@.spec.ports[0].port}')
export KSM_USER=admin
export KSM_PASSWORD=$(kubectl get secret -n ksm ksm-ksm-daemon -o=jsonpath='{@.data.SECURITY_USER_PASSWORD}' | base64 --decode)

4. Verify ksm CLI is configured correctly

$ ksm version
Client Version [0.10.80]
Server Version [0.10.80]

5. Create a YAML file for the KSM service account and ClusterRoleBinding using the following YAML:

ksm-sa.yml

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ksm-admin
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: ksm-cluster-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: ksm-admin
    namespace: kube-system

Apply as follows

$ kubectl apply -f ksm-sa.yml

6. You need a cluster credential file to register and set default Kubernetes clusters that is done as follows

cluster-creds.sh

export kube_config="/Users/papicella/.kube/config"

cluster=`grep current $kube_config|sed "s/ //g"|cut -d ":" -f 2`

echo "Using cluster $cluster"

export server=`grep -B 2 "name: $cluster" $kube_config \
  |grep server|sed "s/ //g"|sed "s/^[^:]*://g"`

export certificate=`grep -B 2 "name: $cluster" $kube_config \
  |grep certificate|sed "s/ //g"|sed "s/.*://"`

export secret_name=$(kubectl get serviceaccount ksm-admin \
   --namespace=kube-system -o jsonpath='{.secrets[0].name}')

export secret_val=$(kubectl --namespace=kube-system get secret $secret_name \
   -o jsonpath='{.data.token}')

export secret_val=$(echo ${secret_val} | base64 --decode)

cat > cluster-creds.yaml << EOF
token: ${secret_val}
server: ${server}
caData: ${certificate}
EOF

echo ""
echo "ready to roll!!!!"
echo ""

Before running this script it's best to make sure you have targeted the correct K8s cluster you wish to. You can run a command as follows to verify that

$ kubectl config current-context
tas4k8s
 
7. Now we have a "cluster-creds.yaml" file we can go ahead and register the Kubernetes cluster with KSM as follows

$ ksm cluster register ksm-svcs ./cluster-creds.yaml
$ ksm cluster set-default ksm-svcs

Verify as follows:

$ ksm cluster list
CLUSTER NAME IP ADDRESS                                      DEFAULT
ksm-svcs    https://tas4k8s.run.haas-210.pez.pivotal.io:8443 true

8. Now we can go ahead and create a Marketplace offering for MySQL. To do that we will use the Bitnami MySQL chart as shown below

$ git clone https://github.com/bitnami/charts.git
$ cd ./charts/bitnami/mysql

** create bind.yaml as follows which is required so our service binding from Tanzu Application Service will inject the right JSON we are expecting or requiring at bind time **

$ cat bind.yaml
template: |
  local filterfunc(j) = std.length(std.findSubstr("mysql", j.name)) > 0;
  local s1 = std.filter(filterfunc, $.services);
  {
    hostname: s1[0].status.loadBalancer.ingress[0].ip,
    name: s1[0].name,
    jdbcUrl: "jdbc:mysql://" + self.hostname + "/my_db?user=" + self.username + "&password=" + self.password + "&useSSL=false",
    uri: "mysql://" + self.username + ":" + self.password + "@" + self.hostname + ":" + self.port + "/my_db?reconnect=true",
    password: $.secrets[0].data['mysql-root-password'],
    port: 3306,
    username: "root"
  }

$ helm package .
# cd ..
$ ksm offer save ./mysql ./mysql/mysql-6.14.7.tgz

Verify MySQL is now part of the offer list as follows
  
$ ksm offer list
MARKETPLACE NAME INCLUDED CHARTS VERSION PLANS
rabbitmq rabbitmq 6.18.1 [persistent ephemeral]
mysql mysql 6.14.7 [default]

9. Now we need to login as an ADMIN user

Verify you are logged in as admin user using the CF CLI:

$ cf target
api endpoint:   https://api.system.run.haas-210.pez.pivotal.io
api version:    2.151.0
user:           admin
org:            system
space:          development

10. At this point you can see the KSM service broker registered with TAS4K8s as follows

$ cf service-brokers
Getting service brokers as admin...

name   url
KSM    http://10.195.93.188

11. Enable access to the MySQL service as follows

$ cf enable-service-access mysql

Verify it's enabled:

$ cf service-access
Getting service access as admin...
broker: KSM
   service    plan         access   orgs
   mysql      default      all
   rabbitmq   ephemeral    all
   rabbitmq   persistent   all

12. At this point it's best to log out of admin and log back in as a user that is not admin

$ cf target
api endpoint:   https://api.system.run.haas-210.pez.pivotal.io
api version:    2.151.0
user:           pas
org:            apples-org
space:          development

13. Create a MySQL service as follows. I passing in some JSON to indicate that my K8s cluster support's a LoadBalancer type so use that as part of the creation of the service.

$ cf create-service mysql default pas-mysql -c '{"service":{"type":"LoadBalancer"}}'

14. Check that the service has created correctly it will take a few minutes

$ cf services
Getting services in org apples-org / space development as pas...

name        service    plan        bound apps          last operation     broker   upgrade available
pas-mysql   mysql      default     my-springboot-app   create succeeded   KSM      no

15. Your service is created in it's own K8s namespace BUT that may not be the case at some point. 
$ kubectl get all -n ksm-2e526124-11a3-4d38-966c-b3ffd45471d7
NAME READY STATUS RESTARTS AGE
pod/k-wqo5mubw-mysql-master-0 1/1 Running 0 15d
pod/k-wqo5mubw-mysql-slave-0 1/1 Running 0 15d

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/k-wqo5mubw-mysql LoadBalancer 10.100.200.12 10.195.93.192 3306:30563/TCP 15d
service/k-wqo5mubw-mysql-slave LoadBalancer 10.100.200.130 10.195.93.191 3306:31982/TCP 15d

NAME READY AGE
statefulset.apps/k-wqo5mubw-mysql-master 1/1 15d
statefulset.apps/k-wqo5mubw-mysql-slave 1/1 15d

16. At this point we can now test our new MySQL service we created and use a Spring Boot application to test this out with. 

The following GitHub repo can be used for that. Ignore the steps to create a service as you have already done that




Finally to define service plans see the link below

More Information
Container Services Manager(KSM)

Tanzu Application Service for Kubernetes

Categories: Fusion Middleware

Using CNCF Sandbox Project Strimzi for Kafka Clusters on VMware Tanzu Kubernetes Grid Integrated Edition (TKGI)

Sun, 2020-08-02 22:45
Strimzi a CNCF sandbox project provides a way to run an Apache Kafka cluster on Kubernetes in various deployment configurations. In this post we will take a look at how to get this running on VMware Tanzu Kubernetes Grid Integrated Edition (TKGI) and consume the Kafka cluster from a Springboot application.

If you have a K8s cluster that's all you need to follow along in this exampleI am using VMware Tanzu Kubernetes Grid Integrated Edition (TKGI) but you can use any K8s cluster you have such as GKE, AKS, EKS etc.

Steps

1. Installing Strimzi is pretty straight forward so we can do that as follows. I am using the namespace "kafka" which needs to be created prior to running this command.

kubectl apply -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka

2. Verify that the operator was installed correctly and we have a running POD as shown below
  
$ kubectl get pods -n kafka
NAME READY STATUS RESTARTS AGE
strimzi-cluster-operator-6c9d899778-4mdtg 1/1 Running 0 6d22h

3. Next let's ensure we have a default storage class for the cluster as shown below.

$ kubectl get storageclass
NAME             PROVISIONER                    AGE
fast (default)   kubernetes.io/vsphere-volume   47d

4. Now at this point we are ready to create a Kafka cluster. For this example we will create a 3 node cluster defined in YML as follows.

kafka-persistent-MULTI_NODE.yaml

apiVersion: kafka.strimzi.io/v1beta1
kind: Kafka
metadata:
  name: apples-kafka-cluster
spec:
  kafka:
    version: 2.5.0
    replicas: 3
    listeners:
      external:
        type: loadbalancer
        tls: false
      plain: {}
      tls: {}
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      log.message.format.version: "2.5"
    storage:
      type: jbod
      volumes:
      - id: 0
        type: persistent-claim
        size: 100Gi
        deleteClaim: false
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 100Gi
      deleteClaim: false
  entityOperator:
    topicOperator: {}
    userOperator: {}

Few things to note:
  • We have enable access to the cluster using the type LoadBalancer which means your K8s cluster needs to support such a Type
  • We need to create dynamic Persistence claim's in the cluster so ensure #3 above is in place
  • We have disabled TLS given this is a demo 
5. Create the Kafka cluster as shown below ensuring we target the namespace "kafka"

$ kubectl apply -f kafka-persistent-MULTI_NODE.yaml -n kafka

6. Now we can view the status/creation of our cluster one of two ways as shown below. You will need to wait a few minutes for everything to start up.

Option 1:
  
$ kubectl get Kafka -n kafka
NAME DESIRED KAFKA REPLICAS DESIRED ZK REPLICAS
apples-kafka-cluster 3 3 1/1 Running 0 6d22h

Option 2:
  
$ kubectl get all -n kafka
NAME READY STATUS RESTARTS AGE
pod/apples-kafka-cluster-entity-operator-58685b8fbd-r4wxc 3/3 Running 0 6d21h
pod/apples-kafka-cluster-kafka-0 2/2 Running 0 6d21h
pod/apples-kafka-cluster-kafka-1 2/2 Running 0 6d21h
pod/apples-kafka-cluster-kafka-2 2/2 Running 0 6d21h
pod/apples-kafka-cluster-zookeeper-0 1/1 Running 0 6d21h
pod/apples-kafka-cluster-zookeeper-1 1/1 Running 0 6d21h
pod/apples-kafka-cluster-zookeeper-2 1/1 Running 0 6d21h
pod/strimzi-cluster-operator-6c9d899778-4mdtg 1/1 Running 0 6d23h

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/apples-kafka-cluster-kafka-0 LoadBalancer 10.100.200.90 10.195.93.200 9094:30362/TCP 6d21h
service/apples-kafka-cluster-kafka-1 LoadBalancer 10.100.200.179 10.195.93.197 9094:32022/TCP 6d21h
service/apples-kafka-cluster-kafka-2 LoadBalancer 10.100.200.155 10.195.93.201 9094:32277/TCP 6d21h
service/apples-kafka-cluster-kafka-bootstrap ClusterIP 10.100.200.77 <none> 9091/TCP,9092/TCP,9093/TCP 6d21h
service/apples-kafka-cluster-kafka-brokers ClusterIP None <none> 9091/TCP,9092/TCP,9093/TCP 6d21h
service/apples-kafka-cluster-kafka-external-bootstrap LoadBalancer 10.100.200.58 10.195.93.196 9094:30735/TCP 6d21h
service/apples-kafka-cluster-zookeeper-client ClusterIP 10.100.200.22 <none> 2181/TCP 6d21h
service/apples-kafka-cluster-zookeeper-nodes ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 6d21h

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/apples-kafka-cluster-entity-operator 1/1 1 1 6d21h
deployment.apps/strimzi-cluster-operator 1/1 1 1 6d23h

NAME DESIRED CURRENT READY AGE
replicaset.apps/apples-kafka-cluster-entity-operator-58685b8fbd 1 1 1 6d21h
replicaset.apps/strimzi-cluster-operator-6c9d899778 1 1 1 6d23h

NAME READY AGE
statefulset.apps/apples-kafka-cluster-kafka 3/3 6d21h
statefulset.apps/apples-kafka-cluster-zookeeper 3/3 6d21h 3 1/1 Running 0 6d22h

7. Our entry point into the cluster is a service of type LoadBalancer which we asked for as per our Kafka cluster YML config. To find the IP address we can run a command as follow using the cluster name from above.

$ kubectl get service -n kafka apples-kafka-cluster-kafka-external-bootstrap -o=jsonpath='{.status.loadBalancer.ingress[0].ip}{"\n"}'
10.195.93.196

Note: Make a not of this IP address as we will need it shortly

8. Let's create a Kafka Topic using YML as follows. In this YML we actually ensure we are using the namespace "kafka".  

create-kafka-topic.yaml

apiVersion: kafka.strimzi.io/v1beta1
kind: KafkaTopic
metadata:
  name: apples-topic
  namespace: kafka
  labels:
    strimzi.io/cluster: apples-kafka-cluster
spec:
  partitions: 1
  replicas: 1
  config:
    retention.ms: 7200000
    segment.bytes: 1073741824


9. Create a Kafka topic as shown below.

$ kubectl apply -f create-kafka-topic.yaml

10. We can view the Kafka topics as shown below.
  
$ kubectl get KafkaTopic -n kafka
NAME PARTITIONS REPLICATION FACTOR
apples-topic 1 1

11. Now at this point we ready to send some messages to our topic "apples-topic" as well as consume messages so to do that we are going to use a Springboot Application in fact two of them which exist on GitHub.


Download or clone those onto your file system. 

12.With both downloaded you will need to set the spring.kafka.bootstrap-servers with the IP address we retrieved from #7 above. That needs to be done in both GitHub downloaded/cloned repo's above. The file we need to edit for both repo's is as follows. 

File: src/main/resources/application.yml 

Example:

spring:
  kafka:
    bootstrap-servers: IP-ADDRESS:9094

Note: Make sure you do this for both downloaded repo application.yml files

13. Now let's run the producer and consumer Springboot application using a command as follows in seperate terminal windows. One will use PORT 8080 while the other uses port 8081.

$ ./mvnw spring-boot:run

Consumer:

papicella@papicella:~/pivotal/DemoProjects/spring-starter/pivotal/KAFKA/demo-kafka-producer$ ./mvnw spring-boot:run

...
2020-08-03 11:41:46.742  INFO 34025 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
2020-08-03 11:41:46.754  INFO 34025 --- [           main] a.a.t.k.DemoKafkaProducerApplication     : Started DemoKafkaProducerApplication in 1.775 seconds (JVM running for 2.102)

Producer:

papicella@papicella:~/pivotal/DemoProjects/spring-starter/pivotal/KAFKA/demo-kafka-consumer$ ./mvnw spring-boot:run

...
2020-08-03 11:43:53.423  INFO 34056 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8081 (http) with context path ''
2020-08-03 11:43:53.440  INFO 34056 --- [           main] a.a.t.k.DemoKafkaConsumerApplication     : Started DemoKafkaConsumerApplication in 1.666 seconds (JVM running for 1.936)

14. Start by opening up the the Producer UI by navigating to http://localhost:8080/



15. Now let's not add any messages yet and also open up the Consumer UI by navigating to http://localhost:8081/



Note: This application will automatically refresh the page every 2 seconds to show which messages have been sent to the Kafka Topic

16. Return to the Producer UI http://localhost:8080/ and add two messages using whatever text you like as shown below.


17. Return to the Consumer UI http://localhost:8081/ to verify the two messages sent to the Kafka topic has been consumed



18. Both these Springboot applications are using "Spring for Apache Kafka


Both Springboot application use a application.yml to bootstrap access to the Kafka cluster

The Producer Springboot application is using a KafkaTemplate to send messages to our Kafka Topic as shown below.
  
@Controller
@Slf4j
public class TopicMessageController {

private KafkaTemplate<String, String> kafkaTemplate;

@Autowired
public TopicMessageController(KafkaTemplate<String, String> kafkaTemplate) {
this.kafkaTemplate = kafkaTemplate;
}

final private String topicName = "apples-topic";

@GetMapping("/")
public String indexPage (Model model){
model.addAttribute("topicMessageAddSuccess", "N");
return "home";
}

@PostMapping("/addentry")
public String addNewTopicMessage (@RequestParam(value="message") String message, Model model){

kafkaTemplate.send(topicName, message);

log.info("Sent single message: " + message);
model.addAttribute("message", message);
model.addAttribute("topicMessageAddSuccess", "Y");

return "home";
}
}

The Consumer Springboot application is configured with a KafkaListener as shown below
  
@Controller
@Slf4j
public class TopicConsumerController {

private static ArrayList<String> topicMessages = new ArrayList<String>();

@GetMapping("/")
public String indexPage (Model model){
model.addAttribute("topicMessages", topicMessages);
model.addAttribute("topicMessagesCount", topicMessages.size());

return "home";
}

@KafkaListener(topics = "apples-topic")
public void listen(String message) {
log.info("Received Message: " + message);
topicMessages.add(message);
}
}

In this post we did not setup any client authentication against the cluster for the producer or consumer given this was just a demo.





More Information

Spring for Apache Kafka

CNCF Sanbox projects

Strimzi
Categories: Fusion Middleware

Stumbled upon this today : Lens | The Kubernetes IDE

Thu, 2020-07-16 21:57
Lens is the only IDE you’ll ever need to take control of your Kubernetes clusters. It is a standalone application for MacOS, Windows and Linux operating systems. It is open source and free.

I installed it today and was impressed. Below is some screen shots of new Tanzu Application Service running on my Kubernetes cluster using Lens IDE. Simply point it to your Kube Config for the cluster you wish to examine.

On Mac SX it's installed as follows

$ brew cask install lens






More Information

https://github.com/lensapp/lens


Categories: Fusion Middleware

Spring Boot Data Elasticsearch using Elastic Cloud on Kubernetes (ECK) on VMware Tanzu Kubernetes Grid Integrated Edition (TKGI)

Mon, 2020-07-13 22:50
VMware Tanzu Kubernetes Grid Integrated Edition (formerly known as VMware Enterprise PKS) is a Kubernetes-based container solution with advanced networking, a private container registry, and life cycle management.

In this post I show how to get Elastic Cloud on Kubernetes (ECK) up and running on VMware Tanzu Kubernetes Grid Integrated Edition and how to access it using a Spring Boot Application using Spring Data Elasticsearch.

With ECK, users now have a seamless way of deploying, managing, and operating the Elastic Stack on Kubernetes.

If you have a K8s cluster that's all you need to follow along.

Steps

1. Let's install ECK on our cluster we do that as follows

Note: There is a 1.1 version as the latest BUT I installing a slightly older one here

$ kubectl apply -f https://download.elastic.co/downloads/eck/1.0.1/all-in-one.yaml

2. Make sure the operator is up and running as shown below
  
$ kubectl get all -n elastic-system
NAME READY STATUS RESTARTS AGE
pod/elastic-operator-0 1/1 Running 0 26d

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/elastic-webhook-server ClusterIP 10.100.200.55 <none> 443/TCP 26d

NAME READY AGE
statefulset.apps/elastic-operator 1/1 26d

3. We can also see a CRD for Elasticsearch as shown below.

elasticsearches.elasticsearch.k8s.elastic.co
  
$ kubectl get crd
NAME CREATED AT
apmservers.apm.k8s.elastic.co 2020-06-17T00:37:32Z
clusterlogsinks.pksapi.io 2020-06-16T23:04:43Z
clustermetricsinks.pksapi.io 2020-06-16T23:04:44Z
elasticsearches.elasticsearch.k8s.elastic.co 2020-06-17T00:37:33Z
kibanas.kibana.k8s.elastic.co 2020-06-17T00:37:34Z
loadbalancers.vmware.com 2020-06-16T22:51:52Z
logsinks.pksapi.io 2020-06-16T23:04:43Z
metricsinks.pksapi.io 2020-06-16T23:04:44Z
nsxerrors.nsx.vmware.com 2020-06-16T22:51:52Z
nsxlbmonitors.vmware.com 2020-06-16T22:51:52Z
nsxlocks.nsx.vmware.com 2020-06-16T22:51:51Z

4. We are now ready to create our first Elasticsearch cluster. To do that create a file YML file as shown below

create-elastic-cluster-from-operator.yaml

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 7.7.0
  http:
    service:
      spec:
        type: LoadBalancer # default is ClusterIP
    tls:
      selfSignedCertificate:
        disabled: true
  nodeSets:
  - name: default
    count: 2
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
    config:
      node.master: true
      node.data: true
      node.ingest: true
      node.store.allow_mmap: false

From the YML a few things to note:

  • We are creating two pods for our Elasticsearch cluster
  • We are using a K8s LoadBalancer to expose access to the cluster through HTTP
  • We are using version 7.7.0 but this is not the latest Elasticsearch version
  • We have disabled the use of TLS given this is just a demo
5. Apply that as shown below.

$ kubectl apply -f create-elastic-cluster-from-operator.yaml

6. After about a minute we should have our Elasticsearch cluster running. The following commands show that
  
$ kubectl get elasticsearch
NAME HEALTH NODES VERSION PHASE AGE
quickstart green 2 7.7.0 Ready 47h

$ kubectl get all -n default
NAME READY STATUS RESTARTS AGE
pod/quickstart-es-default-0 1/1 Running 0 47h
pod/quickstart-es-default-1 1/1 Running 0 47h

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.100.200.1 <none> 443/TCP 27d
service/quickstart-es-default ClusterIP None <none> <none> 47h
service/quickstart-es-http LoadBalancer 10.100.200.92 10.195.93.137 9200:30590/TCP 47h

NAME READY AGE
statefulset.apps/quickstart-es-default 2/2 47h

7. Let's deploy a Kibana instance. To do that create a YML as shown below

create-kibana.yaml

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana-sample
spec:
  version: 7.7.0
  count: 1
  elasticsearchRef:
    name: quickstart
    namespace: default
  http:
    service:
      spec:
        type: LoadBalancer # default is ClusterIP

8. Apply that as shown below.

$ kubectl apply -f create-kibana.yaml

9. To verify everything is up and running we can run a command as follows
  
$ kubectl get all
NAME READY STATUS RESTARTS AGE
pod/kibana-sample-kb-f8fcb88d5-jdzh5 1/1 Running 0 2d
pod/quickstart-es-default-0 1/1 Running 0 2d
pod/quickstart-es-default-1 1/1 Running 0 2d

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kibana-sample-kb-http LoadBalancer 10.100.200.46 10.195.93.174 5601:32459/TCP 2d
service/kubernetes ClusterIP 10.100.200.1 <none> 443/TCP 27d
service/quickstart-es-default ClusterIP None <none> <none> 2d
service/quickstart-es-http LoadBalancer 10.100.200.92 10.195.93.137 9200:30590/TCP 2d

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/kibana-sample-kb 1/1 1 1 2d

NAME DESIRED CURRENT READY AGE
replicaset.apps/kibana-sample-kb-f8fcb88d5 1 1 1 2d

NAME READY AGE
statefulset.apps/quickstart-es-default 2/2 2d

10. So to access out cluster we will need to obtain the following which we can do using a script as follows. This was tested on Mac OSX

What do we need?

  • Elasticsearch password
  • IP address of the LoadBalancer service we created


access.sh

export PASSWORD=`kubectl get secret quickstart-es-elastic-user -o go-template='{{.data.elastic | base64decode}}'`
export IP=`kubectl get svc quickstart-es-http -o jsonpath='{.status.loadBalancer.ingress[0].ip}'`

echo ""
echo $IP
echo ""

curl -u "elastic:$PASSWORD" "http://$IP:9200"

echo ""

curl -u "elastic:$PASSWORD" "http://$IP:9200/_cat/health?v"

Output:

10.195.93.137

{
  "name" : "quickstart-es-default-1",
  "cluster_name" : "quickstart",
  "cluster_uuid" : "Bbpb7Pu7SmaQaCmEY2Er8g",
  "version" : {
    "number" : "7.7.0",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "81a1e9eda8e6183f5237786246f6dced26a10eaf",
    "build_date" : "2020-05-12T02:01:37.602180Z",
    "build_snapshot" : false,
    "lucene_version" : "8.5.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

.....

11. Ideally I would load some data into the Elasticsearch cluster BUT let's do that as part of a sample application using "Spring Data Elasticsearch". Clone the demo project as shown below.

$ git clone https://github.com/papicella/boot-elastic-demo.git
Cloning into 'boot-elastic-demo'...
remote: Enumerating objects: 36, done.
remote: Counting objects: 100% (36/36), done.
remote: Compressing objects: 100% (26/26), done.
remote: Total 36 (delta 1), reused 36 (delta 1), pack-reused 0
Unpacking objects: 100% (36/36), done.

12. Edit "./src/main/resources/application.yml" with your details for the Elasticsearch cluster above.

spring:
  elasticsearch:
    rest:
      username: elastic
      password: {PASSWORD}
      uris: http://{IP}:9200

13. Package as follows

$ ./mvnw -DskipTests package

14. Run as follows

$ ./mvnw spring-boot:run

....
2020-07-14 11:10:11.947  INFO 76260 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8080 (http) with context path ''
2020-07-14 11:10:11.954  INFO 76260 --- [           main] c.e.e.demo.BootElasticDemoApplication    : Started BootElasticDemoApplication in 2.495 seconds (JVM running for 2.778)
....

15. Access application using "http://localhost:8080/"




16. If we look at our code we will see the data was loaded into the Elasticsearch cluster using a java class called "LoadData.java". Ideally data should already exist in the cluster but for demo purposes we load some data as part of the Spring Boot Application and clear the data prior to each application run given it's just a demo.

2020-07-14 11:12:33.109  INFO 76277 --- [           main] com.example.elastic.demo.LoadData        : Pre loading Car{id='OjThSnMBLjyTRl7lZsDL', make='holden', model='commodore', bodystyles=[BodyStyle{type='2-door'}, BodyStyle{type='4-door'}, BodyStyle{type='5-door'}]}
2020-07-14 11:12:33.584  INFO 76277 --- [           main] com.example.elastic.demo.LoadData        : Pre loading Car{id='OzThSnMBLjyTRl7laMCo', make='holden', model='astra', bodystyles=[BodyStyle{type='2-door'}, BodyStyle{type='4-door'}]}
2020-07-14 11:12:34.189  INFO 76277 --- [           main] com.example.elastic.demo.LoadData        : Pre loading Car{id='PDThSnMBLjyTRl7lasCC', make='nissan', model='skyline', bodystyles=[BodyStyle{type='4-door'}]}
2020-07-14 11:12:34.744  INFO 76277 --- [           main] com.example.elastic.demo.LoadData        : Pre loading Car{id='PTThSnMBLjyTRl7lbMDe', make='nissan', model='pathfinder', bodystyles=[BodyStyle{type='5-door'}]}
2020-07-14 11:12:35.227  INFO 76277 --- [           main] com.example.elastic.demo.LoadData        : Pre loading Car{id='PjThSnMBLjyTRl7lb8AL', make='ford', model='falcon', bodystyles=[BodyStyle{type='4-door'}, BodyStyle{type='5-door'}]}
2020-07-14 11:12:36.737  INFO 76277 --- [           main] com.example.elastic.demo.LoadData        : Pre loading Car{id='QDThSnMBLjyTRl7lcMDu', make='ford', model='territory', bodystyles=[BodyStyle{type='5-door'}]}
2020-07-14 11:12:37.266  INFO 76277 --- [           main] com.example.elastic.demo.LoadData        : Pre loading Car{id='QTThSnMBLjyTRl7ldsDU', make='toyota', model='camry', bodystyles=[BodyStyle{type='4-door'}, BodyStyle{type='5-door'}]}
2020-07-14 11:12:37.777  INFO 76277 --- [           main] com.example.elastic.demo.LoadData        : Pre loading Car{id='QjThSnMBLjyTRl7leMDk', make='toyota', model='corolla', bodystyles=[BodyStyle{type='2-door'}, BodyStyle{type='5-door'}]}
2020-07-14 11:12:38.285  INFO 76277 --- [           main] com.example.elastic.demo.LoadData        : Pre loading Car{id='QzThSnMBLjyTRl7lesDj', make='kia', model='sorento', bodystyles=[BodyStyle{type='5-door'}]}
2020-07-14 11:12:38.800  INFO 76277 --- [           main] com.example.elastic.demo.LoadData        : Pre loading Car{id='RDThSnMBLjyTRl7lfMDg', make='kia', model='sportage', bodystyles=[BodyStyle{type='4-door'}]}

LoadData.java
  
package com.example.elastic.demo;

import com.example.elastic.demo.indices.BodyStyle;
import com.example.elastic.demo.indices.Car;
import com.example.elastic.demo.repo.CarRepository;
import org.springframework.boot.CommandLineRunner;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import lombok.extern.slf4j.Slf4j;

import static java.util.Arrays.asList;

@Configuration
@Slf4j
public class LoadData {
@Bean
public CommandLineRunner initElasticsearchData(CarRepository carRepository) {
return args -> {
carRepository.deleteAll();
log.info("Pre loading " + carRepository.save(new Car("holden", "commodore", asList(new BodyStyle("2-door"), new BodyStyle("4-door"), new BodyStyle("5-door")))));
log.info("Pre loading " + carRepository.save(new Car("holden", "astra", asList(new BodyStyle("2-door"), new BodyStyle("4-door")))));
log.info("Pre loading " + carRepository.save(new Car("nissan", "skyline", asList(new BodyStyle("4-door")))));
log.info("Pre loading " + carRepository.save(new Car("nissan", "pathfinder", asList(new BodyStyle("5-door")))));
log.info("Pre loading " + carRepository.save(new Car("ford", "falcon", asList(new BodyStyle("4-door"), new BodyStyle("5-door")))));
log.info("Pre loading " + carRepository.save(new Car("ford", "territory", asList(new BodyStyle("5-door")))));
log.info("Pre loading " + carRepository.save(new Car("toyota", "camry", asList(new BodyStyle("4-door"), new BodyStyle("5-door")))));
log.info("Pre loading " + carRepository.save(new Car("toyota", "corolla", asList(new BodyStyle("2-door"), new BodyStyle("5-door")))));
log.info("Pre loading " + carRepository.save(new Car("kia", "sorento", asList(new BodyStyle("5-door")))));
log.info("Pre loading " + carRepository.save(new Car("kia", "sportage", asList(new BodyStyle("4-door")))));
};
}
}

17. Our CarRepository interface is defined as follows

CarRepository.java
  
package com.example.elastic.demo.repo;

import com.example.elastic.demo.indices.Car;
import org.springframework.data.domain.Page;
import org.springframework.data.domain.Pageable;
import org.springframework.data.elasticsearch.repository.ElasticsearchRepository;
import org.springframework.stereotype.Repository;

@Repository
public interface CarRepository extends ElasticsearchRepository <Car, String> {

Page<Car> findByMakeContaining(String make, Pageable page);

}

18. So let's also via this data using "curl" and Kibana as shown below.

curl -X GET -u "elastic:{PASSWORD}" "http://{IP}:9200/vehicle/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "query": { "match_all": {} },
  "sort": [
    { "_id": "asc" }
  ]
}
'

Output:

{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 10,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [
      {
        "_index" : "vehicle",
        "_type" : "_doc",
        "_id" : "OjThSnMBLjyTRl7lZsDL",
        "_score" : null,
        "_source" : {
          "_class" : "com.example.elastic.demo.indices.Car",
          "make" : "holden",
          "model" : "commodore",
          "bodystyles" : [
            {
              "type" : "2-door"
            },
            {
              "type" : "4-door"
            },
            {
              "type" : "5-door"
            }
          ]
        },
        "sort" : [
          "OjThSnMBLjyTRl7lZsDL"
        ]
      },
      {
        "_index" : "vehicle",
        "_type" : "_doc",
        "_id" : "OzThSnMBLjyTRl7laMCo",
        "_score" : null,
        "_source" : {
          "_class" : "com.example.elastic.demo.indices.Car",
          "make" : "holden",
          "model" : "astra",
          "bodystyles" : [
            {
              "type" : "2-door"
            },
            {
              "type" : "4-door"
            }
          ]
        },
        "sort" : [
          "OzThSnMBLjyTRl7laMCo"
        ]
      },
      {
        "_index" : "vehicle",
        "_type" : "_doc",
        "_id" : "PDThSnMBLjyTRl7lasCC",
        "_score" : null,
        "_source" : {
          "_class" : "com.example.elastic.demo.indices.Car",
          "make" : "nissan",
          "model" : "skyline",
          "bodystyles" : [
            {
              "type" : "4-door"
            }
          ]
        },
        "sort" : [
          "PDThSnMBLjyTRl7lasCC"
        ]
      },
      {
        "_index" : "vehicle",
        "_type" : "_doc",
        "_id" : "PTThSnMBLjyTRl7lbMDe",
        "_score" : null,
        "_source" : {
          "_class" : "com.example.elastic.demo.indices.Car",
          "make" : "nissan",
          "model" : "pathfinder",
          "bodystyles" : [
            {
              "type" : "5-door"
            }
          ]
        },
        "sort" : [
          "PTThSnMBLjyTRl7lbMDe"
        ]
      },
      {
        "_index" : "vehicle",
        "_type" : "_doc",
        "_id" : "PjThSnMBLjyTRl7lb8AL",
        "_score" : null,
        "_source" : {
          "_class" : "com.example.elastic.demo.indices.Car",
          "make" : "ford",
          "model" : "falcon",
          "bodystyles" : [
            {
              "type" : "4-door"
            },
            {
              "type" : "5-door"
            }
          ]
        },
        "sort" : [
          "PjThSnMBLjyTRl7lb8AL"
        ]
      },
      {
        "_index" : "vehicle",
        "_type" : "_doc",
        "_id" : "QDThSnMBLjyTRl7lcMDu",
        "_score" : null,
        "_source" : {
          "_class" : "com.example.elastic.demo.indices.Car",
          "make" : "ford",
          "model" : "territory",
          "bodystyles" : [
            {
              "type" : "5-door"
            }
          ]
        },
        "sort" : [
          "QDThSnMBLjyTRl7lcMDu"
        ]
      },
      {
        "_index" : "vehicle",
        "_type" : "_doc",
        "_id" : "QTThSnMBLjyTRl7ldsDU",
        "_score" : null,
        "_source" : {
          "_class" : "com.example.elastic.demo.indices.Car",
          "make" : "toyota",
          "model" : "camry",
          "bodystyles" : [
            {
              "type" : "4-door"
            },
            {
              "type" : "5-door"
            }
          ]
        },
        "sort" : [
          "QTThSnMBLjyTRl7ldsDU"
        ]
      },
      {
        "_index" : "vehicle",
        "_type" : "_doc",
        "_id" : "QjThSnMBLjyTRl7leMDk",
        "_score" : null,
        "_source" : {
          "_class" : "com.example.elastic.demo.indices.Car",
          "make" : "toyota",
          "model" : "corolla",
          "bodystyles" : [
            {
              "type" : "2-door"
            },
            {
              "type" : "5-door"
            }
          ]
        },
        "sort" : [
          "QjThSnMBLjyTRl7leMDk"
        ]
      },
      {
        "_index" : "vehicle",
        "_type" : "_doc",
        "_id" : "QzThSnMBLjyTRl7lesDj",
        "_score" : null,
        "_source" : {
          "_class" : "com.example.elastic.demo.indices.Car",
          "make" : "kia",
          "model" : "sorento",
          "bodystyles" : [
            {
              "type" : "5-door"
            }
          ]
        },
        "sort" : [
          "QzThSnMBLjyTRl7lesDj"
        ]
      },
      {
        "_index" : "vehicle",
        "_type" : "_doc",
        "_id" : "RDThSnMBLjyTRl7lfMDg",
        "_score" : null,
        "_source" : {
          "_class" : "com.example.elastic.demo.indices.Car",
          "make" : "kia",
          "model" : "sportage",
          "bodystyles" : [
            {
              "type" : "4-door"
            }
          ]
        },
        "sort" : [
          "RDThSnMBLjyTRl7lfMDg"
        ]
      }
    ]
  }
}

Kibana

Obtain Kibana HTTP IP as shown below and login using username "elastic" and password we obtained previously.

$ kubectl get svc kibana-sample-kb-http -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
10.195.93.174




Finally maybe you want to deploy the application to Kubernetes. To do that take a look at Cloud Native Buildpacks CNCF project and/or Tanzu Build Service to turn your code into a Container Image stored in a registry.



More Information

Spring Data Elasticsearch
https://spring.io/projects/spring-data-elasticsearch

VMware Tanzu Kubernetes Grid Integrated Edition Documentation
https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid-Integrated-Edition/index.html
Categories: Fusion Middleware

Multi-Factor Authentication (MFA) using OKTA with Spring Boot and Tanzu Application Service

Thu, 2020-07-09 23:22
Recently I was asked to build a quick demo showing how to use MFA with OKTA and Spring Boot application running on Tanzu Application Service. Here is the demo application plus how to setup and run this yourself.

Steps

1. Clone the existing repo as shown below

$ git clone https://github.com/papicella/mfa-boot-fsi
Cloning into 'mfa-boot-fsi'...
remote: Enumerating objects: 47, done.
remote: Counting objects: 100% (47/47), done.
remote: Compressing objects: 100% (31/31), done.
remote: Total 47 (delta 2), reused 47 (delta 2), pack-reused 0
Unpacking objects: 100% (47/47), done.



2. Create a free account of https://developer.okta.com/

Once created login to the dev account. Your account URL will look like something as follows

https://dev-{ID}-admin.okta.com



3. You will need your default authorization server settings. From the top menu in the developer.okta.com dashboard, go to API -> Authorization Servers and click on the default server


You will need this data shortly. Image above is an example those details won't work for your own setup.

4. From the top menu, go to Applications and click the Add Application button. Click on the Web button and click Next. Name your app whatever you like. I named mine "pas-okta-springapp". Otherwise the default settings are fine. Click Done.

From this screen shot you can see that the default's refer to localhost which for DEV purposes is fine.


You will need the Client ID and Client secret from the final screen so make a note of these

5. Edit the "./mfa-boot-fsi/src/main/resources/application-DEV.yml" to include the details as per #3 and #4 above.

You will need to edit

  • issuer
  • client-id
  • client-secret


application-DEV.yaml

spring:
  security:
    oauth2:
      client:
        provider:
          okta:
            user-name-attribute: email

okta:
  oauth2:
    issuer: https://dev-213269.okta.com/oauth2/default
    redirect-uri: /authorization-code/callback
    scopes:
      - profile
      - email
      - openid
    client-id: ....
    client-secret: ....

6. In order to pick up this application-DEV.yaml we have to set the spring profile correctly. That can be done using a JVM property as follows.

-Dspring.profiles.active=DEV

In my example I use IntelliJ IDEA so I set it on the run configurations dialog as follows



7. Finally let's setup MFA which we do as follows by switching to classic UI as shown below



8. Click on Security -> Multifactor and setup another Multifactor policy. In the screen shot below I select "Email Policy" and make sure it is "Required" along with the default policy



9. Now run the application making sure you set the spring active profile to DEV.

...
2020-07-10 13:34:57.528  INFO 55990 --- [  restartedMain] pas.apa.apj.mfa.demo.DemoApplication     : The following profiles are active: DEV
...

10. Navigate to http://localhost:8080/



11. Click on the "Login" button

Verify you are taken to the default OKTA login page


12. Once logged in the second factor should then ask for a verification code to be sent to your email. Press the "Send me the code" button




13. Once you enter the code sent to your email you will be granted access to the application endpoints







14. Finally to deploy the application to Tanzu Application Service perform these steps below

- Create a manifest.yaml as follows

---
applications:
- name: pas-okta-boot-app 
  memory: 1024M
  buildpack: https://github.com/cloudfoundry/java-buildpack.git#v4.16
  instances: 2
  path: ./target/demo-0.0.1-SNAPSHOT.jar
  env:
    JBP_CONFIG_OPEN_JDK_JRE: '{ jre: { version: 11.+}}'

- Package the application as follows

$ ./mvnw -DskipTests package

- In the DEV OTKA console create a second application which will be for the deployed application on Tanzu Application Service which refers to it's FQDN rather then localhost as shown below



- Edit "application.yml" to ensure you set the following correctly for the new "Application" we created above.

You will need to edit

  • issuer
  • client-id
  • client-secret
- Push the application using "cf push -f manifest.yaml"

$ cf apps
Getting apps in org papicella-org / space apple as papicella@pivotal.io...
OK

name                requested state   instances   memory   disk   urls
pas-okta-boot-app   started           1/1         1G       1G     pas-okta-boot-app.cfapps.io


That's It!!!!

Categories: Fusion Middleware

GitHub Actions to deploy Spring Boot application to Tanzu Application Service for Kubernetes

Wed, 2020-06-17 21:28
In this demo I show how to deploy a simple Spring boot application using GitHub Actions onto Tanzu Application Service for Kubernetes (TAS4K8s).

Steps

Ensure you have Tanzu Application Service for Kubernetes (TAS4K8s) running as shown below.
  
$ kapp list
Target cluster 'https://35.189.13.31' (nodes: gke-tanzu-gke-lab-f67-np-f67b23a0f590-abbca04e-5sqc, 8+)

Apps in namespace 'default'

Name Namespaces Lcs Lca
certmanager-cluster-issuer (cluster) true 8d
externaldns (cluster),external-dns true 8d
harbor-cert harbor true 8d
tas (cluster),cf-blobstore,cf-db,cf-system, false 8d
cf-workloads,cf-workloads-staging,istio-system,kpack,
metacontroller
tas4k8s-cert cf-system true 8d

Lcs: Last Change Successful
Lca: Last Change Age

5 apps

Succeeded

The demo exists on GitHub using the following URL, to follow along simply use your own GitHub repository making the changes as detailed below. The example below is for a Spring Boot application so your YAML file for the action would differ for non Java applications but there are many starter templates to choose from for other programming languages.

https://github.com/papicella/github-boot-demo



GitHub Actions help you automate your software development workflows in the same place you store code and collaborate on pull requests and issues. You can write individual tasks, called actions, and combine them to create a custom workflow. Workflows are custom automated processes that you can set up in your repository to build, test, package, release, or deploy any code project on GitHub

1. Create a folder at the root of your project source code as follows

$ mkdir ".github/workflows"

2. In ".github/workflows" folder, add a .yml or .yaml file for your workflow. For example, ".github/workflows/maven.yml"

3. Use the "Workflow syntax for GitHub Actions" reference documentation to choose events to trigger an action, add actions, and customize your workflow. In this example the YML "maven.yml" looks as follows.

maven.yml
  
name: Java CI with Maven and CD with CF CLI

on:
push:
branches: [ master ]
pull_request:
branches: [ master ]

jobs:
build:

runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2
- name: Set up JDK 11.0.5
uses: actions/setup-java@v1
with:
java-version: 11.0.5
- name: Build with Maven
run: mvn -B package --file pom.xml
- name: push to TAS4K8s
env:
CF_USERNAME: ${{ secrets.CF_USERNAME }}
CF_PASSWORD: ${{ secrets.CF_PASSWORD }}
run: |
curl --location "https://cli.run.pivotal.io/stable?release=linux64-binary&source=github" | tar zx
./cf api https://api.tas.lab.pasapples.me --skip-ssl-validation
./cf auth $CF_USERNAME $CF_PASSWORD
./cf target -o apples-org -s development
./cf push -f manifest.yaml

Few things here around the YML Workflow syntax for the GitHub Action above

  • We are using a maven action sample which will FIRE on a push or pull request on the master branch
  • We are using JDK 11 rather then Java 8
  • 3 Steps exists here
    • Setup JDK
    • Maven Build/Package
    • CF CLI Push to TAS4K8s using the built JAR artifact from the maven build
  • We download the CF CLI into ubuntu image 
  • We have masked the username and password using Secrets

4. Next in the project root add a manifest YAML for deployment to TAS4K8s

- Add a manifest.yaml file in the project root to deploy our simple Spring boot RESTful application

---
applications:
  - name: github-TAS4K8s-boot-demo
    memory: 1024M
    instances: 1
    path: ./target/demo-0.0.1-SNAPSHOT.jar

5. Now we need to add Secrets to the Github repo which are referenced in out "maven.yml" file. In our case they are as follows.
  • CF_USERNAME 
  • CF_PASSWORD
In your GitHub repository click on "Settings" tab then on left hand side navigation bar click on "Secrets" and define your username and password for your TAS4K8s instance as shown below



6. At this point that is all we need to test our GitHub Action. Here in IntelliJ IDEA I issue a commit/push to trigger the GitHub action



7. If all went well using "Actions" tab in your GitHub repo will show you the status and logs as follows






8. Finally our application will be deployed to TAS4K8s as shown below and we can invoke it using HTTPie or CURL for example
  
$ cf apps
Getting apps in org apples-org / space development as pas...
OK

name requested state instances memory disk urls
github-TAS4K8s-boot-demo started 1/1 1G 1G github-tas4k8s-boot-demo.apps.tas.lab.pasapples.me
my-springboot-app started 1/1 1G 1G my-springboot-app.apps.tas.lab.pasapples.me
test-node-app started 1/1 1G 1G test-node-app.apps.tas.lab.pasapples.me

$ cf app github-TAS4K8s-boot-demo
Showing health and status for app github-TAS4K8s-boot-demo in org apples-org / space development as pas...

name: github-TAS4K8s-boot-demo
requested state: started
isolation segment: placeholder
routes: github-tas4k8s-boot-demo.apps.tas.lab.pasapples.me
last uploaded: Thu 18 Jun 12:03:19 AEST 2020
stack:
buildpacks:

type: web
instances: 1/1
memory usage: 1024M
state since cpu memory disk details
#0 running 2020-06-18T02:03:32Z 0.2% 136.5M of 1G 0 of 1G

$ http http://github-tas4k8s-boot-demo.apps.tas.lab.pasapples.me
HTTP/1.1 200 OK
content-length: 28
content-type: text/plain;charset=UTF-8
date: Thu, 18 Jun 2020 02:07:39 GMT
server: istio-envoy
x-envoy-upstream-service-time: 141

Thu Jun 18 02:07:39 GMT 2020



More Information

Download TAS4K8s
https://network.pivotal.io/products/tas-for-kubernetes/

GitHub Actions
https://github.com/features/actions

GitHub Marketplace - Actions
https://github.com/marketplace?type=actions
Categories: Fusion Middleware

Deploying a Spring Boot application to Tanzu Application Service for Kubernetes using GitLab

Mon, 2020-06-15 20:44
In this demo I show how to deploy a simple Springboot application using GitLab pipeline onto Tanzu Application Service for Kubernetes (TAS4K8s).

Steps

Ensure you have Tanzu Application Service for Kubernetes (TAS4K8s) running as shown below
  
$ kapp list
Target cluster 'https://lemons.run.haas-236.pez.pivotal.io:8443' (nodes: a51852ac-e449-40ad-bde7-1beb18340854, 5+)

Apps in namespace 'default'

Name Namespaces Lcs Lca
cf (cluster),build-service,cf-blobstore,cf-db, true 10d
cf-system,cf-workloads,cf-workloads-staging,
istio-system,kpack,metacontroller

Lcs: Last Change Successful
Lca: Last Change Age

1 apps

Succeeded

Ensure you have GitLab running. In this example it's installed on a Kubernetes cluster but it doesn't have to be. All that matters here is that GitLab can access the API endpoint of your TAS4K8s install
  
$ helm ls -A
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
gitlab gitlab 2 2020-05-15 13:22:15.470219 +1000 AEST deployed gitlab-3.3.4 12.10.5

1. First let's create a basic Springboot application with a simple RESTful endpoint as shown below. It's best to use the Spring Initializer to create this application. I simply used the web and lombok dependancies as shown below.

Note: Make sure you select java version 11.

Spring Initializer Web Interface


Using built in Spring Initializer in IntelliJ IDEA.


Here is my simple RESTful controller which simply output's todays date.
  
package com.example.demo;

import lombok.extern.slf4j.Slf4j;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

import java.util.Date;

@RestController
@Slf4j
public class FrontEnd {
@GetMapping("/")
public String index () {
log.info("An INFO Message");
return new Date().toString();
}
}

2. Create an empty project in GitLab using the name "gitlab-TAS4K8s-boot-demo"



3. At this point this add our project files from step #1 above into the empty GitLab project repository. We do that as follows.

$ cd "existing project folder from step #1"
$ git init
$ git remote add origin http://gitlab.ci.run.haas-236.pez.pivotal.io/root/gitlab-tas4k8s-boot-demo.git
$ git add .
$ git commit -m "Initial commit"
$ git push -u origin master

Once done we now have out GitLab project repository with the files we created as part of the project setup


4. It's always worth running the code locally just to make sure it's working so if you like you can do that as follows

RUN:

$ ./mvnw spring-boot:run

CURL:

$ curl http://localhost:8080/
Tue Jun 16 10:46:26 AEST 2020

HTTPie:

papicella@papicella:~$
papicella@papicella:~$
papicella@papicella:~$ http :8080/
HTTP/1.1 200
Connection: keep-alive
Content-Length: 29
Content-Type: text/plain;charset=UTF-8
Date: Tue, 16 Jun 2020 00:46:40 GMT
Keep-Alive: timeout=60

Tue Jun 16 10:46:40 AEST 2020

5. Our GitLab project as no pipelines defined so let's create one as follows in the project root directory using the default pipeline name ".gitlab-ci.yml"

image: openjdk:11-jdk

stages:
  - build
  - deploy

build:
  stage: build
  script: ./mvnw package
  artifacts:
    paths:
      - target/demo-0.0.1-SNAPSHOT.jar

production:
  stage: deploy
  script:
  - curl --location "https://cli.run.pivotal.io/stable?release=linux64-binary&source=github" | tar zx
  - ./cf api https://api.system.run.haas-236.pez.pivotal.io --skip-ssl-validation
  - ./cf auth $CF_USERNAME $CF_PASSWORD
  - ./cf target -o apples-org -s development
  - ./cf push -f manifest.yaml
  only:
  - master


Note: We have not defined any tests in our pipeline which we should do but we haven't written any in this example.

6. For this pipeline to work we will need to do the following

- Add a manifest.yaml file in the project root to deploy our simple Springboot RESTful application

---
applications:
  - name: gitlab-TAS4K8s-boot-demo
    memory: 1024M
    instances: 1
    path: ./target/demo-0.0.1-SNAPSHOT.jar

- Alter the API endpoint to match your TAS4K8s endpoint

- ./cf api https://api.system.run.haas-236.pez.pivotal.io --skip-ssl-validation

- Alter the target to use your ORG and SPACE within TAs4K8s.

- ./cf target -o apples-org -s development

This command shows you what your current CF CLI is targeted to so you can ensure you edit it with correct details
  
$ cf target
api endpoint: https://api.system.run.haas-236.pez.pivotal.io
api version: 2.150.0
user: pas
org: apples-org
space: development

7. For the ".gitlab-ci.yml" to work we need to define two ENV variables for our username and password. Those two are as follows which is our login credentials to TAS4K8s

  • CF_USERNAME 
  • CF_PASSWORD

To do that we need to navigate to "Project Settings -> CI/CD - Variables" and fill in the appropriate details as shown below



8. Now let's add the two new files using git , add a commit message and push the changes

$ git add .gitlab-ci.yml
$ git add manifest.yaml
git commit -m "add pipeline configuration"
$ git push -u origin master

9. Navigate to GitLab UI "CI/CD -> Pipelines" and we should see our pipeline starting to run








10. If everything went well!!!



11. Finally our application will be deployed to TAS4K8s as shown below
  
$ cf apps
Getting apps in org apples-org / space development as pas...
OK

name requested state instances memory disk urls
gitlab-TAS4K8s-boot-demo started 1/1 1G 1G gitlab-tas4k8s-boot-demo.apps.system.run.haas-236.pez.pivotal.io
gitlab-tas4k8s-demo started 1/1 1G 1G gitlab-tas4k8s-demo.apps.system.run.haas-236.pez.pivotal.io
test-node-app started 1/1 1G 1G test-node-app.apps.system.run.haas-236.pez.pivotal.io

$ cf app gitlab-TAS4K8s-boot-demo
Showing health and status for app gitlab-TAS4K8s-boot-demo in org apples-org / space development as pas...

name: gitlab-TAS4K8s-boot-demo
requested state: started
isolation segment: placeholder
routes: gitlab-tas4k8s-boot-demo.apps.system.run.haas-236.pez.pivotal.io
last uploaded: Tue 16 Jun 11:29:03 AEST 2020
stack:
buildpacks:

type: web
instances: 1/1
memory usage: 1024M
state since cpu memory disk details
#0 running 2020-06-16T01:29:16Z 0.1% 118.2M of 1G 0 of 1G

12. Access it as follows.

$ http http://gitlab-tas4k8s-boot-demo.apps.system.run.haas-236.pez.pivotal.io
HTTP/1.1 200 OK
content-length: 28
content-type: text/plain;charset=UTF-8
date: Tue, 16 Jun 2020 01:35:28 GMT
server: istio-envoy
x-envoy-upstream-service-time: 198

Tue Jun 16 01:35:28 GMT 2020

Of course if you wanted to create an API like service you could use the source code at this repo rather then the simple demo shown here using OpenAPI.

https://github.com/papicella/spring-book-service



More Information

Download TAS4K8s
https://network.pivotal.io/products/tas-for-kubernetes/

GitLab
https://about.gitlab.com/
Categories: Fusion Middleware

Installing a UI for Tanzu Application Service for Kubernetes

Thu, 2020-06-04 23:18
Having installed Tanzu Application Service for Kubernetes a few times having a UI is something I must have. In this post I show how to get Stratos deployed and running on Tanzu Application Service for Kubernetes (TAS4K8s) beta 0.2.0.

Steps

Note: It's assumed you have TAS4K8s deployed and running as per the output of "kapp" 

$ kapp list
Target cluster 'https://lemons.run.haas-236.pez.pivotal.io:8443' (nodes: a51852ac-e449-40ad-bde7-1beb18340854, 5+)

Apps in namespace 'default'

Name  Namespaces                                    Lcs   Lca
cf    (cluster),build-service,cf-blobstore,cf-db,   true  2h
      cf-system,cf-workloads,cf-workloads-staging,
      istio-system,kpack,metacontroller

Lcs: Last Change Successful
Lca: Last Change Age

1 apps

Succeeded

1. First let's create a namespace to install Stratos into.

$ kubectl create namespace console
namespace/console created

2. Using helm 3 install Stratos as shown below.

$ helm install my-console --namespace=console stratos/console --set console.service.type=LoadBalancer
NAME: my-console
LAST DEPLOYED: Fri Jun  5 13:18:22 2020
NAMESPACE: console
STATUS: deployed
REVISION: 1
TEST SUITE: None

3. You can verify it installed correctly a few ways as shown below

- Check using "helm ls -A"
$ helm ls -A
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
my-console console 1 2020-06-05 13:18:22.785689 +1000 AEST deployed console-3.2.1 3.2.1
- Check everything in the namespace "console" is up and running
$ kubectl get all -n console
NAME READY STATUS RESTARTS AGE
pod/stratos-0 2/2 Running 0 34m
pod/stratos-config-init-1-mxqbw 0/1 Completed 0 34m
pod/stratos-db-7fc9b7b6b7-sp4lf 1/1 Running 0 34m

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/my-console-mariadb ClusterIP 10.100.200.65 <none> 3306/TCP 34m
service/my-console-ui-ext LoadBalancer 10.100.200.216 10.195.75.164 443:32286/TCP 34m

NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/stratos-db 1/1 1 1 34m

NAME DESIRED CURRENT READY AGE
replicaset.apps/stratos-db-7fc9b7b6b7 1 1 1 34m

NAME READY AGE
statefulset.apps/stratos 1/1 34m

NAME COMPLETIONS DURATION AGE
job.batch/stratos-config-init-1 1/1 28s 34m
4. To invoke the UI run a script as follows.

Script:

export IP=`kubectl -n console get service my-console-ui-ext -ojsonpath='{.status.loadBalancer.ingress[0].ip}'`

echo ""
echo "Stratos URL: https://$IP:443"
echo ""

Output:

$ ./get-stratos-url.sh

Stratos URL: https://10.195.75.164:443

5. Invoking the URL above will take you to a screen as follows where you would select "Local Admin" account



6. Set a password and click "Finish" button


7. At this point we need to get an API endpoint for our TAS4K8s install. Easiest way to get that is to run a command as follows when logged in using the CF CLI as follows

$ cf api
api endpoint:   https://api.system.run.haas-236.pez.pivotal.io
api version:    2.150.0

8. Click on the "Register an Endpoint" + button as shown below


9. Select "Cloud Foundry" as the type you wish to register.

10. Enter details as shown below and click on "Register" button.
 


11. At this point you should connect to Cloud Foundry using your admin credentials for the TAS4K8s instance as shown below.


12. Once connected your good to go and start deploying some applications. 




Categories: Fusion Middleware

Targeting specific namespaces with kubectl

Mon, 2020-06-01 00:45
Note for myself given kubectl does not allow multiple namespaces as per it's CLI

$ eval 'kubectl  --namespace='{cf-system,kpack,istio-system}' get pod;'

OR (get all) if you want to see all resources

$ eval 'kubectl  --namespace='{cf-system,kpack,istio-system}' get all;'
  
$ eval 'kubectl --namespace='{cf-system,kpack,istio-system}' get pod;'
NAME READY STATUS RESTARTS AGE
ccdb-migrate-995n7 0/2 Completed 1 3d23h
cf-api-clock-7595b76c78-94trp 2/2 Running 2 3d23h
cf-api-deployment-updater-758f646489-k5498 2/2 Running 2 3d23h
cf-api-kpack-watcher-6fb8f7b4bf-xh2mg 2/2 Running 0 3d23h
cf-api-server-5dc58fb9d-8d2nc 5/5 Running 5 3d23h
cf-api-server-5dc58fb9d-ghwkn 5/5 Running 4 3d23h
cf-api-worker-7fffdbcdc7-fqpnc 2/2 Running 2 3d23h
cfroutesync-75dff99567-kc8qt 2/2 Running 0 3d23h
eirini-5cddc6d89b-57dgc 2/2 Running 0 3d23h
fluentd-4fsp8 2/2 Running 2 3d23h
fluentd-5vfnv 2/2 Running 1 3d23h
fluentd-gq2kr 2/2 Running 2 3d23h
fluentd-hnjgm 2/2 Running 2 3d23h
fluentd-j6d5n 2/2 Running 1 3d23h
fluentd-wbzcj 2/2 Running 2 3d23h
log-cache-7fd48cd767-fj9k8 5/5 Running 5 3d23h
metric-proxy-695797b958-j7tns 2/2 Running 0 3d23h
uaa-67bd4bfb7d-v72v6 2/2 Running 2 3d23h
NAME READY STATUS RESTARTS AGE
kpack-controller-595b8c5fd-x4kgf 1/1 Running 0 3d23h
kpack-webhook-6fdffdf676-g8v9q 1/1 Running 0 3d23h
NAME READY STATUS RESTARTS AGE
istio-citadel-589c85d7dc-677fz 1/1 Running 0 3d23h
istio-galley-6c7b88477-fk9km 2/2 Running 0 3d23h
istio-ingressgateway-25g8s 2/2 Running 0 3d23h
istio-ingressgateway-49txj 2/2 Running 0 3d23h
istio-ingressgateway-9qsqj 2/2 Running 0 3d23h
istio-ingressgateway-dlbcr 2/2 Running 0 3d23h
istio-ingressgateway-jdn42 2/2 Running 0 3d23h
istio-ingressgateway-jnx2m 2/2 Running 0 3d23h
istio-pilot-767fc6d466-8bzt8 2/2 Running 0 3d23h
istio-policy-66f4f99b44-qhw92 2/2 Running 1 3d23h
istio-sidecar-injector-6985796b87-2hvxw 1/1 Running 0 3d23h
istio-telemetry-d6599c76f-ps6xd 2/2 Running 1 3d23h
Categories: Fusion Middleware

Pages