Kubernetes FAQ

Recently I’ve been working a lot with kubernetes at my job. I’ve written up a bunch of internal documentation based on my work, and because it’s not company-specific I decided I’m going to share it here for anyone trying to find answers on the internet.

Table of Contents:

  1. what is k8s?
  2. view nodes in a nodegroup
  1. view pods on a node
  2. restart deployment
  3. insufficient cpu, insufficient memory errors
  4. deployment is not ready
  5. CrashLoopBackoff
  6. pod didn’t trigger scale-up
  7. how do I avoid accidentally running a kubectl command on the wrong cluster?

What is Kubernetes? How do I use it?

Check out kubernetes docs here: https://kubernetes.io/docs/home

View Nodes in a Nodegroup

Confirm the nodegroup you’re checking actually exists (e.g. you’re looking for pods running under engtools-infra/my-app) by listing all the nodegroups in a certain namespace:

k get nodegroup -n engtools-infra

Get the nodes under that nodegroup:

k get nodes -o wide | grep my-app

View Pods on a Node

View pods under a node with the following command (replace NODE_NAME_GOES_HERE with the node’s name, e.g. ip-10-10-12-123.ec2.internal).

k get pod --field-selector=spec.nodeName=NODE_NAME_GOES_HERE -owide --all-namespaces | grep -v -E "( kube-proxy-| kube2iam-|  local-volume-provisioner-| node-monitoring-| localusers-)"

Restart Deployment

Make sure you’re in the right cluster, get the deployment name, then use the rollout command.

k config current-context
k get deployment -n [NAMESPACE]
k rollout restart deployment [DEPLOYMENT_NAME] -n [NAMESPACE]

insufficient cpu, insufficient memory errors

If you’re seeing errors about insufficient resources, or node(s) had taint … that the pod didn’t tolerate warnings, you probably need to define instanceType and resources attributes in your service k8s objects / charts. For example, something like the following:

  instanceType: c5.2xlarge
      cpu: "3"
      memory: "4G"
      cpu: "4"
      memory: "6G"

Deployment is not ready

A log you might see when deploying with Bazel is Deployment is not ready: <service>. 0 out of 2 expected pods are ready. This usually means k8s pods are in a CLB or some other non-ready status.

Use kubectl commands to check on the pods’ status and logs. Usual status check command:

kubectl -n NAMESPACE get pods


If you see CrashLoopBackOff status on one of your pods, that means it’s crashing (I know, thanks captain obvious).

Check the pod logs: k logs CRASHING_POD -n NAMESPACE.

Find the deployment.yaml file describing the launch instructions for this pod. Find the command attribute in that file, comment out the value underneath it, and add the following:

          - tail
          - "-f"
          - /dev/null

Then re-deploy the pod. This will allow you to SSH into the pod, run the failing command manually, and generally get more info about the problem.

You might also need to comment out the liveness/readiness container specs in deployment.yaml, as those will often cause the pod to crash as well if there’s an issue with the primary container.

More info about troubleshooting crashloopbackoff pods can be found in this releasehub.com article.

pod didn’t trigger scale-up

Normal   NotTriggerScaleUp    19m                  cluster-autoscaler   pod didn’t trigger scale-up: 3 node(s) had taint {node: database-rev-eng-8c38-pool1}, that the pod didn’t tolerate…

Potentially a node group or a scaling group issue. For example, you might have a single AutoScalingGroup (asg) under your nodegroup-controller which has reached its max size. In this case, adding autoscaling: true option to your charts may be required in order to have more ASGs created.

If adding nodegroup options, you may need to delete the existing nodegroups (krm ng NODEGROUP_NAME -n NAMESPACE) for the option to take effect. After creating the new nodegroup, the cluster autoscaler may need to pass by, which can take a few minutes. So if you’re still seeing the scale-up messages and your pod in a Pending status after such a change you may need to give it a little time.

How do I avoid accidentally running a kubectl command on the wrong cluster?

Use Explicit Context in Commands

kubectl –context cluster3.us1.test.com get pods -n mynamespace  


kube-ps1 will show you what cluster and namespace you’ve set in your prompt, which helps avoid this issue:

After installing it with homebrew, add this to your .rc file and restart your terminal to enable it:

source /usr/local/opt/kube-ps1/share/kube-ps1.sh

It works great in conjunction with using kubectx and kubens to change your context/cluster/namespace.

It’s very easy to change contexts in a different terminal and have a stale prompt showing the old context. Press enter to refresh the prompt.


If you use powerlevel10k, it can dynamically update your prompt with the current context when you’re typing a kubectl command – https://github.com/romkatv/powerlevel10k#show-on-command.


‘Clear Formatting’ Shortcut Fix

Recently my ‘clear formatting’ shortcut for most google apps (Command (or Cmd) ⌘ + \) stopped working on my Macbook.

This was very frustrating as I use it all the time.

Apparently the 1password took over this shortcut since it uses cmd+\ to fill logins, silently. I was able to change the shortcut in 1password preferences, at which point clear formatting started working again.

In the future, if you have a similar issue, there’s also an app called ShortcutDetective that can help you find out if a keyboard shortcut is being used for an app on your Mac.

View Pods in a Nodegroup

Confirm the nodegroup you’re checking actually exists (e.g. you’re looking for pods running under product-team/fubar) by listing all the nodegroups in a certain namespace:

get nodegroup -n product-team

Get the nodes under that nodegroup:

get nodes -o wide | grep fubar

Get the pods under those nodes (replace NODE_NAME_GOES_HERE with the name you got from the command above, e.g. ip-10-111-22-333.ec2.internal):

k get pod --field-selector=spec.nodeName=NODE_NAME_GOES_HERE -owide --all-namespaces

Parking Near Airports in NYC

Little trick I learned from living in NYC.

There’s free street parking in neighborhoods like Ozone Park near JFK on streets that don’t have street cleaning, for example on 120th St between 133rd Ave and Sutter Ave. You can park there, and then take public transit or uber to JFK. I’ve done this at least twice now.

This is fairly low risk if you’re not going away for more than a week. The only issue is if you leave the car for too long I imagine it might be reported as abandoned and get towed.

You can do a similar sort of thing near LGA, which I’ve done MANY times now, but there are no streets without street cleaning, so you either need to make sure your trip is 6 days or less, or cough up the $65 ticket for street cleaning.

The nice thing about LGA, is that street parking is close enough to the airport that you can just walk to the terminal, or take a kick scooter and then lock it up at one of the bike racks outside the terminals.

I love not needing to spend $60-80 for airport transportation and being able to drive myself 95% of the way there. I usually walk to LGA too instead of taking an Uber after finding a spot.

Bulk Delete Projects in Jira

Had to do this recently for a lot of test projects that were created in our Jira sandbox instance.

It’s a small python script using Jira API and the requests python library.

import requests
from requests.auth import HTTPBasicAuth

domain = "https://MY_JIRA_INSTANCE.atlassian.net"
admin_user = "JIRA_ADMIN@MY_COMPANY.com"
admin_token = "JIRA_ADMIN_TOKEN"
project_query = "PROJECTS_I_WANT_TO_DELETE"

def getJson(url, auth):
    r = requests.request(
        headers={"Accept": "application/json"},
    return r.json()

def deleteProject(id, auth):
    r = requests.request(
        domain + "/rest/api/3/project/" + id,
    return r.text

search_url = domain + "/rest/api/3/project/search?query=" + project_query
auth = HTTPBasicAuth(admin_user, admin_token)
json = getJson(search_url, auth)

projectIds = []

# append results across all pages, while there's still a nextPage, to projectIds array
while "nextPage" in json:
    nextUrl = json["nextPage"]
    for searchResult in json['values']:
        # optional safety check to make sure the right project is being added to the deletion array
        # if "PROJECT_NAME_I_INTEND_TO_DELETE" in searchResult["name"]:
    print("Number of project IDs found matching the search query: " + str(len(projectIds)))
    json = getJson(nextUrl, auth)

# append a single page, or the last page of results, to projectIds array
for searchResult in json['values']:
    # optional safety check to make sure the right project is being added to the deletion array
    # if "PROJECT_NAME_I_INTEND_TO_DELETE" in searchResult["name"]:
print("Number of project IDs found matching the search query: " + str(len(projectIds)))

# delete projects in projectIds array
for index, id in enumerate(projectIds):
    print("Deleting project " + id + ". Projects remaining: " + str(len(projectIds)-index))
    print(deleteProject(id, auth))

Facebook Login Quick Start – iOS Swift 3 and Xcode 8

Here is yet another entry in my iOS developer journey in which I try to improve upon available official documentation.

Facebook’s login SDK for iOS has been around for a while and has gone through several iterations/updates. There are various tutorials online which have these issues among others:

  • Official Facebook Login SDK for Swift docs do not go over required Info.plist and AppDelegate.swift changes for their SDK to work, and appears to use old Swift 2 syntax.
  • Various online tutorials are still using no longer necessary objective-C bridging headers.
  • Facebook’s developer quick start for iOS steps only use objective-C syntax.

So as with Parse, I will combine the steps for doing a quick and easy Facebook Login setup in Swift 3.

Quick Start

  1. Create new app at https://developers.facebook.com. This is straight-forward and simply requires a unique display name.
  2. Install Facebook libraries using cocoapods. You can do this using carthage or manually, but so far I’ve found cocoapods to be the most ubiquitous and easy way. If you haven’t used Cocoapods before, my previous post on using Parse has some additional information.
    target 'XCODE-PROJECT-NAME' do
      pod 'FacebookCore'
      pod 'FacebookLogin'
      pod 'FacebookShare'
  3. Go to your new facebook app’s dashboard on https://developers.facebook.com and use “Quick Start” button under Facebook Login section to set your bundle identifier and to get the required code for your project’s Info.plist file. You can ignore/skip all the steps except 3 and 5. Make sure you get both sections for your Info.plist file in step 5.
  4. Add AppDelegate.swift methods that enable Facebook to leave and return to your app when your users try to sign in with Facebook.
    import FacebookCore
    func application(_ application: UIApplication, didFinishLaunchingWithOptions launchOptions: [UIApplicationLaunchOptionsKey: Any]?) -> Bool {
            // Override point for customization after application launch.
            SDKApplicationDelegate.shared.application(application, didFinishLaunchingWithOptions: launchOptions)
            return true
    func application(_ app: UIApplication, open url: URL, options: [UIApplicationOpenURLOptionsKey : Any] = [:]) -> Bool {
            let handled = SDKApplicationDelegate.shared.application(app, open: url, options: options)
            return handled
  5. Finally add the code for creating a facebook login button in your app.
    import FacebookLogin
    class ViewController: UIViewController {
        override func viewDidLoad() {
            // Do any additional setup after loading the view, typically from a nib.
                let loginButton = LoginButton(readPermissions: [ .publicProfile ])
                loginButton.center = view.center

You should now have an access token with public profile permissions for the user account that signed in with the facebook login button.


  • You may get the error: “OSStatus error -10814″. Don’t worry, this just means the facebook app isn’t installed in your simulator or on your device. Facebook Login API will still work.
  • [Update September 27, 2017] You might notice 3 errors (“contentDescription” etc) in your xcode project resulting from the facebook cocoapods. There’s an existing issue thread about these due to deprecated code, and it might very well be fixed by the time you’re reading this tutorial. The fix is easy, as you can simply comment out the lines that result in the errors.

Parse Quick Start for iOS Swift 3 and Xcode 8

So now that you set up a free Parse server on Heroku using my previous guide you might be wondering how to start using it in your iOS project. Checking the main Parse documentation shows a bunch of Objective-C, which isn’t helpful if you’re using Swift 3.

Here are the steps I gathered from perusing the web and some testing:

  1. Install Parse libraries (SDK/Framework).
    1. I prefer cocoapods to maintain to a single standard and since so many other frameworks use it.
    2. Create ‘Podfile‘ file in the xcode project folder with the following contents:
      source 'https://github.com/CocoaPods/Specs.git'
      platform :ios, '10.0'
      target 'PROJECT_NAME' do
        pod 'Parse'
    3. Run ‘pod install‘ in terminal in the same directory as the Podfile.
  2. Connect your Swift 3 app to Parse server.
    1. Import Parse framework by adding ‘import Parse‘ at the top of ‘AppDelegate.swift‘ file.
    2. In ‘AppDelegate.swift‘ under function ‘application(… didFinishLaunchingWithOptions …)‘ add the following code:
      let configuration = ParseClientConfiguration {
      	$0.applicationId = "YOUR_PARSE_APP_ID"
      	$0.clientKey = "YOUR_PARSE_CLIENT_KEY"
      	$0.server = "https://PARSE_SERVER_NAME.herokuapp.com/parse"
      Parse.initialize(with: configuration)
  3. Test Parse server connection.
    1. Import Parse framework by adding ‘import Parse‘ at the top of ‘ViewController.swift‘ file.
    2. In ‘ViewController.swift‘ file, add the following code under function ‘ViewDidLoad‘:
      let testObj = PFObject(className: "testObj")
      testObj["foo"] = "bar"
    3. You should see the new object in your Parse Dashboard.


If XCode can’t find the Parse module with the import command after you open the new .xcworkspace file, try the following:

  • Press Command+Option+Shift+K and then Run your app.
  • Or from the menu -> Product, press Option on your keyboard and you’ll see Clean Build Folder.

Free Parse Server on Heroku

Parse Server is a great platform for managing logins and storing data used by your apps in the cloud.

You can install and use parse server on many different cloud hosting providers, but setting up Parse Server on Heroku is one of the few ways you can try out and use it for free.

Additionally you can also install Parse Dashboard on Heroku for managing and viewing data on your Parse Server using a nice GUI.


Install Parse Server on Heroku (easy)

  1. Use Heroku’s helpful “deploy to Heroku” link for Parse: https://heroku.com/deploy?template=https://github.com/ParsePlatform/parse-server-example
  2. Type in a unique app name. You will need this in a bit. screen-shot-2016-11-04-at-10-28-34-am
  3. For config variables, change APP_ID and MASTER_KEY to your own unique values. For SERVER_URL change “yourappname” to whatever you picked for your App Name. screen-shot-2016-11-04-at-10-29-51-am
  4. Click the big Deploy button. A helpful build process will be displayed and your parse server should be ready to go in about 1 minute.

At this point you can start using your Parse Server with its standard SDKs and REST interface.

If you had any issues with the automated deployment, you can also use the manual process in the Heroku devcenter source link above. The only thing you will need to do is make sure you set the Master_Key and App_ID by going to your Config Variables under your Heroku app’s Settings tab, and adding “MASTER_KEY” and “APP_ID” variables.

Install Parse Dashboard on Heroku

  1. Install Node.js, Git and the Heroku Toolbelt on your machine.
  2. Run the following commands in your terminal/command line app. You can use whatever options you want for ‘npm init’.
    mkdir my-parse-dashboard
    cd my-parse-dashboard
    git init
    npm init
    npm install --save parse-dashboard

    This will create a git repository for the parse-dashboard code and download the required code.

  3. In my-parse-dashboard directory, create 2 text files (parse-dashboard-config.json and Procfile) to hold configuration data for Parse Dashboard.
      "apps": [
          "serverURL": "https://parse-server38.herokuapp.com/parse",
          "appId": "parseServer38",
          "masterKey": "secretMasterKeySK$%@F09292",
          "appName": "My Parse Server App"
      "users": [


    web: ./node_modules/.bin/parse-dashboard --config ./parse-dashboard-config.json --allowInsecureHTTP


  4. Commit all files to your git repository.
    git add package.json Procfile parse-dashboard-config.json
    git commit -m "Initial Commit"


  5. Push repository to Heroku.
    heroku login 
    heroku create
    git push heroku master


  6. Deploy app in free tier and open it.
    heroku ps:scale web=1
    heroku open


  7. Use the username and password from the config file to login to Parse Dashboard.

You should now be able to use a GUI to view and manage your Parse Server.

Notes and Potential Errors

  • Parse Dashboard: Application suffered an error. Check Logs.
    • Use heroku logs command in terminal to view logs.
    • This happened to me due to an error in parse-dashboard-config.json. Make sure all fields match what you configured when setting up Parse Server heroku app and look up syntax for further reference.
  • Parse Dashboard: “Server not available” for Parse Server App.
    • Occurred when I put in HTTP instead of HTTPS in the serverURL field in parse-dashboard-config.json.
  • Use the following commands after making changes in parse-dashboard-config.json.
    git add parse-dashboard-config.json
    git commit -m "updated blah blah"
    git push heroku master


Google Maps SDK and APIs using Swift 3 / Xcode 8

While trying out the Google Maps SDK on iOS and Google Maps Directions API I noticed that Google’s Getting Started code hasn’t been updated for Swift 3. Here are the changes required to get the code to work.

Google Maps SDK

  1. Follow first 3 steps (install latest xcode, install sdk, get api key) at https://developers.google.com/maps/documentation/ios-sdk/start.
  2. Create a single view application in Xcode. For viewDidLoad() function in ViewController.swift, add the following code. Note: I decided to include the API key under viewDidLoad() for simplicity.
    let camera = GMSCameraPosition.camera(withLatitude: -33.868,
          longitude:151.2086, zoom:6)
    let mapView = GMSMapView.map(withFrame: CGRect.zero, camera:camera)
    let marker = GMSMarker()
    marker.position = camera.target
    marker.snippet = "Hello World"
    marker.appearAnimation = kGMSMarkerAnimationPop
    marker.map = mapView
    self.view = mapView
  3. Run the app. You should see a map with a single marker centered over Sydney, Australia. If you click on the marker, you should see the text “Hello World” above it. If you see the marker, but the map is not visible, confirm that you have provided your API key.


Google Maps Directions API

I decided to use Alamofire iOS library to make HTTP requests to the Google Maps Directions API as it makes it much easier than the built-in Swift methods.

  1. Use Alamofire installation instructions here: https://github.com/Alamofire/Alamofire. I went with the CocoaPods method as it was what Google Maps SDK used and it worked well for me.
  2. Get an API key for Google Maps Directions API here: https://developers.google.com/maps/documentation/directions/ (about halfway down the page).
  3. Create a single view application in Xcode. For viewDidLoad() function in ViewController.swift, add the following code.
    Alamofire.request("https://maps.googleapis.com/maps/api/directions/json?" +
                "origin=Disneyland&destination=Universal+Studios+Hollywood4&" +
                { response in
                    print(response.request)  // original URL request
                    print(response.response) // HTTP URL response
                    print(response.data)     // server data
                    print(response.result)   // result of response serialization
                    if let JSON = response.result.value {
                        print("JSON: (JSON)")
  4. Run the app. You should see the waypoints and directions in the Xcode Console.


Climbing Mount Elbert

A couple of weeks ago I went to Colorado with the goal of climbing Mount Elbert. At 14,440 feet high it is the tallest mountain in Colorado.

Unfortunately I only made it to about 13,650 feet =). A combination of lack of time and mild altitude sickness (on my part) prevented my brother and I from reaching the summit. However I did get some great photos and fully intend on reaching the summit on my next attempt, whenever that is.

Video correction: 2nd tallest mountain outside of Hawaii/Alaska =).