I've been experiencing this issue recently, when I enabled bpf.masquerade, in the attempt to eliminate completely the iptables usage with cilium. I was wondering if anyone could give me some pointers where is the problem? Thank you for taking the time to look into it.

Found a guide on AWS best practices, and it’s actually really helpful. It’s full of little tips that don’t get mentioned much but make a lot of sense for anyone starting out. Felt like a good find, so I’m sharing it here!

microk8s seems to be easy, everything works out of the box - until you need something a little more specific.

I have setup my k8s in microk8s and now in the process of switching to ceph. I got it up and running with cephadm, which worked like charm. Now it's time to link the two. microk8s enable rook-ceph seemed like the right choice. microk8s connect-external-ceph just worked, the cluster is there in the namespace rook-ceph-external.

main@node01:~$ kubectl get cephcluster -n rook-ceph-external
NAME                 DATADIRHOSTPATH   MONCOUNT   AGE     PHASE   MESSAGE   HEALTH   EXTERNAL   FSID
rook-ceph-external   /var/lib/rook     3          3h55m                              true    

Time to create a static PV as described in the docs. Problem though: wheer are the csidrivers?

main@node01:~$ kubectl get csidriver --all-namespaces
No resources found

Any ideas here? Does microk8s not come with a fully functional rook-ceph?

The last couple of months I was building a platform that uncovers misconfigurations and best practice violations in your K8s cluster.

I'd be really happy if you'd check out the page and let me know what you think of the idea.

Would you use it? If not, what are road-blockers for you? Which questions are unanswered on the landing page? Any kind of feedback is highly appreciated.

I am also looking for people who would like to register for early, so I can get a bit of feedback on the platform itself and new ideas for features to implement.

On the page, it is promised that the agent running in the cluster will be open source - and I intend to keep that promise. For now the repo is still private, since I don't feel the code is ready to be public (yet). It is written in golang. If you are proficient with go, ideally with experience using the k8s API, and you would like to contribute to the project, I'd be happy. Let me know.

Thanks a lot in advance! Hope you like it:)

We have a prometheus instance which is running on some VM and already scrapes for example metrics from databases and kafka.

Is this viable to use the same instance to somehow scrape metrics from k8s cluster (both nodes and pods running in the cluster) or we should rather set up prometheus instance in the cluster and configure federation if needed to store metrics in single place? Keep in mind there is a Grafana with dashboards already in-place which is integrated with external prometheus instance.

Hey everyone,

I have a few clusters, all setup with Karpenter (SpotToSpot consolidation enabled), and I feel like the node utilization is quite low (30% Memory, 20% CPU).

What are your tricks to have a better cost efficiency? Basically with such utilization, it appears that I could run with nearly half the nodes.

I do use Karpenter, and have limits setup for all my nodes, along with spotToSpotConsolidation enabled.

Cheers

Is there any way to control things like `/etc/network/interfaces.d` with K3s? I have edge devices where I need them to communicate with each other over WiFi. In my KCAD training, they didn't really cover this.

Did you learn something new this week? Share here!

I'm creating a simple social application.

I plan to commercialize this project, but it is currently in the early design stage. The structure I came up with is to initially set up docker + jenkins, create a simple pipeline, and build it right away. I have never used K8s yet. Would it be a good idea to set it up in advance?

I read the official doc about clusterip, and read some materials about k8s network. as far as i know, if I establish a conn with cllusterip service, I will get a socket like (my_ip, my_port, cluster_ip, svc_port), and the net packages will be routed to a pod behind the clusterip service.

what I really want to figure out is: how the routing make it to route the package of the conn always to the same pod, there should be some mapping like socket |-> pod_addr.

Is it achieved by iptable? or user space code? I'd be grateful if someone could give some snippets or hints about where to start.

Running EKS with EBS volumes for PVs spanning over 3 AZs. The question is, particularly with spot instances that can go down frequently, what do you do to ensure the pod gets scheduled on a node in the same AZ as the PV. I know of a few options, but I wanted to see if there were other alternatives:

• Use EFS instead (Not something I am looking to do)
• Use Longhorn (Haven't looked much into this tool, but not against it)
• Specify node selectors for each deployment (I'd rather have a solution more dynamic than this)

Also, is there possibly something with Cluster Autoscaler or Karpenter that could assist with this sort of thing as I haven't found anything yet.

Hello. I am looking at using flux to deploy multiple various applications on one cluster.

But looking at all the documentation and articles online, and even searching reddit, I cannot find anyone using this?

Everything is about multiple clusters or multiple environments.

the only method I suspect will work is storing kubernetes configs in the application repository together with the code?

What if I want to have one gitops repository, which will contain 2 clusters (environments) and 5 different applications in their own namespaces?

Please advise, thanks.

I have a situation which I presume is incredibly common, but don't yet have the terminology in my arsenal to figure out what kind of solution to look for.

We have kubernetes clusters in various regions (say X, Y Z) around the US. Each of these clusters run the same applications (deployments), and these applications all communicate with a centralized database, which is effectively a cache. We currently have one centralized database instance (outside of kubernetes, say in region W). We notice that (amongst other obvious issues) the latency to this database varies significantly across regions. Our plan to combat this is to split the database in region W into individual instances in each region (X, Y and Z), and apps running in region X will write to the db in region X. Because of the nature of our application, we should see the behavior that an application running in region X should be able to find _most_ of the data it needs from the database instance in region X. However, it could be the odd case that an application needs to fetch some data in region Y, for example, so we need to be able to support this.

My immediate thought is to leave the database in region W. Instead of communicating directly with the database, our apps will now hit a proxy. The proxy will write data to both the regional db and the centralized db (in region W) (perhaps this is a write-behind cache). When an app instance needs to fetch data, it will hit a proxy instance (can have these proxies set up as a deployment in each regional cluster). The proxy will first check the regional db, and if the data is not present, it will fall back to the centralized db.

How does this sound? Any major issues immediately present with this approach? Any out-of-the-box third party solution or design pattern which covers exactly this scenario? Any kind of information helps, always looking to expand the toolset available to me! (btw the db is redis, so any suggestions/gotchas on running redis in k8s are also welcome).

https://kubernetespodcast.com/episode/241-65k-gke/
https://cloud.google.com/blog/products/containers-kubernetes/gke-65k-nodes-and-counting?e=48754805

Hey r/kubernetes community!

I’m currently working on organizing namespaces in our Kubernetes environment and wanted to get your thoughts on best practices for naming conventions. I know a good naming strategy can make a big difference in managing resources effectively, especially as our setup grows and becomes more complex.

and how about annotations

Thanks :)

Hi everyone here.

I'm trying to deploy some sort of "stack" to allow users to log into a k8s cluster. This is an EKS, deployed obiously in AWS.

Currently, i'm deploying:

dex (https://github.com/helm/charts/tree/master/stable/dex)

dex-k8s-authenticator (https://github.com/mintel/dex-k8s-authenticator)

kube-oidc-proxy, (https://github.com/jetstack/kube-oidc-proxy/tree/master/deploy/charts/kube-oidc-proxy)

oauth2-proxy (https://github.com/oauth2-proxy/oauth2-proxy)

What i'm trying is to use DEX to connect to our LDAP, dex-k8s-authenticator to login users and use token inside kubectl (CLI), oauth2-proxy to login users and authenticate into kubernetes-dashboard.

kube-oidc-proxy is used as a proxy between apicalls to kube-apiserver.

Anyone has as BETTER approach of this? Is driving me crazy because, it sometimes works from CLI, sometimes fails. And it never works from oauth2-proxy because it says that token/user is already claimed and can't be refreshed.

This is a nice blog by Artem Lajko on how to use GitOps plus event framework to create services on demand.

The image is taken from the blog

https://i.redd.it/m8u5ir4hqo0e1.gif

Here is the link to the blog. Its on Medium though

Looking for any information since I'm at the conference and hoping to connect with others / hear about cool things to do in the evenigs.

Thanks.

Hi all,

Consider on prem 3 node k8s cluster. HA via kube-vip. Backed up etcd db and /etc/kbuernetes/pki from original master1. This has only 1 worker node. For testing I deleted all 3 control plane nodes (this is not produciton...this is for testing).

Recreted 3 to be master nodes. On new master1 node - I ran

mkdir -p /etc/kubernetes/pki
cp -r /path/to/backup/pki/* /etc/kubernetes/pki/

And after that I generate kube-vip yaml for kbue-VIP pod. And run init command.

kubeadm init --control-plane-endpoint="10.2.0.10" --upload-certs --pod-network-cidr=10.244.0.0/16 --ignore-preflight-errors=all

.10 is VIP of existing cluster. Still using same CIDR for PODs

And then

kubelet stop
mv /var/lib/etcd /var/lib/etcd.bak

ETCDCTL_API=3 etcdctl snapshot restore <etcd-backup-path> --data-dir /var/lib/etcd
systemctl start kubelet

After some time I could see all nodes are back in kubectl (obvisoly only master1 is present). I generated token again on master 1 and generatd join command for remaining masters

kubeadm init phase upload-certs --upload-certs

kubeadm token create --ttl 0 --certificate-key fc9075792e6be187a43a3f2eba05e75bbffa5981630446ad0bad22cc91dd268f --print-join-command --v=5

I get join command. I ran it on master 2. could not find a JWS signature in the cluster-info ConfigMap for token ID. So its failing

I1113 10:42:11.988587    8558 checks.go:128] validating if the "kubelet" service is enabled and active
I1113 10:42:12.001779    8558 checks.go:201] validating availability of port 10250
I1113 10:42:12.001974    8558 checks.go:428] validating if the connectivity type is via proxy or direct
I1113 10:42:12.002024    8558 checks.go:327] validating the contents of file /proc/sys/net/ipv4/ip_forward
I1113 10:42:12.002074    8558 join.go:536] [preflight] Discovering cluster-info
I1113 10:42:12.002106    8558 token.go:79] [discovery] Created cluster-info discovery client, requesting info from "10.2.0.10:6443"
I1113 10:42:12.002689    8558 token.go:210] [discovery] Waiting for the cluster-info ConfigMap to receive a JWS signaturefor token ID "2j3nar"
I1113 10:42:12.013560    8558 token.go:228] [discovery] Retrying due to error: could not find a JWS signature in the cluster-info ConfigMap for token ID "2j3nar"

I check cluster-info CM in kube-public NS. Sure it doesn't have that token ID. Its showing some old token IDs which doesn't exist. Tried generating new tokens, still don't see any of these new token IDs in cluster-info.

I tried deleting and recreating this CM, still no luck.

Out of curiosity - I rerun the kubeadmin init command again and this time there is no whatsoever token ID present in that CM. This is bit strange. Is it becasue I am using old certs/keys form old pki folder?

Are my steps for recovery are correct one? Can someone give any more hints. I did kubeadm reset and master 1 and reinitited the cluster (w/o using pki files or etcdb) and I could see the entry of token ID in that CM.

Hi,

I am trying to use downward API to capture some annotations and making them available as env var as documented here https://kubernetes.io/docs/tasks/inject-data-application/environment-variable-expose-pod-information/

Let's take an example:

spec:
  containers:
  - env:
    - name: NODE_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: spec.nodeName
    - name: POD_IP
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.annotations['cni.projectcalico.org/podIP']

So I would expect the POD_IP to be available. It is not.

NODE_NAME=i-0e6e718f3ec50e4de
POD_IP=

Can anyone see what is wrong here? And of course these annotations exist

Annotations:  cni.projectcalico.org/containerID: blahblahblah
                      cni.projectcalico.org/podIP: 100.124.112.124/28
                      cni.projectcalico.org/podIPs: 100.124.112.124/28

EDIT: Formatting is slightly screwed

Hi everyone, First of all, here's the link.

https://chatgpt.com/g/g-JPmhB0YiR-kubernetes-cluster-installer

I wanted to share this GPT that I've been working on over the past few days. I'm starting a Master's in AI and Innovation, and one of the Prompt Engineering exercises was to create a commonly used GPT. Without realizing it, I ended up creating this GPT based on a need I often have when testing things in my home lab or controlled work environments.

This idea came up little by little, and I decided to share it to see if anyone else might be interested in trying it out.

What does this GPT do?

This GPT generates several scripts that will help you quickly deploy, on one or more Ubuntu 22.04 LTS systems or Debian-based distributions, a Kubernetes cluster customized to your needs. You decide how many nodes to use (you can configure 2 master nodes, 3 workers, or 1 master and 2 workers, for example).

If you have more than one master node, the GPT will also generate a script to deploy a load balancer.

Why share it?

This GPT has been useful to me, and I thought it could be helpful to others, so I decided to share it. Of course, there are other automation tools that can help you deploy a cluster, but this solution is quite easy to use as long as you have a basic understanding of how to run a script in Linux. If not, ChatGPT can surely help you ;)

https://preview.redd.it/2dkldoxbtn0e1.png?width=792&format=png&auto=webp&s=8ff54c5fba88bd36dbbc93de78ff35ed5a599318

I want to implement a simple GitOps tool like ArgoCD/Flux.

Applying is easy. Prune is hard.

What is prune in this context?

Yesterday I applied 10 manifests.

We decided that one manifests is not needed any more.

Today I apply 9 manifests.

A normal kubectl call would not delete the manifest we no longer need. You need "--prune".

We don't use kubectl, we use Go.

Afaik there is no way to do prune with client-go.

Looking at ApplySet : kubectl apply --prune redesign and graduation strategy I understand that this task is not trivial.

I am unsure. Should I implement that myself or use a library?

I found that: https://github.com/argoproj/gitops-engine. It started as a joint effort with Flux, but afaik Flux does not use that package anymore.

What is your recommendation if you want to reconcile Kubernetes resources via Go?

Where is Kubernetes python package documentation? If anyone is having link for that, please send me. I need to work with some kubernetes api.