Hi all,

I have a small question that I can’t Google.

I want to buy a server with a Core i5-13500 processor for my cluster a a node. The specification states that this processor has 6 performance and 8 efficient cores.

How many cores will Kubernetes see? 6 or 14?

Thanks

I have been deeply involved in this topic, and looked at multiple solutions just to see if it's something doable so I'm really curious of what do you guys think, or what ideas you have for this topic.

If i wanted to provide KaaS.. first step would be obviously to take a look at cluster api, now, let's say I have a hard requirement to use RKE2.. basically what I give to the customer in the end needs to be an RKE2 cluster same as EKS / GKE - abstracting away the control plane nodes. Sadly for RKE2 there seems to be no solution at the moment, so is it worth investing my time and doing something like this, would it be a good project? I know that similar solution exist

-K0S / Kosmotron

-Kamaji

but nothing that basically gives an RKE2 remote control plane cluster.

I've been working with Bitbucket + Helm deployment from the last year and everything was working fine, Suddenly from the past few days am recently getting this error mentioning

Unable to load data: Failed to fetch default: `git fetch origin --tags --force --prune` failed exit status 128:WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!

SHA256:46OSHA1Rmj8E8ERTC6xkNcmGOw9oFxYr0WF6zWW8l1E. Please contact your system administrator.

I have no clue on how to troubleshoot this as this was working till last week and I am sure I didnt do any changes to the repo or the cluster in the past 3 weeks, this setup was deployed by one of an ex-colleague. I did enough googling but I have no clue why it not working.

My private key is synced with the bitbucket repo, So I guess that's not the issue. No changes in CM or PODs. Anyone ever fixed this? or where to look at this?

Feel free to ask for more info, happy to provide and want to resolve this asap I can

ArgoCD Version - v2.6.10+34094a2
Helm - v3.10.3+g835b733

https://preview.redd.it/i96tw7n4nvzc1.png?width=2202&format=png&auto=webp&s=9fb7c58e952836ecb15bd7c1c6f8715fe0f2d86c

https://preview.redd.it/uwoogtt5ovzc1.png?width=642&format=png&auto=webp&s=611f0dd14c654aa2a9c9315774a2b3b0e4d2d00f

As sustainability is a big thing and serverless functions (e.g. with Wasm) is such a great concept, why is nobody doing sth about the obvious lack of K8s to handle function calls in an instant (get a request, launch a function workload, finish it)?

From what I understand all workloads have to be scheduled to a pod which is created descriptively and therefore lazily. That makes it a bad choice for instant function calls and thus solutions as Knative or Spinkube opt for pre-heating pods one or another way.

Wouldn't the obvious choice be to teach K8s a way of instant non-pod run-up-shut-down workloads to achieve real serverless functions capability?

Pretty sure there's just sth I don't know, so please help me understand or pinpoint me to according resources e. g. KEPs.

I have added a new rule about respectful use of AI-generated content. So far, we have been removing obviously LLM-generated content as spam, now we have an explicit rule and removal reason.

Hey folks,

There's been a lot of talk about VictoriaMetrics last year. Is it really worth considering a switch from Prometheus?
What are the advantages of sticking with Prometheus amidst all the buzz surrounding VictoriaMetrics? Will VictoriaMetrics remain free like Prometheus, or are there potential trade-offs to consider?

I would like some insight on that. Thank you very much.

hey folks!

i'm jr. on minikube adn trying to understand whole structure.

i have installed Minikube on Ubuntu 24.04.

After I have tried to deploy rabbitmq using Helm Chart but I got some issues..

Output like here:

kubectl describe pod rabbitmq-0 -n rabbitmq

https://preview.redd.it/4aobjbt09uzc1.png?width=1845&format=png&auto=webp&s=1575e5b2b4473fb25077fadb52b300e2c6c1c84c

kubectl get pods --all-namespaces

https://preview.redd.it/08bw0eja9uzc1.png?width=946&format=png&auto=webp&s=02e1aa7c3217f288d3b0ef8ecdcc1d334912f80b

kubectl logs rabbitmq-0 -n rabbitmq

https://preview.redd.it/sgbvz8nh9uzc1.png?width=1747&format=png&auto=webp&s=938baa09a5d82b67feccd2dfe953f6d665dfb146

Could you please help me the fix the issue ?

Thank you

I know they do they do different things nodes pods <metrics> etc but still...

The other point is why don't pods migrade to different nodes (( bigger nodes etc )) via a memory processes like vMotion rather than killing seems like things should be far more smarter than they currently are etc?

I am thinking of doing quite a bit of testing in bash or golang for helm charts.... Just wondered what was already out there that one could grab some insperation from...

I don't want to reinvent the wheel if there are projects out there, but not seen anything that looks close to what's in the back of my mind.

I have been working with K8s for a couple of years now. I wanted to create a csi-driver from scratch for TrueNas. I have used other csi-drivers at work and I was wondering how one would go about creating one from scratch. Its part of a hackathon challenge at work. I found democratic-csi which helps with TrueNas but I wasn't able to clearly get the design. Any pointers or general guidelines would help.

I don't really get the whole monolithic argument in Kubernetes, and I'm too shy to ask at this point. Every time someone explains it, I act like I know, but I'm actually vague and full of doubts.

As far as I understand, Kubernetes is the management and orchestration of containers. Containers are portable, lightweight applications that are independent of the operating system(RHEL/Suse/Windows); they share the kernel OS. Sometimes, applications can be sliced into microservices, which are small pieces of the application. Am I right at this point/stage?

Okay, is a container considered monolithic in the case of application containers, since they are basically lighter than a VM and independent of a dedicated OS? Is the monolithic argument only for microservice-type pods? Please help me understand this. Can you give me a simple example?

Hi , I have multiple microservices around 25 and they need to be deployed on around 4 server, each server having all 25 services.

Until now I had only two services , so I was using docker containers for deployment but currently I am figuring out what is the best way for my current scenario. I don't have any expert available for k8s.

It would be great help if any of you could help.

Edit: we are not using any external cloud provider , We need to host on the internal servers

Hi,

I wanted to play around with Rancher. So I setup a Rancher docker container on my computer, version v2.8.3. Then I tried using the default Linode template to create a 3 node cluster on Linode, but it seems to be stuck with the two rotating messages:

"Waiting for viable init node" and "Waiting for all etcd machines to be deleted". Nothing is created in my Linode account.

Does anyone have any experience with this template?

i configured the ingress to routes the traffic between the frontend and backend
When i open the front-end in the browser it works correctly however when i enter the same route url in new tab i get Nginx 404

Help needed!!!

https://preview.redd.it/rxfo5umhnozc1.png?width=559&format=png&auto=webp&s=7089e6f7d655e3b052f3808d4a3aba60f5932a67

I am currently in the processing of moving our airflow instances to a K8 cluster. I have been installed the latest version of the helm chart (1.13.1), which gives airflow 2.8.3. I am running this instance on a RHEL 9 server with python 3.11 (base python 3.9, using alias). However after installing airflow, I discovered the pods are running on python 3.8. This does not exist on my system, so It has to come from the helm chart. I have spent two days scrubbing the internet and have found no information on helm requiring python 3.8. I am installing my python dependencies using a dockerfile which specifies 3.11, but during the build it reverts to python 3.8. I feel like I am at my whit's end, has anyone experienced this issue?

pod python version
airflow@airflow-triggerer-0:/opt/airflow$ python --version Python 3.8.18
OS python versions (both for posterity)
[airflow@_______ ~]$ python --version Python 3.9.18
[airflow@_______ ~]$ source ~/.bashrc
[airflow@_______ ~]$ python --version Python 3.11.5

dockerfile

`FROM python:3.11 FROM apache/airflow:2.8.3-python3.11

RUN echo "if [ -f ~/.bashrc ]; then source ~/.bashrc; fi" >> ~/.bash_profile

COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt USER airflow
`

I feel like I am going crazy, I need another perspective. I have rebuild my cluster dozens of times at this point.

EDIT: It appears that my kubernetes instance got corrupted. When attempting to build the dockerfile it kept wanting to run on python 3.8, once I rebuilt the cluster is started running on 3.11 and I can actually install the packages. If you run into something similar I would attempt that first.

I am not even sure how to ask the question, i don’t have the vocabulaires figured out yet.

I am new to k8s and I’m slowly getting the hang of things good news. I like the magic of flux but some things don’t work as expected.

I defined a helm repository and helm release resources for metallb in a single file. I also defined an IPAddressPool and L2Advertisment resources in the same file.

When I commit this to my repo, flux fails to apply the changes it says something like unknown custom resource. If I however remove the address pool and l2 advertisement resources the metalb resources are applied and then if I add my IP and layer m2 resources it works.

This suggests that flux might be trying to deploy the other resources before the helm release resource.

In Terraform, there’s this concept of depends on. I’ve seen that you can use depends on between helm releases but how can I say do not deploy this k8s resources until the helm release is deployed.

If this isn’t possible what’s the way you / industry standard for handling these situations.

Thanks in advance again sorry for the question title I wasn’t even sure how to ask it.

I have a Talos Cluster with 1x control-plane node and 1x worker node. It's running Talos 1.7.1 and Kubernetes 1.30.0. I deployed a plain Cilium install with no network policies yet with the following fluxCD release:

---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: cilium
  namespace: kube-system
spec:
  interval: 5m
  chart:
    spec:
      chart: cilium
      version: ">=1.15.0"
      sourceRef:
        kind: HelmRepository
        name: cilium
        namespace: kube-system
      interval: 1m
  values:
    ipam:
      mode: kubernetes
    hubble:
      relay:
        enabled: true
      ui:
        enabled: true
    kubeProxyReplacement: true
    securityContext:
      capabilities:
        ciliumAgent:
          - CHOWN
          - KILL
          - NET_ADMIN
          - NET_RAW
          - IPC_LOCK
          - SYS_ADMIN
          - SYS_RESOURCE
          - DAC_OVERRIDE
          - FOWNER
          - SETGID
          - SETUID
        cleanCiliumState:
          - NET_ADMIN
          - SYS_ADMIN
          - SYS_RESOURCE
    cgroup:
      autoMount:
        enabled: true
      hostRoot: /sys/fs/cgroup
    k8sServiceHost: localhost
    k8sServicePort: "7445"

I installed fluent-bit also with fluxCD:

---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: fluent-bit
  namespace: kube-system
spec:
  interval: 5m
  chart:
    spec:
      chart: fluent-bit
      version: ">=0.46"
      sourceRef:
        kind: HelmRepository
        name: fluent-bit
        namespace: kube-system
      interval: 1m
  values:
    podAnnotations:
      fluentbit.io/exclude: 'true'
    extraPorts:
      - port: 12345
        containerPort: 12345
        protocol: TCP
        name: talos

    config:
      service: |
        [SERVICE]
          Flush         5
          Daemon        Off
          Log_Level     warn
          Parsers_File  custom_parsers.conf    
          HTTP_Server On
          HTTP_Listen 0.0.0.0
          HTTP_Port 2020
      inputs: |
        [INPUT]
          Name          tcp
          Listen        0.0.0.0
          Port          12345
          Format        json
          Tag           talos.*
        [INPUT]
          Name          tail
          Alias         kubernetes
          Path          /var/log/containers/*.log
          Parser        containerd
          Tag           kubernetes.*
        [INPUT]
          Name          tail
          Alias         audit
          Path          /var/log/audit/kube/*.log
          Parser        audit
          Tag           audit.*    
      filters: |
        [FILTER]
          Name                kubernetes
          Alias               kubernetes
          Match               kubernetes.*
          Kube_Tag_Prefix     kubernetes.var.log.containers.
          Use_Kubelet         Off
          Merge_Log           On
          Merge_Log_Trim      On
          Keep_Log            Off
          K8S-Logging.Parser  Off
          K8S-Logging.Exclude On
          Annotations         Off
          Labels              On
        [FILTER]
          Name          modify
          Match         kubernetes.*
          Add           source kubernetes
          Remove        logtag    
      customParsers: |
        [PARSER]
          Name          audit
          Format        json
          Time_Key      requestReceivedTimestamp
          Time_Format   %Y-%m-%dT%H:%M:%S.%L%z
        [PARSER]
          Name          containerd
          Format        regex
          Regex         ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<log>.*)$
          Time_Key      time
          Time_Format   %Y-%m-%dT%H:%M:%S.%L%z    
      outputs: |
        [OUTPUT]
          Name    stdout
          Alias   stdout
          Match   *
          Format  json_lines    
    daemonSetVolumes:
      - name: varlog
        hostPath:
          path: /var/log

    daemonSetVolumeMounts:
      - name: varlog
        mountPath: /var/log

    tolerations:
      - operator: Exists
        effect: NoSchedule

There are 2x fluent-bit pods getting scheduled. One on the worker node and one on the control-plane node. The one on the worker node gets healthy and I can see logs being gathered. The one on the control-plane node does not get healthy and after a while goes to "CrashLoopBackOff". When describing the pod I can see that the readiness probe fails with "connection refused". This seems like some sort of network issue but there are no network policies. The log output of the pod on the control-plane seems fine aswell:

fluent-bit log output on control-plane

pod that is not getting healthy

What can I do to debug this. Does anybody have any ideas?

Hello.

I'm trying to deploy istio with a cluster created with k0s.

But the following error received with helm and istioctl:

helm install istio-base istio/base -n istio-system --set defaultRevision=default

Error: INSTALLATION FAILED: Kubernetes cluster unreachable: Get "http://localhost:8080/version": dial tcp [::1]:8080: connect: connection refused

istioctl install --set profile=demo

Error: check minimum supported Kubernetes version: error getting Kubernetes version: Get "http://localhost:8080/version?timeout=5s": dial tcp [::1]:8080: connect: connection refused

Obiusly the cluster is up and running:

kubectl get nodes

NAME STATUS ROLES AGE VERSION

kube02 Ready <none> 6d22h v1.29.4+k0s

kube04 Ready <none> 6d22h v1.29.4+k0s

I don't see supported versions here:

https://istio.io/latest/docs/setup/platform-setup/

Anyone tested?

Thanks

I'm really new to this stuff so maybe i'm missing some simple stuff. I had a container and i made a pod with it. Then i also create a service. Here are the YAML files for them:

myflaskapp-pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: myflaskapp-pod
  labels:
    app: myflaskapp
spec:
  containers:
  - name: myflaskapp-container
    image: creativsrwr/myflaskapp_2
    ports:
    - containerPort: 5000

myflaskapp-service.yaml:

apiVersion: v1
kind: Service
metadata:
  name: myflaskapp-service
spec:
  selector:
    app: myflaskapp
  ports:
  - protocol: TCP
    port: 80
    targetPort: 5000
  type: NodePort

then i use $ kubectl apply -f myflaskapp.pod.yaml

$ kubectl apply -f myflaskapp.pod.yaml

now with the command kubectl get services i get the name of the service

NAME: myflaskapp-service TYPE: Nodeport CLUSTER-IP: 10.108.217.13 PORT(S): 80:31779/TCP

i can access the webpage using:

$ minikube service myflaskapp-service

which takes me to the page http://192.168.49.2:31779 where my simple app is and works fine. The problem is i'm running it on a Ubuntu VM in VMbox. I want to access this link from my host windows. But when i type this link it doesn't work. I turned on port forwarding in VMbox settings for this VM as:

but even when i visit 127.0.0.1:8080 on my windows it doesn't open it up. Can someone please tell me how to make the service available on my host machine too. I only found one thing which said to configure NAT port forwarding of VM as above.

what i did.

hey all, we're trying to identify what specific type of workloads do teams find hardest to optimize in terms of resource consumption. We've seen a few culprits so far that tend to be hard to optimize by correcting configuring request / limit values - java /spark being an example.

what workloads do you find hard to optimize when running on k8s, specially on public clouds?

​

View Poll

Hi,

I’ve started to experience a weird situation one my fairly stable k3s cluster started to have one of its node suffering from rebooting.

It happens when the node in question runs a specific workload (nextcloud) and only when I try to sync my local laptop with that nextcloud (so basically trying to pull around 20Gb of data).

Thing to note : it’s a k3s cluster with Ubuntu 22.04 and kernel 6.5 backend. Network plugin is using cilium

What I’ve investigated so far :

• ressources are fine I have plenty of memory and cpu available
• it happens on any of my agent nodes it’s not specific to one of them
• syslog/kernlog aren’t showing anything specific
• kube events only says that “kubelet was rebooted”
• pods logs are not showing anything either. It could be because they crash before the log showing anything ?

I can 100% reproduce it any time so at least any suggestion can be tested easily

Could cilium/hubble ui be struggling with too many data ? I have no more guesses

Want to remove waevenet completely from my K8s cluster and reinstall it.

My cluster details is as below

NAME      STATUS   ROLES           VERSION   INTERNAL-IP    OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME

kmaster   Ready    control-plane   v1.30.0   192.168.0.80   Ubuntu 20.04.6 LTS   5.4.0-152-generic   containerd://1.6.31

kworker   Ready    <none>          v1.30.0   192.168.0.70   Ubuntu 20.04.6 LTS   5.4.0-156-generic   containerd://1.6.31
Any suggestion would be appreciated ..

Hi Folks,
there's been a lot of hype around GitOps and declarative mgmt style and seems like lot of folks are using it with success. While I do see lot of advantages, and I think it covers 99% of update scenarios, I wonder about that 1% cases were doing changes that are more imperative by nature might be required? e.g. like an intermediate step to cleanup something, or say switch default storage, first you have to patch to remove default mark from the old storage class then you can patch new storage class as default, basically the change can't be just declared, or the order is important and etc. Another example that I recall - we had once like milion of admission reports from Kyverno that slowed down ETCD on EKS, and the solution was to run CICD pipeline to clean it up on all clusters and such changes couldn't be just declared in yaml. With CICD I was able to do anything that was ever needed to run, starting with even simple things like run kctl get -A ... across all clusters to find something that we were not able to track via metrics/logs. And of course, with CICD we were still able to do 99% declarative style updates like: kctl apply -k <dir> / or Helm deploy or Terraform for the infra part and all that jazz.

Wonder how folks who do 100% GitOps only, handle these situations or is it that you use both - purpose build CICD pipeline for infra and corner/one-off cases and GitOps only for ordinary yaml deployments?
Or are you using Helm charts with some extra scripts hooks for pre/post deployments?
Are GitOps controllers allowing you to run extra commands "pre/post apply" yaml sync?
Or maybe most of the time you build new clean clusters and then test and cut over traffic to the new clusters?
Or is that GitOps tools like say ArgoCD is just for Devs to do deployments for their apps, give them some nice visiblity and controls of rollbacks and etc., but the platform level stuff & infra are still done via plain old CICD system?

Wonder about this as running clusters for more than a 1-2y you will definitely run into some situations, in which doing just an apply of simple yaml change from repo won't be enough?
Appreciate feedback, thanks!

Hello,

we are administering onprem Kubernetes,

Kustomize Version: v4.5.4

Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.17", GitCommit:"22a9682c8fe855c321be75c5faacde343f909b04", GitTreeState:"clean", BuildDate:"2023-08-23T23:37:25Z", GoVersion:"go1.20.7", Compiler:"gc", Platform:"linux/amd64"}

I want to list all of the flags which are set for etcd, especially the
--etcd-compaction-interval duration

Can you give me an example how to achieve this?

"Hey Kubernetes community! I just published a detailed guide on deploying applications on Kubernetes AKS via terminal. Dive into the world of container orchestration with Azure Kubernetes Service! #Kubernetes #AKS #DevOps"

Include the link to your blog post: Deploying Applications on Kubernetes AKS via Terminal

Greetings to all of you!

I have faith in the power of the Reddit community.

I am writing to request assistance with configuring dynamically provisioned volumes for bare metal Kubernetes (K8s) cluster. Static provisioning works very well. Here is CSI driver: https://github.com/DDNStorage/exa-csi-driver/tree/master?tab=readme-ov-file

Could someone please explain how to allow privileged pods for the kubelet? I have figured out how to do it for the API server, but not for the kubelet.

Here are the logs from the CSI controller:
/var/lib/kubelet/plugins_registry/exa.csi.ddn.com/csi.sock, err: rpc error: code = Unimplemented desc = unknown service pluginregistration.Registration
node4 kubelet[620522]: I0510 15:41:35.629446 620522 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin=

Thanks in advance for your help.

Hello :)

I would like to know the best way to handle Helm charts in Flux that have certain dependencies. Flux installs the charts in a mixed up way and if there are dependencies that should be there before a chart, this is not taken into account at all.

How do you do that? Are there any best practices I could follow?

Thanks in advance! :)