r0b2g1t / k3s-cluster-on-oracle-cloud-infrastructure

A fully automated Kubernetes k3s cluster deployment including IaC on the Oracle Cloud Infrastructure, which is always free.

Hi there 👋

k3s-cluster-on-oracle-cloud-infrastructure's Issues

Error: shape VM.Standard.E2.1.Micro not found

I am getting the following error after updating the ubuntu image to the one for the region -> eu-frankfurt-1

module.compute.oci_core_instance.worker[1]: Creating...
module.compute.oci_core_instance.worker[0]: Creating...
╷
│ Error: 404-NotAuthorizedOrNotFound 
│ Provider version: 4.69.0, released on 2022-03-23.  
│ Service: Core Instance 
│ Error Message: shape VM.Standard.E2.1.Micro not found 
│ OPC request ID: f0e16950924de78391d594d1481FF2194F/DF250747F204905D2CE7474649978B15 
│ Suggestion: Either the resource has been deleted or service Core Instance need policy to access this resource. Policy reference: https://docs.oracle.com/en-us/iaas/Content/Identity/Reference/policyreference.htm
│ 
│ 
│   with module.compute.oci_core_instance.worker[0],
│   on compute/main.tf line 68, in resource "oci_core_instance" "worker":
│   68: resource "oci_core_instance" "worker" {
│ 
╵

This is the worker node config I have;

  worker_instance_config = {
    shape_id = "VM.Standard.E2.1.Micro"
    ocpus    = 1
    ram      = 1
    // Canonical-Ubuntu-20.04-2022.03.02-0
    source_id   = "ocid1.image.oc1.eu-frankfurt-1.aaaaaaaavcrpbwmm75t6azhxgepxah6vigiwwvruti3gj2frhuxnvhzn3e5a"
    source_type = "image"
    worker_ip_0 = "10.0.0.21"
    worker_ip_1 = "10.0.0.22"
    // release: v0.21.5-k3s2r1
    k3os_image = "https://github.com/rancher/k3os/releases/download/v0.21.5-k3s2r1/k3os-amd64.iso"
    metadata = {
      "ssh_authorized_keys" = join("\n", var.ssh_authorized_keys)
    }
  }

The two server nodes VM.Standard.A1.Flex are up and running fine.

I am stuck unfortunately and any help would be gratefully received :)

Also, are starting and destroying instances chargeable :)

It looks like the shape VM.Standard.E2.1.Micro isn't available in your region.

It looks like the shape VM.Standard.E2.1.Micro isn't available in your region.
You can check it by:
Go to Home > Governance > Limits, Quotas and Usage to see where your VM.Standard.E2.1.Micro shapes are available.

_Originally posted by @r0b2g1t in https://github.com/r0b2g1t/k3s-cluster-on-oracle-cloud-infrastructure/issues/2#issuecomment-1083209846_

For others, its available, but not in the availability domain set by the script.

Changing the AD, allowed the instance to be created.

Error: Self-referential block

Hi,

When I try to run TF plan command I get the following error:

terraform plan -out .tfplan
var.private_key_password
  Password for private key to use for signing

  Enter a value: 

module.network.data.oci_identity_availability_domain.ad: Reading...
module.network.data.oci_identity_availability_domain.ad: Read complete after 0s [id=ocid1.availabilitydomain.oc1..xxxxxx]
╷
│ Error: Self-referential block
│ 
│   on compute/main.tf line 24, in resource "oci_core_instance" "server_0":
│   24:           server_0_ip = oci_core_instance.server_0.private_ip,
│ 
│ Configuration for oci_core_instance.server_0 may not refer to itself.
╵
╷
│ Error: Self-referential block
│ 
│   on compute/main.tf line 24, in resource "oci_core_instance" "server_0":
│   24:           server_0_ip = oci_core_instance.server_0.private_ip,
│ 
│ Configuration for oci_core_instance.server_0 may not refer to itself.

I had to change the network/data.tf ad_number = 1 just because the region af-johannesburg-1 doesn't allow/have ad_number=2

Any idea on how I can fix this?

security rules and kubectl

Your security rules are not allowing kubectl to reach the cluster externally.

Also the public ip of the machine would need to be added to its allowed cert/listeners.

Havn't looked into how your creating the K3s but:
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--tls-san x.x.x.x" sh -s -
with x.x.x.x being the public IP would give you access.

or edit what k3s is serving directly:
kubectl -n kube-system edit secrets/k3s-serving

Fails to create Core Instace

I'm using root user access key and then when I try to run the script it creates the network just fine but fails to create the actual instance.

╷
│ Error: 404-NotAuthorizedOrNotFound
│ Provider version: 4.67.0, released on 2022-03-10.
│ Service: Core Instance
│ Error Message: Authorization failed or requested resource not found.
│ OPC request ID: f39808081be0c66f025ee81eb87dbf3a/37EFF46E0A38072AF67D0343D2CDAC94/4F30CD595B5C0CC47C715402A16EA37A
│ Suggestion: Either the resource has been deleted or service Core Instance need policy to access this resource. Policy reference: https://docs.oracle.com/en-us/iaas/Content/Identity/Reference/policyreference.htm
│
│
│   with module.compute.oci_core_instance.server_1,
│   on compute/main.tf line 1, in resource "oci_core_instance" "server_1":
│    1: resource "oci_core_instance" "server_1" {
│

The cluster is deployed but if you try to resolve a service from another host it failed

what I expect : doing nslookup from the worker node to a service should be oki but it is not

Fix:

sudo ufw disbale
sudo service iptables stop
sudo systemctl stop firewalld
Clear IPtables: https://github.com/k3s-io/k3s/issues/535#issuecomment-907789701
sudo netfilter-persistent save

Performance Bottleneck

What is Server Node 1?

scp rancher@<SERVER_NODE_1_PUBLIC_IP>:/etc/rancher/k3s/k3s.yaml ~/.kube/config

and what is server node 1?

The server-nodes are at the availability domain 2 (AD-2) and the agent node are created in AD-1.

Cannot you just say which processor arch is for the server and which one for the agent nudes?

Error: 404;NotAuthorizedOrNotFound - caused by hard-coded compute image IDs in the wrong region

A terraform apply throws this error:

│ Error: 404-NotAuthorizedOrNotFound, Authorization failed or requested resource not found.
│ Suggestion: Either the resource has been deleted or service Core Instance need policy to access this resource. Policy reference: https://docs.oracle.com/en-us/iaas/Content/Identity/Reference/policyreference.htm
│ Documentation: https://registry.terraform.io/providers/oracle/oci/latest/docs/resources/core_instance 
│ API Reference: https://docs.oracle.com/iaas/api/#/en/iaas/20160918/Instance/LaunchInstance 
│ Request Target: POST https://iaas.eu-amsterdam-1.oraclecloud.com/20160918/instances 
│ Provider version: 4.119.0, released on 2023-05-03.  
│ Service: Core Instance 
│ Operation Name: LaunchInstance 
│ OPC request ID: 74975954df7e642d81de039b4c702ba1/380DB4D8255D882C2E7CABB029A00BAF/93E0CE782AD76D87AF42585D88D0D369 
│ 
│ 
│   with module.compute.oci_core_instance.server_0,
│   on compute/main.tf line 1, in resource "oci_core_instance" "server_0":
│    1: resource "oci_core_instance" "server_0" {

Suggestion: instead of hard-coded into a locals section in the variables.tf file, use a data lookup using the correct region / shape.

Workaround: If anyone else is using the eu-amsterdam-1 region you can use these:

    // Canonical-Ubuntu-22.04-Minimal-aarch64-2023.04.18-0 eu-amsterdam-1
    source_id   = "ocid1.image.oc1.eu-amsterdam-1.aaaaaaaa4yojwha4rdsnagwul4ncgy45lx7q2g3pd5ru2io6rsx6pog35mfq"

    // Canonical-Ubuntu-20.04-Minimal-2023.04.19-0 eu-amsterdam-1
    source_id   = "ocid1.image.oc1.eu-amsterdam-1.aaaaaaaaml7w5cdrj2fzoa7yaaa4ynymkolvshyz3cc4rbxymk52kcwxt6ma"

If you're in another region, use the oci command to look them. You'll need your compartment ID.

oci compute image list --shape "VM.Standard.E2.1.Micro" --compartment-id ${YOUR_COMPARTMENT_ID}