nais / aivia

Simple app to mirror a topic from on-prem kafka to Aiven kafka.

AiviA

Simple app to mirror a topic from on-prem kafka to Aiven kafka. It does not support Schema Registry.

!!! This application is intended as a tool to help migrate away from the on-prem Kafka cluster, which is deprecated. !!!

How does it work?

First you need to create a mapping of topics to mirror. The app will read this from a properties-file, mapping a source topic to a destination topic.

Example:

aapen-data-v2 = nais.data-v2
privat-hemmelig-saa-det-saa = nais.hemmelig-v2

With this configuration messages will be copied from aapen-data-v2 on-prem to nais.data-v2 on Aiven, and from privat-hemmelig-saa-det-saa on-prem to nais.hemmelig-v2 on Aiven.

The application will read the file from the path /var/run/configmaps/aivia-topic-mapping/topic_mapping.properties. If you want something else, you can use the environment variable AIVIA_TOPIC_MAPPING_PATH.

If you want to use AiviA to mirror some other direction than on-prem to Aiven, you also need to configure source and target clusters using the environment variables AIVIA_SOURCE and AIVIA_TARGET respectively. Valid values are on-prem and aiven. The default is on-prem for source and aiven for target.

It is possible to mirror any combination of AIVIA_SOURCE and AIVIA_TARGET (even aiven to aiven).

Using AiviA

1. Create a new repository with a nais.yaml file:

apiVersion: "nais.io/v1alpha1"
kind: "Application"
metadata:
  name: aivia
  namespace: myteam
  labels:
    team: myteam
spec:
  image: ghcr.io/nais/aivia:latest
  liveness:
    path: "/internal/isalive"
  readiness:
    path: "/internal/isready"
  replicas:
    min: 1
    max: 1
    cpuThresholdPercentage: 50
  prometheus:
    enabled: true
    path: "/internal/prometheus"
  limits:
    cpu: "200m"
    memory: "256Mi"
  requests:
    cpu: "200m"
    memory: "256Mi"
  env:
    - name: LOG_FORMAT
      value: logstash
  envFrom:
    - secret: aivia-kafka-on-prem
  filesFrom:
    - configmap: aivia-topic-mapping
  kafka:
    pool: nav-dev

It is best to use a specific version, which you can do by getting the latest aivia image from the AiviA package page. If you do so, you must make sure to keep updated with latest versions.

2. Create a configmap with your topic mappings

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    team: myteam
  annotations:
    reloader.stakater.com/match: "true"
  name: aivia-topic-mapping
  namespace: myteam
data:
  topic_mapping.properties: |
    aapen-data-v2 = myteam.data-v2
    privat-hemmelig-saa-det-saa = myteam.hemmelig-v2

3. Create a workflow to deploy the application and configmap

name: "Deploy aivia"
on:
  push:
    branches:
    - "main"
jobs:
  deploy:
    name: "Deploy AiviA"
    runs-on: ubuntu-latest
    steps:
      - uses: "actions/checkout@v3"
      - uses: nais/deploy/actions/deploy@v1
        env:
          APIKEY: ${{ secrets.NAIS_DEPLOY_APIKEY }}
          CLUSTER: dev-gcp
          RESOURCE: configmap.yaml,nais.yaml

4. Create a secret with configuration for on-prem kafka, and apply it to the cluster

apiVersion: v1
kind: Secret
metadata:
  name: aivia-kafka-on-prem
  namespace: myteam
stringData:
  KAFKA_ON_PREM_BROKERS: "b27apvl00045.preprod.local:8443,b27apvl00046.preprod.local:8443,b27apvl00047.preprod.local:8443"
  KAFKA_ON_PREM_USERNAME: "myServiceUser"
  KAFKA_ON_PREM_PASSWORD: "myServiceUsersPassword"

Run kubectl apply -f secret.yaml to insert the secret into the cluster. Do not commit secret.yaml to the repository.

5. Commit and push

Commit nais.yaml, configmap.yaml and .github/workflows/main.yaml and push to github.

What does this do?

AiviA guarantees the following:

• At-least-once delivery of all messages present in the source topic.
• Preservation of message ordering if using the default partitioning strategy.
• Propagation of keys present in messages from the source topic.

AiviA does not guarantee the following:

• Prompt delivery of messages.
• Preservation of offsets. Your application should ideally handle messages idempotently.

Verifying the aivia image and its contents

The image is signed "keylessly" using Sigstore cosign. To verify its authenticity run

cosign verify \
--certificate-identity "https://github.com/nais/aivia/.github/workflows/main.yaml@refs/heads/main" \
--certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
ghcr.io/nais/aivia@sha256:<shasum>

The images are also attested with SBOMs in the CycloneDX format. You can verify these by running

cosign verify-attestation --type cyclonedx \
--certificate-identity "https://github.com/nais/aivia/.github/workflows/main.yaml@refs/heads/main" \
--certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
ghcr.io/nais/aivia@sha256:<shasum>

aivia's Issues

Skrive i README og i about-feltet til repoet at Aivia kan brukes Aiven -> Aiven i tillegg til onprem -> Aiven

I vår app https://github.com/navikt/rekrutteringsbistand-stilling-kafkabro bruker vi Aivia til å viersende fra en Aivien topic til en annen Aivien topic. Vi lurte på om det var mulig før vi startet, for i følge README og about-feltet i repoet til Aivia kan Aivia tilsynelatende bare brukes fra onprem topic til Aiven topic.

ACTION REQUIRED: Changes to pulling Chainguard Images

Hey there Chainguard here.

We noticed that you are using Chainguard Images, thank you! We wanted to make you aware of an upcoming change that will impact your project.

Starting August 16, 2023 public users will no longer be able to pull images from our registry (cgr.dev/chainguard) by tags other than latest or latest-dev. Please see the announcement for more information.

You are currently using the following.

In https://github.com/nais/aivia/blob/0db0c735a85e3273b6d6c9f4781058ab4597b73d/.github/workflows/main.yaml:

• cgr.dev/chainguard/jre:openjdk-

In https://github.com/nais/aivia/blob/0db0c735a85e3273b6d6c9f4781058ab4597b73d/Dockerfile:

• cgr.dev/chainguard/jre:openjdk-

Our goal is to prevent your project from experiencing any disruptions. Please see the migration guide for options.

If there's more we can do to help please reply to this issue or email us at [email protected].

Thank you!

Logge hvilke topics som blir replikert

Det hadde vært veldig fint å fått med navnet på topicene når noe blir replikert. Nå sier den bare Found 3 records to mirror og når det er mange muligheter er ikke så lett å debugge.

Flaky tests

Noen tester feiler innimellom på GHA. En re-run fikser det. Disse bør sees på slik at de blir mer robuste.

Feil format i apploggen?

Det ser ut som at formatet på loggingen til apploggen avviker fra nais-dokumentasjonen, og det ser rart ut i apploggen til min instans av Aivia: Når jeg søker på https://logs.adeo.no/app/discover i logstash-apps-* med envclass : "p" and application : "rekrutteringsbistand-stilling-kafkabro" så er det ingen verdi i feltet level og jeg ser at loglevel og timestamp er inneholdt i feltet message. Prøver å legge ved screenshot.

Loggeformat følger ikke Nais JSON anbefaling

Ser i Kibana at loggelvel (feltet level) er tomt, mens verdien til loglevel "INFO" forekommer i feltet message. Det betyr antageligvis at Aivia ikke følger anbefalingen her: https://doc.nais.io/observability/logs/#log-configuration

Oppskrifta for å deploye fanger ikke opp endringer i ConfigMap

Oppskrifta som ligger på å sette opp appen og deployment fanger ikke opp at man gjør endringer i configmappet, og man må f.eks. manuelt slette podden for at endringen fanges opp.