GithubHelp home page GithubHelp logo

aivia's Introduction

AiviA

Simple app to mirror a topic from on-prem kafka to Aiven kafka. It does not support Schema Registry.

!!! This application is intended as a tool to help migrate away from the on-prem Kafka cluster, which is deprecated. !!!

How does it work?

First you need to create a mapping of topics to mirror. The app will read this from a properties-file, mapping a source topic to a destination topic.

Example:

aapen-data-v2 = nais.data-v2
privat-hemmelig-saa-det-saa = nais.hemmelig-v2

With this configuration messages will be copied from aapen-data-v2 on-prem to nais.data-v2 on Aiven, and from privat-hemmelig-saa-det-saa on-prem to nais.hemmelig-v2 on Aiven.

The application will read the file from the path /var/run/configmaps/aivia-topic-mapping/topic_mapping.properties. If you want something else, you can use the environment variable AIVIA_TOPIC_MAPPING_PATH.

If you want to use AiviA to mirror some other direction than on-prem to Aiven, you also need to configure source and target clusters using the environment variables AIVIA_SOURCE and AIVIA_TARGET respectively. Valid values are on-prem and aiven. The default is on-prem for source and aiven for target.

It is possible to mirror any combination of AIVIA_SOURCE and AIVIA_TARGET (even aiven to aiven).

Using AiviA

1. Create a new repository with a nais.yaml file:

apiVersion: "nais.io/v1alpha1"
kind: "Application"
metadata:
  name: aivia
  namespace: myteam
  labels:
    team: myteam
spec:
  image: ghcr.io/nais/aivia:latest
  liveness:
    path: "/internal/isalive"
  readiness:
    path: "/internal/isready"
  replicas:
    min: 1
    max: 1
    cpuThresholdPercentage: 50
  prometheus:
    enabled: true
    path: "/internal/prometheus"
  limits:
    cpu: "200m"
    memory: "256Mi"
  requests:
    cpu: "200m"
    memory: "256Mi"
  env:
    - name: LOG_FORMAT
      value: logstash
  envFrom:
    - secret: aivia-kafka-on-prem
  filesFrom:
    - configmap: aivia-topic-mapping
  kafka:
    pool: nav-dev

It is best to use a specific version, which you can do by getting the latest aivia image from the AiviA package page. If you do so, you must make sure to keep updated with latest versions.

2. Create a configmap with your topic mappings

apiVersion: v1
kind: ConfigMap
metadata:
  labels:
    team: myteam
  annotations:
    reloader.stakater.com/match: "true"
  name: aivia-topic-mapping
  namespace: myteam
data:
  topic_mapping.properties: |
    aapen-data-v2 = myteam.data-v2
    privat-hemmelig-saa-det-saa = myteam.hemmelig-v2

3. Create a workflow to deploy the application and configmap

name: "Deploy aivia"
on:
  push:
    branches:
    - "main"
jobs:
  deploy:
    name: "Deploy AiviA"
    runs-on: ubuntu-latest
    steps:
      - uses: "actions/checkout@v3"
      - uses: nais/deploy/actions/deploy@v1
        env:
          APIKEY: ${{ secrets.NAIS_DEPLOY_APIKEY }}
          CLUSTER: dev-gcp
          RESOURCE: configmap.yaml,nais.yaml

4. Create a secret with configuration for on-prem kafka, and apply it to the cluster

apiVersion: v1
kind: Secret
metadata:
  name: aivia-kafka-on-prem
  namespace: myteam
stringData:
  KAFKA_ON_PREM_BROKERS: "b27apvl00045.preprod.local:8443,b27apvl00046.preprod.local:8443,b27apvl00047.preprod.local:8443"
  KAFKA_ON_PREM_USERNAME: "myServiceUser"
  KAFKA_ON_PREM_PASSWORD: "myServiceUsersPassword"

Run kubectl apply -f secret.yaml to insert the secret into the cluster. Do not commit secret.yaml to the repository.

5. Commit and push

Commit nais.yaml, configmap.yaml and .github/workflows/main.yaml and push to github.

What does this do?

AiviA guarantees the following:

  • At-least-once delivery of all messages present in the source topic.
  • Preservation of message ordering if using the default partitioning strategy.
  • Propagation of keys present in messages from the source topic.

AiviA does not guarantee the following:

  • Prompt delivery of messages.
  • Preservation of offsets. Your application should ideally handle messages idempotently.

Verifying the aivia image and its contents

The image is signed "keylessly" using Sigstore cosign. To verify its authenticity run

cosign verify \
--certificate-identity "https://github.com/nais/aivia/.github/workflows/main.yaml@refs/heads/main" \
--certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
ghcr.io/nais/aivia@sha256:<shasum>

The images are also attested with SBOMs in the CycloneDX format. You can verify these by running

cosign verify-attestation --type cyclonedx \
--certificate-identity "https://github.com/nais/aivia/.github/workflows/main.yaml@refs/heads/main" \
--certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
ghcr.io/nais/aivia@sha256:<shasum>

aivia's People

Contributors

dependabot[bot] avatar gtcno avatar jhrv avatar jksolbakken avatar mortenlj avatar sechmann avatar sonhal avatar thokra-nav avatar tronghn avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

aivia's Issues

ACTION REQUIRED: Changes to pulling Chainguard Images

Hey there Chainguard here.

We noticed that you are using Chainguard Images, thank you! We wanted to make you aware of an upcoming change that will impact your project.

Starting August 16, 2023 public users will no longer be able to pull images from our registry (cgr.dev/chainguard) by tags other than latest or latest-dev. Please see the announcement for more information.

You are currently using the following.

In https://github.com/nais/aivia/blob/0db0c735a85e3273b6d6c9f4781058ab4597b73d/.github/workflows/main.yaml:

  • cgr.dev/chainguard/jre:openjdk-

In https://github.com/nais/aivia/blob/0db0c735a85e3273b6d6c9f4781058ab4597b73d/Dockerfile:

  • cgr.dev/chainguard/jre:openjdk-

Our goal is to prevent your project from experiencing any disruptions. Please see the migration guide for options.

If there's more we can do to help please reply to this issue or email us at [email protected].

Thank you!

Logge hvilke topics som blir replikert

Det hadde vært veldig fint å fått med navnet på topicene når noe blir replikert. Nå sier den bare Found 3 records to mirror og når det er mange muligheter er ikke så lett å debugge.

Flaky tests

Noen tester feiler innimellom på GHA. En re-run fikser det. Disse bør sees på slik at de blir mer robuste.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.