[schema-registry] Topology with manually created SpecificAvroSerde needs to know the schema-registry URL upfront about fluent-kafka-streams-tests

[AHeise]

The way that we solve that is by using the TestTopology constructor that takes a function to create the topology.

public TestTopology(final Function<? super Properties, ? extends Topology> topologyFactory,
            final Map<?, ?> properties)

This topologyFactory function feeds back the properties in the second parameter with all automatically populated entries, such as SR url.
So if you have some createTopology(Properties properties) in your main application, you can retrieve the SR url with properties.get(AbstractKafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG) or try to directly pass the properties to the configure method of the AvroSerde.

[PHameete]

Thanks for this Pointer. I've got it to work in this way as well now.

Our app is part of a Spring Boot application which requires slightly more effort to allow us to configure it in this way, but its doable :-)

The issue can be closed then. Thanks for the super fast reply!

[kensixx]

Can you share a snippet on how this works? I also have a same scenario, but different to the sample codes, I use @Autowired in my unit test because my Kafka Streams implementation is also @Autowired due to Spring Kafka library.

I am confused right now as to how my @Autowired implementation can reach the mock schema registry URL when overriding Avro Serdes. Hope you guys can guide me on this 🙏🏻

@SpringBootTest
public class HourlyLoginCountProcessorTests {
    private SchemaRegistryMock schemaRegistryMock = new SchemaRegistryMock();

    @Autowired
    HourlyLoginCountProcessor hourlyLoginCountProcessor;

    TestTopology<Object, Object> testTopology = null;

    @BeforeEach
    public void setUp() throws IOException {
        this.schemaRegistryMock.start();

        StreamsBuilder streamsBuilder = new StreamsBuilder();
        this.hourlyLoginCountProcessor.buildPipeline(streamsBuilder);

        Topology topology = streamsBuilder.build();

        this.testTopology = new TestTopology<>(topology, this.getStreamsConfiguration("http://dummy"))
                .withSchemaRegistryMock(this.schemaRegistryMock);

        this.testTopology.start();
    }

    @Test
    void HourlyLoginCountProcessorTest() {
        LoginRaw loginRaw = this.generateLoginRawForSauyo();
        LoginRaw loginRawNullLocation = this.generateLoginRawWithNullLocation();

        // try inserting data into the LOGIN_RAW_STREAM stream / topic
        this.testTopology.input()
                .add("", loginRaw)
                .add("", loginRawNullLocation);

        this.testTopology.input()
                .at(TimeUnit.SECONDS.toMillis(1)).add("", loginRaw)
                .at(TimeUnit.SECONDS.toMillis(2)).add("", loginRawNullLocation);
//                .at(System.currentTimeMillis()).add(dummyInput); // flush KTable

        // see the output of the stream based on input
        ProducerRecord<Object, LoginCount> record =
                this.testTopology.tableOutput().withValueType(LoginCount.class).readOneRecord();

        // should use .toString() method because the field is in CharSequence
        System.out.println("haha");
    }

    private LoginRaw generateLoginRawWithNullLocation() {
        // omitted for brevity
    }

    private LoginRaw generateLoginRawForSauyo() {
        // omitted for brevity
    }

    private Properties getStreamsConfiguration(String url) {
        final Properties streamsConfiguration = new Properties();
        streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, "TopologyTestDriver");
        streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "mock:9092");
        streamsConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
        streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, SpecificAvroSerde.class);
        // giving any value to SCHEMA_REGISTRY_URL_CONFIG will activate a mocked Schema Registry.
        // actual value is ignored
        streamsConfiguration.put(AbstractKafkaSchemaSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, url);

        return streamsConfiguration;
    }
}

[philipp94831]

Hi @kensixx, can you please elaborate what the error is you are getting or what exactly is not working?