makubi / avrohugger-maven-plugin Goto Github PK

View Code? Open in Web Editor NEW

9.0 9.0 9.0 194 KB

Maven plugin for generating Scala case classes and ADTs from Apache Avro schemas, datafiles, and protocols

License: Apache License 2.0

Java 65.37% Scala 34.63%

avrohugger-maven-plugin's People

Contributors

Stargazers

Watchers

Forkers

jozic sini shudas chrisalbright jdoe-id1 algorri94 tyger belint01 lydiaskuse

avrohugger-maven-plugin's Issues

Missing 1.6 version from maven repository

I cannot find the referenced 1.6 version in the usage documentation on the maven central repository:

https://mvnrepository.com/artifact/at.makubi.maven.plugin/avrohugger-maven-plugin

I only see the 1.5 version as the newest.

Is the 1.6 released somewhere else?

Update to Avrohugger 0.15.0

Avrohugger 0.15.0 fixes ordering of AVDL files.

Incorporate avrohugger's support for unions

Avrohugger's PR was merged on Dec.1st to allow support for unions
julianpeeters/avrohugger#50
Would be great if plugin can support it too

Avrohugger is outdated

The version of Avrohugger used by this plugin is quite outdated: 1.0.0-RC24. Moreover, it is not a stable release version.

As a result, the plugin suffers from Avrohugger issues, that have been fixed in later versions (for example, incorrect handling of reserved words: julianpeeters/sbt-avrohugger#86)

Currently Avrohugger version 1.7.0 is available.

Latest version of avrohugger uses Seq instead of List in by default in generated code

I'm using this plugin in production, and have been limited to certain use cases because of List being the default interface for array types. I contributed a change to avrohugger to change the default to Seq which is more general @julianpeeters just published a new RC which has this functionality.

Add a test with reverse file ordering

Currently there is no test to validate that the generator works even if files are in reverse order.

Get stuck when Generating Scala files for schemas

I try to generate class from avdl files, and here is my directory for avro files:
src/main/avro
--common
--xxx
--xxx
--xxx

src/main/avro/ src/main/scala true

It works when I change sourceDirectory into common, but it does not work for other subpath.

All the output log I got is nothing but:
[DEBUG] (f) fileIncludes = []
[DEBUG] (f) limitedNumberOfFieldsInCaseClasses = false
[DEBUG] (f) namespaceMapping = []
[DEBUG] (f) outputDirectory = C:\xxx\src\main\scala
[DEBUG] (f) recursive = true
[DEBUG] (f) sourceDirectory = C:\xxx\src\main\avro
[DEBUG] (f) sourceGenerationFormat = SPECIFIC_RECORD
[DEBUG] (f) typeOverrides = at.makubi.maven.plugin.avrohugger.typeoverride.TypeOverrides@305f031
[DEBUG] -- end configuration --
[INFO] Generating Scala files for schemas in C:\xxx

And then get stuck here, I wait for a whole night, it does not throw any exception.

Can't use plugin with java 7

Hi there,
tried to use the plugin with java 7 and it fails with the following error

[ERROR] Failed to execute goal at.makubi.maven.plugin:avrohugger-maven-plugin_2.11:1.1:generate-scala-sources (default) on project Boo: Execution default of goal at.makubi.maven.plugin:avrohugger-maven-plugin_2.11:1.1:generate-scala-sources failed: Unable to load the mojo 'generate-scala-sources' in the plugin 'at.makubi.maven.plugin:avrohugger-maven-plugin_2.11:1.1' due to an API incompatibility: org.codehaus.plexus.component.repository.exception.ComponentLookupException: at/makubi/maven/plugin/avrohugger/GeneratorMojo : Unsupported major.minor version 52.0
[ERROR] -----------------------------------------------------
[ERROR] realm =    plugin>at.makubi.maven.plugin:avrohugger-maven-plugin_2.11:1.1
[ERROR] strategy = org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy
[ERROR] urls[0] = file:/home/eplatonov/.m2/repository/at/makubi/maven/plugin/avrohugger-maven-plugin_2.11/1.1/avrohugger-maven-plugin_2.11-1.1.jar
[ERROR] urls[1] = file:/home/eplatonov/.m2/repository/org/codehaus/plexus/plexus-utils/3.0.22/plexus-utils-3.0.22.jar
[ERROR] urls[2] = file:/home/eplatonov/.m2/repository/org/apache/commons/commons-lang3/3.4/commons-lang3-3.4.jar
[ERROR] urls[3] = file:/home/eplatonov/.m2/repository/javax/enterprise/cdi-api/1.0/cdi-api-1.0.jar
[ERROR] urls[4] = file:/home/eplatonov/.m2/repository/javax/annotation/jsr250-api/1.0/jsr250-api-1.0.jar
[ERROR] urls[5] = file:/home/eplatonov/.m2/repository/org/eclipse/sisu/org.eclipse.sisu.inject/0.3.2/org.eclipse.sisu.inject-0.3.2.jar
[ERROR] urls[6] = file:/home/eplatonov/.m2/repository/org/codehaus/plexus/plexus-component-annotations/1.5.5/plexus-component-annotations-1.5.5.jar
[ERROR] urls[7] = file:/home/eplatonov/.m2/repository/io/spray/spray-json_2.11/1.3.2/spray-json_2.11-1.3.2.jar
[ERROR] urls[8] = file:/home/eplatonov/.m2/repository/org/scala-lang/scala-library/2.11.8/scala-library-2.11.8.jar
[ERROR] urls[9] = file:/home/eplatonov/.m2/repository/com/julianpeeters/avrohugger-core_2.11/0.14.0/avrohugger-core_2.11-0.14.0.jar
[ERROR] urls[10] = file:/home/eplatonov/.m2/repository/org/scala-lang/scala-compiler/2.11.8/scala-compiler-2.11.8.jar
[ERROR] urls[11] = file:/home/eplatonov/.m2/repository/org/scala-lang/modules/scala-xml_2.11/1.0.4/scala-xml_2.11-1.0.4.jar
[ERROR] urls[12] = file:/home/eplatonov/.m2/repository/org/scala-lang/modules/scala-parser-combinators_2.11/1.0.4/scala-parser-combinators_2.11-1.0.4.jar
[ERROR] urls[13] = file:/home/eplatonov/.m2/repository/org/apache/avro/avro/1.7.7/avro-1.7.7.jar
[ERROR] urls[14] = file:/home/eplatonov/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar
[ERROR] urls[15] = file:/home/eplatonov/.m2/repository/com/thoughtworks/paranamer/paranamer/2.3/paranamer-2.3.jar
[ERROR] urls[16] = file:/home/eplatonov/.m2/repository/org/xerial/snappy/snappy-java/1.0.5/snappy-java-1.0.5.jar
[ERROR] urls[17] = file:/home/eplatonov/.m2/repository/org/apache/commons/commons-compress/1.4.1/commons-compress-1.4.1.jar
[ERROR] urls[18] = file:/home/eplatonov/.m2/repository/org/tukaani/xz/1.0/xz-1.0.jar
[ERROR] urls[19] = file:/home/eplatonov/.m2/repository/org/apache/avro/avro-compiler/1.7.7/avro-compiler-1.7.7.jar
[ERROR] urls[20] = file:/home/eplatonov/.m2/repository/commons-lang/commons-lang/2.6/commons-lang-2.6.jar
[ERROR] urls[21] = file:/home/eplatonov/.m2/repository/org/apache/velocity/velocity/1.7/velocity-1.7.jar
[ERROR] urls[22] = file:/home/eplatonov/.m2/repository/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar
[ERROR] urls[23] = file:/home/eplatonov/.m2/repository/org/scala-lang/scala-reflect/2.11.8/scala-reflect-2.11.8.jar
[ERROR] urls[24] = file:/home/eplatonov/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.9.13/jackson-core-asl-1.9.13.jar
[ERROR] urls[25] = file:/home/eplatonov/.m2/repository/com/eed3si9n/treehugger_2.11/0.4.1/treehugger_2.11-0.4.1.jar
[ERROR] Number of foreign imports: 1
[ERROR] import: Entry[import  from realm ClassRealm[maven.api, parent: null]]
[ERROR] 
[ERROR] -----------------------------------------------------
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginContainerException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :Boo

looks like this line https://github.com/makubi/avrohugger-maven-plugin/blob/master/pom.xml#L17 is the cause.
Is it possible to make it work for java 7 and 8?

Thanks

typeOverrides not working

I am using the following configuration:

<plugin>
                <groupId>at.makubi.maven.plugin</groupId>
                <artifactId>avrohugger-maven-plugin</artifactId>
                <executions>
                    <execution>
                        <phase>generate-sources</phase>
                        <goals>
                            <goal>generate-scala-sources</goal>
                        </goals>
                        <configuration>
                            <sourceDirectory>${project.basedir}/avro</sourceDirectory>
                            <typeOverrides>
                                <timestampMillisType>JAVA_TIME_INSTANT</timestampMillisType>
                            </typeOverrides>
                        </configuration>
                    </execution>
                </executions>
</plugin>

with the following avro schema:

protocol metrics {
  record Metric {
    string metricName;
    @logicalType("timestamp-millis")
    long executionTimestamp;
  }
}

The build and case class generation works fine but I am seeing that the executionTimestamp parameter is Long and not java.time.Instant as I would expect.
Is there something wrong with my configuration ?

ERROR: Not an Avro data file.

When I run mvn avrohugger:generate-scala-sources, I get the error:

Not an Avro data file. -> [Help 1]

Here is the context in the output:

[INFO] Generating Scala files for schemas in /Users/devin.bost/proj/repos/pulsar_workshop/scenarios/native-pulsar/np-common/src/main/avro to /Users/devin.bost/proj/repos/pulsar_workshop/scenarios/spark-scala/target/generated-sources
[INFO] Generating Scala files for /Users/devin.bost/proj/repos/pulsar_workshop/scenarios/native-pulsar/np-common/src/main/avro/DeviceTSNew.avro
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 0.429 s
[INFO] Finished at: 2023-06-27T18:47:58-05:00
[INFO] ------------------------------------------------------------------------
[ERROR] Not an Avro data file. -> [Help 1]

This can be reproduced from this repo branch:

https://github.com/devinbost/pulsar_workshop/tree/spark-scala

To reproduce:
git clone https://github.com/devinbost/pulsar_workshop.git
git checkout spark-scala
cd pulsar_workshop
cd scenarios
cd spark-scala
mvn avrohugger:generate-scala-sources

For reference, here is the Avro file:

{
  "namespace": "com.example.pulsarworkshop",
  "type": "record",
  "name": "DeviceTSNew",
  "fields": [
    {
      "name": "tag_id",
      "type": "string"
    },
    {
      "name": "data_quality",
      "type": "int"
    },
    {
      "name": "event_time",
      "type": "string",
      "logicalType": "timestamp-millis",
      "doc": "The date and time is stored as string in ISO 8601 format"
    },
    {
      "name": "event_value",
      "type": "double"
    },
    {
       "name": "yyyymm",
       "type": "int"
    }
  ]
}

Here is the POM file:

pom.xml.txt

Including a class with the same name as a class in avdl file gets shadowed

I'm trying to use a class from another avdl file in a file with a record of the same name. When this gets generated, the resulting scala uses the class inside the file rather than the class imported.

Example:
fileOne.avdl

@namespace("com.my.package.one")
protocol ProtocolOne {
    record MyRecord {
        ....
    }
}

fileTwo.avdl

@namespace("com.my.package.two")
protocol ProtocolTwo {
    record MyRecord {
        ....
    }

    record UsesMyRecord {
        com.my.package.one.MyRecord myRecord;
    }
}

Resulting case class UsesMyRecord will use com.my.package.two's MyRecord since the generated case class doesn't keep the full class path

New syntax from Avro 1.11.1 not supported

Hello,

    <groupId>at.makubi.maven.plugin</groupId>
    <artifactId>avrohugger-maven-plugin</artifactId>
    <version>1.7</version>

Currently using v1.7, I cannot use latest syntaxt feature form Avro 1.11.1

Avro 1.11.1 has introduced new syntax to create optional fields AVRO-32557

ie:

union {null, string} optionalName = null;

can be writen

string? optionalName = null;

A bump to the latest version of avrohugger should fix the issue.

Thanks

Incorporate avrohugger's namespace mapping feature

Avro Hugger has the ability to change the package name of the generated scala source files.
https://github.com/julianpeeters/avrohugger#customizable-namespace-mapping

Having this functionality exposed in this plugin would be great. I have created a PR to do this

#13

release for java 1.8

Is it possible to build an artefact for the latest (1.7) version of this plugin with Java 1.8 and publish it to Maven?

makubi / avrohugger-maven-plugin Goto Github PK

avrohugger-maven-plugin's People

Contributors

Stargazers

Watchers

Forkers

avrohugger-maven-plugin's Issues

Recommend Projects

Recommend Topics

Recommend Org

Jobs