mennanov / fieldmask-utils

Protobuf Field Mask Go utils

Protobuf Field Mask utils for Go

Features:

• Copy from any Go struct to any compatible Go struct with a field mask applied
• Copy from any Go struct to a map[string]interface{} with a field mask applied
• Extensible masks (e.g. inverse mask: copy all except those mentioned, etc.)
• Supports Protobuf Any message types.

If you're looking for a simple FieldMask library to work with protobuf messages only (not arbitrary structs) consider this tiny repo: https://github.com/mennanov/fmutils

Examples

Copy from a protobuf message to a protobuf message:

// testproto/test.proto

message UpdateUserRequest {
    User user = 1;
    google.protobuf.FieldMask field_mask = 2;
}

package main

import fieldmask_utils "github.com/mennanov/fieldmask-utils"

// A function that maps field mask field names to the names used in Go structs.
// It has to be implemented according to your needs.
// Scroll down for a reference on how to apply field masks to your gRPC services.
func naming(s string) string {
	if s == "foo" {
		return "Foo"
	}
	return s
}

func main () {
	var request UpdateUserRequest
	userDst := &testproto.User{} // a struct to copy to
	mask, _ := fieldmask_utils.MaskFromPaths(request.FieldMask.Paths, naming)
	fieldmask_utils.StructToStruct(mask, request.User, userDst)
	// Only the fields mentioned in the field mask will be copied to userDst, other fields are left intact
}

Copy from a protobuf message to a map[string]interface{}:

package main

import fieldmask_utils "github.com/mennanov/fieldmask-utils"

func main() {
	var request UpdateUserRequest
	userDst := make(map[string]interface{}) // a map to copy to
	mask, _ := fieldmask_utils.MaskFromProtoFieldMask(request.FieldMask, naming)
	err := fieldmask_utils.StructToMap(mask, request.User, userDst)
	// Only the fields mentioned in the field mask will be copied to userDst, other fields are left intact
}

Copy with an inverse mask:

package main

import fieldmask_utils "github.com/mennanov/fieldmask-utils"

func main() {
	var request UpdateUserRequest
	userDst := &testproto.User{} // a struct to copy to
	mask := fieldmask_utils.MaskInverse{"Id": nil, "Friends": fieldmask_utils.MaskInverse{"Username": nil}}
	fieldmask_utils.StructToStruct(mask, request.User, userDst)
	// Only the fields that are not mentioned in the field mask will be copied to userDst, other fields are left intact.
}

Naming function

For developers that are looking for a mechanism to apply a mask field in their update endpoints using gRPC services, there are multiple options for the naming function described above:

• Using the CamelCase function provided in the original protobuf repository. This repository has been deprecated and it will potentially trigger lint errors.
  • You can copy-paste the CamelCase function to your own project or,
  • You can use an Open Source alternative that provides the same functionality, already took care of copying the code, and also added tests.

func main() {
    mask := &fieldmaskpb.FieldMask{Paths: []string{"username"}}
    mask.Normalize()
    req := &UpdateUserRequest{
        User: &User{
            Id:       1234,
            Username: "Test",
        },
    }
    if !mask.IsValid(req) {
        return
    }
    protoMask, err := fieldmask_utils.MaskFromProtoFieldMask(mask, strings.PascalCase)
    if err != nil {
        return
    }
    m := make(map[string]any)
    err = fieldmask_utils.StructToMap(protoMask, req, m)
	if err != nil {
		return
    }
	fmt.Println("Resulting map:", m)
}

This will result in a map that contains the fields that need to be updated with their respective values.

Limitations

1. Larger scope field masks have no effect and are not considered invalid:

   field mask strings "a", "a.b", "a.b.c" will result in a mask a{b{c}}, which is the same as "a.b.c".

2. Masks inside a protobuf Map are not supported.

3. When copying from a struct to struct the destination struct must have the same fields (or a subset) as the source struct. Either of source or destination fields can be a pointer as long as it is a pointer to the type of the corresponding field.

4. oneof fields are represented differently in fieldmaskpb.FieldMask compared to fieldmask_util.Mask. In FieldMask the fields are represented using their property name, in this library they are prefixed with the oneof name matching how Go generated code is laid out. This can lead to issues when converting between the two, for example when using MaskFromPaths or MaskFromProtoFieldMask.

fieldmask-utils's Issues

empty slice

type MomentInfo struct {}
type list struct {
    MomentInfos []*MomentInfo `json:"moment_infos"`
}

old := &list{[]*MomentInfo{}}
new := map[string]interface{}

_ = fieldmask_utils.StructToMap(mask, old, new, fieldmask_utils.WithTag(`json`))

jsonStr, _ := json.Marshal(new)
t.Log(new, "\n", string(jsonStr))

print

map[moment_infos:[]] 
{"moment_infos":null}

why after json.Marshal got a nil pointer but not an empty slice?

golang 1.18
fieldmask-utils 0.5.0

whether add Option SrcTag() for copy from src tag

https://github.com/mennanov/fieldmask-utils/blob/f9573a9a16fd68cd4ab04fca7d1b16dadc7e346a/copy.go#L74C4-L74C14

now mask.Filter use the FiledName of object, However, path also input json_tag, I hope this package can add SrcTag Option for this scene

not support time.Time

time.Time transport to empty map by structx.StructToMapByFieldMask

println(fmt.Sprintf("%T", updateField["CreateTime"]))----------> map[string]interface {}

*int64 panic

*int64 field got panic by StructToMap()，stack as below

example:

import (
	msku "github.com/mennanov/fieldmask-utils"
)

type example struct {
	P2Int64  *int64
	WhatEver string
}

func willPanic() {
	var (
		i         int64 = 666
		updateMap       = make(map[string]interface{})
	)

	mask, _ := msku.MaskFromPaths(
		[]string{"P2Int64", "WhatEver"},
		func(s string) string { return s },
	)

	// panic here
	_ = msku.StructToMap(
		mask,
		&example{
			P2Int64:  &i,
			WhatEver: "what ever",
		},
		updateMap,
	)
}

nil slice in src not working

@mennanov nil slice is being set as empty

 func TestStructToStruct_EntireSlice_NonEmptyDst(t *testing.T) {
	type A struct {
		Field1 string
		Field2 int
	}
	type B struct {
		Field1 string
		A      []A
	}
	src := &B{
		Field1: "src StringerB field1",
		A:      nil,
	}
	dst := &B{
		Field1: "dst StringerB field1",
		A: []A{
			{
				Field1: "dst StringerA field1 0",
				Field2: 10,
			},
			{
				Field1: "dst StringerA field1 1",
				Field2: 20,
			},
			{
				Field1: "dst StringerA field1 2",
				Field2: 30,
			},
		},
	}

	mask := fieldmask_utils.MaskFromString("Field1,A")
	err := fieldmask_utils.StructToStruct(mask, src, dst)
	require.NoError(t, err)
	assert.Equal(t, &B{
		Field1: src.Field1,
		A:      src.A,
	}, dst)
}

output -
@@ -2,3 +2,4 @@ Field1: (string) (len=20) "src StringerB field1", - A: ([]fieldmask_utils_test.A) <nil> + A: ([]fieldmask_utils_test.A) { + } })

CamelCase func in docs is now deprecated

When building a project following the examples, I receive the following notice:
google.golang.org/protobuf/protoc-gen-go/generator: cannot find module providing package google.golang.org/protobuf/protoc-gen-go/generator

Unfortunately, the new package does not export their GoCamelCase func.

Not sure how you prefer to address this, but I do think the docs should be updated to an alternative, if one exists.

Not copying anypb correctly?

I don't believe StructToStruct is handling anypb correctly. Take a look at the attached patch:
any_existing.patch.txt

It appears to be copying other fields. Notice the ExtraUser Role field has changed even though it isn't included in the mask.

Edit: Also, the ExtraUser Name field should be unchanged (since it isn't included in the mask), but is changing to "username".

feat: support protobuf wrapper values for get the value directly.

is it possible to support protobuf wrapper values for get the value directly.for now feildmask-util lib,will get a {"value":somevalue}

question: any way to conver Mask to MaskInverse?

some code may help:

mask, _ := fieldmask_utils.MaskFromProtoFieldMask(GetFieldMask(), generator.CamelCase)

fieldmask_utils.StructToStruct(fieldmask_utils.MaskInverse(mask), src, dst)

FROM readme example:

mask := fieldmask_utils.MaskInverse{"Id": nil, "Friends": fieldmask_utils.MaskInverse{"Username": nil}}

can also write as

fieldmask_utils.Mask{"Id": nil, "Friends": fieldmask_utils.MaskInverse{"Username": nil}}

Example using oneof fields and StructToStruct

I'm trying to validate/limit writes to a proto message that includes a oneof group but can't work out how to combine MaskFromProtoFieldMask and StructToStruct.

The issue I'm having is that fieldmaskpb.New correctly validates the oneof fields as if they were top level fields, but fieldmask-utils expects them to be nested inside the oneof group name.

Example

Given a proto like this

message Msg {
  oneof name {
    string formal = 1;
    string casual = 2;
  }
}

This does not return an error: fieldmaskpb.New(&pb.Msg{}, "formal") but passing "name" or "name.formal" does. On the util side, using fieldMaskUtils.MaskFromProtoFieldMask(fields, strcase.ToCamel) however results in a Mask that doesn't work with that proto.

Hope that all makes sense?

I've also submitted a ticket on the proto side to hopefully make this easier: golang/protobuf#1344

Ambiguous merging of nullable structs with StructToStruct

The following new test case for copy.go is failing. I added it in my local copy.

I wonder what the correct behavior should be in this case.

func TestStructToStruct_NestedNilParentStruct_NonNilDst(t *testing.T) {
	type A struct {
		Field1 string
		Field2 int
	}
	type B struct {
		Field1 string
		Field2 int
		A      *A
	}
	type C struct {
		Field1 string
		B      B
	}

	// Given: src.B.A is nil, dst.B.A is not nil
	src := &C{
		Field1: "src C field1",
		B: B{
			Field1: "src StringerB field1",
			Field2: 1,
			A:      nil, // nil struct
		},
	}
	dst := &C{
		Field1: "dst C field1",
		B: B{
			Field1: "dst StringerB field1",
			Field2: 2,
			A: &A{
				Field1: "dst StringerA field1",
				Field2: 10,
			},
		},
	}

	// Given: mask contains subfield of nil field in src, i.e. src.B.A
	mask := fieldmask_utils.MaskFromString("B{Field1,A{Field2}}")

	// When: StructToStruct is called
	err := fieldmask_utils.StructToStruct(mask, src, dst)

	// Then: no error is returned
	require.NoError(t, err)

	// Then: dst B.A is not modified
	assert.Equal(t, &C{
		Field1: "dst C field1",
		B: B{
			Field1: src.B.Field1,
			Field2: 2,
			A: &A{
				Field1: "dst StringerA field1",
				Field2: 10,
			},
		},
	}, dst)
}

Question: Is there a way to validate Mask fields are correct?

I'd like to be able to easily catch bad paths (e.g., due to a typo or misconfiguration):

       paths := []string{
		"Person.FirstNaaaame",  // bad
		"Person.LastName",
	}
	// create mask
	mask, err := fu.MaskFromPaths(paths, namer)
        // desire
        err = fu.Validate(mask, someStruct}

Is there an existing way to validate that all paths refer to an actual field? I'm asking before I implement something to do this.

Convert protobuf message to map after applying a field mask doesn't work

Context

I'm trying to convert a proto message to map using this library in order to perform a PATCH update in an endpoint. I'm getting an empty map instead of a single field map with the respective value.

Relevant code

mask := &fieldmaskpb.FieldMask{Paths: []string{"title"}}
mask.Normalize()
task := &tasksv1.Task{
  Title: "test",
}
if !mask.IsValid(task) {
  return nil, errors.New("invalid mask")
}
protoMask, err := fieldmask_utils.MaskFromProtoFieldMask(mask, func(s string) string { return s })
if err != nil {
  return nil, err
}
m := make(map[string]any)
if err = fieldmask_utils.StructToMap(protoMask, task, m); err != nil {
  return nil, err
}
fmt.Println("Map:", m)

Current behavior

m is empty

Expected behavior

m := map[string]any{
  "title": "test",
}

Reference

Protobuf messages: https://github.com/marcoshuck/todo/blob/main/api/tasks/v1/tasks.proto#L60

Preserve field values in dst

Hi, thanks for the lib.

Currently, when copying structs, the behavior of preserving fields in dst not consistent. Top-level struct fields are always preserved, but nested struct fields are always zeroed (if mask targets other fields).

Example test case:

func TestStructToStruct(t *testing.T) {
	type (
		Author struct {
			FirstName string
			LastName string
		}
		Document struct {
			Title string
			Date string
			Author Author
		}
	)

	dst := &Document{
		Title:  "Hello, World!",
		Date: "2019-05-01",
		Author: Author{
			FirstName: "John",
			LastName:  "Doe",
		},
	}
	src := &Document{
		Title: "Hello, updated World!",
		Author: Author{
			FirstName: "Johnny",
		},
	}
	expected := &Document{
		Title:  "Hello, updated World!",
		Date: "2019-05-01",
		Author: Author{
			FirstName: "Johnny",
			LastName:  "Doe",
		},
	}

	mask := fieldmask_utils.MaskFromString("Title,Author{FirstName}")
	err := fieldmask_utils.StructToStruct(mask, src, dst)
	require.NoError(t, err)

	assert.Equal(t, expected, dst)
}

In the result of this example, field "Date" is preserved in dst, but field "Author.LastName" is not.

Error introduced upgrading to bazel_rules_go

com_github_mennanov_fieldmask_utils/copy.go:78:28: call of reflect.ValueOf copies lock value: google.golang.org/protobuf/types/known/anypb.Any contains google.golang.org/protobuf/internal/impl.MessageState contains sync.Mutex

Ref: https://stackoverflow.com/questions/64183794/why-do-the-go-generated-protobuf-files-contain-mutex-locks

Sorry I don't know what other sort of context you need here, I'm getting this error in a build for some other service at my org when updating another dependency that then requires me to update bazel_rules_go. It's a mess.

If I can provide any more context please let me know. Although the error is happening because I'm upgrading some bazel package, the error looks to be a gRPC violation that became more strict in later versions.

Unexpected copy results when mask is empty

	type A struct {
		B string
	}
	a := &A{
		B: "b",
	}
	mask, _ := fm.MaskFromPaths([]string{}, generator.CamelCase)
	dest := &A{}
	_ = fm.StructToStruct(mask, a, dest)

	fmt.Println(dest)
	// got: &{b}
	// expect: &{}

I am confused with the copy result. When the mask (paths) is empty, what I expect is nothing will be copied. But the result is everything is copied.

I am wondering what's the purpose to copy everything when the mask is empty?

What I set is NOT what I get ???

package main

import (
	"fmt"

	msku "github.com/mennanov/fieldmask-utils"
)

func main() {
	var (
		err  error
		mask msku.Mask
	)

	if mask, err = msku.MaskFromPaths(
		[]string{"a.b.c"},
		func(s string) string { return s },
	); err != nil {
		panic(err)
	}

	if _, ok := mask.Get("a.b.c"); ok {
                 // I expected this would work
		fmt.Println("That's Good! As Expcted!")
	} else {
                // but I got this instead.
                // Why ???
		fmt.Println("Oops! Suprise!")
	}
}

Why not support mapToStruct?

Why is structToMap supported but not mapToStruct?

Thank you.

protobuf v1.3.5 support

We are using fieldmask-utils here at the New York Times for a proof of concept - it's working well so far, except that it required us to bump up to v1.4.2 of protobuf from v1.3.5. This is causing a big performance issue in our use case (marshaling is almost 2-3x slower). My question is whether fieldmask-utils can work with version 1.3.5? Does it rely on anything specific in v1.4.2?

question: Is it possible to map from a struct to an identical struct with pointer fields?

Hi,

I'm wondering if its possible to apply a field mask and map to a destination struct where all fields are pointers and any fields excluded after applying the mask are set to nil - as shown in the code snippet below:

package main

import (
  "fmt"

  fmutils "github.com/mennanov/fieldmask-utils"
)


func main() {
	type Source struct {
		FieldOne string
		FieldTwo int
	}

	type Dest struct {
		FieldOne *string
		FieldTwo *int
	}

	source := Source{FieldOne: "hello", FieldTwo: 5}
	dest := new(Dest)

	mask := fmutils.MaskFromString("FieldTwo")
	err := fmutils.StructToStruct(mask, source, &dest)
	fmt.Println("error", err)
	fmt.Println(dest)
}

However, I encounter the following error: error src <int Value>, int is not addressable

My use case here is that I'm trying to implement an gRPC Update endpoint that supports partial updates and I'm using a SQL query builder (squirrel) so I need a way that I can dynamically construct the UPDATE query and only include fields that were contained in the update mask. So in the example above, it would be any non nil fields on the Dest struct.

The reason I don't just use an identical struct (no pointers), is because one wouldn't be able to distinguish between go zero values and values that a user actual explicitly provided in their update request (e.g. if they wanted to set an integer field to zero).

I did think about using the StructToMap method but then there is also some challenges with mapping between the field names in my proto message and the column names in my DB.

I also tried and saw your other repo https://github.com/mennanov/fmutils and previously I had pretty much the same pattern you had in this test but I didn't like the fact I had to do an extra request to fetch the existing resource in the database for every update request.

Do you know of any elegant ways to accomplish dynamic UPDATE query generation? I'd love to hear any thoughts you might have on this!

Also a little side question: in fmutils repo I can simply do

fmutils.Filter(protoMessage, req.Msg.UpdateMask.Paths) and everything works correctly, whereas in this repo it seems that one must apply camel casing first to the proto FieldMask?

Thanks!