Null fields lost when uploading datastore backup with repeated nested records about google-bigquery

[GoogleCodeExporter]

Could you share a little additional information:

- the structure of your appengine entity (specifically how it is declared in 
your code)
- the job id of the import

The current implementation turns appengine entities into bigquery nested 
records so if your entity really is:
[ { name, time, error } ]
then you should get a nested record and not a 3 top level list fields:
[ name ]
[ time ]
[ error ]
however this depends on how the top level entity is constructed in appengine 
and that is why I am asking for your appengine entity declaration.

Original comment by [email protected] on 26 Jul 2013 at 5:03

[GoogleCodeExporter]

Glad to provide more info.

Job IDs for upload (from python 2.7 and 2.5 respectively):
  job_b83f91fb3fd94a2fbc7767699e253854
  job_d63291113ca240dc95f3a58be66bd504


Test data creation code:

from datetime import datetime
from google.appengine.ext.ndb.model import Model, StringProperty, 
StructuredProperty, IntegerProperty, DateTimeProperty


class Event(Model):
    eventType = StringProperty()
    eventTime = IntegerProperty()
    eventError = StringProperty()


class Session(Model):
    sessionID = StringProperty()
    startTime = DateTimeProperty()
    events = StructuredProperty(modelclass=Event, repeated=True)


class BQTestHandler(webapp.RequestHandler):

    def get(self):
        event1 = Event(eventType='A', eventTime=1)
        event2 = Event(eventType='B', eventTime=7, eventError='Null Pointer')
        event3 = Event(eventType='A', eventTime=12)
        session = Session(sessionID='23432', startTime=datetime.now(), events=[event1, event2, event3])
        session.put()

Original comment by [email protected] on 26 Jul 2013 at 6:55

[GoogleCodeExporter]

So the mismatch here is due to how StructureProperty lays out the data:
https://developers.google.com/appengine/docs/python/ndb/properties#structured

That doc explains that it lays it out as an array of repeated fields. It also 
calls out the restriction that a repeated structured property cannot contain 
another repeated structured property / field. Which is a result of this layout.

I don't yet have a solution for you but I am investing LocalStructuredProperty 
and its behavior. Also, I am investigating the underlying db.Model library and 
checking if it has a way of expressing structured properties. I will update you 
once I know more.

Original comment by [email protected] on 26 Jul 2013 at 9:03

[GoogleCodeExporter]

It looks like this issue is quite old, and the engineer who was working on it 
has left the company.

Resolving due to lack of activity; please re-open if this is still an issue for 
you.

Original comment by [email protected] on 22 Aug 2014 at 5:45

• Changed state: WontFix