directus-labs / 8-to-9-migration-tool Goto Github PK

View Code? Open in Web Editor NEW

1.0 1.0 0.0 58 KB

Automated script to migrate from Directus v8 to Directus v9

JavaScript 99.58% Shell 0.42%

directus migration

8-to-9-migration-tool's People

Contributors

Stargazers

Watchers

8-to-9-migration-tool's Issues

Migration fails with only a single user in the v8 database

With only a single source user, migration fails at tasks/user.js, line 92:

context.userMap[user.id] = createdUsers.data.data[index].id;

Upon investigation it seems that createdUsers.data.data is an object rather than an array with a single user. I fixed it by changing line 92 to this:

context.userMap[user.id] = createdUsersAsArray[index].id;

createdUsersAsArray is defined but never used, and I assume this to be its purpose.
Line 95 may require the same substitution, although it completed the migration without changing it.

Migrating stops with Status field type.

Hi,

Using the latest code. V8 latest and V9 rc91. When I run the migration it stops on a collection with this error -

 ✖ principal_projects
        → Cannot read property 'value' of undefined

and this stack trace at the end. It looks to be something related to an issue reading option.soft_delete values from a field of type Status. The status mappings all have soft delete in the off state. 26 collections are migrated before this one, so it's working up until then.

TypeError: Cannot read property 'value' of undefined
    at Task.task (file:///home/support/migration-tool/tasks/schema.js:168:53)
    at /home/support/migration-tool/node_modules/listr/lib/task.js:167:30
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  context: [Object: null prototype] {
    completedSteps: {
      schema: true,
      collections: false,
      files: false,
      roles: false,
      users: false,
      relationsv8: false,
      relations: false,
      data: false,
      completed: false
    },

We could possibly just delete the field and recreate on the migrated version, but wondered if there's something I else I can do?

Thanks

Richard

Error when Migrating Relations Field id in collection is a primary key

Managed to fix most problems, but got stuck on this one:

    V9 =>
    /relations
    undefined
    undefined
    V9 <=
    400
    {
    "errors": [
        {
            "message": "Field \"id\" in collection \"songs_directus_midis\" is a primary key",
            "extensions": {
                "code": "INVALID_PAYLOAD"
            }
        }
    ]
}
    
Error: 
    V9 =>
    /relations
    undefined
    undefined
    V9 <=
    400
    {
    "errors": [
        {
            "message": "Field \"id\" in collection \"songs_directus_midis\" is a primary key",
            "extensions": {
                "code": "INVALID_PAYLOAD"
            }
        }
    ]
}
    
    at file:///Users/arillo/dev/aibeatz/backups/migration-tool/api.js:41:9
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async migrateRelationsData (file:///Users/arillo/dev/aibeatz/backups/migration-tool/tasks/relations.js:108:3)

Perhaps somebody can point me into the proper direction on how to fix this error when migrating relations? Thanks.

SyntaxError: Unexpected token '.' on execution

While executing node index.js I get the following error

jatinsinghal@Jatins-MacBook-Pro directus-migration-tool % node index.js 
(node:13171) ExperimentalWarning: The ESM module loader is experimental.
file:///Users/jatinsinghal/Documents/projects/canopy/directus-migration-tool/tasks/schema.js:51
          })?.field || null,
             ^

SyntaxError: Unexpected token '.'
    at Loader.moduleStrategy (internal/modules/esm/translators.js:88:18)

Is there something I am missing?

"Error: Request failed with status code 503" when Migrating Files after Migrating Schema

Console Output: https://gist.github.com/Yuuyuuei/091c313b0fb09393626a1158d63c463f

Migration is Failing with 404

Migrating the Test Directus8 to Directus9 is failing. Has anyone used this tool successfully?

$ node index.js
✨ Migrating https://taxonomy-directus-service.test.xxx.com (v8) to https://taxonomy-directus-service.dev.xxx.com (v9)...
  ❯ Migrating Schema
    ✖ Downloading Schema
      → Request failed with status code 404
      Creating Collections
      Migrating Relations
    Migration Files
    Migrating Users
    Migrating Data
Error: Request failed with status code 404
    at createError (/workdir/migration/migration-tool/node_modules/axios/lib/core/createError.js:16:15)
    at settle (/workdir/migration/migration-tool/node_modules/axios/lib/core/settle.js:17:12)
    at IncomingMessage.handleStreamEnd (/workdir/migration/migration-tool/node_modules/axios/lib/adapters/http.js:260:11)
    at IncomingMessage.emit (node:events:388:22)
    at endReadableNT (node:internal/streams/readable:1295:12)
    at processTicksAndRejections (node:internal/process/task_queues:80:21) {
  config: {
    url: '/collections',
    method: 'get',
    headers: {
      Accept: 'application/json, text/plain, */*',
      Authorization: 'Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6IjJlNTY0N2NkLWFhNTEtNDJmNy05YWExLTBjYzM2NGUwNDFjMiIsImlhdCI6MTYyMDY3NDE4OCwiZXhwIjoxNjIwNjc1MDg4fQ.5--203VXNU1kdKV7HUnWm6cmQ1GkmzYweYlmLAuh988',
      Cookie: 'directus--session=undefined',
      'User-Agent': 'axios/0.21.1'
    },
    baseURL: 'https://taxonomy-directus-service.test.xxx.com/',
    transformRequest: [ [Function: transformRequest] ],
    transformResponse: [ [Function: transformResponse] ],
    timeout: 0,
    adapter: [Function: httpAdapter],
    xsrfCookieName: 'XSRF-TOKEN',
    xsrfHeaderName: 'X-XSRF-TOKEN',
    maxContentLength: -1,
    maxBodyLength: -1,
    validateStatus: [Function: validateStatus],
    data: undefined
  },
request: <ref *1> ClientRequest {
    _events: [Object: null prototype] {
      socket: [Function (anonymous)],
      abort: [Function (anonymous)],
      aborted: [Function (anonymous)],
      connect: [Function (anonymous)],
      error: [Function (anonymous)],
      timeout: [Function (anonymous)],
      prefinish: [Function: requestOnPrefinish]
    },
    _eventsCount: 7,
    _maxListeners: undefined,
    outputData: [],
    outputSize: 0,
    writable: true,
    destroyed: false,
    _last: true,
    chunkedEncoding: false,
    shouldKeepAlive: false,
    _defaultKeepAlive: true,
    useChunkedEncodingByDefault: false,
    sendDate: false,
    _removedConnection: false,
    _removedContLen: false,
    _removedTE: false,
    _contentLength: 0,
    _hasBody: true,
    _trailer: '',
    finished: true,
    _headerSent: true,
    _closed: false,
    socket: TLSSocket {
      _tlsOptions: [Object],
      _secureEstablished: true,
      _securePending: false,
      _newSessionPending: false,
      _controlReleased: true,
      secureConnecting: false,
      _SNICallback: null,
      servername: 'taxonomy-directus-service.test.xxx.com',
      alpnProtocol: false,
      authorized: true,
      authorizationError: null,
      encrypted: true,
      _events: [Object: null prototype],
      _eventsCount: 10,
      connecting: false,
      _hadError: false,
      _parent: null,
      _host: 'taxonomy-directus-service.test.xxx.com',
      _readableState: [ReadableState],
      _maxListeners: undefined,
      _writableState: [WritableState],
      allowHalfOpen: false,
      _sockname: null,
      _pendingData: null,
      _pendingEncoding: '',
      server: undefined,
      _server: null,
      ssl: [TLSWrap],
      _requestCert: true,
      _rejectUnauthorized: true,
      parser: null,
      _httpMessage: [Circular *1],
      [Symbol(res)]: [TLSWrap],
      [Symbol(verified)]: true,
      [Symbol(pendingSession)]: null,
      [Symbol(async_id_symbol)]: 56,
      [Symbol(kHandle)]: [TLSWrap],
      [Symbol(kSetNoDelay)]: false,
      [Symbol(lastWriteQueueSize)]: 0,
      [Symbol(timeout)]: null,
      [Symbol(kBuffer)]: null,
      [Symbol(kBufferCb)]: null,
      [Symbol(kBufferGen)]: null,
      [Symbol(kCapture)]: false,
      [Symbol(kBytesRead)]: 0,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(connect-options)]: [Object],
      [Symbol(RequestTimeout)]: undefined
    },
    _header: 'GET /collections HTTP/1.1\r\n' +
      'Accept: application/json, text/plain, */*\r\n' +
      'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6IjJlNTY0N2NkLWFhNTEtNDJmNy05YWExLTBjYzM2NGUwNDFjMiIsImlhdCI6MTYyMDY3NDE4OCwiZXhwIjoxNjIwNjc1MDg4fQ.5--203VXNU1kdKV7HUnWm6cmQ1GkmzYweYlmLAuh988\r\n' +
      'Cookie: directus--session=undefined\r\n' +
      'User-Agent: axios/0.21.1\r\n' +
      'Host: taxonomy-directus-service.test.xxx.com\r\n' +
      'Connection: close\r\n' +
      '\r\n',
    _keepAliveTimeout: 0,
    _onPendingData: [Function: noopPendingOutput],
    agent: Agent {
      _events: [Object: null prototype],
      _eventsCount: 2,
      _maxListeners: undefined,
      defaultPort: 443,
      protocol: 'https:',
      options: [Object],
      requests: {},
      sockets: [Object],
      freeSockets: {},
      keepAliveMsecs: 1000,
      keepAlive: false,
      maxSockets: Infinity,
      maxFreeSockets: 256,
      scheduling: 'fifo',
      maxTotalSockets: Infinity,
      totalSocketCount: 1,
      maxCachedSessions: 100,
      _sessionCache: [Object],
      [Symbol(kCapture)]: false
    },
    socketPath: undefined,
    method: 'GET',
    maxHeaderSize: undefined,
    insecureHTTPParser: undefined,
    path: '/collections',
    _ended: true,
    res: IncomingMessage {
      _readableState: [ReadableState],
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      socket: [TLSSocket],
      httpVersionMajor: 1,
      httpVersionMinor: 1,
      httpVersion: '1.1',
      complete: true,
      rawHeaders: [Array],
      rawTrailers: [],
      aborted: false,
      upgrade: false,
      url: '',
      method: null,
      statusCode: 404,
      statusMessage: 'Not Found',
      client: [TLSSocket],
      _consuming: false,
      _dumped: false,
      req: [Circular *1],
      responseUrl: 'https://taxonomy-directus-service.test.xxx.com/collections',
      redirects: [],
      [Symbol(kCapture)]: false,
      [Symbol(kHeaders)]: [Object],
      [Symbol(kHeadersCount)]: 14,
      [Symbol(kTrailers)]: null,
      [Symbol(kTrailersCount)]: 0,
      [Symbol(RequestTimeout)]: undefined
    },
    aborted: false,
    timeoutCb: null,
    upgradeOrConnect: false,
    parser: null,
    maxHeadersCount: null,
    reusedSocket: false,
    host: 'taxonomy-directus-service.test.xxx.com',
    protocol: 'https:',
    _redirectable: Writable {
      _writableState: [WritableState],
      _events: [Object: null prototype],
      _eventsCount: 2,
      _maxListeners: undefined,
      _options: [Object],
      _ended: true,
      _ending: true,
      _redirectCount: 0,
      _redirects: [],
      _requestBodyLength: 0,
      _requestBodyBuffers: [],
      _onNativeResponse: [Function (anonymous)],
      _currentRequest: [Circular *1],
      _currentUrl: 'https://taxonomy-directus-service.test.xxx.com/collections',
      [Symbol(kCapture)]: false
    },
    [Symbol(kCapture)]: false,
    [Symbol(kNeedDrain)]: false,
    [Symbol(corked)]: 0,
    [Symbol(kOutHeaders)]: [Object: null prototype] {
      accept: [Array],
      authorization: [Array],
      cookie: [Array],
      'user-agent': [Array],
      host: [Array]
    }
  },
  response: {
    status: 404,
    statusText: 'Not Found',
    headers: {
      date: 'Mon, 10 May 2021 19:18:39 GMT',
      'content-type': 'application/json',
      'content-length': '42',
      connection: 'close',
      server: 'Apache/2.4.38 (Debian)',
      'x-powered-by': 'PHP/7.3.20',
      'access-control-allow-origin': '*'
    },
    config: {
      url: '/collections',
      method: 'get',
      headers: [Object],
      baseURL: 'https://taxonomy-directus-service.test.xxx.com/',
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      adapter: [Function: httpAdapter],
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: -1,
      maxBodyLength: -1,
      validateStatus: [Function: validateStatus],
      data: undefined
    },
    request: <ref *1> ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 7,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: false,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: false,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      _contentLength: 0,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: false,
      socket: [TLSSocket],
      _header: 'GET /collections HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Authorization: Bearer FhNTEtNDJmNy05YWExLTBjYzM2NGUwNDFjMiIsImlhdCI6MTYyMDY3NDE4OCwiZXhwIjoxNjIwNjc1MDg4fQ.5--203VXNU1kdKV7HUnWm6cmQ1GkmzYweYlmLAuh988\r\n' +
        'Cookie: directus--session=undefined\r\n' +
        'User-Agent: axios/0.21.1\r\n' +
        'Host: taxonomy-directus-service.test.xxx.com\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: noopPendingOutput],
      agent: [Agent],
      socketPath: undefined,
      method: 'GET',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      path: '/collections',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: 'taxonomy-directus-service.test.xxx.com',
      protocol: 'https:',
      _redirectable: [Writable],
      [Symbol(kCapture)]: false,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype]
    },
    data: { error: [Object] }
  },
  isAxiosError: true,
  toJSON: [Function: toJSON],
  context: [Object: null prototype] { skipCollections: [] }
}

@rijkvanzanten any thoughts on how to resolve?

Incorrect default value for v8 array type fields

When migrating a schema from v8 that is defined as an array and contains a default value, the creation of the table on v9 fails complaining that the default value is not a valid JSON.

Example field definition to reproduce:
"field_name": { "field": "field_name", "datatype": "VARCHAR", "type": "array", "sort": 2300, "interface": "multiselect", "required": true, "default_value": "some_option1", "options": { "choices": { "some_option1": "some_option1", "some_option2": "some_option2", "some_option3": "some_option3", "some_option4": "some_option4", }, "allow_other": false, "draggable": false, "wrap": false, "formatting": true, "single": false }, "locked": 0, "width": "full", "length": "255" },

Feature request: update interface enum

Since the last update, some interfaces have changed their naming (many-to-one, divider, etc). The migration tool should reflect those name changes.

Migration stops at repeater options

Using latest v8 and v9 RC93. I think this is a repeater field issue

TypeError: fieldDetails.options.fields.map is not a function
    at migrateFieldOptions (file:///home/support/migration-tool/tasks/schema.js:84:43)
    at file:///home/support/migration-tool/tasks/schema.js:203:22
    at Array.map (<anonymous>)
    at Task.task (file:///home/support/migration-tool/tasks/schema.js:177:48)
    at /home/support/migration-tool/node_modules/listr/lib/task.js:167:30
    at processTicksAndRejections (internal/process/task_queues.js:95:5) {
  context: [Object: null prototype] {
    completedSteps: {
      schema: true,
      collections: false,
      files: false,
      roles: false,
      users: false,
      relationsv8: false,
      relations: false,
      data: false,
      completed: false
    },

Error while migrating files

Hey guys!

I'm getting an error when the migration tool reaches the Uploading Files task.

TypeError: Cannot read property 'split' of undefined
    at Task.task (file:///.../migration-tool/tasks/files.js:62:32)

Looking at the code, it's trying to split the string using the property data.asset_url that doesn't exist in the File Object. Shouldn't it be using the property data.full_url instead?

pass file id to v9 file import api to preserve same file id from v8

hello, do not know if this considered a bug but here is what is causing trouble for me, so inside my article collection there is wysiwyg and images in it, each image link by url with sth like this https://directus.myapp/uploads/_/originals/<id>.<extension>, when using the migration tool, everything works as expected, all the data and the images. I'm using gatsby and because i do not want images within the wysiwyg content link directly to directus, so in the old directus v8 I do some regex matching to get the id from url of the format https://directus.myapp/uploads/_/originals/<id>.<extension> and use the gatsby-image plugin to get the image and react-html-parser to change the img tag there. When upgraded to v9, the id that got from regex matching does not work anymore because they are different after file import api from v9.

i looked into the code and v9 doc and I think maybe you can pass additional id to uploadBatch function here so that it uses the old id from v8? Sorry that I have yet try to update uploadBatch function locally to test this because I have been migrating from v8 to v9 locally of course so it does not affect my production one, and update some schema and data along the way so it would take some time for me to try this since I think it would requires me to re-do the v8 to v9 migrate, so if passing additional id to the function does not affect the tool much and if it does fix this problem maybe it would benefit to others as well if they had similar issue like me, or any suggestion for me also good as well

edit0: supposed to be filename_disk instead but without extension from v8 since id from v8 is integer

writeContext() doesn't create context JSON files if none exist

When no JSON files are pre-created for data, these errors occur

[Error: ENOENT: no such file or directory, open './context/state/schema.json'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: './context/state/schema.json',
  context: [Object: null prototype] {
    completedSteps: {
      schema: true,
      collections: false,
      files: false,
      roles: false,
      users: false,
      relationsv8: false,
      relations: false,
      data: false,
      completed: false
    },
    collectionsV9: [],
    skipCollections: [],
    collections: [ [Object], [Object], [Object], [Object] ]
  }
}

At least on Windows. writeContext() doesn't handle creation.

Repeater options are lost during migration

After migration, the JSON content for the field is still there, but the interface/options for the repeater field are missing, so the UI renders each item as "Empty Item".

TypeError: Cannot read property 'split' of undefined

Hi all,

We're trying to migrate a v8 stack of Directus to v9.
Both run in Docker, and we're attempting to migrate from MySQL to PgSQL.
Unfortunately, we seem to have trouble with files, and I was hoping someone may be able to shed some light please?

 ❯ Migration Files
    ✔ Getting File Count
    ❯ Uploading Files
      ✖ Uploading files 1—100
        → Cannot read property 'split' of undefined
        Uploading files 101—200
        Uploading files 201—300
        Uploading files 301—400
        Uploading files 401—500
    Migrating Users
    Migrating Data
TypeError: Cannot read property 'split' of undefined

Status Migration fails creating unarchive values

this piece of code in tasks/schema.jsfails if there isn't a 3rd status value or at least one that matches the unarchive value query. We had a collection with two statuses - published and deleted and it wasn't til I added back a 3rd one that the migration moved on. Took a while to work out why!

                                        Object.entries(collection.fields).find(([field, details]) => {
                                                return (details.type || "").toLowerCase() === "sort";
                                        })?.field || null,
                                ...(statusField
                                        ? {
                                                        archive_field: statusField.field,
                                                        archive_value: Object.values(
                                                                statusField.options.status_mapping
                                                        ).find((option) => option.soft_delete)?.value,
                                                        unarchive_value: Object.values(
                                                                statusField.options.status_mapping
                                                        ).find((option) => !option.soft_delete && !option.published).value,
                                          }
                                        : {}),```

Migration Fails on "Migrating Relations"

While migrating from V8 to V9 I encounter the following error while "migrating relations"

Cannot read propery 'map' of undefined

I have tried specifying the order in which to load collections to make sure a collection is only loaded after tables used by its foreignkey constraints already exist, but no luck.

The issue seems to be that the filter at tasks/relations.js:39 has no matches:

const relationsV9 = context.relationsV8
    .filter((relation) => {
      return (
        (relation.collection_many.startsWith("directus_") &&
          relation.collection_one.startsWith("directus_")) === false
      );
    })

I can see in state/relationsV8.json that this is the case, there are no relations meeting that condition.

@moekify What's the meaning of that condition, why are we filtering down to a relation that has ~~collection_one~~ collection_many starting with "directus_" but a collection_one that does not? What would happen to if I had a many-to-one defined between two collections that I invented, wouldn't that collection be excluded from this filter?

Thank you so much for your time on this! Happy to contribute if this turns out to be a bug, but grateful for any clarification that can help me get migrated.

Migration crashing when running data task: TypeError: Cannot read property '4' of undefined

The data migration is failing with the following error:
❯ Inserting Data
✖ text_classification
→ Cannot read property '4' of undefined
..
Migrating Users
Migration Files
TypeError: Cannot read property '4' of undefined
at file:///Users/acho/workdir/migration/migration-tool/tasks/data.js:87:24
at Array.map ()
at insertBatch (file:///Users/acho/workdir/migration/migration-tool/tasks/data.js:83:32)
at processTicksAndRejections (node:internal/process/task_queues:93:5)
at async Task.task (file:///Users/acho/workdir/migration/migration-tool/tasks/data.js:49:4) {
context: [Object: null prototype] {
skipCollections: [
'testing'
],

SyntaxError: Unexpected identifier

Hey guys,
thanks for the migration tool. Unfortunately I get the error "SyntaxError: Unexpected identifier" when starting, although listr was installed via npm ci. Do you have an idea what could be wrong?

Output starting with node index.js:

import Listr from "listr";
       ^^^^^
SyntaxError: Unexpected identifier
    at Module._compile (internal/modules/cjs/loader.js:723:23)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
    at Module.load (internal/modules/cjs/loader.js:653:32)
    at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
    at Function.Module._load (internal/modules/cjs/loader.js:585:3)
    at Function.Module.runMain (internal/modules/cjs/loader.js:831:12)
    at startup (internal/bootstrap/node.js:283:19)
    at bootstrapNodeJSCore (internal/bootstrap/node.js:623:3)

Migration crashing with read property '0' of undefined when running Users task

✨ Migrating https://taxonomy-directus-service.test.xxx.com (v8) to https://taxonomy-directus-service.dev.xxx.com (v9)...
✔ Migrating Schema
❯ Migrating Users
✔ Downloading Roles
✔ Creating Roles
✔ Downloading Users
✖ Creating Users
→ Cannot read property '0' of undefined
Migrating Data
Migration Files
TypeError: Cannot read property '0' of undefined
at file:///Users/acho/workdir/migration/migration-tool/tasks/users.js:92:52
at Array.forEach ()
at Task.createUsers [as task] (file:///Users/acho/workdir/migration/migration-tool/tasks/users.js:91:16)
at processTicksAndRejections (node:internal/process/task_queues:93:5) {
context: [Object: null prototype] {

Error migrating file relations: "foreign key constraint cannot be implemented"

First I was getting 400 bad request error. Then I tried the branch by @hakan-erdem-temiz based on his PR from a month ago. This got me past the 400 and migrated many of the relations until it hit a new 500. I think this is the only collection that uses the 02M "Files" field where it seems to be snagging.

Here's the log from the script:

 > Migrating Schema
    √ Downloading Schema
    √ Creating Collections
    × Migrating Relations
      → Request failed with status code 500
    Migration Files
    Migrating Users
    Migrating Data
Error: Request failed with status code 500
    at createError (C:\Users\liamh\dev\directus-migration-tool-2\node_modules\axios\lib\core\createError.js:16:15)
    at settle (C:\Users\liamh\dev\directus-migration-tool-2\node_modules\axios\lib\core\settle.js:17:12)
    at IncomingMessage.handleStreamEnd (C:\Users\liamh\dev\directus-migration-tool-2\node_modules\axios\lib\adapters\http.js:260:11)
    at IncomingMessage.emit (events.js:387:35)
    at endReadableNT (internal/streams/readable.js:1317:12)
    at processTicksAndRejections (internal/process/task_queues.js:82:21) {
  config: {
    url: '/relations',
    method: 'post',
    data: '{"meta":{"many_collection":"projects_directus_files","many_field":"directus_files_id","many_primary":"id","one_collection":"directus_files","one_field":"projects","one_primary":"id","junction_field":"projects_id"},"field":"directus_files_id","collection":"projects_directus_files","related_collection":"directus_files","schema":null}',

...and here's the log from the v9 instance:

21:38:44 🚨 error alter table "projects_directus_files" add constraint "projects_directus_files_directus_files_id_foreign" foreign key ("directus_files_id") references "directus_files" ("id") - foreign key constraint "projects_directus_files_directus_files_id_foreign" cannot be implemented
error: alter table "projects_directus_files" add constraint "projects_directus_files_directus_files_id_foreign" foreign key ("directus_files_id") references "directus_files" ("id") - foreign key constraint "projects_directus_files_directus_files_id_foreign" cannot be implemented
    at Parser.parseErrorMessage (/home/node/.npm/_npx/ebcc468e2b0f2b61/node_modules/pg-protocol/dist/parser.js:287:98)
    at Parser.handlePacket (/home/node/.npm/_npx/ebcc468e2b0f2b61/node_modules/pg-protocol/dist/parser.js:126:29)
    at Parser.parse (/home/node/.npm/_npx/ebcc468e2b0f2b61/node_modules/pg-protocol/dist/parser.js:39:38)
    at TLSSocket.<anonymous> (/home/node/.npm/_npx/ebcc468e2b0f2b61/node_modules/pg-protocol/dist/index.js:11:42)
    at TLSSocket.emit (node:events:394:28)
    at addChunk (node:internal/streams/readable:312:12)
    at readableAddChunk (node:internal/streams/readable:287:9)
    at TLSSocket.Readable.push (node:internal/streams/readable:226:10)
    at TLSWrap.onStreamRead (node:internal/stream_base_commons:190:23)

Directus 8 v8.7.2 on MySQL 8
Directus 9 v9.0.0-rc83 on PostgreSQL 12

crash with status 503 service unavailable during "Migration Files, Uploading Files"

migration machine:

MacBook Pro v11.1 big sur
nodejs: v14.15.4

I am migrating v8 to v9 on my local machine, so I take v8 backup from production, make sure everything is restored properly(data, files etc) and after migrate v8 to v9 on local machine, taking v9 backup from local and migrate/restore it onto production

below is the docker-compose.yml setup on my local machine to do the migration

version: '2.4'

volumes:
  directus_db:
  directus_uploads:
  directus_v8_uploads:

networks:
  directus_db:
  directus_v8_db:

services:
  directus_db:
    image: postgres:12-alpine
    container_name: directus_db
    networks:
      - directus_db
    volumes:
      - directus_db:/var/lib/postgresql/data
    environment:
      - POSTGRES_USER=directus
      - POSTGRES_DB=directus
      - POSTGRES_PASSWORD=directus
      - PGDATA=/var/lib/postgresql/data/pg_data
      - POSTGRES_INITDB_ARGS=--data-checksums
      - POSTGRES_INITDB_WALDIR=/var/lib/postgresql/data/pg_wal

  directus:
    image: directus/directus:v9.0.0-rc.53
    networks:
      - directus_db
    ports:
      - 8080:8055
    environment:
      DIRECTUS_APP_ENV: production
      DB_CLIENT: pg
      DB_HOST: directus_db
      DB_PORT: 5432
      DB_USER: directus
      DB_PASSWORD: directus
      DB_DATABASE: directus
    volumes:
      - ./data/config:/var/directus/config:ro
      - directus_uploads:/var/directus/public/uploads

  mysql:
    image: mysql:5.7
    container_name: directus_v8_db
    networks:
      - directus_v8_db
    environment:
      MYSQL_USER: directus
      MYSQL_DATABASE: directus
      MYSQL_PASSWORD: directus
      MYSQL_ROOT_PASSWORD: directus

  directus_v8:
    image: directus/directus:v8-apache
    networks:
      - directus_v8_db
    ports:
      - 8081:80
    environment:
      DIRECTUS_APP_ENV: production
      DIRECTUS_DATABASE_HOST: mysql
      DIRECTUS_DATABASE_PORT: 3306
      DIRECTUS_DATABASE_NAME: directus
      DIRECTUS_DATABASE_USERNAME: directus
      DIRECTUS_DATABASE_PASSWORD: directus
      DIRECTUS_AUTH_PUBLICKEY: some random secret
      DIRECTUS_AUTH_SECRETKEY: another random secret
    volumes:
      - directus_v8_uploads:/var/directus/public/uploads

running migration in the console

node index.js
Migrating http://localhost:8081/my-awesome-project (v8) to http://localhost:8080 (v9)...
  ✔ Migrating Schema
  ❯ Migration Files
    ✔ Getting File Count
    ❯ Uploading Files
      ✖ Uploading files 1—100
        → Request failed with status code 503
        Uploading files 101—200
        Uploading files 101—200
        Uploading files 201—300
        Uploading files 301—400
        Uploading files 401—500
        Uploading files 501—600
        Uploading files 601—700
        Uploading files 701—800
        Uploading files 801—900
        Uploading files 901—1000
        Uploading files 1001—1100
        Uploading files 1101—1200
        Uploading files 1201—1300
        Uploading files 1301—1400
        Uploading files 1401—1500
        Uploading files 1501—1600
    Migrating Users
    Migrating Data
Error: Request failed with status code 503
    at createError (/private/tmp/my-awesome-project-backups/migration-tool/node_modules/axios/lib/core/createError.js:16:15)
    at settle (/private/tmp/my-awesome-project-backups/migration-tool/node_modules/axios/lib/core/settle.js:17:12)
    at IncomingMessage.handleStreamEnd (/private/tmp/my-awesome-project-backups/migration-tool/node_modules/axios/lib/adapters/http.js:260:11)
    at IncomingMessage.emit (events.js:327:22)
    at endReadableNT (internal/streams/readable.js:1327:12)
    at processTicksAndRejections (internal/process/task_queues.js:80:21) {
  config: {
    url: '/files/import',
    method: 'post',
    data: '{"url":"http://localhost:8081/my-awesome-project/assets/ah39dolmx8o4sckg","data":{"filename_download":"book(1).svg","title":"Book(1)","description":""}}',
    headers: {
      Accept: 'application/json, text/plain, */*',
      'Content-Type': 'application/json;charset=utf-8',
      Authorization: 'Bearer 9C18B2F8-AA92-403A-9DBC-1CC3F3288368',
      'User-Agent': 'axios/0.21.1',
      'Content-Length': 142
    },
    baseURL: 'http://localhost:8080',
    transformRequest: [ [Function: transformRequest] ],
    transformResponse: [ [Function: transformResponse] ],
    timeout: 0,
    adapter: [Function: httpAdapter],
    xsrfCookieName: 'XSRF-TOKEN',
    xsrfHeaderName: 'X-XSRF-TOKEN',
    maxContentLength: -1,
    maxBodyLength: -1,
    validateStatus: [Function: validateStatus]
  },
  request: <ref *1> ClientRequest {
    _events: [Object: null prototype] {
      socket: [Function (anonymous)],
      abort: [Function (anonymous)],
      aborted: [Function (anonymous)],
      connect: [Function (anonymous)],
      error: [Function (anonymous)],
      timeout: [Function (anonymous)],
      prefinish: [Function: requestOnPrefinish]
    },
    _eventsCount: 7,
    _maxListeners: undefined,
    outputData: [],
    outputSize: 0,
    writable: true,
    destroyed: false,
    _last: true,
    chunkedEncoding: false,
    shouldKeepAlive: false,
    _defaultKeepAlive: true,
    useChunkedEncodingByDefault: true,
    sendDate: false,
    _removedConnection: false,
    _removedContLen: false,
    _removedTE: false,
    _contentLength: null,
    _hasBody: true,
    _trailer: '',
    finished: true,
    _headerSent: true,
    socket: Socket {
      connecting: false,
      _hadError: false,
      _parent: null,
      _host: 'localhost',
      _readableState: [ReadableState],
      _events: [Object: null prototype],
      _eventsCount: 7,
      _maxListeners: undefined,
      _writableState: [WritableState],
      allowHalfOpen: false,
      _sockname: null,
      _pendingData: null,
      _pendingEncoding: '',
      server: null,
      _server: null,
      parser: null,
      _httpMessage: [Circular *1],
      [Symbol(async_id_symbol)]: 303,
      [Symbol(kHandle)]: [TCP],
      [Symbol(kSetNoDelay)]: false,
      [Symbol(lastWriteQueueSize)]: 0,
      [Symbol(timeout)]: null,
      [Symbol(kBuffer)]: null,
      [Symbol(kBufferCb)]: null,
      [Symbol(kBufferGen)]: null,
      [Symbol(kCapture)]: false,
      [Symbol(kBytesRead)]: 0,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(RequestTimeout)]: undefined
    },
    _header: 'POST /files/import HTTP/1.1\r\n' +
      'Accept: application/json, text/plain, */*\r\n' +
      'Content-Type: application/json;charset=utf-8\r\n' +
      'Authorization: Bearer 9C18B2F8-AA92-403A-9DBC-1CC3F3288368\r\n' +
      'User-Agent: axios/0.21.1\r\n' +
      'Content-Length: 142\r\n' +
      'Host: localhost:8080\r\n' +
      'Connection: close\r\n' +
      '\r\n',
    _keepAliveTimeout: 0,
    _onPendingData: [Function: noopPendingOutput],
    agent: Agent {
      _events: [Object: null prototype],
      _eventsCount: 2,
      _maxListeners: undefined,
      defaultPort: 80,
      protocol: 'http:',
      options: [Object],
      requests: {},
      sockets: [Object],
      freeSockets: {},
      keepAliveMsecs: 1000,
      keepAlive: false,
      maxSockets: Infinity,
      maxFreeSockets: 256,
      scheduling: 'fifo',
      maxTotalSockets: Infinity,
      totalSocketCount: 1,
      [Symbol(kCapture)]: false
    },
    socketPath: undefined,
    method: 'POST',
    maxHeaderSize: undefined,
    insecureHTTPParser: undefined,
    path: '/files/import',
    _ended: true,
    res: IncomingMessage {
      _readableState: [ReadableState],
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      socket: [Socket],
      httpVersionMajor: 1,
      httpVersionMinor: 1,
      httpVersion: '1.1',
      complete: true,
      headers: [Object],
      rawHeaders: [Array],
      trailers: {},
      rawTrailers: [],
      aborted: false,
      upgrade: false,
      url: '',
      method: null,
      statusCode: 503,
      statusMessage: 'Service Unavailable',
      client: [Socket],
      _consuming: false,
      _dumped: false,
      req: [Circular *1],
      responseUrl: 'http://localhost:8080/files/import',
      redirects: [],
      [Symbol(kCapture)]: false,
      [Symbol(RequestTimeout)]: undefined
    },
    aborted: false,
    timeoutCb: null,
    upgradeOrConnect: false,
    parser: null,
    maxHeadersCount: null,
    reusedSocket: false,
    host: 'localhost',
    protocol: 'http:',
    _redirectable: Writable {
      _writableState: [WritableState],
      _events: [Object: null prototype],
      _eventsCount: 2,
      _maxListeners: undefined,
      _options: [Object],
      _ended: true,
      _ending: true,
      _redirectCount: 0,
      _redirects: [],
      _requestBodyLength: 142,
      _requestBodyBuffers: [],
      _onNativeResponse: [Function (anonymous)],
      _currentRequest: [Circular *1],
      _currentUrl: 'http://localhost:8080/files/import',
      [Symbol(kCapture)]: false
    },
    [Symbol(kCapture)]: false,
    [Symbol(kNeedDrain)]: false,
    [Symbol(corked)]: 0,
    [Symbol(kOutHeaders)]: [Object: null prototype] {
      accept: [Array],
      'content-type': [Array],
      authorization: [Array],
      'user-agent': [Array],
      'content-length': [Array],
      host: [Array]
    }
  },
  response: {
    status: 503,
    statusText: 'Service Unavailable',
    headers: {
      'x-powered-by': 'Directus',
      vary: 'Origin',
      'access-control-allow-credentials': 'true',
      'access-control-expose-headers': 'Content-Range',
      'content-type': 'application/json; charset=utf-8',
      'content-length': '184',
      etag: 'W/"b8-0FjMpR/HtK8aMYHaOFBn5EJovcs"',
      date: 'Sun, 04 Apr 2021 07:36:20 GMT',
      connection: 'close'
    },
    config: {
      url: '/files/import',
      method: 'post',
      data: '{"url":"http://localhost:8081/my-awesome-project/assets/ah39dolmx8o4sckg","data":{"filename_download":"book(1).svg","title":"Book(1)","description":""}}',
      headers: [Object],
      baseURL: 'http://localhost:8080',
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      adapter: [Function: httpAdapter],
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: -1,
      maxBodyLength: -1,
      validateStatus: [Function: validateStatus]
    },
    request: <ref *1> ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 7,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: false,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: true,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      _contentLength: null,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      socket: [Socket],
      _header: 'POST /files/import HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Content-Type: application/json;charset=utf-8\r\n' +
        'Authorization: Bearer 9C18B2F8-AA92-403A-9DBC-1CC3F3288368\r\n' +
        'User-Agent: axios/0.21.1\r\n' +
        'Content-Length: 142\r\n' +
        'Host: localhost:8080\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: noopPendingOutput],
      agent: [Agent],
      socketPath: undefined,
      method: 'POST',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      path: '/files/import',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: 'localhost',
      protocol: 'http:',
      _redirectable: [Writable],
      [Symbol(kCapture)]: false,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype]
    },
    data: { errors: [Array] }
  },
  isAxiosError: true,
  toJSON: [Function: toJSON],
  context: [Object: null prototype] {
    collections: [ [Object], [Object], [Object], [Object], [Object], [Object] ],
    relations: [
      [Object], [Object],
      [Object], [Object],
      [Object], [Object],
      [Object], [Object],
      [Object], [Object],
      [Object]
    ],
    fileCount: 1551,
    fileMap: {}
  }
}

the migration crashed half way and only data model are being imported from v8 to v9

edit0: I also made sure the url to the file exists by copy and paste it into browser http://localhost:8081/my-awesome-project/assets/ah39dolmx8o4sckg and both user token are admin level

edit1: Sorry for silly mistake, so I check logs from directus v9 container, it is trying to download the file from 127.0.0.1:8081 which the directus v9 has no access to directus v8, I will fix it and probably close this issue

Add a way to retry on failed inserts

Right now, any failed insert crashes and stops the migration tool. We should catch any failed requests, and try them again later and / or report what failed

Migrating Relations Fails - Request failed with status code 400

The migration was often failing prior to the migrating of the relations, but after learning that it was caused by existing users as well as tokens expiring, I have gotten to the point where the migrating of relations fails. Although this sucks, it's progress. I now need to understand how to resolve the failed migration of relations.

However, it is difficult to debug. Here is what I have to work with:

Feedback from Migration Tool:

` ✔ Saving Relations context

✖ Migrating Relations
  → Request failed with status code 400
  Saving Relations context
Migrating Data
Save final context
Error: Request failed with status code 400`

Feedback from Directus Docker instance:

`23:49:51 ✨ request completed POST 200 /relations 62ms

23:49:51 ✨ request completed POST 200 /relations 70ms
23:49:51 ✨ request completed POST 200 /relations 115ms
23:49:51 ✨ request completed POST 400 /relations 59ms`

I am not sure how to debug any further with just this information. I am unable to know what the issue is.

However, I have attempted to recreate the POST via Postman and notice there is not error. It seems to run fine and returns a schema by the looks of it.

Any help to making progress in resolving this would be much appreciated.

Migrating relationships

So after some investigating and alternative thinking, the way the tool migrates relationships might need a little more love. I've looked at the code myself and only slightly grasp the fact that it's hard. 😂

Have a thread here where I went through my configuration in detail directus/directus#7207

// v8

collection_many   field_many  collection_one  one_field
__________________________________________________________

blog              content     blog_content    NULL


// v9

many_collection   many_field  one_collection  one_field
__________________________________________________________

blog_content      post_id     content         blog_content

moekify (don't want to annoy with notifications so no @) did say some clean up may be required but I have many relations that look like this.

Are they technically invalid despite the fact they worked in v8? Because after my refresher on relationships, it is wrong is v8...
Is there an obvious pattern I haven't noticed that might allow me to correct them on the fly with some code? I'm pretty sure most if not all my relations are o2m so that fact might allow me to do so..?

Edit:

Similarly for an m2m with a junction table

// v8

collection_many           field_many            collection_one    one_field   junction_field
____________________________________________________________________________________________

portfolio_item_types      portfolio_id          portfolio         type        portfolio_types_id
portfolio_item_types      portfolio_types_id    portfolio_types   portfolio   portfolio_id


// v9

many_collection           many_field            one_collection    one_field   junction_field
____________________________________________________________________________________________

portfolio_item_types      portfolio_types_id    portfolio_types   null        portfolio_id
portfolio_item_types      portfolio_id          portfolio         type        portfolio_types_id

Like, this is the payload

{
	"meta": {
		"many_collection": "portfolio_item_types",
		"many_field": "portfolio_id",
		"one_collection": "portfolio",
		"one_field": "type",
		"junction_field": "portfolio_types_id"
	},
	"field": "portfolio_id",
	"collection": "portfolio_item_types",
	"related_collection": "portfolio",
	"schema": null
}

Response:

{
  "errors": [
    {
      "message": "Field \"portfolio_id\" in collection \"portfolio_item_types\" already has an associated relationship",
      "extensions": {
        "code": "INVALID_PAYLOAD"
      }
    }
  ]
}

Migrating translations : What about the parsing of languages code (ex: "fr", "en") to languages ids ?

Hello,

In V8, to translate the contents I made something like that (in my memory it was a good practice) :

page (id, translations (translations with id<->page_translations.page))
page_translations (id, page (int), language_code (language))

So, when I finally succeeded to import my data to the V9 with the migration-tool, I noticed that my collections kept this old schema, and there didn't had any conversion to the new schema with language_id etc of Directus v9.

So, I don't have any idea of how I'm going to restore all my translations because now they do not appear at all (blank space below "Translations" for exemple in each of my page items).

Is it something that someone is working on ?

Migration failed on `Migrating Relations`

Trying to migrate and have this errors. Any ideas where and what I should debug?

  ❯ Migrating Relations
    ✔ Get v8 Relations
    ✔ Saving Relations context
    ✖ Migrating Relations
      → Request failed with status code 500
      Saving Relations context
    Migrating Data
    Save final context
Error: Request failed with status code 500
    at createError (/home/ash/Soft/directus-migration-tool/node_modules/axios/lib/core/createError.js:16:15)
    at settle (/home/ash/Soft/directus-migration-tool/node_modules/axios/lib/core/settle.js:17:12)
    at IncomingMessage.handleStreamEnd (/home/ash/Soft/directus-migration-tool/node_modules/axios/lib/adapters/http.js:260:11)
    at IncomingMessage.emit (events.js:412:35)
    at endReadableNT (internal/streams/readable.js:1317:12)
    at processTicksAndRejections (internal/process/task_queues.js:82:21) {
  config: {
    url: '/relations',
    method: 'post',
    data: '{"meta":{"many_collection":"we_offer","many_field":"title","one_collection":"about","one_field":"we_offer","junction_field":null},"field":"title","collection":"we_offer","related_collection":"about","schema":null}',

Server logs:

21:26:54 🚨 error alter table "we_offer" add constraint "we_offer_title_fore
ign" foreign key ("title") references "about" ("id") - foreign key constrain
t "we_offer_title_foreign" cannot be implemented
error: alter table "we_offer" add constraint "we_offer_title_foreign" foreig
n key ("title") references "about" ("id") - foreign key constraint "we_offer
_title_foreign" cannot be implemented
    at Parser.parseErrorMessage (/home/ash/projects/shaini/shaini.kg/directu
s-shaini/node_modules/pg-protocol/dist/parser.js:287:98)
    at Parser.handlePacket (/home/ash/projects/shaini/shaini.kg/directus-sha
ini/node_modules/pg-protocol/dist/parser.js:126:29)
    at Parser.parse (/home/ash/projects/shaini/shaini.kg/directus-shaini/nod
e_modules/pg-protocol/dist/parser.js:39:38)
    at Socket.<anonymous> (/home/ash/projects/shaini/shaini.kg/directus-shai
ni/node_modules/pg-protocol/dist/index.js:11:42)
    at Socket.emit (events.js:400:28)
    at addChunk (internal/streams/readable.js:290:12)
    at readableAddChunk (internal/streams/readable.js:265:9)
    at Socket.Readable.push (internal/streams/readable.js:204:10)
    at TCP.onStreamRead (internal/stream_base_commons.js:188:23)
21:26:54 ✨ request errored POST 500 /relations 23ms

Error while migrating relations -

yes , i have followed these steps :

i have created a empty database in mysql
import sql dumb files one-by-one
ran v8 directus project and got user auth token by api
ran v9 new fresh project and got user auth token by api
ran migration-tool , its now giving error on migrating relations

v8 Database :

Error Log File While Migration:

V9 =>
/relations
undefined
undefined
V9 <=
400
{
"errors": [
    {
        "message": "Field \"output_destination_id\" in collection \"output_destination_deal_pool\" already has an associated relationship",
        "extensions": {
            "code": "INVALID_PAYLOAD"
        }
    }
]

}

Error:
V9 =>
/relations
undefined
undefined
V9 <=
400
{
"errors": [
{
"message": "Field "output_destination_id" in collection "output_destination_deal_pool" already has an associated relationship",
"extensions": {
"code": "INVALID_PAYLOAD"
}
}
]
}

at file:///C:/Omair%20Folder/Office%20Projects/Directus%20Database%20Migration%20Project/migration-tool/api.js:41:9
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async migrateRelationsData (file:///C:/Omair%20Folder/Office%20Projects/Directus%20Database%20Migration%20Project/migration-tool/tasks/relations.js:116:3)

Crash when migrating users database

Stupidly, I initiated a new database in directus 9 using the same admin email address as my admin user in directus 8. That resulted in a crash without an understandable error message (Error 400) when attempting to migrate the user database. This might need documentation or a quick preflight check before running migration.

Using directus_users in a Many-to-One fails to migrate

Directus Legacy 8.8.1
Directus Node 9.6.0
Migration Tool - current

I have a table in v8 that references users as a many-to-one relationship from directus_users (a project that has a project leader). The mysql database field is an int as the v8 directus_user id field is an int. When the collection is migrated, the /relations migration breaks as the new relationship is incompatible.

V9 debug log:
00:35:22 🚨 alter table `project` add constraint `project_leader_foreign` foreign key (`leader`) references `directus_users` (`id`) - UNKNOWN_CODE_PLEASE_REPORT: Referencing column 'leader' and referenced column 'id' in foreign key constraint 'project_leader_foreign' are incompatible. Error: UNKNOWN_CODE_PLEASE_REPORT: Referencing column 'leader' and referenced column 'id' in foreign key constraint 'project_leader_foreign' are incompatible.

I think this is because the v9 directus_users' id field is now a char(32).

Any thoughts on how I can resolve this (as I've got a bunch of these relationships).

Migrating large files folder

Working on an upgrade with this tool and I've got about 90GB of files to migrate. If I can just move the original folder to the new one, would it be possible to modify the files.js script to create the new record in the database, but not actually handle copying the file to the new location?

With my first test, it looks like it rewrites the filename to the UUID in the new v9 location, so my initial reaction is that I'll need to have them all copied over.

Any additional insight would be appreciated.

Error during migration of collection - POST -> 500

The following request during migration returned a 500

config: {
    url: '/collections',
    method: 'post',
    data: '{"collection":"campaign","meta":{"note":null,"hidden":false,"singleton":false,"translations":null,"sort_field":null},"schema":{},"fields":[{"field":"id","type":"integer","meta":{"note":"","interface":null,"translation":null,"readonly":false,"hidden":true,"width":"full"},"schema":{"has_auto_increment":true,"default_value":null,"is_primary_key":true,"is_nullable":false,"max_length":"10","numeric_precision":"10","numeric_scale":null}},{"field":"status","type":"string","meta":{"note":"","interface":"dropdown","translation":null,"readonly":false,"hidden":false,"width":"full"},"schema":{"has_auto_increment":false,"default_value":"draft","is_primary_key":false,"is_nullable":false,"max_length":"20","numeric_precision":"20","numeric_scale":null}},{"field":"owner","type":"uuid","meta":{"note":"","interface":"user","translation":null,"readonly":true,"hidden":true,"width":"full","special":["user-created"]},"schema":{"has_auto_increment":false,"default_value":null,"is_primary_key":false,"is_nullable":true,"max_length":"10","numeric_precision":"10","numeric_scale":null}},{"field":"modified_by","type":"uuid","meta":{"note":"","interface":"user","translation":null,"readonly":true,"hidden":true,"width":"full","special":["user-updated"]},"schema":{"has_auto_increment":false,"default_value":null,"is_primary_key":false,"is_nullable":true,"max_length":"10","numeric_precision":"10","numeric_scale":null}},{"field":"modified_on","type":"timestamp","meta":{"note":"","interface":"datetime","translation":null,"readonly":true,"hidden":true,"width":"full","special":["date-updated"]},"schema":{"has_auto_increment":false,"default_value":null,"is_primary_key":false,"is_nullable":true,"max_length":null,"numeric_precision":null,"numeric_scale":null}},{"field":"start_date","type":"dateTime","meta":{"note":"","interface":"datetime","translation":null,"readonly":false,"hidden":false,"width":"full"},"schema":{"has_auto_increment":false,"default_value":null,"is_primary_key":false,"is_nullable":false,"max_length":null,"numeric_precision":null,"numeric_scale":null}},{"field":"end_date","type":"dateTime","meta":{"note":"","interface":"datetime","translation":null,"readonly":false,"hidden":false,"width":"full"},"schema":{"has_auto_increment":false,"default_value":null,"is_primary_key":false,"is_nullable":false,"max_length":null,"numeric_precision":null,"numeric_scale":null}},{"field":"internal_name","type":"string","meta":{"note":"Name for internal usage only","interface":"text-input","translation":null,"readonly":false,"hidden":false,"width":"full"},"schema":{"has_auto_increment":false,"default_value":null,"is_primary_key":false,"is_nullable":true,"max_length":"200","numeric_precision":"200","numeric_scale":null}},{"field":"content_items","type":"alias","meta":{"note":"List of Content Items - linked to campaign - usually edited in content item directly.","interface":"one-to-many","translation":null,"readonly":false,"hidden":false,"width":"full"}},{"field":"translations","type":"integer","meta":{"note":null,"interface":"translations","translation":null,"readonly":false,"hidden":false,"width":"full"},"schema":{"has_auto_increment":false,"default_value":null,"is_primary_key":false,"is_nullable":true,"max_length":null,"numeric_precision":null,"numeric_scale":null}}]}',`

Error in lokal server - which is recieving cms instance was 
`14:17:19 ƒÜ¿ table[field.type] is not a function
TypeError: table[field.type] is not a function
    at FieldsService.addColumnToTable (C:\Projekte\XYZ\local-XYZ-cms\XYZ-cms\node_modules\directus\dist\services\fields.js:314:39)
    at TableBuilder._fn (C:\Projekte\XYZ\local-XYZ-cms\XYZ-cms\node_modules\directus\dist\services\collections.js:90:43)
    at TableBuilder.toSQL (C:\Projekte\XYZ\local-XYZ-cms\XYZ-cms\node_modules\knex\lib\schema\tablebuilder.js:44:14)
    at SchemaCompiler_MySQL.createTable (C:\Projekte\XYZ\local-XYZ-cms\XYZ-cms\node_modules\knex\lib\schema\compiler.js:101:25)
    at SchemaCompiler_MySQL.toSQL (C:\Projekte\XYZ\local-XYZ-cms\XYZ-cms\node_modules\knex\lib\schema\compiler.js:63:26)
    at SchemaBuilder.toSQL (C:\Projekte\XYZ\local-XYZ-cms\XYZ-cms\node_modules\knex\lib\schema\builder.js:35:45)
    at ensureConnectionCallback (C:\Projekte\XYZ\local-XYZ-cms\XYZ-cms\node_modules\knex\lib\execution\internal\ensure-connection-callback.js:4:30)
    at Runner.ensureConnection (C:\Projekte\XYZ\local-XYZ-cms\XYZ-cms\node_modules\knex\lib\execution\runner.js:272:20)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async Runner.run (C:\Projekte\XYZ\local-XYZ-cms\XYZ-cms\node_modules\knex\lib\execution\runner.js:30:19)`

Table Schema of DB Table to be migrated
`-- --------------------------------------------------------
-- Host:                         [fcfc::1:1c60:12ff:fe4b:f502]
-- Server Version:               5.7.30 - MySQL Community Server (GPL)
-- Server Betriebssystem:        Linux
-- HeidiSQL Version:             11.2.0.6213
-- --------------------------------------------------------

/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */;
/*!40101 SET NAMES utf8 */;
/*!50503 SET NAMES utf8mb4 */;
/*!40014 SET @OLD_FOREIGN_KEY_CHECKS=@@FOREIGN_KEY_CHECKS, FOREIGN_KEY_CHECKS=0 */;
/*!40101 SET @OLD_SQL_MODE=@@SQL_MODE, SQL_MODE='NO_AUTO_VALUE_ON_ZERO' */;
/*!40111 SET @OLD_SQL_NOTES=@@SQL_NOTES, SQL_NOTES=0 */;


-- Exportiere Datenbank Struktur für directus
CREATE DATABASE IF NOT EXISTS `directus` /*!40100 DEFAULT CHARACTER SET utf8 */;
USE `directus`;

-- Exportiere Struktur von Tabelle directus.campaign
CREATE TABLE IF NOT EXISTS `campaign` (
  `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `status` varchar(20) COLLATE utf8mb4_unicode_ci NOT NULL DEFAULT 'draft',
  `owner` int(10) unsigned DEFAULT NULL,
  `modified_by` int(10) unsigned DEFAULT NULL,
  `modified_on` datetime DEFAULT NULL,
  `start_date` datetime NOT NULL,
  `end_date` datetime NOT NULL,
  `internal_name` varchar(200) COLLATE utf8mb4_unicode_ci DEFAULT NULL COMMENT 'Name for internal usage only',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=5 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci;

-- Exportiere Daten aus Tabelle directus.campaign: ~4 rows (ungefähr)
/*!40000 ALTER TABLE `campaign` DISABLE KEYS */;
INSERT INTO `campaign` (`id`, `status`, `owner`, `modified_by`, `modified_on`, `start_date`, `end_date`, `internal_name`) VALUES
	(1, 'deleted', 1, 4, '2020-06-29 11:25:11', '2020-06-15 00:00:00', '2020-08-31 00:00:00', NULL),
	(2, 'published', 4, 13, '2021-01-07 10:33:10', '2020-07-15 00:01:00', '2020-12-10 00:59:00', 'Campaign_expired'),
	(3, 'published', 3, 1, '2021-02-23 10:12:45', '2020-12-09 00:00:00', '2021-02-28 20:20:00', 'Xmas'),
	(4, 'published', 8, 8, '2021-02-25 14:54:36', '2021-03-01 00:23:00', '2021-04-29 23:59:00', 'Eastern');
/*!40000 ALTER TABLE `campaign` ENABLE KEYS */;

/*!40101 SET SQL_MODE=IFNULL(@OLD_SQL_MODE, '') */;
/*!40014 SET FOREIGN_KEY_CHECKS=IFNULL(@OLD_FOREIGN_KEY_CHECKS, 1) */;
/*!40101 SET CHARACTER_SET_CLIENT=@OLD_CHARACTER_SET_CLIENT */;
/*!40111 SET SQL_NOTES=IFNULL(@OLD_SQL_NOTES, 1) */;

Please let me know if you need any further information.

Choices lost during migration

I have collections that feature muliple choice state selection, as seen in the V8 screenshots. The migration completed successfully but the choices are lost.

V8
Edit item:

Field configuration:

V9
Edit item:

Field configuration:

Migrating to rc70 throws full pool error

Doing a migration to rc70, we get this error while migrating relations:

Any idea on how to fix this?

Migrating fails on data step

So, now when I try and migrate, the whole thing goes smoothly, migrating relations, files, etc. until it gets to the actual data:

This is weird because the collection in question is dead simple, it literally only has an id and one user-editable text field.

Re-running the script from this point gets me the same result. I could try running from scratch again, but I'm migrating a lot of files so it takes a little while. Any ideas?

I should also say, the v9 instance doesn't show any errors and the v8 error logs don't seem to really reveal anything of interest either.

Many thanks to those who've been working on this, I'm closer than ever before to migrating my instance!

Error: Request failed with status code 400

I am getting this error when I try to migrate.

On the V9 side I get

request completed POST 400 /collections 4ms

Any ideas how to fix this?

Unexpected error when migrating colection

We're trying to migrate v8 to v9, and we get an internal server error with this request, generated by the migration tool:

{
    "collection": "circle_badges_component",
    "meta": {
        "note": null,
        "hidden": false,
        "singleton": false,
        "translations": null,
        "sort_field": null
    },
    "schema": {},
    "fields": [
        {
            "field": "id",
            "type": "integer",
            "meta": {
                "note": "",
                "interface": null,
                "translation": null,
                "readonly": false,
                "hidden": true,
                "width": "full"
            },
            "schema": {
                "has_auto_increment": true,
                "default_value": null,
                "is_primary_key": true,
                "is_nullable": false,
                "max_length": "10",
                "numeric_precision": "10",
                "numeric_scale": null
            }
        },
        {
            "field": "status",
            "type": "string",
            "meta": {
                "note": "",
                "interface": "dropdown",
                "translation": null,
                "readonly": false,
                "hidden": false,
                "width": "full"
            },
            "schema": {
                "has_auto_increment": false,
                "default_value": "draft",
                "is_primary_key": false,
                "is_nullable": false,
                "max_length": "20",
                "numeric_precision": "20",
                "numeric_scale": null
            }
        },
        {
            "field": "owner",
            "type": "uuid",
            "meta": {
                "note": "",
                "interface": "user",
                "translation": null,
                "readonly": true,
                "hidden": true,
                "width": "full",
                "special": [
                    "user-created"
                ]
            },
            "schema": {
                "has_auto_increment": false,
                "default_value": null,
                "is_primary_key": false,
                "is_nullable": true,
                "max_length": "10",
                "numeric_precision": "10",
                "numeric_scale": null
            }
        },
        {
            "field": "created_on",
            "type": "timestamp",
            "meta": {
                "note": "",
                "interface": "datetime",
                "translation": null,
                "readonly": true,
                "hidden": true,
                "width": "full",
                "special": [
                    "date-created"
                ]
            },
            "schema": {
                "has_auto_increment": false,
                "default_value": null,
                "is_primary_key": false,
                "is_nullable": true,
                "max_length": null,
                "numeric_precision": null,
                "numeric_scale": null
            }
        },
        {
            "field": "name",
            "type": "string",
            "meta": {
                "note": "",
                "interface": "text-input",
                "translation": null,
                "readonly": false,
                "hidden": false,
                "width": "half"
            },
            "schema": {
                "has_auto_increment": false,
                "default_value": null,
                "is_primary_key": false,
                "is_nullable": true,
                "max_length": "200",
                "numeric_precision": "200",
                "numeric_scale": null
            }
        },
        {
            "field": "order",
            "type": "integer",
            "meta": {
                "note": "",
                "interface": "numeric",
                "translation": null,
                "readonly": false,
                "hidden": false,
                "width": "full"
            },
            "schema": {
                "has_auto_increment": false,
                "default_value": null,
                "is_primary_key": false,
                "is_nullable": true,
                "max_length": "10",
                "numeric_precision": "10",
                "numeric_scale": null
            }
        },
        {
            "field": "margin_top_desktop",
            "type": "integer",
            "meta": {
                "note": "Number of pixels for the top margin of the module (desktop)",
                "interface": "numeric",
                "translation": null,
                "readonly": false,
                "hidden": false,
                "width": "full"
            },
            "schema": {
                "has_auto_increment": false,
                "default_value": null,
                "is_primary_key": false,
                "is_nullable": true,
                "max_length": "10",
                "numeric_precision": "10",
                "numeric_scale": null
            }
        },
        {
            "field": "margin_bottom_desktop",
            "type": "integer",
            "meta": {
                "note": "Number of pixels for the bottom margin of the module (desktop)",
                "interface": "numeric",
                "translation": null,
                "readonly": false,
                "hidden": false,
                "width": "full"
            },
            "schema": {
                "has_auto_increment": false,
                "default_value": null,
                "is_primary_key": false,
                "is_nullable": true,
                "max_length": "10",
                "numeric_precision": "10",
                "numeric_scale": null
            }
        },
        {
            "field": "margin_top_mobile",
            "type": "integer",
            "meta": {
                "note": "Number of pixels for the top margin of the module (mobile)",
                "interface": "numeric",
                "translation": null,
                "readonly": false,
                "hidden": false,
                "width": "full"
            },
            "schema": {
                "has_auto_increment": false,
                "default_value": null,
                "is_primary_key": false,
                "is_nullable": true,
                "max_length": "10",
                "numeric_precision": "10",
                "numeric_scale": null
            }
        },
        {
            "field": "margin_bottom_mobile",
            "type": "integer",
            "meta": {
                "note": "Number of pixels for the bottom margin of the module (mobile)",
                "interface": "numeric",
                "translation": null,
                "readonly": false,
                "hidden": false,
                "width": "full"
            },
            "schema": {
                "has_auto_increment": false,
                "default_value": null,
                "is_primary_key": false,
                "is_nullable": true,
                "max_length": "10",
                "numeric_precision": "10",
                "numeric_scale": null
            }
        },
        {
            "field": "device",
            "type": "string",
            "meta": {
                "note": "The device where the module will appear",
                "interface": "text-input",
                "translation": null,
                "readonly": false,
                "hidden": false,
                "width": "full"
            },
            "schema": {
                "has_auto_increment": false,
                "default_value": "both",
                "is_primary_key": false,
                "is_nullable": true,
                "max_length": "100",
                "numeric_precision": "100",
                "numeric_scale": null
            }
        },
        {
            "field": "date_programations",
            "type": "text",
            "meta": {
                "note": "When the banner will appear and disappear by site (UTC zone time)",
                "interface": "repeater",
                "translation": null,
                "readonly": false,
                "hidden": false,
                "width": "full",
                "special": [
                    "json"
                ]
            },
            "schema": {
                "has_auto_increment": false,
                "default_value": null,
                "is_primary_key": false,
                "is_nullable": true,
                "max_length": "65535",
                "numeric_precision": "65535",
                "numeric_scale": null
            }
        },
        {
            "field": "typography",
            "type": "string",
            "meta": {
                "note": "Choose between standard component typography (openSans, Agaramond...) or Montserrat typography",
                "interface": "radio-buttons",
                "translation": null,
                "readonly": false,
                "hidden": false,
                "width": "full"
            },
            "schema": {
                "has_auto_increment": false,
                "default_value": "standard",
                "is_primary_key": false,
                "is_nullable": true,
                "max_length": "100",
                "numeric_precision": "100",
                "numeric_scale": null
            }
        },
        {
            "field": "platform",
            "type": "string",
            "meta": {
                "note": "",
                "interface": "text-input",
                "translation": null,
                "readonly": false,
                "hidden": false,
                "width": "full"
            },
            "schema": {
                "has_auto_increment": false,
                "default_value": "both",
                "is_primary_key": false,
                "is_nullable": true,
                "max_length": "10",
                "numeric_precision": "10",
                "numeric_scale": null
            }
        },
        {
            "field": "circle_badges",
            "type": "alias",
            "meta": {
                "note": "include the circle badges. max.9 categories to be contributed",
                "interface": "many-to-many",
                "translation": null,
                "readonly": false,
                "hidden": false,
                "width": "full"
            }
        }
    ]
}

and this is the error that we get

Publish to NPM

Would be useful to be able to run this straight through npx: npx migrate-directus <from> <to>

Datetimepicker fields in v8 cannot be imported into v9

Problem
If you have a V8 instance with a MYSQL-DB, it seems that the datetime picker interface in V8 gets saved as a non ISO timestamp, that isn't compatible with the v9 import endpoint, that expects iso formatted timestamps.

This doesn't seem to have timezone information either. Which really makes this annoying :D!

Looks like the datetime field implementation has no concept of i18n.
https://github.com/directus/v8-archive/blob/681196fead1f87e2f73d2cdf64af34543d3140cf/app/src/interfaces/datetime/input.vue

vs
https://github.com/directus/v8-archive/blob/681196fead1f87e2f73d2cdf64af34543d3140cf/app/src/interfaces/datetime-created/input.vue#L21

Related PR that killed the "it works still": directus/directus#7200

Add license

We should be able to use MIT for this

Icons and Translations not migrating correctly

The icons are not being migrated.

The translations for both the collections and fields are not following the correct structure.

v8 translation

{
    "locale": "string",
    "translation": "string"
}

v9 translation

{
    "language": "string",
    "translation": "string"
}

Item Ownership

Hello All,

Firstly, thank you so much for the wonderful directus, can't stop using it.

I have successfully migrated my data to V9, however i am facing a little problem when it comes to ownership of items.
After migration, all items are owned by the admin, however i would like to retain the original owner data.

i tried adding the following to the itesmRecords which is being processed before posting to V9

if(item.owner && item.owner !== 1)
{
	console.log("item being processed has an owner", item.owner, "matching id is", context.userMap[item.owner+""])
	item.owner = context.userMap[item.owner+""]
	console.log("new owner is", item.owner);
}

and i get this in the console

item being processed has an owner 326 matching id is 63e54f56-8d4e-4749-b848-8e1575e6ebd1
VM93 data.js:181 new owner is 63e54f56-8d4e-4749-b848-8e1575e6ebd1
VM93 data.js:179 item being processed has an owner 312 matching id is 60742b25-1ae5-470b-bd05-31d51e7bb70f
VM93 data.js:181 new owner is 60742b25-1ae5-470b-bd05-31d51e7bb70f

which means I do assign the matching owner succssfully.

However, when i check in the db after i am done, it still shows that the admin owns the data,

How can i get past this little issue?

Thank you so much

"Not a function" error when Creating Collections

Hello,

When running the migration tool, everything seems to connect correctly, but then it fails during the Migrating Schema → Creating Collections phase. Collections produce a (details.length || "").split is not a function error. I tried excluding the menu_cats collection (this would be the easiest to reproduce manually), but then the menu_items collection threw the same error.

For full context, the v8 database is MySQL and the v9 is SQLite. I suspect that shouldn't make a difference, but figured that I would mention it anyway. Any guidance would be appreciated.

Thanks!

Loading context
✨ Loaded context succesfully
  ✔ Loading context
  ❯ Migrating Schema
    ✔ Downloading Schema
    ✔ Saving schema context
    ❯ Creating Collections
      ✖ menu_cats
        → (details.length || "").split is not a function
        menu_items
        menu_item_prices
      Saving collections context
    Migration Files
    Migrating Users
    Migrating Relations
    Migrating Data
    Save final context
TypeError: (details.length || "").split is not a function
    at file:///Users/mattbisme/Sites/_global/directus_migration/migration-tool/tasks/schema.js:235:44
    at Array.map (<anonymous>)
    at Task.task (file:///Users/mattbisme/Sites/_global/directus_migration/migration-tool/tasks/schema.js:197:48)
    at /Users/mattbisme/Sites/_global/directus_migration/migration-tool/node_modules/listr/lib/task.js:167:30 {
  context: [Object: null prototype] {
    completedSteps: {
      schema: true,
      collections: false,
      files: false,
      roles: false,
      users: false,
      relationsv8: false,
      relations: false,
      data: false,
      completed: false
    },
    collectionsV9: [],
    skipCollections: [],
    collections: [ [Object], [Object], [Object] ]
  }
}

Creating a Collections: Cannot convert undefined or null to object

Something went wrong when Creating a Collection on the migrating scheme step.

Cannot convert undefined or null to object
TypeError: Cannot convert undefined or null to object
    at Function.values (<anonymous>)
    at Task.task (file:///D:/_git/2023-yondr/directus-migration-tool/tasks/schema.js:175:30)
    at D:\_git\2023-yondr\directus-migration-tool\node_modules\listr\lib\task.js:167:30

"Migrating Relations" fails creating link table with directus_files

The migration tool seems to be failing when trying to create a many-to-many link table with directus_files

➜ node index.js
✨ Migrating https://cms.energyadvisors.com (v8) to https://directus.energyadvisors.com (v9)...
  ❯ Migrating Schema
    ✔ Downloading Schema
    ✔ Creating Collections
    ✖ Migrating Relations
      → Request failed with status code 500
    Migration Files
    Migrating Users
    Migrating Data
Error: Request failed with status code 500
    at createError (/Users/Derek/src/migration-tool/node_modules/axios/lib/core/createError.js:16:15)
    at settle (/Users/Derek/src/migration-tool/node_modules/axios/lib/core/settle.js:17:12)
    at IncomingMessage.handleStreamEnd (/Users/Derek/src/migration-tool/node_modules/axios/lib/adapters/http.js:260:11)
    at IncomingMessage.emit (events.js:388:22)
    at endReadableNT (internal/streams/readable.js:1336:12)
    at processTicksAndRejections (internal/process/task_queues.js:82:21) {
  config: {
    url: '/relations',
    method: 'post',
    data: '{"meta":{"many_collection":"home_page_case_history_directus_files","many_field":"directus_files_id","many_primary":"id","one_collection":"directus_files","one_field":"home_page","one_primary":"id","junction_field":"home_page_id"},"field":"directus_files_id","collection":"home_page_case_history_directus_files","related_collection":"directus_files","schema":null}',
    headers: {
      Accept: 'application/json, text/plain, */*',
      'Content-Type': 'application/json;charset=utf-8',
      Authorization: 'Bearer 81sG3K0mSKBKiMOrZJKF',
      'User-Agent': 'axios/0.21.1',
      'Content-Length': 363
    },

I turned on DEBUG logs in my v9 directus and this is the error it's showing.

alter table `home_page_case_history_directus_files` add constraint `home_page_case_history_directus_files_directus_files_id_foreign` foreign key (`directus_files_id`) references `directus_files` (`id`) - ER_CANT_CREATE_TABLE: Can't create table `directus`.`home_page_case_history_directus_files` (errno: 150 "Foreign key constraint is incorrectly formed")
--
Wed, Jul 7 2021 3:27:32 pm | Error: ER_CANT_CREATE_TABLE: Can't create table `directus`.`home_page_case_history_directus_files` (errno: 150 "Foreign key constraint is incorrectly formed")
Wed, Jul 7 2021 3:27:32 pm | at Query.Sequence._packetToError (/directus/node_modules/mysql/lib/protocol/sequences/Sequence.js:47:14)
Wed, Jul 7 2021 3:27:32 pm | at Query.ErrorPacket (/directus/node_modules/mysql/lib/protocol/sequences/Query.js:79:18)
Wed, Jul 7 2021 3:27:32 pm | at Protocol._parsePacket (/directus/node_modules/mysql/lib/protocol/Protocol.js:291:23)
Wed, Jul 7 2021 3:27:32 pm | at Parser._parsePacket (/directus/node_modules/mysql/lib/protocol/Parser.js:433:10)
Wed, Jul 7 2021 3:27:32 pm | at Parser.write (/directus/node_modules/mysql/lib/protocol/Parser.js:43:10)
Wed, Jul 7 2021 3:27:32 pm | at Protocol.write (/directus/node_modules/mysql/lib/protocol/Protocol.js:38:16)
Wed, Jul 7 2021 3:27:32 pm | at Socket.<anonymous> (/directus/node_modules/mysql/lib/Connection.js:88:28)
Wed, Jul 7 2021 3:27:32 pm | at Socket.<anonymous> (/directus/node_modules/mysql/lib/Connection.js:526:10)
Wed, Jul 7 2021 3:27:32 pm | at Socket.emit (node:events:394:28)
Wed, Jul 7 2021 3:27:32 pm | at addChunk (node:internal/streams/readable:312:12)

There's quite a few images in this link table. Any suggestions on how I can work with this?

Directus 9 version: 9.0.0-RC-83 using latest MariaDB
Directus 8 version: 8.8.1 and using Mysql 8

Cannot migrate v8 tables to v9 with 'CURRENT_TIMESTAMP'

Describe the Bug

All the existing tables with datetime/timestamp fields which contains default values as CURRENT_TIMESTAMP are failed to migrate and getting following error.

ER_INVALID_DEFAULT: Invalid default value for 'createdAt'

To Reproduce

Host directus v9.
Migrate v8 to v9 via migration tool.

What version of Directus are you using?

9.0.0-rc.87

What version of Node.js are you using?

16.6.0

What database are you using?

mysql 8.0.22

What browser are you using?

Chrome

What operating system are you using?

Windows

How are you deploying Directus?

locally

Getting "bad request" on execution

Getting following on execution

MacBook-Pro directus-migration-tool % node index.js
✨ Migrating http://localhost:8090/project (v8) to http://localhost:8055 (v9)...
  ❯ Migrating Schema
    ✔ Downloading Schema
    ❯ Creating Collections
      ✖ test_collection
        → Request failed with status code 400
      Migrating Relations
    Migration Files
    Migrating Users
    Migrating Data
Error: Request failed with status code 400
    at createError (/Users/jatinsinghal/Documents/projects/canopy/directus-migration-tool/node_modules/axios/lib/core/createError.js:16:15)
    at settle (/Users/jatinsinghal/Documents/projects/canopy/directus-migration-tool/node_modules/axios/lib/core/settle.js:17:12)
    at IncomingMessage.handleStreamEnd (/Users/jatinsinghal/Documents/projects/canopy/directus-migration-tool/node_modules/axios/lib/adapters/http.js:260:11)
    at IncomingMessage.emit (events.js:327:22)
    at endReadableNT (internal/streams/readable.js:1327:12)
    at processTicksAndRejections (internal/process/task_queues.js:80:21) {
  config: {
    url: '/collections',
    method: 'post',
    data: '{"collection":"test_collection","meta":{"note":null,"hidden":false,"singleton":false,"translations":null,"sort_field":null},"schema":{},"fields":[{"field":"id","type":"integer","meta":{"note":"","interface":null,"translation":null,"readonly":false,"hidden":true,"width":"full"},"schema":{"has_auto_increment":true,"default_value":null,"is_primary_key":true,"is_nullable":false,"max_length":"10","numeric_precision":"10","numeric_scale":null}},{"field":"status","type":"string","meta":{"note":"","interface":"dropdown","translation":null,"readonly":false,"hidden":false,"width":"full"},"schema":{"has_auto_increment":false,"default_value":"draft","is_primary_key":false,"is_nullable":false,"max_length":"20","numeric_precision":"20","numeric_scale":null}},{"field":"sort","type":"integer","meta":{"note":"","interface":"numeric","translation":null,"readonly":false,"hidden":true,"width":"full"},"schema":{"has_auto_increment":false,"default_value":null,"is_primary_key":false,"is_nullable":true,"max_length":"10","numeric_precision":"10","numeric_scale":null}},{"field":"owner","type":"uuid","meta":{"note":"","interface":"user","translation":null,"readonly":true,"hidden":true,"width":"full","special":["user-created"]},"schema":{"has_auto_increment":false,"default_value":null,"is_primary_key":false,"is_nullable":true,"max_length":"10","numeric_precision":"10","numeric_scale":null}},{"field":"created_on","type":"timestamp","meta":{"note":"","interface":"datetime","translation":null,"readonly":true,"hidden":true,"width":"full","special":["date-created"]},"schema":{"has_auto_increment":false,"default_value":null,"is_primary_key":false,"is_nullable":true,"max_length":null,"numeric_precision":null,"numeric_scale":null}},{"field":"text_1","type":"string","meta":{"note":"","interface":"text-input","translation":null,"readonly":false,"hidden":false,"width":"full"},"schema":{"has_auto_increment":false,"default_value":null,"is_primary_key":false,"is_nullable":true,"max_length":"200","numeric_precision":"200","numeric_scale":null}},{"field":"json_1","type":"text","meta":{"note":"","interface":"json","translation":null,"readonly":false,"hidden":false,"width":"full","special":["json"]},"schema":{"has_auto_increment":false,"default_value":null,"is_primary_key":false,"is_nullable":true,"max_length":null,"numeric_precision":null,"numeric_scale":null}}]}',
    headers: {
      Accept: 'application/json, text/plain, */*',
      'Content-Type': 'application/json;charset=utf-8',
      Authorization: 'Bearer qsQJHjDW8SXwBZqkHTtLp4htq2xj3SfN',
      'User-Agent': 'axios/0.21.1',
      'Content-Length': 2360
    },
    baseURL: 'http://localhost:8055',
    transformRequest: [ [Function: transformRequest] ],
    transformResponse: [ [Function: transformResponse] ],
    timeout: 0,
    adapter: [Function: httpAdapter],
    xsrfCookieName: 'XSRF-TOKEN',
    xsrfHeaderName: 'X-XSRF-TOKEN',
    maxContentLength: -1,
    maxBodyLength: -1,
    validateStatus: [Function: validateStatus]
  },
  request: <ref *1> ClientRequest {
    _events: [Object: null prototype] {
      socket: [Function (anonymous)],
      abort: [Function (anonymous)],
      aborted: [Function (anonymous)],
      connect: [Function (anonymous)],
      error: [Function (anonymous)],
      timeout: [Function (anonymous)],
      prefinish: [Function: requestOnPrefinish]
    },
    _eventsCount: 7,
    _maxListeners: undefined,
    outputData: [],
    outputSize: 0,
    writable: true,
    destroyed: false,
    _last: true,
    chunkedEncoding: false,
    shouldKeepAlive: false,
    _defaultKeepAlive: true,
    useChunkedEncodingByDefault: true,
    sendDate: false,
    _removedConnection: false,
    _removedContLen: false,
    _removedTE: false,
    _contentLength: null,
    _hasBody: true,
    _trailer: '',
    finished: true,
    _headerSent: true,
    socket: Socket {
      connecting: false,
      _hadError: false,
      _parent: null,
      _host: 'localhost',
      _readableState: [ReadableState],
      _events: [Object: null prototype],
      _eventsCount: 7,
      _maxListeners: undefined,
      _writableState: [WritableState],
      allowHalfOpen: false,
      _sockname: null,
      _pendingData: null,
      _pendingEncoding: '',
      server: null,
      _server: null,
      parser: null,
      _httpMessage: [Circular *1],
      [Symbol(async_id_symbol)]: 89,
      [Symbol(kHandle)]: [TCP],
      [Symbol(kSetNoDelay)]: false,
      [Symbol(lastWriteQueueSize)]: 0,
      [Symbol(timeout)]: null,
      [Symbol(kBuffer)]: null,
      [Symbol(kBufferCb)]: null,
      [Symbol(kBufferGen)]: null,
      [Symbol(kCapture)]: false,
      [Symbol(kBytesRead)]: 0,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(RequestTimeout)]: undefined
    },
    _header: 'POST /collections HTTP/1.1\r\n' +
      'Accept: application/json, text/plain, */*\r\n' +
      'Content-Type: application/json;charset=utf-8\r\n' +
      'Authorization: Bearer qsQJHjDW8SXwBZqkHTtLp4htq2xj3SfN\r\n' +
      'User-Agent: axios/0.21.1\r\n' +
      'Content-Length: 2360\r\n' +
      'Host: localhost:8055\r\n' +
      'Connection: close\r\n' +
      '\r\n',
    _keepAliveTimeout: 0,
    _onPendingData: [Function: noopPendingOutput],
    agent: Agent {
      _events: [Object: null prototype],
      _eventsCount: 2,
      _maxListeners: undefined,
      defaultPort: 80,
      protocol: 'http:',
      options: [Object],
      requests: {},
      sockets: [Object],
      freeSockets: {},
      keepAliveMsecs: 1000,
      keepAlive: false,
      maxSockets: Infinity,
      maxFreeSockets: 256,
      scheduling: 'fifo',
      maxTotalSockets: Infinity,
      totalSocketCount: 1,
      [Symbol(kCapture)]: false
    },
    socketPath: undefined,
    method: 'POST',
    maxHeaderSize: undefined,
    insecureHTTPParser: undefined,
    path: '/collections',
    _ended: true,
    res: IncomingMessage {
      _readableState: [ReadableState],
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      socket: [Socket],
      httpVersionMajor: 1,
      httpVersionMinor: 1,
      httpVersion: '1.1',
      complete: true,
      headers: [Object],
      rawHeaders: [Array],
      trailers: {},
      rawTrailers: [],
      aborted: false,
      upgrade: false,
      url: '',
      method: null,
      statusCode: 400,
      statusMessage: 'Bad Request',
      client: [Socket],
      _consuming: false,
      _dumped: false,
      req: [Circular *1],
      responseUrl: 'http://localhost:8055/collections',
      redirects: [],
      [Symbol(kCapture)]: false,
      [Symbol(RequestTimeout)]: undefined
    },
    aborted: false,
    timeoutCb: null,
    upgradeOrConnect: false,
    parser: null,
    maxHeadersCount: null,
    reusedSocket: false,
    host: 'localhost',
    protocol: 'http:',
    _redirectable: Writable {
      _writableState: [WritableState],
      _events: [Object: null prototype],
      _eventsCount: 2,
      _maxListeners: undefined,
      _options: [Object],
      _ended: true,
      _ending: true,
      _redirectCount: 0,
      _redirects: [],
      _requestBodyLength: 2360,
      _requestBodyBuffers: [],
      _onNativeResponse: [Function (anonymous)],
      _currentRequest: [Circular *1],
      _currentUrl: 'http://localhost:8055/collections',
      [Symbol(kCapture)]: false
    },
    [Symbol(kCapture)]: false,
    [Symbol(kNeedDrain)]: false,
    [Symbol(corked)]: 0,
    [Symbol(kOutHeaders)]: [Object: null prototype] {
      accept: [Array],
      'content-type': [Array],
      authorization: [Array],
      'user-agent': [Array],
      'content-length': [Array],
      host: [Array]
    }
  },
  response: {
    status: 400,
    statusText: 'Bad Request',
    headers: {
      'x-powered-by': 'Directus',
      vary: 'Origin',
      'content-type': 'application/json; charset=utf-8',
      'content-length': '113',
      etag: 'W/"71-YartklZ00JpQjys820QI6m1/qgU"',
      date: 'Mon, 05 Apr 2021 10:47:13 GMT',
      connection: 'close'
    },
    config: {
      url: '/collections',
      method: 'post',
      data: '{"collection":"test_collection","meta":{"note":null,"hidden":false,"singleton":false,"translations":null,"sort_field":null},"schema":{},"fields":[{"field":"id","type":"integer","meta":{"note":"","interface":null,"translation":null,"readonly":false,"hidden":true,"width":"full"},"schema":{"has_auto_increment":true,"default_value":null,"is_primary_key":true,"is_nullable":false,"max_length":"10","numeric_precision":"10","numeric_scale":null}},{"field":"status","type":"string","meta":{"note":"","interface":"dropdown","translation":null,"readonly":false,"hidden":false,"width":"full"},"schema":{"has_auto_increment":false,"default_value":"draft","is_primary_key":false,"is_nullable":false,"max_length":"20","numeric_precision":"20","numeric_scale":null}},{"field":"sort","type":"integer","meta":{"note":"","interface":"numeric","translation":null,"readonly":false,"hidden":true,"width":"full"},"schema":{"has_auto_increment":false,"default_value":null,"is_primary_key":false,"is_nullable":true,"max_length":"10","numeric_precision":"10","numeric_scale":null}},{"field":"owner","type":"uuid","meta":{"note":"","interface":"user","translation":null,"readonly":true,"hidden":true,"width":"full","special":["user-created"]},"schema":{"has_auto_increment":false,"default_value":null,"is_primary_key":false,"is_nullable":true,"max_length":"10","numeric_precision":"10","numeric_scale":null}},{"field":"created_on","type":"timestamp","meta":{"note":"","interface":"datetime","translation":null,"readonly":true,"hidden":true,"width":"full","special":["date-created"]},"schema":{"has_auto_increment":false,"default_value":null,"is_primary_key":false,"is_nullable":true,"max_length":null,"numeric_precision":null,"numeric_scale":null}},{"field":"text_1","type":"string","meta":{"note":"","interface":"text-input","translation":null,"readonly":false,"hidden":false,"width":"full"},"schema":{"has_auto_increment":false,"default_value":null,"is_primary_key":false,"is_nullable":true,"max_length":"200","numeric_precision":"200","numeric_scale":null}},{"field":"json_1","type":"text","meta":{"note":"","interface":"json","translation":null,"readonly":false,"hidden":false,"width":"full","special":["json"]},"schema":{"has_auto_increment":false,"default_value":null,"is_primary_key":false,"is_nullable":true,"max_length":null,"numeric_precision":null,"numeric_scale":null}}]}',
      headers: [Object],
      baseURL: 'http://localhost:8055',
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      adapter: [Function: httpAdapter],
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: -1,
      maxBodyLength: -1,
      validateStatus: [Function: validateStatus]
    },
    request: <ref *1> ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 7,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: false,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: true,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      _contentLength: null,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      socket: [Socket],
      _header: 'POST /collections HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Content-Type: application/json;charset=utf-8\r\n' +
        'Authorization: Bearer qsQJHjDW8SXwBZqkHTtLp4htq2xj3SfN\r\n' +
        'User-Agent: axios/0.21.1\r\n' +
        'Content-Length: 2360\r\n' +
        'Host: localhost:8055\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: noopPendingOutput],
      agent: [Agent],
      socketPath: undefined,
      method: 'POST',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      path: '/collections',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: 'localhost',
      protocol: 'http:',
      _redirectable: [Writable],
      [Symbol(kCapture)]: false,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype]
    },
    data: { errors: [Array] }
  },
  isAxiosError: true,
  toJSON: [Function: toJSON],
  context: [Object: null prototype] { collections: [ [Object] ] }
}

"Bad Request" Error on migrating relations

The creation of collections seems to be working fine but when it comes to migrating relations I get this message

migration tool side:

v9 directus project side:

I have noticed that the data sent wasn't in the same format as the users_relations' table of my v9 directus :
sent data :

directus_relations of my v9 directus database:

As you can see there is no 'id' key, 'sort_field' key nor 'one_deselect_action' key in the sent data (the same goes for all the other elements in the array, they're all in the same format)
Is it related to my issue ? Anyway, how can I fix this ?
Thanks in advance.

Migration fails on composite primary key

Migration fails (error 500) if collection has composite primary key.
Error from directus:
Error: ER_MULTIPLE_PRI_KEY: Multiple primary key defined

Data that is posted

{
    "collection":"articles_checked",
    "meta":{"note":null,"hidden":false,"singleton":false,"icon":null,"sort_field":null},
    "schema":{},
    "fields":[{
        "field":"article",
        "type":"integer",
        "meta":{"note":"","readonly":false,"hidden":false,"width":null,"sort":null},
        "schema":{"has_auto_increment":false,"default_value":null,"is_primary_key":true,"is_nullable":false,"max_length":"10","numeric_precision":"10","numeric_scale":null}
    },{
        "field":"user",
        "type":"string",
        "meta":{"note":"","readonly":false,"hidden":false,"width":null,"sort":null},
        "schema":{"has_auto_increment":false,"default_value":null,"is_primary_key":true,"is_nullable":false,"max_length":"36","numeric_precision":"36","numeric_scale":null}
    },{
        "field":"created_on",
        "type":"dateTime",
        "meta":{"note":"","readonly":false,"hidden":false,"width":null,"sort":null},
        "schema":{"has_auto_increment":false,"default_value":"NOW()","is_primary_key":false,"is_nullable":false,"max_length":null,"numeric_precision":null,"numeric_scale":null}
    }]
}