Type "regclass" about pg-mem

[oguimbal]

@MirrorBytes FYI I've created a unit test that matches your use case.

And we're almost there... oid is now supported (1.4.1), but on delete cascade is not.
I'll implement that :)

🔨 Not supported 🔨 : Foreign keys with actions not yet supported

[oguimbal]

Ah ! The important bit is UnhandledPromiseRejectionWarning ... this is fixed in [email protected] ... its just a warning, but agreed, this was not supposed to be here (nor related to any test).

FYI, this is the fix and here is how it is tested

[oguimbal]

Duly noted.
I'll implement that asap.

Thanks :) That's nice to hear.

[BrownKnight]

Another +1 from me, very much looking forward to seeing where this goes - looks to be an incredibly useful tool.

[MirrorBytes]

Based on yarn.lock:

TypeORM: v0.2.29
pg-mem: v1.4.2

Thank you for the tip, I've removed that declaration.

I'm going to try wrapping the tests like you suggest, and running them that way. I'm running quite a number of unit tests with graphql, and the user tests worked perfectly.
The form tests are weird though.. so it pulls the correct forms created by a specified user, but still throws that error.

Let me isolate my tests, then I'll test against the real instance (as a control), then I'll test it against pg-mem.

[MirrorBytes]

IT WORKS! 👍

I can happily say I've converted all my tests to pg-mem without any errors whatsoever!

And their run times are comparable to the real PG instance without the spool up times!

[oguimbal]

Next step to improve testing time: Share the db instance between tests to avoid running .synchronize() multiple times (which probably eats up most of your CPU when running a test).

You can do this with restore points, as documented on the readme.

[MirrorBytes]

I took my tests out of running in band and the time was cut in half. Time for restore points!

Thank you for working on this! Makes testing run quite a bit faster

[oguimbal]

Yup, that's very much in the the scope.

It's not a trivial feature though (but will be necesary to improve how pg-mem deals with reflection), so I cant say exactly when it will be released.

(if someone else also needs this, please comment in order to help priorizing it)

[MirrorBytes]

Understood completely. Regclass is a pain to deal with in regular postgres.

+1 This would be very useful for the project I'm working on.

BTW, great work on all of this. Being able to use postgres instead of sqlite in my unit tests will be amazing.

[oguimbal]

Could you provide examples of how you use regclasses ?

(i'm looking for inspiration to write some tests, and I'd like to cover some actual use cases)

[BrownKnight]

I'll be honest I'm not hugely experienced with how TypeORM uses regclasses, aside from the fact that it not being supported meant that my TypeORM relations didn't work. The sample you already have at /samples/typeorm/joins.ts is pretty much a solid use case. I'm not sure if using the DataMapper approach vs ActiveRecord affects how TypeORM deals with relations, but would be worth testing with both I think.

I have just tried to create a minimal repo with some use cases where I've encountered the need to have regclasses previously, however trying to run that sample file resulted in an error when synchronising (I've most likely done something wrong admittedly):

Error: No alias provided
        at Query.buildSelect (/Users/aman/pg-mem-regclass-usage/node_modules/pg-mem/src/query.ts:354:23)
        at Query.queries (/Users/aman/pg-mem-regclass-usage/node_modules/pg-mem/src/query.ts:98:47)
        at queries.next (<anonymous>)
        at Query.query (/Users/aman/pg-mem-regclass-usage/node_modules/pg-mem/src/query.ts:42:20)

When I've got some more time later this week I'll delve a little deeper and come up with a few use cases.

[MirrorBytes]

Agree with @BrownKnight, the only additional tests around it you would need for basic functionality is using something like:

const photo = await Photo.find({ id }, { relations: [ 'user' ] });

This will trigger a regclass due to named relations. Now this will try to convert the named relations to OID relations.
After that, I would run some basic functions, like test adds and deletes.

[oguimbal]

Hi !

I'm not sure to see which request is failing (the join.ts sample works for me... it actually is a unit test that prevents from deploying if failing).

However, I implemented regclass (at least part of it) in [email protected] , along with several other improvements, could you try it ?

[MirrorBytes]

Tested it out, now I get this error:

QueryFailedError: 🔨 Not supported 🔨 : Type "oid"

This tells me that you're getting closer.
Here's some good info that may help implementing oids:

"The oid type itself has few operations beyond comparison. It can be cast to integer, however, and then manipulated using the standard integer operators. (Beware of possible signed-versus-unsigned confusion if you do this.)

The OID alias types have no operations of their own except for specialized input and output routines. These routines are able to accept and display symbolic names for system objects, rather than the raw numeric value that type oid would use. The alias types allow simplified lookup of OID values for objects. For example, to examine the pg_attribute rows related to a table mytable, one could write:"

SELECT * FROM pg_attribute WHERE attrelid = 'mytable'::regclass;

"rather than:"

SELECT * FROM pg_attribute
  WHERE attrelid = (SELECT oid FROM pg_class WHERE relname = 'mytable');

Info taken from here: oids

[MirrorBytes]

There's absolutely nothing special with my database, I'm just using sync for the time being. Here are the simplified entities I'm using with relations, maybe you can use them for some tests:

base.ts

import {
  BaseEntity,
  PrimaryGeneratedColumn,
  CreateDateColumn,
  BeforeInsert,
  BeforeUpdate,
} from 'typeorm';
import { validateOrReject } from 'class-validator';
import { ObjectType, Field, ID } from 'type-graphql';

@ObjectType()
export abstract class External extends BaseEntity {
  @Field(() => ID)
  @PrimaryGeneratedColumn('uuid')
  readonly id!: string;

  @Field()
  @CreateDateColumn({ type: 'timestamp' })
  createdAt!: Date;

  @Field()
  @CreateDateColumn({ type: 'timestamp' })
  updatedAt!: Date;

  @BeforeInsert()
  @BeforeUpdate()
  async validate(): Promise<void> {
    await validateOrReject(this);
  }
}

user.ts

import { Entity, Column, OneToMany } from 'typeorm';
import { ObjectType, Field } from 'type-graphql';
import { IsEmail, Length } from 'class-validator';

import { External } from './base';
import { Form } from './form';

@Entity()
@ObjectType()
export class User extends External {
  @Field()
  @Column('citext', { unique: true })
  @IsEmail()
  email!: string;

  @Field()
  @Column()
  @Length(1, 64)
  name!: string;

  @Column({ type: 'bytea' })
  password!: Buffer;

  @OneToMany(() => Form, (form) => form.user)
  forms!: Form[];

  ...
}

form.ts

import { Entity, Column, ManyToOne, OneToMany } from 'typeorm';
import { ObjectType, Field } from 'type-graphql';
import { MaxLength } from 'class-validator';

@Entity()
@ObjectType()
export class Form extends External {
  @Field()
  @MaxLength(60)
  @Column()
  name!: string;

  @Field(() => User, { nullable: true })
  @ManyToOne(() => User, (user) => user.forms)
  user!: User;

  ...

  @OneToMany(() => Submission, (submission) => submission.form, {
    cascade: true,
  })
  submissions!: Submission[];

  ...
}

submission.ts

import { Entity, Column, ManyToOne } from 'typeorm';
import { ObjectType } from 'type-graphql';

@Entity()
@ObjectType()
export class Submission extends External {
  ...

  @ManyToOne(() => Form, (form) => form.submissions, { onDelete: 'CASCADE' })
  form!: Form;
}

[BrownKnight]

It appears the issues @MirrorBytes is seeing may be due something TypeORM has changed between v0.2.25 and the latest version. I've run the joins.ts file on every version since v0.2.25, and it only gives the old type not supported error in v0.2.29 (latest)

[oguimbal]

Good catch ! I'm using 0.2.25 ... that's why 👍 thanks, i'll have a look.

[MirrorBytes]

I'm using 0.2.29 as well, but I'm getting the oid not supported error

[MirrorBytes]

@oguimbal Awesome! Really looking forward to it!

[oguimbal]

[email protected] is out. Meaning that all of this should be resolved. Please reopen this (or another ad-hoc issue) if necessary !

[MirrorBytes]

Awesome! Most of my tests are passing! However, I added a line to your test, and I'm getting a strange error now:

failed to cast text to uuid in string

    🐜 This seems to be an execution error, which means that your request syntax seems okay,
    but the resulting statement cannot be executed → Probably not a pg-mem error.

    *️⃣ Failed SQL statement: SELECT "Form"."id" AS "Form_id", "Form"."createdAt" AS "Form_createdAt", "Form"."updatedAt" AS "Form_updatedAt", "Form"."name" AS "Form_name", "Form"."userId" AS "Form_userId", "Form__user"."id" AS "Form__user_id", "Form__user"."createdAt" AS "Form__user_createdAt", "Form__user"."updatedAt" AS "Form__user_updatedAt", "Form__user"."email" AS "Form__user_email", "Form__user"."name" AS "Form__user_name", "Form__user"."password" AS "Form__user_password" FROM "form" "Form" LEFT JOIN "user" "Form__user" ON "Form__user"."id"="Form"."userId" WHERE "Form"."id" IN ('');

Here's the line:

const loaded_form = await Form.find({ user });

Chaging it to:

const loaded_form = await Form.find({ where: { user } });

Results in the same error.

Edit: I should note that it isn't erroring out in your test specifically, but in my use case. That's the only additional step that deviates from your test in terms of order of function call.

[oguimbal]

Are you sure this usecase could work against a real instance of PG ?

The error is due to the WHERE "Form"."id" IN (''); part of the generated request. "id" being an uuid, and the string '' not being a valid uuid value, thus the condition is invalid.

[oguimbal]

I tried adding the line you're refering to to the test.

Works on my machine 😶

(and user.id is not empty, thus the generated request is not the same)

Is there something that changes between your setup & mine ?

Have you registered extensions to pg-mem like uuid-ossp ? (When attaching a debugger, I can see that Typeorm is trying to use those... exceptions like Extension does not exist: uuid-ossp are being thown, but it seems that Typeorm catches them and fallsback to something else to generate uuids)

[MirrorBytes]

Here's my setup for pg-mem:

import { newDb, DataType } from 'pg-mem';
import { v4 } from 'uuid';

import { User } from '../../entities/user';
import { Form } from '../../entities/form';
import { Submission } from '../../entities/submission';

const db = newDb({
  autoCreateForeignKeyIndices: true,
});

export async function setupTest(schema: string): Promise<void> {
  console.log(schema);
  db.getSchema('pg_catalog').registerFunction({
    name: 'col_description',
    args: [DataType.int, DataType.int],
    returns: DataType.text,
    implementation: () => '',
  });

  db.registerExtension('uuid-ossp', (schema) => {
    schema.registerFunction({
      name: 'uuid_generate_v4',
      returns: DataType.uuid,
      implementation: v4,
    });
  });

  db.registerExtension('citext', (schema) => {
    schema.registerFunction({
      name: 'citext',
      args: [DataType.text],
      returns: DataType.text,
      implementation: (arg: string) => arg.toLocaleLowerCase(),
    });
  });

  const connection = await db.adapters.createTypeormConnection({
    type: 'postgres',
    entities: [User, Form, Submission],
  });

  await connection.synchronize();
}

I got it to stop throwing those exceptions with that.

I've been testing my use case with a real PG instance and Jest running in band; going to replace it with pg-mem when able.

I can confirm my user is populated, and its id is generated. It just appears that it's not populating in the generated query.
I'm going to play around with my use case for a little bit and see if I can figure out what's happening. This may just be the way I implemented something

[oguimbal]

That's weird. I now have the exact same setup as yours (no more exceptions), but it works on my side...

Whats is your exact Typeorm version (really installed, not in your package.json) ... cat node_modules/typeorm/package.json| grep version ? I've got 0.2.29 ... and [email protected] ?

Side note 1: The function col_description is now embbeded in pg-mem. No need to declare it.

Side note 2: you should wrap your Typeorm connection in a structure that forces your unit test to close the connection once done. For instance, like this ... if you dont, and forget to close connection.close() at the end of a unit test, it could impact following unit tests (your tests wont be isolated anymore), since Typeorm has static variables that stores the current connection.

[MirrorBytes]

Ok, so in isolation it still throws that error.

[oguimbal]

console.log('User ID', user.id);
const loaded_form = await Form.find({ user });

What is the console.log showing ?

[MirrorBytes]

Wow, I messed up the origin of the error. I need more caffeine, sorry for the confusion lol
Well technically, it failed at the wrong test due to Jest's timing.

So, the error is originating from another test where I'm trying to retrieve a form that doesn't exist:

const form = await Form.findOne(form, { relations: ['user'] });

It's failing because it's still attempting to pull a form from a user (when that user doesn't exist either).

[oguimbal]

Haha that happens.

Isnt it supposed to fail, then ?

If not, could you provide some code to reproduce it ?

[MirrorBytes]

It's supposed to fail and be caught, but here's my check internally in my use case:

    try {
      const check = await Form.findOne(form, { relations: ['user'] });

      if (!check) return new Result(false, 'Form does not exist!');

      return check;
    } catch (e) {
      return new Result(false, 'Form does not exist!');
    }

The error never passed into the catch.

Edit:

Forgot the test:

  setupTest('is not successful with non-existing form', async () => {
    const res = await baseRet<QueryFormArgs>(GET_FORM, {
      form: '',
    });

    expect(res.data?.form).toMatchObject({
      ok: false,
      msg: 'Form does not exist!',
    });
  });

setupTest is nearly identical to your isolation method that you mentioned before.

[oguimbal]

You're sure that its the statement in your try/catch that throws ?? I dont see how this can be 😳
And what is the form argument that you're passing to findOne ? I guess it has something like { id: '' } in it ?

[MirrorBytes]

Positive, and that's correct. The purpose of that try/catch is specifically to catch the string to uuid conversion error that throws when the string does not match a uuid pattern.

Here's the resolver function I'm using with type-graphql:

const FormResultUnion = createUnionType({
  name: 'FormResult',
  types: () => [Result, Form],
  resolveType: (val) => {
    if ('ok' in val) return Result;

    if ('name' in val) return Form;

    return undefined;
  },
});

...

  @Query(() => FormResultUnion)
  async form(@Arg('form') form: string): Promise<typeof FormResultUnion> {
    // Avoid GraphQL UUID error.
    try {
      const check = await Form.findOne(form, { relations: ['user'] });

      if (!check) return new Result(false, 'Form does not exist!');

      return check;
    } catch (e) {
      return new Result(false, 'Form does not exist!');
    }
  }

The weird part is that the test passes while presenting me with the error message. If I put another test after this one (just a blank), it immediately fails with that error message.

And I've confirmed that it does indeed enter the catch, but still throws that error.

[MirrorBytes]

Here's a screenshot of it:

[oguimbal]

Glad to hear it 👍 !