Models need to be updated about pathfinder

[mglinski]

Here is an example database init file using the cortex setUp() commands for each model.

https://gist.github.com/mglinski/d05b7954cb04fbad4d4f

Once all the mods are updated with the properties that you have linked in your schema picture, running that code will build the database tables for the user.

[mglinski]

I have reverse engineered the SQL schema and will be posting a copy once I confirm that most things are working correctly. It is prolly missing some static data, but this will hopefully save some work for someone else in the meantime.

[exodus4d]

I had some problems with creating all the necessary foreign indexes (especially multicolumn indexes) with just the Setup method implemented in Cortex. But it is not required to configure each field in a Cortex-Models.
The script is intelligent enough to parse the table and column types (int, varchar,...) and generate the Schema layer automatically and store it in cache.

An other point is, that static data, has to be imported manually anyway. Therefore I have decided to stick with a single SQL Dump Ill give you once the beta is over.

[mglinski]

I have successfully reverse engineered the entire database, all types and scopes, etc. I have a working install now, and the setup process is very easy now and can be fully automated.

The problem with not fully describing your models is the exact above situation, you rely on a specific database's SQL schema dump to use the program. You could just run setUp on all the models and it will not only build the tables for you but diff any changes and keep you up to date. Foreign keys are really annoying to automate, but it usually just comes down to manually naming them and executing the table scaffolding in the correct order. I did a pass of foreign keys based on your schema screenshot and was having issues saving or deleting data so there is either some schema discrepancies or some bugs that we could be helping squash.

Testing/debugging/contributing to this would be soo much easier though if the schema was available, I will be posting what I have after another day or so of wider testing as there is a lot of interest in trialing this program.

[mglinski]

I made a wiki article to help with setting this app up for people who might not know 100% what to do.

The DB Dump is in there: https://github.com/exodus4d/pathfinder/wiki/beta

[exodus4d]

Official Clean Database Export is in the updated wiki

[ikkez]

hi @exodus4d. Regarding your request, I did a quick look through your models and the DB dump.
It is possible to add indexes and unique indexes with the cortex field configuration like this:

class CharacterModel extends BasicModel {
    protected $fieldConf = [
        'active' => [
            'type' => \DB\SQL\Schema::DT_BOOL,
            'index' => true,
            // 'unique' => true,
        ]
        // ...       
    ];

Once you trigger CharacterModel::setup(), the indexes should be added automatically. I guess I should mention that are is a (maybe stupid) naming convention for the index names that are generated this way (But I'm sure there was a reason behind that). Maybe I should refactor that behaviour, as it already happend to me that I hit the max index key name of 64 chars, but basically it's working like that.
For multi-column indexes, currently the only way is to add them manually:

class AllianceMapModel extends BasicModel {
    // ...
    static public function setup($db=null, $table=null, $fields=null) {
        parent::setup($db,$table,$fields);
        $schema = new \DB\SQL\Schema($db);
        $table = $schema->alterTable('alliance_map');
        if (!in_array('alliance_map___allianceId__mapId',$table->listIndex())) {
            $unique = true;
            $table->addIndex(array('allianceId','mapId'),$unique);
            $table->build();
        }
    }
}

Also not included in the Schema Builder is something like:

updated timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP

because that's something mysql specific, which isn't easy portable for postgre, sqlite and others (only with custom db triggers). Therefore a cross-compatible way would go like this:

'updated' => array(
    'type' => \DB\SQL\Schema::DT_DATETIME
    'default' => \DB\SQL\Schema::DF_CURRENT_TIMESTAMP,
)

and in my model, I used the beforeupdate hook, which calls $this->touch('updated');, which set's the current time to the field.

And regarding foreign key constraints. Unfortunately these are not available at all at the moment, as I mostly use the model event hooks for update and delete cascades for compatibility reasons across the different db engines.

[exodus4d]

Regarding to #11, #3

Im currently working on a "Setup/Bootstrapping" functionality for Pathfinder. This will set up and update the database Tables/Indexes/Relations.... Here is the plan:

1. A new route is added to the routes.ini "/setup", where you will find a new Dashboard with some helpful information:
   • Shows some Environment/OS/Server information
   • Shows some configuration parameters from pathfinder.ini
   • Shows required Server/PHP/MySQL modules and their installed versions.
   • Outdated/Missing modules (Fat Free Framework requirements) are highlighted (orange)
   • Connection status for both required DBs will be shown. This includes a list with all tables.
2. If everything is OK (green), you see some CTA-Buttons for bootstrapping/updating the DB
   • Each child Cortex DB-Model(28) will implement/overwrite its own setup();function if needed, to setup its table and all required dependencies.
   • This will set up and configure all tables in the Pathfinder DB
3. The main DB setup function can be called after each release to receive updates/changes (new columns... )
4. If you are done with the setup process. Uncomment/Delete the route "/setup" from routes.ini to prevent abuse.

As you can imagine, this is a lot of work :( I cant promise when will be released (hope end of 2015). But this is probably the last big new feature before Pathfinder leaves Beta phase.

@ikkez thanks a lot for your post!

• I have found some minor problems with the setup() function while testing the process with the first two models. E.g. Setting up a table for the first time (new table), I´m able to create varchar columns without default values. But if I add a new varcharcolumn afterwards without default value, I run into a "trigger_error()". If you are interested I could provide some more Information and LineNumbers.
• An other problem is, that columns can not be changed once they are created e.g. change default value.
• The "delete" column functionality is commented out. I guess this is for security reasons

Here is a WIP Screenshot with the current state of the Dashboard.

[ikkez]

@exodus4d regarding varchar columns without a default value: Well the underlying schema builder was build to provide an API to create tables across the common different database engines. Maybe MySQL does not shout at you when adding those fields, but PostgreSQL and others don't like that. If you setup your model at a postgre database and would remove that check, you'll get a PDO error instead. It doesn't even make sense to add a column that should not be null, but does not have a default value, as it would be NULL for every existing record, which would violate its own column definition. Just drop the "not null" part if you don't have a value for it, or provide a default.

To update a column, there is currently the ->updateColumn method in the schema builder, which indeed only can update the datatype but not the default value. I can investigate this. Basically the schema plugin is not a fully schema migration tool. I.e. if you plan to rename columns, you have to write something that knows which column to rename (or drop?), because this cannot be made fully automatical in cortex or the schema plugin.

[exodus4d]

@ikkez thanks again, for your explanation! :) Maybe you could help me once more:

I have a Model with a relation that looks like this (simplyfied):

class UserModel extends BasicModel {

    protected $table = 'user';

    protected $fieldConf = [
        'lastLogin' => [
            'type' => Schema::DT_TIMESTAMP,
            'index' => true
        ],
        'active' => [
            'type' => Schema::DT_BOOL,
            'nullable' => false,
            'default' => true
        ],
        'name' => [
            'type' => Schema::DT_VARCHAR128,
            'nullable' => false,
            'default' => '',
            'index' => true,
            'unique' => true
        ],
       'apis' => [
            'has-many' => ['Model\UserApiModel', 'userId']
        ]
    ];
...

... and a counterpart for that relation defined in Model\UserApiModel [1:m]:

class UserApiModel extends BasicModel {

    protected $table = 'user_api';

    protected $fieldConf = [
        'userId' => [
            'belongs-to-one' => 'Model\UserModel'
        ]
    ];

Now i would like to setup Model\UserModel:

Model\UserModel::setup()

The table is successfully created with all indexes and unique keys.
But there is also a column created (named apis, type: Schema::DT_INT) which is in fact a virtual field for that relation.

This happens because setup(); gets all configuration information (like $fieldConf) from calling $self::resolveConfiguration().
Afterwards there is a loop over all received fields and within this loop the following function is called for each field:

$field = static::resolveRelationConf($field);

This function inclues a array_key check for my defined relational field and type: Schema::DT_INT is added:

if (array_key_exists('has-many', $field)){
    $field['relType'] = 'has-many';
    if (!isset($field['type']))
    $field['type'] = Schema::DT_INT;

1. Is this behavior intended?
2. Is there simple way to "skip" relational fields for column creation?

Thanks :)

[ikkez]

yeah that's a known bug which should have been fixed here ikkez/f3-cortex@c7f50e9
So it's in the next version, I can release v1.4.1 soon. In the meantime you can just fetch the current dev version.

[exodus4d]

👍

[ikkez]

Hi @exodus4d
I worked on an update for the updateColumn method. Maybe you can have a look if it works for you. You can find that dev-version right here: https://github.com/ikkez/f3-schema-builder/tree/dev-2.2

Basically I modified the updateColumn method, so that it accepts a Column object as 2nd parameter as well. Within that object, you can just set the new column datatype, null flag, and default value:

$cols = $table->getCols(true);
$foo = $cols['foo'];
$col = new \DB\SQL\Column('foo',$table);
$col->copyfrom($foo); // copy from existing column
// alter what you need:
$col->type_varchar(60); // adjust type
$col->defaults('great'); // change default
$col->nullable(true); // set nullable
$table->updateColumn('foo',$col);

Some unit tests are here. Not the most beautiful code, but successfully tested with mysql,sqlite,pgsql,mssql2008/2012.

[exodus4d]

wow nice! This feature comes at the right time. :) The table/column "check for change" and table setup functionality is working for now. The code is very generic and could be used for any Cortex-Model. The last part is now the updateColumn() function. Ill try your update.

Im currently setting up a fresh CentOS7 server running PHP7 and MariaDB for testing, but i dont expect any major problems:

[exodus4d]

@ikkez

I´m currently testing your new updateColumn() function. It works pretty well so far! There is just one minor thing to mention.
When I use column configs with 'type' = self::DT_JSON and compare the types by $schema->isCompatible($colType,$colDef) I always get false as response. (column type text, schema type JSON)
That´s because DT_JSON and DT_SERIALIZED are special datataypes implemented by the Cortex class and not known by Schema. I made a small "workaround" for my usecase.
So its not a bug ;)

[ikkez]

This thing looks so cool. congrats for the nice work. Yeah the DT_JSON type is just a text type. If you have a workaround for this already, it's fine. Some database engines already support native json data types in their latest versions, so maybe it's worth to check if it's usable with the F3 mappers. I guess there are some things to patch, but basically mysql 5.7, pgsql 9.3 and mariadb 10 already have own JSON datatypes now. unfortunately creating, inserting and reading them works differently everywhere, so this could be something for a future version of F3, SchemaBuilder and Cortex.

[exodus4d]

I´m finally done! All features are implemented and tested.

• The code is already on the develop branch. - master will be updated the next deys
• The WiKi - Setup - Database is updated with the new bootstrapping/update process. -> It can be done by two clicks ;)
• More Screenshots in the Installation Section on the new URL: https://www.pathfinder-w.space
• I´m on a new Youtube video, that demonstrates the process...

[ikkez]

👍