ryangjchandler / orbit Goto Github PK

I released a small package, https://github.com/ryangjchandler/git, for working with Git inside of PHP and would like to refactor to use that with Orbit instead of the package I'm currently using.

Suggestion from @stancl that would be similar to migrate:fresh - clear out the Orbit specific content directories.

Good day!

Are there any plans to add a setting that would prohibit the model during initialization from checking for data updates in files at the time of execution.

trait Orbital

        if (
            Orbit::isTesting() ||
            filemtime($modelFile) > filemtime(Orbit::getDatabasePath()) ||
            $driver->shouldRestoreCache(static::getOrbitalPath()) ||
            ! static::resolveConnection()->getSchemaBuilder()->hasTable((new static)->getTable())
        ) {
            (new static)->migrate();
        }

And add a command, which, for example, at the time of unloading, did this check and, if needed, recreated and updated the data in the cache.

I planned to use your library for storing various small reference books.

Such as types of documents, types of prices, type of goods, services.

Perhaps with the release of PHP 8.1 and ENUM, this will not be necessary, although it is very convenient to pull up data from the relations.

Thank you and sorry for my English

I'm thinking that, in some cases, Driver objects might want to add their own columns to the schema. A good example would be the Markdown driver.

To save some effort from the developer, their should probably be a schema method on the driver that adds the column instead.

Related to #27.

Tests are failing because of ::getAnnotations - need to dive into why.

I just played around with the markdown driver and it seems a bit fragile. When I modify a file directly or add another file to the folder I get an empty collection when running Model::all(). Neither reverting the changed nor deleting the sqlite database or the new file helps. Only when I removed all files and start over the driver starts working again.

No issues with the YAML and JSON driver.

This command, orbit:commit, would force commit any changes inside of the content/ directory. Perfect for when you want to run it manually or periodically via the scheduler.

This would be similar to the Markdown driver - parse out the front-matter and compile the Markdown.

The difference would be that it parses the content as Blade as well, passing through the current model instance. So you could do something like:

---
title: My Awesome Post
tags:
  - laravel
  - php
---

# {{ $this->title }}

@foreach($this->tags as $tag)
    <span>{{ $tag }}</span>
@endforeach

⚠️ The only data being passed through to the Blade would be the current model, but this doesn't stop anybody editing the file from using dangerous elements such as database queries etc, I'd generally recommend against using this driver when editing content via a GUI / CRUD form. If you want to enable that, use the regular Markdown driver and parse the content out yourself with shortcodes (or something similar).

I'd like to add a new Json driver that can read and write JSON files. It's a pretty simple concept and will be a good opportunity to refactor any duplicated code between drivers.

I'd also like to make the driver configurable - specifying the number of spaces for indentation, etc.

I think it would be nice to have some testing utilities for Orbit - especially for testing that content exists.

Added to my project today as was looking for a flat-file version of a db since I really only have one table that I would be creating. Can you please help on getting the migration working, and how exactly I'd be seeding it?

Additionally, since running migrations was still trying to connect to "mysql",
I updated the config > databases > 'default' => 'orbit' // correct or no?

For the migration up() I have:
Schema::create('products', function (Blueprint $table) { $table->string('identifier')->unique(); $table->string('name'); $table->string('decryption_key'); $table->string('api_key'); $table->string('api_end_point'); $table->json('api_data_config'); $table->json('api_registration_config'); });

Running it I get (did a remove of the previous run when I saw that it stating "exists"):

$ rm -rf content/products/
$ php artisan migrate
Migrating: 2021_04_23_000000_create_products_table

Illuminate\Database\QueryException

SQLSTATE[HY000]: General error: 1 table "products" already exists (SQL: create table "products" ("identifier" varchar not null, "name" varchar not null, "decryption_key" varchar not null, "api_key" varchar not null, "api_end_point" varchar not null, "api_data_config" text not null, "api_registration_config" text not null))

at vendor/laravel/framework/src/Illuminate/Database/Connection.php:678
674▕ // If an exception occurs when attempting to run a query, we'll format the error
675▕ // message to include the bindings with SQL, which will make this exception a
676▕ // lot more helpful to the developer instead of just the database's errors.
677▕ catch (Exception $e) {
➜ 678▕ throw new QueryException(
679▕ $query, $this->prepareBindings($bindings), $e
680▕ );
681▕ }
682▕

  +9 vendor frames

10 database/migrations/2021_04_23_000000_create_products_table.php:49
Illuminate\Support\Facades\Facade::__callStatic("create")

  +21 vendor frames

32 artisan:37
Illuminate\Foundation\Console\Kernel::handle(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))

I've encountered some issues but I can't conclusively say if they were Orbit-related. Could be nice to add tests for this.

While it is useful to jump into tinker or tinkerwell and just Post::create() it would be cool if you could have an interactive CLI so you could:

php artisan orbit:generate ModelName

Which would then ask you to give a name/title depending on how the model is routed.

At the moment, a change to a single file on disk will result in the entire Orbit (SQLite) cache being restored.

This can be problematic for large datasets and cause performance issues.

Without looking into it too much, my initial thoughts are:

1. Instead of clearing the cache and re-caching all of the files, find all files that have a "modified at" greater than that of the SQLite cache.
2. Loop through each of those files and cache accordingly (updateOrCreate).

Currently, Orbit will only pick up on content in the top-level of the model directory. This means that nested structures aren't currently supported.

To be completely honest, I'm not sure on the best way to support this. It's going to be a tricky one to get right because of the following:

The filename used for content matches the primary key. If you have nested content, e.g. content/posts/sponsors/hello-world.md - how does you generate the primary key for this when creating? You would need to have some sort of hierarchy that you follow.

Say you had a slug prop as the primary key, the slug for this would technically be sponsors/hello-world, so to get the model you'd need to do Post::find('sponsors/hello-world')...

When using Laravel Scout, search($query)->get() will always return an empty array, even though search($query)->raw() returns a valid result.

Would it be possible to add a 'disable' flag? The reasoning behind this is that if the disable flag is set, the model behavior falls back to the default Eloquent/DB behavior (even if the orbit trait is used in the model). This way, you could build your data using the DB and then export it to orbit flat file storage for use through orbit after applying DB transformations (joins, etc.)

I'm not sure this makes sense but I have a use case for such behavior and if possible, I think it would be a useful addition.

At the moment, the content column that the Markdown field uses for the actual markdown isn't configurable and is hard coded. This should be configurable so that the user can use their own column name.

Keeping this list here to track who is using Orbit.

Hi, thanks for Orbit!
Is there a way to customize / add padded zeroes to the filenames Orbit creates? This would sort the files better in a code editor that doesn't understand numerical sorting (like Sublime). Curious about your opinion.

I think it would be good to distinguish between the 2 events.

This looks super promising for me, but I have a couple of questions (if that's okay!)

I have a database table in MySQL of 391,944 rows (threads), and another of 5,360,127 rows (posts) that I would love to convert over to this flat file format to reduce the stress on my MySQL database. Each one of the 391,944 threads contains some of the 5,360,127 posts.

If we divide the two figures, it works out at around 13.67 posts per thread. Obviously some threads have 1 post while others have thousands.

Can this package support this? I imagine it'd be content/threads/{id} and content/posts/{id} and each thread could read from each post just like with the traditional MySQL driver?

Can we use this Orbit driver on only specific Models? I imagine it's something like this in my ForumPost and ForumThread Model?

    protected $connection= 'orbit';

If the answer to the above is yes, what would the procedure be for converting my MySQL database rows over to flat files? Do you have a tool for this or would I have to do something like change over the connection, then create a command line tool to run through every thread and every post and re-save them, causing the Eloquent methods for saving to be called and write the flat file?

Sorry for lots of questions! I think this could help lessen the load on our database server

The 1.1.0 release adds the following line to the boot method of the OrbitServiceProvider to truncate the _orbit_meta table if it already exists.

https://github.com/ryangjchandler/orbit/pull/95/files#diff-9ab2c0c6607186af00d0a727b04179cd17605199ab1d7662338dd6d7a19ff223R70

Doing so updates the last modified timestamp of the Sqlite database. This introduces a bug in the shouldRestoreCache method of the FileDriver class since it compares the timestamp of the database file to the last modified timestamp of the content files. This will always return false now since the database gets touched at the beginning of each request. This means that the cache never gets updated and we get served stale content until we completely remove the database.

I'm not sure what the best way to fix this is as I didn't look too deeply into the changes in 1.1.0. This might also have slipped through the test suite since the database always gets migrated in tests due to Orbit::isTesting() being checked first.

Hey,
just played a little bit with your package. Either I don't understand the getKeyName() method or it doesn't work. When I use the example from the readme with slug the slug is set to an auto-incrementing ID instead of the value I pass. Is this intended?
I'd expect to pass any value that is used to store the file then.

This is quite a high priority one since the code base is untested at the moment (I'm manually testing things in a workbench application).

Here's everything the test suite should cover:

• Adding Orbital to a model.
• Changing the primary key on a model.
• Changing the driver for a model.
• Changing the name of a model's content directory.
• Creating new models and writing files to disk.
• Updating existing models and updating files on disk.
• Deleting models and removing files from disk.
• Soft deleting models and updating deleted_at.
• Check that all events are dispatched correctly.
• Write tests for the Git suite (?)

If my Model has a column with a default value and I create a new Instance, it doesn't get saved in file.

My Blueprint inside the Model:

public static function schema(Blueprint $table)
{
    $table->string('title');
    $table->string('link');
    $table->string('author')->default('Max Mustermann');
}

The create Method:

Article::create([
    'title' => 'Test',
    'link' => 'test-link'
]);

The generated file:

---
title: Test
link: test-link
---

What I expect:

---
title: Test
link: test-link
author: Max Mustermann
---

It's weird because the sqlite cache actually works like expected and fills the author column with the default value. But the next time it's refreshed, I get a Integrity constraint violation: 19 NOT NULL constraint failed. articles.author

Feature request related to this issue: If a NOT NULL field has a default, don't make it mandatory to have it set in the file. Just return the default if it's missing.

Good package though, big fan of it.

Running the seeder, I get an extra attribute content:

       App\Models\Product {#4278
         identifier: "test",
         name: "test instance",
         decryption_key: "abc",
         api_key: "abc",
         api_end_point: "{"login_validation":{"url":"api\/login","data_mapping":{"username":"username","password":"password"}},"registration":{"url":"api\/create\/","data_mapping":{"first_name":"fname","last_name":"lname","email":"mail","cell_phone":"phoneCell","password":"password"}}}",
         content: null,
         created_at: "2021-04-26 19:27:05",
         updated_at: "2021-04-26 19:27:05",
       },

Using tinker, using the same command, does not add it:

>>> \App\Models\Product::all()->each->delete();  // deleting to clear the table
=> Illuminate\Database\Eloquent\Collection {#3337
     all: [],
   }
>>>         \App\Models\Product::create([
...             'identifier' => 'test',
...             'name' => 'test instance',
...             'decryption_key' => 'abc',
...             'api_key' => 'abc',
...             'api_end_point' => json_encode([
...                 'login_validation' => [
...                     'url' => 'api/login',
...                     'data_mapping' => [
                        'username' => 'username',
                        'password' => 'password',
                    ]
                ],
                'registration' => [
                    'url' => 'api/create/',
                    'data_mapping' => [
                        'first_name' => 'fname',
                        'last_name'  => 'lname',
                        'email'      => 'mail',
                        'cell_phone' => 'phoneCell',
                        'password'   => 'password',
                    ]
                ],
            ]),
        ]);
// below is the record as expected, created within tinker
=> App\Models\Product {#4333
     identifier: "test",
     name: "test instance",
     decryption_key: "abc",
     api_key: "abc",
     api_end_point: "{"login_validation":{"url":"api\/login","data_mapping":{"username":"username","password":"password"}},"registration":{"url":"api\/create\/","data_mapping":{"first_name":"fname","last_name":"lname","email":"mail","cell_phone":"phoneCell","password":"password"}}}",
     updated_at: "2021-04-26 19:31:46",
     created_at: "2021-04-26 19:31:46",
   }
>>> \App\Models\Product::all()->each->delete(); // delete to start clean again
=> Illuminate\Database\Eloquent\Collection {#4275
     all: [
       App\Models\Product {#4276
         identifier: "test",
         name: "test instance",
         decryption_key: "abc",
         api_key: "abc",
         api_end_point: "{"login_validation":{"url":"api\/login","data_mapping":{"username":"username","password":"password"}},"registration":{"url":"api\/create\/","data_mapping":{"first_name":"fname","last_name":"lname","email":"mail","cell_phone":"phoneCell","password":"password"}}}",
         content: null,
         created_at: "2021-04-26 19:31:46",
         updated_at: "2021-04-26 19:31:46",
       },
     ],
   }
>>> \App\Models\Product::all()
=> Illuminate\Database\Eloquent\Collection {#4280
     all: [],
   }
>>> ^D
Exit:  Ctrl+D

$ php artisan db:seed
Seeding: Database\Seeders\ProductSeeder
Seeded:  Database\Seeders\ProductSeeder (15.43ms)
Database seeding completed successfully.

$ php artisan tinker
Psy Shell v0.10.8 (PHP 8.0.2 — cli) by Justin Hileman
>>> \App\Models\Product::all()
=> Illuminate\Database\Eloquent\Collection {#4276
     all: [
       App\Models\Product {#4278
         identifier: "test",
         name: "test instance",
         decryption_key: "abc",
         api_key: "abc",
         api_end_point: "{"login_validation":{"url":"api\/login","data_mapping":{"username":"username","password":"password"}},"registration":{"url":"api\/create\/","data_mapping":{"first_name":"fname","last_name":"lname","email":"mail","cell_phone":"phoneCell","password":"password"}}}",
         content: null,
         created_at: "2021-04-26 19:32:29",
         updated_at: "2021-04-26 19:32:29",
       },
     ],
   }
>>>

See above created_at.
I used the same create() statement in tinker as in the seeder (cut/pasted).

my model:

<?php

namespace App\Models\API;

use Illuminate\Database\Eloquent\Model;
use Illuminate\Database\Schema\Blueprint;
use Orbit\Concerns\Orbital;

class Test extends Model {

    use Orbital;

    /**
     * The database connection that should be used by the model.
     *
     * @var string
     */
    protected $connection = 'orbit';

    /**
     * Indicates if the model's ID is auto-incrementing.
     *
     * @var bool
     */
    public $incrementing = false;

    /**
     * The primary key associated with the table.
     *
     * @var string
     */
    protected $primaryKey = 'name';

    /**
     * The data type of the auto-incrementing ID.
     *
     * @var string
     */
    protected $keyType = 'string';

    /**
     * Indicates if the model should be timestamped.
     *
     * @var bool
     */
    public $timestamps = false;

    /**
     * The attributes that are mass assignable.
     *
     * @var array
     */
    protected $fillable = ['name', 'is_active'];

    /**
     * The model's default values for attributes.
     *
     * @var array
     */
    protected $attributes = [
        'is_active' => 0,
    ];

    /**
     * The attributes that should be cast.
     *
     * @var array
     * */
    protected $casts = [
        'is_active' => 'integer', // same result with boolean and string
    ];

    /**
     *
     * @param Blueprint $table
     */
    public static function schema(Blueprint $table) {
        $table->string('name');
        $table->integer('is_active'); // already checked with bool
    }

}

modifying /vendor/ryangjchandler/orbit/src/Drivers/Json.php to show whats going on:

<?php

namespace Orbit\Drivers;

use Illuminate\Database\Eloquent\Model;
use SplFileInfo;

class Json extends FileDriver {

    protected function dumpContent(Model $model): string {
        $attributes = $this->getModelAttributes($model); // get attributes
        $data = array_filter($attributes); // set attributes
        $output = json_encode($data, JSON_PRETTY_PRINT); // encode
        dd($model, $attributes, $data, $output); // output all
        return $output;
    }

    protected function parseContent(SplFileInfo $file): array {
        $contents = file_get_contents($file->getPathname());

        if (!$contents) {
            return [];
        }

        return json_decode($contents, true);
    }

    protected function extension(): string {
        return 'json';
    }

}

simple code to reproduce:

$test = new Test();
$test->name = 'test';
$test->is_active = '0';
/* // same empty result
$test->is_active = 0;
$test->is_active = false;
$test->is_active = 'false';
*/
$o = $test->save();

this is the model as input for Json-Driver in Json->dumpContent(Model $model)...

App\Models\API\Test {#264 ▼
  #connection: "orbit"
  +incrementing: false
  #primaryKey: "name"
  #keyType: "string"
  +timestamps: false
  #fillable: array:2 [▶]
  #attributes: array:2 [▼
    "is_active" => 0
    "name" => "test"
  ]
  #casts: array:1 [▶]
  #table: null
  #with: []
  #withCount: []
  +preventsLazyLoading: false
  #perPage: 15
  +exists: true
  +wasRecentlyCreated: true
  #original: array:1 [▶]
  #changes: []
  #classCastCache: []
  #dates: []
  #dateFormat: null
  #appends: []
  #dispatchesEvents: []
  #observables: []
  #relations: []
  #touches: []
  #hidden: []
  #visible: []
  #guarded: array:1 [▶]
}

this is the second output, this are the attributes from model, with "is_active=0"

array:2 [▼
  "is_active" => 0
  "name" => "test"
]

after array_filter "is_active" has now been removed

array:1 [▼
  "name" => "test"
]

and this is the result output as json without is_active

"""
{
    "name": "test"
}
"""

I suspect this issue is due to the array_filter here: https://github.com/ryangjchandler/orbit/blob/main/src/Drivers/Json.php#L12 ?

i tried already with other datatypes with same result:
false => empty
(as string) 'false' => true (result as true)
0 => empty
'0' => empty

did I miss something?
thank you for the project!

Currently only files with the right suffix and inside the content/ directory (or whatever set in the config) are accessible. For large projects it would be great if you could sort your database files into subdirectories.

Hi Ryan, great package! I am using this to do some specific tasks, but when trying to fresh seed, the id of the markdown keeps incrementing instead of starting from 1. Is this intentional or... a bug? Thank you

I think something that a flat-file system should have is revisions. Here's what I'm thinking.

---
active_revision: _revisions/20210304-230451.md
---

Then, when the model is loaded in, this revision is pulled in instead as the "active" working copy. If you wanted to rollback to a previous revision, you would be able to with $model->rollback().

The _revisions/* files would actually store all of the data / information.

If you wanted to rollback to a specific version, you could do $model->rollback('20210208-230759') and that would take you back to that specific revision.

You could also provide an integer and that would roll you back X number of times.

This is the last driver that I want to implement as part of the "core". Custom drivers can of course be made.

Here's what I'm thinking:

• Each row in the CSV is a record. So the Driver::all() method would just return a transposed version of each row with the column => value.
• When you create a new record, we just append a new row to the CSV. Probably makes sense to use https://github.com/spatie/simple-excel for this since it relies on LazyCollection already (perfect for huge files).'
• Deleting a record will just filter the rows and write back to the file.
• Updating a record will do the same.

This command will find all of the models in your application that use the Orbital trait and cache the content's accordingly.

This could be good for deployments so instead of caching on the initial request, you instead cache outside.

Currently, if you have some data in a file but the column doesn't exist in the schema, it will throw an exception. I think there's a better way of handling this, but will need to make sure that the performance overhead isn't massive.

I'm wondering if it would be possible to set up configuration where the content files are actually stored somewhere that isn't local to the actual Laravel install, i.e. on S3?

I'm a fan of running Laravel applications serverless using bref, and would love to be able to store small bits of data in S3, instead of having to spin up a DB instance or work with DynamoDB.

Can we use or develop orbit as laravel queue driver this will be really helpful for developer who are deploying their projects on shared hosting, where there is no redis available, we can use sqlite and mysql as laravel queue but problem is both are not good for queues your jobs table get locked after some time.

I am sure queue processing will be really fast with orbit.

It appears the content directories are not being correctly cased as snake_case.

Call to undefined method App\Models\Test::getOrbitOptions()

Replicate:

1. Create fresh Laravel Project
2. Install Orbit
3. Make new Model with Orbital
4. Try to run Model::all() via php artisan tinker

Idk if this maybe an issue with my machine, or if I forgot to configure something?

In many cases, people will want to soft-delete (also restore) records instead of hard-deleting them.
For the markdown driver, we can use soft-delete to move the file into the bin or trash folder for example.

Trying to use Orbit as a flat-file user model doesn't work at the moment because it doesn't store $hidden properties in the file.

This driver would work the same as the normal Markdown driver, but instead of using YAML as the front-matter, it would use JSON instead (YAML with brackets, lol).

would be nice if hasMany() and belongsto() were available in the Model class, like here in docs: https://laravel.com/docs/8.x/eloquent-relationships#one-to-many

Currently, the Markdown driver is turning array casts into strings. The problem is with $model->getAttributes(), which returns the attributes after being casted into their raw form.

Right now there's no License file that indicates how this code is licensed and we are allowed to use it.

would be nice to have Builder for queries in orbit, to do simple queries like:
$post = Comment::where('author', 'John')->get();
i think it's a simple implementation, just foreach all files in db/directory and search for the matching files.
https://laravel.com/docs/8.x/eloquent#retrieving-models