Architecture question; before I fork and start coding about django-podcasting

[rizumu]

This sounds good. I hadn't considered sharing of episodes/enclosures between sites and if it is something you'd like to add I'd be happy to pull it in.

[djangolackey]

The hardest part from my point of view would be the migration. Since I'll be moving the ForeignKey for Enclosure to the other side of the relationship and adding the Many-to-Many on Enclosure for Show and Show for Site,

You don't require South for migrations, so I might need to do some testing locally to see if I'll need custom migration scripts to help people not using south to make the jump.

I may need some feed back on that part especially.

Thanks,

[rizumu]

Migrations have been neglected here so far, but now that 1.7 is out and South has seen its last release, I'd prefer to use the built in Django migrations. A possible idea could be to and the m2m in one migration, perform the data migration in the next, then remove the fk and finally rename the m2m.

[rizumu]

And I can certainly test out the migration with existing data.

[djangolackey]

I've got model changes made with migrations using the Django 1.7 migrations tools. I haven't looked at the tests yet (I'm not really test driven. B) )

I've pushed my changes so far to my fork. Unfortunately, I have lots of commits as I went back and forth with my dev server.

Also, I'm not sure how the migrations will work. One of the catches was that the fk fields had to say on the models while the data migration script ran. Then I removed those fields in separate migration script and code commit; however, if you checkout the Head revision of the code the fields will be gone, which will keep the data migration portion from running correctly.

[rizumu]

On my first pass I hit some errors in initial 0001_initial.py with taggit. I also have taggit installed, so commenting out the lines related to tagging in the 0001_initial.py allowed me to get 0001 to run.

The second migration errors out on a FK constraint:

./manage.py migrate podcasting 0002
Operations to perform:
  Target specific migration: 0002_auto_20140909_0044, from podcasting
Running migrations:
  Applying podcasting.0002_auto_20140909_0044...Traceback (most recent call last):
  File "./manage.py", line 11, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/rizumu/.virtualenvs/scenemachine/lib/python2.7/site-packages/django/core/management/__init__.py", line 385, in execute_from_command_line
    utility.execute()
  File "/Users/rizumu/.virtualenvs/scenemachine/lib/python2.7/site-packages/django/core/management/__init__.py", line 377, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/rizumu/.virtualenvs/scenemachine/lib/python2.7/site-packages/django/core/management/base.py", line 288, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/Users/rizumu/.virtualenvs/scenemachine/lib/python2.7/site-packages/django/core/management/base.py", line 338, in execute
    output = self.handle(*args, **options)
  File "/Users/rizumu/.virtualenvs/scenemachine/lib/python2.7/site-packages/django/core/management/commands/migrate.py", line 160, in handle
    executor.migrate(targets, plan, fake=options.get("fake", False))
  File "/Users/rizumu/.virtualenvs/scenemachine/lib/python2.7/site-packages/django/db/migrations/executor.py", line 63, in migrate
    self.apply_migration(migration, fake=fake)
  File "/Users/rizumu/.virtualenvs/scenemachine/lib/python2.7/site-packages/django/db/migrations/executor.py", line 97, in apply_migration
    migration.apply(project_state, schema_editor)
  File "/Users/rizumu/.virtualenvs/scenemachine/lib/python2.7/site-packages/django/db/backends/schema.py", line 82, in __exit__
    self.execute(sql)
  File "/Users/rizumu/.virtualenvs/scenemachine/lib/python2.7/site-packages/django/db/backends/schema.py", line 98, in execute
    cursor.execute(sql, params)
  File "/Users/rizumu/.virtualenvs/scenemachine/lib/python2.7/site-packages/django/db/backends/utils.py", line 81, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "/Users/rizumu/.virtualenvs/scenemachine/lib/python2.7/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
  File "/Users/rizumu/.virtualenvs/scenemachine/lib/python2.7/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/Users/rizumu/.virtualenvs/scenemachine/lib/python2.7/site-packages/django/db/backends/utils.py", line 65, in execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: insert or update on table "podcasting_episode" violates foreign key constraint "podcas_enclosure_id_1156b89efc0bc8c5_fk_podcasting_enclosure_id"
DETAIL:  Key (enclosure_id)=(0) is not present in table "podcasting_enclosure".

Did these migrations work on your project? If you have finished with the model changes, I could attempt to step through generating the migrations with my project too.

[djangolackey]

That's actually an interesting error. What db are you using?
I'm just running sqlite in my test bed.

Django migrations recommended creating the ForeignKey with default key value of 0, then I populate the real value with the second migration and finally remove the old foreign key with the third migration.

I may have to go further and add the column as a nullible numeric initially then convert it to a ForeignKey after the data migration script is complete.

Thanks,

[rizumu]

Hi @djangolackey I'm using Postgres, haven't tested it on sqlite or mysql, but I do know they both have more relaxed constraints than Postgres, so it is best to generate migrations via Postgres.

[djangolackey]

I believe I've overcome the migration issues by using a single migration file and a slightly different strategy. I'll test on postgres to be sure.

A new issue has arisen that I didn't really anticipate.
The URLs for Shows uses the AutoSlugField of Title field and unique_with="site"
The URLs for Episodes uses the AutoSlugField of Title and unique_with="show__site"

The issue here is Show has a ManyToMany relationship with Site and Episode has a ManyToMany relationship with Show with has a ManytoMany relationship with Site.

I see two options here:

1. Have the Title unique across the board. The AutoSlugField doesn't allow the unique_with option on ManyToMany fields so they can't traverse this problem
2. Don't use the AutoSlugField. Instead use the Form Class to populate the SlugField and traverse the ManyToMany relationships to either fail the form or increment the slug.

Let me know your preference.

Thanks,

[rizumu]

Sounds great! My preference on the slug is the first option because it is the simplest and if users really need the same slug on different sites a manual intervention could be made to change the object in the database.

One other important part of the codebase is feeds.py, just double check the rss/atom feeds are generated as expected.

[djangolackey]

I had a epiphany as to why you set up the foreignkey from enclosures to episodes just now while working on the feeds.

You allow different mimetypes for the same podcast episode.

My changes will not allow that unless I either make it a ManyToMany relationship or revert that change.

I'm going to talk to my primary user. Do you see this as a tell use case?

Thanks

[rizumu]

A m2m with a unique together between episode and mime would work. I am thinking that forms are more involved with m2m, but that is the tradeoff for the flexibility. Although for the amount of times enclosures would be shared on multiple episodes, it could just be easiest to upload again. Maybe lets hold off on Enclosure m2m since it is rather rare, but if you do think you'll use it we can make it work.

Btw, nice call on moving the mime choices to settings.

[djangolackey]

Thanks. My primary use uses m4a files. Go figure.

We should probably either make the M2M work now or declare the new behavior. If someone is using multiple enclosures with an episode they will lose the second relationship with the migration as it stands.

I'm leaning toward the M2M since it would leave existing data functioning in a predicable way. I like the unique together option. I hadn't thought of that.

I promise I'm attempting to take care while still making progress. :-)

[rizumu]

All sounds good to me to cover this ground all at once. I am looking forward to the show to site m2m, and even the enclousre to episode I can think of some potential use cases.

[djangolackey]

in order to make the jump in a single commit a created a new branch that contains all the changes. You can find it here.

https://github.com/djangolackey/django-podcasting/tree/update_models

I haven't touched the tests. I need to go back and look at those. But I'm done for tonight. Let me know what you think so far. Once I look at the tests I'll feel better about an official pull request.

Thanks,

[rizumu]

Okay, looking good. Migrations worked with my data and I was able to traverse the hierarchy of site -> enclosuret as expected in the shell. I can't actually run my site until I update the views, forms and templates to match, but I can wait until this lands. I am using factory boy for testing atm, there aren't many tests tbh, so any improvement there is welcome.

And at any point you want to make a PR, go ahead. Its actually easier to review and comment inline, see the whole diff in one go with a PR vs viewing the branch, even if it is still a WIP. So even if there are a lot of little commits, the main PR the diff updates with each push and we could merge it once everything is reviewed. Thanks for you work on this so far! Its looking great.

[rizumu]

Ah, another benefit of making the PR is that Travis should run the tests and warn inline of any errors.

[djangolackey]

Pull request created. Thanks. #16