How to work with relationships?

[rafis]

Seems support of relationships is very bad. And it is hard to debug because django_sharding_library.router.ShardedRouter.allow_relation is never gets called.

[JBKahn]

I believe it's used to determine if models are allowed to be foreign keyed to eachother.

So probably only used at either migrating creation time or at migration time. I can take a look later and point you to the relevant source code in Django.

Support is meant to be limited to foreign keys on the same database.

[rafis]

Here is my code:
https://github.com/rafis/gmv

It is a bit messy. I have two shard groups sets_shard_group and items_shard_group. And I need many to many relationship between these shards and as well as with users table. When I try to create simple object in shard groups it throws an exception related to relationships:

from sharded_storage.models import Set
set1 = Set.objects.create()
set1.name = 'asdasd'
set1.save()

item1 = set1.items.create()

raises an exception that it can't find main.auth_user table in the database of sets_shard_group. And definitely it can't because auth_user is in other databases which is default database. And also django_sharding_library.router.ShardedRouter.allow_relation is never gets called. Nor then makemigrations, nor migrate, nor experimenting like shown above.

[JBKahn]

I'm not sure about the specific error message but I can checkout your code and see.

[JBKahn]

Are the items and set on the same database? Otherwise it's not meant to be an allowed relationship with an actual foreign key

[rafis]

Ok, I think I got it. I need to change all relationships fields with models.BigIntegerField() and thus to work with cross-shard relationships manually?

But why wouldn't you extend Django relationships classes to allow those cross-shard links natively?

[rafis]

Why do I need to define so much complicated get_shard() and get_shard_from_id() functions in my model? Aren't they should be provided by some class or by some decorator from your library? Can you please take a look and maybe advice how to simplify them:

    def get_shard(self):
        if self.pk is not None:
            shard_group = getattr(self, 'django_sharding__shard_group')
            django_sharding_app = apps.get_app_config('django_sharding')
            bucketer = django_sharding_app.get_bucketer(shard_group)

            return bucketer.get_shard(self)
        else:
            sharded_by_field = getattr(self, 'django_sharding__sharded_by_field')
            sharded_by_field = self._meta.get_field(sharded_by_field)
            self.pk = sharded_by_field.strategy.get_next_id()
            
            return self.get_shard()

    @staticmethod
    def get_shard_from_id(pk):
        if pk is not None:
            shard_group = getattr(Set, 'django_sharding__shard_group')
            django_sharding_app = apps.get_app_config('django_sharding')
            bucketer = django_sharding_app.get_bucketer(shard_group)
    
            obj = Set()
            obj.pk = pk

            return bucketer.get_shard(obj)
        else:
            sharded_by_field = getattr(Set, 'django_sharding__sharded_by_field')
            
            sharded_by_field = Set._meta.get_field(sharded_by_field)
            pk = sharded_by_field.strategy.get_next_id()
            
            return Set.get_shard_from_id(pk)

[JBKahn]

Django doesn't have good multi-db support as it is and I didn't really like the idea of hiding how the DB works from the code. It makes it harder to tweak DB usage, the more magic that happens. But yes, you'd use big integer fields instead. I'll take a look at this code you posted and your example. Is that code block also in your code?

[JBKahn]

For the get_shard_from_id, you can inherit from this field or make your own to inherit from this

and get_shard isn't supposed to be that complicated. If you use that class than you do not need to do pk generation in that method. The package is a set of tools, rather than a framework because how you choose to shard highly impacts the code. If you explain how you're using sharding I'm happy to provide help for best using the package to achieve that.

[mihalt]

@JBKahn as I understand this topic is highly related to this #15

Would be really nice to have cross-database relations working for the box. I have a lot of FK, M2m and Generic fields between near all of my general big models.

The lib is with good potential, but I really don't have idea how to work with it without this feature 😅
Are any better workarounds then implementing appropriate relations via tds_fdw on the postgres side for example?

[JBKahn]

I haven't touched this library in a long time nor Django. I'm not sure if Django has improved it's support or if there are better options than this library.

[mihalt]

I haven't touched this library in a long time nor Django. I'm not sure if Django has improved it's support or if there are better options than this library.

There are a few of distributed databases, that django recommend itself https://docs.djangoproject.com/en/5.2/ref/databases/#using-a-3rd-party-database-backend

They can do JOINS on low level.
But the best option that I found is Citus Postgres plugin on today. I even shouldn't change default 'ENGINE': 'django.db.backends.postgresql'
Can you maybe recommend anything better, by the way?

I like your highlevel opportunity of obvious telling to which shard what data to put. Is the future of the django-sharding to work with low level distributed opportunities by default but that just give highlevel instruments to rule them? What do you think?