Auto-generating Data with Django-Migration and Lookup_Field
As I was making a whole new API and its model table to be used, here came the necessity to make A LOT of data for every single already-existing user accounts. The first thing that came up to my mind was using python manage.py shell to enter some simple scripts with ORMs. However, I felt like I wanted to do something that was more challenging and fun. Plus, this following approach won’t require dev ops or sys admin to manually write ORM script on the dev/production server. They just have to do exactly what they have been doing like any other day in the office.
Overriding `save()` Method to Generate Data in the Future
First, let’s take a look at the case where I made a new table.
class Report(models.Model):
ALL_TABLE_FIELDS = (
'local_ordered_date', 'card', 'country', 'customer', 'supplier',
'amount', 'currency', 'quantity', 'quantity_unit', ... )
DEFAULT_TABLE_FIELDS = {
'OrderTrendReport': ('card', 'board', 'customer', 'amount',
'currency', 'stage', 'owner', 'country',
'local_ordered_date', 'local_delivery_deadline',
'order_number'),
... }
class ReportType(models.TextChoices):
# hard-coded in order to avoid ImportError and AppRegistryNotReady
OrderTrendReport = 'OrderTrendReport', 'Order Trend report'
RevenueTrendReport = 'RevenueTrendReport', 'Revenue Trend report'
DealSuccessRateReport = 'DealSuccessRateReport', 'Deal Success Rate report'
DealLostReasonReport = 'DealLostReasonReport', 'Deal Lost Reason report'
OrderByCategoryReport = 'OrderByCategoryReport', 'Order By Category report'
user = models.ForeignKey(User, on_delete=models.CASCADE)
report_type = models.CharField(choices=ReportType.choices,
max_length=50,
default=ReportType.OrderTrendReport)
table_fields = ArrayField(models.CharField(max_length=50),
default=list)
@property
def all_table_fields(self):
return self.ALL_TABLE_FIELDS
(I had to shorten some parts for brevity)
As you can see, there is the Report table, within which ReportType, a TextChoices model class, resides in order to support Report’s report_type field. The main idea is that a Report instance should have a report_typefield as a text choice, and a corresponding default table_fields field. Let’s say, there is an instance that has a report_type field as ‘OrderTrendReport’. Let’s take a closer look at table_fields:
table_fields = ArrayField(models.CharField(max_length=50),
default=list)You can see that the default is already set for list. You can change this to some callable if you have defined such BEFORE you define the value of default. Surely, I tried that like below:
def get_default_table_fields(obj):
return obj.DEFAULT_TABLE_FIELDS[obj.report_type]However, this will not work because while Django’s default attribute does take a callable as a default, but it DOES NOT take a callable WITH A PARAMETER.
The only way to overcome this predicament would be to override the save() method of the models.Model. Like below:
def save(self, *args, **kwargs):
if not self.pk: # this will ensure that the object is new
self.table_fields = self.DEFAULT_TABLE_FIELDS[self.report_type]
super().save(*args, **kwargs)Awesome! So what now?
Editing Migration File to Automate Data Generation
It’s time to save lots of instances for every single case of report_type for user accounts. Right now, there are about 5 report_types. So if there are 200 users, there should be 1000 instances of Report. Instead of manually doing this on a shell, you can write additional code to execute while migrating.
import django.contrib.postgres.fields
from django.db import migrations, models
import django.db.models.deletion
def make_report_fields(apps, schema_editor):
User = apps.get_model(settings.AUTH_USER_MODEL)
Report = apps.get_model('deal', 'Report')
ReportTypes = ('OrderTrendReport', 'RevenueTrendReport', 'DealSuccessRateReport',
'DealLostReasonReport', 'OrderByCategoryReport')
report_fields = [Report(user=user, report_type=report_type)
for report_type in ReportTypes
for user in User.objects.all()]
Report.objects.bulk_create(report_fields)
class Migration(migrations.Migration):
dependencies = [
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
('deal', '0059_auto_20210610_0948'),
]
operations = [
migrations.CreateModel(
name='Report',
fields=[
('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('report_type', models.CharField(choices=[('OrderTrendReport', 'Order Trend report'), ('RevenueTrendReport', 'Revenue Trend report'), ('DealSuccessRateReport', 'Deal Success Rate report'), ('DealLostReasonReport', 'Deal Lost Reason report'), ('OrderByCategoryReport', 'Order By Category report')], default='OrderTrendReport', max_length=50)),
('table_fields', django.contrib.postgres.fields.ArrayField(base_field=models.CharField(max_length=50), default=list, size=None)),
('user', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)),
],
),
migrations.RunPython(make_report_fields)
]However, the above DID NOT WORK!
What happened?
When you take a look at make_report_fields, you can see that I used bulk_create() in order to make a bunch of data at once. Someone on StackOverflow kindly explained that bulk_create() does not use the save() method of the model but instead works on a SQL level. So I tried putting save() method, running a nested for-loop like so:
for user in User.objects.all():
for report_type in ReportTypes:
report = Report(user=user, report_type=report_type)
report.save()
But alas, this still wouldn’t work!
The reason is that the serialized model in the migrations does not have my custom save() method.
So came the last solution: writing the whole dictionary in the migration file.
DEFAULT_TABLE_FIELDS = {
'OrderTrendReport': ('card', 'board', 'customer', 'amount',
'currency', 'stage', 'owner', 'country',
'local_ordered_date', 'local_delivery_deadline',
'order_number'),
'RevenueTrendReport': ('card', 'customer', 'invoice_total',
'payment_total', 'balance', 'currency',
'owner', 'order_number', 'board'),
'DealSuccessRateReport': ('card', 'board', 'customer', 'amount',
'currency', 'owner', 'result',
'local_result_updated_at'),
'DealLostReasonReport': ('card', 'board', 'customer', 'amount',
'currency', 'owner',
'local_result_updated_at', 'lost_reason',
'lost_description'),
'OrderByCategoryReport': ('card', 'board', 'customer', 'amount',
'currency', 'stage', 'owner', 'country',
'local_ordered_date', 'local_delivery_deadline',
'order_number')
}
def make_report_fields(apps, schema_editor):
User = apps.get_model(settings.AUTH_USER_MODEL)
Report = apps.get_model('deal', 'Report')
ReportTypes = ('OrderTrendReport', 'RevenueTrendReport', 'DealSuccessRateReport',
'DealLostReasonReport', 'OrderByCategoryReport')
for user in User.objects.all():
for report_type in ReportTypes:
report = Report(user=user, report_type=report_type,
table_fields=DEFAULT_TABLE_FIELDS[report_type])
report.save()This guaranteed that each Report instance has correct table_fields recorded.
