Cannot create gloss for fresh install

[Woseseltops]

This is a fresh install of Signbank https://signbank4.vlaamsegebarentaal.be/

If you go to the admin, you can log in with these temporary test credentials:

user: divya
pwd: divya!23

When you try to create a gloss, you get an error, though... any idea what's going on there @susanodd ?

PS
For completeness, the VGT admin team agreed with sharing these credentials here, as this is only an empty test installation.

[susanodd]

I'll look at it.
I know when signbank was new there needs to be some already existing data in some tables.
I will look at my notes from the install up signbank-susan.
Is there an "empty" signbank database anywhere? There are no database scripts in ansible.
Also the necessary folders are not all present in the ansible.

It won't work to start with the "original" signbank db since the migrations have points where they can't migrate from the previous. (That has to do with the populate methods that relied on old fields, then mapped the data to new fields, then removed the old fields. Namely the "name_english" etc fields. So the migrations can't be applied.)

I will check it out.
That's off the top of my head.

In case there is something off about signbank.db (initial database)

It could be possible to create an initial db by copying what create test db does.

The test database removes content from tables. The tables that remain are necessary to have data in them.
This could be a problem with the field choices. At the very least, the empty and N/A choices should exist. Same for Handshapes. (The test database creation does not delete handshapes.)
(If there is nothing in some of the field choices, then it's not going to work.)

We still have the container signbank-test that has not been filled in with anything.
So it's possible to work on this there, from scratch with some kind of (trial and error) "initial database" that we (via testing) determine to be sufficient for it to run.
I think Django creates the anonymous user. To use the admin initially, the user needs to be at least staff.
We do some of this in the virtual environment.

[susanodd]

1. It gives a message that there are no pages, login to define some.
   This should not happen. All of the pages have templates, except the About pages and Copyright do not, and are visible for anonymous users.
2. In the admin, no pages show up. The urls for all pages with and without templates should be here.
3. in the admin, there are no groups (some places check group permissions)
4. in the admin, the field choices table is empty. (as above, this should contain '-' and 'N/A' for each of the predefined field choices.)
5. in the admin, sites has "example.com"

i think a new initial database might be needed.
Or an sql script to add the tables that need populating

[susanodd]

I changed the site name in the "sites" in the admin.
But still when I then type the page after the site name in the browser, to my surprise I get a message to contact Onno (old email).

[susanodd]

In the admin, I added a page /about/copyright/ with some text

But there are no menus.
To see it, you need the url

https://signbank4.vlaamsegebarentaal.be/about/copyright/

[susanodd]

Maybe make a create or insert sql with the required contents for the tables?

[susanodd]

I added the necessary permissions to the user. Since that didn't work, I created a group Editor and added permissions there, too.

It works to add feedback (by typing in the page in the url by hand, since there are no menus)

https://signbank4.vlaamsegebarentaal.be/feedback/overview/

It looks like in our Admin python code we did not implement methods for adding glosses or lemmas. (There are no +add in the column next to the table.)
To create a gloss, you also need to have a lemma, but it's not shown to add those either.

Normally, because creating a gloss involves lots of steps, also some fields need to be filled in before it can be saved.

[susanodd]

The settings file is not properly defined.

I typed in this url (by hand since there are no pages):

https://signbank4.vlaamsegebarentaal.be/signs/search/

It complains here:

language = Language.objects.get(**DEFAULT_KEYWORDS_LANGUAGE)

this setting does not exist.

Here,

DEFAULT_DATASET	
'Your Dataset'
DEFAULT_DATASET_ACRONYM	
'YDS'
DEFAULT_DATASET_LANGUAGE_ID	
1
DEFAULT_DATASET_PK	
1
DEFAULT_KEYWORDS_LANGUAGE	
'********************'

[susanodd]

For adding a new sign this way:

https://signbank4.vlaamsegebarentaal.be/signs/add/

you get this error

default_dataset = Dataset.objects.get(acronym=default_dataset_acronym)

default_dataset_acronym	
'YDS'

For creation of a new signbank, I was able to modify the site "example.com" in the admin.
But the other settings are all in the settings files.
We could add settings to the admin to make this easier.
I thought about doing this for dark mode, since I added css colours to the code as variables.
But those can just as easily be fetched by Django from the database.

[susanodd]

https://signbank4.vlaamsegebarentaal.be/dictionary/test_abstract_machine/1/

url to create a gloss (this is an API testing url) with the dataset id 1 (VGT)

[susanodd]

https://signbank4.vlaamsegebarentaal.be/datasets/select/

I created a group Dataset_Manager to view this page:

https://signbank4.vlaamsegebarentaal.be/datasets/VGT

(That should not require manager group but maybe because other things are missing control flow showed it was necessary. Oh wait, the group needs to exist, but the user does not need to be in it. Let me fix that.)
There is 1 gloss in the dataset now.

[susanodd]

@Woseseltops where did the initial database come from?
Do you want me to work on this?
The create test db script can probably be revised to do this.

The specific interface languages need to be specified in various places.
Do you want the initial (empty signbank) to start out with only one language?
Some initial translations would be needed because the default Django po files seem to be for English.
Actually English needs to be there because all the "trans" code in the templates is in English.

The database model translation fields (i.e., name_en, name_nl, etc) are created by Django based on the settings.
So an initial database (which needs to have Field Choice and Pages tables filled in) needs to have the correct language fields.
(So for example, name_ar and name_he can be removed.)

[ROMAN5]

Hello @susanodd

I was the one that set up https://signbank4.vlaamsegebarentaal.be/

The goal was to move away from an old (~2018) fork of signbank by starting "a new site" from scratch, a clean database and the head of the master branch.
It was a challenge to get an initial database that would get the site to start up. Explained below is the only & quickest way I found to go from cloning to homepage. (and is most likely a source of some of the issues) I pulled together info from the wiki here and various forks, but still not sure what is the correct way to have a properly configured "empty" initial database.

--

Using WSL Ubuntu 22.04
Requires Python 3.9

Run following commands:
git clone https://github.com/Signbank/Global-signbank.git
cd Global-signbank

Remove 'typing-extensions==3.10.0.0' from requirements.txt

Run following command:
pip install -r requirements.txt

Copy signbank/settings/server_specific/default.py to signbank/settings/server_specific/server_specific.py
Edit the following in signbank/settings/server_specific/server_specific.py:
ROOT = ''
BASE_DIR = ROOT
SWITCH_TO_MYSQL = 0
URL = ''
ALLOWED_HOSTS = ['127.0.0.1']
WRITABLE_FOLDER = ROOT+'writable/'
SECRET_KEY = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
SIGNBANK_VERSION_CODE = 'global'

Create writable/database/signbank.db (Empty 0kb)

Run following commands:
find . -path "migrations" -not -regex ".*init.py" -a -not -regex ".*migrations" | xargs rm -rf
python3 bin/develop.py makemigrations
python3 bin/develop.py migrate
python3 bin/develop.py createsuperuser
python3 bin/develop.py runserver

[susanodd]

We're up to Python 3.12 now.
It had to be installed on Ubuntu using Deadsnakes
You'll need a much newer signbank database as the starting point.
We revised the Field Choice and Handshape models to use objects instead of "character" "machine_value" data.
We also added model translations and got rid of the original "name_english" fields, etc.
(Django does the translations now.)

For us, it wasn't possible to migrate from the "pre-field-choice" model and the "pre-model-translation" because a "populate" migration was needed to copy the old values to the new fields, before removing the old fields. Since fields were removed, the migrations can't migrate from original databases anymore.

If possible you would need to export your original database to an SQL file that could then be used to fill a new database. (With some extra programming to do that.)

I browsed at your installation and it looked like you didn't want to show English as a language?
Internally, English needs to be there in order for the translations to work. The templates are in English, with "trans" text in English. Then these are what get translated to your "be" "po" files.

(If this is too jargon-sounding, @Woseseltops can also add some comments.)

We have a "create test db" script that runs as a command in the virtual environment.
It empties most tables except for those necessary for it to run. For example, the Pages table needs to be there in order to show the menus. It looks like your database (as viewed in Admin) was missing these.
I know there are too many languages because we have 5 interface languages.

I'm not sure what order in which to do things.
I assume that if you can export your original database, then devise a new empty database (along the lines of what is included in the "test db", then you should be able to populate it.

To make a "starting non-empty" database into which you could import your own glosses, etc. we would need to remove more things from the test database. (For example, all the languages except English and possible Dutch, plus the users. The groups are needed to run signbank, but not the user permissions.)

If you can get your virtual environment working, you could try to use the create_test_db command on it.
That creates a database without glosses.
More or less.

You also need to keep English.
And change the "model translations". (Django adds and removes the internal "name_en" "name_nl" fields itself. The default is English.)

This is rambling a bit. See if you can somehow create a test database from your original database.
signbank/dictionary/management/commands/create_test_db.py

This gives an idea of what tables are needed in order to run.

The Field Choices are needed but your old Field Choices table won't work. (So even if you can create an initial database from your old one, it still needs some more changes. Django can generate migrations. so if you can get it running with an old database, then Django can generate migrations.)

We can also try to create an initial database. (@Woseseltops has done this before, but I think it was missing some things.)

Most import, you need to export your original glosses data in order to be able to import it.

Can you identify with show_migrations on your original database where exactly it started from?

Internally there should be some Django tables that include the names of the migrations.
Then we can see what data you have and how to map it to the new tables.
This needs an additional migration because of the FieldChoice and Handshape models, and model translations.

The above is needed to make sure whatever you start with (without actual glosses), will run.

As I write this, I'm guessing that it won't run yet.

[susanodd]

@ROMAN5 I understood it that you had an old signbank from 2018. Is that true? Do you already have data that you want to put in a new signbank?

@Woseseltops did you create an empty starting database ever? I know we can make one from the create_test_db but more things need to be removed from it in order for it to be minimal.
Also the model translation columns.
Since you recently removed the sessions data, perhaps you have gained insight in this?

[ROMAN5]

@ROMAN5 I understood it that you had an old signbank from 2018. Is that true? Do you already have data that you want to put in a new signbank?

No. As I have mentioned that we wanted "a new site" from scratch, a clean database and the head of the master branch.

[susanodd]

@ROMAN5 I understood it that you had an old signbank from 2018. Is that true? Do you already have data that you want to put in a new signbank?

No. As I have mentioned that we wanted "a new site" from scratch, a clean database and the head of the master branch.

Okay. I was distracted by the 2018.
I will see if I can create an empty database.

We have ansible scripts for creating a signbank.
@Woseseltops recently added a feature to create an Amazon site for a pull request branch for code review.
He removed sessions data from the database since the database needs to be copied to Amazon for the temporary signbank.
(It has an expiration date.)

I will work on making a trimmed down empty database that does not have the interface translations.
Then you will be able to add your own interface language during setup.
There are also quite a few folders that need to be created on the server. Those are in the ansible script.

Ideally, we would make something that can make a fresh install of an empty signbank.

[susanodd]

@ROMAN5 are any of you currently users of signbank?

I needed to leave the User table existing in order to login myself in order to delete things by hand.

There is a bug in Guardian and most users it does not work to delete them in the Admin.
So I have adapted one of the virtual environment commands in order to delete users.
I can delete all users except for myself, wessel, AnonmousUser, and DivyaKanekal ?

Are there any other users to retain?
(This is in an empty signbank. There needs to be users or signbank won't work.)

I've deleted all Datasets except for VGT, tstMH, NGT.

(But all glosses are deleted in these datasets.)

[susanodd]

This isn't working at all.

For some reason Guardian isn't working correctly.

The shorthand Django view:

{% get_obj_perms request.user for dataset as "dataset_perms" %}

the variable dataset_perms assigned to above, has the form:

['add_permission', 'change_permission', 'delete_permission', 'view_permission']

instead of what is being tested in the template:

{% if "view_dataset" in dataset_perms or "can_view_dataset" in dataset_perms %}

So 'view_permission' does not match "view_dataset"

[This is with a simplified "empty" database, where all the content has been removed except for the "non-data" Django management parts]
I'll need to discuss this with @Woseseltops to figure out what is missing.

This example is in the "Select Datasets" template.

I changed the code to:

{% if "view_permission" in dataset_perms %}

and it works.

It looks like maybe a mismatch of the guardian version or something.

[susanodd]

I modified all the permission tests to match what worked.

It seems to be running okay now with an empty database.

I made a command to empty the tables.
And a command to remove users except for the development users, and Divya.

I need to "automate" the other adjustments I made to the database. I removed the datasets except for NGT, VGT, and tstMH.
(This can be automated. There are no glosses. The Handshapes and Field Choices remain.)

[susanodd]

@ROMAN5 I have the empty database running on signbank-test.
It still needs some work. I removed all the users except our development team and Divya.
The extra interface languages need to be migrated out. (So the empty database itself is not yet ready for you.) I moved the discussion to the pull request.

[ROMAN5]

@ROMAN5 I have the empty database running on signbank-test. It still needs some work. I removed all the users except our development team and Divya. The extra interface languages need to be migrated out. (So the empty database itself is not yet ready for you.) I moved the discussion to the pull request.

@susanodd OK, Thank you for letting me know. I will wait for empty database when it is ready for me.

[susanodd]

@ROMAN5 do you want the glosses of VGT to stay? There are 16928 glosses.
This would have an advantage for cooperation, since the gloss ids would be retained.

[ROMAN5]

@ROMAN5 do you want the glosses of VGT to stay? There are 16928 glosses. This would have an advantage for cooperation, since the gloss ids would be retained.

@susanodd Where did you find 16928 glosses? Normally we need to start new db from scratch.

[susanodd]

Here, the dataset VGT

https://signbank.cls.ru.nl/datasets/VGT

You need to be logged in to see it.
@ocrasborn put the glosses there years ago during collaboration.
It's only the glosses. He may have videos somewhere.

[ROMAN5]

@susanodd Three years ago is too old. As I have mentioned that we need a new db that working with latest version of Global Signbank (problem was "cannot create gloss for fresh install"). We are installing on a new website. New database and the latest version of Global Signbank -- ready? Thanks.

[susanodd]

@susanodd Three years ago is too old. As I have mentioned that we need a new db that working with latest version of Global Signbank (problem was "cannot create gloss for fresh install"). We are installing on a new website. New database and the latest version of Global Signbank -- ready? Thanks.

The glosses in VGT are just data. The question is whether you want the VGT table to be erased in the new empty database.
With or without the VGT content it will still be a database that is for the newest master branch.

The empty database is being created from the most recent signbank database. All of the models will be up to date.
Over the years all the data has been migrated according to any changes in the models. All of the data in signbank is according to the models in master.

The VGT data consists of all the Annotations and Lemmas. These are according to the newest models.

I ask because empty is empty.
You will need to create glosses somehow.

[ROMAN5]

The glosses in VGT are just data. The question is whether you want the VGT table to be erased in the new empty database. With or without the VGT content it will still be a database that is for the newest master branch.

I ask because empty is empty.
You will need to create glosses somehow.

I don't have a strong preference as the plan is to start from zero. We just need way to start a new website that can be configured anew, and that we have glosses and data added manually from frontend.

[susanodd]

The glosses in VGT are just data. The question is whether you want the VGT table to be erased in the new empty database. With or without the VGT content it will still be a database that is for the newest master branch.

I ask because empty is empty.
You will need to create glosses somehow.

I don't have a strong preference as the plan is to start from zero. We just need way to start a new website that can be configured anew, and that we have glosses and data added manually from frontend.

Since there needs to be one dataset in order to work, and there needs to be at least one user that can use the admin, then there needs to already exist some things or it won't work.

I can make the dataset be VGT and just leave the glosses that are already there. It also needs to be made public otherwise the site won't work properly. I need to ask @Woseseltops what to do about the users. You will need to have an account that can use the Admin on your site. Or we can do the same divya account.

Signbank has NGT as public.
(I didn't actually know that it wouldn't work without glosses. The tests are automated testing and those create glosses on an empty database. But they are single tests, not on the entire interface.)

To maintain the structure of signbank, the pages need to be there. Nearly all are templates, except for the About menu. Those are in the admin.
You can turn off pages in the admin and change group permissions on them to hide them.

I am assuming you want to keep being able to pull from master to get new features and bug fixed.
We need to do something about the interface languages.
What languages do you plan to use? I assume you want to keep EN and NL (or change it to be BE). Do you want to add BE and FR? You can start out copying the NL "po" file to start and then just modify them as needed. There are thousands of translations, so I don't advise to start empty. English needs to be there because the templates are written in English.

I suggest that we add extra migration files for BE and FR (make them conditional) and also generate the "po" files.
Then you just have the settings turned on for the languages you need.
This would allow you to keep being able to pull from master.

I need to figure out how to do the model translations to have them conditional.

[susanodd]

The glosses in VGT are just data. The question is whether you want the VGT table to be erased in the new empty database. With or without the VGT content it will still be a database that is for the newest master branch.

I ask because empty is empty.
You will need to create glosses somehow.

I don't have a strong preference as the plan is to start from zero. We just need way to start a new website that can be configured anew, and that we have glosses and data added manually from frontend.

It is proving very difficult to leave the dataset in the database.
This is because when the glosses and datasets are deleted in a "command" in the virtual environment, Django still envokes all the signals on deletion (normally it stores data in a table about things you have deleted.)
So this is causing problems.
So likely there will be no glosses.
The glosses need to be removed before the datasets can be removed.
But to do this in a script is not working.
If all the glosses (of all the datasets) are deleted, then the datasets can be removed.
Again, there needs to be at least one dataset.
And there should be at least one gloss in the dataset.

@Woseseltops any help would be appreciated.

[susanodd]

@ROMAN5 there are some problems now with permissions.
That is being discussed in the pull request.
For some reason, when emptying the glosses, the permissions on the (now empty) datasets have disappeared. It is not possible to create glosses.
I had changed all the permissions in the system to make the code work with an empty database.
But that ended up incompatible with the recent permission upgrades to master, so I undid that work.