Bugfix/localization 926

[mandrade2]

In this PR issue #253 is resolved. Whenever a submission is being saved, a stored procedure checks that with a SELECT ... FOR UPDATE row lock that the submission to be saved is not there yet. This prevents duplicate submission being saved. Making rows unique setting a multi-column index or unique_together in Django would've been preferred, but the older duplicate submissions needed to be there for consistency. Besides, Django does not support QuerySet.select_for_update() For MySQL databases.

Travis jobs where configured to run with Mysql, so the migration and testing job succeed.

All database defaults where set to have a correct character set and collation. Since by default MySQL does not store 4 byte-characters for the utf8 character set. It's explained here https://stackoverflow.com/a/20349552/8763522

[iafan]

Do we need to override CHARSET and COLLATION now that they are in 90-local.conf.template file? (90-local.conf file will be loaded before 91-travis.conf)

[mandrade2]

But is Travis loading 90-local.conf.template on the build environment? It seems to me that It should not run it since it has the .template extension

[mandrade2]

Any input on this Igor? As I stated above It's not clear to me that travis will use the configuration in 90-local.conf.template

[iafan]

Same question as above.

[iafan]

Can you please explain this change? I think with this PR we're moving away from sqlite.

[mandrade2]

It's to maintain sqlite support for user's that are on it. Thought I should keep it for compatibility

[iafan]

Here and below, I'm concerned about these snapshot changes to the source string. They don't seem to be related to the changes this PR is supposed to introduce. We need to understand why this property is reported differently now.

[mandrade2]

Don't know why it happens. Tests wouldn't pass if I kept the old snapshots. Might be because of my environment, but before the changes I introduced the tests would run smoothly.

[mandrade2]

The thing is that when I run the tests without generating a new dump (not snapshots) I get this error on all tests when setting up the fixtures.

Then, when I generate a new dump I can get past the fixtures setup but other errors show up.

When I called the submission procedure outside the save method, generating a new dump and snapshots would fix all tests. However, there's that weird modification of source strings.

So the problem here would be that the original data_dump.json is having troubles when initialized on MySQL. Apparently because of uniqueness constraints. On the other hand when I create a new dump some of this problems dissapear.

[mandrade2]

@iafan It turns out I could not get over the uniqueness constraints colliding from the dump without generating a new dump, so I uploaded a new one. It contains some changes on ids, timestamps and ordering.

The new dump would not pass all tests without regenerating snapshots, so I did regenerate them. The new snapshots have some weird source changes as you pointed out. I've looked back at commit history and found a commit that had a similar impact on snapshots. The last modification of urls to be tested occurs here, so I discard the source modifications are due to different urls being tested and just having out of date snapshots.

[benjamimo1]

Hello @mandrade2 I have a couple of questions:
Just for future reference, why are there some many files being updated?
Have you updated the SQL to look better? (Formatted)
Also please mark as resolved all comments made by other reviewers if they were addressed.

[mandrade2]

I fixed the remaining issues regarding tests. It turns out that I had to modify the submission save method to call for a reload from the database, to avoid Django raising an error because of garbage collecting an unsaved instance. Also added to the stored procedure arguments for a suggestion_id and quality_check_id, I had forgotten them.

I rolled back all the snapshot changes and fixed two tests that kept failing. The _test_user_purging test was failing because when asserting that two datetimes where equal, the dates from MySQL were coming back from the db without microseconds. My solution was to strip microseconds of the object that we test against. The other test was test_timeline_view_unit_with_creation. It failed because the submission unit was saved twice, which in turn called twice a method from the Unit save method called add_initial_submission. Plus, there was another call to add_initial_submission on the test itself. I removed one call to the unit save method and also the extra add_initial_submission call. I'm not sure if this changes too much the tested case on the test. You can see on both tests that it was expected for them to behave different when being tested on MySQL.

done