Error on importing 100k products

[jtcosta]

Hi

My solution involves having products data coming from a queue system.
In that I sent 100k products gathered in packages of 100/500 depending on configurations.
Having that said after importing I would say something like 4000 products i start having - "General system exception happened".
The strange part is - I catched those 100 products and rerun the process and booooom - everything works just fine. - So i have no idea on how to reproduce this problem.

Specs:
Localy no indexers or crons are in place.

What I've tried:

ran those 100 products 7 times - 700 products - no errors
ran 100 products with repeated products - no errors.

Hints:

I've turned general log and realised that magento rollsback sometimes the save of categories - urls or products may lead to this.
Magento 2 has some triggers which could be messing with this process.

Can somebody help me?
TY

[EliasKotlyar]

Hello jtcosta,

I assume that this error occurs not on the Magento-side, but on the database. "General Error" is mostly just passed from the database to Magento. Your database could be not able to handle that amount of data in a small period of time. If you are unable to scale on the Database Size, you should try to make some "sleep" periods before importing a new bunch of data.

Greetings
Elias

[joaotnlima]

Hello Elias,

After some digging I got to this function ->

public function processImport($dataArray) { if($this->_validateData($dataArray)) { $this->_importData(); } }

Which seems that if by any reason the validation of rows fails the Importer will exit and not import that bunch.
ValidateData will return $result = !$errorAggregator->getErrorsCount();

So what I'm trying to do is basically get the rows which are invalid and trying to skip/remove them.
My question is -> Isn't the module already doing this? If so where is that logic?