Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gnarly data still causes problems (DROID CSV) #92

Open
ross-spencer opened this issue Jun 5, 2022 · 0 comments
Open

Gnarly data still causes problems (DROID CSV) #92

ross-spencer opened this issue Jun 5, 2022 · 0 comments
Labels
bug that noise in the pipes was not in your imagination, there's a gremlin loose in the works

Comments

@ross-spencer
Copy link
Member

Some CSVs can still interrupt processing. We handle this now with an error log, but we may want to find a way of proactively identifying the issue when reading the CSV or escaping inserts when there is a problem:

"ID","PARENT_ID","URI","FILE_PATH","NAME","METHOD","STATUS","SIZE","TYPE","EXT","LAST_MODIFIED","EXTENSION_MISMATCH","SHA256_HASH","FORMAT_COUNT","PUID","MIME_TYPE","FORMAT_NAME","FORMAT_VERSION"
"3354","3314","file:/home/ross-spencer/git/test-data/src/sampledata/TestTransfers/badNames/objects/%22quote%22.txt","/home/ross-spencer/git/test-data/src/sampledata/TestTransfers/badNames/objects/""quote"".txt","""quote"".txt","Extension","Done","5","File","txt","2018-08-09T23:42:07","false","6667b2d1aab6a00caa5aee5af8ad9f1465e567abf1c209d15727d57b3e8f6e5f","1","x-fmt/111","text/plain","Plain Text File",""
"3396","3314","file:/home/ross-spencer/git/test-data/src/sampledata/TestTransfers/badNames/objects/%22quote%22/","/home/ross-spencer/git/test-data/src/sampledata/TestTransfers/badNames/objects/""quote""","""quote""","","Done","","Folder","","2018-08-09T23:42:07","false","","","","","",""
"3409","3396","file:/home/ross-spencer/git/test-data/src/sampledata/TestTransfers/badNames/objects/%22quote%22/%40at.txt","/home/ross-spencer/git/test-data/src/sampledata/TestTransfers/badNames/objects/""quote""/@at.txt","@at.txt","Extension","Done","5","File","txt","2018-08-09T23:42:07","false","6667b2d1aab6a00caa5aee5af8ad9f1465e567abf1c209d15727d57b3e8f6e5f","1","x-fmt/111","text/plain","Plain Text File",""
"3399","3396","file:/home/ross-spencer/git/test-data/src/sampledata/TestTransfers/badNames/objects/%22quote%22/control.txt","/home/ross-spencer/git/test-data/src/sampledata/TestTransfers/badNames/objects/""quote""/control.txt","control.txt","Extension","Done","5","File","txt","2018-08-09T23:42:07","false","6667b2d1aab6a00caa5aee5af8ad9f1465e567abf1c209d15727d57b3e8f6e5f","1","x-fmt/111","text/plain","Plain Text File",""
"3435","3314","file:/home/ross-spencer/git/test-data/src/sampledata/TestTransfers/badNames/objects/%22quote.txt","/home/ross-spencer/git/test-data/src/sampledata/TestTransfers/badNames/objects/""quote.txt","""quote.txt","Extension","Done","5","File","txt","2018-08-09T23:42:07","false","6667b2d1aab6a00caa5aee5af8ad9f1465e567abf1c209d15727d57b3e8f6e5f","1","x-fmt/111","text/plain","Plain Text File",""
"3408","3314","file:/home/ross-spencer/git/test-data/src/sampledata/TestTransfers/badNames/objects/%27%20quote.txt","/home/ross-spencer/git/test-data/src/sampledata/TestTransfers/badNames/objects/' quote.txt","' quote.txt","Extension","Done","5","File","txt","2018-08-09T23:42:07","false","6667b2d1aab6a00caa5aee5af8ad9f1465e567abf1c209d15727d57b3e8f6e5f","1","x-fmt/111","text/plain","Plain Text File",""
"3426","3314","file:/home/ross-spencer/git/test-data/src/sampledata/TestTransfers/badNames/objects/%27quote%27.txt","/home/ross-spencer/git/test-data/src/sampledata/TestTransfers/badNames/objects/'quote'.txt","'quote'.txt","Extension","Done","5","File","txt","2018-08-09T23:42:07","false","6667b2d1aab6a00caa5aee5af8ad9f1465e567abf1c209d15727d57b3e8f6e5f","1","x-fmt/111","text/plain","Plain Text File",""
"3382","3314","file:/home/ross-spencer/git/test-data/src/sampledata/TestTransfers/badNames/objects/%27quote%27/","/home/ross-spencer/git/test-data/src/sampledata/TestTransfers/badNames/objects/'quote'","'quote'","","Done","","Folder","","2018-08-09T23:42:07","false","","","","","",""
"3386","3382","file:/home/ross-spencer/git/test-data/src/sampledata/TestTransfers/badNames/objects/%27quote%27/%40at.txt","/home/ross-spencer/git/test-data/src/sampledata/TestTransfers/badNames/objects/'quote'/@at.txt","@at.txt","Extension","Done","5","File","txt","2018-08-09T23:42:07","false","6667b2d1aab6a00caa5aee5af8ad9f1465e567abf1c209d15727d57b3e8f6e5f","1","x-fmt/111","text/plain","Plain Text File",""
"3387","3382","file:/home/ross-spencer/git/test-data/src/sampledata/TestTransfers/badNames/objects/%27quote%27/control.txt","/home/ross-spencer/git/test-data/src/sampledata/TestTransfers/badNames/objects/'quote'/control.txt","control.txt","Extension","Done","5","File","txt","2018-08-09T23:42:07","false","6667b2d1aab6a00caa5aee5af8ad9f1465e567abf1c209d15727d57b3e8f6e5f","1","x-fmt/111","text/plain","Plain Text File",""
"3441","3314","file:/home/ross-spencer/git/test-data/src/sampledata/TestTransfers/badNames/objects/quote%20%22quote%22%20quote.txt","/home/ross-spencer/git/test-data/src/sampledata/TestTransfers/badNames/objects/quote ""quote"" quote.txt","quote ""quote"" quote.txt","Extension","Done","5","File","txt","2018-08-09T23:42:07","false","6667b2d1aab6a00caa5aee5af8ad9f1465e567abf1c209d15727d57b3e8f6e5f","1","x-fmt/111","text/plain","Plain Text File",""
"3113","3096","file:/home/ross-spencer/git/test-data/src/sampledata/TestTransfers/zippedWithBadNames/objects/%22quote%22.zip","/home/ross-spencer/git/test-data/src/sampledata/TestTransfers/zippedWithBadNames/objects/""quote"".zip","""quote"".zip","Signature","Done","344","File","zip","2018-08-09T23:42:07","false","101798294713909b757fae88896ea9ae2795645ac29ea3457a1632eeeb0ee8ea","1","x-fmt/263","application/zip","ZIP Format",""
@ross-spencer ross-spencer added the bug that noise in the pipes was not in your imagination, there's a gremlin loose in the works label Jun 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug that noise in the pipes was not in your imagination, there's a gremlin loose in the works
Projects
None yet
Development

No branches or pull requests

1 participant