[BUG] `gsub!': invalid byte sequence in UTF-8 (ArgumentError)

[boldyrev-insegment]

Some emails trigger the error in the subject in lib/sisimai/mime.rb:343

[boldyrev-insegment]

Can be fixed by adding:

if ! getdecoded.valid_encoding?
  getdecoded = getdecoded.encode("UTF-16be", :invalid=>:replace, :replace=>"?").encode('UTF-8')
end

before:

getdecoded.gsub!(/\r\n/, "\n")  # Convert CRLF to LF

[azumakuniyuki]

@boldyrev-insegment Thanks for the bug report. Does the following code fix the bug ?

getdecoded.scrub!('?')
getdecoded.gsub!(/\r\n/, "\n")  # Convert CRLF to LF

Best regards

[boldyrev-insegment]

@azumakuniyuki yes, it did fix the problem.

Sorry for such a late reply but I've tested the code on a bunch of different emails and found a few more bugs, so I had to tweak the code a little more :

-- 1 --

lib/sisimai/mime.rb:443:in `gsub!': invalid byte sequence in UTF-8 (ArgumentError)

add:
bodystring.scrub!('?')
before:
bodystring.gsub!(%r{^(Content-Type:\s*message/(?:rfc822|delivery-status)).+$}, '\1')

-- 2 --

lib/sisimai/mime.rb:435:in `makeflat': incompatible character encodings: UTF-8 and ASCII-8BIT (Encoding::CompatibilityError)

replace
return hasflatten
with
return hasflatten.to_s.force_encoding("UTF-8")

Hope that helps!

[azumakuniyuki]

@boldyrev-insegment Sorry for late response. Would you show me the email that causes the error (including entire message and email headers) at Private-Gist ? I will try to debug and fix the problem.

Best regards,

[boldyrev-insegment]

I'm sorry but I can't share it as far as the data is very sensitive.