dshield, dshield_top_1000: converted file is empty, etc

[Steve-Newcomb]

On Debian stable (Bullseye) I'm getting frequent log entries like this one:

Mar 4 09:45:02 tigger update-ipsets.sh[616456]: WARNING: firehol_level1: will be generated without 'dshield' - enable 'dshield' it to be included the next time

(ed note: The entry would be more coherent if the word "for" were inserted before "it to be included the next time")
(ed note: On Debian, anyway, there is no update-ipsets.sh, but there is update-ipsets. Is that the case everywhere, or just on Debian?)

So, as root I run update-ipsets, and I see lots of output that includes these lines (fail2ban is an ipset I automagically maintain from various logs, using a large set of regular expressions, and it has nothing to do with the issue under discussion, I think):

Found these ipsets active: fail2ban firehol_level1
(...)
                        dshield| 56 fails so far, time increased from 10 to 460 mins
                                   |  LATER  419/460 mins passed, will fetch in 41 mins
                                   | download is the same, but we need to re-process it
                                   | converting with 'dshield_parser'
                                   |  ERROR  converted file is empty.
                                   |
                   dshield_top_1000|  LATER  50/61 mins passed, will fetch in 11 mins
                                   | forced reprocessing (ignoring download status)
                                   | converting with 'parse_dshield_api'
                                   |  ERROR  converted file is empty.
(...)
                     firehol_level1| examining source ipsets:
                                  |  WARNING  will be generated without 'dshield' - enable 'dshield' it to be included the next time

So, I follow the log entry's suggestion and see that:

230304 09:36 ~ root@tigger{2}# update-ipsets enable dshield
 WARNING : dshield: is already enabled

...and the system log continues to get entries like the one at the top of this post.

So what should I do? Should I simply ignore those log entries?

[ixs]

This issue would be fixed in #488

[brycied00d]

Unfortunately, #488 only corrects the URL for the dshield feed.

With dshield_top_1000 (https://isc.sans.edu/api/sources/attacks/1000/), upstream "deprecated" the current URL and left a helpful note behind.

<?xml version="1.0" encoding="UTF-8"?>
<!-- Because people do not read and keep using this as a blocklist, we discontinued 
            this specific feed. Use 1001 or 999 and you will still get the same list, but DO NOT USE IT AS A 
            BLOCKLIST. PLEASE!!! -->
<sources>

</sources>

To make dshield_top_1000 work again, change the 1000 in the URL to either 999 or 1001.

[FabioPedretti]

This should be solved with #458.