- Code refactor
- Better identification of external URLs
- Fixed a major bug that made several intel URLs pass under the radar
- Fixed a major bug that caused non-html type content to be marked a crawlable URL
- added
--exclude
option - Better regex and code logic to favor performance
- Fixed a bug that caused dnsdumpster to fail if target was a subdomain
- Fixed a bug that caused a crash if run outside "Photon" directory
- Fixed a bug in file saving (specific to python3)
- Added
--timeout
option - Added
--output
option - Added
--user-agent
option - Replaced lxml with regex
- Better logic for favoring performance
- Added bigger and seperate file for user-agents
- Fixed lot of bugs
- Suppress SSL warnings in MAC
- x100 speed by code optimization
- Simplified code of
exporter
plugin
- Added
exporter
plugin - Added seamless update ability
- Fixed a bug in update function
- Fixed an issue which caused regular links to be saved in robots.txt
- Simplified
flash
function - Removed
-n
as an alias of--ninja
- Added
--only-urls
option - Refactored code for readability
- Skip saving files if the content is empty
- Introduced plugins
- Added
dnsdumpster
plugin - Fixed non-ascii character handling, again
- 404 pages are now added to
failed
list - Handling exceptions in
jscanner
- Proper handling of null response from
robots.txt
&sitemap.xml
- Python2 compatibility
- Proper handling of non-ascii chars
- Added ability to specify custom regex pattern
- Display total time taken and average time per request
- Disabled colors on windows and mac
- Cross platform file handling
- First stable release.