发布于 2016-11-21 15:52:49 | 199 次阅读 | 评论: 0 | 来源: 网友投递
Rspamd 反垃圾邮件系统
Rspamd是一个反垃圾邮件系统，因为使用事件模型和正则表达式优化，其设计工作速度比SpamAssassin还要快。目前推出的功能： regexp规则过滤的不同部分的信息;一些内置的功能分析的信息;模糊哈希支持; SURBL滤波器;电子邮件和性质表支持;控制界面进行远程管理和统计信息收集，一个Perl和卢阿插件系统;统计支持（定向结构刨花板/簸扬） ;兼容SpamAssassin ;和一个客户端程序的电子邮件扫描。类似的规则， rspamd约10倍SpamAssassin 。
Rspamd 1.4.0 发布，
该版本包含众多的关键问题修复以及新特性，详细列表如下：
[CritFix] Add guards for inactive redis connections
[CritFix] Another fix for proxying files using rspamd_proxy
[CritFix] Cleanup inactive redis connections
[CritFix] Do not sometimes try to exec posfilters before classification
[CritFix] Fix application of IPv6 mask
[CritFix] Fix chunked encoding when reading messages
[CritFix] Fix file mode for rspamd_proxy
[CritFix] Fix hyperscan compilation on regexp change
[CritFix] Fix issue with finding of end of lines pointers
[CritFix] Fix iteration over headers array (introduced in 1.4)
[CritFix] Fix processing of learned tokens count for redis backend
[CritFix] Fix race condition in checking of cached maps
[CritFix] Fix workers scripts by sharing workers configs
[CritFix] Introduce raw content to text parts
[CritFix] Plug memory leak and potential memory corruption
[Feature] Adaptive ratelimits
[Feature] Add ASN -> rbldnsd script for asn.rspamd.com
[Feature] Add DMARC_NA symbol
[Feature] Add F-Prot support to antivirus module
[Feature] Add HTTP backend to metadata exporter
[Feature] Add Lua API module for monitored objects
[Feature] Add R_DKIM_NA / R_SPF_NA / AUTH_NA symbols
[Feature] Add R_DKIM_PERMFAIL symbol
[Feature] Add R_SPF_PERMFAIL symbol
[Feature] Add Sophos antivirus support
[Feature] Add ZSTD compression to Lua API
[Feature] Add mid Lua module
[Feature] Add one_param flag for metric symbols
[Feature] Add a generic lua classifier
[Feature] Add a very basic interface to access workers data from on_load
[Feature] Add ability to delete a hash by its data to fuzzy_check plugin
[Feature] Add ability to enable/disable symbols via dynamic_conf
[Feature] Add ability to lookup settings by key
[Feature] Add common way to disable Lua modules
[Feature] Add compression support to rspamd client
[Feature] Add condition to do antiviral check
[Feature] Add configuration for lua classifiers
[Feature] Add configuration knobs for the errors circular buffer
[Feature] Add decompression support in rspamd client
[Feature] Add errors exporter to the controller
[Feature] Add expected value for monitored DNS resources
[Feature] Add exporter from error ringbuf to ucl
[Feature] Add extended version for fann creation function
[Feature] Add ffi friendly version of process_regexp function
[Feature] Add frequency and time display to webui
[Feature] Add fuzzy_delhash command to rspamc client
[Feature] Add implementation of redis connections pool
[Feature] Add latency and offline time monitoring
[Feature] Add learning support for lua classifiers
[Feature] Add max-size and timeout options to CGP helper
[Feature] Add method to enable/disable symbols in config
[Feature] Add methods to get metric's actions and symbols from Lua
[Feature] Add mmap support to lua_text
[Feature] Add monitored object for surbl plugin
[Feature] Add more exceptions to surbl whitelist
[Feature] Add more meta-tokens to bayes
[Feature] Add neural net classifier to fann_scores module
[Feature] Add neural net serialization/deserialization
[Feature] Add new dynamic conf module
[Feature] Add periodic events support for lua_config
[Feature] Add plugin to check MX'es for the sender's domain
[Feature] Add preliminary monitored module
[Feature] Add preliminary support of dynamic conf updates in Redis
[Feature] Add preliminary version of clamav plugin
[Feature] Add redis cache to asn module
[Feature] Add replies compression
[Feature] Add spamhaus DROP dnsbl
[Feature] Add support for dictionary in client compression
[Feature] Add support for fuzzy learn and unlearn from lua
[Feature] Add support for input encryption
[Feature] Add support of min_learns to neural net classifier
[Feature] Add termination callbacks for workers
[Feature] Add user-agent for rspamc
[Feature] Add utility to perform classifier tests
[Feature] Add zstd compression library
[Feature] Allow HTTPS requests in lua_http
[Feature] Allow conditions for pre and postfilters
[Feature] Allow custom functions for ratelimits
[Feature] Allow for excluding messages from AV scanning based on size
[Feature] Allow for getting worker stats from Lua
[Feature] Allow getting task UID from Lua
[Feature] Allow parsing of mailbox messages from the commandline
[Feature] Allow plugins to publish their lua API via rspamd_plugins
[Feature] Allow to compare other systems with Rspamd
[Feature] Allow to execute Lua scripts by controller
[Feature] Allow to have a function to set custom greylist message
[Feature] Allow to iterate over multiple tags
[Feature] Allow to pass extra data from plugins to log helper
[Feature] Allow to plan new periodics at different time
[Feature] Allow to reset hashes
[Feature] Allow to run rspamadm lua just as a lua interpreter
[Feature] Allow to store settings in redis
[Feature] Allow to update dynamic conf in Redis
[Feature] Allow to use dictionaries for compression
[Feature] Allow to use md5, sha1, sha256, sha384 and sha512 hashes in Lua
[Feature] Allow whitelisting by IP for greylisting plugin
[Feature] Antivirus: Support whitelists & pattern-matching sig names
[Feature] Backport pack/unpack routines from Lua 5.3
[Feature] Check settings with equal priopities in alphabetical order
[Feature] Compress neural net in redis
[Feature] Consider more tags when doing WHITE_ON_WHITE rule
[Feature] Descriptive options for DMARC failure symbols
[Feature] Descriptive options for RBL symbols
[Feature] Enable configuration for monitored objects
[Feature] Execute on_load scripts with ev_base ready
[Feature] Fann scores now uses metadata from a message
[Feature] Implement FANN threaded learning
[Feature] Implement classifying for lua classifiers
[Feature] Implement finish scripts for worker processes
[Feature] Implement monitoring for DNS resources
[Feature] Implement real priorities for pre and post filters
[Feature] Insert two symbols: FANN_HAM and FANN_SPAM instead of one
[Feature] Module to push metadata/messages to redis pubsub
[Feature] Monitor RBL records
[Feature] Move fann_classifier to a separate plugin
[Feature] Normalize all ANN inputs
[Feature] Preliminary version of metric exporter module
[Feature] Preserve decompression context between tasks
[Feature] Ratelimit: Support dynamic bucket size/leak rate
[Feature] Relax FORGED_RECIPIENTS: allow senders to BCC themselves
[Feature] Remove symbols weights on composites processing
[Feature] Return symbol scores when getting resulting symbols
[Feature] Rework lua tcp module
[Feature] Rule to detect some obvious X-PHP-Originating-Script forgeries
[Feature] Rule to identify some X-PHP-Script forgeries
[Feature] Rules for scoring Google Message-ID fixes
[Feature] Send hashes values to reply
[Feature] Set expire for dmarc reports
[Feature] Stop using cymru zone as it is unstable
[Feature] Stop using of GLists for headers, improve performance
[Feature] Store for in task:get_received_headers
[Feature] Store for part in received headers
[Feature] Store enabled flag for webui session
[Feature] Store error messages in ring buffer
[Feature] Support compressed maps
[Feature] Support excluding selected users from ratelimits
[Feature] Support looking up NS records in lua_dns
[Feature] Support modern style SURBL configuration
[Feature] Support multiple hashes in delhash path
[Feature] Support new messages in rspamc
[Feature] Support requests without reads in lua_tcp
[Feature] Support setting task message from Lua
[Feature] Track visibility of HTML elements
[Feature] Try to add CRLF when checking DKIM
[Feature] Try to guess line endings when folding headers
[Feature] Try to improve normalization function for bayes
[Feature] Use FFI to optimize SA module
[Feature] Use length based arguments for redis, allow lua_text as arg
[Feature] Use more layers for fann and another normalization
[Feature] User-defined ratelimits
[Feature] Utility to convert fuzzy storage from sqlite to redis
[Feature] Yield DMARC_DNSFAIL on lookup failure
[Fix] Adopt fuzzy storage for flexible backends
[Fix] Allow plain IP addresses in Rspamd maps
[Fix] Another fix for brain-damaged hiredis
[Fix] Another fix for rdns write errors
[Fix] Another fix for rdns_make_request_full invocation
[Fix] Another fix in DKIM canonicalization
[Fix] Another memory leak plugged
[Fix] Another try to deal with posix idiotizm
[Fix] Another try to fix RDNS events processing logic
[Fix] Avoid double frees in HEAD requests
[Fix] Avoid extra symbols for RBLs
[Fix] Banish table.maxn from Lua parts
[Fix] Check for socket error before connection in lua_tcp
[Fix] Correctly propagate redis timeouts to Lua
[Fix] Do not add extra newline in MIME mode
[Fix] Do not be cheated by system hiredis
[Fix] Do not classify when a message has not enough tokens
[Fix] Do not crash on redis errors
[Fix] Do not distinguish NXDOMAIN and NOREC for monitored
[Fix] Do not replan retransmits if merely one server is defined
[Fix] Do not use headers to calculate messages digests
[Fix] Don't force action in replies module for authenticated users/local networks
[Fix] Explicitly ban default passwords in webui
[Fix] Finally fix ambiguity between parsed and resolved spf elts
[Fix] Fix 'decoded' value in task:get_header_full()
[Fix] Fix DKIM calculations
[Fix] Fix DKIM signing for messages with no newline at the end
[Fix] Fix DNS request in monitored
[Fix] Fix DNS write errors processing
[Fix] Fix HTTP methods other than GET and POST
[Fix] Fix PERMFAIL for v6/v4 ambiguities
[Fix] Fix absurdic scores for HFILTER_URL_ONLY
[Fix] Fix actions in rolling history
[Fix] Fix actrie patterns
[Fix] Fix applying of lua dynamic confg
[Fix] Fix autolearning errors and redis cache
[Fix] Fix bayes learn_condition
[Fix] Fix build with the recent OpenSSL
[Fix] Fix caching and compressed maps
[Fix] Fix check plain text part
[Fix] Fix crash on OpenBSD in url_email_start
[Fix] Fix double free in SPF
[Fix] Fix extraction of shingles from redis fuzzy storage
[Fix] Fix false sharing for symbols in the cache
[Fix] Fix float usage in util:get_time
[Fix] Fix folding algorithm to deal with empty tokens
[Fix] Fix format string
[Fix] Fix format string usage in controller errors handling
[Fix] Fix handling of '�' in lua_tcp
[Fix] Fix handling of HTTP HEAD methods
[Fix] Fix hash creation
[Fix] Fix hiredis stupidity
[Fix] Fix implicit settings module settingsup
[Fix] Fix interaction with lua GC to avoid craches
[Fix] Fix ip_score module registration
[Fix] Fix issue with empty messages and dkim
[Fix] Fix issues with CGP helper
[Fix] Fix issues with the recent SPF changes
[Fix] Fix key name to load ANN correctly
[Fix] Fix lua tcp module by saving do_read in callback data
[Fix] Fix memory leak in client when using compression
[Fix] Fix min_learns option
[Fix] Fix on_finish scripts and async handlers
[Fix] Fix options for SPF dnsfail symbol
[Fix] Fix parsing includes and redirects in SPF
[Fix] Fix parsing of lua comments with empty lines
[Fix] Fix parsing of unquoted HTML attributes
[Fix] Fix periodic events and redis
[Fix] Fix processing of fuzzy learns from Lua
[Fix] Fix processing of redirect in SPF includes
[Fix] Fix processing of symbols when reject limit is reached
[Fix] Fix refcounts when map is specified by IP
[Fix] Fix rspamd{session} class in Lua API
[Fix] Fix setting ratelimit key for 'ip' bucket
[Fix] Fix some cases of TLD urls detector
[Fix] Fix statconvert tool
[Fix] Fix stats for backend-less classifiers
[Fix] Fix training script for fann_redis
[Fix] Fix variable in ann module
[Fix] Fix various errors in lua dynamic conf plugin
[Fix] Forget old ANN when max_usages is reached to avoid overtrain
[Fix] Further canonicalization fixes
[Fix] Further fixes for fann_redis prefixes
[Fix] Handle failures for inactive pooled connections
[Fix] Improve multimap info message
[Fix] More fixes in ANN loading
[Fix] More fixes to fann_redis
[Fix] More issues in fann_redis
[Fix] More spaces fix in DKIM signature
[Fix] Multiple fixes to asn script, add IPv6 support
[Fix] Multiple issues in fann_redis
[Fix] No greylist rejected messages
[Fix] One more attempt to fix lua_redis
[Fix] One more check for readdir...
[Fix] Params should be treated as a hash
[Fix] Plug memory leak in regexp desctructor
[Fix] Process headers only once
[Fix] Properly handle nil values in ratelimit plugin
[Fix] Really fix redis shingles check
[Fix] Remove fann with incorrect layers count
[Fix] Remove mentions of deleted include
[Fix] Remove some incompatible functions
[Fix] Settings: fix authenticated parameter (#886)
[Fix] Skip MX check for authenticated users and local networks
[Fix] Slightly fix ANN routines
[Fix] Stop caching records with DNS failures
[Fix] Treat all errors in redis_pool as fatal errors for a connection
[Fix] Try avoid false-positives in HEADER_FORGED_MDN rule
[Fix] Try to avoid race condition when using rrd
[Fix] Try to reload redis scripts if they are missing
[Fix] Unbreak once_received skipping for local networks
[Fix] Unlock ANN on error
[Fix] Use memmove for overlapping regions
[Fix] Use real size instead of displayed for core limits
[Fix] Use the correct macro to get the size of control
[Fix] Various fixes for errors ringbuffer
[Fix] Yield R_SPF_DNSFAIL if lookup of included record fails
[Fix] mid: fix map initialization
[Fix] mid: handle incorrect rgexps in the map
[Rework] Add extract training data function to fann_redis
[Rework] Add preliminary train tests
[Rework] Add redis storage feature to fann_redis
[Rework] Adopt fuzzy storage for abstract backend
[Rework] Adopt plugins
[Rework] First reiteration on fann scores
[Rework] Implement loading/invalidating
[Rework] Make lua_redis task agnostic
[Rework] Make rspamd protocol messages useful
[Rework] Massive removal of legacy code
[Rework] More cleanup actions
[Rework] Remove legacy code never used for classifiers
[Rework] Remove outdated and unused lua_session module
[Rework] Reorganize fuzzy backend structure
[Rework] Reorganize the internal backend structure
[Rework] Restore old fann_scores, move common parts
[Rework] Rework and simplify rbl plugin
[Rework] Rework parsing of DMARC records