Repository navigation
Description
16:34:46.561Z ERROR elastalert-server:
ProcessController: ERROR:root:Traceback (most recent call last):
File "/opt/elastalert/elastalert/elastalert.py", line 1173, in run_all_rules
num_matches = self.run_rule(rule, endtime, self.starttime)
File "/opt/elastalert/elastalert/elastalert.py", line 870, in run_rule
if not self.run_query(rule, rule['starttime'], endtime):
File "/opt/elastalert/elastalert/elastalert.py", line 602, in run_query
data = self.get_hits(rule, start, end, index, scroll)
File "/opt/elastalert/elastalert/elastalert.py", line 331, in get_hits
if self.current_es.is_atleastsixsix():
File "elastalert/init.py", line 69, in is_atleastsixsix
major, minor = map(int, self.es_version.split(".")[:2])
File "elastalert/init.py", line 43, in es_version
self._es_version = self.info()['version']['number']
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.2-py2.7.egg/elasticsearch/client/utils.py", line 84, in _wrapped
return func(*args, params=params, **kwargs)
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.2-py2.7.egg/elasticsearch/client/init.py", line 259, in info
return self.transport.perform_request("GET", "/", params=params)
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.2-py2.7.egg/elasticsearch/transport.py", line 353, in perform_request
timeout=timeout,
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.2-py2.7.egg/elasticsearch/connection/http_requests.py", line 138, in perform_request
raise ConnectionTimeout("TIMEOUT", str(e), e)
ConnectionTimeout: ConnectionTimeout caused by - ConnectTimeout(HTTPConnectionPool(host='xxx.xxx.xxx.xxx', port=9200): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7efc1794ced0>, 'Connection to xxx.xxx.xxx.xxx timed out. (connect timeout=20)')))
ERROR:root:Uncaught exception running rule poolqq: ConnectionTimeout caused by - ConnectTimeout(HTTPConnectionPool(host='xxx.xxx.xxx.xxx', port=9200): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7efc1794ced0>, 'Connection to xxx.xxx.xxx.xxx timed out. (connect timeout=20)')))
If this error occurs on my server, the alarm will not be issued again, and only by restarting the server can the alarm be triggered again
However, 9200 is normal. May I ask why?
Activity
nsano-rururu commentedon Jun 23, 2020
It is a sample
Elasticsearch 7.8.0
Kibana 7.8.0
bitsensor/elastalert:3.0.0-beta.0
docker-compose.yml
kibana/config/kibana.yml
es/config/elasticsearch.yml
Dockerfiles/Dockerfile-elastalert
elastalert/bin/elastic_search_status.sh
elastalert/bin/elastalert-start.sh
elastalert/config/config.json
elastalert/config/elastalert-test.yml
elastalert/config/elastalert.yml
kakaNo1 commentedon Jun 24, 2020
Ok, thank you for your timely reply. I deployed the official Docker, and there was no problem at the beginning (the alarm was normal). Within a week or a few days, the above error would appear, and then no alarm would be sent
xuanyuanaosheng commentedon Jun 24, 2020
@nsano-rururu The elastalert now is only support the python 3.
nsano-rururu commentedon Jun 24, 2020
・Which Docker image are you using?
Babel 6 → 7
bug fix
・Cloud (AWS1, Azure, etc.), physical server?
・What are the server specifications? (CPU, number of CPU cores, memory, etc.)
・Are you using Kibana without problems when an error occurs?
・Is Elasticsearch running on docker ps?
・Did you check docker logs to see if there are any errors in the Elasticsearch container?
・How much memory is available by executing the "free -h" command when an error occurs?
nsano-rururu commentedon Jun 24, 2020
Yeah, right,
kakaNo1 commentedon Jun 28, 2020
The mirror is bitsensor/elastalert: 3.0.0 - beta. 1
Elasticsearch version 7.2.0
Elastalert Server is available in the virtual machine
Elasticsearch Cluster in Aliyun (2 8C16G)
There's nothing wrong with Kibana when an error occurs
There's enough storage space
Python3 is used to run
nsano-rururu commentedon Jun 28, 2020
I've never run ElastAlert on an Elasticsearch Cluster, so my current knowledge may not tell...
nsano-rururu commentedon Jun 28, 2020
I'm aware that Elasticsearch, Kibana and ElastAlert are running on the same server.
It seems to be connectivity issue. Can you connect to ES with curl -v http://localhost:9200 from docker host?
nsano-rururu commentedon Jun 28, 2020
Is that server up and running with anything besides Elasticsearch, Kibana and ElatAlert installed?
Could you tell me the information of docker-compose.yml and the configuration file when docker container is started?
nsano-rururu commentedon Jun 28, 2020
Since the error message is connection timeout, the size of the index to be searched may be very large, but I can not clearly say that it is only the information provided so far.
nsano-rururu commentedon Jun 28, 2020
I found a setting in the ElastAlert documentation to change the timeout period.
Add es_conn_timeout to elastalert.yml of ElastAlert and set the setting value to a large number (default is 20)
Try restarting the Docker container for ElastAlert.
kakaNo1 commentedon Jun 29, 2020
Thank you for your timely reply. I will try today. My server is running in K8S, do you need to check YAML
kakaNo1 commentedon Jun 29, 2020
I set it up in config.yaml
Es_conn_timeout: 50
Let's see if we can make the same mistakes tomorrow
kakaNo1 commentedon Jul 3, 2020
I'm very sorry, there is still an error, timeout time is 50, the reason for the problem is again, when timeout occurs, the alarm will not be triggered again, can you configure the retry mechanism?, I delete index again and restart the program, and everything is fine again,
16:01:30.692Z ERROR elastalert-server:
ProcessController: ERROR:root:Traceback (most recent call last):
File "/opt/elastalert/elastalert/elastalert.py", line 1173, in run_all_rules
num_matches = self.run_rule(rule, endtime, self.starttime)
File "/opt/elastalert/elastalert/elastalert.py", line 870, in run_rule
if not self.run_query(rule, rule['starttime'], endtime):
File "/opt/elastalert/elastalert/elastalert.py", line 602, in run_query
data = self.get_hits(rule, start, end, index, scroll)
File "/opt/elastalert/elastalert/elastalert.py", line 331, in get_hits
if self.current_es.is_atleastsixsix():
File "elastalert/init.py", line 69, in is_atleastsixsix
major, minor = map(int, self.es_version.split(".")[:2])
File "elastalert/init.py", line 43, in es_version
self._es_version = self.info()['version']['number']
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.2-py2.7.egg/elasticsearch/client/utils.py", line 84, in _wrapped
return func(*args, params=params, **kwargs)
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.2-py2.7.egg/elasticsearch/client/init.py", line 259, in info
return self.transport.perform_request("GET", "/", params=params)
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.2-py2.7.egg/elasticsearch/transport.py", line 353, in perform_request
timeout=timeout,
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.2-py2.7.egg/elasticsearch/connection/http_requests.py", line 138, in perform_request
raise ConnectionTimeout("TIMEOUT", str(e), e)
ConnectionTimeout: ConnectionTimeout caused by - ConnectTimeout(HTTPConnectionPool(host='xxx.xxx.xxx.xxx', port=9200): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7fb16464b710>, 'Connection to xxx.xxx.xxx.xxx timed out. (connect timeout=50)')))
nsano-rururu commentedon Jul 3, 2020
I need to see if I have the same issue with a Docker image that uses a newer version of ElastAlert
servercentral/elastalert・・・ElastAlert 0.2.1
https://hub.docker.com/r/servercentral/elastalert
johnsusek/elastalert-server・・・ElastAlert 0.2.4
https://hub.docker.com/r/johnsusek/elastalert-server
kakaNo1 commentedon Jul 6, 2020
Hello, I noticed that the official server mirror is Bitsensor /elastalert
May I ask that the two you recommend are the same?
kakaNo1 commentedon Jul 6, 2020
The following error occurs when using the johnsusek/elastalert-server:latest startup:
08:56:06.576Z INFO elastalert-server: ProcessController: Creating index
08:56:08.914Z INFO elastalert-server:
ProcessController: Elastic Version:7
Mapping used for string:{'type': 'keyword'}
Index elastalert_status already exists. Skipping index creation.
08:56:08.915Z INFO elastalert-server: ProcessController: Index create exited with code 0
08:56:08.916Z INFO elastalert-server: ProcessController: Starting elastalert with arguments [none]
08:56:08.931Z INFO elastalert-server: ProcessController: Started Elastalert (PID: 47)
08:56:08.935Z INFO elastalert-server: Server: Server listening on port 3030
08:56:08.937Z INFO elastalert-server: Server: Websocket listening on port 3333
08:56:08.938Z INFO elastalert-server: Server: Server started
08:56:11.534Z ERROR elastalert-server:
ProcessController: Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/opt/elastalert/elastalert/elastalert.py", line 1929, in
08:56:11.535Z ERROR elastalert-server:
ProcessController: sys.exit(main(sys.argv[1:]))
File "/opt/elastalert/elastalert/elastalert.py", line 1925, in main
client.start()
File "/opt/elastalert/elastalert/elastalert.py", line 1106, in start
08:56:11.538Z ERROR elastalert-server:
ProcessController: self.run_all_rules()
File "/opt/elastalert/elastalert/elastalert.py", line 1158, in run_all_rules
self.send_pending_alerts()
File "/opt/elastalert/elastalert/elastalert.py", line 1534, in send_pending_alerts
08:56:11.538Z ERROR elastalert-server:
ProcessController: pending_alerts = self.find_recent_pending_alerts(self.alert_time_limit)
File "/opt/elastalert/elastalert/elastalert.py", line 1526, in find_recent_pending_alerts
08:56:11.540Z ERROR elastalert-server:
ProcessController: size=1000)
File "/usr/lib/python2.7/site-packages/elasticsearch-7.0.1-py2.7.egg/elasticsearch/client/utils.py", line 84, in _wrapped
08:56:11.541Z ERROR elastalert-server:
ProcessController: return func(*args, params=params, **kwargs)
TypeError: search() got an unexpected keyword argument 'doc_type'
08:56:11.604Z ERROR elastalert-server: ProcessController: ElastAlert exited with code 1
08:56:11.605Z INFO elastalert-server: Server: Stopping server
08:56:11.605Z INFO elastalert-server: ProcessController: ElastAlert is not running
08:56:11.606Z INFO elastalert-server: Server: Server stopped. Bye!
nsano-rururu commentedon Jul 6, 2020
Bitsensor/elastalert will no longer be updated
nsano-rururu commentedon Jul 6, 2020
It is difficult to answer without information about what you did. Because it’s not Esper
nsano-rururu commentedon Jul 6, 2020
kakaNo1 commentedon Jul 7, 2020
“
ADD elastalert/bin/elastalert-start.sh /usr/local/bin/
ADD elastalert/bin/elastic_search_status.sh /usr/local/bin/
”
Where are these two files?I didn't find
nsano-rururu commentedon Jul 7, 2020
I can not do it. I have no time to bite. I'm sorry. goodbye.
nsano-rururu commentedon Jul 7, 2020
The file is in the first answer.