Page MenuHome

Elastic: when connection fails there is no indication of what is being connected to
Closed, ResolvedPublic

Description

When running ./manage.sh elastic reset_index inside the blender_cloud docker container, I got this traceback:

Traceback (most recent call last):
  File "/opt/python/lib/python3.6/site-packages/urllib3/connection.py", line 141, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/opt/python/lib/python3.6/site-packages/urllib3/util/connection.py", line 83, in create_connection
    raise err
  File "/opt/python/lib/python3.6/site-packages/urllib3/util/connection.py", line 73, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/python/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py", line 147, in perform_request
    response = self.pool.urlopen(method, url, body, retries=False, headers=self.headers, **kw)
  File "/opt/python/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/opt/python/lib/python3.6/site-packages/urllib3/util/retry.py", line 333, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/opt/python/lib/python3.6/site-packages/urllib3/packages/six.py", line 686, in reraise
    raise value
  File "/opt/python/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/opt/python/lib/python3.6/site-packages/urllib3/connectionpool.py", line 357, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/opt/python/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/opt/python/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/opt/python/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/opt/python/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/opt/python/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/opt/python/lib/python3.6/site-packages/urllib3/connection.py", line 166, in connect
    conn = self._new_conn()
  File "/opt/python/lib/python3.6/site-packages/urllib3/connection.py", line 150, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f666a9fb438>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "manage.py", line 7, in <module>
    cli.manager.run()
  File "/opt/python/lib/python3.6/site-packages/flask_script/__init__.py", line 417, in run
    result = self.handle(argv[0], argv[1:])
  File "/opt/python/lib/python3.6/site-packages/flask_script/__init__.py", line 386, in handle
    res = handle(*args, **config)
  File "/opt/python/lib/python3.6/site-packages/flask_script/commands.py", line 216, in __call__
    return self.run(*args, **kwargs)
  File "/data/git/pillar/pillar/cli/elastic.py", line 39, in reset_index
    task.execute()
  File "/data/git/pillar/pillar/api/search/index.py", line 42, in execute
    idx.delete(ignore=404)
  File "/opt/python/lib/python3.6/site-packages/elasticsearch_dsl/index.py", line 314, in delete
    return self.connection.indices.delete(index=self._name, **kwargs)
  File "/opt/python/lib/python3.6/site-packages/elasticsearch/client/utils.py", line 76, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/opt/python/lib/python3.6/site-packages/elasticsearch/client/indices.py", line 188, in delete
    params=params)
  File "/opt/python/lib/python3.6/site-packages/elasticsearch/transport.py", line 314, in perform_request
    status, headers, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
  File "/opt/python/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py", line 156, in perform_request
    raise ConnectionError('N/A', str(e), e)
elasticsearch.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7f666a9fb438>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7f666a9fb438>: Failed to establish a new connection: [Errno 111] Connection refused)

There is no indication as to what URL the script tries to connect to, making it harder to diagnose.

Details

Type
Bug

Event Timeline

Sybren A. Stüvel (sybren) lowered the priority of this task from Needs Triage by Developer to Normal.Jan 3 2018, 5:57 PM
Sybren A. Stüvel (sybren) created this task.

Another one appears when trying to search for a user:

[Wed Jan 03 17:25:37.002496 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820] Traceback (most recent call last):
[Wed Jan 03 17:25:37.002556 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]   File "/opt/python/lib/python3.6/site-packages/eve/flaskapp.py", line 1033, in __call__
[Wed Jan 03 17:25:37.002561 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]     return super(Eve, self).__call__(environ, start_response)
[Wed Jan 03 17:25:37.002566 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]   File "/opt/python/lib/python3.6/site-packages/flask/app.py", line 1994, in __call__
[Wed Jan 03 17:25:37.002568 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]     return self.wsgi_app(environ, start_response)
[Wed Jan 03 17:25:37.002573 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]   File "/opt/python/lib/python3.6/site-packages/flask/app.py", line 1985, in wsgi_app
[Wed Jan 03 17:25:37.002577 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]     response = self.handle_exception(e)
[Wed Jan 03 17:25:37.002583 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]   File "/opt/python/lib/python3.6/site-packages/flask/app.py", line 1540, in handle_exception
[Wed Jan 03 17:25:37.002587 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]     reraise(exc_type, exc_value, tb)
[Wed Jan 03 17:25:37.002592 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]   File "/opt/python/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
[Wed Jan 03 17:25:37.002595 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]     raise value
[Wed Jan 03 17:25:37.002601 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]   File "/opt/python/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
[Wed Jan 03 17:25:37.002605 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]     response = self.full_dispatch_request()
[Wed Jan 03 17:25:37.002611 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]   File "/opt/python/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request
[Wed Jan 03 17:25:37.002621 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]     rv = self.handle_user_exception(e)
[Wed Jan 03 17:25:37.002627 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]   File "/opt/python/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception
[Wed Jan 03 17:25:37.002629 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]     reraise(exc_type, exc_value, tb)
[Wed Jan 03 17:25:37.002632 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]   File "/opt/python/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
[Wed Jan 03 17:25:37.002634 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]     raise value
[Wed Jan 03 17:25:37.002640 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]   File "/opt/python/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
[Wed Jan 03 17:25:37.002643 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]     rv = self.dispatch_request()
[Wed Jan 03 17:25:37.002650 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]   File "/opt/python/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
[Wed Jan 03 17:25:37.002655 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]     return self.view_functions[rule.endpoint](**req.view_args)
[Wed Jan 03 17:25:37.002662 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]   File "/data/git/pillar/pillar/api/search/routes.py", line 52, in search_user
[Wed Jan 03 17:25:37.002667 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]     data = queries.do_user_search(searchword, terms)
[Wed Jan 03 17:25:37.002673 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]   File "/data/git/pillar/pillar/api/search/queries.py", line 113, in do_user_search
[Wed Jan 03 17:25:37.002676 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]     response = search.execute()
[Wed Jan 03 17:25:37.002681 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]   File "/opt/python/lib/python3.6/site-packages/elasticsearch_dsl/search.py", line 639, in execute
[Wed Jan 03 17:25:37.002684 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]     **self._params
[Wed Jan 03 17:25:37.002688 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]   File "/opt/python/lib/python3.6/site-packages/elasticsearch/client/utils.py", line 76, in _wrapped
[Wed Jan 03 17:25:37.002690 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]     return func(*args, params=params, **kwargs)
[Wed Jan 03 17:25:37.002693 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]   File "/opt/python/lib/python3.6/site-packages/elasticsearch/client/__init__.py", line 636, in search
[Wed Jan 03 17:25:37.002695 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]     doc_type, '_search'), params=params, body=body)
[Wed Jan 03 17:25:37.002700 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]   File "/opt/python/lib/python3.6/site-packages/elasticsearch/transport.py", line 314, in perform_request
[Wed Jan 03 17:25:37.002704 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]     status, headers, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
[Wed Jan 03 17:25:37.002709 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]   File "/opt/python/lib/python3.6/site-packages/elasticsearch/connection/http_urllib3.py", line 156, in perform_request
[Wed Jan 03 17:25:37.002712 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820]     raise ConnectionError('N/A', str(e), e)
[Wed Jan 03 17:25:37.002736 2018] [wsgi:error] [pid 347:tid 140106908223232] [remote 172.18.0.11:63820] elasticsearch.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7f6d3c790ef0>: Failed to establish a new connection: [Errno 111] Connection refused) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7f6d3c790ef0>: Failed to establish a new connection: [Errno 111] Connection refused)

I can't figure out how the search.execute() call knows which ElasticSearch host to connect to. It may connect to localhost or elasticsearch by default, but neither work in our production environment.