Graylog2 Server Currently Unavailable Port 12900

scottalanmiller

I have Graylog installed but am trying to do something a little more complicated than a typical install, using external ElasticSearch cluster. I'm getting this error of "We are experiencing problems connecting to the Graylog server running on http://127.0.0.1:12900/. Please verify that the server is healthy and working correctly."

0_1471705482740_Screenshot from 2016-08-20 11-03-50.png

I've reconfigured to use the private IP address as well, that does not work either, as you can see in the screen shot. I've checked on the box and 12900 is listening and telnet can connect to it. 12900 is the REST API for the system.

# netstat -tulpn | grep 12900
tcp6       0      0 127.0.0.1:12900         :::*                    LISTEN      11769/java

telnet localhost 12900
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

scottalanmiller

Found the issue. Misdocumentation on the Graylog site, big surprise. Same kinds of issues as before. ElasticSearch was not responding locally and it can't talk to the cluster until the local client is up and running. Got that working and we have Graylog2 up and running!

JaredBusch

@scottalanmiller SELinux?

scottalanmiller

@JaredBusch said in Graylog2:

@scottalanmiller SELinux?

Decent guess but that didn't change it

DustinB3403

from the localhost can you see /system/sessions as accessible?

gjacobse

HTTPS vs HTTP?

Is that port blocked in your FW?

scottalanmiller

@gjacobse said in Graylog2:

HTTPS vs HTTP?

Is that port blocked in your FW?

It's unblocked and tested. It's all HTTP. No proxy in place yet.

scottalanmiller

@DustinB3403 said in Graylog2:

from the localhost can you see /system/sessions as accessible?

It doesn't like local...

curl localhost:9000/system/sessions
<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="robots" content="noindex, nofollow">
    <meta charset="UTF-8">
    <title>Graylog Web Interface</title>
    <link rel="shortcut icon" href="/favicon.png">
    
  </head>
  <body>
    <script src="/config.js"></script>
    <script src="/vendor.js"></script>
    
    <script src="/polyfill.f2a88026f0d0e7332f17.js"></script>
    
    <script src="/plugin/org.graylog.plugins.pipelineprocessor.ProcessorPlugin/plugin.org.graylog.plugins.pipelineprocessor.PipelineProcessorPlugin.124facd49c87779f691b.js"></script>
    
    <script src="/plugin/org.graylog.plugins.map.MapWidgetPlugin/plugin.org.graylog.plugins.map.MapWidgetPlugin.3a6e86e960b80407f74e.js"></script>
    
    <script src="/plugin/org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin/plugin.org.graylog.plugins.enterprise_integration.EnterpriseIntegrationPlugin.7f1e11d52a66d206799c.js"></script>
    
    <script src="/plugin/org.graylog.plugins.collector.CollectorPlugin/plugin.org.graylog.plugins.collector.CollectorPlugin.74e8dfee2c1d6b11bc87.js"></script>
    
    <script src="/app.f2a88026f0d0e7332f17.js"></script>
    
  </body>
</html>

scottalanmiller

I've totally disabled the firewall, just to be sure that that is not the problem.

scottalanmiller

Just realized that 12900 is exclusively listening on the loopback.

DustinB3403

But that doesn't look like it's error'd out, what if you use the IP address rather than localhost.

mlnews

@DustinB3403 said in Graylog2:

But that doesn't look like it's error'd out, what if you use the IP address rather than localhost.

It errored out on browser detection.

scottalanmiller

Here is the current status of the configuration file:

grep -v ^# /etc/graylog/server/server.conf 

is_master = true
node_id_file = /etc/graylog/server/node-id
password_secret = [redacted]
root_password_sha2 = [redacted]
root_email = [redacted]
root_timezone = UTC
plugin_dir = /usr/share/graylog-server/plugin
rest_listen_uri = http://127.0.0.1:12900/
web_listen_uri = http://0.0.0.0:9000/
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
rotation_strategy = count
elasticsearch_max_docs_per_index = 20000000
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_max_number_of_indices = 20
retention_strategy = delete
elasticsearch_shards = 4
elasticsearch_replicas = 0
elasticsearch_index_prefix = graylog
allow_leading_wildcard_searches = false
allow_highlighting = false
elasticsearch_cluster_name = ntg-graylog2
elasticsearch_discovery_zen_ping_unicast_hosts = 127.0.0.1:9300, 192.168.1.51:9300, 192.168.1.52:9300, 192.168.1.53:9300, 192.168.1.51:9500
elasticsearch_node_master = false
elasticsearch_node_data = false
elasticsearch_analyzer = standard
output_batch_size = 500
output_flush_interval = 1
output_fault_count_threshold = 5
output_fault_penalty_seconds = 30
processbuffer_processors = 5
outputbuffer_processors = 3
processor_wait_strategy = blocking
ring_size = 65536
inputbuffer_ring_size = 65536
inputbuffer_processors = 2
inputbuffer_wait_strategy = blocking
message_journal_enabled = true
message_journal_dir = /var/lib/graylog-server/journal
lb_recognition_period_seconds = 3
mongodb_uri = mongodb://localhost/graylog
mongodb_max_connections = 1000
mongodb_threads_allowed_to_block_multiplier = 5
content_packs_dir = /usr/share/graylog-server/contentpacks

scottalanmiller

Looking at info here... https://github.com/Graylog2/graylog2-server/issues/1908

scottalanmiller

Here is the log output on a restart. Getting errors trying to connect to a local service, that I know is running.

2016-08-20T17:19:14.300Z INFO  [node] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] started
2016-08-20T17:19:14.547Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
2016-08-20T17:19:16.789Z INFO  [NetworkListener] Started listener bound to [0.0.0.0:9000]
2016-08-20T17:19:16.791Z INFO  [HttpServer] [HttpServer] Started.
2016-08-20T17:19:16.791Z INFO  [WebInterfaceService] Started Web Interface at <http://0.0.0.0:9000/>
2016-08-20T17:19:17.596Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
2016-08-20T17:19:19.310Z WARN  [IndexerSetupService] Could not connect to Elasticsearch
2016-08-20T17:19:19.310Z INFO  [IndexerSetupService] If you're using multicast, check that it is working in your network and that Elasticsearch is accessible. Also check that the cluster name setting is correct.
2016-08-20T17:19:19.310Z INFO  [IndexerSetupService] See http://docs.graylog.org/en/2.0/pages/configuration/elasticsearch.html for details.
2016-08-20T17:19:20.648Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
2016-08-20T17:19:22.266Z INFO  [NetworkListener] Started listener bound to [0.0.0.0:12900]
2016-08-20T17:19:22.266Z INFO  [HttpServer] [HttpServer-1] Started.
2016-08-20T17:19:22.266Z INFO  [RestApiService] Started REST API at <http://0.0.0.0:12900/>
2016-08-20T17:19:22.267Z INFO  [ServiceManagerListener] Services are healthy
2016-08-20T17:19:22.268Z INFO  [ServerBootstrap] Services started, startup times in ms: {OutputSetupService [RUNNING]=196, BufferSynchronizerService [RUNNING]=202, MetricsReporterService [RUNNING]=203, KafkaJournal [RUNNING]=240, InputSetupService [RUNNING]=276, PeriodicalsService [RUNNING]=642, JournalReader [RUNNING]=854, WebInterfaceService [RUNNING]=6864, IndexerSetupService [RUNNING]=9386, RestApiService [RUNNING]=12343}
2016-08-20T17:19:22.272Z INFO  [ServerBootstrap] Graylog server up and running.
2016-08-20T17:19:22.277Z INFO  [InputSetupService] Triggering launching persisted inputs, node transitioned from Uninitialized [LB:DEAD] to Running [LB:ALIVE]
2016-08-20T17:19:23.711Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
2016-08-20T17:19:25.121Z INFO  [IndexRangesCleanupPeriodical] Skipping index range cleanup because the Elasticsearch cluster is unreachable or unhealthy
2016-08-20T17:19:26.736Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
2016-08-20T17:19:29.764Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
2016-08-20T17:19:32.785Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
2016-08-20T17:19:35.810Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
2016-08-20T17:19:38.826Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
2016-08-20T17:19:41.845Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
2016-08-20T17:19:44.860Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
2016-08-20T17:19:47.874Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]
2016-08-20T17:19:50.889Z INFO  [zen] [graylog-e969a669-a37c-4004-9d29-63e99daaf4cf] failed to send join request to master [{prd-lnx-elastic1}{Z3r9bHOgRrGzxzm9J6zJfA}{192.168.1.51}{192.168.1.51:9300}], reason [RemoteTransportException[[prd-lnx-elastic1][192.168.1.51:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[graylog-e969a669-a37c-4004-9d29-63e99daaf4cf][127.0.0.1:9350] connect_timeout[30s]]; nested: NotSerializableExceptionWrapper[connect_exception: Connection refused: /127.0.0.1:9350]; ]

scottalanmiller

I made the same change as the person in the GitHub thread and opened port 12900 to the world so that I could test from a browser. And it indeed can be reached for me, the same as him. So the error appears to be wrong.

0_1471713807470_Screenshot from 2016-08-20 13-23-18.png

scottalanmiller

Found the issue. Misdocumentation on the Graylog site, big surprise. Same kinds of issues as before. ElasticSearch was not responding locally and it can't talk to the cluster until the local client is up and running. Got that working and we have Graylog2 up and running!

StrongBad

Don't forget to mark the thread as solved then

scottalanmiller

@StrongBad said in Graylog2:

Don't forget to mark the thread as solved then

Marked