ELK stops accepting logs

Danp

Ran a plugin update. Will see if that has any effect.

Updated logstash-codec-json 2.1.0 to 2.1.1
Updated logstash-filter-dns 2.0.2 to 2.1.0
Updated logstash-filter-json 2.0.3 to 2.0.4
Updated logstash-input-beats 2.1.3 to 2.2.0
Updated logstash-input-rabbitmq 3.1.4 to 3.3.0
Updated logstash-input-redis 2.0.2 to 2.0.3
Updated logstash-output-elasticsearch 2.5.1 to 2.5.3

Danp

During the command bin/plugin update, received the error

Error Bundler::InstallError, retrying 1/10
An error occurred while installing manticore (0.5.5), and Bundler cannot continue.
Make sure that `gem install manticore -v '0.5.5'` succeeds before bundling.

Discussions on similar issues can be found here and here.

scottalanmiller

Lots of reports on this, although I have a feeling that it might not be a leak but might be just "how much memory Logstash needs."

Try setting it to something high and see if it stops expanding.

Danp

@scottalanmiller said:

Lots of reports on this, although I have a feeling that it might not be a leak but might be just "how much memory Logstash needs."

Try setting it to something high and see if it stops expanding.

I already went from 1g to 2g. I could be off base, but I suspect a memory leak b/c it runs for days before the OOM condition occurs. The only active feed is from my XO server, and that activity should be light.

Danp

Found this current Github discussion that describes the plugin update issue I ran into.

scottalanmiller

@Danp said:

@scottalanmiller said:

Lots of reports on this, although I have a feeling that it might not be a leak but might be just "how much memory Logstash needs."

Try setting it to something high and see if it stops expanding.

I already went from 1g to 2g. I could be off base, but I suspect a memory leak b/c it runs for days before the OOM condition occurs. The only active feed is from my XO server, and that activity should be light.

That would be the expected behaviour whether a leak or just hasn't fully grown yet.

Danp

Finally back up and running...

scottalanmiller

@Danp said:

Finally back up and running...

What did you do?

Danp

@scottalanmiller Reinstalled ruby-maven and manticore.

Danp

No issues in almost a week. I will probably drop the heap size back to 1g and see if it remains stable.