ML
    • Recent
    • Categories
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Building ELK on CentOS 7

    Scheduled Pinned Locked Moved IT Discussion
    scalentg labscale hc3centoscentos 7elklogginglog managementhow tolinuxelasticsearchkibanalogstashkibana 4
    43 Posts 8 Posters 16.9k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • RamblingBipedR
      RamblingBiped
      last edited by

      Nice! I'll try to give this a whirl at some point in the next couple of days.

      Thanks!

      1 Reply Last reply Reply Quote 0
      • DashrenderD
        Dashrender @scottalanmiller
        last edited by

        @scottalanmiller said:

        Half a terabyte is a good starting point for disk space.

        So much for me trying it - I might be lucky if I have 100 GB available for this. šŸ˜ž

        scottalanmillerS 1 Reply Last reply Reply Quote 0
        • scottalanmillerS
          scottalanmiller @Dashrender
          last edited by

          @Dashrender said:

          @scottalanmiller said:

          Half a terabyte is a good starting point for disk space.

          So much for me trying it - I might be lucky if I have 100 GB available for this. šŸ˜ž

          You can do that for seeing what it looks like. 20GB will work for a very tiny test workload. But very tiny.

          1 Reply Last reply Reply Quote 0
          • scottalanmillerS
            scottalanmiller
            last edited by

            Just tested on a fresh build and it works BEAUTIFULLY. I put it into a script and ran it instead of going line by line, worked on the first try, no problems. It stops in the middle and asks for a password, that could be moved to the end or something, but it works just fine and isn't so slow that you'd want to walk away. So I added a BASH script header. If you want, just copy/paste into a text file and run it. Boom, done. Working ELK in a minute.

            JaredBuschJ 1 Reply Last reply Reply Quote 2
            • JaredBuschJ
              JaredBusch @scottalanmiller
              last edited by

              @scottalanmiller so what do you setup your disk partitioning like in CentOS 7?

              On a minimal install left to automatic, if you use a larger drive, it will create a separate partition for all the space after 50gb.

              this is highly annoying because I created a 127GB drive (default in Hyper-V) and now 50GB is separate from all the rest.

              scottalanmillerS 1 Reply Last reply Reply Quote 0
              • JaredBuschJ
                JaredBusch
                last edited by

                like this

                [root@elk ~]# df -h
                Filesystem                   Size  Used Avail Use% Mounted on
                /dev/mapper/centos_elk-root   50G  855M   50G   2% /
                devtmpfs                     906M     0  906M   0% /dev
                tmpfs                        916M     0  916M   0% /dev/shm
                tmpfs                        916M  8.3M  907M   1% /run
                tmpfs                        916M     0  916M   0% /sys/fs/cgroup
                /dev/sda2                    494M   98M  396M  20% /boot
                /dev/sda1                    200M  9.5M  191M   5% /boot/efi
                /dev/mapper/centos_elk-home   75G   33M   75G   1% /home
                tmpfs                        184M     0  184M   0% /run/user/0
                [root@elk ~]#
                
                1 Reply Last reply Reply Quote 0
                • JaredBuschJ
                  JaredBusch
                  last edited by

                  You should at least tell the user that you are asking for the kibana password.

                  htpasswd -c /etc/nginx/htpasswd.users kibanauser

                  0_1456293132171_upload-2a181928-c672-4286-86d6-43b69bd92fc3

                  1 Reply Last reply Reply Quote 0
                  • JaredBuschJ
                    JaredBusch
                    last edited by

                    I had this error.

                    0_1456293268870_upload-fc8bbfa4-c8e3-4c86-8e10-5cd5cf3195c8

                    1 Reply Last reply Reply Quote 0
                    • JaredBuschJ
                      JaredBusch
                      last edited by

                      Looks like maybe you forgot to start firewalld?

                        % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                                       Dload  Upload   Total   Spent    Left  Speed
                      100   814  100   814    0     0   1370      0 --:--:-- --:--:-- --:--:--  1372
                      {
                        "acknowledged" : true
                      }
                      FirewallD is not running
                      FirewallD is not running
                      [root@elk ~]# yum install firewalld
                      Loaded plugins: fastestmirror
                      Loading mirror speeds from cached hostfile
                       * base: mirror.oss.ou.edu
                       * epel: fedora-epel.mirror.lstn.net
                       * extras: centos.mirrors.wvstateu.edu
                       * updates: centos.mirrors.wvstateu.edu
                      Package firewalld-0.3.9-14.el7.noarch already installed and latest version
                      Nothing to do
                      [root@elk ~]# systemctl start firewalld
                      [root@elk ~]# systemctl status firewalld
                      ā— firewalld.service - firewalld - dynamic firewall daemon
                         Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
                         Active: active (running) since Tue 2016-02-23 23:55:11 CST; 14s ago
                       Main PID: 11482 (firewalld)
                         CGroup: /system.slice/firewalld.service
                                 └─11482 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid
                      
                      Feb 23 23:55:09 elk systemd[1]: Starting firewalld - dynamic firewall daemon...
                      Feb 23 23:55:11 elk systemd[1]: Started firewalld - dynamic firewall daemon.
                      [root@elk ~]#
                      
                      scottalanmillerS 1 Reply Last reply Reply Quote 0
                      • JaredBuschJ
                        JaredBusch
                        last edited by

                        Yeah, you set it to install, but you never start or enable it.

                        0_1456293489801_upload-2c71d7be-5435-4f22-ab21-c2912aa344c8

                        1 Reply Last reply Reply Quote 0
                        • JaredBuschJ
                          JaredBusch
                          last edited by JaredBusch

                          Line 109 needs commented out.

                          0_1456293589646_upload-722d8a55-ede0-467f-815e-97aca00bde17

                          add this right after the yum install to fix the firewall.

                          yum -y install wget firewalld epel-release
                          systemctl enable firewalld
                          systemctl start firewalld
                          yum -y install nginx httpd-tools unzip
                          

                          I would just remove line 109 it serves no purpose.

                          Edit: Some dumbass forgot to snapshot the image so he could repeat the install...

                          scottalanmillerS 1 Reply Last reply Reply Quote 0
                          • JaredBuschJ
                            JaredBusch
                            last edited by JaredBusch

                            Why lock out with .htaccess? There is no hint what is needed to log in here.

                            0_1456293980571_upload-16198c5d-89fa-48ac-a702-3d6b2cc05644

                            I hate this level of authentication.

                            Using kibanauser and the password I chose, results in Kibana setup.
                            0_1456294107205_upload-68bec54b-23aa-4026-95d9-8080cfed408d

                            scottalanmillerS 1 Reply Last reply Reply Quote 0
                            • scottalanmillerS
                              scottalanmiller @JaredBusch
                              last edited by

                              @JaredBusch said:

                              @scottalanmiller so what do you setup your disk partitioning like in CentOS 7?

                              If I'm doing this for product, I do 20GB for the OS and 200GB+ on a second VHD for the data. I put it all under LVM and make a XFS filesystem on the secondary mount and mount it to data and make a symlink for the Elasticsearch database directory into there.

                              JaredBuschJ 1 Reply Last reply Reply Quote 0
                              • scottalanmillerS
                                scottalanmiller @JaredBusch
                                last edited by

                                @JaredBusch said:

                                Why lock out with .htaccess? There is no hint what is needed to log in here.

                                It's how Digital Ocean does it as well. Kibana doesn't have a built in authentication scheme that I know of. HTAccess is very simple for someone to just get started.

                                1 Reply Last reply Reply Quote 0
                                • scottalanmillerS
                                  scottalanmiller
                                  last edited by

                                  And simple to remove when you want to move to something else.

                                  1 Reply Last reply Reply Quote 0
                                  • scottalanmillerS
                                    scottalanmiller @JaredBusch
                                    last edited by

                                    @JaredBusch said:

                                    Line 109 needs commented out.

                                    0_1456293589646_upload-722d8a55-ede0-467f-815e-97aca00bde17

                                    add this right after the yum install to fix the firewall.

                                    yum -y install wget firewalld epel-release
                                    systemctl enable firewalld
                                    systemctl start firewalld
                                    yum -y install nginx httpd-tools unzip
                                    

                                    I would just remove line 109 it serves no purpose.

                                    Edit: Some dumbass forgot to snapshot the image so he could repeat the install...

                                    Thanks. That was formatting I had originally put in before scripting it.

                                    1 Reply Last reply Reply Quote 0
                                    • scottalanmillerS
                                      scottalanmiller @JaredBusch
                                      last edited by

                                      @JaredBusch said:

                                      Looks like maybe you forgot to start firewalld?

                                      Fixed

                                      1 Reply Last reply Reply Quote 0
                                      • JaredBuschJ
                                        JaredBusch @scottalanmiller
                                        last edited by

                                        @scottalanmiller said:

                                        @JaredBusch said:

                                        @scottalanmiller so what do you setup your disk partitioning like in CentOS 7?

                                        If I'm doing this for product, I do 20GB for the OS and 200GB+ on a second VHD for the data. I put it all under LVM and make a XFS filesystem on the secondary mount and mount it to data and make a symlink for the Elasticsearch database directory into there.

                                        SO this mean you need to make one of your linux admin setrups on drive settings because that is not what CentOS does by dfault.

                                        DashrenderD 1 Reply Last reply Reply Quote 0
                                        • DashrenderD
                                          Dashrender @JaredBusch
                                          last edited by

                                          @JaredBusch said:

                                          @scottalanmiller said:

                                          @JaredBusch said:

                                          @scottalanmiller so what do you setup your disk partitioning like in CentOS 7?

                                          If I'm doing this for product, I do 20GB for the OS and 200GB+ on a second VHD for the data. I put it all under LVM and make a XFS filesystem on the secondary mount and mount it to data and make a symlink for the Elasticsearch database directory into there.

                                          SO this mean you need to make one of your linux admin setrups on drive settings because that is not what CentOS does by dfault.

                                          Would CentOS do what Scott does if you had two drives you provide CentOS to use? i.e. a 20 GB and a 200+ GB one? Would CentOS install the OS and everything fully on the 20, and then just mount the 200 on some point?

                                          JaredBuschJ 1 Reply Last reply Reply Quote 0
                                          • JaredBuschJ
                                            JaredBusch @Dashrender
                                            last edited by

                                            @Dashrender said:

                                            @JaredBusch said:

                                            @scottalanmiller said:

                                            @JaredBusch said:

                                            @scottalanmiller so what do you setup your disk partitioning like in CentOS 7?

                                            If I'm doing this for product, I do 20GB for the OS and 200GB+ on a second VHD for the data. I put it all under LVM and make a XFS filesystem on the secondary mount and mount it to data and make a symlink for the Elasticsearch database directory into there.

                                            SO this mean you need to make one of your linux admin setrups on drive settings because that is not what CentOS does by dfault.

                                            Would CentOS do what Scott does if you had two drives you provide CentOS to use? i.e. a 20 GB and a 200+ GB one? Would CentOS install the OS and everything fully on the 20, and then just mount the 200 on some point?

                                            Than answer is not by default. It tries to make it's own magic.

                                            You can see here I created a 20gb and a 200GB vhdx and told the install to handle it all for me.

                                            0_1456324763628_upload-a8516602-b5e4-4c0e-9112-caabbb970b80

                                            Guess what, you still end up with a 50GB and a 170GB partitions scheme

                                            [root@elk ~]# df -h
                                            Filesystem                   Size  Used Avail Use% Mounted on
                                            /dev/mapper/centos_elk-root   50G  882M   50G   2% /
                                            devtmpfs                     906M     0  906M   0% /dev
                                            tmpfs                        916M     0  916M   0% /dev/shm
                                            tmpfs                        916M  8.3M  907M   1% /run
                                            tmpfs                        916M     0  916M   0% /sys/fs/cgroup
                                            /dev/sda2                    494M   99M  395M  21% /boot
                                            /dev/sda1                    200M  9.5M  191M   5% /boot/efi
                                            /dev/mapper/centos_elk-home  168G   33M  168G   1% /home
                                            tmpfs                        184M     0  184M   0% /run/user/0
                                            [root@elk ~]#
                                            
                                            1 Reply Last reply Reply Quote 0
                                            • 1
                                            • 2
                                            • 3
                                            • 1 / 3
                                            • First post
                                              Last post