Beats – quick settings

Before first run of any beat from elastic.co is good to set up number of replicas and shards based on your environment (e.g. number of nodes).

settings: {
  index.number_of_replicas: 0,
  index.number_of_shards:0
}

 

Elasticsearch cluster security

Access control

Never ever run elasticsearch as root!

Scripts

script.inline: false  
script.stored: false  
script.file:   true

Script.inline enable run scripts provided inline in the API. Script.stored enable run stored script using API. Script.file  can run script which are stored in filesystem ( /etc/elasticsearch/scripts (rpm or deb), config/scripts (zip, tar).

Read more about scripting in elasticsearch e.g. Java security policy https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-scripting-security.html

 

Allow_explicit_index

rest.action.multi.allow_explicit_index: false

Elasticsearch will reject now multi-search, multi-get and bulk request with explicit index in body.

https://www.elastic.co/guide/en/elasticsearch/reference/current/url-access-control.html

Filtered alias

For specification and  for better user expirience it is possible to use filtred alias.

POST /_aliases
{
    "actions" : [
        { "add" : { "indices" : ["test1", "test2"], "alias" : "alias1" } }
    ]
}

https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-aliases.html#filtered

Service stability:

https://www.digitalocean.com/community/tutorials/how-to-set-up-a-production-elasticsearch-cluster-on-ubuntu-14-04

Sources:

https://www.elastic.co/guide/en/cloud/current/security.html

https://sematext.com/blog/2017/01/18/elasticsearch-security-authentication-encryption-backup/

https://www.opsdash.com/blog/howto-setup-elasticsearch-secure.html

 

 

Zabezpečení Elasticsearch pomocí SearchGuardu

Searchguard

Elasticsearch scripts

cd /opt/elasticsearch
sudo bin/elasticsearch-plugin install -b com.floragunn:search-guard-5:5.3.0-11
vim /etc/elasticsearch/elasticsearch.yml

Searchguard settings in elasticsearch.yml

security.manager.enabled: false

http://floragunncom.github.io/search-guard-ssl-docs/installation.html

root@e58127d1d54e:/opt/elasticsearch# plugins/search-guard-5/tools/sgadmin.sh -cd plugins/search-guard-5/sgconfig/ -ks /etc/elasticsearch/node-1-keystore.jks -kspass 95ba06bb222fd7640283 -ts /etc/elasticsearch/truststore.jks -tspass 25cb9058f1b53dd61c69 -nhnv
 Search Guard Admin v5
 Will connect to localhost:9300 ... done
 Contacting elasticsearch cluster 'elasticsearch' and wait for YELLOW clusterstate ...
 Clustername: elasticsearch
 Clusterstate: YELLOW
 Number of nodes: 1
 Number of data nodes: 1
 searchguard index does not exists, attempt to create it ... done (auto expand replicas is on)
 Populate config from /opt/elasticsearch/plugins/search-guard-5/sgconfig
 Will update 'config' with plugins/search-guard-5/sgconfig/sg_config.yml
 SUCC: Configuration for 'config' created or updated
 Will update 'roles' with plugins/search-guard-5/sgconfig/sg_roles.yml
 SUCC: Configuration for 'roles' created or updated
 Will update 'rolesmapping' with plugins/search-guard-5/sgconfig/sg_roles_mapping.yml
 SUCC: Configuration for 'rolesmapping' created or updated
 Will update 'internalusers' with plugins/search-guard-5/sgconfig/sg_internal_users.yml
 SUCC: Configuration for 'internalusers' created or updated
 Will update 'actiongroups' with plugins/search-guard-5/sgconfig/sg_action_groups.yml
 SUCC: Configuration for 'actiongroups' created or updated
 Done with success

SearchGuard + Kibana + security

Logstash 5.4.0 – offline filter plugins

Exportovaný zip pluginů obsahuje tyto přidané pluginy:

  • logstash-filter-aggregate
  • logstash-filter-de_dot
  • logstash-filter-json_encode
  • logstash-input-jmx

Pluginy lze nainstalovat pomocí tohoto příkazu:

bin/logstash-plugin install file:///<path_to_zip>/logstash-offline-plugins-5.4.0.zip

logstash-offline-plugins-5.4.0.zip

Zdroje:

https://www.elastic.co/guide/en/logstash/current/offline-plugins.html#installing-offline-packs

 

boot2docker – Elasticsearch a max_map_count

Pokud používáte boot2docker, tak určitě narazíte při startu Elasticsearch služby na podobnou chybu:

Exception in thread "main" java.lang.RuntimeException: bootstrap checks failed
 initial heap size [268435456] not equal to maximum heap size [1073741824]; this can cause resize pauses and prevents mlockall from locking the entire heap
 max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]
 at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:93)
 at org.elasticsearch.bootstrap.BootstrapCheck.check(BootstrapCheck.java:66)
 at org.elasticsearch.bootstrap.Bootstrap$5.validateNodeBeforeAcceptingRequests(Bootstrap.java:191)
 at org.elasticsearch.node.Node.start(Node.java:323)
 at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:206)
 at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:269)
 at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:111)
 at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:106)
 at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:91)
 at org.elasticsearch.cli.Command.main(Command.java:53)
 at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:74)
 at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67)
 Refer to the log for complete error details.

Řešení:
Stačí upravit /var/lib/boot2docker/profile přídáním tohoto řádku na konec souboru a následně udělat reboot.

 # Update the vm.max_map_count setting
 sysctl -w vm.max_map_count=262144

Elasticsearch docker container na Ubuntu 16.04

Pokud používáte sebp/elk image, můžete se setkat s chybou

2017-04-05T20:57:53,130][ERROR][o.e.b.Bootstrap          ] [IMZwJd9] node validation exception
bootstrap checks failed
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

Řešení:

sudo sysctl -w vm.max_map_count=262144

nebo

echo 262144 | sudo tee /proc/sys/vm/max_map_count

Permanentní nastavení:

Do /etc/sysctl.conf přidáme parametr

 vm.max_map_count=1048575

Poté je nutné restartovat server nebo pustit

sysctl -p

.

http://elk-docker.readthedocs.io/#troubleshooting

 

Docker na Ubuntu 16.04

Pro rozběhnutí Docker na Ubuntu, lze bez problémů postupovat podle oficiální dokumentace.

https://docs.docker.com/engine/installation/linux/ubuntu/#install-using-the-repository

Pro moje potřeby jsem se nepouštěl do testování trial verze Docker EE, ale rovnou zůstal u Docker CE.

Po úspěšné instalaci jsem, pak ještě nastavil prostředí Ubuntu pro běh Docker jako non-root user. Opět jsem postupoval podle oficiální dokumentace.

https://docs.docker.com/engine/installation/linux/linux-postinstall/

A nezapomenout restartovat stroj!