Backup Elasticsearch on Kubernetes
You can use the instructions on this page to create pre and post backup rules with Portworx Backup, which take application-consistent backups for Elasticsearch on Kubernetes in production.
The Elasticsearch data directory is used prevent permanent data loss and is typically located at /user/share/elasticsearch/data
inside the Kubernetes pod. This will also enable Portworx Backup to back up and restore the data stored in this location.
Before using this guide, make sure and configure PVCs for elasticsearch-data
. Use the below file as an example.
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: elasticsearch
spec:
version: 7.8.0
nodeSets:
- name: default
count: 3
podTemplate:
metadata:
labels:
appname: "elastisearch-app"
spec:
containers:
- name: elasticsearch
volumeMounts:
- name: elasticsearch-backups
mountPath: /usr/share/elasticsearch/backups
volumes:
- name: elasticsearch-backups
persistentVolumeClaim:
claimName: elasticsearch-backups
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
storageClassName: elastic-pwx-storage-class
config:
node.master: true
node.data: true
node.ingest: true
node.store.allow_mmap: false
path.repo: ["/usr/share/elasticsearch/backups"]
Elasticsearch can also create a snapshot repository to store index snapshots produced by the internal snapshot and restore API. For this, the above template uses a Portworx shared volume and named elasticsearch-backups
to create a shared file-system across all Elasticsearch nodes to be used to store the index snapshots.
Installation
Prerequisites
If you have not used Portworx, Portworx Backup, or run Data services on Kubernetes it is a good idea to review the following information:
- Install Portworx on Kubernetes
- Application installation
- Install Portworx Backup
- Add cluster to Portworx Backup
- Creating backup rules
NOTE:
- The template above can not be used alone. Please follow prerequisites from the following elastic on kubernetes operations guide.
- The rules below will use a username
elastic
and password specific to the environment. You will need to modify the rule to use your username and password for your environment. - You should mount a PVC to the elasticsearch data directory. This is typically located at
/user/share/elasticsearch/data
.
Create rules for Elasticsearch
Create rules for Elasticsearch that will run both before and after the backup operation runs:
Create a pre-exec backup rule for Elasticsearch
For the pre-backup rule you will create a rule that performs multiple actions.
- Freeze the index
- Flush all indexes in Elasticsearch
- Create an Elasticsearch index snapshot of all indexes
Create the rule.
- Navigate to Settings → Rules → Add New.
- Add a name for your Rule.
Add the following app label:
appname=elasticsearch-app
Add the following action:
curl -X POST -u "elastic:<password>" -k "https://elasticsearch-es-http:9200/customer/_freeze&pretty"
Add the following additional action:
curl -X PUT -u "elastic:<password>" -k "https://elasticsearch-es-http:9200/all/_flush&pretty"
Add the following additional action:
curl -X PUT -u "elastic:<password>" -k "https://elasticsearch-es-http:9200/_snapshot/es_backups/%3Csnapshot-$(uuidgen)-%7Bnow%2Fd%7D%3E?wait_for_completion=true&pretty"
Create a post-exec backup rule for Elasticsearch
Performing the _freeze
and _flush
operations on your index before calling the snapshot API ensures flexible and accurate restores by making sure the database isn’t currently being written into.
Since you performed a _freeze
operation when you created a backup, you must create a post exec rule to perform _unfreeze
operation. The steps below create a post-exec rule which runs _unfreeze
on the customer
index:
- Navigate to Settings → Rules → Add New.
- Add a name for your Rule.
Add the following app label:
appname=elasticsearch-app
Add the following action:
curl -X POST -u "elastic:<password>" -k "https://elasticsearch-es-http:9200/customer/_unfreeze&pretty"
Associate the pre and post-exec rules with the Elasticsearch backup
Create a backup for your Elasticsearch application. Select the rules you created in the steps above from the pre-exec and post-exec dropdowns:
Once you have filled out the backup form, click Create
Demo
Watch this short demo of the above information.