Elasticsearch Reindex를 사용해 신규 index 생성

2022.07.12 - [docker] - Docker + Elasticsearch, Kibana 구성 (single node)

Docker + Elasticsearch, Kibana 구성 (single node)

 

기존 인덱스에 저장되어 있는 특정 데이터와 필드의 데이터를 커스텀해서 신규 인덱스에 담아야 하는 업무가 생겨서 Reindex를 사용했고 해당 내용을 정리해보려고 합니다.

 

Reindex란?

POST _reindex
{
  "source": {
    "index": "old-index"
  },
  "dest": {
    "index": "new-index"
  }
}

POST _reindex
{
  "source": {
    "remote": {
      "host": "http://remotees:9200",
      "username": "username",
      "password": "password"
    },
    "index": "old_index"
  },
  "dest": {
    "index": "new_index"
  }
}

기존에 존재하는 old-index의 document들을 new-index에 색인하는 것입니다.

 

Reindex를 사용 하는 이유?

mapping, shard 정보를 수정하거나 사전을 변경해 새롭게 색인이 필요할떄 사용

 

사용해보기

old_index 생성 (prod-old)

PUT /prod-old
{
  "mappings": {
    "properties": {
      "productCode": {
      	"type": "keyword"
      },
      "groupCode": {
      	"type": "integer"
      },
      "price": {
      	"type": "integer"
      },
      "productName": {
      	"type": "text",
      	"fields": {
          "field": {
            "type": "keyword"	
          }
      	}
      },
      "maker": {
        "type": "keyword"	
      }
    }
  }
}

POST _bulk
{"index":{"_index":"prod-old"}}
{"productCode":"1", "groupCode": 1,"price": 1500000,"productName":"한성 노트북","maker":"한성"}
{"index":{"_index":"prod-old"}}
{"productCode":"2", "groupCode": 1,"price": 1500000,"productName":"레노버 노트북","maker":"레노버"}
{"index":{"_index":"prod-old"}}
{"productCode":"3", "groupCode": 1,"price": 1500000,"productName":"삼성 노트북","maker":"삼성"}
{"index":{"_index":"prod-old"}}
{"productCode":"4", "groupCode": 1,"price": 1500000,"productName":"lg 노트북","maker":"lg"}
{"index":{"_index":"prod-old"}}
{"productCode":"5", "groupCode": 2,"price": 500000,"productName":"커피 포트","maker":"스타벅스"}

GET prod-old/_search
{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 5,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "prod",
        "_type" : "_doc",
        "_id" : "4fX01YMBq1p887f0yyID",
        "_score" : 1.0,
        "_source" : {
          "productCode" : "1",
          "groupCode" : 1,
          "price" : 1500000,
          "productName" : "한성 노트북",
          "maker" : "한성"
        }
      },
      ...

 

new_index 생성 (prod-new)

PUT /prod_new
{
  "settings": {
    "number_of_shards": 3
  },
  "mappings": {
    "properties": {
      "productCode": {
      	"type": "keyword"
      },
      "groupCode": {
      	"type": "integer"
      },
      "price": {
      	"type": "integer"
      },
      "productName": {
      	"type": "text",
      	"fields": {
          "field": {
            "type": "keyword"	
          }
      	}
      }
    }
  }
}

 

pipline 구성

2022.07.19 - [ELK] - Elasticsearch Ingest Node PipeLine 사용해 보기

Elasticsearch Ingest Node PipeLine 사용해 보기

PUT /_ingest/pipeline/prod-pipline
{
  "description": "prod pipline",
  "processors": [
    {
      "set": {
        "field": "_id",
        "value": "{{productCode}}"
      }
    }
  ]
}

 

Reindex

POST /_reindex?wait_for_completion=false&slices=auto
{
  "source": {
    "index": "prod-old",
    "_source": {
      "includes": ["productCode", "groupCode", "price", "productName"]
    },
    "query": {
      "bool": {
        "filter": [
          {
            "term": {
              "groupCode": 1
            }
          }
        ]
      }
    }
  },
  "dest": {
    "index": "prod-new",
    "pipeline": "prod-pipline"
  }
}


결과
{
  "task" : "R8gQaiWuRKyzGVbEA-oBdA:751579"
}

 

prod-new 조회

{
  "took" : 17,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 4,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "prod-new",
        "_type" : "_doc",
        "_id" : "1",
        "_score" : 1.0,
        "_source" : {
          "productCode" : "1",
          "price" : 1500000,
          "productName" : "한성 노트북",
          "groupCode" : 1
        }
      },
      {
        "_index" : "prod-new",
        "_type" : "_doc",
        "_id" : "2",
        "_score" : 1.0,
        "_source" : {
          "productCode" : "2",
          "price" : 1500000,
          "productName" : "레노버 노트북",
          "groupCode" : 1
        }
      },
      {
        "_index" : "prod-new",
        "_type" : "_doc",
        "_id" : "3",
        "_score" : 1.0,
        "_source" : {
          "productCode" : "3",
          "price" : 1500000,
          "productName" : "삼성 노트북",
          "groupCode" : 1
        }
      },
      {
        "_index" : "prod-new",
        "_type" : "_doc",
        "_id" : "4",
        "_score" : 1.0,
        "_source" : {
          "productCode" : "4",
          "price" : 1500000,
          "productName" : "lg 노트북",
          "groupCode" : 1
        }
      }
    ]
  }
}

 

Reindex 옵션

wait_for_completion=false
- 비동기 실행
- task id로 실행한 명령 상태 확인

GET /_tasks/taskID
{
  "completed" : true,
  "task" : {
    "node" : "R8gQaiWuRKyzGVbEA-oBdA",
    "id" : 751579,
    "type" : "transport",
    "action" : "indices:data/write/reindex",
    "status" : {
      "total" : 7,
      "updated" : 6,
      "created" : 1,
      "deleted" : 0,
      "batches" : 1,
      "version_conflicts" : 0,
      "noops" : 0,
      "retries" : {
        "bulk" : 0,
        "search" : 0
      },
      "throttled_millis" : 0,
      "requests_per_second" : -1.0,
      "throttled_until_millis" : 0
    },
    "description" : "reindex from [prod-old] to [prod-new][_doc]",
    "start_time_in_millis" : 1665743049499,
    "running_time_in_nanos" : 316173000,
    "cancellable" : true,
    "headers" : { }
  },
  "response" : {
    "took" : 315,
    "timed_out" : false,
    "total" : 7,
    "updated" : 6,
    "created" : 1,
    "deleted" : 0,
    "batches" : 1,
    "version_conflicts" : 0,
    "noops" : 0,
    "retries" : {
      "bulk" : 0,
      "search" : 0
    },
    "throttled" : "0s",
    "throttled_millis" : 0,
    "requests_per_second" : -1.0,
    "throttled_until" : "0s",
    "throttled_until_millis" : 0,
    "failures" : [ ]
  }
}

- task 취소

POST /_tasks/taskID/_cancle

slices=auto
- 병렬로 작업할 수를 결정.
- 해당 인덱스에 있는 샤드 수와 동일한 수일 경우 성능이 가장 좋음 
- auto 설정이 샤드 수와 같음