Elasticsearch

When you want to search for data within a certain period in Kibana, the first thing that comes to mind is the built-in Time Picker. It is really handy. But in some cases your data may have more than one date field, and you have to filter with a query instead of the Time Picker. For example, with email data: sending, opening, and clicking each send a log to my server. That gives you two times, one for when the email was sent and one for when the event happened (such as opening or clicking). In a case like this, you may need to search on a date field in the search box. ...

If you found this article through a search engine, you probably feel the same resentment toward X-Pack that I do. At first I thought X-Pack was a free, handy plugin, so I installed it on both Elasticsearch and Kibana. Then 30 days later a message appeared saying the license had expired and I needed to “renew the license key,” otherwise I couldn’t log into Kibana. Up to that point I still assumed I just needed to register an account to get a new license key. But after browsing the official site for a long time, I couldn’t find any text about licensing, pricing, or purchasing (later I found out it does exist — in tiny print… do you people even want to make money?). In the end I only found the pricing page through a Google search. ...

After you install Elasticsearch and Kibana, you will quickly run into a serious problem: “Wait, where are the username and password?” That’s right, Elasticsearch and Kibana don’t come with the usual user management module. You can only restrict access by binding them to an internal IP or port. But if you’re like my company and don’t have a fixed IP, or you want to set up several accounts, keep reading. This post shares how to set up basic user authentication for Elasticsearch and Kibana using HTTP Auth. ...

If you found this article, you have probably been searching for how to use Split in an Elasticsearch (or Kibana) Script Field for a long time! I deliberately included every keyword I could think of, so that engineers who suffer like I did do not waste too much of their lives on this… To cut to the chase: although Painless claims to support the Java Native API, a few things are not supported, and one of them is split. The story goes that the Elasticsearch developers felt the time complexity of split was too poor, so they cut it out. I did not know this at first, and I tried all kinds of approaches, from doubting whether my syntax was correct to wondering whether some special syntax was needed to import the Java API. Finally, on an Elasticsearch Issue, I found this post: [Painless - Request for native String split function], and my feelings at that moment were…… ...

When you work with Elasticsearch in Python, you will inevitably install the official elasticsearch.py library. Recently, while using Python to receive email logs and write them into Elasticsearch, I picked up quite a few lessons worth sharing. As everyone knows, AWS Lambda charges by execution time, so when writing Lambda functions every second counts. Anything that ran over 0.5 seconds had me in agony. So I went from inserting records one at a time with .index, to using streaming_bulk, and finally, unable to stand it anymore, mixing parallel_bulk with streaming_bulk. Here I will talk about the differences between these approaches and the pitfalls I ran into! ...