While working on a common issue that I have been repeatedly experiencing, I started thinking about if there could be a better way to tackle it. Good code should be flexible enough so that it is adaptable to many different situations and not hard-coded for a specific scenario (unless you a tweaking for performance).
Anyone using Lucene-based systems (solr, elasticsearch, zoie) knows that a full re-index is required if the document mapping changes. In elasticsearch, I wrote/executed the same re-indexing code over and over again since my mappings were constantly evolving. Surely there had to be a better way. One nice feature of elasticsearch is the concept of plugins, which allow users of the system to add code to elasticsearch without having to modify the original source code. A re-index plugin would be easy to create, but the code would still not be accessible remotely. Would it be possible to create a new REST endpoint to access the new functionality?
There is little written about elasticsearch plugins, but armed with the source code, I was able to dig in and find out that it is not only indeed possible, but it is also actually quite easy. Here is a quick tutorial on how to do so. We'll create a simple hello, world interface.
First up is the all important es-plugin.properties file. If this file is found as a resource within the classpath (defined from all the jars in the plugin zip file), the file is read and is used to bootstrap the system. The format is as follows:
plugin=org.elasticsearch.plugin.helloworld.HelloWorldPlugin
The file contains a single "plugin" property which defines the main plugin class to be instantiated. Only one plugin class may be defined inside the es-plugin.properties file. The class defined by the plugin property must be a subclass of org.elasticsearch.plugins.Plugin. When the plugin class is instantiated, each module defined in the system is past to it via the processModule(Module module) method. Most modules can be ignored, the one we are interested is the RestModule. The RestModule contains all REST actions in the system. We can now add our not-yet-created REST action.
public class HelloWorldPlugin extends AbstractPlugin {
public String name() {
return "hello-world";
}
public String description() {
return "Hello World Plugin";
}
@Override public void processModule(Module module) {
if (module instanceof RestModule) {
((RestModule) module).addRestAction(HelloWorldAction.class);
}
}
}
Each REST action must be a subclass of BaseRestHandler. When the class is instantiated, the elasticsearch client and RestController are passed in via constructor injection via Google Guice. The restController is where will define the actual endpoints.
@Inject public HelloWorldAction(Settings settings, Client client, RestController controller) {
super(settings, client);
// Define REST endpoints
controller.registerHandler(GET, "/_hello/", this);
controller.registerHandler(GET, "/_hello/{name}", this);
}
One word of advice is to prepend all new endpoints with an underscore '_' in order to not confuse them with actual indices.
From there we can implement the handleRequest method, which handles the request. Our simple example will simply return the first parameter passed in via the url or "world" if not.
public void handleRequest(final RestRequest request, final RestChannel channel) {
logger.debug("HelloWorldAction.handleRequest called");
String name = request.hasParam("name") ? request.param("name") : "world";
try {
XContentBuilder builder = restContentBuilder(request);
builder.startObject().field(new XContentBuilderString("hello"), name).endObject();
channel.sendResponse(new XContentRestResponse(request, OK, builder));
} catch (IOException e) {
onFailure(channel, request, e);
}
}
public void onFailure(RestChannel channel, RestRequest request, Throwable e) {
try {
channel.sendResponse(new XContentThrowableRestResponse(request, e));
} catch (IOException e1) {
logger.error("Failed to send failure response", e1);
}
}
After installing the plugin (please visit the code on Github for building and installation instructions). We can now test the plugin.
$ curl -XGET http://localhost:9200/_hello/mike
{"hello":"mike"}
This example does not interact at all with the underlying system. Let's come up with another simple (and contrived) example that will issue a GET request with the same name parameter passed in.
public void handleRequest(final RestRequest request, final RestChannel channel) {
logger.debug("HelloWorldAction.handleRequest called");
final String name = request.hasParam("name") ? request.param("name") : "world";
final GetRequest getRequest = new GetRequest(INDEX, TYPE, name);
getRequest.listenerThreaded(false);
getRequest.operationThreaded(true);
String[] fields = {"msg"};
getRequest.fields(fields);
client.get(getRequest, new ActionListener<GetResponse>() {
@Override public void onResponse(GetResponse response) {
try {
XContentBuilder builder = restContentBuilder(request);
GetField field = response.field("msg");
String greeting = (field!=null) ? (String)field.values().get(0) : "Sorry, do I know you?";
builder
.startObject()
.field(new XContentBuilderString("hello"), name)
.field(new XContentBuilderString("greeting"), greeting)
.endObject();
if (!response.exists()) {
channel.sendResponse(new XContentRestResponse(request, NOT_FOUND, builder));
} else {
channel.sendResponse(new XContentRestResponse(request, OK, builder));
}
} catch (Exception e) {
onFailure(e);
}
}
@Override public void onFailure(Throwable e) {
try {
channel.sendResponse(new XContentThrowableRestResponse(request, e));
} catch (IOException e1) {
logger.error("Failed to send failure response", e1);
}
}
});
}
Please note most error checking is not done for reasons of brevity. Always check your values! Also for this example, the index and type names are hardcoded.
Reinstall the plugin and restart elasticsearch. Next, create the test index and add a value.
curl -XPUT 'http://localhost:9200/example/'
curl -XPUT http://localhost:9200/example/person/dave -d '{
"msg" : "Affirmative, Dave. I read you."
}'
Now we can query the data
$ curl -XGET http://localhost:9200/_hello/dave
{"hello":"dave","greeting":"Affirmative, Dave. I read you."}
$ curl -XGET http://localhost:9200/_hello/susan
{"hello":"susan","greeting":"Sorry, do I know you?"}
Although only one plugin can be defined by plugin file, multiple actions can be added in the processModule(Module module) method.
Complete code and instructions can be found at https://github.com/brusic/elasticsearch-hello-world-plugin/
I was always wondering how will I create a plugin in ES !
ReplyDeletethat's a great tutorial.
Thanks.
David.
Thank you so much for sharing your knowledge!
ReplyDelete