<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sreekanth T on Medium]]></title>
        <description><![CDATA[Stories by Sreekanth T on Medium]]></description>
        <link>https://medium.com/@sreekantht?source=rss-78021763f6f3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*vS2J10sAMAjgVa4e.jpeg</url>
            <title>Stories by Sreekanth T on Medium</title>
            <link>https://medium.com/@sreekantht?source=rss-78021763f6f3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 12:47:37 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sreekantht/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Indexing log files to Solr using Logstash]]></title>
            <link>https://medium.com/@sreekantht/indexing-log-files-to-solr-using-logstash-5aad3aa3dba?source=rss-78021763f6f3------2</link>
            <guid isPermaLink="false">https://medium.com/p/5aad3aa3dba</guid>
            <category><![CDATA[logs]]></category>
            <category><![CDATA[logstash]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[solr]]></category>
            <category><![CDATA[log-analytics]]></category>
            <dc:creator><![CDATA[Sreekanth T]]></dc:creator>
            <pubDate>Sat, 26 Aug 2017 15:38:53 GMT</pubDate>
            <atom:updated>2019-09-06T05:14:40.513Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_TnYpNvvDGVdwr5USaJBbA.jpeg" /></figure><p>Recently I got a requirement to create a log dashboard for different Hadoop applications and component. By default the tool for parsing log files was Logstash. Usually people use Logstash together with Elastic Search for storage and Kibana for visualizations. But in our case there was slight chnage. Our team has already been using Solr as the search engine, so we needed to use the same setup and instead of Kibana we had to use Banana, a fork of Kibana.</p><p>Everything was fine when started. Logstash has its own output plugin for indexing the data to Solr, named logstash-output-solr_http plugin. But it was not working, the logs were not getting indexed to Solr collection. After hours of head scratching and numerous coffees finally I figured out the issue and successfully indexed the data into Solr.</p><p>In this tutorial I will try to explain the steps I followed to index the log files to Solr collection. I have used linux machines running on Red Hat 7 for this tutorial.</p><p><strong>Installing Logstash</strong></p><p>Before installing Logstash make sure that you have Java installed in your machine.</p><p><em>sudo yum install java-1.8.0-openjdk<br>curl -L -O </em><a href="https://artifacts.elastic.co/downloads/logstash/logstash-5.5.0.rpm"><em>https://artifacts.elastic.co/downloads/logstash/logstash-5.5.0.rpm</em></a><em><br>sudo rpm -i logstash-5.5.0.rpm</em></p><p>The below page explains Logstash installation for other platforms.</p><p><a href="https://www.elastic.co/guide/en/logstash/current/installing-logstash.html">Installing Logstash</a></p><p><strong>Installing Solr</strong></p><p>The next step after installing Logstash is to install Solr. I have installed Solr in a different server but it can be installed on the same server as Logstash.</p><p>For installing Solr you have to create a user other than root user.</p><p><em>wget </em><a href="http://mirror.reverse.net/pub/apache/lucene/solr/6.6.0/solr-6.6.0.tgz"><em>http://mirror.reverse.net/pub/apache/lucene/solr/6.6.0/solr-6.6.0.tgz</em></a><em><br>tar -xvzf solr-6.6.0.tgz</em></p><p><a href="https://cwiki.apache.org/confluence/display/solr/Installing+Solr">Installing Solr - Apache Solr Reference Guide - Apache Software Foundation</a></p><p><strong>Installing solr_http plugin</strong></p><p>Logstash doesnt support indexing to Solr out of the box. You will have to install the official Logstash plugin for Solr names logstash-output-solr_http. To install plugin logstash has its own utility. Go to the logstash installation directory and run the following command.</p><p><em>logstash-plugin install logstash-output-solr_http</em></p><p>The current version of this plugin has few bugs. We will discuss how to fix those and make this plugin work.</p><p><strong>Installing Rsolr 1.1.2</strong></p><p>Logstash will use a ruby library called Rsolr for connecting to Solr. Logstash will install the latest version of Rsolr, but latest Rsolr has a bub with timestamp datatype which will prevent indexing to Solr. To prevent this we will have to manually install the working version of Rsolr. Currently the latest working one is Rsolr 1.1.2.</p><p><em>gem install rsolr -v 1.1.2</em></p><p><strong>Updating Logstash Solr_http plugin</strong></p><p>Once Rsolr 1.1.2 is installed make a note of the path on which the gem file got installed. We have to update this path in the solr_http plugin. Open the solr_http.rb file present in the following directory.</p><p><em>/[logstash_installation_dir]/vendor/bundle/jruby/1.9/gems/logstash-output-solr_http-3.0.1/lib/logstash/outputs</em></p><p>Open this file in a text editor and comment <em>require “rsolr” </em>and<em> </em>add the following line</p><p><em>require “/[path/to]/gems/rsolr-1.1.2/lib/rsolr”</em></p><p>The default gem installation path can be obtained by the following command.</p><p><em>gem environment</em></p><p>On the 67th line of solr_http.rb file comment the <em>.iso8601</em> part. Then comment the line <a href="http://twitter.com/solr"><em>@solr</em></a><em>.add(documents) and </em>add the below line to enable the documents to be committed after adding them to the collection.</p><p><a href="http://twitter.com/solr"><em>@solr</em></a><em>.add(documents, :add_attributes =&gt; {:commitWithin=&gt;1000})</em></p><p>Working version of the file can be accessed from the below github repo.</p><p><a href="https://github.com/sreekanthtdhar/logstash-output-solr_http/blob/master/lib/logstash/outputs/solr_http.rb">sreekanthtdhar/logstash-output-solr_http</a></p><p><strong>Creating the Logstash config file</strong></p><p>Now you can create the Logstash Configuration file with the details of input file directory and output details. Here we wont try to add any filter as we want to index the complete log messages.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/197fe277ddf837f64a895ba4b7d47d25/href">https://medium.com/media/197fe277ddf837f64a895ba4b7d47d25/href</a></iframe><p>Logz.io has a good tutorial on creating grok patterns for fetching speciific fields of a log.</p><p><a href="https://logz.io/blog/logstash-grok/">A Beginner&#39;s Guide to Logstash Grok | Logz.io</a></p><p><strong>Starting Solr nodes and creating a collection</strong></p><p>Now we have completed the logstash setup and created the conf file. Next step is to start the Solr nodes and then create an empty collection to index the log messages. Make sure that the collection name mentioned in the Logstash conf file and the one you are going to create are same.</p><p><em>bin/solr stop -e cloud -noprompt</em></p><p>Now we can create a data driven collection using the below command.</p><p><em>bin/solr create -c collectionname -d server/solr/configsets/data_driven_schema_configs/</em></p><p><strong>Starting Logstash</strong></p><p>Its time to start our Logstash service with our custom conf file.</p><p><em>logstash -f logstash_solr.conf</em></p><p>Logstash will start and read the log files present in the input directory. Each line will be indexed to the Solr collection with a number of metadata fields.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5aad3aa3dba" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>