Wednesday, October 25, 2017

Run shell script using Oozie job

In this I will show how to automate shell scripts using Oozie framework.

First create job.properties file.

1
2
3
4
5
6
nameNode=hdfs://localhost:8020
jobTracker=localhost:8050
queueName=default
oozie.wf.application.path=/<Hdfs Path>/sampleworkflow.xml
oozie.use.system.libpath=true
oozie.libpath=<shared lib path on Hdfs>

Create sampleworkflow.xml and place it in Hdfs.
1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
<workflow-app name="shell-wf" xmlns="uri:oozie:workflow:0.4">
 <start to="shell-node"/>
 <action name="shell-node">
 <shell xmlns="uri:oozie:shell-action:0.2">
 <job-tracker>${jobTracker}</job-tracker>
 <name-node>${nameNode}</name-node>
 <configuration>
    <property>
    <name>mapred.job.queue.name</name>
    <value>${queueName}</value>
    </property>
  </configuration>
 <exec>sample.sh</exec>
 <file>/<Hdfs Path of the shell script>/sample.sh</file>
 </shell>
 <ok to="end"/>
 <error to="fail"/>
 </action>
 <kill name="fail">
 <message>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
 </kill>
 <end name="end"/>
</workflow-app>


Running the job:
1
oozie job -oozie http://localhost:11000/oozie -config job.properties -run


Java Client for Secure Web Socket

This post is how to connect a websocket which is secured by Basic Authentication.

I will be using javax.websocket-api jar file.

Create WebContainer object

WebSocketContainer container = ContainerProvider.getWebSocketContainer();

using the container connect to the service

1
container.connectToServer(endpoint, clientConfig, new URI("wss://hostname:port/demo"));


credentials are passed to clientConfig object.

1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
ClientEndpointConfig.Configurator configurator = new ClientEndpointConfig.Configurator() {
             public void beforeRequest(Map<String, List<String>> headers) {
          String credentials = "username:password";
          headers.put("Authorization", Arrays.asList("Basic " + new BASE64Encoder().encode(credentials.getBytes())));
                 System.out.println("Header set successfully");
             }
         };

         ClientEndpointConfig clientConfig = ClientEndpointConfig.Builder.create()
                 .configurator(configurator)
                 .build();


endpoint is the callback handler. Once the session is established with the service then we pass a message using the session object. onMessage() method should be overridden to receive the message.

1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
Endpoint endpoint = new Endpoint() {
  @Override
                public void onOpen(Session session, EndpointConfig config) {
                    session.addMessageHandler(new MessageHandler.Whole<String>() {
                        @Override
                        public void onMessage(String content) {
                            System.out.println("Received message: "+content);
                        }
                    });
                    try {
                 System.out.println("Sending message to endpoint: " + msg);
                        System.out.println("Session Id:: "+session.getId());
   session.getBasicRemote().sendText(msg);
      } catch (Exception e) {
   e.printStackTrace();
      }
                }
            };

Thursday, October 19, 2017

Connect to Cassandra from RStudio

In order to connect to cassandra from rstudio RJDBC library would help.

Prerequisites:
Install RJDBC library
1
install.packages("RJDBC")


Place cassandra-jdbc.jar library in the cassandra libraries folder. In my environment it's placed under the path '/usr/share/dse/cassandra/lib/'

Make sure thrift protocol is enabled on the cassandra cluster
1
nodetool statusthrift


If thrift is disabled it can be enabled with following command

1
nodetool enablethrift


R Script:

1
2
3
4
library(RJDBC)
cassdrv <- JDBC("org.apache.cassandra.cql.jdbc.CassandraDriver",list.files("/usr/share/dse/cassandra/lib/",pattern="jar$",full.names=T))
casscon <- dbConnect(cassdrv, "jdbc:cassandra://localhost:9160/test")
res <- dbGetQuery(casscon, "select * from emp")

Tuesday, October 17, 2017

Java Custom Accumulators implementation to collect bad records

Accumulators:

During transformations in spark we often encounter a problem where we can use the variables defined outside the function that we pass to map() or filter but cannot pass the data from the function back to driver. Accumulators which is a shared variable in the spark cluster solve this problem.

Accumulators work as follows:
  • We create them in the driver by calling the SparkContext.accumulator(initial Value) method, which produces an accumulator holding an initial value. The return type is an org.apache.spark.Accumulator[T] object, where T is the type of initialValue.
  • Worker code in Spark closures can add to the accumulator with its += method (or add in Java).
  • The driver program can call the value property on the accumulator to access its value (or call value() and setValue() in Java).

Spark’s built-in accumulator types: integers (Accumulator[Int]) with addition. Out of the box, Spark supports accumulators of type Double, Long, and Float. In addition to these, Spark also includes an API to define custom accumulator types.

Custom accumulators need to extend AccumulatorParam, which is covered in the Spark API documentation. Beyond adding to a numeric value, we can use any operation for add, provided
that operation is commutative and associative.

I have a requirement where I have data in a file I want to collect the records who length is greater than characters:

First I create custom accumulator implementing org.apache.spark.AccumulatorParam

1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
public class RecordAccmulator implements AccumulatorParam<Map<String, String>> {
    private static final long serialVersionUID = 1L;

  @Override
  public Map<String, String> addInPlace(Map<String, String> arg0, Map<String, String> arg1) {
    Map<String, String> map = new HashMap<>();
    map.putAll(arg0);
    map.putAll(arg1);
    return map;
  }

  @Override
  public Map<String, String> zero(Map<String, String> arg0) {

    return new HashMap<>();
  }

  @Override
  public Map<String, String> addAccumulator(Map<String, String> arg0, Map<String, String> arg1) {

    return addInPlace(arg0, arg1);
  }
}


I use the custom accumulator in my business class:

1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
JavaSparkContext sc = SparkUtils.createSparkContext(MyTest.class.getName(), "local[*]");
 SQLContext hiveContext = SparkUtils.getSQLContext(sc);
 
 JavaRDD<String> file = sc.textFile("inputfile.txt");
 logger.info("File Record Count:: "+file.count());
 
 
    Accumulator<Map<String, String>> accm = sc.accumulator(new HashMap<>(), new RecordAccmulator());
 
 JavaPairRDD<String, String> filePair = file.mapToPair(new PairFunction<String, String, String>(  ) {

    private static final long serialVersionUID = 1L;

    @Override
    public Tuple2<String, String> call(String t) throws Exception {
        String[] str = StringUtils.split(t,":");
      
      if(str[1].length()>10){
        Map map = new HashMap<>();
        map.put(str[0], str[1]);
        accm.add(map);
      }
      return new Tuple2<String, String>(str[0], str[1]);
    }});
 
 logger.info("Pair Count:: "+filePair.count());
 logger.info("Accumulator Values:: "+accm.value());


Sunday, May 14, 2017

How to process huge files with hadoop

In order to read huge files we cannot afford to load them into memory. One solution would be a BlockingQueue approach.

It's a consumer-producer approach where producer read the file and place in the queue until it's full. Consumer thread is blocked until the queue is non empty.

Queue is created using BlockingQueue interface.

ExecutorService is used to create a consumer thread pool and producer thread pool.

java.util.concurrent.ExecutorService consumer = java.util.concurrent.Executors.newFixedThreadPool(CONSUMERS_COUNT);



Sunday, December 11, 2016

Livy Job Server Implementation

Livy Job Server Installation on Horton Works Data Platform (HDP 2.4):
Download the compressed file from
http://archive.cloudera.com/beta/livy/livy-server-0.2.0.zip

Unzip the file and work on configuration changes.

Configurations:

1
Add 'spark.master=yarn-cluster' in config file '/usr/hdp/current/spark-client/conf/spark-defaults.conf'



livy-env.sh
Add these entries

1
2
3
4
5
6
7
8
export SPARK_HOME=/usr/hdp/current/spark-client
export HADOOP_HOME=/usr/hdp/current/hadoop-client/bin/
export HADOOP_CONF_DIR=/etc/hadoop/conf
export SPARK_CONF_DIR=$SPARK_HOME/conf
export LIVY_LOG_DIR=/jobserver-livy/logs
export LIVY_PID_DIR=/jobserver-livy
export LIVY_MAX_LOG_FILES=10
export HBASE_HOME=/usr/hdp/current/hbase-client/bin


log4j.properties
By default logs roll on console in INFO mode. It can be changed to file rolling with debug enabled with the below configuration:

1
2
3
4
5
6
log4j.rootCategory=DEBUG, NotConsole
log4j.appender.NotConsole=org.apache.log4j.RollingFileAppender
log4j.appender.NotConsole.File=/Analytics/livy-server/logs/livy.log
log4j.appender.NotConsole.maxFileSize=20MB
log4j.appender.NotConsole.layout=org.apache.log4j.PatternLayout
log4j.appender.NotConsole.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n

Copy the following libraries to the path '<LIVY SEVER INSTALL PATH>/rsc-jars'

1
2
3
4
5
6
7
8
hbase-common.jar
hbase-server.jar
hbase-client.jar
hbase-rest.jar
guava-11.0.2.jar
protobuf-java.jar
hbase-protocol.jar
spark-assembly.jar


Integrating web Application with Livy job server:
Copy the following libraries to tomcat classpath (lib folder)

1
2
livy-api-0.2.0.jar
livy-client-http-0.2.0.jar


Servlet invokes the LivyClientUtil

LivyClientUtil:
1
2
3
4
5
String livyUrl = "http://127.0.0.1:8998";
LivyClient client = new HttpClientFactory().createClient(new URL(livyUrl).toURI(), null);
client.uploadJar(new File(jarPath)).get();
TimeUnit time = TimeUnit.SECONDS;
String jsonString = client.submit(new SparkReaderJob("08/19/2010")).get(40,time);

SparkReaderJob:

1
2
3
4
5
public class SparkReaderJob implements Job<String> {
@Override
public String call(JobContext jc) throws Exception {
           JavaSparkContext jsc = jc.sc();
} }


References:
http://livy.io/
https://github.com/cloudera/livy

Monday, November 14, 2016

How to call a java class from scala

Create an Eclipse java maven project.

Add Scala nature to the project Right click on project > Configure > Add Scala Nature

Create project structure as:
src/main/java
src/main/scala

Sample Java class:

1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
package test.java;

public class Hello {

    public static void main(String[] args) {
        Hello hello = new Hello();
        hello.helloUser("Tomcat");

    }

    public void helloUser(String user) {
        System.out.println("Hello " + user);
    }

}

Calling java class from Scala class

1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
package test.scala;

import test.java.Hello

object HelloScala {

  def helloScala() {
    val hello = new Hello()
    hello.helloUser("Scala")
  }
}


POM file:

1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>Java_Scala_Project</groupId>
    <artifactId>Java_Scala_Project</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <dependencies>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>2.10.4</version>
        </dependency>

    </dependencies>

    <!-- <repositories>
        <repository>
            <id>scala</id>
            <name>Scala Tools</name>
            <url>http://scala-tools.org/repo-releases/</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </repository>
    </repositories> -->

    <pluginRepositories>
        <pluginRepository>
            <id>scala</id>
            <name>Scala Tools</name>
            <url>http://scala-tools.org/repo-releases/</url>
            <releases>
                <enabled>true</enabled>
            </releases>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </pluginRepository>
    </pluginRepositories>

    <build>
        <sourceDirectory>src</sourceDirectory>
        <plugins>
            <plugin>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.3</version>
                <configuration>
                    <source>1.7</source>
                    <target>1.7</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <executions>
                    <execution>
                        <goals>
                            <goal>compile</goal>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.scala-tools</groupId>
                <artifactId>maven-scala-plugin</artifactId>
                <executions>
                    <execution>
                        <id>compile</id>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                        <phase>compile</phase>
                    </execution>
                    <execution>
                        <id>test-compile</id>
                        <goals>
                            <goal>testCompile</goal>
                        </goals>
                        <phase>test-compile</phase>
                    </execution>
                    <execution>
                        <phase>process-resources</phase>
                        <goals>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>