Hive installation using MySQL on ubuntu

1. Install MySQL
             $ sudo apt-get install mysql-server
             Note: You will be prompted to seta password for root.

2. Install the MySQL Java Connector –
           $ sudo apt-get install libmysql-java

3. Create soft link for connector in Hive lib directory or copy connector jar to lib folder –
           ln -s /usr/share/java/mysql-connector-java.jar $HIVE_HOME/lib/mysql-connector-java.jar
          Note :- HIVE_HOME points to installed hive folder.

4. Create the Initial database schema using the hive-schema-0.14.0.mysql.sql file ( or the file
corresponding to your installed version of Hive) located in the
           $HIVE_HOME/scripts/metastore/upgrade/mysql directory.
           $ mysql -u root -p
           Enter password:


mysql> CREATE DATABASE metastore;
mysql> USE metastore;
mysql> SOURCE $HIVE_HOME/scripts/metastore/upgrade/mysql/hive-schema-0.14.0.mysql.sql;

5. You also need a MySQL user account for Hive to use to access the metastore. It is very important
to prevent this user account from creating or altering tables in the metastore database schema.

        mysql> CREATE USER 'hiveuser'@'%' IDENTIFIED BY 'hivepassword';
        mysql> GRANT all on *.* to 'hiveuser'@localhost identified by 'hivepassword';
        mysql> flush privileges;
        Note : – hiveuser is the ConnectionUserName in hive-site.xml ( As explained next)

6. Create hive-site.xml ( If not already present) in $HIVE_HOME/conf folder with the
configuration below –

<configuration>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:mysql://localhost/metastore?
createDatabaseIfNotExist=true</value>
<description>metadata is stored in a MySQL server</description>
</property>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>com.mysql.jdbc.Driver</value>
<description>MySQL JDBC driver class</description>
</property>
<property>
<name>javax.jdo.option.ConnectionUserName</name>
<value>hiveuser</value>
<description>user name for connecting to mysql server</description>
</property>
<property>
<name>javax.jdo.option.ConnectionPassword</name>
<value>hivepassword</value>
<description>password for connecting to mysql server</description>
</property>
<property>
<name>datanucleus.autoCreateSchema</name>
<value>false</value>
<description>Creates necessary schema on a startup if one doesn't exist
</ description>
</property>
</configuration>

7. We are all set now. Start the hive console.

Type hive and enter



For any clarification type in comment and keep update with latest information with www.facebook.com/coebda

Get & Set Method in Salesforce

Getter and setter methods are used to pass data from your visualforce page to your controller and vice versa..

Let's take a very simple scenario... Let's assume you want to display a textbox in your visualforce page.. When the user enters some value in this textbox and clicks on a button you want the value entered by the user in your Apex class (ie. basically your controller or extension)

So go ahead and create a simple visualforce page.. the code for this would be

<apex:page controller="simplegetset">
  <apex:form>
    <apex:outputlabel value="Enter your name here"/>
       <apex:inputtext value="{!userinput}"/>          
  </apex:form>    
</apex:page>

The Apex code for this page would be...

public class simplegetset
{
    public String userinput{get; set;}
}

Now, the variable "userinput" in your Apex class will store the value entered by the user....

Let's analyze the methods now...

Get

The "get" method is used to pass data from your Apex code to your Visualforce page.. In our example we are not passing any value.. hence, when your page loads initially the textbox will have a empty value...

So, lets modify our code and pass a default value to the textbox.. Change the Apex code as follows..

public class simplegetset
{  
    public String userinput;
    public String getuserinput(){return 'John';}
   
    public void setuserinput(String userinput)
    {
        this.userinput = userinput;
    }   
}

You can now see that your page loads with a value 'John'...

Set

The "set" method is used to pass values from your visualforce page to the controller... In our example the variable "userinput" will be storing the value entered in the textbox..

Now modify your VF page as below..

<apex:page controller="simplegetset">
  <apex:form>
    <apex:outputlabel value="Enter your name here"/>
       <apex:inputtext value="{!userinput}">
           <apex:actionsupport event="onclick" rerender="display" />
       </apex:inputtext>                   
    <apex:outputpanel id="display">
        <apex:outputtext value="The name entered is {!userinput}"/>
    </apex:outputpanel>                   
  </apex:form>    
</apex:page>

The Apex code would be...

public class simplegetset
{
    public String userinput{get; set;}
}

In this example what happens is.. the variable "userinput" stores the value entered in visualforce page and passes it to the Apex code.. hence you are able to see the entered value in the visualforce page..

I guess you might understand what i am saying.. to make things simple now use the same visualforce page.. but modify the Apex code as below..

public class simplegetset
{  
    public String userinput;
    public String getuserinput(){return 'John';}
   
    public void setuserinput(String userinput)
    {
        this.userinput = userinput;
    }   
}

Now, whatever value you enter the page always displays "The name entered is John".... This is because your get method always returns 'John'... but still your set method will store the value you entered in the variable "userinput"....


for more update keep visiting 
www.facebook.com/coebda

Pig Exercise Part V

        Here I am sharing few exercise that can be helpful for project purpose.
Data analysis can be done by Pig:
Here I am taking weatherData to analysis purpose.

For raw data comment your mail ID in comment section.





----loading and parsing data-----

A = load '/weatherPIG' using TextLoader as (data:chararray);
AF = foreach A generate TRIM(SUBSTRING(data, 6, 14)), TRIM(SUBSTRING(data, 46, 53)), TRIM(SUBSTRING(data, 38, 45));
store AF into '/data9' using PigStorage(',');
S = load '/data9/part-m-00000' using PigStorage(',') as (date:chararray, min:double, max:double);

-------Hot Days------

X = filter S by max > 25;

-------Cold Days------

X = filter S by min < 0;

-------Hottest Day-----

H1 = group S all;     /* puts S's data in H1's Tuple */
I = foreach H1 generate MAX(S.max) as maximum;
X = filter S by max == I.maximum;

-------Coldest Day------

H2 = group S all;
J = foreach H2 generate MIN(S.min) as minimum;
X = filter S by min == J.minimum;

Pig Exercise-Part IV

Here is the exercise for pig language. It is referring to some raw data as it is large in space so it cannot be posted here.

For raw data post your email ID in comment.

A. Load Customer records
========================
cust = load '/input/custs' using PigStorage(',') as (custid:chararray, firstname:chararray, lastname:chararray,age:long,profession:chararray);

B. Select only 100 records
==========================
amt = limit cust 100;
dump amt;

c. Group customer records by profession
=======================================
groupbyprofession = group cust by profession;

D. Count no of customers by profession
======================================
countbyprofession = foreach groupbyprofession generate group, COUNT(cust);
dump countbyprofession;

E. Load transaction records
===========================
txn = load '/input/txns' using PigStorage(',') as(txnid:chararray, date:chararray,custid:chararray,amount:double,category:chararray,product:chararray,city:chararray,state:chararray,type:chararray);

F. Group transactions by customer
=================================
txnbycust = group txn by custid;

G. Sum total amount spent by each customer
==========================================
spendbycust = foreach txnbycust generate group, SUM(txn.amount);

H. Order the customer records beginning from highest spender
============================================================
custorder = order spendbycust by $1 desc;

I. Select only top 100 customers
================================
top100cust = limit custorder 100;

J. Join the transactions with customer details
==============================================
top100join = join top100cust by $0, cust by $0;
describe top100join;

K. Select the required fields from the join for final output
============================================================
top100 = foreach top100join generate $0,$3,$4,$5,$6,$1;
describe top100;

L.Dump the final output
=======================
dump top100;


Update yourself and keep visit
www.facebook.com/coebda

Pig Exercise-part III

Here is Pig programs for wordcount

myinput = load '/sample.txt' as (line);
//TOKENIZE splits the line into a field for each word. 
//flatten will take the collection of records returned by TOKENIZE and
//produce a separate record for each one, calling the single field in the
//record word.

words = foreach myinput generate flatten(TOKENIZE(line)) as word;

grpd = group words by word;

cntd = foreach grpd generate group, COUNT(words);

dump cntd;


Keep updated with

www.facebook.com/coebda

If you need raw data comment here:

Pig Exercise Part-II

Suppose we have 4 column data with field language, website, pagecount and page_size

    en google.com 50 100
    en yahoo.com 60 100
    us google.com 70 100
    en google.com 68 100

and we want


google.com 118
yahoo.com 60 as output.


records = LOAD '/webcount' using PigStorage(' ') as  (projectname:chararray, pagename:chararray, pagecount:int,pagesize:int);

filtered_records = FILTER records by projectname=='en';

grouped_records = GROUP filtered_records by pagename;     

results = FOREACH grouped_records generate group,SUM(filtered_records.pagecount);

sorted_result = ORDER results by $1 desc;

STORE sorted_result INTO '/YOUROUTPUT';


Keep updated with
www.facebook.com/coebda

 

Pig Exercise-part I

Note the following about bags:

• A bag can have duplicate tuples.
  
  
• A bag can have tuples with differing numbers of fields. However, if Pig tries to access a field that does not exist, a null value is substituted.
  
  
• A bag can have tuples with fields that have different data types. However, for Pig to effectively process bags, the schema of the tuples within those bags should be the same. For example, if half of the tuples include chararray fields and while the other half include float fields, only half of the tuples will participate in any kind of computation because the chararray fields will be converted to null.
  
  
  Bags have two forms: outer bag (or relation) and inner bag.
  
  

  Example: Outer Bag

  In this example A is a relation or bag of tuples. You can think of this bag as an outer bag.
  

  A = LOAD 'data' as (f1:int, f2:int, f3;int);
  DUMP A;
  (1,2,3)
  (4,2,1)
  (8,3,4)
  (4,3,3)

  Example: Inner Bag

  Now, suppose we group relation A by the first field to form relation X.
  In this example X is a relation or bag of tuples.
  
  The tuples in relation X have two fields. The first field is type int. The second field is type bag; you can think of this bag as an inner bag.
  

  X = GROUP A BY f1;
  DUMP X;
  (1,{(1,2,3)})
  (4,{(4,2,1),(4,3,3)})
  (8,{(8,3,4)})

  
  
   For more updates visit:
  www.facebook.com/coebda
  
  For raw data comment here.