hbutani / sqlwindowing Goto Github PK

Hello,

I am trying to use the windowing function and wondering how can I save the output into an existing table in hive.

for. eg. Trying to run this
from t_enc
partition by mid
order by tdate
with
sum(tamt) as slsamt
select mid,tdate,slsamt;

I would like to save the output to an existing table and not sure how can I do that.

Please advise.

Thanks!

Hi Harish,

trying linear regression,
this works:
*from <select acct_no, mydt, x, y
from mytab>
partition by acct_no
order by acct_no,mydt
with
linearRegSlope(x,y) as slope,
linearRegIntercept(x,y) as intercept
select acct_no, slope,intercept
into path='ma_9_2011_08'; *

but this does not, which tries to do regression on ALL the records without partitioning
**
from <select 1 as blah,acct_no, mydt, x, y
from mytab>
partition by blah
order by blah
with
linearRegSlope(x,y) as slope,
linearRegIntercept(x,y) as intercept
select blah, slope,intercept
into path='ma_9_2011_08';
**

Error:

12/02/03 12:24:55 INFO mapred.JobClient: Task Id : attempt_201201151510_0220_r_000002_3, Status : FAILED
java.lang.reflect.UndeclaredThrowableException: Unknown exception in doAs
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1139)
at org.apache.hadoop.mapred.Child.main(Child.java:264)
Caused by: java.security.PrivilegedActionException: com.sap.hadoop.ds.list.ByteBasedList$ListFullException
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
... 1 more
Caused by: com.sap.hadoop.ds.list.ByteBasedList$ListFullException
at com.sap.hadoop.ds.list.ByteBasedList.ensureCapacity(ByteBasedList.java:80)
at com.sap.hadoop.ds.list.ByteBasedList.write(ByteBasedList.java:108)
at com.sap.hadoop.ds.list.ByteBasedList.append(ByteBasedList.java:158)
at com.sap.hadoop.ds.list.ByteBasedList$append.call(Unknown Source)
at com.sap.hadoop.windowing.runtime.Partition.leftShift(Partition.groovy:100)
at com.sap.hadoop.windowing.runtime.Partit
attempt_201201151510_0220_r_000002_3: Query:
attempt_201201151510_0220_r_000002_3: tableInput=(hiveTable=WindowingTempTable_1328289790395)
attempt_201201151510_0220_r_000002_3: partitionColumns=all_in_one
attempt_201201151510_0220_r_000002_3: orderColumns=all_in_one ASC
attempt_201201151510_0220_r_000002_3: funcSpecs=[linearregslope(alias=slope, param=[id=bur_fico_scor_no, id=cv_score], type=null, window=null),
attempt_201201151510_0220_r_000002_3: linearregintercept(alias=intercept, param=[id=bur_fico_scor_no, id=cv_score], type=null, window=null)]
attempt_201201151510_0220_r_000002_3: select=slope, intercept
attempt_201201151510_0220_r_000002_3: whereExpr=null

hey Harish,

thanks for the npath function,
a few observations,

i was running this:
from
npath(<select acct_no, my_dt ,my_score
from my_tab>
partition by my_acct_no
order by my_dt,
'a.b.c',
<["a" : "my_score == 444","b" : "my_score == 555","c" : "my_score == 666"]>,
<[ "distinct my_acct_no" ]>)
select my_acct_no into path='npath_10_2011_09';

found that it doesn't like distinct in the final select. In this particular case, may be it doesn't matter since the partition is by the acct number. But how can I do Aster style distinct in the "outer" select?

Tested with CDH3 Update 3 (cdh3u3)

Here is error log, any idea what I did wrong?

~/hive/bin> hive --service windowingCli -w com.sap.hadoop.windowing-0.0.1-SNAPSHOT-jar-with-dependencies.jar
Hive history file=/tmp/yzhang/hive_job_log_yzhang_201202112135_1368506989.txt
Exception in thread "main" com.sap.hadoop.windowing.WindowingException: java.lang.NumberFormatException: null
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77)
at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:54)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:182)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:190)
at com.sap.hadoop.windowing.cli.WindowingClient.connect(WindowingClient.groovy:126)
at com.sap.hadoop.windowing.cli.WindowingClient.(WindowingClient.groovy:38)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77)
at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:54)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:182)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:198)
at com.sap.hadoop.windowing.WindowingHiveCliDriver.setupClient(WindowingHiveCliDriver.groovy:64)
at com.sap.hadoop.windowing.WindowingHiveCliDriver$setupClient.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120)
at com.sap.hadoop.windowing.WindowingHiveCliDriver.main(WindowingHiveCliDriver.groovy:221)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
Caused by: java.lang.NumberFormatException: null
at java.lang.Integer.parseInt(Integer.java:417)
at java.lang.Integer.parseInt(Integer.java:499)
at java_lang_Integer$parseInt.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at com.sap.hadoop.windowing.cli.WindowingClient.readPort(WindowingClient.groovy:110)
at com.sap.hadoop.windowing.cli.WindowingClient.connect(WindowingClient.groovy:117)
... 21 more

Hi,

While running a query, I found that even after completing the CTAS step, the windowing process plods along without any complaint, only to fumble at the end in the reduce stage if a column name is misspelt or does not exist. Shouldn't the parsing in detect that the final select, after the "with" clause has columns that are erroneous? Cause the user has no way of knowing that something errored out unless he looks at the logs. There is a mysterious spew on the screen something like this:
attempt_201201151510_0169_r_000001_0: Query:
attempt_201201151510_0169_r_000001_0: tableInput=(hiveTable=WindowingTempTable_1328129373936)
attempt_201201151510_0169_r_000001_0: partitionColumns=bc_prod_acct_no
attempt_201201151510_0169_r_000001_0: orderColumns=bc_prod_acct_no ASC, bc_date ASC
attempt_201201151510_0169_r_000001_0: funcSpecs=[rank(alias=r, param=[], type=null, window=null),
attempt_201201151510_0169_r_000001_0: avg(alias=ma4_ratio, param=[expr=cv_tot_bal/cv_rtl_bal_am], type=null, window=window(start=range(3 PRECEDING), end=currentRow)),
attempt_201201151510_0169_r_000001_0: last_value(alias=prev_ratio, param=[expr=cv_tot_bal/cv_rtl_bal_am], type=null, window=window(start=range(1 PRECEDING), end=currentRow)),
attempt_201201151510_0169_r_000001_0: ntile(alias=percentile_5, param=[intValue=5], type=null, window=null)]
attempt_201201151510_0169_r_000001_0: select=bc_prod_acct_no, bc_date, cv_ratio, r, ma4_ratio, prev_ratio, percentile_5
attempt_201201151510_0169_r_000001_0: whereExpr=null
attempt_201201151510_0169_r_000001_0: tableOutput=(output(path=ma_6_2011_08, serde=org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, serDeProps={field.delim=,, columns=bc_prod_acct_no,bc_date,cv_ratio,r,ma4_ratio,prev_ratio,percentile_5, columns.types=bigint,string,double,int,double,double,int}, format=org.apache.hadoop.mapred.TextOutputFormat)

thanks

ameet

I am trying to start windowingCli but getting the below error on one of the cluster. It seems to be running fine with the other cluster.

hive --service windowingCli
Hive history file=/tmp/root/hive_job_log_root_201210101710_1530752256.txt
Exception in thread "main" groovy.lang.MissingMethodException: No signature of method: jline.ConsoleReader.addCompletor() is applicable for argument types: ([Ljline.Completor;) values: [[org.apache.hadoop.hive.cli.CliDriver$5@78c08593, org.apache.hadoop.hive.cli.CliDriver$3@2781b3d4]]
Possible solutions: addCompletor(jline.Completor), getCompletors()
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:55)
at org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:46)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at com.sap.hadoop.windowing.WindowingHiveCliDriver.main(WindowingHiveCliDriver.groovy:219)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

here is my windowCli.sh file.

THISSERVICE=windowingCli
export SERVICE_LIST="${SERVICE_LIST}${THISSERVICE} "

windowingCli () {
CLASS=com.sap.hadoop.windowing.WindowingHiveCliDriver
if $cygwin; then
HIVE_LIB=cygpath -w "$HIVE_LIB"
fi
JAR=${HIVE_LIB}/com.sap.hadoop.windowing-0.0.2-SNAPSHOT.jar
exec $HADOOP jar $JAR $CLASS "$@"
}

windowingCli_help () {
windowingCli "--help"
}

hi,

I am able to make a connection from the client to the metastore service and run a hive query
hive --service windowingCli -w /tmp/windowing/com.sap.hadoop.windowing-0.0.1-SNAPSHOT-jar-with-dependencies.jar

however, when I type a query, it does not recognize the syntax,

hive> from <select a1,a2,a8 from matest >
> partition by a1
> order by a1, a2 asc
> with rank() as r,
> select a1,a2,a8,r;
FAILED: Parse Error: line 1:5 cannot recognize input near '<' 'select' 'a1' in join source.

What am I missing?

thanks for your help

ameet

Hey Harish,

I am running a query which has a final where clause

from <select acct_no,to_date(mydt) my_date,
cv_tot_bal from mon_2011_08_all_attrib>
partition by acct_no
order by acct_no, my_date asc
with
RowNumber(cv_tot_bal) as tot_rank, Count(cv_tot_bal) as tot_count
select
acct_no, my_date, cv_tot_bal, tot_rank, tot_count
Where < tot_rank in ( tot_count/2 +1), (tot_count+1)/2) >
into path='ma_7_2011_08'

the query runs and the create table hive job compeltes successfully. But the next windowing Job never runs and there is no indication in the window as well, no error.
Without the *where clause the query runs fine.

Is my where syntax ok?

i am trying to do a median.

ameet

Hi,

I downloaded the jar with dependencies : com.sap.hadoop.windowing-0.0.1-SNAPSHOT-jar-with-dependencies.jar
modified the cli.sh and copied it to windowingCli.sh
I can start the service by:
hive --service windowingCli -w /tmp/windowing/com.sap.hadoop.windowing-0.0.1-SNAPSHOT-jar-with-dependencies.jar
but when I type any command such as show databases, it errors out:
hive> show databases;
FAILED: Error in metadata: javax.jdo.JDOFatalInternalException: Unexpected exception caught.
NestedThrowables:
java.lang.reflect.InvocationTargetException
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

I have also started the metastore as described in the pdf but no dice.

thanks for your help

ameet

is ther anyone to encountered below issue? Any help or point are appreciated. thx.
hive> from
> <select starttime,lac ,ci ,callerno,count(1) callCount
> from ranap
> group by starttime,lac ,ci ,callerno limit 1000
> >
> partition by lac
> order by lac
> with
> rank() as r
> select starttime,lac,ci,callerno,callCount,r
> where <r <=3>
> into path '/tmp/hadhive/'
> serde 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
> with serdeproperties('field.delim'=',')
> format 'org.apache.hadoop.mapred.TextOutputFormat';
BR.recoverFromMismatchedToken
com.sap.hadoop.windowing.WindowingException: line 12:10 mismatched input '/tmp/hadhive/' expecting EQ
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77)
at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:190)
at com.sap.hadoop.windowing.runtime.WindowingShell.parse(WindowingShell.groovy:72)
at com.sap.hadoop.windowing.runtime.WindowingShell.execute(WindowingShell.groovy:126)
at com.sap.hadoop.windowing.runtime.WindowingShell$execute.call(Unknown Source)
at com.sap.hadoop.windowing.cli.WindowingClient3.executeQuery(WindowingClient3.groovy:28)
at com.sap.hadoop.windowing.cli.WindowingClient3$executeQuery.call(Unknown Source)
at com.sap.hadoop.windowing.WindowingHiveCliDriver.processCmd(WindowingHiveCliDriver.groovy:117)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:406)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:341)
at org.apache.hadoop.hive.cli.CliDriver$processLine.call(Unknown Source)
at com.sap.hadoop.windowing.WindowingHiveCliDriver.main(WindowingHiveCliDriver.groovy:235)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

I am running the below on the table temp2 partitioned on Symbol and am continously getting the below error

hieve> from temp2 partition by Symbol order by Day with avg(Volume) over rows between unbounded preceding and current row as rollingavg select Symbol,Day,Rollingavg where <Day \> 3> into path='/home/hive';
com.sap.hadoop.windowing.WindowingException: Unknown partition column Symbol
at sun.reflect.GeneratedConstructorAccessor65.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77)
at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:190)
at com.sap.hadoop.windowing.query.Translator.setupInputColumns(Translator.groovy:129)
at com.sap.hadoop.windowing.query.Translator$setupInputColumns.callCurrent(Unknown Source)
at com.sap.hadoop.windowing.query.Translator.setupQueryInput(Translator.groovy:97)
at com.sap.hadoop.windowing.query.Translator$setupQueryInput.callCurrent(Unknown Source)
at com.sap.hadoop.windowing.query.Translator.translate(Translator.groovy:48)
at com.sap.hadoop.windowing.query.Translator$translate.call(Unknown Source)
at com.sap.hadoop.windowing.runtime.WindowingShell.execute(WindowingShell.groovy:127)
at com.sap.hadoop.windowing.runtime.WindowingShell$execute.call(Unknown Source)
at com.sap.hadoop.windowing.cli.WindowingClient3.executeQuery(WindowingClient3.groovy:28)
at com.sap.hadoop.windowing.cli.WindowingClient3$executeQuery.call(Unknown Source)
at com.sap.hadoop.windowing.WindowingHiveCliDriver.processCmd(WindowingHiveCliDriver.groovy:117)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:338)
at org.apache.hadoop.hive.cli.CliDriver$processLine.call(Unknown Source)
at com.sap.hadoop.windowing.WindowingHiveCliDriver.main(WindowingHiveCliDriver.groovy:235)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

Failed windowing query from temp2 partition by Symbol order by Day with avg(Volume) over rows between unbounded preceding and current row as rollingavg select Symbol,Day,Rollingavg where <Day \> 3> into path='/home/hive' Unknown partition column Symbol

The below Query gives the same error
from < select Day,SUM(Volume) as Volume,Symbol from temp2 GROUP BY Symbol,Day> partition by Symbol order by Day with avg(Volume) over rows between unbounded preceding and current row as rollingavg select Symbol,Day,Rollingavg where <Day \> 3> into path='/home/hive'

reported by achaubal.

I have created external table ext_test_data and using the windowing function and i get following error. It looks for abc.ext_test_data? Why does it default to abc? i tried with databasename.tablename but that didn't work either

hive> from ext_test_data
> partition by acc_id
> order by acc_id,upd_dt desc
> with
> first_value(upd_dt, 'true') over rows between current row and 1 following as eff_end_dt
> select acc_id,acc_name,upd_dt,eff_end_dt;
com.sap.hadoop.windowing.WindowingException: com.sap.hadoop.windowing.WindowingException: NoSuchObjectException(message:abc.ext_test_data table not found)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77)

Rohit

Hi hbutani:
I can't download the subset of the 2011 Census data into my Hive instance. I have found for google, but failed. Can you send me one? My email is [email protected].
Thanks a lot.

Hi,

I tried kill, running job In the middle.

such,
hadoop job -kill

but, sqlwindowing query was running,
The log such as follows was output.

2012-06-21 16:13:52,553 Stage-1 map = 41%, reduce = 0%
2012-06-21 16:14:09,029 Stage-1 map = 42%, reduce = 0%
Ended Job = job_201206201422_29401 with errors
Error during job, obtaining debugging information...
Examining task ID: task_201206201422_29401_m_010818 (and more) from job job_201206201422_29401
Examining task ID: task_201206201422_29401_m_000119 (and more) from job job_201206201422_29401
Examining task ID: task_201206201422_29401_m_000024 (and more) from job job_201206201422_29401
Examining task ID: task_201206201422_29401_m_000028 (and more) from job job_201206201422_29401

Hi,
I have followed the exact procedure and when i start the hive --service command it throws the below error
Exception in thread "main" java.lang.NoClassDefFoundError: JAR
Caused by: java.lang.ClassNotFoundException: JAR
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: JAR. Program will exit.

I have used com.sap.hadoop.windowing-0.0.2-SNAPSHOT.jar and groovy-all-1.8.0.jar and my this is my sh

THISSERVICE=windowingCli
export SERVICE_LIST="${SERVICE_LIST}${THISSERVICE} "

windowingCli () {
CLASS=com.sap.hadoop.windowing.WindowingHiveCliDriver
if $cygwin; then
HIVE_LIB=cygpath -w "$HIVE_LIB"
fi
JAR=${HIVE_LIB}/com.sap.hadoop.windowing-0.0.2-SNAPSHOT.jar
exec $HADOOP jar $JAR $CLASS "$@"
}

windowingCli_help () {
windowingCli "--help"
}

please help me with this error. i am using cdh 4.7 which came with hive 0.10. thank you

hi guys, what may cause this problem ? thank you .
> select * from windowing;
OK
1 cs 1000.0
2 cs 2000.0
3 cs 4000.0
4 ds 3000.0
5 ds 1000.0
6 ds 500.0
7 cs 8000.0
Time taken: 0.731 seconds
hive> from windowing
> partition by dep
> order by salary
> with count() as c
> select dep, salary, c ;
FAILED: Parse Error: line 2:0 required (...)+ loop did not match anything at input 'partition' in statement

Hi Harish,

looking at the windowing.pdf doc, it seemed that the metastore --service was needed to make this work for the windowing jar. Is it mandatory? or is it just a representative picture?

i.e. can I not have just a regular mysql running somewhere and make appropriate changes to hive-site.xml to point to that machine with mysql with appropriate jdbc driver? Or the metastore as a service is NEEDED by the windowing jar to make connections into A metastore.

thanks

ameet

hi , i tested below sql , it will raise below issue. how to fix it?

hive> wmode windowing;
hive> from
> <select starttime,lac ,ci ,callerno,count(1) callCount
> from ranap
> group by starttime,lac ,ci ,callerno limit 1000
> >
> partition by lac
> order by lac
> with
> rank() as r
> select starttime,lac,ci,callerno,callCount,r
> where <r <=3>;
Executing Embedded Hive Query:
Create table WindowingTempTable_1345105836343 as select starttime,lac ,ci ,callerno,count(1) callCount
from ranap
group by starttime,lac ,ci ,callerno limit 1000

Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks not specified. Defaulting to jobconf value of: 8
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=
In order to set a constant number of reducers:
set mapred.reduce.tasks=
Cannot run job locally: Number of reducers (= 8) is more than 1
Starting Job = job_201208161534_0030, Tracking URL = http://server52:50030/jobdetails.jsp?jobid=job_201208161534_0030
Kill Command = /home/hadhive/hadoop-0.20.2/bin/hadoop job -Dmapred.job.tracker=server52:9001 -kill job_201208161534_0030
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 8
2012-08-16 16:30:41,123 Stage-1 map = 0%, reduce = 0%
2012-08-16 16:30:47,145 Stage-1 map = 100%, reduce = 0%
2012-08-16 16:30:56,180 Stage-1 map = 100%, reduce = 13%
2012-08-16 16:30:57,186 Stage-1 map = 100%, reduce = 25%
2012-08-16 16:30:58,191 Stage-1 map = 100%, reduce = 38%
2012-08-16 16:30:59,197 Stage-1 map = 100%, reduce = 50%
2012-08-16 16:31:00,203 Stage-1 map = 100%, reduce = 63%
2012-08-16 16:31:01,213 Stage-1 map = 100%, reduce = 75%
2012-08-16 16:31:05,230 Stage-1 map = 100%, reduce = 88%
2012-08-16 16:31:06,236 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201208161534_0030
Launching Job 2 out of 2
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=
In order to set a constant number of reducers:
set mapred.reduce.tasks=
Cannot run job locally: Number of Input Files (= 8) is larger than hive.exec.mode.local.auto.input.files.max(= 4)
Starting Job = job_201208161534_0031, Tracking URL = http://server52:50030/jobdetails.jsp?jobid=job_201208161534_0031
Kill Command = /home/hadhive/hadoop-0.20.2/bin/hadoop job -Dmapred.job.tracker=server52:9001 -kill job_201208161534_0031
Hadoop job information for Stage-2: number of mappers: 3; number of reducers: 1
2012-08-16 16:31:14,011 Stage-2 map = 0%, reduce = 0%
2012-08-16 16:31:17,022 Stage-2 map = 67%, reduce = 0%
2012-08-16 16:31:18,027 Stage-2 map = 100%, reduce = 0%
2012-08-16 16:31:26,057 Stage-2 map = 100%, reduce = 100%
Ended Job = job_201208161534_0031
Moving data to: hdfs://server52:9000/smp/hive/warehouse/test_js.db/windowingtemptable_1345105836343
Table test_js.windowingtemptable_1345105836343 stats: [num_partitions: 0, num_files: 1, num_rows: 0, total_size: 33224, raw_data_size: 0]
1000 Rows loaded to hdfs://server52:9000/tmp/hive-hadhive/hive_2012-08-16_16-30-36_344_8638175632607957910/-ext-10000
MapReduce Jobs Launched:
Job 0: Map: 1 Reduce: 8 HDFS Read: 37276978 HDFS Write: 348674 SUCCESS
Job 1: Map: 3 Reduce: 1 HDFS Read: 350178 HDFS Write: 33224 SUCCESS
Total MapReduce CPU Time Spent: 0 msec
OK
Time taken: 53.257 seconds
Finished Executing Embedded Hive Query

com.sap.hadoop.windowing.WindowingException: com.sap.hadoop.windowing.WindowingException: NoSuchObjectException(message:big.WindowingTempTable_1345105836343 table not found)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77)
at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:190)
at com.sap.hadoop.windowing.runtime.mr.MRTranslator.setupWindowingInput(MRTranslator.groovy:45)
at com.sap.hadoop.windowing.runtime.mr.MRTranslator$setupWindowingInput.callCurrent(Unknown Source)
at com.sap.hadoop.windowing.query.Translator.setupQueryInput(Translator.groovy:92)
at com.sap.hadoop.windowing.query.Translator$setupQueryInput.callCurrent(Unknown Source)
at com.sap.hadoop.windowing.query.Translator.translate(Translator.groovy:48)
at com.sap.hadoop.windowing.query.Translator$translate.call(Unknown Source)
at com.sap.hadoop.windowing.runtime.WindowingShell.execute(WindowingShell.groovy:127)
at com.sap.hadoop.windowing.runtime.WindowingShell$execute.call(Unknown Source)
at com.sap.hadoop.windowing.cli.WindowingClient3.executeQuery(WindowingClient3.groovy:28)
at com.sap.hadoop.windowing.cli.WindowingClient3$executeQuery.call(Unknown Source)
at com.sap.hadoop.windowing.WindowingHiveCliDriver.processCmd(WindowingHiveCliDriver.groovy:117)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:406)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:341)
at org.apache.hadoop.hive.cli.CliDriver$processLine.call(Unknown Source)
at com.sap.hadoop.windowing.WindowingHiveCliDriver.main(WindowingHiveCliDriver.groovy:235)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Caused by: com.sap.hadoop.windowing.WindowingException: NoSuchObjectException(message:big.WindowingTempTable_1345105836343 table not found)
at com.sap.hadoop.HiveUtils.getTable(HiveUtils.java:190)
at com.sap.hadoop.HiveUtils.getTable(HiveUtils.java:198)
at com.sap.hadoop.HiveUtils$getTable.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
at com.sap.hadoop.HiveUtils$getTable.call(Unknown Source)
at com.sap.hadoop.windowing.runtime.mr.MRTranslator.setupWindowingInput(MRTranslator.groovy:35)
... 19 more
Caused by: NoSuchObjectException(message:big.WindowingTempTable_1345105836343 table not found)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:976)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:735)
at com.sap.hadoop.HiveUtils.getTable(HiveUtils.java:186)
... 24 more
Failed windowing query from
<select starttime,lac ,ci ,callerno,count(1) callCount
from ranap
group by starttime,lac ,ci ,callerno limit 1000

partition by lac
order by lac
with
rank() as r
select starttime,lac,ci,callerno,callCount,r
where <r <=3> com.sap.hadoop.windowing.WindowingException: NoSuchObjectException(message:big.WindowingTempTable_1345105836343 table not found)

Hi，

com.sap.hadoop.windowing-0.0.2-SNAPSHOT.jar from https://github.com/hbutani/SQLWindowing/downloads was built from which tree of the sources ?
could you give me a hint? please

I am using Cloudra hive-0.7.1-cdh3u3/

yzhang@bsdhcp15755:~> hive --service windowingCli -w /home/yzhang/lib/com.sap.hadoop.windowing-0.0.1-SNAPSHOT-jar-with-dependencies.jar
Hive history file=/tmp/yzhang/hive_job_log_yzhang_201204031410_1192503448.txt
hive-log4j.properties not found
hive> use dev;
OK
Time taken: 1.496 seconds
hive> show tables;
OK
auditlog_plus
departments
employees
test
windowingtemptable_1333476532214
Time taken: 0.183 seconds
hive> wmode windowing;
hive> from <select department_id, employee_id, salary from employees>
> partition by department_id
> order by department_id, salary desc
> with rank() as r
> select department_id, employee_id, salary, r
> into path='/tmp/wout';
Executing Embedded Hive Query...
Total MapReduce jobs = 2
Launching Job 1 out of 2
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_201204021126_0014, Tracking URL = http://bsdhcp15755:50030/jobdetails.jsp?jobid=job_201204021126_0014
Kill Command = /home/yzhang/hadoop/bin/hadoop job -Dmapred.job.tracker=bsdhcp15755:8021 -kill job_201204021126_0014
2012-04-03 14:11:09,358 Stage-1 map = 0%, reduce = 0%
2012-04-03 14:11:10,372 Stage-1 map = 100%, reduce = 0%
2012-04-03 14:11:11,378 Stage-1 map = 100%, reduce = 100%
Ended Job = job_201204021126_0014
Ended Job = 189591670, job is filtered out (removed at runtime).
Moving data to: hdfs://bsdhcp15755/tmp/hive-yzhang/hive_2012-04-03_14-11-06_308_5393686420578850454/-ext-10001
Moving data to: hdfs://bsdhcp15755/user/hive/warehouse/dev.db/windowingtemptable_1333476666291
107 Rows loaded to hdfs://bsdhcp15755/tmp/hive-yzhang/hive_2012-04-03_14-11-06_308_5393686420578850454/-ext-10000
OK
Time taken: 5.416 seconds
done executing Embedded Hive Query.
12/04/03 14:11:11 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
12/04/03 14:11:11 INFO metastore.ObjectStore: ObjectStore, initialize called
Failed windowing query from <select department_id, employee_id, salary from employees>
partition by department_id
order by department_id, salary desc
with rank() as r
select department_id, employee_id, salary, r
into path='/tmp/wout' com.sap.hadoop.windowing.WindowingException: javax.jdo.JDOFatalInternalException: Unexpected exception caught.
NestedThrowables:
java.lang.reflect.InvocationTargetException

And here is the log in the /tmp/yzhang/hive.log:

2012-04-03 14:11:11,975 ERROR CliDriver (SessionState.java:printError(365)) - Failed windowing query from <select department_id, employee_id, salary from employees>
partition by department_id
order by department_id, salary desc
with rank() as r
select department_id, employee_id, salary, r
into path='/tmp/wout' com.sap.hadoop.windowing.WindowingException: javax.jdo.JDOFatalInternalException: Unexpected exception caught.
NestedThrowables:
java.lang.reflect.InvocationTargetExceptioncom.sap.hadoop.windowing.WindowingException: com.sap.hadoop.windowing.WindowingException: javax.jdo.JDOFatalInternalException: Unexpected exception caught.
NestedThrowables:
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77)
at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallConstructor(CallSiteArray.java:54)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:182)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:190)
at com.sap.hadoop.windowing.cli.WindowingClient.executeQuery(WindowingClient.groovy:61)
at com.sap.hadoop.windowing.cli.WindowingClient$executeQuery.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at com.sap.hadoop.windowing.WindowingHiveCliDriver.processCmd(WindowingHiveCliDriver.groovy:132)
at com.sap.hadoop.windowing.WindowingHiveCliDriver$processCmd.callCurrent(Unknown Source)
at com.sap.hadoop.windowing.WindowingHiveCliDriver.processLine(WindowingHiveCliDriver.groovy:85)
at com.sap.hadoop.windowing.WindowingHiveCliDriver$processLine.call(Unknown Source)
at com.sap.hadoop.windowing.WindowingHiveCliDriver.main(WindowingHiveCliDriver.groovy:255)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:197)

Hi,

I cannot write to $HIVE_HOME/lib and $HIVE_HOME/bin/ext directory as our environment does not allow custom jars.

I have set up hive --auxpath to point to groovy and windowing jar. Also copied cli.sh to my local hive user directory and renamed to wcli.sh and made the recommended changes.

i get following error:
hive --service wcli
WARN: HADOOP_TOOLS_HOME env not set. Using /home/y/bin/../
Service wcli not found
Available Services: cli help jar lineage rcfilecat

Is there an environment variable to be set so that i can using windowing from non-standard location?

Rohit

I am using hive 0.7.1 with mysql as a metastore
I have copied com.sap.hadoop.windowing-0.0.1-SNAPSHOT.jar,com.sap.hadoop.windowing-0.0.1-SNAPSHOT-jar-with-dependencies.jar and groovy-all-1.8.0.jar files to /usr/lib/hive/lib folder.Also added windowing.sh file.Below is the content of this file :

THISSERVICE=windowingCli
export SERVICE_LIST="${SERVICE_LIST}${THISSERVICE} "

windowingCli () {
CLASS=com.sap.hadoop.windowing.WindowingHiveCliDriver
if $cygwin; then
HIVE_LIB=cygpath -w "$HIVE_LIB"
fi
JAR=/usr/lib/hive/lib/com.sap.hadoop.windowing-0.0.1-SNAPSHOT.jar
exec $HADOOP jar $JAR $CLASS "$@"
}

windowingCli_help () {
windowingCli "--help"
}

Then I started metastore service with hive --service metastore command.

But hive --service windowingCli command does not work and it throws below exception.It seems jar files are corrupt.

[sb25634@hadoop02 ~]$ hive --service windowingCli -w /usr/lib/hive/lib/com.sap.hadoop.windowing-0.0.1-SNAPSHOT-jar-with-dependencies.jar
Exception in thread "main" java.io.IOException: Error opening job jar: /usr/lib/hive/lib/com.sap.hadoop.windowing-0.0.1-SNAPSHOT.jar
at org.apache.hadoop.util.RunJar.main(RunJar.java:124)
Caused by: java.util.zip.ZipException: error in opening zip file
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.(ZipFile.java:127)
at java.util.jar.JarFile.(JarFile.java:135)
at java.util.jar.JarFile.(JarFile.java:72)
at org.apache.hadoop.util.RunJar.main(RunJar.java:122)

What am I doing wrong?Am I using right jar files?Can you provide the above jar files?

hi harish,

the newer jars have started emitting this error on startup,
hive --service windowingCli -w /tmp/windowing/com.sap.hadoop.windowing-0.0.1-SNAPSHOT-jar-with-dependencies.jar
Hive history file=/tmp/ameet/hive_job_log_ameet_201202030904_2090640366.txt
hive-log4j.properties not found

I do have the hive-log4j.properties in /usr/lib/hive/conf/ dir and tried setting HIVE_HOME as well, but to no effect.

any thoughts?

ameet

Hi, I tested with the 0.2 release. Most part of the rank works as expected, but the lag function does not work. I am still use the employees table as my test data. For the following query:

from employees
partition by department_id
order by department_id, salary desc
select r, department_id, employee_id, salary,
< lag('salary',1) - salary > as salary_gap
into path='/tmp/wout';

I expected to see for each department_id, for each employee, I will list the employee's salary, plus the gap of its salary to the highest salary in each department. There is no error from the console about processing this query, but there is no data generated.

hadoop fs -ls /tmp/wout
12/04/18 22:32:24 INFO security.UserGroupInformation: JAAS Configuration already set up for Hadoop, not re-installing.
ls: Cannot access /tmp/wout: No such file or directory.

hive> from employees
> partition by department_id
> order by department_id, salary desc
> select r, department_id, employee_id, salary,
> < lag('salary',1) - salary > as salary_gap
> into path='/tmp/wout';
Starting Job = job_201204182053_0010, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201204182053_0010
Kill Command = /home/yzhang/hadoop/bin/hadoop job -Dmapred.job.tracker=localhost:8021 -kill job_201204182053_0010
Hadoop job information for : number of mappers: 2; number of reducers: 1
2012-04-18 22:33:03,238 map = 0%, reduce = 0%
2012-04-18 22:33:07,249 map = 100%, reduce = 0%
2012-04-18 22:33:14,296 map = 100%, reduce = 33%
2012-04-18 22:33:19,324 map = 100%, reduce = 0%
2012-04-18 22:33:26,372 map = 100%, reduce = 33%
2012-04-18 22:33:31,405 map = 100%, reduce = 0%
2012-04-18 22:33:39,445 map = 100%, reduce = 33%
2012-04-18 22:33:44,470 map = 100%, reduce = 0%
2012-04-18 22:33:51,521 map = 100%, reduce = 33%
2012-04-18 22:33:56,545 map = 100%, reduce = 0%
2012-04-18 22:33:57,549 map = 100%, reduce = 100%
Map: 2 Reduce: 1 HDFS Read: 15850 HDFS Write: 0 SUCCESS
Ended Job = job_201204182053_0010

$ hadoop fs -ls /tmp/wout
12/04/18 22:34:21 INFO security.UserGroupInformation: JAAS Configuration already set up for Hadoop, not re-installing.
Found 1 items
drwxr-xr-x - yzhang supergroup 0 2012-04-18 22:33 /tmp/wout/_logs

As you can see, I did not see the /tmp/wout/part-00000 file, which was generated when I tested the rank function, and contained the correct data as I expected in that case. But in lag function test, there is no file being generated

hi harish

can u enhancement the insert synatax ? like below sql query

from

insert into talb

Hi,

After completion of one windowing query, the second windowing query gets silence from the cli. Only when I type wmode hive and quit and log back in can I execute the query.
Does it make sense? or am I missing something?

ameet

I am trying to use a non-default db to issue the windowing queries. I see there is a patch for that but encountered some issue in building it. Can you send me (kasturi.chatterjee at gmail.com) a latest jar with the fix?

Thanks

Hi Harish,

the doc mentions that there are 16 functions implemented in the jar. Could you please list them with a brief description?
Was wondering whether the statistical functions from the reference ORacle doc are implemented. for e.g. median, mode etc.

thx

ameet