fcorti / alflytics

The Open Source Business Intelligence solution over the Alfresco platform.

Alflytics

Alflytics is the Open Source Business Intelligence (OSBI) solution over Alfresco content and processes, built using the Pentaho platform. With Alflytics is provided a pure Business Intelligence solution to extract, store and enquiry content data (documents, folders, metadata, tags, categories, etc.), audit trail and processes at a very detailed level, with the goal to be easily customized and extended with other entities coming from external Services like: ERP, CRM, custom Applications, etc.

Alflytics uses the data coming from Alfresco platform, the ability of the Pentaho platform to build reports, dashboards and free analysis, and the latest techniques in Data Warehousing, to define a complete set of analytics and insights on top of your preferred ECM+BPM solution.

Technical details

This project develops a Pentaho Sparkl Application to be released into an installation of the Pentaho Analytics platform. To deploy it, please follow the instructions below assuming you have a Pentaho Analytics platform v7.0 available in the <pentaho-server> folder.

Open a terminal and execute the commands detailed below.

cd ~
git clone https://github.com/fcorti/alflytics.git
cd alflytics
mkdir <pentaho-server>/pentaho-solutions/system/Alflytics
cp -R * <pentaho-server>/pentaho-solutions/system/Alflytics
rm -rf <pentaho-server>/pentaho-solutions/system/Alflytics/build
cd <pentaho-server>
./start-pentaho.sh

Documentation and support

For more informations about the project, support or further details, go throught the documentation.

alflytics's Issues

Error importing the audit event.

Corti, good evening.
I am with problems in extraction:

018/08/10 18:37:07 - Write to log.0 - ------------> Linenr 33077------------------------------
2018/08/10 18:37:07 - Write to log.0 - Error importing the audit event.
2018/08/10 18:37:07 - Write to log.0 - Probably the ID is extracted multiple times but at the moment this is how Alfresco works.
2018/08/10 18:37:07 - Write to log.0 -
2018/08/10 18:37:07 - Write to log.0 - entry = '{"id":90260,"application":"alfresco-access","user":"admin","time":"2015-11-07T23:30:59.467-02:00","values":{"/alfresco-access/login/user":"admin"}}'
2018/08/10 18:37:07 - Write to log.0 - fileName = 'source.1.fromId.90000.maxItems.5000.json'
2018/08/10 18:37:07 - Write to log.0 - filePath = /home/opt/pentaho-server/tomcat/temp/Alflytics/Alfresco/AuditTrail/alfresco-access
2018/08/10 18:37:07 - Write to log.0 - instanceId = 1
2018/08/10 18:37:07 - Write to log.0 - id = 90260
2018/08/10 18:37:07 - Write to log.0 - application = 'alfresco-access'

I have't idea how to solve.

Could you help me?

Thank you.

Roberto

File "Alflytics.datasource.analysis.xml" not found during the installation process

Hi,

I'm trying to install Alflytics (locally on my mac) following the instructions on the official manual.

The installation process fails with the following error (taken from catalina.out):

Generate multipart entity.0 - ERROR (version 7.0.0.0-25, build 1 from 2016-11-05 15.35.36 by buildguy) : File 'Users/david/Workspace/work/pentaho-server/pentaho-solutions/system/Alflytics/endpoints/kettle/admin/../src/5.2/repository/Alflytics.datasource.analysis.xml' not found!!

Alflytics: v5.0.EA
Pentaho: pentaho-server-ce-7.0.0.0-25

The error seems to be related to the wrong initialization of the "uploadAnalysis" variable in the _addDataSources.ktr file (an initial '/' is missing)

Windows installation issues

The problems found from some user are:

1. Unzipping Pentaho in Desktop will cause the paths to be too long. Suggest to unzip it in C:.
2. The file name web.xml.2017-07-08T14:09:53.033Z has invalid windows characters.
3. Uncompress pentaho to the desktop is also not a good idea.The problem is that there are jar files inside with big names which makes the destination path too big for windows!

Extraction Error in Alflytics

Hi Francesco,

I tried installing Alflytics with Alfresco community 5.2 . But Ended up in some error . I am new to the solution , request your help in identifying the error .

Alfresco Share 5.2
pentaho 7.1
Postgres DB
Installation Method : Manual

During the process of installation , there was no errors . However , we started having error from extraction process . please find the Extraction image , which show the details below .

I am also attaching the log file for reference .

Pentaho_20190421012018.log

Regards
Shabanash

No extraction activity

Hi:

I've installed Pentaho and Alflytics successfully, but when I try to push the Extract button, Catalina does not show me any REST call.... This is what I see: (I use HSQL)

2018/08/09 18:28:19 - readConfiguration - Spedizione iniziata per la trasformazione [readConfiguration]
2018/08/09 18:28:19 - DWH_ALF_DIM_INSTANCES.0 - Finished reading query, closing connection.
2018/08/09 18:28:19 - DWH_ALF_DIM_INSTANCES.0 - Elaborazione terminata (I=9, O=0, R=0, W=9, U=0, E=0)
2018/08/09 18:28:19 - OUTPUT.0 - Elaborazione terminata (I=0, O=0, R=9, W=9, U=0, E=0)
2018/08/09 18:28:28 - alflyticsExtract - Spedizione iniziata per la trasformazione [alflyticsExtract]
2018/08/09 18:28:28 - Data Grid.0 - Elaborazione terminata (I=0, O=0, R=0, W=1, U=0, E=0)
2018/08/09 18:28:28 - Variable substitution.0 - Elaborazione terminata (I=0, O=0, R=1, W=1, U=0, E=0)
18:28:28,345 ERROR [RepositoryResource] Repository file [:public:Alflytics:ETL:Get all.kjb] not found
2018/08/09 18:28:28 - HTTP Client.0 - Elaborazione terminata (I=0, O=0, R=1, W=1, U=0, E=0)
2018/08/09 18:28:28 - OUTPUT.0 - Elaborazione terminata (I=0, O=0, R=1, W=1, U=0, E=0)
2018/08/09 18:29:36 - readConfiguration - Spedizione iniziata per la trasformazione [readConfiguration]
2018/08/09 18:29:36 - DWH_ALF_DIM_INSTANCES.0 - Finished reading query, closing connection.
2018/08/09 18:29:36 - DWH_ALF_DIM_INSTANCES.0 - Elaborazione terminata (I=9, O=0, R=0, W=9, U=0, E=0)
2018/08/09 18:29:36 - OUTPUT.0 - Elaborazione terminata (I=0, O=0, R=9, W=9, U=0, E=0)
2018/08/09 18:30:01 - writeConfiguration - Inizio dell'esecuzione del job
2018/08/09 18:30:01 - writeConfiguration - Inizio entry [SQL]
2018/08/09 18:30:01 - writeConfiguration - Inizio entry [Success]
2018/08/09 18:30:01 - writeConfiguration - Terminata jobentry [Success] (risultato=[true])
2018/08/09 18:30:01 - writeConfiguration - Terminata jobentry [SQL] (risultato=[true])
2018/08/09 18:30:01 - writeConfiguration - L'esecuzione del job è terminata
2018/08/09 18:31:04 - alflyticsExtract - Spedizione iniziata per la trasformazione [alflyticsExtract]
2018/08/09 18:31:04 - Data Grid.0 - Elaborazione terminata (I=0, O=0, R=0, W=1, U=0, E=0)
2018/08/09 18:31:04 - Variable substitution.0 - Elaborazione terminata (I=0, O=0, R=1, W=1, U=0, E=0)
18:31:04,132 ERROR [RepositoryResource] Repository file [:public:Alflytics:ETL:Get all.kjb] not found
2018/08/09 18:31:04 - HTTP Client.0 - Elaborazione terminata (I=0, O=0, R=1, W=1, U=0, E=0)
2018/08/09 18:31:04 - OUTPUT.0 - Elaborazione terminata (I=0, O=0, R=1, W=1, U=0, E=0)
18:31:34,645 ERROR [GenericServlet] GenericServlet.ERROR_0004 - Resource /saiku-ui/saikuplugin.properties_supported_languages.properties not found in plugin saiku
18:31:34,663 ERROR [GenericServlet] GenericServlet.ERROR_0004 - Resource /saiku-ui/saikuplugin.properties.properties not found in plugin saiku
18:31:34,684 ERROR [GenericServlet] GenericServlet.ERROR_0004 - Resource /saiku-ui/saikuplugin.properties_it.properties not found in plugin saiku
18:31:34,726 ERROR [GenericServlet] GenericServlet.ERROR_0004 - Resource /saiku-ui/saikuplugin.properties_it_IT.properties not found in plugin saiku
2018/08/09 18:46:38 - readConfiguration - Spedizione iniziata per la trasformazione [readConfiguration]
2018/08/09 18:46:38 - DWH_ALF_DIM_INSTANCES.0 - Finished reading query, closing connection.
2018/08/09 18:46:38 - DWH_ALF_DIM_INSTANCES.0 - Elaborazione terminata (I=9, O=0, R=0, W=9, U=0, E=0)
2018/08/09 18:46:38 - OUTPUT.0 - Elaborazione terminata (I=0, O=0, R=9, W=9, U=0, E=0)
2018/08/09 18:47:02 - writeConfiguration - Inizio dell'esecuzione del job
2018/08/09 18:47:02 - writeConfiguration - Inizio entry [SQL]
2018/08/09 18:47:02 - writeConfiguration - Inizio entry [Success]
2018/08/09 18:47:02 - writeConfiguration - Terminata jobentry [Success] (risultato=[true])
2018/08/09 18:47:02 - writeConfiguration - Terminata jobentry [SQL] (risultato=[true])
2018/08/09 18:47:02 - writeConfiguration - L'esecuzione del job è terminata
2018/08/09 18:47:46 - alflyticsExtract - Spedizione iniziata per la trasformazione [alflyticsExtract]
2018/08/09 18:47:46 - Data Grid.0 - Elaborazione terminata (I=0, O=0, R=0, W=1, U=0, E=0)
2018/08/09 18:47:46 - Variable substitution.0 - Elaborazione terminata (I=0, O=0, R=1, W=1, U=0, E=0)
18:47:46,456 ERROR [RepositoryResource] Repository file [:public:Alflytics:ETL:Get all.kjb] not found
2018/08/09 18:47:46 - HTTP Client.0 - Elaborazione terminata (I=0, O=0, R=1, W=1, U=0, E=0)
2018/08/09 18:47:46 - OUTPUT.0 - Elaborazione terminata (I=0, O=0, R=1, W=1, U=0, E=0)

Thank you for your support.

Using a different port for Pentaho from 8080

Some errors appear using a different port for installation than 8080.

alflytics with postgres sql

Hello,
I am on a project where I have to provide statistics from alfresco thanks to alflytics. I used postgres. but I can't extract the data from alfresco. at each extraction my alflytics uses hsqldb.
thank you to enlighten me. thank you

Error fetching Alfresco data

My environment is (only server):
Cent OS 7.6.1810
Alfresco 5.2
Postgresql 9.6.14

In Postgresql it has the Alfresco database (alfresco), the Alflytics database (alflytics) and the postgres database.

I made the installation by connecting to the Postgresql database according to the manual.

Through the Spoon on my desktop I made the connection in Pentaho and changed the 5 scripts it has in Alflytics \ ETL successfully, pointing to postgresql and the Pentaho host url. In PeNtaho I created an ODBC Alflytics pointing to the Alflytics database in Postgresql.

When running Job Get all on Spoon, it does not feed the Warehouse tables, only Operational and Staging. And with that it returns no data to generate report.

And does pentaho.log and catalina.log show no direction for this error?

help me please?

spoon.log

pentaho.log

catalina.log

No jobs files on Pentaho Analytics platform path /public/Alflytics/ETL

Hi,
I'm having issue on importing Pentaho repository (section 3.4.1.4 Importing the Pentaho repository).
After restarting the server, no files generated on ETL and OLAP directories, only sub-folders was generated (as attached).
Is it a normal results or not?

Ubuntu 16.04
Pentaho Analytics 7.1
PostgreSQL 9.5
Alflytics 5.0

Regards,
[bayu]

put reports into Alfresco

hi Corti,

im using a PDI to execute the job ''report all'', and i would like publish the reports into the alfresco, I've followed the steps described on page 44 -5.3.3 of the Alflytics-Manual , also a i've update the table DWH_REPORTS setting the correct values for ALF_FTP_PORT and ALF_FTP_HOST.
the good news is that I got 9 reports published successfully in alfresco, the bad news is had the following error during the execution:

2018/09/19 14:11:01 - Put a file with FTP - ERROR (version 7.1.0.0-12, build 1 from 2017-05-16 17.18.02 by buildguy) : Error putting to FTP : Connection refused (Connection refused)
2018/09/19 14:11:01 - Put a file with FTP - ERROR (version 7.1.0.0-12, build 1 from 2017-05-16 17.18.02 by buildguy) : java.net.ConnectException: Connection refused (Connection refused)
2018/09/19 14:11:01 - Put a file with FTP - at java.net.PlainSocketImpl.socketConnect(Native Method)
2018/09/19 14:11:01 - Put a file with FTP - at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
2018/09/19 14:11:01 - Put a file with FTP - at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
2018/09/19 14:11:01 - Put a file with FTP - at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
2018/09/19 14:11:01 - Put a file with FTP - at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
2018/09/19 14:11:01 - Put a file with FTP - at java.net.Socket.connect(Socket.java:589)
2018/09/19 14:11:01 - Put a file with FTP - at com.enterprisedt.util.proxy.PlainSocket.createPlainSocket(PlainSocket.java:122)
2018/09/19 14:11:01 - Put a file with FTP - at com.enterprisedt.net.ftp.FTPControlSocket.(FTPControlSocket.java:366)
2018/09/19 14:11:01 - Put a file with FTP - at com.enterprisedt.net.ftp.FTPClient.connect(FTPClient.java:970)
2018/09/19 14:11:01 - Put a file with FTP - at org.pentaho.di.job.entries.ftpput.JobEntryFTPPUT.execute(JobEntryFTPPUT.java:597)
2018/09/19 14:11:01 - Put a file with FTP - at org.pentaho.di.job.Job.execute(Job.java:724) at org.pentaho.di.job.entries.job.JobEntryJobRunner.run(JobEntryJobRunner.java:69)
2018/09/19 14:11:01 - Put a file with FTP - at java.lang.Thread.run(Thread.java:748)

Could you help me to solve this issue?

Thank you.

Dercio.

Comma in SQL command on Windows platform

The SQL query (for HyperSql)

MERGE INTO DWH_DIM_DATES_YEARS USING (SELECT -1,0 AS ID, 'Unknown' AS DESCRIPTION FROM (VALUES(0))) AS VALS(ID,DESCRIPTION)

Seems to fail because of the comma in "-1,0" on Windows platform.

& character in endpoint result are badly managed

In some environments (also on linux), endpoint like:

/alfresco/api/-default-/public/alfresco/versions/1/people?maxItems=${maxItems}&skipCount=${skipCount}
/alfresco/api/-default-/public/alfresco/versions/1/people?maxItems=1000000000&skipCount=1&fields=%7B%7D

http://localhost:8080/pentaho/plugin/Alflytics/api/getendpoints?paramendpointId=alfrescoPeople

http://localhost:8080/pentaho/plugin/Alflytics/api/getendpoints?paramendpointId=alfrescoPeopleCount

Can cause a bad result:

http://192.10.2.41:80/alfresco/api/-default-/public/workflow/versions/1skipCount=0
http://192.10.2.41:80/alfresco/api/-default-/public/workflow/versions/1fields=%7B%7D

The & seems to be a problem in some environments.

Bug Wrapped Exception (with status template): null

2018/05/17 21:23:48 - Abort.0 - ERROR (version 7.1.0.0-12, build 1 from 2017-05-16 17.18.02 by buildguy) : Row nr 1 causing abort : [{}], [http://10.39.17.122:8080/alfresco/service/api/audit/query/alfresco-access?fromId=785000&limit=5000&verbose=true], [GET], [785000,0], [{
2018/05/17 21:23:48 - Abort.0 - "status" :
2018/05/17 21:23:48 - Abort.0 - {
2018/05/17 21:23:48 - Abort.0 - "code" : 500,
2018/05/17 21:23:48 - Abort.0 - "name" : "Internal Error",
2018/05/17 21:23:48 - Abort.0 - "description" : "An error inside the HTTP server which prevented it from fulfilling the request."
2018/05/17 21:23:48 - Abort.0 - },
2018/05/17 21:23:48 - Abort.0 -
2018/05/17 21:23:48 - Abort.0 - "message" : "04170003 Wrapped Exception (with status template): null",
2018/05/17 21:23:48 - Abort.0 - "exception" : "",
2018/05/17 21:23:48 - Abort.0 -
2018/05/17 21:23:48 - Abort.0 - "callstack" :
2018/05/17 21:23:48 - Abort.0 - [
2018/05/17 21:23:48 - Abort.0 -
2018/05/17 21:23:48 - Abort.0 - ],
2018/05/17 21:23:48 - Abort.0 -
2018/05/17 21:23:48 - Abort.0 - "server" : "Community v5.2.0 (re21f2be5-b22) schema 10.057",
2018/05/17 21:23:48 - Abort.0 - "time" : "17/05/2018 21:23:48"
2018/05/17 21:23:48 - Abort.0 - }
2018/05/17 21:23:48 - Abort.0 -
2018/05/17 21:23:48 - Abort.0 - ], [500]
2018/05/17 21:23:48 - Abort.0 - ERROR (version 7.1.0.0-12, build 1 from 2017-05-16 17.18.02 by buildguy) : Rest call failed.

Data extraction failed

Hi ,
I am trying to extract data (http://localhost:9000/pentaho/plugin/Alflytics/api/extract) and I get the following exception.

`org.pentaho.di.core.exception.KettleStepException:
Error while running this step!

Couldn't execute SQL: INSERT INTO "DWH_DIM_DATES_YEARS" ("ID","DESCRIPTION")
(
SELECT -1.0 AS "ID", 'Unknown' AS "DESCRIPTION"
)
ON CONFLICT ("ID") DO UPDATE SET
"DESCRIPTION"=EXCLUDED."DESCRIPTION"

ERROR: syntax error at or near "ON"
Position: 111

at org.pentaho.di.trans.steps.sql.ExecSQL.processRow(ExecSQL.java:236)
at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
at java.lang.Thread.run(Thread.java:745)

Caused by: org.pentaho.di.core.exception.KettleDatabaseException:
Couldn't execute SQL: INSERT INTO "DWH_DIM_DATES_YEARS" ("ID","DESCRIPTION")
(
SELECT -1.0 AS "ID", 'Unknown' AS "DESCRIPTION"
)
ON CONFLICT ("ID") DO UPDATE SET
"DESCRIPTION"=EXCLUDED."DESCRIPTION"

ERROR: syntax error at or near "ON"
Position: 111

at org.pentaho.di.core.database.Database.execStatement(Database.java:1561)
at org.pentaho.di.core.database.Database.execStatements(Database.java:1661)
at org.pentaho.di.trans.steps.sql.ExecSQL.processRow(ExecSQL.java:211)
... 2 more

Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at or near "ON"
Position: 111
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2198)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1927)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:561)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:405)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:397)
at org.pentaho.di.core.database.Database.execStatement(Database.java:1535)
... 4 more
2018/01/08 18:48:57 - Execute SQL script.0 - ERROR (version 7.1.0.0-12, build 1 from 2017-05-16 17.18.02 by buildguy) : Unexpected error
2018/01/08 18:48:57 - Execute SQL script.0 - ERROR (version 7.1.0.0-12, build 1 from 2017-05-16 17.18.02 by buildguy) : org.pentaho.di.core.exception.KettleStepException:
2018/01/08 18:48:57 - Execute SQL script.0 - Error while running this step!
2018/01/08 18:48:57 - Execute SQL script.0 -
2018/01/08 18:48:57 - Execute SQL script.0 - Couldn't execute SQL: INSERT INTO "DWH_DIM_DATES_YEARS" ("ID","DESCRIPTION")
2018/01/08 18:48:57 - Execute SQL script.0 - (
2018/01/08 18:48:57 - Execute SQL script.0 - SELECT -1.0 AS "ID", 'Unknown' AS "DESCRIPTION"
2018/01/08 18:48:57 - Execute SQL script.0 - )
2018/01/08 18:48:57 - Execute SQL script.0 - ON CONFLICT ("ID") DO UPDATE SET
2018/01/08 18:48:57 - Execute SQL script.0 - "DESCRIPTION"=EXCLUDED."DESCRIPTION"
2018/01/08 18:48:57 - Execute SQL script.0 -
2018/01/08 18:48:57 - Execute SQL script.0 - ERROR: syntax error at or near "ON"
2018/01/08 18:48:57 - Execute SQL script.0 - Position: 111
2018/01/08 18:48:57 - Execute SQL script.0 -
2018/01/08 18:48:57 - Execute SQL script.0 -
2018/01/08 18:48:57 - Execute SQL script.0 - at org.pentaho.di.trans.steps.sql.ExecSQL.processRow(ExecSQL.java:236)
2018/01/08 18:48:57 - Execute SQL script.0 - at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
2018/01/08 18:48:57 - Execute SQL script.0 - at java.lang.Thread.run(Thread.java:745)
2018/01/08 18:48:57 - Execute SQL script.0 - Caused by: org.pentaho.di.core.exception.KettleDatabaseException:
2018/01/08 18:48:57 - Execute SQL script.0 - Couldn't execute SQL: INSERT INTO "DWH_DIM_DATES_YEARS" ("ID","DESCRIPTION")
2018/01/08 18:48:57 - Execute SQL script.0 - (
2018/01/08 18:48:57 - Execute SQL script.0 - SELECT -1.0 AS "ID", 'Unknown' AS "DESCRIPTION"
2018/01/08 18:48:57 - Execute SQL script.0 - )
2018/01/08 18:48:57 - Execute SQL script.0 - ON CONFLICT ("ID") DO UPDATE SET
2018/01/08 18:48:57 - Execute SQL script.0 - "DESCRIPTION"=EXCLUDED."DESCRIPTION"
2018/01/08 18:48:57 - Execute SQL script.0 -
2018/01/08 18:48:57 - Execute SQL script.0 - ERROR: syntax error at or near "ON"
2018/01/08 18:48:57 - Execute SQL script.0 - Position: 111
2018/01/08 18:48:57 - Execute SQL script.0 -
2018/01/08 18:48:57 - Execute SQL script.0 - at org.pentaho.di.core.database.Database.execStatement(Database.java:1561)
2018/01/08 18:48:57 - Execute SQL script.0 - at org.pentaho.di.core.database.Database.execStatements(Database.java:1661)
2018/01/08 18:48:57 - Execute SQL script.0 - at org.pentaho.di.trans.steps.sql.ExecSQL.processRow(ExecSQL.java:211)
2018/01/08 18:48:57 - Execute SQL script.0 - ... 2 more
2018/01/08 18:48:57 - Execute SQL script.0 - Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at or near "ON"
2018/01/08 18:48:57 - Execute SQL script.0 - Position: 111`

And when I go to url http://localhost:9000/pentaho/plugin/Alflytics/api/check
there are couple of extraction extraction logs failed they are Alfresco users in warehouse, Alfresco audit trail in warehouse, Alfresco repository in warehouse, Alfresco processes in staging and
Data quality is empty.

Any help would be of great help

Thanks.

Get REST in json for audit trail Error

Hi @fcorti ,

At first i would like to congratulate you for the project, it has been very useful.

Recently i've sucessfully install the Alflytics into production enviroment for tests reason, but i have problems while execute the job get+all using the follow script on PDI kitchen:

sudo sh /opt/PENTAHO/data-integration/kitchen.sh -rep=Alflytics_TESTE -user=admin -pass=****-job=Get+all -dir=/public/Alflytics/ETL -logfile=/home/alfrescoadm/alflytics.log -level=Basic -param:alflytics.dbType=mysql -param:alflytics.getData.alfresco.entity.audits=true-param:alflytics.getData.alfresco.entity.processes=true -param:alflytics.getData.alfresco.entity.processes.getItems=true -param:alflytics.getData.alfresco.entity.repository=true -param:alflytics.getData.alfresco.entity.repository.removeDeleted=true -param:alflytics.pentaho.protocol=http -param:alflytics.pentaho.host=****-param:alflytics.pentaho.port=8080 -param:alflytics.pentaho.login=****-param:alflytics.pentaho.password=****

I will share with you some error that I found in the log file jsut bellow:

Unexpected error occurred while launching entry [hasMoreItems?.0]
2019/04/26 15:39:49 - Get REST in json for audit trail -
2019/04/26 15:39:49 - Get REST in json for audit trail - Unexpected error occurred while launching entry [Set targetFile.0]
2019/04/26 15:39:49 - Get REST in json for audit trail -
2019/04/26 15:39:49 - Get REST in json for audit trail - Unexpected error occurred while launching entry [Get REST in json for audit trail.0]
2019/04/26 15:39:49 - Get REST in json for audit trail -
2019/04/26 15:39:49 - Get REST in json for audit trail - Unexpected error occurred while launching entry [hasMoreItems?.0]
2019/04/26 15:39:49 - Get REST in json for audit trail - Unexpected error occurred while launching entry [Get REST in json for audit trail.0]

other wise when i set the param param:alflytics.getData.alfresco.entity.audits=false the job runs perfectly.

I appreciate your help.

No data found

I've installed Alflytics with Java 8 + Pentaho 7.1 + Alfresco CE 5.2
I've manual installed Alflytics, download .zip and uncompress in system directory.
I have performed the steps indicated in the manual and performed an extraction.

Then, I have published the data with the intention of seeing the results obtained ... but the data do not appear.

My objective and reason why I use Alflitycs is because I would like to obtain statistics of workflows that I have created and I would like to obtain execution times, user that executes them, start date / end date. Do you recommend another tool?

Extraction Error in Alflytics with Alfresco v6: Rest call failed

Hi,
I am facing issue with extraction process of alflytics. It works fine with the Alfresco version 5.2. I tried running it with Alfresco 6 and I get below error. Does Alflytics work with Alfresco version 6?
Here is the log

Get Alfresco repository from Alfresco in tmp - Starting entry [Create targetPathForCategories]

2019/08/01 14:33:59 - Get Alfresco repository from Alfresco in tmp - Starting entry [Get REST in json for categories]
2019/08/01 14:33:59 - Get REST in json - Starting entry [Set hasMoreItems]
2019/08/01 14:33:59 - Get REST in json - Starting entry [Set targetFile]
2019/08/01 14:33:59 - Get REST in json - Starting entry [Get REST in json]
2019/08/01 14:33:59 - Get REST in json - Loading transformation from repository [Get REST in json] in directory [/public/Alflytics/ETL/Staging/Util]
2019/08/01 14:33:59 - Get REST in json - Using legacy execution engine
2019/08/01 14:33:59 - Get REST in json - Dispatching started for transformation [Get REST in json]
2019/08/01 14:34:00 - Data Grid.0 - Finished processing (I=0, O=0, R=0, W=1, U=0, E=0)
2019/08/01 14:34:00 - Variables replace.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2019/08/01 14:34:00 - Select values.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2019/08/01 14:34:00 - Abort.0 - ERROR (version 7.1.0.0-12, build 1 from 2017-05-16 17.18.02 by buildguy) : Row nr 1 causing abort : [{"query":{"query":"(TYPE:"cm:category")","language": "afts"},"sort":[{"type":"FIELD","field":"cm:modified","ascending": true},{"type":"FIELD","field":"sys:dbid","ascending":true}],"include":["properties"],"paging":{"maxItems": 5000,"skipCount":0}}], [http://localhost:8081/alfresco/api/-default-/public/search/versions/1/search], [POST], [{"error":{"errorKey":"framework.exception.ApiDefault","statusCode":500,"briefSummary":"07010026 ","stackTrace":"For security reasons the stack trace is no longer displayed, but the property is kept for previous versions","descriptionURL":"https://api-explorer.alfresco.com","logId":"97b9016f-6f9a-4047-bc97-402f806edc49"}}], [500]
2019/08/01 14:34:00 - Abort.0 - ERROR (version 7.1.0.0-12, build 1 from 2017-05-16 17.18.02 by buildguy) : Rest call failed.
2019/08/01 14:34:00 - Abort.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=1)
2019/08/01 14:34:00 - Get REST in json - Transformation detected one or more steps with errors.
2019/08/01 14:34:00 - Get REST in json - Transformation is killing the other steps!
2019/08/01 14:34:00 - Set Variables.0 - We didn't receive a row while setting the default values.
2019/08/01 14:34:00 - Set Variables.0 - Finished after 0 rows.
2019/08/01 14:34:00 - Switch / Case.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2019/08/01 14:34:00 - Get REST in json - ERROR (version 7.1.0.0-12, build 1 from 2017-05-16 17.18.02 by buildguy) : Errors detected!
2019/08/01 14:34:00 - REST Client.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2019/08/01 14:34:00 - Get REST in json - ERROR (version 7.1.0.0-12, build 1 from 2017-05-16 17.18.02 by buildguy) : Errors detected!
2019/08/01 14:34:00 - Get REST in json - Finished job entry [Get REST in json] (result=[false])
2019/08/01 14:34:00 - Get REST in json - Finished job entry [Set targetFile] (result=[false])
2019/08/01 14:34:00 - Get REST in json - Finished job entry [Set hasMoreItems] (result=[false])

Problem with Jobs when running in Data integration !!!!

Good Afertnoon
I have problems with the execution of the Jobs that are generated in the integration with Alfresco, I generated the following problem in the majority of Jobs:

2017/10/11 15:57:40 - Dynamic SQL row.0 - ERROR (version 7.1.0.0-12, build 1 from 2017-05-16 17.18.02 by buildguy) : Because of an error, this step can't continue:
2017/10/11 15:57:40 - Dynamic SQL row.0 - Error obtaining fields for this step
2017/10/11 15:57:40 - Dynamic SQL row.0 -
2017/10/11 15:57:40 - Dynamic SQL row.0 - Couldn't get field info from []
2017/10/11 15:57:40 - Dynamic SQL row.0 -
2017/10/11 15:57:40 - Dynamic SQL row.0 - La consulta no retornó ningún resultado.
2017/10/11 15:57:40 - Get Alfresco classes in variable - ERROR (version 7.1.0.0-12, build 1 from 2017-05-16 17.18.02 by buildguy) : Errors detected! `

also sends me to this route too 👍

http://localhost:8091/alfresco/api/discovery

my port is 8091 where Alfresco runs, but there is a username and password that I already try with the one we normally use with alfresco but it will not let me enter

You could help me with this error Thank you very much

not fetching any data

Hi Francesco ,

Greetings ...

I tried to install AAAR initially , but had some errors . After your guidance I tried installing the Alflytics . But I am facing below issue .

OS : - Ubuntu 18.10 server
Pentaho :- pentaho-server-ce-7.1.0.0-12
Alfresco :- 5.2
Database :- Postgres

Issue :- Not extracting any information and found that there is no files in the :public:Alflytics:ETL:Get all.kjb. I checked the /pentaho-solutions/system/default-content and ensured that there is zip file and are getting renamed as file name with timestamp . but there is not file getting generated in the Public Repositry of Pentaho.

Logs:-
2019/04/14 15:43:03 - Get Dashboard Style.0 - Finished processing (I=0, O=0, R=2, W=2, U=0, E=0)
2019/04/14 15:43:03 - remove auxiliary fields.0 - Finished processing (I=0, O=0, R=2, W=4, U=0, E=0)
2019/04/14 15:43:03 - Detect empty stream.0 - Finished processing (I=0, O=0, R=2, W=0, U=0, E=0)
2019/04/14 15:43:03 - remove cpk_filename.0 - Finished processing (I=0, O=0, R=2, W=2, U=0, E=0)
2019/04/14 15:43:03 - OUTPUT.0 - Finished processing (I=0, O=0, R=2, W=4, U=0, E=0)
2019/04/14 15:43:03 - Clean fields.0 - Finished processing (I=0, O=0, R=2, W=2, U=0, E=0)
2019/04/14 15:43:03 - LOG1 2.0 - Finished processing (I=0, O=0, R=2, W=2, U=0, E=0)
2019/04/14 15:43:03 - getPluginMetadata.0 - Finished processing (I=0, O=0, R=2, W=4, U=0, E=0)
2019/04/14 15:43:03 - Sort by pluginId.0 - Finished processing (I=0, O=0, R=2, W=2, U=0, E=0)
2019/04/14 15:43:03 - OUTPUT.0 - Finished processing (I=0, O=0, R=2, W=2, U=0, E=0)
2019/04/14 15:43:57 - alflyticsExtract - Dispatching started for transformation [alflyticsExtract]
2019/04/14 15:43:57 - Data Grid.0 - Finished processing (I=0, O=0, R=0, W=1, U=0, E=0)
2019/04/14 15:43:57 - Variable substitution.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
15:43:57,835 ERROR [RepositoryResource] Repository file [:public:Alflytics:ETL:Get all.kjb] not found
2019/04/14 15:43:57 - HTTP Client.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2019/04/14 15:43:57 - OUTPUT.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)

Empty JSON input tmp files

In all the "tmp to table" jobs you should check if JSON are empty.
Some users have this issue.
/public/Alflytics/ETL/Get json in table
...

Get data from Alfresco using spoon

hi one more time,

im tryng to get data from the alfresco instance usign spoon and mysql workbench 6.3 as database for data warehouse.
i've execute the job "Get all" and i had the following erros:

No tran_Status found. Transformation status checking not available.
2018/09/19 16:39:49 - Alflytics - launch DDL statement:
2018/09/19 16:39:49 - Alflytics - INSERT INTO OPE_ALF_AUDIT_ACCESSES(INSTANCE_ID,ID,USER,USER_LOGIN,USER_LOGIN_FAILURE,USER_LOGOUT,TIME,ACTION,NODE_PATH) VALUES (1,1,'admin','admin',null,null,CONVERT_TZ(STR_TO_DATE('2018-09-05T15:04:03.10', '%Y-%m-%dT%H:%i:%s.%f'),'+02:00','+00:00'),null,null)
2018/09/19 16:39:50 - Alflytics - 1 statement executed
2018/09/19 16:39:50 - SQL creation.0 - ERROR (version 7.1.0.0-12, build 1 from 2017-05-16 17.18.02 by buildguy) : Unexpected error
2018/09/19 16:39:50 - SQL creation.0 - ERROR (version 7.1.0.0-12, build 1 from 2017-05-16 17.18.02 by buildguy) : org.pentaho.di.core.exception.KettleValueException:
2018/09/19 16:39:50 - SQL creation.0 - Javascript error:
2018/09/19 16:39:50 - SQL creation.0 - SyntaxError: String contains control character (getInserts#3)
2018/09/19 16:39:50 - SQL creation.0 -
2018/09/19 16:39:50 - SQL creation.0 - at org.pentaho.di.trans.steps.scriptvalues_mod.ScriptValuesMod.addValues(ScriptValuesMod.java:475)
2018/09/19 16:39:50 - SQL creation.0 - at org.pentaho.di.trans.steps.scriptvalues_mod.ScriptValuesMod.processRow(ScriptValuesMod.java:541)
2018/09/19 16:39:50 - SQL creation.0 - at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
2018/09/19 16:39:50 - SQL creation.0 - at java.lang.Thread.run(Thread.java:748)
2018/09/19 16:39:50 - SQL creation.0 - Caused by: org.mozilla.javascript.EcmaError: SyntaxError: String contains control character (getInserts#3)
2018/09/19 16:39:50 - SQL creation.0 - at org.mozilla.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3785)
2018/09/19 16:39:50 - SQL creation.0 - at org.mozilla.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3763)
2018/09/19 16:39:50 - SQL creation.0 - at org.mozilla.javascript.NativeJSON.parse(NativeJSON.java:146)
2018/09/19 16:39:50 - SQL creation.0 - at org.mozilla.javascript.NativeJSON.execIdCall(NativeJSON.java:122)
2018/09/19 16:39:50 - SQL creation.0 - at org.mozilla.javascript.IdFunctionObject.call(IdFunctionObject.java:129)
2018/09/19 16:39:50 - SQL creation.0 - at org.mozilla.javascript.optimizer.OptRuntime.call1(OptRuntime.java:66)
2018/09/19 16:39:50 - SQL creation.0 - at org.mozilla.javascript.gen._getInserts__84._c_getInserts_1(getInserts:3)
2018/09/19 16:39:50 - SQL creation.0 - at org.mozilla.javascript.gen._getInserts__84.call(getInserts)
2018/09/19 16:39:50 - SQL creation.0 - at org.mozilla.javascript.optimizer.OptRuntime.callName(OptRuntime.java:97)
2018/09/19 16:39:50 - SQL creation.0 - at org.mozilla.javascript.gen.script_87._c_script_0(script:2)
2018/09/19 16:39:50 - SQL creation.0 - at org.mozilla.javascript.gen.script_87.call(script)
2018/09/19 16:39:50 - SQL creation.0 - at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:426)
2018/09/19 16:39:50 - SQL creation.0 - at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3178)
2018/09/19 16:39:50 - SQL creation.0 - at org.mozilla.javascript.gen.script_87.call(script)
2018/09/19 16:39:50 - SQL creation.0 - at org.mozilla.javascript.gen.script_87.exec(script)
2018/09/19 16:39:50 - SQL creation.0 - at org.pentaho.di.trans.steps.scriptvalues_mod.ScriptValuesMod.addValues(ScriptValuesMod.java:388)
2018/09/19 16:39:50 - SQL creation.0 - ... 3 more
2018/09/19 16:39:50 - Get Alfresco audit trail from staging in operational - ERROR (version 7.1.0.0-12, build 1 from 2017-05-16 17.18.02 by buildguy) : Errors detected!
2018/09/19 16:39:50 - Source table.0 - Finished reading query, closing connection.

2018/09/19 16:39:50 - Get Alfresco audit trail from staging in operational - ERROR (version 7.1.0.0-12, build 1 from 2017-05-16 17.18.02 by buildguy) : Errors detected!

Abort.0 - ERROR (version 7.1.0.0-12, build 1 from 2017-05-16 17.18.02 by buildguy) : Row nr 1 causing abort : [{"query":{"query": "(((TYPE:"cm:content") OR (TYPE:"cm:folder")) AND NOT (TYPE:"cm:systemfolder") AND (@cm:modified:["2001-01-01T00:00:00.000Z" TO MAX]))","language":"afts"},"paging":{"maxItems":1,"skipCount":1000000000}}], [http://172.31.4.120:8080/alfresco/api/-default-/public/search/versions/1/search], [POST], [{"error":{"errorKey":"framework.exception.ApiDefault","statusCode":500,"briefSummary":"08191512 ","stackTrace":"For security reasons the stack trace is no longer displayed, but the property is kept for previous versions","descriptionURL":"https://api-explorer.alfresco.com","logId":"4a8578ac-f28e-4086-9e70-f179ac657326"}}], [500]
2018/09/19 16:39:53 - Abort.0 - ERROR (version 7.1.0.0-12, build 1 from 2017-05-16 17.18.02 by buildguy) : Rest call failed.
2018/09/19 16:39:53 - Abort.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=1)
2018/09/19 16:39:53 - Get Alfresco counter - Transformation detected one or more steps with errors.
2018/09/19 16:39:53 - Get Alfresco counter - Transformation is killing the other steps!
2018/09/19 16:39:53 - Get Alfresco counter - ERROR (version 7.1.0.0-12, build 1 from 2017-05-16 17.18.02 by buildguy) : Errors detected!

after running this job I could notice that some tables were not filled for some reason, I will attach some screen captures to help analyze the problem.

I appreciate your help.

integrity constraint violation: NOT NULL check constraint

hello,

I think this is a bug: when retrieving the tasks from Alfresco, Alflytics is expecting to get an assignee for every task, but some tasks may have no assignee in a specific time, for instance, when a task is sent to a group and no one accepted it yet.

This is the error:

2017-09-18 15:57:54,684 ERROR [org.pentaho.di] 2017/09/18 15:57:54 - Execute row SQL script.0 - ERROR (version 7.1.0.0-12, build 1 from 2017-05-16 17.18.02 by buildguy) : Error in step, asking everyone to stop because of:
2017/09/18 15:57:54 - Execute row SQL script.0 - Couldn't execute SQL: INSERT INTO "OPE_ALF_WF_TASKS"("INSTANCE_ID","ID","WF_INSTANCE_ID","NAME","DESCRIPTION","STARTED_AT","DUE_AT","ASSIGNEE","STATE","PRIORITY","TASK_DEFINITION_ID") VALUES (1,'50505','49571','Description','My title 3213',TO_DATE('2017-09-12T16:20:44.466+0000', 'YYYY-MM-DD"T"HH24:MI:SS.FFF'),null,null,'unclaimed',2,'alfrescoUsertask5')
2017/09/18 15:57:54 - Execute row SQL script.0 -
2017/09/18 15:57:54 - Execute row SQL script.0 - integrity constraint violation: NOT NULL check constraint; SYS_CT_10677 table: "OPE_ALF_WF_TASKS" column: ASSIGNEE

So, it seems the column restrinction should be eliminated.

thank you!

Cannot extract data from alfresco

I have install alflytics manual with postgresql 9.4.
When i extract data, nothing extracted. And in log has an error below:
2018/09/14 22:12:44 - alflyticsExtract - Dispatching started for transformation [alflyticsExtract]
2018/09/14 22:12:44 - Data Grid.0 - Finished processing (I=0, O=0, R=0, W=1, U=0, E=0)
2018/09/14 22:12:44 - Variable substitution.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2018/09/14 22:12:54 - HTTP Client.0 - ERROR (version 7.1.0.0-12, build 1 from 2017-05-16 17.18.02 by buildguy) : Because of an error, this step can't continue:
2018/09/14 22:12:54 - HTTP Client.0 - Unable to get result from specified URL : http://103.237.144.27:9080/pentaho/api/repos/:public:Alflytics:ETL:Get%20all.kjb/viewer
2018/09/14 22:12:54 - HTTP Client.0 - The host did not accept the connection within timeout of 10000 ms
2018/09/14 22:12:54 - HTTP Client.0 - Finished processing (I=0, O=0, R=1, W=0, U=0, E=1)
2018/09/14 22:12:54 - alflyticsExtract - Transformation detected one or more steps with errors.
2018/09/14 22:12:54 - alflyticsExtract - Transformation is killing the other steps!

Add to documentation the update of alflytics.dbPassword

From Pablo:
I have noticed that in addition to the parameter "alflytics.dbType" I have to change the password of the connection data to the database because it comes with the default password (postgres), this is not indicated in the manual, maybe it would be good to include it.

Check the status of auditing on the alfresco platform and be able to activate/deactivate it

It would be nice to be able to check status of auditing on alfresco platform and be able to activate/deactivate it from Alflytics administration.

Extraction error

Hi,

I've managed to install and configure Alflytics_v5_EA but when I try to perform an "extraction" from Alfresco, I get the following error (from catalina.out):

[...]
REST call failed..0 - ERROR (version 7.0.0.0-25, build 1 from 2016-11-05 15.35
.36 by buildguy) : La riga n. 1 ha causato l'annullamento: [{}], [https://xx.xx.xx.xx:8443/alfresco
/api/discovery], [GET], [<html><head>
2018/06/08 16:56:41 - REST call failed..0 - </head><body><p>Please <a href="/alfresco/api/discovery"
>log in</a>.</p>
2018/06/08 16:56:41 - REST call failed..0 - </body></html>
2018/06/08 16:56:41 - REST call failed..0 - ], [401]
2018/06/08 16:56:41 - REST call failed..0 - ERROR (version 7.0.0.0-25, build 1 from 2016-11-05 15.35
.36 by buildguy) : REST call failed. Please check you Alfresco instance is available.
[...]

The Alfresco instance is available and reachable from the alflytics server (it responds with the xml fragment visible in the log).
It seems like the basic authetication used by the api/discovery call does not respond in the correct way.

Is there a workaround to resolve this problem?

Alflytics
Alflytics_v5_EA
pentaho-server-ce-7.0.0.0-25
SO: centos 7.3

Alfresco:
Alfresco Content Service: 5.2.3
SSL enabled on the 8443 port
Authentication: internal + Active Directory