Customizing Oracle ODI Knowledge Modules

Oracle ODI ships with a bundle of very useful and quite diverse knowledge modules but at the same time one of the basic need it to get a customized version of the same knowledge module to get more control and gain more functionality. So how can we do that ?

Problem Statement : 
 
The Scenario we are taking about today is very simple one , if you are using any Integration Knowledge Module it creates a table with I$_  and do the further processing from their on wards. But here is a flaw , if you kick off the same job or any other job which loads the same target table ODI will launch a new instance and mess the current running job.

For example , I am using  "IKM Oracle Incremental Update" and what it does is it creates an table with I$ instead i wish to have it create a table post_fix with current SESSION number. e.g. I$

The steps are as follows
1 - Go to the project folder -->  Knowledge Modules -- > find IKM Oracle Incremental Update
2 - Right click on the selected module above and click on "Duplicate Selection".
3 - Give it a suitable name  e.g. "Custom_IKM Oracle Incremental Update"
4- Double click on the Module and click on "Details" to see the lost of steps for this module.
5- You would find a step with a title as " Create flow table I$".
6- Select that and lets see the code looks like as follows

create table <%=odiRef.getTable("L", "INT_NAME", "W")%>
(
    <%=odiRef.getColList("", "[COL_NAME]\t\t[DEST_WRI_DT] NULL", ",\n\t", "", "")%>,
    IND_UPDATE        CHAR(1)
)
<%=odiRef.getUserExit("FLOW_TABLE_OPTIONS")%>

7- Add the custom logic to append session number with the table being created.

create table <%=odiRef.getTable("L", "INT_NAME", "W")%>_<%=odiRef.getSession("SESS_NO")%>
(
    <%=odiRef.getColList("", "[COL_NAME]\t\t[DEST_WRI_DT] NULL", ",\n\t", "", "")%>,
    IND_UPDATE        CHAR(1)
)
<%=odiRef.getUserExit("FLOW_TABLE_OPTIONS")%>

 8- Save your knowledge module and that is it.
 9- Use the Custom Knowledge Module in your mapping .

You would see that the I$ table is created with a session post fix so any other instance of same job will not disturb the running flow.

Similarly you can add new steps , make them conditional and add new options to customize it more as per requirements.

Hope that helps.

Thanks



 

MS SQL Server Model not showing any objects in ODI

we had a requirement to connect to the MS SQL server 2008 as our source system and once issue we noticed that we were succesfull in connecting to the database while in the model view we were not able to see any objects.

The procedure to connect to the database is standard but there is one trick which we should understand while connecting to the SQL server database.

Please find below the screen shot when u click for new data server Go to Topology - > Physical Architecture - > Technology -> MS Sql Server ->  New Data Server

As below choose the mentioned JDBC drivers for SQL Server and give the IP and port. Plase make sure that remote access is anabled and your user has access on the required database.

Next step is to create a physical schema from this data server here you must keep in mind that in SQL Server database structure is different e.g In our case the database name is "dqaa" and we wish to extract some tables from it. The owner of schema is "DBO" and hence we need to set the setting as below because the property "Local Object Mask" will use that information to extract tables /views/objects from SQL server metadata . This was the main reason why we were not able to see any objects in the Model view after reverse engineering.

Hope that helps.

What are Pluggable database in Oracle & Setting up ODI Repository with Oracle 12c.

Oracle 12 c has recently introduced a new concept of plug-gable databases which actually aims to separate the metadata from business user data. The concept is to create a container that will hold the user data schema. The details about the plug-gable database and how they work can be referenced from below. 

http://oracle-base.com/articles/12c/multitenant-overview-container-database-cdb-12cr1.php

But here are few questions?

How do I connect to a pluggable database using SQL developer or TOAD?

The Answer is simple with oracle installation it creates and ask for the name of the pluggable database e.g. PDBORCL (in our case this is the pluggable database name). If you are not sure you can login to system using SYS user and execute below query. You can see I have a PDB with name as PDBORCL and its open. If it’s not open you can open it by issuing the second SQL statement below

select name, open_mode from v$pdbs;

PDBORCL             READ WRITE

Alter pluggable database all open;

Now you have to define a TNS entry in your oracletns.ora file as follows and you will be able to connect it using TOAD/SQL Developer.

PDBORCL=

  (DESCRIPTION=

    (ADDRESS=

      (PROTOCOL=TCP)

      (HOST=localhost)

      (PORT=1521)

    )

    (CONNECT_DATA=

      (SERVER=dedicated)

      (SERVICE_NAME=PDBORCL)

    )

  )

How do I Setup the ODI repository into a pluggable database using RCU?

The RCU utility will setup the repository for you in the oracle 12c, but if you have configured the pluggable database option the RCU pre check fails with below error.

RCU-6002: The specified database does not meet the minimum requirement to load metadata repository. RCU-6080: Global prerequisite check failed - Check requirement for specified database the selected Oracle database is a multitenant container database (CDB). Connecting to a multitenant container database (CDB) is not supported. Instead, connect to a valid pluggable database (PDB).

So to solve this issue while RCU asks for the details on “Database connection details” step provide the Service Name: PDBORCL instead of your main service name e.g. ORCL which is by default.  Doing so all the repository objects and user will be created in the PDBORCL database and later you will use the below steps to create or connect to your PDB based repository.

How do I connect to a pluggable database using ODI or any other application?

For my case we have an issue while connecting to the PDB database while setting up the ODI repository we did the following while filling up the JDBC URL. Giving only the service name will give error and to resolve it we did the following by providing the full TNS entry details.

jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=localhost)(PORT=1521)))(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=PDBORCL)))

Hope that helps.

Reference.

https://blogs.oracle.com/dataintegration/entry/odi_creating_a_repository_in

http://en.community.dell.com/techcenter/enterprise-solutions/w/oracle_solutions/5063.how-to-connect-to-pluggable-databases-in-oracle-rac-12cr1

http://www.areaetica.com/setup-oracle-12c-enterprise-manager-express-for-a-pdb/

http://blog.e-dba.com/blog/2013/06/29/oracle-database-12c-pluggable-databases/
http://www.youtube.com/watch?v=ynUURa5dA6Q
 

Exporting Data From Oracle to Flat File

Hi here's an example of how an sqlplus  export script should look if you would like to export data from Oracle to a flat file .

------

set colsep |     -- separate columns with a "|" character

set headsep off  -- remove heading separators

set linesize 1000   -- line size set to 1000 or whatever your maximum length is

set numw 20       -- 20 is the length of numbers, you set this at whatever length you want your numbers to be. This should also avoid getting numbers in scientific notations such as 1.77382E... etc.

set pagesize 0   -- no headers

set trimspool on -- get rid of trailing blanks set echo off -- so you don't get the data on the screen that slows down the process significantly... if you want to monitor the file you can use the unix command tail -f /usr/loadfiles/loadfile1.dat set feedback off -- no messages

spool /usr/loadfiles/loadfile1.dat -- location where the flat file will be exported... You might have to convert dates, numbers etc. to varchars

select col1,col2,col3,... from tableX  where ... ; -- query for downloading the data... You can put here your "latest data" vs. "cold data" filtering conditions etc.

spool off

exit

------

If you want everything into a single line you can use the concatenate query (without trailing blanks after every column): select col1|| '|' ||col2|| '|' ||col3,... from tableX  where ... In that case you don't need the "set colsep... statement). There are many ways of specifying the query. Also (this is trivial, but worth mentioning) be sure you have enough space and write permissions in the file system you are exporting the file.

You put the text between the two "------" into a file called (for example) script.sql or something similar, then run from the unix prompt as follows:

unix prompt> sqlplus username/password@oracle_connection @script.sql

You can put multiple sqlplus statements into a unix shell script for exporting from multimple sources/tables. Be aware that I did not test this, so you will have to experiment a bit in your specific environment, but it should give you a good conceptual idea on how to do it...