Recover OIC Process Project from deployed jar

 

The other day I deleted by mistake an OIC process App and did not have a backup.  But luckily, I had some already active versions deployed and was able to recover everything.

 

** I do not know if the next instructions work all the time or for other versions in the future, but work for me now.

 

-       JDeveloper version 12.2.1.4

-       OIC 21.4.1

 

Instructions

 

1.     Configure the “Instance Storage” in OIC. https://docs.oracle.com/en/cloud/paas/integration-cloud/oracle-integration-oci/configure-object-storage.html

 

2.     After that, do an export to the bucket it was created in the previous step.

 

3.     Download the zip file that is on the bucket, and unzipped.

 

4.     Go to the folder ‘Process’, and unzip the file called Process_export.par.

 

5.     Go into that folder and open ‘Process/composite/oracleinternalpcs’ and pick one of the .jar file that you want to use to recover the project.

 

6.     Open the jdeveloper and create a BPM Application and a Project using the empty composite template. (the name does not matter)

 

7.     Click on the Project on the folder structure, so it gets selected for the import.

 

8.     Click on the menu option ‘File->Import…’ and select ’Soa Archive into Soa Project’.

 

9.     Select you .jar file that you choose from before.

 

10.  Some windows with errors message can appear, ignore them.

 

11.  The composite.xml file gets opened. Close the file and save changes. (some other errors again). Close the jdeveloper.

 

12.  Go into the folder structure where the project was. (something like $HOME/jdeveloper/mywork/<app>/<project>)

 

13.  Create a zip file that contains the ‘SOA’ folder and its contents.

 

14.  Finally, change the extension from .zip to .exp.

 

15.  Open OIC and create a new app using the ‘import’ option.

 

That’s it, enjoy.

Weblogicbeat - Weblogic Monitoring with ELK

I had a previous entry about the configuration of logstash to capture the diagnostic, out, log, access, and garbage collector logs http://carlgira.blogspot.com/2017/09/logstash-configuration-for-weblogic.html . The idea of this entry is to explain a tool i just build to monitor a weblogic server using a custom beat of ELK (elasticsearh, logstash, kibana)

The tool is able to capture information about:

• Server state and health
• Memory use
• Threads status
• Datasource health and statistics
• Applications health and statistics

This information is sent to elasticsearch and kibana for metric visualization.

Configuration

- Download the tool from the releases link https://github.com/carlgira/weblogicbeat/releases

- Edit the file weblogicbeat.yml with your configuration.

Sample configuration file

weblogicbeat:
  period: 60s
  host: http://localhost:7001
  username: weblogic
  password: welcome1
  servername: server1
  datasources: ["EssDS", "EDNDataSource"]
  applications: ["ESSAPP", "sample-app"]

• period: How often an event is sent to the output
• host: Admin host and port
• username: Weblogic admin user
• password: Weblogic admin password
• servernam: Name of the server to monitor
• datasources: Array of datasources to monitor
• applications: Array of applications to monitor

- Run the tool

./weblogicbeat

Kibana

After starting weblogicbeat all the information should be arriving to kibana (remember to create an index with the name "weblogicbeat*") I create a set of visualizations and one dashboard to show all the metrics about weblogic. 

You can download the file https://github.com/carlgira/weblogicbeat/blob/master/kibana-weblogic-dashboard-export.json and import it to kibana. (follow instructions here https://www.elastic.co/guide/en/kibana/current/managing-saved-objects.html

Compilation

If you prefer to compile your own version you can go to the github repo and follow the README instructions. (or change it as you want). https://github.com/carlgira/weblogicbeat

Links of interest

• Using WebLogic 12C RESTful management for monitoring WebLogic Domains https://blog.dbi-services.com/using-weblogic-12c-restful-management-for-monitoring-weblogic-domains/
  
  
• Monitoring Domain Resources https://docs.oracle.com/middleware/1221/wls/WLRUR/examples.htm#WLRUR203
  
  
• Weblogic monitoring exporter https://github.com/oracle/weblogic-monitoring-exporter
  
  
• Jolokia https://jolokia.org/
  

Logstash Configuration for Weblogic

Probably the harder part to configure ELK (ElasticSearch, Logstash, Kibana) is to parse logs, get all fields correctly. I could not find a complete configuration to all types of logs of Weblogic for ELK, so i'm sharing mine.

Weblogic has four main different types of logs (server,  out, access, diagnostic),  every one with different format. I'm going to explain briefly the configuration of FileBeat and Logstash (for ElasticSearch and Kibana read their documentation Starting guide)

[update:14-08-2018] Added garbage collection logs patterns.

I'm sharing the configuration of Filebeat (as a first filter of logs), and logstash configuration (to parse the fields on the logs). Github repo https://github.com/carlgira/weblogic-elk

1. Use Filebeat as first filter to logstash. The idea is that you can have lots of these small apps running in every machine where you have your logs, for later feed to an instance of logstash (or elasticsearch directly). My intention was to include:

• The line that starts with one specific pattern (normaly the date)
• Exclude all the "Notification" or "Info" logs. (don't want those on logstash)
• Add a tag so logstash can identify what kind of log is (server,  out, access, diagnostic)
• No stacktraces just error codes.

     2. The information is feed to logstash instance that identify the type of log and using a custom "grok" can identify all fields in message.

• Logstash identify the type of log and apply a filter.
• The grok pattern is configured to parse all the fields of every kind of event and format every field
• Additional pattern file to control some other log structures
• Taking special care to format log dates (some of them were in es-ES locale and got some problems with timezones)
• Add a different prefix to all kinds of log so it can be easily identified in ElasticSearch/Kibana
• "wls" for server logs
• "out" for out logs
• "diag" for diagnostic logs
• "acc" for access logs
•  "gcc" for garbage colletion logs

TEST

Clone the github repo https://github.com/carlgira/weblogic-elk, you can test the configuration with the example logs.

• Download and install Filebeat, Logstash and ElasticSearch
• Install the plugin logstash-filter-translate
•  logstash-plugin install logstash-filter-translate
• Start ElasticSearch
• Start Logstash
• logstash -f weblogic-logstash-pipeline.conf --config.reload.automatic
• Start FileBeat
• filebeat -c weblogic-filebeat.yml

Check the output in the console of logstash or connect a Kibana to your ElasticSearch to see the results.

Thats all

Oracle Soa Suite Unit test with Groovy (11g, 12c) - part 2

A month ago I wrote a blog about how to create dynamic xml request with groovy (http://carlgira.blogspot.com.es/2016/05/oracle-soa-suite-unit-test-with-groovy.html i recommend to read that blog first). Since that, i'been working to make it better.

The idea is that you can use groovy code inside the XMLs of a test case to create dynamic values like dates, numbers, alphanumeric (you can do anything you want with groovy), but i wanted one more thing.

PROBLEM

For testing you probably have some pre and post conditions steps, probably you are going to need to share some information between service calls, and maybe you are going to need to get or assign a value to a field from outside the test case.

SOLUTION

The idea was to create a little framework that helps to create more robust test for the SOA Suite, executing and configuring the test case from a JUnit test. The solution allows to:

1. Create dynamic xml request with groovy.
2. Pre-load information to groovy for the execution of the test case.
3. Get a saved value from groovy to the java client, for post-processing or validation.

With these points you can get two scenarios. 

SAMPLE 1 - Simple Test Case (Execution of groovy code)

In the first scenario you get every variable loaded from the XML, (you don't pre-load any information or share fields between requests). all the information of the test case is on the XML request with the groovy code.

Groovy code on Xml request	Xml generated

SAMPLE 2 - Complex Test Case (Execution of groovy code, pre-load of data, saved data for post-processing)

The second example is more complex

Groovy code on Xml request	Generated XML

• A variable "name" is called but is not initialized in groovy. We are going to set that variable before start of the Test in Java with the value "outName"
• The variable "token" is initialized and is saved so it can be used or obtained later for post processing.

HOW IT WORKS

Those samples are executed from a Java client with the desired configuration.

The first sample just execute the test case as it is, the second one has a little more logic that i'm going to explain.

Java Code of Test Sample 2	Test results

1. The first thing is the definition of the names of the composite, test suite and test case that is going to be executed.
2. Before the execution of Test, the variable "name" is set with value "outName". (like a pre-step for the test)
3. Execution of Test.
4. The variable "token" that is set inside of the XML can be obtained after the execution of the test. You can see the value "74" for the token variable in the generated XML and in the test result. (like a post-step for the test)

I'm using two classes inside of the Java Test.

• GroovyShellService: The class has some utilities to create/delete a groovy shell and also functions to get/set variables inside a groovy shell of a test case. 
• UnitTestManager: It has some functions to execute test suites or test cases for a specific composite.

CODE

In Github you are going to find four projects

https://github.com/carlgira/soa-unit-test

• custom-test-case: Lib used to  to resolve the groovy code in the Xml requests.
• soa-unit-test-webapp: War with REST services to manage the creation/deletion/execution of the groovy shells for every test case.
• soa-unit-test-client: Lib with functions to call the REST services and also a utility class to call the test-suites and test-cases from the SOA suite.
• sample-unit-test-project: A simple class with the execution of the two test cases presented on this blog. Also has the sample SOA app for 11g and 12c so you can test it.

After installation the only thing you need to learn is how to use the two classes inside the soa-unit-test-client so you can build your own test cases. Check the sample inside sample-unit-test-project to know how to use them.

INSTALLATION/CONFIGURATION

 

You need to package, install and deploy the four projects.

• Follow all the instructions from http://carlgira.blogspot.com.es/2016/05/oracle-soa-suite-unit-test-with-groovy.html to install custom-test-case in Weblogic server.
• Open the pom.xml from soa-unit-test-client, and sample-unit-test-project and modify the variable of the "mdw.home" in the profile according with your version (11.1.1.7 or 12.1.3) and put the path to your SOA installation.
• Install in the maven repository the soa-unit-test-client. (11.1.1.7 or 12.1.3)
• mvn clean install -Dsoa-version=11.1.1.7 -Dmaven.test.skip=true
• mvn clean install -Dsoa-version=12.1.3 -Dmaven.test.skip=true
• Package soa-unit-test-webapp.
• mvn clean package -Dmaven.test.skip=true
• Deploy the generated war soa-unit-test-webapp in Weblogic.
• Install the SOA-app using the Enterprise manager from sample-unit-test-project/src/test/resources (11.1.1.7 or 12.1.3)
• soa11.1.1.7/sca_soa-test-project_rev1.0.jar
• soa12.1.3/sca_soa-test-project_rev1.0.jar
  
• Open the project sample-unit-test-project and in the class "SoaUnitTest" modify in the constructor the host, port, username and password. 
• In sample-unit-test-project  execute the tests (takes like 2 minutes)
• mvn clean test -Dsoa-version=11.1.1.7
• mvn clean test -Dsoa-version=12.1.3
• Check the two instances created and check that the two test cases are executed correctly.

Thats all,  thanks and i hope someone find this useful.

Oracle Soa Suite Unit test with Groovy (11g, 12c) - part 1

It's going to be a little long but if you stay until the last line, you are going to know what i did to create dynamic xml requests and responses usign the unit test framework of the Soa Suite with groovy. (yes, groovy!)

I'm going to give some background of the problem and later my own personal solution. Lets begin :D

[Update 30/06/2016] Wrote a second blog about subject with more functionality for the tests; JUnit test execution, pre load of dat and post processing of information. http://carlgira.blogspot.com.es/2016/06/oracle-soa-suite-unit-test-with-groovy.html

PROBLEM

I was trying to create Unit Test for some Bpel with several Web Service calls, human task, JCAs to database, and i get really frustated trying to create dynamic requests or responses using the tool within the Jdeveloper.

I wanted to re-create some fields, update dates etc.

The only thing i found was something that the TestSuites supports but the graphic wizards dont show. You can use small Xpath functions to replace values of the payloads in your TestSuite.

The next image shows the initiation message of a TestCase. You can see that after the payload, there is an element called "update". This element only receives two attributes, the "updateLocation" that refers a XpathLocation of a field to update, and the "updateXpathFunction" with the xpath function with the new value.

You could think that this could work but there is a problem with the Xpath functions you can use, there are only avalaible the "basic" Xpath functions https://www.w3.org/TR/1999/REC-xpath-19991116/#corelib ( Node-Set, String, Boolean and math functions - check the link with the full list)

But there are also some diferences between versions of the Soa Suite that i check:

Soa Suite 11.1.1.7: Only basic Xpath functions
Soa Suite 11.1.1.7.3: After this versions they add several functions of the http://www.oracle.com/XSL/Transform/java/oracle.tip.pc.services.functions.Xpath20 that you can use, (all of them related with dates and durations).
Soa Suite 12.1.3: I check again if there was more supported functions in 12c but it seems that again you only can use the basic Xpath functions. (lost all the Xpath20 functions dont know why)

So, there are few options to create dynamic XML in your unit test.

SOLUTION

I love the way the SoapUI gives you the possibility to create dynamic request using incrusted groovy inside of your XML. So i've spent some time looking for doing this possible with the Soa Suite.

I tried for long to found the "spot" of code that i should replace to make this possible, and i found it :D. The basic idea is that you can create dynamic XML request or response using incrusted groovy.

The best way to explain what it does is with an example.

•  When you build the request add some code of groovy. In the next example you can see how a date is created and later a random integer value.

When you execute the test, those values are replaced and you get a dynamic XML!!

HOW IT WORKS

I had to look for the spot with minimun changes. I found it in the class oracle.integration.platform.testfwk.TestCase in the fabric-ext.jar

I create an exact copy of that class and add some code to execute the groovy code inside the method "populatePayload".

Finally, the idea is to add my jar to the server classpath and make sure that is loaded first that the original one.

You can see everything inside the sources

INSTALLATION/CONFIGURATION

These are all the instructions to make it work.

• Download the sources from github https://github.com/carlgira/soa-unit-test/tree/master/custom-test-case

• Create the jar (Soa 11.1.1.7 or 12.1.3, use a custom profile for your version)
• mvn -Dsoa-version=11.1.1.7 clean package
• mvn -Dsoa-version=12.1.3 clean package

• Copy the file custom-test-case-1.0-jar-with-dependencies.jar to the directory Middleware\Oracle_SOA1\soa\modules\oracle.soa.fabric_11.1.1 in 11g or in the Middleware/soa/soa/modules/oracle.soa.fabric_11.1.1 on 12c.

• Add the custom-test-case-1.0-jar-with-dependencies.jar in the classpath of the MANIFEST file of the jar oracle.soa.fabric.jar. Make sure that this file appears before than the fabric-ext.jar in the classpath variable. (create a backup of jar before the modification)

•  Reboot your server

•  Now you can add to your XML messages in the Jdeveloper the groovy code :D

SUMMARY

• The utility allow you to create dynamic XML messages with groovy.
• You need to change the classpath to make sure to load the modified class.
• This utility is intended to your test environment.

Thanks! i hope someone find this useful.

Full Weblogic Custom Authenticator (11g, 12c)

I could not find a full example of custom weblogic asserter using maven. I got some examples using ant, but i dont like that... it never works for me for some reason, so i simply create a maven project using the sources of [1] getting some ideas of this github snippet wiht the "WebLogic MBean Maker" pom.xml [2], and most important the explanation on the creation of a custom authenticator in the book "Securing WebLogic Server 12c" chapter 4  (really recommended if you want to understand what you are doing).

There are lots of blogs with the same sources (SimpleSampleIdentityAsserterProviderImpl), so you are going to find the same example over and over again on internet, based in a really old sample used with Bea-server. In some moment in time the authenticator sources came with a sample web application to test it but you could not find that application anymore.

I don't want to extend to much with this, so i'll try to remark only the most important, and give you the basic instructions to make it work.

• First understand the difference between and authenticator and a asserter. The asserter is a way to "translate" a key or token to a set of credentials, that can be used to authenticate (lets say a cookie, http header or a certificate). The authenticator has the responsibility to check if with those credentials  the user can continue to the protected resource. (A better explanation can be found in the A-team blog about asserters [5])

          Why this is important? Because on many internet sources you are only going to get the asserter and not the full authenticator (the asserter plus the LoginModule). If you only get the asserter you are half way. Check [6] to get the LoginModule or this one [7] to extend your already mbean asserter to an authenticator provider.

• The second thing was to test it. It was funny, it was harder to find the right configuration for the web application than the sources of the authenticator. 

          With the authenticator we create a custom token. The key that the user must use, so the asserter gets activated.  A token can be a cookie, a http header or a certificate [8], so to test your application you must send the token the correct way (i use the cookie and the http header and both works fine)

        You also need to configure your web application with the protected resources, the roles and the principal mapping.

        In your web.xml add something like this (add the "login-config", the "security-role" and a "security-constraint")

 
  CLIENT-CERT
 

 
  LoggedUsers
 

 
  
   Protected resources
   protected/*
   GET
   POST
  
  
   LoggedUsers
  
 

       In your weblogic.xml add the mapping between the role and the principals.

        
  LoggedUsers
  users
 

**** For a beautiful example with the sources of the web application see [9]

CONFIGURE

The project is on github  https://github.com/carlgira/soa-utils/tree/master/http-token-authenticator

• In the pom.xml configure the "mdw.home" with your Middleware path (on the 11g or 12c profile)
• According with your version probably you'll have to change some jar paths.
• Execute maven install (11.1.1.7 or 12.1.3)
• mvn -Dsoa-version=12.1.3 clean install
• mvn -Dsoa-version=11.1.1.7 clean install
• Copy the jar to the path Middleware\wlserver_10.3\server\lib\mbeantypes
• Reboot your server
• Go to Security Realms->myrealm->Providers and create a SimpleSampleIdentityAsserter.
•  Make sure to put all the authenticators flags to "SUFFICENT"
• Reorder your authenticators and put the new one the last.
• Reboot your server

TEST A PROTECTED APPLICATION

I use the firebug add-on of Firefox to test it. Just create a custom cookie named "PerimeterAtnToken" and value "username=weblogic".

You can also make a http request with a http header named "PerimeterAtnToken" and with a value of "username=weblogic".

That should be enough, if not, make sure to check the "References" read the book and all the examples.

Thanks!

REFERENCES

1. Simple Sample Custom Identity Asserter for Weblogic Server 12c http://weblogic-wonders.com/weblogic/2014/01/13/simple-sample-custom-identity-asserter-weblogic-server-12c/
2. WebLogic MBean Maker. https://gist.github.com/kares/356576
3 Creating a wlfullclient.jar. https://docs.oracle.com/cd/E12840_01/wls/docs103/client/jarbuilder.html
4. Securing WebLogic Server 12c,
5. Why do I need an Authenticator when I have an Identity Asserter?, Oracle A-team http://www.ateam-oracle.com/why-do-i-need-an-authenticator-when-i-have-an-identity-asserter/
6. Do You Need to Develop a Custom Authentication Provider?http://docs.oracle.com/cd/E21764_01/web.1111/e13718/atn.htm#DEVSP220
7. Weblogic Identity Asserter and Athorization Provider in one! http://darylwiest.blogspot.com.es/2015/02/weblogic-identity-asserter-and.html
Extends the mbean to an authenticator
8. Passing Tokens for Perimeter Authentication http://docs.oracle.com/cd/E21764_01/web.1111/e13718/ia.htm#DEVSP254
9. SiteMinder WebLogic Security Provider Mock, https://gibaholms.wordpress.com/2015/01/21/siteminder-weblogic-security-provider-mock/
10. Mock Weblogic Login module - Identity Asserter and Authenticator,  http://danielveselka.blogspot.com.es/2012/04/mock-weblogic-login-module-identity.html

DeadLock Detector

DeadLock Detector

Two utilities to detect a deadlock inside a JVM.

See https://github.com/carlgira/soa-utils/tree/master/deadlock

 

 deadlock-detection

A remote deadlock detector. It connects to a JVM using JMX to detect if a deadlock is ocurring. It needs the host of the JVM and the JMX port.

  $ cd deadlock-detection
  $ mvn clean package
  $ java -jar target/deadlock-detection-1.0-SNAPSHOT.jar localhos:3333

deadlock-detector-service

A Weblogic Rest service to detect if is ocurring a deadlock in the server. It can deployed on Weblogic or tested with spring-boot

• Test with Spring-boot

  $ cd deadlock-detector-service
  $ mvn clean package
  $ java -jar target/deadlock-detector-service-1.0.0.war
  $ curl -X GET http://localhost:8080/deadlock

• Weblogic

  $ cd deadlock-detector-service
  $ mvn clean package
  $ Deploy to Weblogic 
  $ curl -X GET http://localhost:7001/deadlock-detector-service-1.0.0/deadlock

Thanks!