Thursday, 19 January 2017

Installing OpenIDM with a MySQL Database

I mentioned some time ago in the OpenIDM beginner series that although the built in OrientDB is more than sufficient for development purposes as soon as you start getting serious you will want to install a proper database backend.

This is a quick blog to talk you through how to install OpenIDM on a database. In my case I am going to use MySQL ( MariaDB specifically ), steps for installing other databases are broadly similar.


For this exercise please:



1. Create a directory for the environment, e.g. : /usr/local/env/demo
2. Unzip OpenIDM:


Open up openidm/conf/repo.orientdb.json  and take a quick look.

What we have here is a set of queries for the OOTB embedded OrientDB. We are going to replace this with a compatible SQL based version.

3. Run the following script to configure the OpenIDM schema:

mysql -u root -p < /usr/local/env/demo/openidm/db/mysql/scripts/openidm.sql

Here I run it for mysql against the root user. You will be asked for a password, please enter it. This script will create an openidm schema.

4. Next we need to run some additional scripts.

mysql -D openidm -u root -p < /opt/forgerock/openidm/db/mysql/scripts/activiti.mysql55.create.engine.sql
mysql -D openidm -u root -p < /opt/forgerock/openidm/db/mysql/scripts/activiti.mysql55.create.history.sql
mysql -D openidm -u root -p < /opt/forgerock/openidm/db/mysql/scripts/activiti.mysql.create.identity.sql

These scripts set up the Activiti BPMN workflow engine in the openidm database instance we just created.

5. Now lets remove the original repo.orientdb.json file. Don't forget to do this step! I'd also suggest making a backup:

cd /usr/local/env/demo/openidm/conf
mkdir backup
mv repo.orientdb.json backup

6. Next we need to copy over the database specific configuration files:

cd /usr/local/env/demo/openidm
cp db/mysql/conf/repo.jdbc.json conf
cp db/mysql/conf/datasource.jdbc-default.json conf

Again, let's take a quick look at these files , repo.jdbc.json:

You'll notice that what we have is an SQL version of the repo file we looked at earlier, ready to execute queries against a MySQL like database. There is also one other notable difference:

The original repo did not have this line. Well, this line is used to specify the datasource to be used i.e. our MySQL database, and it refers to the second file we copied over datasource.jdbc-default.json:

8. Ensure that the useDataSource (default) value matches the datasource specification datasource.jdbc-default.json

9. Configure the datasource.jdbc-default.json file for your MySQL database. Specifically the jdbcUrl, databaseName, username and password.

10. Update workflow.json to use the same datasource i.e. default.

To be very clear, if you named your datasource specification datasource.jdbc-example.json then your useDataSource value should be example in the files above.

11. That's it. You should be able to start openidm up normally:


Monday, 19 December 2016

OpenAM in 5 Minutes

At ForgeRock we pride ourselves on how quickly you can get up and running with our technology. So in that spirit I thought I would write a very quick, bare bones guide to getting an instance of OpenAM up and running.

For this exercise I am going to use Tomcat 8 as the web container for OpenAM. I am also going to use the embedded OpenDJ as both the user and configuration store. This is not recommended for production but is ideal for development purposes.

I am going to install locally on my Macbook, but the steps for Windows or Linux deployments are basically the same. I am going to cover the preparation of Tomcat and the installation and initial configuration of OpenAM. Everything you need to get up and running.


For this exercise please:
Feel free to swap in your web container of choice, but in the spirit of getting up and running in under 5 minutes I will be using Tomcat.



Make sure you have a sensible hostname configured with a top level domain.

sudo vi /etc/hosts

If not already present edit with:       localhost


1. Create a directory for the environment, I tend to use: /usr/local/env/fivemins
2. Unzip Tomcat:

mv apache-tomcat-8.5.9 tomcatam

3. Unzip OpenAM:

mv openam/OpenAM-13.5.0.war tomcatam/webapps/openam.war

4. Make Tomcat executable:

cd tomcatam/bin
chmod +x *.sh

5. Start Tomcat:

cd tomcatam/bin

6. Navigate to OpenAM:

Installing OpenAM

1. Select Create Default Configuration. By doing this OpenAM will use an embedded instance of OpenDJ as both a directory and user store with no configuration required. This is not recommended for production environments but is great for development.

2. Accept the license agreement and press Continue

3. Enter default passwords for the administrator and policy agent.

The Default User Password is the password you will use to log in to OpenAM as administrator. Make sure you remember it!

The Policy Agent User Password would be used if we were integrating a policy agent with OpenAM.

4. Press Create Configuration and wait a few moments...

5. Press Proceed to Login
6. Log in as amadmin with the Default User Password from earlier.

7. All being well, you should now be logged in as administrator

8. If you logout, you can also log in as the demo user:

9. Enter the username demo and the password changeit. Demo is a normal, non administrator user and as such can only see the user dashboard.


We have just installed and configured a vanilla instance of OpenAM, in around about 5 minutes. Obviously by itself this doesn't do much but in future blogs we will explore how to quickly implement functionality such as social login or two factor authentication and build on top of this simple exercise.

Tuesday, 29 November 2016

A Beginners Guide to OpenIDM - Part 6 - Provisioning to Active Directory

Last time we looked at configuring user registration. So now we have our users? How do we get them into our other systems? For example a user directory, in this case Active Directory (AD) ?

In this blog we will take a look at configuring OpenIDM provisioning, which consists of synchronisation to AD and reconciliation from AD. We will collect those accounts already in Active Directory and also create new AD accounts for our new identities.

Note that the concepts introduced here are easily applied to another LDAP ( such as OpenDJ ) or even other systems ( also likely to be covered in later blogs ). I have chosen Active Directory because it is a something I have commonly encountered throughout my career. Furthermore I am going to assume you have an Active Directory instance up and running. If not feel free to use OpenDJ or any other LDAP as the steps will be broadly similar.

Note: As many of you will be aware OpenIDM 4.5 has recently been released, for now I am going to continue using OpenIDM 4 for this blog but everything we talk about here is still applicable to 4.5, albeit with minor differences. I will likely move to OpenIDM 4.5 (or 5 even) when the beginners series is concluded.

This blog continues my OpenIDM Beginners series, catch up with the links below:

Provisioning in OpenIDM

I have always used provisioning as a bit of an umbrella term for both account creation and reconciliation. As we have discussed in previous blogs OpenIDM is based on the concepts of connectors and mappings. Connectors interface with external systems, while mappings define how attributes flow between OpenIDM and the external system.

So there are a few moving parts here. We need to:
  • Configure an AD connector.
  • Configure a mapping from AD accounts to an OpenIDM user object.
  • Configure a mapping to AD accounts from an OpenIDM user object.
So we have a little bit of work to do here. Lets get started.

Note: I will cover groups in another soon to be written blog entry.

Configuring the Active Directory Connector

Login to OpenIDM as the administrator. Navigate to connectors and create a new LDAP connector:

Select LDAP Type as AD LDAP Configuration, accept any prompts:

Enter your host name or IP, port, and Distinguished Name (DN) of the account you will be connecting as. 

Creating and updating users in Active Directory typically requires you to be connected over LDAPS (LDAP over SSL). For now Lets just get the connector working over LDAP. In a minute, we will configure LDAPS.

And the base context:

Then save your changes. As in the previous blog, to test if the connector is working we can view the accounts.

We should see a list of users in Active Directory:

Configuring LDAP over SSL

Now we have a working integration, we need to go back and configure SSL. First we need to get the AD LDAP SSL certificate into the OpenIDM truststore. The easiest way I have found to do this is as follows:

openssl s_client -connect -showcerts

The above command will retrieve the certificate from AD. You want to copy the certificate:


And copy it into another file somewhere as text e.g. ldaps.pem

Then you need to import into the OpenIDM truststore

keytool -import -alias ldaps -file ldaps.pem -keystore /usr/local/env/blog/openidm/security/truststore

Now, you can go back to the connector configuration and check the Over SSL box (ignore show certificate). Once done, save the connector and test. 

All being well it should still work and we can set about the interesting bit, provisioning!

Creating a Mapping

Now we need a mapping, navigate to configure, mappings, and create a new mapping from the User Object to our new User Directory. Note that this mapping, although data flows from OpenIDM to Active Directory, will manage provisioning of new AD accounts as well as reconciliation of existing ones and matching to user identities.

As before, we need to map AD attributes to user identity attributes. Let's just add the minimum required for now. Press "Add Missing Required Properties". This will add any required properties to the mapping as specified by the schema. In theory this should be the minimum set we need for the reconciliation to work.

Now we need to map dn and sn. Distinguished Name ( dn ) is an attribute that has to be generated from the data in our identity. Select dn and set a source property:

Press Update.

Select dn again and navigate to Transformation Script.

What we have here is an inline way to quickly write scripts to transform attributes.

Note: The source property we set above userName is used as the current value for the source value in the script above.

Enter the following inline script: 

"CN=" + source + ",OU=User Accounts,OU=trustzone1corp,DC=trustzone1,DC=forgedemo,DC=co,DC=uk"

Press Update.

Now we also need to map sn and typically we also need samAccountName. Add both in the usual way and we should have:

Note: I have had at least one report that you may also need to map the __ENABLE__ attribute here. If you get an issue, try that.

Press Save Properties.

Configuring Correlation

Then navigate to Association, Association Rules. Select Correlation Queries and Add Correlation Query.

Remember to set the Link Qualifier to default.

This will have the effect of linking AD accounts to users based on the sAMAccountName in AD matching the username in OpenIDM.

With this done, Submit, Save, then Save and Don't Reconcile ( you will see why in a moment ).

Navigate to Behaviors and as in the earlier blog, set the current policy to Default Actions and Save.

Remember, when a mapping is created this is set to Read-only, effectively a testing state for mappings where no changes will be made to data in OpenIDM or the target system. 

Now, scroll up and press Reconcile Now.

Expand In Progress:

And you should see the results of reconciliation, note we have 100 Absent conditions.

If you look at the Behaviors tab you should see that:

The Absent condition corresponds to the Create operation. So.. we should now have 100 users created in our Active Directory. Sure enough if you take a look in Active Directory Users & Computers that is indeed the case.

Reviewing Associations

Finally, you can take a closer look at the results in OpenIDM, again navigate to our mapping, view the Associations tab and scroll down. You should see something similar to the following:

If you play with the View selector you can see the results for each behavioural situation encountered.

Linked Systems

If you navigate to Manage, User, then select a user you should be able to see that they now have a Linked System for AD:

If matching accounts had already existed in AD, then OpenIDM would not re-create them. Instead they would be linked to a user as above based on the correlation query we defined earlier. If you quickly re-run the synchronization process we can confirm this:

This time we see Confirmed instead of Absent. As the accounts already exist.

All OpenIDM needs to do is Update the account if something has changed.


So in this blog we configured a basic provisioning integration between OpenIDM and Active Directory. Now typically you will want to set up more attributes, including a password. You will also want to reconcile and assign LDAP groups, probably based on roles. I plan to look at all of these requirements in a future blog building on the work we have already done here ( note it does not matter if you used AD or another LDAP for this exercise, any future exercise will still be valid ).

Hope you have found this useful!

Monday, 17 October 2016

OpenIDM Widgets Using the ELK Stack

OpenIDM 4.5 includes support for integration with ElasticSearch. For those not familiar, ElasticSearch is part of what is commonly referred to as the ELK stack:

E - ElasticSearch: Processes incoming data and allows for it to be quickly and RESTfully analysed and searched.
L - LogStash: A logging processor that takes input data, puts it through a pipeline of operations and outputs it somewhere ( usually into ElasticSearch ).
K - Kibana: Visualisation engine that utilises the data provided by ElasticSearch to create visual charts of all different types that can be embedded into web pages.

Together they enable you to create slick visualisations of data.

In the latest 4.5 release of OpenIDM two new features have been introduced:

  • An ElasticSearch audit handler, that can be used to push audit events into ElasticSearch.
  • Configurable dashboard widgets in the admin interface. There are a number of widgets available OOTB, one of which enables the embedding of Kibana visualisations.
This is incredibly powerful and relatively easy to set up, so I wanted to do a quick blog to explain how to do it.

In this blog we do not actually need LogStash as we will use the OpenIDM audit handler to push data to ElasticSearch. However ordinarily may well want to use LogStash as well to process and transform additional logging data ( as well as just audit data ).

Installing ElasticSearch & Kibana

The first step is to download and install ElasticSearch from

Simply unzip ElasticSearch somewhere and start it up:

cd /usr/local/env/elk/
mv elasticsearch-2.4.1 elasticsearch


You can check that it is running by navigating to, and you should see something like this:

Next, download and install Kibana from

Again, just unzip it somewhere:

cd /usr/local/env/elk
tar -xvf kibana-4.6.1-darwin-x86_64.tar.gz

But before starting it up check the following file:


And ensure it is pointing at your elasticsearch instance:

Then start it up:


You can navigate to Kibana on

If you can see this kibana splash page, then everyone is on track.

Configuring OpenIDM 4.5 for ElasticSearch Audit

Log into OpenIDM as the administrator and navigate to Configure, System Preferences

Then select the Audit tab

And add an ElasticSearch event handler

Give your handler a name, configure all the events you are interested in, and make sure Enabled is selected

Note: In earlier builds there was a bug where to enable the handler you had to manually edit conf/audit.json. If you follow all the steps and something is not working, double check that the handler is set to "enabled" : true

Scroll down and configure the handler parameters as below:

host: your host
port: 9200 ( unless you changed it)
indexName: audit
Press Submit then on the next page you must remember to scroll down and press Save

Now, put your elasticsearch process window somewhere you can see it, then log out and back in of OpenIDM and take a look. You should see some activity:

OpenIDM is now successfully pushing audit data to elasticsearch!

Processing Audit Data in Kibana

Now we need to configure Kibana to process and do things with all that juicy audit data from OpenIDM. Navigate to Kibana again and replace "logstash-*" with "audit, then hit Create:

We now have an Audit indices, and you should see the OpenIDM audit attributes have already been catalogued:

And if you click Discover you should be able to browse the data!

Creating a Visualisation

Now with our data, we can quickly create visualisations. There are many blogs on this subject so I won't go into detail here but I will create a quick visualisation to show failed authentications into OpenIDM over time.

The easiest way to do this is to generate such an event ( try logging in with the wrong password for example ) then take a look at the data.

We should see something like this:

And if we drill down further:

In Kibana's language ( based on Lucene to select these events we want a search filter like:

eventName:"authentication" AND result:"FAILED"

Now we have just failed authentication events, ( not successful ones) :

Hit Save and give the search it a name:

Now go to Visualize and select Vertical bar chart:

And use our saved search:

As you can guess this isn't quite right, but if we configure the x-axis and press Apply ( play button )
We get something that looks a bit more useful. Now if you try generating some failed authentications and refreshing the page you should see this update.

Save the visualisation.

Adding the Visualisation to OpenIDM

Now it is time to bring it all together. Select Share Visualisation

Examine the first URL ( the Embed URL ).

Copy the URL somewhere, just the http://localhost... discard the iframe tag e.g.,pause:!f,value:0),time:(from:now-1h,mode:quick,to:now))&_a=(filters:!(),linked:!t,query:(query_string:(query:'*')),uiState:(),vis:(aggs:!((id:'1',params:(),schema:metric,type:count),(id:'2',params:(customInterval:'2h',extended_bounds:(),field:timestamp,interval:m,min_doc_count:1),schema:segment,type:date_histogram)),listeners:(),params:(addLegend:!t,addTimeMarker:!f,addTooltip:!t,defaultYExtents:!f,mode:stacked,scale:linear,setYExtents:!f,shareYAxis:!t,times:!(),yAxis:()),title:'Failed%20IDM%20Authn',type:histogram))

Navigate to OpenIDM and log in as administrator, then Dashboards and New Dashboard:

Call it whatever you want and press Create:

Add a widget to your new dashboard:

Select Embed Webpage:

Then close the widgets menu (x):

Select Settings:

And enter the following values:

URL: The URL from the Kibana visualisation
Height: You may have to play with this a bit, but 200px is a good start.
Title: Whatever you like.

And press Save.

You should now see your visualisation in OpenIDM!


In around an hour, we have configured the ELK stack, integrated it with OpenIDM and created a (very) basic visualisation. This is really the tip of the ice berg of what you could do with OpenIDM and ELK and I plan to explore this in much more detail in future blogs and share any useful visualisations I come up with.