How to generate sub-categories for WSO2 features

We can generate sub-categories for WSO2 features by creating product-specific composite features to group existing features based on their functionality.
You can follow below steps to create a composite-feature and include it as a sub-category under a WSO2 product in P2. 

1. Identify the sub-categories for your product and create corresponding composite features. (eg: For AS, the identified sub-categories are listed in the previous post.)  Also, you can see the list of composite-features for AS category at: https://svn.wso2.org/repos/wso2/carbon/platform/branches/4.0.0/features/composite-features/as 
Similarly we can create composite-features for other products (eg: composite-features/esb, composite-features/greg)

2. In the new composite-feature, import the required features you want to group together. Use importFeatureDef to import required features instead of using includedFeatureDef. This is to avoid tight-coupling to it's dependency features. 

eg: In the "Application and Service Hosting" composite feature defined for AS we can import the set of features we need to group together as below.

        org.wso2.carbon.servicehosting:4.0.0
        org.wso2.carbon.service.mgt:4.0.5
        org.wso2.carbon.dataservices:4.0.5
        org.wso2.carbon.webapp.mgt:4.0.5
        org.wso2.carbon.jaxws.webapp.mgt:4.0.2
        org.wso2.appserver.mashup:4.0.5
        org.jaggeryjs:0.9.0.ALPHA2-wso2v2
        org.wso2.carbon.identity.xacml.filter:4.0.5

We should also make sure that these imported features are already published to the p2-repository under repo-gen goal. Otherwise during feature installations, P2 will not be able to resolve these dependencies as the required features are not in the repository.

So we should make sure the featureArtifacts of above imported features are in the list of published features in the p2-repository.

Note that in the above sample, I have imported org.wso2.appserver.mashup:4.0.5 feature, which is another composite feature introduced to group mashup related features set. Likewise we can nest composite-features in a parent composite-feature to gain a more fine-grained sub-categories.

3. Set the following properties for the composite feature in the p2-advice file (p2.inf)
 

org.eclipse.equinox.p2.type.category:true
org.wso2.carbon.p2.category.type:composite

This is how we are defining this feature is a composite-feature and a sub-category. Above properties are read by the feature manager to identify the feature as a composite type and list as a sub-category.

4. Following is the composite-feature pom created for "Application and Service Hosting" sub-category under AS. Note that the groupID is product specific: org.wso2.appserver, which is necessary to maintain composite-features based on products.

 
  org.wso2.carbon
  carbon-features
  4.0.0
  ../../../../pom.xml
 

 4.0.0
 org.wso2.appserver
 org.wso2.appserver.service.application.hosting.feature
 4.0.5
 pom
 WSO2 Carbon - Application and Service Hosting
 http://wso2.org
 This feature includes service and application hosting
  features

 
  
   
    org.wso2.maven
    carbon-p2-plugin
    ${carbon.p2.plugin.version}
    
     
      4-p2-feature-generation
      package
      
       p2-feature-gen
      
      
       org.wso2.appserver.service.application.hosting
       ../../../../etc/feature.properties
       
        
         org.eclipse.equinox.p2.type.category:true
         org.wso2.carbon.p2.category.type:composite
        
       
       
        org.wso2.carbon.servicehosting:4.0.0
        org.wso2.carbon.service.mgt:4.0.5
        org.wso2.carbon.dataservices:4.0.5
        org.wso2.carbon.webapp.mgt:4.0.5
        org.wso2.carbon.jaxws.webapp.mgt:4.0.2
        org.wso2.appserver.mashup:4.0.5
        org.jaggeryjs:0.9.0.ALPHA2-wso2v2
        org.wso2.carbon.identity.xacml.filter:4.0.5
       
      
     
    
   
  
 

5. Add the newly added composite-feature to the p2-repo pom.xml under <featureArtifacts>section, so the artifact is published to the repository at the repo-gen phase.

                             org.wso2.appserver:org.wso2.appserver.service.application.hosting.feature:${carbon.patch.version.405}

6. Finally, add the new composite-feature as a <catFeature> under the relevant product category.

 org.wso2.appserver.service.application.hosting
 ${carbon.patch.version.405}

Here are the list of new AS-specific composite features published to the p2-repo.

 org.wso2.appserver:org.wso2.appserver.miscellaneous.feature:${carbon.patch.version.405}


 org.wso2.appserver:org.wso2.appserver.mashup.feature:${carbon.patch.version.405}


 org.wso2.appserver:org.wso2.appserver.monitoring.feature:${carbon.patch.version.405}


 org.wso2.appserver:org.wso2.appserver.module.mgt.feature:${carbon.patch.version.405}


 org.wso2.appserver:org.wso2.appserver.tools.feature:${carbon.patch.version.405}


 org.wso2.appserver:org.wso2.appserver.registry.feature:${carbon.patch.version.405}


 org.wso2.appserver:org.wso2.appserver.data.publisher.feature:${carbon.patch.version.405}


 org.wso2.appserver:org.wso2.appserver.transports.feature:${carbon.patch.version.405}


 org.wso2.appserver:org.wso2.appserver.service.application.hosting.feature:${carbon.patch.version.405}


 org.wso2.appserver:org.wso2.appserver.service.application.security.feature:${carbon.patch.version.405}


 org.wso2.appserver:org.wso2.appserver.mandatory.feature:${carbon.patch.version.405}

Now the AS category looks like this with the newly added composite-features to mimic sub-categories

 org.wso2.carbon.appserver.category
 Application Server
 This category contains features required for WSO2
  Application Server
 
 
  
   org.wso2.appserver.monitoring
   ${carbon.patch.version.405}
  
  
   org.wso2.appserver.tools
   ${carbon.patch.version.405}
  
  
   org.wso2.appserver.transports
   ${carbon.patch.version.405}
  
  
   org.wso2.appserver.service.application.hosting
   ${carbon.patch.version.405}
  
  
   org.wso2.appserver.mandatory
   ${carbon.patch.version.405}
  
  
   org.wso2.appserver.miscellaneous
   ${carbon.patch.version.405}
  
 

Here is how AS category is now displayed in Feature manager UI.

WSO2 Carbon Feature Sub-Categories

Equinox P2 provides a concept called P2 category to group related installable units (IUs) in a P2 repository.  In WSO2 Carbon we use the same concept to implement Carbon feature categories in our P2 repo. In our maven based build platform we use carbon-p2-plugin tool, to generate features, feature-repository, product profiles and feature-categories. To generate P2-categories using the Carbon p2-plugin we use P2 CategoryPublisher tool under-the-hood. I have explained how we can create P2-categories for Carbon features to represent a WSO2 product in one of my previous posts.

Following is how 'Application Server' category is shown as a P2-category in the repository. All required features to install Application Server is listed under the AS category as shown below.

 
However there's a limitation in P2 category concept which restricts us from achieving fine-grained feature sub-categorization. Sub-categorization of features under a Product can be really useful from a user's point-of-view to identify the set of minimally required features to gain  functionality of a certain WSO2 product. Using P2 category concept and CategoryPublisher tool we can only categorize features in a single level (Product level) as shown above. We cannot achieve a hierarchical sub-categorization which can map features to user-level feature groups based on their functionality. So to achieve sub-categorization of features we must find an alternate approach.

As a solution for this, and to implement sub-categories for WSO2 Carbon features, we can follow a process to create new set of composite features to group required features as sub-categories. A newly created composite feature will correspond to a sub-category in a certain Product. They can be specific to a product, because each product can have different grouping of the same platform features based on the expected functionality.

For an example if we take Application Server, we can group the list of features under AS category into sub-categories by defining new set of composite features as below.

1. Application and Service Hosting
2. Application Server - Core (contains the core set of features that must be installed with AS)
3. Application Server -Transports (jms, mail transport features)
4. Miscellaneous (dep.sync, event, tasks, ws-discovery features)
5. Monitoring
6. Tools

Please note that it's also possible to define more composite-features to group related carbon features and have them under other composites as imported-features.
For example under Application Server we can create a composite-feature as 'Mashup' and have it as an imported-feature under 'Application and Service Hosting' sub-category.
 
With above sub-categories defined and required features grouped into sub-categories, the AS category is now shown in the P2 repository like below.

In the next post, I will explain a step-by-step approach to create feature sub-categories under Products.

My Presentation at the OSGi Community Event 2012

 OSGi Community Event 2012 on "Reducing Complexity with OSGi" was held in Ludwigsburg, Germany between Oct 23-25, 2012. The event was co-located with EclipseCon Europe and it was a great opportunity for me to represent WSO2 and give a technical talk on how we have reduced complexity in building a server platform using OSGi. Here is the slide-deck I used for my talk.

Building a server platform with osgi from Dileepa Jayakody

How to view disk space in Linux

1. How to list the disk space in linux. Use df (disk filesystem) command
> df -h


Output
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda6              66G   48G   15G  77% /
udev                  3.9G  4.0K  3.9G   1% /dev
tmpfs                 1.6G  912K  1.6G   1% /run

There are many additional parameters you can use with this tool. Just try df --help to list the options.

Guidelines for Carbon Feature Developers

1. Follow the general feature hierarchy when developing Carbon features

• Aggregate/Composite feature
• Server feature
• Console feature
• Common feature (if any)

2.  Follow the general naming convention in features : Just like the feature’s ID is important to identify features at runtime , the name is important for the users to identify features uniquely in the UI. So every feature’s name should follow a convention, be unique and should be self-describing.

3. Use p2 properties meaningfully in the advice file given at the p2-feature-gen goal in your feature’s pom.xml

• org.wso2.carbon.p2.category.type: server | console | common

Above property is used to define whether the feature is a backend (server), UI (console) or a common feature. This property is read by feature manager for BE, FE filtering

• org.eclipse.equinox.p2.type.group: true | false

This property is important to be explicitly defined as ‘false’ when the feature is not a   group type feature and you don’t want it to be listed in feature manager UI. (We list down group type features in the feature manager UI)

For sub features (console, ui) which  are dependencies to the aggregate-feature use this property as false.

Usage Sample :

<adviceFile>

<properties>

               <propertyDef>org.wso2.carbon.p2.category.type:server</propertyDef>

                       <propertyDef>org.eclipse.equinox.p2.type.group:false</propertyDef>

           </properties>

</adviceFile>

4. Correctly give the relative path to the feature.properties in the feature’s pom: This properties file   contains the license and copyright text to be viewed in feature manager UI. The file is located at branches/4.0.0/features/etc/feature.properties.

The relative path from your feature to this file needs to be given correctly in order to load the license and copyright in the UI

Usage Sample:

<propertiesFile>../../../etc/feature.properties</propertiesFile>

5. Use correct carbon-p2-plugin instruction to construct the features

• bundleDef : Use this when you need to include a component/bundle in your feature. The bundle included is considered as an integral part of the feature and doesn’t support updates during a new feature installation.

Usage Sample :

<bundles>

           <bundleDef>org.wso2.carbon:org.wso2.carbon.security.mgt</bundleDef>

</bundles>

• includedFeatureDef : Use this when you need to include a sub-feature in your composite feature. The included feature is considered a part of your composite feature and it is not enforced by new installations and updates are not supported at the time of installation.

(eg: Include the console, server and common features in your composite feature. And set org.eclipse.equinox.p2.type.group: true only in your composite feature)

Usage Sample :

<includedFeatures> <includedFeatureDef>org.wso2.carbon:org.wso2.carbon.core.common.feature:4.0.2</includedFeatureDef>
<includedFeatures>

• importBundleDef : Use this when you require an external bundle outside your component as a dependency for your feature. This is considered as an optional feature dependency and if specified, is enforced by new installations and supports update at the time of installation.

Usage Sample :

<importBundles>

              <importBundleDef>org.wso2.xkms.wso2:xkms</importBundleDef>

</importBundles>

• importFeatureDef : Use this when you require an external feature as a dependency for your feature. This is considered as an optional feature dependency and if specified, is enforced by new installations and supports update at the time of installation.

Usage Sample :

<importFeatures>    <importFeatureDef>org.wso2.carbon.core.server:${wso2carbon.version}</importFeatureDef>

</importFeatures>

6. Use p2-touchpoint instructions to perform additional configuration operations for the feature.

Using touchpoint instructions we can do things like copying configuration files to target system during a p2 phase (collect, configure, install, uninstall, unconfigure). You can define these touchpoint instructions in the p2.inf file of your feature.

Usage Sample :

instructions.configure = \

org.eclipse.equinox.p2.touchpoint.natives.copy(source:${installFolder}/features/org.wso2.carbon.apimgt.publisher_${feature.version}/publisher/,target:${installFolder}/../deployment/server/jaggeryapps/publisher,overwrite:true);\

Common mistakes in creating features

• Not using  org.eclipse.equinox.p2.type.group:false property in sub-features: Because of this we can see low-level  sub-features listed as top-level features in the featuremanager UI.
  
  
• Not giving the relative path correctly to feature.properties. This causes the license and copyright not being loaded in the feature manager UI.

• Including external features instead of properly defining external dependencies as ImportedFeatures: This makes the included features tightly coupled to the composite feature and doesn’t allow to be updated to a newer version during a new feature installation. This leads to further complications in installing several features/categories at once as there could be newer versions of the included features as dependencies which are supposed to be target update

• Not following the naming convention of features: Some of the existing features have repeated names for sub-features as well as it’s composite. This will confuse the user as it will be hard to differentiate features in the UI. It will also give the impression that a certain feature is duplicated.

• Not using p2-touchpoint instructions to copy required configuration files to the target directories during feature installation phase: Some of our features require additional configuration files copied to the target system in-order to the system to function properly. But in many features in our platform the required config files are copied to the target directory at product build time (in product assembly). This will break the feature installation because the required config files are not copied to target system at feature installation time. This could lead to system malfunction after a feature installation.

How to define a Feature Category to represent a WSO2 Product

In the last post I've explained how to install feature-categories in Carbon from Feature Manager. In this blog post I will briefly discuss how to publish a feature category to WSO2 p2-repo.

A feature category in the p2-repo represents a certain WSO2 product. (eg: Application Server, Data Services Server etc..) The category will categorize all required features to constitute the particular product. It makes life easy for a user to install a set of features relevant to a product at once by selecting the relevant product category.

eg: if the user wants to install Application Server on top of Carbon, the user can simply select the "Application Server" category and install all features grouped under that category.

The process of publishing a feature category to p2-repo should follow below steps;

1. First you need to know all the features required to install your product; this is defined in the product's p2-profile-gen.

eg: All features required to construct Application Server is defined in it's p2-profile-gen

Then in the features/repository/pom.xml;

2. Make sure all the required features are published to the repository in the carbon-p2-plugin's p2-repo-gen goal. So all required features should be included as <featureArtifact>s. 

3. Now create a <category> to represent your product under <categories> section which represents all the feature categories published in the p2-repo.

4. Include the relevant feature list as <catFeature>s under your category except the product's styles feature. A catFeature represent a required p2-feature for a category. Please note that, <id> and <version> are required for a <catFeature>. 

<id> is the feature's p2-artifact-id given in the feature's pom under p2-feature-gen goal of the carbon-p2-plugin. If a version for the p2-artifact isn't explicitly defined, it will inherit the feature's maven artifact's version.

(eg: In platform/features/admin-mgt/org.wso2.carbon.admin.mgt.feature/pom.xml the p2 artifact-id given for org.wso2.carbon.admin.mgt.feature is org.wso2.carbon.admin.mgt )

5. The styles feature should not be grouped under the Product category by default, because then if the user selects several product categories at once, it will try to install several product styles at once and break the server's style. The solution is to have a seperate feature category for Product Styles. Then the user can select the product style to install separately.

6. As the final step, it's important to test the installation of your category on Carbon at-least once, to see if there are any missing requirements for the installation. 

So following above 6 simple steps you can define a category of features to cluster the required set of features to install a WSO2 Product on top of Carbon.

How to install features via Feature Manager in Carbon 4.0

Carbon is a modular middleware framework for enterprise applications. It's power yields from it's compose-able architecture using which users can compose their own Carbon Products by installing software components on top of  Carbon. These reusable software units are called 'features'. The process of installing features on a target system is called 'provisioning'.  In Carbon, provisioning is facilitated via the Feature Manager.

Feature Manager

Carbon Feature Manager facilitates the users to easily install features on top of Carbon. It provides the main provisioning actions (install, uninstall, update, revert) using Equinox p2 under the hood. Users can install features from either a remote p2 repository or a local repository. A p2 repository contains the things to be installed (artifacts) and their metadata (InstallableUnits) so a provisioning agent (client) can connect to these p2 repositories and perform provisioning actions.

Basically the functionality of the Feature Manager can be considered as 2 main aspects; 

• Repository Management (Lets you to add,delete,edit p2 repositories. You can add remote repositories giving a URL and local repositories giving the local-path )  
• Provisioning Management (Lets you install, uninstall, update features or revert to a previous installation)

You can follow the Carbon documentation on how to use Feature Manager here.

 

What's new in Feature Manager in Carbon 4.0?

Carbon 4.0 (the upcoming release) has several improvements in Feature Manager.  As I said earlier Feature Manager can be used to compose carbon products (eg: Application Server, Data Services Server etc.) by installing the required list of features on top of Carbon. For this you need to know, the list of features required to install to gain the capabilities of a particular WSO2 Product. 

In Carbon 4.0 this is made easy for the users. Now you can install new products on Carbon using 'Feature Categories' from a p2 repository. You don't need to remember the list of features required for a particular product as the p2 repository for Carbon is now published with feature categories to represent each Carbon product. A feature category is basically a logical group of required features for a particular product. For example all the required features to install Application Server is categorised under 'Application Server' category.


The new Feature Manager comes with an option to enable 'Group features by category' when you list down available features from a p2 repository.  If you disable this option you can still see the flat list of features provided from the repository without the category groupings, just as in previous releases (3.2.0, 3.2.1,..). 

You can follow the below steps to install a WSO2 product on Carbon using feature categories.

1. Add a Carbon p2 repository published with feature category metadata. You can find a successful build of Carbon4 p2 repository with feature categories from here. 

2. Go to 'Available Features' tab and select the newly added p2 repository from the list. 

Enable 'Group features by category' option and click 'Find Features'. You can see the available feature categories from the p2 repository.

3. Select a feature category (eg: Application Server) to install and click install. It will automatically select all the required features for the product to be installed. 

4. Accept the license agreement for the features and complete the installation process.

  

5. When the installation is complete, restart the server. You can now see Application Server features available for your use on Carbon. You can see the list of installed features by navigating to Feature Manager > Installed Features tab.

In the next post I will discuss how to create feature categories to group required features for a Carbon product.

Carbon Notes

It's been a while since I wrote a blog....Thought to give some updates on what I was doing past few months.

I was learning a bunch of new things during past few months; mainly stuff related to web-services, osgi and carbon. Here I would like to give you a basic introduction about WSO2 Carbon.

 WSO2 Carbon

Carbon is the core of all the products in WSO2. It's called a modular middleware platform. Carbon empowers the WSO2 SOA middleware stack by providing a platform with pluggable modules in a compose-able architecture. The users can simply compose their own carbon servers with intended features by installing features into carbon. This powerful feature is enabled via osgi, the dynamic modular system for Java. Carbon Core contains a set of fundamental features such as user management, security, service management, logging, clustering etc, and on top of it we can install additional components to compose our Carbon Server with required functionality. Each component has a clear Front-End, Back-End separation. I will talk about carbon components and features in detail in coming posts.

Component based architecture of Carbon

Technologies Used

So OSGI,.. it's a major topic we need to look at when talking about Carbon. OSGI is actually a set of technical specifications. OSGI Alliance is the official body which pioneer and set standards in OSGI specifications. This is a good index  given by OSGI Alliance for any beginner to learn OSGI.  You can also learn about OSGI and how Carbon uses it to compose it's powerful dynamic modular architecture here.

OSGI comes in several implementations at present; Apache Felix, Knopflerfish and Equinox. WSO2 Carbon uses Equinox as the OSGI implementation as it gives a powerful provisioning platform and many more features. Equinox is the Eclipse implementation of OSGI. So in a way we can say WSO2 Carbon is the Eclipse for SOA middleware products.     

Carbon platform is built on several fundamental technologies including OSGI.  The other technologies include J2EE, web services (WS* specs), Axis2 and equinox p2 (the provisioning platform for OSGI components). Tomcat is the underlying servlet container used in Carbon.

Carbon Products

Using these fundamental technologies WSO2 has bred an amazing set of middleware products and services. The products come in 2 forms, the standalone/on-premise product and its cloud enabled service. We call the standalone products Carbon products and the cloud services Stratos services. Stratos brings all carbon products to the cloud. It can be hosted as a private or public cloud.
The products stack includes all what you need to implement a lean enterprise middleware system from data to the screen. Some of the products can be listed as below in groups based on their role in the middleware platform.

Integration

• WSO2 Enterprise Service Bus
• WSO2 Message Broker
• WSO2 Rules Server
• WSO2 Complex Event Processing Server

Service Orchestration and Composition

• WSO2 Business Process Server
• WSO2 Mashup Server

Repository and Governance

• WSO2  Governance Registry

Identity Management and Security

• WSO2  Identity Server

Application Container

• WSO2  Application Server

Presentation Layer

• WSO2  Gadget Server

Monitoring and Reporting

• WSO2  Business Activity Monitor

Adapters and Data

• WSO2  Data Services Server

Tools

• WSO2  Carbon Studio

In the coming posts I hope to give you more info on developing with Carbon.