fix 3560, Cannot recreate a workflow after a create/delete

fix 35353, Workflows are not notified when their root workflow elements are disabled

The commit will bring the following features : - An updated version of the underlying database. BDB JE 3.3 is now used. - Attribute API refactoring providing a better abstraction and offering improved performances. - A new GUI called the Control-Panel to replace the Status-Panel: the specifications for this GUI are available on OpenDS Wiki and contains a link to a mockup. See <https://www.opends.org/wiki/page/ControlPanelUISpecification>. - Some changes in the replication protocol to implement "Assured Replication Mode". The specifications are on OpenDS Wiki at <https://www.opends.org/wiki/page/AssuredMode> and section 7 described some of the replication changes required to support this. Assured Replication is not finished, but the main replication protocol changes to support it are done. As explained by Gilles on an email on the Dev mailing list (http://markmail.org/message/46rgo3meq3vriy4a), with these changes the newer versions of OpenDS may not be able to replicate with OpenDS 1.0 instances. - Support for Service Tags on the platforms where the functionality is available and enabled. Specifications are published at <https://www.opends.org/wiki/page/OpenDSServiceTagEnabled>. For more information on Service Tags see <http://wikis.sun.com/display/ServiceTag/Sun+Service+Tag+FAQ>. - The Admin Connector service. In order to provide agentry of the OpenDS server at any time, a new service has been added, dedicated to the administration, configuration and monitoring of the server. An overview of the Admin Connector service and it's use is available on the OpenDS wiki <https://www.opends.org/wiki/page/ManagingAdministrationTrafficToTheServer> - Updates to the various command line tools to support the Admin Connector service. - Some internal re-architecting of the server to put the foundation of future developments such as virtual directory services. The new NetworkGroups and WorkFlow internal services which have been specified in <https://www.opends.org/wiki/page/BasicOperationRoutingThroughNetworkGroup> are now implemented. - Many bug fixes... This set of changes may have an impact on current clients and deployments, however it should be possible to upgrade existing 1.0 instances. Some of the changes impacting the clients include : - Scripts using the command line tools including dsconfig will need to be updated (usually to remove some no longer required options).

With this refactoring: - Removed the ability for plugins to directly change the result of an operation - The plugin result API is simplified and more structured. - Pre and Post operation are guaranteed to run in pairs. - Better handling of cancel requests to minimize incorrect result codes. - Operations can indicate too late to cancel from anywhere in the work flow. - Added a non blocking abort method to cancel operations without waiting for a result. - Cancel requests from client disconnect now includes the disconnect reason. Fix for issue 2858

Updated the copyright statement to reflect that Sun owns the full copyright on the project files.

s set of files provides the workflow configuration manual mode. Until now, the workflows were automatically configured-a wokflow was created for each base DN in the backends. When new suffixes were added or when a backend was added, the associated workflows were also created (and simillarly workflows were deleted as suffixes or backends were removed). With the manual mode, each and every workflow in the server must be defined explicitely in the configuration. By default, the server is running in automatic configuration mode. To have a server running with manual configuration mode one must set the attribute in cn=config: dn: cn=config ... ds-cfg-workflow-configuration-mode: auto|manual No attribute means "auto" mode. The workflow configuration consist of 3 parts: - the configuration of workfow elements - the configuration of workfows - the configuration of network groups The Workflow Elements - A workflow element is a basic task in a workflow. The workflow elements are organized in trees and the simplest tree is made of one element. For example, the workflow element that wraps a local backend is configured as follow: dn: ds-cfg-workflow-element-id=userRoot,cn=workflow elements,cn=config objectClass: top objectClass: ds-cfg-workflow-element objectClass: ds-cfg-local-backend-workflow-element ds-cfg-workflow-element-id: userRoot ds-cfg-enabled: true ds-cfg-java-class: org.opends.server.workflowelement.localbackend.LocalBackendWorkflowElement ds-cfg-backend: ds-cfg-backend-id=userRoot,cn=Backends,cn=config From an admin standpoint, the local backend workflow element is an aggregation of a single backend (attribute ds-cfg-backend). So we cannot disable/delete a backend as long as it is used by a local backend workflow element. The Workflows - A workflow is a chain of processing and it's targeting all the entries under a given baseDN. The processing is actually identified by the root node of the task tree described above. The configuration of a workflow looks like: dn: ds-cfg-workflow-id=userRoot,cn=workflows,cn=config objectClass: top objectClass: ds-cfg-workflow ds-cfg-workflow-id: userRoot ds-cfg-enabled: true ds-cfg-workflow-element: ds-cfg-workflow-element-id=userRoot,cn=workflow elements,cn=config ds-cfg-base-dn: dc=example,dc=com From an admin standpoint, the local workflow is an aggregation of a single elements (attribute ds-cfg-workflow-element). So we cannot disable/delete a workflow element as long as it is used by a local workflow. The Network Groups - A network group defines categories for client connection. The network group contains a set of workflows and each client operation is routed to one (or more) workflow(s). By default, the server create a default network group which contains all the workflows defined in the server. The default network group looks like: dn: ds-cfg-id=defaultNetworkGroup2,cn=network groups,cn=config objectClass: top objectClass: ds-cfg-network-group ds-cfg-id: defaultNetworkGroup2 ds-cfg-enabled: true ds-cfg-workflow: ds-cfg-workflow-id=adminRoot,cn=Workflows,cn=config ds-cfg-workflow: ds-cfg-workflow-id=ads-truststore,cn=Workflows,cn=config ds-cfg-workflow: ds-cfg-workflow-id=backup,cn=Workflows,cn=config ds-cfg-workflow: ds-cfg-workflow-id=config,cn=Workflows,cn=config ds-cfg-workflow: ds-cfg-workflow-id=monitor,cn=Workflows,cn=config ds-cfg-workflow: ds-cfg-workflow-id=schema,cn=Workflows,cn=config ds-cfg-workflow: ds-cfg-workflow-id=tasks,cn=Workflows,cn=config ds-cfg-workflow: ds-cfg-workflow-id=userRoot,cn=Workflows,cn=config From an admin standpoint, the network group is an aggregation of several workflows (attribute ds-cfg-workflow). So we cannot disable/delete a workflow as long as it is used by a network group. A unit test named WorkflowConfigurationTest tests the configuration of network groups, workflows and workflow elements.

Correct apparent error in deregisterAllOnShutdown.

There are several improvements to the unit test framework in this commit. * Test methods are no longer interleaved between classes. All test methods in a class are run together, with @BeforeClass and @AfterClass methods called immediately before and after the methods are run. As part of this fix, you are now required to include sequential=true in every class level @Test annotation. If you don't do this, the build will complain. * Added a TestCaseUtils.restartServer() method that will do an in core restart of the directory server during the tests. This can be used in a @BeforeClass method to ensure that the tests start with a clean directory server, and also in an @AfterClass method to cleanup after a test that makes a lot of configuration changes. So if you introduce a new test that runs fine in isolation but fails when run with other tests, you could try calling TestCaseUtils.restartServer() in its @BeforeClass method. The TestCaseUtils.restartServer() method will reinitialize the server and reload the original test configuration, but it's not quite the same as creating a completely new process. Specifically, it cannot undo any changes that were made to static member variables of a class. I've fixed a handful of places in the server where this was a problem, but there might be more lurking. If you write a test that changes static member variables of a class, please make sure that it cleans up after itself in an @AfterMethod or @AfterClass test. * The tests now use significantly much less memory. I saw a peak of only 80MB. There were two main problems. 1) TestNG holds on to all parameters and results for the whole test, and 2) since the test classes themselves live for the duration of the tests, their member variables were holding onto a lot of garbage. The in-core restart made this problem much worse because we ended up with lots of copies of the Schema, ConfigEntryS, etc. I've introduced some hacks to fix this. Basically the code uses devious methos to go in and null out the parameters and member variables after the test has run. If you're curious about the details of how we've addressed this take a look at the comments in DirectoryServerTestCase. From now on, all test classes must inherit directly or indirectly from DirectoryServerTestCase. The build will fail if they don't. * Upgrade to TestNG b5.7. There is a fix in this release that helps our tests to run in order, and I've had to make a couple of more fixes, which they will eventually put back into the trunk. * In classes with a class-level @Test annotation, TestNG treats any public method (except @Before/After* methods) as a test method. The build now points this out and asks you to either add a specific @Test annotation to the method or change the method to be non-public. I've fixed up several places where a test wasn't annotated and others where a non-test method was being treated as a test method. * The tests now report progress as they run. Run 'build testhelp' to see details on how to control the output. I've also added some new test properties, mainly to make debugging the tests easier * test.remote.debug.port: This test property allows you to remotely attach a debugger to the unit tests. If you provide a valid port value, the unit tests will not start to run until the debugger is attached. * testng.verbosity0to5: This test property controls the debugging output of TestNG. This output is useful to check the order in which test methods are invoked or other details on what TestNG is doing. Valid values are integer values from 0 (no output) to 5 (maximum output). Since this implicitly sets -Dorg.opends.test.suppressOutput=false, other stderr/stdout output generated by the unit tests will also be displayed. * org.opends.test.copyClassesToTestPackage=true: This test property copies the classes into the test server root. This enables you to run the server tools on the test server. It can slow down the test startup so the files are no longer copied by default. Thanks to Neil for trying out these changes and to him and Bo for helping me track down some of the memory leaks.

This commit is a step toward getting OpenDS internationalized. There are still issues to be resolved before we can declare that we are internationalized but this commit covers the bulk of changes needed at this time. In general this commit converts any string values that might potentially be shown to an end user (client or administrator) to a new class called Message. This includes exception messages, error log messages, alert and email text, labels, and CLI output. Message's main interface includes methods for rendering itself as a string in the default locale using toString() or a specified locale toString(Locale). In addition to addressing localization, this would allow us to potentially support controls allowing clients to set a preferred locale for server communication or output log messages in different languages. Message extends CharSequence so it can be used in some places where strings are currently used with not code change (e.g. writing to a buffer) as well as allowing messages to be composed of one another. In order to create localized messages, instead of adding them to the *Messages.java class, you define them in the corresponding properties files. In general the property file keys are used to derive the new messages severity, category, and ordinal. There are directives that can be used in the properties file in order to avoid doing this however. For instance all the properties files used internally include the 'global.category' directive at the top of the file that instructs the generator to make all the following message belong to a single category. So for internal messages the property value key will be as follows: [SEVERITY]_[DESCRIPTION]_[ORDINAL] See the properties files themselves for more information. Once your messages are in the properties file you can use the ant target 'generatemessages' to create create MessageDescriptor objects that are referred to in the Java code. In the code you might think of these objects as replacing the int-valued message IDs. To create messages you call MessageDescriptor.get() which replaces MessageHandler.getMessage(). It might be helpful then to try to define your messages before referring to them in your code since until you run 'generatemessages' you won't have any Java-code with with to create your message objects. In order to support non-localizable text in messages (host names, DN's, etc.) Message's can be created from String values by calling Message.raw(String). In this way it is also possible to circumvent the intention of changing the APIs to use Message objects instead of Strings which is to support writing of localizable messages. This method is currently used in the code as a crutch for areas of the code that need internationalization (tagged with a 'TODO: i18n' comment). It might also be used by 3rd party developers that don't intend to localize their products. However for internal use we should not be using this method to create Messages for text that is locale sensitive. Additionally this code includes a MessageBuilder that can be used in much the same way that a StringBuilder can be used. It has a method toMessage() that can be used to generate a message object when needed. At this time this method just converts the buffer contents to a raw, unlocalizable message so this will need to addressed in the future. Some other issues: - While all of the unit tests pass after this commit, there may be issues in the code where we have no or minimal test coverage. This includes much of the setup and GUI tools packages for which I have not done any testing but will begin after this commit. - The new message related Java sources are contained in package org.opends.messages and organized in a new module src/module modeled somewhat after the admin module: src/messages/src (non-generated Java source files) src/messages/generated (generated source files) src/messages/messages (properties files for messages) If you run the code from within your IDE you will have to mark all messages/src, messages/generated and messages as source directories. - To generated the MessageDescriptor files you invoke the 'generatemessages' target. - If you run the code from within your IDE you will probably ne - This commit would change the plugin API. Here are the changes caught by DirectoryServerPluginTestCase: doShutdown(String)->doShutdown(Message) doPostDisconnect(PostDisconnectPluginResult,ClientConnection,DisconnectReason,int,String) -> doPostDisconnect(PostDisconnectPluginResult,ClientConnection,DisconnectReason,Message); - This commit aligns the entire code-base behind the new Formatter class (printf style formattings like %s) as opposed to the older MessageFormat style (using curly braces for message arguments). This allows us to provide more type safety for message arguments. So for instance you can include a %c argument specifier in your format string and the MessageDescriptor will enforce specification of a Character as message creation time. - Not all of the config framework has not been converted to use Messages. Perhaps as some point we might have a 'message' syntax for locale-sensitive textural data. I'm sure Matt will have ideas about this. - In addition to Message, MessageDescriptor and MessageBuilder, the org.opends.server.message package includes enums for Category and Severity that replace the definitions formerly found in MessageHandler. - The interface for the ErrorLogger has been simplified to allow logging of messages using a single Message parameter. The severity and category of the message will be derived from the Message itself instead of specified as arguments when a message is logged. - I will provide more documentation about this plumbing on the Wiki shortly. - I will expand on the included tests soon.

This set of changes add ID to identify network groups, workflows and workflow elements. These identifiers pave the way for the network group and workflow configuration. A workflow that is registered with a network group must have an identifier that is not already registered with the network group. To deregister a workflow one has to provide either a workflow base DN or a workflow identifier. The former method is used when a backend is deregistered with the server while the later may be used when applying workflow configuration changes. The unit-tests are updated accordingly.

Commits the refactoring of the core server to provide support for proxy/distribution/virtual functionnalities. This includes the new set of local operations, as well as the workflow and networkgroup support.