Organic Programming

Intuition about SwarmESB: http://salboaie.github.com/

1. abstraction build over asynchronous messages…. yes you can repeat this but it is not very useful.
2. you put your APIs (functions, calls to web services, basic building blocks of logic,etc) in nodes and  you compose all these in a special language that get executed by jumping from node to node.  Makes sense to you?

Why is called swarming?

1. because I like organic metaphors
2. because jumping  from node to node  (using the network) of a set of values can be seen as sending messages but can be seen also as a swarming of some mysterious bees that care the honey (values ) . After a  swarm reaches a node, it can swarm again and again and you can describe and understand in a single place all the related swarming activities started in a node . What can be cooler than that?

It is not somehow similar to SOA?

Yes it is.  It is trying to solve similar concerns and actually it can work well in a SOA environment.

SwarmESB is really an ESB?

ESB have many meanings but basically, by swarming we are targeting the orchestration and loose coupling issues that all ESB are trying to address.

“Elegance is not a dispensable luxury but a factor that decides between success and failure.” Edsger Dijkstra

We had a discussion today in a meeting and we had an strong insight: we don’t try to create/sell just another RAD framework but we are trying to sell: an environment that will enforce code discipline,an architecture that respects the experience of the industry (like MDA), best practices.

Really! We even decided to sacrifice from the programmer’s immediate convenience in order to deliver what we promise: reduced  maintenance costs for the final applications. That could annoy some users if we fail to communicate what we are really selling: DISCIPLINE in PROGRAMMING with  MVC,  good practices and environment that will inhibit programmers  to do clever tricks. Only at the end we should say that we sell  a simpler programming framework! Sure, we are also targeting Excel users (at least the most sophisticated ones and only for simple things like CRUDs,simple forms) but it just happens that doing a “visual editor for MVC” led us near possible alternatives to spreadsheets space of tools. It is risky,very risky for a business to educate his customers but we are trying our best to do it in a easy and rewarding way.

Yes, a good education is required to appreciate elegance but usually the most elegant solution is the one that is more simple and efficient (almost by definition of the elegance) and we are thriving to achieve just that!

By best practices we mean things like:

• loose object coupling [1]
• real MVC architecture (and as simple as possible)
• MDA
• don’t mix many technologies and APIs in the same place (or like mixing view with controller as in many pseudo MVC frameworks)
• Don’t re-invent the wheel [2] (we have reinvented it for you )
• Keep things as simple as possible but not simpler. [2]
• Learn from others mistakes, not your own. [2]
• Begin with an end in mind [2]
• If it’s not broken, don’t fix it. [2]
• others…[]

References:

[1] http://en.wikipedia.org/wiki/Loose_coupling
[2] http://social.msdn.microsoft.com/Forums/en/Vsexpressvb/thread/78f6a9ba-05eb-45b7-88d5-8bcfcf0643cc

Even if asynchronous programming can get tricky, it enables a beneficial declarative  coding style.  I will present a few use cases for which  benefits outweighs the risks (risks that could come from bad/stupid programmers mainly):

• Form validation.  Data validation is a natural part of the model (remember the fat model idea) and it is not part of view for sure. With triggers in quark, it is possible to easily set metadata about invalidity of fields/objects and the view  will just reflect changes in metadata.
• Loose coupling in MVC and support for View States (quark terminology). The idea is that a type of data (an object) should be  differently viewed by the user depending on internal values.  In quark, asynchronous triggers will watch member values, set  a ViewState metadata and the view is switching to a proper view. Imagine doing this in a pure procedural way… This solution offers  the best possible loose coupling in Quark’s MVC  (only one way loop communication between V-> C ->M->V in this order)  .
• Implementation of the business rules layer.  A way to  enforce specific business rules on the quark models is by using triggers that  will watch values and as result will create new objects,change values.

I was always passioned on how to make programming a easier task. A direction that looked promising to me when I was a student was about programming techniques based on asynchronous behaviors. Asynchronism allows a declarative style that can simplify(minimize) the code,at the expense of  some complexity in behavior.  In many cases that is the right thing to do (such asynchronous code could be expressing business rules (what) and not how you do low level variable management => usually a very good practice ) but you have to simulate it in pure synchronous programming languages.

Many years after school, when I discovered Action Script and his events model based on observer design pattern, I instantly liked it.  I know that many programmers have serious problem dealing with asynchronism in their code and for good reasons (and others are abusing it when simple function calls should be used ). The usual solution is to create synchronous layers  inside of an asynchronous environment and hope for the best

In Quark’s model description language, we have the idea of “trigger”  that is basically a function/observer that is automatically called when other objects get changed. While this is a very natural way for describing model’s behavior, a naive implementation of the triggers can become very tricky to understand. The cause looks to be the intuition of atomicity of changes in a function.

class example
{
 var a:int; //should never take a value greater then 10
 var b:int; //should be always 10 * a, or 0

 [trigger on a changes]
 function trigger1()
 {
   if(a >= 10)
   {
    b = 100;
    a = 10;
   }
   else
    {
     b = a * 10;
    }
 }

 [trigger on b changes]
 function trigger2()
 {
   if(b != a * 10)
   {
    b = 0;
   }
 }
}
 var e:example;
 e.a = 11;

Above code (in a hypothetical action script dialect) should result in e.b having value 100.
Still, a naive implementation of triggers, could lead b having value 0 if trigger2 get called immediate after b = 100. This is just a simple example that reveals how evil the asynchronism can get.
We will take the risk in Quark Framework to implement asynchronous programming but we try to prevent things like unintuitive/unexpected behaviors or infinite loops caused by asynchronous events.
The syntax of Quark Schema language is a mix of XML and Action Script (but not really like MXML) and we will publish/document later (before the first release of quark 2 planed in October).
The way of preserving atomicity intuition about functions/triggers, it is also a good way to minimize the number of function calls (especially calls on intermediate results).

As a MVC framework, quark have a nice solution to the control of views using data from  model.

Firstly, we have a convention that every component we display in a quark based application has a model object instance that keeps the status and current configuration of that component. We call those object instances: shadows and we call the shadow classes,those quark classes that describe shadow objects.  That magic of quark is that it automatically creates UI for editing quark classes and in this way we have cheap UI to configure everything in current and future quark components.

The View is responsible for reflecting changes in model and for simplicity and loose coupling we don’t allow direct access of controller to the view . But if there is no direct connection how we control the UI?

Te answer stays in something we call metadata . A meta-data is like a member of an object,except that it is not a member that get persistence or UI. Meta-data can be attached to any quark object or to object members.  We identified a set of standard meta-data but any view component can need a different set of meta-data:

• ViewState metadata type ( not a very intuitive name,we are still looking for one) is probably the most important type of metadata.  This metadata give a hint to  quark about what shadow clone to use for UI configuration when a class instance  should be  displayed.  It simpler than it looks, basically every class instance in  model should have a ViewState metadata that is a name of a configuration used   to display/edit that object.
• ReadOnly metadata – an object with this meta-data set is obviously a read only objects
• Invisible metadata (for fields or objects)
• Label  metadata – what label get displayed when an object is displayed in a list,etc
• Icon metadata

We will show how it works  in some demos,soon.

The  model is the central part of any MVC application.  But what exactly the models are? Simplistically,one can see the model as  database schemas plus some data structures that reflect data from databases at runtime.

I have tried this “simple” approach in the past frameworks (even in Quark 1.0) and we used classes that mimics the usual OOP classes that languages as Java or Action Script provides by default.  In Quark 2.0 we have created a new XML based, OOP language that we call Quark Schema that is providing support for nice and useful models:

• declaratively describe the persistence and the life of the objects
• declaratively describe the constraints, validation and relations between model’s objects
• describe class behavior
• is fully editable using GUI tools that we are currently building

We found that we can discern 6 categories of objects (they are not orthogonal categories ):

• transient objects – objects that exists only at runtime in one instance of the application and should not be saved in a persistent storage
• globally persistent objects: objects with global identity (even for different users )
• locally persistent objects : objects that have no global identity but have a local identity inside of a global or transient object
• relational objects – objects that mimic the structure of relational tables
• document objects – graph of objects that contain other objects
• shadow objects -objects that doesn’t contain user data but inner information of the system.  Example of shadow objects : configurations and the structure of the view. Shadows get automatically generated and one  is doing customization by using visual tools organically embedded in the application but it doesn’t care much about shadow’s structure.
  
  Observations:
• Global objects that contains other local objects (or are persistence roots for graph of objects as we call it ) should be seen as complex documents structured as graphs that can be used as configurations or data structures. Quark Framework allows you to describe models in the form of normal “relational” objects or in form of  ”document” objects.
• Document objects can be seen as relational objects with fields that are root for trees containing local objects  serialized as XML strings.
• Shadow documents are usually document objects.
• Transient objects can be global or local
• Members of classes can be declared  transient and will not be present in the database schema but their values get serialized depending of their declaration as  local,global( or created as transient objects or no)
• We have discovered that local objects have an interesting property: when they are assigned to a member of a class member from a different persistence root (different persistence global root object) they should be automatically cloned.   Quark Schema is automatically doing just that and we discovered that is nice and useful in real cases.

Apparently all these rules and observations are a bit complex, fortunately the business user will use a simple UI to create such classes (models) and should not worry much about persistence aspects.  It will just communicate his intentions and the system will do “the magic”. Local objects are usually shadow objects and will be a concern for component creators and rarely for the business users (customers or consultants).

This article is long enough so I will not present  any syntax but I want to mention only that for Quark 2.0 we have got all these aspects working and well tested with unit testing.  Drop me an email for any questions.

At the end, take a look at this picture that is saying: “the model should be fat but it should be constructed with skinny,elegant and nicely shaped classes”:

Creating  above picture piece by piece is  a laborious process. For me, it took 1 minute to create using Google search for images and a screenshot. Similarly, our goal is to create a tool that  will allow  creation of a working business application in minutes and we believe that we are not far from it! A new direction I’m contemplating now is to have  Quark Framework  as a tool for creation of mashup applications [1] with data from multiple business applications. It is a logical use of Quark and fortunately it results from our design,without much supplementary programming work.

References:

[1] http://en.wikipedia.org/wiki/Mashup_(web_application_hybrid)

In the quest for eliminating all unnecessary barriers in having ultra rapid application development we realized that the save button is sometimes quite Evil. For web applications the Save button is somehow inefficient and could lead to confusion and loose of data because of browser crushes,network errors,user mistakes. The save button is also increasing the complexity of the system. Many hardcore programmers will say that it is not much but our framework wants to allow even non-programmers to create working applications with an effort equivalent with that of creating a spreadsheet.  We think that the approach of periodic auto-saving is the way to go in Quark for  almost all cases.

If the framework provides cheap (or free)  undo,redo operations for all operations that users can do,naturally that we can have nice applications without the “save button”.

What “save button” provides and “workarounds” could be required:

• the biggest issue: workflow and integration with process management capabilities. If a user is just playing with data in UI,could trigger workflow changes unintentionally. Business application that have workflows are normally used by serios people that will not just click randomly and press keys,but still, this could be a problem.
• protection for “cat” operations – if a user is just playing with his keyboard or a cat is walking on it,bad data could find a way to the database
• implementation of atomic changes:provide consistent persistent storage.  In some cases,changes in a part of an object without changing other parts will lead to an invalid states of the object. This problem will be avoided with a validation, Quark Framework will not auto-save invalid objects.
• implementing transactional operation for graph of objects will require additional,a bit more complex validation

I think that cases that will require a higher level of validation,could have a standard “validate change” button or “apply” button in the UI and the “auto-save” will not save those objects that are not manually validated.

Quark framework 2.0  provides for free (almost) undo,redo and auto-save. These features were not present in Quark 1.0.   So,what do you think?  Do you really need to put users pressing the “save button” after each important change ?

Nothing is wrong! MVC is the most famous design pattern after singleton but people have very diverse opinions and there are quite a lot of misunderstandings about this powerful design pattern.

MVC design pattern is  illustrated by this figure (from wikipedia [1] ):

Even if those dashed and solid lines are looking a bit dubious, in many existing MVC implementations  the reality is even more complex than this, look at the next  image from [2]:

Simple? No? Necessarly complex? There is no final answer yet, but in Quark 2.0 we have a simpler and elegant communication  paradigm between Models,View and Controllers.

In Quark, the controller is controlling the model and the view is strictly rendering the Model while triggering the controller when the user is asking for something. Also, as you can see in the picture, we are fans of fat models [3] ,  skinny controllers and dummy,auto-generated views like in Naked Objects , MDA,DDD [4] .

We know that some programmers do not have the capacity to understand that a “Model” is not just a wrapper of a database table but our approach that we call “Quark Schema”   should make things easier for anybody: advanced programmer or clueless user. Even non professional programmers could do the right things without worrying much about philosophical and technical details of MVC.  Our goal is to make a tool and a programming framework that will look and be simple like doing things in  Excel but resulting in nicely growed Organic Programs.

References:

[1]  MVC in Wikipedia: http://en.wikipedia.org/wiki/Model–view–controller

[2] Pure MVC page on Wikipedia http://en.wikipedia.org/wiki/PureMVC

[3] Links about Fat and skinny Models and controllers, anemic domain model,etc

http://www.martinfowler.com/bliki/AnemicDomainModel.html

http://nuts-and-bolts-of-cakephp.com/2009/01/06/another-way-to-think-about-mvc/

[4] Naked Objects, About MDA and DDD : http://en.wikipedia.org/wiki/Naked_objects

And just for fun (and profit for some): About Fat Models

Many people believe  that programming is like building of houses and programmers are architects. Others, believe that programming should be more like farming and growing of beings.
A perfect explanation for this thinking is expressed in [1]: “Systems grow and evolve. Often programs die, or succumb to disease. Yet, at the same time, I can feel the pulse of life as information permeates through the network. I no longer seek to claim to be a builder, but one who plants seeds and cultivates the soil. The biological metaphor may sound humble when contrasted to the titles of architect or engineer, yet for a system to bear fruit, it will take cultivation.”
Different approaches (but no one being mainstream) tried to integrate programming with NLP or with multi-agent system, but our current view about Organic Programming can be resumed for a set of simple principles:

1. OP should lead to high quality programs,with reduced costs of developing and adapting programs to the needs of the final user.
2. following of the best programming practices while hiding complexity for the developer by using advanced frameworks and good UI
3. tight integration of the tools used for developing the program with the program itself. These tools can be used at runtime to customise or even to create new features.
4. use the right language technology for the right thing. A wider adoption of technologies like rule engines,workflows and well designed domain languages could improve the quality of the programs.

Everybody wants to achieve the good effects of first principle of OP, but few succeeds because they are building programs instead of growing programs. Or at least this is what we believe.
Current mainstream frameworks like Rails or Java framework are not respecting the first principle because they are too focused on code and are using a technology for building frameworks (OOP) for creating programs.
The 3th principles is respected only in some programs, a notable case are spreadsheets and in the next articles we will insist on how our technology called Quark Framework goes towards respecting this fundamental principle of OP.  A second version of the Quark framework will be released soon. The first version will be published as open source software next week.  At least in the near future we will focus only on organic programming for enterprise software [3] and not on organic computing [2].

References:
[1]   http://descmath.com/data/organic.html

[2]   http://en.wikipedia.org/wiki/Organic_computing

[3]  http://en.wikipedia.org/wiki/Enterprise_software