Johlero – Cardoen Lieven

Last week I got a call from a big customer saying that the 100 of the 200 candidates had just received an error log window in our Flex Application. I knew that this could happen someday, but what had caused this error log…?

After a lot of searching in the logs of the two servers at the customer, I found out that at a certain time on one of the two servers all sessions and application pool were ended. Why? Well, after again a lot of searching I found out that IIS recycles an application pool every 29 hours… Man, why is this default enabled???

Apparently a lot of these recycle events are nog logged by default. Only the 29 hour recycle event apparently is logged by default. As for now I haven’t found this log yet, so I will have to enable all those logs to find the actual reason why the application pool recycled…

That’s done by this command on IIS6:

cscript %SYSTEMDRIVE%\inetpub\adminscripts\adsutil.vbs Set w3svc/AppPools/LogEventOnRecycle 255

On IIS7 you can enable them in the Advanced Settings of the application pool.

The problem with our Flex Application was that it uses WebORB Authentication. This authentication is done by the first remote call after which the server side session remembers the credentials. After the recycle the sessions were gone and 100 candidates received a WebORB Security error… We have already solved this problem by re-authenticating when this happens but I’m also really hoping to find out why the application pool recycled in the first place… Any suggestions are welcome… (memory?, too mucht connections?, database problem?, …).

Ciao, Lieven Cardoen aka Johlero

I’ve seen that working with stored procedures in MSSQL2005 sometimes raises a lot of questions in me in how to maintain all those stored procedures. I was wondering if there was a way of checking Data Object Dependencies, such as stored procedures that depend on a table.

I’ve gotten some good answers on StackOverflow.

Apparently there is a database engine stored procedure and two dynamic management functions you can use to do this like:

1. sp_depends: Displays information about database object dependencies, such as the views and procedures that depend on a table or view, and the tables and views that are depended on by the view or procedure. References to objects outside the current database are not reported. –> will be removed in future versions of mssqlserver
2. sys.dm_sql_referencing_entities: Returns one row for each entity in the current database that references another user-defined entity by name. A dependency between two entities is created when one entity, called the referenced entity, appears by name in a persisted SQL expression of another entity, called the referencing entity. For example, if a user-defined type (UDT) is specified as the referenced entity, this function returns each user-defined entity that reference that type by name in its definition. The function does not return entities in other databases that may reference the specified entity. This function must be executed in the context of the master database to return a server-level DDL trigger as a referencing entity.
3. sys.dm_sql_referenced_entities: Returns one row for each user-defined entity referenced by name in the definition of the specified referencing entity. A dependency between two entities is created when one user-defined entity, called the referenced entity, appears by name in a persisted SQL expression of another user-defined entity, called the referencing entity. For example, if a stored procedure is the specified referencing entity, this function returns all user-defined entities that are referenced in the stored procedure such as tables, views, user-defined types (UDTs), or other stored procedures.

Also very usefull is SysComments :

SELECT DISTINCT Object_Name(ID)
FROM SysComments
WHERE text LIKE '%Table%'
AND text LIKE '%Column%'

Apparently this SysComments table is a sql server 200 system table included for backward compatibility. Instead you should be using sys.sql_modules, which is a view object.

Writing tests however seems to me the most reliable way of ensuring that nothing was broken when changing an object in the database.

Lieven Cardoen aka Johlero

In the next example I extended the previous example with declaring an array and invoking a method on an instance of an object.

Download Project

Example

In the ModelLocator I added two variables which will be manipulated by the prana container.

Users is an Array, typedUsers is a TypedCollection of User objects (see the Constructor).

There's also a function to add an instance of a User to the typedUsers collection.

In the PranaExample.mxml (application starting point) I added two List controls with dataproviders users and typedUsers.

The two variables (users and typedUsers) are being populated by the Prana Framework like this:

As you can see the users array is just populated by an array tag with multiple value tags.
To populate the typedUsers TypedCollection, we need to call the addUser function on the JohleroModelLocater. The arg tag defines the argument that will be passed to the addUser function.

In the lib there's a prana-main.swc. This swc is not the same as the swc of the latest production release from prana. There was a bug and it's fixed in this swc. You can allways checkout the prana repository from https://prana.svn.sourceforge.net/svnroot/prana/prana-main/trunk and add a reference to your project. Like that you'll allways have the latest code available (with its risks).

That's it. See the example and project to see it work!

Ciao! Lieven Cardoen aka Johlero

In this post I somewhat extended the previous example to show you how to work with a properties.txt file and interface-based programming.

Download Flex Project

Example (right click swf to see source)

The application-context-johlero.xml looks like this:

There are two new things here. First is the reference to a property file application-context-johlero.properties.txt (containing key value pairs) and the assignment of an instance of a class to a property. Notice that the property calculationFactory in JohleroModelLocator can be set to an instance of DivisionFactory or MultiplyFactory in two ways.

In my experience using a property file is usefull to seperate program specific configuration from environment specific configuration. In my example a have a gatewayUrl which is defined in the property file because this gatewayUrl will probably change when deploying on another server.

To learn some things about interface based programming I suggest you read a book about Design Patterns (Head First, Gang of Four) and OOP (Head First) but you can see that Prana is a great way of keeping your application kind of stupid. The application doesn't know if a division of multiplying will be executed. Only at runtime an instance of DivisionFactory or MultiplyFactory is injected in the application. Because the application doesn't have a reference to DivisionFactory or MultiplyFactory (only a reference to ICalculationFactory exists) you need to add these Classes to your application. If you don't do this, Prana will not be able to create an instance of these classes at runtime.

That's why you'll see in the PranaExample2.mxml:

Well, I think this is about all the explanation you'll need. The example speaks for itself.

Ciao!

If you want to walk your dog, you don't command your dog's legs to walk. Instead you command the dog and trust that it takes care of its legs itself.

Only talk to your immediate friends.

In modern program languages that use dot as field identifier, use only one dot.

Law of Demeter for functions requires that a method M of an object O may only invoke the methods of the following kinds of objects:

1. O itself
2. M's parameters
3. any objects created/instantiated within M
4. O's direct component objects

That summarizes the whole Law Of Demeter, but maybe some more information will enlighten you more...

The purpose of the Law Of Demeter is to minimize coupling between modules. It reduces the size of the response set in the calling class and this tends to reduce errors (a response set is the number of functions directly invoked by methods of the class).

An example in a real programming language makes it all clear for me:

Advantages:

• It'll make your code more maintainable and adaptable. Since coupling is minimized, object containers can be changed without reworking their callers.

Disadvantages:

• It sometimes requiers to write a great deal of small wrapper methods to propagate method calls to the components or subcontractors. As a result, the class's interface becomes rather bulky and performance can decrease. Reversing the Law of Demeter and tightly coupling modules may give you performance gain.

The Behavioral Problem

There are the structural aspects (how tables relate to objects) but there's also a behavioral problem which is harder to solve. How will you get the objects to load and save themselves in the database? If you load a lot of objects into memory, you will have to make sure that the database you are working with stays consistent and synchronized with your memory model. This is not an easy job because for instance sometimes you need to create a row first before you can modify another (because you need the id of the created row). Concurrency is also a big issue here!

The Unit of Work pattern can solve both of these problems. It keeps track of all objects read from the database together with all objects that are modified. It also knows how to make updates to database. The programmer then tells the Unit of Work to commit. It is the controller of the database mapping.

An Identity Map is needed to keep a record of every row you read. If you read some data, you can check this Identity Map to know if you don't allready have it.

When using a Domain Model, usually you will load linked objects together. Without Lazy Load however, this would mean that enormous object graphs would be loaded out of the database. Lazy Load relies on having a placeholder for a reference to an object. If you try to follow the placeholder, the real object will get pulled from the database. A good example here is when in Flex we have to load big trees. If you have a tree with 100.000 folders, loading them all together would take a while. If we load a folder, we give back its children but not the children of the children. So a depth 1 is returned.

In a previous project created with Flex/Fluorine Gateway/.NET C#/SQL2005 I just save the whole model. Having to program a Unit of Work keeping track of changes would have taken me too long. I realize that if the project would be used by thousand concurrent users, changes would have to be made. For now, it isn't a problem, but I can imagine that working out a Unit of Work would not be an easy job. In the current project a similar problem arises when editing exams. An exam consists of some components that contain items (exercises). A Unit of Work would have to keep track of deleting items, moving items, changing items, changing components, ... Now we just save the whole Exam object (this means mapping it to DTO's in Flex, sending it to the service layer, converting it to Table Data Gateway, deleting the Exam and saving the Table Data Gateway). A fast solution, but it actually takes 30 seconds to do it... No need to say we'll have to change this in the future.

A big part in the role of the data source layer is talking to a database, most of the times a relational database. I have done some reading about how to implement this data source layer because it can become pretty messy in big projects. It's also hard to refactor once a choice of pattern is made so it's important to make the right choice from the beginning. For my current project I'll look into some patterns, ideas, new technologies and write down a resume here.

Architectural patterns

They define how the domain logic talks to the database. In my experience there should be data access layer (.NET, Java, ...) developers and database developers. Most of the times a programmer also does the database side as a result of which they don't become database experts. When you have data access layer developers and database developers, you will want to seperate SQL from the code, so that a database developer can have a look at the SQL. In this way, a Database administrator can understand how best to tune it and how to arrange indexes.

We had a performance problem with lazy loading a tree. It went so slow that usability was down to nothing. The trees are saved as left right trees in the database (preorder tree traversal algorithm). Normally the advantage of these trees are that loading the whole tree goes very fast. However, for the lazy loading, we only need the children of the node we are loading, and this took a very long time. After searching a bit on indexes in SQL, I managed to improve the performance to a level that usability is really very good now. Lazy loading the trees is now a pleasure.

A good way to do this is to have classes based on the tables (one class for each table). These classes form a gateway to the table. Developers who specialize in the database have a clear place to go.

1. Row Data Gateway: An instance for each row that's returned by a query.
2. Table Data Gateway: Single object for each table in the database. This provides methods to query the database that return a Record Set.

This will only work for applications where the domain model is allmost equal to the database structure, having one class for each table.

A Table Data Gateway can also be used to organise Stored Procedures. You could treat the Stored Procedures as Tables and have a Table Data Gateway to wrap the calls to the stored procedures.

Active Record is having a Row Data Gateway and then adding domain logic to the class. Again, if your Domain Model corresponds closely to the database structure, this option is usefull. However, if your Domain Model becomes more and more complicated, the Active Record approach will break down sooner or later. As you start to implement inheritance, OO patterns, stategies, ... in your Domain Model, things will get out of control. Another thing is that O/R mapping tools generate classes that you will need to extend to add Domain Logic. Eventually sending these objects to Flex will get messy as you will have to exclude certain properties that do not need to be send. A solution is to isolate your Domain Model from the database by making an indirection layer responsible for the mapping between domain objects and db tables. This is called a Data Mapper and it completly isolates the two layers.

There are tools that create these patterns(O/R mapping tools). I have used MyGeneration together with Entity Spaces and SubSonic (both Table & Row Data Gateways I think). Both were very helpfull but not ideal. They are third party tools and when you change the database you need to recreate all the classes. The biggest problem is that if the database changes, compiling the project will be no problem. At runtime however problems will start and errors will follow. All the projects that I've done are RIA using Flex and Remote Calls to the Server. This means having DTO's. The classes created by Entity Spaces and SubSonic are too large and complicated too send to Flex. So I had to create Mappers from Entity Spaces/SubSonic to the DTO's, which made it complicated. I'm kind of hoping that LINQ solves this. In the current project I'm working on the complexity is even bigger. I'm using Stored Procedures to retrieve data and SubSonic to save data and sometimes retrieve data. This means I have to convert DataSets and SubSonic Objects to DTO's before sending them to Flex. Now we are also porting the model from Flex to .NET, which means that converters also need to be able to convert datasets and subsonic objects to the Model. It gets kind of complicated and I'm searching for a good solution here.

Another thing are OO databases. I haven't experimented with them yet, but the whole problem of mapping the database to the model would disappear with a OO database (I think). The whole mapping thing is only relevant because of the fundamental difference between objects and relations. Objects hold a reference to another object while tables hold a primary id to another record. I guess not many projects use OO databases, maybe because relational databases have been around for a long time and are proven technology.

Concurrency is a big issue if you have business applications created in Flex. If two clients are editing the same data then the last client that saves the data will win.

Optimistic Offline Lock has the best liveness but you will only find out that a business transaction fails when you commit (save). So if you have been working for half an hour on some data the system will not commit the transaction and users won't like this. Possibilites here are:

• Letting the user save his data under another name
• Letting the user decide whether his data should overwrite the existing data
• Letting the user compare the two versions and resolve the problems

Optimistic Offline Lock is easy to program. If you commit data, you need to check if the data in the database has changed since you loaded the data. Providing a solution however is more difficult.

That's why we choose for a combination of Pessimistic Offline Lock and Implicit Lock. It's harder to program but you don't need any of the solutions above and it's much more userfriendly. We combined it however with Coarse-Grained Lock, managing the concurrency of a group of objects together.

If a user opens an object to edit its data, then a message is send to all other clients saying that object of type A with id x is locked. A visual notification is shown in the other clients. When the user navigates away from editing the object, a message is send saying that the object of type A with id x is unlocked. We also send the id of the user and client locking the object. A drawback is that when a client opens an object for editing and leaves his desk for instance, then the object stays locked. However, after a half hour of inactivity, the user is logged out automatically.

The server side also holds a model of objects that are locked. If a client connects, it gets all the currently locked objects from the server. If a client disconnects, then the server cleans up the locked objects from that client and notifies the other clients. When two clients request a lock for the same object, then the client that triggers first the server side function to lock an object wins.

Concurrency was one of the biggest challenges so far and I'm not quite sure if our method is the most appropriate. It's hard to test, hard to debug and your application starts to rely for a part on being able to send and receive messages (that's why we made it configurable). We have implemented it using WebORB and it seems pretty stable.

Cheers, Lieven Cardoen

In my Company we develop educational software. Authors can create content in a backoffice and present them to students in different kinds of contexts.

The first and second version use the same backoffice created mainly in ASP. In spite of the fact that usability isn't that good, it is still being used by our customers. The frontend for the first version was written in flash using Actionscript 1, the second version used Actionscript 2. In AS1 we had an swf for each type of exercise, in AS2 we had one big swf.

For the third version we decided to use Flex for backend and frontend. There have been hours of discussion about this decision in our company and even customers tend to argue about using Flex. This post is about this discussion, about why we made that choice and the consequences of this choice.

First let me state some of the requirements.

• Customers ask for interactive exercises having dragndrop, video, audio, images, connecting things, ...
• An author need an easy tool to create these exercises and preview them.
• Scenarios can be created that generate all sorts of tests and these tests can be previewed.
• An intelligent player can present these tests to frontend users.
• The application may need to run offline in the future (frontend and backend)
• CD-roms must be made with the content

For the frontend we didn't have a lot of options two years ago. Silverlight wasn't even born yet (or maybe just), haven't read a lot of good things about Java FX although I haven't tested it myself, Javascript and AJAX do not seem suited for this kind of applicatoin... So Flash seemed the best choice and as of today this is still the best choice. Having an application running in the browser is a big advantage for these kind of applications. The application is loaded once into the browser which increases usability a lot, the different parts of the application can talk easily with each other, communication with the server is easy with remoting technology like WebORB or FluorineFX and interactivity like Drag and Drop, audio, video, ... is a pleasure to develop, you don't need to worry about different browsers, Linux, Mac and Windows supported and creating CD-roms is quite easy with tools like ZINC of Adobe Air. The downside is that all logic lives on the client side. First of all this means that reverse engineering the swf is not hard at all for someone who knows how to do it, even if the swf is encrypted, security needs to get more attention, programming an application is harder than programming pages so you need experienced developers, the Flash Player Plugin needs to be installed (in big companies this is often a big problem because users of a computer do not have rights to install a new plugin. The admin of the company needs to do this and often these admins are not eager to do this...) and few graphical designers have experience with Flex.

Creating the backend with .NET technology would probably be twice as fast. You can easily seperate work without affecting other parts of the project (with an application changing something often affects other parts of the application), there are a lot of .NET developers an few Flex developers, doing it in Flex still needs a server layer (WebORB with Services and some logic),... So why did we choose Flex to create our backend? Well, there are a few arguments that are important enough for this decision:

1. A lot of logic from the player can be reused in the backend, like serializers, deserialers, our framework, ...
2. Creating interactive exercises with Flash is a hundred times easer than doing it with Javascript.
3. Showing a preview of the exercise can be done without having to show a popup loading an swf to show the exercise. (as we had to do in our first version).
4. Bringing it offline to Mac, Linux and Windows is a piece of cake with Adobe Air. If you would develop the backend in .NET, bringing it offline to Windows would be possible, but bringing it to Mac and Linux wouldn't. (Even with Silverlight you would be doing the same as in Flex+ no Linux with Silverlight and not as mature yet as Flex).

It's true that some parts like reports, access and rights management, creating scenarios, ... would be a lot easier with .NET but then you would be mixing technolgy and that wouldn't look all that nice now would it?

It's not easy to defend such choices. For small projects backends are usually made with .NET and frontends with Flash if interactivity is needed. As a project like ours gets very big, you need to be thinking about maintenance, design patterns, different customers, budget, resources, ... and then Flex doesn't allways appear ideal... Ok, there's no problem with different browsers, you have an application feeling in the browser, it's  supported on Mac, Linux, Windows, but is it worth it? I find it allready difficult to explain the decision on my blog here to developers, imagine explaining it to a customer... Switching to product development in our case would be the best move according to me. The bigger a project comes, the more time it takes to add functionality. And then there's this worry of me about how big an swf can get before having problems? Our swf is currently about 1meg big. The backend also loads a lot of stuff from the server into the memory (clientside)... Are there limitations?

I have played a little bit with silverlight in the last days and it looks promising. If it would ever get as mature as Flex, then I would certainly switch. A tool like Visual Studio is a treat to work with and having all the logic in one technology would be great. Now we have to port a lot of logic from Flex to .NET to create reports (for instance in Reporting Services), to have exports to paper from tests, ...

I do not intend to start a discussion about which technology is better. For me there are customers, budget and demands and trying to meet those demands. All technologies will have their advantages and disadvantages and it's up to the team to choose the right technology to do the job.

To the future!

Ciao!

A big problem in my current project is Concurrency. A client loads a Tree in his Flex application and starts making changes to this tree. If another client also makes changes in the Tree, then we have a Concurrency problem. We thought of two approaches of dealing with this problem. One of them is using Messaging with MSMQ. We did however had some issues developing this and are still trying to make it work.

We mainly worked with two examples provided by WebORB to figure things out. One of them is the Flex Messaging and MSMQ  example from the WebORBConsole --> Examples --> Real-time Messaging. The other one is the MSMQ to Flex Data Push example from the site of the midnight coders. In the first example Flex is the Producer and Consumer of Messages. In the second example the Server produces Messages and Flex consumes them.

What do you need to run these examples:

1. MSMQ: If you look into Computer Management under Services and Applications you should see a folder Message Queuing. If not I suggest you google to find out how to install this onto your computer. For Vista you'll need to go to the Control Panel --> Programs and Features --> Turn windows features on or off. There you should find a checkbox for installing MSMQ.
2. WebOrb for .NET  --> In your site you only need the weborb.dll, so this installation isn't really needed. But it comes with a lot of documentation and examples, so if you haven't installed it yet, go and do so now!
3. The Messaging Server (WebOrb) should be online. If you're using the WebOrb30 installation to run your examples, then the Messaging Server is started by weborbee.exe. In your project website however, you shouldn't use this weborbee.exe. You only need to copy the global.asax. If you look into this global.asax, you'll see the weborbMessagingServer is started there. I'll show in the examples how to make sure the MessaginServer is online.

 When you open the Flex Consumer Resource from the second example you'll see in the properties of the Project --> Flex Compiler that a services-config.xml file is compiles with the application. In this file Channels and Destinations are declared. We left this out so we could choose at runtime which Channel and Destination to use.

Ok, let's start with the Flex Client Side. Here's the example somehow adjusted to our means. I'll go through it piece by piece. For now don't bother too much about the server side. I'll go deeper into that later.

We register an Alias for the Car class so it'll be mapped correctly to the C# Messaging.Car class.
Next we call the function pingWeborb() which will ensure us that the MessagingServer is started before we start to send Messages. By triggering the weborb gateway (weborb.aspx) in your site, you'll be starting the MessagingServer (in Global.asax). If you are not using the weborb30 site, make sure you haven't started the MessagingServer with the Weborbconsole in weborb30 site. If you want to be sure of this, go to Task Manager --> Processes and kill the w3wp.exe process.

The GenericDestination can be found in the remoting-config.xml in the WEB-INF\flex folder along the server side. More on this later. The source is the Class instance that will be created and the endpoint the url of the gateway. Make sure you use the RemoteObject class from the mx.rpc.remoting.mxml package, as this RemoteObject has an endpoint property. There's also a RemoteObject class in the mx.rpc.remoting package which has no endpoint property. Adding two EventListeners and executing the ping() function in the Weborb.Management.ManagementService class instance.

 If the Ping was successfull, then onPingResult should be triggered, in which we'll connect as a Consumer and as a Producer. If the Ping was unsuccessfull, onPingFault will be triggerd and an Alert Box will be shown with the error. If something goes wrong here, I would start a http sniffer like Charles to see if the Gateway is called. If it is, start by creating some breakpoints in Flex and certainly in the global.asax on the serverside. Make sure to keep on reading even if things don't work as I will talk about some other problems I've encountered. Maybe that one of these problems will be the issue that's causing your problem.

Ok, first we connect as a Consumer, because the connectAsProducer function will also be sending a message. First we create a ChannelSet containing a WeborbMessagingChannel. I haven't gone deep into this Channel thing, but one thing that was strange to me was that no site is given in the rtmp uri. Apparently the MessagingServer is listening to a port. The /MessagingService points to a folder in the Applications folder under the root of your site. In this MessagingService folder it's possible to place an app.config where you can change some configurations. The channel triggers a MessageEvent.MESSAGE event when a message is Consumed.

The destination can be found in the messaging-config.xml under WEB-INF\flex. In there you'll find an entry path with the name of the queue. This path together with the subqueue (in my case johleroqueue) will be the name of your queue. If the path is .\private$\WEBORB-[destinationName] and the destination name "ExampleDestination" then a queue weborb-exampledestination-johleroqueue will be created. The first time the queue is used it will automatically be created by weborb. However, if weborb creates the queue, you will not be able to access it with the Computer Management tool. Apparently access is denied. A workaround for this is to manually create this queue before running the example. If you manually create it, you'll have access rights. However, you'll need to give the Network Service rights to the queue, otherwise the messages will not arrive (you can check this!). Weborb will then use the queue you created if you use the same name off course. The ACKNOWLEDGE event listener will let the consumer know if subscription was successfull. The other listeners are clear enough I guess.

Connecting as Producer is quite similar to the Consumer with the exception of the send function. Meanless to say that the Producer and Consumer need to connect to the same queue. Next the Producers sends a Car instance.

If the Consumer has successfully suscribed, then a onMessageAck should be triggered. If the Producer has successfully connected, then onMessageAck should be triggered a second time. The onMessageAck from the Producer does not mean that the message has successfully been send, it only means that the Producer has been able to connect to the MessagingService. There doesn't seem to be a clear way of knowing if the Message has successfully been send. However, if the Consumer gets the message, then other Consumers will have gotten the message too. After this onMessage should be triggered and a Car (if the server side is configured right) should be delivered.

Adios, Johlero!

Thx to the midnight coders!