Csharp-sqlite+ADO.NET = Valentines Day Win!

As part of my work on OpenSimulator, we ran into an issue with the sqlite/native wrapper in Mono.Data.SQLiteClient so..  to get to the bottom of it..     I molded the Mono.Data.SQLiteClient over top of csharp-sqlite to enable full ADO.NET support with the managed sqlite library, csharp-sqlite.

It turns out, that while csharp-sqlite isn't quite ready for prime time, it's good enough for basic function.

The result is you can now use csharp-sqlite in ADO.NET.

The following zip file contains a modified csharp-sqlite.   The namespace for which is csSQLite and Community.Data.SQLite.   It also contains contains a set of ADO.NET classes to make use of csharp-sqlite; http://teravus.wmcv.com/googletester/csharp-sqliteado.zip

Open The Community.Data.SQLiteClient folder and load the solution.   It's already configured to load the csharp-sqlite project.   When you compile, the result of both builds will be in the bin folder in the csharp-sqlite/Community.Data.SQLiteClient folder.

Before you jump in, just a word of warning, this doesn't seem to be able to successfully load sqlite databases created with the native sqlite library yet.

For more information on csharp-sqlite, you can visit the project home page at: http://code.google.com/p/csharp-sqlite/

To see my e-mail to the C# SQLite google group containing all of the information, you can visit the message at: http://groups.google.com/group/csharp-sqlite/browse_thread/thread/e775361c98307176

 It turns out that the OpenSimulator issue was a sqlite file locking issue that's manifesting itself as an ugly error message.    So, while this was an interesting diversion and got us one step closer to figuring out what went wrong with the native library version in OpenSimulator, it doesn't immediately solve the issue with the SQLite database connector in OpenSimulator with Mono 2.4.3 and above.

With all of the hype leading up to the ipad release culminating in Valleywag's bounty offer, it was hard not to follow suit and anticipate how awesome a touch screen tablet from apple would be.   I know that I was anticipating it's release.    The marketing succeeded.   It had my attention.   After a few weeks of hearing about it, I thought to myself, "if it is an Apple Mac Computer with OSX on it, I'll definitely be in the market".

After reading the articles and news reports about what an IPad is, however, I don't think it's for me.   Sure, it's nice looking..   ultra portable, has a vibrant screen, and responds to touch..  like an iTouch.  Unfortunately..   it's /too/ much like an iTouch.  In fact, it's simply a larger, faster iTouch.     Take an iTouch, stretch it out..    and you have an iPad.   I know iTouch and iPhones are all the craze now..   but I refuse to get one for various reasons.   Here's 3 of them.

1. Apple wants to say what I can and can't run on the iPad. 

Ahh, the app store.   The App Store is one of the biggest examples of a single content clearing house.  There are a number of benefits to this approach for Apple and relatively few for consumers and developers.  Apple can say what you can run on a device that you purchase.   If you install something un-approved, you void your warranty.   For consumers, this means that there's a single place to go to get content on their iphone.  It's searchable, sufficient for general use, and virus free.    The problem isn't that the App Store exists.  The problem is that it's the /only/ way to do things.    There's no way to download your app and run it(except javascript browser apps) .   One argument that could be made..   is that it reduces piracy for apps.  Piracy is a huge problem with PC/Mac applications so that /seems/ to make sense..     until you search the term 'Iphone App Piracy' on google.    Headlines like: Apple App Store Has Lost $450 Million to Piracy, and  New Iphone App Piracy Statistics reveal try before you buy myth  .    The articles show that, in this case, a single content clearing house is failing to prevent piracy.  Sure, it could be argued that, if the iPhone wasn't jailbroken, then all app piracy would be stopped.   However..    if the iPhone was not locked to the App Store, developers would have a better ability to secure their applications from piracy.    Crackers can always 'break' their DRM schemes..   so, what would at least slow the crackers down is diversity of techniques.    Crackers can currently..   simply pull the app from their jailbroken iphone with a 'certain app cracker' and put it on an app aggregation service.   The simplicity of this makes cracked app publication easy for an average user.

One comment on a forum:
I keep getting an error when trying to crack any application.
The error starts with:
"/var/stash/Applications.SZDqB9/----------/De---ptApp.sh: line 166: 75 Done od -A n -N 4096 -t......"
Please help. Cheers

The comment demonstrates that an average user is running the cracker app and asking for support on a forum.

One app publisher's piracy story:  Tap-Fu
"As another side note, the wait time for Tap-Fu to show up on the various sites from the time of release was about 40 minutes."

This demonstrates that the piracy argument is pretty lame.  

Next, what happens if Apple decides to close down the App Store?   I purchased an IPod Nano when it came out several years ago.   Not a whole lot of work has gone into ipod games :).  They actually used to exist.   What happens when the next big thing is something different?   How bloated is Itunes going to get? 

2. iPhone apps for a tablet based personal computer are not as useful as personal computer applications.

This one is pretty much a given.   Applications have been around on the personal computer for decades.   Companies have poured millions of dollars into applications to make them useful.    We're starting all over again for the iphone and iPad.  This may be good for apple and competitors of the market leaders in the industry, but this isn't necessarily good for users of the apps.  Pretty much, for a word processor, you have iWork(coming soon), Quickoffice, DocumentsToGo and gOffice.  As far as I know, the open source alternative, OpenOffice will not be available on the iPhone

3. It's hard to get user and document files onto and off of your iPhone/iPad

Most blog posters mention that one of the only 'official' ways to get user files, like word documents onto your iphone is to put it on your Apple iDisk.   Yay, more $$$ for Apple.

In Conclusion, iPad is not a Personal Computer.   I was really, really, really hoping that it would be one.   For the person who has everything, another 'gameboy' would be great.     The iPad is a bit more like a cross between a digital assistant/business assistant and game console then a personal computer.   The marketing for the device was effective in getting me interested that it was going to be released...   but the actual device itself doesn't suit me well.    For more info on the iPad, please visit Apple's iPad website.

Thank you everyone for contacting the opensim developers and community.

Recently, there have been a few incidents of IRC bot spam.    In each of these instances, we've had to set the channel +R, which means that only users who are registered on freenode can join the channel.   All most all of these bots are not programmed to create a registered freenode account first.   This keeps them off the channel.   If you do find that the channels are set +R, you will get a message when you try to join saying that you must be registered.      Please either register your nickname with freenode or wait a few hours and come back.   Usually the channel will be opened up to everyone again in 10-12 hours.     

One more thing!    If you see a bot spamming the channel, don't click the link!   Even if the spam message seems like it's coming from someone or somewhere official.   It's a social engineering trick to get you to click the link.   The link usually contains a web page with a zero day browser exploit that will take over your computer and turn your computer into an IRC bot that does the same spamming to a different IRC channel.

I was watching the news and blogosphere lately and it seems that some people have the opinion that OpenSimulator /could/ have done something interesting but didn't move fast enough to capitalize on the first mover advantage...       Additionally, I've had people tell me personally that there hasn't been much news lately on OpenSimulator..   so it must be loosing steam.

The problem with these statements..   is they're entirely false.   There's been a TON of new press about OpenSimulator during 2008, 2009 and several important press items in 2010 already.   So, why the disconnect between perception and reality?   

One potential candidate for the disconnect..    is the OpenSimulator site itself.        Many people probably looked at the site, clicked the News link and seen that it hadn't been updated since September of 2008!   I have a google alert set up for news and blog items about OpenSimulator and I literally get 3 to 10 new blog posts or articles a day.   People are writing about OpenSimulator.   People are using OpenSimulator..     maybe it's time to update the news page.

So, I've updated the news page with a quick selection of news and press items.     A few items from 2009 that stuck out in my head as most important

The problem, of course, is we're all developers, engineers, scientists.   We're not PR people.  

This means that our effort is split like this:

• 97% - Code
• 3% - PR and Community Management

It also means that we probably let many opportunities to promote OpenSimulator pass us by.   In the past, I don't remember there being this disconnect of perception vs reality.   Then again, sdague was consistently updating the various news, public relations pages, and the community.   He used various techniques to communicate with the community.

• IRC
• Mailing Lists
• News and Information pages on OpenSimulator.org
• The Planet.us blog entry RSS feed aggregator 

I guess, if I looked at the News page alone, I might make the same conclusion as the people who are telling me that nothing new is coming out of OpenSimulator.

Another way that we can improve this perception is engaging the community.   The more they're engaged, the more they tend to be interested and promote OpenSimulator through word of mouth.

Lately, I've been getting a lot of reports of being unable to log into various websites.   

Most people, however, don't have an issue.     Just a few.  

So, I set out to locate the common element here.    

It turns out that some browsers occasionally get errors in the index of cookies that cause them not to expire from the cookie folder but, when the browser reads the cookie from the cookie store, it immediately expires them and doesn't send them back to the server.    Unfortunately, it also doesn't get rid of the cookie in the cookie store!   This 'state' of cookie doesn't allow the browser to accept a new cookie from the server.   This presents itself to users in the inability to log-in to a website or a website's inability to remember the user if they've checked 'remember me'.

You might think that simply clearing cookies would solve the issue.    In some cases, it does.     In some cases, it doesn't.    

When clearing cookies fails to resolve the problem, there's a workaround for Internet Explorer, the user needs to open an Explorer window to the user profile directory on their computer.   Commonly (on windows XP) , this is C:\Documents And Settings\YourUsername.    From there, go to the 'Cookies' folder/Directory.     Delete all of the items in there except Desktop.ini and Index.dat.  

There's also a workaround for FireFox:
Tools > Options > Advanced > Network > Offline Storage (Cache): "Clear Now"
Tools > Options > Privacy > Cookies: "Show Cookies"

It's also possible that the file cookies.sqlite that stores the cookies is corrupted.
You can try to rename or delete cookies.sqlite and cookies.txt in your Profile Folder.

See http://kb.mozillazine.org/Profile_folder_-_Firefox for more information about how to find the profile folder on your computer 

Two days ago, I got a call from the customer service of one client that they were getting an ASP .NET error page (with the custom error page) when they were trying to use the account administrative service to manage user account data through a secure ASP.NET application.     With ASP .NET on a windows box, the actual error message will appear in the Event Log in the Application section so I took a look there.

The error message was an InvalidOperationException when using a System.Data.SqlClient.SqlConnection Open method.      The issue was that it wasn't able to get a pooled connection from the .NET SQL pool.    When recycling the application pool for this application in the IIS manager, the page would load up to the point when it required SQL Connections and then would wait.   I waited for a minute or two and it continued to try to load.   Upon refreshing the page, the ASP .NET error page appeared again.     It was clear that the server was unable make a connection with the database and that the connection pool was being exausted.    This wasn't immediately picked up by the application monitoring systems because they're on the public side of the server cluster.    

What struck me as odd was that the application is designed to handle those kinds of InvalidOperationExceptions gracefully by creating a new connection.   The idea here, is that; while, it's important to make sure that all database resources are managed and properly closed and disposed, it would be impossible to crash the application by exausting the sql connection pool.

The default in this application is to use a pooled connection but if the pooled connections are unavailable and .NET throws an InvalidOperationException on SqlConnection.Open, modify the connection string to inform .NET to create a new unpooled connection.

        private SqlConnection GetSQLConnection(Database db, bool pooled)
        {
            SqlConnection dbconn = null;

            lock (openConnections)
            {
                if (openConnections.ContainsKey(db))
                {
                    dbconn = openConnections[db];
                }
                else
                {
                    string connectionstring = ConnectionStrings[(int)db];
                   
                    if (pooled)
                        connectionstring += ";Min Pool Size=5;Max Pool Size=60;Connect Timeout=2;Connection Timeout=45";
                    else
                        connectionstring += ";Pooling=false;Connect Timeout=45;Connection Timeout=45";
                    dbconn = new SqlConnection(ConnectionStrings[(int)db]);
                    openConnections.Add(db, dbconn);
                }
            }
            return dbconn;
        }

The important bit about that code is:

• If the connection is pooled, append the pool size and pool timeout settings to the connection string.
• If the connection is not to be pooled, make sure to tell the .NET library to skip the pool.

Then, the connection is opened inside a Try/Catch for InvalidOperationException.   If an error is thrown by the try/catch, it gets a non-pooled connection.

                    try
                    {
                        conn.Open();
                    }
                    catch (InvalidOperationException)
                    {
                        // Try to start a non pooled database connection.
                        conn = GetSQLConnection(db, false);
                        if (conn.State == ConnectionState.Closed)
                        {
                            conn.Open();
                        }
                    }

When everyone is doing their job, this extra protection is not needed and rarely, if ever, executes.   40 database connections simultaniously open is Slashdot Effect territory.   It doesn't happen often.

This bit of code was acting exactly as designed.  It just so happens however, that a Microsoft Update got applied a few hours earlier and the server was waiting for the next planned outage window to reboot.  After rebooting the server, the server was once again able to resume normal operations.

Shortly after, I started getting complaints that a separate application wasn't allowing users to log-in.      After debugging for a bit, it turned out that the state was no longer being kept in the cluster for users.  This issue was on a different server in the cluster.   When a user 'switched' which 'behind the scenes' server in the cluster, the server didn't know what the user did last and therefore reported to the application that the user wasn't logged in.   It just so happened that a Microsoft Update got applied several hours earlier and the server was waiting for the next planned outage window to reboot.     After restarting the second server, I stopped getting complaints.

So far, there's been no re-occurance.   All servers in the cluster have been updated and rebooted.      I can't wait to see what sort of system specific, untestable situation will happen on the next round of 'Microsoft Updates'

A friend of mine, recently mentioned that he gets a bit annoyed when people complain about bugs and demand them to be fixed so they can run their businesses but never offer to support the developers or say "thank you."    

Now, I agree with the statement.   I don't like it when people demand bugs be fixed for free so they can run their business either.  But then, as the conversation continued, it sounded more and more like my friend was confronting me on providing free code and that somehow, I was feeding into the expectation of slave labor.  It got to a point where he told me that what I'm doing is somehow more expensive then people getting paid to code because it feeds the people who expect code for free.   So, the bulk of the rest of this entry will be analysis of why I do what I do, what I think I'm getting out of it and an analysis of the conversation.

So, I guess, to understand why I code on OpenSimulator for free, what I get out of it needs to be mentioned.   I have a core desire to feel productive.  If I'm working on OpenSimulator, I'm producing something and therefore being productive.    I like the fact that I /can/ fix issues in OpenSimulator.   While, it doesn't make me feel superior in any way to anyone else, it makes me feel useful.  I like the community of smart people that surrounds this project.   It's a breath of fresh air knowing that the people that I spend the most time with are big thinkers.   I like the dedication that the community has for this project.  People actually care about how well OpenSimulator performs.  It is evident that people care by the sheer amount of bug reports that we get, the forks that people base on OpenSimulator but extend well beyond or original ideas,  the press that OpenSimulator gets, and the activity of the IRC channels and mailing lists.   This is a fabulous and active community.   If I'm working on OpenSimulator, i'm not bored.  I'm sure that there are more that I havn't thought of yet..    but lets move on. 

So my friend said that he didn't mind contributing code that he developed for himself to an open source project as long as it helped him somehow in something that he's doing.    He required reciprocity.  To which I replied that I didn't really need reciprocity.   As long as I was capable of and had a desire to give code, I would.   He replied with, 'giving is one thing, slave labor is entirely another affair'.    I thought about that for a second..   and then said..    " If you're doing it for other people, then I can see where you are coming from.   If you're doing it for yourself, what they want done..  doesn't really matter the same way. "   (This is one of the reasons..   that I've refused the job offers that I've gotten on OpenSimulator.   Once I take a job on OpenSimulator, it stops being what I want to do and becomes what someone else wants to do that I need to do to survive.   That distinction is important for me and, when I do take a job on OpenSimulator, it will be either an overpaid job, or one of last resort.)

At this point, I want to point out, that I don't think that taking money for working on OpenSimulator is bad.    Everyone who is working on OpenSimulator is doing it for their own reasons, whatever they may be..   from personal enjoyment, to providing a service for money.

So, for me,  if I get all of that from spending hour after hour working on OpenSimulator..     /and/ it helps the project along, it's a win-win for me.  I feel productive. I get to hang out with a great community of developers and enthusiasts.   I get to learn new coding techniques from some of the best systems designers out there.    And, so what if some people are using my work to build a business and make money?  Why should that fact appear in my mind at all?   I'm not doing it for them specifically.  While, I'm empathetic to businesses reporting bugs in OpenSimulator, what I actually work on is a combination of work(including bug fixes) that will improve the OpenSimulator experience for everyone and what tickles my fancy at that moment.  Sometimes, that results in big things..   like MegaRegions..   Sometimes it results in unfinished features..   And, it almost always results in a better experience using OpenSimulator.

Just because I code OpenSimulator for my personal enjoyment, and not for a business, doesn't mean that I'm not getting anything out of it.   So, I guess, for me in retrospect,  that's my reciprocity.  Because I'm getting something out of it, it doesn't seem like slave labor to me.   I can see how business oriented people might look at my situation from the outside and see what I'm doing as slave labor..  but I don't consider it to be.   

Finally, on the topic of what I'm doing being expensive because it feeds the people who expect slave labor 'coding for free'...     Logically, I have a hard time with that statement because for people who expect 'coding for free' not to be fed..   there would have to be no coding being done. No progress or no free download.   The license is BSD-ish, so people can pretty much do what they want with the code.   Does that mean that people should stop paying developers to work on OpenSimulator because it feeds the people who demand slave labor?   To me, whether a developer is coding for free or being paid to code, there is coding being done and the people who want slave labor benefit.   If developers are developing, whether free or paid, the community gets code, the platform gets better and it reinforces that the project is active, innovative, and that people care.  This draws more attention to the project through media and word of mouth which produces more developers to work on it.  It seems to me that being paid to work on OpenSimulator and working on OpenSimulator for free..  are equally expensive in that regard.   To not feed the people who want to benefit from slave labor, you have to prevent them from benefiting..   on a BSD-ish licensed project, the only way to do that is to take away the developers who work on it.  Make them a scarce resource.   If you take away developers producing code, you end up with a non-project.   Which is more expensive?     If you were to compare them to a spiral..     progress and people benefitting..  whether the progress was free or paid..  would be the upward spiral.      No progress..   would lead to the downward spiral..      Is my logic flawed?

Yay, It turns out that there were two bugs with regards to terse updates.    The first one was in SceneObjectPart.SendScheduledUpdates().    In some cases, it didn't clear the UpdateSchedule using ClearUpdateSchedule().   This means that objects that failed the duplicate test were never getting the m_updateFlag set back to zero.       In this state, the call to ScheduleTerseUpdate would fail because of the m_updateFlag < 1 test.  It would skip entirely over the code that schedules terse updates.

The second update was adding:

Velocity.ApproxEquals(Vector3.Zero, VELOCITY_TOLERANCE) ||

to the duplicate check in SendScheduledUpdates().   This ensures that Updates with the velocity <0,0,0> get sent.    Without that, the Zero velocity vector isn't guarenteed to get out.    Thirdly, I added a check in the ODEPrim.Velocity Getter.  If _ZeroFlag, then return <0,0,0> for the velocity.   If _zeroFlag is set, it has come to a rest regardless of what the tiny amount of velocity it might have.   Here, I think the averaging of the previous velocities was working against us.     "returnVelocity.X = (m_lastVelocity.X + _velocity.X)/2;"   will return a higher velocity when the object is stopped then it should for the last update with a zero velocity.   This causes the object to keep moving.    The averaging was put in place to make object motion appear more smooth.

After debugging positions and velocities between the various layers of OpenSimulator today, it appears that things are slightly better then I previously thought right now.

It looks like there's an error with ZeroFlag in one spot in UpdatePositionAndVelocity in ODEPrim that, when a prim is rezzed and made physical, causes it to drop through the terrain.

It also looks like there's an issue..   where if you drag the hand cursor over a prim the 'right' way, the SOG/SOP and ODEPrim no longer synchronize again.   If you bump the prim with your avatar where you would expect it to be, physics knows where the object is..   but the SOG/SOP doesn't.    So, that's our break.    Some kind of reference loss.    Isolating the specific state where this occurs should be an interesting task tomorrow, however, I think I've discovered enough for one day.

06:27:40 - !ZeroFlag <116.5574, 106.5194, 21.29421>

06:27:40 - !ZeroFlag <116.7664, 106.5418, 21.3075>

06:27:40 - !ZeroFlag <117.0615, 106.5566, 21.3036>

06:27:40 - !ZeroFlag <117.3505, 106.5721, 21.30312>

06:27:40 - !ZeroFlag <117.6379, 106.5873, 21.30292>

06:27:40 - !ZeroFlag <117.9243, 106.604, 21.30146>

06:27:40 - !ZeroFlag <117.9579, 106.6056, 21.30099>

06:27:41 - !ZeroFlag <117.9578, 106.6032, 21.30212>

06:27:41 - !ZeroFlag <118.1797, 106.6338, 21.31411>

06:27:41 - !ZeroFlag <118.4731, 106.6529, 21.30904>

06:27:42 - !ZeroFlag <118.7649, 106.6682, 21.30845>

06:27:42 - !ZeroFlag <119.0538, 106.6863, 21.30784>

06:27:42 - !ZeroFlag <119.3399, 106.699, 21.30552>

06:27:42 - !ZeroFlag <119.3714, 106.6974, 21.30613>

06:27:43 - !ZeroFlag <119.3718, 106.6955, 21.30722>

06:27:43 - !ZeroFlag <119.6014, 106.6931, 21.31141>

06:27:43 - !ZeroFlag <119.8856, 106.7092, 21.31145>

06:27:43 - !ZeroFlag <120.1725, 106.7264, 21.31157>

06:27:43 - !ZeroFlag <120.2315, 106.7134, 21.33288>

06:27:43 - !ZeroFlag <120.2231, 106.6987, 21.32208>

06:27:43 - !ZeroFlag <120.2056, 106.7055, 21.30926>

06:27:43 - !ZeroFlag <120.1871, 106.7195, 21.29576>

06:27:43 - !ZeroFlag <120.1709, 106.7317, 21.282>

06:27:45 - !ZeroFlag <120.1709, 106.7309, 21.28252>

06:27:45 - !ZeroFlag <120.3771, 106.7588, 21.27705>

06:27:45 - !ZeroFlag <120.6764, 106.7718, 21.27605>

06:27:45 - !ZeroFlag <120.9664, 106.7914, 21.27538>

06:27:45 - !ZeroFlag <121.2548, 106.8121, 21.27536>

06:27:45 - !ZeroFlag <121.3039, 106.814, 21.29639>

06:27:45 - !ZeroFlag <121.2975, 106.8158, 21.28955>

I had a look at the most recent trunk of OpenSimulator and I'm really surprised at how massively broken it is compared with previous versions.   Particularly the object and update pipeline is broken. 

It's broken enough that it will take a lot of work to unbreak preliminarily and then several weeks after that to test the system fully.   When it gets in this state, I get torn between frustration at how it got that way and tenacity to see the project succeed.     On the one hand, I'm extremely frustrated because it was working fine and I wasn't tracking the changes close enough to be able to figure out how it broke easily.  I wonder if someone else can resolve it easier because they were tracking the changes.   Then I remember that few people know that part of OpenSimulator like I do so, ultimately, the task will fall on me to resolve it.   

I think I'll debug it more this weekend.  Work comes first.     One of the reasons that I have not been tracking OpenSimulator changes closely is work has required significantly more of my time to complete then usual.   Normally, I have a full day's work that I get done. After that, I work on OpenSimulator. For weeks, however, it's been a full day's worth of work and then more work.   My work schedule is frustrating all by itself lately.  I'm not complaining really.  I'm arguing with myself.   About how long a task takes in estimated time, how long it takes in actual time.   Keeping projects on time, on budget and clients happy.   Maybe where I'm failing is managing expectations and i'm compensating by using time allocated for personal time.   I do have a vested interest in seeing my clients succeed in business beyond just 'finishing a project'.