Posts from September 2005.

1 Year Old Today!

The current incarnation of this website is one year old today, happy birthday ianmjones.net!

Although ianmjones.net has been around for a few years with a blog like appearance, it wasn’t until September 30th 2004 that I started to properly “blog”.

Since last September I’ve always intended this site to stay on track, to talk about how things were going with my quest to become an independent software vendor (ISV). I’m not sure I’ve managed that very well, I seem to have strayed quite a bit, really haven’t written a lot about what I’ve learnt along the way and haven’t written as often I hoped I would.

What have I achieved in my first year?

  • I’ve read and listened to tonnes of marketing and business material in many different forms of media and from many sources (which I’ve failed to talk about here when I really should have, slap my wrist).
  • I’ve researched and picked suitable tools to develop, source control, package and sell my software.
  • I have designed and developed a software application in a programming language and with other tools that I hadn’t used before.
  • I’ve learnt a lot about the FogBugz database schema and data usage and installed it way too many times on far too many platforms!
  • I’ve registered something like 35 separate domains and set up the websites.
  • I’ve set up an online store to sell my software (visited a few times by people, but never used the way it should be).
  • I’ve set up a few channels for public communication, such as the Newsletter and Yahoo! group (there’s a proper forum waiting in the wings).
  • I’ve released three public betas of my software, with the second and third showing a lot of improvement over the first!
  • I’ve even released some open source code as a by-product of my software development.
  • I’ve met (in a virtual way) a lot of really nice people from all over the world.
  • I’ve had very little sleep compared to the year before, and seen a lot less of my family too (this isn’t a good one).
  • And tonnes of other bits and bobs that escape me just now.

Some of this stuff I’m very proud of, some not so. But having a real direction in my life and being able to continuously step closer towards my goal of becoming an ISV while having lots of support from my family and friends, and new friends too, has made this a fabulous year.

Really looking forward to the next year, it should be good.

Quick Update / FogBugz & LDAP / Selling Beta Software

Working hard on getting the final few things sorted for CaseDetective 1.0.

Just got to learn about and write some LDAP code for Active Directory and OpenLDAP authentication now that FB4SP1 has it, add automatic version update checks, and update all my Help Docs and I’ll be all set ;-)

I’ve already knocked off a few other little bits and bobs, but still have quite a bit to go by the looks of things!

The final beta has proven to be a real non-event, which is great! Eh? Well, no one has raised any bugs at all on this version, there’s only been a couple of questions about how things may go in the future and some really nice comments. So I’m very happy.

I’m obviously going to be taking a little bit of a risk by adding the LDAP stuff to v1.0 without having a public beta, but before going live I’ll seed it out to a few current testers that I know use domains etc and might be using this new FogBugz feature.

The thing is, I doubt a huge number of FogBugz users will start using this LDAP authentication straight away, it does take a little bit of planning to make sure existing users are correctly configured for when you switch. I found this out the hard way, locking myself out of FogBugz in the process. It was easy to get back in by updating a couple of records directly in the database though. Second attempt worked much better once I’d sorted out how the user names should be set. I guess a network professional would have no problems, but I’m not one of those by any stretch of the imagination.

One disappointing but expected thing about the betas is that I’ve made no sales. Well, I say no sales, I know of plenty beta users who have said they’ll be buying it, but none have yet.

I kind of expected this, it won’t be until v1.0 is out the door before I actually start to get sales. As I see it there are two primary reasons:

Firstly, people don’t like to pay for something which is in Beta as that label means “not quite finished” and who knows when it actually will be finished? Paying up is a risk.

Secondly, I gave the beta users way too much trial time to play with. 60 usage days of trial time is a very long time, I should have kept it at the default of 30 days. But it’s too late now, the deal was done and I promised 60 days, and a further 30 once v1.0 was out there, so that’s what I’ll do.

If someone uses CaseDetective only once every month they’ll be able to use the v1.0 trial for over two and a half years! It’s a very unlikely scenario, but is possible. I’ll just have to work hard to make CaseDetective irresistible and so useful that people will feel like paying for it just to say thank you :-)

Yay! Free alcohol!

I got a delivery today of a nice little box with the word “fragile” written down it’s sides, it could only be one thing … Yay! Free alcohol!

yay_free_alcohol_cartoon.jpg

Thanks to Hugh Macleod at gapingvoid.com bloggers in the UK and Ireland are getting the chance to sample some Stormhoek wine supplied by Orbital Wines. Each bottle comes with a little booklet and a personalised label:

stormhoek_wine.jpg

Ordinarily I’m a red wine drinker, it has to be pretty good and plenty chilled before I’ll usually enjoy a white.

I’ve just returned from a hard game of squash and we’ve just had our first glass of Stormhoek with dinner, with the bottle having been thoroughly chilled in the fridge since lunch time. What a lovely wine, it really does taste very fresh and fruity. Mandy (my wife) says “it’s alright”, which believe me is quite a compliment!

So it’s a thumbs up from the Jones household, we’ll be keeping our eyes out for Stormhoek when next shopping for wine.

CaseDetective Beta 3 Released

As well as releasing my UNIXCrypt RealBasic module, I’ve also released the latest, and hopefully last beta of CaseDetective.

This one doesn’t look like much on the outside, but has a few fixes on the inside, and a bit of mutli-threading chucked in to make things a little more responsive in the GUI while grabbing data from the database.

ALso, I’ve put in the framework for the Help system. It’s a little sparse at the moment, but should be improved plenty in time for the 1.0 release.

I had seriously considered making some big changes in this version (as discussed in an earlier post), but decided that they ought to wait until 1.0 is out the door. I’ve got users who are pretty happy with the way CaseDetective works at the moment, so I don’t want to rock the boat just yet, I’ll leave that until 1.1, giving me plenty of time to make sure I get it right. This release is made with RelBasic 2005r2, but doesn’t include any of the stuff I need to do to support MS Access databases.

I’m chomping at the bit to get onto v1.1, but I know I should finish off the docs and final polish of 1.0 first, and make sure this final beta is as solid as I hope.

UNIX Crypt For RealBasic

A little while ago I mentioned that I had initially built the user authentication in CaseDetective so that it used the user’s FogBugz website to do an automated physical login, and then parsed the resulting page for specific variable names that can only be there if the user had logged in successfully. This was a bad idea, and not even my preferred method, but I’d done this way to avoid asking Fog Creek about the encryption they use for user passwords stored in the database. Anyway, once I wised up and asked Michael Pryor if he’d let me in on the secret, he of course told me that it was the UNIX Crypt algorithm (funnily enough, I’ve recently found a discussion forum post where he had already mentioned it, but it was after I released Beta 1).

I was over the moon when Michael told me it was UNIX Crypt because I figured there’d be thousands of ready made modules that I could choose from that would give me instant access to the one-way crypt I needed, it’s so old, so ingrained in the UNIX world that it was surely going to be a breeze.

So I checked out my Einhugur plugins only to find no simple “UNIX Crypt” functions. Not worry, there are a tonne of different algorithms in the e-CryptIt module, it’s probably just down to me not knowing what the underlying cryptographic algorithm is, so I search the net…

I spent a few hours searching the net, eventually finding out that UNIX Crypt is a derivative of the Data Encryption Standard (DES). I learnt a lot about encryption in these few hours, including that UNIX Crypt is pretty weak, but also that it’s pretty easy to password check with because the “salt” is embedded in the encrypted string as the first two characters. I also leant that in a lot of systems the DES algorithm has been superseded by using Blowfish or Twofish.

Back to my plugins … nope, nothing about DES, not a sausage. Bugger! So I sent an email off to the Einhugur list asking if I was right, maybe I’d missed something, but alas, DES and UNIX Crypt are not supported by e-CryptIt, and no-one knew of another plugin that would help. Double bugger! At least e-CryptIt has Blowfish and Twofish, but that doesn’t help me just now as I need the DES version.

There are very few modules out there (”out there” being the internet) that do UNIX Crypt, and most are for Windows and would require that I bundle another dll with my application, which I really didn’t want to do, I already bundle the eSellerate dll and didn’t want to have to bundle any more.

Even if I did buy a dll (most of these dll files require payment) and distribute it for Windows, it would still have left me with having to find a solution on Mac OS X.

I played around with RealBasic Declares on Mac OS X as it comes with a “unistd” library that I found I could potentially call to do a crypt. But I had very sporadic results, and many crashes, and relying on the placement of a library outside of my control gave me the hebees (you’re supposed to be able to just use the sort name for the library without it’s full path, but that never worked for me, I had to use the full path to get anywhere at all).

What next, write my own UNIX Crypt module from scratch, in RealBasic? Surely not, that’s got to be painful!

Well, I had no choice, I either wrote a module to do UNIX Crypt from within my RealBasic application where I had full control and no cross-platform woes, compile up some freely available C code for both Windows and Mac OS X, keep searching fruitlessly for an existing robust cross-platform solution (that isn’t there), or forget the whole thing and go back to looking at building Integrated Windows authentication into my FogBugz web logon code (shivers with dread).

During my research on the web I found various bits of C code that did DES and UNIX Crypt that had BSD or other unencumbered licenses, so I had at least some code I could look at for inspiration. Now, I’m no C coder, I can read it a bit as long as it doesn’t get too hairy, and I’ve even written a few modules and applications for clients where we needed to wrap third-party libraries with C code, and then in some cases wrapped this with Informix 4GL code. I’ve had some great success with that kind of thing. But I’m telling you now, people that devise encryption algorithms are seriously deranged, how on Earth they come up with this stuff beats me, and how some of these C developers got to the place where they do certain things they do I don’t know. It’s just plain scary looking at encryption code.

I’d found via Wikipedia.org a great DES Algorithm Illustrated page (WARNING: if you go to that site, don’t bother looking at other parts unless you’re in the privacy of your own home as your boss may not appreciate the scantily clad ladies) that explained bit by bit (literally) how the DES algorithm worked with a known input and output. Without that site I’d have been right royally stumped, it allowed me to build a stand alone project in which I could test my UNIXCrypt module with a known input and output not only to the entire algorithm, but to each little segment of the calculations.

It took a few days, and a lot of hair loss, but eventually I had straight DES working fine, I could perfectly replicate the inputs and outputs from DES Algorithm Illustrated. Woohoo!

All I had to do then was implement UNIX Crypt. Sounds easy don’t it? I thought so too. The problem is that UNIX Crypt doesn’t use straight DES, oh no, that would be too easy, it has to be mucked about a bit first! ;-)

UNIX Crypt sets the DES key in just the same manner as normal DES does using the string (password) passed in. So far so good. Then it runs DES encrypt 25 times using that key, but starting off with a message containing just zeros. This also sounds pretty straight forward doesn’t it? Well, on each of those 25 runs of DES crypt, at each point that the “E-Box” is expanded (you’ll have to go look at that DES Algorithm Illustrated site to see what that means) the resulting bits (which by the way have already been mucked about enough thank you very much) are mucked about again depending on some XOr criteria derived from the two “salt” characters.

Luckily I had various bits of C code to check out, but they don’t really explain anything, and try as I might I just couldn’t get the bloomin’ thing working.

I tried for two weeks to get UNIX Crypt working, it was so frustrating. I had DES working, I knew it, so why could I not get these seemingly small changes to adhere to the UNIX Crypt algorithm working?! I think I must have lost at least half my hair over those two weeks (that’s my story and I’m sticking to it, I haven’t been losing my hair at all before that, not one ;-) ).

Finally I sent out a plea to the RealBasic Network Users Group (NUG) for anyone who knows any way of getting UNIX Crypt working reliably cross platform from RealBasic. I was prepared to throw all my work away if anyone came up with anything good. No one had suggestion that would work for me, it seems no one on the group had needed to do it before cross platform, although there was a Mac OS X solution for encrypting passwords.

Eventually a guy called Mark Nutter found some Java code I hadn’t seen before that did UNIX Crypt, I looked at it and found that it very closely matched the C code I’d used as a template for my own development. I followed it through and compared to my own RealBasic code, just like I had the C code, but alas could not find anything that looked wrong.

When I reported back to the NUG that I still couldn’t find the problem Mark then offered to take a look at my code to see if anything jumped out, so I took him up on his offer and sent him my test project. Later that day he wrote back, unfortunately not having been able to find the problem. Here’s a quote from his email:

… I’m checking your RB code against that, and
everything seems to check out, except that the results
aren’t matching. But encryption algorithms are so
twisted! My brain hurts after only a couple hours, I
can imagine you’ve been going bonkers.

At this point I’m fairly stumped also. :(

I wrote back thanking him for trying and telling him not to try looking at it any more, I didn’t want to be held responsible for his decline into madness! I’d just have to try and step through the code bit by bit and see if I can find the place where it went astray.

When Mark wrote back to say good luck he casually mentioned that I should try and compile up the C code I’d used as a template and output to a buffer every now and then and compare with my RealBasic code. Doh! Why the hell didn’t think of that sooner, I had code that I knew should work, tried and tested, but just in another language.

So I jumped into XCode created a new project and copied in the C code I’d been using and set about creating a command line app that would simply print out the bit strings at various points. I started off by making it simply do the UNIX crypt to make sure it was doing the right thing, outputting the input and output strings. It worked.

Then I decided I’d start off slowly, I just added one intermediate output of a “bit string”, the bits as accepted by the setkey function which is called right at the beginning before doing the real DES stuff. Then did the same in my RealBasic code. Ran the two of them, and guess what, I already had a miss-match! Bugger, bugger bugger! If I had a problem this early on in the process, it was going to take me bloody ages to find and fix all the other errors.

When I looked closely at the two bit strings, that should have been the same, but weren’t, I noticed that my code was creating a string very similar, in fact it was almost identical except for an extra “0″ at the beginning of the string, and then again before the next recognizable pattern of “1″s and “0″s. Damn, my code was passing in 8bits per character rather than the 7bits per character for the key that the C code was using. Checked my code, sure enough, there it was, an extra “0″ in my bit mask when converting the ascii characters to binary!

Oh well, delete one “O”, recompile and run again to find the next error. Except there wasn’t an error, my test project only bloody went and spat out the correct encrypted string! Woohoo! Do a little jigg!

It only took me a couple of minute to import my UNIXCrypt module into my CaseDetective project and hook it up to get it working, finally I could authenticate a user’s FogBugz login id and password against the database, cross platform and without any dependance on external modules or applications. Phew, I can’t tell you how much of a relief that was, I was so happy.

So, after all that hard work, fretting and nearly going insane, I’m going to do one last insane thing, I’m releasing my UNIXCrypt RealBasic module as a free download. You can grab it from my IMiJ Software website, it’s in the “Free Stuff” section.

You can download or check out the UNIXCrypt REALbasic module and test project that I created from the rbunixcrypt Google Code project. It’s under a MIT license, so you can do pretty much anything you like with it.

You can use the UNIXCrypt RealBasic module to simulate UNIX Crypt by simply calling “UNIXCrypt” with a string and two character salt.

Thank you to everyone who helped or tried to help me get this done, hope someone finds it useful.

Started to use RealBasic 2005r2

Just yesterday I started to use RealBasic 2005r2 for my development of CaseDetective.

I had been holding off from using the latest version of RealBasic because of two things:

  1. I was happily using v5.5.5 for my project and didn’t want to risk breaking anything by moving to new versions of my tools.
  2. Didn’t feel v2005 was quite stable enough (from comments in the RealBasic mail lists).

So given those two very important reasons, why have I decided to move to v2005r2 before v2005r3 has been released?

The prime reason is that there are features in v2005 that I think I need now to be able to develop features I’m planning and to improve some other areas of my software.

In particular, I’ve been struggling with trying to support Microsoft Access databases as a data source in CaseDetective. It’s not that I can’t connect to the database etc, that’s easy with ODBC, it’s that Microsoft Access does not support the advanced SQL I can use in the other supported FogBugz databases, namely Microsoft SQL Server and MySQL.

By moving to RealBasic v2005 I think I can now make changes to the way I handle the data so that I can keep all the extra fields I give the users of CaseDetective (such as Last Updated By, Closed By, Resolved By etc) and also support MS Access. Whereas at the moment I simply can not keep those fields and support MS Access too, the SQL I’m using just doesn’t work no matter which way I do it, so I’m going to have to split the grabbing of the extra data into separate queries.

Splitting the retrieval of data into separate queries means I can’t use SQL “sort by” for when the user wants to sort by those extra fields, I’m going to have to bring the sorting back into the application only. This means sorting arrays of data, which will hopefully be easier to do using RealBasic 2005’s new “SortWith” method.

I first came across the new array SortWith method in a post about new features in RealBasic 2005 by Jonathan Johnson on his NilObject blog. The new SortWith method allows you to sort an array and keep other arrays that have the same number of elements in step with the re-organising. So, if you sort the LastNameArray you can keep the FirstNameArray in step, so that elements 1, 2, 3 … n are always locked together across the two arrays. And you can do this for multiple associated arrays at the same time…

LastNameArray.SortWith(FirstNameArray, MiddleNamesArray, EmailAddressArray)

This would mean if the 3rd element of LastNameArray was moved into position 1 as a result of the sort, then elements 3 of FirstNameArray, MiddleNamesArray and EmailAddressArray would also move into position 1 of their respective arrays.

I’ve yet to try this out (so I don’t know whether sorting by one array and then another would allow me to get what you’d expect from “sort by LastName, FirstName”), and haven’t fully planned my new architecture, but I think this will help me get to where I want to be, and will hopefully open up the possibility to add more sort fields too as once this is implemented I can’t see anything stopping me from sorting by every field I make available, whether basic FogBugz or derived field. It’ll just take a little extra work to make sure all dates and times have a SQL format string (or TotalSeconds integer) companion to get around the restrictions of the SortWith method, but I already have this for most fields.

Update: Stuff arrays, me thinks RealSQLDatabase (a.k.a SQLite3) is the way to go, can you say “super fast in-memory SQL database”! This to be honest was my prime reason for moving to RB2005, but I wasn’t sure it would work. I’ve just spent my lunch hour looking into this further, and as soon as I found out that I can do in-memory SQLite3 databases in RB2005, and can then save the contents out to disk by attaching a file based database and inserting my in-memory table’s data into the file based database’s tables, I was sold. This totally opens the door to some features I want to implement in the future. I’m a very very happy man! :-)

A nice thing I’ve already noticed with RB 2005 is that a disabled listbox on WinXP actually looks disabled now, whereas in RB 5.5.5 it looked active, but you couldn’t do anything with it, which was very confusing. On the flip side I seem to have lost one of the separators in a menu, so I need to sort that one out.

I think I’ll finish off a couple of small changes to CaseDetective and release another (final) beta made with RB 2005, and then I’ll tackle the MS Access problem for a separate release, so look out for a new version of CaseDetective in the next week or so with some minor updates and fixes.

OpenBase Adds RealBasic Stored Procedures Support

In the wee hours of this morning, while I was sound asleep, my server received an email from OpenBase announcing that OpenBase 9.1 will be released on September 14th.

Usually I’d just delete such an email as I’ve yet to use OpenBase in anger, and haven’t really played with it for a couple of years now, when I first had a dabble with WebObjects. However, in their list of new features for this version was support for RealBasic Stored Procedures, which caught my eye.

This sounds great, I’ve spent the last year writing RealBasic code, and so am slowly getting more and more comfortable with it, to the point that I miss it’s features when working in other languages. So this announcement along with the much anticipated “Swordfish” web application framework from Real Software is starting to make my investment in time and money in RealBasic a good bet.

Of course, I haven’t got OpenBase, don’t as yet have a really good use for it that would warrant spending that kind of money (I’ve always felt it costs just a little more than I can afford), but with all the other improvements and new features that are in v9.1, I’ll certainly be looking at it again soon. It’s maturing nicely.