Category: Uncategorized

I was getting very confused with my django DB quite happily setting the date in SQLite, but displaying a ” when loading it back from SQLite.

The SQLite field was a DATETIME, and in Django a ‘DateField, which I would have thought should work.

Alas, no.

Apparently the SQLite field must be a varchar(20) (or similar) for a date w/o time.

Strange.

I made a quick python script to convert OpenLP song (lyric) databases into the presentation format used by ProPresenter.

https://github.com/danthedeckie/OpenLP-To-ProPresenter5-Converter

is the link.  I put it together in a few hours, it should have been quicker, but I’m still re-acquainting myself with python.

It is a nice language.

One thing I did today, while cleaning up a bit, was wonder about something I remember from doing python years ago – string concatenation.  Joining two strings (texts) together.

“Hello” + “World” -> “HelloWorld”

I remembered something about it being slow, and python recommending using

”.join((“Hello”, “World”))

which seems to me one of the most blatently ugly obscure gotchas I’ve come across in a long while.

Anyway, I refactored my code into that style – converting databases and song lyrics and writing XML stuff is pretty much all Text formatting and concatination.

It made no discernable difference.  So I went back to normal easier to read x += y, and x = y + z type of code.

I made a quick python script to convert OpenLP song (lyric) databases into the presentation format used by ProPresenter.

https://github.com/danthedeckie/OpenLP-To-ProPresenter5-Converter

is the link.  I put it together in a few hours, it should have been quicker, but I’m still re-acquainting myself with python.

It is a nice language.

One thing I did today, while cleaning up a bit, was wonder about something I remember from doing python years ago – string concatenation.  Joining two strings (texts) together.

“Hello” + “World” -> “HelloWorld”

I remembered something about it being slow, and python recommending using

”.join((“Hello”, “World”))

which seems to me one of the most blatently ugly obscure gotchas I’ve come across in a long while.

Anyway, I refactored my code into that style – converting databases and song lyrics and writing XML stuff is pretty much all Text formatting and concatination.

It made no discernable difference.  So I went back to normal easier to read x += y, and x = y + z type of code.

I appologise for how techy and geeky and boring this post is.

I even find it so myself.

But I need to get this off my chest, so I can actually write about interesting stuff later…

And some geeky people may actually find something comprehensible in here.  Perhaps even interesting.

If you want to read it, then if your eyes glaze over with techiness, then stop reading, drink a glass of water, scroll down to ‘And another thing‘, and read from there.

So then.

Most of my recent work has been on the omnitube.org website.

It’s a site I put together with David when I visited OMNIvision 4 years ago (now that seems weird!).  We put it together in about 2 weeks, based on a well known large complex CMS (Content Management System) called Joomla, which was the OM standard web-site-software-package, a few very large complex modules for that, specifically “VirtueMart”, plus a bunch of my own hacks in php/javascript, plus a load of very clever HTML/CSS from David.  Since then, I’d done a few fixes for them, logging on from a ‘net cafe in Singapore, and when Becky and I arrived at OMNIvision 2 years ago, it was basically in the same state.

The trouble was that in the 2 years since we’d put it together, the internet had changed quite a lot, and we hadn’t had the time to really work on OMNItube to change with it.  Joomla had advanced significantly, as had Virtuemart, and alas, we couldn’t smoothly upgrade to the latest versions of those without majorly breaking everything.

About a year ago (from now), I decided it was about time to do something about this, as I was scared that a now 3 year old install of joomla/etc without being able to have security updates was a bit scary.  Especially as we wanted to sell videos online, and DVDs, and so were doing credit-card type security stuff.  And the system admin wanted to upgrade the server, and I thought doing that would probably break everything again.

I spent about a week trying to upgrade things to the latest versions of joomla and virtuemart.  Some bits I got almost close to working, but in the end it just totally bogged me down.  Joomla + Virtuemart + all the plugins and modules which were installed to make everything work as we wanted ended up at around 7000+ files, and 800’000 lines of code.  Not joking.  So tracking down bugs and errors and trying even to find which function I was looking for took forever, and I simply don’t have a big enough brain to hold that much in my head at once.

So after a week or two of working hard to try and upgrade things, I gave up, it was just too big a job for me to do on my own.  I’d made no progress, and it was still a big stinkin’ mess.

We also had some new features we wanted to get working. Specifically, iTunes subscriptions, XML sitemap, facebook commenting/liking/etc, and related product lists.  The “search” system was also very broken, giving pretty much useless results most of the time.

I’d just read an article online about making stuff simpler, ‘the no-framework framework’ or something like that.  And looking at the SQL database, I figured I could write the iTunes and sitemap features we wanted in just pure basic php.  So I did.  It took about 3 days, which was pretty good, I thought, considering I’d not really done significant amounts of programming in about 4 years.

So then – perhaps alas? – I thought, well, why not re-write the whole of OMNItube in a similar manner?  I mean, how hard can it be?  It’s a very simple website, after all….

In 2 weeks after that, I’d put together a quick ‘demo’ of the site, which took it’s data from the joomla database, had all the basic functionality of the front end,  but in 700 lines of code, TOTAL.  No Joomla, No Virtuemart, nothing.

OK, so it was only the front end, if you wanted to actually add a new video or something you needed to edit the database manually, rather than use the horrendously over-complex somewhat-broken-by-our-CSS-mods joomla back-end, but hey!

700 lines of code is something I can hold in my head, no problem.  Way better than 800 thousand!

So anyway, that’s been the big project for me, over the last year.  It’s taken me 3 re-writes, which is what took so long.  I really should have spent a lot longer learning stuff and looking at alternatives before launching in to it as I did…

I first wrote it doing all the SQL in-line, sort of saying “I need a list of the products, so the SQL for that is SELECT…blah.blah.blah” and so writing it then and there.  Which works for very simple stuff.  But I began to find patterns, of course, and so wrote functions and classes to abstract stuff out, which was good!  But each time I did something like that, it meant re-factoring and writing stuff I’d already done, to make use of the new abstract methods I was coming up with.

Eventually, though, it was all working, pretty much.

 I slowly switched over bits of the live system to my new creation over time,  so the video embedding system was first, then the iTunes feeds, then the front end, eventually, and finally I was able to delete the joomla system, and run entirely on my much smaller simpler pure-php non-frameworked version.  I guess about 6 to 8 months ago?

So after the crazy summer, I did a bit of cleaning up, backups, version-management, etc, And a guy called Chris came to work with us, who is interested in doing web / programming stuff.   So I chatted with him, showed him OMNItube, as it was then, and told him about my future plans.

I’d found a cool looking database library for PHP called ‘Propel‘, which did much of the grunt work of doing SQL database querying instead of us having to do it.  I thought it looked cool, and would save us a lot of work in the future, and so Chris started work on porting OMNItube over to using Propel.  He also figured out a bunch of stupid stuff I’d been doing before, and found much better, simpler ways to do it.  Templating things, for the big example.

In some ways, I’d not really wanted to be working on any more OMNItube stuff for a while though.. I kind of thought ‘it’s working, I should probably just leave it until next year or something’.  But then after Chris left, I wanted to get his project all merged in to the main system, and not leave things in a mixed state, with 2 versions, his almost-done version, and the online-one with changes to it that I’d needed to make since he started….

Anyway.  I’ve spent the last few weeks hacking hard at OMNItube, until now it’s stable and running well using the new Propel based system.  It’s much simpler, much better, the administration/back-end thing is very cool, and generally just works great.

The whole thing is now 7000 lines of PHP, plus a few hundred of javascript and html, but still, that’s quite manageable, and I’m reasonably happy with it.  I’ve even open-sourced one of the smaller more generic bits of the system, and plan to put a few more modules online too, in the hope other people can find them useful in the future.

I’ve learned a lot, during this project.  I understand from experience a load of programming concepts a lot better than I used to before.  I’ve learned quite how much I dislike PHP, for one thing.  It really is a botch on top of a hack on top of a temporary way to fix a problem.  If I were starting the project now, I think I would pick a cleaner, simpler, better language.  Possibly ruby, python, LISP, javascript, perl5, or even haskell.

Another thing though, I’m really bored of web development.  It’s so much effort, and in the end, all you’re left with is a bunch of patterns of pixels on a screen.

I SO miss acting, theatre, drama, etc.

And the trouble is, after a day fighting with programming problems, I don’t then feel capable of doing anything creative.  It’s like my brain has just given up, as has my emotional capacity to care about creativity.  All I want to do is crash, which isn’t very nice for Becky, nor very helpful in building relationships, in staying up to date with anything actually interesting, or in keeping in practice with Clarinet or juggling or writing plays or whatever else I’d prefer, if I were less frustrated and computered-out to actually do.

Writing, for instance.  When I’ve been writing code, or trying to write code all day, I have no ability to write on this blog, say.  Or write newletters, or emails, letters, or even SMSs.

I’ve not really touched OMNItube now in about a week, which is why I’m able to write this post… but even still, I’m not quite able to break my brain out of that mould enough to write about anything interesting… all I can do is write about writing code.

Which I guess is a step in the right direction.

I’ll write a more general, non-techy post soon

using the normal ‘migration assistant’ didn’t work, so we had to do it manually. Possibly due to moving from a Intel based OSX-Server machine to a PPC based non-Server?

All user details are kept in

/private/var/db/dslocal/nodes/Default/users

in .plist files (XML, human readable).

and all passwords are kept in hash files in

/private/var/db/shadow/hash

The hash files are named after the “generateduid” found inside the users .plist file.

If you’re using home directories on the computer, you’d need to move them across too.

Ditto for user groups.  User groups are kept in

/private/var/db/dslocal/nodes/Default/groups

Move the files across to the new machine, and it all just works.

if you use “cp -n” on the command line, it won’t write over the top of files already there, so shouldn’t muck anything up (hopefully!)

NOTE:  One weird thing was that a bunch of the users didn’t appear in the Accounts Preference Panel.  This seems to be linked whether or not the account .plist file has a default shell set up:

    <key>shell</key>
    <array>
        <string>/bin/bash</string>
    </array>

it goes after the “realname” key/array pair.

Dear Internet,

You’ve been such a help in me finding stuff, random answers to obscure questions, that I should also submit random answers I’ve found out myself, so others can find them.

This is it.

I’m not good with computers!  Honest I’m not!

Dear Internet,

You’ve been such a help in me finding stuff, random answers to obscure questions, that I should also submit random answers I’ve found out myself, so others can find them.

This is it.

I’m not good with computers!  Honest I’m not!

There is a reason why I prefer text based / commandline / unix interfaces.

“Why?” You may ask?

So I had to update a website – basically changing from ‘2011’ to ‘2012’ – for someone.  Once I got in to the file manager (web based) I could find the graphics files to replace, reasonably easily, and upload new ones.

But I needed to update the text of the banner.  No .html files, or .inc or anything to edit.

So I went to the “WYSIWYG” editor section of the site.  I could edit everything, except that part.  And the footer.  And the page title. So, I could edit the “articles”, but not much else.

After hunting around for a while, I decided to try the “Website Management” area.  Total different interface again, loads of settings boxes and tabs.  I found there on different pages where I could set the page title, and so on.  But I couldn’t find the banner! Anywhere!

Eventually, this morning, I found it.

In “Website Management” -> “Structure” -> “Website Settings” -> “Code Injection” -> “Advanced” -> “Site Banner HTML Override — Edited”

I hadn’t found it before, because the section I needed was “Site Banner HTML Override”, and to find that, I needed to look through a drop-down box which was displaying “Extra Header Code (within )” with some javascript includes.

SERIOUSLY.

What is wrong with just having a bunch of text files, and then letting me type

“grep 2011 *”

And it giving me a list of files to edit?  HOW is this graphical interface any easier?

OK.  So ‘grep’ isn’t common English, I know.  Learning basic unix does have a steep looking learning curve.  At first.

In case you’re interested, I started, months and months ago, writing an “English-like” command shell.

So you could type commands like:

“list all files which contain 2011”

and it should tell you.

It’s on sourceforge, the first alpha versions of the code.


http://sourceforge.net/projects/daftshell/

The whole idea of ‘natural language’ interfaces fascinates me.  I should work on daftshell again, one of these days.

So. Yeah.  Graphical “easy” systems aren’t always easy.