su.percilio.us

I recently built an Arduino stoplight web server so we could monitor our builds at work. I found a great tutorial at Instructables.com (http://www.instructables.com/id/Arduino-stoplight-web-server/) it was put together by mkanoap and is pretty damn good. Basically in the end you have a traffic light with an IP address that sits on your network and you can change the lights by sending it HTTP requests.

It was a fairly straightforward tutorial, although it is not necessarily recommended for a beginner in amateur electronics as I was. I tried to wire up the circuit myself, but failed miserably and ended up purchasing a printed circuit board (PCB) on line.

We use Jenkins for our build server, so I wrote a PHP script to poll Jenkins every 5 minutes and change the traffic light color based on our thresholds. I even wrote a PHP object to make sending the HTTP requests to the Arduino even easier. Check out the photos below. I’ve even included the PHP script in case you ever decide to do this yourself and need an easy PHP abstraction to make the pretty colors change.

At the link below you can download the PHP class I use to change the lights in my build monitor polling script. Below the link you can see some examples of how to use it. It requires php-curl, but other than that, it’s pretty much good to go and very easy to use.

PHP Stoplight Script

$stoplight = new Stoplight('192.168.1.111', 'p@ssw0rd');
 
// Turn on one light and turn off all others
$stoplight->red();
$stoplight->yellow();
$stoplight->green();
 
// Turn on any combination of lights
$stoplight->on(array('red', 'green'));
 
// Turn off all lights
$stoplight->off();

I ran across an interesting issue today. An issue, that when I Googled it, appeared to have been resolved 2 years ago. However, I was using up-to-date versions of PHP, ImageMagick and the Imagick PECL library.

So, on to my issue. I have an abstract piece of code that when given the proper variables will set set the locale for my application. It sets the locale thusly:

setlocale(LC_ALL, 'fr_CA.utf8');

Later in the application, I was trying to draw a rectangle for a gauge that would graph a number against another number. When I tried to call the Imagick rectangle method, all hell broke loose.

$gauge = new ImagickDraw();
 
$x1 = $config->position->x1;
$y1 = $config->position->y1;
$x2 = $config->position->x2;
$y2 = $config->position->y2;
 
$gauge->rectangle($x1,$y1,$x2,$y2);

I then got this lovely message:

Fatal error: Uncaught exception 'ImagickException' with message 'Non-conforming drawing primitive definition `,' @ draw.c/DrawImage/3123'

Googled a bit and everything said this was fixed in the PECL library as of 2007. But it’s 2010 and I’m still having the issue!

Turns out if you set “LC_NUMERIC” to any French language locale, it can make interpreting floats difficult, because the French use commas (,) instead of periods (.) to delimit their fractions numerically. This caused the “rectangle” function of the “Imagick” PECL library to go crazy.

My Solution?

I altered my brash locale setting code to be a little more granular:

setlocale(LC_TIME, 'fr_CA.utf8');
setlocale(LC_MONETARY, 'fr_CA.utf8');

Setting locales for “LC_TIME” and “LC_MONETARY” was enough for me to get the programmatic localization I needed. And it made sure that “LC_NUMERIC” did not get set and cause localization on floats to interfere with external libraries.

This is “bang your head against the wall” stuff people. I hope this helps someone.

Throughout my many development projects I have produced applications in many different fashions. I’ve written code by shelling into the production server and editing files that were serving up content to end users in real time (YIKES!). I’ve developed directly on test machines and SCPed files directly up to production servers (YIKES AGAIN!). In the last few years, however, I’ve worked with different companies, many of whom have much more formalized development procedures, and I’ve learned a few things … to put it lightly.

One of the more profound techniques I’ve brought away from my tenure as a web software developer, is the use of a Virtual Machine or Virtual Development Environment as the first step in the development process. I won’t go into the details of how Virtual Machines are utilized in a development procedure (hopefully if you’re reading you are already using a VM), but I will touch on a few of the basic concepts as I describe my procedure for developing Facebook applications using a VM.

Virtual Machines are great for development. Each engineer gets their very own server to develop on, that mirrors the test and production servers. They can hack away at core code (or installation specific code) in a completely sanitized environment and only add their code back into the shared pool when they’ve tested it thoroughly themselves. They could even create their own continuous integration server if they wanted to. I say these things only to reinforce the fact that I like to use VMs and even on small-scale projects I implement them. This brings us forward to the point of this post:

Developing a Facebook Application on a Virtual Machine

NOTE: The examples below make some broad assumptions of your level of knowledge, like it assumes you have mod_proxy already enabled and it assumes that you already have your VM user talking successfuly to your test server user. This is just a high-level overview. Hopefully you can use your highly developed inference skills to fill in the gaps.

In my most recent incarnation as a “Web Developer,” I am the “Senior Developer” (read “Only Developer”) producing and maintaining a small-scale Facebook application. Please keep in mind that the following procedure is based on my current system at work and to follow it exactly would mean you have to have a similar infrastructure. I have a little Linux test machine on-site. It’s an old Dell desktop running Ubuntu DESKTOP, so nothing fancy at all. If you’re using an off-site test machine or Windows servers, you can probably still get some insight from this post, but the specific examples won’t be of much help.

Now, scale of the project aside, I like VMs, so I implemented one for developing this application. I set it up as a simple LAMP stack using Ubuntu Server 9.10 and turned on all the necessary pieces. I set up my standard framework and put everything into place. “This is great!” I thought. “I have completely streamlined this process! Look at this great little VM I have set up to develop on.”

Very soon after that I got my first ticket telling me we had to launch a new customer installation of the app. “All right!” I exclaimed. “I get to use the new VM!”

NOTE: We do a white-label application, so it needs to be completely devoid of our influence. No corporate name or logo apart from the client’s is allowed to be seen, in doing this we create an application on Facebook for each customer installation. This way we can have custom logos, description, and canvas page URLs per client. It get a bit unwieldy as we have to create TWO applications per client, a production application whose canvas callback URL points at our production system and a test application whose canvas callback URL points at our test server, but it works out well for our size.

The first step in launching a new client installation of our app is to create the test application. I log into our Facebook developer account and proceed to begin setting up the application. I get to the “Canvas Callback URL” section and a moment of severe disappointment grips me. The VM is completely internal and there is no way for Facebook to talk to it. Unless I’m willing to create a “Facebook VM application” for each of our developers (I know it’s only me, but I’m thinking of scalability here, people) and then there’s the problem of DNS entries, because every VM has an internal IP on our network, which is not visible to Facebook anyway.

BOO. It looked as if continuing to develop code directly on the test server was going to be the only way to test our Facebook application properly.

I wasn’t satisfied with that conclusion. I liked VMs. I liked sanitized development environments, especially when working with entry level developers. There had to be away around this.

I started down the road of trying to jury rig the application so it could function outside of Facebook. Meaning there didn’t need to be a Facebook frame around it and there didn’t have to be a valid Facebook session present. Then we could just access it via an internal IP like: http://192.168.54.75:/facebook-app or if we messed with our host file a bit http://nick-vm/facebook-app.

In the VERY long-term this could have been a viable option. Development would have been simple and the application could have had more breadth as it could have functioned as a standalone website. Due to time constraints, however, this was not going to work. It was surprisingly difficult to add code that could properly spoof a Facebook session and even if I had achieved that goal, it was turning my session code into spaghetti with all sorts of nasty environment based conditionals that added NO VALUE (and actually decreased value because of the increasingly unreadable code and creeping file size) to the final product that would live on the production server.

In a stroke of “genius” (read “Nick figuring something out”) I decided that the way to go about this would be to trick Facebook into thinking the content served to it was coming from the test server, when in fact, it was coming from my VM.

With some tricky BASH scripting and Apache’s very convenient mod_proxy module, I was able to create an automated check in/check out system that would tell the test server to take Facebook’s incoming HTTP request and forward it along to an internal machine … my VM, on a temporary basis. This meant that the content being served to Facebook would be coming from my virtual machine using the test server as the intermediary.

There was a number of things gained by this approach. The canvas callback URL in Facebook could permanently remain as “http://application.testserver.com/,” there would not need to be any changes to the application business logic, and core code could be developed by an engineer with no harm coming to code on the test server that was currently powering the test installations.

The check in/check out scripts were superbly easy to write too and were easily executable directly from the VM. Each script, when you boil it right down, needed to accomplish one thing. The “check out” script needed to alter the Apache virtual host on the test server to proxy the request through to the virtual machine – and the “check in” script simply needed to reverse this process.

So, the first thing we needed was a vanilla virtual host file that could be used as the proxy template:

<VirtualHost *:80>
    ServerName {appname}.testserver.com
    RewriteEngine On
    RewriteRule ^(.*)$ http://{user}-vm$1 [P]
</VirtualHost>

Replacing the {appname} and {user} variables using a BASH script, AND making sure that the necessary entries were present in the test server’s host file, AND ensuring my virtual machine had a static internal IP, AND ensuring that my virtual machine had a virtual host file that accepted the HTTP request “nick-vm”, AND ensuring that Apache got bounced on both my virtual machine and the test server … made everything work perfectly. Facebook never knew the difference.

Here’s a snippet of the BASH check out script that shows how to edit this file. You’ll have to visualize in your minds how the variables get populated:

ssh testserver.com mv $VHOSTDIR/$APPNAME.conf $VHOSTDIR/$APPNAME.conf.checked.out.by.$USER
ssh testserver.com cp -f $VHOSTDIR/vhost-checkedout-template.conf $VHOSTDIR/$APPNAME.conf
ssh testserver.com sed -i "s/\{appname\}/$NAME/g" $VHOSTDIR/$APPNAME.conf
ssh testserver.com sed -i "s/\{user\}/$USER/g" $VHOSTDIR/$APPNAME.conf
ssh testserver.com chmod 644 $VHOSTDIR/$APPNAME.conf
ssh testserver.com chown $USER:dev $VHOSTDIR/$APPNAME.conf

That would spit out a vhost that looked like this:

<VirtualHost *:80>
    ServerName app1.testserver.com
    RewriteEngine On
    RewriteRule ^(.*)$ http://nick-vm$1 [P]
</VirtualHost>

The original virtual host file would be safe and sound – copied over to a file called “app1.checked.out.by.nick.” This would make it instantly apparent just by looking at the file system that the app was being edited by someone. This was also the way the check out script would know it could not check that file out until it was checked back in again.

After bouncing Apache, this virtual host would make sure all requests to app1 from Facebook went through to my virtual machine. Of course, the test server needed to know that “nick-vm” pointed to my VM. So, I gave my virtual machine a static IP and made sure the test machine’s host file had that record:

nick-vm    192.168.54.75

This is really the only “unscalable” piece of this system. For every engineer you onboard, you need to allocate a unique static internal IP for their VM and add the record to your test server’s host file. But the onboarding process will include creating accounts for them anyway, so you can just make this part of that procedure.

Once the app was checked out and the necessary code edited, all the check in script had to do was reverse the process:

ssh testserver.com mv $VHOSTDIR/$APPNAME.conf.checked.out.by.$USER $VHOSTDIR/$APPNAME.conf

Then bounce Apache on both the test server and the VM. Done and done.

Now, my specific check in/check out scripts did other things like packaging up databases and checking app code out of Subversion. They also fixed permission and did everything needed to properly configure an installation to run on a server. But the key point was it used the Apache virtual host files on the test server to create that connection between Facebook and the virtual machine.

A relatively large caveat to this system is that only one engineer can work on an application at any one time and the virtual machine version of the installation becomes visible to ANYONE in the world who adds the test application to their Facebook account. Since, it’s only a test application, this isn’t too big a deal, and can be combated by adding some sort of staging environment to your development process. This will ensure that clients never see test versions of the application, but it will also mean that each application installation will require three Facebook applications instead of two.

In the end I found this to be a pretty elegant solution. Multiple engineers could work on multiple apps and test them directly through Facebook. The biggest gain was that the core application code that was shared between all installations could be edited without ever impacting the stability of the core code on the test server. This meant that if you were changing core functionality, only one app at a time would be affected. This was a HUGELY significant, because clients could be testing their applications on the test server and I could be developing core code on my VM and the integrity of their applications would not be affected.

You may think that this is a bit of overkill for your application, but try to look at it this way: too many applications are not given the serious attention they need at the beginning and wind up being amateurishly written applications that have important jobs to do that they are incapable of doing. If you take some large-scale approaches in the beginning, you can save yourself A LOT of headaches later on. And we all do want our Facebook applications to become the next big thing … don’t we?

INTERESTING NOTE: I should add that I have successfully deployed this technique via VPN. So, I can have my VM running on my laptop at home, VPN into my office, and have the test server proxy the HTTP request from Facebook over VPN back to my VM at home. This means you do not even have to be at the office to check out an application installation and work on it. There might be an eensy-weensie loss of performance this way, but that’s a small price to pay to be able to write software in your skivvies.

Expanding on my last post about being able to successfully connect to Convio Web Services in three lines, I’d like to get a little more in depth and actually construct the necessary SOAP headers and make a simple call. I’ve written a small PHP implementation as well. I was able to make a successful log in to CWS in just 2 lines of PHP code!

Now, as far as CWS goes, Python and PHP may not instantly be an obvious choice as far as languages go, mainly because they are both largely scripting languages. CWS is mainly for robust data syncing operations and those tasks generally go to languages like Java or C#, however, I’m a high level language guy, so I go with what I know. Also, Python and PHP both make SOAP exceedingly easy to work with, so there’s no real reason why you can’t, with a bit of discipline, write a perfectly excellent and robust data sync operation in either of these languages … especially Python.

So, here are my examples. They are exceedingly simple and easy once again. The Python example leverages suds once again and the PHP example is using PHP’s pretty impressive built in SOAP library. Each example shows you how to create the SOAP client using the Convio WSDL, make a successful log in to CWS, create a valid session header for subsequent calls, and how to make a call that grabs the number of constituents that have been added to the Convio database.

If you find either of these examples helpful, please let me know! Keep in mind these are just functional examples. There is no proper error handling or any sort of architecture here; they are just some bare bones examples on how to get data from CWS.

Python

from suds.client import Client
 
# Load the WSDL                                                                                                                                                                                                                                
client = Client('https://secureX.convio.net/1.0/CLIENT/wsdl')
 
# Log in to CWS                                                                                                                                                                                                                                
login = client.service.Login('apiadmin', 'password')
 
# Set the session header                                                                                                                                                                                                                       
session = client.factory.create('Session')
session.SessionId = login.SessionId
client.set_options(soapheaders=session)
 
# Get the number of constituents added since last sync                                                                                                                                                                                         
data = dict(PartitionId=1001, RecordType='Constituent', PageSize=1)
response = client.service.GetIncrementalInsertsCount(**data);
 
# Show the number of constituents added                                                                                                                                                                                                        
print '%d constituents have been added since the last sync.' % (int(response.RecordCount))

PHP

// Load the WSDL                                                                                                                                                                                                                               
$client = new SoapClient('https://secureX.convio.net/1.0/CLIENT/wsdl');
 
// Log in to CWS                                                                                                                                                                                                                               
$login = $client->Login(array('UserName'=>'apiadmin', 'Password'=>'password'));
 
// Set the session header                                                                                                                                                                                                                      
$headerBody = array('SessionId'=>$login->Result->SessionId);
$header     = new SoapHeader('urn:soap.convio.com', 'Session', $headerBody);
$client->__setSoapHeaders($header);
 
// Get the number of constituents added since last sync                                                                                                                                                                                        
$data = array
(
    'PartitionId' => 1001,
    'RecordType'  => 'Constituent',
    'PageSize'    => 1
);
$response = $client->GetIncrementalInsertsCount($data);
 
// Show the number of constituents added                                                                                                                                                                                                       
print(sprintf("%d constituents have been added since the last sync\n", (int) $response->Result->RecordCount));

I was on the look out for a good SOAP library for Python now that I am the new Convio “API Guy.”

In my search I discovered this library:

Suds

To test it out I tried making successful connection to Convio Web Services.

1
2
3

from suds.client import Client
client = Client('https://secureX.convio.net/1.0/CLIENT/wsdl')
result = client.service.Login('apiadmin', 'password')

Boom. Straight in, no fuss no muss. Pretty impressive. I’m going to play with Suds and CWS a bit more to see if there is actually a robust Python based syncing application that can be made here. I’m led to believe Suds is not great for creating SOAP servers, but it makes for a very “Pythonic” and easy to understand way to create a SOAP client.