Why post it here?  Well, PFUG regulars will be very familiar with our own Jerry Salem, who will be hosting a webinar January 18th at 2 PM EST, called Playing Nicely with Others – FileMaker and Content Management Systems.

There’s still time to register!  Go here for details:

http://www.filemakeracademy.com/?page_id=49

 

This week we are going to look at the integration of a Drupal web site and a FileMaker database.  We had a client with a specific need; an art school that offers classes to adults in their community.  They have a FileMaker back-end database that they use for registration and membership.  They also have a web site where people can view class offerings and enroll in classes.  The client would like to create a way to link these two independent databases.

There are two points where the two systems touch each other: presenting the classes offered for a given semester and the registration of students into specific classes.  The client and I decided to leave the first goal for the moment, and concentrate on the second goal of getting the registration data into FileMaker.

I am not working in a vacuum, of course, as there is already a workflow in place.  The accountant logs into their PayPal portal and does a screen print of the daily transactions.  This is handed off to a secretary who keys the information into the FileMaker database.

My suggestion to the client was to remove the excess printing and keystroke entry.  The workflow I suggested was:

1. A user (the secretary who used to key the data into FileMaker) clicks a button and is shown a list view of transactions from the web site.
2. The user then navigates to each record in turn and checks the student registering for the class.  Are they currently in the FileMaker database?  If they are, compare the existing FileMaker data with the data that the student entered on the site. Does the data match, has the address updated? etc.  Or is the student new to the school?  If so, add the student to the FileMaker database.
3. After reviewing the data entered on the web site, the user creates a new Order in the FileMaker and imports all the data.  If there is a problem with the transaction, it is flagged for follow up later.

PROPOSED METHOD

The first thing to do is figure out how to view the registration data from inside the database.  There are two basic ways to do this: import the web data into temporary tables or directly view the data using ESS.  I chose the second method.  The client is using a Drupal plugin called UberCart to manage the registrations.  This plugin is very popular for Drupal sites that sell things; this site is no exception.  Viewing MySQL data using ESS has been covered before but one think I needed to discover for myself was the data structure for UberCart.  This was done by trial and error, reading the documentation, and asking knowledgeable people.  Eventually I came up with this realtionship graph of the UberCart tables:

UberCart explanation;  The base table for the Order (or Invoice) is called uc_Order.  Each LineItem is in the uc_XXX table.  The table that has the products, i.e. the things that are sold (in this case classes) are in the uc_Products table.

When integrating with Drupal it is easy to get burned by placing your own fields in the native Drupal tables.  This is not a good practice.  In most situations, you don’t have much control over the Drupal site.  The administrator may not give you permission to create fields, and since plugins get updated all the time, they may rewrite the Drupal table and you would lose your custom fields.  If you need to create custom fields, do them in your FileMaker database.  To do this you may need to create a special table with a one-to-one relationship to the Drupal table.

In our case, I needed to create a table (Called zWebOrderLineItems, highlighted above)  to hold the status of each registration from the web site (the Status field), and to hold the FileMaker ID of the student taking the class (_kf_ContactID).  From the utility table we can make a relationship to the native FileMaker table holding the students in our database.

WORKFLOW

Once I had figured out the relationships between all the various tables in UberCart, then the rest was just FileMaker.  The workflow looked like this:

1. The user clicks on button on home page.
2. He/she is taken to a list view of the Web Orders that need to be processed.
3. The user clicks on the first order and confirms that the student is in the FileMaker database.
4. If the student is not in the database, the user clicks a button to add the student.
5. If the student is in the database, the data (address, DOB, cell phone, etc.) is confirmed as correct or updated.
6. The user clicks a button to create a new invoice.
7. Then, the user goes to next Order in list.

Here is an illustration of the user interface.  The left side is all info from the Drupal site.  The right side is the corresponding info for that student in the FileMaker database.

DISCUSSION

Creating an ESS link from the Drupal web site to the client’s back-end FileMaker database has certainly made their job easier.  Now students are registered for classes in near real-time.  Before the integration, registrations were done in batch mode the day after the registrations were entered on the web site.  Using a server side script, the main page in the database checks for new web orders and alerts the users within 5 minutes of record creation.  In addition, the registrations are entered into the database using two button clicks, instead of completely re-keying all of the information by hand.

Having a user review the data before it goes into the database is always a good idea.  Unlike using FileMaker as an interface, you don’t have the same control over data input in a web application.  For example, a user may put in their address as Rock Ln., while your business rule (or your direct mail service) may require addresses to be spelled out, i.e. Rock Lane.  In addition, user review will let you have the user search your database for existing fuzzy matches (I may be in the database as Jerome Salem, but registered for a class as Jerry Salem) where you wouldn’t want your database to be freely searchable on the web!  For these reasons, I would always recommend a “scrubber layer” between any external database and your in-house database.

Finally, this technique allows both systems to do what they do well, with minimal amount of interaction between the two.  In this case the data flow is one-way, from Drupal into the FileMaker database.  Next time we will look at the opposite, letting data flow from the back-end database into Drupal.

 

Problem

In this episode, I will present a technique to show data that is not evenly spaced. As a typical example we are going to look at pricing data. For our example, we are building a solution for a company that sells items. The items are sold at a particular price that is entered into the database. However, the price changes over time. Our task is to present the user with a graph of the longitudinal price changes. As you read through this canonical example, keep in mind we can use the same technique for any variable that changes over time. Here are some other examples where this would be useful: a student’s test scores, the number of orders a customer has placed, a patient’s blood pressure, or the number of times a product has been ordered. Basically any data that is related to the parent record and changes over time. Here is the data we are going to be working with. These are the dates when new pricing has been implemented over the past year for a product:

One might argue that showing the data in a portal like this is all you really need. In many cases I agree with the “less is more” motif. However, you can sometimes learn more from the graphed data than from just looking at the data in a table. First, let’s just graph the data using the new FileMaker Charts tool:

If you don’t look closely at the X (horizontal) axis, you may conclude from this graph that the price has gone up steadily over the past year. Here is the data presented with the correct X axis (I will call this “spaced data”):

From the spaced data, it is much easier to see a couple of trends that were not obvious from looking at the table data or the simple bar graph. First, you can see that there were way more price changes in the first half of the year than in the second half of the year. Second, since about May 2011 the price was relatively stable; the price only changed 3 times and stayed at about $60. Performing this type of analysis can be invaluable for the end users of the database. There are a couple of steps to do to present data in this way. It isn’t too hard once you have done it a couple of times. Method In order to present the data that is spaced properly (as in the “spaced data” graph) we need to be able to tell the database that the date values are actually part of a larger picture. To do that we need to build a “helper table” of date values. In the example file, the table is called Daily. It has a minimum number of fields: a serial number key field (probably not needed, but it is a good habit to have a key field in all tables), a date, a number field set to global storage, and a display calculation showing just the month name from the date field. For this table, you will need to do some maintenance to make sure the dates are always up to date. Typically I would run a server side script that creates new date record every day. What we are really going to plot along the X axis are these date records. The Y values will be the pricing data. Here is how everything is related: The Item table is related to the Pricing table using the “item_PRICING” table occurrence, this makes the portal shown at the beginning of this post. The X axis is the daily records that fall between a Max and Min date. Those dates are in the global date fields gStart and gEnd and are set in the background using a script trigger (although you could set them manually). This is represented by the item_DAILY table occurrence. Finally the Y axis data is that Pricing data that corresponds to the date record and the Item record we are on. To make that relationship we have a helper global field in the Daily table (gLink) which is also set via the script trigger. The Y data is represented by the item_Daily_PRICING table occurrence. Here is the detail for the item_Daily relationship. I am including this, since it usually trips me up too.

Here is the setup for the actual graph. There are a couple of interesting points here. The data in the horizontal axis is “filtered” so only the first day of each month shows a value. Depending on the date range you may want to just show every date. The Y axis is the data from the item_daily_PRICING table. The table object is on a layout linked to the Item TO. Note that we are graphing data from the related table ‘item_DAILY’.

For all this to work, three things need to be set on record load. In the current (Item) table, we need to set the date range for the X axis of the graph. This would probably be the Max and Min of the dates of the things you want to graph (in this case, the dates from the item_PRICING relationship). I set my Min date to the last day of the month before the minimum date, and the Max to the second day of the month after the maximum date. This is just so my fancy axis labels look nice. Also you need to set the ‘helper global’ in the Daily table to be the current item key value.

Once these are all set, the graph will look something like this:

This technique is a bit trickier than the sorting or setting a window title tips presented earlier on this forum, it is definitely worth the trouble. You can get the demo file to get more details on the X axis labels.

 

Run this code snippet to navigate to a list view containing a sub-summary to illustrate my point.

Goto layout (List view with Sub Summaries)
Sort
Goto Record[First]

When the user does this the first record is at the top of the list.  But the first sub summary is not visible, until the user moves the scroll bar.  This is especially evident if the user has a couple of screens full of records in the list.

the result SHOULD look like this

To solve this is not too hard.  actually it only takes two script steps.

Goto layout (List view with Sub Summaries)
Sort
Show Record[First]
Enter Preview Mode[]
Enter Browse Mode[]

 

I remember designing databases in the 1990′s that would let you do exactly that.  It was only for very forward thinking companies back then.  Shortly after that phones became way more complicated for this to be really practical.

Then came the iPhone and Filemaker GO. Once again, it is practical to dial a phone number directly from a database.

There are two script steps you can accomplish this: Dial Phone or Open URL.  Both script steps work the same as far as I can tell, but in either method the trick is that a phone number needs to be referenced as just digits.  You must first remove anything that isn’t a number. For example, a phone number field that has the contents 215-555-1234 will not dial correctly.   In FileMaker you can strip non-numeric characters out of the phone number using the Filter function.

Here is an example script using the Dial Phone and one using Open URL:

Attach a script like one of these examples to a button next to a phone number and you are on your way.  If anyone knows any functional difference between Dial Phone or Open URL and when one is more appropriate than the other please leave a comment or drop me a line.

 

You, or someone else, are staring at the computer screen looking at a record that has changed since the last time you looked at it and you mutter to yourself, “Who changed this record?  This is not the same as the last time I looked at it!”

An audit log is like looking at a record through time.  It will let you trace any changes that have been made on a piece of data in your database.  For example, in a contact management system an audit log will let you see address changes over time for a particular person.  You can use an audit log in an ordering system to track changes made to each invoice over time.  It is also an important tool for determining workflow processes. Remember the old adage, “who what when where why and how”.  Who changed this record? What data was changed? When was it changed? Why was it changed?and How?  An audit log can tell you all of these things (except probably the “Why”).

There are many tools around the web availible for building audit logs. Here is an technique that I used to use, and here is a  commercial product. My current favorite audit log tool is called UltraLog from Nightwing Enterprises and is the basis for the rest of this article.

This tool uses a single field to track changes.  In this field is a concatenated list that shows the changes that have been made to specified fields for the record.  Unlike other methods this technique doesn’t rely on a separate table to write the changes.  It is simple to use and implement, but there are some drawbacks with this method.  Since it writes the changes directly to the record, it can’t track deleted records.  This is an unfair criticism, though, as there are plenty of techniques to block record deletion, hide deleted records, or move deleted records to an archive table.  These are all fine ways to take care of this situation.  Keep your eyes on the reason for this technique to simply tell the story of the record.

But what good is an audit trail if you don’t know where it is?  Depending on the database and situation, there are two ways I will display the audit trail.  Either as a tab, or a pop up.

Almost every database I have seen has Last Modified fields in each table and displayed on layouts.   Why not use this piece of information, but make it more powerful? I do this by making the Modified Date label a button.  When you click that button it brings up the modification history of the record.  This is a subtle and intuitive touch that you can show users once and they will almost always remember that clicking on the Modification date displays the modification history.  The way I do this is also very simple.  I don’t bother with a script or even a layout for displaying the information.  Make the label a button and set it to bring up a custom dialog box.  Make the title of the box ‘Record History’ or something like that.  For the body of the dialog box, use the Modification History field.  Of course you could also use a script and a separate layout which will look much nicer, but take more time.  Here is an example:

If your database incorporates tabs then you have a better option.  Simply make a tab for modification changes.  This is more user friendly and also adds significant advantages.  For example you can do finds for records modified by a certain person, or records modified in a date range.

Here is another example showing how you can display an audit trail for an invoice with line items.  This tab shows the audit trail, along with a portal underneath that shows the invoices for the contact. The portal on the right shows the line items for the selected invoice.

SOMETHING EXTRA

I have also realized that using the UltraLog technique, it is possible to write to the Modification History field in your own scripts.  In the case illustrated (an invoice), when a line item is deleted from the invoice, it also writes to the Modification History field.  As here is the script fragment;

 

Compare relationships and querries

During the last episode we learned that Drupal is most powerful when the knowledgeable developer uses plug-ins to extend the abilities of the web site.  Some of the most useful plug-ins are CCK (adds the ability to create custom fields), Fivestar (adds a voting widget to your website) and Views (a powerful way to view data from different nodes).  We used the CCK to create a couple of new fields in the Things content type.

This episode we will show a basic technique to display information.  Lets say you want to see a list of content types.  To use FileMaker-speak, lets say you want to see a list of related records.

On our test web site, I have created a number of blog entries.  I would like to view just the three most current blog entries in a list on the home page.  If I was doing this in FileMaker I would create a relationship from the base table occurrence of the home layout to the blog table.  And then put a portal on the home layout showing the first three records. Right now, the web site has nothing on the right side.

Doing a similar task in Drupal is also a multi step process.  We will first need to use the Views plug in to create the relationship or portal.  Then to put it on the home screen, we will use the Blocks module to define the block and place it on the home page.  Here are the steps;

1. Create a view of just blog entries (remember, in our web site, we have two content types (i.e. tables), Stories and Things.  We are using Stories for our blog entries
2. Refine the view to just show the most recent records, and decide on what fields we are going to show
3. Create a Block to define the object (a link, or a list)
4. Put the Block on the screen using Menus

To navigate to the Views area, and create a new view; select Site Building –> Views –> List –> Add

As before, give the view a Human readable name, a computer readable name, a tag and define the View type.  The default type is Node; that is for just regular tables. If you add new data sources, they will show up on the list too.  This is similar to defining the context of the relationship. What type of data are you going to view?

From here you need to give the View a name, I called mine, ‘Recent_Stories’.  Lets do the most basic setup for the View.  It is a little backwards, but here is what you do; Define Fields then Filters, then go to the basic section for the formatting.  Once the Defaults are done, we can then define new types of displays:  a page and a block.  The workflow goes from column 3, to 4, then 2 and finally 1.

1. Add Fields (define what fields are in the View)
2. Filter for just Stories
3. Add a Display Type ( for this example a Page)

Now that the view is built, we have to put it somewhere.  We are going to add it to the right side of the standard web page.

Select –> Site building –> Blocks –> List  From the admin banner across the top.  Move the Recent_Stories view from the bottom section to the right side, at first I was surprised how slick it is to just drag something.  Then I remembered, this is only novel for a web page.  You could drag and drop things in computer programs since 1984.

Using these techniques you can also create views to show more specific information, such as posts with or without comments.  As your site grows you will certainly have many views. You can also pass parameters into the view for more power.  For example, pass the user name to only show posts that were written by the current user.  We will be seeing a lot more of views in future installments Crossing Drupal.

Here is a great shot of our progress, the block on the right shows the most recent three posts, I clicked on the first one to show it in the main part of the screen.

Today’s big lesson is that Views in Drupal  are similar to relationships in Filemaker.  A view also contains formatting and style information.  You can pass arguments to a view for more control over what the user sees.  The view can be placed on a page using Blocks, or as we will learn next time, using the Menus.

 

I’ve never tried the createRecord method, but the ability to send an array of field values in the method seems very interesting. How would you format this data in the $values array?
$rec =& $fm->createRecord(‘Form View’, $values);
The PHP API lacks good examples of each of the methods in the class, and this post brings attention to an alternate method to set data, though I’m not certain when it might be useful, aside from bypassing the large XML array in the $result. Might be worth a try to see the differences in performance.

So two worthy questions, for which I have answers!

First off, the $values array.

In the previous example, we had the following code snippet:

$fm &= new Filemaker('myDB', 'localhost', 'web', 'web123');
$rec = $fm->createRecord('web_layout');
$rec->setField('Name', 'Robin');
$result = $rec->commit();

You could create consecutive lines of setField calls for each field that you wanted to edit. But you could alternately add the fields as an array when you call the createRecord (or newAddCommand) functions.

Here’s how that would look (I’m going to use a few extra fields to illustrate).

$valuesArray = array('Name' => 'Robin', 'Year' => 2011, 'Week' => '6/5/2011')
$rec = $fm->createRecord('web_layout'; $valuesArray);
$result = $rec->commit();

FileMaker, in fact, will automatically treat each value as a separate array. When you’re hand-coding, you don’t have to do this unless you’re trying to write to a repeating field — let’s say I wanted to make an repeating field to store the haul of veggies I got from this week’s CSA. So your $valuesArray could in fact look like this:
$valuesArray = array('Name' => 'Robin', 'Year' => 2011, 'Week' => '6/5/2011', 'CSA' => array('Spinach', 'Lettuce', 'Radishes', 'Arugula', 'Salad Mix'));
(Just like when you’re in a calculation dialog, you want to be careful about closing off your nested arrays, hence the double-thesis at the end!)

PHP has many robust array manipulation functions, and the $_POST and $_GET variables when you submit a form are stored as arrays. With only a little manipulation and careful programming, you could theoretically just dump that $_POST data directly into your new record, a web form with hundreds of fields could process in just a half-dozen or so lines of PHP.

But as to the second question, what is the performance difference between these two commands?

You can’t really test this on single-record adds: You have to have a respectable amount of data that you can add. At first I thought 300 songs (or about the entire song catalog of The Beatles) would be a good benchmark to test on, but I kept bumping my head on the 30 second maximum execution time (I’ll see what I can do to beef that up in a future post). So I pared it down to just 150 records.

I whipped up a PHP script to move those songs (song, year or release, album, author(s), lead vocal(s), peak position in both UK and US charts, and notes. Thanks, Wikipedia!) from a .csv file into FileMaker. Once with createRecord() and a bunch of setField() commands, once with createRecord() with the values passed in as an array, once with newAddCommand() and a bunch of setField() commands, and once with newAddCommand() and a value array and a microtime timer to test how long it took each task to complete.

Obviously, 150 records in 30 seconds isn’t particularly stellar performance, but it is worth pointing out that I am probably not working on an optimal setup. Unsurprisingly, createRecord() appears to have an advantage over newAddCommand (because it’s a bit more “trim” as a function), but I admit I wasn’t expecting the value array to underperform compared with multiple setField() statements, which I thought would cause incremental execution time run-ups. But as neither the webserver nor the FileMaker server were dedicated solely to this task and may have had to stretch its resources at any time, I would caution against taking this table as the definitive guide to performance on any of these values, and I would urge you to test these methods out on your own and post your results in comments. (For information on how to put a timer on your scripts, check out the microtime examples at php.net).

By the way — did you know you can shave seconds off the processing time by keeping your web-facing layout in form view?

FMPHP add records benchmark sample files

 

There are two ways to create a record:

• Use the createRecord() method, specifying a layout name, and optionally specifying an array of field values. You can also set values individually in the new record object.The createRecord() method does not save the new record to the database. To save the record to the database, call the commit() method.
  For example:
  $rec =& $fm->createRecord('Form View', $values);
$result = $rec->commit();
• Use the Add command. Use the newAddCommand() method to create a FileMaker_Command_Add object, specifying the layout name and an array with the record data. To save the record to the database, call the execute() method.
  For example:
  $newAdd =& $fm->newAddCommand('Respondent', $respondent_data);
$result = $newAdd->execute();

Which is somewhat helpful, but many people wonder what the substantive difference is between these two statements.

Honestly, not a whole lot, but there is one important distinction between these two commands. The first indication of difference is in the way the two statements are “fired off.” The createRecord() statement is completed with a commit() call, whereas the newAddCommand uses an execute() call.

Let’s say you had the code

$fm = new Filemaker('myDB', 'localhost', 'web', 'web123');
$rec = $fm->createRecord('web_layout');
$rec->setField('Name', 'Robin');
$result = $rec->commit();


and the code

$fm = new Filemaker('myDB', 'localhost', 'web', 'web123');
$rec = $fm->newAddCommand('web_layout');
$rec->setField('Name', 'Robin');
$result = $rec->execute();


Both would basically perform the same task: They would add a record to the table on web_layout, set the “Name” field to “Robin”. In both instances, the record will not be created in FileMaker until the $result variable is set, at which point all the fields will be written.

The big difference is what comes back to $result: In createRecord(), result returns a simple integer value letting you know if the add was successful (e.g, 1 if the commit was successful). But for newAddCommand, the $result variable is an array containing a bunch of information about that new record, including the record ID. This means that you could follow that bit of code with this:

$newRecord = current($result->getRecords());
$recID = $newRecord->getRecordID();
$serial = $newRecord->getField('SerialNumber');
echo "<p>The record was added successfully: The new recordID is $recID and the auto-assigned serial number is $serial.</p>";

With createRecord(), there won’t be much of a handle for you to hold onto if you want to report back on the record that was created. Not the worst thing in the world, certainly, and if you don’t need it, then there no reason to clutter up your memory space with the massive array that the newAddCommand() will return.

If you’re curious to see the sort of information that is passed back to the $result, using a echo print_r($result); command after you execute the newAddCommand() statement will give you a good idea of what is being returned, including table structure information!

 

Example

I have a client that is a museum.  They host many events, such as tours for schools or camps, overnight visits for schools, and birthday parties.  All of these events are in a single table.  However each event type has its own layout with different business rules and different groups of fields that need to be filled out.

The client wants to see all of the events in a list view, and also be able to move easily between events in form view.

Each type of event has its own layout. As shown in this image:

Technique

Instead of building custom forward and back buttons and hiding the Status Toolbar, use a script trigger to decide what type of event layout to display.  In FileMaker version 10 and later we have new layout options.  Today we will take advantage of OnRecordLoad.  This trigger is a layout trigger, found on the Layout Setup dialog box in the third tab, Script Triggers.

Whenever a new record is selected, or made the active record this script fires.  Here is the script in our example file:

Using this new technique we can give regular users the ability to use native FileMaker navigation buttons. In the old days we created our own forward and back buttons and put them on each layout.  If a user happened to open the status area, then there was a good chance he or she would eventually land on the wrong layout for the event being viewed.  This could be a disaster.

There are a couple of things I especially like about this technique.

1) Any time I can use the native FileMaker tools I feel like I am giving my users more of the power that comes with using a FileMaker solution.
2) I don’t need to reinvent anything.  The status area already has a navigation tool and it works great.

I am posting an example file so you can get the full experience.  Try going to the form view and click from record to record using the Status Toolbar navigation tool. Enjoy!