This week we are going to look at the integration of a Drupal web site and a FileMaker database.  We had a client with a specific need; an art school that offers classes to adults in their community.  They have a FileMaker back-end database that they use for registration and membership.  They also have a web site where people can view class offerings and enroll in classes.  The client would like to create a way to link these two independent databases.

There are two points where the two systems touch each other: presenting the classes offered for a given semester and the registration of students into specific classes.  The client and I decided to leave the first goal for the moment, and concentrate on the second goal of getting the registration data into FileMaker.

I am not working in a vacuum, of course, as there is already a workflow in place.  The accountant logs into their PayPal portal and does a screen print of the daily transactions.  This is handed off to a secretary who keys the information into the FileMaker database.

My suggestion to the client was to remove the excess printing and keystroke entry.  The workflow I suggested was:

1. A user (the secretary who used to key the data into FileMaker) clicks a button and is shown a list view of transactions from the web site.
2. The user then navigates to each record in turn and checks the student registering for the class.  Are they currently in the FileMaker database?  If they are, compare the existing FileMaker data with the data that the student entered on the site. Does the data match, has the address updated? etc.  Or is the student new to the school?  If so, add the student to the FileMaker database.
3. After reviewing the data entered on the web site, the user creates a new Order in the FileMaker and imports all the data.  If there is a problem with the transaction, it is flagged for follow up later.

PROPOSED METHOD

The first thing to do is figure out how to view the registration data from inside the database.  There are two basic ways to do this: import the web data into temporary tables or directly view the data using ESS.  I chose the second method.  The client is using a Drupal plugin called UberCart to manage the registrations.  This plugin is very popular for Drupal sites that sell things; this site is no exception.  Viewing MySQL data using ESS has been covered before but one think I needed to discover for myself was the data structure for UberCart.  This was done by trial and error, reading the documentation, and asking knowledgeable people.  Eventually I came up with this realtionship graph of the UberCart tables:

UberCart explanation;  The base table for the Order (or Invoice) is called uc_Order.  Each LineItem is in the uc_XXX table.  The table that has the products, i.e. the things that are sold (in this case classes) are in the uc_Products table.

When integrating with Drupal it is easy to get burned by placing your own fields in the native Drupal tables.  This is not a good practice.  In most situations, you don’t have much control over the Drupal site.  The administrator may not give you permission to create fields, and since plugins get updated all the time, they may rewrite the Drupal table and you would lose your custom fields.  If you need to create custom fields, do them in your FileMaker database.  To do this you may need to create a special table with a one-to-one relationship to the Drupal table.

In our case, I needed to create a table (Called zWebOrderLineItems, highlighted above)  to hold the status of each registration from the web site (the Status field), and to hold the FileMaker ID of the student taking the class (_kf_ContactID).  From the utility table we can make a relationship to the native FileMaker table holding the students in our database.

WORKFLOW

Once I had figured out the relationships between all the various tables in UberCart, then the rest was just FileMaker.  The workflow looked like this:

1. The user clicks on button on home page.
2. He/she is taken to a list view of the Web Orders that need to be processed.
3. The user clicks on the first order and confirms that the student is in the FileMaker database.
4. If the student is not in the database, the user clicks a button to add the student.
5. If the student is in the database, the data (address, DOB, cell phone, etc.) is confirmed as correct or updated.
6. The user clicks a button to create a new invoice.
7. Then, the user goes to next Order in list.

Here is an illustration of the user interface.  The left side is all info from the Drupal site.  The right side is the corresponding info for that student in the FileMaker database.

DISCUSSION

Creating an ESS link from the Drupal web site to the client’s back-end FileMaker database has certainly made their job easier.  Now students are registered for classes in near real-time.  Before the integration, registrations were done in batch mode the day after the registrations were entered on the web site.  Using a server side script, the main page in the database checks for new web orders and alerts the users within 5 minutes of record creation.  In addition, the registrations are entered into the database using two button clicks, instead of completely re-keying all of the information by hand.

Having a user review the data before it goes into the database is always a good idea.  Unlike using FileMaker as an interface, you don’t have the same control over data input in a web application.  For example, a user may put in their address as Rock Ln., while your business rule (or your direct mail service) may require addresses to be spelled out, i.e. Rock Lane.  In addition, user review will let you have the user search your database for existing fuzzy matches (I may be in the database as Jerome Salem, but registered for a class as Jerry Salem) where you wouldn’t want your database to be freely searchable on the web!  For these reasons, I would always recommend a “scrubber layer” between any external database and your in-house database.

Finally, this technique allows both systems to do what they do well, with minimal amount of interaction between the two.  In this case the data flow is one-way, from Drupal into the FileMaker database.  Next time we will look at the opposite, letting data flow from the back-end database into Drupal.

 

Problem

In this episode, I will present a technique to show data that is not evenly spaced. As a typical example we are going to look at pricing data. For our example, we are building a solution for a company that sells items. The items are sold at a particular price that is entered into the database. However, the price changes over time. Our task is to present the user with a graph of the longitudinal price changes. As you read through this canonical example, keep in mind we can use the same technique for any variable that changes over time. Here are some other examples where this would be useful: a student’s test scores, the number of orders a customer has placed, a patient’s blood pressure, or the number of times a product has been ordered. Basically any data that is related to the parent record and changes over time. Here is the data we are going to be working with. These are the dates when new pricing has been implemented over the past year for a product:

One might argue that showing the data in a portal like this is all you really need. In many cases I agree with the “less is more” motif. However, you can sometimes learn more from the graphed data than from just looking at the data in a table. First, let’s just graph the data using the new FileMaker Charts tool:

If you don’t look closely at the X (horizontal) axis, you may conclude from this graph that the price has gone up steadily over the past year. Here is the data presented with the correct X axis (I will call this “spaced data”):

From the spaced data, it is much easier to see a couple of trends that were not obvious from looking at the table data or the simple bar graph. First, you can see that there were way more price changes in the first half of the year than in the second half of the year. Second, since about May 2011 the price was relatively stable; the price only changed 3 times and stayed at about $60. Performing this type of analysis can be invaluable for the end users of the database. There are a couple of steps to do to present data in this way. It isn’t too hard once you have done it a couple of times. Method In order to present the data that is spaced properly (as in the “spaced data” graph) we need to be able to tell the database that the date values are actually part of a larger picture. To do that we need to build a “helper table” of date values. In the example file, the table is called Daily. It has a minimum number of fields: a serial number key field (probably not needed, but it is a good habit to have a key field in all tables), a date, a number field set to global storage, and a display calculation showing just the month name from the date field. For this table, you will need to do some maintenance to make sure the dates are always up to date. Typically I would run a server side script that creates new date record every day. What we are really going to plot along the X axis are these date records. The Y values will be the pricing data. Here is how everything is related: The Item table is related to the Pricing table using the “item_PRICING” table occurrence, this makes the portal shown at the beginning of this post. The X axis is the daily records that fall between a Max and Min date. Those dates are in the global date fields gStart and gEnd and are set in the background using a script trigger (although you could set them manually). This is represented by the item_DAILY table occurrence. Finally the Y axis data is that Pricing data that corresponds to the date record and the Item record we are on. To make that relationship we have a helper global field in the Daily table (gLink) which is also set via the script trigger. The Y data is represented by the item_Daily_PRICING table occurrence. Here is the detail for the item_Daily relationship. I am including this, since it usually trips me up too.

Here is the setup for the actual graph. There are a couple of interesting points here. The data in the horizontal axis is “filtered” so only the first day of each month shows a value. Depending on the date range you may want to just show every date. The Y axis is the data from the item_daily_PRICING table. The table object is on a layout linked to the Item TO. Note that we are graphing data from the related table ‘item_DAILY’.

For all this to work, three things need to be set on record load. In the current (Item) table, we need to set the date range for the X axis of the graph. This would probably be the Max and Min of the dates of the things you want to graph (in this case, the dates from the item_PRICING relationship). I set my Min date to the last day of the month before the minimum date, and the Max to the second day of the month after the maximum date. This is just so my fancy axis labels look nice. Also you need to set the ‘helper global’ in the Daily table to be the current item key value.

Once these are all set, the graph will look something like this:

This technique is a bit trickier than the sorting or setting a window title tips presented earlier on this forum, it is definitely worth the trouble. You can get the demo file to get more details on the X axis labels.

 

Also appearing on this special DevCon 2011 episode are our good friends Tim Neudecker from Kyo Logic, Lee Lukehart from SavvyData, and Kirk Bowman from MightData! Check it out here.

 

This year’s conference was held at the gorgeous San Diego Hilton Bayfront where we enjoyed temperatures in the 70s with low humidity and a pleasant breeze. The accommodations and facility was top notch, with plenty of space for the conference and all of the normal hotel patrons.

For me, it’s really about being together. This is an amazing and supportive community of people who share common interests, face similar hurdles, and are working toward many of the same goals. While the FileMaker online community is vibrant, nothing beats face time, and DevCon offers plenty of opportunities to hang out and chat with 1,000+ of your closest FileMaker friends! Here are some of my highlights from the 2011 FileMaker Developer Conference.

Performance

The three sessions that had the most impact for me this year all had to do with performance. FileMaker, Inc.’s Jon Thatcher kicked off the first slot on Wednesday, the first official conference day, with an “under the hood” session focused on WAN performance. He explored some of the behind-the-scenes changes that they are working on to improve WAN performance in future versions and explained the ins and outs of the FileMaker Server logs. Understanding the logs is the first step in analyzing and ultimately correcting performance issues.

On Thursday Mark Richman from Skeleton Key in St. Louis, MO presented “Practical Techniques for Improving FileMaker WAN Performance”. Mark shared some of the most “expensive” features in FileMaker (from a performance perspective) and how to work around them, or simply avoid them when deploying over a wide area network.

Jason Young from Terapico Data in Boulder, CO expanded on last year’s more general “What’s Faster” session with a focus this year on “What’s Faster for FileMaker Go”. Jason provided some comparative speed metrics when running FileMaker processes in FileMaker Go on an iPad locally, on a WLAN, and over 3G. The results in relation to each other were as expected with performance being worse over 3G than on a WLAN, and both being dramatically slower than a locally run process. Jason also shared a case study where performance was a major hurdle for  a FileMaker Go deployment. He explained what techniques they used to overcome performance issues by simply rethinking the way they approached certain features.

Some common themes emerged across all three of these sessions. Primarily, with more and more movement toward WAN deployments, FileMaker developers need to rethink some common practices based on an understanding of what causes performance to drag. For example, all three presenters talked about the increased use of one-to-one relationships to help keep tables thin, push fields that contain a lot of data out to related tables, and ultimately reduce the amount of data that needs to be pulled down from the server when records load.

This topic is very important for the development community right now and it’s great to see so much attention being placed on it. Thanks to Jon, Mark, and Jason for their hard work and for bringing real solutions for dealing with performance issues to the community.

FileMaker Feud

The most fun I had at a session this year was FileMaker Feud, presented by our friends at MightyData in Dallas, TX. This was one of several vendor sessions that gave vendors an opportunity to showcase their products and services. MightyData had some fun with their slot by recruiting eight contestants and two of their FileMaker coaches, Martha Zink and Anders Monsen, who competed as “families” guessing the most popular answers to many FileMaker related questions. The answers came from a survey sent out to the greater FileMaker community months in advance. Based on a tournament format, the winning team was then split up for a second round, and then those winners went head-to-head for the final rounds. In the end Laura Betz from Cross IT beat Lee Lukehart, SaavyData, in a suspense-filled final round that could have been the basis for a fine come-from-behind, underdog film! All contestants received great gifts, with the grand prize being a brand new iPad 2!

Next Year

The conference ended with the announcement that the 2012 FileMaker Developer Conference will be held in Miami, on July 16-19. If you are part of the FileMaker community but haven’t yet made it to a DevCon, do yourself a favor and try to get out to Miami next summer. You won’t regret it!

 

One hallmark of DevCon is the lack of available sleep time.  This year was no exception with the time zone change (for us east-coasters) and the number of sessions and informal gatherings going on this year.

The sessions I chose to go to on the first day covered a wide variety of FileMaker use cases.  The day started with a session on WAN performance by Jon Thatcher.  After a primer on the differences between data transfer on Local versus Wide Area Networks, we learned how to use the Stats page of the Filemaker Server admin tool to diagnose problems and improve serving data.  Throughout the talk, Jon gave design tips that would improve overall performance of a solution.

The main sessions of the day covered the relationship graph and script triggers.  While both of these topics are very different, the interesting conclusion of the day was the same from both sessions.  All of the action should be in your scripts.  If your business logic lives in scripts, then it is much easier to modify, document, and even move it to other sections of a solution.  There are also many performance benefits you can get doing finds and sorts at run time in scripts versus relying on the relationship graph for these things.

But one of the best parts of DevCon was seeing all of the people I have spoken to on the net, or in real life, and catching up with old friends. Also, we had several members of the Philly FileMaker User Group attend the conference this year (pictured above). It was great to see everyone in San Diego!

 

Our good friend, Don Levan, Vanguard Custom Software, is hosting the next Craft of FileMaker Design Studio on 

Register before the end of the FileMaker Developer’s conference and save $300.

Attendees of The Design Studio will learn a formal process for designing exceptional FileMaker solutions:

• Conduct design research
• Derive requirements from personas and scenarios
• Identify and use design patterns
• Develop and refine ideas
• Prototype and  test designs
• Utilize feedback to improve solutions

 Eight FileMaker Inc. staff have attended the Design Studio (formerly known as the “Fundamentals of Exceptional Interface Design Workshop”). Jesse Barnum, President of 360Works, had this to say of the workshop, “This workshop really helped us in the design of the document check-in screen of DocuBin, our newest product. After implementing what we learned, we simplified, re-worded, and rearranged the buttons so that the result of each choice is much more obvious. It’s little touches like this that make any software appealing, both to look at and to use.”

What is design?

Design is an intentional process in which you elicit, derive, decide, clarify, and plan the solution to a given problem. Sketches, prototypes, technical plans, and finished solutions are the artifacts that result from a design process.

Design is also a key differentiator. A minimum level of design used to be enough to demonstrate professionalism, craft, and care. The overwhelming success of IOS, the growing acceptance of web-standards, and the sophistication of the user base, has significantly raised what is considered acceptable. Indeed, applications and websites that look amateurish or have poor interaction design are often viewed with contempt and distrust.

How can learning a formal design process help me?

It is almost impossible to hit the right solution the first time. Most people and businesses don’t know exactly what they want or need. As a result, a lot of time, money, and effort is spent undoing the initial attempts or working around ingenious solutions to the wrong problem. An effective design process saves money by identifying the right problems to solve and by developing effective solutions early when it is most cost effective.

Additionally, a repeatable design process makes it possible to have confidence in your estimates and projected timelines, and makes it significantly easier to sell better projects.

Who is teaching this course?

Dr. Don Levan, President of Vanguard Custom Software and the Founder of the Craft of FileMaker is a Licensed Clinical Psychologist, FileMaker 7, 8, 9, 10, and 11 Certified Developer, and former photo journalist. Dr. Levan was a speaker at the 2010 FileMaker Developer’s Conference, will be speaking again at the 2011 FileMaker Developer’s Conference, and is an accomplished presenter and teacher. He has taught courses and seminars on interaction design, programming skills, and psychology. Dr. Levan founded Vanguard Custom Software after creating a series of practice management systems for the mental health and medical clinics where he was treating as a Clinical Psychologist.

What are past attendees saying?

“This course is a must-have for all serious developers.”

— Paul Spafford, FM Layoutmode.com

“This workshop changed the way I work forever!”

— Marty Thomasson, Gearbox Solutions

“Absolutely fantastic material, professionally delivered.”

— James Hea, Project Manager, FileMaker Systems at HSBC Bank Bermuda Limited

“I would wholeheartedly recommend this workshop to any serious FileMaker developer trying to bring the best user experience possible to his or her clients.”

— Richard Carlton, President of Carlton Consulting, Platinum member of the FileMaker Business Alliance

“This workshop provided clear and tangible examples and exercises that engaged us to rethink how we approach requirements gathering and dovetail that into intentional system design.”

— Adam Aronson, President of FullCity Consulting, Platinum member of the FileMaker Business Alliance

“The workshop was fun and stimulating and will raise the game of those who care about designing databases at the highest level… I learned a lot in a short time.”

— Doug Gardner, President of Exoteric, Six-time speaker at the FileMaker Developer’s Conference

“It’s a ‘must do’ for FileMaker developers who do not have a background in formal software development. It will save them and their clients much time and aggravation if they apply the process presented in the workshop. Otherwise, they will repeatedly re-invent the common mistakes inherent in a free-form design development process.”

— Dale Stuart, Ph.D., Sc.D., President, CEO, StarQuick Solutions, Inc., Makers of TheraQuick

What does the workshop cost?
Registration for the class is $1,499 US. Register by the end of the 2011 FileMaker Developer’s Conference and save $300 discount. To receive the discount, enter the discount code “DevCon” on the registration form.

Where can I get more information about the Design Studio?

Dr. Levan will be presenting “The Fundamentals of Exceptional Interface Design” at the 2011 FileMaker Developer’s Conference (Thursday, August 4th, 9AM). This session will provide a high level overview of the process and practices covered in the Design Studio.

Paul Spafford, of Spafford Data Solutions, has written a review of his experience attending the the first Design Studio on his blog, FMLayoutmode.com. You can read his review here.

Matt Navarre, of MSN Media, interviewed Don Levan on the FM Talk Podcast. You can listen to that interview here.

How can I save $300

Registration for the Design Studio is $1,499. Register by August 13th and save $300. To receive the discount, enter the Discount Code “DevCon” on the registration form.

What else do attendees need to know?

The Studio is being held at the 825 3rd Avenue (50th Street) in the heart of New York City.

The Studio will run from 9 a.m. – 5 p.m. each day. Participants are encouraged to bring their laptop computers. Paper, sharpies, sticky notes, and six-up templates will be provided. Participants will receive a recommended reading list in advance of the Studio.

Participants are responsible for their own meals.

Relevant Links

Craft of FileMaker , http://fmcraft.com/
Vanguard Custom Software, http://vanguardcs.net/

The Craft of FileMaker is a service of Vanguard Custom Software. Vanguard’s mission is to “design simple solutions to complex problems.” They incorporate software development best practices, interaction design, and cognitive psychology to create data-driven applications and websites that are efficient for business and easy for people.

 

Run this code snippet to navigate to a list view containing a sub-summary to illustrate my point.

Goto layout (List view with Sub Summaries)
Sort
Goto Record[First]

When the user does this the first record is at the top of the list.  But the first sub summary is not visible, until the user moves the scroll bar.  This is especially evident if the user has a couple of screens full of records in the list.

the result SHOULD look like this

To solve this is not too hard.  actually it only takes two script steps.

Goto layout (List view with Sub Summaries)
Sort
Show Record[First]
Enter Preview Mode[]
Enter Browse Mode[]

 

I remember designing databases in the 1990′s that would let you do exactly that.  It was only for very forward thinking companies back then.  Shortly after that phones became way more complicated for this to be really practical.

Then came the iPhone and Filemaker GO. Once again, it is practical to dial a phone number directly from a database.

There are two script steps you can accomplish this: Dial Phone or Open URL.  Both script steps work the same as far as I can tell, but in either method the trick is that a phone number needs to be referenced as just digits.  You must first remove anything that isn’t a number. For example, a phone number field that has the contents 215-555-1234 will not dial correctly.   In FileMaker you can strip non-numeric characters out of the phone number using the Filter function.

Here is an example script using the Dial Phone and one using Open URL:

Attach a script like one of these examples to a button next to a phone number and you are on your way.  If anyone knows any functional difference between Dial Phone or Open URL and when one is more appropriate than the other please leave a comment or drop me a line.

 

You, or someone else, are staring at the computer screen looking at a record that has changed since the last time you looked at it and you mutter to yourself, “Who changed this record?  This is not the same as the last time I looked at it!”

An audit log is like looking at a record through time.  It will let you trace any changes that have been made on a piece of data in your database.  For example, in a contact management system an audit log will let you see address changes over time for a particular person.  You can use an audit log in an ordering system to track changes made to each invoice over time.  It is also an important tool for determining workflow processes. Remember the old adage, “who what when where why and how”.  Who changed this record? What data was changed? When was it changed? Why was it changed?and How?  An audit log can tell you all of these things (except probably the “Why”).

There are many tools around the web availible for building audit logs. Here is an technique that I used to use, and here is a  commercial product. My current favorite audit log tool is called UltraLog from Nightwing Enterprises and is the basis for the rest of this article.

This tool uses a single field to track changes.  In this field is a concatenated list that shows the changes that have been made to specified fields for the record.  Unlike other methods this technique doesn’t rely on a separate table to write the changes.  It is simple to use and implement, but there are some drawbacks with this method.  Since it writes the changes directly to the record, it can’t track deleted records.  This is an unfair criticism, though, as there are plenty of techniques to block record deletion, hide deleted records, or move deleted records to an archive table.  These are all fine ways to take care of this situation.  Keep your eyes on the reason for this technique to simply tell the story of the record.

But what good is an audit trail if you don’t know where it is?  Depending on the database and situation, there are two ways I will display the audit trail.  Either as a tab, or a pop up.

Almost every database I have seen has Last Modified fields in each table and displayed on layouts.   Why not use this piece of information, but make it more powerful? I do this by making the Modified Date label a button.  When you click that button it brings up the modification history of the record.  This is a subtle and intuitive touch that you can show users once and they will almost always remember that clicking on the Modification date displays the modification history.  The way I do this is also very simple.  I don’t bother with a script or even a layout for displaying the information.  Make the label a button and set it to bring up a custom dialog box.  Make the title of the box ‘Record History’ or something like that.  For the body of the dialog box, use the Modification History field.  Of course you could also use a script and a separate layout which will look much nicer, but take more time.  Here is an example:

If your database incorporates tabs then you have a better option.  Simply make a tab for modification changes.  This is more user friendly and also adds significant advantages.  For example you can do finds for records modified by a certain person, or records modified in a date range.

Here is another example showing how you can display an audit trail for an invoice with line items.  This tab shows the audit trail, along with a portal underneath that shows the invoices for the contact. The portal on the right shows the line items for the selected invoice.

SOMETHING EXTRA

I have also realized that using the UltraLog technique, it is possible to write to the Modification History field in your own scripts.  In the case illustrated (an invoice), when a line item is deleted from the invoice, it also writes to the Modification History field.  As here is the script fragment;

 

Whether you want to be a spectator or an actual contestant is your choice, but MightyData (who is sponsoring the event) would love for your answers to be part of the survey.

You can check out all the details at MightyData’s FileMaker Feud.