UPDATED: FAIL: How Passage Technology's Storage Helper Torched My Relationship with Salesforce

UPDATE: So, after sending this link over the @asksalesforce support team, they re-referred it back to Salesforce customer support. I received another call from a Salesforce customer support person who took some more time to listen to the issue (apparently, he couldn't view the blog post because of security controls on their systems).

As he listened to parts of my story, he was surprised by a couple of points. First, he was surprised that I wasn't able to use Data Loader to download the deleted records on Friday, while they were still in the table. When I explained that they were gone now, he was surprised but noted that the sweeper utility may have just come through our org. According to him, I wouldn't have been able to simply restore the records by toggling the IsDeleted flag (but being able to download them while they were still in the table would have been useful - where were you on Friday?).

When I mentioned the "invalid cross reference id" error, he noted that the issue there was trying to re-import the data with an ID - that the system had to assign an ID. I told him that this would make it impossible to re-link the "Invoice Line" child records, but he said that it could be done with a bit of work in Excel. He then asked me to send those sandbox files that I needed to re-upload and offered to solve the import issue for me.

In all, I'm feeling a bit better about all this than I was yesterday at 6:30, but we'll see how it all unfolds. I'll keep you updated.

----- Origitnal Post ------

If you've been watching my Twitter feed, you've already seen parts of this story, but you may find more coherence (and a few more details) in this version.

A couple of years ago, following a mention at Dreamforce, I purchased a suite of software extensions for Salesforce from Passage Technology. As a Salesforce Admin, what drew me in was a product that they offered, Rollup Helper, that made it possible to build data roll-ups of a variety of data even across custom objects. To translate this into simpler language, a roll up field enables you to do something like total the value of all opportunities on an account. Salesforce has always had roll-up fields but the functionality was limited to records that had parent-child relationships (with the roll-up on the parent). Rollup Helper enables you to create roll-ups on records, even if there isn't a parent child relationship.

The pricing for Rollup Helper was about $2K for a one-year license, but they also offered two other software utilities, Lookup Helper and Storage Helper. If you bought them as a bundle, the other two software tools (each about $800), were included at a discount. Storage Helper seemed like it held some promise. Essentially, what it enables you to do is set up a filter condition and automate the deletion of records. Since the base data storage allocation for Salesforce.com is 1GB, depending upon your record usage, this could represent a savings for you versus paying for added storage.

For the most part, our data all fit easily within Salesforce data storage limits, with the exception of Orders and Invoices that were brought in through our Oracle back-end system integration. While the header record volume wasn't large, each of those records also included multiple lines and, as this data accumulated over the years, we hit our data limits. Initially, I manually deleted old records, but eventually wound up buying more storage space. In this environment, a software utility that we could configure to erase records that were over five years old seemed like a handy utility.

Some "In Hindsight" Data Points I Should Have Weighed More Heavily
I had mixed results with Passage Technology software, and I probably should have given more consideration to that when I ran Storage Helper on Friday. I actually created custom fields and set up Storage Helper back in October of 2016, but with our added storage, I didn't really need to run it, so the software sat, unused in my org for a couple of years. Meanwhile, I made a couple of separate attempts to use Lookup Helper, none of which were successful despite connecting with Passage Technology customer support both times.

I should have known...

In many ways, Storage Helper should be pretty simple software. The premise - set up a set of conditions to filter a search, then delete the results of the search. The thing that should make it cool is that you can set it to run on an automated script, so it will just work for you. Initially, I'd built a filter for both Orders and Invoices that looked at the year in the date record, then deleted things greater than 6 years old. Part of the reason why I'd waited to run the script was, after talking to some of my colleagues, I'd thought about possibly trying to calculate the age down to the month.

Anyway, background aside, I was getting ready to do an evaluation of Salesforce.com's Wave Analytics module. As part of the set-up for that, Salesforce also provided a trial license to a full Sandbox. But since we were going to evaluate the analytics, it seemed like it might be a good idea to remove some of the old order and invoice data, both to streamline the data migration and to simplify charting functionality. And so I ran Storage Helper on Invoices.

This was my first mistake.

Before I ran the software for the first time, I did previewed the data that would be deleted, something that's included in the software and recommended by the wizard. A quick sample of the data preview showed records from 2010 - so that looked good. It also triggered an error alert because the data set that it was getting to run on was greater that 50,000 records. I okayed that - figuring that it would require multiple runs of the software to get the job done.

After running the software, the interface offers a little dashboard that says, "Storage Helper saved you this much money and freed up this much data storage space". I was curious if it was deleting the child "Invoice Line" records and how much space that accounted for, so I went over to the Salesforce System Monitor and looked at the storage usage. Unfortunately, when stuff is in your recycle bin, it's still included in your storage count on System Monitor. I'd previewed the data (if you've ever looked at data in your recycle bin, there isn't a lot of helpful record detail info there - practically speaking, the only way to really know the details of a record is to undelete it), so I decided to go ahead and delete it. Now the system monitor showed a change and, for a moment, I was happy.

Back to Storage Helper, and I previewed the data again. Again, records from 2011, so I ran the script again. I should say that the script seemed to lag while it ran, but that's not unusual with some of these Salesforce processes. Because of the lag (thinking back, I think it lagged the first time it ran as well), I wasn't really sure whether it was working or not. Again, I deleted the records in the recycle bin. With the lag, I might even have pushed the "Run" button multiple times. Somewhere in there though, it finished it's second or third run, and I was curious how much data was left in these old records.

So I ran preview again. This time, I noticed an Invoice Owner who joined the company more recently. I was surprised -- how could this guy have owned a record from >6 years ago. I clicked through the preview to the record and, sure enough, the record was from last year. Last year? Then I looked at the Storage Helper script and the filter that I'd built -- it was gone. There was no filter.

I quickly went over to the recycle bin and "undeleted" all of the records in there. I also emailed the Storage Helper product manager from Passage Tech. This was before lunch. He emailed me about 10 minutes later to say that he was no longer the product manager for that product -- and forwarded my email to the current product manager and their customer support team. Meanwhile, I started to assess the damage.

A little after noon, I sent an follow up email to Passage Tech with these screenshots, the first being a Pivot Table report of Invoices from Live, the second being an extraction from my Sandbox that was refreshed last September:

That's an 84,948 record difference between live and seven month old data.

It goes without saying that, after that first contact with Passage Technology, I didn't hear back from them until Monday morning when I got an email from their customer support.

Salesforce Steps Up To Take Things From Bad to Worse
After not hearing anything back from Passage Technology (guessing that they were on East Coast Time and had probably gone home), I reached out to my Salesforce AE. Was there any way that Salesforce could help?

She suggested creating a case. Once there was a case, she said, they could escalate the case. And so I wrestled with the stupid subject tree to find a case and submit it. By 4:00pm, I finally had a case number. I forwarded the case number to her and she escalated it. Salesforce customer support reacted quickly, but the guy who called me talked with me as we both went to the recycle bin and looked to see if the records were there (they weren't - I'd already 'undeleted' what records were there when I realized what Storage Helper had done). Since there were no records there, he told me that Salesforce did offer a paid data recovery option, but I'd need to speak with my AE for pricing on that option. He also suggested that I consider the Idea Exchange (seriously).

So I emailed my AE again. She let me know that the data recovery thing started at $10K. Other than that, she reached out to some other Salesforce colleagues for suggestions. Queue the crickets.

Seeing that this wasn't going well, I attempted to restore some of the missing data by "Upserting" my old data from Sandbox. If you aren't familiar with the term, upserting essentially updates a record if it finds one, otherwise it inserts it. Unfortunately, this didn't work because all of the deleted records were still in the database table, they just had a field called "IsDeleted" flagged with true. Realizing this, I tried to fix it by updating all of the "IsDeleted" to false. Unfortunately, that didn't work because Salesforce won't let you modify the "IsDeleted" field.

So I searched for ways to update the IsDeleted field. I found a couple of suggestions. One suggestion was the Developer Workbench, an online tool that provides you access to a bunch of the underlying aspects of Salesforce. Using this tool, I was able to verify that the records were there. I could even look at the records individually. Thinking that this would be helpful for knowing exactly which records had been deleted, I tried to Bulk Export a .csv file of the deleted records. Unfortunately, bulk exports of this type of data weren't supported. I tried to work with specific records to change the IsDeleted flag, but that wouldn't work either. I even tried to generate a list of the deleted records so that I could copy and paste them into a spreadsheet, but that wouldn't work.

From an older post that I found online, I saw where somebody had written that some third-party data loader tools might enable you to modify IsDeleted. I tried rerunning an older version of LexiLoader that I had. It said that I needed an older version of Java to run it - so I installed that. Even then, while LexiLoader would run, it couldn't log in. By 6:30 on Friday, I'd exhausted all of the options that I could find. In another exchange with my AE, she said she was out of ideas and that we'd have to wait for technical support - on Monday.

At that point, I posted this on Twitter with what I expected to happen:

So here's what I expect to happen. Despite me trying to work with them on the issue immediately, @salesforce will do nothing and, by Monday my data will only be accessible through tape back-ups and the $10K recovery. #FAIL #sucks

— yaMarketingGoomer (@MarketingToMe) April 7, 2018

There are a couple of things worth noting at this point:

1. What I expected to happen is that at some point that night, Salesforce's system would process (and erase) the data that was flagged with IsDeleted. This would mean that data that was easily accessible and a situation that was potentially correctable with the right tools and/or permissions would soon be out of reach.
2. Once this data was deleted, depending upon how the Salesforce system works, it's possible that they might recycle the ID numbers of the records that were deleted. I don't know about this aspect of how Salesforce works and, if they do, how quickly those ID values are recycled. However, if an ID value was recycled, that would essentially mean that you couldn't just "re-upload" the data. And worse, any data that depended on being linked to that ID (like my Invoice Line child records) couldn't be re-uploaded.
3. Once this data was wiped from the data table, it would be significantly more difficult to determine which records had been erased and which ones had not, making a recovery and reset even more difficult.

Monday Arrives
As you can guess, I didn't hear anything back from Salesforce or Passage Technology until Monday morning. After I granted Passage Tech customer support login access, they came back with a grand toreador...

Unfortunately, we have no history to determine as to whether the filter existed, or was setup correctly to begin with since the job has not been successfully run in the past.

See, it's not their fault.

Meanwhile, I hadn't heard back from my Salesforce AE Monday morning, so I emailed her. I emailed her a second time after I used the Developer's Workbench tool and did indeed verify that all of the deleted records were now gone. My AE responded after 11:00am, noting that she'd been in a customer meeting. By lunchtime, she emailed me again with more scoping questions. I didn't hear back again until later in the day when I emailed a second follow up. What did seem clear from her afternoon email was that, for any data restoration being done, it would probably be on us to do it.

So yesterday evening I made a second attempt to upsert the older header records from my Sandbox instance using Data Loader. Errors again, only this time a different error message, "invalid cross reference id". It goes without saying that, it's not just a handful of records either.

At this point, it seems like the whole thing is just F@#ked!
Because of the inter-relationship of records, I don't think there's a tool (short of our integration orchestration), that can upload the data and the related. While we have the source data - we didn't loose that, it looks like there is no easy way to reload it back into Salesforce. And, essentially, all of the data is worthless because you can't easily tell what data is missing.

It goes without saying that I'm so pissed off at this point that it's even difficult for me to go back through and write about it. I'm trying really hard to keep to keep my vocabulary restrained and professional. What more can I say about Passage Technology and their software -- I think that they can expect that I won't be renewing our license for their software, but I don't expect much from them since we're really talking about, at most, a $3K per year customer.

But Salesforce? What can I say. We often talk about Salesforce.com being an unsupported platform -- when the rubber met the road, their customer support FAILED. They followed up the case on Friday night with ANOTHER email about the Idea Exchange. And, at 6:30pm on Friday when, if they had a resource that could extract or modify the IsDeleted field on two tables in our org, we could have recovered from this entire mess. Instead, they were done for the week. TGIF. Pick up the pieces next week.

At this point, I don't know how the rest of this will work out, but I do know one thing -- I don't expect to have good words about Salesforce.com going forward.

Yet Another Dreamforce FAIL

I've posted about this quite a bit on Twitter, but I felt it was time to put together a longer synopsis of the great FAIL that is Dreamforce registration.

First, background. I've been eight Dreamforce events over the years. There are upsides and downsides to the conference, but if there is one overarching theme for Dreamforce, it's oversized crowds overwhelming insufficient resources. And it's been that way at every Dreamforce I've been to. Sometimes it is simply an annoyance -- like long lines or the inability to get into a session. At other times, like at last year's Dreamforce Gala, it's kind of surprising that people haven't gotten hurt.

When you look at the history of Dreamforce, you see aspects where things evolve to address the crowds. Aspects like multiple locations for lunches and conference sessions reflect thoughtful efforts to address the crowds and adapt. Even the one year where they brought in the cruise ship as an additional hotel, while it may have been a failed experiment, was a step to addressing the oversubscribed hotel market.

Last year was the first year when I actually saw Salesforce change the conference web site to say that the conference was sold out. This happened a couple of weeks prior to the event (it's possible that it may have happened in year's past, but I didn't see it). Practically speaking, the conference was sold out long before that time, as the available hotel space was essentially gone about a day after registration was opened.

Salesforce has known about the sold out hotel problem for several years (thus, the cruise ship), but when you're using (and selling out) all of the available space, broadly, across the region. there are actual commodity limits that you face. And this brings me to the core strategy that Salesforce has employed in an effort to address this. Their strategy is essentially this:

"Surprise, registration is open!"

Last year, Salesforce introduced a tool that would email you as soon as registration opened. As it happened, I was at San Jose airport waiting to board an early morning flight when I received the email (6:05am), and I jumped through hoops to register before my flight boarded. I was able to book the Hotel Nikko (my preferred hotel) that morning. I also reached out to my colleagues, but they didn't register until that afternoon and the nearest good hotel they could find was the Hilton Financial District.

This year, we'd been planning to send a larger group to Dreamforce and help build some sales manager / power user / evangelists (over the years, one thing that we've found is that when our staff attends Dreamforce, it really opens their eyes to the much broader potential of enterprise software). So, the morning that I received the email announcing that Dreamforce registration had started, the first thing that I did was email my colleagues.

It's worth noting that this year's announcement and registration opening came on the Thursday, June 29th, right before what was for most people in the states, a four-day fourth of July weekend. Many people take the entire week of the fourth off and quite a few leave before that weekend arrives. Not only was this true for our staff, I later learned that our Salesforce AE was also out on vacation at that time. Again, remember the strategy:

"Surprise, registration is open!"

By the time I'd emailed my colleagues and gotten through the registration screen, I couldn't find the Hotel Nikko on the list. Part of the problem is that they way that the Dreamforce registration portal shows available hotels, it can be difficult to sort them. As I searched through the list, it wasn't just the Nikko that was unavailable, all of the Marriott properties in the convention center area were gone. I opted for the Hilton, knowing that some of my colleagues were Hilton loyalists (I used to be, but having "lost" hundreds of thousands of Hilton points multiple times, they're no longer a preferred choice).

While dealing with my own registration, I decided to text one of my colleagues to see if she'd seen my email. She replied via text, asking me to register her as she only had her phone and poor network access. One of the great failures of the Dreamforce registration process is that there is no way to register multiple attendees at the same time. This is also true for their room reservation system. That means that, if you want to make reservations for your entire team so that they can stay at the same hotel, you are hosed. By the time I'd finished my colleagues reservation, the Hilton was no longer an option and I had to put her in the Parc 55.

As the minutes ticked away, I realized that, if my colleagues that I'd emailed hadn't started the registration process on their own already, they were hosed. And if they were on vacation, forget about it. Since some of our colleagues were coming from the east coast, they needed hotel rooms -- commuting wasn't an option. In short, any plans that we might have had for a bigger, more useful Dreamforce -- hosed.

The Aftermath of Registration
As you can imagine, I was feeling pretty frustrated with Salesforce that morning. To have registered twice in succession and not be able to have our team in the same hotel? What's worse, it just kept eating at me, particularly as it became clear that my colleagues hadn't seen my email, hadn't registered, and were probably on vacation. I decided to draft an email to our Salesforce AE with a CC to ceo@salesforce.com. It was not a short email.

Of course, I didn't hear back from our AE until a week and half later, on Monday July 10. Like many of my colleagues, he was on vacation.

When it became clear that I wasn't going to get a quick response back from my AE, I turned to Twitter. I sent out my first tweet at 9:43 that morning. After numerous tweets that day, @Dreamforce began following my Twitter feed. I continued to post complaints about the process and my experience. On July 10th when our Salesforce AE replied, his only real response to the Dreamforce issue was, "Unfortunately, I have no control over the SF hotel market but I had clients last year that had success renting places on Airbnb or VRBO, so if you are still looking you could explore that as an option." 

I continued to post my issues about Dreamforce on and off again until, on Thursday July 27th, almost a month after registration opened, Salesforce's Twitter Customer Support team @asksalesforce replied to me asking if there was anything that they could do to help. They actually referred me to @Dreamforce, where they'd been following me the entire time (hats off to their team for actually trying to help).

But, in case you missed it -- or anyone at Salesforce is actually listening -- I'm still pissed off about the registration process. In this report card, you failed to meet expectations. To make it easy for you, here is a simple list of things that I think would make this better.

• Why can't Salesforce Account Execs work with customers to understand their expectations and intentions for Dreamforce? Your internal team knows when registration will open. If they are intend to bring a block of users, why not create a select number of carve-outs for pre-registration and hotel booking? 
• Why not let customers know when registration will open in advance of the date? That way, rather than being stuck with whatever situation you're in when registration opens, you can plan for it -- if it's important to you. Seriously, even concerts let you know in advance what day that tickets will go on sale.
• Why not allow people to register and reserve rooms for up to four people? Dreamforce is better when it's groups of colleagues attending at the same time. And frankly, the idea that each user gets stuck with whatever hotel (as opposed to housing in groups) is just ridiculous.
• The Promo code option in the registration on the first day is not helpful. I spent several minutes searching the Internet for a promo code to apply. If nothing else, allow people to apply one retroactively, since the housing block allocation is the problematic window.
• There should be something like a waiting list for hotel rooms such that, if a reservation opens up at a hotel, you draw from the waiting list pool first. That way more people could have access to their preferred locations.

Frankly, I suspect that it's likely that Salesforce does provide carve-outs for their top tier customers. I'm sure that their sales teams also work closely with them to ensure that their needs for Dreamforce are met. What that says to me -- along with the "that's a bummer man. I don't control the San Francisco hotel market, so I can't do anything for you" response from our Salesforce AE -- is that there is a layer of customer lip service going on here. As customers, we matter in that way of "customers matter" slogans, but not in us, the guys who've been on the platform for over 10 years now, but who don't have hundreds or thousands of seat licenses. At Dreamforce last year, they gave us this 10 year customer award.

At the time, I couldn't help but notice the phallic shape of the award. It was probably the funniest aspect of Dreamforce last year, as I couldn't help but feel like it doubled as a reminder for all of the times I felt like I'd been screwed by Salesforce. But hey, I'm sure it wasn't intentional. 

Just for the record, it's now been over two months since registration opened. I'm still pissed off. And I still haven't heard anything from Salesforce except for the aforementioned contacts.

No More United - Extracting Myself from The UA Ecosystem

Despite previously having frequently held status on United Airlines, over the past couple of years, I've completely quit flying the airline. United was an acceptable airline in the old days when you had status and they'd automatically put you in Economy Plus. However, once they stopped doing that and started putting me in the seats with less space in the back of the plane -- let's just say that I don't consider their standard economy seats to be a minimum viable product.

Unfortunately, I'm still somewhat rooted the United Airlines ecosystem. I have a couple hundred thousand Mileage Plus miles. I still haven't ever used miles to book a flight. What's more, I've been carrying a Chase United Explorer card for several years, but it's become increasingly useless for any of the ancillary benefits -- really, the "no international transaction fees" is the only practical thing it does for me.

Then recently while searching Hipmunk for flights for a family trip, I noticed the results that were showing up from United Airlines included this rollover caveat: "Basic Coach: No Overhead Carry-ons Allowed; Seats Assigned at Check-in; Last to Board; No Elite Qualifying Miles; No Changes or Refunds." I'd read something about United Airlines adding this "Basic Coach" ticket, but this was really the first time that I'd seen mention of it appearing in my flight searches. And when I saw it, it spoke to me like a vision that said, "You will never fly on United again."

In that moment, I realized that I needed to extract myself from the United Airlines ecosystem.
So the first problem is, what do I do with about 200K of United Miles that I have no interest in using to fly on the airline. I always used to spend miles for flight upgrades, but somewhere along the way, that became a pain in the ass, particularly once I quit flying United. So, what do I do? Do I subject friends or family to an experience on an airline that I don't believe meets a minimum threshold? Do I save the miles to use on some international code-share flight? Do I just burn the miles on some overpriced merchandise I don't need from the miles-for-products catalog? The reality is, if there was a simple answer, I would have probably cleared the bank out some time ago.

Next there is the Chase United Explorer card. While I haven't received those passes for the United Lounge for two or three years now (what's up with that?), if I do happen to use my United miles on a United flight, I think that the Chase card may provide some ancillary benefits. In that way, it's probably best not to cancel it until I resolve the mileage bank. And yet, every day that I hold that card instead of another card that could provide me with a more useful return is another wasted day in the United Airlines ecosystem.

And while all of this may seem crazy, it's weighing on my mind enough that last night I had a dream about trying to resolve the whole situation. I was at an airport trying to get help from the United customer service counter and they just kept ignoring me, talking among themselves. They reminded me of Salesforce employees at Dreamforce. I gotta get out of this crazy United ecosystem!

British Airways F'ed Up Online & Customer Support System

For the past 14 hours, I've been wrestling with the British Airways online system, trying to correct a problem that their system created. In the process of attempting to correct the seating assignment, I've learned a few things about their system -- and about why you might want to avoid booking flights with them. Put simply, their system is not designed for customer support or customer satisfaction and I've now been told repeatedly that, "if you have an issue in the 24 hours before your flight", your fucked. Harsh words, I know, but British Airways "customer service" people say that they can't do anything in that 24 hour window. If you had one person tell you that, you might attribute it "a bad apple" customer service rep. But when multiple customer service people tell you the same thing, then you know that this is a business strategy.

When you're checking in for a flight, there are many aspects and errors that can go wrong. It's especially challenging for international flights because you have an added layers of documentation and permissions, concerns about connections. Tensions are higher because the costs are higher -- your $1000+ flight is not equivalent to a $100+ domestic flight. You might expect that in this environment, there would be added pressure on an airline to do things right -- or to make things right when things don't go according to plan. This is what makes British Airways system unusual. Rather than being engineered to make things right, the system seems designed to generate customer frustration.

Here's the story of my experience with British Airways and some insight into the underlying mechanisms in their system that appear to be optimized for frustration.

My wife and I booked trips on British Airways about a month ago. My wife needed to go to Helsinki for a business trip and I decided to accompany her and take the time for tourism. Because of the different purposes of our trip, we had to book our tickets separately. Hers through her work travel system, and me selecting the corresponding flight directly from the airline. She opted for the American codeshare version of the BA flight because AA is one of their preferred corporate airlines and she already has a lot of American miles. I opted for British Airways because I'd already flown on them a couple of months earlier on a trip to Brussels.

Our frustrations started with navigating British Airways ticket pricing, which I already wrote about and you can find more about here.

So, one aspect of British Airways default ticket is that you can't select a seat until 24 hours before your flight. Should you choose to, you can pay something like $30 to select a seat in advance, but the default window is 24 hours before. We opted not to pay this as it would be an upcharge that my wife's work would be unwilling to reimburse -- and having one assigned seat without it's mate is kind of stupid.

In a discussion with customer service (either through her travel agent, AA, or BA), my wife was able to address the issue of us traveling together and get our two seats grouped. When we went to check in, it assigned her one seat and me the seat next to her. Unfortunately, when I went to create my online boarding pass, British Airways system counted it as a second check in and relocated my seat (and hers) to an entirely different location. The first one was better, the second one seemed to be triggered just a few hours before the departure, and those seats weren't very good. After spending some time on the phone with British Airways customer support, they assured as that everything was fine. But when we went to the airport service counter to get boarding passes -- you guessed it, they were messed up and wrong. Eventually the service counter people just manually moved us to the original seats that we had together -- but they acted rather pained to have had to do that. They also gave us the explanation that "they'd changed planes, so that probably messed up the process".

On the return flight, the leg from Helsinki to London is a codeshare operated by Finnair. As we ticked off the time, trying to get into the system right after it opened, we hit a couple of roadblocks as British Airways system seemed to struggle with the record exchange with Finnair. Eventually, we were able to both get checked in -- and change seats since the first leg of the flight was only about half full. I say change seats because, despite many empty rows, British Airways auto-seat assignment software assigned me a middle seat near the middle of the plane. It also assigned me a middle seat for the second leg of the trip, the transcontinental flight back to San Jose.

No matter how I tried, I couldn't change the BA seat assignment. I couldn't access it. And they stuck me in a middle seat. Despite trying to check in right near the start of the 24 hour window, British Airways stuck me in a middle seat.

So I reached out to British Airways and Finnair customer service on Twitter. The Finnair customer service team responded quickly, and when I provided them with my flight details, they told me that the couldn't access the British Airways seat assignment system. They couldn't change it. I would need to work with British Airways to change that seat. So I continued to reach out to British Airways -- that all started at about 11:00 in the morning.

My wife and I speculated that we might be able to access the BA flight once the 24 hour mark for that flight arrived. Meanwhile, I continued to try and reach out to British Airways. When the 24 hour mark before the second leg of the flight came and we still couldn't access our seat assignment, I attempted to call the airline. I'd tried calling earlier, but the local customer support number was only good for Monday through Friday, so the only number "for travel problems within the 24 hour period" was an international number. When I called that number, I waited on hold for 12 minutes before a woman from their customer service team took my info, then told me that she couldn't make any changes during the 24 hours before the flight. I told her about how my wife had tried to purchase an upgraded meal, but the transaction didn't seem like it would go all the way through. She said she couldn't fix that because it was within the 24 hours before the flight. In short, it was a 15 minute international call -- at my expense -- to learn that British Airways could not help me. They were unwilling or unable to make any changes within the 24 hour window.

So I continued on to post complaints on Twitter. Eventually, around 6pm local time, British Airways Twitter support team responded to me. After sending a direct message to them with my flight details, an hour or so later, they finally responded with this:

"as you're due to travel within 24 hours, we're unable to amend your seat. The airport have control of the flight and seating. You'll need to make any amendments to your seats at the airport. Please accept our apologies for any inconvenience caused."

Somewhere in all of this (perhaps on the telephone), it became clear that it would never have been possible for us to edit the second leg seat assignment. Whatever we were assigned, we were assigned and their "24 hour lock" on the system meant that, for your second leg seat, you were essentially hosed. Suddenly it made more sense why, on my return trip from Brussels, I was stuck in a middle seat in the back of the section despite aisle and window seats still open. And how, when I'd tried to change my seat online, their system wouldn't let me.

Keep in mind that my wife and I travel on airlines frequently. We often manage our seats using online the online interface after check-in has opened. In fact, my wife has sometimes made multiple changes. I mention this only to underscore a point -- while many airlines block out a wait time and only allow check-ins 24 hours before a flight, most flights will allow you to manage and change your seat up until the time you're at the airport getting on the plane. As long as there is space available. Unless you're attempting to change class -- where they'll upcharge you. So this limitation in the BA system is both unusual and bizarre. It's also a giant FU to their "customers", the people who are paying for seats on the plane.

This experience comes on a flight that is supposed to transport me into "bronze" status in British Airways frequent flier program. Another special British Airways surprise for me was that I fully expected to have already achieved status on my flight to Helsinki. However, surprisingly, the fare that we purchased for these tickets meant that the "points" value for my transcontinental flight on the carrier was only 20 points, what looks to be the minimum value. My previous one hour flight from London to Brussels was actually worth more "points". Yet another FU to their customers.

This is truly "Customer Lip Service".

While we can never be certain of what the future holds, despite my "frequent flier" status that I will have earned after this flight, I won't be rushing to rebook another flight on British Airways and, as I mentioned in my previous blog post about the business, I would advise anyone considering doing business with British Airways -- Caveat Emptor!

United's "Re-accommodate" and the "Customer" Lie in Airline Doublespeak

The Internet is abuzz with the story of the doctor on the United Airlines flight who was "asked" to give up his seat, then beaten and forcibly removed from the plane. If you haven't seen it, the Jimmy Kimmel show had a pretty good synopsis. As shocking as the video footage of the event is, perhaps what might be even more surprising was when United came out with the statement:

Flight 3411 from Chicago to Louisville was overbooked. After our team looked for volunteers, one customer refused to leave the aircraft voluntarily and law enforcement was asked to come to the gate. We apologize for the overbook situation.

I say shock, but I don't really mean shock, because in many respects while we're surprised that a business might say something like this, we're not surprised to see this kind of language and attitude from an airline. In that way, what I should probably say is that, while the footage and story is distressing, it's not really surprising.

The Grand Airline Lie: "Customer"
Most of us have expectations for "customer service". Inherent in that relationship is the principle that, as a customer, you have options and by choosing to spend your money with a vendor, part of your decision will be based on what you get for the service and how you are treated throughout the transactional experience. Or as you'd find on Wikipedia:

Customer service is the provision of service to customers before, during and after a purchase. The perception of success of such interactions is dependent on employees "who can adjust themselves to the personality of the guest". Customer service concerns the priority an organization assigns to customer service relative to components such as product innovation and pricing. In this sense, an organization that values good customer service may spend more money in training employees than the average organization or may proactively interview customers for feedback.

While you'll hear the airlines use the word "customer" a lot, for most travelers, there is little meaning beyond an exchange of money. Rather than being treated like customers, most airline passengers are treated more like cattle or sheep. We probably have more in common with the animals at factory farms than we do with the classic notion of a "customer".

And it's not just United Airlines. On CNN during the news cycle around the United story, the network ran a series of clips of people being thrown off the plane, all from different airlines. American, Delta, Spirit. They even lumped in a woman who was forced to buy pajama pants because JetBlue wouldn't let her on the plane wearing shorts that they didn't like.

Let's face it. They all suck. Some suck worse than others. People joke about it, but nothing gets better. Instead, things just deteriorate further.

So how did we get to this point? Here's a piece from Wired that does a good job of characterizing the problems with United. How United Turned the Friendly Skies into a Flying Hellscape is an interesting look at the recent history of the carrier and the impact of the merger with Continental. But in some sense, as noted, this issue transcends United.

The Inherent Tension of Flying
Traveling can be stressful. Some stresses are common -- we all struggle with time zone changes -- and some stresses are individual -- I may not be afraid of flying, but you might be. Time and schedules also ratchet up stress levels -- from concerns about making a flight or connecting flights to factors outside of the flight itself like business meetings, appointments or even vacation itineraries.

In the midst of this stressful environment for travelers, the airlines have been doing everything they can to ease traveler stresses (haha). Seriously though, rather than that, for the past 10 years or so, the airlines have increasingly pursued strategies of testing the limits of what passengers will tolerate, all with an eye towards increasing profitability. The Wired article talks about "Calculated Misery" and links to this New Yorker article, Why Airlines Want to Make You Suffer. The article talks about JetBlue being a hold-out on charging baggage fees and what drove them to change:

Wall Street analysts, however, accused JetBlue of being “overly brand-conscious and customer-focussed.”

In short, not only do airlines, as a business, not care about you, the "customer", they're business model is increasingly built on taking advantage of you. United Airlines profited to the tune of nearly $10 billion dollars over the past two years with that business strategy.

And the effect on "customers" is to put them on edge, forcing them into a combative, defensive mindset. It's not really difficult to see the equation, build systems that impose misery, tensions rise. Add in a good portion of unequal treatment -- like walking past the first class section and seeing spacious seating that is such a contrast to the cramped seating that you're being put in -- raises tensions higher. That's partly why we have more incidents of "air rage" taking place.

You Must Obey the Uniformed Flight Crew
Since we returned to flying following the 9-11 highjackings, we've all been chartered with paying special deference to the uniformed flight crew. It goes without saying that it's an essential rule for the safety and security of the flight. At the same time, there are flights that it seems like turn into something just short of the Stanford prison experiment. Whether it's a result of the "psychological effects of perceived power", or simply a flight attendant having a bad day, airline passengers have to be aware that something that starts as a simple customer service incident can easily escalate to an event that brings in law enforcement. After all, who could forget the guy in the "Princess Bride" shirt?

These incidents between passengers and airlines staff will keep coming up because that's the business model. To quote from the Wikipedia page on the Stanford prison experiment:

The experiment's results favor situational attribution of behavior over dispositional attribution (a result caused by internal characteristics). It seemed that the situation, rather than their individual personalities, caused the participants' behavior.

The "customer" environment created by the airlines, while endorsed by Wall Street, is toxic. These days, when you engage with an airline as a "customer", you have to devote an inordinate amount of energy to mitigating the stresses inherent in the experience. That's why "the friendly skies" is laughable. There is no joy in the base level experience. There isn't really much that an airline can do to "wow" you. And the hostile environment turns minor events into seeming random acts of cruelty.

Take the United Airlines incident as an example. When United decided to remove those people, was it clear who they chose and why? Were they flying on standby? Did they pick Dr. Dao because he's asian? Did he pay less for his ticket than other passengers? While news stories may relate different answers to this, that we can believe that it was a random act of cruelty speaks volumes to the toxic  customer environment.

But it isn't going to change. Because profits and "shareholders enthusiasm" for more profits. Despite the tremendous first day dip in United Airlines stock value, it's value is returning. And it's unlikely that you'll see any government regulation that makes it better. At least, anytime soon. So, until such time as there is something like governmental regulation that forces the airline industry to end their "calculated misery" approach, you need to recognize that when the airline industry uses the word "customer", they really mean something "transactional actor". Sure, they will do things that seem kind of like customer service -- like having people respond to your frustrated posts on Twitter -- but their ability to do much of anything beyond kind words is quite limited. In the end, their goal is not really to make you happy, it's to get you to stop squeaking.

TockTix and The French Laundry Reservation We Never Got

As you'll note from previous posts that I've written as well as the food blog that we don't really update much anymore, my wife and I enjoy food and dining. Leading up to her birthday this year, she had one simple idea for what she wanted -- for us to share a dinner at The French Laundry.

The French Laundry holds a special position of royalty in the food pantheon, one of those places we revere and, even as restaurant ratings change and prestige hops from one establishment to the next, it remains a benchmark for once-in-a-lifetime dining. We've never been there, but we've seen glimpses of the food in film and video, tasted approximations in dishes we made from the cookbook.

As one of the world's greatest restaurants, The French Laundry has always been difficult to get reservations at, and that's been true since well before the first time my wife and I went to Napa together in 2006 (before there was OpenTable and smart phones). And over the years, we've made a few casual attempts to get reservations, but never anything to concerted as the actual prospect of getting reservations, combined with a window of opportunity to go, seemed unlikely.

The Plan is Made
Once we decided that we would try to do The French Laundry for my wife's birthday dinner, I needed to prepare the support logistics. After an epic meal with wine pairings, nobody has any business driving, so you need to stay in close proximity to the restaurant, so I booked a hotel in Yountville. And, while the price for the hotel rooms seemed extremely high -- we even considered abandoning the Napa plan and heading for Hawaii -- we decided that it would be worth it. Additionally, since we were already going to be up there, it seemed worthwhile to book reservations at another restaurant on Friday night, the 28th. As we'd always wanted to try Redd (heard great things about it), that was the first place I checked -- and booked a reservation. Finally, my wife requested a couple's massage, so I tracked down a spa in Yountville for that -- and booked at the beginning of March. All that remained was to book our dinner at The French Laundry.

While I don't follow their site and I haven't ever successfully gotten a reservation there, I can say with some confidence that, for the longest time The French Laundry maintained a policy of taking reservations one month to the calendar date before the actual date -- as you'll note from this snapshot grabbed from the WayBack Machine.

This was the case until March 20 of this year. Why? Because while I planned and prepared to go on OpenTable and make a reservation at 12:01 on March 27 -- and I kept checking on The French Laundry website to verify that information; suddenly, on March 20. everything changed. On that day, preceded by one day's notice on their Twitter feed, The French Laundry rolled out a new ticketing system by Tock. And suddenly, reservations were open through June 30.

While this was probably a windfall for some seeking reservations, if you were like me and last visited The French Laundry site on say, March 19, you didn't know anything about it. And, if you were like me, you sat, thinking you were in the non-existent virtual line waiting for the reservation day opening to appear. Perhaps I should have called. Pestered the staff at the restaurant. Ah, hindsight perhaps. But it wasn't until Sunday evening, March 26, when I decided to visit the site again, thinking I would check on the Open Table portal, that I discovered the new ticketing system... and that all of the reservation opportunities were now gone.

Perhaps, I thought, I'm just missing something. Perhaps if I call the restaurant in the morning at 10:00am (as the web site used to say), I can speak with a human. Perhaps they can help me understand what's happening. Maybe they have some tables set aside for something special like my wife's birthday. And, having spent the night wracked with unpleasant French Laundry dreams, I got up, went to work, and called The French Laundry at 10:00am. Of course, the number doesn't do anything now except take you to an electronic message detailing the new reservation system.

#FAIL -- Suprises in Queueing are a Recipe for Customer Frustration
You don't have to be an expert in queue theory to know that people in a queue don't like surprises. Imaging if the window for taking food orders has a long line and then, suddenly, that window closes and another window opens up on the other side of the building. Without careful line management, the odds are unlikely that the people who were nearest the front of the line will find themselves near the front of the line at the new window. And there will be much frustration.

But it doesn't have to be that way. With careful queue management, a business can exert control over the line, manage the changes, and maintain customer satisfaction (like the clerk who comes to get the next person in line). It's not difficult for a business to do this, it's just that customer satisfaction and customer experience must be a consideration for the business.

And that's why, as I wrote on Twitter, it's surprising that The French Laundry would be so haphazard with their roll-out of this new ticketing system. Surely, The French Laundry, the restaurant that lands the plates of diners with such precisely timed synchronization, would be the pinnacle of customer awareness. Sadly, it looks as though all of that careful attention to detail and customer experience only happens in the dining room. It certainly didn't carry over to the roll-out of this ticketing system.

Consider the timeline as I can backdate it from news and communication reports.

• March 19, 2017 -- @_TFL_ announces the new ticketing system on their Twitter feed
• March 20, 2017 -- Local news outlets, SFGate and NBC Bay Area pick up the story
• March 23, 2017 -- Wine Spectator does a piece on the new ticketing system
• March 24, 2017 -- Food and Wine piece notes all reservations through June 30 are booked

Remember, as of March 19, The French Laundry web site still said you need to wait until the calendar day one month ahead. So, when did they decide on this system? It's hard for me to believe that they just found it and decided to activate it on March 19. Wouldn't it have been a bit more customer-friendly to provide some sort of transition -- say 30 days notice -- before rolling out the new ticketing system?

What's up with TockTix?
Fundamentally, the difference between the Open Table system and the TockTix system is that users must essentially purchase their spot at the restaurant -- their experience -- at the time of booking. The strategy behind this, from the restaurant's perspective, is to prevent no-shows.

When I first encountered the Tocktix system on Sunday night, March 26, I had a number of reactions. My initial reaction was, what happened to the old reservations screen? Initially, I thought that perhaps this was the result of the opening of a new window of bookings for April 27th, and I was excited to see a booking time/interface. Of course, that was not the case.

Here's an example (taken today) of the Tocktix interface.

Initially, I think I accidentally tried to make a reservation on March 27 because I actually thought it was just the opening of the calendar month window interface. What I didn't see was that this was further down the screen.

Instead, when I clicked the date, the calendar interface was gone. Same result, sort of, but up until this point, I thought I was making progress. So I clicked on "Add Me to the Wait List."

It was at this moment that I realized that things were unfolding badly for me. I went ahead and added my name to the wait list. At the same time, I was extremely frustrated that I couldn't add anything, no notes or comments, to clarify that this wasn't just any day, that it was my wife's birthday. Sure, I could add us to a wait list a week later, but that day wouldn't be anywhere nearly as important as the first day that I signed up for.

Since that time, I've also realized that there are other issues with the TockTix waiting line system.

1. I can't tell what position in line that I am.
2. I can't drop my listing from the wait list, like if say, we decided to cancel the entire trip to Yountville and go somewhere else.
3. I get no sense of movement or updates from the list. I don't even get an initial confirmation email confirming my place in line. What if all of the other "people" in front of me are ticketing bots?

Needless to say, I my frustration with the entire experience hasn't dissipated.

Prospective Customers and The French Laundry
As a business, The French Laundry doesn't have to care about potential customers in the queue in the same way that some businesses do. As long as they have limited seats and a consistent parade of butts in the seats, and as long as they deliver a premium, desirable experience to them, they don't really need to care about the people in line. If I get frustrated or you get frustrated, who cares. It won't stop the guy behind you from happily filling a seat, given the opportunity. And maybe you're so frustrated that you write a review on Yelp, but you haven't even been there, so your complaints are hollow. In short, even if you say, "they suck," it's pretty unlikely that you're voice will be heard over the din of oohs and ahs from people who have been there. In the end, it doesn't matter. The French Laundry brand will transcend your unhappiness. They don't need you.

For me, this whole experience diminished the brand. Ultimately, that won't matter to you, the next guy in line, or probably anyone considering going to the restaurant.

And so you're probably thinking that, if they called me today and said, we've got a table for you and your wife, I would change my tune. To that, I'd say, you're probably right about going -- we still haven't canceled all of our Yountville plans yet. And at the same time, The French Laundry brand will never hold the same elevated benchmark status for me as it once did. And I think it's unlikely that I ever will go there. I certainly won't pursue reservations or try to get into the line again.

REI and Online Customer Experience WTF in 3D

For many years, REI has been a business that held a special relationship for me. There were no Recreational Equipment Inc. stores in the south when I moved back out here from Tennessee way back in 1991. REI was a welcome friend to so many awesome aspects of the California experience -- hiking and camping gear, rain and snow gear, rock climbing shoes, and pretty much anything in between. It quickly became my go-to resource for all of my activities here, enabling me to go where the climate suits my clothes.

Among friends over the years, we've often found ourselves in discussions and debates about whether the REI of today is as good as the REI of yesteryear. From discussions with touring cyclists as to whether the REI rain gear was as good as the older Goretex gear from the previous generation of REI rain gear, to musings about the changing nature of the store as fashion seemed to be an increasing emphasis rather than gear.

Myself, I've been a member of REI since 1992 and over the years, I've bought a fair amount of stuff from them. REI has held such a significant role in my life that, often, if there was something that REI carried, I would buy it from them, even if it was slightly cheaper somewhere else. A lot of things figured into this calculation, such as the annual dividend as a discount, and REI's strong return policy. In the old days, there are stories of people returning gear after months and years. In looking at the specifics of their return policy now, that rock-solid guarantee has been reduced to something like a one year warranty. Problems with your REI gear? Good luck with that... Of course, I've never had to return anything I've bought from REI, but the belief that they have a strong return policy is probably a widespread belief. They even referenced REI returns in that Reese Witherspoon movie, Wild.

REI Delivers a Series of Failures
Recently, we'd been in the market for a jogging stroller. We just missed an opportunity to one of the top-rated ones from Target as they were closing out a previous model year. We actually could have probably gotten one, but we decided to visit our local REI store and check them out before we purchased. Suitably impressed, we ran around to various Target locations, but all of the discounted strollers were gone. After more research and study, we decided we would probably just go ahead and purchase the stroller from REI as soon as one of their periodic 20% off one full priced item sales took place. Sure enough, in the weeks before Thanksgiving, REI offered the 20% discount, so after waiting several weeks, we decided to move forward with our purchase.

The first surprise was that while REI used to carry the BOB Flex, they had stopped carrying that model and now only carried the Pro model, about $70 more expensive than the Flex. The Pro model includes a hand-brake, something many online reviews say is somewhat unnecessary. It also ways about 5 pounds more, something we really felt like we didn't need. After discovering this, we took a step back and reviewed all of the different options for jogging strollers. Ultimately, what we decided was that we would go with REI because of our long relationship with the brand, their strong return policy, and because my unused dividend benefit would provide some additional discount on the purchase.

On November 17th, I finally placed the order. We were anxious to pick up the stroller and would have loved to pick it up at one of REI's retail locations, but despite having four REIs in the area, the stroller was unavailable in all of them. They were gone from the Saratoga store where we'd previously looked at them. Our only choice was to order for in-store pick-up. But forget about Amazone Prime or other fast delivery services that you might expect, REI's initial estimate was that the unit would be available for pick-up on Sunday, November 27th.

My gut told me that this date was suspect. One week later -- typical ground shipping would have been Thanksgiviving Thursday, the 24th. Why they couldn't ship product to one of their locations in less than a week was a bit puzzling, but assuming that the timeline was tight -- REI had already published that they would be closed on Black Friday, so assuming a Saturday delivery, Sunday seemed possible, but unlikely.

So as the week or so of waiting went on, I decided to check with REI customer service, since I hadn't received any specific confirmation about pick-up times. Unfortunately, what I discovered during my chat with customer service was that REI had screwed up the order. They reprocessed it, and now my expected delivery date was scheduled for December 4th. As a courtesy, they sent me a $20 REI gift card.

Of course, while they were busy screwing up my order and being closed, Amazon featured the BOB strollers on a black Friday deal for less than we were paying REI (since they offered the Flex that we actually wanted and the Pro for the same discounted price that we got through REI). Instead, thanks to REI, we got to wait an extra week.

Still feeling frustrated with REI, I decided to contact their online chat Customer Support a second time on Friday, Dec. 2. After chatting with the customer service rep and relaying my frustration, his only response was,

REI Customer Service Rep: Occasionally an order may be delayed in our system, entirely our fault. But we would certainly hope our members and customers can understand that we are doing the best we can. When possible we do try to upgrade shipping or make things right.
REI Customer Service Rep: Occasionally these types of errors cannot be helped. Is there something particular regarding your order that I can help you with now?

In short, "bummer for you".

So Sunday, December 4th came, and we decided to head over to the REI store to pick up the stroller. My expectation was that there would be no stroller there since REI's order confirmation says something about expecting an email when the stroller finally arrives, but I thought it was worth a try since they'd given me that date.

What I learned in the store was that, no, the stroller wasn't there and that there was no way the stroller could have been there on the date promised since the store only received shipments on Monday, Wednesday and Friday. In short, REI's own system was making promises and writing checks that it's ass couldn't cash.

This isn't my first frustration with REI this year. Earlier, I went there trying to replace my Teva sandals, something that I'd been resisting doing for a long time. Imagine my surprise when my go-to gear store suddenly didn't have ANY Teva sandals. I wound up replacing my Teva's at Sports Basement.

REI has destroyed their brand loyalty with me
While I'm sure that I will continue to purchase things from REI -- it is one of the only places where some of this gear is available -- the business is no longer my preferred supplier. In a couple of moves, they have eroded all credibility with me. I wish that I could point to a specific reason or mark it as a trend, but I don't have anything like that. This is just one anecdote, but based on my recent experiences, I would advise Caveat Emptor.

Enterprise Car Rental: Customer Lip Service

As I wrote a couple of weeks ago, I was finally contacted by Enterprise Car Rental. Back at the beginning of June, I spent some time on the phone with Jose, who claimed to be a regional manager for Enterprise in the Miami region. Jose listened, he was apologetic, and he told me that he would take care of everything for me. He also asked for my home address, the implication being that he was going to send me something as a symbolic apology. I also asked for a complete postmortem in an email.

And so, I left it there, waiting to see what Enterprise would do, what their response was. And, as things happen, I was very busy over the next couple of weeks. But I didn't forget. A week or so ago, I even called Jose on the cell number that he provided me, and left a voice mail requesting follow-up.

Since we first spoke, however, I have not heard anything nor received anything. Enterprise still has not credited my card for the incorrect charges. In short, they haven't done anything.

Earlier this week, I tweeted multiple follow-ups to @enterprisecares. They told me that they would follow up with Miami. To date, I still haven't received any response.

As I mentioned on Twitter, Enterprise Car Rental has inspired me to add a new tag to my blog, Customer Lip Service. This is one of those when they say that they'll take care of you in an attempt to make you feel better, but don't actually follow up with any real action. Remember when Iomega customers actually had to sue the company (and win) in order to get support for their ZIP Drives?

If you're considering renting a car, you should seriously consider this customer experience horror story as a cautionary tale.