User Testing of Shipment Tracking with Estimated Delivery Date
User Testing Status Key
- PASS
- BARELY PASS
- INDETERMINATE
- FAIL
Statuses key of user testing questions:
- PASS – Validated by most or all participants
- BARELY PASS – Validated by at least 5 participants, with a tiebreaker of one and a group size under 10 this pass 1s questionable or barely passing validation.
- INDETERMINATE – Not enough information, Agent uncertainty or a mix of answers given that cannot be quantified
- FAIL – Validation failure by most or all participants
User Testing Questions & Synthesis
QUESTION
SYNTHESIS
STATUS
How do you know that a package is delayed with the systems you use today? For instance, in the Chewy shipment tracking drawer, shipper dashboard, Fed Ex or other carrier system?
- 5 participants use FedEx tracking
- 1 participant uses the tracking drawer
- 3 participants use all three tools (FedEx Tracking, our CRM Tracking Drawer and the Oracle System Shipper Dashboard).
FAIL
This question is marked as fail due to only one CSR using our shipment tracking drawer exclusively, where as 3 CSRs use a combination of systems
When a customer asks why the package is late with carrier, where do you look for an explanation?
- 4 participants use FedEx tracking
- 3 participants use the Shipper Dashboard and then FedEx
- 1 participant uses the Shipper Dashboard
- 1 participant looks at the warehouse the package is coming from
FAIL
CS agents do not trust the shipment tracking drawer to provide accurate info. They are currently using Google, FedEx tracking, other carrier systems (if applicable) and our Shipper Dashboard inside of Oracle.
The following question is in reference to the tracking tools you use today. Do you feel you currently have enough information about shipment delay to explain it to a customer?
- 7 participants stated yes
- I participant stated no
- 1 participant was undecided stating there is enough information 50% of the time
PASS
Agents felt they mostly have accurate information. However, sometimes FedEx does report inaccurately.
How are you using the travel history in the Tracking Drawer today?
-
4 participants do not use the travel history
-
5 participants do use the travel history
BARELY PASS
This is pass but barely. Agents do not trust the shipment tracking drawer for the most up to date and accurate travel history.
What does the term “Scheduled Delivery Date” mean to you?
- 4 agents understand scheduled meaning “estimated to arrive”
- 1 agent stated “supposed to arrive”
- 2 agent stated “guaranteed by carrier”
- 1 agent stated “expected to arrive”
- 1 agent stated “CST scheduled package arrival”
FAIL
It is very apparent we use a lot of different terminology inside of our tools. We use original, scheduled, estimated promised and actual.
See appendix A at the end of this document
What does the term “Promised Delivery Date” mean to you?
- 8 out of 9 understand and describe promised as guaranteed day, specific day or latest day.
-
1 agent relayed he/ she was unsure what promised delivery means and how this is communicated
PASS
Although this is a strong pass, guaranteed, specific day, latest shouldn’t use the term “Promised” we try but can’t always assure a package will be there within a promised time frame. We should revisit our vernacular by surveying our agents to see what terminology makes the most sense to them. Design proposes using only 2 dates “Original” and “Estimated”
See appendix A at the end of this document
Can you tell the difference between the grey timestamp (time of the scan) and the scheduled/ estimated delivery date in red?
- All participants stated that they understood the difference between the time of the scan and estimated delivery date change in red
PASS
Although this is a pass the two dates side-by-side were confusing and overwhelming to most of our agents. Agents understand there’s a difference but it doesn’t mean the two columns is the right solution
Where do you currently go to see indication of date changes that happened on the day of a delivery exception or event? (Reference grey column).
- 3 participants refer to FedEx
- 3 participants refer to the Shipper Dashboard
- 3 participants refer to both FedEx and the Shipper Dashboard
- 1 participant simply stated tracking info
INDETERMINATE
The confusion around this question seemed to be that agents were looking for 1:1 information between our tools, how that information is shown and what they were seeing in the prototype.
Do you feel that the original scan and the new scheduled delivery date are both needed in the shipment tracking drawer?
- 7 out of 9 agents said yes and, it wouldn’t hurt and feels the info is helpful
- 2 agents stated no, the scan is not needed while one of those said it might be helpful to others
PASS
The new scheduled delivery date and how the date changes was seen as secondary information to the agents we tested with. They liked having it or thought it would be helpful to new agents but didn’t feel like experienced agents would need the info as much or reference it as often.
Are you aware that Chewy promises a delivery date at shipment scan?
- 5 out of 9 say yes, 2 of these say they know of a date but were not sure of a promised date
- 1 participant said no
FAIL
We should revisit our vernacular by surveying our agents to see what terminology makes sense to them. Design proposes using only two dates “Original” and “Estimated”
See appendix A at the end of this document
Is it relevant to you to know how many times the promised date changes?
- 7 out 0f 9 agents say yes its needed, helpful and especially helpful for escalated orders
- 2 say no, while one agent specifically stated it isn’t really relevant to her
PASS
The promised or expected date changes would be helpful, especially for escalated orders, returns, refunds or concessions. It is helpful to have all information to relay to a Customer but Customer Service Agents felt it’s not always needed at first glance.
What is more important to you, how may times the date changed or how many days late the package is?
- 8 out of 9 agents stated the amount of days the package is late would be more important than how many times the date had changed
- 1 participant said both the date changes and days late are equally important
PASS
Our agents overwhelmingly felt that if they had to choose between how many times the delivery date changed and how many days late the package is, our Agents spoke to how many days late being what our Customers would resonate with and care about the most.
Do you need to know which warehouse the package is coming from? If yes, why? List all of the use cases.
All participants said yes, this is helpful to know the approximate time frame and how fast the package will arrive. The facility helps them to see the bigger picture, this is often the first thing they look for.
PASS
Something unexpected that came out of our user testing was that Agents feel the facility information is valuable to helping them to solve shipment delay and tracking issues that arise for the Customer. The majority spoke to the lettering and codes we use internally. The desire is that if we continue to use our abbreviated facility names we should add the city and state information to the data we display in our internal tools. Most agents suggested getting rid of the acronyms entirely.
How do you feel about the alternate experience of the popup modal?
7 out of 9 agents preferred the popup experience
2 agents preferred the side-by-side estimated delivery date experience because the information would be helpful to have in all use cases.
FAIL
This is a fail because the primary experience used in our testing was the most vetted and is what most questions were driven by. An alternate experience (a modal dialog) was shown to agents at the end, meant to be a bonus question if there was enough time.
We were able to ask all agents about the new experience. We specifically asked what agents thought of the new display of information and which they preferred. Agents commented that the popup is easier to read and the columns therein line up reducing cognitive load by eliminating the work of the agent trying to do that on their own. Overall they liked the cleaner and easier to read display of information.