How to work effectively with informatics consultants in life sciences?

As a leader in a pharmaceutical or life sciences organisation, getting the most out of your team and resources is always a top priority. After making the decision to proceed with a critical investment in consulting services, there may even be more pressure to find the optimal use of these time-limited external resources. So, how can you make sure you are using these resources to their full potential? In this blog, our industry expert Micah Rimer will show you how.

During Micah’s 20 years’ working at big pharma & vaccines corporations, including Bayer, Chiron, Novartis and GSK, he has successfully deployed consultancy groups within lab informatics and clinical projects. Micah has worked with Scimcon to support his teams on high profile critical projects

Frame the problem – what does your implementation project need to achieve?

As with any business situation, it is important that there is a common goal that everyone is aligned around.

It is essential that you do not waste valuable time revisiting the same conversations. Ask yourself: “Is it obvious what problem we are trying to solve?” Often, issues can arise when people are arguing about implementing a solution, whilst losing sight of the challenge at hand. 

Take the example of Remote Clinical Monitoring: You might decide that it would be beneficial to have your Clinical Research Associates (CRAs) track and monitor the progress of a clinical study without traveling to clinical sites. That sounds like it could be very promising, but what is the problem that needs to be solved?

Without clear goals on what you want to accomplish with Remote Clinical Monitoring, it will be difficult to declare an implementation a success. In addition, if you and your organisation do not know what you are trying to achieve with a particular technical solution, it will be impossible to give your informatics consultants a clear set of deliverables.

So, first things first, agree on the problem statement!

One of the first times I hired Scimcon to support me with an informatics project, I had recently joined a pharma company and found myself in the middle of conflicting department objectives, with what seemed to be no clear path out of the mess I had inherited. The organisation had purchased an expensive new software system that had already become a failed implementation. After spending a year continuously configuring and programming it, it was no closer to meeting the business needs than when the project had started. There were two loud criticisms to address on that point:

This also highlighted a far wider range of issues, such as some people who felt their skills were not being properly utilised while problems went unsolved, and that the bioinformatics department might not have the right goals to begin with.

To solve this challenge, we sat down with Scimcon to identify all the different problems associated with the inherited project, and to clarify what we needed to do to turn it into a success. In taking time to review the situation and without too much effort, we were able to come up with four key areas to address: 

  1. Understand how bioinformatics/ IT priorities should map to the organisation’s priorities – before we spent any more time and money, what did the organisation actually need?
  2. Solve the bioinformatics problem that the software had been purchased for (assuming that was indeed a verified need).  
  3. Determine how roles and work could be shifted and changed so that we were utilising the talents and the resources in the department.
  4. If possible, put the purchased software to use! 

With the help of Scimcon, we were able to define these problems and then focus on finding answers to each of the questions. In the end it turned out to be one of our most successful engagements together, award winning even. By just asking senior management what their biggest challenge was, we found their overriding priority was to have an overview of all the R&D projects going on. And while the new software was not particularly well suited for solving the bioinformatics problem that it had been acquired for, it could easily be used to map out the R&D process for portfolio tracking. Then, we turned our attention to the bioinformatics problem, which was easily solved by a bit of custom code from one of the bioinformatics programmers who felt that previously his skills were not being properly utilised.

Once we knew where we were, and where we wanted to get to, all we had to do was get there one challenge at a time.

Manage internal expectations – how will the informatics consultants work with your clincial/analytical teams?

Once you have identified and agreed on the problem that you want to solve, the next step is making sure the organisation is ready to work with your consultants. As with all relationships, business or otherwise, a crucial step is to make sure that everyone has the same expectations, and that all the relevant stakeholders are on the same page.

People have many different perspectives on why consultants are brought in.

As there can be so many different roles and perspectives on the use of consultants, you need to make sure that you address all the different stakeholder perspectives. It is important to establish a positive situation, as you want the consultants to be able to work with your teams without unnecessary tension.

When I was just starting out with my first LIMS implementation (Laboratory Information Management System), I remember being impressed that you could hire someone who had the specific experience and expertise to guide you on something they had done before but that was new to you. I wondered, “why was that not done all the time? Why do so many implementation projects fail when you can bring people in who had solved that particular problem before?” When I asked Russell Hall, a consultant at Scimcon for us on that first project, he said that not everyone is comfortable admitting they need help. As my career has progressed, I have come to value that feedback more and more. There are many people who are highly competent and effective in their jobs, but are not comfortable with the appearance that they are not sufficient on their own. It is always important to manage for those situations, rather than assuming that everyone will welcome external help.

Lastly, it is also critical to manage expectations, regarding the use of consultants. Your boss may need to defend the budget, or be prepared to stand behind recommendations or conclusions that are delivered from people outside of the organisation. It should also be considered that management might not readily accept something that might seem obvious to employees working at a different level. By liaising with senior leaders from the outset, you can make sure both parties are aligned how the consultants will interact with people in the company, and what their role will be. This is important both to achieve what you want internally and also to make sure the consultants have a proper expectation of how their efforts will be utilised. 

Communicate and adjust – how is your information managed between your team and consultants?

While it can be very tempting to feel that you can leave the majority of the project to the experts, the reality is things rarely go as smoothly as planned. As the life science business and information management have advanced over the last few decades, the amount of complexity and details has grown tremendously. It is more and more difficult for a single person to maintain an overview of all the relevant facts. The only way to be successful is to communicate and make sure that the right people have the right information at the right time. Your consultants are no different. 

Many organisations have challenges in terms of taking decisions and communicating them effectively. For your consultants who do not typically have all the same access and networks in the organisation that internal staff do, it is imperative that you make sure they are kept up to date. You want to avoid them spending valuable time on focusing on areas and deliverables that have shifted to being less important. Finding ways to keep consultants informed on all the latest developments is absolutely necessary for them to be able to deliver successfully. Figure out what makes sense by considering the organisation culture and the consulting engagement setup. Whether it is by use of frequent check-ins or online collaboration, be prepared to put in additional efforts to make sure that the information gets to where it needs to go.

As well as good communication, organisations have to be able to adjust as needed. Occasionally everything does work out according to plan, but that is more the exception than the rule when it comes to complex life science informatics projects. While timelines and commitments are critical, it is important to view any project as a collaboration. There will be unexpected software issues. There will be unplanned organisational changes and problems. People get sick, life happens. By having open and continuous dialogue, you can be best prepared to make the adjustments needed to find solutions together to unexpected problems.

Ensuring success in your informatics projects

Consultants can be hugely valuable to you and your organisation.

But you have to setup the right conditions for everything to work out well.

  1. Know what the problem you are trying to solve is, and make sure you have as much alignment around the problem statement as possible.  
  2. Make sure the organisation is ready for the collaboration by ensuring that your team and management know what to expect out of the engagement, and that your consultants similarly know the scope and what their mission is.
  3. Lastly, you need to keep in constant communication and make sure that you are ready to work together to adjust to the inevitable bumps that will come up on the road.

Working together, you can get to where you need to go.

If you’re interested in working with Scimcon on your upcoming informatics project, contact us today for a no-commitment chat about how we can help you succeed.

Meet Scimcon: Geoff Parker?

Profile

  • Name: Geoff Parker
  • Job title: Co-Founder of Scimcon
  • Pets: A Chocolate Labrador, who eats enough for 10
  • Favourite animal: Red squirrel
  • Favourite food: Sushi
  • Favourite film: Forbidden Planet
  • Fun fact: I played Angel Gabriel in the school nativity.

What do you enjoy the most about working at Scimcon?

That’s an easy question, I love the constantly changing challenges that informatics consultancy throws up and helping customers and our own internal teams to creatively meet those challenges.

We help customers in pharma, biopharma, and clinical trials to identify and evaluate the goals and challenges ahead and use this to influence information system strategies. We assist with programme and project direction; vendor and product selection (of LIMS, SDMS, ELNs and ePRO solutions); and to drive successful software implementations.

The works is often equal parts frustrating, challenging, and exciting but as people who have been involved in this type of work will recognise, it is ultimately extremely rewarding.

What do you enjoy doing in your spare time?

I am somewhat cursed by having multiple passions to fit into my spare time. Being a keen mountain biker, avid photographer and relentless traveller does present some time-management dilemmas.

Photography is something I have been fascinated by since I was at school, and is something that I have picked up and put down intermittently since then. However, three years ago I was lucky enough to spend a week riding bikes with a professional photographer and this really inspired me to develop my skills and style. You can view some of my results of my photography journey on my portfolio site.

My addiction to mountain biking started when I moved to Scotland ten years ago, as there is some of the best riding in the world, right here. The mixture of physical demands, technical riding skills and the pure pleasure of being surrounded by forest and mountains has proved irresistible. Not that the abundance of local riding has stopped me traveling with my bike. Ten years of truly amazing riding has taken me to some of the best trails in Morocco, Nepal, USA, New Zealand, Switzerland, Norway, Iceland, and Chile! Like most people, I am looking forward to when travel once again becomes possible, then maybe I can put my planned trips to Kyrgyzstan and Bhutan into action.

What is your favourite travel destination?

I’m not sure I can pick a favourite, as it depends on the activity involved. For a chilled afternoon walk drinking coffee, I would have to say San Francisco. Relaxed attitude, wide open streets and great culture make it a hard place to beat.

For mountain biking, it could be any of those places I mentioned above. My trip spending a week on HMS Gassten, a converted World War Two minesweeper, with a group of good friends riding high above the fjords in Norway deserves special mention.  Also, I think Nepal’s Mustang Valley was a special trip. Amazing culture, fantastic people, and as for those mountains – they build them big over there!

2020/21 has been an interesting year – how has it impacted you outside of work?

The biggest difference is that I have been at home for the longest period in about 22 years! In the last 12 months, I have only managed to travel down to our office in Newmarket twice (from my home in Scotland). I would normally travel 2 or 3 weeks out of 4, so it has been a big change.

Distancing from family and friends has also been difficult, as it has for most. I have stayed in touch as much as possible with family and friends the same way anyone else has; telephone calls, meeting in car parks and standing 4 metres apart, in addition to several Zoom quizzes!

Scimcon as a business is deeply rooted in technology – but how technology-oriented are you? What devices do you use?

I have had a long fascination with technology, which of course in part has led to my role at Scimcon. We have obviously come a long way since I first became involved in technology. I started programming on home computers during the explosion of affordable devices in the UK during the early 1980s.

Everything has continued to stem from there, to a point where I’m now the co-founder of a scientific technology consultancy, using technology to drive change and deliver scientific value.

In terms of devices, I use pretty much anything you can imagine. I have electronic shifting on a couple of my bikes, and I have some amazing camera equipment that even five years ago would have been inconceivable.

Does your use of technology differ outside of work?

I don’t think my use of technology in my everyday is too different to how I would use it at work. At Scimcon we try to use technology in a way that enhances people’s work lives, so that they can focus on what is important. If I look at technology in bikes and cameras, what excites me is the same thing – not the technology itself, but what it allows you to do.

For example, when I started photography at school, the process was incredibly different to how it is now. You tried different methods of taking photographs and would be left with a roll of film afterwards that if you were lucky, you’d get developed a week later. Of course, you could guarantee that you would get the photos back after all that time and wonder “what on earth was I trying to do?” Nowadays, we have digital cameras, so you take the photo, see the result, and if it does not work you can just try again instantly.

That’s exactly how we try to use technology at Scimcon. We don’t want technology to get in the way or slow down the result you want to achieve, we want it to enable you to do the work you are trying to do – whether that’s riding bikes, taking photographs, or integrating your LIMS to streamline your workflow.

The challenges of implementing ePRO – part two?

Cell network and WIFI connectivity

Clearly good cell network or WIFI connectivity is of primary importance when thinking about using ePRO in a clinical study. Sites should conduct a feasibility activity in their local vicinity during the study initiation phase to understand if cell network strengths are good enough for ePRO and if not, if there is likely to be a large population of subjects that have home WIFI. For remote locations with possible issues with either the cell network coverage or even with the availability of electricity, these sites should be reconsidered before a decision is made to use ePRO.

However, even with bad cell network coverage, it is possible to conduct a study with ePRO by ensuring the correct messaging appears to the subjects on the device. All ePRO systems allow the subjects to access and complete their diaries on a daily basis without cell or WIFI connectivity. The data entered by the subject is stored on the device until connectivity is established. There are circumstances where the site may need to be made aware of certain responses in real time in order to maintain subject safety. If connectivity is established and the data is sent in real time, the site staff may receive an alert from the system asking them to contact the participant. In order to ensure participant safety is maintained it is good practice to alert the participant to contact the sites / study doctor if certain responses in the diary are provided, such as if the participant required emergency medical attention.

Translations

As mentioned earlier in an earlier instalment, translation is an aspect not typically associated with clinical systems such as EDC. Additionally, because the ePRO devices are in the hands of the participants, the screens, in their local languages, must be submitted to the local ethics committees within the countries that the study is to take place in.

It should not be forgotten that paper PRO is also subject to a translation process. However, the additional complexity with ePRO is the need to apply the translated text to the software in order to generate the questionnaire screens. This involves multiple review and update rounds between the ePRO vendor and the translation vendor which increases time, effort and therefore cost. Once the screens are generated there may be further review rounds during the in-country review between the local sponsor representatives, ePRO vendor and translation vendor. All of these review rounds increase the timescales for an activity that is already time critical.

Often the date for submission to the ethics committee is set during the planning phase of the study by the sponsor study team. If the submission date is missed it could result in a delay to the study start. Consequently, it is important to ensure clear timelines are in place between agreeing the ePRO requirements, which will decide what is displayed on each of the screens, and submitting the screen report to the ethics committees. The ePRO vendor will need to be made aware of these timelines at the earliest opportunity. It can take more than 12 weeks for the completion of the translations resulting in the creation of the finalized screen reports. These timelines are difficult to manage if the process includes an in-country review round which allows for local country representatives from the sponsor to review the translations. It is important to make clear to the reviewers what it is that these stakeholders are reviewing. Translation is not an exact science. There are many ways of writing the same sentence. As the translator is the trained expert in translations it should be left to them to choose the most appropriate wording in the local language which most faithfully represents the English version. The in-country review, if indeed one is required, should only be conducted to ensure the correct screens are displayed in the screen reports. Allowing the in-country reviewer to make suggestions on preferred wording risks multiple back and forth review rounds, increasing timelines and jeopardizing study start dates.

End User Acceptance

As briefly mentioned in part one of this series, possibly the most crucial aspect when introducing a new system or process to a user base is change management. If you do not bring your users along the journey with you, you are less likely to gain their acceptance.

How does this manifest itself? With ePRO there is plenty of opportunity for issues to arise, from delays in getting devices to sites, to errors in the software, to usability and connectivity issues. All of which can affect the investigators ability to get on with their daily work. If you have not put in place a good change management process you will find the investigators very quickly become disenchanted with the system and even the smallest of issues will become magnified, resulting in escalations to the sponsors senior management team.

It is important from the outset to set the stage with the investigators. Why are we using ePRO? What benefits does it bring to the sponsor? It may require more work on behalf of the investigator which will need to be compensated for. It is also about setting expectations. Things will go wrong, issues will need to be resolved, backup processes may need to be utilized, but the investigator must be aware that they have the support when needed and how they can access that support.

When implementing ePRO thought should go into understanding how to improve the investigators experience. For example, investigators can be working on multiple studies at the same time for different sponsors all using ePRO, so adding labels onto the packaging of shipped devices so that investigators can quickly store them together is a quick win.

As a sponsor it may also be necessary to implement an additional layer of support for the study team. When issues arise the investigator will contact the vendor helpdesk. Often the helpdesk are unable to provide an immediate remedy so the investigator will then contact the sponsor’s study team representatives. The additional layer of support sits between the study team and the vendor, collating issues, communicating technical information back and forth in a manner that is easily digestible and holding the ePRO vendor to account. This relieves the frustration experienced within the sponsor’s study team and reduces the likelihood of escalation.  

Qualification/Validation

It is important to remember that in accordance with ICH GCP guidance section 5.2.1 [1] the sponsor may transfer trial-related duties to a vendor, however the ultimate responsibility for the quality and data integrity of the trial data always resides with the sponsor. In order to ensure data is collected, stored and transferred in accordance with ICH-GCP guidance the sponsor will need a suitable oversight strategy of the vendor’s processes.  This may, at a minimum, include a qualification activity which involves auditing the ePRO vendor on a regular basis. It may also invoke some internal validation work to ensure, as the sponsor, your own organisation’s technical and study specific requirements for clinical systems are met.

It is not recommended for the sponsor to leave the responsibility of vendor oversight to the individual study teams. The study teams may not have the experience or technical know-how necessary to understand the challenges and considerations discussed earlier in this blog.

We would recommend a two stage process.

Stage one is the qualification of the vendors system by a centralized team including stakeholders from the data management and computer system quality departments. Once qualified, the vendor’s system can be classed as a platform that can subsequently be used on multiple studies.

Stage two is to have experienced stakeholders conduct study specific User Acceptance Testing (UAT) in order to test the many scenarios and nuances associated with ePRO and to ensure the system meets the requirements of the study Protocol. Once implemented and running live on a study it is far more difficult to update the system than finding and fixing the issue during the implementation phase.

However, UAT is the very least the Sponsor must do as part of the overall study specific validation activity (Stage two). The Sponsor’s quality management system (QMS) may also mandate that the implementation of any clinical system (which would include ePRO) requires internal validation documentation to supplement the vendor’s validation package. This can include documents such as a sponsor Validation Plan, Requirements Specification, Risk Assessment, Traceability Matrix and Validation Report.

ePRO vendors will often offer to provide a service to create UAT script on behalf of the Sponsor, however this is not recommended as a misunderstanding of requirements by the vendor may also manifest itself in the scripting. This can result in the script passing when executed when in fact it does not meet the requirements of the Sponsor.

For these reasons, having experienced validation professionals generating the deliverables and executing the testing is recommended.

Summary

The two parts of this blog have detailed, at a high level, many of the challenges that can be experienced during the implementation and use of ePRO. These challenges are summarized below:

These challenges, and others, often create frustration for the investigators and study teams, and in the worst case scenarios, studies can be put on hold or stopped completely, costing millions of dollars and preventing a product from going to market.

A mechanism to help mitigate, or at the very least support the investigators and study teams through, many of these challenges is for the sponsor to put in place a centralized ePRO team within their organization. This team, consisting of internal sponsor stakeholders or perhaps supplementing Scimcon resources into the team, would be responsible for the qualification of the ePRO vendor, the strategic implementation of libraries and standards and for supporting the implementation of ePRO study-specifically, providing help with creating validation deliverables and executing UAT, and being a second level of support during the conduct phase of a study. ePRO is fast becoming the standard and it is worthwhile investing early on in an internal team to ensure success.

To discuss how Scimcon can support your organization with the implementation of ePRO, please contact one of our consultants for an informal conversation.

References

[1] ICH harmonised guideline integrated addendum to ICH E6(R1): Guideline for Good Clinical Practice ICH E6(R2) ICH Consensus Guideline, https://ichgcp.net

The challenges of implementing ePRO – part one?

Benefits of Implementing ePRO 

When it comes to documenting the advantages of using ePRO over paper in clinical trials, the benefits are clear. 

With all the advantages to using ePRO over paper it seems to be a no-brainer to use ePRO whenever possible. However, it’s important to be mindful of certain considerations and challenges that come with the implementation of ePRO within your organization before jumping in.   

Challenges and Considerations 

Historically the implementation of electronic clinical systems in general has been challenging. In the majority of cases it requires the move from a paper-based process to an electronic system in an environment where the reliance has always been on paper, hindering the adoption of computer systems that are seen as alien. Taking EDC as an example, the response to an international survey cited that 46% of respondents identified inertia or concern with changing current process, and 40% identified resistance from investigative sites as the major causes for adoption delays [2].    

ePRO is not immune to these challenges. In fact, it could be argued that ePRO is even more susceptible. While ePRO suffers from the traditional technical issues and user acceptance that EDC experiences, ePRO is also placed in the hands of potentially thousands of study participants many of whom may have little technical understanding. Additionally, ePRO relies on hardware (mobile device or tablet), cell network or WIFI connectivity, translation into the participant’s local language, multiple userbase (study teams, investigators and participants) and local helpdesk support, all of which comes with their own set of challenges and associated costs and few, if any, of which are encountered with EDC.  

ePRO is one of the few electronic systems that directly collects source data and as a result comes under increased scrutiny from a data integrity and quality perspective, especially when used for primary or secondary end point data collection. The system must always be available in order to allow subjects to be activated on ePRO devices. If a participant leaves a clinical site without an active device, this can result in missed data which can be construed as a serious quality issue and perhaps put subject safety at risk.  

Before I go into the more detailed challenges associated with ePRO, let’s first consider the financial costs. 

Cost   

On the surface of it, it would appear that implementing ePRO is significantly more costly than paper. The expense of the devices, associated logistics and data usage (monthly SIM costs), the licenses, helpdesk and translations all contribute to costs that range from hundreds of thousands of dollars to multi-million-dollar contracts per study.  

When making a business case for ePRO it is important to take into account the hidden costs associated with paper in order to compare the two.  

  1. The additional eCRF’s that must be created in your EDC to house the data transcribed from the paper PRO.  
  1. The time spent by the site staff transcribing data from the paper PRO into the EDC and the associated monitoring required to source data verify the transcribed data. For example; this activity requires onsite visits by the CRA.  
  1. The data cleaning at the end of the study by the Data Management team, which takes time and effort with multiple communications back and forth to the investigator. ePRO can significantly reduce the timeline associated with data cleaning due to the nature of these data being electronic source data.  

When conducting a full assessment, the gap between the cost of implementing ePRO vs paper reduces significantly. ePRO vendors have attempted to provide examples which result in paper diaries actually contributing more cost to a study budget than ePRO.  

The business case for implementing ePRO should not be solely based on raw cost. This will likely result in failure to get agreement at the leadership level. You will find it easier to get acceptance if you can prove that ePRO costs are comparable to paper while also concentrating on the non-tangible benefits as, in the case of ePRO, these are the real reasons for its consideration. Increasing the quality of your data collection results in more confidence in that data, which in turn reduces the likelihood of rejection when submitted to the regulators (predominantly for primary and secondary end point data). Receiving the data in real time and reducing the need for data cleaning can aid the ability to get a product to market quicker by shortening the timelines to close the study, which in turn results in cost avoidance.   

Many ePRO vendors will provide a cost calculator; a spreadsheet where the sponsor can plug in parameters associated with their study to provide an estimate of costs before engagement with the vendor. Only a small number of parameters are required to calculate a good estimate with the most important being the length of the study in months and the number of participants. The length of the study drives the helpdesk, data usage and PM costs, whereas the number of participants drives the device, logistics and shipping costs. There are other costs associated with the configuration of the system, translation, shipping, number of sites etc, however these are often negligible in comparison for larger studies.   

In summary; it is important to build a business case for ePRO within your organization in order to assist in gaining acceptance at a leadership level. The business case should include areas of efficiency over paper together with examples of ePRO costs using the cost calculators provided by the vendors, as well as emphasizing the other benefits of ePRO, such as subject safety, compared to paper solutions.  

Software 

In the past, ePRO implementations were customized pretty much from the ground up, coding the study specifics into the vendor’s study builder toolkit. This resulted in a huge effort required to validate the system to ensure errors and bugs were captured before studies went live. Inevitably despite all this testing some issues did make it through to the live study, causing frustration for the participants, investigators and study teams.   

Over the past decade the systems have become more sophisticated. Less code is required during the implementation phase, which has been replaced with configuration. Vendors have also introduced library functionality which allow sponsors to define questionnaires up front that can be reused across studies. As the questionnaire is not rebuilt every time there is less opportunity to introduce errors. Additionally, the ability to reuse questionnaires from a library also results in less work by the vendor per study, less validation on behalf of the sponsor and can reduce the time and costs during the implementation phase.  

It may also be possible to standardize other areas of functionality, perhaps the workflow as to when questionnaires are made available to the participants, or the alerting system, or the visit schedule. It may not be possible to standardize across therapeutic areas, but within a therapeutic area where multiple studies collect the same data this approach can result in substantially reduced timelines during the implementation phase, while reducing the risk of software errors on the studies.     

Hardware (mobile device or tablet) 

ePRO can be implemented in a number of different modalities. In this blog, we are concentrating on provisioned devices; devices that are provided by the ePRO vendor at a cost to the study Sponsor, and “bring your own device” (BYOD); where a subject’s own device is used as the ePRO instrument. It should be noted that all studies require provisioned devices to a certain degree in order to cater for cases where a subject either does not own a compatible mobile device or does not own a mobile device at all. 

When provisioning devices, ePRO vendors are responsible for the associated logistics such as software installation and shipping. Vendors are generally very knowledgeable when it comes to the customs regulations in many countries, including the average timelines required to get a shipment to a site. 

In scenarios where competitive recruitment between sites is employed, it is particularly important to plan ahead.  As it may not be possible to predict the number of subjects that will be recruited at a specific site and therefore the number of devices required at that site, it is necessary to purchase a sensible overage of provisioned devices. Although costly it ensures sites will not run out of devices.  

With an increasing number of the world’s population now owning smart phones, BYOD was seen as the natural progression for ePRO. It reduces the costs and burden of acquiring devices and the associated logistics and also reduces the monthly costs for data usage. These costs do not completely disappear as a certain level of provisioning is required for those cases where participants don’t own a compatible smart phone. BYOD also reduces risks associated with not having enough devices on site, especially, as mentioned above, with competitive recruitment. However, BYOD does come with own set of unique challenges, mainly associated with data integrity and privacy. Some considerations might be: 

  1. How do you ensure equivalence between participants devices to guarantee the questionnaire is displayed to each participant without the introduction of bias? 
  1. As the participants devices are not locked down participants can turn off alarms/alerts, change the date and time etc. How can this be managed? 
  1. Who pays for the data usage on the participants SIM card? 
  1. Participants are not required to carry multiple devices (study device and their own smart phone). This is seen as a benefit and can increase compliance further. 
  1. As the participant will use their own personal smart phone and it is not dedicated to the study, the participant may have concerns over the security of their non-study related private information. 

There are clearly a lot of benefits of using BYOD over provisioned devices with more and more sponsors feeling comfortable moving into this space, however it is important to consider the implications before doing so.    

In the next instalment of this blog, we will discuss some other challenges of implementing ePRO in your organisation, such as connectivity, translations, and end user acceptance testing. Keep an eye on our Opinion page for part two of the series, coming soon. 

References 

[1]  ‘Electronic for Industry – Electronic Source Data in Clinical Investigations’, FDA 2013 http://www.fda.gov/downloads/drugs/guidancecomplianceregulatoryinformation/guidances/ucm328691.pdf  

[2]  ‘Welker JA. Implementation of electronic data capture systems: barriers and solutions.’ Contemp Clin Trials. 2007 May;28(3):329-36. doi: 10.1016/j.cct.2007.01.001. Epub 2007 Jan 11. PMID: 17287151. 

Planning for successful User Acceptance Testing in a lab or clinical setting?

What is User Acceptance Testing?

User Acceptance Testing (UAT) is one of the latter stages of a software implementation project. UAT fits in the project timeline between the completion of configuration / customisation of the system and go live. Within a regulated lab or clinical setting UAT can be informal testing prior to validation, or more often forms the Performance Qualification (PQ).

Whether UAT is performed in a non-regulated or regulated environment it is important to note that UAT exists to ensure that business processes are correctly reflected within the software. In short, does the new software function correctly for your ways of working?

Identifying and managing your requirements

You would never go into any project without clear objectives, and software implementations are no exception. It is important to understand exactly how you need software workflows and processes to operate.

To clarify your needs, it is essential to have a set of requirements outlining the intended outcomes of the processes. How do you want each workflow to perform? How will you use this system? What functionality do you need and how do you need the results presented? These are all questions that must be considered before going ahead with a software implementation project.

Creating detailed requirements will highlight areas of the business processes that will need to be tested within the software by the team leading the User Acceptance Testing.

Requirements, like the applications they describe, have a lifecycle and they are normally defined early in the purchase phase of a project. These ‘pre-purchase’ requirements will be product independent and will evolve multiple times as the application is selected, and implementation decisions are made.

While it is good practice to constantly revise the requirements list as the project proceeds, it is often the case that they are not well maintained. This can be due to a variety of reasons, but regardless of the reason you should ensure the system requirements are up to date before designing your plan for UAT.

Assessing your requirements

A common mistake for inexperienced testing teams is to test too many items or outcomes. It may seem like a good idea to test as much as possible, but this invariably means all requirements from critical to the inconsequential are tested to the same low level.

Requirements are often prioritised during product selection and implementation phases according to MoSCoW analysis. This divides requirements into Must-have, Should-have, Could-have and Wont-have and is a great tool for assessing requirements in these earlier phases.

During the UAT phase these classifications are less useful, for example there may be requirements for a complex calculation within a LIMS, ELN or ePRO system. These calculations may be classified as ‘Could-have’ or low priority because there are other options to perform the calculations outside of the system. However, if these calculations are added to the system during implementation, they are most likely, due to their complexity, a high priority for testing.

To avoid this the requirements, or more precisely their priorities, need to be re-assessed as part of the initial UAT phase.

A simple but effective way to set priority is to assess each requirement against the risk criteria and assign a testing score. The following criteria are often used together to assess risk:

Once the priority of the requirements has been classified the UAT team can then agree how to address the requirements in each category.

A low score could mean the requirement is not tested or included in a simple checklist.

A medium score could mean the requirement is included in a test script with several requirements.

A high score could mean the requirement is the subject of a dedicated test script.

Planning UAT

A key question often asked of our team is how many test scripts will be needed and in what order should they be executed? These questions can be answered by creating a Critical Test Plan (CTP). The CTP approach requires that you first rise above the requirements and identify the key business workflows you are replicating in the system. For a LIMS system these would include:

Sample creation, Sample Receipt, Sample Prep, Testing, Result Review, Approval and Final Reporting.

Next the test titles required for each key workflow are added in a logical order to a CTP diagram, which assists in clarifying the relationship between each test. The CTP is also a great tool to communicate the planned testing and helps to visualise any workflows that may have been overlooked.

Now that the test titles have been decided upon, requirements can be assigned to a test title and we are ready to start authoring the scripts.

Choosing the right test script format

There are several different approaches to test script formats. These range from simple checklists, ‘objective based’ where an overview of the areas to test are given but not the specifics of how to test, to very prescriptive step by step instruction-based scripts.

When testing a system within the regulated space you generally have little choice but to use the step by step approach.

Test scripts containing step by step instruction should have a number of elements for each step:

A typical example is given below.

A screenshot of a cell phone
Description automatically generated

However, when using the step by step format for test scripts, there are still pragmatic steps that can be taken to ensure efficient testing.

Data Setup – Often it is necessary to create system objects to test within a script. In an ELN this could be an experiment, reagent or instrument, or in ePRO a subject or site. If you are not directly testing the creation of these system objects in the test script, their creation should be detailed in a separate data setup section outside of the ‘step by step’ instructions. Besides saving time during script writing, any mistakes made in the data setup will not be classified as script errors and can be quickly corrected without impacting test execution.

Low Risk Requirements – If you have decided to test low risk requirements then consider the most appropriate way to demonstrate that they are functioning correctly. A method we have used successfully is to add low risk requirements to a table outside of the step by step instructions. The table acts as a checklist with script executers marking each requirement that they see working correctly during executing the main body of step by step instructions. This avoids adding the low requirements into the main body of the test script but still ensures they are tested.

Test Script Length – A common mistake made during script writing is to make them too long. If a step fails while executing a script, one of the resulting actions could be to re-run the script. This is onerous enough when you are on page 14 of a 15 page script. However, this is significantly more time-consuming if you are on page 99 out of 100. While there is no hard and fast rule on number of steps or pages to have within a script, it is best to keep them to a reasonable length. An alternative way to deal with longer scripts is to separate them into sections which allows the option of restarting the current block of instructions within a script, instead of the whole script.

Are all the requirements covered?

An important task when co-ordinating UAT is be fully transparent about which requirements are to be tested and in which scripts. We recommend adding this detail against each requirement in the User Requirements Specification (URS). This appended URS is often referred to as a Requirements Trace Matrix. For additional clarity we normally add a section to each test script that details all the requirements tested in the script as well as adding the individual requirement identifiers to the steps in the scripts that test them.

What comes next?

UAT is an essential phase in implementing new software, and for inexperienced users it can become time-consuming and difficult to progress. However, following the above steps from our team of experts will assist in authoring appropriate test scripts and leading to the overall success of a UAT project. In a future blog we will look at dry running scripts and formal test execution, so keep an eye on our Opinion page for further updates.

Industry leader interviews – Ajit Nagral. An insight into the world of scientific entrepreneurship?

Ajit, please introduce yourself. 

My career history is uncomplicated because straight out of college I started working for myself, and it has been that way for my entire career. I graduated in the US with a background in computing science and I decided I did not want to take the traditional path of finding a job and building a career that way. I always liked to be doing things differently. Coming out of college with very little money and a limited skillset, the reasonable thing to do was to get into software consulting because that did not require a whole lot of capital. Since then, I have founded four companies in the life science sector.

What led you to the science industry?

I ended up in the pharma and life science industry very early on by chance. After I graduated, I was in Boston with database and computing skills, and I started a small consulting company called Megaware. I found out there was a large life science vendor in Massachusetts which had some opportunities around a new life science product they were building. After a lot of persistence, the CIO reluctantly gave me 15 minutes to speak with him. I told him about my background and what I was attempting to do – he said they didn’t have anything for me within his organisation, but he would connect me to his counterpart at their analytical instrument division. I then got a contract to help this analytical instrument company build a part of their (then) new product, the first database driven instrument software, and that was my entry into the pharma world. That seems simple, but it was fortuitous and persistence more than anything else.

Can you give us a potted history of each of your companies? 

My first company was Megaware. Back in those days labs were making the transition from VAX.VMS-based systems to PC-based systems. Enterprises had huge investments in VAX.VMS systems and in HP printers. We produced a product to be able to print from a VAX.VMS system onto a HP printer. It seems straightforward (but it was not!) because you are printing to a Windows-based printer but from a non-Windows system. We built a whole system to rectify this issue. I ran and built Megaware for 4 – 5 years and it was then sold to a boutique consulting company.

NuGenesis was my second company. We identified the issue that labs had several different instruments, from different vendors, but they did not talk to each other. You had large pharma companies, printing reams of paper, spending $millions of dollars on running labs across the globe and eventually all of that intellectual property ended up on paper! In those times when they submitted a new drug application, those applications were on tens of hundreds of pieces of paper which were carried to the regulatory agencies on trucks for review! It made no sense that something started out electronically and ended up on paper to be read by somebody manually. We were able to intercept print streams and capture a lot of information to make the data live. It is remarkable that 20 years later it is still being used – that says a lot about the value and sustainability of NuGenesis. I sold the business to Waters in 2004.

After I sold NuGenesis I was clear I wanted to stay in life sciences but do something different. I went back to many of my clients and ask what problems they are facing – that is when I landed into outsourcing in the area of drug safety, clinical and regulatory.

What was new and different was an area called pharmacovigilance. If you recall, there were a couple of landmark cases related to drugs in the market that had caused deaths. That is when the regulatory agencies realised they do not have a handle around adverse events. They approve drugs, they come to market and years later you start seeing adverse reactions that you did not see during the trial period. The regulatory agencies started mandating reporting of all adverse effects. With the visibility and potential liability, the biopharma industry sprang into action and the flood gates opened to drug safety outsourcing. This was when we launched Sciformix – a scientific knowledge-based outsourcing provider for the life science industry. Any given year when we got to the maturity stage we were doing around 1 million cases. We did everything from cancer drugs to consumer products, from cancer medication to sunscreen lotion. Our success at Sciformix was due to our ability to combine enough science and a very good process. Again, the company grew very rapidly to over 1300-1400 people globally, and it came to a point where it felt like the time was right to divest in 2018. Sciformix was acquired by LabCorp/Covance, a top three CRO (and currently a leader in COVID19 diagnostic testing).

What was the motivation behind the launch of Scitara? 

Having done tech, services and global delivery, I thought I should combine these skills and focus on my finale!

Our core team believes Scitara is more than just a business, it is a goal of ours to solve a major problem that still exists in the scientific laboratory: data connectivity. We are pioneering a new digital revolution when it comes to lab data connectivity. We have invented a platform called Scitara DLX (data lab exchange), and our goal with this platform is to connect your instrument, application, or anything else you use in the lab to our platform and we guarantee they can talk to each other.

Our goal is that science labs can log into any system that they are currently using and can access data from any other system that is in the lab. We have a mantra of ‘no application or instrument left behind’. For us to achieve this goal we need cooperation from the industry, which is why I am calling this a finale. It will require all our connections we have made over the years and the reputation we have built to reach out to everyone in the ecosystem. Companies are making significant headway in their digital transformation initiatives, except they do not know how to get their lab data onto their digital platforms, and that is where we come in.

How did you find your entrepreneurial drive?

I am very driven to be independent. I am useless when it comes to working for someone else and fortunately, I have never had to. My personality drives me to try new things and dive into uncertainty and this has always pushed me into something completely new.

The building of my companies motivated me. What excites me is the building from the ground up. Each time the building is easier, but the expectations are higher. I do not build to divest – I build to create value, disrupt, and hopefully deliver a meaningful impact, and the rest takes care of itself.

If anyone comes to me for advice or mentoring, I ask them why? Why do you want to do it? Why you? What is the motivation? That tells a lot very early on about the chances of success that a person may or may not achieve. It does not guarantee success, but if you have a good understanding of the ‘why’ it helps you go a long way. Beyond that I’d say it is important to find a mentor from the industry – people need to recognise that investments happen in teams not necessarily ideas. Do not latch onto an idea too much because things can change.

Create a loyal fanbase, people often think I have 500+ clients, but it is not the number that counts, it is whether you have a handful of loyal clients who make a lot of noise and reopen doors. That becomes exceedingly important.

What would you say makes you a successful entrepreneur? 

We do not rely on big sales engines in our industry. It is about building solid connections and networks. When clients learn that I created the concept behind several successful companies, people admire that. There is no better way to connect with a client than something that they are fond of and that I am proud of.

I have learnt the hard way; you build the best partnerships in tough times. When things do not go right, it is how you react that defines not only your relationship but your career as an entrepreneur. I have sold to the same clients across multiple companies. Most of those clients I have had difficult moments with, and it has made our relationship much more resilient.

Having a non-scientific viewpoint has also really helped, particularly when it comes to products. To be able to look at the consumer world, or industrial world or finance world and understand how technology has evolved there and bring those learnings into the scientific world is invaluable.

What does the future hold for Ajit Nagral? 

This is the first time after having done this for 20+ years that I have the liberty and luxury to say if this part of my journey were to end, what would be my new journey? It is the first time I have thought about it, and I think it comes with experience and the safety net I have built for me and my family. I am eternally grateful to my customer, employees and investors to put me in this position.

Hopefully Scitara is my last company, as an operating founder. There are many other things I want to do. In addition to being a tech guy I am also a musician. There are things I am doing in music production that I have started already – hopefully in a few years once I am done with Scitara, that is where I will end up!

Industry leader interviews – Andrew Miles. The role of pre-sales demos in successful vendor selection events?

Andrew, please introduce yourself.

I am Andrew Miles and I have worked with commercial vendors in informatics pre-sales for over 20 years. I originally trained for seven years as a biomedical scientist in haematology and blood transfusion science, however, I made the decision to move into an IT role, and enjoyed my responsibilities implementing IT systems into the lab.

Tell us more about your role in the lab software sector

I was in my first IT role for seven years, training scientists on how to roll out lab software. As my role progressed, I started to work on pre-sales, and I really enjoyed this aspect of the job. I then started looking for a pre-sale focused role and landed a position at LabVantage Solutions in 2000, which was a relatively small vendor at the time. Most of my experience comes from here, as it was a hugely diverse role working with customers from every area of lab testing imaginable.

In 2016 I moved to the healthcare sector on diagnostics solutions at InterSystems, before I made the decision to retire in April 2020.

Do all Lab Informatics vendors have pre-sales specialists?

I would say the majority of vendors do have specialists in this field. You can split the sector into big industry players which offer a broad range of solutions, and then smaller companies which may offer simpler or more niche solutions, or only operate in one field. These big vendors would certainly have a dedicated team, and this team will be made up of individuals (like myself) with lab experience, as an understanding of how labs operate is vital.

In addition to pre-sales and sales, some labs involve consultancies like Scimcon to help lead their projects and advise. This is how I met and built a relationship with the Scimcon team – Scimcon was consulting on a project with one of our prospective clients and was impressed with our demo, and we got to know each other quite well.

How long is the process?

In terms of timing, it really varies. I worked on one project which ran for about five to six weeks in total. I have also worked on a project which spanned over 18 months until it was completed. It depends on the complexity of the project being undertaken, and the size. A small project in the UK covering only one or two labs will take much less time than a global project encompassing multiple sites.

If a pre-sales demo is successful, there is an official handover when the contract has been signed between pre-sales and sales teams, and the implementation teams. However, it would not be a clean-cut finish point, and pre-sales and sales would normally remain involved in some way as the implementation teams will have questions as the project moves forward. For example, if we had offered bespoke functionality, we would need to be held accountable and help the implementation teams understand what was offered and why. This means that we would be involved in a process for much longer in some cases.

How are customer expectations managed?

When it comes to customers, it is important to remember that labs have a clearer idea of where they are heading in the future than software vendors do, so it’s about doing your research and making sure you understand your customer as well as they understand what they need to achieve. It is difficult to create a product which ticks every single box, but most vendors are now aiming to create a product that works straight out of the box and can be configured to meet requirements. This is particularly helpful in demos, as you can offer a wide range of features within the product as it comes, but you can still offer configuration if a particular feature is needed.

What best practice have you observed in how potential clients prepare internally and brief vendors during selection processes?

I think it is particularly important that multiple people from the lab are involved in the demo. Not just the decision makers, but the lab technicians, the managers and IT departments who can understand the tech. Everyone who will be impacted by new software and products at all levels should be involved in the process, and in the past, we have found it really unusual to give a demo to only one or two people.

It also helps when clients have a clear idea of what they are looking for, and what they want to get out of their product. With this kind of information, it makes it much easier for pre-sales teams to advise further and offer a solution that is most appropriate for the client’s need. However, at the same time, a client does need to be prepared to compromise. It is difficult to create a one-size-fits-all product, and pre-sales teams have to be prepared to walk away if they are unable to meet the customers’ requirements without creating a hugely complex product. If a client is willing to negotiate, and is happy to cover additional costs, then technology vendors could then consider a more bespoke approach and solution.

How can you tell when a demo has gone well?

As I said before, a demo should be reaching the whole room, not just the decision makers. A sale is obviously a clear sign you have presented a successful demo, but you can normally grasp the general feeling in the room about halfway through. You will find that people start leaning forward and engaging more through questions and comments, which shows you have piqued their interest. It also helps if you build allies in the room, as you will find they are able to counter any negative opinions, and interest in the product spreads around the room.

Were there ever any times when you believe a demo could have gone better?

Of course, we have not always got it right, and those are the times that taught us how important market knowledge is in securing a prospective client. One instance I recall was when myself and member of the sales team were asked to do a last-minute demo, and my colleague said that the client was in the mineral mining industry. The first thing the coordinator asked us during the demo was if we had any experience with customers in the prospecting and diamond mining industry, which we did not. We managed to recover with some relevant content, but even so, it is still a situation we would definitely want to avoid. If you are not showing relevance or credibility as a vendor to prospective clients, you really set yourself up for a fall.

What is the outlook for the Lab Informatics industry, where do you see it going?

I have already seen a lot of change in my 20 years in pre-sales. Technological changes are definitely the biggest area, but more recently we’re seeing surges in specific scientific areas such as genomics and the use of biorepositories for specimen storage, donor consent and retrieval.

As technology has advanced, we are also seeing more intelligent technology come into the field. As artificial intelligence algorithms are programmed into systems such as LIMS, we could expect another surge towards extended automation, which will save both vendors and client’s money in the long term.

Do you believe the current pandemic situation will have lasting effect on technology vendors?

I do not believe the current situation will have a long-lasting impact on vendors. In the short term we may see issues with implementation of systems, as current restrictions only allow necessary work to be carried out, adhering to social distancing measures. In the long-term implementation activities will return to normal, there may be additional opportunities for vendors to provide data handling system as some labs have geared specifically towards COVID-19 testing. I am confident that the LIMS market will eventually return to something like normal.

Digital transformation: Revolutionising the labs of the future?

Scimcon has worked with many lab-based clients throughout our 20 years in the industry, across a vast range of projects. Here we discuss the current challenges that labs are facing in 2020, and the work that needs to be done through digital transformation to ensure that labs in the future can streamline and manage their data.


The limitations of the current laboratory information systems landscape

Today’s labs are facing similar challenges as camera companies. Camera manufacturers such as Nikon and Canon are now faced with the challenge of selling to a new generation of budding photographers, most of whom by now have grown up with increasingly higher-quality smartphone cameras. As a result of having access to technology that is designed for ease of use, this generation of users find themselves progressively more frustrated with traditional technology and methods required to operate today’s ‘real’ cameras. Where smartphones can offer instant uploads to online services; amazing results that leverage computational photography; and synchronicity between multiple devices, traditional cameras appear complicated, difficult to control and impractical. Camera companies therefore face the challenge of building usability, such as that found in smartphone cameras into their existing products, otherwise they risk losing a whole demographic of potential customers.

The analogy is that modern labs are facing a similar problem. As new generations of scientists join laboratory settings, many are finding the lack of synchronicity and usability of information management systems increasingly frustrating. Why can’t we check instruments remotely whenever we want? Why can’t data be easily transferred between devices or colleagues? Why isn’t all this information seamless? Limitations such as these can be hugely time-consuming, as well as resulting in reduced productivity and security risks for data with minimal protection. Similar to the camera makers, we are risking losing the best new talent to other areas of science. Digital transformation addresses these challenges head on, with the proficiency to make your lab more intuitive and efficient.

What is digital transformation and how can it enhance your current lab setup?

Digital transformation involves the integration of new technology and methods into existing lab technology. Although this advancement in technology is a relatively new development within the laboratory setting, lab managers are quick to realise that digital transformation is essential in optimising workflows and productivity. In 2018, 70% of labs were reported to have a digital lab strategy in place or were working towards one1– a number that we can only expect to have significantly increased since then.

Significant effort has taken place in laboratories over the past two decades or more, which has delivered substantial benefits. This effort has been focused on the key lab workflows and the matching informatics systems such as CDS, LIMS, ELN, LES and SDMS, to mention a few. The next decade needs to build on this success to create a true digital laboratory.

Digital labs of the future: what can we expect?

Digital lab transformation is more than just implementing informatics systems, it involves taking these systems and pushing them a step further. For example, a lab could connect instruments bi-directionally to LIMS or ELN, but digital lab transformation would also facilitate online monitoring of instrument status, automatic ordering of consumables, reserving instrument time, auto-tracking utilisation and the use of telemetry data to predict faults before they happen.

A digital lab may also utilise a feature rich LIMS, ELN or LES that enables collation and review of all results for an experiment, but a digitally transformed lab would also be able to collate results across potentially several LIMS and ELNs throughout an organisation. This would allow the promotion of internal and external collaborations, enabling the ‘science later’ paradigm of cross team, cross technique and cross experiment data mining. This, in turn, will progress artificial intelligence and machine learning.  

Overall, a digital transformation is more than just providing scientists with the means to spend more time on actual science. It provides the complete toolset of a lab wherever a scientist may be, whether that is in the lab itself, in an off-site office, in a café or even at the kitchen table.

At present, even top laboratories face problems with a lack of modernisation, and this is a problem that is slowly trickling down to smaller labs that are starting to face similar challenges. If we continue to drive forward with the help of innovative technology, we could expect to see many labs becoming more efficient, more supportive of science and more reliable than ever before.

However, to do this, it is up to laboratory leaders to have a clear vision of where they see their lab going. It is hard to transform any business by only doing little bits, so it is up to the higher levels of lab personnel to decide what steps to take to ensure that their labs are working at optimal capacity and potential. This is where Scimcon can help.

How can Scimcon help to revolutionise your lab?

Scimcon is proud to offer a range of digital lab services to assist in digitising a lab, many of which are outlined in our introductory blog. We are also able to help labs go that step further, with our collective wealth of experience in the lab, both as scientists and project leaders. Whether it is the development of the strategy, the running of the programme, or providing resources and leadership for your projects, Scimcon can help you understand what you want to achieve, and how to reach it.

To find out more about types of projects we support, and how we can help you to transform your lab, get in touch.

Reference:

1 ‘Despite steady growth in digital transformation initiatives, companies face budget and buy-in challenges’, https://www.zdnet.com/article/survey-despite-steady-growth-in-digital-transformation-initiatives-companies-face-budget-and-buy-in/

In order to work as intended, this site stores cookies on your device. Accepting improves our site and provides you with personalized service.
Click here to learn more