In order to work as intended, this site stores cookies on your device. Accepting improves our site and provides you with personalized service.
Click here to learn more
Click here to learn more
Our team at Scimcon is made up of a talented group of interesting individuals – and our newest recruit Ben Poynter certainly does not disappoint!
Ben joined our Scimcon team in July 2022 as an associate consultant, and has been working with the lab informatics specialists to get up to speed on all things Scimcon. We spoke to Ben about his experience so far, his interests, background, and what he hopes to achieve during his career as an informatics consultant.
So, I studied Biomedical Science at Sheffield Hallam University, which was a four-year course and allowed me to specialise in neuroscience. During my time at university, I created abstracts that were presented in neuroscience conferences in America, which was a great opportunity for me to present what I was working on. My final year dissertation was on bioinformatics in neuroscience, as I was always interested in the informatics side of biomedical science as well.
Once COVID hit, I moved into code work, and worked in specimen processing, and then as a supervisor for PerkinElmer who were undertaking some of the virus research. When things started to die down, I began working for a group called Test and Travel (not the infamous Track and Trace initiative, but a similar idea!). I started there as a lab manager, training new staff on lab protocols for COVID-19, and then a month into that I started working more on the LIMS side – which is where I ended up staying. I wrote the UAT scripts for 3 different companies, I performed validation on the systems, I would process change controls. I then moved to Acacium as LIMS lead there, so over the course of my career I’ve worked with a number of LIMS and bioinformatics systems, including LabWare 7, LIMS X, Labcentre, WinPath Enterprise, and Nautilus (ThermoFisher Scientific).
In the early stages, I would have to say it was when Jon and Dave led my first interview, and Jon asked me a question I hadn’t been asked in an interview setting before. He asked me ‘who is Ben Poynter?’. The first time I answered, I discussed my degree, my professional experience with LIMS and other informatics systems, and how that would apply within Scimcon’s specialism in lab informatics consultancy. Then he asked me again and I realised he was really asking what my hobbies were, and how I enjoyed spending my free time. Since starting at Scimcon, I’ve been introduced to the full team and everyone is happy to sit and talk about your life both inside and outside of work, which makes for a really pleasant environment to work in. Also, it seems as though everyone has been here for decades – some of the team have even been here since Scimcon’s inception back in 2000, which shows that people enjoy their time enough to stay here.
I’ve been given a really warm welcome by everyone on the team, and it’s really nice to see that everyone not only enjoys their time here, but actively engages with every project that’s brought in. It’s all hands on deck!
So, my main hobbies and interests outside of work are game design, as well as gaming in general. I run a YouTube account with friends, and we enjoy gaming together after work and then recording the gameplay and uploading to YouTube. We are also working on a tower defence game at the moment, with the aim to move into more open world games using some of the new engines that are available for game development.
In addition to gaming and development, I also enjoy 3D printing. I have a 3D printer which allows me to design my own pieces and print them. It’s a bit noisy, so I can’t always have it running depending on what meetings I have booked in!
Technology is a real interest of mine, and I’m really fortunate to have a role where my personal interests cross-over into my career. The language I use for game design is similar to what I work with at Scimcon, and the language skills I’ve developed give me a fresh perspective on some of the coding we use.
At the moment, I’m working on configuration for some of the LIMS systems I’ll be working with at customer sites, which I really enjoy as it gives me the chance to work with the code and see what I can bring to the table with it. Other projects include forms for Sample Manager (ThermoFisher Scientific), making it look more interesting, moving between systems, and improving overall user experience. It’s really interesting being able to get to grips with the systems and make suggestions as to where improvements can be made.
My first week mainly consisted of shadowing other Scimcon lab informatics consultants to get me up to speed on things. I have been working with the team on the UK-EACL project, which has been going really well, and it’s been great to get that 1-2-1 experience with different members of the team, and I feel like we have a real rapport with each other. I’ve been motoring through my training plan quite quickly, so I’m really looking forward to seeing the different roles and projects I’ll be working on.
I’d really like to get to grips with the project management side of things, and also love to get to grips with the configuration side as well. It’s important to me that I can be an all-round consultant, who’s capable at both managing projects and configuration. No two projects are the same at Scimcon, so having the capability to support clients with all their needs, to be placed with a client and save them time and money, is something I’m keen to work towards.
For more information about Scimcon and how our dedicated teams can support on your lab informatics or other IS projects, contact us today.
The challenges of implementing ePRO – part two?Clearly good cell network or WIFI connectivity is of primary importance when thinking about using ePRO in a clinical study. Sites should conduct a feasibility activity in their local vicinity during the study initiation phase to understand if cell network strengths are good enough for ePRO and if not, if there is likely to be a large population of subjects that have home WIFI. For remote locations with possible issues with either the cell network coverage or even with the availability of electricity, these sites should be reconsidered before a decision is made to use ePRO.
However, even with bad cell network coverage, it is possible to conduct a study with ePRO by ensuring the correct messaging appears to the subjects on the device. All ePRO systems allow the subjects to access and complete their diaries on a daily basis without cell or WIFI connectivity. The data entered by the subject is stored on the device until connectivity is established. There are circumstances where the site may need to be made aware of certain responses in real time in order to maintain subject safety. If connectivity is established and the data is sent in real time, the site staff may receive an alert from the system asking them to contact the participant. In order to ensure participant safety is maintained it is good practice to alert the participant to contact the sites / study doctor if certain responses in the diary are provided, such as if the participant required emergency medical attention.
As mentioned earlier in an earlier instalment, translation is an aspect not typically associated with clinical systems such as EDC. Additionally, because the ePRO devices are in the hands of the participants, the screens, in their local languages, must be submitted to the local ethics committees within the countries that the study is to take place in.
It should not be forgotten that paper PRO is also subject to a translation process. However, the additional complexity with ePRO is the need to apply the translated text to the software in order to generate the questionnaire screens. This involves multiple review and update rounds between the ePRO vendor and the translation vendor which increases time, effort and therefore cost. Once the screens are generated there may be further review rounds during the in-country review between the local sponsor representatives, ePRO vendor and translation vendor. All of these review rounds increase the timescales for an activity that is already time critical.
Often the date for submission to the ethics committee is set during the planning phase of the study by the sponsor study team. If the submission date is missed it could result in a delay to the study start. Consequently, it is important to ensure clear timelines are in place between agreeing the ePRO requirements, which will decide what is displayed on each of the screens, and submitting the screen report to the ethics committees. The ePRO vendor will need to be made aware of these timelines at the earliest opportunity. It can take more than 12 weeks for the completion of the translations resulting in the creation of the finalized screen reports. These timelines are difficult to manage if the process includes an in-country review round which allows for local country representatives from the sponsor to review the translations. It is important to make clear to the reviewers what it is that these stakeholders are reviewing. Translation is not an exact science. There are many ways of writing the same sentence. As the translator is the trained expert in translations it should be left to them to choose the most appropriate wording in the local language which most faithfully represents the English version. The in-country review, if indeed one is required, should only be conducted to ensure the correct screens are displayed in the screen reports. Allowing the in-country reviewer to make suggestions on preferred wording risks multiple back and forth review rounds, increasing timelines and jeopardizing study start dates.
As briefly mentioned in part one of this series, possibly the most crucial aspect when introducing a new system or process to a user base is change management. If you do not bring your users along the journey with you, you are less likely to gain their acceptance.
How does this manifest itself? With ePRO there is plenty of opportunity for issues to arise, from delays in getting devices to sites, to errors in the software, to usability and connectivity issues. All of which can affect the investigators ability to get on with their daily work. If you have not put in place a good change management process you will find the investigators very quickly become disenchanted with the system and even the smallest of issues will become magnified, resulting in escalations to the sponsors senior management team.
It is important from the outset to set the stage with the investigators. Why are we using ePRO? What benefits does it bring to the sponsor? It may require more work on behalf of the investigator which will need to be compensated for. It is also about setting expectations. Things will go wrong, issues will need to be resolved, backup processes may need to be utilized, but the investigator must be aware that they have the support when needed and how they can access that support.
When implementing ePRO thought should go into understanding how to improve the investigators experience. For example, investigators can be working on multiple studies at the same time for different sponsors all using ePRO, so adding labels onto the packaging of shipped devices so that investigators can quickly store them together is a quick win.
As a sponsor it may also be necessary to implement an additional layer of support for the study team. When issues arise the investigator will contact the vendor helpdesk. Often the helpdesk are unable to provide an immediate remedy so the investigator will then contact the sponsor’s study team representatives. The additional layer of support sits between the study team and the vendor, collating issues, communicating technical information back and forth in a manner that is easily digestible and holding the ePRO vendor to account. This relieves the frustration experienced within the sponsor’s study team and reduces the likelihood of escalation.
It is important to remember that in accordance with ICH GCP guidance section 5.2.1 [1] the sponsor may transfer trial-related duties to a vendor, however the ultimate responsibility for the quality and data integrity of the trial data always resides with the sponsor. In order to ensure data is collected, stored and transferred in accordance with ICH-GCP guidance the sponsor will need a suitable oversight strategy of the vendor’s processes. This may, at a minimum, include a qualification activity which involves auditing the ePRO vendor on a regular basis. It may also invoke some internal validation work to ensure, as the sponsor, your own organisation’s technical and study specific requirements for clinical systems are met.
It is not recommended for the sponsor to leave the responsibility of vendor oversight to the individual study teams. The study teams may not have the experience or technical know-how necessary to understand the challenges and considerations discussed earlier in this blog.
We would recommend a two stage process.
Stage one is the qualification of the vendors system by a centralized team including stakeholders from the data management and computer system quality departments. Once qualified, the vendor’s system can be classed as a platform that can subsequently be used on multiple studies.
Stage two is to have experienced stakeholders conduct study specific User Acceptance Testing (UAT) in order to test the many scenarios and nuances associated with ePRO and to ensure the system meets the requirements of the study Protocol. Once implemented and running live on a study it is far more difficult to update the system than finding and fixing the issue during the implementation phase.
However, UAT is the very least the Sponsor must do as part of the overall study specific validation activity (Stage two). The Sponsor’s quality management system (QMS) may also mandate that the implementation of any clinical system (which would include ePRO) requires internal validation documentation to supplement the vendor’s validation package. This can include documents such as a sponsor Validation Plan, Requirements Specification, Risk Assessment, Traceability Matrix and Validation Report.
ePRO vendors will often offer to provide a service to create UAT script on behalf of the Sponsor, however this is not recommended as a misunderstanding of requirements by the vendor may also manifest itself in the scripting. This can result in the script passing when executed when in fact it does not meet the requirements of the Sponsor.
For these reasons, having experienced validation professionals generating the deliverables and executing the testing is recommended.
The two parts of this blog have detailed, at a high level, many of the challenges that can be experienced during the implementation and use of ePRO. These challenges are summarized below:
These challenges, and others, often create frustration for the investigators and study teams, and in the worst case scenarios, studies can be put on hold or stopped completely, costing millions of dollars and preventing a product from going to market.
A mechanism to help mitigate, or at the very least support the investigators and study teams through, many of these challenges is for the sponsor to put in place a centralized ePRO team within their organization. This team, consisting of internal sponsor stakeholders or perhaps supplementing Scimcon resources into the team, would be responsible for the qualification of the ePRO vendor, the strategic implementation of libraries and standards and for supporting the implementation of ePRO study-specifically, providing help with creating validation deliverables and executing UAT, and being a second level of support during the conduct phase of a study. ePRO is fast becoming the standard and it is worthwhile investing early on in an internal team to ensure success.
To discuss how Scimcon can support your organization with the implementation of ePRO, please contact one of our consultants for an informal conversation.
[1] ICH harmonised guideline integrated addendum to ICH E6(R1): Guideline for Good Clinical Practice ICH E6(R2) ICH Consensus Guideline, https://ichgcp.net
Planning for successful User Acceptance Testing in a lab or clinical setting?User Acceptance Testing (UAT) is one of the latter stages of a software implementation project. UAT fits in the project timeline between the completion of configuration / customisation of the system and go live. Within a regulated lab or clinical setting UAT can be informal testing prior to validation, or more often forms the Performance Qualification (PQ).
Whether UAT is performed in a non-regulated or regulated environment it is important to note that UAT exists to ensure that business processes are correctly reflected within the software. In short, does the new software function correctly for your ways of working?
You would never go into any project without clear objectives, and software implementations are no exception. It is important to understand exactly how you need software workflows and processes to operate.
To clarify your needs, it is essential to have a set of requirements outlining the intended outcomes of the processes. How do you want each workflow to perform? How will you use this system? What functionality do you need and how do you need the results presented? These are all questions that must be considered before going ahead with a software implementation project.
Creating detailed requirements will highlight areas of the business processes that will need to be tested within the software by the team leading the User Acceptance Testing.
Requirements, like the applications they describe, have a lifecycle and they are normally defined early in the purchase phase of a project. These ‘pre-purchase’ requirements will be product independent and will evolve multiple times as the application is selected, and implementation decisions are made.
While it is good practice to constantly revise the requirements list as the project proceeds, it is often the case that they are not well maintained. This can be due to a variety of reasons, but regardless of the reason you should ensure the system requirements are up to date before designing your plan for UAT.
A common mistake for inexperienced testing teams is to test too many items or outcomes. It may seem like a good idea to test as much as possible, but this invariably means all requirements from critical to the inconsequential are tested to the same low level.
Requirements are often prioritised during product selection and implementation phases according to MoSCoW analysis. This divides requirements into Must-have, Should-have, Could-have and Wont-have and is a great tool for assessing requirements in these earlier phases.
During the UAT phase these classifications are less useful, for example there may be requirements for a complex calculation within a LIMS, ELN or ePRO system. These calculations may be classified as ‘Could-have’ or low priority because there are other options to perform the calculations outside of the system. However, if these calculations are added to the system during implementation, they are most likely, due to their complexity, a high priority for testing.
To avoid this the requirements, or more precisely their priorities, need to be re-assessed as part of the initial UAT phase.
A simple but effective way to set priority is to assess each requirement against the risk criteria and assign a testing score. The following criteria are often used together to assess risk:
Once the priority of the requirements has been classified the UAT team can then agree how to address the requirements in each category.
A low score could mean the requirement is not tested or included in a simple checklist.
A medium score could mean the requirement is included in a test script with several requirements.
A high score could mean the requirement is the subject of a dedicated test script.
A key question often asked of our team is how many test scripts will be needed and in what order should they be executed? These questions can be answered by creating a Critical Test Plan (CTP). The CTP approach requires that you first rise above the requirements and identify the key business workflows you are replicating in the system. For a LIMS system these would include:
Sample creation, Sample Receipt, Sample Prep, Testing, Result Review, Approval and Final Reporting.
Next the test titles required for each key workflow are added in a logical order to a CTP diagram, which assists in clarifying the relationship between each test. The CTP is also a great tool to communicate the planned testing and helps to visualise any workflows that may have been overlooked.
Now that the test titles have been decided upon, requirements can be assigned to a test title and we are ready to start authoring the scripts.
There are several different approaches to test script formats. These range from simple checklists, ‘objective based’ where an overview of the areas to test are given but not the specifics of how to test, to very prescriptive step by step instruction-based scripts.
When testing a system within the regulated space you generally have little choice but to use the step by step approach.
Test scripts containing step by step instruction should have a number of elements for each step:
A typical example is given below.
However, when using the step by step format for test scripts, there are still pragmatic steps that can be taken to ensure efficient testing.
Data Setup – Often it is necessary to create system objects to test within a script. In an ELN this could be an experiment, reagent or instrument, or in ePRO a subject or site. If you are not directly testing the creation of these system objects in the test script, their creation should be detailed in a separate data setup section outside of the ‘step by step’ instructions. Besides saving time during script writing, any mistakes made in the data setup will not be classified as script errors and can be quickly corrected without impacting test execution.
Low Risk Requirements – If you have decided to test low risk requirements then consider the most appropriate way to demonstrate that they are functioning correctly. A method we have used successfully is to add low risk requirements to a table outside of the step by step instructions. The table acts as a checklist with script executers marking each requirement that they see working correctly during executing the main body of step by step instructions. This avoids adding the low requirements into the main body of the test script but still ensures they are tested.
Test Script Length – A common mistake made during script writing is to make them too long. If a step fails while executing a script, one of the resulting actions could be to re-run the script. This is onerous enough when you are on page 14 of a 15 page script. However, this is significantly more time-consuming if you are on page 99 out of 100. While there is no hard and fast rule on number of steps or pages to have within a script, it is best to keep them to a reasonable length. An alternative way to deal with longer scripts is to separate them into sections which allows the option of restarting the current block of instructions within a script, instead of the whole script.
An important task when co-ordinating UAT is be fully transparent about which requirements are to be tested and in which scripts. We recommend adding this detail against each requirement in the User Requirements Specification (URS). This appended URS is often referred to as a Requirements Trace Matrix. For additional clarity we normally add a section to each test script that details all the requirements tested in the script as well as adding the individual requirement identifiers to the steps in the scripts that test them.
UAT is an essential phase in implementing new software, and for inexperienced users it can become time-consuming and difficult to progress. However, following the above steps from our team of experts will assist in authoring appropriate test scripts and leading to the overall success of a UAT project. In a future blog we will look at dry running scripts and formal test execution, so keep an eye on our Opinion page for further updates.
