Return on Time Investment: Research Resources
Leslie Nicoll, Sulochana Naidoo, Qing Yang
Nurse Author & Editor, 2021, 31(1), 6
In an earlier article,1 the concept of “return on time investment” was introduced (ROTI) to consider the time it takes to learn and use a software program or application, balanced against the hours saved for your work or project. For people in academia, or those who anticipate a career that will involve writing on a regular basis, having a strong knowledge of a word processing program and a citation manager is essential. These programs have a high ROTI—that is, it is worth the time investment to learn to use them well, as they will save you many hours of effort in the long run.
What if your career will involve active engagement in research activities? Or perhaps you are a student, embarking on a PhD dissertation or DNP capstone project. These types of projects usually involve data-related activities—typically data collection, management, and analysis. There are myriad applications for each of these tasks. Sorting out the pros and cons of each—and balancing that against cost and ROTI—can be overwhelming. In this article, we will provide a framework that can be used as a starting point for thinking through some of these issues. Note that a comprehensive review of the various applications available is beyond the scope of this article. Our goal, instead, is to give some representative examples and provide criteria that you can use to guide decision-making.
Decision Making Algorithm
Figure 1 is a starting point for evaluation. Note that your goal should always be to increase your efficiency and accuracy. This is balanced against availability, cost, frequency of use, and the learning curve of the application you plan to use. All of this taken together will help you determine your ROTI.
In contrast to the applications discussed previously (word processors, citation managers, and graphical design programs) which are widely available and may be free, low cost, or affordable, programs used for research are less ubiquitous and often expensive. Therefore, availability and cost are both important considerations and may drive your decision making. If your employer has an institutional license for program “X,” then you need a really good reason to buck the system to use program “Y.” Institutionally licensed programs may also include training options and support. On the other hand, if nothing is available to you locally, then cost may very well be the deciding factor as to which program you choose, which is why it is the first column in Figure 1.
Form Development and Data Collection
There are several applications that can be used to create forms for data collection, such as Survey Monkey, Qualtrics, JotForm, and Wufoo (which is owned by Survey Monkey). All these applications work basically the same way and allow you to create forms quickly and easily. They have pre-set fields such as name, address, email address, and phone number, which you can drag and drop onto your form. They also all have branching logic, which means that if a person answers a question a certain way (eg, yes or no) other fields for answers may be displayed or hidden. Most include a feature to integrate with a payment system, such as Stripe or PayPal. This is likely not important for research purposes, but if you become proficient with a form application, you may find yourself developing forms for conference registrations or to sell tickets to an event. Once a form has been developed, it can be embedded on a website or a link for completion can be sent to by email to identified participants.
In our experience, these programs are quite intuitive and easy-to-use—in fact, developing a questionnaire or survey can be fun. That said, it is important to keep in mind good survey principles. If you are developing something from scratch, you want to be thinking of pilot testing your survey and establishing reliability and validity. If you are using a pre-existing survey, make sure you have permission to reproduce it in another format and use it in your study.
Many institutions have licenses for Qualtrics and Survey Monkey, so by default, you may be using one of those applications. But if you need to search further afield, the good news is that you will not necessarily be deterred by the cost as prices vary widely. Many applications have a free trial period to let you test the program; after that they are priced per month which means you can cancel your subscription when your project is done without being tied into an expensive long-term contract. Different price points may be determined by the number of questions on a survey or the number of responses you can receive in a month. Payment integration and HIPAA security are usually features offered at slightly higher price points. With free trial periods and flexible payment options, you should be able to find something that works for your particular situation—but you need to do your homework to find the best solution. Spend some time doing your research on this issue!
One application you may have heard a lot about (or be familiar with) is Research Electronic Data Capture (REDCap).2 This is a web-based application to collect data for clinical research, create databases, and produce simple reports, graphs, and descriptive statistics. It was developed in 2004 at Vanderbilt University by a group of clinical researchers who needed a secure, HIPAA compliant resource for data collection. REDCap is a little different from the previously discussed form development programs in that it is data driven and has a different underlying architecture from software that generates forms.
REDCap has many positive features for data collection and management:
- It is HIPAA compliant and highly secure, so it is an excellent choice if your study contains sensitive data.
- It can be easily organized by different instruments or case report forms (demographic, medical history, and so on). Many clinical research related instrument forms are available to help orient you to the database building process.
- While REDCap is useful for collecting survey data from study participants, it can also be used for data entry by study team members.
- If your study has a longitudinal design, that is, collecting data on the same variables at different times, it can be easily incorporated in the REDCap database design. Doing so ensures that all the records collected at different times can be linked appropriately and accurately.
- You can automatically produce an easy-to-read data dictionary or code book.
- You can produce simple reports, graphs, and descriptive statistics.
- Data from REDCap can be exported into a wide range of formats that are used by different statistical software programs, including: EXCEL, IBM SPSS Statistics, R, STATA, and SAS.
Things you need to consider before using:
- REDCap is free, but to use it, there needs to be a license agreement between your institution and REDCap. There are over 3200 institutions worldwide that have license agreements. You should check if it is available at your institution before you consider using REDCap for data collection, storage, and analysis. This licensing structure, unfortunately, makes REDCap inaccessible to those who are not at an institution with a license—it is available to non-profits only. Keep this in mind as you consider your options.
- REDCap is intuitive to use, but there is a learning curve. The best way to get started is watch from videos that are offered by REDCap (https://projectredcap.org/resources/videos/). You should be able to get started with building a basic database in a few days. More advanced functions may take longer to learn over time.
Compared to other data collection resources, REDCap has many appealing features for researchers. If you have a working knowledge about study design, have also used other data collection resources, and have it available to you at your institution, then it is worth it to learn REDCap which should prove to be a good ROTI. On the other hand, if it is not available to you then you may need to consider another option, as previously discussed.
Data Analysis: Qualitative Data
If your research data frequently includes open-ended responses to interviews and survey questions, or other forms of text such as journal articles, you value the richness and comprehensiveness of qualitative data. You probably also appreciate how cumbersome and time-consuming the subjective process of manually organizing and analyzing qualitative data can be. If you value time as a precious commodity, then using software for qualitative data analysis is highly recommended. Such software helps to keep data organized, enables more methodical and effective analysis, and consequently yields richer insights into the phenomena being studied.
The NVivo software package (QSR International, Melbourne, Australia) is one of the most widely used qualitative and mixed-methods data analysis tools in academia and healthcare. It offers the convenience of seven interface languages and audio transcription services in 28 languages.
NVivo facilitates qualitative and mixed methods research by allowing you to:
- Import text, image, audio, and video data from multiple sources including reference management software.
- Organize data from multiple sources within a project file.
- Code data to specific nodes to better recognize inherent trends and themes.
- Record insights and interpretations in memos, and link them to specific data, to create the basis for your written project.
- Query and search data and save the results.
- Create data visualizations such as word clouds, word trees, concept maps and graphs to illustrate trends and relationships.
- Work collaboratively in teams of up to five people with options to upgrade for larger teams.
Qualitative data analysis is typically an immersive process. It is very likely, therefore, that researchers will want to use NVivo themselves rather than handing off the analysis to a consultant. Thus, having a working knowledge of NVivo before using it is essential. This knowledge may be gained by enrolling in workshops offered at your institution (if there is an institutional subscription), tutorials offered by QSR international, and paid or freely available online tutorials. You may also have the opportunity to become familiar with NVivo through collaboration with someone else—for example, being a graduate assistant on a faculty member’s qualitative research project.
NVivo will prove to be intuitive and user friendly for researchers with an understanding of how to process qualitative data manually, and who are familiar with the interface of commonly used computer programs. Learning how to utilize its various unique features and applications requires a moderate initial time investment. The learning curve will continue with use but will flatten considerably.
NVivo is sold through both institutional and individual licenses; the cost for a non-academic, individual purchase is approximately $1250. Learning how to use NVivo is undoubtedly a high gain ROTI for qualitative and mixed-method researchers.
Data Analysis: Quantitative Data
Nowadays, you almost always need to collect quantitative data in your research. That means you will have to perform some level of statistics, from just describing your sample characteristics by using descriptive statistics, to hypothesis testing between two variables by using bivariate inferential statistics, and to trying to explore more complex relationships among multiple variables by using advanced statistical models. Before doing any of this, you may also need to do some simple data cleaning and data management, such as taking care of missing data or outliers, and creating new variables. Statistical software is crucial in all of these processes.
There are many statistical software programs on the market and those most frequently used by nursing researchers are IBM SPSS Statistics (IBM Inc., Armonk, NY, USA), SAS (SAS Institute, Cary, NC, USA), and Excel (Microsoft, Inc., Redmond, WA, USA). You may need to have taken at least one statistical course so you can try to do some descriptive and bivariate inferential statistics yourself. For more complex inferential statistics or models, you may need more systematic training or consultation with a statistician. The table below compares three of the most frequently used software programs. Use this to decide which one to choose according to what types of statistics you are trying to generate. You can use this table together with Figure 1 to make the best ROTI decision for yourself.
Table 1. Comparison of Statistical Programs
|Cost and availability||Data Management||Descriptive Statistics
|Inferential Statistics||Modeling Techniques||Learning Curve|
|Excel||Included in Windows Office||Fair||Yes||Some||No||Easy to learn|
|SPSS||Expensive but academic discounts can make the price reasonable for students. May be available through an institutional license.||Good||Yes||Yes||Most||Moderate:
Can self-teach by using online resources if you already have some statistical knowledge. Many stats courses include using SPSS as part of the course.
|SAS||Expensive, but sometimes free if your institution already has a license||Excellent||Yes||Yes||Yes||Very steep:
Recommend a formal training program
When you embark on a project that involves data collection, storage, manipulation, and analysis, you need to think very carefully about how you will manage each step. You could ask participants in study to complete a questionnaire using pencil and paper, but what would be your next step? Similarly, if they complete a survey electronically, where do the data go from there and how will they be stored and analyzed? If you interview people and tape record the interviews, who will do the transcription? How will interviews move from a conversation log to a program such as NVivo? You need visualize each of these steps, how the work will be managed, and who will do it. Taking the time to learn a program such as NVivo or SPSS might be a worthwhile investment in yourself, because you can use this knowledge to consult with others and maybe even become a research collaborator on a variety of projects. On the other hand, if you are a conceptual “big thinker” and not a detail-oriented analyst, then your time might be better used in another way, while you use financial resources to hire expert consultants for handle this aspect of the project. Whatever path you choose, what is most important is to have a clear understanding of the issues at play and use this knowledge to make an informed decision about your ROTI.
- Nicoll LH. Return on Time Investment: Writing Resources. Nurse Author Ed. 2021;31(1):5. Accessed March 12, 2021. https://naepub.com/wp-content/uploads/2021/03/2021-31-1-5-Nicoll.pdf
- Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)–a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377-381. doi:10.1016/j.jbi.2008.08.010
About the Authors
Leslie Nicoll, PhD, MBA, RN, FAAN, is Editor-in-Chief of Nurse Author & Editor and CIN: Computers, Informatics, Nursing. She is co-author of The Editor’s Handbook, 3rd ed. which was published in July 2019. You can reach Leslie via the contact form on this site, and she will respond to you directly.
Sulochana Naidoo, PhD, MS, serves as the Associate Director, Global Education for Duke- NUS. In this role she oversees and supports the medical education collaboration between Duke University School of Medicine and Duke-NUS Medical School (Singapore).
Qing Yang, PhD, is an Associate Professor and Biostatistician at Duke University, School of Nursing. She teaches statistical courses to PhD students and DNP students. She also serves as statistical co-investigator on many NIH and DOD funded grants.
Copyright 2021: The Authors. May not be reproduced without permission.
Journal Complication Copyright 2021: John Wiley and Son Ltd.