As required by Utah Code (62A-4a-118) we have completed our review of the Bureau of Services Review (BSR) within the Department of Human Services. Utah's child welfare system has been subject to much public interest and legislative reform. As part of that reform effort, the Legislature established the Bureau of Services Review within the Department of Human Services to monitor compliance with legislative and legal mandates. Our office conducted a previous audit of BSR (Report #95-07). In our prior review we found that BSR was doing a good job of monitoring the Division of Child and Family Services (DCFS) but was not specifically reporting on how well the system was protecting children from abuse or neglect and preserving families.
In this audit, the Legislature wanted us to not only report on BSR's performance but also on whether the state's child welfare system is improving. Our current audit found that BSR is accurately reporting compliance in most cases with requirements of the David C. et al. v. Leavitt lawsuit settlement agreement, current legislation, and DCFS policy. BSR has also incorporated performance aspects into their review as recommended in our previous audit. However, we note areas where BSR's ratings need more definitions and clarifications. Finally, we found that some child welfare system outcome measures have shown improvement since our 1993 audit of that system (Report #93-06).
The following summary identifies the findings of this audit:
BSR Appears to Effectively Monitor Compliance. Our evaluation of BSR's 1996 review concluded that BSR appears to be effective in monitoring most cases for compliance with child welfare requirements by DCFS. To make this conclusion, we evaluated BSR's methodology for developing their review instruments (assessment questionnaires), for selecting the cases to be reviewed and for rating cases. We found that BSR's methodology for developing the assessment questionnaire and selecting cases appears sound. In addition, we found that BSR accurately applied assessment questionnaires to case file information.
BSR Appears to Effectively Monitor Performance. Our evaluation of BSR's 1996 review also concluded that in most cases BSR appears to be effective in monitoring for caseworker performance in achieving the mission of DCFS. BSR's 1996 report indicates that DCFS caseworkers are performing at acceptable levels in most aspects of providing child welfare services. However, BSR did identify areas where caseworkers could improve. Unlike the compliance section, BSR's review of performance is not specifically related to the settlement agreement, legislation, or policy but resulted from our 1995 review of BSR. By incorporating performance based questions into the assessment questionnaires, BSR has expanded the review to include a conclusion regarding whether the child was protected, if adequate services were provided to preserve the family, and whether a permanent family was provided for foster children in a timely manner. In this regard, BSR's review is now used for more than just scoring compliance/noncompliance. It is now a tool that assists regional managers and supervisors in training caseworkers.
BSR Accurately Reports Problems In DCFS. Besides reviewing the case files appropriately, we also found no evidence to suggest that BSR fails to report noncompliance or poor performance by DCFS caseworkers. The high level of agreement between our review and BSR's review of case files indicates that BSR reviewers are finding the same deficiencies we found.
BSR Reviews Can Be Strengthened. In our opinion, some fine-tuning of definitions and methods used in scoring BSR's assessment questionnaires would strengthen the bureau's review. In the compliance review, better definitions are needed and BSR reviewers need training on when to apply each response. We also found cases that did not technically meet compliance standards, but were still given points for compliance because of the current method used to score some compliance areas. In the performance review, most of our disagreements occurred where the rating did not match the shortcomings documented in the case. We found instances where BSR reviewers documented serious problems with a caseworker's performance, but were hesitant to assign a "Poor" rating. We believe if BSR would develop a rating classification which identifies cases meeting the minimal level of acceptability but needing improvement, their review process would be strengthened. Lastly, a few case files were discovered missing in DCFS offices and were, therefore, unavailable for BSR to review. We recommend that BSR develop a section in their next review to report missing case files.
System Outcome Measures in DCFS Suggest Improvement has Been Made in Some Areas. Our evaluation of DCFS indicates that significant progress has occurred in implementing the recommendations made in our 1993 review of the state's child welfare system. Of the 37 recommendations presented in our 1993 audit of Utah's Child Welfare System, we found that all but three have been adequately incorporated into policy or otherwise implemented. Two of the three exceptions are related recommendations that deal with conducting home visits during the CPS investigation. These two recommendations are addressed in policy, but not adequately. The third exception is a recommendation to train new workers on the elements needed to substantiate a referral. Despite incorporating our recommendations into policy and training, there are nonetheless some instances where the policy is not being observed or practiced. In addition to reviewing implementation of our 1993 audit recommendations, we also compared outcome measures from our 1993 report with similar outcome measures in BSR's 1996 report. Six outcome measures were selected for comparison and the results suggest that improvement has been made with the level of compliance in each area.