dhs_seal&femabluetext_new Risk MAP MIP USER GUIDANCE DFIRM UPLOAD GUIDANCE Automated QA review process through the MIP and DFIRM Verification Tool (DVT) Table of Contents I. OVERVIEW..............................................................................................................4 II. GENERAL GUIDELINES.............................................................................................5 III. UPLOADING DATA OPTIONS....................................................................................81. MIP Studies Workflow.....................................................................................................................................8 Highlights........................................................................................................................................................................8Process..............................................................................................................................................................................82. Citrix........................................................................................................................................................................9 Highlights........................................................................................................................................................................9Process..............................................................................................................................................................................93. MIP Studies Workflow – Mail Data to the Data Depot......................................................................10 Highlights......................................................................................................................................................................10Process............................................................................................................................................................................104. DFIRM Tools......................................................................................................................................................10 Highlights......................................................................................................................................................................10 Process............................................................................................................................................................................11 IV. FOLDER STRUCTURE AND INCLUDED FILES............................................................12 V. TIPS FOR FASTER PROCESSING..............................................................................14 VI. ADDITIONAL MIP FUNCTPROCESS....... IONALITY RELATED TO THE AUTOMATED QA REVIEW Tools & Links T ..............................................................................................................15est SReporting Tool......... ubmissions..........................................................................................................................15................................................................................................................................................15DVT Administration................................................................................................................................................15 II. TRACKING SUBMISSIONS......................................................................................16 V I. OVERVIEW This document is intended to provide guidance on the options for uploading DFIRM submissions on to the MIP, and submitting them for automated QA review through the DFIRM Verification Tool (DVT). Following these guidelines will lead to increased efficiency in receiving results for and getting through the automated QA review process. There are four options for uploading submissions to the MIP – 1. Upload through the MIP Submit Data Files portlet 2. Upload files to the J drive through Citrix 3. Mail data to the Data Depot (only method for submissions above 2 GB) 4. Use DFIRM Tools Each option is applicable for all three DFIRM database-related tasks in the MIP, including Develop Draft DFIRM DB, Produce Preliminary Map Products, and Produce Final Map Products. Included in this document are the correct procedures for submitting files, and information on how to resubmit data and track submission status. Also included are descriptions of the correct folder structure and which files to include with each DFIRM submission for automated QA review. If you encounter a scenario that is not covered in the documented options, please contact your MIP Champion and/or Black Belt for assistance. The list of MIP Champions and Black Belts can be found in the MIP Help Escalation Plan document on hazards.fema.gov >MIP User Care > Guides & Documentation > MIP Help Escalation Plan REMINDER The MIP-DVT tool is an aid in QA/QC and does not take the place of visual quality control checks that are performed by each of the PTS Firms. Ultimate responsibility for the quality of the DFIRMs lies with each of the PTS Firms. II. GENERAL GUIDELINES 1. Recommended file size limits for each option: – Upload data through the MIP portlet (Submit Data Files): upload individual files and/or zip files smaller than 1GB and with fewer than 8000 individual files (at one time, in one upload submission) – Upload through Citrix: upload individual files and/or zip files smaller than 1GB and fewer than 8000 individual files (at one time, in one upload submission) – Mail data to Data Depot: recommended for file packages larger than 1 GB . Reminder: allow two weeks for mail and upload process – DFIRM Tools: depends on the upload process, see DFIRM Tools section 2. The validation process begins when the user clicks Validate Contents, no matter whether the files were uploaded through the MIP, Citrix or the Data Depot. – Do not click Validate Contents until all files needed for Metaman and DFIRM DB validation have been uploaded to the J drive and do not resubmit any files, through any submission process, until results of the previous submission are returned. 3. Metadata must pass Metaman validation before the DFIRM DB is automatically submitted to the MIP-DVT engine for review. 4. Plan ahead! Take into consideration your file submission size (to determine whether the submission can be submitted through the MIP or must be mailed to the Data Depot) and the turnaround times for the DVT review, and plan time into your schedule to allow ample time for the upload and validation process. – The DVT is more sensitive and accurate than its predecessor, QA/QC Pro. That means that the DVT may catch errors for a DFIRM submission for the same Verification Check that its predecessor missed. Thus, users may experience a larger number of errors or warnings on submissions than in the past. Please account for that as you plan the amount of time you need for the automated QA review process. – Average turnaround times: . The turnaround time for an automated review is expected to be significantly faster than the turnaround time for automated reviews through the previous automated QA review tool called QA/QC Pro. During the review, the status of the DFIRM Review in the MIP will read as In-Progress (in the Submission Status section). If the MIP still reads In-Progress after two business days, contact your MIP Black Belt. If necessary, they will contact MIP Help. Do not upload another submission package or click Validate Contents while the Submission Status section reads In-Progress. 5. For assistance with the automated QA review process, each PTS Firm has defined an “auto QC coordinator.” Please see the table below for the auto QC Coordinator for your organization: Regions PTS Contractor QR5 Point of Contact Contact Information 4, 8, 9 BakerAECOM Jerry McCarty jmccarty@mbakercorp.com 2, 3, 6 RAMPP Lynn Russell lrussell@dewberry.com 1, 5, 7, 10 STARR Eric Marcoux eric.marcoux@starr-team.com 6. For assistance with final reviews, each PTS Firm has defined a “QR5 Coordinator.” Please see the table below for the QR5 coordinator for your organization: Regions PTS Contractor QR5 Point of Contact Contact Information 4, 8, 9 BakerAECOM Kate Metznik QR5@mbakercorp.com 2, 3, 6 RAMPP Lynn Russell QR5@rampp-team.com 1, 5, 7, 10 STARR April Gaffney QR5@STARR-team.com 7. Any changes on the J drive to files validated by Metaman and/or the DVT will change the submission from the current status (In-Progress, Passed, etc) to Validation Required. This indicates to the MIP that a file(s) has been altered and requires the user to click Validate Contents to restart the validation process. Changed files not validated by the Metaman or the DVT will not change the submission status to Validation Required. 8. If you have corrections to make to only certain files, it is not necessary to upload all files again, only the files that have the corrections. Make sure these files have the same file name as the incorrect submissions and when you upload the new files, they will overwrite the old files. 9. If you have a submission currently In-Progress, do not upload corrected and/or additional files until a Passed or Failed status is returned. Any resubmission of valid files prior to the return of a Passed or Failed status, even if they are the same file names and file creation dates, will change the status to Validation Required. 10. A notification of DFIRM DB validation results will be sent to the email address associated with the MIP user ID who submitted the DFIRM for validation. 11. Both the Metadata Profile Validation and DFIRM DB QA Validation fields in the Submission Status section of the task must read Passed before the user can complete the Workflow task. 12. Click on the links in the Submission Status section (i.e., Passed, Failed) to view the validation report. 13. Data is copied from the J to K drive for final storage when the user completes the task on the MIP (i.e., clicks Complete Task on the Review screen). III. UPLOADING DATA OPTIONS 1. MIP Studies Workflow Highlights . A new DFIRM Verification Tool (DVT) has been integrated with the MIP. The process to upload and submit DFIRM submissions for automated QA review has not changed, rather, only the tool that performs the review against FEMA’s DFIRM Verification Check standard has changed (one associated change is the files to be included in a DFIRM submission package – see Section IV: Folder Structure and Included Files for specific information). . Detailed results are now available right on the MIP rather than through the QA/QC Pro website. The QA/QC Pro website will be available for a short time after the deployment of the DVT over the weekend of December 11, 2009 but eventually will go away. . Use the MIP Submit Data Files portlet to upload: o Zipped files 1GB or smaller, with fewer than 8000 individual files o Individual files 1GB or smaller Process . Use the Submit Data Files button on the MIP to upload files. This button is found by opening the Submit Data twistie. . When the project is created, the MIP automatically creates a path to where your files will go on the J drive, as indicated in the Submit Data area. Submit Data screen shot showing where your files will be uploaded. o This path is not a link to anything, but indicates where your files will be uploaded. . The Submit Data Files screen is a tool to calculate the approximate amount of time a data upload will take. The Internet Connection Speed and File Size fields on the data submission page are optional. . Users can navigate/browse to any drive (including the J or K drive) to find files to upload the submission. . If the DFIRM submission or parts of it are uploaded as a zip file, the MIP automatically unzips the files once the upload process is complete. While the MIP unzips the submission, the status of the submission will display as “unzipping” (See Section VIII: Tracking Submissions, for more information). The time it takes to complete the unzipping process will vary and is based on the number of files submitted and the size of those files. . Click Update Contents to confirm the files were uploaded and to display the Validate Contents button. . Click Validate Contents to start the metadata validation process. If it is a DFIRM DB task, upon a passing metadata report, relevant files from the DFIRM DB submission are automatically bundled and then submitted to the MIP-DVT engine for review and validation (see Section IV: Folder Structure and Included Files for the files that are required with a DFIRM submission package). . Once the user receives a passing report for both Metadata Profile Validation and DFIRM DB QA Validation, click Complete Task. This will copy the files from the J drive to the K drive and allow the user to move forward in the MIP Studies workflow. For the Produce Final Map Products submission, the user should wait to click Complete Task until they receive a passing report from the visual QC performed by the PTS Firms as well as for Metadata Profile Validation and DFIRM DB QA Validation. 2. Citrix Highlights . Use Citrix to upload individual files larger than 1 GB to the J drive. . Do not upload zipped files or add subfolders to the directory structure. . When all files required for Metaman and DFIRM DB validation are copied to the appropriate folder on the J drive, return to the MIP workflow step and click Validate Contents to kick off the metadata validation. Process . Upload files from your local drive through Citrix to the correct TaskSysID folder on the J drive . Once ALL files are ready for DVT and Metaman review, return to the MIP workflow step and click Validate Contents to start the validation process. If it is a DFIRM DB task, upon a passing metadata report, the DFIRM DB is automatically submitted to the DVT for review. . Once the user receives a passing report for both Metadata Profile Validation and DFIRM DB QA Validation, click Complete Task. This will copy the files from the J drive to the K drive and allow the user to move forward in the MIP Study workflow. For the Produce Final Map Products submission, the user should wait to click Complete Task until they receive a passing report from the visual QC performed by the PTS Firms as well as for Metadata Profile Validation and DFIRM DB QA Validation. 3. MIP Studies Workflow – Mail Data to the Data Depot Highlights . Mail submissions that are larger than 1 GB to the Data Depot. . It typically takes two weeks from the time you mail the data to the Data Depot to the time that data is uploaded into the MIP. If you have not received notification of the upload within 10 business days of mailing your submission, please contact MIP Help to determine the progress on your submittal. . Allow sufficient time in your schedule to accommodate the mail and upload process for these large file submissions. Process . Users who mail submissions to the Data Depot must indicate so through the MIP workflow. . To indicate that files are being mailed, click on the Submit Data Files button from the Submit Data section, click the Mail the Data radio button and click Submit to automatically create a MIP Help ticket. The Data Depot address and all required information are displayed on this screen. Users should not send data to the Data Depot without completing this information on the MIP. . Do not forget to mail the submission! . Once the mailed data is received and uploaded to the MIP, MIP Help will send an email notification to the user to inform them that their data has been uploaded. . Upon notification that the data was uploaded, return to the MIP workflow step and click Validate Contents to kick off the metadata validation. . Once the user receives a passing report for both Metadata Profile Validation and DFIRM DB QA Validation, click Complete Task to copy the files from the J drive to the K drive. For the Produce Final Map Products submission, the user should wait to click Complete Task until they receive a passing report from the visual QC performed by the PTS Firms as well as for Metadata Profile Validation and DFIRM DB QA Validation. 4. DFIRM Tools Highlights . Use DFIRM tools to upload your submission when you have used the DFIRM Tools to create your files. . Remember, after you upload through DFIRM Tools, you must return to the MIP workflow to start the validation process by clicking the Validate Contents button. Process . Once the files are uploaded in DFIRM Tools, click Export. This places the submission on the J drive. . Users must convert the .txt metadata files into .xml prior to submission. . The user must return to the MIP workflow step and click Validate Contents to kick off the validation. . Once the user receives a passing report for both Metadata Profile Validation and DFIRM DB QA Validation, click Complete Task to copy the files from the J drive to the K drive. For the Produce Final Map Products submission, the user should wait to click Complete Task until they receive a passing report from the visual QC performed by the PTS Firms as well as for Metadata Profile Validation and DFIRM DB QA Validation. IV. FOLDER STRUCTURE AND INCLUDED FILES When submitting to the DVT for automated QA review only submit the required files and in the proper file structure, as outlined below. This will reduce excess overload on the DVT and speed up processing. Do not include subfolders within the zipped file that you submit. For the DVT, users are required to submit DFIRM Submissions in Shapefile format as opposed to e00s as before. Shapefile DFIRMs are required in the Draft, Preliminary, and Final stages of the Studies Workflow. There is no change in the Map Service Center (MSC) deliverable requirements with the introduction of the DVT. From a DFIRM format standpoint, users will be required to submit: Shapefile, e00 and MIF (no change from today). FOR ALL SUBMISSIONS: . Required files for DVT: o DFIRM database metadata file (files are required in an .xml file format). The metadata file should be included in the data submission as a separate file in the root directory (See Section V: Tips for Faster Processing for more information). o DFIRM database files in Shapefile format (SHP, SHX, DBF, and PRJ files are required for each Shapefile). The files should be placed in the appropriate TaskSysID folder (see below). Draft Submissions . When preparing a valid submission to the MIP-DVT engine, verify that your submission is located in the following folder: J:\FEMA\[region]\[state]\[county]\[county or community]\[FEMA Case Number]\SubmissionUpload\Mapping.Draft_DFIRM_DB\[task system ID] Prelim Submissions: . When preparing a valid submission to the MIP-DVT engine, verify that your submission is located in the following folder: J:\FEMA\[region]\[state]\[county]\[county or community]\[FEMA Case Number]\SubmissionUpload\Mapping.Prelimanry_DFIRM_DB\[task system ID] Final Submissions: . When preparing a valid submission for the MIP-DVT engine, verify that your submission is located in the following folder: J:\FEMA\[region]\[state]\[county]\[county or community]\[FEMA Case Number]\SubmissionUpload\Mapping.Final_DFIRM_DB\[task system ID]. . Required files for DVT: o DFIRM database metadata file (Files are required in an .xml file format. The metadata file should be included in the data submission as a separate file in the root directory (See Section V: Tips for Faster Processing for more information). o DFIRM database files in Shapefile format (SHP, SHX, DBF, and PRJ files are required for each Shapefile). The files should be placed in the appropriate TaskSysID folder as specified above. . Required files for Task Sys ID folder: o SHP, SHX, DBF o e00 o MIF, MID Additional Required Files . Required files for RFIRM folder: o Ortho-base: PNG, PGW o Vector: TIF, TFW Final DFIRM DB and FIRM panel images (RFIRMs) are retrieved from the J-drive by the PTS contractor responsible for the independent visual QC process. You can now upload the data necessary for visual reviews at the same time as you upload the data for automated QA reviews – the MIP will know not to send the DVT the panel images. When you receive a passing result from the DVT through the MIP that the automated review for the DFIRM submission has passed, and assuming the data needed for the visual review process has been uploaded, notify the PTS QR5 Coordinator (see bullet 6 of Section II: General Guidelines for the name of the QR5 Coordinator for each PTS Firm) table below) that the submission has passed automated validation and that the final DFIRM database and FIRM panel images are on the J-drive and are ready for visual QC. Note that if during the visual review process changes are required to the contents of the data that is reviewed during the automated QA review process, the MIP will require the metadata and DFIRM database files to be re-submitted and re-validated. Note: Georeferenced FIRM panels are still required to be submitted with Final DFIRM submissions. Panel images that are included with the submission will be ignored by the DVT tool (since they are part of the visual QC process, not the automated QA review). V. TIPS FOR FASTER PROCESSING The following are some examples as to how to improve processing time for your submission. Before you upload your submission, remember the following: 1. Metaman will perform validation on the following: Existence of a metadata file in XML format (text files are no longer supported to meet this requirement), DFIRM Table, Source Citation, Metadata Filename and Publication Date, Coordinate System Information : Horizontal Datum, Vertical Datum, Bounding Coordinates, Projection. 2. Confirm that your .xml metadata files are readable. An unreadable XML file will result in a failed submission. 3. Metadata files should be included in the data submission as a separate file in the root directory. Submit Data screen shot showing good metadata root directory information. Submit Data screen shot showing bad metadata root directory information. 4. The FIS folder is no longer created. FIS submissions should be loaded into the appropriate folder within the DFIRM submission. 5. Wait until the metadata and DFIRM DB uploads are complete prior to starting validation. Check this by clicking the Validate Update Contents button. 6. Format projection information correctly. State plane submissions are supposed to define the grid system as “State Plane Coordinate System 1983”. Often users will use some other phrasing, which causes the legacy system to stop processing the study. DVT will now flag this as an error but will still process the study and return a result. 7. Check for typos in included tables. The DVT system checks the metadata to determine which DFIRM layers are supposed to be in the submission. It looks for the string “The following tables are included in this data set:”. If this phrase has a typo, the submission will complete, but DVT will flag an error and send back an error report. BAD–The metadata file will not be found in this folder. If the folder read General, the MIP could find the file. Rootdirectory GOOD–The metadata file is a separate file in the root directory. VI. ADDITIONAL MIP FUNCTIONALITY RELATED TO THE AUTOMATED QA REVIEW PROCESS Tools & Links Test Submissions – The Tools and Links Test Submissions module can be used as a tool for “practicing” DFIRM DB validations for auto review. This allows mapping partners to see what errors they need to fix, if any, prior to submitting the DFIRM DB to the DVT for review. – Uploading DFIRM submissions through Tools and Links does not place data on the J drive, it does not create any records in the MIP, nor does it validate metadata. – The user will receive an email notification of a passing or failed report with a link to the report in the MIP report area. – Changes made to the Verification Check settings in the DVT Administration module do not apply to the Test Submission module. In other words, if a Check is turned off for Workflow submissions it will still be executed for Test Submissions. – Changes made to the error threshold in the DVT Administration module do apply to the Test Submission module. Reporting Tool Automated QA review reports will now be available on the MIP, rather than the QA/QC Pro website. The QA/QC Pro website will remain available for a short time after the deployment of the MIP – DVT Integration release over the weekend of December 11, 2009. During that time, “old” links in the MIP to QA review reports in QA/QC Pro will continue to function as before. Approximately two months after the deployment of the MIP – DVT Integration release, QA/QC Pro will be shut down completely and clicking on those old links to QA review reports in QA/QC Pro will route users to a page that lets them know that the legacy system is no longer available. DVT Administration The DVT Administration module is a tool used to customize DVT verification checks (i.e., turn verification checks on or off), modify DVT Submission Results from failed to passed, and to modify the DVT error threshold (the number of errors a DFIRM submission for automated QA review can still have and still produce a passing result). Any changes to the DVT administration settings will have to be authorized by FEMA. VII. TRACKING SUBMISSIONS The MIP displays the progress of the submission (Uploading, Unzipping, In-Progress, Passed, Failed) in the Submission Status section of the MIP workflow task. . Uploading: Applicable for submissions uploaded through the MIP Submit Data Files portlet o Indicates the submission is being uploaded and will appear as long as the blue status submission bar is displayed. . Unzipping: Applicable for submissions uploaded through the MIP Submit Data Files portlet o Indicates the MIP is unzipping the submission package. o Depending on the size of the zipped file, this process may take up to 30 minutes. . In-Progress: Applicable for all submission options o Indicates the submission package has been submitted to the DVT for review . Passed: Applicable for all submission options o Indicates a passing report from the DVT . Failed: Applicable for all submission options o Indicates a failed report from the DVT o Users may click on the link provided for more information on the reasons for failure. . Validation Required (formerly Data Revised): Applicable for all submission options o Indicates files to be validated by Metaman and/or the DVT have been changed and need to be resubmitted o Click Validate Contents to start the validation process The user who started the validation process in the MIP will receive an email notification when the DFIRM validation is complete. In addition, the status of the DFIRM review in the MIP workflow will be updated with the pass/fail result and a link to the review report.