Lines Batch QC

Batch QC allows you to rapidly address large data sets with many records. You can run Batch QC on an entire service layer or an existing queue. It is a tool designed to save you time. For example, if you have a thousand lines to address, you may not have time to manually address all of them. Let Batch QC do the work of automatically snapping your lines to flowlines within the snap tolerance. Then you can manually inspect only those lines that did not automatically snap.

If your data has been previously been addressed and already has a ReachCode or GNIS_NAME, HydroAdd will snap it to the flowline with a matching ReachCode or GNIS_NAME. Batch QC prioritizes ReachCode over GNIS_NAME but can use either. For example, you may have line events that have previously been addressed to the NHDPlus V2. Batch QC will snap the line events to a matching ReachCode in the NHD, if it is within the snap tolerance.

Batch QC results reside on the HydroAdd application server—your data are not updated to AGOL until you publish. Results of Batch QC jobs are always available in the Batch QC queue--even after you publish or remove the corresponding service layer from AGOL. You should always review the results of Batch QC before you publish the results to your service layer on AGOL. This acts as a safeguard to protect your data because it allows to you to try different snap tolerances before you publish. You cannot undo the edits to your service layer once HydroAdd publishes the updates. In any case, you should always keep a copy of your original data and ArcGIS Pro project.

Lines Batch QC Pane

The Batch QC panes lists all the batch jobs you have run. You can filter the jobs list by date range, or sort by created/completed date. The Batch QC results listed in the Batch QC Pane reside on the HydroAdd application server—your data are not updated to AGOL until you publish. Results of Batch QC jobs are always available in the Batch QC pane--even after you publish or remove the corresponding service layer from AGOL.

Create Lines Batch QC Job

1. Click Lines Batch QC job
   
2. The Create Batch QC Job window pops up.
3. Enter a queue or an entire service.
4. Enter a nickname the Batch QC job.
5. Enter the Snap Tolerance (meters).
6. Enter the QC Tolerance (meters).
7. Enter the Length Change Tolerance (% change). 
   
8. Click Create.
   The Batch Job runs in the background.
9. Check the Batch QC Jobs Pane to see progress.
10. The Job Status updates in real time as the job as runs.
    ‎ The Batch QC Job Status shows:
    1. Created date
    2. Edit queue or service 
    3. QC Tolerance, Snap Tolerance, and Length Change Tolerance
    4. Percent of features processed
    5. Percent of features approved
    6. Percent of features published to AGOL
    7. Job status
       

See also HydroAdd Line Schema.

See also Lines Batch QC Logic.

Snap Tolerance for lines

User input. The maximum distance you allow Batch QC to move both start and end points of a line event. Snap tolerance is essentially a buffer around the start and end points. If there is a flowline feature within the snap tolerance of both start and end points, Batch QC will snap the points, then draw a line event connecting the points. Both start and end points must be in snap tolerance for HydroAdd to draw a new line event. If one of the points is not in snap tolerance, HydroAdd will not draw a new line event. 

• If SnapDistance <= SnapTolerance, then InSnapTolerance = Yes.
• If no flowlines are within the SnapTolerance, then InSnapTolereance = No, InQCTolerance = No, and QCApproved = No
• If the input line already has a ReachCode, Batch QC will snap it to the NHDFlowline with matching ReachCode, if it is within the SnapTolerance.
• If the input line already has a GNIS_NAME, Batch QC will snap it to the NHDFlowline with matching GNIS_NAME, if it is within the Snap Tolerance.
• If the input line already has both a ReachCode and GNIS_NAME, ReachCode takes priority.
• If the input line does not have a preexisting ReachCode or GNIS_NAME, Batch QC will snap it to the nearest flowline within SnapTolerance regardless of the ReachCode or GNIS_NAME on the flowline.

See also HydroAdd Line Schema.

See also Lines Batch QC Logic.

QC Tolerance for lines

User input. The maximum snap distance where both start and end points of a line event are automatically approved. If there is a flowline feature within the snap tolerance of both start and end points, Batch QC will snap the points, then draw a line event connecting the points. However, automatic approval by Batch QC depends on the QC tolerance. If the snap distance is greater than the QC tolerance, then the lines is not approved.

• If SnapDistance <= QCTolerance, then InQCTolerance = Yes, and QCApproved = Yes
• If no flowlines are in the SnapTolerance, then HydroAdd will not draw a new line event. In this case, InSnapTolereance = No, InQCTolerance = No, and QCApproved = No

Tips for setting QC tolerance

• QC Tolerance is important when your input lines do not have a preexisting ReachCode or GNIS_NAME.
• QC Tolerance is especially useful in cases where the snap tolerance is large. When the Snap Tolerance is large, there are likely to be multiple flowlines within the snap tolerance. If the start or end points moves very far, there is a possibility they will snap to the wrong flowline. For this reason, when using a large snap tolerance, a good practice is to set the QC tolerance at about one half or more of the snap distance.

Length Change Tolerance

User input. Percent length change acceptable during batch QC. HydroAdd splits lines at ReachCodes. A single long line event may be split into several shorter line events--one on each ReachCode. Length Change is the percent length change of newly addressed line from original input line. 

• LengthChange = Abs[length new feature - length original feature)/length original feature] * 100
• If LengthChange <= LengthChangeTolerance, then InLengthChangeTolerance=Yes

Tips for setting Length Change Tolerance

Length Change Tolerance is useful for finding very short line events produced by Batch QC. For example, an original line event may be split into several shorter line events during Batch QC--one on each ReachCode. Sometimes the the original line event overlaps a short segment of an adjacent ReachCode. The most common place to find such short segments is on tributaries to the main stem or at confluences where two or more ReachCodes meet. Compared to the original line event, these short segments have large percent length change, usually greater than 80%. 

• To find very short segments during QC, set the Length Change Tolerance high, up to 80-90%. New line events with Length Change less than the Length Change Tolerance are approved, but very short segments where the Length Change is greater than Length Change Tolerance are not approved.

• To QC line events by Snap Distance alone, set Length Change Tolerance to 100%. All lengths of new line events are thus approved, and QC approval status depends solely on QC Tolerance.  

• To QC line events by Length Change alone, set the QC Tolerance equal to the Snap Tolerance. All snap distances are thus approved, and QC approval status depends solely on Length Change Tolerance.
• To QC line events that changed length only slightly, set the Length Change Tolerance low, as low as 5-10%. 

See also HydroAdd Line Schema.

See also Lines Batch QC Logic.

Lines Batch QC Job Status

The Lines Batch QC Job Status shows:

• Created date
• Edit queue or service 
• QC Tolerance, Snap Tolerance, and Length Change Tolerance
• Percent of features processed
• Percent of features approved
• Percent of features published to AGOL
• Job status
  

Lines Batch QC Review Queue

The Batch QC Review Queue allows you to review the Batch QC results before you publish to AGOL. Batch QC results reside on the HydroAdd application server—your data are not updated to AGOL until you publish. Results of Batch QC jobs are always available in the Batch QC pane—even after you publish or remove the corresponding service layer from AGOL. You should always review the results of Batch QC before you publish the results to your service layer on AGOL. This acts as a safeguard to protect your data because it allows to you to try different tolerances before you publish. You cannot undo the edits to your data once HydroAdd publishes the updates. In any case, you should always keep a copy of your original data and ArcGIS Pro project.

Batch QC splits very long line events into several shorter line events--one on each ReachCode. The Batch QC queue displays these shorter segments as a line group under the original. You may find that some of these shorter segments are not approved, even if they are in the right place. This is likely because the shorter segments are outside the Length Change Tolerance. Compared to the original line event, these short segments have large percent length change. To find very short segments during QC, set the Length Change Tolerance high, up to 80-90%.

See also Tips for lines Batch QC.

• Click a row in the Batch QC queue to zoom to the feature. 

• Hover the cursor over a row to highlight features yellow.

• Original features are cyan.

• Results are red. 

• Queues are paginated. You can choose the number of rows per page.

Lines Batch QC Feature Details

You can see details of the Batch QC for each feature.

1. Click the details button from a row in the Batch QC queue
   
2. The Batch QC details pane opens
    
3. The Batch QC details for each feature indicates:
   1. Snap Tolerance
   2. QC Tolerance
   3. On Network (Y/N)
   4. Snap Distance
   5. Length Change Percent
   6. Original ReachCode Match (Y/N)
   7. In Snap Tolerance (Y/N)
   8. In QC Tolerance (Y/N)
   9. In Length Change Tolerance (Y/N)
   10. Equidistant - start point or end point is equidistant to locations on one or more flowlines (Y/N) 
   11. QC Approved (Y/N)

See also HydroAdd Line Schema.

See also Lines Batch QC Logic.

Publishing QC Results to ArcGIS Online

Publishing from HydroAdd updates the features in your service layer at AGOL. However, the results displayed in the Batch QC queue do not change after publishing. The results are essentially a static snapshot of the state of your service layer before publishing. Even when the features in the map update to new locations after publishing, results of Batch QC queue remain unchanged.

You can always run Batch QC again after publishing, but you will get different results because publishing overwrites your service layer—many of the feature will have been addressed to flowlines. To see the actual approval status of the features after publishing, you must examine the features in an editing queue (not the Batch QC queue) or use the identify tool in the map.

NOTE: Only published lines can be added to editing queues. This is because Batch QC results reside only on the HydroAdd application server—they are virtual objects, not real features in your service layer. Your data are not updated to AGOL until you publish. However, only "real" line features with geometry can be added to queues. Points do not have this limitation because their geometry is simple; unpublished points can be added to editing queues.

To publish one line

1. The original line is cyan.
   The new lines are red.
   Hover the cursor over a row to highlight features yellow.
   
2. Click the cloud icon in the row of the Batch QC results queue.
   
3. The cloud icon updates briefly to show the item is publishing.
4. When the item is published, the could icon updates to show published status.
   
5. A popup, lower right, indicates when the feature updates.
    
6. After a moment, the map updates to show the updated location.
7. Confirm the updates in Search/Select or an editing queue.
   Notice the original line event is gone from the map. Also, notice the newly published line events have the same SourceID
   as the original line events.
   
8. You can return to the Batch QC queue at any time to publish the remaining results.

To publish multiple lines

Publishing from HydroAdd updates the features in your service layer at AGOL. However, the results displayed in the Batch QC queue do not change after publishing. The results are essentially a static snapshot of the state of your service layer before publishing. Even when the features in the map update to new locations after publishing, results of Batch QC queue remain unchanged.

You can always run Batch QC again after publishing, but you will get different results because publishing overwrites your service layer—many of the feature will have been addressed to flowlines. To see the actual approval status of the features after publishing, you must examine the features in an editing queue (not the Batch QC queue) or use the identify tool in the map.

• Use the Pick All checkbox to pick all rows on a page or use a row checkbox to pick single features. Only the features in the page will be picked. For example, if there are 10 rows in the Batch QC queue only 10 features will be picked. To pick more rows, increase the number of rows per page or go to the next page. You can pick up to 2,000 rows per page.
• Alternatively, you can pick only the items in a line group. Batch QC splits very long line events into several shorter line events--one on each ReachCode. The Batch QC queue displays these shorter segments as a line group under the original.
• Once you have picked multiple rows, you can publish, or add to/remove from a queue.
• You can publish approved features, unapproved, or both.
• You can add published items to editing queues. NOTE: Only published lines can be added to editing queues. This is because Batch QC results reside only on the HydroAdd application server—they are virtual objects, not real features in your service layer. Your data are not updated to AGOL until you publish. However, only "real" line features with geometry can be added to queues. Points do not have this limitation because their geometry is simple; unpublished points can be added to editing queues.
• NOTE: Publishing takes time. It is recommended to select the maximum number of items (2,000) then let publishing run in the background or run overnight.  
• Best practice is to publish all lines, even unapproved lines. This allows you to add unapproved lines to a queue after publishing is complete. 

To publish multiple lines

1. Select multiple lines, a line group, or select all rows.
2. The maximum number of items that can be selected for publishing is 2,000. 
3. Click the cloud icon at the top of the Batch QC queue.‎
   
4. The Public Batch QC Job window pop up.
5. Click Publish. HydroAdd updates the selected features to your service layer at AGOL.
   
6. The cloud icon updates briefly to show the item is publishing.
7. When the item is published, the could icon updates to show published status.
   
8. A popup, lower right, indicates when the feature updates.
    
9. NOTE: Publishing takes time. It is recommended to select the maximum number of items (2,000) then let publishing run in the background or run overnight.  
10. Best practice is to publish all lines, even unapproved lines. This allows you to add unapproved lines to a queue after publishing is complete. 

See also Tips for lines Batch QC.

See also HydroAdd Line Schema.

See also Lines Batch QC Logic.

Publishing Status

The publishing status of the entire job is displayed with job statistics in the list of all Batch QC jobs.  

The publishing status of individual features is always shown in the Batch QC queue:

Making queues from Lines Batch QC Jobs

You can add selected lines to editing queues. NOTE: Only published lines can be added to editing queues. This is because Batch QC results reside only on the HydroAdd application server—they are virtual objects, not real features in your service layer. Your data are not updated to AGOL until you publish. However, only "real" line features with geometry can be added to queues. Points do not have this limitation because their geometry is simple; unpublished points can be added to editing queues.

Because only published lines can be added to queues, best practice is to publish all lines, even unapproved lines. Once publishing is complete, add the unapproved lines to an editing queue for manual inspection. 

1. Select a line group, or select all rows.

2. The maximum number of items that can be selected per page is 2,000. 
3. Queues with more than 2,000 items are possible, but not recommended. 
4. Click the queue button icon at the top of the Batch QC queue.‎
   
5. The Create Edit Queue window pop up.
6. Click Create Edit Queue.
   
7. HydroAdd creates an editing queue of the selected features.
   A popup, lower right, indicates the queue is created.
   
8. The Editing queue appears in the editing queues pane. 
   

Lines Batch QC logic table

Overview of Lines Batch QC Logic. Note that InSnapTolerace refers to both start and end points of the line.

• If SnapDistance <= SnapTolerance, then InSnapTolerance = Yes 
• If no flowlines are in within the SnapTolerance, then InQCTolerance = No and QCApproved = No
• If SnapDistance <= QCTolerance, then InQCTolerance = Yes.
• If QCApproved = No, then the user should manually QC the item in an editing queue. 
• QCApproved = Yes only when both InQCTolerance and InLengthTolerance = Yes.