--- title: "Merging v3.0.3" canonical: "https://servicedesk.maxgeo.com/space/DS/1437335579/Merging%20v3.0.3" format: markdown --- > Macro (toc)  When DataShed5 runs the assay import via the single file import option the following steps are carried out. 1. The assays are loaded into a buffer so that checks can be made to ensure the integrity of the data.  DataShed5 checks the lab method, element, unit and lab libraries to ensure that the data conforms to these libraries.  The Generic Method is updated from the Original Lab Method and the Element and Repeat fields are populated from the Lab Element library. New data records are loaded to the Append Data tab and Existing records are loaded into the Merge Data tab on the Buffer Table datagrid. 2. The incoming sample ids are matched with the sample ids in each of the assay tables. 3. DataShed5 calculates the sysResult from the Assay Result.  Any text fields are converted to the corresponding substitution value and the text equivalent is entered into the sysAssayStatus field. 4. DataShed5 merges the records from the Merge Data tab that have been successfully checked.  To view the unsuccessful records, go to the **Assay Manager** and any unmerged data is listed.  Once the unmerged data has been sorted out, click the **Process Batch** button next to the batch in the Batch List.  DataShed5 checks and merges the data.   **NOTE: After every import, open the appropriate tables and DataSet in DataShed5 and view the imported records to confirm they are correct** ![image-20241008-034847.png](media://f0f96ce7-6ddd-43f6-a02d-c5768902a285) ## Viewing Unmerged Data After importing assays, DataShed5 retains any assays that have not merged into the main assay tables in the Incoming Assays table. To view the unmerged assay data go to **Assay Manager. ** In the Unmerged Assay Results panel the unmerged assays are listed by the various categories of: Missing library entries, No matching samples or Other as shown in the diagram above. To establish the reason for the data not merging, go through the process that DataShed5 follows in checking and validating each record.  Each of the steps, with the associated error message if they fail, is described in detail below: ## Missing Library Entries  If there are records in the missing library entries, the exception type is set to Missing Library Entry.  The library tables can be accessed from the **Assay Management** menu. ![image-20241008-034930.png](media://8e0e29fc-a552-40b2-859f-6b9b4cf01e09) ![image-20241008-034944.png](media://43d71545-8f1e-454f-bb57-8fbf6f8bd73f) **Element Groups:**                              Names of the element groups **Elements:**                            Names of the elements and their associated groups **Method Priorities:**            Ranking of generic methods for assay processing **Generic Methods:**            Translation of laboratory methods into standard methods **Lab Methods Alias:**          Analytical methods used by the laboratory **Labs:**                                     Analytical laboratory **Lab Element Alias***:*           Element name used by the laboratory          **Units:**                                    A list of the units of measurement **Conversion Factors:**       A list of conversions between measurement units The Comment field gets populated with a list of the codes that are missing from the library tables. ![image-20241008-035009.png](media://c7cde8e7-47e0-40e5-ba8f-7056de72e332) - Is there a Laboratory with the Code of the Incoming Assays? - Is there a method for that LabCode in the library that matches the “OrigMethod” of the assay? - Is there a LabElement for that LabCode that matches the LabElement of the assay?  The Comments field will contain information about what library values are missing: - Missing Lab Element Alias; Unknown: The lab element does not have an element and repeat specified in the Lab Element Alias library - Original Lab Method; Missing Method Alias: the Lab method does not have a Generic method specified in the Lab Method Alias library.  Any missing library entries listed in the Unmerged Assay Results can be added via the Assay Libraries tab.  To Add a Lab Element: - Click on the Lab Element Alias - Enter the Lab Code, this must be a lab that is in the Labs library - Enter the Lab Element code from the lab file - Assign an element and a repeat code. The Element must be a code in the Element Assay library. ![image-20241008-035058.png](media://aca64618-3d7d-4bf0-90ec-c8e692365b9d) ## No Matching Samples DataShed5 references the sample register to match the assay results with a SampleID in the database.  This is commonly tblSampleRegister and is a table that is managed by triggers in the SQL database.  A SampleID must be present in this table before the assay result will be merged from the Incoming Assay table into the normalised assay table. If there are records in the no matching samples, the exception type is updated to No Matching Samples. If the same SampleID has been used in different DataSets exception type is set to More Than One Sample Match.  To manage the SampleIDs that are available to DataShed5 a DataSet must be  “active”. This is managed in the **DataSet Manager**. ![image-20241008-035121.png](media://9428d93a-1c40-4020-93a9-875f161019ef) ## Other  If there are records listed, it often indicates that samples have a null result or a text result such as NL or SNR, which is not listed in the substitution table. There is no error message, but the sysResult field remains NULL. **Substitutions** - If the assay result is 0 or NULL, then DataShed5 is unable to do a calculation on this record. If the Assay Result is NULL, then these records can be deleted. If the Assay Result is 0, check if it means that the sample was not assayed or whether it is below detection limits. - BDL substitutions ensure that the lower limit is populated and that the substitution for the import layout has been populated correctly. The sysAssayStatus will be populated with BDL if the substitutions table has been populated correctly.  For example, a “<” before the results indicates that it is Below Detection Level for that method and “BDL” is inserted in the sysAssayStatus field. - OR substitution must have the upper limit populated so that the sysresult can be calculated.  **Units:** - The unit the result recorded in must be in the Units library ![image-20241008-035206.png](media://6cfb4d0f-d677-4b67-b4bf-34aed01e8229) - There must be a conversion from the unit code reported to the system unit specified in the Elements library. - Find out what the system unit for the element is from the Elements library.  This information is stored in the Elements library. ![image-20241008-035241.png](media://61050093-60c9-4ec9-b0d8-3004d4db3124) - Add the conversion from the unit the result was reported by that laboratory to the preferred unit in the elements table. In the example above, the conversion from Kg to g is added. ![image-20241008-035303.png](media://2c5cc61f-4859-4f67-ba12-0dfc692de8b8)