I have been looking for a good way to convert a log file using powershell into a csv so I can import data correctly into Teradata. I have never seen a file like this before so I am going to try to explain as best as I can.
Here is a sample from the logfile
[XXXXX:11aa11a-123-12a4-12a3-12323aabb123:4] 2021-02-05 00:00:00,000: [DEBUG] Somesystem::Request:
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Header>
<ctx:App xmlns:ctx="http://namespace">
<ctx:Session>
<ctx:UN>XXXXX</ctx:UN>
<ctx:SId>11aa11a-123-12a4-12a3-12323aabb123</ctx:SId>
<ctx:Creation>2021-02-05T00:00:00+11:00</ctx:Creation>
</ctx:Session>
<ctx:IC>
<ctx:TId>11aa1aa-1aa1-2a22-3aa1-aa1b1233456</ctx:TId>
<ctx:Call>
<ctx:BI>13</ctx:BI>
<ctx:CN>Somesystem</ctx:CN>
</ctx:Call>
<ctx:Call>
<ctx:BI>1</ctx:BI>
</ctx:Call>
<ctx:Call>
<ctx:BI>1</ctx:BI>
</ctx:Call>
<ctx:Call>
<ctx:BI>1</ctx:BI>
</ctx:Call>
<ctx:Call>
<ctx:BI>1</ctx:BI>
<ctx:CN>Somesystem</ctx:CN>
<ctx:CL>UnknownLocation</ctx:CL>
</ctx:Call>
<ctx:Call>
<ctx:BI>0</ctx:BI>
</ctx:Call>
</ctx:IC>
</ctx:App>
<wsse:Security xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">
<wsse:UT>
<wsse:UN>SomesystemUSR</wsse:UN>
<wsse:P>Somepassword</wsse:P>
</wsse:UT>
</wsse:Security>
</soapenv:Header>
<soapenv:Body>
<ns5:ListRequest xmlns:ns5="http://Namespace/Service/V1.0" xmlns="http://www.immi.gov.au/Namespace/Enterprise/ErrorMessages/V1.0" xmlns:ns6="http://Namespace/Core/V1.0" xmlns:ns8="http://Namespace/Service/V1.0" xmlns:ns7="http://Namespace/Core/V1.0" xmlns:ns9="http://Namespace/Core/V1.0"
xmlns:ns10="http://Namespace/Core/Messaging/V1.0" xmlns:ns2="http://Namespace/Enterprise/V1.0" xmlns:ns4="http://Namespace/Core/V1.0" xmlns:ns3="http://Namespace/WarningMessages/V1.0">
<ns4:FromDate>2021-01-28</ns4:FromDate>
<ns4:ToDate>2021-01-28</ns4:ToDate>
<ns4:Code>0123</ns4:Code>
<ns4:Type>U</ns4:Type>
<ns4:Record>S</ns4:Record>
<ns4:AnotherCode>D</ns4:AnotherCode>
</ns5:ListRequest>
</soapenv:Body>
</soapenv:Envelope> (jaxws.LoggingJaxWsHandler) [WContainer : 0] [XXXXX:11aa11a-123-12a4-12a3-12323aabb123BB:4] 2021-02-05 01:00:00,000: [DEBUG] Somesystem::Request:
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Header>
<ctx:App xmlns:ctx="http://namespace">
<ctx:Session>
<ctx:UN>XXXXX</ctx:UN>
<ctx:SId>11aa11a-123-12a4-12a3-12323aabb123BB</ctx:SId>
<ctx:Creation>2021-02-05T01:00:00+11:00</ctx:Creation>
</ctx:Session>
<ctx:IC>
<ctx:TId>1as23bb3-1er2-234d-234e-bb8b20995147</ctx:TId>
<ctx:Call>
<ctx:BI>25</ctx:BI>
<ctx:CN>Somesystem</ctx:CN>
</ctx:Call>
<ctx:Call>
<ctx:BI>1</ctx:BranchIndex>
</ctx:Call>
<ctx:Call>
<ctx:BI>1</ctx:BI>
</ctx:Call>
<ctx:Call>
<ctx:BI>1</ctx:BI>
</ctx:Call>
<ctx:Call>
<ctx:BI>1</ctx:BI>
<ctx:CN>Somesystem</ctx:CN>
<ctx:CL>UnknownLocation</ctx:CL>
</ctx:Call>
<ctx:Call>
<ctx:BI>0</ctx:BI>
</ctx:Call>
</ctx:IC>
</ctx:App>
<wsse:Security xmlns:wsse="http://xt-1.0.xsd">
<wsse:UT>
<wsse:UN>SomeUSR</wsse:UN>
</wsse:UT>
</wsse:Security>
</soapenv:Header>
<soapenv:Body>
<s:ListResponse xmlns:s="http://Namespace/Service/V1.0" xmlns:t="http://Namespace/Core/V1.0" xmlns:m="http://Namespace/Core/Messaging/V1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://Namespace/Service/V1.0 ..\Service\V1.0\Response.xsd">
<s:List>
<s:FL>
<t:lDate>2021-01-05</t:lDate>
<t:Code>N</t:Code>
<t:lID>AB123</t:lID>
</s:FL>
<s:FL>
<t:lDate>2021-01-05</t:lDate>
<t:Code>G</t:Code>
<t:lID>CD456</t:lID>
</s:FL>
<s:FL>
<t:Date>2021-01-05</t:Date>
<t:Code>W</t:Code>
<t:lID>EF654</t:lID>
</s:FL>
<s:FL>
<t:Date>2021-01-05</t:Date>
<t:Code>P</t:Code>
<t:lID>TR123</t:lID>
</s:FL>
<s:FL>
<t:lDate>2021-01-05</t:lDate>
<t:Code>N</t:Code>
<t:lID>AB123</t:lID>
</s:FL>
<s:FL>
<t:lDate>2021-01-05</t:lDate>
<t:Code>N</t:Code>
<t:lID>AB123</t:lID>
</s:FL>
<s:FL>
<t:Date>2021-01-05</t:Date>
<t:Code>M</t:Code>
<t:lID>AB345</t:lID>
</s:FL>
<s:FL>
<t:Date>2021-01-05</t:Date>
<t:Code>C</t:Code>
<t:lID>AB456</t:lID>
</s:FL>
</s:List>
</s:ListResponse>
</soapenv:Body>
</soapenv:Envelope> (jaxws.LoggingJaxWsHandler) [WContainer : 3]
I want to be able to extract data in a format that is easy to import into Teradata for example
UN, SId, Creation, TId, BI, CN, FromDate, ToDate, Code, Type, Record, AnotherCode, FL, Ldate, Iid,...... so-on
I am open to any suggestions as to having a better way of getting a flat structure of out this soap tree. As you can see the second SoapEnvelope has a lot more data in soapBody than the first one. I am unable to find any question similar to this one.
Note : I do not have access to command line
As Olaf has stated it might be hard to convert xml into flat csv records. xml is usually made up of many parent/child object whereas csv is typically single object per row
To help maybe guide your progress you can try taking one of the xml envelopes and converting that to an xml object in powershell by doing the following.
$xml = [xml]@"
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
<soapenv:Header>
<ctx:App xmlns:ctx="http://namespace">
<ctx:Session>
<ctx:UN>XXXXX</ctx:UN>
<ctx:SId>11aa11a-123-12a4-12a3-12323aabb123</ctx:SId>
<ctx:Creation>2021-02-05T00:00:00+11:00</ctx:Creation>
</ctx:Session>
<ctx:IC>
<ctx:TId>11aa1aa-1aa1-2a22-3aa1-aa1b1233456</ctx:TId>
<ctx:Call>
<ctx:BI>13</ctx:BI>
<ctx:CN>Somesystem</ctx:CN>
</ctx:Call>
<ctx:Call>
<ctx:BI>1</ctx:BI>
</ctx:Call>
<ctx:Call>
<ctx:BI>1</ctx:BI>
</ctx:Call>
<ctx:Call>
<ctx:BI>1</ctx:BI>
</ctx:Call>
<ctx:Call>
<ctx:BI>1</ctx:BI>
<ctx:CN>Somesystem</ctx:CN>
<ctx:CL>UnknownLocation</ctx:CL>
</ctx:Call>
<ctx:Call>
<ctx:BI>0</ctx:BI>
</ctx:Call>
</ctx:IC>
</ctx:App>
<wsse:Security xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">
<wsse:UT>
<wsse:UN>SomesystemUSR</wsse:UN>
<wsse:P>Somepassword</wsse:P>
</wsse:UT>
</wsse:Security>
</soapenv:Header>
<soapenv:Body>
<ns5:ListRequest xmlns:ns5="http://Namespace/Service/V1.0" xmlns="http://www.immi.gov.au/Namespace/Enterprise/ErrorMessages/V1.0" xmlns:ns6="http://Namespace/Core/V1.0" xmlns:ns8="http://Namespace/Service/V1.0" xmlns:ns7="http://Namespace/Core/V1.0" xmlns:ns9="http://Namespace/Core/V1.0"
xmlns:ns10="http://Namespace/Core/Messaging/V1.0" xmlns:ns2="http://Namespace/Enterprise/V1.0" xmlns:ns4="http://Namespace/Core/V1.0" xmlns:ns3="http://Namespace/WarningMessages/V1.0">
<ns4:FromDate>2021-01-28</ns4:FromDate>
<ns4:ToDate>2021-01-28</ns4:ToDate>
<ns4:Code>0123</ns4:Code>
<ns4:Type>U</ns4:Type>
<ns4:Record>S</ns4:Record>
<ns4:AnotherCode>D</ns4:AnotherCode>
</ns5:ListRequest>
</soapenv:Body>
</soapenv:Envelope>
"@
From there you can examine what the object looks like by drilling down into the properties
PS Env:\> $xml
Envelope
--------
Envelope
PS Env:\> $xml.Envelope
soapenv Header Body
------- ------ ----
http://schemas.xmlsoap.org/soap/envelope/ Header Body
PS Env:\> $xml.Envelope.Body
ListRequest
-----------
ListRequest
PS Env:\> $xml.Envelope.Body.ListRequest
ns5 : http://Namespace/Service/V1.0
xmlns : http://www.immi.gov.au/Namespace/Enterprise/ErrorMessages/V1.0
ns6 : http://Namespace/Core/V1.0
ns8 : http://Namespace/Service/V1.0
ns7 : http://Namespace/Core/V1.0
ns9 : http://Namespace/Core/V1.0
ns10 : http://Namespace/Core/Messaging/V1.0
ns2 : http://Namespace/Enterprise/V1.0
ns4 : http://Namespace/Core/V1.0
ns3 : http://Namespace/WarningMessages/V1.0
FromDate : 2021-01-28
ToDate : 2021-01-28
Code : 0123
Type : U
Record : S
AnotherCode : D
That's all I got. I don't know of any way to convert this type of multidimensional object into flat csv other than picking out what you want from the xml object and creating a custom object with those properties. I'm sure there's someone smarter out there that might be able to offer more :)
UPDATE: So to take this idea a step further and explain what I meant by creating the custom object I've created this function which will take in the xml object and convert it to a custom object with properties I've picked out of the xml. After that you can use either the ConvertTo-Csv
or Export-Csv
cmdlets to generate the csv/csv file
function Convert-SoapXmlToCustomObject
{
[cmdletbinding()]
param(
[Parameter(ValueFromPipeline = $true, Mandatory = $true)]
[xml[]]$XmlObject
)
Process
{
if ($XmlObject)
{
foreach ($xml in $XmlObject)
{
[pscustomobject][ordered]@{
Sid = $xml.Envelope.Header.App.Session.SId
Creation = $xml.Envelope.Header.App.Session.Creation
Tid = $xml.Envelope.Header.App.IC.TId
Calls = @() #if each call needs to be separate record maybe use loop to create
Username = $xml.Envelope.Header.Security.ut.UN
FromDate = $xml.Envelope.Body.ListRequest.FromDate
ToDate = $xml.Envelope.Body.ListRequest.ToDate
Code = $xml.Envelope.Body.ListRequest.Code
Type = $xml.Envelope.Body.ListRequest.Type
Record = $xml.Envelope.Body.ListRequest.Record
AnotherCode = $xml.Envelope.Body.ListRequest.AnotherCode
}
}
}
}
}
$xml, $xml2 | Convert-SoapXmlToCustomObject | ConvertTo-Csv -NoTypeInformation
Output looks like this
"Sid","Creation","Tid","Calls","Username","FromDate","ToDate","Code","Type","Record","AnotherCode"
"11aa11a-123-12a4-12a3-12323aabb123","2021-02-05T00:00:00+11:00","11aa1aa-1aa1-2a22-3aa1-aa1b1233456","System.Object[]","SomesystemUSR","2021-01-28","2021-01-28","0123","U","S","D"
"11aa11a-123-12a4-12a3-12323aabb123BB","2021-02-05T01:00:00+11:00","1as23bb3-1er2-234d-234e-bb8b20995147","System.Object[]","SomeUSR",,,,,,
Collected from the Internet
Please contact [email protected] to delete if infringement.
Comments