convert soap log file to csv using powershell

User56756

I have been looking for a good way to convert a log file using powershell into a csv so I can import data correctly into Teradata. I have never seen a file like this before so I am going to try to explain as best as I can.

Here is a sample from the logfile

[XXXXX:11aa11a-123-12a4-12a3-12323aabb123:4] 2021-02-05 00:00:00,000: [DEBUG] Somesystem::Request:
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
  <soapenv:Header>
    <ctx:App xmlns:ctx="http://namespace">
      <ctx:Session>
        <ctx:UN>XXXXX</ctx:UN>
        <ctx:SId>11aa11a-123-12a4-12a3-12323aabb123</ctx:SId>
        <ctx:Creation>2021-02-05T00:00:00+11:00</ctx:Creation>
      </ctx:Session>
      <ctx:IC>
        <ctx:TId>11aa1aa-1aa1-2a22-3aa1-aa1b1233456</ctx:TId>
        <ctx:Call>
          <ctx:BI>13</ctx:BI>
          <ctx:CN>Somesystem</ctx:CN>
        </ctx:Call>
        <ctx:Call>
          <ctx:BI>1</ctx:BI>
        </ctx:Call>
        <ctx:Call>
          <ctx:BI>1</ctx:BI>
        </ctx:Call>
        <ctx:Call>
          <ctx:BI>1</ctx:BI>
        </ctx:Call>
        <ctx:Call>
          <ctx:BI>1</ctx:BI>
          <ctx:CN>Somesystem</ctx:CN>
          <ctx:CL>UnknownLocation</ctx:CL>
        </ctx:Call>
        <ctx:Call>
          <ctx:BI>0</ctx:BI>
        </ctx:Call>
      </ctx:IC>
    </ctx:App>
    <wsse:Security xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">
      <wsse:UT>
        <wsse:UN>SomesystemUSR</wsse:UN>
        <wsse:P>Somepassword</wsse:P>
      </wsse:UT>
    </wsse:Security>
  </soapenv:Header>
  <soapenv:Body>
    <ns5:ListRequest xmlns:ns5="http://Namespace/Service/V1.0" xmlns="http://www.immi.gov.au/Namespace/Enterprise/ErrorMessages/V1.0" xmlns:ns6="http://Namespace/Core/V1.0" xmlns:ns8="http://Namespace/Service/V1.0" xmlns:ns7="http://Namespace/Core/V1.0" xmlns:ns9="http://Namespace/Core/V1.0"
      xmlns:ns10="http://Namespace/Core/Messaging/V1.0" xmlns:ns2="http://Namespace/Enterprise/V1.0" xmlns:ns4="http://Namespace/Core/V1.0" xmlns:ns3="http://Namespace/WarningMessages/V1.0">
      <ns4:FromDate>2021-01-28</ns4:FromDate>
      <ns4:ToDate>2021-01-28</ns4:ToDate>
      <ns4:Code>0123</ns4:Code>
      <ns4:Type>U</ns4:Type>
      <ns4:Record>S</ns4:Record>
      <ns4:AnotherCode>D</ns4:AnotherCode>
    </ns5:ListRequest>
  </soapenv:Body>
</soapenv:Envelope> (jaxws.LoggingJaxWsHandler) [WContainer : 0] [XXXXX:11aa11a-123-12a4-12a3-12323aabb123BB:4] 2021-02-05 01:00:00,000: [DEBUG] Somesystem::Request:
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
  <soapenv:Header>
    <ctx:App xmlns:ctx="http://namespace">
      <ctx:Session>
        <ctx:UN>XXXXX</ctx:UN>
        <ctx:SId>11aa11a-123-12a4-12a3-12323aabb123BB</ctx:SId>
        <ctx:Creation>2021-02-05T01:00:00+11:00</ctx:Creation>
      </ctx:Session>
      <ctx:IC>
        <ctx:TId>1as23bb3-1er2-234d-234e-bb8b20995147</ctx:TId>
        <ctx:Call>
          <ctx:BI>25</ctx:BI>
          <ctx:CN>Somesystem</ctx:CN>
        </ctx:Call>
        <ctx:Call>
          <ctx:BI>1</ctx:BranchIndex>
        </ctx:Call>
        <ctx:Call>
          <ctx:BI>1</ctx:BI>
        </ctx:Call>
        <ctx:Call>
          <ctx:BI>1</ctx:BI>
        </ctx:Call>
        <ctx:Call>
          <ctx:BI>1</ctx:BI>
          <ctx:CN>Somesystem</ctx:CN>
          <ctx:CL>UnknownLocation</ctx:CL>
        </ctx:Call>
        <ctx:Call>
          <ctx:BI>0</ctx:BI>
        </ctx:Call>
      </ctx:IC>
    </ctx:App>
    <wsse:Security xmlns:wsse="http://xt-1.0.xsd">
      <wsse:UT>
        <wsse:UN>SomeUSR</wsse:UN>
      </wsse:UT>
    </wsse:Security>
  </soapenv:Header>
  <soapenv:Body>
    <s:ListResponse xmlns:s="http://Namespace/Service/V1.0" xmlns:t="http://Namespace/Core/V1.0" xmlns:m="http://Namespace/Core/Messaging/V1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://Namespace/Service/V1.0 ..\Service\V1.0\Response.xsd">
      <s:List>
        <s:FL>
          <t:lDate>2021-01-05</t:lDate>
          <t:Code>N</t:Code>
          <t:lID>AB123</t:lID>
        </s:FL>
        <s:FL>
          <t:lDate>2021-01-05</t:lDate>
          <t:Code>G</t:Code>
          <t:lID>CD456</t:lID>
        </s:FL>
        <s:FL>
          <t:Date>2021-01-05</t:Date>
          <t:Code>W</t:Code>
          <t:lID>EF654</t:lID>
        </s:FL>
        <s:FL>
          <t:Date>2021-01-05</t:Date>
          <t:Code>P</t:Code>
          <t:lID>TR123</t:lID>
        </s:FL>
        <s:FL>
          <t:lDate>2021-01-05</t:lDate>
          <t:Code>N</t:Code>
          <t:lID>AB123</t:lID>
        </s:FL>
        <s:FL>
          <t:lDate>2021-01-05</t:lDate>
          <t:Code>N</t:Code>
          <t:lID>AB123</t:lID>
        </s:FL>
        <s:FL>
          <t:Date>2021-01-05</t:Date>
          <t:Code>M</t:Code>
          <t:lID>AB345</t:lID>
        </s:FL>
        <s:FL>
          <t:Date>2021-01-05</t:Date>
          <t:Code>C</t:Code>
          <t:lID>AB456</t:lID>
        </s:FL>
      </s:List>
    </s:ListResponse>
  </soapenv:Body>
</soapenv:Envelope> (jaxws.LoggingJaxWsHandler) [WContainer : 3]

I want to be able to extract data in a format that is easy to import into Teradata for example

UN, SId, Creation, TId, BI, CN, FromDate, ToDate, Code, Type, Record, AnotherCode, FL, Ldate, Iid,...... so-on

I am open to any suggestions as to having a better way of getting a flat structure of out this soap tree. As you can see the second SoapEnvelope has a lot more data in soapBody than the first one. I am unable to find any question similar to this one.

Note : I do not have access to command line

Daniel

As Olaf has stated it might be hard to convert xml into flat csv records. xml is usually made up of many parent/child object whereas csv is typically single object per row

To help maybe guide your progress you can try taking one of the xml envelopes and converting that to an xml object in powershell by doing the following.

$xml = [xml]@"
<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
  <soapenv:Header>
    <ctx:App xmlns:ctx="http://namespace">
      <ctx:Session>
        <ctx:UN>XXXXX</ctx:UN>
        <ctx:SId>11aa11a-123-12a4-12a3-12323aabb123</ctx:SId>
        <ctx:Creation>2021-02-05T00:00:00+11:00</ctx:Creation>
      </ctx:Session>
      <ctx:IC>
        <ctx:TId>11aa1aa-1aa1-2a22-3aa1-aa1b1233456</ctx:TId>
        <ctx:Call>
          <ctx:BI>13</ctx:BI>
          <ctx:CN>Somesystem</ctx:CN>
        </ctx:Call>
        <ctx:Call>
          <ctx:BI>1</ctx:BI>
        </ctx:Call>
        <ctx:Call>
          <ctx:BI>1</ctx:BI>
        </ctx:Call>
        <ctx:Call>
          <ctx:BI>1</ctx:BI>
        </ctx:Call>
        <ctx:Call>
          <ctx:BI>1</ctx:BI>
          <ctx:CN>Somesystem</ctx:CN>
          <ctx:CL>UnknownLocation</ctx:CL>
        </ctx:Call>
        <ctx:Call>
          <ctx:BI>0</ctx:BI>
        </ctx:Call>
      </ctx:IC>
    </ctx:App>
    <wsse:Security xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">
      <wsse:UT>
        <wsse:UN>SomesystemUSR</wsse:UN>
        <wsse:P>Somepassword</wsse:P>
      </wsse:UT>
    </wsse:Security>
  </soapenv:Header>
  <soapenv:Body>
    <ns5:ListRequest xmlns:ns5="http://Namespace/Service/V1.0" xmlns="http://www.immi.gov.au/Namespace/Enterprise/ErrorMessages/V1.0" xmlns:ns6="http://Namespace/Core/V1.0" xmlns:ns8="http://Namespace/Service/V1.0" xmlns:ns7="http://Namespace/Core/V1.0" xmlns:ns9="http://Namespace/Core/V1.0"
      xmlns:ns10="http://Namespace/Core/Messaging/V1.0" xmlns:ns2="http://Namespace/Enterprise/V1.0" xmlns:ns4="http://Namespace/Core/V1.0" xmlns:ns3="http://Namespace/WarningMessages/V1.0">
      <ns4:FromDate>2021-01-28</ns4:FromDate>
      <ns4:ToDate>2021-01-28</ns4:ToDate>
      <ns4:Code>0123</ns4:Code>
      <ns4:Type>U</ns4:Type>
      <ns4:Record>S</ns4:Record>
      <ns4:AnotherCode>D</ns4:AnotherCode>
    </ns5:ListRequest>
  </soapenv:Body>
</soapenv:Envelope>
"@

From there you can examine what the object looks like by drilling down into the properties

    PS Env:\> $xml

Envelope
--------
Envelope

PS Env:\> $xml.Envelope

soapenv                                   Header Body
-------                                   ------ ----
http://schemas.xmlsoap.org/soap/envelope/ Header Body

PS Env:\> $xml.Envelope.Body

ListRequest
-----------
ListRequest

PS Env:\> $xml.Envelope.Body.ListRequest

ns5         : http://Namespace/Service/V1.0
xmlns       : http://www.immi.gov.au/Namespace/Enterprise/ErrorMessages/V1.0
ns6         : http://Namespace/Core/V1.0
ns8         : http://Namespace/Service/V1.0
ns7         : http://Namespace/Core/V1.0
ns9         : http://Namespace/Core/V1.0
ns10        : http://Namespace/Core/Messaging/V1.0
ns2         : http://Namespace/Enterprise/V1.0
ns4         : http://Namespace/Core/V1.0
ns3         : http://Namespace/WarningMessages/V1.0
FromDate    : 2021-01-28
ToDate      : 2021-01-28
Code        : 0123
Type        : U
Record      : S
AnotherCode : D

That's all I got. I don't know of any way to convert this type of multidimensional object into flat csv other than picking out what you want from the xml object and creating a custom object with those properties. I'm sure there's someone smarter out there that might be able to offer more :)

UPDATE: So to take this idea a step further and explain what I meant by creating the custom object I've created this function which will take in the xml object and convert it to a custom object with properties I've picked out of the xml. After that you can use either the ConvertTo-Csv or Export-Csv cmdlets to generate the csv/csv file

function Convert-SoapXmlToCustomObject
{
    [cmdletbinding()]
    param(
        [Parameter(ValueFromPipeline = $true, Mandatory = $true)]
        [xml[]]$XmlObject
    )

    Process
    {
        if ($XmlObject)
        {
            foreach ($xml in $XmlObject)
            {
                [pscustomobject][ordered]@{
                    Sid         = $xml.Envelope.Header.App.Session.SId
                    Creation    = $xml.Envelope.Header.App.Session.Creation
                    Tid         = $xml.Envelope.Header.App.IC.TId
                    Calls       = @() #if each call needs to be separate record maybe use loop to create
                    Username    = $xml.Envelope.Header.Security.ut.UN
                    FromDate    = $xml.Envelope.Body.ListRequest.FromDate
                    ToDate      = $xml.Envelope.Body.ListRequest.ToDate
                    Code        = $xml.Envelope.Body.ListRequest.Code
                    Type        = $xml.Envelope.Body.ListRequest.Type
                    Record      = $xml.Envelope.Body.ListRequest.Record
                    AnotherCode = $xml.Envelope.Body.ListRequest.AnotherCode

                }
            }
        }
    }
}

$xml, $xml2 | Convert-SoapXmlToCustomObject | ConvertTo-Csv -NoTypeInformation

Output looks like this

"Sid","Creation","Tid","Calls","Username","FromDate","ToDate","Code","Type","Record","AnotherCode"
"11aa11a-123-12a4-12a3-12323aabb123","2021-02-05T00:00:00+11:00","11aa1aa-1aa1-2a22-3aa1-aa1b1233456","System.Object[]","SomesystemUSR","2021-01-28","2021-01-28","0123","U","S","D"
"11aa11a-123-12a4-12a3-12323aabb123BB","2021-02-05T01:00:00+11:00","1as23bb3-1er2-234d-234e-bb8b20995147","System.Object[]","SomeUSR",,,,,,

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

From Dev

Writing to a log file using powershell

From Dev

create a log file using powershell

From Dev

Convert CSV to JSON and JSON to CSV using PowerShell

From Dev

CSV file editing using PowerShell

From Dev

Convert multiple xls to csv using powershell

From Dev

How to convert this XML to CSV using PowerShell 2.0?

From Java

Convert Azure DevOps rest api test plans for all projects to csv file using powershell

From Dev

How can I convert a csv file to lowercase or uppercase maintaining it's structure using powershell?

From Dev

How to convert text file containing double quotes to csv format using powershell

From Dev

Convert JSON file to a CSV file using R

From Dev

Convert an XML file to CSV file using java

From Dev

Remove Columns from CSV File Using Powershell

From Dev

Appending a CSV file to end of another, using Powershell

From Dev

How to convert a text file to CSV using Bash

From Dev

How to convert a CSV file using a PHP script

From Dev

Unable to convert JSON file to CSV using Python

From Dev

Convert Resultset to CSV file using Java

From Dev

Using PHP how to convert csv file

From Dev

How to convert XML file to CSV using Perl?

From Dev

convert a text file to csv using shell script

From Dev

How to convert a CSV file using a PHP script

From Dev

How to convert a text file to CSV using Bash

From Dev

Unable to convert JSON file to CSV using Python

From Dev

Convert .csv file to .ods using terminal

From Dev

PowerShell - Convert CSV to XLSX

From Dev

Create a CSV file from a specific log file using shell script

From Dev

C# convert csv to xls (using existing csv file)

From Dev

How to convert .xls to .csv using Powershell without Excel installed

From Dev

How to convert the csv to lower case using Powershell script

Related Related

  1. 1

    Writing to a log file using powershell

  2. 2

    create a log file using powershell

  3. 3

    Convert CSV to JSON and JSON to CSV using PowerShell

  4. 4

    CSV file editing using PowerShell

  5. 5

    Convert multiple xls to csv using powershell

  6. 6

    How to convert this XML to CSV using PowerShell 2.0?

  7. 7

    Convert Azure DevOps rest api test plans for all projects to csv file using powershell

  8. 8

    How can I convert a csv file to lowercase or uppercase maintaining it's structure using powershell?

  9. 9

    How to convert text file containing double quotes to csv format using powershell

  10. 10

    Convert JSON file to a CSV file using R

  11. 11

    Convert an XML file to CSV file using java

  12. 12

    Remove Columns from CSV File Using Powershell

  13. 13

    Appending a CSV file to end of another, using Powershell

  14. 14

    How to convert a text file to CSV using Bash

  15. 15

    How to convert a CSV file using a PHP script

  16. 16

    Unable to convert JSON file to CSV using Python

  17. 17

    Convert Resultset to CSV file using Java

  18. 18

    Using PHP how to convert csv file

  19. 19

    How to convert XML file to CSV using Perl?

  20. 20

    convert a text file to csv using shell script

  21. 21

    How to convert a CSV file using a PHP script

  22. 22

    How to convert a text file to CSV using Bash

  23. 23

    Unable to convert JSON file to CSV using Python

  24. 24

    Convert .csv file to .ods using terminal

  25. 25

    PowerShell - Convert CSV to XLSX

  26. 26

    Create a CSV file from a specific log file using shell script

  27. 27

    C# convert csv to xls (using existing csv file)

  28. 28

    How to convert .xls to .csv using Powershell without Excel installed

  29. 29

    How to convert the csv to lower case using Powershell script

HotTag

Archive